Skip to content

fix: avoid OSS multipart limit in Parquet writes - #713

Open
jerry-024 wants to merge 1 commit into
apache:mainfrom
jerry-024:fix/opendal-writer-chunk-size
Open

fix: avoid OSS multipart limit in Parquet writes#713
jerry-024 wants to merge 1 commit into
apache:mainfrom
jerry-024:fix/opendal-writer-chunk-size

Conversation

@jerry-024

@jerry-024 jerry-024 commented Aug 14, 2026

Copy link
Copy Markdown
Contributor

Purpose

Paimon Rust previously created OpenDAL writers without an explicit chunk size. Parquet writes use OpenDAL FuturesAsyncWriter, which buffers 256 KiB at a time. Because the minimum OSS multipart part size is 100 KiB, OpenDAL forwarded each 256 KiB buffer as a separate part instead of coalescing it.

OSS allows at most 10,000 parts per multipart upload, which limited a single Parquet file to about 2.44 GiB before the upload exceeded the part-count limit.

Brief change log

  • Create the OpenDAL writer with writer_with.
  • Set the writer chunk size to 8 MiB so OpenDAL coalesces Parquet writes into larger multipart parts.
  • Increase the resulting multipart size ceiling from about 2.44 GiB to about 78.125 GiB.

Tests

  • cargo fmt --all -- --check
  • The targeted FileIO write/read test was blocked locally by the existing paimon-vindex-core use of the unstable stdarch_neon_f16 feature on aarch64 Rust 1.92.0; CI will run the repository test suite.

API and Format

No public API or storage format changes.

Documentation

No documentation changes are required.

@jerry-024 jerry-024 changed the title fix: configure OpenDAL writer chunk size fix: avoid OSS multipart limit in Parquet writes Aug 14, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant