Skip to content

perf(parquet): reuse DataPageV2 assembly buffer for eager writes - #1192

Draft
fallintoplace wants to merge 1 commit into
apache:mainfrom
fallintoplace:perf/parquet-datapage-v2-buffer
Draft

perf(parquet): reuse DataPageV2 assembly buffer for eager writes#1192
fallintoplace wants to merge 1 commit into
apache:mainfrom
fallintoplace:perf/parquet-datapage-v2-buffer

Conversation

@fallintoplace

Copy link
Copy Markdown
Contributor

Rationale for this change

buildDataPageV2 creates a new bytes.Buffer for every page when joining levels with encoded values. Pages without active dictionary encoding are written synchronously, so their assembly storage does not need to survive the call. Reallocating it for every page adds page-sized allocation churn.

What changes are included in this PR?

  • Reuse the column writer scratch buffer for eagerly written DataPageV2 pages.
  • Keep an owned buffer for pages retained while dictionary encoding is active.
  • Add a benchmark that writes 64 eager 64 KiB pages with uncompressed and Snappy codecs.

Medians from 6 runs on an Apple M1 Pro were:

codec metric main this PR change
uncompressed time/op 1.08 ms 650 us -39.6%
uncompressed B/op 4,547,602 383,763 -91.6%
uncompressed allocs/op 461 321 -30.4%
Snappy time/op 9.05 ms 8.88 ms -1.9%
Snappy B/op 3,027,460 489,403 -83.8%
Snappy allocs/op 459 325 -29.2%

Are these changes tested?

  • go test ./parquet/... -count=1
  • go test ./parquet/file -run "^$" -bench "^BenchmarkWriteDataPageV2Eager$" -benchmem -count=6

The Parquet suite includes multi-page DataPageV2 dictionary coverage for the retained-page path.

Are there any user-facing changes?

No. The Parquet output and public API are unchanged.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant