Skip to content

perf(parquet): batch plain BYTE_ARRAY encoding - #1193

Draft
fallintoplace wants to merge 1 commit into
apache:mainfrom
fallintoplace:perf/parquet-plain-byte-array-batch
Draft

perf(parquet): batch plain BYTE_ARRAY encoding#1193
fallintoplace wants to merge 1 commit into
apache:mainfrom
fallintoplace:perf/parquet-plain-byte-array-batch

Conversation

@fallintoplace

Copy link
Copy Markdown
Contributor

Rationale for this change

PlainByteArrayEncoder.Put currently calls PutByteArray for every value. Each call checks sink capacity, writes the 4-byte length, then writes the payload. Large batches repeat that sink work for every value.

What changes are included in this PR?

  • Calculate the encoded batch size in one pass.
  • Reserve the sink capacity once.
  • Write lengths and payloads directly into the reserved region.
  • Add benchmarks for Put and all-valid PutSpaced with 1,048,576 values at three widths.

The single-value PutByteArray path is unchanged. PutSpaced benefits through its existing valid-run batching.

Medians from 6 runs on an Apple M1 Pro were:

method value width main this PR change
Put 4 bytes 10.25 ms 3.85 ms -62.4%
Put 16 bytes 11.00 ms 4.38 ms -60.2%
Put 64 bytes 13.77 ms 7.00 ms -49.2%
PutSpaced 4 bytes 10.33 ms 3.95 ms -61.8%
PutSpaced 16 bytes 11.13 ms 4.44 ms -60.1%
PutSpaced 64 bytes 13.54 ms 7.45 ms -45.0%

Allocation counts are unchanged.

Are these changes tested?

  • go test ./parquet/... -count=1
  • go test ./parquet/internal/encoding -run "^$" -bench "^BenchmarkEncodePlainByteArray$" -benchmem -count=6

The existing encoding suite covers plain BYTE_ARRAY round trips and spaced validity patterns.

Are there any user-facing changes?

No. The encoded Parquet bytes and public API are unchanged.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant