Skip to content

perf(parquet): batch DELTA_BYTE_ARRAY encoding - #1179

Draft
fallintoplace wants to merge 1 commit into
apache:mainfrom
fallintoplace:perf/delta-byte-array-batching
Draft

perf(parquet): batch DELTA_BYTE_ARRAY encoding#1179
fallintoplace wants to merge 1 commit into
apache:mainfrom
fallintoplace:perf/delta-byte-array-batching

Conversation

@fallintoplace

@fallintoplace fallintoplace commented Aug 12, 2026

Copy link
Copy Markdown
Contributor

Summary

  • Batch DELTA_BYTE_ARRAY prefix lengths and suffixes in chunks of 256 values.
  • Reuse fixed scratch space for DELTA_LENGTH_BYTE_ARRAY lengths.
  • Preserve the previous value across batch boundaries and separate Put calls.
  • Add a boundary round-trip test and representative benchmarks.

Why

DELTA_BYTE_ARRAY was calling the prefix and suffix encoders once per value. Large pages therefore paid for many small calls and temporary length slices.

Apache Arrow C++ already uses fixed-size batching for this encoder. This change follows the same shape while keeping the existing Go encoding format and last-value behavior.

Correctness

  • The encoded layout is unchanged: prefix lengths first, followed by the suffix stream.
  • The first value still has a zero prefix.
  • Empty suffixes still work as before.
  • The new test compares one large Put call with Put calls split around a batch boundary, then decodes and compares every value.

Benchmark

64K values on an Apple M1 Pro, 1 second per sample, 3 samples:

Case Before After Change
prefix-heavy 2.09 ms/op 1.75 ms/op ~16% faster
low-prefix 1.68 ms/op 1.47 ms/op ~12% faster

Allocation counts stayed the same in both cases.

Checks

  • PARQUET_TEST_DATA=<parquet-testing-data> go test ./parquet/...
  • go test -race ./parquet/internal/encoding -count=1
  • git diff --check

No public API changes.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant