Skip to content

perf(parquet): encode spaced booleans without compaction - #1175

Open
fallintoplace wants to merge 2 commits into
apache:mainfrom
fallintoplace:perf/parquet-boolean-spaced-runs
Open

perf(parquet): encode spaced booleans without compaction#1175
fallintoplace wants to merge 2 commits into
apache:mainfrom
fallintoplace:perf/parquet-boolean-spaced-runs

Conversation

@fallintoplace

Copy link
Copy Markdown
Contributor

Rationale for this change

Plain Boolean spaced encoding currently counts valid bits, allocates a compacted bitmap, copies valid runs into it, and then copies that bitmap into the encoder buffer.

Short unaligned runs also create bitmap readers and writers for every copy. This is expensive for common nullable patterns.

What changes are included in this PR?

  • Append valid bitmap runs directly to the existing encoder buffer.
  • Accumulate the valid count while encoding instead of scanning validity first.
  • Write short runs directly to avoid unaligned bitmap-copy allocations.
  • Keep bulk bitmap copies for longer runs.
  • Share bitmap writer initialization between the Boolean input paths.
  • Add benchmarks for 1K, 64K, and 1M values across several null patterns.
  • Add tests for bitmap offsets, validity offsets, encoder-buffer boundaries, and repeated calls.

Apple M1 Pro results for 65,536 values with -cpu=1:

Validity pattern Before After Speedup
All valid 6.77 us 3.00 us 2.26x
1% null 224 us 182 us 1.23x
10% null 912 us 307 us 2.97x
Alternating 50% null 3.86 ms 394 us 9.79x
90% null 1.16 ms 95.8 us 12.1x
Clustered nulls 7.53 us 3.65 us 2.06x

For the short-run patterns:

  • 10% null: 847 KB and 13,109 allocations to 23 B and 0 allocations.
  • 50% null: 4.20 MB and 65,542 allocations to 14 B and 0 allocations.
  • 90% null: 735 KB and 11,469 allocations to 0 B and 0 allocations.

Are these changes tested?

Yes.

  • go test ./parquet/...
  • go test -race ./parquet/internal/encoding
  • go vet -composites=false ./parquet/internal/encoding
  • Cross-compiled the encoding tests for linux/amd64 and linux/s390x.

Are there any user-facing changes?

No.

@fallintoplace
fallintoplace marked this pull request as ready for review August 12, 2026 21:09
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant