Skip to content

perf(parquet): encode RLE levels in batches - #1177

Draft
fallintoplace wants to merge 1 commit into
apache:mainfrom
fallintoplace:perf/parquet-batched-rle-levels
Draft

perf(parquet): encode RLE levels in batches#1177
fallintoplace wants to merge 1 commit into
apache:mainfrom
fallintoplace:perf/parquet-batched-rle-levels

Conversation

@fallintoplace

Copy link
Copy Markdown
Contributor

What changed

  • add a batch path for repetition and definition level encoding
  • scan repeated runs once instead of calling the scalar encoder for every level
  • keep the scalar path around literal boundaries
  • preserve the existing encoded bytes across split streaming calls
  • add level encoder and scalar-versus-batch benchmarks

Benchmarks

Apple M1 Pro:

RLE workload Before After Change
65,536 all defined 151,051 ns/op 31,143 ns/op -79%
65,536 mostly defined 324,228 ns/op 252,108 ns/op -22%
65,536 alternating 339,128 ns/op 341,149 ns/op +0.6%

The alternating case is the fallback path. Allocations remain at zero.

@fallintoplace
fallintoplace marked this pull request as draft August 11, 2026 22:14
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant