Skip to content

perf(compute): batch contiguous filter take indices - #1197

Draft
fallintoplace wants to merge 1 commit into
apache:mainfrom
fallintoplace:perf/compute-filter-take-runs
Draft

perf(compute): batch contiguous filter take indices#1197
fallintoplace wants to merge 1 commit into
apache:mainfrom
fallintoplace:perf/compute-filter-take-runs

Conversation

@fallintoplace

Copy link
Copy Markdown
Contributor

Summary

  • Add a range writer for uint16 and uint32 take indices.
  • Use it when a filter block or bitmap run is fully selected.
  • Keep the scalar path for mixed blocks.
  • Add benchmarks for dense, clustered, alternating, short, and nullable filters.

Benchmark

On an Apple M1 Pro, the 1M-row all-selected case improved from about 3.0 ms to 0.5 ms locally. Fragmented filters stayed close to the baseline.

Tests

  • go test ./arrow/compute/internal/kernels ./arrow/compute
  • go test -race ./arrow/compute/internal/kernels ./arrow/compute
  • Full package test run passed with the local Parquet test data checkout, excluding the CSV example that needs a separate Arrow CSV fixture.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant