Skip to content

perf(arrow/compute): take chunked binary values without concatenation - #1199

Draft
fallintoplace wants to merge 1 commit into
apache:mainfrom
fallintoplace:perf/compute-chunked-binary-take
Draft

perf(arrow/compute): take chunked binary values without concatenation#1199
fallintoplace wants to merge 1 commit into
apache:mainfrom
fallintoplace:perf/compute-chunked-binary-take

Conversation

@fallintoplace

Copy link
Copy Markdown
Contributor

What

Chunked string and binary take currently concatenates all value chunks before selecting rows.

This adds a chunk-aware path for:

  • STRING
  • BINARY
  • LARGE_STRING
  • LARGE_BINARY

The new path resolves global indices to source chunks and copies only selected values. It keeps one output chunk per index chunk and preserves null and bounds behavior.

Benchmark

64 chunks x 4096 rows, 32-byte values, random indices, Apple M1 Pro:

  • 1% selected: 0.87 ms / 9.6 MB -> 0.08 ms / 0.11 MB
  • 10% selected: 2.36 ms / 10.4 MB -> 1.43 ms / 0.96 MB

The benchmark covers 8 and 64 chunks, 1% / 10% / 50% / 100% selection, and all four types.

Tests

  • go test ./arrow/compute/... -count=1

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant