Skip to content

perf(arrow/array): reuse TableReader scratch slices - #1200

Draft
fallintoplace wants to merge 1 commit into
apache:mainfrom
fallintoplace:perf/array-table-reader-scratch
Draft

perf(arrow/array): reuse TableReader scratch slices#1200
fallintoplace wants to merge 1 commit into
apache:mainfrom
fallintoplace:perf/array-table-reader-scratch

Conversation

@fallintoplace

Copy link
Copy Markdown
Contributor

What

TableReader.Next allocated two column slices for every record batch.

This keeps the batch slice on TableReader and reads the current chunks again after finding the batch size. NewRecordBatch copies the input slice, so the scratch slice can be safely reused on the next call.

Benchmark

The benchmark reuses one reader across passes. It uses Int32 columns with 256 rows per chunk.

256 columns x 256 chunks:

  • 10.2 ms -> 3.0 ms
  • 3.76 MB -> 1.27 MB
  • 1024 -> 512 allocations

The benchmark covers 4, 32, and 256 columns with 32 and 256 chunks.

Tests

  • go test ./arrow/array -count=1
  • go test -race ./arrow/array -run TestTableReader -count=1

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant