Skip to content

perf(arrow/array): avoid temporary buffers when concatenating - #1194

Draft
fallintoplace wants to merge 1 commit into
apache:mainfrom
fallintoplace:perf/array-concatenate-buffer-ranges
Draft

perf(arrow/array): avoid temporary buffers when concatenating#1194
fallintoplace wants to merge 1 commit into
apache:mainfrom
fallintoplace:perf/array-concatenate-buffer-ranges

Conversation

@fallintoplace

Copy link
Copy Markdown
Contributor

Summary

  • copy fixed-width values directly from source buffer ranges into the output buffer
  • avoid creating one temporary memory.Buffer for every input chunk
  • add a benchmark with a fixed number of values and increasing chunk counts

Benchmarks

Apple M1 Pro, 65,536 int64 values:

chunks old time new time delta old allocs new allocs
1 65.92 us 59.86 us ~ 7 5
8 48.58 us 51.12 us ~ 15 6
64 64.33 us 64.18 us ~ 71 6
1,024 154.88 us 97.89 us -36.80% 1,031 6
8,192 571.7 us 214.5 us -62.48% 8,199 6

At 1,024 chunks, B/op drops by 14.22%. At 8,192 chunks, it drops by 52.06%.

Tests

  • go test ./...

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant