Skip to content

perf(arrow/array): bulk append list nulls and empty values - #1187

Draft
fallintoplace wants to merge 1 commit into
apache:mainfrom
fallintoplace:perf/arrow-list-bulk-appends
Draft

perf(arrow/array): bulk append list nulls and empty values#1187
fallintoplace wants to merge 1 commit into
apache:mainfrom
fallintoplace:perf/arrow-list-bulk-appends

Conversation

@fallintoplace

Copy link
Copy Markdown
Contributor

What

  • reserve List and ListView batches once
  • update parent validity in bulk
  • append repeated offsets and zero sizes through the unsafe builder path
  • cover List, LargeList, ListView, and LargeListView

These bulk APIs currently call the scalar append path once per list slot. Empty and null variable-sized lists do not append child values, so only the parent validity and dimensions need updating.

This follows the batching shape used by the C++ list builders:

https://github.com/apache/arrow/blob/485499fd02ea2b0c323d67871fbe96aae4232504/cpp/src/arrow/array/builder_nested.h#L138-L160

For ListView, this keeps Go's current physical representation by repeating the current child offset with a zero size.

Benchmark

Apple M1 Pro, Go 1.26.3. Representative medians from three to five runs with 65,536 slots:

Builder Operation Before After Change
List nulls 848 us 201 us -76%
List empty 848 us 197 us -77%
LargeList nulls 1.08 ms 241 us -78%
LargeList empty 1.05 ms 245 us -77%
ListView nulls 1.35 ms 414 us -69%
ListView empty 1.24 ms 415 us -67%
LargeListView nulls 1.47 ms 509 us -65%
LargeListView empty 1.50 ms 505 us -66%

Regular-list allocations drop from 45 to 18 per batch. List-view allocations drop from 68 to 25. The n=1 cases stay within benchmark noise.

Tests

  • go test ./arrow/array -count=1
  • go test -race ./arrow/array -run 'TestListBuilderBulkAppendNullsAndEmptyValues|TestList|TestLargeList' -count=1
  • go vet ./arrow/array
  • go test -p 1 ./...

@fallintoplace
fallintoplace marked this pull request as draft August 12, 2026 13:07
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant