Skip to content

perf(arrow/array): optimize bulk appends for struct builders - #1189

Draft
fallintoplace wants to merge 1 commit into
apache:mainfrom
fallintoplace:perf/arrow-struct-bulk-appends
Draft

perf(arrow/array): optimize bulk appends for struct builders#1189
fallintoplace wants to merge 1 commit into
apache:mainfrom
fallintoplace:perf/arrow-struct-bulk-appends

Conversation

@fallintoplace

Copy link
Copy Markdown
Contributor

Summary

  • reserve the struct and its children once for the full batch
  • update the parent validity bitmap and counters once
  • dispatch one bulk null or empty append to each child builder
  • preserve the existing Go behavior where null structs append null child values
  • add mixed nested-struct coverage and width-scaled benchmarks

This uses the same bulk-dispatch orientation as Arrow C++ StructBuilder, while keeping Go child-null semantics unchanged.

Benchmark

65,536 appended structs, representative medians on Apple M1 Pro:

Fields Operation Before After Change
1 nulls 457 us 245 us -46%
1 empty 639 us 301 us -53%
4 nulls 1.40 ms 0.82 ms -41%
4 empty 1.93 ms 1.07 ms -44%
16 nulls 5.27 ms 2.86 ms -46%
16 empty 6.96 ms 3.86 ms -45%
64 nulls 17.37 ms 11.20 ms -36%
64 empty 27.31 ms 15.25 ms -44%

Allocations per build also drop:

  • 1 field: 48 to 23
  • 4 fields: 132 to 53
  • 16 fields: 470 to 175
  • 64 fields: 1,846 to about 687

Single-value performance stays about the same.

Tests

  • go test -p 1 ./...
  • go test -race ./arrow/array -run TestStructBuilderBulkAppendNullsAndEmptyValues -count=1
  • go vet ./arrow/array

@fallintoplace
fallintoplace marked this pull request as draft August 12, 2026 13:45
@fallintoplace fallintoplace changed the title perf(arrow/array): bulk append struct nulls and empty values perf(arrow/array): optimize bulk appends for struct builders Aug 12, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant