Skip to content

perf(arrow/csv): avoid materializing rows before writing - #1190

Draft
fallintoplace wants to merge 1 commit into
apache:mainfrom
fallintoplace:perf/csv-stream-rows
Draft

perf(arrow/csv): avoid materializing rows before writing#1190
fallintoplace wants to merge 1 commit into
apache:mainfrom
fallintoplace:perf/csv-stream-rows

Conversation

@fallintoplace

@fallintoplace fallintoplace commented Aug 12, 2026

Copy link
Copy Markdown
Contributor

Rationale for this change

Writer.Write currently converts columns to strings, then copies every string header into a row-major [][]string before calling WriteAll. This allocates one []string for every row. Large record batches spend a lot of memory on this temporary matrix.

What changes are included in this PR?

  • Keep the converted strings in their existing column slices.
  • Reuse one row-sized []string while writing each CSV record.
  • Preserve the existing flush and error behavior.

For the existing 1,000-row, 16-column benchmark, medians from 6 runs on an Apple M1 Pro were:

main this PR change
time/op 4.78 ms 4.61 ms -3.5%
B/op 2,292,279 2,012,183 -12.2%
allocs/op 41,514 40,515 -2.4%

This removes 999 allocations per call for 1,000-row batches.

Are these changes tested?

  • go test ./arrow/... -count=1
  • go test ./arrow/csv -run "^$" -bench "^BenchmarkWrite$" -benchmem -count=6

Are there any user-facing changes?

No. The generated CSV output is unchanged.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant