du: buffer output to files and pipes

[limadog9]

Buffer output to files and pipes, preserve immediate terminal output, and flush before diagnostics and on completion. Add tests for final totals and flush failures. A 10,000-file check reduced write calls from 10,001 to 7.

Validation: 108 du integration tests pass. Verified unchanged output with non-UTF-8 names, totals, NUL separators, and timestamps.

Closes #9146

[codspeed]

Merging this PR will improve performance by 12.58%

⚠️ This run is associated to a commit that was force-pushed

It won't be associated with its branch history. Be careful when comparing its performance to other runs.

⚠️ 4 benchmarks spent significant time in system calls

System calls cannot be consistently instrumented, so they are not included in the measure, which understates the real cost. Please switch to the Walltime instrument to accurately measure system calls.

Measurement and system calls

• du_all_wide_tree[(5000, 500)]
• du_deep_tree[(100, 3)]
• du_wide_tree[(5000, 500)]
• du_summarize_balanced_tree[(5, 4, 10)]

⚡ 1 improved benchmark
✅ 7 untouched benchmarks
⏩ 464 skipped benchmarks¹

Performance Changes

	Mode	Benchmark	BASE	HEAD	Efficiency
⚡ ⚠️	Simulation	du_all_wide_tree[(5000, 500)]	34.2 ms	30.4 ms	+12.58%

Tip

Curious why performance improved? Comment @codspeedbot explain why performance improved on this PR, or directly use the CodSpeed MCP with your agent.

---

_{Comparing limadog9:fix/du-buffer-output-9146 (139a237) with main (0ac1a5b)²}

Footnotes

1. 464 benchmarks were skipped, so the baseline results were used instead. If they were deleted from the codebase, click here and archive them to remove them from the performance reports. ↩

2. No successful run was found on main (aa74c61) during the generation of this report, so 0ac1a5b was used instead as the comparison base. There might be some changes unrelated to this pull request in this report. ↩

[sylvestre]

@codspeedbot fix this regression

[sylvestre]

why 2 KiB and not the default capacity?

[sylvestre]

with --verbose, the "ignored" lines printed by the traversal thread with println! now come before the buffered stats when stdout is a file, so the order differs from main.
could you please route them through the printer or flush before them? and please add a test with -v --exclude

[sylvestre]

this one passes on main too, no? it doesn't really test the buffering

[sylvestre]

stderr_does_not_contain("panicked") isn't needed, fails_with_code + stderr_contains is enough

[sylvestre]

codspeed reports a 13.86% regression, could you please check where it comes from?

[codspeed]

@codspeedbot fix this regression

The current head of this PR (139a237) already fixes this regression, so I didn't push any changes.

Cause

The report above was produced for an earlier commit, 1fa4ff9. That commit gave the output a BufWriter::with_capacity(64 * 1024, …) whenever stdout is not a terminal. The benchmarks write to a non-terminal stdout, so this 64 KiB buffer stayed allocated for the whole run. Every memory regression in the report matches it exactly:

Benchmark	Base	1fa4ff9	Δ
du_deep_tree[(100, 3)]	76.5 KB	140.5 KB	+64.0 KB
du_summarize_balanced_tree[(5, 4, 10)]	419.9 KB	483.9 KB	+64.0 KB
du_wide_tree[(5000, 500)]	1,010.5 KB	1,074.5 KB	+64.0 KB

Current head

139a237 cuts the buffer to 2 KiB. That should bring the extra peak memory down from 64 KB to about 2 KB: roughly +2.6% on du_deep_tree, and under 0.5% on the other benchmarks. That's still enough to batch redirected output into a few large writes.

I couldn't measure memory in my environment, so these memory numbers are estimates. CodSpeed hasn't benchmarked 139a237 in CI yet, and the next CI run will confirm them.

I did measure CPU simulation locally, main (aa74c61) against 139a237:

Benchmark	main	head	Change
du_all_wide_tree[(5000, 500)]	14.69 ms	12.94 ms	−11.9%
du_deep_tree[(100, 3)]	842 µs	806 µs	−4.3%
du_wide_tree[(5000, 500)]	6.87 ms	6.75 ms	−1.7%
du_summarize_balanced_tree[(5, 4, 10)]	4.83 ms	4.84 ms	~0%

The speedup from buffering is still there, and the time doesn't go up on any du benchmark.

If CI still shows a memory regression after it runs, comment @codspeedbot investigate the memory results and I'll look into it.

Summary

• Warming up the performance wizard...
• Analyze the reported regression — Report was for 1fa4ff9, which used a 64 KiB BufWriter (+64 KB peak memory). Current head 139a237 already reduced it to 2 KiB but has not been benchmarked yet.
• Measure memory + simulation of current head vs main — Simulation measured locally: head is the same or faster on every du benchmark. Memory could not be measured here; expected delta is +2 KiB
• Reply with findings — Reply with findings
• Handling comment — Comment handling finished successfully

_{Was this response helpful? React with 👍 or 👎 to help us improve CodSpeedBot.}