refactor(benchmarks): isolate enqueue timing in data-pipeline by BridgeAR · Pull Request #2499 · DataDog/libdatadog

BridgeAR · 2026-09-09T22:44:41Z

What does this PR do?

Measure trace-buffer enqueue throughput with an empty buffer sized for each iteration. Sender threads prepare before timing, start together, and flush after timing. Rename the benchmark group to trace_buffer_enqueue.

Motivation

Exporter scheduling previously determined whether sends queued spans or returned BatchFull. Fresh runs dropped 62–73% of spans, so identical code measured different mixtures of paths.

The revised normal-mode medians spanned 444–470 µs across seven fresh runs, a 5.8% range.

Additional Notes

Benchmark-only; no production code changes. An RAII completion guard ensures sender failures propagate instead of hanging the benchmark.

How to test the change?

cargo nextest run -p libdd-data-pipeline --lib
cargo clippy -p libdd-data-pipeline --all-targets -- -D warnings
Run all 12 trace_buffer benchmark cases.

The broader package suite passed 209 of 214 tests. Five integration snapshot tests require the CI-only /snapshots fixture.

github-actions · 2026-09-09T22:46:29Z

📚 Documentation Check Results

⚠️ 1304 documentation warning(s) found

📦 `libdd-data-pipeline` - 1304 warning(s)

Updated: 2026-09-14 15:53:05 UTC | Commit: 394875b | missing-docs job results

datadog-prod-us1-5 · 2026-09-09T22:48:21Z

✅ All CI checks and tests passed.

🎉 All green!

🧪 All tests passed
❄️ No new flaky tests detected

🎯 Code Coverage (details)
• Patch Coverage: 100.00%
• Overall Coverage: 77.74% (-0.01%)

_{This comment will be updated automatically if new data arrives.

🔗 Commit SHA: ffd814c | Docs | View more details | Give us feedback!}

github-actions · 2026-09-09T22:48:33Z

🔒 Cargo Deny Results

✅ No issues found!

📦 `libdd-data-pipeline` - ✅ No issues

Updated: 2026-09-14 15:54:37 UTC | Commit: 394875b | dependency-check job results

dd-octo-sts · 2026-09-09T23:08:39Z

Artifact Size Benchmark Report

aarch64-alpine-linux-musl

Artifact	Baseline	Commit	Change
/aarch64-alpine-linux-musl/lib/libdatadog_profiling.a	96.01 MB	96.01 MB	0% (0 B) 👌
/aarch64-alpine-linux-musl/lib/libdatadog_profiling.so	9.02 MB	9.02 MB	0% (0 B) 👌

aarch64-unknown-linux-gnu

Artifact	Baseline	Commit	Change
/aarch64-unknown-linux-gnu/lib/libdatadog_profiling.a	107.42 MB	107.42 MB	0% (0 B) 👌
/aarch64-unknown-linux-gnu/lib/libdatadog_profiling.so	12.19 MB	12.19 MB	0% (0 B) 👌

libdatadog-x64-windows

Artifact	Baseline	Commit	Change
/libdatadog-x64-windows/debug/dynamic/datadog_profiling_ffi.dll	29.04 MB	29.04 MB	0% (0 B) 👌
/libdatadog-x64-windows/debug/dynamic/datadog_profiling_ffi.lib	96.08 KB	96.08 KB	0% (0 B) 👌
/libdatadog-x64-windows/debug/dynamic/datadog_profiling_ffi.pdb	191.77 MB	191.76 MB	-0% (-16.00 KB) 👌
/libdatadog-x64-windows/debug/static/datadog_profiling_ffi.lib	818.09 MB	818.09 MB	0% (0 B) 👌
/libdatadog-x64-windows/release/dynamic/datadog_profiling_ffi.dll	9.69 MB	9.69 MB	0% (0 B) 👌
/libdatadog-x64-windows/release/dynamic/datadog_profiling_ffi.lib	96.08 KB	96.08 KB	0% (0 B) 👌
/libdatadog-x64-windows/release/dynamic/datadog_profiling_ffi.pdb	27.51 MB	27.51 MB	0% (0 B) 👌
/libdatadog-x64-windows/release/static/datadog_profiling_ffi.lib	55.58 MB	55.58 MB	0% (0 B) 👌

libdatadog-x86-windows

Artifact	Baseline	Commit	Change
/libdatadog-x86-windows/debug/dynamic/datadog_profiling_ffi.dll	25.41 MB	25.41 MB	0% (0 B) 👌
/libdatadog-x86-windows/debug/dynamic/datadog_profiling_ffi.lib	97.58 KB	97.58 KB	0% (0 B) 👌
/libdatadog-x86-windows/debug/dynamic/datadog_profiling_ffi.pdb	197.08 MB	197.07 MB	-0% (-16.00 KB) 👌
/libdatadog-x86-windows/debug/static/datadog_profiling_ffi.lib	800.38 MB	800.38 MB	0% (0 B) 👌
/libdatadog-x86-windows/release/dynamic/datadog_profiling_ffi.dll	7.51 MB	7.51 MB	0% (0 B) 👌
/libdatadog-x86-windows/release/dynamic/datadog_profiling_ffi.lib	97.58 KB	97.58 KB	0% (0 B) 👌
/libdatadog-x86-windows/release/dynamic/datadog_profiling_ffi.pdb	29.63 MB	29.63 MB	0% (0 B) 👌
/libdatadog-x86-windows/release/static/datadog_profiling_ffi.lib	52.55 MB	52.55 MB	0% (0 B) 👌

x86_64-alpine-linux-musl

Artifact	Baseline	Commit	Change
/x86_64-alpine-linux-musl/lib/libdatadog_profiling.a	85.97 MB	85.97 MB	0% (0 B) 👌
/x86_64-alpine-linux-musl/lib/libdatadog_profiling.so	10.03 MB	10.03 MB	0% (0 B) 👌

x86_64-unknown-linux-gnu

Artifact	Baseline	Commit	Change
/x86_64-unknown-linux-gnu/lib/libdatadog_profiling.a	101.85 MB	101.85 MB	0% (0 B) 👌
/x86_64-unknown-linux-gnu/lib/libdatadog_profiling.so	12.25 MB	12.25 MB	0% (0 B) 👌

pr-commenter · 2026-09-14T12:35:51Z

Benchmarks

Comparison

Benchmark execution time: 2026-09-14 16:03:01

Comparing candidate commit ffd814c in PR branch BridgeAR/2026-09-09-trace-buffer-benchmark-stability with baseline commit e0a0134 in branch main.

📊 Benchmarking dashboard

Found 0 performance improvements and 0 performance regressions! Performance is the same for 24 metrics, 0 unstable metrics.

Explanation

This is an A/B test comparing a candidate commit's performance against that of a baseline commit. Performance changes are noted in the tables below as:

🟩 = significantly better candidate vs. baseline
🟥 = significantly worse candidate vs. baseline

We compute a confidence interval (CI) over the relative difference of means between metrics from the candidate and baseline commits, considering the baseline as the reference.

If the CI is entirely outside the configured SIGNIFICANT_IMPACT_THRESHOLD (or the deprecated UNCONFIDENCE_THRESHOLD), the change is considered significant.

Feel free to reach out to #apm-benchmarking-platform on Slack if you have any questions.

More details about the CI and significant changes

You can imagine this CI as a range of values that is likely to contain the true difference of means between the candidate and baseline commits.

CIs of the difference of means are often centered around 0%, because often changes are not that big:

---------------------------------(------|---^--------)-------------------------------->
                              -0.6%    0%  0.3%     +1.2%
                                 |          |        |
         lower bound of the CI --'          |        |
sample mean (center of the CI) -------------'        |
         upper bound of the CI ----------------------'

As described above, a change is considered significant if the CI is entirely outside the configured SIGNIFICANT_IMPACT_THRESHOLD (or the deprecated UNCONFIDENCE_THRESHOLD).

For instance, for an execution time metric, this confidence interval indicates a significantly worse performance:

----------------------------------------|---------|---(---------^---------)---------->
                                       0%        1%  1.3%      2.2%      3.1%
                                                  |   |         |         |
       significant impact threshold --------------'   |         |         |
                      lower bound of CI --------------'         |         |
       sample mean (center of the CI) --------------------------'         |
                      upper bound of CI ----------------------------------'

Candidate

Candidate benchmark details

Group 1

cpu_model	git_commit_sha	git_commit_date	git_branch
Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz	`ffd814c`	1789401016	BridgeAR/2026-09-09-trace-buffer-benchmark-stability

scenario	metric	min	mean ± sd	median ± mad	p75	p95	p99	max	peak_to_median_ratio	skewness	kurtosis	cv	sem	runs	sample_size
trace_buffer/1_senders/10us_delay	execution_time	56.222ms	56.495ms ± 0.155ms	56.464ms ± 0.071ms	56.536ms	56.846ms	56.951ms	57.196ms	1.30%	1.390	2.308	0.27%	0.011ms	1	200
trace_buffer/1_senders/10us_delay	throughput	15735.477op/s	15930.865op/s ± 43.586op/s	15939.372op/s ± 20.104op/s	15959.149op/s	15979.087op/s	15992.422op/s	16008.049op/s	0.43%	-1.371	2.227	0.27%	3.082op/s	1	200
trace_buffer/1_senders/1us_delay	execution_time	49.958ms	49.990ms ± 0.023ms	49.986ms ± 0.009ms	49.995ms	50.029ms	50.090ms	50.124ms	0.28%	2.696	10.806	0.05%	0.002ms	1	200
trace_buffer/1_senders/1us_delay	throughput	17955.607op/s	18003.558op/s ± 8.246op/s	18005.201op/s ± 3.065op/s	18008.124op/s	18012.534op/s	18014.209op/s	18015.264op/s	0.06%	-2.688	10.753	0.05%	0.583op/s	1	200
trace_buffer/1_senders/no_delay	execution_time	346.307µs	347.732µs ± 0.789µs	347.634µs ± 0.302µs	347.939µs	348.732µs	349.302µs	355.510µs	2.27%	5.328	47.356	0.23%	0.056µs	1	200
trace_buffer/1_senders/no_delay	throughput	2531577.028op/s	2588213.908op/s ± 5801.354op/s	2588931.715op/s ± 2249.068op/s	2591112.933op/s	2593838.935op/s	2596227.771op/s	2598850.388op/s	0.38%	-5.188	45.513	0.22%	410.218op/s	1	200
trace_buffer/2_senders/10us_delay	execution_time	56.478ms	57.008ms ± 0.231ms	56.992ms ± 0.189ms	57.190ms	57.416ms	57.490ms	57.532ms	0.95%	0.256	-0.798	0.40%	0.016ms	1	200
trace_buffer/2_senders/10us_delay	throughput	31286.755op/s	31575.172op/s ± 127.559op/s	31583.223op/s ± 104.673op/s	31683.097op/s	31754.768op/s	31794.116op/s	31871.019op/s	0.91%	-0.242	-0.805	0.40%	9.020op/s	1	200
trace_buffer/2_senders/1us_delay	execution_time	49.965ms	50.004ms ± 0.028ms	49.996ms ± 0.017ms	50.021ms	50.049ms	50.098ms	50.143ms	0.29%	1.550	3.524	0.06%	0.002ms	1	200
trace_buffer/2_senders/1us_delay	throughput	35897.356op/s	35996.965op/s ± 20.447op/s	36002.870op/s ± 12.365op/s	36011.535op/s	36020.181op/s	36022.917op/s	36025.532op/s	0.06%	-1.544	3.497	0.06%	1.446op/s	1	200
trace_buffer/2_senders/no_delay	execution_time	882.599µs	922.492µs ± 14.013µs	922.458µs ± 9.064µs	931.682µs	945.869µs	950.879µs	962.317µs	4.32%	0.013	0.043	1.52%	0.991µs	1	200
trace_buffer/2_senders/no_delay	throughput	1870484.987op/s	1951684.692op/s ± 29661.293op/s	1951308.990op/s ± 19169.904op/s	1970285.361op/s	2003911.786op/s	2021585.254op/s	2039431.271op/s	4.52%	0.080	0.049	1.52%	2097.370op/s	1	200
trace_buffer/4_senders/10us_delay	execution_time	57.114ms	57.401ms ± 0.125ms	57.392ms ± 0.079ms	57.474ms	57.605ms	57.785ms	57.894ms	0.88%	0.671	1.656	0.22%	0.009ms	1	200
trace_buffer/4_senders/10us_delay	throughput	62182.577op/s	62717.037op/s ± 136.664op/s	62726.836op/s ± 86.333op/s	62798.160op/s	62917.667op/s	63026.868op/s	63032.245op/s	0.49%	-0.650	1.609	0.22%	9.664op/s	1	200
trace_buffer/4_senders/1us_delay	execution_time	49.988ms	50.035ms ± 0.034ms	50.031ms ± 0.016ms	50.045ms	50.110ms	50.139ms	50.169ms	0.28%	1.550	2.615	0.07%	0.002ms	1	200
trace_buffer/4_senders/1us_delay	throughput	71757.049op/s	71950.174op/s ± 48.685op/s	71955.833op/s ± 22.486op/s	71981.911op/s	72008.057op/s	72016.460op/s	72017.170op/s	0.09%	-1.545	2.600	0.07%	3.443op/s	1	200
trace_buffer/4_senders/no_delay	execution_time	2.003ms	2.279ms ± 0.095ms	2.287ms ± 0.065ms	2.346ms	2.413ms	2.468ms	2.524ms	10.35%	-0.306	-0.169	4.14%	0.007ms	1	200
trace_buffer/4_senders/no_delay	throughput	1426545.637op/s	1582564.020op/s ± 66859.164op/s	1574173.045op/s ± 44532.666op/s	1631309.362op/s	1690515.316op/s	1749191.687op/s	1796932.636op/s	14.15%	0.530	0.119	4.21%	4727.657op/s	1	200
trace_buffer/8_senders/10us_delay	execution_time	57.465ms	57.844ms ± 0.320ms	57.722ms ± 0.164ms	58.063ms	58.430ms	58.663ms	59.212ms	2.58%	1.134	1.044	0.55%	0.023ms	1	200
trace_buffer/8_senders/10us_delay	throughput	121596.551op/s	124477.335op/s ± 685.357op/s	124736.113op/s ± 355.050op/s	125030.070op/s	125177.509op/s	125286.870op/s	125294.117op/s	0.45%	-1.105	0.913	0.55%	48.462op/s	1	200
trace_buffer/8_senders/1us_delay	execution_time	50.004ms	50.090ms ± 0.059ms	50.078ms ± 0.036ms	50.125ms	50.195ms	50.284ms	50.337ms	0.52%	1.342	2.322	0.12%	0.004ms	1	200
trace_buffer/8_senders/1us_delay	throughput	143036.961op/s	143740.848op/s ± 168.391op/s	143775.374op/s ± 104.031op/s	143864.853op/s	143940.298op/s	143970.110op/s	143987.289op/s	0.15%	-1.333	2.286	0.12%	11.907op/s	1	200
trace_buffer/8_senders/no_delay	execution_time	5.390ms	6.228ms ± 0.181ms	6.280ms ± 0.067ms	6.344ms	6.389ms	6.410ms	6.413ms	2.12%	-2.179	5.928	2.91%	0.013ms	1	200
trace_buffer/8_senders/no_delay	throughput	1122701.211op/s	1157205.324op/s ± 36172.068op/s	1146466.420op/s ± 12202.778op/s	1164207.358op/s	1223073.289op/s	1332238.232op/s	1335882.560op/s	16.52%	2.490	7.932	3.12%	2557.751op/s	1	200

scenario	metric	95% CI mean	Shapiro-Wilk pvalue	Ljung-Box pvalue (lag=1)	Dip test pvalue
trace_buffer/1_senders/10us_delay	execution_time	[56.473ms; 56.516ms] or [-0.038%; +0.038%]	None	None	None
trace_buffer/1_senders/10us_delay	throughput	[15924.824op/s; 15936.905op/s] or [-0.038%; +0.038%]	None	None	None
trace_buffer/1_senders/1us_delay	execution_time	[49.987ms; 49.993ms] or [-0.006%; +0.006%]	None	None	None
trace_buffer/1_senders/1us_delay	throughput	[18002.415op/s; 18004.701op/s] or [-0.006%; +0.006%]	None	None	None
trace_buffer/1_senders/no_delay	execution_time	[347.623µs; 347.841µs] or [-0.031%; +0.031%]	None	None	None
trace_buffer/1_senders/no_delay	throughput	[2587409.896op/s; 2589017.920op/s] or [-0.031%; +0.031%]	None	None	None
trace_buffer/2_senders/10us_delay	execution_time	[56.976ms; 57.040ms] or [-0.056%; +0.056%]	None	None	None
trace_buffer/2_senders/10us_delay	throughput	[31557.494op/s; 31592.851op/s] or [-0.056%; +0.056%]	None	None	None
trace_buffer/2_senders/1us_delay	execution_time	[50.000ms; 50.008ms] or [-0.008%; +0.008%]	None	None	None
trace_buffer/2_senders/1us_delay	throughput	[35994.131op/s; 35999.799op/s] or [-0.008%; +0.008%]	None	None	None
trace_buffer/2_senders/no_delay	execution_time	[920.550µs; 924.434µs] or [-0.211%; +0.211%]	None	None	None
trace_buffer/2_senders/no_delay	throughput	[1947573.922op/s; 1955795.462op/s] or [-0.211%; +0.211%]	None	None	None
trace_buffer/4_senders/10us_delay	execution_time	[57.384ms; 57.418ms] or [-0.030%; +0.030%]	None	None	None
trace_buffer/4_senders/10us_delay	throughput	[62698.097op/s; 62735.978op/s] or [-0.030%; +0.030%]	None	None	None
trace_buffer/4_senders/1us_delay	execution_time	[50.030ms; 50.039ms] or [-0.009%; +0.009%]	None	None	None
trace_buffer/4_senders/1us_delay	throughput	[71943.427op/s; 71956.921op/s] or [-0.009%; +0.009%]	None	None	None
trace_buffer/4_senders/no_delay	execution_time	[2.266ms; 2.292ms] or [-0.575%; +0.575%]	None	None	None
trace_buffer/4_senders/no_delay	throughput	[1573297.983op/s; 1591830.057op/s] or [-0.586%; +0.586%]	None	None	None
trace_buffer/8_senders/10us_delay	execution_time	[57.799ms; 57.888ms] or [-0.077%; +0.077%]	None	None	None
trace_buffer/8_senders/10us_delay	throughput	[124382.351op/s; 124572.319op/s] or [-0.076%; +0.076%]	None	None	None
trace_buffer/8_senders/1us_delay	execution_time	[50.082ms; 50.098ms] or [-0.016%; +0.016%]	None	None	None
trace_buffer/8_senders/1us_delay	throughput	[143717.511op/s; 143764.186op/s] or [-0.016%; +0.016%]	None	None	None
trace_buffer/8_senders/no_delay	execution_time	[6.202ms; 6.253ms] or [-0.404%; +0.404%]	None	None	None
trace_buffer/8_senders/no_delay	throughput	[1152192.224op/s; 1162218.425op/s] or [-0.433%; +0.433%]	None	None	None

Baseline

Baseline benchmark details

Group 1

cpu_model	git_commit_sha	git_commit_date	git_branch
Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz	`e0a0134`	1789387904	main

scenario	metric	min	mean ± sd	median ± mad	p75	p95	p99	max	peak_to_median_ratio	skewness	kurtosis	cv	sem	runs	sample_size
trace_buffer/1_senders/10us_delay	execution_time	56.352ms	56.669ms ± 0.225ms	56.610ms ± 0.128ms	56.792ms	57.089ms	57.454ms	57.531ms	1.63%	1.302	1.964	0.40%	0.016ms	1	200
trace_buffer/1_senders/10us_delay	throughput	15643.728op/s	15881.821op/s ± 62.804op/s	15898.295op/s ± 35.907op/s	15927.234op/s	15956.851op/s	15968.737op/s	15971.132op/s	0.46%	-1.276	1.864	0.39%	4.441op/s	1	200
trace_buffer/1_senders/1us_delay	execution_time	50.025ms	50.082ms ± 0.029ms	50.081ms ± 0.011ms	50.088ms	50.143ms	50.178ms	50.191ms	0.22%	0.998	2.158	0.06%	0.002ms	1	200
trace_buffer/1_senders/1us_delay	throughput	17931.579op/s	17970.649op/s ± 10.386op/s	17970.962op/s ± 4.078op/s	17975.688op/s	17988.556op/s	17990.602op/s	17990.934op/s	0.11%	-0.992	2.146	0.06%	0.734op/s	1	200
trace_buffer/1_senders/no_delay	execution_time	330.081µs	337.540µs ± 6.122µs	336.859µs ± 1.191µs	338.095µs	342.245µs	351.906µs	410.441µs	21.84%	8.841	99.848	1.81%	0.433µs	1	200
trace_buffer/1_senders/no_delay	throughput	2192762.846op/s	2667109.570op/s ± 42109.057op/s	2671738.558op/s ± 9479.922op/s	2681034.466op/s	2703960.366op/s	2723199.981op/s	2726600.293op/s	2.05%	-7.631	80.174	1.57%	2977.560op/s	1	200
trace_buffer/2_senders/10us_delay	execution_time	56.810ms	57.294ms ± 0.188ms	57.294ms ± 0.142ms	57.435ms	57.585ms	57.670ms	57.699ms	0.71%	-0.107	-0.655	0.33%	0.013ms	1	200
trace_buffer/2_senders/10us_delay	throughput	31196.207op/s	31417.430op/s ± 103.173op/s	31416.769op/s ± 77.909op/s	31494.968op/s	31597.444op/s	31627.937op/s	31684.311op/s	0.85%	0.120	-0.651	0.33%	7.295op/s	1	200
trace_buffer/2_senders/1us_delay	execution_time	50.062ms	50.117ms ± 0.037ms	50.109ms ± 0.017ms	50.123ms	50.203ms	50.215ms	50.277ms	0.33%	1.586	2.706	0.07%	0.003ms	1	200
trace_buffer/2_senders/1us_delay	throughput	35801.683op/s	35916.246op/s ± 26.814op/s	35921.450op/s ± 12.095op/s	35934.202op/s	35944.274op/s	35948.556op/s	35955.069op/s	0.09%	-1.581	2.685	0.07%	1.896op/s	1	200
trace_buffer/2_senders/no_delay	execution_time	873.054µs	915.773µs ± 16.391µs	916.647µs ± 9.257µs	925.461µs	936.953µs	942.679µs	1023.966µs	11.71%	1.041	8.583	1.79%	1.159µs	1	200
trace_buffer/2_senders/no_delay	throughput	1757871.344op/s	1966170.308op/s ± 34691.530op/s	1963677.808op/s ± 19741.706op/s	1983927.979op/s	2027447.353op/s	2054241.479op/s	2061727.091op/s	4.99%	-0.593	5.838	1.76%	2453.062op/s	1	200
trace_buffer/4_senders/10us_delay	execution_time	57.145ms	57.551ms ± 0.134ms	57.547ms ± 0.081ms	57.628ms	57.798ms	57.954ms	57.968ms	0.73%	0.441	0.609	0.23%	0.009ms	1	200
trace_buffer/4_senders/10us_delay	throughput	62103.257op/s	62553.449op/s ± 145.759op/s	62557.080op/s ± 87.816op/s	62642.674op/s	62763.106op/s	62831.219op/s	62997.214op/s	0.70%	-0.424	0.591	0.23%	10.307op/s	1	200
trace_buffer/4_senders/1us_delay	execution_time	50.108ms	50.171ms ± 0.045ms	50.162ms ± 0.018ms	50.181ms	50.246ms	50.332ms	50.495ms	0.66%	2.938	14.466	0.09%	0.003ms	1	200
trace_buffer/4_senders/1us_delay	throughput	71294.004op/s	71754.124op/s ± 64.520op/s	71767.900op/s ± 25.413op/s	71791.745op/s	71817.413op/s	71831.248op/s	71844.804op/s	0.11%	-2.917	14.270	0.09%	4.562op/s	1	200
trace_buffer/4_senders/no_delay	execution_time	2.208ms	2.377ms ± 0.052ms	2.379ms ± 0.037ms	2.414ms	2.458ms	2.482ms	2.512ms	5.59%	-0.210	-0.200	2.19%	0.004ms	1	200
trace_buffer/4_senders/no_delay	throughput	1433021.900op/s	1515184.124op/s ± 33484.371op/s	1513178.608op/s ± 23447.244op/s	1539433.585op/s	1569507.291op/s	1592574.162op/s	1630241.524op/s	7.74%	0.328	-0.075	2.20%	2367.703op/s	1	200
trace_buffer/8_senders/10us_delay	execution_time	57.602ms	57.920ms ± 0.211ms	57.854ms ± 0.113ms	58.037ms	58.318ms	58.549ms	58.670ms	1.41%	1.062	0.740	0.36%	0.015ms	1	200
trace_buffer/8_senders/10us_delay	throughput	122719.386op/s	124310.033op/s ± 451.138op/s	124451.188op/s ± 244.565op/s	124646.473op/s	124815.980op/s	124945.651op/s	124996.144op/s	0.44%	-1.045	0.686	0.36%	31.900op/s	1	200
trace_buffer/8_senders/1us_delay	execution_time	50.199ms	50.284ms ± 0.057ms	50.272ms ± 0.034ms	50.317ms	50.392ms	50.479ms	50.504ms	0.46%	1.335	2.316	0.11%	0.004ms	1	200
trace_buffer/8_senders/1us_delay	throughput	142563.267op/s	143187.739op/s ± 161.942op/s	143220.645op/s ± 96.318op/s	143302.755op/s	143388.848op/s	143416.865op/s	143428.501op/s	0.15%	-1.326	2.283	0.11%	11.451op/s	1	200
trace_buffer/8_senders/no_delay	execution_time	5.729ms	6.346ms ± 0.170ms	6.391ms ± 0.095ms	6.466ms	6.547ms	6.610ms	6.654ms	4.11%	-1.000	0.743	2.67%	0.012ms	1	200
trace_buffer/8_senders/no_delay	throughput	1082106.699op/s	1135399.016op/s ± 31258.594op/s	1126545.223op/s ± 16733.572op/s	1151492.825op/s	1196296.811op/s	1222927.204op/s	1256801.054op/s	11.56%	1.147	1.191	2.75%	2210.316op/s	1	200

scenario	metric	95% CI mean	Shapiro-Wilk pvalue	Ljung-Box pvalue (lag=1)	Dip test pvalue
trace_buffer/1_senders/10us_delay	execution_time	[56.638ms; 56.701ms] or [-0.055%; +0.055%]	None	None	None
trace_buffer/1_senders/10us_delay	throughput	[15873.117op/s; 15890.525op/s] or [-0.055%; +0.055%]	None	None	None
trace_buffer/1_senders/1us_delay	execution_time	[50.078ms; 50.086ms] or [-0.008%; +0.008%]	None	None	None
trace_buffer/1_senders/1us_delay	throughput	[17969.210op/s; 17972.088op/s] or [-0.008%; +0.008%]	None	None	None
trace_buffer/1_senders/no_delay	execution_time	[336.691µs; 338.388µs] or [-0.251%; +0.251%]	None	None	None
trace_buffer/1_senders/no_delay	throughput	[2661273.660op/s; 2672945.480op/s] or [-0.219%; +0.219%]	None	None	None
trace_buffer/2_senders/10us_delay	execution_time	[57.268ms; 57.320ms] or [-0.045%; +0.045%]	None	None	None
trace_buffer/2_senders/10us_delay	throughput	[31403.131op/s; 31431.729op/s] or [-0.046%; +0.046%]	None	None	None
trace_buffer/2_senders/1us_delay	execution_time	[50.111ms; 50.122ms] or [-0.010%; +0.010%]	None	None	None
trace_buffer/2_senders/1us_delay	throughput	[35912.530op/s; 35919.963op/s] or [-0.010%; +0.010%]	None	None	None
trace_buffer/2_senders/no_delay	execution_time	[913.501µs; 918.044µs] or [-0.248%; +0.248%]	None	None	None
trace_buffer/2_senders/no_delay	throughput	[1961362.395op/s; 1970978.220op/s] or [-0.245%; +0.245%]	None	None	None
trace_buffer/4_senders/10us_delay	execution_time	[57.532ms; 57.570ms] or [-0.032%; +0.032%]	None	None	None
trace_buffer/4_senders/10us_delay	throughput	[62533.248op/s; 62573.649op/s] or [-0.032%; +0.032%]	None	None	None
trace_buffer/4_senders/1us_delay	execution_time	[50.165ms; 50.178ms] or [-0.012%; +0.012%]	None	None	None
trace_buffer/4_senders/1us_delay	throughput	[71745.182op/s; 71763.065op/s] or [-0.012%; +0.012%]	None	None	None
trace_buffer/4_senders/no_delay	execution_time	[2.370ms; 2.384ms] or [-0.304%; +0.304%]	None	None	None
trace_buffer/4_senders/no_delay	throughput	[1510543.512op/s; 1519824.736op/s] or [-0.306%; +0.306%]	None	None	None
trace_buffer/8_senders/10us_delay	execution_time	[57.891ms; 57.950ms] or [-0.050%; +0.050%]	None	None	None
trace_buffer/8_senders/10us_delay	throughput	[124247.509op/s; 124372.556op/s] or [-0.050%; +0.050%]	None	None	None
trace_buffer/8_senders/1us_delay	execution_time	[50.276ms; 50.292ms] or [-0.016%; +0.016%]	None	None	None
trace_buffer/8_senders/1us_delay	throughput	[143165.296op/s; 143210.183op/s] or [-0.016%; +0.016%]	None	None	None
trace_buffer/8_senders/no_delay	execution_time	[6.323ms; 6.370ms] or [-0.371%; +0.371%]	None	None	None
trace_buffer/8_senders/no_delay	throughput	[1131066.875op/s; 1139731.156op/s] or [-0.382%; +0.382%]	None	None	None

yannham · 2026-09-14T13:23:35Z

-            tokio::time::sleep(Duration::from_millis(2)).await;
-            Ok(AgentResponse::Unchanged)
-        })
+        Box::pin(async { Ok(AgentResponse::Unchanged) })


Any reason to remove the 2 ms sleep here? I guess it's supposed to simulate at least a tiny bit of work.

yannham · 2026-09-14T13:34:17Z

+    // Keep the historical group name for pairwise CI comparisons; the custom timer isolates sender
+    // enqueue work.
    let mut group = c.benchmark_group("trace_buffer");
+    group.sampling_mode(SamplingMode::Flat);


Any reason for this addition? On one hand it does help CI times, but it's also a double edged sword, as it usually makes bench flakier.

yannham · 2026-09-14T13:45:46Z

+                            let input = Vec::from_iter(
                                (0..num_senders)
                                    .map(|_| (0..CHUNKS_PER_SENDER).map(|_| vec![make_span()]))
                                    .map(Vec::from_iter),


Unrelated, but while we're at it, this looks simpler (since make_span() is basically constant, we just clone the same bytes, this should be equivalent)

Suggested change

let input = Vec::from_iter(

(0..num_senders)

.map(|_| (0..CHUNKS_PER_SENDER).map(|_| vec![make_span()]))

.map(Vec::from_iter),

let input = vec![ vec![make_span(); CHUNKS_PER_SENDER]; num_senders];

The concurrent trace-buffer benchmark let exporter scheduling decide whether each send queued a span or hit BatchFull. Fresh runs dropped 62-73% of spans, so identical code measured a different mix of paths. Give each iteration an empty buffer sized for its workload, release ready sender threads inside a custom timer, and flush after measurement. Rename the series because it now measures successful enqueue throughput instead of a scheduler-dependent mix. Benchmark-only; no production code changes.

Pairwise benchmark analysis joins baseline and candidate scenarios by their full Criterion group name. Renaming the group leaves no comparable scenarios and makes analysis fail.

Criterion keeps regression analysis for shorter workloads and selects flat sampling when wall time requires it.

github-actions Bot added the data-pipeline label Sep 9, 2026

BridgeAR changed the title ~~bench(data-pipeline): isolate enqueue timing~~ refactor(benchmarks): isolate enqueue timing in data-pipeline Sep 9, 2026

BridgeAR marked this pull request as ready for review September 9, 2026 22:55

BridgeAR requested a review from a team as a code owner September 9, 2026 22:55

BridgeAR force-pushed the BridgeAR/2026-09-09-trace-buffer-benchmark-stability branch from 17fa829 to f9aaf1e Compare September 14, 2026 12:18

BridgeAR commented Sep 14, 2026

View reviewed changes

Comment thread libdd-data-pipeline/benches/trace_buffer.rs Outdated

yannham approved these changes Sep 14, 2026

View reviewed changes

BridgeAR added 4 commits September 14, 2026 11:50

ci(data-pipeline): preserve trace buffer benchmark series

d618e66

Pairwise benchmark analysis joins baseline and candidate scenarios by their full Criterion group name. Renaming the group leaves no comparable scenarios and makes analysis fail.

Apply suggestion from @BridgeAR

2a9b9cd

bench(data-pipeline): use adaptive trace buffer sampling

ffd814c

Criterion keeps regression analysis for shorter workloads and selects flat sampling when wall time requires it.

BridgeAR force-pushed the BridgeAR/2026-09-09-trace-buffer-benchmark-stability branch from effbd53 to ffd814c Compare September 14, 2026 15:51

Navigation Menu

Uh oh!

refactor(benchmarks): isolate enqueue timing in data-pipeline - #2499

refactor(benchmarks): isolate enqueue timing in data-pipeline#2499
BridgeAR wants to merge 4 commits into
mainfrom
BridgeAR/2026-09-09-trace-buffer-benchmark-stability

BridgeAR commented Sep 9, 2026

Uh oh!

github-actions Bot commented Sep 9, 2026 •

edited

Loading

Uh oh!

datadog-prod-us1-5 Bot commented Sep 9, 2026 •

edited by datadog-prod-us1-6 Bot

Loading

Uh oh!

github-actions Bot commented Sep 9, 2026 •

edited

Loading

Uh oh!

dd-octo-sts Bot commented Sep 9, 2026 •

edited

Loading

Uh oh!

Uh oh!

pr-commenter Bot commented Sep 14, 2026 •

edited

Loading

Explanation

More details about the CI and significant changes

Group 1

Group 1

Uh oh!

yannham Sep 14, 2026

Uh oh!

yannham Sep 14, 2026

Uh oh!

yannham Sep 14, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Uh oh!

Conversation

BridgeAR commented Sep 9, 2026

What does this PR do?

Motivation

Additional Notes

How to test the change?

Uh oh!

github-actions Bot commented Sep 9, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

📚 Documentation Check Results

📦 libdd-data-pipeline - 1304 warning(s)

Uh oh!

datadog-prod-us1-5 Bot commented Sep 9, 2026 • edited by datadog-prod-us1-6 Bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions Bot commented Sep 9, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

🔒 Cargo Deny Results

📦 libdd-data-pipeline - ✅ No issues

Uh oh!

dd-octo-sts Bot commented Sep 9, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Artifact Size Benchmark Report

Uh oh!

Uh oh!

pr-commenter Bot commented Sep 14, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Benchmarks

Comparison

Explanation

More details about the CI and significant changes

Candidate

Group 1

Baseline

Group 1

Uh oh!

yannham Sep 14, 2026

Choose a reason for hiding this comment

Uh oh!

yannham Sep 14, 2026

Choose a reason for hiding this comment

Uh oh!

yannham Sep 14, 2026

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

github-actions Bot commented Sep 9, 2026 •

edited

Loading

📦 `libdd-data-pipeline` - 1304 warning(s)

datadog-prod-us1-5 Bot commented Sep 9, 2026 •

edited by datadog-prod-us1-6 Bot

Loading

github-actions Bot commented Sep 9, 2026 •

edited

Loading

📦 `libdd-data-pipeline` - ✅ No issues

dd-octo-sts Bot commented Sep 9, 2026 •

edited

Loading

pr-commenter Bot commented Sep 14, 2026 •

edited

Loading