Repository navigation
bench: criterion suite for the extension hot paths - #1474
Open
duncanista wants to merge 34 commits into
Open
duncanista wants to merge 34 commits into
duncanista wants to merge 34 commits into
Conversation
…act tests that pass today
Codex Review Summary: ✅ CompletedThis comment shows the latest Codex review activity on this pull request.
ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings. |
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 341f23b39d
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
duncanista
changed the title
- bench-docker-arm64.sh: pass the cargo arguments to the container shell as positional parameters, so criterion filters are not re-parsed. - bench-compare.sh: make a relative CARGO_TARGET_DIR absolute before cd, so cargo, critcmp and the baseline notes use the same directory. - bench-compare.sh: record a dirty marker and a hash of the uncommitted diff in the baseline notes. - benches/README.md: drop the old branch name.
- dogstatsd: advance the metric name every 100 lines so n lines give min(n, 5000) contexts - traces: set BASE_NS to 2026-10-27T12:00:00Z and encode meta/metrics in key order - events: pad the SQS body to the exact size and put the LWA deadline 60 s from now - telemetry: emit extension logs in the logger::Formatter JSON shape
- fault_injection_test: measure the /next budget from when the INVOKE was queued - harness: shutdown_cleanly asserts the exit came within the SHUTDOWN deadline
- write_corpus: 600 events for the JSON shapes so each POST fits the 1 MiB maxBytes - traces: TRACE_HEADERS matches the reference payload's 100 traces
- oversized trace: wait for /next against the 10 s INVOKE deadline, and assert the rejected trace is never sent again after the rejection is lifted - lifecycle: read the intake before SHUTDOWN so the final flush cannot hide a missing end-of-invocation flush
Keep the #1467 dependency pins without ddsketch-agent, regenerate Cargo.lock and LICENSE-3rdparty.csv, and list libddwaf-sys in the cargo-machete ignore list: it is a dependency only to enable its aws-lc-rs feature.
…modes in blackhole
…/cargo-machete/feature
…-fixtures/feature
…tdown deadline in shutdown()
…TDOWN, deliver telemetry only after an accepted subscribe
…t-harness/feature
…h-suite/feature Adapt the benches to what the merge brings in from main and the fixtures: - init: the fixtures build the enrichment tags with enrich_tags(), not ENRICH_TAGS. - traces: the libdatadog bump (#1467) renamed obfuscate_span to obfuscate_pb_span, HttpConfig.remove_paths_with_digits to remove_path_digits, and new_without_connection_pooling to new_periodic. tag_replace_rules is now built by startup::build_tag_replace_rules, made doc(hidden) pub so the bench builds it as production does.
duncanista
added a commit
that referenced
this pull request
Nov 5, 2026
**TL;DR:** One shared set of generated test inputs: Telemetry API batches, Lambda log lines, traces, DogStatsD packets and trigger events. **Why:** The fault-injection harness (#1473) and the benchmarks (#1474) need the same realistic inputs. Today each test builds its own by hand. The extension does not change. ## Overview Jira: [SVLS-9977](https://datadoghq.atlassian.net/browse/SVLS-9977) Stacked on #1471 (`jordan.gonzalez/cargo-machete/feature`). This PR does not need the CI PRs below it. It sits on this stack so that the bench PR can build on both. ### Why this PR We are adding test tools for the extension: - criterion benchmarks for the hot paths (#1474) - a fault-injection harness that runs the real `bottlecap` binary against a fake Lambda API and an intake that fails on demand (#1473) - later, a local Lambda simulator All of them need the same realistic inputs: Telemetry API batches, Lambda log lines, trace payloads, DogStatsD packets and trigger events. Today each test builds its own inputs by hand. This PR adds one small set of generators for those inputs, so every tool tests and measures the same shapes. Large inputs, such as a 220 KB SQS batch, are generated at test time instead of being checked into git. On its own, this PR only adds test code. The extension does not change. The only non-test change is 9 lines that make the existing `stitch` module `#[doc(hidden)] pub`, so integration tests and benches can call it. What to check here: that the generated shapes match what Lambda sends (for example the split-record format that the stitcher reassembles since #1415), and the self-test in `tests/fixtures_corpus.rs`. ### What is in it `bottlecap/tests/fixtures/` generates the inputs that the benches, the fault-injection harness and the Lambda simulator share, so no large blob is in git: - `telemetry`: Telemetry API batches (platform events, function logs in 3 formats, split records for the stitcher, `runtimeDone` with or without metrics, `logsDropped`, unknown events); - `logs`: Lambda text and JSON lines, one OOM line per runtime; - `traces`: synthetic traces and the 2000-span v0.4 msgpack payload, at a fixed base time; - `dogstatsd`: datagrams for 50 metric names x 100 endpoints; - `events`: the checked-in trigger payloads plus a 220 KB SQS and a 100-record DynamoDB event. `tests/fixtures_corpus.rs` checks the generators (for example, a split record stitches back). It needs `extension::telemetry::stitch` from the integration tests, so the module and `FragmentBuffer` become `#[doc(hidden)] pub`. No behavior change. Size: 13 core lines, 1058 lines in total. ## Testing - applies on the previous PR of the stack: pass - `cargo fmt --all -- --check`: pass - `cargo clippy --workspace --all-targets --features default -- -D warnings`: pass - `cargo clippy --workspace --all-targets --features default,test-mode -- -D warnings`: pass - `cargo nextest run --workspace`: pass [SVLS-9977]: https://datadoghq.atlassian.net/browse/SVLS-9977?atlOrigin=eyJpIjoiNWRkNTljNzYxNjVmNDY3MDlhMDU5Y2ZhYzA5YTRkZjUiLCJwIjoiZ2l0aHViLWNvbS1KU1cifQ
Base automatically changed from
jordan.gonzalez/fault-harness/feature
to
main
November 6, 2026 20:32
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: e8c95bd0b6
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
Contributor
There was a problem hiding this comment.
🟡 Changes recommended
The trace benchmark filters out every chunk, while the initialization benchmark drops clients before RSS sampling and includes teardown in its timing, making the performance and memory comparisons misleading.
4 open issues
What changed in this PR
Adds Criterion benchmarks for the Lambda extension’s hot paths, using shared fixtures from the preceding test work to support before-and-after performance comparisons. Production changes expose existing helpers without changing behavior.
Changes:
- Adds six benchmark targets covering logs, traces, invocation parsing, enhanced metrics, configuration, and initialization.
- Adds baseline comparison tooling and a Linux arm64 Docker runner.
- Documents measurement practices, initial results, and benchmark coverage.
| File | Description |
|---|---|
scripts/bench-docker-arm64.sh |
Runs benchmarks in an isolated Linux arm64 container. |
scripts/bench-compare.sh |
Saves, compares, exports, and lists baselines. |
bottlecap/src/startup.rs |
Exposes tag-replacement rule construction. |
bottlecap/src/proc/mod.rs |
Exposes path-based readers for fixture benchmarks. |
bottlecap/src/logs/lambda/processor.rs |
Exposes the OOM detection helper. |
bottlecap/Cargo.toml |
Adds Criterion and registers six benchmark targets. |
bottlecap/Cargo.lock |
Locks Criterion and its dependencies. |
bottlecap/benches/traces.rs |
Benchmarks trace decoding, processing, stats, and flushing. |
bottlecap/benches/README.md |
Documents execution, comparisons, coverage, and baseline results. |
bottlecap/benches/logs.rs |
Benchmarks log processing, stitching, batching, and compression. |
bottlecap/benches/invocation.rs |
Benchmarks trigger identification and payload handling. |
bottlecap/benches/init.rs |
Measures initialization components and client construction. |
bottlecap/benches/enhanced_metrics.rs |
Benchmarks resource readers and metric tag construction. |
bottlecap/benches/config.rs |
Benchmarks configuration loading and rule compilation. |
bottlecap/benches/common/mod.rs |
Shares allocator, runtime, configuration, and benchmark setup. |
bottlecap/benches/baselines/mac-main.txt |
Records initial baseline metadata. |
.gitignore |
Excludes baseline exports and arm64 build output. |
Review effort: Balanced
- bench-docker-arm64.sh writes the baseline-<name>.txt note (machine, container, load, profile, commit) that bench-compare.sh compare prints; fix its BENCH_FAST profile label. - bench-compare.sh records every CARGO_PROFILE_BENCH_* override in the note. - init: read RSS from /proc/self/status on Linux, so images without ps work. Keep what each step builds alive, and drop the warm-build clients outside the measurement. - traces: the regex require filter matches the fixture service; assert that every span is kept. - The shared setup and the config bench take the function identity from the fixtures and do not set the variables Lambda gives only to the runtime (AWS_EXECUTION_ENV, LAMBDA_TASK_ROOT and others). The config benches drop their results outside the measurement. - README: per-element time is the batch time divided by the count, the FIPS command needs --no-default-features, and the numbers recorded before these fixes are marked pre-fix.
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: fc8831c964
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
…fuse emulated arm64 - bench-compare.sh save: capture the git revision before cargo bench, and flag the note when the tree changes during the run. Record the cargo arguments in the note. - save and bench-docker-arm64.sh: remove every result and the note saved under the same baseline name before the run, so a baseline holds one run. Reject names that are not plain directory names. - bench-docker-arm64.sh: stop when the Docker daemon is not arm64 (QEMU or binfmt emulation). BENCH_ALLOW_EMULATION=1 runs anyway and marks the note as emulated. - profile note: also flag CARGO_PROFILE_RELEASE_* and RUSTFLAGS overrides, which the bench profile inherits. - benches/common: setup_env removes AWS_EXECUTION_ENV and the other runtime-only variables that Lambda does not give to extensions.
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 1db5865481
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
…s, drop numbers - scripts/bench-note.sh holds the baseline note for bench-compare.sh and bench-docker-arm64.sh: name check, removal of an older baseline, git revision before the run, post-run edit check, profile and Rust flag overrides, machine and container, and the note file. The Docker run sources it inside the container, and a second container marks the note when a tracked file changed during the run. - export copies the note next to the JSON as <file>.txt. - bench-docker-arm64.sh checks the image for bash, and apt-get when it has no protoc, before the run; a custom RUST_IMAGE must be Debian-based. - logs benches over a Telemetry POST use LogKind::max_events_per_post, the cap the fixture corpus uses: 600 JSON events stay under the 1 MiB maxBytes. Throughput uses the count per shape. - benches/README.md no longer has measured numbers: the findings table maps review items to bench ids, and points to the Confluence page and the bench CI workflow for numbers. benches/baselines/mac-main.txt is removed.
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: f13318dfa1
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
TL;DR: Criterion benchmarks for the hot paths: log processing, trace processing, invocation parsing, enhanced metrics, config load and init.
Why: The extension runs on about 0.07 vCPU at 128 MB, so per-line and per-span costs show up in the customer's billed duration. We need before and after numbers for each performance fix.
Overview
Jira: SVLS-9977
The extension runs at about 0.07 vCPU at 128 MB, so per-line, per-span and per-invocation
costs show up in billed duration. This PR adds 6 criterion benches (
bottlecap/benches/):logs,traces,invocation,enhanced_metrics,config,init. Benches namedalt_*measure an alternative from the review, as the target for a fix.
bottlecap/benches/README.md: what each bench measures, how to run and compare it, noisecontrol, and a map from each review item to its bench id. It has no measured numbers: the
investigation numbers are in the Confluence page "Bottlecap review: combined findings", and
the bench CI workflow (ci: informational benchmark workflow #1475) has the current ones. No baseline is committed.
scripts/bench-compare.sh: save, compare and export criterion baselines with critcmp.scripts/bench-docker-arm64.sh: the suite in a 2-CPU Linux arm64 container (Debian-basedimage). Both source
scripts/bench-note.sh, which writes the note of each baseline: themachine, the profile and Rust flag overrides, the git revision before the run, a mark when a
tracked file changed during the run, and the cargo arguments.
exportcopies the note nextto the JSON.
tests/fixtures:LogKind::max_events_per_post(), the most events one Telemetry POST holds(600 for the JSON shapes, under the 1 MiB
maxBytes). The corpus and the logs benches use it.Cargo.toml:criterion0.5 as a dev-dependency (no plotters, no rayon), the 6[[bench]]targets, and
bench = falseon the lib and binaries socargo benchruns only criterion.#[doc(hidden)] pubso the benches can reach them(
is_oom_error, the/procreaders*_from_path,startup::build_tag_replace_rules).No behavior change.
.gitignore: the Linux arm64 target dir.Size: 3281 insertions and 20 deletions in total. Production code: 32 changed lines in
src/.Scripts: 377 lines. The rest is benches, fixtures, docs and
Cargo.lock.Testing
cargo fmt --all -- --check: passcargo metadata --locked: passcargo clippy --workspace --all-targets --features default -- -D warnings: passcargo clippy --workspace --all-targets --features default,test-mode -- -D warnings: passcargo nextest run --workspace: passcargo bench --no-run(LTO off, 16 codegen units): passcargo bench --no-runandcargo bench -- --test, production profile: passcargo test --test fixtures_corpus: passcargo deny --locked check: passcargo machete: passdd-rust-license-tool check: pass-x), andbash -non/bin/bash3.2: passcargo,critcmpanddocker(notes, overrides,edit during the run, export with the note, image check): pass
bench-docker-arm64.shwith a stub Debian-based image, and the Alpineimage check: pass
cargo clippy --workspace --all-targets --no-default-features --features fips -- -D warnings: pass. It cannot run on macOS (libddwaf-sys load error).