feat: introduce metabench, a unified benchmarking framework

[Martin Taillefer (geeknoid)]

No description provided.

[github-actions]

⚠️ Manual proc-macro SemVer review required

cargo semver-checks intentionally does not analyse proc-macro-only targets. 1 of 3 publishing crate(s) require manual review either for their own procedural macro contract or because they directly consume a manually reviewed breaking release. Ordinary libraries retain their automated result, and review advances to the next dependency edge only when the current release is breaking.

Crate	Baseline	Baseline commit	This PR	Minimum required	Status
metabench	new crate	—	0.1.0	0.1.0	✅ ok
metabench_macros	new crate	—	0.1.0	—	⚠️ manual proc-macro review required
metabench_macros_impl	new crate	—	0.1.0	0.1.0	✅ ok

⚠️ metabench_macros — manual proc-macro review detail

`metabench_macros` is a proc-macro-only crate. cargo-semver-checks intentionally skips proc-macro targets because they have no supported library API surface. Review exported macro names, accepted input syntax, diagnostics, and generated output manually; build and test results do not establish public API SemVer compatibility.

Proc-macro API compatibility must be reviewed manually; successful builds and tests do not establish SemVer compatibility. This check is informational and does not block the merge.

View the check run

[Copilot]

🟡 Changes recommended

There are a few repo-convention and operational issues (coverage-gate test-module annotations, very deep committed fixture paths, and a more robust workflow invocation) that should be addressed before merging.

Once you've addressed the issues Copilot identified, you can request another Copilot review.

Pull request overview

Introduces metabench, a new unified benchmarking framework intended to run the same workload across Criterion, Gungraun/Callgrind, Linux perf, and allocation tracking, and integrates it into the workspace + CI.

Changes:

• Adds new metabench runtime crate plus metabench_macros / metabench_macros_impl proc-macro crates, with benches and integration tests.
• Adds extensive fixture-based parsing tests for Criterion/Gungraun/perf/allocation artifacts and a CI smoke-check for benchmark registration output.
• Updates workspace dependencies/lockfile and wires metabench into mutation-testing grouping, spellcheck dictionary, and Miri exclusions.

File summaries

File	Description
scripts/test-metabench-registration.sh	New CI script that asserts stable --list benchmark identities for metabench bench targets.
scripts/mutants.rs	Adds metabench crates to cargo-mutants test grouping.
README.md	Documents metabench as a primary crate in the repo.
crates/metabench/tests/report_api.rs	Integration tests for report wire format + IO behavior and invariants.
crates/metabench/tests/fixtures/artifacts/traversal/too_deep/d1/d2/d3/d4/d5/d6/d7/d8/d9/d10/d11/d12/d13/d14/d15/d16/d17/d18/d19/d20/d21/d22/d23/d24/d25/d26/d27/d28/d29/d30/d31/d32/d33/d34/d35/d36/d37/d38/d39/d40/d41/d42/d43/d44/d45/d46/d47/d48/d49/d50/d51/d52/d53/d54/d55/d56/d57/d58/d59/d60/d61/d62/d63/d64/d65/.keep	Depth-limit sentinel fixture for artifact traversal.
crates/metabench/tests/fixtures/artifacts/traversal/cycle/new/estimates.json	Traversal fixture: Criterion estimates file for cycle/symlink scenario.
crates/metabench/tests/fixtures/artifacts/traversal/cycle/new/benchmark.json	Traversal fixture: Criterion benchmark metadata for cycle/symlink scenario.
crates/metabench/tests/fixtures/artifacts/perf/unicode.jsonl	Perf fixture: unicode event name JSONL record.
crates/metabench/tests/fixtures/artifacts/perf/minimal.jsonl	Perf fixture: minimal valid perf JSONL record.
crates/metabench/tests/fixtures/artifacts/perf/malformed.jsonl	Perf fixture: malformed JSONL record.
crates/metabench/tests/fixtures/artifacts/perf/invalid_numeric.jsonl	Perf fixture: invalid numeric counter value.
crates/metabench/tests/fixtures/artifacts/perf/empty.jsonl	Perf fixture: empty file for “no records” error path.
crates/metabench/tests/fixtures/artifacts/perf/duplicate.jsonl	Perf fixture: duplicate event records.
crates/metabench/tests/fixtures/artifacts/gungraun/wrong_mapping/summary.json	Gungraun fixture: summary with out-of-range case mapping.
crates/metabench/tests/fixtures/artifacts/gungraun/minimal/summary.json	Gungraun fixture: minimal valid v6 summary.
crates/metabench/tests/fixtures/artifacts/gungraun/manifest_wrong_set/metabench-identities.json	Gungraun fixture: identity manifest with wrong set.
crates/metabench/tests/fixtures/artifacts/gungraun/manifest_order/metabench-identities.json	Gungraun fixture: identity manifest for ordering test.
crates/metabench/tests/fixtures/artifacts/gungraun/manifest_order/b/summary.json	Gungraun fixture: summary part of manifest ordering scenario.
crates/metabench/tests/fixtures/artifacts/gungraun/manifest_order/a/summary.json	Gungraun fixture: summary part of manifest ordering scenario.
crates/metabench/tests/fixtures/artifacts/gungraun/manifest_malformed/metabench-identities.json	Gungraun fixture: malformed identity manifest JSON.
crates/metabench/tests/fixtures/artifacts/gungraun/manifest_duplicate/metabench-identities.json	Gungraun fixture: duplicate identities in manifest.
crates/metabench/tests/fixtures/artifacts/gungraun/manifest_count/metabench-identities.json	Gungraun fixture: wrong manifest count vs selected benchmarks.
crates/metabench/tests/fixtures/artifacts/gungraun/malformed/summary.json	Gungraun fixture: malformed summary JSON.
crates/metabench/tests/fixtures/artifacts/gungraun/invalid_numeric/summary.json	Gungraun fixture: invalid numeric metric type.
crates/metabench/tests/fixtures/artifacts/gungraun/empty/.keep	Gungraun fixture: empty directory sentinel.
crates/metabench/tests/fixtures/artifacts/gungraun/duplicate/b/summary.json	Gungraun fixture: duplicate case index scenario (b).
crates/metabench/tests/fixtures/artifacts/gungraun/duplicate/a/summary.json	Gungraun fixture: duplicate case index scenario (a).
crates/metabench/tests/fixtures/artifacts/criterion/unicode/new/estimates.json	Criterion fixture: unicode identity estimates.
crates/metabench/tests/fixtures/artifacts/criterion/unicode/new/benchmark.json	Criterion fixture: unicode identity metadata.
crates/metabench/tests/fixtures/artifacts/criterion/missing_function/new/estimates.json	Criterion fixture: missing function id estimates.
crates/metabench/tests/fixtures/artifacts/criterion/missing_function/new/benchmark.json	Criterion fixture: missing function id metadata.
crates/metabench/tests/fixtures/artifacts/criterion/minimal/new/estimates.json	Criterion fixture: minimal valid estimates.
crates/metabench/tests/fixtures/artifacts/criterion/minimal/new/benchmark.json	Criterion fixture: minimal valid metadata.
crates/metabench/tests/fixtures/artifacts/criterion/malformed/new/estimates.json	Criterion fixture: malformed estimates.
crates/metabench/tests/fixtures/artifacts/criterion/malformed/new/benchmark.json	Criterion fixture: malformed metadata.
crates/metabench/tests/fixtures/artifacts/criterion/invalid_numeric/new/estimates.json	Criterion fixture: invalid numeric duration.
crates/metabench/tests/fixtures/artifacts/criterion/invalid_numeric/new/benchmark.json	Criterion fixture: metadata paired with invalid numeric estimates.
crates/metabench/tests/fixtures/artifacts/criterion/empty/.keep	Criterion fixture: empty directory sentinel.
crates/metabench/tests/fixtures/artifacts/criterion/duplicate/b/new/estimates.json	Criterion fixture: duplicate identity scenario (b) estimates.
crates/metabench/tests/fixtures/artifacts/criterion/duplicate/b/new/benchmark.json	Criterion fixture: duplicate identity scenario (b) metadata.
crates/metabench/tests/fixtures/artifacts/criterion/duplicate/a/new/estimates.json	Criterion fixture: duplicate identity scenario (a) estimates.
crates/metabench/tests/fixtures/artifacts/criterion/duplicate/a/new/benchmark.json	Criterion fixture: duplicate identity scenario (a) metadata.
crates/metabench/tests/fixtures/artifacts/allocations/minimal.json	Allocation fixture: minimal valid allocation metrics.
crates/metabench/tests/fixtures/artifacts/allocations/malformed.json	Allocation fixture: malformed JSON.
crates/metabench/tests/fixtures/artifacts/allocations/invalid_numeric.json	Allocation fixture: invalid numeric values.
crates/metabench/tests/fixtures/artifacts/allocations/empty.json	Allocation fixture: empty set.
crates/metabench/tests/fixtures/artifacts/allocations/duplicate.json	Allocation fixture: duplicate benchmark entries.
crates/metabench/src/mode.rs	Mode enum + parsing/display and unit tests.
crates/metabench/src/lib.rs	Public crate docs/exports + main! entry macro for benchmarks.
crates/metabench/src/group.rs	Runtime benchmark registry, filtering, and wildcard matching + tests.
crates/metabench/src/fixture.rs	Benchmark case/fixture traits + prepared output helper.
crates/metabench/src/error.rs	Public error model and formatting + tests.
crates/metabench/src/engines.rs	bitflags engine selection model and mapping to Mode.
crates/metabench/src/artifact.rs	Artifact parsing/writing for Criterion/Gungraun/perf/allocations + tests.
crates/metabench/src/arguments.rs	CLI argument parsing/routing + tests.
crates/metabench/README.md	Generated crate README for metabench.
crates/metabench/CHANGELOG.md	New changelog entry for crate introduction.
crates/metabench/Cargo.toml	New metabench crate manifest, benches, deps, and metadata.
crates/metabench/build.rs	Build script exporting target/profile metadata.
crates/metabench/benches/stateless_no_cases.rs	Example bench target: stateless/no-cases registration.
crates/metabench/benches/stateless_cases.rs	Example bench target: stateless/data-driven cases.
crates/metabench/benches/stateful_no_cases.rs	Example bench target: stateful/no-cases via SimpleFixture.
crates/metabench/benches/stateful_cases/main.rs	Example multi-file bench target entrypoint.
crates/metabench/benches/stateful_cases/hashmap.rs	Example stateful data-driven bench group + cases.
crates/metabench_macros/tests/ui/support.rs	UI-test support “mini runtime” surface for macro expansion.
crates/metabench_macros/tests/ui/pass/registration_selection.rs	Pass UI test: group naming + selection behavior.
crates/metabench_macros/tests/ui/pass/fixtures.rs	Pass UI test: fixtures/cases/stateful patterns.
crates/metabench_macros/tests/ui/fail/unsafe_method.stderr	Fail UI test output: unsafe method rejection.
crates/metabench_macros/tests/ui/fail/unsafe_method.rs	Fail UI test: unsafe method in benchmarks impl.
crates/metabench_macros/tests/ui/fail/unknown_argument.stderr	Fail UI test output: unknown attribute argument.
crates/metabench_macros/tests/ui/fail/unknown_argument.rs	Fail UI test: unsupported benchmarks attribute args.
crates/metabench_macros/tests/ui/fail/trait_impl.stderr	Fail UI test output: trait impl rejected.
crates/metabench_macros/tests/ui/fail/trait_impl.rs	Fail UI test: benchmarks applied to trait impl.
crates/metabench_macros/tests/ui/fail/non_path_group.stderr	Fail UI test output: non-path self type rejected.
crates/metabench_macros/tests/ui/fail/non_path_group.rs	Fail UI test: invalid group type.
crates/metabench_macros/tests/ui/fail/mutable_case.stderr	Fail UI test output: mutable case ref rejected.
crates/metabench_macros/tests/ui/fail/mutable_case.rs	Fail UI test: &mut Case parameter.
crates/metabench_macros/tests/ui/fail/multiple_arguments.stderr	Fail UI test output: multiple params rejected.
crates/metabench_macros/tests/ui/fail/multiple_arguments.rs	Fail UI test: multiple parameters.
crates/metabench_macros/tests/ui/fail/invalid_signature.stderr	Fail UI test output: async method rejected.
crates/metabench_macros/tests/ui/fail/invalid_signature.rs	Fail UI test: async method in benchmarks impl.
crates/metabench_macros/tests/ui/fail/invalid_receiver.stderr	Fail UI test output: by-value receiver rejected.
crates/metabench_macros/tests/ui/fail/invalid_receiver.rs	Fail UI test: self receiver.
crates/metabench_macros/tests/ui/fail/invalid_group_name.stderr	Fail UI test output: group name validation.
crates/metabench_macros/tests/ui/fail/invalid_group_name.rs	Fail UI test: group name contains /.
crates/metabench_macros/tests/ui/fail/invalid_benchmark_name.stderr	Fail UI test output: benchmark name validation.
crates/metabench_macros/tests/ui/fail/invalid_benchmark_name.rs	Fail UI test: benchmark name contains /.
crates/metabench_macros/tests/ui/fail/inline_method.stderr	Fail UI test output: inline attribute rejected.
crates/metabench_macros/tests/ui/fail/inline_method.rs	Fail UI test: method annotated #[inline].
crates/metabench_macros/tests/ui/fail/generic_method.stderr	Fail UI test output: generic method rejected.
crates/metabench_macros/tests/ui/fail/generic_method.rs	Fail UI test: generic method.
crates/metabench_macros/tests/ui/fail/generic_impl.stderr	Fail UI test output: generic impl rejected.
crates/metabench_macros/tests/ui/fail/generic_impl.rs	Fail UI test: generic impl block.
crates/metabench_macros/tests/ui/fail/empty_impl.stderr	Fail UI test output: empty impl rejected.
crates/metabench_macros/tests/ui/fail/empty_impl.rs	Fail UI test: empty impl.
crates/metabench_macros/tests/ui/fail/duplicate_names.stderr	Fail UI test output: duplicate benchmark names.
crates/metabench_macros/tests/ui/fail/duplicate_names.rs	Fail UI test: duplicate names via benchmark overrides.
crates/metabench_macros/tests/ui/fail/duplicate_benchmark_attribute.stderr	Fail UI test output: duplicate benchmark attr.
crates/metabench_macros/tests/ui/fail/duplicate_benchmark_attribute.rs	Fail UI test: duplicate #[benchmark] usage.
crates/metabench_macros/tests/ui/fail/duplicate_argument.stderr	Fail UI test output: duplicate attribute argument.
crates/metabench_macros/tests/ui/fail/duplicate_argument.rs	Fail UI test: repeated name = argument.
crates/metabench_macros/tests/ui/fail/const_method.stderr	Fail UI test output: const method rejected.
crates/metabench_macros/tests/ui/fail/const_method.rs	Fail UI test: const method.
crates/metabench_macros/tests/ui/fail/benchmarks_on_function.stderr	Fail UI test output: attribute placement error.
crates/metabench_macros/tests/ui/fail/benchmarks_on_function.rs	Fail UI test: benchmarks attribute on fn.
crates/metabench_macros/tests/ui/fail/benchmark_outside_impl.stderr	Fail UI test output: benchmark attribute placement error.
crates/metabench_macros/tests/ui/fail/benchmark_outside_impl.rs	Fail UI test: benchmark attribute outside benchmarks impl.
crates/metabench_macros/tests/ui/fail/associated_item.stderr	Fail UI test output: associated items rejected.
crates/metabench_macros/tests/ui/fail/associated_item.rs	Fail UI test: associated const in impl.
crates/metabench_macros/tests/ui.rs	trybuild harness for downstream macro UI tests.
crates/metabench_macros/src/lib.rs	Proc-macro entry points delegating to impl crate, with coverage exclusions.
crates/metabench_macros/README.md	Generated README for metabench_macros.
crates/metabench_macros/CHANGELOG.md	New changelog for metabench_macros.
crates/metabench_macros/Cargo.toml	Proc-macro crate manifest + coverage-gate override.
crates/metabench_macros_impl/src/lib.rs	Macro implementation: validates impls/methods and generates registrations.
crates/metabench_macros_impl/README.md	Generated README for metabench_macros_impl.
crates/metabench_macros_impl/CHANGELOG.md	New changelog for metabench_macros_impl.
crates/metabench_macros_impl/Cargo.toml	Macro-impl crate manifest and deps (syn2/quote/etc).
Cargo.toml	Workspace deps updated and new workspace members/deps added for metabench.
Cargo.lock	Lockfile updated for new/updated dependencies.
.spelling	Adds metabench-related vocabulary to spellcheck allowlist.
.github/workflows/main.yml	Adds metabench registration step; excludes metabench from Miri.

Review details

• Files reviewed: 122/126 changed files
• Comments generated: 8
• Review effort level: Lite

---

💡 Configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

[Copilot]

🔵 Needs a closer look

It introduces a large new benchmarking subsystem (multiple crates + workspace/CI integration) that warrants focused human review of overall design, long-term maintenance costs, and operational behavior.

Review details

Suppressed comments (1)

crates/metabench/src/group.rs:357

• When selecting without an identity manifest, an out-of-range cases.nth(index) is reported as Error::UnknownBenchmark(index.to_string()), which produces a confusing message (treating an index as a benchmark name). Consider including context that this is a case index.

    let (group, benchmark) = cases.nth(index).ok_or_else(|| Error::UnknownBenchmark(index.to_string()))?;

• Files reviewed: 127/131 changed files
• Comments generated: 2
• Review effort level: Lite

[Copilot]

🟡 Changes recommended

The new public-facing rustdoc examples in metabench use ? without a fn main() -> Result<...> wrapper, so the documentation snippets should be adjusted to be self-contained.

Once you've addressed the issues Copilot identified, you can request another Copilot review.

Review details

• Files reviewed: 127/131 changed files
• Comments generated: 2
• Review effort level: Lite

[Copilot]

🔵 Needs a closer look

It introduces multiple new crates, a new macro-based registration surface, and CI/dependency graph changes that warrant final human validation.

Review details

• Files reviewed: 127/131 changed files
• Comments generated: 0 new
• Review effort level: Lite

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 100.0%. Comparing base (bdafbc4) to head (6ee7fe1).

Additional details and impacted files

@@           Coverage Diff           @@
##             main     #732   +/-   ##
=======================================
  Coverage   100.0%   100.0%           
=======================================
  Files         583      583           
  Lines       62930    62930           
=======================================
  Hits        62930    62930           

Flag	Coverage Δ
linux	99.9% <ø> (+<0.1%)	⬆️
linux-arm	100.0% <ø> (ø)
scheduled	?
windows	100.0% <ø> (?)

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[Copilot]

🟡 Changes recommended

There are correctness/maintainability issues in new code (notably a JSON assertion comparing Value to an integer and insufficiently-informative expect() messages) that should be addressed before approval.

Once you've addressed the issues Copilot identified, you can request another Copilot review.

Review details

• Files reviewed: 127/131 changed files
• Comments generated: 2
• Review effort level: Lite

[Copilot]

🟡 Changes recommended

The report baseline logic can classify allocation regressions based on allocated_bytes shifts without exposing that shift in the public report model, which risks confusing downstream consumers.

Once you've addressed the issues Copilot identified, you can request another Copilot review.

Review details

• Files reviewed: 127/131 changed files
• Comments generated: 1
• Review effort level: Lite

[Copilot]

🟡 Changes recommended

The runner uses predictable temp-file/FIFO paths in a shared temp directory, which is vulnerable to interference on multi-user systems and should be hardened before approval.

Once you've addressed the issues Copilot identified, you can request another Copilot review.

Review details

Suppressed comments (1)

crates/metabench/src/runner.rs:705

• create_worker_token uses a predictable path under the global temp directory. On multi-user systems this can be interfered with (pre-creating the path, symlink tricks, etc.), causing spurious failures or writing/deleting an unintended file. Prefer a securely-created unique temp file (O_EXCL + random name) from tempfile and pass that path to the worker.

fn create_worker_token() -> Result<std::path::PathBuf, Error> {
    let sequence = WORKER_SEQUENCE.fetch_add(1, Ordering::Relaxed);
    let token = env::temp_dir().join(format!("metabench-worker-{}-{sequence}.token", std::process::id()));
    fs::write(&token, []).map_err(Error::CreateWorkerToken)?;
    Ok(token)
}

• Files reviewed: 127/131 changed files
• Comments generated: 1
• Review effort level: Lite

[Copilot]

🟡 Changes recommended

A few newly added tests/docs include clear correctness and maintainability issues (non-existent Duration constructors and session-specific documentation in repo code) that should be corrected before approval.

Once you've addressed the issues Copilot identified, you can request another Copilot review.

Review details

Suppressed comments (1)

crates/metabench/src/arguments.rs:628

• std::time::Duration has no from_mins/from_hours constructors. Use from_secs here so the test expresses the same durations without relying on non-existent APIs.

        for (value, expected) in [
            ("1ms", Duration::from_millis(1)),
            ("2s", Duration::from_secs(2)),
            ("3m", Duration::from_mins(3)),
            ("4h", Duration::from_hours(4)),
        ] {

• Files reviewed: 79/83 changed files
• Comments generated: 4
• Review effort level: Lite

[Copilot]

🟡 Changes recommended

It includes PR/session-specific internal process commentary in a committed test doc comment (and minor trailing-whitespace), which should be cleaned up before approval.

Once you've addressed the issues Copilot identified, you can request another Copilot review.

Review details

Suppressed comments (1)

crates/metabench/tests/spawned_benchmark.rs:13

• The module docs include PR/session-specific process notes ("write_unverified", "WSL-crash policy", "this session forbids") that aren't relevant to the repository and will become stale/noisy over time. Please remove this internal workflow commentary and keep the doc comment focused on what the test validates.

• Files reviewed: 80/84 changed files
• Comments generated: 1
• Review effort level: Lite

[martintmk]

[AI AGENT]: Review of the new metabench crate, public-API first.

What I verified. I read CI on this exact head (35f3d96) rather than re-running the suite, and reproduced the two concrete failures locally in a clean worktree: cargo test -p metabench --doc gives test result: FAILED. 18 passed; 7 failed (every crate-level example), and cargo test -p metabench --lib exact_identity_and_mode_filters_report_whether_work_remains reproduces the CI assertion failure. I also probed the builder surface from a consumer's position with a throwaway integration test, and verified my replacement assertions for the broken test pass. All probes have been reverted.

Public surface reviewed. The crate root exports Suite, SuiteBuilder, Case, case, Engines, Throughput, ThroughputSpec and the main! macro, plus the #[doc(hidden)] __private module. The typestate builder chain (GroupBuilder, BenchmarkBuilder and ~20 lifecycle marker types) is pub inside the private builder module and reachable through return types but not nameable. Two findings land here: serde is an unapproved external type on the re-exported Throughput, and the unnameable builder types make suites unfactorable. I also reviewed dependencies and features, the report/artifact comparison logic, the Miri and coverage gate changes, and the test fixtures.

Areas with no finding. classify/percentage_shift in report.rs handle the zero-baseline and threshold boundaries correctly and are covered by targeted tests; the Gungraun identity-manifest staleness paths in group.rs are guarded and have fixtures for the count, order and wrong-set cases; the unsafe env mutation in runner.rs is genuinely the sole CARGO_TARGET_DIR writer in the suite, as its SAFETY comment claims. No telemetry or resilience surface is touched, so those gates did not apply.

What I could not assess. I did not execute the Criterion, Gungraun, perf or allocation backends end to end — they need Linux and Callgrind — so the measurement-boundary changes made in earlier review rounds are reasoned about from the source only, not measured. I also did not evaluate the report's numerical output against real engine artifacts.

Verdict: changes requested. The three blocking items are the seven broken doctests, the failing retain_mode test that leaves mode filtering untested, and serde in the public API. CI on this head is red across pr-test, testing, static-analysis, spell-check and external-type-exposure; the findings below cover the ones that point at a design or correctness problem rather than restating tooling output. Separately, this PR adds ~12k lines and a new published crate with an empty description — a short summary of the engine model and the reporting contract would help future readers considerably.

[Copilot]

🟡 Changes recommended

Artifact path selection and Criterion estimate validation have correctness issues that can produce misleading or nonsensical reports in real-world runs.

Once you've addressed the issues Copilot identified, you can request another Copilot review.

Review details

• Files reviewed: 69/73 changed files
• Comments generated: 2
• Review effort level: Lite

[Copilot]

🟡 Changes recommended

Artifact isolation/validation and worker-token hardening issues should be addressed to prevent stale/misleading reports and strengthen internal marker handling.

Once you've addressed the issues Copilot identified, you can request another Copilot review.

Review details

Suppressed comments (4)

crates/metabench/src/artifact.rs:50

• Artifact output directories are derived from pre-existing CRITERION_HOME/GUNGRAUN_HOME and default to shared paths under the report root (e.g., target/metabench//criterion). This makes it easy to accidentally mix stale artifacts from earlier runs (or from a user’s environment overrides) into a new combined report. Prefer per-run, metabench-owned directories so each invocation parses only the artifacts it just produced.

        let run = root.join("runs").join(format!("{timestamp}-{}", std::process::id()));
        let criterion = std::env::var_os("CRITERION_HOME").map_or_else(|| root.join("criterion"), PathBuf::from);
        let gungraun = std::env::var_os("GUNGRAUN_HOME").map_or_else(|| root.join("gungraun"), PathBuf::from);
        fs::create_dir_all(&run).map_err(|source| Error::ArtifactIo { path: run.clone(), source })?;

crates/metabench/src/artifact.rs:314

• Criterion estimates are only validated for finiteness; negative medians/interval bounds will currently be accepted. The fixtures include an "invalid_numeric" case with a negative point_estimate, and for time-based measurements a negative estimate is nonsensical—treat these as invalid artifact data.

fn validate_finite(path: &Path, value: f64) -> Result<(), Error> {
    if value.is_finite() {
        Ok(())
    } else {
        Err(Error::ArtifactFormat {

crates/metabench/tests/spawned_benchmark.rs:26

• This spawns cargo bench from within the test without pinning the package, which can become ambiguous or run the wrong target when invoked from a workspace root. Add -p metabench so the test always targets this crate’s benches.
  crates/metabench/tests/spawned_benchmark.rs:37
• This spawns cargo bench from within the test without pinning the package, which can become ambiguous or run the wrong target when invoked from a workspace root. Add -p metabench so the test always targets this crate’s benches.

• Files reviewed: 68/72 changed files
• Comments generated: 2
• Review effort level: Lite

[Copilot]

🟡 Changes recommended

Workspace dependency entries were added that appear unused in-tree (e.g., syn2, nix, bitflags), which should be removed or justified to avoid unnecessary maintenance/audit surface.

Once you've addressed the issues Copilot identified, you can request another Copilot review.

Review details

Suppressed comments (1)

Cargo.toml:169

• nix was added as a workspace dependency, but nothing in the repo currently depends on it directly (no use nix::... and no workspace = true dependency entries). As a result, the lockfile only carries nix 0.27.1 from command-group, so the workspace nix = 0.31.3 entry is effectively unused and the nearby comment about “duplicate” versions becomes misleading.

Unless you’re about to add a direct nix usage, remove this workspace dependency entry.

nix = { version = "0.31.3", default-features = false }

• Files reviewed: 82/86 changed files
• Comments generated: 2
• Review effort level: Lite

[Copilot]

🟡 Changes recommended

The new subprocess-spawning integration tests should be made robust under parallel test execution to avoid nondeterministic contention/flakiness.

Once you've addressed the issues Copilot identified, you can request another Copilot review.

Review details

• Files reviewed: 82/85 changed files
• Comments generated: 1
• Review effort level: Lite

[Copilot]

🟡 Changes recommended

The repo README entry for metabench appears to overstate supported engines by mentioning perf, which should be corrected to avoid misleading documentation.

Once you've addressed the issues Copilot identified, you can request another Copilot review.

Review details

• Files reviewed: 82/85 changed files
• Comments generated: 1
• Review effort level: Lite

[Copilot]

🟡 Changes recommended

Artifact run retention uses a time-based “active” heuristic that can expire during long runs, allowing pruning logic to delete still-in-use run directories.

Once you've addressed the issues Copilot identified, you can request another Copilot review.

Review details

• Files reviewed: 82/85 changed files
• Comments generated: 1
• Review effort level: Lite

[Copilot]

🟡 Changes recommended

There are a few correctness/robustness issues in newly added orchestration and test code (notably error classification on worker kill failures and lock poisoning in subprocess tests) that should be addressed before approval.

Once you've addressed the issues Copilot identified, you can request another Copilot review.

Review details

Suppressed comments (1)

crates/metabench/tests/spawned_benchmark.rs:41

• Same lock-poisoning concern as in run(): Mutex::lock().unwrap() will cascade failures after a single panic. Consider tolerating poisoning when the lock is only used for cross-test subprocess serialization.

• Files reviewed: 82/85 changed files
• Comments generated: 3
• Review effort level: Lite

[Copilot]

🟡 Changes recommended

The artifact retention logic appears to depend on file-locking methods (try_lock/lock) without an explicit locking API wired in, which can undermine the “don’t delete active runs” safety guarantee.

Once you've addressed the issues Copilot identified, you can request another Copilot review.

Review details

• Files reviewed: 82/85 changed files
• Comments generated: 1
• Review effort level: Lite

[Copilot]

🔵 Needs a closer look

crates/metabench/src/artifact.rs relies on non-std File::lock()/File::try_lock() APIs without an evident supporting dependency/trait import, so the active-run marker locking needs to be corrected.

Review details

Suppressed comments (1)

crates/metabench/src/artifact.rs:566

• ArtifactDirectory's active-run marker logic calls try_lock()/lock() on std::fs::File (e.g. file.lock() here, and marker.try_lock() a few lines above). Those methods are not on File in std; they require an extension trait (for example from fs4/fs2) to be in scope and the corresponding crate to be a dependency. Please either add the needed dependency + use …::FileExt as _; import, or switch to a lock approach that doesn't rely on these non-std methods.

            source,
        })?;
    file.lock().map_err(|source| Error::ArtifactIo { path: marker, source })?;
    Ok(file)

• Files reviewed: 82/85 changed files
• Comments generated: 0 new
• Review effort level: Lite

[Copilot]

🟡 Changes recommended

There is at least one user-facing error message that is misleading in some single-engine multi-error scenarios and should be corrected before approval.

Once you've addressed the issues Copilot identified, you can request another Copilot review.

Review details

• Files reviewed: 82/85 changed files
• Comments generated: 1
• Review effort level: Lite

[Copilot]

🔵 Needs a closer look

The PR introduces a large new benchmarking subsystem (runtime orchestration + proc macros + CI workflow changes), which warrants final human review despite no specific defects found in the inspected diffs.

Review details

• Files reviewed: 82/85 changed files
• Comments generated: 0 new
• Review effort level: Lite

[Copilot]

🔵 Needs a closer look

It introduces multiple new crates (runtime + proc-macros), substantial orchestration/reporting logic, and CI/workspace dependency changes that merit final human review despite no specific blocking defects found in this pass.

Review details

• Files reviewed: 83/86 changed files
• Comments generated: 0 new
• Review effort level: Lite

[Copilot]

🔵 Needs a closer look

It introduces a sizable new framework (new crates + proc macros + CI/tooling changes) that warrants final human review for correctness and long-term maintainability.

Review details

• Files reviewed: 84/87 changed files
• Comments generated: 0 new
• Review effort level: Lite

[Copilot]

🔵 Needs a closer look

crates/metabench/Cargo.toml doesn’t allow criterion in allowed_external_types despite metabench::__private publicly re-exporting it, which is likely to trip the repo’s external-type exposure guard.

Review details

Suppressed comments (1)

crates/metabench/Cargo.toml:27

• metabench publicly re-exports the criterion crate via src/__private.rs (pub use criterion;), but package.metadata.cargo_check_external_types.allowed_external_types does not allow any criterion types. This will cause the external-types guard to flag metabench::__private::criterion::* as exposed types unless criterion is explicitly allowed (or the re-export is narrowed).

[package.metadata.cargo_check_external_types]
allowed_external_types = ["gungraun", "metabench_macros::*"]

• Files reviewed: 84/87 changed files
• Comments generated: 0 new
• Review effort level: Lite

[Copilot]

🟡 Changes recommended

Workspace/dependency metadata has at least two concrete issues (unused nix workspace dependency and incomplete external-types allowlist for a publicly re-exported criterion).

Once you've addressed the issues Copilot identified, you can request another Copilot review.

Review details

• Files reviewed: 84/87 changed files
• Comments generated: 2
• Review effort level: Lite

[Copilot]

🟡 Changes recommended

A production mapping/test expectation mismatch in crates/metabench/src/metric_display.rs needs to be resolved to keep tests consistent with the implemented metric labels.

Once you've addressed the issues Copilot identified, you can request another Copilot review.

Review details

• Files reviewed: 87/90 changed files
• Comments generated: 1
• Review effort level: Lite

[Copilot]

🟡 Changes recommended

The metabench crate-level documentation contains at least one incorrect CLI flag name and a doc typo that should be corrected to avoid user confusion.

Once you've addressed the issues Copilot identified, you can request another Copilot review.

Review details

Suppressed comments (1)

crates/metabench/src/lib.rs:180

• Typo in the doc comment: “prints a consolidate result table” should be “prints a consolidated result table”.

//! Each run prints a consolidate result table to the terminal and writes `report.json` and `report.md`

• Files reviewed: 88/91 changed files
• Comments generated: 1
• Review effort level: Lite

[Copilot]

🟡 Changes recommended

The metabench crate-level docs contain a user-facing CLI option mismatch (--allocation vs --allocations) that should be corrected before merge.

Once you've addressed the issues Copilot identified, you can request another Copilot review.

Review details

Suppressed comments (1)

crates/metabench/src/lib.rs:101

• The docs list a --allocation engine selector, but the actual CLI option is --allocations (see crates/metabench/src/arguments.rs:157 and the help text in crates/metabench/src/runner.rs:29). This mismatch is likely to confuse users and contradicts the later options list which already uses --allocations.

//! Criterion, Gungraun, and allocation tracking are enabled by default. You can
//! control the specific engines to run from the command-line by passing the
//! `--criterion`, `--gungraun`, `--allocation`, `--perf`, or `--all-engines` options:

• Files reviewed: 86/95 changed files
• Comments generated: 1
• Review effort level: Lite

[Copilot]

🔵 Needs a closer look

It introduces new core functionality plus multiple proc-macro crates and CI workflow changes, which warrants final human review despite no specific defects identified in the reviewed diffs.

Review details

• Files reviewed: 92/95 changed files
• Comments generated: 0 new
• Review effort level: Lite

[Copilot]

🟡 Changes recommended

The subprocess test harness currently leaks temp directories by storing a TempDir in a static, which should be adjusted to avoid accumulating artifacts across repeated runs.

Once you've addressed the issues Copilot identified, you can request another Copilot review.

Review details

• Files reviewed: 92/95 changed files
• Comments generated: 1
• Review effort level: Lite

[Copilot]

🟡 Changes recommended

New metabench_macros_impl sources add inline #[cfg(test)] mod tests blocks without the repo’s #[cfg_attr(coverage_nightly, coverage(off))] convention, which risks violating the coverage gate policy.

Once you've addressed the issues Copilot identified, you can request another Copilot review.

Review details

• Files reviewed: 92/95 changed files
• Comments generated: 3
• Review effort level: Lite

[Copilot]

🟡 Changes recommended

Allocation/perf probe workers currently drop the configured Criterion arguments (including derived --bench/--test and forwarded --criterion-arg), which can make probe behavior diverge from the main Criterion run.

Once you've addressed the issues Copilot identified, you can request another Copilot review.

Review details

Suppressed comments (1)

crates/metabench/src/runner.rs:531

• Perf probe workers similarly invoke the Criterion harness with only the filter/--exact plus a hard-coded --test, dropping the parent invocation’s --bench/--test selection and any forwarded --criterion-arg ... options. Passing the Criterion args through keeps perf probes aligned with the target’s configured Criterion behavior.

        let native_arguments = [
            OsString::from(&target.native_identity),
            OsString::from("--exact"),
            OsString::from("--test"),
        ];

• Files reviewed: 92/95 changed files
• Comments generated: 1
• Review effort level: Lite

[Copilot]

🔵 Needs a closer look

It introduces substantial new functionality (new crates, macro expansions, subprocess/perf orchestration, and CI wiring) that warrants careful human validation beyond automated review.

Review details

• Files reviewed: 92/95 changed files
• Comments generated: 0 new
• Review effort level: Lite

[Copilot]

🟡 Changes recommended

parse_gungraun can currently accept and emit results with an empty metrics map in cases where the summary has no current metrics, which should be treated as an artifact-format failure.

Once you've addressed the issues Copilot identified, you can request another Copilot review.

Review details

• Files reviewed: 92/95 changed files
• Comments generated: 1
• Review effort level: Lite

[Copilot]

🔵 Needs a closer look

It introduces multiple new crates (including proc-macros), CI workflow changes, and new dependency/version interactions that warrant final human review despite no specific defects found in this pass.

Review details

• Files reviewed: 92/95 changed files
• Comments generated: 0 new
• Review effort level: Lite