Repository navigation
[Cycle 2/5] Add a standing test-integrity review lane #28
Description
Goal
Extend the standing tests review lane with a named test-integrity sub-check. The sub-check must audit evidence quality, not only test pass results.
Scope
- Audit network isolation in tests.
- Audit shared temporary-directory contention.
- Audit verifier scripts for fail-open behavior.
- Audit assertion specificity and hidden environment dependencies.
- Audit fixture repositories for inherited Git signing and hook configuration.
- Reuse
schemas/lane-findings.schema.json, its P0-P3 definitions, and the existing merge pipeline. - Do not introduce a second finding or severity schema.
Dependency
Depends only on released eval-contract/v1. It does not depend on Cycle 1 routing or locking changes.
Dogfooding gate
Run ai-orchestration against an isolated compound-loop checkout. Ask it to produce the lane design, test matrix, and candidate changes.
Use a versioned fixture manifest with at least two seeded defects and one unseeded holdout fixture per audit class. The release gate requires:
- The lane has 100 percent P0/P1 recall and at least 90 percent overall precision across clean and defective fixtures.
- The lane produces zero P0/P1 false positives on clean fixtures.
- Three repeated reviews detect the same seed IDs. Free-text wording is not part of the determinism check.
- The compound-loop validation suite passes.
- A dedicated
C2-TIdogfood corpus directly exercises all audit classes and the holdouts.
Acceptance criteria
- The reviewing protocol invokes the lane on every full phase-gate review.
- The lane persists its raw evidence and normalized findings.
- A missing, cancelled, or malformed lane result fails closed.
- The documentation explains how to add a new seeded test-integrity fixture.
- Repository fixtures can commit without host signing keys or user hooks.
Out of scope
- Changes to
ai-orchestrationruntime behavior. - Compound-loop embedding inside
ai-orchestration.
Activity
feat(review): add a model-based release gate for test integrity
feat(review): add a model-based release gate for test integrity
feat(review): add a model-based release gate for test integrity
feat(review): standing test-integrity sub-check (#44)
teslamint commentedon Nov 6, 2026
Status after #44 (merged as e4f41da): the sub-check, the per-review record, the offline regression evaluator, and the model release gate are on main. The issue's gate is not met yet.
Best measured result: gpt-6.1-sol, tool-free, temperature 0, 3 runs × 79 fixtures:
The remaining errors are mostly environment-dependency cases: three missed and three false alarms. Git fixtures that already isolate themselves were also flagged. The last checklist edit (externally contacted Git remotes count as Network isolation) has not been measured with a model yet. Closing this issue needs either a measured pass on a new blind set, or revised thresholds.
fix(review): make the test-integrity gate report and corpora match wh…
fix(review): score every test-integrity corpus alone and keep worst f…
teslamint commentedon Nov 6, 2026
Second pass measured. The gate is still not met; this issue stays open.
Work is in PR #45 (
feat/c2ti-gate), currently open: CodeRabbit's review of the first commits requested three changes, all fixed in 1574d9e, but the re-review is rate-limited and the stale review blocks merge.Changes: determinism now compares detected seed IDs across repeats (as this issue specifies) and every repeat is scored; three class boundaries were specified; three blind labels were corrected against executed reproductions; two new blind corpora were added, and clean-case file names that announced the answer were neutralised.
Whole registry, 145 cases × 3 repeats, gpt-6.1-sol, temperature 0, tool-free:
Out-of-sample (
test-integrity-blind-gate3.json, 36 cases, scored alone): recall 22–24 of 24, precision 80.0–85.7%, one clean-case P1 false positive in two of three repeats.Remaining blockers:
C2-TI-G3-AST-P2andC2-TI-G3-GIT-P3are detected in two of three repeats at temperature 0, so the determinism criterion fails.core.hooksPath.Dogfooding:
ai-orchestrationran against an isolated export and produced a 323-line document with a design, a test matrix, and a candidate-change manifest, but the document's own status section records that no implementation files were inspected and lists blockers B-01 to B-12, so it is generic evidence, not checkout-grounded candidate changes. The exercise also exposed two orchestrator defects: a planner that emitsfile_path: "."is rejected by the executor's path guard, and default stage models (gemini-3.1-pro-low,gpt-5.5) no longer exist in the proxy catalog.