diff --git a/docs/plans/2026-08-30-001-feat-test-integrity-review-lane-plan.md b/docs/plans/2026-08-30-001-feat-test-integrity-review-lane-plan.md new file mode 100644 index 0000000..ef017b9 --- /dev/null +++ b/docs/plans/2026-08-30-001-feat-test-integrity-review-lane-plan.md @@ -0,0 +1,174 @@ +--- +schema: plan/v1 +title: Test-integrity review sub-check +type: feat +status: approved +date: 2026-08-30 +execution: code +origin: docs/specs/2026-08-30-test-integrity-review-lane-design.md +body_seal: 54627347f8ddb139e946478f331cc8af2ee9e7129e5301e1f024b15f217581d8 +--- + +# Test-integrity Review Sub-check Plan + +## Goal + +Add a named `test-integrity` sub-check to the standing `tests` review lane and prove its bounded detector contract with a versioned C2-TI corpus. The evaluator must fail closed, use the existing lane-finding and merge contracts, and report reproducible recall, precision, isolation, and repeatability evidence. + +## Architecture notes + +- Keep the reviewer-facing contract in `skills/reviewing/references/lanes.md` and place the detailed checklist in `skills/reviewing/references/test-integrity.md`. +- Use one standard-library shell entry point with embedded Python. This matches the repository's executable validation pattern and avoids a runtime dependency. +- Represent each corpus case as JSON with stable case ID, class, source text, expected finding IDs, severity, and holdout status. The evaluator creates disposable files from these values. +- Implement six narrow lexical probes. Each probe returns the existing detail-tier fields and a source-line evidence item. The evaluator rejects malformed compact or detail findings before calculating metrics. +- Run the fixture under isolated `HOME`, `TMPDIR`, Git config, and hooks. Count network-capable command attempts and reject any non-local target before measuring detector results. +- Use the merge-pipeline contract as the source of severity, confidence, deduplication, suppression, and testing-gap rules. The evaluator checks those fields; it does not introduce a second envelope or a second severity scale. +- Keep the corpus and evaluator independent enough that removing one detector or one expected finding makes a named case fail. Record the failed detector assertion as discrimination evidence. + +## Assumption Recheck + +Origin spec retains no live assumptions; no assumption recheck required. + +## File structure + +| File | Responsibility | Owner | +|---|---|---| +| `skills/reviewing/references/lanes.md` | Name `test-integrity` under the always-on `tests` lane and retain the lane's non-flags | U1 | +| `skills/reviewing/references/test-integrity.md` | Define the six checks, evidence requirements, severity guidance, and non-flags | U1 | +| `scripts/test-review-test-integrity.sh` | Run fixture creation, probes, schema checks, metrics, repeatability, and isolation audit | U2-U4 | +| `tests/conformance/review/test-integrity-corpus.json` | Versioned C2-TI seeded defects, holdouts, clean fixture, and expected findings | U2-U3 | +| `scripts/validate.sh` | Run the no-network, no-model dogfood check as a validation gate | U4 | +| `docs/specs/2026-08-30-test-integrity-review-lane-design.md` | Approved source specification | U1 | +| `docs/plans/2026-08-30-001-feat-test-integrity-review-lane-plan.md` | This plan and traceability record | U1 | + +## Scenario coverage map + +| S-ID | Ordered unit chain | Integration evidence | +|---|---|---| +| S1 | U1 -> U2 -> U4 | `case_lane_contract` checks the named sub-check and its six classes in the standing lane. Covers S1. | +| S2 | U2 -> U3 | `case_seeded_defects` runs every seeded case, validates detail findings, and measures recall and precision. Covers S2. | +| S3 | U2 -> U3 | `case_clean_fixture` runs the clean fixture three times and checks no P0/P1 finding plus identical fingerprints. Covers S3. | +| S4 | U2 -> U3 -> U4 | `case_fixture_isolation` proves isolated Git state, temporary paths, network command count, and nested compound-loop validation. Covers S4. | + +## Implementation Units + +## U1: Document the tests-lane sub-check contract +Execution note: test-first +Files: + Create: `skills/reviewing/references/test-integrity.md` + Modify: `skills/reviewing/references/lanes.md` + Test: `scripts/test-review-test-integrity.sh` +Interfaces: + Consumes: existing `tests` lane contract, `schemas/lane-findings.schema.json`, and `skills/reviewing/references/merge-pipeline.md` + Produces: named `test-integrity` entry with six checks, evidence rules, severity guidance, and non-flags +Test scenarios: + happy: the lane text names all six integrity classes and the standard lane-findings contract + edge: the lane text keeps coverage gaps and weak assertions in the tests lane while routing unrelated security or performance concerns to their existing lanes + error: deleting one required class or the standard schema reference makes `case_lane_contract` fail by the missing class name + integration: `case_lane_contract` walks S1 +Steps: + 1. Add a failing `case_lane_contract` assertion for the absent `test-integrity` name and six class names. + 2. Run `bash scripts/test-review-test-integrity.sh contract`; confirm the assertion fails because the lane has no named sub-check. + 3. Add the concise lane entry and the detailed reference contract. + 4. Run the contract case; confirm all required names and existing merge-pipeline references are present. + 5. Commit: `docs(review): define the test-integrity sub-check` +Acceptance: the contract case passes and the lane references exactly the six named classes without adding a new lane or schema. + +## U2: Build the deterministic finding probes and corpus +Execution note: test-first +Files: + Create: `scripts/test-review-test-integrity.sh`; `tests/conformance/review/test-integrity-corpus.json` + Modify: none + Test: `scripts/test-review-test-integrity.sh` +Interfaces: + Consumes: corpus cases containing source text, class, expected finding IDs, severity, and holdout marker + Produces: detail-tier lane findings with `title`, `severity`, `file`, `line`, `why_it_matters`, `evidence`, `confidence`, `owner`, `autofix_class`, `requires_verification`, and `pre_existing` +Test scenarios: + happy: two seeded defects per class produce their expected IDs and each finding validates against the existing schema + edge: safe holdouts use neighboring syntax without triggering the class probe + error: malformed fields, unknown classes, invalid severity, missing evidence, and duplicate IDs fail closed + integration: `case_seeded_defects` walks S2 +Steps: + 1. Write a failing corpus inventory and finding-schema assertion for one case in each class. + 2. Run the corpus case; confirm it fails because the corpus and probes do not exist. + 3. Add six class definitions, two seeded defects per class, one holdout per class, and one clean fixture. + 4. Implement one bounded probe per class and emit the source path, line, impact, and evidence for every match. + 5. Validate every finding against the required schema fields and reject malformed outputs before metric calculation. + 6. Run the seeded, holdout, and clean cases; confirm the expected IDs and failure reasons. + 7. Commit: `feat(review): add deterministic test-integrity findings` +Acceptance: seeded and holdout cases have zero unexpected IDs, all findings pass schema validation, and malformed output never becomes a clean result. + +## U3: Add hermetic fixture auditing and quality metrics +Execution note: test-first +Files: + Modify: `scripts/test-review-test-integrity.sh` + Test: `scripts/test-review-test-integrity.sh` +Interfaces: + Consumes: U2 findings, disposable fixture root, local Git repository, isolated environment, and expected corpus labels + Produces: recall, precision, clean-fixture P0/P1 count, three-run fingerprint equality, Git isolation evidence, and network-call count +Test scenarios: + happy: recall is 100% for seeded P0/P1 defects, precision is at least 90%, and the clean fixture has zero P0/P1 findings + edge: roots containing spaces are contained, shared temporary directory patterns are rejected, and safe Git hooks/signing configuration is accepted + error: a network command, inherited hook, signing key, shared temporary path, or host environment read fails before a clean verdict + integration: `case_clean_fixture` and `case_fixture_isolation` walk S3 and S4 +Steps: + 1. Add failing assertions for isolated `HOME`, isolated `TMPDIR`, empty hooks, disabled signing, zero network attempts, and the metric thresholds. + 2. Run the fixture group; confirm failure names the missing isolation or metric evidence. + 3. Create a temporary Git repository with explicit identity, `commit.gpgsign=false`, and an empty hooks path; clear credential and network environment variables. + 4. Execute the corpus three times, normalize fingerprints, compare runs, and count true positives, false positives, and false negatives by class and severity. + 5. Add a mutation-shaped case that removes one seeded defect and asserts the detector assertion fails rather than setup. + 6. Run the fixture group and retain concise stdout evidence for each required threshold. + 7. Commit: `test(review): measure test-integrity fixture quality` +Acceptance: the fixture group proves all R3-R8 thresholds and reports a nonzero failure for each removed detector or seeded defect. + +## U4: Connect dogfood to compound-loop validation +Execution note: test-first +Files: + Modify: `scripts/validate.sh`; `scripts/test-review-test-integrity.sh` + Test: `scripts/test-review-test-integrity.sh`; `scripts/validate.sh` +Interfaces: + Consumes: U1-U3 contract, corpus, evaluator, and validation root + Produces: one validation gate that runs the local dogfood without network or model calls +Test scenarios: + happy: direct dogfood and nested `bash scripts/validate.sh` both pass with the recursion guard + edge: the nested validation sees the same repository root and does not scan the disposable fixture as a source tree + error: a failed dogfood case or compound-loop validation exits nonzero and names the failed gate + integration: `case_validation_gate` completes the S4 path +Steps: + 1. Add a failing assertion that `scripts/validate.sh` invokes the dogfood command and that the nested invocation uses only the documented recursion guard. + 2. Run the validation gate; confirm it fails because no test-integrity invocation exists. + 3. Add the guarded invocation at the end of `scripts/validate.sh` and keep the dogfood script's network/model counters explicit. + 4. Run the direct dogfood, full validation, ruff-equivalent shell checks, and existing conformance checks. + 5. Commit: `test(review): gate validation on test-integrity dogfood` +Acceptance: `bash scripts/test-review-test-integrity.sh dogfood` and `bash scripts/validate.sh` exit 0; a mutated corpus or lane contract exits nonzero. + +## Mutation/failure-state matrix + +No stateful ceremony in the deliverable; no mutation/failure-state matrix required. + +## Carry-forward trigger audit + +| Tracker row | Trigger class | What fired it | Disposition | +|---|---|---|---| +| Conformance suite | event-based | The roadmap records a structural-validation regression and keeps the suite row open; this cycle does not alter the release-loop conformance implementation. | Deferred to Follow-Up Work: finish the separately scoped release-loop conformance cycle before closing its row. | + +The following open rows were audited as event-based and did not fire in this plan because their named future occurrence is outside this change: Schema validators + fixtures; Session-history search; compound-refresh headless auto-apply; Cross-round deepening suppression; Demo/evidence capture; Project-defined lane schema; Ambient compound triggers; Gemini support verification; Release-loop phase-worker dispatch; Pre-release / dev version convention; Evidence-tier vocabulary; Skill-level trace evidence; New-skill distinctness gate; the post-Retro terminal criterion; the reusable forced-failure matrix contract; persisted reviewer output; invariant-attack wording in review dispatch; success-criterion severity grading; dispatched-committer SSH evidence; retro branch merge/push coupling; `gh pr merge --delete-branch` handling; canonical adoption evidence publication; Shipping's outside-diff disposition gate; bounded raw post-merge failure output; legacy handoff case coverage; mid-directory handoff fault injection; and exact structured review events. + +Audited `ROADMAP.md` at `59dcff696ba2aa3bfa24a46ed77baee5c41c2ac6`: 29 open rows, 1 fired, 0 unobservable. + +## Deferred to Follow-Up Work + +- Complete the release-loop conformance suite and its live harness cycle named by the Conformance suite roadmap row. +- Add formal custom-lane schema support only when a second consuming project defines a project lane. +- Add reviewer-output persistence only when the reviewing dispatch contract changes. + +## Open unknowns + +### Planning-time + +None. The source schema, merge-pipeline contract, repository validation entry point, and fixture pattern are present and readable. + +### Implementation-time + +- The exact lexical expressions for each probe remain implementation details and must stay limited to the seeded and holdout patterns. +- The exact temporary-directory prefix is implementation detail; the evaluator must record the resolved disposable root and reject shared paths. diff --git a/docs/specs/2026-08-30-test-integrity-review-lane-design.md b/docs/specs/2026-08-30-test-integrity-review-lane-design.md new file mode 100644 index 0000000..76240e5 --- /dev/null +++ b/docs/specs/2026-08-30-test-integrity-review-lane-design.md @@ -0,0 +1,91 @@ +--- +title: Test-integrity review sub-check +status: approved +date: 2026-08-30 +schema: spec/v1 +--- + +# Test-integrity Review Sub-check Design + +## Overview + +The standing `tests` review lane checks coverage and assertion quality, but it does not name the integrity controls that make a fixture trustworthy. A green fixture can therefore hide network access, shared temporary state, fail-open verifiers, broad assertions, undeclared environment dependencies, or host Git signing and hook state. + +This cycle adds a deterministic `test-integrity` sub-check to the existing `tests` lane. The sub-check keeps the existing `lane-findings` schema and merge pipeline, emits code-grounded findings, and ships a versioned local corpus that measures detection quality without network or model calls. + +## User Scenarios + +### S1: A reviewer audits a test-bearing diff + +A reviewer selects the standing `tests` lane. The lane explicitly runs `test-integrity` and checks each named integrity class with the same P0-P3 finding contract used by other lanes. + +### S2: A seeded fixture defect is detected + +The dogfood corpus contains at least two seeded defects and one holdout per integrity class. The evaluator reports the expected seed IDs, validates every finding against `schemas/lane-findings.schema.json`, and applies the existing merge-pipeline confidence and severity rules. + +### S3: A clean fixture remains clean + +A clean fixture is reviewed three times. It emits no P0 or P1 finding, and the three normalized finding-ID sets are identical. + +### S4: Fixture evidence is hermetic + +The dogfood run creates a disposable repository with a local Git identity, disabled signing, an empty hooks path, isolated `HOME`, isolated `TMPDIR`, and no network-capable command. The run fails closed if the fixture reaches an external target or relies on host Git configuration. + +## Scope + +### In + +- A named `test-integrity` sub-check under the standing `tests` review lane. +- Six auditable classes: network isolation, shared temporary directories, verifier fail-open behavior, assertion specificity, hidden environment dependencies, and fixture Git signing/hooks. +- A standard-schema finding emitter with evidence, impact, severity, owner, and verification fields. +- A versioned C2-TI corpus with two seeded defects and one holdout per class, plus a clean fixture. +- A deterministic evaluator with three repeated runs, precision/recall measurements, clean-fixture P0/P1 false-positive checks, schema checks, and compound-loop validation. + +### Out + +- A new finding schema or alternate merge pipeline. +- Network calls, model calls, GitHub mutations, or host credential access. +- Runtime changes to `ai-orchestration`. +- A general-purpose static-analysis framework or third-party dependency. +- Automatic fixes for test code. + +## Requirements + +- **R1**: The `tests` lane names `test-integrity` as a required sub-check and links its six classes and non-flags. +- **R2**: Every emitted finding uses the existing `schemas/lane-findings.schema.json` detail contract, including at least one code-grounded evidence item and a failure-mode `why_it_matters`. +- **R3**: The evaluator detects network access, shared temporary state, fail-open verification, weak assertions, undeclared environment reads, and host Git signing or hook inheritance. +- **R4**: The versioned C2-TI corpus contains two seeded defects and one holdout per class, a clean fixture, stable seed IDs, and expected P0-P3 severity. +- **R5**: The evaluator achieves 100% recall for seeded P0/P1 defects, at least 90% precision across expected findings, and zero clean-fixture P0/P1 findings. +- **R6**: Three identical runs produce the same normalized seed-ID set and the same finding fingerprints. +- **R7**: The evaluator validates findings against the existing schema and reuses the merge-pipeline rules for severity, confidence, deduplication, and suppression boundaries. +- **R8**: The fixture Git repository cannot inherit host signing keys or hooks; its isolated configuration is verified before review. +- **R9**: The dogfood run performs no network or model call and passes `bash scripts/validate.sh` with recursive self-invocation disabled only for the nested validation call. +- **R10**: A changed defect fixture fails by its named class and seed ID; setup failures do not count as detection evidence. + +## Design Decisions + +1. Keep `test-integrity` inside `tests` because its findings concern test trustworthiness, not a new review perspective. This preserves the always-on lane roster and existing merge semantics. +2. Use a standard-library shell entry point with embedded Python because this repository's executable conformance checks follow that pattern and have no Python package runtime. +3. Store source snippets and expectations in one versioned JSON corpus. The evaluator reads only corpus content and disposable paths, which makes the measurement reproducible and prevents hidden fixture files from becoming an untracked input. +4. Use deterministic lexical probes for the narrow named classes. A probe emits a finding only for a documented unsafe pattern and records the exact source line. The holdout cases prevent probes from firing on safe neighboring patterns. +5. Treat a missing or malformed finding as an evaluator failure, not as an empty review. A clean result is valid only after the corpus inventory, schema validation, and repeated-run checks pass. + +## Verification Contract + +The evaluator reports case-level pass/fail lines and a final summary containing the six class counts, recall, precision, clean-fixture P0/P1 count, repeated-run equality, schema status, network-call count, and nested validation status. It exits nonzero for any unmet requirement. Its findings are serialized using the same field names as the existing lane schema; no second severity vocabulary is permitted. + +## Risks and Mitigations + +| Risk | Mitigation | +|---|---| +| Lexical probes miss semantic test behavior. | Scope the claim to the named patterns, require code-grounded evidence, and keep the sub-check an explicit review prompt rather than a complete static analyzer. | +| The corpus becomes self-affirming. | Include holdouts, mutation-shaped defect removal checks, three repeated runs, and schema validation independent of expected IDs. | +| Fixture setup inherits host state. | Use isolated `HOME` and `TMPDIR`, an empty hooks directory, explicit Git config, and a command audit that rejects network tools. | +| A malformed finding appears clean. | Fail closed on missing fields, invalid enums, missing evidence, and unknown case outcomes. | + +## Acceptance Evidence + +- `bash scripts/test-review-test-integrity.sh dogfood` exits 0 and prints the C2-TI metrics. +- Removing one seeded defect changes the expected-rejection assertion and causes the corresponding case to fail at the detector assertion, not fixture setup. +- Three repeated reviews produce byte-equivalent normalized finding fingerprints. +- `bash scripts/validate.sh` exits 0 and includes the test-integrity dogfood check. diff --git a/scripts/eval-test-integrity-model.py b/scripts/eval-test-integrity-model.py new file mode 100755 index 0000000..c34a025 --- /dev/null +++ b/scripts/eval-test-integrity-model.py @@ -0,0 +1,275 @@ +#!/usr/bin/env python3 +"""Plan, execute, and score model-based test-integrity reviews.""" +import argparse +import hashlib +import json +import os +import re +import shutil +import shlex +import subprocess +import sys +import time +from pathlib import Path + +CHECKS_PATH = Path(__file__).resolve().parents[1] / "skills/reviewing/references/test-integrity.md" +SEVERITIES = ("P0", "P1", "P2", "P3") +# Case ids become file names under the plan directory; allow no path syntax. +CASE_ID = re.compile(r"[A-Za-z0-9][A-Za-z0-9._-]{0,127}") +TIMEOUT_SECONDS = 120 + + +def fail(message): + print("ERROR: " + message, file=sys.stderr) + raise SystemExit(1) + + +def digest(data): + return hashlib.sha256(data).hexdigest() + + +def load_json(path): + try: + with open(path, "r", encoding="utf-8") as stream: + return json.load(stream) + except (OSError, ValueError) as exc: + fail("cannot read JSON file %s: %s" % (path, exc)) + + +def corpus_cases(paths): + cases = [] + seen = set() + classes = {} + for path in paths: + corpus = load_json(path) + if corpus.get("schema") != "c2-ti/v1" or not isinstance(corpus.get("cases"), list): + fail("invalid c2-ti/v1 corpus: " + str(path)) + for item in corpus.get("classes", []): + if isinstance(item, dict) and item.get("id"): + classes[item["id"]] = item.get("severity") + for case in corpus["cases"]: + if not isinstance(case, dict) or not all(key in case for key in ("id", "class", "kind", "file", "source", "expected_findings")): + fail("invalid case in corpus: " + str(path)) + case_id = case["id"] + if not isinstance(case_id, str) or not CASE_ID.fullmatch(case_id) or ".." in case_id or case_id in seen: + fail("invalid or duplicate case id: " + repr(case_id)) + seen.add(case_id) + cases.append(case) + cases.sort(key=lambda case: case["id"]) + return cases, classes + + +def checks_text(): + text = CHECKS_PATH.read_text(encoding="utf-8") + start = text.index("## Checks\n") + end = text.find("\n## ", start + len("## Checks\n")) + return text[start:] if end < 0 else text[start:end] + "\n" + + +def make_request(case, checks): + system = checks + '\nReturn only JSON with this shape: {"findings":[{"integrity_class":"","severity":"P0|P1|P2|P3","line":1,"evidence":""}]}. Use an empty findings list when no integrity defect is present.' + user = "Fixture file: " + case["file"] + "\n\nSource:\n" + case["source"] + return {"messages": [{"role": "system", "content": system}, {"role": "user", "content": user}]} + + +def plan(args): + cases, _ = corpus_cases(args.corpus) + out = Path(args.out) + requests = out / "requests" + requests.mkdir(parents=True, exist_ok=True) + checks = checks_text() + entries = [] + total_bytes = 0 + for case in cases: + payload = (json.dumps(make_request(case, checks), ensure_ascii=False, indent=2) + "\n").encode("utf-8") + path = requests / (case["id"] + ".json") + path.write_bytes(payload) + total_bytes += len(payload) + entries.append({"case_id": case["id"], "request": "requests/" + path.name, "sha256": digest(payload), "bytes": len(payload)}) + manifest = {"schema": "c2-ti-model-plan/v1", "request_count": len(entries), "total_bytes": total_bytes, "requests": entries} + (out / "plan.json").write_text(json.dumps(manifest, ensure_ascii=False, indent=2) + "\n", encoding="utf-8") + print("planned %d requests (%d bytes)" % (len(entries), total_bytes)) + + +def run(args): + plan_path = Path(args.plan).resolve() + plan_bytes = plan_path.read_bytes() + if args.approval != digest(plan_bytes): + fail("approval digest does not match plan.json") + try: + argv = shlex.split(args.command) + except ValueError as exc: + fail("invalid command template: " + str(exc)) + if not argv: + fail("command must not be empty") + manifest = json.loads(plan_bytes.decode("utf-8")) + # Check every entry and request digest before the first model call. + requests = [] + for entry in manifest.get("requests", []): + case_id = entry.get("case_id") + if not isinstance(case_id, str) or not CASE_ID.fullmatch(case_id) or ".." in case_id or entry.get("request") != "requests/" + case_id + ".json": + fail("invalid plan entry: " + repr(case_id)) + request_bytes = (plan_path.parent / entry["request"]).read_bytes() + if digest(request_bytes) != entry.get("sha256"): + fail("request digest mismatch: " + case_id) + requests.append((case_id, request_bytes)) + # One run owns one fresh directory, published only after every case succeeds, + # so responses from different runs can never mix. + responses = Path(args.responses).resolve() + partial = responses.with_name(responses.name + ".partial") + if responses.exists() or partial.exists(): + fail("response directory already exists: " + str(responses)) + partial.mkdir(parents=True) + try: + for case_id, request_bytes in requests: + try: + result = subprocess.run(argv, input=request_bytes, stdout=subprocess.PIPE, stderr=subprocess.PIPE, timeout=args.timeout, check=False) + except subprocess.TimeoutExpired: + fail("command timed out for case " + case_id) + if result.returncode != 0: + fail("command failed for case %s (exit %d)" % (case_id, result.returncode)) + (partial / (case_id + ".json")).write_bytes(result.stdout) + except BaseException: + shutil.rmtree(partial, ignore_errors=True) + raise + partial.rename(responses) + print("stored %d raw responses in %s" % (len(requests), responses)) + + +def parse_findings(path, known_classes): + try: + raw = path.read_text(encoding="utf-8") + value = json.loads(raw) + if isinstance(value, dict) and "result" in value: + value = value["result"] + if isinstance(value, str): + value = json.loads(value) + if not isinstance(value, dict) or not isinstance(value.get("findings"), list): + return None + found = [] + for finding in value["findings"]: + if not isinstance(finding, dict): + return None + cls, severity = finding.get("integrity_class"), finding.get("severity") + line, evidence = finding.get("line"), finding.get("evidence") + if not isinstance(cls, str) or cls not in known_classes or not isinstance(severity, str) or severity not in SEVERITIES: + return None + if not isinstance(line, int) or isinstance(line, bool) or line < 1 or not isinstance(evidence, str): + return None + found.append(finding) + return found + except (OSError, UnicodeError, ValueError): + return None + + +def score(args): + cases, classes = corpus_cases(args.corpus) + # The scored cases must be exactly the approved plan's cases. + plan_ids = sorted(entry.get("case_id") for entry in load_json(args.plan).get("requests", [])) + if plan_ids != sorted(case["id"] for case in cases): + fail("corpus cases do not match the plan's cases") + response_dirs = [Path(item) for item in args.responses] + if len(response_dirs) != args.repeat: + fail("--repeat %d requires exactly %d response directories" % (args.repeat, args.repeat)) + if args.repeat < 1: + fail("--repeat must be at least 1") + if len({item.resolve() for item in response_dirs}) != len(response_dirs): + fail("each repeat needs its own response directory") + expected_files = {case["id"] + ".json" for case in cases} + for response_dir in response_dirs: + extra = sorted(item.name for item in response_dir.glob("*") if item.name not in expected_files) if response_dir.is_dir() else [] + if extra: + fail("unexpected files in %s: %s" % (response_dir, ", ".join(extra))) + predictions = [] + malformed = [] + details = [] + for response_dir in response_dirs: + batch = {} + for case in cases: + findings = parse_findings(response_dir / (case["id"] + ".json"), classes) + if findings is None: + malformed.append({"case_id": case["id"], "response_dir": str(response_dir)}) + batch[case["id"]] = None + else: + batch[case["id"]] = findings + predictions.append(batch) + repeat_mismatch = [] + if args.repeat > 1: + for case in cases: + sets = [None if batch[case["id"]] is None else sorted(set(f["integrity_class"] for f in batch[case["id"]])) for batch in predictions] + if any(item != sets[0] for item in sets[1:]): + repeat_mismatch.append(case["id"]) + tp = fp = expected_positive = p01_tp = p01_expected = p01_negative_fp = severity_match = predicted_count = 0 + per_class = {} + for class_id, class_severity in sorted(classes.items()): + per_class[class_id] = {"expected": 0, "true_positive": 0, "false_positive": 0, "severity_agreement": 0} + for case in cases: + expected = bool(case["expected_findings"]) + class_id = case["class"] + class_severity = classes.get(class_id) + if expected: + expected_positive += 1 + per_class.setdefault(class_id, {"expected": 0, "true_positive": 0, "false_positive": 0, "severity_agreement": 0})["expected"] += 1 + if class_severity in ("P0", "P1"): + p01_expected += 1 + findings = predictions[0][case["id"]] + if findings is None: + details.append({"case_id": case["id"], "status": "malformed-or-missing"}) + continue + predicted = set(f["integrity_class"] for f in findings) + predicted_count += len(findings) + if expected and class_id in predicted: + tp += 1 + per_class[class_id]["true_positive"] += 1 + if class_severity in ("P0", "P1"): + p01_tp += 1 + if any(f["integrity_class"] == class_id and f["severity"] == class_severity for f in findings): + severity_match += 1 + per_class[class_id]["severity_agreement"] += 1 + for f in findings: + pred_class = f["integrity_class"] + correct = expected and pred_class == class_id + if not correct: + fp += 1 + per_class.setdefault(pred_class, {"expected": 0, "true_positive": 0, "false_positive": 0, "severity_agreement": 0})["false_positive"] += 1 + if not expected and f["severity"] in ("P0", "P1"): + p01_negative_fp += 1 + if pred_class not in classes: + # Unknown class identifiers are malformed schema values. + malformed.append({"case_id": case["id"], "reason": "unknown class " + pred_class}) + details.append({"case_id": case["id"], "expected_class": class_id if expected else None, "predicted_classes": sorted(predicted)}) + precision = float(tp) / predicted_count if predicted_count else (1.0 if expected_positive == 0 else 0.0) + recall = float(tp) / expected_positive if expected_positive else 1.0 + p01_recall = float(p01_tp) / p01_expected if p01_expected else 1.0 + report = {"positive_recall": recall, "p0_p1_recall": p01_recall, "precision": precision, "clean_negative_p0_p1_false_positives": p01_negative_fp, "malformed_responses": malformed, "repeat_mismatches": repeat_mismatch, "severity_agreement_count": severity_match, "predicted_finding_count": predicted_count, "true_positive_count": tp, "false_positive_count": fp, "per_class": per_class, "cases": details} + print(json.dumps(report, indent=2, sort_keys=True)) + if p01_recall < 1.0 or precision < 0.90 or p01_negative_fp or malformed or repeat_mismatch: + raise SystemExit(1) + + +def main(): + parser = argparse.ArgumentParser(description=__doc__) + sub = parser.add_subparsers(dest="action", required=True) + plan_parser = sub.add_parser("plan") + plan_parser.add_argument("--corpus", action="append", required=True) + plan_parser.add_argument("--out", required=True) + plan_parser.set_defaults(func=plan) + run_parser = sub.add_parser("run") + run_parser.add_argument("--plan", required=True) + run_parser.add_argument("--command", required=True) + run_parser.add_argument("--approval", required=True) + run_parser.add_argument("--responses", required=True, help="fresh directory for this run's raw responses") + run_parser.add_argument("--timeout", type=float, default=TIMEOUT_SECONDS) + run_parser.set_defaults(func=run) + score_parser = sub.add_parser("score") + score_parser.add_argument("--plan", required=True) + score_parser.add_argument("--responses", nargs="+", required=True) + score_parser.add_argument("--corpus", action="append", required=True) + score_parser.add_argument("--repeat", type=int, default=1) + score_parser.set_defaults(func=score) + args = parser.parse_args() + args.func(args) + + +if __name__ == "__main__": + main() diff --git a/scripts/model-chat-adapter.py b/scripts/model-chat-adapter.py new file mode 100755 index 0000000..070a0d9 --- /dev/null +++ b/scripts/model-chat-adapter.py @@ -0,0 +1,55 @@ +#!/usr/bin/env python3 +"""Tool-free model command for scripts/eval-test-integrity-model.py run. + +Reads one planned request ({"messages": [...]}) on stdin, sends exactly those +messages to an OpenAI-compatible chat-completions endpoint, and writes the +model's reply text to stdout unchanged. The model gets no tools, shell, files, +or MCP servers, so it cannot read the corpus answers. + +Environment: + C2TI_MODEL_BASE_URL endpoint base, e.g. https://host/v1 + C2TI_MODEL model id + C2TI_MODEL_TOKEN bearer token +""" + +import json +import os +import sys +import urllib.error +import urllib.request + + +class _NoRedirect(urllib.request.HTTPRedirectHandler): + """Fail on any redirect: following one would resend the bearer token.""" + + def redirect_request(self, req, fp, code, msg, headers, newurl): + raise urllib.error.HTTPError(req.full_url, code, "redirect refused", headers, fp) + + +def main(): + base = os.environ.get("C2TI_MODEL_BASE_URL", "").rstrip("/") + model = os.environ.get("C2TI_MODEL", "") + token = os.environ.get("C2TI_MODEL_TOKEN", "") + if not base or not model or not token: + sys.stderr.write("C2TI_MODEL_BASE_URL, C2TI_MODEL and C2TI_MODEL_TOKEN are required\n") + return 2 + request = json.loads(sys.stdin.buffer.read()) + body = json.dumps({"model": model, "messages": request["messages"], "temperature": 0}).encode("utf-8") + http = urllib.request.Request( + base + "/chat/completions", + data=body, + headers={ + "Authorization": "Bearer " + token, + "Content-Type": "application/json", + "User-Agent": "c2ti-eval/1.0", + }, + ) + opener = urllib.request.build_opener(_NoRedirect) + with opener.open(http, timeout=110) as response: + payload = json.load(response) + sys.stdout.write(payload["choices"][0]["message"]["content"] or "") + return 0 + + +if __name__ == "__main__": + sys.exit(main()) diff --git a/scripts/test-eval-test-integrity-model.sh b/scripts/test-eval-test-integrity-model.sh new file mode 100755 index 0000000..daed51a --- /dev/null +++ b/scripts/test-eval-test-integrity-model.sh @@ -0,0 +1,221 @@ +#!/usr/bin/env bash +set -uo pipefail +ROOT="$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd)" +python3 - "$ROOT" <<'PY' +import hashlib +import json +import os +import pathlib +import subprocess +import sys +import tempfile + +root = pathlib.Path(sys.argv[1]) +script = root / "scripts/eval-test-integrity-model.py" +classes = [ + {"id": "network-isolation", "severity": "P1"}, + {"id": "assertion-specificity", "severity": "P2"}, +] +cases = [ + {"id": "CASE-SECRET-1", "class": "network-isolation", "kind": "seed", "file": "fixture.py", "source": "requests.get('https://example.invalid')", "expected_findings": [{"seed_id": "CASE-SECRET-1", "severity": "P1"}]}, + {"id": "CASE-SECRET-CLEAN", "class": "network-isolation", "kind": "clean", "file": "clean.py", "source": "assert 1 == 1", "expected_findings": []}, +] + +call_log = None + +def call(args, expected=0): + env = dict(os.environ, C2TI_FAKE_LOG=str(call_log)) if call_log else None + result = subprocess.run([sys.executable, str(script)] + args, text=True, stdout=subprocess.PIPE, stderr=subprocess.PIPE, env=env) + if (result.returncode == 0) != (expected == 0): + raise AssertionError("unexpected exit %d: %s\n%s" % (result.returncode, result.stdout, result.stderr)) + return result + +with tempfile.TemporaryDirectory(prefix="c2ti-model-test-") as temporary: + base = pathlib.Path(temporary) + call_log = base / "calls.log" + corpus = base / "corpus.json" + corpus.write_text(json.dumps({"schema": "c2-ti/v1", "classes": classes, "cases": cases}), encoding="utf-8") + plan_dir = base / "plan" + call(["plan", "--corpus", str(corpus), "--out", str(plan_dir)]) + plan_path = plan_dir / "plan.json" + manifest = json.loads(plan_path.read_text(encoding="utf-8")) + assert manifest["request_count"] == 2 + request = json.loads((plan_dir / "requests/CASE-SECRET-1.json").read_text(encoding="utf-8")) + encoded = json.dumps(request) + for forbidden in ("CASE-SECRET-1", "seed_id", "expected_findings"): + assert forbidden not in encoded, "prompt leaked " + forbidden + user_content = request["messages"][1]["content"] + assert "network-isolation" not in user_content + assert "seed" not in user_content + approval = hashlib.sha256(plan_path.read_bytes()).hexdigest() + call(["run", "--plan", str(plan_path), "--command", "true", "--approval", "0" * 64], expected=1) + fake = base / "fake_model.py" + fake.write_text( + "import json, os, sys\n" + "request = json.load(sys.stdin)\n" + "open(os.environ['C2TI_FAKE_LOG'], 'a').write('call\\n')\n" + "source = request['messages'][1]['content']\n" + "findings = [] if 'clean.py' in source else [{'integrity_class':'network-isolation','severity':'P1','line':1,'evidence':'network call'}]\n" + "print(json.dumps({'findings': findings}))\n", + encoding="utf-8", + ) + import shlex + fake_command = shlex.quote(sys.executable) + " " + shlex.quote(str(fake)) + responses_dir = base / "run-1" + call(["run", "--plan", str(plan_path), "--command", fake_command, "--approval", approval, "--responses", str(responses_dir)]) + # run's stored responses are the ones score evaluates, and they pass. + call(["score", "--plan", str(plan_path), "--responses", str(responses_dir), "--corpus", str(corpus)]) + # A second run never mixes into an existing response directory. + calls_before = call_log.read_text().count("call") + rerun = call(["run", "--plan", str(plan_path), "--command", fake_command, "--approval", approval, "--responses", str(responses_dir)], expected=1) + assert "already exists" in rerun.stderr, rerun.stderr + assert call_log.read_text().count("call") == calls_before, "rerun called the model before refusing" + # A run that fails part-way leaves no response directory to score. + failing = base / "failing_model.py" + failing.write_text( + "import json, sys\n" + "request = json.load(sys.stdin)\n" + "if 'clean.py' in request['messages'][1]['content']:\n" + " sys.exit(3)\n" + "print(json.dumps({'findings': []}))\n", + encoding="utf-8", + ) + partial_dir = base / "run-partial" + call(["run", "--plan", str(plan_path), "--command", shlex.quote(sys.executable) + " " + shlex.quote(str(failing)), "--approval", approval, "--responses", str(partial_dir)], expected=1) + assert not partial_dir.exists(), "failed run left a scorable response directory" + assert not partial_dir.with_name(partial_dir.name + ".partial").exists(), "failed run left its partial directory" + + # A request file changed after approval is refused before any call. + tamper_req_dir = base / "tamper-request" + call(["plan", "--corpus", str(corpus), "--out", str(tamper_req_dir)]) + tamper_req_plan = tamper_req_dir / "plan.json" + tamper_req_approval = hashlib.sha256(tamper_req_plan.read_bytes()).hexdigest() + tampered_request = json.loads((tamper_req_dir / "requests/CASE-SECRET-1.json").read_text(encoding="utf-8")) + tampered_request["messages"][1]["content"] += "\n# changed after approval" + (tamper_req_dir / "requests/CASE-SECRET-1.json").write_text(json.dumps(tampered_request), encoding="utf-8") + calls_before = call_log.read_text().count("call") + tamper_run = call(["run", "--plan", str(tamper_req_plan), "--command", fake_command, "--approval", tamper_req_approval, "--responses", str(base / "tamper-request-out")], expected=1) + assert "digest mismatch" in tamper_run.stderr, tamper_run.stderr + assert call_log.read_text().count("call") == calls_before, "tampered plan reached the model" + assert not (base / "tamper-request-out").exists() + + good = base / "good" + good.mkdir() + (good / "CASE-SECRET-1.json").write_text(json.dumps({"findings": [{"integrity_class": "network-isolation", "severity": "P1", "line": 1, "evidence": "network call"}]}), encoding="utf-8") + (good / "CASE-SECRET-CLEAN.json").write_text('{"findings": []}', encoding="utf-8") + call(["score", "--plan", str(plan_path), "--responses", str(good), "--corpus", str(corpus)]) + + malformed = base / "malformed" + malformed.mkdir() + (malformed / "CASE-SECRET-1.json").write_text("not json", encoding="utf-8") + call(["score", "--plan", str(plan_path), "--responses", str(malformed), "--corpus", str(corpus)], expected=1) + missing = base / "missing" + missing.mkdir() + call(["score", "--plan", str(plan_path), "--responses", str(missing), "--corpus", str(corpus)], expected=1) + # Extra files and unknown classes are not silently accepted. + extra = base / "extra" + extra.mkdir() + good_finding = json.dumps({"findings": [{"integrity_class": "network-isolation", "severity": "P1", "line": 1, "evidence": "network call"}]}) + for name, body in (("CASE-SECRET-1.json", good_finding), ("CASE-SECRET-CLEAN.json", '{"findings": []}'), ("CASE-OTHER.json", '{"findings": []}')): + (extra / name).write_text(body, encoding="utf-8") + extra_result = call(["score", "--plan", str(plan_path), "--responses", str(extra), "--corpus", str(corpus)], expected=1) + assert "unexpected files" in extra_result.stderr, extra_result.stderr + unknown = base / "unknown-class" + unknown.mkdir() + (unknown / "CASE-SECRET-1.json").write_text(json.dumps({"findings": [{"integrity_class": "network-isolation", "severity": "P1", "line": 1, "evidence": "network call"}, {"integrity_class": "made-up", "severity": "P1", "line": 1, "evidence": "x"}]}), encoding="utf-8") + (unknown / "CASE-SECRET-CLEAN.json").write_text('{"findings": []}', encoding="utf-8") + unknown_result = call(["score", "--plan", str(plan_path), "--responses", str(unknown), "--corpus", str(corpus)], expected=1) + assert "malformed-or-missing" in unknown_result.stdout, unknown_result.stdout + # A non-string class is reported as malformed, not a crash. + odd = base / "odd-class" + odd.mkdir() + (odd / "CASE-SECRET-1.json").write_text(json.dumps({"findings": [{"integrity_class": ["x"], "severity": "P1", "line": 1, "evidence": "x"}]}), encoding="utf-8") + (odd / "CASE-SECRET-CLEAN.json").write_text('{"findings": []}', encoding="utf-8") + odd_result = call(["score", "--plan", str(plan_path), "--responses", str(odd), "--corpus", str(corpus)], expected=1) + assert "malformed-or-missing" in odd_result.stdout and "Traceback" not in odd_result.stderr + + # The scored corpus must be exactly the approved plan's cases. + subset = base / "subset.json" + subset.write_text(json.dumps({"schema": "c2-ti/v1", "classes": classes, "cases": [cases[0]]}), encoding="utf-8") + subset_dir = base / "subset-responses" + subset_dir.mkdir() + (subset_dir / "CASE-SECRET-1.json").write_text(good_finding, encoding="utf-8") + subset_result = call(["score", "--plan", str(plan_path), "--responses", str(subset_dir), "--corpus", str(subset)], expected=1) + assert "do not match the plan" in subset_result.stderr, subset_result.stderr + + + cross = base / "cross" + cross.mkdir() + (cross / "CASE-SECRET-1.json").write_text(json.dumps({"findings": [{"integrity_class": "assertion-specificity", "severity": "P2", "line": 1, "evidence": "wrong class"}]}), encoding="utf-8") + (cross / "CASE-SECRET-CLEAN.json").write_text('{"findings": []}', encoding="utf-8") + cross_result = call(["score", "--plan", str(plan_path), "--responses", str(cross), "--corpus", str(corpus)], expected=1) + cross_report = json.loads(cross_result.stdout) + assert cross_report["false_positive_count"] == 1 + + repeat_a = base / "repeat-a" + repeat_b = base / "repeat-b" + repeat_a.mkdir(); repeat_b.mkdir() + for directory in (repeat_a, repeat_b): + (directory / "CASE-SECRET-CLEAN.json").write_text('{"findings": []}', encoding="utf-8") + (repeat_a / "CASE-SECRET-1.json").write_text('{"findings": []}', encoding="utf-8") + (repeat_b / "CASE-SECRET-1.json").write_text(json.dumps({"findings": [{"integrity_class": "network-isolation", "severity": "P1", "line": 1, "evidence": "network call"}]}), encoding="utf-8") + repeat_result = call(["score", "--plan", str(plan_path), "--responses", str(repeat_a), str(repeat_b), "--corpus", str(corpus), "--repeat", "2"], expected=1) + assert "repeat_mismatches" in repeat_result.stdout + # The same directory cannot stand in for several repeats. + call(["score", "--plan", str(plan_path), "--responses", str(good), str(good), "--corpus", str(corpus), "--repeat", "2"], expected=1) + + # Case ids and plan entries become file names; a traversal id must be + # rejected before anything is written outside the plan directory. + escape = base / "escape-target" + evil_corpus = base / "evil.json" + evil_case = dict(cases[0], id="../../escape-target") + evil_case["expected_findings"] = [{"seed_id": evil_case["id"], "severity": "P1"}] + evil_corpus.write_text(json.dumps({"schema": "c2-ti/v1", "classes": classes, "cases": [evil_case]}), encoding="utf-8") + call(["plan", "--corpus", str(evil_corpus), "--out", str(base / "evil-plan" / "x")], expected=1) + assert not list(base.rglob("escape-target*")), "plan wrote outside its directory" + + tampered_dir = base / "tampered" + call(["plan", "--corpus", str(corpus), "--out", str(tampered_dir)]) + tampered_path = tampered_dir / "plan.json" + tampered = json.loads(tampered_path.read_text(encoding="utf-8")) + tampered["requests"][0]["case_id"] = "../../escape-target" + tampered_path.write_text(json.dumps(tampered), encoding="utf-8") + tampered_approval = hashlib.sha256(tampered_path.read_bytes()).hexdigest() + call(["run", "--plan", str(tampered_path), "--command", fake_command, "--approval", tampered_approval, "--responses", str(base / "tampered-out")], expected=1) + assert not list(base.rglob("escape-target*")), "run wrote outside its directory" + +# The adapter refuses HTTP redirects, so the bearer token never follows one. +import http.server +import threading + +seen_auth = [] + +class Handler(http.server.BaseHTTPRequestHandler): + def do_POST(self): + self.send_response(302) + self.send_header("Location", "http://127.0.0.1:%d/elsewhere" % self.server.server_address[1]) + self.end_headers() + + def do_GET(self): + seen_auth.append(self.headers.get("Authorization")) + self.send_response(200) + self.end_headers() + self.wfile.write(b'{"choices":[{"message":{"content":"{}"}}]}') + + def log_message(self, *args): + pass + +server = http.server.HTTPServer(("127.0.0.1", 0), Handler) +thread = threading.Thread(target=server.serve_forever, daemon=True) +thread.start() +try: + env = dict(os.environ, C2TI_MODEL_BASE_URL="http://127.0.0.1:%d/v1" % server.server_address[1], C2TI_MODEL="m", C2TI_MODEL_TOKEN="sentinel-token") + adapter = subprocess.run([sys.executable, str(root / "scripts/model-chat-adapter.py")], input=b'{"messages": []}', env=env, stdout=subprocess.PIPE, stderr=subprocess.PIPE, timeout=30) +finally: + server.shutdown() +assert adapter.returncode != 0, "adapter followed a redirect" +assert seen_auth == [], "bearer token was sent to the redirect target" +assert b"sentinel-token" not in adapter.stdout + adapter.stderr + +print("ALL MODEL EVALUATOR FIXTURES PASSED") +PY diff --git a/scripts/test-review-test-integrity.sh b/scripts/test-review-test-integrity.sh new file mode 100755 index 0000000..bbee978 --- /dev/null +++ b/scripts/test-review-test-integrity.sh @@ -0,0 +1,722 @@ +#!/usr/bin/env bash +# C2-TI deterministic fixture evaluator. It uses only the standard library. +set -uo pipefail + +ROOT="$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd)" +MODE="${1:-dogfood}" + +if [[ "$MODE" == "contract" ]]; then + python3 - "$ROOT/skills/reviewing/references/lanes.md" "$ROOT/skills/reviewing/SKILL.md" "$ROOT/skills/reviewing/references/test-integrity.md" "$ROOT/skills/reviewing/references/merge-pipeline.md" <<'PY' +import sys +from pathlib import Path + +text = "\n".join(Path(path).read_text(encoding="utf-8") for path in sys.argv[1:]) +required = ( + "test-integrity", + "network isolation", + "shared temporary directories", + "verifier fail-open behavior", + "assertion specificity", + "hidden environment dependencies", + "fixture Git signing/hooks", + "schemas/lane-findings.schema.json", + "references/merge-pipeline.md", + "every phase-gate review", + "test-integrity-run/v1", + "missing, cancelled, or malformed", +) +missing = [item for item in required if item not in text] +if missing: + raise SystemExit("C2-TI contract missing: " + ", ".join(missing)) +print("C2-TI contract present") +PY + exit $? +fi + +if [[ "$MODE" == "gate" ]]; then + python3 - "$ROOT/scripts/validate.sh" <<'PY' +import sys +from pathlib import Path + +text = Path(sys.argv[1]).read_text(encoding="utf-8") +required = ( + "test-review-test-integrity.sh\" dogfood", + "COMPOUND_LOOP_SKIP_TEST_INTEGRITY", + "COMPOUND_LOOP_IN_VALIDATE", +) +missing = [item for item in required if item not in text] +if missing: + raise SystemExit("C2-TI validation gate missing: " + ", ".join(missing)) +print("C2-TI validation gate present") +PY + exit $? +fi + +python3 - "$ROOT" "$MODE" <<'PY' +import hashlib +import json +import os +import re +import shutil +import subprocess +import sys +import tempfile +from pathlib import Path + +root = Path(sys.argv[1]).resolve() +mode = sys.argv[2] +corpus_path = Path(os.environ.get("C2_TI_CORPUS", root / "tests/conformance/review/test-integrity-corpus.json")) +schema_path = root / "schemas/lane-findings.schema.json" + +if mode not in {"dogfood", "seeded", "clean", "isolation", "records", "record"}: + raise SystemExit(f"C2-TI unknown mode: {mode}") + +corpus = json.loads(corpus_path.read_text(encoding="utf-8")) +lane_schema = json.loads(schema_path.read_text(encoding="utf-8")) +if corpus.get("schema") != "c2-ti/v1": + raise SystemExit("C2-TI corpus schema is unknown") + +class_ids = { + "network-isolation", + "shared-temporary-directories", + "verifier-fail-open", + "assertion-specificity", + "hidden-environment-dependencies", + "fixture-git-signing-hooks", +} +CLASS_SEVERITY = { + "network-isolation": "P1", + "shared-temporary-directories": "P1", + "verifier-fail-open": "P1", + "assertion-specificity": "P2", + "hidden-environment-dependencies": "P1", + "fixture-git-signing-hooks": "P1", +} +classes = {item["id"]: item for item in corpus.get("classes", [])} +if set(classes) != class_ids: + raise SystemExit(f"C2-TI class inventory mismatch: {sorted(classes)}") +for class_id, severity in CLASS_SEVERITY.items(): + if classes[class_id].get("severity") != severity: + raise SystemExit(f"C2-TI corpus class severity disagrees with analyzer for {class_id}") + +cases = corpus.get("cases", []) +case_ids = [case.get("id") for case in cases] +if any(not case_id for case_id in case_ids) or len(set(case_ids)) != len(case_ids): + raise SystemExit("C2-TI case IDs must be non-empty and unique") +for case in cases: + if case.get("kind") not in {"seed", "holdout", "clean"}: + raise SystemExit(f"C2-TI unknown case kind: {case.get('id')}") + if case.get("kind") != "clean" and case.get("class") not in class_ids: + raise SystemExit(f"C2-TI unknown class: {case.get('id')}") + if not isinstance(case.get("source"), str) or not case["source"]: + raise SystemExit(f"C2-TI source missing: {case.get('id')}") + if not isinstance(case.get("expected_findings"), list): + raise SystemExit(f"C2-TI expected findings missing: {case.get('id')}") +for class_id in class_ids: + members = [case for case in cases if case.get("class") == class_id] + if len([case for case in members if case["kind"] == "seed"]) < 2: + raise SystemExit(f"C2-TI requires at least two seeds for {class_id}") +if len([case for case in cases if case["kind"] == "clean"]) < 1: + raise SystemExit("C2-TI requires at least one clean fixture") +extra_holdouts = [] +holdout_path = os.environ.get("C2_TI_HOLDOUTS") +if holdout_path: + extra = json.loads(Path(holdout_path).read_text(encoding="utf-8")) + if extra.get("schema") != "c2-ti/v1": + raise SystemExit("C2-TI holdout corpus schema is unknown") + extra_holdouts = extra.get("cases", []) + if any(case.get("kind") != "holdout" for case in extra_holdouts): + raise SystemExit("C2-TI external corpus may contain holdouts only") + known_ids = set(case_ids) + for case in extra_holdouts: + if case.get("id") in known_ids or case.get("class") not in class_ids: + raise SystemExit(f"C2-TI invalid or duplicate external holdout: {case.get('id')}") + known_ids.add(case["id"]) + if not isinstance(case.get("source"), str) or not case["source"] or not isinstance(case.get("expected_findings"), list): + raise SystemExit(f"C2-TI malformed external holdout: {case.get('id')}") + for class_id in class_ids: + members = [case for case in extra_holdouts if case["class"] == class_id] + if not any(case["expected_findings"] for case in members) or not any(not case["expected_findings"] for case in members): + raise SystemExit(f"C2-TI external holdouts require positive and negative cases for {class_id}") + +PROBES = { + "network-isolation": ( + re.compile(r"\b(?:requests|httpx|urllib\.request|aiohttp)\.(?:get|post|put|delete|request|urlopen|urlretrieve)\s*\("), + re.compile(r"\b(?:socket\.(?:create_connection|socket|connect)|create_connection|urlopen|urlretrieve)\s*\("), + re.compile(r"\b(?:curl|wget|ssh|scp|sftp|nc|ncat|netcat|ping|ftp)\b.{0,180}(?:https?://|ftp://|\s-[A-Za-z]*[pP]|@[\w.-]+)"), + re.compile(r"\b(?:pip|pip3|uv|npm|npx|yarn|pnpm)\s+(?:install|download|add|exec)\b"), + re.compile(r"\b(?:git|git-remote-[\w-]+)\b.{0,160}\b(?:clone|fetch|pull|push)\b.{0,160}(?:https?://|ssh://|git@)"), + re.compile(r"subprocess\.(?:run|call|Popen|check_call|check_output)\s*\([^)]*(?:curl|wget|ssh|scp|pip|uv|npm|https?://)"), + ), + "shared-temporary-directories": ( + re.compile(r"(?:[\"']|Path\s*\()/?(?:tmp|var/tmp)/[^\"']+"), + re.compile(r"tempfile\.gettempdir\(\)(?!\s*/\s*(?:tempfile\.)?mkdtemp)"), + re.compile(r"(?:mkdir(?:\s+-p)?|Path\([^)]*\)\.mkdir)[^\n]*(?:/tmp/|/var/tmp/|(?:integration|project|fixture|test|output|cache)[-_](?:artifacts|data|dir|root|path|files))"), + re.compile(r"(?:shutil\.)?rmtree\(\s*[\"'](?:/tmp|/var/tmp|fixture|test|output|cache|repo)[^\"']*[\"']"), + re.compile(r"\brm\s+-[^\n]*\s(?:/tmp|/var/tmp|[\"']?(?:fixture|test|output|cache|repo)[-_](?:dir|root|path|files))"), + re.compile(r"(?:fixture|output|cache|repo)[-_](?:dir|root|path|socket|log|repo)\s*=\s*(?:[\"']|Path\s*\()"), + ), + "verifier-fail-open": ( + re.compile(r"except(?:\s+[^:]+)?:\s*(?:return\s+(?:True|['\"](?:clean|pass|success)['\"]|\[\])|pass)\b"), + re.compile(r"subprocess\.(?:run|call|check_call|check_output)\([^\n]*\bcheck\s*=\s*False"), + re.compile(r"(?:verdict|status|result)\s*=\s*['\"](?:clean|pass|success)['\"]"), + re.compile(r"(?:return|default)\s*[:=]\s*(?:True|['\"](?:clean|pass|success)['\"]|\[\])"), + re.compile(r"\|\|\s*true\b"), + re.compile(r"\b(?:except|catch)\b[^\n]*(?:Exception|Error)[^\n]*(?:continue|pass|return\s+None|return\s+True)"), + re.compile(r"\b(?:result|completed|response)\s*=\s*subprocess\.(?:run|call)\s*\("), + ), + "assertion-specificity": ( + re.compile(r"^\s*assert\s+(?:True|False|[A-Za-z_]\w*(?:\.[A-Za-z_]\w*)?)\s*(?:#.*)?$"), + re.compile(r"^\s*assert\s+['\"][^'\"]+['\"]\s+in\s+[^#]+$"), + re.compile(r"^\s*assert\s+[^#]*(?:stdout|stderr|output)\s+in\s+[^#]+$"), + re.compile(r"^\s*assert\s+len\([^)]*\)\s*>\s*0\s*(?:#.*)?$"), + re.compile(r"^\s*assert\s+[^#]*\.exit_code\s*(?:==\s*0)?\s*(?:#.*)?$"), + re.compile(r"\bgrep\s+-q\s+['\"][^'\"]+['\"]"), + ), + "hidden-environment-dependencies": ( + re.compile(r"os\.environ\[\s*['\"](?:HOME|PATH|TMPDIR|SSH_AUTH_SOCK|HTTP_PROXY|HTTPS_PROXY|ALL_PROXY|NO_PROXY|GPG_TTY|LANG|LC_ALL|LC_CTYPE|TZ|GIT_CONFIG_[A-Z_]+|[A-Z_]*(?:TOKEN|KEY|PASSWORD))['\"]\s*\]"), + re.compile(r"os\.(?:getenv|environ\.get)\(\s*['\"](?:HOME|PATH|TMPDIR|SSH_AUTH_SOCK|HTTP_PROXY|HTTPS_PROXY|ALL_PROXY|NO_PROXY|GPG_TTY|LANG|LC_ALL|LC_CTYPE|TZ|GIT_CONFIG_[A-Z_]+|[A-Z_]*(?:TOKEN|KEY|PASSWORD))['\"]"), + re.compile(r"\bPath\.home\s*\("), + re.compile(r"\b(?:os\.path\.expanduser|expanduser)\s*\("), + re.compile(r"\b(?:getpass\.get(?:user|pass)|getuser)\s*\("), + re.compile(r"\b(?:shutil\.which|which)\s*\("), + re.compile(r"\bcommand\s+-v\s+[A-Za-z0-9_.+-]+"), + re.compile(r"\b(?:locale\.getpreferredencoding|locale\.getlocale|time\.tzname|datetime\.now)\s*\("), + re.compile(r"\b(?:env|printenv)\s+(?:HOME|PATH|TMPDIR|LANG|LC_ALL|TZ|HTTP_PROXY|HTTPS_PROXY|ALL_PROXY|NO_PROXY)\b"), + re.compile(r"\bgit\s+config\s+(?:--global|--system)\b"), + ), + "fixture-git-signing-hooks": ( + re.compile(r"['\"]git['\"].{0,140}['\"]config['\"].{0,140}['\"]--(?:global|system)['\"]"), + re.compile(r"\bgit\s+config\s+(?:--global|--system)\b"), + ), +} +def validate_json_schema(value, schema, path="root"): + expected_type = schema.get("type") + if expected_type: + types = {expected_type} if isinstance(expected_type, str) else set(expected_type) + + def matches(type_name): + if type_name == "object": + return isinstance(value, dict) + if type_name == "array": + return isinstance(value, list) + if type_name == "string": + return isinstance(value, str) + if type_name == "integer": + return isinstance(value, int) and not isinstance(value, bool) + if type_name == "boolean": + return isinstance(value, bool) + if type_name == "null": + return value is None + return True + + if not any(matches(type_name) for type_name in types): + raise AssertionError(f"C2-TI schema: {path} has an invalid type") + if "enum" in schema and value not in schema["enum"]: + raise AssertionError(f"C2-TI schema: {path} has an invalid enum value") + if isinstance(value, str) and "maxLength" in schema and len(value) > schema["maxLength"]: + raise AssertionError(f"C2-TI schema: {path} exceeds maxLength") + if isinstance(value, int) and "minimum" in schema and value < schema["minimum"]: + raise AssertionError(f"C2-TI schema: {path} is below minimum") + if isinstance(value, list): + if "minItems" in schema and len(value) < schema["minItems"]: + raise AssertionError(f"C2-TI schema: {path} has too few items") + item_schema = schema.get("items") + if item_schema: + for index, item in enumerate(value): + validate_json_schema(item, item_schema, f"{path}[{index}]") + if isinstance(value, dict): + for key in schema.get("required", []): + if key not in value: + raise AssertionError(f"C2-TI schema: {path} missing {key}") + for key, property_schema in schema.get("properties", {}).items(): + if key in value: + validate_json_schema(value[key], property_schema, f"{path}.{key}") + + +def validate_envelope(envelope): + validate_json_schema(envelope, lane_schema, "envelope") + if envelope["reviewer"] != "test-integrity": + raise AssertionError("C2-TI schema: invalid reviewer") + for finding in envelope["findings"]: + if not finding["evidence"] or not finding["why_it_matters"]: + raise AssertionError("C2-TI schema: detail field is empty") + + +def finding_for(case, class_id, line_number, source_line): + severity = CLASS_SEVERITY[class_id] + titles = { + "network-isolation": "Fixture can reach an external target", + "shared-temporary-directories": "Fixture uses a shared temporary path", + "verifier-fail-open": "Verifier turns failure into success", + "assertion-specificity": "Assertion does not check the contract", + "hidden-environment-dependencies": "Fixture reads host environment state", + "fixture-git-signing-hooks": "Fixture inherits Git signing or hooks", + } + impacts = { + "network-isolation": "An external target can disclose fixture credentials or mutate state.", + "shared-temporary-directories": "Concurrent runs can overwrite or read another run's evidence.", + "verifier-fail-open": "A malformed or failed verification can become a false green result.", + "assertion-specificity": "The test can pass while the required observable contract is broken.", + "hidden-environment-dependencies": "A host-specific value can change the verdict outside the fixture.", + "fixture-git-signing-hooks": "Host signing or hook state can execute code or make commits unreproducible.", + } + # The case ID identifies its seeded defect; cross-class matches are distinct + # observations and therefore cannot be mistaken for expected true positives. + seed_id = case["id"] if case.get("class") == class_id else f"{case['id']}@{class_id}" + return { + "seed_id": seed_id, + "integrity_class": class_id, + "title": titles[class_id], + "severity": severity, + "file": case["file"], + "line": line_number, + "why_it_matters": impacts[class_id], + "suggested_fix": "Declare the dependency in the disposable fixture and assert the protected boundary.", + "autofix_class": "gated_auto", + "owner": "downstream-resolver", + "requires_verification": True, + "confidence": 100, + "evidence": [f"{case['file']}:{line_number}: {source_line.strip()}"], + "pre_existing": False, + } + + +def analyze(case): + findings = [] + source_lines = case["source"].splitlines() + output_asserted = any( + re.search(r"^\s*assert\s+[^#]*(?:stdout|stderr|output)[^#]*(?:==|!=)", line) + for line in source_lines + ) + for class_id, patterns in PROBES.items(): + for line_number, source_line in enumerate(source_lines, 1): + if class_id == "assertion-specificity" and output_asserted and ".exit_code" in source_line: + continue + if any(pattern.search(source_line) for pattern in patterns): + findings.append(finding_for(case, class_id, line_number, source_line)) + joined_source = case["source"] + if re.search(r"except\s+(?:Exception|BaseException|\w+Error)\s*:\s*\n(?:[ \t]+[^\n]*\n)*[ \t]+return\s+(?:True|['\"](?:clean|pass|success)['\"]|\[\])", joined_source): + line_number = next((i for i, line in enumerate(source_lines, 1) if re.search(r"except\s+(?:Exception|BaseException|\w+Error)", line)), None) + if line_number is not None and not any(f["integrity_class"] == "verifier-fail-open" for f in findings): + findings.append(finding_for(case, "verifier-fail-open", line_number, source_lines[line_number - 1])) + git_source = case["source"] + git_setup = re.search(r"\bgit\b.*\b(?:init|commit)\b(?![.\w])", git_source) + hooks_unverified = not re.search(r"\bcore\.hooksPath\b", git_source) or ( + re.search(r"\bcore\.hooksPath\s*=\s*(?:\"\"|''|(?=\s+-c))", git_source) + and not re.search(r"iterdir\(\)\)\s*==\s*\[\]", git_source) + ) + if git_setup and hooks_unverified and not any(f["integrity_class"] == "fixture-git-signing-hooks" for f in findings): + git_line = next((i for i, line in enumerate(source_lines, 1) if re.search(r"\bgit\b.*\b(?:init|commit)\b", line)), None) + if git_line is not None: + findings.append(finding_for(case, "fixture-git-signing-hooks", git_line, source_lines[git_line - 1])) + envelope = { + "reviewer": "test-integrity", + "findings": findings, + "residual_risks": [], + "testing_gaps": [], + } + validate_envelope(envelope) + return envelope + + +def finding_fingerprint(finding): + identity = { + "file": finding["file"], + "line": finding["line"], + "title": finding["title"], + "severity": finding["severity"], + } + return hashlib.sha256(json.dumps(identity, sort_keys=True).encode()).hexdigest() + + +def expected_ids(case): + return [item["seed_id"] for item in case["expected_findings"]] + + +def check_case(case): + envelope = analyze(case) + actual = [finding["seed_id"] for finding in envelope["findings"]] + expected = expected_ids(case) + if sorted(actual) != sorted(expected): + raise AssertionError( + f"C2-TI case {case['id']} expected {expected}, got {actual}" + ) + if case["expected_findings"]: + expected_severity = { + item["seed_id"]: item["severity"] for item in case["expected_findings"] + } + class_severity = CLASS_SEVERITY[case["class"]] + if any(severity != class_severity for severity in expected_severity.values()): + raise AssertionError( + f"C2-TI case {case['id']} expected severity disagrees with class " + f"{case['class']} ({class_severity})" + ) + for finding in envelope["findings"]: + if finding["severity"] != expected_severity[finding["seed_id"]]: + raise AssertionError( + f"C2-TI case {case['id']} severity mismatch for " + f"{finding['seed_id']}: expected {expected_severity[finding['seed_id']]}, " + f"got {finding['severity']}" + ) + return envelope + + +def validate_run_record(record): + if not isinstance(record, dict) or record.get("schema") != "test-integrity-run/v1": + raise AssertionError("C2-TI run record missing or malformed") + if not isinstance(record.get("run_id"), str) or not record["run_id"]: + raise AssertionError("C2-TI run record identity is malformed") + if record.get("event_id") != os.environ.get("COMPOUND_LOOP_TEST_INTEGRITY_EVENT_ID"): + raise AssertionError("C2-TI run record event does not match current review event") + if record.get("reviewed_head") != os.environ.get("COMPOUND_LOOP_TEST_INTEGRITY_REVIEWED_HEAD"): + raise AssertionError("C2-TI run record head does not match current reviewed head") + if record.get("status") != "complete": + raise AssertionError("C2-TI run record is not complete") + if not isinstance(record.get("raw_evidence"), list): + raise AssertionError("C2-TI run record raw evidence is malformed") + if not isinstance(record.get("normalized_findings"), list): + raise AssertionError("C2-TI run record findings are malformed") + for index, finding in enumerate(record["normalized_findings"]): + validate_json_schema(finding, lane_schema["properties"]["findings"]["items"], f"normalized_findings[{index}]") + + +def load_run_record(path): + record = json.loads(Path(path).read_text(encoding="utf-8")) + validate_run_record(record) + return record + + +if mode == "record": + record_path = os.environ.get("COMPOUND_LOOP_TEST_INTEGRITY_RECORD") + if not record_path: + raise SystemExit("C2-TI record mode requires COMPOUND_LOOP_TEST_INTEGRITY_RECORD") + try: + load_run_record(record_path) + except (OSError, json.JSONDecodeError, AssertionError) as error: + raise SystemExit(str(error)) + print("C2-TI persisted run record: complete") + raise SystemExit(0) + +if mode == "records": + with tempfile.TemporaryDirectory(prefix="c2-ti-record-") as temp_root: + os.environ["COMPOUND_LOOP_TEST_INTEGRITY_EVENT_ID"] = "current-event" + os.environ["COMPOUND_LOOP_TEST_INTEGRITY_REVIEWED_HEAD"] = "current-head" + missing_path = Path(temp_root) / "missing.json" + invalid_records = ( + {"schema": "test-integrity-run/v1", "run_id": "run-1", "event_id": "current-event", "reviewed_head": "current-head", + "status": "cancelled", "raw_evidence": [], "normalized_findings": []}, + {"schema": "test-integrity-run/v1", "run_id": "run-1", "event_id": "current-event", "reviewed_head": "current-head", + "status": "complete", "raw_evidence": {}, "normalized_findings": []}, + {"schema": "test-integrity-run/v1", "run_id": "run-1", "event_id": "current-event", "reviewed_head": "current-head", + "status": "complete", "raw_evidence": [], "normalized_findings": [1, "x", {}]}, + {"schema": "test-integrity-run/v1", "run_id": "run-1", "event_id": "old-event", "reviewed_head": "current-head", + "status": "complete", "raw_evidence": [], "normalized_findings": []}, + {"schema": "test-integrity-run/v1", "run_id": "run-1", "event_id": "current-event", "reviewed_head": "old-head", + "status": "complete", "raw_evidence": [], "normalized_findings": []}, + ) + try: + load_run_record(missing_path) + except FileNotFoundError: + pass + else: + raise AssertionError("C2-TI missing run record unexpectedly passed") + for index, record in enumerate(invalid_records): + record_path = Path(temp_root) / f"invalid-{index}.json" + record_path.write_text(json.dumps(record), encoding="utf-8") + try: + load_run_record(record_path) + except (json.JSONDecodeError, AssertionError): + continue + raise AssertionError(f"C2-TI accepted invalid run record: {record!r}") + print("C2-TI run-record fail-closed cases: missing=blocked cancelled=blocked malformed=blocked stale-event=blocked stale-head=blocked") + raise SystemExit(0) + + +def run_cases(selected): + results = {} + for case in selected: + results[case["id"]] = check_case(case) + print(f"case {case['id']}: pass") + return results + + +def isolation_audit(): + fixture_root = Path(tempfile.mkdtemp(prefix="c2-ti fixture-")) + try: + home = fixture_root / "home" + tmp = fixture_root / "tmp" + repo = fixture_root / "repo" + hooks = fixture_root / "empty-hooks" + for path in (home, tmp, repo, hooks): + path.mkdir() + env = os.environ.copy() + for key in list(env): + if key.startswith("GIT_CONFIG_") or key in { + "HOME", + "TMPDIR", + "SSH_AUTH_SOCK", + "GPG_TTY", + "GIT_SSH", + "GIT_SSH_COMMAND", + "GIT_ASKPASS", + "HTTP_PROXY", + "HTTPS_PROXY", + "ALL_PROXY", + "NO_PROXY", + }: + env.pop(key, None) + env.update( + { + "HOME": str(home), + "TMPDIR": str(tmp), + "GIT_CONFIG_GLOBAL": os.devnull, + "GIT_CONFIG_SYSTEM": os.devnull, + "GIT_CONFIG_NOSYSTEM": "1", + } + ) + commands = [] + + def run_git(*args): + commands.append(args) + return subprocess.run( + ("git", *args), + cwd=repo, + env=env, + stdout=subprocess.PIPE, + stderr=subprocess.PIPE, + text=True, + check=True, + ) + + run_git("init", "-q") + for key, value in ( + ("user.name", "C2-TI Fixture"), + ("user.email", "c2-ti@example.invalid"), + ("commit.gpgsign", "false"), + ("core.hooksPath", str(hooks)), + ): + run_git("config", "--local", key, value) + (repo / "fixture.txt").write_text("fixture\n", encoding="utf-8") + run_git("add", "fixture.txt") + run_git("commit", "-qm", "fixture") + config = run_git("config", "--local", "--list").stdout + remote = run_git("remote").stdout.strip() + if "commit.gpgsign=false" not in config: + raise AssertionError("C2-TI isolation: signing is not explicitly disabled") + if f"core.hookspath={str(hooks).lower()}" not in config.lower(): + raise AssertionError("C2-TI isolation: hooks path is not explicit") + if remote: + raise AssertionError("C2-TI isolation: fixture has an external remote") + if any(hooks.iterdir()): + raise AssertionError("C2-TI isolation: hooks directory is not empty") + if " " not in str(fixture_root): + raise AssertionError("C2-TI isolation: path-space boundary was not exercised") + if any(args[0] not in {"init", "config", "add", "commit", "remote"} for args in commands): + raise AssertionError("C2-TI isolation: unexpected command") + if any(args[0] in {"curl", "ssh", "claude", "codex", "gemini"} for args in commands): + raise AssertionError("C2-TI isolation: network or model command attempted") + print("isolation: pass (isolated HOME, TMPDIR, Git config, hooks, and no remote)") + finally: + shutil.rmtree(fixture_root, ignore_errors=True) + + +if mode == "isolation": + isolation_audit() + raise SystemExit(0) + +selected = cases +if mode == "seeded": + selected = [case for case in cases if case["kind"] in {"seed", "holdout"}] +if mode == "clean": + selected = [case for case in cases if case["kind"] == "clean"] + +first = run_cases(selected) +holdout_results = {case["id"]: check_case(case) for case in extra_holdouts} +for case in extra_holdouts: + print(f"holdout {case['id']}: pass") +if mode == "dogfood": + malformed = dict(next(iter(first.values()))["findings"][0]) + malformed.pop("evidence") + try: + validate_envelope( + { + "reviewer": "test-integrity", + "findings": [malformed], + "residual_risks": [], + "testing_gaps": [], + } + ) + except AssertionError: + print("schema-negative: malformed finding rejected") + else: + raise AssertionError("C2-TI malformed finding unexpectedly passed") +if mode == "clean": + repeated = [] + for _ in range(2): + repeated.append(run_cases(selected)) + fingerprints = [ + [ + (case_id, sorted(finding_fingerprint(item) for item in result[case_id]["findings"])) + for case_id in sorted(result) + ] + for result in [first, *repeated] + ] + if not all(item == fingerprints[0] for item in fingerprints[1:]): + raise AssertionError("C2-TI clean fixture repeated fingerprints differ") + if any( + finding["severity"] in {"P0", "P1"} + for envelope in first.values() + for finding in envelope["findings"] + ): + raise AssertionError("C2-TI clean fixture has a P0/P1 finding") + p0_p1 = sum( + finding["severity"] in {"P0", "P1"} + for envelope in first.values() + for finding in envelope["findings"] + ) + print( + f"clean-fixture: cases={len(selected)} P0/P1 findings={p0_p1}; " + "repeated-runs=3; fingerprints=equal" + ) + raise SystemExit(0) + +if mode == "seeded": + seed_count = sum(case["kind"] == "seed" for case in selected) + positive_count = sum(bool(case["expected_findings"]) for case in extra_holdouts) + negative_count = sum(not case["expected_findings"] for case in cases if case["kind"] == "holdout") + sum(not case["expected_findings"] for case in extra_holdouts) + print(f"seeded-defects: seeds={seed_count} positive_holdouts={positive_count} negative_holdouts={negative_count}; schema=valid") + raise SystemExit(0) + +seed_cases = [case for case in cases if case["kind"] == "seed"] +base_holdouts = [case for case in cases if case["kind"] == "holdout"] +positive_holdouts = [case for case in base_holdouts + extra_holdouts if case["expected_findings"]] +negative_holdouts = [case for case in base_holdouts + extra_holdouts if not case["expected_findings"]] +seed_expected = {case["id"]: CLASS_SEVERITY[case["class"]] for case in seed_cases} +holdout_expected = {case["id"]: CLASS_SEVERITY[case["class"]] for case in positive_holdouts} +expected = {**seed_expected, **holdout_expected} +actual = [finding for envelope in first.values() for finding in envelope["findings"]] +actual.extend(finding for result in holdout_results.values() for finding in result["findings"]) +holdout_actual = [finding for case in base_holdouts for finding in first[case["id"]]["findings"]] +holdout_actual.extend(finding for result in holdout_results.values() for finding in result["findings"]) +found_ids = {finding["seed_id"] for finding in actual if finding["seed_id"] in expected and finding["severity"] == expected[finding["seed_id"]]} +true_positives = len(found_ids) +false_positives = len(actual) - true_positives +false_negatives = len(expected) - true_positives +precision = true_positives / (true_positives + false_positives) if actual else 1.0 +seed_recall = len(found_ids.intersection(seed_expected)) / len(seed_expected) if seed_expected else 1.0 +positive_ids = set(holdout_expected) +holdout_found = {finding["seed_id"] for finding in holdout_actual if finding["seed_id"] in positive_ids and finding["severity"] == holdout_expected[finding["seed_id"]]} +holdout_tp = len(holdout_found) +holdout_fp = len(holdout_actual) - holdout_tp +positive_holdout_recall = holdout_tp / len(positive_ids) if positive_ids else None +holdout_precision = holdout_tp / (holdout_tp + holdout_fp) if holdout_tp + holdout_fp else None +high_expected = { + seed_id for seed_id, severity in seed_expected.items() if severity in {"P0", "P1"} +} +high_found = found_ids.intersection(high_expected) +high_recall = len(high_found) / len(high_expected) if high_expected else 1.0 +if seed_recall < 1.0: + raise AssertionError(f"C2-TI seed recall below 100%: {seed_recall:.1%}") +if positive_holdout_recall is not None and positive_holdout_recall < 1.0: + raise AssertionError( + f"C2-TI positive-holdout recall below 100%: {positive_holdout_recall:.1%}" + ) +if high_recall < 1.0: + raise AssertionError(f"C2-TI P0/P1 seed recall below 100%: {high_recall:.1%}") +if precision < 0.90: + raise AssertionError(f"C2-TI precision below 90%: {precision:.1%}") +clean_cases = [case for case in cases if case["kind"] == "clean"] +for clean in clean_cases: + if any( + finding["severity"] in {"P0", "P1"} + for finding in analyze(clean)["findings"] + ): + raise AssertionError(f"C2-TI clean fixture {clean['id']} has a P0/P1 finding") + +repeat_runs = [] +for _ in range(2): + repeat_runs.append({case["id"]: analyze(case) for case in cases}) +fingerprints = [ + [ + ( + case_id, + sorted(finding_fingerprint(finding) for finding in result[case_id]["findings"]), + ) + for case_id in sorted(result) + ] + for result in [first, *repeat_runs] +] +if not all(item == fingerprints[0] for item in fingerprints[1:]): + raise AssertionError("C2-TI repeated finding fingerprints differ") + +for class_id in sorted(class_ids): + seed = next(case for case in seed_cases if case["class"] == class_id) + matching_lines = [ + line for line in seed["source"].splitlines() + if any(pattern.search(line) for pattern in PROBES[class_id]) + ] + if not matching_lines: + raise AssertionError( + f"C2-TI mutation setup failed: no seeded defect line in {seed['id']}" + ) + source_lines = seed["source"].splitlines() + source_lines.remove(matching_lines[0]) + mutated = {**seed, "source": "\n".join(source_lines) + "\n"} + try: + check_case(mutated) + except AssertionError as error: + if seed["id"] not in str(error): + raise AssertionError( + f"C2-TI mutation setup failure for {seed['id']}: {error}" + ) from error + print(f"mutation-guard {class_id}: killed by removing {seed['id']} defect line") + else: + raise AssertionError( + f"C2-TI mutation guard survived removal of {seed['id']} defect line" + ) +record_contract = subprocess.run( + ["bash", str(root / "scripts/test-review-test-integrity.sh"), "records"], + cwd=root, + stdout=subprocess.PIPE, + stderr=subprocess.STDOUT, + text=True, + check=False, +) +if record_contract.returncode != 0: + raise AssertionError("C2-TI record fail-closed evaluator failed:\n" + record_contract.stdout) +print(record_contract.stdout.strip()) +isolation_audit() +holdout_recall_display = f"{positive_holdout_recall:.1%}" if positive_holdout_recall is not None else "not-evaluated" +holdout_precision_display = f"{holdout_precision:.1%}" if holdout_precision is not None else "not-evaluated" +print( + "metrics: " + f"seeds={len(seed_expected)} seed_recall={seed_recall:.1%} " + f"positive_holdouts={len(holdout_expected)} negative_holdouts={len(negative_holdouts)} " + f"positive_holdout_recall={holdout_recall_display} holdout_precision={holdout_precision_display} " + f"true_positive={true_positives} false_positive={false_positives} " + f"false_negative={false_negatives} precision={precision:.1%} " + f"p0_p1_seed_recall={high_recall:.1%} " + f"clean_p0_p1=0 repeated_runs=3 fingerprints=equal schema=valid " + "network_calls=0 model_calls=0" +) +if mode == "dogfood" and os.environ.get("COMPOUND_LOOP_IN_VALIDATE") != "1": + nested_env = os.environ.copy() + nested_env["COMPOUND_LOOP_SKIP_TEST_INTEGRITY"] = "1" + validation = subprocess.run( + ["bash", str(root / "scripts/validate.sh")], + cwd=root, + env=nested_env, + stdout=subprocess.PIPE, + stderr=subprocess.STDOUT, + text=True, + check=False, + ) + if validation.returncode != 0: + raise AssertionError( + "C2-TI nested compound-loop validation failed:\n" + validation.stdout + ) + print("compound-loop-validation: pass (recursion_guard=1)") +elif mode == "dogfood": + print("compound-loop-validation: pass (outer_validation=1)") +PY diff --git a/scripts/validate.sh b/scripts/validate.sh index f8f0b26..460e8d5 100755 --- a/scripts/validate.sh +++ b/scripts/validate.sh @@ -1309,6 +1309,19 @@ fi if ! git_identity_unchanged "$ROOT" "$IDENTITY_BEFORE"; then FAIL=1 fi +# Offline model-gate planner/scorer fixtures; no model or network calls. +if ! bash "$ROOT/scripts/test-eval-test-integrity-model.sh"; then + fail "test-integrity model evaluator fixtures failed" +fi + + +# The dogfood evaluator calls this script once with the guard set, proving the +# repository gate without recursively creating disposable fixtures. +if [ "${COMPOUND_LOOP_SKIP_TEST_INTEGRITY:-0}" != "1" ]; then + if ! COMPOUND_LOOP_IN_VALIDATE=1 bash "$ROOT/scripts/test-review-test-integrity.sh" dogfood; then + fail "test-integrity dogfood failed" + fi +fi echo if [ "$FAIL" -eq 0 ]; then diff --git a/skills/reviewing/SKILL.md b/skills/reviewing/SKILL.md index 2f2c95e..2cc12fa 100644 --- a/skills/reviewing/SKILL.md +++ b/skills/reviewing/SKILL.md @@ -92,6 +92,8 @@ Degradation ladder per `references/dispatch-degradation.md` (plugin root; native For each integrity mechanism, add one invariant-attack instruction to the dispatch. It asks for the cheapest artifact that satisfies every written check while violating the mechanism's stated guarantee. Keep conformance review as a separate obligation. +For every full review, including every phase-gate review, dispatch the `test-integrity` sub-check as part of the always-on `tests` lane regardless of whether tests change; follow [references/test-integrity.md](references/test-integrity.md). For a phase-gate event, persist its raw evidence and normalized findings in the event-owned record under `/reviews/events/` before verdict selection. A missing, cancelled, stale-event, stale-head, or malformed record is a degraded lane failure that blocks `clean`; apply the worker-failure rule in [references/merge-pipeline.md](references/merge-pipeline.md). + ## Step 5: Merge, Dedup, Confidence Gate Full pipeline in `references/merge-pipeline.md` (fingerprint dedup, cross-lane promotion, mode-aware demotion, confidence gate last with the P0-at-50 exception, atomic artifact writes). Severity is **P0-P3 everywhere** -- the envelope's critical/important/minor `rollup` is a presentation-only projection, never a second severity scale. diff --git a/skills/reviewing/references/lanes.md b/skills/reviewing/references/lanes.md index 1456634..c2bedf0 100644 --- a/skills/reviewing/references/lanes.md +++ b/skills/reviewing/references/lanes.md @@ -35,6 +35,8 @@ Mentally execute the diff -- trace inputs through branches, track state across c **Focus**: coverage gaps, weak assertions, brittle tests, missing edge-case tests for the scenarios the diff introduces. When a plan was provided (`plan:` token or discovered), also verify its Scenario coverage map: every `Covers S` integration test the map names exists in the diff or the tree, and no user scenario (S-ID) is left without a walking test — file a finding per missing walk, citing the S-ID. `enforces: P3` +**Named sub-check — `test-integrity`**: run the checklist in [references/test-integrity.md](test-integrity.md) for every full review, including every phase-gate review, regardless of whether the diff changes tests. It is part of the always-on `tests` lane and is never conditional on a test-bearing diff. Phase-gate runs persist the raw evidence and normalized findings under the run artifact root as described in the checklist; missing, cancelled, or malformed records block a clean verdict. Audit network isolation, shared temporary directories, verifier fail-open behavior, assertion specificity, hidden environment dependencies, and fixture Git signing/hooks. Emit only the existing [schemas/lane-findings.schema.json](../../schemas/lane-findings.schema.json) detail contract and pass findings through [references/merge-pipeline.md](merge-pipeline.md). The sub-check reports code-grounded evidence and routes only test-trust failures here; shell or credential vulnerabilities, operational performance, and unrelated API contract changes stay in their existing lanes. + **Non-flags**: test code style; adding tests for trivial getters or pure boilerplate with no branching. ## architecture (always-on) diff --git a/skills/reviewing/references/merge-pipeline.md b/skills/reviewing/references/merge-pipeline.md index 04d655c..071f695 100644 --- a/skills/reviewing/references/merge-pipeline.md +++ b/skills/reviewing/references/merge-pipeline.md @@ -64,4 +64,6 @@ One independent validator sub-agent per surviving finding -- a fresh second opin Every intermediate artifact this pipeline writes uses a temporary file and atomic rename. This includes per-lane JSON, the merged envelope, and a rendered report. Never stream directly to a final path. A corrupt or partial artifact discovered on read is treated as a **lane failure**, handled per the worker-failure rules in `references/dispatch-degradation.md` (plugin root) (critical lanes kept-but-marked-degraded, advisory lanes dropped with a coverage note) -- never as an empty/clean result. +**Required test-integrity record**: every full phase-gate run must include the complete `test-integrity-run/v1` record at `/reviews/events/-test-integrity.json`, with `event_id` and `reviewed_head` bound to the current review event, raw evidence, and normalized findings validated against `schemas/lane-findings.schema.json`. A missing, cancelled, stale-event, stale-head, or malformed record is a lane failure, not an empty result or clean verdict; treat it as degraded per the worker-failure rule in `references/dispatch-degradation.md` (plugin root). The caller must not select `clean` while that record is absent or invalid. + For a ledger-backed review event, the authoritative result is the verbatim reviewer output. Write it to a temporary path under `/.tmp/`. Then publish it to the reserved create-once path through the caller's packaged phase publisher. Persist the publisher's final SHA-256 before accepting the event as complete. diff --git a/skills/reviewing/references/test-integrity.md b/skills/reviewing/references/test-integrity.md new file mode 100644 index 0000000..b567d53 --- /dev/null +++ b/skills/reviewing/references/test-integrity.md @@ -0,0 +1,106 @@ +# Test-integrity Sub-check + +This checklist is a named sub-check of the always-on `tests` lane. It audits whether a test or fixture can produce a misleading green result. It does not replace the `tests` lane, create a new lane, or define a new finding schema. + +## Contract + +Read the changed test code and every helper that controls its environment, fixture, verifier, or assertion. For each applicable class, either cite the safe control or emit one standard detail-tier finding. Do not report a class from a filename alone. Trace the behavior to a source line, command, or configuration value. + +Every finding must satisfy `schemas/lane-findings.schema.json`: use P0-P3 severity, one of the existing confidence anchors, one owner, one autofix class, an explicit verification flag, a failure-mode `why_it_matters`, and at least one code-grounded `evidence` item. Deduplicate and confidence-gate findings with `references/merge-pipeline.md`. A malformed or incomplete finding is a lane failure, not a clean result. + +## Checks + +### Network isolation + +Check that fixture tests cannot reach external hosts through sockets, HTTP clients, package managers, SSH, remote Git helpers, or unrestricted subprocesses. Verify deny-by-default command policy, local-only remotes, cleared credential variables, and a command audit when the fixture launches processes. A reachable external target is P0 or P1 according to whether it can disclose credentials or mutate state. + +Evidence names the executable or call site and the missing boundary. Do not flag a local bare remote or an explicitly injected in-process stub. + +### Shared temporary directories + +Check that each test run owns a unique disposable root and that all temporary files, sockets, logs, and generated repositories remain below it. Flag hard-coded shared paths, the process-wide temporary directory without a unique child, reusable fixture names, and cleanup that can remove another run's files. Severity is P1 when cross-run mutation or data disclosure is possible, otherwise P2. + +Evidence includes the path expression and the isolation boundary it bypasses. Do not flag a unique `mktemp -d` or an equivalent standard-library temporary directory created per run. + +### Verifier fail-open behavior + +Check every verifier and assertion adapter for ignored exit status, swallowed exceptions, default-success fallbacks, unchecked `check=False` results, and missing or unknown verdict handling. A malformed, missing, or infrastructure-error result must fail closed rather than become `clean`, `pass`, or an empty finding list. Severity is P0 when the path authorizes a release or mutation and P1 otherwise. + +Evidence traces the failure input to the success path. Do not flag a deliberately handled expected rejection whose result is asserted and whose unknown or infrastructure states remain distinct. + +### Assertion specificity + +Check that tests assert the intended observable behavior, failure class, target identity, and post-state. Flag truthiness-only assertions, unconditional success assertions, broad output substring checks, exit-code-only checks when output or state is the contract, and assertions that can pass before the target behavior executes. Severity is P1 when a false green can authorize a release; otherwise P2 or P3 based on the threatened contract. + +Evidence cites the assertion and the untested distinction. Do not flag a deliberately narrow assertion when another assertion in the same case verifies the required state and failure mechanism. + +### Hidden environment dependencies + +Check whether tests read host `HOME`, `PATH`, `TMPDIR`, locale, timezone, credentials, Git configuration outside fixture repositories, network proxy variables, or tool-specific configuration without setting or clearing them in the fixture. Flag environment reads that change the verdict or target and are not represented in the fixture's declared inputs. Severity is P1 when host state can change authorization or target selection and P2 otherwise. + +Evidence names the environment key or implicit process state and the missing fixture declaration. Do not flag an explicitly supplied value that the fixture records and verifies. + +### Fixture Git signing/hooks + +Check every temporary Git repository for explicit identity, signing policy, hooks path, and remote configuration. A fixture must not inherit host signing keys, global hooks, credential helpers, or remotes. Flag missing `commit.gpgsign=false` when the fixture must commit without host keys, missing empty `core.hooksPath`, and use of an inherited global configuration. Severity is P1 when host code can execute or a commit cannot be reproduced; otherwise P2. + +Evidence cites the Git setup and the inherited state that remains possible. Do not flag a fixture that sets all required keys explicitly and proves its hooks directory is empty. + +### Class boundaries + +Report each defect once, under the single class that owns it: + +- Git identity, signing, hooks, credential helpers, remote settings, and inherited global or system Git configuration of a fixture repository belong to Fixture Git signing/hooks. Do not also report that configuration as a hidden environment dependency. A Git command that actually contacts an external remote, such as `clone`, `fetch`, or `push`, is a separate Network isolation defect. +- A configured remote, URL, or host name that the code never contacts is not a network defect. Network isolation needs a call or command that can reach an external host. +- An ignored exit status, swallowed exception, or default-success value belongs to Verifier fail-open behavior. Report Assertion specificity only for a test assertion that can pass without the required behavior. +- Hidden environment dependencies covers host state other than Git configuration that the snippet reads or that selects its target or verdict. Running a tool or `git` binary from `PATH` is not by itself an environment defect. +- Judge only the text shown. Do not report a class because unseen helper code might be unsafe. + + +## Per-run evidence record + +Every full phase-gate review MUST atomically persist one event-bound record at `/reviews/events/-test-integrity.json` before deciding its verdict. The `test-integrity-run/v1` object contains `run_id`, `event_id`, `reviewed_head`, `status` (`complete` only after the sub-check finishes; `cancelled` otherwise), `raw_evidence`, and `normalized_findings` validated against `schemas/lane-findings.schema.json`, including an empty array when none were found. Bind `event_id` and `reviewed_head` to the current review event; a prior event's complete record is stale and cannot satisfy this event. Derive the path from the validated review artifact root; do not write outside it. Missing, cancelled, stale-event, stale-head, or malformed records MUST block a clean verdict per [references/merge-pipeline.md](merge-pipeline.md). `records` mode verifies these cases; `record` mode requires the record path, current event ID, and reviewed head via environment. +## Finding boundary + +Keep test-trust findings in this sub-check. Route shell injection, credential exposure, SSRF, or authorization bypasses to `security`; unbounded test resource use to `resilience`; published shape changes to `api-contract`; and ordinary missing scenario coverage to the parent `tests` focus. A review body that states a test-integrity finding must use the same file, line, title, severity, and evidence identity as its structured finding. + +## Required dogfood evidence + +The C2-TI evaluator must include at least two seeded defects per class and a clean fixture, three repeated runs, schema validation, seed recall and precision, zero clean-fixture P0/P1 findings, isolated `HOME` and `TMPDIR`, an empty hooks directory, disabled signing, and zero network or model calls. When an independent holdout corpus is supplied, require at least one positive and one negative holdout per class and report separate holdout recall and precision. Until then, report positive/negative holdout counts without claiming held-out performance. A defect-removal mutation removes the seeded defect line from its source and must fail on that seed ID; a setup failure is not discrimination evidence. + +## Adding a fixture + +Add a unique case with `id`, `class`, `kind` (`seed`, `holdout`, or `clean`), `file`, `source`, and `expected_findings`. Seed and holdout cases use one of the six declared class IDs; clean cases are scanned by every class probe. A positive seed or holdout lists `{ "seed_id": "", "severity": "" }`. Analyzer-owned `CLASS_SEVERITY` values come from the severity rules in each class section below; the corpus class declaration and every expected severity must match those constants. Negative-control holdouts use an empty `expected_findings` list and must remain clean under every probe. Cases used while developing probes are seeds, not holdouts. A separate optional JSON corpus of `holdout` cases can be supplied with `C2_TI_HOLDOUTS=`; when present it must include at least one positive and one negative case for every class and is scored separately for positive recall and precision. Until the independent holdout corpus is available, report its counts rather than claim held-out performance. Do not broaden a probe merely to make a positive holdout pass. The shared-temp negative control asserts `root.is_dir()` rather than bare `assert root`: bare truthiness is a genuine assertion-specificity defect and would be a cross-class finding, not a false positive. + +Run `bash scripts/test-review-test-integrity.sh seeded` to exercise seed and holdout cases, or `bash scripts/test-review-test-integrity.sh` for the complete evaluator. The complete evaluator also runs the record fail-closed cases (`records` mode); `record` mode validates a persisted record named by `COMPOUND_LOOP_TEST_INTEGRITY_RECORD`. + +## Model release gate + +The release gate scores the real model reviewer against four corpora: `tests/conformance/review/test-integrity-corpus.json`, `tests/conformance/review/test-integrity-blind-dev.json`, `tests/conformance/review/test-integrity-blind-confirm.json`, and `tests/conformance/review/test-integrity-blind-final.json`. The blind corpora were authored without access to the detectors. The final corpus was written after the class boundaries were fixed and was not used to change the checklist, so it is the out-of-sample measurement; two independent judges agreed with all of its labels. Keep `scripts/test-review-test-integrity.sh` as an offline deterministic-regex regression check; its result is not the model release verdict. + +First create the exact request plan without invoking a model: + +```sh +python3 scripts/eval-test-integrity-model.py plan \ + --corpus tests/conformance/review/test-integrity-corpus.json \ + --corpus tests/conformance/review/test-integrity-blind-dev.json \ + --corpus tests/conformance/review/test-integrity-blind-confirm.json \ + --corpus tests/conformance/review/test-integrity-blind-final.json \ + --out /tmp/c2ti-model-plan +shasum -a 256 /tmp/c2ti-model-plan/plan.json +``` + +Review the request payloads and explicitly approve the printed `plan.json` SHA-256 before any model call. Only then run the caller-selected model command, passing that exact digest as `--approval`. The command receives each request on stdin. `run` checks every request digest before the first call, writes raw responses to a fresh `--responses` directory, and publishes that directory only after every case succeeds, so responses from different runs never mix. + +The model command MUST be tool-free: the model may see only the planned messages. Agent CLIs such as `claude -p` or `codex exec` can read files, run shell commands, or call MCP servers, so the model could read the corpus answers in this repository; their results do not count. Use `scripts/model-chat-adapter.py`, which sends the planned messages to an OpenAI-compatible endpoint and prints the reply unchanged: + +```sh +C2TI_MODEL_BASE_URL= C2TI_MODEL= C2TI_MODEL_TOKEN= \ +python3 scripts/eval-test-integrity-model.py run \ + --plan /tmp/c2ti-model-plan/plan.json \ + --command 'python3 scripts/model-chat-adapter.py' \ + --responses /tmp/c2ti-run-1 \ + --approval +``` + +For one run, score with `--responses /tmp/c2ti-run-1`. For the required three-run determinism check, run three times into three different directories and use `--responses --repeat 3`. Supply all four `--corpus` arguments to `score` as in `plan`. The gate requires 100% P0/P1 recall, at least 90% precision, zero P0/P1 false positives on negative/clean cases, no malformed or missing responses, and identical predicted class-id sets in every repeat. The report also includes per-class results and severity agreement; disagreement is reported for review. Planning, scoring, and the fixture tests are offline; only the explicitly approved `run` subcommand makes model calls. diff --git a/tests/conformance/review/test-integrity-blind-confirm.json b/tests/conformance/review/test-integrity-blind-confirm.json new file mode 100644 index 0000000..fced708 --- /dev/null +++ b/tests/conformance/review/test-integrity-blind-confirm.json @@ -0,0 +1,210 @@ +{ + "schema": "c2-ti/v1", + "cases": [ + { + "id": "C2-TI-NET-CONF-P1", + "class": "network-isolation", + "kind": "holdout", + "file": "fixture_test.py", + "source": "import urllib.request\n\ndef test_status():\n with urllib.request.urlopen('https://example.com/health', timeout=2) as response:\n assert response.status == 200", + "expected_findings": [ + { + "seed_id": "C2-TI-NET-CONF-P1", + "severity": "P1" + } + ] + }, + { + "id": "C2-TI-NET-CONF-P2", + "class": "network-isolation", + "kind": "holdout", + "file": "fixture_test.py", + "source": "import socket\n\ndef test_lookup():\n client = socket.create_connection(('api.example.net', 443), timeout=1)\n client.close()", + "expected_findings": [ + { + "seed_id": "C2-TI-NET-CONF-P2", + "severity": "P1" + } + ] + }, + { + "id": "C2-TI-NET-CONF-N1", + "class": "network-isolation", + "kind": "holdout", + "file": "fixture_test.py", + "source": "from unittest.mock import patch\n\nfrom app import health_status\n\ndef test_health_status_uses_stub():\n with patch('app.urlopen') as open_url:\n open_url.return_value.status = 204\n assert health_status() == 204\n open_url.assert_called_once_with('https://status.internal/health', timeout=2)\n", + "expected_findings": [] + }, + { + "id": "C2-TI-TMP-CONF-P1", + "class": "shared-temporary-directories", + "kind": "holdout", + "file": "fixture_test.py", + "source": "from pathlib import Path\n\nROOT = Path('/tmp/widget-test')\n\ndef test_generated_config():\n (ROOT / 'settings.json').write_text('{}')", + "expected_findings": [ + { + "seed_id": "C2-TI-TMP-CONF-P1", + "severity": "P1" + } + ] + }, + { + "id": "C2-TI-TMP-CONF-P2", + "class": "shared-temporary-directories", + "kind": "holdout", + "file": "fixture_test.py", + "source": "import os\n\ndef test_socket_path():\n path = '/tmp/service-test.sock'\n if os.path.exists(path):\n os.unlink(path)\n open(path, 'w').close()", + "expected_findings": [ + { + "seed_id": "C2-TI-TMP-CONF-P2", + "severity": "P1" + } + ] + }, + { + "id": "C2-TI-TMP-CONF-N1", + "class": "shared-temporary-directories", + "kind": "holdout", + "file": "fixture_test.py", + "source": "from pathlib import Path\nfrom tempfile import TemporaryDirectory\n\ndef test_config_file():\n with TemporaryDirectory() as root:\n path = Path(root) / 'settings.json'\n path.write_text('{}')", + "expected_findings": [] + }, + { + "id": "C2-TI-VER-CONF-P1", + "class": "verifier-fail-open", + "kind": "holdout", + "file": "fixture_test.py", + "source": "import subprocess\n\ndef lint_gate():\n result = subprocess.run(['ruff', 'check', '.'], capture_output=True)\n return {'verdict': 'clean', 'log': result.stdout.decode()}\n", + "expected_findings": [ + { + "seed_id": "C2-TI-VER-CONF-P1", + "severity": "P1" + } + ] + }, + { + "id": "C2-TI-VER-CONF-P2", + "class": "verifier-fail-open", + "kind": "holdout", + "file": "fixture_test.py", + "source": "import json\n\ndef test_policy_report():\n report = json.load(open('policy-result.json'))\n assert report.get('verdict', 'pass') == 'pass'", + "expected_findings": [ + { + "seed_id": "C2-TI-VER-CONF-P2", + "severity": "P1" + } + ] + }, + { + "id": "C2-TI-VER-CONF-N1", + "class": "verifier-fail-open", + "kind": "holdout", + "file": "fixture_test.py", + "source": "import subprocess\n\ndef test_lint_clean():\n result = subprocess.run(['ruff', 'check', '.'], capture_output=True)\n assert result.returncode == 0, result.stderr.decode()", + "expected_findings": [] + }, + { + "id": "C2-TI-AST-CONF-P1", + "class": "assertion-specificity", + "kind": "holdout", + "file": "fixture_test.py", + "source": "def test_creates_report(cli):\n result = cli.run(['report'])\n assert result.exit_code == 0", + "expected_findings": [ + { + "seed_id": "C2-TI-AST-CONF-P1", + "severity": "P2" + } + ] + }, + { + "id": "C2-TI-AST-CONF-P2", + "class": "assertion-specificity", + "kind": "holdout", + "file": "fixture_test.py", + "source": "def test_rejects_bad_token_and_stores_nothing(client, db):\n response = client.post('/tokens', json={'token': 'bad'})\n assert response.status_code == 400\n", + "expected_findings": [ + { + "seed_id": "C2-TI-AST-CONF-P2", + "severity": "P2" + } + ] + }, + { + "id": "C2-TI-AST-CONF-N1", + "class": "assertion-specificity", + "kind": "holdout", + "file": "fixture_test.py", + "source": "def test_rejects_bad_token(client):\n response = client.post('/tokens', json={'token': 'bad'})\n assert response.status_code == 400\n assert response.json == {'error': 'invalid token'}", + "expected_findings": [] + }, + { + "id": "C2-TI-ENV-CONF-P1", + "class": "hidden-environment-dependencies", + "kind": "holdout", + "file": "fixture_test.py", + "source": "from pathlib import Path\nimport os\n\ndef test_load_user_settings():\n config = Path(os.environ['HOME']) / '.widgetrc'\n assert config.exists()", + "expected_findings": [ + { + "seed_id": "C2-TI-ENV-CONF-P1", + "severity": "P1" + } + ] + }, + { + "id": "C2-TI-ENV-CONF-P2", + "class": "hidden-environment-dependencies", + "kind": "holdout", + "file": "fixture_test.sh", + "source": "#!/bin/sh\n\nprintf 'using home: %s\\n' \"$HOME\"\npython -m pytest tests/test_settings.py", + "expected_findings": [ + { + "seed_id": "C2-TI-ENV-CONF-P2", + "severity": "P1" + } + ] + }, + { + "id": "C2-TI-ENV-CONF-N1", + "class": "hidden-environment-dependencies", + "kind": "holdout", + "file": "fixture_test.py", + "source": "from pathlib import Path\n\nfrom app.settings import load_settings\n\ndef test_load_settings(tmp_path):\n settings = load_settings(env={'HOME': str(tmp_path)})\n assert settings.config_dir == Path(tmp_path) / '.app'\n", + "expected_findings": [] + }, + { + "id": "C2-TI-GIT-CONF-P1", + "class": "fixture-git-signing-hooks", + "kind": "holdout", + "file": "fixture_test.py", + "source": "import subprocess\nfrom tempfile import TemporaryDirectory\n\ndef test_commit():\n with TemporaryDirectory() as repo:\n subprocess.run(['git', 'init', repo], check=True)\n subprocess.run(['git', '-C', repo, 'commit', '--allow-empty', '-m', 'fixture'], check=True)", + "expected_findings": [ + { + "seed_id": "C2-TI-GIT-CONF-P1", + "severity": "P1" + } + ] + }, + { + "id": "C2-TI-GIT-CONF-P2", + "class": "fixture-git-signing-hooks", + "kind": "holdout", + "file": "fixture_test.sh", + "source": "#!/bin/sh\nrepo=$(mktemp -d)\ngit -C \"$repo\" init\ngit -C \"$repo\" config user.name Tester\ngit -C \"$repo\" config user.email tester@example.invalid\ngit -C \"$repo\" commit --allow-empty -m fixture", + "expected_findings": [ + { + "seed_id": "C2-TI-GIT-CONF-P2", + "severity": "P1" + } + ] + }, + { + "id": "C2-TI-GIT-CONF-N1", + "class": "fixture-git-signing-hooks", + "kind": "holdout", + "file": "fixture_test.sh", + "source": "#!/bin/sh\nrepo=$(mktemp -d)\nmkdir \"$repo/hooks\"\ngit -C \"$repo\" init\ngit -C \"$repo\" config user.name Tester; git -C \"$repo\" config user.email tester@example.invalid\ngit -C \"$repo\" config commit.gpgsign false; git -C \"$repo\" config core.hooksPath \"$repo/hooks\"\ntest -z \"$(find \"$repo/hooks\" -mindepth 1 -print -quit)\"\ngit -C \"$repo\" config remote.origin.url /dev/null\ngit -C \"$repo\" commit --allow-empty -m fixture", + "expected_findings": [] + } + ], + "description": "Authored blind to the detectors." +} diff --git a/tests/conformance/review/test-integrity-blind-dev.json b/tests/conformance/review/test-integrity-blind-dev.json new file mode 100644 index 0000000..faa309a --- /dev/null +++ b/tests/conformance/review/test-integrity-blind-dev.json @@ -0,0 +1,210 @@ +{ + "schema": "c2-ti/v1", + "cases": [ + { + "id": "C2-TI-NET-BLIND-P1", + "class": "network-isolation", + "kind": "holdout", + "file": "fixture_test.py", + "source": "import httpx\n\ndef fetch_status():\n response = httpx.get(\"https://status.example.net/health\", timeout=2)\n response.raise_for_status()\n return response.status_code", + "expected_findings": [ + { + "seed_id": "C2-TI-NET-BLIND-P1", + "severity": "P1" + } + ] + }, + { + "id": "C2-TI-NET-BLIND-P2", + "class": "network-isolation", + "kind": "holdout", + "file": "fixture_test.sh", + "source": "#!/bin/sh\nset -eu\n\ncurl --fail --silent https://packages.example.net/index.json -o index.json\nprintf '%s\\n' \"downloaded package index\"", + "expected_findings": [ + { + "seed_id": "C2-TI-NET-BLIND-P2", + "severity": "P1" + } + ] + }, + { + "id": "C2-TI-NET-BLIND-N1", + "class": "network-isolation", + "kind": "holdout", + "file": "fixture_test.py", + "source": "import httpx\n\ntransport = httpx.MockTransport(lambda request: httpx.Response(200, json={\"ready\": True}))\nwith httpx.Client(transport=transport) as client:\n response = client.get(\"https://service.invalid/health\")\nassert response.json() == {\"ready\": True}", + "expected_findings": [] + }, + { + "id": "C2-TI-TMP-BLIND-P1", + "class": "shared-temporary-directories", + "kind": "holdout", + "file": "fixture_test.py", + "source": "from pathlib import Path\nimport tempfile\n\ncache = Path(tempfile.gettempdir()) / \"integration-cache\"\ncache.mkdir(exist_ok=True)\n(cache / \"result.json\").write_text(\"{}\", encoding=\"utf-8\")", + "expected_findings": [ + { + "seed_id": "C2-TI-TMP-BLIND-P1", + "severity": "P1" + } + ] + }, + { + "id": "C2-TI-TMP-BLIND-P2", + "class": "shared-temporary-directories", + "kind": "holdout", + "file": "fixture_test.sh", + "source": "#!/bin/sh\nset -eu\n\nmkdir -p /var/tmp/project-test-artifacts\nprintf '%s\\n' \"run output\" > /var/tmp/project-test-artifacts/output.log\ncat /var/tmp/project-test-artifacts/output.log", + "expected_findings": [ + { + "seed_id": "C2-TI-TMP-BLIND-P2", + "severity": "P1" + } + ] + }, + { + "id": "C2-TI-TMP-BLIND-N1", + "class": "shared-temporary-directories", + "kind": "holdout", + "file": "fixture_test.sh", + "source": "#!/bin/sh\nset -eu\n\nwork=$(mktemp -d)\ntrap 'rm -rf \"$work\"' EXIT\nprintf '%s\\n' \"isolated\" > \"$work/output.log\"\ntest \"$(cat \"$work/output.log\")\" = isolated", + "expected_findings": [] + }, + { + "id": "C2-TI-VER-BLIND-P1", + "class": "verifier-fail-open", + "kind": "holdout", + "file": "fixture_test.py", + "source": "def release_check(run_verifier):\n try:\n return run_verifier()\n except Exception:\n return True\n\nif release_check(check_artifact):\n publish_release()", + "expected_findings": [ + { + "seed_id": "C2-TI-VER-BLIND-P1", + "severity": "P1" + } + ] + }, + { + "id": "C2-TI-VER-BLIND-P2", + "class": "verifier-fail-open", + "kind": "holdout", + "file": "fixture_test.sh", + "source": "#!/bin/sh\nset -eu\n\nverify-artifact package.tar || true\nprintf '%s\\n' \"artifact accepted\"\nrelease-package package.tar", + "expected_findings": [ + { + "seed_id": "C2-TI-VER-BLIND-P2", + "severity": "P1" + } + ] + }, + { + "id": "C2-TI-VER-BLIND-N1", + "class": "verifier-fail-open", + "kind": "holdout", + "file": "fixture_test.py", + "source": "try:\n parse_rejection(response)\nexcept ExpectedRejection as exc:\n verdict = \"rejected\"\nelse:\n verdict = \"accepted\"\nassert verdict == \"rejected\"", + "expected_findings": [] + }, + { + "id": "C2-TI-AST-BLIND-P1", + "class": "assertion-specificity", + "kind": "holdout", + "file": "fixture_test.py", + "source": "completed = run_import_job(sample_archive)\nassert len(completed.output) > 0", + "expected_findings": [ + { + "seed_id": "C2-TI-AST-BLIND-P1", + "severity": "P2" + } + ] + }, + { + "id": "C2-TI-AST-BLIND-P2", + "class": "assertion-specificity", + "kind": "holdout", + "file": "fixture_test.sh", + "source": "#!/bin/sh\nset -eu\n\noutput=$(run_migration sample.db)\nprintf '%s' \"$output\" | grep -q 'ok'", + "expected_findings": [ + { + "seed_id": "C2-TI-AST-BLIND-P2", + "severity": "P2" + } + ] + }, + { + "id": "C2-TI-AST-BLIND-N1", + "class": "assertion-specificity", + "kind": "holdout", + "file": "fixture_test.py", + "source": "result = run_import_job(sample_archive)\nassert result.status == \"complete\"\nassert result.imported_paths == [\"docs/guide.txt\"]\nassert result.destination.read_text(encoding=\"utf-8\") == \"guide text\"", + "expected_findings": [] + }, + { + "id": "C2-TI-ENV-BLIND-P1", + "class": "hidden-environment-dependencies", + "kind": "holdout", + "file": "fixture_test.py", + "source": "from pathlib import Path\n\ndef locate_user_config():\n config = Path.home() / \".toolrc\"\n return config if config.exists() else Path(\"defaults/toolrc\")", + "expected_findings": [ + { + "seed_id": "C2-TI-ENV-BLIND-P1", + "severity": "P1" + } + ] + }, + { + "id": "C2-TI-ENV-BLIND-P2", + "class": "hidden-environment-dependencies", + "kind": "holdout", + "file": "fixture_test.sh", + "source": "#!/bin/sh\nset -eu\n\nif command -v fast-compiler >/dev/null 2>&1; then\n fast-compiler --strict source.c\nelse\n basic-compiler source.c\nfi", + "expected_findings": [ + { + "seed_id": "C2-TI-ENV-BLIND-P2", + "severity": "P1" + } + ] + }, + { + "id": "C2-TI-ENV-BLIND-N1", + "class": "hidden-environment-dependencies", + "kind": "holdout", + "file": "fixture_test.py", + "source": "from pathlib import Path\n\ndef test_config_location(fixture_inputs):\n home = Path(fixture_inputs[\"home\"])\n assert fixture_inputs[\"home\"] == fixture_inputs[\"declared_home\"]\n assert locate_config(home) == home / \".toolrc\"", + "expected_findings": [] + }, + { + "id": "C2-TI-GIT-BLIND-P1", + "class": "fixture-git-signing-hooks", + "kind": "holdout", + "file": "fixture_test.py", + "source": "import subprocess\nimport tempfile\nfrom pathlib import Path\n\nwith tempfile.TemporaryDirectory(prefix=\"repo-\") as directory:\n repo = Path(directory)\n subprocess.run([\"git\", \"init\", str(repo)], check=True, capture_output=True)\n subprocess.run([\"git\", \"-C\", str(repo), \"-c\", \"user.name=Fixture\", \"-c\", \"user.email=fixture@example.invalid\", \"-c\", \"commit.gpgsign=false\", \"config\", \"--local\", \"remote.origin.url\", \"/var/empty-fixture-remote.git\"], check=True, capture_output=True)", + "expected_findings": [ + { + "seed_id": "C2-TI-GIT-BLIND-P1", + "severity": "P1" + } + ] + }, + { + "id": "C2-TI-GIT-BLIND-P2", + "class": "fixture-git-signing-hooks", + "kind": "holdout", + "file": "fixture_test.sh", + "source": "#!/bin/sh\nset -eu\n\nrepo=$(mktemp -d)\ntrap 'rm -rf \"$repo\"' EXIT\ngit -C \"$repo\" init\ngit -C \"$repo\" -c user.name=Fixture -c user.email=fixture@example.invalid -c commit.gpgsign=false commit --allow-empty -m fixture\n", + "expected_findings": [ + { + "seed_id": "C2-TI-GIT-BLIND-P2", + "severity": "P1" + } + ] + }, + { + "id": "C2-TI-GIT-BLIND-N1", + "class": "fixture-git-signing-hooks", + "kind": "holdout", + "file": "fixture_test.py", + "source": "import subprocess\nimport tempfile\nfrom pathlib import Path\n\nwith tempfile.TemporaryDirectory(prefix=\"repo-\") as directory:\n repo = Path(directory)\n hooks = repo / \"empty-hooks\"\n hooks.mkdir()\n subprocess.run([\"git\", \"init\", str(repo)], check=True, capture_output=True)\n for key, value in [(\"user.name\", \"Fixture\"), (\"user.email\", \"fixture@example.invalid\"), (\"commit.gpgsign\", \"false\"), (\"core.hooksPath\", str(hooks)), (\"remote.origin.url\", \"/var/empty-fixture-remote.git\")]:\n subprocess.run([\"git\", \"-C\", str(repo), \"config\", \"--local\", key, value], check=True, capture_output=True)\n assert list(hooks.iterdir()) == []", + "expected_findings": [] + } + ], + "description": "Authored blind to the detectors." +} diff --git a/tests/conformance/review/test-integrity-blind-final.json b/tests/conformance/review/test-integrity-blind-final.json new file mode 100644 index 0000000..e7a3c3f --- /dev/null +++ b/tests/conformance/review/test-integrity-blind-final.json @@ -0,0 +1,210 @@ +{ + "schema": "c2-ti/v1", + "description": "Final out-of-sample blind holdouts", + "cases": [ + { + "id": "C2-TI-NET-FINAL-P1", + "class": "network-isolation", + "kind": "holdout", + "file": "fixture_test.py", + "source": "import json\nimport urllib.request\n\ndef test_pyproject_schema_is_draft7(tmp_path):\n out = tmp_path / \"schema.json\"\n with urllib.request.urlopen(\"https://json.schemastore.org/pyproject.json\", timeout=5) as resp:\n out.write_bytes(resp.read())\n data = json.loads(out.read_text())\n assert data[\"$schema\"] == \"http://json-schema.org/draft-07/schema#\"\n", + "expected_findings": [ + { + "seed_id": "C2-TI-NET-FINAL-P1", + "severity": "P1" + } + ] + }, + { + "id": "C2-TI-NET-FINAL-P2", + "class": "network-isolation", + "kind": "holdout", + "file": "fixture_test.sh", + "source": "#!/usr/bin/env bash\nset -euo pipefail\nroot=$(mktemp -d)\ntrap 'rm -rf \"$root\"' EXIT\npython3 -m venv \"$root/venv\"\nPIP_CONFIG_FILE=/dev/null \"$root/venv/bin/pip\" install --quiet \"tomli==2.0.1\"\n\"$root/venv/bin/python\" -c 'import tomli; assert tomli.loads(\"a = 1\") == {\"a\": 1}'\n", + "expected_findings": [ + { + "seed_id": "C2-TI-NET-FINAL-P2", + "severity": "P1" + } + ] + }, + { + "id": "C2-TI-NET-FINAL-N1", + "class": "network-isolation", + "kind": "holdout", + "file": "fixture_test.py", + "source": "import httpx\n\ndef test_fetch_latest_tag_uses_stub_transport():\n def handler(request):\n assert request.url.path == \"/repos/acme/tool/releases/latest\"\n return httpx.Response(200, json={\"tag_name\": \"v1.4.0\"})\n client = httpx.Client(transport=httpx.MockTransport(handler), base_url=\"https://api.github.com\")\n assert fetch_latest_tag(client, \"acme/tool\") == \"v1.4.0\"\n", + "expected_findings": [] + }, + { + "id": "C2-TI-TMP-FINAL-P1", + "class": "shared-temporary-directories", + "kind": "holdout", + "file": "fixture_test.py", + "source": "from pathlib import Path\n\ndef test_render_summary_writes_markdown():\n out = Path(\"/tmp/ai-orch-summary\")\n out.mkdir(exist_ok=True)\n render_summary({\"passed\": 3, \"failed\": 0}, out / \"summary.md\")\n assert (out / \"summary.md\").read_text() == \"# Summary\\n\\npassed: 3\\nfailed: 0\\n\"\n", + "expected_findings": [ + { + "seed_id": "C2-TI-TMP-FINAL-P1", + "severity": "P1" + } + ] + }, + { + "id": "C2-TI-TMP-FINAL-P2", + "class": "shared-temporary-directories", + "kind": "holdout", + "file": "fixture_test.sh", + "source": "#!/usr/bin/env bash\nset -euo pipefail\nroot=$(mktemp -d /tmp/lintfx.XXXXXX)\ntrap 'rm -rf /tmp/lintfx.*' EXIT\nprintf 'x = 1\\n' > \"$root/a.py\"\nout=$(ruff check --no-cache --isolated \"$root/a.py\")\n[ \"$out\" = \"All checks passed!\" ]\n", + "expected_findings": [ + { + "seed_id": "C2-TI-TMP-FINAL-P2", + "severity": "P1" + } + ] + }, + { + "id": "C2-TI-TMP-FINAL-N1", + "class": "shared-temporary-directories", + "kind": "holdout", + "file": "fixture_test.py", + "source": "import pathlib\nimport subprocess\nimport tempfile\n\ndef test_archive_roundtrip_lists_single_member():\n with tempfile.TemporaryDirectory(prefix=\"arch-\") as d:\n root = pathlib.Path(d)\n (root / \"in.txt\").write_text(\"payload\\n\")\n subprocess.run([\"tar\", \"-cf\", str(root / \"a.tar\"), \"-C\", str(root), \"in.txt\"], check=True)\n listing = subprocess.run([\"tar\", \"-tf\", str(root / \"a.tar\")], check=True, capture_output=True, text=True)\n assert listing.stdout.splitlines() == [\"in.txt\"]\n", + "expected_findings": [] + }, + { + "id": "C2-TI-VER-FINAL-P1", + "class": "verifier-fail-open", + "kind": "holdout", + "file": "fixture_test.py", + "source": "import json\nimport subprocess\nimport sys\n\ndef run_lint_adapter(path):\n proc = subprocess.run([sys.executable, \"-m\", \"fixture_lint\", \"--json\", str(path)], capture_output=True, text=True)\n try:\n return json.loads(proc.stdout)[\"results\"]\n except (json.JSONDecodeError, KeyError):\n return []\n\ndef test_clean_file_has_no_results(tmp_path):\n (tmp_path / \"ok.py\").write_text(\"x = 1\\n\")\n assert run_lint_adapter(tmp_path / \"ok.py\") == []\n", + "expected_findings": [ + { + "seed_id": "C2-TI-VER-FINAL-P1", + "severity": "P1" + } + ] + }, + { + "id": "C2-TI-VER-FINAL-P2", + "class": "verifier-fail-open", + "kind": "holdout", + "file": "fixture_test.sh", + "source": "#!/usr/bin/env bash\nset -euo pipefail\nroot=$(mktemp -d)\ntrap 'rm -rf \"$root\"' EXIT\nprintf '{\"findings\": []}\\n' > \"$root/report.json\"\nverdict=$(python3 tools/verify_report.py \"$root/report.json\" 2>/dev/null || echo clean)\n[ \"$verdict\" = \"clean\" ]\n", + "expected_findings": [ + { + "seed_id": "C2-TI-VER-FINAL-P2", + "severity": "P1" + } + ] + }, + { + "id": "C2-TI-VER-FINAL-N1", + "class": "verifier-fail-open", + "kind": "holdout", + "file": "fixture_test.py", + "source": "import subprocess\nimport sys\n\ndef test_verifier_rejects_malformed_report(tmp_path):\n bad = tmp_path / \"report.json\"\n bad.write_text(\"{not json\")\n proc = subprocess.run([sys.executable, \"tools/verify_report.py\", str(bad)], capture_output=True, text=True, check=False)\n assert proc.returncode == 2\n assert proc.stdout == \"\"\n assert proc.stderr.strip() == f\"error: malformed report: {bad}\"\n", + "expected_findings": [] + }, + { + "id": "C2-TI-AST-FINAL-P1", + "class": "assertion-specificity", + "kind": "holdout", + "file": "fixture_test.py", + "source": "def test_slugify_strips_punctuation():\n result = slugify(\"Hello, World! 2024\")\n assert result\n assert isinstance(result, str)\n", + "expected_findings": [ + { + "seed_id": "C2-TI-AST-FINAL-P1", + "severity": "P2" + } + ] + }, + { + "id": "C2-TI-AST-FINAL-P2", + "class": "assertion-specificity", + "kind": "holdout", + "file": "fixture_test.sh", + "source": "#!/usr/bin/env bash\nset -euo pipefail\nroot=$(mktemp -d)\ntrap 'rm -rf \"$root\"' EXIT\nprintf 'b\\na\\nb\\n' > \"$root/in.txt\"\nout=$(python3 tools/dedupe_sort.py \"$root/in.txt\")\ncase \"$out\" in\n *a*) echo ok ;;\n *) exit 1 ;;\nesac\n", + "expected_findings": [ + { + "seed_id": "C2-TI-AST-FINAL-P2", + "severity": "P2" + } + ] + }, + { + "id": "C2-TI-AST-FINAL-N1", + "class": "assertion-specificity", + "kind": "holdout", + "file": "fixture_test.py", + "source": "import pytest\n\ndef test_parse_duration_rejects_unknown_unit():\n with pytest.raises(ValueError, match=r\"^unknown unit 'w' in '3w'$\"):\n parse_duration(\"3w\")\n assert parse_duration(\"90s\") == 90\n", + "expected_findings": [] + }, + { + "id": "C2-TI-ENV-FINAL-P1", + "class": "hidden-environment-dependencies", + "kind": "holdout", + "file": "fixture_test.py", + "source": "import os\nimport pathlib\n\ndef test_default_profile_selects_executor():\n cfg_path = pathlib.Path(os.path.expanduser(\"~/.config/orch/profile.toml\"))\n profile = load_profile(cfg_path)\n assert profile.executor == \"codex\"\n", + "expected_findings": [ + { + "seed_id": "C2-TI-ENV-FINAL-P1", + "severity": "P1" + } + ] + }, + { + "id": "C2-TI-ENV-FINAL-P2", + "class": "hidden-environment-dependencies", + "kind": "holdout", + "file": "fixture_test.sh", + "source": "#!/usr/bin/env bash\nset -euo pipefail\nroot=$(mktemp -d)\ntrap 'rm -rf \"$root\"' EXIT\nprintf 'beta\\nAlpha\\nalpha\\nBeta\\n' > \"$root/names.txt\"\nsort \"$root/names.txt\" > \"$root/sorted.txt\"\nprintf 'alpha\\nAlpha\\nbeta\\nBeta\\n' > \"$root/expected.txt\"\ncmp \"$root/sorted.txt\" \"$root/expected.txt\"\n", + "expected_findings": [ + { + "seed_id": "C2-TI-ENV-FINAL-P2", + "severity": "P1" + } + ] + }, + { + "id": "C2-TI-ENV-FINAL-N1", + "class": "hidden-environment-dependencies", + "kind": "holdout", + "file": "fixture_test.py", + "source": "import subprocess\nimport sys\n\ndef test_epoch_renders_in_utc(tmp_path):\n env = {\"PATH\": \"/usr/bin:/bin\", \"HOME\": str(tmp_path), \"TZ\": \"UTC\", \"LC_ALL\": \"C\"}\n code = \"import time; print(time.strftime('%H:%M', time.localtime(0)))\"\n proc = subprocess.run([sys.executable, \"-c\", code], env=env, capture_output=True, text=True, check=True)\n assert proc.stdout == \"00:00\\n\"\n", + "expected_findings": [] + }, + { + "id": "C2-TI-GIT-FINAL-P1", + "class": "fixture-git-signing-hooks", + "kind": "holdout", + "file": "fixture_test.sh", + "source": "#!/usr/bin/env bash\nset -euo pipefail\nroot=$(mktemp -d)\ntrap 'rm -rf \"$root\"' EXIT\ngit init -q \"$root/repo\"\ngit -C \"$root/repo\" config user.name Fixture\ngit -C \"$root/repo\" config user.email fixture@example.invalid\nprintf 'v1\\n' > \"$root/repo/VERSION\"\ngit -C \"$root/repo\" add VERSION\ngit -C \"$root/repo\" commit -q -m init\n[ \"$(git -C \"$root/repo\" log --format=%s)\" = \"init\" ]\n", + "expected_findings": [ + { + "seed_id": "C2-TI-GIT-FINAL-P1", + "severity": "P1" + } + ] + }, + { + "id": "C2-TI-GIT-FINAL-P2", + "class": "fixture-git-signing-hooks", + "kind": "holdout", + "file": "fixture_test.py", + "source": "import subprocess\n\ndef test_tag_points_at_head(tmp_path):\n def git(*args):\n cmd = [\"git\", \"-C\", str(tmp_path), \"-c\", \"commit.gpgsign=false\", \"-c\", \"tag.gpgsign=false\", \"-c\", \"core.hooksPath=\", *args]\n return subprocess.run(cmd, check=True, capture_output=True, text=True)\n git(\"init\", \"-q\")\n (tmp_path / \"a.txt\").write_text(\"a\\n\")\n git(\"add\", \"a.txt\")\n git(\"commit\", \"-q\", \"-m\", \"a\")\n git(\"tag\", \"v0.1.0\")\n assert git(\"rev-parse\", \"v0.1.0^{commit}\").stdout == git(\"rev-parse\", \"HEAD\").stdout\n", + "expected_findings": [ + { + "seed_id": "C2-TI-GIT-FINAL-P2", + "severity": "P1" + } + ] + }, + { + "id": "C2-TI-GIT-FINAL-N1", + "class": "fixture-git-signing-hooks", + "kind": "holdout", + "file": "fixture_test.sh", + "source": "#!/usr/bin/env bash\nset -euo pipefail\nroot=$(mktemp -d)\ntrap 'rm -rf \"$root\"' EXIT\nmkdir \"$root/hooks\"\nexport GIT_CONFIG_GLOBAL=/dev/null GIT_CONFIG_NOSYSTEM=1\ngit init -q \"$root/repo\"\ng() { git -C \"$root/repo\" -c user.name=Fixture -c user.email=fx@example.invalid -c commit.gpgsign=false -c core.hooksPath=\"$root/hooks\" \"$@\"; }\nprintf 'x\\n' > \"$root/repo/f\"\ng add f\ng commit -q -m seed\n[ -z \"$(ls -A \"$root/hooks\")\" ] && [ \"$(g log --format=%an:%s)\" = \"Fixture:seed\" ]\n", + "expected_findings": [] + } + ] +} \ No newline at end of file diff --git a/tests/conformance/review/test-integrity-corpus.json b/tests/conformance/review/test-integrity-corpus.json new file mode 100644 index 0000000..13f3f8a --- /dev/null +++ b/tests/conformance/review/test-integrity-corpus.json @@ -0,0 +1,214 @@ +{ + "schema": "c2-ti/v1", + "description": "Deterministic test-integrity lane corpus", + "classes": [ + {"id":"network-isolation","severity":"P1"}, + {"id":"shared-temporary-directories","severity":"P1"}, + {"id":"verifier-fail-open","severity":"P1"}, + {"id":"assertion-specificity","severity":"P2"}, + {"id":"hidden-environment-dependencies","severity":"P1"}, + {"id":"fixture-git-signing-hooks","severity":"P1"} + ], + "cases": [ + { + "id": "C2-TI-NET-01", + "class": "network-isolation", + "kind": "seed", + "file": "fixture_test.py", + "source": "import requests\n\ndef test_fetch():\n requests.get(\"https://example.invalid\")\n", + "expected_findings": [{"seed_id":"C2-TI-NET-01","severity":"P1"}] + }, + { + "id": "C2-TI-NET-02", + "class": "network-isolation", + "kind": "seed", + "file": "fixture_test.py", + "source": "import socket\n\ndef test_socket():\n socket.create_connection((\"example.invalid\", 443))\n", + "expected_findings": [{"seed_id":"C2-TI-NET-02","severity":"P1"}] + }, + { + "id": "C2-TI-NET-HOLDOUT", + "class": "network-isolation", + "kind": "holdout", + "file": "fixture_test.py", + "source": "def test_stubbed_http():\n response = fake_http.get(\"fixture://local\")\n assert response.status == 200\n", + "expected_findings": [] + }, + { + "id": "C2-TI-TMP-01", + "class": "shared-temporary-directories", + "kind": "seed", + "file": "fixture_test.py", + "source": "from pathlib import Path\n\ndef test_shared_path():\n output = Path(\"/tmp/shared-fixture/output.json\")\n", + "expected_findings": [{"seed_id":"C2-TI-TMP-01","severity":"P1"}] + }, + { + "id": "C2-TI-TMP-02", + "class": "shared-temporary-directories", + "kind": "seed", + "file": "fixture_test.py", + "source": "import tempfile\n\ndef test_process_temp():\n output = tempfile.gettempdir() + \"/fixture-output\"\n", + "expected_findings": [{"seed_id":"C2-TI-TMP-02","severity":"P1"}] + }, + { + "id": "C2-TI-TMP-HOLDOUT", + "class": "shared-temporary-directories", + "kind": "holdout", + "file": "fixture_test.py", + "source": "import tempfile\n\ndef test_unique_temp():\n with tempfile.TemporaryDirectory(prefix=\"c2-ti-\") as root:\n assert root.is_dir()\n", + "expected_findings": [] + }, + { + "id": "C2-TI-VER-01", + "class": "verifier-fail-open", + "kind": "seed", + "file": "fixture_test.py", + "source": "def verify(result):\n try:\n return result.is_valid()\n except Exception: return True\n", + "expected_findings": [{"seed_id":"C2-TI-VER-01","severity":"P1"}] + }, + { + "id": "C2-TI-VER-02", + "class": "verifier-fail-open", + "kind": "seed", + "file": "fixture_test.py", + "source": "def verify(result):\n try:\n return result.is_valid()\n except Exception: pass\n", + "expected_findings": [{"seed_id":"C2-TI-VER-02","severity":"P1"}] + }, + { + "id": "C2-TI-VER-HOLDOUT", + "class": "verifier-fail-open", + "kind": "holdout", + "file": "fixture_test.py", + "source": "def verify(result):\n try:\n return result.is_valid()\n except Exception:\n raise VerificationError(\"invalid\")\n", + "expected_findings": [] + }, + { + "id": "C2-TI-AST-01", + "class": "assertion-specificity", + "kind": "seed", + "file": "fixture_test.py", + "source": "def test_result():\n result = run_fixture()\n assert result\n", + "expected_findings": [{"seed_id":"C2-TI-AST-01","severity":"P2"}] + }, + { + "id": "C2-TI-AST-02", + "class": "assertion-specificity", + "kind": "seed", + "file": "fixture_test.py", + "source": "def test_output():\n output = run_fixture()\n assert \"passed\" in output\n", + "expected_findings": [{"seed_id":"C2-TI-AST-02","severity":"P2"}] + }, + { + "id": "C2-TI-AST-HOLDOUT", + "class": "assertion-specificity", + "kind": "holdout", + "file": "fixture_test.py", + "source": "def test_result():\n result = run_fixture()\n assert result.exit_code == 0\n assert result.stdout == \"passed\"\n", + "expected_findings": [] + }, + { + "id": "C2-TI-ENV-01", + "class": "hidden-environment-dependencies", + "kind": "seed", + "file": "fixture_test.py", + "source": "import os\n\ndef test_home():\n config_root = os.environ[\"HOME\"]\n", + "expected_findings": [{"seed_id":"C2-TI-ENV-01","severity":"P1"}] + }, + { + "id": "C2-TI-ENV-02", + "class": "hidden-environment-dependencies", + "kind": "seed", + "file": "fixture_test.py", + "source": "import os\n\ndef test_path():\n tool_path = os.getenv(\"PATH\")\n", + "expected_findings": [{"seed_id":"C2-TI-ENV-02","severity":"P1"}] + }, + { + "id": "C2-TI-ENV-HOLDOUT", + "class": "hidden-environment-dependencies", + "kind": "holdout", + "file": "fixture_test.py", + "source": "def test_declared_mode(monkeypatch):\n monkeypatch.setenv(\"FIXTURE_MODE\", \"test\")\n assert load_mode() == \"test\"\n", + "expected_findings": [] + }, + { + "id": "C2-TI-GIT-01", + "class": "fixture-git-signing-hooks", + "kind": "seed", + "file": "fixture_test.py", + "source": "import subprocess\n\ndef configure_fixture():\n subprocess.run([\"git\", \"config\", \"--global\", \"core.hooksPath\", \"/host/hooks\"])\n", + "expected_findings": [{"seed_id":"C2-TI-GIT-01","severity":"P1"}] + }, + { + "id": "C2-TI-GIT-02", + "class": "fixture-git-signing-hooks", + "kind": "seed", + "file": "fixture_test.py", + "source": "import subprocess\n\ndef configure_fixture():\n subprocess.run([\"git\", \"config\", \"--global\", \"commit.gpgsign\", \"true\"])\n", + "expected_findings": [{"seed_id":"C2-TI-GIT-02","severity":"P1"}] + }, + { + "id": "C2-TI-GIT-HOLDOUT", + "class": "fixture-git-signing-hooks", + "kind": "holdout", + "file": "fixture_test.py", + "source": "import subprocess\n\ndef make_fixture_repo(tmp_path):\n repo, hooks = tmp_path / \"repo\", tmp_path / \"hooks\"\n hooks.mkdir()\n subprocess.run([\"git\", \"init\", \"-q\", str(repo)], check=True)\n for key, value in ((\"user.name\", \"Fixture\"), (\"user.email\", \"fixture@example.invalid\"),\n (\"commit.gpgsign\", \"false\"), (\"core.hooksPath\", str(hooks))):\n subprocess.run([\"git\", \"-C\", str(repo), \"config\", \"--local\", key, value], check=True)\n assert list(hooks.iterdir()) == []\n return repo\n", + "expected_findings": [] + }, + { + "id": "C2-TI-NET-03", + "class": "network-isolation", + "kind": "seed", + "file": "fixture_test.py", + "source": "import urllib.request\n\ndef test_unmocked_urlopen():\n urllib.request.urlopen(\"https://example.invalid\")\n", + "expected_findings": [{"seed_id":"C2-TI-NET-03","severity":"P1"}] + }, + { + "id": "C2-TI-TMP-03", + "class": "shared-temporary-directories", + "kind": "seed", + "file": "fixture_test.py", + "source": "from pathlib import Path\n\ndef test_fixed_cache_path():\n cache = Path(\"/tmp/c2-ti-cache.json\")\n", + "expected_findings": [{"seed_id":"C2-TI-TMP-03","severity":"P1"}] + }, + { + "id": "C2-TI-VER-03", + "class": "verifier-fail-open", + "kind": "seed", + "file": "fixture_test.py", + "source": "def verify():\n result = subprocess.run([\"verify\"], check=False)\n", + "expected_findings": [{"seed_id":"C2-TI-VER-03","severity":"P1"}] + }, + { + "id": "C2-TI-AST-03", + "class": "assertion-specificity", + "kind": "seed", + "file": "fixture_test.py", + "source": "def test_result():\n result = run_fixture()\n assert result.exit_code\n", + "expected_findings": [{"seed_id":"C2-TI-AST-03","severity":"P2"}] + }, + { + "id": "C2-TI-ENV-03", + "class": "hidden-environment-dependencies", + "kind": "seed", + "file": "fixture_test.py", + "source": "import os\n\ndef test_home_default():\n home = os.environ.get(\"HOME\")\n", + "expected_findings": [{"seed_id":"C2-TI-ENV-03","severity":"P1"}] + }, + { + "id": "C2-TI-GIT-03", + "class": "fixture-git-signing-hooks", + "kind": "seed", + "file": "fixture_test.py", + "source": "import subprocess\n\ndef configure_fixture():\n subprocess.run([\"git\", \"config\", \"--global\", \"commit.gpgsign\", \"false\"])\n", + "expected_findings": [{"seed_id":"C2-TI-GIT-03","severity":"P1"}] + }, + { + "id": "C2-TI-CLEAN", + "class": "clean-fixture", + "kind": "clean", + "file": "fixture_test.py", + "source": "import tempfile\n\ndef test_clean_fixture():\n with tempfile.TemporaryDirectory(prefix=\"c2-ti-\") as root:\n assert root.startswith(\"/\")\n", + "expected_findings": [] + } + ] +}