test(early-stop): CE036 enforces the live_verdict determinism + monotonicity contract - #126
Conversation
…onicity contract Closes the two asks of issue #61 still open after PRs #74/#78 (which shipped the weighted gate, decide_within, and the armed-subset bounds): item 2's enforcement half, and item 4's design decision. Item 2 — enforcement. EarlyStopWatcher's verdict latching, deferred fail-stop, and pass-stop flip-attribution are correct only while every armed criterion's live_verdict is a deterministic, monotonic function of the trajectory prefix. That contract was documented on LiveVerdict/BaseCriterion.live_verdict but nothing enforced it: a third criterion implementing it non-monotonically would type-check, pass CE025, and silently corrupt the stop logic. Monotonicity over arbitrary Python is undecidable, so there is no sound static rule to write. CE036 instead REPLAYS each live criterion against every prefix of recorded trajectories and asserts the property directly, plus two registry-derived coverage checks that stop the fixture table from decaying into a vacuous always-undecided replay: - every LiveSuccessCriterion in the SuccessCriterion union must have cases (a property test over random trajectories would pass vacuously); - every polarity an instance claims via live_decidable_polarities() must actually be reached by some case. Each case also pins the verdict it reaches (so a rotted fixture fails loudly) and is checked for polarity honesty — a terminal decision outside the instance's declared polarities means the watcher would treat a live trigger as inert while the checker decides it. A raise from live_verdict is itself reported as a labeled violation (case + prefix length) and the walk continues, so one bad prefix cannot mask breaches elsewhere — the contract is degrade to 'undecided', never raise, exactly the shape the malformed-regex fixture pins. Six detect-tests with synthetic checkers prove the harness fires rather than passing vacuously. Both existing checkers (skill_triggered, command_executed) are confirmed clean across 13 cases. docs/EXTENDING.md now tells a live-criterion author the fixtures are required in the same change (and that a plugin criterion, invisible to the union walk, should reuse contract_violations in its own suite). Item 4 — design decision, recorded in TASK_DEFINITION_GUIDE § stop_early: the ceiling/floor bounds deliberately stop at the armed subset. A non-observable criterion's bound can never tighten past the vacuous [0, 1] without end-state peeking or per-tool-call judge runs (non-monotonic — the exact false-stop risk the bound design rules out), and permanently-vacuous bounds folded into the gate degenerate to never-stop: an undecided criterion holds the ceiling up and the floor down for the whole run. Arming is the author's declaration of which criteria the smoke verdict may hinge on; the weighted-score break the issue asked about is arm + weight + stop_early_gate_threshold, expressed over the subset that can actually decide mid-run. Honest limits, documented on the rule and the contract: replay proves the contract on supplied trajectories, not in general, and only for in-tree types. Telemetry/turn builders are shared with test_early_stop.py via tests/_fixtures/live_criteria.py (frozen timestamp; thin wrappers keep the 95 existing call sites and their tool-<name> ids byte-identical), and the determinism probe's docstring states its wall-clock limit honestly instead of overclaiming it. Two hardening layers on the rule itself: permuted_violations re-runs the determinism+monotonicity walk under seeded shuffles of every case (an order-sensitive verdict — e.g. read off the latest command — is monotone on the authored ordering and only a reordering exposes it; terminal-verdict and polarity checks stay authored-ordering-only where they are sound), and make typecheck now pyright-checks the contract engine + shared fixtures explicitly (the config's tests/ exclude beats include, so a second invocation with file args is the only working mechanism).
… + CE036 cases Three NON-CI example tasks (tasks/early_stop_contract_*.yaml), each a live-run counterfactual demonstrating the verdict corruption a live_verdict contract violation causes — the concrete harm CE036 (issue #61 item 2) exists to prevent. Each was executed twice against a real agent: once on clean code (correct verdict, correct stop) and once with a deliberately contract-breaking mutant (verdict silently flipped), with the mutant passing the entire pre-CE036 test surface and failing only CE036: - require_success: a checker that counts a crashed match latches a false live PASS, pass-stops before the deliverable exists, and flips SUCCESS to FAILURE (evidence truncation). Caught by the require_success replay cases. - bounded_pass: a two-sided mutant (model claims the pass polarity for a bounded range + checker latches pass at min_count) pass-stops on the first match and freezes a still-compliant count, flipping a deserved FAILURE to SUCCESS (evidence freezing). Correct code instead fail-stops on the max_count overrun, verdict-preserving. Caught as NON-MONOTONIC. - any_engagement: a first-engagement regression on skill_triggered fail-stops on a foreign skill-path read before the expected skill can engage, flipping SUCCESS to FAILURE. Caught as NON-MONOTONIC on the distractor-first fixture. The two trajectories not already pinned verbatim are added as ContractCases (bounded window-then-overrun for command_executed; foreign-path-read-first for skill_triggered), so the exact live-demonstrated walks stay enforced. The probes also documented a runtime finding: the watcher's polarity filtering (live_decidable_polarities-derived triggers) already defends against checker-only out-of-polarity mutants, so the mutants that reach production are two-sided — model claim + checker drift together — which is exactly the class the monotonicity replay catches independently of polarity claims.
|
Claude finished @mohsen-uipath's task in 1m 21s —— View job Code Review in Progress
|
…mport it tests/ is not shipped in the PyPI wheel, so pointing an out-of-tree plugin at 'reuse contract_violations from that module' promised an import an installed consumer cannot make. Both surfaces (EXTENDING.md + the module's honest-limits docstring) now say: copy the replay pattern (ContractCase-style fixture + the prefix walk) into the plugin's own test suite, with this module as the reference implementation.
bai-uipath
left a comment
bai-uipath
left a comment
There was a problem hiding this comment.
Drop the three example tasks; fix the rest if you agree, otherwise lgtm. The replay approach is right — monotonicity isn't statically decidable, and the registry-derived coverage checks are what keep the fixture table from rotting into a vacuous always-undecided pass. That part is clean.
Drop the three tasks/early_stop_contract_*.yaml. The mutants aren't checked in, so running them only shows the clean-code half — the counterfactual can't be reproduced from the repo, and the enforced version of that evidence is already the two new ContractCases. One of them also fails by design in the directory people point coder-eval run at. Fix: delete all three; keep the narrative in the PR body.
The second pyright invocation analyzes zero files. Pyright's exclude beats an explicitly-passed CLI file arg, so the extra make typecheck pass exits 0 having checked nothing — I confirmed with a probe file (filesAnalyzed: 0 under tests/, 1 under src/). Both new modules are clean when actually checked, so this costs nothing to fix. Fix: a second config file with those paths in include, or drop "tests" from the exclude list.
Drop the shared telemetry-builder module. It shares two six-line constructors between exactly two files, and the wrappers still have to reconcile two different tool_id conventions — while the judge test files keep their own builders. Fix: build the telemetry inline in the contract module and revert the early-stop test file entirely.
Minor: the empty-table test hardcodes today's two type names, so adding a third live criterion reds an unrelated test at exactly the wrong moment — compare against the registry instead. A raise on the final prefix leaves the terminal verdict stale and stacks a bogus reaches violation on the real one.
Worth considering: you extracted 650+ real activation-suite pairs to validate this and checked in none. Three of them as ContractCases would move the standing gate off 38 hand-written tool calls onto trajectory shapes nobody would think to author.
…-engine gaps Review follow-ups from #126. 1. The second `make typecheck` pass analyzed ZERO files. pyright's `exclude` beats an explicitly-passed CLI file arg, so `pyright tests/lint/...` exited 0 having checked nothing -- and an `include` entry naming the file is dropped the same way (both verified with a probe file carrying a deliberate error). The pass now runs under its own config, generated by `tests/lint/pyright_config.py` from `[tool.pyright]` so the two passes cannot drift apart. It is also wired into CI (Linux + Windows), which invokes pyright directly and never ran the Makefile pass at all. Once actually checking, it caught six real `reportImplicitStringConcatenation` errors in the contract engine; those are fixed. 2. The empty-table test pinned today's two type names, so adding a live criterion would red the harness self-test at exactly the moment `test_every_live_criterion_type_has_cases` is already failing the author with the actionable message. It now compares against the registry. 3. A raise on the TERMINAL prefix left `_walk_prefixes` returning the previous prefix's stale verdict, stacking a phantom `reaches` breach on top of the real RAISED one. The walk now reports no terminal verdict for that case and `contract_violations` skips checks 4 and 5. 4. `permuted_violations` shuffled commands but left their original `sequence_number` values attached -- trajectories the watcher cannot produce, since `EarlyStopWatcher._collect_verdicts` keeps its partial trajectory sorted by that field. Worse, the layer would degrade to a silent no-op for any future checker that sorts by it (the shuffle sorts straight back). Each shuffle is now renumbered 0..N-1. Fixes 3 and 4 are pinned by new tests, both confirmed to fail against the pre-fix engine. `make verify` green: 4,165 tests, 360 lint, pyright 0 errors on both passes.
Review follow-up from #126. The mutant half of each probe is not checked in, so running these from the repo only shows the clean-code side -- the counterfactual they document cannot be reproduced by a reader. The enforced form of that evidence is already in the fixture table: the two live-demonstrated trajectories stay pinned as ContractCases, and the third was already pinned verbatim. Their provenance comments now cite the PR rather than the deleted files, and say why the probe tasks are not checked in. The narrative stays in the PR body. `make verify` green: 4,162 tests, 357 lint (both -3, purely the per-task parametrization over tasks/), pyright 0 errors on both passes.
|
Thanks — four fixed, one left for your call. Pyright. Confirmed, and worse than it looked: Final-prefix raise. Fixed. The existing raise test passed only by accident of its fixture, so the case had no coverage; it does now. Hardcoded type names. Fixed — compared against the registry. The three probe tasks. Dropped. The Shared telemetry module. Left as is — your facts hold (2 of 20 files). The one thing worth keeping is the frozen timestamp, which CE036 needs. Say the word and the builders go inline in the contract module and Real trajectories. Agreed, follow-up issue rather than holding this PR. One more:
|
2 participants
Closes the two asks of #61 still open after #74/#78: item 2's enforcement half (the
live_verdictcontract was documented but nothing enforced it) and item 4's design decision (recorded in TASK_DEFINITION_GUIDE §stop_early: the ceiling/floor bounds deliberately stop at the armed subset — the weighted-score break is arm + weight +stop_early_gate_threshold).Zero runtime change: the only
src/diff is comments/docstrings incriteria/base.py(AST-verified identical to main with docstrings stripped). The deliverable is a CI-time gate: monotonicity over arbitrary Python is undecidable statically, so CE036 replays each live criterion against every prefix of recorded trajectories (authored ordering + seeded shuffles) and asserts determinism/monotonicity/no-raise directly, with registry-derived coverage (everyLiveSuccessCriterionneeds cases; every claimed polarity must be reached) so the fixture table can't decay into a vacuous always-undecidedreplay.Verification
Sensitivity — mutation testing. Six contract-breaking mutants injected into the real shipped checkers; CE036 caught all six with actionable messages, including a two-sided class (model polarity claim + checker drift together) that defeats the watcher's own polarity filtering: non-monotonic, non-deterministic (RNG), raising instead of degrading, polarity-dishonest, missing fixtures, vacuous all-undecided fixtures.
Specificity + ecological validity. Shipped checkers replay clean across all 15 fixtures and shuffles, and across 650+ (row × criterion) pairs extracted from real live activation-suite runs (26 rows, one per skill stratum, real agent): 0 violations.
Counterfactual harm — live probes (2nd commit). Three NON-CI example tasks (
tasks/early_stop_contract_*.yaml), each executed live twice — clean code vs a contract-breaking mutant. Every mutant passed the entire pre-CE036 test surface and silently flipped the live verdict; only CE036 fails:require_success(crash must not count)reachespinbounded_pass(bounded pass not final)max_countoverrun → FAILUREreachesany_engagement(first-engagement regression)The two trajectories not already pinned verbatim are added as
ContractCases, so the live-demonstrated walks stay enforced.Related context. A kill-switch vs armed A/B on the activation suite (same 26 sampled rows, both arms) re-validated the runtime semantics this contract protects: 25/26 identical verdicts (the one diff traced to agent stochasticity — the watcher never fired), at −73% cost / −60% wall time. The tc-1 pass-stops that deliver those savings are trustworthy precisely because
live_verdictis monotone — which is what CE036 keeps true as new live criteria are added.Gates:
make verifygreen — 4,165 tests, 358 lint (CE036 included), pyright 0 errors (incl. the new explicit pass over the contract engine).Honest limits (documented on the rule): replay proves the contract on supplied trajectories, not in general; in-tree union types only — and since
tests/isn't shipped in the wheel, docs/EXTENDING.md tells plugin authors to copy the replay pattern (aContractCase-style fixture + the prefix walk) into their own test suite, withtests/lint/live_verdict_contract.pyas the reference implementation.