test(early-stop): CE036 enforces the live_verdict determinism + monotonicity contract

.github/workflows/pr-checks.yml

@@ -96,6 +96,15 @@ jobs:
       - name: Type check with pyright
         run: .venv/bin/pyright
 
+      # The CE036 contract engine lives under tests/, which [tool.pyright] excludes
+      # -- and `exclude` beats both a CLI file arg and an `include` entry, so it can
+      # only be reached through a config of its own, derived from [tool.pyright] so
+      # the two passes cannot drift. Mirrors `make typecheck`.
+      - name: Type check the CE036 contract engine
+        run: |
+          .venv/bin/python -m tests.lint.pyright_config .pyright-tests.json
+          .venv/bin/pyright -p .pyright-tests.json
+
       # PHASE 3: Security scanning
       - name: Security - Dependency vulnerabilities (pip-audit)
         run: .venv/bin/pip-audit --desc --skip-editable --ignore-vuln CVE-2026-4539 --ignore-vuln CVE-2026-3219 --ignore-vuln PYSEC-2025-183 # pygments 2.19.2 ReDoS + pip 26.0.1 tar/ZIP ambiguity + pyjwt 2.12.1 weak-encryption (disputed by supplier; key length is application-chosen); no fixes available on PyPI yet — revisit quarterly
@@ -387,6 +396,11 @@ jobs:
       - name: Type check with pyright
         run: .venv/Scripts/pyright
 
+      - name: Type check the CE036 contract engine
+        run: |
+          .venv/Scripts/python -m tests.lint.pyright_config .pyright-tests.json
+          .venv/Scripts/pyright -p .pyright-tests.json
+
       - name: Run test suite
         run: .venv/Scripts/pytest tests/ -v -m "not live and not lint"
 

.gitignore

@@ -69,3 +69,6 @@ refs/
 
 # SkillsBench tasks for testing
 /resources/
+
+# Derived pyright config for the CE036 contract engine (tests/lint/pyright_config.py)
+.pyright-tests.json

CLAUDE.md

@@ -214,7 +214,7 @@ make plugin-reference  # the plugin's bundled criteria reference from the models
 
 Editing `src/coder_eval/pricing.py` means editing `evalboard/lib/pricing.ts` too — it is a hand-copied mirror, and `evalboard/lib/__tests__/pricing-parity.test.ts` fails the build on drift in either direction.
 
-When fixing a bug, ask: *could a custom lint rule have prevented this?* If the root cause is a mechanically detectable pattern (e.g., "always import from `coder_eval.models`", "never call blocking IO in async"), add a rule to `tests/lint/rules/` following the CE001+ pattern and wire it up in `tests/lint/runner.py`. This turns a one-time fix into permanent enforcement. See `tests/test_custom_lint.py` for how rules are tested. (Doc-surface / whole-tree rules that reason over Markdown/YAML or the entire `src/` tree rather than one `.py` AST at a time — CE026–CE031, CE033, CE034, CE035 — are not `BaseRule`s in the runner; they are wired as dedicated `@pytest.mark.lint` test classes. CE035 resolves every `steps.<id>.outputs.<key>` / `needs.<job>.outputs.<key>` reference in `.github/workflows/**` to a writer that actually produces that key — GitHub expands an unwritten output to the empty string, so a typo degrades a gate silently and actionlint models `steps.*.outputs` as an open string map. CE034 scans `tasks/` and forces an armed, live-*passable* `command_executed` to set `require_success` — a crashed invocation would otherwise latch a live PASS, fire `on_pass: stop`, and let FIRED-ONLY armed gating report SUCCESS without ever consulting the unarmed criteria (negative assertions are fail-only and are exempt). CE033 keeps the plugin's bundled `reference/criteria.md` in parity with the `SuccessCriterion` union that generates it (`make plugin-reference` writes it; the rule re-renders and diffs — never hand-edit the file). CE031 guards against dead config: a behavior-driving field on `SimulationConfig`/`RunLimits`/`Dataset` that no code reads by name. CE026 keeps the GitHub Action's onboarding surfaces honest — `README.md`, `docs/CI_GATE.md`, `docs/tutorials/02-ci-pipeline.md`, and the plugin's `ci` skill, whose emitted workflow users copy into their own repos: a page's *first* Action snippet must show the agent-runtime prerequisite steps (pinned to the `action-dogfood` job that proves them in CI), a zero-install absolute next to such a snippet must name the channel it means, every `github.com/marketplace/actions/<slug>` link plus the shields badge label must match `action.yml`'s `name:`, and every `with:` key on a snippet's action step must be a real `action.yml` input (GitHub ignores unknown inputs, so a rename would silently degrade every copied workflow). Renaming an action input or changing its runtime prerequisites therefore means updating the skill too.)
+When fixing a bug, ask: *could a custom lint rule have prevented this?* If the root cause is a mechanically detectable pattern (e.g., "always import from `coder_eval.models`", "never call blocking IO in async"), add a rule to `tests/lint/rules/` following the CE001+ pattern and wire it up in `tests/lint/runner.py`. This turns a one-time fix into permanent enforcement. See `tests/test_custom_lint.py` for how rules are tested. (Doc-surface / whole-tree rules that reason over Markdown/YAML or the entire `src/` tree rather than one `.py` AST at a time — CE026–CE031, CE033–CE036 — are not `BaseRule`s in the runner; they are wired as dedicated `@pytest.mark.lint` test classes. CE036 enforces the `live_verdict` determinism + monotonicity contract (`criteria/base.py`) that `EarlyStopWatcher`'s latching, deferred fail-stop, and flip-attribution silently depend on: monotonicity over arbitrary Python is undecidable, so instead of a static check it REPLAYS each live criterion against every prefix of recorded trajectories (`tests/lint/live_verdict_contract.py::CASES`) — on the authored ordering AND under seeded shuffles (`permuted_violations`, which catch order-sensitive bugs the authored walk misses) — and asserts the property directly, plus registry-derived coverage — every `LiveSuccessCriterion` in the union must have cases, and every polarity its instances claim via `live_decidable_polarities()` must actually be reached by one (otherwise a single always-`undecided` fixture would "cover" a type while proving nothing). Adding a live criterion therefore means adding `ContractCase`s in the same change. CE035 resolves every `steps.<id>.outputs.<key>` / `needs.<job>.outputs.<key>` reference in `.github/workflows/**` to a writer that actually produces that key — GitHub expands an unwritten output to the empty string, so a typo degrades a gate silently and actionlint models `steps.*.outputs` as an open string map. CE034 scans `tasks/` and forces an armed, live-*passable* `command_executed` to set `require_success` — a crashed invocation would otherwise latch a live PASS, fire `on_pass: stop`, and let FIRED-ONLY armed gating report SUCCESS without ever consulting the unarmed criteria (negative assertions are fail-only and are exempt). CE033 keeps the plugin's bundled `reference/criteria.md` in parity with the `SuccessCriterion` union that generates it (`make plugin-reference` writes it; the rule re-renders and diffs — never hand-edit the file). CE031 guards against dead config: a behavior-driving field on `SimulationConfig`/`RunLimits`/`Dataset` that no code reads by name. CE026 keeps the GitHub Action's onboarding surfaces honest — `README.md`, `docs/CI_GATE.md`, `docs/tutorials/02-ci-pipeline.md`, and the plugin's `ci` skill, whose emitted workflow users copy into their own repos: a page's *first* Action snippet must show the agent-runtime prerequisite steps (pinned to the `action-dogfood` job that proves them in CI), a zero-install absolute next to such a snippet must name the channel it means, every `github.com/marketplace/actions/<slug>` link plus the shields badge label must match `action.yml`'s `name:`, and every `with:` key on a snippet's action step must be a real `action.yml` input (GitHub ignores unknown inputs, so a rename would silently degrade every copied workflow). Renaming an action input or changing its runtime prerequisites therefore means updating the skill too.)
 
 Adding a user-facing field to one of the models CE030 tracks (`TaskDefinition`, `RunLimits`, `Dataset`, `SimulationConfig` — see `tests/lint/doc_schema_parity.py`) means documenting it in its guide (mention the field name as inline code) or adding an `EXEMPT` entry with a reason it is not user-authored. `make lint` fails otherwise.
 

Makefile

@@ -38,6 +38,15 @@ plugin-reference:  ## Regenerate the plugin's bundled criteria reference from th
 
 typecheck:  ## Run type checking with pyright
 	uv run pyright
+	# The CE036 contract engine executes checker code and feeds the early-stop
+	# design; it is the one tests/ surface worth type-checking. It needs its own
+	# config: pyproject.toml excludes "tests", and pyright's `exclude` beats BOTH
+	# an explicitly-passed CLI file arg AND an `include` entry naming the file --
+	# either shortcut analyzes ZERO files and exits 0, a gate that checks nothing.
+	# The config below is DERIVED from [tool.pyright] (same rules, only
+	# include/exclude swapped), so the two passes cannot drift apart.
+	uv run python -m tests.lint.pyright_config .pyright-tests.json
+	uv run pyright -p .pyright-tests.json
 
 test:  ## Run test suite (excludes live + lint tests; run `make lint` for those)
 	uv run pytest -n auto -m "not live and not lint" tests/

docs/EXTENDING.md

@@ -220,6 +220,21 @@ Notes:
   LiveSuccessCriterion)` directly, no separate checker-side flag. A lint rule
   (`tests/test_custom_lint.py::TestCE025LiveVerdictConsistency`) keeps the
   model subclassing and the checker's `live_verdict` override paired.
+- Your `live_verdict` must be **deterministic** (a pure function of the
+  `turn_records` prefix — no wall-clock, randomness, or hidden instance state)
+  and **monotonic** (once it returns `"pass"`/`"fail"` for some prefix, every
+  longer prefix returns that same verdict) — `EarlyStopWatcher`'s verdict
+  latching and deferred stops silently depend on both. Lint rule CE036
+  (`tests/lint/live_verdict_contract.py`) enforces this by replaying each live
+  criterion against every prefix of recorded trajectories, and **fails until
+  you add `ContractCase` fixtures** for the new type in the same change,
+  reaching every polarity its instances claim via
+  `live_decidable_polarities()`. An out-of-tree plugin criterion is invisible
+  to CE036's union walk — and the module lives under `tests/`, which is not
+  shipped in the PyPI wheel — so copy the replay pattern (a `ContractCase`-style
+  fixture plus the prefix-by-prefix determinism/monotonicity walk) into your
+  plugin's own test suite, using `tests/lint/live_verdict_contract.py` in this
+  repo as the reference implementation.
 
 > A duplicate `criterion_type` **overwrites** the earlier checker with a warning (not
 > a hard error, unlike agents) — keep type strings unique.

docs/TASK_DEFINITION_GUIDE.md

@@ -468,6 +468,26 @@ Semantics:
   before weighting) — only the combination rule (weighted average vs strict
   AND) changes, which is what makes the `gate_threshold=1.0` default an exact
   equivalence with the strict `all(...)` rule.
+- **Why the bounds stop at the armed subset (design decision).** The
+  ceiling/floor rule deliberately does **not** extend to unarmed or
+  non-observable criteria (`llm_judge`, `reference_comparison`, the file
+  checks). A mid-run score bound for those would require either scoring the
+  unfinished sandbox (the end-state peeking `live_verdict` forbids by
+  construction — it reads only `turn_records`) or re-running judges on every
+  tool call (expensive, and judge scores over a partial trajectory are not
+  monotonic — exactly the false-stop risk the bound design exists to rule
+  out). So a non-observable criterion's bound can never tighten past the
+  vacuous `[0, 1]` — and folding permanently-vacuous bounds into the gate
+  degenerates to "never stop": an undecided criterion holds the ceiling up
+  (suppressing every fail-stop) and the floor down (vetoing every pass-stop)
+  for the whole run. Scoping the gate to the armed subset is therefore not a
+  simplification but the design: **arming is the author's declaration of
+  which criteria the smoke verdict is allowed to hinge on**, and the
+  authoritative full-set score always comes from the kill-switched run. If a
+  run should end early on overall-score grounds, arm the observable criteria
+  with appropriate `weight`s and lower `stop_early_gate_threshold` — that is
+  the weighted-score break, expressed over the subset that can actually
+  decide mid-run.
 - **Decision-step timeout.** `stop_early: {decide_within: N}`. If the
   criterion is still **undecided**
   after N tool-call steps, the watcher latches an **effective fail** for it and

src/coder_eval/criteria/base.py

@@ -37,8 +37,16 @@
 # (early_stop.py::_prev_verdicts) are correct only because both existing
 # implementations (skill_triggered, command_executed) honor this. A non-monotonic or
 # non-deterministic override compiles and passes CE025 (which only checks
-# LiveSuccessCriterion subclassing / live_verdict pairing, not this) but silently corrupts the stop
-# logic — there is currently no automated enforcement beyond this docstring.
+# LiveSuccessCriterion subclassing / live_verdict pairing, not this) but silently corrupts
+# the stop logic.
+#
+# ENFORCEMENT: lint rule CE036 (tests/lint/live_verdict_contract.py) replays every live
+# criterion against every prefix of recorded trajectories and asserts both properties —
+# monotonicity over arbitrary Python is undecidable, so replay is the only sound check.
+# Adding a LiveSuccessCriterion REQUIRES adding ContractCase fixtures for it in the same
+# change (CE036 fails on a live type with no cases, and on a polarity its instances claim
+# decidable but no fixture reaches). Note the limit: CE036 proves the contract on the
+# trajectories an author supplied, not in general — honoring it is still on the author.
 LiveVerdict = Literal["pass", "fail", "undecided"]
 
 
@@ -438,8 +446,9 @@ def live_verdict(
         source of truth for "is this criterion type live-observable", checked
         by ``validate_early_stop`` / ``EarlyStopWatcher`` and enforced by lint
         rule CE025. An override MUST also satisfy the deterministic + monotonic
-        contract documented on the ``LiveVerdict`` type above (not enforced by
-        CE025 or any other automated check).
+        contract documented on the ``LiveVerdict`` type above, enforced by lint
+        rule CE036 — which requires this criterion type to supply replay fixtures
+        (``tests/lint/live_verdict_contract.py::CASES``) in the same change.
         """
         return "undecided"
 

tests/_fixtures/live_criteria.py

@@ -0,0 +1,47 @@
+"""Shared builders for live-criterion trajectories (early-stop tests + CE036).
+
+``tests/test_early_stop.py`` (the watcher's behavioral suite) and
+``tests/lint/live_verdict_contract.py`` (the CE036 contract-replay fixtures) both
+hand-build ``CommandTelemetry``/``TurnRecord`` trajectories for the same two live
+checkers. The primitives live here so a telemetry field addition is threaded
+through once; the *criterion* builders deliberately stay in each file — they
+encode different defaults (armed with ``stop_early`` blocks vs unarmed contract
+instances) and sharing them would just move the divergence into keyword soup.
+
+The timestamp is frozen: CE036's determinism replay requires fixtures that carry
+no nondeterminism of their own, and the watcher tests never read wall-clock off
+telemetry either.
+"""
+
+from __future__ import annotations
+
+from datetime import datetime
+from typing import Any, Literal
+
+from coder_eval.models import CommandTelemetry, TurnRecord
+
+
+FROZEN_TS = datetime(2026, 1, 1, 0, 0, 0)
+
+
+def make_command(
+    tool_name: str,
+    parameters: dict[str, Any],
+    *,
+    tool_id: str | None = None,
+    sequence_number: int = 0,
+    result_status: Literal["success", "error", "unknown"] = "success",
+) -> CommandTelemetry:
+    """One recorded tool call. ``tool_id`` defaults to ``tool-<sequence_number>``."""
+    return CommandTelemetry(
+        tool_name=tool_name,
+        tool_id=tool_id if tool_id is not None else f"tool-{sequence_number}",
+        timestamp=FROZEN_TS,
+        parameters=parameters,
+        result_status=result_status,
+        sequence_number=sequence_number,
+    )
+
+
+def make_turn(*commands: CommandTelemetry, iteration: int = 1) -> TurnRecord:
+    return TurnRecord(iteration=iteration, user_input="", agent_output="", commands=list(commands))