Repository navigation
Description
Sub-issue of #937, the run-health slice. #991 lands only the artifact-change slice of #937 (Partial: artifact-change truthfulness for #937), so the rest of #937 is split out one behaviour per issue; this is the second one.
Problem
Provider and harness failures do not reach the decision. When a provider crashes, an eval raises, or a worker is retired, the task today either raises out of the run or lands as a scored 0. Either way the failure rate is invisible at the point where promote() reads the evidence, so a run in which a third of the tasks never produced a real answer can still hand a clean mean gain to the promoter. The 2026-11-06 benchmark session recorded exactly this, twice: exceptions folded into the mean, and a content-filter crash that produced a final answer no agent generated.
Scope (one behaviour)
Run health becomes evidence that can fail the gate:
- the evidence bundle records the health of the run: how many tasks failed and out of how many, and whether that is above a threshold the caller sets;
- a run above the threshold fails its gate, so
promote()cannot returnpromote; - failed tasks are not scored as
0and are not folded into the mean — the reported gain is over the tasks that actually produced an answer, and the failure count is reported next to it; - callers that pass no health data keep today's evidence exactly: no new keys, existing hashes and decisions unchanged.
Out of scope here: agentdescent/pilot.py, the pilot CLI, the disjoint-test-split rule (#843), censoring policy, and any change to how evolve counts an exception. Those stay on #937 or get their own issue.
Acceptance criteria
A new test file, offline and model-free, that fails on live main and passes on the tip:
- Same scores with a failure rate above the threshold:
promote(build_trust_evidence(...).to_promotion_request(...))hasaction != "promote", and the reason carries the health shortfall rather than a bare score. - Control: the same scores with failures at or below the threshold still promote.
- Failures are not zeroes: with
kfailed tasks, the reported gain is the mean over the tasks that produced a result, and the failure count is on the evidence. - Default is unchanged: with no health data passed, the bundle is byte-identical to main's today (same keys, same hash), asserted not described.
pytest -qstays offline and green; no model, no container.
Verification the PR must carry
- red at the base it was cut from, with the new test file as the only change on the tree, quoting the failure lines;
- the tip's focused suite and the changelog fragment check;
- size and risk tier computed from paths.
Related
- feat(pilot): one evidence-bound decision surface with run-health and artifact-change truthfulness #937 — parent;
pilotcomposition and the remaining criteria. - fix(trust-evidence): an unchanged artifact cannot report gain or promote #991 — the sibling slice already in review.
- evolve: same held-out used for select and reported score (optimistic generalisation) #843 — held-out split, still open.
Written-by: AgentDescent Lead
Bot: AgentDescent Dev-1
Activity
HugoXK commentedon Nov 12, 2026
Claimed: branch
agent/993-run-health-threshold, cut from live main271c9346841ef044ed64565b4ca15594bb161cea; draft PR follows withFixes #993.Red first:
tests/test_run_health_evidence.pyalone at271c9346-> 4 failed, 2 passed, all assertion-level (a run with 20 of 60 tasks failed returnsaction == "promote"; the gain is 0.333 where the 40 answered tasks give 0.5;eval_result.passedisTrue; malformed health is not validated). No TypeError/ImportError; the two controls (2 failures, no health data) pass on main.feat(trust-evidence): a run whose failures exceed the threshold canno…