Skip to content

feat(trust-evidence): a run whose failures exceed the threshold cannot promote #993

Closed

Description

@HugoXK

Sub-issue of #937, the run-health slice. #991 lands only the artifact-change slice of #937 (Partial: artifact-change truthfulness for #937), so the rest of #937 is split out one behaviour per issue; this is the second one.

Problem

Provider and harness failures do not reach the decision. When a provider crashes, an eval raises, or a worker is retired, the task today either raises out of the run or lands as a scored 0. Either way the failure rate is invisible at the point where promote() reads the evidence, so a run in which a third of the tasks never produced a real answer can still hand a clean mean gain to the promoter. The 2026-11-06 benchmark session recorded exactly this, twice: exceptions folded into the mean, and a content-filter crash that produced a final answer no agent generated.

Scope (one behaviour)

Run health becomes evidence that can fail the gate:

  • the evidence bundle records the health of the run: how many tasks failed and out of how many, and whether that is above a threshold the caller sets;
  • a run above the threshold fails its gate, so promote() cannot return promote;
  • failed tasks are not scored as 0 and are not folded into the mean — the reported gain is over the tasks that actually produced an answer, and the failure count is reported next to it;
  • callers that pass no health data keep today's evidence exactly: no new keys, existing hashes and decisions unchanged.

Out of scope here: agentdescent/pilot.py, the pilot CLI, the disjoint-test-split rule (#843), censoring policy, and any change to how evolve counts an exception. Those stay on #937 or get their own issue.

Acceptance criteria

A new test file, offline and model-free, that fails on live main and passes on the tip:

  1. Same scores with a failure rate above the threshold: promote(build_trust_evidence(...).to_promotion_request(...)) has action != "promote", and the reason carries the health shortfall rather than a bare score.
  2. Control: the same scores with failures at or below the threshold still promote.
  3. Failures are not zeroes: with k failed tasks, the reported gain is the mean over the tasks that produced a result, and the failure count is on the evidence.
  4. Default is unchanged: with no health data passed, the bundle is byte-identical to main's today (same keys, same hash), asserted not described.
  5. pytest -q stays offline and green; no model, no container.

Verification the PR must carry

  • red at the base it was cut from, with the new test file as the only change on the tree, quoting the failure lines;
  • the tip's focused suite and the changelog fragment check;
  • size and risk tier computed from paths.

Related

Written-by: AgentDescent Lead
Bot: AgentDescent Dev-1

Activity

  1. added
    enhancementNew feature or request
    area:evolveThe evolve loop: evolution, async_evolve, aggregator, population, policies, strategies
    on Nov 12, 2026
  2. added
    agent-workingAn agent has claimed this issue (the claim comment names the branch)
    on Nov 12, 2026
  3. HugoXK commented on Nov 12, 2026

    @HugoXK
    CollaboratorAuthor

    Claimed: branch agent/993-run-health-threshold, cut from live main 271c9346841ef044ed64565b4ca15594bb161cea; draft PR follows with Fixes #993.

    Red first: tests/test_run_health_evidence.py alone at 271c9346 -> 4 failed, 2 passed, all assertion-level (a run with 20 of 60 tasks failed returns action == "promote"; the gain is 0.333 where the 40 answered tasks give 0.5; eval_result.passed is True; malformed health is not validated). No TypeError/ImportError; the two controls (2 failures, no health data) pass on main.

  4. removed
    agent-workingAn agent has claimed this issue (the claim comment names the branch)
    on Nov 13, 2026
  5. added a commit that references this issue on Nov 16, 2026
    ad30f3a
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

area:evolveThe evolve loop: evolution, async_evolve, aggregator, population, policies, strategiesenhancementNew feature or request

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions