Repository navigation
Open
1 / 11 of 1 issue completedOpen
feat(pilot): one evidence-bound decision surface with run-health and artifact-change truthfulness#937
1 / 11 of 1 issue completed
Copy link
Labels
agent-workingAn agent has claimed this issue (the claim comment names the branch)An agent has claimed this issue (the claim comment names the branch)area:cliThe agentdescent command line (cli.py, doctor, plan/evolve/status/apply)The agentdescent command line (cli.py, doctor, plan/evolve/status/apply)area:evolveThe evolve loop: evolution, async_evolve, aggregator, population, policies, strategiesThe evolve loop: evolution, async_evolve, aggregator, population, policies, strategiesenhancementNew feature or requestNew feature or request
Description
Problem
A run can move a metric and still leave the user unable to answer: did it improve, is the evidence strong enough, should I ship it, what did it cost, what changed? The pieces exist (evolve, trust_evidence, promotion, manifest, runstore) but a caller must wire five to eight of them in the right order. reports/agentdescent-pilot-next-pr-2026-11-10.md proposes pilot as the composition surface.
This issue is not a new optimizer. It is the last mile: one command returns one decision (promote | defer | reject | rollback), the evidence, the cost, the artifact diff and the next action.
It also closes three honesty gaps found in the 2026-11-06 benchmark session:
- a run with
accepted=0and an unchanged artifact can still report a non-zeroquality_gain; - provider/eval exceptions are counted as 0 and disappear into the mean;
- a provider content-filter crash can yield a final answer no agent produced.
Scope
agentdescent/pilot.py:PilotConfig,PilotReport(schema_version: 1),pilot(policy, config),pilot_spec(...).- CLI:
agentdescent pilot --spec ... --test-data ... --dry-run; no--applyin v1. - Compose existing modules only:
evolve/async_evolve,trust_evidence.build_trust_evidence,promotion.promote,manifest.build_manifest,RunStore/DecisionLog(optional). - Run health is first-class evidence: provider failures, harness/sandbox failures, missing outputs, retired workers, eval exceptions, censored runs,
artifact_changed. A run above a configurable failure threshold cannot promote. - Artifact-change truthfulness: when the artifact is byte-identical before and after,
gain=0andnoise_only=true; never reprice evaluation noise as learning. - Disjoint test split: before/after maps come from the same disjoint test tasks, not the engine's held-out split (see evolve: same held-out used for select and reported score (optimistic generalisation) #843).
- Honesty rules: one seed with a CI straddling zero ->
defer; missing contamination check ->defer; contaminated ->reject; no prompts/responses/credentials in the report.
Acceptance criteria
- Offline fake policy: strong gain ->
promote; flat/negative ->reject/defer; contaminated ->reject; missing contamination ->defer. - A run with provider/harness failures above threshold, or an unchanged artifact, never returns
promote. --dry-runvalidates the spec/test split/budget without a model call.- Same spec/data/model/seeds -> same
manifest_idand decision. pytest -qstays offline and green.
Source papers
- Dependable Long-Horizon LLM Agents: A Trajectory-Centered Empirical Study of Where Errors Enter and Which Controls Reach Them -- arXiv:2611.13254 (2026-11-09). Errors enter early; evidence often exists unused; provider crashes can produce phantom final answers. Motivation for run-health/censoring.
- Replay Fidelity in LLM Agent Evaluation: Randomness, Hidden State, and Test Oracles in AppWorld -- arXiv:2611.11216 (2026-11-08). A run is only replayable when RNG and hidden state are controlled.
- Agent Benchmarks Stop Ranking Before They Run Out of Tasks -- arXiv:2611.13541 (2026-11-10). Pass/fail saturates; report graded credit and cost on separate axes.
Related
- evolve: same held-out used for select and reported score (optimistic generalisation) #843 (
evolve: same held-out used for select and reported score) - feat(api): add a stable embedding facade and API version #920 (stable embedding facade)
Non-goals
No --apply, no auto-merge, no new algorithm, no dashboard, no multi-tenant control plane.
Metadata
Metadata
Assignees
Labels
agent-workingAn agent has claimed this issue (the claim comment names the branch)An agent has claimed this issue (the claim comment names the branch)area:cliThe agentdescent command line (cli.py, doctor, plan/evolve/status/apply)The agentdescent command line (cli.py, doctor, plan/evolve/status/apply)area:evolveThe evolve loop: evolution, async_evolve, aggregator, population, policies, strategiesThe evolve loop: evolution, async_evolve, aggregator, population, policies, strategiesenhancementNew feature or requestNew feature or request
Activity
HugoXK commentedon Nov 12, 2026
Implementation order for this issue (keeps PRs small and avoids the cli.py/mkdocs.yml rebase churn):
TrustEvidencerun health +artifact_changed(no CLI change). First test must fail on main: an unchanged artifact reportsgain=0/noise_only=true; a run above the failure threshold cannot promote.pilot.pycore + fake-policy tests + dry-run (the only new module).pilot+ docs + changelog (one surface PR).WIP limit: at most two open PRs; no two PRs touch
cli.pyormkdocs.ymlat the same time. Supporting issues: #939 for the replay verdict, #940 for the verifier evidence block, #941 for the long-horizon proof asset. Do not add a dashboard, auth, billing, multi-tenancy or a Postgres adapter here — those belong to the control plane.HugoXK commentedon Nov 12, 2026
Backlog from the 2026-10-13 -> 11-12 frontier survey (do not start until the #937/#938/#941 batch lands):
WIP stays at two PRs; no two PRs touch cli.py/mkdocs.yml. #942/#943/#944 are the next trust layer; the rest are backlog.
feat(promotion): report and budget the review cost of a diff (#960)
feat(ledger): record a policy digest for every decision and round (#975)
HugoXK commentedon Nov 12, 2026
Claimed: branch
agent/937-artifact-truthfulness, cut from live main98c6b805c90b6436234e305953daba2077612312, tipa4b4d19c4ec8b664aad6ace47e443bb026605950; draft PR #991.Slice, one behaviour: an artifact that is byte-identical before and after cannot be repriced as learning —
build_trust_evidence(..., artifact_changed=False)reportsgain == 0andnoise_only is True, and that bundle cannot returnpromote(the zeroed delta fails the eval gate). Acceptance predicate:promote(build_trust_evidence(before, after, artifact_changed=False).to_promotion_request("x"))never hasaction == "promote", while the same scores withartifact_changed=Truedo promote.Red at
98c6b805, the new test file alone against unmodified main: 5 failed, 4 of themTypeError: build_trust_evidence() got an unexpected keyword argument 'artifact_changed'(test idstest_an_unchanged_artifact_reports_zero_gain_and_noise_only,test_an_unchanged_artifact_cannot_promote,test_a_changed_or_unknown_artifact_keeps_the_measured_gain,test_the_artifact_comparison_travels_with_the_evidence) and 1AttributeError: 'TrustEvidence' object has no attribute 'artifact_changed'(test_the_default_bundle_is_the_one_that_existed_before). All TypeError/AttributeError-shaped, none assertion-level; the file collects because it imports only names that exist on main.Correction to my first version of this comment: it named a local commit
ce583455as the red-only commit. That sha was never pushed (the direct push to github.com timed out and the branch was created through the git data API), so the two-commit red/fix split did not survive; the branch tipa4b4d19ccarries the test and the fix together. The red is reconstructible only by copying the test file onto98c6b805, which is what the PR body documents.Rest of #937 stays unbuilt: run-health evidence,
pilot.py, and thepilotCLI are separate slices.HugoXK commentedon Nov 12, 2026
Acceptance criterion for the wiring slice, found in review of #991 (
a4b4d19c):When
TrustEvidence.noise_onlyis true,to_dict()exportsdelta: 0.0beside the measuredscore,baseline,ciandper_task. A consumer that readsscore - baselinetherefore sees the moved number whiledeltasays zero. The slice that wiresartifact_changedinto a product path must make one of these true and test it: either the exported payload is internally consistent (measured values under names that cannot be read as gain), ordeltastays the measured number andnoise_onlyis the only thing that stops it being reported. Until then the field has no in-repo producer, so no bundle in main is affected.Not a defect in #991: its stated scope is the bundle, and the flag is inert there.
fix(trust-evidence): an unchanged artifact cannot report gain or prom…
feat(trust-evidence): a run whose failures exceed the threshold canno…
fix(cli): refuse an unwritable skill-library seed in one line (#1004)
feat(cost): price a diff and gate acceptance on quality per cost (#945)…
HugoXK commentedon Nov 16, 2026
Next slice (step 2):
agentdescent/pilot.pycore. Branchagent/937-pilot-core, cut from live mainf5cc52e; tip46f46a78dc46a52a94d7b3c2c64b1fa419993b18; PR #1121.Scope:
PilotConfig/PilotPolicy/PilotReport,pilot()+pilot_scores(), fake-policy tests and dry-run. It runs oneevolve(), scores the seed and the result on the same disjoint test tasks, buildsTrustEvidence, and askspromote()for one decision. No CLI, noapply, no engine change; step 3 (CLI + docs) stays open in this issue.Known limit left for the next slice: a non-significant eval gate is
rejecttoday; the issue's single-seeddeferdistinction needs aPromotionRulesoption.HugoXK commentedon Nov 16, 2026
Handoff: the
pilot.pycore slice is up as PR #1121 (branchagent/937-pilot-core, tip46f46a78). Birfy reviews; the deployed FlyBot agentdescent team takes maintenance after that. Step 3 of this issue (CLIpilot+ docs/nav + changelog surface) is still open and is the next slice. Additive follow-ups filed separately: #1111, #1112, #1113.