Skip to content

feat(pilot): one evidence-bound decision surface with run-health and artifact-change truthfulness #937

Open
1 of 1 issue completed

Description

@HugoXK

Problem

A run can move a metric and still leave the user unable to answer: did it improve, is the evidence strong enough, should I ship it, what did it cost, what changed? The pieces exist (evolve, trust_evidence, promotion, manifest, runstore) but a caller must wire five to eight of them in the right order. reports/agentdescent-pilot-next-pr-2026-11-10.md proposes pilot as the composition surface.

This issue is not a new optimizer. It is the last mile: one command returns one decision (promote | defer | reject | rollback), the evidence, the cost, the artifact diff and the next action.

It also closes three honesty gaps found in the 2026-11-06 benchmark session:

  • a run with accepted=0 and an unchanged artifact can still report a non-zero quality_gain;
  • provider/eval exceptions are counted as 0 and disappear into the mean;
  • a provider content-filter crash can yield a final answer no agent produced.

Scope

  • agentdescent/pilot.py: PilotConfig, PilotReport (schema_version: 1), pilot(policy, config), pilot_spec(...).
  • CLI: agentdescent pilot --spec ... --test-data ... --dry-run; no --apply in v1.
  • Compose existing modules only: evolve/async_evolve, trust_evidence.build_trust_evidence, promotion.promote, manifest.build_manifest, RunStore/DecisionLog (optional).
  • Run health is first-class evidence: provider failures, harness/sandbox failures, missing outputs, retired workers, eval exceptions, censored runs, artifact_changed. A run above a configurable failure threshold cannot promote.
  • Artifact-change truthfulness: when the artifact is byte-identical before and after, gain=0 and noise_only=true; never reprice evaluation noise as learning.
  • Disjoint test split: before/after maps come from the same disjoint test tasks, not the engine's held-out split (see evolve: same held-out used for select and reported score (optimistic generalisation) #843).
  • Honesty rules: one seed with a CI straddling zero -> defer; missing contamination check -> defer; contaminated -> reject; no prompts/responses/credentials in the report.

Acceptance criteria

  • Offline fake policy: strong gain -> promote; flat/negative -> reject/defer; contaminated -> reject; missing contamination -> defer.
  • A run with provider/harness failures above threshold, or an unchanged artifact, never returns promote.
  • --dry-run validates the spec/test split/budget without a model call.
  • Same spec/data/model/seeds -> same manifest_id and decision.
  • pytest -q stays offline and green.

Source papers

  • Dependable Long-Horizon LLM Agents: A Trajectory-Centered Empirical Study of Where Errors Enter and Which Controls Reach Them -- arXiv:2611.13254 (2026-11-09). Errors enter early; evidence often exists unused; provider crashes can produce phantom final answers. Motivation for run-health/censoring.
  • Replay Fidelity in LLM Agent Evaluation: Randomness, Hidden State, and Test Oracles in AppWorld -- arXiv:2611.11216 (2026-11-08). A run is only replayable when RNG and hidden state are controlled.
  • Agent Benchmarks Stop Ranking Before They Run Out of Tasks -- arXiv:2611.13541 (2026-11-10). Pass/fail saturates; report graded credit and cost on separate axes.

Related

Non-goals

No --apply, no auto-merge, no new algorithm, no dashboard, no multi-tenant control plane.

Activity

  1. added
    enhancementNew feature or request
    area:evolveThe evolve loop: evolution, async_evolve, aggregator, population, policies, strategies
    area:cliThe agentdescent command line (cli.py, doctor, plan/evolve/status/apply)
    on Nov 12, 2026
  2. HugoXK commented on Nov 12, 2026

    @HugoXK
    CollaboratorAuthor

    Implementation order for this issue (keeps PRs small and avoids the cli.py/mkdocs.yml rebase churn):

    1. TrustEvidence run health + artifact_changed (no CLI change). First test must fail on main: an unchanged artifact reports gain=0/noise_only=true; a run above the failure threshold cannot promote.
    2. pilot.py core + fake-policy tests + dry-run (the only new module).
    3. CLI pilot + docs + changelog (one surface PR).

    WIP limit: at most two open PRs; no two PRs touch cli.py or mkdocs.yml at the same time. Supporting issues: #939 for the replay verdict, #940 for the verifier evidence block, #941 for the long-horizon proof asset. Do not add a dashboard, auth, billing, multi-tenancy or a Postgres adapter here — those belong to the control plane.

  3. HugoXK commented on Nov 12, 2026

    @HugoXK
    CollaboratorAuthor
  4. added
    agent-workingAn agent has claimed this issue (the claim comment names the branch)
    on Nov 12, 2026
  5. HugoXK commented on Nov 12, 2026

    @HugoXK
    CollaboratorAuthor

    Claimed: branch agent/937-artifact-truthfulness, cut from live main 98c6b805c90b6436234e305953daba2077612312, tip a4b4d19c4ec8b664aad6ace47e443bb026605950; draft PR #991.

    Slice, one behaviour: an artifact that is byte-identical before and after cannot be repriced as learning — build_trust_evidence(..., artifact_changed=False) reports gain == 0 and noise_only is True, and that bundle cannot return promote (the zeroed delta fails the eval gate). Acceptance predicate: promote(build_trust_evidence(before, after, artifact_changed=False).to_promotion_request("x")) never has action == "promote", while the same scores with artifact_changed=True do promote.

    Red at 98c6b805, the new test file alone against unmodified main: 5 failed, 4 of them TypeError: build_trust_evidence() got an unexpected keyword argument 'artifact_changed' (test ids test_an_unchanged_artifact_reports_zero_gain_and_noise_only, test_an_unchanged_artifact_cannot_promote, test_a_changed_or_unknown_artifact_keeps_the_measured_gain, test_the_artifact_comparison_travels_with_the_evidence) and 1 AttributeError: 'TrustEvidence' object has no attribute 'artifact_changed' (test_the_default_bundle_is_the_one_that_existed_before). All TypeError/AttributeError-shaped, none assertion-level; the file collects because it imports only names that exist on main.

    Correction to my first version of this comment: it named a local commit ce583455 as the red-only commit. That sha was never pushed (the direct push to github.com timed out and the branch was created through the git data API), so the two-commit red/fix split did not survive; the branch tip a4b4d19c carries the test and the fix together. The red is reconstructible only by copying the test file onto 98c6b805, which is what the PR body documents.

    Rest of #937 stays unbuilt: run-health evidence, pilot.py, and the pilot CLI are separate slices.

  6. HugoXK commented on Nov 12, 2026

    @HugoXK
    CollaboratorAuthor

    Acceptance criterion for the wiring slice, found in review of #991 (a4b4d19c):

    When TrustEvidence.noise_only is true, to_dict() exports delta: 0.0 beside the measured score, baseline, ci and per_task. A consumer that reads score - baseline therefore sees the moved number while delta says zero. The slice that wires artifact_changed into a product path must make one of these true and test it: either the exported payload is internally consistent (measured values under names that cannot be read as gain), or delta stays the measured number and noise_only is the only thing that stops it being reported. Until then the field has no in-repo producer, so no bundle in main is affected.

    Not a defect in #991: its stated scope is the bundle, and the flag is inert there.

  7. removed
    agent-workingAn agent has claimed this issue (the claim comment names the branch)
    on Nov 12, 2026
  8. HugoXK commented on Nov 16, 2026

    @HugoXK
    CollaboratorAuthor

    Next slice (step 2): agentdescent/pilot.py core. Branch agent/937-pilot-core, cut from live main f5cc52e; tip 46f46a78dc46a52a94d7b3c2c64b1fa419993b18; PR #1121.

    Scope: PilotConfig/PilotPolicy/PilotReport, pilot() + pilot_scores(), fake-policy tests and dry-run. It runs one evolve(), scores the seed and the result on the same disjoint test tasks, builds TrustEvidence, and asks promote() for one decision. No CLI, no apply, no engine change; step 3 (CLI + docs) stays open in this issue.

    Known limit left for the next slice: a non-significant eval gate is reject today; the issue's single-seed defer distinction needs a PromotionRules option.

  9. added
    agent-workingAn agent has claimed this issue (the claim comment names the branch)
    on Nov 16, 2026
  10. HugoXK commented on Nov 16, 2026

    @HugoXK
    CollaboratorAuthor

    Handoff: the pilot.py core slice is up as PR #1121 (branch agent/937-pilot-core, tip 46f46a78). Birfy reviews; the deployed FlyBot agentdescent team takes maintenance after that. Step 3 of this issue (CLI pilot + docs/nav + changelog surface) is still open and is the next slice. Additive follow-ups filed separately: #1111, #1112, #1113.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

agent-workingAn agent has claimed this issue (the claim comment names the branch)area:cliThe agentdescent command line (cli.py, doctor, plan/evolve/status/apply)area:evolveThe evolve loop: evolution, async_evolve, aggregator, population, policies, strategiesenhancementNew feature or request

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions