Skip to content

feat(pilot): one decision surface for an evolution run (#937) - #1121

Open
HugoXK wants to merge 1 commit into
mainfrom
agent/937-pilot-core
Open

HugoXK wants to merge 1 commit into
mainfrom
agent/937-pilot-core

Conversation

@HugoXK

@HugoXK HugoXK commented Nov 16, 2026

Copy link
Copy Markdown
Collaborator

Part of #937 (step 2 of the implementation order: pilot.py core + fake-policy tests + dry-run).

What changed

  • New agentdescent/pilot.py: PilotConfig, PilotPolicy, PilotReport, pilot(), pilot_scores().
  • pilot() runs one evolve(), scores the seed and the result on the same disjoint test tasks, builds TrustEvidence, and asks promote() for one promote / defer / reject verdict.
  • pilot_scores() exposes the same decision/report seam for a run that already happened elsewhere.
  • Honesty rules are enforced where the composition happens: a test task that raised is run-health evidence (not a zero), an unchanged artifact reports noise_only, a missing contamination check defers, a contaminated dataset rejects before any model call, and a test/evolve id overlap is refused before any call.
  • to_dict() is the schema_version: 1 report; to_markdown() is the human card. The report contains scores, hashes, counts and cost only -- no prompts, responses or credentials.

No CLI, no apply, no engine change, no new top-level export. The CLI/docs slice stays in #937.

Why this shape

It composes existing modules only: evolve, build_trust_evidence, promote, build_manifest. The test split is explicit and disjoint; the engine's held-out split decides what commits and is never reused for the number the report quotes.

Known limits

  • A non-significant eval gate is reject in promote() today. The issue's "one seed with a CI straddling zero -> defer" distinction needs a PromotionRules option and is left for the next slice.
  • Multi-seed pooling repeats the same test tasks; the interval is a paired bootstrap over seed-task pairs, not cluster-robust. The report says so.
  • Token cost is real only when the model adapter shares the run's Usage; the report says so.
  • No docs page or CLI yet (step 3 of feat(pilot): one evidence-bound decision surface with run-health and artifact-change truthfulness #937) to avoid cli.py/mkdocs.yml conflicts with the open PRs.

Verification at the tip

Tip SHA: 46f46a7

Ran:

  • pytest tests/test_pilot.py -q -> 11 passed
  • pytest tests/test_pilot.py tests/test_trust_evidence.py tests/test_trust_evidence_artifact_truth.py tests/test_run_health_evidence.py tests/test_promotion.py tests/test_promotion_review_cost.py tests/test_manifest.py tests/test_facade.py tests/test_evidence_card.py -> 108 passed
  • pytest tests/test_module_audit.py tests/test_public_surface.py tests/test_layer_boundary.py tests/test_api_docs.py tests/test_api_reference.py tests/test_docs_examples.py tests/test_changelog_fragments.py -> 133 passed
  • python -m tools.changelog_fragments check -> OK
  • python -m tools.gen_api_docs --check -> OK

Not run:

  • full pytest -q -- still running locally at the tip; CI runs it on 3.9-3.13
  • mkdocs build --strict -- no docs/ or mkdocs.yml change
  • a live-model pilot -- the tests use an offline fake policy

The engine can evolve an artifact, but a user still has to wire the paired test score, trust evidence, promotion and manifest in the right order. Add agentdescent.pilot: pilot() runs one evolution, scores the seed and the result on the same disjoint test tasks, and returns one promote/defer/reject report with the evidence, cost, manifest and next action.

A test task that raised is run-health evidence rather than a zero, an unchanged artifact reports noise_only, a missing contamination check defers, a contaminated dataset rejects before any model call, and an overlapping test split is refused before spending. pilot_scores() exposes the same decision seam for a run that happened elsewhere.

No CLI, no apply, no engine change: the CLI/docs slice stays in #937.
@HugoXK HugoXK added enhancement New feature or request area:evolve The evolve loop: evolution, async_evolve, aggregator, population, policies, strategies agent-pr Pull request opened by an agent labels Nov 16, 2026
@HugoXK

HugoXK commented Nov 16, 2026

Copy link
Copy Markdown
Collaborator Author

Handoff to the FlyBot agentdescent team (@flybot) — development is complete from my side; Birfy reviews, then the deployed team owns maintenance.

State

  • Branch agent/937-pilot-core; tip 46f46a78dc46a52a94d7b3c2c64b1fa419993b18.
  • Risk tier T3 (832 lines / 3 files; over the T2 size cap). No self-PASS posted: T3 needs Birfy's PASS at this tip and Danyang's explicit click.
  • Green: build, test (3.9), test (3.10), test (3.12), test (3.13), otel-wire. test (3.11) failed only on Docker Hub toomanyrequests in tests/test_sandbox_container.py; the failed job is being re-run.

Landed here

  • agentdescent/pilot.py: PilotConfig, PilotPolicy, PilotReport, pilot(), pilot_scores().
  • Same disjoint test tasks before/after; TrustEvidence + promote(); honest run-health, noise_only, contamination and overlap handling; offline fake-policy tests and dry-run.

Still open (same issue #937)

Offline evidence at the tip

  • pytest tests/test_pilot.py → 11 passed.
  • pytest evidence/promotion/manifest/facade set → 108 passed.
  • module/public-surface/api/docs/changelog set → 133 passed.
  • tools.changelog_fragments check and tools.gen_api_docs --check → OK.
  • Full local pytest -q still running; CI is the gate.

@HugoXK

HugoXK commented Nov 16, 2026

Copy link
Copy Markdown
Collaborator Author

Status update at tip 46f46a78: all required checks are green — build, test (3.9), test (3.10), test (3.11) (re-run after the Docker Hub rate limit), test (3.12), test (3.13), plus otel-wire. The test (3.11) failure was infrastructure only: toomanyrequests pulling python:3.11-slim in tests/test_sandbox_container.py; the re-run passed.

Local full pytest -q finished with 4 failures, all in modules this PR does not touch and all reproduced without it: test_era_algotune (local numba SystemError), test_gskill, test_integrations (local CLI launcher traceback), test_porous_molecules (numerical bond-length assertion). CI on 3.9–3.13 is green, so these are local-environment results, not this PR's behavior. Listing them so the reviewer sees the whole picture.

Ready for Birfy's review; no self-PASS was posted. @flybot agentdescent team owns maintenance after the merge.

@github-actions github-actions Bot added the risk:T3 Risk tier T3 (core loop, release, process; needs Danyang's click), set by the controller label Nov 16, 2026

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

agent-pr Pull request opened by an agent area:evolve The evolve loop: evolution, async_evolve, aggregator, population, policies, strategies enhancement New feature or request risk:T3 Risk tier T3 (core loop, release, process; needs Danyang's click), set by the controller

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant