[Eval] Reconcile planned attempts with results

[Birfy]

Current status (2026-10-10)

hsliuustc0106 marked this PR Ready for review at 11:34:24 UTC on 2026-10-10. The earlier Draft wording below describes the prior state. Exact-head CI for cafcb947dc6e22df2b70e284b08ea2a7ac93c616 is still absent. This clarification changes no source or PR readiness state.

Why

Related to #13. The saved-results table cannot see a planned attempt that has no result.json, so an interrupted or absent result can silently disappear from the denominator. This is one bounded reconciliation slice; it does not complete the task-suite acceptance criteria.

How

Adds a version-1 attempt manifest and reconciler keyed by the existing (eval folder, model label, task_name) artifact fields. It reports recorded/error, missing, interrupted, unsupported, contradictory unsupported results, ambiguous duplicate matches, and unplanned artifacts. When multiple results share a plan key, the attempt is now marked ambiguous, all candidate paths are listed deterministically, and no artifact is assigned based on ordering. The existing score table remains unchanged; --attempt-plan appends a separate coverage report. Plan keys must be unique because current result artifacts do not carry a planned-attempt ID.

What

• evals/reconcile.py: manifest validation, artifact matching, and Markdown coverage report.
• evals/table.py: optional --attempt-plan output after the existing table.
• evals/README.md: schema, command, outcome meanings, and identity limitation.
• tests/test_evals_reconcile.py: missing/terminal/error/extra/mismatch cases, score-semantics regression, and an order-invariant duplicate-artifact case.

This branch is stacked on PR #59 (codex/eval-attempt-reporting) because the reconciler depends on its result-table artifact conventions. Retarget after #59 lands if maintaining the series in the same order.

Verification

On head cafcb947dc6e22df2b70e284b08ea2a7ac93c616: ruff format --check on the changed Python files, ruff check on those files, ty check on those files, and focused tests (tests/test_evals_reconcile.py tests/test_evals_table.py) passed; 12 tests passed, with 3 dependency deprecation warnings. git diff --check passed.

The full suite on the immediately preceding head 284e3d727562423cba44be1076afd97c786ef7d8 passed: 792 passed, 47 skipped, 153 subtests passed, with 3 dependency deprecation warnings. The new collision path is covered by the focused test on this head.

scripts/smoke.sh passed on the preceding head (smoke: ok); it reported the Jev key absent, so no model call was made.

Repository Actions are currently disabled according to read-only GET /repos/ThinkFlowLab/system1-agents/actions/permissions (HTTP 200, enabled: false). There are no workflow runs/status checks on the current head. This PR remains Draft pending the repository's normal Actions process and PR #59 dependency. No repository settings or credentials were changed.

Demo / evidence

N/A: this change adds offline result reconciliation/reporting and does not change an agent task, decision model, serving path, or inference behavior. Deterministic JSON artifact fixtures and tests are the evidence for its observable output; no live model or paid API was used.

Cloud validation (2026-10-10)

Independent run on exact head cafcb947dc6e22df2b70e284b08ea2a7ac93c616 on the assistant's cloud computer (Linux, CPython 3.12.14), using the frozen core + dev install:

• Full default suite (pytest -q -rs): 793 passed, 47 skipped, 3 warnings, 153 subtests passed.
• Repository-wide ruff format --check ., ruff check ., ty check, uv lock --check, and git diff --check: passed.
• scripts/smoke.sh: passed (smoke: ok), with no Jev key and no live model call.
• uv build: passed for source distribution and wheel.

The 47 skips include optional game/report extras and intentionally unsupported backend contract cases; this is the full default suite on the core install, not all optional configurations. Writable cloud caches and socksio==1.0.0 were used only in the isolated test environment so HTTPX could honor the existing cloud SOCKS proxy. No source change was needed.

This closes the previous full-suite/smoke verification gap for the collision-fix head. The historical preceding-head and focused-test results above are retained. Browser integration was not run on this PR; the separate cloud attempt for PR #63 was blocked before page load by Chromium's local-socket runtime restriction, including after a supported elevated-execution retry. That is not a browser pass for either PR, and no security settings were weakened.

These results do not establish GitHub Actions success or a live-model baseline. No Actions settings, credentials, PR readiness/state, merges or releases were changed.