[Feat] Replay changed-image decision requests with retained evidence

[linear3735]

Required human contributor answers

1. Outcome: complete the online-image path, compare complete-request latency for new images and repeated questions on the same image, and provide reproducible changed-screenshot replay.
2. Highest-leverage variable: acceleration is the first priority; measure complete-request latency before choosing hotspots to optimize.
3. How this work moves it: follow the contributor's sequence: reuse Omni's shared vision backend, validate the three decision types and image/cache changes against the official model, measure both request patterns, then connect Agents replay for reproducible comparison.
4. Customer and need: System1 community inference and evaluation developers who need the online-image path and reproducible full-request comparisons.

Is a PR needed? Yes. The human contributor explicitly requested upstream submission and confirmed the customer on October 12, 2026. These summarize their recorded instructions; private conversation material is not reproduced.

Observed failure evidence: the pre-fix collector at f51b82d accepted probabilities [0.2, 0.2] as ok. Running the new failure-retention regression against that parser produced 1 failed, 46 passed: expected parse_error, observed ok. With the normalization check, the same suite passes all 47 cases; fork PR CI passes all 129 benchmark cases without skips.

Why

The benchmark in #52 does not replay changing screenshot sequences. This adds an ordered collector for official JEV's /v1/decide and native JEV-VL's /v1/systemone, preserving requests and failures for subsequent comparison.

Depends on #52. This Draft targets main because #52's branch exists only in the contributor's fork. The current main-to-head diff therefore includes #52. The incremental change after #52 (5c59dc4) is 4 files, +605/-4, of which 533 changed lines are code/tests/CI. The displayed main-to-head diff is 12 files, +1861/-0, including 1613 code/test/CI lines. Rebase after #52 merges; do not merge this Draft independently of that dependency review.

How

Start with benchmarks/legs/changed_image_replay.py, then its tests and benchmarks/README.md. The collector reuses DecisionClient, validates the full manifest before sending anything, preserves repeated images and option order, and stores raw responses, hashes, timings and completion counts. Invalid responses are retained and counted as failures while the sequence continues.

What

• Add python -m benchmarks.legs.changed_image_replay and document the two flat JEV-VL contracts separately from the typed decision APIs.
• Require normalized finite response probabilities, allowing 1e-6 rounding error.
• Run all tests/benchmarks in the existing CI benchmark job with its fail-on-skip check.

The incremental change does not modify the Agents runtime, policy selection, or application actions. --cache-state records the caller's declaration; it does not reset server caches. ok checks response validity, not reference agreement, cache correctness, or task success. Image identifiers hash the URI, not decoded pixels. No speedup is claimed by this collector PR.

Verification

Head: f09748b967792d52014abb3b0c3b1acbd0ba069c; dependency: #52 at 5c59dc4229c47d2deea31bd3fdb70ce844eaa3d5.

Local Ruff format/check, ty, full pytest and scripts/smoke.sh passed using the existing contributor environment. The local full suite reported 1100 passed, 198 subtests passed, 43 optional/environment skips; benchmarks reported 129 passed, zero skipped. Fresh Kimi and Codex reviews of the normalization/CI follow-up found no blocking defects.

Fork PR CI checked out merge commit 515f46bcf407665117ba303e44e038302d51d02b (head f09748b with dependency 5c59dc4). Its tree is identical to f09748b; it does not validate merging with the latest upstream main. Benchmarks, Linux core 3.11/3.13 and full succeeded. Browser and Windows jobs were skipped by workflow conditions. The full CI job reported 1035 passed, 107 skipped and 199 subtests passed; these environment-dependent skips are separate from the zero-skip benchmark job.

uv run ruff format --check .
uv run ruff check .
uv run ty check
uv run pytest -q
scripts/smoke.sh

The benchmark job pins the public231 reference and runs tests/benchmarks, followed by the XML skip guard. Replay setup and commands specify the manifest, provenance and fresh output directory.

• Contributor has reviewed the final diff and verification results.
• Required application demo is supplied or a maintainer has accepted an exception.

Demo / evidence

Draft blocker: the required application → Agents decision-agent → Omni inference → independently checked application result recording is not yet available. CPU tests use HTTP fixtures; the collector does not itself execute application actions. A same-run demo and trace must be provided, or the documented exception accepted by a maintainer, before marking ready. This PR does not request an exemption based on a historical ticket replay.

The related online JEV-VL implementation is currently in Omni fork #7, built on Omni #117. Historical H800 screenshot replays used an earlier collector and Omni tree; they are not claimed as validation of this head or as an application demo. Fresh GPU inference, full-request latency and application task quality remain unverified for this submission. Existing private run archives are not presented as public evidence links.