Repository navigation
[Eval] Add repeatable browser and desktop task suites #13
Description
Why
Browser and desktop improvements need a fixed set of tasks with explicit success checks and repeatable starting conditions. The existing evaluation runner, job records, and replay tools provide a starting point.
Scope
Create a small suite covering browser forms, date selection and search, plus desktop document editing and file dialogs. Define the required operations for each task and record unsupported operations explicitly. Establish the baseline on tasks the current agent can execute.
Reuse the existing evaluation runner and artifact formats. Record completion rate, total time, decision latency, model cost, and failure reason. Record hardware and model configuration for local inference.
Acceptance criteria
- Every task defines its initial conditions, reset procedure, required operations, and an independent success check.
- Repeated runs use the same task definitions and budgets across models. All attempts count toward completion rate, including failures, timeouts, and unsupported tasks.
- Results distinguish model inference, environment execution, and total task time. Local compute cost assumptions are stated.
- Saved records and replays are sufficient to inspect failures and reproduce the result table.
- The documented command runs the initial suite and produces a baseline report. Desktop results identify the operating system.
Starting points
evals/README.md, s1a/agents/desktop.py, and docs/browser-front.md.
This suite supplies the baseline for Cua-S1 4B 0.2 evaluation, desktop operation extensions, and bounded recovery.
Activity
Birfy commentedon Oct 9, 2026
Triage against main
320bb086: the existing table is a useful starting point, but its score/time/cost columns cannot yet serve as this issue's application-completion report.evals.table.rowsremoves errored trials before calculating time, calls and cost, as well as score. The current regression deliberately expects one scored trial plus two errors to showN=1, score=1.0; an all-error group loses its resource totals. This is existing game-score behavior, so application reporting should add explicit fields rather than silently redefine those metrics.Suggested report acceptance checks:
result.jsonmust not silently shrink the denominator.Overlap: #31 already introduces shared recovery records and paired summaries with independent verification and failure-inclusive denominators. Reuse that contract where it fits, while leaving its recovery-specific comparisons in #16. #42 supplies a final-page artifact; #52 measures decision-service quality. Neither replaces this suite's independent application verdict.
Review/acceptance clarification only; no implementation is claimed here.