Repository navigation
[Feat] Make the decision deadline configurable for slower local System One servers - #36
Conversation
JevModel cut every decision at 5 s, a deadline sized for Jev (0.3 s). A local System One server behind TYPESAFE_API_URL can be slower: on Google Flights, OneJev-27B on an A100 takes about 3.7 s a decision and more on the calendar page, so every run stopped at the eighth step. With 30 s it completed the task with the same 11 actions as Jev. S1A_DECISION_TIMEOUT_S sets the deadline; unset, it stays 5 s, so Jev is unchanged. A value that is not a positive number is a config error. An explicit timeout_s passed to JevModel.from_env still wins. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
| value = float(raw) | ||
| except ValueError: | ||
| value = 0.0 | ||
| if value <= 0: |
There was a problem hiding this comment.
float() accepts nan, inf, and overflow such as 1e9999; neither nan nor positive infinity satisfies value <= 0, so they pass this guard. nan then makes the computed deadline and remaining timeout nan, while inf removes the intended bound. Please also reject not math.isfinite(value) and extend the config-error regression with nan, inf, and 1e9999.
float() reads nan, inf and 1e9999; nan passed the `<= 0` guard and made the deadline nan, inf removed the bound. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
|
Non-blocking:
FYI, out of scope for this PR: an explicit |
|
@chaimaerachdi could you add a Demo / evidence section using the self-review guidance now merged in #39? The measurements in the description are helpful; a short recording, screenshots with logs, or a reproducible terminal trace would make the deadline change easier to verify. On the same task and local System One server, please show:
Please include the invocation/config, model and server versions, hardware, exact baseline/head commits, actual elapsed times and outcomes, and links to the raw traces. For the reported Flights comparison, retain the three interrupted runs in the run record alongside the seven completed runs. Label any injected-delay fixture separately from real-server evidence. Existing artifacts are fine; no new A100 run or video is required. If a real-server run is unavailable, note the gap. Please redact secrets/private data and say whether any attached media may be reused in public project updates, with any attribution requirements. |
…_TIMEOUT_S Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
|
Thanks for the review. Updated docs/browser-front.md in de5f5d2 to mention S1A_DECISION_TIMEOUT_S. The explicit timeout_s validation can go in a separate PR, as suggested. |
|
@chaimaerachdi Please try this deadline change with system1-omni's supported Open-Jev-27B-v1.1 worker, and attach a short video of the Flights agent. The reported local inference uses The video should show the same task with the default and configured deadlines, the agent's actions and the actual final outcome. Link the raw traces and exact agents/engine commits, command and hardware; retain failures and interrupted runs in the evidence. |
|
@chaimaerachdi We're prioritizing the new backends and agent improvements for v0.2.0. This deadline change supports the local/served backend work. Please resolve the current merge conflicts, rerun the required checks, and respond to the outstanding System1-Omni Open-Jev/default-versus-configured-deadline evidence request. If a real-server recording is unavailable, state the gap and the precise blocker; keep injected-delay fixtures distinct from real-server evidence and retain the interrupted runs. Please reply with the remaining work and expected completion time. |
# Conflicts: # CHANGELOG.md
|
Thanks for merging. The real-server evidence you asked for, through system1-omni, is in the follow-up PR (commit 34cc19b, docs/results/decision-timeout/2026-10-07-a40/). Video (attached): the same Flights task, three runs one after the other, only the deadline changing, real time, no cuts. Path: s1a main at 22e685b (this merge) -> system1-omni's Rust frontend omni-jev (99865743) -> Open-Jev's reference server (3308a15) with Open-Jev-9B, one NVIDIA A40 46 GB. No injected delay; the unit tests stay the separate injected-value evidence.
Precise blocker for Open-Jev-27B-v1.1 through system1-omni's native worker: I built it for sm_86 and exported the checkpoint on this machine (37 s, peak 59 GiB RAM, against the 110 GB in the recipe), but the 48 GiB export does not fit the A40: the worker stops at load with "cudaMalloc: out of memory" (log in the same folder). It needs a single GPU above 48 GB (A100 80 GB, H100, H200). Open-Jev-9B is a different checkpoint from both Open-Jev-27B-v1.1 and the OneJev-27B / qev serve runs of the description. The OneJev-27B records of the description, including the three interrupted runs, are not in the repo yet; I'll add them to the follow-up PR. The video may be reused in project updates with a link to that folder. PR36_deadline_system1-omni_A40_2026-10-07.mp4 |
|
Update: the same three runs through system1-omni's own native Open-Jev worker (4a79980, Open-Jev-9B) on the A40, video attached, records in docs/results/decision-timeout/2026-10-08-a40-native/ (commit 389f346).
PR36_deadline_system1-omni_native_A40_2026-10-08.mp4 |
[Fix] Reject a non-finite timeout_s passed in code; real-server deadline evidence for #36
|
As promised: every OneJev-27B run behind this PR's table is now in #56, the three interrupted 30 s runs included (10 started, 7 completed and counted, 3 interrupted), each with its log, profile and final screenshot. |
…the three interrupted ones included Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
[Docs] Every OneJev-27B run behind #36's deadline table, the three interrupted ones included
Why
JevModelcuts every decision at 5 s (DECISIONS_TIMEOUT_S), a deadline sized for Jev, which answers in about0.3 s. A local System One server reached through
TYPESAFE_API_URLcan be slower. On Google Flights, OneJev-27B(OmniJev's new 27B model, which speaks the same API) on an A100 takes about 3.7 s a decision and more on the calendar
page, which offers 66 more elements. Every run stopped at the eighth step, right after the calendar opened.
How
s1a/decision_models/wire.py:decisions_timeout_from_env()readsS1A_DECISION_TIMEOUT_S; unset, it returnsDECISIONS_TIMEOUT_S(5 s). A value that is not a finite, positive number (nan,infand1e9999included) isMODEL_SERVICE_CONFIG_ERROR, naming the variable.s1a/decision_models/jev.py:JevModel.from_env(timeout_s=None)uses it; an explicittimeout_sstill wins.What
S1A_DECISION_TIMEOUT_S(docs/configuration.md,.env.example,CHANGELOG.md).Measured (Google Flights, Zurich to London, one way; A100 80 GB; headless Chromium)
Success is judged by a vision model (Gemini 2.5 Flash) on the final screenshot. Time is split with the repo's
profiler (
--profile-out), as indocs/benchmarks.md.qev serve)qev serve)Times are medians over the runs. At 30 s, ten runs were started and three were interrupted by hand before the end;
those three are not counted.
Verification
uv run ruff format --check . && uv run ruff check .: clean.uv run ty check: 1 diagnostic ins1a/decision_models/cua.py, the same onmain.uv run pytest -q --ignore=tests/test_browser_policy.py: 394 passed, 16 failed;mainon this Windows machinefails the same 16 (environment), so none is new. 4 new tests in
tests/test_decision_models_jev.py(
TestDecisionTimeout): default 5 s, the variable, an explicit value winning, invalid values (abc,0,-3,nan,inf,-inf,1e9999).