From abe2a88d658025db6d23e1601fbab74c48fab4a4 Mon Sep 17 00:00:00 2001 From: chaimaerachdi Date: Fri, 9 Oct 2026 10:25:19 +0200 Subject: [PATCH 1/2] Decision-deadline evidence: every OneJev-27B run behind #36's table, the three interrupted ones included Co-Authored-By: Claude Opus 5.5 --- .../2026-10-01-a100-qev/README.md | 55 ++++++++ .../2026-10-01-a100-qev/bench.sh | 132 ++++++++++++++++++ .../2026-10-01-a100-qev/bench_30s.csv | 16 +++ .../2026-10-01-a100-qev/bench_30s_fla.csv | 6 + .../2026-10-01-a100-qev/bench_5s.csv | 6 + .../2026-10-01-a100-qev/compact_ok.py | 60 ++++++++ .../2026-10-01-a100-qev/slow_ok.py | 12 ++ 7 files changed, 287 insertions(+) create mode 100644 docs/results/decision-timeout/2026-10-01-a100-qev/README.md create mode 100644 docs/results/decision-timeout/2026-10-01-a100-qev/bench.sh create mode 100644 docs/results/decision-timeout/2026-10-01-a100-qev/bench_30s.csv create mode 100644 docs/results/decision-timeout/2026-10-01-a100-qev/bench_30s_fla.csv create mode 100644 docs/results/decision-timeout/2026-10-01-a100-qev/bench_5s.csv create mode 100644 docs/results/decision-timeout/2026-10-01-a100-qev/compact_ok.py create mode 100644 docs/results/decision-timeout/2026-10-01-a100-qev/slow_ok.py diff --git a/docs/results/decision-timeout/2026-10-01-a100-qev/README.md b/docs/results/decision-timeout/2026-10-01-a100-qev/README.md new file mode 100644 index 0000000..18b2fe8 --- /dev/null +++ b/docs/results/decision-timeout/2026-10-01-a100-qev/README.md @@ -0,0 +1,55 @@ +# The #36 description's OneJev-27B runs, every one kept (2026-10-01 and 2026-10-02, one A100 80 GB) + +The records behind the table in #36's description ("OneJev-27B, 5 s: 0 / 5; 30 s: 7 / 7"), including the three 30 s +runs that were interrupted by hand and left out of the count, plus two later OneJev-27B batches that are not in that +table. Real server, no injected delay. + +## Setup + +- Model and server: OneJev-27B (`OmniJev/OneJev-27B`) through `qev serve --multimodal` on port 8000; s1a reaches it + with `TYPESAFE_API_URL=http://127.0.0.1:8000/v1/systemone TYPESAFE_MODEL=jev-latest` and `--model jev`. + This is not system1-omni and not Open-Jev; see `../2026-10-08-a40-native/` for the system1-omni runs. +- Task: `s1a run flights --timeout 600 --profile-out .json`, one-way Zurich to London on November 1, 2026. +- Hardware: one NVIDIA A100 80 GB PCIe. Success is judged by Gemini 2.5 Flash on the final screenshot + (`runs/*.png`), as in the `bench*.csv` files. +- **How the 30 s deadline was set:** these runs predate #36's code. A test-only launcher, `slow_ok.py`, replaced + `JevModel.from_env`'s default deadline with 30 s; `S1A_DECISION_TIMEOUT_S=30` does the same since #36. + The 5 s runs used the default deadline. `compact_ok.py` is the same launcher plus a #9-style compact page state. + `bench.sh` is the batch script as it is now, after these runs. + +## The table in #36, run by run + +| run (file prefix) | deadline | decisions | outcome | counted in #36 | +|---|---|---|---|---| +| `onejev27B_125333` to `onejev27B_125659` (5 runs started 2026-10-01 12:53 to 12:56) | 5 s (default) | 7 each | BLOCKED at decision 8, the date on the calendar page: `decisions connection failed` | yes, 0 / 5 | +| `onejev27B_135818`, `onejev27B_135952` | 30 s | 11 each | flights shown | yes | +| `onejev27B_140127` | 30 s | none recorded | **interrupted by hand** (Ctrl+C, `KeyboardInterrupt` in the log) | no | +| `onejev27B_140231`, `onejev27B_140246` | 30 s | 0 | **interrupted**: the first request failed 3 to 8 s after start (`decisions connection failed`) while the batch was being stopped by hand, as #36's description says | no | +| `onejev27B_151748` to `onejev27B_152407` (5 runs) | 30 s | 11 each | flights shown | yes | + +30 s: ten runs started, seven completed and counted (7 / 7), three interrupted and not counted. Over the seven: +model time per task 60.3 to 62.8 s (median 61.2 s), median decision 3.2 to 3.8 s (median of medians 3.7 s), page +11.3 to 11.4 s, browser 9.6 to 9.9 s (`bench_30s.csv`). #36's description gives 9.7 s for the browser median; the +seven values give 9.9 s. + +Where the 5 s deadline fails, decision by decision (run `onejev27B_151748`, 30 s): decisions 1 to 7 take 1.0 to +3.8 s; decision 8, picking November 1 on the calendar page (15,062 input tokens against 5,745 before it), takes +13.2 s; decision 9 (Done) 17.3 s. With 5 s, decision 8 is cut, which is where every 5 s run stops. + +## Not in the #36 table + +| run (file prefix) | deadline | outcome | +|---|---|---| +| `onejev27Bc_152539` to `onejev27Bc_153103` (5 runs started 2026-10-01 15:25 to 15:31) | 30 s, compact page state (`compact_ok.py`) | 4 flights shown; 1 Google error page ("Oops, something went wrong") | +| `onejev27B_094702` to `onejev27B_095250` (5 runs started 2026-10-02 09:47 to 09:52) | 30 s, after installing flash-linear-attention on the server | 5 flights shown; median decision 2.6 to 2.7 s (`bench_30s_fla.csv`) | + +## Files + +- `runs/.log`: each run's stdout and stderr (the final JSON with status, answer and actions, or the + traceback). `runs/.json`: the `--profile-out` record with every decision's time, tokens and choice (absent + for the 5 s runs, which were not profiled, and for `140127`). `runs/.png`: the final screenshot. +- `bench_5s.csv`, `bench_30s.csv`, `bench_30s_fla.csv`: the batch rows (time is the run's end), with the judge's + verdict. +- `slow_ok.py`, `compact_ok.py`, `bench.sh`: the launchers and batch script. Home paths are shortened to `~/`. + +Privacy and reuse: the browser was signed out; screenshots are Google Flights pages, kept as evaluation evidence. diff --git a/docs/results/decision-timeout/2026-10-01-a100-qev/bench.sh b/docs/results/decision-timeout/2026-10-01-a100-qev/bench.sh new file mode 100644 index 0000000..d1da1b5 --- /dev/null +++ b/docs/results/decision-timeout/2026-10-01-a100-qev/bench.sh @@ -0,0 +1,132 @@ +#!/usr/bin/env bash +# Google Flights benchmark: success rate (vision judge on the final screenshot) and time, per decision model. +# Usage: bash bench.sh "jev omnijev clm" 5 (models, runs per model); results in results/bench.csv +W=~/chaimae; K=$W/gpu_kit; E=$W/clm-env +MODELS=${1:-"jev omnijev clm"}; RUNS=${2:-5} +export PATH=$HOME/.local/bin:$W/node-v22.12.0-linux-x64/bin:$PATH +export LD_LIBRARY_PATH=$W/libs/usr/lib/x86_64-linux-gnu +export PLAYWRIGHT_MCP_ARGS="-y @playwright/mcp@0.0.78 --cdp-endpoint=http://localhost:9222" HF_HUB_OFFLINE=1 +mkdir -p $K/results +CSV=$K/results/bench2.csv +[ -s $CSV ] || echo "time,model,run,success,total_s,decisions,median_decision_ms,model_s,page_s,browser_s,reason" > $CSV + +# CLM needs its two servers (Qwen3-8B embeddings + clm-serve); started once, left running +if [[ " $MODELS " == *" clm "* ]]; then + if [ ! -x $E/bin/clm-serve ]; then echo "== installing vLLM + CLM (~5 min)"; uv venv -q --python 3.12 $E && uv pip install -q --python $E/bin/python vllm contrastive-lm || exit 1; fi + curl -s localhost:8090/v1/models >/dev/null || { echo "== starting Qwen3-8B (download ~16 GB the first time)"; HF_HUB_OFFLINE=0 nohup $E/bin/vllm serve Qwen/Qwen3-8B --served-model-name qwen3-8b --runner pooling --max-model-len 2048 --gpu-memory-utilization 0.5 --port 8090 > $K/results/vllm.log 2>&1 & } + until curl -s localhost:8090/v1/models >/dev/null; do sleep 15; echo " waiting for Qwen3-8B..."; done + curl -s localhost:8700/health >/dev/null || { echo "== starting CLM"; HF_HUB_OFFLINE=0 nohup $E/bin/clm-serve --port 8700 --emb-url http://127.0.0.1:8090/v1/embeddings > $K/results/clm.log 2>&1 & } + until curl -s localhost:8700/health >/dev/null; do sleep 5; echo " waiting for CLM..."; done +fi + +# CLM with a window that holds the whole page (8,192 tokens instead of 2,048): both servers restarted with it +if [[ " $MODELS " == *" clm8k "* ]]; then + pkill -f "vllm serve" ; pkill -f clm-serve ; pkill -f "qev serve" ; sleep 5 + [ -x $E/bin/clm-serve ] || { echo "== installing vLLM + CLM (~5 min)"; uv venv -q --python 3.12 $E && uv pip install -q --python $E/bin/python vllm contrastive-lm || exit 1; } + echo "== starting Qwen3-8B with an 8,192-token window" + HF_HUB_OFFLINE=0 nohup $E/bin/vllm serve Qwen/Qwen3-8B --served-model-name qwen3-8b --runner pooling --max-model-len 8192 --gpu-memory-utilization 0.5 --port 8090 > $K/results/vllm8k.log 2>&1 & + until curl -s localhost:8090/v1/models >/dev/null; do sleep 15; echo " waiting for Qwen3-8B..."; done + echo "== starting CLM reading up to 8,192 tokens" + HF_HUB_OFFLINE=0 nohup $E/bin/clm-serve --port 8700 --emb-url http://127.0.0.1:8090/v1/embeddings --max-tokens 8192 > $K/results/clm8k.log 2>&1 & + until curl -s localhost:8700/health >/dev/null; do sleep 5; echo " waiting for CLM..."; done +fi + +# OneJev (27B or 9B): served by its own qev server on port 8000, which speaks Jev's API; the GPU cannot hold it next +# to CLM's servers, so those are stopped first. Only one OneJev size per call. +Q=$W/qev-env +for size in 27B 9B 4B 0.8B; do + if [[ " $MODELS " == *" onejev$size "* || " $MODELS " == *" onejev${size}c "* ]]; then + pkill -f "vllm serve" ; pkill -f clm-serve ; pkill -f "qev serve" ; sleep 5 + if [ ! -x $Q/bin/qev ]; then echo "== installing OneJev's server (qev)"; uv venv -q --python 3.12 $Q && uv pip install -q --python $Q/bin/python "qev[torch] @ git+https://github.com/OmniJev/OneJev.git" || exit 1; fi + echo "== starting OneJev-$size (download the first time: 55 GB for 27B, 19 GB for 9B)" + HF_HUB_OFFLINE=0 nohup $Q/bin/qev serve --model OmniJev/OneJev-$size --multimodal --port 8000 > $K/results/onejev_$size.log 2>&1 & + until curl -s localhost:8000/health >/dev/null; do sleep 15; echo " waiting for OneJev-$size... ($(tail -c 100 $K/results/onejev_$size.log | tr '\n' ' '))"; done + curl -s localhost:8000/health; echo + fi +done + +# One headless Chrome with its own profile, cookies refused once +C=$(ls -d $HOME/.cache/ms-playwright/chromium-*/chrome-linux*/chrome | head -1) +pgrep -f "remote-debugging-port=9222" >/dev/null || { "$C" --headless=new --no-sandbox --remote-debugging-port=9222 --user-data-dir="$K/chrome-profile" --window-size=1280,900 --no-first-run about:blank >/dev/null 2>&1 & sleep 5; } +cd $K/system1-agents +# DRIVER=fast: the team's best driver in this repo (docs/benchmarks.md arm C''): Playwright MCP without the 500 ms +# settle sleeps (scripts/pw_mcp_nosettle.sh) and --batch on. Rows are labelled -fast. +EXTRA=""; TAG="" +if [ "${DRIVER:-stock}" = fast ]; then + eval "$(bash scripts/pw_mcp_nosettle.sh)" + export PLAYWRIGHT_MCP_ARGS="$PLAYWRIGHT_MCP_ARGS --cdp-endpoint=http://localhost:9222" + EXTRA="--batch on"; TAG="-fast" + echo "== driver: no settle + batched ($PLAYWRIGHT_MCP_COMMAND $PLAYWRIGHT_MCP_ARGS)" +fi +uv run --with websocket-client python $K/reject_cookies.py >/dev/null +TASK=$(uv run python -c "from s1a.agents.flights import SPEC; print(SPEC.goal)" 2>/dev/null | tail -1) +echo "== task: $TASK" + +for model in $MODELS; do + for i in $(seq 1 $RUNS); do + log=$K/results/bench_${model}${TAG}_$(date +%H%M%S).log + prof=${log%.log}.json # the team's profiler: where the seconds go (model, page, browser) + case $model in + jev) env -u TYPESAFE_API_URL -u TYPESAFE_MODEL uv run s1a run flights --model jev --timeout 600 $EXTRA --profile-out "$prof" > "$log" 2>&1 ;; + clm|clm8k) TYPESAFE_API_URL=http://127.0.0.1:8700/v1/systemone TYPESAFE_MODEL=clm-latest uv run python $K/slow_ok.py run flights --model jev --timeout 600 $EXTRA --profile-out "$prof" > "$log" 2>&1 ;; + onejev27B|onejev9B|onejev4B|onejev0.8B)TYPESAFE_API_URL=http://127.0.0.1:8000/v1/systemone TYPESAFE_MODEL=jev-latest uv run python $K/slow_ok.py run flights --model jev --timeout 600 $EXTRA --profile-out "$prof" > "$log" 2>&1 ;; + onejev27Bc|onejev9Bc) TYPESAFE_API_URL=http://127.0.0.1:8000/v1/systemone TYPESAFE_MODEL=jev-latest uv run python $K/compact_ok.py run flights --model jev --timeout 600 $EXTRA --profile-out "$prof" > "$log" 2>&1 ;; + omnijev) OMNIJEV_REPO=$K/OmniJev OMNIJEV_CHECKPOINT=$K/ckpt-4b OMNIJEV_BASE=$K/base-4b uv run s1a run flights --model omnijev --timeout 600 --profile-out "$prof" > "$log" 2>&1 ;; + esac + shot=${log%.log}.png + uv run --with websocket-client python $K/final_shot.py "$shot" >/dev/null 2>&1 + verdict=$(uv run python $K/judge.py "$shot" "$TASK" 2>/dev/null | tail -1) + line=$(uv run python - "$log" "$verdict" "$model$TAG" "$i" <<'PY' +import json, sys, time +log, verdict, model, run = sys.argv[1:5] +text = open(log, encoding="utf-8", errors="replace").read() +k = text.rfind('{"ok"') +try: + answer = json.loads(text[k:text.index("\n", k)] if "\n" in text[k:] else text[k:]) if k >= 0 else {} +except json.JSONDecodeError: + answer = {} +report = answer.get("report") or {} +try: + judged = json.loads(verdict) +except json.JSONDecodeError: + judged = {"success": False, "reason": "judge failed"} +reason = str(judged.get("reason", "")).replace(",", ";").replace("\n", " ") +try: # the team's profiler (docs/benchmarks.md): decision = waiting on the model, probe = page, tool = browser + phases = json.load(open(log[:-4] + ".json", encoding="utf-8")).get("phases") or {} +except (OSError, ValueError): + phases = {} +model_s = phases.get("decision", phases.get("jev", 0)) / 1000 +print(f"{time.strftime('%H:%M:%S')},{model},{run},{int(bool(judged.get('success')))}," + f"{report.get('elapsed_ms', 0) / 1000:.1f},{report.get('decisions', 0)},{report.get('median_decision_ms', 0)}," + f"{model_s:.1f},{phases.get('probe', 0) / 1000:.1f},{phases.get('tool', 0) / 1000:.1f},{reason}") +PY +) + echo "$line" >> $CSV + echo "$line" + # the actions this run took, in order, to see where it goes wrong + uv run python - "$log" <<'PY' +import json, sys +text = open(sys.argv[1], encoding="utf-8", errors="replace").read() +k = text.rfind('{"ok"') +try: + history = (json.loads(text[k:].split("\n", 1)[0]).get("report") or {}).get("history") or [] +except (json.JSONDecodeError, ValueError): + history = [] +steps = [(h.get("text") and f"type {h['text']}") or f"{h.get('kind')} {str(h.get('action'))[:30]}" for h in history] +print(" actions: " + " > ".join(steps[:16]) + (" > ..." if len(steps) > 16 else "")) +PY + done +done + +echo "== summary (all rows in results/bench2.csv)" +uv run python - "$CSV" <<'PY' +import csv, statistics, sys +rows = list(csv.DictReader(open(sys.argv[1], encoding="utf-8"))) +print(f"{'model':<10} {'runs':>4} {'success':>8} {'rate':>6} {'total s':>8} {'model s':>8} {'page s':>7} {'browser s':>10} {'per decision ms':>16}") +for model in dict.fromkeys(r["model"] for r in rows): + rs = [r for r in rows if r["model"] == model] + ok = sum(int(r["success"]) for r in rs) + med = lambda key: statistics.median(float(r[key]) for r in rs) + print(f"{model:<10} {len(rs):>4} {ok:>8} {100 * ok / len(rs):>5.0f}% {med('total_s'):>8.1f} {med('model_s'):>8.1f} " + f"{med('page_s'):>7.1f} {med('browser_s'):>10.1f} {med('median_decision_ms'):>16.0f}") +PY diff --git a/docs/results/decision-timeout/2026-10-01-a100-qev/bench_30s.csv b/docs/results/decision-timeout/2026-10-01-a100-qev/bench_30s.csv new file mode 100644 index 0000000..ae48ea5 --- /dev/null +++ b/docs/results/decision-timeout/2026-10-01-a100-qev/bench_30s.csv @@ -0,0 +1,16 @@ +time,model,run,success,total_s,decisions,median_decision_ms,model_s,page_s,browser_s,reason +13:59:52,onejev27B,1,1,86.6,11,3682,61.2,11.4,9.9,The page displays one-way flight options from Zurich to London on November 1; 2026; for one adult in economy; matching all criteria. +14:01:27,onejev27B,2,1,87.4,11,3732,62.1,11.4,9.9,The page displays one-way flight results from Zurich to London on November 1; with multiple options visible; matching all specified criteria. +14:02:31,onejev27B,3,0,0.0,0,0,0.0,0.0,0.0,The agent has selected the correct date and origin; but has not yet entered the destination or displayed flight results. +14:02:46,onejev27B,4,0,7.7,0,0,0.0,0.0,0.0,The agent stopped on the initial flight search page; with no flight results visible and the form not yet filled out. +14:02:56,onejev27B,5,0,2.6,0,0,0.0,0.0,0.0,The agent stopped on the initial flight search page; with no flight results visible and the form not yet filled out. +15:19:23,onejev27B,1,1,85.6,11,3662,61.2,11.3,9.6,The page displays a list of one-way flight results from Zurich to London on November 1; with prices and flight details. +15:20:57,onejev27B,2,1,86.8,11,3733,62.1,11.4,9.7,The page displays a list of one-way flight results from Zurich to London on November 1; with prices and flight details. +15:22:32,onejev27B,3,1,87.3,11,3769,62.6,11.4,9.9,The page displays a list of one-way flight results from Zurich to London on November 1; with the correct origin; destination; and date. +15:24:07,onejev27B,4,1,87.5,11,3787,62.8,11.4,9.9,The page displays a list of one-way flight options from Zurich to London on November 1; with prices and flight details visible. +15:25:39,onejev27B,5,1,84.6,11,3237,60.3,11.4,9.7,The page shows a list of one-way flight results from Zurich to London on November 1; with prices and flight details visible. +15:27:00,onejev27Bc,1,1,73.6,11,3082,48.7,11.4,10.0,The page displays a list of one-way flight options from Zurich to London on November 1; with prices and flight details visible. +15:28:21,onejev27Bc,2,1,73.6,11,3079,48.6,11.4,10.0,The page displays a list of one-way flight options from Zurich to London on November 1; with prices and flight details visible. +15:29:42,onejev27Bc,3,1,73.4,11,3076,48.6,11.5,9.9,The page displays a list of one-way flight options from Zurich to London on November 1; with prices and flight details visible. +15:31:03,onejev27Bc,4,0,74.3,11,3073,46.6,15.4,8.9,The page shows an error message "Oops; something went wrong" instead of flight results. +15:32:24,onejev27Bc,5,1,73.8,11,3070,48.6,11.4,9.9,The page shows a list of one-way flight results from Zurich to London on November 1; with the correct number of adults and economy class selected. diff --git a/docs/results/decision-timeout/2026-10-01-a100-qev/bench_30s_fla.csv b/docs/results/decision-timeout/2026-10-01-a100-qev/bench_30s_fla.csv new file mode 100644 index 0000000..7cb2c61 --- /dev/null +++ b/docs/results/decision-timeout/2026-10-01-a100-qev/bench_30s_fla.csv @@ -0,0 +1,6 @@ +time,model,run,success,total_s,decisions,median_decision_ms,model_s,page_s,browser_s,reason +09:49:01,onejev27B,1,1,111.0,11,2697,81.8,11.7,9.5,The page displays a list of one-way flight options from Zurich to London on November 1; with prices and flight details visible. +09:50:17,onejev27B,2,1,67.9,11,2639,43.8,11.2,9.7,The page displays a list of one-way flight options from Zurich to London on November 1; with prices and flight details visible. +09:51:34,onejev27B,3,1,68.9,11,2666,44.2,11.8,9.7,The page displays a list of one-way flight options from Zurich to London on November 1; with prices and flight details visible. +09:52:50,onejev27B,4,1,69.0,11,2677,44.3,11.8,9.8,The page displays a list of one-way flight options from Zurich to London on November 1; with prices and flight details visible. +09:54:06,onejev27B,5,1,68.1,11,2651,43.8,11.2,9.7,The page displays a list of one-way flight options from Zurich to London on November 1; with prices and flight details visible. diff --git a/docs/results/decision-timeout/2026-10-01-a100-qev/bench_5s.csv b/docs/results/decision-timeout/2026-10-01-a100-qev/bench_5s.csv new file mode 100644 index 0000000..ad1bf20 --- /dev/null +++ b/docs/results/decision-timeout/2026-10-01-a100-qev/bench_5s.csv @@ -0,0 +1,6 @@ +time,model,run,success,total_s,decisions,median_decision_ms,reason +12:54:27,onejev27B,1,0,47.2,7,2984,The agent has not yet selected the destination or the date; and no flight options are visible. +12:55:17,onejev27B,2,0,43.0,7,3022,The agent has not yet selected the destination or the date; and no flight options are visible. +12:56:08,onejev27B,3,0,43.4,7,3049,The agent has not yet selected the destination or the date; and no flight options are visible. +12:56:59,onejev27B,4,0,43.4,7,3073,The agent has not yet selected the destination or the date; and no flight options are visible. +12:57:50,onejev27B,5,0,43.8,7,3086,The agent has not yet selected the destination or the date; and no flight options are visible. diff --git a/docs/results/decision-timeout/2026-10-01-a100-qev/compact_ok.py b/docs/results/decision-timeout/2026-10-01-a100-qev/compact_ok.py new file mode 100644 index 0000000..becc62a --- /dev/null +++ b/docs/results/decision-timeout/2026-10-01-a100-qev/compact_ok.py @@ -0,0 +1,60 @@ +# Test-only launcher: s1a with a 30 s decision limit AND the browser page compressed before every decision, the way +# PR #9 folds it for Laya: url and title, one short line per element, the last 3 actions, no page text. +# The questions and the model are unchanged; only the state the model reads is smaller. +import sys + +import s1a.decision_models.jev as jev + +LABEL_CHARS, TITLE_CHARS, HISTORY_KEPT = 40, 80, 3 +FLAGS = (("checked", "C"), ("selected", "S"), ("expanded", "X"), ("click_did_nothing", "D")) + + +def row_line(row): + label = str(row.get("label") or "")[:LABEL_CHARS] + value = str(row.get("value") or "")[:LABEL_CHARS] + flags = "".join(letter for key, letter in FLAGS if row.get(key)) + parts = [str(row.get("index", "")), str(row.get("role") or ""), label] + if value: + parts.append(f"={value}") + if flags: + parts.append(f"[{flags}]") + if row.get("blocked_by"): + parts.append(f"(blocked by {str(row['blocked_by'])[:24]})") + return " ".join(p for p in parts if p) + + +def compact(state): + if not ( + isinstance(state, dict) and isinstance(state.get("page"), dict) and isinstance(state.get("elements"), list) + ): + return state + folded = { + "page": {"url": str(state["page"].get("url", "")), "title": str(state["page"].get("title", ""))[:TITLE_CHARS]}, + "elements": [row_line(row) for row in state["elements"]], + } + recent = state.get("recent_actions") + if recent: + folded["recent_actions"] = [ + f"{e.get('kind', '')}:{e.get('action', '')}" + ("" if e.get("page_changed") else " (no change)") + for e in recent[-HISTORY_KEPT:] + ] + return folded + + +_orig_from_env = jev.JevModel.from_env.__func__ +jev.JevModel.from_env = classmethod(lambda cls, timeout_s=30.0: _orig_from_env(cls, timeout_s=timeout_s)) + +_orig_decide = jev.JevModel._decide + + +async def _decide_compact(self, observation, questions): + folded = type(observation)(compact(observation.state), images=observation.images) + return await _orig_decide(self, folded, questions) + + +jev.JevModel._decide = _decide_compact + +from s1a.entry import s1a # noqa: E402 + +sys.argv = ["s1a", *sys.argv[1:]] +sys.exit(s1a()) diff --git a/docs/results/decision-timeout/2026-10-01-a100-qev/slow_ok.py b/docs/results/decision-timeout/2026-10-01-a100-qev/slow_ok.py new file mode 100644 index 0000000..fea6616 --- /dev/null +++ b/docs/results/decision-timeout/2026-10-01-a100-qev/slow_ok.py @@ -0,0 +1,12 @@ +# Test-only launcher: s1a with a 30 s limit per decision instead of 5 s, for local models slower than Jev. +import sys + +import s1a.decision_models.jev as jev + +_orig = jev.JevModel.from_env.__func__ +jev.JevModel.from_env = classmethod(lambda cls, timeout_s=30.0: _orig(cls, timeout_s=timeout_s)) + +from s1a.entry import s1a # noqa: E402 + +sys.argv = ["s1a", *sys.argv[1:]] +sys.exit(s1a()) From cfc83d2e1a33dfd9170ccdb34d223cd5119fd9e4 Mon Sep 17 00:00:00 2001 From: chaimaerachdi Date: Mon, 12 Oct 2026 17:27:52 +0200 Subject: [PATCH 2/2] Decision-deadline evidence: link the raw run records archive The 69 raw files (logs, profiles, final screenshots) stay outside the source tree, per the repository's artifact hygiene; the README links the archive attached to the pull request, with its sha256. Co-Authored-By: Claude Opus 5.5 --- docs/results/decision-timeout/2026-10-01-a100-qev/README.md | 6 +++++- 1 file changed, 5 insertions(+), 1 deletion(-) diff --git a/docs/results/decision-timeout/2026-10-01-a100-qev/README.md b/docs/results/decision-timeout/2026-10-01-a100-qev/README.md index 18b2fe8..8cc70c4 100644 --- a/docs/results/decision-timeout/2026-10-01-a100-qev/README.md +++ b/docs/results/decision-timeout/2026-10-01-a100-qev/README.md @@ -47,7 +47,11 @@ Where the 5 s deadline fails, decision by decision (run `onejev27B_151748`, 30 s - `runs/.log`: each run's stdout and stderr (the final JSON with status, answer and actions, or the traceback). `runs/.json`: the `--profile-out` record with every decision's time, tokens and choice (absent - for the 5 s runs, which were not profiled, and for `140127`). `runs/.png`: the final screenshot. + for the 5 s runs, which were not profiled, and for `140127`). `runs/.png`: the final screenshot. These 69 + files (25 logs, 19 profiles, 25 screenshots) are kept outside the source tree, in + [decision-timeout-2026-10-01-a100-qev-runs.zip](https://github.com/user-attachments/files/33344291/decision-timeout-2026-10-01-a100-qev-runs.zip) + (sha256 `372ff9d828250c2b146f341578a2c7bfbaac00fcf443bcb6aa6291986896fe57`), attached to the pull request; the only + edit is the run machine's home directory written `~` in five Python library paths. - `bench_5s.csv`, `bench_30s.csv`, `bench_30s_fla.csv`: the batch rows (time is the run's end), with the judge's verdict. - `slow_ok.py`, `compact_ok.py`, `bench.sh`: the launchers and batch script. Home paths are shortened to `~/`.