[Docs] Every OneJev-27B run behind #36's deadline table, the three interrupted ones included

docs/results/decision-timeout/2026-10-01-a100-qev/README.md

@@ -0,0 +1,59 @@
+# The #36 description's OneJev-27B runs, every one kept (2026-10-01 and 2026-10-02, one A100 80 GB)
+
+The records behind the table in #36's description ("OneJev-27B, 5 s: 0 / 5; 30 s: 7 / 7"), including the three 30 s
+runs that were interrupted by hand and left out of the count, plus two later OneJev-27B batches that are not in that
+table. Real server, no injected delay.
+
+## Setup
+
+- Model and server: OneJev-27B (`OmniJev/OneJev-27B`) through `qev serve --multimodal` on port 8000; s1a reaches it
+  with `TYPESAFE_API_URL=http://127.0.0.1:8000/v1/systemone TYPESAFE_MODEL=jev-latest` and `--model jev`.
+  This is not system1-omni and not Open-Jev; see `../2026-10-08-a40-native/` for the system1-omni runs.
+- Task: `s1a run flights --timeout 600 --profile-out <run>.json`, one-way Zurich to London on November 1, 2026.
+- Hardware: one NVIDIA A100 80 GB PCIe. Success is judged by Gemini 2.5 Flash on the final screenshot
+  (`runs/*.png`), as in the `bench*.csv` files.
+- **How the 30 s deadline was set:** these runs predate #36's code. A test-only launcher, `slow_ok.py`, replaced
+  `JevModel.from_env`'s default deadline with 30 s; `S1A_DECISION_TIMEOUT_S=30` does the same since #36.
+  The 5 s runs used the default deadline. `compact_ok.py` is the same launcher plus a #9-style compact page state.
+  `bench.sh` is the batch script as it is now, after these runs.
+
+## The table in #36, run by run
+
+| run (file prefix) | deadline | decisions | outcome | counted in #36 |
+|---|---|---|---|---|
+| `onejev27B_125333` to `onejev27B_125659` (5 runs started 2026-10-01 12:53 to 12:56) | 5 s (default) | 7 each | BLOCKED at decision 8, the date on the calendar page: `decisions connection failed` | yes, 0 / 5 |
+| `onejev27B_135818`, `onejev27B_135952` | 30 s | 11 each | flights shown | yes |
+| `onejev27B_140127` | 30 s | none recorded | **interrupted by hand** (Ctrl+C, `KeyboardInterrupt` in the log) | no |
+| `onejev27B_140231`, `onejev27B_140246` | 30 s | 0 | **interrupted**: the first request failed 3 to 8 s after start (`decisions connection failed`) while the batch was being stopped by hand, as #36's description says | no |
+| `onejev27B_151748` to `onejev27B_152407` (5 runs) | 30 s | 11 each | flights shown | yes |
+
+30 s: ten runs started, seven completed and counted (7 / 7), three interrupted and not counted. Over the seven:
+model time per task 60.3 to 62.8 s (median 61.2 s), median decision 3.2 to 3.8 s (median of medians 3.7 s), page
+11.3 to 11.4 s, browser 9.6 to 9.9 s (`bench_30s.csv`). #36's description gives 9.7 s for the browser median; the
+seven values give 9.9 s.
+
+Where the 5 s deadline fails, decision by decision (run `onejev27B_151748`, 30 s): decisions 1 to 7 take 1.0 to
+3.8 s; decision 8, picking November 1 on the calendar page (15,062 input tokens against 5,745 before it), takes
+13.2 s; decision 9 (Done) 17.3 s. With 5 s, decision 8 is cut, which is where every 5 s run stops.
+
+## Not in the #36 table
+
+| run (file prefix) | deadline | outcome |
+|---|---|---|
+| `onejev27Bc_152539` to `onejev27Bc_153103` (5 runs started 2026-10-01 15:25 to 15:31) | 30 s, compact page state (`compact_ok.py`) | 4 flights shown; 1 Google error page ("Oops, something went wrong") |
+| `onejev27B_094702` to `onejev27B_095250` (5 runs started 2026-10-02 09:47 to 09:52) | 30 s, after installing flash-linear-attention on the server | 5 flights shown; median decision 2.6 to 2.7 s (`bench_30s_fla.csv`) |
+
+## Files
+
+- `runs/<prefix>.log`: each run's stdout and stderr (the final JSON with status, answer and actions, or the
+  traceback). `runs/<prefix>.json`: the `--profile-out` record with every decision's time, tokens and choice (absent
+  for the 5 s runs, which were not profiled, and for `140127`). `runs/<prefix>.png`: the final screenshot. These 69
+  files (25 logs, 19 profiles, 25 screenshots) are kept outside the source tree, in
+  [decision-timeout-2026-10-01-a100-qev-runs.zip](https://github.com/user-attachments/files/33344291/decision-timeout-2026-10-01-a100-qev-runs.zip)
+  (sha256 `372ff9d828250c2b146f341578a2c7bfbaac00fcf443bcb6aa6291986896fe57`), attached to the pull request; the only
+  edit is the run machine's home directory written `~` in five Python library paths.
+- `bench_5s.csv`, `bench_30s.csv`, `bench_30s_fla.csv`: the batch rows (time is the run's end), with the judge's
+  verdict.
+- `slow_ok.py`, `compact_ok.py`, `bench.sh`: the launchers and batch script. Home paths are shortened to `~/`.
+
+Privacy and reuse: the browser was signed out; screenshots are Google Flights pages, kept as evaluation evidence.

docs/results/decision-timeout/2026-10-01-a100-qev/bench.sh

@@ -0,0 +1,132 @@
+#!/usr/bin/env bash
+# Google Flights benchmark: success rate (vision judge on the final screenshot) and time, per decision model.
+# Usage: bash bench.sh "jev omnijev clm" 5      (models, runs per model); results in results/bench.csv
+W=~/chaimae; K=$W/gpu_kit; E=$W/clm-env
+MODELS=${1:-"jev omnijev clm"}; RUNS=${2:-5}
+export PATH=$HOME/.local/bin:$W/node-v22.12.0-linux-x64/bin:$PATH
+export LD_LIBRARY_PATH=$W/libs/usr/lib/x86_64-linux-gnu
+export PLAYWRIGHT_MCP_ARGS="-y @playwright/mcp@0.0.78 --cdp-endpoint=http://localhost:9222" HF_HUB_OFFLINE=1
+mkdir -p $K/results
+CSV=$K/results/bench2.csv
+[ -s $CSV ] || echo "time,model,run,success,total_s,decisions,median_decision_ms,model_s,page_s,browser_s,reason" > $CSV
+
+# CLM needs its two servers (Qwen3-8B embeddings + clm-serve); started once, left running
+if [[ " $MODELS " == *" clm "* ]]; then
+  if [ ! -x $E/bin/clm-serve ]; then echo "== installing vLLM + CLM (~5 min)"; uv venv -q --python 3.12 $E && uv pip install -q --python $E/bin/python vllm contrastive-lm || exit 1; fi
+  curl -s localhost:8090/v1/models >/dev/null || { echo "== starting Qwen3-8B (download ~16 GB the first time)"; HF_HUB_OFFLINE=0 nohup $E/bin/vllm serve Qwen/Qwen3-8B --served-model-name qwen3-8b --runner pooling --max-model-len 2048 --gpu-memory-utilization 0.5 --port 8090 > $K/results/vllm.log 2>&1 & }
+  until curl -s localhost:8090/v1/models >/dev/null; do sleep 15; echo "   waiting for Qwen3-8B..."; done
+  curl -s localhost:8700/health >/dev/null || { echo "== starting CLM"; HF_HUB_OFFLINE=0 nohup $E/bin/clm-serve --port 8700 --emb-url http://127.0.0.1:8090/v1/embeddings > $K/results/clm.log 2>&1 & }
+  until curl -s localhost:8700/health >/dev/null; do sleep 5; echo "   waiting for CLM..."; done
+fi
+
+# CLM with a window that holds the whole page (8,192 tokens instead of 2,048): both servers restarted with it
+if [[ " $MODELS " == *" clm8k "* ]]; then
+  pkill -f "vllm serve" ; pkill -f clm-serve ; pkill -f "qev serve" ; sleep 5
+  [ -x $E/bin/clm-serve ] || { echo "== installing vLLM + CLM (~5 min)"; uv venv -q --python 3.12 $E && uv pip install -q --python $E/bin/python vllm contrastive-lm || exit 1; }
+  echo "== starting Qwen3-8B with an 8,192-token window"
+  HF_HUB_OFFLINE=0 nohup $E/bin/vllm serve Qwen/Qwen3-8B --served-model-name qwen3-8b --runner pooling --max-model-len 8192 --gpu-memory-utilization 0.5 --port 8090 > $K/results/vllm8k.log 2>&1 &
+  until curl -s localhost:8090/v1/models >/dev/null; do sleep 15; echo "   waiting for Qwen3-8B..."; done
+  echo "== starting CLM reading up to 8,192 tokens"
+  HF_HUB_OFFLINE=0 nohup $E/bin/clm-serve --port 8700 --emb-url http://127.0.0.1:8090/v1/embeddings --max-tokens 8192 > $K/results/clm8k.log 2>&1 &
+  until curl -s localhost:8700/health >/dev/null; do sleep 5; echo "   waiting for CLM..."; done
+fi
+
+# OneJev (27B or 9B): served by its own qev server on port 8000, which speaks Jev's API; the GPU cannot hold it next
+# to CLM's servers, so those are stopped first. Only one OneJev size per call.
+Q=$W/qev-env
+for size in 27B 9B 4B 0.8B; do
+  if [[ " $MODELS " == *" onejev$size "* || " $MODELS " == *" onejev${size}c "* ]]; then
+    pkill -f "vllm serve" ; pkill -f clm-serve ; pkill -f "qev serve" ; sleep 5
+    if [ ! -x $Q/bin/qev ]; then echo "== installing OneJev's server (qev)"; uv venv -q --python 3.12 $Q && uv pip install -q --python $Q/bin/python "qev[torch] @ git+https://github.com/OmniJev/OneJev.git" || exit 1; fi
+    echo "== starting OneJev-$size (download the first time: 55 GB for 27B, 19 GB for 9B)"
+    HF_HUB_OFFLINE=0 nohup $Q/bin/qev serve --model OmniJev/OneJev-$size --multimodal --port 8000 > $K/results/onejev_$size.log 2>&1 &
+    until curl -s localhost:8000/health >/dev/null; do sleep 15; echo "   waiting for OneJev-$size... ($(tail -c 100 $K/results/onejev_$size.log | tr '\n' ' '))"; done
+    curl -s localhost:8000/health; echo
+  fi
+done
+
+# One headless Chrome with its own profile, cookies refused once
+C=$(ls -d $HOME/.cache/ms-playwright/chromium-*/chrome-linux*/chrome | head -1)
+pgrep -f "remote-debugging-port=9222" >/dev/null || { "$C" --headless=new --no-sandbox --remote-debugging-port=9222 --user-data-dir="$K/chrome-profile" --window-size=1280,900 --no-first-run about:blank >/dev/null 2>&1 & sleep 5; }
+cd $K/system1-agents
+# DRIVER=fast: the team's best driver in this repo (docs/benchmarks.md arm C''): Playwright MCP without the 500 ms
+# settle sleeps (scripts/pw_mcp_nosettle.sh) and --batch on. Rows are labelled <model>-fast.
+EXTRA=""; TAG=""
+if [ "${DRIVER:-stock}" = fast ]; then
+  eval "$(bash scripts/pw_mcp_nosettle.sh)"
+  export PLAYWRIGHT_MCP_ARGS="$PLAYWRIGHT_MCP_ARGS --cdp-endpoint=http://localhost:9222"
+  EXTRA="--batch on"; TAG="-fast"
+  echo "== driver: no settle + batched ($PLAYWRIGHT_MCP_COMMAND $PLAYWRIGHT_MCP_ARGS)"
+fi
+uv run --with websocket-client python $K/reject_cookies.py >/dev/null
+TASK=$(uv run python -c "from s1a.agents.flights import SPEC; print(SPEC.goal)" 2>/dev/null | tail -1)
+echo "== task: $TASK"
+
+for model in $MODELS; do
+  for i in $(seq 1 $RUNS); do
+    log=$K/results/bench_${model}${TAG}_$(date +%H%M%S).log
+    prof=${log%.log}.json  # the team's profiler: where the seconds go (model, page, browser)
+    case $model in
+      jev)     env -u TYPESAFE_API_URL -u TYPESAFE_MODEL uv run s1a run flights --model jev --timeout 600 $EXTRA --profile-out "$prof" > "$log" 2>&1 ;;
+      clm|clm8k) TYPESAFE_API_URL=http://127.0.0.1:8700/v1/systemone TYPESAFE_MODEL=clm-latest uv run python $K/slow_ok.py run flights --model jev --timeout 600 $EXTRA --profile-out "$prof" > "$log" 2>&1 ;;
+      onejev27B|onejev9B|onejev4B|onejev0.8B)TYPESAFE_API_URL=http://127.0.0.1:8000/v1/systemone TYPESAFE_MODEL=jev-latest uv run python $K/slow_ok.py run flights --model jev --timeout 600 $EXTRA --profile-out "$prof" > "$log" 2>&1 ;;
+      onejev27Bc|onejev9Bc) TYPESAFE_API_URL=http://127.0.0.1:8000/v1/systemone TYPESAFE_MODEL=jev-latest uv run python $K/compact_ok.py run flights --model jev --timeout 600 $EXTRA --profile-out "$prof" > "$log" 2>&1 ;;
+      omnijev) OMNIJEV_REPO=$K/OmniJev OMNIJEV_CHECKPOINT=$K/ckpt-4b OMNIJEV_BASE=$K/base-4b uv run s1a run flights --model omnijev --timeout 600 --profile-out "$prof" > "$log" 2>&1 ;;
+    esac
+    shot=${log%.log}.png
+    uv run --with websocket-client python $K/final_shot.py "$shot" >/dev/null 2>&1
+    verdict=$(uv run python $K/judge.py "$shot" "$TASK" 2>/dev/null | tail -1)
+    line=$(uv run python - "$log" "$verdict" "$model$TAG" "$i" <<'PY'
+import json, sys, time
+log, verdict, model, run = sys.argv[1:5]
+text = open(log, encoding="utf-8", errors="replace").read()
+k = text.rfind('{"ok"')
+try:
+    answer = json.loads(text[k:text.index("\n", k)] if "\n" in text[k:] else text[k:]) if k >= 0 else {}
+except json.JSONDecodeError:
+    answer = {}
+report = answer.get("report") or {}
+try:
+    judged = json.loads(verdict)
+except json.JSONDecodeError:
+    judged = {"success": False, "reason": "judge failed"}
+reason = str(judged.get("reason", "")).replace(",", ";").replace("\n", " ")
+try:  # the team's profiler (docs/benchmarks.md): decision = waiting on the model, probe = page, tool = browser
+    phases = json.load(open(log[:-4] + ".json", encoding="utf-8")).get("phases") or {}
+except (OSError, ValueError):
+    phases = {}
+model_s = phases.get("decision", phases.get("jev", 0)) / 1000
+print(f"{time.strftime('%H:%M:%S')},{model},{run},{int(bool(judged.get('success')))},"
+      f"{report.get('elapsed_ms', 0) / 1000:.1f},{report.get('decisions', 0)},{report.get('median_decision_ms', 0)},"
+      f"{model_s:.1f},{phases.get('probe', 0) / 1000:.1f},{phases.get('tool', 0) / 1000:.1f},{reason}")
+PY
+)
+    echo "$line" >> $CSV
+    echo "$line"
+    # the actions this run took, in order, to see where it goes wrong
+    uv run python - "$log" <<'PY'
+import json, sys
+text = open(sys.argv[1], encoding="utf-8", errors="replace").read()
+k = text.rfind('{"ok"')
+try:
+    history = (json.loads(text[k:].split("\n", 1)[0]).get("report") or {}).get("history") or []
+except (json.JSONDecodeError, ValueError):
+    history = []
+steps = [(h.get("text") and f"type {h['text']}") or f"{h.get('kind')} {str(h.get('action'))[:30]}" for h in history]
+print("   actions: " + " > ".join(steps[:16]) + (" > ..." if len(steps) > 16 else ""))
+PY
+  done
+done
+
+echo "== summary (all rows in results/bench2.csv)"
+uv run python - "$CSV" <<'PY'
+import csv, statistics, sys
+rows = list(csv.DictReader(open(sys.argv[1], encoding="utf-8")))
+print(f"{'model':<10} {'runs':>4} {'success':>8} {'rate':>6} {'total s':>8} {'model s':>8} {'page s':>7} {'browser s':>10} {'per decision ms':>16}")
+for model in dict.fromkeys(r["model"] for r in rows):
+    rs = [r for r in rows if r["model"] == model]
+    ok = sum(int(r["success"]) for r in rs)
+    med = lambda key: statistics.median(float(r[key]) for r in rs)
+    print(f"{model:<10} {len(rs):>4} {ok:>8} {100 * ok / len(rs):>5.0f}% {med('total_s'):>8.1f} {med('model_s'):>8.1f} "
+          f"{med('page_s'):>7.1f} {med('browser_s'):>10.1f} {med('median_decision_ms'):>16.0f}")
+PY

docs/results/decision-timeout/2026-10-01-a100-qev/bench_30s.csv

@@ -0,0 +1,16 @@
+time,model,run,success,total_s,decisions,median_decision_ms,model_s,page_s,browser_s,reason
+13:59:52,onejev27B,1,1,86.6,11,3682,61.2,11.4,9.9,The page displays one-way flight options from Zurich to London on November 1; 2026; for one adult in economy; matching all criteria.
+14:01:27,onejev27B,2,1,87.4,11,3732,62.1,11.4,9.9,The page displays one-way flight results from Zurich to London on November 1; with multiple options visible; matching all specified criteria.
+14:02:31,onejev27B,3,0,0.0,0,0,0.0,0.0,0.0,The agent has selected the correct date and origin; but has not yet entered the destination or displayed flight results.
+14:02:46,onejev27B,4,0,7.7,0,0,0.0,0.0,0.0,The agent stopped on the initial flight search page; with no flight results visible and the form not yet filled out.
+14:02:56,onejev27B,5,0,2.6,0,0,0.0,0.0,0.0,The agent stopped on the initial flight search page; with no flight results visible and the form not yet filled out.
+15:19:23,onejev27B,1,1,85.6,11,3662,61.2,11.3,9.6,The page displays a list of one-way flight results from Zurich to London on November 1; with prices and flight details.
+15:20:57,onejev27B,2,1,86.8,11,3733,62.1,11.4,9.7,The page displays a list of one-way flight results from Zurich to London on November 1; with prices and flight details.
+15:22:32,onejev27B,3,1,87.3,11,3769,62.6,11.4,9.9,The page displays a list of one-way flight results from Zurich to London on November 1; with the correct origin; destination; and date.
+15:24:07,onejev27B,4,1,87.5,11,3787,62.8,11.4,9.9,The page displays a list of one-way flight options from Zurich to London on November 1; with prices and flight details visible.
+15:25:39,onejev27B,5,1,84.6,11,3237,60.3,11.4,9.7,The page shows a list of one-way flight results from Zurich to London on November 1; with prices and flight details visible.
+15:27:00,onejev27Bc,1,1,73.6,11,3082,48.7,11.4,10.0,The page displays a list of one-way flight options from Zurich to London on November 1; with prices and flight details visible.
+15:28:21,onejev27Bc,2,1,73.6,11,3079,48.6,11.4,10.0,The page displays a list of one-way flight options from Zurich to London on November 1; with prices and flight details visible.
+15:29:42,onejev27Bc,3,1,73.4,11,3076,48.6,11.5,9.9,The page displays a list of one-way flight options from Zurich to London on November 1; with prices and flight details visible.
+15:31:03,onejev27Bc,4,0,74.3,11,3073,46.6,15.4,8.9,The page shows an error message "Oops; something went wrong" instead of flight results.
+15:32:24,onejev27Bc,5,1,73.8,11,3070,48.6,11.4,9.9,The page shows a list of one-way flight results from Zurich to London on November 1; with the correct number of adults and economy class selected.

docs/results/decision-timeout/2026-10-01-a100-qev/bench_30s_fla.csv

@@ -0,0 +1,6 @@
+time,model,run,success,total_s,decisions,median_decision_ms,model_s,page_s,browser_s,reason
+09:49:01,onejev27B,1,1,111.0,11,2697,81.8,11.7,9.5,The page displays a list of one-way flight options from Zurich to London on November 1; with prices and flight details visible.
+09:50:17,onejev27B,2,1,67.9,11,2639,43.8,11.2,9.7,The page displays a list of one-way flight options from Zurich to London on November 1; with prices and flight details visible.
+09:51:34,onejev27B,3,1,68.9,11,2666,44.2,11.8,9.7,The page displays a list of one-way flight options from Zurich to London on November 1; with prices and flight details visible.
+09:52:50,onejev27B,4,1,69.0,11,2677,44.3,11.8,9.8,The page displays a list of one-way flight options from Zurich to London on November 1; with prices and flight details visible.
+09:54:06,onejev27B,5,1,68.1,11,2651,43.8,11.2,9.7,The page displays a list of one-way flight options from Zurich to London on November 1; with prices and flight details visible.

docs/results/decision-timeout/2026-10-01-a100-qev/bench_5s.csv

@@ -0,0 +1,6 @@
+time,model,run,success,total_s,decisions,median_decision_ms,reason
+12:54:27,onejev27B,1,0,47.2,7,2984,The agent has not yet selected the destination or the date; and no flight options are visible.
+12:55:17,onejev27B,2,0,43.0,7,3022,The agent has not yet selected the destination or the date; and no flight options are visible.
+12:56:08,onejev27B,3,0,43.4,7,3049,The agent has not yet selected the destination or the date; and no flight options are visible.
+12:56:59,onejev27B,4,0,43.4,7,3073,The agent has not yet selected the destination or the date; and no flight options are visible.
+12:57:50,onejev27B,5,0,43.8,7,3086,The agent has not yet selected the destination or the date; and no flight options are visible.