From b1d99606de515621a2e878036e6c358b6ea1024a Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?=C3=81lvaro=20Ortiz?= Date: Mon, 16 Nov 2026 23:21:50 +0100 Subject: [PATCH] Allow benchup.io in the bridge and record the live agent round The GitHub Pages address of the web app now redirects to https://benchup.io, so the published app connects to the bridge from that origin. The bridge only allowed https://benchup3d.github.io and answered it with HTTP 403, which would have kept the published app from ever pairing. Both origins are now allowed, with a test. The agent evaluation can now run against the project open in BenchUp: setup.py --live --token writes prompts that use `benchup ... --live`, and gives new cases an empty project to open. Round 7 reran cases 01-12 this way on the desktop and in Chrome, for item 3 of #55; Safari and the local network access prompt from an https page are still to be checked. --- crates/benchup-cli/evals/README.md | 2 + crates/benchup-cli/evals/prompt.md | 2 +- .../benchup-cli/evals/results/round-7-live.md | 76 +++++++++++++++++++ crates/benchup-cli/evals/setup.py | 45 +++++++++-- crates/benchup-cli/src/bridge.rs | 5 +- crates/benchup-cli/src/bridge_tests.rs | 4 + docs/cli.md | 2 +- 7 files changed, 127 insertions(+), 9 deletions(-) create mode 100644 crates/benchup-cli/evals/results/round-7-live.md diff --git a/crates/benchup-cli/evals/README.md b/crates/benchup-cli/evals/README.md index 48b8161..f928548 100644 --- a/crates/benchup-cli/evals/README.md +++ b/crates/benchup-cli/evals/README.md @@ -13,6 +13,8 @@ There are two layers: # give each agent the prompt in /tmp/benchup-eval5/prompts/.md, verbatim ``` + To run the cases against the project open in BenchUp (desktop or web) instead of a file, add `--live --token ` (and `--port` if BenchUp uses another one). Open each case's `file` in BenchUp before starting its agent, and run the cases one at a time, since one BenchUp serves one bridge. [Round 7](results/round-7-live.md) was run this way. + ## Files - `cases.json` — the cases: request, seed project, mode (`cli`, or `mcp` through `mcp_call.py`), the tools it is expected to exercise, and the reference solution with what `check_model` must say about it. diff --git a/crates/benchup-cli/evals/prompt.md b/crates/benchup-cli/evals/prompt.md index 207e6c5..ee0a44f 100644 --- a/crates/benchup-cli/evals/prompt.md +++ b/crates/benchup-cli/evals/prompt.md @@ -4,7 +4,7 @@ USER REQUEST: "{request}" SETUP AND RULES -- Your working folder is {dir} . Work only there. Start EVERY shell command with: export PATH={bin}:$PATH && cd {dir} && +- Your working folder is {dir} . Work only there. Start EVERY shell command with: export {exports} && cd {dir} && {access} - Do NOT read any source code, README, docs or file of the BenchUp repository, ~/.cargo, or other evaluation folders ({root}), or anywhere else about benchup. No web searches. A real user would not have them. - Do NOT run `benchup open` (several evaluations run at once and would fight over the desktop app). Everything else is allowed. diff --git a/crates/benchup-cli/evals/results/round-7-live.md b/crates/benchup-cli/evals/results/round-7-live.md new file mode 100644 index 0000000..77d0617 --- /dev/null +++ b/crates/benchup-cli/evals/results/round-7-live.md @@ -0,0 +1,76 @@ +# Round 7, live (16 November 2026) + +This round is item 3 of [#55](https://github.com/BenchUp3D/benchup/issues/55): the twelve original cases (01–12) rerun against the project open in BenchUp instead of a file, through `benchup … --live` and `benchup bridge`. Each agent got the usual prompt, prepared with `setup.py --live`, so it learned the tools only from `benchup --help`, the guide and the tool output, and was told the project was open in the app. The requests were not changed. + +The cases ran one at a time on each surface, because one BenchUp serves one bridge: + +- **Desktop:** a `dev-desktop` build of `main` on macOS, started for each case with its own `BENCHUP_DATA_DIR`, the agent connection turned on and the case's file passed on the command line. +- **Web, Chrome 157:** the same commit served by `trunk serve` on `http://127.0.0.1:8090`, with its own bridge port (47616) and token. Each case's file was opened through File → Open. +- **Web, Safari:** not run yet (see below). + +Each cell is the number of `benchup` commands, with those that returned an error in brackets. Live counts include two to four `agent_activity` calls per case, which file mode does not have. Every result was checked independently afterwards with `check_model` and an iso render taken live from the app. + +| # | Case | R5 (file) | R7 desktop | R7 Chrome | Result in R7 (both surfaces) | +|---|---|---|---|---|---| +| 01 | Lengthen the desk | 23 (1) | 29 (0) | 32 (0) | ✅ lengthened at the front, top kept full width past the post | +| 02 | Five shelves to four | 17 (0) | 21 (0) | 20 (0) | ✅ gaps of 345.6 mm, joints flipped in a separate batch | +| 03 | Dining table | 18 (0) | 20 (1) | 21 (0) | ✅ 8/8 joints | +| 04 | Chest of drawers | 20 (0) | 27 (1) | 24 (0) | ✅ 24/24 joints, one component | +| 05 | Back in a rabbet and plinth | 43 (2) | 21 (0) | 20 (0) | ✅ routed rabbet, back whole, plinth in two halves | +| 06 | Solid oak 19 mm | 25 (3) | 24 (0) | 20 (0) | ✅ `change_thickness` without workarounds | +| 07 | Bedside table with curved sides | 16 (0) | 22 (1) | 22 (1) | ✅ 6/6 joints, 2 profiles | +| 08 | Leaning ladder shelf | 27 (1) | 34 (0) | 26 (0) | ✅ 8/8 stopped housings | +| 09 | Workshop dimensions | 18 (0) | 13 (0) | 16 (0) | ✅ the ten reversed joints of the seed are reported, not touched | +| 10 | Modules, components and tags | 25 (2) | 27 (1) | 27 (0) | ⚠️ partial on purpose, alternative saved, as in every round | +| 11 | Cut list and sheets | 12 (0) | 13 (0) | 15 (0) | ✅ 2 sheets, dry run only, project unchanged | +| 12 | A shelf that does not fit | 14 (0) | 15 (0) | 12 (0) | ✅ refused, 491 mm alternative saved | +| | **Total** | **258 (9)** | **266 (4)** | **255 (1)** | 11 ✅, 1 ⚠️ on each surface | + +Live mode behaved like file mode. No agent hit a bridge or connection error, and no call timed out. The extra commands are the `agent_activity` reports and, on the desktop, one agent (08) that placed its dimensions three times. The error in 10 on the desktop was a deliberate trial. + +## What was checked about live mode + +- **Undo:** a batch is one step in BenchUp, for example `Agent: push_pull, split_face, merge_faces, …`, and `benchup undo --live` / `redo --live` step through it. +- **Dry runs:** `call --live --dry-run` leaves the open model and its undo history untouched (checked by counting objects before and after, and by undoing the previous step). Agents in 02, 04 and 11 relied on this. +- **Reconnection:** in Chrome, `trunk` reloaded the page right after case 02. BenchUp restored the project from local storage and connected to the bridge again by itself; the change was kept. +- **Opening another project:** on the web, File → Open while the agent's changes were unsaved showed the unsaved-changes modal (Discard changes, Save, Cancel), as for any other change. +- **Status:** the web status bar showed "Agente conectado" (the app was in Spanish) while the bridge served it. + +Not covered here: the grant/deny prompt and Take control (they need someone drawing in the app at the time), shared projects and leases (they need the server). + +## Transport from the published web app + +- **Origin:** `https://benchup3d.github.io/benchup/` now redirects to `https://benchup.io`, so the published app loads from that origin. The bridge only allowed `https://benchup3d.github.io` and answered the published app with HTTP 403. Fixed in this branch: the bridge allows both origins (`bridge.rs`, with a test). +- **Chrome 157:** a WebSocket from `https://benchup.io` to `ws://127.0.0.1` waits for the local network access permission (`local-network-access` is `prompt`). Nobody answered the prompt in this run; after about 240 s the socket failed with code 1006. Still to do: allow the prompt and confirm the connection opens, then check that BenchUp says what is happening while the prompt waits. +- **Safari:** still to do. The question from #73 is whether Safari lets an `https` page open `ws://127.0.0.1`. The local build on `http://127.0.0.1` does not answer it. + +## Findings + +### Tool bugs and unclear output (benchup-tools / CLI) + +1. **`agent_activity` is missing from `benchup tools`**, and `benchup tools agent_activity` fails with "unknown tool agent_activity", although the guide tells agents to call it in live mode (03, 04, 07 and others). +2. **`agent_activity` inside a `--batch` fails** with "unknown tool agent_activity" and aborts the batch; the guide's "use `{"phase":"preparing"}` while preparing changes" made an agent put it there, and it then had to renumber every `$N` (07, desktop). +3. **`add_joint` accepts a `tongue_groove` whose `a` runs right through `b`**, and `check_model` then names only one of the two boards the shelf passes through (12, desktop). +4. **`flip_joint` accepts `{"id": …}`** although its schema only has `ids` and `additionalProperties: false`, without a warning (10, Chrome). +5. **`check_model only_new`** still reports existing issues as new when the boards are recreated (10, desktop; the `group` case from round 5 is fixed). +6. **`push_pull` on a face that borders a notch** stretches the neighbouring faces and returns `created: []`; it also returns both `face_id` and `new_face_id` with the same value (01). +7. **`change_thickness`** reports an intermediate `length_mm` (481.5) in `joint_adjustments` and the final one (481) in `butt_adjustments` (06, Chrome). +8. **`add_dimension` with `axis`** anchors the line at the midpoint of its points when they differ off the axis, so offsets need to be compensated by hand; ends attach only when given to about five decimals on rotated geometry (08). +9. **`render`** clips long text notes at the image edge without saying so (03, 04, 08), and lists edge-on dimensions of other planes as overlaps (09). +10. **`add_profile` and `get_object`** do not say which face received `width_mm` and which `depth_mm` (05, both surfaces, as in round 6). + +### Guide gaps (docs/agent-guide.md) + +- Say that `agent_activity` is a live-only call that is not listed by `benchup tools` and must not go in a batch, and give its phases. +- Say that dry-run ids match the real run in live mode; the guide says they differ (04). +- The "Lengthening a piece" recipe should cover an end face next to a notch: push, split the new side face at the notch line, push the piece out (01, both surfaces). +- The `rabbet` row of the joint table still reads as a rabbet but cuts a tongue in a slot; lead with the routed profile (05). +- The drawer recipe should say how drawers are supported (runners, rails) and give the gap arithmetic for N equal drawers (04). +- The plinth recipe assumes a bottom panel; the bookcase has none (05). +- Organising an existing model: parts shared between modules, and the order of tag, rename and group calls since grouping changes ids (10). +- `benchup guide` is about 29 KB, and every agent had to read it from a saved file; a table of contents or `benchup guide
` would help. + +### Live-mode setup + +- The pairing token is the only thing an agent needs besides the command, and every agent found the `--live` mode from `benchup --help` alone. +- The project name in results is the file stem (`table`), not a title (`Table`); agents noticed but were not confused. diff --git a/crates/benchup-cli/evals/setup.py b/crates/benchup-cli/evals/setup.py index 9f71d7d..1133ba1 100755 --- a/crates/benchup-cli/evals/setup.py +++ b/crates/benchup-cli/evals/setup.py @@ -2,13 +2,19 @@ """Prepare a round of the agent evaluation (see README.md). python3 crates/benchup-cli/evals/setup.py --round 5 [--cases 01,13] [--binary target/release/benchup] + python3 crates/benchup-cli/evals/setup.py --round 7 --live --token TOKEN [--port 47615] [--cases 01,13] Creates /tmp/benchup-eval/ with the binary in bin/, one folder per case with its seed project, and prompts/.md with the exact prompt to -give each agent. Prints a JSON list of {id, dir, prompt}. Run it from the +give each agent. Prints a JSON list of {id, dir, prompt, file}. Run it from the repository root; it never builds anything. + +With --live, the agents edit the project open in BenchUp through +`benchup ... --live` instead of the file: open each case's `file` in BenchUp +(new cases get an empty project to open) and run the cases one at a time, +since one BenchUp serves one bridge. The requests stay the same. """ -import argparse, json, os, shutil, sys +import argparse, json, os, shlex, shutil, subprocess, sys from pathlib import Path HERE = Path(__file__).resolve().parent @@ -19,6 +25,11 @@ "`python3 {bin}/mcp_call.py FILE list` (tools and schemas), `... FILE call TOOL 'JSON'` and " "`... FILE render VIEW OUT.png`. Start with the server's get_guide tool. Do NOT run the `benchup` " "command directly.") +LIVE_ACCESS = ("- The project is open in the BenchUp app on this computer: {opened}. Edit it there, live, with " + "`benchup` commands that take `--live` instead of a file; the pairing token and port are already set by " + "the command prefix. Do not create, read or write .bup files on disk, and do not start or stop a bridge " + "yourself. Learn the CLI ONLY from `benchup --help`, what the help points you to (e.g. `benchup guide`, " + "`benchup tools`, `benchup tools `) and the outputs/errors it gives you.") def main(): @@ -27,7 +38,15 @@ def main(): parser.add_argument("--cases", help="comma-separated case number prefixes, e.g. 01,13 (default: all)") parser.add_argument("--binary", default=str(ROOT / "target/release/benchup")) parser.add_argument("--out", help="default /tmp/benchup-eval") + parser.add_argument("--live", action="store_true", help="agents edit the project open in BenchUp (--live)") + parser.add_argument("--token", help="with --live: the pairing token shown in BenchUp") + parser.add_argument("--port", type=int, help="with --live: the bridge port set in BenchUp (default 47615)") args = parser.parse_args() + if args.live and not args.token: + sys.exit("--live needs --token, the pairing token from BenchUp (Settings → Agent connection)") + if args.live and any(c["mode"] == "mcp" for c in json.loads((HERE / "cases.json").read_text())["cases"] + if not args.cases or any(c["id"].startswith(w) for w in args.cases.split(","))): + sys.exit("--live runs CLI cases only; leave MCP cases out with --cases") binary = Path(args.binary) if not binary.is_file(): sys.exit(f"no benchup binary at {binary}; build it with `cargo build --release -p benchup-cli` first") @@ -49,17 +68,31 @@ def main(): seed = manifest["seeds"][case["seed"]] if seed: shutil.copy2(ROOT / seed, folder / case["file"]) + elif args.live: + # BenchUp needs a project to open; the request still asks to create it. + subprocess.run([str(out / "bin/benchup"), "new", str(folder / case["file"]), + "--name", Path(case["file"]).stem.capitalize()], check=True, capture_output=True) mcp = case["mode"] == "mcp" + exports = f"PATH={out / 'bin'}:$PATH" + if args.live: + exports += f" BENCHUP_AGENT_TOKEN={shlex.quote(args.token)}" + if args.port: + exports += f" BENCHUP_BRIDGE_PORT={args.port}" + opened = (f"`{case['file']}`" if seed else + f"an empty project named `{Path(case['file']).stem.capitalize()}`, standing for `{case['file']}`") + access = LIVE_ACCESS.format(opened=opened) + else: + access = (MCP_ACCESS if mcp else CLI_ACCESS).format(bin=out / "bin") prompt = template.format( request=case["request"], dir=folder, bin=out / "bin", root=out.parent / "benchup-eval*", - access=(MCP_ACCESS if mcp else CLI_ACCESS).format(bin=out / "bin"), - unit="MCP tool calls" if mcp else "benchup commands", + access=access, exports=exports, unit="MCP tool calls" if mcp else "benchup commands", ) - if not seed: + if not seed and not args.live: prompt = prompt.replace("Work only there.", "Work only there. The project does not exist yet; create it there.") path = out / "prompts" / f"{case['id']}.md" path.write_text(prompt) - listing.append({"id": case["id"], "title": case["title"], "dir": str(folder), "prompt": str(path)}) + listing.append({"id": case["id"], "title": case["title"], "dir": str(folder), "prompt": str(path), + "file": str(folder / case["file"])}) print(json.dumps(listing, indent=1, ensure_ascii=False)) diff --git a/crates/benchup-cli/src/bridge.rs b/crates/benchup-cli/src/bridge.rs index 4e74a3f..7eefa3c 100644 --- a/crates/benchup-cli/src/bridge.rs +++ b/crates/benchup-cli/src/bridge.rs @@ -54,6 +54,9 @@ use tungstenite::{ pub const DEFAULT_PORT: u16 = pairing::DEFAULT_PORT; /// The published web app. Local development servers are always allowed too. pub const DEFAULT_ORIGIN: &str = "https://benchup3d.github.io"; +/// Origins that serve the published web app: the GitHub Pages address +/// redirects to the custom domain, so pages actually load from the latter. +pub const PUBLISHED_ORIGINS: [&str; 2] = [DEFAULT_ORIGIN, "https://benchup.io"]; pub const TOKEN_ENV: &str = "BENCHUP_AGENT_TOKEN"; pub const PORT_ENV: &str = "BENCHUP_BRIDGE_PORT"; pub const MIN_TOKEN_LEN: usize = 16; @@ -152,7 +155,7 @@ fn now_ms() -> u64 { /// `--allow-origin`. pub fn origin_allowed(origin: Option<&str>, extra: &[String]) -> bool { let Some(origin) = origin else { return true }; - if origin == DEFAULT_ORIGIN || extra.iter().any(|o| o == origin) { + if PUBLISHED_ORIGINS.contains(&origin) || extra.iter().any(|o| o == origin) { return true; } ["http://localhost", "http://127.0.0.1"].iter().any(|base| { diff --git a/crates/benchup-cli/src/bridge_tests.rs b/crates/benchup-cli/src/bridge_tests.rs index a97210f..6f46a21 100644 --- a/crates/benchup-cli/src/bridge_tests.rs +++ b/crates/benchup-cli/src/bridge_tests.rs @@ -81,6 +81,10 @@ fn echo_app(hub: &Hub) -> std::thread::JoinHandle<()> { fn origins_are_limited_to_the_web_app_and_local_servers() { assert!(origin_allowed(None, &[])); assert!(origin_allowed(Some(DEFAULT_ORIGIN), &[])); + // The GitHub Pages address redirects there, so the app loads from it. + assert!(origin_allowed(Some("https://benchup.io"), &[])); + assert!(!origin_allowed(Some("https://benchup.io.evil.example"), &[])); + assert!(!origin_allowed(Some("http://benchup.io"), &[])); assert!(origin_allowed(Some("http://localhost:8080"), &[])); assert!(origin_allowed(Some("http://127.0.0.1"), &[])); assert!(!origin_allowed(Some("https://evil.example"), &[])); diff --git a/docs/cli.md b/docs/cli.md index 6a179ce..fd31d2d 100644 --- a/docs/cli.md +++ b/docs/cli.md @@ -131,7 +131,7 @@ How it behaves: - **Scope and selection:** every call starts in the scope the designer is in, so put `enter_object` and the calls that need it in one batch. The agent sees the designer's selection (`get_selection`), so the designer can say "change this". - **Project switches:** every result names the open project. A call that was waiting when another project was opened is refused, so the agent can look again with `get_scene`. A collaborator's new revision of the same shared project does not refuse waiting calls. - **Timeouts and size:** a call waits up to 15 seconds for BenchUp to connect and up to 120 seconds for its answer, then fails with a hint. A call larger than 16 MB is refused by the bridge; split it into smaller batches. An answer larger than 32 MB comes back as an error (with `applied: true` when the call still changed the model); filter `get_scene` or render a smaller view. -- **Security:** only programs that know the token can use the bridge, and the token is never sent: both sides prove they know it with a challenge-response (HMAC-SHA256 over fresh nonces). Browsers must come from `https://benchup3d.github.io` or a local development server (`--allow-origin` adds another), so other web pages cannot reach it. Pass the token in the environment, never as a command-line argument, which other users on the computer can see. +- **Security:** only programs that know the token can use the bridge, and the token is never sent: both sides prove they know it with a challenge-response (HMAC-SHA256 over fresh nonces). Browsers must come from the published web app (`https://benchup3d.github.io`, which redirects to `https://benchup.io`) or a local development server (`--allow-origin` adds another), so other web pages cannot reach it. Pass the token in the environment, never as a command-line argument, which other users on the computer can see. - **Web browsers:** Chrome and Safari may ask whether the site may connect to devices on your local network the first time BenchUp connects to the bridge; allow it. ## Using it from an agent