Skip to content

Live bridge and in-app agent connection (#55) - #73

Merged
dannycoulombe merged 8 commits into
BenchUp3D:mainfrom
furilo:implement-benchup-issue-55
Nov 16, 2026
Merged

dannycoulombe merged 8 commits into
BenchUp3D:mainfrom
furilo:implement-benchup-issue-55

Conversation

@furilo

@furilo furilo commented Nov 15, 2026

Copy link
Copy Markdown
Collaborator

Implements items 1 and 2 of #55: agents can edit the project open in BenchUp, on the desktop and on the web, without a central server. It ports the bridge from #3 (live.rs, protocol.rs, pairing.rs) onto current main, as a subcommand of the existing CLI rather than a separate benchup-mcp binary. The add-on system and MIRA from #3 are left out.

agent ──MCP (stdio) / benchup … --live──▶ benchup bridge (127.0.0.1:47615) ◀──WebSocket── BenchUp (web or desktop)

1. benchup bridge

  • Listens on 127.0.0.1 only. Pairing is a challenge-response with HMAC-SHA256 over fresh nonces from both sides, so the token is never sent; proofs are separated by role (app, client, bridge).
  • Browsers must come from https://benchup3d.github.io or a local dev server (--allow-origin adds more); anything else gets HTTP 403.
  • Limits before authentication (5 s, 1 MiB, 8 pending connections), size limits on calls and results, and deadlines: a call waits up to 15 s for BenchUp to connect and 120 s for the answer, then is cancelled.
  • One BenchUp window per bridge: a second one is refused instead of replacing the first.
  • Serves MCP on stdio. If another bridge holds the port it forwards to it, and takes the port over if that bridge exits.
  • scene, get, call (--batch, --dry-run), undo, redo and render take --live. The first one starts a background bridge, which exits after 30 minutes without calls. The token comes only from BENCHUP_AGENT_TOKEN, never from arguments.
  • Batch $N references and reports move from the CLI to benchup_tools::batch, so files, MCP and the app answer the same way.

2. Agent connection in the app

  • Settings → Agent connection: off by default; the setting and port persist (GeneralSettings, with older preferences loading as off). The pairing token is generated by the app and kept in this device's private storage (0600 file on the desktop, localStorage on the web), with buttons to copy the token, the Claude Code command, an MCP config or shell lines.
  • Each changing call is one undo step, Agent: <tool>, and a batch is one step. Edits go through the same commit path as the designer's, so shared projects keep their read-only state and leases, and the agent is treated like any other user (as suggested in the issue). An edit waits for a selection lease on its way to the server instead of failing.
  • Open question 1, following the suggestion in the issue: reading calls always run. A changing call runs at once unless the designer is mid-operation (drawing, push/pull, typing a value, renaming…) or has taken control; then a shared modal asks Deny access / Grant access. Granting cancels their operation and lets the agent continue until it goes quiet; denying returns an error the agent can read. While the agent edits, the status bar shows Take control.
  • Calls run from App::logic, so a minimized window or hidden tab still answers. Renders run off the UI thread on the desktop. A call queued before another project was opened is refused.
  • eframe's clipboard feature is enabled: without it, copied text (the token, but also geometry and text fields) only reached BenchUp's internal clipboard on the desktop.
  • All new text is translated in every catalog. docs/cli.md, docs/agent-guide.md and the skill cover live mode.

Testing

  • New tests: bridge pairing, origins, timeouts, forwarding and takeover; CLI --live against a stand-in app; app handshake, undo steps, batches and dry runs, grant/deny/take control, shared read-only and leases (including an agent selection requesting its lease), result size limits, token storage, preference persistence.
  • cargo test --locked -p benchup-core -p benchup-tools -p benchup-cli passes. In benchup-app --lib, the only failures are 11 that already fail on a9d9c65 (Chinese strings, the dimension_input_overlay render audit, move/copy arrays…), unrelated to this change.
  • Checked by hand on desktop: CLI --live calls, MCP, undo steps, the grant/deny prompt and Take control. The web build compiles for wasm32 with no warnings and is served by trunk serve.

Not done yet

  • Item 3: rerun the 12 agent scenarios from Add benchup, a CLI and MCP server for editing .bup projects #41 against the live app on desktop and on the web in Chrome and Safari, including the local network access prompt. Safari in particular may not allow ws://127.0.0.1 from an https page.
  • Item 4 (MIRA and in-app chat) stays separate, as planned.
  • Distribution: users must install the benchup command to use the bridge, which today means cargo install --path crates/benchup-cli. Shipping it with the desktop packages would be a follow-up.

Closes items 1 and 2 of #55.

🤖 Generated with Claude Code

furilo and others added 6 commits November 15, 2026 08:38
Agents can now edit the project open in BenchUp, on the desktop and on
the web, without a central server. BenchUp connects out to a local
bridge, because a browser tab cannot listen on a port.

benchup bridge (CLI):
- Listens on 127.0.0.1 only, with challenge-response pairing
  (HMAC-SHA256; the token is never sent), an Origin allowlist,
  pre-authentication limits and call deadlines. One BenchUp window
  per bridge.
- Serves MCP on stdio. When another bridge holds the port it forwards
  to it, and takes the port over if that bridge exits.
- scene, get, call (--batch, --dry-run), undo, redo and render take
  --live. The first one starts a background bridge, which exits after
  30 minutes without calls.
- Batch references and reports move to benchup_tools::batch, so files,
  MCP and the app answer the same way.

Agent connection (app):
- Settings -> Agent connection: off by default; the setting and the
  port persist. The pairing token stays in this device's private
  storage, with copy buttons for the token and setup snippets.
- Each changing call, and each batch, is one undo step "Agent: ...",
  recorded like the designer's own edits, so shared projects keep their
  read-only state and edit leases.
- Reads always run. A changing call asks the designer to grant or deny
  access when they are mid-operation or have taken control; granting
  cancels their operation. The status bar offers Take control.
- Calls run from App::logic, so a minimized window or hidden tab still
  answers. Renders run off the UI thread on the desktop.
- eframe's clipboard feature is enabled so copied text reaches the
  system clipboard on the desktop.
- All text is translated in every catalog; docs and the skill cover
  live mode.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@dannycoulombe

dannycoulombe commented Nov 16, 2026 •

Copy link
Copy Markdown
Collaborator

Doing more tests, but meanwhile I couldn't resist:

image

@furilo

furilo commented Nov 16, 2026 via email

Copy link
Copy Markdown
Collaborator Author

@dannycoulombe

Copy link
Copy Markdown
Collaborator
untitled.mp4

@dannycoulombe
dannycoulombe merged commit 664756d into BenchUp3D:main Nov 16, 2026
2 checks passed
@dannycoulombe

Copy link
Copy Markdown
Collaborator

The CLI/MCP is such a nice feature! I spent a good hour playing around with it yesterday.

  • I was able to test it with Codex and build all sorts of little things, both from scratch and using existing projects. Really cool!
  • I was able to test the grant/deny access mechanism and collaborate with the AI while it was working.

As you saw in the video, I took the initiative to make the process a little more visible directly in the viewport, without making it too overwhelming. If you think it's too much, feel free to adjust!

Alright, I'm merging this!

Awesome work!

@furilo

furilo commented Nov 16, 2026 via email

Copy link
Copy Markdown
Collaborator Author

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants