Skip to content

Live bridge and in-app Agent connection #55

Open

Description

@furilo

Follow-up to #13, #40 and #41. Agents can now edit .bup files with the benchup CLI. The next step is letting them edit the project open in the app, on both web and desktop, without a central server.

Approach

agent ──CLI / MCP──▶ benchup bridge (127.0.0.1)  ◀──WebSocket── BenchUp (web or desktop)

A browser tab cannot listen on a port, so the app connects out to a local bridge. This reuses the bridge code from #3 (live.rs, protocol.rs, pairing.rs) on top of what is now on main.

Plan

  • 1. benchup bridge: a subcommand of the existing CLI, not a separate benchup-mcp binary.
    • It listens on 127.0.0.1 only, uses challenge-response pairing, has an Origin allowlist, and gives calls deadlines.
    • It serves MCP, and also lets the existing commands target the open app, e.g. benchup call --live create_board '{...}' and benchup scene --live. CLI-first agents then work the same way, and docs/agent-guide.md covers both modes.
  • 2. Agent connection in the app, in its own PR.
    • It is off by default, its setting persists, and pairing uses a one-time token shown in the app.
    • Each call is one undo step (Agent: …), and a --batch counts as one step.
    • It respects the read-only state of shared projects and their leases.
    • All text is translated and the dialogs use the shared modal.
    • The generic add-on system from Add an MCP server and optional agent add-ons (Agent connection, Build with MIRA) #3 stays out unless we decide we want it.
  • 3. Validation: rerun the 12 agent scenarios from Add benchup, a CLI and MCP server for editing .bup projects #41 against the live app, on desktop and on the web build in Chrome and Safari (local network access prompt).
  • 4. Later, separate: MIRA and other in-app chat ideas, built on the same tool layer.

Open questions

  • What should happen when an agent operation arrives while the user is in the middle of a tool: queue it, reject it, or apply it?
  • How should agent edits interact with real-time collaboration on shared projects?
  • Who takes what? @aarvee11 wrote the bridge. Would you like to port 1 and 2, or split them?

Meanwhile, #46 (reload on external change) gives desktop users a near-live loop with the CLI.

Activity

  1. dannycoulombe commented on Nov 14, 2026

    @dannycoulombe
    Collaborator
    • What should happen when an agent operation arrives while the user is in the middle of a tool: queue it, reject it, or apply it?

    I'm thinking of a "Grant access" (or "Deny access") prompt that would appear only when the user is actively drawing or editing something. Granting access would cancel the user's current operation and hand control over to the AI. Conversely, the user could request control from the AI when trying to do something while it's performing operations.

    • How should agent edits interact with real-time collaboration on shared projects?

    I don't think an agent should be treated any differently from a real user.

  2. moved this from Backlog to Ready in Roadmapon Nov 14, 2026
  3. self-assigned this
    on Nov 14, 2026
  4. furilo commented on Nov 14, 2026

    @furilo
    CollaboratorAuthor

    On it.

  5. added 2 commits that reference this issue on Nov 16, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

No labels
No labels

Type

No type

Projects

  • Status
    Ready

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions