Open the Bot's screen only when there is something to watch, and offer an unconnected app in the conversation

[guidovizoso]

What this changes

Asking a Bot whether it could use Notion, before any Notion tool was granted, opened its computer beside the conversation — on a still screen, for an action that never happened. Three things behind that, and the third is the reply the Bot should have given instead.

• A call that never reaches the screen no longer opens it. callComputer announced the live screen before every request. /files/list, /files/read, /files/write and /exec happen on the computer rather than on its display, and announcing them opened a pane on whatever still page was already there. The off-screen set is a list of exceptions, not of permits: a browser path added later and never listed goes on announcing, so forgetting the list opens a pane rather than silently stopping one.
• A refused action takes its screen back. The announcement goes out before the request on purpose — a page is worth watching while it loads — but a 403 is a policy, a capability switch or a network rule saying no before the browser is touched. Retraction is per announcement rather than per run: a Bot that opens a page and then has a click refused made both calls in one run, and closing on the run would shut a pane showing the page the first call loaded. An action that was permitted and failed keeps its screen, because the page may have moved. The route's effect moved into useAutoWatchScreen so these rules can be stated in tests without a router, a channel and a transcript around them.
• A Bot is no longer offered computer tools this deployment would refuse. useComputerAvailable only ever asked whose computer it is. It never consulted the capability switches, so with cloudBrowser off the model was still handed computer_navigate, reached for it, was refused, and spent the turn saying so. computerToolsOffered mirrors the server's own split in decideFromSnapshot, which is by intent rather than by tool: files and commands need cloudComputer, everything else needs cloudBrowser, opening a page needs cloudNetwork on top. Unresolved reads as allowed in both halves — the server decides; this only declines to offer what it would certainly refuse. report_refusal stays on the ownership question alone, since recording that a Bot declined something is audit bookkeeping rather than a computer action.
• An app you have not connected is offered where you are reading. The refusal was "Connect it in Settings and ask again" — a correct sentence, and a bad one to end on. It now carries a connect prompt (shared/connect-prompt.ts, declared once and read from both sides for the reason shared/handoff-markers.ts sets out), and the transcript draws a card naming the app with a Connect button. tool-activity keeps it out of the fold for the same reason a consent card is kept out. Four refusals carry it: never connected (brokered and user-OAuth alike), disconnected mid-call, and access withdrawn. A Shared app's deployment account deliberately does not — the person reading cannot connect it however clearly they are asked.

Connect opens the vendor in a new tab rather than navigating this one. A consent screen is a full page load, so sending this tab would replace the conversation and land somebody on the connected-accounts page with their question behind them. This tab stays put and watches this person's own connections, and says connected only when the server lists one — never because somebody came back, since the return trip is an unsigned redirect in a tab this one cannot read. That also meant leaving the sealed OAuth returnTo state alone, which is a closed two-name set for reasons documented at length in plugins/oauth.ts.

Not covered: Gmail still has no native catalogue entry, so "what emails did I get" reaches the card through the Composio broker rather than a first-party entry.

Verified

• Formatter, lint and bun run typecheck (all five workspaces) clean; bun run build succeeds.
• App and shared: 1305 pass, 0 fail (30 new tests across six files).
• Server, the files this touches — plugin-store.integration, plugin-user-credential.integration, plugin-connect-route, plugin-oauth, shared-use-store.integration, shared-brokered-store.integration, composio-adapter, composio-connections: 534 pass, 0 fail.
• A new server test asserts the refusal actually carries a parseable prompt naming the right server. That string is the only thing telling the card which app to connect, so without it the feature degrades silently: the English still reads correctly and the button quietly stops being offered.
• The whole server/tests suite run in parallel shows Failed query: failures against the shared openbot_test database; composio-connections.test.ts fails there and passes alone. That is contention on a database other local sessions share, not these changes.

Where it runs

• New state that outlives a request? None. The announcement epoch and the pane's own bookkeeping are per browser tab and die with it; the capability read is a cached query of an existing endpoint.
• What happens on the second replica? The same thing. GET /api/admin/enterprise/me and GET /api/plugins/connections both answer from Postgres for whoever is asking, and the refusals are built per call from the same rows the boundary already read.
• Anything serialised? No. Nothing here writes. Connecting an account is the existing /connect route, unchanged.
• Anything fanned out to a browser? No. The screen announcement is in-tab only — one module, one set of listeners, same document.
• New listener, port, or schedule? No. The card polls /api/plugins/connections every two seconds, and only while a consent is open in another tab; otherwise it is an ordinary cached read.

Boundary and audit

• Every acting call still goes through the gateway. Nothing moved out of PluginStore.callTool or past the computer boundary — the capability gate withholds tools in front of a decision the server still makes for itself.
• New refusals and failures each write a row. The refusals here are the existing ones with a marker added; none is new, and the audit rows are unchanged.
• Nothing new is trusted from the client. The serverId in the card comes from the server's own refusal, and pressing Connect re-resolves it server-side as it always did.

Changelog

• Two entries under Unreleased: "The Bot's screen opens only when there is something to watch" and "An app you have not connected is offered where you are reading".

Proof

The screen-opening rules are pinned by tests rather than by a screenshot, since what changed is when a pane appears:

• app/tests/computer-screen-announcement.test.ts — a browser action announces; a failed one still announces because the page may have moved; a refused one withdraws; a refusal does not withdraw a sibling's announcement in the same run; /files/* and /exec never announce; an unrecognised path still does.
• app/tests/auto-watch-screen.test.tsx — opens on activity, does not take a pane opened for something else, does not reopen what is being watched, closes only on the announcement that opened it, and honours a dismissal for the rest of the run.
• app/tests/computer-tools-offered.test.ts — each switch withdraws exactly its own tools, and unresolved offers everything.
• app/tests/connect-app-card.test.tsx — asserts the card never navigates this tab, and that "connected" follows the server listing the account rather than the person returning.

I have not re-run the original conversation against a live deployment; the local instance I checked has no activity since 14 Oct.

Review follow-ups

Three P2 findings on the connect card, all confirmed against the code and fixed:

• Connect did nothing for an app that takes a typed key. POST /connect answers { fields } for those, so the mutation unwrapped nothing and the card returned silently. It now says the app is connected with a key and links to the page holding the form.
• A withdrawn credential read as connected. connectionsFor selected from the join table alone, so a revoked credential still listed a connection — the card drew "connected" over the very refusal saying access was withdrawn, removing the only way back. Root-caused on the server: the listing now reports revoked, additively, so the apps-ever-connected readers and memory ingestion keep their behaviour. The join is inner and total because credential_id is a real non-cascading foreign key, which is also why there is no "credential is gone" state; a test pins that, since the field's completeness rests on it.
• The OAuth tab was blocked on a slow request. window.open ran after the network call, by which time Chrome's transient activation (~5s) could have expired. The blank tab is claimed on the click itself and navigated when the url arrives. noopener can't be passed — it makes window.open answer null, and null is a handle nothing can navigate later — so the opener is severed on the handle instead. A tab blocked anyway leaves the url as a link to click.

Card tests went from 4 to 9, covering the claim-on-click ordering, the blocked-tab link, the key-entry link and its target, tab cleanup on failure, and the revoked row. The test harness now mounts a real memory router, since the card links to a connector's settings page.