Repository navigation
Conversation
Adds a guest namespace desktop.* (list_apps, get_app_state, click, set_value, type_text, press_key, scroll, drag, select_text, perform_secondary_action, run, status) backed by the signed codex-computer-use-mcp broker. - One broker session per server process so element indexes survive across execute_code calls; the CLI closes it on exit. - Screenshots are metadata unless requested; desktop.run batches steps and stops at the first failure or missing expect text. - Off by default (config.desktop.enabled). Optional disclaimLauncher works around macOS making the host app the responsible process. - Contract doc structure/desktop-surface.md, catalog and coverage rows, tests.
Codex Review SummaryThis comment shows the latest Codex review activity on this pull request.
ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings. |
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 0a63df9c50
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| const ex = await getExecutor(); | ||
| const raw = await ex.execute(name, args, { signal }); |
There was a problem hiding this comment.
When an MCP request is cancelled or execute_code reaches its deadline, the desktop operation is not cancelled: production constructs the singleton with createDesktop({ config }), so this closure's signal is always undefined. A click, keypress, or text entry can therefore execute after the caller has received a cancellation or timeout; on the CLI path, closing the executor may also wait behind that uncancelled operation. Preserve the shared executor while forwarding each execution's abort signal to its calls.
Useful? React with 👍 / 👎.
| if (executor) return executor; | ||
| const launcher = caps.disclaimLauncher || env.CU_DISCLAIM_LAUNCHER; | ||
| if (launcher) process.env.CU_DISCLAIM_LAUNCHER = launcher; | ||
| broker = await loader(caps.packageDir); | ||
| executor = new broker.DirectSessionExecutor({ idleTimeoutMs: caps.idleMs ?? 120000 }); |
There was a problem hiding this comment.
When the first two desktop calls overlap—through Promise.all or concurrent MCP requests—both can observe executor === null, await the loader, and construct separate broker sessions. The last assignment wins, so an earlier get_app_state can return element indexes from the discarded session while a subsequent click uses the retained session, defeating the cross-call index guarantee and potentially targeting the wrong element. Memoize the initialization promise or otherwise lock this section.
Useful? React with 👍 / 👎.
| results.push({ step: i, method: r.method, ok: r.ok, ...(r.error ? { error: r.error } : {}), | ||
| text: opts.keepText ? r.text : String(r.text || '').slice(0, opts.textChars ?? 300) }); | ||
| if (!r.ok) return { ok: false, complete: false, failedAt: i, results }; | ||
| } | ||
| return { ok: true, complete: true, results }; |
There was a problem hiding this comment.
When a successful step returns more than the default 300 characters, or more than an explicit textChars limit, this slice silently drops the remainder but the method still returns complete: true with no truncation indicator. That contradicts the completeness contract in structure/result-completeness.md and prevents callers from distinguishing a complete accessibility tree from a clipped one. Report truncation and lower complete, or retain the full text whenever claiming completeness.
Useful? React with 👍 / 👎.
1 participant
Adds
desktop.*so one execute_code can read a native macOS app, filter its accessibility tree in-guest, act, and verify, returning only the answer.Why not a separate MCP: one model turn per click, and every tree/screenshot lands in context. Measured on a chat app: tree 4.8K chars + 140 KB screenshot per call; in code mode the guest returned 2 lines.
Transport: signed broker from
codex-computer-use-mcp(the Sky client refuses untrusted parents). One session per server process so element indexes stay valid across calls.macOS: host app becomes TCC responsible process; optional
disclaimLauncherdisclaims responsibility for the signed codex binary. Documented in structure/desktop-surface.md.Off by default. Tests: 1107/1107 locally (+9 new). Live-checked list_apps / get_app_state / run on macOS.