Skip to content

feat(plugin): integrate Computer Use and migrate Apexo paths - #209

Open
cluster1900 wants to merge 14 commits into
mainfrom
computer-use-plan
Open

cluster1900 wants to merge 14 commits into
mainfrom
computer-use-plan

Conversation

@cluster1900

@cluster1900 cluster1900 commented Nov 16, 2026 •

Copy link
Copy Markdown
Owner

Combined scope

This is the single review for the local Computer Use integration and the Apexo directory and Desktop origin migration from #210. The branch retains both original commit histories and includes the latest main at consolidation.

  • Desktop and CLI use the Apexo data paths, and the Desktop renderer loads from apexo://renderer. The upgrade preserves saved settings, session history, managed worktrees, installed plugins, and existing absolute paths through compatibility aliases.
  • Computer Use 0.1.1 is available from the Apexo Skills Catalog. The compatibility backend resolves the local MCP plugin's data directory, and Settings can connect, disconnect, or retry that MCP server. The Auto-accept permissions switch uses the active session or project scope.
  • The combined plugin registry first resolves migrated package and catalog paths, then initializes the built-in Apexo Skills source when needed.

Combined runtime verification

  • Migrated copies of Computer Use 0.1.0 and 0.1.1 from a synthetic legacy installation to the canonical Apexo directory. Both kept their installation identity, connected and reconnected through the real V1 backend, and returned version 1.3.1 from the actual MCP tool.
  • Rebuilt macOS arm64 Electron and verified its apexo://renderer origin, the status and Settings MCP controls, and Auto-accept permissions for an existing session and an unsent draft.
  • Passed 72 focused migration, CORS, MCP, and plugin tests, 114 App unit tests, all 21 typecheck tasks, and both generated-client checks. Mini and fullscreen TUI ran the published plugin and rendered the verified tool result.
  • Passed all 360 scenarios in each of HttpApi coverage, auth, and effect modes. Five cases failed on the first aggregate run, and the three affected batches passed unchanged on replay.

Original Computer Use review and acceptance

Issue for this PR

No linked issue.

Type of change

  • New feature
  • Bug fix
  • Documentation

What does this PR do?

Adds an optional local macOS Computer Use plugin through Apexo's existing plugin loader and stdio MCP runtime. The bundle packages unmodified vercel-labs/agent-desktop 1.3.1 with a small native launcher, installed Skill, pinned downloads/checksums, licenses, and V1/V2 permission profiles. Apexo continues to own Sessions, model execution, permissions, history, cancellation, and plugin lifecycle.

The initial host changes were two acceptance-driven corrections: Desktop invokes the current service password command so its compiled V2 backend starts, and Responses lowering omits tool_choice: none when there are no tool definitions, which xAI rejects. Public Protocol/HttpApi and generated clients are unchanged.

Adds reproducible installed-bundle, Client, TUI, browser, packaged Desktop, and live xAI acceptance scripts, plus a dated result matrix and artifact/source identities. Recommended usage is sequential, local native V2 execution with explicit tool permissions. The original packaged Desktop acceptance used its V2 sidecar. Native binaries and assembled artifacts remain ignored build output.

Acceptance is bounded: existing-file Unicode edit/save, scrolling, randomized Calculator, and pixel-only vision passed. New-document Save-dialog navigation failed and remains unresolved. The separate signed Desktop now has denied, Accessibility-only, and granted-state evidence. CLI grant attribution and its Accessibility-only stage, Intel graphical execution, and several failure-injection cases remain uncertified; the report preserves each partial case and failed attempt. This PR does not claim all 18 feature acceptance cases pass.

How did you verify your code works?

  • Real upstream arm64 archive downloaded, checksum/attestation verified, assembled, installed, and run; all 12 final bundle checksums passed. x64 download/version also checked under Rosetta, as packaging evidence only.
  • Plugin packaging: 4 tests / 68 assertions; profile compatibility and Responses regression: 3 tests / 84 assertions.
  • Existing MCP/plugin/Client regression batch: 21 tests / 139 assertions.
  • Installed native CLI: 9 checks; Electron backend: 8 checks; Promise, Effect, and legacy SDK clients, manual permission accept/reject, denied launch, real descendant cancellation, recovery, and lifecycle covered.
  • Actual mini and fullscreen TUI: 3 checks each; browser client E2E: 6; rebuilt packaged Desktop with its own compiled daemon: 7. UI tests cover installation/activation, tool results, decoded inline images and next-request media, Stop, and manual permissions.
  • Packaged Desktop with the user's existing xAI/Grok sign-in: 4 checks. Additional live xAI CLI cases verify exact saved Unicode/newline bytes, scrolling, 137 × 52 = 7124 from real Calculator observations, a randomized pixel-only marker, and a no-tool final turn.
  • bun turbo typecheck --concurrency=2: 21 tasks passed.
  • GITHUB_ACTIONS=false bun turbo test --concurrency=2: final rerun passed all 18 Apexo batches; six successful tasks. The first aggregate run had three lifecycle timeouts; unchanged focused and aggregate reruns passed, with original causes still recorded as unknown.
  • Client and legacy SDK check:generated: passed unchanged. HttpApi coverage/auth/effect: 360 cases each, zero failures/skips/missing.
  • App unit tests: 114 passed / 1,034 assertions. Desktop runtime-profile tests: 3 passed / 8 assertions.
  • Formatting, source/artifact hash checks, script syntax, and Git whitespace checks passed.

OS permission follow-up

Added a manual CLI permission probe with a temporary AppKit launch identity, explicit state assertions, atomic phase receipts, and controller resume. With user-operated System Settings, the separate packaged Desktop returned permission errors while denied, allowed document snapshots with only Accessibility, and returned a real PNG after both grants and a controlled relaunch. CLI denied and granted-state probes also passed, but its exact OS grant attribution and intermediate state remain uncertified. D01 remains partial; the follow-up record preserves the initial capture failure, setup failures, identity limits, and cleanup. No production/plugin payload changed in this follow-up.

Screenshots / recordings

Actual packaged Desktop rendering the installed plugin's native screenshot; synthetic fixture data only:

Packaged Desktop Computer Use result

Browser evidence is included in the acceptance report.

Checklist

  • I have tested my changes locally
  • I have not included unrelated changes in this PR

@cluster1900 cluster1900 changed the title docs(plugin): plan computer use integration and acceptance docs(plugin): plan direct MCP computer use integration Nov 16, 2026
@cluster1900 cluster1900 changed the title docs(plugin): plan direct MCP computer use integration feat(plugin): integrate local macOS computer use Nov 16, 2026
@cluster1900 cluster1900 changed the title feat(plugin): integrate local macOS computer use feat(plugin): integrate Computer Use and migrate Apexo paths Nov 19, 2026

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant