Ghosthand

Reliable remote macOS control for AI agents. Acked, idempotent input; accessibility-element targets; host-side settle detection — instead of screenshot → guess pixels → click → sleep(2).

Measured (same Mac, same network, same 6-step GUI task)	VNC (tuned baseline)	Ghosthand
Agent knows the action landed (p50)	766 ms	139 ms (~5.5×)
6-step GUI task, confirmed at every step (p50)	16.5 s	3.9 s (~4×)
Mid-action connection kill × 15	15 duplicated (retry) or 5 lost (no retry)	0 duplicated, 0 lost

Full methodology, tails, and the cases where VNC wins: docs/BENCHMARKS.md.

Ghosthand lets an LLM agent running somewhere else drive a real macOS desktop reliably.

The problem

Every remote computer-use stack today runs the same loop: screenshot → guess pixel coordinates → synthetic click → sleep(2) → screenshot again. That loop is tolerable when the agent runs on the machine it drives. Over a network it degrades badly, and for reasons that are structural, not tunable:

• No acknowledgement. VNC-class protocols fire input and forget it. The agent cannot distinguish "click landed", "click lost", and "click landed twice" — and retrying makes the last case more likely.
• No settle signal. sleep(2) is a guess. Too short and the agent reasons over a half-rendered screen; too long and every step pays a 2-second tax.
• Pixel targeting is brittle. A button is a rectangle of pixels whose position depends on scale factor, window layout, and whatever animated since the last screenshot. Retina backing-store math breaks coordinate mapping in ways that fail silently.
• Stale-frame actions. The agent reasons over frame N and acts on frame N+k. If a dialog appeared in between, the click goes somewhere it was never aimed.

The insight

Move the capture-and-input loop onto the target Mac, where the OS can actually answer these questions, and expose it as a small, boring HTTP/JSON API (with an MCP façade for agent clients):

• Acknowledged, idempotent actions. Every mutating call carries a client-generated action_id. Re-sending it returns the stored result and performs nothing. Duplicate clicks become impossible by construction.
• Semantic targets. Click the Save button (via the Accessibility API's AXUIElementPerformAction), not a pixel. Coordinates are the fallback, not the interface.
• Real settle detection. ScreenCaptureKit dirty rects fused with AXObserver notifications, computed host-side at full frame rate. The agent is told when the screen stops changing instead of guessing.
• Frame-bound actions. An action carries the frame the agent reasoned over; if the target changed since, Ghosthand rejects the action instead of misclicking.
• Logical coordinates only. The daemon owns all Retina/backing-store math. Clients never see a 2× framebuffer.
• Input lease. One holder drives at a time; a human touching the mouse suspends the lease so the agent and the owner never fight over the cursor.

Screen Sharing / VNC stays exactly where it is — as the channel a human watches, never the path an agent acts through.

Install

One-liner (downloads the latest signed release, verifies its SHA-256, installs the LaunchAgent, and walks you through the TCC grants) — not live yet: no public release has been published; until then build from source below:

curl -fsSL https://raw.githubusercontent.com/BariBariGood/ghosthand/main/scripts/install.sh | bash

Homebrew (tap pending publication):

brew install baribarigood/tap/ghosthandd

Details, options, --check, and uninstall: docs/INSTALL.md.

30-second quickstart

On the target Mac (needs Accessibility + Screen Recording permission — see docs/OPERATIONS.md):

swift build -c release --product ghosthandd
umask 077; mkdir -p ~/.ghosthand; head -c 32 /dev/urandom | xxd -p -c 64 > ~/.ghosthand/token
.build/release/ghosthandd --token-file ~/.ghosthand/token

From anywhere that can reach it (loopback or your private network overlay):

TOKEN=$(cat ~/.ghosthand/token); H="Authorization: Bearer $TOKEN"; URL=http://127.0.0.1:7434
curl -H "$H" $URL/v1/session                       # resync: display, frontmost app, epochs
curl -H "$H" $URL/v1/tree | less                   # find your target's element_id ("ax-…")
LEASE=$(curl -H "$H" -X POST $URL/v1/lease \
  -d '{"holder":"quickstart","ttl_ms":300000,"purpose":"demo"}' | jq -r .lease_id)
curl -H "$H" -X POST $URL/v1/actions \
  -d '{"action_id":"a-1","lease_id":"'$LEASE'","verb":"click","target":{"element_id":"ax-38f2"}}'

(Semantic label-based targeting — "click the Save button" — lives in the MCP tools, which resolve labels to element ids for you.)

Or point an MCP client — Claude Code, Codex, Cursor, Devin, or anything that speaks stdio MCP — at the daemon in one config block: mcp/README.md. With a fleet config the same MCP server fronts several Macs behind one endpoint (machine parameter per call, per-machine leases, central audit log).

How it compares

	VNC / Screen Sharing	RustDesk-class remote desktop	Local computer-use agent (runs on the box)	Ghosthand
Input acknowledged	No — fire and forget	No	Implicitly (same process)	Yes — per-action ack with outcome
Duplicate input on retry/reconnect	Possible, silent	Possible	N/A	Impossible (action_id replay returns stored result)
Targeting	Pixels	Pixels	Pixels (screenshot-based)	Accessibility elements; pixels as fallback
"Screen is done changing"	Guess (sleep)	Guess	Guess or poll screenshots	Host-side settle: dirty rects + AX notifications
Stale-view protection	None	None	None	Frame-bound actions rejected if view changed
Retina/scale handling	Client's problem	Client's problem	Mostly correct locally	Daemon-owned; logical points only
Human/agent contention	Fight over cursor	Fight over cursor	N/A	Input lease; human touch suspends agent
Designed for	Humans	Humans	Agent on its own machine	Remote agent driving a real desktop

Measured reliability numbers — including the tails, byte accounting, and the cases where VNC wins — live in docs/BENCHMARKS.md.

Honest limitations

• macOS only, one GUI session. The protocol is OS-neutral by design, but the only backend today is macOS (13+). It runs as a LaunchAgent inside the logged-in user's session — no login window, no fast-user-switched sessions.
• Accessibility quality is app-dependent. Apps with poor AX trees (many games, some Electron apps, custom-drawn UIs) degrade Ghosthand to coordinate-fallback mode, where it is still acknowledged and frame-bound but no longer semantic.
• Not a streaming protocol. GET /v1/frame serves keyframes and dirty-rect diffs for an agent's observe-act loop; it is not a low-latency video feed for humans. Keep VNC for watching.
• Trust model is "trusted client". Bearer token over loopback/private overlay network. There is no per-action authorization, no multi-tenant isolation, no TLS termination in the daemon itself. Do not expose port 7434 to a network you don't trust end-to-end.
• TCC permissions are manual-ish. Accessibility and Screen Recording grants require either a human click or MDM/scripted provisioning (scripts/tcc-provision.sh).
• Secure input fields. When macOS secure input is active (password fields), Ghosthand refuses to type rather than pretending it worked.

Layout

Path	What
Sources/GhosthandCore	Platform-independent logic — protocol types, action ledger, settle state machine, tree pruning. Builds and unit-tests on Linux.
Sources/GhosthandMac	macOS-only adapters — ScreenCaptureKit capture, Accessibility tree/actions, CGEvent input, AXObserver.
Sources/ghosthandd	The daemon executable: HTTP server, routing, auth, wiring.
Tests/GhosthandCoreTests	Linux-runnable unit tests for everything in GhosthandCore.
mcp/	MCP server exposing the daemon as agent tools.
harness/	Reliability benchmark + chaos test harness (runs from Linux against a Mac).
docs/	Protocol (normative), architecture, operations, roadmap.

Design rules

1. Never link GPL/AGPL code. First-party Apple frameworks plus MIT/Apache/BSD/MPL dependencies only (manifest).
2. Logic lives in GhosthandCore so it can be tested without a Mac. GhosthandMac is thin adapters over OS APIs.
3. Small files. One concrete implementation per file, wired through a registry/index.
4. The protocol is OS-neutral. Linux and Windows backends must be addable without changing the wire format.

License

Apache-2.0.