Add Debug-only MCP integration for AI-assisted UI inspection - #26
Open
TastyHeadphones wants to merge 2 commits into
Open
Add Debug-only MCP integration for AI-assisted UI inspection#26TastyHeadphones wants to merge 2 commits into
TastyHeadphones wants to merge 2 commits into
Conversation
Introduce `lookinside-mcp`, an optional MCP server that lets AI agents inspect a running Debug build's UI through the same Peertalk plumbing the macOS app uses — hierarchy, search, element details, screenshots, highlight, layout/accessibility diagnostics, and a one-shot bug report. Two new SPM units: LookinMCPCore (headless inspection client, JSON shaping, diagnostics, secure-text redaction) and lookinside-mcp (executable using modelcontextprotocol/swift-sdk over stdio). Reuses LookinCore data models and the existing in-app LookinServer; license gate is bypassed because it is enforced client-side, not by the in-process server.
Codex CLI and Claude Code both use the canonical `<cli> mcp add <name> [--env K=V] -- <command> [args]` form. The previous claude-code snippet omitted the `--` separator, which works in practice but isn't the documented syntax. Aligns both clients on the same shape and documents the env-var passthrough.
TastyHeadphones
force-pushed
the
claude/eager-robinson-c731e7
branch
2 times, most recently
from
May 14, 2026 02:28
aac150e to
d432066
Compare
hxhlb
pushed a commit
to hxhlb/LookInside
that referenced
this pull request
Aug 13, 2026
Adds the third inspection method on top of the v1 routing scaffold —
exposing per-view attribute groups, the surface where LookInside has its
largest lead over comparable tools (~400 built-in attributes across
UIKit and AppKit, plus user-custom hooks).
Wire shape
----------
LKMCPBridgeAttributeGroup → LKMCPBridgeAttributeSection → LKMCPBridgeAttribute,
flattened into JSON. Each attribute carries a `kind` discriminator string
plus a JSON value whose shape correlates with kind, so consumers can
dispatch on `kind` without re-implementing LookinAttrType numerics.
Routing
-------
`attributes.read { targetIdentifier, objectIdentifier, includeUserCustom? }`:
- targetIdentifier picks a LookinLiveDocument (same scheme as
hierarchy.read).
- objectIdentifier picks one LookinDisplayItem via BFS from the
document's top-level windows.
- includeUserCustom (default true) controls whether lookin_customDebugInfos
cards are merged in alongside built-in ones.
- Response carries `groups: [LKMCPBridgeAttributeGroup]` plus a
`detailsCached: Bool` flag so an agent that gets back an empty `groups`
array knows to ask the user to open the view in the host inspector
first (host fetches per-item attrs on demand via RPC 203; v2 reads the
cache without triggering a fresh fetch).
Encoder coverage (LKMCPBridgeAttributeEncoder)
----------------------------------------------
First-class projections for the structurally distinct LookinAttrType
cases:
- numeric kinds (char / int / short / long / longLong / unsigned*) →
`integer` with NSNumber.int64Value
- float, double → `double`
- BOOL → `bool` (NS_ENUM imports as `.BOOL` with the importer keeping
the all-caps initialism)
- Sel, Class, NSString, EnumString → `string`
- EnumInt, EnumLong → `integer` with `kind: "enum"`
- CGPoint / CGVector / CGSize / CGRect → object with named axes
- CGAffineTransform → six-component object
- UIEdgeInsets, UIOffset → named-axis object (shares NSEdgeInsets
memory layout on AppKit hosts)
- UIColor → `{ red, green, blue, alpha }` (server encodes as a 4-element
NSNumber array of 0...1 components)
- Shadow → `{ offset, opacity, radius, color }` matching
LKS_CustomAttrGroupsMaker.m construction
- Json → round-tripped through JSONSerialization
- CustomObj → `{ className, description }` reflection fallback
- None, Void → `null` with `kind: "void"`
- @unknown default → `{ "rawDescription": "..." }`
`extraValue` is preserved (used by enumString to ship allCases) and
custom-setter identifiers pass through unchanged so future write-path
tools can address them.
Verification
------------
xcodebuild on the LookInside scheme (Debug, CODE_SIGNING_ALLOWED=NO)
compiles all new sources cleanly. The remaining single build failure
is the pre-existing FrameworkToolbox macro resolution error in the
sibling LookInside-Injector project's derived directory (see prior
commits 48ee8fe and 34530d2 for the same observation).
Out of scope (deferred)
-----------------------
- Triggering an RPC 203 fetch when the cache is empty — v2 only reads.
- Secure-text redaction at the bridge boundary (UITextField
isSecureTextEntry / NSSecureTextField). PR LookInsideApp#26 redacts client-side;
on this branch we will land it at the host UDS boundary so every
current and future consumer is protected uniformly.
- LKMCPBridgeEntitlement license gate (every target still reports
"licensed").
Claude-Session: https://claude.ai/code/session_017FNZVjcwEMXin8B7qi2rqB
hxhlb
pushed a commit
to hxhlb/LookInside
that referenced
this pull request
Aug 13, 2026
Closes a v2 gap noted at the time attributes.read landed: secure-input views (passwords, OTPs, payment fields) could send their values across the bridge socket verbatim. Borrowing the safety floor from the upstream PR LookInsideApp#26 reference implementation, but pushing the redaction down into the host so every current and future bridge consumer inherits the protection — clients can't reach around the data layer to leak content because the data layer never emits it. New file: LKMCPBridgeSecureContentDetector - Single classifier: `isSecure(displayItem:)` returns true iff the inspected view's class chain contains `NSSecureTextField`. Matches AppKit subclasses too, including the private `_NSSecureTextField_*` variants that ship inside system views. - UIKit `UITextField.isSecureTextEntry` is documented as a known gap: LookinServer does not surface `isSecureTextEntry` as a `LookinAttr_*` today (verified by grepping `LookinAttrIdentifiers.h`), so the bridge has no reliable way to read the bit. The reference PR LookInsideApp#26 redactor declares the same UIKit branch but it silently never fires there either for the same reason. A follow-up commit on LookInside- Server will surface the property; this detector can opt the UIKit case in once it exists. LKMCPBridgeAttributeEncoder - `encode(_:)` becomes `encode(_:redactingSecureContent:)`. When the flag is true, projections of kind `string` / `selector` / `class` / `enum` are replaced with `{ kind: "redacted", value: nil }` before the DTO leaves the encoder. Numeric, boolean, geometry, color, shadow, json, and custom-object projections fall through unchanged — they don't carry user-visible text. - The `enum` redaction is intentionally conservative: the kind covers both EnumInt/EnumLong (numeric) and EnumString (textual). We can't cheaply tell them apart at the boundary, so we redact both. The cost is losing the integer enum value; the upside is no path can leak an EnumString through this kind by accident in the future. - `extraValue` is also nilled when redacting so the EnumString allCases table doesn't ship with the redacted view. LKMCPBridgeInspectionService.handleAttributesRead - Evaluates `isSecure(displayItem:)` once per display item, then threads that single decision through `encodeGroup` → `encodeSection` → `encode(attribute:)` so every attribute on that view shares the same redaction state. A secure UITextField/NSSecureTextField can't have its `placeholder` leak when its `text` is redacted, for example. - Response now carries an explicit `secureContent: Bool` field so an agent looking at a `kind: "redacted"` value understands why and can surface that to the user. Verification: xcodebuild on the LookInside scheme compiles all new sources cleanly; the only remaining build failure is the pre-existing FrameworkToolbox macro resolution issue in the sibling LookInside-Injector project's derived directory (same observation as prior commits 48ee8fe, 34530d2, 9c6824a). Claude-Session: https://claude.ai/code/session_017FNZVjcwEMXin8B7qi2rqB
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Adds an optional MCP server,
lookinside-mcp, so AI agents (Claude Desktop, Claude Code, Codex CLI, Cursor, Windsurf, VS Code, …) can inspect a running Debug build's UI through LookInside — hierarchy, screenshots, search, highlight, layout/accessibility diagnostics, and a one-shot bug report.What's new
Two new SPM units, layered:
LookinMCPCore(Swift library) — headless inspection client built onLookinCore. Includes aHierarchyProviderseam with two implementations:LiveLookinClient(Peertalk over loopback to a runningLookinServer) andFileHierarchyProvider(reads.lookinsnapshot files for offline analysis). Also contains the canonical JSON shapes, hierarchy index, element search, layout/accessibility diagnostics, bug-report builder, and aSecureTextRedactorthat strips secure-text-field contents at the model boundary.lookinside-mcp(executable) — stdio MCP server using modelcontextprotocol/swift-sdk. One file per tool, registered via a smallToolRegistry, so adding a tool later is one new conformance.11 tools shipped:
health_check,list_apps,current_screen,get_hierarchy,search_elements,get_element,capture_screenshot,highlight_element,diagnose_layout,diagnose_accessibility,export_bug_report.Architecture
The MCP server is a parallel consumer of
LookinServeralongside the macOS LookInside.app. The license handshake inLKConnectionManager.mis purely client-side —Sources/LookinServer/Server/Connection/LKS_RequestHandler.mhas no license check — so a separate Debug tooling client can speak the protocol directly without dragging the auth helper into a CLI.Safety / Debug-only
LookinServer, so there's nothing to talk to.UITextField.isSecureTextEntry,NSSecureTextField) are redacted bySecureTextRedactorinsideJSONShape.nodeso every current and future tool inherits the protection.Notes & limitations
macOSminimum from 11 to 13, required by the swift-sdk transport. SPM has no per-target deployment target; this is the cleanest option. iOS/tvOS minimums are unchanged.highlight_elementrequires a new server-side request type that doesn't exist yet; it returns{ ok: false, reason: "…" }for now rather than silently lying. Tracked as a follow-up.ToolRegistrypattern.Install + try
./Scripts/build-mcp-server.sh ./build/lookinside-mcp health ./build/lookinside-mcp print-config claude-desktop # or claude-code, codex, cursor, windsurf, vscodeThen paste the snippet (or run the printed command) into your client's MCP config and ask:
Test plan
swift build --product lookinside-mcp— cleanswift test— 15/15 tests pass (hierarchy index, JSON shape, secure-text redaction, element search, layout/a11y diagnostics, provider error paths)./build/lookinside-mcp health—no_targetpath renders correctly with nonzero exit./build/lookinside-mcp print-config <client>— emits absolute-path snippets for all 6 supported clientsinitialize→notifications/initialized→tools/listreturns all 11 tools with valid JSON SchemaLookinServerembedded (requires user environment)export_bug_reportagainst a real screenlookinside-mcp serve --snapshot some.lookinfrom a captured archiveDocs
healthinterpretation, common errors