Skip to content

Read typed agent-device refusals instead of the text older releases needed - #90

Open
thymikee wants to merge 8 commits into
expo:mainfrom
thymikee:agent-device-drop-legacy
Open

thymikee wants to merge 8 commits into
expo:mainfrom
thymikee:agent-device-drop-legacy

Conversation

@thymikee

@thymikee thymikee commented Nov 17, 2026 •

Copy link
Copy Markdown

Summary

We'd like Explore to use agent-device directly, without workarounds. Explore pins one exact agent-device version, so code written for older releases is dead. This removes the parts that recognised three iOS failures from their message text, because those releases didn't type them. All three now read typed fields:

-return details.recovery === "runner_reported_failure" && details.lifecycleState === "failed"
-  && (native.message === RUNNER_BUSY_MESSAGE || native.message === "main thread execution timed out");
+return details.runnerErrorCode === "RUNNER_BUSY" || details.reason === "runner_busy";

-return /requires a valid viewport/i.test(`${error.message}\n${error.nextAction ?? ""}`);
+return details.reason === "missing-viewport" || details.reason === "invalid-viewport" || sparseSystemSurface;
  • Runner busy: runnerBusyRefusal now serves both read recovery and the explore:binary retry loop, which had its own regex.
  • Watchdog expiry: reads MAIN_THREAD_TIMEOUT / runner_main_thread_timeout.
  • Unreadable surface (Save Password over the app): reads the snapshot engine's viewport reason, or a sparse capture that carries systemSurface.

Outdated version comments are corrected. 8 files changed. No visible change. No change to usage.

Stacked on #87: review after it. The diff includes #87's commits until it merges.

Validation

Head 2adb9ba. Each typed field is guaranteed by agent-device 0.21.40 (tag v0.21.40):

  • busy: packages/apple-runner/src/runner-contract.ts:33,436,463
  • watchdog: runner-contract.ts:42,437
  • viewport: capture-kit/src/ios-snapshot-engine/invariants.ts:21,28 and types.ts:44-45,82
  • sparse system surface: platform-apple/src/snapshot-presentation.ts:246-266

Two things are kept on purpose:

  • alert accept's 15 s process bound, as an outer guard over accept's own retry window.
  • controllerWedged's text check, which also sees errors Explore re-wraps itself.

Build, typecheck and lint pass. npm run test:node: 1718 tests. The only failures are the same 29 that fail on main on this host. The tests build their fixtures from real 0.21.39 envelopes, and 0.21.40 does not change these fields. There's no device run: this removes dead branches and doesn't change what runs on 0.21.40.

🤖 Generated with Claude Code


👀 Devin's session did not pick up the findings · 3 findings for you to triage · Follow Devin's work

Devin Review

thymikee and others added 4 commits November 17, 2026 15:31
…rol back names

0.21.39 no longer says "in-app back control is not available" or "Unable to dismiss the iOS
keyboard", so the fallbacks Explore keyed on those messages would never run. They now read
error.details.reason:

- back: on iOS, agent-device presses the navigation bar back button, else swipes from the left
  edge, and refuses with back_swipe_no_change when nothing changed. It lists the screen's dismiss
  controls (Close, Cancel, Back) in details.dismissControls without pressing them. Explore presses
  the one it names when its label is safe by Explore's destructive and external rules, requires
  the destination to change, and otherwise swipes a sheet down as before.
- keyboard dismiss refuses with keyboard_dismiss_unavailable; the return-key fallback keys on it.
- the remote /health refusal is remote_daemon_unavailable with dispatched "no", which is what
  Explore inferred from the message before resending.

The other controller's bridge gives the same reasons.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
As in source-driven capture, explore:binary compares the screen before and after it presses the
dismiss control agent-device named, and fails the back with agent-device's refusal when nothing
changed.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ipes after a press

A modal kept in a screen's own state keeps the route when its Close control dismisses it, so the
destination check alone read that as a failure and fell through to the sheet swipe-down with the
geometry read before the press. A press now counts when the route changes or the pressed control
is gone. If neither happens, back fails with AGENT_DISMISS_CONTROL_FAILED instead of swiping.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…eaving the screen

The pressed control disappearing counted as back, but a confirmation such as "Discard changes?"
also hides it while the screen stays. The control leaving now counts only when no alert took its
place; otherwise back fails with AGENT_DISMISS_CONTROL_FAILED.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@chatgpt-codex-connector

chatgpt-codex-connector Bot commented Nov 17, 2026 •

Copy link
Copy Markdown
Codex Review Summary: ✅ Completed

This comment shows the latest Codex review activity on this pull request.

Review Status Commit Review trigger
📝 Code Review ✅ Completed 2026-11-17T16:36:37.001506Z 5db688d PR opened
ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

@devin-ai-integration devin-ai-integration Bot left a comment •

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Note

Newer findings are available below. Devin Review posted a newer report on this PR, in addition to the findings presented here.

✅ Devin Review: No Issues Found

Devin Review analyzed this PR and found no bugs or issues to report.

Devin Review

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 5db688d2d8

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread src/binary/run.ts Outdated
const before = screenEvidenceKey((await observe(false)).nodes);
controller(["press", "--level", "full", "--json", "--", control.selector], 120_000);
await settled();
if (screenEvidenceKey((await observe(false)).nodes) === before) throw error;

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Reject confirmations before accepting binary back

In the explore:binary path, when a named Close/Cancel control opens a “Discard changes?” confirmation instead of dismissing the modal, that confirmation changes screenEvidenceKey, so this check completes as a successful back even though the blocking UI remains. Validate that the original modal was actually left (and that no confirmation/Alert replaced it), rather than treating every changed screen as proof that back succeeded.

Useful? React with 👍 / 👎.

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Agreed. This code comes from #87, which this PR stacks on, so I fixed it there in add2d04; this branch is rebased on it. A dismiss-control press now never counts as back while an Alert is showing, in both paths. explore:binary fails the back when the screen after the press shows an alert. In source-driven capture, the alert check now covers the route-change case as well as the control-disappearance case, so a confirmation that hides the observer no longer reads as success. A new test in ci-map.test.mjs covers the hidden-observer confirmation.

Comment thread src/capture/route-capture.ts Outdated
for (let attempt = 0; attempt < 4; attempt++) {
await sleep(500, undefined, { signal: liveSignal(signal) });
after = (await readSnapshot(signal)).snapshot.nodes;
if (observerIdentity(after) !== observerIdentity(before) || !controlShown(after) && !alertShown(after)) return { via: "dismiss-control" };

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Apply the alert guard to observer-identity changes

In the source-agent path, if pressing the dismiss control opens a system confirmation whose accessibility surface hides the app observer, observerIdentity(after) changes (often to an empty marker list) while alertShown(after) is true. The first side of this || therefore returns dismiss-control despite the blocking confirmation; require the alert to be absent for both the route-change and control-disappearance success cases.

Useful? React with 👍 / 👎.

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Agreed. This code comes from #87, which this PR stacks on, so I fixed it there in add2d04; this branch is rebased on it. A dismiss-control press now never counts as back while an Alert is showing, in both paths. explore:binary fails the back when the screen after the press shows an alert. In source-driven capture, the alert check now covers the route-change case as well as the control-disappearance case, so a confirmation that hides the observer no longer reads as success. A new test in ci-map.test.mjs covers the hidden-observer confirmation.

A "Discard changes?" confirmation after the press changes the screen, and can hide the route
observer, so either signal alone read it as having left. Both code paths now require that no
alert is showing before a press counts as back.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@thymikee
thymikee force-pushed the agent-device-drop-legacy branch from 5db688d to 45ca402 Compare November 17, 2026 16:39
agent-device names a dismiss control by its identifier when it has
one, and its label may then be empty. The press check matched only
the label, so a button with no label was never seen and the press
counted as leaving even when the modal stayed. It now matches the
control the way agent-device named it, and the control leaving counts
only when it was on screen before the press.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@thymikee
thymikee force-pushed the agent-device-drop-legacy branch from 45ca402 to a16b520 Compare November 19, 2026 15:28
thymikee and others added 2 commits November 19, 2026 17:03
0.21.40 adds the keyboard fix found while testing this bump: keyboard
dismiss no longer presses the return key, so on a field whose return
key is Done it refuses with keyboard_dismiss_unavailable instead of
submitting the form. The typed reasons Explore reads are unchanged.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Explore recognised three iOS controller failures from the prose the
controller wrote beside them: the message "still finishing a previous
command", the run's derived "runner_main_thread_execution_timeout",
and "requires a valid viewport". agent-device 0.21.40 publishes each
verdict as typed details, so the prose was only ever a fallback for
the releases the pin no longer allows.

The busy refusal is read as runnerErrorCode/reason runner_busy through
a new runnerBusyRefusal, shared by the read recovery and the binary
CLI's retry loop. A snapshot that could not read the system surface
over the app is read as the snapshot engine's reason,
missing-viewport or invalid-viewport, through a new
unreadableSystemSurfaceFailure. The alert watchdog answer is read as
runnerErrorCode MAIN_THREAD_TIMEOUT plus its reason. No message text
decides any of them now.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@thymikee
thymikee force-pushed the agent-device-drop-legacy branch from a16b520 to 2adb9ba Compare November 19, 2026 16:28

@devin-ai-integration devin-ai-integration Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Devin Review found 3 findings. Devin's session did not pick them up; all 3 findings are left for you to triage below.

View findings in Devin Review · Follow Devin's work

Devin Review

* all be this version; each caller checks it by exact equality.
*/
export const AGENT_DEVICE_VERSION = "0.21.29-nightly.20261018.74";
export const AGENT_DEVICE_VERSION = "0.21.40";

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔴 Keyboard dismissal submits forms

On iOS, dismissKeyboard falls back to Return even when the keyboard has a separate Hide Keyboard control. The pinned runner refuses automatic dismissal whenever it sees a visible keyboard. If the observed Return key submits the form, exploration submits it instead of just hiding the keyboard.

Learn more

The binary explorer exposes dismiss-keyboard as an action that hides the keyboard. dismissKeyboard interprets keyboard_dismiss_unavailable as a missing dismiss key and presses a matching Return key by coordinate. The 0.21.40 Apple runner returns that same refusal for any keyboard it recognizes as visible, including keyboards with a dedicated dismissal control that 0.21.39 could press. That sends more keyboards down a fallback that can submit the focused field.

Example: An iPad form exposes both Hide Keyboard and a Go key that submits a request. If both appear in the accessibility snapshot, dismiss-keyboard can press Go instead of the dedicated Hide Keyboard key.

Recommended fix: Remove the unconditional Return-key fallback from dismissal. Use a fresh observation to press a positively identified, dedicated keyboard hide control and verify the keyboard disappears. If no safe control can be identified, report that dismissal is unavailable and leave form submission to a separate explicit action. Add a regression test with both a hide control and a submitting Return key.

Devin Review


Was this helpful? React with 👍 or 👎 to provide feedback.

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Right: 0.21.40 dismisses no visible iOS keyboard itself. It refuses with keyboard_dismiss_unavailable, dispatched: "no", and a hint saying how to proceed. Explore's return-key fallback is removed in #91, which stacks on this PR. There, dismiss-keyboard sends only keyboard dismiss, and the model now gets agent-device's hint with every failed action ("press the visible Hide Keyboard key or an app control that ends editing"). We are deliberately not adding a hide-control matcher to Explore. On iPad the hide key isn't even in the snapshot while the app is windowed, and agent-device dropped label matching on purpose, so the agent picks the control from what it sees.

* Older controllers report one Keyboard element; agent-device 0.21.15 reports only its
* Key elements, which its own occlusion check also treats as the keyboard.
* The controller's keyboard elements: a `Keyboard` container, its `Key` children, or both. A real
* agent-device 0.21.40 capture projects the `Keyboard` plane (role UIKeyboardAutomatic) together

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔴 Keyboard dismissal blocks iOS setup

When an iOS keyboard covers a configured tap or fill target, executeCaptureSetup aborts even with a dedicated Hide Keyboard control. The pinned runner refuses automatic dismissal for visible keyboards, so setup never reaches the target.

Learn more

Route capture passes configured UI prerequisites to executeCaptureSetup. When a requested control overlaps the software keyboard, the tap and fill branch relies on keyboard dismiss before it reselects the target. The pinned Apple runner refuses this command whenever it detects a visible keyboard, even if the screen exposes a dedicated hide control. The executor treats the refusal as SETUP_KEYBOARD_BLOCKING, so the configured action never runs.

Example: A setup sequence focuses an input on an iPad and then taps a Continue button underneath the keyboard. Even with a dedicated Hide Keyboard key visible, setup stops before tapping Continue.

Recommended fix: On the iOS refusal, use a fresh snapshot to locate a dedicated, unambiguous hide control. Press only that control, then take another snapshot and reselect the original target. Keep the existing refusal when no safe control is available; do not substitute Return. Cover the typed refusal and subsequent re-selection in the setup tests.

Devin Review


Was this helpful? React with 👍 or 👎 to provide feedback.

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Intended. On iOS, agent-device 0.21.40 no longer dismisses a visible keyboard itself, so setup now stops with SETUP_KEYBOARD_BLOCKING, with agent-device's refusal as its cause. It does not guess a control to press. A setup sequence is explicit configuration: when a target sits under the keyboard, the sequence should tap the app's own control that ends editing first. The existing next action already says so. We don't add Hide Keyboard matching here, for the reasons in the other thread: label matching is what agent-device removed, and on a windowed iPad the hide key isn't in the snapshot at all.

* all be this version; each caller checks it by exact equality.
*/
export const AGENT_DEVICE_VERSION = "0.21.29-nightly.20261018.74";
export const AGENT_DEVICE_VERSION = "0.21.40";

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔍 Local recompare rejects older captures

Local recompare reads saved captures through parseRuntimeExecutionContext. The parser requires the current agent-device version even though recompare does not drive a simulator.

Devin Review


Was this helpful? React with 👍 or 👎 to provide feedback.

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is how it works today and isn't new here: parseRuntimeExecutionContext has required the pinned controller version since before this stack, and every pin bump retires older saved captures for local recompare. Leaving it as is in this PR.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant