Repository navigation
Conversation
…rol back names 0.21.39 no longer says "in-app back control is not available" or "Unable to dismiss the iOS keyboard", so the fallbacks Explore keyed on those messages would never run. They now read error.details.reason: - back: on iOS, agent-device presses the navigation bar back button, else swipes from the left edge, and refuses with back_swipe_no_change when nothing changed. It lists the screen's dismiss controls (Close, Cancel, Back) in details.dismissControls without pressing them. Explore presses the one it names when its label is safe by Explore's destructive and external rules, requires the destination to change, and otherwise swipes a sheet down as before. - keyboard dismiss refuses with keyboard_dismiss_unavailable; the return-key fallback keys on it. - the remote /health refusal is remote_daemon_unavailable with dispatched "no", which is what Explore inferred from the message before resending. The other controller's bridge gives the same reasons. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
As in source-driven capture, explore:binary compares the screen before and after it presses the dismiss control agent-device named, and fails the back with agent-device's refusal when nothing changed. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ipes after a press A modal kept in a screen's own state keeps the route when its Close control dismisses it, so the destination check alone read that as a failure and fell through to the sheet swipe-down with the geometry read before the press. A press now counts when the route changes or the pressed control is gone. If neither happens, back fails with AGENT_DISMISS_CONTROL_FAILED instead of swiping. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…eaving the screen The pressed control disappearing counted as back, but a confirmation such as "Discard changes?" also hides it while the screen stays. The control leaving now counts only when no alert took its place; otherwise back fails with AGENT_DISMISS_CONTROL_FAILED. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Codex Review Summary: ✅ CompletedThis comment shows the latest Codex review activity on this pull request.
ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings. |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 5db688d2d8
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| const before = screenEvidenceKey((await observe(false)).nodes); | ||
| controller(["press", "--level", "full", "--json", "--", control.selector], 120_000); | ||
| await settled(); | ||
| if (screenEvidenceKey((await observe(false)).nodes) === before) throw error; |
There was a problem hiding this comment.
Reject confirmations before accepting binary back
In the explore:binary path, when a named Close/Cancel control opens a “Discard changes?” confirmation instead of dismissing the modal, that confirmation changes screenEvidenceKey, so this check completes as a successful back even though the blocking UI remains. Validate that the original modal was actually left (and that no confirmation/Alert replaced it), rather than treating every changed screen as proof that back succeeded.
Useful? React with 👍 / 👎.
There was a problem hiding this comment.
Agreed. This code comes from #87, which this PR stacks on, so I fixed it there in add2d04; this branch is rebased on it. A dismiss-control press now never counts as back while an Alert is showing, in both paths. explore:binary fails the back when the screen after the press shows an alert. In source-driven capture, the alert check now covers the route-change case as well as the control-disappearance case, so a confirmation that hides the observer no longer reads as success. A new test in ci-map.test.mjs covers the hidden-observer confirmation.
| for (let attempt = 0; attempt < 4; attempt++) { | ||
| await sleep(500, undefined, { signal: liveSignal(signal) }); | ||
| after = (await readSnapshot(signal)).snapshot.nodes; | ||
| if (observerIdentity(after) !== observerIdentity(before) || !controlShown(after) && !alertShown(after)) return { via: "dismiss-control" }; |
There was a problem hiding this comment.
Apply the alert guard to observer-identity changes
In the source-agent path, if pressing the dismiss control opens a system confirmation whose accessibility surface hides the app observer, observerIdentity(after) changes (often to an empty marker list) while alertShown(after) is true. The first side of this || therefore returns dismiss-control despite the blocking confirmation; require the alert to be absent for both the route-change and control-disappearance success cases.
Useful? React with 👍 / 👎.
There was a problem hiding this comment.
Agreed. This code comes from #87, which this PR stacks on, so I fixed it there in add2d04; this branch is rebased on it. A dismiss-control press now never counts as back while an Alert is showing, in both paths. explore:binary fails the back when the screen after the press shows an alert. In source-driven capture, the alert check now covers the route-change case as well as the control-disappearance case, so a confirmation that hides the observer no longer reads as success. A new test in ci-map.test.mjs covers the hidden-observer confirmation.
A "Discard changes?" confirmation after the press changes the screen, and can hide the route observer, so either signal alone read it as having left. Both code paths now require that no alert is showing before a press counts as back. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
5db688d to
45ca402
Compare
agent-device names a dismiss control by its identifier when it has one, and its label may then be empty. The press check matched only the label, so a button with no label was never seen and the press counted as leaving even when the modal stayed. It now matches the control the way agent-device named it, and the control leaving counts only when it was on screen before the press. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
45ca402 to
a16b520
Compare
0.21.40 adds the keyboard fix found while testing this bump: keyboard dismiss no longer presses the return key, so on a field whose return key is Done it refuses with keyboard_dismiss_unavailable instead of submitting the form. The typed reasons Explore reads are unchanged. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Explore recognised three iOS controller failures from the prose the controller wrote beside them: the message "still finishing a previous command", the run's derived "runner_main_thread_execution_timeout", and "requires a valid viewport". agent-device 0.21.40 publishes each verdict as typed details, so the prose was only ever a fallback for the releases the pin no longer allows. The busy refusal is read as runnerErrorCode/reason runner_busy through a new runnerBusyRefusal, shared by the read recovery and the binary CLI's retry loop. A snapshot that could not read the system surface over the app is read as the snapshot engine's reason, missing-viewport or invalid-viewport, through a new unreadableSystemSurfaceFailure. The alert watchdog answer is read as runnerErrorCode MAIN_THREAD_TIMEOUT plus its reason. No message text decides any of them now. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
a16b520 to
2adb9ba
Compare
There was a problem hiding this comment.
Devin Review found 3 findings. Devin's session did not pick them up; all 3 findings are left for you to triage below.
| * all be this version; each caller checks it by exact equality. | ||
| */ | ||
| export const AGENT_DEVICE_VERSION = "0.21.29-nightly.20261018.74"; | ||
| export const AGENT_DEVICE_VERSION = "0.21.40"; |
There was a problem hiding this comment.
🔴 Keyboard dismissal submits forms
On iOS, dismissKeyboard falls back to Return even when the keyboard has a separate Hide Keyboard control. The pinned runner refuses automatic dismissal whenever it sees a visible keyboard. If the observed Return key submits the form, exploration submits it instead of just hiding the keyboard.
Learn more
The binary explorer exposes dismiss-keyboard as an action that hides the keyboard. dismissKeyboard interprets keyboard_dismiss_unavailable as a missing dismiss key and presses a matching Return key by coordinate. The 0.21.40 Apple runner returns that same refusal for any keyboard it recognizes as visible, including keyboards with a dedicated dismissal control that 0.21.39 could press. That sends more keyboards down a fallback that can submit the focused field.
Example: An iPad form exposes both Hide Keyboard and a Go key that submits a request. If both appear in the accessibility snapshot, dismiss-keyboard can press Go instead of the dedicated Hide Keyboard key.
Recommended fix: Remove the unconditional Return-key fallback from dismissal. Use a fresh observation to press a positively identified, dedicated keyboard hide control and verify the keyboard disappears. If no safe control can be identified, report that dismissal is unavailable and leave form submission to a separate explicit action. Add a regression test with both a hide control and a submitting Return key.
Was this helpful? React with 👍 or 👎 to provide feedback.
There was a problem hiding this comment.
Right: 0.21.40 dismisses no visible iOS keyboard itself. It refuses with keyboard_dismiss_unavailable, dispatched: "no", and a hint saying how to proceed. Explore's return-key fallback is removed in #91, which stacks on this PR. There, dismiss-keyboard sends only keyboard dismiss, and the model now gets agent-device's hint with every failed action ("press the visible Hide Keyboard key or an app control that ends editing"). We are deliberately not adding a hide-control matcher to Explore. On iPad the hide key isn't even in the snapshot while the app is windowed, and agent-device dropped label matching on purpose, so the agent picks the control from what it sees.
| * Older controllers report one Keyboard element; agent-device 0.21.15 reports only its | ||
| * Key elements, which its own occlusion check also treats as the keyboard. | ||
| * The controller's keyboard elements: a `Keyboard` container, its `Key` children, or both. A real | ||
| * agent-device 0.21.40 capture projects the `Keyboard` plane (role UIKeyboardAutomatic) together |
There was a problem hiding this comment.
🔴 Keyboard dismissal blocks iOS setup
When an iOS keyboard covers a configured tap or fill target, executeCaptureSetup aborts even with a dedicated Hide Keyboard control. The pinned runner refuses automatic dismissal for visible keyboards, so setup never reaches the target.
Learn more
Route capture passes configured UI prerequisites to executeCaptureSetup. When a requested control overlaps the software keyboard, the tap and fill branch relies on keyboard dismiss before it reselects the target. The pinned Apple runner refuses this command whenever it detects a visible keyboard, even if the screen exposes a dedicated hide control. The executor treats the refusal as SETUP_KEYBOARD_BLOCKING, so the configured action never runs.
Example: A setup sequence focuses an input on an iPad and then taps a Continue button underneath the keyboard. Even with a dedicated Hide Keyboard key visible, setup stops before tapping Continue.
Recommended fix: On the iOS refusal, use a fresh snapshot to locate a dedicated, unambiguous hide control. Press only that control, then take another snapshot and reselect the original target. Keep the existing refusal when no safe control is available; do not substitute Return. Cover the typed refusal and subsequent re-selection in the setup tests.
Was this helpful? React with 👍 or 👎 to provide feedback.
There was a problem hiding this comment.
Intended. On iOS, agent-device 0.21.40 no longer dismisses a visible keyboard itself, so setup now stops with SETUP_KEYBOARD_BLOCKING, with agent-device's refusal as its cause. It does not guess a control to press. A setup sequence is explicit configuration: when a target sits under the keyboard, the sequence should tap the app's own control that ends editing first. The existing next action already says so. We don't add Hide Keyboard matching here, for the reasons in the other thread: label matching is what agent-device removed, and on a windowed iPad the hide key isn't in the snapshot at all.
| * all be this version; each caller checks it by exact equality. | ||
| */ | ||
| export const AGENT_DEVICE_VERSION = "0.21.29-nightly.20261018.74"; | ||
| export const AGENT_DEVICE_VERSION = "0.21.40"; |
There was a problem hiding this comment.
🔍 Local recompare rejects older captures
Local recompare reads saved captures through parseRuntimeExecutionContext. The parser requires the current agent-device version even though recompare does not drive a simulator.
Was this helpful? React with 👍 or 👎 to provide feedback.
There was a problem hiding this comment.
This is how it works today and isn't new here: parseRuntimeExecutionContext has required the pinned controller version since before this stack, and every pin bump retires older saved captures for local recompare. Leaving it as is in this PR.
Summary
We'd like Explore to use agent-device directly, without workarounds. Explore pins one exact agent-device version, so code written for older releases is dead. This removes the parts that recognised three iOS failures from their message text, because those releases didn't type them. All three now read typed fields:
runnerBusyRefusalnow serves both read recovery and theexplore:binaryretry loop, which had its own regex.MAIN_THREAD_TIMEOUT/runner_main_thread_timeout.systemSurface.Outdated version comments are corrected. 8 files changed. No visible change. No change to usage.
Stacked on #87: review after it. The diff includes #87's commits until it merges.
Validation
Head 2adb9ba. Each typed field is guaranteed by agent-device 0.21.40 (tag
v0.21.40):packages/apple-runner/src/runner-contract.ts:33,436,463runner-contract.ts:42,437capture-kit/src/ios-snapshot-engine/invariants.ts:21,28andtypes.ts:44-45,82platform-apple/src/snapshot-presentation.ts:246-266Two things are kept on purpose:
alert accept's 15 s process bound, as an outer guard over accept's own retry window.controllerWedged's text check, which also sees errors Explore re-wraps itself.Build, typecheck and lint pass.
npm run test:node: 1718 tests. The only failures are the same 29 that fail onmainon this host. The tests build their fixtures from real 0.21.39 envelopes, and 0.21.40 does not change these fields. There's no device run: this removes dead branches and doesn't change what runs on 0.21.40.🤖 Generated with Claude Code
👀 Devin's session did not pick up the findings · 3 findings for you to triage · Follow Devin's work