Skip to content

dismiss-keyboard never taps the return key, and the model sees agent-device's hints - #91

Open
thymikee wants to merge 10 commits into
expo:mainfrom
thymikee:agent-device-keyboard-status
Open

thymikee wants to merge 10 commits into
expo:mainfrom
thymikee:agent-device-keyboard-status

Conversation

@thymikee

@thymikee thymikee commented Nov 19, 2026 •

Copy link
Copy Markdown

Summary

Fixes #88. We want Explore to use agent-device directly, without workarounds. This removes one. When agent-device refused keyboard dismiss with keyboard_dismiss_unavailable, dismiss-keyboard found the return key in the tree and tapped it. That key never only closes the keyboard. On next it moves focus, and on done or go it submits the form.

From 0.21.40 (pinned in #87), agent-device dismisses no visible iOS keyboard itself. Instead, its refusal tells the caller what to do. Explore now passes that on rather than guessing:

  • dismiss-keyboard sends keyboard dismiss and nothing else. A refusal reaches the model unchanged.
  • A failed action shows the model agent-device's hint after the message. Before this, the model saw only the message, for every action. The hint also covers back on a modal.
before: dismiss-keyboard → Explore taps return → form submitted → Home ("Hi, Review")
after:  "keyboard: The Apple runner cannot dismiss this keyboard automatically Next: Inspect the
         screen and press the visible Hide Keyboard key or an app control that ends editing. …"
        → still on sign-up

This deletes the coordinate tap, KEYBOARD_RETURN_KEY and the multi-line guard. 6 files changed.

No change to usage.

Stacked on #87: review after it. The diff includes #87's commits until it merges.

Validation

Tested at head 3813b5a (the bridge hint in 68d2d18 is covered by unit tests) with explore:binary --simulator local on an iPhone 17 Pro (iOS 27.0). The Wallet plan opens sign-up, fills name, email and password, then runs an optional dismiss-keyboard and a capture:

#87 (0.21.40) this PR
dismiss-keyboard "succeeds" by tapping return refused, and the model gets the hint
Final capture Home, account created sign-up, password filled, keyboard up

build, typecheck and eslint pass. test:node: 1692 pass. The 29 failures are the known "evidence comparison did not run" set. Explore's CI needs maintainer approval for fork PRs.

🤖 Generated with Claude Code


🟢 Devin Review found nothing to fix · View findings · Follow Devin's work

Devin Review

thymikee and others added 5 commits November 17, 2026 15:31
…rol back names

0.21.39 no longer says "in-app back control is not available" or "Unable to dismiss the iOS
keyboard", so the fallbacks Explore keyed on those messages would never run. They now read
error.details.reason:

- back: on iOS, agent-device presses the navigation bar back button, else swipes from the left
  edge, and refuses with back_swipe_no_change when nothing changed. It lists the screen's dismiss
  controls (Close, Cancel, Back) in details.dismissControls without pressing them. Explore presses
  the one it names when its label is safe by Explore's destructive and external rules, requires
  the destination to change, and otherwise swipes a sheet down as before.
- keyboard dismiss refuses with keyboard_dismiss_unavailable; the return-key fallback keys on it.
- the remote /health refusal is remote_daemon_unavailable with dispatched "no", which is what
  Explore inferred from the message before resending.

The other controller's bridge gives the same reasons.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
As in source-driven capture, explore:binary compares the screen before and after it presses the
dismiss control agent-device named, and fails the back with agent-device's refusal when nothing
changed.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ipes after a press

A modal kept in a screen's own state keeps the route when its Close control dismisses it, so the
destination check alone read that as a failure and fell through to the sheet swipe-down with the
geometry read before the press. A press now counts when the route changes or the pressed control
is gone. If neither happens, back fails with AGENT_DISMISS_CONTROL_FAILED instead of swiping.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…eaving the screen

The pressed control disappearing counted as back, but a confirmation such as "Discard changes?"
also hides it while the screen stays. The control leaving now counts only when no alert took its
place; otherwise back fails with AGENT_DISMISS_CONTROL_FAILED.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A "Discard changes?" confirmation after the press changes the screen, and can hide the route
observer, so either signal alone read it as having left. Both code paths now require that no
alert is showing before a press counts as back.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@chatgpt-codex-connector

chatgpt-codex-connector Bot commented Nov 19, 2026 •

Copy link
Copy Markdown
Codex Review Summary: ✅ Completed

This comment shows the latest Codex review activity on this pull request.

Review Status Commit Review trigger
📝 Code Review ✅ Completed 2026-11-19T15:27:56.080162Z 0d7c688 PR opened
ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

devin-ai-integration[bot]

This comment was marked as resolved.

agent-device names a dismiss control by its identifier when it has
one, and its label may then be empty. The press check matched only
the label, so a button with no label was never seen and the press
counted as leaving even when the modal stayed. It now matches the
control the way agent-device named it, and the control leaving counts
only when it was on screen before the press.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@thymikee
thymikee force-pushed the agent-device-keyboard-status branch from 0d7c688 to be04540 Compare November 19, 2026 15:28
thymikee and others added 2 commits November 19, 2026 17:03
0.21.40 adds the keyboard fix found while testing this bump: keyboard
dismiss no longer presses the return key, so on a field whose return
key is Done it refuses with keyboard_dismiss_unavailable instead of
submitting the form. The typed reasons Explore reads are unchanged.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
When agent-device reports keyboard_dismiss_unavailable, the fallback
tapped the keyboard's return key by coordinates. That key never only
closes the keyboard: next moves focus to the following field, and
done or go submits the form (expo#88). The step now fails with
KEYBOARD_DISMISS_UNAVAILABLE and names the app's own controls as the
next action. This also removes the multi-line editor guard and the
return-key label pattern, which only served the coordinate tap.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@thymikee
thymikee force-pushed the agent-device-keyboard-status branch from be04540 to 20cc5fc Compare November 19, 2026 16:28
@thymikee thymikee changed the title Keep dismiss-keyboard from submitting forms: pin agent-device 0.21.40, drop the return-key tap Keep dismiss-keyboard from submitting forms: drop the return-key tap Nov 19, 2026
The model read only an error's message, so agent-device's next step
never reached it: how to leave a modal back cannot close, or that a
visible iOS keyboard needs its Hide Keyboard key or an app control
that ends editing. A failed action now reads the message followed by
that hint. dismiss-keyboard passes agent-device's refusal through as
it is, instead of rewording it.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@thymikee thymikee changed the title Keep dismiss-keyboard from submitting forms: drop the return-key tap dismiss-keyboard never taps the return key, and the model sees agent-device's hints Nov 19, 2026
devin-ai-integration[bot]

This comment was marked as resolved.

The other controller's bridge refused keyboard dismiss without a hint,
so the model got generic advice there. Its refusals now give the same
next step as agent-device's.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

dismiss-keyboard presses the return key, which submits a form on its last field

1 participant