Skip to content

fix(control): refuse discovery with a reason when the screen can't be read - #48

Open
easyhooon wants to merge 2 commits into
toss:mainfrom
easyhooon:fix/control-discovery-unavailable
Open

easyhooon wants to merge 2 commits into
toss:mainfrom
easyhooon:fix/control-discovery-unavailable

Conversation

@easyhooon

@easyhooon easyhooon commented Nov 18, 2026 •

Copy link
Copy Markdown
Contributor

Summary

Resolves #45

Discovery (actionTargets, readAccessibility) returned an empty list when the screen couldn't be read — locked, in the background, covered by a dialog, or mid-action. It now fails with OPERATION_UNAVAILABLE and the reason on Android and iOS, so an empty list again means "no controls". The output schema is unchanged, and perform still doesn't refuse.

Also changed:

  • The iOS NectoUIControlPlugin initializer now takes throwing closures, so a custom runtime can report the reason too. Existing non-throwing closures still compile.
  • The Control panel keeps "Input sent" when the screen after an action can't be read, and shows the reason in the list, since the action itself was delivered.
  • The actionTargets and readAccessibility manifest descriptions say when they fail and what an empty list means, so an agent knows before calling.

Changelog

Control: discovery reports why the screen can't be read instead of returning an empty list.

Test Plan

  • script/build, script/test: all passed
  • SDK/Android/gradlew -p SDK/Android build: passed (NectoAndroidControlRuntimeTest 17/17)
  • script/test e2e on iPhone 17 Pro, iOS 26.2: 9 tests passed
  • node script/tests/control-e2e.mjs on the same simulator, which script/test e2e does not run:
PASS SwiftUI: tab, tap, Unicode replace/append/clear, navigation, edge-back, swipe, stale ID
PASS multi-tap: 1–5 simultaneous fingers × 1–3 successive taps
PASS positioned tap: target-relative and screen-relative coordinates
PASS UIKit: navigation bar, accessible gesture, input, secure value redaction, push/back, swipe
PASS discovery during an action: refused with the reason
PASS sheet: covered background is excluded from targets and content

New tests cover each refusal reason on Android, and discovery during a swipe on both Android and iOS (e2e). With the in-progress check removed from discovery, both the Android test and the iOS e2e check fail.

On a Galaxy S25+ (Android, USB) with the sample app, both actionTargets and readAccessibility:

State Result
App in front 9 targets, 14 items
Notification shade pulled down OPERATION_UNAVAILABLE: Another window, such as a dialog or the notification shade, has input focus. …
Home pressed (app in background) OPERATION_UNAVAILABLE: The app is not in the foreground, is covered by a system dialog, or the screen is locked. …
Screen locked Same as above

Both recovered to the full list once the app was back in front.

On the iOS 26.2 simulator, the app in front lists targets as before. In the background or behind the lock screen, iOS usually suspends the app within seconds, so the CLI reports TIMEOUT rather than the reason; the reason appears only while the app is still running. Either way the result is an error, not an empty list.

Not verified on an iPhone.

… read

control.actionTargets and control.readAccessibility returned an empty
list when the runtime could not read the app's window: the screen was
locked, the app was in the background, a dialog or the notification
shade had focus, or an action was in progress. That looked the same as
a screen without controls, so an agent had no hint to unlock the device
or bring the app back.

Discovery now throws operationUnavailable with the reason, as the
action operations already do. The output schema is unchanged, and an
empty list again means the screen has no controls. perform still reads
the screen without refusing, so an action that moves the app to the
background is reported as dispatched.

The iOS NectoUIControlPlugin initializer now takes throwing closures.
Existing non-throwing closures still compile.

The panel keeps "Input sent" after an action whose next screen can't be
read, and shows the reason in the list. The Control e2e retries a
content read that fails while the screen is in transition.

Refs toss#45
Discovery refuses with "Another action is in progress" while an action
runs, but no test exercised that path. Android gets a Robolectric test
that lists targets mid-swipe, and the Control e2e lists targets during a
slow swipe on iOS, where the UIKit runtime has no unit tests.

The actionTargets and readAccessibility descriptions now say that an
unreadable screen fails with OPERATION_UNAVAILABLE and the reason, and
that an empty list means nothing is there, so an agent knows before it
calls.

Refs toss#45
@easyhooon
easyhooon marked this pull request as ready for review November 18, 2026 06:39

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Control: return an error instead of an empty list when the app window can't be read

1 participant