Ground capability checks on the outcome, not a planned step's label - #213
Merged
Merged
Conversation
A plan step names a control before any page has been seen. When that exact label was absent, capability grounding read it as "the feature is missing" and authorized a stop, even with a working control for the same outcome visible in ARIA, in the UI map and in Pilot's own state block. Grounding now binds to the outcome the scenario names; a predicted label is a route, not a requirement. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01S8G4zEAoemvTAjFLJSMfWc
DenysKuchma
approved these changes
Sep 15, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
A plan step names a control before any page has been seen. When that exact label turned out not to exist,
capabilityGroundingRuleread it as "the feature is missing" and authorized a stop — even with a working control for the same outcome sitting in ARIA, in the UI map and in Pilot's own state block.What happened
Session
IllegalSorryBeige706, scenario "Create a sibling suite from an existing suite node". The plan's step 3 wasClick Create next suite— a label that exists nowhere on the page in any form. Pilot turned it into a gate:Meanwhile
textbox "Add new suite",button "Suite"andlink "New suite"were in the Tester's<page_aria>, in the Researcher UI map, and in the<state>block of both Pilot's guidance call and the stop-review that failed the test.The Tester found the right control and discarded it on the record:
Three failed clicks hunting a label that never existed, then
stop()at 93s.The change
One line,
src/ai/rules.ts. The rule never separated a capability the scenario names from a control label a plan step predicted sight-unseen, so "do not replace a requested action with a synonym" and "record the missing capability and stop" both bound to the prediction.It sits under the paragraph defining what grounding binds to, so it scopes that definition and redefines "not available" for the stop rule below without touching it. Planner, Pilot and Tester all read this same string, so one insertion covers every agent that drove the stop.
Trade-off
Grounding gets weaker where it was strongest: a Tester that cannot find the scenario's action may now substitute a control it argues reaches "the same outcome". False passes that the strict reading blocked fall to the goal-rewrite paragraph in the same rule instead.
Verification
bun test tests/integration/— 146 pass, 0 failbun test tests/unit/— 1407 pass, 0 failbun run format— cleanNo test asserts on this rule string. Worth a
regressionlabel before merge — this loosens a guardrail that exists to prevent false passes, and only a real run exercises that.🤖 Generated with Claude Code
https://claude.ai/code/session_01S8G4zEAoemvTAjFLJSMfWc