Skip to content

Ground capability checks on the outcome, not a planned step's label - #213

Merged
DavertMik merged 1 commit into
mainfrom
fix/capability-grounding-scope
Sep 15, 2026
Merged

DavertMik merged 1 commit into
mainfrom
fix/capability-grounding-scope

Conversation

@DavertMik

Copy link
Copy Markdown
Contributor

A plan step names a control before any page has been seen. When that exact label turned out not to exist, capabilityGroundingRule read it as "the feature is missing" and authorized a stop — even with a working control for the same outcome sitting in ARIA, in the UI map and in Pilot's own state block.

What happened

Session IllegalSorryBeige706, scenario "Create a sibling suite from an existing suite node". The plan's step 3 was Click Create next suite — a label that exists nowhere on the page in any form. Pilot turned it into a gate:

Choose Create next suite only if explicitly listed; if absent, stop and report that the required capability is unavailable.

Meanwhile textbox "Add new suite", button "Suite" and link "New suite" were in the Tester's <page_aria>, in the Researcher UI map, and in the <state> block of both Pilot's guidance call and the stop-review that failed the test.

The Tester found the right control and discarded it on the record:

maybe via the "New suite" link... But the plan said only if Create next suite visible else stop.

Three failed clicks hunting a label that never existed, then stop() at 93s.

The change

One line, src/ai/rules.ts. The rule never separated a capability the scenario names from a control label a plan step predicted sight-unseen, so "do not replace a requested action with a synonym" and "record the missing capability and stop" both bound to the prediction.

Ground on the scenario's outcome, not a planned step's control label — a missing label is not a missing capability.

It sits under the paragraph defining what grounding binds to, so it scopes that definition and redefines "not available" for the stop rule below without touching it. Planner, Pilot and Tester all read this same string, so one insertion covers every agent that drove the stop.

Trade-off

Grounding gets weaker where it was strongest: a Tester that cannot find the scenario's action may now substitute a control it argues reaches "the same outcome". False passes that the strict reading blocked fall to the goal-rewrite paragraph in the same rule instead.

Verification

  • bun test tests/integration/ — 146 pass, 0 fail
  • bun test tests/unit/ — 1407 pass, 0 fail
  • bun run format — clean

No test asserts on this rule string. Worth a regression label before merge — this loosens a guardrail that exists to prevent false passes, and only a real run exercises that.

🤖 Generated with Claude Code

https://claude.ai/code/session_01S8G4zEAoemvTAjFLJSMfWc

A plan step names a control before any page has been seen. When that exact
label was absent, capability grounding read it as "the feature is missing"
and authorized a stop, even with a working control for the same outcome
visible in ARIA, in the UI map and in Pilot's own state block.

Grounding now binds to the outcome the scenario names; a predicted label is
a route, not a requirement.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01S8G4zEAoemvTAjFLJSMfWc
@DavertMik
DavertMik merged commit 71ffcc8 into main Sep 15, 2026
2 checks passed
@DavertMik
DavertMik deleted the fix/capability-grounding-scope branch September 15, 2026 23:09
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants