Skip to content

Read which precondition a disabled control names - #218

Open
DavertMik wants to merge 1 commit into
mainfrom
fix/pilot-disabled-cause
Open

DavertMik wants to merge 1 commit into
mainfrom
fix/pilot-disabled-cause

Conversation

@DavertMik

Copy link
Copy Markdown
Contributor

Pilot had one explanation for a disabled control:

- Click failed + button in "disabled buttons" → required field missing. Instruct fill first.

A form can withhold its action for reasons that are not a field at all — an empty selection, an unmet count, a choice made elsewhere — and the page usually says which. The control stating the unsatisfied constraint is itself in the disabled list, and active form already marks the fields the form requires. Both were sitting in the same <state> block Pilot reads.

With one cause on offer, Pilot attributed the block to whichever field the running scenario happened to be about, then steered the whole test at that field, hunting validation feedback the app had no reason to show. A scenario premised on a field being required survived contact with a page that marks the field optional, and the run was judged on a guard that does not exist. Pilot's own reasoning surfaced the real constraint mid-run and dropped it — the rule gave it nowhere to take that.

Change

One diagnostic bullet in getSystemPrompt:

- Scenario's target control in "disabled buttons" → a precondition is unmet; identify which before acting.
  Other disabled controls often name the unsatisfied constraint; "active form" marks [required] fields.
  Aim Tester at the constraint the page names, not the one the scenario assumed — note the difference in PROGRESS.

The trigger widens from Click failed + to the target control appearing disabled, because the attribution forms before any click is attempted, and stays scoped to the scenario's target so unrelated disabled controls elsewhere on a page do not fire it.

Trade-off

A form whose disabled control is generically named and which marks required fields only visually gives Pilot nothing to read; it falls back to the scenario's field exactly as today. The rule helps where the app states its constraint, and cannot invent one where it does not.

Not in this PR

buildVerdictSystemPrompt is a separate prompt from getSystemPrompt and never received the negative-goal rule added in #215, so the final verdict still reasons on the predicted guard. The two prompts diverging is worth its own change.

Verification

  • bun test tests/unit/ — 1415 pass, 0 fail
  • bun test tests/integration/ — 149 pass, 0 fail
  • bun run format, bun run lint:fix — clean

🤖 Generated with Claude Code

Pilot had one explanation for a disabled control: a required field is missing,
fill it. A form can withhold its action for reasons that are not a field at all
— an empty selection, an unmet count, a choice made elsewhere — and the page
usually says which: the control that states the unsatisfied constraint is
itself in the disabled list, and the form marks the fields it actually
requires.

With a single cause on offer, Pilot attributed the block to whichever field the
running scenario happened to be about. It then steered the entire test at that
field, looking for validation feedback the app had no reason to show, while the
real constraint sat unread in the same state block it was given. A scenario
premised on a field being required survived contact with a page that marks the
field optional, and the run was judged on a guard that does not exist.

Pilot now identifies which precondition is unmet before acting on it, and
reports the difference when the page names a constraint the scenario did not
predict.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants