Conversation
Pilot had one explanation for a disabled control: a required field is missing, fill it. A form can withhold its action for reasons that are not a field at all — an empty selection, an unmet count, a choice made elsewhere — and the page usually says which: the control that states the unsatisfied constraint is itself in the disabled list, and the form marks the fields it actually requires. With a single cause on offer, Pilot attributed the block to whichever field the running scenario happened to be about. It then steered the entire test at that field, looking for validation feedback the app had no reason to show, while the real constraint sat unread in the same state block it was given. A scenario premised on a field being required survived contact with a page that marks the field optional, and the run was judged on a guard that does not exist. Pilot now identifies which precondition is unmet before acting on it, and reports the difference when the page names a constraint the scenario did not predict. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
DenysKuchma
approved these changes
Sep 15, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Pilot had one explanation for a disabled control:
A form can withhold its action for reasons that are not a field at all — an empty selection, an unmet count, a choice made elsewhere — and the page usually says which. The control stating the unsatisfied constraint is itself in the disabled list, and
active formalready marks the fields the form requires. Both were sitting in the same<state>block Pilot reads.With one cause on offer, Pilot attributed the block to whichever field the running scenario happened to be about, then steered the whole test at that field, hunting validation feedback the app had no reason to show. A scenario premised on a field being required survived contact with a page that marks the field optional, and the run was judged on a guard that does not exist. Pilot's own reasoning surfaced the real constraint mid-run and dropped it — the rule gave it nowhere to take that.
Change
One diagnostic bullet in
getSystemPrompt:The trigger widens from
Click failed +to the target control appearing disabled, because the attribution forms before any click is attempted, and stays scoped to the scenario's target so unrelated disabled controls elsewhere on a page do not fire it.Trade-off
A form whose disabled control is generically named and which marks required fields only visually gives Pilot nothing to read; it falls back to the scenario's field exactly as today. The rule helps where the app states its constraint, and cannot invent one where it does not.
Not in this PR
buildVerdictSystemPromptis a separate prompt fromgetSystemPromptand never received the negative-goal rule added in #215, so the final verdict still reasons on the predicted guard. The two prompts diverging is worth its own change.Verification
bun test tests/unit/— 1415 pass, 0 failbun test tests/integration/— 149 pass, 0 failbun run format,bun run lint:fix— clean🤖 Generated with Claude Code