diff --git a/CHANGELOG.md b/CHANGELOG.md index ac5a9a00..4f9f6a06 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -2,6 +2,13 @@ ## 2026-09-15 +### Changes + +- [Pilot] A test that stops part-way through a multi-step flow is no longer failed outright when the + control for the next step is still on the page. Pilot now tells the tester which step is missing and + lets it continue, so flows that pass through a dialog, confirmation, or wizard step reach their goal + instead of ending at the intermediate screen. A test is still failed when nothing on the page can take + it further. ### Fixes - [Researcher] A list that already shows everything it holds is no longer described as paginated. The page diff --git a/src/ai/pilot.ts b/src/ai/pilot.ts index fe9e5cb3..8a0b00f1 100644 --- a/src/ai/pilot.ts +++ b/src/ai/pilot.ts @@ -152,14 +152,6 @@ export class Pilot implements Agent { ${sessionLog || 'No actions recorded'} - Decide and commit. "continue" extends the loop and burns iterations — choose it only when - evidence is genuinely insufficient to call pass/fail, not as a safety hedge. - - "pass" if final state proves the SCENARIO GOAL is accomplished. Set requestVerification. - - "fail" if scenario was attempted but goal not achieved. - - "skipped" if scenario is irrelevant/inapplicable, OR systematic infrastructure failures. - - "continue" only when a concrete missing piece of evidence (a verify/see) would change your verdict. - - Mixed evidence + final state shows success → pass. Mixed + final state unclear → continue with guidance. - When deciding "pass", you MUST also set requestVerification to a one-sentence natural-language claim about the current page (e.g., "New item Foo is visible in the items list"). NOT code — do not write I.*, expect(), .then(), or any JavaScript. Choose the strongest single @@ -401,7 +393,7 @@ export class Pilot implements Agent { private buildVerdictSystemPrompt(task: Test): string { return dedent` You are Pilot — final decision maker for test pass/fail. Review the evidence and commit to a - verdict; "continue" only when evidence is genuinely insufficient. + verdict. ${capabilityGroundingRule} @@ -415,10 +407,11 @@ export class Pilot implements Agent { DOM assertion can't be made. Do not pass when Tester achieved only a related navigation/filter/tab/status outcome instead of the requested action, workflow, or entity detail goal. - - "fail": scenario was attempted but the goal was not achieved. + - "fail": goal not achieved and no further step toward it is available on the current page. - "skipped": scenario is irrelevant to the app, OR systematic infrastructure failures (LLM errors, crashes) prevented testing. NOT for "test failed to interact" — that's "fail" or "continue". - - "continue": tester hasn't completed the goal; provide concrete guidance (which tool, what to check). + - "continue": goal incomplete but the control for the NEXT step is present on the current page, or a + concrete missing check would change your verdict. Guidance must name that step. If a verify() asserted a state that was ALREADY TRUE before the test, it proves nothing — reject. reason field: one short sentence, maximum 120 characters. Do NOT restate the decision