From dbb250a95071422172b8d06bbd0e6d5114868f3b Mon Sep 17 00:00:00 2001
From: DavertMik <davert@testomat.io>
Date: Tue, 15 Sep 2026 11:25:39 +0300
Subject: [PATCH] fix(pilot): allow continue when the next step's control is on
 the page
MIME-Version: 1.0
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 8bit

Pilot's verdict prompt defined "fail" as "scenario was attempted but the
goal was not achieved" and "continue" as "tester hasn't completed the
goal" — indistinguishable for an incomplete flow. Three further clauses
narrowed "continue" to an evidence gap only, so a tester that stopped one
action short of the goal, with the remaining control visible and enabled,
could only be failed.

Make the two verdicts exact complements: "fail" when no further step
toward the goal is available on the current page, "continue" when the
control for the next step is present (guidance must name it) or a missing
check would change the verdict. Drop the duplicated decision list from the
user message, which restated the system prompt with the narrower wording.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
---
 CHANGELOG.md    | 10 ++++++++++
 src/ai/pilot.ts | 15 ++++-----------
 2 files changed, 14 insertions(+), 11 deletions(-)

diff --git a/CHANGELOG.md b/CHANGELOG.md
index 054ca15e..ea4d7f3e 100644
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@@ -1,5 +1,15 @@
 # Changelog
 
+## 2026-09-15
+
+### Changes
+
+- [Pilot] A test that stops part-way through a multi-step flow is no longer failed outright when the
+  control for the next step is still on the page. Pilot now tells the tester which step is missing and
+  lets it continue, so flows that pass through a dialog, confirmation, or wizard step reach their goal
+  instead of ending at the intermediate screen. A test is still failed when nothing on the page can take
+  it further.
+
 ## 2026-09-14
 
 ### Changes
diff --git a/src/ai/pilot.ts b/src/ai/pilot.ts
index 2421e77f..559fba71 100644
--- a/src/ai/pilot.ts
+++ b/src/ai/pilot.ts
@@ -152,14 +152,6 @@ export class Pilot implements Agent {
       ${sessionLog || 'No actions recorded'}
       </session_log>
 
-      Decide and commit. "continue" extends the loop and burns iterations — choose it only when
-      evidence is genuinely insufficient to call pass/fail, not as a safety hedge.
-      - "pass" if final state proves the SCENARIO GOAL is accomplished. Set requestVerification.
-      - "fail" if scenario was attempted but goal not achieved.
-      - "skipped" if scenario is irrelevant/inapplicable, OR systematic infrastructure failures.
-      - "continue" only when a concrete missing piece of evidence (a verify/see) would change your verdict.
-      - Mixed evidence + final state shows success → pass. Mixed + final state unclear → continue with guidance.
-
       When deciding "pass", you MUST also set requestVerification to a one-sentence natural-language
       claim about the current page (e.g., "New item Foo is visible in the items list"). NOT
       code — do not write I.*, expect(), .then(), or any JavaScript. Choose the strongest single
@@ -401,7 +393,7 @@ export class Pilot implements Agent {
   private buildVerdictSystemPrompt(task: Test): string {
     return dedent`
       You are Pilot — final decision maker for test pass/fail. Review the evidence and commit to a
-      verdict; "continue" only when evidence is genuinely insufficient.
+      verdict.
 
       ${capabilityGroundingRule}
 
@@ -415,10 +407,11 @@ export class Pilot implements Agent {
         DOM assertion can't be made.
         Do not pass when Tester achieved only a related navigation/filter/tab/status outcome instead of the
         requested action, workflow, or entity detail goal.
-      - "fail": scenario was attempted but the goal was not achieved.
+      - "fail": goal not achieved and no further step toward it is available on the current page.
       - "skipped": scenario is irrelevant to the app, OR systematic infrastructure failures (LLM errors,
         crashes) prevented testing. NOT for "test failed to interact" — that's "fail" or "continue".
-      - "continue": tester hasn't completed the goal; provide concrete guidance (which tool, what to check).
+      - "continue": goal incomplete but the control for the NEXT step is present on the current page, or a
+        concrete missing check would change your verdict. Guidance must name that step.
         If a verify() asserted a state that was ALREADY TRUE before the test, it proves nothing — reject.
 
       reason field: one short sentence, maximum 120 characters. Do NOT restate the decision