Report a failed verification model call as a failure

[DavertMik]

Found while investigating a failed Tester session (Langfuse trace 078285ca5b32d556be2695c50141c5c8, session VariableIncreasedBrown198). Around 6 of the run's 13 minutes went into verify(), and three of the long ones returned "No assertion could express this claim" — for claims that passed minutes later.

The bug

verifyState wrapped its loop in a handler that swallowed everything:

catch: async (error) => {
  debugLog(error);
},

invokeConversation throws, so a rate-limited or aborted model call landed there, was logged, and the loop moved on with codeBlocks still empty. That leaves totalAttempted === 0, which the next lines read as:

const inexpressible = !alreadyVerified && totalAttempted === 0;

So an infrastructure failure came back to Tester as a verdict about the page, with a suggestion pointing the wrong way: "This is not evidence the page is wrong — restate it." Tester duly reworded an assertion that was never the problem.

Two further costs:

• It amplified what it hid. Each iteration retried the model, so one claim spent verifyAttempts x the retries the provider already performs — all against the token limit that had just been hit. In the trace, one verify() produced 9 rate-limit errors and ran for 2m28s; another for 3m41s.
• It ate fatal browser errors. action.attempt rethrows those deliberately; the handler caught them and carried on.

The fix

Drop the handler. loop() rethrows when none is given, and handles StopError before it, so the stop() calls in verifyState are unaffected. Both callers already cope:

• tools.ts — the verify tool's own catch reports Verify tool failed: <error> after throwIfFatalBrowserError
• pilot.ts — guards its call with .catch(() => null)

inexpressible now means only what it says: the model answered without usable assertion code.

Also makes the cached early return match the declared return type — it omitted inexpressible and results, which tsc --noCheck never flagged. tsc --noEmit is clean on this file now.

Tests

New tests/unit/navigator-verify-failure.test.ts: a throwing model call must reject rather than resolve as inexpressible (fails on main), and a model answer with no assertion code must still resolve as inexpressible. Existing navigator unit tests and tests/integration/ pass.

Related

Stacks naturally with #209, which fixes a separate cause of the same inexpressible symptom.

🤖 Generated with Claude Code

https://claude.ai/code/session_01JCYmFZvUsENZBx71Yqojs6