Conversation
extractCodeBlocks matched an opening fence only when the info string was js, javascript or empty, so a ```html block never matched as an opener. The scan then paired that block's closing fence with the next block's closer, shifting every later fence by one and producing text that failed the JavaScript parse guard. One markup snippet in an answer was therefore enough to drop every assertion after it. Navigator read that as zero usable code blocks and reported the claim as inexpressible, which sent Tester off to reword an assertion that had been correct. Match any info string and keep the block only when it names JavaScript, so a foreign block is consumed and skipped instead of desynchronising the scan. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01JCYmFZvUsENZBx71Yqojs6
…tion # Conflicts: # CHANGELOG.md
DenysKuchma
approved these changes
Sep 15, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Found while investigating a failed Tester session (Langfuse trace
078285ca5b32d556be2695c50141c5c8, sessionVariableIncreasedBrown198), where fourverify()calls came back as "No assertion could express this claim" for claims the model had, in fact, answered correctly.The bug
extractCodeBlocksmatched an opening fence only when the info string wasjs,javascriptor empty:/```(?:js|javascript)?\s*\n([\s\S]*?)\n```/gA
```htmlopener never matches — after the backticks the pattern needs\n, and findsh. The scan then walks forward and pairs that block's closing fence with the next block's closer, so every later fence shifts by one. The captured text is prose plus a nested fence, which fails thenew Function('I', code)guard and is dropped.One markup snippet in an answer is therefore enough to drop every assertion after it.
What it cost in the trace
Navigator wrote three valid assertions and got zero back, so
totalAttempted === 0andverifyStatereturnedinexpressible. Tester was told the claim was unexpressible and rewrote an assertion that had been fine. Reproduced against the real response from that trace:The fix
Match any info string, then keep the block only when it names JavaScript. A foreign block is consumed and skipped, so the scan stays in sync. The language filter also stops a
```textblock whose content happens to parse as JavaScript from becoming an assertion.Tests
New
tests/unit/code-extractor.test.tscovers the js block, the unlabelled block, js blocks after a foreign block, a foreign block whose content parses as JS, and a js block that is not valid JS. The last test in that file fails onmain.tests/unit/research-parser.test.ts(which feeds research markdown through the same function) still passes.🤖 Generated with Claude Code
https://claude.ai/code/session_01JCYmFZvUsENZBx71Yqojs6