Ask the chatbot the questions it has got wrong before

[adamjohnwright]

Every regression this week was found the same way: someone asked beta a question and the answer was wrong.

• the safety checker refusing "can you run gsea for me"
• ordinary retrieval taken down for a day by a shared Chroma Settings object
• a live answer that was correct and displayed nothing

Each was caught by a person noticing, which is slow and only covers questions people happen to ask.

reactome-mcp has a sweep that calls every tool and checks the answers contain what they should. It has caught several real bugs, including one I'd introduced that morning. This is the same idea for the chatbot.

./bin/answer-sweep              # all of them
./bin/answer-sweep --only gsea  # one

11/11 against the deployed container in 151s, exit 0.

must_not earns its place

Most of the failures this replaces produced confident, plausible prose. "Reactome does not provide a specific tool" is a fluent sentence and a false one about the flagship feature. Checking only for presence would have passed every one of them.

Each expectation also records why it's checked, printed on failure — so whoever sees a red line learns what broke last time rather than guessing at intent:

FAIL  13.5s  what species are in reactome
      missing: '96'
      why this is checked: Retrieval answered 'primarily Homo sapiens ... no
      indications of other species'. Reactome has 96. A sample of the content
      cannot describe the scope.
      answered: I could not find out the specific species represented in Reactome
      due to a service error.

It found a real degradation on its first production run

That failure above is genuine. reactome.org served a Cloudflare challenge to a burst of requests, and the species answer degraded — which is the error handling working rather than a regression.

So transient upstream failures are retried once, and the retry is reported. A sweep that cries wolf gets ignored; that's exactly how reactome-mcp's own sweep would have gone had it stayed red from day one. The retry is deliberately narrow — only an exception, or an answer that says the lookup failed.

Not the evaluator

evaluator.py scores answer quality with ragas and costs real money. This asks something cheaper — is the chatbot still doing the thing it was fixed to do — in two and a half minutes, and is meant to run before every deploy.

🤖 Generated with Claude Code

---

What an adversarial review of this PR found

I attacked this file on the grounds that a regression gate nobody has seen fail is not yet a gate. Five things, four of them mine:

The matching comment was false. It claimed "96" would not match "1996". It did — re.escape escapes metacharacters and adds no boundaries. That mattered because the release check was must=("9",), which matches any answer containing a digit 9. The earlier 11/11 was real, but two of those checks were asserting close to nothing.

A retried question reported as a clean pass. retried was set on the Result the retry then threw away, so the (retried once) note could never print — the silent retry the comment right above it promises not to do.

max_seconds was declared and never read, and the docstring advertised a --in-container flag argparse would reject.

Fixing the first one weakened a safety guard. Word boundaries at both ends stopped "consult" matching "consulting your physician" — and "consult" is the must_not guarding the medical-advice answer. The end is now closed only for a number, where a longer one is a different number.

Two checks failed on correct answers. Run against live MCP, the release question failed on "The current version of Reactome is 97" because I required the literal word "release"; the service says "version". And must=("96",) for the species count goes up over time. Both now match a pattern. A check that fails on a correct answer is one that gets switched off.

Skipping, rather than crying wolf

From a plain checkout there is no REACTOME_MCP_URL, so the two live-service questions could never pass — and were reported as regressions under a line reading "a question the chatbot used to get wrong and does again". They're now needs_live and skipped loudly. The skip is deliberately narrow, and a test covers the dangerous direction: that they still run when MCP is configured, so the container never silently stops checking them.

Evidence

• 9/9 passed, 2 skipped from a plain local checkout
• 11/11 passed against the running MCP sibling, with the stricter checks
• 304 tests, ruff and mypy clean
• The two tests that matter were run against the bugs they were written for: each fails when the fix is reverted

The tests drive run() against a stub graph, so they test the harness rather than any answer the real chatbot gives — the sweep can now be shown to fail, which is the only thing that makes a green one mean anything.