Skip to content

Fix a live answer that was correct and invisible - #218

Merged
adamjohnwright merged 2 commits into
mainfrom
fix/live-answer-invisible
Sep 15, 2026
Merged

adamjohnwright merged 2 commits into
mainfrom
fix/live-answer-invisible

Conversation

@adamjohnwright

Copy link
Copy Markdown
Contributor

#216 shipped a live destination that answers correctly and shows the user nothing.

The Chainlit UI displays only what the callback handlers stream — bin/chat-chainlit.py reads chainlit_cb.final_stream and has no path that posts result["answer"]. Callbacks travel in the RunnableConfig, and the live path took no config at all.

Measured through the graph, the way the app calls it:

Q: What species are included in the Reactome database?
   answer present  : True  'Reactome covers a total of 96 species...'
   streamed tokens : 0        <-- what the user sees

After: 103 tokens, streamed text matching the answer exactly.

Three things, one cause

  • config is now threaded from the graph node through the tool loop into every model call — including the forced final answer after the round cap, which is the call that produces the text.
  • A streaming copy of the llm, the same model_copy(update={"streaming": True}) every RAG chain uses. The live path was using the shared non-streaming instance.
  • The unreachable-MCP fallback passed a freshly constructed RunnableConfig() instead of the real one, so even the fallback would have been invisible.

Also fixed

chat_history was a parameter of answer_from_live_services that nothing ever passed, so every live answer was asked cold. A follow-up like "can you tell me?" — a real question from a real session — had no context. Typed Sequence rather than list so a caller holding list[HumanMessage] can pass it without a cast.

Tests

Three regression tests. Two fail against the shipped code:

FAILED test_the_config_reaches_every_model_call
FAILED test_the_config_reaches_the_forced_final_answer_too

How this got merged is worth more than the fix

I tested answer_from_live_services directly, it returned the right text, and I reported it working. The constitution's first principle:

A component test passing is not evidence the feature works... Before claiming something works, run it the way a user or the server would.

I wrote that principle into this repository this week and then didn't follow it. The test that would have caught it is the one that drives the compiled graph with a callback handler and counts tokens — which is what I ran only when reviewing my own merged work.

Not affected

Beta is unaffected: it has no REACTOME_MCP_SERVER set, so the live destination is not offered there at all. Verified both retrievers still work in the running container (user guide 6 hits, reactome 40).

🤖 Generated with Claude Code

adamjohnwright and others added 2 commits September 15, 2026 15:11
#216 shipped a `live` destination that answers these questions correctly and
shows the user nothing.

The Chainlit UI displays only what the callback handlers stream --
`bin/chat-chainlit.py` reads `chainlit_cb.final_stream` and has no path that
posts `result["answer"]`. The callbacks travel in the RunnableConfig. The live
path took no config at all: `_answer_from_live_services(state)` never received
one, `answer_from_live_services` never passed one to any model call, and the
model was the shared non-streaming instance rather than a streaming copy like
every other answer path uses.

Measured through the graph, the way the app calls it:

    Q: What species are included in the Reactome database?
       answer present  : True  'Reactome covers a total of 96 species...'
       streamed tokens : 0        <-- what the user sees

After: 103 tokens, and the streamed text matches the answer exactly.

Three things, one cause:

  - `config` is threaded from the graph node through the tool loop into every
    model call, including the forced final answer after the round cap -- that
    call is the one that produces the text.
  - the live path uses a streaming copy of the llm, the same
    `model_copy(update={"streaming": True})` the RAG chains use.
  - the unreachable-MCP fallback passed a freshly constructed `RunnableConfig()`
    instead of the real one, so even the fallback would have been invisible.

Also fixed while here: `chat_history` was a parameter of
`answer_from_live_services` that nothing ever passed, so every live answer was
asked cold. A follow-up like "can you tell me?" -- which is a real question
from a real session -- had no context at all. Typed `Sequence` rather than
`list` so a caller holding `list[HumanMessage]` can pass it.

Three regression tests; two of them fail against the shipped code.

**How this got merged is worth more than the fix.** I tested
`answer_from_live_services` directly, it returned the right text, and I reported
it working. The constitution's first principle says a component test passing is
not evidence the feature works, and to run it the way a user or the server
would. I wrote that principle into a repository this week and then did not
follow it.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@adamjohnwright
adamjohnwright merged commit f10d975 into main Sep 15, 2026
10 checks passed
@adamjohnwright
adamjohnwright deleted the fix/live-answer-invisible branch September 15, 2026 15:20
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant