Fix a live answer that was correct and invisible - #218
Merged
Merged
Conversation
#216 shipped a `live` destination that answers these questions correctly and shows the user nothing. The Chainlit UI displays only what the callback handlers stream -- `bin/chat-chainlit.py` reads `chainlit_cb.final_stream` and has no path that posts `result["answer"]`. The callbacks travel in the RunnableConfig. The live path took no config at all: `_answer_from_live_services(state)` never received one, `answer_from_live_services` never passed one to any model call, and the model was the shared non-streaming instance rather than a streaming copy like every other answer path uses. Measured through the graph, the way the app calls it: Q: What species are included in the Reactome database? answer present : True 'Reactome covers a total of 96 species...' streamed tokens : 0 <-- what the user sees After: 103 tokens, and the streamed text matches the answer exactly. Three things, one cause: - `config` is threaded from the graph node through the tool loop into every model call, including the forced final answer after the round cap -- that call is the one that produces the text. - the live path uses a streaming copy of the llm, the same `model_copy(update={"streaming": True})` the RAG chains use. - the unreachable-MCP fallback passed a freshly constructed `RunnableConfig()` instead of the real one, so even the fallback would have been invisible. Also fixed while here: `chat_history` was a parameter of `answer_from_live_services` that nothing ever passed, so every live answer was asked cold. A follow-up like "can you tell me?" -- which is a real question from a real session -- had no context at all. Typed `Sequence` rather than `list` so a caller holding `list[HumanMessage]` can pass it. Three regression tests; two of them fail against the shipped code. **How this got merged is worth more than the fix.** I tested `answer_from_live_services` directly, it returned the right text, and I reported it working. The constitution's first principle says a component test passing is not evidence the feature works, and to run it the way a user or the server would. I wrote that principle into a repository this week and then did not follow it. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This was referenced Sep 15, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
#216 shipped a
livedestination that answers correctly and shows the user nothing.The Chainlit UI displays only what the callback handlers stream —
bin/chat-chainlit.pyreadschainlit_cb.final_streamand has no path that postsresult["answer"]. Callbacks travel in theRunnableConfig, and the live path took no config at all.Measured through the graph, the way the app calls it:
After: 103 tokens, streamed text matching the answer exactly.
Three things, one cause
configis now threaded from the graph node through the tool loop into every model call — including the forced final answer after the round cap, which is the call that produces the text.model_copy(update={"streaming": True})every RAG chain uses. The live path was using the shared non-streaming instance.RunnableConfig()instead of the real one, so even the fallback would have been invisible.Also fixed
chat_historywas a parameter ofanswer_from_live_servicesthat nothing ever passed, so every live answer was asked cold. A follow-up like "can you tell me?" — a real question from a real session — had no context. TypedSequencerather thanlistso a caller holdinglist[HumanMessage]can pass it without a cast.Tests
Three regression tests. Two fail against the shipped code:
How this got merged is worth more than the fix
I tested
answer_from_live_servicesdirectly, it returned the right text, and I reported it working. The constitution's first principle:I wrote that principle into this repository this week and then didn't follow it. The test that would have caught it is the one that drives the compiled graph with a callback handler and counts tokens — which is what I ran only when reviewing my own merged work.
Not affected
Beta is unaffected: it has no
REACTOME_MCP_SERVERset, so thelivedestination is not offered there at all. Verified both retrievers still work in the running container (user guide 6 hits, reactome 40).🤖 Generated with Claude Code