feat: implement multi-hop pathway traversal for deeper mechanistic reasoning

[bhavyakeerthi3]

Fixes #152

Description

This PR implements recursive traversal in ReactomeTopologyTool. Previously the tool fetched only immediate local context direct inputs, outputs, and one level of preceding events. This enhancement allows the agent to follow both causal and hierarchical links recursively, providing significantly deeper mechanistic context.

Changes

• Recursive Traversal: Added recursive helper to get_flow_context to follow multi-step hops through the Reactome graph
• Hierarchical & Causal Support: Follows both hasEvent (Pathway → Sub-segment) and precedingEvent (Reaction → Upstream trigger)
• Cycle Detection: Prevents infinite loops in circular biological network structures
• Token Safety: max_depth=2 default with per-level breadth cap of 3 events — balances context richness against LLM window limits

Impact

The agent can now answer complex process-oriented queries like "What triggers this cascade?" by tracing causal flow across multiple hops in the Reactome pathway network — rather than seeing only the immediate reaction context.

Verification

• Hierarchical expansion: Apoptosis (R-HSA-109581) correctly expanded into sub-pathways
• Upstream traversal: Causal preceding events traced for reaction R-HSA-111452

Files Affected

• src/tools/reactome_topology.py
• tests/test_flow_reasoning.py

[adamjohnwright]

Staying open as the live version of this work — #144 is closed as superseded by it.

Being straight about where it stands: this needs reworking rather than rebasing, for two separate reasons.

main has moved 140 commits since this branch's base, including the retriever rewrite and the LangChain 1.x migration. Every file this touches has changed underneath it.

src/util/langchain_compat.py should come out. It wraps the AttributeInfo import in three nested try/except blocks and falls back to defining a stub class of the same name if all of them fail. That fallback is the part to remove: if the import is genuinely unavailable, the code proceeds with an object that is not AttributeInfo, and whatever breaks later will give no hint that this is why.

It is also solved already. main fixed this properly during the LangChain 1.x migration, with a single direct import in src/retrievers/reactome/metadata_info.py:

from langchain_classic.chains.query_constructor.schema import AttributeInfo

When this PR was opened those import paths really were moving, so the shim was a fair response to the situation at the time. It just is not the situation any more.

On the feature itself, which is the part worth keeping: multi-hop traversal changes what reaches the LLM, so it needs a before-and-after on the golden questions like every other retrieval change here — see the note on #116 and #133. The evaluator that produces that measurement is being fixed in #211. It also overlaps in intent with specs/007-answer-cascade, which routes mechanistic questions deliberately rather than by retrieval alone, so it is worth reading that spec before reworking — the two should agree on where this kind of question goes.

Thank you for this, and sorry for the long silence.

[adamjohnwright]

The idea in here was right and is now shipped — as reactome-mcp#35, reactome_preceding_events, credited to you.

What was right

Containment was already covered — "what is this pathway made of" — and ordering was not. "What leads up to this" is a different question, it is genuinely mechanistic, and nothing in the system could answer it. You found a real gap.

Multi-hop was also the right instinct. One step back is already sitting in the raw object; the cascade is what has value:

## What happens before R-HSA-69205
### 1 step back
- Cyclin D:CDK4/6 phosphorylates RB1 ... (R-HSA-69227)
### 2 steps back
- CDK4/6:CCND complexes are activated by T-loop phosphorylation (R-HSA-8942836)
- RB1 translocates to the nucleus (R-HSA-9659820)
### 3 steps back
- Translocation of CDK4/6:CCND complexes ... (R-HSA-8942803)

Why it moved rather than merged

ReactomeTopologyTool is a second Content Service client living inside the chatbot, with its own requests.Session and no tests. There is already one in reactome-mcp — 59 tools over the same service, every response shape verified against a live payload, with a weekly sweep that calls all of them. A parallel client is a second place for the same class of bug, and that class is not hypothetical: ten of those tools were reading fields the API never returns, and it took a live sweep to find them.

One thing I learned putting it there, which your version could not have known: followingEvent is not symmetrically available. It is absent at the top level and appears nested as bare dbIds with no stable IDs. So the traversal has to run backwards, and the tool says so — a model asked "what happens after X" needs to know it is getting the answer to a different question.

One thing to carry forward

src/util/langchain_compat.py wraps the AttributeInfo import in three nested try/except blocks and falls back to defining a stub class of the same name. That last branch is the problem: if the import is genuinely unavailable, the code proceeds with an object that is not AttributeInfo, and whatever breaks later gives no hint why. When this PR was opened those import paths really were moving, so it was a fair response at the time. main settled it during the LangChain 1.x migration with a single direct import from langchain_classic.

Worth knowing either way: main has moved 140+ commits since this branch's base, including the retriever rewrite and that migration, so anything from here needs reworking rather than rebasing.

Thank you for this — it is the second of your PRs this week whose idea landed. Sorry it took six months to say so.