feat: add EmbeddingsFilter contextual compression to HybridRetriever …

[bhavyakeerthi3]

🐛Summary

Closes #132

Wraps the HybridRetriever with a ContextualCompressionRetriever using
LangChain's EmbeddingsFilter to discard low-relevance documents before
they reach the LLM.

Changes

• Added EmbeddingsFilter compressor with similarity_threshold=0.76
• Wrapped HybridRetriever with ContextualCompressionRetriever
• Both Reactome and UniProt RAG chains benefit automatically

Impact

• Fewer off-topic documents reach the LLM
• Directly complements Hallucination Grader (PR feat: add hallucination grader and fix web search dead branch #123): better inputs = fewer hallucinations
• Complementary to reranking (PR Add FlashRank reranker to HybridRetriever to improve retrieval quality #116): compression removes noise, reranking orders what remains

[adamjohnwright]

Still open deliberately, and queued rather than ignored — sorry it has been quiet for six months.

This PR changes what reaches the LLM, and this repository's constitution requires a before-and-after measurement on real questions for exactly that kind of change. "This should be better" is explicitly not a finding here, because retrieval quality has no right answer, only "did this move".

The reason there has been no verdict is that the tool which produces that measurement was not trustworthy. src/evaluation/evaluator.py built its own retriever instead of the shipping pipeline, so it measured a configuration that no longer existed; that is fixed. It then scored the raw question rather than the rephrased one, while production always retrieves on rewritten text; that is fixed too. What remained was that a single failed question discarded the entire run, with nothing written to disk until the very end — which is why nobody wanted to run it.

#211 fixes that last part. Once it lands, this PR gets a real answer: the golden questions, before and after, with the noise floor reported, and the result is a number rather than an opinion.

Two things that will need doing first, so they are not a surprise:

• main has moved 140 commits since this branch's base, including the retriever rewrite and the LangChain 1.x migration. This will need reworking, not a straight rebase.
• Add FlashRank reranker to HybridRetriever to improve retrieval quality #116 and feat: add EmbeddingsFilter contextual compression to HybridRetriever … #133 both modify HybridRetriever in src/retrievers/csv_chroma.py, so they conflict with each other. They are two different answers to the same question — whether reranking or filtering the retrieved set improves what the model sees — and the evaluation is what decides between them, or whether both earn their place.

Nothing is needed from you right now. Thank you for the contribution, and for your patience with the delay.

[adamjohnwright]

Measured before reviewing, and the result settles it: at similarity_threshold=0.76 this filter keeps nothing at all.

Run against the current bundle (text-embedding-3-large, Release95) over the first eight golden questions:

EmbeddingsFilter(similarity_threshold=0.76)
  What does CDK5 phosphorylate in Alzheimer's     40 ->   0
  How does TP53 regulate PTEN transcription?      40 ->   0
  What is the role of CDK12 in DNA repair gene    40 ->   0
  Which proteins are in the RNA polymerase II     40 ->   0
  What happens during Golgi fragmentation in n    40 ->   0
  How is oxidative stress handled by peroxired    40 ->   0
  What reactions involve EGFR autophosphorylat    40 ->   0
  Describe the components of the proteasome       40 ->   0
  TOTAL 320 retrieved -> 0 kept (0%)

Merged as written, every question would reach the LLM with no context whatsoever. For comparison, 0.5 keeps 32% and 0.3 keeps 50% — and even 0.3 caps at 20 per question, because EmbeddingsFilter has its own k default that would silently truncate on top of the threshold.

The number isn't wrong through carelessness; it's wrong because a cosine threshold is not portable. 0.76 was presumably reasonable against whatever model and bundle it was tuned on. text-embedding-3-large has a different similarity distribution, and nothing warns you — the retriever returns documents, the filter drops them all, and the chatbot answers from nothing while looking like it worked. This repo's spec on the answer cascade argues the same point about distance thresholds for a different purpose:

the number is arbitrary and drifts with the embedding model. A threshold tuned on Release95 and text-embedding-3-large means nothing after either changes, and nothing warns you.

What was right here, and has already landed

The other half of this PR — doc_lists.extend([bm25_docs, vector_docs]) instead of doc_lists.append(bm25_docs + vector_docs) — was a real bug, and it's on main. Reciprocal rank fusion scores by position within each list, so concatenating the two before fusing threw away the ranking. It landed independently via #169/#170, with per-entity de-duplication added on top. Same insight, arrived at twice.

The problem you were solving is real

Beta transcripts from this week show exactly the noise you were targeting: a question about analysis tools retrieved estrogen-responsive gene expression and RAB GTPase trafficking, and the model wrote a confident answer around them. That is worth fixing.

What it needs is a threshold derived from this corpus rather than carried in — or a mechanism that doesn't have a magic number, which is the open question in specs/007-answer-cascade (D1). The measurement above took minutes and costs pennies; whoever picks this up should start there rather than from a literal.

Also worth knowing before reworking: langchain.retrievers.contextual_compression and langchain.retrievers.document_compressors no longer exist on main. The LangChain 1.x migration moved them to langchain_classic.*.

Closing because it cannot land in this form, not because the idea is wrong. Thank you for it, and sorry it took six months to get a real answer.