Add the disease_variants collection, built from the release file

[adamjohnwright]

The chatbot answers about disease one level above the question. Asked to list the ABCA1 variants in Reactome it names none of the six and says "Defective ABCA1 causes Tangier Disease" instead.

	before	after
ABCA1	"Defective ABCA1 causes Tangier Disease… Other diseases may be associated with ABCA1, bu[t]"	C1417R, Q537R, N935S, R587W, S1446L, each with its disease and loss-of-function status
PTEN	"PTEN Loss of Function in Cancer" — a pathway	Q17*, Q97*, Q171* named against endometrial cancer

The four existing collections hold pathway- and reaction-level prose about disease, so the chatbot talks about disease fluently and has no document for the variant itself. It answers a level up and hedges — a fluent answer that doesn't contain the fact asked for.

Why this one could be done now

It's the only embeddings work that isn't blocked. The four existing collections are generated from Neo4j and we have no credentials; this is a flat TSV, so it follows the alliance path — MetaDataCSVLoader and build_embeddings, no database. 6,294 variants, 400 genes, cost in cents.

The two decisions, delegated back and decided

Which columns are embedded. The loader renders each field as name: value, so the column names are embedded too — and the release names are the query paths that produced them, up to 104 characters of entityWithAccessionedSequence_reactionLikeEvent_entityFunctionalStatus_…. Left alone they'd contribute more tokens than the values, identically in every document. Renamed to what a person would call them:

gene: ABCA1
variant: ABCA1 W590S [plasma membrane]
residue_change: L-tryptophan 590 replaced with L-serine
mutation_type: ReplacedResidue
disease: Tangier disease
functional_status: loss_of_function
normal_reaction: 4xPALM-C-p-2S-ABCA1 tetramer transports CHOL from transport vesicle...

Identifiers are metadata, not content — nobody types R-HSA-5682201 at a chatbot. The variant's own identifier is named st_id because that's the key csv_chroma de-duplicates on; a different name would silently disable de-duplication for this collection.

The pipe-delimited disease field. The spec's own recommendation — split it into a list for metadata — turned out to be impossible: Chroma accepts only str, int, float or bool and rejects a list outright. So I measured the alternative rather than guessing: fanning out to one document per variant-disease pair takes 6,294 documents to 10,500, and repeats p16INK4A R80* 45 times — enough near-identical documents to fill an entire result set. One document per variant, pipes rewritten as commas.

Verification

Two answer-sweep expectations match a pattern for a named variant (\bABCA1 [A-Z]\d{2,4}[A-Z*]) rather than a specific one, because which of the six come back depends on retrieval order and pinning one would fail a good answer — the mistake made in #223 and fixed there. Checked against the recorded answers: the pattern doesn't match the old one and does match the new.

They carry needs_collection="disease_variants" and skip loudly until the bundle ships, rather than turning the deploy gate red for a reason nobody can act on. A test covers the direction that matters — that they run once it's installed. The skip summary is now grouped by reason, since it previously said "they need the live service" for every skip.

313 tests, ruff and mypy clean.

Installing it

The embeddings tree is root-owned, so installing needs sudo — and using sudo is what keeps it root-owned. It is not the container's doing: the image has run as appuser (uid 3001) since 2025-04-17 and the bundle was created 2026-09-02. ~/fix-embeddings-ownership.sh sets awright:reactome with world-read, which suits both — the owner manages bundles without sudo, and the container, whose uid isn't a host user and which only reads at runtime, still can.

🤖 Generated with Claude Code