Skip to content

Give each Chroma store its own Settings; a shared one crossed the bundles - #220

Merged
adamjohnwright merged 1 commit into
mainfrom
fix/chroma-settings-shared
Sep 15, 2026
Merged

adamjohnwright merged 1 commit into
mainfrom
fix/chroma-settings-shared

Conversation

@adamjohnwright

Copy link
Copy Markdown
Contributor

Reactome questions are broken on beta right now. This fixes a live outage.

Asked "What does CDK5 phosphorylate in Alzheimer disease?", the deployed container raises:

PermissionError: [Errno 13] Permission denied:
'/app/embeddings/.../userguide/Release95/sections/9c574827-...'

A reactome question failing inside the user guide bundle — Chroma trying to create a segment folder for a collection that isn't there, in a bundle it should never have opened.

Cause

chroma_settings was one module-level Settings object shared by every store. langchain_chroma mutates what it is handed:

client_settings.persist_directory = (
    persist_directory or client_settings.persist_directory
)

So the last store constructed rewrote the persist directory for every earlier one, and chromadb returned a client keyed on those settings.

With only the reactome bundle installed this was invisible — every store pointed into the same tree, so crossing them changed nothing. Installing the user guide bundle yesterday gave them different trees and turned a latent bug into an outage.

chroma_settings() is now a factory returning a fresh object per store.

Verified in the order that broke beta

User guide first, then reactome, then user guide again:

userguide hits: 6
reactome hits : 40
and again, user guide after reactome: 6

Three tests, all of which fail against the shared constant — including one asserting telemetry stays off, so that isn't lost while fixing the sharing.

I saw this yesterday and dismissed it

The same PermissionError appeared while I was reviewing #216. I wrote it off as "a local artifact of the root-owned bundle" after a check that built the two retrievers and queried each once. That check didn't reproduce the ordering, so it proved nothing — and I reported the system healthy. The bug was in production for a day.

The check that found it was running the three real question types through the deployed container, which is the one I should have run first.

🤖 Generated with Claude Code

…dles

Reactome questions are broken on beta right now. Asked "What does CDK5
phosphorylate in Alzheimer disease?", the deployed container raises:

    PermissionError: [Errno 13] Permission denied:
    '/app/embeddings/.../userguide/Release95/sections/9c574827-...'

A reactome question, failing inside the user guide bundle -- Chroma trying to
create a segment folder for a collection that is not there, in a bundle it
should never have opened.

`chroma_settings` was one module-level Settings object shared by every store,
and `langchain_chroma` mutates what it is handed:

    client_settings.persist_directory = (
        persist_directory or client_settings.persist_directory
    )

So the last store constructed rewrote the persist directory for every earlier
one, and chromadb returned a client keyed on those settings. With only the
reactome bundle installed this was invisible: every store pointed into the same
tree, so crossing them changed nothing. Installing the user guide bundle
yesterday gave them different trees and turned a latent bug into an outage.

`chroma_settings()` is now a factory returning a fresh object per store.

Three tests, all of which fail against the shared constant. Verified in the
order that broke beta -- user guide first, then reactome, then user guide
again: 6 hits, 40 hits, 6 hits.

**I saw this yesterday and dismissed it.** The same PermissionError appeared
while I was reviewing #216, and I wrote it off as "a local artifact of the
root-owned bundle" after a check that built the two retrievers and queried each
once. That check did not reproduce the ordering, so it proved nothing, and I
reported the system healthy. The bug was in production for a day.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@adamjohnwright
adamjohnwright merged commit 7bc4804 into main Sep 15, 2026
10 checks passed
@adamjohnwright
adamjohnwright deleted the fix/chroma-settings-shared branch September 15, 2026 17:31
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant