feat: local Ollama + Claude-headless LLM/embedding backends (perception-rust) - #15
Open
lollinng wants to merge 1 commit into
Open
feat: local Ollama + Claude-headless LLM/embedding backends (perception-rust)#15lollinng wants to merge 1 commit into
lollinng wants to merge 1 commit into
Conversation
lollinng
force-pushed
the
feat/local-ollama-embeddings
branch
from
June 19, 2026 09:43
bc9d6c5 to
b7a9908
Compare
Lets patcha run without the patcha-cloud login. Opt-in, config-selectable backends; defaults (fastembed embeddings, patcha-cloud LLM) unchanged. Embeddings (EMBEDDING_PROVIDER): - "ollama": Embedder gains an enum backend; the Ollama variant calls /api/embed over a blocking ureq client (the embedder runs inside block_in_place/ spawn_blocking and is dropped inside async, so a runtime-free client avoids drop panics). Fails fast if Ollama is unreachable or the model isn't 768-dim. LLM (LLM_PROVIDER), routed inside PatchaApiClient::chat_completion (no caller churn): - "ollama": OpenAI-compatible /v1/chat/completions, no auth. - "claude": Claude Code headless — shells out to `claude -p --system-prompt … --model <opus> --output-format text` via tokio::process, using the local Claude login (no API key). --system-prompt overrides Claude Code's default agent prompt with patcha's. Config (env-driven): EMBEDDING_PROVIDER, LLM_PROVIDER, OLLAMA_URL, OLLAMA_EMBEDDING_MODEL, OLLAMA_LLM_MODEL, CLAUDE_BIN, CLAUDE_MODEL. Cargo: add ureq for the blocking embed call. Verified end-to-end with no cloud login: nomic-embed-text (768) embeds -> sqlite-vec store -> semantic search returns correct ranked results; summarize generates via both Ollama (llama3.2) and Claude headless (Opus). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Tq11wGi71nR57bXi4bwcCz
lollinng
force-pushed
the
feat/local-ollama-embeddings
branch
from
June 19, 2026 11:06
b7a9908 to
23beb61
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Lets patcha run without the patcha-cloud login by making the embedding and LLM backends pluggable. All opt-in via config; defaults (fastembed embeddings, patcha-cloud LLM) are unchanged.
Backends
Embeddings —
EMBEDDING_PROVIDERfastembed(default) — unchanged, on-device.ollama—Embeddergains an enum backend; the Ollama variant calls/api/embedvia a blockingureqclient. Rationale:embed_one/embed_manyare sync, run insideblock_in_place/spawn_blocking, and are dropped inside async in the CLI paths — a client that owns no tokio runtime avoids both the block-in-runtime and drop-panic failure modesreqwest::blockingwould hit. Fails fast if Ollama is unreachable or the model isn't 768-dim (the store's fixedsqlite-vecwidth).LLM —
LLM_PROVIDER(routed insidePatchaApiClient::chat_completion; the 8 call sites are untouched)patcha(default) — unchanged cloud path.ollama— OpenAI-compatible/v1/chat/completions, no auth.claude— Claude Code headless: shells out toclaude -p --system-prompt … --model <opus> --output-format textviatokio::process, using the local Claude login (no API key).--system-promptoverrides Claude Code's default agent prompt with patcha's.Config (env-driven, all optional)
EMBEDDING_PROVIDER,LLM_PROVIDER,OLLAMA_URL,OLLAMA_EMBEDDING_MODEL,OLLAMA_LLM_MODEL,CLAUDE_BIN,CLAUDE_MODEL.Cargo: addsureq(rustls, no extra runtime).Example — fully local embeddings + Opus reasoning, no keys:
Verification (real binary, no cloud login)
cargo buildclean;cargo testpasses (incl. sqlite-vec roundtrip).collect→ 4 events embedded → stored;searchreturns the correct top match for each query (redis/jwt/ffmpeg/kubectl).llama3.2) and via Claude headless (claude -p, Opus):summarizegenerates a coherent daily summary through both. The Opus path produced an honest summary (declined to invent missing detail) where the small local model hallucinated.Notes / scope
db/migrations.rs, so Ollama embedding models must be 768-dim (nomic-embed-textfits; clear error otherwise).claudebackend spawns oneclaude -psubprocess per LLM call — fine for nightly summarize/compaction; for per-event categorization you'd likely keepLLM_PROVIDER=ollama.find_git_reposskips every dotfile before its.gitcheck, so the git collector never discovers repos.