Skip to content

feat: local Ollama + Claude-headless LLM/embedding backends (perception-rust) - #15

Open
lollinng wants to merge 1 commit into
feat/perception-rustfrom
feat/local-ollama-embeddings
Open

feat: local Ollama + Claude-headless LLM/embedding backends (perception-rust)#15
lollinng wants to merge 1 commit into
feat/perception-rustfrom
feat/local-ollama-embeddings

Conversation

@lollinng

@lollinng lollinng commented Jun 19, 2026

Copy link
Copy Markdown
Collaborator

Rebased onto feat/perception-rust (the Python tree is deleted on this branch; this re-implements local-backend support in Rust).

What

Lets patcha run without the patcha-cloud login by making the embedding and LLM backends pluggable. All opt-in via config; defaults (fastembed embeddings, patcha-cloud LLM) are unchanged.

Backends

Embeddings — EMBEDDING_PROVIDER

  • fastembed (default) — unchanged, on-device.
  • ollamaEmbedder gains an enum backend; the Ollama variant calls /api/embed via a blocking ureq client. Rationale: embed_one/embed_many are sync, run inside block_in_place/spawn_blocking, and are dropped inside async in the CLI paths — a client that owns no tokio runtime avoids both the block-in-runtime and drop-panic failure modes reqwest::blocking would hit. Fails fast if Ollama is unreachable or the model isn't 768-dim (the store's fixed sqlite-vec width).

LLM — LLM_PROVIDER (routed inside PatchaApiClient::chat_completion; the 8 call sites are untouched)

  • patcha (default) — unchanged cloud path.
  • ollama — OpenAI-compatible /v1/chat/completions, no auth.
  • claudeClaude Code headless: shells out to claude -p --system-prompt … --model <opus> --output-format text via tokio::process, using the local Claude login (no API key). --system-prompt overrides Claude Code's default agent prompt with patcha's.

Config (env-driven, all optional)

EMBEDDING_PROVIDER, LLM_PROVIDER, OLLAMA_URL, OLLAMA_EMBEDDING_MODEL, OLLAMA_LLM_MODEL, CLAUDE_BIN, CLAUDE_MODEL. Cargo: adds ureq (rustls, no extra runtime).

Example — fully local embeddings + Opus reasoning, no keys:

# ~/.patcha/.env
EMBEDDING_PROVIDER=ollama
OLLAMA_EMBEDDING_MODEL=nomic-embed-text   # 768-dim, matches the store
LLM_PROVIDER=claude
CLAUDE_MODEL=opus

Verification (real binary, no cloud login)

  • cargo build clean; cargo test passes (incl. sqlite-vec roundtrip).
  • Ollama embeddings: collect → 4 events embedded → stored; search returns the correct top match for each query (redis/jwt/ffmpeg/kubectl).
  • LLM via Ollama (llama3.2) and via Claude headless (claude -p, Opus): summarize generates a coherent daily summary through both. The Opus path produced an honest summary (declined to invent missing detail) where the small local model hallucinated.

Notes / scope

  • Dimension is fixed at 768 in db/migrations.rs, so Ollama embedding models must be 768-dim (nomic-embed-text fits; clear error otherwise).
  • The claude backend spawns one claude -p subprocess per LLM call — fine for nightly summarize/compaction; for per-event categorization you'd likely keep LLM_PROVIDER=ollama.
  • Unrelated pre-existing bug (left untouched): find_git_repos skips every dotfile before its .git check, so the git collector never discovers repos.
  • Implemented with AI assistance (Claude Code).

@lollinng
lollinng force-pushed the feat/local-ollama-embeddings branch from bc9d6c5 to b7a9908 Compare June 19, 2026 09:43
@lollinng lollinng changed the title feat: add local Ollama backend for embeddings and LLM features feat: add local Ollama backend for embeddings and LLM (perception-rust) Jun 19, 2026
@lollinng
lollinng changed the base branch from main to feat/perception-rust June 19, 2026 09:43
Lets patcha run without the patcha-cloud login. Opt-in, config-selectable
backends; defaults (fastembed embeddings, patcha-cloud LLM) unchanged.

Embeddings (EMBEDDING_PROVIDER):
- "ollama": Embedder gains an enum backend; the Ollama variant calls /api/embed
  over a blocking ureq client (the embedder runs inside block_in_place/
  spawn_blocking and is dropped inside async, so a runtime-free client avoids
  drop panics). Fails fast if Ollama is unreachable or the model isn't 768-dim.

LLM (LLM_PROVIDER), routed inside PatchaApiClient::chat_completion (no caller churn):
- "ollama": OpenAI-compatible /v1/chat/completions, no auth.
- "claude": Claude Code headless — shells out to `claude -p --system-prompt …
  --model <opus> --output-format text` via tokio::process, using the local
  Claude login (no API key). --system-prompt overrides Claude Code's default
  agent prompt with patcha's.

Config (env-driven): EMBEDDING_PROVIDER, LLM_PROVIDER, OLLAMA_URL,
OLLAMA_EMBEDDING_MODEL, OLLAMA_LLM_MODEL, CLAUDE_BIN, CLAUDE_MODEL.
Cargo: add ureq for the blocking embed call.

Verified end-to-end with no cloud login: nomic-embed-text (768) embeds ->
sqlite-vec store -> semantic search returns correct ranked results; summarize
generates via both Ollama (llama3.2) and Claude headless (Opus).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Tq11wGi71nR57bXi4bwcCz
@lollinng
lollinng force-pushed the feat/local-ollama-embeddings branch from b7a9908 to 23beb61 Compare June 19, 2026 11:06
@lollinng lollinng changed the title feat: add local Ollama backend for embeddings and LLM (perception-rust) feat: local Ollama + Claude-headless LLM/embedding backends (perception-rust) Jun 19, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant