Skip to content

Add DuckDB embedding backend - #61

Merged
gkennos merged 13 commits into
mainfrom
34-duckdb-support
Nov 5, 2026
Merged

Add DuckDB embedding backend#61
gkennos merged 13 commits into
mainfrom
34-duckdb-support

Conversation

@nicoloesch

@nicoloesch nicoloesch commented Oct 16, 2026 •
edited
Loading

Copy link
Copy Markdown
Collaborator

Summary

Changes

  • Adds a DuckDB embedding backend (duckdbvss) with HNSW indexes through the DuckDB VSS extension.
  • Supports filtered and unfiltered nearest-neighbour search, and upserts onto indexed rows.
  • INSTALL vss needs network access once.
  • Reader connections are read-only at the database level.
  • Opening a reader on a missing file uses an empty in-memory database instead of creating the file.
  • The model registry and maintenance CLI work with the new backend.
  • The embedding backend config accepts DuckDB.

Checklist

  • Applied exactly one label (breaking, feature, fix, dependencies, or chore)
  • Tests pass locally (uv run pytest -q)
  • Lint passes (uv run ruff check .)

@nicoloesch nicoloesch added the feature New backwards-compatible functionality. MINOR: x.y+1.z label Oct 16, 2026
This was referenced Oct 16, 2026
Merged
Merged
@gkennos
gkennos self-requested a review November 3, 2026 08:03
nicoloesch and others added 6 commits November 5, 2026 02:14
@nicoloesch
Sync DuckDB VSS's HNSW index after every embedding write
d99fbeb 
Reloading a persisted HNSW index desyncs it from the table on the next
write, undercounting later searches. upsert/delete now resync it after
each write; bulk upserts defer until the end.
@gkennos
dependency bump
a1c4ac0
@gkennos
dependabot
8126313
@gkennos
api docs addition + stop re-creating index batchwise
32d789e
@nicoloesch
Remove duplicated pbar
e921be7
@nicoloesch
Thread CLI disabling through
7866902
@gkennos
gkennos merged commit 5642667 into main Nov 5, 2026
4 checks passed
@gkennos
gkennos deleted the 34-duckdb-support branch November 5, 2026 21:25
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@gkennos gkennos gkennos approved these changes

Assignees

No one assigned

Labels

feature New backwards-compatible functionality. MINOR: x.y+1.z

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

Support DuckDB backend

2 participants

@nicoloesch @gkennos