feat: calibrate abstention per index, compare with other stacks, and rebuild the landing drawer

AGENTS.md

@@ -59,6 +59,7 @@
 ## Validation
 
 - Run the smallest relevant check while editing and `pnpm validate` before a release pull request.
+- Keep `pnpm knip` clean: delete unused files, exports, and dependencies instead of ignoring them.
 - Reconcile `git status` with the intended scope before staging. Never stage secrets or generated local state.
 - Use Conventional Commits. Commit and push only when the user authorizes them.
 - Every commit promoted to `main` that can trigger semantic-release must include these exact body

CLAUDE.md

@@ -5,6 +5,7 @@
 ```bash
 pnpm bootstrap
 pnpm validate
+pnpm knip
 pnpm dev:landing
 pnpm example
 ```

CONTRIBUTING.md

@@ -50,9 +50,13 @@ Before opening a pull request, run:
 pnpm validate
 ```
 
-`pnpm validate` runs Biome, a dependency security audit, TypeScript checks, coverage-gated Vitest,
-production CLI and MCP smoke tests, the static landing build, public API checks, and npm package and
-release-artifact checks.
+`pnpm validate` runs Biome, Knip, a dependency security audit, TypeScript checks, coverage-gated
+Vitest, production CLI and MCP smoke tests, the static landing build, public API checks, and npm
+package and release-artifact checks.
+
+`pnpm knip` reports unused files, exports, dependencies, and duplicate exports across the workspace.
+Remove what it finds rather than ignoring it. Tag an export `@public` only when a benchmark or
+script imports it from the build, and `@alias` only for a deliberate public alias.
 
 To smoke-test the library API against your local build while developing Ragmir Core, run
 `pnpm example` (see

README.md

@@ -11,8 +11,8 @@
 </p>
 
 <p align="center">
-  93% of real developer questions find the right file in the top 10 · 40 ms median search on a
-  43,874-chunk monorepo · <a href="#benchmarks">benchmarks</a>
+  93% of real developer questions find the right file in the top 10 · 39 ms median search on a
+  29,778-chunk monorepo · <a href="#benchmarks">benchmarks</a>
 </p>
 
 <p align="center">
@@ -56,12 +56,18 @@ npx rgr search "how are permissions checked?" --compact
 - **Code and documents in one index.** Hybrid BM25 and vector search understands paths and
   camelCase identifiers and boosts the code that defines what you ask about, so "how are
   permissions checked" reaches `PermissionService.java`, not only its tests and notes.
-- **Fast on real repositories.** Median search takes 40 ms on a 43,874-chunk Java and TypeScript
+- **Fast on real repositories.** Median search takes 39 ms on a 29,778-chunk Java and TypeScript
   monorepo with Office files, and the right file reaches the top 10 for 93% of 101 real developer
-  questions (87% with 6.1.6).
+  questions.
+- **Ahead of the usual RAG stacks.** On this repository, the right file reaches the top 3 for 76%
+  of developer questions, against 56% for a LangChain.js pipeline and 70% for MiniSearch BM25 with
+  the same embedding model, and Ragmir refuses 27 of 30 off-topic questions where they refuse at
+  most 3.
 - **Evidence you can verify.** Citations point to source lines, PDF pages, slides, sheet cells, or
   EPUB sections. Results say when their file changed after indexing (`sourceStatus`) and when
-  their support is thin (`weakEvidence`); unrelated questions return nothing instead of noise.
+  their support is thin (`weakEvidence`). Questions your sources cannot answer return nothing
+  instead of noise: 58 of 90 off-topic and 24 of 90 near-domain questions on real bases, up from
+  41 and 1.
 - **Always current.** `rgr doctor` compares the files on disk with the index on every run, and
   `rgr hooks install` refreshes it in the background after commits, merges, and checkouts.
 - **Private by default.** No account, telemetry, or hosted storage. The default `local-hash`
@@ -180,25 +186,50 @@ use the same retrieval API without one.
 
 ## Benchmarks
 
-Ragmir 6.2 against 6.1.6 on the same machine, configuration, and index, rebuilt by 6.2. Each
-answerable question has one expected file; "top 3" and "top 10" count the questions whose file
-appears there. Unrelated questions (nonsense, weather, recipes, sports) should return nothing.
-Latency is the median of warm sequential searches in one process.
-
-| Corpus | Embeddings | Answer in top 3 | Answer in top 10 | nDCG@10 | Unrelated questions refused | Median search |
-| --- | --- | ---: | ---: | ---: | ---: | ---: |
-| Java/TypeScript monorepo with Office files and scans: 2,117 files, 43,874 chunks, 101 developer questions | multilingual-e5-small | 70% → **73%** | 87% → **93%** | 0.639 → **0.670** | 0 → **3** of 30 | 31 → 40 ms |
-| French and English product documentation: 298 files, 20,931 chunks, 60 questions | mxbai-embed-xsmall | 100% → **100%** | 100% → **100%** | 0.932 → **0.938** | 3 → **15** of 30 | 25 → 31 ms |
-
-Measured on October 27, 2026 on an Apple M3 Max with Node.js 22.23 and LanceDB 0.30, with the
-`quality` profile these projects use and 30 unrelated questions per corpus. The two private corpora
-are described rather than published. Of the passages 6.2 still returned for unrelated questions,
-it flagged 41 of 81 on the monorepo and 37 of 38 on the documentation as weak evidence, against
-7 of 94 and 0 of 60 correct answers. On the monorepo it also applied six exclusion rules that
-6.1.6 silently missed on accented file names and cited 27 decomposed paths in canonical form. The
-extra milliseconds pay for word statistics, code-definition boosts, and crowding checks. Run the
-[benchmarks](./packages/ragmir-core/benchmarks/README.md) and `rgr evaluate` on your own corpus
-before relying on any number.
+### Real projects
+
+Ragmir 6.4 against 6.3.1 on the same machine, configuration, and indexes. Each answerable question
+lists the files that answer it; "top 3" and "top 10" count the questions with one of them there.
+Off-topic questions (sports, cooking, weather, nonsense) and near-domain questions about features
+the project does not have should return nothing. Latency is the median of warm sequential searches
+in one process.
+
+| Corpus | Embeddings | Answer in top 3 | Answer in top 10 | Off-topic refused | Near-domain refused | Correct answers flagged weak | Median search |
+| --- | --- | ---: | ---: | ---: | ---: | ---: | ---: |
+| Java/TypeScript monorepo with Office files and scans: 2,297 files, 29,778 chunks, 101 developer questions | multilingual-e5-base | 74% → **74%** | 93% → **93%** | 0 → **16** of 30 | 1 → **11** of 30 | 11 → **8** of 94 | 39 → 39 ms |
+| French and English product documentation: 299 files, 21,011 chunks, 60 questions | mxbai-embed-xsmall | 100% → **100%** | 100% → **100%** | 16 → **17** of 30 | 0 → **6** of 30 | 0 → **0** of 60 | 28 → 28 ms |
+| This repository: 229 files, 2,490 chunks, 99 developer questions | mxbai-embed-xsmall | 75% → **75%** | 97% → **97%** | 25 → **25** of 30 | 0 → **7** of 30 | 32 → **6** of 96 | 22 → 22 ms |
+
+Measured on November 1, 2026 on an Apple M3 Max with Node.js 26 and LanceDB 0.30, with the
+`quality` profile these projects use. Near-domain questions pair the project's own vocabulary with
+an integration, service, or format it does not have, and each was checked absent from every
+indexed file. 6.4 kept every answer in place and caught them by measuring each index's noise
+floor: a passage only counts as a strong vector match when it is nearer than 24 off-topic probe
+questions, and a search whose first five results are all weak answers nothing. The two private
+corpora are described rather than published. Run the [benchmarks](./packages/ragmir-core/benchmarks/README.md)
+and `rgr evaluate` on your own corpus before relying on any number.
+
+### Against other retrieval stacks
+
+`pnpm bench:competitors` indexes the 210 tracked documentation and source files of this
+repository and asks its 99 developer questions plus 30 off-topic and 30 near-domain questions. Every
+vector side uses the same model and vectors (`mxbai-embed-xsmall-v1`); the other stacks read the same
+files split by LangChain's `RecursiveCharacterTextSplitter` (1,000 characters, 200 overlap).
+
+| System | Answer in top 3 | Answer in top 10 | MRR@10 | Off-topic refused | Near-domain refused | Median search |
+| --- | ---: | ---: | ---: | ---: | ---: | ---: |
+| **Ragmir**, hybrid with `mxbai-embed-xsmall-v1` | **76%** | **97%** | **0.647** | **27** of 30 | **5** of 30 | 22 ms |
+| **Ragmir**, `local-hash` without a model | **79%** | 96% | 0.611 | 9 of 30 | 0 of 30 | 20 ms |
+| LangChain.js splitter with exact cosine search, as `MemoryVectorStore` | 56% | 89% | 0.509 | 0 of 30 | 0 of 30 | 5 ms |
+| Orama 3 hybrid, default similarity 0.8 | 18% | 35% | 0.144 | 2 of 30 | 0 of 30 | 5 ms |
+| Orama 3 hybrid, similarity 0.1 | 38% | 64% | 0.317 | 0 of 30 | 0 of 30 | 6 ms |
+| MiniSearch 7 BM25 | 70% | 92% | 0.566 | 3 of 30 | 0 of 30 | 1 ms |
+
+In-memory libraries answer faster, but they return results for almost every off-topic question,
+and none ranks the answering file as high: it reaches the top 3 for 76 to 79% of questions with Ragmir,
+against 56% with the LangChain.js pipeline and 70% with MiniSearch. Measured on the same machine on
+November 1, 2026; [the benchmark guide](./packages/ragmir-core/benchmarks/README.md) explains every
+setting.
 
 ## TypeScript
 

docs/agent-integration.md

@@ -134,7 +134,8 @@ questions appears once. Within one MCP session, passages already returned come b
 Results flag two conditions. `sourceStatus: "modified"` or `"missing"` means the cited file changed
 after indexing; with `mcpAutoIngest: true`, the server then starts one background incremental
 ingest (at most once a minute). `weakEvidence: true` means the passage holds few of the question's
-distinctive words and has no strong semantic match.
+distinctive words and has no strong semantic match. An empty answer means the base holds no
+evidence for the question: say so instead of answering from the nearest passages.
 
 The tools accept common argument spellings (`top_k`, `radius`, `max_bytes`, `paths`, `q`) and
 clamp out-of-range numbers instead of failing; the response notes each adjustment and lists ignored

docs/api-reference.md

@@ -71,7 +71,8 @@ It identifies indexed evidence, not the current file on disk. Each result whose
 disappeared since indexing also carries `sourceStatus: "modified"` or `"missing"`, from one `stat`
 per cited file; `doctor` checks the whole corpus by size and modification time, and `audit` by
 checksum. `weakEvidence: true` marks a passage with few of the question's distinctive words and
-no strong vector match. Compact results retain these fields.
+no strong vector match; a Transformers search whose first five results are all weak returns an
+empty list instead. Compact results retain these fields.
 
 For XLSX tables, full results attach the header as a separate `context` passage with its own cell
 citation, including at `contextRadius: 0`. Expansion also includes it; compact search requires
@@ -119,6 +120,8 @@ it excludes paths, source text, and queries. No telemetry is transmitted.
 Operations accepting `OperationOptions` support `signal` and `timeoutMs`. The library returns
 values or throws errors; it does not write to stdout/stderr. Use `isRagmirError(error)` to inspect
 structured codes such as `TIMEOUT`, `OVERLOADED`, `INDEX_BUSY`, `INDEX_UNAVAILABLE`, and `CLIENT_CLOSED`.
+A config that still holds retired fields throws `CONFIG_MIGRATION_REQUIRED`: run `rgr upgrade` to
+back it up, migrate it, and rebuild the index.
 
 Per-project process-local queues bound search, embeddings, and ingestion. A private writer lock
 serializes index writers across local OS processes. Readers keep generation leases through their

docs/cli-reference.md

@@ -46,8 +46,9 @@ rgr search "migration" --exact-vector-search
 
 Text output marks a result whose file changed or disappeared after indexing
 (`source modified since indexing`) and a result with thin support (`weak evidence`). JSON results
-carry the same information as `sourceStatus` and `weakEvidence`. `--scope` applies a named scope
-from the config; `--include-path` accepts backslashes on every platform.
+carry the same information as `sourceStatus` and `weakEvidence`. A semantic search whose first five
+results are all weak prints no result: the base holds no evidence for the question. `--scope`
+applies a named scope from the config; `--include-path` accepts backslashes on every platform.
 
 `sources add` accepts paths, globs, and `!` exclusions. Search accepts `--top-k`,
 `--include-path`, `--exclude-path`, and repeatable `--context-path`. Search accepts
@@ -202,7 +203,9 @@ doctor and `rgr audit` label their O(corpus) cost in text and JSON output. A mis
 manifest always yields `ready=false`, including legacy tables that predate manifest activation.
 `readiness.lexicalIndexCurrent` is `false` for an index built under an older keyword policy, such as
 one without accent-folded terms or path and identifier terms. Search keeps using that index; the
-next `rgr ingest` or `rgr upgrade` stages and validates a rebuild before activating it.
+next `rgr ingest` or `rgr upgrade` stages and validates a rebuild before activating it. Its next
+steps also ask for `rgr storage optimize` when the index is split into many small fragments, and for
+`rgr ingest` when a semantic index has no stored noise floor yet.
 
 `rgr ingest --rebuild` writes batches into an isolated LanceDB generation. The existing index stays
 active until the new table and manifest pass row-count, checksum, and duplicate-ID validation. The

docs/configuration.md

@@ -149,11 +149,20 @@ distinctive word counts more than a common one:
   `mxbai-embed-xsmall-v1` uses 1.1. Other models keep the previous rule: any related query word,
   or a distance up to 1.1.
 
+Every Transformers index also measures its own noise floor: `rgr ingest` embeds 24 fixed off-topic
+probe questions (sports, cooking, geography, nonsense, in English and French) and stores the
+distance of the nearest passage any of them reaches. A model packs texts into its own band and a
+corpus shifts it, so only a passage nearer than every off-topic probe counts as a strong vector
+match, and a question with no known word needs one. An index built before this release measures
+the floor once per process on its first search; the next `rgr ingest` stores it.
+
 Each result whose passage holds few of the question's distinctive words and has no strong vector
-match carries `weakEvidence: true`. It is still returned; agents should verify it first. On a
-French and English golden set over a Java and TypeScript monorepo, these rules kept every
-answerable question's source, and with `mxbai-embed-xsmall-v1` they rejected 16 of 30 unrelated
-questions instead of 3. Evaluate your own corpus with `rgr evaluate`.
+match carries `weakEvidence: true`. When the first five results of a Transformers search are all
+weak, the answer holds no real evidence and search returns nothing. `local-hash` search keeps such
+results. On a Java and TypeScript monorepo, a product documentation base, and this repository
+(260 answerable questions), these rules kept every answer in place while refusing 58 of 90
+off-topic and 24 of 90 near-domain unanswerable questions, against 41 and 1 before, and flagged
+14 correct answers as weak instead of 43. Evaluate your own corpus with `rgr evaluate`.
 
 Changing an embedding provider, model, revision, digest, or chunking field requires
 `rgr ingest --rebuild`. Revision and artifact digest participate in the index, vector-index, and

docs/migration.md

@@ -3,6 +3,22 @@
 This is a breaking surface change. Ragmir now concentrates on local indexing, search, exact
 citations, OCR, and integration with developer agents.
 
+## Index noise floor and weak answers (6.4)
+
+Existing indexes stay searchable after the update; nothing needs deleting.
+
+- **Ranking policy version 8.** Transformers search caps strong vector evidence with each index's
+  noise floor and answers nothing when its first five results are all `weakEvidence`.
+  `mxbai-embed-xsmall-v1` flags a passage as weak below 30% of the question's IDF-weighted words
+  instead of 50%. Rerun `rgr evaluate`: stored quality reports no longer verify the active policy.
+- **Noise floor.** `rgr ingest` stores the floor in the index manifest. An older index measures it
+  once per process on its first search until the next `rgr ingest`.
+- **Typed migration error.** A config with retired fields now throws `CONFIG_MIGRATION_REQUIRED`
+  instead of an internal error. Run `rgr upgrade`.
+- **Fragmented indexes.** `rgr doctor` asks for `rgr storage optimize` when an index is split into
+  many small fragments, which slowed searches up to five times on indexes built before automatic
+  compaction.
+
 ## Canonical paths, live freshness, and calibrated evidence (6.2)
 
 Existing indexes stay searchable after the update; nothing needs deleting.

docs/troubleshooting.md

@@ -94,7 +94,9 @@ need explicitly configured local extractors.
 ## Search is weak
 
 Results marked `weakEvidence` hold few of the question's distinctive words and have no strong
-semantic match: rephrase with the project's own terms or narrow the search with a `scope`. When one
+semantic match: rephrase with the project's own terms or narrow the search with a `scope`. A
+semantic search returns nothing when its first five results are all weak; a rephrased question
+with the project's vocabulary usually finds the evidence if it exists. When one
 export or log dominates results, `rgr audit` reports index concentration; exclude it, lower it
 with `sourceWeights`, or cap it with `maxChunksPerDataFile`.
 

knip.json

@@ -0,0 +1,21 @@
+{
+  "$schema": "https://unpkg.com/knip@6/schema.json",
+  "ignoreExportsUsedInFile": {
+    "interface": true,
+    "type": true
+  },
+  "ignoreBinaries": ["mise", "textutil"],
+  "workspaces": {
+    ".": {
+      "entry": ["scripts/*.mjs", "tests/public-api-consumer/consumer.ts"],
+      "project": ["scripts/**/*.mjs", "tests/**/*.ts"]
+    },
+    "packages/ragmir-core": {
+      "entry": ["benchmarks/*.mjs", "scripts/*.mjs", "examples/**/*.{mjs,ts}"],
+      "project": ["src/**/*.ts", "benchmarks/**/*.mjs", "scripts/**/*.mjs"]
+    },
+    "packages/ragmir-landing": {
+      "ignoreDependencies": ["@astrojs/check"]
+    }
+  }
+}

llms.txt

@@ -31,8 +31,9 @@ the query. It owns reasoning, generation, and actions, with the model chosen by
   session as short stubs. Instructions name the base; unrelated workspaces get a scope notice, or
   a refusal with `mcpAllowedRoots`. Register Ragmir per project, not for every project.
 - Results carry `sourceStatus` when the cited file changed after indexing and `weakEvidence` when
-  support is thin. `rgr doctor` checks freshness on every run; `rgr hooks install` refreshes the
-  index after Git operations.
+  support is thin. An empty semantic answer means the base holds no evidence for the question.
+  `rgr doctor` checks freshness on every run; `rgr hooks install` refreshes the index after Git
+  operations.
 - Agents search, expand selected evidence, and refine queries. Ragmir does not run the agent loop.
 - Present four workflow examples: build a feature, diagnose an incident, and plan a migration with
   Claude Code or Codex using cited project evidence; then make a confidential architecture decision