fix(search): anchor rare-identifier symbols into pre-rerank pool (P2)

[ManSio]

Mechanism

P2 query (\hybrid_search_async reciprocal_rank_fusion FTS5 BM25) never had its gold chunk (\src/core/search/engine.py) in the 5-chunk pre-rerank pool, so no pool-stage boost (O1 included) could rescue it. New: _anchor_identifier_chunks_async\ resolves exact-symbol chunks for rare identifier tokens DIRECTLY (single-token FTS fetch, ~70ms warm) and appends them pre-rerank, before O1. Definition chunks outrank caller chunks sharing the symbol; docs/data never anchor; caps 2/token + 3 total + MAX_RERANKER_INPUT; any failure degrades to the unanchored pool.

Root cause (numbers, fresh-process + discarded warm-up + void-checked)

• Pool cut is
  rf_results[:limit]\ with
  aw_limit=min(limit*2,30)\ (engine.py).
• Gold depth: BM25#126, FTS#74, dense absent @200. Index is polluted by own experiment outputs quoting the query verbatim. Old standalone-BM25-rank-0 was measured on a cleaner index — stale.
• Pool-depth sweep (limit 5/10/20/50): gold absent at ALL depths.
• O1 cannot rescue: its candidate is
  eciprocal_rank_fusion\ (df=4, strictly rarest) while gold's symbol is \hybrid_search_async\ (df=100) — no exact match; live O1 fired on \scoring.py\ instead.
• Per-tier top-1 anchoring (adopt/reranker-threshold-and-pool idea) refuted live for this index: tier tops are polluted junk; gold sits at #74-126.

Design choice: (b) tier-winner anchoring for identifier tokens

• (a) widen pool — REFUTED by measurement (see latency below).
• (b) identifier-token anchoring — CHOSEN. Cost: +1-2 FTS fetches ~70ms each (warm), +<=3 pool slots.
• (c) hybrid (b + O1) — what this is: anchors run first, O1 boosts after, unchanged.

Latency cost (reranker_ms vs pool; llama.cpp BGE-M3 live)

limit	5	10	20	50
rerank	1.7s	4.0s	7.7s	22.9s

~0.4s/chunk => reaching gold at depth 126 would cost ~50s. Anchors cost ~0.14s. P2 wall with fix: ~2.6-3.0s (unchanged profile).

Holdout (\scripts/p2_holdout_gate.py, fresh process, GATE PASS)

id	rank	wall		id	rank	wall
P2	1	2.9s		H7	None	2.8s
H1	None	3.2s		H8	1	3.9s
H2	None (no-boost OK)	2.4s		H9	None	9.3s
H3	None	1.8s		H10	None	1.6s
H4	2	2.3s		H11	2	2.4s
H5	1	2.9s		H12	None	1.6s
H6	2	3.5s		N1	None (no-boost OK)	2.1s
				DOC	None (no-doc-boost OK)	3.3s

Contract: P2==1, H2/N1 unboosted, DOC never doc-boosted, no void, no degraded rows. H-ranks without expectations are informational (gate asserts no unexpected boost, not ranks).

Red-team (doc-guard + H2 lessons applied)

• A1 H2-collision: \BM25\ (df=327), \FTS5\ (df=140) excluded by df<=120 cap; unit test pins it.
• A2 doc/data pollution: docs/*.md citing symbols + \canary_set.json\ bogus exact symbol both ineligible (extension blocklist + _is_doc_chunk); unit tests pin both.
• A3 caller-vs-def: \live_search_audit.py:4\ and \engine.py:18\ share symbol \hybrid_search_async\ — def-pattern orders engine.py first; unit test pins order.
• A4 cost/degrade: caps + pool ceiling + FTS-failure passthrough; unit tests pin limit=0, cap, failure paths.

Caveats

• df cap 120 is heuristic and index-dependent (100 vs 140 is a thin boundary).
• P2 rank 1 needs a live reranker; without it gold is in-pool but off-top.
• Found along the way (NOT fixed here, pre-existing): \�syncio.run()\ per query silently degrades even-positioned queries to reranker passthrough (ms=0) — the gate runs one loop + degraded flag. \o1_holdout_gate.py\ has the same harness flaw, so its table understates O1; left untouched (merged file).
• Tests: 13 new unit + 13 O1 + 38 reranker = 64 passed; ruff check clean. Diff: +171/-0 engine.py (a ruff-format +447 churn was reverted to keep the diff minimal).

DO NOT MERGE — awaiting review.

[coderabbitai]

Important

• 🔍 Trigger review

This repository does not receive automatic reviews because it has fewer than 10 stars.

⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Advanced

Run ID: 7de3e642-8478-47f8-968b-6b2244777ec4

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}