Adopt O1: rare-identifier x100 boost at pre-rerank pool

[ManSio]

Mechanism (spec O1, final — no redesign)

Multi-token queries containing a rare identifier-shaped token (snake_case / camelCase / PascalCase / CONSTANT / --flag) now boost exact symbol matches at the PRE-RERANK pool stage (after MMR, before _apply_multi_reranker_async), reusing the proven x100 factor convention.

• D1 rarity gate: candidate only if STRICTLY rarest query term (df from existing BM25 stats, no new index structures). H2-style collision (common term like BM25) never boosts.
• D2 exact gate: full-token equality token == symbol_name, never substring; _is_doc_chunk exclusion inherited (docs never boosted).
• D3 pre-rerank: boost runs on the pool before the reranker, never post-cut (post-cut _boost_exact_name_matches untouched).
• Reranker-cache trap documented in KNOWN_ISSUES (top entry, Open): retriever measurements fresh-process or explicit cache reset; wall<2s without rerank timing = void.
• Note: main HEAD has no anchor_tier_winners (that is unmerged PR52); O1 hooks the MMR->reranker seam and composes after anchors when PR52 lands.

Numbers

Unit: tests/test_o1_identifier_boost.py 13/13 green (extraction, rarity incl. H2 collision, tie, single-term; exact gate; doc-guard negative control; pre-rerank pool assertion; Searcher method e2e + degraded-BM25).
Related search suites: 54 passed.
Live (fresh-process gate scripts/o1_holdout_gate.py, services 8080/8081, warm-up discarded): P2 boost fired (candidate reciprocal_rank_fusion df=4 < hybrid_search_async df=100 — correct per spec), boosted scoring.py chunk to pool front.

Holdout table (branch vs main worktree a027996, same index, read-only)

id	branch rank	main rank	boost	doc_boost
P2	None	None	1/0	0/0
H1	None	None	0/0	0/0
H2	None	None	0/0	0/0
H3	None	None	0/0	0/0
H4	None	None	0/0	0/0
H5	1	1	0/0	0/0
H6	None	None	0/0	0/0
H7	None	None	0/0	0/0
H8	4	4	0/0	0/0
H9	None	None	0/0	0/0
H10	None	None	0/0	0/0
H11	1	1	0/0	0/0
H12	2	2	0/0	0/0
N1	None	None	0/0	0/0
DOC	None	None	0/0	0/0
No regressions vs main (all ranks identical). H1-H12 = O1-holdout-v1 defined in scripts/o1_holdout_gate.py (no canonical H-set exists on main — verified by grep).

Caveats

• P2 gold rank 1 NOT achieved (branch and main alike): engine.py gold chunk is absent from the pre-rerank pool (verified via reranker-input spy: 5-chunk pool without engine.py), so no pool-stage boost can rescue it. Upstream pool-recall issue, PR52 territory (pool anchors / reranker threshold). O1 mechanism itself verified firing correctly at unit + live level.
• Live ranks show run-to-run variance (reranker/dense nondeterminism); table is one paired fresh-process comparison.
• DO NOT MERGE yet — awaiting owner review.

[coderabbitai]

Important

• 🔍 Trigger review

This repository does not receive automatic reviews because it has fewer than 10 stars.

⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Advanced

Run ID: f188837b-ed82-4b04-947f-35deeca7e55e

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}