Adopt ①: P2 pre-rerank pool anchors + top-N recall floor + holdout calibration

[ManSio]

Root cause P2 (verified, was Open)

3-way RRF rewards multi-tier consensus: the P2 target is found by ONE tier only (BM25 rank 0 for \src/core/search/engine.py\ -> 1/(60+1)≈0.0164) while junk present in 2-3 tiers at mediocre ranks accumulates 2-3x that.
rf_results[:limit]\ (\engine.py:746) then amputates the single-tier winner before the reranker ever sees it.

Ruled out by reading: MMR is reorder-only (no drops); bucket weights favour the target (.py 1.0 vs .txt/.md 0.5); query expansion keeps the verbatim query as variants[0]. This resolves the open contradiction in KNOWN_ISSUES (standalone BM25 rank 0 vs hybrid loss): hybrid never returns raw BM25 order. Regression test failed before (target cut, pool = 10x experiments artifacts) / passes after.

Design

• \�nchor_tier_winners()\ (\scoring.py): pool = MMR-ordered RRF top-limit (order preserved for the no-reranker path) + missing per-tier top-1 winners appended from the full RRF list (reconstructed as fused entries when outside it), capped at \MAX_RERANKER_INPUT. Reranker \ op_n\ (=limit) unchanged. No P2 remainder — the fix is complete and cheap.
• Top-N recall floor \MAX_RERANKER_TOPN\ (default 0 = current behavior exactly): union of threshold-passers with top-N by score. Absolute 0.3 cut untouched.
• \ hreshold_calibration.calibrate_threshold()\ (F1-max on labeled holdout) raises \ValueError\ on eval sources (\eval/frozen/token_reduction_v3/…) — the sweep-on-eval rejection (EXPERIMENTS_LOG 2026-09-27) is encoded as a test, not a comment.
• Holdout protocol + sizes (>=10 queries disjoint from frozen eval-16, sha 31f1b0c9) documented in \ hreshold_calibration.py\ docstring. Default 0.3 unchanged pending a real holdout.
• Commit 1 also carries the previously-uncommitted sigmoid normalization found in the working tree (llama.cpp raw logits -> [0,1]); all 7 sigmoid tests green.

Numbers

• New suite: 18 tests in \ est_reranker_pool_and_threshold.py\ (P2 regression red->green, anchors/dedup/cap/fallback/limit=0, top-N off/on/cap on logged P3 logits, F1-selection, 6 eval markers -> ValueError).
• Affected suites: 143 passed (reranker 45 + pool/threshold 18 + searcher + hardening + ubatch + bs_audit); ruff clean; pre-commit gates green on both commits.
• Holdout/eval before-after: synthetic only. P3 logged values: target 0.271 stays cut at default (documented ranking failure, not scale); with \MAX_RERANKER_TOPN=4\ it returns as 4th by score. Live P2/P3 on the real index + holdout calibration NOT run here.

Caveats (open)

• Live validation on real index + llama.cpp needed (synthetic mechanism only).
• Holdout set (>=10 queries outside eval-16) not collected — owner's call.
• No split needed: P2 remainder is zero; threshold part ships as mechanism with default-preserving behavior.

DO NOT MERGE yet — review requested.

[coderabbitai]

Important

• 🔍 Trigger review

This repository does not receive automatic reviews because it has fewer than 10 stars.

⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Advanced

Run ID: 86bd06f3-a26c-4306-9ac9-fb8a866501db

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}