Skip to content

paper: rerank ceiling as headline; SweRankEmbed-Small short-list vote - #153

Merged
ParsaVictor merged 1 commit into
mainfrom
s19-paper-ceiling
Oct 17, 2026
Merged

ParsaVictor merged 1 commit into
mainfrom
s19-paper-ceiling

Conversation

@ParsaVictor

Copy link
Copy Markdown
Owner

Brings the measured results from contributions-log §8.19–§8.21 (#150, #152) into paper/. This PR changes no benchmarks or code.

New tables

  • tables/ceiling.tex, the rerank ceiling (§8.20). It shows the share of issues whose every gold function appears in the engine's first k definitions:

    • Lite-274: 0.828 at k=300, 0.672 at k=30;
    • Verified: 0.790 at k=300;
    • dev: 0.776; SWE-Gym: 0.748.

    The text compares this with SweRank's best function Acc@5 of 0.814, but only for scale. It says explicitly that a recall ceiling is not an accuracy. It also names two higher systems, SweRank+ at 0.854 and SWE-Adept at 0.878, so we do not overclaim. The ceiling is now in the abstract, the introduction and the contributions list (new item 3).

  • tables/swerank.tex, SweRankEmbed-Small as the short-list encoder (§8.19, §8.21).

    • The k=30 vote passes the pre-registered bar on all four tuning halves. The k=100 vote fails on dev-rest and is reported as a negative result.
    • Run once on each holdout, function Acc@5 goes from 0.460 to 0.496 on Lite (+0.036 [0.011, 0.062], p=0.013) and from 0.428 to 0.472 on Verified (+0.044 [0.024, 0.066], p=0.0001).
    • It is marked research-only because of its cc-by-nc licence. The jina vote stays as the shippable row.

Other updates

  • Table 1 gets a SweRankEmbed-Small vote row. The text gives the gap to the same model used as a whole-repository retriever: −0.135.
  • The cost table now includes SweRank's per-issue cost (11–15 s on CPU, fp32).
  • The text notes that the jina holdout gains are a lower bound, because the stored runs kept only 20 candidates per issue.
  • A footnote and a new threat paragraph explain that the lists were regenerated with the release binary (299 of 300 file lists match the stored run).
  • The threats section has a new forking-paths paragraph.
  • REVIEWER-RISKS.md: P0-1: stable_project_id + .neuromesh/id #1 is rewritten around the ceiling and includes the counter-objection "recall@300 is not Acc@5". P0-5: stop workspace_root drift in reindex_incremental #5 is updated.

python paper/check.py passes: 39/39 citations and 25 labels. The paper has not been compiled, because there is no TeX on this machine.

🤖 Generated with Claude Code

…list vote

- New Table (ceiling.tex, log sec. 8.20): every gold function in the engine's top 300 for 0.828
  of Lite-274 (Verified 0.790; top 30 0.672 / 0.627), set against SweRank's 0.814 func Acc@5,
  explicitly as scale, not like-for-like; SweRank+ 0.854 and SWE-Adept 0.878 named as higher.
- New Table (swerank.tex, log sec. 8.19, 8.21): SweRankEmbed-Small vote k=30 passes the
  pre-registered bar on all four halves; k=100 fails dev-rest; run once: Lite-274 func@5
  0.460 -> 0.496 (+0.036 [0.011,0.062], p=0.013), Verified-458 0.428 -> 0.472 (+0.044, p=0.0001).
  jina vote stays the shippable row; its holdout gains are a lower bound (20 stored candidates).
- Footnote + threat: lists regenerated with the release binary (299/300 identical file lists).
- Abstract, intro, contributions, h2h, discussion, threats, conclusion, Table 1, cost table
  and REVIEWER-RISKS #1/#5 updated.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@ParsaVictor
ParsaVictor merged commit d26d3e0 into main Oct 17, 2026
3 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant