[ParsaVictor]

Rebased onto main 6a5c072 (§8.19 now sits before §8.20). There are 7 commits; the pre-registration was first pushed as fca506a at 2026-10-17 09:04:54 +0330, before any SweRank number. It is now 79b257d after the rebases, with its author date kept.

Question: does Salesforce SweRankEmbed-Small (137M) do better than jina-code-v2 as the embedder of the dense short-list vote (§8.18)? Everything else stays fixed: RRF k=60, w=0.5, same texts.

Pre-registered bar (docs/research/prereg-s19-swerank.md): func Acc@5 at least engine + 0.02 on each of dev-fast, dev-rest, gym-a and gym-b.

Results

func Acc@5	dev-fast (52)	dev-rest (158)	gym-a (132)	gym-b (134)	pooled (476)
engine	0.308	0.329	0.258	0.254	0.286
+ jina vote k=30	0.327 (+0.019)	0.342 (+0.013)	0.288 (+0.030)	0.313 (+0.060)	0.317
+ SweRank vote k=30	0.346 (+0.038)	0.361 (+0.032)	0.318 (+0.061)	0.306 (+0.052)	0.332
+ SweRank vote k=100	0.346 (+0.038)	0.335 (+0.006)	0.326 (+0.068)	0.328 (+0.075)	0.332
SweRank order alone, top 30	0.288	0.361	0.280	0.313	–
jina order alone, top 30	0.269	0.278	0.258	0.269	–
SweRank vote w=1, k=30 / k=100	0.346 / 0.346	0.354 / 0.367	0.318 / 0.318	0.321 / 0.328	–

Verdicts

k=30 passes on all four halves. On dev, jina misses the bar on both halves.

• Pooled vs engine: func@5 +0.046 [0.027, 0.067], 24 issues gained / 2 lost, p<0.0001. func@10 +0.027 (p=0.0002). func@1 +0.013 (p=0.24).
• Pooled vs the jina vote: func@5 +0.015 [−0.002, 0.032], 11 gained / 4 lost, p=0.12, so better but not significantly. func@1 −0.008.

k=100 fails on dev-rest (+0.006). Pooled func@5 equals k=30 (0.332, 6 gained / 6 lost head-to-head).

Holdout variant, by the rule committed before any k=100 number (480e92f): k=30, w=0.5. No Lite or Verified run was done here.

Seconds per issue (p50, CPU)

	dev	gym
SweRank k=30 (fp32 torch, 6 threads)	10.8 s	17.9 s
jina k=30 (int8 ONNX)	10.3 s	6.2 s
SweRank k=100 (k=30 texts cached)	14.6 s	20.0 s

The gym k=30 SweRank time was measured with another heavy job on the CPU. The k=100 times are upper bounds: an orphaned second copy of the run competed for the CPU, and its duplicate rows were identical and were dropped.

Caveats

• Licence: the model is cc-by-nc-4.0, so it is not shippable as an engine default.
• Implementation: swerank_embed.py is my own torch re-implementation of NomicBERT and needs no trust_remote_code. It was checked by a sanity test (model-card example: diagonal 0.60/0.63 vs ≤0.03 off-diagonal; padding gives the same output), not by a numerical comparison against the reference code.
• Speed: an int8 ONNX export would be the fair speed comparison with jina.

Changes

• dense_shortlist.py --model swerank-embed-small (the jina default is unchanged)
• swerank_embed.py
• dense_compare.py
• paired.py: a vote-vs-vote mode via a fourth --set field
• §8.19 in contributions-log
• prereg-s19-swerank.md

paper-draft.md, README and config are untouched. Data files (outside git, in swebench/): dense-sw-{dev,gym}.jsonl, dense-sw100-{dev,gym}.jsonl, cache emb-cache/shortlist-swerank.npz.

🤖 Generated with Claude Code