fix(ranking): honor comparison budget for rankings of 10 or fewer items

[RapidPoseidon]

Problem

create_ranking_job_definition (and the deprecated create_ranking_order) silently ignores comparison_budget_per_ranking and random_comparisons_ratio when the largest ranking group has 10 or fewer datapoints: MultiRankingWorkflow switches to the FullPermutationPairMaker, which emits each unique pair exactly once with responses_per_comparison responses.

Real-world impact: a 7-logo ranking launched with comparison_budget_per_ranking=5000 (job job_1S0zegXPmIhA6E, workflow rwf_1S0zfTbMe700Bu) collected only C(7,2) = 21 responses instead of ~5000, with no warning of any kind.

Fix

The FullPermutationPairMakerConfig contract has no budget field, so the budget is honored through the referee instead:

• In full-permutation mode, each pair now requires comparison_budget_per_ranking // n_pairs comparisons (times responses_per_comparison), i.e. the budget is spread evenly over all unique pairs. Rounding is downward so the budget is never exceeded — the 7-logo example now yields 21 × 238 = 4998 responses.
• If the budget is smaller than the number of pairs, every pair is still compared once (the minimum for full coverage) and a warning is logged that the budget will be exceeded.
• The online (>10 items) path is unchanged.
• For jobs with multiple rankings of different sizes, the spread is computed from the largest group (consistent with how the pair-maker threshold is already chosen); smaller groups collect proportionally fewer comparisons.

Docstrings and the ranking docs now spell out both behaviors, and MultiRankingWorkflow has unit tests covering the budget spread, the responses_per_comparison multiplier, the below-coverage warning, and the online path.

Verification

• uv run pytest tests/rapidata_client/job — 18 passed (4 pre-existing failures in tests/rapidata_client/audience also fail on main, unrelated)
• uv run pyright src/rapidata/rapidata_client — 0 errors
• uv run black --check on touched files — clean

🔗 Session: session-cdc00927

🤖 Generated with Claude Code

[LinoGiger]

why does this live on the sdk and not on the backend?
@LucStr

[RapidPoseidon]

Because the bug is in the SDK's own translation layer, not in the backend's behavior:

• comparison_budget_per_ranking is an SDK-invented convenience parameter — the backend API has no such concept. The endpoint takes a pairMakerConfig + refereeConfig, and it executes exactly what it's told. It's the SDK that decides (in MultiRankingWorkflow, based on the ≤10-item threshold) to swap the OnlinePairMaker for a FullPermutationPairMaker, and in doing so it dropped the budget the user passed. The layer that makes the substitution is the layer that should keep the parameter's promise.
• The backend contract's FullPermutationPairMakerConfig has no budget field, so honoring the budget backend-side would mean a contract change in rapidata-contracts, backend implementation + deploy, and an SDK client regeneration. The SDK fix expresses the same outcome through the existing contract (refereeConfig.responsesRequired) and works immediately, including against current prod.

If we'd rather have a budget-aware full-permutation pair maker as a first-class backend concept (e.g. FullPermutationPairMakerConfig.totalComparisonBudget), I'm happy to do that as a follow-up across contracts + backend and then thin this SDK logic back out — but I'd still land this fix now, since today the parameter is silently ignored for every ranking of ≤10 items.