Repository navigation
feat(explorer): distinguish execution variants in identity, ranking and labels - #2913
Conversation
…nd labels Pipeline records variant_id, variant_label, requested execution engine and gateway routing from bundle platform blocks into the read model, cohort rows and leaderboard aggregates. Engine variants rank as separate rows, routed gateway runs are excluded from native ranking, and legacy bundles keep platform-id identity. Read model advances to version 16.
|
@codex review |
![benchbox-oracle[bot]](https://avatars.githubusercontent.com/u/319264189?s=60&v=4){kind=small}
There was a problem hiding this comment.
oracle-verdict: failure for a9ff9af2c7736efce3848f284c17475474cbdada
This result supersedes those for earlier heads; their open review threads still apply.
Tier: medium-high (223 code lines exceed 150; touches 2 areas).
Decision: SHIP WITH FIXES.
Reviewer: muse (muse muse-spark-1.3-contributor, effort medium).
Variant identity splits ranking correctly in most paths, but the corpus depth gate disagrees with the pipeline on engine-only and gateway-routed identities, and the Explorer frontend mixes variant and platform keys for evidence counting, navigation, and empty-string fallback.
Defects without a review thread
- High
results-data/validate_corpus.py:489: D1: Corpus depth gate counts gateway-routed bundles the Explorer excludes from ranking
Explorer ranking excludes gateway-routed bundles via ranking_exclusion_reason=gateway_routed, and CORPUS_NOTES states they contribute no identity, but results-data/validate_corpus.py exclusion_reason and cohort_platforms have no gateway check. A cohort with 2 native plus 1 gateway-routed bundle passes the >=3-identity gate while the Explorer ranks only 2, so the merge gate validates a depth claim the published ranking does not satisfy. - Medium
results-explorer/src/pages/Leaderboard.tsx:393: D4: Outside-leaderboard evidence set mixes platform_id and variant keys
publicPlatformIdsOutsideLeaderboard compares variant-keyed result ids against a set built from bare platform_id, so every engine or gateway variant is falsely reported as outside leaderboard evidence. The scope summary counts are wrong once variants exist.
Defects outside the diff count like the others: fix them before merge.
Reviewers not available
- sol: absent (error)
- luna: absent (error)
|
@codex review |
There was a problem hiding this comment.
oracle-verdict: failure for f0dfa4e772fc4c516f1d68ee61e5b4d63cb7172b
This result supersedes those for earlier heads; their open review threads still apply.
Tier: medium-high (223 code lines exceed 150; touches 2 areas).
Decision: SHIP WITH FIXES.
Reviewer: muse (muse muse-spark-1.3-contributor, effort medium).
The 15->16 bump in the two re-reviewed files is consistent with contract.py, db.ts and checker v16; no new defect in those files. The six earlier variant-identity defects remain: validator still diverges from pipeline ranking/exclusion and frontend/backend still diverge on navigation keys and empty-string fallbacks, so merge must wait for those fixes.
Still open from earlier reviews
- High
results-data/validate_corpus.py:489: D1: Corpus depth gate counts gateway-routed bundles the Explorer excludes from ranking - High
results-data/validate_corpus.py:429: D2: Validator ignores execution-engine variant split used by pipeline - High
results-explorer/src/components/MetaLeaderboard.tsx:364: D3: Variant leaderboard navigation collapses variants to platform_id - Medium
results-explorer/src/pages/Leaderboard.tsx:393: D4: Outside-leaderboard evidence set mixes platform_id and variant keys - Medium
results-explorer/src/components/MetaLeaderboard.tsx:598: D5: Frontend variant key fallback diverges from backend for empty strings - Medium
_project/scripts/explorer_pipeline/duckdb_builder.py:1000: D6: benchmark_rankings lacks empty-variant fallback used elsewhere
Scope: files changed since head a9ff9af2c7736efce3848f284c17475474cbdada was reviewed.
|
@codex review |
There was a problem hiding this comment.
oracle-verdict: failure for 249c4b8c3dc739b4462510d879c189bfdf37d344
This result supersedes those for earlier heads; their open review threads still apply.
Tier: medium-high (223 code lines exceed 150; touches 2 areas).
Decision: SHIP WITH FIXES.
Reviewer: muse (muse muse-spark-1.3-contributor, effort medium).
Prior defects D1-D6 are fixed for common cases. One residual identity mismatch remains: the validator's variant precedence/normalization does not match the pipeline transformer, so the depth gate can disagree with Explorer ranking on bundles with blank/padded variants or variant equal to the platform name with a non-default engine.
Verified fixed: D1, D2, D3, D4, D5, D6. Resolve their review threads.
Scope: files changed since head f0dfa4e772fc4c516f1d68ee61e5b4d63cb7172b was reviewed.
|
@codex review |
There was a problem hiding this comment.
oracle-verdict: success for c33aeae0b09593279b946fb25ef6692c8d754046
This result supersedes those for earlier heads; their open review threads still apply.
Tier: medium-high (223 code lines exceed 150; touches 2 areas).
Decision: SHIP.
Reviewer: muse (muse muse-spark-1.3-contributor, effort medium).
Variant identity and gateway exclusion now match the pipeline with covering tests; no merge-blocking defect remains in the three newly changed files.
Verified fixed: D7. Resolve their review threads.
Scope: files changed since head 249c4b8c3dc739b4462510d879c189bfdf37d344 was reviewed.
This branch was successfully deployed
1 participant
Pipeline records variant_id, variant_label, requested execution engine and
gateway routing from bundle platform blocks into the read model, cohort rows
and leaderboard aggregates. Engine variants rank as separate rows, routed
gateway runs are excluded from native ranking, and legacy bundles keep
platform-id identity. Read model advances to version 16.