Repository navigation
fix(validation): require complete stream ids and query counts in clean throughput bundles - #2921
Conversation
…n throughput bundles Clean-claimed bundles with a completed throughput phase must carry contiguous stream ids 1..S, and every stream of a TPC-H or TPC-DS bundle must cover the full canonical query set (22 and 99). Streams referencing unknown ids, empty stream results, and non-integer ids are rejected. Adds eleven rejection tests alongside the existing coverage suite.
![benchbox-oracle[bot]](https://avatars.githubusercontent.com/u/319264189?s=60&v=4){kind=small}
There was a problem hiding this comment.
oracle-verdict: pending for f44d3d563f68c135f4a2ddcadacd6d7ee239dfea
This result supersedes those for earlier heads; their open review threads still apply.
Tier: medium-high (changes comparison, validation or result logic: benchbox/validation/bundle.py; 151 code lines exceed 150).
Reviewers not available
- sol: absent (error)
- luna: absent (error)
- muse: absent (timeout)
- sonnet: skipped: same family as the author (claude)
- agy: disabled (CI authentication and quota output are not yet calibrated, and the workflow has no agy job)
There was a problem hiding this comment.
oracle-verdict: failure for f44d3d563f68c135f4a2ddcadacd6d7ee239dfea
This result supersedes those for earlier heads; their open review threads still apply.
Tier: medium-high (changes comparison, validation or result logic: benchbox/validation/bundle.py; 151 code lines exceed 150).
Decision: SHIP WITH FIXES.
Reviewer: muse (muse muse-spark-1.3-contributor, effort medium).
New per-stream throughput gate is bypassable and false-positive: it rejects valid combined bundles via stream-0 power rows, counts zero-timing rows as covered, ignores per-stream success flags, and allows single-stream throughput below the TPC minimum.
Reviewers not available
- sol: absent (error)
- luna: absent (error)
Ignore power-phase stream-0 queries when grouping throughput observations, require usable measurement timings per stream, reject completed phases with unsuccessful streams, and enforce the two-stream TPC minimum for TPC-H and TPC-DS coverage.
There was a problem hiding this comment.
oracle-verdict: success for 4bc5127cfd437ea421f32a792d16c4698aa8700c
This result supersedes those for earlier heads; their open review threads still apply.
Tier: medium-high (changes comparison, validation or result logic: benchbox/validation/bundle.py; 168 code lines exceed 150).
Decision: SHIP.
Reviewer: muse (muse muse-spark-1.3-contributor, effort medium).
All four earlier throughput-gate defects are fixed in the head commit and no new merge-blocking defects were found in the changed files.
Verified fixed: D1, D2, D3, D4. Resolve their review threads.
The publication parity harness replays validation from a slim checkout containing only the validator runtime files. Import the TPC stream minimum defensively with the same value as fallback so parity holds.
There was a problem hiding this comment.
oracle-verdict: success for 9b75926d336ad5c638de4d91bd8ac89a238c05c7
This result supersedes those for earlier heads; their open review threads still apply.
Tier: medium-high (changes comparison, validation or result logic: benchbox/validation/bundle.py; 171 code lines exceed 150).
Decision: SHIP.
Reviewer: muse (muse muse-spark-1.3-contributor, effort medium).
New per-stream throughput completeness check fails closed (usable-timing filter, contiguous 1..N streams, success, per-stream canonical coverage with TPC-DS variant folding, unknown-stream rejection) and refactors preserve existing coverage semantics. No correctness, verdict, or security defect requiring a fix before merge was found.
This branch was successfully deployed
1 participant
Clean-claimed bundles with a completed throughput phase must carry
contiguous stream ids 1..S, and every stream of a TPC-H or TPC-DS bundle
must cover the full canonical query set (22 and 99). Streams referencing
unknown ids, empty stream results, and non-integer ids are rejected.
Adds eleven rejection tests alongside the existing coverage suite.