Repository navigation
Description
Is there an existing issue for this?
- I have searched the existing issues.
What area does this task affect?
CI/CD
Task description
Audit EV-07 precision and calibration failures before another measured campaign. Deliver one independently reviewable result at the qualification prerequisite stage.
Scope and acceptance criteria
- Reproduce the frozen score without model calls and give every unmatched prediction/expected finding an evidence row: actual causal defect, disproved premise, missed hypothesis, taxonomy mismatch, or unresolved. Keep scorer labels separate from independent repository truth.
- For each proposed substantive correction, name its smallest reproducible control and owning production boundary. File only independently supported, narrow corrective follow-ups; do not turn this audit into a general prompt rewrite.
- Retain at least three scoring controls: current results reproduce, prior results reproduce under their original rubric, and category matching differences cannot retroactively pass a frozen campaign.
- Keep the 174-case corpus, old labels, prompts and results immutable. Any independently justified evaluator revision needs separate review/versioning and a new preregistration; no tuning of thresholds or disclosure of expected answers to model inputs.
Current issues or inefficiencies
The source-bound campaign at d26d0d6 completed 174 cases but scored precision 80.23%, F1 83.13% and Brier 0.1883, below the published gates. Its 17 unmatched predictions and 11 unmatched expected findings include 11 nearby category mismatches; these counts are not independent proof of 17 false bugs or 11 absent hypotheses.
Observed baseline: unreleased develop ccefb27627cd627ccdb7d5666fd174c2ac1da07b from merged PR #951.
- quality/bugbot/evidence/issue-950/ev07-shadow/campaign-provenance-results.md
- quality/bugbot/evidence/issue-950/ev07-shadow/campaign-provenance-attempts.jsonl
Expected impact
Make this one pending result independently inspectable and reproducible, with its actual passed, failed or unavailable evidence.
Completion does not authorize release, producer trust, profile qualification from partial measurements, or production activation.
Additional context or comments
Priority and dependencies
- Priority: high. Stage: Qualification prerequisite.
- Blocked by: None; may be prepared independently.
- Follow-up to ✨🧑💻 - Verify Bugbot findings against repository evidence before publication #950 and merged PR feat: Verify Bugbot findings against repository evidence before publication (#950) #951, not a request to reopen or extend that delivery.
- Final closure and observed limitations.
Documentation and implementation boundary
Documentation: docs/bugbot/quality-observability.mdx.
Keep Clean Architecture and current coverage/contract gates. When a later implementation changes a public contract, update its catalogued SDD, tests and docs together and validate specifications; update Graphify after code changes.
Read the effective #950 contract in the owning SDD and its current reconciliation amendments; historical pilot deadlines are not current investigation limits.
Fixed invariants and planning state
Backlog only: no in-progress, branch creation, model execution, deployment or activation requested by this planning step.
Keep BUGBOT_VERIFICATION_MODE=disabled. Evaluated PRs must have zero write attempts; retain frozen pilots and independent evaluator/oracle evidence.
Token usage is optional telemetry and cannot stop investigation or reject a supported verdict. Elapsed time and turn count do not impose research cutoffs; explicit cancellation and evidence/SHA integrity remain mandatory. Post-run latency qualification criteria remain distinct from session deadlines.
Metadata
Metadata
Assignees
Labels
Type
Projects
- StatusShow more project fieldsIn Progress
Activity
[-]Audit EV-07 precision and calibration failures before another measured campaign[/-][+]🔧 - Audit EV 07 precision and calibration failures before another measured campaign[/+]vypbot commentedon Nov 19, 2026
Hi! I’m @vypbot, the Copilot assistant for this repository.
I can answer questions, explain the codebase, propose implementation and test plans, review issues and pull requests for potential bugs or security problems, and help authorized maintainers apply changes.
Try
/copilot helpto see the available commands, or mention @vypbot with your question.[-]🔧 - Audit EV 07 precision and calibration failures before another measured campaign[/-][+]🔧🧑💻 - Audit EV 07 precision and calibration failures before another measured campaign[/+]vypbot commentedon Nov 19, 2026
Implementation plan
Acceptance: A reviewable audit record reproduces the frozen campaign without model calls, includes an evidence row for every unmatched prediction and expected finding, demonstrates all three scoring controls, and leaves the 174-case corpus, labels, prompts, and results unchanged.
Need something else? Mention the bot with a question or use
/copilot help.chore-971-audit-ev-07-precision-and-calibration-failures-before-anoth…
vypbot commentedon Nov 19, 2026
Progress · 62% estimated
Assessed head
1f3fbdde· 1/10 supported0 need implementation · 9 need verification or clarification
Verification or clarification needed
3 more pending requirements.
Issue · Linked pull request · Full assessment
chore-971-audit-ev-07-precision-and-calibration-failures-before-anoth…