Skip to content

🔧🧑‍💻 - Audit EV 07 precision and calibration failures before another measured campaign #971

Description

@efraespada

Is there an existing issue for this?

  • I have searched the existing issues.

What area does this task affect?

CI/CD

Task description

Audit EV-07 precision and calibration failures before another measured campaign. Deliver one independently reviewable result at the qualification prerequisite stage.

Scope and acceptance criteria

  • Reproduce the frozen score without model calls and give every unmatched prediction/expected finding an evidence row: actual causal defect, disproved premise, missed hypothesis, taxonomy mismatch, or unresolved. Keep scorer labels separate from independent repository truth.
  • For each proposed substantive correction, name its smallest reproducible control and owning production boundary. File only independently supported, narrow corrective follow-ups; do not turn this audit into a general prompt rewrite.
  • Retain at least three scoring controls: current results reproduce, prior results reproduce under their original rubric, and category matching differences cannot retroactively pass a frozen campaign.
  • Keep the 174-case corpus, old labels, prompts and results immutable. Any independently justified evaluator revision needs separate review/versioning and a new preregistration; no tuning of thresholds or disclosure of expected answers to model inputs.

Current issues or inefficiencies

The source-bound campaign at d26d0d6 completed 174 cases but scored precision 80.23%, F1 83.13% and Brier 0.1883, below the published gates. Its 17 unmatched predictions and 11 unmatched expected findings include 11 nearby category mismatches; these counts are not independent proof of 17 false bugs or 11 absent hypotheses.

Observed baseline: unreleased develop ccefb27627cd627ccdb7d5666fd174c2ac1da07b from merged PR #951.

Expected impact

Make this one pending result independently inspectable and reproducible, with its actual passed, failed or unavailable evidence.
Completion does not authorize release, producer trust, profile qualification from partial measurements, or production activation.

Additional context or comments

Priority and dependencies

Documentation and implementation boundary

Documentation: docs/bugbot/quality-observability.mdx.
Keep Clean Architecture and current coverage/contract gates. When a later implementation changes a public contract, update its catalogued SDD, tests and docs together and validate specifications; update Graphify after code changes.
Read the effective #950 contract in the owning SDD and its current reconciliation amendments; historical pilot deadlines are not current investigation limits.

Fixed invariants and planning state

Backlog only: no in-progress, branch creation, model execution, deployment or activation requested by this planning step.
Keep BUGBOT_VERIFICATION_MODE=disabled. Evaluated PRs must have zero write attempts; retain frozen pilots and independent evaluator/oracle evidence.
Token usage is optional telemetry and cannot stop investigation or reject a supported verdict. Elapsed time and turn count do not impose research cutoffs; explicit cancellation and evidence/SHA integrity remain mandatory. Post-run latency qualification criteria remain distinct from session deadlines.

Activity

  1. moved this to Todo in vypdevon Nov 19, 2026
  2. changed the title [-]Audit EV-07 precision and calibration failures before another measured campaign[/-] [+]🔧 - Audit EV 07 precision and calibration failures before another measured campaign[/+] on Nov 19, 2026
  3. vypbot commented on Nov 19, 2026

    @vypbot
    Contributor

    Hi! I’m @vypbot, the Copilot assistant for this repository.

    I can answer questions, explain the codebase, propose implementation and test plans, review issues and pull requests for potential bugs or security problems, and help authorized maintainers apply changes.

    Try /copilot help to see the available commands, or mention @vypbot with your question.

  4. moved this from Todo to In Progress in vypdevon Nov 19, 2026
  5. added
    branchedIndicates that a branch has been created for this issue.
    on Nov 19, 2026
  6. changed the title [-]🔧 - Audit EV 07 precision and calibration failures before another measured campaign[/-] [+]🔧🧑‍💻 - Audit EV 07 precision and calibration failures before another measured campaign[/+] on Nov 19, 2026
  7. added
    state:workingWork can proceed on the verified branch or without a branch.
    on Nov 19, 2026
  8. vypbot commented on Nov 19, 2026

    @vypbot
    Contributor

    Implementation plan

    Current status: Ready to start. No action is required from maintainers before implementation.

    1. Reproduce the frozen campaign
      • Replay the 174-case results at d26d0d6 without model calls and record scorer outputs separately from independent repository evidence.
      • Preserve the corpus, prompts, labels, results, and published gates unchanged.
    2. Build an evidence row for every unmatched finding
      • Classify each unmatched prediction and expected finding as an actual defect, disproved premise, missed hypothesis, taxonomy mismatch, or unresolved.
      • Link each classification to inspectable repository evidence; do not infer truth from scorer labels or category proximity alone.
    3. Identify and scope supported corrections
      • For each supported substantive correction, name its smallest reproducible control and owning production boundary.
      • File only narrow corrective follow-ups backed by independent evidence; do not broaden this audit into a prompt rewrite.
    4. Retain frozen and historical scoring controls
      • Verify current results reproduce, prior results reproduce under their original rubric, and category matching cannot retroactively pass a frozen campaign.
      • Keep any independently justified evaluator revision separately versioned and reviewed, with a new preregistration.
    5. Publish the qualification prerequisite record
      • Document passed, failed, and unavailable evidence and link the campaign provenance artifacts and relevant quality-observability guidance.
      • Keep BUGBOT_VERIFICATION_MODE=disabled and record that this result does not authorize release, producer trust, profile qualification from partial measurements, or activation.

    Acceptance: A reviewable audit record reproduces the frozen campaign without model calls, includes an evidence row for every unmatched prediction and expected finding, demonstrates all three scoring controls, and leaves the 174-case corpus, labels, prompts, and results unchanged.

    Need something else? Mention the bot with a question or use /copilot help.

  9. added
    size: XXLIndicates an extremely large task or issue that requires significant time and effort to complete.
    55%Progress: 55%
    and removed on Nov 19, 2026
  10. vypbot commented on Nov 19, 2026

    @vypbot
    Contributor

    Progress · 62% estimated

    Assessed head 1f3fbdde · 1/10 supported
    0 need implementation · 9 need verification or clarification

    Verification or clarification needed

    • R15 · - [ ] Reproduce the frozen score without model calls and give every unmatched prediction/expected finding an evidence row: actual causal defect, disproved premise, missed hypothesi — Independently reproduce the frozen score and validate every unmatched row against repository truth. Requirement source · Evidence at assessed head
    • R16 · - [ ] For each proposed substantive correction, name its smallest reproducible control and owning production boundary. File only independently supported, narrow corrective follow-u — Verify each substantive correction has a smallest control, an owning boundary, and an independently supported narrow follow-up. Requirement source · Evidence at assessed head
    • R17 · - [ ] Retain at least three scoring controls: current results reproduce, prior results reproduce under their original rubric, and category matching differences cannot retroactively — Independently run all three controls, including frozen category-match behavior. Requirement source · Evidence at assessed head
    • R18 · - [ ] Keep the 174-case corpus, old labels, prompts and results immutable. Any independently justified evaluator revision needs separate review/versioning and a new preregistration — Verify the corpus, labels, prompts, and results remain unchanged and any evaluator revision is separately reviewed, versioned, and preregistered. Requirement source · Evidence at assessed head
    • R31 · Make this one pending result independently inspectable and reproducible, with its actual passed, failed or unavailable evidence. — Independently reproduce the record and verify its passed, failed, or unavailable evidence. Requirement source · Evidence at assessed head
    • R46 · Keep Clean Architecture and current coverage/contract gates. When a later implementation changes a public contract, update its catalogued SDD, tests and docs together and validate — Verify current coverage and contract gates and applicable specification and Graphify checks. Requirement source · Evidence at assessed head

    3 more pending requirements.

    Issue · Linked pull request · Full assessment

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

60%Progress: 60%branchedIndicates that a branch has been created for this issue.chorein-progressStart work on an admitted issue.maintenancepriority: highCritical issue, needs immediate attention.size: XXLIndicates an extremely large task or issue that requires significant time and effort to complete.state:workingWork can proceed on the verified branch or without a branch.

Projects

  • Status
    In Progress

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions