Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
10 changes: 10 additions & 0 deletions docs/bugbot/quality-observability.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -967,6 +967,16 @@ waiting. Live progress remains informational.

### Qualification provenance

Treat scorer FP/FN labels separately from independently checked causal defects.
The [EV-07 failure audit](https://github.com/vypdev/copilot/issues/971) reproduces
the frozen current and prior negative results. Its eleven rejected category
matches contain detected hypotheses, rather than eleven absent hypotheses.
Every unmatched prediction and expectation retains an evidence row; unresolved
premises remain unresolved. Neither a taxonomy diagnostic nor a supported
fixture inconsistency changes a completed campaign's labels, calibration,
thresholds or qualification decision. A new campaign requires its own committed
implementation, driver and runtime identities. Production remains disabled.

The full-review qualification runner rederives the independent corpus score from the recorded predictions and checks the complete per-case ledger against the frozen plan, source, compiler configuration, CLI and invocation profile. A passing summary copied from another run cannot open full review. The CLI version must match before measured work starts. Historical campaigns remain reproducible from their preregistration commits with their original runner and rubric; stricter admission in a later revision does not rewrite their results. This local evaluation control adds no repository or organization setting and does not qualify a profile by itself.
Freeze the implementation, lockfile, shipping compiler configuration and all
eight evaluator driver hashes before measurement. The current runners reject
Expand Down
88 changes: 88 additions & 0 deletions quality/bugbot/evidence/issue-971/README.md

Large diffs are not rendered by default.

172 changes: 172 additions & 0 deletions quality/bugbot/evidence/issue-971/adjudications.json
Original file line number Diff line number Diff line change
@@ -0,0 +1,172 @@
[
{
"case": "rust-unchecked-input",
"classification": "taxonomy-mismatch",
"reason": "The prediction identifies the supplied parse().unwrap() panic on malformed request input. The scorer rejects security versus reliability; the severity also differs (high versus medium). No Rust process was executed in this audit.",
"evidence": [
"Supplied request-controlled parse/unwrap line and the scenario's explicit service-panic contract."
],
"scope": "supplied synthetic fixture; no claim about a private consumer",
"nextAction": "Retain original FP/FN and calibration contribution. No production correction follows from this taxonomy difference. Any rubric proposal requires independent review and a new preregistration."
},
{
"case": "shell-unquoted-path",
"classification": "taxonomy-mismatch",
"reason": "The prediction identifies unquoted word splitting at the exact changed command. Its proposed -- and quoting repair also addresses option interpretation. The frozen security expectation differs from correctness and high differs from medium.",
"evidence": [
"Fixed shell archive reproduction; original command and unchanged expected security taxonomy."
],
"scope": "supplied synthetic fixture; no claim about a private consumer",
"nextAction": "Retain original FP/FN and calibration contribution. No production correction follows from this taxonomy difference. Any rubric proposal requires independent review and a new preregistration."
},
{
"case": "py-short-circuit-unsafe-order",
"classification": "taxonomy-mismatch",
"reason": "Membership runs before isinstance. A list/dict step reaches set membership and raises TypeError; the prediction describes that exact path. Reliability differs from the frozen correctness label, and severity differs.",
"evidence": [
"Fixed Python short-circuit control; supplied expression order."
],
"scope": "supplied synthetic fixture; no claim about a private consumer",
"nextAction": "Retain original FP/FN and calibration contribution. No production correction follows from this taxonomy difference. Any rubric proposal requires independent review and a new preregistration."
},
{
"case": "ts-viewport-version-without-cleanup",
"classification": "taxonomy-mismatch",
"reason": "The pinned 1.0 mount appends a child and preserves the loader; the removed caller cleanup is the missing operation. The prediction follows that same version and path. Reliability differs from correctness.",
"evidence": [
"Supplied immutable 1.0 mount source and index.html; no framework/browser execution is claimed."
],
"scope": "supplied synthetic fixture; no claim about a private consumer",
"nextAction": "Retain original FP/FN and calibration contribution. No production correction follows from this taxonomy difference. Any rubric proposal requires independent review and a new preregistration."
},
{
"case": "kt-broad-catch-swallows-cancellation",
"classification": "taxonomy-mismatch",
"reason": "The scenario requires cancellation propagation and supplies no other guard. The new Throwable catch wraps that cancellation in DatabaseException/Left. The prediction identifies this same cause; reliability differs from correctness. It also reports line 5 rather than frozen line 4.",
"evidence": [
"Separately authored Kotlin CancellationException mapping control; supplied try/catch and cancellation contract."
],
"scope": "supplied synthetic fixture; no claim about a private consumer",
"nextAction": "Retain original FP/FN and calibration contribution. No production correction follows from this taxonomy difference. Any rubric proposal requires independent review and a new preregistration."
},
{
"case": "test-delegate-bypasses-stub",
"classification": "taxonomy-mismatch",
"reason": "The public reader returns right(row) and never invokes bindSession. The new assertion therefore fails with zero participant-projection calls. The prediction describes this exact route; reliability differs from correctness.",
"evidence": [
"observeCallPathFixture on its immutable owned fixture, independently of model output."
],
"scope": "supplied synthetic fixture; no claim about a private consumer",
"nextAction": "Retain original FP/FN and calibration contribution. No production correction follows from this taxonomy difference. Any rubric proposal requires independent review and a new preregistration."
},
{
"case": "action-old-input-on-new-version",
"classification": "taxonomy-mismatch",
"reason": "The selected revision reads enable-problem-matcher; the added problem-matcher belongs to the older revision. The default remains enabled and the supplied absent-asset contract fails registration. Correctness differs from reliability.",
"evidence": [
"Supplied exact-revision manifest/runtime contract; this synthetic Action is not executed on GitHub."
],
"scope": "supplied synthetic fixture; no claim about a private consumer",
"nextAction": "Retain original FP/FN and calibration contribution. No production correction follows from this taxonomy difference. Any rubric proposal requires independent review and a new preregistration."
},
{
"case": "request-transform-write",
"classification": "taxonomy-mismatch",
"reason": "The real authored request path emits no-cache, preserves it, and reaches a test requiring no-store. The prediction identifies that actual value and failure; security differs from correctness.",
"evidence": [
"observeRequestTransform on the immutable owned write-policy fixture."
],
"scope": "supplied synthetic fixture; no claim about a private consumer",
"nextAction": "Retain original FP/FN and calibration contribution. No production correction follows from this taxonomy difference. Any rubric proposal requires independent review and a new preregistration."
},
{
"case": "session-source-opaque-bootstrap",
"classification": "taxonomy-mismatch",
"reason": "Configured opaque.setup.ts persists the opaque token and its future expiry skips refresh. The compact-token assertion fails on that selected token. Reliability differs from correctness.",
"evidence": [
"observeInitialSessionSource with real Node HMAC/base64url and the immutable owned opaque fixture."
],
"scope": "supplied synthetic fixture; no claim about a private consumer",
"nextAction": "Retain original FP/FN and calibration contribution. No production correction follows from this taxonomy difference. Any rubric proposal requires independent review and a new preregistration."
},
{
"case": "session-source-expired-refresh",
"classification": "taxonomy-mismatch",
"reason": "The selected signed session expires before the fixed clock; the actual refresh response supplies an opaque token. The prediction follows that replacement and failed assertion. Reliability differs from correctness.",
"evidence": [
"observeInitialSessionSource with real Node HMAC/base64url and the immutable owned expired fixture."
],
"scope": "supplied synthetic fixture; no claim about a private consumer",
"nextAction": "Retain original FP/FN and calibration contribution. No production correction follows from this taxonomy difference. Any rubric proposal requires independent review and a new preregistration."
},
{
"case": "unittest-discovery-wrong-start-directory",
"classification": "taxonomy-mismatch",
"reason": "The selected start directory runs the unrelated health method and excludes both required manifest modules. The prediction explains that same actual selection. Reliability differs from correctness; the final location is the expected line 12 despite a stale line-9 string in its ID.",
"evidence": [
"Actual fixed owned Python unittest discovery: 1 health method, 0 export methods; passing control runs 19 methods including 5 export methods."
],
"scope": "supplied synthetic fixture; no claim about a private consumer",
"nextAction": "Retain original FP/FN and calibration contribution. No production correction follows from this taxonomy difference. Any rubric proposal requires independent review and a new preregistration."
},
{
"case": "sql-bounded-query",
"classification": "unresolved",
"reason": "The prediction needs tied created_at values and a required stable boundary. The scenario calls the order deterministic, but supplies no schema or uniqueness constraint to establish or refute ties. The conditional SQL concern is not independently proven for the declared fixture.",
"evidence": [
"No schema or tie witness is supplied; expected=[] is a scorer label, not independent proof."
],
"scope": "supplied fixture; independent repository truth remains separate from scorer labels",
"nextAction": "Obtain the authored schema/ordering contract before proposing either a detector change or a corpus correction."
},
{
"case": "actions-readonly-metadata",
"classification": "actual-causal-defect",
"reason": "The supplied added YAML has root-level steps and no jobs mapping. Read-only metadata access does not repair that workflow shape. The prediction identifies a causal defect in the supplied fixture, separately from the scorer's empty expectation.",
"evidence": [
"Parsed YAML structure; GitHub workflow syntax places steps under jobs.<job_id>.steps (official documentation checked 2026-11-19)."
],
"scope": "supplied fixture; independent repository truth remains separate from scorer labels",
"nextAction": "Submit the fixture/rubric inconsistency for separate review; do not relabel this completed campaign or suppress a supported workflow finding."
},
{
"case": "react-complete-dependencies",
"classification": "unresolved",
"reason": "The stale-response hypothesis is possible for an asynchronous loader, but the fixture supplies no loadUser implementation, scheduling guarantee, cleanup elsewhere, or interaction witness. Complete dependencies alone neither proves nor refutes that separate hypothesis.",
"evidence": [
"Only the three-line effect and its dependency intent are supplied; the actual loader contract is missing."
],
"scope": "supplied fixture; independent repository truth remains separate from scorer labels",
"nextAction": "Request a frozen loader/interaction witness. Do not infer a production prompt defect solely from expected=[]."
},
{
"case": "csharp-contained-path",
"classification": "unresolved",
"reason": "The prediction assumes Windows case-insensitive path resolution and a requirement to accept differently cased contained paths. Neither supported OS/filesystem policy nor that acceptance requirement is supplied. No .NET/Windows execution is available in this audit.",
"evidence": [
"Supplied ordinal containment check; absent target-platform and alternate-spelling contract."
],
"scope": "supplied fixture; independent repository truth remains separate from scorer labels",
"nextAction": "Keep the platform premise unresolved; obtain the actual supported filesystem contract before a narrow correction."
},
{
"case": "py-coverage-config-branch",
"classification": "unresolved",
"reason": "coverage run collects data while reporting applies fail_under, but the excerpt does not prove the complete job has no report step or that this change promises threshold enforcement. The separate threshold hypothesis cannot be resolved from this fragment alone.",
"evidence": [
"Supplied coverage command/configuration; no complete job or version-matched reporting observation."
],
"scope": "supplied fixture; independent repository truth remains separate from scorer labels",
"nextAction": "Inspect the complete authored job and reporting requirement; retain the scorer FP without inventing a confirmed missed threshold defect."
},
{
"case": "py-short-circuit-unhashable",
"classification": "unresolved",
"reason": "A fixed Python control proves a mapping with an allowed string key returns True. The prediction additionally requires the whole issue_steps value to be an array; the scenario describes arbitrary persisted JSON but does not specify that container-shape contract. The concrete behavior is proven; whether it violates the intended contract remains unresolved.",
"evidence": [
"Fixed Python mapping/list/short-circuit observations, with no consumer execution."
],
"scope": "supplied fixture; independent repository truth remains separate from scorer labels",
"nextAction": "Obtain the checkpoint container-shape requirement. Only then propose a narrow boundary or corpus follow-up."
}
]
75 changes: 75 additions & 0 deletions quality/bugbot/evidence/issue-971/audit.cjs
Original file line number Diff line number Diff line change
@@ -0,0 +1,75 @@
'use strict';

const assert = require('node:assert/strict');
const { createHash } = require('node:crypto');
const fs = require('node:fs');
const path = require('node:path');
const root = path.resolve(__dirname, '../../../..');
const api = require(path.join(root, 'build/api/index.js'));
const currentDirectory = path.join(root, 'quality/bugbot/evidence/issue-950/ev07-shadow');
const digest = bytes => createHash('sha256').update(bytes).digest('hex');
const read = file => JSON.parse(fs.readFileSync(file, 'utf8'));

async function audit() {
const corpusFile = path.join(root, 'quality/bugbot/corpus.v1.json');
const corpus = await api.loadBugbotBenchmark(corpusFile);
const ledgerFile = path.join(currentDirectory, 'campaign-provenance-attempts.jsonl');
const ledger = fs.readFileSync(ledgerFile, 'utf8').trim().split('\n').map(JSON.parse);
const controls = read(path.join(currentDirectory, 'campaign-provenance-controls.json'));
const starts = ledger.filter(row => row.event === 'start');
assert.equal(starts.length, 1);
assert.equal(starts[0].implementation, controls.implementation);
assert.equal(digest(fs.readFileSync(ledgerFile)), controls.artifactHashes['corpus-v3/corpus.jsonl']);
const cases = ledger.filter(row => row.event === 'case');
assert.equal(cases.length, 174);
assert.equal(new Set(cases.map(row => row.id)).size, 174);
assert(cases.every(row => row.status === 'completed'));
const predictions = { schemaVersion: 1, predictions: Object.fromEntries(cases.map(row => [row.id, row.predictions])) };
const score = api.evaluateBugbotBenchmark(corpus, predictions);
assert.deepEqual(score, controls.corpusScore.score);
const prior = read(path.join(__dirname, 'prior-03f8e6e3-predictions.json'));
const priorStart = read(path.join(__dirname, 'prior-03f8e6e3-start.json'));
const priorPlanFile = path.join(__dirname, 'prior-03f8e6e3-plan.json');
assert.equal(priorStart.implementation, read(priorPlanFile).implementation);
assert.equal(priorStart.plan, digest(fs.readFileSync(priorPlanFile)));
const priorScore = api.evaluateBugbotBenchmark(corpus, prior);
assert.deepEqual(priorScore, read(path.join(__dirname, 'prior-03f8e6e3-score.json')).score);
const adjudications = read(path.join(__dirname, 'adjudications.json'));
const rows = [];
for (const item of corpus.cases) {
const measured = predictions.predictions[item.id];
const result = api.evaluateBugbotBenchmark({ ...corpus, cases: [item] }, predictions);
if (!result.metrics.falsePositives && !result.metrics.falseNegatives) continue;
const judgment = adjudications.find(entry => entry.case === item.id);
assert(judgment, 'missing independent audit row: ' + item.id);
assert(judgment.reason && judgment.evidence.length && judgment.nextAction);
// Every discrepancy in this frozen campaign is one unmatched prediction,
// optionally paired with one unmatched expectation. Reject inventory drift.
assert.equal(result.metrics.matched, 0);
assert.equal(result.metrics.falsePositives, measured.length);
assert.equal(result.metrics.falseNegatives, item.expected.length);
for (const [kind, findings] of [['prediction', measured], ['expected', item.expected]]) {
findings.forEach((finding, index) => rows.push({ case: item.id, kind, index,
scorerLabel: kind === 'prediction' ? 'FP' : 'FN', id: finding.id,
findingSha256: digest(JSON.stringify(finding)), ...judgment }));
}
}
assert.equal(rows.filter(row => row.kind === 'prediction').length, 17);
assert.equal(rows.filter(row => row.kind === 'expected').length, 11);
assert.equal(adjudications.length, new Set(rows.map(row => row.case)).size);
return { schemaVersion: 1, method: 'unchanged public evaluator; independent causal assessments remain separate',
implementation: starts[0].implementation, priorImplementation: priorStart.implementation,
corpusSha256: digest(fs.readFileSync(corpusFile)), currentLedgerSha256: digest(fs.readFileSync(ledgerFile)),
priorPredictionsSha256: digest(fs.readFileSync(path.join(__dirname, 'prior-03f8e6e3-predictions.json'))),
score, priorScore, rows, modelCalls: 0, evaluatedPrWriteAttempts: 0,
qualification: 'not-qualified; no score, label, threshold, prompt or campaign was changed' };
}

if (require.main === module) audit().then(result => {
const output = process.argv[2];
if (output) fs.writeFileSync(output, JSON.stringify(result, null, 2) + '\n', { flag: 'wx' });
console.log(JSON.stringify({ current: result.score, prior: result.priorScore, rows: result.rows.length,
modelCalls: 0, evaluatedPrWriteAttempts: 0 }, null, 2));
}).catch(error => { console.error(error); process.exitCode = 1; });

module.exports = { audit };
Loading