Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
17 changes: 17 additions & 0 deletions docs/bugbot/failure-scenarios.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -193,3 +193,20 @@ confirmation. Correct the named provider/response cause and recheck from the
current source; preserve identity and reconciliation checks. Completed writes
remain in place and must not be repeated blindly. See the
[structured error reference](/security-operations/operations/error-reference#conflict-eligibility-and-review-freshness).


### Review the default writer installation separately

The #977 artifact records an authenticated default-branch baseline, exact source
Action/bundle/workflow hashes and a proposed workflow that invokes the reviewed
Action SHA. Copying a workflow with a local `./` action onto an older default can
execute older code. Review that source pin together with its destination SHA;
stop and prepare a new review if either changes. Keep the source runner route,
serialization, private read permissions and native source checks intact.

A prepared artifact, an exact-byte fixture or a develop merge does not prove
installed readiness. Use the supported setup inspection/review/receipt route
coordinated with #814, with specific authorization, maintainer default merge and
a real canary only on an explicitly authorized test PR. Record the actual installed
SHA, run and comment result. No native canary or installation has occurred in
this proposal; production verification remains disabled.
2 changes: 1 addition & 1 deletion package.json
Original file line number Diff line number Diff line change
Expand Up @@ -69,7 +69,7 @@
"test": "jest",
"test:watch": "jest --watch",
"test:coverage": "jest --coverage --json --outputFile=coverage/setup-acceptance-jest.json && node scripts/validate-coverage-budgets.cjs && node scripts/validate-setup-assistant-acceptance.cjs coverage/setup-acceptance-jest.json && pnpm run test:extension",
"test:bugbot-shadow-verification": "node --test quality/bugbot/evidence/issue-950/shadow-verification/verification.test.cjs quality/bugbot/evidence/issue-950/normal-flow-shadow/config.test.cjs quality/bugbot/evidence/issue-950/normal-flow-shadow/paths.test.cjs quality/bugbot/evidence/issue-950/ev07-shadow/launch-full-review.test.cjs quality/bugbot/evidence/issue-950/ev07-shadow/review-runtime.test.cjs quality/bugbot/evidence/issue-950/ev07-shadow/score-review.test.cjs quality/bugbot/evidence/issue-950/ev07-shadow/plan.test.cjs quality/bugbot/evidence/issue-950/status-card-publication/publisher.test.cjs",
"test:bugbot-shadow-verification": "node --test quality/bugbot/evidence/issue-977/installation-state.test.cjs quality/bugbot/evidence/issue-950/shadow-verification/verification.test.cjs quality/bugbot/evidence/issue-950/normal-flow-shadow/config.test.cjs quality/bugbot/evidence/issue-950/normal-flow-shadow/paths.test.cjs quality/bugbot/evidence/issue-950/ev07-shadow/launch-full-review.test.cjs quality/bugbot/evidence/issue-950/ev07-shadow/review-runtime.test.cjs quality/bugbot/evidence/issue-950/ev07-shadow/score-review.test.cjs quality/bugbot/evidence/issue-950/ev07-shadow/plan.test.cjs quality/bugbot/evidence/issue-950/status-card-publication/publisher.test.cjs",
"eval:bugbot": "jest src/tooling/__tests__/bugbot_quality_eval.test.ts src/tooling/__tests__/bugbot_benchmark.test.ts src/tooling/__tests__/bugbot_request_transform_evidence.test.ts --runInBand",
"typecheck": "tsc --noEmit && pnpm run typecheck:extension",
"metrics:architecture": "node scripts/collect-architecture-metrics.cjs",
Expand Down
86 changes: 86 additions & 0 deletions quality/bugbot/evidence/issue-977/README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,86 @@
# Reviewed installation proposal — #977

**Prepared only; not installed, authorized or natively verified.** The authenticated
destination is vypdev/copilot (891781467), master at
ace49e4a7ce0f6a007fef5d11479eb9b7a5846e3. A readable workflow-directory listing
at that exact SHA proves the writer filename absent (`observed-default.json`).
A 404 alone is not absence proof. No default file, Variable, Secret, protection,
runner policy, release, consumer template or production mode was changed.

## Exact proposed artifact

`proposed-workflow.yml` is the #976 source workflow with exactly one substitution:
`uses: ./` becomes `uses: vypdev/copilot@1b024481d9d86cb4d4b10eb7f8569f9c8e2f0981`.
Copying the local-action workflow onto the older default revision would load the
older executable. The explicit reviewed SHA avoids that dependency. Preserve the
existing self-hosted/codex/Linux route, default-ref admission, per repository/PR
serialized queue, operation correlation, native evidence revalidation, private-PR
read permissions, publication PAT and last verified comment behavior. No new
credential scope is requested.

`manifest.json` binds the destination, before-state, exact Action manifest/bundle,
source workflow and proposed workflow hashes. Its SHA-256 is
3738d8b9c823739c745ad30e4bf438afdd7d8d3361f2abcd6e71430d0dad13b8. #976 / PR #989 is still unmerged; maintainer review/approval
of this source revision is pending. This is not a producer code attestation.
Do not install a different source or update a moved default from this receipt.

Coordinate the installation with #814's shared inspection/review/diff/receipt and
drift fences. This proposal does not implement a parallel updater, provision
resources or create a setup branch. The maintainer must authorize the exact
manifest/source/target and use the existing supported reviewed installation PR
route. If that route cannot preserve this exact reviewed pin, stop for its
supported migration rather than substituting a floating tag or a blanket update.
A future installation needs its own prepared-commit receipt and fresh authenticated
destination read; a develop merge or this artifact alone cannot close #977.

## Executed controls

Nine deterministic artifact controls pass: authenticated absence, exact installed
bytes, foreign workflow, foreign repository, moved default SHA, renamed default,
inaccessible parent, contradictory listing and the real YAML's source pin /
admission / runner / queue / credentials. `exact-installed-bytes` is only a
synthetic byte comparison, not evidence that a workflow is enabled or installed.
The first test attempt failed before running because it used an uninstalled YAML
package; it now uses the project's existing js-yaml dependency. Both logs remain.

The exact #976 Action bundle (SHA in manifest) also passes **14 packaged localhost
scenarios**, recorded in `packaged-results.json` and the per-scenario byte-exact
compressed logs. They cover progress update, duplicate no-write, independent
refresh, terminal update, delayed progress, replaced attempt, replacement retry,
push during preflight, denied private PR read, refused comment write, verified
empty conversation, missing conversation and legacy card progress/upgrade.
The copied historical canary was adapted only for the new operation correlation,
fixed source-bundle digest, new-empty-output requirement and denied private read;
the original #950 driver/history are unchanged. Fake comment PATCH/POST attempts,
including a refused write and an explicitly rejected GraphQL mutation audit,
are recorded as fake. Real GitHub writes and model calls are zero.

| Required distinction | Actual evidence |
| --- | --- |
| Absent | Authenticated exact-default parent listing and classifier control |
| Exact installed | Synthetic exact-byte classifier; actual installation pending |
| Incompatible/foreign | Foreign source pin and repository controls |
| Stale source | Moved default / renamed default controls; packaged push-preflight |
| Denied private read | Actual packaged worker gets local fake 403, nonzero exit, zero writes, prior card retained |
| Duplicate/replaced wakeup | Actual packaged identical-progress no write; prior attempt refused; new attempt updates |

Run the classifier with `node --test quality/bugbot/evidence/issue-977/installation-state.test.cjs`.
For the packaged audit, create a new empty directory and run
`node quality/bugbot/evidence/issue-977/packaged-canary.cjs /absolute/reviewed/build/github_action/index.js /absolute/new-empty-directory`.
The bundle digest must match the manifest before any case; never point it at an
unreviewed runtime and reinterpret its output as native installation evidence.

## Remaining acceptance

1. Maintainer review and integration of #976 at the exact approved revision.
2. Specific installation authorization for the exact manifest and fresh default.
3. Reviewed installation PR/default merge by the maintainer/Action, with its actual
prepared commit, matching receipt and resulting destination SHA.
4. An explicitly authorized test PR, then a genuinely run native dispatch/canary,
recorded run/job/result and current-source comment readback. No PR has been
chosen or mutated automatically; no native canary is claimed.
5. Read back the installed exact workflow/code and inspect native readiness.

Production verification remains disabled. EV-07 and qualification remain separate.

All 149 shadow controls pass, including the nine artifact cases; specification/catalog/documentation validators and Graphify pass. These local results do not change the pending installation/native acceptance.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
20 changes: 20 additions & 0 deletions quality/bugbot/evidence/issue-977/installation-state.cjs
Original file line number Diff line number Diff line change
@@ -0,0 +1,20 @@
'use strict';

// Read-only evidence classification for the reviewed installation proposal.
// It neither changes a target nor authorizes the existing setup installer.
const { createHash } = require('node:crypto');
const hash = value => createHash('sha256').update(value).digest('hex');

function installationState(plan, observed) {
if (observed.repositoryId !== plan.destination.repositoryId
|| observed.repository !== plan.destination.repository) return 'incompatible';
if (observed.defaultBranch !== plan.destination.defaultBranch
|| observed.head !== plan.destination.head) return 'stale-source';
if (typeof observed.workflow === 'string') return hash(observed.workflow) === plan.proposedWorkflowSha256
? 'exact-installed-bytes' : 'incompatible';
return observed.workflowParentReadable === true && Array.isArray(observed.workflowNames)
&& observed.workflowNames.length < 1000 && observed.workflowNames.every(name => typeof name === 'string')
&& !observed.workflowNames.includes(plan.path.split('/').at(-1)) ? 'absent' : 'unknown';
}

module.exports = { installationState };
48 changes: 48 additions & 0 deletions quality/bugbot/evidence/issue-977/installation-state.test.cjs
Original file line number Diff line number Diff line change
@@ -0,0 +1,48 @@
'use strict';

const assert = require('node:assert/strict');
const { readFileSync } = require('node:fs');
const { createHash } = require('node:crypto');
const path = require('node:path');
const { test } = require('node:test');
const { load: parse } = require('js-yaml');
const { installationState } = require('./installation-state.cjs');
const plan = require('./manifest.json');
const baseline = require('./observed-default.json');
const workflow = readFileSync(path.join(__dirname, 'proposed-workflow.yml'), 'utf8');

for (const [name, observed, expected] of [
['authenticated absent', baseline, 'absent'],
['exact installed bytes', { ...baseline, workflow }, 'exact-installed-bytes'],
['foreign workflow', { ...baseline, workflow: workflow.replace(plan.source.commit, 'f'.repeat(40)) }, 'incompatible'],
['foreign repository', { ...baseline, repositoryId: 1 }, 'incompatible'],
['moved default head', { ...baseline, head: 'a'.repeat(40) }, 'stale-source'],
['renamed default', { ...baseline, defaultBranch: 'other' }, 'stale-source'],
['inaccessible parent', { ...baseline, workflowParentReadable: false }, 'unknown'],
['contradictory listing', { ...baseline, workflowNames: ['copilot_bugbot_status_card.yml'] }, 'unknown'],
]) test('installation classification distinguishes ' + name, () => {
assert.equal(installationState(plan, observed), expected);
assert.equal(plan.installationAuthorized, false);
assert.equal(plan.installedDestinationSha, null);
assert.equal(plan.nativeCanary.status, 'not-run');
});

test('the minimal proposed workflow pins reviewed code and preserves serialization, admission and narrow credentials', () => {
assert.equal(createHash('sha256').update(workflow).digest('hex'), plan.proposedWorkflowSha256);
const yaml = parse(workflow), job = yaml.jobs.publish;
assert.deepEqual(Object.keys(yaml.on), ['workflow_dispatch']);
assert.deepEqual(Object.keys(yaml.on.workflow_dispatch.inputs).sort(), ['operation', 'pull_number', 'work']);
assert.equal(job.if, "${{ github.ref == format('refs/heads/{0}', github.event.repository.default_branch) }}");
assert.deepEqual(job['runs-on'], ['self-hosted', 'codex', 'Linux']);
assert.equal(job.concurrency.group, 'copilot-bugbot-status-${{ github.repository_id }}-${{ inputs.pull_number }}');
assert.equal(job.concurrency.queue, 'max');
assert.equal(job.concurrency['cancel-in-progress'], false);
assert.deepEqual(job.permissions, { actions: 'read', checks: 'read', contents: 'read', issues: 'read', 'pull-requests': 'read' });
const step = job.steps.find(step => step.name === 'Publish canonical comment');
assert.equal(step.uses, 'vypdev/copilot@' + plan.source.commit);
assert.equal(step.with.token, '${{ secrets.PAT }}');
assert.equal(step.env.COPILOT_EVIDENCE_TOKEN, '${{ github.token }}');
assert.equal(step.env.COPILOT_BUGBOT_STATUS_CARD_WORKFLOW, 'write-v1');
assert.equal(plan.productionVerification, 'disabled');
assert.equal(plan.consumerTemplatesChanged, false);
});
Binary file not shown.
Binary file not shown.
35 changes: 35 additions & 0 deletions quality/bugbot/evidence/issue-977/manifest.json
Original file line number Diff line number Diff line change
@@ -0,0 +1,35 @@
{
"schemaVersion": 1,
"kind": "review-only-publisher-installation-proposal",
"destination": {
"repository": "vypdev/copilot",
"repositoryId": 891781467,
"defaultBranch": "master",
"head": "ace49e4a7ce0f6a007fef5d11479eb9b7a5846e3"
},
"path": ".github/workflows/copilot_bugbot_status_card.yml",
"baseline": "absent with authenticated readable parent",
"source": {
"repository": "vypdev/copilot",
"commit": "1b024481d9d86cb4d4b10eb7f8569f9c8e2f0981",
"pullRequest": 989,
"merged": false,
"maintainerApproval": "pending",
"workflowSha256": "7a6b58ec81c41b49e8b4eed59b285f83da71251174fef2ee069138860fcc5b81",
"actionManifestSha256": "0875913058675d1a0e68117a8c98bb7508cf548f5775b645c5352bbd3bce127f",
"actionBundleSha256": "8bd2355979f551142848a0c9fdc3f5f8690fecb8a12da4910a0aad65e155e930"
},
"proposedWorkflowSha256": "6329588dc571722aba3c97ce3be8d1f5e84bff2fbf7ecb4da39ad4f36c66da90",
"sourceChange": "replace local ./ invocation with exact reviewed Action SHA; preserve all other workflow bytes",
"consumerTemplatesChanged": false,
"productionVerification": "disabled",
"installationAuthorized": false,
"installedDestinationSha": null,
"nativeCanary": {
"status": "not-run",
"explicitlyAuthorizedPullRequest": null,
"runId": null
},
"dependency": 976,
"coordination": 814
}
Binary file not shown.
Binary file not shown.
32 changes: 32 additions & 0 deletions quality/bugbot/evidence/issue-977/observed-default.json
Original file line number Diff line number Diff line change
@@ -0,0 +1,32 @@
{
"repository": "vypdev/copilot",
"repositoryId": 891781467,
"defaultBranch": "master",
"head": "ace49e4a7ce0f6a007fef5d11479eb9b7a5846e3",
"observedAt": "2026-11-19T20:33:51.219529+00:00",
"workflowParentReadable": true,
"workflowNames": [
"agent-cli-provisioning.yml",
"ci_check.yml",
"copilot_branch_sync.yml",
"copilot_close_inactive_issues.yml",
"copilot_comment_runtime.yml",
"copilot_commit.yml",
"copilot_credential_health.yml",
"copilot_deployment_orchestration.yml",
"copilot_issue.yml",
"copilot_issue_comment.yml",
"copilot_pull_request.yml",
"copilot_pull_request_approval.yml",
"copilot_pull_request_comment.yml",
"copilot_pull_request_merge_queue.yml",
"copilot_pull_request_review_state.yml",
"copilot_runner_diagnostics.yml",
"hotfix_workflow.yml",
"release_workflow.yml",
"repowise.yml",
"setup_platform_smoke.yml"
],
"workflowAbsent": true,
"evidence": "authenticated GitHub contents listing at exact default SHA; no installation mutation"
}
Binary file not shown.
Loading