Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
15 changes: 15 additions & 0 deletions docs/bugbot/quality-observability.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -1049,3 +1049,18 @@ Future qualification plans also pin the actual prepared executable's digest
and any launcher artifact. A version string alone cannot admit a model call.
Record and check the runtime proof for every corpus invocation; frozen
campaigns continue to replay through their own preregistration commits.

## Producer identity and receipt truth

A native receipt's matching run/App/SHA/artifact context is distinct from
independent evidence of the producer's executed code. The bounded #980 audit
runs the real packaged approval consumer with fabricated current receipts. Three
protected edits remain `protected-path` with no approval attempts while every
head read stays current; invalid run/SHA/App/policy/attempt and human/rollback
controls retain their existing decisions. Its positive cases use only a local
fake service, with no real GitHub write or qualification model call.

Establish the reviewed producer revision and acceptable independent trust before
any future enforced adoption. An artifact match or passing fake receipt case
cannot qualify a model profile or authorize activation. No new attestation
system or production policy is enabled here.
21 changes: 21 additions & 0 deletions docs/pull-requests/guarded-approval.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -248,3 +248,24 @@ Without a job-local token, the historical PAT-only path remains available for
credentials that support Checks; it cannot convert a denied read into approval.
GitHub documents the fine-grained PAT limitation in
[Managing personal access tokens](https://docs.github.com/en/authentication/keeping-your-account-and-data-secure/managing-your-personal-access-tokens#fine-grained-personal-access-tokens-limitations).

## Receipt producer trust

The receipt consumer authenticates the run, repository, attempt, SHA, App,
artifact and recorded custom check ID. A matching artifact is data provenance;
it does not independently establish which executable producer ran or whether a
fabricated clean result is truthful. The existing operator producer declaration
is a trust assumption, not an independent code attestation.

The [#980 packaged audit](https://github.com/vypdev/copilot/issues/980) keeps the
head current and fabricates clean current receipts. Fixed exclusions still
return `protected-path` for bundle, workflow and `action.yml` edits, with zero
approval attempts; foreign and stale receipt contexts also withhold approval.
Human decisions and same-SHA policy/rollback fences remain in force. These are
local fake-provider observations, not live approvals or profile qualification.

Before any future enforced profile, the operator must establish the exact
reviewed workflow/executable revision and acceptable producer trust outside
candidate-controlled artifact assertions. That prerequisite is not supplied by
this audit. A stronger attestation design would need its own SDD review; none is
automatically introduced or enabled. Production verification remains disabled.
2 changes: 1 addition & 1 deletion package.json
Original file line number Diff line number Diff line change
Expand Up @@ -69,7 +69,7 @@
"test": "jest",
"test:watch": "jest --watch",
"test:coverage": "jest --coverage --json --outputFile=coverage/setup-acceptance-jest.json && node scripts/validate-coverage-budgets.cjs && node scripts/validate-setup-assistant-acceptance.cjs coverage/setup-acceptance-jest.json && pnpm run test:extension",
"test:bugbot-shadow-verification": "node --test quality/bugbot/evidence/issue-950/shadow-verification/verification.test.cjs quality/bugbot/evidence/issue-950/normal-flow-shadow/config.test.cjs quality/bugbot/evidence/issue-950/normal-flow-shadow/paths.test.cjs quality/bugbot/evidence/issue-950/ev07-shadow/launch-full-review.test.cjs quality/bugbot/evidence/issue-950/ev07-shadow/review-runtime.test.cjs quality/bugbot/evidence/issue-950/ev07-shadow/score-review.test.cjs quality/bugbot/evidence/issue-950/ev07-shadow/plan.test.cjs quality/bugbot/evidence/issue-950/status-card-publication/publisher.test.cjs",
"test:bugbot-shadow-verification": "node --test quality/bugbot/evidence/issue-950/shadow-verification/verification.test.cjs quality/bugbot/evidence/issue-950/normal-flow-shadow/config.test.cjs quality/bugbot/evidence/issue-950/normal-flow-shadow/paths.test.cjs quality/bugbot/evidence/issue-950/ev07-shadow/launch-full-review.test.cjs quality/bugbot/evidence/issue-950/ev07-shadow/review-runtime.test.cjs quality/bugbot/evidence/issue-950/ev07-shadow/score-review.test.cjs quality/bugbot/evidence/issue-950/ev07-shadow/plan.test.cjs quality/bugbot/evidence/issue-950/status-card-publication/publisher.test.cjs quality/bugbot/evidence/issue-980/packaged-trust.test.cjs",
"eval:bugbot": "jest src/tooling/__tests__/bugbot_quality_eval.test.ts src/tooling/__tests__/bugbot_benchmark.test.ts src/tooling/__tests__/bugbot_request_transform_evidence.test.ts --runInBand",
"typecheck": "tsc --noEmit && pnpm run typecheck:extension",
"metrics:architecture": "node scripts/collect-architecture-metrics.cjs",
Expand Down
93 changes: 93 additions & 0 deletions quality/bugbot/evidence/issue-980/README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,93 @@
# Approval receipt producer trust — #980

The existing packaged approval consumer passes **14 controlled cases** against
only a loopback fake GitHub service. This audit changes no production approval
code or policy. The application baseline is ccefb27627cd627ccdb7d5666fd174c2ac1da07b;
its exact Action bundle and this driver's SHA-256 are in attempt-4.jsonl and
summary.json. Node v24.20.0 / macOS arm64 is the executed environment; no other
OS, model profile or installed default workflow is claimed.

## Exact trust claim

Authenticated run/repository/SHA/App/artifact/check identities bind a receipt to
its declared native context. They do not independently attest the executable
producer, its actual checkout, or the truth of a fabricated clean result. The
fixture deliberately supplies a syntactically clean current receipt. The custom
Review check may be in a different suite, as the existing contract permits;
its exact ID is bound through the authoritative run artifact.

The guarded policy's fixed protected-path decision independently prevents
approval when build/**, the workflow or action.yml changes. All head readbacks
remain current, so the result does not depend on a later stale-head fence.
An operator-attested/configured producer is still a trust assumption. Before a
future enforced profile, maintainers must establish the exact reviewed workflow
and executable revision and an execution context whose producer trust is
acceptable independently of candidate-controlled artifact contents. This audit
supplies no new code attestation or externally enforced control. If that
prerequisite needs an architecture/policy change, review its SDD first.

The current qualification campaigns are still unsuccessful. These fabricated
receipts are not model measurements, independent model oracles or authority to
activate any profile. Actual production verification remains disabled.

## Executed matrix

| Cases | Observed result |
| --- | --- |
| Changed build bundle, source workflow, action.yml | recommend / protected-path; zero approval attempts in every case |
| Foreign receipt run, foreign SHA, foreign App | pending / bugbot-missing; zero approval attempts |
| Foreign artifact run | blocked / evidence-unavailable; zero approval attempts |
| Prior policy/generation, prior attempt | pending / bugbot-missing at the same head; zero approval attempts |
| Ordinary current authorized producer data | One fake approval, observed back at the exact head |
| Rollback with stale same-head receipt | pending / bugbot-missing; zero approval attempts |
| Rollback with fresh current disabled-mode receipt | One fake approval with current policy identity |
| Existing human approval / changes requested | recommend / human-approved, or blocked / changes-requested; no competing approval |

Every observed head remains fixed and every case publishes its assessment to
the fake service. Local fake assessment/approval POSTs are intentionally counted
in the ledger; they are not writes to a real evaluated PR. There are **zero real
GitHub write attempts, zero qualification model calls and no production-mode
change**. The authority stays disabled; a stale enforced receipt is only a
negative data fixture. No human decision or verified real review is modified.

The first attempt could not bind localhost inside the sandbox and executed no
case. Its start ledger and error are retained. After permitting only this local
test execution, all 14 cases passed. The reusable Node test wrapper also passed.
Three existing consumer/domain/use-case Jest suites pass **236 tests**, retaining
the same-SHA policy, latest-attempt and rollback controls. Those existing counts
are not added again to the 14 packaged scenarios.

Run:

```bash
node quality/bugbot/evidence/issue-980/packaged-trust.cjs build/github_action/index.js /absolute/new-attempt.jsonl
node --test quality/bugbot/evidence/issue-980/packaged-trust.test.cjs
```

Each attempt requires a new ledger path and records the package/driver/runtime
identity before cases. The existing shadow-verification script includes the
wrapper for CI. The old #950 rollback script and its historical artifacts remain
unchanged. This result preserves the refuted broad approval-exploit claim as
refuted; it documents the narrower producer-trust limitation.

The source diff check identified an extra blank line at the driver EOF. Its removal changes no case or decision. The exact final driver was rerun: all 14 cases pass in `attempt-3.jsonl`; the previous ledgers/logs and original driver remain in history. The initial mandatory hook passed all 943 suites / 18,695 tests, build and lint before this formatting correction.


## Assessment publication counter-control

Native review of 79396c2 correctly found that blocked evidence and failed
assessment publication both exit 1. The audit now independently asserts
`publication: published` for every case and exactly one received fake assessment
POST carrying the canonical marker. All fourteen cases pass again in attempt 4
with the final driver hash. The source approval implementation is unchanged.

The causal control makes only the local foreign-artifact-run assessment POST
return 403. An isolated audit copy with only the new publication assertions
removed falsely reports 14 passed while its recorded assessment is failed. The
corrected driver rejects those same inputs at that case and records no passing
finish. Both exact ledgers and raw compressed outputs are retained, together
with source hashes and a deterministic derivation recipe. That expected rejection
is not a new positive scenario or a model qualification measurement. Fake write
attempts remain counted; real GitHub/evaluated-PR writes and model calls stay zero.

All 141 shadow controls pass with the publication assertions. The earlier sandbox run passed 140 and could not bind loopback for the packaged wrapper (EPERM); its exact raw output is retained, followed by the permitted localhost-only full pass.
1 change: 1 addition & 0 deletions quality/bugbot/evidence/issue-980/attempt-1.jsonl
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
{"event":"start","at":"2026-11-19T19:47:11.588Z","node":"v24.20.0","platform":"darwin","architecture":"arm64","sourceBaseline":"ccefb27627cd627ccdb7d5666fd174c2ac1da07b","bundleSha256":"c5fe206b2027c951b4f22c64d210ea540b6777eee3c7cd754ba944a771f0f0e6","driverSha256":"ee5cf50d010603b2430151a3074aef95a9065bc66b83d72a2787acfb3bc36f0f","authority":{"mode":"disabled","generation":2,"policyDigest":"2222222222222222222222222222222222222222222222222222222222222222","workflowId":31,"sourceAppId":12},"fixturePolicy":"guarded","realGitHubWriteAttempts":0,"qualificationModelCalls":0}
Binary file added quality/bugbot/evidence/issue-980/attempt-1.log.gz
Binary file not shown.
Loading