Skip to content

feat: Verify Bugbot findings against repository evidence before publication (#950) - #951

Merged
efraespada merged 119 commits into
developfrom
feature/950-verify-bugbot-findings-against-repository-evidence-before-publication
Nov 19, 2026
Merged

efraespada merged 119 commits into
developfrom
feature/950-verify-bugbot-findings-against-repository-evidence-before-publication

Conversation

@efraespada

@efraespada efraespada commented Nov 9, 2026 •

Copy link
Copy Markdown
Member

Refs #950.

Review corrections

  • Reflowed the issue's over-limit source line. The real capture rule accepted the resulting body, and the Action published SDD revision f8de480 on the existing branch with first commit 80407b7 preserved.
  • Defined the policy-generation and latest-attempt approval receipt in the Bugbot SDDs. Tested it through the real approval repository and decision policy, including same-head enforcement, incomplete and failed later attempts, live policy reread and rollback.
  • Preregistered equivalent detector inspection and matched closed verifier tools. The first profile uses only pinned source, resolved contracts and matching CI evidence. Local reproductions remain evaluator-only.

#895 deterministic verdict and coverage audit

One verdict-limit definition drives the prompt and application checks.
100 simulated calls passed the deterministic boundary and diagnostic
controls with zero runner writes and zero model calls. The historical
generic rejections remain indeterminate.

The actual pinned commands show the mutant CSF entry arriving through
executed coverage, remap and map merge, then JSON-summary and inventory
rejection. The observed and baseline reports match. Source establishes
the conditional route. An actual report-entry claim needs authenticated
execution evidence. The PR stays in draft and activation stays disabled.

Contract, link audit and reproduction ·
Controls · Source and execution receipts ·
Artifacts · CI ·
Head checks

#895 configureVitest lifecycle replay

Six fresh Sol sessions used compact packets that add lock-authenticated
evidence of project resolution, awaited configureVitest hooks, and V8
initialization and effective options. The controls verify the packet
and prompt budgets and that the historical packets fail the new timing
checks. The runner recorded zero GitHub write attempts.

Supported original refutations went from 3/3 to 2/3
and supported mutant confirmations from 0/3 to 0/3.
Sol did not meet the entry criterion for a separate runner-directed search.
The PR stays in draft and production activation stays disabled.

Preregistration · Simulated controls ·
Historical packet check · Trial ledger ·
Report · Deterministic CI ·
Commit checks

#895 complete-evidence regression

Phase 1 ran 12 closed one-call trials. All stayed within budget and the
runner recorded zero GitHub write attempts.

Model Supported original refutations Supported mutant confirmations
gpt-6-luna 1/3 0/3
gpt-6.1-sol 3/3 0/3

Sol confirmed one mutant with the correct downstream explanation, but
the packet omitted the lifecycle evidence linking its hook to the
effective coverage options. The other two Sol mutant responses were
inconclusive and named that gap. Luna refuted all three mutants. Neither
profile passed, so phase 2 was gated out. The PR stays in draft and
production activation stays disabled.

Preregistration · Simulated controls ·
Trial ledger · Lifecycle audit ·
Report and per-trial scores · Deterministic CI ·
Commit checks

#895 path-pattern regression pilot

The known-case regression kept the same pinned #895 candidate and initial
evidence while exposing the path-pattern search contract. The 36 slots
recorded 44 provider attempts and zero GitHub writes.

Model Admitted on demand Config recovered / delivered / cited Supported refutations Supported confirmations Gate
gpt-6-luna 0/6 0 / 0 / 0 0/3 0/3 Did not pass
gpt-6.1-sol 3/6 0 / 0 / 0 0/3 0/3 Did not pass

The report compares this run with the historical pilot and lists each
trial's first stopping stage. The PR remains in draft and production
activation stays disabled.

Preregistration · Simulated controls ·
Trial ledger · Scores · Report ·
Deterministic CI · Commit checks

#895 bounded Git path pattern controls

The versioned common adapter matches scope-relative paths with one-segment
wildcards and can filter a literal search before opening blobs. Exact
basename matching and all 17 historical replay outcomes are preserved.

Both new modes delivered the decisive configuration on each pinned tree
in four separate simulated controls. Each filtered to one path and opened
one file. The verifier cited the Git receipt alongside the authenticated
dependencies. The largest final package and prompt were 14,275 and 15,104
bytes. Both modes returned a budget result with zero opened files when
the pattern admitted 123 paths. The runner made zero model calls and
recorded zero GitHub write attempts. Production activation stays disabled
and the PR remains in draft.

Search contract, results and reproduction · Control record ·
Deterministic controls · Versioned adapter ·
Deterministic CI · Commit checks

#895 deterministic Git request replay

The frozen runner reproduced all 17 historical denials. The corrected
adapter admitted 16 searches after normalizing the authorized root alias.
Seven had no exact basename match, seven hit the literal search file
budget, and two combined both outcomes. The three-file read remains denied.
None of the historical inputs recovered the decisive configuration.

Separate exact-name, one-file, and partial-result controls delivered the
pinned configuration on both trees and validated the Git and dependency
receipts within the final packet and prompt limits. This replay made zero
model calls and recorded zero GitHub write attempts. The PR stays in draft
with production activation disabled.

Before/after table and reproduction · Exact request and delivery record ·
Deterministic controls · Versioned adapter ·
Deterministic CI · Commit checks

#895 reserved model diagnostic

Both profiles received the same pinned evidence and first-call schema
within each tree. The initial packet withheld the effective project
configuration. The on-demand arm could add it only through one typed
request to the common Git adapter.

Model On-demand refutations On-demand confirmations Fixed supported Repeat supported Result
gpt-6-luna 0/3 0/3 0/6 0/6 Did not pass
gpt-6.1-sol 0/3 0/3 0/6 0/6 Did not pass

The 36 scheduled trials recorded 48 provider attempts and zero
GitHub write attempts. The report gives each model/tree/arm result and
the first stopping stage for every trial. PR #951 remains in draft and
production activation stays disabled.

Preregistration · Simulated controls ·
Trial ledger · Scores · Report ·
Deterministic CI · Commit checks

#895 deterministic reserved coverage fixture

The reserved pair is ready for a separate diagnostic under the tested
prepared-dependency profile. Both pinned ha-ui trees passed typecheck
and 1,155 Vitest tests. The original report contains 52 runtime modules
and passes inventory. The mutant preserves all 52 entries and adds one
CSF file through the real V8 pipeline. The unchanged inventory rejects
it. The evaluator attributes the aggregate branch change to that file.

A simulated verifier accepted all ten dependency fragments on both
trees through lockfile-authenticated delivery and rejected wrong-version
and cross-revision receipts. This iteration made zero model calls.
PR #951 stays in draft and production activation remains disabled.

Report and reproduction · Subject lock ·
Mutation · Evaluator record ·
Dependency delivery · Future pilot protocol ·
Deterministic CI · Commit checks

#892 Sol retrieval with adapter limits disclosed

The first prompt adds only the common Git adapter's effective 80-line
and 4,400-byte limits. The same pinned trees, initial packet, schemas,
retrieval rules and scoring apply. The historical run admitted 0/18
requests. The replay admitted 18/18 under the same rule.

Arm Tree and expected verdict Admitted Link recovered Link delivered Link cited Supported correct
fixed N1 · refuted 3/3 0/3 0/3 0/3 0/3
fixed N2 · confirmed 3/3 0/3 0/3 0/3 0/3
repeat N1 · refuted 3/3 0/3 0/3 0/3 0/3
repeat N2 · confirmed 3/3 0/3 0/3 0/3 0/3
on-demand N1 · refuted 3/3 3/3 3/3 3/3 3/3
on-demand N2 · confirmed 3/3 3/3 3/3 3/3 3/3

The 18 dry-run slots recorded 30 provider attempts and zero GitHub
write attempts. Sol passed the known-case control. The report recommends a separate reserved case. Production activation remains disabled and
PR #951 remains a draft.

Preregistration · Deterministic controls ·
Trial ledger · Causal audit ·
Report and historical comparison · Deterministic CI · Commit checks

#892 Sol on-demand viewport evidence

Sol starts with the same three receipts in every arm. The viewport
selector and listener are withheld. An admitted on-demand request runs
against the pinned tree through the common Git adapter. The second-call
control receives status without new source.

Arm Tree and expected verdict Requested Link recovered Link delivered Link cited Supported correct
fixed N1 · refuted 3/3 0/3 0/3 0/3 0/3
fixed N2 · confirmed 3/3 0/3 0/3 0/3 0/3
repeat N1 · refuted 3/3 0/3 0/3 0/3 0/3
repeat N2 · confirmed 3/3 0/3 0/3 0/3 0/3
on-demand N1 · refuted 3/3 0/3 0/3 0/3 0/3
on-demand N2 · confirmed 3/3 0/3 0/3 0/3 0/3

The 18 dry-run slots recorded 30 provider attempts and zero
GitHub write attempts. Sol did not pass this known-case retrieval control. All requests named the correct file but requested lines 1–88. The 80-line admission limit blocked every read. The initial prompt described a bounded read without stating the numeric limit. Production activation remains
disabled and PR #951 remains a draft.

Preregistration · Deterministic controls ·
Trial ledger · Causal audit ·
Report · Deterministic CI · Commit checks

#892 Sol verdict control with viewport identity

The final-only verifier receives the same seeded candidate, anchor and
schema as the previous Sol control. Only S1 expands from lines 26–32
to 21–32. Both receipts were verified against their pinned Git trees.
The updated evaluator requires cited viewport identity and checks the
causal explanation for the new mousedown and selection decision.

Tree and expected verdict Historical Sol Sol with expanded S1 Valid responses
N1 · refuted 2/3 3/3 3/3
N2 · confirmed 3/3 3/3 3/3

6 one-call dry-run attempts and zero GitHub write attempts.
Sol passed the prepared-packet control. The report specifies a separate test for independent discovery of the viewport link. The historical results use the previous range rule.
The PR remains in draft and production activation stays disabled.

Preregistration · Deterministic controls ·
Trial ledger · Causal audit ·
Report · Deterministic CI · Commit checks

#892 final verdict with identical complete evidence

The historical matched-source ledger groups its four rejections. The exact
failed checks are indeterminate. The new final-only schema binds candidate,
revision and anchor in the runner and offers four receipt aliases as the
only citation choices. The preflight checked source identity against the
historical ledger and verified byte-identical inputs within each tree.

Verifier Tree Valid Correct Contract rejects Valid inconclusive Unsupported conclusive Judgment errors
GPT-6 Luna N1 3/3 0/3 0 0 3 0
GPT-6 Luna N2 3/3 0/3 0 0 3 0
GPT-6.1 Sol N1 3/3 2/3 0 1 0 0
GPT-6.1 Sol N2 3/3 3/3 0 0 0 0

12 single-call dry-run attempts and zero GitHub write attempts.
Neither profile passed this control. Luna omitted one or more required link citations on every
trial. Sol's inconclusive N1 trial identified viewport selection, which the
historical packet did not show directly. All attempts retained the anchor
and had measured usage within the budget. This result covers judgment on
prepared evidence for the known seeded candidate. Automatic recovery and
detector handoff are separate.
The PR remains a draft and production activation stays disabled.

Preregistration · Frozen manifest ·
Deterministic controls · Trial ledger ·
Report · Deterministic CI · Commit checks

Luna detector with paired #892 verifiers

One Luna detection call per pinned tree produced the handoff before either
verifier ran. A missing hypothesis uses a labeled seeded replay. Both
verifiers receive the same hypothesis, anchor and initial evidence. The
experimental adaptive arm permits two typed reads under one shared budget.
The matched-source control gives both models identical complete source.

Mode Verifier Revision Handoff Supported correct All three links cited
adaptive GPT-6 Luna N1 seeded 0/2 0/2
adaptive GPT-6 Luna N2 seeded 0/2 0/2
adaptive GPT-6.1 Sol N1 seeded 0/2 0/2
adaptive GPT-6.1 Sol N2 seeded 0/2 0/2
matched GPT-6 Luna N1 seeded 0/1 0/1
matched GPT-6 Luna N2 seeded 0/1 0/1
matched GPT-6.1 Sol N1 seeded 0/1 0/1
matched GPT-6.1 Sol N2 seeded 0/1 0/1

The report distinguishes whole-flow performance from judging the same
sources, and records failed and unmetered attempts. #892 is a known-case
diagnostic. The PR stays in draft and production activation remains
disabled.
Preregistration · Runner controls ·
Detection ledger · Frozen candidates ·
Trial ledger · Report and recommendation ·
Deterministic CI · Commit checks

Reserved #892 common Git diagnostic

The reserved pair checks whether the first fresh pointer click works after a
drag is released outside the tab bar. The browser control observed one
selection request on the original and zero on the one-line mutation in all
four mode and direction combinations. Each model ran 18 dry-run slots using
the corrected common Git adapter with the same candidate and initial packet
across arms.

Profile Fixed Second call without sources On demand Decisive recovered / delivered / cited Advance
GPT-6 Luna 0/6 0/6 1/6 5/6 / 5/6 / 5/6 no
GPT-6.1 Sol 0/6 0/6 0/6 1/6 / 1/6 / 1/6 no

The counts require the expected verdict, exact anchor and proved budget.
The on-demand gate additionally requires the decisive source in all six
slots and a higher count than both controls. Neither profile met the gate for full-review evaluation.
The PR stays in draft and production activation remains disabled.
Preregistration · Independent control ·
Trial journal · Profile report and failed stages ·
Deterministic CI · Commit checks

GPT-6.1 Sol automatic #913 replay

The two typed requests that produced the prepared #913 reference went
through the automatic runner in a model-free control. It delivered
the form, parser and precision decoder with exact pinned receipts.
The six measured calls used those frozen second prompts with Sol,
low effort and the prepared diagnostic's verdict schema and budgets.

Revision Expected verdict Prepared Automatic
H1 refuted 3/3 3/3
H2 confirmed 3/3 3/3

Counts require the expected verdict, a valid decisive citation, the
exact anchor and demonstrated budget. Sol reproduced both 3/3 results and met the gate to try a newly
reserved pair.
The PR stays in draft and production activation remains disabled.
Preregistration · Frozen packets ·
Six-call journal · Report ·
Deterministic CI · Commit checks

Automatic quantity chain and pagination oracle

The #913 dry-run derives the source chain from the selected anchor and the
request. The TypeScript checker ties the input and submit callbacks to
the imported parser and the typed precision decoder. All ten historical
requests delivered the three revision-bound receipts to a simulated
verifier. The largest complete packet was 12,658 bytes of the 15,500-byte
limit. There were zero model calls and zero GitHub write attempts in the
replay. With the pinned decoder and component callbacks in a Lit shim,
H1 accepted both six-place inputs and H2 rejected both.
Implementation ·
Replay ·
Consumer observations

The revised #880 oracle records page replacement on P1 and P2. The
label-only mutation suggests accumulation and conflicts with the pinned
navigation label test. The accumulation finding is inconclusive on P2
because an independent requirement to retain prior rows is missing. The
original remains refuted. Historical pilots retain their frozen scores.
Oracle and reproduction ·
Deterministic checks ·
Commit checks

Production activation remains disabled.

Prepared evidence second-call diagnostic

The four historical second-call prompts match their recorded SHA-256
digests. The preregistered dry-run replayed them and the prepared-chain
packets in fresh closed sessions with the same candidate, anchor and
response schema per revision. The 48 slots include 45 valid responses,
one citation integrity failure and two unmetered timeouts. The GitHub
write ports recorded zero attempts.
Preregistered protocol ·
Frozen packets and prompt hashes ·
Trial journal ·
Comparative report

Usable verdicts require an exact anchor, the decisive receipt and proved
budget. Counts are out of three, historical → prepared.

Pair Model Original refutations Mutant confirmations
#913 GPT-6 Luna 0 → 0 1 → 3
#913 GPT-6.1 Sol 2 → 3 0 → 3
#880 GPT-6 Luna 1 → 0 3 → 3
#880 GPT-6.1 Sol 3 → 3 0 → 0

For #913, Sol's three prepared mutant confirmations cite the form path
and the adapter that admits precision six. Luna improved mutant
confirmation and falsely confirmed all three prepared originals. For
#880, Sol cited the button-to-controller path but still left the
mutant inconclusive. The safe premise codes point to page replacement
and the product promise of accumulation. This was a prepared-source
diagnostic. Automatic retrieval and the separate reserved-pair
advancement gate remain pending. #918 stayed outside the matrix and
its read result remains conditional on a persisted retired schedule
that retains its slot. The PR stays in draft and production activation
remains disabled.
Premise signals and interpretation ·
Separate #918 audit ·
Traceability matrix

All 74 focused tests passed and the code graph was updated. The
commit hook passed build, lint and 18,322 tests.
Deterministic CI for 4fe9a40e8 ·
Final SHA checks

Partial common Git diagnostic v1

The versioned runner uses the corrected common Git adapter and preserves
the frozen pilots. Its simulated verifier received the decisive fragment
on all ten original #913 requests, including the six partial responses.
Both calls respected their complete budgets. The largest second packet
was 12,712 bytes. The mutation ports recorded zero write attempts.
Runner · Ten simulated controls

The frozen dry-run matrix used #913 as the known regression and #880 as
the reserved TypeScript pair. It records all 72 slots, with 68 completed,
four failed, zero unmetered attempts and zero GitHub write attempts.
The closed-session gate rejected one unexpected MCP event on Luna.
The journal preserves the event type and integrity errors.
Preregistration ·
Raw trial journal · Comparative report

Neither profile meets the reserved gate for full-review evaluation.
On #880, Luna recorded zero decisive recoveries and citations, and two
correct on-demand verdicts. Sol recovered and cited two decisive receipts
and returned one correct on-demand verdict. All selected anchors were
exact. For the known #913 regression, Sol recovered and cited all six
decisive receipts but returned only three correct verdicts.
Per-profile advancement ·
Traceability

The separate #918 audit shows that the reviewed fixed-schedule creation
and state-change commands start active and reject retirement as a
destination. The SQLite result is conditional on the seeded persisted row
and retained slot. Auditor-selected fragments demonstrate bounded packet
construction. This evidence stays outside the model matrix.
Separate reachability audit

The 71 focused tests, freeze checks and Graphify update completed.
The deterministic CI Check was dispatched on this head. Automatic
Copilot and Setup Platform Smoke checks remain unrun because of
[skip ci]. This PR remains draft and production activation is disabled.
Deterministic CI Check ·
Checks for 5aed48f77

Bounded Git packing and T2 packet audit

On the pinned #913 tree, adding the broader taskQuantity( lookup made the
frozen adapter lose a valid declaration receipt. The successor bounds
each match window before merging and retains valid receipts when auxiliary
lookups exceed a budget. Integrity failures still withhold the result.
All ten recorded requests return the decisive source, compared with one
before. Six responses retain partial per-query outcomes.
Regression tests ·
Ten request results

The three historical Sol T2 prompts match the recorded hashes. The missing
source links were the concrete slot read, persisted state decoding and the
returned action branch. A separate source-only simulation with
auditor-selected ranges delivered those links in all three complete
packets. The largest used 15,329 of 15,500 bytes. The packet proves the
read path from a valid persisted retired row that still owns its slot.
The public transition to that state and automated discovery of the added
ranges remain unproved. SQLite execution stays with the evaluator.
Audit and conclusions ·
Historical and revised T2 packets ·
Evaluator-only observations

All 12 original #918 requests kept the chain, anchor and valid receipts
within the complete packet limits. The simulated runner recorded zero
write attempts. The 71 focused tests and Graphify update completed.
The frozen pilot checksums still verify. This iteration made zero model
calls. The report defines the checks for a new reserved non-Python pair
and leaves the run decision pending review.
12 complete packet measurements ·
Simulated runner

The deterministic CI Check was dispatched for this head. Automatic
Copilot checks and Setup Platform Smoke remain unrun because of
[skip ci]. This PR remains draft and production activation remains
disabled.
Deterministic CI Check ·
Checks for e99e87efb

Module bindings and the reserved TypeScript diagnostic

At 580811b61, Python constructed AlphaDatabase after
BetaDatabase = AlphaDatabase but the resolver supplied BetaRepo.active_key.
The corrected prototype follows the direct alias and cites the binding.
A dynamic reassignment is unresolved and the simulated runner supplies
no effective implementation receipt. All 12 original #918 requests
retained the SQLite chain, anchor and valid receipts within budget.
Regression ·
12 deterministic controls ·
Simulated runner controls

The preregistration was committed before measurement. The dry run used
#918 as a known regression and #913 as the reserved TypeScript pair,
with common Git retrieval only for #913. All 72 planned slots have recorded
outcomes. There were 62 completed, eight expired and two failed slots,
37 unmetered attempts and zero write attempts.
Preregistered commit ·
Protocol · Raw journal ·
Report

Neither profile meets the threshold for full-review evaluation. For the
six on-demand #913 slots per profile, Luna retrieved decisive evidence once
and Sol did not retrieve it. Neither profile cited it or returned a correct
verdict. The selected anchors were exact in all six slots for each model.
The common adapter hit its fragment byte limit in nine of the 12 slots.
Per-profile scoring and limits ·
27-issue traceability

All 66 focused tests, specification validation and Graphify completed.
The deterministic CI Check was dispatched for the exact head. The automatic
Commit, Pull Request and Branch Sync workflows, Copilot / Verification
and Setup Platform Smoke are unrun for this head because of [skip ci].
This PR remains draft and production activation remains disabled.
Deterministic CI Check ·
Checks for d10251508

Effective entry origins and language-independent retrieval

The pinned regression at 2d1d92c22 ignored the reachable
wire(get_alpha()) branch and reported BetaRepo.active_key.
The corrected prototype traces the actual value at the argument use.
It reports unresolved when any relevant origin is unproved and
ambiguous for competing proved origins. A unique proved origin can
differ from the Python annotation.
Executable regression ·
Baseline observation ·
Resolver tests

All 12 original #918 requests retained a sufficient chain to the SQLite
implementation in their original order, with the anchor and receipt
bindings intact. Each replay checked four entry origins. SQLite returned
zero actions on T1 and seven on T2 for both date-only and timed schedules.
The successor runner delivered the causal packets, exercised common Git
retrieval and exposed unresolved or ambiguous cases without an effective
implementation receipt. The runner exercised all rejection paths with
zero write attempts and zero real model calls.
12 replay records ·
SQLite controls ·
Runner tests ·
Simulated exchanges ·
Report and reproduction

The owning SDD separates fragment provenance from effective-route
attribution and requires a reserved non-Python pair in the next diagnostic.
EV-07 compares both models and qualifies each proposed profile independently
with explicit maintainer approval before enablement. The full-review and
174-case corpus thresholds remain in force. The traceability matrix
separates the SQLite control, replay, simulated delivery and pending model
evaluation.
SDD amendment ·
27-issue traceability

All 63 focused tests, specification validation, the frozen preflight and
Graphify completed. The deterministic CI Check was dispatched for the exact
head. The automatic Commit, Pull Request and Branch Sync workflows,
Copilot / Verification and Setup Platform Smoke are unrun for this head
because of [skip ci]. The PR remains draft and production activation
remains disabled.
Deterministic CI Check ·
Checks for 580811b61

Call-site attribution and simulated causal runner

The resolver at 232b8eb35 attributed a later assignment to Inspection(store)
and selected BetaRepo.active_key. The corrected prototype evaluates the
assignment at the argument use. The late assignment is unresolved, a
dominating prior assignment resolves, and competing branches are ambiguous.
Executable baseline reproduction ·
Baseline observation ·
Resolver tests

The successor runner exercised both calls with a simulated verifier. It
delivered the same initial evidence plus five bound causal receipts for
each tree, preserved the anchor and kept the combined packet within
budget. It rejected a second request, a foreign receipt and an oversized
packet. Legacy typed retrieval remains available. The tests recorded zero
write attempts and zero real model calls.
Runner integration test ·
Packets delivered to the simulated transport

The separate replay attested sufficient source chains for all 12 original
#918 requests. SQLite returned zero actions on T1 and seven on T2 for both
date-only and timed schedules. The report includes the reproduction commands.
Report ·
Replay journal ·
SQLite controls

All 56 focused tests passed locally, and the frozen runner preflight and
Graphify update completed. The commit uses [skip ci] to avoid automatic
model workflows. Automatic PR and commit checks including
Copilot / Verification and Setup Platform Smoke remain unrun for this
head. The deterministic CI Check was dispatched for the exact SHA.
This PR remains draft and production activation remains disabled.
Deterministic CI Check ·
Checks for 2d1d92c22

Deterministic causal replay for #918

All 12 original requests returned search-file-budget at f1b782117.
All 12 returned a sufficient source chain through the effective SQLite adapter
after the correction. The exact request, anchor, operation and pinned Git tree
were retained. The largest request opened 37 of 80 files and completed within
the four-second limit. The persisted SQLite control refuted the retired-schedule
premise on T1 and reached the fixed action branch on T2 for both date-only and
timed cases. The replay and tests invoked zero models.

Report and reproducible commands ·
Before · After with bound receipts ·
Focused tests · SQLite controls

The push omitted automatic model workflows. The deterministic CI workflow was
dispatched on the same head. This pull request stays draft and production
activation stays disabled.
Manual CI Check ·
Checks at 232b8eb35

Bounded tree retrieval diagnostic

The prototype now requires membership in the same contiguous diff block
before relocating a candidate. The closed verifier cannot change the frozen
anchor. One typed request can resolve an exact filename or literal within a
bounded pinned Git scope and return complementary receipt-bound fragments.
The offline #869 and #875 controls passed before any measured call.

All 72 replay slots and all provider attempts are retained. The held-out
#918 decision separates decisive retrieval, citation and correct metered
verdicts across the three arms.

Model On-demand retrieved / cited Fixed / repeat / on-demand correct and metered Advance
gpt-6-luna 0/6 · 0/6 0/6 · 0/6 · 0/6 no
gpt-6.1-sol 0/6 · 0/6 0/6 · 0/6 · 0/6 no

The PR is draft and production activation is disabled.
Design freeze ·
Preregistration commit ·
Protocol ·
Deterministic tests ·
Runner parity test ·
Attempt journal ·
Report ·
Reproduction ·
Checks at f1b782117

On-demand evidence diagnostic

The application owns one typed follow-up and validates the final citations before
writes. It normalizes eligible anchors on the pinned diff before freezing the
candidate. The owning SDD defines independent qualification and explicit
maintainer approval for each profile. Intact mixed reviews can publish
confirmed findings while incomplete verification blocks clean approval.
Owning SDD ·
Reconciliation SDD ·
Framed review SDD

The frozen replay completed 72 slots and 100 provider calls. Two slots failed
response validation. Both models returned zero decisive citations for the
held-out #869 pair. Luna scored 1/6 on demand, 2/6 on the repeat control and 1/6
on the fixed packet. Sol scored 0/6 on all three. The measured time and token
ceilings held. Neither profile earned progression to full-review evaluation.
Production remains disabled and this pull request is draft.
Preregistration commit ·
Protocol · Manifest ·
Attempt journal · Report ·
Reproduction ·
27-issue matrix

The specification and catalog validators, focused pilot and approval suites,
local control probes and Graphify update passed.
Copilot / Verification ·
CI Check ·
Platform smoke ·
Checks at e477e0e4b

Expanded v4 pilot

Activation remains disabled. The 504 preregistered slots include 475 completions, 4 failures and 25 aborts. Replay resumed from 207 saved slots with the same frozen inputs. The independent Sol source supplement completed all 18 slots.

Full review Luna Sol
Historical alerts reproduced by the current baseline 1 0
Source/CI positives at the exact anchor 17/21 12/21
Exact anchors in the nine original positive exposures 9/9 9/9
Source/CI verifier gross input p95 with known usage 10,271 10,104

The baseline missed the preregistered reproduction threshold in full review. Incomplete comparisons, missed positive exposures and unmetered calls also prevent qualification. The separate 174-case detection corpus remains a requirement. The replay recovered 25 Luna Source/CI slots after verifier timeouts, and full review recovered three. In the Sol supplement, the added exact-head source excerpts changed nine inconclusive mutant findings to confirmed and retained all nine refutations on the originals.

Final report and ledgers · Sol source supplement · Preregistered protocol · Current checks

Local checks passed for the 12 verifier-policy and adapter tests, 42 dry-run preflights, 18 predecessor prompt hashes, 134 evaluator tests, and the specification, documentation, workflow, build synchronization and evidence checksums.

The repository build and lint passed. The first test run passed 919 suites. The two suites that require loopback passed when rerun with loopback access.

Bounded v3 retest

Activation remains gated. The 216 preregistered slots include 161 completions, 12 failures and 43 aborts. The experimental arms now share a bounded diff projection and closed citation schema. The source/CI arm also supplies the unchanged pytest caller and the exact-head restoration test. The current arm keeps the production detector and literal verifier.

Model End-to-end positives retained v2 → v3 Exact v3 Verifier gross input p95 v2 → v3
Luna 1 → 3 of 9 3 of 9 213,032 → 9,412
Sol 1 → 6 of 9 5 of 9 434,624 → 9,350

The evidence arm detected all nine positive exposures per model. Luna's six lost positives expired in the verifier before a completed provider turn. Sol left three inconclusive, and one confirmed pytest finding used a neighboring line. The observed overhead is lower, but incomplete attempts, positive preservation, exact anchors and baseline reproduction still prevent qualification. The unchanged corpus and production activation remain gated.

Preregistration commit · Protocol · Manifest · Results, ledgers and reproduction · 27-issue matrix

Verification at 864bbbac95d632818554303f78a2d47467912589

  • Six deterministic verifier-policy tests, 18 controlled dry-run preflights and 134 evaluator tests passed. Specification, documentation, workflow, build synchronization, lint, checksums and Graphify validations completed.
  • The Action-published owning SDD and the production activation policy are preserved. GitHub verification and CI passed on the preregistration commit.

Passed Copilot / Verification · Passed CI Check · Current platform checks · Passed preregistration verification

Earlier v2 pilot outcome

Do not advance to activation. Recorded 216 primary attempts and 144 more across the #859 and #861 expansions. The primary matrix retained 50 failures and one abort. End-to-end, the source-and-CI arm retained only one of nine positive controls per model, exceeded the input-token budget and still lacked sufficient historical baseline reproduction after expansion. Production policy binding and receipt emission remain gated.

Pilot protocol · Manifest · Comparison and reproduction · Measured ledger · 27-issue matrix

Verification at 5bf303fcdf854c6aa0bcd0457316a4656be56c82

  • Local specification, documentation, workflow contract, typecheck and targeted lint passed.
  • The full local test suite passed 18,322 tests across 921 suites, with 35 skipped. The 203 focused approval tests, 134 Bugbot evaluator tests and 30 controlled runner preflights passed separately.
  • The CLI and GitHub Action bundles were regenerated with the locked dependencies and pass validate:build.
  • The original scope failure was resolved. The official run at 0b984e8 verified the SDD receipt and completed the progress assessment. The next official run also completed the assessment at 1c85dd51. At the final head, the official check succeeded after reconciling the same SDD receipt and completing all four assessment batches.

Passed Copilot / Verification check · Official assessment logs · Passed CI Check · Final Setup Platform Smoke · Verified pilot commit · Final PR checks · Verified scope recovery

Loading
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

60% Progress: 60% size: XXL Indicates an extremely large task or issue that requires significant time and effort to complete. state:verified

Projects

Status: Done

Development

Successfully merging this pull request may close these issues.

✨🧑‍💻 - Verify Bugbot findings against repository evidence before publication

3 participants