Skip to content

✨🧑‍💻 - Verify Bugbot findings against repository evidence before publication #950

Description

@efraespada

Is there an existing issue or feature request for this?

  • I have searched the existing issues and feature requests.

What area does this improvement affect?

New Feature

What actions does this improvement affect?

Pull Request, Commit

Description of the idea or improvement

Add a language- and framework-independent verification stage between Bugbot candidate detection and publication. The diff defines which changes are under review. The pinned repository revision and relevant project evidence establish whether a change introduces or exposes a reachable defect. The product outcome is a PR review in which every published or reopened defect has a traceable basis, and a reviewer can distinguish a refuted candidate, an unresolved question, and an interrupted run.

Scope and ownership

  • Apply the rule to every eligible finding that would become active, including new findings, updated active findings, reopened findings, and findings counted as overflow. Cover ordinary and framed PR reviews, incremental review and recheck, and the supported Commit review path with or without an owning PR. Preserve the existing canonical PR ownership and fallback rules so push and PR jobs cannot both publish the same review. Apply explicit evidence requirements to any proposed fixed or obsolete transition on an existing finding. Preserve human dismissal authority.
  • Extend the existing Bugbot analysis and publication capability. Generalize the existing candidate verification boundary and reuse the current context, source identity, partitioning, finding identity, history, freshness, approval and presentation contracts. A file extension may select a source adapter or an evidence method. Eligibility for verification follows the finding, regardless of language, framework, severity rank or whether a comment slot is available.

End-to-end flow

Pin the review target and source → detect candidate hypotheses in the diff
→ merge equivalent candidates and plan evidence → retrieve project evidence
→ independently try to refute each candidate → classify every eligible candidate
→ recheck freshness → publish and reconcile one result for that revision
  1. Bind the review. Carry the existing repository, target, event, reviewed head and applicable base or comparison identity through detection, evidence reads, decisions and publication. Use base evidence when required to establish whether the change introduced the defect or made it reachable. A broad checkout, a branch name, or a similarly named CI check is insufficient provenance. For PR merge checkouts or CI builds using a synthetic merge, require verified source or tree equivalence before treating the result as evidence for the relevant revision and path.
  2. Prepare a testable candidate. Preserve its stable identity and changed-code anchor. Capture the change, triggering input or state, reachable path and user-visible or security impact. Apply existing admission rules and deduplicate equivalent hypotheses before verification. Retain relevant evidence across duplicates and distinguish different root causes at the same location. Apply ranking and the visible comment limit after verification. Every retained overflow finding that affects review status must also be confirmed. A candidate rejected by existing scope or shape rules does not consume a verifier slot. Malformed or missing required batch results remain workflow failures under the existing integrity rules.
  3. Acquire evidence according to the claim. The application owns bounded, read-only evidence acquisition and supplies the result to the verifier. The model may ask for evidence through a closed, typed request within the bound review target. Support immutable source reads and bounded searches across changed and unchanged files, callers, guards, configuration, tests and contracts. Use the actual import and resolved dependency or runtime version for framework, language and API claims. Use matching native CI job, suite or step evidence for claims about a build or test, and distinguish a matching execution from a test that was skipped, cached, run against another revision, or irrelevant. A green check in general carries no verdict about a specific claim. Where a new reproduction is needed and supported, run only a preauthorized, trusted, bounded verification command or probe with pinned inputs in an isolated environment. Existing source and CI evidence should work on repositories without an execution profile. Unavailable or denied evidence yields a bounded diagnostic and an honest disposition.
  4. Try to falsify the claim. Use a fresh verification task, independent of the detection conversation, with the concrete hypothesis, its source binding and evidence actually fetched by the application. Examine earlier guards, alternate paths, matching executed counterevidence, effective configuration and contract exceptions. Keep a defect when the evidence establishes the causal path and its impact. Verify the proposed replacement separately before offering an executable suggestion. Keep a confirmed defect with a prose explanation if the proposed replacement cannot be verified. Allow a short bounded additional evidence round for a material gap or conflict. The SDK verifies provenance and response integrity. The verifier evaluates whether the evidence supports or contradicts the claim. A valid link or a confidence score alone cannot establish the verdict.
  5. Validate an explicit disposition for every eligible ID. Use confirmed, refuted or inconclusive. Confirmed requires evidence of the reachable defect and a valid anchor. Refuted requires evidence that contradicts a material premise, such as an effective guard, the correct versioned API behavior, or a matching execution that directly falsifies the predicted failure. Inconclusive is a plausible, eligible hypothesis whose decisive evidence is missing, unavailable, conflicting or outside the allowed evidence budget. It carries a reason and a practical next step. The application checks that IDs, scope, artifact identities and source revisions match the request, that referenced evidence was actually supplied, and that each required disposition occurs once. Treat forged, missing, duplicate or cross-run verdicts as an integrity failure. A verifier cannot introduce a new finding or change the historical identity to make its claim fit.
  6. Publish once after all required work is accounted for. With an integral and current result, publish or reopen only confirmed findings. Discard refuted new candidates and summarize inconclusive ones without opening defect threads for them. If two defects are confirmed and one candidate is inconclusive, publish the two confirmed findings, update the canonical summary once and mark the review incomplete for automatic approval. Track diff coverage, candidate verification and provider publication separately.
    Automatic clean approval requires complete required coverage and verification, no actionable or unresolved finding state, and all existing approval gates. Keep existing branch protection and repository ownership rules. A workflow-level evidence integrity failure, missing required partition, supersession or cancellation before publication produces no new findings or resolution mutations. Keep the last verified result identified by its own revision and surface the failure or supersession through the existing run/check behavior. A publication failure after a provider write follows existing recovery and re-read semantics, records observed effects and never claims a clean result from intended writes.
  7. Reconcile history and retries. Verify a proposed reopen, fixed or obsolete transition against the same pinned project evidence and the explicit retained historical ID. A contradicted historical premise may qualify as obsolete only through the established transition policy with explicit evidence. An unresolved question or omission preserves the existing thread state and any verification requirement. Honor verified human decisions and native thread facts. Recheck the same head after recoverable evidence becomes available and update existing comments without duplicate writes. A new commit supersedes in-flight work. Preserve a single publication owner and the existing race fences around irreversible effects.

Evidence, security and operating limits

  • Use evidence records issued by the application and bound to the review and candidate. Include source kind and safe locator, such as repository plus commit, path and line range, a resolved dependency version and contract source, or a CI producer/job and reviewed tree. Record enough provenance to detect evidence swapping and support operator review. The published view uses concise safe links and a causal explanation. Telemetry may use counts, safe reason categories, timing and correlation identifiers. Keep raw prompts, command output, credentials and unnecessary repository content out of public and durable records.
  • Resolve only authorized repositories, Git objects, paths, versioned dependency sources and CI producers. Treat repository content, logs and model output as untrusted data. Enforce traversal, symlink, link and credential boundaries and validate evidence before any provider write. Execution is opt-in through an existing or explicitly reviewed trusted repository capability with a disposable, isolated runtime and bounded output, time and access. Model text and PR content cannot authorize commands, new network destinations or credentials.
  • Put per-candidate and aggregate limits on source reads, retained evidence, requests, tokens, time, concurrency and retry. Reuse existing frame and traffic accounting for ordinary and framed reviews and verify limit accounting at the actual adapter boundary. Deduplicate safe identical reads within the same pinned scope. Distinguish a candidate-level evidence limit from an invalid or interrupted review. Report the exhausted resource and next supported action. Set numerical limits and any rollout setting from pilot evidence in the owning SDD. Snapshot effective settings per review. Evidence provenance and approval integrity are required in every enabled mode.

What GitHub should show

  • Pending. “Changes analyzed. Checking 3 candidate findings against this commit.” Link to the current run where available.
  • Complete. “1 confirmed defect. 2 candidates refuted.” Include the confirmed comment with a source link and a short explanation of the reachable impact.
  • Incomplete. “2 defects confirmed and published. 1 candidate still needs the test result for this commit.” Link to the affected job or give the supported next action. Show why automatic approval is unavailable.
  • Failed or superseded. Identify the affected revision, the failure or newer revision, and the last verified review where available. Report whether any GitHub publication effects were actually observed.

Reuse the canonical status card, Copilot / Review, approval inputs and Job Summary. Keep candidate dispositions separate from durable historical finding states. Update status once per result using the existing locale and sanitization contracts. Keep operator diagnostics expandable and avoid publishing a separate comment for each rejected or inconclusive hypothesis.

Current limitations or challenges

The audit used Copilot v3.10.13 as the published source baseline (ace49e4). Re-establish the baseline at the selected development revision before implementation. Bugbot currently plans review work from the diff, passes surrounding project guidance and lets its reviewer inspect the checkout. Partition attestations and coverage establish which inputs the run processed. They do not establish which supporting files, dependency contracts or executions the reviewer consulted for a particular claim.

The existing generated-source verification is enabled mainly for partitions containing Python. It reuses the detection context and the configured reviewer model. Its response checks enforce identity, a complete decision set and nonempty evidence text for retained candidates. The application does not currently bind that evidence text to sources it retrieved for the candidate. In #914, the native job reported complete context, retained two candidates in that verification step and published both. The installed pytest fixture discovery excluded the defaulted arguments on which both findings depended. In #919, the exact reviewed source checks retained SQLite history and stops before the claimed failure. #898 documents the same wrong Unicode premise in ha-ui and Domus, with executable counterexamples.

Relevant implementation and contracts: candidate routing, existing verifier, verification response policy, publication workflow, and quality tooling. The audited historical reviews used GPT-6 Luna. The causal contribution of model choice remains an empirical question.

Expected impact

Reviewers get fewer misleading comments, evidence they can inspect for each confirmed defect, and a clear next action when Bugbot cannot establish a verdict. Repository maintainers can keep reliable merge gates while distinguishing an incomplete review from both a clean review and a workflow failure. The feature applies consistently across languages and frameworks.

Pilot before enabling publication

  1. Freeze at least three historical case pairs with pinned source, relevant surrounding files and versioned or executed evidence. Use 🐛 - Bugbot reports valid supplementary Unicode as rejected by a Unicode aware regular expression #898 for Unicode-aware matching, 🐛 - Bugbot reports defaulted pytest arguments as missing fixtures #914 for fixture discovery with defaulted arguments, and 🐛 - Bugbot misses restored fixed replacement source guard #919 for the earlier SQLite guard. For each historical false finding, author a genuine corresponding defect with a predeclared expected condition and anchor. Examples include a validator that mishandles supplementary characters or accepts malformed UTF-16, a genuinely required unresolved fixture, and a reachable restore/replacement failure when the earlier guard is removed. Keep the expected verdict and scoring metadata outside the model input. Keep evidence from the original reported revision separate from later reproductions.
  2. Compare three variants using GPT-6 Luna and GPT-6.1 Sol. Use the existing flow as the baseline, a general second pass with the detection context as the extra-inference control, and a general second pass with application-retrieved project evidence as the proposed flow. Keep the source, task scope, effort, tool permissions and evidence conditions comparable within each model. Record exact SDK, provider, model, effort, runner and fixture identities. Record any unavailable model or unavoidable configuration difference explicitly.
  3. First replay the same frozen eligible candidates through the verification variants to isolate their effect. Then run the full detection-to-result path to measure missed real defects and final output. Predeclare three repetitions per matrix cell and retain failures, aborts, divergences and inconclusive results. Measure unsupported active findings, confirmed real defects, missed defects, inconclusive rate, evidence requests, latency and provider usage or cost where available. Report per-case and per-model results and aggregate counts. Use an isolated dry run with zero GitHub review, status-card, check, label, approval or finding-state writes. Reuse the existing evaluator where it applies and add a repository-shaped runner for cross-file evidence. A diff-only benchmark cannot qualify the new behavior by itself.
  4. The qualification must demonstrate fewer unsupported published findings in a reproducible paired comparison and preserve the three genuine defect controls and the existing applicable recall floors. Report false positives and recall together. Set explicit evidence, latency and usage budgets before production enablement and report observed overhead against those budgets. If the baseline does not reproduce enough false findings to demonstrate a reduction, report that limitation, expand the frozen cohort under the same scoring rules and avoid claiming an improvement from that cohort. A historical replay, a deterministic test and a real model trial are separate evidence.

Acceptance criteria

  • A non-Python eligible finding and a Python eligible finding take the same verification path. Evidence from unchanged files can refute either. A positive control that removes the decisive guard or violates the actual versioned contract remains actionable and correctly anchored.
  • Every eligible new, updated, reopened and overflow candidate has exactly one verdict bound to the current review and to evidence actually supplied by the application. Confidence, a bare link and an unexecuted test claim cannot substitute for that evidence. A failed replacement check preserves the confirmed finding as prose and prevents an unverified executable suggestion.
  • Evidence from a different commit, unrelated CI job, wrong resolved dependency, unsafe path, forged locator, stale or replayed receipt, or unauthorized execution cannot support publication. Missing material evidence yields an inconclusive result with a named next step. A required missing or malformed verifier result fails the run before new finding mutations.
  • A completed run containing confirmed, refuted and inconclusive candidates publishes the confirmed findings in one coordinated result, omits refuted and inconclusive inline defects, and shows the separate verification status. Inconclusive review evidence prevents an automatic clean approval. With only refuted candidates, an otherwise complete clean review is possible under the existing approval rules. Provider publication failures report observed effects and remain recoverable.
  • The canonical PR, framed and ordinary paths, the supported Commit fallback and an exact-head recheck share the verification rule. Push and PR paths cannot duplicate publication. Cancellation, supersession, invalid integrity evidence and aggregate budget failure before publication preserve the last verified result and historical findings. A fresh retry for the same revision is idempotent and a newer revision wins.
  • A historical finding changes to fixed, obsolete or reopened only with explicit verified evidence, its retained identity and the existing native thread rules. Omission, uncertainty and verifier failure preserve state. A protected human dismissal remains effective. Pending, complete, incomplete, failed and superseded states are legible in the existing GitHub surfaces and localization path.
  • The pilot has a reproducible manifest, paired controls, frozen run protocol, raw outcome counts and a reviewable decision on verifier configuration and rollout limits. The SDD maps each acceptance requirement to automated evidence. Add at least 30 distinct deterministic cases, separate from model trials, allocated as 8 provenance and source cases, 6 verdict and publication cases, 6 lifecycle and concurrency cases, 5 permission and security cases, and 5 presentation and compatibility cases. Preserve applicable repository coverage and CI gates. Exercise the production orchestration with a controlled GitHub provider and failure injection, in addition to pure policy tests.
  • Update the owning SDDs and catalog evidence, Bugbot detection and finding-publication docs, operator recovery and quality documentation. Cover evidence prerequisites, rollout, limits, historical compatibility, and rollback using the approved runtime route. Validate the specification and generated catalog as applicable, run the affected test and CI suites, and provide the final-head qualification evidence in the PR.

Alternatives considered

The pilot treats a model change and an extra call on the same context as controls. Repository-wide input expansion could improve discovery, but it does not identify the material evidence supporting a particular finding and makes review cost harder to bound. Language-specific evidence adapters remain useful underneath a common verification contract. Publishing all candidates as actionable findings would expose unresolved hypotheses to users and approval consumers. Holding every confirmed finding until all hypotheses are settled would delay useful feedback when the run is otherwise complete. The proposed product flow publishes only confirmed findings and makes uncertainty visible in the shared review result.

Version of copilot

Audited baseline v3.10.13. Feature applies to the supported Bugbot review workflow.

Additional context or comments

Affected reports and regression map

These are the 27 finding-quality reports identified in the audit. They are linked as concrete inputs to this feature. Record for each report the decisive evidence, expected disposition, and the regression or remaining limitation that covers it. Keep the original reports available for case-specific validation and handle their closure based on evidence for each report.

Repository behavior, guards, and project context (14)

  • #859 — Follow the imported schema to the existing unique index before claiming a composite foreign key is invalid.
  • #861 — Resolve the migration guide in the reviewed Git tree before reporting a missing link target.
  • #872 — Follow the UI state transition and distinguish a rendered confirmation value from an editable input.
  • #879 — Trace inherited permissions and distinguish the incorrect default-role premise from any separately supported coupling defect.
  • #880 — Read the actual pagination contract and UI wording before interpreting a page replacement as a load-more defect.
  • #891 — Trace the measured padding and native click target rather than assuming the click lands on the input.
  • #892 — Trace the full event sequence, including the next mousedown that clears suppression.
  • #893 — Follow Angular's actual native-reset directive and control-value accessor ordering.
  • #896 — Follow the shared listener options to the capture flag. Preserve any separately evidenced shadow-boundary defect.
  • #897 — Inspect the rendered light DOM and the native submit-button proxy.
  • #902 — Trace the stale-trigger guard and the rendered state at each click.
  • #917 — Check the atomic snapshot restore invariant and distinguish retained history from current effects.
  • #918 — Follow the current-schedule query and its prefilter for retired schedules before evaluating offered actions.
  • #919 — Follow the earlier replacement-source guard and the SQLite query over retained fixed exceptions.

Language, browser, and toolchain behavior (13)

  • #867 — Check the resolved Playwright matcher contract for the supported enabled option.
  • #869 — Trace the proxied readiness route and the pinned Playwright HTTP readiness predicate.
  • #873 — Check the rendered Markdown before treating fenced sample links as live navigation.
  • #875 — Inspect the actual JSON value types and exact-integer guard.
  • #888 — Distinguish native requestSubmit and reset from click activation and preserve the genuine duplicate-click case.
  • #890 — Check native implicit submission and the role of disabled text inputs in the blocker count.
  • #894 — Use the pinned Vitest Browser matcher semantics for aria-disabled.
  • #895 — Inspect resolved Vitest and Storybook collection and coverage settings.
  • #898 — Verify supplementary Unicode matching and retain detection of malformed UTF-16.
  • #900 — Verify FormData with a null submitter and the form's actual remaining controls.
  • #904 — Check the exact compiler version and relevant TypeScript job before claiming a type failure.
  • #913 — Evaluate the complete fractional pattern and the concrete six-digit witness.
  • #914 — Check fixture discovery in the installed pytest release and compare against a genuinely required missing fixture.

Implementation handoff

Follow the existing feature admission and pre-branch SDD flow. Use the existing Bugbot analysis and publication capability and coordinate with finding-state reconciliation and framed review. Record the decisions for evidence profiles and limits, execution authorization, mixed publication and approval behavior, and rollout in the SDD. Keep pure admission and disposition policies separate from source and execution adapters, evidence orchestration, provider mutations and presentation. Add an executable boundary check so verifier ports and untrusted response handling cannot invoke publication or arbitrary commands. Keep the pilot separate from activation and preserve the approved release and rollout process.

On-demand evidence revision decisions

  • The first verifier response is a final verdict or one typed request. The application reads only the authorized repository and revision within the candidate budget and returns a candidate-bound receipt with the fragment. The follow-up is final and cites its actual evidence. Missing, denied or exhausted decisive evidence remains inconclusive with cause and next step. Invalid citations stop finding, history and approval writes for the entire batch.
  • Before freezing identity, retain a valid substantive anchor. A syntactic separator or non-addressable line may move to a unique eligible causal line within two lines of the same change block. Record the original and selected positions. Ambiguity withholds an inline location. The verifier cannot move the anchor. Deterministic tests include the pytest separator, an eligible line to preserve and two plausible lines.
  • V4 and the Sol supplement are diagnostic and v4 did not qualify. Compare Luna and Sol at low effort in the replay, using at most 90 seconds, 32,768 gross input and 4,096 output tokens per case including both rounds and retries. A favorable held-out replay would allow full-review evaluation. Require all existing historical reproduction, paired improvement, positive control, exact-anchor, integrity, cost, latency and independent 174-case detection gates for each exact profile independently. Activation remains disabled and requires explicit maintainer approval after qualification.

Bugbot verification integration and rollout for #950

1. Executive summary

Bugbot detects candidates, collects applicable project and official evidence,
independently verifies each candidate, and coordinates the result before any
publication. This amendment records the current delivery and rollout decisions.
The existing owner retains the historical cases and earlier pilot evidence.

4. Goals

Requirement Behavior
SDD-I950-ROLLOUT-R1 Bind candidate preparation, evidence, verdict and publication to the same immutable review head and anchor.
SDD-I950-ROLLOUT-R2 Allow independent investigation under a resolved verifier profile, with a broker that validates project, official and CI evidence.
SDD-I950-ROLLOUT-R3 Preserve shadow isolation, qualification and approval fences across policy changes and rollback.

6. Functional behavior and state model

A review prepares its candidates before verification. The normal reviewer
configuration is inherited unless the repository selects a compatible verifier
profile. Open a fresh session for each candidate and supply its hypothesis,
anchor and pinned source. Follow the effective implementation, configuration
and dependency versions. Consult applicable official versioned sources or CI
when decisive. Use a typed, read-only evidence broker with provenance and
explicit missing or ambiguous results. The same investigation may refine an
incomplete citation before closing. Time and token use are telemetry, with
explicit cancellation and provider failures recorded separately.

Each candidate gets a confirmed, refuted or inconclusive verdict. The closure
binds material claims to delivered source ranges and preserves the anchor.
A complete mixed review can publish confirmed findings when enforcement is
active. An inconclusive review does not authorize a clean approval. An
integrity failure, cancel or moved head preserves the prior verified result.
In shadow, record the outcome separately from ordinary publication. The
default remains disabled. Enforcement requires a qualified exact profile and
explicit maintainer approval. Approval uses current review and policy evidence;
a previous receipt cannot authorize a new policy or rollback generation.

8. Clean Architecture

Application use cases own sequencing, source identity, verdict completeness,
publication and approval. Read-only adapters acquire Git, official and CI
evidence. Provider adapters perform investigation and return bounded structured
verdicts. GitHub adapters write only through the publication boundary. The
release decision remains with the maintainer. The source CI and review checks
remain independent of feature activation.

14. Testing strategy

Area Minimum distinct cases Behaviors/risks covered
Source and identity 2 Candidate binding and evidence provenance
Investigation and closure 2 Profile inheritance and a missing material citation
Rollout and approval 2 Shadow isolation and rollback invalidation
Total 6 Deterministic coverage separate from the qualification corpus
Case ID Area Requirements Scenario
bbe950-rollout-C01 Source and identity SDD-I950-ROLLOUT-R1 A new candidate retains its exact reviewed head and anchor through closure.
bbe950-rollout-C02 Source and identity SDD-I950-ROLLOUT-R1 A foreign receipt or moved head prevents verified publication.
bbe950-rollout-C03 Investigation and closure SDD-I950-ROLLOUT-R2 With no override the verifier inherits the effective reviewer profile.
bbe950-rollout-C04 Investigation and closure SDD-I950-ROLLOUT-R2 If a required versioned source or material citation remains unavailable, the result identifies the gap and is inconclusive.
bbe950-rollout-C05 Rollout and approval SDD-I950-ROLLOUT-R3 Shadow verdicts are recorded without changing publication or authorizing clean approval.
bbe950-rollout-C06 Rollout and approval SDD-I950-ROLLOUT-R3 A policy change or rollback requires a valid current receipt even at the same head.

15. Documentation

Path Change
specs/bugbot-analysis-publication-and-autofix.md update
docs/bugbot/detection.mdx reference
docs/bugbot/configuration.mdx reference
docs/bugbot/quality-observability.mdx reference

16. Acceptance

Validate the source, exact anchor, verdict and policy bindings in the ordinary
and framed routes. Keep the 27 historical reports traceable and keep recorded
pilots reproducible. Each profile proposed for enforcement must independently
satisfy the existing historical reproduction, paired improvement, positive
control, integrity and latency criteria and the independent 174-case detection
corpus. Evidence for readiness and authorization for production are distinct.

17. Requirements traceability

Requirement Cases
SDD-I950-ROLLOUT-R1 bbe950-rollout-C01, bbe950-rollout-C02
SDD-I950-ROLLOUT-R2 bbe950-rollout-C03, bbe950-rollout-C04
SDD-I950-ROLLOUT-R3 bbe950-rollout-C05, bbe950-rollout-C06

20. References and decisions

The existing Bugbot SDD and its evidence reports document the qualification
thresholds, supported profiles, approval policy and rollback protocol.
This revision neither enables production verification nor qualifies a profile.

Activity

  1. added
    enhancementNew feature or request
    priority: lowNot urgent, can be addressed later.
    featureLabel to manage feature branches
    on Nov 9, 2026
  2. moved this to Todo in vypdevon Nov 9, 2026
  3. changed the title [-]Verify Bugbot findings against repository evidence before publication[/-] [+]✨ - Verify Bugbot findings against repository evidence before publication[/+] on Nov 9, 2026
  4. vypbot commented on Nov 9, 2026

    @vypbot
    Contributor

    Hi! I’m @vypbot, the Copilot assistant for this repository.

    I can answer questions, explain the codebase, propose implementation and test plans, review issues and pull requests for potential bugs or security problems, and help authorized maintainers apply changes.

    Try /copilot help to see the available commands, or mention @vypbot with your question.

  5. added
    SDDAn SDD update is required before branch work.
    on Nov 9, 2026
  6. vypbot commented on Nov 9, 2026

    @vypbot
    Contributor

    SDD work status

    Current status: Waiting for specification answers. No SDD draft or branch exists yet.

    Owning SDD: specs/bugbot-analysis-publication-and-autofix.md

    Clarification round: 1

    Draft SHA-256: d8c73180d00c2e2a119bdbe33b3ba4ad2c0e813b97189ca047df829414dba420

    • Q1 · Maintainer: Review the draft and reply exactly with use seed 69d3a5481a4b443faf19ff7bcdb268330adc355ee0e7c34cccc7d2d697ddcd12 or generate 69d3a5481a4b443faf19ff7bcdb268330adc355ee0e7c34cccc7d2d697ddcd12.

    Reply with SDD Q1: your answer (one line per question). The Action will continue after the required people answer every question.

  7. added
    state:specifyingThe issue contract is being clarified and specified.
    on Nov 9, 2026
  8. 242 remaining items

  9. added a commit that references this issue on Nov 19, 2026
  10. vypbot commented on Nov 19, 2026

    @vypbot
    Contributor

    Issue closed: synchronization recommendation retired

    No branch synchronization is required for this closed issue.

    The previous recommendation is historical. Issue closure does not certify that the branches are synchronized.

  11. moved this from In Progress to Done in vypdevon Nov 19, 2026
  12. efraespada commented on Nov 19, 2026

    @efraespada
    MemberAuthor

    Follow-up backlog after #951 — qualification first

    PR #951 is merged; its source/CI closure is complete. The verifier profile is not qualified and BUGBOT_VERIFICATION_MODE remains disabled.
    This is a backlog, not an instruction to start branches or model runs. All 13 issues were created from the installed bug/chore forms, with one priority each.

    Start next: #970 and #971 can be prepared in parallel. Finish the measurement-integrity controls before freezing another measured source.

    Qualification path

    Order Issue Priority Closure needed
    1a #970 — Executed source and sanitized Git provenance High Proven source/lock/driver identity before imports or calls
    1b #971 — Diagnose failed precision, F1 and calibration High Independent per-error evidence and justified narrow corrections
    2a #972 — Propagate explicit cancellation in the shadow runner Medium An interrupted operation cannot be reported complete
    2b #973 — Correct deletion-only citation metadata Medium Grounding metadata agrees with the actual fixed diff
    3 #974 — New preregistered 174-case corpus run High Every unchanged detection gate passes on the exact new pin
    4 #975 — The 42 paired full-review slots High Profile-specific false-positive, positive-control, citation, integrity and latency gates
    flowchart LR
        A["#970 Source integrity"] --> E["#974 Corpus gate"]
        B["#971 Audit and justified corrections"] --> E
        C["#972 Cancellation"] --> E
        D["#973 Diff provenance"] --> E
        E -->|"passing score and identical pins"| F["#975 Paired full reviews"]
    
    Loading

    Text equivalent: complete #970-#973 and any independently justified corrections from #971; freeze the exact implementation/drivers/runtime; run #974. Admit #975 only if the corpus actually passes for those identical pins. Six native GitHub blocked-by relationships record this sequence and the writer-install dependency below.
    Closing #974 with a failed report does not satisfy the passing-score condition. A changed implementation invalidates reuse of the earlier source-bound qualification.

    The previous campaign completed 174/174 but failed precision (80.23% <90%), F1 (83.13% <87%) and Brier (0.1883 >0.16). Its 42 review slots were unstarted and Sol measurements were zero. Preserve these negative results; do not turn unmatched scorer categories into independently established false bugs.
    The existing p95 qualification threshold is evaluated after investigation. Do not restore token, elapsed-time or turn-count research cutoffs. Unknown token usage never rejects a supported verdict.

    Operational follow-up and rollout prerequisites

    Issue Priority Relationship to qualification
    #976 — Accepted dispatch without run ID Medium Recovery correctness before installing the new writer
    #977 — Trusted-default status-writer installation Medium Blocked by #976; operational installation, not a prerequisite for local corpus/full-review dry-run
    #980 — Approval-receipt producer trust boundary Medium Required trust review before any proposed enforced activation; matching run/artifact metadata is not code attestation
    #979 — Credential-health trusted-ref policy Medium Separate operator/runner hardening; a branch-local condition is not an independent security boundary
    #978 — CI aggregate-predicate validator Low Regression protection; the currently shipped gate already fails each worker failure

    The reported broad action-edit approval route was not reproduced by the guarded-policy packaged controls: protected bundle/workflow/action.yml edits return protected-path with zero mutations. #980 retains the narrower producer-trust limitation rather than asserting a proven approval exploit.
    #977 must inspect the current trusted default revision; the absence recorded in #951 is a historical snapshot, not an assumption about a later installation. Coordinate installation tooling with existing #814. No deployment, default-branch merge or activation is authorized by this index.

    Separate Setup acceptance, outside EV-07

    Issue Priority Existing open observations
    #981 — Real screen-reader and 200% zoom journeys Low U076/U077, U081/U082, U086/U087, U091/U092: eight observations in four locales
    #982 — Packaged platform and Windows symlink checks Low I039-I042 and X023/X026: six actual Linux/Windows observations

    These two issues account for all 14 open Setup ledger cases (336/350 passed). They do not block qualification of the Linux Bugbot profile. DOM inspection, another OS, a skipped test or a queued runner cannot replace the missing observation; these tasks invoke no real models.

    Preserved scope and decision

    Active execution — 2026-11-19 23:42 UTC

    The GitHub Action has now created and linked all twelve implementation/evidence branches (#970–#973, #976–#982 and causal #985). #974 and #975 remain unstarted. All contributor pushes update those exact existing refs normally. No manual remote lifecycle, merge, installation, release, deployment or activation has occurred.

    Issue PR / published SHA Real state and executed evidence Pending / next action
    #970 #983 · published 01097099d220bd481f6e50b58d2213165bcf6194 Published SHA passes required CI/Commit/Verification/Branch Sync, including 187 Linux controls, but final Review is NEUTRAL with a valid MEDIUM checkout-receipt finding. The real old verifier accepts a fabricated file and the direct runner imports checkout code; causal failures are preserved. Local correction removes file authority, rejects direct CLIs before imports and routes each worker through the external bootstrap with process-local IPC context and repeated byte proof. 193 combined controls pass, including six new forgery/IPC/nested/cancellation cases; the actual legacy verifier is frozen for squash-safe replay. New correction is not yet committed/published. Documentation/SDD/catalog and Graphify updated; final frozen-counter replay and mandatory complete build/Jest/lint hook precede a new push and final-SHA native review. Earlier CI does not validate this new correction. #974 remains unstarted.
    #971 #984 · 1f3fbddeb4b84dea231ddeb7dc18af54fa33a565 All 28 scorer rows audited; frozen scores reproduce; three audit controls and full hook pass. Required final-head CI/Commit/Review/Verification succeed; no open review threads. Automatic approval is neutral, not a maintainer approval. Ready for maintainer review/merge. Labels, five unresolved material premises and negative results unchanged.
    #972 #988 · e1b7d2d22c9e5cca34fa85bbfcf9f7b439c0ac88 Seven cancellation / 147 shadow controls, causal false-success control and full 943-suite hook pass. Exact compressed failed/positive logs retained. Every final-head native check including aggregate CI and Review succeeds; no open review threads. Ready for maintainer review/merge. Combined prerequisite integration still requires Action/maintainer integration and verification before #974.
    #973 #986 · dfe3e4aaa08e6cc95ddbeaa55c7daf833653446a Six citation/receipt / 146 shadow controls and full 943-suite hook pass. All required native CI/Commit/Review/Verification/Branch Sync succeed, including aggregate23:07:32 after exactly one static retry and the separate single Branch Sync retry. Original provider failure, static timeout, failed whole run and 134-test isolated pass retained. Ready for maintainer review/merge and combined prerequisite validation. Automatic approval is neutral, not a maintainer approval. No further retry or watcher is needed.
    #974 No PR; unstarted No new qualification campaign; required deterministic/causal prerequisites are not merged. Integrate #970–#973 and #985, then commit exact implementation/drivers/runtime/corpus/prompts/schemas/order/retry policy before measuring.
    #975 No PR; 42/42 slots unstarted No passing new corpus on identical pins; zero new qualification verifier measurements. Admit only after a genuinely passing #974 result. No profile is qualified.
    #976 #989 · 1b024481d9d86cb4d4b10eb7f8569f9c8e2f0981 62 dispatch/worker/handoff/publication controls and full 943-suite / 18,715-test hook pass. Required final-head CI/Commit/Review/Verification succeed; no open threads. Automatic approval is neutral, not a maintainer approval. Ready for maintainer review/merge. Does not establish default installation or durable exactly-once across arbitrary process restart.
    #977 #993 · c7d6d6a2eb0db6d02376df8348707e5795a36999 Complete review-only artifact, exact source/default manifest and #976 Action pin. Nine artifact / 149 shadow / 14 packaged localhost controls plus full 943-suite / 18,695-test hook pass. Native Commit/Verification/Review succeed, no open threads; all required native CI including aggregate and Review now succeed. Actual installation remains open: #976 integration/approval, exact-manifest installation authorization, supported #814-coordinated installer/default merge and actual native canary on an explicitly authorized test PR. No installation or native canary performed.
    #978 #987 · 0b99506bf04b345b1c5d027bef6a02b6cb1874bc 360 focused / 36 workflows / full 943-suite / 18,703-test hook pass. Native Commit/Review/Verification, all CI workers and aggregate CI now succeed, including the single retained coverage retry. Current automatic Approval check is failure, recorded20:20 before CI succeeded. The trusted-default observer is queued (44611328444) with a newer pending run (44629019683); its reviewed default YAML serializes that observer. These are waiting states, not a fresh code failure or an approval. No approval observer was manually dispatched or canceled. Source implementation is prepared for maintainer review. No gate or timeout weakened.
    #979 #991 · f0655e7f64d7d141de2e943b68db21e8fdb32392 Source-only policy proposal: exact default/workflow/ref, two actor IDs plus rerun identity, self-hosted vypdev group. 503 focused / 944-suite / 18,706-test hook pass. Native Commit retry, Review/Verification and all CI including aggregate succeed. Original provider assessment failure retained. Maintainer policy decision pending; editable guard is not independent workflow-editor enforcement. Org group policy is unreadable403. No health dispatch, installation or hardened claim.
    #980 #992 · 6efac73417df179a3153b526ae6df9b0b53c976b Final publication-assertion correction: 14 packaged cases / 141 shadow / full 943-suite / 18,695-test build/lint pass; old false-pass control and new rejection retained. Native Commit/Verification/Branch Sync/Review/static CI succeed. Bot resolved its earlier LOW finding on this exact SHA; no open review threads. Bundles/production behavior unchanged. All required native checks including build/coverage/aggregate and Review now succeed; ready for maintainer review/merge. Independent producer-code attestation remains unestablished; no activation is claimed.
    #981 #994 · aa5c03551d283ad62fa36d8d80912090065d212d Frozen source/package/CLI/UI packet and eight blank human rows published. Acceptance/docs/spec/graph and full 943-suite / 18,695-test hook pass; ledger/bundles byte-unchanged at 336/350, 14 open. Native Commit/Verification/Branch Sync/Review and all three CI workers succeed on aa5; no human pass is claimed. All required native checks including aggregate CI now succeed. 0/8 new actual observations: complete speech / actual 200% journeys with named reviewer in en/es/fr/pt remain required; request unanswered. Separate from Linux Bugbot.
    #982 #996 · bbf3bf1660a4f53e194bc2eb965287c70439741a Independent six-case source/package packet, exact Windows assertion commands and blank evidence rows published. Acceptance/docs/spec/graph and full 943-suite / 18,695-test build/lint retry pass; first worker SIGSEGV and six-test isolated pass retained. Ledger/bundles unchanged at 336/350, 14 open. Native Commit/Verification/Branch Sync/Review and static CI succeed on the exact bbf3 head; Review completed without findings. All required native CI including coverage/build/aggregate now succeed. 0/6 new actual observations: actual Linux/Windows desktops and Windows-service symlink capability/reviewer still required. No platform-observation job dispatched; no privilege/runner/template change; Windows service stays off codex label.

    Causal #971 correction: #985 / PR #990, ddc768b2e57922ff153bf90553472cc0e849755d, publishes a new 174-case v2 artifact correcting only malformed workflow context, with unchanged IDs/labels and immutable v1/history. Six controls, 146 shadow controls, full 943-suite hook and every native check including aggregate CI pass; Review succeeds without open findings. Maintainer review/merge is pending. Future #974 must explicitly preregister the new corpus/driver identity; old defaults and old negative scores are unchanged.

    Qualification: no profile qualified. The audit separates 17 FP / 11 FN scorer rows from independently established defects; 22 rows are paired taxonomy mismatches, one malformed fixture has a causal follow-up, five material premises remain unresolved. No retrospective relabeling or threshold change.

    External prerequisites: #970–#973 and causal #985 must be integrated by the maintainer/Action before #974 can register and measure its exact combined source. #977 additionally needs the exact manifest/source/default approval, supported installation receipt and actual native canary on an explicitly authorized PR. #979 policy approval and the 8+6 real Setup observations remain pending. A queued run or prepared packet establishes none of those facts. #973's original CI run is now terminal failure, with successful build/coverage and its preserved static timeout. The one same-SHA static retry and its aggregate now succeed. Branch Sync's separate single retry also succeeds. No further retry is requested.

    Combined-source preview: a disposable local git merge-tree preview of exact #970 01097099d… and #972 e1b7d2d… reports conflicts in normal-flow-shadow/run.cjs, package.json and docs/development/testing.mdx. This generated no commit, remote ref, test result or model call. Action/maintainer integration must preserve both authenticated bootstrap admission and explicit operation cancellation, then run the combined controls. Separate passing branches do not establish the source identity needed by #974.

    Shared-runner coordination (#995): after checking workflow role, exact owned branch/SHA and the already-running current-head successor, canceled only the initial duplicate #981 Branch Sync observer 44613039500 at old base ccefb. It finished cancelled at 22:06; current-head successor 44616045862 succeeded at 22:10, and remote aa5c035 remains unchanged. Before/cancel/after records are retained locally. After publishing final #970, its old37560 CI run 44613311674 had all three workers already terminal/cancelled and only the always() aggregate queued, retaining concurrency ahead of the new01097099 CI 44627348541. Normal cancellation was accepted but did not close that aggregate; after rechecking no active job and unchanged current head, one documented force-cancel returned202. Old run completed cancelled at22:26:49 and the current-head CI moved from pending to queued. No active test or model investigation was interrupted; records were retained. No other repository, active current-head review, runner placement, concurrency, token or protection was changed.

    Current issue and review audit: all thirteen issues #970–#982 plus causal #985 remain open. Their complete available comments contain no new human decision or observation since goal execution began; PR991 has no submitted maintainer review. Current descriptions of preparation PR993/994/996 explicitly retain installation or real-observation requirements and claim no acceptance from the packets. Managed PR/issue relationships are preserved; actual acceptance remains pending. Source snapshots and comments/reviews were retained locally.

    Safety state: repository Variable independently read at this update: BUGBOT_VERIFICATION_MODE=disabled (unchanged since 2026-11-18). No new qualification model calls and zero evaluated-PR write attempts. Fixture writes are local fakes and explicitly counted; native delivery reviews are separate from qualification/platform pilots. Frozen campaigns, negative attempts, preregistrations and the 27-issue matrix remain unchanged. No prerequisite PR has merged at the last authenticated read. Approval, merge and installation remain with the maintainer/Action.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

100%Progress: 100%SDDAn SDD update is required before branch work.branchedIndicates that a branch has been created for this issue.enhancementNew feature or requestfeatureLabel to manage feature branchespriority: lowNot urgent, can be addressed later.size: XXLIndicates an extremely large task or issue that requires significant time and effort to complete.

Projects

  • Status
    Done

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions