Skip to content

πŸ›πŸ§‘β€πŸ’» - Repeated native citations pass the assessment schema then abort same head progressΒ #944

Closed

Description

@efraespada

Is there an existing issue for this?

  • I have searched the existing issues.

Which actions are affected?

Commits

Which platforms are affected?

Linux, macOS

Description

After restoration to the published v3 channel, the native Domus progress evaluation completed its 17 source partitions, then stopped in acceptance batch 1/4 with validation.evidence-unavailable, requirement R63, reason evidence-missing. Existing progress was correctly retained. The consumer's ordinary CI, both actual container architectures and full PR review completed successfully; progress validation remains red.

A read-only deterministic probe of the published SDK exposes a contract mismatch: progressBatchResponseSchema permits citation arrays of 1–8 admitted IDs, including repeated IDs; assertAgentResponseSchema accepts those repetitions. Both the prose and closed-fact profiles preserve those citation arrays. readProgressAssessmentEvidence subsequently rejects repetitions as evidence-missing, aborting the complete evaluation without the existing complete-batch correction allowance.

The original native response was not retained and this report does not claim a raw citation capture. Repetition is inferred from the native failure path after adapter schema validation; the synthetic counterexample independently proves the schema/host mismatch and reproduces the same R63/batch1 diagnostic. A valid unique-citation control succeeds. Missing, foreign, generated, stale or otherwise malformed evidence must continue to block publication.

Expected: state the distinct-citation contract explicitly and handle an otherwise fully valid, attributable repeated-citation response within the existing single complete-batch correction allowance, shared with length/completion recovery. Preserve every requirement, distinct admitted source reference, identity, response bound, the 120-call/30-minute ceiling and final publication gate. A repeat after the shared allowance is consumed must produce a precise content-free, nonretryable diagnostic. Do not silently deduplicate model output, invent citations, accept invalid receipts, add another allowance, weaken coverage or rerun unchanged heads until green.

Reproducing the issue

  1. Inspect native Commit 42518152136, job128845593649, on consumer 193bd116a0d3e36b8db31f818ce179d12c37e7d8. It downloads vypdev/copilot@v3 at d2f7523d1f2f8a302130d91f15b421e74b43632a. The original complete issue body remains SHA256 f5c01613456235e349179026a023ad992b3d95fc427568f27c8f7e31fbf8b2d0.
  2. Build an ordinary one-row batched manifest with stable R63 and one valid native source reference using createProgressScopeManifest, planProgressAssessmentBatches and progressBatchResponseSchema.
  3. Encode a valid closed-fact response: assessment: source-unverified, progress50, all exact manifest/batch/head identities, and citations: [reference, reference].
  4. The actual assertAgentResponseSchema(response, compactPlan.schema) succeeds. Decode through decodeCompactProgressAssessment and pass through parseProgressAssessmentReceipt: it throws validation.evidence-unavailable, progress-batch-retained, evidence-missing, R63/batch1.
  5. Change only the array to [reference]: the exact same adapter/schema/decoder/receipt chain succeeds. The standalone probe makes zero model queries, zero GitHub mutations and no repository source changes. Probe and native log evidence are retained in the task's release evidence directory.

copilot Version

3.10.12, v3 and v3.10.12 resolve to accepted production d2f7523d1f2f8a302130d91f15b421e74b43632a. Native agent codex-cli0.167.0, model gpt-6-luna. Adapter/control counterexample executed on macOS with Node24.20.0.

Relevant Log Output

Assessing progress evidence part 17 of 17, round 2 of 2.
Assessing acceptance batch 1 of 4; verified 0. Previous progress is retained.
Complete same-head progress evidence was not verified. Existing progress was retained.
Impact: Progress retained. Verified 0 of 4 batches.
Error code: validation.evidence-unavailable
Action: Inspect requirement R63, batch 1, reason evidence-missing.
Retryable: No
Reference: 0ba6f456-c65b-440d-8eae-3f5e91bfcdcb

Additional context and comments

Follow-up discovered while carrying out the maintainer's authorized operational investigation and patch940 publication. This differs from the completed-field/UTF-8 quota failure fixed in #937/#938; no quota failure was reported in this run. Do not change or replace the immutable published 3.10.12 artifact. Source correction must use the normal Action-managed bugfix branch and PR, with full checks, changed-line coverage and zero active findings. No additional release number is authorized by this report.

Update the catalogued progress/recovery SDD and operator documentation together. Cover both wire profiles, the unique-citation control, repeated-citation recovery and exhaustion, mixed failures sharing the same allowance, stale/foreign/malformed/forged failures, complete frozen batches and no partial publication. Existing repository test budgets and all coverage gates remain in force. Preserve the failed native run instead of retrying it for a green status.

Activity

  1. added
    bugSomething isn't working
    bugfixSomething isn't working
    priority: highCritical issue, needs immediate attention.
    on Nov 8, 2026
  2. added theissue type on Nov 8, 2026
  3. moved this to Todo in vypdevon Nov 8, 2026
  4. changed the title [-]Repeated native citations pass the assessment schema then abort same-head progress[/-] [+]πŸ› - Repeated native citations pass the assessment schema then abort same head progress[/+] on Nov 8, 2026
  5. vypbot commented on Nov 8, 2026

    @vypbot
    Contributor

    Hi! I’m @vypbot, the Copilot assistant for this repository.

    I can answer questions, explain the codebase, propose implementation and test plans, review issues and pull requests for potential bugs or security problems, and help authorized maintainers apply changes.

    Try /copilot help to see the available commands, or mention @vypbot with your question.

  6. added
    contract-changeThe issue changes a product or engineering contract.
    in-progressStart work on an admitted issue.
    on Nov 8, 2026
  7. added
    SDDAn SDD update is required before branch work.
    on Nov 8, 2026
  8. moved this from Todo to In Progress in vypdevon Nov 8, 2026
  9. 7 remaining items

  10. added
    size: XXLIndicates an extremely large task or issue that requires significant time and effort to complete.
    75%Progress: 75%
    and removed on Nov 8, 2026
  11. vypbot commented on Nov 8, 2026

    @vypbot
    Contributor

    Issue closed β€” historical progress

    Current status: closed as completed.

    The last estimate is retained below as history. Closure does not certify a new percentage, tests, or branch alignment.

    Action required: none for this closed issue. If work continues, reopen the issue; a fresh assessment can update this same card.

    Last assessment before closure (historical)

    scope:2d9a691a1561e5eecf5208460526159046a67e86080efd22eb950224c1e252c7

    Progress: 65% β€” in progress

    > Current status: [R19] The original native response was not retained and this report does not claim a raw citation capture. Repetition is inferred from the native failure path after adapter schema validation; the synthetic counterexample independently proves the schema/host mismatch and reproduces the same R63/batch1 diagnostic. A valid unique-citation control succeeds. Missing, foreign, generated, stale or otherwise malformed evidence must continue to block publication. β€” Source preserves blocking checks for invalid evidence and adds distinct-citation recovery; the original native response remains unavailable.
    [R21] Expected: state the distinct-citation contract explicitly and handle an otherwise fully valid, attributable repeated-citation response within the existing single complete-batch correction allowance, shared with length/completion recovery. Preserve every requirement, distinct admitted source reference, identity, response bound, the 120-call/30-minute ceiling and final publication gate. A repeat after the shared allowance is consumed must produce a precise content-free, nonretryable diagnostic. Do not silently deduplicate model output, invent citations, accept invalid receipts, add another allowance, weaken coverage or rerun unchanged heads until green. β€” Source implements one shared correction allowance, distinct citations, and a nonretryable exhaustion diagnostic.
    [R52] Update the catalogued progress/recovery SDD and operator documentation together. Cover both wire profiles, the unique-citation control, repeated-citation recovery and exhaustion, mixed failures sharing the same allowance, stale/foreign/malformed/forged failures, complete frozen batches and no partial publication. Existing repository test budgets and all coverage gates remain in force. Preserve the failed native run instead of retrying it for a green status. β€” The catalogued SDD and operator documentation cover both profiles, recovery boundaries, invalid evidence, and publication behavior.

    Next: [R19] The original native response was not retained and this report does not claim a raw citation capture. Repetition is inferred from the native failure path after adapter schema validation; the synthetic counterexample independently proves the schema/host mismatch and reproduces the same R63/batch1 diagnostic. A valid unique-citation control succeeds. Missing, foreign, generated, stale or otherwise malformed evidence must continue to block publication. β€” Independent verification of the required evidence gates is unavailable.
    [R21] Expected: state the distinct-citation contract explicitly and handle an otherwise fully valid, attributable repeated-citation response within the existing single complete-batch correction allowance, shared with length/completion recovery. Preserve every requirement, distinct admitted source reference, identity, response bound, the 120-call/30-minute ceiling and final publication gate. A repeat after the shared allowance is consumed must produce a precise content-free, nonretryable diagnostic. Do not silently deduplicate model output, invent citations, accept invalid receipts, add another allowance, weaken coverage or rerun unchanged heads until green. β€” Required execution and independent checks for recovery, mixed failures, and publication gates are unverified.
    [R52] Update the catalogued progress/recovery SDD and operator documentation together. Cover both wire profiles, the unique-citation control, repeated-citation recovery and exhaustion, mixed failures sharing the same allowance, stale/foreign/malformed/forged failures, complete frozen batches and no partial publication. Existing repository test budgets and all coverage gates remain in force. Preserve the failed native run instead of retrying it for a green status. β€” Required repository checks, coverage gates, and independent native consumer qualification remain unverified.

  12. vypbot commented on Nov 8, 2026

    @vypbot
    Contributor

    Issue closed: synchronization recommendation retired

    No branch synchronization is required for this closed issue.

    The previous recommendation is historical. Issue closure does not certify that the branches are synchronized.

  13. moved this from In Progress to Done in vypdevon Nov 8, 2026
  14. added
    100%Progress: 100%
    and removed
    in-progressStart work on an admitted issue.
    state:workingWork can proceed on the verified branch or without a branch.
    on Nov 8, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

100%Progress: 100%SDDAn SDD update is required before branch work.branchedIndicates that a branch has been created for this issue.bugSomething isn't workingbugfixSomething isn't workingcontract-changeThe issue changes a product or engineering contract.priority: highCritical issue, needs immediate attention.size: XXLIndicates an extremely large task or issue that requires significant time and effort to complete.

Type

Projects

  • Status
    Done

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions