Skip to content

Request Copilot at Defined Moments and Cover Fix Pushes With an Attested Local Pass - #2278

Merged
ptr727 merged 9 commits into
developfrom
feature/review-moments-2261
Oct 2, 2026
Merged

ptr727 merged 9 commits into
developfrom
feature/review-moments-2261

Conversation

@ptr727

@ptr727 ptr727 commented Oct 2, 2026

Copy link
Copy Markdown
Owner

Summary

This is the defined-moments half of #2261, per the maintainer's answers recorded on that issue. A Copilot round is requested when a pull request opens and on the head of a pull request into the default branch, a promotion among them, rather than on every push. On a pull request into any other branch, a fix push after the first round is covered by the recorded local strict-review pass the push already owes. The PR publishes that pass so the review gate can read it.

  • Ruleset. repo-config/develop.json and main.json set review_on_push and review_draft_pull_requests to false and keep review on open. Applying the change to live repositories is a separate config run, which needs its own go-ahead. Narrowing the account-level setting is a UI step for the maintainer.

  • pr_review.py attest. It publishes the local pass as a comment carrying <!-- fleet-local-review: head=<sha> findings=<n> -->, after confirming four things:

    • the checkout is at the pushed head
    • the checkout holds no other change
    • its merge base is the pull request's own
    • local_review.py check passes against the base

    The comment carries the findings count the pass recorded. Per the maintainer's answer, the count is shown and not gated, since local findings are advisory. local_review.py status now reports the per-reviewer findings map that this reads.

  • status. An attested head on a pull request into a non-default branch, after Copilot's first round, reads as review_on_head=local and coverage=local. That holds only for an owner, member, or collaborator comment whose marker stands as a line of its own outside a fence, and only where the whole review history is in view and no round on record states or appears to state partial coverage.

  • wait. On such a head wait requests nothing. An attested head ends as covered, and one with no attestation exits 49 (AWAITING_LOCAL_PASS). --request asks for a round anyway. A promotion, a first round, a partial on record, and a history past the window are all requested as before.

  • A post-merge finding on Stop Requesting Copilot Into a Reached Rate Limit #2277, fixed here. A newer quota or error refusal now outranks older genuine coverage of the same head, in the digest's refusal= field and in wait's verdict, since it is the newest word on the pull request.

  • CodeRabbit. A drive prompts it at most once per pull request, on the head it judges final, and never after a rate-limit notice.

  • Docs. The rule is stated in GOVERNANCE.md "PR Review Etiquette", pr-review-conduct (loop step 3, step 8, Merge Gate item 2, and the reviewer bullets), the Copilot runbook, scripts/README.md, repo-config/README.md, docs/pr-reviewer-reference.md, and the one-liners in README.md, RESYNC.md, and AUDIT.md.

Verification

  • tests.test_pr_review and tests.test_local_review pass (658 tests). Each new behavior was shown to fail with it reverted or mutated, including the stale-refusal, partial, old-answer, and marker-shape cases.
  • A live GraphQL read confirmed that baseRepository.defaultBranchRef and IssueComment.authorAssociation exist.
  • Two local strict-review passes ran. Round 1 raised 16 findings: 15 are fixed, and Develop #5 went to the maintainer, who chose to show the count without gating on it. Round 2 raised 3 low findings, all fixed.

Closes #2261

🤖 Generated with Claude Code

ptr727 and others added 5 commits October 1, 2026 20:44
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ted Local Pass

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
An attested head is covered whatever a stale refusal or an old plain answer says, a partial
on record or a truncated history requests a round instead of holding, the marker counts only
as a line of its own outside a fence, and attest checks the merge base and an unconfigured
clean tree. The docs name a pull request into the default branch rather than a promotion
alone, and drop the per-push wording left beside the new rule.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Per the maintainer's answer, attest reads the covering passes' recorded findings from
local_review.py status and carries the total in its marker, which status prints beside the
local-pass reading and nothing gates on. Unread merge-base cases report apart, the remote
ref is named in full, and the header states the new refusal and the requested cases.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Copilot AI lite review requested due to automatic review settings October 2, 2026 13:21
@coderabbitai

coderabbitai Bot commented Oct 2, 2026 •

Copy link
Copy Markdown

Important

  • 🔍 Trigger review

This repository does not receive automatic reviews because it has fewer than 10 stars.

⚙️ Run configuration

Configuration used: Repository: ptr727/ProjectTemplate/.coderabbit.yaml

Review profile: CHILL

Plan: Advanced

Run ID: 5c02f1d0-e473-4695-af6f-070d06d180e4

  • Autopilot · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Autopilot is currently an internal CodeRabbit preview.


Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot review overview

🟡 Changes recommended

Unresolved critical and moderate findings remain in attestation and review-handling behavior.

Review effort: Lite
Findings: 2 High severity · 1 Medium severity · 1 Low severity

Open (4)
What changed in this PR

This PR defines Copilot review moments and adds attested local-review coverage for fix pushes.

Changes:

  • Adds local-pass attestation and findings reporting.
  • Updates review logic, rulesets, tests, documentation, and skills.
  • Adds regression coverage for refusal precedence and attestation behavior.
File Description
tests/​test_pr_review.py Tests review and attestation behavior.
tests/​test_local_review.py Tests findings reporting.
scripts/​README.md Documents the attestation workflow.
scripts/​pr_review.py Implements attestation and local coverage.
scripts/​local_review.py Reports recorded findings.
RESYNC.md Updates review-loop guidance.
repo-config/​README.md Documents ruleset behavior.
repo-config/​main.json Disables push-triggered reviews.
repo-config/​develop.json Disables push-triggered reviews.
README.md Updates review-loop guidance.
GOVERNANCE.md Defines review moments and coverage.
docs/​pr-reviewer-reference.md Updates review behavior guidance.
AUDIT.md Updates convergence guidance.
.github/​skills/​pr-review-conduct/​SKILL.md Updates the review skill.
.github/​copilot-instructions.md Updates the Copilot runbook.
.claude-plugin/​fleet-skills/​skills/​pr-review-conduct/​SKILL.md Updates the generated skill.
.claude-plugin/​fleet-skills/​.source-digests/​pr-review-conduct Updates the source digest.
.agents/​skills/​pr-review-conduct/​SKILL.md Updates the canonical review skill.

💡 Configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

Comment thread scripts/pr_review.py Outdated
Comment thread scripts/pr_review.py
Comment thread scripts/pr_review.py
Comment thread scripts/pr_review.py Outdated
…ttestation

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Copilot AI lite review requested due to automatic review settings October 2, 2026 13:31
@ptr727

ptr727 commented Oct 2, 2026

Copy link
Copy Markdown
Owner Author

Answering "Reject --request outside wait" (scripts/pr_review.py:4355), which the reviewer resolved on the fix push: fixed in 26831ab. --request now errors on any command but wait, the same validation --checkout has, with a test.

@codecov

codecov Bot commented Oct 2, 2026 •

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 89.07563% with 13 lines in your changes missing coverage. Please review.
⚠️ Please upload report for BASE (develop@5a814c1). Learn more about missing BASE report.

Files with missing lines Patch % Lines
scripts/pr_review.py 89.07% 13 Missing ⚠️
Additional details and impacted files
@@            Coverage Diff             @@
##             develop    #2278   +/-   ##
==========================================
  Coverage           ?   58.22%           
==========================================
  Files              ?       16           
  Lines              ?     7821           
  Branches           ?        0           
==========================================
  Hits               ?     4554           
  Misses             ?     3267           
  Partials           ?        0           
Flag Coverage Δ
python-3.13 58.22% <89.07%> (?)

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot review overview

🟡 Changes recommended

Unresolved critical and moderate review findings remain in scripts/pr_review.py.

Review effort: Lite
Findings: 1 High severity

Open (1)
Resolved since last review (4)
Previously missed (1)

In code that hasn't changed since last review

Low severity Document all four attestation checks

scripts/​pr_review.py:3788

[P2] Document the fourth attestation check. This docstring says the attestation is gated by three checks, but the function also validates that the checkout's merge base matches the pull request's merge base before running local_review.py check. The count and the documented conditions are therefore incomplete. Describe all four checks so the public command documentation matches the refusal behavior.

Comment thread scripts/pr_review.py Outdated
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Copilot AI lite review requested due to automatic review settings October 2, 2026 13:41
@ptr727

ptr727 commented Oct 2, 2026

Copy link
Copy Markdown
Owner Author

Answering the previously missed finding "Document all four attestation checks" (scripts/pr_review.py:3788): fixed in 60a624f. The attest docstring now names all four checks, the merge base among them, and says the first two are read again once the check returns.

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot review overview

🟡 Changes recommended

Unresolved critical and moderate review findings remain in scripts/pr_review.py.

Review effort: Lite
Findings: 2 High severity

Open (2)
Resolved since last review (1)

Comment thread scripts/pr_review.py
Comment thread scripts/pr_review.py
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Copilot AI lite review requested due to automatic review settings October 2, 2026 13:52

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot review overview

🟡 Changes recommended

Unresolved attestation parsing, receipt validation, failure handling, and documentation findings remain.

Review effort: Lite
Findings: 2 High severity

Open (2)
Resolved since last review (2)

Comment thread scripts/pr_review.py Outdated
Comment thread scripts/pr_review.py Outdated
…und the Marker

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Copilot AI lite review requested due to automatic review settings October 2, 2026 14:02
@ptr727

ptr727 commented Oct 2, 2026

Copy link
Copy Markdown
Owner Author

Answering "Mask inline code spans before parsing the marker" (scripts/pr_review.py:3953), which the reviewer resolved on the fix push: fixed in 793f50f. The marker is now parsed after CODE_SPAN.sub, so a multi-line span no longer leaves a quoted marker on a line of its own, and a test case fails with the mask removed.

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot review overview

🟡 Changes recommended

Unresolved attestation, wait-path, comment-history, and test-helper issues block approval.

Review effort: Lite
Findings: 2 High severity

Open (2)
Resolved since last review (2)

Comment thread scripts/pr_review.py
Comment thread tests/test_pr_review.py
@ptr727
ptr727 merged commit 644b297 into develop Oct 2, 2026
11 checks passed
@ptr727
ptr727 deleted the feature/review-moments-2261 branch October 2, 2026 15:48
ptr727 added a commit that referenced this pull request Oct 2, 2026
…eview (#2284)

## Summary

Promotes develop to main with the pull requests below. Each was already
reviewed and merged into develop.

- [#2271](#2271): Correct
the false no-vocabulary claims about environment secrets.
- [#2276](#2276): A full
Copilot file table on an earlier round carries forward to a later head
when the pull request changes the same set of files at both commits, the
same bound a coverage statement uses.
- [#2275](#2275):
`install-tools.sh` no longer exits 1 silently when the tools named leave
out the last managed tool, and its report notes show a home path as `~`.
- [#2277](#2277): A
Copilot round that says only "encountered an error" is read as a
possible quota hit. `pr_review.py` reads the reviewer's own Actions run
log, which states the rate limit and the time it resets, and `wait` no
longer requests a review into either case.
- [#2278](#2278): A
Copilot round is requested when a pull request opens and on the head of
a pull request into the default branch, rather than on every push.
Covering a fix push into develop:
- A recorded local strict-review pass covers the push, published with
`pr_review.py attest` and read by `status` as `review_on_head=local`.
- The Copilot rule in `repo-config/develop.json` and `main.json` now
reviews on open only, not on push or for drafts.

Applying that ruleset change to the live fleet repositories is a
separate config run after this merges, and it needs the maintainer's
go-ahead. Until it runs, GitHub still reviews every push, including
pushes to this pull request.

Closes #2256
Closes #2260
Closes #2261
Closes #2268
Closes #2272

🤖 Generated with [Claude Code](https://claude.com/claude-code)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants