This runbook sets NCode Review up for an organization and a repository, top to bottom, without reading code. README.md is the reference for each part (Using it, Interface, Approvals, Opting a repository in), and this page is the order to do them in. NCodeGroup/project-template's Adopt a Project, End to End covers the rest of a repository's adoption.
Each step lists Who, Where, Do, Check and Goes wrong. Commands run in PowerShell 7 from the folder the step's Where names. <owner>/<name> is the repository being set up and <number> a pull request. Each entry in section 5 lists Symptom, Cause and Fix.
- How to Read It
- 1. Once per Organization
- 2. Once per Repository
- 3. Keys
- 4. How a Review Behaves
- 5. Troubleshooting
Do these once, in order.
- Who: each maintainer who runs the scripts.
- Where: a PowerShell 7 prompt.
- Do:
gh auth login, thengh auth refresh -h github.com -s admin:org. What each scope is for:repo, whichgh auth logingives, covers the repository's environment and environment secrets and, with the admin role on the repository, project-template'stools/Set-RepositoryBaseline.ps1.admin:orgcovers reading and setting the organization secrets and choosing which repositories can read them, andtools/Enable-CodeReview.ps1stops without it.- GitHub lists and changes an app installation's repositories only for a GitHub App's user token or a personal access token, never for
gh auth login's token, whatever its scopes (read:user,userandadmin:orgdon't help), so a repository is added to a selected-repositories installation on the app's installation page (2.1), and the script warns and goes on (#236).
- Check:
gh auth statuslistsadmin:orgamong the token's scopes, andgh secret list --org NCodeGroupsucceeds. - Goes wrong: a
GH_TOKENorGITHUB_TOKENenvironment variable replaces the signed-in account's token and its scopes; see 5.3.
- Who: an organization owner.
- Where:
https://github.com/organizations/NCodeGroup/settings/apps/new(Organization settings > Developer settings > GitHub Apps > New GitHub App). - Do:
- Name it NCode Review; it posts as
ncode-review[bot]. - Clear Webhook > Active.
- Set the repository permissions Checks: Read and write (the "Code review" check run), Contents: Read-only, Issues: Read and write (the issue
/review teachopens), Metadata: Read-only and Pull requests: Read and write (threads, the summary comment, reactions and its approval). Grant no other permission and subscribe to no events. - Make it installable only on this account.
- Then Install App > NCodeGroup > All repositories, which covers every repository, NCodeGroup/code-review included, where
/review teach orgopens its issues. - Its client ID, on the app's General page, is
Iv23liAbZt44tgephpLC,app-client-id's default inaction.yml; an app with another ID needsapp-client-idin every caller, or the default changed in a release. - Its private key is stored in 1.5.
- Name it NCode Review; it posts as
- Check:
gh api orgs/NCodeGroup/installations --jq '.installations[] | select(.app_slug == "ncode-review") | {repository_selection, permissions, events}'shows"repository_selection":"all", those five permissions and"events":[]. - Goes wrong: an installation on selected repositories only must have each repository added, as 2.1 does for the approver.
- Who: an organization owner.
- Where:
https://github.com/organizations/NCodeGroup/settings/apps/new(Organization settings > Developer settings > GitHub Apps > New GitHub App). - Do:
- Name it NCode Review Approver; it approves as
ncode-review-approver[bot]. - Clear Webhook > Active.
- Set the repository permissions Contents: Read and write, Metadata: Read-only and Pull requests: Read and write, nothing else, and subscribe to no events.
- Make it installable only on this account.
- Install App > NCodeGroup > Only select repositories, and select only the repositories that require the approval.
- Its client ID is
Iv23liDPFX1K6NjGIr9i,approver-client-id's default. - Why Contents: write is needed: GitHub counts an approval toward a ruleset's required approval only from a reviewer that can write to the repository, and the NCode Review app can't (#232).
- Why it is safe: the action mints the approver's token after the agents exit, for the one repository, with Pull requests: write alone, so no token it holds can push, and its key is a secret of the
code-reviewenvironment, which only the default branch can deploy to (ADR-1011).
- Name it NCode Review Approver; it approves as
- Check: the 1.2 command with
"ncode-review-approver"shows"repository_selection":"selected",{"contents":"write","metadata":"read","pull_requests":"write"}and"events":[]. - Goes wrong: installing it on all repositories widens what a leaked key reaches, since a leaked approver key can mint a Contents: write token for every repository it is installed on.
- Who: an admin of the Anthropic organization.
- Where: the Claude Console's Workload identity page,
https://platform.claude.com/settings/workload-identity-federation(Connect workload > GitHub Actions), with Anthropic's guide Use WIF with GitHub Actions. - Do: the review pays from a workspace and logs in with no secret, through workload identity federation.
- Register the issuer
https://token.actions.githubusercontent.com. - Create the workspace that pays for reviews and a service account.
- Create a federation rule that maps to that service account and workspace and matches only jobs in a NCodeGroup repository's
code-reviewenvironment, whose GitHub token has the audiencehttps://api.anthropic.comand the subjectrepo:NCodeGroup/<name>:environment:code-review. So match the audience, the subject prefixrepo:NCodeGroup/, and the claimsrepository_ownerNCodeGroupandenvironmentcode-review. - NCodeGroup's are the action's defaults: rule
github-code-review(federation-rule-idfdrl_019dXaBRa2Zo1UazugiQwWZ3),organization-ide9301a37-092f-47c8-9b24-188808b1d576,service-account-idsvac_012yav3w3rfhH5i9bAgoWLHeandworkspace-idwrkspc_019f5RLk5SG4XWE8XKwWFHZh. Other values need those inputs in every caller, or the defaults changed in a release.
- Register the issuer
- Check: comment
/wif-teston an issue in NCodeGroup/code-review, as an owner or member; the WIF smoke test'swith-environmentjob gets a token and itswithout-environmentjob is refused. - Goes wrong:
- A rule that doesn't require the environment lets any branch's workflow get a token, and the
without-environmentjob fails with "The rule accepted a job outside the code-review environment". - A caller whose review job lacks
environment: code-reviewgets no token, and the review pays on Bedrock or stops as "Review budget used up".
- A rule that doesn't require the environment lets any branch's workflow get a token, and the
- Who: an organization owner (
admin:org), and an admin of the AWS account for the Bedrock key. - Where: the AWS account's Amazon Bedrock console, in
us-west-2; then an up-to-date clone of NCodeGroup/code-review (rungit pullfirst). - Do:
- Generate a long-term Bedrock API key (it starts with
ABSK) that can call the global inference profilesglobal.anthropic.claude-haiku-5-5,global.anthropic.claude-sonnet-5-5andglobal.anthropic.claude-opus-5-5inus-west-2. - Generate the NCode Review app's private key (3.1).
- Run
.\tools\Enable-CodeReview.ps1 -AppPrivateKeyPath <path to the NCode Review .pem> -WhatIf, then without-WhatIf. It storesNCODE_REVIEW_APP_PRIVATE_KEYand, when the organization has none yet,AWS_BEARER_TOKEN_BEDROCK, read from the environment variable of that name or a hidden prompt, as organization secrets visible only to the repositories that opted in, passing each value on standard input. - Then delete the
.pem(3.1).
- Generate a long-term Bedrock API key (it starts with
- Check:
gh secret list --org NCodeGrouplists both secrets;/wif-test(1.4) also makes a one-token Haiku call on Bedrock with the key and passes. - Goes wrong:
- "That doesn't look like a Bedrock API key (they start with ABSK)."
- "NCodeGroup has no NCODE_REVIEW_APP_PRIVATE_KEY yet. Pass -AppPrivateKeyPath with the app's .pem file."
- "The NCode Review app isn't installed on NCodeGroup. Install it from the app's settings page first."
- Scope errors, 5.3.
Do these for each repository, in order, after section 1.
- Who: an organization owner.
- Where:
https://github.com/organizations/NCodeGroup/settings/installations> Configure next to each app > Repository access. - Do: NCode Review is installed on all repositories, so it needs nothing. For NCode Review Approver, choose the repository under Only select repositories and save, only where the approval will be required.
Enable-CodeReview.ps1also tries to add it, but gh's token can't list the installation's repositories (1.1), so it warns and goes on. - Check: the approver's installation page lists the repository.
- Goes wrong: without the approver's installation the action's approver token step fails, and the publish step warns and approves as
ncode-review[bot], whose approval meets no required approval (2.5).
- Who: an organization owner (
admin:org: the script checks the organization secrets first). - Where: an up-to-date clone of NCodeGroup/code-review: run
git pullfirst, since a clone from before v1.5.0 has no-ApproverPrivateKeyPath(5.1). - Do:
- Run
.\tools\Enable-CodeReview.ps1 -Repo <name> -ApproverPrivateKeyPath <path to the NCode Review Approver .pem> -WhatIf, then without-WhatIf. For several repositories, pass one value,-Repo a,b. - It creates or updates the
code-reviewenvironment so only the default branch can deploy to it, deletes environment secrets named like the organization secrets (they would hide them), gives the repository both organization secrets, and storesNCODE_REVIEW_APPROVER_PRIVATE_KEYas a secret of that environment, never of the organization, so only default-branch jobs can read it. - Without the script, the key alone goes in with
Get-Content -Raw -LiteralPath <path to the NCode Review Approver .pem> | gh secret set NCODE_REVIEW_APPROVER_PRIVATE_KEY --env code-review --repo <owner>/<name>, on standard input, never on the command line. - Then delete the
.pem(3.1). A repository opted in later gets a newly generated key, since the app can hold several, and the others keep theirs.
- Run
- Check:
gh secret list --env code-review --repo <owner>/<name>listsNCODE_REVIEW_APPROVER_PRIVATE_KEY, andgh api orgs/NCodeGroup/actions/secrets/NCODE_REVIEW_APP_PRIVATE_KEY/repositories --jq '.repositories[].full_name'lists<owner>/<name>. - Goes wrong: 5.1, 5.2, 5.3.
- Who: a maintainer of the repository.
- Where:
.github/workflows/code-review.yml, changed in a pull request (both triggers run the file from the default branch). - Do:
-
Start from Using it, or project-template's
code_reviewanswer. -
Pin a release's full commit SHA with its tag in a comment, never a tag or a branch (Versions). The approver needs v1.5.0 or later:
NCodeGroup/code-review@a4cdf65d2450e4e1b2b92ed53430ceb8026d4f97 # v1.5.0.gh api repos/NCodeGroup/code-review/commits/<tag> --jq .shaprints a release's SHA. -
In the
reviewjob'swith:only, which keepsenvironment: code-review, add these two inputs:approve: true approver-private-key: ${{ secrets.NCODE_REVIEW_APPROVER_PRIVATE_KEY }}
-
Never put them in the
outcomesjob: it runs outside the environment, so it can't read the key, and it never approves.
-
- Check: once merged,
gh api repos/<owner>/<name>/contents/.github/workflows/code-review.yml -H "Accept: application/vnd.github.raw"shows both lines in the review job. - Goes wrong: merged before 2.2, the input is empty and NCode Review approves as its own app, which is harmless until 2.4.
- Who: an admin of the repository (
repo). - Where: an up-to-date clone of NCodeGroup/project-template (run
git pullfirst). - Do:
- Only once 2.1 to 2.3 are on the default branch, run
.\tools\Set-RepositoryBaseline.ps1 -Repo <name> -WhatIf, with the-RequireChecksvalue project-template's Run the Baseline gives, then without-WhatIf. - It reads the default branch's
code-review.ymland the environment secret before it requires anything. Withapprove: true, a non-emptyapprover-private-keyand the secret, it requires one approval, of the last push, and prints "NCode Review approves as the NCode Review Approver, which can write, so the ruleset requires one approval, of the last push". Without the secret it prints that the environment has noNCODE_REVIEW_APPROVER_PRIVATE_KEYand requires no approval for it. - It also creates the
pull_request_targetevent policy a public repository needs from 2026-12-02. - For NCodeGroup/code-review, which reviews itself with a released pin, the order is:
- Set the "Protect main" ruleset's required approvals to 0 (Settings > Rules > Rulesets > Protect main).
- Merge and release the version that passes the key and pin it.
- Store the key (2.2).
- Run the baseline again.
- Only once 2.1 to 2.3 are on the default branch, run
- Check:
gh api repos/<owner>/<name>/rules/branches/main --jq '.[] | select(.type == "pull_request") | .parameters.required_approving_review_count'prints1. - Goes wrong: an approval required before the approver is live blocks every merge, which is what blocked this repository's pull requests on 2026-11-21.
- Who: a maintainer.
- Where: the repository's next pull request.
- Do: mark it ready, or comment
/review, and let the review finish. - Check:
- After a clean review,
gh api repos/<owner>/<name>/pulls/<number>/reviews --jq '.[] | .user.login + " " + .state'showsncode-review-approver[bot] APPROVED. gh pr view <number> --repo <owner>/<name> --json reviewDecisionshowsAPPROVED.- The merge box shows the requirement met.
- After a clean review,
- Goes wrong:
- An approval from
ncode-review[bot]means no approver key reached the review (2.2, 2.3, a pin older than v1.5.0) or the approver's token step failed, which the run's log warns about (2.1). - When the approver's review doesn't count,
gh api graphql -f query='query($o:String!,$n:String!,$p:Int!){repository(owner:$o,name:$n){pullRequest(number:$p){reviews(last:20){nodes{author{login} state authorCanPushToRepository}}}}}' -f o=<owner> -f n=<name> -F p=<number>showsauthorCanPushToRepositoryfalse when the approver lacks Contents: write there (1.3). - A missing approval was dismissed by a later review that didn't approve, or by a push. The summary comment's
Approval:line says why.
- An approval from
- Who: an organization owner.
- Where: the app's settings page,
https://github.com/organizations/NCodeGroup/settings/apps/ncode-reviewor.../ncode-review-approver, Private keys; and the clone from 1.5. - Do:
- Generate a private key, which downloads a
.pem. - Store it, the NCode Review app's with
-AppPrivateKeyPath(1.5) and the approver's with-Repo <name> -ApproverPrivateKeyPath(2.2). - Then
Remove-Item -LiteralPath <path to the .pem>. - Why: GitHub keeps the secret and gives it only to the workflows; a new key can be generated at any time; and a leaked approver key can approve on every repository the approver is installed on.
- Generate a private key, which downloads a
- Check:
Test-Path -LiteralPath <path to the .pem>printsFalse. - Goes wrong: a copy kept in Downloads or a backup outlives the deletion.
-
Who: an organization owner.
-
Where: the app's settings page,
https://github.com/organizations/NCodeGroup/settings/apps/ncode-reviewor.../ncode-review-approver, Private keys; and the clone from 1.5. -
Do:
- Generate a new key.
- Store it everywhere (the approver's with
-Reponaming every repository its installation lists). - Delete the
.pem. - Only then delete the old key on the app's page.
The Bedrock key rotates with
.\tools\Enable-CodeReview.ps1 -UpdateBedrockKey, then the old key is revoked in AWS. -
Check: the app's page lists only the new key, and 2.5's check passes on the next review.
-
Goes wrong: deleting the old key first stops every repository that still has it from approving, or, for the NCode Review app, from posting.
What a maintainer sees and does once the repository is set up; Commands and Approvals are the reference.
- Opening, reopening or marking a pull request ready starts a review, drafts included.
- An automatic run on a commit already reviewed stops, and its log says " was already reviewed; comment /review to review it again."
- A push starts nothing, so comment
/reviewas a top-level comment whose first line is exactly/review; lines below it, such as a---footer, are fine. - Only people with write access can. A private organization membership counts as a contributor, so make it public or be added as a collaborator.
- A pull request a bot opens isn't reviewed when it opens, so a maintainer approves Dependabot's where an approval is required.
- The
skip-reviewlabel skips the automatic reviews. - Pull requests from forks are never reviewed.
- After three reviews with no comment from a writer, an automatic review posts "Paused: comment
/reviewto resume".
- With
approve: true, a clean review approves and every other review dismisses the earlier approval. - A push dismisses it when the ruleset dismisses stale approvals, so comment
/reviewafter a push. - Nits don't block: a review whose only findings are nits approves when every reviewer ran in full and verification lowered or refuted nothing.
- A later review can raise a problem's severity. A finding repeats an earlier thread only at the same severity unless a writer dismissed that thread (resolved it while still on its code, or a 👎), so an unanswered nit can come back as a should-fix in a new thread.
- Everything a review publishes, its trace included, is checked for anything shaped like a credential, which a test fixture in the diff can trip, and the review then fails (#233, 5.5).
- ADR-1009 records why a clean review approves.
- Symptom: "A parameter cannot be found that matches parameter name 'ApproverPrivateKeyPath'."
- Cause: the clone's
Enable-CodeReview.ps1is older than v1.5.0. - Fix:
git pullin the clone and run it again.
- Symptom: a warning "Couldn't list the repositories of the NCode Review Approver installation (…): GitHub lists them only for a GitHub App's user token or a personal access token. Make sure / is selected at https://github.com/organizations/NCodeGroup/settings/installations/.", or, from a clone older than #236, the run stops on HTTP 403 from
user/installations/<id>/repositories. - Cause: gh's token can't list an installation's repositories, whatever its scopes.
- Fix: select the repository on the page the warning names (2.1), and
git pullan older clone; adding scopes doesn't help.
- Symptom: "gh can't read NCodeGroup's organization secrets. Grant the scope with: gh auth refresh -h github.com -s admin:org", or an error ending "If it's a 403, the token needs the admin:org scope: gh auth refresh -h github.com -s admin:org".
- Cause: the token lacks
admin:org, or aGH_TOKENorGITHUB_TOKENvariable replaces it. - Fix: run that command, or remove the variable, and check with
gh auth status.
- Symptom: no 👀 and no "Code review" check run, or a check run that ends without findings.
- Cause: one of these, each with its fix:
- a push starts nothing: comment
/review. - the commit was already reviewed: comment
/review. - the comment isn't top-level, doesn't have
/reviewalone on its first line, or comes from someone without write access: see 4.1. - a bot's pull request, a fork or the
skip-reviewlabel: see 4.1. - "Paused": comment
/review. - "Review budget used up", when the plan's API credits are used up or the Anthropic login failed and the Bedrock key is missing, refused or throttled: see 1.4 and 1.5.
- a public repository without the
pull_request_targetevent policy from 2026-12-02: see 2.4. - a
code-review.ymlthat isn't on the default branch yet, since both triggers run it from there: merge it to the default branch.
- a push starts nothing: comment
- Fix: the fix is the one listed with the cause.
- Symptom: the "Code review" check run is "Review failed" with "The trace contains something shaped like a credential (…)." (the same check names "The review", "The summary comment" or "The check run").
- Cause: before it publishes anything, the review refuses text shaped like an Anthropic API key, a GitHub token, a Bedrock API key, a PEM private key's BEGIN line or a JWT (
src/outputs.ts), and a test fixture in the diff is the usual source. - Fix: build the fixture from parts, as
test/enable-code-review.test.tsbuilds its key from five repeated dashes and the words around them, push, and comment/review(#233).