Repository navigation
Description
Problem
Setting up NCode Review takes manual steps across GitHub's UI, gh, the apps' settings pages and other repositories. Today they are spread over the README's "Using it", "Approvals" and "Opting a repository in" sections, tools/Enable-CodeReview.ps1's help, ADR-1009 and ADR-1011, and issue comments. Setting up the NCode Review Approver on 2026-11-21 (#232, NCodeGroup/project-template#460) showed what that costs:
- A stale clone ran an
Enable-CodeReview.ps1without-ApproverPrivateKeyPath. ghscopes were guessed:admin:org, thenread:user, thenuser, none of which lets gh's token list an installation's repositories (Warn and go on when GitHub refuses to list an app installation's repositories #236 makes the script warn and go on).- The approver's key was stored by hand with
gh secret set. - The question "do I delete the
.pem?" came up. - A required approval blocked this repository's own pull requests before the approver was live, because the order wasn't written down.
Change
Write one runbook, docs/setup.md or a README section the README's table of contents links, that a maintainer follows top to bottom without reading code. Give each step: who does it, where (a URL, or a command and the folder it runs from), what to check afterwards, and what goes wrong. Cover:
Once per organization
- Create and install the NCode Review app (permissions, webhook off, install scope), and keep its client ID as
app-client-id's default. - Create the NCode Review Approver app: Contents: write, Pull requests: write, Metadata: read, webhook off, installed on selected repositories only. Say why Contents: write is needed and is safe (Submit the approval as a dedicated approver app, so it meets a ruleset's required approval #232, ADR-1011).
- Set up Anthropic: the workspace, the service account and the federation rule that trusts only jobs in the
code-reviewenvironment. - Store the Bedrock key,
AWS_BEARER_TOKEN_BEDROCK, and the NCode Review app key,NCODE_REVIEW_APP_PRIVATE_KEY, as organization secrets withEnable-CodeReview.ps1. - List the
ghsetup the scripts need, and what each scope is for:gh auth refresh -h github.com -s admin:org. Also say that GitHub lists an app installation's repositories only to a GitHub App's user token or a personal access token, never togh auth login's token, so a repository is added to a selected-repositories installation on the app's settings page.
Once per repository
6. Add the repository to both apps' installations.
7. Run Enable-CodeReview.ps1 -Repo <name> -ApproverPrivateKeyPath <pem> from an up-to-date clone (git pull first), or set NCODE_REVIEW_APPROVER_PRIVATE_KEY in the code-review environment with gh secret set --env code-review and the key on standard input. Then check it with gh secret list --env code-review --repo <owner>/<name>.
8. code-review.yml: pin a release's full SHA, pass approve: true and approver-private-key: ${{ secrets.NCODE_REVIEW_APPROVER_PRIVATE_KEY }}, and put both inputs in the review job only.
9. Require the approval only after steps 6 to 8 are on the default branch, with the project template's tools/Set-RepositoryBaseline.ps1 -Repo <name>. It checks the input and the secret before it requires anything. For this repository, which reviews itself, give the bootstrap order: drop the required approval to 0, merge and release the version that passes the key, store the key, run the baseline again.
10. Check it: the next clean review's approval comes from ncode-review-approver[bot], and the merge box shows the requirement met. When it doesn't, check where the approval came from (ncode-review[bot] means no approver key or token), authorCanPushToRepository, and whether a later review dismissed it.
Keys
11. The private key: generate it, store it, then delete the local .pem. Say why: GitHub keeps the secret, a new key can be generated at any time, and a leaked approver key can approve on every repository the app is installed on. Rotate a key by generating a new one, storing it everywhere, then deleting the old one on the app's page.
How a review behaves
12. What a maintainer sees and does:
- Marking a pull request ready starts a review, which stops when the commit was already reviewed.
- A push needs a /review comment, first line only.
- A review that doesn't approve dismisses the earlier approval.
- Nits don't block, and a re-review can raise a finding's severity.
- A bot's pull request is skipped, and the maintainer approves it.
- The trace refuses to publish anything shaped like a credential, which a test fixture in the diff can trip (#233).
Troubleshooting
13. Each error met so far, with its fix: a missing -ApproverPrivateKeyPath parameter (stale clone), the user/installations/.../repositories 403, admin:org errors, a review that skips, and "Review failed: the trace contains something shaped like a credential".
Link it from the README's "Opting a repository in" and from NCodeGroup/project-template#468, the project template's end-to-end adoption runbook, which links here for NCode Review's steps. Keep the existing reference sections as the reference, and let the runbook be the procedure.
Refs #232, #236, NCodeGroup/project-template#460, NCodeGroup/project-template#468
Generated by Claude Code
Activity
polewskm commentedon Nov 21, 2026
Plan snapshot. The working copy is
PLAN.mdin the plan's checkout; this comment is not updated as steps run.Plan: Write the end-to-end NCode Review setup runbook (10 steps)
Plan: Write the end-to-end NCode Review setup runbook
Issue: #237
Refs: #232 #236 NCodeGroup/project-template#460 NCodeGroup/project-template#468
Goal
Setting up NCode Review takes manual steps spread over the README,
tools/Enable-CodeReview.ps1's help, ADR-1009, ADR-1011 and issue comments, and setting up the NCode Review Approver on 2026-11-21 showed what that costs. Write one runbook,docs/setup.md, that a maintainer follows top to bottom without reading code: once per organization, once per repository, the keys, how a review behaves, and troubleshooting. Link it from the README and the docs index; the README's sections stay the reference, the runbook is the procedure.Context for the executor
./.template/Use-Node.ps1npm cionce in this worktree before step 9 (its Done-when runsnode --test test/command.test.ts, which needs theyamlpackage). The main checkout already hasnode_modules.npm run typechecknpm testCHANGELOG.mdand the README's newer paragraphs do.<owner>/<name>(the repository being set up),<name>,<number>(a pull request),<path to the NCode Review .pem>,<path to the NCode Review Approver .pem>.###heading then five bullets, each bold label followed by a colon inside the bold:- **Symptom:**,- **Cause:**and- **Fix:**instead. A bullet with several items may hold a nested list, indented two spaces.docs/setup.mdare relative:../README.md#<anchor>; links to a decision record aredecisions/<file>.md; links to issues are full URLs, such ashttps://github.com/NCodeGroup/code-review/issues/232.src/outputs.ts(secretPatterns), and this pull request's own review readsdocs/setup.md. Never write text those patterns match: no PEM header line with only capital letters and spaces between-----BEGINandPRIVATE KEY-----, nosk-ant-followed by 8 or more characters, noABSK,ghp_,gho_,ghu_,ghs_,ghr_orgithub_pat_followed by 20 or more characters, no twoeyJ-prefixed dotted segments. Step 8 says how to describe them in words.test/command.test.tsreads the first```yamlblock after## Using itinREADME.md. Step 9 adds a sentence before that block and must not add another code block between the heading and it.[IO.File]::ReadAllText,[IO.File]::WriteAllTextor[IO.Directory]::GetFiles: .NET resolves it against the process's folder, whichSet-Locationdoesn't move and which in a Copilot session is the main checkout, not the plan's worktree. UseGet-Content -LiteralPath,Set-Content -LiteralPathorGet-ChildItem, or make the path absolute first withJoin-Path (Get-Location) <path>.Decisions
docs/setup.md, not a README section: the README has no table of contents,docs/holds the repository's documents anddocs/README.mdindexes each one, and the project template's runbook is the same kind of file (docs/adopt.mdin NCodeGroup/project-template).docs/adopt.md, without its Applies to (every step here applies to every NCode Review setup). Troubleshooting entries list Symptom, Cause and Fix.gh.gh api orgs/NCodeGroup/installationson 2026-11-21: NCode Review has Checks: write, Contents: read, Issues: write, Metadata: read and Pull requests: write, no webhook events, and is installed on all repositories; NCode Review Approver has Contents: write, Metadata: read and Pull requests: write, no webhook events, and is installed on selected repositories. So 2.1 adds a repository to the approver's installation only./wif-test(.github/workflows/wif-smoke-test.yml) is its check, because it proves both that acode-reviewjob gets a token and that a job without the environment is refused..pemwas deleted gets a newly generated approver key: an app can hold several active keys, so the other repositories keep theirs.repeats()insrc/threads.ts), so an undismissed nit can come back as a should-fix in a new thread.test/enable-code-review.test.ts).docs/README.md's Rules say.Release:line: the plan changes only repository docs, which ADR-1006 andAGENTS.md's Conventions say release nothing; its changelog entry goes under## [Unreleased].Out of scope
action.yml,tools/Enable-CodeReview.ps1, workflows or tests.docs/adopt.md, which links issue Document every manual step to set up NCode Review, end to end, in one runbook #237 for these steps; a session in that repository points it atdocs/setup.md.docs/README.mdtext.Steps
[ ] 1. Start docs/setup.md with its preamble and gh's scopes
docs/setup.md(new)# Set Up NCode Review, End to End, then a paragraph: this runbook sets NCode Review up for an organization and a repository, top to bottom, without reading code; README.md is the reference for each part (Using it, Interface, Approvals, Opting a repository in), and this page is the order to do them in; NCodeGroup/project-template's Adopt a Project, End to End covers the rest of a repository's adoption.## How to Read It: each step lists Who, Where, Do, Check and Goes wrong; commands run in PowerShell 7 from the folder the step's Where names;<owner>/<name>is the repository being set up and<number>a pull request; each entry in section 5 lists Symptom, Cause and Fix.## Contents: bullets[How to Read It](#how-to-read-it),[1. Once per Organization](#1-once-per-organization),[2. Once per Repository](#2-once-per-repository),[3. Keys](#3-keys),[4. How a Review Behaves](#4-how-a-review-behaves),[5. Troubleshooting](#5-troubleshooting).## 1. Once per Organization, a line "Do these once, in order.", then### 1.1 Give gh the Scopes the Scripts Need:gh auth login, thengh auth refresh -h github.com -s admin:org. What each scope is for:repo, whichgh auth logingives, covers the repository's environment and environment secrets and, with the admin role on the repository, project-template'stools/Set-RepositoryBaseline.ps1;admin:orgcovers reading and setting the organization secrets and choosing which repositories can read them, andtools/Enable-CodeReview.ps1stops without it. GitHub lists and changes an app installation's repositories only for a GitHub App's user token or a personal access token, never forgh auth login's token, whatever its scopes (read:user,userandadmin:orgdon't help), so a repository is added to a selected-repositories installation on the app's installation page (2.1), and the script warns and goes on (#236).gh auth statuslistsadmin:orgamong the token's scopes, andgh secret list --org NCodeGroupsucceeds.GH_TOKENorGITHUB_TOKENenvironment variable replaces the signed-in account's token and its scopes; see 5.3.$t = Get-Content -Raw -LiteralPath docs/setup.md; $need = '# Set Up NCode Review, End to End', '## How to Read It', '## Contents', '](#5-troubleshooting)', '## 1. Once per Organization', '### 1.1 Give gh the Scopes the Scripts Need', 'gh auth refresh -h github.com -s admin:org', 'personal access token', '- **Goes wrong:**'; $miss = $need | Where-Object { -not $t.Contains($_) }; if ($miss) { $miss; exit 1 }[ ] 2. Add the two apps to section 1
docs/setup.md(edit)### 1.2 Create and Install the NCode Review App. Who: an organization owner. Where:https://github.com/organizations/NCodeGroup/settings/apps/new(Organization settings > Developer settings > GitHub Apps > New GitHub App). Do: name it NCode Review (it posts asncode-review[bot]); clear Webhook > Active; repository permissions Checks: Read and write (the "Code review" check run), Contents: Read-only, Issues: Read and write (the issue/review teachopens), Metadata: Read-only, Pull requests: Read and write (threads, the summary comment, reactions and its approval); no other permission and no events; installable only on this account. Then Install App > NCodeGroup > All repositories, which covers every repository, NCodeGroup/code-review included, where/review teach orgopens its issues. Its client ID, on the app's General page, isIv23liAbZt44tgephpLC,app-client-id's default inaction.yml; an app with another ID needsapp-client-idin every caller, or the default changed in a release. Its private key is stored in 1.5. Check:gh api orgs/NCodeGroup/installations --jq '.installations[] | select(.app_slug == "ncode-review") | {repository_selection, permissions, events}'shows"repository_selection":"all", those five permissions and"events":[]. Goes wrong: an installation on selected repositories only must have each repository added, as 2.1 does for the approver.### 1.3 Create the NCode Review Approver App. Who and Where: as in 1.2. Do: name it NCode Review Approver (it approves asncode-review-approver[bot]); clear Webhook > Active; repository permissions Contents: Read and write, Metadata: Read-only, Pull requests: Read and write, nothing else, no events; installable only on this account; Install App > NCodeGroup > Only select repositories, and select only the repositories that require the approval. Its client ID isIv23liDPFX1K6NjGIr9i,approver-client-id's default. Why Contents: write is needed: GitHub counts an approval toward a ruleset's required approval only from a reviewer that can write to the repository, and the NCode Review app can't (#232). Why it is safe: the action mints the approver's token after the agents exit, for the one repository, with Pull requests: write alone, so no token it holds can push, and its key is a secret of thecode-reviewenvironment, which only the default branch can deploy to (ADR-1011). Check: the 1.2 command with"ncode-review-approver"shows"repository_selection":"selected",{"contents":"write","metadata":"read","pull_requests":"write"}and"events":[]. Goes wrong: installing it on all repositories widens what a leaked key reaches, since a leaked approver key can mint a Contents: write token for every repository it is installed on.### 1.1block.$t = Get-Content -Raw -LiteralPath docs/setup.md; $need = '### 1.2 Create and Install the NCode Review App', '### 1.3 Create the NCode Review Approver App', 'Iv23liAbZt44tgephpLC', 'Iv23liDPFX1K6NjGIr9i', 'Contents: Read and write', '"repository_selection":"selected"', '](decisions/1011-the-approval-is-submitted-by-an-approver-app-that-can-write.md)'; $miss = $need | Where-Object { -not $t.Contains($_) }; if ($miss) { $miss; exit 1 }[ ] 3. Add Anthropic, Bedrock and the organization secrets to section 1
docs/setup.md(edit)### 1.4 Set Up Anthropic. Who: an admin of the Anthropic organization. Where: the Claude Console's Workload identity page,https://platform.claude.com/settings/workload-identity-federation(Connect workload > GitHub Actions), with Anthropic's guide Use WIF with GitHub Actions. Do: the review pays from a workspace and logs in with no secret, through workload identity federation: register the issuerhttps://token.actions.githubusercontent.com; create the workspace that pays for reviews and a service account; create a federation rule that maps to that service account and workspace and matches only jobs in a NCodeGroup repository'scode-reviewenvironment, whose GitHub token has the audiencehttps://api.anthropic.comand the subjectrepo:NCodeGroup/<name>:environment:code-review, so match the audience, the subject prefixrepo:NCodeGroup/, and the claimsrepository_ownerNCodeGroupandenvironmentcode-review. NCodeGroup's are the action's defaults: rulegithub-code-review(federation-rule-idfdrl_019dXaBRa2Zo1UazugiQwWZ3),organization-ide9301a37-092f-47c8-9b24-188808b1d576,service-account-idsvac_012yav3w3rfhH5i9bAgoWLHe,workspace-idwrkspc_019f5RLk5SG4XWE8XKwWFHZh; other values need those inputs in every caller, or the defaults changed in a release. Check: comment/wif-teston an issue in NCodeGroup/code-review, as an owner or member; the WIF smoke test'swith-environmentjob gets a token and itswithout-environmentjob is refused. Goes wrong: a rule that doesn't require the environment lets any branch's workflow get a token, and thewithout-environmentjob fails with "The rule accepted a job outside the code-review environment"; a caller whose review job lacksenvironment: code-reviewgets no token, and the review pays on Bedrock or stops as "Review budget used up".### 1.5 Store the Organization Secrets. Who: an organization owner (admin:org), and an admin of the AWS account for the Bedrock key. Where: the AWS account's Amazon Bedrock console, inus-west-2; then an up-to-date clone of NCodeGroup/code-review (rungit pullfirst). Do: generate a long-term Bedrock API key (it starts withABSK) that can call the global inference profilesglobal.anthropic.claude-haiku-5-5,global.anthropic.claude-sonnet-5-5andglobal.anthropic.claude-opus-5-5inus-west-2; generate the NCode Review app's private key (3.1); run.\tools\Enable-CodeReview.ps1 -AppPrivateKeyPath <path to the NCode Review .pem> -WhatIf, then without-WhatIf. It storesNCODE_REVIEW_APP_PRIVATE_KEYand, when the organization has none yet,AWS_BEARER_TOKEN_BEDROCK, read from the environment variable of that name or a hidden prompt, as organization secrets visible only to the repositories that opted in, passing each value on standard input. Then delete the.pem(3.1). Check:gh secret list --org NCodeGrouplists both secrets;/wif-test(1.4) also makes a one-token Haiku call on Bedrock with the key and passes. Goes wrong: "That doesn't look like a Bedrock API key (they start with ABSK)."; "NCodeGroup has no NCODE_REVIEW_APP_PRIVATE_KEY yet. Pass -AppPrivateKeyPath with the app's .pem file."; "The NCode Review app isn't installed on NCodeGroup. Install it from the app's settings page first."; scope errors, 5.3.### 1.1block.$t = Get-Content -Raw -LiteralPath docs/setup.md; $need = '### 1.4 Set Up Anthropic', '### 1.5 Store the Organization Secrets', 'fdrl_019dXaBRa2Zo1UazugiQwWZ3', 'svac_012yav3w3rfhH5i9bAgoWLHe', 'wrkspc_019f5RLk5SG4XWE8XKwWFHZh', 'repo:NCodeGroup/<name>:environment:code-review', '/wif-test', '-AppPrivateKeyPath <path to the NCode Review .pem>', 'global.anthropic.claude-opus-5-5'; $miss = $need | Where-Object { -not $t.Contains($_) }; if ($miss) { $miss; exit 1 }[ ] 4. Start section 2 with the installations and the approver's key
docs/setup.md(edit)## 2. Once per Repository, a line "Do these for each repository, in order, after section 1.", then two steps.### 2.1 Add the Repository to Both Apps' Installations. Who: an organization owner. Where:https://github.com/organizations/NCodeGroup/settings/installations> Configure next to each app > Repository access. Do: NCode Review is installed on all repositories, so it needs nothing; for NCode Review Approver, choose the repository under Only select repositories and save, only where the approval will be required.Enable-CodeReview.ps1also tries to add it, but gh's token can't list the installation's repositories (1.1), so it warns and goes on. Check: the approver's installation page lists the repository. Goes wrong: without the approver's installation the action's approver token step fails, and the publish step warns and approves asncode-review[bot], whose approval meets no required approval (2.5).### 2.2 Store the Approver's Key. Who: an organization owner (admin:org: the script checks the organization secrets first). Where: an up-to-date clone of NCodeGroup/code-review: rungit pullfirst, since a clone from before v1.5.0 has no-ApproverPrivateKeyPath(5.1). Do:.\tools\Enable-CodeReview.ps1 -Repo <name> -ApproverPrivateKeyPath <path to the NCode Review Approver .pem> -WhatIf, then without-WhatIf; several repositories as one value,-Repo a,b. It creates or updates thecode-reviewenvironment so only the default branch can deploy to it, deletes environment secrets named like the organization secrets (they would hide them), gives the repository both organization secrets, and storesNCODE_REVIEW_APPROVER_PRIVATE_KEYas a secret of that environment, never of the organization, so only default-branch jobs can read it. Without the script, the key alone goes in withGet-Content -Raw -LiteralPath <path to the NCode Review Approver .pem> | gh secret set NCODE_REVIEW_APPROVER_PRIVATE_KEY --env code-review --repo <owner>/<name>, on standard input, never on the command line. Then delete the.pem(3.1); a repository opted in later gets a newly generated key, since the app can hold several, and the others keep theirs. Check:gh secret list --env code-review --repo <owner>/<name>listsNCODE_REVIEW_APPROVER_PRIVATE_KEY, andgh api orgs/NCodeGroup/actions/secrets/NCODE_REVIEW_APP_PRIVATE_KEY/repositories --jq '.repositories[].full_name'lists<owner>/<name>. Goes wrong: 5.1, 5.2, 5.3.### 1.1block.$t = Get-Content -Raw -LiteralPath docs/setup.md; $need = '## 2. Once per Repository', "### 2.1 Add the Repository to Both Apps' Installations", "### 2.2 Store the Approver's Key", '-ApproverPrivateKeyPath <path to the NCode Review Approver .pem> -WhatIf', 'gh secret list --env code-review --repo <owner>/<name>', 'gh secret set NCODE_REVIEW_APPROVER_PRIVATE_KEY --env code-review --repo <owner>/<name>'; $miss = $need | Where-Object { -not $t.Contains($_) }; if ($miss) { $miss; exit 1 }[ ] 5. Add the workflow's inputs and the required approval to section 2
docs/setup.md(edit)### 2.3 Pass the Approval Inputs in code-review.yml. Who: a maintainer of the repository. Where:.github/workflows/code-review.yml, changed in a pull request (both triggers run the file from the default branch). Do: start from Using it, or project-template'scode_reviewanswer. Pin a release's full commit SHA with its tag in a comment, never a tag or a branch (Versions); the approver needs v1.5.0 or later:NCodeGroup/code-review@a4cdf65d2450e4e1b2b92ed53430ceb8026d4f97 # v1.5.0, andgh api repos/NCodeGroup/code-review/commits/<tag> --jq .shaprints a release's SHA. In thereviewjob'swith:only, which keepsenvironment: code-review, add a fencedyamlblock withapprove: trueandapprover-private-key: ${{ secrets.NCODE_REVIEW_APPROVER_PRIVATE_KEY }}. Never put them in theoutcomesjob: it runs outside the environment, so it can't read the key, and it never approves. Check: once merged,gh api repos/<owner>/<name>/contents/.github/workflows/code-review.yml -H "Accept: application/vnd.github.raw"shows both lines in the review job. Goes wrong: merged before 2.2, the input is empty and NCode Review approves as its own app, which is harmless until 2.4.### 2.4 Require the Approval. Who: an admin of the repository (repo). Where: an up-to-date clone of NCodeGroup/project-template (rungit pullfirst). Do: only once 2.1 to 2.3 are on the default branch, run.\tools\Set-RepositoryBaseline.ps1 -Repo <name> -WhatIf, with the-RequireChecksvalue project-template's Run the Baseline gives, then without-WhatIf. It reads the default branch'scode-review.ymland the environment secret before it requires anything: withapprove: true, a non-emptyapprover-private-keyand the secret, it requires one approval, of the last push, and prints "NCode Review approves as the NCode Review Approver, which can write, so the ruleset requires one approval, of the last push"; without the secret it prints that the environment has noNCODE_REVIEW_APPROVER_PRIVATE_KEYand requires no approval for it. It also creates thepull_request_targetevent policy a public repository needs from 2026-12-02. For NCodeGroup/code-review, which reviews itself with a released pin, the order is a numbered list: set the "Protect main" ruleset's required approvals to 0 (Settings > Rules > Rulesets > Protect main); merge and release the version that passes the key and pin it; store the key (2.2); run the baseline again. Check:gh api repos/<owner>/<name>/rules/branches/main --jq '.[] | select(.type == "pull_request") | .parameters.required_approving_review_count'prints1. Goes wrong: an approval required before the approver is live blocks every merge, which is what blocked this repository's pull requests on 2026-11-21.### 1.1block.$t = Get-Content -Raw -LiteralPath docs/setup.md; $need = '### 2.3 Pass the Approval Inputs in code-review.yml', '### 2.4 Require the Approval', 'a4cdf65d2450e4e1b2b92ed53430ceb8026d4f97 # v1.5.0', 'approver-private-key: ${{ secrets.NCODE_REVIEW_APPROVER_PRIVATE_KEY }}', 'Set-RepositoryBaseline.ps1 -Repo <name> -WhatIf', 'required_approving_review_count', 'Protect main'; $miss = $need | Where-Object { -not $t.Contains($_) }; if ($miss) { $miss; exit 1 }[ ] 6. Add the approval's check and section 3, the keys
docs/setup.md(edit)### 2.5 Check the Approval. Who: a maintainer. Where: the repository's next pull request. Do: mark it ready, or comment/review, and let the review finish. Check: after a clean review,gh api repos/<owner>/<name>/pulls/<number>/reviews --jq '.[] | .user.login + " " + .state'showsncode-review-approver[bot] APPROVED, andgh pr view <number> --repo <owner>/<name> --json reviewDecisionshowsAPPROVED; the merge box shows the requirement met. Goes wrong: an approval fromncode-review[bot]means no approver key reached the review (2.2, 2.3, a pin older than v1.5.0) or the approver's token step failed, which the run's log warns about (2.1); when the approver's review doesn't count,gh api graphql -f query='query($o:String!,$n:String!,$p:Int!){repository(owner:$o,name:$n){pullRequest(number:$p){reviews(last:20){nodes{author{login} state authorCanPushToRepository}}}}}' -f o=<owner> -f n=<name> -F p=<number>showsauthorCanPushToRepositoryfalse when the approver lacks Contents: write there (1.3); a missing approval was dismissed by a later review that didn't approve, or by a push. The summary comment'sApproval:line says why.## 3. Keys, then### 3.1 Store a Key, Then Delete the .pem. Who: an organization owner. Where: the app's settings page,https://github.com/organizations/NCodeGroup/settings/apps/ncode-reviewor.../ncode-review-approver, Private keys; and the clone from 1.5. Do: Generate a private key, which downloads a.pem; store it, the NCode Review app's with-AppPrivateKeyPath(1.5) and the approver's with-Repo <name> -ApproverPrivateKeyPath(2.2); thenRemove-Item -LiteralPath <path to the .pem>. Why: GitHub keeps the secret and gives it only to the workflows; a new key can be generated at any time; and a leaked approver key can approve on every repository the approver is installed on. Check:Test-Path -LiteralPath <path to the .pem>printsFalse. Goes wrong: a copy kept in Downloads or a backup outlives the deletion.### 3.2 Rotate a Key. Who and Where: as in 3.1. Do, as a numbered list: generate a new key; store it everywhere (the approver's with-Reponaming every repository its installation lists); delete the.pem; only then delete the old key on the app's page. The Bedrock key rotates with.\tools\Enable-CodeReview.ps1 -UpdateBedrockKey, then the old key is revoked in AWS. Check: the app's page lists only the new key, and 2.5's check passes on the next review. Goes wrong: deleting the old key first stops every repository that still has it from approving, or, for the NCode Review app, from posting.### 1.1block.$t = Get-Content -Raw -LiteralPath docs/setup.md; $need = '### 2.5 Check the Approval', 'ncode-review-approver[bot] APPROVED', 'authorCanPushToRepository', '## 3. Keys', '### 3.1 Store a Key, Then Delete the .pem', '### 3.2 Rotate a Key', '-UpdateBedrockKey'; $miss = $need | Where-Object { -not $t.Contains($_) }; if ($miss) { $miss; exit 1 }[ ] 7. Add section 4, how a review behaves
docs/setup.md(edit)## 4. How a Review Behaves, a line "What a maintainer sees and does once the repository is set up; Commands and Approvals are the reference.", then two subsections of plain bullets (no Who/Where labels):### 4.1 Starting a Review: opening, reopening or marking a pull request ready starts a review, drafts included; an automatic run on a commit already reviewed stops, and its log says " was already reviewed; comment /review to review it again."; a push starts nothing, so comment/reviewas a top-level comment whose first line is exactly/review(lines below it, such as a---footer, are fine); only people with write access can, and a private organization membership counts as a contributor, so make it public or be added as a collaborator; a pull request a bot opens isn't reviewed when it opens, so a maintainer approves Dependabot's where an approval is required; theskip-reviewlabel skips the automatic reviews; pull requests from forks are never reviewed; after three reviews with no comment from a writer, an automatic review posts "Paused: comment/reviewto resume".### 4.2 Findings and the Approval: withapprove: true, a clean review approves and every other review dismisses the earlier approval; a push dismisses it when the ruleset dismisses stale approvals, so comment/reviewafter a push; nits don't block: a review whose only findings are nits approves when every reviewer ran in full and verification lowered or refuted nothing; a later review can raise a problem's severity, since a finding repeats an earlier thread only at the same severity unless a writer dismissed that thread (resolved it while still on its code, or a 👎), so an unanswered nit can come back as a should-fix in a new thread; everything a review publishes, its trace included, is checked for anything shaped like a credential, which a test fixture in the diff can trip, and the review then fails (#233, 5.5). Link ADR-1009 after the approval bullets.## CommandsinREADME.md.$t = Get-Content -Raw -LiteralPath docs/setup.md; $need = '## 4. How a Review Behaves', '### 4.1 Starting a Review', '### 4.2 Findings and the Approval', 'was already reviewed; comment /review to review it again.', 'skip-review', 'Paused: comment/reviewto resume', 'same severity', '](decisions/1009-a-clean-review-approves-the-pull-request.md)'; $miss = $need | Where-Object { -not $t.Contains($_) }; if ($miss) { $miss; exit 1 }[ ] 8. Add section 5, troubleshooting, and check every link
docs/setup.md(edit)## 5. Troubleshooting, then five entries with Symptom, Cause and Fix:### 5.1 A Missing ApproverPrivateKeyPath Parameter: Symptom "A parameter cannot be found that matches parameter name 'ApproverPrivateKeyPath'."; Cause: the clone'sEnable-CodeReview.ps1is older than v1.5.0; Fix:git pullin the clone and run it again.### 5.2 A 403 Listing an Installation's Repositories: Symptom: a warning "Couldn't list the repositories of the NCode Review Approver installation (…): GitHub lists them only for a GitHub App's user token or a personal access token. Make sure / is selected at https://github.com/organizations/NCodeGroup/settings/installations/.", or, from a clone older than #236, the run stops on HTTP 403 fromuser/installations/<id>/repositories; Cause: gh's token can't list an installation's repositories, whatever its scopes; Fix: select the repository on the page the warning names (2.1), andgit pullan older clone; adding scopes doesn't help.### 5.3 admin:org Errors: Symptom "gh can't read NCodeGroup's organization secrets. Grant the scope with: gh auth refresh -h github.com -s admin:org", or an error ending "If it's a 403, the token needs the admin:org scope: gh auth refresh -h github.com -s admin:org"; Cause: the token lacksadmin:org, or aGH_TOKENorGITHUB_TOKENvariable replaces it; Fix: run that command, or remove the variable, and check withgh auth status.### 5.4 A Review That Doesn't Start or Skips: Symptom: no 👀 and no "Code review" check run, or a check run that ends without findings; Causes and fixes as a nested list: a push (comment/review); a commit already reviewed (/review); a comment that isn't top-level, doesn't have/reviewalone on its first line, or comes from someone without write access (4.1); a bot's pull request, a fork or theskip-reviewlabel (4.1); "Paused" (comment/review); "Review budget used up", when the plan's API credits are used up or the Anthropic login failed and the Bedrock key is missing, refused or throttled (1.4, 1.5); a public repository without thepull_request_targetevent policy from 2026-12-02 (2.4); acode-review.ymlthat isn't on the default branch yet, since both triggers run it from there.### 5.5 The Trace Contains Something Shaped Like a Credential: Symptom: the "Code review" check run is "Review failed" with "The trace contains something shaped like a credential (…)." (the same check names "The review", "The summary comment" or "The check run"); Cause: before it publishes anything, the review refuses text shaped like an Anthropic API key, a GitHub token, a Bedrock API key, a PEM private key's BEGIN line or a JWT (src/outputs.ts), and a test fixture in the diff is the usual source; Fix: build the fixture from parts, astest/enable-code-review.test.tsbuilds its key from five repeated dashes and the words around them, push, and comment/review(#233). Describe the shapes only in these words, never with an example (see Gotchas).### 1.1block, with Symptom, Cause and Fix as its labels.$t = Get-Content -Raw -LiteralPath docs/setup.md; $need = '## 5. Troubleshooting', '### 5.1 A Missing ApproverPrivateKeyPath Parameter', "### 5.2 A 403 Listing an Installation's Repositories", '### 5.3 admin:org Errors', "### 5.4 A Review That Doesn't Start or Skips", '### 5.5 The Trace Contains Something Shaped Like a Credential', "A parameter cannot be found that matches parameter name 'ApproverPrivateKeyPath'.", '- **Symptom:**', '- **Fix:**'; $miss = @($need | Where-Object { -not $t.Contains($_) }); if ('-----BEGIN', 'sk-ant-', 'eyJ' | Where-Object { $t.Contains($_) }) { $miss += 'credential-shaped text' }; $slug = { param($h) (($h.Trim().ToLowerInvariant() -replace '[^a-z0-9 _-]', '') -replace ' ', '-') }; $heads = { param($f) Select-String -LiteralPath $f -Pattern '^#{1,6} (.+)$' | ForEach-Object { & $slug $_.Matches[0].Groups[1].Value } }; $own = & $heads docs/setup.md; $readme = & $heads README.md; foreach ($m in [regex]::Matches($t, '\]\(([^)\s]+)\)')) { $u = $m.Groups[1].Value; if ($u -match '^https?://') { continue }; $p, $a = $u -split '#', 2; if ($p -and -not (Test-Path -LiteralPath (Join-Path docs $p))) { $miss += $u; continue }; if ($a -and -not $p -and $a -notin $own) { $miss += $u }; if ($a -and $p -eq '../README.md' -and $a -notin $readme) { $miss += $u } }; if ($miss) { $miss; exit 1 }[ ] 9. Link the runbook from the README and the docs index
README.md(edit),docs/README.md(edit)README.md,## Using it: insert, between the heading and the```yamlblock, one paragraph followed by a blank line: "To set NCode Review up for an organization or a repository, follow docs/setup.md, the runbook that orders these steps; this README is their reference."README.md,## Opting a repository in: after its first paragraph (ending`gh auth refresh -h github.com -s admin:org`.), insert the paragraph "docs/setup.md gives every step in order, with what to check and what goes wrong; this section and Approvals are their reference." In thepowershellblock, after the-SkipBedrockexample and its blank line, insert two lines:# Store the NCode Review Approver's key in each repository's code-review environment, and add it to the approver's installation.and.\tools\Enable-CodeReview.ps1 -Repo identity-platform -ApproverPrivateKeyPath <path to the NCode Review Approver .pem>, then a blank line.docs/README.md,## What Lives Where: insert, after thebest-practices-criteria.mdbullet and before thedecisions/bullet,- [setup.md](setup.md): the runbook that sets NCode Review up for the organization and for each repository, top to bottom, with its troubleshooting., wrapped at 120 characters with two-space continuation indent, as the other bullets there are.docs/README.md.$r = Get-Content -Raw -LiteralPath README.md; $d = Get-Content -Raw -LiteralPath docs/README.md; if (([regex]::Matches($r, '\]\(docs/setup\.md\)')).Count -lt 2 -or -not $r.Contains('-ApproverPrivateKeyPath <path to the NCode Review Approver .pem>') -or -not $d.Contains('[setup.md](setup.md)')) { exit 1 }; node --test test/command.test.ts[ ] 10. Record the runbook in the changelog
CHANGELOG.md(edit)## [Unreleased], insert a### Addedheading before the existing### Changedheading, with a blank line after the heading and after the entry, holding this entry:-docs/setup.mdis the runbook that sets NCode Review up end to end: once per organization (gh's scopes, the NCode Review and NCode Review Approver apps, Anthropic's workload identity federation and the organization secrets), once per repository (the approver's installation and key,code-review.yml's approval inputs, requiring the approval and checking it), the private keys, how a review behaves, and the errors met so far with their fixes;README.mdanddocs/README.mdlink it ([#237](https://github.com/NCodeGroup/code-review/issues/237)).Change nothing else in the file.
### Changedin## [Unreleased].$t = Get-Content -Raw -LiteralPath CHANGELOG.md; $u = $t.Substring(0, $t.IndexOf('## [1.5.0]')); if (([regex]::Matches($u, '(?m)^### Added$')).Count -ne 1 -or $u.IndexOf('### Added') -gt $u.IndexOf('### Changed') -or -not $u.Contains('-docs/setup.mdis the runbook') -or -not $u.Contains('issues/237')) { exit 1 }Review checklist
docs/setup.mdmatches its source: messages matchtools/Enable-CodeReview.ps1,src/command.tsandsrc/outputs.ts; IDs and inputs matchaction.yml; permissions matchgh api orgs/NCodeGroup/installations; the baseline's behavior matches NCodeGroup/project-template'stools/Set-RepositoryBaseline.ps1.src/outputs.ts, so this pull request's own review can publish.test/command.test.tspasses.Follow-ups
README.md's Using it workflow pins v1.0.0 (d14405f…) and passes neitherapprove: truenorapprover-private-key, and its Interface section's "A caller needs" list leaves out the optionalNCODE_REVIEW_APPROVER_PRIVATE_KEYenvironment secret, so the reference lagsdocs/setup.md's step 2.3; bring both to the current release.