Skip to content

Document every manual step to set up NCode Review, end to end, in one runbook #237

Closed

Description

@polewskm

Problem

Setting up NCode Review takes manual steps across GitHub's UI, gh, the apps' settings pages and other repositories. Today they are spread over the README's "Using it", "Approvals" and "Opting a repository in" sections, tools/Enable-CodeReview.ps1's help, ADR-1009 and ADR-1011, and issue comments. Setting up the NCode Review Approver on 2026-11-21 (#232, NCodeGroup/project-template#460) showed what that costs:

  • A stale clone ran an Enable-CodeReview.ps1 without -ApproverPrivateKeyPath.
  • gh scopes were guessed: admin:org, then read:user, then user, none of which lets gh's token list an installation's repositories (Warn and go on when GitHub refuses to list an app installation's repositories #236 makes the script warn and go on).
  • The approver's key was stored by hand with gh secret set.
  • The question "do I delete the .pem?" came up.
  • A required approval blocked this repository's own pull requests before the approver was live, because the order wasn't written down.

Change

Write one runbook, docs/setup.md or a README section the README's table of contents links, that a maintainer follows top to bottom without reading code. Give each step: who does it, where (a URL, or a command and the folder it runs from), what to check afterwards, and what goes wrong. Cover:

Once per organization

  1. Create and install the NCode Review app (permissions, webhook off, install scope), and keep its client ID as app-client-id's default.
  2. Create the NCode Review Approver app: Contents: write, Pull requests: write, Metadata: read, webhook off, installed on selected repositories only. Say why Contents: write is needed and is safe (Submit the approval as a dedicated approver app, so it meets a ruleset's required approval #232, ADR-1011).
  3. Set up Anthropic: the workspace, the service account and the federation rule that trusts only jobs in the code-review environment.
  4. Store the Bedrock key, AWS_BEARER_TOKEN_BEDROCK, and the NCode Review app key, NCODE_REVIEW_APP_PRIVATE_KEY, as organization secrets with Enable-CodeReview.ps1.
  5. List the gh setup the scripts need, and what each scope is for: gh auth refresh -h github.com -s admin:org. Also say that GitHub lists an app installation's repositories only to a GitHub App's user token or a personal access token, never to gh auth login's token, so a repository is added to a selected-repositories installation on the app's settings page.

Once per repository
6. Add the repository to both apps' installations.
7. Run Enable-CodeReview.ps1 -Repo <name> -ApproverPrivateKeyPath <pem> from an up-to-date clone (git pull first), or set NCODE_REVIEW_APPROVER_PRIVATE_KEY in the code-review environment with gh secret set --env code-review and the key on standard input. Then check it with gh secret list --env code-review --repo <owner>/<name>.
8. code-review.yml: pin a release's full SHA, pass approve: true and approver-private-key: ${{ secrets.NCODE_REVIEW_APPROVER_PRIVATE_KEY }}, and put both inputs in the review job only.
9. Require the approval only after steps 6 to 8 are on the default branch, with the project template's tools/Set-RepositoryBaseline.ps1 -Repo <name>. It checks the input and the secret before it requires anything. For this repository, which reviews itself, give the bootstrap order: drop the required approval to 0, merge and release the version that passes the key, store the key, run the baseline again.
10. Check it: the next clean review's approval comes from ncode-review-approver[bot], and the merge box shows the requirement met. When it doesn't, check where the approval came from (ncode-review[bot] means no approver key or token), authorCanPushToRepository, and whether a later review dismissed it.

Keys
11. The private key: generate it, store it, then delete the local .pem. Say why: GitHub keeps the secret, a new key can be generated at any time, and a leaked approver key can approve on every repository the app is installed on. Rotate a key by generating a new one, storing it everywhere, then deleting the old one on the app's page.

How a review behaves
12. What a maintainer sees and does:
- Marking a pull request ready starts a review, which stops when the commit was already reviewed.
- A push needs a /review comment, first line only.
- A review that doesn't approve dismisses the earlier approval.
- Nits don't block, and a re-review can raise a finding's severity.
- A bot's pull request is skipped, and the maintainer approves it.
- The trace refuses to publish anything shaped like a credential, which a test fixture in the diff can trip (#233).

Troubleshooting
13. Each error met so far, with its fix: a missing -ApproverPrivateKeyPath parameter (stale clone), the user/installations/.../repositories 403, admin:org errors, a review that skips, and "Review failed: the trace contains something shaped like a credential".

Link it from the README's "Opting a repository in" and from NCodeGroup/project-template#468, the project template's end-to-end adoption runbook, which links here for NCode Review's steps. Keep the existing reference sections as the reference, and let the runbook be the procedure.

Refs #232, #236, NCodeGroup/project-template#460, NCodeGroup/project-template#468


Generated by Claude Code

Activity

  1. polewskm commented on Nov 21, 2026

    @polewskm
    ContributorAuthor

    Plan snapshot. The working copy is PLAN.md in the plan's checkout; this comment is not updated as steps run.

    Plan: Write the end-to-end NCode Review setup runbook (10 steps)

    Plan: Write the end-to-end NCode Review setup runbook

    Issue: #237
    Refs: #232 #236 NCodeGroup/project-template#460 NCodeGroup/project-template#468

    Goal

    Setting up NCode Review takes manual steps spread over the README, tools/Enable-CodeReview.ps1's help, ADR-1009, ADR-1011 and issue comments, and setting up the NCode Review Approver on 2026-11-21 showed what that costs. Write one runbook, docs/setup.md, that a maintainer follows top to bottom without reading code: once per organization, once per repository, the keys, how a review behaves, and troubleshooting. Link it from the README and the docs index; the README's sections stay the reference, the runbook is the procedure.

    Context for the executor

    • Prelude: ./.template/Use-Node.ps1
    • Setup: run npm ci once in this worktree before step 9 (its Done-when runs node --test test/command.test.ts, which needs the yaml package). The main checkout already has node_modules.
    • Build: npm run typecheck
    • Test: npm test
    • Conventions:
      • Markdown, UTF-8 without BOM, LF line endings, a final newline. Write one line per paragraph or bullet, with no hard wrap, as CHANGELOG.md and the README's newer paragraphs do.
      • Write the facts each step gives, in plain sentences; don't add facts the step doesn't give. Keep every command, message, ID, URL and heading exactly as written in the step, including backticks.
      • Placeholders: <owner>/<name> (the repository being set up), <name>, <number> (a pull request), <path to the NCode Review .pem>, <path to the NCode Review Approver .pem>.
      • A runbook step has this shape, a ### heading then five bullets, each bold label followed by a colon inside the bold:
        ### 2.5 Check the Approval
        
        - **Who:** a maintainer.
        - **Where:** the next pull request.
        - **Do:** ...
        - **Check:** ...
        - **Goes wrong:** ...
        A troubleshooting entry in section 5 has the bullets - **Symptom:**, - **Cause:** and - **Fix:** instead. A bullet with several items may hold a nested list, indented two spaces.
      • Links to the README from docs/setup.md are relative: ../README.md#<anchor>; links to a decision record are decisions/<file>.md; links to issues are full URLs, such as https://github.com/NCodeGroup/code-review/issues/232.
    • Gotchas:
      • Everything NCode Review publishes, the trace included, is checked against the patterns in src/outputs.ts (secretPatterns), and this pull request's own review reads docs/setup.md. Never write text those patterns match: no PEM header line with only capital letters and spaces between -----BEGIN and PRIVATE KEY-----, no sk-ant- followed by 8 or more characters, no ABSK, ghp_, gho_, ghu_, ghs_, ghr_ or github_pat_ followed by 20 or more characters, no two eyJ-prefixed dotted segments. Step 8 says how to describe them in words.
      • test/command.test.ts reads the first ```yaml block after ## Using it in README.md. Step 9 adds a sentence before that block and must not add another code block between the heading and it.
      • A script or command never passes a relative path to a .NET method, such as [IO.File]::ReadAllText, [IO.File]::WriteAllText or [IO.Directory]::GetFiles: .NET resolves it against the process's folder, which Set-Location doesn't move and which in a Copilot session is the main checkout, not the plan's worktree. Use Get-Content -LiteralPath, Set-Content -LiteralPath or Get-ChildItem, or make the path absolute first with Join-Path (Get-Location) <path>.

    Decisions

    • The runbook is a new file, docs/setup.md, not a README section: the README has no table of contents, docs/ holds the repository's documents and docs/README.md indexes each one, and the project template's runbook is the same kind of file (docs/adopt.md in NCodeGroup/project-template).
    • Each step lists Who, Where, Do, Check and Goes wrong, the shape of project-template's docs/adopt.md, without its Applies to (every step here applies to every NCode Review setup). Troubleshooting entries list Symptom, Cause and Fix.
    • gh's scopes come first (1.1), not fifth as the issue lists them, because every later step's check runs gh.
    • The apps' permissions are the ones their NCodeGroup installations hold, read with gh api orgs/NCodeGroup/installations on 2026-11-21: NCode Review has Checks: write, Contents: read, Issues: write, Metadata: read and Pull requests: write, no webhook events, and is installed on all repositories; NCode Review Approver has Contents: write, Metadata: read and Pull requests: write, no webhook events, and is installed on selected repositories. So 2.1 adds a repository to the approver's installation only.
    • The Anthropic step says what the federation rule must match and names the action's default IDs; /wif-test (.github/workflows/wif-smoke-test.yml) is its check, because it proves both that a code-review job gets a token and that a job without the environment is refused.
    • A repository opted in after the approver's .pem was deleted gets a newly generated approver key: an app can hold several active keys, so the other repositories keep theirs.
    • "A re-review can raise a finding's severity" is documented as built: a finding repeats an earlier thread only at the same severity, unless a writer dismissed that thread (repeats() in src/threads.ts), so an undismissed nit can come back as a should-fix in a new thread.
    • The credential refusal's fix is the one Approve as a dedicated approver app #233 used: build the fixture from parts (test/enable-code-review.test.ts).
    • The runbook links the README's sections and the decision records instead of restating them, as docs/README.md's Rules say.
    • No Release: line: the plan changes only repository docs, which ADR-1006 and AGENTS.md's Conventions say release nothing; its changelog entry goes under ## [Unreleased].

    Out of scope

    • Any change to code, action.yml, tools/Enable-CodeReview.ps1, workflows or tests.
    • The README's Using it example and Interface lists, beyond the one sentence step 9 adds (see Follow-ups).
    • NCodeGroup/project-template's docs/adopt.md, which links issue Document every manual step to set up NCode Review, end to end, in one runbook #237 for these steps; a session in that repository points it at docs/setup.md.
    • A decision record: the runbook decides nothing new.
    • Re-wrapping or rewording any existing README or docs/README.md text.

    Steps

    [ ] 1. Start docs/setup.md with its preamble and gh's scopes

    • Files: docs/setup.md (new)
    • Change: Write, in order:
      • # Set Up NCode Review, End to End, then a paragraph: this runbook sets NCode Review up for an organization and a repository, top to bottom, without reading code; README.md is the reference for each part (Using it, Interface, Approvals, Opting a repository in), and this page is the order to do them in; NCodeGroup/project-template's Adopt a Project, End to End covers the rest of a repository's adoption.
      • ## How to Read It: each step lists Who, Where, Do, Check and Goes wrong; commands run in PowerShell 7 from the folder the step's Where names; <owner>/<name> is the repository being set up and <number> a pull request; each entry in section 5 lists Symptom, Cause and Fix.
      • ## Contents: bullets [How to Read It](#how-to-read-it), [1. Once per Organization](#1-once-per-organization), [2. Once per Repository](#2-once-per-repository), [3. Keys](#3-keys), [4. How a Review Behaves](#4-how-a-review-behaves), [5. Troubleshooting](#5-troubleshooting).
      • ## 1. Once per Organization, a line "Do these once, in order.", then ### 1.1 Give gh the Scopes the Scripts Need:
        • Who: each maintainer who runs the scripts. Where: a PowerShell 7 prompt.
        • Do: gh auth login, then gh auth refresh -h github.com -s admin:org. What each scope is for: repo, which gh auth login gives, covers the repository's environment and environment secrets and, with the admin role on the repository, project-template's tools/Set-RepositoryBaseline.ps1; admin:org covers reading and setting the organization secrets and choosing which repositories can read them, and tools/Enable-CodeReview.ps1 stops without it. GitHub lists and changes an app installation's repositories only for a GitHub App's user token or a personal access token, never for gh auth login's token, whatever its scopes (read:user, user and admin:org don't help), so a repository is added to a selected-repositories installation on the app's installation page (2.1), and the script warns and goes on (#236).
        • Check: gh auth status lists admin:org among the token's scopes, and gh secret list --org NCodeGroup succeeds.
        • Goes wrong: a GH_TOKEN or GITHUB_TOKEN environment variable replaces the signed-in account's token and its scopes; see 5.3.
    • Pattern: the step shape in Context for the executor.
    • Done when: $t = Get-Content -Raw -LiteralPath docs/setup.md; $need = '# Set Up NCode Review, End to End', '## How to Read It', '## Contents', '](#5-troubleshooting)', '## 1. Once per Organization', '### 1.1 Give gh the Scopes the Scripts Need', 'gh auth refresh -h github.com -s admin:org', 'personal access token', '- **Goes wrong:**'; $miss = $need | Where-Object { -not $t.Contains($_) }; if ($miss) { $miss; exit 1 }

    [ ] 2. Add the two apps to section 1

    • Files: docs/setup.md (edit)
    • Change: Append two steps.
      • ### 1.2 Create and Install the NCode Review App. Who: an organization owner. Where: https://github.com/organizations/NCodeGroup/settings/apps/new (Organization settings > Developer settings > GitHub Apps > New GitHub App). Do: name it NCode Review (it posts as ncode-review[bot]); clear Webhook > Active; repository permissions Checks: Read and write (the "Code review" check run), Contents: Read-only, Issues: Read and write (the issue /review teach opens), Metadata: Read-only, Pull requests: Read and write (threads, the summary comment, reactions and its approval); no other permission and no events; installable only on this account. Then Install App > NCodeGroup > All repositories, which covers every repository, NCodeGroup/code-review included, where /review teach org opens its issues. Its client ID, on the app's General page, is Iv23liAbZt44tgephpLC, app-client-id's default in action.yml; an app with another ID needs app-client-id in every caller, or the default changed in a release. Its private key is stored in 1.5. Check: gh api orgs/NCodeGroup/installations --jq '.installations[] | select(.app_slug == "ncode-review") | {repository_selection, permissions, events}' shows "repository_selection":"all", those five permissions and "events":[]. Goes wrong: an installation on selected repositories only must have each repository added, as 2.1 does for the approver.
      • ### 1.3 Create the NCode Review Approver App. Who and Where: as in 1.2. Do: name it NCode Review Approver (it approves as ncode-review-approver[bot]); clear Webhook > Active; repository permissions Contents: Read and write, Metadata: Read-only, Pull requests: Read and write, nothing else, no events; installable only on this account; Install App > NCodeGroup > Only select repositories, and select only the repositories that require the approval. Its client ID is Iv23liDPFX1K6NjGIr9i, approver-client-id's default. Why Contents: write is needed: GitHub counts an approval toward a ruleset's required approval only from a reviewer that can write to the repository, and the NCode Review app can't (#232). Why it is safe: the action mints the approver's token after the agents exit, for the one repository, with Pull requests: write alone, so no token it holds can push, and its key is a secret of the code-review environment, which only the default branch can deploy to (ADR-1011). Check: the 1.2 command with "ncode-review-approver" shows "repository_selection":"selected", {"contents":"write","metadata":"read","pull_requests":"write"} and "events":[]. Goes wrong: installing it on all repositories widens what a leaked key reaches, since a leaked approver key can mint a Contents: write token for every repository it is installed on.
    • Pattern: step 1's ### 1.1 block.
    • Done when: $t = Get-Content -Raw -LiteralPath docs/setup.md; $need = '### 1.2 Create and Install the NCode Review App', '### 1.3 Create the NCode Review Approver App', 'Iv23liAbZt44tgephpLC', 'Iv23liDPFX1K6NjGIr9i', 'Contents: Read and write', '"repository_selection":"selected"', '](decisions/1011-the-approval-is-submitted-by-an-approver-app-that-can-write.md)'; $miss = $need | Where-Object { -not $t.Contains($_) }; if ($miss) { $miss; exit 1 }

    [ ] 3. Add Anthropic, Bedrock and the organization secrets to section 1

    • Files: docs/setup.md (edit)
    • Change: Append two steps.
      • ### 1.4 Set Up Anthropic. Who: an admin of the Anthropic organization. Where: the Claude Console's Workload identity page, https://platform.claude.com/settings/workload-identity-federation (Connect workload > GitHub Actions), with Anthropic's guide Use WIF with GitHub Actions. Do: the review pays from a workspace and logs in with no secret, through workload identity federation: register the issuer https://token.actions.githubusercontent.com; create the workspace that pays for reviews and a service account; create a federation rule that maps to that service account and workspace and matches only jobs in a NCodeGroup repository's code-review environment, whose GitHub token has the audience https://api.anthropic.com and the subject repo:NCodeGroup/<name>:environment:code-review, so match the audience, the subject prefix repo:NCodeGroup/, and the claims repository_owner NCodeGroup and environment code-review. NCodeGroup's are the action's defaults: rule github-code-review (federation-rule-id fdrl_019dXaBRa2Zo1UazugiQwWZ3), organization-id e9301a37-092f-47c8-9b24-188808b1d576, service-account-id svac_012yav3w3rfhH5i9bAgoWLHe, workspace-id wrkspc_019f5RLk5SG4XWE8XKwWFHZh; other values need those inputs in every caller, or the defaults changed in a release. Check: comment /wif-test on an issue in NCodeGroup/code-review, as an owner or member; the WIF smoke test's with-environment job gets a token and its without-environment job is refused. Goes wrong: a rule that doesn't require the environment lets any branch's workflow get a token, and the without-environment job fails with "The rule accepted a job outside the code-review environment"; a caller whose review job lacks environment: code-review gets no token, and the review pays on Bedrock or stops as "Review budget used up".
      • ### 1.5 Store the Organization Secrets. Who: an organization owner (admin:org), and an admin of the AWS account for the Bedrock key. Where: the AWS account's Amazon Bedrock console, in us-west-2; then an up-to-date clone of NCodeGroup/code-review (run git pull first). Do: generate a long-term Bedrock API key (it starts with ABSK) that can call the global inference profiles global.anthropic.claude-haiku-5-5, global.anthropic.claude-sonnet-5-5 and global.anthropic.claude-opus-5-5 in us-west-2; generate the NCode Review app's private key (3.1); run .\tools\Enable-CodeReview.ps1 -AppPrivateKeyPath <path to the NCode Review .pem> -WhatIf, then without -WhatIf. It stores NCODE_REVIEW_APP_PRIVATE_KEY and, when the organization has none yet, AWS_BEARER_TOKEN_BEDROCK, read from the environment variable of that name or a hidden prompt, as organization secrets visible only to the repositories that opted in, passing each value on standard input. Then delete the .pem (3.1). Check: gh secret list --org NCodeGroup lists both secrets; /wif-test (1.4) also makes a one-token Haiku call on Bedrock with the key and passes. Goes wrong: "That doesn't look like a Bedrock API key (they start with ABSK)."; "NCodeGroup has no NCODE_REVIEW_APP_PRIVATE_KEY yet. Pass -AppPrivateKeyPath with the app's .pem file."; "The NCode Review app isn't installed on NCodeGroup. Install it from the app's settings page first."; scope errors, 5.3.
    • Pattern: step 1's ### 1.1 block.
    • Done when: $t = Get-Content -Raw -LiteralPath docs/setup.md; $need = '### 1.4 Set Up Anthropic', '### 1.5 Store the Organization Secrets', 'fdrl_019dXaBRa2Zo1UazugiQwWZ3', 'svac_012yav3w3rfhH5i9bAgoWLHe', 'wrkspc_019f5RLk5SG4XWE8XKwWFHZh', 'repo:NCodeGroup/<name>:environment:code-review', '/wif-test', '-AppPrivateKeyPath <path to the NCode Review .pem>', 'global.anthropic.claude-opus-5-5'; $miss = $need | Where-Object { -not $t.Contains($_) }; if ($miss) { $miss; exit 1 }

    [ ] 4. Start section 2 with the installations and the approver's key

    • Files: docs/setup.md (edit)
    • Change: Append ## 2. Once per Repository, a line "Do these for each repository, in order, after section 1.", then two steps.
      • ### 2.1 Add the Repository to Both Apps' Installations. Who: an organization owner. Where: https://github.com/organizations/NCodeGroup/settings/installations > Configure next to each app > Repository access. Do: NCode Review is installed on all repositories, so it needs nothing; for NCode Review Approver, choose the repository under Only select repositories and save, only where the approval will be required. Enable-CodeReview.ps1 also tries to add it, but gh's token can't list the installation's repositories (1.1), so it warns and goes on. Check: the approver's installation page lists the repository. Goes wrong: without the approver's installation the action's approver token step fails, and the publish step warns and approves as ncode-review[bot], whose approval meets no required approval (2.5).
      • ### 2.2 Store the Approver's Key. Who: an organization owner (admin:org: the script checks the organization secrets first). Where: an up-to-date clone of NCodeGroup/code-review: run git pull first, since a clone from before v1.5.0 has no -ApproverPrivateKeyPath (5.1). Do: .\tools\Enable-CodeReview.ps1 -Repo <name> -ApproverPrivateKeyPath <path to the NCode Review Approver .pem> -WhatIf, then without -WhatIf; several repositories as one value, -Repo a,b. It creates or updates the code-review environment so only the default branch can deploy to it, deletes environment secrets named like the organization secrets (they would hide them), gives the repository both organization secrets, and stores NCODE_REVIEW_APPROVER_PRIVATE_KEY as a secret of that environment, never of the organization, so only default-branch jobs can read it. Without the script, the key alone goes in with Get-Content -Raw -LiteralPath <path to the NCode Review Approver .pem> | gh secret set NCODE_REVIEW_APPROVER_PRIVATE_KEY --env code-review --repo <owner>/<name>, on standard input, never on the command line. Then delete the .pem (3.1); a repository opted in later gets a newly generated key, since the app can hold several, and the others keep theirs. Check: gh secret list --env code-review --repo <owner>/<name> lists NCODE_REVIEW_APPROVER_PRIVATE_KEY, and gh api orgs/NCodeGroup/actions/secrets/NCODE_REVIEW_APP_PRIVATE_KEY/repositories --jq '.repositories[].full_name' lists <owner>/<name>. Goes wrong: 5.1, 5.2, 5.3.
    • Pattern: step 1's ### 1.1 block.
    • Done when: $t = Get-Content -Raw -LiteralPath docs/setup.md; $need = '## 2. Once per Repository', "### 2.1 Add the Repository to Both Apps' Installations", "### 2.2 Store the Approver's Key", '-ApproverPrivateKeyPath <path to the NCode Review Approver .pem> -WhatIf', 'gh secret list --env code-review --repo <owner>/<name>', 'gh secret set NCODE_REVIEW_APPROVER_PRIVATE_KEY --env code-review --repo <owner>/<name>'; $miss = $need | Where-Object { -not $t.Contains($_) }; if ($miss) { $miss; exit 1 }

    [ ] 5. Add the workflow's inputs and the required approval to section 2

    • Files: docs/setup.md (edit)
    • Change: Append two steps.
      • ### 2.3 Pass the Approval Inputs in code-review.yml. Who: a maintainer of the repository. Where: .github/workflows/code-review.yml, changed in a pull request (both triggers run the file from the default branch). Do: start from Using it, or project-template's code_review answer. Pin a release's full commit SHA with its tag in a comment, never a tag or a branch (Versions); the approver needs v1.5.0 or later: NCodeGroup/code-review@a4cdf65d2450e4e1b2b92ed53430ceb8026d4f97 # v1.5.0, and gh api repos/NCodeGroup/code-review/commits/<tag> --jq .sha prints a release's SHA. In the review job's with: only, which keeps environment: code-review, add a fenced yaml block with approve: true and approver-private-key: ${{ secrets.NCODE_REVIEW_APPROVER_PRIVATE_KEY }}. Never put them in the outcomes job: it runs outside the environment, so it can't read the key, and it never approves. Check: once merged, gh api repos/<owner>/<name>/contents/.github/workflows/code-review.yml -H "Accept: application/vnd.github.raw" shows both lines in the review job. Goes wrong: merged before 2.2, the input is empty and NCode Review approves as its own app, which is harmless until 2.4.
      • ### 2.4 Require the Approval. Who: an admin of the repository (repo). Where: an up-to-date clone of NCodeGroup/project-template (run git pull first). Do: only once 2.1 to 2.3 are on the default branch, run .\tools\Set-RepositoryBaseline.ps1 -Repo <name> -WhatIf, with the -RequireChecks value project-template's Run the Baseline gives, then without -WhatIf. It reads the default branch's code-review.yml and the environment secret before it requires anything: with approve: true, a non-empty approver-private-key and the secret, it requires one approval, of the last push, and prints "NCode Review approves as the NCode Review Approver, which can write, so the ruleset requires one approval, of the last push"; without the secret it prints that the environment has no NCODE_REVIEW_APPROVER_PRIVATE_KEY and requires no approval for it. It also creates the pull_request_target event policy a public repository needs from 2026-12-02. For NCodeGroup/code-review, which reviews itself with a released pin, the order is a numbered list: set the "Protect main" ruleset's required approvals to 0 (Settings > Rules > Rulesets > Protect main); merge and release the version that passes the key and pin it; store the key (2.2); run the baseline again. Check: gh api repos/<owner>/<name>/rules/branches/main --jq '.[] | select(.type == "pull_request") | .parameters.required_approving_review_count' prints 1. Goes wrong: an approval required before the approver is live blocks every merge, which is what blocked this repository's pull requests on 2026-11-21.
    • Pattern: step 1's ### 1.1 block.
    • Done when: $t = Get-Content -Raw -LiteralPath docs/setup.md; $need = '### 2.3 Pass the Approval Inputs in code-review.yml', '### 2.4 Require the Approval', 'a4cdf65d2450e4e1b2b92ed53430ceb8026d4f97 # v1.5.0', 'approver-private-key: ${{ secrets.NCODE_REVIEW_APPROVER_PRIVATE_KEY }}', 'Set-RepositoryBaseline.ps1 -Repo <name> -WhatIf', 'required_approving_review_count', 'Protect main'; $miss = $need | Where-Object { -not $t.Contains($_) }; if ($miss) { $miss; exit 1 }

    [ ] 6. Add the approval's check and section 3, the keys

    • Files: docs/setup.md (edit)
    • Change: Append one step, then a section.
      • ### 2.5 Check the Approval. Who: a maintainer. Where: the repository's next pull request. Do: mark it ready, or comment /review, and let the review finish. Check: after a clean review, gh api repos/<owner>/<name>/pulls/<number>/reviews --jq '.[] | .user.login + " " + .state' shows ncode-review-approver[bot] APPROVED, and gh pr view <number> --repo <owner>/<name> --json reviewDecision shows APPROVED; the merge box shows the requirement met. Goes wrong: an approval from ncode-review[bot] means no approver key reached the review (2.2, 2.3, a pin older than v1.5.0) or the approver's token step failed, which the run's log warns about (2.1); when the approver's review doesn't count, gh api graphql -f query='query($o:String!,$n:String!,$p:Int!){repository(owner:$o,name:$n){pullRequest(number:$p){reviews(last:20){nodes{author{login} state authorCanPushToRepository}}}}}' -f o=<owner> -f n=<name> -F p=<number> shows authorCanPushToRepository false when the approver lacks Contents: write there (1.3); a missing approval was dismissed by a later review that didn't approve, or by a push. The summary comment's Approval: line says why.
      • ## 3. Keys, then ### 3.1 Store a Key, Then Delete the .pem. Who: an organization owner. Where: the app's settings page, https://github.com/organizations/NCodeGroup/settings/apps/ncode-review or .../ncode-review-approver, Private keys; and the clone from 1.5. Do: Generate a private key, which downloads a .pem; store it, the NCode Review app's with -AppPrivateKeyPath (1.5) and the approver's with -Repo <name> -ApproverPrivateKeyPath (2.2); then Remove-Item -LiteralPath <path to the .pem>. Why: GitHub keeps the secret and gives it only to the workflows; a new key can be generated at any time; and a leaked approver key can approve on every repository the approver is installed on. Check: Test-Path -LiteralPath <path to the .pem> prints False. Goes wrong: a copy kept in Downloads or a backup outlives the deletion.
      • ### 3.2 Rotate a Key. Who and Where: as in 3.1. Do, as a numbered list: generate a new key; store it everywhere (the approver's with -Repo naming every repository its installation lists); delete the .pem; only then delete the old key on the app's page. The Bedrock key rotates with .\tools\Enable-CodeReview.ps1 -UpdateBedrockKey, then the old key is revoked in AWS. Check: the app's page lists only the new key, and 2.5's check passes on the next review. Goes wrong: deleting the old key first stops every repository that still has it from approving, or, for the NCode Review app, from posting.
    • Pattern: step 1's ### 1.1 block.
    • Done when: $t = Get-Content -Raw -LiteralPath docs/setup.md; $need = '### 2.5 Check the Approval', 'ncode-review-approver[bot] APPROVED', 'authorCanPushToRepository', '## 3. Keys', '### 3.1 Store a Key, Then Delete the .pem', '### 3.2 Rotate a Key', '-UpdateBedrockKey'; $miss = $need | Where-Object { -not $t.Contains($_) }; if ($miss) { $miss; exit 1 }

    [ ] 7. Add section 4, how a review behaves

    • Files: docs/setup.md (edit)
    • Change: Append ## 4. How a Review Behaves, a line "What a maintainer sees and does once the repository is set up; Commands and Approvals are the reference.", then two subsections of plain bullets (no Who/Where labels):
      • ### 4.1 Starting a Review: opening, reopening or marking a pull request ready starts a review, drafts included; an automatic run on a commit already reviewed stops, and its log says " was already reviewed; comment /review to review it again."; a push starts nothing, so comment /review as a top-level comment whose first line is exactly /review (lines below it, such as a --- footer, are fine); only people with write access can, and a private organization membership counts as a contributor, so make it public or be added as a collaborator; a pull request a bot opens isn't reviewed when it opens, so a maintainer approves Dependabot's where an approval is required; the skip-review label skips the automatic reviews; pull requests from forks are never reviewed; after three reviews with no comment from a writer, an automatic review posts "Paused: comment /review to resume".
      • ### 4.2 Findings and the Approval: with approve: true, a clean review approves and every other review dismisses the earlier approval; a push dismisses it when the ruleset dismisses stale approvals, so comment /review after a push; nits don't block: a review whose only findings are nits approves when every reviewer ran in full and verification lowered or refuted nothing; a later review can raise a problem's severity, since a finding repeats an earlier thread only at the same severity unless a writer dismissed that thread (resolved it while still on its code, or a 👎), so an unanswered nit can come back as a should-fix in a new thread; everything a review publishes, its trace included, is checked for anything shaped like a credential, which a test fixture in the diff can trip, and the review then fails (#233, 5.5). Link ADR-1009 after the approval bullets.
    • Pattern: the bullet lists under ## Commands in README.md.
    • Done when: $t = Get-Content -Raw -LiteralPath docs/setup.md; $need = '## 4. How a Review Behaves', '### 4.1 Starting a Review', '### 4.2 Findings and the Approval', 'was already reviewed; comment /review to review it again.', 'skip-review', 'Paused: comment /review to resume', 'same severity', '](decisions/1009-a-clean-review-approves-the-pull-request.md)'; $miss = $need | Where-Object { -not $t.Contains($_) }; if ($miss) { $miss; exit 1 }

    [ ] 8. Add section 5, troubleshooting, and check every link

    • Files: docs/setup.md (edit)
    • Change: Append ## 5. Troubleshooting, then five entries with Symptom, Cause and Fix:
      • ### 5.1 A Missing ApproverPrivateKeyPath Parameter: Symptom "A parameter cannot be found that matches parameter name 'ApproverPrivateKeyPath'."; Cause: the clone's Enable-CodeReview.ps1 is older than v1.5.0; Fix: git pull in the clone and run it again.
      • ### 5.2 A 403 Listing an Installation's Repositories: Symptom: a warning "Couldn't list the repositories of the NCode Review Approver installation (…): GitHub lists them only for a GitHub App's user token or a personal access token. Make sure / is selected at https://github.com/organizations/NCodeGroup/settings/installations/.", or, from a clone older than #236, the run stops on HTTP 403 from user/installations/<id>/repositories; Cause: gh's token can't list an installation's repositories, whatever its scopes; Fix: select the repository on the page the warning names (2.1), and git pull an older clone; adding scopes doesn't help.
      • ### 5.3 admin:org Errors: Symptom "gh can't read NCodeGroup's organization secrets. Grant the scope with: gh auth refresh -h github.com -s admin:org", or an error ending "If it's a 403, the token needs the admin:org scope: gh auth refresh -h github.com -s admin:org"; Cause: the token lacks admin:org, or a GH_TOKEN or GITHUB_TOKEN variable replaces it; Fix: run that command, or remove the variable, and check with gh auth status.
      • ### 5.4 A Review That Doesn't Start or Skips: Symptom: no 👀 and no "Code review" check run, or a check run that ends without findings; Causes and fixes as a nested list: a push (comment /review); a commit already reviewed (/review); a comment that isn't top-level, doesn't have /review alone on its first line, or comes from someone without write access (4.1); a bot's pull request, a fork or the skip-review label (4.1); "Paused" (comment /review); "Review budget used up", when the plan's API credits are used up or the Anthropic login failed and the Bedrock key is missing, refused or throttled (1.4, 1.5); a public repository without the pull_request_target event policy from 2026-12-02 (2.4); a code-review.yml that isn't on the default branch yet, since both triggers run it from there.
      • ### 5.5 The Trace Contains Something Shaped Like a Credential: Symptom: the "Code review" check run is "Review failed" with "The trace contains something shaped like a credential (…)." (the same check names "The review", "The summary comment" or "The check run"); Cause: before it publishes anything, the review refuses text shaped like an Anthropic API key, a GitHub token, a Bedrock API key, a PEM private key's BEGIN line or a JWT (src/outputs.ts), and a test fixture in the diff is the usual source; Fix: build the fixture from parts, as test/enable-code-review.test.ts builds its key from five repeated dashes and the words around them, push, and comment /review (#233). Describe the shapes only in these words, never with an example (see Gotchas).
    • Pattern: step 1's ### 1.1 block, with Symptom, Cause and Fix as its labels.
    • Done when: $t = Get-Content -Raw -LiteralPath docs/setup.md; $need = '## 5. Troubleshooting', '### 5.1 A Missing ApproverPrivateKeyPath Parameter', "### 5.2 A 403 Listing an Installation's Repositories", '### 5.3 admin:org Errors', "### 5.4 A Review That Doesn't Start or Skips", '### 5.5 The Trace Contains Something Shaped Like a Credential', "A parameter cannot be found that matches parameter name 'ApproverPrivateKeyPath'.", '- **Symptom:**', '- **Fix:**'; $miss = @($need | Where-Object { -not $t.Contains($_) }); if ('-----BEGIN', 'sk-ant-', 'eyJ' | Where-Object { $t.Contains($_) }) { $miss += 'credential-shaped text' }; $slug = { param($h) (($h.Trim().ToLowerInvariant() -replace '[^a-z0-9 _-]', '') -replace ' ', '-') }; $heads = { param($f) Select-String -LiteralPath $f -Pattern '^#{1,6} (.+)$' | ForEach-Object { & $slug $_.Matches[0].Groups[1].Value } }; $own = & $heads docs/setup.md; $readme = & $heads README.md; foreach ($m in [regex]::Matches($t, '\]\(([^)\s]+)\)')) { $u = $m.Groups[1].Value; if ($u -match '^https?://') { continue }; $p, $a = $u -split '#', 2; if ($p -and -not (Test-Path -LiteralPath (Join-Path docs $p))) { $miss += $u; continue }; if ($a -and -not $p -and $a -notin $own) { $miss += $u }; if ($a -and $p -eq '../README.md' -and $a -notin $readme) { $miss += $u } }; if ($miss) { $miss; exit 1 }

    [ ] 9. Link the runbook from the README and the docs index

    • Files: README.md (edit), docs/README.md (edit)
    • Change:
      • README.md, ## Using it: insert, between the heading and the ```yaml block, one paragraph followed by a blank line: "To set NCode Review up for an organization or a repository, follow docs/setup.md, the runbook that orders these steps; this README is their reference."
      • README.md, ## Opting a repository in: after its first paragraph (ending `gh auth refresh -h github.com -s admin:org`.), insert the paragraph "docs/setup.md gives every step in order, with what to check and what goes wrong; this section and Approvals are their reference." In the powershell block, after the -SkipBedrock example and its blank line, insert two lines: # Store the NCode Review Approver's key in each repository's code-review environment, and add it to the approver's installation. and .\tools\Enable-CodeReview.ps1 -Repo identity-platform -ApproverPrivateKeyPath <path to the NCode Review Approver .pem>, then a blank line.
      • docs/README.md, ## What Lives Where: insert, after the best-practices-criteria.md bullet and before the decisions/ bullet, - [setup.md](setup.md): the runbook that sets NCode Review up for the organization and for each repository, top to bottom, with its troubleshooting., wrapped at 120 characters with two-space continuation indent, as the other bullets there are.
    • Pattern: the existing bullets in docs/README.md.
    • Done when: $r = Get-Content -Raw -LiteralPath README.md; $d = Get-Content -Raw -LiteralPath docs/README.md; if (([regex]::Matches($r, '\]\(docs/setup\.md\)')).Count -lt 2 -or -not $r.Contains('-ApproverPrivateKeyPath <path to the NCode Review Approver .pem>') -or -not $d.Contains('[setup.md](setup.md)')) { exit 1 }; node --test test/command.test.ts

    [ ] 10. Record the runbook in the changelog

    • Files: CHANGELOG.md (edit)
    • Change: Under ## [Unreleased], insert a ### Added heading before the existing ### Changed heading, with a blank line after the heading and after the entry, holding this entry:
      - docs/setup.mdis the runbook that sets NCode Review up end to end: once per organization (gh's scopes, the NCode Review and NCode Review Approver apps, Anthropic's workload identity federation and the organization secrets), once per repository (the approver's installation and key,code-review.yml's approval inputs, requiring the approval and checking it), the private keys, how a review behaves, and the errors met so far with their fixes; README.mdanddocs/README.md link it ([#237](https://github.com/NCodeGroup/code-review/issues/237)).
      Change nothing else in the file.
    • Pattern: the entries under ### Changed in ## [Unreleased].
    • Done when: $t = Get-Content -Raw -LiteralPath CHANGELOG.md; $u = $t.Substring(0, $t.IndexOf('## [1.5.0]')); if (([regex]::Matches($u, '(?m)^### Added$')).Count -ne 1 -or $u.IndexOf('### Added') -gt $u.IndexOf('### Changed') -or -not $u.Contains('- docs/setup.md is the runbook') -or -not $u.Contains('issues/237')) { exit 1 }

    Review checklist

    • Every fact in docs/setup.md matches its source: messages match tools/Enable-CodeReview.ps1, src/command.ts and src/outputs.ts; IDs and inputs match action.yml; permissions match gh api orgs/NCodeGroup/installations; the baseline's behavior matches NCodeGroup/project-template's tools/Set-RepositoryBaseline.ps1.
    • Every one of the issue's 13 items has its step, with Who, Where, Do, Check and Goes wrong (or Symptom, Cause and Fix in section 5).
    • Nothing in the new text matches a pattern in src/outputs.ts, so this pull request's own review can publish.
    • The README's Using it caller block is unchanged and test/command.test.ts passes.
    • The runbook links the README and the decision records rather than restating them, and claims only what is built.

    Follow-ups

    • Show the approval inputs in the README's caller example: README.md's Using it workflow pins v1.0.0 (d14405f…) and passes neither approve: true nor approver-private-key, and its Interface section's "A caller needs" list leaves out the optional NCODE_REVIEW_APPROVER_PRIVATE_KEY environment secret, so the reference lags docs/setup.md's step 2.3; bring both to the current release.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    documentationImprovements or additions to documentation

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions