Skip to content

Fill in the OpenSSF Best Practices documents #206

Closed

Description

@polewskm

GOVERNANCE.md, ROADMAP.md, docs/architecture.md, docs/assurance-case.md and docs/best-practices-badge.md still hold the template's placeholders and [//]: # prompts. Write them for this repository, covering the maintainers, the roadmap, the components and the assurance case, so the silver-level criteria they map to are met.

Found while planning Update to project-template v0.96.0; it landed in #205 (release https://github.com/NCodeGroup/code-review/releases/tag/v1.4.1).

Activity

  1. polewskm commented on Nov 16, 2026

    @polewskm
    ContributorAuthor

    Plan snapshot. The working copy is PLAN.md in the plan's checkout; this comment is not updated as steps run.

    Plan: Fill in the OpenSSF Best Practices documents (8 steps)

    Plan: Fill in the OpenSSF Best Practices documents

    Issue: #206

    Goal

    GOVERNANCE.md, ROADMAP.md, docs/architecture.md and docs/assurance-case.md still hold the project template's placeholders and [//]: # prompts. This plan writes them for code-review itself: its maintainer and owner, its plans for the next twelve months, its components and design rules, and its security requirements, threat model, design principles, the weaknesses it counters and its evidence. Those are the documents the OpenSSF Best Practices silver criteria point to.

    Context for the executor

    • Prelude: ./.template/Use-Node.ps1
    • Build: npm run typecheck
    • Test: npm test
    • Setup: this is a fresh worktree; run npm ci once before the gate step if node_modules is missing.
    • Conventions: UTF-8, LF, no BOM. Markdown prose wraps at 120 columns, and a bullet's continuation lines are indented by two spaces. Copy every text block in a step exactly, line for line: the fenced block is the text, and its fence lines are not part of it. Step 3's block is fenced with four backticks because it holds a mermaid block, whose three-backtick fences are part of the text.
    • Gotchas: a [//]: # "…" line is a Markdown comment the template left as a prompt. A step that fills its section deletes that comment line, and the blank line after it, so a section never has two blank lines in a row. Change no file the step doesn't list, and no line it doesn't name.

    Decisions

    • The plan is a docs plan with no Release: line: it changes only documentation, which AGENTS.md says releases nothing (ADR-1006). Its changelog entry goes under ## [Unreleased].
    • @polewskm is named as the only maintainer and owner. On 2026-11-15 the repository's only collaborator was polewskm (admin), and so was the NCodeGroup organization's only admin.
    • The roadmap lists the five items the maintainer chose on 2026-11-15, and "Not Planned" lists reviewing outside NCodeGroup repositories and running as a hosted service.
    • docs/best-practices-badge.md stays as it is. The project isn't registered at bestpractices.dev yet, so its PROJECT_ID and the registration prompt stay until a maintainer registers it. Every other line of it is already true for this repository.
    • The assurance case states only what the code and README already say, citing the modules that enforce each requirement. It says plainly that test coverage isn't measured, because nothing measures it.
    • Each threat names the module or setting that counters it, so a reviewer can check the claim against the code.

    Out of scope

    • Any code, workflow, action.yml, README.md or SECURITY.md change.
    • docs/best-practices-badge.md, CODE_OF_CONDUCT.md and every template-owned file.
    • Registering at bestpractices.dev, or adding a second owner.

    Steps

    [ ] 1. Name the maintainer and owner in GOVERNANCE.md

    • Files: GOVERNANCE.md (edit)
    • Change:
      • Replace line 3, code-review belongs to the repository's owners. This page says who does what and how decisions are made., with code-review belongs to the NCodeGroup organization, and @polewskm maintains it. This page says who does what and how and, on the next line, decisions are made.
      • Delete the line [//]: # "Maintainers: name each maintainer in the Who column, such as @octocat." and the blank line after it.
      • In the Maintainer table row, replace | Maintainer | The repository's owners | with | Maintainer | @polewskm |; the rest of the row stays.
      • Delete the line [//]: # "Maintainers: name the owners, and keep the list current." and the blank line after it.
      • Replace Owners today: the repository's owners. Continuity needs with Owners today: @polewskm, the only owner of the repository and the NCodeGroup organization's only administrator. Continuity needs, then rewrap that paragraph at 120 columns, keeping every word.
    • Done when: if (-not (Select-String -LiteralPath GOVERNANCE.md -Pattern "repository's owners|\[//\]: #" -Quiet) -and (Select-String -LiteralPath GOVERNANCE.md -Pattern '^\| Maintainer \| @polewskm \|' -Quiet) -and (Select-String -LiteralPath GOVERNANCE.md -Pattern '^Owners today: @polewskm' -Quiet)) { exit 0 } else { exit 1 }

    [ ] 2. Write the roadmap

    • Files: ROADMAP.md (edit)
    • Change: delete the three [//]: # "Maintainers: …" lines, each with the blank line after it. Replace the two bullets under ## Next Twelve Months (from - **Supply chain.** through the line ending docs/best-practices-badge.md)).) with exactly:
      - **Review quality.** Grow the eval set, starting with must-not cases for the patterns the maintainer declines
        ([#191](https://github.com/NCodeGroup/code-review/issues/191)), and keep recall and precision measured on every
        prompt change through the smoke eval gate.
      - **Re-reviews and deep reviews.** Catch the expected concern of the `re-review-sensitive-path` eval case, which the
        re-review runs of 2026-11-01 missed ([#123](https://github.com/NCodeGroup/code-review/issues/123)).
      - **Cost and latency.** Read production's stage times from a week of v1 traces with `node tools/timings.ts`, and keep
        a normal review between $0.60 and $2.10.
      - **Learned guidance.** Retrain `guidance/` on every NCodeGroup repository's decision records and review outcomes.
      - **Supply chain and OpenSSF.** Keep external inputs pinned and current
        ([ADR-0008](docs/decisions/project-template/0008-external-inputs-are-pinned-and-kept-current.md),
        [ADR-0009](docs/decisions/project-template/0009-new-dependency-versions-wait-a-cooldown.md)), raise the OpenSSF
        Scorecard score where the repository controls it
        ([ADR-0020](docs/decisions/project-template/0020-github-projects-are-scored-by-openssf-scorecard.md)), and reach the
        Best Practices silver level, which needs a second owner for continuity
        ([GOVERNANCE.md](GOVERNANCE.md#continuity), [docs/best-practices-badge.md](docs/best-practices-badge.md)).
      
      Replace - Nothing is ruled out yet. with - Reviewing pull requests outside NCodeGroup repositories: the review relies on the organization's keys, app and and the next line learned guidance., then a third line - Running as a hosted service: it runs as a composite action in each repository's own workflow.
    • Done when: if (-not (Select-String -LiteralPath ROADMAP.md -Pattern '\[//\]: #|Nothing is ruled out' -Quiet) -and (Select-String -LiteralPath ROADMAP.md -Pattern '^- \*\*Review quality\.\*\*' -Quiet) -and (Select-String -LiteralPath ROADMAP.md -Pattern '^- Running as a hosted service' -Quiet)) { exit 0 } else { exit 1 }

    [ ] 3. Describe the components in docs/architecture.md

    • Files: docs/architecture.md (edit)
    • Change: delete lines 3 and 4 (the [//]: # "Maintainers: describe the components, … comment and its keeps." line) and the blank line after them. Under ## Components, replace Not described yet. with the text in step 3's block below, the mermaid fences included.
      A consumer's `.github/workflows/code-review.yml` calls the composite action in `action.yml`, which runs these steps in
      one job, each a Node.js script under `src/` run directly with `node`:
      
      ```mermaid
      flowchart LR
        event[Pull request or /review comment] --> ack[Acknowledge]
        ack --> review[Review step]
        review --> publish[Publish]
        publish --> finish[Finish]
        review -.-> agents[Agents: Read, Grep, Glob]
        agents -.-> proxy[Localhost proxy] -.-> model[Anthropic or Bedrock]
      ```
      
      | Component | Role |
      | --- | --- |
      | Acknowledgement (`src/acknowledge.ts`) | Within seconds, puts 👀 on a `/review` comment and the "Code review" check run in progress, with the runner's own `node` and no npm package |
      | Review step (`src/main.ts`) | Gathers the diff and context, picks the provider, runs the review and writes `results.json` and `trace.json`, or `status.json` with why there's no review |
      | Gather (`src/gather.ts`, `src/fetch-head.ts`, `src/writers.ts`, `src/settings.ts`) | Fetches the pull request's head only to read it, keeps only writers' text, and reads the repository's settings from the default branch |
      | Provider and proxy (`src/provider.ts`, `src/credentials.ts`, `src/proxy.ts`) | Picks Anthropic while the plan's credits last, then Bedrock, and adds the credentials to each model request on localhost |
      | Review pipeline (`src/context.ts`, `src/review.ts`, `src/verify.ts`, `src/rereview.ts`) | Context, selection, the reviewers in `reviewers/`, rule checks, synthesis and verification; a re-review reads only what changed |
      | Agents (`src/agent.ts`) | One model call each, through the Claude Agent SDK, with `prompts/` and `guidance/`, three read-only tools and a path guard |
      | Publish (`src/publish.ts`, `src/threads.ts`, `src/comments.ts`, `src/checks.ts`, `src/outputs.ts`) | Posts the review threads and summary comment, completes the check run, and scans everything it posts for secrets |
      | Finish (`src/finish.ts`) | Completes the check run with why when there was no review to publish |
      | Outcomes (`src/outcomes.ts`) | Records what became of each finding when the pull request closes, by rules alone |
      | Training and evals (`training/`, `evals/`, `tools/`) | Offline: learn `guidance/` from NCodeGroup repositories, and measure each change to the review |
      
    • Done when: if (-not (Select-String -LiteralPath docs/architecture.md -Pattern 'Not described yet|\[//\]: #' -Quiet) -and (Select-String -LiteralPath docs/architecture.md -Pattern '^flowchart LR$' -Quiet) -and ((Select-String -LiteralPath docs/architecture.md -Pattern '^\| ' | Measure-Object).Count -eq 12)) { exit 0 } else { exit 1 }

    [ ] 4. List the design rules in docs/architecture.md

    • Files: docs/architecture.md (edit)
    • Change: under ## Design Rules, keep the existing bullet - Decisions the design rests on are recorded in [decisions/](decisions/README.md). and add these bullets before it:
      - The agents hold no secret: the review step's process holds the model credentials, and the agents get the proxy's
        address and a placeholder. No app token is ever in the review step's environment.
      - Code from a pull request is read, never built, run or installed, and its text is data, never instructions.
      - A pull request can't change its own review: the workflow runs from the default branch, and so do the settings.
      - Everything published, comments, check runs and the trace, is scanned for secrets first.
      - A `/review` command is the comment's first line
        ([ADR-1001](decisions/1001-a-review-command-is-the-comments-first-line.md)), and one problem gets one thread
        ([ADR-1008](decisions/1008-one-problem-gets-one-thread.md)).
      - A re-review whose changed hunks are more than one part of a first review is a full review
        ([ADR-1007](decisions/1007-a-re-review-larger-than-one-part-is-a-full-review.md)), and a clean review approves only
        when the caller sets `approve: true` ([ADR-1009](decisions/1009-a-clean-review-approves-the-pull-request.md)).
      - Training learns from the bot's findings as the maintainer answered them
        ([ADR-1005](decisions/1005-training-learns-from-the-bots-findings-as-the-maintainer-answered-them.md)).
      
    • Done when: if ((Select-String -LiteralPath docs/architecture.md -Pattern '^- The agents hold no secret' -Quiet) -and (Select-String -LiteralPath docs/architecture.md -Pattern '1009-a-clean-review-approves' -Quiet) -and (Select-String -LiteralPath docs/architecture.md -Pattern '^- Decisions the design rests on' -Quiet)) { exit 0 } else { exit 1 }

    [ ] 5. Write the security requirements and threat model

    • Files: docs/assurance-case.md (edit)
    • Change: under ## Security Requirements, delete the [//]: # line and the blank line after it, and replace Not written yet. with:
      - **R1.** The model credentials never reach an agent, a log or anything published.
      - **R2.** The app's private key reaches only the steps that mint its tokens, and no app token is in the review step's
        environment.
      - **R3.** Code from a pull request is read, never built, run or installed, and an agent reads nothing outside its
        checkout.
      - **R4.** Text from a pull request is data: only the text of authors who can write to the repository reaches a
        prompt, and no such text reaches a shell unquoted.
      - **R5.** A pull request can't change how it is reviewed.
      - **R6.** A review's cost is bounded.
      
      Users must not rely on a review to find every defect: its findings are advice, and nothing requires its check run.
      
      Under ## Threat Model, delete the [//]: # line and the blank line after it, and replace Not written yet. with:
      The trust boundary is the pull request: its author, code and text are untrusted, and the review step, which holds the
      model credentials, reads them. Out of scope: a compromised GitHub, Anthropic or AWS account, a maintainer with write
      access acting against the project, and a vulnerability in a dependency itself ([SECURITY.md](../SECURITY.md)).
      
      | Threat | In scope | Countered by |
      | --- | --- | --- |
      | Prompt injection makes an agent leak the model credentials | Yes | R1: `src/proxy.ts` and `src/credentials.ts` keep them in the review step's process; agents get a placeholder |
      | An agent reads a file outside the checkout, through `..` or a symlink | Yes | R3: the path guard in `src/agent.ts` resolves symlinks and refuses any path outside the checkout |
      | The pull request's code runs in the job that holds the credentials | Yes | R3: `src/fetch-head.ts` fetches the head only to read; `pull_request_target` runs the default branch's workflow |
      | A stranger steers the review through a description, comment or issue | Yes | R4: `src/writers.ts` keeps only writers' text; `/review` runs only for writers; a fork's pull request isn't reviewed |
      | A pull request softens its own review | Yes | R5: `src/settings.ts` reads `.github/code-review/` from the default branch only |
      | A secret reaches a comment, the check run or the trace artifact | Yes | R1: `src/outputs.ts` scans everything published |
      | An app token is misused by an agent | Yes | R2: tokens are minted per step, revoked before the review step, and minted again only after the agents exit |
      | A poisoned dependency cache reaches the credentials | Yes | `action.yml` turns off setup-node's cache (`package-manager-cache: false`) |
      | Runaway cost or a review loop | Yes | R6: per-call caps, a cost ceiling per review (`CODE_REVIEW_MAX_RUN_USD`) and at most three reviews without a writer's comment (`src/pause.ts`) |
      | Quoted text notifies people or forges log lines | Yes | `src/markdown.ts` escapes @mentions; `src/log.ts` writes each line as one printable line |
      
    • Done when: if (-not (Select-String -LiteralPath docs/assurance-case.md -Pattern 'Maintainers: list what|Maintainers: add a table' -Quiet) -and (Select-String -LiteralPath docs/assurance-case.md -Pattern '^- \*\*R6\.\*\*' -Quiet) -and (Select-String -LiteralPath docs/assurance-case.md -Pattern '^\| Threat \| In scope \| Countered by \|$' -Quiet)) { exit 0 } else { exit 1 }

    [ ] 6. Write the design principles and the weaknesses countered

    • Files: docs/assurance-case.md (edit)
    • Change: under ## Secure Design Principles Applied, delete the two-line [//]: # "Maintainers: say how the design applies … comment and the blank line after it, and replace Not written yet. with:
      - **Economy of mechanism:** each agent has three read-only tools, and the acknowledgement runs with no npm package.
      - **Fail-safe defaults:** an author whose permission can't be read counts as no writer; with no provider the check run
        says so and nothing is charged; an incomplete review doesn't approve, and verification never turns a review into
        an approval.
      - **Complete mediation:** the path guard checks every tool call, and the writer check applies to every text source.
      - **Least privilege:** the workflow starts with `permissions: {}`; each app token can write only what its step needs,
        on the one repository under review, and is revoked; only default-branch jobs get an Anthropic token.
      - **Open design:** the code, prompts, reviewers, guidance, evals and decision records are public.
      
      Under ## Common Weaknesses Countered, delete the [//]: # line and the blank line after it, and replace Not written yet. with:
      - CWE-22 and CWE-59, path traversal and link following: R3.
      - CWE-77, CWE-78 and CWE-94, command and code injection: R3 and R4.
      - CWE-1427, improper neutralization of input used for LLM prompting: R4.
      - CWE-200, CWE-522 and CWE-532, exposed information, unprotected credentials and secrets in logs: R1 and R2.
      - CWE-117, improper output neutralization for logs: `src/log.ts`.
      - CWE-400, uncontrolled resource consumption: R6.
      - CWE-829, inclusion of functionality from an untrusted control sphere: R5, and actions pinned by full commit SHA.
      
    • Done when: if (-not (Select-String -LiteralPath docs/assurance-case.md -Pattern 'Not written yet|Maintainers: say how|Maintainers: list the CWE' -Quiet) -and (Select-String -LiteralPath docs/assurance-case.md -Pattern '^- \*\*Least privilege:\*\*' -Quiet) -and (Select-String -LiteralPath docs/assurance-case.md -Pattern '^- CWE-1427' -Quiet)) { exit 0 } else { exit 1 }

    [ ] 7. Add the tests to the evidence

    • Files: docs/assurance-case.md (edit)
    • Change: under ## Evidence, delete the line [//]: # "Maintainers: add the tests and the coverage the project keeps." and the blank line after it. Replace the bullet - **Gate:** CI runs the project's commands on every pull request. with:
      - **Gate:** CI runs `npm run typecheck` and `npm test`, the `node:test` suite in `test/`, on every pull request;
        `test/acknowledge.test.ts` checks that `action.yml` keeps every app token out of the review step's environment.
        Test coverage isn't measured.
      - **Evals:** the smoke eval runs on every push to `main` that changes the prompts, the reviewers, the guidance or the
        models, and fails when it catches more than two fewer expected findings than its baseline or flags its must-not case.
      
    • Done when: if (-not (Select-String -LiteralPath docs/assurance-case.md -Pattern '\[//\]: #' -Quiet) -and (Select-String -LiteralPath docs/assurance-case.md -Pattern 'Test coverage isn''t measured' -Quiet) -and (Select-String -LiteralPath docs/assurance-case.md -Pattern '^- \*\*Evals:\*\*' -Quiet)) { exit 0 } else { exit 1 }

    [ ] 8. Write the changelog entry and run the gate

    • Files: CHANGELOG.md (edit)
    • Change: under ## [Unreleased], which is empty, add a blank line, ### Changed, a blank line and this entry on one line: - GOVERNANCE.md, ROADMAP.md, docs/architecture.mdanddocs/assurance-case.md describe code-review itself: its maintainer, its plans for the next twelve months, its components and design rules, and its security requirements, threat model and the weaknesses it counters ([#206](https://github.com/NCodeGroup/code-review/issues/206)). Keep one blank line before ## [1.4.1] - 2026-11-15. Then run the gate.
    • Done when: if (-not (Select-String -LiteralPath CHANGELOG.md -Pattern 'issues/206\)\)\.$' -Quiet)) { exit 1 }; npm run typecheck; if (-not $? -or $LASTEXITCODE) { exit 1 }; npm test; if (-not $? -or $LASTEXITCODE) { exit 1 } else { exit 0 }

    Review checklist

    • Every claim in the four documents matches the code or README: module names exist, permissions: {} is in .github/workflows/code-review.yml, package-manager-cache: false is in action.yml, and maxUnansweredReviews is 3 in src/pause.ts.
    • No [//]: # prompt is left in GOVERNANCE.md, ROADMAP.md, docs/architecture.md or docs/assurance-case.md. docs/best-practices-badge.md is unchanged.
    • The mermaid diagram renders on GitHub, and every relative link resolves from its file's folder.
    • Prose lines stay within 120 columns. Table rows may be longer.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    follow-upWork a plan found and did not do

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions