feat(repo): pin rat-stack and measure the static scorecard rows

[systemfsoftware-maker]

Layer 2 of the rat-stack scorecard stack (plan: docs/plans/2026-10-05-2151-feat-ratstack-scorecard-plan.md, unit U2). It sits on #44. It stays inert until U3 adds the workflow.

What lands

• evals/ratstack-scorecard/flake.nix has its own lock. It pins nixpkgs to 4975466 (the root's rev) and the launcher to pnpm-release-management/1801228 (PR ci(release): keep the release pipeline off the template placeholder #5).
  • ratstack-src fetches rat-stack at ratstack.pin.json (54d3560, rat-stack main HEAD) as a fixed-output fetch.
  • tools-store is the offline pnpm store for the instrument's npm tools. It is built by the launcher repo's mkPnpmWorkspacePackages (nixpkgs fetchPnpmDeps, fetcher v4).
  • scorecard runs the Deno orchestrator with those paths baked in. Its --allow-run is limited to git and the pinned launcher.
• The static family (src/families/static.ts) measures two rows on both sides, three runs each:
  • M23: suppression directives. src/tools/extract-comments.mjs runs oxc-parser inside the launcher and returns comment text. The orchestrator classifies that text with src/model/directives.ts, using rat-stack's three ledger families plus Stryker, ESLint, Biome, dprint, deno-lint-ignore and coverage ignores.
  • M28: distinct name@version packages. src/tools/parse-lockfile.mjs runs yaml inside the launcher and reads both YAML documents of a pnpm 11/12 lockfile. src/model/lockfile.ts deduplicates the keys.
• Exclusions are declared, with reasons, in src/sides/{ratstack,starter}.ts:
  • rat-stack: tools/oxlint/anti-slop/ and vendor/, the same trees its own ledger excludes.
  • starter: repos/ (subtrees) and evals/ (the instrument itself).
• verifyCitation (src/model/verify-citation.workflow.ts) re-checks a cited rat-stack line range at the pin. M12 uses it in U7.

Deviations from the plan, declared (CONST-W3)

• oxc-parser is 0.152.0, not 0.153.0. 0.153.0 was published at 2026-10-05T10:53Z, and pnpm's minimumReleaseAge (1 day) refused it inside fetchPnpmDeps (exact error in the session: @oxc-parser/binding-*@0.153.0 was published at 2026-10-05T10:2x, within the minimumReleaseAge cutoff). I kept the policy and used the newest version older than a day.
• The tools copy into a temp work dir before the offline install. The plan named a ratstack-toolchain lockfile-hash derivation. Here the tool node_modules are installed inside the sandbox from tools-store. ratstack-toolchain (Node 24.20.0 plus the pnpm 11.3.0 tarball) lands with U4, which is the first unit that runs rat-stack's own install.
• The directive counter is a pure module, not a workflow. It counts and makes no decision, so it is directives.ts. Citation checking is a decision (Verified | Contradicted), so it is a workflow.

Verification (run in this session)

$ scorecard=$(nix build --no-link --print-out-paths ./evals/ratstack-scorecard#scorecard)
$ $scorecard/bin/scorecard measure --family static --out static.json      # wallMs 3052
M23 ratstack 54d3560 runs [219,219,219]   detail: oxlint 138, effect-diagnostics 75, typescript 5, eslint 1;
                                          453 files counted, 0 parse errors; anti-slop 36 files excluded
M28 ratstack 54d3560 runs [957,957,957]   2 lockfile documents, lockfileVersion 9.0
M23 starter  395bab5 runs [0,0,0]         15 files counted; repos/ 18 and evals/ 28 files excluded
M28 starter  395bab5 runs [392,392,392]

$ cd evals/ratstack-scorecard
$ nix develop --command sh -c 'SANDBOX_PROJECT=$PWD sandbox --pnpm-store "$SANDBOX_PNPM_STORE" -- pnpm install --frozen-lockfile'
The integrity of 1515 files was checked in 4.7s.
$ nix develop --command sh -c 'SANDBOX_PROJECT=$PWD sandbox -- pnpm vitest run'
 Test Files  8 passed (8)
      Tests  30 passed (30)
$ DENO_NO_PACKAGE_JSON=1 deno check src/      (exit 0)
$ pnpm format:check && pnpm gate:tasks && pnpm gate:dist      (all green; mutation not run locally, per Kiro)

M23 compared with rat-stack's own published total. https://ratstack.sh/debt.md today shows 217: effect-diagnostics 75, oxlint 138, typescript 4. The neutral counter gets 219 at the pin. The two extra directives are one @ts-* and one eslint-disable. Both sit outside rat-stack's ledger roots (it counts only apps|packages|scripts|tools/ and root *.config.ts). KTD8 counts every tracked source file except declared vendored trees. The site also serves ed63ba3, while the pin is 54d3560.

Sabotage

1. Source-file filter. Making isSourceFile also accept .md turns J2 red (only the directive in a comment counts …: files gained README.md). Reverted, and J2 is green.
2. Citation window. Widening the cited window by one line (slice(lines[0] - 1, lines[1] + 1)) turns the citation law red. The previous version of the law checked only the line after, missed this sabotage, and was strengthened to check both neighbours. Reverted, and the suite is green (30 passed).
3. Launcher removed. Pointing the wrapper at another launcher binary (SCORECARD_SANDBOX=/nonexistent/sandbox) is refused by Deno itself: NotCapable: Requires run access to "/nonexistent/sandbox". The orchestrator can spawn only git and the pinned launcher, so no tool has an unsandboxed path.

Review fixes (Kiro rulings, 2026-10-06)

Commits on this stack, per ruling. Local gate at bd50211: pnpm format:check ok, gate:tasks and gate:dist green, pnpm scorecard:check 40/40 (import rules, then journeys), tsc -p evals/ratstack-scorecard clean, deno check src/ clean, actionlint clean. Mutation runs in CI only.

Ruling	Commit	Proof
#1+#3 one vitest project, journeys by import	b937803 (#44), 849d77b (#46)	renaming static-family.test.ts to measure.test.ts still runs it from its record; a src/model/misplaced.test.ts importing the fixture fails scorecard check (exit 1)
#4 absolute bar for Unsupported rows	9bfdcfd (#44)	sabotage meetsBar => true: 2 tests red; reverted green
#18 duplicate cell refused	9bfdcfd, bd50211	aggregate on families with a second ratstack M23 cell: DuplicateCell {"id":"M23","side":"ratstack"}, exit 1
#11 static family end to end	849d77b (#46)	deleted record: MissingLauncherRecord, red; edited fixture repo without re-producing: StaleLauncherRecord, red; failing tool gives InstrumentError with exited 3
#2 cache never holds InstrumentError	bd50211	cache-check on a poisoned entry: CacheUnusable … M23, M28, exit 1; clean entry exit 0; save is gated on the same check
#5 SHA decode + leased bot push	bd50211	pin check live: PinMoved 54d3560 -> 65e9465; GitRefSchema requires 40 hex; push uses --force-with-lease=<branch>:<tip> and refuses a tip by another author
#6 token via credential helper	bd50211	git config --get credential.helper stores the literal ${GH_TOKEN}; git credential fill yields the secret from env only
#8 HttpClient timeout + retry, typed	bd50211	missing repo: GithubApiError naming the request, exit 1 (no silent baseline); 15 s timeout, 3 transient retries
#9 definition hash scope, base-branch instrument, drift, preview	ac93e77, bd50211	editing src/harness/sandbox.ts leaves M23/M28 hashes unchanged; editing src/families/static.ts moves both; grading jobs check out base.sha; instrument preview job is continue-on-error
Orchestrator split	849d77b, bd50211	adding import 'effect' to src/families/static.ts fails the import check (exit 1); driver flake has no --allow-net