feat(repo): pin rat-stack and measure the static scorecard rows

.github/workflows/ci.yml

@@ -45,6 +45,9 @@ jobs:
     steps:
       - uses: actions/checkout@v7
       - uses: cachix/install-nix-action@v31
+      - name: Allow the sandbox launcher's unprivileged user namespaces
+        if: runner.os == 'Linux'
+        run: sudo sysctl -w kernel.apparmor_restrict_unprivileged_userns=0
       - name: dprint from the flake
         run: echo "$(nix build --no-link --print-out-paths .#dprint)/bin" >> "$GITHUB_PATH"
       # pnpm/setup replaces pnpm/action-setup for pnpm v11+ and folds

docs/plans/2026-10-05-2151-feat-ratstack-scorecard-plan.md

@@ -155,6 +155,7 @@ Local rows run each side with its documented local defaults plus throwaway value
 - KTD14. **Readiness calls URL Scanner v2 with `agentReadiness: true`, then polls the result.** It submits `POST /accounts/{account_id}/urlscanner/v2/scan` with `agentReadiness: true` and polls `GET …/v2/result/{scan_id}` every 15 s. The first live call fixes where the option sits in the request (top level or `options`) and the result path. The cross-check posts `{ "url": … }` to `https://isitagentready.com/api/scan` and reads `level`, as rat-stack's `apps/mischief/scripts/smoke.sh` does. When the levels differ, the row is flagged. This implements the scanner Key Decision (session-settled: user-directed — chosen over isitagentready as primary: first-party reproducible JSON through our token).
 - KTD15. **Tests are admitted by layer, with refusal as the default** (`skill://test-layer-selection`). Every decision lives in a `*.workflow.ts` module with a colocated `*.workflow.property.test.ts`, mutated at break 100 by sfs stryker-js in CI on push to `main` (D5), never locally. Family runners, harness, side adapters and REST clients are executors and adapters: they get no tests of their own, and each unit's family smoke run plus its PR sabotage proves them. Three process-isolated journeys under `evals/ratstack-scorecard/journeys/` observe what only the seam can see: J1 launcher isolation (U4), J2 the static family over a fixture tree through the real parser in the sandbox (U2), and J3 the CLI's exit status and schema-valid JSON on a ratchet failure (U3). Refused: tests of `registry.ts` (a declaration), unit tests of runners (they would spawn processes or mock the subject), and assertions on rendered Markdown wording.
 - KTD16. **A mutated gate is graded only against a green baseline.** Every family that edits a clone and runs a gate (M13, M20, M22, M24, M25) first runs the unmodified gate on that side over warm caches, and a red baseline makes those cells `unmeasurable` with the first failing task. The cold path (M26) is its own baseline: a red cold gate makes M26 `unmeasurable` and leaves the other families alone. This keeps a kill rate or removal rate from being computed over a gate that was already failing.
+- KTD17. **Journeys that need the launcher run in two phases; only the host driver starts the launcher** (Kiro ruling, 2026-10-06). The launcher cannot nest: `bwrap --unshare-all` inside the prm#5 sandbox (`1801228`, bwrap 0.12.0) fails with `bwrap: setting up uid map: Operation not permitted`, and loosening the launcher is ruled out. So `scorecard journeys` (the zero-third-party Deno driver) first produces each journey declared as data in `journeys/manifest.json`, running the real launcher work (for example `measureStatic` end to end over two fixture git repos built from `journeys/__fixtures__/repos/`), and writes one record per journey to `journeys/__records__/<id>.json`: argv, exit code, stdout, stderr, output files, egress log, wall time, and an input hash over the declared inputs plus the launcher store path. It then runs the single vitest project inside the launcher. A test is a journey because it imports `journeys/launcher-run.ts`, whose `launcherRun(id)` decodes the record with Effect Schema and fails red with `MissingLauncherRecord` or `StaleLauncherRecord`, never a skip (CONST-T12). `scorecard check` (in the root `check:ci`) first runs the import-graph rules (the driver's graph from `src/main.ts` loads no third-party module; no file under `src/` imports the journey fixture), then `scorecard journeys`.
 
 ### High-Level Technical Design
 

evals/ratstack-scorecard/.gitignore

@@ -1 +1,2 @@
 .cache/
+journeys/__records__/

evals/ratstack-scorecard/README.md

@@ -19,12 +19,21 @@ The scorecard measures the starter against [rat-stack](https://github.com/joelho
 
 ## Running it
 
-Everything that loads third-party code runs inside the sandbox launcher from `systemfsoftware/pnpm-release-management` (`packages.<system>.sandbox`), with this directory as the sandbox project. The instrument's own flake (`flake.nix`) pins nixpkgs (pnpm 12.9.0, Node 24) and the launcher, and builds the tools' pnpm store from `pnpm-lock.yaml` as a fixed-output derivation (`tools-store`). The install is offline from that store; the sandbox gets no network at all.
+Everything that loads third-party code runs inside the sandbox launcher from `systemfsoftware/pnpm-release-management` (`packages.<system>.sandbox`).
+
+The instrument has its own flake (`flake.nix`, pinned nixpkgs and launcher). Its `scorecard` package bundles Deno, the launcher, the rat-stack source at `ratstack.pin.json`, and the offline pnpm store for the tools (`tools-store`). The family runners install the tools from that store inside the sandbox, so nothing reaches the registry at measurement time.
+
+```sh
+scorecard=$(nix build --no-link --print-out-paths ./evals/ratstack-scorecard#scorecard)
+$scorecard/bin/scorecard measure --family static --out static.json
+```
+
+The instrument's own tests run inside the launcher against the same offline store:
 
 ```sh
 cd evals/ratstack-scorecard
 nix develop --command sh -c 'SANDBOX_PROJECT=$PWD sandbox --pnpm-store "$SANDBOX_PNPM_STORE" -- pnpm install --frozen-lockfile'
 nix develop --command sh -c 'SANDBOX_PROJECT=$PWD sandbox -- pnpm vitest run'
 ```
 
-The decision modules (`src/model/*.workflow.ts`) and the orchestrator import only Deno APIs, `node:` builtins and each other, so `deno check src/` type-checks them without any third-party code.
+`src/main.ts` and everything it imports use only Deno APIs, `node:` builtins and each other, so `DENO_NO_PACKAGE_JSON=1 deno check src/` type-checks the orchestrator and the decision core without third-party code. The Node scripts in `src/tools/` are the only code that loads npm packages, and they only ever run inside the launcher.

evals/ratstack-scorecard/deno.json

@@ -4,6 +4,6 @@
     "exactOptionalPropertyTypes": true,
     "noUncheckedIndexedAccess": true
   },
-  "exclude": ["node_modules/", "**/*.test.ts", "**/*.arbitrary.ts", "vitest.config.ts"],
+  "exclude": ["node_modules/", "src/tools/", "journeys/", "**/*.test.ts", "**/*.arbitrary.ts", "vitest.config.ts"],
   "lock": false
 }

evals/ratstack-scorecard/flake.nix

@@ -4,7 +4,7 @@
   inputs = {
     nixpkgs.url = "github:NixOS/nixpkgs/494ce7fd23ff6a5dff39e1fb11e9b6f2ac74bf25";
     pnpm-release-management = {
-      url = "github:systemfsoftware/pnpm-release-management/180122866dd537fa728b5563fb1820fbd2af88cc";
+      url = "github:systemfsoftware/pnpm-release-management/8f1984418fef130956a3d1f50dc471fd1984d2ec";
       inputs.nixpkgs.follows = "nixpkgs";
     };
   };
@@ -13,6 +13,12 @@
     let
       systems = [ "x86_64-linux" "aarch64-linux" "aarch64-darwin" ];
       forEachSystem = fn: nixpkgs.lib.genAttrs systems (system: fn nixpkgs.legacyPackages.${system});
+      pin = builtins.fromJSON (builtins.readFile ./ratstack.pin.json);
+      toolsStoreHash = {
+        x86_64-linux = "sha256-jUyIV2ysOxkIij/Ff5VGuIADTibbY0SsCPGglmIvdn0=";
+        aarch64-linux = "sha256-jUyIV2ysOxkIij/Ff5VGuIADTibbY0SsCPGglmIvdn0=";
+        aarch64-darwin = "sha256-XWYBTy3xZbQwJxdGbAK4+R69XNmL3RRyXu6+nY+XPXc=";
+      };
     in
     {
       packages = forEachSystem (pkgs:
@@ -24,11 +30,34 @@
             src = self;
             pname = "ratstack-scorecard";
             pnpm = pkgs.pnpm_12;
-            hash = "sha256-TylxLEQflTlKx6QDk9mFlBOEInUvvBBeOVFUYCURmYo=";
+            hash = toolsStoreHash.${system};
           }).pnpm-store;
+          ratstack-src = pkgs.fetchFromGitHub {
+            inherit (pin) owner repo;
+            rev = pin.commit;
+            hash = pin.narHash;
+          };
+          scorecard = pkgs.writeShellApplication {
+            name = "scorecard";
+            runtimeInputs = [ pkgs.deno pkgs.git ];
+            text = ''
+              export SCORECARD_INSTRUMENT=${self}
+              export SCORECARD_SANDBOX=${sandbox}/bin/sandbox
+              export SCORECARD_TOOLS_STORE=${tools-store}
+              export SCORECARD_RATSTACK_SRC=${ratstack-src}
+              export SCORECARD_TOOL_PATH=${pkgs.lib.makeBinPath [ pkgs.nodejs_24 pkgs.pnpm_12 pkgs.coreutils ]}
+              export SCORECARD_NODE_VERSION=${pkgs.nodejs_24.version}
+              export SCORECARD_PNPM_VERSION=${pkgs.pnpm_12.version}
+              export DENO_NO_PACKAGE_JSON=1
+              exec deno run --no-config --allow-read --allow-write --allow-env --allow-sys=hostname \
+                --allow-run=git,${sandbox}/bin/sandbox \
+                ${self}/src/main.ts "$@"
+            '';
+          };
         in
         {
-          inherit sandbox tools-store;
+          inherit sandbox tools-store ratstack-src scorecard;
+          default = scorecard;
         });
 
       devShells = forEachSystem (pkgs:

evals/ratstack-scorecard/journeys/__fixtures__/lockfiles/pnpm11.yaml

@@ -0,0 +1,73 @@
+---
+lockfileVersion: "9.0"
+
+importers:
+  .:
+    configDependencies: {}
+    packageManagerDependencies:
+      "@pnpm/exe":
+        specifier: 11.3.0
+        version: 11.3.0
+      pnpm:
+        specifier: 11.3.0
+        version: 11.3.0
+
+packages:
+  "@pnpm/exe@11.3.0":
+    resolution: {
+      integrity: sha512-1ItrG3GdA8HC7IUMy79SmYqynjjfwXtIMlbpx9MzrU3ZXMYMfw3yuwjIXW7Aw2z0rRxxa2KtTYcLxqjIaVjUmg==,
+    }
+    hasBin: true
+
+  pnpm@11.3.0:
+    resolution: { integrity: sha512-AAAA }
+    hasBin: true
+
+snapshots:
+  "@pnpm/exe@11.3.0": {}
+
+  pnpm@11.3.0: {}
+
+---
+lockfileVersion: "9.0"
+
+settings:
+  autoInstallPeers: true
+  excludeLinksFromLockfile: false
+
+importers:
+  .:
+    dependencies:
+      effect:
+        specifier: 4.0.0
+        version: 4.0.0
+    devDependencies:
+      "@effect/vitest":
+        specifier: 4.0.0
+        version: 4.0.0(effect@4.0.0)(vitest@5.0.3)
+      vitest:
+        specifier: 5.0.3
+        version: 5.0.3
+
+packages:
+  "@effect/vitest@4.0.0":
+    resolution: { integrity: sha512-BBBB }
+    peerDependencies:
+      effect: ^4.0.0
+      vitest: ^5.0.0
+
+  effect@4.0.0:
+    resolution: { integrity: sha512-CCCC }
+
+  vitest@5.0.3:
+    resolution: { integrity: sha512-DDDD }
+
+snapshots:
+  "@effect/vitest@4.0.0(effect@4.0.0)(vitest@5.0.3)":
+    dependencies:
+      effect: 4.0.0
+      vitest: 5.0.3
+
+  effect@4.0.0: {}
+
+  vitest@5.0.3: {}

evals/ratstack-scorecard/journeys/__fixtures__/lockfiles/pnpm12.yaml

@@ -0,0 +1,78 @@
+---
+lockfileVersion: "9.0"
+
+importers:
+  .:
+    configDependencies: {}
+    packageManagerDependencies:
+      pnpm:
+        specifier: 12.4.2
+        version: 12.4.2
+
+packages:
+  "@pnpm/exe.linux-x64@12.4.2":
+    resolution: { integrity: sha512-EEEE }
+    cpu: [x64]
+    os: [linux]
+
+  pnpm@12.4.2:
+    resolution: { integrity: sha512-FFFF }
+    hasBin: true
+
+snapshots:
+  "@pnpm/exe.linux-x64@12.4.2":
+    optional: true
+
+  pnpm@12.4.2:
+    optionalDependencies:
+      "@pnpm/exe.linux-x64": 12.4.2
+
+---
+lockfileVersion: "9.0"
+
+settings:
+  autoInstallPeers: true
+  excludeLinksFromLockfile: false
+
+catalogs:
+  default:
+    effect:
+      specifier: ^4.0.0
+      version: 4.0.0
+
+importers:
+  .:
+    dependencies:
+      effect:
+        specifier: "catalog:"
+        version: 4.0.0
+    devDependencies:
+      "@effect/vitest":
+        specifier: 4.0.0
+        version: 4.0.0(effect@4.0.0)(vitest@5.0.3)
+      vitest:
+        specifier: 5.0.3
+        version: 5.0.3
+
+packages:
+  "@effect/vitest@4.0.0":
+    resolution: { integrity: sha512-BBBB }
+    peerDependencies:
+      effect: ^4.0.0
+      vitest: ^5.0.0
+
+  effect@4.0.0:
+    resolution: { integrity: sha512-CCCC }
+
+  vitest@5.0.3:
+    resolution: { integrity: sha512-DDDD }
+
+snapshots:
+  "@effect/vitest@4.0.0(effect@4.0.0)(vitest@5.0.3)":
+    dependencies:
+      effect: 4.0.0
+      vitest: 5.0.3
+
+  effect@4.0.0: {}
+
+  vitest@5.0.3: {}

evals/ratstack-scorecard/journeys/__fixtures__/repos/ratstack/src/intake.ts

@@ -0,0 +1,4 @@
+// oxlint-disable-next-line no-console -- counted
+console.log('intake')
+// @ts-expect-error counted
+export const n: number = 'x'

evals/ratstack-scorecard/journeys/__fixtures__/repos/ratstack/vendor/copied.ts

@@ -0,0 +1,2 @@
+// @ts-nocheck vendored, excluded
+export const v = 1