Skip to content

fix(repo): make mutation CI fail truthfully and actually mutate - #117

Merged
ryanleecode merged 7 commits into
mainfrom
lying-mutations
Sep 25, 2026
Merged

ryanleecode merged 7 commits into
mainfrom
lying-mutations

Conversation

@systemfsoftware-maker

@systemfsoftware-maker systemfsoftware-maker commented Sep 25, 2026 •

Copy link
Copy Markdown
Collaborator

Summary

Mutation CI now fails when a package produces no mutation report, and every package really gets mutated. Before this, run 36160660790 showed green while all three jobs had failed: the Mutation step had continue-on-error, the report check accepted the stream file every run writes, and merge-reports exited 0 after merging 0 of 3 reports.

What changes

  • Mutation lane. It now follows the systemfsoftware setup. A plan job sizes jobs from recorded timings, a matrix runs them, then the timings and report jobs run. There is no continue-on-error. A package that leaves no mutation-report.json fails its job with a ::error annotation, while a low score alone does not. Outputs from an earlier run are deleted before each run, including when a package is skipped because its job budget ran out. Oversized packages are split with shardMutate. It runs on pushes to main and manual dispatch only.
  • Plugins load from node_modules. Each package's stryker.config.ts called import.meta.resolve with its own package name. Node answered through the package's own exports, so the config loaded the package's unbuilt dist/ (exit 4). installedPlugin now resolves the published catalog:stryker copy.
  • Dry runs finish. stryker-js and stryker-vm-harness use a vitest.mutation.config.ts limited to unit and property tests. The integration and differential suites had pushed the dry run past its 5-minute limit (exit 3).
  • CLI output (minor bump). Piped stdout now gets human output. NDJSON goes to stdout only with --json or STRYKER_MODE=machine; in that mode stdout carries nothing else, and stderr carries no echo of each event. reports/mutation-stream.jsonl is written in both modes. Anything that parsed NDJSON from a piped stryker run without asking for it must add --json.
  • Partial reports rebuild. stryker merge-reports now reads the stream's _tag:"mutant" records. It had been decoding kind:"mutant", so a stream never rebuilt anything.
  • TypeScript checker. Each mutant is now checked against the files it changes plus their dependents, instead of asking for diagnostics of the whole program every time.
Checker on its own package (157 mutants) Result
Before 0/157 after 240 s, killed
After 157/157 in 35 s

On a 4-mutant sample, the new checker returned the same statuses as the published 7.1.0.

Review notes

A multi-reviewer pass found 10 actionable findings, and all of them are fixed on this branch. The fixes include decoding every script input with effect Schema instead of casting it, and adding tests for shardMutate, installedPlugin, buildSummary, the incomplete-part rebuild, and the --json argv probe. One finding claimed that the merge-reports flags changed. They did not: the base commit already declares --parts/--out (Cli.parts.ts), and this branch only corrects a stale README example.

Validation

  • pnpm format:check, pnpm typecheck, pnpm check:ci, pnpm test and pnpm test:scripts pass on this head.
  • A local mutation-job run on the checker package, using the published 7.1.0 checker, printed Mutation produced no report and exited 1. A stubbed run that crashed and left an old report behind also failed.
  • The CLI and checker changes reach this workflow only after they are released, because mutation runs the published CLI through catalog:stryker.

Compound Engineering


View with [code]smith Autofix with [code]smith
Need help on this PR? Tag @codesmith-bot with what you need. Autofix is disabled.

- Port the systemfsoftware mutation lane: timed plan, matrix, timings, report;
  no continue-on-error, and a package that leaves no report fails its job
- Resolve runner/checker plugins from node_modules so a package never loads its
  own unbuilt dist; dry runs use unit/property suites only
- Piped stdout gets human output; JSONL only with --json or STRYKER_MODE=machine,
  and the stream file is written in both modes
- merge-reports rebuilds partial reports from the stream's _tag records
- TypeScript checker checks mutated files and their dependents instead of the
  whole program per mutant (0/157 in 240s, now 157/157 in 35s)
- Share one stdout-or-drain sink between the file and stdout drains
- Decide the stderr echo with a boolean match instead of an orElse fallback
- Decode part meta, reports, JOB/JOBS, timings and workspace inputs with
  effect Schema instead of casting parsed JSON
- Clear earlier reports before a budget-skipped package so it fails the gate
- Name shardMutate and the path when package.json is missing or unparsable
- Cover shardMutate, installedPlugin, buildSummary, the incomplete-part
  rebuild, writeEncoded modes and the --json argv probe
- Accept --json=true like the CLI parser; rename the stream rebuild module
  to report-from-stream.steps.ts
macOS links /tmp to /private/tmp, and module resolution reports the real
path, so the installedPlugin and package.json diagnostics tests failed on
the macOS runner
…graph

Dependents that import the mutated file through a `compilerOptions.paths` alias
were missing from the affected-file set, so a mutant that the whole-program
check reported as a compile error was handed to the test runner instead

Every tsconfig the checker walks now contributes its alias patterns, and a
non-relative specifier resolves through them to the same candidate files the
compiler resolves it to, alongside the existing relative and re-export edges
@ryanleecode
ryanleecode merged commit f806528 into main Sep 25, 2026
6 checks passed
@ryanleecode
ryanleecode deleted the lying-mutations branch September 25, 2026 20:31
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants