Skip to content

perf(repo): halve vm runner mutation wall time with identical verdicts - #118

Merged
ryanleecode merged 29 commits into
mainfrom
optimize/vm-runner-mutation-wall-time
Sep 26, 2026
Merged

ryanleecode merged 29 commits into
mainfrom
optimize/vm-runner-mutation-wall-time

Conversation

@systemfsoftware-maker

@systemfsoftware-maker systemfsoftware-maker commented Sep 25, 2026 •

Copy link
Copy Markdown
Collaborator

Summary

A stryker run with testRunner: 'vm' now takes about half the wall time, and every mutant verdict is the same as before. The saving comes from idle time: checkers now run in parallel and release their share of concurrency when they finish, and workers start faster (a shared compile cache, and a worker thread booted before the test file that needs it).

Layer 3 of stack #112, on top of #111, which made vm run the real Vitest runner and fixed its verdicts but left it about as slow as testRunner: 'vitest'.

Measurements

Enterprise fixture (4 packages, 319 mutants, TypeScript checker), stryker.vm.config.ts, 5 interleaved pairs of the #111 head against this branch. The host load was 29-55 on 8 cores, so absolute times are noisy; the paired gap is not.

#111 head This PR
Wall time (median) 24.94 s 12.64 s
First tested mutant 14.24 s 6.81 s
Verdict differences (12 runs, 319 mutants) reference 0

A rebuild of the final head measured 10.87 to 11.42 s over 4 runs, again with 0 differences.

What each change bought, each measured against the previous best:

Change Before After
Check mutant groups on every checker process at once 20.67 s 17.95 s
Stop the checker processes as soon as checking ends 17.95 s 16.45 s
Share a Node compile cache with every worker 15.79 s 14.87 s
Give the checkers' share of concurrency to test runners after checking 15.64 s 12.76 s
Boot the next isolated worker thread ahead of time 11.98 s 11.24 s

Two other experiments were measured and dropped: one scheduler streaming checked mutants straight into test runs (no gain, because the host is CPU-bound), and grouped type-checking in the TypeScript checker (11 % slower).

Design decisions

  • Isolation is unchanged. Each test file still gets a brand-new worker thread. The standby pool only boots that thread while the previous file runs. A spare is handed out only when the project, env and execArgv match what it was booted with, because the thread bakes those in at boot. Vitest sends the config and environment in the start message for each file, after the claim, so a spare booted during the dry run cannot carry stale settings into a mutant run.
  • Vitest API risk. The standby pool plugs in through Vitest's poolRunner option and ThreadsPoolWorker, both marked @experimental in Vitest 5.0.1. A Vitest minor could break it; the vm parity differential would catch a verdict change.
  • Brief oversubscription. When checking ends, the test-runner pool can grow from testRunners to testRunners + checkers (idle runners retire after 1 minute) while the checker processes are still shutting down in the background. Until they exit, more processes than concurrency can be alive.
  • Compile cache location. The CLI calls module.enableCompileCache() and passes the directory to every worker through NODE_COMPILE_CACHE. It uses Node's default directory under the OS temp dir, as TypeScript's own tsc does. NODE_DISABLE_COMPILE_CACHE=1 turns it off, and on Node older than 22.8 the CLI runs without it instead of crashing before the version check.
  • Cleanup on failure. A runtime that fails after Vitest starts, such as a refused browser-mode project, now closes its driver. That stops its standby threads and removes the stryker-setup-*.js file it copied into the project.

Validation

pnpm check:ci (format, lint, typecheck, every test suite including the vm parity differential, build) and the changeset gate pass on the head of this branch.

New tests, each shown to fail against the behaviour it pins:

  • Standby pool properties: spare reuse only on an identical request with exactly one replacement booted, a boot or post-claim failure rejecting start(), and disposal waiting for every stop.
  • Checker properties: results stay in plan order when later groups finish first; checks spread over distinct pool slots without exceeding the pool; a crashed checker slot is replaced; checker contract breaches become mutation-test stage errors; the checkers are released exactly once when checking finishes, fails or is interrupted.
  • The browser-mode refusal scenario now also asserts that no setup file is left in the project. On the previous head it found stryker-setup-<uuid>.js left behind.

Not covered by a test: the order in which the worker environment is merged, which is what lets a user's NODE_COMPILE_CACHE or NODE_DISABLE_COMPILE_CACHE win. The node driver that builds it cannot host in-source tests under the repo's lint rules; a smoke run confirmed that 4 test-runner workers, 4 checker workers and their Vitest threads all read the same cache directory.

New concepts

Node's module compile cache. Since Node 22.8, module.enableCompileCache() stores V8 code cache for the CommonJS and ESM modules Node loads in a directory on disk. The next process that loads the same file skips parsing and compiling it. A child process joins the same cache when NODE_COMPILE_CACHE names the directory.

Why here: a mutation run starts dozens of short-lived processes and worker threads that each load the same Vitest, Vite and TypeScript bundles. A thread-reuse scheme would also save that time, but it would weaken per-file isolation. The cache speeds up every fresh start and leaves isolation alone.

Example: packages/stryker-js/src/bin/enable-compile-cache.ts enables it at CLI start, and the node driver adds NODE_COMPILE_CACHE to every worker it spawns.

When not to use it: when coverage depends on V8 source positions (Vitest turns the cache off itself for v8 coverage), or in a library that other code imports, where turning it on would be a side effect on the host process.

Related: #100


View with [code]smith Autofix with [code]smith
Need help on this PR? Tag @codesmith-bot with what you need. Autofix is disabled.

… vm harness

The vm runner re-implemented Vitest behaviour and reported 144 of 319 enterprise-fixture mutant statuses differently from Vitest. testRunner vm now runs the vitest runner on Vitest's isolated threads pool, so its verdicts are Vitest's own, and @systemfsoftware/stryker-vm-harness is removed

Refs #100
…test runner

A test file that fails as a whole (a suite registering no tests, a load-time throw, a beforeAll or afterAll failure) now reaches the mutant decision as a failing test, so the mutant is reported Killed instead of Surviving, and a dry run loading such a file fails naming it

Skipped tests that never start are collected as skipped instead of being dropped from the run results

A related-only start whose filter matches no test file is recognised from the rendered error code, so the run continues with no tests to kill the mutant instead of stopping with a runner crash

Refs #100
A Vitest test declared with test.todo carries mode todo and no result, which the run-status mapping read as failed; it is now reported as skipped, as Vitest itself reports it

Refs #100
… harness

The harness compared a killed mutant's status reason against the deleted vm harness's literal; it now uses TestRunner.HitLimitReasonPrefix so a runner hit-limit kill is recognised as the runaway verdict

Refs #100
Also regenerates the stryker-js API report for the new Plugin exports

Refs #100
The vm test runner config is a pure rewrite, not a blueprint, so it moves out of the blueprint kind file and the blueprint-kind lint gate stops applying; the parity candidate child runner is built with a piped Option so the effect-tsgo pipeability diagnostic is gone

Refs #100
…mutated code

The vm runner now runs Vitest, which narrows the dry run to tests related to the mutated files; the fixture's test imported nothing, so the dry run found no tests. The workspaces also run the real platform ports now that vm spawns a worker

Refs #100
… a real worker

Replaces the scenarios that pinned the resolved runner config and the runner kind with a dry run of a browser-mode project, and moves the pool characterization out of the runner's src tests, which the test-discipline rules reject

Refs #100
The vm runner narrows the dry run to tests related to the mutated files, and the workspace's only test imported nothing, so the run stopped before writing its report

Refs #100
… failure

The bail setting was fixed when the Vitest instance was created, so it applied to the dry run too and a failed dry run named only its first failing test. The runner now sets each project's bail per run: never for the dry run, as configured for mutant runs. The dry-run-failure workspace's tests now import the module they cover, since the runner only runs related tests

Refs #100
Real Vitest fails a file whose empty describe registers no tests, and the vitest runner reports such a file as one failed test named by its path. The reference arm now records the same outcome from Vitest's file result, under a key both sandboxes share

Refs #100
After merging main, a mutate set whose files produce no mutants reached the dry run with those files as its related set. The Vitest runner found no related test and the run failed with No tests were executed, where main expects an empty report. The dry run now relates tests only to files that carry mutants, and the runner runs every test when that list is empty

Refs #100
With AGENT set the shared config bailed after one failure. The cancelled run then waited forever on the vm parity differential's worker, so a single failing test turned pnpm test into a hang until the caller's timeout. Agent runs now finish and report every failure, as CI runs do

Refs #100
…sses

Stryker spawned one checker process per concurrency slot but checked every group on a single slot, so 321 checks ran one after another. Each group now takes its own pool slot, bounded by the pool size, and results keep plan order

Refs #100
The checker pool lived in the mutation-test stage scope, so its processes shut down only after the last test run and their closes added up to three seconds to every run. The pool now has its own scope, closed in a background fiber as soon as the checks return, and the stage still waits for that close before it ends

Refs #100
Every Vitest worker thread reloaded the Vitest runtime from source, and most of each mutant run went to module compilation. The CLI now enables Node's compile cache first thing and passes its directory to every worker it spawns, so the checker and test-runner processes and their Vitest threads reuse compiled code. NODE_DISABLE_COMPILE_CACHE and a preset NODE_COMPILE_CACHE still win. The library's nodePlatformLayer is unchanged

Refs #100
…er checking

The resolved concurrency is split between checkers and test runners, but checking ends before the first test run, so the checker share sat idle for the whole test phase. The test-runner pool now warms the test-runner share and grows on demand up to the full concurrency, and the mutant stream runs at that total

Refs #100
Every test file of a vm-runner mutant run gets a brand-new Vitest worker thread, and most of that thread's life was importing Vitest's runtime before the file started. The runner now supplies its own threads pool that hands each file a fresh thread spawned in the background while the previous file ran, so isolation is unchanged and the start-up leaves the critical path

Refs #100
A runtime that failed after createVitest, such as a refused browser-mode project, never closed its driver, so its standby threads and the copied stryker-setup file outlived the failure. The runtime now closes on any failure after creation and removes the setup file on any failure after the copy

A spare thread is reused only when the claiming request has the same project, env and execArgv it was booted with, because the worker bakes those in at boot. Disposal waits for every stop before reporting the first failure, a boot that fails during a claim rejects start(), and a death names whether it happened before or after the claim

Refs #100
The CLI called module.enableCompileCache at import, before the Node version check, so a runtime older than 22.8 crashed with a TypeError instead of reporting the unsupported version. It now feature-detects the API and runs without the cache when it is missing or reports a failure

Refs #100
Properties pin the checker fan-out: results keep plan order when later groups finish first, checks spread over distinct pool slots without exceeding the pool, a crashed checker slot is replaced, and checker breaches become mutation-test stage errors

The checker scope lifetime moves into checkerResourcesInOwnScope so one helper owns it: the checkers are released in the background once checking finishes, and exactly once when checking fails or the stage is interrupted

Refs #100
@systemfsoftware-maker
systemfsoftware-maker added this pull request to stack #112 September 25, 2026 22:30
…ne console

The capture layer ran every first string argument through printf-style substitution, so console.log('%%') recorded a single percent sign where Node prints the string unchanged. The merge-reports summary property found it on macOS when it drew a summary containing %%. A single string argument is now recorded as given, as Node does

Refs #100
Four e2e scenarios hit the 120 s test timeout on run 36198645464 while the same code passed e2e on the previous head; a personal token cannot rerun the job

Refs #100
Base automatically changed from refactor/vm-runner-vitest-delegation to main September 25, 2026 23:31
@ryanleecode
ryanleecode merged commit a1057e2 into main Sep 26, 2026
6 checks passed
@ryanleecode
ryanleecode deleted the optimize/vm-runner-mutation-wall-time branch September 26, 2026 02:16
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants