perf(repo): halve vm runner mutation wall time with identical verdicts - #118
Merged
Merged
Conversation
… vm harness The vm runner re-implemented Vitest behaviour and reported 144 of 319 enterprise-fixture mutant statuses differently from Vitest. testRunner vm now runs the vitest runner on Vitest's isolated threads pool, so its verdicts are Vitest's own, and @systemfsoftware/stryker-vm-harness is removed Refs #100
…test runner A test file that fails as a whole (a suite registering no tests, a load-time throw, a beforeAll or afterAll failure) now reaches the mutant decision as a failing test, so the mutant is reported Killed instead of Surviving, and a dry run loading such a file fails naming it Skipped tests that never start are collected as skipped instead of being dropped from the run results A related-only start whose filter matches no test file is recognised from the rendered error code, so the run continues with no tests to kill the mutant instead of stopping with a runner crash Refs #100
A Vitest test declared with test.todo carries mode todo and no result, which the run-status mapping read as failed; it is now reported as skipped, as Vitest itself reports it Refs #100
… harness The harness compared a killed mutant's status reason against the deleted vm harness's literal; it now uses TestRunner.HitLimitReasonPrefix so a runner hit-limit kill is recognised as the runaway verdict Refs #100
Also regenerates the stryker-js API report for the new Plugin exports Refs #100
The vm test runner config is a pure rewrite, not a blueprint, so it moves out of the blueprint kind file and the blueprint-kind lint gate stops applying; the parity candidate child runner is built with a piped Option so the effect-tsgo pipeability diagnostic is gone Refs #100
…mutated code The vm runner now runs Vitest, which narrows the dry run to tests related to the mutated files; the fixture's test imported nothing, so the dry run found no tests. The workspaces also run the real platform ports now that vm spawns a worker Refs #100
… a real worker Replaces the scenarios that pinned the resolved runner config and the runner kind with a dry run of a browser-mode project, and moves the pool characterization out of the runner's src tests, which the test-discipline rules reject Refs #100
The vm runner narrows the dry run to tests related to the mutated files, and the workspace's only test imported nothing, so the run stopped before writing its report Refs #100
… failure The bail setting was fixed when the Vitest instance was created, so it applied to the dry run too and a failed dry run named only its first failing test. The runner now sets each project's bail per run: never for the dry run, as configured for mutant runs. The dry-run-failure workspace's tests now import the module they cover, since the runner only runs related tests Refs #100
Real Vitest fails a file whose empty describe registers no tests, and the vitest runner reports such a file as one failed test named by its path. The reference arm now records the same outcome from Vitest's file result, under a key both sandboxes share Refs #100
After merging main, a mutate set whose files produce no mutants reached the dry run with those files as its related set. The Vitest runner found no related test and the run failed with No tests were executed, where main expects an empty report. The dry run now relates tests only to files that carry mutants, and the runner runs every test when that list is empty Refs #100
With AGENT set the shared config bailed after one failure. The cancelled run then waited forever on the vm parity differential's worker, so a single failing test turned pnpm test into a hang until the caller's timeout. Agent runs now finish and report every failure, as CI runs do Refs #100
…sses Stryker spawned one checker process per concurrency slot but checked every group on a single slot, so 321 checks ran one after another. Each group now takes its own pool slot, bounded by the pool size, and results keep plan order Refs #100
The checker pool lived in the mutation-test stage scope, so its processes shut down only after the last test run and their closes added up to three seconds to every run. The pool now has its own scope, closed in a background fiber as soon as the checks return, and the stage still waits for that close before it ends Refs #100
Every Vitest worker thread reloaded the Vitest runtime from source, and most of each mutant run went to module compilation. The CLI now enables Node's compile cache first thing and passes its directory to every worker it spawns, so the checker and test-runner processes and their Vitest threads reuse compiled code. NODE_DISABLE_COMPILE_CACHE and a preset NODE_COMPILE_CACHE still win. The library's nodePlatformLayer is unchanged Refs #100
…er checking The resolved concurrency is split between checkers and test runners, but checking ends before the first test run, so the checker share sat idle for the whole test phase. The test-runner pool now warms the test-runner share and grows on demand up to the full concurrency, and the mutant stream runs at that total Refs #100
Every test file of a vm-runner mutant run gets a brand-new Vitest worker thread, and most of that thread's life was importing Vitest's runtime before the file started. The runner now supplies its own threads pool that hands each file a fresh thread spawned in the background while the previous file ran, so isolation is unchanged and the start-up leaves the critical path Refs #100
A runtime that failed after createVitest, such as a refused browser-mode project, never closed its driver, so its standby threads and the copied stryker-setup file outlived the failure. The runtime now closes on any failure after creation and removes the setup file on any failure after the copy A spare thread is reused only when the claiming request has the same project, env and execArgv it was booted with, because the worker bakes those in at boot. Disposal waits for every stop before reporting the first failure, a boot that fails during a claim rejects start(), and a death names whether it happened before or after the claim Refs #100
The CLI called module.enableCompileCache at import, before the Node version check, so a runtime older than 22.8 crashed with a TypeError instead of reporting the unsupported version. It now feature-detects the API and runs without the cache when it is missing or reports a failure Refs #100
Properties pin the checker fan-out: results keep plan order when later groups finish first, checks spread over distinct pool slots without exceeding the pool, a crashed checker slot is replaced, and checker breaches become mutation-test stage errors The checker scope lifetime moves into checkerResourcesInOwnScope so one helper owns it: the checkers are released in the background once checking finishes, and exactly once when checking fails or the stage is interrupted Refs #100
systemfsoftware-maker
added this pull request to stack #112
September 25, 2026 22:30
…ne console
The capture layer ran every first string argument through printf-style substitution, so console.log('%%') recorded a single percent sign where Node prints the string unchanged. The merge-reports summary property found it on macOS when it drew a summary containing %%. A single string argument is now recorded as given, as Node does
Refs #100
Four e2e scenarios hit the 120 s test timeout on run 36198645464 while the same code passed e2e on the previous head; a personal token cannot rerun the job Refs #100
Base automatically changed from
refactor/vm-runner-vitest-delegation
to
main
September 25, 2026 23:31
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
A
stryker runwithtestRunner: 'vm'now takes about half the wall time, and every mutant verdict is the same as before. The saving comes from idle time: checkers now run in parallel and release their share ofconcurrencywhen they finish, and workers start faster (a shared compile cache, and a worker thread booted before the test file that needs it).Layer 3 of stack #112, on top of #111, which made
vmrun the real Vitest runner and fixed its verdicts but left it about as slow astestRunner: 'vitest'.Measurements
Enterprise fixture (4 packages, 319 mutants, TypeScript checker),
stryker.vm.config.ts, 5 interleaved pairs of the #111 head against this branch. The host load was 29-55 on 8 cores, so absolute times are noisy; the paired gap is not.A rebuild of the final head measured 10.87 to 11.42 s over 4 runs, again with 0 differences.
What each change bought, each measured against the previous best:
concurrencyto test runners after checkingTwo other experiments were measured and dropped: one scheduler streaming checked mutants straight into test runs (no gain, because the host is CPU-bound), and grouped type-checking in the TypeScript checker (11 % slower).
Design decisions
envandexecArgvmatch what it was booted with, because the thread bakes those in at boot. Vitest sends the config and environment in the start message for each file, after the claim, so a spare booted during the dry run cannot carry stale settings into a mutant run.poolRunneroption andThreadsPoolWorker, both marked@experimentalin Vitest 5.0.1. A Vitest minor could break it; the vm parity differential would catch a verdict change.testRunnerstotestRunners + checkers(idle runners retire after 1 minute) while the checker processes are still shutting down in the background. Until they exit, more processes thanconcurrencycan be alive.module.enableCompileCache()and passes the directory to every worker throughNODE_COMPILE_CACHE. It uses Node's default directory under the OS temp dir, as TypeScript's owntscdoes.NODE_DISABLE_COMPILE_CACHE=1turns it off, and on Node older than 22.8 the CLI runs without it instead of crashing before the version check.stryker-setup-*.jsfile it copied into the project.Validation
pnpm check:ci(format, lint, typecheck, every test suite including the vm parity differential, build) and the changeset gate pass on the head of this branch.New tests, each shown to fail against the behaviour it pins:
start(), and disposal waiting for every stop.stryker-setup-<uuid>.jsleft behind.Not covered by a test: the order in which the worker environment is merged, which is what lets a user's
NODE_COMPILE_CACHEorNODE_DISABLE_COMPILE_CACHEwin. The node driver that builds it cannot host in-source tests under the repo's lint rules; a smoke run confirmed that 4 test-runner workers, 4 checker workers and their Vitest threads all read the same cache directory.New concepts
Node's module compile cache. Since Node 22.8,
module.enableCompileCache()stores V8 code cache for the CommonJS and ESM modules Node loads in a directory on disk. The next process that loads the same file skips parsing and compiling it. A child process joins the same cache whenNODE_COMPILE_CACHEnames the directory.Why here: a mutation run starts dozens of short-lived processes and worker threads that each load the same Vitest, Vite and TypeScript bundles. A thread-reuse scheme would also save that time, but it would weaken per-file isolation. The cache speeds up every fresh start and leaves isolation alone.
Example:
packages/stryker-js/src/bin/enable-compile-cache.tsenables it at CLI start, and the node driver addsNODE_COMPILE_CACHEto every worker it spawns.When not to use it: when coverage depends on V8 source positions (Vitest turns the cache off itself for v8 coverage), or in a library that other code imports, where turning it on would be a side effect on the host process.
Related: #100
Need help on this PR? Tag
@codesmith-botwith what you need. Autofix is disabled.