Skip to content

feat(repo): judge every e2e journey by authored mutant annotations - #128

Merged
ryanleecode merged 27 commits into
mainfrom
e2e-followup
Sep 28, 2026
Merged

ryanleecode merged 27 commits into
mainfrom
e2e-followup

Conversation

@systemfsoftware-maker

@systemfsoftware-maker systemfsoftware-maker commented Sep 27, 2026 •

Copy link
Copy Markdown
Collaborator

Summary

Every e2e journey now checks each reported mutant against an outcome a person wrote beside the mutated code, instead of against numbers recorded from an earlier engine run. A misclassified status therefore fails its journey with the file, line and both outcomes, where before it was absorbed into a re-blessed baseline. The Stryker CLI and plugin interface now publish their wire formats, stream, report, span taxonomy and mutator catalog as versioned contracts, and the lane decodes only through them.

This is the follow-up to #127's review, delivered as one wave per docs/plans/2026-09-27-0424-refactor-contract-first-mutation-oracles-plan.md. Releasing waits for your approval.

What reviewers should check

  • Published surface breaks (BREAK-1). Mutant identity, statuses and locations move from the instrumenter to stryker-js-plugin-interface. New @systemfsoftware/stryker-js-cli-contract owns the stream schema, span taxonomy and stock catalog, and commits each as a JSON document under contract/. A guard fails a contract document change that has no matching version bump. Changesets: stryker-js, plugin-interface and instrumenter major; cli-contract and typescript-checker minor.
  • Two statuses gain producers. perTest coverage now reports an uncovered mutant as NoCoverage without running it. An interrupted run's checkpoint lists each unsettled planned mutant as Pending.
  • Mutator providers. A plugin can contribute mutators named <namespace>/<Name>. optInMutations and excludedMutations are validated against the loaded catalogs before instrumentation.
  • Annotation grammar. // @stryker-expect <scope> <outcome>: <mutators> lives in fixture source. CompileError(TSnnnn) and RuntimeError(Type) name their cause. An in-process placement check confirms every authored compile code with the real checker, and status closure proves every status has a witness journey.

Engine bugs the authored oracle found

Every one of these was hidden by the recorded baselines, and each is fixed here:

  • A mutant on a ternary's test printed without parens, so it replaced the whole conditional.
  • A // Stryker disable that ignored a prefix of a node's mutants shifted the remaining replacements onto the wrong mutant ids.
  • The typescript-checker's per-mutant CompileError reason dropped the TS code.
  • Machine-mode stdout carried a non-event line (the json reporter's report path).

The first two have solution docs under docs/solutions/runtime-errors/.

Seeded faults

pnpm --filter @systemfsoftware/stryker-e2e seed-status-fault <status> patches one misclassification into a scratch copy and runs the journey the witness registry assigns. A run counts as red only when the failure names that misclassification. Every status shipped by the product was caught:

Status Misclassification Witness journey Result Lane Trace
CompileError CompileError reported with a reason naming no diagnostic enterprise-composite-checker.e2e.test.ts red 158 s 26688e163a88053919a7f36d68beffb9
Ignored Directive-ignored mutant reported as Survived enterprise-mutation-lifecycle.e2e.test.ts red 276 s e109bc3699c8cbf0249ca2cb00846297
Killed Killed reported as Survived mutation-run.e2e.test.ts red 165 s 785d36c8bcdd635597ee0f4e71766df2
NoCoverage NoCoverage reported as Survived enterprise-mutation-lifecycle.e2e.test.ts red 253 s fd97f9916f143c69e10f86fee39b0a64
Pending Pending omitted from the interrupted checkpoint enterprise-runner-resilience.e2e.test.ts red 207 s be0fb03ed075b193cd457849e4797f3d
RuntimeError RuntimeError reported as Survived enterprise-mutation-lifecycle.e2e.test.ts red 260 s 047804c98e400b3ba05713d5c08c7ee5
Survived Survived reported as Killed svelte-app.e2e.test.ts red 174 s d18916e557590d25723197e71e6e17f0
Timeout Timeout reported as Survived enterprise-runner-resilience.e2e.test.ts red 193 s fd6f90bee6198a7bffb95a8a888ccaf4

Verification

  • CI on 585ed775b: both check jobs and all five e2e shards pass.
  • Full lane from a fresh bake at 585ed775b: 11 files / 16 tests passed in 297 s, under the 1200 s shard cap.
  • pnpm test across the workspace passes, and so do the format check, typecheck, check:ci, guard:projects selftests and check-changeset.

Found while getting CI green

  • The CLI lost spans against a slow collector. It exported through a SimpleSpanProcessor, one OTLP request per span (about 11k per run), and the exporter refuses any export past 30 in flight. On CI the lifecycle journey's rerun lost stryker.run_request and its write span, which cut every phase off from stryker.cli.run in the trace contract (trace 50a1eccb8432e61850b11c34e754192c). The CLI now uses a BatchSpanProcessor. A probe against a collector that answers in 300 ms received 30 of 400 spans before the change and 400 of 400 after.
  • The stream schema no longer uses report shapes with open JSON records. CI flaked ∀x_RunEventWireLine_=x because verdict lines reused report shapes whose rest records let the generator write escaped keys that V8's JSON.parse misreads (nodejs/node#63785). The verdict now uses the fixed-field VerdictThresholds and VerdictLocation structs. stream-document.differential.test.ts compares the committed document with the wire codec on lines generated from both: they must refuse the same lines and reproduce the same JSON. A scratch candidate that drops one verdict field turns it red.
  • e2e-core's generated report laws get the same 180 s budget as plugin-interface's under CI coverage. Locally they take about 5 s; on a contended runner they passed 30 s.

Dependencies and test names

  • Every @systemfsoftware/* dependency is on its latest release; the catalog:stryker dogfood entries stay latest, now resolving to stryker-js 13.1.0 and the 8.0.1 runner and checker. The differential-spec host-bound patch is ported unchanged from 0.6.0 to 0.6.1, which still runs host-bound comparisons on the simulation kernel.
  • No suite here compared a Fake with a Real, so none was a contract test. The golden-line wire codec suite is now run-event-wire-line.integration.test.ts, the report codec suite is report-codec.integration.test.ts, and the document-versus-codec parity check is a Differential.compare spec. The new test-suffix-outside-src lint rule now refuses the *.contract.test.ts suffix.
  • The new lint rules require one cell per *.cell.ts and pure imports in *.schema.ts. Helpers moved out of 12 cell files into plain modules, Checker.cell.ts split into check.cell.ts and group.cell.ts, validate-options.cell.ts split from load-config.cell.ts, and the generated report module is now generated/report.schema.ts. No published API report changed.

Residuals

  • A local api:check sometimes renders stryker-js's Plugin.Node export without the Node_2 as alias. The committed report keeps the alias, which is what CI renders on main.
  • The checkpoint re-reads every source file each time a mutant settles (Θ(M·N) per run). This PR leaves it unchanged.
  • A mutant placed in a class extends slot can hit the same missing-parens hole as the ternary test. It is documented, not fixed.

Session-settled decisions carried from planning: authored contracts that producers compile against (user-directed, over contracts generated from the producer); annotations with a coverage model (user-directed, over runner snapshots); no engine-output snapshot anywhere (user-approved); annotated causes for CompileError/RuntimeError (user-approved); one catalog model with the provider loader in this wave (user-directed); two contracts (user-approved); Effect core for schema codegen (user-approved); one wave (user-directed); NoCoverage and Pending producers (user-directed).

Related: #127


View with [code]smith Autofix with [code]smith
Need help on this PR? Tag @codesmith-bot with what you need. Autofix is disabled.

…y catalog names and add the e2e core oracles
# Conflicts:
#	.changeset/mutant-ids-are-decimal.md
#	.changeset/thresholds-are-one-schema.md
@ryanleecode
ryanleecode merged commit 6f8a052 into main Sep 28, 2026
10 checks passed
@ryanleecode
ryanleecode deleted the e2e-followup branch September 28, 2026 01:07
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants