Skip to content

feat(repo): run the e2e lane as effect specs over traces it owns - #127

Merged
ryanleecode merged 17 commits into
mainfrom
e2e-rehaul
Sep 27, 2026
Merged

ryanleecode merged 17 commits into
mainfrom
e2e-rehaul

Conversation

@systemfsoftware-maker

@systemfsoftware-maker systemfsoftware-maker commented Sep 26, 2026 •

Copy link
Copy Markdown
Collaborator

Summary

The e2e lane now runs as Effect effect-gherkin-spec features on one Effect harness. Each CLI run owns its trace: the harness hands the CLI a TRACEPARENT, and the lifecycle journey is judged by a trace-spec contract over that trace as Tempo returns it, instead of by span names found through search. To make that possible, the stryker CLI now continues a caller's trace from TRACEPARENT/TRACESTATE (minor bump for @systemfsoftware/stryker-js).

What reviewers should check

  • The CLI change is the only published surface. stryker.cli.run takes an external parent decoded through the existing Trace.Traceparent schema. An absent or malformed carrier keeps today's fresh root, as the OpenTelemetry environment-carrier spec asks.
  • No assertion was dropped. Each journey keeps its authored literals, timeouts and ORACLE-LITERALS blocks byte-identical (check:oracle-drift: no drift). gherkin-spec refuses a second assertion on one state, so each old bddStep('And', …) became a When that derives a new state followed by one composite Then.
  • The five typescript-checker-* journeys are one feature: a scenario outline over the build-mode, include and preset configs, plus the broken-checker and vm-run scenarios. It runs in 261 s, inside the rest shards' 1200 s cap, with one forked microVM at a time.
  • The collector is now mandatory. Global setup probes Tempo and fails naming pnpm lgtm:up; it sets OTEL_ENABLED=true when unset and refuses an explicit false.
  • Oracle derivations are differential-spec relations. The eight properties became Metamorphic.on / Differential.compare relations with shrinking. The oxc-vs-ts-morph comparison is hostBound rather than runBudget-only (a deviation from the plan): the oxc instrumenter needs a real Effect runtime, and on the simulation kernel it deadlocks on async finalizers.

Plan: docs/plans/2026-09-26-1939-refactor-e2e-effect-spec-libraries-plan.md.

Verification

Check Result
Full lane with LGTM (vitest run, all journeys) 11 files, 15 tests pass; 309 s, then 418 s after the review fixes (baseline before conversion: 657 s)
Lifecycle contract falsified Renaming a required span in a scratch copy of the contract gave a Break naming that conjunct, the trace id and the dump path
CLI carrier smoke against Tempo TRACEPARENT set: stryker.cli.run parented on the carried span id. Unset or malformed: fresh root
pnpm --filter @systemfsoftware/stryker-js test 76 files, 802 tests pass
test:oracle 6 files, 39 tests; each relation broken on purpose reported a shrunk counterexample
bless-oracle -- --verify edge 2/2 runs agree byte-for-byte; committed baseline unchanged
format:check, typecheck, test, check:ci, changeset gate pass

Each live journey runs its CLI twice: @systemfsoftware/vitest re-runs a passing live test once on a fresh build as a leaked-state check. Expect two owned trace ids per journey in Tempo.

New concepts

Owned-trace contracts (trace-spec). A test that asserts on spans owns the trace it reads, rather than searching the collector for one. Stimulus.make mints a trace id per run and hands it to the system as a W3C traceparent; a Contract of declared spans and relations (Rel.exists, Rel.descendant) is then judged against exactly that trace, read back through RemoteObservation until it settles. Here it replaces a span-name search over Tempo that could match another run's trace. It fits whenever the system under test runs in another process and can accept a trace carrier. It does not fit when the system cannot be told its parent context: then there is nothing to own.

Unapplied review findings

Applied in this branch: #1 (E2E-2 rule text reverted to main; the checker counts already lived in the fixture helper there) and #2 (one shared src/Harness/harness-layers.ts for the fixture, global setup and blessed-baseline.ts).

  • P1 — test/e2e/tests/fixtures/machine-stream.fixture.ts:1 — E2E-3: lane assertions import @systemfsoftware/stryker-js / stryker-js-plugin-interface as values
    Pre-existing on main (the journeys already decoded RunEvent.RunEventWireLine from the workspace package); this branch only moved the import into one fixture. Either amend E2E-3 to admit the published wire-line and report codecs or move the decode behind testResources/, as a separate owner decision.
  • P1 — test/e2e/tests/fixtures/stryker-trace.fixture.ts:95 — contract could pass on a stale trace
    Not reproduced: each StrykerRun mints a fresh random 128-bit trace id and the contract reads only that id; four lifecycle runs each produced exactly one owned trace.
  • P1 — test/e2e/tests/fixtures/trace-observation.fixture.ts:13 — Tempo 404 would end remote observation on the first poll
    Not reproduced on the pinned Tempo: GET /api/v2/traces/<unknown id> answers 200 {"trace":{},"metrics":{}}, which reads as zero spans and keeps polling. Would bite only on a Tempo that returns 404; the fix would be upstream in trace-spec's TempoTraceStore.source.
  • P2 — packages/stryker-js/src/bin/main.ts:221 — pass explicit fields to OtelTracer.makeExternalSpan
    Suggested: makeExternalSpan({ traceId, spanId, traceFlags, traceState }) instead of passing the decoded parts record.
  • P2 — packages/stryker-js/src/reporter-stream.service.ts:505 — malformed TRACEPARENT falls back to a fresh root silently
    This is the specified behaviour (plan KTD4, OpenTelemetry env-carrier spec); a warn log would be a product decision.
  • P2 — test/e2e/scripts/derive-oracle.differential.test.ts:289 — relations registered without a describe
    Each relation registers as a named vitest test (test:oracle: 6 files, 39 tests; each selectable with vitest -t <relation>).
  • P2 — test/e2e/tests/fixtures/global-setup.ts:30 — Tempo preflight and OTEL_ENABLED refusal have no automated test
    Both were exercised by hand this run (TEMPO_URL=http://127.0.0.1:1 fails naming pnpm lgtm:up; OTEL_ENABLED=false is refused).
  • P2 — test/e2e/tests/fixtures/global-setup.ts:47 — probe checks only TimedOut
    @systemfsoftware/effect-readiness 0.4.1 types the verdict as Satisfied | TimedOut, so every non-ready verdict is covered.
  • P2 — test/e2e/tests/fixtures/stryker-trace.fixture.ts:78 — lifecycle contract has no taxonomy-exclusivity conjunct
    The plan keeps the contract to the claims the old span-name check made; an exclusivity relation would be a new claim.

Review run: 20260926-211536-c46880af (ce-code-review, mode:agent; reviewers: correctness, project-standards, testing, maintainability, learnings, reliability, adversarial — in-process, no cross-model peer CLI on the host). Artifacts: /tmp/compound-engineering-0/ce-code-review/20260926-211536-c46880af/.


Compound Engineering


View with [code]smith Autofix with [code]smith
Need help on this PR? Tag @codesmith-bot with what you need. Autofix is disabled.

stryker.cli.run takes an external parent decoded from TRACEPARENT and
TRACESTATE through Trace.Traceparent; an absent or malformed carrier
keeps the fresh root span
Adds the Effect harness layer, the StrykerRun stimulus that hands the
CLI a TRACEPARENT per run, stream decoding, Tempo remote observation
and a collector preflight, and converts the mutation-run journey into
an effect-gherkin-spec feature
…ions

Ports the eight derive-oracle properties to Metamorphic.on and
Differential.compare with shrunk counterexamples; the oxc-vs-ts-morph
comparison is host-bound because the oxc instrumenter needs a real
Effect runtime
The five typescript-checker journeys become one feature: a scenario
outline over the build-mode, include and preset configs, the broken
checker scenario and the vm-run report scenario, each keeping its
authored counts
blessed-baseline runs under NodeRuntime.runMain on the harness services
and the node FileSystem, with the bake as a scoped layer; refusals of
non-verdict runs, non-zero exits and the sabotage slice are unchanged
failing-run, vm-vitest, vitest-nested-describe and svelte-app become
features on the Effect harness; persisted reports are derived states
read from the run's own fork and every authored literal is kept
edge, composite-checker, resilience and sabotage become features on
the Effect harness with their oracle literal blocks byte-identical,
so reconcile-oracle and the drift check still splice them
A test-side trace-taxonomy declares the lifecycle spans the exporter
writes; the lifecycle feature judges its one CLI run through that
contract over Tempo, replacing the hand-rolled span-name checks and
the OTEL_ENABLED gate
Every journey now runs on the Effect harness, so microvm-harness.ts and
tempo.ts go; the terminal-verdict accessor the enterprise journeys
shared moves into the stream fixture, and trace break dumps are ignored
The carrier decode changed the declaration graph, and api-extractor now
renders the exported Node without the Node_2 alias on every forced run;
the API itself is unchanged
…ctor

The lane docs name the Tempo preflight, TEMPO_URL, the owned trace id
per run, Break dumps, the targeted-run command, and the widened E2E-2
oracle-literal rule for shared verify helpers
reportedMutantsOf and runIdsIn move into the stream fixture; the
journeys and the checker helper that carried identical copies import
them
The harness fixture, global setup and the bless script build their
layers from HarnessServicesLive and HarnessPlatformLive instead of
three hand-copied compositions
The checker counts already lived in the fixture verify helper on main,
so the lane rule stays as it was rather than widening a judgment
surface
@ryanleecode
ryanleecode merged commit 6118d89 into main Sep 27, 2026
10 checks passed
@ryanleecode
ryanleecode deleted the e2e-rehaul branch September 27, 2026 03:57
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants