chore: Add backend coverage and mutation testing harness by jeromevdl · Pull Request #416 · aws-samples/sample-collaborative-ai-dlc

jeromevdl · 2026-08-21T15:28:09Z

Summary

measure backend coverage in CI and retain detailed reports
compare PR coverage with the latest successful main baseline
add Stryker mutation testing across backend domains
run an informational mutation pilot on changed backend files

Mutation PR pilot

mutates entire changed production files
ignores static mutants
skips draft PRs
cancels obsolete runs after new pushes
stops mutation execution after 30 minutes
publishes partial results after timeout
never blocks a PR because of surviving mutants
exposes HTML/JSON reports as workflow artifacts

Validation

2,498 backend tests passing
601 shared tests passing
sso-token: 23 mutants, 78.26% mutation score
lint, formatting, secret scanning and package audits passing

Follow-up verification

This PR adds the coverage reporting workflow, but GitHub only runs a workflow_run workflow after it exists on main.

After this PR is merged, the next PR modifying backend code must confirm that:

the coverage comment is posted on the correct PR;
it lists the backend files modified by that PR;
it compares their coverage against the latest successful run on main.

By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.

JWThewes

Went through this on a scratch worktree — installed deps and actually ran the harness (Stryker on sso-token, ws-authorizer, building-blocks; dry runs on shared, v2-orchestrator, untested; the changed-files script incl. --report-only; plus lint/format/coverage). Comments inline are backed by those runs.

What I like: the privileged/unprivileged split is done properly (compute on the PR token, commenting via workflow_run), actions are SHA-pinned in line with the rest of the repo, break: null means nothing can block a PR, and lint/format pass clean. Coverage trending + mutation testing are both worth having.

Why I'm requesting changes: a few concrete defects, and the expensive/risky paths were never exercised — validation only covers sso-token (23 mutants, no testcontainers).

Blockers

ignorePatterns misses terraform/ (stryker.config.mjs:32-43). Stryker doesn't read .gitignore; on my checkout terraform/ is 2.7 GB (1.0 GB of .terraform providers) and gets copied into the sandbox on every scope run.
.build/ bundles get mutated (stryker.config.mjs:12-16). The mutate glob only excludes test/**. The changed-files script filters /.build/ already.
incremental makes local scores non-reproducible (stryker.config.mjs:18). --include-static is one-way sticky — reproduced: 66.78% -> 62.42% -> 62.42% on building-blocks. Zero CI benefit since reports/ isn't cached.
--report-only destroys failure detail and reports a false cause (mutation-pr.yml:52-61). A failed scope gets reported as a 30-minute timeout, and the scope name is dropped.
No tests for 377 lines of new script code (scripts/). scripts/test/ has tests for all four existing scripts, run in CI via npm run test:release.

Strongly want

One infrastructure scope validated in CI — concurrency: 4 means 4 gremlin + 4 DynamoDB-Local JVMs on a 4-vCPU runner, and that's untested.
infrastructureLambdas in vitest.config.js derived or guarded by a test (correct today, will rot).
Harness files added to mutation-pr.yml's paths, plus workflow_dispatch (both are existing conventions here).
.stryker-tmp in .oxlintignore.
if-no-files-found: error on the coverage upload.

Nice to have

Drop the untested scope — 902 mutants + Docker to learn what all: true coverage already reports.
Dedupe the two scripts (~2/3 overlap, and lambda enumeration is a third copy of vitest.config.js).
Temper the npm run test:mutation README line — shared alone is 13,890 mutants.

Happy to pair on the ignorePatterns inversion if that's easiest.

JWThewes · 2026-09-01T12:55:25Z

+  ignorePatterns: [
+    '.cache/**',
+    '.claude/**',
+    '.codegraph/**',
+    '.idea/**',
+    '.kiro/**',
+    '.opencode/**',
+    '.venv/**',
+    '.worktrees/**',
+    'frontend/**',
+    'site/**',
+  ],


Blocker. Stryker doesn't read .gitignore — per the shipped schema the only always-ignored paths are node_modules, .git, /reports, *.tsbuildinfo, /stryker.log, .stryker-tmp. So terraform/ gets copied into the sandbox on every scope run. On my checkout that's 2.7 GB (1.0 GB of .terraform provider binaries), ×29 scopes for a full run.

Also .codegraph, .kiro, .opencode, .worktrees don't exist in this repo, while the 2.7 GB dir that does isn't listed — the denylist is going to rot. Can we invert it?

ignorePatterns: ['**', '!lambda/**', '!test/**', '!package.json', '!package-lock.json', '!vitest*.config.js']

Missing today beyond terraform: docs/ (3.5 MB), lambda/*/.build/, .husky/.

JWThewes · 2026-09-01T12:55:25Z

+const mutate =
+  explicitMutate ??
+  (scope === 'untested'
+    ? untestedDirectories.map((directory) => `lambda/${directory}/**/*.js`)
+    : [`lambda/${scope}/**/*.js`, `!lambda/${scope}/test/**`]);


Blocker. lambda/${scope}/**/*.js doesn't exclude .build/, which terraform populates with bundled Lambda output (.gitignore:55, and it's in .oxlintignore too). Combined with the ignorePatterns gap above, anyone who has run terraform and then npm run test:mutation will mutate minified bundles.

run-changed-backend-mutations.mjs already filters /.build/ — can we apply the same exclusion here?

JWThewes · 2026-09-01T12:55:25Z

+    ? untestedDirectories.map((directory) => `lambda/${directory}/**/*.js`)
+    : [`lambda/${scope}/**/*.js`, `!lambda/${scope}/test/**`]);
+const reportBase = process.env.STRYKER_REPORT_BASE ?? `reports/mutation/${scope}`;
+const incremental = process.env.STRYKER_INCREMENTAL !== 'false';


Blocker — please drop incremental. reports/ is gitignored and not cached in CI, so this buys us nothing there, but it makes local scores non-reproducible. run-backend-mutations.mjs:65-66 deletes ${scope}.json and .html before each run but not ${scope}-incremental.json.

Reproduced on building-blocks:

--scope=building-blocks -> 66.78% --scope=building-blocks --include-static -> 62.42% --scope=building-blocks -> 62.42% <- should be 66.78 INFO IncrementalDiffer Result: 330 of 330 mutant result(s) are reused.

So --include-static is one-way sticky. Either drop incremental entirely or rmSync the incremental file alongside the other two.

JWThewes · 2026-09-01T12:55:25Z

+  },
+  mutate,
+  coverageAnalysis: 'perTest',
+  concurrency: 4,


Nit: hardcoded. os.availableParallelism() - 1? Related: on a 4-vCPU runner this means 4 test-runner processes, which multiplies the testcontainer count for the infra scopes (see my note on vitest.mutation.config.js).

JWThewes · 2026-09-01T12:55:25Z

+const selectedProjects =
+  scope === 'untested'
+    ? backendProjects
+    : backendProjects.filter((project) => project.name === scope);


Suggest dropping the untested scope. Selecting all 28 projects forces globalSetup + Docker. I measured 902 mutants across 7 files — and those 7 are standalone API handlers nothing else imports, so every mutant comes back NoCoverage. Coverage already tells us that for free (all: true, include: lambda/**/*.js).

Same cost lands in the PR path: touching lambda/users/index.js (326 lines) spins up the full suite just to conclude "still untested".

JWThewes · 2026-09-01T12:55:26Z

+const percent = (metrics) =>
+  metrics.totalMutants === 0 ? 'n/a' : `${metrics.mutationScore.toFixed(1)}%`;


Returns NaN% when totalMutants > 0 but everything is ignored — mutationScore is NaN in that case, and the === 0 guard doesn't catch it.

JWThewes · 2026-09-01T12:55:26Z

+
+  for (const [scope, files] of filesByScope) {
+    const reportPath = join(reportRoot, `${scope}.json`);
+    if (!existsSync(reportPath)) {


Nothing clears reports/mutation/pr/*.json before a run. If a scope's Stryker run fails locally, this existsSync picks up the previous run's report and the summary presents stale numbers as current. Fine in CI (fresh checkout), but a footgun locally — run-backend-mutations.mjs:65-66 does clean up, so let's be consistent.

JWThewes · 2026-09-01T12:55:26Z

 # Test coverage output
 coverage/
+reports/
+.stryker-tmp/


.stryker-tmp should go in .oxlintignore too. Stryker leaves it behind when a run crashes, and oxlint would then lint 500+ sandbox copies. coverage and lambda/*/.build are already in there.

JWThewes · 2026-09-01T12:55:26Z

 npm test                 # run all unit tests
 npm run test:coverage    # run tests with a coverage report (HTML in coverage/)
+npm run test:mutation:dry # validate the mutation-testing harness without running mutants
+npm run test:mutation     # mutate all backend JavaScript sources


This isn't really a runnable command. Measured mutant counts on this branch:

Scope Mutants Note

shared 13,890

v2-orchestrator 4,050 baseline test run alone takes 61 s

untested 902 full suite + Docker

building-blocks 330 15 s end-to-end

Extrapolating from building-blocks, v2-orchestrator on its own is hours. Can we either add an explicit "expect many hours, use --scope=" warning or drop this line and document only the scoped form?

JWThewes · 2026-09-01T12:55:26Z

+Backend coverage includes untested Lambda source files and produces text, HTML,
+LCOV, JSON summary, and JSON detail reports. CI retains the full HTML report as
+a workflow artifact. Every successful push to `main` also publishes the latest
+coverage summary as the comparison baseline. Pull requests receive a
+non-blocking coverage comment showing the trend for changed backend files. The
+first PR after these workflows reach `main` must confirm that the third-party
+report action correctly resolves changed files from its `workflow_run` context.
+Mutation testing uses StrykerJS with the existing Vitest tests across all Lambda
+projects. It runs each Lambda domain sequentially to keep the instrumented
+process bounded, writes per-domain HTML and JSON reports under
+`reports/mutation/`, then generates `reports/mutation/summary.json`. Pass a
+single domain when needed, for example
+`npm run test:mutation -- --scope=v2-orchestrator`. Static mutants are ignored
+by default because they require reloading the test environment and dominate the
+runtime on the largest domains. Pass `--include-static` for an exhaustive run.
+Mutation scores are initially informational so the first runs establish a
+baseline before any blocking threshold is chosen. Pull requests that change
+backend JavaScript also run an informational mutation campaign over each
+changed production file. The PR comment uses the official Stryker metrics and
+highlights surviving or uncovered mutants without blocking the pull request.
+Draft pull requests are skipped, static mutants are excluded, previous runs are
+cancelled after a new push, and the pilot stops after 30 minutes. A timeout
+produces a partial informational report rather than failing the pull request.


27 lines in essentially one paragraph, mixing local commands, CI mechanics, the static-mutant rationale and a follow-up TODO. The rest of this file is short paragraph + code block. Worth splitting into a few paragraphs, or moving the CI mechanics over to CONTRIBUTING.md.

The "first PR after these workflows reach main must confirm..." sentence can go — see my note in coverage-report.yml.

test: add backend coverage and mutation reporting

ac306ce

jeromevdl requested review from JWThewes, eipasteur, leandrodamascena and svozza as code owners August 21, 2026 15:28

JWThewes requested changes Sep 1, 2026

View reviewed changes

Navigation Menu

Uh oh!

chore: Add backend coverage and mutation testing harness - #416

chore: Add backend coverage and mutation testing harness#416
jeromevdl wants to merge 1 commit into
mainfrom
chore/improve-test-harness

jeromevdl commented Aug 21, 2026

Uh oh!

JWThewes left a comment

Uh oh!

JWThewes Sep 1, 2026

Uh oh!

JWThewes Sep 1, 2026

Uh oh!

JWThewes Sep 1, 2026

Uh oh!

JWThewes Sep 1, 2026

Uh oh!

JWThewes Sep 1, 2026

Uh oh!

JWThewes Sep 1, 2026

Uh oh!

JWThewes Sep 1, 2026

Uh oh!

JWThewes Sep 1, 2026

Uh oh!

JWThewes Sep 1, 2026

Uh oh!

JWThewes Sep 1, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

		const percent = (metrics) =>
		metrics.totalMutants === 0 ? 'n/a' : `${metrics.mutationScore.toFixed(1)}%`;

Scope	Mutants	Note
`shared`	13,890
`v2-orchestrator`	4,050	baseline test run alone takes 61 s
`untested`	902	full suite + Docker
`building-blocks`	330	15 s end-to-end

Uh oh!

Conversation

jeromevdl commented Aug 21, 2026

Summary

Mutation PR pilot

Validation

Follow-up verification

Uh oh!

JWThewes left a comment

Choose a reason for hiding this comment

Blockers

Strongly want

Nice to have

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants