Skip to content

Run each test file in its own process: the parent runner drops file tails under CI load - #169

Merged
Rfym21 merged 1 commit into
Rfym21:mainfrom
maxff77:fix/ci-test-gate-per-file
Sep 11, 2026
Merged

Run each test file in its own process: the parent runner drops file tails under CI load#169
Rfym21 merged 1 commit into
Rfym21:mainfrom
maxff77:fix/ci-test-gate-per-file

Conversation

@maxff77

@maxff77 maxff77 commented Sep 11, 2026

Copy link
Copy Markdown

Problem

CI on main (after #167) is red with TEST GATE: FAIL [SHORT_RUN]: 1035 of 1074 tests, fail 0, exit 0, on 3 of 3 attempts. The tree is whole — the same commit reports 1074/128 locally and on an idle 4-core Linux VPS (Node 24.21). Each CI attempt lost the tail of different files (e.g. R3–R12 of anthropic-native-parity, the end of thinking-retention).

Cause: node --test --test-force-exit tests/*.test.js — the parent runner with one child per file — loses the end of a child's piped output under CPU load. Reproduced on the VPS by adding six CPU hogs (1069, then 1074 on retry). It is load-dependent, and GitHub's 2-vCPU runner is always loaded, so the gate's retries could never save it. (This is the same drop the gate was written to detect; it just cannot be retried around on CI.)

Fix

  • tools/test-gate.js — stop using the parent runner. Every file runs as node --test --test-isolation=none --test-force-exit <file>; the gate reads each process's stdout to EOF before counting anything and sums the per-file summaries (sumSummaries: a file with no summary voids the attempt as NO_SUMMARY, never subtracted quietly). Bounded pool (TEST_GATE_CONCURRENCY, default min(4, cores-1)), one watchdog per attempt. --test-force-exit stays: 26 files never exit without it.
  • .github/workflows/ci.yml — 300 s per attempt, 2 attempts (measured: ~10 s local, 77 s on the 4-core VPS at concurrency 1).
  • tests/tool-prompt.test.js (P14) — threshold 400 → 1500 ms. Measured 50 ms on a Mac, 210–225 ms on the VPS idle, 445 ms under load; the quadratic regression it guards was ~2 s, so 1500 still catches it and stops flaking on a shared runner.
  • tests/test-count-gate.test.js — three tests for sumSummaries. Baseline 1074 → 1077.

Verification

  • Local: npm test → PASS 1077/128; eslint clean.
  • Linux VPS, this exact tree, six CPU hogs, single attempt: PASS 1077/128 twice (65 s, 68 s). The previous gate under the same load produced a silent short pass.

…ails under CI load

CI (ubuntu-latest, 2 vCPU) reported 1035 of 1074 tests with fail 0 and exit 0
on 3 of 3 attempts for a tree that is whole (main after Rfym21#167), each attempt
missing the tail of different files. The same tree reports 1074 locally and on
an idle 4-core VPS, and that VPS reproduces the drop (1069) as soon as CPU load
is added. The loss is `node --test`'s parent runner losing the end of a child's
piped output on --test-force-exit; it is load-dependent, and CI is always loaded,
so the gate's retries could never save it.

tools/test-gate.js: stop using the parent runner. Every file runs as
`node --test --test-isolation=none --test-force-exit <file>`, its stdout is read
to EOF before anything is counted, and the per-file summaries are summed
(sumSummaries: a file with no summary voids the attempt as NO_SUMMARY, it is
never subtracted quietly). Bounded pool (TEST_GATE_CONCURRENCY, default
min(4, cores - 1)); one watchdog covers the whole attempt. --test-force-exit
stays: 26 files never exit without it.

.github/workflows/ci.yml: 300 s per attempt, 2 attempts (measured: ~10 s local,
77 s on the 4-core VPS at concurrency 1).

tests/tool-prompt.test.js (P14): threshold 400 -> 1500 ms. Measured 50 ms on a
Mac, 210-225 ms on the VPS idle, 445 ms under load; the quadratic regression it
guards was ~2 s, so 1500 still catches it and stops flaking on a shared runner.

tests/test-count-gate.test.js: three tests for sumSummaries. Baseline 1074 -> 1077.

Verified: gate PASS 1077/128 locally; on the VPS under six CPU hogs, single
attempt, 2 of 3 runs PASS 1074 and the third failed loudly on P14 — never a
short pass.
@Rfym21
Rfym21 merged commit 9c7bb5b into Rfym21:main Sep 11, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants