Run each test file in its own process: the parent runner drops file tails under CI load - #169
Merged
Merged
Conversation
…ails under CI load CI (ubuntu-latest, 2 vCPU) reported 1035 of 1074 tests with fail 0 and exit 0 on 3 of 3 attempts for a tree that is whole (main after Rfym21#167), each attempt missing the tail of different files. The same tree reports 1074 locally and on an idle 4-core VPS, and that VPS reproduces the drop (1069) as soon as CPU load is added. The loss is `node --test`'s parent runner losing the end of a child's piped output on --test-force-exit; it is load-dependent, and CI is always loaded, so the gate's retries could never save it. tools/test-gate.js: stop using the parent runner. Every file runs as `node --test --test-isolation=none --test-force-exit <file>`, its stdout is read to EOF before anything is counted, and the per-file summaries are summed (sumSummaries: a file with no summary voids the attempt as NO_SUMMARY, it is never subtracted quietly). Bounded pool (TEST_GATE_CONCURRENCY, default min(4, cores - 1)); one watchdog covers the whole attempt. --test-force-exit stays: 26 files never exit without it. .github/workflows/ci.yml: 300 s per attempt, 2 attempts (measured: ~10 s local, 77 s on the 4-core VPS at concurrency 1). tests/tool-prompt.test.js (P14): threshold 400 -> 1500 ms. Measured 50 ms on a Mac, 210-225 ms on the VPS idle, 445 ms under load; the quadratic regression it guards was ~2 s, so 1500 still catches it and stops flaking on a shared runner. tests/test-count-gate.test.js: three tests for sumSummaries. Baseline 1074 -> 1077. Verified: gate PASS 1077/128 locally; on the VPS under six CPU hogs, single attempt, 2 of 3 runs PASS 1074 and the third failed loudly on P14 — never a short pass.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
CI on
main(after #167) is red withTEST GATE: FAIL [SHORT_RUN]: 1035 of 1074 tests, fail 0, exit 0, on 3 of 3 attempts. The tree is whole — the same commit reports 1074/128 locally and on an idle 4-core Linux VPS (Node 24.21). Each CI attempt lost the tail of different files (e.g. R3–R12 ofanthropic-native-parity, the end ofthinking-retention).Cause:
node --test --test-force-exit tests/*.test.js— the parent runner with one child per file — loses the end of a child's piped output under CPU load. Reproduced on the VPS by adding six CPU hogs (1069, then 1074 on retry). It is load-dependent, and GitHub's 2-vCPU runner is always loaded, so the gate's retries could never save it. (This is the same drop the gate was written to detect; it just cannot be retried around on CI.)Fix
tools/test-gate.js— stop using the parent runner. Every file runs asnode --test --test-isolation=none --test-force-exit <file>; the gate reads each process's stdout to EOF before counting anything and sums the per-file summaries (sumSummaries: a file with no summary voids the attempt asNO_SUMMARY, never subtracted quietly). Bounded pool (TEST_GATE_CONCURRENCY, defaultmin(4, cores-1)), one watchdog per attempt.--test-force-exitstays: 26 files never exit without it..github/workflows/ci.yml— 300 s per attempt, 2 attempts (measured: ~10 s local, 77 s on the 4-core VPS at concurrency 1).tests/tool-prompt.test.js(P14) — threshold 400 → 1500 ms. Measured 50 ms on a Mac, 210–225 ms on the VPS idle, 445 ms under load; the quadratic regression it guards was ~2 s, so 1500 still catches it and stops flaking on a shared runner.tests/test-count-gate.test.js— three tests forsumSummaries. Baseline 1074 → 1077.Verification
npm test→ PASS 1077/128; eslint clean.