Repository navigation
ci: shard the dev checks in the Python 3.14 test parts - #108
Merged
Merged
Conversation
The whole dev/tests suite ran inside 3.14 part 2 beside two other shards and set the length of every merge-queue run: 76-97 s, against 60 s or less for every other job (10 merge_group runs, 2026-11-09/10). Inside that job the dev suite took 57-87 s; the engine and audit-tools shards ended by 59 s. dev/test_shard.py now knows the dev suite, and test-part shards it (TEST_SHARDED_SUITES). Shard 1 is test_implement_eval alone (38 s locally); shard 2 holds the other 20 modules (14 s), so the small whole targets (test-implement-tools, verify-rules) move to part 2. The 3.11 test-dev job, coverage-part and coverage-combine are unchanged; coverage.py never measured dev/. Locally, on 4 CPUs with Python 3.14, the slower part drops from 53 s to 38 s, with the same test counts per suite. Roadmap-Step: CS6 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Bottom line
test (part 2, Python 3.14)set the length of 9 of the last 10 merge-queue runs (76-97 s). Every other job took 60 s or less, except one test-dev outlier at 84 s. This PR splitsdev/testsover the two 3.14 parts. Expected result: the 3.14 parts drop to about 60 s, and the critical path drops to about 60-70 s (inferred). It adds no job.Measured cause (checked)
dev/testssuite took 57-87 s. The engine shard and the audit-tools shard in the same job finished by 59 s (job logs, 6 runs).test_implement_evaltakes 38 s of 53 s on 3.14 and 32 s of 44 s on 3.11 (local). Each of its 58 tests builds a fresh git scenario.Change
dev/test_shard.pyknows thedevsuite;dev/test_shard_weights.jsonhas its weights (--measure).Makefile:TEST_SHARDED_SUITES := engine audit-tools dev, whichtest-partuses, and a newtest-dev-shardtarget.test-implement-toolsandverify-rulesmove to part 2: dev shard 1 istest_implement_evalalone.test-devjob,coverage-part,coverage-combine SHARDED=1(coverage.py never measureddev/), and the nightly unsharded run.dev/tests/test_ci_parity.pypins the new layout: test-part shards every runner suite; coverage-part shards a subset.Local A/B, 4 CPUs, Python 3.14 (checked)
The test counts per suite are the same (806 engine, 1102 audit tools, 395 dev, 135 implement tools).
Gates (checked, local)
make check(with workflow-lint),make security,make docs-lint,make self-audit,make coverage(92%, 9633 statements, 801 missed) andmake commit-lint RANGE=origin/dev..HEADall pass.🤖 Generated with Claude Code