diff --git a/.agents/docs/2026-10-02-pr-ci-acceleration-and-the-toolchain-specification-design.md b/.agents/docs/2026-10-02-pr-ci-acceleration-and-the-toolchain-specification-design.md new file mode 100644 index 00000000..2f2647d6 --- /dev/null +++ b/.agents/docs/2026-10-02-pr-ci-acceleration-and-the-toolchain-specification-design.md @@ -0,0 +1,1039 @@ +--- +subject: design +status: active +--- + +# PR CI acceleration and the toolchain specification (#756, #757, #669) + +- Date: 2026-10-02. Status: proposed for review; revision 2. Nothing in this record is implemented. +- Base: `main` at `4d81d062` (2026.10.1.3). +- Inputs: + - CI jobs 110514498085, 110514498447 and 110514498760 on `main`. + - Issues #756, #757 and #669, with #669's 2026-10-01 measurement. + - A census of every place mcpp states a toolchain version. + - The packaging pipeline of the `gcc` and `llvm` payloads, and the upstream state of GCC 16.2 and + LLVM 23. + - The GitHub Actions record of seven head commits: four pull-request heads (#758, #754, #752, #745) and + three pushes to `main` (`4d81d062`, `68e49981`, `23c9590b`). Every job, step, queue time, cache lookup + and ninja count of these runs was measured. + +**Revision 2.** Revision 1 proposed moving the default toolchains to GCC 16.2 and LLVM 23. Review on +2026-10-02 settled the toolchain work as a specification only: SPEC-009 is written, and no version moves +in this work (Part V keeps the analysis for the move that will follow it). The subject of this record +therefore became the time a pull request waits for CI. That time is dominated by work the CI repeats, +and the triage of the three red jobs that started this record was the first evidence of it. + +Reading order: +- Part I triages the inputs. +- Part II is the main design: where PR CI time goes, the rules that remove the repetition, and an optional + external lane. +- Part III states the defects the next release repairs. +- Part IV is the toolchain specification, as approved. +- Part V keeps the toolchain candidates for later. +- Part VI orders the work, Part VII lists what is not yet measured, and Part VIII asks the open questions. + +## 0. Decisions + +### 0.1 Settled in review (2026-10-02) + +| # | Decision | +|---|---| +| T1 | The toolchain specification (Part IV) is published as a new specification, SPEC-009; SPEC-006 keeps identity, selection and the payload contract, and its §7 becomes a reference to SPEC-009 | +| T2 | No default toolchain moves in this work. GCC 16.2 and LLVM 23 are candidates (Part V), each subject to the gate of TS-10.5 (G1 to G6, tolerance 10 percent) | +| T3 | LLVM moves only to 23.1.3, once it passes the gate; 23.1.2 is measured as an early reading; `23.1.2-r1` only on explicit request | +| T4 | A default that an earlier mcpp recorded on a first run is announced once when a newer default exists, and the user chooses to move or keep it (TS-11.2); a fresh install announces nothing | +| T5 | When LLVM moves, the bare-metal and iOS rows lag with the reason recorded | +| T6 | GCC 16.2 is a candidate only if Bug 126577 does not reproduce in mcpp's own build and suite; otherwise the GCC line waits for 16.3 | +| T7 | The GCC recipe and builder live in `xlings-res/gcc`, as `musl-gcc`'s do | +| T8 | The macOS e2e step limit rises now; Part II replaces it with measured shards | + +### 0.2 Requested now + +| # | Decision | Recommendation | +|---|---|---| +| D1 | Each commit builds mcpp once per host, and every other job consumes that build (§2.4.1) | Yes | +| D2 | One top-level workflow orchestrates the stages; per-area work moves into reusable workflows (§2.4.2) | Yes | +| D3 | A pull request that changes only documentation runs the documentation checks and nothing else (§2.4.3) | Yes | +| D4 | Only pushes to `main` write caches; one job writes each key; the `target/` caches are removed (§2.4.4) | Yes | +| D5 | The e2e suite is sharded by measured duration, and a shard's step limit follows from its budget (§2.4.5) | Yes | +| D6 | Every e2e test runs on some host or carries a recorded reason, checked in CI; the `llvm` capability is granted (§2.4.6) | Yes | +| D7 | Legs that are known red run on `main` and on dispatch, not on pull requests (§2.4.7) | Yes | +| D8 | An external lane under `speak-agent` is designed now and built only if the rule of §2.5.5 holds after D1 to D7 | Yes | +| D9 | The next release repairs #757 and #756 (Part III) | Yes | +| D10 | The single line table of TS-3 waits for the first toolchain move | Yes | + +## Part I. Triage + +### 1.1 The three CI jobs + +| Job | Leg | Failing step | Cause | Class | +|---|---|---|---|---| +| 110514498085 | `ci-macos`, `xcode-27` (known red #669) | Test: non-module C++23 compilation | `ld64.lld` 22.1.8 cannot load `libc++.tbd` and `libSystem.tbd` of the Xcode 27 SDK (`unknown architecture`, `arm64e.x1`); every undefined symbol after it is a consequence | external, tracked by #669 | +| 110514498447 | `ci-macos-e2e`, `xcode-27` (known red #669) | Build mcpp from source | the same signature while linking `bin/mcpp` | external, tracked by #669 | +| 110514498760 | `ci-macos-e2e`, `macos-15` | E2E suite | the step reached its 25-minute limit at test 867 of 877; every earlier test passed, and test 867 was still printing readings when it was stopped | a time budget in this repository | + +The third job is not a hang. The suite started at 18:30:52 and was stopped at 18:55:54. The last green +run of the same suite (36895492498, the head of #755) took 21 min 01 s, so the budget left 19 percent +for runner variance, and this runner used more. The workflow's header still states that the suite +takes about 3.5 minutes on macOS and is therefore not sharded (`ci-macos-e2e.yml:6-7`); that was true +when it was written and is now false by a factor of six. + +None of the three is a usage error or a non-conforming project. + +### 1.2 #756: a provider's `.ixx` host module is invisible to the fast path + +**Conformance.** The reproducer conforms to the manifest specification. `.ixx` is opt-in through +`[build] module_extensions` (the built-in table is `.cppm` only, `modules/source-kind/src/source_kind.cppm:121`); +`host-module = true` is the Cargo-inherited kebab key and `module_extensions` the mcpp-owned snake key +(SPEC-004, naming table); `[lib] path` and `mcpp::define` (`src/build/hostprogram.cppm:48`) exist. The +workaround the report describes, declaring `.ixx` in the consumer, is excluded: the consumer has no +`.ixx` source, and the dead-entry warning it then receives is correct. + +**Mechanism.** `fast_path_identity` builds one extension table from the consumer's effective manifest +(`src/build/execute.cppm:1418-1437`). `dep_sources_newer_than` (`execute.cppm:1172-1215`) classifies +every file under every path-dependency root with that table. A `.ixx` file that only its own package +declares is classified as not affecting the graph's shape and is skipped. The host module is compiled +into the consumer's build program, which no edge of `build.ninja` names, so ninja cannot see the edit +either, and the fast path replays the old graph with the old build program. + +Classification is a property of the package that owns the file; the sweep asks a different package. +The mechanism does not depend on the platform, although the report was made on Windows. + +**Verdict.** An engine defect. Repair in §3.2. + +### 1.3 #757: an upgraded engine replays a graph that names the removed one + +**Conformance.** Upgrading mcpp through xlings and removing the previous version is ordinary use. + +**Mechanism.** The emitter writes the absolute path of the running engine into `build.ninja` for the +`__action` wrapper and for `stage` (`src/build/ninja_backend.cppm:3502-3512`), and states why that is +safe: "A version change regenerates the file (the version is in the fingerprint)." The fingerprint does +contain the version (`modules/toolchain-model/src/fingerprint.cppm:125`), but no fast path computes a +fingerprint. `try_fast_build` (`execute.cppm:1594`), `try_fast_workspace_build` (`:1749`) and +`try_fast_run` (`:1884`) select a recorded entry by target, profile, cache mode, features and toolchain +request, and then compare the recorded fingerprint with the recorded directory's basename, which is the +record compared with itself. `BuildCacheEntry` (`execute.cppm:150`) records no engine identity. The +emitter assumed a property that the fast path never promised. When the old executable still exists the +failure is silent instead: a newer front end drives the actions of an older engine. + +**A structural observation.** The three fast paths carry 23, 30 and 30 hand-written declines. A +property added to one must be added to the others by hand, and every field of `BuildCacheEntry` carries +its own paragraph explaining what its absence means. #757 is the case in which no path received the +property at all. + +**Verdict.** An engine defect. Repair in §3.1. Recognising `CreateProcess failed` in +`is_stale_ninja_failure` is rejected: it is a criterion over error text, and a launch failure of a +program the user declared must not be retried as a stale graph. + +### 1.4 #669: the macOS 27 link + +- The Xcode 27 SDKs list `arm64e.x1-macos` and `arm64e.x1-maccatalyst` in their `.tbd` stubs. + `ld64.lld` up to 23.1.2 rejects them (`unknown architecture` in 22.1.8, `unknown target` in 23.1.2). +- The fix, llvm/llvm-project#222721, merged to `main` on 2026-09-11. It reached `release/23.x` on + 2026-09-29 as `532fa5afbe2b`, `8f747d63ffb2` (lldb) and `ee66426152f9` (no ABI break), after + `llvmorg-23.1.2` was tagged on 2026-09-22. The 23.1.2 announcement schedules 23.1.3 for Tuesday + 2026-10-06. +- Measured (speak-agent/llvm-macos27-lab, run 36878561820): an `ld64.lld` built from `release/23.x` at + `21ef2ddb8060`, with every other component taken from the official 23.1.2 package, links and runs C, + C++23 and `import std` on the `xcode-27` image (macOS 27.0, Xcode 27.0, SDK 27.0). The stock 23.1.2 + `ld64.lld` fails there; both link on `macos-15` (SDK 15.5). The Command Line Tools `MacOSX26.5.sdk` + does not list the slice, so the failure follows the SDK, not the operating system. +- Other distributors carry the backport as a patch: Homebrew's `llvm` formula from 23.1.1_1, and + hermetic-llvm for 22 and 23. Apple's own linker reads the slice (third-party reports, not Apple + documentation). + +It is neither an mcpp defect nor the SDK-selection defect that xim-pkgindex#858 repaired. + +### 1.5 What the census found + +1. **The host defaults are older than the title of this work presumes.** macOS and Windows with MSVC + default to `llvm@20.1.7` (`modules/toolchain-model/src/triple.cppm:1045`, `:1049`); Linux on an + architecture other than x86_64 defaults to `gcc@15.1.0-musl` (`:1057`). `llvm@22.1.8` appears only as + the pin of 17 target rows and as mcpp's own `[toolchain] macos`. +2. **The macOS default cannot build mcpp.** libc++ 20's `std` module does not expose the comparison of + `directory_iterator`, so `llvm@20.1.7` cannot compile mcpp's sources (`.github/workflows/ci-linux.yml:278-284`); + mcpp's own manifest therefore builds with 22.1.8 while it gives its users 20.1.7. +3. **Defaults are stated in two tables and copied widely.** Table A is `pins::kFirstRun*` with duplicate + `kSuggest*` literals (`triple.cppm:1034-1081`); table B is `kKnownTargets[].pin` (`triple.cppm:447-845`). + Beyond them: 21 literals in nine source files, `mcpp.toml`, 175 rows of `tests/matrix/expected.tsv`, + seven workflows, one action, six CI tools, at least eight e2e scripts, examples, and 33 documentation + files. Two couplings are checked: `kFirstRunWinGnu` against the `x86_64-windows-gnu` row + (`tests/unit/test_windows_defaults.cpp:55-61`), and the host default against four documentation + statements (`.github/tools/check_default_toolchain_docs.py`). +4. **Two capability probes name payload versions.** `tests/e2e/run_all.sh:70-71` grants `musl` only if + `musl-gcc/15.1.0` is installed, and `:78-79` grants `mingw-cross` only for `mingw-cross-gcc/16.1.0`. + When the default moves, sixteen tests skip and report nothing. +5. **A first-run default is written once.** `src/build/prepare/toolchain.cpp:1607` persists it to + `config.toml`, and nothing revisits it. A default that moves therefore reaches fresh homes only. +6. **The pack ABI tag carries the compiler major** (`src/pack/abi_tag.cppm:215-224`). GCC 16.1 to 16.2 + keeps `gcc16-libstdcxx16`; LLVM 22 to 23 changes `clang22-libcxx22` to `clang23-libcxx23`, and a + prebuilt artefact with the old tag is refused (`src/pack/prebuilt.cppm:117`). `mcpp-index`'s + `main` holds no such artefact today. +7. **The GCC payload has no recipe that runs anywhere but one machine.** 16.1.0 was built by hand from + the `fromsource` recipe (`make -j8`, about 28 minutes), then stripped and its `specs` rewritten by + steps no repository scripts; SPEC-006 §5.1 already records the gap. `musl-gcc` and + `aarch64-linux-musl-gcc` have dispatchable builder workflows in `xlings-res`; `mingw-cross-gcc` and the + Windows-host Canadian cross are manual; `mingw-gcc` mirrors winlibs, which publishes 16.2.0. + +## Part II. PR CI acceleration + +### 2.1 What was measured + +Seven head commits were measured from the GitHub Actions API and the job logs. The four pull-request heads +are the final heads of #758 (`51b66cec`), #754 (`71d09f0b`), #752 (`72e238dc`) and #745 (`ac4beb88`). +The three pushes to `main` are `4d81d062`, `68e49981` (documentation only) and `23c9590b`. Only runs with +event `pull_request` or `push` are counted. The release workflow, which a tag push starts beside the CI of +the same commit, is reported separately. The repository has no required status check, so a pull request +waits for the last job to finish; the four pull requests were merged between 32 s and 183 s after their +last job ended. + +| | #758 | #754 | #752 | #745 | `4d81d062` | `68e49981` (docs) | `23c9590b` | +|---|---|---|---|---|---|---|---| +| jobs | 47 | 45 | 46 | 46 | 35 | 35 | 35 | +| wall clock, first attempt (min) | 49.6 | 42.0 | 37.1 | 35.7 | 59.5 | 42.9 | 27.0 | +| wall clock with reruns (min) | 79.5 | 42.0 | 37.1 | 50.7 | 59.5 | 42.9, left red | 27.0 | +| wall clock without queueing, modelled (min) | 36.3 | 28.6 | 26.7 | 25.5 | 59.4 | 33.5 | 27.0 | +| runner minutes | 568 | 477 | 467 | 501 | 478 | 436 | 417 | +| builds of mcpp from source | 37 + 6 | 35 + 6 | 36 + 6 | 36 + 6 | 30 + 6 | 30 + 6 | 30 + 6 | + +The second figure in the last row counts the deliberate rebuilds after `mcpp clean` or with another +toolchain. + +Where the runner minutes go, as a share of each commit's total: + +| category | share | +|---|---| +| building mcpp (primary builds, then the deliberate rebuilds) | 34-40 percent, then 8-11 percent: 44-49 percent together | +| the e2e suite | 18-22 percent | +| `mcpp test` | 6-7 percent | +| setup actions | 4-14 percent; the high values are `Install wine` (below) | +| cache saves | 1-5 percent | +| everything else | 13-17 percent | + +Median duration of the primary build step, over all seven commits: + +| host | median | range | +|---|---|---| +| Linux | 6.6 min | 4.1 to 16.3 min; the maximum is the aarch64 fresh install | +| Windows | 4.8 min | 2.2 to 6.0 min | +| macOS | 2.7 min | 1.9 to 4.9 min | + +### 2.2 What the measurements show + +**F1. Every job builds mcpp, and no job reuses another's build.** +- Thirty to thirty-seven jobs per commit compile mcpp from source with the pinned bootstrap. +- Only three hand-offs of a built binary exist: ci-windows `build-test` to `no-msvc-fallback` + (`ci-windows.yml:113-165`), the openkal `run` legs, and the Windows-to-Linux cross build. +- The present shape was chosen deliberately (`ci-linux.yml:8-15`), on the premise that a warm rebuild costs + about 2.5 minutes, which is cheaper than serialising the legs behind a shared artifact. The premise no + longer holds: the measured median is 6.6 minutes on Linux, and F2 shows that the rebuild is not warm. + +**F2. The `target/` cache makes no build incremental.** +- On the documentation-only commit `68e49981`, the compiled inputs were identical to the previous commit's, + and 14 jobs restored their `target/` by exact key. In the Linux `build + unit tests` job of that commit, + ninja still ran 830 of 830 edges. +- On #758, a restore-key hit ran 834 of 834. +- In #758's `toolchain: gcc`, the first build (with 94 MB of restored `target/`) took 362.75 s. The cold + rebuild after `mcpp clean` took 381 s, only 5 percent more. +- Build steps after an exact hit are no faster than after a miss (Linux 488 s against 340 s, where the six + exact hits also missed the sandbox; Windows 293 s against 324 s). +- The cause is not measured. Checkout giving every source a modification time newer than the restored + outputs is the likely one. + +**F3. The caches evict each other.** +- The repository held 10.78 GB of caches against a 10 GB limit. +- A pull-request run saves 34 to 42 caches, of 17 MB to 1.6 GB each; `target/` caches alone range from + 46 MB to 3.3 GB. 26 job identifiers form 26 `target/` lineages on Linux. +- In 155 job logs: + - the sandbox cache missed 44 percent of lookups and the xlings cache 36 percent; + - a Windows sandbox key that hit at 17:02 and 17:11 was gone by 17:44; + - after #758's saves, the next push to `main` missed 20 of 26 sandbox lookups. +- Parallel jobs race to save one key: two to twenty-seven "Unable to reserve cache" failures per commit. +- A cancelled run saves nothing, and every CI workflow cancels in progress, including on `main`. + +**F4. The 20-job limit is reached by this repository alone.** +- On each of the seven commits, the peak number of running jobs was exactly 20; macOS reached its own + limit of 5 on three of them. +- Sixty of the 67 Linux and Windows jobs that queued 90 s or more started within 8 s of another of our + jobs finishing, which is the signature of a slot cap. +- Queueing added 9 to 13 minutes of wall clock on five of the seven commits. +- macOS jobs queued up to 12.7 minutes. 15 of the 18 macOS jobs that queued over two minutes carry + GitHub's capacity notice, so part of the macOS wait is GitHub's capacity rather than ours. + +**F5. The e2e suites are the last jobs, and they hit their limit.** +- An e2e shard was the last job on three of the seven commits (Windows 2/2 at 27 to 34 minutes; Linux 2/2 + at 28.6). +- The 25-minute step limit was reached on three of the seven commits, and a fourth failed a test that + passed on rerun. Two were rerun, at +30 and +15 minutes of wall clock; two stayed red on `main`. +- On Windows, about 15 tests of over 30 seconds take 58 to 64 percent of a 15 to 18 minute shard. The same + tests take about one second on Linux, for example `36_llvm_toolchain` and + `687_the_vendored_xlings_probe_is_an_argument_vector`. +- Shards are assigned round-robin (`tests/e2e/run_all.sh:379-414`), and the two shards differ by 2.7 + minutes on average on Linux and 3.6 on Windows. +- macOS is not sharded: one job of 242 tests runs 17 to 21 minutes. The workflow header still gives + 3.5 minutes (`ci-macos-e2e.yml:6-7`). + +**F6. A documentation-only commit costs as much as a code commit.** +- No workflow has a `paths-ignore`. +- `68e49981`, which changed two files under `.agents/docs`, ran 35 jobs and 36 builds in 436 runner + minutes, against 417 for the code commit before it, and it ended red on a Windows shard timeout. + +**F7. One evicted cache costs up to fifty minutes.** +- When the `wine-debs` cache is evicted, `apt` downloads the Wine packages at stalled speeds: + - 25.8 minutes on #758; + - 49.9 minutes on `4d81d062`, where libwine (105 MB) alone took 841 s; + - 40 seconds when the cache hits. +- On two of the seven commits this made `mingw-cross-wine` the last job (59.5 minutes of wall on + `4d81d062`). + +**F8. Known-red legs spend macOS slots on pull requests.** +- The two `xcode-27` legs fail on every commit (#669), after 2 to 4 minutes, each holding a macOS slot. +- A pull request cannot change their outcome unless it is about #669. + +**F9. Seven e2e tests never run in CI.** +- `run_all.sh` lists `llvm` as a capability a test may require (`run_all.sh:263`), but no line grants it. +- Of the 32 tests that require it, 25 are invoked directly by name in a dedicated job. +- Seven are invoked nowhere: 134, 135, 136, 137, 741, 875 and 876. Among them are 875 and 876, the tests + #755 added for a toolchain named by path and for the toolchain phase. +- Their skip lines read like legitimate ones. + +### 2.3 Rules + +These rules govern PR CI. They are stated so that a later change of the CI can be checked against them. + +| Rule | Statement | +|---|---| +| R1 | A commit builds mcpp from source once per host; every job that needs the commit's mcpp consumes that build. A job builds mcpp again only when the rebuild is what it tests: another toolchain, a cold build, a cross target. | +| R2 | What a job runs follows from what the commit changed. A change that cannot affect a job's outcome does not start the job. | +| R3 | A cache has one writer, the `main` lineage. Pull requests restore; they do not save. A cache that does not shorten the job that restores it is removed. | +| R4 | A job's time limit follows from its measured duration; a limit that is reached without a hang is a budget defect, not a test failure. | +| R5 | Every test runs on at least one host, or carries a recorded reason why no host runs it; CI checks this. | +| R6 | The scarcest resource is a macOS slot (5 per organisation). A macOS job runs only work that needs macOS. | +| R7 | A pull request waits only for work that can change its outcome. | + +### 2.4 The design inside the repository + +#### 2.4.1 One build per host (R1) + +Each commit has one build job per host: + +| host | runner | +|---|---| +| Linux x86_64 | ubuntu-24.04 | +| Linux aarch64 | ubuntu-24.04-arm | +| macOS arm64 | macos-15 | +| Windows x86_64 | windows-latest | + +A build job: +1. restores the sandbox; +2. runs the bootstrap; +3. builds the commit's mcpp; +4. packs it with `mcpp pack`, as ci-windows `build-test` already does for `no-msvc-fallback` + (`ci-windows.yml:68-119`), so that the artifact carries its own runtime closure; +5. uploads the result as an artifact. + +Every consumer downloads and unpacks that artifact and uses it as `$MCPP`. A consumer still restores the +sandbox when its tests need toolchain payloads. The `find … -newer mcpp.toml` and newest-binary selections +that protect against a stale cached binary (`ci-target-matrix.yml:59-63`, `ci-windows.yml:43-48`, +`openkal-cross.yml:150-180`) are removed, because the binary no longer comes from a restored `target/`. + +Jobs whose subject is a rebuild keep it, and start from the artifact instead of a bootstrap build: +- the GCC cold rebuild; +- the musl and LLVM builds; +- the cross builds; +- the macOS rebuild with the LLVM default; +- the Windows LLVM rebuild. + +This removes the bootstrap build that each of them pays first. + +`mcpp test` keeps a job of its own per host in the first wave. It compiles the code under test and the +144 unit-test programs itself, so the artifact would spare it only the bootstrap build, at the price of +waiting for the build stage. On Linux it is the longest job today: 24.7 minutes, of which 8.2 are the +build and 15.4 the tests. If it becomes the critical path, it splits into the root suite and the +per-member suites (`ci-linux.yml:196-218`). + +The target-matrix and openkal legs on `macos-14` and `windows-2022` consume the artifact of their host +family. For `macos-14` this holds only if the artifact's minimum macOS version is at most 14 (Part VII). +Whether these legs must build with `--dev` is not recorded anywhere; they move to the artifact unless a +reason is found (Part VII). The target matrix's `invariants` and `scan` become one job per host, because +`scan` already waits for `invariants` on the same host (`ci-target-matrix.yml:233`). + +Counted per pull-request commit, the change is: +- builds: from 41 to 43 down to about 16, which is four artifact builds, three `mcpp test` jobs, and about + nine rebuilds that are a job's subject; +- build runner minutes: from about 235 down to about 90. + +#### 2.4.2 One workflow, stages in order (R1, R7) + +An artifact is shared without extra machinery only within one workflow run. The CI therefore becomes +one top-level workflow, `ci.yml`, with three stages: + +1. `changes`: classify the commit's paths (§2.4.3), in seconds. +2. `docs`: the text checks that need no binary. They are the 14 checks `build-test` runs today before + its bootstrap (`ci-linux.yml:55-158`). +3. `build`: a matrix over the four hosts. + +The present per-area workflows become reusable workflows that the top-level workflow calls with the +artifact names: Linux, Linux e2e, Windows, Windows e2e, macOS, macOS e2e, iOS, target matrix, cross +build, openkal cross, and the MSVC xlings test. Their jobs keep their names, so that the checks a reader +knows remain recognisable. Path-gated workflows (`ci-aarch64-fresh-install`, +`measure-windows-tool-crt`, `pypi-publish`) stay separate. + +The wall clock of a pull request becomes the sum of three stages: + +| stage | duration | +|---|---| +| `changes` | under a minute | +| the slowest build | about 10 minutes: Linux setup, a 6.6-minute build, packing, upload | +| the slowest consumer | setup, plus the longest shard of §2.4.5 | + +The first wave of jobs is the four builds, the docs job and the three `mcpp test` jobs. The first wave +today is all 46 jobs at once, which is what saturates the 20 slots (F4). + +#### 2.4.3 What a change starts (R2, R7) + +`changes` sorts the changed paths into classes: + +| class | paths | starts | +|---|---|---| +| documentation | `**/*.md`, `docs/**`, `.agents/**`, `LICENSE*` | the `docs` stage | +| documentation that a check reads | a changed documentation path that a script under `tests/` or `.github/tools/` names, found by searching for it; today this includes `docs/01-getting-started.md`, `docs/20-toolchains.md` and `docs/03-examples.md`, with their `docs/zh/` copies | the `docs` stage, plus the Linux build and every test and tool that names the path, among them the default-toolchain check (`check_default_toolchain_docs.py`) and e2e 616 | +| everything else | | the whole CI | + +A commit that changes nothing outside the documentation class costs one docs job of about two minutes, +instead of 35 jobs and 436 runner minutes (F6). No check is required by the repository's rules, so a +workflow that does not start leaves no check pending. + +The second class is derived by a search, not listed by hand, so that a test added later to read a +document is found without editing the classifier. The classes are deliberately coarse. A finer selection, such as running only the e2e tests a change +could affect, is rejected. A differential selection cannot see a test that is red on both sides, and the +repository has paid for that before. + +#### 2.4.4 Caches with one writer (R3) + +- **Who writes.** Sandbox (`~/.mcpp`) and xlings (`~/.xlings`) caches are saved only by the build job of + each host, and only on a push to `main` (`actions/cache/save` after the build). +- **Who reads.** Every other job, and every pull-request job, uses `actions/cache/restore`. A pull request + reads the `main` lineage, which GitHub scopes to every branch, and adds nothing to the store. +- **Saves per run.** From 34 to 42 down to at most three per host (sandbox, xlings and the e2e durations of + §2.4.5), each key with one writer, on `main` only. +- **The `target/` caches are removed.** F2 shows they do not shorten a build, and at up to 3.3 GB each they + are the main cause of F3. They return only if the build becomes incremental across checkouts. That needs + a measured cause (Part VII) and a fix that keeps the stale-object hazards already recorded in + `.agents/docs/2026-05-15-stdcompat-restat-e2e.md` and `cross-build-test.yml:416-434` out of the result. +- **Concurrency.** `cancel-in-progress` becomes `${{ github.event_name == 'pull_request' }}`, and a push + to `main` has a group of its own commit. A push to `main` runs to completion and writes its caches; a + superseded pull-request run is still cancelled. One group for all pushes to `main` would not be enough, + because GitHub keeps one pending run per group and cancels the older pending run when another arrives. +- **Keys name what fills the cache.** The sandbox key hashes `ci.yml` as well as `mcpp.toml` and + `.xlings.json`, because `ci.yml` lists what the build job installs before it saves. A cache saved under + an unchanged key is never saved again, so a toolchain added to that list would otherwise never reach it. +- **Wine.** The Wine packages come from one pinned archive, published once as a release asset of this + repository, instead of from the `apt` mirrors with a cache that eviction removes (F7). The archive's + sha256 is checked before installation. + +#### 2.4.5 Shards by measured duration (R4) + +- `run_all.sh` already reports each test's duration (`run_all.sh:452-462`). On a push to `main`, each shard + writes its durations under a key of its own, and the next pull request merges the shards of its host. +- A pull request assigns tests to shards greedily by duration, longest first. This is the plan recorded + in `.agents/docs/todos/2026-06-24-e2e-suite-sharding.md`, of which only round-robin was built. +- A test without a recorded duration counts as the host's median. +- Assignment changes only balance, never which tests run, so a stale timing file cannot hide a test. + +Once §2.4.1 lands, a shard no longer pays a build, so more shards cost only their setup. The proposed +counts: + +| host | shards | each shard | +|---|---|---| +| Linux | 3 | about 10 minutes | +| Windows | 3 | about 12 minutes, the long tests spread first | +| macOS | 2 | about 10 minutes | + +Each shard's step limit is twice its budgeted duration, so a limit is reached only by a hang (R4). The +25-minute limits that F5 measured being reached become 20 to 24 minutes on shards half as long. + +The 15 slow Windows tests are a finding in their own right. A test that takes one second on Linux and +up to 104 seconds on Windows measures something about Windows, and Part VII lists it for investigation. +It is not a reason to remove them from the suite. + +#### 2.4.6 No test is silently absent (R5) + +- `run_all.sh` grants `llvm` when the sandbox holds an LLVM payload with `clang++`, as it already does for + `scan-deps` and `import-std-libcxx` (`run_all.sh:221-230`). The Linux e2e shards install the LLVM + payload beside GCC. +- Every shard writes the list of tests it ran and skipped, with the reason for each skip, as a step + summary and as a small artifact. +- A final `e2e-coverage` job collects the lists of every host. It fails if a test ran on no host and is not + listed in `tests/e2e/never-in-ci.tsv`, which states for each such test why no hosted runner can run it. +- The seven tests of F9 must either run or be listed. On Linux, 134, 135, 136, 137, 741, 875 and 876 are + expected to run once `llvm` is granted. + +The run_all.sh comment at `:290-305` records why a hard-required capability was rejected: a token cannot +tell a misconfigured runner from a platform that lacks the capability. The coverage job answers the +question that the token could not, because it knows every host's list. + +#### 2.4.7 macOS slots (R6, R7) + +- The `xcode-27` legs, which are known red, run on pushes to `main` and on dispatch, and on a pull request + only when the pull request carries the label `macos-27`. A pull request that addresses #669 sets the + label. +- The macOS legs of the target matrix and of openkal consume the macOS artifact. macOS builds of mcpp per + commit fall from eight to one. +- The number of macOS jobs per pull-request commit changes less than their minutes do: from 9 or 10 to + about 8. + - The build. + - Two e2e shards. + - `ci-macos` integration, which keeps its fresh sandbox (`setup-macos-llvm/action.yml:7-9` states that + a fresh sandbox is what it proves). + - iOS. + - The target matrix as one job. + - openkal's build and run legs. +- What falls is the time each macOS job holds a slot: no job but the build compiles mcpp, and the two + known-red legs leave pull requests. + +### 2.5 An external lane under `speak-agent` (optional) + +#### 2.5.1 What it would buy + +The 20-job and 5-macOS limits apply per account or organisation, not per repository. `mcpp-community` is +on the Free plan. A repository under `speak-agent`, a machine account, runs against a second pool of +the same size. One repository is enough, because more repositories under one account add no capacity. +The lane can therefore raise the parallelism of a pull request. It cannot shorten a job. + +#### 2.5.2 Constraints + +- **Terms of Service.** GitHub allows one free account per person, plus a machine account that is used + only for automated tasks; `speak-agent` is one. The Actions terms forbid using hosted runners for + activity unrelated to the testing of the software project associated with the repository in which the + Actions run. A lane repository whose stated purpose is to test `mcpp-community/mcpp`, which runs only + mcpp's tests at mcpp's commits, satisfies the wording. Spreading one project's CI over two accounts to + exceed one plan's concurrency is not addressed by the terms, and that is a risk this record cannot + resolve. The lane is therefore optional. It has an off switch, and the repository's own CI stays + complete without it. +- **Forks.** A `pull_request` run from a fork receives no secrets, so it cannot dispatch the lane. Fork + pull requests run every leg inside the repository. +- **Authority.** A push to `main` always runs every leg inside the repository. The lane serves pull + requests only. + +#### 2.5.3 Shape + +- **Repository.** `speak-agent/mcpp-ci-lane`. It is separate from the validation labs, whose `main` + holds only rules and a guard; the lane's workflows live on its `main`, because they are dispatched there. +- **Dispatch.** A job in mcpp's CI (seconds, no slot held while the lane runs) calls the lane's + `workflow_dispatch` with the commit, the pull request number and the legs. The token is a fine-grained + token of `speak-agent` with Actions write on the lane repository only, stored as an mcpp secret. +- **Lane run.** It checks out `mcpp-community/mcpp` at that commit (public, no token). It builds once per + host as in §2.4.1, and runs the legs as consumers. +- **Test jobs.** They have `permissions: {}` and no secret. +- **Report.** A last job, which runs no code from the commit, writes one commit status per leg to the mcpp + commit (`lane/macos-e2e-1` and so on), linking to the lane run. Its token is a second fine-grained token + with commit-status write on `mcpp-community/mcpp` only. That job reads its inputs from + `needs..result` only. +- **In mcpp.** When the lane is on (a repository variable), the legs it carries are skipped on same-repo + pull requests and run as usual everywhere else. + +#### 2.5.4 Costs + +- Two places to read logs. +- Two secrets to rotate. +- A cold cache lineage in the lane until its own runs warm it. +- Contention with the validation labs, which use the same `speak-agent` pool. + +#### 2.5.5 When to build it + +Only if, after D1 to D7 have landed, the median first-attempt wall clock of five consecutive code pull +requests exceeds 30 minutes, and queueing accounts for more than a fifth of it. The first legs to move +are then the macOS and Windows e2e shards, because macOS slots are the scarcest and Windows shards the +longest. + +### 2.6 Expected effect and how it is checked + +An estimate, from the medians of §2.1, to be replaced by measurement: + +| | today (median of the 4 pull requests) | after D1 to D7 | +|---|---|---| +| from-source builds of mcpp | 42 | about 16 | +| runner minutes | 489 | about 330 | +| first-attempt wall clock | 39.6 min | 25 to 28 min, of which about 10 is the build stage | +| macOS jobs, and macOS builds of mcpp | 9 to 10, and 7 | about 8, and 1 | +| caches saved | 34 to 42 | none on a pull request; on `main`, the sandbox and xlings caches of each host and one durations key per shard | +| a documentation-only commit | 35 jobs, 436 runner minutes | 1 to 3 jobs, under 15 runner minutes | + +The work is accepted when five consecutive code pull requests after it show all of the following. The +measurement uses the same scripts as §2.1. +- A median first-attempt wall clock of at most 28 minutes. +- At most 16 from-source builds per commit. +- No e2e step reaching its limit. +- A green `e2e-coverage` job. +- Cache usage below 8 GB. +- A documentation-only pull request finishing under 5 minutes. + +### 2.7 What was built, and where it departs from §2.4 + +Implemented in mcpp#759. Each departure below was decided by a measurement or a +reading taken while building it. + +- **The artifact is the binary itself, not a packed copy (§2.4.1).** The Linux + self-host binary has the interpreter + `~/.mcpp/registry/data/xpkgs/xim-x-glibc/2.44/lib64/ld-linux-x86-64.so.2`, needs + `libgcc_s.so.1` from the `xim-x-gcc/16.1.0` payload, and links libstdc++ + statically (measured locally, `readelf`). A consumer that restores the + sandbox runs it as it is. Packing it would test a different binary from the + one every self-host build produces. `use-built-mcpp` runs the binary first, + and only when it does not run does the bootstrap install the toolchain + `mcpp.toml` names for the host; a binary that still does not run fails the + step. The binary is 25 MB. +- **The Wine packages keep their cache (§2.4.4).** Their eviction was a + consequence of F3, not a property of the cache. With one writer per key, the + `wine-debs` cache is saved only by `mingw-cross-wine` on main and is no longer + displaced. A release asset would have added a second thing to publish and + keep current. +- **The timing tables are in the repository (§2.4.5).** They live under + `tests/e2e/timings/.tsv`, seeded from the per-test lines of the + 2026-10-01 logs. A shard's membership is then a function of the commit, + which makes it reproducible (`E2E_LIST=1` prints it). The `e2e-coverage` job + uploads the merged durations of each run as the artifact `e2e-timings`, and + refreshing a table is copying a file. The tables were seeded from the + 2026-10-01 logs, then replaced by the durations of the first run of this + change, in which 515 Linux tests ran instead of 466. Linux therefore has four + shards rather than three, each budgeted at 9.9 minutes. Windows has three at + 14.4 to 14.5, and macOS two at 9.1 and 9.3. The step limits are about twice + the budgets: 22, 30 and 20 minutes. +- **The classifier searches exact paths (§2.4.3).** It searches for the + changed path and for its translation (`docs/X` and `docs/zh/X`), and not for + a bare file name. A search on the name made `.agents/docs/README.md`, which + every new record regenerates, a code change, because release packaging names + `README.md`. The checks of the `docs` job do not count as readers, because + they run on every change. A document that any other script or source names + starts the whole CI rather than a subset of it, which is simpler and errs + towards running more. +- **The macOS legs of the target matrix and of openkal still build (§2.4.1).** + They run on `macos-14`, and the artifact is built on `macos-15`. release.yml + records that a bootstrap build linking the system libc++ with a minimum + version of 14 failed at launch on `macos-14`. Until the artifact is measured + there, those two legs build their own, and the gate carries + `ci-lint: allow-r1` with that reason. Their Linux and Windows legs consume the + artifact. +- **`invariants` and `scan` remain two jobs (§2.4.1).** Merging them is a + rewrite of the target-matrix workflow beside a change that already rewrites + twelve; it is left for a change of its own. +- **A test that never ran was broken.** 741, one of the seven `llvm` tests, + failed on its first run in CI. The root package of its fixture was a binary, + so the build linked `cabi-probe.exe` for `x86_64-windows-gnu`, and with + `allow_host_libs` the link found the host's mingw-w64 libraries. It passed on a + machine that has them and failed on every runner, and nothing noticed, because + no runner ran it. The fixture's root is now a library, which is what the test + says it is: "this test compiles only". +- **The review of the change** (a code review of mcpp#759) found eight defects. Each was repaired before + merging: + - the coverage check matched test names inside comments and inside longer names, and now matches whole + tokens outside comments; + - `changes` failed, rather than running the whole CI, when the GitHub API did not answer; + - pushes to `main` shared a concurrency group; + - the cache keys did not name the install list; + - the Linux shards' toolchain installs shared the suite's time limit, and now have a step of their own; + - the `macos-27` label's effect was undocumented (it is read on the next push); + - the CHANGELOG gave three Linux shards; + - a path-dependency root with a tab in its path could be cut short in the build record, and is now left + unrecorded, which declines the fast path. +- **Coverage found more than F9.** Classifying the tests of the 2026-10-01 + logs found 24 that ran on no runner and were named by no workflow. The + seven `llvm` tests are among them, and so are three `musl` tests: the probe + named 15.1.0 while the runners installed 16.1.0. Seven `mingw-cross` tests + are in the list too, a toolchain no shard installed. The rest were 105 (nasm), + 65 (scan-deps), 239 (named by its `E2E_ONLY` pattern, which the check now + reads), 257 (needs wine and a Linux-hosted MinGW), 658 (an attached Android + device), and 873-877 (added after the logs). The capability probes now ask for + a family. The Linux shards install musl, llvm, mingw-cross and nasm. 257 runs + in `mingw-cross-wine`. 658 is the one entry of + `tests/e2e/coverage-exceptions.tsv`. + +## Part III. Defects repaired by the next release + +### 3.1 #757: the engine's identity is part of a build record + +1. `BuildCacheEntry` records the engine that wrote the graph: its version (`MCPP_VERSION`) and the + canonical path of its executable (`self_exe_path()`, the same function the emitter uses). An entry + without them declines once: "the recorded build predates the engine identity". +2. **One admission predicate.** The gates common to the three fast paths are one function: the engine + identity, the record's version fields, the toolchain request, the runtime binding, the graph's + existence, mode and request tag, and the freshness sweeps. `try_fast_build`, + `try_fast_workspace_build` and `try_fast_run` call it and add only their own gates (run targets, a + runner, the run tier, the selection). A field added later is then checked in one place, which is the + structural repair of the shape §1.3 describes. +3. Criteria: an e2e test builds with an engine copied to one path, deletes it, and builds again with the + same binary at another path; the second build regenerates the graph and succeeds, and no command in + the new graph names the old path. A unit test of the predicate covers a recorded version that differs + from the running one, and an entry that lacks the fields. + +### 3.2 #756: each path dependency is classified by its own package + +1. When the plan records the path-dependency roots (`src/build/prepare/plan.cpp:139-177`), it records with + each root that package's own `module_extensions` and `device_extensions`, from its effective manifest. +2. `dep_sources_newer_than` classifies the files under each root with that root's table. The record's + block is count-prefixed like `depSourceRoots=`, and an entry without the tables declines once. +3. Criteria: an e2e test with a provider that declares `.ixx` and a consumer that declares nothing. + Editing the provider's host module re-runs the consumer's build program, the program's output + changes, and no dead-entry warning is printed. + +## Part IV. The mcpp toolchain specification (SPEC-009, as approved) + +This part is normative. It is to be published as SPEC-009 in `docs/specs/`, in that directory's +language and format, with an implementation status on each rule. SPEC-006 continues to define what a +toolchain is, how one is named and selected, and what a payload must contain; SPEC-009 defines how the +set of supported toolchains changes over time. Terms follow RFC 2119. + +### TS-1 Terms + +| Term | Meaning | +|---|---| +| Family | `gcc`, `llvm`, `msvc`, `emsdk`, `android-ndk` (SPEC-006 §2.1) | +| Line | a family and a major version: GCC 16, LLVM 23 | +| Release | one upstream version of a line: 16.2.0, 23.1.3 | +| Payload | an installable tree for a release, a host and a variant: glibc `gcc`, native `musl-gcc`, cross `-linux-musl-gcc`, `mingw-gcc`, `mingw-cross-gcc`, `llvm` | +| Row | one row of the target matrix (`kKnownTargets`), or a host default | +| Default | the release a row resolves when nothing is declared | +| Line table | the one table in the engine that states every row's default (TS-3) | +| Self-host toolchain | the toolchains mcpp's own `mcpp.toml` declares | + +### TS-2 Support tiers + +| Tier | Which releases | Obligation | +|---|---|---| +| Default | the release the line table names for a row | the full e2e suite on that row's CI host; the build of mcpp itself where that host builds mcpp; release artefacts are built with it | +| Supported | the previous Default of each row, and releases the line table lists explicitly | the acceptance programs (SPEC-006 §6.2) in CI whenever the line table or one of its payloads changes; a regression is a defect | +| Available | every other release in the index | no CI. Reports are accepted. A known defect is stated as a capability of the payload (`.mcpp-toolchain.json`), never as a version branch in the engine | + +Each family's floor, the oldest line whose `import std` mcpp accepts, MUST be stated in the line +table. Today the floor is implicit. + +### TS-3 One line table + +1. Every default and every row pin MUST be read from one table in the engine. For each row it states the + family, the release, the variant, the tier, and, for a row whose release differs from its family's + Default, the reason and the condition under which the difference ends. +2. Help text, install suggestions, error messages and `mcpp self env --format json` MUST be formatted + from the table. A second literal of a default inside the engine is a defect. +3. Everything outside the engine that names a default (documentation, workflows, tests, examples and + mcpp's own manifest) MUST either read it from `mcpp self env --format json` or be compared with it by + a CI check. A literal that is neither read nor checked is a defect. +4. The index's `latest` is not a default. mcpp pins exact releases, and moving `latest` changes nothing + mcpp builds. + +### TS-4 A line moves as a whole + +1. On one host, the rows of one family SHOULD resolve one release. A row MAY lag only with a reason in + the line table, and the reason MUST be revisited at the next move. +2. A release enters the Default of a row only when every payload that row needs exists at that release, + on both mirrors (TS-9). The payloads of one GCC line (glibc, native musl, cross musl, mingw) are one + line for this purpose. + +### TS-5 Provenance + +1. A payload MUST derive from an upstream release: an official tag and its official source or binary + archive, whose sha256 the payload's description records (SPEC-006 §4.5). +2. A payload MUST be produced by a recipe in a repository, run by a CI job or by a script a CI job could + run. A manual step is not part of a recipe (SPEC-006 §5.1). +3. A patched payload, an upstream release plus upstream commits that no release carries yet, MAY be + published only when all of the following hold: + - every patch is a commit on the upstream release branch, or on its main branch with an open + backport, cited by hash; + - it is published as a revision of that release (`revision = N`, assets under `-r`), never + under the official asset name, and its description lists the patches; + - it is placed on the rows that need it and on no other; + - it has an exit: when an upstream release carries the patches, the Default moves to that release, + and the revision remains in the index only for those who pinned it. +4. The engine MUST NOT work around a toolchain defect by changing the platform's inputs (selecting an + older SDK, rewriting SDK files, substituting another linker) unless that is the row's declared design. + A toolchain defect is repaired by a toolchain release, or by a patched payload under rule 3. + +### TS-6 What it means that mcpp supports a line + +1. The engine MUST derive what a toolchain can do from the payload, not from its version: the standard + library module from the library's manifest or layout, the scanner from the driver, flags from probes + measured on the row. A comparison of version numbers is permitted only for a defect registered under + TS-7, or in a language-feature table whose rows cite the release notes. +2. On every row where a line is Default, the engine MUST pass the acceptance programs (a program using + ``, `` and ``; `import std`; `import std.compat`), the e2e suite, and, where the + host builds mcpp, the build of mcpp itself. +3. Every output that encodes a release MUST be reviewed at a move: the pack ABI tag (compiler major), + BMI and cache identity (the version enters the fingerprint and every cache key, so a move needs no + epoch change), and diagnostics that quote a version. + +### TS-7 The compiler-defect register + +1. Every engine behaviour, and every shape of mcpp's own sources, that exists because of a compiler + defect MUST have an entry: family, first release observed, last release verified, upstream report, + a minimal reproduction under `tests/` that fails while the defect exists, and the sites that depend + on it. +2. At every move of a line, every reproduction runs against the new release, and the entry records the + result. +3. A workaround is removed only when every Supported release is past the fix. + +Entries known today, from source comments and earlier records: an instantiation dropped along a module +import chain (GCC 16.1); a `FILE`-typed entity in a module interface breaking a later +`#include ` (GCC PR 99000, open); a nested `std::map` member truncating a BMI (GCC 16.1); a new, +widely imported module with standard types in its interface poisoning downstream BMIs (GCC 16.1); a +segmentation fault on a new interface unit (GCC 16.1, `src/build/prepare.cppm:37-50`); a miscompile with +a full BMI, which is why clang uses the two-phase reduced interface (clang 22.1.8, +`src/build/ninja_backend.cppm:1331`); a crash on an inline helper in a module purview (clang 20.1.7 on +Windows); the 27.0 SDK leaving `INFINITY` and `NAN` to `` under modules +(`src/toolchain/hostflags.cppm:162-181`); the missing `directory_iterator` comparison in libc++ 20's +`std`; an ICE on `AMDGPUAsmParser.cpp` (GCC 16.1, the reason `llvm-dev` is built with GCC 15.1). + +### TS-8 Host platform releases + +1. A new operating system or SDK release that a row's users will meet gets a CI leg as soon as a hosted + image exists, before that image becomes the runner default. +2. A leg that is red for an external cause carries `known_red: '#'`, the workflow assertion keeps + the issue open, and the leg leaves the list when the issue closes. +3. The repair follows TS-5. + +### TS-9 Mirrors + +A version row MUST NOT reach the index before every asset it names is present on the GLOBAL and CN +mirrors and verified on both by a GET (status 200, byte size and sha256). An upload of more than about +8 MiB to the CN mirror is made from a host inside the CN network. + +### TS-10 Moving a Default + +1. The upstream release exists. +2. Its payloads are built by their recipes (TS-5) and pass the admission of SPEC-006 §6; each records its + inputs. +3. The assets are on both mirrors and verified (TS-9). +4. The index gains the version rows; `latest` is unchanged. +5. The engine is verified on every row that will take the release as Default, and the release passes + the gate below. The payloads MAY be named by path (SPEC-006 §2.2.1), so that this step does not wait + for step 4. +6. One mcpp pull request moves the line table and every reader and checked copy of it; mcpp is + released. +7. The index's `latest` moves after that release has shipped. +8. The tiers shift: the previous Default becomes Supported, and the line before it becomes Available. + +A move is reverted by reverting step 6. Payloads and index rows stay. + +**TS-10.5 The gate: a new release does not make the module experience worse.** A candidate release R is +compared with the row's current Default D on the same mcpp commit, on the same runner image, in the +same job, so that the comparison spans only the change of release. R passes when all six hold: + +| Gate | Criterion | +|---|---| +| G1 | every e2e test that passes with D passes with R, and no test that runs with D is skipped with R | +| G2 | where the row builds mcpp, mcpp builds itself with R, and that binary passes the suite | +| G3 | the acceptance programs of TS-6.2 build and run | +| G4 | every TS-7 reproduction is run, and none of G1 to G3 needs a new workaround or a reshaped source; a defect that does is a failure of the gate unless the review accepts it with a register entry | +| G5 | the scanned module graph (each unit's provided and required modules) of mcpp's own sources and of the e2e module fixtures is identical under R and D | +| G6 | the cold and the warm build of mcpp and of the `bench/` projects take no more than 10 percent longer with R, as the median of three runs, and the BMIs are no more than 10 percent larger | + +A release that fails the gate stays Available. The next release of the same line becomes the +candidate. + +### TS-11 Machines that already have a default + +1. A release that a user or a project states (`[toolchain]`, `--toolchain`, `MCPP_TOOLCHAIN`, + `mcpp toolchain default`) MUST NOT move. +2. A default that mcpp wrote on a first run is not a statement by the user, and its record MUST say so. + - On a home with no recorded default, a first run installs and records the line table's answer, as + it does today, and states nothing more. + - The notice exists for one case only: the home holds a default recorded by an earlier mcpp's first + run, and the running mcpp's line table names a newer release for that row. + - In that case mcpp MUST state once the recorded and the newer release, together with the two + commands that settle the question: `mcpp toolchain default ` moves the record, and + `mcpp toolchain default --keep` keeps it. Either command turns the record into a statement by the + user. + - The notice does not block, does not prompt, and is not repeated. Until the user answers, the + recorded release stays in use. +3. Payloads already installed stay; removing them is the user's decision. + +### TS-12 mcpp builds with what it gives + +mcpp's own manifest MUST use the Default release of each row it is built on. A divergence is a line-table +entry with a reason and an exit. + +### TS-13 Conformance checks + +The rules a program can check MUST be checked in CI: + +| Check | Rule | +|---|---| +| C1 | no default literal in `src/` or `modules/` outside the line table (TS-3.2) | +| C2 | every documentation statement of a default, including the target-row tables of `docs/21` and the README, matches `mcpp self env --format json` (TS-3.3) | +| C3 | workflows, actions and tests take versions from the line table; a capability probe asks for a family, not a version (TS-3.3) | +| C4 | `mcpp.toml`'s `[toolchain]` equals the line table or a recorded divergence (TS-12) | +| C5 | a `known_red` leg names an open issue (TS-8; exists) | +| C6 | the TS-7 reproductions run on every Default row | +| C7 | the gate of TS-10.5 is one workflow that takes a row and a candidate payload, and the pull request of TS-10.6 cites its run | + +## Part V. The toolchain candidates, kept for the move that follows this work + +Nothing in this part is done in this work (T2). It states where each line would go and what each move +needs, so that a candidate that passes the gate (TS-10.5) can move without another design. + +### 5.1 The line table if both candidates pass + +| Row | Today | After | Payloads it needs | +|---|---|---|---| +| Linux x86_64 (host default) | `gcc@16.1.0` | `gcc@16.2.0` | `gcc` 16.2.0 | +| Linux, other architectures (host default) | `gcc@15.1.0-musl` | `gcc@16.2.0-musl` | `musl-gcc` 16.2.0 (native) | +| Windows without MSVC (host default) | `gcc@16.1.0` | `gcc@16.2.0` | `mingw-gcc` 16.2.0 | +| Windows with MSVC (host default) | `llvm@20.1.7` | `llvm@23.1.3` | `llvm` 23.1.3 windows | +| macOS (host default) | `llvm@20.1.7` | `llvm@23.1.3` | `llvm` 23.1.3 macosx-arm64 | +| `x86_64-linux-musl`, `aarch64-linux-musl`, `x86_64-windows-gnu` | `gcc@16.1.0` | `gcc@16.2.0` | `musl-gcc`, `aarch64-linux-musl-gcc`, `mingw-gcc`, `mingw-cross-gcc` 16.2.0 | +| `x86_64-windows-musl` | `llvm@22.1.8` | `llvm@23.1.3` | `llvm` 23.1.3 | +| bare-metal rows (12) and iOS rows (3) | `llvm@22.1.8` | `llvm@22.1.8`, lagging (T5) | picolibc sysroots carry compiler-rt builtins built with 22.1.8; iOS rows need `llvm.libcxx` and `llvm.compiler-rt-builtins` packages at 23 (`src/build/prepare/scan.cpp:631-632`) | +| mcpp's own `[toolchain]` | `gcc@16.1.0`, `macos = llvm@22.1.8`, `windows = llvm@20.1.7`, musl `gcc@16.1.0-musl` | the Default of each row | none beyond the above | + +Not moved: `android-ndk` and `emsdk` rows; the `bench/` pins, which are fixed on purpose so that a +benchmark does not report a toolchain change as an engine change; fixtures that pin a release to test a +specific behaviour. + +The move from 20.1.7 to 23.1.3 on macOS and Windows also removes finding 2 of §1.5: the macOS default +will be able to build mcpp. + +### 5.2 GCC 16.2 + +**Upstream.** GCC 16.2.0 was released on 2026-08-07, a bug-fix release with 672 commits since 16.1.0. +The commit range does not touch the p1689 output, the module mapper or the driver options mcpp uses. +It repairs nine module defects (among them 124953, failed to load pendings; 124981, an undefined +reference for an instantiation streamed from a BMI; 125768 and 126209, ICEs). libstdc++ adds the symbol +version `GLIBCXX_3.4.36`. + +**The risk that decides T6.** Bug 126577, a corrupt CMI when `-fmodule-mapper`, a re-exported partition, +a `chrono` alias and `unique_ptr` meet, is a regression of the 16 branch after 16.1.0, present in 16.2.0, +and retargeted to 16.3. mcpp passes `-fmodule-mapper` to GCC for every unit with a module scope +(`src/build/plan.cppm:3402`). The rule: if measurement M1 reproduces it, the GCC Default stays at +16.1.0 and 16.2.0 is Available, until a 16.3 release or a TS-5 revision carrying the fix. + +**Payloads.** + +| Payload | How it is produced this round | +|---|---| +| `gcc` 16.2.0 (glibc) | the `fromsource` recipe at 16.2.0, with the strip and the `specs` canonicalisation turned into a script (the musl builders' strip loop is the model), run by a builder workflow in `xlings-res/gcc` in the shape of `xlings-res/musl-gcc`'s; the description records the configure line, the source sha256 and the build's C library | +| `musl-gcc`, `aarch64-linux-musl-gcc` 16.2.0 | dispatch of the existing builder workflows with `gcc_ver=16.2.0` | +| `mingw-gcc` 16.2.0 | mirror of winlibs `16.2.0posix-14.0.0-ucrt-r2` | +| `mingw-cross-gcc` 16.2.0 | a builder workflow for the recipe that 16.1.0 followed by hand (GCC, binutils 2.44, mingw-w64 v12 CRT); if it is not ready, the `x86_64-windows-gnu` row lags under TS-4 | + +Out of this round: `riscv64-linux-musl-gcc`, the Windows-host `x86_64-linux-musl-gcc`, `gcc-runtime` +(only 15.1.0 exists), and rebuilding `llvm-dev` and `libllvm` with 16.2 (they are built with 15.1 +because of the 16.1 ICE in TS-7). + +### 5.3 LLVM 23 + +**Upstream.** 23.1.0, 23.1.1 and 23.1.2 were released on 2026-08-25, 09-08 and 09-22; 23.1.3 is +scheduled for 2026-10-06 and is expected to carry the `arm64e.x1` repair, which reached +`release/23.x` after the 23.1.2 tag. Release archive names and layouts are unchanged from 22.1.8; 23 +adds `.tar.zst` siblings and `.msi` installers. The Windows archive carries no +libc++ and no `std` module, as before, so `import std` on that row continues to come from the MSVC +STL. + +**What changes for a build tool**, from the 23 release notes and the sources: + +- P1857R3 changes how `module` and `import` directives are lexed during dependency discovery. The + scanner's output must be measured (M2, §5.5). +- `__has_feature(modules)` is no longer true with only `-std=c++20`. The SDK 27 `INFINITY`/`NAN` + workaround may therefore become unnecessary on 23; it stays harmless, because its values equal the + SDK's, and its TS-7 entry is re-measured. +- `export` inside a module implementation partition is now rejected. +- libc++ removed many transitive includes, and `std.cppm` now includes `` and + ``. +- The libc++ `[abi:...]` tag follows `_LIBCPP_VERSION` and changes with every patch release. The + fingerprint and the cache keys already contain the version. +- Reduced BMIs became the default in 22. `-fmodule-output` and `-fmodules-reduced-bmi` remain. + `GetStdModuleManifestPath` is unchanged between 22.1.8 and 23.1.2. + +**Payloads.** The carve script (`xim-pkgindex/.agents/tools/build-llvm-subpkg.sh`) applies unchanged to +the three official archives. The Linux carve still injects `libatomic` from a GCC on the build host +(23.1.2's `libc++.so.1` still names `libatomic.so.1`, and the archive carries none). That GCC should be +16.2 once its payload exists. `llvm-tools` and `mcpp-vscode-clangd` take the same version. The index's +parity test requires the new version on the linux, macosx and windows sections together. The existing +22.1.8 and 20.1.7 rows stay, because index tests name their assets and pinned users rely on them. + +### 5.4 The macOS 27 link + +The repair is the official 23.1.3 (TS-5.4, T3). Four days separate this record from its scheduled tag, +and the `xcode-27` legs are known red without blocking anything. Every LLVM row therefore moves to +23.1.3 when 23.1.3 passes the gate; the 23.1.2 measurement only shows early whether the line is likely +to pass. + +A patched payload is available under TS-5.3 if it is asked for: `llvm@23.1.2` revision 1 for the macOS +row. It would be the official 23.1.2 archive carved as usual, with `ld64.lld` and `llvm-otool` rebuilt +from `release/23.x` at a recorded commit containing `532fa5afbe2b` and `ee66426152f9`, published as +`23.1.2-r1` assets whose description names the commits. It is not part of the plan unless 23.1.3 is +delayed and macOS 27 users must be served before it. + +Rejected alternatives: selecting the Command Line Tools 26.5 SDK, rewriting the SDK's `.tbd` files, +and linking with Apple's `ld` instead of `ld64.lld`. Each changes the platform's inputs to hide a +toolchain defect (TS-5.4), and each would survive the repair as a second behaviour. + +When both `xcode-27` legs are green with the new release, #669 closes and the `known_red` entries leave +the workflows (TS-8.2). + +### 5.5 Measurements that start the move + +Each one decides something. They run when the first move begins, not in this work. + +| # | Measurement | Decides | +|---|---|---| +| M1 | mcpp built by itself, and its e2e suite, with GCC 16.2.0 on Linux x86_64; the 126577 reproducer | T6 | +| M2 | the same with the official LLVM 23.1.x on Linux x86_64, named by path: scanning, partitions, transitive includes | the engine changes of §5.3 | +| M3 | the same on Windows with LLVM 23.1.x and the MSVC STL | the Windows row | +| M4 | macOS 15 with 23.1.x; `xcode-27` with 23.1.3 | T3, #669 | +| M5 | every TS-7 reproduction against the candidates | the register's entries | +| M6 | a program referencing a `GLIBCXX_3.4.36` symbol, built with 16.2.0 and packed in each pack mode, run on a clean container | the runtime closure of the GCC row | +| M7 | a bare-metal row and an iOS row with the 23 compiler and their 22.1.8 companion packages | whether T5 can be shortened | + +## Part VI. Order of the work + +Each item is one pull request unless stated otherwise. CI changes need no mcpp release; the engine +repairs do. + +| # | Item | Contents | Depends on | +|---|---|---|---| +| 1 | SPEC-009 | Part IV in `docs/specs/`, in that directory's language and format, with an implementation status on each rule (most are "not implemented"); SPEC-006 §7 becomes a reference to it | none | +| 2 | Engine repairs | Part III; released as the next mcpp version | none | +| 3 | CI, low-risk rules | D4 (cache writers, `target/` removal, concurrency on `main`, the Wine archive), D6 (the `llvm` capability and the coverage job), D7 (known-red legs off pull requests), and the macOS step limit of T8 | none | +| 4 | CI, the build stage | D1, D2 and D3: `ci.yml`, the build matrix with packed artifacts, the reusable workflows, path classes | item 3, so that its measurements start from the new cache policy | +| 5 | CI, shards | D5: duration-based shards and their limits | item 4 | +| 6 | Measurement | §2.6, over five consecutive code pull requests | item 5 | +| 7 | External lane | §2.5, only if the rule of §2.5.5 holds | item 6 | + +Items 1, 2 and 3 run in parallel. + +The first application of SPEC-009 (Part V) is not in this list. It starts when LLVM 23.1.3 is tagged, under +a record of its own, and it begins with the measurements of §5.5 and the single line table (D10). + +## Part VII. Not yet measured + +- **Why a restored `target/` does not make a build incremental (F2).** Checkout modification times are the + likely cause, and are not verified. The answer decides whether `target/` caching can return, and in what + form. +- **Whether every consumer works with the packed artifact (§2.4.1).** The Windows hand-off proves it for + one job. The e2e suite on Linux and macOS with a packed `$MCPP` is the measurement, run once in a + validation pull request before item 4. +- **Why the target-matrix and openkal legs build with `--dev`.** Nothing records it. +- **The minimum macOS version of an mcpp built on `macos-15`.** It decides whether the `macos-14` legs can + consume it. +- **Why about 15 e2e tests take 39 to 104 seconds on Windows and about one second on Linux.** +- **The cost of the e2e-coverage job's lists**, and whether any host-specific skip is missing from them. +- **Toolchains.** + - Whether mcpp builds itself with GCC 16.2.0 and with LLVM 23 (§5.5, M1 to M3). + - The `libstdc++.modules.json` relative path in a 16.2.0 install. + - Whether the clang 23 driver changes a default that the generated `clang++.cfg` relies on. + - Whether the build-program cache and the host-module store key the engine version as well as the + compiler. If they do not, §3.1 extends to them. + +## Part VIII. Questions for review + +1. One top-level `ci.yml` with reusable workflows (recommended), or each platform workflow with a build job + of its own and artifacts shared only within it. +2. The artifact as the packed self-host build (recommended), or the raw binary next to a restored + sandbox. +3. Removing the `target/` caches now (recommended), or keeping them until F2's cause is measured. +4. The documentation classes of §2.4.3, with the second class derived by searching `tests/` and + `.github/tools/` for the changed path. +5. The shard counts of §2.4.5 (Linux 3, Windows 3, macOS 2), and step limits at twice the budget. +6. The known-red legs on pull requests only with the label `macos-27` (recommended). +7. The external lane: designed now and built under §2.5.5 (recommended), or built now. If it is built, the + repository name `speak-agent/mcpp-ci-lane`. +8. The single line table (TS-3) deferred to the first toolchain move (recommended), or built in item 2. diff --git a/.agents/docs/README.md b/.agents/docs/README.md index db02537d..f9ace8e8 100644 --- a/.agents/docs/README.md +++ b/.agents/docs/README.md @@ -18,7 +18,7 @@ superseded_by: 2026-09-07-....md # when status is superseded --- ``` -323 records. +324 records. ## By subject @@ -30,6 +30,7 @@ Records that declare one. Everything else is listed by date below. ### design +- [PR CI acceleration and the toolchain specification (#756, #757, #669)](2026-10-02-pr-ci-acceleration-and-the-toolchain-specification-design.md) — active - [工具与工具链的来源：声明、编程决定、可观察](2026-10-01-tool-and-toolchain-sources-design.md) — landed - [A pack's build reported as a build, and a unit's compile independent of the member selection: triage and design (#753, #751)](2026-10-01-pack-drive-and-selection-independent-compile-design.md) — landed - [Member selection, build programs prepared once, a pack over several members, and the output streams of `mcpp run`: the plan for the release after 2026.9.30.2 (#748, #749, #750)](2026-09-30-member-selection-and-build-program-cost-plan.md) — landed @@ -114,6 +115,7 @@ Records that declare one. Everything else is listed by date below. ### 2026-10 +- [PR CI acceleration and the toolchain specification (#756, #757, #669)](2026-10-02-pr-ci-acceleration-and-the-toolchain-specification-design.md) — active - [工具与工具链的来源：声明、编程决定、可观察](2026-10-01-tool-and-toolchain-sources-design.md) — landed - [A pack's build reported as a build, and a unit's compile independent of the member selection: triage and design (#753, #751)](2026-10-01-pack-drive-and-selection-independent-compile-design.md) — landed ### 2026-09 diff --git a/.agents/skills/mcpp-contributing/SKILL.md b/.agents/skills/mcpp-contributing/SKILL.md index e747335a..d6ebec57 100644 --- a/.agents/skills/mcpp-contributing/SKILL.md +++ b/.agents/skills/mcpp-contributing/SKILL.md @@ -182,14 +182,16 @@ gh pr checks # 查看状态 gh run view --log-failed # 查看失败日志 ``` -CI 由分平台的基础构建/单元集成检查与独立 E2E 检查组成： -| Workflow | 平台 | 内容 | -|----------|------|------| -| `ci-linux` / `ci-linux-e2e` | Linux x86_64 | 自举构建、unit/integration / 分片 E2E | -| `ci-macos` / `ci-macos-e2e` | macOS ARM64 | 自举构建、unit/integration / E2E | -| `ci-windows` / `ci-windows-e2e` | Windows x86_64 | 自举构建、toolchain 回归 / E2E | -| `cross-build-test` | Linux/Windows cross targets | 交叉构建、产物运行与 MinGW/Wine 检查 | -| `ci-aarch64-fresh-install` | Linux ARM64 native | path-filtered fresh install、原生自举与 musl `build.mcpp` host-helper 回归 | +一次提交的 CI 是 `ci.yml` 的一次运行，分段执行（设计见 `.agents/docs/2026-10-02-pr-ci-acceleration-and-the-toolchain-specification-design.md` 第二部分）： +| 阶段 | 内容 | +|------|------| +| `changes` | 按改动路径分类；只改了没有任何脚本、测试或源码读取的文档时，只跑 `docs` | +| `docs` | 不需要二进制的检查（版本钉、文档风格与结构、工作流断言等） | +| `build-*` | 每个宿主构建一次 mcpp（`build.yml`），上传为 `mcpp-built-` | +| `linux` / `linux-e2e`、`macos` / `macos-e2e` / `macos-ios`、`windows` / `windows-e2e` / `windows-msvc-xlings`、`cross`、`target-matrix`、`openkal` | 各领域的可复用工作流（原来的 `ci-*.yml`），通过 `.github/actions/use-built-mcpp` 使用上面那次构建，不再各自构建 | +| `e2e-coverage` | 每个 e2e 测试都在某个宿主上运行、由专门 job 运行，或在 `tests/e2e/coverage-exceptions.tsv` 中写明原因 | + +`ci-aarch64-fresh-install`、`measure-windows-tool-crt` 与 `pypi-publish` 仍是按路径触发的独立工作流。缓存只在 main 上由一个 job 保存；PR 只恢复。 **以 PR 实际 required checks 为准，所有未跳过的 required checks 必须通过。** 如果某个平台失败： 1. 下载日志分析原因 diff --git a/.agents/skills/mcpp-release/SKILL.md b/.agents/skills/mcpp-release/SKILL.md index e28e37d0..abe28c57 100644 --- a/.agents/skills/mcpp-release/SKILL.md +++ b/.agents/skills/mcpp-release/SKILL.md @@ -73,9 +73,9 @@ gh run list --branch main --limit 3 ``` 以分支保护和 `gh pr checks ` 显示的 actual required checks 为准。 -在 main 上监控当前运行时，检查 `ci-linux`、`ci-linux-e2e`、`ci-macos`、 -`ci-macos-e2e`、`ci-windows`、`ci-windows-e2e` 与 `cross-build-test` 的结果； -跳过或非 required 的 workflow 不是合入 gate。不要在 required CI 红的时候发版。 +在 main 上监控当前运行时，检查 `ci` 这一个工作流的运行（它包含各平台、e2e 分片、 +交叉构建与 `e2e-coverage`）；known-red 的腿（名字里带 issue 号）允许失败。跳过或 +非 required 的 workflow 不是合入 gate。不要在 `ci` 红的时候发版。 ### 2. bump 版本号（第一组两处，单个 commit，走 PR） diff --git a/.github/actions/bootstrap-mcpp/action.yml b/.github/actions/bootstrap-mcpp/action.yml index 15ef29bf..0b7887ab 100644 --- a/.github/actions/bootstrap-mcpp/action.yml +++ b/.github/actions/bootstrap-mcpp/action.yml @@ -1,13 +1,21 @@ name: bootstrap-mcpp description: > - Restore the shared CI cache lineage (mcpp sandbox + xlings + target/) and - bootstrap a released mcpp via xlings. Exports MCPP and XLINGS_BIN. + Restore the shared CI cache lineage (mcpp sandbox and xlings) and bootstrap + a released mcpp via xlings. Exports MCPP and XLINGS_BIN. - Extracted so the split CI jobs (build / toolchain legs / e2e shards / - integration) share ONE definition instead of copy-pasting a 40-line - preamble per job. Every job that uses it lands on the same cache keys, - which is what makes splitting cheap: each job restores a warm sandbox - and only pays one incremental `mcpp build`. + The caches are RESTORED here and never saved. One job per host writes them, + the build job of .github/workflows/build.yml, and only on a push to main + (rule R3 of .agents/docs/2026-10-02-pr-ci-acceleration-and-the-toolchain- + specification-design.md). Before that rule every job that used this action + saved the same key on success: thirty-four to forty-two saves per pull + request against a 10 GB repository limit, parallel saves of one key racing + each other, and the main lineage evicted within forty minutes (measured + 2026-10-01). The keys are outputs so that the one writer saves exactly what + was restored. + + `target/` is no longer cached. A restored `target/` made no build + incremental: on an exact hit ninja still ran 830 of 830 edges, while the + caches themselves were up to 3.3 GB each. inputs: xlings-version: @@ -26,10 +34,20 @@ inputs: # depended on the machine, which is why CI failed on `compat:lua` on # Windows and `mcpplibs.capi:lua` on Linux. Never pin below that. default: '2026.9.30.1' - cache-target: - description: also restore/save target/ (build artifacts + BMIs) - required: false - default: 'true' + +outputs: + sandbox-key: + description: the exact key of the mcpp sandbox cache + value: ${{ steps.sandbox.outputs.cache-primary-key }} + sandbox-hit: + description: "'true' when the sandbox was restored by its exact key" + value: ${{ steps.sandbox.outputs.cache-hit }} + xlings-key: + description: the exact key of the xlings cache + value: ${{ steps.xlings.outputs.cache-primary-key }} + xlings-hit: + description: "'true' when xlings was restored by its exact key" + value: ${{ steps.xlings.outputs.cache-hit }} runs: using: composite @@ -38,8 +56,9 @@ runs: # "-release-" caches. A bare "mcpp-sandbox--" restore prefix used to # match the release sandbox too, silently swapping in a differently # populated registry (issue #120). - - name: Cache mcpp sandbox - uses: actions/cache@v4 + - name: Restore the mcpp sandbox + id: sandbox + uses: actions/cache/restore@v4 with: path: ~/.mcpp # `runner.arch` IS PART OF EVERY KEY, AND WAS NOT. @@ -66,12 +85,16 @@ runs: # sandbox — which is what actually resolves dependencies — would # silently stay behind (observed: a 0.4.30 sandbox surviving under a # 0.4.69 bootstrap for weeks). - key: mcpp-sandbox-${{ runner.os }}-${{ runner.arch }}-ci-xl${{ inputs.xlings-version }}-${{ hashFiles('mcpp.toml', '.xlings.json') }} + # ci.yml is part of the key because it names the toolchains the build + # job installs before it saves (`prewarm`): a sandbox saved under an + # unchanged key would never gain one added there. + key: mcpp-sandbox-${{ runner.os }}-${{ runner.arch }}-ci-xl${{ inputs.xlings-version }}-${{ hashFiles('mcpp.toml', '.xlings.json', '.github/workflows/ci.yml') }} restore-keys: | mcpp-sandbox-${{ runner.os }}-${{ runner.arch }}-ci-xl${{ inputs.xlings-version }}- - - name: Cache xlings - uses: actions/cache@v4 + - name: Restore xlings + id: xlings + uses: actions/cache/restore@v4 with: path: ~/.xlings key: xlings-${{ runner.os }}-${{ runner.arch }}-v2-xl${{ inputs.xlings-version }}-${{ hashFiles('.xlings.json') }} @@ -109,6 +132,37 @@ runs: esac tarball="xlings-${XLINGS_VERSION}-linux-${xa}.tar.gz" ;; esac + # FAST PATH: the xlings cache already holds the pinned version, so the + # tarball fetch + extract + `self install` are skipped. Without this + # guard every job of every CI run paid the download and the extract, + # measured at 5 to 30 seconds per job on Linux and ~30 seconds on + # Windows, across thirty jobs per run — about half the bootstrap-mcpp + # step on Windows, more the 7 seconds the unix leg pays. The cache key + # is `xl$VER` already; the check is the one case this guard would + # otherwise miss: a stale `xlings` cache from BEFORE the pin was + # bumped (a partial restore-key match hands the same OS/ARCH cache + # back, but with the previous release), or a binary that no longer runs + # because its dynamic loader is gone. + XL_BIN_PATH="$HOME/.xlings/subos/default/bin/xlings" + if [ -x "$XL_BIN_PATH" ]; then + xl_ver="$("$XL_BIN_PATH" --version 2>/dev/null | head -1 || true)" + if [ -n "$xl_ver" ] && echo "$xl_ver" | grep -qF "$XLINGS_VERSION"; then + export PATH="$HOME/.xlings/subos/default/bin:$PATH" + echo "$HOME/.xlings/subos/default/bin" >> "$GITHUB_PATH" + "$XL_BIN_PATH" --version + MCPP=$(bash "$REPO_DIR/.github/tools/install_pinned_mcpp.sh" "$REPO_DIR") + echo "system xlings: $("$XL_BIN_PATH" --version 2>/dev/null | head -1)" + if [ -x "$HOME/.mcpp/registry/bin/xlings" ]; then + echo "sandbox xlings: $("$HOME/.mcpp/registry/bin/xlings" --version 2>/dev/null | head -1)" + else + echo "sandbox xlings: (not initialised yet)" + fi + echo "MCPP=$MCPP" >> "$GITHUB_ENV" + echo "XLINGS_BIN=$XL_BIN_PATH" >> "$GITHUB_ENV" + exit 0 + fi + fi + WORK=$(mktemp -d) # Retried and verified — see .github/tools/fetch_release.sh. A bare curl # here was the single largest source of unexplained CI red on this repo @@ -179,6 +233,37 @@ runs: XLINGS_VERSION: ${{ inputs.xlings-version }} run: | REPO_DIR="$(pwd)" + # FAST PATH: see the unix leg for the reasoning. The cost on Windows is + # larger (the zip is bigger and the runner's network path to + # github.com is slower), measured at ~30 s per job. + # + # The fast path addresses xlings by ABSOLUTE PATH, not by bare + # `xlings.exe`. The cold path runs `xlings self install` first, which + # writes the dir into Windows PATH via `[Environment]::SetEnvironmentVariable`; + # without that step a bare `xlings.exe` call depends on the bash + # export PATH, which Git Bash re-derives from Windows on every child + # shell and drops the mixed-separator entry. install_pinned_mcpp.sh + # carries the same note. + XL_BIN_PATH="$USERPROFILE/.xlings/subos/default/bin/xlings.exe" + if [ -x "$XL_BIN_PATH" ]; then + xl_ver="$("$XL_BIN_PATH" --version 2>/dev/null | head -1 || true)" + if [ -n "$xl_ver" ] && echo "$xl_ver" | grep -qF "$XLINGS_VERSION"; then + export PATH="$USERPROFILE/.xlings/subos/default/bin:$PATH" + echo "$USERPROFILE/.xlings/subos/default/bin" >> "$GITHUB_PATH" + "$XL_BIN_PATH" --version + MCPP=$(bash "$REPO_DIR/.github/tools/install_pinned_mcpp.sh" "$REPO_DIR") + echo "system xlings: $("$XL_BIN_PATH" --version 2>/dev/null | head -1)" + if [ -x "$USERPROFILE/.mcpp/registry/bin/xlings.exe" ]; then + echo "sandbox xlings: $("$USERPROFILE/.mcpp/registry/bin/xlings.exe" --version 2>/dev/null | head -1)" + else + echo "sandbox xlings: (not initialised yet)" + fi + echo "MCPP=$MCPP" >> "$GITHUB_ENV" + echo "XLINGS_BIN=$(cygpath -w "$XL_BIN_PATH")" >> "$GITHUB_ENV" + exit 0 + fi + fi + WORK=$(mktemp -d) zipfile="xlings-${XLINGS_VERSION}-windows-x86_64.zip" # Same helper as the unix leg. This is the leg that kept failing, and a @@ -207,17 +292,3 @@ runs: # Precise key on src/ + manifest so a no-source-change run lands on a full # hit; layered restore-keys let partial hits keep BMI/dyndep state for a # proper incremental build. - - name: Cache target/ (build artifacts + BMIs) - if: inputs.cache-target == 'true' - uses: actions/cache@v4 - with: - path: target - # `modules/**` belongs here as much as `src/**` does. mcpp's own - # source lives in both since the subsystem split, and a key that hashed - # only one of them would restore a target/ built from different sources - # and report success — the failure mode a cache key exists to prevent, - # arriving silently. - key: mcpp-target-${{ runner.os }}-${{ runner.arch }}-${{ github.job }}-${{ hashFiles('src/**', 'modules/**', 'tests/**', 'mcpp.toml', 'mcpp.lock') }} - restore-keys: | - mcpp-target-${{ runner.os }}-${{ runner.arch }}-${{ github.job }}- - mcpp-target-${{ runner.os }}-${{ runner.arch }}- diff --git a/.github/actions/setup-macos-llvm/action.yml b/.github/actions/setup-macos-llvm/action.yml index e79a10cb..b43cd7de 100644 --- a/.github/actions/setup-macos-llvm/action.yml +++ b/.github/actions/setup-macos-llvm/action.yml @@ -24,11 +24,22 @@ inputs: required: false default: 'macos-15' +outputs: + xlings-key: + description: the exact key of the xlings cache + value: ${{ steps.xlings.outputs.cache-primary-key }} + xlings-hit: + description: "'true' when xlings was restored by its exact key" + value: ${{ steps.xlings.outputs.cache-hit }} + runs: using: composite steps: - - name: Cache xlings - uses: actions/cache@v4 + # Restored, never saved here: the macOS build job of + # .github/workflows/build.yml is the one writer, on a push to main (rule R3). + - name: Restore xlings + id: xlings + uses: actions/cache/restore@v4 with: path: ~/.xlings key: xlings-${{ inputs.image }}-arm64-v3-xl${{ inputs.xlings-version }}-${{ hashFiles('.xlings.json') }} diff --git a/.github/actions/use-built-mcpp/action.yml b/.github/actions/use-built-mcpp/action.yml new file mode 100644 index 00000000..7a87ff6f --- /dev/null +++ b/.github/actions/use-built-mcpp/action.yml @@ -0,0 +1,44 @@ +name: use-built-mcpp +description: > + Put this commit's mcpp, built once per host by .github/workflows/build.yml, + in place of a build of the job's own (rule R1 of + .agents/docs/2026-10-02-pr-ci-acceleration-and-the-toolchain-specification-design.md). + + Run it after bootstrap-mcpp or setup-macos-llvm, which restore the sandbox + and install the released bootstrap. It exports MCPP_BOOT (that bootstrap), + MCPP and MCPP_FRESH (this commit's binary, at an absolute path), and + MCPP_VENDORED_XLINGS, and sets the given mirror on xlings and on the binary. + + The binary is the one the build job produced, not a repackaging of it, so + every consumer tests what a self-host build makes. On Linux that binary's + interpreter and runtime libraries live in payloads of the sandbox (glibc and + the GCC runtime of the toolchain mcpp.toml names). A restored sandbox holds + them; when it does not, the bootstrap installs that toolchain and the binary + is run again. A binary that still does not run fails this step. + + The toolchain mcpp.toml names for the host is then installed with the + binary, as a job that built mcpp used to install it as a side effect. + +inputs: + host: + description: > + The host the artifact was built on: linux-x86_64, linux-aarch64, + macos-arm64 or windows-x86_64. + required: true + mirror: + description: The mirror xlings and mcpp use in this job. + required: false + default: GLOBAL + +runs: + using: composite + steps: + - name: Download this commit's mcpp (${{ inputs.host }}) + uses: actions/download-artifact@v4 + with: + name: mcpp-built-${{ inputs.host }} + path: ${{ runner.temp }}/mcpp-built + + - name: Use this commit's mcpp + shell: bash + run: bash "$GITHUB_ACTION_PATH/use.sh" "${{ inputs.host }}" "${{ inputs.mirror }}" diff --git a/.github/actions/use-built-mcpp/use.sh b/.github/actions/use-built-mcpp/use.sh new file mode 100644 index 00000000..99641571 --- /dev/null +++ b/.github/actions/use-built-mcpp/use.sh @@ -0,0 +1,82 @@ +#!/usr/bin/env bash +# The body of the use-built-mcpp action; see action.yml for what it provides. +# +# Usage: use.sh +set -euo pipefail + +host="$1" +mirror="$2" + +dir="$RUNNER_TEMP/mcpp-built" +case "$host" in + windows-*) exe=mcpp.exe; dir="$(cygpath -u "$dir")" ;; + *) exe=mcpp ;; +esac +bin="$dir/$exe" +if [ ! -f "$bin" ]; then + echo "::error::the artifact mcpp-built-$host holds no $exe" + ls -la "$dir" || true + exit 1 +fi +chmod +x "$bin" + +boot="${MCPP:-}" +if [ -z "$boot" ]; then + echo "::error::MCPP is unset: run bootstrap-mcpp or setup-macos-llvm before use-built-mcpp" + exit 1 +fi + +# The toolchain mcpp.toml names for this host is the one the build used, so it +# is the one whose payloads hold the binary's runtime. +manifest_toolchain() { + local key + case "$host" in + macos-*) key=macos ;; + windows-*) key=windows ;; + *) key=default ;; + esac + awk -v k="$key" ' + /^\[/ { in_tc = ($0 == "[toolchain]") ; next } + in_tc && $1 == k { gsub(/"/, "", $3); print $3; exit } + ' mcpp.toml +} + +if ! out=$("$bin" --version 2>&1); then + tc="$(manifest_toolchain)" + echo "this commit's mcpp does not run yet ($out); installing ${tc:-the default toolchain} with the bootstrap" + if [ -n "$tc" ]; then + "$boot" toolchain install "${tc%@*}" "${tc#*@}" + fi + if ! out=$("$bin" --version 2>&1); then + echo "::error::this commit's mcpp does not run on this runner: $out" + exit 1 + fi +fi +echo "this commit's mcpp: $out ($bin)" + +# The mirror first: the runners are outside CN, and the install below and +# every later download read it. +if [ -n "${XLINGS_BIN:-}" ]; then + "$XLINGS_BIN" config --mirror "$mirror" 2>/dev/null || true +fi +MCPP_VENDORED_XLINGS="${XLINGS_BIN:-}" "$bin" self config --mirror "$mirror" + +# THE TOOLCHAIN THE BUILD USED IS INSTALLED, AS IT WAS WHEN EVERY JOB BUILT. +# A job that built mcpp itself installed this toolchain as a side effect, and +# the steps after the build relied on it without saying so: measured on the +# first run of this action, the aarch64 leg of the target matrix restored no +# sandbox, its binary ran without any payload, and the invariants that list the +# host's toolchains found none ("gcc is not installed here"). Installing it here +# keeps every consumer's environment what it was. It is a lookup when the +# toolchain is present. +tc="$(manifest_toolchain)" +if [ -n "$tc" ]; then + MCPP_VENDORED_XLINGS="${XLINGS_BIN:-}" "$bin" toolchain install "${tc%@*}" "${tc#*@}" +fi + +{ + echo "MCPP_BOOT=$boot" + echo "MCPP=$bin" + echo "MCPP_FRESH=$bin" + if [ -n "${XLINGS_BIN:-}" ]; then echo "MCPP_VENDORED_XLINGS=$XLINGS_BIN"; fi +} >> "$GITHUB_ENV" diff --git a/.github/tools/check_e2e_coverage.py b/.github/tools/check_e2e_coverage.py new file mode 100644 index 00000000..c98c9869 --- /dev/null +++ b/.github/tools/check_e2e_coverage.py @@ -0,0 +1,166 @@ +#!/usr/bin/env python3 +"""Every e2e test runs somewhere, or says why it cannot (rule R5). + +WHY THIS EXISTS + +`run_all.sh` exits 0 on a skip, and a skip line reads the same whether the +host legitimately lacks a capability or the runner was set up wrong. Measured +on 2026-10-01, before this check: 24 of 564 tests ran on no host and were named +by no workflow. Seven declared `llvm`, which no line granted; three declared +`musl`, whose probe named a release (15.1.0) the runners no longer installed; +seven declared `mingw-cross`, which no shard installed. Every one of those runs +was green. + +WHAT IT CHECKS + +It reads the per-test reports the shards write (`E2E_REPORT` of run_all.sh, one +`\\t\\t\\t` line per test) and decides, for every test +under tests/e2e: + + ran some report says pass, fail or timeout; + job no report ran it, but a workflow names it outside a comment, as a + whole token: by file name, by name without `.sh`, or by its number + as an `E2E_ONLY` pattern such as `239_*.sh`. A dedicated job runs it + and asserts its result itself. A name in a comment does not count, + and neither does a longer name that contains it; + excused tests/e2e/coverage-exceptions.tsv lists it with the reason no hosted + runner can run it; + uncovered none of these. The check fails. + +An exception for a test that ran, or for a test that does not exist, also fails +the check, so that the list cannot outlive the reason it records. + +With --timings-out DIR it writes `.tsv`, the measured duration of every +test that ran, merged across that host's shards, in the format run_all.sh reads +from E2E_TIMINGS; refreshing tests/e2e/timings/ is copying those files. A report +file is named `e2e-report--.tsv`. + +Usage: + check_e2e_coverage.py --reports DIR [--root DIR] [--timings-out DIR] +""" +from __future__ import annotations + +import argparse +import re +import sys +from collections import defaultdict +from pathlib import Path + +RAN = {"pass", "fail", "timeout"} +REPORT_NAME = re.compile(r"e2e-report-(?P[a-z0-9-]+?)-(?P\d+)\.tsv$") + + +def read_reports(directory: Path): + """Yield (host, shard, status, test, ms, detail) for every report line.""" + for path in sorted(directory.rglob("e2e-report-*.tsv")): + m = REPORT_NAME.search(path.name) + if not m: + continue + for line in path.read_text(encoding="utf-8", errors="replace").splitlines(): + parts = line.split("\t") + if len(parts) < 3: + continue + ms = int(parts[2]) if parts[2].isdigit() else 0 + yield m["host"], int(m["shard"]), parts[0], parts[1], ms, parts[3] if len(parts) > 3 else "" + + +def read_exceptions(path: Path) -> dict[str, str]: + out: dict[str, str] = {} + if not path.exists(): + return out + for line in path.read_text(encoding="utf-8").splitlines(): + if not line.strip() or line.startswith("#"): + continue + test, _, reason = line.partition("\t") + out[test.strip()] = reason.strip() + return out + + +TOKEN = re.compile(r"(? set[str]: + """Every test name or `_*` pattern named outside a comment.""" + tokens: set[str] = set() + for text in texts: + for line in text.splitlines(): + code = line.split("#", 1)[0] if line.lstrip().startswith("#") else line + if not code.strip(): + continue + tokens.update(m.group(1) for m in TOKEN.finditer(code)) + return tokens + + +def named_by_a_workflow(test: str, tokens: set[str]) -> bool: + stem = test[:-3] + number = stem.split("_", 1)[0] + return stem in tokens or f"{number}_*" in tokens + + +def main() -> int: + parser = argparse.ArgumentParser() + parser.add_argument("--reports", required=True) + parser.add_argument("--root", default=".") + parser.add_argument("--timings-out") + args = parser.parse_args() + root = Path(args.root).resolve() + + tests = sorted(p.name for p in (root / "tests" / "e2e").glob("[0-9]*.sh")) + workflows = workflow_tokens([p.read_text(encoding="utf-8") + for p in sorted((root / ".github" / "workflows").glob("*.yml"))]) + exceptions = read_exceptions(root / "tests" / "e2e" / "coverage-exceptions.tsv") + + ran: dict[str, set[str]] = defaultdict(set) + skipped: dict[str, set[str]] = defaultdict(set) + shard_ms: dict[tuple[str, int], int] = defaultdict(int) + timings: dict[str, dict[str, int]] = defaultdict(dict) + reports = 0 + for host, shard, status, test, ms, detail in read_reports(Path(args.reports)): + reports += 1 + if status in RAN: + ran[test].add(host) + shard_ms[(host, shard)] += ms + timings[host][test] = ms + elif status == "skip": + skipped[test].add(f"{host}: {detail}") + if reports == 0: + print(f"no report under {args.reports}: nothing can be said about coverage") + return 1 + + uncovered, stale = [], [] + counts = defaultdict(int) + for test in tests: + if test in ran: + counts["ran"] += 1 + if test in exceptions: + stale.append(f"{test} is excused but ran on {', '.join(sorted(ran[test]))}") + elif named_by_a_workflow(test, workflows): + counts["job"] += 1 + elif test in exceptions: + counts["excused"] += 1 + else: + uncovered.append(f"{test}: {'; '.join(sorted(skipped[test])) or 'in no report'}") + for test in exceptions: + if test not in tests: + stale.append(f"{test} is excused but does not exist") + + print(f"{len(tests)} tests: {counts['ran']} ran on a shard, {counts['job']} run by a " + f"dedicated job, {counts['excused']} excused, {len(uncovered)} uncovered") + for (host, shard), ms in sorted(shard_ms.items()): + print(f" {host} shard {shard}: {ms / 60000:.1f} min of tests") + for line in uncovered: + print(f"UNCOVERED: {line}") + for line in stale: + print(f"STALE EXCEPTION: {line}") + + if args.timings_out: + out = Path(args.timings_out) + out.mkdir(parents=True, exist_ok=True) + for host, table in timings.items(): + (out / f"{host}.tsv").write_text( + "".join(f"{t}\t{ms}\n" for t, ms in sorted(table.items())), encoding="utf-8") + return 1 if uncovered or stale else 0 + + +if __name__ == "__main__": + sys.exit(main()) diff --git a/.github/tools/classify_changes.py b/.github/tools/classify_changes.py new file mode 100644 index 00000000..b6d5a6ff --- /dev/null +++ b/.github/tools/classify_changes.py @@ -0,0 +1,140 @@ +#!/usr/bin/env python3 +"""Decide whether a change can affect anything but the documentation checks. + +Rule R2 of .agents/docs/2026-10-02-pr-ci-acceleration-and-the-toolchain- +specification-design.md: what a job runs follows from what the commit changed. +Measured before this rule (2026-10-01): a commit that changed two files under +`.agents/docs` ran 35 jobs, 36 builds of mcpp and 436 runner minutes. + +THE CLASSES + + documentation `*.md` anywhere, `docs/**`, `.agents/**`, `LICENSE*`. + code everything else, and every documentation file that a script, + a test, a workflow or a source file names. The second half is + DERIVED, by searching the repository, rather than listed: a test + added later that reads a document is found without editing this + program. The search looks for the file's path and for the path of + its translation (`docs/X` and `docs/zh/X` name each other), since + a check that reads one language usually reads both. It does not + search for a bare file name: `README.md` is named by release + packaging, and a match on the name alone would make every design + record, whose index is `.agents/docs/README.md`, a code change. + +A change whose every path is documentation that nothing names starts the +documentation checks only. Anything else, an empty change and a change that +cannot be listed included, starts the whole CI. + +Usage: + classify_changes.py [--root DIR] [--github-output FILE] < changed-paths + One path per line on standard input, relative to the repository root. + Prints `code=true` or `code=false` and one line per reason; with + --github-output, appends `code=...` to that file as well. +""" +from __future__ import annotations + +import argparse +import fnmatch +import os +import sys +from pathlib import Path + +DOC_PATTERNS = ("*.md", "docs/*", ".agents/*", "LICENSE*") +SEARCHED = ("tests", ".github", "src", "modules", "tools", "mcpp.toml", "bench/src") +SKIPPED_DIRS = {".git", "target", "node_modules", "__pycache__"} + +# A document read only by these files is still documentation: this program and +# its test name paths as data, and the checks of the `docs` job in ci.yml run on +# every change, documentation-only ones included. tests/scripts/ +# test_classify_changes.py requires each check listed here to appear in ci.yml. +DOCS_JOB_CHECKS = ( + ".github/tools/check_docs_style.sh", + ".github/tools/check_docs_structure.sh", + ".github/tools/gen_agents_index.py", + ".github/tools/check_target_tiers.py", + ".github/tools/check_reason_tokens.sh", + ".github/tools/check_matrix_reasons.sh", + ".github/tools/check_version_pins.sh", +) +NOT_READERS = (".github/tools/classify_changes.py", "tests/scripts/test_classify_changes.py") \ + + DOCS_JOB_CHECKS + + +def is_documentation(path: str) -> bool: + return any(fnmatch.fnmatchcase(path, p) for p in DOC_PATTERNS) or \ + fnmatch.fnmatchcase(Path(path).name, "*.md") + + +def searched_files(root: Path): + for entry in SEARCHED: + base = root / entry + if base.is_file(): + yield base + continue + if not base.is_dir(): + continue + for dirpath, dirnames, filenames in os.walk(base): + dirnames[:] = [d for d in dirnames if d not in SKIPPED_DIRS] + for name in filenames: + if name.endswith(".md"): + continue + yield Path(dirpath) / name + + +def names_of(path: str) -> tuple[str, ...]: + """The path, and the path of its translation under docs/.""" + if path.startswith("docs/zh/"): + return path, "docs/" + path[len("docs/zh/"):] + if path.startswith("docs/"): + return path, "docs/zh/" + path[len("docs/"):] + return (path,) + + +def readers_of(root: Path, docs: list[str]) -> dict[str, list[str]]: + """Map each documentation path to the files that name it.""" + needles = {d: names_of(d) for d in docs} + found: dict[str, list[str]] = {d: [] for d in docs} + for f in searched_files(root): + if str(f.relative_to(root)).replace(os.sep, "/") in NOT_READERS: + continue + try: + text = f.read_text(encoding="utf-8", errors="ignore") + except OSError: + continue + for d, names in needles.items(): + if any(n in text for n in names): + found[d].append(str(f.relative_to(root))) + return found + + +def classify(root: Path, paths: list[str]) -> tuple[bool, list[str]]: + paths = [p.strip() for p in paths if p.strip()] + if not paths: + return True, ["the change lists no path"] + code = [p for p in paths if not is_documentation(p)] + if code: + shown = ", ".join(code[:5]) + (" ..." if len(code) > 5 else "") + return True, [f"{len(code)} path(s) outside the documentation: {shown}"] + named = {d: r for d, r in readers_of(root, paths).items() if r} + if named: + return True, [f"{d} is read by {', '.join(r[:3])}" for d, r in sorted(named.items())] + return False, [f"{len(paths)} documentation path(s), none named by a script, test or source"] + + +def main() -> int: + parser = argparse.ArgumentParser() + parser.add_argument("--root", default=".") + parser.add_argument("--github-output") + args = parser.parse_args() + code, reasons = classify(Path(args.root).resolve(), sys.stdin.read().splitlines()) + value = "true" if code else "false" + print(f"code={value}") + for r in reasons: + print(f" {r}") + if args.github_output: + with open(args.github_output, "a", encoding="utf-8") as out: + out.write(f"code={value}\n") + return 0 + + +if __name__ == "__main__": + sys.exit(main()) diff --git a/.github/workflows/build.yml b/.github/workflows/build.yml new file mode 100644 index 00000000..15cfc57f --- /dev/null +++ b/.github/workflows/build.yml @@ -0,0 +1,121 @@ +name: build + +# ONE BUILD OF THIS COMMIT'S MCPP PER HOST (rule R1 of +# .agents/docs/2026-10-02-pr-ci-acceleration-and-the-toolchain-specification-design.md). +# +# Called by ci.yml once per host. The binary is uploaded as the artifact +# `mcpp-built-`, and every job of the commit that needs this commit's mcpp +# takes it with .github/actions/use-built-mcpp instead of building its own. +# Measured before this workflow existed (2026-10-01): thirty to thirty-seven +# jobs per commit built mcpp from source, which was 44 to 49 percent of the +# commit's runner minutes, and none of those builds was incremental. +# +# This job is also the ONE WRITER of the host's sandbox and xlings caches (rule +# R3), and it writes them only on a push to main. Before it saves, it installs +# the toolchains its consumers need, so that the cache a pull request restores +# already holds them. + +on: + workflow_call: + inputs: + host: + description: linux-x86_64, linux-aarch64, macos-arm64 or windows-x86_64 + type: string + required: true + runs-on: + description: the runner label + type: string + required: true + prewarm: + description: > + Toolchains installed before the caches are saved on main, separated + by ';', each as `mcpp toolchain install` takes it (`gcc 16.1.0-musl`). + type: string + required: false + default: '' + +jobs: + build: + name: build mcpp (${{ inputs.host }}) + runs-on: ${{ inputs.runs-on }} + timeout-minutes: 45 + steps: + - uses: actions/checkout@v4 + + - id: boot + if: ${{ !startsWith(inputs.host, 'macos-') }} + uses: ./.github/actions/bootstrap-mcpp + + - id: mac + if: ${{ startsWith(inputs.host, 'macos-') }} + uses: ./.github/actions/setup-macos-llvm + with: + image: ${{ inputs.runs-on }} + + - name: Build mcpp from source (self-host) + shell: bash + run: | + set -euo pipefail + export MCPP_VENDORED_XLINGS="$XLINGS_BIN" + "$XLINGS_BIN" config --mirror GLOBAL 2>/dev/null || true + "$MCPP" self config --mirror GLOBAL 2>/dev/null || true + "$MCPP" build + # target/ is not restored from a cache, so the tree holds exactly the + # binary this build linked. + case "${{ inputs.host }}" in windows-*) exe=mcpp.exe ;; *) exe=mcpp ;; esac + built=$(find target -type f -name "$exe" -path '*/bin/*' | grep -v '/dist/' || true) + [ "$(printf '%s\n' "$built" | grep -c .)" = 1 ] || { + echo "::error::expected one $exe under target/, found: ${built:-none}"; exit 1; } + mkdir -p dist + cp "$built" "dist/$exe" + "dist/$exe" --version + + - name: Upload this commit's mcpp + timeout-minutes: 5 + uses: actions/upload-artifact@v4 + with: + name: mcpp-built-${{ inputs.host }} + path: dist/${{ startsWith(inputs.host, 'windows-') && 'mcpp.exe' || 'mcpp' }} + if-no-files-found: error + retention-days: 3 + + # Only on main, and only for the caches this job restored: a pull request + # reads the main lineage and adds nothing to the store. + - name: Install the toolchains the consumers use + if: ${{ github.event_name == 'push' && inputs.prewarm != '' }} + shell: bash + run: | + set -euo pipefail + IFS=';' read -ra specs <<< "${{ inputs.prewarm }}" + for spec in "${specs[@]}"; do + spec="$(echo "$spec" | xargs)" + [ -n "$spec" ] || continue + echo "::group::mcpp toolchain install $spec" + # shellcheck disable=SC2086 -- a spec is ` ` + "$MCPP" toolchain install $spec + echo "::endgroup::" + done + + - name: Save the sandbox cache + if: ${{ github.event_name == 'push' && steps.boot.outcome == 'success' && steps.boot.outputs.sandbox-hit != 'true' }} + uses: actions/cache/save@v4 + with: + path: ~/.mcpp + key: ${{ steps.boot.outputs.sandbox-key }} + + - name: Save the xlings cache + if: ${{ github.event_name == 'push' && steps.boot.outcome == 'success' && steps.boot.outputs.xlings-hit != 'true' }} + uses: actions/cache/save@v4 + with: + path: ~/.xlings + key: ${{ steps.boot.outputs.xlings-key }} + + # macOS keeps no sandbox cache: setup-macos-llvm builds the sandbox from + # the freshly resolved LLVM on every run, which is what the macOS jobs + # exist to prove. Its xlings cache has this job as its writer. + - name: Save the xlings cache (macOS) + if: ${{ github.event_name == 'push' && steps.mac.outcome == 'success' && steps.mac.outputs.xlings-hit != 'true' }} + uses: actions/cache/save@v4 + with: + path: ~/.xlings + key: ${{ steps.mac.outputs.xlings-key }} diff --git a/.github/workflows/ci-linux-e2e.yml b/.github/workflows/ci-linux-e2e.yml index a26be330..8c5b3966 100644 --- a/.github/workflows/ci-linux-e2e.yml +++ b/.github/workflows/ci-linux-e2e.yml @@ -1,40 +1,31 @@ name: ci-linux-e2e -# The e2e suite (tests/e2e/run_all.sh) split out of ci-linux.yml so it runs in -# PARALLEL with the build/unit/toolchain jobs instead of tacked on after them, -# and SHARDED across two runners on top of that. Both workflows share the same -# cache lineage (mcpp sandbox + xlings + target/), so each shard restores a warm -# build and the only added wall-clock vs. the inline version is one extra warm -# `mcpp build` per runner. +# The e2e suite (tests/e2e/run_all.sh) on Linux, called by ci.yml beside +# ci-linux.yml after the Linux build. Four shards, assigned by measured +# duration (tests/e2e/timings/linux.tsv, rule R4 of the 2026-10-02 CI record), +# each running the one binary build.yml produced (use-built-mcpp, rule R1). +# Each shard writes a per-test report that the e2e-coverage job of ci.yml reads +# (rule R5). # -# Paired workflows: ci-linux.yml (build + unit + toolchain legs + integration), +# Paired workflows: ci-linux.yml (unit + toolchain legs + integration), # ci-macos.yml / ci-macos-e2e.yml, ci-windows.yml / ci-windows-e2e.yml. on: - push: - branches: [ main ] - pull_request: - branches: [ main ] - workflow_dispatch: - -concurrency: - group: ci-${{ github.workflow }}-${{ github.ref }} - cancel-in-progress: true + workflow_call: jobs: e2e: - name: e2e ${{ matrix.shard }}/2 (linux x86_64, self-host) + name: e2e ${{ matrix.shard }}/4 (linux x86_64, self-host) runs-on: ubuntu-24.04 timeout-minutes: 45 strategy: fail-fast: false matrix: - shard: [1, 2] + shard: [1, 2, 3, 4] env: - # Round-robin slice of tests/e2e (see run_all.sh). Two shards halve the - # suite's wall-clock; the split is computed on the full file list, so a - # test's shard does not move when host capabilities differ. - E2E_SHARD: ${{ matrix.shard }}/2 + E2E_SHARD: ${{ matrix.shard }}/4 + E2E_TIMINGS: tests/e2e/timings/linux.tsv + E2E_REPORT: ${{ github.workspace }}/e2e-report-linux-${{ matrix.shard }}.tsv MCPP_HOME: /home/runner/.mcpp # NOTE: do NOT force MCPP_VERBOSE here. The e2e suite includes tests that # assert mcpp's DEFAULT (quiet) output — e.g. 48_build_error_output and @@ -45,74 +36,71 @@ jobs: # `submodules: recursive` so tests/e2e/233_bench_matrix.sh can check that # each `hub`/`body` in bench/matrix.json exists in the tree it names. # Without the trees that check reads "submodule not initialised" and - # reports nothing, which is how a hub path written for the CURRENT - # layout stayed in matrix.json while the workload it named is a - # HISTORICAL mcpp -- three cells reporting `skipped` on every bench run - # and the job still green (#599). - # - # It is not free and it is not expensive: the three pinned workloads are - # 725 + 701 + 806 tracked files, under 10 MB of source in total, and - # nothing here builds them. Both shards carry it because run_all.sh - # slices the file list round-robin, so which shard holds 233 moves when - # a test is added. + # reports nothing (#599). Under 10 MB of source; every shard carries it, + # because which shard holds 233 follows from the timing table. - uses: actions/checkout@v4 with: submodules: recursive - # Same cache lineage as ci-linux.yml so this job lands on a warm - # toolchain/sandbox instead of re-installing it. - uses: ./.github/actions/bootstrap-mcpp - - name: Configure mirror + Build mcpp from source (self-host) + - uses: ./.github/actions/use-built-mcpp + with: + host: linux-x86_64 + + # Its own step and its own limit, so that downloads on a cold sandbox are + # not taken from the suite's budget. + - name: The toolchains the suite probes + timeout-minutes: 15 run: | - export MCPP_VENDORED_XLINGS="$XLINGS_BIN" - "$XLINGS_BIN" config --mirror GLOBAL 2>/dev/null || true - "$MCPP" self config --mirror GLOBAL 2>/dev/null || true - "$MCPP" build + set -euo pipefail + # Pin the global default so test 28 (default-toolchain path) gets a + # deterministic GNU answer instead of an auto-install pick. Installed + # explicitly, not assumed: the restored sandbox is the build job's, + # and a toolchain that only happens to be in it is a dependency on a + # cache. + "$MCPP" toolchain install gcc 16.1.0 + "$MCPP" toolchain default gcc@16.1.0 + # The toolchains whose capabilities the suite probes (run_all.sh): + # musl, llvm (which also yields scan-deps and import-std-libcxx) and + # the Linux-hosted MinGW cross compiler. Without them their tests + # skip on every runner; measured 2026-10-01, seventeen such tests ran + # nowhere. Warm runs re-install nothing. + "$MCPP" toolchain install gcc 16.1.0-musl + "$MCPP" toolchain install llvm 22.1.8 + "$MCPP" toolchain install mingw-cross 16.1.0 + XLINGS_HOME="$MCPP_HOME/registry" "$MCPP_VENDORED_XLINGS" install xim:nasm -y - name: E2E suite - # Per-test 600s timeout lives in tests/e2e/run_all.sh and identifies - # WHICH test hung; this caps the whole suite so a hang fails fast. - timeout-minutes: 25 + # About twice the shard's budget of ten minutes (R4): reached only by a + # hang. The per-test 600 s limit in run_all.sh names the test that hung. + timeout-minutes: 22 run: | - # Point the e2e runner at the freshly-built binary, not the - # bootstrap one. Tests cd into mktemp -d, so $MCPP must be - # absolute or the relative path breaks under the temp cwd. - # The RELEASED mcpp this job bootstrapped from, kept for e2e 252: the - # claim that an older client can still build against a package the PR - # produces is only worth making if something checks it against a real - # old binary. Captured before $MCPP is repointed at the fresh build. - export MCPP_BOOT="$MCPP" - MCPP=$(realpath "$(find target -type f -name mcpp -printf '%T@ %p\n' | sort -rn | head -1 | cut -d' ' -f2)") - test -x "$MCPP" - export MCPP - # Tests that set MCPP_HOME to a fresh tmpdir need an xlings to - # bootstrap from; surface the xlings binary installed above. - export MCPP_VENDORED_XLINGS="$XLINGS_BIN" + set -euo pipefail + # MCPP is this commit's binary and MCPP_BOOT the released bootstrap + # (use-built-mcpp). e2e 252 needs the latter: the claim that an older + # client can build against a package this commit produces is only + # worth making against a real old binary. + export MCPP MCPP_BOOT test -x "$MCPP_VENDORED_XLINGS" # GitHub-hosted runners are outside CN; keep CI toolchain downloads on # the global mirror while mcpp's default remains CN for fresh local # sandboxes. E2E tests with their own MCPP_HOME read this variable. export MCPP_E2E_TOOLCHAIN_MIRROR=GLOBAL - "$MCPP" self config --mirror "$MCPP_E2E_TOOLCHAIN_MIRROR" "$MCPP" self config - # Pin the global default so test 28 (default-toolchain path) gets a - # deterministic GNU answer instead of an auto-install pick. - # - # Installed explicitly, not assumed. `toolchain default` requires the - # toolchain to be present, and nothing in this job puts it there: - # building mcpp above uses whatever first-run picks (llvm). It worked - # only because the sandbox cache happened to carry a gcc some earlier - # run had installed -- so the moment the cache key changed (an xlings - # pin bump moves the whole prefix), every e2e shard failed with - # `gcc@16.1.0 is not installed` before running a single test. - # Warm runs re-install nothing; this costs a lookup. - "$MCPP" toolchain install gcc 16.1.0 - "$MCPP" toolchain default gcc@16.1.0 - # Warm musl once so fresh-home e2e tests inherit the payload. - "$MCPP" toolchain install gcc 16.1.0-musl bash tests/e2e/run_all.sh + # One file, no glob: the report the e2e-coverage job of ci.yml reads. + - name: Upload the shard's report + if: always() + timeout-minutes: 5 + uses: actions/upload-artifact@v4 + with: + name: e2e-report-linux-${{ matrix.shard }} + path: e2e-report-linux-${{ matrix.shard }}.tsv + if-no-files-found: warn + retention-days: 7 + # ────────────────────────────────────────────────────────────────── # Bare metal: the one chain the sharded suite above cannot be trusted # to exercise. @@ -138,12 +126,9 @@ jobs: - uses: actions/checkout@v4 - uses: ./.github/actions/bootstrap-mcpp - - name: Build mcpp from source (self-host) - run: | - export MCPP_VENDORED_XLINGS="$XLINGS_BIN" - "$XLINGS_BIN" config --mirror GLOBAL 2>/dev/null || true - "$MCPP" self config --mirror GLOBAL 2>/dev/null || true - "$MCPP" build + - uses: ./.github/actions/use-built-mcpp + with: + host: linux-x86_64 - name: Install the emulator (xim:qemu-riscv) run: | @@ -184,12 +169,8 @@ jobs: - name: Bare-metal e2e timeout-minutes: 25 run: | - MCPP=$(realpath "$(find target -type f -name mcpp -printf '%T@ %p\n' | sort -rn | head -1 | cut -d' ' -f2)") - test -x "$MCPP" - export MCPP - export MCPP_VENDORED_XLINGS="$XLINGS_BIN" + export MCPP MCPP_VENDORED_XLINGS export MCPP_E2E_TOOLCHAIN_MIRROR=GLOBAL - "$MCPP" self config --mirror "$MCPP_E2E_TOOLCHAIN_MIRROR" # llvm is the toolchain a freestanding target pins; install it # explicitly rather than relying on whatever the sandbox cache holds. "$MCPP" toolchain install llvm 22.1.8 @@ -278,12 +259,9 @@ jobs: - uses: actions/checkout@v4 - uses: ./.github/actions/bootstrap-mcpp - - name: Build mcpp from source (self-host) - run: | - export MCPP_VENDORED_XLINGS="$XLINGS_BIN" - "$XLINGS_BIN" config --mirror GLOBAL 2>/dev/null || true - "$MCPP" self config --mirror GLOBAL 2>/dev/null || true - "$MCPP" build + - uses: ./.github/actions/use-built-mcpp + with: + host: linux-x86_64 - name: Install the NDK (xim:android-ndk) timeout-minutes: 20 @@ -296,12 +274,8 @@ jobs: - name: Android e2e (no device) timeout-minutes: 40 run: | - MCPP=$(realpath "$(find target -type f -name mcpp -printf '%T@ %p\n' | sort -rn | head -1 | cut -d' ' -f2)") - test -x "$MCPP" - export MCPP - export MCPP_VENDORED_XLINGS="$XLINGS_BIN" + export MCPP MCPP_VENDORED_XLINGS export MCPP_E2E_TOOLCHAIN_MIRROR=GLOBAL - "$MCPP" self config --mirror "$MCPP_E2E_TOOLCHAIN_MIRROR" "$MCPP" toolchain install gcc 16.1.0 "$MCPP" toolchain default gcc@16.1.0 # Run directly, not through run_all.sh, so a skip cannot pass as a @@ -374,11 +348,12 @@ jobs: # Payload cache (downloads only — the container still has no host # toolchain, which is the property under test). - - name: Cache mcpp sandbox payloads - uses: actions/cache@v4 + - name: Restore mcpp sandbox payloads + id: hermetic-cache + uses: actions/cache/restore@v4 with: path: ~/.mcpp - key: mcpp-hermetic-${{ hashFiles('mcpp.toml') }} + key: mcpp-hermetic-${{ hashFiles('mcpp.toml', '.github/workflows/ci-linux-e2e.yml') }} restore-keys: | mcpp-hermetic- @@ -393,6 +368,8 @@ jobs: "$MCPP_BOOT" self config --mirror GLOBAL echo "MCPP_BOOT=$MCPP_BOOT" >> "$GITHUB_ENV" + # This job builds its own mcpp on purpose: the build is part of what it + # tests, a compiler-free container in which only sandbox payloads exist. - name: Build PR mcpp from source (sandbox gcc only) run: | "$MCPP_BOOT" build @@ -495,3 +472,12 @@ jobs: "ok: provides['std'] names the std-cache BMI" \ "ok: build-id is present and stable across two runs" \ "PASS: 786 the std unit in the database, emit/build agreement, provides and build-id" + + # Last, so that the payloads the steps above installed are in it. This job + # is the one writer of its cache, on main only (R3). + - name: Save mcpp sandbox payloads + if: ${{ github.event_name == 'push' && steps.hermetic-cache.outputs.cache-hit != 'true' }} + uses: actions/cache/save@v4 + with: + path: ~/.mcpp + key: ${{ steps.hermetic-cache.outputs.cache-primary-key }} diff --git a/.github/workflows/ci-linux.yml b/.github/workflows/ci-linux.yml index 448787e3..d4dade3d 100644 --- a/.github/workflows/ci-linux.yml +++ b/.github/workflows/ci-linux.yml @@ -5,30 +5,25 @@ name: ci-linux # workflow no longer depends on a previous-release tarball — the # chicken-and-egg now lives upstream in the xlings index. # -# SHAPE: four INDEPENDENT jobs, no `needs:` between them. Each restores the -# same cache lineage (see .github/actions/bootstrap-mcpp) and pays one warm -# `mcpp build` (~2.5 min) to get the PR's own binary, then does its own leg. -# That warm rebuild is far cheaper than serialising the legs behind a shared -# artifact would be: +# SHAPE: called by ci.yml after the Linux build (build.yml). No job here builds +# mcpp to obtain the commit's binary: each takes the one binary that build +# produced (.github/actions/use-built-mcpp, rule R1 of the 2026-10-02 CI +# record). A job builds mcpp only when the build is what it tests: a cold +# rebuild, another toolchain, a cross target. # -# before: build → unit → gcc → musl → llvm → xlings ≈ 18 min (one job) -# after: max(build+unit, gcc, musl+llvm, xlings) ≈ 7-8 min +# This replaced four independent jobs that each paid a "warm" rebuild, chosen +# when that rebuild cost about 2.5 minutes. Measured on 2026-10-01 it cost 6.6 +# minutes, and on an exact hit of the restored target/ ninja still ran 830 of +# 830 edges. # -# The ~18 min e2e suite is a SEPARATE workflow (ci-linux-e2e.yml, sharded) -# that runs in parallel on the same caches. +# The checks that need no binary run in the `docs` job of ci.yml. +# +# The e2e suite is ci-linux-e2e.yml, called beside this one. # # Paired workflows: ci-linux-e2e.yml, ci-macos.yml, ci-windows.yml. on: - push: - branches: [ main ] - pull_request: - branches: [ main ] - workflow_dispatch: - -concurrency: - group: ci-${{ github.workflow }}-${{ github.ref }} - cancel-in-progress: true + workflow_call: env: # MCPP_HOME pinned so the cache keys below restore into the same path @@ -41,138 +36,20 @@ env: jobs: build-test: - name: build + unit tests (linux x86_64, self-host) + name: unit tests (linux x86_64, self-host) runs-on: ubuntu-24.04 timeout-minutes: 45 steps: - uses: actions/checkout@v4 - # Before the bootstrap, not after: this needs no toolchain and no mcpp, - # takes under a second, and the drift it catches (a stale xlings pin) - # would otherwise surface minutes later as an unrelated-looking - # dependency-resolution failure. Pure text extraction on purpose — it - # has to work when the build is broken. - - name: Check version / xlings pin consistency - run: bash .github/tools/check_version_pins.sh - - # Same placement and the same reason as the pin check above: adding a - # package under `modules/` means editing four files, and three of those - # omissions fail later and elsewhere — the worst of them only on a macOS - # machine that has no mcpp to rebuild with. - - name: Check modules/ wiring - run: bash .github/tools/check_modules_wiring.sh - - # Same placement, same reason: pure text, no toolchain, under a second. - # Catches the prepare.cppm decomposition growing back into one huge - # file one commit at a time — see the script's header. - - name: Check src/build/prepare* file lengths - run: bash .github/tools/check_file_lengths.sh - - # Same placement, same reason: pure text, no toolchain, under a second. - # - # This one is a HARD gate (unlike lint-ci-assertions.sh below) because it - # has no false positives left — its scope was cut to the two directories - # that walk trees mcpp does not control, and the one legitimate site - # carries a NARROW-OK rationale. See the script's header for why the - # scope is that narrow, and mcpp#516 for what it costs when it is missed. - # - # It runs on LINUX on purpose even though the bug it guards is - # Windows-only: it is text analysis, and putting it where the fast leg is - # means a violation is reported in seconds rather than after a Windows - # bootstrap. - - name: Check no walk-derived path is narrowed directly - run: bash .github/tools/check_narrow_conversions.sh - - # Same placement, same reason: pure text, no toolchain. - # - # IT PRINTS AND DOES NOT FAIL, DELIBERATELY. The three rules it carries - # each have real false positives — a control step legitimately pins one - # target, a `grep -q .` legitimately expects output — and the whole value - # of the thing is that somebody READS it. A hard gate would be routed - # around with a suppression within a month, and the suppression would then - # be the only record that a rule ever existed. - - name: Where the CI assertions live - run: bash tools/lint-ci-assertions.sh - - # A GATE, unlike the warnings above: a step asserts what its name says - # (the 2026-09-28 design, WS7, #729). A build piped into `tee` without - # pipefail, a build step whose status is discarded before a grep, and a - # job allowed to fail without an open issue are refused. The fixture - # tests run first, so a lint that stopped reading files fails here - # rather than passing every workflow. - - name: Steps assert what their names say - env: - GH_TOKEN: ${{ github.token }} - run: | - python3 tests/scripts/test_check_workflow_assertions.py - python3 .github/tools/check_workflow_assertions.py --check-open - - - name: The release canary runner runs each command under the named bash - run: python3 tests/scripts/test_release_canaries.py - - - name: The protocol table of SPEC-007 names the engine's protocol - run: python3 tests/scripts/test_protocol_table.py - - # Text-only, like the two steps around it, and it belongs here rather - # than in the target-matrix workflow: that workflow runs the matrix, and - # this asserts a property of the TABLE, which is readable without a - # single toolchain. - - name: The target matrix names every refusal - run: bash .github/tools/check_matrix_reasons.sh - - # Same placement and the same reason: pure text, no toolchain, and the - # drift it catches — a 简体中文 page that has silently fallen behind its - # English original — is invisible to every other job. - - name: Check documentation style and bilingual parity - run: bash .github/tools/check_docs_style.sh - - # The architecture half: which tree a document lives in, whether every - # path it names resolves, and whether each specification is reachable - # from all three indexes. Same placement and the same reason -- it needs - # no toolchain, and what it catches is invisible to every other job. - - name: Check documentation structure - run: bash .github/tools/check_docs_structure.sh - - # Every documented tier agrees with kKnownTargets. - # - # Four documents restate a row's tier and nothing compared them to - # the table. When wasm32-emscripten became `verified` and the - # Android rows gained tiers, docs/21 was updated and both READMEs - # were not -- so the front page said three targets were `planned` - # while the engine had built and run two of them. On its first run - # this check also found two Cortex-A rows missing from docs/21 - # entirely, which predates that work. - - name: Documented target tiers agree with the table - run: python3 .github/tools/check_target_tiers.py - - # The reason-token table in docs/50 is a MACHINE INTERFACE: mcpp-index's - # compatibility measurement reads a token out of a refusal to tell "this - # graph does not supply what the member asked for" from "the member did - # not build", and that distinction decides a published figure. A token - # the engine emits and the table omits is a promise nobody can rely on. - # - # Four tokens were added to that table by hand as "the ones it was - # missing"; a later enumeration found four more that had been absent the - # whole time. A set compared by reading is a set compared by sampling. - - name: Reason tokens agree with the engine, and with the mirror - run: bash .github/tools/check_reason_tokens.sh - - uses: ./.github/actions/bootstrap-mcpp - - name: Configure mirror + Build mcpp from source (self-host) - run: | - export MCPP_VENDORED_XLINGS="$XLINGS_BIN" - # Set GLOBAL mirror via xlings directly (bootstrap mcpp may lack --mirror flag) - "$XLINGS_BIN" config --mirror GLOBAL 2>/dev/null || true - "$MCPP" self config --mirror GLOBAL 2>/dev/null || true - "$MCPP" build + - uses: ./.github/actions/use-built-mcpp + with: + host: linux-x86_64 - name: Unit + integration tests via `mcpp test` run: | - # Use freshly-built mcpp for test (it has --mirror support) - MCPP_FRESH=$(realpath "$(find target -type f -name mcpp -printf '%T@ %p\n' | sort -rn | head -1 | cut -d' ' -f2)") - echo "MCPP_FRESH=$MCPP_FRESH" >> "$GITHUB_ENV" - "$MCPP_FRESH" self config --mirror GLOBAL "$MCPP_FRESH" test # Each package under `modules/` carries its own tests, and they are built @@ -196,17 +73,8 @@ jobs: - name: Per-subsystem tests (`mcpp test -p `) run: | set -euo pipefail - # `$MCPP_FRESH` comes from the step above, which takes the NEWEST - # mcpp anywhere under target/. That is correct in this job because - # nothing between the two steps writes another one — but `mcpp pack` - # leaves a copy under `target/dist/`, and a pack step inserted above - # would silently redirect both this loop and the suite before it to - # the packaged binary. Observed locally; kept as a note because the - # expression is shared with three other jobs and this is not the - # place to change it. - case "$MCPP_FRESH" in - */target/dist/*) echo "MCPP_FRESH points into target/dist — a pack step now runs before the tests"; exit 1 ;; - esac + # `$MCPP_FRESH` is the binary of build.yml (use-built-mcpp), not one + # found under target/. members=$(sed -n '/^\[workspace\]/,/^\[/p' mcpp.toml \ | grep -oE '"modules/[a-z0-9-]+"' | tr -d '"' | sed 's|modules/||') [ -n "$members" ] || { echo "no workspace members found in mcpp.toml"; exit 1; } @@ -217,11 +85,10 @@ jobs: echo "::endgroup::" done - # A cold, from-scratch self-host build with the manifest-pinned GCC: the - # property `build-test` cannot cover, because it builds incrementally on a - # restored target/. `mcpp test` is deliberately NOT repeated here — it would - # be the same suite, same toolchain, same driver binary as `build-test`, - # differing only in incremental state. + # A cold, from-scratch self-host build by this commit's binary with the + # manifest-pinned GCC: mcpp building itself, where build.yml has the + # bootstrap build it. `mcpp test` is deliberately NOT repeated here; it would + # be the same suite, toolchain and binary as `build-test`. toolchain-gcc: name: "toolchain: gcc (cold self-host)" runs-on: ubuntu-24.04 @@ -230,15 +97,9 @@ jobs: - uses: actions/checkout@v4 - uses: ./.github/actions/bootstrap-mcpp - - name: Build mcpp from source (self-host) - run: | - export MCPP_VENDORED_XLINGS="$XLINGS_BIN" - "$XLINGS_BIN" config --mirror GLOBAL 2>/dev/null || true - "$MCPP" self config --mirror GLOBAL 2>/dev/null || true - "$MCPP" build - MCPP_FRESH=$(realpath "$(find target -type f -name mcpp -printf '%T@ %p\n' | sort -rn | head -1 | cut -d' ' -f2)") - cp "$MCPP_FRESH" /tmp/mcpp-fresh - echo "MCPP=/tmp/mcpp-fresh" >> "$GITHUB_ENV" + - uses: ./.github/actions/use-built-mcpp + with: + host: linux-x86_64 - name: "Toolchain: GCC — cold rebuild with the PR binary" run: | @@ -248,8 +109,8 @@ jobs: "$MCPP" clean "$MCPP" build 2>&1 | tee build.log; grep -q "Resolved gcc@16.1.0" build.log - # The two cheap legs share one runner: each is ~1 min after the warm build, - # so a runner apiece would cost more in setup than it saves in wall-clock. + # The two rebuilds share one runner, each with this commit's binary as the + # builder: a runner apiece would cost more in setup than it saves. toolchain-cross: name: "toolchain: musl + llvm" runs-on: ubuntu-24.04 @@ -258,15 +119,9 @@ jobs: - uses: actions/checkout@v4 - uses: ./.github/actions/bootstrap-mcpp - - name: Build mcpp from source (self-host) - run: | - export MCPP_VENDORED_XLINGS="$XLINGS_BIN" - "$XLINGS_BIN" config --mirror GLOBAL 2>/dev/null || true - "$MCPP" self config --mirror GLOBAL 2>/dev/null || true - "$MCPP" build - MCPP_FRESH=$(realpath "$(find target -type f -name mcpp -printf '%T@ %p\n' | sort -rn | head -1 | cut -d' ' -f2)") - cp "$MCPP_FRESH" /tmp/mcpp-fresh - echo "MCPP=/tmp/mcpp-fresh" >> "$GITHUB_ENV" + - uses: ./.github/actions/use-built-mcpp + with: + host: linux-x86_64 # Auto-installs gcc@16.1.0-musl on demand (cached across runs). - name: "Toolchain: musl-gcc — build mcpp (--target)" @@ -313,15 +168,9 @@ jobs: - uses: actions/checkout@v4 - uses: ./.github/actions/bootstrap-mcpp - - name: Build mcpp from source (self-host) - run: | - export MCPP_VENDORED_XLINGS="$XLINGS_BIN" - "$XLINGS_BIN" config --mirror GLOBAL 2>/dev/null || true - "$MCPP" self config --mirror GLOBAL 2>/dev/null || true - "$MCPP" build - MCPP_FRESH=$(realpath "$(find target -type f -name mcpp -printf '%T@ %p\n' | sort -rn | head -1 | cut -d' ' -f2)") - cp "$MCPP_FRESH" /tmp/mcpp-fresh - echo "MCPP=/tmp/mcpp-fresh" >> "$GITHUB_ENV" + - uses: ./.github/actions/use-built-mcpp + with: + host: linux-x86_64 - name: "Integration: mcpp builds & runs xlings (openxlings/xlings)" run: | @@ -353,15 +202,9 @@ jobs: - uses: actions/checkout@v4 - uses: ./.github/actions/bootstrap-mcpp - - name: Build mcpp from source (self-host) - run: | - export MCPP_VENDORED_XLINGS="$XLINGS_BIN" - "$XLINGS_BIN" config --mirror GLOBAL 2>/dev/null || true - "$MCPP" self config --mirror GLOBAL 2>/dev/null || true - "$MCPP" build - MCPP_FRESH=$(realpath "$(find target -type f -name mcpp -printf '%T@ %p\n' | sort -rn | head -1 | cut -d' ' -f2)") - cp "$MCPP_FRESH" /tmp/mcpp-fresh - echo "MCPP=/tmp/mcpp-fresh" >> "$GITHUB_ENV" + - uses: ./.github/actions/use-built-mcpp + with: + host: linux-x86_64 - name: Build every example the runner can build run: | diff --git a/.github/workflows/ci-macos-e2e.yml b/.github/workflows/ci-macos-e2e.yml index 62e53072..10548091 100644 --- a/.github/workflows/ci-macos-e2e.yml +++ b/.github/workflows/ci-macos-e2e.yml @@ -1,28 +1,31 @@ name: ci-macos-e2e -# The e2e suite on macOS ARM64, split out of ci-macos.yml (where it was 3.4 of -# the job's 8.4 min) so it runs in parallel with the toolchain/integration -# work — same shape as ci-linux-e2e.yml / ci-windows-e2e.yml. +# The e2e suite on macOS ARM64, called by ci.yml beside ci-macos.yml after the +# macOS build. Two shards, assigned by measured duration +# (tests/e2e/timings/macos.tsv, rule R4 of the 2026-10-02 CI record), each +# running the one binary build.yml produced (use-built-mcpp, rule R1), and +# each writing the per-test report the e2e-coverage job of ci.yml reads. # -# Not sharded: the macOS runners finish the suite in ~3.5 min, which is under -# this workflow's setup+build floor, so a second runner would buy nothing. +# The suite took about 3.5 minutes when this workflow was split out of +# ci-macos.yml, and it was not sharded for that reason. Measured on 2026-10-01 +# it took 17 to 21 minutes in one job, and reached the 25-minute limit once. # # Paired workflows: ci-macos.yml, ci-linux-e2e.yml, ci-windows-e2e.yml. on: - push: - branches: [ main ] - pull_request: - branches: [ main ] - workflow_dispatch: - -concurrency: - group: ci-${{ github.workflow }}-${{ github.ref }} - cancel-in-progress: true + workflow_call: + inputs: + known-red: + description: > + Include the legs that are known red (#669). ci.yml passes true on + main, on dispatch and on a pull request labelled `macos-27` (rule R7). + type: boolean + required: false + default: true jobs: e2e: - name: e2e suite (macOS ARM64, self-host, ${{ matrix.image }}${{ matrix.known_red != '' && format(', known red {0}', matrix.known_red) || '' }}) + name: e2e suite ${{ matrix.shard }}/${{ matrix.shards }} (macOS ARM64, self-host, ${{ matrix.image }}${{ matrix.known_red != '' && format(', known red {0}', matrix.known_red) || '' }}) # The same two images as ci-macos.yml; `xcode-27` is macOS 27 (see there). # KNOWN RED on xcode-27, along with ci-macos.yml's own job: the image's # Command Line Tools SDK ships an `arm64e.x1` .tbd stub ld64.lld 22.1.8 @@ -31,11 +34,10 @@ jobs: strategy: fail-fast: false matrix: - include: - - image: macos-15 - known_red: '' - - image: xcode-27 - known_red: '#669' + # macos-15 in two shards. The known-red leg is in the matrix only when + # the caller asks for it, and as one shard: it fails at its first link + # (#669), and two shards of it would hold two of the five macOS slots. + include: ${{ fromJSON(inputs.known-red && '[{"image":"macos-15","known_red":"","shard":1,"shards":2},{"image":"macos-15","known_red":"","shard":2,"shards":2},{"image":"xcode-27","known_red":"#669","shard":1,"shards":1}]' || '[{"image":"macos-15","known_red":"","shard":1,"shards":2},{"image":"macos-15","known_red":"","shard":2,"shards":2}]') }} runs-on: ${{ matrix.image }} # KNOWN RED, MACHINE-READABLY (the 2026-09-28 design, WS7). A leg whose # failure has a tracked external cause carries that issue in `known_red`: @@ -44,6 +46,10 @@ jobs: # the issue to be open. The leg leaves the list when #669 closes. continue-on-error: ${{ matrix.known_red != '' }} timeout-minutes: 60 + env: + E2E_SHARD: ${{ matrix.shard }}/${{ matrix.shards }} + E2E_TIMINGS: tests/e2e/timings/macos.tsv + E2E_REPORT: ${{ github.workspace }}/e2e-report-${{ matrix.image == 'macos-15' && 'macos' || matrix.image }}-${{ matrix.shard }}.tsv # NOTE: no MCPP_VERBOSE — the e2e suite asserts mcpp's default quiet # output (tests 48/53). steps: @@ -59,29 +65,21 @@ jobs: with: image: ${{ matrix.image }} - - name: Build mcpp from source (self-host) - run: | - export MCPP_VENDORED_XLINGS="$XLINGS_BIN" - "$MCPP" build + - uses: ./.github/actions/use-built-mcpp + with: + host: macos-arm64 - name: E2E suite - # Fail-fast on hung tests instead of burning the whole job budget. - # Per-test 600s timeout lives in run_all.sh. - timeout-minutes: 25 + # Twice a shard's budget of about nine minutes (R4); the per-test 600 s + # limit in run_all.sh names a test that hangs. + timeout-minutes: 20 run: | - # The RELEASED mcpp this job bootstrapped from, kept for e2e 252: the - # claim that an older client can still build against a package the PR - # produces is only worth making if something checks it against a real - # old binary. Captured before $MCPP is repointed at the fresh build. - export MCPP_BOOT="$MCPP" - MCPP=$(find target -path "*/bin/mcpp" | head -1) - MCPP=$(cd "$(dirname "$MCPP")" && pwd)/$(basename "$MCPP") - test -x "$MCPP" - export MCPP - export MCPP_VENDORED_XLINGS="$XLINGS_BIN" + # MCPP is this commit's binary and MCPP_BOOT the released bootstrap + # (use-built-mcpp); e2e 252 checks an older client against a package + # this commit produces, so it needs a real old binary. + export MCPP MCPP_BOOT MCPP_VENDORED_XLINGS test -x "$MCPP_VENDORED_XLINGS" export MCPP_E2E_TOOLCHAIN_MIRROR=GLOBAL - "$MCPP" self config --mirror "$MCPP_E2E_TOOLCHAIN_MIRROR" "$MCPP" self config # macOS default toolchain is LLVM "$MCPP" toolchain default "llvm@${MCPP_LLVM_VER}" @@ -89,6 +87,19 @@ jobs: set -o pipefail bash tests/e2e/run_all.sh 2>&1 | tee "$RUNNER_TEMP/e2e-suite.log" + # One file, no glob: the report the e2e-coverage job of ci.yml reads. A + # glob upload once hung a macOS job for ten minutes, and cancelling it lost + # the job's whole log. + - name: Upload the shard's report + if: always() + timeout-minutes: 5 + uses: actions/upload-artifact@v4 + with: + name: e2e-report-${{ matrix.image == 'macos-15' && 'macos' || matrix.image }}-${{ matrix.shard }} + path: e2e-report-${{ matrix.image == 'macos-15' && 'macos' || matrix.image }}-${{ matrix.shard }}.tsv + if-no-files-found: warn + retention-days: 7 + # Measurement legs print READING lines and pass whatever they read; the # readings are what a decision is taken from (#646 F2: whether a C++ # exception thrown in a dylib is caught by its class under the payload's @@ -108,6 +119,8 @@ jobs: # NDK is a large download the suite's per-test bound does not allow for, # and the shard does not carry it (the script declares android-ndk). - name: "Android row on a macOS host (721)" + # Once per image, not once per shard: the step installs the NDK. + if: matrix.shard == 1 # ci-lint: allow-r1: one NDK install per image; the shards share it timeout-minutes: 30 shell: bash run: | diff --git a/.github/workflows/ci-macos-ios.yml b/.github/workflows/ci-macos-ios.yml index 5d32e7d1..980191e8 100644 --- a/.github/workflows/ci-macos-ios.yml +++ b/.github/workflows/ci-macos-ios.yml @@ -14,15 +14,7 @@ name: ci-macos-ios # assertion before. on: - push: - branches: [ main ] - pull_request: - branches: [ main ] - workflow_dispatch: - -concurrency: - group: ci-macos-ios-${{ github.ref }} - cancel-in-progress: true + workflow_call: jobs: # THE PREMISES, AS A PROBE THAT RUNS ON REQUEST. @@ -168,11 +160,9 @@ jobs: submodules: recursive - uses: ./.github/actions/setup-macos-llvm - - name: Build mcpp from source (self-host) - run: | - export MCPP_VENDORED_XLINGS="$XLINGS_BIN" - "$MCPP" build - echo "MCPP_DEV=$(ls -td "$PWD"/target/*/*/bin/mcpp | head -1)" >> "$GITHUB_ENV" + - uses: ./.github/actions/use-built-mcpp + with: + host: macos-arm64 # NO TOOLCHAIN IS DECLARED, AND THAT IS WHAT MAKES THIS THE USER'S PATH. # @@ -288,7 +278,7 @@ jobs: run: | set -euo pipefail cd /tmp/iostest - "$MCPP_DEV" build --target aarch64-ios + "$MCPP_FRESH" build --target aarch64-ios /tmp/assert-artefact.sh aarch64-ios arm64 2 18.0 # `platform 7` is IOSSIMULATOR. The number is what separates this row from @@ -297,7 +287,7 @@ jobs: run: | set -euo pipefail cd /tmp/iostest - "$MCPP_DEV" build --target aarch64-ios-sim 2>&1 | tee build-sim.log + "$MCPP_FRESH" build --target aarch64-ios-sim 2>&1 | tee build-sim.log /tmp/assert-artefact.sh aarch64-ios-sim arm64 7 18.0 # THE TWO LAYERS THE PACKAGES SUPPLY, READ FROM THE REPORT AND FROM THE @@ -335,14 +325,14 @@ jobs: version = "0.1.0" TOML printf 'import std;\nint main() { std::print("x\\n"); }\n' > src/main.cpp - "$MCPP_DEV" build --target aarch64-ios-sim > mixed.log 2>&1 || { echo "FAIL: import std without llvm.libcxx no longer builds"; cat mixed.log; exit 1; } + "$MCPP_FRESH" build --target aarch64-ios-sim > mixed.log 2>&1 || { echo "FAIL: import std without llvm.libcxx no longer builds"; cat mixed.log; exit 1; } # A degradation renders its `what` text, not its domain, so the # sentence is what a log can be asked for. grep -q "links the SDK's libc++ under the toolchain payload's libc++ headers" mixed.log || { echo "FAIL: no degradation named the mixed libc++"; cat mixed.log; exit 1; } grep -q 'llvm.libcxx' mixed.log || { echo "FAIL: the degradation does not name llvm.libcxx"; cat mixed.log; exit 1; } printf '#include \n#include \nint main() { std::string s = "1-2-3"; std::puts(s.c_str()); }\n' > src/main.cpp rm -rf target - "$MCPP_DEV" build --target aarch64-ios-sim 2>&1 | tee plain.log + "$MCPP_FRESH" build --target aarch64-ios-sim 2>&1 | tee plain.log grep -q 'carries no compiler runtime for aarch64-ios-sim' plain.log || { echo "FAIL: no degradation named the missing compiler runtime"; exit 1; } grep -q "links the SDK's libc++ under" plain.log && { echo "FAIL: a program without import std was reported as mixing libc++"; exit 1; } ninja=$(ls target/aarch64-ios-sim/*/build.ninja | head -1) @@ -365,7 +355,7 @@ jobs: run: | set -uo pipefail cd /tmp/iostest - out=$("$MCPP_DEV" run --target aarch64-ios-sim 2>&1) && rc=0 || rc=$? + out=$("$MCPP_FRESH" run --target aarch64-ios-sim 2>&1) && rc=0 || rc=$? printf '%s\n' "$out" | tail -20 if [ "$rc" -ne 0 ]; then echo "FAIL: mcpp run --target aarch64-ios-sim exited $rc" @@ -384,5 +374,5 @@ jobs: run: | set -euo pipefail cd /tmp/iostest - "$MCPP_DEV" build --target x86_64-ios-sim + "$MCPP_FRESH" build --target x86_64-ios-sim /tmp/assert-artefact.sh x86_64-ios-sim x86_64 7 18.0 diff --git a/.github/workflows/ci-macos.yml b/.github/workflows/ci-macos.yml index b7c0c8e6..3167e3aa 100644 --- a/.github/workflows/ci-macos.yml +++ b/.github/workflows/ci-macos.yml @@ -3,19 +3,21 @@ name: ci-macos # macOS CI for mcpp — validates LLVM/Clang as the default macOS toolchain. # Tests the full xlings → LLVM → C++23 import std pipeline on macOS ARM64. # -# The e2e suite runs in PARALLEL in ci-macos-e2e.yml (same setup via -# .github/actions/setup-macos-llvm), not tacked onto this job. +# Called by ci.yml after the macOS build (build.yml): the steps that need this +# commit's mcpp take that build (use-built-mcpp, rule R1 of the 2026-10-02 CI +# record) instead of building one. The e2e suite is ci-macos-e2e.yml, called +# beside this one with the same setup (.github/actions/setup-macos-llvm). on: - push: - branches: [ main ] - pull_request: - branches: [ main ] - workflow_dispatch: - -concurrency: - group: ci-macos-${{ github.ref }} - cancel-in-progress: true + workflow_call: + inputs: + known-red: + description: > + Include the legs that are known red (#669). ci.yml passes true on + main, on dispatch and on a pull request labelled `macos-27` (rule R7). + type: boolean + required: false + default: true jobs: macos-xlings-llvm: @@ -32,11 +34,10 @@ jobs: strategy: fail-fast: false matrix: - include: - - image: macos-15 - known_red: '' - - image: xcode-27 - known_red: '#669' + # The known-red leg is in the matrix only when the caller asks for it: + # a pull request cannot change its outcome unless it addresses #669, + # and on every other one it held a macOS slot to fail in three minutes. + include: ${{ fromJSON(inputs.known-red && '[{"image":"macos-15","known_red":""},{"image":"xcode-27","known_red":"#669"}]' || '[{"image":"macos-15","known_red":""}]') }} runs-on: ${{ matrix.image }} # KNOWN RED, MACHINE-READABLY (the 2026-09-28 design, WS7). A leg whose # failure has a tracked external cause carries that issue in `known_red`: @@ -292,25 +293,19 @@ jobs: *) echo "FAIL: unexpected platform"; exit 1 ;; esac - - name: Build mcpp from source (self-host) - run: | - export MCPP_VENDORED_XLINGS="$XLINGS_BIN" - "$MCPP" build + - uses: ./.github/actions/use-built-mcpp + with: + host: macos-arm64 - name: Unit + integration tests via `mcpp test` run: | - # Use freshly-built mcpp (has --mirror support) - MCPP=$(find target -path "*/bin/mcpp" | head -1) - MCPP=$(cd "$(dirname "$MCPP")" && pwd)/$(basename "$MCPP") - "$MCPP" self config --mirror GLOBAL - "$MCPP" test + "$MCPP_FRESH" test # WS8: this host's row of docs/01 and docs/20 is checked against the # one answer the resolver gives (`self env --format json`). - name: The documented default toolchain is this host's answer run: | - MCPP=$(find target -path "*/bin/mcpp" | head -1) - python3 .github/tools/check_default_toolchain_docs.py --mcpp "$MCPP" + python3 .github/tools/check_default_toolchain_docs.py --mcpp "$MCPP_FRESH" - name: Forensics — test-binary link + load state (on failure) if: failure() @@ -359,17 +354,13 @@ jobs: - name: "Toolchain: LLVM — build mcpp (self-host)" run: | - MCPP=$(find target -path "*/bin/mcpp" | head -1) - MCPP=$(cd "$(dirname "$MCPP")" && pwd)/$(basename "$MCPP") - test -x "$MCPP" - cp "$MCPP" /tmp/mcpp-fresh + cp "$MCPP_FRESH" /tmp/mcpp-fresh MCPP=/tmp/mcpp-fresh "$MCPP" toolchain default "llvm@${MCPP_LLVM_VER}" "$MCPP" clean "$MCPP" build "$MCPP" --version - # GRAPHICS ON THIS HOST, BUILD ONLY, AND THAT IS THE WHOLE CLAIM. # # This runner has no Vulkan device, so what is asserted is what this diff --git a/.github/workflows/ci-target-matrix.yml b/.github/workflows/ci-target-matrix.yml index 62f22b9b..ad10d2bb 100644 --- a/.github/workflows/ci-target-matrix.yml +++ b/.github/workflows/ci-target-matrix.yml @@ -6,14 +6,7 @@ name: target matrix # 装了某个载荷」而通过,或因为没装而跳过,而两者在退出码上与「全部正确」没有区别。 # 三个宿主各扫一遍,把结果与仓库里的期望表比对,差异即失败。 on: - pull_request: - push: - branches: [ main ] - workflow_dispatch: - -concurrency: - group: ci-${{ github.workflow }}-${{ github.ref }} - cancel-in-progress: true + workflow_call: env: XLINGS_NON_INTERACTIVE: '1' @@ -50,7 +43,22 @@ jobs: - uses: actions/checkout@v4 - uses: ./.github/actions/bootstrap-mcpp + # This commit's mcpp: the one build.yml produced (use-built-mcpp, rule R1 + # of the 2026-10-02 CI record), except on the macOS leg. That leg runs on + # macos-14 and the macOS artifact is built on macos-15; whether it runs on + # macos-14 is not measured (the record's Part VII), so the leg builds its + # own until it is. + - if: matrix.host != 'macos-arm64' + uses: ./.github/actions/use-built-mcpp + with: + host: ${{ matrix.host }} + + - name: Name this commit's mcpp MCPP_UNDER_TEST + if: matrix.host != 'macos-arm64' + run: echo "MCPP_UNDER_TEST=$MCPP_FRESH" >> "$GITHUB_ENV" + - name: Build the mcpp in this pull request + if: matrix.host == 'macos-arm64' # ci-lint: allow-r1: the macOS leg builds its own mcpp until the artifact is measured on macos-14 run: | set -euo pipefail export MCPP_VENDORED_XLINGS="$XLINGS_BIN" @@ -258,7 +266,22 @@ jobs: - uses: actions/checkout@v4 - uses: ./.github/actions/bootstrap-mcpp + # This commit's mcpp: the one build.yml produced (use-built-mcpp, rule R1 + # of the 2026-10-02 CI record), except on the macOS leg. That leg runs on + # macos-14 and the macOS artifact is built on macos-15; whether it runs on + # macos-14 is not measured (the record's Part VII), so the leg builds its + # own until it is. + - if: matrix.host != 'macos-arm64' + uses: ./.github/actions/use-built-mcpp + with: + host: ${{ matrix.host }} + + - name: Name this commit's mcpp MCPP_UNDER_TEST + if: matrix.host != 'macos-arm64' + run: echo "MCPP_UNDER_TEST=$MCPP_FRESH" >> "$GITHUB_ENV" + - name: Build the mcpp in this pull request + if: matrix.host == 'macos-arm64' # ci-lint: allow-r1: the macOS leg builds its own mcpp until the artifact is measured on macos-14 run: | set -euo pipefail export MCPP_VENDORED_XLINGS="$XLINGS_BIN" diff --git a/.github/workflows/ci-windows-e2e.yml b/.github/workflows/ci-windows-e2e.yml index da100042..592e5543 100644 --- a/.github/workflows/ci-windows-e2e.yml +++ b/.github/workflows/ci-windows-e2e.yml @@ -1,37 +1,33 @@ name: ci-windows-e2e -# The e2e suite on Windows, split out of ci-windows.yml (where it was 9.7 of -# the job's 20.4 min) and sharded across two runners — same shape as -# ci-linux-e2e.yml. Shares the cache lineage in .github/actions/bootstrap-mcpp, -# so each shard restores a warm sandbox and pays one incremental `mcpp build`. +# The e2e suite on Windows x64, called by ci.yml beside ci-windows.yml after +# the Windows build. Three shards, assigned by measured duration +# (tests/e2e/timings/windows.tsv, rule R4 of the 2026-10-02 CI record): about +# fifteen tests of over thirty seconds took 58 to 64 percent of a round-robin +# shard, and the two shards differed by 3.6 minutes on average. Each shard runs +# the one binary build.yml produced (use-built-mcpp, rule R1) and writes the +# per-test report the e2e-coverage job of ci.yml reads. # -# Paired workflows: ci-windows.yml (build + unit + package, toolchains + -# regressions), ci-linux-e2e.yml, ci-macos-e2e.yml. +# Paired workflows: ci-windows.yml, ci-linux-e2e.yml, ci-macos-e2e.yml. on: - push: - branches: [ main ] - pull_request: - branches: [ main ] - workflow_dispatch: - -concurrency: - group: ci-${{ github.workflow }}-${{ github.ref }} - cancel-in-progress: true + workflow_call: jobs: e2e: - name: e2e ${{ matrix.shard }}/2 (windows x64, self-host) + name: e2e ${{ matrix.shard }}/3 (windows x64, self-host) runs-on: windows-latest timeout-minutes: 45 strategy: fail-fast: false matrix: - shard: [1, 2] + shard: [1, 2, 3] env: MCPP_HOME: C:\Users\runneradmin\.mcpp # Round-robin slice of tests/e2e (see run_all.sh). - E2E_SHARD: ${{ matrix.shard }}/2 + E2E_SHARD: ${{ matrix.shard }}/3 + E2E_TIMINGS: tests/e2e/timings/windows.tsv + E2E_REPORT: ${{ github.workspace }}/e2e-report-windows-${{ matrix.shard }}.tsv # NOTE: do NOT force MCPP_VERBOSE here. The e2e suite includes tests that # assert mcpp's DEFAULT (quiet) output — e.g. 48_build_error_output and # 53_namespaced_cache_label — which forced verbose would break. @@ -46,23 +42,15 @@ jobs: submodules: recursive - uses: ./.github/actions/bootstrap-mcpp - - name: Build mcpp from source (self-host) + - uses: ./.github/actions/use-built-mcpp + with: + host: windows-x86_64 + + - name: Name this commit's mcpp MCPP_SELF shell: bash run: | - export MCPP_VENDORED_XLINGS="$XLINGS_BIN" - "$MCPP" build - # Pick the NEWEST mcpp.exe, not an arbitrary one: `target/` is - # restored from cache and keeps a directory per build fingerprint, - # so after a version bump the freshly built binary sits alongside - # the previous release's. `find | head -1` returned whichever the - # directory walk hit first — which is how a 0.0.106 build ran the - # 0.0.105 binary and failed 01_help_and_version. - MCPP_SELF=$(find target -name "mcpp.exe" -path "*/bin/*" -printf "%T@ %p\n" \ - | sort -rn | head -1 | cut -d" " -f2-) - test -n "$MCPP_SELF" || { echo "FAIL: no mcpp.exe"; exit 1; } - MCPP_SELF=$(cd "$(dirname "$MCPP_SELF")" && pwd)/$(basename "$MCPP_SELF") - "$MCPP_SELF" --version - echo "MCPP_SELF=$MCPP_SELF" >> "$GITHUB_ENV" + "$MCPP_FRESH" --version + echo "MCPP_SELF=$MCPP_FRESH" >> "$GITHUB_ENV" # MinGW-w64 GCC (xim:mingw-gcc). Installed here so the `mingw` capability # is GRANTED rather than left to whatever the shared sandbox cache happens @@ -78,30 +66,43 @@ jobs: - name: "Toolchain: MinGW payload for the fat-package e2e" shell: bash run: | + # The build job's prewarm installs `mingw 16.1.0` on `main` and saves + # it into the sandbox cache, so every PR restores a sandbox that + # already holds it. The install call is then a no-op but still pays + # the mcpp-toolchain startup cost; measured 68 s on three shards per + # PR. Skip the call when the payload is already reachable, and keep + # the verification that names the missing-capability case (the whole + # reason this step exists is so a skipped shard does not read like a + # passing one). export MCPP_VENDORED_XLINGS="$XLINGS_BIN" - "$MCPP_SELF" toolchain install mingw 16.1.0 - # Verified through the SAME two locations run_all.sh probes — checking - # only one of them would let the step pass while the capability stays - # ungranted, which is the shape of a green run that tested nothing. found="" for c in "${MCPP_HOME:-$HOME/.mcpp}"/registry/data/xpkgs/xim-x-mingw-gcc/*/bin/g++.exe \ "$HOME"/.xlings/data/xpkgs/xim-x-mingw-gcc/*/bin/g++.exe; do [[ -x "$c" ]] && { found="$c"; break; } done + if [ -z "$found" ]; then + "$MCPP_SELF" toolchain install mingw 16.1.0 + for c in "${MCPP_HOME:-$HOME/.mcpp}"/registry/data/xpkgs/xim-x-mingw-gcc/*/bin/g++.exe \ + "$HOME"/.xlings/data/xpkgs/xim-x-mingw-gcc/*/bin/g++.exe; do + [[ -x "$c" ]] && { found="$c"; break; } + done + fi + # Verified through the SAME two locations run_all.sh probes — checking + # only one of them would let the step pass while the capability stays + # ungranted, which is the shape of a green run that tested nothing. test -n "$found" || { echo "FAIL: mingw installed but not where run_all.sh looks"; exit 1; } echo "mingw payload: $found" - name: E2E suite shell: bash - # Fail-fast on hung tests instead of burning the whole job budget. - # Per-test 600s timeout lives in run_all.sh. - timeout-minutes: 25 + # About twice the shard's budget of fifteen minutes (R4): reached only by + # a hang. The per-test 600 s limit in run_all.sh names the test that hung. + timeout-minutes: 30 run: | - # The RELEASED mcpp this job bootstrapped from, kept for e2e 252: the - # claim that an older client can still build against a package the PR - # produces is only worth making if something checks it against a real - # old binary. Captured before $MCPP is repointed at the fresh build. - export MCPP_BOOT="${MCPP:-$MCPP_BOOT}" + # MCPP_BOOT is the released bootstrap (use-built-mcpp). e2e 252 checks + # an older client against a package this commit produces, so it needs + # a real old binary. + export MCPP_BOOT export MCPP="$MCPP_SELF" export MCPP_VENDORED_XLINGS="$XLINGS_BIN" export MCPP_E2E_TOOLCHAIN_MIRROR=GLOBAL @@ -112,6 +113,17 @@ jobs: # Measurement legs print READING lines (#646 F2 across PE images, #649 E10 # the llvm row's recorded CRT); collected where a reader finds them. + # One file, no glob: the report the e2e-coverage job of ci.yml reads. + - name: Upload the shard's report + if: always() + timeout-minutes: 5 + uses: actions/upload-artifact@v4 + with: + name: e2e-report-windows-${{ matrix.shard }} + path: e2e-report-windows-${{ matrix.shard }}.tsv + if-no-files-found: warn + retention-days: 7 + - name: Measurement readings if: always() shell: bash diff --git a/.github/workflows/ci-windows-msvc-xlings.yml b/.github/workflows/ci-windows-msvc-xlings.yml index 1ed1ff1f..f8b50da4 100644 --- a/.github/workflows/ci-windows-msvc-xlings.yml +++ b/.github/workflows/ci-windows-msvc-xlings.yml @@ -31,15 +31,7 @@ name: ci-windows-msvc-xlings # Paired workflows: ci-windows.yml, ci-windows-e2e.yml. on: - push: - branches: [ main ] - pull_request: - branches: [ main ] - workflow_dispatch: - -concurrency: - group: ci-${{ github.workflow }}-${{ github.ref }} - cancel-in-progress: true + workflow_call: jobs: xlings-msvc: @@ -52,21 +44,15 @@ jobs: - uses: actions/checkout@v4 - uses: ./.github/actions/bootstrap-mcpp - - name: Build mcpp from source (self-host) + - uses: ./.github/actions/use-built-mcpp + with: + host: windows-x86_64 + + - name: Name this commit's mcpp MCPP_SELF shell: bash run: | - export MCPP_VENDORED_XLINGS="$XLINGS_BIN" - "$MCPP" build - # Newest mcpp.exe, not an arbitrary one — `target/` is restored from - # cache and keeps a directory per build fingerprint. Same reasoning - # as ci-windows-e2e.yml, where picking wrong ran the previous - # release's binary. - MCPP_SELF=$(find target -name "mcpp.exe" -path "*/bin/*" -printf "%T@ %p\n" \ - | sort -rn | head -1 | cut -d" " -f2-) - test -n "$MCPP_SELF" || { echo "FAIL: no mcpp.exe"; exit 1; } - MCPP_SELF=$(cd "$(dirname "$MCPP_SELF")" && pwd)/$(basename "$MCPP_SELF") - "$MCPP_SELF" --version - echo "MCPP_SELF=$MCPP_SELF" >> "$GITHUB_ENV" + "$MCPP_FRESH" --version + echo "MCPP_SELF=$MCPP_FRESH" >> "$GITHUB_ENV" - name: xlings-managed msvc e2e shell: bash diff --git a/.github/workflows/ci-windows.yml b/.github/workflows/ci-windows.yml index e4fc311e..9122ebb3 100644 --- a/.github/workflows/ci-windows.yml +++ b/.github/workflows/ci-windows.yml @@ -3,55 +3,37 @@ name: ci-windows # Windows CI for mcpp — same flow as Linux (ci-linux.yml) and macOS (ci-macos.yml): # xlings install mcpp → self-host build → smoke → package # -# SHAPE: three INDEPENDENT jobs, no `needs:` between them; each restores the -# shared cache lineage (.github/actions/bootstrap-mcpp) and pays one warm -# `mcpp build` to get the PR's own binary. The e2e suite moved to -# ci-windows-e2e.yml (sharded ×2) — it was 9.7 of this job's 20.4 min. -# -# before: build → unit → xlings → stdin → e2e → toolchains → package ≈ 20 min -# after: max(build+unit+package, toolchains+integration) ≈ 8 min -# in parallel with ci-windows-e2e (≈ 8 min) +# SHAPE: called by ci.yml after the Windows build (build.yml). No job here +# builds mcpp to obtain the commit's binary: each takes the one binary that +# build produced (use-built-mcpp, rule R1 of the 2026-10-02 CI record) and names +# it MCPP_SELF. A job builds mcpp only when the build is what it tests (the LLVM +# rebuild of `toolchains`). The e2e suite is ci-windows-e2e.yml, called beside +# this one. on: - push: - branches: [ main ] - pull_request: - branches: [ main ] - workflow_dispatch: - -concurrency: - group: ci-windows-${{ github.ref }} - cancel-in-progress: true + workflow_call: env: MCPP_HOME: C:\Users\runneradmin\.mcpp jobs: build-test: - name: build + test + package (windows x64, self-host) + name: test + package (windows x64, self-host) runs-on: windows-latest timeout-minutes: 45 steps: - uses: actions/checkout@v4 - uses: ./.github/actions/bootstrap-mcpp - - name: Build mcpp from source (self-host) + - uses: ./.github/actions/use-built-mcpp + with: + host: windows-x86_64 + + - name: Name this commit's mcpp MCPP_SELF shell: bash run: | - export MCPP_VENDORED_XLINGS="$XLINGS_BIN" - "$MCPP" build - # Pick the NEWEST mcpp.exe, not an arbitrary one: `target/` is - # restored from cache and keeps a directory per build fingerprint, - # so after a version bump the freshly built binary sits alongside - # the previous release's. `find | head -1` returned whichever the - # directory walk hit first — which is how a 0.0.106 build ran the - # 0.0.105 binary and failed 01_help_and_version. - MCPP_SELF=$(find target -name "mcpp.exe" -path "*/bin/*" -printf "%T@ %p\n" \ - | sort -rn | head -1 | cut -d" " -f2-) - test -n "$MCPP_SELF" || { echo "FAIL: no mcpp.exe"; exit 1; } - MCPP_SELF=$(cd "$(dirname "$MCPP_SELF")" && pwd)/$(basename "$MCPP_SELF") - "$MCPP_SELF" --version - echo "MCPP_SELF=$MCPP_SELF" >> "$GITHUB_ENV" + "$MCPP_FRESH" --version + echo "MCPP_SELF=$MCPP_FRESH" >> "$GITHUB_ENV" - name: Unit + integration tests via mcpp test shell: bash @@ -72,15 +54,11 @@ jobs: VERSION=$(awk -F '"' '/^version[[:space:]]*=/{print $2; exit}' mcpp.toml) WRAPPER="mcpp-${VERSION}-windows-x86_64" ZIPNAME="${WRAPPER}.zip" - # Pick the NEWEST mcpp.exe, not an arbitrary one: `target/` is - # restored from cache and keeps a directory per build fingerprint, - # so after a version bump the freshly built binary sits alongside - # the previous release's. `find | head -1` returned whichever the - # directory walk hit first — which is how a 0.0.106 build ran the - # 0.0.105 binary and failed 01_help_and_version. - MCPP_BIN=$(find target -name "mcpp.exe" -path "*/bin/*" -printf "%T@ %p\n" \ - | sort -rn | head -1 | cut -d" " -f2-) - test -n "$MCPP_BIN" || { echo "FAIL: no mcpp.exe in target/"; exit 1; } + # The binary of build.yml (use-built-mcpp), not one found under + # target/: nothing restores target/ any more, and this is the binary + # every other job of the commit tests. + MCPP_BIN="$MCPP_SELF" + test -f "$MCPP_BIN" || { echo "FAIL: no mcpp.exe at $MCPP_BIN"; exit 1; } STAGING=$(mktemp -d) mkdir -p "$STAGING/$WRAPPER/bin" "$STAGING/$WRAPPER/registry/bin" @@ -239,22 +217,15 @@ jobs: - uses: actions/checkout@v4 - uses: ./.github/actions/bootstrap-mcpp - - name: Build mcpp from source (self-host) + - uses: ./.github/actions/use-built-mcpp + with: + host: windows-x86_64 + + - name: Name this commit's mcpp MCPP_SELF shell: bash run: | - export MCPP_VENDORED_XLINGS="$XLINGS_BIN" - "$MCPP" build - # Pick the NEWEST mcpp.exe, not an arbitrary one: `target/` is - # restored from cache and keeps a directory per build fingerprint, - # so after a version bump the freshly built binary sits alongside - # the previous release's. `find | head -1` returned whichever the - # directory walk hit first — which is how a 0.0.106 build ran the - # 0.0.105 binary and failed 01_help_and_version. - MCPP_SELF=$(find target -name "mcpp.exe" -path "*/bin/*" -printf "%T@ %p\n" \ - | sort -rn | head -1 | cut -d" " -f2-) - test -n "$MCPP_SELF" || { echo "FAIL: no mcpp.exe"; exit 1; } - MCPP_SELF=$(cd "$(dirname "$MCPP_SELF")" && pwd)/$(basename "$MCPP_SELF") - echo "MCPP_SELF=$MCPP_SELF" >> "$GITHUB_ENV" + "$MCPP_FRESH" --version + echo "MCPP_SELF=$MCPP_FRESH" >> "$GITHUB_ENV" # Integration: the mcpp built from THIS PR's source ($MCPP_SELF, the # self-hosted binary) builds & runs a real external C++ project — xlings @@ -430,7 +401,6 @@ jobs: export MCPP_VENDORED_XLINGS="$XLINGS_BIN" MCPP="$MCPP_SELF" bash .github/tools/check_unicode_paths.sh - # GRAPHICS ON THIS HOST, BUILD ONLY, AND THAT IS THE WHOLE CLAIM. # # This runner has no Vulkan device, so what is asserted is what this diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml new file mode 100644 index 00000000..77f622f2 --- /dev/null +++ b/.github/workflows/ci.yml @@ -0,0 +1,266 @@ +name: ci + +# THE CI OF ONE COMMIT, IN STAGES. The design and its measurements are in +# .agents/docs/2026-10-02-pr-ci-acceleration-and-the-toolchain-specification-design.md +# (Part II); the rules it cites are R1-R7 there. +# +# changes what the commit changed decides what runs (R2). A change of +# documentation that nothing reads runs `docs` and nothing else. +# docs the checks that need no binary, on every change. +# build-* this commit's mcpp, built ONCE per host (R1) by build.yml and +# uploaded as `mcpp-built-`. +# the rest the per-area workflows, called as reusable workflows. Each waits +# for its own host's build only, takes the binary with +# .github/actions/use-built-mcpp, and keeps the job names it had as +# a workflow of its own. +# e2e-coverage every e2e test ran somewhere or says why not (R5). +# +# Measured before this shape (2026-10-01, seven commits): 35 to 47 jobs started +# at once against the organisation's 20 slots, 30 to 37 of them built mcpp +# from source, and a commit waited 36 to 50 minutes, 9 to 13 of them in the +# queue. + +on: + push: + branches: [ main ] + pull_request: + branches: [ main ] + workflow_dispatch: + +# A superseded pull-request run is cancelled. Every push to main runs to the +# end, in a group of its own commit: its build jobs are the only writers of the +# caches (R3), and a cancelled run saves nothing. A group shared by all pushes +# to main would still lose runs, because GitHub keeps one pending run per group +# and cancels the older pending one when a third arrives. +concurrency: + group: ci-${{ github.event_name == 'push' && github.sha || github.ref }} + cancel-in-progress: ${{ github.event_name == 'pull_request' }} + +permissions: + contents: read + # `changes` lists the files of the pull request. + pull-requests: read + +jobs: + changes: + name: what the change starts + runs-on: ubuntu-24.04 + timeout-minutes: 5 + outputs: + code: ${{ steps.classify.outputs.code }} + steps: + - uses: actions/checkout@v4 + + - name: Classify the changed paths + id: classify + env: + GH_TOKEN: ${{ github.token }} + EVENT: ${{ github.event_name }} + REPO: ${{ github.repository }} + PR: ${{ github.event.pull_request.number }} + BEFORE: ${{ github.event.before }} + SHA: ${{ github.sha }} + run: | + set -euo pipefail + full() { echo "code=true" >> "$GITHUB_OUTPUT"; echo "code=true ($1)"; } + # Anything this step cannot list runs the whole CI: an API error, a + # force-push whose previous commit is gone, an event without a diff. + case "$EVENT" in + pull_request) + gh api --paginate "repos/$REPO/pulls/$PR/files" \ + --jq '.[] | .filename, (.previous_filename // empty)' > changed.txt \ + || { full "the files of the pull request could not be listed"; exit 0; } ;; + push) + if [ -z "$BEFORE" ] || [ "$BEFORE" = 0000000000000000000000000000000000000000 ]; then + full "a push with no previous commit"; exit 0 + fi + gh api "repos/$REPO/compare/$BEFORE...$SHA" > compare.json \ + || { full "the push could not be compared with its previous commit"; exit 0; } + # The compare API lists at most 300 files; a longer list is + # treated as a change of everything. + if [ "$(jq '.files | length' compare.json)" -ge 300 ]; then + full "300 or more changed files"; exit 0 + fi + jq -r '.files[] | .filename, (.previous_filename // empty)' compare.json > changed.txt ;; + *) + full "event $EVENT"; exit 0 ;; + esac + python3 .github/tools/classify_changes.py --github-output "$GITHUB_OUTPUT" < changed.txt + + docs: + name: documentation and repository checks + runs-on: ubuntu-24.04 + timeout-minutes: 15 + steps: + - uses: actions/checkout@v4 + + - name: Check version / xlings pin consistency + run: bash .github/tools/check_version_pins.sh + + - name: Check modules/ wiring + run: bash .github/tools/check_modules_wiring.sh + + - name: Check src/build/prepare* file lengths + run: bash .github/tools/check_file_lengths.sh + + - name: Check no walk-derived path is narrowed directly + run: bash .github/tools/check_narrow_conversions.sh + + - name: Where the CI assertions live + run: bash tools/lint-ci-assertions.sh + + - name: Steps assert what their names say + env: + GH_TOKEN: ${{ github.token }} + run: | + python3 tests/scripts/test_check_workflow_assertions.py + python3 .github/tools/check_workflow_assertions.py --check-open + + - name: The release canary runner runs each command under the named bash + run: python3 tests/scripts/test_release_canaries.py + + - name: The protocol table of SPEC-007 names the engine's protocol + run: python3 tests/scripts/test_protocol_table.py + + - name: The target matrix names every refusal + run: bash .github/tools/check_matrix_reasons.sh + + - name: Check documentation style and bilingual parity + run: bash .github/tools/check_docs_style.sh + + - name: Check documentation structure + run: bash .github/tools/check_docs_structure.sh + + - name: Documented target tiers agree with the table + run: python3 .github/tools/check_target_tiers.py + + - name: Reason tokens agree with the engine, and with the mirror + run: bash .github/tools/check_reason_tokens.sh + + - name: The tools of the CI stages pass their fixtures + run: | + python3 tests/scripts/test_classify_changes.py + python3 tests/scripts/test_check_e2e_coverage.py + python3 tests/scripts/test_check_default_toolchain_docs.py + + build-linux: + needs: changes + if: needs.changes.outputs.code == 'true' + uses: ./.github/workflows/build.yml + with: + host: linux-x86_64 + runs-on: ubuntu-24.04 + # What the Linux consumers install beyond the build's own toolchain: the + # e2e shards (musl, llvm, mingw-cross), the toolchain legs (musl, llvm). + prewarm: gcc 16.1.0-musl; llvm 22.1.8; mingw-cross 16.1.0 + + build-linux-arm: + needs: changes + if: needs.changes.outputs.code == 'true' + uses: ./.github/workflows/build.yml + with: + host: linux-aarch64 + runs-on: ubuntu-24.04-arm + + build-macos: + needs: changes + if: needs.changes.outputs.code == 'true' + uses: ./.github/workflows/build.yml + with: + host: macos-arm64 + runs-on: macos-15 + + build-windows: + needs: changes + if: needs.changes.outputs.code == 'true' + uses: ./.github/workflows/build.yml + with: + host: windows-x86_64 + runs-on: windows-latest + # The MinGW payload every Windows e2e shard installs. + prewarm: mingw 16.1.0 + + linux: + needs: build-linux + uses: ./.github/workflows/ci-linux.yml + + linux-e2e: + needs: build-linux + uses: ./.github/workflows/ci-linux-e2e.yml + + macos: + needs: build-macos + uses: ./.github/workflows/ci-macos.yml + with: + # R7: the known-red legs (#669) run where they can change a decision: on + # main, on dispatch, and on a pull request labelled `macos-27`. The label + # is read when the pull request is pushed to; adding it starts no run, + # because a run on every label of every pull request would cost a whole + # CI each time. + known-red: ${{ github.event_name != 'pull_request' || contains(github.event.pull_request.labels.*.name, 'macos-27') }} + + macos-e2e: + needs: build-macos + uses: ./.github/workflows/ci-macos-e2e.yml + with: + known-red: ${{ github.event_name != 'pull_request' || contains(github.event.pull_request.labels.*.name, 'macos-27') }} + + macos-ios: + needs: build-macos + uses: ./.github/workflows/ci-macos-ios.yml + + windows: + needs: build-windows + uses: ./.github/workflows/ci-windows.yml + + windows-e2e: + needs: build-windows + uses: ./.github/workflows/ci-windows-e2e.yml + + windows-msvc-xlings: + needs: build-windows + uses: ./.github/workflows/ci-windows-msvc-xlings.yml + + cross: + needs: [build-linux, build-windows] + uses: ./.github/workflows/cross-build-test.yml + + target-matrix: + needs: [build-linux, build-linux-arm, build-macos, build-windows] + uses: ./.github/workflows/ci-target-matrix.yml + + # Pull requests and dispatch only, as when it was a workflow of its own. + openkal: + needs: [build-linux, build-macos, build-windows] + if: github.event_name != 'push' + uses: ./.github/workflows/openkal-cross.yml + + e2e-coverage: + name: every e2e test runs somewhere + needs: [changes, linux-e2e, windows-e2e, macos-e2e] + if: ${{ always() && needs.changes.outputs.code == 'true' && !cancelled() }} + runs-on: ubuntu-24.04 + timeout-minutes: 10 + steps: + - uses: actions/checkout@v4 + + - name: Fetch the shards' reports + uses: actions/download-artifact@v4 + with: + pattern: e2e-report-* + path: reports + merge-multiple: true + + - name: Every test ran, is run by a dedicated job, or is excused + run: python3 .github/tools/check_e2e_coverage.py --reports reports --timings-out timings + + # The measured durations, merged per host, in the format of + # tests/e2e/timings/; refreshing those tables is copying these files. + - name: Upload the measured durations + if: always() + uses: actions/upload-artifact@v4 + with: + name: e2e-timings + path: timings/ + if-no-files-found: ignore + retention-days: 14 diff --git a/.github/workflows/cross-build-test.yml b/.github/workflows/cross-build-test.yml index 24e26bd2..ca61fc70 100644 --- a/.github/workflows/cross-build-test.yml +++ b/.github/workflows/cross-build-test.yml @@ -52,15 +52,7 @@ name: cross-build-test # xlings-res + xim-pkgindex. on: - push: - branches: [ main ] - pull_request: - branches: [ main ] - workflow_dispatch: - -concurrency: - group: ci-${{ github.workflow }}-${{ github.ref }} - cancel-in-progress: true + workflow_call: jobs: cross-build: @@ -81,21 +73,25 @@ jobs: steps: - uses: actions/checkout@v4 - - name: Cache mcpp sandbox - uses: actions/cache@v4 + # Restored here and saved by this job alone, on main, after its last + # step (rule R3 of the 2026-10-02 CI record). + - name: Restore mcpp sandbox + id: sandbox + uses: actions/cache/restore@v4 with: path: ~/.mcpp - key: mcpp-sandbox-${{ runner.os }}-cross-${{ matrix.target }}-${{ hashFiles('mcpp.toml', '.xlings.json') }} + key: mcpp-sandbox-${{ runner.os }}-cross-${{ matrix.target }}-${{ hashFiles('mcpp.toml', '.xlings.json', '.github/workflows/cross-build-test.yml') }} restore-keys: | mcpp-sandbox-${{ runner.os }}-cross-${{ matrix.target }}- - - name: Cache xlings - uses: actions/cache@v4 + - name: Restore xlings + id: xlings + uses: actions/cache/restore@v4 with: path: ~/.xlings - key: xlings-${{ runner.os }}-v2-${{ hashFiles('.xlings.json') }} + key: xlings-${{ runner.os }}-cross-v3-${{ hashFiles('.xlings.json') }} restore-keys: | - xlings-${{ runner.os }}-v2- + xlings-${{ runner.os }}-cross-v3- - name: Install qemu-user-static run: | @@ -161,16 +157,9 @@ jobs: echo "XLINGS_BIN=$HOME/.xlings/subos/default/bin/xlings" >> "$GITHUB_ENV" echo "MCPP_BOOT=$MCPP_BOOT" >> "$GITHUB_ENV" - - name: Self-host build (bootstrap mcpp -> fresh host mcpp) - run: | - export MCPP_VENDORED_XLINGS="$XLINGS_BIN" - "$XLINGS_BIN" config --mirror GLOBAL 2>/dev/null || true - "$MCPP_BOOT" self config --mirror GLOBAL 2>/dev/null || true - "$MCPP_BOOT" build - MCPP=$(realpath "$(find target -type f -name mcpp -printf '%T@ %p\n' | sort -rn | head -1 | cut -d' ' -f2)") - test -x "$MCPP" - "$MCPP" self config --mirror GLOBAL - echo "MCPP=$MCPP" >> "$GITHUB_ENV" + - uses: ./.github/actions/use-built-mcpp + with: + host: linux-x86_64 - name: "Cross-build mcpp -> ${{ matrix.target }}" run: | @@ -216,6 +205,20 @@ jobs: xver=$($RUN "$XLINGS_XBIN" --version 2>&1 || true) echo "$xver" + - name: Save mcpp sandbox + if: ${{ github.event_name == 'push' && steps.sandbox.outputs.cache-hit != 'true' }} + uses: actions/cache/save@v4 + with: + path: ~/.mcpp + key: ${{ steps.sandbox.outputs.cache-primary-key }} + + - name: Save xlings + if: ${{ github.event_name == 'push' && steps.xlings.outputs.cache-hit != 'true' }} + uses: actions/cache/save@v4 + with: + path: ~/.xlings + key: ${{ steps.xlings.outputs.cache-primary-key }} + # ── Linux → Windows MinGW cross (OS-cross, same arch: ELF→PE) ───────────── # Builds a demo project for x86_64-w64-mingw32 with the from-source GCC-16 # MSVCRT cross toolchain, asserts the artefact is a fully-static PE, and runs @@ -232,32 +235,40 @@ jobs: steps: - uses: actions/checkout@v4 - - name: Cache mcpp sandbox - uses: actions/cache@v4 + # Restored here and saved by this job alone, on main, after its last + # step (rule R3). The Wine packages were evicted with everything else + # while the repository's caches exceeded their limit, and `apt` then + # took 26 to 50 minutes (2026-10-01); with one writer per key they stay. + - name: Restore mcpp sandbox + id: sandbox + uses: actions/cache/restore@v4 with: path: ~/.mcpp - key: mcpp-sandbox-${{ runner.os }}-mingw-cross-${{ hashFiles('mcpp.toml', '.xlings.json') }} + key: mcpp-sandbox-${{ runner.os }}-mingw-cross-${{ hashFiles('mcpp.toml', '.xlings.json', '.github/workflows/cross-build-test.yml') }} restore-keys: | mcpp-sandbox-${{ runner.os }}-mingw-cross- - - name: Cache xlings - uses: actions/cache@v4 + - name: Restore xlings + id: xlings + uses: actions/cache/restore@v4 with: path: ~/.xlings - key: xlings-${{ runner.os }}-v2-${{ hashFiles('.xlings.json') }} + key: xlings-${{ runner.os }}-mingw-cross-v3-${{ hashFiles('.xlings.json') }} restore-keys: | - xlings-${{ runner.os }}-v2- + xlings-${{ runner.os }}-mingw-cross-v3- # wine 的包集固定不变 —— 缓存整个 .deb 依赖闭包,命中时跳过 apt update # 与下载(每轮省 ~1-2min)。镜像月度更新可能改变依赖缺口,dpkg -i 失败时 # 由 apt-get -f 兜底并重新回填缓存。 - - name: Cache wine debs - uses: actions/cache@v4 + - name: Restore wine debs + id: wine + uses: actions/cache/restore@v4 with: path: ~/wine-debs key: wine-debs-${{ runner.os }}-ubuntu24.04-v1 - name: Install wine + timeout-minutes: 20 run: | sudo dpkg --add-architecture i386 || true if ls ~/wine-debs/*.deb >/dev/null 2>&1; then @@ -311,15 +322,9 @@ jobs: echo "XLINGS_BIN=$HOME/.xlings/subos/default/bin/xlings" >> "$GITHUB_ENV" echo "MCPP_BOOT=$MCPP_BOOT" >> "$GITHUB_ENV" - - name: Self-host build (fresh host mcpp) - run: | - export MCPP_VENDORED_XLINGS="$XLINGS_BIN" - "$MCPP_BOOT" self config --mirror GLOBAL 2>/dev/null || true - "$MCPP_BOOT" build - MCPP=$(realpath "$(find target -type f -name mcpp -printf '%T@ %p\n' | sort -rn | head -1 | cut -d' ' -f2)") - test -x "$MCPP" - "$MCPP" self config --mirror GLOBAL - echo "MCPP=$MCPP" >> "$GITHUB_ENV" + - uses: ./.github/actions/use-built-mcpp + with: + host: linux-x86_64 - name: Install mingw-cross toolchain run: | @@ -402,6 +407,40 @@ jobs: grep -qxF "$line" "$log" || { echo "::error::797 did not print: $line"; exit 1; } done + # 257 needs a Linux-hosted MinGW compiler AND wine, which only this job + # has; on the shards it skipped, and it had run on no runner (finding F9 + # of the 2026-10-02 CI record). + - name: "e2e: a shared library across the PE boundary (257)" + run: | + set -o pipefail + export MCPP_VENDORED_XLINGS="$XLINGS_BIN" + log="$RUNNER_TEMP/257.log" + bash tests/e2e/257_shared_library_pe.sh 2>&1 | tee "$log" + if grep -q '^SKIP' "$log"; then + echo "::error::257 skipped on the job that can run it"; exit 1 + fi + + - name: Save mcpp sandbox + if: ${{ github.event_name == 'push' && steps.sandbox.outputs.cache-hit != 'true' }} + uses: actions/cache/save@v4 + with: + path: ~/.mcpp + key: ${{ steps.sandbox.outputs.cache-primary-key }} + + - name: Save xlings + if: ${{ github.event_name == 'push' && steps.xlings.outputs.cache-hit != 'true' }} + uses: actions/cache/save@v4 + with: + path: ~/.xlings + key: ${{ steps.xlings.outputs.cache-primary-key }} + + - name: Save wine debs + if: ${{ github.event_name == 'push' && steps.wine.outputs.cache-hit != 'true' }} + uses: actions/cache/save@v4 + with: + path: ~/wine-debs + key: ${{ steps.wine.outputs.cache-primary-key }} + # ── windows → linux ─────────────────────────────────────────────────────── # The mirror of mingw-cross-wine. Two jobs because a Windows runner cannot # execute the ELF it produces; the artefact is handed to a Linux job and @@ -412,41 +451,20 @@ jobs: timeout-minutes: 60 steps: - uses: actions/checkout@v4 + # No job restores target/ any more. This one was the first to stop: a + # restored BMI tree beside a fresh one made GCC report + # `import 'std' has CRC mismatch` on the second of its two builds. - uses: ./.github/actions/bootstrap-mcpp + + - uses: ./.github/actions/use-built-mcpp with: - # Do NOT restore target/ in a cross job. This job builds twice — once - # for the host, then once for x86_64-linux-musl — and a restored BMI - # tree makes the second build read `std` BMIs that no longer match - # what the dependency BMIs were compiled against: - # - # mcpplibs.cmdline: error: import 'std' has CRC mismatch - # - # GCC bakes a CRC of each imported module's BMI into the importer, so - # the two have to come from the same build round; a cache that - # restores one without the other is not a partial speedup, it is an - # unbuildable tree. It reproduced on rerun, and only in this job — - # the other two cross jobs below cache ~/.mcpp and ~/.xlings but - # deliberately never target/, which is the convention this now follows. - # - # It stayed hidden until a PR touched neither mcpp.toml nor - # .xlings.json: those two files key the sandbox cache, so every - # earlier run had been a cold miss. - cache-target: 'false' + host: windows-x86_64 - - name: Build mcpp from source (self-host) + - name: Name this commit's mcpp MCPP_SELF shell: bash run: | - export MCPP_VENDORED_XLINGS="$XLINGS_BIN" - "$MCPP" build - # Newest, not first: target/ is cache-restored and keeps a directory - # per build fingerprint, so `find | head -1` can hand back the - # PREVIOUS release's binary (that is how a 0.0.106 build ran 0.0.105). - MCPP_SELF=$(find target -name "mcpp.exe" -path "*/bin/*" -printf "%T@ %p\n" \ - | sort -rn | head -1 | cut -d" " -f2-) - test -n "$MCPP_SELF" || { echo "FAIL: no mcpp.exe"; exit 1; } - MCPP_SELF=$(cd "$(dirname "$MCPP_SELF")" && pwd)/$(basename "$MCPP_SELF") - "$MCPP_SELF" --version - echo "MCPP_SELF=$MCPP_SELF" >> "$GITHUB_ENV" + "$MCPP_FRESH" --version + echo "MCPP_SELF=$MCPP_FRESH" >> "$GITHUB_ENV" - name: Install the linux-musl cross toolchain (windows-hosted canadian) shell: bash diff --git a/.github/workflows/measure-windows-tool-crt.yml b/.github/workflows/measure-windows-tool-crt.yml index f3a95551..c3a06c92 100644 --- a/.github/workflows/measure-windows-tool-crt.yml +++ b/.github/workflows/measure-windows-tool-crt.yml @@ -104,8 +104,6 @@ jobs: if ($left) { Write-Host "FAIL: VC tools still present after masking"; exit 1 } - uses: ./.github/actions/bootstrap-mcpp - with: - cache-target: 'false' - name: Install xim:qt-base 6.11.1 and remove its copy of the runtime shell: bash diff --git a/.github/workflows/openkal-cross.yml b/.github/workflows/openkal-cross.yml index dcd4923b..aed6e099 100644 --- a/.github/workflows/openkal-cross.yml +++ b/.github/workflows/openkal-cross.yml @@ -55,17 +55,7 @@ name: openkal cross-build (3 hosts × 3 targets) # means the unwinder found this image's own frame descriptions. on: - pull_request: - workflow_dispatch: - inputs: - openkal_ref: - description: "Branch of the openkal packages to build against (empty = main)" - required: false - default: "" - -concurrency: - group: ci-${{ github.workflow }}-${{ github.ref }} - cancel-in-progress: true + workflow_call: env: # No mcpp or xlings version here. `bootstrap-mcpp` owns both, and a second @@ -138,7 +128,22 @@ jobs: # THE mcpp UNDER TEST. Everything after this uses the binary this step # produces; the bootstrapped one above is only what compiles it. + # This commit's mcpp: the one build.yml produced (use-built-mcpp, rule R1 + # of the 2026-10-02 CI record), except on the macOS leg. That leg runs on + # macos-14 and the macOS artifact is built on macos-15; whether it runs on + # macos-14 is not measured (the record's Part VII), so the leg builds its + # own until it is. + - if: matrix.host != 'macos' + uses: ./.github/actions/use-built-mcpp + with: + host: ${{ matrix.host == 'linux' && 'linux-x86_64' || 'windows-x86_64' }} + + - name: Name this commit's mcpp MCPP_UNDER_TEST + if: matrix.host != 'macos' + run: echo "MCPP_UNDER_TEST=$MCPP_FRESH" >> "$GITHUB_ENV" + - name: Build the mcpp in this pull request + if: matrix.host == 'macos' # ci-lint: allow-r1: the macOS leg builds its own mcpp until the artifact is measured on macos-14 run: | set -euo pipefail export MCPP_VENDORED_XLINGS="$XLINGS_BIN" @@ -481,26 +486,12 @@ jobs: - uses: actions/checkout@v4 - uses: ./.github/actions/bootstrap-mcpp - - name: Build the mcpp in this pull request - run: | - set -euo pipefail - export MCPP_VENDORED_XLINGS="$XLINGS_BIN" - "$XLINGS_BIN" config --mirror GLOBAL 2>/dev/null || true - "$MCPP" self config --mirror GLOBAL 2>/dev/null || true - "$MCPP" build --dev - # NEWEST BY MTIME, NOT FIRST BY DIRECTORY ORDER. `target/` holds one - # directory per fingerprint and the runner restores a cache of it, so - # `find … | head -1` can return a binary an earlier run left behind. - # Measured: it reported the right VERSION STRING — the stale copy was - # built from an earlier push of this same release — while missing the - # last two commits, so one new test passed and two failed for reasons - # that were nowhere in the source. - BUILT=$(find target -type f -name 'mcpp' -printf '%T@ %p\n' \ - | sort -rn | head -1 | cut -d' ' -f2) - [ -n "$BUILT" ] || { echo "::error::mcpp did not build"; exit 1; } - BUILT=$(cd "$(dirname "$BUILT")" && pwd)/$(basename "$BUILT") - echo "MCPP_UNDER_TEST=$BUILT" >> "$GITHUB_ENV" - "$BUILT" --version + - uses: ./.github/actions/use-built-mcpp + with: + host: linux-x86_64 + + - name: Name this commit's mcpp MCPP_UNDER_TEST + run: echo "MCPP_UNDER_TEST=$MCPP_FRESH" >> "$GITHUB_ENV" - name: Install what the scripts declare run: | diff --git a/CHANGELOG.md b/CHANGELOG.md index 75f2b13c..461a702c 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -4,6 +4,70 @@ > Each `## []` section is that release's notes. Entries are written in English > from 2026.9.28.3 on; earlier entries remain as written. +## [2026.10.2.1] - 2026-10-02 + +This release repairs two fast-path defects (mcpp#756, mcpp#757), publishes the +specification by which toolchains are supported and maintained (SPEC-009), and +restructures the repository's CI so that a commit builds mcpp once per host +(`.agents/docs/2026-10-02-pr-ci-acceleration-and-the-toolchain-specification-design.md`). +No default toolchain changes. + +### Fixed + +- **An upgraded mcpp no longer replays a graph that names the previous one** + (mcpp#757). The record of a build names the engine that wrote it, its version + and the path of its executable, and the fast paths of `mcpp build`, of a + workspace build and of `mcpp run` decline a record that another engine wrote. + The graph's own `$mcpp` binding is asked as well, since `--configure-only` + from another install rewrites the graph without the record. Before, the fast + path compared the recorded fingerprint with the recorded directory, and after + an upgrade that removed the old install every action that starts mcpp failed + with `CreateProcess failed` or `not found`. A record written by an earlier + release declines once. +- **One predicate admits a recorded build for all three fast paths**, so a + property of the record is checked in one place rather than three. +- **A path dependency's sources are classified by its own package** (mcpp#756). + A provider that declares `.ixx` in `[build] module_extensions` has its edits + seen by the fast path whatever the consumer declares; before, the sweep used + the consumer's table, and the edit of a `.ixx` host module was replayed as + "no work to do". + +### Added + +- **SPEC-009, the support and maintenance of toolchains** + (`docs/specs/toolchain-maintenance.md`). It states the support tiers, the one + table every default is read from, how a toolchain line moves as a whole, the + provenance of a payload and when a patched payload may be published, what it + means that mcpp supports a line, the compiler-defect register, the handling + of a new host platform release, the mirror rule, and the order in which a + default moves. A default moves only after the candidate passes a measured + gate: the module experience is not worse. SPEC-006 §7 now refers to it. Most + rules are not implemented yet, and each says so. + +### Changed (repository CI) + +- **One build of mcpp per host.** `ci.yml` runs a commit's CI in stages. + `changes` classifies the changed paths, and a change of documentation that + nothing reads runs the documentation checks only. `docs` runs the checks that + need no binary. `build.yml` builds mcpp once per host, and the per-area + workflows, now reusable, take that binary through + `.github/actions/use-built-mcpp`. Measured before this change, 30 to 37 jobs + of a commit built mcpp from source, which was 44 to 49 percent of its runner + minutes. +- **One writer per cache.** Every job restores. One job per key saves, on a + push to `main` only, and `target/` is no longer cached, because a restored + `target/` made no build incremental. +- **E2E shards by measured duration**, four on Linux, three on Windows and two + on macOS, from `tests/e2e/timings/`. `run_all.sh` takes `E2E_TIMINGS`, + `E2E_REPORT` and `E2E_LIST`. +- **Every e2e test runs somewhere.** The `e2e-coverage` job fails when a test + ran on no host, is named by no workflow, and is not excused in + `tests/e2e/coverage-exceptions.tsv`. Before it, 24 tests ran on no runner. + `run_all.sh` grants `llvm` on Linux, and it probes `musl` and `mingw-cross` + by family rather than by one release. +- The legs that are known red (#669) run on `main`, on dispatch, and on a pull + request labelled `macos-27` (the label is read on the next push). + ## [2026.10.1.3] - 2026-10-01 This release gives every tool a build uses a source that can be declared, diff --git a/README.md b/README.md index 14a62860..06306cd0 100644 --- a/README.md +++ b/README.md @@ -12,7 +12,7 @@ | [Documentation](docs/) · [Getting Started](docs/01-getting-started.md) · [mcpp.toml Guide](docs/04-mcpp-toml.md) · [Examples](docs/03-examples.md) · [Toolchains](docs/20-toolchains.md) | |:---:| | [Package index mcpp-index](https://mcpplibs.github.io/mcpp-index/) · [Module libraries mcpplibs](https://github.com/mcpplibs) · [Community Forum](https://forum.d2learn.org/category/20) · [Issues](https://github.com/mcpp-community/mcpp/issues) · [Releases](https://github.com/mcpp-community/mcpp/releases) | -| [![ci-linux](https://github.com/mcpp-community/mcpp/actions/workflows/ci-linux.yml/badge.svg?branch=main)](https://github.com/mcpp-community/mcpp/actions/workflows/ci-linux.yml) [![ci-macos](https://github.com/mcpp-community/mcpp/actions/workflows/ci-macos.yml/badge.svg?branch=main)](https://github.com/mcpp-community/mcpp/actions/workflows/ci-macos.yml) [![ci-windows](https://github.com/mcpp-community/mcpp/actions/workflows/ci-windows.yml/badge.svg?branch=main)](https://github.com/mcpp-community/mcpp/actions/workflows/ci-windows.yml) | +| [![ci](https://github.com/mcpp-community/mcpp/actions/workflows/ci.yml/badge.svg?branch=main)](https://github.com/mcpp-community/mcpp/actions/workflows/ci.yml) | | Plugins · [mcpp-language-server (mcppls)](https://github.com/Sunrisepeak/mcpp-language-server) — a C++20/23 modules language server for VS Code, Zed, CLion, Neovim, AI agents (MCP) and CI |

diff --git a/README.zh-CN.md b/README.zh-CN.md index 521dfcb9..5c0fdb1f 100644 --- a/README.zh-CN.md +++ b/README.zh-CN.md @@ -12,7 +12,7 @@ | [文档](docs/zh/) · [快速开始](docs/zh/01-getting-started.md) · [mcpp.toml 指南](docs/zh/04-mcpp-toml.md) · [示例项目](docs/zh/03-examples.md) · [工具链管理](docs/zh/20-toolchains.md) | |:---:| | [包索引 mcpp-index](https://mcpplibs.github.io/mcpp-index/) · [模块化库 mcpplibs](https://github.com/mcpplibs) · [社区论坛](https://forum.d2learn.org/category/20) · [Issues](https://github.com/mcpp-community/mcpp/issues) · [Releases](https://github.com/mcpp-community/mcpp/releases) | -| [![ci-linux](https://github.com/mcpp-community/mcpp/actions/workflows/ci-linux.yml/badge.svg?branch=main)](https://github.com/mcpp-community/mcpp/actions/workflows/ci-linux.yml) [![ci-macos](https://github.com/mcpp-community/mcpp/actions/workflows/ci-macos.yml/badge.svg?branch=main)](https://github.com/mcpp-community/mcpp/actions/workflows/ci-macos.yml) [![ci-windows](https://github.com/mcpp-community/mcpp/actions/workflows/ci-windows.yml/badge.svg?branch=main)](https://github.com/mcpp-community/mcpp/actions/workflows/ci-windows.yml) | +| [![ci](https://github.com/mcpp-community/mcpp/actions/workflows/ci.yml/badge.svg?branch=main)](https://github.com/mcpp-community/mcpp/actions/workflows/ci.yml) | | 支持的插件 · [mcpp-language-server（mcppls）](https://github.com/Sunrisepeak/mcpp-language-server) —— C++20/23 模块语言服务器，面向 VS Code、Zed、CLion、Neovim、AI Agent（MCP）与 CI |

diff --git a/docs/04-mcpp-toml.md b/docs/04-mcpp-toml.md index b79af2dc..48be22ce 100644 --- a/docs/04-mcpp-toml.md +++ b/docs/04-mcpp-toml.md @@ -906,7 +906,11 @@ one key rather than several: 2. those units compile with the **module** rule — they emit a BMI and their objects are linked unconditionally; 3. the freshness fast path watches them, so adding an `import` to one - invalidates the build graph instead of silently reusing a stale one. + invalidates the build graph instead of silently reusing a stale one. A + package's files are classified by that package's own table: a `path` + dependency that declares `.ixx` for its sources is watched for `.ixx` + whatever its consumer declares, and the consumer needs no entry of its own + for it. Any extension is accepted **except** ones that already name a non-module role (`.cpp` `.cc` `.cxx` `.c` `.m` `.mm` `.h` `.hpp` `.hh` `.hxx` `.S` `.s` diff --git a/docs/09-commands-by-scenario.md b/docs/09-commands-by-scenario.md index 86c19fce..6b64d59f 100644 --- a/docs/09-commands-by-scenario.md +++ b/docs/09-commands-by-scenario.md @@ -123,6 +123,13 @@ For that invocation the option takes the place of `[toolchain] default` in `MCPP_TOOLCHAIN`. Each toolchain builds into its own output directory, and a recorded build is replayed only for the toolchain request that recorded it. +A recorded build is replayed only by the engine that recorded it. The record +names the engine by version and by the path of its executable, because the +build graph names that executable and starts it. After an upgrade, or once the +executable has moved or been removed, the next `mcpp build` plans the graph +again and writes the record anew; nothing under `target/` needs to be deleted +by hand. + ## Explaining a resolution `mcpp why` reports what a build would resolve, and builds nothing: diff --git a/docs/README.md b/docs/README.md index 1ad37ccc..4eea17b3 100644 --- a/docs/README.md +++ b/docs/README.md @@ -164,3 +164,4 @@ downstream tooling. - [SPEC-006 — Toolchain management: identity, origin, selection and the payload contract](specs/toolchain-management.md) - [SPEC-007 — Build plugins: configuration, construction and verification, and the runtime and planning obligations](specs/build-plugins.md) - [SPEC-008 — A library's interface: public modules, the published closure, and one interface in both forms](specs/library-interface.md) + - [SPEC-009 — The support and maintenance of toolchains: lines, defaults, provenance, moves and retirement](specs/toolchain-maintenance.md) diff --git a/docs/specs/README.md b/docs/specs/README.md index 0647d87e..7e5e77be 100644 --- a/docs/specs/README.md +++ b/docs/specs/README.md @@ -35,9 +35,10 @@ | [SPEC-003](exit-codes.md) | 退出码契约(分类、语义、稳定性承诺) | 评审中 v1.0 | 2026-09-01 | mcpp >= 2026.9.1.1 | | [SPEC-004](manifest-semantics.md) | `mcpp.toml` 的平面划分、条件化形状、解析轴与命名规约 | 草案 v1.10 | 2026-09-28 | 条件化形状 mcpp >= 2026.8.29.1;目标轴 mcpp >= 2026.9.6.4;`linkage` 默认值 mcpp >= 2026.9.15.2;链接 flag 的词读法 mcpp >= 2026.9.26.2;条件化的 `dialect_cxxflags` 与 `-p` 的包身份 mcpp >= 2026.9.28.1;条件表按具体程度生效 mcpp >= 2026.9.28.2 | | [SPEC-005](build-database.md) | 构建数据库:`mcpp emit build-database` 的内容、取值规则与不写工程目录的保证 | 评审中 v1.6 | 2026-09-29 | mcpp >= 2026.9.15.1;v1.3 条款 mcpp >= 2026.9.26.2;v1.4 条款 mcpp >= 2026.9.27.1;v1.5 条款 mcpp >= 2026.9.28.1;v1.6 条款 mcpp >= 2026.9.29.5 | -| [SPEC-006](toolchain-management.md) | 工具链管理:身份、来源、选择与载荷契约 | 草案 v0.4 | 2026-09-28 | 逐条标注;已实现条款 mcpp >= 2026.9.24.1;§3.7 mcpp >= 2026.9.28.1;§3.7.1 mcpp >= 2026.9.28.2 | +| [SPEC-006](toolchain-management.md) | 工具链管理:身份、来源、选择与载荷契约 | 草案 v0.6 | 2026-10-02 | 逐条标注;已实现条款 mcpp >= 2026.9.24.1;§3.7 mcpp >= 2026.9.28.1;§3.7.1 mcpp >= 2026.9.28.2;§2.2.1 与 §3.3 的非缺省来源 mcpp >= 2026.10.1.3 | | [SPEC-007](build-plugins.md) | 构建插件:配置、施工与校验的分工,运行时与规划期的义务 | 草案 v0.6 | 2026-09-28 | 逐条标注;mcpp >= 2026.9.26.2;v0.3 条款 mcpp >= 2026.9.27.1;v0.4 条款 mcpp >= 2026.9.28.1;v0.5 条款 mcpp >= 2026.9.28.2;v0.6(§9)mcpp >= 2026.9.28.3 | | [SPEC-008](library-interface.md) | 库的接口:公开模块、发布闭包与两种形态的一致 | 草案 v0.1 | 2026-09-28 | 第一阶段(只警告)mcpp >= 2026.9.28.3 | +| [SPEC-009](toolchain-maintenance.md) | 工具链的支持与维护:版本线、默认值、来源、移动与退役 | 草案 v0.1 | 2026-10-02 | 逐条标注;本版只有规范,多数条款未实现 | ## 文档约定 diff --git a/docs/specs/toolchain-maintenance.md b/docs/specs/toolchain-maintenance.md new file mode 100644 index 00000000..f2ff6140 --- /dev/null +++ b/docs/specs/toolchain-maintenance.md @@ -0,0 +1,386 @@ +# SPEC-009:工具链的支持与维护 + +| 项 | 值 | +|---|---| +| 规范编号 | SPEC-009 | +| 标题 | 工具链的支持与维护:版本线、默认值、来源、移动与退役 | +| 状态 | 草案 v0.1 | +| 最后修改 | 2026-10-02 | +| 对应实现 | 逐条标注;本版只有规范,多数条款未实现 | +| 相关设计文档 | `.agents/docs/2026-10-02-pr-ci-acceleration-and-the-toolchain-specification-design.md`(第 IV 部分) | +| 相关 issue | mcpp#669(macOS 27 的链接)、mcpp#685、mcpp#687、mcpp#755 | +| 使用文档 | [docs/20 - 工具链](../zh/20-toolchains.md) | + +本规范定义 mcpp 支持的工具链集合如何随时间变化:哪些版本被支持、默认值由哪一张表给出、载荷从哪里来、 +一个默认值如何移动,以及已经记录了默认值的机器如何得知新的默认值。 +[SPEC-006](toolchain-management.md) 定义工具链是什么、如何命名与选择、载荷必须包含什么,本规范不重复这些内容。 + +本规范不指定任何一个具体版本为默认值。具体版本由一张线表给出(§3);本版发布时线表尚不存在, +默认值仍由两张表分别给出(§3.1)。各节编号与设计过程中的条款编号 TS-n 一一对应。 + +用语按 RFC 2119:**必须 / 禁止**(强制)、**应当**(强烈建议)、**可以**(可选)。实现状态标记见 [规范索引](README.md)。 + +--- + +## 1. 术语 + +族与载荷沿用 [SPEC-006](toolchain-management.md) §1 与 §2.1 的定义。本节增加以下术语。 + +| 术语 | 含义 | +|---|---| +| 族(family) | `gcc`、`llvm`、`msvc`、`emsdk`、`android-ndk` | +| 线(line) | 一个族加一个主版本:GCC 16、LLVM 23 | +| 发布(release) | 一条线上的一个上游版本:16.2.0、23.1.3 | +| 载荷(payload) | 一个发布在某个宿主与变体上的可安装目录树:glibc 的 `gcc`、原生 `musl-gcc`、交叉的 `-linux-musl-gcc`、`mingw-gcc`、`mingw-cross-gcc`、`llvm` | +| 行(row) | 目标矩阵(`kKnownTargets`)的一行,或一个宿主默认值 | +| 默认(default) | 一行在没有任何声明时解析到的发布 | +| 线表(line table) | 引擎中唯一一张给出每一行默认的表(§3) | +| 自举工具链(self-host toolchain) | mcpp 自己的 `mcpp.toml` 所声明的工具链 | + +--- + +## 2. 支持分级 + +### 2.1 三个等级 未实现 + +发布按下列三个等级分级。 + +| 等级 | 范围 | 义务 | +|---|---|---| +| Default | 线表为一行所指名的发布 | 该行 CI 宿主上的完整 e2e 套件;该宿主构建 mcpp 时,mcpp 自身的构建;发布产物以它构建 | +| Supported | 每一行上一个 Default,以及线表明确列出的发布 | 线表或其载荷变化时,CI 运行验收程序(SPEC-006 §6.2);回归是缺陷 | +| Available | 索引中的其余所有发布 | 无 CI,接受报告。已知缺陷**必须**以载荷的能力(`.mcpp-toolchain.json`)陈述,**禁止**写成引擎里的版本分支 | + +当前:引擎不按发布分级。`kKnownTargets` 的 `tier`(`verified`、`preview`、`planned`)是目标行的验证级别,与本节的发布分级是两个概念。 + +### 2.2 下限 未实现 + +每个族的下限,即 mcpp 接受其 `import std` 的最老的线,**必须**写在线表中。 + +当前:下限是隐含的,没有一处陈述。 + +--- + +## 3. 一张线表 + +### 3.1 唯一的来源 未实现 + +每一个默认与每一个行的钉住版本,**必须**读自引擎中的同一张表。表为每一行写明族、发布、变体与等级; +行的发布不同于其族的 Default 时,**必须**另写明理由,以及差异在何种条件下终止。 + +当前:默认值由两张表分别给出,均在 `modules/toolchain-model/src/triple.cppm`:表 A 是 `pins::kFirstRun*`(附带重复的 `kSuggest*` 字面量), +表 B 是 `kKnownTargets[].pin`。两张表之间只有一处耦合被检查:`kFirstRunWinGnu` 与 `x86_64-windows-gnu` 行的一致 +(`tests/unit/test_windows_defaults.cpp`)。 + +### 3.2 引擎内的读者 未实现 + +帮助文本、安装建议、错误信息与 `mcpp self env --format json` **必须**由线表格式化。引擎内出现第二处默认值字面量是缺陷。 + +当前:`mcpp self env --format json` 的 `data.defaultToolchain` 取自 `pins::host_default_toolchain`(表 A)。安装建议使用 `kSuggest*` 字面量; +两张表之外,另有 21 处默认值字面量分布在九个源文件中。 + +### 3.3 引擎外的读者 部分实现 + +引擎之外凡写出默认值者(文档、工作流、测试、示例与 mcpp 自己的清单),**必须**读自 `mcpp self env --format json`, +或者由一项 CI 检查与它比对。既不读取也不比对的字面量是缺陷。 + +当前:只有宿主默认值在文档中的四条陈述被比对:`.github/tools/check_default_toolchain_docs.py` 在每个 CI 宿主上检查该宿主的行, +在 `docs/01`、`docs/20` 及其 `docs/zh/` 副本中各一条。其余读者既不读取也不比对:`mcpp.toml`、`tests/matrix/expected.tsv` 的 175 行、 +七个工作流、一个 action、六个 CI 工具、至少八个 e2e 脚本、示例,以及 33 个文档文件中的其余陈述。 + +### 3.4 索引的 latest 不是默认值 已实现 + +索引的 `latest` **禁止**作为默认值。mcpp 钉住精确的发布;移动 `latest` 不改变 mcpp 构建的任何结果。 + +--- + +## 4. 版本线整体移动 + +### 4.1 一个族在一台宿主上解析到一个发布 未实现 + +在一台宿主上,同一个族的各行**应当**解析到同一个发布。一行**可以**落后,但**必须**在线表中写明理由, +且理由**必须**在下一次移动时重新评估。 + +当前:不成立。macOS 与带 MSVC 的 Windows 默认 `llvm@20.1.7`,而 17 个目标行钉住 `llvm@22.1.8`。Linux 在 x86_64 之外的架构默认 +`gcc@15.1.0-musl`,其余 gcc 行为 16.1.0。这些落后都没有记录理由。 + +### 4.2 载荷齐备 未实现 + +一个发布进入某一行的 Default,**必须**以该行所需的每一个载荷在该发布上都存在为前提,且两个镜像上都有(§9)。 +同一条 GCC 线的载荷(glibc、原生 musl、交叉 musl、mingw)在此处视为同一条线。 + +当前:没有程序检查此条件。 + +--- + +## 5. 来源 + +### 5.1 上游发布 部分实现 + +载荷**必须**出自一个上游发布:官方标签,及其官方的源码或二进制归档;载荷的描述**必须**记录该归档的 sha256(SPEC-006 §4.5)。 + +当前:索引以 sha256 固定资产(SPEC-006 §4.6),但描述文件尚不记录上游源码的 sha256(SPEC-006 §4.5 未实现)。 + +### 5.2 配方 未实现 + +载荷**必须**由检入仓库的配方产出,由一个 CI 作业运行,或由 CI 作业可以运行的脚本运行。人工步骤不属于配方(SPEC-006 §5.1)。 + +当前:glibc 的 `gcc` 载荷由人工按 `fromsource` 配方构建,随后的 strip 与 `specs` 改写没有任何仓库中的脚本(16.1.0 如此)。 +`musl-gcc` 与 `aarch64-linux-musl-gcc` 在 `xlings-res` 中有可派发的构建工作流;`mingw-cross-gcc` 与 Windows 宿主上的 Canadian cross 是手工的; +`mingw-gcc` 转存 winlibs 的发布。 + +### 5.3 打补丁的载荷 部分实现 + +打补丁的载荷,即一个上游发布加上尚无发布携带的上游提交,**可以**发布,当且仅当以下条件全部成立: + +- 每一个补丁都是上游发布分支上的一个提交,或是主分支上带有未关闭的回移(backport)的提交,并以哈希引用; +- 它作为该发布的修订发布(`revision = N`,资产名为 `<发布>-r`),**禁止**使用官方资产名,其描述列出全部补丁; +- 它只放在需要它的行上,不放在其他行上; +- 它有退出条件:当某个上游发布携带了这些补丁,Default 移到该发布,修订只为已钉住它的使用者留在索引中。 + +当前:索引已有修订的先例(xim-pkgindex 的 glibc 2.44.3-r1,`revision` 字段与 `-rN` 资产名),机制存在;上述四项条件没有程序检查。 + +### 5.4 引擎不改动平台的输入 部分实现 + +引擎**禁止**以改变平台输入的方式绕开工具链缺陷,例如选择更旧的 SDK、改写 SDK 的文件、替换另一个链接器,除非那就是该行声明的设计。 +工具链缺陷由工具链的发布修复,或由 §5.3 的打补丁载荷修复。 + +当前:mcpp#669 没有以改动平台输入的方式处理:引擎没有为它选择更旧的 SDK、改写 SDK 的文件或替换链接器。全引擎范围的符合性没有核对,也没有检查。 + +--- + +## 6. 支持一条版本线的含义 + +### 6.1 能力取自载荷 部分实现 + +引擎**必须**从载荷推导工具链能做什么,而不是从版本:标准库模块取自库的清单或布局,扫描器取自驱动,flag 取自在该行上测得的探针。 +版本号的比较只在两种情形被允许:一项登记于 §7 的缺陷,或一张语言特性表,其每一行引用发布说明。 + +当前:标准库模块、扫描器与描述文件的字段按载荷取得(SPEC-006 §4.4、§4.5)。引擎中是否还有按版本号的分支,没有逐处核对;已知的一处是按 `cl.exe` 横幅的版本选择 std 模块的语言级别(`src/toolchain/msvc.cppm`)。 + +### 6.2 Default 行的验收 部分实现 + +在一个线是 Default 的每一行上,引擎**必须**通过验收程序(一个使用 ``、`` 与 `` 的程序;`import std`;`import std.compat`)与 e2e 套件; +该宿主构建 mcpp 时,还**必须**通过 mcpp 自身的构建。 + +当前:e2e 套件在各 CI 宿主上运行;验收程序的矩阵部分实现(SPEC-006 §6.2)。macOS 的 Default `llvm@20.1.7` 无法构建 mcpp 自身 +(其 libc++ 的 `std` 模块不暴露 `directory_iterator` 的比较),mcpp 的清单因此以 22.1.8 构建;该行的第三项验收不成立。 + +### 6.3 随发布编码的输出 部分实现 + +每一个编码了发布的输出,**必须**在移动时被评审:包的 ABI 标签(编译器主版本)、BMI 与缓存的身份(版本进入指纹与每一个缓存键,因此移动不需要更换纪元),以及引用版本的诊断。 + +当前:ABI 标签带编译器主版本(`src/pack/abi_tag.cppm`):GCC 16.1 到 16.2 保持 `gcc16-libstdcxx16`;LLVM 22 到 23 由 `clang22-libcxx22` +变为 `clang23-libcxx23`,带旧标签的预制产物被拒绝(`src/pack/prebuilt.cppm`)。版本已进入指纹(`modules/toolchain-model/src/fingerprint.cppm`)。评审本身没有清单,也没有检查。 + +--- + +## 7. 编译器缺陷登记 + +### 7.1 登记项 未实现 + +凡因一个编译器缺陷而存在的引擎行为,以及凡因此而采取的 mcpp 自身源码的形状,**必须**有一个登记项,写明:族、首次观察到的发布、最后验证过的发布、上游报告、 +`tests/` 下一个在缺陷存在时失败的最小复现,以及依赖它的位置。 + +当前:没有登记表。这些行为的原因散见于源码注释与早先的记录,不保证每一项都有最小复现。 + +### 7.2 移动时重跑 未实现 + +每一次移动一条线时,**必须**对新发布运行每一个复现,并在登记项中记录结果。 + +当前:没有登记表,也就没有可运行的集合。 + +### 7.3 移除条件 未实现 + +一个绕开缺陷的做法,**必须**在每一个 Supported 发布都越过修复之后才移除。 + +当前:没有此规则;各处绕行的移除时机由修改者判断。 + +### 7.4 已知条目 + +以下条目取自源码注释与早先的记录,待 §7.1 落地后写成正式的登记项。 + +| 缺陷 | 族与发布 | 位置 | +|---|---|---| +| 一个实例化沿模块导入链被丢弃 | GCC 16.1 | — | +| 模块接口中的 `FILE` 类型实体使其后的 `#include ` 失败 | GCC PR 99000(未关闭) | — | +| 嵌套的 `std::map` 成员使 BMI 被截断 | GCC 16.1 | — | +| 一个新的、被广泛导入且接口含标准类型的模块毒化下游的 BMI | GCC 16.1 | — | +| 新接口单元上的段错误 | GCC 16.1 | `src/build/prepare.cppm` | +| 完整 BMI 下的误编译,因此 clang 使用两阶段的精简接口 | clang 22.1.8 | `src/build/ninja_backend.cppm` | +| 模块 purview 中的内联辅助函数导致崩溃 | clang 20.1.7(Windows) | — | +| 27.0 SDK 在模块下把 `INFINITY` 与 `NAN` 留给 `` | clang 22 与 macOS 27.0 SDK | `src/toolchain/hostflags.cppm` | +| libc++ 20 的 `std` 缺少 `directory_iterator` 的比较 | libc++ 20 | `mcpp.toml` 的 `[toolchain]` | +| `AMDGPUAsmParser.cpp` 的内部编译器错误,因此 `llvm-dev` 以 GCC 15.1.0 构建 | GCC 16.1 | — | + +--- + +## 8. 宿主平台的新版本 + +### 8.1 新版本的 CI 腿 部分实现 + +一个行的使用者将会遇到的新操作系统或新 SDK 发布,**必须**在存在托管的 runner 镜像时尽早得到一条 CI 腿,早于该镜像成为 runner 的默认。 + +当前:macOS 27 在 `xcode-27` 镜像上已有 CI 腿,因 mcpp#669 为已知红色。没有一般性的清单规定哪些宿主版本须有腿。 + +### 8.2 已知红色的腿 已实现 + +因外部原因而红色的腿**必须**带 `known_red: '#'`,工作流断言(`.github/tools/check_workflow_assertions.py`)**必须**保证该 issue 未关闭, +issue 关闭时该腿**必须**离开已知红色的列表。 + +### 8.3 修复 部分实现 + +宿主平台新版本引出的工具链缺陷,其修复遵循 §5.3 与 §5.4。 + +当前:状态同 §5.3 与 §5.4。mcpp#669 尚未修复,两条 `xcode-27` 腿保持已知红色。 + +--- + +## 9. 镜像 + +### 9.1 先镜像,后入索引 未实现 + +一个版本行**禁止**在它所指名的每一个资产都出现在 GLOBAL 与 CN 两个镜像上、并在两者上经 GET 验证(状态码 200、字节数与 sha256)之前进入索引。 + +当前:没有程序检查镜像的存在与哈希,顺序由发布者保证。 + +### 9.2 CN 镜像的上传 未实现 + +向 CN 镜像上传大于约 8 MiB 的文件,**必须**从 CN 网络内的主机进行。 + +当前:没有检查;该条是一项操作约束。 + +--- + +## 10. 移动默认版本 未实现 + +移动一行的 Default **必须**按以下顺序进行。当前:没有程序串联这些步骤;没有线表,步骤 10.6 须同时修改两张表及其全部副本(§3);步骤 10.5 见该节。 + +### 10.1 上游发布 + +该上游发布已经存在。 + +### 10.2 载荷 + +其载荷由各自的配方构建(§5),通过 SPEC-006 §6 的准入,并各自记录其输入。 + +### 10.3 镜像 + +资产已在两个镜像上并通过验证(§9)。 + +### 10.4 索引 + +索引增加该版本的行;`latest` 不变。 + +### 10.5 门:新发布不使模块体验变差 + +引擎在每一个将以该发布为 Default 的行上得到验证,且该发布通过下列的门。载荷**可以**按路径命名(SPEC-006 §2.2.1),因此本步不必等待 10.4。 + +候选发布 R 与该行当前的 Default D 在同一个 mcpp 提交、同一个 runner 镜像、同一个作业中比较,使比较只跨越发布的变化。 +R 通过,当且仅当下列六项全部成立。 + +| 门 | 判据 | +|---|---| +| G1 | 凡以 D 通过的 e2e 测试,以 R 也通过;凡以 D 运行的测试,以 R 不被跳过 | +| G2 | 该行构建 mcpp 时,mcpp 以 R 构建自身,且该二进制通过套件 | +| G3 | §6.2 的验收程序能够构建并运行 | +| G4 | 运行 §7 的每一个复现,且 G1 至 G3 不需要新的绕行或重塑的源码;需要者使门失败,除非评审接受并附登记项 | +| G5 | mcpp 自身源码与 e2e 模块夹具所扫描出的模块图(每个单元提供与需要的模块),在 R 与 D 下相同 | +| G6 | mcpp 与 `bench/` 工程的冷构建和暖构建,以 R 所用时间不超过 D 的 110%(三次运行的中位数),BMI 的体积不超过 110% | + +未通过门的发布留在 Available。同一条线的下一个发布成为候选。 + +当前:没有门的工作流,G1 至 G6 均未实现;CI 不在同一作业中比较两个发布。 + +### 10.6 引擎 + +一个 mcpp 拉取请求移动线表,以及线表的每一个读者与每一个被检查的副本;随后发布 mcpp。 + +### 10.7 latest + +在该 mcpp 发布之后,索引的 `latest` 移动。 + +### 10.8 等级移位 + +此前的 Default 成为 Supported,再之前的线成为 Available。 + +### 10.9 撤销 + +移动通过回退 10.6 撤销。载荷与索引的行保留。 + +### 10.10 C 库绑定 + +本条承接 SPEC-006 §7 原有的规则:移动 C 库绑定,或某个载荷的 `latest`,之前,SPEC-006 §6.2 的矩阵**必须**在新版本上全部通过。 +状态:未实现(§6.2 的矩阵部分实现,此顺序没有检查)。 + +--- + +## 11. 已有默认值的机器 + +### 11.1 声明的发布不移动 已实现 + +使用者或项目所声明的发布(`[toolchain]`、`--toolchain`、`MCPP_TOOLCHAIN`、`mcpp toolchain default`)**禁止**移动。 + +当前:已记录的默认值只由首次运行与 `mcpp toolchain default` 写入,没有路径因线表的变化而改写它。 + +### 11.2 首次运行记录的默认值 未实现 + +mcpp 在首次运行时写下的默认值不是使用者的声明,其记录**必须**如实表明这一点。 + +- 在没有已记录默认值的主目录上,首次运行安装并记录线表的答案,一如今日,此外不说任何话。 +- 通知只为一种情形存在:主目录里是由更早的 mcpp 的首次运行记录的默认值,而正在运行的 mcpp 的线表为该行指名了更新的发布。 +- 此时 mcpp **必须**陈述一次:已记录的发布、更新的发布,以及解决这个问题的两条命令。`mcpp toolchain default <更新的发布>` 移动记录; + `mcpp toolchain default --keep` 保留记录。任何一条命令都使该记录成为使用者的声明。 +- 通知不阻塞,不提示输入,不重复。使用者作答之前,已记录的发布继续使用。 + +当前:`src/build/prepare/toolchain.cpp:1607` 在首次运行时把默认值写入 `config.toml` 的 `[toolchain] default`(Windows 的 MinGW 回退另有两处写入), +此后没有任何路径按线表重新评估它,因此移动后的默认值只到达全新的主目录。该键与 `mcpp toolchain default ` 所写的是同一个键,记录无法区分首次运行与使用者的声明; +`mcpp toolchain default --keep` 不存在。 + +### 11.3 已安装的载荷 已实现 + +已安装的载荷保留;移除它们是使用者的决定。 + +当前:移除载荷的入口只有 `mcpp toolchain remove`。 + +--- + +## 12. mcpp 用自己给出的工具链构建 + +### 12.1 清单取线表 部分实现 + +mcpp 自己的清单**必须**使用其构建所在的每一行的 Default 发布。偏离是线表中的一项,带理由与退出条件。 + +当前:`mcpp.toml` 的 `[toolchain]` 为 `default = "gcc@16.1.0"`、`macos = "llvm@22.1.8"`、`windows = "llvm@20.1.7"`。`default` 等于 Linux x86_64 的默认, +`windows` 等于带 MSVC 的 Windows 的默认;`macos` 偏离该行的默认 `llvm@20.1.7`,原因见 §6.2,但没有线表项记录理由与退出条件。 + +--- + +## 13. 一致性检查 + +程序能够检查的规则**必须**在 CI 中被检查。 + +| 检查 | 内容 | 条款 | 状态 | +|---|---|---|---| +| C1 | `src/` 与 `modules/` 中,线表之外没有默认值字面量 | §3.2 | 未实现 | +| C2 | 文档中每一处对默认值的陈述,包括 `docs/21` 的目标行表与 README,与 `mcpp self env --format json` 一致 | §3.3 | 部分实现 | +| C3 | 工作流、action 与测试的版本取自线表;能力探针询问族,不询问版本 | §3.3 | 未实现 | +| C4 | `mcpp.toml` 的 `[toolchain]` 等于线表或一项已记录的偏离 | §12 | 未实现 | +| C5 | 一条 `known_red` 的腿指名一个未关闭的 issue | §8.2 | 已实现 | +| C6 | §7 的复现在每一个 Default 行上运行 | §7 | 未实现 | +| C7 | §10.5 的门是一个以行与候选载荷为输入的工作流,§10.6 的拉取请求引用它的运行 | §10 | 未实现 | + +当前: + +- C2:`check_default_toolchain_docs.py` 只比对宿主默认值的四条陈述(§3.3);`docs/21` 的目标行表与 README 没有覆盖。 +- C3:`tests/e2e/run_all.sh` 仅当 `musl-gcc/15.1.0` 已安装时授予 `musl`,仅当 `mingw-cross-gcc/16.1.0` 已安装时授予 `mingw-cross`。 + 默认值移动后,十六个测试被跳过且不报告。 +- C5:`check_workflow_assertions.py` 要求允许失败的作业在名字中指名一个 issue,并在检查 issue 状态时报告已不再打开的 issue。 + +--- + +## 变更记录 + +| 版本 | 日期 | 变更 | +|---|---|---| +| v0.1 | 2026-10-02 | 初版 | diff --git a/docs/specs/toolchain-management.md b/docs/specs/toolchain-management.md index 6c90275a..c2a3bd4e 100644 --- a/docs/specs/toolchain-management.md +++ b/docs/specs/toolchain-management.md @@ -4,10 +4,10 @@ |---|---| | 规范编号 | SPEC-006 | | 标题 | 工具链管理:身份、来源、选择与载荷契约 | -| 状态 | 草案 v0.5 | -| 最后修改 | 2026-10-01 | +| 状态 | 草案 v0.6 | +| 最后修改 | 2026-10-02 | | 对应实现 | 逐条标注;标为「已实现」的条款对应 mcpp >= 2026.9.24.1。标为「未实现」的条款计划与下一批 LLVM 工具链一同落地,届时按实测修订本规范 | -| 相关设计文档 | `.agents/docs/2026-09-24-toolchain-selection-and-payload-trust-design.md`、`.agents/docs/2026-09-24-685-687-msvc-stl-and-toolchain-payloads.md`、`.agents/docs/2026-09-28-ecosystem-design-and-optimisation-plan.md` | +| 相关设计文档 | `.agents/docs/2026-09-24-toolchain-selection-and-payload-trust-design.md`、`.agents/docs/2026-09-24-685-687-msvc-stl-and-toolchain-payloads.md`、`.agents/docs/2026-09-28-ecosystem-design-and-optimisation-plan.md`、`.agents/docs/2026-10-02-pr-ci-acceleration-and-the-toolchain-specification-design.md` | | 相关 issue | mcpp#685、mcpp#687、mcpp#718、mcpp#755 | | 使用文档 | [docs/20 - 工具链](../zh/20-toolchains.md)、[docs/32 - 编写载荷](../zh/32-authoring-a-payload.md)、[docs/91 - 工具链内部](../zh/91-toolchain-internals.md) | @@ -304,7 +304,8 @@ xim-pkgindex 的准入脚本 `verify-toolchain.sh` 对一个载荷归档做一 ## 7. 发布顺序 未实现 -移动 C 库绑定或某个载荷的 `latest` 之前,§6.2 的矩阵**必须**在新版本上全部通过。 +移动一个默认版本或一个 C 库绑定的顺序,由 [SPEC-009](toolchain-maintenance.md) §10 规定;§6.2 的矩阵是该顺序中的一道要求。 +本节不再自有条款,实现状态同 SPEC-009 §10:未实现。 --- @@ -317,3 +318,4 @@ xim-pkgindex 的准入脚本 `verify-toolchain.sh` 对一个载荷归档做一 | v0.3 | 2026-09-28 | 随 mcpp 2026.9.28.1:新增 §3.7,MSVC ABI 的 CRT 模型是目标 ABI 的性质,cl 与 clang++ 同样收到,默认 `toolchain-coupled`(mcpp#718)。 | | v0.5 | 2026-10-01 | 随 mcpp 2026.10.1.3(mcpp#755):新增 §2.2.1,工具链可由路径命名,并说明 `bootstrap` 与 `configure = "build.mcpp"`;§3.3 增加非缺省来源的陈述、汇总、记录与 `--managed-only`。 | | v0.4 | 2026-09-28 | 随 mcpp 2026.9.28.2:新增 §3.7.1,程序旁的文件由一个解析器决定;MSVC C++ 运行时是一个带版本的集合;契约决定种类;声明的运行时文件与 toolset 的版本比较;读不出的版本不作决定;action 的 `PATH` 首位是 toolset 的运行时目录(2026-09-28 设计 WS1)。 | +| v0.6 | 2026-10-02 | §7 移入 SPEC-009(工具链的支持与维护):移动默认版本或 C 库绑定的顺序、门与撤销由 SPEC-009 §10 规定,本节改为引用。 | diff --git a/docs/zh/README.md b/docs/zh/README.md index 8fac9f6b..9af4ddca 100644 --- a/docs/zh/README.md +++ b/docs/zh/README.md @@ -159,3 +159,4 @@ - [SPEC-006 —— 工具链管理：身份、来源、选择与载荷契约](../specs/toolchain-management.md) - [SPEC-007 —— 构建插件：配置、施工与校验的分工，运行时与规划期的义务](../specs/build-plugins.md) - [SPEC-008 —— 库的接口：公开模块、发布闭包与两种形态的一致](../specs/library-interface.md) + - [SPEC-009 —— 工具链的支持与维护：版本线、默认值、来源、移动与退役](../specs/toolchain-maintenance.md) diff --git a/mcpp.toml b/mcpp.toml index 7bd71af1..687186da 100644 --- a/mcpp.toml +++ b/mcpp.toml @@ -1,6 +1,6 @@ [package] name = "mcpp" -version = "2026.10.1.3" +version = "2026.10.3.1" description = "Modern C++ build & package management tool" license = "Apache-2.0" authors = ["mcpp-community"] diff --git a/modules/versioning/src/version.cppm b/modules/versioning/src/version.cppm index 0c24be54..0ed6c554 100644 --- a/modules/versioning/src/version.cppm +++ b/modules/versioning/src/version.cppm @@ -31,6 +31,6 @@ import std; export namespace mcpp { -inline constexpr std::string_view MCPP_VERSION = "2026.10.1.3"; +inline constexpr std::string_view MCPP_VERSION = "2026.10.3.1"; } // namespace mcpp diff --git a/src/build/execute.cppm b/src/build/execute.cppm index ea1e65ef..d58b9ca2 100644 --- a/src/build/execute.cppm +++ b/src/build/execute.cppm @@ -39,6 +39,7 @@ import mcpp.source_kind; import mcpp.modgraph.scanner; import mcpp.toolchain.post_install; import mcpp.toolchain.stdmod; +import mcpp.version; // MCPP_VERSION — the engine a build record names import mcpp.xlings; import mcpp.xlings.subos_info; import mcpp.runtime.binding; @@ -147,7 +148,55 @@ std::string toolchain_request_identity() { return std::format("cli={};default={}", cli, machineDefault); } -struct BuildCacheEntry { +// THE ENGINE THAT WROTE A BUILD RECORD. +// +// build.ninja names the engine that wrote it by absolute path: the `$mcpp` +// rules (dyndep, stage, the BMI schedule), and the `__action` wrapper of an +// action that declares an environment or a stamp, all start that executable. +// The emitter said a version change regenerates the file, because the version +// is a fingerprint input, and the fast paths never computed a fingerprint: they +// matched an entry by target, profile, cache mode, features and toolchain +// request, and compared its recorded fingerprint with its own directory's name. +// After an upgrade that removed the previous install the entry was replayed, and +// every action started a program that no longer existed (#757). When the old +// executable still exists the failure is silent instead: a newer front end +// drives the actions, and the graph, of an older engine. +// +// So the record carries the engine itself, and the one admission predicate +// below compares it. The version and the executable's path are both kept: the +// same version at another path (a moved or reinstalled binary) leaves the same +// stale text in the graph that a new version leaves, and only the path says so. +export struct EngineIdentity { + std::string version; // MCPP_VERSION + std::string exe; // mcpp_exe_path(), the spelling the graph uses + bool operator==(const EngineIdentity&) const = default; +}; + +// The engine this process is: the one that would write a record now, and the +// one that would replay it. +export EngineIdentity running_engine() { + return {std::string(mcpp::MCPP_VERSION), mcpp_exe_path().string()}; +} + +// Why a record written by `recorded` is not replayed by `running`; nullopt when +// they are the same engine. `recorded` is empty for an entry written before the +// field existed, which is not the same answer as an engine that happens to +// match -- the entry then declines once and the next write records it, the +// discipline of every field of the entry below. +export std::optional +engine_declined_because(const std::optional& recorded, + const EngineIdentity& running) { + if (!recorded) return std::string("the recorded build predates the engine identity"); + if (recorded->version != running.version) + return std::format("the recorded build was written by mcpp {}, and this is mcpp {}", + recorded->version, running.version); + if (recorded->exe != running.exe) + return std::format("the recorded build was written by the engine at {}, and this one is at {}", + recorded->exe, running.exe); + return std::nullopt; +} + +export struct BuildCacheEntry { std::string targetTriple; // "" for default target std::string outputDir; std::string ninjaProgram; @@ -204,7 +253,14 @@ struct BuildCacheEntry { // invisible: ninja has no edge for a file that did not exist when // build.ninja was written, so `mcpp build` replays the stale graph and // reports success. See BuildContext::depSourceRoots. - std::vector depSourceRoots; + // + // Each root carries the extension tables of its own package, which is what + // the sweep classifies the root's files with (#756): the consumer's table + // says nothing about a provider's `.ixx`. Written as `depSources=`; the + // block of an engine that recorded the paths alone (`depSourceRoots=`) is + // read past and counts as absent, since a root without its tables cannot + // be swept. + std::vector depSourceRoots; // Was the block present at all? An EMPTY list is a legitimate answer — a // project with no path dependencies has none — so it cannot stand in for // "this cache predates the field", and the two need opposite treatment: @@ -271,9 +327,41 @@ struct BuildCacheEntry { // fast path matches every entry of its selection. std::string selection; std::string group; + // The engine that wrote the graph this entry records, absent for an entry + // written before the field (see EngineIdentity). Optional rather than an + // empty identity, because an engine whose version is the empty string does + // not exist and "no record" must not be representable as a value. + std::optional engine; }; -std::vector read_build_cache(const std::filesystem::path& projectRoot) { +// One list of extensions as a line of the record carries it: joined by a unit +// separator. An extension is a dot and a name, so it holds neither that +// character nor a tab or a newline, which the line's own structure relies on. +// The two directions are kept together so that the reader and the writer cannot +// disagree about the separator. +constexpr char kExtensionSeparator = '\x1f'; + +std::vector split_extensions(std::string_view joined) { + std::vector out; + while (!joined.empty()) { + const auto at = joined.find(kExtensionSeparator); + out.emplace_back(joined.substr(0, at)); + if (at == std::string_view::npos) break; + joined.remove_prefix(at + 1); + } + return out; +} + +std::string join_extensions(const std::vector& extensions) { + std::string out; + for (auto const& x : extensions) { + if (!out.empty()) out += kExtensionSeparator; + out += x; + } + return out; +} + +export std::vector read_build_cache(const std::filesystem::path& projectRoot) { auto path = projectRoot / kBuildCacheFile; std::ifstream f(path); if (!f) return {}; @@ -368,12 +456,38 @@ std::vector read_build_cache(const std::filesystem::path& proje // zero-length list and an absent block must not read the same. Absent // means the cache predates the field, and the fast path then declines // once so the next write records it. + // + // The block that listed the paths alone is read past and left + // unrecorded: its roots have no tables, so the entry declines once and + // the next write records both. if (haveNextLine && line.starts_with("depSourceRoots=")) { std::size_t n = 0; try { n = std::stoul(line.substr(15)); } catch (...) { n = 0; } - for (std::size_t i = 0; i < n && std::getline(f, line); ++i) - e.depSourceRoots.push_back(line); - e.depSourceRootsRecorded = true; + for (std::size_t i = 0; i < n && std::getline(f, line); ++i) {} + haveNextLine = static_cast(std::getline(f, line)); + } + // `depSources=`, then one line per root: the path, the module + // extensions and the device extensions of the root's package, separated + // by a tab, the extensions of one list by a unit separator. A line that + // is not of that shape makes the whole block unrecorded rather than + // read with a table nobody wrote. + if (haveNextLine && line.starts_with("depSources=")) { + std::size_t n = 0; + try { n = std::stoul(line.substr(11)); } catch (...) { n = 0; } + bool wellFormed = true; + std::vector roots; + for (std::size_t i = 0; i < n && std::getline(f, line); ++i) { + const auto t1 = line.find('\t'); + const auto t2 = t1 == std::string::npos ? t1 : line.find('\t', t1 + 1); + if (t2 == std::string::npos) { wellFormed = false; continue; } + roots.push_back({std::filesystem::path(line.substr(0, t1)), + split_extensions(std::string_view(line).substr(t1 + 1, t2 - t1 - 1)), + split_extensions(std::string_view(line).substr(t2 + 1))}); + } + if (wellFormed && roots.size() == n) { + e.depSourceRoots = std::move(roots); + e.depSourceRootsRecorded = true; + } haveNextLine = static_cast(std::getline(f, line)); } // Optional `runner=0|1` (#544). Absent ⇒ false; see the field. @@ -417,6 +531,15 @@ std::vector read_build_cache(const std::filesystem::path& proje e.group = line.substr(6); haveNextLine = static_cast(std::getline(f, line)); } + // `engine=`, a tab, and the executable's path; see + // EngineIdentity. The version holds no tab, so the path is everything + // after the first one. Absent, or without the tab, the entry declines + // once. + if (haveNextLine && line.starts_with("engine=")) { + if (const auto tab = line.find('\t'); tab != std::string::npos) + e.engine = EngineIdentity{line.substr(7, tab - 7), line.substr(tab + 1)}; + haveNextLine = static_cast(std::getline(f, line)); + } entries.push_back(std::move(e)); if (!haveNextLine || line.empty()) break; } @@ -425,8 +548,8 @@ std::vector read_build_cache(const std::filesystem::path& proje // Serialize the P3 format. Declared ahead of its single caller so the reader // and the writer of this file sit next to each other. -void write_build_cache_entries(const std::filesystem::path& path, - const std::vector& entries); +export void write_build_cache_entries(const std::filesystem::path& path, + const std::vector& entries); // `a, b` and `b a` are one request. Normalised on both sides of the comparison // — the entry stores this form and the fast path computes it — so a cache hit @@ -460,7 +583,7 @@ void write_build_cache(const std::filesystem::path& projectRoot, const std::string& profile = "", const std::string& cacheMode = "", const mcpp::platform::runtime::RuntimeBinding& runtimeBinding = {}, - std::vector depSourceRoots = {}, + std::vector depSourceRoots = {}, bool runnerDeclared = false, bool runTierPending = false, const std::string& features = {}, @@ -503,6 +626,9 @@ void write_build_cache(const std::filesystem::path& projectRoot, newEntry.xlingsPayloadsRecorded = true; newEntry.selection = selection; newEntry.group = group; + // The engine that wrote the graph is the one writing the record: both + // happen in this process, after the graph was emitted. + newEntry.engine = running_engine(); entries.insert(entries.begin(), std::move(newEntry)); // Trim to LRU capacity. @@ -541,8 +667,21 @@ void write_build_cache_entries(const std::filesystem::path& path, << '\n'; f << "profile=" << e.profile << '\n'; f << "cacheMode=" << e.cacheMode << '\n'; - f << "depSourceRoots=" << e.depSourceRoots.size() << '\n'; - for (auto& r : e.depSourceRoots) f << r << '\n'; + // A root whose path holds a tab or a line break cannot be written in + // this line format: read back, the path would be cut at the tab, and a + // directory that happened to exist under the shorter name would be + // swept with the wrong table. Such an entry records no roots, which + // reads as "predates the list" and declines the fast path, the safe + // direction. + const bool writable = std::ranges::none_of(e.depSourceRoots, [](auto const& r) { + return r.root.generic_string().find_first_of("\t\n\r") != std::string::npos; + }); + if (writable) { + f << "depSources=" << e.depSourceRoots.size() << '\n'; + for (auto& r : e.depSourceRoots) + f << r.root.generic_string() << '\t' << join_extensions(r.moduleExtensions) + << '\t' << join_extensions(r.deviceExtensions) << '\n'; + } f << "runner=" << (e.runnerDeclared ? 1 : 0) << '\n'; f << "runtier=" << (e.runTierPending ? 1 : 0) << '\n'; f << "features=" << e.features << '\n'; @@ -553,6 +692,8 @@ void write_build_cache_entries(const std::filesystem::path& path, f << "selection=" << e.selection << '\n'; f << "group=" << e.group << '\n'; } + if (e.engine) + f << "engine=" << e.engine->version << '\t' << e.engine->exe << '\n'; } } @@ -1012,13 +1153,7 @@ export int run_build_plan(BuildContext& ctx, bool verbose, bool no_cache, std::move(runTargets), runEnvKey, runEnvValue, ctx.profile, std::string(cache_mode_name(ctx.cacheMode)), ctx.plan.runtimeBinding, - [&] { - std::vector v; - v.reserve(ctx.depSourceRoots.size()); - for (auto const& r : ctx.depSourceRoots) - v.push_back(r.generic_string()); - return v; - }(), + ctx.depSourceRoots, // #544: the run fast path declines an entry whose // target has a runner declared — see the field. !choose_runner(ctx).tmpl.empty(), @@ -1169,12 +1304,21 @@ bool sources_newer_than(const std::filesystem::path& projectRoot, // The dependency's own manifest is swept too. A member that gains a target, a // `[modules] sources` glob or a `[build]` flag changes what the graph SHOULD // be, and none of that is visible from its source files' timestamps. -bool dep_sources_newer_than(const std::vector& depSourceRoots, - std::filesystem::file_time_type ninjaTime, - const mcpp::ExtensionTable& extTable) { +// +// EACH ROOT IS CLASSIFIED BY ITS OWN PACKAGE (#756). Which of a tree's files can +// change the graph depends on the extensions the package that owns the tree +// declares: a provider's `.ixx` is a module interface because the provider says +// so, whatever the consumer declares, and a consumer with only `.cpp` sources +// has no reason to. The sweep used the consumer's table for every root, so an +// edit to the provider's host module was classified as a file of no interest and +// replayed as "no work". The tables are the ones prepare recorded with the root. +export bool dep_sources_newer_than(const std::vector& depSourceRoots, + std::filesystem::file_time_type ninjaTime) { std::error_code ec; - for (auto const& rootStr : depSourceRoots) { - std::filesystem::path depRoot(rootStr); + for (auto const& dep : depSourceRoots) { + const std::filesystem::path& depRoot = dep.root; + const auto extTable = mcpp::extension_table_for(dep.moduleExtensions, + dep.deviceExtensions); // A dependency directory that has gone away is a resolution question, // not a staleness one: fall through to prepare_build, which reports it // with the dependency's name instead of a bare missing path. @@ -1583,14 +1727,146 @@ bool local_toolchain_unchanged(const std::filesystem::path& outputDir) { return true; } -// Why the project fast path declined, under `-v` (#734 E5). Each refusal names -// its condition, so a platform on which the fast path never serves shows which +// Why a fast path declined, under `-v` (#734 E5). Each refusal names its +// condition, so a platform on which the fast path never serves shows which // precondition it fails instead of only the slower build. std::optional fast_path_declined(std::string_view path, std::string_view why) { mcpp::log::verbose("fast-path", std::format("{} declined: {}", path, why)); return std::nullopt; } +// WHAT A RECORDED BUILD MUST SATISFY BEFORE ANY FAST PATH REPLAYS IT. +// +// `try_fast_build`, `try_fast_workspace_build` and `try_fast_run` each carried +// a list of declines of their own, 23, 30 and 30 of them, and a property added +// to one had to be added to the others by hand. #757 is the case in which no +// path received the property at all: the engine that wrote a graph was recorded +// nowhere and compared nowhere. The gates the three share are this one +// function, and a path keeps only what is its own: the selection a workspace +// command matches, a program to run, a runner, the run tier, and the project's +// own sources. A field added to the record is then checked in one place. +// +// The order is the cost of the question: the record itself first, then the +// files it names, then the sweeps of the trees it names, and last the snapshot +// of the artifacts, which reads the build directory. +export struct ReplayAsk { + // What the graph must have been planned for (`request_tag`): a project's + // features, or a workspace group's members. + std::string request; + // The manifest whose edit re-plans, and how a decline names it: the + // project's, or the workspace's. + std::filesystem::path manifest; + std::string_view manifestName = "mcpp.toml"; + // How a decline names the trees the recorded roots stand for: a project + // has path dependencies, a workspace has members besides. + std::string_view treesName = "a path dependency's"; + // The project's own tree, for the paths that sweep one: asked with the time + // of build.ninja, true when a source of it is newer. Empty for a path whose + // trees are all recorded roots, which is what a workspace's members are. + std::function projectSourcesNewer; +}; + +// What a path needs of an admitted record to go on: where the graph is, the +// ninja that drives it, and the artifacts' state before ninja runs, which is +// what tells afterwards whether an artifact was relinked. +export struct ReplayableBuild { + std::filesystem::path outputDir; + std::filesystem::path ninjaPath; + std::string ninjaProgram; + mcpp::build::runtime_validation::ArtifactSnapshot validated; +}; + +export std::expected +admit_recorded_build(const BuildCacheEntry& e, const ReplayAsk& ask) { + // The engine that wrote the graph is the engine that replays it (#757). + // First, because every later question is about a graph this engine did not + // write when this one fails. + if (auto why = engine_declined_because(e.engine, running_engine())) return std::unexpected(*why); + + // The runtime the recorded build ran under. An entry written before the + // immutable snapshot cannot say which environment the program needs, and + // running it with another is worse than not using the cache: it works once + // and then silently stops finding its runtime data (mcpp#352). + if (!e.runtimeBinding) return std::unexpected("the recorded build predates the runtime binding"); + if (e.runtimeEnvKey.empty()) + return std::unexpected("the recorded build predates the runtime environment key"); // regenerate build.ninja once + + // P1: verify fingerprint matches the outputDir basename. + const std::filesystem::path outputDir(e.outputDir); + if (!e.fingerprint.empty() && outputDir.filename().string() != e.fingerprint) + return std::unexpected("the recorded build directory is not the one for this fingerprint"); + + std::error_code ec; + const auto ninjaPath = outputDir / "build.ninja"; + if (!std::filesystem::exists(ninjaPath, ec)) return std::unexpected("build.ninja does not exist"); + + // The graph names the engine it runs, and that is asked of the graph, not + // only of the record: another engine's `--configure-only` rewrites the + // graph and leaves the record that names this one. + if (const auto named = mcpp::build::read_engine_binding(ninjaPath); + !named.empty() && named != mcpp_exe_path().generic_string()) + return std::unexpected(std::format("build.ninja runs another engine ({})", named)); + + // #407. Freshness is measured against the SOURCES, which says nothing + // about what kind of graph this is. `mcpp test` and + // `mcpp build --configure-only` write their plan -- dev-deps, test targets, + // `default` naming the test binaries and NOT the package's target -- into + // this same file, because the fingerprint covers neither input. Replaying + // that for a plain build linked the tests, never linked the target, and + // printed `Finished`; and a broken file under tests/ (never scanned here) + // failed a plain `mcpp build` outright. + if (!mcpp::build::is_plain_build_graph(ninjaPath)) + return std::unexpected("build.ninja was written by another mode (test, pack or a named target)"); + // What the graph was planned for (workspace design 2026-09-29 §3): the + // features no longer name the directory, so the graph says which it has. + if (mcpp::build::read_request(ninjaPath) != ask.request) + return std::unexpected("build.ninja was written for another request (features or workspace members)"); + + const auto ninjaTime = std::filesystem::last_write_time(ninjaPath, ec); + if (ec) return std::unexpected("the time of build.ninja cannot be read"); + + const auto runtimeTime = std::filesystem::last_write_time( + e.runtimeBinding->subosDir / ".xlings.json", ec); + if (ec || runtimeTime > ninjaTime) + return std::unexpected("the runtime's .xlings.json is newer than build.ninja"); + + const auto manifestTime = std::filesystem::last_write_time(ask.manifest, ec); + if (ec || manifestTime > ninjaTime) + return std::unexpected(std::format("{} is newer than build.ninja", ask.manifestName)); + + // mcpp#225: bounded + vcs/build-dir-excluded walk (see sources_newer_than) + // instead of a hand-rolled recursive_directory_iterator over src/. + if (ask.projectSourcesNewer && ask.projectSourcesNewer(ninjaTime)) + return std::unexpected("a project source, build.mcpp, a build-program input or a resource script is newer than build.ninja"); + + // A cache written before this field existed cannot say whether the build + // had `path` dependencies, and answering "assume none" is the wrong half of + // that guess: it would keep replaying a stale graph for exactly the projects + // the field was added for. Decline once; the write below records the list + // and every later invocation is fast again. It has to be every path: `mcpp + // run` reaches its binary through the same check, and a `run` that skipped + // it would execute an artifact built from a source set that no longer + // exists. + if (!e.depSourceRootsRecorded) + return std::unexpected("the recorded build predates the list of path-dependency roots and their extension tables"); + if (dep_sources_newer_than(e.depSourceRoots, ninjaTime)) + return std::unexpected(std::format("{} manifest or source is newer than build.ninja", ask.treesName)); + if (!xlings_payloads_present(e)) return std::unexpected("a recorded xlings payload is missing"); + if (!local_toolchain_unchanged(e.outputDir)) + return std::unexpected("a program of the toolchain named by path changed"); + + auto validated = mcpp::build::runtime_validation::validated_artifact_snapshot( + outputDir, *e.runtimeBinding); + if (!validated) return std::unexpected("no validated artifact snapshot is recorded for this build"); + + auto ninjaProgram = e.ninjaProgram; + // Legacy caches stored a shell-quoted path; execvp needs the raw path. + if (ninjaProgram.size() >= 2 && ninjaProgram.front() == '\'' + && ninjaProgram.back() == '\'') + ninjaProgram = ninjaProgram.substr(1, ninjaProgram.size() - 2); + return ReplayableBuild{outputDir, ninjaPath, std::move(ninjaProgram), std::move(*validated)}; +} + export std::optional try_fast_build(const std::filesystem::path& projectRoot, bool verbose, bool no_cache, std::string_view currentTarget = "") { @@ -1641,80 +1917,23 @@ export std::optional try_fast_build(const std::filesystem::path& projectRoo } } if (!match) return fast_path_declined("build", "no recorded build matches this request"); - if (!match->runtimeBinding) return fast_path_declined("build", "the recorded build predates the runtime binding"); - auto outputDirStr = match->outputDir; - auto ninjaProgram = match->ninjaProgram; - // Legacy caches stored a shell-quoted path; execvp needs the raw path. - if (ninjaProgram.size() >= 2 && ninjaProgram.front() == '\'' - && ninjaProgram.back() == '\'') - ninjaProgram = ninjaProgram.substr(1, ninjaProgram.size() - 2); - auto cachedFingerprint = match->fingerprint; + // The gates every fast path shares (see admit_recorded_build); what is left + // here is the project's own tree. + auto admitted = admit_recorded_build(*match, { + .request = mcpp::build::request_tag({}, want->features), + .manifest = projectRoot / "mcpp.toml", + .projectSourcesNewer = [&](std::filesystem::file_time_type ninjaTime) { + return sources_newer_than(projectRoot, ninjaTime, want->resourceScripts, + want->extTable); + }, + }); + if (!admitted) return fast_path_declined("build", admitted.error()); + const auto& outputDir = admitted->outputDir; + const auto& ninjaPath = admitted->ninjaPath; + const auto& validatedBefore = admitted->validated; auto runtimeEnvKey = match->runtimeEnvKey; auto runtimeEnvValue = match->runtimeEnvValue; - if (runtimeEnvKey.empty()) - return fast_path_declined("build", "the recorded build predates the runtime environment key"); // old cache entry; regenerate build.ninja once - - // P1: verify fingerprint matches the outputDir basename. - if (!cachedFingerprint.empty()) { - auto dirBasename = std::filesystem::path(outputDirStr).filename().string(); - if (dirBasename != cachedFingerprint) { - return fast_path_declined("build", "the recorded build directory is not the one for this fingerprint"); - } - } - - std::error_code ec; - std::filesystem::path outputDir(outputDirStr); - - auto ninjaPath = outputDir / "build.ninja"; - if (!std::filesystem::exists(ninjaPath, ec)) return fast_path_declined("build", "build.ninja does not exist"); - - // #407. Freshness is measured against the SOURCES, which says nothing - // about what kind of graph this is. `mcpp test` and - // `mcpp build --configure-only` write their plan — dev-deps, test targets, - // `default` naming the test binaries and NOT the package's target — into - // this same file, because the fingerprint covers neither input. Replaying - // that for a plain build linked the tests, never linked the target, and - // printed `Finished`; and a broken file under tests/ (never scanned here) - // failed a plain `mcpp build` outright. - if (!mcpp::build::is_plain_build_graph(ninjaPath)) return fast_path_declined("build", "build.ninja was written by another mode (test, pack or a named target)"); - // What the graph was planned for (workspace design 2026-09-29 §3): the - // features no longer name the directory, so the graph says which it has. - if (mcpp::build::read_request(ninjaPath) != mcpp::build::request_tag({}, want->features)) - return fast_path_declined("build", "build.ninja was written for another request (features or workspace members)"); - - auto ninjaTime = std::filesystem::last_write_time(ninjaPath, ec); - if (ec) return fast_path_declined("build", "the time of build.ninja cannot be read"); - - auto runtimeManifest = match->runtimeBinding->subosDir / ".xlings.json"; - auto runtimeTime = std::filesystem::last_write_time(runtimeManifest, ec); - if (ec || runtimeTime > ninjaTime) return fast_path_declined("build", "the runtime's .xlings.json is newer than build.ninja"); - - // Check mcpp.toml - auto tomlPath = projectRoot / "mcpp.toml"; - auto tomlTime = std::filesystem::last_write_time(tomlPath, ec); - if (ec || tomlTime > ninjaTime) return fast_path_declined("build", "mcpp.toml is newer than build.ninja"); - - // mcpp#225: bounded + vcs/build-dir-excluded walk (see sources_newer_than) - // instead of a hand-rolled recursive_directory_iterator over src/. - if (sources_newer_than(projectRoot, ninjaTime, want->resourceScripts, - want->extTable)) return fast_path_declined("build", "a project source, build.mcpp, a build-program input or a resource script is newer than build.ninja"); - // A cache written before this field existed cannot say whether the build - // had `path` dependencies, and answering "assume none" is the wrong - // half of that guess: it would keep replaying a stale graph for exactly - // the projects the field was added for. Decline once; the write below - // records the list and every later invocation is fast again. - if (!match->depSourceRootsRecorded) return fast_path_declined("build", "the recorded build predates the list of path-dependency roots"); - if (dep_sources_newer_than(match->depSourceRoots, ninjaTime, want->extTable)) - return fast_path_declined("build", "a path dependency's manifest or source is newer than build.ninja"); - if (!xlings_payloads_present(*match)) return fast_path_declined("build", "a recorded xlings payload is missing"); - if (!local_toolchain_unchanged(match->outputDir)) - return fast_path_declined("build", "a program of the toolchain named by path changed"); - - auto validatedBefore = - mcpp::build::runtime_validation::validated_artifact_snapshot( - outputDir, *match->runtimeBinding); - if (!validatedBefore) return fast_path_declined("build", "no validated artifact snapshot is recorded for this build"); // All inputs are older than build.ninja → fast-path: just run ninja. // C1: this configuration is confirmed current, so the root database is @@ -1723,12 +1942,12 @@ export std::optional try_fast_build(const std::filesystem::path& projectRoo // see it back. restore_root_compile_commands(projectRoot, outputDir); std::chrono::milliseconds elapsed{}; - auto rc = run_ninja_fast(ninjaProgram, outputDir, ninjaPath, verbose, + auto rc = run_ninja_fast(admitted->ninjaProgram, outputDir, ninjaPath, verbose, runtimeEnvKey, runtimeEnvValue, &elapsed); if (!rc) return fast_path_declined("build", "ninja reported a stale graph"); if (*rc != 0) return rc; if (!mcpp::build::runtime_validation::artifact_snapshot_unchanged( - *validatedBefore)) + validatedBefore)) return fast_path_declined("build", "ninja relinked an artifact, whose closure the full path validates"); // relinked: full path reconstructs + validates closure // The descriptor the plan recorded (revision 3, §7.3); empty for a record @@ -1794,43 +2013,18 @@ export std::optional try_fast_workspace_build( && e.toolchainRecorded && e.toolchainRequest == want->toolchainRequest && e.selection == selection && e.group == group) { match = &e; break; } if (!match) return fast_path_declined("workspace", "no recorded build matches this selection"); - if (!match->runtimeBinding || match->runtimeEnvKey.empty()) - return fast_path_declined("workspace", "the recorded build predates the runtime binding"); - const std::filesystem::path outputDir(match->outputDir); - if (!match->fingerprint.empty() && outputDir.filename().string() != match->fingerprint) - return fast_path_declined("workspace", "the recorded build directory is not the one for this fingerprint"); - const auto ninjaPath = outputDir / "build.ninja"; - if (!std::filesystem::exists(ninjaPath, ec)) - return fast_path_declined("workspace", "build.ninja does not exist"); - if (!mcpp::build::is_plain_build_graph(ninjaPath)) - return fast_path_declined("workspace", "build.ninja was written by another mode (test, pack or a named target)"); - if (mcpp::build::read_request(ninjaPath) != mcpp::build::request_tag(group, {})) - return fast_path_declined("workspace", "build.ninja was written for another request (features or workspace members)"); - const auto ninjaTime = std::filesystem::last_write_time(ninjaPath, ec); - if (ec) return fast_path_declined("workspace", "the time of build.ninja cannot be read"); - const auto runtimeTime = std::filesystem::last_write_time( - match->runtimeBinding->subosDir / ".xlings.json", ec); - if (ec || runtimeTime > ninjaTime) - return fast_path_declined("workspace", "the runtime's .xlings.json is newer than build.ninja"); - if (wsToml > ninjaTime) - return fast_path_declined("workspace", "the workspace's mcpp.toml is newer than build.ninja"); - if (!match->depSourceRootsRecorded - || dep_sources_newer_than(match->depSourceRoots, ninjaTime, want->extTable)) - return fast_path_declined("workspace", "a member's or a path dependency's manifest or source is newer than build.ninja"); - if (!xlings_payloads_present(*match)) - return fast_path_declined("workspace", "a recorded xlings payload is missing"); - if (!local_toolchain_unchanged(match->outputDir)) - return fast_path_declined("workspace", "a program of the toolchain named by path changed"); - auto validated = mcpp::build::runtime_validation::validated_artifact_snapshot( - outputDir, *match->runtimeBinding); - if (!validated) - return fast_path_declined("workspace", "no validated artifact snapshot is recorded for this build"); - auto ninjaProgram = match->ninjaProgram; - if (ninjaProgram.size() >= 2 && ninjaProgram.front() == '\'' - && ninjaProgram.back() == '\'') - ninjaProgram = ninjaProgram.substr(1, ninjaProgram.size() - 2); - ready.push_back({outputDir, ninjaProgram, match->runtimeEnvKey, - match->runtimeEnvValue, std::move(*validated)}); + // The gates every fast path shares (see admit_recorded_build). A + // workspace has no project tree of its own to sweep: its members are + // recorded roots, each classified by its own package. + auto admitted = admit_recorded_build(*match, { + .request = mcpp::build::request_tag(group, {}), + .manifest = wsRoot / "mcpp.toml", + .manifestName = "the workspace's mcpp.toml", + .treesName = "a member's or a path dependency's", + }); + if (!admitted) return fast_path_declined("workspace", admitted.error()); + ready.push_back({admitted->outputDir, admitted->ninjaProgram, match->runtimeEnvKey, + match->runtimeEnvValue, std::move(admitted->validated)}); profile = want->profile; } @@ -1941,33 +2135,6 @@ std::optional try_fast_run(const std::filesystem::path& projectRoot, // declared tool absent. prepare_build provisions the difference. if (match->runTierPending) return fast_path_declined("run", "the run tier is not yet decided"); - auto outputDirStr = match->outputDir; - auto ninjaProgram = match->ninjaProgram; - // Legacy caches stored a shell-quoted path; execvp needs the raw path. - if (ninjaProgram.size() >= 2 && ninjaProgram.front() == '\'' - && ninjaProgram.back() == '\'') - ninjaProgram = ninjaProgram.substr(1, ninjaProgram.size() - 2); - if (match->runtimeEnvKey.empty()) - return fast_path_declined("run", "the recorded build predates the runtime environment key"); // old cache entry; go through prepare_build once - // Written before this mcpp knew about subos environments (mcpp#352). Taking - // the fast path here would run the program without them -- which is the - // defect this field exists to fix, surviving an upgrade. - // - // It survives it for a long time, too: the fast path's identity is the - // profile, the cache mode and the resource list, and its fingerprint check - // compares a cached entry against ITSELF. Neither notices that a different - // mcpp wrote the entry, so without this line an upgraded mcpp would reuse a - // pre-upgrade build until something else happened to invalidate it. Measured - // on a real upgrade from 2026.8.7.1, not reasoned about. - if (!match->runtimeBinding) - return fast_path_declined("run", "the recorded build predates the runtime binding"); // predates the immutable snapshot; rebuild once - - // P1: verify fingerprint matches the outputDir basename. - if (!match->fingerprint.empty()) { - auto dirBasename = std::filesystem::path(outputDirStr).filename().string(); - if (dirBasename != match->fingerprint) return fast_path_declined("run", "the recorded build directory is not the one for this fingerprint"); - } - // Locate the requested run-target before doing any filesystem freshness // work — an unrecognized name falls back to prepare_build, which gives // a proper "no binary target 'x' found" error instead of a silent miss. @@ -1979,55 +2146,35 @@ std::optional try_fast_run(const std::filesystem::path& projectRoot, } if (!chosen) return fast_path_declined("run", "the requested program is not among the recorded ones"); - std::error_code ec; - std::filesystem::path outputDir(outputDirStr); - auto ninjaPath = outputDir / "build.ninja"; - if (!std::filesystem::exists(ninjaPath, ec)) return fast_path_declined("run", "build.ninja does not exist"); - // #407, same reason as try_fast_build: a test-shaped graph does not build - // the run target at all, so running ninja against it would report success - // and then exec a stale (or absent) binary. - if (!mcpp::build::is_plain_build_graph(ninjaPath)) return fast_path_declined("run", "build.ninja was written by another mode (test, pack or a named target)"); - // What the graph was planned for (workspace design 2026-09-29 §3): the - // features no longer name the directory, so the graph says which it has. - if (mcpp::build::read_request(ninjaPath) != mcpp::build::request_tag({}, want->features)) - return fast_path_declined("run", "build.ninja was written for another request (features or workspace members)"); - auto ninjaTime = std::filesystem::last_write_time(ninjaPath, ec); - if (ec) return fast_path_declined("run", "the time of build.ninja cannot be read"); - - auto runtimeManifest = match->runtimeBinding->subosDir / ".xlings.json"; - auto runtimeTime = std::filesystem::last_write_time(runtimeManifest, ec); - if (ec || runtimeTime > ninjaTime) return fast_path_declined("run", "the runtime's .xlings.json is newer than build.ninja"); - - auto tomlPath = projectRoot / "mcpp.toml"; - auto tomlTime = std::filesystem::last_write_time(tomlPath, ec); - if (ec || tomlTime > ninjaTime) return fast_path_declined("run", "mcpp.toml is newer than build.ninja"); - - if (sources_newer_than(projectRoot, ninjaTime, want->resourceScripts, - want->extTable)) return fast_path_declined("run", "a project source, build.mcpp, a build-program input or a resource script is newer than build.ninja"); - // Same gate as try_fast_build's, and it has to be BOTH places: `mcpp run` - // reaches its binary through this path, so a `run` that skipped the check - // would execute an artifact built from a source set that no longer exists. - if (!match->depSourceRootsRecorded) return fast_path_declined("run", "the recorded build predates the list of path-dependency roots"); - if (dep_sources_newer_than(match->depSourceRoots, ninjaTime, want->extTable)) - return fast_path_declined("run", "a path dependency's manifest or source is newer than build.ninja"); - if (!xlings_payloads_present(*match)) return fast_path_declined("run", "a recorded xlings payload is missing"); - if (!local_toolchain_unchanged(match->outputDir)) - return fast_path_declined("run", "a program of the toolchain named by path changed"); - - auto validatedBefore = - mcpp::build::runtime_validation::validated_artifact_snapshot( - outputDir, *match->runtimeBinding); - if (!validatedBefore) return fast_path_declined("run", "no validated artifact snapshot is recorded for this build"); + // The gates every fast path shares (see admit_recorded_build), for the + // reason `mcpp run` is among the paths that need them: it execs the + // artifact itself, so an entry that cannot say which engine wrote it, or + // which runtime the program needs, must not be taken. What is left here is + // the project's own tree, as for try_fast_build; a graph that is not the + // plain build's does not build the run target at all (#407), so ninja + // against it would report success and then exec a stale or absent binary. + auto admitted = admit_recorded_build(*match, { + .request = mcpp::build::request_tag({}, want->features), + .manifest = projectRoot / "mcpp.toml", + .projectSourcesNewer = [&](std::filesystem::file_time_type ninjaTime) { + return sources_newer_than(projectRoot, ninjaTime, want->resourceScripts, + want->extTable); + }, + }); + if (!admitted) return fast_path_declined("run", admitted.error()); + const auto& outputDir = admitted->outputDir; + const auto& ninjaPath = admitted->ninjaPath; + const auto& validatedBefore = admitted->validated; // Fresh → run ninja (picks up any incremental object/link work) then // exec the cached exe path directly. C1, same reason as try_fast_build's. restore_root_compile_commands(projectRoot, outputDir); - auto rc = run_ninja_fast(ninjaProgram, outputDir, ninjaPath, /*verbose=*/false, + auto rc = run_ninja_fast(admitted->ninjaProgram, outputDir, ninjaPath, /*verbose=*/false, match->runtimeEnvKey, match->runtimeEnvValue); if (!rc) return fast_path_declined("run", "ninja reported a stale graph"); if (*rc != 0) return kRunBuildFailed; if (!mcpp::build::runtime_validation::artifact_snapshot_unchanged( - *validatedBefore)) + validatedBefore)) return fast_path_declined("run", "ninja relinked an artifact, whose closure the full path validates"); // never execute an artifact not validated for this binding auto exe = outputDir / chosen->second; diff --git a/src/build/graph_shape.cppm b/src/build/graph_shape.cppm index 16a0be93..24a2c353 100644 --- a/src/build/graph_shape.cppm +++ b/src/build/graph_shape.cppm @@ -133,6 +133,36 @@ std::string read_request(const std::filesystem::path& ninjaPath) { return {}; } +// The engine this graph runs as `$mcpp`, the binding every graph carries (see +// the emitter in ninja_backend.cppm), unescaped and in generic form; empty when +// the file cannot be read or binds no engine. The record of a build names the +// engine that wrote the record, and that is not always the one that wrote the +// graph: another engine's `--configure-only` rewrites build.ninja and leaves the +// record as it was. The graph is the authority on what it will run. +std::string read_engine_binding(const std::filesystem::path& ninjaPath) { + std::ifstream input(ninjaPath); + if (!input) return {}; + std::string line; + while (std::getline(input, line)) { + if (!line.starts_with("mcpp ") && !line.starts_with("mcpp=")) continue; + const auto eq = line.find('='); + if (eq == std::string::npos) continue; + std::string_view value = std::string_view(line).substr(eq + 1); + while (!value.empty() && value.front() == ' ') value.remove_prefix(1); + while (!value.empty() && (value.back() == '\r' || value.back() == ' ')) + value.remove_suffix(1); + // The inverse of escape_ninja_path: `$$`, `$:` and `$ ` stand for the + // character after the dollar. + std::string out; + for (std::size_t i = 0; i < value.size(); ++i) { + if (value[i] == '$' && i + 1 < value.size()) { out += value[++i]; continue; } + out += value[i]; + } + return out; + } + return {}; +} + // Read the shape back. `nullopt` means "this file does not say" — a build.ninja // written before this line existed, an unreadable file, or something that is // not a mcpp graph at all. Callers must treat that as a MISS, never as diff --git a/src/build/ninja_backend.cppm b/src/build/ninja_backend.cppm index fbf3822e..bc15b36b 100644 --- a/src/build/ninja_backend.cppm +++ b/src/build/ninja_backend.cppm @@ -283,10 +283,20 @@ std::optional ninja_encoding_mismatch(std::string_view reported, unsigned processCodePage, std::string_view ninjaProgram); +// Where this engine is: the absolute path the emitter writes into build.ninja +// for the `$mcpp` rules and the `__action` wrapper. It is exported so that the +// record of a build (`.build_cache`) states the engine by the same function the +// graph does; two spellings of "where am I" would be one decision derived twice. +std::filesystem::path mcpp_exe_path(); + } // namespace mcpp::build namespace mcpp::build { +std::filesystem::path mcpp_exe_path() { + return mcpp::platform::fs::self_exe_path(); +} + namespace { // U+FEFF in UTF-8. The response files of the MSVC tools begin with it (#693). @@ -655,10 +665,6 @@ bool dyndep_mode_enabled() { return !(sv == "0" || sv == "off" || sv == "false"); } -std::filesystem::path mcpp_exe_path() { - return mcpp::platform::fs::self_exe_path(); -} - bool is_c_source(const mcpp::build::CompileUnit& cu) { return cu.kind == mcpp::SourceKind::C; } @@ -3504,10 +3510,12 @@ std::string emit_ninja_string(const BuildPlan& plan, std::string* placements, // the same decision derived twice. // // The absolute path is baked into build.ninja, as every other tool - // path in it is. A version change regenerates the file (the version - // is in the fingerprint); moving the binary without changing its - // version would leave a stale path here, exactly as it would for - // the compiler. + // path in it is. A version change moves the build directory (the + // version is in the fingerprint), but that is not what keeps a + // stale path out of a replayed graph: the fast paths compute no + // fingerprint. They compare the engine the build record names (its + // version and `mcpp_exe_path()`, see `admit_recorded_build`), and a + // different engine plans the graph again (#757). const auto q = [](const std::string& v) { return shell_quote_arg(escape_ninja_chars(v)); }; diff --git a/src/build/prepare.cppm b/src/build/prepare.cppm index 4342a38d..21578941 100644 --- a/src/build/prepare.cppm +++ b/src/build/prepare.cppm @@ -412,6 +412,24 @@ export struct PlanNote { std::string path; }; +// A source tree outside the project that a build read, with the classification +// of its own package. +// +// WHICH FILES OF A TREE CAN CHANGE THE GRAPH IS A PROPERTY OF THE PACKAGE THAT +// OWNS THEM. `.ixx` is a module interface only where a package declares it in +// `[build] module_extensions` (the built-in table is `.cppm`), and a provider +// declares it for its own sources without the consumer having a `.ixx` file at +// all. The freshness sweep used the consumer's table for every tree and so +// classified a provider's `.ixx` as a file of no interest (#756). The tables +// are recorded with the root, from the package's effective manifest, so the +// sweep asks each tree the question its owner answers. +export struct DepSourceRoot { + std::filesystem::path root; + std::vector moduleExtensions; + std::vector deviceExtensions; + bool operator==(const DepSourceRoot&) const = default; +}; + export struct BuildContext { // THE PER-MACHINE JOB DEFAULT, carried so it is read once. // @@ -451,7 +469,7 @@ export struct BuildContext { // cannot resolve dependencies without becoming prepare_build, and a second // derivation would drift from the first exactly when a resolution rule // changes. Written into `.build_cache`; see BuildCacheEntry::depSourceRoots. - std::vector depSourceRoots; + std::vector depSourceRoots; // `/bin` and then `` of every installed `[xlings] deps` // payload of the runtime-owner manifest, in declaration order (#544); the // pair comes from runner_lookup::payload_search_dirs. Read by diff --git a/src/build/prepare/plan.cpp b/src/build/prepare/plan.cpp index 52306850..ad2f93a1 100644 --- a/src/build/prepare/plan.cpp +++ b/src/build/prepare/plan.cpp @@ -138,7 +138,14 @@ static std::expected step13_source_packages(PrepareState& sta { std::vector owned = state.storeRoots; owned.push_back(mcpp::home::root()); - std::vector roots; + std::vector roots; + auto rootOf = [&](std::size_t i, std::filesystem::path n) { + const auto& bc = state.packages[i].manifest.buildConfig; + return DepSourceRoot{std::move(n), bc.moduleExtensions, bc.deviceExtensions}; + }; + auto recorded = [&](const std::filesystem::path& n) { + return std::ranges::any_of(roots, [&](const DepSourceRoot& r) { return r.root == n; }); + }; // The same enumeration answers a second reader: which packages were // read from an editable tree, with their source globs (the build // database lists them as the inputs that change the plan). @@ -161,8 +168,8 @@ static std::expected step13_source_packages(PrepareState& sta normalized, state.packages[i].manifest.modules.sources}); if (i == 0 || normalized == state.root->lexically_normal()) continue; - if (std::find(roots.begin(), roots.end(), normalized) == roots.end()) - roots.push_back(std::move(normalized)); + if (!recorded(normalized)) + roots.push_back(rootOf(i, std::move(normalized))); } // A workspace plan's members are its projects: their trees are what // the fast path sweeps, the workspace's own package included. @@ -170,8 +177,8 @@ static std::expected step13_source_packages(PrepareState& sta for (std::size_t i = 1; i < state.packages.size(); ++i) { if (!state.packages[i].selectedMember) continue; auto normalized = state.packages[i].root.lexically_normal(); - if (std::find(roots.begin(), roots.end(), normalized) == roots.end()) - roots.push_back(normalized); + if (!recorded(normalized)) + roots.push_back(rootOf(i, std::move(normalized))); } ctx.depSourceRoots = std::move(roots); } diff --git a/tests/e2e/741_c_abi_environment_declared_by_the_c_library.sh b/tests/e2e/741_c_abi_environment_declared_by_the_c_library.sh index e2abb14a..5514439a 100755 --- a/tests/e2e/741_c_abi_environment_declared_by_the_c_library.sh +++ b/tests/e2e/741_c_abi_environment_declared_by_the_c_library.sh @@ -63,6 +63,12 @@ cabi_probe_asm_marker: ret EOF +# THE ROOT IS A LIBRARY, SO NOTHING LINKS. With the inferred binary target the +# build linked `cabi-probe.exe` for x86_64-windows-gnu, which needs a MinGW +# runtime: it passed on a machine with the host's mingw-w64 installed and failed +# on every CI runner, and nothing noticed because no runner ever ran this test +# (it declared `llvm`, which no runner granted, until 2026-10-02). Its subject +# is the compile lines, which a library has as well. cat > mcpp.toml <<'EOF' [package] name = "cabi-probe" @@ -72,6 +78,10 @@ version = "0.1.0" fakemusl = { path = "fakemusl" } openkalwin = { path = "openkalwin" } +[targets.cabi-probe] +kind = "lib" +sources = ["src/*.cpp", "src/*.S"] + [build] allow_host_libs = true EOF diff --git a/tests/e2e/878_an_upgraded_engine_does_not_replay_the_old_graph.sh b/tests/e2e/878_an_upgraded_engine_does_not_replay_the_old_graph.sh new file mode 100755 index 00000000..b4bbc39c --- /dev/null +++ b/tests/e2e/878_an_upgraded_engine_does_not_replay_the_old_graph.sh @@ -0,0 +1,125 @@ +#!/usr/bin/env bash +# requires: unix-shell +# 878_an_upgraded_engine_does_not_replay_the_old_graph.sh -- #757. +# +# build.ninja names the engine that wrote it by absolute path: the `$mcpp` rules +# (dyndep, stage, the BMI schedule) and the `__action` wrapper start that +# executable. The emitter assumed that a new engine plans the graph again, "the +# version is in the fingerprint", and no fast path computes a fingerprint: each +# matched the recorded entry by target, profile, cache mode, features and +# toolchain request, and replayed it. After an upgrade that removed the previous +# install, `mcpp build` ran the old graph and every `stage` action failed +# because the program it names was gone; with the old install still present the +# new front end drove the actions of an older engine without saying so. +# +# The record of a build now carries the engine that wrote it, its version and the +# path of its executable (the path the graph uses), and one predicate +# (`admit_recorded_build`) compares it for every fast path. +# +# One program is built with an engine copied to one path, and the same binary is +# then run from another path, which is what a reinstall into another directory +# looks like to the record (the version is unchanged, so the path is what +# differs, and it is the part that breaks the graph). +# +# A the engine is moved and nothing else changes: the build succeeds, plans +# the graph again, and no command of the new graph names the old path; +# B after that one build the next is replayed by the fast path again, and a +# second move together with an edit of a deployed file (an action that +# starts the engine has to run) succeeds with the new content in place; +# C a record written before the engine was recorded declines once (under -v +# the reason is printed), and the build after it is replayed; +# D `mcpp run` is asked the same question, and runs the program; +# E a graph that another engine rewrote while the record still names this +# one (what `--configure-only` from another install does) is not replayed: +# the graph's own `$mcpp` binding is asked as well. +set -e + +TMP=$(mktemp -d) +trap 'rm -rf "$TMP"' EXIT +fail() { echo "FAIL: $1"; shift; for f in "$@"; do echo "--- $f ---"; cat "$f" 2>/dev/null; done; exit 1; } +MCPP="${MCPP:-mcpp}" +export NO_COLOR=1 +# The engine is copied out of the directory a home would be derived from, so the +# home the build uses is stated rather than found from the copy's location. +export MCPP_HOME="${MCPP_HOME:-$HOME/.mcpp}" + +EXE="" +case "$(uname -s)" in MINGW*|MSYS*|CYGWIN*) EXE=".exe" ;; esac + +replayed() { # replayed : the build was served by the fast path + ! grep -q "declined" "$1" && ! grep -q "Resolving toolchain" "$1" +} +names() { # names

: does a graph under target/ name the directory + grep -rl --include=build.ninja -F -- "$1" target 2>/dev/null | grep -q . +} + +mkdir -p "$TMP/engine-one" "$TMP/engine-two" "$TMP/engine-three" "$TMP/app/src" "$TMP/app/assets" +cp "$MCPP" "$TMP/engine-one/mcpp$EXE" +cd "$TMP/app" +cat > mcpp.toml <<'TOML' +[package] +name = "upgrade878" +version = "0.1.0" + +[runtime] +deploy = [ { from = "assets/data.txt", to = "." } ] +TOML +printf '#include \nint main() { std::puts("hello"); return 0; }\n' > src/main.cpp +printf 'one\n' > assets/data.txt +deployed() { find target -name data.txt -path '*/bin/*' | head -1; } + +ONE="$TMP/engine-one/mcpp$EXE" +"$ONE" build > b0.log 2>&1 || fail "the first build failed" b0.log +"$ONE" build -v > b1.log 2>&1 || fail "the warm build failed" b1.log +replayed b1.log || fail "the warm build was not replayed, so nothing below is measured against the fast path" b1.log +names "engine-one" || fail "the graph does not name the engine that wrote it, so the criterion has nothing to detect" b0.log +grep -q '^engine=' target/.build_cache || fail "the record does not name the engine that wrote it" target/.build_cache + +# A. The engine moves; no source, manifest or deployed file changes. +mv "$TMP/engine-one/mcpp$EXE" "$TMP/engine-two/mcpp$EXE" +TWO="$TMP/engine-two/mcpp$EXE" +"$TWO" build -v > a1.log 2>&1 || fail "A: the build with the engine at its new path failed" a1.log +names "engine-one" && fail "A: a graph still names the path the engine was moved from" a1.log +names "engine-two" || fail "A: the regenerated graph does not name the engine that built it" a1.log +grep -q "fast-path: build declined" a1.log || fail "A: the build did not say that the fast path declined" a1.log + +# B. Replayed again, and an action that starts the engine runs. +"$TWO" build -v > b2.log 2>&1 || fail "B: the build after the regeneration failed" b2.log +replayed b2.log || fail "B: the build after the regeneration was not replayed by the fast path" b2.log +mv "$TMP/engine-two/mcpp$EXE" "$TMP/engine-three/mcpp$EXE" +THREE="$TMP/engine-three/mcpp$EXE" +sleep 1.1 +printf 'two\n' > assets/data.txt +"$THREE" build > b3.log 2>&1 || fail "B: the build after the second move and an edit of a deployed file failed" b3.log +[ "$(cat "$(deployed)")" = "two" ] || fail "B: the edit of the deployed file did not reach the output directory" b3.log +names "engine-two" && fail "B: a graph still names the path the engine was moved from" + +# C. A record written before the engine was recorded declines once. +grep -v '^engine=' target/.build_cache > target/.build_cache.aged +mv target/.build_cache.aged target/.build_cache +"$THREE" build -v > c1.log 2>&1 || fail "C: the build with an aged record failed" c1.log +grep -q "predates the engine identity" c1.log || fail "C: the decline did not name the missing engine identity" c1.log +grep -q '^engine=' target/.build_cache || fail "C: the build did not record the engine" target/.build_cache +"$THREE" build -v > c2.log 2>&1 || fail "C: the build after the aged record failed" c2.log +replayed c2.log || fail "C: the record written after the decline was not replayed" c2.log + +# D. `mcpp run`, whose fast path is a different function, asks the same question. +mv "$TMP/engine-three/mcpp$EXE" "$TMP/engine-one/mcpp$EXE" +"$ONE" run -v > d1.log 2>&1 || fail "D: the run with the engine at another path failed" d1.log +grep -q '^hello$' d1.log || fail "D: the program did not run" d1.log +grep -q "fast-path: run declined" d1.log || fail "D: the run did not say that the fast path declined" d1.log +names "engine-three" && fail "D: a graph still names the path the engine was moved from" + +# E. The graph names its engine itself. Another engine rewrote the graph and +# left the record: the fast path asks the graph, declines, and plans again. +"$ONE" build > e0.log 2>&1 || fail "E: the build before the rewrite failed" e0.log +for g in $(find target -name build.ninja); do + awk -v p="$TMP/engine-gone/mcpp$EXE" '/^mcpp *=/ { print "mcpp = " p; next } { print }' \ + "$g" > "$g.rewritten" && mv "$g.rewritten" "$g" +done +names "engine-gone" || fail "E: the rewrite did not take" +"$ONE" build -v > e1.log 2>&1 || fail "E: the build of a graph another engine wrote failed" e1.log +grep -q "runs another engine" e1.log || fail "E: the decline did not name the graph's engine" e1.log +names "engine-gone" && fail "E: a graph still names the engine that rewrote it" + +echo "OK" diff --git a/tests/e2e/879_a_providers_module_extension_reaches_the_fast_path.sh b/tests/e2e/879_a_providers_module_extension_reaches_the_fast_path.sh new file mode 100755 index 00000000..b97fbc8d --- /dev/null +++ b/tests/e2e/879_a_providers_module_extension_reaches_the_fast_path.sh @@ -0,0 +1,186 @@ +#!/usr/bin/env bash +# requires: unix-shell +# 879_a_providers_module_extension_reaches_the_fast_path.sh -- #756. +# +# A host module that a `path` build-dependency provides is compiled into the +# consumer's build program, which no edge of the consumer's build.ninja names, +# so the fast path has to see an edit to it itself. It sweeps the dependency's +# tree and decides which files can change the graph with an extension table. It +# used the CONSUMER's table, so a `.ixx` that only the provider declares was +# classified as a file of no interest, the edit was replayed as "no work", and +# the build program kept the old module. The consumer cannot be asked to declare +# `.ixx` as a workaround: it has no `.ixx` source, and the dead-entry warning it +# would then receive is correct. +# +# Classification belongs to the package that owns the file. Each root the build +# cache records now carries its own package's `module_extensions` and +# `device_extensions`, and the sweep classifies the files below a root with that +# root's table. +# +# The same provider and consumer are built as a project (the project fast path +# and `mcpp run`'s) and as a workspace member (the workspace fast path). +# +# A after a build and a warm build, an edit to `rules.ixx` re-runs the +# consumer's build program: the program prints the new value, in all three +# forms; +# B after one build that confirmed the edit the next build is replayed again +# (under -v: no "declined", no "Resolving toolchain"), and the next edit is +# seen as well; +# C the consumer declares nothing about `.ixx`, and no build prints the +# dead-entry warning; +# D a record that lists the roots without their tables declines once, and +# the build after it is replayed. +set -e + +TMP=$(mktemp -d) +trap 'rm -rf "$TMP"' EXIT +fail() { echo "FAIL: $1"; shift; for f in "$@"; do echo "--- $f ---"; cat "$f" 2>/dev/null; done; exit 1; } +MCPP="${MCPP:-mcpp}" +export NO_COLOR=1 + +EXE="" +case "$(uname -s)" in MINGW*|MSYS*|CYGWIN*) EXE=".exe" ;; esac +bin_of() { find "$1" -path "*/bin/*" -name "$2$EXE" -type f | head -1; } + +# A provider that declares `.ixx` and has one source; a consumer with only a +# `.cpp` source whose build program imports the provider's module. +provider() { + mkdir -p "$1" + cat > "$1/mcpp.toml" <<'TOML' +[package] +namespace = "repro" +name = "rules" +version = "0.1.0" +standard = "c++23" + +[lib] +path = "rules.ixx" + +[build] +sources = ["rules.ixx"] +module_extensions = [".ixx"] +TOML + value "$1" 1 +} +value() { + printf 'export module repro.rules;\n\nexport const char* repro_definition() {\n return "REPRO_VALUE=%s";\n}\n' "$2" > "$1/rules.ixx" +} +consumer() { + local dir="$1" + mkdir -p "$dir" + cat > "$dir/mcpp.toml" <<'TOML' +[package] +namespace = "repro" +name = "app" +version = "0.1.0" +standard = "c++23" + +[build] +sources = ["main.cpp"] + +[build-dependencies] +"repro.rules" = { path = "../rules", host-module = true } + +[targets.app] +kind = "bin" +main = "main.cpp" +TOML + cat > "$dir/build.mcpp" <<'CPP' +import mcpp; +import repro.rules; + +int main() { + mcpp::define(repro_definition()); + return 0; +} +CPP + cat > "$dir/main.cpp" <<'CPP' +#include + +int main() { + std::printf("REPRO_VALUE=%d\n", REPRO_VALUE); +} +CPP +} + +replayed() { # replayed : the build was served by the fast path + ! grep -q "declined" "$1" && ! grep -q "Resolving toolchain" "$1" +} + +cd "$TMP" + +# ── A project: the consumer builds alone, `mcpp run` runs its program ─────── +provider proj/rules +consumer proj/app +cd proj/app +run_value() { "$MCPP" run 2>/dev/null | grep '^REPRO_VALUE=' | tail -1; } + +"$MCPP" build > p0.log 2>&1 || fail "project: the first build failed" p0.log +[ "$(run_value)" = "REPRO_VALUE=1" ] || fail "project: the program did not print the provider's value" p0.log +"$MCPP" build -v > p1.log 2>&1 || fail "project: the warm build failed" p1.log +replayed p1.log || fail "project: the warm build was not replayed, so the next assertion measures nothing" p1.log + +# A, project build +sleep 1.1 +value ../rules 2 +"$MCPP" build > p2.log 2>&1 || fail "project: the build after the edit failed" p2.log +bin="$(bin_of target app)" +[ -n "$bin" ] || fail "project: the program was not found" p2.log +[ "$("$bin")" = "REPRO_VALUE=2" ] || fail "A: the edit of the provider's rules.ixx did not reach the program (build)" p2.log + +# B +"$MCPP" build -v > p3.log 2>&1 || fail "project: the build after the confirmed edit failed" p3.log +replayed p3.log || fail "B: the build after a confirmed edit was not replayed by the fast path" p3.log + +# A, project run: the fast path of `mcpp run` +sleep 1.1 +value ../rules 3 +out="$("$MCPP" run 2>&1)" || fail "project: the run after the second edit failed"$'\n'"$out" +grep -q '^REPRO_VALUE=3$' <<<"$out" || fail "A: the edit of the provider's rules.ixx did not reach the program (run)"$'\n'"$out" + +# D. A record that listed the roots without their tables (an engine before this +# one wrote that) declines once, and the record written after it is replayed. +"$MCPP" build > p4.log 2>&1 || fail "project: the build before ageing the record failed" p4.log +grep -q '^depSources=1$' target/.build_cache || fail "D: the record does not carry the root with its tables" target/.build_cache +awk '/^depSources=/ { n = substr($0, 12) + 0; print "depSourceRoots=" n + while (n-- > 0) { getline; split($0, f, "\t"); print f[1] } + next } + { print }' target/.build_cache > target/.build_cache.aged +mv target/.build_cache.aged target/.build_cache +grep -q '^depSourceRoots=1$' target/.build_cache || fail "D: the record was not aged" target/.build_cache +"$MCPP" build -v > d1.log 2>&1 || fail "D: the build with an aged record failed" d1.log +grep -q "predates the list of path-dependency roots" d1.log || fail "D: the decline did not name the missing tables" d1.log +"$MCPP" build -v > d2.log 2>&1 || fail "D: the build after the aged record failed" d2.log +replayed d2.log || fail "D: the record written after the decline was not replayed" d2.log +cd "$TMP" + +# ── A workspace member: the workspace fast path ───────────────────────────── +mkdir ws +printf '[workspace]\nmembers = ["app"]\n' > ws/mcpp.toml +provider ws/rules +consumer ws/app +cd ws + +"$MCPP" build -p app > w0.log 2>&1 || fail "workspace: the first build failed" w0.log +"$MCPP" build -p app -v > w1.log 2>&1 || fail "workspace: the warm build failed" w1.log +replayed w1.log || fail "workspace: the warm build was not replayed, so the next assertion measures nothing" w1.log +bin="$(bin_of target app)" +[ -n "$bin" ] && [ "$("$bin")" = "REPRO_VALUE=1" ] || fail "workspace: the program did not print the provider's value" w0.log + +# A, workspace +sleep 1.1 +value rules 2 +"$MCPP" build -p app > w2.log 2>&1 || fail "workspace: the build after the edit failed" w2.log +[ "$("$bin")" = "REPRO_VALUE=2" ] || fail "A: the edit of the provider's rules.ixx did not reach the program (workspace)" w2.log + +# B +"$MCPP" build -p app -v > w3.log 2>&1 || fail "workspace: the build after the confirmed edit failed" w3.log +replayed w3.log || fail "B: the workspace build after a confirmed edit was not replayed by the fast path" w3.log + +# C +for f in "$TMP"/proj/app/*.log "$TMP"/ws/*.log; do + if grep -q "dead entry" "$f"; then fail "C: a build printed the dead-entry warning" "$f"; fi +done +grep -q "module_extensions" "$TMP/proj/app/mcpp.toml" && fail "C: the consumer declares module_extensions, which the criterion forbids" + +echo "OK" diff --git a/tests/e2e/coverage-exceptions.tsv b/tests/e2e/coverage-exceptions.tsv new file mode 100644 index 00000000..13a168b7 --- /dev/null +++ b/tests/e2e/coverage-exceptions.tsv @@ -0,0 +1,4 @@ +# Tests that no hosted runner can run, read by .github/tools/check_e2e_coverage.py. +# One `\t` line each. A test that runs on some shard, or that a +# workflow names, does not belong here, and the check fails if it is listed. +658_a_program_runs_on_an_attached_android_device.sh needs an Android device attached over adb; the android job of ci-linux-e2e.yml runs `mcpp test` on an emulator instead diff --git a/tests/e2e/run_all.sh b/tests/e2e/run_all.sh index a928404c..57c6f3dc 100755 --- a/tests/e2e/run_all.sh +++ b/tests/e2e/run_all.sh @@ -65,20 +65,34 @@ case "$OS" in CAPS+=(elf unix-shell fresh-sandbox) command -v g++ &>/dev/null && CAPS+=(gcc) command -v patchelf &>/dev/null && CAPS+=(patchelf) - # musl-gcc: check both system PATH and xlings-managed locations + # musl-gcc: check both system PATH and xlings-managed locations. + # + # ANY RELEASE OF THE FAMILY, NOT ONE VERSION. This probe named + # 15.1.0, and the mingw-cross probe below named 16.1.0: a capability + # that names a release turns every test that declares it into a silent + # skip the day the default moves (SPEC-009, check C3). if command -v x86_64-linux-musl-g++ &>/dev/null \ - || [[ -x "$HOME/.xlings/data/xpkgs/xim-x-musl-gcc/15.1.0/bin/x86_64-linux-musl-g++" ]] \ - || [[ -x "${MCPP_HOME}/registry/data/xpkgs/xim-x-musl-gcc/15.1.0/bin/x86_64-linux-musl-g++" ]]; then + || ls "$HOME"/.xlings/data/xpkgs/xim-x-musl-gcc/*/bin/x86_64-linux-musl-g++ 2>/dev/null | head -1 | grep -q . \ + || ls "${MCPP_HOME}"/registry/data/xpkgs/xim-x-musl-gcc/*/bin/x86_64-linux-musl-g++ 2>/dev/null | head -1 | grep -q .; then CAPS+=(musl) fi # mingw-cross: the Linux-hosted MinGW-w64 cross toolchain (xim # mingw-cross-gcc, GCC 16 MSVCRT). Must be the xim-managed GCC-16 build, # NOT the distro apt g++-mingw-w64 (GCC 13 — no `import std`). Probe the # xlings/mcpp payload location, mirroring the musl probe above. - if [[ -x "$HOME/.xlings/data/xpkgs/xim-x-mingw-cross-gcc/16.1.0/bin/x86_64-w64-mingw32-g++" ]] \ - || [[ -x "${MCPP_HOME}/registry/data/xpkgs/xim-x-mingw-cross-gcc/16.1.0/bin/x86_64-w64-mingw32-g++" ]]; then + if ls "$HOME"/.xlings/data/xpkgs/xim-x-mingw-cross-gcc/*/bin/x86_64-w64-mingw32-g++ 2>/dev/null | head -1 | grep -q . \ + || ls "${MCPP_HOME}"/registry/data/xpkgs/xim-x-mingw-cross-gcc/*/bin/x86_64-w64-mingw32-g++ 2>/dev/null | head -1 | grep -q .; then CAPS+=(mingw-cross) fi + # llvm: an LLVM payload in the home mcpp uses. It was a legal token that + # no line granted, so the seven tests that need nothing else (134-137, + # 741, 875, 876) ran on no runner while their skip lines read like + # legitimate ones (finding F9 of the 2026-10-02 CI record). Granted on + # Linux only: the macOS runner wires its LLVM into the sandbox for the + # build, and the tests that declare `llvm` were written for an ELF host. + if ls "${MCPP_HOME}"/registry/data/xpkgs/xim-x-llvm/*/bin/clang++ 2>/dev/null | head -1 | grep -q .; then + CAPS+=(llvm) + fi # mingw-host-headers: this Linux HOST's own mingw-w64 headers # (`apt install mingw-w64`, distro package). Distinct from both # `mingw-cross` above (an xim-managed cross GCC) and `mingw` below (a @@ -395,10 +409,80 @@ if [[ -n "${E2E_SHARD:-}" ]]; then echo "FATAL: bad E2E_SHARD='$E2E_SHARD' (want /, 1-based)" exit 1 fi - echo "Shard: ${SHARD_IDX}/${SHARD_TOTAL} (round-robin over the suite)" fi SHARD_POS=0 +# BY MEASURED DURATION WHEN A TIMING TABLE IS GIVEN (rule R4 of the 2026-10-02 +# CI record). Round-robin balances counts, not time: measured on 2026-10-01 the +# two Windows shards differed by 3.6 minutes on average, and about fifteen tests +# of over thirty seconds took 58-64 percent of a shard. E2E_TIMINGS names a +# table of `\t` lines (tests/e2e/timings/.tsv); the +# tests are assigned longest first, each to the shard with the least time so +# far, and a test the table does not know counts as the table's median. The +# assignment depends only on the file list and the table, both in the +# repository, so a shard's membership is reproducible from the commit, and it +# changes which shard runs a test, never whether one does. +SHARD_MEMBERS="" +if (( SHARD_TOTAL > 1 )) && [[ -n "${E2E_TIMINGS:-}" ]]; then + if [[ ! -r "$E2E_TIMINGS" ]]; then + echo "FATAL: E2E_TIMINGS='$E2E_TIMINGS' is not readable" + exit 1 + fi + SHARD_MEMBERS="$( + for t in "$HERE"/[0-9]*.sh; do basename "$t"; done \ + | awk -v tf="$E2E_TIMINGS" ' + BEGIN { FS = "\t"; n = 0 + while ((getline line < tf) > 0) { + split(line, f, "\t"); if (f[2] ~ /^[0-9]+$/) { ms[f[1]] = f[2] + 0; v[++n] = f[2] + 0 } } + # median of the known durations + for (i = 1; i <= n; i++) for (j = i + 1; j <= n; j++) if (v[j] < v[i]) { x = v[i]; v[i] = v[j]; v[j] = x } + med = n ? v[int((n + 1) / 2)] : 1000 } + { w = ($0 in ms) ? ms[$0] : med; printf "%d\t%s\n", w, $0 }' \ + | sort -t "$(printf '\t')" -k1,1nr -k2,2 \ + | awk -v total="$SHARD_TOTAL" -v me="$SHARD_IDX" ' + BEGIN { FS = "\t"; for (i = 1; i <= total; i++) load[i] = 0 } + { best = 1; for (i = 2; i <= total; i++) if (load[i] < load[best]) best = i + load[best] += $1; if (best == me) print $2 }')" + echo "Shard: ${SHARD_IDX}/${SHARD_TOTAL} (by measured duration, $E2E_TIMINGS)" +elif (( SHARD_TOTAL > 1 )); then + echo "Shard: ${SHARD_IDX}/${SHARD_TOTAL} (round-robin over the suite)" +fi +in_shard() { + local name="$1" + if [[ -n "$SHARD_MEMBERS" ]]; then + grep -qxF "$name" <<<"$SHARD_MEMBERS" + return + fi + local mine=$(( (SHARD_POS % SHARD_TOTAL) + 1 )) + SHARD_POS=$(( SHARD_POS + 1 )) + (( mine == SHARD_IDX )) +} + +# What each test of this run did, for the coverage check of ci.yml +# (.github/tools/check_e2e_coverage.py): one `\t\t\t` +# line per test of this shard, status pass, fail, timeout or skip. +report() { + [[ -n "${E2E_REPORT:-}" ]] || return 0 + printf '%s\t%s\t%s\t%s\n' "$1" "$2" "$3" "$4" >> "$E2E_REPORT" +} +if [[ -n "${E2E_REPORT:-}" ]]; then : > "$E2E_REPORT"; fi + +# E2E_LIST=1 prints the tests this invocation would run (after E2E_ONLY and the +# shard, before the capability check) and exits, so a shard's membership can be +# read without running it. +if [[ "${E2E_LIST:-}" == "1" ]]; then + for test in "$HERE"/[0-9]*.sh; do + name="$(basename "$test")" + if [[ -n "${E2E_ONLY:-}" ]]; then + # shellcheck disable=SC2053 -- glob match is the point + [[ "$name" == $E2E_ONLY ]] || continue + fi + if (( SHARD_TOTAL > 1 )); then in_shard "$name" || continue; fi + echo "$name" + done + exit 0 +fi + # Optional name filter: E2E_ONLY="" runs just the matching tests. # Used by the workflows that own ONE subject (see ci-windows-msvc-xlings.yml) # so the job name and the thing it runs cannot drift apart. @@ -407,17 +491,16 @@ for test in "$HERE"/[0-9]*.sh; do # shellcheck disable=SC2053 — glob match is the point [[ "$(basename "$test")" == $E2E_ONLY ]] || continue fi + name="$(basename "$test")" if (( SHARD_TOTAL > 1 )); then - _mine=$(( (SHARD_POS % SHARD_TOTAL) + 1 )) - SHARD_POS=$(( SHARD_POS + 1 )) - (( _mine == SHARD_IDX )) || continue + in_shard "$name" || continue fi - name="$(basename "$test")" echo missing_cap="$(check_requires "$test")" if [[ -n "$missing_cap" ]]; then echo "SKIP: $name (missing capability: $missing_cap)" ((SKIP++)) + report skip "$name" 0 "missing capability: $missing_cap" continue fi echo "=== $name ===" @@ -434,13 +517,16 @@ for test in "$HERE"/[0-9]*.sh; do if [[ $rc -eq 0 ]]; then echo "PASS: $name (${_dur})" ((PASS++)) + report pass "$name" "$_dur_ms" "" elif [[ $rc -eq 124 ]]; then + report timeout "$name" "$_dur_ms" "exceeded ${E2E_TEST_TIMEOUT}s" # GNU timeout: 124 = killed after deadline (TERM); 137 = SIGKILL after grace. echo "TIMEOUT: $name (exceeded ${E2E_TEST_TIMEOUT}s — likely network / xlings stall)" ((FAIL++)) FAILED_TESTS+=("$name (TIMEOUT)") TIMED_OUT_TESTS+=("$name") else + report fail "$name" "$_dur_ms" "exit $rc" echo "FAIL: $name (exit $rc, ${_dur})" ((FAIL++)) FAILED_TESTS+=("$name (exit $rc)") diff --git a/tests/e2e/timings/linux.tsv b/tests/e2e/timings/linux.tsv new file mode 100644 index 00000000..770bdb79 --- /dev/null +++ b/tests/e2e/timings/linux.tsv @@ -0,0 +1,515 @@ +00_fixture_path_hygiene.sh 2543 +01_help_and_version.sh 12 +02_new_build_run.sh 5529 +03_multi_module.sh 5933 +04_incremental.sh 3342 +05_errors.sh 1223 +06_emit_xpkg.sh 15 +07_static_library.sh 133 +08_shared_library.sh 189 +09_path_dependency.sh 2437 +100_cppfly_reflection.sh 12254 +100_feature_sources_test_mode.sh 3887 +101_cppfly_llvm_soft.sh 3836 +103_target_vocabulary.sh 11073 +104_asm_sources_gas.sh 2741 +105_asm_sources_nasm.sh 8738 +106_feature_gated_sources_toml.sh 6781 +107_generated_files_toml.sh 1598 +108_cfg_conditional_sources.sh 1198 +109_per_glob_flags.sh 1218 +10_env_command.sh 5999 +110_build_mcpp_env_contract.sh 8839 +111_dep_build_mcpp.sh 2142 +112_build_mcpp_cross.sh 7194 +113_scanner_mcpp_dir_prune.sh 179 +114_run_scan_scope.sh 1563 +115_manifest_syntax_aot_brace.sh 1308 +116_include_family_and_define_spaces.sh 1322 +117_object_path_collision.sh 1311 +118_purview_include_rebuild.sh 8825 +119_dep_cfg_sources.sh 1351 +11_index_list.sh 7507 +120_ws_root_indices.sh 722 +121_default_ns_redirect.sh 491 +122_run_member.sh 2372 +123_same_named_main_across_dep.sh 1290 +124_gen_source_out_of_root_object_path.sh 1698 +125_build_mcpp_dep_dir.sh 1814 +126_default_features_opt_out.sh 391 +127_transitive_default_features_optout.sh 2577 +128_feature_forwarding.sh 279 +12_add_command.sh 450 +134_target_sysroot_and_queries.sh 4462 +135_diagnostic_promise_resolves.sh 7996 +136_aarch64_zero_libc_target.sh 577 +137_x86_64_zero_libc_target.sh 539 +138_feature_sources_gate_vs_provider.sh 601 +139_build_program_advisory.sh 637 +13_toolchain_pin.sh 8 +141_include_dirs_after.sh 1319 +142_generated_files_incremental.sh 2431 +143_build_mcpp_source_directive.sh 2716 +144_build_mcpp_include_dir.sh 3689 +145_root_build_mcpp_dep_dirs.sh 1621 +146_feature_flags.sh 8190 +147_per_os_features_xpkg.sh 504 +148_wide_include_list.sh 1408 +149_conditional_glob_flags.sh 1264 +14_toolchain_fallback.sh 7356 +150_clang_module_operator_template.sh 207 +151_index_url_artifact_migration.sh 5984 +152_test_subdir_names.sh 141 +153_test_isolation.sh 222 +154_test_filter.sh 150 +155_test_json.sh 188 +156_nested_ld_library_path.sh 124 +157_test_glob_flags.sh 140 +158_test_signal_and_duration.sh 308 +159_test_list.sh 24 +15_test_passing.sh 2176 +160_test_timeout.sh 4651 +161_xpkg_name_form.sh 63 +162_bare_name_namespace_scope.sh 5879 +163_identity_first_resolution.sh 25198 +164_ninja_goal_command_length.sh 529 +165_bare_name_cross_namespace_wire_address.sh 6126 +166_run_env_no_private_glibc.sh 353 +167_build_defines_module_scan.sh 813 +168_build_mcpp_musl_host_static.sh 5623 +169_semver_project_index.sh 511 +16_test_failing.sh 2361 +170_bmi_staging_no_cascade.sh 1320 +171_bmi_staging_locked_dest.sh 2585 +172_build_cache_cross_project.sh 9478 +173_build_profile_isolation.sh 1114 +173_index_refresh_policy.sh 791 +174_cache_modes_and_commands.sh 17259 +175_cpp20_import_std.sh 7801 +176_cpp20_std_bmi_isolation.sh 8248 +178_test_observability.sh 10866 +179_spaced_paths.sh 1435 +17_test_no_tests.sh 16 +181_build_mcpp_import_std.sh 5332 +183_cross_no_relink.sh 1167 +183_cxx_runtime_contract.sh 4291 +184_build_cache_object_layout.sh 12353 +185_index_floor_degrades.sh 5647 +186_build_mcpp_protocol_and_bound.sh 7594 +187_dep_host_tool.sh 3472 +188_build_actions.sh 8591 +189_host_module_rules.sh 1893 +18_devdeps_isolation.sh 1181 +190_link_rspfile_newlines.sh 1278 +191_link_scale.sh 20371 +192_install_path_namespace.sh 3686 +193_provision_reexport.sh 8603 +194_build_mcpp_glob_input.sh 2471 +195_target_cfg_feature_deps.sh 410 +196_version_identity_and_lock.sh 15118 +198_windows_resources_cross.sh 3729 +199_resources_validation.sh 222 +19_bmi_cache_reuse.sh 6699 +200_subos_env_reaches_program.sh 1778 +201_gcc_no_specs_pollution.sh 8564 +202_machine_output_contract.sh 2576 +203_exact_selector_lock_migration.sh 358 +204_new_transactional_scaffold.sh 9698 +205_root_local_subos.sh 732 +206_runtime_binding_physics.sh 1177 +207_runtime_contract_provenance.sh 200 +208_private_libc_stays_in_the_binary.sh 314 +209_runtime_verdict_never_reruns_green.sh 723 +20_p1689_scanner.sh 4818 +210_local_index_addressing_on_every_host.sh 248 +211_configure_only_cdb.sh 486 +212_cached_dep_std_is_ordered.sh 5431 +213_build_after_test_is_not_the_test_graph.sh 396 +214_executable_carries_dt_rpath.sh 1506 +215_pack_has_no_build_machine_paths.sh 3483 +216_selfcontained_refuses_host_capability.sh 545 +217_module_extensions.sh 9988 +218_module_extensions_graph_shape.sh 5117 +219_runtime_search_farm_is_last.sh 345 +21_ninja_dyndep.sh 4670 +220_farm_only_library_runs.sh 880 +221_subos_without_info_still_builds.sh 157 +222_shared_library_cxx_runtime_contract.sh 2362 +22_doctor_cache_publish.sh 47292 +230_bench_harness.sh 55578 +231_jobs_option.sh 2197 +232_workflow_syntax.sh 325 +233_bench_matrix.sh 224 +234_bmi_schedule_on.sh 6695 +235_std_object_only_when_needed.sh 1071 +236_module_extensions_default_toolchain.sh 3494 +237_default_subos_without_info.sh 287 +238_c_only_unit_links_with_c_driver.sh 3775 +23_remove_update.sh 12842 +240_pack_pe_zip_cross.sh 414 +242_pack_library_interface_and_headers.sh 779 +243_pack_library_interface_closure.sh 681 +244_pack_library_gate.sh 536 +245_pack_library_fat_target_selection.sh 763 +246_explicit_empty_sources.sh 261 +247_bare_triple_conditional_native.sh 389 +248_pack_library_fat_pe_leg.sh 513 +249_pack_workspace_root_unchanged.sh 217 +24_git_dependency.sh 10859 +250_pack_names_the_target.sh 261 +251_pack_library_shared.sh 375 +252_pack_library_old_client.sh 308 +253_pack_library_undetermined_partition.sh 532 +254_pack_library_platform_coverage.sh 503 +25_convention_mode.sh 4566 +260_module_extension_is_configuration.sh 198 +261_pack_follows_the_module_extension.sh 465 +263_lib_root_follows_the_extension.sh 1252 +264_pack_library_is_relocatable.sh 424 +265_pack_strips_but_stays_usable.sh 916 +267_target_requested_is_empty_for_a_native_build.sh 744 +268_target_side_resolution.sh 242 +269_openkal_llvm_spelling_still_resolves.sh 485 +26_c_language_support.sh 1293 +26_toolchain_management.sh 40215 +27_namespace_dependencies.sh 8849 +27_self_contained_home.sh 16869 +280_target_side_layers.sh 2345 +281_target_side_rules.sh 223 +282_target_is_a_request.sh 67 +283_run_target_flag_owns_its_cache_slot.sh 323 +284_env_segment_is_optional_everywhere.sh 3132 +285_kernel_abi_from_graph_keeps_the_payload_c_library.sh 4622 +286_the_openkal_stack_still_builds.sh 40171 +287_the_openkal_stack_crosses_to_aarch64.sh 38940 +289_one_host_reaches_every_openkal_target.sh 77137 +28_target_static.sh 4 +290_the_declaration_puts_an_environment_in_front.sh 1998 +291_dynamic_linkage_is_refused_only_when_the_c_library_is_the_graphs.sh 30147 +292_a_package_that_names_a_layer_does_not_lose_the_targets_compiler.sh 355 +293_the_requested_target_and_the_resolved_one_name_one_os.sh 9515 +294_the_list_answers_what_can_be_built_not_what_has_a_payload.sh 23 +295_naming_the_host_target_changes_nothing.sh 1370 +296_what_the_report_names_is_what_the_link_line_uses.sh 26315 +297_a_capability_pin_is_not_a_preference.sh 251 +298_overriding_a_convention_requires_replacing_it.sh 28550 +299_a_request_that_named_no_c_library_resolves_to_a_row_that_exists.sh 9159 +29_toolchain_partial_versions.sh 5553 +300_a_registered_family_is_not_reported_unknown.sh 43 +301_the_graphs_compiler_is_taken_and_nothing_is_written.sh 298 +302_a_stated_compiler_outranks_the_graph_and_two_requirements_do_not_stack.sh 85 +303_the_query_gives_one_answer_for_the_c_library.sh 107 +304_the_target_side_reaches_every_unit.sh 188 +305_built_from_is_not_published.sh 228 +306_dependency_linkage_axis.sh 376 +307_symbol_provision.sh 551 +308_dependency_required_features.sh 213 +309_host_module_identity.sh 987 +30_dev_binary_home.sh 20 +30_pack_modes.sh 5699 +310_build_dependencies.sh 749 +311_rule_build_dependencies.sh 3019 +312_build_rules_example.sh 3353 +313_check_stamp_on_every_platform.sh 639 +314_dependency_action_generated_header.sh 2676 +315_blocking_check_gates_compilation.sh 4923 +316_link_unit_with_no_inputs_is_refused.sh 144 +317_project_build_hooks.sh 18666 +31_transitive_deps.sh 142 +321_workspace_inheritance.sh 687 +322_dialect_flag_refusal.sh 10582 +323_post_link_record_durability.sh 422 +324_fast_path_sees_path_deps.sh 3532 +325_system_toolchain_refused.sh 222 +326_dependency_standard_floor.sh 740 +327_declared_xlings_deps_report_their_own_failure.sh 7813 +328_layer_predicates_apply_and_unknown_ones_say_so.sh 808 +329_exit_codes_match_the_contract.sh 20 +32_semver_merge.sh 9628 +330_runner_hosted_targets.sh 1428 +331_xlings_workspace_one_table.sh 1855 +333_named_runners_locked_and_sbom.sh 7286 +334_dependency_declared_tool_is_found.sh 251 +335_xlings_tool_tiers.sh 37 +337_run_takes_features_and_profile.sh 961 +339_run_reports_the_program_status.sh 956 +33_multi_version_mangling.sh 15107 +340_no_orphan_survives_a_killed_mcpp.sh 11268 +35_workspace.sh 1807 +36_llvm_toolchain.sh 1246 +37_llvm_import_std.sh 3445 +38_llvm_modules.sh 4184 +38_self_config_mirror.sh 258 +39_llvm_incremental.sh 5127 +39_xlings_index_migration.sh 220 +40_llvm_bmi_cache.sh 5557 +41_llvm_std_compat.sh 2913 +42_custom_local_index.sh 8323 +43_indices_lockfile.sh 8452 +44_indices_e2e_integration.sh 7633 +45_install_platform_mapping.sh 12 +46_self_config_mirror_no_bootstrap.sh 6953 +47_cdb_prebuilt_module_path_abs.sh 1236 +47_llvm_atomic_link.sh 669 +48_build_error_output.sh 296 +49_bmi_cache_nested_custom_index.sh 6825 +50_package_owned_build_flags.sh 182 +51_package_generated_files.sh 351 +52_local_path_namespaced_index.sh 486 +53_namespaced_cache_label.sh 454 +54_package_owned_ldflags.sh 262 +55_dependency_shared_artifact.sh 206 +56_transitive_shared_artifact.sh 237 +57_static_dep_shared_artifact.sh 208 +58_preinstall_mcpp_deps_for_hooks.sh 295 +59_cpp_standard_config.sh 6668 +600_accel_variant_selection.sh 415 +601_exclusive_capability.sh 172 +603_version_floor.sh 303 +604_offline_skips_sandbox_bootstrap.sh 434 +605_probe_channel_and_accel_reach_the_build_program.sh 1349 +606_constrained_source_globs_narrow_to_the_build.sh 1145 +607_chained_actions_form_the_device_link.sh 1132 +608_object_actions_reach_a_static_library.sh 532 +609_clean_stale.sh 4532 +609_shader_sources_are_device_sources.sh 1679 +60_stale_xpkg_cache_reinstall.sh 330 +610_feature_controlled_rule_collection.sh 4584 +611_variant_switch_is_not_replayed.sh 601 +612_build_program_file_input_fast_path.sh 1232 +613_sycl_sources_are_device_sources.sh 1808 +614_feature_xlings_reaches_xpkg_dir.sh 1043 +615_feature_deps_shared_library_is_linked.sh 481 +616_examples_and_their_index_agree.sh 292 +617_build_program_knows_the_cxx_stdlib.sh 566 +618_dependency_feature_tools_reach_its_build_program.sh 45430 +619_a_tool_may_not_be_conditioned_on_a_layer.sh 29 +61_dependency_visibility.sh 476 +620_link_flag_reaches_the_link_line.sh 579 +621_exports_narrows_the_symbol_surface.sh 570 +622_rule_declared_payload_reaches_the_consumers_build_program.sh 50647 +623_device_sources_must_reach_an_action.sh 292 +624_a_missing_host_module_names_the_key.sh 304 +625_a_payload_can_be_gated_on_the_accelerator.sh 44636 +626_naming_a_subset_of_backends_is_not_a_mismatch.sh 434 +627_one_package_one_version.sh 46552 +628_a_pin_below_a_stated_floor_is_refused.sh 47993 +629_a_version_range_is_answered_by_xpkg_dir.sh 47222 +62_dotted_dependency_selector_priority.sh 420 +62_runtime_library_dirs.sh 414 +630_a_rule_brings_its_own_environment.sh 44172 +631_action_depfile_tracks_a_runtime_discovered_include.sh 944 +632_a_build_program_is_told_its_package_name.sh 550 +633_host_modules_are_ordered_by_what_they_import.sh 1288 +634_a_private_module_fragment_is_not_a_partition.sh 130 +635_a_byte_order_mark_does_not_hide_a_module.sh 305 +636_a_module_extension_file_need_not_provide_a_module.sh 165 +637_a_conditional_runtime_table_alone_is_applied.sh 519 +638_pack_format_dispatch.sh 5742 +639_the_scanner_does_not_read_inside_a_comment.sh 790 +63_bare_dependency_peer_root_priority.sh 378 +640_a_capability_pin_explains_its_own_row.sh 9920 +641_the_android_rows_are_wired_and_the_simulator_is_a_row.sh 81543 +643_windows_subsystem_cross.sh 1455 +644_windows_keys_are_inert_elsewhere.sh 598 +645_the_fast_path_compares_the_toolchain_request.sh 684 +646_runtime_deploy_places_files_in_a_directory.sh 1881 +647_abi_threads_is_one_switch_for_the_graph.sh 642 +648_an_install_hook_sees_the_build_target.sh 868 +649_pack_carries_runtime_deploy_files.sh 3179 +64_shared_soname_runtime_alias.sh 404 +650_wasm_row_names_its_launcher.sh 21 +651_a_build_program_deploys_what_it_generated.sh 1872 +652_an_application_is_a_kind.sh 514 +653_exceptions_is_the_second_abi_member.sh 6 +654_a_requirement_on_the_target_axis.sh 715 +656_run_hands_the_distributable_to_the_runner.sh 2616 +657_a_build_program_under_the_web_row_uses_the_host_compiler.sh 3 +659_an_action_declaration_has_no_size_limit.sh 1262 +65_toolchain_runtime_dirs_for_run.sh 661 +660_an_action_names_the_engine_through_mcpp_self.sh 1358 +661_one_identity_two_declarations.sh 754 +662_pack_stages_declared_files_before_the_closure.sh 7113 +663_a_graph_libcxx_over_the_payloads_c_library.sh 17317 +665_a_host_tool_follows_its_source.sh 9337 +669_an_elf_host_pack_states_its_closure.sh 1078 +66_runtime_provides.sh 1271 +670_an_rpath_loader_token_is_left_as_written.sh 189 +671_toolchain_is_an_option_of_run_test_and_pack.sh 4156 +672_a_runner_receives_the_files_the_artifact_carries.sh 9538 +673_test_discover_names_where_tests_are.sh 1177 +674_a_format_is_run_by_the_runner_named_after_it.sh 8008 +677_a_conditional_dependency_replaces_the_unconditional_one.sh 728 +678_a_row_states_a_library_form.sh 860 +679_a_path_dependency_takes_its_manifests_identity.sh 669 +67_features_strict.sh 7264 +681_index_repos_reach_an_existing_home.sh 9473 +682_why_deps_prints_the_resolved_graph.sh 213 +683_a_build_program_reads_the_payload_pkg_config_view.sh 514 +684_a_runner_supplied_twice_is_refused.sh 518 +685_a_nested_test_loads_a_graph_shared_library.sh 405 +686_an_activated_xlings_subos_does_not_redirect_the_registry.sh 7823 +687_the_vendored_xlings_probe_is_an_argument_vector.sh 629 +688_emit_build_database.sh 4886 +689_pack_takes_features.sh 2163 +68_profile_passthrough.sh 8118 +690_a_shared_library_over_a_graph_cxx_runtime.sh 29859 +692_a_package_states_its_default_link_form.sh 1247 +693_the_root_build_program_reads_dependency_link_forms.sh 4248 +696_a_cxx_layer_provider_keeps_its_own_standard.sh 33280 +698_a_tool_source_under_its_dependency_has_a_bounded_address.sh 4322 +69_package_templates.sh 9948 +700_a_program_over_a_cxx_shared_library_has_one_cxx_runtime.sh 1123 +701_symbol_provision_does_not_report_what_the_build_shares_by_construction.sh 1437 +702_a_static_package_under_a_shared_library_is_linked_into_it.sh 747 +70_per_target_flags.sh 231 +710_a_forward_reaches_every_dependency_table.sh 454 +711_a_feature_deps_restatement_names_one_source.sh 5820 +712_a_tool_package_contributes_nothing_to_its_consumer.sh 6191 +713_a_git_dependency_selects_a_repository_member.sh 6173 +714_features_names_a_dependency_feature.sh 5951 +71_required_features_gate.sh 253 +720_the_root_build_program_reads_the_resolved_graph.sh 2778 +722_pack_strips_what_the_graph_built.sh 3741 +723_pack_reports_what_it_produced.sh 857 +72_test_profile_features.sh 337 +730_bare_compat_dependency_is_not_a_refresh_miss.sh 97 +731_planning_children_do_not_inherit_the_callers_pipe.sh 438 +732_xlings_refresh_is_bounded_and_owned.sh 4999 +733_envelope_reports_observed_network_access.sh 344 +734_auto_refresh_false_governs_first_index_sync.sh 159 +735_offline_missing_package_has_its_own_code.sh 118 +736_compile_flag_words_reach_the_compiler_and_the_databases.sh 414 +737_the_declared_runtime_payload_is_installed.sh 72189 +739_a_private_feature_dep_does_not_reach_the_consumer.sh 296 +73_issue131_per_target_cxxflag.sh 177 +740_gcc_over_a_graph_c_library_is_degraded_not_refused.sh 123 +741_c_abi_environment_declared_by_the_c_library.sh 177 +742_gcc_accepted_when_the_realisation_is_empty.sh 112 +743_kernel_abi_interfaces_are_resolved_not_preprocessed.sh 205 +744_a_declared_absence_explains_the_link_that_asked_for_it.sh 90 +745_no_run_builds_the_tests_and_says_so.sh 367 +746_macos_deployment_target_is_target_keyed_not_host_keyed.sh 233 +74_run_no_loader_env_leak.sh 1306 +75_index_status_offline.sh 8087 +765_a_consumer_include_directory_stays_in_the_consumer.sh 675 +76_compile_commands_generated.sh 1245 +76_qualified_custom_ns_noncanonical_filename.sh 458 +770_workspace_member_as_dependency.sh 1139 +772_a_published_member_is_self_contained.sh 5544 +773_commands_outside_the_build_read_the_effective_manifest.sh 349 +774_an_index_member_inherits_its_archive_workspace.sh 324 +775_a_host_tool_merges_its_conditional_sections_once.sh 5157 +776_a_path_with_no_utf8_spelling_is_named.sh 314 +777_c_standard_applies_to_the_package_that_declares_it.sh 187 +778_a_graph_link_searches_no_host_directory.sh 99076 +779_a_build_program_declares_a_runtime_search_dir.sh 1953 +77_cdb_preserves_test_entries.sh 4460 +780_a_passing_check_moves_its_stamp_past_a_changed_input.sh 1689 +781_deleted_root_cdb_is_restored.sh 2308 +782_foreign_root_cdb_entries_are_replaced.sh 3587 +783_cdb_switches_whole_with_the_configuration.sh 3204 +784_cdb_replays_from_its_directory.sh 445 +785_cdb_interface_flag_module_extensions.sh 221 +786_std_unit_in_the_database_and_build_id.sh 1191 +787_emit_plans_every_member_on_its_own.sh 165 +788_emit_defers_an_unbuilt_host_tool.sh 1183 +789_emit_describes_a_member_past_a_failed_build_program.sh 327 +78_test_main_combinations.sh 11880 +790_a_prepare_action_populates_an_unknown_directory.sh 1823 +791_a_prepare_action_that_creates_nothing_fails.sh 955 +792_a_rerun_input_inside_a_prepare_directory_is_warned.sh 1743 +793_an_unknown_action_role_is_refused.sh 2595 +795_a_link_flag_reaches_the_linker_as_written.sh 960 +798_a_rule_applies_through_a_source_it_claims.sh 13286 +799_an_action_runs_with_its_env_and_cwd.sh 1024 +79_gtest_regular_dep_feature_main.sh 14959 +800_a_feature_provides_its_host_tools.sh 1031 +801_a_dependency_program_is_shipped_with_the_consumer.sh 1851 +802_a_host_build_applies_its_host_row.sh 1695 +803_a_recorded_payload_that_is_gone_is_refused_offline.sh 111 +804_a_path_host_tool_builds_with_its_chosen_toolchain.sh 1148 +805_rooted_workspace_reaches_its_own_path_dependency.sh 481 +806_dash_p_resolves_the_package_first.sh 755 +807_a_lib_root_imports_its_own_package.sh 508 +808_device_source_is_not_a_compile_unit.sh 15188 +809_failed_build_program_hides_no_device_source.sh 89 +80_feature_defines.sh 259 +810_two_packages_deploy_the_same_bytes_to_one_name.sh 1941 +812_an_index_floor_is_a_closing_tip.sh 252 +813_conditional_dialect_cxxflags_root_only.sh 23819 +815_the_database_describes_what_a_rule_generates.sh 970 +816_a_git_dependency_reports_its_fetch.sh 221 +817_emit_writes_no_xlings_json_under_private_work_dir.sh 283 +818_a_declared_deploy_outranks_a_search_dir_dll_cross.sh 949 +819_the_crt_is_placed_by_its_rule_not_by_search_order.sh 377 +81_capability_binding.sh 573 +821_an_edge_states_its_advice_on_both_paths.sh 1614 +822_a_package_states_the_oldest_mcpp_it_supports.sh 288 +823_mcpp_core_and_mcpp_name_one_interface.sh 765 +824_a_build_program_reads_the_resolved_toolchain.sh 1076 +826_plugin_diagnostics_features_and_names.sh 726 +827_many_placements_run_as_one_process.sh 2327 +828_a_library_states_its_interface.sh 638 +829_pack_takes_a_member.sh 278 +82_feature_optional_deps.sh 264 +831_a_path_dependency_host_module_outside_src.sh 4402 +832_the_fast_path_resumes_after_an_edit.sh 2677 +833_a_workspace_is_one_graph_per_configuration.sh 1703 +834_a_workspace_plans_once_and_replays_in_one_check.sh 1180 +835_a_shared_library_is_placed_by_a_link.sh 319 +836_a_members_own_statements_reach_its_build.sh 1367 +837_a_members_resources_reach_its_image.sh 170 +838_a_program_shipped_through_artifacts_links_its_own_closure.sh 366 +839_a_workspaces_build_programs_are_named_ordered_and_cached.sh 743 +83_feature_defines_propagate.sh 141 +840_a_workspace_database_describes_each_package_once.sh 602 +841_pack_summarizes_many_outputs.sh 3206 +842_the_build_reports_each_step_once.sh 7211 +843_a_terminal_names_the_action_that_runs.sh 4684 +844_a_planned_build_keeps_one_lock_entry_per_identity.sh 794 +845_play_game_restores_the_terminal.sh 12308 +846_the_vendored_xlings_is_replaced_from_the_newest_source.sh 758 +847_a_module_name_is_unique_within_a_program.sh 1960 +849_a_staged_bmi_waits_for_a_module_compiled_here.sh 1607 +84_runtime_dll_deploy.sh 161 +850_a_selected_member_declares_as_the_root.sh 760 +851_a_shared_member_is_compiled_once_per_configuration.sh 536 +852_a_repeated_dash_p_selects_every_member_it_names.sh 273 +853_run_keeps_one_member_and_exclude_removes_members.sh 764 +854_a_test_over_several_members_plans_and_builds_once.sh 1333 +855_a_shared_test_build_is_reported_once_per_group.sh 346 +856_a_test_over_several_members_continues_past_a_failing_one.sh 1528 +857_a_host_module_several_members_import_is_compiled_once.sh 807 +858_the_engines_module_is_compiled_once_per_home.sh 4172 +859_a_host_module_entry_is_reused_while_its_inputs_agree.sh 3220 +85_target_cfg_build_flags.sh 129 +860_a_workspaces_programs_compile_at_the_same_time.sh 16698 +861_concurrent_compiles_plan_what_serial_compiles_plan.sh 1228 +862_run_q_writes_exactly_the_programs_stdout.sh 199 +863_a_run_whose_build_failed_exits_101.sh 662 +864_a_builds_status_is_on_standard_error.sh 753 +865_mcpp_test_reports_its_steps_on_stderr_and_its_results_on_stdout.sh 575 +866_both_streams_on_one_pipe_keep_their_order_and_json_stays_clean.sh 3940 +867_a_pack_over_several_members_plans_builds_and_runs_programs_once.sh 10859 +868_a_pack_over_several_members_is_refused_before_anything_is_compiled.sh 5486 +869_a_pack_over_several_members_names_each_member_in_its_report.sh 1582 +86_llvm_hermetic_link.sh 3796 +86_target_cfg_dependencies.sh 156 +870_a_pack_over_several_members_resolves_and_fails_per_member.sh 6416 +871_a_pack_states_its_build_and_every_drive_takes_the_job_count.sh 11625 +872_a_member_compiles_once_whatever_the_selection.sh 1966 +873_payload_overrides.sh 1826 +874_payload_on_request.sh 2050 +875_toolchain_by_path.sh 665 +876_toolchain_phase.sh 1138 +877_why_sources.sh 83 +87_build_default_profile.sh 502 +88_xlings_environment.sh 9400 +89_build_mcpp.sh 3422 +90_workspace_test.sh 636 +91_target_bare_alias.sh 1021 +92_build_mcpp_import.sh 269 +93_xpkg_parse.sh 84 +96_msvc_unavailable_nonwindows.sh 38 +98_reflection_import_std.sh 6868 diff --git a/tests/e2e/timings/macos.tsv b/tests/e2e/timings/macos.tsv new file mode 100644 index 00000000..c227203a --- /dev/null +++ b/tests/e2e/timings/macos.tsv @@ -0,0 +1,242 @@ +00_fixture_path_hygiene.sh 4000 +01_help_and_version.sh 0 +02_new_build_run.sh 5000 +04_incremental.sh 6000 +05_errors.sh 2000 +06_emit_xpkg.sh 0 +100_feature_sources_test_mode.sh 5000 +101_cppfly_llvm_soft.sh 3000 +10_env_command.sh 6000 +117_object_path_collision.sh 2000 +118_purview_include_rebuild.sh 9000 +11_index_list.sh 7000 +122_run_member.sh 3000 +126_default_features_opt_out.sh 2000 +128_feature_forwarding.sh 1000 +12_add_command.sh 1000 +138_feature_sources_gate_vs_provider.sh 3000 +139_build_program_advisory.sh 3000 +13_toolchain_pin.sh 0 +148_wide_include_list.sh 2000 +149_conditional_glob_flags.sh 2000 +14_toolchain_fallback.sh 5000 +150_clang_module_operator_template.sh 0 +151_index_url_artifact_migration.sh 6000 +152_test_subdir_names.sh 1000 +153_test_isolation.sh 1000 +154_test_filter.sh 1000 +155_test_json.sh 1000 +156_nested_ld_library_path.sh 0 +157_test_glob_flags.sh 1000 +158_test_signal_and_duration.sh 2000 +159_test_list.sh 0 +15_test_passing.sh 2000 +160_test_timeout.sh 5000 +161_xpkg_name_form.sh 0 +162_bare_name_namespace_scope.sh 7000 +163_identity_first_resolution.sh 10000 +164_ninja_goal_command_length.sh 2000 +165_bare_name_cross_namespace_wire_address.sh 6000 +166_run_env_no_private_glibc.sh 1000 +167_build_defines_module_scan.sh 3000 +16_test_failing.sh 2000 +170_bmi_staging_no_cascade.sh 2000 +171_bmi_staging_locked_dest.sh 4000 +178_test_observability.sh 13000 +179_spaced_paths.sh 2000 +17_test_no_tests.sh 0 +181_build_mcpp_import_std.sh 6000 +183_cxx_runtime_contract.sh 9000 +18_devdeps_isolation.sh 2000 +190_link_rspfile_newlines.sh 2000 +191_link_scale.sh 38000 +195_target_cfg_feature_deps.sh 2000 +199_resources_validation.sh 1000 +19_bmi_cache_reuse.sh 9000 +202_machine_output_contract.sh 3000 +204_new_transactional_scaffold.sh 7000 +210_local_index_addressing_on_every_host.sh 0 +211_configure_only_cdb.sh 4000 +221_subos_without_info_still_builds.sh 2000 +230_bench_harness.sh 36000 +231_jobs_option.sh 7000 +232_workflow_syntax.sh 0 +233_bench_matrix.sh 1000 +234_bmi_schedule_on.sh 2000 +236_module_extensions_default_toolchain.sh 2000 +23_remove_update.sh 14000 +242_pack_library_interface_and_headers.sh 4000 +243_pack_library_interface_closure.sh 4000 +244_pack_library_gate.sh 3000 +246_explicit_empty_sources.sh 1000 +247_bare_triple_conditional_native.sh 2000 +249_pack_workspace_root_unchanged.sh 1000 +24_git_dependency.sh 9000 +250_pack_names_the_target.sh 1000 +251_pack_library_shared.sh 1000 +252_pack_library_old_client.sh 2000 +253_pack_library_undetermined_partition.sh 2000 +254_pack_library_platform_coverage.sh 2000 +259_shared_library_macho.sh 3000 +25_convention_mode.sh 5000 +260_module_extension_is_configuration.sh 1000 +261_pack_follows_the_module_extension.sh 2000 +263_lib_root_follows_the_extension.sh 2000 +266_pack_reads_a_macho_program_without_running_it.sh 2000 +267_target_requested_is_empty_for_a_native_build.sh 2000 +268_target_side_resolution.sh 4000 +269_openkal_llvm_spelling_still_resolves.sh 3000 +27_namespace_dependencies.sh 8000 +280_target_side_layers.sh 4000 +299_a_request_that_named_no_c_library_resolves_to_a_row_that_exists.sh 0 +300_a_registered_family_is_not_reported_unknown.sh 0 +301_the_graphs_compiler_is_taken_and_nothing_is_written.sh 0 +302_a_stated_compiler_outranks_the_graph_and_two_requirements_do_not_stack.sh 0 +303_the_query_gives_one_answer_for_the_c_library.sh 2000 +304_the_target_side_reaches_every_unit.sh 0 +305_built_from_is_not_published.sh 1000 +313_check_stamp_on_every_platform.sh 2000 +317_project_build_hooks.sh 30000 +327_declared_xlings_deps_report_their_own_failure.sh 7000 +328_layer_predicates_apply_and_unknown_ones_say_so.sh 6000 +329_exit_codes_match_the_contract.sh 0 +32_semver_merge.sh 6000 +330_runner_hosted_targets.sh 5000 +331_xlings_workspace_one_table.sh 1000 +35_workspace.sh 2000 +36_llvm_toolchain.sh 1000 +37_llvm_import_std.sh 3000 +38_llvm_modules.sh 4000 +38_self_config_mirror.sh 0 +39_llvm_incremental.sh 6000 +39_xlings_index_migration.sh 1000 +40_llvm_bmi_cache.sh 6000 +41_llvm_std_compat.sh 4000 +42_custom_local_index.sh 8000 +43_indices_lockfile.sh 6000 +44_indices_e2e_integration.sh 8000 +45_install_platform_mapping.sh 0 +46_self_config_mirror_no_bootstrap.sh 6000 +47_cdb_prebuilt_module_path_abs.sh 1000 +47_llvm_atomic_link.sh 0 +48_build_error_output.sh 2000 +601_exclusive_capability.sh 0 +603_version_floor.sh 2000 +604_offline_skips_sandbox_bootstrap.sh 0 +609_clean_stale.sh 4000 +616_examples_and_their_index_agree.sh 1000 +628_a_pin_below_a_stated_floor_is_refused.sh 32000 +645_the_fast_path_compares_the_toolchain_request.sh 4000 +646_runtime_deploy_places_files_in_a_directory.sh 4000 +659_an_action_declaration_has_no_size_limit.sh 4000 +660_an_action_names_the_engine_through_mcpp_self.sh 3000 +666_a_macho_program_reaches_a_dispatched_format_with_its_tree.sh 4000 +668_a_macho_program_is_staged_with_its_closure.sh 2000 +66_runtime_provides.sh 1000 +671_toolchain_is_an_option_of_run_test_and_pack.sh 3000 +672_a_runner_receives_the_files_the_artifact_carries.sh 13000 +673_test_discover_names_where_tests_are.sh 2000 +674_a_format_is_run_by_the_runner_named_after_it.sh 15000 +679_a_path_dependency_takes_its_manifests_identity.sh 3000 +67_features_strict.sh 8000 +681_index_repos_reach_an_existing_home.sh 8000 +682_why_deps_prints_the_resolved_graph.sh 1000 +683_a_build_program_reads_the_payload_pkg_config_view.sh 1000 +684_a_runner_supplied_twice_is_refused.sh 3000 +686_an_activated_xlings_subos_does_not_redirect_the_registry.sh 4000 +687_the_vendored_xlings_probe_is_an_argument_vector.sh 1000 +688_emit_build_database.sh 7000 +68_profile_passthrough.sh 5000 +698_a_tool_source_under_its_dependency_has_a_bounded_address.sh 8000 +69_package_templates.sh 11000 +702_a_static_package_under_a_shared_library_is_linked_into_it.sh 2000 +704_cxx_identity_across_macho_images.sh 4000 +720_the_root_build_program_reads_the_resolved_graph.sh 4000 +721_an_android_row_links_on_a_macos_host.sh 107000 +732_xlings_refresh_is_bounded_and_owned.sh 5000 +733_envelope_reports_observed_network_access.sh 1000 +734_auto_refresh_false_governs_first_index_sync.sh 0 +735_offline_missing_package_has_its_own_code.sh 0 +736_compile_flag_words_reach_the_compiler_and_the_databases.sh 3000 +739_a_private_feature_dep_does_not_reach_the_consumer.sh 3000 +746_macos_deployment_target_is_target_keyed_not_host_keyed.sh 1000 +75_index_status_offline.sh 6000 +765_a_consumer_include_directory_stays_in_the_consumer.sh 2000 +76_compile_commands_generated.sh 1000 +770_workspace_member_as_dependency.sh 5000 +774_an_index_member_inherits_its_archive_workspace.sh 1000 +775_a_host_tool_merges_its_conditional_sections_once.sh 4000 +77_cdb_preserves_test_entries.sh 5000 +780_a_passing_check_moves_its_stamp_past_a_changed_input.sh 3000 +781_deleted_root_cdb_is_restored.sh 3000 +782_foreign_root_cdb_entries_are_replaced.sh 4000 +787_emit_plans_every_member_on_its_own.sh 1000 +78_test_main_combinations.sh 9000 +799_an_action_runs_with_its_env_and_cwd.sh 2000 +79_gtest_regular_dep_feature_main.sh 9000 +809_failed_build_program_hides_no_device_source.sh 0 +80_feature_defines.sh 2000 +812_an_index_floor_is_a_closing_tip.sh 1000 +815_the_database_describes_what_a_rule_generates.sh 2000 +816_a_git_dependency_reports_its_fetch.sh 1000 +817_emit_writes_no_xlings_json_under_private_work_dir.sh 0 +819_the_crt_is_placed_by_its_rule_not_by_search_order.sh 0 +81_capability_binding.sh 3000 +821_an_edge_states_its_advice_on_both_paths.sh 3000 +822_a_package_states_the_oldest_mcpp_it_supports.sh 2000 +823_mcpp_core_and_mcpp_name_one_interface.sh 4000 +824_a_build_program_reads_the_resolved_toolchain.sh 2000 +826_plugin_diagnostics_features_and_names.sh 3000 +827_many_placements_run_as_one_process.sh 4000 +828_a_library_states_its_interface.sh 4000 +829_pack_takes_a_member.sh 2000 +82_feature_optional_deps.sh 1000 +831_a_path_dependency_host_module_outside_src.sh 5000 +832_the_fast_path_resumes_after_an_edit.sh 5000 +833_a_workspace_is_one_graph_per_configuration.sh 5000 +834_a_workspace_plans_once_and_replays_in_one_check.sh 10000 +836_a_members_own_statements_reach_its_build.sh 4000 +839_a_workspaces_build_programs_are_named_ordered_and_cached.sh 5000 +83_feature_defines_propagate.sh 1000 +840_a_workspace_database_describes_each_package_once.sh 8000 +841_pack_summarizes_many_outputs.sh 5000 +842_the_build_reports_each_step_once.sh 13000 +843_a_terminal_names_the_action_that_runs.sh 5000 +845_play_game_restores_the_terminal.sh 13000 +846_the_vendored_xlings_is_replaced_from_the_newest_source.sh 2000 +847_a_module_name_is_unique_within_a_program.sh 5000 +849_a_staged_bmi_waits_for_a_module_compiled_here.sh 3000 +84_runtime_dll_deploy.sh 1000 +851_a_shared_member_is_compiled_once_per_configuration.sh 3000 +852_a_repeated_dash_p_selects_every_member_it_names.sh 3000 +853_run_keeps_one_member_and_exclude_removes_members.sh 6000 +854_a_test_over_several_members_plans_and_builds_once.sh 4000 +855_a_shared_test_build_is_reported_once_per_group.sh 2000 +856_a_test_over_several_members_continues_past_a_failing_one.sh 9000 +857_a_host_module_several_members_import_is_compiled_once.sh 3000 +858_the_engines_module_is_compiled_once_per_home.sh 8000 +859_a_host_module_entry_is_reused_while_its_inputs_agree.sh 7000 +85_target_cfg_build_flags.sh 1000 +860_a_workspaces_programs_compile_at_the_same_time.sh 10000 +861_concurrent_compiles_plan_what_serial_compiles_plan.sh 4000 +862_run_q_writes_exactly_the_programs_stdout.sh 1000 +863_a_run_whose_build_failed_exits_101.sh 4000 +864_a_builds_status_is_on_standard_error.sh 4000 +865_mcpp_test_reports_its_steps_on_stderr_and_its_results_on_stdout.sh 4000 +866_both_streams_on_one_pipe_keep_their_order_and_json_stays_clean.sh 6000 +867_a_pack_over_several_members_plans_builds_and_runs_programs_once.sh 16000 +868_a_pack_over_several_members_is_refused_before_anything_is_compiled.sh 14000 +869_a_pack_over_several_members_names_each_member_in_its_report.sh 10000 +86_llvm_hermetic_link.sh 0 +86_target_cfg_dependencies.sh 0 +870_a_pack_over_several_members_resolves_and_fails_per_member.sh 16000 +872_a_member_compiles_once_whatever_the_selection.sh 12000 +87_build_default_profile.sh 3000 +88_xlings_environment.sh 2000 +89_build_mcpp.sh 4000 +90_workspace_test.sh 5000 +91_target_bare_alias.sh 0 +92_build_mcpp_import.sh 1000 +93_xpkg_parse.sh 1000 +94_macos_test_stdlib.sh 3000 +96_msvc_unavailable_nonwindows.sh 0 diff --git a/tests/e2e/timings/windows.tsv b/tests/e2e/timings/windows.tsv new file mode 100644 index 00000000..8eb6380c --- /dev/null +++ b/tests/e2e/timings/windows.tsv @@ -0,0 +1,182 @@ +00_fixture_path_hygiene.sh 33892 +01_help_and_version.sh 184 +04_incremental.sh 9490 +05_errors.sh 2616 +06_emit_xpkg.sh 211 +100_feature_sources_test_mode.sh 10409 +101_cppfly_llvm_soft.sh 63285 +10_env_command.sh 7244 +117_object_path_collision.sh 11002 +118_purview_include_rebuild.sh 14229 +11_index_list.sh 5582 +122_run_member.sh 2332 +126_default_features_opt_out.sh 2747 +128_feature_forwarding.sh 2165 +12_add_command.sh 2435 +13_toolchain_pin.sh 112 +148_wide_include_list.sh 4911 +149_conditional_glob_flags.sh 2369 +14_toolchain_fallback.sh 8046 +150_clang_module_operator_template.sh 497 +151_index_url_artifact_migration.sh 10016 +152_test_subdir_names.sh 874 +153_test_isolation.sh 1594 +154_test_filter.sh 1385 +155_test_json.sh 1233 +156_nested_ld_library_path.sh 86 +157_test_glob_flags.sh 887 +159_test_list.sh 282 +15_test_passing.sh 1669 +161_xpkg_name_form.sh 632 +162_bare_name_namespace_scope.sh 10735 +163_identity_first_resolution.sh 38467 +164_ninja_goal_command_length.sh 1749 +165_bare_name_cross_namespace_wire_address.sh 10356 +166_run_env_no_private_glibc.sh 978 +167_build_defines_module_scan.sh 3560 +16_test_failing.sh 1861 +170_bmi_staging_no_cascade.sh 1813 +171_bmi_staging_locked_dest.sh 4057 +177_cpp20_msvc.sh 9276 +179_spaced_paths.sh 3756 +17_test_no_tests.sh 235 +180_msvc_build_mcpp.sh 22389 +183_cxx_runtime_contract.sh 9771 +18_devdeps_isolation.sh 1459 +190_link_rspfile_newlines.sh 3424 +191_link_scale.sh 27144 +195_target_cfg_feature_deps.sh 3223 +197_windows_resources.sh 8404 +199_resources_validation.sh 1740 +19_bmi_cache_reuse.sh 68995 +202_machine_output_contract.sh 2952 +204_new_transactional_scaffold.sh 9113 +210_local_index_addressing_on_every_host.sh 1074 +211_configure_only_cdb.sh 7706 +221_subos_without_info_still_builds.sh 2254 +230_bench_harness.sh 71394 +231_jobs_option.sh 9198 +232_workflow_syntax.sh 304 +233_bench_matrix.sh 653 +234_bmi_schedule_on.sh 4292 +236_module_extensions_default_toolchain.sh 3768 +23_remove_update.sh 20063 +241_windows_ucrt_runtime_identity.sh 18807 +242_pack_library_interface_and_headers.sh 4069 +243_pack_library_interface_closure.sh 8029 +244_pack_library_gate.sh 8759 +246_explicit_empty_sources.sh 1734 +247_bare_triple_conditional_native.sh 2950 +249_pack_workspace_root_unchanged.sh 1248 +250_pack_names_the_target.sh 1736 +251_pack_library_shared.sh 1204 +252_pack_library_old_client.sh 2007 +253_pack_library_undetermined_partition.sh 3015 +254_pack_library_platform_coverage.sh 3324 +255_pack_library_msvc_archiver.sh 2483 +256_pack_library_fat_windows.sh 5554 +258_shared_library_msvc_auto_def.sh 7734 +25_convention_mode.sh 5300 +260_module_extension_is_configuration.sh 1544 +261_pack_follows_the_module_extension.sh 2876 +262_pack_consumed_by_native_cl.sh 2052 +263_lib_root_follows_the_extension.sh 7435 +280_target_side_layers.sh 3141 +313_check_stamp_on_every_platform.sh 5557 +317_project_build_hooks.sh 27463 +327_declared_xlings_deps_report_their_own_failure.sh 10378 +328_layer_predicates_apply_and_unknown_ones_say_so.sh 4863 +329_exit_codes_match_the_contract.sh 305 +35_workspace.sh 2263 +36_llvm_toolchain.sh 114148 +38_self_config_mirror.sh 64649 +39_xlings_index_migration.sh 66344 +42_custom_local_index.sh 8209 +43_indices_lockfile.sh 11686 +44_indices_e2e_integration.sh 12795 +45_install_platform_mapping.sh 206 +46_self_config_mirror_no_bootstrap.sh 7828 +47_cdb_prebuilt_module_path_abs.sh 1915 +48_build_error_output.sh 63119 +601_exclusive_capability.sh 1806 +603_version_floor.sh 2585 +604_offline_skips_sandbox_bootstrap.sh 1823 +609_clean_stale.sh 6691 +628_a_pin_below_a_stated_floor_is_refused.sh 49452 +642_windows_subsystem.sh 5247 +645_the_fast_path_compares_the_toolchain_request.sh 110810 +646_runtime_deploy_places_files_in_a_directory.sh 5219 +659_an_action_declaration_has_no_size_limit.sh 11091 +660_an_action_names_the_engine_through_mcpp_self.sh 2681 +66_runtime_provides.sh 2221 +671_toolchain_is_an_option_of_run_test_and_pack.sh 3428 +673_test_discover_names_where_tests_are.sh 2238 +679_a_path_dependency_takes_its_manifests_identity.sh 3719 +67_features_strict.sh 8619 +681_index_repos_reach_an_existing_home.sh 10633 +682_why_deps_prints_the_resolved_graph.sh 1655 +683_a_build_program_reads_the_payload_pkg_config_view.sh 1509 +687_the_vendored_xlings_probe_is_an_argument_vector.sh 64285 +688_emit_build_database.sh 74035 +68_profile_passthrough.sh 5954 +698_a_tool_source_under_its_dependency_has_a_bounded_address.sh 13584 +69_package_templates.sh 18548 +703_the_llvm_row_on_the_msvc_abi_records_its_static_crt.sh 6359 +705_cxx_identity_across_pe_images.sh 5435 +720_the_root_build_program_reads_the_resolved_graph.sh 5906 +735_offline_missing_package_has_its_own_code.sh 640 +736_compile_flag_words_reach_the_compiler_and_the_databases.sh 2679 +739_a_private_feature_dep_does_not_reach_the_consumer.sh 2267 +75_index_status_offline.sh 6268 +760_msvc_toolset_is_chosen_once.sh 8601 +765_a_consumer_include_directory_stays_in_the_consumer.sh 68276 +76_compile_commands_generated.sh 1575 +770_workspace_member_as_dependency.sh 4873 +774_an_index_member_inherits_its_archive_workspace.sh 65596 +775_a_host_tool_merges_its_conditional_sections_once.sh 75532 +77_cdb_preserves_test_entries.sh 4293 +780_a_passing_check_moves_its_stamp_past_a_changed_input.sh 2885 +781_deleted_root_cdb_is_restored.sh 2778 +782_foreign_root_cdb_entries_are_replaced.sh 4060 +787_emit_plans_every_member_on_its_own.sh 1293 +78_test_main_combinations.sh 10826 +794_a_windows_program_finds_a_dll_through_runtime_search_dir.sh 2965 +799_an_action_runs_with_its_env_and_cwd.sh 2956 +79_gtest_regular_dep_feature_main.sh 11982 +809_failed_build_program_hides_no_device_source.sh 919 +80_feature_defines.sh 1649 +811_place_dlls_never_overwrites_a_declared_deploy.sh 1912 +814_the_llvm_row_defaults_to_the_dynamic_crt.sh 12482 +819_the_crt_is_placed_by_its_rule_not_by_search_order.sh 2294 +81_capability_binding.sh 4393 +820_a_windows_program_takes_its_runtime_from_one_resolver.sh 11452 +825_a_build_program_reads_the_msvc_toolset.sh 3484 +82_feature_optional_deps.sh 5090 +836_a_members_own_statements_reach_its_build.sh 3069 +839_a_workspaces_build_programs_are_named_ordered_and_cached.sh 4317 +83_feature_defines_propagate.sh 1073 +840_a_workspace_database_describes_each_package_once.sh 7573 +841_pack_summarizes_many_outputs.sh 8552 +842_the_build_reports_each_step_once.sh 15936 +846_the_vendored_xlings_is_replaced_from_the_newest_source.sh 73849 +847_a_module_name_is_unique_within_a_program.sh 8333 +848_msvc_binds_a_module_name_per_program.sh 1399 +849_a_staged_bmi_waits_for_a_module_compiled_here.sh 71571 +84_runtime_dll_deploy.sh 961 +857_a_host_module_several_members_import_is_compiled_once.sh 72667 +858_the_engines_module_is_compiled_once_per_home.sh 235616 +859_a_host_module_entry_is_reused_while_its_inputs_agree.sh 74384 +85_target_cfg_build_flags.sh 942 +860_a_workspaces_programs_compile_at_the_same_time.sh 85295 +86_target_cfg_dependencies.sh 1033 +872_a_member_compiles_once_whatever_the_selection.sh 76821 +87_build_default_profile.sh 3359 +88_xlings_environment.sh 2315 +89_build_mcpp.sh 6883 +90_workspace_test.sh 3360 +91_target_bare_alias.sh 915 +92_build_mcpp_import.sh 1371 +93_xpkg_parse.sh 685 +95_msvc_system_toolchain.sh 3086 +97_mingw_toolchain.sh 11239 +99_msvc_native_build.sh 8995 diff --git a/tests/scripts/test_check_e2e_coverage.py b/tests/scripts/test_check_e2e_coverage.py new file mode 100644 index 00000000..e27bd644 --- /dev/null +++ b/tests/scripts/test_check_e2e_coverage.py @@ -0,0 +1,99 @@ +#!/usr/bin/env python3 +"""Fixture tests for .github/tools/check_e2e_coverage.py (rule R5). + +A test that ran, a test a workflow names, and a test the exceptions excuse are +covered; a test that is none of these fails the check, and so does an exception +for a test that ran or that does not exist. +""" + +from __future__ import annotations + +import subprocess +import sys +import tempfile +import unittest +from pathlib import Path + +REPO_ROOT = Path(__file__).resolve().parents[2] +SCRIPT = REPO_ROOT / ".github" / "tools" / "check_e2e_coverage.py" + + +class E2ECoverage(unittest.TestCase): + def setUp(self) -> None: + self._dir = tempfile.TemporaryDirectory() + self.root = Path(self._dir.name) + e2e = self.root / "tests" / "e2e" + e2e.mkdir(parents=True) + for name in ("10_runs.sh", "20_in_a_job.sh", "30_excused.sh", "40_by_pattern.sh"): + (e2e / name).write_text("#!/usr/bin/env bash\n# requires: unix-shell\n") + wf = self.root / ".github" / "workflows" + wf.mkdir(parents=True) + (wf / "ci.yml").write_text("run: bash tests/e2e/20_in_a_job.sh\nE2E_ONLY: '40_*.sh'\n") + (e2e / "coverage-exceptions.tsv").write_text("# test\treason\n30_excused.sh\tneeds a device\n") + self.reports = self.root / "reports" + self.reports.mkdir() + (self.reports / "e2e-report-linux-1.tsv").write_text( + "pass\t10_runs.sh\t1200\t\nskip\t20_in_a_job.sh\t0\tmissing capability: qemu-riscv\n") + + def tearDown(self) -> None: + self._dir.cleanup() + + def run_check(self, *extra: str) -> subprocess.CompletedProcess: + return subprocess.run([sys.executable, str(SCRIPT), "--root", str(self.root), + "--reports", str(self.reports), *extra], + capture_output=True, text=True, check=False) + + def test_ran_named_and_excused_tests_are_covered(self) -> None: + r = self.run_check() + self.assertEqual(r.returncode, 0, r.stdout) + self.assertIn("1 ran on a shard, 2 run by a dedicated job, 1 excused, 0 uncovered", r.stdout) + + def test_a_name_in_a_comment_or_inside_a_longer_name_does_not_count(self) -> None: + e2e = self.root / "tests" / "e2e" + (e2e / "60_commented.sh").write_text("# x\n# requires: llvm\n") + (e2e / "70_short.sh").write_text("# x\n# requires: llvm\n") + with (self.root / ".github" / "workflows" / "ci.yml").open("a") as f: + f.write(" # 60_commented.sh is mentioned in a comment only\n" + "run: bash tests/e2e/170_short_but_longer.sh\n") + (self.reports / "e2e-report-linux-2.tsv").write_text( + "skip\t60_commented.sh\t0\tmissing capability: llvm\n" + "skip\t70_short.sh\t0\tmissing capability: llvm\n") + r = self.run_check() + self.assertEqual(r.returncode, 1) + self.assertIn("UNCOVERED: 60_commented.sh", r.stdout) + self.assertIn("UNCOVERED: 70_short.sh", r.stdout) + + def test_a_test_that_runs_nowhere_fails(self) -> None: + (self.root / "tests" / "e2e" / "50_nowhere.sh").write_text("# x\n# requires: llvm\n") + (self.reports / "e2e-report-linux-2.tsv").write_text( + "skip\t50_nowhere.sh\t0\tmissing capability: llvm\n") + r = self.run_check() + self.assertEqual(r.returncode, 1) + self.assertIn("UNCOVERED: 50_nowhere.sh: linux: missing capability: llvm", r.stdout) + + def test_an_exception_for_a_test_that_ran_fails(self) -> None: + (self.reports / "e2e-report-macos-1.tsv").write_text("pass\t30_excused.sh\t10\t\n") + r = self.run_check() + self.assertEqual(r.returncode, 1) + self.assertIn("STALE EXCEPTION: 30_excused.sh is excused but ran on macos", r.stdout) + + def test_an_exception_for_a_missing_test_fails(self) -> None: + with (self.root / "tests" / "e2e" / "coverage-exceptions.tsv").open("a") as f: + f.write("99_gone.sh\tremoved\n") + r = self.run_check() + self.assertEqual(r.returncode, 1) + self.assertIn("STALE EXCEPTION: 99_gone.sh is excused but does not exist", r.stdout) + + def test_no_report_says_nothing_and_fails(self) -> None: + for p in self.reports.iterdir(): + p.unlink() + self.assertEqual(self.run_check().returncode, 1) + + def test_merged_timings_are_written_per_host(self) -> None: + out = self.root / "timings" + self.assertEqual(self.run_check("--timings-out", str(out)).returncode, 0) + self.assertEqual((out / "linux.tsv").read_text(), "10_runs.sh\t1200\n") + + +if __name__ == "__main__": + unittest.main() diff --git a/tests/scripts/test_classify_changes.py b/tests/scripts/test_classify_changes.py new file mode 100644 index 00000000..961d0170 --- /dev/null +++ b/tests/scripts/test_classify_changes.py @@ -0,0 +1,82 @@ +#!/usr/bin/env python3 +"""Fixture tests for .github/tools/classify_changes.py (rule R2). + +A change of documentation that nothing names is `code=false`; a change of any +other path, of a document a test reads, or of nothing at all is `code=true`. +""" + +from __future__ import annotations + +import subprocess +import sys +import tempfile +import unittest +from pathlib import Path + +REPO_ROOT = Path(__file__).resolve().parents[2] +SCRIPT = REPO_ROOT / ".github" / "tools" / "classify_changes.py" + + +def classify(root: Path, paths: list[str]) -> str: + out = subprocess.run([sys.executable, str(SCRIPT), "--root", str(root)], + input="\n".join(paths), capture_output=True, text=True, check=True) + return out.stdout.splitlines()[0] + + +class ClassifyChanges(unittest.TestCase): + def setUp(self) -> None: + self._dir = tempfile.TemporaryDirectory() + self.root = Path(self._dir.name) + (self.root / "docs").mkdir() + (self.root / "docs" / "guide.md").write_text("guide\n") + (self.root / "docs" / "examples.md").write_text("examples\n") + (self.root / "tests" / "e2e").mkdir(parents=True) + (self.root / "tests" / "e2e" / "616_examples.sh").write_text( + 'grep -q x "$ROOT/docs/examples.md"\n') + (self.root / "src").mkdir() + (self.root / "src" / "main.cpp").write_text("int main() {}\n") + + def tearDown(self) -> None: + self._dir.cleanup() + + def test_documentation_that_nothing_names_is_not_code(self) -> None: + self.assertEqual(classify(self.root, ["docs/guide.md", ".agents/docs/x.md", + "README.md", "LICENSE"]), "code=false") + + def test_a_document_a_test_reads_is_code(self) -> None: + self.assertEqual(classify(self.root, ["docs/examples.md"]), "code=true") + + def test_the_translation_of_a_document_a_test_reads_is_code(self) -> None: + # The test names docs/examples.md; docs/zh/examples.md is its translation. + self.assertEqual(classify(self.root, ["docs/zh/examples.md"]), "code=true") + + def test_any_other_path_is_code(self) -> None: + self.assertEqual(classify(self.root, ["docs/guide.md", "src/main.cpp"]), "code=true") + + def test_an_empty_change_is_code(self) -> None: + self.assertEqual(classify(self.root, []), "code=true") + + def test_the_repository_classifies_its_own_documents(self) -> None: + # docs/20-toolchains.md is read by check_default_toolchain_docs.py. + self.assertEqual(classify(REPO_ROOT, ["docs/20-toolchains.md"]), "code=true") + # Release packaging copies the root README; the design-record index is + # read by nothing that a documentation check does not already run. + self.assertEqual(classify(REPO_ROOT, ["README.md"]), "code=true") + self.assertEqual(classify(REPO_ROOT, [".agents/docs/README.md", + ".agents/docs/2026-10-02-x.md"]), "code=false") + + def test_every_documentation_check_it_trusts_runs_in_the_docs_job(self) -> None: + sys.path.insert(0, str(SCRIPT.parent)) + import classify_changes # noqa: E402 + ci = (REPO_ROOT / ".github" / "workflows" / "ci.yml").read_text(encoding="utf-8") + docs_job = ci[ci.index("\n docs:"):ci.index("\n build-linux:")] + for check in classify_changes.DOCS_JOB_CHECKS: + with self.subTest(check=check): + name = Path(check).name + self.assertIn(name, docs_job if name != "gen_agents_index.py" + else (REPO_ROOT / ".github" / "tools" / "check_docs_structure.sh") + .read_text(encoding="utf-8")) + + +if __name__ == "__main__": + unittest.main() diff --git a/tests/unit/test_build_cache_record.cpp b/tests/unit/test_build_cache_record.cpp new file mode 100644 index 00000000..60d80e66 --- /dev/null +++ b/tests/unit/test_build_cache_record.cpp @@ -0,0 +1,317 @@ +#include + +import std; +import mcpp.build.execute; +import mcpp.build.prepare; +import mcpp.build.graph_shape; + +using namespace mcpp::build; + +// The record of a build (`target/.build_cache`) and the one predicate every fast +// path asks of it. Three properties are fixed here: +// +// - the engine that wrote a record is part of it, and a record whose engine is +// not the running one is not replayed (#757); +// - each path-dependency root is recorded with, and swept by, the extension +// table of its own package (#756); +// - an entry written before either field declines, once, and does not +// misparse what follows it. + +namespace { + +struct Tmp { + std::filesystem::path path; + Tmp() { + path = std::filesystem::temp_directory_path() + / std::format("mcpp_cache_record_test_{}", std::random_device{}()); + std::filesystem::create_directories(path); + } + ~Tmp() { + std::error_code ec; + std::filesystem::remove_all(path, ec); + } +}; + +void write_file(const std::filesystem::path& p, std::string_view body) { + std::filesystem::create_directories(p.parent_path()); + std::ofstream os(p, std::ios::binary); + os << body; +} + +BuildCacheEntry minimal_entry() { + BuildCacheEntry e; + e.targetTriple = ""; + e.outputDir = "/work/target/x86_64-linux-gnu/abc"; + e.ninjaProgram = "/usr/bin/ninja"; + e.fingerprint = "abc"; + e.runtimeEnvKey = "-"; + return e; +} + +} // namespace + +// ─── the engine ───────────────────────────────────────────────────────────── + +TEST(EngineIdentity, TheRunningEngineIsNotDeclined) { + const auto engine = running_engine(); + EXPECT_FALSE(engine.version.empty()); + EXPECT_FALSE(engine.exe.empty()); + EXPECT_FALSE(engine_declined_because(engine, engine).has_value()); +} + +TEST(EngineIdentity, AnEntryWithoutTheFieldDeclinesAndSaysWhy) { + const auto why = engine_declined_because(std::nullopt, running_engine()); + ASSERT_TRUE(why.has_value()); + EXPECT_NE(why->find("predates the engine identity"), std::string::npos) << *why; +} + +TEST(EngineIdentity, AnotherVersionDeclinesAndNamesBoth) { + const auto running = running_engine(); + auto recorded = running; + recorded.version = "2026.1.1.1"; + const auto why = engine_declined_because(recorded, running); + ASSERT_TRUE(why.has_value()); + EXPECT_NE(why->find("2026.1.1.1"), std::string::npos) << *why; + EXPECT_NE(why->find(running.version), std::string::npos) << *why; +} + +// The path alone is enough. A binary moved to another directory, or reinstalled +// there, is the same version and leaves the same stale text in the graph that a +// new version does: the graph names the executable by absolute path. +TEST(EngineIdentity, AnotherPathAtTheSameVersionDeclinesAndNamesBoth) { + const auto running = running_engine(); + auto recorded = running; + recorded.exe = "/removed/install/bin/mcpp"; + const auto why = engine_declined_because(recorded, running); + ASSERT_TRUE(why.has_value()); + EXPECT_NE(why->find("/removed/install/bin/mcpp"), std::string::npos) << *why; + EXPECT_NE(why->find(running.exe), std::string::npos) << *why; +} + +// ─── the admission predicate ──────────────────────────────────────────────── + +// The engine is the first thing asked: when it is not the same engine, nothing +// else about the record is read, so the reason is the engine's whatever else is +// wrong with the entry. This is what makes the one predicate cover an upgrade +// for the three fast paths that call it. +TEST(AdmitRecordedBuild, AnotherEngineIsDeclinedBeforeAnythingElseIsRead) { + auto e = minimal_entry(); + e.engine = running_engine(); + e.engine->version = "2026.1.1.1"; + const auto admitted = admit_recorded_build(e, ReplayAsk{.manifest = "mcpp.toml"}); + ASSERT_FALSE(admitted.has_value()); + EXPECT_NE(admitted.error().find("2026.1.1.1"), std::string::npos) << admitted.error(); +} + +TEST(AdmitRecordedBuild, AnEntryWithoutTheEngineIsDeclined) { + auto e = minimal_entry(); + const auto admitted = admit_recorded_build(e, ReplayAsk{.manifest = "mcpp.toml"}); + ASSERT_FALSE(admitted.has_value()); + EXPECT_NE(admitted.error().find("predates the engine identity"), std::string::npos) + << admitted.error(); +} + +// With the engine matching, the question moves on to the entry's other fields: +// the decline is no longer the engine's. +TEST(AdmitRecordedBuild, TheSameEngineIsAskedTheNextQuestion) { + auto e = minimal_entry(); + e.engine = running_engine(); + const auto admitted = admit_recorded_build(e, ReplayAsk{.manifest = "mcpp.toml"}); + ASSERT_FALSE(admitted.has_value()); + EXPECT_EQ(admitted.error().find("engine"), std::string::npos) << admitted.error(); + EXPECT_NE(admitted.error().find("runtime binding"), std::string::npos) << admitted.error(); +} + +// ─── the record, written and read ─────────────────────────────────────────── + +TEST(BuildCacheRecord, TheEngineAndEachRootsTablesSurviveAWriteAndARead) { + Tmp tmp; + auto e = minimal_entry(); + e.engine = EngineIdentity{"2026.10.2.1", "/opt/with space/bin/mcpp"}; + e.depSourceRoots = { + {"/work/rules", {".ixx", ".ccm"}, {}}, + {"/work/plain", {}, {}}, + {"/work/device", {}, {".cu"}}, + }; + e.depSourceRootsRecorded = true; + e.toolchainRecorded = true; + e.toolchainRequest = "cli=;default=llvm@22.1.8"; + write_build_cache_entries(tmp.path / "target" / ".build_cache", {e}); + + const auto read = read_build_cache(tmp.path); + ASSERT_EQ(read.size(), 1u); + ASSERT_TRUE(read[0].engine.has_value()); + EXPECT_EQ(*read[0].engine, *e.engine); + EXPECT_TRUE(read[0].depSourceRootsRecorded); + EXPECT_EQ(read[0].depSourceRoots, e.depSourceRoots); + // The fields written after the two above were read too: the blocks do not + // swallow one another. + EXPECT_TRUE(read[0].toolchainRecorded); + EXPECT_EQ(read[0].toolchainRequest, e.toolchainRequest); +} + +// A path the line format cannot hold is not written at all, so the entry reads +// as one that predates the list and declines, rather than as a shorter path. +TEST(BuildCacheRecord, ARootWithATabInItsPathIsLeftUnrecorded) { + Tmp tmp; + auto e = minimal_entry(); + e.depSourceRoots = {{"/work/a\tb", {".ixx"}, {}}}; + e.depSourceRootsRecorded = true; + e.toolchainRecorded = true; + e.toolchainRequest = "cli=;default=gcc@16.1.0"; + write_build_cache_entries(tmp.path / "target" / ".build_cache", {e}); + const auto read = read_build_cache(tmp.path); + ASSERT_EQ(read.size(), 1u); + EXPECT_FALSE(read[0].depSourceRootsRecorded); + EXPECT_TRUE(read[0].depSourceRoots.empty()); + EXPECT_TRUE(read[0].toolchainRecorded); +} + +TEST(BuildCacheRecord, AnEmptyListOfRootsIsRecordedNotAbsent) { + Tmp tmp; + auto e = minimal_entry(); + e.depSourceRootsRecorded = true; + write_build_cache_entries(tmp.path / "target" / ".build_cache", {e}); + const auto read = read_build_cache(tmp.path); + ASSERT_EQ(read.size(), 1u); + EXPECT_TRUE(read[0].depSourceRootsRecorded); + EXPECT_TRUE(read[0].depSourceRoots.empty()); + EXPECT_FALSE(read[0].engine.has_value()); +} + +// An engine that recorded the paths alone wrote `depSourceRoots=`. Its roots +// have no tables, so the block is read past and left unrecorded, and the fields +// after it still parse. +TEST(BuildCacheRecord, TheOlderBlockOfPathsIsReadPastAndUnrecorded) { + Tmp tmp; + write_file(tmp.path / "target" / ".build_cache", + "[target=]\n" + "/work/target/x86_64-linux-gnu/abc\n" + "/usr/bin/ninja\n" + "abc\n" + "-\n" + "\n" + "runTargets=0\n" + "runEnv=\n" + "\n" + "subos=\n" + "profile=dev\n" + "cacheMode=global\n" + "depSourceRoots=2\n" + "/work/rules\n" + "/work/other\n" + "runner=0\n" + "runtier=0\n" + "features=\n" + "toolchain=cli=;default=gcc@16.1.0\n" + "xlingsPayloads=0\n"); + const auto read = read_build_cache(tmp.path); + ASSERT_EQ(read.size(), 1u); + EXPECT_FALSE(read[0].depSourceRootsRecorded); + EXPECT_TRUE(read[0].depSourceRoots.empty()); + EXPECT_FALSE(read[0].engine.has_value()); + EXPECT_EQ(read[0].profile, "dev"); + EXPECT_TRUE(read[0].toolchainRecorded); + EXPECT_EQ(read[0].toolchainRequest, "cli=;default=gcc@16.1.0"); + EXPECT_TRUE(read[0].xlingsPayloadsRecorded); +} + +// A line of the block that is not a path and two lists leaves the block +// unrecorded rather than read with a table nobody wrote. +TEST(BuildCacheRecord, AMalformedRootLineLeavesTheBlockUnrecorded) { + Tmp tmp; + write_file(tmp.path / "target" / ".build_cache", + "[target=]\n" + "/work/target/x86_64-linux-gnu/abc\n" + "/usr/bin/ninja\n" + "abc\n" + "-\n" + "\n" + "depSources=1\n" + "/work/rules\n" + "runner=0\n"); + const auto read = read_build_cache(tmp.path); + ASSERT_EQ(read.size(), 1u); + EXPECT_FALSE(read[0].depSourceRootsRecorded); +} + +// ─── the sweep ────────────────────────────────────────────────────────────── + +namespace { + +// A dependency tree whose manifest is older than `ninjaTime` and whose one +// source, `rules.ixx`, is newer. +struct EditedProvider { + Tmp tmp; + std::filesystem::file_time_type ninjaTime; + EditedProvider() { + ninjaTime = std::filesystem::file_time_type::clock::now(); + write_file(tmp.path / "mcpp.toml", "[package]\nname = \"rules\"\n"); + write_file(tmp.path / "rules.ixx", "export module rules;\n"); + std::filesystem::last_write_time(tmp.path / "mcpp.toml", ninjaTime - std::chrono::seconds(20)); + std::filesystem::last_write_time(tmp.path / "rules.ixx", ninjaTime + std::chrono::seconds(20)); + } +}; + +} // namespace + +// The defect of #756: the provider declares `.ixx` and the consumer does not. +// The root is swept with its own package's table, so the edit is seen. +TEST(DepSourcesNewerThan, AProvidersOwnExtensionMakesItsEditCount) { + EditedProvider p; + EXPECT_TRUE(dep_sources_newer_than({{p.tmp.path, {".ixx"}, {}}}, p.ninjaTime)); +} + +// A root whose package declares nothing about `.ixx` does not have an `.ixx` of +// interest, whatever any other package declares: the table is the root's own. +TEST(DepSourcesNewerThan, APackageThatDeclaresNothingHasNoInterestInTheFile) { + EditedProvider p; + EXPECT_FALSE(dep_sources_newer_than({{p.tmp.path, {}, {}}}, p.ninjaTime)); +} + +// One root's table does not leak into the next: of two roots holding the same +// file, the one whose package declares the extension is the one that answers. +TEST(DepSourcesNewerThan, EachRootIsClassifiedByItsOwnTable) { + EditedProvider declares; + EditedProvider silent; + const auto ninjaTime = declares.ninjaTime; + EXPECT_FALSE(dep_sources_newer_than({{silent.tmp.path, {}, {}}}, ninjaTime)); + EXPECT_TRUE(dep_sources_newer_than( + {{silent.tmp.path, {}, {}}, {declares.tmp.path, {".ixx"}, {}}}, ninjaTime)); +} + +// What was always swept still is, with any table. +TEST(DepSourcesNewerThan, TheBuiltInInterfaceExtensionIsSweptWithoutADeclaration) { + EditedProvider p; + write_file(p.tmp.path / "m.cppm", "export module m;\n"); + std::filesystem::last_write_time(p.tmp.path / "m.cppm", p.ninjaTime + std::chrono::seconds(20)); + EXPECT_TRUE(dep_sources_newer_than({{p.tmp.path, {}, {}}}, p.ninjaTime)); +} + +TEST(DepSourcesNewerThan, ANewerManifestCounts) { + EditedProvider p; + std::filesystem::last_write_time(p.tmp.path / "mcpp.toml", p.ninjaTime + std::chrono::seconds(20)); + EXPECT_TRUE(dep_sources_newer_than({{p.tmp.path, {}, {}}}, p.ninjaTime)); +} + +// The graph names the engine it runs as `$mcpp`, escaped for ninja; the reader +// returns the path the emitter escaped, so that it compares with +// `mcpp_exe_path()` in generic form. +TEST(ReadEngineBinding, TheEscapedPathIsReadBack) { + Tmp tmp; + write_file(tmp.path / "build.ninja", + "# mcpp:graph=normal\ncxx = /opt/llvm/bin/clang++\n" + "mcpp = C$:/Program$ Files/mcpp$$1/mcpp.exe\n\nrule cxx\n"); + EXPECT_EQ(read_engine_binding(tmp.path / "build.ninja"), "C:/Program Files/mcpp$1/mcpp.exe"); +} + +TEST(ReadEngineBinding, AGraphWithoutTheBindingSaysNothing) { + Tmp tmp; + write_file(tmp.path / "build.ninja", "cxx = clang++\nmcpp_extra = x\nrule cxx\n"); + EXPECT_EQ(read_engine_binding(tmp.path / "build.ninja"), ""); +} + +TEST(ReadEngineBinding, AMissingFileSaysNothing) { + Tmp tmp; + EXPECT_EQ(read_engine_binding(tmp.path / "absent.ninja"), ""); +}