diff --git a/.agents/docs/2026-09-30-member-selection-and-build-program-cost-plan.md b/.agents/docs/2026-09-30-member-selection-and-build-program-cost-plan.md new file mode 100644 index 000000000..e944dec19 --- /dev/null +++ b/.agents/docs/2026-09-30-member-selection-and-build-program-cost-plan.md @@ -0,0 +1,943 @@ +--- +subject: design +status: landed +--- + +# Member selection, build programs prepared once, a pack over several members, and the output streams of `mcpp run`: the plan for the release after 2026.9.30.2 (#748, #749, #750) + +- Status: implemented as 2026.10.1.1. Section 15 records what was built, + what was measured, and where the implementation departs from sections 3 to + 6. Revision 3, below, is the plan as it was implemented. + - Revision 3 settles the open decisions by their recommendations (D3, D6, + D7), since the review directed that the plan be implemented. It removes + the stage-specific downstream verification from the plan, adds a review + from several angles (section 12), the tasks with their ownership and + dependencies (section 13), and the cross-repository work and its + verification (section 14). It folds #750, filed after revision 1, into S, + and records #751 as outside this release (section 8). + - Revision 1 was reviewed on 2026-10-01. D1, D2 and D5 were accepted. + - D4 was settled by the reviewer's rule: what the project owns is kept in + the project, and what comes from the index is kept globally. + - Status output moves to stderr on every command (R3, option (b)). + - D3 and D6 were asked about. Revision 2 explains D3 and replaces D6: the + revision 1 answer would have changed the meaning of a JSON field that + `docs/50-machine-output.md` §7 fixes. + - The review also asked for `mcpp run -q` to be aligned with established + practice. That is section 6, from a measurement. + - Section 11 is the self-review of revision 2. +- Date: 2026-09-30, revised 2026-10-01. +- Origin: + 1. A question on how to build or test several workspace members at once + (`mcpp test -p a -p b`), and the follow-up questions on what Cargo does + and what mcpp's own design admits. + 2. mcpp#748: a workspace's build programs are prepared and compiled one + after another, although only their runs have an order. + 3. mcpp#749: `mcpp pack` packs one member per invocation, so packing + several members plans the graph and runs the build programs once per + member. + 4. Review round 1: whether `mcpp run -q` follows the conventions of + comparable tools. + 5. mcpp#750, filed independently of this plan: a repeated `-p` keeps only + the last value (F1). +- Task: one plan for the next release that treats the four as one subject. + The subject is which members a command acts on, what a command pays once + per selection rather than once per member, and which stream carries what. + +## 0. Summary + +| # | Finding | Item | +|---|---|---| +| F1 | `-p` takes one value on every command. A repeated `-p` is accepted, and all but the last are dropped without a word (measured) | S1, S2 | +| F2 | The selection is computed by two functions. `build` and `emit` use `workspace_selection` and plan once per configuration group. `test --workspace` uses `workspace_fanout_members` and plans each member alone (code) | S1, S4 | +| F3 | Each build program compiles the bundled `mcpp` module, and every host module it imports, into its own directory before its own compile. #748 measured about 7.8 s per program on its runner, repeated for every program and every invocation. The attribution comes from code reading and is to be confirmed | B1, B2 | +| F4 | `pack` has one `-p`, no `--workspace`, and one stage directory per invocation (code, #749) | K1 | +| F5 | `mcpp run` writes its status lines to stdout. With `-q` it still writes an empty line to stdout before the program's output. A failed build exits 1, as a program that returns 1 does (measured) | R1, R2, R3 | + +The plan has four parts. No manifest key is added. + +- **S, the selection.** + - S1: one selection shared by every command. + - S2: `-p` is repeatable. + - S3: `--exclude`. + - S4: `test` plans a selection once. +- **B, the build programs (#748).** + - B1: the `mcpp` module and host modules are compiled once per agreeing + flag set. The output is kept globally when it comes from the engine or the + index, and in the workspace otherwise. + - B2: programs are compiled concurrently, after the process launcher is + made safe for concurrent callers (B2-0). + - B3: runs in dependency waves, deferred. +- **K, pack over a selection (#749).** + - K1: `pack --workspace` and a repeated `-p`, with one plan and one stage + directory per member. +- **R, the output streams.** + - R1: no empty line under `-q`. + - R2: a distinct exit status for a `run` whose build failed. + - R3: status output on stderr, on every command. + +Three existing behaviours change: + +- A repeated `-p` selects every member it names (F1). +- Status lines leave stdout (R3). +- A `run` whose build failed exits 101 (D7). + +## 1. Findings + +### F1. A repeated `-p` keeps the last value + +`src/cli.cppm` declares `package` with `.takes_value()` and without +`.multiple()`. It does so on `build` (390), `run` (436), `test` (538), `pack` +(660) and `describe` (707). + +Measured with mcpp 2026.9.30.1 on a virtual workspace of three members `a`, +`b` and `c`. The option declaration is the same at 8e00a183. + +``` +$ mcpp build -p a -p b + Workspace building member 'b' + ... + Compiling b v0.1.0 (b) + Finished dev [unoptimized + debuginfo] in 0.30s +``` + +`a` is not built, and nothing says so. #750 reports the same defect on +`mcpp test -p a -p b`, which tests `b` alone and exits 0. + +### F2. Two selection functions, and `test` plans each member alone + +`src/cli/cmd_build.cppm` has two functions: + +- `workspace_selection` (79) returns the workspace root and the selected + members. `build` (324) and `emit build-database` (612) pass the members to + `workspace_groups`. They then plan once per configuration group through + `BuildOverrides::workspace_members` (workspace design 2026-09-29, §15). +- `workspace_fanout_members` (50) returns the member list alone. `test` (998) + loops over it and calls `run_tests` once per member with + `package_filter = member`. Each call resolves the toolchain, plans that + member's closure, and runs the build programs of the closure. + +The planner already accepts a group together with the test targets of each +member (`BuildOverrides::member_targets`, read in +`src/build/prepare/manifest.cpp:187`). `--configure-only` and +`emit build-database` use it. `test` does not. + +This has two consequences, the same ones #749 states for `pack`: + +- A shared member's build program runs once per member that reaches it. +- A shared member's active features are those of one closure, not the union + across the selection. + +### F3. The preparation of a build program is repeated per program + +When the program's cache is stale, `run_build_program` +(`src/build/build_program.cppm`) performs four steps: + +1. It compiles the bundled `mcpp` module and its `mcpp.core` alias + (`build_mcpp_module`, 1401). +2. It obtains the `std` module, which is already cached globally by + `stdmod::ensure_built`. +3. It compiles every host module the program imports (`build_host_module`, + 1518). +4. It compiles `build.mcpp` and runs it. + +Steps 1 and 3 write into `/target/.build-mcpp`. Neither step checks +whether its output is already current. `program_compiling` is called only at +step 4 (1665), so the reported `ran` covers step 4 alone. Steps 1 to 3 are +counted as `plan` in `Finished`. `step9_member_build_programs` +(`src/build/prepare/target_side.cpp:1945`) runs the programs one after +another. + +#748 measured about 7.8 s outside `ran` for each of four programs on a +4-vCPU `windows-2025` runner, and about 8 s again in each `mcpp pack`. The +four programs import the same host module with the same flags. + +### F4. `pack` acts on one member + +`pack` has one `-p` and no `--workspace` (`src/cli.cppm:660`). The stage +directory is one value per invocation (`BuildOverrides::pack_stage_dir`, +`src/build/prepare.cppm:687`). `step9_member_build_programs` gives that one +value to every program (`bpEnv.packStageDir`, target_side.cpp:1992). + +#749 measured two consecutive `mcpp pack -p` invocations on a five-member +workspace at 146 s. Distribution accounts for about 35 s of that. Most of the remainder is +planning and build programs repeated per invocation, plus a 29.2 s gap +between the two processes. + +### F5. The output streams of `mcpp run` + +The measurements used mcpp 2026.9.30.2 on member `a` of the F1 workspace. The +program prints `OUT` on stdout and `ERR` on stderr. + +| Command | stdout | Note | +|---|---|---| +| `mcpp run 2>/dev/null` | the `Workspace`, `Resolving`, `Resolved`, `Target`, `Inferred`, `Finished` and `Running` lines, an empty line, then `OUT` | status is on stdout | +| `mcpp run -q 2>/dev/null` | `\n` `OUT` `\n` (checked with `od -c`) | one empty line before the program's output | +| `mcpp run -q -- x` (the program returns 3) | exit 3 | the program's status passes through | +| `mcpp run -q` with a compile error | exit 1, and the diagnostic is shown | the build's status | +| `mcpp run -q` (the program returns 1) | exit 1 | indistinguishable from the previous row | + +Where each behaviour comes from: + +- **Status on stdout.** This was decided on purpose by the observability + design of 2026-05-22: `status`, `info`, `finished` and progress go to + stdout, and `warning` and `error` go to stderr. +- **The empty line.** `src/build/execute.cppm:2183` prints `std::println("")` + after the `Running` line, and it does so unconditionally. Under `-q`, only + the empty line remains. +- **Arguments after `--`.** The `--quiet`/`-q` pre-scan (`src/cli.cppm:161`) + stops at `--`, so arguments after it reach the program. + +## 2. What the design already states, and what follows from it + +| # | Principle | Where | Consequence here | +|---|---|---|---| +| P1 | An input mcpp cannot serve, or cannot read unambiguously, is refused by name. mcpp does not guess | `docs/00-what-mcpp-is.md`, the guarantee; `docs/07-workspace.md` §5.3, where an ambiguous `-p` is refused, naming every match | F1 is a defect. A name that matches no member is refused, and so is a selection a command cannot act on | +| P2 | One graph per configuration. The selection only chooses the roots | `docs/07-workspace.md` §5.4; workspace design 2026-09-29 §15 | Several members are one plan, never N invocations. The selection is a set, so argument order does not change the plan | +| P3 | A command over several members continues past a failing member, reports each member, and ends with a summary | `docs/07-workspace.md` §5.3 | A multi-member `-p` inherits this report. There is no fail-fast switch | +| P4 | `-p` names a package, resolved among the members | `docs/07-workspace.md` §5.3 | `-p` does not select a dependency, unlike Cargo | +| P5 | The engine carries general capabilities. A new manifest key is a compatibility cost on engines already released | the engine-and-plugin rule; the `[c-abi]` precedent | No `default-members` key | +| P6 | A BMI is usable only by a compile that agrees with it. The agreement is produced from one set of flags, not checked afterwards | `build_host_module` (`src/build/hostprogram.cppm:918`) | B1 shares a BMI only between compiles whose key, which includes the flags, is equal | +| P7 | Only what comes from the immutable store may enter the global cache, and only when nothing it was built against is local | `src/build/prepare/plan.cpp:1964` | B1 applies the same rule: engine and index output is global, and everything else stays in the workspace | +| P8 | The JSON streams add fields and never remove or redefine one | `docs/50-machine-output.md` §7 | D6 adds records and fields. `build_ms` keeps its meaning | + +### Compared with Cargo + +| Cargo | mcpp in this plan | Reason | +|---|---|---| +| `-p a -p b` | same | P2 | +| `--workspace --exclude c` | same, and also with the implicit whole selection at a virtual root (D2) | a virtual root without `-p` already means every member | +| `-p` accepts a dependency | refused; members only | P4 | +| `-p 'foo-*'` | not in this release | a fourth resolution form would make a pattern ambiguous between a name and a path | +| `[workspace] default-members` | not in this release | P5 | +| stops at the first failing test binary unless `--no-fail-fast` | continues and reports every member (D1) | P3 | +| status on stderr, and stdout is the program's | the same after R3 | section 6 | +| a failed build exits 101, and a program's status passes through | the same after R2 (D7) | section 6 | + +## 3. The selection (S1 to S4) + +### S1. One selection, one function + +`workspace_fanout_members` and `workspace_selection` are replaced by one +function. + +- Input: `(wantAll, packages[], excludes[])`. +- Output: `{root, members}`. `members` is a set, kept in `[workspace] + members` order (D5). A rooted workspace's own package comes first, as `"."`. + +| Input | Members | +|---|---| +| `--workspace` | all | +| a virtual root, no `-p` | all | +| a rooted root, no `-p` | `"."` | +| inside member X, no `-p` | X | +| `-p X -p Y` | {X, Y}, each resolved by the §5.3 order | +| any "all" form above with `--exclude Z` | all minus Z | + +The following are refused before any planning: + +| Input | Refusal | +|---|---| +| `-p N`, where N matches no member | refused, listing the members | +| N is ambiguous | refused, naming every match, as today | +| `--exclude` together with `-p` | refused | +| `--exclude Z`, where Z matches no member | refused | +| every member excluded | refused | + +Two `-p` values that resolve to one member select it once. + +### S2. `-p` is repeatable + +`.multiple()` is added to `package` on `build`, `test`, `pack` and +`describe`. `BuildOverrides::package_filter` stays a single string. The plan +reads it in 56 places, and a selection of one member still sets it. A +selection of several members goes through `workspace_members`, as +`--workspace` does now. + +`run` keeps one member. On `run`, a second `-p` is refused, naming both. + +### S3. `--exclude` + +The option is added to `build`, `test`, `pack` and `describe`. Its value is +resolved like `-p`, and it may be repeated. It applies to every "all" form +(D2) and is refused with `-p`. + +### S4. `test` plans a selection once + +A `test` over more than one member proceeds in four steps: + +1. It plans once per configuration group, with each member's test targets in + `member_targets`, as `--configure-only` does. +2. It builds each group's graph once. +3. It runs each member's tests in member order, continuing past failures + (D1). +4. Test discovery stays scoped to each member. + +The report keeps its shape, with one change for the shared build (D6, +section 7). The timeouts are divided between the two phases: + +- `--build-timeout` bounds the build. +- `--workspace-timeout` bounds the runs. A member not started before the + deadline is listed as `not run`, as before. + +A member that fails to plan fails alone, by the R5.2 rule `emit` already +uses. The other members of its group are planned without it. + +## 4. Build programs (#748) + +### B0. The attribution, measured first + +Before B1 is written, `mcpp build --workspace` runs with `MCPP_VERBOSE=1` on +a fixture of #748's shape: four members, each with a build program that +imports one member host module, which imports the bundled `mcpp` module. The +`buildmcpp-host` lines are timestamped and summed per step: the `mcpp` +module, `std`, each host module, and `build.mcpp`. + +If steps 1 and 3 of F3 are not most of the preparation, B1 is re-scoped. The +measurement is recorded in section 15, which the landing revision adds. +B0 also records the `base` flags of the bundled module's compile (see +section 11, item 10). + +### B1. The `mcpp` module and host modules are compiled once per key, and kept by provenance + +Each output of `build_mcpp_module` and `build_host_module` moves from the +program's `bdir` to an entry addressed by a key. The key is built from: + +- the host compiler's identity (`compilerHash`); +- the standard flag, `base`, and the `use` flags of the compile; +- the SHA-256 of the interface file; +- for the bundled module, the mcpp version, which determines its text; +- for a host module, the providing package's identity (index, name, + version). + +Under P6, a program whose flags differ has a different key. Agreement is +therefore a consequence of the key and is never checked separately. + +**Where the entry lives (D4, as decided): by provenance, under P7.** + +| What | Where | Why | +|---|---|---| +| The bundled `mcpp` module and `mcpp.core` | the global cache, next to the `std` module | the engine owns its text. It is identical in every project for one mcpp version and one host compiler | +| A host module from an index package whose sources are in the immutable store | the global cache, beside the package's cached objects | the same admission rule as the dependency cache (`plan.cpp:1964`) | +| A host module from a path or git dependency, or from a workspace member | `/target/.build-mcpp/host-modules//` | its sources can change without its name and version changing | + +A host module compiled "alone" imports only `std` and `mcpp` +(`build_host_module`: "a rule package is a leaf by construction"). Its local +taint is therefore its own package's alone, so the dependency cache's rule +applies without a closure walk. + +The cache mode applies as it does for dependencies: + +- `--cache global`: global when admissible, otherwise the workspace. +- `local` and `off`: the workspace. Programs of one invocation still share + entries. + +The global entries reuse `mcpp.bmi_cache`: + +- the entry layout and `entry.json`, whose recorded inputs are compared field + by field on a hit; +- the LRU stamp, so `mcpp cache gc` collects them; +- the write to a temporary name followed by a rename. + +On GCC, a consumer stages BMIs into its `gcm.cache`, which `bmi_cache` +already does for cached dependencies. On clang and MSVC, the BMIs are +referenced by path. + +Expected effect, which is an estimate, not a measurement: + +- On a fresh runner, the first program pays steps 1 and 3 of F3, and every + later program of the build pays neither. For the #748 workspace this saves + 25 to 30 s. +- A later `mcpp pack` or `mcpp test` in the same checkout pays neither. +- On a developer machine, a second project with the same mcpp and host + compiler does not compile the bundled module again. + +### B2. Programs are compiled concurrently + +**B2-0, the launcher is safe under concurrent use.** Two threads that start +children at the same time must not pass one child's pipe to the other. + +- `capture_exec` on Linux and macOS creates its pipe with `::pipe`, without + `O_CLOEXEC` (`modules/platform/src/process.cppm`). +- The bounded launchers on both families create an inheritable write end and + start the child with handle inheritance + (`modules/platform/src/windows/bounded_process.cppm`, `CreateProcessA` with + `bInheritHandles = TRUE`). + +A child started by one thread while another thread's pipe is open inherits +that pipe's write end. The other thread's reader then waits for end of file +until the unrelated child exits. The pipes are therefore created close-on-exec +where the platform offers it (`pipe2` with `O_CLOEXEC`; `posix_spawn`'s `dup2` +clears the flag on the child's descriptors). Where the platform does not +offer it (macOS pipes, Windows handle inheritance), the creation of the pipe, +the start of the child and the parent's close of the write end form one +critical section under a process-wide mutex. The registry of children for +signal forwarding (`guard_group_on_signal`) is made safe for concurrent +callers in the same change. + +`step9_member_build_programs` is split into two phases: + +- **Phase A** compiles every stale program concurrently, up to the build's + job count, once B1's entries exist. No compile depends on another program's + run. +- **Phase B** runs the programs in the present serial order and applies + their directives in that order. The plan and `build.ninja` are therefore + byte-identical to the serial form's (#748 D). + +A compile's output is captured and printed as a whole. When several compiles +fail, the failure reported is the first in the serial order (#748 E). + +### B3. Runs in dependency waves: deferred (D3) + +B3 would run programs with no dependency between them at the same time. In +the #748 workspace, gpp.core and gpp.updater would run together, then gpp.cli +and gpp.gui. + +#748 estimates the saving at 12.9 s to 10.8 s. The cost is paid in +correctness: + +- Directives from concurrent runs would have to be buffered and applied in + the serial order. +- Programs write generated files into their packages. Nothing now states + that two programs' writes do not interfere, because they have always run + one at a time. +- Reports, the first reported failure, and program timeouts would each need + an ordering rule. + +B3 is reconsidered only if a measurement after B1 and B2 shows the runs to be +the dominant remaining cost. + +## 5. Pack over a selection (#749) + +### K1 + +`mcpp pack --workspace`, `--exclude`, and a repeated `-p` select members +through S1. For `pack`, "all" means every member with a packable target. + +1. **Refusals before any compile.** + - A selected member that does not provide the requested `--format` is + refused, and the refusal names it. + - `--output` with more than one member must name a directory. + - Two members whose staging writes one destination from different sources + are refused, naming both, by the rule `mcpp stage` applies. +2. **One plan per configuration group, and one build.** Each build program + runs once. +3. **A stage directory and a format per member.** + - `BuildOverrides::pack_stage_dir` becomes a map from member to + directory. + - `step9_member_build_programs` already visits one package at a time, and + sets `bpEnv.packStageDir` and `bpEnv.packFormat` from the map. +4. **Distribution per member, in member order.** A failing member is + reported, and the others continue (P3). The exit status is non-zero if any + member failed. +5. **`mcpp pack -p X` keeps its meaning.** It plans X's closure alone. + +## 6. The output streams (R1 to R3) + +### Established practice + +The practice below is recalled, not measured here. + +- **Cargo.** `cargo run` writes every status line (`Compiling`, + `Finished`, `Running`) to stderr. The program owns stdout. `-q` suppresses + cargo's own messages, not errors. A failed build exits 101, and a program's + exit status passes through. +- **`go run` and `zig run`.** Both print nothing of their own on success and + write diagnostics to stderr. +- **The common rule.** stdout carries a command's result: the program's + output for `run`, and the document for a command that emits one. Progress + goes to stderr, where a redirection or a pipe does not capture it. + +### R1. No empty line under `-q` + +The separator after the `Running` line is written only when the `Running` +line was written. It goes to the same stream as the `Running` line, which is +stderr after R3. + +Criterion R-A: `mcpp run -q 2>/dev/null` produces exactly the program's +stdout, compared byte for byte. + +### R2. A `run` whose build failed exits with a status of its own (D7) + +A `run` whose planning or build fails exits **101**. This is Cargo's value +and an established convention. It is also rare among programs' own exit +statuses. The program's own status continues to pass through unchanged. +`build`, `test` and `pack` keep their exit statuses; `test`'s 0, 1 and 2 are +a documented contract. + +Criterion R-B: with a compile error, `mcpp run` exits 101. With a program +that returns 1, it exits 1. + +### R3. Status output on stderr, on every command (option (b), decided) + +Every line the 2026-05-22 design sends to stdout moves to stderr: `status`, +`info`, `finished`, `line`, the progress bar, and the live region. This +covers every command, so no two commands differ. Four changes follow: + +- **The live region's terminal test** follows stderr instead of stdout. + Output revision 3 (2026-09-30) draws a frame in one write, and that is + unaffected. +- **stdout keeps only a command's result.** This means the program's output + for `run`, the JSON streams, and the documents that `emit`, `describe` and + `--list-runners` print. +- **JSON modes.** stdout would now stay clean without `set_quiet`. The + existing `set_quiet` calls are kept, so that a JSON run also leaves stderr + free of human status, as it does today. +- **Records.** This plan supersedes the stream table of the 2026-05-22 + design, and that record is not edited (the record rule). The user-facing + docs that describe streams are updated in the same pull request: + `docs/50-machine-output.md` and `docs/09-commands-by-scenario.md`. + +**R3-0, a census first.** 46 e2e scripts mention `Compiling`, `Finished` or +`Running`. Most capture `2>&1`, which is indifferent to the change. The +scripts that capture stdout alone and assert a status line are counted +before the change, and each is corrected in the same pull request. + +**User-facing change.** `mcpp build | tee log` no longer records the status +lines, and becomes `mcpp build 2>&1 | tee log`. CI logs, which capture both +streams, are unaffected. The CHANGELOG states this under a breaking-change +heading. + +Criterion R-C: on a warning-free project, `mcpp build >/dev/null` still +shows `Compiling` and `Finished`, and `mcpp build 2>/dev/null` writes +nothing. + +## 7. Decisions + +| # | Question | Outcome | +|---|---|---| +| D1 | Should a multi-member `test` continue past a failing member? | **Accepted**: yes, and there is no switch | +| D2 | May `--exclude` be used with every "all" form? | **Accepted**: yes. It is refused with `-p` | +| D3 | Should B3, runs in dependency waves, be deferred? | **Settled by the recommendation**: deferred (§4 B3) | +| D4 | Where does B1's output live? | **Decided by the reviewer's rule**: engine and index output is global, and project-owned output stays in the workspace (§4 B1) | +| D5 | Are reports and packs in manifest order or command-line order? | **Accepted**: manifest order | +| D6 | How is a shared build reported in `test`? | **Settled by the recommendation**: revised form below | +| D7 | Should a `run` whose build failed exit 101? | **Settled by the recommendation**: yes (§6 R2) | +| D8 | Should status move to stderr on one command or on every command? | **Decided**: every command, option (b) (§6 R3) | + +**D6, revised.** Revision 1 proposed reporting each member's `elapsed_ms` as +its run time alone. That redefines a field, which P8 forbids. Revision 2 +adds and does not redefine: + +- **A group record** comes before the group's first test record: + `{"group_build": {"group": 0, "members": [...], "build_ms": N}}`. +- **Each member's `build_ms`** is its group's build wall time. That is still + true to the field's definition: the wall time this member's tests waited + for their build. A new field, `build_group`, names the group. + - A consumer that sums `build_ms` over members deduplicates by + `build_group`. + - A consumer that does not sum is unaffected. +- **Each test's `duration_ms`** keeps the meaning the code gave it: the run of + a test that ran, the build of a `compile_fail`. (The documentation's + "build+run" did not describe the measured value; it is corrected.) The test + binary's own build time, the sum of its edges in `.ninja_log`, is the added + field `build_ms`. +- **The human report** prints one line per group: members, build time, and + the slowest edges, for example `slowest: libs/jsc link 88s`. That keeps the + signal the per-member split existed for: a member whose link, not its + tests, is slow. Each member's line then states its run time. + +## 8. What this release does not do + +| Item | Reason | +|---|---| +| `-p` selecting a dependency | P4 | +| glob patterns in `-p` and `--exclude` | a fourth resolution form, and no demand | +| `[workspace] default-members` | P5 | +| `--fail-fast` / `--no-fail-fast` | P3; `--workspace-timeout` bounds the fan-out | +| several members as N internal invocations | P2 | +| a multi-member `run` | an artifact to execute is one program | +| B3 | D3 | +| a global home for project-owned host modules | P7 | +| #751, a shared member recompiled when another member compiles a module of the same name | see below | + +**#751 is outside this release.** W10 of 2026.9.30.2 moves two providers of +one module name below their packages' directories only when the plan holds +both (`src/build/plan.cppm`, the block after the product directories). A +plan of `-p app` holds one provider and a plan of `--workspace` holds two, +so the shared member's command lines differ between the two selections. A +command line independent of the selection needs one of two things: + +- the set of module names of the whole workspace, including the closures of + members that are not selected, at every plan; or +- every module placed below its provider and named explicitly to every + importer. On GCC this means a module mapper file on every compile, which is + the default path of every GCC build. + +Both change the planning of every workspace build, and neither is a +consequence of this plan's items. #751 remains open for its own design. The +criteria S-B and S-G below are stated on fixtures in which every module name +has one provider. + +## 9. Delivery + +One pull request in mcpp carries every item of this plan, the documentation, +and the version. The items share `src/cli/cmd_build.cppm`, `src/cli.cppm` +and the e2e suite, and R3 changes what every e2e script reads, so separate +pull requests would correct the same scripts more than once. Section 13 +divides the work into tasks and states their order; section 14 states what +the other repositories do. + +## 10. Acceptance criteria + +Every new e2e fixture writes its paths through named `*_HOST` variables and +passes the `00` path lint. + +**Selection** +- S-A. `mcpp build -p a -p b` in a workspace of `a`, `b` and `c` builds `a` + and `b`. `c`'s object directory stays absent. +- S-B. `mcpp build -p a -p b`, followed by `mcpp build -p b -p a`, adds no + compile edge to `.ninja_log`. +- S-C. `mcpp build -p nosuch` exits non-zero before planning. Its message + names `nosuch` and lists the members. +- S-D. `mcpp run -p a -p b` is refused, naming both. +- S-E. `mcpp build --workspace --exclude c` builds `a` and `b`. + `--exclude nosuch` and `-p a --exclude b` are refused. +- S-F. `a` and `b` share `core`, whose build program appends a line to a + file under `OUT_DIR` on each run. After `mcpp test --workspace`, the file + has one line. Today it has two. +- S-G. After `mcpp build --workspace`, `mcpp test --workspace` adds no + compile edge for `core`'s sources. The fixture has no dev-dependency, so + the test build does not change `core`'s features. +- S-H. The JSON stream of S-F has one `group_build` record, and both + members' summaries carry its `build_group` and its `build_ms`. +- S-I. #750's reproduction: `mcpp test -p a -p b` runs the tests of `a` and + of `b`, and the workspace summary counts both members. + +**Build programs (#748)** +- B-A. Two members' programs import one host module from a path dependency + with identical flags. The module's compile appears once in the + `buildmcpp-host` verbose log, and its entry is under the workspace's + `target/`. +- B-B. A program whose standard differs gets its own entry. +- B-C. After one build, a second project with the same mcpp and host + compiler does not compile the bundled `mcpp` module. There is no + `mcpp module precompile` or `compile` line in its verbose log. +- B-D. A host module from a path dependency is never written under the + global cache root, with `--cache global` set. +- B-E. Two independent programs record their compile start and end. The two + intervals overlap when `-j` is 2 or more. The test compares timestamps, not + wall time. +- B-F. `build.ninja` and the applied directives are byte-identical to those + of a build at concurrency 1, and the `ran` lines are in the same order. +- B-G. When two programs fail to compile, the reported failure is the first + in the serial order. +- B-H. Two children started at once from two threads, one of which exits at + once and one of which sleeps: the reader of the first returns when the + first exits, not when the second does (B2-0). + +**Pack (#749)** +- K-A. #749's criteria A to E. The stage directories are read through + `pack_stage_dir()` and written to `OUT_DIR`. + +**Output streams** +- R-A, R-B and R-C (section 6). + +## 11. Self-review of revision 2 + +1. **The D4 rule versus the #748 measurement.** The rule sends the measured + workspace's host module to the workspace store, because that module is a + workspace member and its sources are local. It is still compiled once per + build, which is #748's saving. The global part adds the bundled module and + index rules such as `mcpp.plugins`. No gain claimed for #748 depends on the + global part. +2. **Taint of an index host module.** P7's dependency cache also requires + that nothing a package was built against is local. A host module is + compiled alone, against `std` and `mcpp` only. `std` is already global, + and the bundled module is global under B1. The rule therefore holds + without a walk. If host modules ever gain imports of other packages, which + `build_host_module` states they cannot, this must be re-derived. B-D + guards the local side. +3. **Upgrade debris.** Every mcpp release leaves one stale entry for the + bundled module per host compiler. It is collected by `mcpp cache gc`, as a + stale `std` entry is. There is no new eviction mechanism. +4. **R3 and the live region.** When stdout is a pipe and stderr is a + terminal, as in `mcpp run | less` or `mcpp emit ... > file`, the region is + now drawn. Today it is not drawn in that case. That is the intended + effect, and it is also Cargo's behaviour. The reverse case, stderr + redirected and stdout a terminal, loses the region, which is also + intended. +5. **R3 and `mcpp test`.** Test programs' stdout is captured into the JSON + stream or printed on failure; its routing is not changed by R3. Only + mcpp's own status lines move. R-C checks `build` only. A test-side + criterion is not added, because the human test report + (`test result ok. …`) is itself a result and stays on stdout. **This + boundary, the report as result or as status, is the one open point of R3, + and PR 4 settles it by the census, following Cargo, which prints + `test result` on stdout.** +6. **S4 and `--workspace-timeout`.** Today the timeout can stop the fan-out + between member builds. After S4, the group build is one step and cannot + be interrupted by it. A workspace whose build alone exceeds the deadline + used to stop early and now overruns until `--build-timeout`. This is + stated in `docs/07-workspace.md` §5.3 by PR 1. +7. **S-G under dev-dependencies.** Revision 1 stated S-G without condition. + A dev-dependency can activate a feature of a shared package and + legitimately recompile it, as in Cargo. Revision 2 states the fixture's + condition. +8. **The exit status 101 and `run` under a runner.** With + `[target.].runner` or `--runner`, the status that passes through + is the runner's. A runner that itself returns 101 would read as a failed + build. This is accepted, as in Cargo, and documented. +9. **Numbers are estimates.** The 25 to 30 s of B1 and #749's 30 to 45 s are + estimates. B0 replaces the first by a measurement in the landing + revision. +10. **Cross-project reuse depends on `base`.** The key includes `base`, the + flags the bundled module is compiled with. By code reading, `base` is + `host_base_flags(tc, macosDeploymentTarget)` + (`src/build/build_program.cppm:534`). It is built from the host toolchain + and the macOS deployment target only, and so carries no project path. + Cross-project reuse therefore holds. B0 records `base` to confirm this; + if a project path appears, the bundled module stays in the workspace until + that is resolved. + +## 12. Review from several angles (revision 3) + +**Architecture.** +- One selection function replaces two (S1). The selection is data, a set of + members, and the planner already accepts a set; no planning mode is added. +- B1 is one keyed store with two homes, chosen by provenance (P7). The global + home reuses `mcpp.bmi_cache`'s entry layout, its recorded inputs and its + LRU stamp. +- B2 splits `run_build_program` into a compile phase and a run phase. Only + `step9_member_build_programs` schedules the two apart. The root package's + program and the dependencies' programs keep the single call. +- K1 divides `build_and_pack` into a build, a stage per member, and a + dispatch. A single member is the same code with one member. +- R3 is one decision in `mcpp.ui`: the stream that narrates. Every narrating + function reads it. + +**Stability.** +- Directives are applied in the serial order, so the plan of a concurrent + build is byte-identical to the serial plan (B-F). +- Concurrency is confined to compiles. Runs stay serial (B3 deferred). +- The launcher is made safe for concurrent callers before any concurrent + caller exists (B2-0). +- A store entry is written under a temporary name and renamed into place. + Its `entry.json` is written last, so a partial entry is never read. +- S4 reuses the group planning that `emit build-database` and + `--configure-only` already exercise. +- A member that fails to plan fails alone. + +**Simplicity.** +- No manifest key is added. +- The options added are a repeatable `-p` and `--exclude`. +- There is no switch for failing fast, for the stream of status output, or + for the store's location. + +**User experience.** +- A repeated `-p` does what it states. A misspelt member is refused, and the + refusal lists the members. +- `mcpp run -q > file` writes exactly the program's output. A failed build is + distinguishable from a failing program (101). +- `test` and `pack` over several members plan once. +- A pipe or a redirection receives a command's result, and progress stays on + the terminal. + +**Compatibility.** +- A single `-p` keeps its meaning. A repeated `-p`, which acted on the last + member alone, now acts on every member it names; the old behaviour was the + defect #750 reports. +- The JSON streams gain a record and a field and lose nothing (P8). +- Status lines move from stdout to stderr. This is the one change a script + can observe. It is stated in the CHANGELOG with its migration (`2>&1`). + Machine consumers read the JSON forms, which do not move. Section 14 + records that no ecosystem consumer reads status lines from stdout. +- The exit status 101 applies to `run` alone. `build`, `test` and `pack` keep + theirs. +- Nothing is written to a manifest, so an older engine reads every project + this release reads. + +**Cross-platform.** +- B1 covers the three BMI families. GCC finds BMIs under the compile's + `gcm.cache`, so a store entry is staged there, as `bmi_cache` stages a + cached dependency. clang names a BMI with `-fmodule-file==`, and + MSVC with `/reference =`. +- Store keys use `u8string()`. A path given to a tool is narrowed with + `try_narrow`, by the path narrowing rule of the contributing guide. +- B2-0 differs by platform: `pipe2` on Linux, a mutex on macOS, and a mutex or + an explicit handle list on Windows. +- R3's live region follows the stream it is drawn on. The Windows console + test `can_move_cursor` is asked of stderr. + +**Consistency.** +- `build`, `test`, `pack`, `describe` and `emit build-database` accept the + same selection. `run` refuses a second member. +- Every report is in manifest order (D5). +- Every command narrates on stderr. + +**Upgrade.** +- No project needs an edit, and no cache needs a migration. +- The first build after the upgrade compiles the bundled module once per host + compiler, because the key holds the mcpp version. +- A script that read status lines from stdout adds `2>&1`. + +**Test coverage.** +- Each criterion of section 10 has an e2e script. The selection function and + the store key have unit tests. +- The census of R3-0 corrects the existing scripts that read status from + stdout. Each correction captures both streams. +- B1's clang and MSVC paths run in the macOS and Windows e2e shards. B2-0 has + a test that starts two children at once and requires each reader to finish + when its own child exits. + +## 13. Tasks, ownership and dependencies + +| Task | Items | Files owned | Depends on | +|---|---|---|---| +| T1 | S1 to S4, D6, #750 | `src/cli.cppm` (the `package` and `exclude` options of `build`, `run`, `test`, `describe`, `emit build-database`), `src/cli/cmd_build.cppm`, a new `src/cli/selection.cppm` (module `mcpp.cli.selection`), `src/build/execute.cppm` (`run_tests` and its summary only), `src/build/prepare/manifest.cpp` and `src/build/prepare.cppm` (test targets of a group), `src/project.cppm` | none | +| T2 | B0, B2-0, B1, B2 | `src/build/build_program.cppm`, `src/build/hostprogram.cppm`, `src/build/prepare/target_side.cpp` (`step9_member_build_programs`), `src/bmi_cache.cppm`, `modules/platform/src/process.cppm`, `modules/platform/src/unix/*`, `modules/platform/src/windows/*`, `src/build/progress.cppm` (program lines) | none | +| T3 | R1, R2, R3 | `src/ui.cppm`, `modules/platform/src/terminal.cppm`, `src/build/execute.cppm` (the `run` path only), the e2e scripts of the R3-0 census | none | +| T4 | K1 | `src/pack/pipeline.cppm`, `src/cli/cmd_publish.cppm`, `src/cli.cppm` (the options of `pack`), `src/build/prepare.cppm` (the stage directory per member), one hunk in `step9_member_build_programs` and in `src/build/prepare/features.cpp` (the stage directory a program receives) | T1 (the selection module), T2 (the final form of `step9`) | +| T5 | integration | `mcpp.toml` and `modules/versioning/src/version.cppm` (the version), `CHANGELOG.md`, `docs/` and `docs/zh/`, this record | T1 to T4 | + +- T1, T2 and T3 proceed at the same time, each on its own branch from the + integration branch. +- They are merged in the order T2, T1, T3, so that the census of T3 is taken + against the scripts T1 and T2 add. New scripts capture both streams when + they assert a status line, so that they hold before and after T3. +- T4 starts on the integration branch once T1 and T2 are merged. +- T5 takes the full e2e suite on the merged branch, locally in shards, then + opens the pull request. + +## 14. Cross-repository work and verification + +| Repository | Change | When | +|---|---|---| +| mcpp-community/mcpp | the pull request of section 9, then a release by tag | first | +| xlings-res/mcpp (GitHub and GitCode) | release assets mirrored by `publish-ecosystem`. The GitCode assets are uploaded with a local `gtc` as soon as each appears on the GitHub release | during the release | +| openxlings/xim-pkgindex | the bump pull request that `publish-ecosystem` opens, merged by a maintainer | after the mirror is verified by GET | +| mcpp-community/mcpp-index | `MCPP_VERSION` and `latest_mcpp` move to the new version, and a full scan runs by `workflow_dispatch` | after the index entry is live | +| openxlings/xlings | none. `kXlingsVersion` 2026.9.30.1 is xlings' latest release. xlings' CI reads mcpp's exit status and not its status lines | none | +| mcpp-language-server | none. It reads the `emit build-database --format json` envelope from stdout and the exit status | none | + +The release canaries (xlings, mcppls) build with the tagged mcpp before the +platform builds start. + +**Verification of the released artifact.** In an xlings subos sandbox, with +the released mcpp addressed at its store path and the CN mirror set inside +the sandbox (`mcpp self config --mirror CN`), a script runs: + +1. `mcpp test -p a -p b` on a three-member workspace: both members tested, + the third untouched (#750). +2. `--exclude`, and the refusal of an unknown member. +3. A workspace of four members whose programs import one host module: the + host module compiled once (#748). +4. `mcpp pack --workspace` over two members that share a member with a build + program: each program run once, and one package per member (#749). +5. `mcpp run -q` with stdout redirected: exactly the program's output, and + exit 101 for a compile error. + +Each section reports ok, failed or not run separately. The script's result is +posted on #748, #749 and #750, which are then closed. + +## 15. Implementation record (2026.10.1.1) + +The four tasks of section 13 were implemented on separate branches and merged +into one integration branch in the order T1, T3, T2, T4, followed by an +independent review of the merged code and its corrections. + +### 15.1 B0, measured + +A virtual workspace of four members, each with a build program that imports +one host module of a path dependency, which imports `std` and `mcpp`; clang +22.1.8, four jobs. Compiler time per step, summed from a trace of the build: + +| Step | Before | After, first build | After, second workspace | +|---|---|---|---| +| bundled `mcpp` module | 16 commands, 0.559 s | 4 commands, 0.118 s | none | +| host module | 8 commands, 0.245 s | 2 commands, 0.046 s | 2 commands, 0.054 s | +| `build.mcpp` compile and link | 4 commands, 0.297 s | 0.306 s, concurrent | 0.290 s | + +Steps 1 and 3 of F3 are 73% of the compiler time and all of the preparation +outside a program's own compile, which confirms the attribution; the `std` +module is a cache hit. The whole build took 0.89 to 1.22 s before, 0.48 s +after on a cold home and 0.33 to 0.39 s on a warm one. On this machine the +preparation is about 0.2 s per program against #748's 7.8 s on a Windows +runner, so the measurement confirms the shape, not the absolute cost. The +bundled module's `base` flags contain only payload paths below the home, none +of the project's (section 11, item 10). + +### 15.2 Departures from the plan + +- **Selection.** `--workspace` together with `-p` is refused, naming both, + like the other contradictions of S1; the S1 table did not list it. `mcpp + test --list` over several members keeps its per-member form. +- **The test stream.** Revision 3 kept `duration_ms` as "build+run". The code + of 2026.9.30.2 recorded the run of a test that ran and the build of a + `compile_fail`, and that is the meaning kept; the documentation's + "build+run" is corrected. The build time of a test's own binary is the + added field `build_ms`. +- **The store key (B1).** Besides the inputs of section 4, the key of a + workspace entry includes a digest of the provider package's whole tree, so + that an edit of a header the interface includes is not served a stale BMI. + A tree of more than 4096 files or 64 MiB is keyed for the process only and + loses reuse across commands. The compile code moved from + `hostprogram.cppm` into `src/build/host_module_compile.cppm`, and the store + is `src/build/host_module_store.cppm`. +- **Result lines (R3).** A result line is written by `mcpp::ui::result`, + which the caller chooses; the stream is not inferred from a verb. +- **Pack (K1).** + - A bare `mcpp pack` at a virtual root keeps its behaviour of 2026.9.30.2 and + packs the first member with a program; several members are packed with + `--workspace`, `--exclude` or several `-p`. + - An action belongs to the package that submitted it. A package acts for + itself when it is a packed member, otherwise for the one packed member + whose closure reaches it. A provider that several packed members reach is + refused, because one run of its program cannot stage a tree per member. + - An `--output` that does not exist is created as a directory. + - `member_request` and `workspace_groups` moved into `mcpp.cli.selection`: + an import of `mcpp.cli.cmd_build` from `mcpp.cli.cmd_publish` made GCC + 16.1 fail with an internal compiler error. + - A provider's program runs once in each of the two passes a dispatched + format has always had; a shared member that provides nothing runs once. + +### 15.3 Review + +An independent review of the merged code found no data race, deadlock or +reuse of a stale compile. It found, and the integration corrected: + +- a member of a group whose package failed after the group's first failure + was reported without its own diagnostics; +- the `duration_ms` change above; +- a GCC host-module entry that kept the copies of the imported BMIs staged for + its compile, the `std` module's among them; +- the `Finished` breakdown, which added the overlapping compile times of the + concurrent phase; the phase is now counted once, as its wall time, and it + states no warning of its own; +- the announcement of a multi-member test plan, which named the virtual root; +- comments and documentation that no longer described the code. + +A second review, of the pack change alone, found no change to the pack of one +member, and three defects in the pack of several, which are corrected (e2e 870): + +- `${mcpp.target_file:}` resolved to the last link unit of that name in + the plan, so a member's distributable could be built from another member's + program of the same target name. The name is now resolved for the member the + action's package acts for, and refused when that is none of the members that + define it. +- One member's failing distribution step, or a member without a staged tree + whose provider reads it, failed every member of its configuration. The drive + keeps going when several members are packed, each member's declared products + are removed before it, so that a file present after it is this pack's, and a + member without a tree is failed alone and the pass prepared again for the + others. +- The members were reported in the order of their configurations. They are + reported in `[workspace] members` order. + +### 15.4 Verification before the pull request + +- Unit tests: 142 passed. +- The e2e scripts added by this plan (852 to 870) pass under clang and, for + those that depend on the family, under GCC. +- The full e2e suite on the integration of T1 to T3, on a machine whose + default toolchain is clang: 473 passed, 19 failed, 61 skipped. The 19 fail + with the same message on 2026.9.30.2; 17 of them pass with GCC selected, and + the other two need an Android NDK or a Windows host. +- The sandbox script `.agents/docs/2026-10-01-member-selection-verify.sh`, + run on the host against the integration binary: every section passes. + Against 2026.9.30.2, every section marked CHANGE fails and every other + section passes. + +### 15.5 Open + +- #751 (section 8). +- The Windows and macOS paths of B2-0, B1 and R3 are exercised by CI only. diff --git a/.agents/docs/2026-10-01-member-selection-verify.sh b/.agents/docs/2026-10-01-member-selection-verify.sh new file mode 100644 index 000000000..edf0e64c3 --- /dev/null +++ b/.agents/docs/2026-10-01-member-selection-verify.sh @@ -0,0 +1,156 @@ +#!/usr/bin/env bash +# Sandbox verification of the release that implements +# .agents/docs/2026-09-30-member-selection-and-build-program-cost-plan.md +# (#748, #749, #750), run against the PUBLISHED release inside an xlings +# sandbox: +# +# B64=$(base64 -w0 .agents/docs/2026-10-01-member-selection-verify.sh) +# xlings subos new v1001 2>/dev/null || true +# xlings subos use v1001 --sandbox --cmd "echo $B64 | base64 -d > /tmp/v.sh && VER= bash /tmp/v.sh" +# +# VER selects the release under test. Running it with VER=2026.9.30.2 is the +# control: every section marked CHANGE must fail there, and every other +# section must pass on both. +# +# Every probe directory is removed at the start of its section, because the +# sandbox's $HOME persists between runs of the same subos. A section that does +# not run is reported as SKIP and counted apart from a pass. +set -u +VER="${VER:?VER=}" +W="${W:-$HOME/v1001}" +fails=0; passes=0; skips=0 +pass() { echo "PASS $1"; passes=$((passes+1)); } +fail() { echo "FAIL $1"; [ -n "${2:-}" ] && [ -f "$2" ] && tail -20 "$2"; fails=$((fails+1)); } +skip() { echo "SKIP $1"; skips=$((skips+1)); } + +# 0. The release under test, from the published channel, with the CN mirror. +if [ -n "${MCPP_OVERRIDE:-}" ]; then + MCPP="$MCPP_OVERRIDE" +else + xlings config --mirror CN >/dev/null 2>&1 || true + xlings update >/dev/null 2>&1 || true + xlings install "mcpp@$VER" -y > /tmp/v1001-install.log 2>&1 || true + MCPP="$HOME/.xlings/data/xpkgs/xim-x-mcpp/$VER/bin/mcpp" +fi +if [ ! -x "$MCPP" ]; then + echo "FATAL: mcpp $VER is not installable from the index"; tail -20 /tmp/v1001-install.log; exit 2 +fi +got=$("$MCPP" --version 2>&1 | head -1) +case "$got" in *"$VER"*) pass "0 installed: $got";; *) fail "0 version: $got";; esac +# The sandbox's home is empty, so the mirror is set inside it. A run on a host +# with MCPP_OVERRIDE leaves the host's configuration as it is. +[ -n "${MCPP_OVERRIDE:-}" ] || "$MCPP" self config --mirror CN >/dev/null 2>&1 || true +mkdir -p "$W" + +lib_member() { # : a library member with one module and one test + mkdir -p "$1/src" "$1/tests" + printf '[package]\nname = "%s"\nversion = "0.1.0"\n\n[targets.%s]\nkind = "lib"\n' "$2" "$2" > "$1/mcpp.toml" + printf 'export module %s;\nexport int %s_value() { return %s; }\n' "$2" "$2" "$3" > "$1/src/$2.cppm" + printf 'import %s;\nint main() { return %s_value() == %s ? 0 : 1; }\n' "$2" "$2" "$3" > "$1/tests/test_$2.cpp" +} + +# 1. CHANGE (#750): a repeated -p selects every member it names. +rm -rf "$W/s1"; mkdir -p "$W/s1"; cd "$W/s1" +printf '[workspace]\nmembers = ["a", "b", "c"]\n' > mcpp.toml +lib_member a a 1; lib_member b b 2; lib_member c c 3 +if "$MCPP" test -p a -p b > s1.log 2>&1 \ + && grep -q 'test_a' s1.log && grep -q 'test_b' s1.log && ! grep -q 'test_c' s1.log; then + pass "1 CHANGE: mcpp test -p a -p b tests a and b, and not c" +else fail "1 CHANGE: a repeated -p" s1.log; fi + +# 2. CHANGE: --exclude, and a member name that matches nothing. +cd "$W/s1" +if "$MCPP" test --workspace --exclude c > s2a.log 2>&1 \ + && grep -q 'test_a' s2a.log && grep -q 'test_b' s2a.log && ! grep -q 'test_c' s2a.log; then + pass "2a CHANGE: --workspace --exclude c tests a and b" +else fail "2a CHANGE: --exclude" s2a.log; fi +if "$MCPP" build -p nosuch > s2b.log 2>&1; then fail "2b an unknown member was accepted" s2b.log +elif grep -q 'nosuch' s2b.log; then pass "2b an unknown member is refused by name" +else fail "2b the refusal does not name the member" s2b.log; fi + +# 3. CHANGE (R1, R2): run -q writes exactly the program's output, and a +# failed build exits 101. +rm -rf "$W/s3"; mkdir -p "$W/s3/src"; cd "$W/s3" +printf '[package]\nname = "p3"\nversion = "0.1.0"\n\n[targets.p3]\nkind = "bin"\nmain = "src/main.cpp"\n' > mcpp.toml +printf '#include \nint main() { std::puts("OUT"); std::fputs("ERR\\n", stderr); return 0; }\n' > src/main.cpp +"$MCPP" run -q > s3.out 2> s3.err; rc=$? +if [ "$rc" = 0 ] && [ "$(od -c s3.out | head -1)" = "$(printf 'OUT\n' | od -c | head -1)" ]; then + pass "3a CHANGE: run -q writes exactly the program's stdout" +else fail "3a CHANGE: run -q stdout (rc=$rc)" s3.out; fi +printf 'int main() { x }\n' > src/main.cpp +"$MCPP" run -q > s3b.log 2>&1; rc=$? +if [ "$rc" = 101 ]; then pass "3b CHANGE: a run whose build failed exits 101" +else fail "3b CHANGE: exit status $rc for a failed build" s3b.log; fi + +# 4. CHANGE (R3): a build narrates on stderr. +cd "$W/s1" +out=$("$MCPP" build --workspace 2>/dev/null) +if [ -z "$out" ] && "$MCPP" build --workspace 2>&1 >/dev/null | grep -q 'Finished'; then + pass "4 CHANGE: status lines are on stderr and stdout is empty" +else fail "4 CHANGE: status stream"; echo "$out" | head -5; fi + +# 5. index packages build and run with the release. +rm -rf "$W/s5"; mkdir -p "$W/s5/src"; cd "$W/s5" +printf '[package]\nname = "eco"\nversion = "0.1.0"\n\n[dependencies]\n"compat.zlib" = "*"\n"mcpplibs.cmdline" = "*"\n\n[targets.eco]\nkind = "bin"\nmain = "src/main.cpp"\n' > mcpp.toml +cat > src/main.cpp <<'EOF' +#include +#include +import mcpplibs.cmdline; +int main() { std::printf("zlib %s\n", zlibVersion()); return 0; } +EOF +if "$MCPP" run > s5.log 2>&1 && grep -q '^zlib ' s5.log; then + pass "5 compat.zlib and mcpplibs.cmdline from the index build and run" +else fail "5 index packages" s5.log; fi + +# 6. CHANGE (#748): four members' build programs import one host module of a +# path dependency; the module is compiled once for the four, and the engine's +# own module is kept in the global cache. +rm -rf "$W/s6"; mkdir -p "$W/s6/rules/src"; cd "$W/s6" +printf '[workspace]\nmembers = ["m1", "m2", "m3", "m4"]\n' > mcpp.toml +printf '[package]\nname = "rule6"\nversion = "0.1.0"\n\n[targets.rule6]\nkind = "lib"\n' > rules/mcpp.toml +printf 'export module v6.rules;\nimport mcpp;\nexport void apply(const char* d) { mcpp::define(d); }\n' > rules/src/rule6.cppm +for m in m1 m2 m3 m4; do + mkdir -p $m/src + printf '[package]\nname = "%s"\nversion = "0.1.0"\n\n[build-dependencies]\nrule6 = { path = "../rules", host-module = true }\n\n[targets.%s]\nkind = "bin"\nmain = "src/main.cpp"\n' $m $m > $m/mcpp.toml + printf 'int main() { return 0; }\n' > $m/src/main.cpp + printf 'import mcpp;\nimport v6.rules;\nint main() { apply("V6_%s=1"); return 0; }\n' $m > $m/build.mcpp +done +if MCPP_VERBOSE=1 "$MCPP" build --workspace > s6.log 2>&1; then + compiles=$(grep -cE "host module 'v6\.rules' (precompile|compile):" s6.log || true) + ran=$(grep -cE "^ *build\.mcpp m[1-4] .* ran " s6.log || true) + if [ "$compiles" = 1 ] && [ "$ran" = 4 ]; then + pass "6 CHANGE: one host module compile for four build programs" + else fail "6 CHANGE: $compiles host module compiles for $ran programs" s6.log; fi +else fail "6 the workspace of four build programs did not build" s6.log; fi + +# 7. CHANGE (#749): one pack of two program members that share a member with a +# build program: one package per member, and the shared program runs once. +rm -rf "$W/s7"; mkdir -p "$W/s7/core/src"; cd "$W/s7" +printf '[workspace]\nmembers = ["core", "cli", "gui"]\n' > mcpp.toml +printf '[package]\nname = "core"\nversion = "0.1.0"\n\n[targets.core]\nkind = "lib"\n' > core/mcpp.toml +printf 'export module v7core;\nexport int core_answer() { return 42; }\n' > core/src/core.cppm +cat > core/build.mcpp <<'EOF' +import mcpp; +#include +#include +int main() { + std::ofstream log(std::string(mcpp::out_dir()) + "/runs.log", std::ios::app); + log << "ran\n"; + return 0; +} +EOF +for m in cli gui; do + mkdir -p $m/src + printf '[package]\nname = "%s"\nversion = "0.1.0"\n\n[dependencies]\ncore = { path = "../core" }\n\n[targets.%s]\nkind = "bin"\nmain = "src/main.cpp"\n' $m $m > $m/mcpp.toml + printf '#include \nimport v7core;\nint main() { std::printf("%s %%d\\n", core_answer()); }\n' $m > $m/src/main.cpp +done +if "$MCPP" pack --workspace --format dir > s7.log 2>&1; then + packed=$(grep -cE '^ *Packed ' s7.log || true) + runs=$(find core -name runs.log -exec cat {} + 2>/dev/null | grep -c ran || true) + if [ "$packed" = 2 ] && [ "$runs" = 1 ]; then + pass "7 CHANGE: one pack of two members, and the shared build program ran once" + else fail "7 CHANGE: $packed packages, the shared program ran $runs times" s7.log; fi +else fail "7 CHANGE: mcpp pack --workspace" s7.log; fi + +echo "---- $passes passed, $fails failed, $skips skipped" +[ "$fails" = 0 ] diff --git a/.agents/docs/README.md b/.agents/docs/README.md index 203802292..c357062c8 100644 --- a/.agents/docs/README.md +++ b/.agents/docs/README.md @@ -18,7 +18,7 @@ superseded_by: 2026-09-07-....md # when status is superseded --- ``` -320 records. +321 records. ## By subject @@ -30,6 +30,7 @@ Records that declare one. Everything else is listed by date below. ### design +- [Member selection, build programs prepared once, a pack over several members, and the output streams of `mcpp run`: the plan for the release after 2026.9.30.2 (#748, #749, #750)](2026-09-30-member-selection-and-build-program-cost-plan.md) — landed - [The build's wall time, its progress count, a hang after the build, and #732 and #744: measurements and a remediation plan](2026-09-30-build-wall-time-progress-count-and-hang-plan.md) — landed - [Build output, revision 3: every package that does work is named, the live display is one line drawn in one write, and a repeated warning is stated once per file](2026-09-30-build-output-refinement-design.md) — landed - [The workspace as the unit of build: one graph per configuration, one scheduler, product directories, and a reusable graph module](2026-09-29-workspace-build-graph-design.md) — landed @@ -111,6 +112,7 @@ Records that declare one. Everything else is listed by date below. ### 2026-09 +- [Member selection, build programs prepared once, a pack over several members, and the output streams of `mcpp run`: the plan for the release after 2026.9.30.2 (#748, #749, #750)](2026-09-30-member-selection-and-build-program-cost-plan.md) — landed - [The build's wall time, its progress count, a hang after the build, and #732 and #744: measurements and a remediation plan](2026-09-30-build-wall-time-progress-count-and-hang-plan.md) — landed - [Build output, revision 3: every package that does work is named, the live display is one line drawn in one write, and a repeated warning is stated once per file](2026-09-30-build-output-refinement-design.md) — landed - [The workspace as the unit of build: one graph per configuration, one scheduler, product directories, and a reusable graph module](2026-09-29-workspace-build-graph-design.md) — landed diff --git a/.github/workflows/ci-windows.yml b/.github/workflows/ci-windows.yml index 8d3534181..e4fc311e4 100644 --- a/.github/workflows/ci-windows.yml +++ b/.github/workflows/ci-windows.yml @@ -394,7 +394,9 @@ jobs: # the global default in `toolchain list` / doctor output. TMP=$(mktemp -d); cd "$TMP" - out=$("$MCPP_SELF" toolchain default msvc); echo "$out" + # Both streams: `Detected` and `Default set to` are narration, on + # standard error since 2026.10.1.1. + out=$("$MCPP_SELF" toolchain default msvc 2>&1); echo "$out" grep -q "Detected" <<<"$out" grep -q "msvc@system" <<<"$out" diff --git a/CHANGELOG.md b/CHANGELOG.md index 8b5543584..87425db74 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -4,6 +4,98 @@ > Each `## []` section is that release's notes. Entries are written in English > from 2026.9.28.3 on; earlier entries remain as written. +## [2026.10.1.1] - 2026-10-01 + +This release implements the plan for member selection, build programs prepared +once, a pack over several members, and the output streams +(`.agents/docs/2026-09-30-member-selection-and-build-program-cost-plan.md`), +and resolves mcpp#748, mcpp#749 and mcpp#750. Two changes are observable by +scripts: narration moves to standard error, and a `mcpp run` whose build failed +exits 101 (both under **Changed**). + +### Changed + +- **Narration is written to standard error, and a command's result to + standard output.** The narration is what says what a command is doing: the + status lines that begin with a verb (`Resolving`, `Compiling`, `Finished`, + `Running`, `Packed`, `Downloading` and the others), the progress bars, the + status row of a terminal, and the blank line after `Running`. The result is + what the command was asked to produce: the program's output under `mcpp run`, + a document or a listing, and for `mcpp test` each verdict, the `test result` + line and the `workspace result` line. Releases up to 2026.9.30.2 wrote the + narration to standard output. A script that read a status line from standard + output changes: `mcpp build | tee log` becomes `mcpp build 2>&1 | tee log`. A + CI log, which captures both streams, is unaffected. The live status row is + drawn on standard error, whether or not standard output is a terminal + (e2e 864 to 866). +- **A `mcpp run` whose planning or build failed exits 101**, as `cargo run` + does. It exited 1 or 2, which a program that returns 1 also does. The + program's own status still passes through, a refused start keeps 125 to 127, + and `build`, `test` and `pack` keep their statuses. A program or runner that + itself returns 101 reads as a failed build (e2e 863). +- **`mcpp test` over several members plans once per configuration.** The + members of one configuration are one plan with each member's tests, so a + package they share is compiled once with the union of their features and its + build program runs once; then each member's tests run in `[workspace] + members` order with that member's runtime directories, continuing past a + member that fails. The JSON stream adds a `group_build` record before a + group's first test and a `build_group` field in each member's summary, whose + `build_ms` is the group's build time; each test record adds `build_ms`, the + build time of its own binary. `--workspace-timeout` bounds the runs and + `--build-timeout` the build (e2e 854 to 856). +- **A workspace's build programs share what they import and compile at the + same time (mcpp#748).** The bundled `mcpp` module and each host module are + compiled once per key: the host compiler, the standard and every flag of the + compile, the imported BMIs and the interface's content. The engine's module + and the host modules of index packages are kept in the global cache, where + `mcpp cache gc` collects them; a host module of a path or git dependency or + of a workspace member is kept under the workspace's + `target/.build-mcpp/host-modules/`. The members' programs are compiled + together, up to the job count, and run in the order they always ran, so the + plan is the one a serial build writes. On a fixture of #748's shape the whole + build takes 0.33 to 0.48 s instead of 0.89 to 1.22 s (e2e 857 to 861). + +### Added + +- **A repeated `-p` selects every member it names (mcpp#750).** `build`, + `test`, `pack` and `mcpp emit build-database` read one selection: a set of + members in `[workspace] members` order, whatever the order of the `-p` + values. A value that names no member is refused before planning, and the + refusal lists the members. `mcpp run` executes one program and refuses a + second `-p`, naming both; `--workspace` together with `-p` is refused + (e2e 852, 853). +- **`--exclude `** removes members from a whole-workspace selection + (`--workspace`, or a virtual root without `-p`), on `build`, `test`, `pack` + and `mcpp emit build-database`. It is refused with `-p`, for a name that + matches no member, and when it leaves no member (e2e 853). +- **`mcpp pack` over several members (mcpp#749).** `--workspace`, a repeated + `-p` and `--exclude` pack several members with one plan per configuration and + one build; each member is staged in a tree of its own, its build program sees + its own `pack_stage_dir()`, and a dispatched format runs one second pass per + configuration. A positional target, several `--target` values, an `--output` + that is a file, a member no provider of the requested format acts for, and + two members that would write one destination are refused before anything is + compiled. A member whose distribution step fails is reported by name and the + others are packed; the members are reported in `[workspace] members` order, + and `${mcpp.target_file:}` names the target of the member an action is + for. `mcpp pack -p ` keeps its meaning. The JSON envelope adds + `data.stages` and a `member` field per artifact when several members are + packed (e2e 867 to 870). + +### Fixed + +- **A repeated `-p` no longer keeps only its last value (mcpp#750).** + `mcpp test -p a -p b` tested `b` alone and exited 0. +- **Starting a child process is safe for concurrent callers.** A pipe created + by one thread could be inherited by a child another thread started, and the + first thread's reader then waited for end of file until the unrelated child + exited. Pipes are created close-on-exec on Linux; on macOS and Windows the + creation of a pipe, the start of the child and the parent's close of the + write end form one critical section. The registry of children for signal + forwarding is serialised and holds 256 entries. +- **`mcpp run -q` writes exactly the program's standard output.** It began + with an empty line (e2e 862). + ## [2026.9.30.2] - 2026-09-30 This release answers five reports on 2026.9.30.1 while building xlings diff --git a/docs/04-mcpp-toml.md b/docs/04-mcpp-toml.md index f8d58a023..ab90baa69 100644 --- a/docs/04-mcpp-toml.md +++ b/docs/04-mcpp-toml.md @@ -1391,6 +1391,13 @@ What is **not** cached: `path` and `git` dependencies, at any depth, and workspace members. Their sources can change without their `name@version` changing, so no key over that identity could notice the change. +The modules a build program imports follow the same rule (2026.10.1.1+). The +bundled `mcpp` module and the host modules of index packages are kept in the +global cache, and a host module of a `path` or `git` dependency or of a workspace +member is kept under `target/.build-mcpp/host-modules/`, never in the global +cache. `--cache local` and `--cache off` keep all of them in the project. See +[30 — Placement of compiled host modules](30-build-mcpp.md). + Inspection and reclamation: ``` diff --git a/docs/07-workspace.md b/docs/07-workspace.md index 99f6854ac..1626bb292 100644 --- a/docs/07-workspace.md +++ b/docs/07-workspace.md @@ -326,18 +326,22 @@ a descriptor generator rather than by the person reading the message. ```bash mcpp build # virtual workspace → builds ALL members; rooted → the root package mcpp build -p server # build a specific member and its dependencies +mcpp build -p server -p cli # build several members, in one plan mcpp build --workspace # build every member explicitly +mcpp build --workspace --exclude legacy # every member but legacy mcpp test # virtual workspace → tests ALL members; rooted → the root package mcpp test -p core # test a single member +mcpp test -p core -p http # test several members: one plan, one build, one report per member mcpp test --workspace # test every member (one report per member; continues past failures) ``` At a **virtual** workspace root (only `[workspace]`, no `[package]`), bare `mcpp build` / `mcpp test` act on **all** members. At a **rooted** workspace (`[package]` + `[workspace]`), they act on the root package; `--workspace` -acts on the root package and every member. `mcpp test --workspace` builds + runs each member's -`tests/**/*.cpp` independently — discovery is scoped per member, so two members may -each have a `tests/main.cpp` without colliding. +acts on the root package and every member. `mcpp test` over several members +plans them together and builds once (§5.4), then runs each member's +`tests/**/*.cpp` — discovery is scoped per member, so two members may each have +a `tests/main.cpp` without colliding. ### 5.2 Building from a Member Subdirectory @@ -350,9 +354,9 @@ mcpp searches upward from the current directory; if it finds an `mcpp.toml` cont ### 5.3 The `-p, --package` Option -`-p` works with `build`, `test`, `run`, and other commands to select the target -member. Its value is resolved in one order, because the option names a -*package*: +`-p` works with `build`, `test`, `run`, `mcpp emit build-database` and other +commands to select the target member. Its value is resolved in one order, +because the option names a *package*: 1. a member's qualified name, `.` (only meaningful for a member that declares a namespace); @@ -372,32 +376,76 @@ selects the member named by the package, with a warning naming the other one — the option promises a package, so an exact package-name match outranks a directory that merely happens to share the spelling. -`--workspace` (on `build` and `test`) is the fan-out form: it acts on **every** -member. `mcpp test --workspace` reports each member separately and continues past a -failing member, exiting non-zero if any member failed — ideal as a single, -shell-free CI step for a workspace that tests many libraries. +#### Several members (mcpp 2026.10.1.1+) + +`-p` may be repeated on `build`, `test` and `mcpp emit build-database`. Each +value names one member, resolved by the order above, and the command acts on +all of them. The selection is a **set**, kept in `[workspace] members` order +whatever order `-p` was written in; a member named twice, by two spellings, is +selected once. + +```bash +mcpp build -p server -p cli # the same members as -p cli -p server +mcpp test -p core -p http +``` + +A value that names no member is refused before anything is planned, and the +refusal lists the members. `-p` together with `--workspace` is refused as +well: the two state two selections, and neither is taken over the other. +`mcpp run` executes one program, so it acts on one +member: a second `-p` is refused, naming every member asked for, and is never +read as "the last one". + +#### Leaving members out: `--exclude` + +```bash +mcpp build --workspace --exclude legacy # every member but legacy +mcpp test --exclude legacy --exclude bench # at a virtual root: every member but two +``` + +`--exclude ` may be repeated on `build`, `test` and `mcpp emit +build-database`. Its value is resolved like `-p`, and it removes members from +a selection of every member: `--workspace`, or a virtual root without `-p`. +It is refused, before anything is planned, together with `-p`, where neither +form applies (inside a member, or at a rooted root without `--workspace`), for +a name that matches no member, and when it leaves no member. + +`--workspace` (on `build`, `test` and `mcpp emit build-database`) is the fan-out +form: it acts on **every** member. `mcpp test --workspace` reports each member +separately and continues past a failing member, exiting non-zero if any member +failed — ideal as a single, shell-free CI step for a workspace that tests many +libraries. A member fails alone whatever failed in it: a test, its package's +build, or its plan (a member that cannot be planned is planned without, and the +others are planned together again). #### The fan-out report ``` + Workspace building 97 members: libs/core, libs/http, ... + Workspace built members libs/core, libs/http, ... in 120.40s; slowest: obj/libs/jsc/tests/jsc.o 88.0s Workspace testing member 'libs/core' (3/97) test_paths ... ok (0.31s) - test result ok. 7 passed; 0 failed; finished in 9.50s (build 8.90s + run 0.60s) - Workspace member 'libs/core' (3/97) ok — 7 passed in 9.50s + test result ok. 7 passed; 0 failed; finished in 121.10s (build 120.40s + run 0.60s) + Workspace member 'libs/core' (3/97) ok — 7 passed, run 0.60s ... workspace result ok. 97 member(s); 412 passed; 0 failed; finished in 355.20s - slowest: libs/jsc 93.5s, libs/install 32.2s, libs/http 24.1s + slowest: libs/install 32.2s, libs/http 24.1s ``` -`M/N` progress, per-test durations, and a per-member time split into **build** vs -**run**. The split is the useful part: a member whose tests take milliseconds but -whose link takes 90 seconds looks identical to a slow test suite in a single merged -number, and only one of those is worth investigating. +`M/N` progress and per-test durations. The members of one configuration are +built once, so the build is reported once, in the group's line: its members, its +wall time, and the edges that took the most of it. That is the signal the +per-member split used to give: a member whose link takes 90 seconds, and not its +tests, is named by the group's `slowest:` edges. Each member's own line states +the time of its **run**, and the final `slowest:` line ranks members by it. `--message-format json` carries the same data as NDJSON. Every test record is -member-qualified (`"member"`), and the stream ends with a `workspace_summary` -record naming the failed and not-run members — a bare test name is ambiguous the -moment two members both have a `smoke`. +member-qualified (`"member"`), a `group_build` record states each group's build +before its first test record, each member's summary names its group +(`build_group`), and the stream ends with a `workspace_summary` record naming +the failed and not-run members — a bare test name is ambiguous the moment two +members both have a `smoke`. The fields are in +[50 — Machine-Readable Output](50-machine-output.md#mcpp-test---message-format-json--the-test-stream). #### Bounding the fan-out @@ -407,11 +455,20 @@ mcpp test --workspace --build-timeout 300 # per-ninja-drive deadline (default 0 mcpp test --workspace --workspace-timeout 1800 # whole fan-out (default 0 = no limit) ``` -The fan-out is serial, so an unbounded member stalls every member after it. All -three deadlines report rather than abort: a timed-out test fails that test and the -fan-out continues; a timed-out build fails that member; `--workspace-timeout` stops -the fan-out and lists what did not run instead of leaving the CI job to kill the -process (which discards everything it had to say). +The members' runs are serial, so an unbounded member stalls every member after +it. All three deadlines report rather than abort: a timed-out test fails that +test and the fan-out continues; a timed-out build fails the members that waited +for it; `--workspace-timeout` stops the fan-out and lists what did not run +instead of leaving the CI job to kill the process (which discards everything it +had to say). + +`--workspace-timeout` bounds the runs, and it is measured from the start of the +command: it is checked before each member's tests start, and a member not +started by then is listed as not run (2026.10.1.1+). The members of a group are +built in one step, which the deadline cannot interrupt, so a workspace whose +build alone exceeds it runs to the end of that build, bounded by +`--build-timeout`, and then starts no member. Before 2026.10.1.1 each member was +built and run in turn, and the deadline could stop the fan-out between builds. ### 5.4 One graph per configuration (mcpp 2026.9.29.1+) @@ -428,7 +485,8 @@ member that several members use is compiled once. directory. - **Selection.** `--workspace`, and a virtual root without `-p`, select every member. `-p X`, and a command run in X's directory, plan X and what X - reaches. The two share the build directory, so `mcpp build --workspace` + reaches; `-p X -p Y` plans both together, as one selection (§5.3). The + selections share the build directory, so `mcpp build --workspace` followed by `mcpp build -p X` compiles nothing, and a package is compiled again only when its active features differ between the two commands. - **Flags.** A member's `cflags`, `cxxflags`, `ldflags` and defines apply to @@ -454,7 +512,20 @@ member that several members use is compiled once. into that member's programs and shared libraries only (2026.9.29.2+). - **Build programs.** The members' build programs run dependencies first, and a program's result is reused by every command whose inputs to it are - unchanged, whichever members the command selects (2026.9.29.5+). + unchanged, whichever members the command selects (2026.9.29.5+). Their + compiles run at the same time, up to the job count, and only their runs + follow that order, so the plan is the one a serial build writes; a host + module that several programs import is compiled once for all of them + (2026.10.1.1+; see [30 — Build programs](30-build-mcpp.md)). +- **Tests.** `mcpp test` over several members plans as `build` does: once per + configuration, with each member's tests, so a package the members share is + compiled once, its build program runs once, and its features are the union + the selection asks for (2026.10.1.1+). The configuration's packages and test + binaries are built once; then each member's tests run, in member order, with + that member's runtime directories and no other member's. A test run after + `mcpp build --workspace` compiles nothing the build compiled, unless a + dev-dependency changes a package's features. `mcpp test` of one member is one + plan of that member, as it always was. - **Compile database.** `mcpp build --configure-only` and `mcpp emit build-database` plan as the build does, one plan per configuration with each member's tests, so a package the members share is described once per @@ -519,7 +590,9 @@ myproject/ - `mcpp.lock` at the workspace root records the resolution of every member. `mcpp build --workspace` writes the whole record; `mcpp build -p X` updates the entries of X's graph. -- A member's build program writes to `/target/.build-mcpp/`. +- A member's build program writes to `/target/.build-mcpp/`. The host + modules the programs import are compiled into the workspace's own + `target/.build-mcpp/host-modules/`, once for all of them. - Build directories a member held under its own `target/` with an earlier mcpp are not read; `mcpp clean --stale` removes them. diff --git a/docs/08-testing.md b/docs/08-testing.md index 04fe48492..bc523ab00 100644 --- a/docs/08-testing.md +++ b/docs/08-testing.md @@ -82,6 +82,31 @@ configuration it is meant to check rather than against the default one: `--build-timeout ` bounds the compile. A test that hangs is reported as a failure with its own name, not as a job that stopped. +The report is the command's result and is written to standard output: each +test's verdict, the output of a test that failed, the `test result` line and, for +`--workspace`, the `workspace result` line. The build's steps (`Compiling`, +`Running`) are narration and are written to standard error, so `mcpp test > report.txt` +collects the report, and `mcpp test 2>&1 | tee test.log` collects both +([09 — Commands by scenario](09-commands-by-scenario.md#output-streams)). + +### In a workspace + +```bash +mcpp test -p core -p http # the tests of two members +mcpp test --workspace --exclude legacy # every member's but legacy's +``` + +`-p` may be repeated, and `--exclude` leaves members out of `--workspace` or of +a virtual root's whole-workspace selection; both are resolved as +[07 — Workspaces](07-workspace.md#53-the--p---package-option) states. A test over +several members plans them together and builds once: the members of one +configuration are one build graph, so a package they share is compiled once, +and its build program runs once. Then each member's tests run, in +`[workspace] members` order, each with that member's own runtime directories. +The command continues past a member that fails, whether a test, its package's +build or its plan failed, reports each member, and exits non-zero if any did. +Discovery is scoped per member, so two members may each have a `tests/main.cpp`. + ## Tests that reach packages the artifact does not ```toml @@ -169,5 +194,7 @@ here is only that the flag exists and that the human format is the default. - `mcpp test --list` over a manifest that does not load lists `tests/**/*.cpp`, not the `[test] discover` set it cannot read. - `--build-timeout` is POSIX-only. -- `--workspace-timeout` bounds a `--workspace` fan-out and reports what did run; - it does not attribute the timeout to a member. +- `--workspace-timeout` bounds the runs of a `--workspace` fan-out and reports + what did run; it does not attribute the timeout to a member. It is checked + before each member's tests start, so it cannot interrupt the build the + members share, which `--build-timeout` bounds. diff --git a/docs/09-commands-by-scenario.md b/docs/09-commands-by-scenario.md index dca833e5f..70bd75666 100644 --- a/docs/09-commands-by-scenario.md +++ b/docs/09-commands-by-scenario.md @@ -31,7 +31,7 @@ them. Confusing them costs a full rebuild. | Store | Scope | Growth trigger | Emptied by | |---|---|---|---| | `target///` | one project | a configuration fingerprint changes and opens a new directory | `mcpp clean`, `mcpp clean --stale` | -| the build cache (`mcpp cache dir`) | the whole machine | any project compiles a dependency or a `std` module, or builds a host tool | `mcpp cache gc`, `mcpp cache prune`, `mcpp cache clean` | +| the build cache (`mcpp cache dir`) | the whole machine | any project compiles a dependency or a `std` module, or builds a host tool, or compiles the `mcpp` module its build program imports | `mcpp cache gc`, `mcpp cache prune`, `mcpp cache clean` | `mcpp clean` removes `target/` entirely, and the next build recompiles everything. `mcpp clean --stale` removes only the fingerprint directories that @@ -218,10 +218,12 @@ Every acquisition is reported by one renderer (2026.9.28.1+): - the clone of a `git` dependency; - the sandbox's first-run tools. -On a terminal each item is a bar drawn in place. When stdout is not a terminal, -as in a CI log or a pipe, each item prints one line when it starts, with its -size when known, and one line when it finishes, with its duration. That output -carries no carriage return and no erase sequence. `--quiet` prints neither. +On a terminal each item is a bar drawn in place. When standard error is not a +terminal, as in a CI log or when it is redirected, each item prints one line +when it starts, with its size when known, and one line when it finishes, with +its duration. That output carries no carriage return and no erase sequence. +`--quiet` prints neither. The bars are narration, and narration is written to +standard error (see "Output streams" below). An index refresh is reported step by step when the xlings that mcpp drives emits progress events for it (xlings 2026.9.28.1+). With an older xlings it @@ -296,8 +298,9 @@ On a terminal one status row is drawn below the output and updated in place: there. - Last comes the longest-running `check` or `prepare` action. ninja reports every other step only when it finishes. -- The row is first drawn half a second into the command. Every change leaves - in one write that overwrites the row in place, so the row never flickers. +- The row is drawn on standard error, and is first drawn half a second into the + command. Every change leaves in one write that overwrites the row in place, + so the row never flickers. `MCPP_PROGRESS` chooses the screen: @@ -307,10 +310,10 @@ On a terminal one status row is drawn below the output and updated in place: - `off`: no live row, only the lines of a log. The screen needs a terminal that draws braille: a UTF-8 locale, or Windows -Terminal. Elsewhere the row is plain. When the output is not a terminal (a CI -log, a pipe), only final lines are written, and the status row is written -when the output has been silent for a minute. `TERM=dumb` selects that form -on a terminal too. +Terminal. Elsewhere the row is plain. When standard error is not a terminal (a +CI log, a redirection), only final lines are written, and the status row is +written when the output has been silent for a minute. `TERM=dumb` selects that +form on a terminal too. `--play-game` plays a game on the screen while the build runs. It is accepted by `build`, `run` and `test`, and `--play-game=NAME` names the game; @@ -333,8 +336,9 @@ The game runs at its own speed; the counts beside it state the build. Keys are read without echo, and Ctrl-C still stops the build. The terminal's mode is restored when the build ends or is interrupted; a process killed outright cannot restore it, and `stty sane` does. The game needs standard input and -standard output on a terminal, with mcpp in its foreground (not a background -job); otherwise one line says why, and the build proceeds. +standard error (where the screen is drawn) on a terminal, with mcpp in its +foreground (not a background job); otherwise one line says why, and the build +proceeds. `--verbose` names every package: `Fresh` for those with nothing to do, and `Compiled` with the steps and span of each that did work. It also states each @@ -342,6 +346,65 @@ build program's compile and run times, and prints every step as ninja reports it (`[f/t] ` and its output). `--quiet` prints none of this. Machine output (`--message-format json`) is unchanged. +## Output streams + +Every command writes its **narration** to standard error and its **result** to +standard output (mcpp 2026.9.30.2 and earlier wrote the narration to standard +output). The narration is what says what the command is doing: the status lines +that begin with a verb, the progress bars, the status row, `Finished`, and the +blank line after `Running`. The result is what the command was asked to +produce: + +- under `mcpp run`, what the program prints, and nothing else; +- a document or a listing: `mcpp run --list-runners`, `mcpp toolchain list`, + `mcpp cache list`, `mcpp search`, `mcpp test --list`, `mcpp emit`; +- for `mcpp test`, the report: each test's verdict, the output of a test that + failed, the `test result` line, and for `--workspace` the `workspace result` + line. + +```console +$ mcpp run -q 2>/dev/null > out.txt # out.txt holds the program's output, byte for byte +$ mcpp build >/dev/null # the steps and Finished are still shown +$ mcpp build 2>&1 | tee build.log # record the steps: both streams on one pipe +``` + +- `mcpp build | tee build.log` no longer records the steps, which are on + standard error; write `mcpp build 2>&1 | tee build.log`. A CI log captures + both streams and is unaffected. +- `--quiet` suppresses the narration. Warnings, errors and compiler + diagnostics are written to standard error as before, and `--quiet` leaves + them. +- Both streams keep the order of the writes on one pipe, and on a terminal. +- On a terminal the status row is drawn on standard error. `mcpp build | less` + (standard output a pipe) draws it; `mcpp build 2>log` (standard error a file) + does not. +- Machine output (`--message-format json`, `--format json`) is unchanged: + standard output carries the document alone + ([50 — Machine-Readable Output](50-machine-output.md)). + +### The exit status of `mcpp run` + +`mcpp run` exits with the program's own status, passed through unchanged. A +`run` whose planning or build failed did not start the program, and exits +**101**, which is the status a failed `cargo run` build has: + +```console +$ mcpp run # a compile error +error: build failed in app v0.1.0 (.) +$ echo $? +101 +$ mcpp run -- 1 # a program that returns 1 +$ echo $? +1 +``` + +A program that could not be started keeps the status of the refusal (`127` +not found, `126` not executable, `125` otherwise), and `mcpp build`, `mcpp +test` and `mcpp pack` keep their statuses. A program, or a runner +(`--runner`, `[target.].runner`), that itself returns 101 reads as a +failed build; [50 — Machine-Readable Output](50-machine-output.md) §6 gives the +bands. + ## Validating a descriptor before publishing `mcpp xpkg parse` reads a descriptor with the resolver's own grammar, so what diff --git a/docs/10-pack-and-release.md b/docs/10-pack-and-release.md index 56b76134c..18d7c0cd0 100644 --- a/docs/10-pack-and-release.md +++ b/docs/10-pack-and-release.md @@ -148,6 +148,8 @@ mcpp pack --dev # the same, as `build` and `run` spell it mcpp pack --message-format json # one mcpp.pack envelope on stdout (mcpp 2026.9.16.1+) mcpp pack --no-strip # ship the artifacts as built mcpp pack -p app --format release # a workspace member, as if run in its directory +mcpp pack --workspace --format release # every member with a program, planned and built once +mcpp pack -p cli -p gui -o dist/ # two members, each archive below dist/ mcpp pack --debug-symbols dbg/ # write the separated *.debug files under dbg/ mcpp pack --format msi --features installer # activate root-package features for the pack ``` @@ -168,12 +170,82 @@ relative `-o` keeps meaning the directory the command was typed in. take, with the same precedence: `--profile` wins over either, on all three commands. +### Packing several members (mcpp 2026.10.1.1+) + +`-p` may be repeated, and `--workspace` and `--exclude` select members as they do +for `mcpp build` ([07 — Workspaces](07-workspace.md), §5.3). The members are +planned as one selection and built once, and each is then staged, and packed, in +a tree of its own: + +```bash +mcpp pack --workspace --format release # every member with a program +mcpp pack --workspace --exclude updater # every such member but one +mcpp pack -p cli -p gui --format release +``` + +Packing each member with its own `mcpp pack -p ` plans the graph, runs +the build programs and starts the build again for every member, including the +members they share. A pack over several members plans once for each +configuration (the grouping `mcpp build` uses), builds once, and runs each build +program once per pass: a package that several members reach, such as a shared +library with a build program, is prepared and compiled once, and its program +does not run again for the pack. + +What each member receives is what `mcpp pack -p ` gives it: + +- **The product.** A member's archive or tree is the one the member makes when + packed alone, below its own `target/dist/`. With `--output ` it is written + below the directory, which is created when it is missing, under the name it has + by default. +- **A staged tree of its own.** With a dispatched `--format`, the pack prepares + twice, as above, and the second pass tells each build program the staged tree + of the member it acts for. `mcpp::pack_stage_dir()` and `${mcpp.stage_dir}` + answer per member. A program acts for the member it belongs to; a package that + only one packed member reaches (a provider the member depends on) acts for that + member; a package that several packed members reach acts for none of them, and + is told neither the format nor a tree, so one run of its program serves them + all. +- **Its own distributable.** An `artifact` action the request introduced belongs + to the member whose program, or whose provider, submitted it. Each member's + outputs are verified to exist and are reported as `Packed` for that member. + +`--workspace` packs a member only if it has a program target, and says which it +skipped. A member named with `-p` that has none is refused. Members are packed +one configuration at a time and reported in `[workspace] members` order, whatever +order `-p` names them in. A member that +fails, for instance one whose provider submitted nothing for the requested +format, is reported by name and the others are packed; the exit status is +non-zero if any member failed. One member, whether named with one `-p` or by the +directory the command runs in, is packed exactly as before: `mcpp pack -p X` +plans X's closure alone. `mcpp pack` with no selector at a virtual workspace +root packs the first member that has a program, as it always did; `--workspace` +packs them all. + +**Refused before anything is compiled**, naming what was refused: + +| Input | Reason | +|---|---| +| a positional target name | it names a target of one package | +| more than one `--target` | a program is built for one target, and the several-target Android pack stages the legs of one member | +| `--output` that exists as a file | each member's archive or tree is written below the directory | +| a member named with `-p` that has no program target | there is nothing to pack for it | +| with a dispatched `--format`, a member no provider acts for | the format is provided by a package the member does not reach, or by a package several members reach | +| two members that would write one archive or tree | `--output` names one directory, and two packages of one name, version and target share an archive name | + +A package with a build program that provides a format for several members cannot +serve them: a provider receives one staged tree. Provide the format from each +member's own build program (a helper both call is enough), or pack the members +one at a time. + `--message-format json` (mcpp 2026.9.16.1+) prints one `mcpp.pack` envelope on stdout after the command finishes and sends every human line to stderr. Its `data.artifacts` holds each produced file or directory with its absolute path, its `type` (`file` or `directory`), the `--format` value and the triples of its legs; `data.stage` holds the staged tree, its manifest and whether the closure -was walked ([50 — Machine Output](50-machine-output.md)). `--format` names the +was walked ([50 — Machine Output](50-machine-output.md)). A pack of several +members lists every member's artifacts, each naming its `member`, leaves +`data.stage` null and names one tree per member in `data.stages` +(2026.10.1.1+). `--format` names the package format on this command, so machine output is asked for the way `mcpp test` asks for it. diff --git a/docs/30-build-mcpp.md b/docs/30-build-mcpp.md index 253a912b5..9886c0854 100644 --- a/docs/30-build-mcpp.md +++ b/docs/30-build-mcpp.md @@ -141,8 +141,9 @@ interface and belongs in the declarative manifest/descriptor Instead of printing raw strings, `build.mcpp` can be written **modules-first** — `import mcpp;`, no `#include` needed. The `mcpp` module is bundled in the -mcpp binary (so it always matches that mcpp's protocol) and is compiled on demand; -its functions just emit the directives above: +mcpp binary (so it always matches that mcpp's protocol) and is compiled on demand, +once per mcpp version and host compiler, into the build cache (see *Where compiled +host modules are kept* below); its functions just emit the directives above: ```cpp // build.mcpp @@ -1526,6 +1527,18 @@ A program whose result is reused did no work and has a line, `build.mcpp reads `Running` with the programs finished and scheduled, and names the running program with its state (`compiling` or `running`) and a clock. +**A workspace's programs are compiled at the same time and run one after +another** (2026.10.1.1+). A member's program runs after the programs of the +members it depends on, because it reads their directives, and that order is +kept: the directives are applied in it, so the plan, and with it `build.ninja`, +is the one a build that compiled the programs one by one would write. The +compiles have no such order, so every program whose result is stale is compiled +at once, up to the job count (`--jobs`, `MCPP_JOBS`, `[build] jobs`). The +`compiled` time of a program includes preparing what it imports, which is no +longer left to `plan`. A compile's output is printed whole; when several +programs fail to compile, the failure reported is the first in the order the +programs run in, whichever compile finished first. + ## Host tools from a dependency (mcpp 2026.8.5.1+) A package can build a binary its consumers need *at build time* — `protoc`, a @@ -1632,10 +1645,12 @@ import protobufgen; int main() { return protobufgen::generate({"schema"}) ? 0 : 1; } ``` -mcpp compiles that package's lib-root module **for the host, in the same -command as `build.mcpp`** — which is what makes the BMI usable at all, since a +mcpp compiles that package's lib-root module **for the host, with the same +flags as `build.mcpp`** — which is what makes the BMI usable at all, since a module interface is only importable by a compile that agrees with it on -standard, dialect and compiler identity. +standard, dialect and compiler identity. The compiled module is kept by key, so +it is compiled once for every program that agrees on those flags (see +*Placement of compiled host modules*). Rules are therefore versioned, testable and distributable through the package manager already in use, written in **C++** — no second language, which is the @@ -1670,6 +1685,43 @@ internal fork are all legitimate and indistinguishable from here. The lib root must be at `src/.cppm` (or wherever `[lib] path` points); a missing one is reported as *"host module 'x': no interface unit at …"*. +#### Placement of compiled host modules (2026.10.1.1+) + +The bundled `mcpp` module and the host modules a program imports are not +compiled into each program's own directory. Each is an entry of a store, +addressed by everything that reaches its compile: the host compiler's identity, +the standard flag, the flags the compile carries, the BMIs it imports, the +SHA-256 of the interface file, the mcpp version (for the bundled module) and the +providing package (for a host module). The inputs are recorded in the entry's +`entry.json`, and a hit compares them field by field, never the hash alone. Two +programs whose compiles agree on all of these share one entry, and a program +whose standard or flags differ has an entry of its own, so a BMI is never taken +by a compile that does not agree with it. + +Where an entry lives depends on where its text comes from: + +| Module | Kept in | Reason | +|---|---|---| +| the bundled `mcpp` module and its `mcpp.core` alias | the global cache: `$MCPP_HOME/build-cache/v1/pkg/_engine/mcpp-build-module@//` | the engine owns its text, which is identical in every project for one mcpp version and one host compiler | +| a host module of an index package whose sources are in the immutable store | the global cache: `…/pkg//@//` | the dependency cache's rule: a name and version identify the bytes | +| a host module of a `path` or `git` dependency, or of a workspace member | `/target/.build-mcpp/host-modules//` | its sources can change without its name and version changing | + +The cache mode applies as it does to dependencies (see +[04 §2.10](04-mcpp-toml.md)). With `--cache global`, the default, an entry goes to +the global cache where the table says so. With `--cache local` or `--cache off` +every entry is kept in the workspace's store, and the programs of one invocation +still share them. A host module of a `path` dependency is never written to the +global cache. The key of an entry in the workspace's store also holds a digest of +the package's source tree, so an edit to a header that the interface includes +compiles a new entry, and the old one is left where it is. + +An entry in the global cache is an entry like a dependency's. `mcpp cache list` +shows the bundled module as `_engine/mcpp-build-module@`, `mcpp cache +verify` checks it, and `mcpp cache gc` collects it by last use; an upgrade of mcpp +leaves the previous version's entry for `gc`. The workspace's store is removed +with its `target/` by `mcpp clean`. A program sees what it saw before: the same +modules, compiled with the same flags. + **A package may offer several rules, selected by features** (mcpp 2026.9.5.3+). Every module interface unit among the package's resolved `[build] sources` — including the sources a feature adds — is compiled as a host module under the diff --git a/docs/50-machine-output.md b/docs/50-machine-output.md index 012fd243c..b8fa80b47 100644 --- a/docs/50-machine-output.md +++ b/docs/50-machine-output.md @@ -137,6 +137,38 @@ the answer, and the exit code says the answer is a rejection. §1 still holds parse stdout, do not branch on the code — but a client that treats any non-zero exit as "no output" will discard a document it was given. +### Standard output carries the result, standard error the narration + +Every command writes to two streams, and the division is the same for all of +them. Releases up to 2026.9.30.2 wrote the narration to standard output, so a +consumer that read it there reads standard error now. + +| stream | carries | +|---|---| +| standard output | the command's **result**: what a program prints under `mcpp run`; a document (an envelope, `mcpp emit`, `mcpp run --list-runners`, `--version`, `--help`); a listing (`mcpp toolchain list`, `mcpp cache list`, `mcpp search`, `mcpp test --list`); and the report of `mcpp test`: each test's verdict, the output of a test that failed, the `test result` line, and the `workspace result` line of a workspace run | +| standard error | the **narration**: the status lines (`Resolving`, `Compiling`, `Finished`, `Running`, `Packing`, `Packed`, `Downloading`, `Updating` and the others that begin a line with a verb), the progress bars and the status row of a terminal, the blank line that follows `Running`, `warning:`, `error:`, `note:` and `tip:` lines, the compilers' diagnostics, and what `--verbose` adds | + +The test for a line is what it is for. A line that says what the command is +doing or has done is narration. A line that is what the command was asked to +produce is a result; Cargo prints libtest's report on standard output for the +same reason, and so does `mcpp test`. + +- **A pipe receives the result alone.** `mcpp run -q 2>/dev/null` writes + exactly the program's standard output, and `mcpp build >/dev/null` still + shows the steps. On a terminal both streams reach the screen, and the status + row is drawn on standard error, so `mcpp build 2>/dev/null` shows no row. +- **`--quiet` suppresses the narration**, including the blank line after + `Running`, and leaves warnings, errors and diagnostics. +- **The machine-readable modes** (`--format json`, `--message-format json`) + keep standard output for their document, as before. What they narrate on + standard error is unchanged: `mcpp test --message-format json` narrates + nothing, and `mcpp pack --message-format json` and `mcpp emit + build-database` narrate there while they plan and build. +- **Both streams on one pipe keep the order of the writes.** `mcpp build 2>&1 + | tee log` records the steps, and is the spelling for a script that used to + write `mcpp build | tee log` (a CI log, which captures both streams, is + unaffected). + ## 4. Effects — what a command does before it prints An IDE with an untrusted-workspace gate has to decide **before** running. @@ -204,14 +236,20 @@ same thing — one answer, two shapes. ## 6. Exit status -`mcpp run` REPORTS THE PROGRAM'S OWN EXIT STATUS. Three bands divide the space, +`mcpp run` REPORTS THE PROGRAM'S OWN EXIT STATUS. Four bands divide the space, and only the first belongs to the program: | range | meaning | |---|---| -| `0`–`124` | the program ran; this is its own status, passed through unchanged | +| `0`–`124` | the program ran; this is its own status, passed through unchanged. The exception is `101`, below | +| `101` | `mcpp run` could not build the program: its planning or its build failed, and nothing was started. This is the status Cargo gives a failed `cargo run` build | | `125`–`127` | the spawn was attempted and refused — `127` not found, `126` found but not executable, `125` anything else; `126` also answers `mcpp run --format ` for a distributable that is a directory and meets no runner, refused before the spawn with the same meaning (2026.9.14.2+) | -| `2` | mcpp refused before attempting anything: a usage, configuration or resolution error | +| `2` | mcpp refused the request before building anything, or after building it and before starting anything: a usage error, `--format` with `--no-runner`, a program or a runner that the project does not declare | + +Until 2026.9.30.2 a failed build exited `1` and a failed planning `2`, so that +a compile error and a program that returns `1` could not be told apart by a +script. A failed build is now `101`; `mcpp build`, `mcpp test` and `mcpp pack` +keep their statuses. Until 2026.9.4.3 every non-zero status was folded to `1`, so that `2` could mean "could not start" as distinct from "ran and failed". The distinction was worth @@ -221,13 +259,17 @@ well — so the command this project tells people to type could not be branched The middle band is the one `env`, `timeout` and `nice` already use and that shells document, so `126` and `127` arrive with their usual meanings rather than -as numbers this project allocated. +as numbers this project allocated. `101` is Cargo's, for the same reason. + +A PROGRAM MAY ITSELF EXIT `101` OR `125`–`127`, AND mcpp DOES NOT TRY TO +DISAMBIGUATE BY NUMBER. What separates the two is that a launcher or build +failure always writes a reason to stderr and a program's own status never does. +A client that must be certain should read stderr, or use `--format json` where +the status is a field rather than a channel. -A PROGRAM MAY ITSELF EXIT `125`–`127`, AND mcpp DOES NOT TRY TO DISAMBIGUATE BY -NUMBER. What separates the two is that a launcher failure always writes a reason -to stderr and a program's own status never does. A client that must be certain -should read stderr, or use `--format json` where the status is a field rather -than a channel. +With a runner (`--runner`, `[target.].runner`) the status that passes +through is the runner's, and a runner that itself returns `101` reads as a +failed build. This is accepted, as it is in Cargo. `mcpp test` is unchanged and remains `0` or `1`: it aggregates many programs, so there is no single status to pass through. Per-test codes are in the JSON @@ -539,10 +581,19 @@ to stderr, including what the build programs and tools the pack starts print. | field | | |---|---| | `artifacts` | one record per produced artifact: `path` (absolute), `type` (`file` or `directory`), `format` (the `--format` value, `tar` when omitted) and `targets` (the canonical triple of each leg that went into it). A dispatched format reports the terminal outputs of the actions the request introduced; a several-`--target` Android pack reports one artifact whose `targets` lists every leg | -| `stage` | the tree the artifact was made from: `dir`, `manifest` (the stage manifest below) and `closure` (`walked` or `not-walked`); `null` for a library package and when no tree was staged | +| `stage` | the tree the artifact was made from: `dir`, `manifest` (the stage manifest below) and `closure` (`walked` or `not-walked`); `null` for a library package, when no tree was staged, and for a pack of several members | +| `stages` *(2026.10.1.1+)* | present only for a pack of several members (`--workspace`, or `-p` repeated): one record per member in `[workspace] members` order, with `member` (the qualified package name) and the `dir`, `manifest` and `closure` of `stage` | + +For a pack of several members `artifacts` lists every member's artifacts, in the +same member order, and each record gains `member`, the qualified package name of +the member it belongs to. A pack of one member has neither `stages` nor +`member`: the envelope is the one it always was. A failure omits `data`, exits with the command's exit status and carries the -diagnostic code `MCPP_PACK_FAILED`; the reason is on stderr. The per-run +diagnostic code `MCPP_PACK_FAILED`; the reason is on stderr. A pack of several +members that failed for some of them carries one `MCPP_PACK_FAILED` diagnostic +more for each such member, naming it, and omits `data` as well: the members that +were packed are on disk and named on stderr. The per-run `effects` are `read-project`, `write-project` and `write-global-cache`, with `exec-build-script` when a build program ran. `--protocol-version` declares `init-mcpp-home`, `read-project`, `write-project`, `network`, @@ -556,9 +607,11 @@ mcpp test [pattern] [--workspace] --message-format json This stream predates the envelope of §2 and is not wrapped in it: it is NDJSON, one record per test as each finishes, then one summary record per member. A -`--workspace` run ends with one `workspace_summary` record. The §7 guarantees -apply to it — fields are added and never removed, and a field's meaning never -changes — and the fields below are the contract as of 2026.9.2.1. +`--workspace` run ends with one `workspace_summary` record. A test over several +members adds one `group_build` record per group, before the group's first test +record. The §7 guarantees apply to it — fields are added and never removed, and +a field's meaning never changes — and the fields below are the contract as of +2026.9.2.1, with the additions each row dates. Per test: @@ -569,7 +622,8 @@ Per test: | `status` | `pass`, `compile_fail`, `run_fail`, `not_run`, or `built` | | `exit_code` | the test's exit status; `0` for `not_run` and `built` | | `signal` | the signal number when the status encodes one, else `null` | -| `duration_ms` | build+run wall time of this test | +| `duration_ms` | the wall time of the step that decided the status: the run for a test that ran, the build for a `compile_fail` | +| `build_ms` | the build time of this test's own binary in this invocation, the sum of its link edge and its main unit's compile edge in `.ninja_log`; `0` when neither was rebuilt *(2026.10.1.1+)* | | `timed_out` | `true` when `--timeout` killed it (`run_fail`) | | `compile_output`, `run_output` | captured diagnostics | | `reason` | `not_run` only: why, in one sentence; `""` otherwise | @@ -583,6 +637,37 @@ Summary record, `{"summary": {...}}`: | `not_run_reason` | the reason shared by all of them, or `""` | | `built` | tests built under `--no-run`, which were not to be executed | | `elapsed_ms`, `build_ms`, `run_ms` | wall time, split | +| `build_group` | *(2026.10.1.1+)* present when the member was planned with others: the `group` of the `group_build` record whose build its tests waited for | + +`build_ms` is the wall time this member's tests waited for their build. For a +member planned alone that is its own build. For a member planned with others it +is the build of its whole group, the same number for every member of the +group, and a consumer that sums `build_ms` over members deduplicates by +`build_group`; a consumer that does not sum it is unaffected. `elapsed_ms` is +the wall clock for the whole member, planning and the group's build included. + +### The group record *(2026.10.1.1+)* + +`mcpp test` over several members plans the members of one configuration +together and builds them once, so the build's time belongs to the group and is +stated once. One record per group precedes the group's first test record: + +```json +{"group_build":{"group":0,"members":["libs/a","libs/b"],"build_ms":8210}} +``` + +| field | | +|---|---| +| `group` | the group's number, from 0, in the order the groups were built | +| `members` | the members planned in it, as `[workspace] members` spells them | +| `build_ms` | the wall time of the group's build: its packages and every test binary | + +A member whose package does not build in a group reports +`{"error":"package","member":"…","compile_output":"…"}`, and the members of the +group whose packages build still run. A test of one member, named with `-p` or +by the command's directory, has neither the record nor `build_group`; a +whole-workspace selection (`--workspace`, or a virtual root without `-p`) +reports in the per-group form even when it holds one member. **`built` and `not_run` are different answers and are counted apart.** Both describe a test that was compiled and not executed, and that is where the diff --git a/docs/specs/exit-codes.md b/docs/specs/exit-codes.md index 6d5975ba0..4c7168c93 100644 --- a/docs/specs/exit-codes.md +++ b/docs/specs/exit-codes.md @@ -4,9 +4,9 @@ |---|---| | 规范编号 | SPEC-003 | | 标题 | mcpp 的进程退出码:分类、语义与稳定性承诺 | -| 状态 | 评审中 v1.0 | -| 版本 | 1.0 | -| 最后修改 | 2026-09-01 | +| 状态 | 评审中 v1.1 | +| 版本 | 1.1 | +| 最后修改 | 2026-10-01 | | 对应实现 | mcpp >= 2026.9.1.1 | | 相关设计文档 | `.agents/docs/2026-08-08-machine-readable-output-protocol-design.md` §R4、`.agents/docs/2026-08-31-issue540-seven-audit-findings.md` §4 | | 相关 issue | #379、#540 | @@ -49,6 +49,7 @@ usage / internal 一半(`2`、`70`、`127`),runtime 的一半 —— 也就是 | `2` | 用法错误 | 未知选项、不支持的选项值、缺少必需参数 | stderr | | `4` | 环境未就绪 | 全局配置加载 / 首次初始化失败(`$MCPP_HOME` 不可写、config.toml 损坏、引导 xlings 失败) | stderr | | `70` | 内部错误 | 未捕获异常。`EX_SOFTWARE` | stderr | +| `101` | 构建失败(仅 `mcpp run`) | `mcpp run` 的规划或构建失败,程序没有启动。Cargo 对 `cargo run` 构建失败给出同一个值 | stderr | | `127` | 未知命令 | 第一个位置参数不是一个子命令 | stderr | `4` 与 `1` 的分界是**谁需要被修**:`4` 说明 mcpp 自己的家还没有准备好,任何命令都会 @@ -58,6 +59,12 @@ usage / internal 一半(`2`、`70`、`127`),runtime 的一半 —— 也就是 `70` 与 `1` 的分界是**这是不是一个缺陷**:`70` 一律意味着 mcpp 有 bug,值得开 issue;`1` 通常不是。 +`101` 只属于 `mcpp run`。`run` 透传程序自身的状态(`docs/50-machine-output.md` +§6),所以它需要一个程序不常返回的值来表示「程序没有被构建出来」,使脚本能把一次 +编译错误和一个返回 `1` 的程序区分开。`mcpp build`、`mcpp test`、`mcpp pack` 的状态 +不变。自己返回 `101` 的程序,或返回 `101` 的 runner,读起来与构建失败相同;区分 +它们靠 stderr,构建失败一定在那里写出原因。 + ## 3. 规则 ### 3.1 分类必须稳定 已实现 @@ -90,7 +97,7 @@ issue;`1` 通常不是。 ## 4. 当前实现与本规范的差异 -无。§2 的六个码是 `src/` 中出现的全部进程退出码。 +无。§2 的码是 `src/` 中出现的全部进程退出码(2026-09-01 核对时是六个;1.1 起增加 `101`,见 `execute.cppm` 的 `kRunBuildFailed`)。 2026-09-01 穷举核对,记录方法与读数,因为「我数了一遍」不是判据: @@ -117,4 +124,5 @@ issue;`1` 通常不是。 | 版本 | 日期 | 变更 | |---|---|---| +| 1.1 | 2026-10-01 | 新增 `101`:`mcpp run` 的规划或构建失败(此前为 `1` 或 `2`,与返回同一个值的程序无法区分)。 | | 1.0 | 2026-09-01 | 首版。落地 2026-08-08 协议设计文档 §R4 指派而未写的契约;补上 `1` 与 `4`,并说明为什么 `4` 不属于 `docs/11` 那张按信封命令划定的表(#540)。 | diff --git a/docs/zh/04-mcpp-toml.md b/docs/zh/04-mcpp-toml.md index 42ab32097..17b7c2027 100644 --- a/docs/zh/04-mcpp-toml.md +++ b/docs/zh/04-mcpp-toml.md @@ -1322,6 +1322,11 @@ cache = "global" # "global" (default) | "local" | "off" workspace 成员。它们的源码可以在 `name@version` 不变的情况下改变， 所以没有任何基于那个身份的键能察觉到变化。 +构建程序 import 的模块遵循同一条规则（2026.10.1.1+）。内置的 `mcpp` 模块与索引包的 host +模块保存在全局缓存中；`path` 或 `git` 依赖、或 workspace 成员的 host 模块保存在 +`target/.build-mcpp/host-modules/` 下，绝不进入全局缓存。`--cache local` 与 `--cache off` +把它们全部保存在工程内。见 [30 — 已编译 host 模块的存放位置](30-build-mcpp.md)。 + 查看与回收： ``` diff --git a/docs/zh/07-workspace.md b/docs/zh/07-workspace.md index 6470fec5c..172e9c035 100644 --- a/docs/zh/07-workspace.md +++ b/docs/zh/07-workspace.md @@ -308,17 +308,20 @@ warning: dependency `render` declares standard = "c++26", and this graph is ```bash mcpp build # virtual workspace → builds ALL members; rooted → the root package mcpp build -p server # build a specific member and its dependencies +mcpp build -p server -p cli # build several members, in one plan mcpp build --workspace # build every member explicitly +mcpp build --workspace --exclude legacy # every member but legacy mcpp test # virtual workspace → tests ALL members; rooted → the root package mcpp test -p core # test a single member +mcpp test -p core -p http # test several members: one plan, one build, one report per member mcpp test --workspace # test every member (one report per member; continues past failures) ``` 在**虚拟**工作空间根（只有 `[workspace]`、没有 `[package]`）下，裸 `mcpp build` / `mcpp test` 作用于**全体**成员；在**带根包**的工作空间（`[package]` + `[workspace]`）下，两者作用于根包；`--workspace` 作用于根包与全体成员。 -`mcpp test --workspace` 独立构建并运行每个成员的 `tests/**/*.cpp`——发现按成员 -隔离，因此两个成员各有一个 `tests/main.cpp` 也不冲突。 +对多个成员的 `mcpp test` 把它们放在一起规划、只构建一次（§5.4），然后运行每个成员的 +`tests/**/*.cpp`——发现按成员隔离，因此两个成员各有一个 `tests/main.cpp` 也不冲突。 ### 5.2 从成员子目录构建 @@ -333,8 +336,8 @@ mcpp 从当前目录向上搜索；若发现某个 `mcpp.toml` 含 `[workspace]` ### 5.3 `-p, --package` 选项 -`-p` 可用于 `build`、`test`、`run` 等命令，指定目标成员。选项名说的是**包**， -参数值按下述顺序解析： +`-p` 可用于 `build`、`test`、`run`、`mcpp emit build-database` 等命令，指定目标成员。 +选项名说的是**包**，参数值按下述顺序解析： 1. 成员的限定名 `.`（只有声明了 namespace 的成员才有这个拼法）； 2. 否则，成员裸的 `package.name`——如果两个以上成员共享它，拒绝并点名每一个匹配； @@ -350,29 +353,63 @@ mcpp run -p server -- --port 8080 参数值若既是某个成员的包名，又是另一个成员的目录，选中包名所命名的那个成员，并给 出警告点名另一个成员——选项名的是包，包名的精确匹配压过恰好同名的目录。 -`--workspace`（用于 `build` 与 `test`）是扇出形式：作用于**每个**成员。 -`mcpp test --workspace` 逐成员分别汇报，遇失败继续，只要有任一成员失败就非零 -退出——很适合作为"一个测试众多库的工作空间"单条、无需 shell 的 CI 步骤。 +#### 多个成员（mcpp 2026.10.1.1+） + +`-p` 可以在 `build`、`test` 与 `mcpp emit build-database` 上重复。每个值指定一个成员， +按上述顺序解析，命令作用于所有这些成员。选择是一个**集合**，无论 `-p` 怎么排列，都按 +`[workspace] members` 的顺序保存；同一个成员被两种拼法各指定一次，只选中一次。 + +```bash +mcpp build -p server -p cli # the same members as -p cli -p server +mcpp test -p core -p http +``` + +指定不到任何成员的值会在规划任何内容之前被拒绝，拒绝信息列出各成员。`-p` 与 `--workspace` 同时出现也会被拒绝：两者陈述了两种选择，任何一方都不会被默认取代另一方。`mcpp run` 执行 +一个程序，所以只作用于一个成员：第二个 `-p` 被拒绝，并点名所有被指定的成员，绝不会被 +理解为"取最后一个"。 + +#### 排除成员：`--exclude` + +```bash +mcpp build --workspace --exclude legacy # every member but legacy +mcpp test --exclude legacy --exclude bench # at a virtual root: every member but two +``` + +`--exclude ` 可以在 `build`、`test` 与 `mcpp emit build-database` 上重复。它的值按 +与 `-p` 相同的方式解析，从"选中每个成员"的选择中去掉成员：`--workspace`，或不带 `-p` 的 +虚拟根。下列情形在规划任何内容之前被拒绝：与 `-p` 同时出现；两种形式都不适用时（在成员 +目录中，或在带根包的根下且没有 `--workspace`）；名字不匹配任何成员；以及它去掉了所有成员。 + +`--workspace`（用于 `build`、`test` 与 `mcpp emit build-database`）是扇出形式：作用于 +**每个**成员。`mcpp test --workspace` 逐成员分别汇报，遇失败继续，只要有任一成员失败就 +非零退出——很适合作为"一个测试众多库的工作空间"单条、无需 shell 的 CI 步骤。成员无论因 +什么失败都只让自己失败：一个测试、它的包的构建，或者它的规划（无法规划的成员被排除在外， +其余成员重新放在一起规划）。 #### 扇出的汇报 ``` + Workspace building 97 members: libs/core, libs/http, ... + Workspace built members libs/core, libs/http, ... in 120.40s; slowest: obj/libs/jsc/tests/jsc.o 88.0s Workspace testing member 'libs/core' (3/97) test_paths ... ok (0.31s) - test result ok. 7 passed; 0 failed; finished in 9.50s (build 8.90s + run 0.60s) - Workspace member 'libs/core' (3/97) ok — 7 passed in 9.50s + test result ok. 7 passed; 0 failed; finished in 121.10s (build 120.40s + run 0.60s) + Workspace member 'libs/core' (3/97) ok — 7 passed, run 0.60s ... workspace result ok. 97 member(s); 412 passed; 0 failed; finished in 355.20s - slowest: libs/jsc 93.5s, libs/install 32.2s, libs/http 24.1s + slowest: libs/install 32.2s, libs/http 24.1s ``` -`M/N` 进度、逐测试耗时，以及按 **build** 与 **run** 拆开的成员耗时。拆开才是有用 -的部分：一个测试只要几毫秒、但链接要 90 秒的成员，在单个合并数字里与"测试套件本身 -很慢"长得一模一样，而这两种情形只有一种值得去查。 +`M/N` 进度与逐测试耗时。同一配置的成员只构建一次，因此构建只在组的那一行里报告一次： +它的成员、它的墙钟时间，以及占用时间最多的那些边。这就是原先逐成员拆分所给出的信号： +链接要 90 秒、而不是测试慢的成员，会被组的 `slowest:` 边点名。每个成员自己的那一行写 +出它的**运行**耗时，最后的 `slowest:` 一行按这个耗时给成员排名。 -`--message-format json` 以 NDJSON 承载同样的数据。每条 test 记录都带成员限定 -字段（`"member"`），流的末尾是一条 `workspace_summary` 记录，列出失败成员与未 -运行成员——一旦两个成员都有一个叫 `smoke` 的测试，裸测试名就不再能归因。 +`--message-format json` 以 NDJSON 承载同样的数据。每条 test 记录都带成员限定字段 +（`"member"`），`group_build` 记录在每个组的第一条测试记录之前说明该组的构建，每个成员的 +汇总指明它的组（`build_group`），流的末尾是一条 `workspace_summary` 记录，列出失败成员与 +未运行成员——一旦两个成员都有一个叫 `smoke` 的测试，裸测试名就不再能归因。各字段见 +[50 —— 机器可读输出](50-machine-output.md#mcpp-test---message-format-json--测试流)。 #### 给扇出设期限 @@ -382,11 +419,17 @@ mcpp test --workspace --build-timeout 300 # per-ninja-drive deadline (default 0 mcpp test --workspace --workspace-timeout 1800 # whole fan-out (default 0 = no limit) ``` -扇出是串行的，所以一个没有上界的成员会拖住排在它后面的每一个成员。三个期限都是 -**汇报而非中止**：测试超时只判该测试失败，扇出继续；构建超时只判该成员失败； +各成员的运行是串行的，所以一个没有上界的成员会拖住排在它后面的每一个成员。三个期限都是 +**汇报而非中止**：测试超时只判该测试失败，扇出继续；构建超时判等待这次构建的成员失败； `--workspace-timeout` 停止扇出并列出未运行的成员，而不是把进程留给 CI 去 kill—— 那样会把进程本该说出的话一并丢掉。 +`--workspace-timeout` 限制的是运行，从命令开始时计时：它在每个成员的测试开始前检查，到那 +时还没有开始的成员被列为未运行（2026.10.1.1+）。一个组的成员在同一步中构建，期限无法打断 +这一步，因此仅构建就超过期限的工作空间会把这次构建做完（由 `--build-timeout` 限制），然后 +不再启动任何成员。2026.10.1.1 之前每个成员依次构建、依次运行，期限可以在两次构建之间停止 +扇出。 + ### 5.4 每个配置一张构建图（mcpp 2026.9.29.1+） 对工作空间的命令把成员放在一起规划：被选中的成员及其全部依赖构成一张构建图，只有一个 @@ -397,7 +440,8 @@ mcpp test --workspace --workspace-timeout 1800 # whole fan-out (default 0 = no 在各自的图中构建，这些图同时进行，共享命令的并行任务数。成员写下的相对路径（例如它自己的 `[indices]` 路径）按该成员的目录解析。 - **选择。** `--workspace`，以及虚拟工作空间根下不带 `-p` 的命令，选中全体成员；`-p X` 与在 - X 的目录中执行的命令规划 X 及其所依赖的一切。两者共用构建目录，因此先执行 + X 的目录中执行的命令规划 X 及其所依赖的一切；`-p X -p Y` 把两者放在一起规划，作为一个 + 选择（§5.3）。这些选择共用构建目录，因此先执行 `mcpp build --workspace` 再执行 `mcpp build -p X` 不编译任何内容；只有当某个包在两条命令中 启用的 feature 不同时，它才会被重新编译。 - **编译参数。** 成员的 `cflags`、`cxxflags`、`ldflags` 与 defines 作用于该成员自己的命令。 @@ -413,7 +457,15 @@ mcpp test --workspace --workspace-timeout 1800 # whole fan-out (default 0 = no - **资源。** 成员的 `[resources]` 与 `windows_code_page` 按该成员的目录与 include 目录编译， 只嵌入该成员自己的程序与共享库（2026.9.29.2+）。 - **构建程序。** 成员的构建程序按依赖在前的顺序运行；只要程序的输入不变，无论命令选中哪些 - 成员，程序的结果都被复用（2026.9.29.5+）。 + 成员，程序的结果都被复用（2026.9.29.5+）。它们的编译同时进行，数量以作业数为上限，只有运行 + 遵循这个顺序，因此计划与串行构建写出的相同；多个程序 import 的同一个 host 模块只为它们 + 编译一次（2026.10.1.1+；见 [30 — 构建程序](30-build-mcpp.md)）。 +- **测试。** 对多个成员的 `mcpp test` 按 `build` 的方式规划：每个配置一次，包含各成员的 + 测试，因此成员共用的包只编译一次，它的构建程序只运行一次，它的 feature 是选择所请求的 + 并集（2026.10.1.1+）。该配置的包与测试二进制只构建一次；然后每个成员的测试按成员顺序 + 运行，使用该成员自己的运行时目录，而不是其他成员的。在 `mcpp build --workspace` 之后运行 + 的测试不会编译构建已经编译过的任何内容，除非某个 dev-dependency 改变了包的 feature。 + 对一个成员的 `mcpp test` 是该成员的一次规划，一如既往。 - **编译数据库。** `mcpp build --configure-only` 与 `mcpp emit build-database` 按构建的方式 规划，每个配置一次规划并包含各成员的测试，因此成员共用的包在每个配置中只描述一次。规划了 多个配置的命令只写一次根目录的 `compile_commands.json`，内容为各配置数据库的并集 @@ -468,7 +520,8 @@ myproject/ - 工作空间根的 `compile_commands.json` 覆盖所有已构建或已配置的成员。 - 工作空间根的 `mcpp.lock` 记录全体成员的解析结果。`mcpp build --workspace` 写入完整记录； `mcpp build -p X` 更新 X 所在图中的条目。 -- 成员的构建程序写入 `/target/.build-mcpp/`。 +- 成员的构建程序写入 `/target/.build-mcpp/`。各程序 import 的 host 模块编译进 + workspace 自己的 `target/.build-mcpp/host-modules/`，为全体程序只编一次。 - 早期版本的 mcpp 在成员自己的 `target/` 下留下的构建目录不再被读取；`mcpp clean --stale` 会删除它们。 diff --git a/docs/zh/08-testing.md b/docs/zh/08-testing.md index 9ce4b0c1a..cc7542b23 100644 --- a/docs/zh/08-testing.md +++ b/docs/zh/08-testing.md @@ -76,6 +76,27 @@ mcpp test -- --verbose # everything after `--` goes to each test binary `--timeout ` 杀掉仍在运行的测试（默认 300；`0` 关闭），`--build-timeout ` 限制编译耗时。挂起的测试以它自己的名字被报为失败，而不是报成一个停止的任务。 +报告是命令的结果，写到标准输出：每个测试的结论、失败测试的输出、`test result` 行， +以及 `--workspace` 时的 `workspace result` 行。构建的各步骤（`Compiling`、`Running`） +属于叙述，写到标准错误，因此 `mcpp test > report.txt` 收集报告， +`mcpp test 2>&1 | tee test.log` 两者都收集 +（见 [09 —— 按场景选命令](09-commands-by-scenario.md#输出流)）。 + +### 在工作空间中 + +```bash +mcpp test -p core -p http # the tests of two members +mcpp test --workspace --exclude legacy # every member's but legacy's +``` + +`-p` 可以重复，`--exclude` 把成员从 `--workspace` 或虚拟根的全体选择中去掉；两者都按 +[07 —— 工作空间](07-workspace.md#53--p---package-选项)所述解析。对多个成员的测试把它们放在 +一起规划、只构建一次：同一配置的成员是一张构建图，因此它们共用的包只编译一次，它的构建 +程序只运行一次。然后每个成员的测试按 `[workspace] members` 的顺序运行，各自使用该成员 +自己的运行时目录。命令在某个成员失败后继续，无论失败的是测试、它的包的构建还是它的规划， +逐成员汇报，只要有一个失败就以非零退出。发现按成员隔离，因此两个成员可以各有一个 +`tests/main.cpp`。 + ## 取到产物取不到的包的测试 ```toml @@ -153,6 +174,7 @@ mcpp test --message-format json - manifest 加载不了时，`mcpp test --list` 列出的是 `tests/**/*.cpp`，而不是它读不到的 `[test] discover` 集合。 - `--build-timeout` 只在 POSIX 上有效。 -- `--workspace-timeout` 限制 `--workspace` 扇出的耗时，并报告哪些成员跑完了；它不把 - 超时归因到某一个成员。 +- `--workspace-timeout` 限制 `--workspace` 扇出的运行，并报告哪些成员跑完了；它不把 + 超时归因到某一个成员。它在每个成员的测试开始前检查，因此不能打断成员共用的那次构建， + 那次构建由 `--build-timeout` 限制。 diff --git a/docs/zh/09-commands-by-scenario.md b/docs/zh/09-commands-by-scenario.md index 0dc141c0c..a47dc3e24 100644 --- a/docs/zh/09-commands-by-scenario.md +++ b/docs/zh/09-commands-by-scenario.md @@ -28,7 +28,7 @@ | 存储 | 作用域 | 增长时机 | 清空方式 | |---|---|---|---| | `target/<三元组>/<指纹>/` | 单个工程 | 一个配置指纹变化，开出一个新目录 | `mcpp clean`、`mcpp clean --stale` | -| 构建缓存（`mcpp cache dir`） | 整台机器 | 任何工程编译一个依赖、一个 `std` 模块，或构建一个 host 工具 | `mcpp cache gc`、`mcpp cache prune`、`mcpp cache clean` | +| 构建缓存（`mcpp cache dir`） | 整台机器 | 任何工程编译一个依赖、一个 `std` 模块，或构建一个 host 工具，或编译其构建程序 import 的 `mcpp` 模块 | `mcpp cache gc`、`mcpp cache prune`、`mcpp cache clean` | `mcpp clean` 整个删掉 `target/`，下次构建重编一切。`mcpp clean --stale` 只删已无构建记录使用的指纹目录，在用的配置保留： @@ -202,9 +202,10 @@ commit 记进 `mcpp.toml`;`mcpp index unpin` 移除它。 - `git` 依赖的克隆； - 沙箱首次运行时的工具。 -在终端上，每一项是一个原地重绘的进度条。标准输出不是终端时（例如 CI 日志或 -管道），每一项在开始时打印一行（已知时带上大小），结束时打印一行（带上耗时）。 -这样的输出不含回车符，也不含擦除序列。`--quiet` 两者都不打印。 +在终端上，每一项是一个原地重绘的进度条。标准错误不是终端时（例如 CI 日志或 +重定向），每一项在开始时打印一行（已知时带上大小），结束时打印一行（带上耗时）。 +这样的输出不含回车符，也不含擦除序列。`--quiet` 两者都不打印。进度条属于叙述， +叙述写到标准错误（见下文「输出流」）。 当 mcpp 驱动的 xlings 为索引刷新发出进度事件时（xlings 2026.9.28.1+），索引刷新 逐步报告。较旧的 xlings 下，它显示其状态行，然后安静地结束，与以前相同。 @@ -257,7 +258,7 @@ $ mcpp build 动画在 mcpp 工作时缓慢移动，有步骤完成时加快，构建等待时静止。 - **计数与时间**：随后是已完成与计划的步骤数，以及自命令开始的时间。当没有待启动的步骤时，追加 `last N running`。`Building` 的步骤是构建的工作：编译、链接、归档与 action（2026.9.30.2+）。放置全局缓存提供的内容由 `Cached` 行报告，不计入；依赖扫描先行，以 `Scanning` 显示，并有自己的计数；需要等待某个包的 `prepare` 或 `check` action 的扫描随构建一起运行，并计入构建。 - **正在运行的动作**：最后是运行最久的 `check` 或 `prepare` 动作。其余步骤只在完成时由 ninja 报告。 -- **绘制方式**：状态行在命令开始半秒后才首次绘制；每次更新都以一次写入原地覆盖，不会闪烁。 +- **绘制方式**：状态行画在标准错误上，在命令开始半秒后才首次绘制；每次更新都以一次写入原地覆盖，不会闪烁。 `MCPP_PROGRESS` 选择点阵屏的内容： @@ -266,7 +267,7 @@ $ mcpp build - `plain`：保留状态行，不显示点阵屏； - `off`：不绘制实时行，与日志输出相同。 -点阵屏需要能绘制盲文点字的终端，即 UTF-8 locale 或 Windows Terminal；否则状态行不含点阵屏。输出不是终端（CI 日志、管道）时，只写出最终的行，并在输出静默一分钟时写出状态行。在终端上设置 `TERM=dumb` 同样选择这种形式。 +点阵屏需要能绘制盲文点字的终端，即 UTF-8 locale 或 Windows Terminal；否则状态行不含点阵屏。标准错误不是终端（CI 日志、重定向）时，只写出最终的行，并在输出静默一分钟时写出状态行。在终端上设置 `TERM=dumb` 同样选择这种形式。 `--play-game` 在构建期间于点阵屏上玩一个游戏。`build`、`run`、`test` 都接受该选项；`--play-game=NAME` 指定游戏，否则随机选择： @@ -285,10 +286,46 @@ $ mcpp build --play-game=snake - **速度与计数**：游戏以自身的速度运行，旁边的计数陈述构建的进展。 - **按键读取**：按键不回显；Ctrl-C 仍然中断构建。 - **终端模式**：构建结束或被中断时，终端模式会恢复；被强制杀死的进程无法恢复，此时可执行 `stty sane`。 -- **使用条件**：游戏要求标准输入和标准输出都是终端，且 mcpp 位于该终端的前台（不是后台作业）；否则写出一行说明原因，构建照常进行。 +- **使用条件**：游戏要求标准输入和标准错误（点阵屏画在其上）都是终端，且 mcpp 位于该终端的前台（不是后台作业）；否则写出一行说明原因，构建照常进行。 `--verbose` 列出每个包：无事可做的包记为 `Fresh`，做了事的包记为 `Compiled`，并给出其步骤数和耗时跨度。它还给出每个构建程序的编译与运行时间，并按 ninja 的报告打印每个步骤（`[f/t] <命令>` 及其输出）。`--quiet` 不输出以上内容。机器输出（`--message-format json`）不变。 +## 输出流 + +每条命令都把**叙述**写到标准错误，把**结果**写到标准输出（mcpp 2026.9.30.2 及更早的版本把叙述写到标准输出）。叙述说明命令正在做什么：以动词开头的状态行、进度条、状态行、`Finished`，以及 `Running` 之后的空行。结果是命令被要求产生的东西： + +- `mcpp run` 之下，是程序打印的内容，不含其他任何东西； +- 文档或列表：`mcpp run --list-runners`、`mcpp toolchain list`、`mcpp cache list`、`mcpp search`、`mcpp test --list`、`mcpp emit`； +- 对 `mcpp test` 而言，是报告：每个测试的结论、失败测试的输出、`test result` 行，以及 `--workspace` 时的 `workspace result` 行。 + +```console +$ mcpp run -q 2>/dev/null > out.txt # out.txt 逐字节等于程序的输出 +$ mcpp build >/dev/null # 各步骤与 Finished 仍然显示 +$ mcpp build 2>&1 | tee build.log # 记录各步骤：两个流合到一条管道 +``` + +- **迁移**：`mcpp build | tee build.log` 不再记录各步骤，它们在标准错误上；应写作 `mcpp build 2>&1 | tee build.log`。同时捕获两个流的 CI 日志不受影响。 +- **`--quiet`**：抑制叙述。警告、错误和编译器诊断仍照常写到标准错误，`--quiet` 不影响它们。 +- **顺序**：在同一条管道上和在终端上，两个流都保持写入的先后顺序。 +- **状态行**：在终端上，状态行画在标准错误上。`mcpp build | less`（标准输出是管道）会画出状态行；`mcpp build 2>log`（标准错误是文件）不会。 +- **机器输出**：`--message-format json` 与 `--format json` 不变，标准输出只携带文档（见 [50 —— 机器可读输出](50-machine-output.md)）。 + +### `mcpp run` 的退出状态 + +`mcpp run` 以程序自身的状态退出，原样透传。规划或构建失败的 `run` 没有启动程序，退出状态为 **101**，这与 `cargo run` 构建失败时的状态相同： + +```console +$ mcpp run # 编译错误 +error: build failed in app v0.1.0 (.) +$ echo $? +101 +$ mcpp run -- 1 # 返回 1 的程序 +$ echo $? +1 +``` + +无法启动的程序保留拒绝时的状态（`127` 找不到，`126` 不可执行，`125` 其他），`mcpp build`、`mcpp test`、`mcpp pack` 的状态不变。程序本身，或者 runner（`--runner`、`[target.].runner`），自己返回 101 时，读起来与构建失败相同；各区间见 [50 —— 机器可读输出](50-machine-output.md) §6。 + ## 发布前校验描述符 `mcpp xpkg parse` 用解析器自己的文法读一个描述符，所以它报告的就是解析时 diff --git a/docs/zh/10-pack-and-release.md b/docs/zh/10-pack-and-release.md index b5714a3fa..0c9294b43 100644 --- a/docs/zh/10-pack-and-release.md +++ b/docs/zh/10-pack-and-release.md @@ -136,6 +136,8 @@ mcpp pack --dev # the same, as `build` and `run` spell it mcpp pack --message-format json # one mcpp.pack envelope on stdout (mcpp 2026.9.16.1+) mcpp pack --no-strip # ship the artifacts as built mcpp pack -p app --format release # a workspace member, as if run in its directory +mcpp pack --workspace --format release # every member with a program, planned and built once +mcpp pack -p cli -p gui -o dist/ # two members, each archive below dist/ mcpp pack --debug-symbols dbg/ # write the separated *.debug files under dbg/ mcpp pack --format msi --features installer # activate root-package features for the pack ``` @@ -154,11 +156,64 @@ feature：每一条 `--target` 腿，以及被分派格式的两遍构建。它 `--release` 与 `--dev`（mcpp 2026.9.16.1+）是 `build`、`run` 已接受的简写， 优先级相同：三条命令上都是 `--profile` 优先于它们。 +### 打包多个成员（mcpp 2026.10.1.1+） + +`-p` 可以重复，`--workspace` 与 `--exclude` 选择成员的方式与 `mcpp build` 相同 +（[07 —— 工作区](07-workspace.md)，§5.3）。这些成员作为一个选择一起规划、只构建一次， +然后每个成员在各自的暂存树中暂存并打包： + +```bash +mcpp pack --workspace --format release # 每个有程序的成员 +mcpp pack --workspace --exclude updater # 除一个之外的每个这样的成员 +mcpp pack -p cli -p gui --format release +``` + +对每个成员分别运行 `mcpp pack -p `，每次都会重新规划图、重新运行构建程序、 +重新启动构建，它们共用的成员也不例外。对多个成员的打包，每个配置（即 `mcpp build` +使用的分组）规划一次、构建一次，每个构建程序每遍运行一次：被多个成员依赖的包， +例如带构建程序的共享库，只准备并编译一次，它的程序在这次打包里不会再次运行。 + +每个成员得到的，就是 `mcpp pack -p ` 给它的： + +- **产物。** 成员的归档或目录，就是它单独打包时得到的那一份，位于它自己的 + `target/dist/` 之下。给出 `--output ` 时，它写在该目录之下（目录不存在时会创建），使用默认的名字。 +- **自己的暂存树。** 使用被分派的 `--format` 时，打包如上所述准备两次，第二遍告诉每个 + 构建程序它所代表的成员的暂存树。`mcpp::pack_stage_dir()` 与 `${mcpp.stage_dir}` + 按成员回答。一个程序代表它所属的成员；只有一个被打包成员依赖到的包（成员所依赖的 + 提供者）代表那个成员；被多个被打包成员依赖到的包不代表其中任何一个，它既不会被告知 + 格式，也不会被告知任何树，因此它的程序运行一次就够所有成员使用。 +- **自己的可分发物。** 本次请求引入的 `artifact` action，属于提交它的那个程序或提供者 + 所代表的成员。每个成员的输出都会核实存在，并以该成员的 `Packed` 报告。 + +`--workspace` 只打包有程序 target 的成员，并说明跳过了哪些；用 `-p` 点名的成员若没有 +程序 target 则被拒绝。成员按配置逐组打包，按 `[workspace] members` 的顺序报告，与 `-p` 的书写顺序无关。 +某个成员失败（例如它的提供者对所请求的格式什么也没有提交）时，按名字报告，其余成员 +照常打包；只要有一个成员失败，退出状态就非零。一个成员 —— 无论是用一个 `-p` 点名，还是 +由命令所在的目录决定 —— 与以前完全一样地打包：`mcpp pack -p X` 只规划 X 的闭包。在虚拟 +工作区根目录不带任何选择器的 `mcpp pack` 仍旧打包第一个有程序的成员；`--workspace` 则 +打包全部。 + +**在任何东西被编译之前拒绝**，并点名被拒绝的对象： + +| 输入 | 原因 | +|---|---| +| 位置参数 target 名 | 它点名的是某一个包的 target | +| 多于一个 `--target` | 一个程序只为一个 target 构建，而多 target 的 Android 打包暂存的是一个成员的各条腿 | +| 已存在且是文件的 `--output` | 每个成员的归档或目录写在该目录之下 | +| 用 `-p` 点名、却没有程序 target 的成员 | 它没有可打包的东西 | +| 使用被分派的 `--format` 时，没有任何提供者代表它的成员 | 该格式由成员依赖不到的包提供，或由被多个成员依赖到的包提供 | +| 两个会写同一个归档或目录的成员 | `--output` 只点名一个目录，而名字、版本与 target 相同的两个包共用同一个归档名 | + +一个为多个成员提供格式、带构建程序的包，无法同时服务它们：提供者只会得到一棵暂存树。 +请从每个成员自己的构建程序提供该格式（两者共用一个辅助函数即可），或者逐个打包成员。 + `--message-format json`（mcpp 2026.9.16.1+）在命令结束后于 stdout 上输出一个 `mcpp.pack` 信封，所有给人看的行都改走 stderr。它的 `data.artifacts` 列出 产出的每一个文件或目录，带绝对路径、`type`（`file` 或 `directory`）、 `--format` 取值以及各条腿的三元组；`data.stage` 给出暂存树、它的 manifest， -以及闭包是否已经走通（见 [50 —— 机器输出](50-machine-output.md)）。这条命令 +以及闭包是否已经走通（见 [50 —— 机器输出](50-machine-output.md)）。对多个成员的 +打包列出每个成员的产物，每一条点名它的 `member`，`data.stage` 为 null，并在 +`data.stages` 中为每个成员给出一棵树（2026.10.1.1+）。这条命令 上的 `--format` 表示的是包格式，因此机器输出改用 `mcpp test` 请求它的那种 方式。 diff --git a/docs/zh/30-build-mcpp.md b/docs/zh/30-build-mcpp.md index e165ae9f7..a2347f4c9 100644 --- a/docs/zh/30-build-mcpp.md +++ b/docs/zh/30-build-mcpp.md @@ -126,7 +126,8 @@ manifest/描述符里（`[build] include_dirs`），而不是构建期程序里 除打印裸字符串外，`build.mcpp` 也可以写成**模块优先**形式——`import mcpp;`，不需要 `#include`。`mcpp` 模块**内置在 mcpp 二进制里**（因此始终与当前这版 mcpp -的协议匹配），按需编译；它的函数只是 emit 上面那些指令： +的协议匹配），按需编译——每个 mcpp 版本与 host 编译器组合只编一次，存入构建缓存 +（见下文*已编译 host 模块的存放位置*）；它的函数只是 emit 上面那些指令： ```cpp // build.mcpp @@ -1282,6 +1283,13 @@ mcpp **不会**每次构建都重跑 `build.mcpp`。它会缓存程序产出的 程序运行期间，状态行显示 `Running`、已完成与已安排的程序数，并给出正在运行的程序、它的状态 （`compiling` 或 `running`）及计时。 +**工作空间的构建程序同时编译，依次运行**（2026.10.1.1+）。成员的程序在它所依赖的成员的程序 +之后运行，因为它要读那些程序的指令；这个顺序保持不变：指令按该顺序应用，因此计划，也就是 +`build.ninja`，与逐个编译程序的构建所写出的完全相同。编译之间没有这样的顺序，所以每个结果已 +过期的程序同时编译，数量以作业数为上限（`--jobs`、`MCPP_JOBS`、`[build] jobs`）。程序报告的 +`compiled` 耗时包含准备它所 import 的模块的时间，不再留给 `plan` 计入。一次编译的输出整体 +打印；若多个程序编译失败，报告的是按运行顺序排在最前的那个，而不论哪次编译先结束。 + ## 依赖产出的 host 工具（mcpp 2026.8.5.1+） 一个包能构建出消费者在**构建期**需要的二进制 —— `protoc`、`grpc_cpp_plugin`、 @@ -1370,9 +1378,10 @@ import protobufgen; int main() { return protobufgen::generate({"schema"}) ? 0 : 1; } ``` -mcpp 会把该包的 lib 根模块**为 host 编译，且与 `build.mcpp` 在同一条命令里** —— +mcpp 会把该包的 lib 根模块**为 host 编译，且与 `build.mcpp` 使用相同的标志** —— 这正是 BMI 能用的前提：一个模块接口只对「在 standard / dialect / 编译器身份上与 -它一致」的编译可导入。 +它一致」的编译可导入。编译结果按键保存，因此对每组在这些标志上一致的程序只编一次 +（见*已编译 host 模块的存放位置*）。 于是规则**有版本、能测试、能通过既有的包管理器分发**，而且是用 **C++** 写的 —— 不引入第二门语言，这正是 `build.mcpp` 存在的理由。 @@ -1398,6 +1407,34 @@ mcpp 拒绝这种情形，并点名两个包与各自的 interface 路径。检 lib 根必须在 `src/.cppm`（或 `[lib] path` 指向的位置）；缺失时报 *"host module 'x': no interface unit at …"*。 +#### 已编译 host 模块的存放位置（2026.10.1.1+） + +内置的 `mcpp` 模块与程序 import 的 host 模块不再编译进每个程序自己的目录。每一个都是存储中 +的一个条目，条目的地址由进入它那次编译的一切决定：host 编译器的身份、standard 标志、该次 +编译携带的标志、它 import 的 BMI、interface 文件的 SHA-256、mcpp 版本（对内置模块）与提供者 +包（对 host 模块）。这些输入记录在条目的 `entry.json` 中，命中时逐字段比较，而不只比较哈希。 +两个程序的编译在所有这些上一致时共用一个条目；standard 或标志不同的程序有自己的条目，因此 +BMI 不会被与它不一致的编译取用。 + +条目放在哪里取决于它的文本从哪里来： + +| 模块 | 存放位置 | 依据 | +|---|---|---| +| 内置的 `mcpp` 模块及其 `mcpp.core` 别名 | 全局缓存：`$MCPP_HOME/build-cache/v1/pkg/_engine/mcpp-build-module@//` | 引擎拥有这段文本，对同一 mcpp 版本与同一 host 编译器，它在每个工程中都相同 | +| 源码位于不可变 store 的索引包的 host 模块 | 全局缓存：`…/pkg//@//` | 依赖缓存的规则：名称与版本确定了字节 | +| `path` 或 `git` 依赖、或 workspace 成员的 host 模块 | `/target/.build-mcpp/host-modules//` | 它的源码可以在名称与版本不变的情况下改变 | + +缓存模式的作用与对依赖相同（见 [04 §2.10](04-mcpp-toml.md)）。`--cache global`（默认）下， +条目按上表放入全局缓存。`--cache local` 或 `--cache off` 下，所有条目都保存在 workspace 的 +存储里，同一次调用中的各程序仍共用它们。`path` 依赖的 host 模块绝不写入全局缓存。workspace +存储中条目的键还包含包源码树的摘要，因此修改 interface 所 include 的头文件会编出新条目， +旧条目原样保留。 + +全局缓存中的条目与依赖的条目一样：`mcpp cache list` 把内置模块显示为 +`_engine/mcpp-build-module@`，`mcpp cache verify` 检查它，`mcpp cache gc` 按最近 +使用回收它；升级 mcpp 会留下上一版本的条目，交给 `gc`。workspace 的存储随 `target/` 由 +`mcpp clean` 一并删除。程序看到的与以前相同：同样的模块，用同样的标志编译。 + **一个包可以提供多条规则，由 feature 选择**(mcpp 2026.9.5.3+)。包解析后的 `[build] sources` 里 —— 含 feature 加入的源文件 —— 每一个模块接口单元都以它自己声明的 名字编成一个 host 模块，lib 根排在最前。feature 单元可以 import lib 根；除此之外每个 diff --git a/docs/zh/50-machine-output.md b/docs/zh/50-machine-output.md index 76002d1b4..80279f329 100644 --- a/docs/zh/50-machine-output.md +++ b/docs/zh/50-machine-output.md @@ -125,6 +125,22 @@ $ echo $? 答案是一次拒绝。§1 依然成立 —— 解析 stdout，不要按退出码分支 —— 但一个把任何 非零退出都当成「没有输出」的客户端，会丢掉它已经拿到手的文档。 +### 标准输出携带结果，标准错误携带叙述 + +每条命令都写两个流，划分方式对所有命令相同。2026.9.30.2 及更早的版本把叙述写到标准输出，所以原先在那里读取叙述的使用方，现在应当读标准错误。 + +| 流 | 携带的内容 | +|---|---| +| 标准输出 | 命令的**结果**：`mcpp run` 下程序打印的内容；文档（信封、`mcpp emit`、`mcpp run --list-runners`、`--version`、`--help`）；列表（`mcpp toolchain list`、`mcpp cache list`、`mcpp search`、`mcpp test --list`）；以及 `mcpp test` 的报告：每个测试的结论、失败测试的输出、`test result` 行，以及工作区运行的 `workspace result` 行 | +| 标准错误 | **叙述**：状态行（`Resolving`、`Compiling`、`Finished`、`Running`、`Packing`、`Packed`、`Downloading`、`Updating` 及其他以动词开头的行）、终端上的进度条与状态行、`Running` 之后的空行、`warning:`、`error:`、`note:`、`tip:` 行、编译器诊断，以及 `--verbose` 增加的内容 | + +判断一行属于哪个流，依据它的用途。说明命令正在做什么或已经做了什么的行是叙述；命令被要求产生的东西是结果。Cargo 把 libtest 的报告写到标准输出，`mcpp test` 同理。 + +- **管道只收到结果。** `mcpp run -q 2>/dev/null` 写出的恰好是程序的标准输出；`mcpp build >/dev/null` 仍然显示各步骤。在终端上两个流都到达屏幕，状态行画在标准错误上，所以 `mcpp build 2>/dev/null` 不显示状态行。 +- **`--quiet` 抑制叙述**，包括 `Running` 之后的空行，保留警告、错误和诊断。 +- **机器可读模式**（`--format json`、`--message-format json`）仍把标准输出留给自己的文档。它们在标准错误上叙述什么没有变化：`mcpp test --message-format json` 不叙述；`mcpp pack --message-format json` 与 `mcpp emit build-database` 在规划与构建期间在标准错误上叙述。 +- **两个流在同一条管道上保持写入顺序。** `mcpp build 2>&1 | tee log` 记录各步骤，这是原先写 `mcpp build | tee log` 的脚本应改成的写法（同时捕获两个流的 CI 日志不受影响）。 + ## 4. effects —— 命令在打印结果之前执行的动作 一个带有 untrusted-workspace 门禁的 IDE，必须在**运行之前**就做出决定。等 @@ -188,14 +204,19 @@ mcpp cache list --json -> {"root": …, "entries": [ … ]} ## 6. 退出状态 -`mcpp run` 报告的是程序自身的退出状态。整个取值空间分三段，只有第一段属于 +`mcpp run` 报告的是程序自身的退出状态。整个取值空间分四段，只有第一段属于 程序： | 区间 | 含义 | |---|---| -| `0`–`124` | 程序运行过了；这是它自己的状态，原样透传 | +| `0`–`124` | 程序运行过了；这是它自己的状态，原样透传。例外是 `101`，见下一行 | +| `101` | `mcpp run` 无法构建程序：规划或构建失败，没有启动任何东西。这是 `cargo run` 构建失败时 Cargo 给出的状态 | | `125`–`127` | 尝试过启动但被拒绝 —— `127` 是找不到，`126` 是找到了但不可执行，`125` 是其他原因；对 `mcpp run --format ` 而言，一个是目录、且没有任何 runner 能到达的分发物同样得到 `126`，它在启动之前就被拒绝，含义相同（2026.9.14.2+） | -| `2` | mcpp 在尝试启动任何东西之前就拒绝了：用法、配置或解析错误 | +| `2` | mcpp 在构建任何东西之前拒绝了请求，或在构建之后、启动任何东西之前拒绝了请求：用法错误、`--format` 与 `--no-runner` 同用、项目没有声明的程序或 runner | + +在 2026.9.30.2 之前，构建失败退出 `1`，规划失败退出 `2`，脚本因此无法区分 +一次编译错误和一个返回 `1` 的程序。现在构建失败是 `101`；`mcpp build`、 +`mcpp test`、`mcpp pack` 保持各自的状态。 在 2026.9.4.3 之前，所有非零状态都被折叠成 `1`，目的是让 `2` 能表示「起不来」 以区别于「跑了但失败」。这个区分值得保留，但代价不值得：`main` 返回 `3` 的 @@ -204,12 +225,15 @@ mcpp cache list --json -> {"root": …, "entries": [ … ]} 中间那一段，是 `env`、`timeout`、`nice` 早已在用、并且被 shell 文档化的取 值，因此 `126` 与 `127` 到达时带着它们惯常的含义，而不是本项目自行分配的 -编号。 +编号。`101` 同理，它是 Cargo 的取值。 -**程序自身也可以以 `125`–`127` 退出，mcpp 不试图靠数字去区分两者。** 区分它 -们的是：启动失败一定会向 stderr 写出原因，而程序自身的退出状态从不这样做。 -需要确定结果的客户端应当读 stderr，或者使用 `--format json` —— 那里退出 -状态是一个字段，不是一条通道。 +**程序自身也可以以 `101` 或 `125`–`127` 退出，mcpp 不试图靠数字去区分两者。** +区分它们的是：构建或启动失败一定会向 stderr 写出原因，而程序自身的退出状态从 +不这样做。需要确定结果的客户端应当读 stderr，或者使用 `--format json` —— 那里 +退出状态是一个字段，不是一条通道。 + +使用 runner（`--runner`、`[target.].runner`）时，透传的是 runner 的状态； +自己返回 `101` 的 runner 读起来与构建失败相同。这一点被接受，Cargo 也是如此。 `mcpp test` 保持不变，仍然是 `0` 或 `1`：它聚合了多个程序的结果，没有单一的 状态可以透传。每个测试各自的退出码在 JSON 流的 `exit_code` 字段里（见 @@ -496,10 +520,17 @@ mcpp pack [target] [--format ] [--target ...] --message-format json | 字段 | | |---|---| | `artifacts` | 每个产出的产物一条记录：`path`（绝对路径）、`type`（`file` 或 `directory`）、`format`（`--format` 的取值，省略时为 `tar`），以及 `targets`（进入该产物的每条腿的规范三元组）。被分派的格式报告本次请求引入的那些 action 的终端输出；一个多 `--target` 的 Android 打包报告一个产物，其 `targets` 列出每一条腿 | -| `stage` | 该产物所来自的那棵树：`dir`、`manifest`（即下文的暂存清单）与 `closure`（`walked` 或 `not-walked`）；库包，以及没有暂存任何树的情形，此字段为 `null` | +| `stage` | 该产物所来自的那棵树：`dir`、`manifest`（即下文的暂存清单）与 `closure`（`walked` 或 `not-walked`）；库包、没有暂存任何树的情形，以及对多个成员的打包，此字段为 `null` | +| `stages` *（2026.10.1.1+）* | 只在对多个成员的打包（`--workspace`，或重复的 `-p`）中出现：每个成员一条记录，按 `[workspace] members` 的顺序，带 `member`（限定的包名），以及与 `stage` 相同的 `dir`、`manifest`、`closure` | + +对多个成员的打包，`artifacts` 按同样的成员顺序列出每个成员的产物，每条记录多出 +`member`，即它所属成员的限定包名。对一个成员的打包既没有 `stages` 也没有 `member`： +信封与以往完全一样。 失败时省略 `data`，以命令自身的退出码退出，并携带诊断码 -`MCPP_PACK_FAILED`；原因写在 stderr 上。每次运行的 `effects` 是 +`MCPP_PACK_FAILED`；原因写在 stderr 上。对多个成员的打包若有成员失败，则对每个这样的成员 +再多带一条点名它的 `MCPP_PACK_FAILED` 诊断，同样省略 `data`：已经打包的成员在磁盘上， +并在 stderr 上点名。每次运行的 `effects` 是 `read-project`、`write-project` 与 `write-global-cache`，若运行了某个构建 程序则再加上 `exec-build-script`。`--protocol-version` 为 `pack` 声明 `init-mcpp-home`、`read-project`、`write-project`、`network`、 @@ -513,8 +544,9 @@ mcpp test [pattern] [--workspace] --message-format json 这条流早于 §2 的信封，也不被它包裹：它是 NDJSON，每个测试结束时一条记录，随后 每个成员一条汇总记录。`--workspace` 运行以一条 `workspace_summary` 记录 -结束。§7 的保证对它同样成立 —— 字段只增不减，字段含义永不改变 —— 下表是 -2026.9.2.1 时点的契约。 +结束。对多个成员的测试会在每个组的第一条测试记录之前增加一条 `group_build` 记录。 +§7 的保证对它同样成立 —— 字段只增不减，字段含义永不改变 —— 下表是 +2026.9.2.1 时点的契约，各行注明其后的新增。 每个测试： @@ -525,7 +557,8 @@ mcpp test [pattern] [--workspace] --message-format json | `status` | `pass`、`compile_fail`、`run_fail`、`not_run` 或 `built` | | `exit_code` | 该测试的退出状态；`not_run` 与 `built` 时为 `0` | | `signal` | 状态编码了信号时为信号编号，否则为 `null` | -| `duration_ms` | 该测试构建加运行的墙钟时间 | +| `duration_ms` | 决定该测试状态的那一步的墙钟时间：运行过的测试是运行，`compile_fail` 是构建 | +| `build_ms` | 本次调用中该测试自身二进制的构建耗时，即它的链接边与主单元编译边在 `.ninja_log` 里的耗时之和；两者都未重新构建时为 `0` *（2026.10.1.1+）* | | `timed_out` | 被 `--timeout` 杀掉时为 `true`（`run_fail`） | | `compile_output`、`run_output` | 捕获到的诊断输出 | | `reason` | 仅 `not_run` 时：一句话说明原因；其余情况为 `""` | @@ -539,6 +572,32 @@ mcpp test [pattern] [--workspace] --message-format json | `not_run_reason` | 它们共同的原因，或 `""` | | `built` | 在 `--no-run` 下构建、本就不打算执行的测试数 | | `elapsed_ms`、`build_ms`、`run_ms` | 墙钟时间，分段给出 | +| `build_group` | *（2026.10.1.1+）* 成员与其他成员一起规划时出现：它的测试所等待的那次构建所属的 `group_build` 记录的 `group` | + +`build_ms` 是该成员的测试等待它们的构建所用的墙钟时间。单独规划的成员，它就是自己 +的构建；与其他成员一起规划的成员，它是整个组的构建时间，同一组的每个成员数字相同， +对 `build_ms` 按成员求和的消费方按 `build_group` 去重；不求和的消费方不受影响。 +`elapsed_ms` 是整个成员的墙钟时间，包含规划与组的构建。 + +### 组记录*（2026.10.1.1+）* + +对多个成员的 `mcpp test` 把同一配置的成员放在一起规划、只构建一次，因此构建的时间 +属于组，只说一次。每个组一条记录，位于该组第一条测试记录之前： + +```json +{"group_build":{"group":0,"members":["libs/a","libs/b"],"build_ms":8210}} +``` + +| 字段 | | +|---|---| +| `group` | 组的编号，从 0 起，按构建顺序 | +| `members` | 该组规划的成员，按 `[workspace] members` 的写法 | +| `build_ms` | 该组构建的墙钟时间：它的包与每个测试二进制 | + +组内某个成员的包构建失败时，报告 +`{"error":"package","member":"…","compile_output":"…"}`，该组中包能构建的成员照常 +运行。用 `-p` 或命令所在目录指定的单个成员的测试，既没有这条记录，也没有 `build_group`； +全体选择（`--workspace`，或虚拟根下不带 `-p`）即使只含一个成员，也按分组形式报告。 **`built` 与 `not_run` 是两个不同的答案，分开计数。** 两者描述的都是一个 编译过、没有执行的测试，相似之处到此为止：`not_run` 意味着 mcpp 试过而做不 diff --git a/mcpp.toml b/mcpp.toml index 0b595e0a7..9422646b9 100644 --- a/mcpp.toml +++ b/mcpp.toml @@ -1,6 +1,6 @@ [package] name = "mcpp" -version = "2026.9.30.2" +version = "2026.10.1.1" description = "Modern C++ build & package management tool" license = "Apache-2.0" authors = ["mcpp-community"] diff --git a/modules/platform/src/process.cppm b/modules/platform/src/process.cppm index d559cded3..730f58d1c 100644 --- a/modules/platform/src/process.cppm +++ b/modules/platform/src/process.cppm @@ -364,6 +364,16 @@ std::string windows_shell_command_line(std::string_view command) { namespace { +// The section every child start holds from the creation of its pipe to the +// parent's close of the write end (see mcpp.platform.unix.bounded_process, which +// states why). Each platform's module defines a class on every platform and +// empties the one that does not apply, so the uses below need no branch. +#if defined(_WIN32) +using LaunchSection = mcpp::platform::winproc::LaunchSection; +#else +using LaunchSection = mcpp::platform::unixproc::LaunchSection; +#endif + // Append a non-interactive stdin redirect to prevent child processes from // blocking on terminal input. // - POSIX: "< /dev/null" — fixes macOS xcrun / xcode-select hangs. @@ -537,7 +547,13 @@ RunResult capture(std::string_view command) { auto cmd = finalize_shell_command(command); RunResult result; - std::FILE* fp = ::popen(cmd.c_str(), "r"); + std::FILE* fp = nullptr; + { + // `popen` creates the pipe and starts the child in one call, so the + // section is exactly that call. + LaunchSection launch; + fp = ::popen(cmd.c_str(), "r"); + } if (!fp) { result.exit_code = -1; return result; @@ -564,8 +580,21 @@ RunResult capture_with_env( const std::vector>& env) { #if defined(_WIN32) - for (auto& [k, v] : env) - _putenv_s(k.c_str(), v.c_str()); + // This mutates the calling process's environment, as it always has, and it + // is called from several threads at once when a workspace's build programs + // are compiled together. A variable is therefore set only when its value + // differs, so callers that pass one value (the toolchain's INCLUDE and LIB, + // the same for every compile of a build) stop touching the environment block + // after the first, and the writes that remain are made under the section the + // children are started in. + { + LaunchSection launch; + for (auto& [k, v] : env) { + const char* now = std::getenv(k.c_str()); + if (now && v == now) continue; + _putenv_s(k.c_str(), v.c_str()); + } + } return capture(command); #else std::string prefixed; @@ -595,7 +624,11 @@ int run_streaming(std::string_view command, std::function on_line) { auto cmd = finalize_shell_command(command); - std::FILE* fp = ::popen(cmd.c_str(), "r"); + std::FILE* fp = nullptr; + { + LaunchSection launch; + fp = ::popen(cmd.c_str(), "r"); + } if (!fp) return -1; std::array buf{}; @@ -624,7 +657,11 @@ int run_streaming(std::string_view command, int run_passthrough(std::string_view command, std::string* output) { auto cmd = finalize_shell_command(command); - std::FILE* fp = ::popen(cmd.c_str(), "r"); + std::FILE* fp = nullptr; + { + LaunchSection launch; + fp = ::popen(cmd.c_str(), "r"); + } if (!fp) return -1; std::array buf{}; @@ -695,7 +732,13 @@ int run_exec(const std::vector& argv, ::posix_spawnattr_setflags(&attr, POSIX_SPAWN_SETPGROUP); pid_t pid = 0; - int sp = ::posix_spawnp(&pid, cargv[0], nullptr, &attr, cargv.data(), envp.data()); + int sp = 0; + { + // A child with no pipe of its own is still started outside the window + // another thread's pipe is open in. + LaunchSection launch; + sp = ::posix_spawnp(&pid, cargv[0], nullptr, &attr, cargv.data(), envp.data()); + } ::posix_spawnattr_destroy(&attr); if (sp != 0) { // Reported once: by the caller when it asked for the errno, here @@ -756,9 +799,6 @@ RunResult capture_exec( #if defined(__linux__) || defined(__APPLE__) // posix_spawn + a pipe; stdout and stderr both go to the pipe so the // captured text is combined (replaces the old `2>&1`). - int fds[2]; - if (::pipe(fds) != 0) { result.exit_code = 127; return result; } - auto envStore = merged_environ(extraEnv); std::vector envp; for (auto& s : envStore) envp.push_back(s.data()); @@ -767,6 +807,18 @@ RunResult capture_exec( for (auto& a : argv) cargv.push_back(const_cast(a.c_str())); cargv.push_back(nullptr); + // Several threads call this at once (a workspace's build programs are + // compiled together). The pipe is close-on-exec, so a child another thread + // starts while this one is open does not keep it open (see + // mcpp.platform.unix.bounded_process for the rule and for what a platform + // without `pipe2` does in addition). + LaunchSection launch; + int fds[2]; + if (mcpp::platform::unixproc::make_pipe(fds) != 0) { + result.exit_code = 127; + return result; + } + posix_spawn_file_actions_t fa; ::posix_spawn_file_actions_init(&fa); // Run the child in `cwd` when requested (e.g. build.mcpp, whose relative @@ -794,6 +846,7 @@ RunResult capture_exec( ::posix_spawnattr_destroy(&attr); ::posix_spawn_file_actions_destroy(&fa); ::close(fds[1]); + launch.release(); if (sp == 0) mcpp::platform::unixproc::guard_group_on_signal(pid); if (sp != 0) { ::close(fds[0]); @@ -845,9 +898,6 @@ RunResult capture_stdout( if (spawn_error) *spawn_error = 0; if (argv.empty()) { result.exit_code = 127; return result; } #if defined(__linux__) || defined(__APPLE__) - int fds[2]; - if (::pipe(fds) != 0) { result.exit_code = 127; return result; } - auto envStore = merged_environ(extraEnv); std::vector envp; for (auto& s : envStore) envp.push_back(s.data()); @@ -856,6 +906,14 @@ RunResult capture_stdout( for (auto& a : argv) cargv.push_back(const_cast(a.c_str())); cargv.push_back(nullptr); + // The same rule as capture_exec's. + LaunchSection launch; + int fds[2]; + if (mcpp::platform::unixproc::make_pipe(fds) != 0) { + result.exit_code = 127; + return result; + } + posix_spawn_file_actions_t fa; ::posix_spawn_file_actions_init(&fa); ::posix_spawn_file_actions_addopen(&fa, 0, "/dev/null", O_RDONLY, 0); @@ -876,6 +934,7 @@ RunResult capture_stdout( ::posix_spawnattr_destroy(&attr); ::posix_spawn_file_actions_destroy(&fa); ::close(fds[1]); + launch.release(); if (sp != 0) { ::close(fds[0]); result.exit_code = 127; diff --git a/modules/platform/src/terminal.cppm b/modules/platform/src/terminal.cppm index 3adc2f95b..b42df85ce 100644 --- a/modules/platform/src/terminal.cppm +++ b/modules/platform/src/terminal.cppm @@ -4,7 +4,7 @@ // is_tty() — whether stdout is a terminal // is_terminal(s) — whether a standard stream is a terminal // can_move_cursor(s)— whether a live display may be drawn on it -// cols(), rows() — the terminal's size +// cols(s), rows(s) — the size of the terminal a stream reaches // write(s, text) — UTF-8 text to a standard stream // write_frame(s, b) — a frame of a live display, in one write // same_terminal() — whether stdout and stderr reach one terminal @@ -63,12 +63,15 @@ bool is_terminal(Stream s); // that the escape sequences mcpp writes are interpreted rather than printed. bool can_move_cursor(Stream s); -// Returns the terminal width in columns. Tries the terminal first (TIOCGWINSZ, -// or the console's window on Windows), falls back to $COLUMNS, then to 80. -std::size_t cols(); +// The width in columns of the terminal the stream reaches. Tries that terminal +// first (TIOCGWINSZ on the stream's descriptor, or the console's window on +// Windows), falls back to $COLUMNS, then to 80. The stream is a parameter +// because the display is drawn on one stream while the other may be a pipe: a +// width asked of a descriptor that is not a terminal is always the fallback. +std::size_t cols(Stream s); // The terminal's height in rows, by the same order, with $LINES and 24. -std::size_t rows(); +std::size_t rows(Stream s); // Writes UTF-8 text to the stream. A Windows console receives it as UTF-16 // through WriteConsoleW, so that text outside ASCII (`·`, `→`, a path in @@ -90,17 +93,18 @@ void write_frame(Stream s, std::string_view bytes); // Whether standard error reaches the terminal standard output reaches: both // are terminals and, on POSIX, the same device; on Windows both are console -// handles, and a process has at most one console. A line for standard error -// can then travel in a frame written to standard output. +// handles, and a process has at most one console. A line for the stream a live +// display is not drawn on can then travel in the frame written to the stream it +// is drawn on. bool same_terminal(); -// Whether the terminal is likely to draw East Asian ambiguous-width -// characters (`·`, `…`, `→`, the box-drawing block) two columns wide: a -// Windows console whose output code page is 932, 936, 949 or 950, or, on -// POSIX, a locale (LC_ALL, LC_CTYPE, then LANG) for Chinese, Japanese or +// Whether the terminal the stream reaches is likely to draw East Asian +// ambiguous-width characters (`·`, `…`, `→`, the box-drawing block) two columns +// wide: a Windows console whose output code page is 932, 936, 949 or 950, or, +// on POSIX, a locale (LC_ALL, LC_CTYPE, then LANG) for Chinese, Japanese or // Korean. The answer is a likelihood; a live display that budgets its width // by it is never wider than the terminal either way. -bool ambiguous_wide(); +bool ambiguous_wide(Stream s); // Whether the terminal can be expected to draw characters beyond ASCII from // its font or a fallback, braille included: on POSIX a UTF-8 locale (LC_ALL, @@ -122,15 +126,15 @@ std::vector decode_keys(std::string& pending); // KEYS READ FROM THE TERMINAL FOR THE LIFETIME OF THE OBJECT, without echo and // without waiting for a line end; Ctrl-C still interrupts. Active only when -// standard input and standard output are both terminals and, on POSIX, mcpp -// is in the terminal's foreground process group: a background job that -// changed the terminal's mode would be stopped by SIGTTOU. The mode the -// terminal had is restored when the object is destroyed, and by the signal -// handler if a signal ends mcpp first (POSIX: `unixproc::guard_terminal_mode`; -// Windows: a console control handler). +// standard input and the stream the display is drawn on (`display`) are both +// terminals and, on POSIX, mcpp is in the terminal's foreground process group: +// a background job that changed the terminal's mode would be stopped by +// SIGTTOU. The mode the terminal had is restored when the object is destroyed, +// and by the signal handler if a signal ends mcpp first (POSIX: +// `unixproc::guard_terminal_mode`; Windows: a console control handler). class KeyInput { public: - KeyInput(); + explicit KeyInput(Stream display); ~KeyInput(); KeyInput(const KeyInput&) = delete; KeyInput& operator=(const KeyInput&) = delete; @@ -234,28 +238,32 @@ std::size_t from_env(const char* name, std::size_t fallback) { } } // namespace -std::size_t cols() { +std::size_t cols(Stream s) { #if defined(_WIN32) CONSOLE_SCREEN_BUFFER_INFO info{}; - if (HANDLE h; console_of(Stream::Out, &h) && ::GetConsoleScreenBufferInfo(h, &info)) + if (HANDLE h; console_of(s, &h) && ::GetConsoleScreenBufferInfo(h, &info)) return static_cast(info.srWindow.Right - info.srWindow.Left + 1); #elif defined(__unix__) || defined(__APPLE__) struct winsize w{}; - if (::ioctl(::fileno(stdout), TIOCGWINSZ, &w) == 0 && w.ws_col > 0) + if (::ioctl(::fileno(file_of(s)), TIOCGWINSZ, &w) == 0 && w.ws_col > 0) return w.ws_col; +#else + (void)s; #endif return from_env("COLUMNS", 80); } -std::size_t rows() { +std::size_t rows(Stream s) { #if defined(_WIN32) CONSOLE_SCREEN_BUFFER_INFO info{}; - if (HANDLE h; console_of(Stream::Out, &h) && ::GetConsoleScreenBufferInfo(h, &info)) + if (HANDLE h; console_of(s, &h) && ::GetConsoleScreenBufferInfo(h, &info)) return static_cast(info.srWindow.Bottom - info.srWindow.Top + 1); #elif defined(__unix__) || defined(__APPLE__) struct winsize w{}; - if (::ioctl(::fileno(stdout), TIOCGWINSZ, &w) == 0 && w.ws_row > 0) + if (::ioctl(::fileno(file_of(s)), TIOCGWINSZ, &w) == 0 && w.ws_row > 0) return w.ws_row; +#else + (void)s; #endif return from_env("LINES", 24); } @@ -332,12 +340,14 @@ bool same_terminal() { #endif } -bool ambiguous_wide() { +bool ambiguous_wide(Stream s) { #if defined(_WIN32) - if (console_of(Stream::Out)) { + if (console_of(s)) { const UINT cp = ::GetConsoleOutputCP(); if (cp == 932 || cp == 936 || cp == 949 || cp == 950) return true; } +#else + (void)s; #endif for (const char* name : {"LC_ALL", "LC_CTYPE", "LANG"}) { const char* v = std::getenv(name); @@ -418,8 +428,8 @@ std::vector decode_keys(std::string& pending) { return keys; } -KeyInput::KeyInput() { - if (!is_terminal(Stream::Out)) return; +KeyInput::KeyInput(Stream display) { + if (!is_terminal(display)) return; #if defined(_WIN32) const HANDLE h = ::GetStdHandle(STD_INPUT_HANDLE); DWORD mode = 0; diff --git a/modules/platform/src/unix/bounded_process.cppm b/modules/platform/src/unix/bounded_process.cppm index 9757652db..1de9f7253 100644 --- a/modules/platform/src/unix/bounded_process.cppm +++ b/modules/platform/src/unix/bounded_process.cppm @@ -79,6 +79,43 @@ struct DeadlineRun { using OutputSink = void (*)(void* ctx, const char* data, unsigned long len); +// ─── Starting a child is safe for concurrent callers ───────────────────── +// +// Two threads that start children at the same time must not pass one child's +// pipe to the other. A child inherits every descriptor of its parent that is not +// close-on-exec, so a child started by thread B while thread A holds the write +// end of A's capture pipe keeps that end open for as long as it lives. A's +// reader then waits for end of file until B's unrelated child exits: a compile +// that finished at once is reported only when a longer one does, and a deadline +// the reader is meant to enforce is enforced late. +// +// The pipes are therefore created close-on-exec, and `posix_spawn`'s `dup2` +// action, which clears the flag on the descriptor it creates, is the only way a +// descriptor reaches the child. Where the platform has `pipe2` (Linux) that is +// one atomic call. Where it does not (macOS) the flag is set by a second call, +// which leaves a window in which another thread's child could be started; every +// start on such a platform therefore holds one process-wide section, from the +// creation of the pipe to the parent's close of its write end. On Linux the +// section is empty. +// +// `make_pipe` returns 0 on success and -1 on failure, as `pipe` does. Both ends +// are close-on-exec. The caller holds a `LaunchSection` across this call and the +// start of the child. +int make_pipe(int fds[2]); + +class LaunchSection { +public: + LaunchSection(); + ~LaunchSection(); + // Leaves the section before the destructor does: a caller that goes on to + // wait for its child must not hold it for the child's whole life. + void release(); + LaunchSection(const LaunchSection&) = delete; + LaunchSection& operator=(const LaunchSection&) = delete; +private: + bool held_ = false; +}; + // `argvEntries` is `argvCount` NUL-terminated strings; `envEntries` is // `envCount` "KEY=VALUE" strings applied on top of the current environment. // `cwd` may be null. A non-positive `deadlineMs` is rejected with @@ -173,6 +210,13 @@ void background_stop(long long group, long long graceMs); // Fixed capacity and no allocation: the handler reads this array and may not // allocate. Registering past capacity fails loudly rather than silently // dropping a group, because a dropped group is an orphan nobody will find. +// +// SAFE FOR CONCURRENT CALLERS. Children are started from several threads at once +// (a workspace's build programs are compiled concurrently), so claiming and +// releasing a slot is serialized by a mutex, and two threads can no longer claim +// the same slot. The handler takes no lock: it reads slots that are written one +// `sig_atomic_t` at a time. The capacity is 256, well above any job count, since +// one group is held per running child. void guard_group_on_signal(long long group); void unguard_group(long long group); void clear_group_guard(); @@ -262,8 +306,10 @@ DeadlineRun capture_with_deadline(const char* const* argvEntries, cargv.push_back(nullptr); const bool capture = (sink != nullptr); + // From the pipe to the parent's close of its write end: see LaunchSection. + LaunchSection launch; int fds[2] = {-1, -1}; - if (capture && ::pipe(fds) != 0) return out; + if (capture && make_pipe(fds) != 0) return out; posix_spawn_file_actions_t fa; ::posix_spawn_file_actions_init(&fa); @@ -292,6 +338,7 @@ DeadlineRun capture_with_deadline(const char* const* argvEntries, ::posix_spawnattr_destroy(&attr); ::posix_spawn_file_actions_destroy(&fa); if (capture) ::close(fds[1]); + launch.release(); if (sp != 0) { out.spawn_error = sp; if (capture) ::close(fds[0]); return out; } if (ownGroup) guard_group_on_signal(pid); @@ -364,8 +411,14 @@ namespace { // Read by a signal handler, so `volatile sig_atomic_t` and nothing else: the // handler may run between any two instructions and may not lock, allocate, or // call anything that is not async-signal-safe. 0 means "nothing to clean up". -constexpr int kMaxGuardedGroups = 8; +constexpr int kMaxGuardedGroups = 256; volatile sig_atomic_t g_guardedGroups[kMaxGuardedGroups] = {}; +// Serializes the WRITERS of the registry and of the handlers it installs. The +// handler never takes it. +std::mutex g_guardMutex; + +// The platforms without `pipe2` share one section across every child start. +std::mutex g_launchMutex; // The terminal whose mode the handler restores (-1: none), and that mode. volatile sig_atomic_t g_terminalFd = -1; @@ -396,6 +449,32 @@ extern "C" void background_signal_handler(int sig) { } // namespace +int make_pipe(int fds[2]) { +#if defined(__linux__) + return ::pipe2(fds, O_CLOEXEC); +#else + if (::pipe(fds) != 0) return -1; + ::fcntl(fds[0], F_SETFD, FD_CLOEXEC); + ::fcntl(fds[1], F_SETFD, FD_CLOEXEC); + return 0; +#endif +} + +LaunchSection::LaunchSection() { +#if !defined(__linux__) + g_launchMutex.lock(); + held_ = true; +#endif +} + +LaunchSection::~LaunchSection() { release(); } + +void LaunchSection::release() { + if (!held_) return; + held_ = false; + g_launchMutex.unlock(); +} + BackgroundChild spawn_background(const char* const* argvEntries, unsigned long argvCount, const char* cwd, @@ -430,8 +509,14 @@ BackgroundChild spawn_background(const char* const* argvEntries, ::posix_spawnattr_setflags(&attr, POSIX_SPAWN_SETPGROUP); pid_t pid = 0; - const int sp = ::posix_spawnp(&pid, cargv[0], &fa, &attr, - cargv.data(), current_environ()); + int sp = 0; + { + // A child that holds no pipe of its own must still not be started in + // the window another thread's pipe is open in (see LaunchSection). + LaunchSection launch; + sp = ::posix_spawnp(&pid, cargv[0], &fa, &attr, + cargv.data(), current_environ()); + } ::posix_spawn_file_actions_destroy(&fa); ::posix_spawnattr_destroy(&attr); if (sp != 0) return out; @@ -490,17 +575,20 @@ void background_stop(long long group, long long graceMs) { void guard_group_on_signal(long long group) { if (group <= 0) return; - for (int i = 0; i < kMaxGuardedGroups; ++i) { - if (g_guardedGroups[i] == 0) { - g_guardedGroups[i] = static_cast(group); - install_handlers(); - return; + { + std::lock_guard lock(g_guardMutex); + for (int i = 0; i < kMaxGuardedGroups; ++i) { + if (g_guardedGroups[i] == 0) { + g_guardedGroups[i] = static_cast(group); + install_handlers(); + return; + } } } // Out of slots. Say so rather than return silently: an unguarded group is // a process that outlives mcpp, and the whole point of this file is that // such a process is never acceptable. - std::fputs("mcpp: internal: more than 8 concurrently guarded process " + std::fputs("mcpp: internal: more than 256 concurrently guarded process " "groups; the newest is NOT guarded and may outlive mcpp\n", stderr); } @@ -510,6 +598,7 @@ void guard_group_on_signal(long long group) { // the guard an outer one still needs. void unguard_group(long long group) { if (group <= 0) return; + std::lock_guard lock(g_guardMutex); bool any = false; for (int i = 0; i < kMaxGuardedGroups; ++i) { if (g_guardedGroups[i] == static_cast(group)) @@ -525,6 +614,7 @@ void unguard_group(long long group) { } void clear_group_guard() { + std::lock_guard lock(g_guardMutex); for (int i = 0; i < kMaxGuardedGroups; ++i) g_guardedGroups[i] = 0; if (g_terminalFd >= 0) return; // the terminal's guard still needs the handler ::signal(SIGINT, SIG_DFL); @@ -533,12 +623,14 @@ void clear_group_guard() { } void guard_terminal_mode(int fd) { + std::lock_guard lock(g_guardMutex); if (fd < 0 || ::tcgetattr(fd, &g_terminalMode) != 0) return; g_terminalFd = fd; install_handlers(); } void unguard_terminal_mode() { + std::lock_guard lock(g_guardMutex); if (g_terminalFd < 0) return; ::tcsetattr(g_terminalFd, TCSANOW, &g_terminalMode); g_terminalFd = -1; @@ -575,6 +667,10 @@ BackgroundChild spawn_background(const char* const*, unsigned long, } int background_running(long long, int*) { return -1; } void background_stop(long long, long long) {} +int make_pipe(int[2]) { return -1; } +LaunchSection::LaunchSection() {} +LaunchSection::~LaunchSection() {} +void LaunchSection::release() {} void guard_group_on_signal(long long) {} void unguard_group(long long) {} void clear_group_guard() {} diff --git a/modules/platform/src/windows/bounded_process.cppm b/modules/platform/src/windows/bounded_process.cppm index 3fdb7444a..6f34b4e8a 100644 --- a/modules/platform/src/windows/bounded_process.cppm +++ b/modules/platform/src/windows/bounded_process.cppm @@ -82,6 +82,36 @@ struct DeadlineRun { int spawn_error = 0; }; +// ─── Starting a child is safe for concurrent callers ───────────────────── +// +// `CreateProcess` with `bInheritHandles = TRUE` hands a child EVERY inheritable +// handle of its parent, and a capture pipe's write end has to be inheritable for +// its own child. A child started by thread B while thread A holds that write end +// keeps it open for as long as it lives, and A's reader then waits for end of +// file until B's unrelated child exits. (`SetHandleInformation` cannot close the +// window: a handle is inheritable from its creation to the moment the parent +// closes it, and the start of the child lies between the two.) +// +// So the starts that can run at the same time as another, the ones below and +// the `_popen` calls of mcpp.platform.process, hold one process-wide section +// from the creation of the pipe to the parent's close of its write end. The +// alternative, `PROC_THREAD_ATTRIBUTE_HANDLE_LIST`, would restrict what OUR +// launchers pass and leave `_popen` and any third party passing everything; the +// section is one rule for all of them. +// +// On another platform the class exists and does nothing. +class LaunchSection { +public: + LaunchSection(); + ~LaunchSection(); + // Leaves the section before the destructor does. + void release(); + LaunchSection(const LaunchSection&) = delete; + LaunchSection& operator=(const LaunchSection&) = delete; +private: + bool held_ = false; +}; + // Receives stdout+stderr as it arrives. Called on the calling thread only. // // A NULL sink means "do not capture": the child inherits the caller's stdio and @@ -169,6 +199,10 @@ void background_stop(unsigned long long job, unsigned long long process, // A REGISTRY AND NOT ONE SLOT, for the reason the POSIX peer gives: a spanning // `[hooks]` command and the build's own ninja are guarded at the same time, and // a single slot lets the second registration disarm the first. +// +// SAFE FOR CONCURRENT CALLERS, as the POSIX peer is: claiming and releasing a +// slot is serialized by a mutex and the console handler takes no lock. The +// capacity is 256. void guard_job_on_signal(unsigned long long job); void unguard_job(unsigned long long job); void clear_job_guard(); @@ -238,6 +272,9 @@ std::string environment_block(const char* const* envEntries, return block; } +// The one section every child start shares (see LaunchSection). +std::mutex g_launchMutex; + struct Handle { HANDLE h = nullptr; Handle() = default; @@ -252,6 +289,19 @@ struct Handle { } // namespace +LaunchSection::LaunchSection() { + g_launchMutex.lock(); + held_ = true; +} + +LaunchSection::~LaunchSection() { release(); } + +void LaunchSection::release() { + if (!held_) return; + held_ = false; + g_launchMutex.unlock(); +} + DeadlineRun capture_with_deadline(const char* commandLine, const char* const* envEntries, unsigned long envCount, @@ -273,6 +323,9 @@ DeadlineRun capture_with_deadline(const char* commandLine, sa.nLength = sizeof(sa); sa.bInheritHandle = TRUE; + // From the creation of the pipe to the parent's close of its write end: no + // other child may be started while this pipe is inheritable. + LaunchSection launch; Handle readEnd, writeEnd; if (capture) { if (!::CreatePipe(&readEnd.h, &writeEnd.h, &sa, 0)) return out; @@ -350,6 +403,7 @@ DeadlineRun capture_with_deadline(const char* commandLine, // The parent must drop its copy of the write end or the pipe never reaches // EOF, even after every child has exited. writeEnd.reset(); + launch.release(); const auto started = std::chrono::steady_clock::now(); const auto until = deadlineMs > 0 @@ -414,8 +468,11 @@ namespace { // Read by a console control handler, which runs on a thread of the OS's // choosing. Only the handle is shared, and closing a job handle is atomic from // the caller's point of view. -constexpr int kMaxGuardedJobs = 8; +constexpr int kMaxGuardedJobs = 256; volatile unsigned long long g_guardedJobs[kMaxGuardedJobs] = {}; +// Serializes the WRITERS of the registry and of the handler it installs; the +// console handler never takes it. +std::mutex g_guardMutex; BOOL WINAPI background_console_handler(DWORD) { // TerminateJobObject, not CloseHandle: this handler races `background_stop` @@ -477,12 +534,19 @@ BackgroundChild spawn_background(const char* commandLine, // CREATE_NEW_PROCESS_GROUP is the peer of POSIX_SPAWN_SETPGROUP: the child // stops receiving the console's Ctrl-C, which is what makes the guard // below necessary and what stops a stray Ctrl-C from half-killing the tree. - const BOOL ok = ::CreateProcessA( - nullptr, cmdBuf.data(), nullptr, nullptr, - /*bInheritHandles=*/TRUE, - CREATE_SUSPENDED | CREATE_NEW_PROCESS_GROUP - | (inheritStdio ? 0u : CREATE_NO_WINDOW), - nullptr, (cwd && *cwd) ? cwd : nullptr, &si, &pi); + BOOL ok = FALSE; + { + // This child inherits every inheritable handle, another thread's + // capture pipe included, unless it is started outside the window that + // pipe is open in (see LaunchSection). + LaunchSection launch; + ok = ::CreateProcessA( + nullptr, cmdBuf.data(), nullptr, nullptr, + /*bInheritHandles=*/TRUE, + CREATE_SUSPENDED | CREATE_NEW_PROCESS_GROUP + | (inheritStdio ? 0u : CREATE_NO_WINDOW), + nullptr, (cwd && *cwd) ? cwd : nullptr, &si, &pi); + } if (!ok) { out.refused = ::GetLastError(); if (job) ::CloseHandle(job); @@ -555,19 +619,23 @@ void background_stop(unsigned long long job, unsigned long long process, void guard_job_on_signal(unsigned long long job) { if (!job) return; - for (int i = 0; i < kMaxGuardedJobs; ++i) { - if (g_guardedJobs[i] == 0) { - g_guardedJobs[i] = job; - ::SetConsoleCtrlHandler(background_console_handler, TRUE); - return; + { + std::lock_guard lock(g_guardMutex); + for (int i = 0; i < kMaxGuardedJobs; ++i) { + if (g_guardedJobs[i] == 0) { + g_guardedJobs[i] = job; + ::SetConsoleCtrlHandler(background_console_handler, TRUE); + return; + } } } - std::fputs("mcpp: internal: more than 8 concurrently guarded jobs; the " + std::fputs("mcpp: internal: more than 256 concurrently guarded jobs; the " "newest is NOT guarded and may outlive mcpp\n", stderr); } void unguard_job(unsigned long long job) { if (!job) return; + std::lock_guard lock(g_guardMutex); bool any = false; for (int i = 0; i < kMaxGuardedJobs; ++i) { if (g_guardedJobs[i] == job) g_guardedJobs[i] = 0; @@ -577,6 +645,7 @@ void unguard_job(unsigned long long job) { } void clear_job_guard() { + std::lock_guard lock(g_guardMutex); for (int i = 0; i < kMaxGuardedJobs; ++i) g_guardedJobs[i] = 0; ::SetConsoleCtrlHandler(background_console_handler, FALSE); } @@ -599,6 +668,10 @@ DeadlineRun capture_with_deadline(const char*, const char* const*, unsigned long return {}; } +LaunchSection::LaunchSection() {} +LaunchSection::~LaunchSection() {} +void LaunchSection::release() {} + BackgroundChild spawn_background(const char*, const char*, int) { return {}; } int background_running(unsigned long long, int*) { return -1; } void background_stop(unsigned long long, unsigned long long, long long) {} diff --git a/modules/platform/tests/test_process_concurrent_children.cpp b/modules/platform/tests/test_process_concurrent_children.cpp new file mode 100644 index 000000000..69f2d33d6 --- /dev/null +++ b/modules/platform/tests/test_process_concurrent_children.cpp @@ -0,0 +1,105 @@ +#include + +#if !defined(_WIN32) +#include +#include +#endif + +import std; +import mcpp.platform.process; +import mcpp.platform.unix.bounded_process; + +// SUBSYSTEM-LEVEL, and the contract is B2-0 of the member-selection and +// build-program-cost plan (#748): the launcher is safe for concurrent callers. +// +// A workspace's build programs are compiled together, so several threads start +// children at the same moment. A child inherits every descriptor (every +// inheritable handle on Windows) its parent holds, so a child started by one +// thread while another thread's capture pipe is open keeps that pipe's write end +// open for as long as it lives. The other thread's reader then waits for end of +// file until an unrelated child exits. + +namespace proc = mcpp::platform::process; + +namespace { + +using Clock = std::chrono::steady_clock; + +#if defined(_WIN32) +// `hostname` exits at once on every Windows; `ping -n 2` answers the second +// time one second after the first, which is the slowest child this needs. +const std::vector kQuick{"hostname"}; +const std::vector kSlow{"ping", "-n", "2", "127.0.0.1"}; +constexpr auto kSlowFor = std::chrono::milliseconds(1000); +constexpr int kRounds = 6; +#else +const std::vector kQuick{"/bin/sh", "-c", "echo quick"}; +const std::vector kSlow{"/bin/sh", "-c", "sleep 0.5"}; +constexpr auto kSlowFor = std::chrono::milliseconds(500); +constexpr int kRounds = 12; +#endif + +struct Outcome { + Clock::time_point end{}; + int exit_code = -1; + std::string output; +}; + +// Runs `argv` once the gate opens and records when its reader returned. +Outcome run_when_open(const std::atomic& gate, const std::vector& argv) { + while (!gate.load(std::memory_order_acquire)) std::this_thread::yield(); + auto r = proc::capture_exec(argv); + return {Clock::now(), r.exit_code, std::move(r.output)}; +} + +} // namespace + +// Two children started at once from two threads, one of which exits at once and +// one of which sleeps: the reader of the first returns when the first child +// exits, not when the second does. +// +// The window a leaked descriptor needs is the few hundred microseconds between +// the creation of a pipe and the parent's close of its write end, so one round +// cannot be expected to fall inside it. Each round releases both threads at one +// instant, which makes an overlap likely, and the rounds are repeated. With the +// launcher as it was, an overlap turns the quick reader's return into the slow +// child's exit; with it fixed, no round can. +TEST(ConcurrentChildren, AReaderReturnsWhenItsOwnChildExits) { + for (int round = 0; round < kRounds; ++round) { + std::atomic gate{false}; + Outcome quick, slow; + std::thread tSlow([&] { slow = run_when_open(gate, kSlow); }); + std::thread tQuick([&] { quick = run_when_open(gate, kQuick); }); + gate.store(true, std::memory_order_release); + tQuick.join(); + tSlow.join(); + + ASSERT_EQ(quick.exit_code, 0) << "round " << round; + ASSERT_EQ(slow.exit_code, 0) << "round " << round; + // Each reader is handed its own child's output and nothing else. + EXPECT_NE(quick.output.find("quick"), std::string::npos) << quick.output; + // The quick child was gone long before the slow one. Half the slow + // child's life is the margin: it does not depend on how fast the + // machine starts a process, only on the two readers not sharing a pipe. + const auto gap = std::chrono::duration_cast( + slow.end - quick.end); + EXPECT_GT(gap, kSlowFor / 2) + << "round " << round << ": the reader of the child that exited at once " + << "returned only " << gap.count() << " ms before the slow child did; " + << "it waited for a child another thread started"; + } +} + +#if !defined(_WIN32) +// The mechanism behind the test above, stated directly. A pipe the launcher +// creates is close-on-exec on both ends, so the only descriptors a child can +// inherit are the ones `posix_spawn` duplicates onto its standard streams. +TEST(ConcurrentChildren, APipeTheLauncherCreatesClosesOnExec) { + int fds[2] = {-1, -1}; + ASSERT_EQ(mcpp::platform::unixproc::make_pipe(fds), 0); + EXPECT_TRUE(::fcntl(fds[0], F_GETFD) & FD_CLOEXEC) << "the read end is inheritable"; + EXPECT_TRUE(::fcntl(fds[1], F_GETFD) & FD_CLOEXEC) << "the write end is inheritable"; + ::close(fds[0]); + ::close(fds[1]); +} +#endif diff --git a/modules/versioning/src/version.cppm b/modules/versioning/src/version.cppm index 2b3e3e890..6ea6470ae 100644 --- a/modules/versioning/src/version.cppm +++ b/modules/versioning/src/version.cppm @@ -31,6 +31,6 @@ import std; export namespace mcpp { -inline constexpr std::string_view MCPP_VERSION = "2026.9.30.2"; +inline constexpr std::string_view MCPP_VERSION = "2026.10.1.1"; } // namespace mcpp diff --git a/src/bmi_cache.cppm b/src/bmi_cache.cppm index 20101bbb5..fb4001f88 100644 --- a/src/bmi_cache.cppm +++ b/src/bmi_cache.cppm @@ -41,6 +41,18 @@ // concurrent builds racing to fill the same entry cannot interleave writes, and // writes entry.json last so a crash mid-populate leaves a miss, not a // half-populated hit. +// +// A SECOND KIND OF ENTRY IS PUBLISHED WHOLE (#748, B1). What a build program +// imports, the bundled `mcpp` module and the host modules of rule packages, is +// compiled in a directory of its own and then moved into place, so the entry +// never exists half-written and two threads (or two mcpp processes) compiling +// the same key cannot interleave their files. `stage_entry` creates that +// directory beside the entry, on the same file system, and `publish_staged` +// writes entry.json into it, last, and renames it to the entry's address. The +// entry has the layout above, so `probe_cached`, `touch_accessed`, +// `mcpp cache gc`, `list`, `info` and `verify` treat it as they treat any other. +// Its address is either below the package address, in the global cache, or +// exactly `CacheKey::directDir`, in a workspace's own store. module; @@ -73,8 +85,16 @@ struct CacheKey { nlohmann::json inputs; std::string bmiDirName = "gcm.cache"; // consumer-side directory name std::string manifestTag = "gcm"; // "gcm" | "pcm" + // When set, the entry lives exactly here rather than below the package + // address: a workspace's own store of what belongs to that workspace (a host + // module from a path dependency, whose sources can change without its name + // and version changing). Everything below it is laid out as in the global + // cache. `cacheRoot` then names where staging directories go, and is the + // workspace store's own root. + std::filesystem::path directDir; std::filesystem::path dir() const { + if (!directDir.empty()) return directDir; return cacheRoot / "pkg" / indexName / std::format("{}@{}", packageName, version) / keyHex; } @@ -165,6 +185,26 @@ populate_from(const CacheKey& key, const std::filesystem::path& projectTargetDir, const DepArtifacts& artifacts); +// A fresh directory a producer writes an entry's `bmi/` and `obj/` into, on the +// same file system as the entry so that `publish_staged` can rename it into +// place. It is outside every directory `mcpp cache` walks, so a producer that is +// interrupted leaves nothing for `gc` to report as an incomplete entry. +std::expected +stage_entry(const CacheKey& key); + +// Writes entry.json into `staged`, last, and renames `staged` to the entry's +// address. `artifacts` lists what `staged` holds (`objFiles[].cacheRel` names +// the file below `obj/`; `buildRel` is not used). Returns true when this call +// put the entry in place and false when an entry that satisfies `artifacts` was +// already there, another thread's or another process's, in which case `staged` +// is removed and the entry in place is the one to use. An entry that is there +// and does not satisfy it (a crash that predates the rename, an older layout) is +// replaced. +std::expected +publish_staged(const CacheKey& key, + const std::filesystem::path& staged, + const DepArtifacts& artifacts); + // Absolute paths of an entry's artifacts, for the stage edges. std::filesystem::path cached_bmi_path(const CacheKey& key, std::string_view basename); std::filesystem::path cached_obj_path(const CacheKey& key, std::string_view rel); @@ -381,4 +421,115 @@ populate_from(const CacheKey& key, return write_entry(key.entryFile(), j); } +std::expected +stage_entry(const CacheKey& key) { + // Unique across threads, processes and calls without asking the OS for a + // process id: a counter for the threads of this process, and the clock and a + // random number for the others. + static std::atomic counter{0}; + static const unsigned long long salt = [] { + std::random_device rd; + return (static_cast(rd()) << 32) ^ rd(); + }(); + const auto ticks = std::chrono::steady_clock::now().time_since_epoch().count(); + // Short: the compiler is handed paths below this directory, and a path of a + // few hundred characters is one some Windows tools cannot open. + const auto unique = std::format("{:.8}.{:08x}.{}", key.keyHex, + static_cast(salt ^ static_cast(ticks)), + counter++); + const auto base = key.directDir.empty() + ? key.cacheRoot / "tmp" + : key.directDir.parent_path() / ".tmp"; + const auto staged = base / unique; + std::error_code ec; + // A producer that was interrupted (Ctrl-C, a killed build) leaves its staging + // directory behind, and nothing else removes it: no listing of the cache + // reaches it. One that is a day old belongs to no producer still running. + { + const auto cutoff = std::filesystem::file_time_type::clock::now() - std::chrono::hours(24); + std::error_code lec; + for (auto const& old : std::filesystem::directory_iterator(base, lec)) { + std::error_code tec; + if (std::filesystem::last_write_time(old.path(), tec) < cutoff && !tec) + std::filesystem::remove_all(old.path(), tec); + } + } + std::filesystem::create_directories(staged / "bmi", ec); + if (ec) return std::unexpected(std::format( + "cannot create a staging directory under '{}': {}", base.string(), ec.message())); + std::filesystem::create_directories(staged / "obj", ec); + if (ec) return std::unexpected(std::format( + "cannot create a staging directory under '{}': {}", base.string(), ec.message())); + return staged; +} + +std::expected +publish_staged(const CacheKey& key, + const std::filesystem::path& staged, + const DepArtifacts& arts) +{ + std::error_code ec; + // Every artifact the entry is about to claim is in the directory now, before + // anything names it. + for (auto& g : arts.bmiFiles) + if (!std::filesystem::exists(staged / "bmi" / g, ec)) { + std::filesystem::remove_all(staged, ec); + return std::unexpected(std::format("expected build output missing: bmi/{}", g)); + } + for (auto& o : arts.objFiles) + if (!std::filesystem::exists(staged / "obj" / std::filesystem::path(o.cacheRel), ec)) { + std::filesystem::remove_all(staged, ec); + return std::unexpected(std::format("expected build output missing: obj/{}", o.cacheRel)); + } + + nlohmann::json j; + j["created"] = now_iso8601(); + j["schema"] = kEntrySchema; + j["key"] = key.keyHex; + j["package"] = std::format("{}/{}@{}", key.indexName, key.packageName, key.version); + j["bmi_dir"] = key.bmiDirName; + j["tag"] = key.manifestTag; + j["inputs"] = key.inputs; + j["bmi"] = arts.bmiFiles; + { + auto objs = nlohmann::json::array(); + for (auto& o : arts.objFiles) objs.push_back(o.cacheRel); + j["obj"] = std::move(objs); + } + j["accessed"] = now_iso8601(); + // entry.json LAST: it is the sentinel, and it is written into the directory + // before the directory has an address, so the entry is never seen without it. + if (auto w = write_entry(staged / "entry.json", j); !w) { + std::filesystem::remove_all(staged, ec); + return std::unexpected(w.error()); + } + + const auto target = key.dir(); + std::filesystem::create_directories(target.parent_path(), ec); + std::filesystem::rename(staged, target, ec); + if (!ec) return true; + + // The target exists (a directory is not replaced by a rename). Another + // producer finished first, and its entry is as good as this one when it + // satisfies what was asked: use it. + if (probe_cached(key, arts).ok) { + std::filesystem::remove_all(staged, ec); + return false; + } + // It does not, so it is a remnant, and the next step replaces it. + std::filesystem::remove_all(target, ec); + std::filesystem::rename(staged, target, ec); + if (!ec) return true; + // A third producer may have won in between; the entry is usable then. + if (probe_cached(key, arts).ok) { + std::error_code rm; + std::filesystem::remove_all(staged, rm); + return false; + } + std::error_code rm; + std::filesystem::remove_all(staged, rm); + return std::unexpected(std::format( + "cannot publish cache entry '{}': {}", target.string(), ec.message())); +} + } // namespace mcpp::bmi_cache diff --git a/src/build/build_program.cppm b/src/build/build_program.cppm index 53ee91d21..6336b349a 100644 --- a/src/build/build_program.cppm +++ b/src/build/build_program.cppm @@ -14,6 +14,8 @@ export module mcpp.build.build_program; import std; import mcpp.diag; // structured diagnostics of build programs (#734 E11) +import mcpp.home; // cache_root — the global home of the engine's compiled module (#748) +import mcpp.log; // the compile of a build program is timestamped in the verbose log import mcpp.manifest; import mcpp.platform; import mcpp.pm.mangle; // imported_module_names -- what build.mcpp asks for @@ -24,7 +26,8 @@ import mcpp.toolchain.fingerprint; // hash_file / hash_string (FNV-1a, 16 hex) import mcpp.build.directives; // the directive definition table (own module: see its header) import mcpp.build.progress; // the program's line (build progress design 2026-09-29) import mcpp.build.refusal; // the machine-readable identity of a refusal -import mcpp.build.hostprogram; // bundled `mcpp` module compile (own module: see its header) +import mcpp.build.hostprogram; // the bundled `mcpp` module's text (own module: see its header) +import mcpp.build.host_module_compile; // the bundled module and the host modules, compiled once per key (#748) import mcpp.build.resources; // compile_utf8_manifest — the build program speaks UTF-8 (#693) import mcpp.toolchain.hostflags; // the shared host-compile flag producer import mcpp.toolchain.linkmodel; // shared C-library / clang-cfg-bypass model @@ -367,8 +370,39 @@ struct BuildProgramEnv { // platform rather than leaving the rule true only where the compiler // happens to help. bool importable = true; + // WHERE THE COMPILED MODULE MAY BE KEPT (#748, B1). An entry from an + // index package whose sources are in the immutable store goes to the + // global cache, where every project of the machine reuses it; everything + // else (a path or git dependency, a workspace member) is kept under the + // workspace's `target/`, because its sources can change without its name + // and version changing. The rule is the dependency cache's own + // (plan.cpp), and a host module is compiled alone, so the local taint it + // asks about is its own package's. + // + // The provider's identity is part of the entry's key either way: the + // index, name and version for an index package, and the manifest's own + // name and version otherwise. + std::string providerIndex; + std::string providerName; + std::string providerVersion; + bool immutableSource = false; + // The provider's package root. For a module whose sources can change in + // place, the tree below it is part of the entry's key: an edit to a header + // the interface includes is an edit to what was compiled. + std::filesystem::path providerRoot; }; std::vector hostModules; + // WHERE THE MODULES THIS PROGRAM IMPORTS ARE COMPILED TO (#748, B1). + // + // `moduleCacheRoot` is the global cache root, or empty when the build's cache + // mode (`--cache local` or `off`) asks for nothing to be written there. + // `moduleStore` is `/target/.build-mcpp/host-modules`, which the + // programs of one workspace share, so that a host module several members + // import is compiled once for all of them. Empty means the program's own + // artifacts directory, which is what a program that is not part of a + // workspace had before. + std::filesystem::path moduleCacheRoot; + std::filesystem::path moduleStore; }; // The env-var name `hostprogram::xpkg_dir` reads back. One spelling of the @@ -425,7 +459,43 @@ bool mentions_missing_mcpp_api(std::string_view compilerOutput); std::vector> install_hook_env(const BuildProgramEnv& env); -std::expected run_build_program( +// THE COMPILE OF A PROGRAM, MADE AHEAD OF ITS RUN (#748, B2). +// +// A workspace's programs run in an order: a member's program reads the directives +// of the programs of the members it depends on. Their compiles have none. This is +// what the compile leaves behind, so that `step9_member_build_programs` can +// compile every program that needs it at the same time, and then run the programs +// in the order they always ran in, each taking its compile from here. +// +// NOTHING IS REPORTED OR APPLIED BY THE COMPILE. A compile that failed leaves its +// message here, and the program's turn reports it, so the failure that reaches +// the user is the first in the order the programs run in, whichever compile +// finished first. The warnings and the refusal the compile produced are kept for +// the same reason: they are said, and recorded, by the thread that runs the +// programs, in that order. +struct PrecompiledProgram { + // A compile was attempted. False when the program's result was still valid + // when it was asked (nothing to compile), or when a check that comes before + // the compile refused it (which the program's turn refuses again, in order). + bool compiled = false; + // What the compile was made for: the program's text, the host compiler, the + // host modules' text. The program's turn uses the compile only when its own + // computation of this agrees, so a compile made for something that changed in + // the meantime is made again rather than trusted. + std::string stamp; + std::filesystem::path bin; + std::chrono::milliseconds compile{0}; + // The compile's own failure, in the words a serial build gives it. + std::string error; + std::optional refusal; + std::vector warnings; +}; + +// Everything `run_build_program` does before it runs the program, and stops there: +// the checks, the cache, and, for a program that needs it, the compile. Safe to +// call from several threads at once for different programs. A program that +// compiles shows as compiling; nothing else is shown. +PrecompiledProgram precompile_build_program( mcpp::manifest::Manifest& m, const std::filesystem::path& root, const std::filesystem::path& hostCompiler, @@ -433,6 +503,18 @@ std::expected run_build_program( const mcpp::manifest::CppStandardConfig& cppStandard, const BuildProgramEnv& env); +// `precompiled`, when given, is what `precompile_build_program` made for this +// program. The program is compiled here only when it is absent or made for +// something else. +std::expected run_build_program( + mcpp::manifest::Manifest& m, + const std::filesystem::path& root, + const std::filesystem::path& hostCompiler, + const mcpp::toolchain::Toolchain& tc, + const mcpp::manifest::CppStandardConfig& cppStandard, + const BuildProgramEnv& env, + const PrecompiledProgram* precompiled = nullptr); + // Has any recorded build-program input changed since its build.mcpp cache was // written: a glob's path SET (#359), a declared file's CONTENT, or a declared // environment variable's value? @@ -609,8 +691,11 @@ std::string sanitize_feature_env(std::string f) { } // The injected contract values, as (NAME, value) pairs for the child process. +// `warn` is false in the concurrent compile phase (#748, B2): the values are +// computed again at the program's turn, which states each warning once. std::vector> -contract_env(const fs::path& root, const fs::path& outDir, const BuildProgramEnv& env) { +contract_env(const fs::path& root, const fs::path& outDir, const BuildProgramEnv& env, + bool warn = true) { std::vector> e; auto hostT = mcpp::toolchain::triple::host_triple().str(); // The toolchain and target names an install hook also receives, from the @@ -729,7 +814,7 @@ contract_env(const fs::path& root, const fs::path& outDir, const BuildProgramEnv if (inserted) { e.emplace_back(var, dir.string()); } else if (it->second != dir.string()) { - mcpp::ui::warning(std::format( + if (warn) mcpp::ui::warning(std::format( "build.mcpp: dependency name collides on {} (kept '{}', ignored " "'{}') — rename one dependency to disambiguate", var, it->second, dir.string())); @@ -746,7 +831,7 @@ contract_env(const fs::path& root, const fs::path& outDir, const BuildProgramEnv if (inserted) { e.emplace_back(var, form); } else if (it->second != form) { - mcpp::ui::warning(std::format( + if (warn) mcpp::ui::warning(std::format( "build.mcpp: dependency name collides on {} (kept '{}', ignored " "'{}') — rename one dependency to disambiguate", var, it->second, form)); @@ -790,7 +875,7 @@ contract_env(const fs::path& root, const fs::path& outDir, const BuildProgramEnv if (inserted) { e.emplace_back(var, path); } else if (it->second != path) { - mcpp::ui::warning(std::format( + if (warn) mcpp::ui::warning(std::format( "build.mcpp: tool name collides on {} (kept '{}', ignored '{}')", var, it->second, path)); } @@ -996,13 +1081,22 @@ install_hook_env(const BuildProgramEnv& env) { }; } -std::expected run_build_program( +// WHICH PART OF A PROGRAM'S LIFE ONE CALL RUNS. `Whole` is the program as it has +// always been run: check, compile if the cache is stale, run, apply. `Compile` +// stops at the end of the compile and leaves its outcome in `*made`; it reports, +// records and applies nothing. `Run` is `Whole` with a compile taken from `*made`. +enum class ProgramMode { Whole, Compile, Run }; + +std::expected run_build_program_impl( + ProgramMode mode, mcpp::manifest::Manifest& m, const fs::path& root, const fs::path& hostCompiler, const mcpp::toolchain::Toolchain& tc, const mcpp::manifest::CppStandardConfig& cppStandard, - const BuildProgramEnv& env) { + const BuildProgramEnv& env, + const PrecompiledProgram* given, + PrecompiledProgram* made) { fs::path src = root / "build.mcpp"; std::error_code ec; @@ -1048,7 +1142,7 @@ std::expected run_build_program( fs::path bdir = build_dir(root, env); fs::path outDir = bdir / "out"; - auto childEnv = contract_env(root, outDir, env); + auto childEnv = contract_env(root, outDir, env, mode != ProgramMode::Compile); std::string ctxHash = contract_hash(childEnv); // THE GRAPH DOCUMENT'S CONTENT, NOT ITS PATH. The path is the same on // every run; what the document says is what the program's answer depends @@ -1253,6 +1347,9 @@ std::expected run_build_program( // directives, no run. CacheRecord cache = read_cache(bdir); if (cache_fresh(root, bdir, cache, programHash, compilerHash, ctxHash)) { + // Nothing to compile. Applying, advising and reporting the cached result + // belong to the program's turn, which does them in order. + if (mode == ProgramMode::Compile) return {}; if (auto terr = dirs::target_directive_error(m, cache.directives); !terr.empty()) return std::unexpected(terr); // #622 A4. Checked on the cache-hit path too, so a `deploy` a fresh @@ -1284,6 +1381,9 @@ std::expected run_build_program( const std::string& who; bool requested; ProgressClock::time_point compileStart{}, runStart{}; + // The compile was made elsewhere, by the thread that compiled it: its + // duration is stated, and is not read off this thread's clock. + std::optional compileDuration; bool ran = false; bool reported = false; ~ProgramReport() { report(); } @@ -1293,7 +1393,8 @@ std::expected run_build_program( using namespace std::chrono; const auto now = ProgressClock::now(); const auto zero = ProgressClock::time_point{}; - const auto compile = compileStart == zero ? milliseconds{0} + const auto compile = compileDuration ? *compileDuration + : compileStart == zero ? milliseconds{0} : duration_cast((runStart == zero ? now : runStart) - compileStart); const auto run = runStart == zero ? milliseconds{0} : duration_cast(now - runStart); @@ -1301,9 +1402,12 @@ std::expected run_build_program( who, requested, ran ? mcpp::build::progress::ProgramOutcome::Ran : mcpp::build::progress::ProgramOutcome::Failed, - compile, run); + compile, run, /*compiledAside=*/compileDuration.has_value()); } } programReport{who, env.requested}; + // The compile phase says nothing: what it made is reported, or refused, when + // the program's turn comes. + if (mode == ProgramMode::Compile) programReport.reported = true; fs::create_directories(outDir, ec); // creates bdir too // #230: on Windows the capture_exec shell is cmd.exe, which can only launch @@ -1315,38 +1419,9 @@ std::expected run_build_program( fs::path bin = bdir / (mcpp::platform::is_windows ? "build.mcpp.exe" : "build.mcpp.bin"); - // ── Compile build.mcpp with the host toolchain ────────────────────────── - // Spelled by the dialect layer, not concatenated here: the canonical of - // `standard = "c++fly"` / `"c++latest"` is not a valid -std= spelling, so - // the old `"-std=" + canonical` produced `-std=c++fly` and the host compile - // died on an unknown dialect. cppfly::std_flag resolves those against the - // toolchain that will actually run the compile — the host one, here. - std::string std_flag = mcpp::toolchain::cppfly::std_flag( - tc, cppStandard.canonical.empty() ? std::string_view("c++23") - : std::string_view(cppStandard.canonical), - cppStandard.level); - // One resolution of the deployment target, used by every compile below - // and by the std module it asks stdmod to build — they must agree or - // clang rejects the BMI. + // The driver's dialect: what the compile is spelled in, and how the program's + // output is read (`accept_output`). Used on both sides of the compile. // - // LEGITIMATELY HOST-KEYED, unlike the main build's readers (#685). `tc` - // here is always the HOST toolchain (see this function's own doc above - // `run_build_program` / `host_base_flags`'s header) -- build.mcpp is - // compiled AND run on the machine doing the build, so "the target" this - // one compile is for IS the host, and `tc.targetTriple` already names - // it. Asking `tc`'s own resolved target keeps this correct without - // reading a compile-time `__APPLE__`/`is_macos` constant, which would - // silently disagree with `tc` the day build.mcpp gains a host toolchain - // resolved for something other than the machine mcpp itself runs on. - const bool buildProgramTargetIsMacos = [&] { - auto bpTt = mcpp::toolchain::triple::parse(tc.targetTriple); - return bpTt && bpTt->os == "macos"; - }(); - const std::string macosDeploymentTarget = - mcpp::platform::macos::deployment_target( - buildProgramTargetIsMacos, m.buildConfig.macosDeploymentTarget); - auto base = host_base_flags(tc, macosDeploymentTarget); - // The host compile has always been spelled in GNU driver syntax with no // dialect branch at all — `grep -i msvc` over this file used to hit only // comments. Under cl.exe every one of `-O0` / `-x c++` / `-static` / `-o` @@ -1355,342 +1430,462 @@ std::expected run_build_program( const auto& dial = mcpp::toolchain::dialect_for(tc); const bool msvcHost = dial.id == std::string_view("msvc"); - // Only wire the bundled `mcpp` module when build.mcpp actually imports it — - // so the common `#include`-based program compiles exactly as before (no - // -fmodules, cwd = project root). When it does `import mcpp;`, compile the - // module, link its object, and run the build.mcpp compile from `bdir` so GCC - // finds gcm.cache/mcpp.gcm. - // `srcText` was read above the cache fast path, which the prerequisite - // check needs to run ahead of. - bool usesModule = srcText.find("import mcpp") != std::string::npos; - bool usesStdCompat = imports_module(srcText, "std.compat"); - bool usesStd = usesStdCompat || imports_module(srcText, "std"); - - // A rule package's interface is compiled by this same function, so what IT - // imports decides what has to be built just as much as what build.mcpp - // imports. Scanning only build.mcpp made a rule that said `import std;` - // fail with `module 'std' not found` — the std module was never built, - // because the program that triggers the build did not mention it. - for (auto const& hm : env.hostModules) { - std::ifstream is(hm.interface); - if (!is) continue; // a missing interface is diagnosed by build_host_module - std::ostringstream ss; ss << is.rdbuf(); - const std::string t = ss.str(); - if (t.find("import mcpp") != std::string::npos) usesModule = true; - if (imports_module(t, "std.compat")) usesStdCompat = true; - if (imports_module(t, "std")) usesStd = true; - } - - usesStd = usesStd || usesStdCompat; - - // The toolchain's own environment (MSVC's INCLUDE / LIB / VSLANG, which - // detection synthesized from the located VC tools + Windows SDK). Needed - // by every compile below, the module precompile included. - std::vector> compileEnv; - for (auto const& ev : tc.envOverrides) - compileEnv.emplace_back(ev.key, ev.value); - - // Named modules dispatch on the same BmiTraits/CommandDialect rows the - // main build uses, so there is no per-family gate here: cl.exe's - // .ifc + /reference works because the table already describes it, not - // because build.mcpp grew a second implementation of it. - std::vector moduleFlags; - fs::path mcppModuleObject; - fs::path mcppCoreObject; // `mcpp.core`, the same interface (#734, E8) - if (usesModule) { - auto mf = build_mcpp_module(bdir, hostCompiler, base, std_flag, tc, - compileEnv); - if (!mf) return std::unexpected(mf.error()); - moduleFlags = std::move(mf->useFlags); - mcppModuleObject = std::move(mf->object); - mcppCoreObject = std::move(mf->aliasObject); - } - - // ── `import std;` in build.mcpp ───────────────────────────────────────── - // - // mcpp asks projects to `import std;` everywhere and then made their build - // script fall back to `#include` — the bundled `mcpp` module even says so - // in its own header comment. The std module the main build already uses is - // reusable verbatim: stdmod::ensure_built caches on - // (toolchain × standard × dialect), so for a native build this is a cache - // HIT on the very artifact the project's own TUs import. Only a cross - // build pays for a second one, which is unavoidable — see below. - // - // `tc` here is the HOST toolchain: prepare.cppm's - // host_tc_for_build_program() resolves the spec WITHOUT the --target axis - // and hands it in. That is load-bearing. build.mcpp is compiled AND run on - // the machine doing the build, so a std BMI built for the target would - // produce a helper that cannot execute — the same host≠target mistake the - // mingw-cross work had to fix in four separate places. - std::vector stdFlags; - std::vector stdObjects; - // GCC finds staged BMIs by cwd; Clang/MSVC get an explicit path flag. - bool stdStagedInBdir = false; - if (usesStd) { - if (!tc.hasImportStd) { - return std::unexpected(std::format( - "build.mcpp uses `import std;` but the host toolchain ({}) " - "ships no std module.\n" - " Use #include in build.mcpp, or switch to a toolchain " - "that provides one.", tc.label())); + // THE COMPILE, AS ONE STEP. The preparation of what the program imports and + // the compile of the program itself are one piece of work with one duration, + // which is what `ran` reports; it used to report the compile of the program + // alone and leave the preparation (seven seconds a program, measured on + // #748's runner) to be counted in `plan`. It is a lambda so that it can be + // made ahead of the program's run, on another thread (`precompile_build_ + // program`), and used by the run without being made again. + ProgressClock::time_point compileBegan{}, compileEnded{}; + std::vector compileWarnings; + std::optional compileRefusal; + auto compile_binary = [&]() -> std::expected { + compileBegan = ProgressClock::now(); + mcpp::build::progress::program_compiling(who, env.requested); + mcpp::log::verbose("buildmcpp-host", std::format("build.mcpp {}: compile start", who)); + // ── Compile build.mcpp with the host toolchain ────────────────────────── + // Spelled by the dialect layer, not concatenated here: the canonical of + // `standard = "c++fly"` / `"c++latest"` is not a valid -std= spelling, so + // the old `"-std=" + canonical` produced `-std=c++fly` and the host compile + // died on an unknown dialect. cppfly::std_flag resolves those against the + // toolchain that will actually run the compile — the host one, here. + std::string std_flag = mcpp::toolchain::cppfly::std_flag( + tc, cppStandard.canonical.empty() ? std::string_view("c++23") + : std::string_view(cppStandard.canonical), + cppStandard.level); + // One resolution of the deployment target, used by every compile below + // and by the std module it asks stdmod to build — they must agree or + // clang rejects the BMI. + // + // LEGITIMATELY HOST-KEYED, unlike the main build's readers (#685). `tc` + // here is always the HOST toolchain (see this function's own doc above + // `run_build_program` / `host_base_flags`'s header) -- build.mcpp is + // compiled AND run on the machine doing the build, so "the target" this + // one compile is for IS the host, and `tc.targetTriple` already names + // it. Asking `tc`'s own resolved target keeps this correct without + // reading a compile-time `__APPLE__`/`is_macos` constant, which would + // silently disagree with `tc` the day build.mcpp gains a host toolchain + // resolved for something other than the machine mcpp itself runs on. + const bool buildProgramTargetIsMacos = [&] { + auto bpTt = mcpp::toolchain::triple::parse(tc.targetTriple); + return bpTt && bpTt->os == "macos"; + }(); + const std::string macosDeploymentTarget = + mcpp::platform::macos::deployment_target( + buildProgramTargetIsMacos, m.buildConfig.macosDeploymentTarget); + auto base = host_base_flags(tc, macosDeploymentTarget); + + // Only wire the bundled `mcpp` module when build.mcpp actually imports it — + // so the common `#include`-based program compiles exactly as before (no + // -fmodules, cwd = project root). When it does `import mcpp;`, compile the + // module, link its object, and run the build.mcpp compile from `bdir` so GCC + // finds gcm.cache/mcpp.gcm. + // `srcText` was read above the cache fast path, which the prerequisite + // check needs to run ahead of. + bool usesModule = srcText.find("import mcpp") != std::string::npos; + bool usesStdCompat = imports_module(srcText, "std.compat"); + bool usesStd = usesStdCompat || imports_module(srcText, "std"); + + // A rule package's interface is compiled by this same function, so what IT + // imports decides what has to be built just as much as what build.mcpp + // imports. Scanning only build.mcpp made a rule that said `import std;` + // fail with `module 'std' not found` — the std module was never built, + // because the program that triggers the build did not mention it. + for (auto const& hm : env.hostModules) { + std::ifstream is(hm.interface); + if (!is) continue; // a missing interface is diagnosed by provide_host_module + std::ostringstream ss; ss << is.rdbuf(); + const std::string t = ss.str(); + if (t.find("import mcpp") != std::string::npos) usesModule = true; + if (imports_module(t, "std.compat")) usesStdCompat = true; + if (imports_module(t, "std")) usesStd = true; } - auto sm = mcpp::toolchain::ensure_built( - tc, cppStandard.canonical, std_flag, macosDeploymentTarget); - if (!sm) { - // The second branch of the same refusal, and it is named for the - // same reason: both `run_build_program` call sites in prepare.cppm - // return `std::unexpected` unconditionally, so this error always - // reaches the layer that reads the code. An unnamed one here would - // reproduce, for the host std module, exactly the gap the target - // std module had. - refusal::record(refusal::Code::StdModulePrecompile); - return std::unexpected(std::format( - "build.mcpp uses `import std;` but the std module could not be " - "built for the host toolchain: {}", sm.error().message)); + + usesStd = usesStd || usesStdCompat; + + // The toolchain's own environment (MSVC's INCLUDE / LIB / VSLANG, which + // detection synthesized from the located VC tools + Windows SDK). Needed + // by every compile below, the module precompile included. + std::vector> compileEnv; + for (auto const& ev : tc.envOverrides) + compileEnv.emplace_back(ev.key, ev.value); + + // Named modules dispatch on the same BmiTraits/CommandDialect rows the + // main build uses, so there is no per-family gate here: cl.exe's + // .ifc + /reference works because the table already describes it, not + // because build.mcpp grew a second implementation of it. + std::vector moduleFlags; + fs::path mcppModuleObject; + fs::path mcppCoreObject; // `mcpp.core`, the same interface (#734, E8) + // The BMIs a compile can see, in the order they were made: what GCC finds by + // name under `gcm.cache`, and what the other two families are told by flag. + // A host module is compiled with every one before it. + std::vector visibleBmis; + // Where what this program imports is kept: the engine's module globally when + // the cache mode allows it, and the workspace's store for everything that is + // the project's own (#748, B1). + mcpp::build::ModuleStores stores; + stores.cacheRoot = env.moduleCacheRoot; + stores.workspaceStore = env.moduleStore.empty() ? bdir / "host-modules" : env.moduleStore; + if (usesModule) { + auto mf = mcpp::build::provide_mcpp_module(stores, bdir, hostCompiler, base, + std_flag, tc, compileEnv); + if (!mf) return std::unexpected(mf.error()); + moduleFlags = std::move(mf->useFlags); + mcppModuleObject = std::move(mf->object); + mcppCoreObject = std::move(mf->aliasObject); + for (auto& b : mf->bmis) visibleBmis.push_back(std::move(b)); } - auto traits = mcpp::toolchain::bmi_traits(tc); - if (traits.stdBmiUsePrefix.empty()) { - // GCC: BMIs are found implicitly under /gcm.cache, so stage - // the cached ones where the compile will look. Copy rather than - // symlink — this mirrors the main build's staging edge, and a - // stale copy is caught by ensure_built's own cache key. - std::error_code ec; - fs::path gcmDir = bdir / traits.bmiDir; - fs::create_directories(gcmDir, ec); - auto stage = [&](const fs::path& from, std::string_view name) - -> std::expected { - if (from.empty() || !fs::exists(from)) return {}; - fs::path to = gcmDir / std::format("{}{}", name, traits.bmiExt); - fs::copy_file(from, to, fs::copy_options::overwrite_existing, ec); - if (ec) return std::unexpected(std::format( - "staging {} for build.mcpp failed: {}", name, ec.message())); - return {}; - }; - if (auto r = stage(sm->bmiPath, "std"); !r) - return std::unexpected(r.error()); - if (usesStdCompat) { - if (auto r = stage(sm->compatBmiPath, "std.compat"); !r) - return std::unexpected(r.error()); + // ── `import std;` in build.mcpp ───────────────────────────────────────── + // + // mcpp asks projects to `import std;` everywhere and then made their build + // script fall back to `#include` — the bundled `mcpp` module even says so + // in its own header comment. The std module the main build already uses is + // reusable verbatim: stdmod::ensure_built caches on + // (toolchain × standard × dialect), so for a native build this is a cache + // HIT on the very artifact the project's own TUs import. Only a cross + // build pays for a second one, which is unavoidable — see below. + // + // `tc` here is the HOST toolchain: prepare.cppm's + // host_tc_for_build_program() resolves the spec WITHOUT the --target axis + // and hands it in. That is load-bearing. build.mcpp is compiled AND run on + // the machine doing the build, so a std BMI built for the target would + // produce a helper that cannot execute — the same host≠target mistake the + // mingw-cross work had to fix in four separate places. + std::vector stdFlags; + std::vector stdObjects; + // GCC finds staged BMIs by cwd; Clang/MSVC get an explicit path flag. + bool stdStagedInBdir = false; + if (usesStd) { + if (!tc.hasImportStd) { + return std::unexpected(std::format( + "build.mcpp uses `import std;` but the host toolchain ({}) " + "ships no std module.\n" + " Use #include in build.mcpp, or switch to a toolchain " + "that provides one.", tc.label())); } - // -fmodules may already be present from the `mcpp` module path; - // GCC tolerates the repeat, but keep the argv honest. - if (!usesModule) stdFlags.push_back("-fmodules"); - stdStagedInBdir = true; - } else { - // Through bmi_reference_tokens, not string concatenation: the - // traits spell these for the ninja STRING channel, where - // `-fmodule-file=std=

` (one word) and `/reference std=

` - // (two) are indistinguishable. Concatenating produced a single - // argv element with a space inside it, and cl answered - // "C2230: could not find module 'std'". - for (auto& t : mcpp::toolchain::bmi_reference_tokens( - traits.stdBmiUsePrefix, sm->bmiPath)) - stdFlags.push_back(t); - if (usesStdCompat && !sm->compatBmiPath.empty()) + // One thread builds the std module, and the others find it: the first + // build in a fresh home would otherwise have every program's compile + // write the same cache directory. + std::optional smValue; + std::optional smError; + { + static std::mutex stdModuleMutex; + std::lock_guard stdLock(stdModuleMutex); + auto built = mcpp::toolchain::ensure_built( + tc, cppStandard.canonical, std_flag, macosDeploymentTarget); + if (built) smValue = std::move(*built); + else smError = built.error().message; + } + if (!smValue) { + // The second branch of the same refusal, and it is named for the + // same reason: both `run_build_program` call sites in prepare.cppm + // return `std::unexpected` unconditionally, so this error always + // reaches the layer that reads the code. An unnamed one here would + // reproduce, for the host std module, exactly the gap the target + // std module had. + compileRefusal = refusal::Code::StdModulePrecompile; + return std::unexpected(std::format( + "build.mcpp uses `import std;` but the std module could not be " + "built for the host toolchain: {}", *smError)); + } + auto sm = std::move(smValue); + + auto traits = mcpp::toolchain::bmi_traits(tc); + if (traits.stdBmiUsePrefix.empty()) { + // GCC: BMIs are found implicitly under /gcm.cache, so stage + // the cached ones where the compile will look. Copy rather than + // symlink — this mirrors the main build's staging edge, and a + // stale copy is caught by ensure_built's own cache key. + std::error_code ec; + fs::path gcmDir = bdir / traits.bmiDir; + fs::create_directories(gcmDir, ec); + auto stage = [&](const fs::path& from, std::string_view name) + -> std::expected { + if (from.empty() || !fs::exists(from)) return {}; + fs::path to = gcmDir / std::format("{}{}", name, traits.bmiExt); + fs::copy_file(from, to, fs::copy_options::overwrite_existing, ec); + if (ec) return std::unexpected(std::format( + "staging {} for build.mcpp failed: {}", name, ec.message())); + return {}; + }; + if (auto r = stage(sm->bmiPath, "std"); !r) + return std::unexpected(r.error()); + if (usesStdCompat) { + if (auto r = stage(sm->compatBmiPath, "std.compat"); !r) + return std::unexpected(r.error()); + } + // -fmodules may already be present from the `mcpp` module path; + // GCC tolerates the repeat, but keep the argv honest. + if (!usesModule) stdFlags.push_back("-fmodules"); + stdStagedInBdir = true; + } else { + // Through bmi_reference_tokens, not string concatenation: the + // traits spell these for the ninja STRING channel, where + // `-fmodule-file=std=

` (one word) and `/reference std=

` + // (two) are indistinguishable. Concatenating produced a single + // argv element with a space inside it, and cl answered + // "C2230: could not find module 'std'". for (auto& t : mcpp::toolchain::bmi_reference_tokens( - traits.stdCompatBmiUsePrefix, sm->compatBmiPath)) + traits.stdBmiUsePrefix, sm->bmiPath)) stdFlags.push_back(t); + if (usesStdCompat && !sm->compatBmiPath.empty()) + for (auto& t : mcpp::toolchain::bmi_reference_tokens( + traits.stdCompatBmiUsePrefix, sm->compatBmiPath)) + stdFlags.push_back(t); + } + if (!sm->objectPath.empty() && fs::exists(sm->objectPath)) + stdObjects.push_back(sm->objectPath.string()); + if (usesStdCompat && !sm->compatObjectPath.empty() + && fs::exists(sm->compatObjectPath)) + stdObjects.push_back(sm->compatObjectPath.string()); + // The std cache's directory is named by its own identity hash, which is + // what a dependent entry's key records for the BMIs inside it. + { + const auto id8 = sm->cacheDir.filename().generic_u8string(); + const std::string stdIdentity(reinterpret_cast(id8.data()), id8.size()); + visibleBmis.push_back({"std", sm->bmiPath, stdIdentity}); + if (usesStdCompat && !sm->compatBmiPath.empty()) + visibleBmis.push_back({"std.compat", sm->compatBmiPath, stdIdentity}); + } } - if (!sm->objectPath.empty() && fs::exists(sm->objectPath)) - stdObjects.push_back(sm->objectPath.string()); - if (usesStdCompat && !sm->compatObjectPath.empty() - && fs::exists(sm->compatObjectPath)) - stdObjects.push_back(sm->compatObjectPath.string()); - } - // #355 step 5: dependency-provided host modules (reusable build rules - // shipped as ordinary packages). Compiled HERE, with `base` and `std_flag` - // — the same flags the build.mcpp compile below gets — because a BMI is - // only usable by a compile that agrees with it. Doing this in a separate - // sub-build would leave that agreement to chance, and disagreement shows - // up as `module X CRC mismatch`, not as a clear error. - // - // AFTER the std block, and that ordering is load-bearing: a rule may - // `import std;` just as build.mcpp may, and it can only do so once the std - // BMI exists and `stdFlags` names it. Compiling rules first — which is what - // 2026.8.5.1 did — handed them an empty `stdFlags` and failed with - // `module 'std' not found`. - std::vector hostModuleObjects; - for (auto const& ref : env.hostModules) { - std::vector use = moduleFlags; - use.insert(use.end(), stdFlags.begin(), stdFlags.end()); - auto hm = build_host_module(bdir, hostCompiler, base, std_flag, tc, - compileEnv, ref.logical, ref.interface, use); - if (!hm) return std::unexpected(hm.error()); - // APPENDED VERBATIM, and nothing is de-duplicated. - // - // This filtered per TOKEN, and it was written for the one family whose - // marker is a single idempotent word: GCC's `-fmodules`, already - // present because the bundled `mcpp` module put it there. The other - // two families do not have that shape. - // - // GCC `-fmodules` 1 token, idempotent - // Clang `-fmodule-file==` 1 token, unique - // MSVC `/reference`, `=` 2 tokens, FIRST REPEATS - // - // So on `windows = "msvc@system"` the pair arrived, `/reference` was - // found already in the list, and only the pair's second half was - // appended. cl.exe received `=.ifc` with no switch in - // front of it and read it as a source file name: - // - // c1xx: fatal error C1083: Cannot open source file: - // 'huxerui.rules.sources=...\huxerui.rules.sources.ifc' - // - // Clang was immune by construction -- one word, never equal to an - // existing element -- which is why the defect was specific to the one - // toolchain selection that reaches `import std;` at c++20 on Windows. + // #355 step 5: dependency-provided host modules (reusable build rules + // shipped as ordinary packages). Compiled HERE, with `base` and `std_flag` + // — the same flags the build.mcpp compile below gets — because a BMI is + // only usable by a compile that agrees with it. Doing this in a separate + // sub-build would leave that agreement to chance, and disagreement shows + // up as `module X CRC mismatch`, not as a clear error. // - // The comment this replaces stated the whole value of the filter: - // "repeating it is harmless but noisy". It bought a tidier argv and - // paid with a broken command line. De-duplicating by logical module - // name, or by contiguous subsequence, would both be correct -- and - // would both be a new rule kept for the same cosmetic reason. The rule - // is gone instead. - for (auto& f : hm->useFlags) - moduleFlags.push_back(f); - hostModuleObjects.push_back(std::move(hm->object)); - } + // AFTER the std block, and that ordering is load-bearing: a rule may + // `import std;` just as build.mcpp may, and it can only do so once the std + // BMI exists and `stdFlags` names it. Compiling rules first — which is what + // 2026.8.5.1 did — handed them an empty `stdFlags` and failed with + // `module 'std' not found`. + std::vector hostModuleObjects; + for (auto const& ref : env.hostModules) { + std::vector use = moduleFlags; + use.insert(use.end(), stdFlags.begin(), stdFlags.end()); + mcpp::build::HostModuleSource source; + source.logical = ref.logical; + source.interface = ref.interface; + source.index = ref.providerIndex; + source.package = ref.providerName; + source.version = ref.providerVersion; + source.immutableSource = ref.immutableSource; + source.sourceRoot = ref.providerRoot; + auto hm = mcpp::build::provide_host_module(stores, source, bdir, hostCompiler, + base, std_flag, tc, compileEnv, use, + visibleBmis); + if (!hm) return std::unexpected(hm.error()); + for (auto& b : hm->bmis) visibleBmis.push_back(b); + // APPENDED VERBATIM, and nothing is de-duplicated. + // + // This filtered per TOKEN, and it was written for the one family whose + // marker is a single idempotent word: GCC's `-fmodules`, already + // present because the bundled `mcpp` module put it there. The other + // two families do not have that shape. + // + // GCC `-fmodules` 1 token, idempotent + // Clang `-fmodule-file==` 1 token, unique + // MSVC `/reference`, `=` 2 tokens, FIRST REPEATS + // + // So on `windows = "msvc@system"` the pair arrived, `/reference` was + // found already in the list, and only the pair's second half was + // appended. cl.exe received `=.ifc` with no switch in + // front of it and read it as a source file name: + // + // c1xx: fatal error C1083: Cannot open source file: + // 'huxerui.rules.sources=...\huxerui.rules.sources.ifc' + // + // Clang was immune by construction -- one word, never equal to an + // existing element -- which is why the defect was specific to the one + // toolchain selection that reaches `import std;` at c++20 on Windows. + // + // The comment this replaces stated the whole value of the filter: + // "repeating it is harmless but noisy". It bought a tidier argv and + // paid with a broken command line. De-duplicating by logical module + // name, or by contiguous subsequence, would both be correct -- and + // would both be a new rule kept for the same cosmetic reason. The rule + // is gone instead. + for (auto& f : hm->useFlags) + moduleFlags.push_back(f); + hostModuleObjects.push_back(std::move(hm->object)); + } - // `-x c++` is required: the `.mcpp` extension is unknown to the compiler, so - // without it the driver hands build.mcpp to the linker as a linker script. - std::vector compileArgv = { hostCompiler.string() }; - if (msvcHost) { - // /nologo /EHsc /utf-8 — cl.exe needs these to behave like the other - // two drivers do by default (quiet, exceptions on, UTF-8 sources). - for (auto f : dial.alwaysFlagsArgv) compileArgv.emplace_back(f); - } - compileArgv.push_back(std_flag); - // No optimization: this program runs once per build and its compile time - // is on the critical path. MSVC spells "off" /Od, not /O0. - compileArgv.push_back(msvcHost ? std::string("/Od") - : std::string(dial.optPrefix) + "0"); - for (auto& bf : base) compileArgv.push_back(bf); - for (auto& mf : moduleFlags) compileArgv.push_back(mf); - for (auto& sf : stdFlags) compileArgv.push_back(sf); - // The `.mcpp` extension is unknown to every driver, so without this the - // file is handed to the linker as a linker script. - // Per-file where the driver has that form (cl's /Tp), positional - // otherwise. Object files follow on this same command line, and cl's - // global /TP would compile them as C++ source. - if (!dial.perFileCxxPrefix.empty()) { - compileArgv.push_back(std::string(dial.perFileCxxPrefix) + src.string()); - } else { - for (auto f : dial.forceCxxLangArgv) compileArgv.emplace_back(f); - compileArgv.push_back(src.string()); - } - if (usesModule || !stdObjects.empty() || !hostModuleObjects.empty()) { - // Link the module objects. GNU drivers need the input language reset - // first, or the .o that follows `-x c++` is handed to the frontend as - // C++ source; cl.exe has no `-x` at all and infers from the extension. - // This used to be unconditional and was only harmless while MSVC could - // not reach it — removing that gate made the dead branch live, and cl - // answered with `D9002: ignoring unknown option '-x'`. - if (!msvcHost) { compileArgv.push_back("-x"); compileArgv.push_back("none"); } - if (usesModule) compileArgv.push_back(mcppModuleObject.string()); - if (usesModule && !mcppCoreObject.empty()) compileArgv.push_back(mcppCoreObject.string()); - for (auto& hmo : hostModuleObjects) compileArgv.push_back(hmo.string()); - for (auto& so : stdObjects) compileArgv.push_back(so); - } - // THE BUILD PROGRAM SPEAKS THE ENCODING mcpp SPEAKS (#693, D4). - // - // mcpp hands a build program its paths through the environment and reads - // its directives from stdout, and on Windows mcpp runs with a UTF-8 process - // code page. A build program without the same application manifest reads - // the environment through the machine's ANSI code page instead. Measured on - // a cp1252 runner: a narrow build.mcpp in `C:\w\caf` printed its - // path in cp1252 bytes that mcpp could not decode, and in a directory - // outside the code page it received `??` in place of the name. So the - // program carries the manifest; where no resource compiler stands beside - // the compiler, it is built without one and the build says so. - if constexpr (mcpp::platform::is_windows) { - auto manifestRes = mcpp::build::resources::compile_utf8_manifest( - tc, dial.id, bdir, "build.mcpp.utf8"); - if (manifestRes) { + // `-x c++` is required: the `.mcpp` extension is unknown to the compiler, so + // without it the driver hands build.mcpp to the linker as a linker script. + std::vector compileArgv = { hostCompiler.string() }; + if (msvcHost) { + // /nologo /EHsc /utf-8 — cl.exe needs these to behave like the other + // two drivers do by default (quiet, exceptions on, UTF-8 sources). + for (auto f : dial.alwaysFlagsArgv) compileArgv.emplace_back(f); + } + compileArgv.push_back(std_flag); + // No optimization: this program runs once per build and its compile time + // is on the critical path. MSVC spells "off" /Od, not /O0. + compileArgv.push_back(msvcHost ? std::string("/Od") + : std::string(dial.optPrefix) + "0"); + for (auto& bf : base) compileArgv.push_back(bf); + for (auto& mf : moduleFlags) compileArgv.push_back(mf); + for (auto& sf : stdFlags) compileArgv.push_back(sf); + // The `.mcpp` extension is unknown to every driver, so without this the + // file is handed to the linker as a linker script. + // Per-file where the driver has that form (cl's /Tp), positional + // otherwise. Object files follow on this same command line, and cl's + // global /TP would compile them as C++ source. + if (!dial.perFileCxxPrefix.empty()) { + compileArgv.push_back(std::string(dial.perFileCxxPrefix) + src.string()); + } else { + for (auto f : dial.forceCxxLangArgv) compileArgv.emplace_back(f); + compileArgv.push_back(src.string()); + } + if (usesModule || !stdObjects.empty() || !hostModuleObjects.empty()) { + // Link the module objects. GNU drivers need the input language reset + // first, or the .o that follows `-x c++` is handed to the frontend as + // C++ source; cl.exe has no `-x` at all and infers from the extension. + // This used to be unconditional and was only harmless while MSVC could + // not reach it — removing that gate made the dead branch live, and cl + // answered with `D9002: ignoring unknown option '-x'`. if (!msvcHost) { compileArgv.push_back("-x"); compileArgv.push_back("none"); } - compileArgv.push_back(manifestRes->string()); + if (usesModule) compileArgv.push_back(mcppModuleObject.string()); + if (usesModule && !mcppCoreObject.empty()) compileArgv.push_back(mcppCoreObject.string()); + for (auto& hmo : hostModuleObjects) compileArgv.push_back(hmo.string()); + for (auto& so : stdObjects) compileArgv.push_back(so); + } + // THE BUILD PROGRAM SPEAKS THE ENCODING mcpp SPEAKS (#693, D4). + // + // mcpp hands a build program its paths through the environment and reads + // its directives from stdout, and on Windows mcpp runs with a UTF-8 process + // code page. A build program without the same application manifest reads + // the environment through the machine's ANSI code page instead. Measured on + // a cp1252 runner: a narrow build.mcpp in `C:\w\caf` printed its + // path in cp1252 bytes that mcpp could not decode, and in a directory + // outside the code page it received `??` in place of the name. So the + // program carries the manifest; where no resource compiler stands beside + // the compiler, it is built without one and the build says so. + if constexpr (mcpp::platform::is_windows) { + auto manifestRes = mcpp::build::resources::compile_utf8_manifest( + tc, dial.id, bdir, "build.mcpp.utf8"); + if (manifestRes) { + if (!msvcHost) { compileArgv.push_back("-x"); compileArgv.push_back("none"); } + compileArgv.push_back(manifestRes->string()); + } else { + compileWarnings.push_back(std::format( + "build.mcpp: built without the UTF-8 code page ({}); a path outside " + "this machine's ANSI code page will not reach it intact", + manifestRes.error())); + } + } + // Self-contained helper link — see the staticHostHelper doctrine above. + // Deliberately NOT in `base`: that also feeds the bundled module's + // compile/precompile commands, where a link flag has no business (and for + // Clang would perturb the default PIC/PIE codegen of mcpp.o). + if (staticHostHelper) compileArgv.push_back(std::string(dial.staticRuntime)); + // A DYNAMIC HELPER ON LINUX GETS `DT_RPATH`, NOT `DT_RUNPATH`. + // + // The driver's default is the new tag, and a runpath is consulted only for + // the helper's OWN needed libraries. A build program that opens a host + // library at run time -- a rule package reading a driver's version through + // the driver itself -- then fails one hop later, because that library's + // own dependencies (`libdl.so.2`, `libpthread.so.0`) are looked up without + // the helper's search path and the payload loader has no default that + // reaches them. Measured: `dlopen("/lib/libcuda.so.1")` from a + // build.mcpp answered `libdl.so.2: cannot open shared object file` while + // the very same directories sat in the helper's RUNPATH. The artifacts + // mcpp links carry DT_RPATH for this reason (loader_contract's Rpath tag); + // the helper now does too. Driver-only spelling: the helper is always + // linked through the compiler driver, never through the linker directly. + if (!staticHostHelper && !msvcHost + && !mcpp::platform::is_windows && !mcpp::platform::is_macos) + compileArgv.push_back("-Wl,--disable-new-dtags"); + if (msvcHost) { + // /Fe: takes its value attached, not as a separate argv token. + compileArgv.push_back(std::string(dial.outputExePrefix) + bin.string()); } else { - mcpp::ui::warning(std::format( - "build.mcpp: built without the UTF-8 code page ({}); a path outside " - "this machine's ANSI code page will not reach it intact", - manifestRes.error())); + compileArgv.push_back("-o"); compileArgv.push_back(bin.string()); + } + // A `=` with no switch in front of it, checked before the + // command runs rather than diagnosed from cl.exe's answer to it. cl reports + // such a token as `C1083: Cannot open source file`, which names the module + // and the BMI and never names the missing flag -- so it reads as a broken + // build tree rather than as a broken command line. See + // mcpp::toolchain::orphaned_reference; this is the reader that makes the + // rule enforced rather than merely stated. + if (auto orphan = mcpp::toolchain::orphaned_reference(compileArgv)) { + return std::unexpected(std::format( + "build.mcpp: the module reference '{}' reached the compiler with no " + "switch in front of it.\n" + " This is an mcpp defect, not a problem with the project: the " + "reference is\n" + " assembled as a pair (`/reference =` on MSVC) and " + "only one half\n" + " arrived. Please report it with the toolchain name and this " + "line.", *orphan)); + } + // GCC resolves imported BMIs via gcm.cache/ relative to the compile cwd, so + // any compile that imports a module — `mcpp`, `std`, a build rule's host + // module, or any mix — has to run from bdir, where they were staged or + // compiled. One condition: a build.mcpp that imports only a rule needs + // exactly the same cwd as one that imports only mcpp (a rule-only program + // compiled in the project root and failed with "failed to read compiled + // module", e2e 807 under GCC). Otherwise the project root is fine. + const bool needsBmiCwd = usesModule || stdStagedInBdir || !env.hostModules.empty(); + std::string compileCwd = needsBmiCwd ? bdir.string() : root.string(); + auto cres = mcpp::platform::process::capture_exec(compileArgv, compileEnv, + compileCwd); + mcpp::log::verbose("buildmcpp-host", std::format("build.mcpp {}: compile end", who)); + if (cres.exit_code != 0) { + std::string msg = std::format("build.mcpp failed to compile (exit {}):\n{}", + cres.exit_code, cres.output); + if (mentions_missing_mcpp_api(cres.output)) { + msg += std::format( + "\n The `mcpp` build module this engine bundles does not have " + "that name.\n" + " Either the package was written for a newer mcpp (try " + "`mcpp self update`;\n" + " this is mcpp {}), or the name is misspelled — the compiler " + "cannot tell\n" + " the two apart, because the module is generated by whichever " + "mcpp is running.", + mcpp::MCPP_VERSION); + } + return std::unexpected(std::move(msg)); + } + return {}; + }; + + // The compile is taken from the phase that made it ahead of this run, or made + // here. It is taken only when it was made for what this run computes: the + // program's text, the host compiler and the text of the host modules. + const std::string stamp = programHash + '|' + compilerHash; + if (mode == ProgramMode::Run && given && given->compiled && given->stamp == stamp) { + for (auto const& w : given->warnings) mcpp::ui::warning(w); + programReport.compileDuration = given->compile; + if (!given->error.empty()) { + if (given->refusal) mcpp::build::refusal::record(*given->refusal); + return std::unexpected(given->error); } - } - // Self-contained helper link — see the staticHostHelper doctrine above. - // Deliberately NOT in `base`: that also feeds the bundled module's - // compile/precompile commands, where a link flag has no business (and for - // Clang would perturb the default PIC/PIE codegen of mcpp.o). - if (staticHostHelper) compileArgv.push_back(std::string(dial.staticRuntime)); - // A DYNAMIC HELPER ON LINUX GETS `DT_RPATH`, NOT `DT_RUNPATH`. - // - // The driver's default is the new tag, and a runpath is consulted only for - // the helper's OWN needed libraries. A build program that opens a host - // library at run time -- a rule package reading a driver's version through - // the driver itself -- then fails one hop later, because that library's - // own dependencies (`libdl.so.2`, `libpthread.so.0`) are looked up without - // the helper's search path and the payload loader has no default that - // reaches them. Measured: `dlopen("/lib/libcuda.so.1")` from a - // build.mcpp answered `libdl.so.2: cannot open shared object file` while - // the very same directories sat in the helper's RUNPATH. The artifacts - // mcpp links carry DT_RPATH for this reason (loader_contract's Rpath tag); - // the helper now does too. Driver-only spelling: the helper is always - // linked through the compiler driver, never through the linker directly. - if (!staticHostHelper && !msvcHost - && !mcpp::platform::is_windows && !mcpp::platform::is_macos) - compileArgv.push_back("-Wl,--disable-new-dtags"); - if (msvcHost) { - // /Fe: takes its value attached, not as a separate argv token. - compileArgv.push_back(std::string(dial.outputExePrefix) + bin.string()); } else { - compileArgv.push_back("-o"); compileArgv.push_back(bin.string()); - } - // A `=` with no switch in front of it, checked before the - // command runs rather than diagnosed from cl.exe's answer to it. cl reports - // such a token as `C1083: Cannot open source file`, which names the module - // and the BMI and never names the missing flag -- so it reads as a broken - // build tree rather than as a broken command line. See - // mcpp::toolchain::orphaned_reference; this is the reader that makes the - // rule enforced rather than merely stated. - if (auto orphan = mcpp::toolchain::orphaned_reference(compileArgv)) { - return std::unexpected(std::format( - "build.mcpp: the module reference '{}' reached the compiler with no " - "switch in front of it.\n" - " This is an mcpp defect, not a problem with the project: the " - "reference is\n" - " assembled as a pair (`/reference =` on MSVC) and " - "only one half\n" - " arrived. Please report it with the toolchain name and this " - "line.", *orphan)); - } - mcpp::build::progress::program_compiling(who, env.requested); - programReport.compileStart = ProgressClock::now(); - // GCC resolves imported BMIs via gcm.cache/ relative to the compile cwd, so - // any compile that imports a module — `mcpp`, `std`, a build rule's host - // module, or any mix — has to run from bdir, where they were staged or - // compiled. One condition: a build.mcpp that imports only a rule needs - // exactly the same cwd as one that imports only mcpp (a rule-only program - // compiled in the project root and failed with "failed to read compiled - // module", e2e 807 under GCC). Otherwise the project root is fine. - const bool needsBmiCwd = usesModule || stdStagedInBdir || !env.hostModules.empty(); - std::string compileCwd = needsBmiCwd ? bdir.string() : root.string(); - auto cres = mcpp::platform::process::capture_exec(compileArgv, compileEnv, - compileCwd); - if (cres.exit_code != 0) { - std::string msg = std::format("build.mcpp failed to compile (exit {}):\n{}", - cres.exit_code, cres.output); - if (mentions_missing_mcpp_api(cres.output)) { - msg += std::format( - "\n The `mcpp` build module this engine bundles does not have " - "that name.\n" - " Either the package was written for a newer mcpp (try " - "`mcpp self update`;\n" - " this is mcpp {}), or the name is misspelled — the compiler " - "cannot tell\n" - " the two apart, because the module is generated by whichever " - "mcpp is running.", - mcpp::MCPP_VERSION); + auto compiled = compile_binary(); + compileEnded = ProgressClock::now(); + if (mode == ProgramMode::Compile) { + made->compiled = true; + made->stamp = stamp; + made->bin = bin; + made->compile = std::chrono::duration_cast( + compileEnded - compileBegan); + made->warnings = std::move(compileWarnings); + made->refusal = compileRefusal; + if (!compiled) made->error = compiled.error(); + return {}; } - return std::unexpected(std::move(msg)); + for (auto const& w : compileWarnings) mcpp::ui::warning(w); + if (compileRefusal) mcpp::build::refusal::record(*compileRefusal); + if (!compiled) return std::unexpected(compiled.error()); + programReport.compileStart = compileBegan; } // ── Run it; capture stdout(+stderr) and parse directives ──────────────── @@ -1813,6 +2008,34 @@ std::expected run_build_program( return {}; } +PrecompiledProgram precompile_build_program( + mcpp::manifest::Manifest& m, + const fs::path& root, + const fs::path& hostCompiler, + const mcpp::toolchain::Toolchain& tc, + const mcpp::manifest::CppStandardConfig& cppStandard, + const BuildProgramEnv& env) { + PrecompiledProgram made; + // A refusal that comes before the compile is not carried: the program's turn + // runs the same checks and refuses in its own order. + (void)run_build_program_impl(ProgramMode::Compile, m, root, hostCompiler, tc, + cppStandard, env, nullptr, &made); + return made; +} + +std::expected run_build_program( + mcpp::manifest::Manifest& m, + const fs::path& root, + const fs::path& hostCompiler, + const mcpp::toolchain::Toolchain& tc, + const mcpp::manifest::CppStandardConfig& cppStandard, + const BuildProgramEnv& env, + const PrecompiledProgram* precompiled) { + return run_build_program_impl(precompiled ? ProgramMode::Run : ProgramMode::Whole, + m, root, hostCompiler, tc, cppStandard, env, + precompiled, nullptr); +} + std::vector declared_program_inputs(const fs::path& workRoot) { std::vector out; diff --git a/src/build/execute.cppm b/src/build/execute.cppm index be6b92abf..808003efa 100644 --- a/src/build/execute.cppm +++ b/src/build/execute.cppm @@ -90,6 +90,23 @@ int launcher_status(int spawnErrno) { } } +// THE STATUS OF A `run` WHOSE PLANNING OR BUILD FAILED (output streams plan +// 2026-10-01, D7, R2). +// +// It used to be the build's own status, 1 or 2, and a program that returns 1 +// reads the same: a script that runs `mcpp run -q` could not tell a compile +// error from a program that failed. 101 is the status Cargo gives a failed +// `cargo run` build, and an established convention. It is also rare among the +// statuses programs return themselves, which is what makes it tell the two +// apart; it does not collide with the launcher's band above. Only `run` uses +// it: `build`, `test` and `pack` keep their statuses, and so does a program's +// own, which passes through unchanged. +// +// With a runner (`--runner`, `[target.].runner`) the status that passes +// through is the runner's, and a runner that itself returns 101 reads as a +// failed build. That is accepted, as in Cargo, and documented. +constexpr int kRunBuildFailed = 101; + // ─── P0: build cache for fast-path rebuilds ───────────────────────── constexpr std::string_view kBuildCacheFile = "target/.build_cache"; @@ -669,9 +686,13 @@ compute_subos_env(const mcpp::build::BuildPlan& plan) { // tell an engine that states "nothing" from one that predates the variable. constexpr std::string_view kRuntimeFilesEnv = "MCPP_RUNTIME_FILES"; +// `owner`, for a plan of several workspace members, is the member whose +// artifact this is: the shared libraries another member's targets link are not +// this artifact's files. std::vector> runtime_files_for(const mcpp::build::BuildContext& ctx, - const std::filesystem::path& artifact) { + const std::filesystem::path& artifact, + std::string_view owner = {}) { std::vector> out; const auto artifactDir = artifact.parent_path().lexically_normal(); const auto artifactNorm = artifact.lexically_normal(); @@ -689,6 +710,7 @@ runtime_files_for(const mcpp::build::BuildContext& ctx, for (auto const& d : mcpp::build::compute_flags(ctx.plan).runtimeDeploy) add(d.dest); for (auto const& lu : ctx.plan.linkUnits) { if (lu.kind != mcpp::build::LinkUnit::SharedLibrary) continue; + if (!owner.empty() && !lu.memberOf.empty() && lu.memberOf != owner) continue; add(lu.output); for (auto const& alias : lu.runtimeAliases) add(alias); } @@ -1374,8 +1396,10 @@ std::optional run_ninja_fast(const std::string& ninjaProgram, } return 1; } + // Verbose ninja output is narration, so it goes to the narration stream, + // whether or not `--quiet` is also given (verbose output survives it). if (verbose && !reporting && !out.empty()) - std::fputs(out.c_str(), stdout); + mcpp::ui::block(out); // What the edges that ran had to say on success: the same reader the full // path calls (mcpp.build.advice), because this path skips `prepare` and a // report attached to one path only appears or not depending on whether @@ -1826,6 +1850,20 @@ export std::optional try_fast_workspace_build( return 0; } +// THE BLANK LINE AFTER THE `Running` LINE BELONGS TO THAT LINE. It separates +// mcpp's narration from the program's output on a terminal, so it is narration +// as well: written to the stream the `Running` line was written to (standard +// error) and, like it, not under `--quiet`. It used to be a bare `println` to +// standard output, so `mcpp run -q` wrote one empty line in front of the +// program's own output, and `mcpp run -q > file` began with it (measured with +// 2026.9.30.2, `od -c`). Both streams are flushed after it, so that nothing +// mcpp wrote is still buffered when the program starts writing to the same +// terminal. +void run_separator() { + mcpp::ui::line(""); + mcpp::ui::flush(); +} + // mcpp#225 (E2): `mcpp run`'s fast path. Mirrors try_fast_build's // fingerprint/freshness gate against the SAME cache entry `mcpp build` // wrote (targetTriple == "" — a HOST build; see the precondition below), then @@ -1972,7 +2010,7 @@ std::optional try_fast_run(const std::filesystem::path& projectRoot, auto rc = run_ninja_fast(ninjaProgram, outputDir, ninjaPath, /*verbose=*/false, match->runtimeEnvKey, match->runtimeEnvValue); if (!rc) return fast_path_declined("run", "ninja reported a stale graph"); - if (*rc != 0) return rc; + if (*rc != 0) return kRunBuildFailed; if (!mcpp::build::runtime_validation::artifact_snapshot_unchanged( *validatedBefore)) return fast_path_declined("run", "ninja relinked an artifact, whose closure the full path validates"); // never execute an artifact not validated for this binding @@ -1982,8 +2020,7 @@ std::optional try_fast_run(const std::filesystem::path& projectRoot, mcpp::build::progress::close(); // the program owns the terminal mcpp::ui::status("Running", std::format("`{}`", mcpp::ui::shorten_path(exe, pathCtx))); - std::println(""); - std::fflush(stdout); + run_separator(); std::vector argv; argv.push_back(exe.string()); for (auto& a : passthrough) argv.push_back(a); @@ -2180,8 +2217,7 @@ int run_artifact_via_runner(mcpp::build::BuildContext& ctx, mcpp::ui::status("Running", std::format("`{}`", mcpp::ui::shorten_path(exe, pathCtx))); } - std::println(""); - std::fflush(stdout); + run_separator(); std::vector> childEnv; auto [runEnvKey, runEnvValue] = compute_run_env(ctx.plan); @@ -2336,7 +2372,7 @@ export int build_run_target(const std::optional& targetName, popts.features = features; auto outcome = mcpp::pack::build_and_pack( std::move(popts), /*modeFromUser=*/false, targetName.value_or(std::string{})); - if (outcome.rc != 0) return outcome.rc; + if (outcome.rc != 0) return kRunBuildFailed; if (outcome.artifacts.empty()) { // Not reached today: `build_and_pack` returns rc=0 only after // confirming at least one reported artifact exists on disk. Kept @@ -2363,10 +2399,9 @@ export int build_run_target(const std::optional& targetName, ov2.will_run = true; auto ctx2 = prepare_build(/*print_fp=*/false, /*includeDevDeps=*/false, /*extraTargets=*/{}, ov2); - if (!ctx2) { mcpp::ui::error(std::format("{}", ctx2.error())); return 2; } - if (auto rc = run_build_plan(*ctx2, /*verbose=*/false, no_cache, target_triple); - rc != 0) - return rc; + if (!ctx2) { mcpp::ui::error(std::format("{}", ctx2.error())); return kRunBuildFailed; } + if (run_build_plan(*ctx2, /*verbose=*/false, no_cache, target_triple) != 0) + return kRunBuildFailed; // ONE DISTRIBUTABLE, OR A SENTENCE. The pack pipeline reports the // terminal artifacts of the request (outputs no other introduced // action consumes); a format that ends in two files has no single @@ -2418,7 +2453,7 @@ export int build_run_target(const std::optional& targetName, ov.will_run = true; auto ctx = prepare_build(/*print_fp=*/false, /*includeDevDeps=*/false, /*extraTargets=*/{}, ov); - if (!ctx) { mcpp::ui::error(std::format("{}", ctx.error())); return 2; } + if (!ctx) { mcpp::ui::error(std::format("{}", ctx.error())); return kRunBuildFailed; } // `target_triple` IS PASSED, AND OMITTING IT WROTE A CROSS BUILD INTO // THE HOST'S CACHE SLOT. // @@ -2437,9 +2472,8 @@ export int build_run_target(const std::optional& targetName, // The comment on `try_fast_run` records the same defect reached through the // MANIFEST's default target, and guards that door alone. This is the other // door: the flag. Measured on 2026.8.24.3, from a clean `target/`. - if (auto rc = run_build_plan(*ctx, /*verbose=*/false, no_cache, target_triple); - rc != 0) - return rc; + if (run_build_plan(*ctx, /*verbose=*/false, no_cache, target_triple) != 0) + return kRunBuildFailed; // The program run is the selected member's (workspace design 2026-09-29 // §15), with its closure's runtime. focus_on_member(*ctx); @@ -2552,10 +2586,42 @@ export struct TestRunSummary { // `notRun` so the workspace total cannot add a stated build-only result to // a run that mcpp could not perform. int built = 0; - long long buildMs = 0; // Phase A + bulk pass + per-test drives + // The wall time this member's tests waited for their build. Phase A + bulk + // pass + per-test drives for a member planned alone; for a member planned + // with others, the build of its whole group (`buildGroup`), plus its own + // per-test drives. + long long buildMs = 0; long long runMs = 0; // the test binaries' own execution long long elapsedMs = 0; // wall clock for the whole member bool packageError = false; // Phase A failed: no test ever ran + // The configuration group whose one build this member's tests waited for, + // when `mcpp test` planned several members together; -1 for a member + // planned and built alone. Members of one group report the same + // `buildMs`, so a consumer that sums it over members deduplicates by this + // number. + int buildGroup = -1; +}; + +// One selected member of a `mcpp test` over several members, as the command +// layer found it: the path `[workspace] members` spells, and what discovery +// read from the member's own directory (two members may each have a +// `tests/main.cpp`). +export struct WorkspaceTestMember { + std::string path; + std::vector targets; + // Discovery failed: the member fails alone, with this message, and the + // others are planned without it. + std::string error; + // The line a member with no tests reports, naming where discovery looked. + std::string noTests; +}; + +// What the command layer reports around each member's run. `begin` is asked +// before a member's tests start, and a member it answers false for is not run +// (`--workspace-timeout`); `end` receives the member's exit status and summary. +export struct WorkspaceTestHooks { + std::function begin; + std::function end; }; // Minimal JSON string escaping for the --message-format json records. Same @@ -2580,127 +2646,136 @@ static std::string test_json_escape(std::string_view s) { return out; } -// `mcpp test` driver: discover tests/**/*.cpp, synthesize targets, build -// with dev-deps, run each test binary, summarize. -export int run_tests(std::span passthrough, - BuildOverrides overrides = {}, - TestOptions testOpts = {}, - TestRunSummary* summaryOut = nullptr) { - const bool json = (testOpts.format == TestMessageFormat::Json); - // The member this call is scoped to (empty outside a workspace). Threaded - // into every JSON record so a `--workspace` stream can be attributed: a - // bare test name is ambiguous the moment two members both have a `smoke`. - const std::string memberName = overrides.package_filter; - TestRunSummary summary; - struct SummaryWriter { - TestRunSummary* out; const TestRunSummary* src; - ~SummaryWriter() { if (out) *out = *src; } - } summaryWriter{summaryOut, &summary}; - // Wall clock for the WHOLE member, started before Phase A. The old `t0` - // sat after Phase A and the bulk pass, so `finished in` reported only the - // per-test loop: measured on one member, 6.53s printed against 93.5s - // actual — a 14x understatement, and worst exactly on the build-heavy - // members where the number matters. - auto tMember = std::chrono::steady_clock::now(); - auto member_ms = [&tMember] { - return std::chrono::duration_cast( - std::chrono::steady_clock::now() - tMember).count(); - }; - // JSON mode: stdout carries NDJSON only. All ui::status/info lines print - // to stdout, so silence them wholesale; errors already go to stderr. - if (json) mcpp::ui::set_quiet(true); - // The report covers the planning and the package's own build (Phase A); - // it is closed before the tests' own lines. - mcpp::build::progress::open(mcpp::log::is_verbose()); - - auto root = mcpp::project::find_manifest_root(std::filesystem::current_path()); - if (!root) { - mcpp::ui::error("no mcpp.toml found in current directory or any parent"); - return 2; - } +// A test's result, as the records and the summary read it. +struct TestResult { + std::string name; + // `NotRun` (#544): built, and not executed — the host cannot load the + // artifact, or the declared runner could not be found or started. + // Reporting that as `RunFail (exit 127)` states that the test ran and + // returned 127, which is false and indistinguishable from a missing + // program; reporting it as a pass would be read as one. + // `Built` (`--no-run`): compiled and linked, and deliberately not + // executed. Distinct from `NotRun`, which means mcpp tried and could + // not --- the difference is whether anything was left unanswered. + enum class St { Pass, CompileFail, RunFail, NotRun, Built } status; + int exitCode = 0; + std::string compileOutput; + std::string runOutput; + // The wall time of the step that decided the status: the run for a test + // that ran, the build for a `compile_fail`. The meaning `duration_ms` has + // always had; the test binary's own build time is `buildMs`. + long long durationMs = 0; + bool timedOut = false; // killed by --timeout + std::string reason; // NotRun only: why, in one sentence + // The build time of this test's own binary in this invocation: the sum of + // its link edge and its main unit's compile edge in `.ninja_log`, 0 when + // neither was rebuilt (`build_ms`, 2026.10.1.1+). + long long buildMs = 0; +}; - auto discovered = mcpp::build::discover_test_targets( - *root, overrides.package_filter); - if (!discovered) { - mcpp::ui::error(discovered.error()); - return 2; - } - auto testRoot = discovered->packageRoot; - auto testTargets = std::move(discovered->targets); - if (testTargets.empty()) { - // Names where it looked when the manifest chose the place, so that a - // glob that matches nothing is not read as a project without tests. - if (discovered->discoverDeclared) { - std::string globs; - for (auto const& g : discovered->discover) - globs += std::format("{}\"{}\"", globs.empty() ? "" : ", ", g); - std::println("no tests found ([test] discover = [{}])", globs); - } else { - std::println("no tests found in tests/"); - } - return 0; - } - // --list: enumerate (filtered) tests and stop — no toolchain resolution, - // no build. Names/paths come straight from discovery, so this also works - // on tests that do not currently compile. - if (testOpts.list) { - std::size_t total = 0; - for (auto& t : testTargets) { - if (!testOpts.filter.empty() - && t.name.find(testOpts.filter) == std::string::npos) continue; - ++total; - auto abs = std::filesystem::absolute(testRoot / t.main) - .lexically_normal().generic_string(); - if (json) - std::println("{{\"member\":\"{}\",\"test\":\"{}\",\"main\":\"{}\"}}", - test_json_escape(memberName), - test_json_escape(t.name), test_json_escape(abs)); - else - std::println("{}", t.name); - } - if (json) { - std::println("{{\"summary\":{{\"total\":{}}}}}", total); - std::fflush(stdout); - } - return 0; - } +// Streaming NDJSON: one record per test, emitted as it finishes — a +// consumer (e.g. the d2x provider) sees progress live, and a crash +// mid-run still leaves the completed records on stdout. +static void emit_test_json(const std::string& memberName, const TestResult& r) { + const char* st = r.status == TestResult::St::Pass ? "pass" + : r.status == TestResult::St::CompileFail ? "compile_fail" + : r.status == TestResult::St::NotRun ? "not_run" + : r.status == TestResult::St::Built ? "built" + : "run_fail"; + std::string signal = (r.exitCode > 128 && r.exitCode < 128 + 65) + ? std::to_string(r.exitCode - 128) : "null"; + std::println("{{\"member\":\"{}\",\"test\":\"{}\",\"status\":\"{}\"," + "\"exit_code\":{},\"signal\":{}," + "\"duration_ms\":{},\"timed_out\":{}," + "\"compile_output\":\"{}\",\"run_output\":\"{}\"," + "\"reason\":\"{}\",\"build_ms\":{}}}", + test_json_escape(memberName), + test_json_escape(r.name), st, r.exitCode, signal, r.durationMs, + r.timedOut ? "true" : "false", + test_json_escape(r.compileOutput), test_json_escape(r.runOutput), + test_json_escape(r.reason), r.buildMs); + std::fflush(stdout); +} - // 3. prepare_build with dev-deps enabled + synthetic targets. - // A test binary is executed, so the run tier applies here exactly as it - // does to `mcpp run` — `[xlings.workspace]` has no separate `test` tier - // because there is no separate need. - overrides.will_run = true; - auto ctx = prepare_build(/*print_fp=*/false, - /*includeDevDeps=*/true, - std::move(testTargets), - std::move(overrides)); - if (!ctx) { mcpp::ui::error(ctx.error()); return 2; } +// An edge ninja recorded in `.ninja_log`: what it made, and how long it took. +struct NinjaEdge { + std::string output; + long long ms = 0; +}; - // Filter guard. The filter selects at the build/run stage ONLY — the plan - // above always contains every test, so build.ninja and - // compile_commands.json stay complete (clangd depends on the latter; a - // filtered run must not clobber it down to one entry). - auto filter_match = [&](const mcpp::build::LinkUnit& lu) { - return lu.kind == mcpp::build::LinkUnit::TestBinary - && (testOpts.filter.empty() - || lu.targetName.find(testOpts.filter) != std::string::npos); - }; - if (!testOpts.filter.empty()) { - bool any = false; - for (auto& lu : ctx->plan.linkUnits) - if (filter_match(lu)) { any = true; break; } - if (!any) { - if (json) - std::println("{{\"error\":\"no-tests-matched\",\"filter\":\"{}\"}}", - test_json_escape(testOpts.filter)); - mcpp::ui::error(std::format("no tests match '{}'", testOpts.filter)); - return 2; +// The edges ninja appended to `log` after byte `from`: the ones the drives +// since that point ran. The log accumulates across invocations, so an edge +// that was not rebuilt has an old entry that says how long it took once; the +// offset taken before the first drive is what tells this run's edges from +// those. A log that ninja rewrote (recompaction) is shorter than the offset +// and reads as no edges, which is an absent measurement and not a zero. +// +// Format (ninja log v5): `start_ms TAB end_ms TAB mtime TAB output TAB hash`. +static std::vector ninja_edges_since(const std::filesystem::path& log, + std::uintmax_t from) { + std::vector edges; + std::error_code ec; + const auto size = std::filesystem::file_size(log, ec); + if (ec || size <= from) return edges; + std::ifstream is(log, std::ios::binary); + if (!is) return edges; + is.seekg(static_cast(from)); + std::string line; + while (std::getline(is, line)) { + std::array f{}; + std::size_t n = 0, b = 0; + std::string_view v = line; + while (n < f.size()) { + const auto t = v.find('\t', b); + f[n++] = v.substr(b, t == std::string_view::npos ? std::string_view::npos : t - b); + if (t == std::string_view::npos) break; + b = t + 1; } + if (n < 4) continue; + long long start = 0, end = 0; + auto [p1, e1] = std::from_chars(f[0].data(), f[0].data() + f[0].size(), start); + auto [p2, e2] = std::from_chars(f[1].data(), f[1].data() + f[1].size(), end); + if (e1 != std::errc{} || e2 != std::errc{} || end < start) continue; + edges.push_back({std::string(f[3]), end - start}); } + return edges; +} - // 4. "Compiling test_X (test)" lines for the test binaries. +// A planned test build, and what the tests run against it need: the plan, the +// backend that drives it, how long planning and the build took, and how the +// test binaries are executed. One per configuration group of a `mcpp test` +// over several members; `run_tests` holds one for its one member. +struct TestBuild { + std::optional ctx; + std::unique_ptr backend; + long long prepareMs = 0; + // Phase A, the bulk pass, and any attribution drive. + long long buildMs = 0; + bool bulkBuiltEverything = false; + std::filesystem::path ninjaLog; + std::uintmax_t logFrom = 0; + + // How the test binaries are executed, resolved once for the build + // (#544): every test of one build shares a target. + RunnerChoice runnerChoice; + std::vector runnerTmpl; + // Non-empty ⇒ no test is spawned; every one is reported NotRun with it. + std::string invocationNotRunReason; + // Non-zero: there is nothing to execute the tests with, and this is the + // exit status every member of the build returns. + int runnerFatal = 0; + // Set by the first worker whose spawn the kernel refused; every worker + // checks it before spawning. Workers already past the check may be + // refused the same way — harmless, a refused spawn has no side effects — + // and each such result is NotRun, not RunFail. The reason is printed once. + std::atomic hostCannotRun{false}; + std::string hostCannotRunReason; +}; + +// The "Compiling " lines the tests' own lines follow. +static void test_announce(const BuildContext& ctx) { std::map cachedUnits; - for (auto& dep : ctx->cachedDeps) cachedUnits[dep.name] = dep.units; + for (auto& dep : ctx.cachedDeps) cachedUnits[dep.name] = dep.units; auto announce = [&](const std::string& name, const mcpp::manifest::DependencySpec& spec, std::string_view suffix) { @@ -2716,184 +2791,247 @@ export int run_tests(std::span passthrough, } }; std::set announced; - announced.insert(ctx->manifest.package.name); - mcpp::ui::status("Compiling", - std::format("{} v{} (.)", - ctx->manifest.package.name, ctx->manifest.package.version)); - for (auto& [name, spec] : ctx->manifest.dependencies) { - if (announced.contains(name)) continue; - announced.insert(name); - announce(name, spec, ""); - } - for (auto& [name, spec] : ctx->manifest.devDependencies) { - if (announced.contains(name)) continue; - announced.insert(name); - announce(name, spec, " (dev)"); - } - // List test binaries. - // (Per-test "Compiling" lines print in Phase B, interleaved with each - // test's own result — announcing them all up front separated the three - // pieces of one test's story across the whole output.) - - // 5. Two-phase build. Phase A: package-level artifacts (everything that - // is not a test binary — libs, deps). A failure here is the PACKAGE's - // fault, not any single test's: report it as a build error, never as - // N red tests. Phase B (below): each test is built as its own ninja - // goal, so a compile failure is attributed to exactly that test and - // the rest still build and run. - struct TestResult { - std::string name; - // `NotRun` (#544): built, and not executed — the host cannot load the - // artifact, or the declared runner could not be found or started. - // Reporting that as `RunFail (exit 127)` states that the test ran and - // returned 127, which is false and indistinguishable from a missing - // program; reporting it as a pass would be read as one. - // `Built` (`--no-run`): compiled and linked, and deliberately not - // executed. Distinct from `NotRun`, which means mcpp tried and could - // not --- the difference is whether anything was left unanswered. - enum class St { Pass, CompileFail, RunFail, NotRun, Built } status; - int exitCode = 0; - std::string compileOutput; - std::string runOutput; - long long durationMs = 0; // build+run wall time for THIS test - bool timedOut = false; // killed by --timeout - std::string reason; // NotRun only: why, in one sentence + auto announce_package = [&](const mcpp::manifest::Manifest& m, std::string_view where) { + announced.insert(m.package.name); + mcpp::ui::status("Compiling", + std::format("{} v{} ({})", m.package.name, m.package.version, where)); }; - std::vector results; - - // Streaming NDJSON: one record per test, emitted as it finishes — a - // consumer (e.g. the d2x provider) sees progress live, and a crash - // mid-run still leaves the completed records on stdout. - auto emit_json = [&](const TestResult& r) { - if (!json) return; - const char* st = r.status == TestResult::St::Pass ? "pass" - : r.status == TestResult::St::CompileFail ? "compile_fail" - : r.status == TestResult::St::NotRun ? "not_run" - : r.status == TestResult::St::Built ? "built" - : "run_fail"; - std::string signal = (r.exitCode > 128 && r.exitCode < 128 + 65) - ? std::to_string(r.exitCode - 128) : "null"; - std::println("{{\"member\":\"{}\",\"test\":\"{}\",\"status\":\"{}\"," - "\"exit_code\":{},\"signal\":{}," - "\"duration_ms\":{},\"timed_out\":{}," - "\"compile_output\":\"{}\",\"run_output\":\"{}\"," - "\"reason\":\"{}\"}}", - test_json_escape(memberName), - test_json_escape(r.name), st, r.exitCode, signal, r.durationMs, - r.timedOut ? "true" : "false", - test_json_escape(r.compileOutput), test_json_escape(r.runOutput), - test_json_escape(r.reason)); - std::fflush(stdout); + auto announce_dependencies = [&](const mcpp::manifest::Manifest& m) { + for (auto& [name, spec] : m.dependencies) { + if (announced.contains(name)) continue; + announced.insert(name); + announce(name, spec, ""); + } + for (auto& [name, spec] : m.devDependencies) { + if (announced.contains(name)) continue; + announced.insert(name); + announce(name, spec, " (dev)"); + } }; + // A plan of several members has a virtual root, which declares only its + // members: the packages are the members, named as their directories. + if (ctx.manifest.package.virtualRoot && !ctx.workspaceMembers.empty()) { + for (auto const& wm : ctx.workspaceMembers) announce_package(wm.manifest, wm.memberPath); + for (auto const& wm : ctx.workspaceMembers) announce_dependencies(wm.manifest); + return; + } + announce_package(ctx.manifest, "."); + announce_dependencies(ctx.manifest); +} - auto backend = mcpp::build::make_ninja_backend(); - - // Phase A goal set: every shared prerequisite — all package/dep compile - // units EXCEPT the tests' own main TUs, plus any non-test link outputs. - // In test mode the lib link unit is skipped entirely (plan.cppm), so the - // package's module objects are the only place shared breakage can show - // up; building them here is what keeps a broken src/ module a PACKAGE - // error instead of N identical per-test compile failures. +// Phase A goal set: every shared prerequisite — all package/dep compile +// units EXCEPT the tests' own main TUs, plus any non-test link outputs. +// In test mode the lib link unit is skipped entirely (plan.cppm), so the +// package's module objects are the only place shared breakage can show +// up; building them here is what keeps a broken src/ module a PACKAGE +// error instead of N identical per-test compile failures. +static std::vector test_package_goals(const BuildContext& ctx) { std::set testMains; - for (auto& lu : ctx->plan.linkUnits) + for (auto& lu : ctx.plan.linkUnits) if (lu.kind == mcpp::build::LinkUnit::TestBinary && lu.entryMain) testMains.insert(*lu.entryMain); - std::vector pkgTargets; - for (auto& cu : ctx->plan.compileUnits) + std::vector goals; + for (auto& cu : ctx.plan.compileUnits) if (!testMains.contains(cu.source)) - pkgTargets.push_back(cu.object.generic_string()); - for (auto& lu : ctx->plan.linkUnits) + goals.push_back(cu.object.generic_string()); + for (auto& lu : ctx.plan.linkUnits) if (lu.kind != mcpp::build::LinkUnit::TestBinary) - pkgTargets.push_back(lu.output.generic_string()); - mcpp::build::progress::programs_done(); - if (!pkgTargets.empty()) { - mcpp::build::BuildOptions aOpts; - aOpts.ninjaTargets = pkgTargets; - aOpts.buildTimeoutSecs = static_cast(testOpts.buildTimeoutSecs); - // Phase A is the package's own build, and is reported as `mcpp build` - // reports one (build progress design 2026-09-29); the tests' own - // builds and runs below keep their per-test lines. - std::optional phaseReport; - if (!json && !mcpp::ui::is_quiet()) { - phaseReport.emplace(ctx->outputDir); - aOpts.progress = &*phaseReport; - } - auto tPhaseA = std::chrono::steady_clock::now(); - auto a = backend->build(ctx->plan, aOpts); - summary.buildMs += std::chrono::duration_cast( - std::chrono::steady_clock::now() - tPhaseA).count(); - if (!a) { - summary.packageError = true; - summary.elapsedMs = member_ms(); - std::fflush(stdout); - if (json) - std::println("{{\"error\":\"package\",\"compile_output\":\"{}\"}}", - test_json_escape(a.error().diagnosticOutput)); - // Surface the compiler/linker stderr (parity with run_build_plan) — - // otherwise `mcpp test` failures show only "build failed" with no - // diagnostic, which is undebuggable (notably on CI). A failed step - // was reported when it failed. - if (!a.error().reported) mcpp::ui::error(a.error().message); - mcpp::ui::block(a.error().diagnosticOutput); - return 1; - } + goals.push_back(lu.output.generic_string()); + return goals; +} - // M3.2: populate BMI cache for deps that did NOT hit cache — deps - // are package-level artifacts, so this belongs right after Phase A. - for (auto& task : ctx->depsToPopulate) { - auto pr = mcpp::bmi_cache::populate_from(task.key, ctx->outputDir, task.artifacts); - if (!pr) { - mcpp::ui::warning(std::format( - "bmi cache populate failed for {}@{}: {}", - task.key.packageName, task.key.version, pr.error())); - } +// The part of Phase A that one member's tests need: the objects of the +// member's closure, which are the ones its test binaries link other than the +// tests' own main TUs. A member whose package does not build is found by +// building these alone, so that a group's other members still run. +static std::vector member_package_goals(const BuildContext& ctx, + std::string_view owner) { + std::set testMains; + for (auto& lu : ctx.plan.linkUnits) + if (lu.kind == mcpp::build::LinkUnit::TestBinary && lu.entryMain) + testMains.insert(*lu.entryMain); + std::set mainObjects; + for (auto& cu : ctx.plan.compileUnits) + if (testMains.contains(cu.source)) mainObjects.insert(cu.object); + std::set seen; + std::vector goals; + for (auto& lu : ctx.plan.linkUnits) { + if (lu.kind != mcpp::build::LinkUnit::TestBinary || lu.memberOf != owner) continue; + for (auto& o : lu.objects) + if (!mainObjects.contains(o) && seen.insert(o).second) + goals.push_back(o.generic_string()); + } + return goals; +} + +// Phase A, run against `tb` (see the note at its two callers): everything +// every test shares, built once. Nullopt when it built; else the failure, +// which the caller reports. Its wall time is added to the build's. +static std::optional test_phase_a(TestBuild& tb, const TestOptions& testOpts, + bool json) { + auto* ctx = &*tb.ctx; + auto& backend = tb.backend; + auto pkgTargets = test_package_goals(*ctx); + if (pkgTargets.empty()) return std::nullopt; + mcpp::build::BuildOptions aOpts; + aOpts.ninjaTargets = pkgTargets; + aOpts.buildTimeoutSecs = static_cast(testOpts.buildTimeoutSecs); + // Phase A is the package's own build, and is reported as `mcpp build` + // reports one (build progress design 2026-09-29); the tests' own + // builds and runs below keep their per-test lines. + std::optional phaseReport; + if (!json && !mcpp::ui::is_quiet()) { + phaseReport.emplace(ctx->outputDir); + aOpts.progress = &*phaseReport; + } + auto tPhaseA = std::chrono::steady_clock::now(); + auto a = backend->build(ctx->plan, aOpts); + tb.buildMs += std::chrono::duration_cast( + std::chrono::steady_clock::now() - tPhaseA).count(); + if (!a) return std::move(a.error()); + + // M3.2: populate BMI cache for deps that did NOT hit cache — deps + // are package-level artifacts, so this belongs right after Phase A. + for (auto& task : ctx->depsToPopulate) { + auto pr = mcpp::bmi_cache::populate_from(task.key, ctx->outputDir, task.artifacts); + if (!pr) { + mcpp::ui::warning(std::format( + "bmi cache populate failed for {}@{}: {}", + task.key.packageName, task.key.version, pr.error())); } + } + + // No "Finished test" line here: Phase A only built the shared + // prerequisites. Printing a success banner right before per-test + // failures read as a contradiction; the final summary carries timing. + return std::nullopt; +} - // No "Finished test" line here: Phase A only built the shared - // prerequisites. Printing a success banner right before per-test - // failures read as a contradiction; the final summary carries timing. +// 6. Phase B. First a single keep-going bulk build over every selected +// test goal — ninja parallelizes across tests and a failing test does +// not stop the rest (-k 0). The result is deliberately ignored: the +// per-test loop below re-drives each goal so a failure is attributed to +// exactly one test. +// +// ...but ONLY when this bulk build failed. A re-drive was assumed to be +// a near no-op, and it is not: a drive re-emits build.ninja, rewrites +// compile_commands.json, spawns ninja and re-validates the runtime +// closure. Measured on the 83-test suite AFTER the rule E fix, that is +// still ~39ms x 83 = 3.2s of a 5.3s hot run — spent re-asking a question +// the bulk build just answered for every test at once. +// +// `-k 0` means the bulk exit code is 0 IFF every selected goal built, so +// it carries exactly the information the loop was re-deriving. When it +// is non-zero the loop runs as before and each failure still names its +// own test. +// `keep` says which test binaries are goals. +template +static void test_bulk(TestBuild& tb, const TestOptions& testOpts, Keep&& keep) { + auto* ctx = &*tb.ctx; + auto& backend = tb.backend; + mcpp::build::BuildOptions bulk; + bulk.keepGoing = true; + bulk.buildTimeoutSecs = static_cast(testOpts.buildTimeoutSecs); + for (auto& lu : ctx->plan.linkUnits) + if (keep(lu)) + bulk.ninjaTargets.push_back(lu.output.generic_string()); + if (!bulk.ninjaTargets.empty()) { + auto tBulk = std::chrono::steady_clock::now(); + tb.bulkBuiltEverything = backend->build(ctx->plan, bulk).has_value(); + tb.buildMs += std::chrono::duration_cast( + std::chrono::steady_clock::now() - tBulk).count(); } - // The tests' own lines follow, as they always have. - mcpp::build::progress::close(); +} - // 6. Phase B. First a single keep-going bulk build over every selected - // test goal — ninja parallelizes across tests and a failing test does - // not stop the rest (-k 0). The result is deliberately ignored: the - // per-test loop below re-drives each goal so a failure is attributed to - // exactly one test. - // - // ...but ONLY when this bulk build failed. A re-drive was assumed to be - // a near no-op, and it is not: a drive re-emits build.ninja, rewrites - // compile_commands.json, spawns ninja and re-validates the runtime - // closure. Measured on the 83-test suite AFTER the rule E fix, that is - // still ~39ms x 83 = 3.2s of a 5.3s hot run — spent re-asking a question - // the bulk build just answered for every test at once. - // - // `-k 0` means the bulk exit code is 0 IFF every selected goal built, so - // it carries exactly the information the loop was re-deriving. When it - // is non-zero the loop runs as before and each failure still names its - // own test. - bool bulkBuiltEverything = false; - { - mcpp::build::BuildOptions bulk; - bulk.keepGoing = true; - bulk.buildTimeoutSecs = static_cast(testOpts.buildTimeoutSecs); - for (auto& lu : ctx->plan.linkUnits) - if (filter_match(lu)) - bulk.ninjaTargets.push_back(lu.output.generic_string()); - if (!bulk.ninjaTargets.empty()) { - auto tBulk = std::chrono::steady_clock::now(); - bulkBuiltEverything = backend->build(ctx->plan, bulk).has_value(); - summary.buildMs += std::chrono::duration_cast( - std::chrono::steady_clock::now() - tBulk).count(); - } +// How the test binaries are executed — the SAME runner `mcpp run` uses, +// resolved ONCE per build (#544). One read point, two callers; and +// one lookup, because every test of a build shares a target, so +// "the runner's program is not there" is a fact about the build and +// is reported once rather than once per test. +// +// Nothing else about the test model changes, and that is a measured +// result rather than a simplification: semihosting propagates the +// firmware's `main` return value to the emulator's exit code +// (`return 7` → qemu exits 7, verified), so "exit code is the verdict" +// holds under a runner exactly as it does on the host. +static void test_resolve_runner(TestBuild& tb, const TestOptions& testOpts, bool json) { + auto* ctx = &*tb.ctx; + tb.runnerChoice = choose_runner(*ctx, testOpts.noRunner); + auto& runnerChoice = tb.runnerChoice; + if (runnerChoice.ignored && !json) + mcpp::ui::info("note", std::format( + "--no-runner: ignoring the runner declared for {}", runnerChoice.tripleKey)); + if (runnerChoice.fromManifest && !json) + mcpp::ui::info("note", std::format( + "[target.{}].runner overrides the runner a dependency supplied", + runnerChoice.tripleKey)); + if (runnerChoice.freestanding && runnerChoice.tmpl.empty()) { + std::println(stderr, "error: {}", + mcpp::freestanding::no_runner_message(runnerChoice.tripleKey)); + tb.runnerFatal = 2; + return; + } + tb.runnerTmpl = runnerChoice.tmpl; + auto& runnerTmpl = tb.runnerTmpl; + if (!runnerTmpl.empty()) { + const char* pathEnv = std::getenv("PATH"); + auto found = mcpp::build::runner_lookup::locate( + runnerTmpl.front(), ctx->xlingsDepBinDirs, pathEnv ? pathEnv : ""); + if (found.program) runnerTmpl.front() = found.program->string(); + else tb.invocationNotRunReason = mcpp::build::runner_lookup::not_found_message( + runnerChoice.tripleKey, runnerTmpl.front(), found.searched); } +} + +// One member's tests, against a build that exists: the per-test builds that +// the bulk pass did not answer, then the runs, then the member's summary. +// `owner` is the member's package name in the plan, which picks its test +// binaries out of a plan that holds several members' (empty: every test +// binary of the plan). `carriedMs` is the time the member has already spent +// on planning and building, which `elapsed_ms` counts; `summary.buildMs` +// arrives holding the build's wall time. +static int test_run_member(TestBuild& tb, const TestOptions& testOpts, + std::span passthrough, bool json, + const std::string& memberName, const std::string& owner, + long long carriedMs, TestRunSummary& summary) { + auto* ctx = &*tb.ctx; + auto& backend = tb.backend; + const auto tLoop = std::chrono::steady_clock::now(); + if (tb.runnerFatal) return tb.runnerFatal; + const auto& runnerChoice = tb.runnerChoice; + auto& runnerTmpl = tb.runnerTmpl; + auto& invocationNotRunReason = tb.invocationNotRunReason; + auto& hostCannotRun = tb.hostCannotRun; + auto& hostCannotRunReason = tb.hostCannotRunReason; + const bool bulkBuiltEverything = tb.bulkBuiltEverything; - // Then build + run each test in sequence; collect results. + // Filter guard. The filter selects at the build/run stage ONLY — the plan + // always contains every test, so build.ninja and compile_commands.json + // stay complete (clangd depends on the latter; a filtered run must not + // clobber it down to one entry). + auto filter_match = [&](const mcpp::build::LinkUnit& lu) { + return lu.kind == mcpp::build::LinkUnit::TestBinary + && (owner.empty() || lu.memberOf == owner) + && (testOpts.filter.empty() + || lu.targetName.find(testOpts.filter) != std::string::npos); + }; + std::vector results; + auto emit_json = [&](const TestResult& r) { + if (!json) return; + emit_test_json(memberName, r); + }; + // The runtime of THIS member: in a plan of several members, the plan's own + // directories are the union over every member, and a member's tests are + // told about its closure's alone. auto runtimeEnvKey = mcpp::platform::env::runtime_library_path_key(); - auto runtimeEnvValue = mcpp::platform::env::prepend_path_list( - runtimeEnvKey, ctx->plan.runtimeLibraryDirs); + std::string runtimeEnvValue; + bool hasRuntimeDirs = false; + with_member(*ctx, owner, [&] { + runtimeEnvValue = mcpp::platform::env::prepend_path_list( + runtimeEnvKey, ctx->plan.runtimeLibraryDirs); + hasRuntimeDirs = !ctx->plan.runtimeLibraryDirs.empty(); + }); // Read once for the whole run rather than per test: it is one file, and // every test in a run belongs to the same subos. const auto subosEnv = compute_subos_env(ctx->plan); @@ -2905,7 +3043,7 @@ export int run_tests(std::span passthrough, // here with a dyld error that names neither the cause nor the platform — // so say it out loud rather than leaving the difference silent. if constexpr (mcpp::platform::is_macos) { - if (runtimeEnvKey.empty() && !ctx->plan.runtimeLibraryDirs.empty()) { + if (runtimeEnvKey.empty() && hasRuntimeDirs) { mcpp::diag::warning("test/runtime-path", "macOS does not inject a runtime library path for test binaries " "(DYLD_LIBRARY_PATH is deliberately not set); dependencies must be " @@ -2945,6 +3083,7 @@ export int run_tests(std::span passthrough, std::string name; std::vector argv; std::vector> env; + long long buildMs = 0; // this test's own edges in .ninja_log }; std::vector runnable; @@ -2955,47 +3094,6 @@ export int run_tests(std::span passthrough, // the terminal interleaves them line by line, which does not just look // untidy — it makes a failing assertion unattributable, and the whole // reason the per-test loop exists is attribution. - // How the test binaries are executed — the SAME runner `mcpp run` uses, - // resolved ONCE per invocation (#544). One read point, two callers; and - // one lookup, because every test of an invocation shares a target, so - // "the runner's program is not there" is a fact about the invocation and - // is reported once rather than once per test. - // - // Nothing else about the test model changes, and that is a measured - // result rather than a simplification: semihosting propagates the - // firmware's `main` return value to the emulator's exit code - // (`return 7` → qemu exits 7, verified), so "exit code is the verdict" - // holds under a runner exactly as it does on the host. - const auto runnerChoice = choose_runner(*ctx, testOpts.noRunner); - if (runnerChoice.ignored && !json) - mcpp::ui::info("note", std::format( - "--no-runner: ignoring the runner declared for {}", runnerChoice.tripleKey)); - if (runnerChoice.fromManifest && !json) - mcpp::ui::info("note", std::format( - "[target.{}].runner overrides the runner a dependency supplied", - runnerChoice.tripleKey)); - if (runnerChoice.freestanding && runnerChoice.tmpl.empty()) { - std::println(stderr, "error: {}", - mcpp::freestanding::no_runner_message(runnerChoice.tripleKey)); - return 2; - } - std::vector runnerTmpl = runnerChoice.tmpl; - // Non-empty ⇒ no test is spawned; every one is reported NotRun with it. - std::string invocationNotRunReason; - if (!runnerTmpl.empty()) { - const char* pathEnv = std::getenv("PATH"); - auto found = mcpp::build::runner_lookup::locate( - runnerTmpl.front(), ctx->xlingsDepBinDirs, pathEnv ? pathEnv : ""); - if (found.program) runnerTmpl.front() = found.program->string(); - else invocationNotRunReason = mcpp::build::runner_lookup::not_found_message( - runnerChoice.tripleKey, runnerTmpl.front(), found.searched); - } - // Set by the first worker whose spawn the kernel refused; every worker - // checks it before spawning. Workers already past the check may be - // refused the same way — harmless, a refused spawn has no side effects — - // and each such result is NotRun, not RunFail. The reason is printed once. - std::atomic hostCannotRun{false}; - std::string hostCannotRunReason; auto run_tests_now = [&](std::vector& list) { if (list.empty()) return; @@ -3105,7 +3203,7 @@ export int run_tests(std::span passthrough, } if (!json) mcpp::ui::plain(std::format("{} ... not run", r.name)); results.push_back({r.name, TestResult::St::NotRun, 0, {}, {}, ms, false, - reason}); + reason, r.buildMs}); std::fflush(stdout); emit_json(results.back()); continue; @@ -3114,18 +3212,18 @@ export int run_tests(std::span passthrough, if (!json) mcpp::ui::plain(std::format( "{} ... FAIL (timeout after {}s)", r.name, testOpts.timeoutSecs)); results.push_back({r.name, TestResult::St::RunFail, exitCode, {}, - runOutput, ms, true}); + runOutput, ms, true, {}, r.buildMs}); } else if (exitCode == 0) { if (!json) mcpp::ui::plain(std::format( "{} ... ok ({:.2f}s)", r.name, static_cast(ms) / 1000.0)); results.push_back({r.name, TestResult::St::Pass, 0, {}, - runOutput, ms}); + runOutput, ms, false, {}, r.buildMs}); } else { if (!json) mcpp::ui::plain(std::format( "{} ... FAIL (exit {}, {:.2f}s)", r.name, exitCode, static_cast(ms) / 1000.0)); results.push_back({r.name, TestResult::St::RunFail, exitCode, {}, - runOutput, ms}); + runOutput, ms, false, {}, r.buildMs}); } // The captured output belongs directly under its own line, or // it is attributable to nothing. @@ -3153,6 +3251,19 @@ export int run_tests(std::span passthrough, std::chrono::steady_clock::now() - tRunPhase).count(); }; + // The build part of a test's duration is that test binary's own edges, its + // main TU and its link, among the ones ninja logged for this build. Read + // once when the bulk pass built every test; after each drive otherwise, + // since a drive logs its own. + std::map edgeMs; + auto read_edges = [&] { + edgeMs.clear(); + for (auto& e : ninja_edges_since(tb.ninjaLog, tb.logFrom)) edgeMs[e.output] += e.ms; + }; + read_edges(); + std::map objectOf; + for (auto& cu : ctx->plan.compileUnits) objectOf[cu.source] = cu.object; + for (auto& lu : ctx->plan.linkUnits) { if (!filter_match(lu)) continue; @@ -3200,6 +3311,11 @@ export int run_tests(std::span passthrough, } auto exe = ctx->outputDir / lu.output; + if (!bulkBuiltEverything) read_edges(); + long long buildMsOfTest = edgeMs[lu.output.generic_string()]; + if (lu.entryMain) + if (auto o = objectOf.find(*lu.entryMain); o != objectOf.end()) + buildMsOfTest += edgeMs[o->second.generic_string()]; // Through the runner resolved once above, or bare. The runner's // program was located already; only the artifact changes per test. @@ -3219,8 +3335,9 @@ export int run_tests(std::span passthrough, // `mcpp run` hands them over (see `runtime_files_for`). Written here, // in the single-threaded pass, one list per test program. if (!runnerTmpl.empty()) { - if (auto listed = write_runtime_files_list(*ctx, exe, - runtime_files_for(*ctx, exe))) + std::vector> carried; + with_member(*ctx, owner, [&] { carried = runtime_files_for(*ctx, exe, owner); }); + if (auto listed = write_runtime_files_list(*ctx, exe, carried)) childEnv.emplace_back(std::string(kRuntimeFilesEnv), listed->string()); else if (invocationNotRunReason.empty()) invocationNotRunReason = listed.error(); @@ -3242,9 +3359,9 @@ export int run_tests(std::span passthrough, } } - runnable.push_back({lu.targetName, std::move(argv), std::move(childEnv)}); + runnable.push_back({lu.targetName, std::move(argv), std::move(childEnv), + buildMsOfTest}); } - // Pass 2: run them. Concurrently unless there is exactly one — see // `runJobs` for why the single-test case is deliberately different. // @@ -3255,11 +3372,12 @@ export int run_tests(std::span passthrough, if (testOpts.noRun) { for (auto& r : runnable) results.push_back({r.name, TestResult::St::Built, 0, {}, {}, 0, - false, {}}); + false, {}, r.buildMs}); } else { run_tests_now(runnable); } - summary.elapsedMs = member_ms(); + summary.elapsedMs = carriedMs + std::chrono::duration_cast( + std::chrono::steady_clock::now() - tLoop).count(); // 7. Summary. int passed = 0; @@ -3300,13 +3418,18 @@ export int run_tests(std::span passthrough, const int rc = failed ? 1 : (notRun ? 2 : 0); if (json) { + // `build_group` names the configuration group whose one build this + // member's tests waited for, when it was planned with others; its + // `build_ms` is then the group's (docs/50 §8). + const auto group = summary.buildGroup >= 0 + ? std::format(",\"build_group\":{}", summary.buildGroup) : std::string{}; std::println("{{\"summary\":{{\"member\":\"{}\",\"passed\":{},\"failed\":{}," "\"not_run\":{},\"not_run_reason\":\"{}\"," "\"built\":{}," - "\"elapsed_ms\":{},\"build_ms\":{},\"run_ms\":{}}}}}", + "\"elapsed_ms\":{},\"build_ms\":{},\"run_ms\":{}{}}}}}", test_json_escape(memberName), passed, failed, notRun, test_json_escape(notRunReason), built, - summary.elapsedMs, summary.buildMs, summary.runMs); + summary.elapsedMs, summary.buildMs, summary.runMs, group); std::fflush(stdout); return rc; } @@ -3323,7 +3446,7 @@ export int run_tests(std::span passthrough, std::println(""); if (rc == 0) { - mcpp::ui::status("test result", + mcpp::ui::result("test result", std::format("ok. {}; finished in {}", counts, timing)); return 0; } @@ -3340,6 +3463,466 @@ export int run_tests(std::span passthrough, return rc; } +// `mcpp test` driver: discover tests/**/*.cpp, synthesize targets, build +// with dev-deps, run each test binary, summarize. +export int run_tests(std::span passthrough, + BuildOverrides overrides = {}, + TestOptions testOpts = {}, + TestRunSummary* summaryOut = nullptr) { + const bool json = (testOpts.format == TestMessageFormat::Json); + // The member this call is scoped to (empty outside a workspace). Threaded + // into every JSON record so a `--workspace` stream can be attributed: a + // bare test name is ambiguous the moment two members both have a `smoke`. + const std::string memberName = overrides.package_filter; + TestRunSummary summary; + struct SummaryWriter { + TestRunSummary* out; const TestRunSummary* src; + ~SummaryWriter() { if (out) *out = *src; } + } summaryWriter{summaryOut, &summary}; + // Wall clock for the WHOLE member, started before Phase A. The old `t0` + // sat after Phase A and the bulk pass, so `finished in` reported only the + // per-test loop: measured on one member, 6.53s printed against 93.5s + // actual — a 14x understatement, and worst exactly on the build-heavy + // members where the number matters. + auto tMember = std::chrono::steady_clock::now(); + auto member_ms = [&tMember] { + return std::chrono::duration_cast( + std::chrono::steady_clock::now() - tMember).count(); + }; + // JSON mode: stdout carries NDJSON only. All ui::status/info lines print + // to stdout, so silence them wholesale; errors already go to stderr. + if (json) mcpp::ui::set_quiet(true); + // The report covers the planning and the package's own build (Phase A); + // it is closed before the tests' own lines. + mcpp::build::progress::open(mcpp::log::is_verbose()); + + auto root = mcpp::project::find_manifest_root(std::filesystem::current_path()); + if (!root) { + mcpp::ui::error("no mcpp.toml found in current directory or any parent"); + return 2; + } + + auto discovered = mcpp::build::discover_test_targets( + *root, overrides.package_filter); + if (!discovered) { + mcpp::ui::error(discovered.error()); + return 2; + } + auto testRoot = discovered->packageRoot; + auto testTargets = std::move(discovered->targets); + if (testTargets.empty()) { + // Names where it looked when the manifest chose the place, so that a + // glob that matches nothing is not read as a project without tests. + if (discovered->discoverDeclared) { + std::string globs; + for (auto const& g : discovered->discover) + globs += std::format("{}\"{}\"", globs.empty() ? "" : ", ", g); + std::println("no tests found ([test] discover = [{}])", globs); + } else { + std::println("no tests found in tests/"); + } + return 0; + } + // --list: enumerate (filtered) tests and stop — no toolchain resolution, + // no build. Names/paths come straight from discovery, so this also works + // on tests that do not currently compile. + if (testOpts.list) { + std::size_t total = 0; + for (auto& t : testTargets) { + if (!testOpts.filter.empty() + && t.name.find(testOpts.filter) == std::string::npos) continue; + ++total; + auto abs = std::filesystem::absolute(testRoot / t.main) + .lexically_normal().generic_string(); + if (json) + std::println("{{\"member\":\"{}\",\"test\":\"{}\",\"main\":\"{}\"}}", + test_json_escape(memberName), + test_json_escape(t.name), test_json_escape(abs)); + else + std::println("{}", t.name); + } + if (json) { + std::println("{{\"summary\":{{\"total\":{}}}}}", total); + std::fflush(stdout); + } + return 0; + } + // 3. prepare_build with dev-deps enabled + synthetic targets. + // A test binary is executed, so the run tier applies here exactly as it + // does to `mcpp run` — `[xlings.workspace]` has no separate `test` tier + // because there is no separate need. + overrides.will_run = true; + auto prepared = prepare_build(/*print_fp=*/false, + /*includeDevDeps=*/true, + std::move(testTargets), + std::move(overrides)); + if (!prepared) { mcpp::ui::error(prepared.error()); return 2; } + TestBuild tb; + tb.ctx.emplace(std::move(*prepared)); + tb.backend = mcpp::build::make_ninja_backend(); + tb.ninjaLog = tb.ctx->outputDir / ".ninja_log"; + { + std::error_code ec; + tb.logFrom = std::filesystem::file_size(tb.ninjaLog, ec); + if (ec) tb.logFrom = 0; + } + auto* ctx = &*tb.ctx; + + // Filter guard. The filter selects at the build/run stage ONLY — the plan + // above always contains every test, so build.ninja and + // compile_commands.json stay complete (clangd depends on the latter; a + // filtered run must not clobber it down to one entry). + auto filter_match = [&](const mcpp::build::LinkUnit& lu) { + return lu.kind == mcpp::build::LinkUnit::TestBinary + && (testOpts.filter.empty() + || lu.targetName.find(testOpts.filter) != std::string::npos); + }; + if (!testOpts.filter.empty()) { + bool any = false; + for (auto& lu : ctx->plan.linkUnits) + if (filter_match(lu)) { any = true; break; } + if (!any) { + if (json) + std::println("{{\"error\":\"no-tests-matched\",\"filter\":\"{}\"}}", + test_json_escape(testOpts.filter)); + mcpp::ui::error(std::format("no tests match '{}'", testOpts.filter)); + return 2; + } + } + + // 4. "Compiling test_X (test)" lines for the test binaries. + test_announce(*ctx); + // List test binaries. + // (Per-test "Compiling" lines print in Phase B, interleaved with each + // test's own result — announcing them all up front separated the three + // pieces of one test's story across the whole output.) + + // 5. Two-phase build. Phase A: package-level artifacts (everything that + // is not a test binary — libs, deps). A failure here is the PACKAGE's + // fault, not any single test's: report it as a build error, never as + // N red tests. Phase B (below): each test is built as its own ninja + // goal, so a compile failure is attributed to exactly that test and + // the rest still build and run. + mcpp::build::progress::programs_done(); + if (auto a = test_phase_a(tb, testOpts, json)) { + summary.packageError = true; + summary.buildMs = tb.buildMs; + summary.elapsedMs = member_ms(); + std::fflush(stdout); + if (json) + std::println("{{\"error\":\"package\",\"compile_output\":\"{}\"}}", + test_json_escape(a->diagnosticOutput)); + // Surface the compiler/linker stderr (parity with run_build_plan) — + // otherwise `mcpp test` failures show only "build failed" with no + // diagnostic, which is undebuggable (notably on CI). A failed step + // was reported when it failed. + if (!a->reported) mcpp::ui::error(a->message); + mcpp::ui::block(a->diagnosticOutput); + return 1; + } + // The tests' own lines follow, as they always have. + mcpp::build::progress::close(); + + test_bulk(tb, testOpts, filter_match); + test_resolve_runner(tb, testOpts, json); + summary.buildMs = tb.buildMs; + return test_run_member(tb, testOpts, passthrough, json, memberName, /*owner=*/"", + member_ms(), summary); +} + +// `mcpp test` over several members: the members are planned once per +// configuration group, each group's Phase A and test goals are built once, and +// then each member's tests run in member order, continuing past a failing +// member (member-selection design 2026-09-30, S4 and D1). +// +// `groups` holds the selected members by configuration, as `mcpp build` groups +// them, and `members` the same members in `[workspace] members` order with +// what discovery found in each. A member with no tests, or whose discovery +// failed, is not planned, as it never was: it reports its own result when its +// turn comes. A group that fails to plan is planned again member by member, +// so a member that fails to plan fails alone, and the members that plan are +// planned together again without it. +export void run_workspace_tests(std::span passthrough, + const BuildOverrides& base, + const TestOptions& testOpts, + const std::filesystem::path& wsRoot, + const std::vector>& groups, + std::vector members, + const WorkspaceTestHooks& hooks) { + const bool json = (testOpts.format == TestMessageFormat::Json); + if (json) mcpp::ui::set_quiet(true); + // The report covers the planning and the packages' own builds (Phase A); + // it is closed before the tests' own lines. + mcpp::build::progress::open(mcpp::log::is_verbose()); + + // What each member has to say when its turn comes, decided before any + // build: a member whose tests cannot be planned says why, and one with + // nothing to run says so. + struct Slot { + int session = -1; // the build that holds the member's tests + std::string owner; // the member's package name in that plan + std::string error; // it failed before its tests could run + std::string note; // it has no tests + bool noMatch = false; // no test matches the filter + bool packageFailed = false; + std::string packageOutput; // the diagnostics of its package's build + // Whether `packageOutput` was already printed, as the group's Phase A + // failure; a member's own failure that differs from it is printed at + // the member's turn, so a second broken member is not reported bare. + bool packageOutputShown = false; + }; + std::vector slots(members.size()); + std::map indexOf; + std::vector planned(members.size(), false); + for (std::size_t i = 0; i < members.size(); ++i) { + indexOf[members[i].path] = i; + if (!members[i].error.empty()) { slots[i].error = members[i].error; continue; } + if (members[i].targets.empty()) { slots[i].note = members[i].noTests; continue; } + if (!testOpts.filter.empty() + && std::ranges::none_of(members[i].targets, [&](auto const& t) { + return t.name.find(testOpts.filter) != std::string::npos; })) { + slots[i].noMatch = true; + continue; + } + planned[i] = true; + } + + std::vector> sessions; + std::vector> sessionMembers; + + // One plan of the given members, with each member's tests. + auto plan_session = [&](const std::vector& who) + -> std::expected, std::string> { + BuildOverrides mo = base; + mo.package_filter.clear(); + mo.project_root = wsRoot; + mo.will_run = true; + mo.workspace_members.clear(); + mo.workspace_request.clear(); + mo.member_targets.clear(); + for (auto i : who) { + mo.workspace_members.push_back(members[i].path); + mo.member_targets[members[i].path] = members[i].targets; + } + for (auto const& m : members) mo.workspace_request.push_back(m.path); + const auto t0 = std::chrono::steady_clock::now(); + auto prepared = prepare_build(/*print_fp=*/false, /*includeDevDeps=*/true, + /*extraTargets=*/{}, std::move(mo)); + if (!prepared) return std::unexpected(prepared.error()); + auto tb = std::make_unique(); + tb->prepareMs = std::chrono::duration_cast( + std::chrono::steady_clock::now() - t0).count(); + tb->ctx.emplace(std::move(*prepared)); + tb->backend = mcpp::build::make_ninja_backend(); + tb->ninjaLog = tb->ctx->outputDir / ".ninja_log"; + std::error_code ec; + tb->logFrom = std::filesystem::file_size(tb->ninjaLog, ec); + if (ec) tb->logFrom = 0; + return tb; + }; + auto attach = [&](std::unique_ptr tb, const std::vector& who) { + mcpp::build::progress::programs_done(); + const int id = static_cast(sessions.size()); + for (auto i : who) { + slots[i].session = id; + for (auto const& wm : tb->ctx->workspaceMembers) + if (wm.memberPath == members[i].path) slots[i].owner = wm.name; + // An empty owner selects every test binary of the plan, which is + // another member's as well: a member the plan does not name runs + // nothing. + if (slots[i].owner.empty()) { + slots[i].error = std::format( + "member '{}' is not among the members its plan holds", members[i].path); + slots[i].session = -1; + } + } + sessionMembers.push_back(who); + sessions.push_back(std::move(tb)); + }; + + for (auto const& g : groups) { + std::vector who; + for (auto const& mp : g) + if (auto it = indexOf.find(mp); it != indexOf.end() && planned[it->second]) + who.push_back(it->second); + if (who.empty()) continue; + auto tb = plan_session(who); + if (tb) { attach(std::move(*tb), who); continue; } + if (who.size() == 1) { slots[who.front()].error = tb.error(); continue; } + // The group did not plan, and the message does not say which member is + // to blame. Each member is planned alone to find out: the ones that + // plan are planned together again without the others, and the ones + // that do not are reported with their own reason. + std::vector survivors; + std::vector> alone; + for (auto i : who) { + auto one = plan_session({i}); + if (one) { survivors.push_back(i); alone.push_back(std::move(*one)); } + else slots[i].error = one.error(); + } + if (survivors.size() > 1) { + if (auto together = plan_session(survivors)) { + attach(std::move(*together), survivors); + continue; + } + } + for (std::size_t k = 0; k < survivors.size(); ++k) + attach(std::move(alone[k]), {survivors[k]}); + } + // The groups' compile databases are published once, as the union, below: + // a group's own build must not publish the root's as if it were the only + // one (`mcpp build` does the same). + if (sessions.size() > 1) + for (auto& s : sessions) s->ctx->plan.publishRootCompileDb = false; + + // Build every group: Phase A once, then one keep-going pass over the test + // goals of the members whose package built. + for (std::size_t s = 0; s < sessions.size(); ++s) { + auto& tb = *sessions[s]; + const auto& who = sessionMembers[s]; + test_announce(*tb.ctx); + if (auto a = test_phase_a(tb, testOpts, json)) { + // A failure of the package level is the package's fault, never N + // red tests. A group's Phase A stops at its first failure, and the + // failure says nothing of which member it belongs to: each + // member's own part of it is built alone, so that a member whose + // package builds still runs, and the one whose package does not is + // reported as failed, alone. + if (!json) { + if (!a->reported) mcpp::ui::error(a->message); + mcpp::ui::block(a->diagnosticOutput); + } + if (who.size() > 1) { + for (auto i : who) { + if (slots[i].session < 0) continue; + const auto goals = member_package_goals(*tb.ctx, slots[i].owner); + if (goals.empty()) continue; + mcpp::build::BuildOptions own; + own.ninjaTargets = goals; + own.buildTimeoutSecs = static_cast(testOpts.buildTimeoutSecs); + const auto t0 = std::chrono::steady_clock::now(); + auto r = tb.backend->build(tb.ctx->plan, own); + tb.buildMs += std::chrono::duration_cast( + std::chrono::steady_clock::now() - t0).count(); + if (!r) { + slots[i].packageFailed = true; + slots[i].packageOutput = r.error().diagnosticOutput; + slots[i].packageOutputShown = + r.error().diagnosticOutput == a->diagnosticOutput; + } + } + } + // Nothing pointed at a member (a failure outside every member's + // own objects), or the group is one member: they fail together. + if (std::ranges::none_of(who, [&](auto i) { return slots[i].packageFailed; })) + for (auto i : who) { + slots[i].packageFailed = true; + slots[i].packageOutput = a->diagnosticOutput; + slots[i].packageOutputShown = true; + } + } + // The test goals of the members that can run. + std::set runnable; + for (auto i : who) + if (!slots[i].packageFailed && slots[i].session >= 0) + runnable.insert(slots[i].owner); + test_bulk(tb, testOpts, [&](const mcpp::build::LinkUnit& lu) { + return lu.kind == mcpp::build::LinkUnit::TestBinary + && runnable.contains(lu.memberOf) + && (testOpts.filter.empty() + || lu.targetName.find(testOpts.filter) != std::string::npos); + }); + if (!runnable.empty()) test_resolve_runner(tb, testOpts, json); + } + // The tests' own lines follow, as they always have. + mcpp::build::progress::close(); + + if (sessions.size() > 1) { + std::vector dirs; + for (auto& s : sessions) dirs.push_back(s->ctx->outputDir); + publish_workspace_compile_commands(wsRoot, dirs); + } + + // One record or line per group, before its first member's tests: the + // members it built and the wall time of the build (member-selection design + // D6). The time is the group's and not any member's, so it is stated once; + // each member's summary names the group it waited for. + for (std::size_t s = 0; s < sessions.size(); ++s) { + auto& tb = *sessions[s]; + const auto& who = sessionMembers[s]; + if (json) { + std::string list; + for (auto i : who) { + if (!list.empty()) list += ','; + list += std::format("\"{}\"", test_json_escape(members[i].path)); + } + std::println("{{\"group_build\":{{\"group\":{},\"members\":[{}],\"build_ms\":{}}}}}", + s, list, tb.buildMs); + std::fflush(stdout); + continue; + } + std::string names; + for (auto i : who) names += (names.empty() ? "" : ", ") + members[i].path; + // The edges that took the build its time, for the question the per + // member split used to answer: which member's link, and not its tests, + // is slow. + auto edges = ninja_edges_since(tb.ninjaLog, tb.logFrom); + std::ranges::sort(edges, [](auto const& a, auto const& b) { return a.ms > b.ms; }); + std::string slowest; + for (std::size_t k = 0; k < edges.size() && k < 3; ++k) { + if (edges[k].ms < 1000) break; + slowest += std::format("{}{} {:.1f}s", slowest.empty() ? "" : ", ", + edges[k].output, static_cast(edges[k].ms) / 1000.0); + } + mcpp::ui::status("Workspace", std::format( + "{}built {} {} in {:.2f}s{}", + sessions.size() > 1 ? std::format("group {}/{} ", s + 1, sessions.size()) + : std::string{}, + who.size() == 1 ? "member" : "members", names, + static_cast(tb.buildMs) / 1000.0, + slowest.empty() ? std::string{} : std::format("; slowest: {}", slowest))); + } + + for (std::size_t i = 0; i < members.size(); ++i) { + auto& m = members[i]; + auto& slot = slots[i]; + if (hooks.begin && !hooks.begin(i, m.path)) continue; + TestRunSummary sum; + int rc = 0; + if (!slot.error.empty()) { + mcpp::ui::error(slot.error); + rc = 2; + } else if (slot.noMatch) { + if (json) + std::println("{{\"error\":\"no-tests-matched\",\"filter\":\"{}\"}}", + test_json_escape(testOpts.filter)); + mcpp::ui::error(std::format("no tests match '{}'", testOpts.filter)); + rc = 2; + } else if (!slot.note.empty()) { + if (!json) std::println("{}", slot.note); + } else if (slot.session >= 0) { + auto& tb = *sessions[static_cast(slot.session)]; + sum.buildGroup = slot.session; + sum.buildMs = tb.buildMs; + if (slot.packageFailed) { + sum.packageError = true; + sum.elapsedMs = tb.prepareMs + tb.buildMs; + if (json) + std::println("{{\"error\":\"package\",\"member\":\"{}\",\"compile_output\":\"{}\"}}", + test_json_escape(m.path), test_json_escape(slot.packageOutput)); + mcpp::ui::error(std::format( + "member '{}': its package did not build, so its tests did not run", m.path)); + if (!json && !slot.packageOutputShown) mcpp::ui::block(slot.packageOutput); + rc = 1; + } else { + rc = test_run_member(tb, testOpts, passthrough, json, m.path, slot.owner, + tb.prepareMs + tb.buildMs, sum); + } + } + if (hooks.end) hooks.end(i, m.path, rc, sum); + } +} + // `mcpp clean` driver. export int clean_project(bool wipe_bmi) { auto root = mcpp::project::find_manifest_root(std::filesystem::current_path()); @@ -3350,14 +3933,14 @@ export int clean_project(bool wipe_bmi) { std::println(stderr, "error: cannot remove target/: {}", ec.message()); return 1; } - std::println("Cleaned: {}", (*root / "target").string()); + mcpp::ui::line(std::format("Cleaned: {}", (*root / "target").string())); if (wipe_bmi) { auto cache = mcpp::toolchain::default_cache_root(); std::filesystem::remove_all(cache, ec); - std::println("Cleaned build cache: {}", cache.string()); - std::println(" (`mcpp cache clean --legacy` also removes the unused " - "pre-v1 cache, if any)"); + mcpp::ui::line(std::format("Cleaned build cache: {}", cache.string())); + mcpp::ui::line(" (`mcpp cache clean --legacy` also removes the unused " + "pre-v1 cache, if any)"); } return 0; } diff --git a/src/build/host_module_compile.cppm b/src/build/host_module_compile.cppm new file mode 100644 index 000000000..4aa6f5fa8 --- /dev/null +++ b/src/build/host_module_compile.cppm @@ -0,0 +1,622 @@ +// mcpp.build.host_module_compile — the bundled `mcpp` module and the host modules +// a build program imports, compiled once per key and kept by provenance (#748, B1). +// +// Split out of hostprogram.cppm, where `build_mcpp_module` and `build_host_module` +// compiled into each program's own directory. That file's header states why it +// must not grow (a clang 22 miscompile that followed the growth of the module it +// was in); the compile moved here, and the store it reads and writes is +// mcpp.build.host_module_store, which states what an entry is and where it lives. +// +// THE COMPILE IS UNCHANGED, AND ITS PLACE IS NOT. The argument vectors are the +// ones the two functions built, per compiler family, from the same BmiTraits and +// CommandDialect rows. What moved is the directory they run in and write to: a +// scratch directory of the store, where the files are made under the names the +// entry will have, and which is renamed into place when the compile succeeds. A +// consumer then names the entry's files: +// +// GCC finds BMIs by name under `/gcm.cache`, so a consumer is staged a +// copy of each BMI it imports in its own `gcm.cache`, as bmi_cache's +// consumers are. The compile of a host module is given the same staging +// in its scratch directory, so it finds the BMIs it imports the way its +// consumer will. +// Clang names a BMI with `-fmodule-file==`, and the path is the +// entry's. +// MSVC names it with `/reference =`. +// +// WHAT A KEY HOLDS. The store records the inputs and compares them field by field +// on a hit; this file decides what they are (`inputs_for`): the host compiler's +// identity, the standard flag, the flags of the compile, the flags that name the +// BMIs it imports and which BMIs those are, the digest of the text it compiles, +// and the provenance (the engine's version for the bundled module, the providing +// package for a host module). For GCC, whose use flags name nothing, the BMIs a +// compile can see are recorded by identity: an entry's key, or the std cache's. +// The compile of a host module is given every BMI made before it, in order, as +// it always was; that is the mechanism by which a rule can import another rule. +// +// A host module is compiled ALONE: its interface may import `std` and the bundled +// `mcpp` module and not a third package. A rule package is a leaf by construction. + +export module mcpp.build.host_module_compile; + +import std; +import mcpp.build.directives; // kProtocolVersion — the announced value has ONE source +import mcpp.build.host_module_store; +import mcpp.build.hostprogram; // the bundled module's text +import mcpp.libs.json; +import mcpp.log; +import mcpp.platform; +import mcpp.platform.process; +import mcpp.toolchain.dialect; +import mcpp.toolchain.fingerprint; // hash_file / hash_string +import mcpp.toolchain.hostflags; // bmi_reference_tokens +import mcpp.toolchain.model; +import mcpp.version; // MCPP_VERSION — the bundled module's text follows it + +export namespace mcpp::build { + +namespace fs = std::filesystem; + +// A BMI a compile can see: what GCC would find under `gcm.cache`, and what the +// other two families are told by flag. `identity` is what a dependent entry's +// key records for it: the key of the entry it came from, or, for `std`, the name +// of the std cache's directory (which is its own identity hash). +struct VisibleBmi { + std::string name; + fs::path path; + std::string identity; +}; + +// What one of these modules contributes to the build.mcpp compile. +struct McppModule { + std::vector useFlags; // how the consumer names the BMIs + fs::path object; // linked alongside build.mcpp + // `mcpp.core` (#734, E8): the same interface under the layer's name, a unit + // whose whole body re-exports `mcpp`. Empty for a host module. + fs::path aliasObject; + // The BMIs it made visible, for the compile after it. + std::vector bmis; + std::string entryKey; + bool reused = false; // found in the store, not compiled here + bool global = false; // kept in the global cache +}; + +// Where a build program's compiled imports may be kept. +struct ModuleStores { + // The global cache root; empty when `--cache local` or `off` asked for no + // global entry, in which case everything is kept in the workspace store. + fs::path cacheRoot; + // `/target/.build-mcpp/host-modules`. + fs::path workspaceStore; +}; + +// One host module to provide, and where its text came from (D4 of the plan). +struct HostModuleSource { + std::string logical; // the module name its source declares + fs::path interface; + // The provider's identity: the index, package and version for an index + // package, and the manifest's own name and version otherwise. + std::string index; + std::string package; + std::string version; + // True when the provider is an index package whose sources are in the + // immutable store. Only then may the entry be kept globally. + bool immutableSource = false; + // The provider's package root; the interface's directory when empty. + fs::path sourceRoot; +}; + +using CompileEnv = std::vector>; + +// The bundled `mcpp` module and its `mcpp.core` alias, for the build.mcpp +// compile in `consumerDir`. +std::expected +provide_mcpp_module(const ModuleStores& stores, const fs::path& consumerDir, + const fs::path& compiler, const std::vector& base, + const std::string& stdFlag, const mcpp::toolchain::Toolchain& tc, + const CompileEnv& env); + +// One host module. `useFlags` and `visible` are what the compile of the program +// would carry before this module: the flags that name the BMIs made so far, and +// those BMIs. +std::expected +provide_host_module(const ModuleStores& stores, const HostModuleSource& source, + const fs::path& consumerDir, const fs::path& compiler, + const std::vector& base, const std::string& stdFlag, + const mcpp::toolchain::Toolchain& tc, const CompileEnv& env, + const std::vector& useFlags, + const std::vector& visible); + +} // namespace mcpp::build + +namespace mcpp::build { + +namespace { + +namespace hm = mcpp::build::hostmods; + +constexpr int kStoreEpoch = 1; + +std::string join(const std::vector& argv) { + std::string joined; + for (auto const& a : argv) { if (!joined.empty()) joined += ' '; joined += a; } + return joined; +} + +// The host compiler's identity, as the std module's metadata states it: the +// compiler, its version, its driver (the declared identity, else a hash of the +// binary), the target, and the standard library. +// +// The driver's bytes are read once per path and process. A workspace of N +// programs asks N times and the binary is megabytes. +std::string driver_identity(const mcpp::toolchain::Toolchain& tc) { + if (!tc.driverIdent.empty()) return mcpp::toolchain::hash_string(tc.driverIdent); + if (tc.binaryPath.empty()) return {}; + static std::mutex m; + static std::map seen; + // UTF-8, never the code page: the name is an identity. + const auto u8 = tc.binaryPath.generic_u8string(); + const std::string key(reinterpret_cast(u8.data()), u8.size()); + std::lock_guard lock(m); + if (auto it = seen.find(key); it != seen.end()) return it->second; + return seen.emplace(key, mcpp::toolchain::hash_file(tc.binaryPath)).first->second; +} + +nlohmann::json toolchain_identity(const mcpp::toolchain::Toolchain& tc) { + return { + {"compiler", std::string(tc.compiler_name())}, + {"compiler_version", tc.version}, + {"driver_identity", driver_identity(tc)}, + {"target_triple", tc.targetTriple}, + {"stdlib", tc.stdlibId}, + {"stdlib_version", tc.stdlibVersion}, + }; +} + +// The inputs that hold for every entry of this file: what the compile is run +// with, apart from the text it compiles and where the text came from. +nlohmann::json common_inputs(std::string_view role, const mcpp::toolchain::Toolchain& tc, + const std::vector& base, const std::string& stdFlag, + const CompileEnv& env, const std::vector& use, + const std::vector& visible) { + nlohmann::json j; + j["epoch"] = kStoreEpoch; + j["role"] = std::string(role); + j["toolchain"] = toolchain_identity(tc); + j["std_flag"] = stdFlag; + j["base"] = base; + j["use"] = use; + // The toolchain's own environment (MSVC's INCLUDE and LIB) names the headers + // the compile reads, and two installs of one cl.exe can differ in it. + nlohmann::json envj = nlohmann::json::array(); + for (auto const& [k, v] : env) envj.push_back({k, v}); + j["env"] = std::move(envj); + nlohmann::json imports = nlohmann::json::array(); + for (auto const& b : visible) imports.push_back({{"module", b.name}, {"identity", b.identity}}); + j["imports"] = std::move(imports); + return j; +} + +// `-fmodule-file==` and `/reference =`; nothing for GCC, +// which finds the BMI by name under `gcm.cache`. +std::vector use_flags(const mcpp::toolchain::Toolchain& tc, std::string_view name, + const fs::path& bmi) { + if (tc.compiler == mcpp::toolchain::CompilerId::MSVC) + return mcpp::toolchain::bmi_reference_tokens(std::format(" /reference {}=", name), bmi); + if (mcpp::toolchain::is_clang(tc)) + return mcpp::toolchain::bmi_reference_tokens(std::format("-fmodule-file={}=", name), bmi); + return {"-fmodules"}; +} + +// What GCC finds by name: copies of the BMIs a compile imports, where the +// compile will look. A copy rather than a link, as the std module is staged, and +// for the same reason: a BMI that is rebuilt in place must not change under a +// compile that is reading it. +std::expected +stage_for_gcc(const fs::path& cwd, std::string_view bmiDir, + const std::vector>& bmis) { + std::error_code ec; + const fs::path dir = cwd / std::string(bmiDir); + fs::create_directories(dir, ec); + for (auto const& [file, from] : bmis) { + if (from.empty() || !fs::exists(from, ec)) continue; + fs::copy_file(from, dir / file, fs::copy_options::overwrite_existing, ec); + if (ec) + return std::unexpected(std::format("staging {} for a build program failed: {}", + file, ec.message())); + } + return {}; +} + +std::string substituted(std::string_view text, std::string_view placeholder, + std::string_view value) { + std::string out(text); + if (auto p = out.find(placeholder); p != std::string::npos) + out.replace(p, placeholder.size(), value); + return out; +} + +// The text of the bundled module and of its alias, as they are written and +// compiled. The placeholders stand for what the engine's own scanner would read +// as a second module of hostprogram.cppm (see kMcppModuleSource). +std::pair module_texts() { + auto mod = substituted(kMcppModuleSource, "@MODULE@", "export module"); + // Substituted rather than hardcoded so the announced version can never + // drift from the one the engine checks against. + mod = substituted(mod, "@PROTOCOL@", + std::to_string(mcpp::build::directives::kProtocolVersion)); + auto alias = substituted(kMcppCoreAliasSource, "@MODULE@", "export module"); + alias = substituted(alias, "@EXPORT@", "export"); + return {std::move(mod), std::move(alias)}; +} + +hm::Home home_for(const ModuleStores& stores, const mcpp::toolchain::Toolchain& tc, + bool global, std::string_view index, std::string_view package, + std::string_view version) { + const auto traits = mcpp::toolchain::bmi_traits(tc); + hm::Home h; + h.bmiDirName = std::string(traits.bmiDir); + h.manifestTag = std::string(traits.manifestPrefix); + if (global && !stores.cacheRoot.empty()) { + h.kind = hm::Home::Kind::Global; + h.cacheRoot = stores.cacheRoot; + h.index = std::string(index); + h.package = std::string(package); + h.version = std::string(version); + } else { + h.kind = hm::Home::Kind::Workspace; + h.workspaceStore = stores.workspaceStore; + } + return h; +} + +// Runs one compile of the store's producer. `what` names the step in the log +// and in the failure. The command is logged, always: it is the only place the +// argv of a SUCCESSFUL compile is observable (the `-isystem` rows that say +// whether a host toolchain's C library was attached, #622). +std::expected +run_step(std::string_view subject, const fs::path& cwd, const CompileEnv& env, + std::vector argv, const char* what) { + mcpp::log::verbose("buildmcpp-host", + std::format("{} {}: {}", subject, what, join(argv))); + auto r = mcpp::platform::process::capture_exec(argv, env, cwd.string()); + if (r.exit_code != 0) + return std::unexpected(std::format("{} {} failed (exit {}):\n{}", + subject, what, r.exit_code, r.output)); + return {}; +} + +// Moves what GCC wrote under `gcm.cache` to the entry's `bmi/`, and removes +// what is left there: the copies of the imported BMIs `stage_for_gcc` put in +// place for the compile (the std module's among them, tens of megabytes). They +// are inputs of the compile, not part of the entry, and an entry that kept them +// would carry a copy of `std` for every host module. +std::expected +collect_gcc_bmis(const fs::path& scratch, std::string_view bmiDir, + const std::vector& files) { + std::error_code ec; + for (auto const& f : files) { + fs::rename(scratch / std::string(bmiDir) / f, scratch / "bmi" / f, ec); + if (ec) + return std::unexpected(std::format("the compiler wrote no {}: {}", f, ec.message())); + } + fs::remove_all(scratch / std::string(bmiDir), ec); + return {}; +} + +} // namespace + +std::expected +provide_mcpp_module(const ModuleStores& stores, const fs::path& consumerDir, + const fs::path& compiler, const std::vector& base, + const std::string& stdFlag, const mcpp::toolchain::Toolchain& tc, + const CompileEnv& env) +{ + const auto traits = mcpp::toolchain::bmi_traits(tc); + const auto& dial = mcpp::toolchain::dialect_for(tc); + const bool msvc = tc.compiler == mcpp::toolchain::CompilerId::MSVC; + const bool clang = mcpp::toolchain::is_clang(tc); + const std::string ext(traits.bmiExt), obj(dial.objExt); + + const auto texts = module_texts(); + const std::string& moduleSrc = texts.first; + const std::string& aliasSrc = texts.second; + + // What the entry holds, under the names a consumer will use. + hm::Files files; + files.bmi = {"mcpp" + ext, "mcpp.core" + ext}; + files.obj = {"mcpp" + obj, "mcpp_core" + obj}; + + auto inputs = common_inputs("build-module", tc, base, stdFlag, env, {}, {}); + // The text the entry is compiled from: the module and its alias, exactly as + // written. The version is there too, for the reader of entry.json, and because + // the text follows it. + inputs["mcpp_version"] = std::string(mcpp::MCPP_VERSION); + inputs["interface_sha256"] = hm::sha256_hex(moduleSrc + '\x1f' + aliasSrc); + + // The bundled module is the engine's own text: identical in every project for + // one mcpp version and one host compiler, so it is kept globally (D4). + const auto home = home_for(stores, tc, /*global=*/true, "_engine", "mcpp-build-module", + std::string(mcpp::MCPP_VERSION)); + + auto entry = hm::obtain(home, inputs, files, + [&](const fs::path& S) -> std::expected { + constexpr std::string_view kSubject = "mcpp module"; + std::error_code ec; + { + std::ofstream os(S / "mcpp.cppm", std::ios::trunc); + os << moduleSrc; + if (!os) return std::unexpected(std::string("could not write mcpp module source")); + } + { + std::ofstream os(S / "mcpp_core.cppm", std::ios::trunc); + os << aliasSrc; + if (!os) return std::unexpected(std::string("could not write the mcpp.core unit")); + } + const fs::path bmiOut = S / "bmi", objOut = S / "obj"; + auto with_base = [&](std::vector head) { + for (auto const& b : base) head.push_back(b); + return head; + }; + auto step = [&](std::vector argv, const char* what) { + return run_step(kSubject, S, env, with_base(std::move(argv)), what); + }; + + if (msvc) { + // cl produces the .ifc and the .obj in one step. + const fs::path ifc = bmiOut / ("mcpp" + ext), coreIfc = bmiOut / ("mcpp.core" + ext); + const fs::path o1 = objOut / ("mcpp" + obj), o2 = objOut / ("mcpp_core" + obj); + std::vector argv{compiler.string()}; + for (auto f : dial.alwaysFlagsArgv) argv.emplace_back(f); + argv.push_back(stdFlag); + argv.push_back("/interface"); + for (auto f : dial.forceCxxLangArgv) argv.emplace_back(f); + argv.push_back(dial.compileOnly == std::string_view("/c") ? "/c" : "-c"); + argv.push_back("mcpp.cppm"); + argv.push_back("/ifcOutput"); argv.push_back(ifc.string()); + argv.push_back(std::string(dial.outputObjPrefix) + o1.string()); + if (auto r = step(std::move(argv), "compile"); !r) return r; + auto use = mcpp::toolchain::bmi_reference_tokens(" /reference mcpp=", ifc); + std::vector av{compiler.string()}; + for (auto f : dial.alwaysFlagsArgv) av.emplace_back(f); + av.push_back(stdFlag); + av.push_back("/interface"); + for (auto f : dial.forceCxxLangArgv) av.emplace_back(f); + av.push_back(dial.compileOnly == std::string_view("/c") ? "/c" : "-c"); + av.push_back("mcpp_core.cppm"); + av.push_back("/ifcOutput"); av.push_back(coreIfc.string()); + av.push_back(std::string(dial.outputObjPrefix) + o2.string()); + for (auto& f : use) av.push_back(f); + return step(std::move(av), "mcpp.core compile"); + } + + if (clang) { + const fs::path pcm = bmiOut / ("mcpp" + ext), corePcm = bmiOut / ("mcpp.core" + ext); + const fs::path o1 = objOut / ("mcpp" + obj), o2 = objOut / ("mcpp_core" + obj); + if (auto r = step({compiler.string(), stdFlag, "--precompile", + "mcpp.cppm", "-o", pcm.string()}, "precompile"); !r) return r; + if (auto r = step({compiler.string(), stdFlag, "-c", + pcm.string(), "-o", o1.string()}, "object"); !r) return r; + auto use = mcpp::toolchain::bmi_reference_tokens("-fmodule-file=mcpp=", pcm); + std::vector pre{compiler.string(), stdFlag, "--precompile", + "mcpp_core.cppm", "-o", corePcm.string()}; + for (auto& f : use) pre.push_back(f); + if (auto r = step(std::move(pre), "mcpp.core precompile"); !r) return r; + std::vector ob{compiler.string(), stdFlag, "-c", + corePcm.string(), "-o", o2.string()}; + for (auto& f : use) ob.push_back(f); + return step(std::move(ob), "mcpp.core object"); + } + + // GCC: BMIs are implicit under /gcm.cache, so nothing to name. + const fs::path o1 = objOut / ("mcpp" + obj), o2 = objOut / ("mcpp_core" + obj); + if (auto r = step({compiler.string(), stdFlag, "-fmodules", "-c", + "mcpp.cppm", "-o", o1.string()}, "compile"); !r) return r; + if (auto r = step({compiler.string(), stdFlag, "-fmodules", "-c", + "mcpp_core.cppm", "-o", o2.string()}, "mcpp.core compile"); !r) return r; + return collect_gcc_bmis(S, traits.bmiDir, files.bmi); + }); + if (!entry) return std::unexpected(entry.error()); + mcpp::log::verbose("buildmcpp-host", + std::format("bundled module mcpp: entry {} {} ({})", entry->key, + entry->global ? "global" : "workspace", + entry->reused ? "reused" : "compiled")); + + McppModule out; + out.entryKey = entry->key; + out.reused = entry->reused; + out.global = entry->global; + out.object = entry->obj(files.obj[0]); + out.aliasObject = entry->obj(files.obj[1]); + out.bmis = {{"mcpp", entry->bmi(files.bmi[0]), entry->key}, + {"mcpp.core", entry->bmi(files.bmi[1]), entry->key}}; + if (msvc || clang) { + out.useFlags = use_flags(tc, "mcpp", out.bmis[0].path); + for (auto& f : use_flags(tc, "mcpp.core", out.bmis[1].path)) out.useFlags.push_back(f); + } else { + out.useFlags = {"-fmodules"}; + if (auto r = stage_for_gcc(consumerDir, traits.bmiDir, + {{files.bmi[0], out.bmis[0].path}, + {files.bmi[1], out.bmis[1].path}}); !r) + return std::unexpected(r.error()); + } + return out; +} + +std::expected +provide_host_module(const ModuleStores& stores, const HostModuleSource& source, + const fs::path& consumerDir, const fs::path& compiler, + const std::vector& base, const std::string& stdFlag, + const mcpp::toolchain::Toolchain& tc, const CompileEnv& env, + const std::vector& useFlags, + const std::vector& visible) +{ + std::error_code ec; + if (!fs::exists(source.interface, ec)) { + return std::unexpected(std::format( + "host module '{}': no interface unit at {}\n" + " A package offering build rules must have a lib root " + "(src/.cppm or [lib] path).", + source.logical, source.interface.string())); + } + const auto traits = mcpp::toolchain::bmi_traits(tc); + const auto& dial = mcpp::toolchain::dialect_for(tc); + const bool msvc = tc.compiler == mcpp::toolchain::CompilerId::MSVC; + const bool clang = mcpp::toolchain::is_clang(tc); + const std::string ext(traits.bmiExt), objExt(dial.objExt); + + // A filesystem-safe stem. Partition separators and any path separator that + // sneaks into a logical name would otherwise create directories that do + // not exist. Dots are left ALONE on purpose: `a.b.rules.o` is a legal + // filename, GCC's own gcm.cache uses the dotted module name verbatim, and + // rewriting them would make the object name disagree with the BMI name for + // no gain. + std::string stem(source.logical); + for (auto& c : stem) if (c == ':' || c == '/' || c == '\\') c = '-'; + + hm::Files files; + files.bmi = {stem + ext}; + files.obj = {stem + objExt}; + + const auto digest = hm::sha256_file(source.interface); + if (digest.empty()) + return std::unexpected(std::format("host module '{}': cannot read {}", + source.logical, source.interface.string())); + + auto inputs = common_inputs("host-module", tc, base, stdFlag, env, useFlags, visible); + inputs["module"] = source.logical; + inputs["interface_sha256"] = digest; + inputs["provider"] = {{"index", source.index}, {"name", source.package}, + {"version", source.version}}; + // A package whose sources can change in place: what the interface includes is + // part of what was compiled, and it may sit anywhere in the package, so the + // package's tree is part of the key. + const bool global = source.immutableSource && !stores.cacheRoot.empty(); + inputs["source"] = global ? "immutable" : "workspace"; + if (!global) { + auto tree = hm::tree_digest(source.sourceRoot.empty() ? source.interface.parent_path() + : source.sourceRoot); + inputs["source_tree"] = tree.complete ? tree.hex : ("unbounded:" + hm::process_nonce()); + } + + const auto home = home_for(stores, tc, global, source.index, source.package, source.version); + + // The interface's LANGUAGE, stated rather than inferred from its extension. + // + // Measured on macOS CI: a rule package whose lib root is `rulepkg.ixx` + // made `clang++ --precompile rulepkg.ixx -o rulepkg.pcm` EXIT 0 AND WRITE + // NOTHING — clang's driver does not recognise `.ixx`, so it treated the file + // as a linker input, warned that it was unused, and succeeded. The failure + // surfaced one step later as `no such file or directory: …/rulepkg.pcm`, + // naming an output rather than the input that was never read. + // + // Every other module compile in mcpp already says this (BmiTraits:: + // moduleInterfaceLangFlag — `/interface /TP`, `-x c++-module`, `-x c++`); + // the host-module path was the one place that still let the driver guess. + // It is positional on GNU-style drivers, so it goes immediately before the + // input. + std::vector langArgv; + { + std::string_view lang = traits.moduleInterfaceLangFlag; + for (std::size_t i = 0; i < lang.size(); ) { + while (i < lang.size() && lang[i] == ' ') ++i; + auto j = lang.find(' ', i); + if (j == std::string_view::npos) j = lang.size(); + if (j > i) langArgv.emplace_back(lang.substr(i, j - i)); + i = j; + } + } + + auto entry = hm::obtain(home, inputs, files, + [&](const fs::path& S) -> std::expected { + const std::string subject = std::format("host module '{}'", source.logical); + const fs::path bmiOut = S / "bmi", objOut = S / "obj"; + auto with_base = [&](std::vector head) { + for (auto const& b : base) head.push_back(b); + for (auto const& f : useFlags) head.push_back(f); + return head; + }; + auto step = [&](std::vector argv, const char* what) { + return run_step(subject, S, env, with_base(std::move(argv)), what); + }; + const fs::path o = objOut / files.obj[0]; + + if (msvc) { + const fs::path ifc = bmiOut / files.bmi[0]; + std::vector argv{compiler.string()}; + for (auto f : dial.alwaysFlagsArgv) argv.emplace_back(f); + argv.push_back(stdFlag); + argv.push_back("/interface"); + for (auto f : dial.forceCxxLangArgv) argv.emplace_back(f); + argv.push_back("/c"); + argv.push_back(source.interface.string()); + argv.push_back("/ifcOutput"); argv.push_back(ifc.string()); + argv.push_back(std::string(dial.outputObjPrefix) + o.string()); + return step(std::move(argv), "compile"); + } + + if (clang) { + const fs::path pcm = bmiOut / files.bmi[0]; + std::vector pre{compiler.string(), stdFlag, "--precompile"}; + for (auto const& l : langArgv) pre.push_back(l); + pre.push_back(source.interface.string()); + pre.push_back("-o"); pre.push_back(pcm.string()); + if (auto r = step(std::move(pre), "precompile"); !r) return r; + // The precompile can succeed and write nothing when the driver + // ignored the input, which is exactly what happened above. + // Checked here so the diagnostic names the interface rather than + // a missing output. + std::error_code e2; + if (!fs::exists(pcm, e2)) { + return std::unexpected(std::format( + "host module '{}': the compiler accepted '{}' and produced no " + "BMI.\n" + " The interface's language is passed explicitly, so this " + "is not an extension\n" + " the driver failed to recognise — check that the file " + "really is a module interface.", + source.logical, source.interface.string())); + } + return step({compiler.string(), stdFlag, "-c", pcm.string(), "-o", o.string()}, + "object"); + } + + // GCC: BMIs are implicit under /gcm.cache, so nothing to name — + // which is also why the compile has to find every BMI it imports + // there. + { + std::vector> staged; + for (auto const& b : visible) + staged.emplace_back(b.path.filename().string(), b.path); + if (auto r = stage_for_gcc(S, traits.bmiDir, staged); !r) return r; + } + std::vector gccArgv{compiler.string(), stdFlag, "-fmodules", "-c"}; + for (auto const& l : langArgv) gccArgv.push_back(l); + gccArgv.push_back(source.interface.string()); + gccArgv.push_back("-o"); gccArgv.push_back(o.string()); + if (auto r = step(std::move(gccArgv), "compile"); !r) return r; + return collect_gcc_bmis(S, traits.bmiDir, files.bmi); + }); + if (!entry) return std::unexpected(entry.error()); + mcpp::log::verbose("buildmcpp-host", + std::format("host module '{}': entry {} {} ({})", source.logical, entry->key, + entry->global ? "global" : "workspace", + entry->reused ? "reused" : "compiled")); + + McppModule out; + out.entryKey = entry->key; + out.reused = entry->reused; + out.global = entry->global; + out.object = entry->obj(files.obj[0]); + out.bmis = {{source.logical, entry->bmi(files.bmi[0]), entry->key}}; + if (msvc || clang) { + out.useFlags = use_flags(tc, source.logical, out.bmis[0].path); + } else { + out.useFlags = {"-fmodules"}; + if (auto r = stage_for_gcc(consumerDir, traits.bmiDir, + {{files.bmi[0], out.bmis[0].path}}); !r) + return std::unexpected(r.error()); + } + return out; +} + +} // namespace mcpp::build diff --git a/src/build/host_module_store.cppm b/src/build/host_module_store.cppm new file mode 100644 index 000000000..268d9dcf9 --- /dev/null +++ b/src/build/host_module_store.cppm @@ -0,0 +1,381 @@ +// mcpp.build.host_module_store — where what a build program imports is kept once +// it is compiled (#748, B1). +// +// A build program that imports `mcpp`, or a host module of a rule package, needs +// that module's BMI and object before its own compile. Until this module, each +// program compiled them into its own directory, before its own compile and +// whether or not an identical copy sat in the directory next door: four members +// that import one host module compiled the `mcpp` module four times and the host +// module four times. #748 measured 7.8 s per program outside the program's own +// `ran`, repeated for every program and every invocation. +// +// AN ENTRY IS ADDRESSED BY ITS INPUTS, AND THE INPUTS ARE RECORDED. The key is a +// hash of everything that reaches the compile: the host compiler's identity, the +// standard flag, the flags the compile carries, the BMIs it imports, and the text +// it compiles. The same inputs are written to entry.json, and a hit compares +// them field by field (mcpp.bmi_cache::probe_cached), never the hash alone. The +// flags of one compile are the flags of every consumer of its BMI, so a BMI is +// shared only between compiles that agree with it (hostprogram.cppm, P6): the +// agreement is a consequence of the key and is not checked afterwards. +// +// WHERE AN ENTRY LIVES DEPENDS ON WHERE ITS TEXT CAME FROM, and on nothing else. +// +// Global the text comes from the engine (the bundled `mcpp` module) or from +// an index package whose sources sit in the immutable store. The +// same name and version are the same bytes, so one entry serves every +// project of this machine. It lives in the global cache, in the +// layout of a dependency's entry, so `mcpp cache gc` collects it. +// Workspace the text comes from a path or git dependency or from a workspace +// member. Its sources can change without its name and version +// changing, which is the rule the dependency cache applies +// (plan.cpp, the admission of a package to the cache), so it is kept +// under the workspace's `target/` and is never written to the global +// cache. +// +// A host module is compiled ALONE, against `std` and `mcpp` only, so the local +// taint the dependency cache walks the closure for is, for a host module, its +// own package's alone. +// +// AN ENTRY IS PUBLISHED WHOLE. It is compiled into a staging directory and +// renamed into place with entry.json written last (mcpp.bmi_cache:: +// publish_staged), so it is never seen half-written. Threads of one process that +// want one key take one lock, so the key is compiled once, and the others find +// it; two processes that race both compile, and the second to publish finds the +// first's entry in place and discards its own. + +export module mcpp.build.host_module_store; + +import std; +import mcpp.bmi_cache; +import mcpp.libs.json; +import mcpp.toolchain.fingerprint; // hash_string — the one key hash of the build cache + +export namespace mcpp::build::hostmods { + +namespace fs = std::filesystem; + +// ─── SHA-256 ───────────────────────────────────────────────────────────── +// +// The digest of the text an entry was compiled from. The build's other content +// identities are 64-bit FNV-1a, which answers "has this changed"; an entry that +// other projects and other users of the machine reuse is held to "is this the +// same text" and records a digest that answers that. +std::string sha256_hex(std::string_view bytes); +// Empty when the file cannot be read: an unreadable file has no identity, and a +// caller that keyed on the empty string would key every unreadable file alike. +std::string sha256_file(const fs::path& p); + +// The digest of every regular file below `dir`: the relative name (UTF-8) and +// the content of each, in name order. A host module is compiled from one +// interface file, and the file may include what sits beside it; for a package +// whose sources can change in place, the interface's digest alone would not see +// an edit to an included file. `target`, `.git` and `.mcpp` are not entered. +// +// A tree of more than `kTreeFileLimit` files, or of more than `kTreeByteLimit` +// bytes, is not digested: `complete` is false, and the caller keys the entry on +// something that cannot match a later invocation, so the entry is reused only +// within this one. +inline constexpr std::size_t kTreeFileLimit = 4096; +inline constexpr std::uintmax_t kTreeByteLimit = 64ull * 1024 * 1024; +struct TreeDigest { + std::string hex; + bool complete = true; +}; +TreeDigest tree_digest(const fs::path& dir); + +// A stand-in identity that matches within this process and nowhere else. +std::string process_nonce(); + +// ─── Where an entry lives ──────────────────────────────────────────────── + +struct Home { + enum class Kind { Global, Workspace }; + Kind kind = Kind::Workspace; + + // Global: the cache root (mcpp::home::cache_root()) and the package address + // below it, `pkg//@//`. + fs::path cacheRoot; + std::string index; + std::string package; + std::string version; + + // Workspace: `/target/.build-mcpp/host-modules`; an entry is + // `//`. + fs::path workspaceStore; + + // What entry.json records about the BMI family, as for a dependency's entry. + std::string bmiDirName = "gcm.cache"; + std::string manifestTag = "gcm"; +}; + +// What an entry holds: file names below its `bmi/` and `obj/`. +struct Files { + std::vector bmi; + std::vector obj; +}; + +struct Entry { + fs::path dir; + std::string key; // 16 hex; what a dependent entry records as its import + bool reused = false; // found, and not compiled by this call + bool global = false; + + fs::path bmi(std::string_view name) const { return dir / "bmi" / std::string(name); } + fs::path obj(std::string_view name) const { return dir / "obj" / std::string(name); } +}; + +// Writes `files` into `scratch/bmi` and `scratch/obj`. The scratch directory is +// the producer's own, and is gone when the call returns. +using Producer = std::function(const fs::path& scratch)>; + +// The entry for `inputs` in `home`: found, or produced once and published. +// +// `inputs` is the complete description of what `produce` will compile. Two calls +// whose inputs are equal get one entry, and `produce` runs at most once for them +// in this process. `files` names what it writes. +std::expected +obtain(const Home& home, const nlohmann::json& inputs, const Files& files, + const Producer& produce); + +// The key `obtain` derives for `inputs`. +std::string key_of(const nlohmann::json& inputs); + +} // namespace mcpp::build::hostmods + +namespace mcpp::build::hostmods { + +// ─── SHA-256 (FIPS 180-4) ──────────────────────────────────────────────── + +namespace { + +constexpr std::array kSha256K = { + 0x428a2f98u, 0x71374491u, 0xb5c0fbcfu, 0xe9b5dba5u, 0x3956c25bu, 0x59f111f1u, + 0x923f82a4u, 0xab1c5ed5u, 0xd807aa98u, 0x12835b01u, 0x243185beu, 0x550c7dc3u, + 0x72be5d74u, 0x80deb1feu, 0x9bdc06a7u, 0xc19bf174u, 0xe49b69c1u, 0xefbe4786u, + 0x0fc19dc6u, 0x240ca1ccu, 0x2de92c6fu, 0x4a7484aau, 0x5cb0a9dcu, 0x76f988dau, + 0x983e5152u, 0xa831c66du, 0xb00327c8u, 0xbf597fc7u, 0xc6e00bf3u, 0xd5a79147u, + 0x06ca6351u, 0x14292967u, 0x27b70a85u, 0x2e1b2138u, 0x4d2c6dfcu, 0x53380d13u, + 0x650a7354u, 0x766a0abbu, 0x81c2c92eu, 0x92722c85u, 0xa2bfe8a1u, 0xa81a664bu, + 0xc24b8b70u, 0xc76c51a3u, 0xd192e819u, 0xd6990624u, 0xf40e3585u, 0x106aa070u, + 0x19a4c116u, 0x1e376c08u, 0x2748774cu, 0x34b0bcb5u, 0x391c0cb3u, 0x4ed8aa4au, + 0x5b9cca4fu, 0x682e6ff3u, 0x748f82eeu, 0x78a5636fu, 0x84c87814u, 0x8cc70208u, + 0x90befffau, 0xa4506cebu, 0xbef9a3f7u, 0xc67178f2u, +}; + +constexpr std::uint32_t rotr(std::uint32_t x, int n) { return (x >> n) | (x << (32 - n)); } + +class Sha256 { +public: + void update(const unsigned char* data, std::size_t len) { + total_ += len; + while (len > 0) { + const std::size_t take = std::min(len, 64 - fill_); + std::memcpy(block_.data() + fill_, data, take); + fill_ += take; data += take; len -= take; + if (fill_ == 64) { compress(); fill_ = 0; } + } + } + + std::string finish() { + const std::uint64_t bits = total_ * 8; + const unsigned char one = 0x80; + update(&one, 1); + const unsigned char zero = 0; + while (fill_ != 56) update(&zero, 1); + unsigned char len[8]; + for (int i = 0; i < 8; ++i) len[i] = static_cast(bits >> (56 - 8 * i)); + update(len, 8); + std::string out; + out.reserve(64); + for (auto w : h_) + for (int i = 3; i >= 0; --i) + out += std::format("{:02x}", static_cast((w >> (8 * i)) & 0xffu)); + return out; + } + +private: + void compress() { + std::array w{}; + for (int i = 0; i < 16; ++i) + w[i] = (std::uint32_t{block_[4 * i]} << 24) | (std::uint32_t{block_[4 * i + 1]} << 16) + | (std::uint32_t{block_[4 * i + 2]} << 8) | std::uint32_t{block_[4 * i + 3]}; + for (int i = 16; i < 64; ++i) { + const auto s0 = rotr(w[i - 15], 7) ^ rotr(w[i - 15], 18) ^ (w[i - 15] >> 3); + const auto s1 = rotr(w[i - 2], 17) ^ rotr(w[i - 2], 19) ^ (w[i - 2] >> 10); + w[i] = w[i - 16] + s0 + w[i - 7] + s1; + } + auto [a, b, c, d, e, f, g, h] = h_; + for (int i = 0; i < 64; ++i) { + const auto S1 = rotr(e, 6) ^ rotr(e, 11) ^ rotr(e, 25); + const auto ch = (e & f) ^ (~e & g); + const auto t1 = h + S1 + ch + kSha256K[i] + w[i]; + const auto S0 = rotr(a, 2) ^ rotr(a, 13) ^ rotr(a, 22); + const auto maj = (a & b) ^ (a & c) ^ (b & c); + const auto t2 = S0 + maj; + h = g; g = f; f = e; e = d + t1; d = c; c = b; b = a; a = t1 + t2; + } + h_[0] += a; h_[1] += b; h_[2] += c; h_[3] += d; + h_[4] += e; h_[5] += f; h_[6] += g; h_[7] += h; + } + + std::array h_ = { + 0x6a09e667u, 0xbb67ae85u, 0x3c6ef372u, 0xa54ff53au, + 0x510e527fu, 0x9b05688cu, 0x1f83d9abu, 0x5be0cd19u, + }; + std::array block_{}; + std::size_t fill_ = 0; + std::uint64_t total_ = 0; +}; + +// The file name as UTF-8, never through the code page (the path narrowing rule: +// a name that is an identity is `u8string()`). +std::string utf8(const fs::path& p) { + const auto u8 = p.generic_u8string(); + return std::string(reinterpret_cast(u8.data()), u8.size()); +} + +} // namespace + +std::string sha256_hex(std::string_view bytes) { + Sha256 h; + h.update(reinterpret_cast(bytes.data()), bytes.size()); + return h.finish(); +} + +std::string sha256_file(const fs::path& p) { + std::ifstream is(p, std::ios::binary); + if (!is) return {}; + Sha256 h; + std::array buf{}; + while (is.read(buf.data(), buf.size()) || is.gcount() > 0) + h.update(reinterpret_cast(buf.data()), + static_cast(is.gcount())); + return h.finish(); +} + +std::string process_nonce() { + static const std::string nonce = [] { + std::random_device rd; + return std::format("{:08x}{:08x}", rd(), rd()); + }(); + return nonce; +} + +TreeDigest tree_digest(const fs::path& dir) { + TreeDigest out; + std::vector> rows; // (relative name, file digest) + std::uintmax_t bytes = 0; + std::error_code ec; + fs::recursive_directory_iterator it(dir, fs::directory_options::skip_permission_denied, ec); + if (ec) { out.complete = false; return out; } + for (; it != fs::recursive_directory_iterator(); it.increment(ec)) { + if (ec) { out.complete = false; break; } + const auto name = utf8(it->path().filename()); + std::error_code tec; + if (it->is_directory(tec)) { + if (name == "target" || name == ".git" || name == ".mcpp") it.disable_recursion_pending(); + continue; + } + if (!it->is_regular_file(tec)) continue; + if (rows.size() >= kTreeFileLimit) { out.complete = false; break; } + if (const auto size = it->file_size(tec); !tec) bytes += size; + if (bytes > kTreeByteLimit) { out.complete = false; break; } + rows.emplace_back(utf8(it->path().lexically_relative(dir)), sha256_file(it->path())); + } + std::ranges::sort(rows); + Sha256 h; + for (auto const& [name, digest] : rows) { + const auto line = std::format("{}\t{}\n", name, digest); + h.update(reinterpret_cast(line.data()), line.size()); + } + out.hex = h.finish(); + return out; +} + +// ─── The entries ───────────────────────────────────────────────────────── + +std::string key_of(const nlohmann::json& inputs) { + // `dump()` is canonical: an object's members are kept in key order. + return mcpp::toolchain::hash_string("mcpp-host-module-store-v1\x1f" + inputs.dump()); +} + +namespace { + +mcpp::bmi_cache::CacheKey cache_key_of(const Home& home, const nlohmann::json& inputs) { + mcpp::bmi_cache::CacheKey ck; + ck.keyHex = key_of(inputs); + ck.inputs = inputs; + ck.bmiDirName = home.bmiDirName; + ck.manifestTag = home.manifestTag; + if (home.kind == Home::Kind::Global) { + ck.cacheRoot = home.cacheRoot; + ck.indexName = home.index; + ck.packageName = home.package; + ck.version = home.version; + } else { + ck.cacheRoot = home.workspaceStore; + ck.directDir = home.workspaceStore / ck.keyHex; + ck.indexName = "workspace"; + ck.packageName = "host-modules"; + ck.version = ""; + } + return ck; +} + +mcpp::bmi_cache::DepArtifacts artifacts_of(const Files& files) { + mcpp::bmi_cache::DepArtifacts a; + a.bmiFiles = files.bmi; + for (auto const& o : files.obj) a.objFiles.push_back({o, {}}); + return a; +} + +// One lock per entry address, for the threads of this process. A lock is never +// removed: there are a handful of entries, and a thread still waiting on a +// removed lock would be waiting on a destroyed one. +std::mutex& lock_for(const fs::path& dir) { + static std::mutex mapMutex; + static std::map> locks; + std::lock_guard guard(mapMutex); + auto& slot = locks[utf8(dir)]; + if (!slot) slot = std::make_unique(); + return *slot; +} + +} // namespace + +std::expected +obtain(const Home& home, const nlohmann::json& inputs, const Files& files, + const Producer& produce) +{ + const auto ck = cache_key_of(home, inputs); + const auto want = artifacts_of(files); + Entry entry; + entry.dir = ck.dir(); + entry.key = ck.keyHex; + entry.global = home.kind == Home::Kind::Global; + + // One thread at a time per address: the thread that finds nothing compiles, + // and the threads behind it find the entry. + std::lock_guard guard(lock_for(entry.dir)); + + if (mcpp::bmi_cache::probe_cached(ck, want).ok) { + mcpp::bmi_cache::touch_accessed(ck); + entry.reused = true; + return entry; + } + + auto staged = mcpp::bmi_cache::stage_entry(ck); + if (!staged) return std::unexpected(staged.error()); + if (auto r = produce(*staged); !r) { + std::error_code ec; + fs::remove_all(*staged, ec); + return std::unexpected(r.error()); + } + auto placed = mcpp::bmi_cache::publish_staged(ck, *staged, want); + if (!placed) return std::unexpected(placed.error()); + // Another process published first: its entry is the one in place. + entry.reused = !*placed; + return entry; +} + +} // namespace mcpp::build::hostmods diff --git a/src/build/hostprogram.cppm b/src/build/hostprogram.cppm index 8c5a05703..0f773fdfb 100644 --- a/src/build/hostprogram.cppm +++ b/src/build/hostprogram.cppm @@ -9,17 +9,16 @@ // solid, and the cheap response is to stop growing that namespace. // // See .agents/docs/2026-08-02-host-compile-single-producer-design.md §6.2. +// +// WHAT IS LEFT HERE IS THE TEXT, AND THE COMPILE OF IT IS ELSEWHERE. The bundled +// module's source and the question "does this program import that module" stay; +// `build_mcpp_module` and `build_host_module`, which compiled the module and the +// host modules into each program's own directory, moved to +// mcpp.build.host_module_compile (#748), which keeps them in a store by key. export module mcpp.build.hostprogram; import std; -import mcpp.build.directives; // kProtocolVersion — the announced value has ONE source -import mcpp.log; -import mcpp.platform; -import mcpp.platform.process; -import mcpp.toolchain.dialect; -import mcpp.toolchain.hostflags; -import mcpp.toolchain.model; export namespace mcpp::build { @@ -860,11 +859,6 @@ static ProtocolAnnouncer mcpp_protocol_announcer; } )CPP"; -// Compile the bundled `mcpp` module into `bdir` and return the extra flags the -// build.mcpp compile needs to import it (the object `mcpp.o` is linked alongside). -// GCC : -fmodules → gcm.cache/mcpp.gcm + mcpp.o; build.mcpp compiles from -// `bdir` (cwd) so GCC finds gcm.cache/mcpp.gcm. -// Clang : --precompile → mcpp.pcm, then -c → mcpp.o; pass -fmodule-file=mcpp=. // Does the source contain `import ;`? // // A plain substring search is not enough here: "import std" is a prefix of @@ -894,15 +888,6 @@ bool imports_module(std::string_view src, std::string_view name) { } -// What the bundled `mcpp` module contributes to the build.mcpp compile. -struct McppModule { - std::vector useFlags; // how the consumer names the BMI - fs::path object; // linked alongside build.mcpp - // `mcpp.core` (#734, E8): the same interface under the layer's name, a unit - // whose whole body re-exports `mcpp`. Empty for a host module. - fs::path aliasObject; -}; - // The unit that makes `import mcpp.core;` and `import mcpp;` name one // interface. Written beside `mcpp.cppm` and compiled after it, so either // spelling -- or both in one program -- reaches the same symbols. Placeholders @@ -911,300 +896,4 @@ struct McppModule { inline constexpr std::string_view kMcppCoreAliasSource = "@MODULE@ mcpp.core;\n@EXPORT@ import mcpp;\n"; -// Compile ONE dependency-provided module interface for the host, into `bdir`, -// with the SAME flags build.mcpp itself gets. Returns how to name its BMI plus -// the object to link. -// -// Shares build_mcpp_module's per-family dispatch deliberately: a BMI is only -// usable by a compile that agrees with it on standard, dialect and compiler -// identity, and the cheapest way to guarantee that is to produce both from one -// set of flags rather than to check afterwards. -// -// Limitation, stated rather than hidden: the interface is compiled ALONE, so it -// may import `std` and the bundled `mcpp` module but not a third package. A -// rule package is a leaf by construction; a transitive host module graph would -// need the sub-build machinery and its own BMI-agreement story. -std::expected -build_host_module(const fs::path& bdir, const fs::path& compiler, - const std::vector& base, const std::string& stdFlag, - const mcpp::toolchain::Toolchain& tc, - const std::vector>& env, - std::string_view logicalName, const fs::path& interfacePath, - const std::vector& extraUseFlags); - -std::expected -build_mcpp_module(const fs::path& bdir, const fs::path& compiler, - const std::vector& base, const std::string& stdFlag, - const mcpp::toolchain::Toolchain& tc, - const std::vector>& env) { - std::error_code ec; - fs::path cppm = bdir / "mcpp.cppm"; - std::string moduleSrc(kMcppModuleSource); - if (auto p = moduleSrc.find("@MODULE@"); p != std::string::npos) - moduleSrc.replace(p, std::string_view("@MODULE@").size(), "export module"); - // Substituted rather than hardcoded so the announced version can never - // drift from the one the engine checks against. - if (auto p = moduleSrc.find("@PROTOCOL@"); p != std::string::npos) - moduleSrc.replace(p, std::string_view("@PROTOCOL@").size(), - std::to_string(mcpp::build::directives::kProtocolVersion)); - { std::ofstream os(cppm, std::ios::trunc); - os << moduleSrc; - if (!os) return std::unexpected(std::string("could not write mcpp module source")); } - { - std::string aliasSrc(kMcppCoreAliasSource); - if (auto p = aliasSrc.find("@MODULE@"); p != std::string::npos) - aliasSrc.replace(p, std::string_view("@MODULE@").size(), "export module"); - if (auto p = aliasSrc.find("@EXPORT@"); p != std::string::npos) - aliasSrc.replace(p, std::string_view("@EXPORT@").size(), "export"); - std::ofstream os(bdir / "mcpp_core.cppm", std::ios::trunc); - os << aliasSrc; - if (!os) return std::unexpected(std::string("could not write the mcpp.core unit")); - } - - auto run = [&](std::vector argv, const char* what) - -> std::expected { - // THE ONLY PLACE THIS ARGV IS EVER OBSERVABLE ON A SUCCESSFUL RUN. - // A failing compile shows its own command implicitly (the compiler's - // diagnostics name the headers it looked for and did not find); a - // passing one otherwise leaves no trace of which `-isystem` rows it - // carried — which is exactly the fact that distinguishes a host - // toolchain whose C library was attached from one whose wasn't (#622, - // `host_tc_for_build_program`'s cross branch). `verbose()` always logs - // it; MCPP_VERBOSE=1 additionally echoes it to stderr. - std::string joined; - for (auto const& a : argv) { if (!joined.empty()) joined += ' '; joined += a; } - mcpp::log::verbose("buildmcpp-host", - std::format("mcpp module {}: {}", what, joined)); - auto r = mcpp::platform::process::capture_exec(argv, env, bdir.string()); - if (r.exit_code != 0) - return std::unexpected(std::format("mcpp module {} failed (exit {}):\n{}", - what, r.exit_code, r.output)); - return {}; - }; - auto with_base = [&](std::vector head) { - for (auto& b : base) head.push_back(b); - return head; - }; - - // Dispatch on the SAME module table the main build uses (BmiTraits + - // CommandDialect), not on a local is_clang/else. That is what makes a - // toolchain family work here as soon as it works there — adding cl.exe - // needed no new pipeline, only this row. - const auto traits = mcpp::toolchain::bmi_traits(tc); - const auto& dial = mcpp::toolchain::dialect_for(tc); - McppModule out; - - if (tc.compiler == mcpp::toolchain::CompilerId::MSVC) { - // cl produces the .ifc and the .obj in one step. - fs::path ifc = bdir / ("mcpp" + std::string(traits.bmiExt)); - out.object = bdir / ("mcpp" + std::string(dial.objExt)); - std::vector argv{compiler.string()}; - for (auto f : dial.alwaysFlagsArgv) argv.emplace_back(f); - argv.push_back(stdFlag); - argv.push_back("/interface"); - for (auto f : dial.forceCxxLangArgv) argv.emplace_back(f); - argv.push_back(dial.compileOnly == std::string_view("/c") ? "/c" : "-c"); - argv.push_back("mcpp.cppm"); - argv.push_back("/ifcOutput"); argv.push_back(ifc.string()); - argv.push_back(std::string(dial.outputObjPrefix) + out.object.string()); - if (auto r = run(with_base(std::move(argv)), "compile"); !r) - return std::unexpected(r.error()); - out.useFlags = mcpp::toolchain::bmi_reference_tokens(" /reference mcpp=", ifc); - fs::path coreIfc = bdir / ("mcpp.core" + std::string(traits.bmiExt)); - out.aliasObject = bdir / ("mcpp_core" + std::string(dial.objExt)); - std::vector av{compiler.string()}; - for (auto f : dial.alwaysFlagsArgv) av.emplace_back(f); - av.push_back(stdFlag); - av.push_back("/interface"); - for (auto f : dial.forceCxxLangArgv) av.emplace_back(f); - av.push_back(dial.compileOnly == std::string_view("/c") ? "/c" : "-c"); - av.push_back("mcpp_core.cppm"); - av.push_back("/ifcOutput"); av.push_back(coreIfc.string()); - av.push_back(std::string(dial.outputObjPrefix) + out.aliasObject.string()); - for (auto& f : out.useFlags) av.push_back(f); - if (auto r = run(with_base(std::move(av)), "mcpp.core compile"); !r) - return std::unexpected(r.error()); - for (auto& f : mcpp::toolchain::bmi_reference_tokens(" /reference mcpp.core=", coreIfc)) - out.useFlags.push_back(f); - return out; - } - - out.object = bdir / ("mcpp" + std::string(dial.objExt)); - if (mcpp::toolchain::is_clang(tc)) { - fs::path pcm = bdir / ("mcpp" + std::string(traits.bmiExt)); - if (auto r = run(with_base({compiler.string(), stdFlag, "--precompile", - "mcpp.cppm", "-o", pcm.string()}), "precompile"); !r) - return std::unexpected(r.error()); - if (auto r = run(with_base({compiler.string(), stdFlag, "-c", - pcm.string(), "-o", out.object.string()}), "object"); !r) - return std::unexpected(r.error()); - out.useFlags = mcpp::toolchain::bmi_reference_tokens("-fmodule-file=mcpp=", pcm); - fs::path corePcm = bdir / ("mcpp.core" + std::string(traits.bmiExt)); - out.aliasObject = bdir / ("mcpp_core" + std::string(dial.objExt)); - std::vector pre{compiler.string(), stdFlag, "--precompile", - "mcpp_core.cppm", "-o", corePcm.string()}; - for (auto& f : out.useFlags) pre.push_back(f); - if (auto r = run(with_base(std::move(pre)), "mcpp.core precompile"); !r) - return std::unexpected(r.error()); - std::vector obj{compiler.string(), stdFlag, "-c", - corePcm.string(), "-o", out.aliasObject.string()}; - for (auto& f : out.useFlags) obj.push_back(f); - if (auto r = run(with_base(std::move(obj)), "mcpp.core object"); !r) - return std::unexpected(r.error()); - for (auto& f : mcpp::toolchain::bmi_reference_tokens("-fmodule-file=mcpp.core=", corePcm)) - out.useFlags.push_back(f); - return out; - } - - // GCC: BMIs are implicit under /gcm.cache, so nothing to name. - if (auto r = run(with_base({compiler.string(), stdFlag, - std::string(mcpp::toolchain::bmi_traits(tc).compileModulesFlag).empty() - ? "-fmodules" : "-fmodules", - "-c", "mcpp.cppm", "-o", out.object.string()}), "compile"); !r) - return std::unexpected(r.error()); - out.useFlags = {"-fmodules"}; - // GCC finds both BMIs under /gcm.cache; the alias only has to exist. - out.aliasObject = bdir / ("mcpp_core" + std::string(dial.objExt)); - if (auto r = run(with_base({compiler.string(), stdFlag, "-fmodules", - "-c", "mcpp_core.cppm", "-o", out.aliasObject.string()}), - "mcpp.core compile"); !r) - return std::unexpected(r.error()); - return out; -} - - -} // namespace mcpp::build - -namespace mcpp::build { - -std::expected -build_host_module(const fs::path& bdir, const fs::path& compiler, - const std::vector& base, const std::string& stdFlag, - const mcpp::toolchain::Toolchain& tc, - const std::vector>& env, - std::string_view logicalName, const fs::path& interfacePath, - const std::vector& extraUseFlags) { - std::error_code ec; - if (!fs::exists(interfacePath, ec)) { - return std::unexpected(std::format( - "host module '{}': no interface unit at {}\n" - " A package offering build rules must have a lib root " - "(src/.cppm or [lib] path).", - logicalName, interfacePath.string())); - } - // A filesystem-safe stem. Partition separators and any path separator that - // sneaks into a logical name would otherwise create directories that do - // not exist. Dots are left ALONE on purpose: `a.b.rules.o` is a legal - // filename, GCC's own gcm.cache uses the dotted module name verbatim, and - // rewriting them would make the object name disagree with the BMI name for - // no gain. - std::string stem(logicalName); - for (auto& c : stem) if (c == ':' || c == '/' || c == '\\') c = '-'; - - auto run = [&](std::vector argv, const char* what) - -> std::expected { - auto r = mcpp::platform::process::capture_exec(argv, env, bdir.string()); - if (r.exit_code != 0) - return std::unexpected(std::format( - "host module '{}' {} failed (exit {}):\n{}", - logicalName, what, r.exit_code, r.output)); - return {}; - }; - auto with_base = [&](std::vector head) { - for (auto& b : base) head.push_back(b); - for (auto& f : extraUseFlags) head.push_back(f); - return head; - }; - - const auto traits = mcpp::toolchain::bmi_traits(tc); - const auto& dial = mcpp::toolchain::dialect_for(tc); - McppModule out; - out.object = bdir / (stem + std::string(dial.objExt)); - - if (tc.compiler == mcpp::toolchain::CompilerId::MSVC) { - fs::path ifc = bdir / (stem + std::string(traits.bmiExt)); - std::vector argv{compiler.string()}; - for (auto f : dial.alwaysFlagsArgv) argv.emplace_back(f); - argv.push_back(stdFlag); - argv.push_back("/interface"); - for (auto f : dial.forceCxxLangArgv) argv.emplace_back(f); - argv.push_back("/c"); - argv.push_back(interfacePath.string()); - argv.push_back("/ifcOutput"); argv.push_back(ifc.string()); - argv.push_back(std::string(dial.outputObjPrefix) + out.object.string()); - if (auto r = run(with_base(std::move(argv)), "compile"); !r) - return std::unexpected(r.error()); - out.useFlags = mcpp::toolchain::bmi_reference_tokens( - std::format(" /reference {}=", logicalName), ifc); - return out; - } - - // The interface's LANGUAGE, stated rather than inferred from its extension. - // - // Measured on macOS CI: a rule package whose lib root is `rulepkg.ixx` - // made `clang++ --precompile rulepkg.ixx -o rulepkg.pcm` EXIT 0 AND WRITE - // NOTHING — clang's driver does not recognise `.ixx`, so it treated the file - // as a linker input, warned that it was unused, and succeeded. The failure - // surfaced one step later as `no such file or directory: …/rulepkg.pcm`, - // naming an output rather than the input that was never read. - // - // Every other module compile in mcpp already says this (BmiTraits:: - // moduleInterfaceLangFlag — `/interface /TP`, `-x c++-module`, `-x c++`); - // the host-module path was the one place that still let the driver guess. - // It is positional on GNU-style drivers, so it goes immediately before the - // input. - std::vector langArgv; - { - std::string_view lang = traits.moduleInterfaceLangFlag; - for (std::size_t i = 0; i < lang.size(); ) { - while (i < lang.size() && lang[i] == ' ') ++i; - auto j = lang.find(' ', i); - if (j == std::string_view::npos) j = lang.size(); - if (j > i) langArgv.emplace_back(lang.substr(i, j - i)); - i = j; - } - } - - if (mcpp::toolchain::is_clang(tc)) { - fs::path pcm = bdir / (stem + std::string(traits.bmiExt)); - std::vector pre{compiler.string(), stdFlag, "--precompile"}; - for (auto const& l : langArgv) pre.push_back(l); - pre.push_back(interfacePath.string()); - pre.push_back("-o"); pre.push_back(pcm.string()); - if (auto r = run(with_base(std::move(pre)), "precompile"); !r) - return std::unexpected(r.error()); - // The precompile can succeed and write nothing when the driver ignored - // the input, which is exactly what happened above. Checked here so the - // diagnostic names the interface rather than a missing output. - if (!fs::exists(pcm, ec)) { - return std::unexpected(std::format( - "host module '{}': the compiler accepted '{}' and produced no " - "BMI.\n" - " The interface's language is passed explicitly, so this " - "is not an extension\n" - " the driver failed to recognise — check that the file " - "really is a module interface.", - logicalName, interfacePath.string())); - } - if (auto r = run(with_base({compiler.string(), stdFlag, "-c", - pcm.string(), "-o", out.object.string()}), - "object"); !r) - return std::unexpected(r.error()); - out.useFlags = mcpp::toolchain::bmi_reference_tokens( - std::format("-fmodule-file={}=", logicalName), pcm); - return out; - } - - // GCC: BMIs are implicit under /gcm.cache, so nothing to name — which - // is also why the compile has to happen in bdir (it already does). - std::vector gccArgv{compiler.string(), stdFlag, "-fmodules", "-c"}; - for (auto const& l : langArgv) gccArgv.push_back(l); - gccArgv.push_back(interfacePath.string()); - gccArgv.push_back("-o"); gccArgv.push_back(out.object.string()); - if (auto r = run(with_base(std::move(gccArgv)), "compile"); !r) - return std::unexpected(r.error()); - out.useFlags = {"-fmodules"}; - return out; -} - } // namespace mcpp::build diff --git a/src/build/ninja_backend.cppm b/src/build/ninja_backend.cppm index ae7728550..c12e46cd5 100644 --- a/src/build/ninja_backend.cppm +++ b/src/build/ninja_backend.cppm @@ -4584,7 +4584,7 @@ std::expected NinjaBackend::build(const BuildPlan& plan stage("symbol-provision"); // Under a report the lines were printed as ninja wrote them. if (opts.verbose && !opts.progress && !out.empty()) - std::fputs(out.c_str(), stdout); + mcpp::ui::block(out); std::set want(opts.ninjaTargets.begin(), opts.ninjaTargets.end()); for (auto& lu : plan.linkUnits) { if (!want.empty() && !want.contains(lu.output.generic_string())) continue; diff --git a/src/build/plan.cppm b/src/build/plan.cppm index be1001d4e..410bbc394 100644 --- a/src/build/plan.cppm +++ b/src/build/plan.cppm @@ -686,6 +686,29 @@ expand_manifest_include_entry(const std::filesystem::path& root, // ends up spelled two ways. std::string qualified_package_name(const mcpp::manifest::Manifest& manifest); +// What `${mcpp.target_file:}` names in `plan`: the build-dir-relative +// output of the link unit of that target name. A plan of several members may +// hold a target of one name in each of them; the name then means the unit of +// `actingMember`, the member the referring package acts for. `output` is empty +// when no unit has the name. `members` is non-empty when the name is ambiguous: +// several members define it and none of them is `actingMember`. +struct TargetFileAnswer { + std::string output; + std::vector members; +}; +TargetFileAnswer resolve_target_file(const BuildPlan& plan, std::string_view name, + std::string_view actingMember); + +// The references an action's arguments made that the plan cannot answer, as +// the refusal a user reads: names no link unit has (`unresolvedTargets`), names +// several members define for a package that acts for none of them +// (`ambiguousTargets`, from `resolve_target_file`), and `${mcpp.artifact:}` +// references no requested artifact matches (`unresolvedArtifacts`). +std::expected refuse_unresolved_references( + const BuildPlan& plan, const std::set& unresolvedTargets, + const std::map>& ambiguousTargets, + const std::set& unresolvedArtifacts); + // The objects a package contributes to an image that links it whole: its // module units, which link unconditionally, then its implementation units, in // plan order. A shared library that carries a private copy of a graph C++ @@ -3289,6 +3312,64 @@ make_plan(const mcpp::manifest::Manifest& manifest, return plan; } +TargetFileAnswer resolve_target_file(const BuildPlan& plan, std::string_view name, + std::string_view actingMember) { + TargetFileAnswer answer; + std::set members; + for (auto const& lu : plan.linkUnits) { + if (lu.targetName != name) continue; + answer.output = lu.output.generic_string(); + if (!actingMember.empty() && lu.memberOf == actingMember) return {answer.output, {}}; + if (!lu.memberOf.empty()) members.insert(lu.memberOf); + } + // One member's plan, and a name one member defines, resolve as they always + // have: to the one unit, or to the last of the units that share the name. + if (members.size() > 1) answer.members.assign(members.begin(), members.end()); + return answer; +} + +std::expected refuse_unresolved_references( + const BuildPlan& plan, const std::set& unresolvedTargets, + const std::map>& ambiguousTargets, + const std::set& unresolvedArtifacts) { + if (!unresolvedTargets.empty()) { + std::string bad, known; + for (auto const& n : unresolvedTargets) bad += (bad.empty() ? "" : ", ") + n; + for (auto const& lu : plan.linkUnits) + known += (known.empty() ? "" : ", ") + lu.targetName; + return std::unexpected(std::format( + "build.mcpp action references unknown target(s) via " + "${{mcpp.target_file:...}}: {}\n" + " targets in this build: [{}]\n" + " (a target gated by required_features is absent unless those " + "features are active)", + bad, known.empty() ? std::string("none") : known)); + } + + for (auto const& [n, in] : ambiguousTargets) + return std::unexpected(std::format( + "build.mcpp action references ${{mcpp.target_file:{}}}, a target of " + "each of the members {}, and its package acts for none of them.\n" + " use: select one of the members, or give the targets distinct names", + n, std::format("{}", in))); + + if (!unresolvedArtifacts.empty()) { + std::string bad, known; + for (auto const& n : unresolvedArtifacts) bad += (bad.empty() ? "" : ", ") + n; + for (auto const& lu : plan.linkUnits) + if (!lu.artifactOf.empty()) + known += (known.empty() ? "" : ", ") + lu.artifactOf + "/" + lu.targetName; + return std::unexpected(std::format( + "build.mcpp action references unknown artifact(s) via " + "${{mcpp.artifact:/}}: {}\n" + " artifacts in this build: [{}]\n" + " (an artifact exists when a dependency edge requests it with " + "`artifacts = [\"\"]`)", + bad, known.empty() ? std::string("none") : known)); + } + return {}; +} + std::vector package_link_objects(const BuildPlan& plan, std::string_view packageName) { std::vector objects; diff --git a/src/build/prepare.cppm b/src/build/prepare.cppm index 0e33e6cd7..8d043bd02 100644 --- a/src/build/prepare.cppm +++ b/src/build/prepare.cppm @@ -493,8 +493,33 @@ export struct BuildContext { // record is written, and matched, per selection and group (§15). std::string workspaceRequest; std::string workspaceGroup; + // Which packed member each package acts for, in a plan of several + // selected members (member selection design 2026-09-30, K1): for every + // package of the plan, the selected members whose dependency closure + // reaches it, in discovery order, a member reaching itself. Empty for a + // plan of one member or none, where the plan's one subject is what every + // package acts for. Read through `pack_owner`. + std::map> packReach; + // The packages that declared each pack format with + // `mcpp::provides_pack_format`, by format, as qualified package names. + // Collected on every pass, with `plan.providedPackFormats`, which it + // refines: the plan knows the set of formats and this knows who provides + // each. + std::map> packFormatProviders; }; +// The selected member a package acts for in a packaging pass: the package +// itself when it is one of the members, the one member that reaches it when +// exactly one does, and none (empty) when several do. `reach` is the package's +// entry of `BuildContext::packReach`. Written once and read by prepare, which +// hands each program the stage of the member it acts for, and by `mcpp pack`, +// which attributes each action a program submitted to a member. +export inline std::string pack_owner(std::string_view package, + const std::vector& reach) { + if (std::ranges::find(reach, package) != reach.end()) return std::string(package); + return reach.size() == 1 ? reach.front() : std::string{}; +} + // The ONE cache-mode resolver, for the same reason resolve_profile_name exists: // execute.cppm's fast paths deliberately skip prepare_build, so they need to // settle the mode from the same rule. Pure in (manifest, override, environment). @@ -515,6 +540,50 @@ export struct BuildContext { // lost. A context of any other shape is left as it is. export void focus_on_member(BuildContext& ctx); +// Reads `fn` with the plan describing one workspace member (workspace design +// 2026-09-29 §15): the member's link group is exchanged into the plan's own +// fields for the call, so what a member's tests run against, or what its +// package is made of -- its runtime directories, the files its runtime needs -- +// is its closure's, and no other member's; and the member's manifest and root +// become the context's, so a reader that asks "the package being built" is +// answered about the member. `owner` is the member's qualified package name. +// Outside a workspace plan, for an empty `owner` and for a name the plan does +// not hold, the plan is read as it is. +// +// Every exchange is undone before the call returns, also when `fn` throws: a +// drive emits the plan and must see its own fields, and the next member is +// read from the plan as the group left it. The exchange is of fields, not of +// copies, so a member's view costs no more than a swap. This is +// `focus_on_member` for a plan of several members, and scoped. +export template +void with_member(BuildContext& ctx, std::string_view owner, F&& fn) { + BuildPlan::LinkGroup* group = nullptr; + BuildContext::WorkspaceMember* member = nullptr; + if (!owner.empty()) { + for (auto& g : ctx.plan.linkGroups) + if (!g.linkOnly && g.member == owner) { group = &g; break; } + for (auto& m : ctx.workspaceMembers) + if (m.name == owner) { member = &m; break; } + } + if (!group && !member) { fn(); return; } + struct Exchange { + BuildContext& ctx; + BuildPlan::LinkGroup* group; + BuildContext::WorkspaceMember* member; + void flip() { + if (group) swap_link_group(ctx.plan, *group); + if (member) { + std::swap(ctx.manifest, member->manifest); + std::swap(ctx.projectRoot, member->root); + } + } + Exchange(BuildContext& c, BuildPlan::LinkGroup* g, BuildContext::WorkspaceMember* m) + : ctx(c), group(g), member(m) { flip(); } + ~Exchange() { flip(); } + } exchange{ctx, group, member}; + fn(); +} + export CacheMode resolve_cache_mode(const mcpp::manifest::Manifest& m, std::string_view override_mode); @@ -676,32 +745,49 @@ export struct BuildOverrides { // and after staging, so the claiming member submits an action whose input // is a directory that by then exists. // - // NEITHER VALUE IS DERIVED HERE, AND THAT IS THE POINT. `pack_stage_dir` is - // a function of the package name, the version, the resolved triple and the - // mode, and the resolved triple is not known until a prepare has run. - // Computing it a second time before prepare -- from the host triple, say -- - // is the shape where two derivations of one value agree on every machine - // the author has. Both are read out of what the first pass and `make_plan` - // already answered. + // NEITHER VALUE IS DERIVED HERE, AND THAT IS THE POINT. A member's stage + // directory is a function of the package name, the version, the resolved + // triple and the mode, and the resolved triple is not known until a + // prepare has run. Computing it a second time before prepare -- from the + // host triple, say -- is the shape where two derivations of one value agree + // on every machine the author has. Both are read out of what the first pass + // and `make_plan` already answered. std::string pack_format; - std::filesystem::path pack_stage_dir; - // WHY THERE IS NO STAGED TREE, when there is none and a format was still - // requested. Empty otherwise. - // - // A dispatched format does not require the built-in bundling to have - // succeeded -- see the note in `mcpp.pack.pipeline`. When it did not, the - // reason travels here so `${mcpp.stage_dir}`'s refusal can name it instead - // of saying only that the placeholder is unavailable. A member author - // reading "this build is not packaging" for a build that plainly is would - // be sent looking in the wrong place. - std::string pack_stage_reason; - // #649 E5: the strip decision and the debug-symbol directory that pass - // resolved, "1" or "0" and absolute, set only beside `pack_format`. A - // member that stages libraries of its own reads them through - // `mcpp::pack_strip()` and `mcpp::pack_debug_symbols_dir()`, so - // `--no-strip` reaches its files as it reaches the engine's. - std::string pack_strip; - std::filesystem::path pack_debug_symbols_dir; + // What a packaging pass knows of one packed member: where its tree is + // staged, and what the staging resolved for the programs that act for it. + struct PackStage { + // The staged tree, absolute. Empty when staging was refused, which is + // what makes `${mcpp.stage_dir}` refuse with `reason` attached rather + // than expand to a directory that does not exist. + std::filesystem::path dir; + // WHY THERE IS NO STAGED TREE, when there is none and a format was + // still requested. Empty otherwise. + // + // A dispatched format does not require the built-in bundling to have + // succeeded -- see the note in `mcpp.pack.pipeline`. When it did not, + // the reason travels here so `${mcpp.stage_dir}`'s refusal can name it + // instead of saying only that the placeholder is unavailable. A member + // author reading "this build is not packaging" for a build that + // plainly is would be sent looking in the wrong place. + std::string reason; + // #649 E5: the strip decision and the debug-symbol directory that + // member's staging resolved, "1" or "0" and absolute. A member that + // stages libraries of its own reads them through `mcpp::pack_strip()` + // and `mcpp::pack_debug_symbols_dir()`, so `--no-strip` reaches its + // files as it reaches the engine's. + std::string strip; + std::filesystem::path debugSymbolsDir; + }; + // The packed members' stages, by qualified package name. One entry is what a + // pack of one member sets, and then every program of the plan receives it, + // as the plan's only package being packed. With several entries (`mcpp + // pack` over several members, member selection design 2026-09-30, K1) a + // program receives the stage of the member it acts for: its own when it is + // a packed member, and otherwise the one packed member whose dependency + // closure reaches it. A package that several packed members reach acts for + // none of them, and receives no stage at all, which is what lets one run of + // its program serve every member. + std::map pack_stages; }; // ── git dependency helpers ────────────────────────────────────────────────── diff --git a/src/build/prepare/features.cpp b/src/build/prepare/features.cpp index fefc5a4d6..1f1aa8f63 100644 --- a/src/build/prepare/features.cpp +++ b/src/build/prepare/features.cpp @@ -1187,10 +1187,14 @@ step6_host_module_registration(PrepareState& state) { } std::vector ordered; + // The package each entry of `ordered` came from, for where its + // compiled module may be kept (#748, B1). + std::vector orderedProvider; for (auto p : *topo) { for (auto& hm : units(p)) { hm.importable = isDirect.contains(p); ordered.push_back(std::move(hm)); + orderedProvider.push_back(p); } } @@ -1202,7 +1206,8 @@ step6_host_module_registration(PrepareState& state) { if (auto clash = prov::host_module_collision(ordered)) return std::unexpected(*clash); - for (auto const& hm : ordered) { + for (std::size_t k = 0; k < ordered.size(); ++k) { + auto const& hm = ordered[k]; // Warned once per (package, module), not once per consumer: // a rule re-exported down a chain is visible to every // package on it, and repeating one naming remark N times @@ -1212,8 +1217,31 @@ step6_host_module_registration(PrepareState& state) { if (prefixWarned.insert(hm.package + "\x1e" + hm.module).second) mcpp::diag::warning("build/rule-namespace", *w); } - state.hostModulesByConsumer[c].push_back( - {hm.module, hm.interface, hm.importable}); + mcpp::build::BuildProgramEnv::HostModuleRef ref{ + hm.module, hm.interface, hm.importable}; + // Where the compiled module may be kept (#748, B1): in the + // global cache only when the provider is an index package + // whose sources are in the immutable store, the rule the + // dependency cache applies to a package (plan.cpp). The + // module is compiled alone, so nothing it was built against + // can be local: the provider's own location decides. + { + const auto pi = orderedProvider[k]; + const auto& provider = state.packages[pi]; + const auto* ident = pi >= 1 && pi - 1 < state.dep_cache_identities.size() + ? &state.dep_cache_identities[pi - 1] : nullptr; + ref.providerName = identity(pi); + ref.providerVersion = provider.manifest.package.version; + ref.providerRoot = provider.root; + if (ident && ident->sourceKind == "version" + && mcpp::build::path_is_under_any(provider.root, state.storeRoots)) { + ref.immutableSource = true; + ref.providerIndex = ident->indexName; + ref.providerName = ident->packageName; + if (!ident->version.empty()) ref.providerVersion = ident->version; + } + } + state.hostModulesByConsumer[c].push_back(std::move(ref)); } } @@ -1883,10 +1911,7 @@ static std::expected step6_dependency_build_programs(PrepareS // generating a declaration for this package must match how this // package is compiled. fill_package_build_env(bpEnv, pkg.manifest); - bpEnv.packFormat = state.overrides.pack_format; - bpEnv.packStageDir = state.overrides.pack_stage_dir; - bpEnv.packStrip = state.overrides.pack_strip; - bpEnv.packDebugSymbolsDir = state.overrides.pack_debug_symbols_dir; + state.fillPackEnv(bpEnv, i); bpEnv.requested = pkg.selectedMember; bpEnv.languageModules = pkg.manifest.language.modules; bpEnv.ruleModules = pkg.manifest.buildConfig.ruleModules; @@ -1896,6 +1921,11 @@ static std::expected step6_dependency_build_programs(PrepareS bpEnv.artifactsDir = state.workRoot / "target" / ".build-mcpp" / "deps" / (dirSafe(pkg.manifest.package.name) + "@" + pkg.manifest.package.version); bpEnv.genBase = bpEnv.artifactsDir / "out"; + // What the program imports is kept once for the workspace, and in the + // global cache where it comes from the engine or the index (#748). + bpEnv.moduleStore = state.workRoot / "target" / ".build-mcpp" / "host-modules"; + if (state.cacheMode == CacheMode::Global) + bpEnv.moduleCacheRoot = mcpp::home::cache_root(); // mcpp#241: this package's resolved dependencies as // MCPP_DEP__DIR, from the authoritative edge graph (no // name-guessing); covers feature-activated deps too diff --git a/src/build/prepare/plan.cpp b/src/build/prepare/plan.cpp index a0d7084ee..cb627ba58 100644 --- a/src/build/prepare/plan.cpp +++ b/src/build/prepare/plan.cpp @@ -200,6 +200,9 @@ static std::expected step13_source_packages(PrepareState& sta break; } } + // Which member each package acts for in a pack of several members. + state.computePackReach(); + ctx.packReach = state.packReach; } return {}; } @@ -914,6 +917,9 @@ static std::expected step13_build_graph_actions(PrepareState& // become an edge with a blank path, and ninja reports that far away // from the typo that caused it. std::set unresolvedTargets; + // Names several members define, used by a package acting for none of + // them (`resolve_target_file`): refused, never resolved by member order. + std::map> ambiguousTargets; std::set unresolvedArtifacts; // `${mcpp.stage_dir}` used where there is no staged tree, and used by an // action whose role runs before the link. Both are refusals rather than @@ -923,9 +929,13 @@ static std::expected step13_build_graph_actions(PrepareState& // Section 2 of the design record measured that shape: a valid, empty, // 52 KB installer with nothing said about it. std::set stageDirNoPass, stageDirWrongRole; - // Carried from `state.overrides` so the refusal below can say WHY there is - // no tree, which is a different sentence from "you are not packaging". + // Carried from the stage of the package that declared the action, so the + // refusal below can say WHY there is no tree, which is a different + // sentence from "you are not packaging". std::string stageDirWhy; + // A package several packed members reach has no stage of its own; its + // refusal names the package and those members. + std::string stageDirShared; // WHETHER *THIS* ACTION REFERENCED THE STAGED TREE, and deliberately a // flag rather than a set keyed on the action's id: an id is unique // within the package that declared it and nothing more, so two packages @@ -943,7 +953,12 @@ static std::expected step13_build_graph_actions(PrepareState& // the action (§15 of the 2026-09-29 workspace design). Set per // package by `collect`. std::filesystem::path binDir = ctx.plan.outputDir / "bin"; - const bool stagePass = !state.overrides.pack_stage_dir.empty(); + // The stage of the package being collected, as `collect` sets it: that of + // the member it acts for (`PrepareState::packStageOf`), or why it has none. + const BuildOverrides::PackStage* stage = nullptr; + std::string actingMember; // the member the package acts for (actingMemberOf) + std::string stageShared; + bool stagePass = false; auto substitute = [&](std::string s, const char* actionId, mcpp::manifest::BuildAction::Role role) { auto rep = [&](std::string_view what, const std::string& with) { @@ -973,13 +988,14 @@ static std::expected step13_build_graph_actions(PrepareState& if (s.find("${mcpp.stage_dir}") != std::string::npos) { if (!stagePass) { stageDirNoPass.insert(actionId); - stageDirWhy = state.overrides.pack_stage_reason; + stageDirWhy = stage ? stage->reason : std::string{}; + if (!stageShared.empty()) stageDirShared = stageShared; } else if (role != mcpp::manifest::BuildAction::Role::Artifact) { stageDirWrongRole.insert(actionId); } else { thisActionUsesStageDir = true; } - rep("${mcpp.stage_dir}", state.overrides.pack_stage_dir.string()); + rep("${mcpp.stage_dir}", stage ? stage->dir.string() : std::string{}); } constexpr std::string_view kTf = "${mcpp.target_file:"; for (std::size_t p; (p = s.find(kTf)) != std::string::npos; ) { @@ -993,12 +1009,10 @@ static std::expected step13_build_graph_actions(PrepareState& // "missing and no known rule to make it". Commands run with // cwd = the build dir, so the relative form is also what the // tool being invoked should receive. - std::string resolved; - for (auto const& lu : ctx.plan.linkUnits) - if (lu.targetName == name) - resolved = lu.output.generic_string(); - if (resolved.empty()) unresolvedTargets.insert(name); - s.replace(p, close - p + 1, resolved); + auto answer = mcpp::build::resolve_target_file(ctx.plan, name, actingMember); + if (answer.output.empty()) unresolvedTargets.insert(name); + else if (!answer.members.empty()) ambiguousTargets[name] = answer.members; + s.replace(p, close - p + 1, answer.output); } // `${mcpp.artifact:/}` (mcpp#711): a dependency's // program that an edge requested with `artifacts = [...]`, spelled @@ -1029,12 +1043,17 @@ static std::expected step13_build_graph_actions(PrepareState& } return s; }; - auto collect = [&](const mcpp::manifest::Manifest& mm) { + auto collect = [&](const mcpp::manifest::Manifest& mm, std::size_t packageIndex) { // The declaring package, recorded here because this is the only // place that knows it: the build program emitted the action, and a // program has no idea which package the engine loaded it for. // mcpp#534's ordering edge is scoped to this name. auto owner = mcpp::build::qualified_package_name(mm); + stage = state.packStageOf(packageIndex); + actingMember = state.actingMemberOf(packageIndex); + stagePass = stage && !stage->dir.empty(); + stageShared = stage || state.overrides.pack_stages.empty() + ? std::string{} : state.packSharedWhy(packageIndex); binDir = ctx.plan.outputDir / "bin"; for (auto const& g : ctx.plan.linkGroups) if (!g.linkOnly && g.member == owner) binDir = ctx.plan.outputDir / g.productDir; @@ -1066,7 +1085,7 @@ static std::expected step13_build_graph_actions(PrepareState& if (thisActionUsesStageDir) { a.consumesStageDir = true; a.inputs.push_back( - mcpp::pack::stage_manifest_path(state.overrides.pack_stage_dir).string()); + mcpp::pack::stage_manifest_path(stage->dir).string()); } a.packageName = owner; ctx.plan.actions.push_back(std::move(a)); @@ -1074,12 +1093,17 @@ static std::expected step13_build_graph_actions(PrepareState& // Every package's declaration, on every pass. Sorted and de-duplicated // below so the refusal's list reads the same whatever order resolution // walked the graph in. - for (auto const& f : mm.buildConfig.packFormats) + // ...and who declared each, which a pack over several members reads + // to tell which member a format is provided for. + for (auto const& f : mm.buildConfig.packFormats) { ctx.plan.providedPackFormats.push_back(f); + auto& who = ctx.packFormatProviders[f]; + if (std::ranges::find(who, owner) == who.end()) who.push_back(owner); + } }; - collect(*state.m); + collect(*state.m, 0); for (std::size_t i = 1; i < state.packages.size(); ++i) - collect(state.packages[i].manifest); + collect(state.packages[i].manifest, i); std::ranges::sort(ctx.plan.providedPackFormats); ctx.plan.providedPackFormats.erase( std::ranges::unique(ctx.plan.providedPackFormats).begin(), @@ -1088,6 +1112,13 @@ static std::expected step13_build_graph_actions(PrepareState& if (!stageDirNoPass.empty()) { std::string ids; for (auto const& n : stageDirNoPass) ids += (ids.empty() ? "" : ", ") + n; + if (!stageDirShared.empty()) + return std::unexpected(std::format( + "build.mcpp action(s) [{}] reference ${{mcpp.stage_dir}}, and {}.\n" + " A staged tree belongs to one member, and one run of this package's " + "program serves all of them.\n" + " use: provide the format from each member's own build program, or pack " + "the members one at a time", ids, stageDirShared)); if (!stageDirWhy.empty()) { return std::unexpected(std::format( "build.mcpp action(s) [{}] reference ${{mcpp.stage_dir}}, and no " @@ -1124,34 +1155,10 @@ static std::expected step13_build_graph_actions(PrepareState& "there is anything to stage.\n" " use: role = \"artifact\"", ids)); } - if (!unresolvedTargets.empty()) { - std::string bad, known; - for (auto const& n : unresolvedTargets) bad += (bad.empty() ? "" : ", ") + n; - for (auto const& lu : ctx.plan.linkUnits) - known += (known.empty() ? "" : ", ") + lu.targetName; - return std::unexpected(std::format( - "build.mcpp action references unknown target(s) via " - "${{mcpp.target_file:...}}: {}\n" - " targets in this build: [{}]\n" - " (a target gated by required_features is absent unless those " - "features are active)", - bad, known.empty() ? std::string("none") : known)); - } - - if (!unresolvedArtifacts.empty()) { - std::string bad, known; - for (auto const& n : unresolvedArtifacts) bad += (bad.empty() ? "" : ", ") + n; - for (auto const& lu : ctx.plan.linkUnits) - if (!lu.artifactOf.empty()) - known += (known.empty() ? "" : ", ") + lu.artifactOf + "/" + lu.targetName; - return std::unexpected(std::format( - "build.mcpp action references unknown artifact(s) via " - "${{mcpp.artifact:/}}: {}\n" - " artifacts in this build: [{}]\n" - " (an artifact exists when a dependency edge requests it with " - "`artifacts = [\"\"]`)", - bad, known.empty() ? std::string("none") : known)); - } + if (auto refused = mcpp::build::refuse_unresolved_references( + ctx.plan, unresolvedTargets, ambiguousTargets, unresolvedArtifacts); + !refused) + return std::unexpected(refused.error()); // role = "object": the outputs are LINK inputs, so attach them to the // link units that should receive them. diff --git a/src/build/prepare/state.cppm b/src/build/prepare/state.cppm index 2846e4fdb..147c70377 100644 --- a/src/build/prepare/state.cppm +++ b/src/build/prepare/state.cppm @@ -317,6 +317,109 @@ struct PrepareState { // which a root receives; in a workspace plan every selected member does. std::vector profileCflags, profileCxxflags; bool workspacePlan() const { return !selectedMemberPaths.empty(); } + // ── The packaging pass: which packed member a package acts for ────────── + // + // `mcpp pack` over several members plans them as one graph (member + // selection design 2026-09-30, K1), and a package of it may serve one + // member, several, or be a member itself. A program that provides a pack + // format submits an action against the staged tree of the member it packs, + // so each program must be told which member that is: its own when it is a + // selected member, the one selected member whose closure reaches it when + // exactly one does, and none when several do (`pack_owner`). The closures + // are read from the dependency edges recorded while the graph was walked, + // which is why this is asked for only after the graph is complete. + // + // Empty, and never asked for, when the plan holds fewer than two selected + // members: its one subject is what every package acts for. + std::map> packReach; + bool packReachComputed = false; + std::size_t selectedMemberCount() const { + std::size_t n = 0; + for (std::size_t i = 1; i < packages.size(); ++i) + if (packages[i].selectedMember) ++n; + return n; + } + void computePackReach() { + if (packReachComputed) return; + packReachComputed = true; + if (selectedMemberCount() < 2) return; + for (std::size_t m = 1; m < packages.size(); ++m) { + if (!packages[m].selectedMember) continue; + std::vector reached(packages.size(), false); + reached[m] = true; + for (bool grew = true; grew;) { + grew = false; + for (auto const& r : graphRequests) + if (r.consumerPackageIndex < reached.size() && reached[r.consumerPackageIndex] + && r.dependencyPackageIndex < reached.size() + && !reached[r.dependencyPackageIndex]) { + reached[r.dependencyPackageIndex] = true; + grew = true; + } + } + const auto member = mcpp::build::qualified_package_name(packages[m].manifest); + for (std::size_t i = 1; i < packages.size(); ++i) + if (reached[i]) + packReach[mcpp::build::qualified_package_name(packages[i].manifest)] + .push_back(member); + } + } + // Why package `i` has no stage when several packed members reach it: such a + // package acts for none of them (`pack_owner`). Empty otherwise. + std::string packSharedWhy(std::size_t i) { + computePackReach(); + if (i >= packages.size()) return {}; + const auto name = mcpp::build::qualified_package_name(packages[i].manifest); + auto reach = packReach.find(name); + if (reach == packReach.end() || reach->second.size() < 2) return {}; + std::string members; + for (auto const& m : reach->second) members += (members.empty() ? "'" : ", '") + m + "'"; + return std::format("package '{}' is reached by the packed members {}", name, members); + } + // What the packaging pass tells the programs that act for package `i`, or + // null when there is none to tell: outside a packaging pass, for a package + // several packed members reach, and for a member that has no stage. + const BuildOverrides::PackStage* packStageOf(std::size_t i) { + auto& stages = overrides.pack_stages; + if (stages.empty()) return nullptr; + if (selectedMemberCount() < 2) return &stages.begin()->second; + computePackReach(); + if (i >= packages.size()) return nullptr; + const auto name = mcpp::build::qualified_package_name(packages[i].manifest); + auto reach = packReach.find(name); + if (reach == packReach.end()) return nullptr; + auto stage = stages.find(pack_owner(name, reach->second)); + return stage == stages.end() ? nullptr : &stage->second; + } + // The selected member package `i` acts for in a plan of several members: + // itself when it is one, the one member whose closure reaches it when + // exactly one does, and none (empty) otherwise, by `pack_owner`'s rule. It + // is what a name the package's actions use is resolved against when two + // members give that name to different things (`${mcpp.target_file:}`). + std::string actingMemberOf(std::size_t i) { + if (selectedMemberCount() < 2 || i >= packages.size()) return {}; + computePackReach(); + const auto name = mcpp::build::qualified_package_name(packages[i].manifest); + auto reach = packReach.find(name); + return reach == packReach.end() ? std::string{} : pack_owner(name, reach->second); + } + // The packaging pass's values of a build program's environment, for the + // program of package `i`. A package that acts for no packed member is told + // nothing, not even the format: its answer cannot depend on a request it + // has no member to serve, so its program is not run again for it, and one + // run serves every member that reaches it. + void fillPackEnv(mcpp::build::BuildProgramEnv& env, std::size_t i) { + if (overrides.pack_stages.empty()) { + env.packFormat = overrides.pack_format; + return; + } + if (const auto* stage = packStageOf(i)) { + env.packFormat = overrides.pack_format; + env.packStageDir = stage->dir; + env.packStrip = stage->strip; + env.packDebugSymbolsDir = stage->debugSymbolsDir; + } + } // `--features` as the command gave it. A workspace plan hands the tokens to // its members and clears `overrides.features`; the request is still what // the build was asked for, and what its fast-path record names. diff --git a/src/build/prepare/target_side.cpp b/src/build/prepare/target_side.cpp index 1b2557977..922ca6a94 100644 --- a/src/build/prepare/target_side.cpp +++ b/src/build/prepare/target_side.cpp @@ -20,6 +20,8 @@ import mcpp.source_kind; import mcpp.modgraph.glob; import mcpp.graph; import mcpp.build.progress; // the members' programs wait in order (design 2026-09-29 §4.2) +import mcpp.build.schedule.policy; // resolve_jobs — how many programs compile at once (#748, B2) +import mcpp.platform.capacity; // the host fallback when no job count was stated import mcpp.modgraph.graph; import mcpp.modgraph.scanner; import mcpp.modgraph.validate; @@ -1589,8 +1591,7 @@ static std::expected step9_root_build_program(PrepareState& s bpEnv.profile = state.effectiveProfile; bpEnv.accel = state.resolvedAccel(); fill_package_build_env(bpEnv, *state.m); - bpEnv.packFormat = state.overrides.pack_format; - bpEnv.packStageDir = state.overrides.pack_stage_dir; + state.fillPackEnv(bpEnv, 0); bpEnv.languageModules = state.m->language.modules; bpEnv.ruleModules = state.m->buildConfig.ruleModules; if (auto dit = state.deviceSourcesByPackage.find(state.root->string()); dit != state.deviceSourcesByPackage.end()) @@ -1601,6 +1602,11 @@ static std::expected step9_root_build_program(PrepareState& s // helper straight into it. Same value as the default when work_dir is // unset, so an ordinary build is unchanged. bpEnv.artifactsDir = state.workRoot / "target" / ".build-mcpp"; + // What the program imports is kept once for the workspace, and in the + // global cache where it comes from the engine or the index (#748). + bpEnv.moduleStore = bpEnv.artifactsDir / "host-modules"; + if (state.cacheMode == CacheMode::Global) + bpEnv.moduleCacheRoot = mcpp::home::cache_root(); // Root mode keeps genBase empty: a relative `generated=` from the ROOT // package resolves against the package root (the documented contract), // not against OUT_DIR. @@ -1636,9 +1642,6 @@ static std::expected step9_root_build_program(PrepareState& s bpEnv.dormantFeatures = state.dormantFeaturesByConsumer.count(0u) ? state.dormantFeaturesByConsumer.at(0u) : decltype(bpEnv.dormantFeatures){}; - // #649 E5: the packaging pass's strip decision, beside its format. - bpEnv.packStrip = state.overrides.pack_strip; - bpEnv.packDebugSymbolsDir = state.overrides.pack_debug_symbols_dir; // #647 E1: THE RESOLVED GRAPH, FOR THE ROOT'S PROGRAM ONLY. // // Every package, dependencies before the packages that request them @@ -1964,9 +1967,10 @@ static std::expected step9_member_build_programs(PrepareState return std::filesystem::exists(state.packages[i].root / "build.mcpp", bpEc) || !state.packages[i].manifest.buildConfig.ruleModules.empty(); }; - // mcpp runs them one after another, in this order, so every program not - // yet started is truly waiting; its line says so (build progress design - // 2026-09-29, §3.2), named as the program names itself. + // Their compiles may overlap (below), and their runs follow this order, so + // every program not yet run is waiting for its turn; its line says so + // (build progress design 2026-09-29, §3.2), named as the program names + // itself. for (auto const i : order) { if (!hasProgram(i)) continue; const auto& m = state.packages[i].manifest; @@ -1975,11 +1979,11 @@ static std::expected step9_member_build_programs(PrepareState : m.package.namespace_ + "." + m.package.name, state.packages[i].selectedMember); } - for (auto const i : order) { - if (!hasProgram(i)) continue; + // THE ENVIRONMENT A MEMBER'S PROGRAM IS GIVEN. Computed from the plan as it + // stands when it is asked, so the compile phase and the run each ask for it. + auto program_env = [&](std::size_t i) + -> std::expected { auto& pkg = state.packages[i]; - auto host = state.host_tc_for_build_program(); - if (!host) return std::unexpected(host.error()); mcpp::build::BuildProgramEnv bpEnv; bpEnv.targetTriple = state.resolvedTargetCanonical; fill_target_build_env(bpEnv, *state.m, state.tc ? &*state.tc : nullptr, @@ -1988,10 +1992,9 @@ static std::expected step9_member_build_programs(PrepareState bpEnv.profile = state.effectiveProfile; bpEnv.accel = state.resolvedAccel(); fill_package_build_env(bpEnv, pkg.manifest); - bpEnv.packFormat = state.overrides.pack_format; - bpEnv.packStageDir = state.overrides.pack_stage_dir; - bpEnv.packStrip = state.overrides.pack_strip; - bpEnv.packDebugSymbolsDir = state.overrides.pack_debug_symbols_dir; + // The stage of the member this program acts for: its own, or the one + // packed member that reaches it (`PrepareState::fillPackEnv`). + state.fillPackEnv(bpEnv, i); bpEnv.requested = pkg.selectedMember; bpEnv.languageModules = pkg.manifest.language.modules; bpEnv.ruleModules = pkg.manifest.buildConfig.ruleModules; @@ -2020,6 +2023,108 @@ static std::expected step9_member_build_programs(PrepareState if (!graph) return std::unexpected(graph.error()); bpEnv.graphFile = graph->first; bpEnv.graphDigest = graph->second; + // What the program imports is kept once for the workspace, so that a host + // module several members import is compiled once for all of them, and in + // the global cache where it comes from the engine or the index (#748). + bpEnv.moduleStore = state.workRoot / "target" / ".build-mcpp" / "host-modules"; + if (state.cacheMode == CacheMode::Global) + bpEnv.moduleCacheRoot = mcpp::home::cache_root(); + return bpEnv; + }; + + // THE COMPILES COME FIRST, AT THE SAME TIME (#748, B2). What a program needs + // before it runs is a compile, and a compile depends on no other program's + // run: only the runs have an order. So every program that needs compiling is + // compiled now, up to the build's job count at once, and the programs are then + // run in the order above, each taking the compile made for it. + // + // What the compile phase leaves out is what keeps the plan the serial build's + // plan: it applies no directive, reports no outcome, states no warning and + // writes no program cache. (It writes what the compile reads: the graph + // document and a program synthesised from rules.) The runs below do the + // rest, in the same order as before, so `build.ninja` and the directives + // applied are byte for byte the serial build's. A program's environment is computed again at its turn, from + // the plan as the programs before it left it; the compile is used only when + // it was made for what that computes, and is made again otherwise. + std::vector turns; // the programs, in the order they run + for (auto const i : order) + if (hasProgram(i)) turns.push_back(i); + std::vector precompiled(turns.size()); + if (turns.size() >= 2) { + if (auto host = state.host_tc_for_build_program()) { + std::vector> envs(turns.size()); + for (std::size_t k = 0; k < turns.size(); ++k) + if (auto e = program_env(turns[k])) envs[k] = std::move(*e); + + int globalDefaultJobs = 0; + if (auto c = state.get_cfg(/*requireBootstrap=*/false)) + globalDefaultJobs = static_cast((*c)->defaultJobs); + int jobs = mcpp::build::schedule::resolve_jobs(*state.m, {}, globalDefaultJobs); + if (jobs <= 0) + jobs = mcpp::platform::capacity::recommended_jobs( + mcpp::platform::capacity::host_capacity()); + const std::size_t workers = std::min( + turns.size(), static_cast(std::max(jobs, 1))); + + std::atomic next{0}; + // The first program, in the order they run, whose compile failed. A + // program after it is never reached by a build that stops at the + // first failure, so its compile is not started. + std::atomic firstFailure{turns.size()}; + auto work = [&] { + for (;;) { + const auto k = next.fetch_add(1); + if (k >= turns.size()) return; + if (k > firstFailure.load() || !envs[k]) continue; + auto& pkg = state.packages[turns[k]]; + try { + precompiled[k] = mcpp::build::precompile_build_program( + pkg.manifest, pkg.root, host->first, host->second, + pkg.manifest.cppStandard, *envs[k]); + } catch (const std::exception& ex) { + // A compile that threw is a compile that failed. Its + // empty stamp matches nothing, so the program's turn + // compiles it again, alone, and reports what that + // compile says. + precompiled[k] = {}; + precompiled[k].compiled = true; + precompiled[k].stamp = {}; + precompiled[k].error = std::format( + "build.mcpp failed to compile: {}", ex.what()); + } + if (precompiled[k].compiled && !precompiled[k].error.empty()) { + auto seen = firstFailure.load(); + while (k < seen && !firstFailure.compare_exchange_weak(seen, k)) {} + } + } + }; + const auto phaseBegan = std::chrono::steady_clock::now(); + std::vector pool; + for (std::size_t w = 1; w < workers; ++w) { + // A thread the system cannot start is one fewer worker, and the + // compiles are shared by the ones that did. + try { pool.emplace_back(work); } catch (const std::system_error&) { break; } + } + work(); + for (auto& t : pool) t.join(); + // The compiles overlapped, so the time they took is the phase's wall + // time, stated once; each program's line still shows its own. + mcpp::build::progress::programs_compiled( + std::chrono::duration_cast( + std::chrono::steady_clock::now() - phaseBegan)); + } + } + + std::size_t turn = 0; + for (auto const i : order) { + if (!hasProgram(i)) continue; + const auto& made = precompiled[turn++]; + auto& pkg = state.packages[i]; + auto host = state.host_tc_for_build_program(); + if (!host) return std::unexpected(host.error()); + auto envOf = program_env(i); + if (!envOf) return std::unexpected(envOf.error()); + auto& bpEnv = *envOf; auto& bc = pkg.manifest.buildConfig; const auto mark = state.markDirectiveTail(pkg.manifest); @@ -2029,7 +2134,7 @@ static std::expected step9_member_build_programs(PrepareState const bool exclusiveBefore = bc.runExclusive; auto bp = mcpp::build::run_build_program( pkg.manifest, pkg.root, host->first, host->second, - pkg.manifest.cppStandard, bpEnv); + pkg.manifest.cppStandard, bpEnv, made.compiled ? &made : nullptr); if (!bp) { if (!state.overrides.plan_only) return std::unexpected(std::format( @@ -2277,10 +2382,11 @@ static std::expected step9_rerun_input_prepare_dir(PrepareSta if (f.enabled && !f.flags.empty()) dst += std::format(" ({})", f.flags); if (!f.enabled) dst += std::format(" ({})", f.reason); } - std::println("c++fly on {}: {}; enabled: {}; skipped: {}", - state.tc->label(), state.stdFlagAndDialect, - enabled.empty() ? "(none)" : enabled, - skipped.empty() ? "(none)" : skipped); + // Narration, like every line that says what the build is doing. + mcpp::ui::line(std::format("c++fly on {}: {}; enabled: {}; skipped: {}", + state.tc->label(), state.stdFlagAndDialect, + enabled.empty() ? "(none)" : enabled, + skipped.empty() ? "(none)" : skipped)); } for (auto& f : mcpp::toolchain::cppfly::effective_dialect_flags( *state.tc, state.m->cppStandard.experimental, diff --git a/src/build/progress.cppm b/src/build/progress.cppm index e52da6b37..c08557d46 100644 --- a/src/build/progress.cppm +++ b/src/build/progress.cppm @@ -201,8 +201,16 @@ void configurations(std::size_t n); void program_scheduled(std::string_view package, bool requested); void program_compiling(std::string_view package, bool requested); void program_running(std::string_view package, bool requested); +// `compiledAside`: the program was compiled in the concurrent compile phase of +// a workspace's programs (#748, B2), whose wall time `programs_compiled` states +// once. Its own `compile` is still shown on its line, and is not added to the +// command's program time a second time. void program_finished(std::string_view package, bool requested, ProgramOutcome outcome, - std::chrono::milliseconds compile, std::chrono::milliseconds run); + std::chrono::milliseconds compile, std::chrono::milliseconds run, + bool compiledAside = false); +// The wall time of the concurrent compile phase of a workspace's programs: +// compiles that overlapped are counted once, as the time they took together. +void programs_compiled(std::chrono::milliseconds wall); void programs_done(); // The validations after ninja. @@ -1106,7 +1114,7 @@ mcpp::ui::Frame frame() { r.lastDone = done; // The screen takes 25 columns; a terminal narrower than 60 keeps the // counts and the clock instead. - if (mcpp::platform::terminal::cols() >= 60) { + if (mcpp::platform::terminal::cols(mcpp::ui::narration_stream()) >= 60) { screen::Screen sc; r.animation->draw(sc); cells = sc.render(r.animationColour); @@ -1160,8 +1168,9 @@ std::unique_ptr choose_animation() { // `--play-game[=NAME]` (revision 3, §5.14): the CLI publishes the request as // MCPP_PLAY_GAME (`random` or a name). The game needs what the screen needs, -// and keys: standard input and standard output on a terminal, with mcpp in -// its foreground. Otherwise it is off, and one line says why. +// and keys: standard input and the stream the screen is drawn on (standard +// error) on a terminal, with mcpp in its foreground. Otherwise it is off, and +// one line says why. void choose_game(Report& r, std::vector& notes) { auto want = mcpp::platform::env::get("MCPP_PLAY_GAME").value_or(""); if (want.empty()) return; @@ -1183,7 +1192,8 @@ void choose_game(Report& r, std::vector& notes) { "draws braille, without --quiet and MCPP_PROGRESS=plain or off, is needed)"); return; } - auto keys = std::make_unique(); + auto keys = std::make_unique( + mcpp::ui::narration_stream()); if (!keys->active()) { notes.push_back("--play-game: standard input is not a terminal in the foreground, " "so no key can be read"); @@ -1263,7 +1273,8 @@ void program_running(std::string_view package, bool /*requested*/) { } void program_finished(std::string_view package, bool /*requested*/, ProgramOutcome outcome, - std::chrono::milliseconds compile, std::chrono::milliseconds run) { + std::chrono::milliseconds compile, std::chrono::milliseconds run, + bool compiledAside) { auto& r = report(); std::vector out; { @@ -1273,7 +1284,7 @@ void program_finished(std::string_view package, bool /*requested*/, ProgramOutco p.outcome = outcome; p.compile = compile; p.run = run; - r.programTime += compile + run; + r.programTime += run + (compiledAside ? std::chrono::milliseconds{0} : compile); // A program whose result is reused did no work (revision 3, §7.1). if (outcome != ProgramOutcome::Cached || r.verbose) out.push_back(program_line(r, p)); } @@ -1284,6 +1295,12 @@ void program_finished(std::string_view package, bool /*requested*/, ProgramOutco : std::format("compiled {}ms ran {}ms", compile.count(), run.count()))); } +void programs_compiled(std::chrono::milliseconds wall) { + auto& r = report(); + std::lock_guard lock(r.m); + r.programTime += wall; +} + // The build programs are done and the plan continues. Measured with // 2026.9.29.5: the status row read `Running build programs` for 13 s after the // only program finished, because nothing returned the phase to planning. diff --git a/src/cli.cppm b/src/cli.cppm index 212b96d74..790d17dc8 100644 --- a/src/cli.cppm +++ b/src/cli.cppm @@ -387,8 +387,8 @@ int run(int argc, char** argv) { .help("Target no accelerator, ignoring [build] accel")) .option(cl::Option("static").help( "Force static linking (-static). On Linux, prefer pairing with --target -linux-musl")) - .option(cl::Option("package").short_name('p').takes_value().value_name("NAME") - .help("Build only the named workspace member (namespace.name or package name, then directory)")) + .option(cl::Option("package").short_name('p').takes_value().multiple().value_name("NAME") + .help("Build the named workspace member (namespace.name or package name, then directory); repeat to build several")) .option(cl::Option("profile").takes_value().value_name("NAME") .help("Build profile: dev (default) | release | dist | <[profile.*] name>")) .option(cl::Option("release").help("Shorthand for --profile release")) @@ -401,6 +401,8 @@ int run(int argc, char** argv) { .help("Treat manifest schema warnings (unknown feature/platform) as errors")) .option(cl::Option("workspace") .help("Build all workspace members")) + .option(cl::Option("exclude").takes_value().multiple().value_name("NAME") + .help("With --workspace (or at a virtual workspace root), leave the named member out; repeatable, refused with -p")) .option(cl::Option("play-game") .help("Play a game in the status row while it builds: --play-game=snake|stack|runner, or one at random")) .action(wrap_rc(cmd_build))) @@ -433,8 +435,8 @@ int run(int argc, char** argv) { // `run` accepted, and scripts written against it must keep working. .option(cl::Option("target-triple").takes_value().value_name("TRIPLE") .help("Alias for --target")) - .option(cl::Option("package").short_name('p').takes_value().value_name("NAME") - .help("Run only the named workspace member (namespace.name or package name, then directory; single-member, no --workspace fan-out)")) + .option(cl::Option("package").short_name('p').takes_value().multiple().value_name("NAME") + .help("Run only the named workspace member (namespace.name or package name, then directory; one member: a second -p is refused)")) // DECLARED ON THE THREE COMMANDS THAT BUILD BEFORE THEY ACT, AS ON // `build`. The value has always reached them: the pre-parse loop // above publishes it as MCPP_TOOLCHAIN for every command, and @@ -526,7 +528,7 @@ int run(int argc, char** argv) { .option(cl::Option("build-timeout").takes_value().value_name("SECS") .help("Kill a compile/link drive still running after SECS seconds (default 0 = no limit; POSIX only)")) .option(cl::Option("workspace-timeout").takes_value().value_name("SECS") - .help("Stop the --workspace fan-out after SECS seconds and report what did run (default 0 = no limit)")) + .help("Start no further member's tests after SECS seconds since the command began, and report what did not run (default 0 = no limit); the build is bounded by --build-timeout")) .option(cl::Option("profile").takes_value().value_name("NAME") .help("Build profile for the test build: dev (default) | release | dist | <[profile.*] name>")) .option(cl::Option("features").takes_value().value_name("LIST") @@ -535,8 +537,8 @@ int run(int argc, char** argv) { .help("Pin capability providers (e.g. blas=openblas,lapack=mkl)")) .option(cl::Option("strict") .help("Treat manifest schema warnings (unknown feature/platform) as errors")) - .option(cl::Option("package").short_name('p').takes_value().value_name("NAME") - .help("Run tests only for the named workspace member (namespace.name or package name, then directory)")) + .option(cl::Option("package").short_name('p').takes_value().multiple().value_name("NAME") + .help("Run the tests of the named workspace member (namespace.name or package name, then directory); repeat to test several")) .option(cl::Option("toolchain").takes_value().value_name("SPEC") .help("Build the tests with this toolchain for one invocation, e.g. llvm@22.1.8")) .option(cl::Option("cache").takes_value().value_name("MODE") @@ -545,6 +547,8 @@ int run(int argc, char** argv) { .help("Deprecated alias for --cache=off (also clears the build dir)")) .option(cl::Option("workspace") .help("Run tests for all workspace members")) + .option(cl::Option("exclude").takes_value().multiple().value_name("NAME") + .help("With --workspace (or at a virtual workspace root), leave the named member out; repeatable, refused with -p")) .option(cl::Option("play-game") .help("Play a game in the status row while it builds: --play-game=snake|stack|runner, or one at random")) .action(wrap_rc([&passthrough](const cl::ParsedArgs& p) { @@ -637,7 +641,7 @@ int run(int argc, char** argv) { .help("tar (default; .zip for a Windows target) | dir | any " "format the resolved graph provides (e.g. appimage, msi)")) .option(cl::Option("output").short_name('o').takes_value() - .help("Override output path")) + .help("Override output path; with several members, the directory each archive or tree is written below")) // Packaging builds RELEASE by default — the artifact leaves this // machine. `[build] default-profile` still wins when it is set; // this only replaces the "dev" fallback every other command uses. @@ -657,8 +661,12 @@ int run(int argc, char** argv) { .help("Output format: human (default) | json (one mcpp.pack envelope on stdout; narration on stderr)")) .option(cl::Option("no-strip") .help("Ship the artifacts as built (default: strip debug info)")) - .option(cl::Option("package").short_name('p').takes_value().value_name("NAME") - .help("Pack the named workspace member (namespace.name or package name, then directory), as if run in its directory")) + .option(cl::Option("package").short_name('p').takes_value().multiple().value_name("NAME") + .help("Pack the named workspace member (namespace.name or package name, then directory), as if run in its directory; repeat to pack several, planned and built once")) + .option(cl::Option("workspace") + .help("Pack every workspace member that has a program target, planned and built once")) + .option(cl::Option("exclude").takes_value().multiple().value_name("NAME") + .help("With --workspace, leave the named member out; repeatable, refused with -p")) .option(cl::Option("debug-symbols").takes_value().value_name("DIR") .help("Write the separated *.debug files here (default: discard)")) .action(wrap_rc(cmd_pack))) @@ -704,8 +712,8 @@ int run(int argc, char** argv) { .option(cl::Option("no-accel") .help("Describe the variant built for no accelerator")) .option(cl::Option("static").help("Describe the build with --static")) - .option(cl::Option("package").short_name('p').takes_value().value_name("NAME") - .help("Describe only the named workspace member (namespace.name or package name, then directory)")) + .option(cl::Option("package").short_name('p').takes_value().multiple().value_name("NAME") + .help("Describe the named workspace member (namespace.name or package name, then directory); repeat to describe several")) .option(cl::Option("profile").takes_value().value_name("NAME") .help("Build profile: dev (default) | release | dist | <[profile.*] name>")) .option(cl::Option("release").help("Shorthand for --profile release")) @@ -717,7 +725,9 @@ int run(int argc, char** argv) { .option(cl::Option("strict") .help("Treat manifest schema warnings (unknown feature/platform) as errors")) .option(cl::Option("workspace") - .help("Describe all workspace members in one document"))) + .help("Describe all workspace members in one document")) + .option(cl::Option("exclude").takes_value().multiple().value_name("NAME") + .help("With --workspace (or at a virtual workspace root), leave the named member out; repeatable, refused with -p"))) .action(wrap_rc([&dispatch_sub](const cl::ParsedArgs& p) { return dispatch_sub("emit", p, {{"xpkg", cmd_emit_xpkg}, {"sbom", mcpp::cli::cmd_sbom}, diff --git a/src/cli/cmd_build.cppm b/src/cli/cmd_build.cppm index 0e3f94de7..9fee26df3 100644 --- a/src/cli/cmd_build.cppm +++ b/src/cli/cmd_build.cppm @@ -19,6 +19,7 @@ import mcpp.build.coff_exports; import mcpp.build.stage; import mcpp.build.schedule.detach_codegen; import mcpp.build.test_targets; +import mcpp.cli.selection; import mcpp.build.build_database; import mcpp.build.build_program; import mcpp.build.progress; // the report every building command opens @@ -39,103 +40,22 @@ import mcpp.wire; namespace mcpp::cli { -// Decide whether a build/test invocation acts on several workspace members, and -// if so which. It does when `--workspace` is given, or at a *virtual* workspace -// root with no `-p` (the intuitive "act on the whole workspace"). Returns the -// member paths as `[workspace] members` writes them -- a rooted workspace's own -// package first, as "." (workspace design 2026-09-29 §7.1) -- or nullopt for -// the single-package / single-`-p` / rooted-bare path. Inside a member, the -// workspace is the one that lists it. -std::optional> -workspace_fanout_members(bool wantAll, const std::string& package_filter) { - auto root = mcpp::project::find_manifest_root(std::filesystem::current_path()); - if (!root) return std::nullopt; - auto m = mcpp::manifest::load(*root / "mcpp.toml"); - if (m && !m->workspace.present && wantAll) { - auto wsRoot = mcpp::project::find_workspace_root(*root); - if (wsRoot.empty()) return std::nullopt; - m = mcpp::manifest::load(wsRoot / "mcpp.toml"); - } - if (!m || !m->workspace.present || m->workspace.members.empty()) return std::nullopt; - bool virtualWs = m->package.name.empty(); - if (!(wantAll || (virtualWs && package_filter.empty()))) return std::nullopt; - std::vector members; - if (!virtualWs) members.push_back("."); - members.insert(members.end(), m->workspace.members.begin(), m->workspace.members.end()); - return members; -} - -// The workspace a build command acts on and the members it selects (workspace -// design 2026-09-29 §7.1, §15): `--workspace`, and a virtual root without -// `-p`, select every member (a rooted workspace's own package first, as "."); -// `-p X` selects X; a command in a member's directory selects that member; a -// command at a rooted workspace's root selects the workspace's own package. -// nullopt outside a workspace. -struct WorkspaceSelection { - std::filesystem::path root; - std::vector members; -}; -std::expected, std::string> -workspace_selection(bool wantAll, const std::string& package_filter) { - auto root = mcpp::project::find_manifest_root(std::filesystem::current_path()); - if (!root) return std::optional{}; - auto m = mcpp::manifest::load(*root / "mcpp.toml", {.insideWorkspace = true}); - if (!m) return std::optional{}; - WorkspaceSelection sel{*root, {}}; - std::string inside; - if (!m->workspace.present) { - auto wsRoot = mcpp::project::find_workspace_root(*root); - if (wsRoot.empty()) return std::optional{}; - sel.root = wsRoot; - m = mcpp::manifest::load(wsRoot / "mcpp.toml"); - if (!m || !m->workspace.present) return std::optional{}; - const auto rel = root->lexically_normal() - .lexically_relative(wsRoot.lexically_normal()); - for (auto const& mp : m->workspace.members) - if (std::filesystem::path(mp).lexically_normal() == rel) inside = mp; - if (inside.empty()) return std::optional{}; - } - const bool rooted = !m->package.name.empty(); - std::vector all; - if (rooted) all.push_back("."); - all.insert(all.end(), m->workspace.members.begin(), m->workspace.members.end()); - if (wantAll) { sel.members = std::move(all); return sel; } - if (!package_filter.empty()) { - auto dir = mcpp::project::resolve_member_dir(*m, sel.root, package_filter); - if (!dir) return std::unexpected(dir.error()); - auto rel = dir->empty() ? std::string(".") - : dir->lexically_normal().lexically_relative(sel.root.lexically_normal()).generic_string(); - if (rel.empty()) rel = "."; - for (auto const& mp : m->workspace.members) - if (std::filesystem::path(mp).lexically_normal() == std::filesystem::path(rel)) - rel = mp; - sel.members = {rel}; - return sel; - } - if (!inside.empty()) { sel.members = {inside}; return sel; } - if (!rooted) { sel.members = std::move(all); return sel; } - sel.members = {"."}; - return sel; -} - -// The workspace a fan-out acts on, and its members grouped by configuration -// (workspace design 2026-09-29 §15): members whose root-position values are -// equal are planned together, in one graph, in one build directory. -std::expected>, std::string> -workspace_groups(const std::filesystem::path& wsRoot, const std::vector& members) { - auto ws = mcpp::manifest::load(wsRoot / "mcpp.toml"); - if (!ws) return std::unexpected(ws.error().format()); - std::vector> groups; - std::map byKey; - for (auto const& mp : members) { - auto mm = mcpp::project::load_member_manifest(*ws, wsRoot, mp); - if (!mm) return std::unexpected(mm.error()); - const auto key = mcpp::project::root_position_key(*mm); - auto [it, fresh] = byKey.try_emplace(key, groups.size()); - if (fresh) groups.emplace_back(); - groups[it->second].push_back(mp); - } - return groups; +// A member's tests, discovered from the member's own directory. Discovery +// resolves the path it is given as `-p` would, so a member whose path is also +// another member's package name would be read as that member: a member is +// discovered from its own directory or refused, and never from another's. +std::expected +discover_member_tests(const std::filesystem::path& wsRoot, const std::string& mp) { + auto d = mcpp::build::discover_test_targets(wsRoot, mp); + if (!d) return d; + std::error_code ec; + const bool same = std::filesystem::equivalent(d->packageRoot, wsRoot / mp, ec); + if (!ec && !same) + return std::unexpected(std::format( + "the path '{}' is also the name of another member's package, so its tests " + "cannot be told from that member's; select it by its package name (-p )", + mp)); + return d; } // The tests of each member of a configuration group, for a plan of the group @@ -149,7 +69,7 @@ std::expected group_tests(const std::filesystem::path& wsRoot, const std::vector& group) { GroupTests out; for (auto const& mp : group) { - auto d = mcpp::build::discover_test_targets(wsRoot, mp); + auto d = discover_member_tests(wsRoot, mp); if (!d) return std::unexpected(std::format("{}: {}", mp, d.error())); if (!d->targets.empty()) out.targets[mp] = std::move(d->targets); out.discovery.emplace_back(d->packageRoot, std::move(d->discover)); @@ -321,7 +241,7 @@ export int cmd_build(const mcpplibs::cmdline::ParsedArgs& parsed) { // Groups are independent (a member reached from two groups is a node of // each), and a failed group does not stop the others; the first non-zero // exit wins. - auto selection = workspace_selection(parsed.is_flag_set("workspace"), ov.package_filter); + auto selection = mcpp::cli::select_members(member_request(parsed)); if (!selection) { mcpp::ui::error(std::format("{}", selection.error())); return 2; } if (*selection) { auto const& members = (*selection)->members; @@ -609,7 +529,8 @@ export int cmd_emit_build_database(const mcpplibs::cmdline::ParsedArgs& parsed) requests.push_back({mp, {}, std::move(one)}); }; std::filesystem::path wsRoot = *root; - auto selection = workspace_selection(parsed.is_flag_set("workspace"), ov.package_filter); + const auto request = member_request(parsed); + auto selection = mcpp::cli::select_members(request); if (!selection) return failed("MCPP_BUILD_DATABASE_PLAN_FAILED", selection.error()); if (*selection && (*selection)->members.size() > 1) { @@ -629,9 +550,10 @@ export int cmd_emit_build_database(const mcpplibs::cmdline::ParsedArgs& parsed) requests.push_back({g.size() == 1 ? g.front() : std::string{}, g, std::move(mo)}); } } - } else if (auto members = workspace_fanout_members(parsed.is_flag_set("workspace"), - ov.package_filter)) { - plan_alone(members->front()); + } else if (*selection && (*selection)->whole) { + // The whole workspace, which lists one member (or one is left by + // `--exclude`): that member's plan, as a selection of several plans. + plan_alone((*selection)->members.front()); } else { requests.push_back({std::string{}, {}, ov}); } @@ -676,8 +598,9 @@ export int cmd_emit_build_database(const mcpplibs::cmdline::ParsedArgs& parsed) } }; { - // Planning narrates on stdout and may start programs that inherit it; - // the document is printed after this scope, alone. + // mcpp narrates on standard error, but planning may start programs + // that inherit standard output (build programs, installers); the + // document is printed after this scope, alone. mcpp::platform::terminal::StdoutToStderr narration; // A group whose plan fails is planned member by member (appended to // `requests` and reached by this same loop), so a member's failure @@ -825,12 +748,21 @@ export int cmd_emit_build_database(const mcpplibs::cmdline::ParsedArgs& parsed) } // One line per selector: a value never spans lines, and a `\x1f` separator // before `f`, `c` or `a` reads as a longer hex escape (clang refuses it). + // The members the flags name, each as written and in command-line order. + // One `-p` reads as it always has, so the fingerprint of a command that + // names one member does not change; `--exclude` adds a line only when it + // is given. + std::string packages; + for (auto const& p : request.packages) packages += (packages.empty() ? "" : ",") + p; + std::string excludes; + for (auto const& e : request.excludes) excludes += (excludes.empty() ? "" : ",") + e; const auto selector = std::format( "spec={}\ntarget={}\ntoolchain={}\nprofile={}\nfeatures={}\n" - "cap={}\naccel={}\nstatic={}\npackage={}\nworkspace={}", + "cap={}\naccel={}\nstatic={}\npackage={}\nworkspace={}{}", spec, ov.target_triple, mcpp::platform::env::get("MCPP_TOOLCHAIN").value_or(""), ov.profile, ov.features, ov.capabilities, ov.accel, ov.force_static, - ov.package_filter, parsed.is_flag_set("workspace")); + packages, parsed.is_flag_set("workspace"), + excludes.empty() ? std::string{} : std::format("\nexclude={}", excludes)); auto rendered = mcpp::build::database::render(members, failedMemberRoots, wsRoot, selector); // A note's severity is its own (E3's program-failure note is an error; @@ -884,9 +816,23 @@ export int cmd_run(const mcpplibs::cmdline::ParsedArgs& parsed, if (parsed.positional_count() > 0) targetName = parsed.positional(0); // -p/--package : scope to one workspace member, same flag/rule // as `mcpp build -p` / `mcpp test -p` (mcpp::project::resolve_member_dir). - // `mcpp run` is single-member only — no `--workspace` fan-out. + // `mcpp run` is single-member only — no `--workspace` fan-out — because an + // artifact to execute is one program. The option is repeatable on the + // commands that act on several members, so a second `-p` here is refused, + // naming every member asked for, and never read as "the last one". + const auto packages = parsed.option_or_empty("package").values; + if (packages.size() > 1) { + std::string named; + for (std::size_t i = 0; i < packages.size(); ++i) + named += std::format("{}'{}'", + i == 0 ? "" : (i + 1 == packages.size() ? " and " : ", "), packages[i]); + mcpp::ui::error(std::format( + "mcpp run runs one program, so it acts on one workspace member, and -p names {}: " + "pass one -p (mcpp build and mcpp test accept several)", named)); + return 2; + } std::string package_filter; - if (auto p = parsed.value("package")) package_filter = *p; + if (!packages.empty()) package_filter = packages.front(); std::string cache_mode; bool no_cache = parsed.is_flag_set("no-cache"); if (auto c = parsed.value("cache")) cache_mode = *c; @@ -992,11 +938,19 @@ export int cmd_test(const mcpplibs::cmdline::ParsedArgs& parsed, } } - // Workspace fan-out: test every member through run_tests (which scopes its - // discovery to the member). Continue-on-failure + per-member summary so one - // red member never hides the rest. - if (auto members = workspace_fanout_members(parsed.is_flag_set("workspace"), - ov.package_filter)) { + // The members this command tests, by the one selection every command reads + // (`mcpp::cli::select_members`). A selection of one member, named or implied + // by the directory, is that member's own test run, as it always was. A + // selection of several members, or of the whole workspace, fans out: the + // members are planned once per configuration group and each group is built + // once, then each member's tests run in member order, continuing past a + // failing member so that one red member never hides the rest (member + // selection design 2026-09-30, S4). + auto selection = mcpp::cli::select_members(member_request(parsed)); + if (!selection) { mcpp::ui::error(std::format("{}", selection.error())); return 2; } + if (*selection && ((*selection)->whole || (*selection)->members.size() > 1)) { + auto const& sel = **selection; + auto const& members = sel.members; const bool json = (to.format == mcpp::build::TestMessageFormat::Json); // Silence the ui BEFORE the first member, not inside run_tests. The // quiet flag used to be set by run_tests itself, so the fan-out's own @@ -1023,28 +977,37 @@ export int cmd_test(const mcpplibs::cmdline::ParsedArgs& parsed, const long long wsDeadlineMs = static_cast(workspaceTimeoutSecs) * 1000; - std::size_t idx = 0; - for (auto& mp : *members) { - ++idx; - // Checked BEFORE starting a member rather than after: stopping - // mid-member would leave a half-built member reported as neither - // run nor skipped. + // Asked before a member's tests start. The deadline is checked BEFORE + // starting a member rather than after: stopping mid-member would leave + // a half-tested member reported as neither run nor skipped. It is + // measured from the start of the command, so the build the members + // share counts against it, and a member not started by then is listed + // as not run; the build itself is bounded by --build-timeout. + auto member_begin = [&](std::size_t i, const std::string& mp) -> bool { if (wsDeadlineMs > 0 && ws_ms() >= wsDeadlineMs) { notRun.push_back(mp); - continue; + return false; } - mcpp::build::BuildOverrides mo = ov; - mo.package_filter = mp; mcpp::ui::status("Workspace", - std::format("testing member '{}' ({}/{})", mp, idx, members->size())); - mcpp::build::TestRunSummary sum; - int r = mcpp::build::run_tests(passthrough, mo, to, &sum); + std::format("testing member '{}' ({}/{})", mp, i + 1, members.size())); + return true; + }; + // The member's line: how it ended, and how long its own tests ran. The + // build is the group's, stated once by the group's line, so a member's + // line does not state it again. + auto member_end = [&](std::size_t i, const std::string& mp, int r, + const mcpp::build::TestRunSummary& sum) { + const auto idx = i + 1; totalPassed += sum.passed; totalFailed += sum.failed; totalNotRun += sum.notRun; totalBuilt += sum.built; - memberTimes.emplace_back(mp, sum.elapsedMs); - auto secs = static_cast(sum.elapsedMs) / 1000.0; + // Ranked by its run, which is the member's own: a build a group + // shares belongs to no one member. + memberTimes.emplace_back(mp, sum.buildGroup >= 0 ? sum.runMs : sum.elapsedMs); + auto secs = static_cast(sum.buildGroup >= 0 ? sum.runMs : sum.elapsedMs) / 1000.0; + const char* took = sum.buildGroup >= 0 ? "run " : ""; + const char* in = sum.buildGroup >= 0 ? ", " : " in "; if (r == 2 && sum.failed == 0 && sum.notRun > 0) { // Built and not executed (#544): the member did not fail, and // it did not pass. 2 outranks 0 and yields to 1, as it does @@ -1052,25 +1015,79 @@ export int cmd_test(const mcpplibs::cmdline::ParsedArgs& parsed, if (rc == 0) rc = 2; unrunnable.push_back(mp); mcpp::ui::status("Workspace", - std::format("member '{}' ({}/{}) NOT RUN — {} passed, {} not run in {:.2f}s", - mp, idx, members->size(), sum.passed, sum.notRun, secs)); + std::format("member '{}' ({}/{}) NOT RUN — {} passed, {} not run{}{}{:.2f}s", + mp, idx, members.size(), sum.passed, sum.notRun, in, took, secs)); } else if (r != 0) { rc = r; failed.push_back(mp); mcpp::ui::status("Workspace", - std::format("member '{}' ({}/{}) FAILED — {} passed, {} failed in {:.2f}s", - mp, idx, members->size(), sum.passed, sum.failed, secs)); + std::format("member '{}' ({}/{}) FAILED — {} passed, {} failed{}{}{:.2f}s", + mp, idx, members.size(), sum.passed, sum.failed, in, took, secs)); } else { // Under `--no-run` nothing passed and nothing was meant to: // reporting "0 passed" for a member whose tests all built is // the same sentence a member with no tests would produce. mcpp::ui::status("Workspace", sum.built - ? std::format("member '{}' ({}/{}) ok — {} built, not run in {:.2f}s", - mp, idx, members->size(), sum.built, secs) - : std::format("member '{}' ({}/{}) ok — {} passed in {:.2f}s", - mp, idx, members->size(), sum.passed, secs)); + ? std::format("member '{}' ({}/{}) ok — {} built, not run{}{}{:.2f}s", + mp, idx, members.size(), sum.built, in, took, secs) + : std::format("member '{}' ({}/{}) ok — {} passed{}{}{:.2f}s", + mp, idx, members.size(), sum.passed, in, took, secs)); + } + }; + + if (to.list) { + // A listing builds nothing, so there is nothing to plan once: each + // member lists its own tests. + for (std::size_t i = 0; i < members.size(); ++i) { + if (!member_begin(i, members[i])) continue; + mcpp::build::BuildOverrides mo = ov; + mo.package_filter = members[i]; + mcpp::build::TestRunSummary sum; + int r = mcpp::build::run_tests(passthrough, mo, to, &sum); + member_end(i, members[i], r, sum); + } + } else { + // Each member's own tests, discovered from the member's own + // directory: two members may each have a `tests/main.cpp`. + std::vector inputs; + for (auto const& mp : members) { + mcpp::build::WorkspaceTestMember wm; + wm.path = mp; + auto d = discover_member_tests(sel.root, mp); + if (!d) { + wm.error = std::format("{}: {}", mp, d.error()); + } else { + wm.targets = std::move(d->targets); + if (wm.targets.empty()) { + // Names where it looked when the manifest chose the + // place, so that a glob that matches nothing is not + // read as a project without tests. + if (d->discoverDeclared) { + std::string globs; + for (auto const& g : d->discover) + globs += std::format("{}\"{}\"", globs.empty() ? "" : ", ", g); + wm.noTests = std::format("no tests found ([test] discover = [{}])", globs); + } else { + wm.noTests = "no tests found in tests/"; + } + } + } + inputs.push_back(std::move(wm)); } + // Members whose root-position values are equal are planned together, + // as `mcpp build` plans them. A member whose manifest cannot be read + // has no configuration: each member is then its own group, and it + // fails alone when it is planned. + std::vector> groups; + if (auto g = workspace_groups(sel.root, members)) groups = std::move(*g); + else for (auto const& mp : members) groups.push_back({mp}); + + mcpp::build::WorkspaceTestHooks hooks; + hooks.begin = member_begin; + hooks.end = member_end; + mcpp::build::run_workspace_tests(passthrough, ov, to, sel.root, groups, + std::move(inputs), hooks); } auto wsElapsed = ws_ms(); @@ -1101,7 +1118,7 @@ export int cmd_test(const mcpplibs::cmdline::ParsedArgs& parsed, "\"tests_not_run\":{},\"tests_built\":{}," "\"failed_members\":[{}],\"unrunnable_members\":[{}]," "\"not_run\":[{}],\"elapsed_ms\":{}}}}}", - members->size(), totalPassed, totalFailed, totalNotRun, + members.size(), totalPassed, totalFailed, totalNotRun, totalBuilt, join(failed), join(unrunnable), join(notRun), wsElapsed); std::fflush(stdout); @@ -1134,15 +1151,15 @@ export int cmd_test(const mcpplibs::cmdline::ParsedArgs& parsed, if (totalBuilt) notRunCounts += std::format("; {} built, not run", totalBuilt); if (failed.empty() && notRun.empty() && unrunnable.empty()) - mcpp::ui::status("workspace result", + mcpp::ui::result("workspace result", std::format("ok. {} member(s); {} passed; 0 failed{}; finished in {:.2f}s", - members->size(), totalPassed, notRunCounts, + members.size(), totalPassed, notRunCounts, static_cast(wsElapsed) / 1000.0)); else mcpp::ui::error(std::format( "workspace test: {}/{} member(s) failed; {} passed; {} failed{}; " "finished in {:.2f}s", - failed.size(), members->size(), totalPassed, totalFailed, notRunCounts, + failed.size(), members.size(), totalPassed, totalFailed, notRunCounts, static_cast(wsElapsed) / 1000.0)); if (!failed.empty()) mcpp::ui::plain(std::format(" failed members: {}", join_names(failed))); diff --git a/src/cli/cmd_publish.cppm b/src/cli/cmd_publish.cppm index aac28a609..b61d3388a 100644 --- a/src/cli/cmd_publish.cppm +++ b/src/cli/cmd_publish.cppm @@ -12,6 +12,7 @@ import mcpp.build.advice; import mcpplibs.cmdline; import mcpp.build.prepare; // profile_override_from_flags import mcpp.build.progress; // the build's report (build progress design 2026-09-29) +import mcpp.cli.selection; // which members `pack` acts on, as `build` plans them import mcpp.log; import mcpp.libs.json; import mcpp.pack; @@ -224,6 +225,14 @@ export int cmd_pack(const mcpplibs::cmdline::ParsedArgs& parsed) { env.data = nullptr; env.diagnostics.push_back({"MCPP_PACK_FAILED", mcpp::wire::Severity::Error, "mcpp pack did not produce a package; the reason is on standard error"}); + // A pack of several members names each member that failed, which the + // code above cannot: the others may have been packed. + if (outcome.members.size() > 1) + for (auto const& m : outcome.members) + if (m.rc != 0) + env.diagnostics.push_back({"MCPP_PACK_FAILED", mcpp::wire::Severity::Error, + std::format("mcpp pack did not produce a package for member '{}'; " + "the reason is on standard error", m.name)}); mcpp::wire::emit(env); return rc; } @@ -242,9 +251,30 @@ export int cmd_pack(const mcpplibs::cmdline::ParsedArgs& parsed) { }; nlohmann::json artifacts = nlohmann::json::array(); nlohmann::json stage = nullptr; + nlohmann::json stages = nullptr; if (libraryRoute) { const auto fmt = parsed.value("format").value_or("tar"); artifacts.push_back(artifact_json(library.artifact, fmt, library.targets)); + } else if (outcome.members.size() > 1) { + // Several members (member selection design 2026-09-30, K1). Fields are + // added and none is redefined (docs/50 §7): every member's artifacts + // are listed, each naming its member, and `stage`, which names one + // tree, stays null while `stages` names one per member. + stages = nlohmann::json::array(); + for (auto const& m : outcome.members) { + for (auto const& a : m.artifacts) { + auto j = artifact_json(a, m.format, m.targets); + j["member"] = m.name; + artifacts.push_back(std::move(j)); + } + if (!m.stageDir.empty()) + stages.push_back(nlohmann::json{ + {"member", m.name}, + {"dir", m.stageDir.lexically_normal().string()}, + {"manifest", m.stageManifest.lexically_normal().string()}, + {"closure", m.closure}, + }); + } } else { for (auto const& a : outcome.artifacts) artifacts.push_back(artifact_json(a, outcome.format, outcome.targets)); @@ -256,12 +286,122 @@ export int cmd_pack(const mcpplibs::cmdline::ParsedArgs& parsed) { }; } env.data = nlohmann::json{{"artifacts", std::move(artifacts)}, {"stage", std::move(stage)}}; + if (!stages.is_null()) env.data["stages"] = std::move(stages); mcpp::wire::emit(env); return 0; } namespace { +// The members `mcpp pack` packs, from the selectors of its command line (member +// selection design 2026-09-30, K1). +// +// A command without a selector acts on the package of its directory, as it +// always has, and so does a selection of one member: `-p X` packs X as if the +// command ran in X's directory, which is what the command does. Several members +// are planned and built once, each in its own stage. For `pack`, "every member" +// means every member with a program target to pack: `--workspace` skips a member +// that has none, and `-p` names a member that has none in a refusal. +struct PackMembers { + // Set when several members are packed; null for one, which is the package of + // the directory the command runs in, entered when `-p` named it. + std::optional several; + int rc = 0; // non-zero: refused, and reported +}; + +PackMembers pack_members(const mcpplibs::cmdline::ParsedArgs& parsed) { + PackMembers out; + const auto request = mcpp::cli::member_request(parsed); + auto selection = mcpp::cli::select_members(request); + if (!selection) { + mcpp::ui::error(selection.error()); + out.rc = 2; + return out; + } + if (!*selection) { + // Outside a workspace there is nothing for `-p` to name. + if (!request.packages.empty()) { + auto root = mcpp::project::find_manifest_root(std::filesystem::current_path()); + if (!root) { + mcpp::ui::error("-p needs a workspace; no mcpp.toml was found here or above"); + } else if (auto rm = mcpp::manifest::load(*root / "mcpp.toml"); !rm) { + mcpp::ui::error(rm.error().format()); + } else { + mcpp::ui::error(std::format("-p {}: {} is not a workspace", + request.packages.front(), root->string())); + } + out.rc = 2; + } + return out; + } + const auto& sel = **selection; + const bool selected = request.all || !request.packages.empty() || !request.excludes.empty(); + if (!selected) return out; + + // The members with a program to pack. + auto ws = mcpp::manifest::load(sel.root / "mcpp.toml"); + if (!ws) { + mcpp::ui::error(ws.error().format()); + out.rc = 2; + return out; + } + std::vector packable; + for (auto const& mp : sel.members) { + auto mm = mcpp::project::load_member_manifest(*ws, sel.root, mp); + if (!mm) { + mcpp::ui::error(mm.error()); + out.rc = 2; + return out; + } + if (std::ranges::any_of(mm->targets, [](auto const& t) { return t.is_program(); })) { + packable.push_back(mp); + continue; + } + // One member named is packed as it always was, a library package + // included; with several, a member that has no program is skipped when + // the selection is every member, and refused by name when `-p` named it. + if (sel.members.size() == 1) packable.push_back(mp); + else if (sel.whole) + mcpp::ui::info("Skipping", std::format( + "{}: no program target to pack", mm->package.name)); + else { + mcpp::ui::error(std::format( + "member '{}' has no program target to pack: `mcpp pack` over several members " + "packs programs.\n Pack it alone with `-p {}`, or leave it out.", + mm->package.name, mp)); + out.rc = 2; + return out; + } + } + if (packable.empty()) { + mcpp::ui::error("no workspace member has a program target to pack"); + out.rc = 2; + return out; + } + + if (packable.size() == 1) { + // One member: the package of its directory, as if the command ran there. + // A relative `--output` keeps meaning the directory the user typed it in + // (the caller resolves it before this changes the directory). + std::error_code ec; + std::filesystem::current_path(sel.root / packable.front(), ec); + if (ec) { + mcpp::ui::error(std::format("-p {}: cannot enter {}: {}", packable.front(), + (sel.root / packable.front()).string(), ec.message())); + out.rc = 2; + } + return out; + } + auto groups = mcpp::cli::workspace_groups(sel.root, packable); + if (!groups) { + mcpp::ui::error(groups.error()); + out.rc = 2; + return out; + } + out.several = mcpp::pack::MemberPack{sel.root, std::move(*groups), {}, packable}; + return out; +} + int cmd_pack_body(const mcpplibs::cmdline::ParsedArgs& parsed, mcpp::pack::PackOutcome* report, mcpp::pack::LibraryPackReport* libraryReport, @@ -270,31 +410,14 @@ int cmd_pack_body(const mcpplibs::cmdline::ParsedArgs& parsed, // other `-p` uses, and the pack then runs in its directory, so the result // is by construction the one `mcpp pack` in that directory produces. A // relative `--output` keeps meaning the directory the user typed it in. + std::optional typedOutput; + if (auto v = parsed.value("output")) typedOutput = std::filesystem::absolute(*v); + const auto startedIn = std::filesystem::current_path(); + auto members = pack_members(parsed); + if (members.rc != 0) return members.rc; std::optional outputFromUser; - if (auto pkg = parsed.option_or_empty("package").value(); !pkg.empty()) { - auto root = mcpp::project::find_manifest_root(std::filesystem::current_path()); - if (!root) { - mcpp::ui::error("-p needs a workspace; no mcpp.toml was found here or above"); - return 2; - } - auto rm = mcpp::manifest::load(*root / "mcpp.toml"); - if (!rm) { mcpp::ui::error(rm.error().format()); return 2; } - auto member = mcpp::project::resolve_member_dir(*rm, *root, pkg); - if (!member) { mcpp::ui::error(member.error()); return 2; } - if (member->empty()) { - mcpp::ui::error(std::format("-p {}: {} is not a workspace", pkg, root->string())); - return 2; - } - if (auto v = parsed.value("output")) - outputFromUser = std::filesystem::absolute(*v); - std::error_code ec; - std::filesystem::current_path(*member, ec); - if (ec) { - mcpp::ui::error(std::format("-p {}: cannot enter {}: {}", pkg, - member->string(), ec.message())); - return 2; - } - } + if (typedOutput && std::filesystem::current_path() != startedIn) + outputFromUser = typedOutput; // ─── Resolve mode ──────────────────────────────────────────────── mcpp::pack::Options opts; bool modeFromUser = false; @@ -357,6 +480,48 @@ int cmd_pack_body(const mcpplibs::cmdline::ParsedArgs& parsed, if (auto o = parsed.option("target")) triples = o->get().values; if (!triples.empty()) opts.targetTriple = triples.back(); + // ─── Several members ───────────────────────────────────────────── + // + // Each is refused before anything is planned, let alone compiled: the + // command names one thing where several members are packed, and the + // members cannot share it. + if (members.several) { + if (const auto name = parsed.positional(0); !name.empty()) { + mcpp::ui::error(std::format( + "a target name cannot be given when several members are packed: '{}' names " + "a target of one package.\n" + " use: -p {}, or leave the name out to pack each member's program", + name, name)); + return 2; + } + if (triples.size() > 1) { + mcpp::ui::error( + "--target may be given once when several members are packed: each member's " + "program is built for one target,\n" + " and a pack for several targets stages one member's legs into one tree.\n" + " use: pack that member alone with -p "); + return 2; + } + opts.output.clear(); + if (typedOutput) { + std::error_code ec; + if (std::filesystem::exists(*typedOutput, ec) + && !std::filesystem::is_directory(*typedOutput, ec)) { + mcpp::ui::error(std::format( + "--output names a file, '{}', and several members are packed.\n" + " Each member's archive or tree is written below the directory --output " + "names, under the name\n" + " it has by default. use: --output ", typedOutput->string())); + return 2; + } + members.several->outputDir = *typedOutput; + } + auto out = mcpp::pack::build_and_pack_members(std::move(opts), modeFromUser, + *members.several); + if (report) *report = out; + return out.rc; + } + // ─── Which target, and therefore which kind of package ─────────── // // The positional is a target NAME. Its `kind` decides everything: a diff --git a/src/cli/selection.cppm b/src/cli/selection.cppm new file mode 100644 index 000000000..db1165422 --- /dev/null +++ b/src/cli/selection.cppm @@ -0,0 +1,229 @@ +// mcpp.cli.selection — which workspace members a command acts on. +// +// `build`, `test`, `mcpp emit build-database` and `pack` read the same +// selection from the same flags, so the members a command plans are the ones +// the flags name whatever the command is (member selection design 2026-09-30, +// S1). The selection is a SET of members: it is kept in `[workspace] members` +// order whatever order `-p` names them in, and a member named twice is +// selected once, so the plan does not depend on how the command line was +// spelled. +// +// The function is split in two. `select_members(ws, ...)` reads nothing but +// the workspace manifest it is given and its members' manifests, which is what +// the unit tests exercise; the overload that takes a directory finds the +// workspace the directory belongs to and the member the directory is inside, +// and is the one the commands call. + +export module mcpp.cli.selection; + +import std; +import mcpplibs.cmdline; +import mcpp.manifest; +import mcpp.project; + +export namespace mcpp::cli { + +// The selectors of a command line, as written. +struct MemberRequest { + bool all = false; // --workspace + std::vector packages; // every -p/--package, in command-line order + std::vector excludes; // every --exclude, in command-line order +}; + +// The workspace a command acts on and the members it selects. +struct MemberSelection { + std::filesystem::path root; + // Each member as `[workspace] members` spells it, in that order; a rooted + // workspace's own package comes first, as "." (workspace design + // 2026-09-29 §7.1). + std::vector members; + // True when the selection is the workspace itself less what `--exclude` + // removed: `--workspace`, or a virtual root without `-p`. A command that + // reports per member (`test`) reports in its fan-out form for such a + // selection even when the workspace has one member. + bool whole = false; +}; + +// The member path `value` names, spelled as `[workspace] members` spells it +// ("." for a rooted workspace's own package). The value is resolved in the +// order docs/07 §5.3 states, by `mcpp::project::resolve_member_dir`: a +// qualified name, then a package name (refused, naming every match, when +// several members share it), then a directory. +std::expected +member_path_of(const mcpp::manifest::Manifest& ws, + const std::filesystem::path& wsRoot, + std::string_view value) { + auto dir = mcpp::project::resolve_member_dir(ws, wsRoot, value); + if (!dir) return std::unexpected(dir.error()); + std::string rel = "."; + if (!dir->empty()) { + const auto u8 = dir->lexically_normal() + .lexically_relative(wsRoot.lexically_normal()) + .generic_u8string(); + rel.assign(reinterpret_cast(u8.data()), u8.size()); + if (rel.empty()) rel = "."; + } + // The spelling the manifest uses, so that "./libs/core" and "libs/core" + // are one member to every reader of the selection. + for (auto const& mp : ws.workspace.members) + if (std::filesystem::path(mp).lexically_normal() == std::filesystem::path(rel)) + return mp; + return rel; +} + +// The selection over a workspace whose manifest is `ws` and whose root is +// `wsRoot`. `inside` is the member, as `[workspace] members` spells it, that +// the command's directory is inside; empty at the workspace root. +// +// | request | members | +// |------------------------------------------|----------------------------------| +// | `--workspace` | every member | +// | a virtual root, no `-p` | every member | +// | a rooted root, no `-p` | "." | +// | inside member X, no `-p` | X | +// | `-p X -p Y` | {X, Y} | +// | an "all" form above, with `--exclude Z` | every member but Z | +// +// Refused, before anything is planned: `--workspace` together with `-p`, a +// `-p` that names no member (the refusal lists the members) or several (it +// names every match), `--exclude` together with `-p`, `--exclude` without an +// "all" form, an `--exclude` that names no member, and an `--exclude` that +// removes every member. +std::expected +select_members(const mcpp::manifest::Manifest& ws, + const std::filesystem::path& wsRoot, + std::string_view inside, + const MemberRequest& req) { + // `--workspace` selects every member and `-p` names some of them. The two + // together state two selections, and taking either one would drop the + // other without a word, which is the defect a repeated `-p` had (#750). + if (req.all && !req.packages.empty()) + return std::unexpected(std::string( + "--workspace cannot be combined with -p: --workspace selects every member, " + "and -p names the members a command acts on")); + if (!req.excludes.empty() && !req.packages.empty()) + return std::unexpected(std::string( + "--exclude cannot be combined with -p: -p names the members a command acts on, " + "and --exclude removes members from a whole-workspace selection")); + + const bool rooted = !ws.package.name.empty(); + std::vector all; + if (rooted) all.push_back("."); + all.insert(all.end(), ws.workspace.members.begin(), ws.workspace.members.end()); + + MemberSelection sel{wsRoot, {}, false}; + if (req.all) { + sel.members = all; + sel.whole = true; + } else if (!req.packages.empty()) { + std::set named; + for (auto const& p : req.packages) { + auto mp = member_path_of(ws, wsRoot, p); + if (!mp) return std::unexpected(mp.error()); + named.insert(std::move(*mp)); + } + // Manifest order, and once each: the selection is a set. + for (auto const& mp : all) + if (named.contains(mp)) sel.members.push_back(mp); + } else if (!inside.empty()) { + sel.members = {std::string(inside)}; + } else if (!rooted) { + sel.members = all; + sel.whole = true; + } else { + sel.members = {"."}; + } + + if (sel.members.empty()) + return std::unexpected(std::string("the workspace lists no members")); + if (req.excludes.empty()) return sel; + + if (!sel.whole) + return std::unexpected(std::string( + "--exclude removes members from a whole-workspace selection: " + "add --workspace (a virtual workspace root selects every member without it)")); + std::set removed; + for (auto const& e : req.excludes) { + auto mp = member_path_of(ws, wsRoot, e); + if (!mp) return std::unexpected(std::format("--exclude '{}': {}", e, mp.error())); + removed.insert(std::move(*mp)); + } + std::erase_if(sel.members, [&](const std::string& mp) { return removed.contains(mp); }); + if (sel.members.empty()) + return std::unexpected(std::string( + "--exclude removes every member of the workspace, so nothing is left to act on")); + return sel; +} + +// The selection for a command run in `cwd`: nullopt outside a workspace (the +// command then acts on the one package it is in). Inside a member's directory +// the workspace is the one that lists the member, and `--workspace` there +// still means the whole of it. +std::expected, std::string> +select_members(const MemberRequest& req, + const std::filesystem::path& cwd = std::filesystem::current_path()) { + auto root = mcpp::project::find_manifest_root(cwd); + if (!root) return std::optional{}; + auto m = mcpp::manifest::load(*root / "mcpp.toml", {.insideWorkspace = true}); + // A manifest that cannot be read is reported by the planner, with its own + // diagnostic; it is not a selection question. + if (!m) return std::optional{}; + + auto outside = [&]() -> std::expected, std::string> { + if (!req.excludes.empty()) + return std::unexpected(std::string( + "--exclude names members of a workspace, and this directory is not in one")); + return std::optional{}; + }; + + std::filesystem::path wsRoot = *root; + std::string inside; + if (!m->workspace.present) { + wsRoot = mcpp::project::find_workspace_root(*root); + if (wsRoot.empty()) return outside(); + m = mcpp::manifest::load(wsRoot / "mcpp.toml"); + if (!m || !m->workspace.present) return outside(); + const auto rel = root->lexically_normal().lexically_relative(wsRoot.lexically_normal()); + for (auto const& mp : m->workspace.members) + if (std::filesystem::path(mp).lexically_normal() == rel) inside = mp; + if (inside.empty()) return outside(); + } + auto sel = select_members(*m, wsRoot, inside, req); + if (!sel) return std::unexpected(sel.error()); + return std::optional{std::move(*sel)}; +} + + +// The selectors of a command line, as `mcpp::cli::select_members` reads them +// (member selection design 2026-09-30, S1): every command that acts on members +// reads its `-p`, `--workspace` and `--exclude` the same way, so the members a +// command plans are the ones the flags name whatever the command is. +MemberRequest member_request(const mcpplibs::cmdline::ParsedArgs& parsed) { + MemberRequest req; + req.all = parsed.is_flag_set("workspace"); + req.packages = parsed.option_or_empty("package").values; + req.excludes = parsed.option_or_empty("exclude").values; + return req; +} + +// The workspace a fan-out acts on, and its members grouped by configuration +// (workspace design 2026-09-29 §15): members whose root-position values are +// equal are planned together, in one graph, in one build directory. +std::expected>, std::string> +workspace_groups(const std::filesystem::path& wsRoot, const std::vector& members) { + auto ws = mcpp::manifest::load(wsRoot / "mcpp.toml"); + if (!ws) return std::unexpected(ws.error().format()); + std::vector> groups; + std::map byKey; + for (auto const& mp : members) { + auto mm = mcpp::project::load_member_manifest(*ws, wsRoot, mp); + if (!mm) return std::unexpected(mm.error()); + const auto key = mcpp::project::root_position_key(*mm); + auto [it, fresh] = byKey.try_emplace(key, groups.size()); + if (fresh) groups.emplace_back(); + groups[it->second].push_back(mp); + } + return groups; +} + +} // namespace mcpp::cli diff --git a/src/config.cppm b/src/config.cppm index 57f669e56..ed8c05412 100644 --- a/src/config.cppm +++ b/src/config.cppm @@ -328,13 +328,15 @@ namespace { // bootstrap status lines line up under the cyan "Downloading …" lines // produced via the BootstrapProgressCallback. We can't import mcpp.ui // from here (cyclic dep), so this is a tiny duplicate of that helper — -// no color, no fanciness. +// no color, no fanciness. It writes to standard error, which is the stream +// mcpp.ui narrates on: a status line is never a command's result, and a +// pipe that reads the result must not receive it. void print_status(std::string_view verb, std::string_view msg) { constexpr std::size_t W = 12; if (verb.size() >= W) { - std::println("{} {}", verb, msg); + std::println(stderr, "{} {}", verb, msg); } else { - std::println("{}{} {}", std::string(W - verb.size(), ' '), verb, msg); + std::println(stderr, "{}{} {}", std::string(W - verb.size(), ' '), verb, msg); } } @@ -954,8 +956,8 @@ bool ensure_project_index_dir( repo.artifact = spec.artifact; repo.source = spec.source; } else if (!spec.artifact.empty()) { - std::println("warning: [indices].{}: artifact source ignored " - "(rev/tag/branch/path pins force git)", name); + std::println(stderr, "warning: [indices].{}: artifact source ignored " + "(rev/tag/branch/path pins force git)", name); } customRepos.push_back(std::move(repo)); } diff --git a/src/fallback/xlings_binary.cppm b/src/fallback/xlings_binary.cppm index d17d64795..51e020f09 100644 --- a/src/fallback/xlings_binary.cppm +++ b/src/fallback/xlings_binary.cppm @@ -176,13 +176,14 @@ acquire_xlings_binary(const std::filesystem::path& destBin, bool quiet = false, std::error_code ec; std::filesystem::create_directories(destBin.parent_path(), ec); - // Right-pad verb to 12 columns (matches mcpp::ui::verb_padded layout). + // Right-pad verb to 12 columns (matches mcpp::ui::verb_padded layout), on + // standard error: the stream mcpp.ui narrates on. auto print_status = [](std::string_view verb, std::string_view msg) { constexpr std::size_t W = 12; if (verb.size() >= W) - std::println("{} {}", verb, msg); + std::println(stderr, "{} {}", verb, msg); else - std::println("{}{} {}", std::string(W - verb.size(), ' '), verb, msg); + std::println(stderr, "{}{} {}", std::string(W - verb.size(), ' '), verb, msg); }; // The first acquisition takes the source a replacement would take diff --git a/src/pack/pipeline.cppm b/src/pack/pipeline.cppm index 13321aba0..73dc263f1 100644 --- a/src/pack/pipeline.cppm +++ b/src/pack/pipeline.cppm @@ -1,6 +1,8 @@ -// mcpp.pack.pipeline — pack orchestration: build (re-preparing for musl static -// when needed), pick the main binary, plan + run the bundler. -// Bodies moved verbatim from the CLI layer. Zero behavior change. +// mcpp.pack.pipeline — pack orchestration: the plan of each configuration group +// of the members being packed (re-preparing for musl static when needed), what is +// refused before anything is compiled, one build per group, then for each member +// its program, its plan and the bundler, and one dispatch pass per group. The +// package of one directory is a group of one member, and runs the same steps. module; #include @@ -60,6 +62,22 @@ export struct PackOutcome { std::string closure; // Whether a build program ran in this pack, for the envelope's `effects`. bool ranBuildPrograms = false; + // What one packed member answered (member selection design 2026-09-30, + // K1). A pack of several members reports each in member order, and the + // fields above then describe the pack as a whole: `artifacts` holds every + // member's, and the stage fields, which name one tree, are the first + // member's. A pack of one member holds that member here as well. + struct Member { + std::string name; // qualified package name + int rc = 0; + std::vector artifacts; + std::string format; + std::vector targets; + std::filesystem::path stageDir; + std::filesystem::path stageManifest; + std::string closure; + }; + std::vector members; }; // #634 A3: the two directory lists an Android row's closure is read against, @@ -250,175 +268,131 @@ void report_packed(const std::vector& outputs, e.outputs == 1 ? "file" : "files")); } -// Everything after CLI option parsing for `mcpp pack`. +// What `mcpp pack` over several workspace members packs (member selection +// design 2026-09-30, K1), as the command selected it. // -// `wantTarget` is the target NAME the user asked for, empty when they did not. -// It exists because `mcpp pack ` now routes on `[targets.].kind`: -// a name that resolves to a program has to reach the binary selection below, -// or a project with two `bin` targets would accept `mcpp pack app2` and -// silently bundle app1 — the shape where the command succeeds and the answer -// is wrong. -// -// `extraLegs` (#630 A9) is every OTHER triple a several-`--target` app-pack -// request named, already built by `build_extra_android_legs` above. This call -// still does exactly one build — of `opts.targetTriple`, the PRIMARY leg — and -// stages the primary's own artifact and the declared deploy files as it always -// has; `extraLegs`, when non-empty, only changes WHERE the primary's shared -// object lands (`Plan::extraSharedLegs`, read by `run_shared_program`) and adds -// the other legs, each with its own closure, beside it in the same staged tree, -// before the one dispatch pass runs. Empty for a single-`--target` pack. -export PackOutcome build_and_pack(Options opts, bool modeFromUser, - const std::string& wantTarget = {}, - std::vector extraLegs = {}) { - // `--target *-linux-musl` without an explicit `--mode` implies - // `--mode static` — packaging a musl-static ELF as bundle-project - // would feed patchelf a static binary and crash. The docs treat - // this pair as equivalent; surface it in the code path too. - if (!modeFromUser && opts.targetTriple.find("-musl") != std::string::npos) { - opts.mode = mcpp::pack::Mode::Static; - modeFromUser = true; // user-equivalent intent — block manifest override - } - - // ─── Build first (pack implies a fresh build) ──────────────────── - mcpp::build::BuildOverrides ov; - if (opts.mode == mcpp::pack::Mode::Static && opts.targetTriple.empty()) - ov.target_triple = "x86_64-linux-musl"; - else - ov.target_triple = opts.targetTriple; - // A bundled program leaves this machine: release is the fallback, not dev. - // `[build] default-profile` still decides when the project states one. - ov.profile = opts.profile; - ov.profile_fallback = "release"; - // The same features on this pass and on the dispatched format's second - // pass below, which reuses `ov`. - ov.features = opts.features; +// The members are planned as the selection `mcpp build` plans: once per +// configuration group, one graph and one build for the members of a group, so +// a package that several members reach is compiled once and its build program +// runs once. Packing each member alone would plan the graph, run the build +// programs and start the build again for each (#749). +export struct MemberPack { + // The workspace's root. + std::filesystem::path root; + // The packed members, each as `[workspace] members` spells it, by + // configuration group: members whose root-position values are equal are + // planned together. Every member has a program target to pack. + std::vector> groups; + // `--output` when several members are packed: the directory each member's + // archive or tree is written below, under the name it would have by + // default. Empty: each member's own `target/dist`. + std::filesystem::path outputDir; + // Every packed member, as `[workspace] members` spells it, in that order: + // the order the members are reported in, whichever group each is in. + std::vector order; +}; - // QUIET FOR A DISPATCHED FORMAT, AND ONLY UNTIL THE VALUE IS VALIDATED. - // - // `mcpp pack --format bogus` must write NOTHING to stdout and exit 2 -- - // the machine-output contract, asserted by - // tests/e2e/202_machine_output_contract.sh, because this is the path a - // client hits when it probes an mcpp for a capability. The set of valid - // values is a property of the resolved graph, so the refusal cannot be - // decided until prepare has run, and prepare narrates what it resolves. - // - // Nothing is lost when the value IS valid: the dispatch pass prepares a - // second time and prints the same lines, so a successful - // `pack --format ` narrates once rather than twice. - const bool quietUntilValidated = - opts.format == mcpp::pack::Format::Dispatched && !mcpp::ui::is_quiet(); - if (quietUntilValidated) mcpp::ui::set_quiet(true); - auto ctx = mcpp::build::prepare_build(/*print_fp=*/false, /*includeDevDeps=*/false, - /*extraTargets=*/{}, ov); - if (quietUntilValidated) mcpp::ui::set_quiet(false); - if (!ctx) { - mcpp::ui::error(ctx.error()); - return PackOutcome{2}; - } +// One member of a pack, and what each step learned of it. A project that is not +// a workspace member has one, which is the plan's own package. +struct MemberJob { + // The member's qualified package name. + std::string name; + // The member as `[workspace] members` spells it; empty outside a workspace. + std::string path; + // Whether the plan holds workspace members, and the member is read through + // `with_member`. False for a plain project, whose plan is its own view. + bool inWorkspace = false; + // This member's request: the command's, with the mode the member's manifest + // and the command line resolved to. + Options request; + bool modeFromUser = false; + // The request with what the plan answered added to it (`member_plan`). + Options opts; + // The program the member packs, from the member's link units. + std::filesystem::path mainBinary; + bool programIsSharedObject = false; + std::optional plan; + // Set only when NO tree exists at all; see `stage_member`. + std::string stageFailure; + ClosureStatus closure; + mcpp::ui::PathContext pathCtx; + bool ranBuildPrograms = false; + // What the member's pack answered: its status, artifacts and stage. + PackOutcome::Member result; +}; - // Manifest may override mode only when neither --mode nor an - // equivalent flag (--target *-musl → static) was given. - // A workspace plan's subject is its one selected member (§15). - const auto& subjectManifest = ctx->workspaceMembers.size() == 1 - ? ctx->workspaceMembers.front().manifest : ctx->manifest; - if (!modeFromUser && !subjectManifest.packConfig.defaultMode.empty()) { - if (auto m = mcpp::pack::parse_mode(subjectManifest.packConfig.defaultMode)) - opts.mode = *m; - } +// One configuration group: the plan of its members, what produced the plan, +// and the members. A plan of one package, or of the member a command runs in, +// is a group of one member. +struct GroupJob { + // The members as `[workspace] members` spells them; empty for the package + // of the directory the command runs in. + std::vector paths; + // What produced `ctx`. It stays the record of it, because the dispatch pass + // prepares again with the same overrides plus the packaging pass's values. + mcpp::build::BuildOverrides ov; + std::optional ctx; + std::vector members; + // The group failed as a whole: its plan, or its build. + int rc = 0; +}; - // Re-derive target triple: if mode is Static we force the musl - // triple even when the manifest's [pack].default_mode bumped us - // here after `prepare_build` ran with the host toolchain. - // - // ...but NOT over a target the user asked for. `--mode static` on its own - // has always meant "the musl-static ELF", and that stays; `--mode static - // --target x86_64-windows-gnu` used to silently become a Linux build, - // which was invisible while PE packaging did not exist and is a wrong - // answer now that it does. An explicit `--target` is an instruction. - if (opts.mode == mcpp::pack::Mode::Static - && opts.targetTriple.empty() - && ctx->tc.targetTriple.find("-musl") == std::string::npos) { - // Need to re-prepare the build with the musl target. - // - // `ov` IS MUTATED RATHER THAN SHADOWED. It has to stay the record of - // what produced `ctx`, because the dispatch pass below re-enters - // prepare with the same overrides plus two fields -- and a second - // overrides object left behind here would make that pass differ from - // this build in a way nothing states. - ov.target_triple = "x86_64-linux-musl"; - // Quiet on the same grounds as the first prepare: this one also runs - // before `--format` has been validated. - if (quietUntilValidated) mcpp::ui::set_quiet(true); - auto ctx2 = mcpp::build::prepare_build(false, false, {}, ov); - if (quietUntilValidated) mcpp::ui::set_quiet(false); - if (!ctx2) { mcpp::ui::error(ctx2.error()); return PackOutcome{2}; } - ctx = std::move(ctx2); - } +// What the command asked for, and what every group of it shares. +struct PackRun { + Options opts; + bool modeFromUser = false; + std::string wantTarget; + std::vector extraLegs; + const MemberPack* selection = nullptr; + // More than one member is packed, so a line about one names it. + bool several = false; + // The value of a dispatched `--format` is not known to be one until prepare + // has run; see `prepare_group`. + bool quietUntilValidated = false; + std::optional cfg; +}; - // ─── Is the requested format one anything provides? ────────────── - // - // BEFORE THE BUILD, because a refusal that arrives after a full compile is - // a worse refusal, and because this is the earliest point at which it can - // be exact: build programs have now run and declared what they provide. - // - // The set is read from a pass that asked for NOTHING. That is what the - // "declare unconditionally, submit conditionally" rule buys -- a member - // that declared only when asked would leave this list empty exactly when a - // user names a format, and the refusal would name nothing. - if (opts.format == mcpp::pack::Format::Dispatched) { - auto const& provided = ctx->plan.providedPackFormats; - if (std::ranges::find(provided, opts.formatName) == provided.end()) { - std::string avail; - for (auto b : mcpp::pack::kBuiltinPackFormats) - avail += (avail.empty() ? "" : ", ") + std::string(b); - for (auto const& f : provided) { - if (mcpp::pack::is_builtin_pack_format(f)) continue; - avail += ", " + f; - } - mcpp::ui::error(std::format( - "unknown --format '{}'.\n" - " available in this build: {}\n" - " A format past `tar` and `dir` comes from a package in the " - "resolved graph, which declares\n" - " it with `mcpp::provides_pack_format(\"\")` in its build " - "program. Add the package\n" - " that provides '{}' to [build-dependencies] and activate its " - "feature.", - opts.formatName, avail, opts.formatName)); - return PackOutcome{2}; - } - } +// A refusal: what the command exits with, and the message. The message is +// reported where the refusal is made. +struct Refusal { + int rc = 0; + std::string message; +}; - // A package claiming a built-in name is silently unreachable, since the - // parser resolves `tar` and `dir` before consulting the graph at all. - // - // OUTSIDE THE DISPATCH BRANCH ABOVE, because the mistake is in the PACKAGE - // and does not depend on what this invocation asked for. Reported on every - // pack, so the author hears it on the plain `mcpp pack` they are most - // likely to run. - for (auto const& f : ctx->plan.providedPackFormats) - if (mcpp::pack::is_builtin_pack_format(f)) - mcpp::ui::warning(std::format( - "a package in this graph declares `mcpp:pack-format={}`, which " - "is one of the archive shapes `mcpp pack` owns; `--format {}` " - "will always select the built-in and never that package", f, f)); +// A line about a member names it when several are packed, and stays as it has +// always been for one. +std::string about(const PackRun& run, const MemberJob& m, std::string_view text) { + return run.several ? std::format("member '{}': {}", m.name, text) : std::string(text); +} - auto be = mcpp::build::make_ninja_backend(); - mcpp::build::BuildOptions bo; - auto br = be->build(ctx->plan, bo); - if (!br) { - // The compiler's own output, not just "build failed" — same reason as - // in the library pipeline. - if (!br.error().diagnosticOutput.empty()) { - std::fputs(br.error().diagnosticOutput.c_str(), stderr); - if (br.error().diagnosticOutput.back() != '\n') std::fputs("\n", stderr); - } - mcpp::ui::error(br.error().message); - return PackOutcome{1}; +// Does the member's closure carry this program shipped through `artifacts`? In a +// plan of several members each program a member ships is placed in that member's +// product directory (`BuildPlan::LinkGroup::placements`), and the member's own +// files are the ones that directory holds. +bool member_carries(const mcpp::build::BuildContext& ctx, const MemberJob& m, + const mcpp::build::LinkUnit& u) { + if (!m.inWorkspace || ctx.workspaceMembers.size() < 2) return true; + for (auto const& g : ctx.plan.linkGroups) { + if (g.linkOnly || g.member != m.name) continue; + if (u.output.parent_path() == g.productDir) return true; + return std::ranges::any_of(g.placements, + [&](auto const& pl) { return pl.source == u.output; }); } - // Everything below reads the package being packed: in a workspace plan, - // its selected member (workspace design 2026-09-29 §15). - mcpp::build::focus_on_member(*ctx); + return false; +} + +// The program a member packs, and everything the member's pack reads from the +// plan: its request resolved against the flags, and the pack plan. It reads the +// plan and no built file but the program's format (`make_plan`), so it is asked +// before the build of a pack of several members, for what it refuses and for +// where it writes, and again after it, for the plan `pack::run` executes. +// +// Called inside the member's view (`with_member`): `ctx.manifest`, +// `ctx.projectRoot` and the plan's link group are the member's. +std::optional member_plan(PackRun& run, GroupJob& g, MemberJob& m) { + auto& ctx = *g.ctx; + const bool sharedPlan = ctx.workspaceMembers.size() > 1; + Options opts = m.request; // ─── Pick the main binary target ───────────────────────────────── // @@ -433,64 +407,67 @@ export PackOutcome build_and_pack(Options opts, bool modeFromUser, // everywhere else this record sweeps (§2.3). `dependencyOwned` is // excluded: a dependency's own `shared` target contributes a link unit to // this plan too, and it is never the package being packed. + // + // In a plan of several members the link units are every member's, and a + // member packs the ones `memberOf` names it for. auto is_program_link_unit = [&](const mcpp::build::LinkUnit& lu) { + if (sharedPlan && lu.memberOf != m.name) return false; // A dependency's program shipped beside this one (mcpp#711) is staged // as a file, never packed as the program. if (lu.kind == mcpp::build::LinkUnit::Binary) return lu.artifactOf.empty(); if (lu.kind != mcpp::build::LinkUnit::SharedLibrary || lu.dependencyOwned) return false; - for (auto const& t : ctx->manifest.targets) + for (auto const& t : ctx.manifest.targets) if (t.name == lu.targetName) return t.kind == mcpp::manifest::Target::Application; return false; }; std::filesystem::path mainBinary; const mcpp::build::LinkUnit* chosenLu = nullptr; - if (!wantTarget.empty()) { - for (auto& lu : ctx->plan.linkUnits) { - if (is_program_link_unit(lu) && lu.targetName == wantTarget) { - mainBinary = ctx->outputDir / lu.output; + if (!run.wantTarget.empty()) { + for (auto& lu : ctx.plan.linkUnits) { + if (is_program_link_unit(lu) && lu.targetName == run.wantTarget) { + mainBinary = ctx.outputDir / lu.output; chosenLu = &lu; break; } } - if (mainBinary.empty()) { - mcpp::ui::error(std::format( - "target '{}' is not a program in this build", wantTarget)); - return PackOutcome{2}; - } + if (mainBinary.empty()) + return Refusal{2, about(run, m, std::format( + "target '{}' is not a program in this build", run.wantTarget))}; } - for (auto& lu : ctx->plan.linkUnits) { + for (auto& lu : ctx.plan.linkUnits) { if (!mainBinary.empty()) break; - if (is_program_link_unit(lu) && lu.targetName == ctx->manifest.package.name) { - mainBinary = ctx->outputDir / lu.output; + if (is_program_link_unit(lu) && lu.targetName == ctx.manifest.package.name) { + mainBinary = ctx.outputDir / lu.output; chosenLu = &lu; break; } } if (mainBinary.empty()) { // Fall back to the first binary target if package.name doesn't match. - for (auto& lu : ctx->plan.linkUnits) { + for (auto& lu : ctx.plan.linkUnits) { if (is_program_link_unit(lu)) { - mainBinary = ctx->outputDir / lu.output; + mainBinary = ctx.outputDir / lu.output; chosenLu = &lu; break; } } } - if (mainBinary.empty()) { - mcpp::ui::error("no binary target to pack"); - return PackOutcome{1}; - } + if (mainBinary.empty()) + return Refusal{1, about(run, m, "no binary target to pack")}; // Passed to `make_plan` rather than re-derived from the file: `make_plan` // has only `mainBinary` and would otherwise have to ask the triple and // the manifest the same question a second time. const bool programIsSharedObject = chosenLu && chosenLu->kind == mcpp::build::LinkUnit::SharedLibrary; - auto cfg = mcpp::config::load_or_init(/*quiet=*/false, - mcpp::fetcher::make_bootstrap_progress_callback()); - if (!cfg) { mcpp::ui::error(cfg.error().message); return PackOutcome{4}; } + if (!run.cfg) { + auto cfg = mcpp::config::load_or_init(/*quiet=*/false, + mcpp::fetcher::make_bootstrap_progress_callback()); + if (!cfg) return Refusal{4, cfg.error().message}; + run.cfg = std::move(*cfg); + } // ─── What the build promised, and where its runtime lives ──────── // @@ -502,7 +479,7 @@ export PackOutcome build_and_pack(Options opts, bool modeFromUser, // /MD, say) must not make the package behave as though it had been // honoured. { - const auto flags = mcpp::build::compute_flags(ctx->plan); + const auto flags = mcpp::build::compute_flags(ctx.plan); opts.carryToolchainRuntime = flags.contractByRole[static_cast( mcpp::build::dist::Role::Distributable)] @@ -519,15 +496,15 @@ export PackOutcome build_and_pack(Options opts, bool modeFromUser, && !flags.programCxxRuntimeStated) { opts.carryToolchainRuntime = false; } - opts.toolchainRuntimeDirs = ctx->plan.toolchain.linkRuntimeDirs; - if (!ctx->plan.toolchain.msvcRedistDir.empty()) - opts.toolchainRuntimeDirs.push_back(ctx->plan.toolchain.msvcRedistDir); + opts.toolchainRuntimeDirs = ctx.plan.toolchain.linkRuntimeDirs; + if (!ctx.plan.toolchain.msvcRedistDir.empty()) + opts.toolchainRuntimeDirs.push_back(ctx.plan.toolchain.msvcRedistDir); // Where a third-party dependency's shared library may be found. Both // channels, because they answer for different things: the runtime // library dirs are what `mcpp run` puts on the loader's path, and the // link intent's search dirs are what a dependency package declared. - opts.depSearchDirs = ctx->plan.runtimeLibraryDirs; - for (auto const& d : ctx->plan.linkIntent.runtimeSearchDirs) + opts.depSearchDirs = ctx.plan.runtimeLibraryDirs; + for (auto const& d : ctx.plan.linkIntent.runtimeSearchDirs) opts.depSearchDirs.push_back(d); // What the build placed relative to the executable (#615): the // runtime placement resolver's answer (`CompileFlags::runtimeDeploy`), @@ -542,10 +519,10 @@ export PackOutcome build_and_pack(Options opts, bool modeFromUser, // copy. A copy found in a dependency's directory never enters a // package that the contract says the host serves. namespace rp = mcpp::build::runtime_placement; - const bool msvcAbi = mcpp::toolchain::is_msvc_target(ctx->plan.toolchain); + const bool msvcAbi = mcpp::toolchain::is_msvc_target(ctx.plan.toolchain); // Relative to the program's directory: `bin`, or a workspace member's // product directory (§15 of the 2026-09-29 workspace design). - const auto& productDir = ctx->plan.productDir; + const auto& productDir = ctx.plan.productDir; for (auto const& d : flags.runtimeDeploy) { if (msvcAbi && d.dest.parent_path() == productDir && rp::is_msvc_crt_name(d.dest.filename().string())) @@ -558,8 +535,8 @@ export PackOutcome build_and_pack(Options opts, bool modeFromUser, // `artifacts = [...]`) is linked into `bin/` beside the executable, so // it is staged the way a deployed file is. // In a workspace member's product directory it is the placed copy. - for (auto const& u : ctx->plan.linkUnits) - if (!u.artifactOf.empty()) + for (auto const& u : ctx.plan.linkUnits) + if (!u.artifactOf.empty() && member_carries(ctx, m, u)) opts.runtimeFiles.push_back(productDir == "bin" ? u.output.lexically_relative("bin") : std::filesystem::path(u.output.filename())); @@ -568,54 +545,286 @@ export PackOutcome build_and_pack(Options opts, bool modeFromUser, // link_library_dirs` is a file the link used and the device does not // have -- and against its driver's; see `driver_library_dirs`. if (programIsSharedObject) { - for (auto const& d : ctx->plan.linkIntent.linkLibraryDirs) + for (auto const& d : ctx.plan.linkIntent.linkLibraryDirs) opts.depSearchDirs.push_back(d); - for (auto const& d : ctx->plan.linkIntent.transitiveNeededDirs) + for (auto const& d : ctx.plan.linkIntent.transitiveNeededDirs) opts.depSearchDirs.push_back(d); - auto dirs = driver_library_dirs(ctx->tc); + auto dirs = driver_library_dirs(ctx.tc); opts.toolchainLibraryDirs = std::move(dirs.search); opts.platformLibraryDirs = std::move(dirs.platform); } } - // ─── Build the plan + run ──────────────────────────────────────── - auto plan = mcpp::pack::make_plan(ctx->manifest, *cfg, opts, - mainBinary, ctx->projectRoot, ctx->tc.targetTriple, + // ─── Build the plan ────────────────────────────────────────────── + auto plan = mcpp::pack::make_plan(ctx.manifest, *run.cfg, opts, + mainBinary, ctx.projectRoot, ctx.tc.targetTriple, // From the RESOLVED graph. `mcpp why runtime` on a real imgui project // lists `capability:opengl.glx.driver <- compat.glfw@3.4` — none of // which appears in the project's own manifest. - ctx->plan.runtimeRequirements, programIsSharedObject); - if (!plan) { mcpp::ui::error(plan.error().message); return PackOutcome{1}; } + ctx.plan.runtimeRequirements, programIsSharedObject); + if (!plan) return Refusal{1, about(run, m, plan.error().message)}; + // `--output

` with several members: each member's archive, or its tree + // for `--format dir`, is written below the directory under the name it would + // have by default. The staging tree of an archive, and of a dispatched + // format, stays in the member's `target/dist`, as it does for `-o `. + if (run.selection && !run.selection->outputDir.empty()) { + plan->archivePath = run.selection->outputDir / plan->archivePath.filename(); + if (opts.format == mcpp::pack::Format::Dir) + plan->stagingRoot = run.selection->outputDir / plan->stagingRoot.filename(); + } + m.opts = std::move(opts); + m.mainBinary = std::move(mainBinary); + m.programIsSharedObject = programIsSharedObject; + m.plan = std::move(*plan); + return std::nullopt; +} + +// Prepares one group of the pack: the plan of its members, and what is decided +// from it alone. +std::optional prepare_group(PackRun& run, GroupJob& g) { + // QUIET FOR A DISPATCHED FORMAT, AND ONLY UNTIL THE VALUE IS VALIDATED. + // + // `mcpp pack --format bogus` must write NOTHING to stdout and exit 2 -- + // the machine-output contract, asserted by + // tests/e2e/202_machine_output_contract.sh, because this is the path a + // client hits when it probes an mcpp for a capability. The set of valid + // values is a property of the resolved graph, so the refusal cannot be + // decided until prepare has run, and prepare narrates what it resolves. + // + // Nothing is lost when the value IS valid: the dispatch pass prepares a + // second time and prints the same lines, so a successful + // `pack --format ` narrates once rather than twice. + auto prepare = [&]() { + if (run.quietUntilValidated) mcpp::ui::set_quiet(true); + auto ctx = mcpp::build::prepare_build(/*print_fp=*/false, /*includeDevDeps=*/false, + /*extraTargets=*/{}, g.ov); + if (run.quietUntilValidated) mcpp::ui::set_quiet(false); + return ctx; + }; + auto ctx = prepare(); + if (!ctx) return Refusal{2, ctx.error()}; + g.ctx = std::move(*ctx); + + // The members of the group: the workspace members of its plan, in + // selection order, or the plan's own package. + if (g.ctx->workspaceMembers.empty()) { + MemberJob m; + m.name = mcpp::build::qualified_package_name(g.ctx->manifest); + g.members.push_back(std::move(m)); + } else { + for (auto const& wm : g.ctx->workspaceMembers) { + MemberJob m; + m.name = wm.name; + m.path = wm.memberPath; + m.inWorkspace = true; + g.members.push_back(std::move(m)); + } + } + auto manifest_of = [&](const MemberJob& m) -> const mcpp::manifest::Manifest& { + if (!m.inWorkspace) return g.ctx->manifest; + for (auto const& wm : g.ctx->workspaceMembers) + if (wm.name == m.name) return wm.manifest; + return g.ctx->manifest; + }; + // Manifest may override mode only when neither --mode nor an + // equivalent flag (--target *-musl → static) was given. + // A workspace plan's subject is its selected member (§15). + for (auto& m : g.members) { + m.request = run.opts; + m.modeFromUser = run.modeFromUser; + const auto& mf = manifest_of(m); + if (!m.modeFromUser && !mf.packConfig.defaultMode.empty()) { + if (auto mode = mcpp::pack::parse_mode(mf.packConfig.defaultMode)) + m.request.mode = *mode; + } + } + + // Re-derive target triple: if mode is Static we force the musl + // triple even when the manifest's [pack].default_mode bumped us + // here after `prepare_build` ran with the host toolchain. + // + // ...but NOT over a target the user asked for. `--mode static` on its own + // has always meant "the musl-static ELF", and that stays; `--mode static + // --target x86_64-windows-gnu` used to silently become a Linux build, + // which was invisible while PE packaging did not exist and is a wrong + // answer now that it does. An explicit `--target` is an instruction. + // + // One group is one target, so the members of a group that want the static + // row must all want it. + const bool anyStatic = std::ranges::any_of(g.members, [](const MemberJob& m) { + return m.request.mode == mcpp::pack::Mode::Static; }); + if (anyStatic + && run.opts.targetTriple.empty() + && g.ctx->tc.targetTriple.find("-musl") == std::string::npos) { + if (auto other = std::ranges::find_if(g.members, [](const MemberJob& m) { + return m.request.mode != mcpp::pack::Mode::Static; }); + other != g.members.end()) { + auto first = std::ranges::find_if(g.members, [](const MemberJob& m) { + return m.request.mode == mcpp::pack::Mode::Static; }); + return Refusal{2, std::format( + "member '{}' packs as --mode static, which links for the musl target, and " + "member '{}' does not, and the two share one build.\n" + " use: --mode for both, or pack them separately", + first->name, other->name)}; + } + // Need to re-prepare the build with the musl target. + // + // `ov` IS MUTATED RATHER THAN SHADOWED. It has to stay the record of + // what produced `ctx`, because the dispatch pass below re-enters + // prepare with the same overrides plus two fields -- and a second + // overrides object left behind here would make that pass differ from + // this build in a way nothing states. + g.ov.target_triple = "x86_64-linux-musl"; + // Quiet on the same grounds as the first prepare: this one also runs + // before `--format` has been validated. + auto ctx2 = prepare(); + if (!ctx2) return Refusal{2, ctx2.error()}; + g.ctx = std::move(*ctx2); + } + return std::nullopt; +} + +// Is the requested format one the group provides, and for which members? The +// refusals of a dispatched `--format`, before the build. +std::optional check_format(PackRun& run, GroupJob& g) { + auto const& ctx = *g.ctx; + const auto& formatName = run.opts.formatName; + // ─── Is the requested format one anything provides? ────────────── + // + // BEFORE THE BUILD, because a refusal that arrives after a full compile is + // a worse refusal, and because this is the earliest point at which it can be + // exact: build programs have now run and declared what they provide. + // + // The set is read from a pass that asked for NOTHING. That is what the + // "declare unconditionally, submit conditionally" rule buys -- a member + // that declared only when asked would leave this list empty exactly when a + // user names a format, and the refusal would name nothing. + if (run.opts.format == mcpp::pack::Format::Dispatched) { + auto const& provided = ctx.plan.providedPackFormats; + if (std::ranges::find(provided, formatName) == provided.end()) { + std::string avail; + for (auto b : mcpp::pack::kBuiltinPackFormats) + avail += (avail.empty() ? "" : ", ") + std::string(b); + for (auto const& f : provided) { + if (mcpp::pack::is_builtin_pack_format(f)) continue; + avail += ", " + f; + } + return Refusal{2, std::format( + "unknown --format '{}'.\n" + " available in this build: {}\n" + " A format past `tar` and `dir` comes from a package in the " + "resolved graph, which declares\n" + " it with `mcpp::provides_pack_format(\"\")` in its build " + "program. Add the package\n" + " that provides '{}' to [build-dependencies] and activate its " + "feature.", + formatName, avail, formatName)}; + } + // A member is packed into its own staged tree, by a provider that acts + // for it: the member's own build program, or a package that only that + // member reaches. A package several members reach has no one tree to + // name, and a member served by no provider would be reported as a pack + // that produced nothing once the whole build had been paid for. + if (g.members.size() > 1) { + std::string missing; + bool anyShared = false; + for (auto const& m : g.members) { + bool served = false; + std::string sharedProvider; + if (auto it = ctx.packFormatProviders.find(formatName); + it != ctx.packFormatProviders.end()) + for (auto const& p : it->second) { + auto reach = ctx.packReach.find(p); + if (reach == ctx.packReach.end()) continue; + const auto owner = mcpp::build::pack_owner(p, reach->second); + if (owner == m.name) served = true; + else if (owner.empty() && sharedProvider.empty() + && std::ranges::find(reach->second, m.name) + != reach->second.end()) + sharedProvider = p; + } + if (served) continue; + anyShared = anyShared || !sharedProvider.empty(); + missing += std::format("\n member '{}' has no package providing it{}", m.name, + sharedProvider.empty() ? std::string{} + : std::format(" ('{}' provides it, and other packed members reach it too)", + sharedProvider)); + } + if (!missing.empty()) + return Refusal{2, std::format( + "--format '{}' is not provided for every packed member.{}\n" + " A member is packed into its own staged tree by a provider in its own build " + "program, or in a\n" + " package that only that member reaches.{}", + formatName, missing, + anyShared + ? "\n A package that several packed members reach acts for none of them: " + "one run of its program\n" + " serves them all, and it is given no staged tree. Provide the format from " + "each member's own\n" + " build program, or pack the members one at a time." + : std::string{})}; + } + } + + // A package claiming a built-in name is silently unreachable, since the + // parser resolves `tar` and `dir` before consulting the graph at all. + // + // OUTSIDE THE DISPATCH BRANCH ABOVE, because the mistake is in the PACKAGE + // and does not depend on what this invocation asked for. Reported on every + // pack, so the author hears it on the plain `mcpp pack` they are most + // likely to run. + for (auto const& f : ctx.plan.providedPackFormats) + if (mcpp::pack::is_builtin_pack_format(f)) + mcpp::ui::warning(std::format( + "a package in this graph declares `mcpp:pack-format={}`, which " + "is one of the archive shapes `mcpp pack` owns; `--format {}` " + "will always select the built-in and never that package", f, f)); + return std::nullopt; +} + +// Stages one member from the build of its group: the pack plan, the tree, and +// for a built-in format the product. +// +// Called inside the member's view (`with_member`). +std::optional stage_member(PackRun& run, GroupJob& g, MemberJob& m) { + auto& ctx = *g.ctx; + if (auto refused = member_plan(run, g, m)) return refused; + auto& opts = m.opts; + auto& plan = m.plan; + const auto& cfg = *run.cfg; + + // The RESOLVED debug-information decision. On the plan, not in Options: + // Options is the request, this is what it came out as once the manifest + // and the toolchain had their say. Tools come from the build's own + // toolchain so a cross bundle is stripped by the cross tool. // #630 A9: see the field comment on `Plan::extraSharedLegs` and the // parameter comment on `extraLegs` above. A no-op (default-constructed, // empty) for every caller before this item. - plan->extraSharedLegs = std::move(extraLegs); + plan->extraSharedLegs = std::move(run.extraLegs); // #649 E5: the shared libraries this graph built, primary leg. From the // plan's link units, never from a directory listing: a vendor library // deployed beside the program is not one of them and stays as shipped. - for (auto const& u : ctx->plan.linkUnits) - if (u.kind == mcpp::build::LinkUnit::SharedLibrary) - plan->graphSharedLibraries.push_back(ctx->outputDir / u.output); + for (auto const& u : ctx.plan.linkUnits) + if (u.kind == mcpp::build::LinkUnit::SharedLibrary + && (ctx.workspaceMembers.size() < 2 || u.memberOf.empty() || u.memberOf == m.name)) + plan->graphSharedLibraries.push_back(ctx.outputDir / u.output); - // The RESOLVED debug-information decision. On the plan, not in Options: - // Options is the request, this is what it came out as once the manifest - // and the toolchain had their say. Tools come from the build's own - // toolchain so a cross bundle is stripped by the cross tool. - plan->strip = mcpp::pack::resolve_strip(opts, ctx->manifest.packConfig); - plan->debugDir = mcpp::pack::resolve_debug_dir(opts, ctx->manifest.packConfig, - ctx->projectRoot); + plan->strip = mcpp::pack::resolve_strip(opts, ctx.manifest.packConfig); + plan->debugDir = mcpp::pack::resolve_debug_dir(opts, ctx.manifest.packConfig, + ctx.projectRoot); plan->stripTools = mcpp::pack::StripTools{ - .strip = mcpp::toolchain::binutils_tool(ctx->tc, "strip"), - .objcopy = mcpp::toolchain::binutils_tool(ctx->tc, "objcopy"), + .strip = mcpp::toolchain::binutils_tool(ctx.tc, "strip"), + .objcopy = mcpp::toolchain::binutils_tool(ctx.tc, "objcopy"), // The CANONICAL triple, resolved the same way the library packer // resolves it: an empty `targetTriple` means "this host", and asking // the empty string would answer "in-band" for macOS and MSVC alike. .inBandDebugInfo = mcpp::pack::debug_info_is_in_band( - ctx->tc.targetTriple.empty() + ctx.tc.targetTriple.empty() ? mcpp::toolchain::triple::host_triple().str() : [&] { - auto t = mcpp::toolchain::triple::parse(ctx->tc.targetTriple); - return t ? t->str() : ctx->tc.targetTriple; + auto t = mcpp::toolchain::triple::parse(ctx.tc.targetTriple); + return t ? t->str() : ctx.tc.targetTriple; }()), }; @@ -650,38 +859,40 @@ export PackOutcome build_and_pack(Options opts, bool modeFromUser, // question the provider had not been asked. // // So a missing tree is REPORTED AND CARRIED rather than swallowed: the - // reason is printed as a warning, `pack_stage_dir` stays empty, and + // reason is printed as a warning, the stage directory stays empty, and // `${mcpp.stage_dir}` then refuses at expansion naming that reason. A // provider that reads the tree gets a precise diagnostic; one that does not // proceeds. Nothing is silently degraded -- what changes is who decides. - std::string stageFailure; // set only when NO tree exists at all. - mcpp::pack::ClosureStatus closure; - if (auto r = mcpp::pack::run(*plan, *cfg); !r) { - if (opts.format != mcpp::pack::Format::Dispatched) { - mcpp::ui::error(r.error().message); - return PackOutcome{1}; - } - stageFailure = r.error().message; - mcpp::ui::warning(std::format( + if (run.selection && !run.selection->outputDir.empty()) { + std::error_code dirEc; + std::filesystem::create_directories(plan->archivePath.parent_path(), dirEc); + if (opts.format == mcpp::pack::Format::Dir) + std::filesystem::create_directories(plan->stagingRoot.parent_path(), dirEc); + } + if (auto r = mcpp::pack::run(*plan, cfg); !r) { + if (opts.format != mcpp::pack::Format::Dispatched) + return Refusal{1, about(run, m, r.error().message)}; + m.stageFailure = r.error().message; + mcpp::ui::warning(about(run, m, std::format( "no staged tree for --format {}: {}\n" " A format that consumes ${{mcpp.stage_dir}} cannot be produced " "here; one that names a\n" " built file with ${{mcpp.target_file:}} is unaffected.", - opts.formatName, stageFailure)); + opts.formatName, m.stageFailure))); } else if (!r->walked) { // The tree exists; only its dependency closure does not. Distinct // warning text -- "staged" is true here, unlike the branch above. - closure = mcpp::pack::ClosureStatus{false, r->reason, r->needs}; - mcpp::ui::warning(std::format( + m.closure = mcpp::pack::ClosureStatus{false, r->reason, r->needs}; + mcpp::ui::warning(about(run, m, std::format( "staged without its dependency closure: {}\n" " A format that consumes ${{mcpp.stage_dir}} sees the program and its " "declared\n" " runtime files but not its discovered dependencies; one that names a " "built file\n" " with ${{mcpp.target_file:}} is unaffected.", - closure.reason)); + m.closure.reason))); } else { - closure.needs = r->needs; + m.closure.needs = r->needs; } // The staged tree is now on disk and final -- past the closure (walked or @@ -690,102 +901,151 @@ export PackOutcome build_and_pack(Options opts, bool modeFromUser, // when the staged set does, and so a provider can read whether the // closure was walked. Best-effort: see write_stage_manifest. Skipped when // no tree exists, so no manifest describes a tree that is not there. - if (stageFailure.empty()) mcpp::pack::write_stage_manifest(plan->stagingRoot, closure); + if (m.stageFailure.empty()) mcpp::pack::write_stage_manifest(plan->stagingRoot, m.closure); // #649 E9: the outcome a machine reader receives, from values this pass - // has already answered. Both returns below go through it. - auto outcome_with = [&](std::vector artifacts) { - PackOutcome o; - o.artifacts = std::move(artifacts); - o.format = opts.format == mcpp::pack::Format::Tar ? std::string("tar") - : opts.format == mcpp::pack::Format::Dir ? std::string("dir") - : opts.formatName; - const auto canonical = [](std::string const& t) { - if (t.empty()) return mcpp::toolchain::triple::host_triple().str(); - auto parsed = mcpp::toolchain::triple::parse(t); - return parsed ? parsed->str() : t; - }; - o.targets.push_back(canonical(plan->triple)); - for (auto const& leg : plan->extraSharedLegs) o.targets.push_back(leg.triple); - if (stageFailure.empty()) { - o.stageDir = plan->stagingRoot; - o.stageManifest = mcpp::pack::stage_manifest_path(plan->stagingRoot); - o.closure = closure.walked ? "walked" : "not-walked"; - } - std::error_code bec; - o.ranBuildPrograms = std::filesystem::exists(ctx->projectRoot / "build.mcpp", bec) - || !ctx->manifest.buildConfig.ruleModules.empty(); - for (auto const& sp : ctx->sourcePackages) - if (std::filesystem::exists(sp.root / "build.mcpp", bec)) o.ranBuildPrograms = true; - return o; + // has already answered. Every return goes through it. + auto& out = m.result; + out.name = m.name; + out.format = opts.format == mcpp::pack::Format::Tar ? std::string("tar") + : opts.format == mcpp::pack::Format::Dir ? std::string("dir") + : opts.formatName; + const auto canonical = [](std::string const& t) { + if (t.empty()) return mcpp::toolchain::triple::host_triple().str(); + auto parsed = mcpp::toolchain::triple::parse(t); + return parsed ? parsed->str() : t; }; + out.targets.push_back(canonical(plan->triple)); + for (auto const& leg : plan->extraSharedLegs) out.targets.push_back(leg.triple); + if (m.stageFailure.empty()) { + out.stageDir = plan->stagingRoot; + out.stageManifest = mcpp::pack::stage_manifest_path(plan->stagingRoot); + out.closure = m.closure.walked ? "walked" : "not-walked"; + } + std::error_code bec; + m.ranBuildPrograms = std::filesystem::exists(ctx.projectRoot / "build.mcpp", bec) + || !ctx.manifest.buildConfig.ruleModules.empty(); + for (auto const& sp : ctx.sourcePackages) + if (std::filesystem::exists(sp.root / "build.mcpp", bec)) m.ranBuildPrograms = true; + // Paths are shown from the directory the command was typed in: a workspace's + // root when several members are packed, and otherwise the package's own. + m.pathCtx = mcpp::fetcher::make_path_ctx( + &cfg, run.selection ? run.selection->root : ctx.projectRoot); - auto pathCtx = mcpp::fetcher::make_path_ctx(&*cfg, ctx->projectRoot); + if (opts.format != mcpp::pack::Format::Dispatched) { + auto outPath = (opts.format == mcpp::pack::Format::Tar) + ? plan->archivePath : plan->stagingRoot; + mcpp::ui::status("Packed", mcpp::ui::shorten_path(outPath, m.pathCtx)); + out.artifacts.push_back(std::move(outPath)); + } + return std::nullopt; +} - // ─── The dispatch pass ─────────────────────────────────────────── +// ─── The dispatch pass ─────────────────────────────────────────────── +// +// A `role = "artifact"` action is a ninja edge, and the staged tree is +// produced here, in C++, AFTER ninja has finished. So an artifact action +// cannot depend on the staged tree in the pass that built it, and a +// single-pass `--format ` is not expressible. Two passes are, and +// every value this one needs was answered by the first: +// +// `ov` the overrides that produced the build above +// `plan->stagingRoot` from make_plan, which resolved the triple +// `opts.formatName` the request, already checked against the graph +// +// NOTHING IS RE-DERIVED, and that is the whole discipline of this block. +// `stagingRoot` is a function of the package name, the version, the resolved +// triple and the mode; the resolved triple is not known until a prepare has +// run, so computing it a second time before prepare -- from the host triple, +// say -- is the shape where two derivations of one value agree on every +// machine the author has and disagree on one they do not. +// +// One pass serves every member of the group: each member has its own stage, +// the programs act for the member they belong to (`BuildOverrides:: +// pack_stages`), and one ninja drive builds every member's edges. +std::optional dispatch_group(PackRun& run, GroupJob& g, mcpp::build::Backend& be) { + // WHICH ARTIFACT ACTIONS THIS BUILD ALREADY HAD, before a format was + // requested. The dispatch below reports what the REQUEST introduced, + // and this is the other half of that subtraction. + std::set> preexistingArtifacts; + for (auto const& a : g.ctx->plan.actions) + if (a.role == mcpp::manifest::BuildAction::Role::Artifact) + preexistingArtifacts.emplace(a.packageName, a.id); + + // Nothing was staged, so there is nothing to hand a provider. + if (std::ranges::none_of(g.members, [](const MemberJob& m) { return m.result.rc == 0; })) + return std::nullopt; + + g.ov.pack_format = run.opts.formatName; + auto stage_members = [&] { + g.ov.pack_stages.clear(); + for (auto const& m : g.members) { + if (m.result.rc != 0) continue; + mcpp::build::BuildOverrides::PackStage stage; + // Empty when staging was refused, which is what makes + // `${mcpp.stage_dir}` refuse with the reason attached rather than + // expand to a directory that does not exist. + if (m.stageFailure.empty()) stage.dir = m.plan->stagingRoot; + stage.reason = m.stageFailure; + // #649 E5: the RESOLVED strip decision and debug directory, for a + // member that stages libraries of its own and must follow the same + // switch `--no-strip` and `--debug-symbols` set for this tree. + stage.strip = m.plan->strip ? "1" : "0"; + stage.debugSymbolsDir = m.plan->debugDir; + g.ov.pack_stages[m.name] = std::move(stage); + } + }; + stage_members(); + auto distCtx = mcpp::build::prepare_build(false, false, {}, g.ov); + // A MEMBER WITHOUT A TREE FAILS ALONE (P3). Its provider may name only a + // built file and proceed, which is why the member is kept in the pass; when + // the pass is refused instead -- a provider read `${mcpp.stage_dir}` -- the + // members without a tree are failed by name and the pass is prepared again + // for the others. + if (!distCtx && run.several + && std::ranges::any_of(g.members, [](const MemberJob& m) { + return m.result.rc == 0 && !m.stageFailure.empty(); })) { + for (auto& m : g.members) { + if (m.result.rc != 0 || m.stageFailure.empty()) continue; + mcpp::ui::error(about(run, m, std::format( + "no staged tree for --format {}: {}", run.opts.formatName, m.stageFailure))); + m.result.rc = 1; + } + if (std::ranges::none_of(g.members, [](const MemberJob& m) { return m.result.rc == 0; })) + return std::nullopt; + stage_members(); + distCtx = mcpp::build::prepare_build(false, false, {}, g.ov); + } + if (!distCtx) return Refusal{2, distCtx.error()}; + + // WHICH ACTIONS ARE THE DISTRIBUTABLE: the artifact actions the REQUEST + // INTRODUCED. An action present in both passes existed before anyone + // asked for a format -- a codesign stamp, a size budget -- and + // reporting one as the package would be a wrong answer that looks like a + // right one. // - // A `role = "artifact"` action is a ninja edge, and the staged tree is - // produced here, in C++, AFTER ninja has finished. So an artifact action - // cannot depend on the staged tree in the pass that built it, and a - // single-pass `--format ` is not expressible. Two passes are, and - // every value this one needs was answered by the first: + // THE FIRST VERSION ASKED A NARROWER QUESTION AND GOT IT WRONG. It + // collected only actions naming `${mcpp.stage_dir}`, on the assumption + // that a distributable consumes the staged closure. Not every format + // does: an `.msi` built from ONE named program takes + // `${mcpp.target_file:}` and never looks at the tree, which is + // the shape section 6 of the design record recommends -- "name the + // input, do not harvest a directory", after a bind path that resolved + // to nothing produced a valid, empty, 52 KB installer. So the member + // that followed the guidance was the member the check refused, and the + // workaround was to name the placeholder as an unused input purely to + // satisfy it. Presence-in-this-pass is the property actually wanted, + // and it needs nothing of the member. // - // `ov` the overrides that produced the build above - // `plan->stagingRoot` from make_plan, which resolved the triple - // `opts.formatName` the request, already checked against the graph + // Identity is (package, id): an id is unique within the package that + // declared it and nothing more. // - // NOTHING IS RE-DERIVED, and that is the whole discipline of this block. - // `stagingRoot` is a function of the package name, the version, the - // resolved triple and the mode; the resolved triple is not known until a - // prepare has run, so computing it a second time before prepare -- from the - // host triple, say -- is the shape where two derivations of one value agree - // on every machine the author has and disagree on one they do not. - if (opts.format == mcpp::pack::Format::Dispatched) { - // WHICH ARTIFACT ACTIONS THIS BUILD ALREADY HAD, before a format was - // requested. The dispatch below reports what the REQUEST introduced, - // and this is the other half of that subtraction. - std::set> preexistingArtifacts; - for (auto const& a : ctx->plan.actions) - if (a.role == mcpp::manifest::BuildAction::Role::Artifact) - preexistingArtifacts.emplace(a.packageName, a.id); - - ov.pack_format = opts.formatName; - // #649 E5: the RESOLVED strip decision and debug directory, for a - // member that stages libraries of its own and must follow the same - // switch `--no-strip` and `--debug-symbols` set for this tree. - ov.pack_strip = plan->strip ? "1" : "0"; - ov.pack_debug_symbols_dir = plan->debugDir; - // Empty when staging was refused, which is what makes - // `${mcpp.stage_dir}` refuse with the reason attached rather than - // expand to a directory that does not exist. - ov.pack_stage_dir = stageFailure.empty() ? plan->stagingRoot - : std::filesystem::path{}; - ov.pack_stage_reason = stageFailure; - auto distCtx = mcpp::build::prepare_build(false, false, {}, ov); - if (!distCtx) { mcpp::ui::error(distCtx.error()); return PackOutcome{2}; } - - // WHICH ACTIONS ARE THE DISTRIBUTABLE: the artifact actions the REQUEST - // INTRODUCED. An action present in both passes existed before anyone - // asked for a format -- a codesign stamp, a size budget -- and - // reporting one as the package would be a wrong answer that looks like - // a right one. - // - // THE FIRST VERSION ASKED A NARROWER QUESTION AND GOT IT WRONG. It - // collected only actions naming `${mcpp.stage_dir}`, on the assumption - // that a distributable consumes the staged closure. Not every format - // does: an `.msi` built from ONE named program takes - // `${mcpp.target_file:}` and never looks at the tree, which is - // the shape section 6 of the design record recommends -- "name the - // input, do not harvest a directory", after a bind path that resolved - // to nothing produced a valid, empty, 52 KB installer. So the member - // that followed the guidance was the member the check refused, and the - // workaround was to name the placeholder as an unused input purely to - // satisfy it. Presence-in-this-pass is the property actually wanted, - // and it needs nothing of the member. - // - // Identity is (package, id): an id is unique within the package that - // declared it and nothing more. - std::vector distOutputs; + // AN ACTION BELONGS TO THE MEMBER THE PACKAGE THAT SUBMITTED IT ACTS FOR: + // the member whose own build program submitted it, or the one member whose + // closure reaches the package (`pack_owner`). A plan of one member has no + // other to belong to. + struct Submitted { + std::vector outputs; // THE DISTRIBUTABLE IS THE TERMINAL ARTIFACT. A provider may submit a // chain (`dist-apk`: link, add libraries, align, sign); every output // is verified below, but the thing a user installs, and the operand @@ -793,90 +1053,395 @@ export PackOutcome build_and_pack(Options opts, bool modeFromUser, // introduced action consumes. Measured 2026-09-12: with the first // output taken as the operand, `adb-run` received the unsigned // `base.apk` and `adb install` refused it. - std::vector distInputs; - for (auto const& a : distCtx->plan.actions) { - if (a.role != mcpp::manifest::BuildAction::Role::Artifact) continue; - if (preexistingArtifacts.contains({a.packageName, a.id})) continue; - for (auto const& o : a.outputs) distOutputs.push_back(o); - for (auto const& i : a.inputs) distInputs.push_back(i); + std::vector inputs; + }; + std::map submitted; + const bool sharedPlan = distCtx->workspaceMembers.size() > 1; + for (auto const& a : distCtx->plan.actions) { + if (a.role != mcpp::manifest::BuildAction::Role::Artifact) continue; + if (preexistingArtifacts.contains({a.packageName, a.id})) continue; + std::string member = g.members.front().name; + if (sharedPlan) { + auto reach = distCtx->packReach.find(a.packageName); + member = reach == distCtx->packReach.end() ? std::string{} + : mcpp::build::pack_owner(a.packageName, reach->second); } - auto absolute_of = [&](std::string const& p) { - auto q = std::filesystem::path(p).is_absolute() - ? std::filesystem::path(p) : distCtx->plan.outputDir / p; - return q.lexically_normal(); - }; - std::set consumed; - for (auto const& i : distInputs) consumed.insert(absolute_of(i)); - // DECLARED AND THEN SUBMITTED NOTHING. The half of the contract a - // member is most likely to get wrong is the gate, and a member whose - // gate never opens leaves a pass that succeeds and produces no - // package. Refused by name rather than reported as success. - if (distOutputs.empty()) { - mcpp::ui::error(std::format( - "no action claimed --format '{}'.\n" - " A package declared it provides this format, and no build " - "program submitted a new\n" - " `role = \"artifact\"` action when it was asked for.\n" - " The provider must gate on the request and not on anything " - "else:\n" - " mcpp::provides_pack_format(\"{}\"); " - "// always\n" - " if (std::string_view(mcpp::pack_format()) == \"{}\") ..." - " // then submit", - opts.formatName, opts.formatName, opts.formatName)); - return PackOutcome{1}; + if (member.empty()) continue; + auto& s = submitted[member]; + for (auto const& o : a.outputs) s.outputs.push_back(o); + for (auto const& i : a.inputs) s.inputs.push_back(i); + } + auto absolute_of = [&](std::string const& p) { + auto q = std::filesystem::path(p).is_absolute() + ? std::filesystem::path(p) : distCtx->plan.outputDir / p; + return q.lexically_normal(); + }; + // Two members' providers writing one file are refused, naming both: one + // edge declares an output once, and the second declaration would be ninja's + // refusal of the whole group. + { + std::map writer; + for (auto const& m : g.members) { + auto it = submitted.find(m.name); + if (m.result.rc != 0 || it == submitted.end()) continue; + for (auto const& o : it->second.outputs) { + auto [at, fresh] = writer.try_emplace(absolute_of(o), m.name); + if (!fresh && at->second != m.name) + return Refusal{1, std::format( + "members '{}' and '{}' would both write {} for --format '{}'.\n" + " A file has one producer: write each member's output below a " + "directory of its own,\n" + " for example under `mcpp::out_dir()`, which is the member's.", + at->second, m.name, at->first.string(), run.opts.formatName)}; + } } + } - mcpp::ui::info("Distributing", std::format("{} v{} (--format {})", - plan->packageName, plan->packageVersion, opts.formatName)); - - // NO EXPLICIT GOALS. Everything but the dist edges is already up to - // date from the build above, so a full drive costs a graph scan and - // nothing else -- and an explicit goal set is how the 0.0.104 soname - // aliases went missing, because an edge reachable only through - // `default` is skipped under one. - mcpp::build::BuildOptions dbo; - auto dr = be->build(distCtx->plan, dbo); - if (!dr) { - if (!dr.error().diagnosticOutput.empty()) { - std::fputs(dr.error().diagnosticOutput.c_str(), stderr); - if (dr.error().diagnosticOutput.back() != '\n') std::fputs("\n", stderr); - } - mcpp::ui::error(dr.error().message); - return PackOutcome{1}; + // DECLARED AND THEN SUBMITTED NOTHING. The half of the contract a + // member is most likely to get wrong is the gate, and a member whose + // gate never opens leaves a pass that succeeds and produces no + // package. Refused by name rather than reported as success. + // + // A MEMBER THAT GOT NOTHING FAILS ALONE: the members that were claimed + // continue (P3). + bool any = false; + for (auto& m : g.members) { + if (m.result.rc != 0) continue; + auto it = submitted.find(m.name); + if (it != submitted.end() && !it->second.outputs.empty()) { any = true; continue; } + mcpp::ui::error(about(run, m, std::format( + "no action claimed --format '{}'.\n" + " A package declared it provides this format, and no build " + "program submitted a new\n" + " `role = \"artifact\"` action when it was asked for.\n" + " The provider must gate on the request and not on anything " + "else:\n" + " mcpp::provides_pack_format(\"{}\"); " + "// always\n" + " if (std::string_view(mcpp::pack_format()) == \"{}\") ..." + " // then submit", + run.opts.formatName, run.opts.formatName, run.opts.formatName))); + m.result.rc = 1; + } + if (!any) return std::nullopt; + + for (auto const& m : g.members) + if (m.result.rc == 0) + mcpp::ui::info("Distributing", std::format("{} v{} (--format {})", + m.plan->packageName, m.plan->packageVersion, run.opts.formatName)); + + // NO EXPLICIT GOALS. Everything but the dist edges is already up to + // date from the build above, so a full drive costs a graph scan and + // nothing else -- and an explicit goal set is how the 0.0.104 soname + // aliases went missing, because an edge reachable only through + // `default` is skipped under one. + // + // SEVERAL MEMBERS KEEP GOING (P3): one member's distribution step that + // fails must not stop another member's, so the drive continues past a + // failure, and each member is then judged by its own files below. + // + // A drive that keeps going and fails cannot say which member's step + // failed, and a file a previous pack left in place would then read as this + // pack's product. So the members' declared products are removed before the + // drive, and a file present after it is one this drive made. + if (run.several) { + std::error_code rmEc; + for (auto const& [name, s] : submitted) + for (auto const& o : s.outputs) std::filesystem::remove_all(absolute_of(o), rmEc); + } + mcpp::build::BuildOptions dbo; + dbo.keepGoing = run.several; + auto dr = be.build(distCtx->plan, dbo); + const bool driveFailed = !dr.has_value(); + if (!dr) { + if (!dr.error().diagnosticOutput.empty()) { + std::fputs(dr.error().diagnosticOutput.c_str(), stderr); + if (dr.error().diagnosticOutput.back() != '\n') std::fputs("\n", stderr); } + if (!run.several) return Refusal{1, dr.error().message}; + if (!dr.error().reported) mcpp::ui::error(dr.error().message); + } - // THE CRITERION IS THE FILE, NOT THE EXIT CODE. A cached build program - // replaying the first pass's answer, or a tool that writes nothing and - // exits 0, both leave ninja reporting success -- and section 2's - // measured failure was a packaging step that succeeded while carrying - // nothing. + // THE CRITERION IS THE FILE, NOT THE EXIT CODE. A cached build program + // replaying the first pass's answer, or a tool that writes nothing and + // exits 0, both leave ninja reporting success -- and section 2's + // measured failure was a packaging step that succeeded while carrying + // nothing. + for (auto& m : g.members) { + if (m.result.rc != 0) continue; + const auto& s = submitted.at(m.name); + std::set consumed; + for (auto const& i : s.inputs) consumed.insert(absolute_of(i)); std::error_code ec; std::vector reported; std::vector intermediate; - for (auto const& o : distOutputs) { + bool produced = true; + for (auto const& o : s.outputs) { auto abs = absolute_of(o); if (!std::filesystem::is_regular_file(abs, ec) && !std::filesystem::is_directory(abs, ec)) { - mcpp::ui::error(std::format( - "--format {} reported success and produced nothing at {}", - opts.formatName, abs.string())); - return PackOutcome{1}; + // A drive that failed says why above; its missing file is + // the member's failure, not a success that carried nothing. + mcpp::ui::error(about(run, m, driveFailed + ? std::format("--format {} did not produce {}: its step failed", + run.opts.formatName, abs.string()) + : std::format("--format {} reported success and produced nothing at {}", + run.opts.formatName, abs.string()))); + m.result.rc = 1; + produced = false; + break; } if (consumed.contains(abs)) { intermediate.push_back(std::move(abs)); continue; } reported.push_back(std::move(abs)); } - report_packed(reported, pathCtx); + if (!produced) continue; + report_packed(reported, m.pathCtx); // Every output consumed by another: a cycle a provider should not // write, reported as all outputs rather than as nothing. if (reported.empty()) reported = std::move(intermediate); - return outcome_with(std::move(reported)); + m.result.artifacts = std::move(reported); + } + return std::nullopt; +} + +// The refusals that need only the plans of the groups, before anything is +// compiled, for a pack of several members: what each member would write, and +// that no two write one destination. +std::optional check_destinations(PackRun& run, std::vector& groups) { + std::map writer; + for (auto& g : groups) { + if (g.rc != 0) continue; + for (auto& m : g.members) { + std::optional refused; + mcpp::build::with_member(*g.ctx, m.inWorkspace ? m.name : std::string_view{}, [&] { + refused = member_plan(run, g, m); + }); + if (refused) return refused; + // The staging tree and the product: a tree for `--format dir`, an + // archive for `tar`; a dispatched format's products are its + // provider's, and are compared when they are submitted. + std::vector writes{m.plan->stagingRoot}; + if (m.opts.format == mcpp::pack::Format::Tar) writes.push_back(m.plan->archivePath); + for (auto const& w : writes) { + auto [at, fresh] = writer.try_emplace(w.lexically_normal(), m.name); + if (!fresh) + return Refusal{2, std::format( + "members '{}' and '{}' would both write {}.\n" + " Two packages of one name, version and target share an archive name: " + "pack them one at\n" + " a time, or without --output, which writes each below its own " + "`target/dist`.", + at->second, m.name, at->first.string())}; + } + m.plan.reset(); + } + } + return std::nullopt; +} + +// The pack: the plan of each configuration group and what is refused from it, +// then for each group the build, each member's staging and the one dispatch +// pass. `selection` is null for the package of the directory the command runs +// in, which is one member; a single member is the same steps with one member. +PackOutcome run_pack(PackRun run) { + auto be = mcpp::build::make_ninja_backend(); + + // ─── Build first (pack implies a fresh build) ──────────────────── + mcpp::build::BuildOverrides base; + if (run.opts.mode == mcpp::pack::Mode::Static && run.opts.targetTriple.empty()) + base.target_triple = "x86_64-linux-musl"; + else + base.target_triple = run.opts.targetTriple; + // A bundled program leaves this machine: release is the fallback, not dev. + // `[build] default-profile` still decides when the project states one. + base.profile = run.opts.profile; + base.profile_fallback = "release"; + // The same features on this pass and on the dispatched format's second + // pass below, which reuses `ov`. + base.features = run.opts.features; + + std::vector groups; + if (run.selection) { + std::vector request; + for (auto const& members : run.selection->groups) + for (auto const& mp : members) request.push_back(mp); + for (auto const& members : run.selection->groups) { + GroupJob g; + g.paths = members; + g.ov = base; + g.ov.package_filter.clear(); + g.ov.project_root = run.selection->root; + g.ov.workspace_members = members; + g.ov.workspace_request = request; + groups.push_back(std::move(g)); + } + } else { + GroupJob g; + g.ov = base; + groups.push_back(std::move(g)); + } + std::size_t memberCount = 0; + run.quietUntilValidated = + run.opts.format == mcpp::pack::Format::Dispatched && !mcpp::ui::is_quiet(); + + // ─── The plans, and what is refused before anything is compiled ── + for (auto& g : groups) { + if (auto refused = prepare_group(run, g)) { + mcpp::ui::error(refused->message); + if (groups.size() == 1) return PackOutcome{refused->rc}; + // A configuration that cannot be planned fails for its members alone, + // as a member that fails to build does (P3): the members are named + // as the command named them, and the other groups are packed. + g.rc = refused->rc; + g.members.clear(); + for (auto const& mp : g.paths) g.members.push_back(MemberJob{.name = mp, .path = mp}); + } + memberCount += g.members.size(); + } + run.several = memberCount > 1; + for (auto& g : groups) + if (g.rc == 0) + if (auto refused = check_format(run, g)) { + mcpp::ui::error(refused->message); + return PackOutcome{refused->rc}; + } + if (run.several) + if (auto refused = check_destinations(run, groups)) { + mcpp::ui::error(refused->message); + return PackOutcome{refused->rc}; + } + + // ─── One build per group, each member staged, one dispatch ─────── + for (auto& g : groups) { + if (g.rc != 0) continue; + mcpp::build::BuildOptions bo; + auto br = be->build(g.ctx->plan, bo); + if (!br) { + // The compiler's own output, not just "build failed" — same reason as + // in the library pipeline. + if (!br.error().diagnosticOutput.empty()) { + std::fputs(br.error().diagnosticOutput.c_str(), stderr); + if (br.error().diagnosticOutput.back() != '\n') std::fputs("\n", stderr); + } + mcpp::ui::error(br.error().message); + g.rc = 1; + if (!run.several) return PackOutcome{1}; + continue; + } + // Everything below reads the package being packed: in a workspace plan, + // each selected member in turn (workspace design 2026-09-29 §15). + for (auto& m : g.members) { + std::optional refused; + mcpp::build::with_member(*g.ctx, m.inWorkspace ? m.name : std::string_view{}, [&] { + refused = stage_member(run, g, m); + }); + if (!refused) continue; + mcpp::ui::error(refused->message); + m.result.name = m.name; + m.result.rc = refused->rc; + if (!run.several) return PackOutcome{refused->rc}; + } + if (run.opts.format != mcpp::pack::Format::Dispatched) continue; + if (auto refused = dispatch_group(run, g, *be)) { + mcpp::ui::error(refused->message); + for (auto& m : g.members) + if (m.result.rc == 0) m.result.rc = refused->rc; + if (!run.several) return PackOutcome{refused->rc}; + } } - auto outPath = (opts.format == mcpp::pack::Format::Tar) - ? plan->archivePath : plan->stagingRoot; - mcpp::ui::status("Packed", mcpp::ui::shorten_path(outPath, pathCtx)); - return outcome_with({outPath}); + // ─── What was packed ───────────────────────────────────────────── + // + // A failing member is reported and the others continue (P3): the status is + // that of the first member, in member order, that failed. + // + // The members are reported in `[workspace] members` order, which the groups, + // each in that order, keep only within themselves. + std::vector reported; + for (auto& g : groups) + for (auto& m : g.members) { + m.result.name = m.name; + if (g.rc != 0 && m.result.rc == 0) m.result.rc = g.rc; + reported.push_back(&m); + } + if (run.selection && !run.selection->order.empty()) { + auto rank = [&](const MemberJob* m) { + auto it = std::ranges::find(run.selection->order, m->path); + return static_cast(it - run.selection->order.begin()); + }; + std::ranges::stable_sort(reported, {}, rank); + } + PackOutcome out; + for (auto* m : reported) { + if (m->result.rc != 0 && out.rc == 0) out.rc = m->result.rc; + out.ranBuildPrograms = out.ranBuildPrograms || m->ranBuildPrograms; + out.members.push_back(m->result); + } + if (out.rc != 0) return out; + for (auto const& m : out.members) + out.artifacts.insert(out.artifacts.end(), m.artifacts.begin(), m.artifacts.end()); + const auto& first = out.members.front(); + out.format = first.format; + out.targets = first.targets; + out.stageDir = first.stageDir; + out.stageManifest = first.stageManifest; + out.closure = first.closure; + return out; +} + +// Everything after CLI option parsing for `mcpp pack`, for the package of the +// directory the command runs in (a workspace member included). +// +// `wantTarget` is the target NAME the user asked for, empty when they did not. +// It exists because `mcpp pack ` now routes on `[targets.].kind`: +// a name that resolves to a program has to reach the binary selection below, +// or a project with two `bin` targets would accept `mcpp pack app2` and +// silently bundle app1 — the shape where the command succeeds and the answer +// is wrong. +// +// `extraLegs` (#630 A9) is every OTHER triple a several-`--target` app-pack +// request named, already built by `build_extra_android_legs` above. This call +// still does exactly one build — of `opts.targetTriple`, the PRIMARY leg — and +// stages the primary's own artifact and the declared deploy files as it always +// has; `extraLegs`, when non-empty, only changes WHERE the primary's shared +// object lands (`Plan::extraSharedLegs`, read by `run_shared_program`) and adds +// the other legs, each with its own closure, beside it in the same staged tree, +// before the one dispatch pass runs. Empty for a single-`--target` pack. +export PackOutcome build_and_pack(Options opts, bool modeFromUser, + const std::string& wantTarget = {}, + std::vector extraLegs = {}) { + // `--target *-linux-musl` without an explicit `--mode` implies + // `--mode static` — packaging a musl-static ELF as bundle-project + // would feed patchelf a static binary and crash. The docs treat + // this pair as equivalent; surface it in the code path too. + if (!modeFromUser && opts.targetTriple.find("-musl") != std::string::npos) { + opts.mode = mcpp::pack::Mode::Static; + modeFromUser = true; // user-equivalent intent — block manifest override + } + PackRun run; + run.opts = std::move(opts); + run.modeFromUser = modeFromUser; + run.wantTarget = wantTarget; + run.extraLegs = std::move(extraLegs); + return run_pack(std::move(run)); +} + +// `mcpp pack` over several workspace members: the same pipeline, with each +// configuration group planned once and built once (member selection design +// 2026-09-30, K1). Each member is staged from the group's build, read through +// its own view of the plan, and the members of a group are dispatched by one +// second pass. +export PackOutcome build_and_pack_members(Options opts, bool modeFromUser, + const MemberPack& selection) { + if (!modeFromUser && opts.targetTriple.find("-musl") != std::string::npos) { + opts.mode = mcpp::pack::Mode::Static; + modeFromUser = true; + } + PackRun run; + run.opts = std::move(opts); + run.modeFromUser = modeFromUser; + run.selection = &selection; + return run_pack(std::move(run)); } } // namespace mcpp::pack diff --git a/src/pm/commands.cppm b/src/pm/commands.cppm index bd06494f3..5997228a5 100644 --- a/src/pm/commands.cppm +++ b/src/pm/commands.cppm @@ -363,8 +363,8 @@ inline int cmd_add(const mcpplibs::cmdline::ParsedArgs& parsed) { mcpp::ui::status("Adding", std::format( "{} v{} to {}", canonicalSelector, version, table)); - std::println(""); - std::println("Run `mcpp build` to fetch and build with the new dependency."); + mcpp::ui::line(""); + mcpp::ui::line("Run `mcpp build` to fetch and build with the new dependency."); return 0; } @@ -610,8 +610,8 @@ inline int cmd_update(const mcpplibs::cmdline::ParsedArgs& parsed) { std::filesystem::remove(lockPath, ec); mcpp::ui::status("Updating", "all dependencies (mcpp.lock cleared)"); } - std::println(""); - std::println("Run `mcpp build` to re-resolve and rewrite mcpp.lock."); + mcpp::ui::line(""); + mcpp::ui::line("Run `mcpp build` to re-resolve and rewrite mcpp.lock."); return 0; } diff --git a/src/publish/pipeline.cppm b/src/publish/pipeline.cppm index ee177dc30..b4623a52b 100644 --- a/src/publish/pipeline.cppm +++ b/src/publish/pipeline.cppm @@ -86,7 +86,7 @@ export int emit_xpkg_to(std::string version, const std::filesystem::path& output std::ofstream os(output); if (!os) { std::println(stderr, "error: cannot write '{}'", output.string()); return 1; } os << lua; - std::println("Wrote {}", output.string()); + mcpp::ui::line(std::format("Wrote {}", output.string())); } return 0; } @@ -211,30 +211,31 @@ export int publish_package(bool dry_run, bool allow_dirty) { std::println("--- end ---"); } - // 5. Print step-by-step PR instructions. + // 5. Print step-by-step PR instructions: guidance about what to do next, + // which is narration. The document a `--dry-run` prints above is the result. char first = pkg.name.empty() ? '?' : pkg.name[0]; - std::println(""); - std::println("Next steps to publish to mcpp-index:"); - std::println(""); - std::println(" 1. Tag this commit and push:"); - std::println(" git tag -a v{0} -m \"v{0}\"", pkg.version); - std::println(" git push --tags"); - std::println(""); - std::println(" 2. Upload the tarball to your repo's GitHub Release:"); - std::println(" URL: {}/releases/new?tag=v{}", pkg.repo, pkg.version); - std::println(" Attach: {}", tarball.string()); - std::println(""); - std::println(" 3. Open a PR to mcpp-index:"); - std::println(" Fork: https://github.com/mcpplibs/mcpp-index"); - std::println(" Add: pkgs/{}/{}.lua", first, pkg.name); - std::println(" (file content is in {})", xpkgPath.string()); - std::println(""); + mcpp::ui::line(""); + mcpp::ui::line("Next steps to publish to mcpp-index:"); + mcpp::ui::line(""); + mcpp::ui::line(" 1. Tag this commit and push:"); + mcpp::ui::line(std::format(" git tag -a v{0} -m \"v{0}\"", pkg.version)); + mcpp::ui::line(" git push --tags"); + mcpp::ui::line(""); + mcpp::ui::line(" 2. Upload the tarball to your repo's GitHub Release:"); + mcpp::ui::line(std::format(" URL: {}/releases/new?tag=v{}", pkg.repo, pkg.version)); + mcpp::ui::line(std::format(" Attach: {}", tarball.string())); + mcpp::ui::line(""); + mcpp::ui::line(" 3. Open a PR to mcpp-index:"); + mcpp::ui::line(" Fork: https://github.com/mcpplibs/mcpp-index"); + mcpp::ui::line(std::format(" Add: pkgs/{}/{}.lua", first, pkg.name)); + mcpp::ui::line(std::format(" (file content is in {})", xpkgPath.string())); + mcpp::ui::line(""); // TODO(post-v0.0.3): if `gh` CLI is on PATH and authenticated, offer // `mcpp publish --auto` to: // - gh release create v // - fork mcpp-index, add pkg lua, gh pr create // See docs/11-publishing-a-library.md. - std::println("Tip: future versions of mcpp may automate steps 2-3 via the gh CLI."); + mcpp::ui::line("Tip: future versions of mcpp may automate steps 2-3 via the gh CLI."); return 0; } diff --git a/src/scaffold/create.cppm b/src/scaffold/create.cppm index 9d7d4f7d1..c47a76a4f 100644 --- a/src/scaffold/create.cppm +++ b/src/scaffold/create.cppm @@ -285,13 +285,17 @@ export int new_from_package_template( "{} (template {}@{}:{})", project.qualifiedName, pkg->selector, pkg->version, chosen->name)); - std::println("Resolved template package: namespace={} name={} route={} " - "descriptor={} payload={}", - pkg->id.namespace_, pkg->id.shortName, pkg->indexRoute, - pkg->descriptorDigest, - pkg->payloadDigest.empty() ? "unavailable" : pkg->payloadDigest); + // What `new` says about what it made is narration, like the `Created` + // line above it: standard error, so that the project's creation is not + // part of what a pipe receives. + mcpp::ui::line(std::format( + "Resolved template package: namespace={} name={} route={} " + "descriptor={} payload={}", + pkg->id.namespace_, pkg->id.shortName, pkg->indexRoute, + pkg->descriptorDigest, + pkg->payloadDigest.empty() ? "unavailable" : pkg->payloadDigest)); if (!chosen->meta.postMessage.empty()) - std::println("{}", chosen->meta.postMessage); + mcpp::ui::line(chosen->meta.postMessage); return 0; } @@ -428,10 +432,10 @@ int main() { return 1; } - std::println("Created {} package '{}' at {}", gui ? "gui" : "bin", - project.qualifiedName, tx.final_path().string()); - std::println("Next: cd {} && mcpp build && mcpp run (or `mcpp test`)", - project.directoryName); + mcpp::ui::line(std::format("Created {} package '{}' at {}", gui ? "gui" : "bin", + project.qualifiedName, tx.final_path().string())); + mcpp::ui::line(std::format("Next: cd {} && mcpp build && mcpp run (or `mcpp test`)", + project.directoryName)); return 0; } diff --git a/src/toolchain/lifecycle.cppm b/src/toolchain/lifecycle.cppm index dd6091b03..f3d8e58e3 100644 --- a/src/toolchain/lifecycle.cppm +++ b/src/toolchain/lifecycle.cppm @@ -207,9 +207,9 @@ void msvc_print_detected(const mcpp::toolchain::msvc::MsvcInstallation& inst, inst.display_version(), inst.vsProduct.empty() ? "" : std::format(" (VS {})", inst.vsProduct), inst.toolsVersion)); - std::println(" cl: {}", inst.clPath.string()); - std::println(" import std: {}", - inst.hasStdModules ? "available (std.ixx)" : "not available"); + mcpp::ui::line(std::format(" cl: {}", inst.clPath.string())); + mcpp::ui::line(std::format(" import std: {}", + inst.hasStdModules ? "available (std.ixx)" : "not available")); } // The Windows SDK is the OTHER half of a usable MSVC, and a payload that @@ -398,8 +398,8 @@ void msvc_warn_if_sdk_missing(const mcpp::toolchain::msvc::MsvcInstallation& ins // is what the user will believe. auto choice = mcpp::toolchain::msvc::resolve_sdk_for(inst.clPath); if (choice.sdk) { - std::println(" windows sdk: {} ({})", - choice.sdk->version, choice.sdk->root.string()); + mcpp::ui::line(std::format(" windows sdk: {} ({})", + choice.sdk->version, choice.sdk->root.string())); if (!choice.note.empty()) mcpp::ui::info("note", choice.note); return; } @@ -947,11 +947,11 @@ export int toolchain_install(const mcpp::config::GlobalConfig& cfg, if (!mcpp::platform::is_windows) return msvc_wrong_host(); if (auto inst = mcpp::toolchain::msvc::detect_installation()) { msvc_print_detected(*inst); - std::println(""); - std::println("This is the machine's own Visual Studio — mcpp does not manage it."); - std::println("Tip: `mcpp toolchain default msvc` to make it the default,"); - std::println(" or `mcpp toolchain install msvc ` for a pinned one"); - std::println(" that does not depend on what this machine has installed."); + mcpp::ui::line(""); + mcpp::ui::line("This is the machine's own Visual Studio — mcpp does not manage it."); + mcpp::ui::line("Tip: `mcpp toolchain default msvc` to make it the default,"); + mcpp::ui::line(" or `mcpp toolchain install msvc ` for a pinned one"); + mcpp::ui::line(" that does not depend on what this machine has installed."); return 0; } mcpp::ui::error(mcpp::toolchain::msvc::install_guidance()); @@ -1040,9 +1040,10 @@ export int toolchain_install(const mcpp::config::GlobalConfig& cfg, mcpp::ui::status("Installed", std::format("{} → {}", pkg.display_spec(), inst->clPath.string())); if (cfg.defaultToolchain.empty()) { - std::println(""); - std::println("Tip: `mcpp toolchain default {}` to make this the default.", - spec->spec_str()); + mcpp::ui::line(""); + mcpp::ui::line(std::format( + "Tip: `mcpp toolchain default {}` to make this the default.", + spec->spec_str())); } return 0; } @@ -1116,12 +1117,13 @@ export int toolchain_install(const mcpp::config::GlobalConfig& cfg, mcpp::ui::status("Installed", std::format("{} → {}", pkg.display_spec(), bin.string())); if (cfg.defaultToolchain.empty()) { - std::println(""); - std::println("Tip: `mcpp toolchain default {}{}` to make this the default.", - spec->spec_str(), - spec->target.empty() - ? std::string{} - : std::format(" --target {}", spec->target.str())); + mcpp::ui::line(""); + mcpp::ui::line(std::format( + "Tip: `mcpp toolchain default {}{}` to make this the default.", + spec->spec_str(), + spec->target.empty() + ? std::string{} + : std::format(" --target {}", spec->target.str()))); } return 0; } diff --git a/src/ui.cppm b/src/ui.cppm index a933a5148..0feb5c489 100644 --- a/src/ui.cppm +++ b/src/ui.cppm @@ -3,6 +3,15 @@ // All user-visible status lines from CLI / fetcher / build go through // here. TTY auto-detect; MCPP_NO_COLOR / --no-color disables colors. // +// TWO STREAMS, ONE RULE (output streams plan 2026-10-01, §6, R3). A line that +// narrates what the command is doing -- a verb in the status column, a +// progress bar, the live region, the status row -- goes to the NARRATION +// stream, which is standard error; `narration_stream()` is the one place that +// says so. A line that states the command's result -- what a program prints +// under `run`, a document, a listing, a test's verdict -- stays on standard +// output, where a pipe or a redirection receives it and progress does not. +// Every function below names which of the two it writes to. +// // ONE RENDERER, TWO MEDIA (build progress design 2026-09-29, §5, and its // revision 3, 2026-09-30, §9). Every line goes through `emit`. On a terminal // that can move the cursor, the rows that are still changing -- a download @@ -16,7 +25,7 @@ // when the log has been silent for a minute. module; -#include // fileno, stdout +#include // fileno, stdout, stderr export module mcpp.ui; @@ -35,16 +44,28 @@ void disable_color(); // Check if color is enabled. bool is_color_enabled(); -// Verb-style status ("Compiling foo v0.1.0" pattern). +// The stream that narrates: standard error. The live region, the progress +// bars, the status row and every line below that is not a result are written +// to it, and the region's terminal test, colour and size follow it. +mcpp::platform::terminal::Stream narration_stream(); + +// Verb-style status ("Compiling foo v0.1.0" pattern), on the narration stream. // verb verb word, padded right-aligned in 12-char column // message metadata after the verb void status(std::string_view verb, std::string_view message); -// Cyan verb (Updating, Downloading, Cleaned). +// A verb-style line that states the command's RESULT rather than a step of its +// work, in the layout of `status`, on standard output: the `test result` and +// `workspace result` lines of `mcpp test`, which Cargo's libtest also prints +// on standard output. The caller says which of the two a line is; the stream +// is never inferred from the verb's spelling. +void result(std::string_view verb, std::string_view message); + +// Cyan verb (Updating, Downloading, Cleaned), on the narration stream. void info(std::string_view verb, std::string_view message); -// Bold green Finished line, preceded by a blank line when the command wrote a -// line before it (design §4.5). +// Bold green Finished line, on the narration stream, preceded by a blank line +// when the command narrated a line before it (design §4.5). // `descriptor` annotates the profile's actual effect (e.g. "optimized", // "unoptimized + debuginfo"). Empty = print the profile name alone; callers // that never resolved the profile knobs must not invent one. `detail` follows @@ -105,14 +126,17 @@ struct Diagnostic { }; void diagnostic(const Diagnostic& d); -// Plain output (no verb), respecting -q flag. +// Plain output (no verb), respecting -q flag: a line of a command's RESULT, on +// standard output. `mcpp test` prints each test's verdict and the members' +// summary lines through it. void plain(std::string_view message); -// Flush stdout. Every stdout-writing function above already calls this, so -// callers only need it when they wrote to stdout directly (std::println) and -// want that line visible now rather than whenever the libc buffer happens to -// fill. main() also sets stdout line-buffered, which covers the POSIX -// platforms; this is what makes the guarantee hold on Windows too, where +// Flush both standard streams. Every writing function above already flushes +// what it wrote, so callers only need it when they wrote directly +// (std::println) and want that line visible now rather than whenever the libc +// buffer happens to fill, and before a child process that writes to the same +// terminal starts. main() also sets stdout line-buffered, which covers the +// POSIX platforms; this is what makes the guarantee hold on Windows too, where // MSVCRT treats _IOLBF as _IOFBF. Progress-driven output must be visible while // the process is still running: a build that is killed mid-flight is exactly // when its last lines matter most. @@ -212,7 +236,9 @@ public: SuspendRegion& operator=(const SuspendRegion&) = delete; }; -// One final line to stdout, through the region; suppressed by --quiet. +// One final line of narration, through the region; suppressed by --quiet. A +// line of a command's result is `plain`. An empty `text` is the blank line +// that separates a step from what follows it. void line(std::string_view text); // The bytes of one frame: from the first row of a region of `previousRows` @@ -236,10 +262,10 @@ std::vector region_rows(const std::vector& bars, // --- progress bar (single-line, \r-rewritten) --- // // ONE RENDERER, TWO OUTPUT MODES. On a terminal the bar is a line of the -// region, redrawn in place. When stdout is not a terminal (a CI log, a pipe, a -// file) it prints one line when the item finishes, with its duration: no `\r`, -// no erase sequence, no repaint per frame. The mode follows stdout; -// `set_live_progress` overrides it for tests. +// region, redrawn in place. When the narration stream is not a terminal (a CI +// log, a pipe, a file) it prints one line when the item finishes, with its +// duration: no `\r`, no erase sequence, no repaint per frame. The mode follows +// the narration stream; `set_live_progress` overrides it for tests. void set_live_progress(bool live); bool live_progress(); @@ -330,7 +356,7 @@ private: std::unordered_set finished_; }; -// --- quiet flag (suppresses status / info / finished) --- +// --- quiet flag (suppresses status / info / finished / line: the narration) --- void set_quiet(bool q); bool is_quiet(); @@ -363,10 +389,13 @@ namespace { namespace term = mcpp::platform::terminal; -bool g_color = false; +// Colour is decided per stream, by what that stream is: a line for a pipe must +// not carry escape sequences because the OTHER stream is a terminal. +// Indexed by `term::Stream`. +bool g_color[2] = {false, false}; bool g_quiet = false; bool g_inited = false; -// -1: follow stdout; 0 / 1: set by set_live_progress. +// -1: follow the narration stream; 0 / 1: set by set_live_progress. int g_liveOverride = -1; // East Asian ambiguous characters count two columns (terminal::ambiguous_wide). bool g_ambiguousWide = false; @@ -382,16 +411,29 @@ constexpr std::string_view kYellow = "\033[33m"; constexpr std::string_view kRed = "\033[31m"; constexpr std::string_view kBrightRed = "\033[91m"; -bool detect_color() { +// The stream narration is written to (output streams plan 2026-10-01, R3): +// standard error, so that standard output carries a command's result alone. +// This is the one place that says so; changing it moves the region, the bars, +// the status row, the colour decision and the size the region is fitted to. +constexpr term::Stream kNarration = term::Stream::Err; +// The stream a command's result is written to. +constexpr term::Stream kResult = term::Stream::Out; + +bool& color_flag(term::Stream s) { return g_color[s == term::Stream::Out ? 0 : 1]; } +bool colored(term::Stream s) { return color_flag(s); } +// The colour of what the region and the narration lines draw. +bool narration_colored() { return colored(kNarration); } + +bool detect_color(term::Stream s) { if (auto* e = std::getenv("MCPP_NO_COLOR"); e && *e == '1') return false; if (auto* e = std::getenv("NO_COLOR"); e && *e) return false; // On Windows this also turns the console's escape processing on, without // which the colour sequences would be printed as text. - return term::can_move_cursor(term::Stream::Out); + return term::can_move_cursor(s); } std::string with_color(std::string_view code, std::string_view text) { - if (!g_color) return std::string(text); + if (!narration_colored()) return std::string(text); std::string out; out.reserve(code.size() + text.size() + kReset.size()); out.append(code).append(text).append(kReset); @@ -407,12 +449,19 @@ std::string verb_padded(std::string_view verb) { } std::string verb_line(std::string_view colour, std::string_view verb, - std::string_view message) { + std::string_view message, bool color) { auto v = verb_padded(verb); - if (g_color) return std::format("{}{}{}{} {}", kBold, colour, v, kReset, message); + if (color) return std::format("{}{}{}{} {}", kBold, colour, v, kReset, message); return std::format("{} {}", v, message); } +// A verb line for the narration stream, which is where every verb line but +// those `result` writes is written. +std::string verb_line(std::string_view colour, std::string_view verb, + std::string_view message) { + return verb_line(colour, verb, message, narration_colored()); +} + // The configuration groups of one workspace command build on threads // (workspace design 2026-09-29 §6), and each narrates its build: one line is // written as a whole. The region below is guarded by the same lock. @@ -433,7 +482,7 @@ struct Region { std::vector> bars; std::size_t drawnRows = 0; // rows of the region on the screen now std::vector lastRows; // the rows drawn last - bool anythingAbove = false; // this command wrote a line to stdout + bool anythingAbove = false; // this command narrated a line std::chrono::steady_clock::time_point lastDraw{}; std::chrono::steady_clock::time_point lastLine{}; std::chrono::milliseconds heartbeat{60'000}; @@ -462,12 +511,12 @@ std::chrono::steady_clock::time_point& start_point() { } // Whether the bars draw on a terminal: the region's decision while it is -// open, stdout's otherwise. +// open, the narration stream's otherwise. bool bars_live() { auto& r = region(); if (r.open) return r.live; if (g_liveOverride >= 0) return g_liveOverride == 1; - return term::can_move_cursor(term::Stream::Out); + return term::can_move_cursor(kNarration); } std::string erase_bytes(std::size_t rows) { @@ -479,7 +528,7 @@ std::string erase_bytes(std::size_t rows) { } std::size_t max_live_lines() { - const auto rows = term::rows(); + const auto rows = term::rows(kNarration); return std::min(10, rows > 3 ? rows - 3 : 1); } @@ -510,7 +559,7 @@ std::vector current_rows_locked() { std::vector bars; for (auto const& [who, text] : r.bars) bars.push_back(text); auto rows = region_rows(bars, frame, max_live_lines()); - const auto width = term::cols() > 1 ? term::cols() - 1 : 1; + const auto width = term::cols(kNarration) > 1 ? term::cols(kNarration) - 1 : 1; for (auto& row : rows) row = fit(row, width); return rows; } @@ -529,44 +578,51 @@ void redraw_locked() { if (!may_draw_locked()) return; auto rows = current_rows_locked(); if (rows.size() == r.drawnRows && rows == r.lastRows) return; - term::write_frame(term::Stream::Out, frame_bytes(r.drawnRows, {}, rows)); + term::write_frame(kNarration, frame_bytes(r.drawnRows, {}, rows)); drawn_locked(std::move(rows)); } void erase_locked() { auto& r = region(); if (r.drawnRows == 0) return; - term::write_frame(term::Stream::Out, erase_bytes(r.drawnRows)); + term::write_frame(kNarration, erase_bytes(r.drawnRows)); r.drawnRows = 0; r.lastRows.clear(); } // Writes `text` (whole lines) above the region; line_mutex() held. With the // region on a terminal, the lines and the region's new rows leave in one -// write. A line for standard error travels in that write when standard error -// is the same terminal; otherwise it goes to standard error alone, which is -// not the screen the region is on. -void emit_locked(term::Stream s, std::string_view text) { +// write. A line for the other stream travels in that write when both streams +// reach the same terminal; otherwise it goes to its own stream alone, which is +// not the screen the region is on. `narrates` marks a line of the narration +// (as opposed to a warning, an error or a result), which is what `Finished`'s +// blank line answers to. +void emit_locked(term::Stream s, std::string_view text, bool narrates = false) { auto& r = region(); - if (s == term::Stream::Out && !text.empty()) r.anythingAbove = true; + if (narrates && !text.empty()) r.anythingAbove = true; r.lastLine = std::chrono::steady_clock::now(); const bool framed = may_draw_locked() - && (s == term::Stream::Out || term::same_terminal()); + && (s == kNarration || term::same_terminal()); if (!framed) { + // What was written to standard output before, by a caller that did not + // come through here, arrives before this line whichever stream it is + // for: on a terminal that both streams reach, the two are one column of + // text. Standard error is unbuffered. + std::fflush(stdout); term::write(s, text); std::fflush(s == term::Stream::Out ? stdout : stderr); return; } auto rows = current_rows_locked(); - term::write_frame(term::Stream::Out, frame_bytes(r.drawnRows, text, rows)); + term::write_frame(kNarration, frame_bytes(r.drawnRows, text, rows)); drawn_locked(std::move(rows)); } -void emit(term::Stream s, std::string_view text); +void emit(term::Stream s, std::string_view text, bool narrates = false); // The terminal side of mcpp.log's verbose records: through the one writer. void verbose_record(const mcpp::log::Record& record) { - emit(term::Stream::Err, mcpp::log::verbose_line(record, g_color)); + emit(term::Stream::Err, mcpp::log::verbose_line(record, colored(term::Stream::Err))); } std::atomic g_frameIntervalMs{100}; @@ -607,31 +663,41 @@ void tick(std::stop_token stop) { if (g_quiet || !r.source || now - r.lastLine < r.heartbeat) continue; auto frame = r.source(); if (frame.status.empty()) { r.lastLine = now; continue; } - emit_locked(term::Stream::Out, frame.status + "\n"); + emit_locked(kNarration, frame.status + "\n", /*narrates=*/true); } } } // namespace namespace { -void emit(term::Stream s, std::string_view text) { +void emit(term::Stream s, std::string_view text, bool narrates) { std::lock_guard line(line_mutex()); - emit_locked(s, text); + emit_locked(s, text, narrates); +} +// A line of the narration: the stream is the narration stream's. +void narrate(std::string_view text) { + emit(kNarration, text, /*narrates=*/true); } } // namespace +term::Stream narration_stream() { return kNarration; } + void init() { if (g_inited) return; - g_color = detect_color(); - g_ambiguousWide = term::ambiguous_wide(); + color_flag(term::Stream::Out) = detect_color(term::Stream::Out); + color_flag(term::Stream::Err) = detect_color(term::Stream::Err); + g_ambiguousWide = term::ambiguous_wide(kNarration); g_inited = true; mcpp::log::set_terminal_sink(&verbose_record); } void set_ambiguous_wide(bool wide) { g_ambiguousWide = wide; } -void disable_color() { g_color = false; } -bool is_color_enabled() { return g_color; } +void disable_color() { + color_flag(term::Stream::Out) = false; + color_flag(term::Stream::Err) = false; +} +bool is_color_enabled() { return narration_colored(); } void set_quiet(bool q) { g_quiet = q; } @@ -639,11 +705,14 @@ void set_live_progress(bool live) { g_liveOverride = live ? 1 : 0; } bool live_progress() { if (g_liveOverride >= 0) return g_liveOverride == 1; - return term::can_move_cursor(term::Stream::Out); + return term::can_move_cursor(kNarration); } bool is_quiet() { return g_quiet; } -void flush() { std::fflush(stdout); } +void flush() { + std::fflush(stdout); + std::fflush(stderr); +} void set_line_buffered() { #if defined(_WIN32) @@ -667,13 +736,19 @@ void set_line_buffered() { void status(std::string_view verb, std::string_view message) { if (g_quiet) return; init(); - emit(term::Stream::Out, verb_line(kBrightGreen, verb, message) + "\n"); + narrate(verb_line(kBrightGreen, verb, message) + "\n"); +} + +void result(std::string_view verb, std::string_view message) { + if (g_quiet) return; + init(); + emit(kResult, verb_line(kBrightGreen, verb, message, colored(kResult)) + "\n"); } void info(std::string_view verb, std::string_view message) { if (g_quiet) return; init(); - emit(term::Stream::Out, verb_line(kBrightCyan, verb, message) + "\n"); + narrate(verb_line(kBrightCyan, verb, message) + "\n"); } void finished(std::string_view profile, std::chrono::milliseconds elapsed, @@ -691,28 +766,29 @@ void finished(std::string_view profile, std::chrono::milliseconds elapsed, if (!detail.empty()) msg += std::format(" · {}", detail); std::lock_guard line(line_mutex()); // The summary is separated from the steps above it by one blank line - // (design §4.5); a command that wrote nothing before it writes none. + // (design §4.5); a command that narrated nothing before it writes none. const std::string head = region().anythingAbove ? "\n" : ""; - emit_locked(term::Stream::Out, head + verb_line(kBrightGreen, "Finished", msg) + "\n"); + emit_locked(kNarration, head + verb_line(kBrightGreen, "Finished", msg) + "\n", + /*narrates=*/true); } void warning(std::string_view message) { init(); - emit(term::Stream::Err, g_color + emit(term::Stream::Err, colored(term::Stream::Err) ? std::format("{}{}warning:{} {}\n", kBold, kYellow, kReset, message) : std::format("warning: {}\n", message)); } void error(std::string_view message) { init(); - emit(term::Stream::Err, g_color + emit(term::Stream::Err, colored(term::Stream::Err) ? std::format("{}{}error:{} {}\n", kBold, kBrightRed, kReset, message) : std::format("error: {}\n", message)); } void note(std::string_view message) { init(); - emit(term::Stream::Err, g_color + emit(term::Stream::Err, colored(term::Stream::Err) ? std::format("{}{}note:{} {}\n", kBold, kCyan, kReset, message) : std::format("note: {}\n", message)); } @@ -747,7 +823,7 @@ void print_closing_notices() { if (g_quiet) return; init(); for (auto const& n : notices) { - emit(term::Stream::Err, g_color + emit(term::Stream::Err, colored(term::Stream::Err) ? std::format("{}{}tip:{} {}\n", kBold, kCyan, kReset, n.message) : std::format("tip: {}\n", n.message)); } @@ -755,23 +831,24 @@ void print_closing_notices() { void plain(std::string_view message) { if (g_quiet) return; - emit(term::Stream::Out, std::string(message) + "\n"); + emit(kResult, std::string(message) + "\n"); } void line(std::string_view text) { if (g_quiet) return; - emit(term::Stream::Out, std::string(text) + "\n"); + narrate(std::string(text) + "\n"); } void diagnostic(const Diagnostic& d) { init(); + const bool color = colored(term::Stream::Err); auto bold_red = [&](std::string_view s) { - return g_color ? std::format("{}{}{}{}", kBold, kBrightRed, s, kReset) - : std::string(s); + return color ? std::format("{}{}{}{}", kBold, kBrightRed, s, kReset) + : std::string(s); }; auto blue = [&](std::string_view s) { - return g_color ? std::format("{}{}{}{}", kBold, kBrightCyan, s, kReset) - : std::string(s); + return color ? std::format("{}{}{}{}", kBold, kBrightCyan, s, kReset) + : std::string(s); }; std::string out; std::string head = "error"; @@ -940,7 +1017,7 @@ std::string step_line(std::string_view verb, std::string_view subject, : tone == Tone::Muted ? kDim : tone == Tone::Bad ? kRed : std::string_view{}; - if (g_color && !colour.empty()) s += std::format("{}{}{}", colour, state, kReset); + if (narration_colored() && !colour.empty()) s += std::format("{}{}{}", colour, state, kReset); else s += state; return s; } @@ -948,15 +1025,15 @@ std::string step_line(std::string_view verb, std::string_view subject, std::string status_line(std::string_view phase, std::string_view rest) { init(); const auto verb = std::format("{:>12}", phase); - std::string s = g_color ? std::format("{}{}{}{}", kBold, kBrightCyan, verb, kReset) - : verb; + std::string s = narration_colored() + ? std::format("{}{}{}{}", kBold, kBrightCyan, verb, kReset) : verb; if (!rest.empty()) s += std::format(" {}", rest); return s; } std::string hue(std::string_view text, Hue h) { init(); - if (!g_color || h == Hue::Plain || text.empty()) return std::string(text); + if (!narration_colored() || h == Hue::Plain || text.empty()) return std::string(text); std::string_view code = h == Hue::Cyan ? "\033[36m" : h == Hue::Magenta ? "\033[95m" : h == Hue::Blue ? "\033[94m" @@ -1068,6 +1145,7 @@ void set_heartbeat(std::chrono::milliseconds interval) { SuspendRegion::SuspendRegion() { std::fflush(stdout); + std::fflush(stderr); std::lock_guard line(line_mutex()); erase_locked(); ++region().suspended; @@ -1106,16 +1184,16 @@ std::string fmt_bytes(std::size_t b) { return std::format("{:.2f} GB", static_cast(b) / (1024.0*1024.0*1024.0)); } -// Best-effort terminal width. Tries TIOCGWINSZ first; on failure (e.g., -// stdout is a pipe) honours $COLUMNS so users can clamp the width -// manually for testing or when running under CI loggers that don't -// propagate winsize. Falls back to 80 cols. +// Best-effort width of the terminal narration is drawn on. Tries TIOCGWINSZ +// first; on failure (e.g., standard error is a pipe) honours $COLUMNS so users +// can clamp the width manually for testing or when running under CI loggers +// that don't propagate winsize. Falls back to 80 cols. // // 80 is the right safe default for a "fixed-shape" status line — we'd // rather collapse the bar than wrap into a second row that `\r\033[2K` // can't clean up later. std::size_t terminal_cols() { - return term::cols(); + return term::cols(kNarration); } // Truncate a "visible" string (no ANSI codes inside) to `max` chars, replacing diff --git a/src/xlings/xlings.cppm b/src/xlings/xlings.cppm index a88cd7df7..dd78df163 100644 --- a/src/xlings/xlings.cppm +++ b/src/xlings/xlings.cppm @@ -622,7 +622,7 @@ bool is_official_package_index_fresh(const Env& env, std::string_view packageName, std::int64_t ttlSeconds); -// Run `xlings update` to refresh all index repos. Streams output to stdout. +// Run `xlings update` to refresh all index repos. Streams output to stderr. // Returns the xlings exit code. int update_index(const Env& env, bool quiet = false); @@ -676,13 +676,14 @@ namespace mcpp::xlings { namespace { -// Right-pad a verb to 12 columns for bootstrap status lines. +// Right-pad a verb to 12 columns for bootstrap status lines, on standard +// error: the stream mcpp.ui narrates on. void print_status(std::string_view verb, std::string_view msg) { constexpr std::size_t W = 12; if (verb.size() >= W) { - std::println("{} {}", verb, msg); + std::println(stderr, "{} {}", verb, msg); } else { - std::println("{}{} {}", std::string(W - verb.size(), ' '), verb, msg); + std::println(stderr, "{}{} {}", std::string(W - verb.size(), ' '), verb, msg); } } @@ -1856,8 +1857,9 @@ int install_direct(const Env& env, std::string_view target, bool quiet) { bool timedOut = false; // The streaming runner seals stdin itself. Lines are passed through as the // inherited terminal showed them before, unless the caller asked for quiet. + // They are xlings' own progress text, narration: standard error. int rc = mcpp::platform::process::run_streaming_bounded(cmd, - [quiet](std::string_view line) { if (!quiet) std::println("{}", line); }, + [quiet](std::string_view line) { if (!quiet) std::println(stderr, "{}", line); }, std::chrono::duration_cast(kDirectInstallTimeout), std::chrono::milliseconds{0}, &timedOut); if (timedOut) diff --git a/tests/e2e/122_run_member.sh b/tests/e2e/122_run_member.sh index 1e1e93a62..5c8b7d85a 100755 --- a/tests/e2e/122_run_member.sh +++ b/tests/e2e/122_run_member.sh @@ -47,9 +47,11 @@ EOF # `-p memberB` must run memberB's binary — assert its DISTINCT output, not # memberA's (and not "no binary target found"). +# Standard output is the program's alone: the status lines are on standard +# error (output streams plan 2026-10-01, R3), so the capture is exact. OUT=$("$MCPP" run -p memberB 2>run_b.log) || { cat run_b.log; echo "FAIL: run -p memberB failed"; exit 1; } echo "$OUT" -[[ "$OUT" == *"hello from memberB"* ]] || { +[[ "$OUT" == "hello from memberB" ]] || { echo "FAIL: expected memberB's output, got: $OUT" exit 1 } @@ -61,7 +63,7 @@ echo "$OUT" # `-p memberA` runs the other one. OUT=$("$MCPP" run -p memberA 2>run_a.log) || { cat run_a.log; echo "FAIL: run -p memberA failed"; exit 1; } echo "$OUT" -[[ "$OUT" == *"hello from memberA"* ]] || { +[[ "$OUT" == "hello from memberA" ]] || { echo "FAIL: expected memberA's output, got: $OUT" exit 1 } diff --git a/tests/e2e/180_msvc_build_mcpp.sh b/tests/e2e/180_msvc_build_mcpp.sh index 946a23707..c63fab6f8 100755 --- a/tests/e2e/180_msvc_build_mcpp.sh +++ b/tests/e2e/180_msvc_build_mcpp.sh @@ -132,8 +132,11 @@ run_out=$("$MCPP" run 2>&1) || { echo "FAIL: run (modules): $run_out"; exit 1; } [[ "$run_out" == *"msvc-modules-ok"* ]] \ || { echo "FAIL: run output (modules): $run_out"; exit 1; } -# The .ifc really came from the msvc module pipeline, not a silent fallback. -find target/.build-mcpp -name "*.ifc" | grep -q . \ +# The .ifc really came from the msvc module pipeline, not a silent fallback. The +# bundled module is kept by key, in the global cache or the workspace's own store +# (#748), so both are searched. +CACHE_ROOT="$("$MCPP" cache dir | head -1)" +find "$CACHE_ROOT/pkg/_engine" target/.build-mcpp -name "*.ifc" 2>/dev/null | grep -q . \ || { echo "FAIL: no .ifc produced for the bundled mcpp module"; exit 1; } echo "PASS: MSVC build.mcpp — include path, link-lib translation, named modules" diff --git a/tests/e2e/309_host_module_identity.sh b/tests/e2e/309_host_module_identity.sh index 6640d1595..e52a1d7e7 100755 --- a/tests/e2e/309_host_module_identity.sh +++ b/tests/e2e/309_host_module_identity.sh @@ -92,13 +92,17 @@ out="$("$MCPP" run 2>&1 | grep '^DIVERGENT=' | tail -1)" # rule the object was `protobufgen.o`; under this one it is named for what the # source declares. A file listing is platform-independent evidence about which # name the engine used, where the successful import is not. -BM=target/.build-mcpp -ls "$BM"/acme.rules.protobuf.* >/dev/null 2>&1 || { +# +# A host module is kept as an entry of the workspace's store, keyed by what it +# was compiled from (#748), and the entry names its files as before: the BMI +# under `bmi/` and the object under `obj/`. +BM=target/.build-mcpp/host-modules +[ -n "$(find "$BM" -name 'acme.rules.protobuf.*' 2>/dev/null)" ] || { echo "FAIL: the host module was not named for its DECLARED module name" - ls -la "$BM" 2>/dev/null + find "$BM" 2>/dev/null exit 1; } -if ls "$BM"/protobufgen.* >/dev/null 2>&1; then - ls -la "$BM" +if [ -n "$(find "$BM" -name 'protobufgen.*' 2>/dev/null)" ]; then + find "$BM" echo "FAIL: the host module is still named for the PACKAGE name" exit 1 fi diff --git a/tests/e2e/312_build_rules_example.sh b/tests/e2e/312_build_rules_example.sh index 8b9e4975e..f89470772 100755 --- a/tests/e2e/312_build_rules_example.sh +++ b/tests/e2e/312_build_rules_example.sh @@ -36,7 +36,9 @@ out="$("$MCPP" run 2>&1 | tail -1)" # The rules are build-time only. Asserted on the ARTIFACT: a rule's objects have # no business in the consumer's binary, and `Compiling rules-embed (path)` in # the log is the host-module compile, not evidence either way. -mapfile -t objs < <(find target -path '*obj*' -name '*.o' -printf '%f\n' | sort) +# The store a host module is kept in (`.build-mcpp/host-modules`, #748) has an +# `obj/` of its own and is not the consumer's: it is left out. +mapfile -t objs < <(find target -path '*obj*' -not -path '*/.build-mcpp/*' -name '*.o' -printf '%f\n' | sort) for o in "${objs[@]}"; do case "$o" in main.o|embed_greeting.o) ;; diff --git a/tests/e2e/852_a_repeated_dash_p_selects_every_member_it_names.sh b/tests/e2e/852_a_repeated_dash_p_selects_every_member_it_names.sh new file mode 100755 index 000000000..9474cea7e --- /dev/null +++ b/tests/e2e/852_a_repeated_dash_p_selects_every_member_it_names.sh @@ -0,0 +1,79 @@ +#!/usr/bin/env bash +# requires: unix-shell +# 852 -- a repeated `-p` selects every member it names, as a set, and a `-p` +# that names no member is refused before anything is planned. +# +# `-p` was declared with `takes_value()` and without `multiple()`, so a +# repeated `-p` kept the last value and dropped the others without a word +# (mcpp#750): `mcpp build -p a -p b` built `b` alone and exited 0. The +# selection is now one function for every command (member selection design +# 2026-09-30, S1 and S2); the members it returns are a set in `[workspace] +# members` order. +# +# Criteria: +# A. `mcpp build -p a -p b` in a workspace of `a`, `b` and `c` builds `a` and +# `b`. `c`'s object directory stays absent. +# B. `mcpp build -p a -p b` followed by `mcpp build -p b -p a` adds no +# compile edge to `.ninja_log`: the order `-p` was written in is not part +# of the selection, and a member named twice is one member. +# C. `mcpp build -p nosuch` exits non-zero before planning. Its message names +# `nosuch` and lists the members; a `-p` that is valid beside it does not +# rescue the command. +set -e + +TMP=$(mktemp -d) +trap 'rm -rf "$TMP"' EXIT +fail() { echo "FAIL: $1"; shift; for f in "$@"; do echo "--- $f ---"; cat "$f" 2>/dev/null; done; exit 1; } +cd "$TMP" + +cat > mcpp.toml <<'EOF' +[workspace] +members = ["a", "b", "c"] +EOF +for m in a b c; do + mkdir -p $m/src + printf '[package]\nname = "%s"\nversion = "0.1.0"\n\n[targets.%s]\nkind = "lib"\n' $m $m > $m/mcpp.toml + printf 'export module sel852_%s;\nexport int %s_value() { return 1; }\n' $m $m > $m/src/$m.cppm +done + +# The build directory of a command: the one directory under target/ holding a +# build.ninja. +build_dir() { find target -name build.ninja -exec dirname {} \; | head -1; } +# How many times ninja built the object of member $2 in build directory $1. +object_edges() { awk -F'\t' -v m="$2" '$4 ~ ("(^|/)obj/" m "/src/" m "\\.m\\.o$")' "$1/.ninja_log" | wc -l | tr -d ' '; } + +# ── C ────────────────────────────────────────────────────────────────────── +# First, so that nothing has been planned, or built, that could be mistaken for +# what the refusal left behind. +rc=0 +"$MCPP" build -p a -p nosuch > c.log 2>&1 || rc=$? +[ "$rc" -ne 0 ] || fail "C: a -p that names no member was accepted" c.log +grep -q "nosuch" c.log || fail "C: the refusal does not name the member it could not find" c.log +for m in a b c; do + grep -qE "'$m'" c.log || fail "C: the refusal does not list member $m" c.log +done +! grep -q "Resolving toolchain" c.log || fail "C: the command planned before it refused" c.log +[ ! -d target ] || fail "C: the refused command left a build directory" c.log +echo "ok: C, an unknown -p is refused by name, with the members listed, before planning" + +# ── A ────────────────────────────────────────────────────────────────────── +"$MCPP" build -p a -p b > a.log 2>&1 || fail "A: -p a -p b did not build" a.log +dir=$(build_dir) +[ -n "$dir" ] || fail "A: no build directory" a.log +[ "$(object_edges "$dir" a)" = 1 ] || fail "A: a was not built" "$dir/.ninja_log" +[ "$(object_edges "$dir" b)" = 1 ] || fail "A: b was not built (the repeated -p kept one value)" "$dir/.ninja_log" +[ ! -e "$dir/obj/c" ] || fail "A: c was built although no -p named it" a.log +grep -q "Workspace building 2 members: a, b" a.log || fail "A: the plan does not state the two members" a.log +echo "ok: A, -p a -p b builds a and b, and c is untouched" + +# ── B ────────────────────────────────────────────────────────────────────── +"$MCPP" build -p b -p a > b.log 2>&1 || fail "B: -p b -p a did not build" b.log +[ "$(object_edges "$dir" a)" = 1 ] && [ "$(object_edges "$dir" b)" = 1 ] \ + || fail "B: the reverse order compiled an edge again" "$dir/.ninja_log" +"$MCPP" build -p a -p ./b -p b -p a > b2.log 2>&1 || fail "B: a member named twice did not build" b2.log +[ "$(object_edges "$dir" a)" = 1 ] && [ "$(object_edges "$dir" b)" = 1 ] \ + || fail "B: naming a member twice compiled an edge again" "$dir/.ninja_log" +[ "$(find target -name build.ninja | wc -l | tr -d ' ')" = 1 ] || fail "B: the order made a second build directory" b.log +echo "ok: B, the order of -p, and a member named twice, do not change the selection" + +echo "PASS: 852_a_repeated_dash_p_selects_every_member_it_names" diff --git a/tests/e2e/853_run_keeps_one_member_and_exclude_removes_members.sh b/tests/e2e/853_run_keeps_one_member_and_exclude_removes_members.sh new file mode 100755 index 000000000..8afa2d24f --- /dev/null +++ b/tests/e2e/853_run_keeps_one_member_and_exclude_removes_members.sh @@ -0,0 +1,109 @@ +#!/usr/bin/env bash +# requires: unix-shell +# 853 -- `mcpp run` keeps one member, and `--exclude` removes members from a +# whole-workspace selection. +# +# `-p` is repeatable on `build`, `test` and `mcpp emit build-database`, and +# one selection function serves all of them (member selection design +# 2026-09-30, S1 to S3). `run` executes one program, so a second `-p` there is +# refused, naming both, and never read as "the last one". `--exclude ` +# removes members from the forms that select every member: `--workspace`, or a +# virtual root without `-p`. +# +# Criteria: +# D. `mcpp run -p a -p b` is refused, naming both, before anything is +# planned; `mcpp run -p a` runs `a`. +# E. `mcpp build --workspace --exclude c` builds `a` and `b` and leaves `c` +# untouched; so does `--exclude c` alone at a virtual root. `--exclude +# nosuch`, `-p a --exclude b`, and an `--exclude` that leaves no member +# are refused, each before planning; so is `--exclude` where no +# whole-workspace form applies, and `--workspace` together with `-p`. +# F. `mcpp test --workspace --exclude c` tests `a` and `b`, and `mcpp emit +# build-database` describes the members a selection names and no others. +set -e + +TMP=$(mktemp -d) +trap 'rm -rf "$TMP"' EXIT +fail() { echo "FAIL: $1"; shift; for f in "$@"; do echo "--- $f ---"; cat "$f" 2>/dev/null; done; exit 1; } +cd "$TMP" + +cat > mcpp.toml <<'EOF' +[workspace] +members = ["a", "b", "c"] +EOF +for m in a b c; do + mkdir -p $m/src $m/tests + printf '[package]\nname = "%s"\nversion = "0.1.0"\n\n[targets.%s]\nkind = "bin"\nmain = "src/main.cpp"\n' $m $m > $m/mcpp.toml + printf '#include \nint main() { std::puts("hello from %s"); return 0; }\n' $m > $m/src/main.cpp + printf 'int main() { return 0; }\n' > $m/tests/smoke.cpp +done + +build_dir() { find target -name build.ninja -exec dirname {} \; | head -1; } +refused() { # refused