2026.10.1.3: a source for every tool a build uses, declared, programmable and observable (#755) - #758
Merged
Merged
Conversation
…d observable
A build uses a toolchain, the payloads its plugins declare, and the tools those
plugins run. Each of them now has one source that a project can state, a build
program can decide, and anyone can read back; a project that writes none of the
new keys builds exactly as before, with the same output.
- `[xlings.overrides]`, `MCPP_XLINGS_OVERRIDE_<NS>_<NAME>` and config.toml state
where a declared payload comes from. An overridden payload is not provisioned
and does not reach the offline gate; `xpkg_program` and `xpkg_source` answer
the program it named and `override`. A stated version is checked against every
requirement the graph made, and a dependency that writes the table is refused.
- `provision = "on-request"` installs a payload when a build program asks for it
with `xpkg_request`: one batch per invocation, only the programs that asked run
again, and planning records MCPP_BUILD_DATABASE_PAYLOAD_DEFERRED instead.
- `[toolchain] <key> = { path = ... }` and `MCPP_TOOLCHAIN=path:<dir>` name a
toolchain this machine already has; mcpp probes it, drives it with its own link
model, writes nothing into the tree, keys the fingerprint and the fast paths on
its programs' content, and records it in mcpp.lock as local. `bootstrap` names
the toolchain that builds build programs, and `{ configure = "build.mcpp" }`
hands the build toolchain to the root program's toolchain phase.
- A build reports a source that is not the ecosystem's on its own line, sums them
on the `Finished` line, and writes the record to resolution.json; `mcpp why
sources|tool|payload` and the `mcpp.why.sources` kind read it, and
`--managed-only` refuses a build that uses any.
- Protocol 15: xpkg_source, xpkg_program, xpkg_request, xpkg_pending, phase,
decision, toolchain.
Specifications and both documentation trees are updated: SPEC-004 §4.7,
SPEC-006 §2.2.1 and §3.3, SPEC-007 R6.2/R6.5/R6.6/R9.9 and the protocol table,
docs/09, 20, 23, 30, 31, 32, 50 and their 简体中文 mirrors.
Closes #755
Test plan
- `mcpp test`: 144 passed, including the new tests/unit/test_sources.cpp (15).
- New e2e 873 (overrides), 874 (on-request), 875 (a toolchain by path),
876 (the toolchain phase), 877 (`mcpp why` and its machine output), each with
MCPP_NO_AUTO_INSTALL=1 as the criterion and a control that must refuse.
- 62 existing e2e cases selected by keyword pass; 219 fails identically on the
released 2026.10.1.2 on this machine, and 658 needs an attached Android device.
- mcpp-plugins 0.19.0 against this engine: 11 consumer fixtures and 27
plugin-logic cases pass; tests/cmake-consumer builds under
MCPP_NO_AUTO_INSTALL=1 when its build program names its own cmake.
…gn record carries front matter `std::to_string` over a filesystem clock's rep and over `uintmax_t` is ambiguous on libc++, so the three sites that write or compare the stamp of a toolchain named by path format the two values through `std::format` with an explicit type. The design record gains the `subject`/`status` front matter every record dated 2026-09-08 or later carries.
…air specialization Exporting `std::vector<std::pair<std::string, std::filesystem::path>>` from mcpp.toolchain.model made clang 20.1.7 on Windows crash while generating code for `mcpp::pack::interface_set_digest`, a function of another module that instantiates the same specialization and sorts by a pointer to its `first`. The report named a file this branch never touched, which is how this hazard always reads; `modules/manifest/src/types.cppm` records a GCC 16 case of the same shape. The field is now a `ToolOverride` of two named members, and every reader uses the same structured binding it used before.
7 tasks done
…member A `std::ranges` projection spelled as a pointer-to-member into a type the module imports makes clang 20.1.7 crash in code generation, and the report names an unrelated function: first `mcpp::pack::interface_set_digest`, then `mcpp::doctor::why_report`, both on windows-2022. Every lookup this branch added over the decision record now passes a predicate, which reads the same and compiles on every host.
…p.lock is not claimed to hold it The chapter, its mirror, SPEC-006 and the release notes said a build records such a toolchain in mcpp.lock as `local`. It does not: the lock holds the result of dependency resolution, a toolchain is not a resolved dependency, and nothing in this branch writes one there. What is true is that the driver and each stated tool enter the fingerprint and are recorded beside the build, so the fast paths decline once one of them changed, and a machine without the tree is refused where the declaration is read. The design record states the correction.
The third place an override may be stated was parsed inside `load_or_init`, which bootstraps a home, so no unit test could reach it. `parse_payload_overrides` is now a pure function of the parsed document, and three cases cover the two shapes it accepts, the key it refuses, and a config with no such table.
… when it outgrows the channel
A build program that imports many host modules carries one
`-fmodule-file=<name>=<path>` per module, with absolute paths, and on Windows
`capture_exec` reaches a shell that tolerates 8191 bytes. mcpp-plugins'
all-rules-compile fixture imports fifteen and crossed that line the moment the
collection gained one more module, reporting only
The command line is too long.
build.mcpp failed to compile (exit 1)
which names neither the length nor the cause -- the family mcpp.build.cmdlimits
exists to make legible. The command now goes through `@file` when it is over
the budget that module states, which every driver mcpp supports reads, and the
file stays beside the program for a failed compile to show. Its quoting is
`response_file_body`, exported and covered by a unit case, because the command
it serves cannot be run on a host whose limit it does not cross.
`command -v true` prints `true`, not a path: a shell answers with what it would run, and for a builtin that is the word. which() then found no file and reported the name missing, so a bare-name payload override of such a name was refused with `'true' is not found on PATH` on a machine carrying /usr/bin/true -- an answer that sends the reader to the wrong place. Found while verifying the host class of mcpp#755. PATH is now walked for a bare word the shell returned, and two cases cover it: the builtin name resolves to its program, and a name no host has still answers nothing.
A response file is not one format. clang and GCC tokenize it the GNU way, where
a backslash escapes the next character, so the Windows paths written plainly came
back with their separators eaten:
clang++: error: no such file or directory:
'D:amcpp-pluginsmcpp-pluginstestsall-rules-compiletarget.build-mcpp...'
Each argument is therefore wrapped in single quotes for those drivers, inside
which nothing is special, and an embedded single quote is closed, escaped and
reopened; cl and clang-cl keep Windows quoting, where a backslash is literal.
One case per grammar.
…ace manifest where a member is built Both are what the code reads (prepare's runtime owner is the workspace manifest when there is one); the table named only the root.
…ge needs
Measured with mcpp-plugins 0.19.0 on the released 2026.10.1.2: the reader is told
error: mcpp.toml: error: [feature-xlings.deps-archive] xim:cmake:
unknown key 'provision' in a scoped entry; expected 'version' and 'when'
and nothing about the version, because the floor check needs the document that
this very parse failed to produce. Every release of a plugin collection raises
its floor, so this is the first thing a user on an older engine meets.
The floor is therefore read from the file's text in the parse-failure path --
`mcpp` inside `[package]`, nothing else -- and when this engine is below it, the
refusal says so and names the upgrade, in the words the floor check already uses.
`stated_mcpp_floor` is exported and seven cases pin the shapes it reads.
…er escapes them inside quotes too The first form single-quoted each argument, which a POSIX shell would take literally and this tokenizer does not: LLVM's GNU tokenizer escapes a backslash inside quotes as well as outside, so the Windows paths still arrived with their separators eaten. Measured with clang 22.1.8 -- a response file holding `'-DX=a\b'` yields `X=ab`, one holding `-DX=a\\b` yields `X=a\b` -- so every backslash is doubled, every quote escaped, and whitespace handled by quoting the whole argument. The case states the measurement.
…ays it cannot `--ld-path` was appended inside the Linux clang branch, the only one that consumes `link_toolchain_flags`. On macOS the stated linker therefore entered the fingerprint -- touching the wrapper declined the fast path -- and took no part in the link, with nothing said; the toolchain lab measured it on macos-15, where build.ninja held no `--ld-path`. e2e 875 asserts that flag but skips on a host without the llvm payload, which macOS CI is. The flag belongs to the driver, not to a platform, so it is added once after every shape has built its line. For a gcc toolchain the declaration is now refused where it is read: gcc selects a linker by the name `ld` inside a `-B` directory, so a program named anything else could not be chosen, and a silent `-B` would be the same defect in the other direction.
…its gcc example no longer states one
The cross example named `tools = { ld = ... }` on a gcc tree, which the engine now
refuses, and nothing said which trees read that role.
Every refusal this feature adds has a case; this one did not. It runs against an installed gcc payload and says so when none is present, rather than passing silently on a host without one.
…ppends itself
A shell on Windows answers `C:/Program Files/CMake/bin/cmake` for a `cmake.exe`,
and process creation there appends `.exe`, so a path stated without it names a
program the machine would run; refusing it answers about spelling rather than
about the machine. The plugin-side resolver gained the same rule in
mcpp-plugins 0.19.0, where CI measured the refusal.
Verified against a declared payload overridden by `{ program = "bin/faketool" }`
with only `bin/faketool.exe` present: the build reports
`Using xim:prec-absent <- .../bin/faketool.exe [custom . mcpp.toml:13]`.
…generate `00_fixture_path_hygiene.sh` caught three of them: 873 and 874 interpolated `$work` into a `[build-dependencies]` path, and 875 wrote the toolchain root, the linker wrapper and a driver straight from shell variables. MSYS rewrites paths in argv and the environment but not in file content, so on Windows a native mcpp would read `/d/a/...` and resolve it against the current drive -- the failure the helper's own header records costing a day. Each path now goes through `host_path`, and the assertions that compare mcpp's output compare host spellings too.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
A build uses a toolchain, the payloads its plugins declare, and the tools those plugins run. Each of them now has one source: a project can state it, a build program can decide it, and anyone can read it back. A project that writes none of the new keys builds exactly as before, and its output is unchanged byte for byte.
The problem this closes is a timing one. A payload a plugin declares is provisioned before any
build.mcppruns, so a build program that names its own tool downloaded the payload anyway, and offline the build was refused before the program could run at all.Payloads
[xlings.overrides]in the root manifest (also under[target.'cfg(..)']),MCPP_XLINGS_OVERRIDE_<NS>_<NAME>, and~/.mcpp/config.tomlstate where a declared payload comes from. An overridden payload is not provisioned and does not reach the offline gate;mcpp::xpkg_diranswers the root it implies, and the newxpkg_program/xpkg_sourceanswer the program it named andoverride. A statedversionis checked against every requirement the graph made. A dependency that writes the table is refused: which payloads a package needs is its own statement, where they come from is the project's.provision = "on-request"installs a payload when a build program asks for it withmcpp::xpkg_request. Every request of one invocation is installed together, only the programs that asked run again, and the run that asked is discarded.mcpp emit build-databaseinstalls nothing and recordsMCPP_BUILD_DATABASE_PAYLOAD_DEFERRED.Toolchains
[toolchain] <key> = { path = "<dir>", prefix, sysroot, family, launcher, tools }andMCPP_TOOLCHAIN=path:<dir>name a toolchain this machine already has. mcpp probes the drivers, identifies them, drives them with its own link model and hermetic check, and writes nothing into the tree. The driver and each stated tool enter the fingerprint by content, the fast paths decline when one changed, andmcpp.lockrecords it aslocal. This is not= "system": it is a tree the project names, in the shapemsvc@systemalready had.bootstrapnames the toolchain that compiles and runs build programs when it should not be the one building the project.{ configure = "build.mcpp" }hands the build toolchain to the root build program: it runs once in a toolchain phase (mcpp::phase()is"toolchain") and states the toolchain, and may state nothing else there.Observability
A source that is not the ecosystem's gets a line of its own, the
Finishedline summarises them, and the record is written toresolution.json:mcpp why sources,mcpp why tool <name>andmcpp why payload <ns:name>report it, including asmcpp.why.sourcesunder--format json.--managed-only/MCPP_MANAGED_ONLY=1refuses a build whose sources are not the ecosystem's, naming each.Protocol 15 adds
xpkg_source,xpkg_program,xpkg_request,xpkg_pending,phase,decisionandtoolchain.Specifications and documentation
SPEC-004 §4.7, SPEC-006 §2.2.1 and §3.3, SPEC-007 R6.2/R6.5/R6.6/R9.9 and the protocol table, SPEC build-database 1.5; docs/09, 20, 23, 30, 31, 32, 50 and their 简体中文 mirrors. The design record is
.agents/docs/2026-10-01-tool-and-toolchain-sources-design.md.Also fixed, each found by a platform or a measurement rather than by review
-fmodule-file=<name>=<path>per host module the program imports; mcpp-plugins'all-rules-compileimports fifteen and crossed the 8191 bytes the Windows shell tolerates the moment the collection gained one more module, reporting onlyThe command line is too long.The file is written in the grammar its driver reads — single quotes for clang and GCC, which treat a backslash as an escape, Windows quoting for cl and clang-cl — with a unit case per grammar. This repository's own CI cannot reach the path (the POSIX budget is 128 KiB); the plugins fixture does.unknown key 'provision' in a scoped entryand nothing about the version, because the engine floor is checked on the document that very parse failed to produce — and raising the floor is what every plugin release does. The floor is now read from the file's text in that path, and seven cases pin the shapes read.which()resolves a name that is also a shell builtin.command -v trueprintstrue, not a path, so a bare-name payload override of such a name was refused as not found on a machine carrying/usr/bin/true.Closes #755
Test plan
mcpp test: 144 passed, 0 failed, including the newtests/unit/test_sources.cpp(15 cases).mcpp whyand its machine output). Each usesMCPP_NO_AUTO_INSTALL=1as the criterion — a build that succeeds asked for no download — and each has a control that must refuse.xpkg_dir,feature-xlings,mcpp why,protocol=,MCPP_TOOLCHAIN,fast-path,emit build-database) pass.219_runtime_search_farm_is_lastfails identically on the released 2026.10.1.2 on this machine, and658needs an attached Android device.check_version_pins.sh,check_docs_style.sh,check_docs_structure.sh,check_reason_tokens.sh,check_file_lengths.sh,check_modules_wiring.sh,test_protocol_table.pyand the othertests/scripts/*.py.plugin-logiccases pass, andtests/cmake-consumerbuilds underMCPP_NO_AUTO_INSTALL=1when its build program names its own cmake, which is the behaviour this change exists for.