Repository navigation
fix: resolve the openkal index and runtime revisions once per run - #804
Sunrisepeak merged 3 commits into
Conversation
aff91b0 to
608d307
Compare
CI analysis: the two red checks are flaky, and neither one is this diffRun 38406939721 (this PR's head 1.
|
| time (UTC) | runner image | renaming ...\18\Enterprise\VC |
|---|---|---|
| 09:57 | windows-2025-vs2026, commit 20260925.250 |
ok |
| 13:59 (this run) | same | denied |
| 14:32 | same | ok |
ci-windows.yml is not in this diff, and the same step on the same image passed 33 minutes later (run 38411372563). The mask's own postcondition is deliberate — an incomplete mask must not read as a green "bare Windows" row — so a transient denial becomes a red check.
2. macos-e2e / e2e suite 2/2 (xcode-27) — 845 kills mcpp before the phase its assertion names
E2E Summary: 144 passed, 1 failed, 170 skipped, Failed: 845_play_game_restores_the_terminal.sh, from the script's own play.py:71:
AssertionError: G1: no status row carries the score
assert re.search(r"Building .* · snake \d+", plain), ...
The captured g3.log stops in the Programs phase:
Running ⠀⠐⠒⠂… 0/1 · 0:01 · build.mcpp app compiling 0:01 · snake 0
build.mcpp app ran 1.24s
Running ⠀⠀⠒⠒… 1/1 · 0:01 · snake 0
play.py sends the arrows and the Ctrl-C at a fixed t0 + 1.5s, while the build reaches Phase::Building only after the Programs phase (build.mcpp compile + run, plus toolchain resolution), which took ~1.24s here. The rows do carry the score in the Programs frames; phase_status() prints Running for Phase::Programs and Building only for Phase::Building (src/build/progress.cppm:1074), so the only missing token is the phase word. A 1.5s deadline against ~1.24s of work is decided by host jitter:
- the macos-15 twin of this job passed in this same run;
- on main (38378041638) the macos-15 twin failed with the identical assertion while xcode-27 passed;
- 2 of the last 11
ciruns failede2e suite 2/2, the two hosts alternating.
Next
Both are fixed in a follow-up PR: e2e 845 waits for the frame its assertion names instead of for a wall-clock delay, and the Visual Studio mask — currently three copies of the same block, in ci-windows.yml, ci-fresh-install.yml and measure-windows-tool-crt.yml — gains retries with a one-level-deeper fallback, with the postcondition kept.
|
Both are fixed in #806: e2e 845 now waits for the status row its assertion names instead of for a wall-clock delay, and the Visual Studio mask (three copies, now one action) retries a refused rename and falls back to masking |
|
@Sunrisepeak 我把 #806 的修改 cherry pick 过来后, CI 现已通过 |
|
5ff0678 to
7102abb
Compare
The four build jobs of openkal-cross.yml each cloned `main` of the index and of the runtime of their own accord, at four different moments on four runners that become available at four different times, and the run jobs then asserted that all four had built the same sources. On 2026-10-13 that assertion failed for a reason the change under test had nothing to do with: run 38399400601 started at 12:43 UTC, its build jobs at 12:52:08 through 13:01:33, and mcpp-index@main took four commits inside that window (21b9a77a 12:56:12, d3a328c8 13:04:42, cfb808bc 13:10:06, 0340de61 13:17:58), so the four clones straddled pushes and the run jobs reported `index sources differ`. Resolve both refs once, in a new `revisions` job, and have every host materialise the commit it names. OPENKAL_BRANCH keeps its meaning: an override still verifies exactly what it named, now as one commit rather than as whatever the branch had become when each runner woke up. The run jobs keep their cross-host comparison; it is what proves a host used the pin rather than re-resolving for itself. git_clone_retry.sh grows the two primitives that needs: `--rev <ref> <url>`, which prints the commit a ref names without downloading the tree, and `--commit <sha> <url> <dir>`, which materialises exactly that commit, both under the retry the helper already exists for. Writing a SHA into the workflow instead would be the pin-that-keeps-a-check-green-by-keeping-it- out-of-date that the file itself warns about.
7102abb to
6ff9cca
Compare
The job died on 2026-10-15 (run 38695300904) in apt's unpack of wine's
i386 closure, before a single byte was compiled:
dpkg: error processing archive
.../libxml2_2.9.14+dfsg-1.3ubuntu3.10_i386.deb (--unpack):
trying to overwrite shared
'/usr/share/doc/libxml2/changelog.Debian.gz', which is different
from other instances of package libxml2:i386
libpng16-16t64, libx11-6 and libfreetype6, all alike. Multi-Arch: same
instances must share every file bit-for-bit, and the i386 and amd64 debs
in the Ubuntu archive embed changelogs that differ (Launchpad #1574351
and its kin). That is a property of the archive, not of the runner: the
step's own `|| sudo apt-get install -y wine` retry failed with the
identical error, so a rerun re-rolls nothing.
Give dpkg --force-overwrite on every install this step makes, the cold
path and the cached-debs path alike. The changelogs are not what this
job reads; wine is.
@Sunrisepeak 改好了 |
Run 38695300904 failed on a cache HIT, not in the cold path: the cached libxml2:i386 2.9.14+dfsg-1.3ubuntu3.9 was installed with `dpkg -i` next to the image's libxml2:amd64 ...3.10, then `apt-get install -f` upgraded it, and that upgrade's shared changelog was refused. The cause is the key: `wine-debs-...-v1` names nothing that fills the cache, and the save runs only on a miss, so the packages were saved once and never again while the image moved on (the 2026-10-02 CI record, 2.4.4: a cache saved under an unchanged key is never saved again). `--force-overwrite` let the stale install through instead of removing it. - The key carries the runner's ImageVersion; restore-keys hands a new image the previous image's packages until main saves its own. - The cache seeds /var/cache/apt/archives and apt installs, every time. A cached package still at the current version is used as is, a newer one is fetched, and nothing older than the image is installed, so there is no conflict to force through. - The third-party apt lists are dropped on both paths, since both now run `apt-get update`. - What main saves is what this image installs; `autoclean` drops packages no index offers any more, so the cache does not grow with each image. This PR's own run exercises the stale path: its key misses, and restore-keys returns the `-v1` cache that failed.
2 participants
Summary
openkal-cross.ymlhas fourbuildjobs that each clonedmainofmcpp-community/mcpp-indexand ofmcpplibs/openkal-llvm-runtimeof their ownaccord, and three
runjobs that then assert all four hosts built the samesources. Four clones of a moving branch are not four copies of one revision.
That is what failed in run 38399400601
(PR #803, 2026-10-13): all three
runjobs reportedThe run started at 12:43 UTC and its build jobs at 12:52:08, 12:53:44,
12:58:20 and 13:01:33 UTC, and they do not clone at the same point in their own
step sequence (the macOS leg builds its own mcpp first), while
mcpp-index@maintook four commits inside that window —21b9a77a12:56:12,d3a328c813:04:42,cfb808bc13:10:06,0340de6113:17:58. Two clonesstraddling one push is all it takes, and the reading arrives 40 minutes later
in a job that built nothing. Nothing about the change under test was involved,
and a rerun only re-rolls the dice.
The fix
revisionsjob runs first and resolvesmcpp-index@mainandopenkal-llvm-runtime@$OPENKAL_BRANCHto one commit each.buildjobs materialise those commits instead of cloning a branch.The
runjobs keep their cross-host comparison: it is now what proves a hostused the pin rather than re-resolving for itself.
OPENKAL_BRANCHkeeps its meaning — an override still verifies exactly whatit named, resolved once instead of once per runner.
git_clone_retry.shgrows the two primitives that needs, under the retry thehelper already exists for (a resolver blip is not a red build):
--rev <ref> <url>— print the commit a ref names, without downloading thetree (
ls-remote; both repositories are ~17 MB).--commit <sha> <url> <dir>— materialise exactly that commit(
init+fetch --depth 1 origin <sha>, since--branchtakes a branch ora tag and a commit is neither).
Pinning a SHA into the workflow was the alternative and it is the wrong one:
it is the pin-nobody-is-reminded-to-move that the file's own
OPENKAL_BRANCHcomment is annotated with.
Verification
git_clone_retry.sh: 16 cases against throwaway local repositories —--revfor a branch name, a full ref andHEAD; a ref that does not exist(exit 1, no retry); an unreachable URL (retries, then gives up);
--commitfor an older commit (detached, no stray branch ref), the tip, a
0…0SHA anda non-SHA; a non-empty destination (self-heals); the plain-clone path and its
partial-clone removal. All pass;
bash -nclean.--rev mainonmcpp-indexresolved to0340de61ffc83f0eeeae459e74a4156b11935897, and--commitmaterialisedexactly that commit (813 files, detached HEAD) — the fetch-by-SHA the pin
depends on works on the host the jobs run against.
revisionsstep's shell was executed with a stub helper: it emitsindex=/runtime=to$GITHUB_OUTPUT, calls the helper with the two refs,and fails with
::error::not a commiton a non-commit.tests/e2e/232_workflow_syntax.shpasses (24 workflows load);.github/tools/check_workflow_assertions.pyreports 0 problems;tools/lint-ci-assertions.shreports the same two pre-existing warnings asbefore this change;
git diff --checkclean..github/**classifies ascode=true, so this PR's own CI runs the fullmatrix — including the four-host openkal build and the three
runjobs thepin is for.
Test plan
mcpp test(unit/integration): not applicable, no C++ changed