Repository navigation
Fix the n8n, Euro-Office plus, BuildKit prune and Zitadel rolling failures - #96
Merged
Merged
Conversation
From n8n commit c86615646b (2026-11-10), the n8n and runners Dockerfiles
declare BUILDER_IMAGE, RUNTIME_IMAGE and NODE_IMAGE without defaults.
docker/docker-bake.hcl holds the pins. The plans ran docker buildx build
with three fixed build arguments, so every lane of every n8n case failed
with "base name (${BUILDER_IMAGE}) should not be blank".
Preparation now resolves the n8n, runners and runners-distroless Bake
targets at the checked-out commit and adds their arguments to the run's
plan command. All lanes build the same images. The fixed NODE_VERSION,
N8N_VERSION and N8N_RELEASE_TYPE arguments are removed, because the
targets set them. Preparation stops if a target builds a different
Dockerfile or the plan already sets one of its arguments.
From Euro-Office core commit 2a3371b60b (2026-11-05), the core stage builds through scripts/build.sh. The stage has an ARG CORE_CCACHE switch and a core-ccache cache mount. The CMake lines that the plus overlay edited are gone, so preparation for every plus lane of euro-office and euro-office-cache stopped with "core Dockerfile changed: expected one "-DCMAKE_BUILD_TYPE=Release \\\n"". On core revisions with that switch, the overlay copies the pinned ccache binaries into the core stage and sets CORE_CCACHE=1. Lanes that do not archive local compiler caches also drop the core-ccache mount, as they already did for the WASM ccache mount. Earlier core revisions keep the existing edits. Base lanes still build the upstream Dockerfile unchanged.
The prune job refused to delete anything unless 95% of the blobs that the newest active indexes reference were listed. From 2026-11-05 GitHub evicted cache entries that those indexes still reference. Coverage fell from 100% to 61.9%, and every scheduled run since has failed without deleting. The stale backlog grew to 217 GB, which adds to the eviction. The guard now requires 50%. A share below that still stops the job, because it means the index format and the blob keys no longer match. Unlisted blobs are never deletion candidates, so a lower share does not put referenced blobs at risk.
Upstream replaced moon with Turborepo in zitadel/nextgen commit 8ff8273110 (2026-11-12). Every rolling run since fails with Command "moon" not found, so the project leaves the rolling schedule. The case definition and its records stay.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Four harness fixes for rolling projects and the prune job. Each one has failed on every run for days because upstream changed.
Docker n8n: Dockerfiles need Bake arguments
All 18 jobs of
Docker - n8nhave failed since 2026-11-10 (run 43431250145):From n8n commit
c86615646b,docker/images/n8n/Dockerfileand the runners Dockerfiles declareBUILDER_IMAGE,RUNTIME_IMAGEandNODE_IMAGEwithout defaults. The pins are only indocker/docker-bake.hcl. Our plans randocker buildx buildwith three fixed build arguments.tools/docker/shared/bake-build-args.rbresolves then8n,runnersorrunners-distrolesstarget withdocker buildx bake --printat the checked-out commit. It adds the target's arguments to the run's plan command after--file. Every lane reads that command, so all lanes build the same image.NODE_VERSION=26.7.0,N8N_VERSIONandN8N_RELEASE_TYPEarguments are removed. The targets set all three.start_shathe three targets build the same Dockerfiles. Their arguments are the ones the plans used to pass, so fresh runs are unchanged.Docker Euro-Office plus lanes: the core stage now has its own ccache switch
Every plus lane of
euro-officeandeuro-office-cachehas failed in preparation since 2026-11-04 (run 43428499245, run 43428502714):From Euro-Office core commit
2a3371b60b, the core stage runsscripts/build.sh. That script adds the ccache launchers whenARG CORE_CCACHEis set, and the stage has acore-ccachemount at/ccache.CORE_CCACHE, the overlay copies the pinned ccache binaries into the core stage and setsCORE_CCACHE=1.core-ccachemount, as they already drop the WASM ccache mount.Against the core Dockerfile at
651145267514(the revision in the failing runs), the generated file differs from upstream by the twoCOPYlines andARG CORE_CCACHE=1. Without local caches, it also drops the mount line.Prune BuildKit cache: guard blocked every run
Every scheduled prune has failed since 2026-11-05 (run 43427645619):
Through 2026-11-04, coverage was 100% and each run deleted 400 entries. After that, GitHub began evicting cache entries: the stale blob count fell from 20,158 (2026-11-04 18:43 UTC) to 2,665 (2026-11-05 08:43 UTC), and this job deleted nothing in that time. Some of the evicted blobs are still referenced by the newest indexes, so coverage dropped and the guard stopped all cleanup. The repository's Actions cache now holds 532 GB in 27,000 entries.
The guard now requires 50%. Its purpose is to catch an index format that no longer matches the blob keys, and a mismatch like that shows up near 0%. Unlisted blobs are never deletion candidates, so a lower share does not put referenced blobs at risk.
AGENTS.mddescribes the rule.moon Zitadel: stop rolling
Upstream replaced moon with Turborepo in
zitadel/nextgencommit8ff8273110on 2026-11-12. Every run since fails withCommand "moon" not found(run 43311950530). From 2026-11-03 the builds passed but the output check failed, because the console output moved toapi/internal/staticui/console/dist. The project leavesrolling.toml; the case and its records stay.PLAN.mdlists it under stopped cases.Not changed here
boringcache plus: needs the same CLI release.bitrise(workspace build cache not activated) and Nix Zedcachix(expired token): account issues, the lanes keep failing.Checks
bundle exec ruby -Ilib -e 'Dir["test/bench/*_test.rb"].each { require_relative it }': 215 runs, 0 failures, 3 skips.bin/bench check: ok.test/bench/bake_build_args_test.rb; build-script core cases intest/bench/euro_office_preparation_test.rb; an eviction case intest/bench/buildkit_prune_test.rb.Docker - n8n,ghalane on GitHub x64 (run 43439161934): all three cases passed cold and warm. Preparation resolved each Bake target atstart_sha, and the builds used the samenode:26.7.0base as before. The Euro-Office change is covered by unit tests only, because a fresh run buildsstart_sha, whose core stage predatesCORE_CCACHE.After merge
The next rolling step for n8n and Euro-Office runs with these changes. The next scheduled prune should delete up to 400 stale entries.