Skip to content

Fix the n8n, Euro-Office plus, BuildKit prune and Zitadel rolling failures - #96

Merged
gauravtiwari merged 4 commits into
mainfrom
fix-rolling-harness-failures
Nov 13, 2026
Merged

gauravtiwari merged 4 commits into
mainfrom
fix-rolling-harness-failures

Conversation

@gauravtiwari

@gauravtiwari gauravtiwari commented Nov 13, 2026 •

Copy link
Copy Markdown
Contributor

Four harness fixes for rolling projects and the prune job. Each one has failed on every run for days because upstream changed.

Docker n8n: Dockerfiles need Bake arguments

All 18 jobs of Docker - n8n have failed since 2026-11-10 (run 43431250145):

ERROR: failed to build: failed to solve: base name (${BUILDER_IMAGE}) should not be blank
ERROR: failed to build: failed to solve: base name (${NODE_IMAGE}) should not be blank

From n8n commit c86615646b, docker/images/n8n/Dockerfile and the runners Dockerfiles declare BUILDER_IMAGE, RUNTIME_IMAGE and NODE_IMAGE without defaults. The pins are only in docker/docker-bake.hcl. Our plans ran docker buildx build with three fixed build arguments.

  • tools/docker/shared/bake-build-args.rb resolves the n8n, runners or runners-distroless target with docker buildx bake --print at the checked-out commit. It adds the target's arguments to the run's plan command after --file. Every lane reads that command, so all lanes build the same image.
  • The fixed NODE_VERSION=26.7.0, N8N_VERSION and N8N_RELEASE_TYPE arguments are removed. The targets set all three.
  • Preparation stops if the target builds a different Dockerfile from the plan, or if the plan already sets one of the target's arguments.
  • At start_sha the three targets build the same Dockerfiles. Their arguments are the ones the plans used to pass, so fresh runs are unchanged.

Docker Euro-Office plus lanes: the core stage now has its own ccache switch

Every plus lane of euro-office and euro-office-cache has failed in preparation since 2026-11-04 (run 43428499245, run 43428502714):

core Dockerfile changed: expected one "-DCMAKE_BUILD_TYPE=Release \\\n"

From Euro-Office core commit 2a3371b60b, the core stage runs scripts/build.sh. That script adds the ccache launchers when ARG CORE_CCACHE is set, and the stage has a core-ccache mount at /ccache.

  • For core revisions that declare CORE_CCACHE, the overlay copies the pinned ccache binaries into the core stage and sets CORE_CCACHE=1.
  • Lanes that do not archive local compiler caches drop the core-ccache mount, as they already drop the WASM ccache mount.
  • Earlier core revisions keep the existing edits. The WASM Dockerfile still matches. Base lanes build upstream unchanged.

Against the core Dockerfile at 651145267514 (the revision in the failing runs), the generated file differs from upstream by the two COPY lines and ARG CORE_CCACHE=1. Without local caches, it also drops the mount line.

Prune BuildKit cache: guard blocked every run

Every scheduled prune has failed since 2026-11-05 (run 43427645619):

bench: not deleting: index coverage is below 95%
50 active BuildKit scopes, 49 with an index; 1809 referenced blobs, 690 missing (coverage 61.9%)
stale: 2209 blobs and 92 indexes, 217.0 GB

Through 2026-11-04, coverage was 100% and each run deleted 400 entries. After that, GitHub began evicting cache entries: the stale blob count fell from 20,158 (2026-11-04 18:43 UTC) to 2,665 (2026-11-05 08:43 UTC), and this job deleted nothing in that time. Some of the evicted blobs are still referenced by the newest indexes, so coverage dropped and the guard stopped all cleanup. The repository's Actions cache now holds 532 GB in 27,000 entries.

The guard now requires 50%. Its purpose is to catch an index format that no longer matches the blob keys, and a mismatch like that shows up near 0%. Unlisted blobs are never deletion candidates, so a lower share does not put referenced blobs at risk. AGENTS.md describes the rule.

moon Zitadel: stop rolling

Upstream replaced moon with Turborepo in zitadel/nextgen commit 8ff8273110 on 2026-11-12. Every run since fails with Command "moon" not found (run 43311950530). From 2026-11-03 the builds passed but the output check failed, because the console output moved to api/internal/staticui/console/dist. The project leaves rolling.toml; the case and its records stay. PLAN.md lists it under stopped cases.

Not changed here

  • Pants backend-ai: waiting for a CLI with the larger archive metadata limit, then a series restart.
  • Rust/Cargo Zed boringcache plus: needs the same CLI release.
  • Bazel gRPC bitrise (workspace build cache not activated) and Nix Zed cachix (expired token): account issues, the lanes keep failing.

Checks

  • bundle exec ruby -Ilib -e 'Dir["test/bench/*_test.rb"].each { require_relative it }': 215 runs, 0 failures, 3 skips. bin/bench check: ok.
  • New tests: test/bench/bake_build_args_test.rb; build-script core cases in test/bench/euro_office_preparation_test.rb; an eviction case in test/bench/buildkit_prune_test.rb.
  • No workflow files changed.
  • Branch fresh run of Docker - n8n, gha lane on GitHub x64 (run 43439161934): all three cases passed cold and warm. Preparation resolved each Bake target at start_sha, and the builds used the same node:26.7.0 base as before. The Euro-Office change is covered by unit tests only, because a fresh run builds start_sha, whose core stage predates CORE_CCACHE.

After merge

The next rolling step for n8n and Euro-Office runs with these changes. The next scheduled prune should delete up to 400 stale entries.

From n8n commit c86615646b (2026-11-10), the n8n and runners Dockerfiles
declare BUILDER_IMAGE, RUNTIME_IMAGE and NODE_IMAGE without defaults.
docker/docker-bake.hcl holds the pins. The plans ran docker buildx build
with three fixed build arguments, so every lane of every n8n case failed
with "base name (${BUILDER_IMAGE}) should not be blank".

Preparation now resolves the n8n, runners and runners-distroless Bake
targets at the checked-out commit and adds their arguments to the run's
plan command. All lanes build the same images. The fixed NODE_VERSION,
N8N_VERSION and N8N_RELEASE_TYPE arguments are removed, because the
targets set them. Preparation stops if a target builds a different
Dockerfile or the plan already sets one of its arguments.
From Euro-Office core commit 2a3371b60b (2026-11-05), the core stage
builds through scripts/build.sh. The stage has an ARG CORE_CCACHE switch
and a core-ccache cache mount. The CMake lines that the plus overlay
edited are gone, so preparation for every plus lane of euro-office and
euro-office-cache stopped with "core Dockerfile changed: expected one
"-DCMAKE_BUILD_TYPE=Release \\\n"".

On core revisions with that switch, the overlay copies the pinned ccache
binaries into the core stage and sets CORE_CCACHE=1. Lanes that do not
archive local compiler caches also drop the core-ccache mount, as they
already did for the WASM ccache mount. Earlier core revisions keep the
existing edits. Base lanes still build the upstream Dockerfile unchanged.
The prune job refused to delete anything unless 95% of the blobs that the
newest active indexes reference were listed. From 2026-11-05 GitHub
evicted cache entries that those indexes still reference. Coverage fell
from 100% to 61.9%, and every scheduled run since has failed without
deleting. The stale backlog grew to 217 GB, which adds to the eviction.

The guard now requires 50%. A share below that still stops the job,
because it means the index format and the blob keys no longer match.
Unlisted blobs are never deletion candidates, so a lower share does not
put referenced blobs at risk.
Upstream replaced moon with Turborepo in zitadel/nextgen commit
8ff8273110 (2026-11-12). Every rolling run since fails with Command
"moon" not found, so the project leaves the rolling schedule. The case
definition and its records stay.
@gauravtiwari
gauravtiwari merged commit 6ab3dd3 into main Nov 13, 2026
15 checks passed
@gauravtiwari
gauravtiwari deleted the fix-rolling-harness-failures branch November 13, 2026 15:30
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant