Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion AGENTS.md
Original file line number Diff line number Diff line change
Expand Up @@ -35,7 +35,7 @@ This repository benchmarks BoringCache against other cache providers on real pro
- Each regular lane rolls on one fixed cache scope, `<lane>-<runner>-rolling`. `[samples]` reserves selected lanes for weekly `mode=sample` builds of the latest completed GitHub source commit; samples also run when upstream has no new commit and do not advance rolling. They share the project's rolling concurrency group. BoringCache samples restore the regular GitHub cache without publishing; native caches persist between samples. Fresh runs use their own scopes and exclude sample lanes. Legacy `[nightly]` behavior remains supported for historical configurations.
- When a lane, runner or rolling series leaves `[runs]`, delete its BoringCache tags with `boringcache rm --no-platform --no-git boringcache/benchmarks <tags>`, listing them by exact name. Its records stay in `results/`.
- `schedule.yml` runs `bin/bench tick` every 15 minutes. It checks each idle project in `rolling.toml` against its upstream branch and dispatches only when a commit is due at its configured stride. Active or queued rolling and sample projects wait for a later tick. An upstream check failure is reported and fails the tick without dispatching that project. The workflow's concurrency group runs one rolling run per project at a time.
- `prune-buildkit.yml` deletes stale BuildKit `type=gha` blobs every 2 hours.
- `prune-buildkit.yml` deletes stale BuildKit `type=gha` blobs every 2 hours. It deletes only when at least half of the blobs referenced by the newest active indexes are listed. A lower share means the index and blob keys no longer match. GitHub can evict blobs that an index still references, so some missing blobs are expected.
- `report.yml` regenerates `data/results.json` and `data/report.md` hourly from recorded results. Experimental lanes are excluded.
- A failed build or output check is recorded first, then fails its job. A failure in `prepare` leaves no record; that lane continues at the next step.
- Re-run a failed rolling job only before the next step of that project starts. A re-run of an older step skips its build once a newer step is recorded for that lane, so it can never overwrite the newer cache.
Expand Down
1 change: 1 addition & 0 deletions PLAN.md
Original file line number Diff line number Diff line change
Expand Up @@ -45,6 +45,7 @@ The superseded measurements are available in [Git history](https://github.com/bo
| Docker Vocion | 2026-10-28 | Removed from the rolling schedule at the user’s request. Definitions and measurements remain in `tools/docker/vocion` and `results/docker/vocion`. Last completed publication run: [40686377453](https://github.com/boringcache/benchmarks/actions/runs/40686377453). Filesystem diagnostic [40676640825](https://github.com/boringcache/benchmarks/actions/runs/40676640825) stopped; its dedicated workflow is disabled. |
| Docker Eclipse OpenSOVD | 2026-10-28 | Removed from the rolling schedule at the user’s request. Definitions and measurements remain in `tools/docker/opensovd` and `results/docker/opensovd`. Last completed run: [40673476105](https://github.com/boringcache/benchmarks/actions/runs/40673476105). |
| Cargo Bevy Explorer | 2026-10-28 | Dropped because [the upstream repository](https://github.com/decentraland/bevy-explorer) is archived; GitHub API verified `archived: true` on this date. Removed from the rolling schedule. Definitions and measurements remain in `tools/cargo/bevy-explorer` and `results/cargo/bevy-explorer`. Last completed run: [40686361441](https://github.com/boringcache/benchmarks/actions/runs/40686361441). |
| moon Zitadel | 2026-11-13 | Removed from the rolling schedule. Upstream replaced moon with Turborepo in `zitadel/nextgen` commit `8ff8273110` on 2026-11-12, and every rolling run since fails with `Command "moon" not found` ([run 43311950530](https://github.com/boringcache/benchmarks/actions/runs/43311950530)). From 2026-11-03 the builds passed but the output check failed, because the console build moved to `api/internal/staticui/console/dist` ([run 41580361010](https://github.com/boringcache/benchmarks/actions/runs/41580361010)). Definitions and measurements remain in `tools/moon/zitadel` and `results/moon/zitadel`. Last completed run: [41579157320](https://github.com/boringcache/benchmarks/actions/runs/41579157320). |

## Removed cases

Expand Down
2 changes: 1 addition & 1 deletion lib/bench/buildkit_prune.rb
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,7 @@ class BuildkitPrune
INDEX = /\Aindex-(?<scope>.+)-1-[0-9a-f]{8}(?:-sig)?#(?<version>\d+)\z/
MIN_AGE = 6 * 3600
KEEP_INDEXES = 2
MIN_COVERAGE = 0.95
MIN_COVERAGE = 0.5

Plan = Struct.new(:blobs, :indexes, :referenced, :missing, :coverage, :deletable, keyword_init: true) do
def bytes
Expand Down
1 change: 0 additions & 1 deletion rolling.toml
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,6 @@ docker = ["chroma", "discourse", "dspace", "duckgres", "euro-office", "euro-offi
go = ["hugo"]
gradle = ["opentelemetry-java"]
maven = ["spring-ai"]
moon = ["zitadel"]
nix = ["zed"]
nx = ["storybook"]
pants = ["backend-ai"]
Expand Down
67 changes: 67 additions & 0 deletions test/bench/bake_build_args_test.rb
Original file line number Diff line number Diff line change
@@ -0,0 +1,67 @@
require_relative "test_helper"
require "open3"

class BakeBuildArgsTest < Minitest::Test
SCRIPT = File.join(Bench::ROOT, "tools/docker/shared/bake-build-args.rb")

def test_n8n_plans_build_with_the_arguments_their_bake_target_resolves
{ "n8n" => "n8n", "n8n-runners" => "runners", "n8n-runners-distroless" => "runners-distroless" }.each do |kase, target|
prepare = TomlRB.load_file(File.join(Bench::ROOT, "tools/docker", kase, "case.toml")).fetch("prepare")
assert_equal %(ruby "$BENCH_ROOT/tools/docker/shared/bake-build-args.rb" upstream/docker/docker-bake.hcl #{target}), prepare.last
command = TomlRB.load_file(File.join(Bench::ROOT, "tools/docker", kase, ".boringcache.toml")).dig("adapters", "docker", "command")
refute_includes command, "--build-arg"
end
end

def test_resolved_arguments_follow_the_dockerfile
run_script(args: { "BUILDER_IMAGE" => "node:26.13.0-alpine3.24@sha256:aa", "RUNTIME_IMAGE" => "n8nio/base:26.13.0@sha256:bb" }) do |command, output, status|
assert status.success?, output
assert_equal ["docker", "buildx", "build", "--file", "upstream/docker/images/n8n/Dockerfile",
"--build-arg", "BUILDER_IMAGE=node:26.13.0-alpine3.24@sha256:aa", "--build-arg", "RUNTIME_IMAGE=n8nio/base:26.13.0@sha256:bb",
"--tag", "n8n-benchmark:local", "--load", "upstream"], command
end
end

def test_failed_bake_resolution_stops_preparation
run_script(resolve: false) do |command, output, status|
refute status.success?
assert_includes output, "Cannot resolve the n8n Bake target"
refute_includes command, "--build-arg"
end
end

def test_a_target_for_another_dockerfile_stops_preparation
run_script(dockerfile: "docker/images/n8n/Dockerfile.debian") do |command, output, status|
refute status.success?
assert_includes output, "builds upstream/docker/images/n8n/Dockerfile.debian"
refute_includes command, "--build-arg"
end
end

def test_an_argument_set_by_the_plan_and_the_target_stops_preparation
run_script(planned: ["--build-arg", "BUILDER_IMAGE=node:22"], args: { "BUILDER_IMAGE" => "node:26" }) do |command, output, status|
refute status.success?
assert_includes output, "both set BUILDER_IMAGE"
assert_equal 1, command.count("--build-arg")
end
end

private
def run_script(args: {}, resolve: true, dockerfile: "docker/images/n8n/Dockerfile", planned: [])
Dir.mktmpdir("bake-args") do |dir|
FileUtils.mkdir_p(File.join(dir, "bin"))
command = ["docker", "buildx", "build", "--file", "upstream/docker/images/n8n/Dockerfile", *planned, "--tag", "n8n-benchmark:local", "--load", "upstream"]
File.write(File.join(dir, ".boringcache.toml"), TomlRB.dump("adapters" => { "docker" => { "tag" => "n8n-docker", "command" => command } }))
definition = { "target" => { "n8n" => { "context" => ".", "dockerfile" => dockerfile, "args" => args } } }
File.write(File.join(dir, "bin/docker"), <<~RUBY)
#!#{RbConfig.ruby}
abort "Expected Bake inspection" unless ARGV == %w[buildx bake --file upstream/docker/docker-bake.hcl --print n8n]
exit 1 unless #{resolve}
puts #{JSON.generate(definition).inspect}
RUBY
File.chmod(0o755, File.join(dir, "bin/docker"))
output, status = Open3.capture2e({ "PATH" => "#{dir}/bin:#{ENV.fetch("PATH")}" }, RbConfig.ruby, SCRIPT, "upstream/docker/docker-bake.hcl", "n8n", chdir: dir)
yield TomlRB.load_file(File.join(dir, ".boringcache.toml")).dig("adapters", "docker", "command"), output, status
end
end
end
9 changes: 9 additions & 0 deletions test/bench/buildkit_prune_test.rb
Original file line number Diff line number Diff line change
Expand Up @@ -41,6 +41,15 @@ def test_nothing_is_deletable_when_the_index_does_not_match_the_listed_blobs
assert_equal [2, 0.0, false], [result.missing, result.coverage, result.deletable]
end

def test_blobs_evicted_after_their_index_was_written_do_not_stop_the_prune
indexes = [index("docker-app-gha-github-rolling", 1)]
layers = { indexes.first["key"] => %w[sha256:aa sha256:bb sha256:cc sha256:dd sha256:ee] }
result = plan(layers:, blobs: [blob("aa"), blob("bb"), blob("cc"), blob("ff")], indexes:)

assert_equal [2, 0.6, true], [result.missing, result.coverage, result.deletable]
assert_equal %w[sha256:ff], result.blobs.map { it["key"].delete_prefix("buildkit-blob-1-") }
end

def test_the_newest_index_is_chosen_by_its_version_number
indexes = [index("docker-app-gha-github-rolling", 9), index("docker-app-gha-github-rolling", 10)]

Expand Down
65 changes: 61 additions & 4 deletions test/bench/euro_office_preparation_test.rb
Original file line number Diff line number Diff line change
Expand Up @@ -17,6 +17,27 @@ def test_plus_preserves_source_and_adds_compiler_and_mount_caches
end
end

def test_plus_turns_on_the_upstream_ccache_switch_of_the_build_script_core
prepare_core(source: build_script_core_source) do |dir, source, output, status|
assert status.success?, output
assert_equal source, File.read(File.join(dir, "upstream/core/.docker/core.bake.Dockerfile"))
dockerfile = File.read(File.join(dir, "overlay/core.Dockerfile"))
assert_includes dockerfile, "FROM core-base AS core\nCOPY bench-ccache/ccache /usr/local/bin/ccache\nCOPY bench-ccache/ccache-storage-http /usr/local/bin/ccache-storage-http\n"
assert_includes dockerfile, " ARG CORE_CCACHE=1\n"
assert_includes dockerfile, "--mount=type=cache,target=/ccache,id=core-ccache"
assert_includes dockerfile, "id=vcpkg-binary-cache"
assert_equal source.lines.size + 2, dockerfile.lines.size
end
end

def test_build_script_core_without_its_ccache_switch_stops_preparation
prepare_core(source: build_script_core_source.sub(" ARG CORE_CCACHE\n", "")) do |dir, _, output, status|
refute status.success?
assert_includes output, "core Dockerfile changed"
refute File.exist?(File.join(dir, "overlay/core.Dockerfile"))
end
end

def test_changed_compile_step_stops_preparation
prepare_core(replace: "cmake --build . --parallel") do |dir, _, output, status|
refute status.success?
Expand Down Expand Up @@ -97,13 +118,30 @@ def test_native_control_archives_compiler_caches_as_well_as_build_state
end
end

def test_combined_cache_leaves_the_build_script_core_compiler_cache_remote
prepare_standalone(plus: true, core: build_script_core_source) do |dir, _, output, status|
assert status.success?, output
core = File.read(File.join(dir, "overlay/core.Dockerfile"))
assert_includes core, " ARG CORE_CCACHE=1\n"
assert_includes core, "target=/build-cache"
refute_includes core, "target=/ccache"
end
end

def test_native_control_keeps_the_build_script_core_compiler_cache_mount
prepare_standalone(plus: true, local_cache: true, core: build_script_core_source) do |dir, _, output, status|
assert status.success?, output
assert_includes File.read(File.join(dir, "overlay/core.Dockerfile")), "--mount=type=cache,target=/ccache,id=core-ccache"
end
end

private
def prepare_standalone(resolve: true, plus: false, local_cache: false)
def prepare_standalone(resolve: true, plus: false, local_cache: false, core: core_source)
Dir.mktmpdir("euro-office") do |dir|
%w[upstream/core/.docker overlay bin].each { FileUtils.mkdir_p(File.join(dir, it)) }
File.write(File.join(dir, "upstream/VERSION"), "9.3.6-rc.1\n")
File.write(File.join(dir, ".boringcache.toml"), "[adapters.docker]\ntool-cache = #{plus ? '["ccache:test"]' : '[]'}\n")
File.write(File.join(dir, "upstream/core/.docker/core.bake.Dockerfile"), core_source)
File.write(File.join(dir, "upstream/core/.docker/core.bake.Dockerfile"), core)
File.write(File.join(dir, "upstream/core/.docker/core-wasm.bake.Dockerfile"), <<~'DOCKERFILE')
FROM emsdk AS core-wasm
COPY core /core
Expand Down Expand Up @@ -134,11 +172,10 @@ def prepare_standalone(resolve: true, plus: false, local_cache: false)
end
end

def prepare_core(plus: true, replace: nil)
def prepare_core(plus: true, replace: nil, source: core_source)
Dir.mktmpdir("euro-office") do |dir|
FileUtils.mkdir_p(File.join(dir, "upstream/core/.docker"))
FileUtils.mkdir_p(File.join(dir, "overlay"))
source = core_source
source = source.sub("cmake --build .", replace) if replace
File.write(File.join(dir, "upstream/core/.docker/core.bake.Dockerfile"), source)
File.write(File.join(dir, ".boringcache.toml"), "[adapters.docker]\nmount-cache = #{plus}\n")
Expand All @@ -161,6 +198,26 @@ def core_source
DOCKERFILE
end

def build_script_core_source
<<~'DOCKERFILE'
FROM core-base AS core

COPY core /core

ARG CORE_CCACHE
RUN --mount=type=cache,target=/build-cache \
--mount=type=cache,target=/nuget-cache,id=vcpkg-binary-cache \
--mount=type=cache,target=/ccache,id=core-ccache \
--mount=type=secret,id=remote_cache_s3_access_key \
export CCACHE_DIR=/ccache CCACHE_MAXSIZE=2G && \
mkdir -p ${BUILD_ROOT} && \
${CORE_CCACHE:+ccache --zero-stats} && \
/core/scripts/build.sh --build-dir /build-cache --output /build-cache/package ${CORE_CCACHE:+--ccache} && \
${CORE_CCACHE:+ccache --show-stats} && \
cp -r /build-cache/package/* ${BUILD_ROOT}
DOCKERFILE
end

def stage_stub(dir)
helper = File.join(dir, "tools/docker/shared/stage-ccache-binary.sh")
FileUtils.mkdir_p(File.dirname(helper))
Expand Down
12 changes: 12 additions & 0 deletions tools/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -131,6 +131,14 @@ objects before the next target. These files were already excluded from every
provider's cache; the shared dependency artifacts remain. Failed targets stop
the script before verification or cleanup.

The n8n Docker cases build each upstream Dockerfile with the build arguments
that its target in `docker/docker-bake.hcl` resolves at the checked-out commit:
`n8n`, `runners` and `runners-distroless`. Upstream keeps its base image pins
only in that file, and the Dockerfiles have no defaults for them. Preparation
adds the resolved arguments to the run's plan command, so every lane builds
the same images. It stops if the target builds a different Dockerfile or the
plan already sets one of its arguments.

## Adding

The Euro-Office Docker cases start at DocumentServer
Expand All @@ -141,6 +149,10 @@ includes the seed and thirty later commits, with four distinct core revisions.
resolved by its Bake target, then checks `x2t` and `allfontsgen`. Its plus plan
adds ccache launchers and a ccache mount to a generated Dockerfile; both plus
providers use that file and preserve the existing build and vcpkg mounts.
Core revisions that declare `CORE_CCACHE` compile through ccache themselves.
For those, the generated Dockerfile copies the pinned ccache binaries and sets
`CORE_CCACHE=1`. It keeps upstream's `core-ccache` mount only for lanes that
archive local compiler caches.
All lanes retain the dependency downloads and builds defined at each revision.
The history crosses upstream's migration to anonymous S3 dependency downloads.
`euro-office` builds the upstream `standalone` Bake target, including packaging,
Expand Down
3 changes: 0 additions & 3 deletions tools/docker/n8n-runners-distroless/.boringcache.toml
Original file line number Diff line number Diff line change
Expand Up @@ -8,9 +8,6 @@ metadata-hints = ["benchmark=n8n-runners-distroless", "upstream-job=build-and-pu
command = [
"docker", "buildx", "build",
"--file", "upstream/docker/images/runners/Dockerfile.distroless",
"--build-arg", "NODE_VERSION=26.7.0",
"--build-arg", "N8N_VERSION=snapshot",
"--build-arg", "N8N_RELEASE_TYPE=dev",
"--provenance", "false",
"--sbom", "false",
"--tag", "n8n-runners-distroless-benchmark:local",
Expand Down
2 changes: 1 addition & 1 deletion tools/docker/n8n-runners-distroless/case.toml
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,7 @@ project = "n8n"
repo = "n8n-io/n8n"
branch = "master"
start_sha = "0976d925b91c0652f2fd774737d0cc3ae620f59b"
prepare = ["mise trust && mise install", "bash \"$BENCH_ROOT/tools/shared/stage-vscode-ripgrep.sh\" upstream", "cd upstream && CI=1 pnpm install --frozen-lockfile && CI=1 N8N_SKIP_LICENSES=true pnpm build:n8n"]
prepare = ["mise trust && mise install", "bash \"$BENCH_ROOT/tools/shared/stage-vscode-ripgrep.sh\" upstream", "cd upstream && CI=1 pnpm install --frozen-lockfile && CI=1 N8N_SKIP_LICENSES=true pnpm build:n8n", "ruby \"$BENCH_ROOT/tools/docker/shared/bake-build-args.rb\" upstream/docker/docker-bake.hcl runners-distroless"]
check = "docker image inspect n8n-runners-distroless-benchmark:local"

[runs]
Expand Down
3 changes: 0 additions & 3 deletions tools/docker/n8n-runners/.boringcache.toml
Original file line number Diff line number Diff line change
Expand Up @@ -8,9 +8,6 @@ metadata-hints = ["benchmark=n8n-runners", "upstream-job=build-and-push-docker-i
command = [
"docker", "buildx", "build",
"--file", "upstream/docker/images/runners/Dockerfile",
"--build-arg", "NODE_VERSION=26.7.0",
"--build-arg", "N8N_VERSION=snapshot",
"--build-arg", "N8N_RELEASE_TYPE=dev",
"--provenance", "false",
"--sbom", "false",
"--tag", "n8n-runners-benchmark:local",
Expand Down
2 changes: 1 addition & 1 deletion tools/docker/n8n-runners/case.toml
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,7 @@ project = "n8n"
repo = "n8n-io/n8n"
branch = "master"
start_sha = "0976d925b91c0652f2fd774737d0cc3ae620f59b"
prepare = ["mise trust && mise install", "bash \"$BENCH_ROOT/tools/shared/stage-vscode-ripgrep.sh\" upstream", "cd upstream && CI=1 pnpm install --frozen-lockfile && CI=1 N8N_SKIP_LICENSES=true pnpm build:n8n"]
prepare = ["mise trust && mise install", "bash \"$BENCH_ROOT/tools/shared/stage-vscode-ripgrep.sh\" upstream", "cd upstream && CI=1 pnpm install --frozen-lockfile && CI=1 N8N_SKIP_LICENSES=true pnpm build:n8n", "ruby \"$BENCH_ROOT/tools/docker/shared/bake-build-args.rb\" upstream/docker/docker-bake.hcl runners"]
check = "docker image inspect n8n-runners-benchmark:local"

[runs]
Expand Down
3 changes: 0 additions & 3 deletions tools/docker/n8n/.boringcache.toml
Original file line number Diff line number Diff line change
Expand Up @@ -8,9 +8,6 @@ metadata-hints = ["benchmark=n8n-docker", "upstream-job=build-and-push-docker-im
command = [
"docker", "buildx", "build",
"--file", "upstream/docker/images/n8n/Dockerfile",
"--build-arg", "NODE_VERSION=26.7.0",
"--build-arg", "N8N_VERSION=snapshot",
"--build-arg", "N8N_RELEASE_TYPE=dev",
"--provenance", "false",
"--sbom", "false",
"--tag", "n8n-benchmark:local",
Expand Down
2 changes: 1 addition & 1 deletion tools/docker/n8n/case.toml
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
repo = "n8n-io/n8n"
branch = "master"
start_sha = "0976d925b91c0652f2fd774737d0cc3ae620f59b"
prepare = ["mise trust && mise install", "bash \"$BENCH_ROOT/tools/shared/stage-vscode-ripgrep.sh\" upstream", "cd upstream && CI=1 pnpm install --frozen-lockfile && CI=1 N8N_SKIP_LICENSES=true pnpm build:n8n"]
prepare = ["mise trust && mise install", "bash \"$BENCH_ROOT/tools/shared/stage-vscode-ripgrep.sh\" upstream", "cd upstream && CI=1 pnpm install --frozen-lockfile && CI=1 N8N_SKIP_LICENSES=true pnpm build:n8n", "ruby \"$BENCH_ROOT/tools/docker/shared/bake-build-args.rb\" upstream/docker/docker-bake.hcl n8n"]
check = "docker image inspect n8n-benchmark:local"

[runs]
Expand Down
Loading
Loading