Skip to content
Closed
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
17 changes: 17 additions & 0 deletions docs/results.md
Original file line number Diff line number Diff line change
Expand Up @@ -23,6 +23,23 @@ Experimental lanes are excluded from the report. A matched step must use the sam

New Actions records set `cache_restore_tracked` when the harness ran an Actions cache restore. A missing key after that scope's first step is a seed failure even if the scope has never restored successfully. Older records lack that distinction; the report identifies their restoring scopes from successful restored keys. Providers without a tracked restored key cannot be classified by this field.

## Product unit evidence

Wrapped BoringCache lanes request the CLI's existing native-evidence JSON export.
The result retains its `unit_evidence.v1` summary under
`provider_reported.native_evidence`; the complete product output, including any
`unit_records.v1` carrier, is listed in `evidence`. The harness reads the common
product format and contains no build-tool log parser.

Writing this requested export is included in each BoringCache lane's timed
command. Other providers keep their existing native evidence commands.

Missing, unreadable and older exports stay explicit and never become zero work.
Script and separate restore/save lanes do not request this local export. Other
providers keep their native commands. These files contain only our benchmark
jobs' evidence. Source-specific collection options require a CLI version that
supports them before they are added to a rolling plan.

## Runner samples

Weekly and manual samples appear as individual runs in `samples` and the Runner samples table, apart from rolling medians. Rows retain source commit, cache source, observed CPU and cores, timed operations, exit status and output check. Missing post-job timing is unmeasured, not zero. The selected GitHub source run is metadata for a mutable cache tag, not an immutable version identifier.
Expand Down
30 changes: 27 additions & 3 deletions lib/bench/phase_run.rb
Original file line number Diff line number Diff line change
Expand Up @@ -54,6 +54,7 @@ def start
end

def build
FileUtils.rm_f(native_evidence_path) if native_export?
return $?&.exitstatus || 127 unless seed && system(environment, *command, chdir: build_dir)

lane.finish(phase).all? { shell(it, chdir: build_dir) } ? 0 : 1
Expand Down Expand Up @@ -183,7 +184,7 @@ def seed
def command
return [*lane.wrap(phase, environment), *kase.script] if kase.script
return [*lane.wrap(phase, environment), *lane_program, *lane.args(phase, environment)] unless lane.boringcache?
return ["boringcache", kase.tool.name, *lane.cli_args, *read_only] unless lane.phases?
return ["boringcache", kase.tool.name, *lane.cli_args, "--native-tool-evidence-json", native_evidence_path, *read_only] unless lane.phases?

restore, save = %w[restore save].map { Shellwords.join(["boringcache", kase.tool.name, *lane.cli_args, "--phase", it]) }
["bash", "-c", [restore, Shellwords.join(adapter.fetch("command")), (save unless read_only)].compact.join(" && ")]
Expand Down Expand Up @@ -235,6 +236,29 @@ def evidence_path
record_path.sub(/\.json\z/, ".boringcache.jsonl")
end

def native_export?
lane.boringcache? && !lane.phases? && !kase.script
end

def native_evidence_path
File.expand_path(record_path.sub(/\.json\z/, ".native-tool.json"))
end

def native_evidence
return {} unless native_export?
return { "state" => "missing" } unless File.file?(native_evidence_path)
return { "state" => "unavailable", "reason" => "file_limit" } if File.size(native_evidence_path) > 64 * 1024 * 1024

native = JSON.parse(File.read(native_evidence_path))
block = native.is_a?(Hash) ? native["unit_evidence"] : nil
return { "state" => "not_reported", "reason" => "unit_evidence_not_exported" } unless block.is_a?(Hash) && block["schema_version"] == "unit_evidence.v1"

{ "state" => block["collection_state"], "unit_evidence" => block,
"build_actions" => native.dig("build_actions", "unit_evidence"), "file" => File.basename(native_evidence_path) }.compact
rescue JSON::ParserError, SystemCallError, IOError
{ "state" => "unavailable", "reason" => "unreadable_export" }
end

def write_record(started_at:, seconds:, exit_status:)
record = {
"adapter_command" => kase.tool.name,
Expand Down Expand Up @@ -264,7 +288,7 @@ def write_record(started_at:, seconds:, exit_status:)
"output_ok" => output_ok(exit_status),
"provider_reported" => provider_reported,
"observed" => observed,
"evidence" => File.exist?(evidence_path) ? [File.basename(evidence_path)] : [],
"evidence" => [evidence_path, *(native_export? ? [native_evidence_path] : [])].select { File.file?(it) }.map { File.basename(it) },
"versions" => { "boringcache" => boringcache_version, "boringcache_release" => (@env["BENCH_CLI_RELEASE"] if boringcache_version) }.compact,
**({ "product_env" => product_env } if product_env),
**({ "plan_settings" => plan_settings } if plan_settings),
Expand All @@ -289,7 +313,7 @@ def plan_settings
end

def provider_reported
{ "cache_session_summary" => session_summary }.compact
{ "cache_session_summary" => session_summary, "native_evidence" => (native_evidence if native_export?) }.compact
end

def observed
Expand Down
32 changes: 32 additions & 0 deletions measurements/llvm-native-2026-11-16/README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,32 @@
# LLVM native build controls, 16 November 2026

This experiment measures a native Clang build with a local ccache. It is separate
from rolling provider results. The source commit, machine, flags, timed commands,
output identity and exclusions are in `result.json`. Each seed and compiler-warm
arm ran once. The two unchanged-build reads use the seeded build directory.

The host uses Clang 18 and lld 18 on ARM64, targeting `clang`, with precompiled
headers disabled. The rolling case uses different host tools, architecture and
targets. These numbers do not compare cache providers or represent that case.

The seed took 779.788 seconds with 9 existing cache hits and 3,121 misses; it
was not an empty-cache cold run. A newly configured build directory using its local
compiler cache took 19.722 seconds, plus 17.511 seconds for configuration. ccache
reported 2,227 direct hits, 903 preprocessed hits, zero cacheable misses and one
unsupported code directive. The resulting executable matched the seed SHA256.
Output verification is outside the timed build.

To repeat, check out the recorded upstream commit, install the recorded tools,
set the three cache environment variables and substitute local absolute paths in
the CMake command. Start with an empty build directory. The recorded seed reused a cache from the
preceding cohort; an empty cache gives a separate cold measurement. Run CMake and
Ninja, record the executable hash, then run Ninja twice unchanged. Move the build
directory aside, preserving the cache and source tree. Run CMake at the original
build path, zero ccache statistics, run the same Ninja command, and compare the
executable hash. Read statistics with `ccache --print-stats --format=json`.
The raw warm Ninja log contains one invocation; its edge durations overlap.

A preceding cohort with default precompiled headers had almost no cacheable
compilations. It is excluded from this compiler-warm comparison. Local internal
build-state diagnostics and infrastructure records are kept outside this
benchmark measurement.
55 changes: 55 additions & 0 deletions measurements/llvm-native-2026-11-16/ccache-seed.json
Original file line number Diff line number Diff line change
@@ -0,0 +1,55 @@
{
"autoconf_test": 0,
"bad_compiler_arguments": 0,
"bad_input_file": 0,
"bad_output_file": 0,
"cache_miss": 3121,
"cache_size_kibibyte": 156336,
"called_for_link": 0,
"called_for_preprocessing": 0,
"cleanups_performed": 0,
"compile_failed": 0,
"compiler_check_failed": 0,
"compiler_produced_empty_output": 0,
"compiler_produced_no_output": 0,
"compiler_produced_stdout": 0,
"could_not_find_compiler": 0,
"could_not_use_modules": 0,
"could_not_use_precompiled_header": 0,
"direct_cache_hit": 9,
"direct_cache_miss": 3122,
"disabled": 0,
"error_hashing_extra_file": 0,
"files_in_cache": 5495,
"internal_error": 0,
"local_storage_hit": 9,
"local_storage_miss": 3121,
"local_storage_read_hit": 18,
"local_storage_read_miss": 6236,
"local_storage_write": 5339,
"max_cache_size_kibibyte": 19531250,
"max_files_in_cache": 0,
"missing_cache_file": 0,
"modified_input_file": 0,
"multiple_source_files": 0,
"no_input_file": 0,
"output_to_stdout": 0,
"preprocessed_cache_hit": 0,
"preprocessed_cache_miss": 3121,
"preprocessor_error": 0,
"recache": 0,
"remote_storage_error": 0,
"remote_storage_hit": 0,
"remote_storage_miss": 0,
"remote_storage_read_hit": 0,
"remote_storage_read_miss": 0,
"remote_storage_timeout": 0,
"remote_storage_write": 0,
"stats_updated_timestamp": 1794798276,
"stats_zeroed_timestamp": 1794797497,
"unsupported_code_directive": 1,
"unsupported_compiler_option": 0,
"unsupported_environment_variable": 0,
"unsupported_source_encoding": 0,
"unsupported_source_language": 0
}
12 changes: 12 additions & 0 deletions measurements/llvm-native-2026-11-16/ccache-version.txt
Original file line number Diff line number Diff line change
@@ -0,0 +1,12 @@
ccache version 4.14.3
Features: crsh-storage file-storage http-storage redis+unix-storage redis-storage remote-storage

Copyright (C) 2002-2007 Andrew Tridgell
Copyright (C) 2009-2026 Joel Rosdahl and other contributors

See <https://ccache.dev/credits.html> for a complete list of contributors.

This program is free software; you can redistribute it and/or modify it under
the terms of the GNU General Public License as published by the Free Software
Foundation; either version 3 of the License, or (at your option) any later
version.
55 changes: 55 additions & 0 deletions measurements/llvm-native-2026-11-16/ccache-warm.json
Original file line number Diff line number Diff line change
@@ -0,0 +1,55 @@
{
"autoconf_test": 0,
"bad_compiler_arguments": 0,
"bad_input_file": 0,
"bad_output_file": 0,
"cache_miss": 0,
"cache_size_kibibyte": 156336,
"called_for_link": 0,
"called_for_preprocessing": 0,
"cleanups_performed": 0,
"compile_failed": 0,
"compiler_check_failed": 0,
"compiler_produced_empty_output": 0,
"compiler_produced_no_output": 0,
"compiler_produced_stdout": 0,
"could_not_find_compiler": 0,
"could_not_use_modules": 0,
"could_not_use_precompiled_header": 0,
"direct_cache_hit": 2227,
"direct_cache_miss": 904,
"disabled": 0,
"error_hashing_extra_file": 0,
"files_in_cache": 5495,
"internal_error": 0,
"local_storage_hit": 3130,
"local_storage_miss": 0,
"local_storage_read_hit": 5357,
"local_storage_read_miss": 897,
"local_storage_write": 0,
"max_cache_size_kibibyte": 19531250,
"max_files_in_cache": 0,
"missing_cache_file": 0,
"modified_input_file": 0,
"multiple_source_files": 0,
"no_input_file": 0,
"output_to_stdout": 0,
"preprocessed_cache_hit": 903,
"preprocessed_cache_miss": 0,
"preprocessor_error": 0,
"recache": 0,
"remote_storage_error": 0,
"remote_storage_hit": 0,
"remote_storage_miss": 0,
"remote_storage_read_hit": 0,
"remote_storage_read_miss": 0,
"remote_storage_timeout": 0,
"remote_storage_write": 0,
"stats_updated_timestamp": 1794798317,
"stats_zeroed_timestamp": 1794798298,
"unsupported_code_directive": 1,
"unsupported_compiler_option": 0,
"unsupported_environment_variable": 0,
"unsupported_source_encoding": 0,
"unsupported_source_language": 0
}
10 changes: 10 additions & 0 deletions measurements/llvm-native-2026-11-16/checksums.sha256
Original file line number Diff line number Diff line change
@@ -0,0 +1,10 @@
18118cd88800da0177497e3c1173b3381e9b39af058c0ba586fc2b4228e9a991 README.md
160c9a661816aa989ff5d71d0c0d06ec4dd9a577a2f7d74d7047fc85293bb192 ccache-seed.json
deb1f7dd10f232c95b78607177019d523bdeef15a8172b2629743ae5c5b9f3d4 ccache-version.txt
6c329a8a7e96e7a80f9145578537f6a4da62556b890f948f695e0bb504e0b090 ccache-warm.json
d5c9ff6b83f2bce4db34258b8ec7262de63337cd67f348391549086ce2ef3052 configure.stdout
54aa0b540490e74b700615d86c8a66ed7a70303888a6cbdd76d7c3e2ae895324 ninja.json
0c407f2609741d6126c2a57aa2dda2c1fbea0cab930c3ca657afce55b17e3a1c result.json
52cee13fac65f864bf8c8b88e2bb980118f528e5355aed0e770dd7a79df1f908 tool-versions.txt
90980127b671199bcf83c29cb2097a0054d9687c3a8093d599ecd05f054a210e warm.ninja.log.gz
86b3c5451a70f77215c9f020dd44d06bf9ee0a468e3e4f7bd9a3fcf9a9c24721 warm.stdout
Loading
Loading