From 03b1e082895cc66a5220788ce7530639a5e69068 Mon Sep 17 00:00:00 2001 From: Ulf Adams Date: Sat, 1 Aug 2026 16:45:37 +0000 Subject: [PATCH 01/59] ladybird: reconcile with upstream's rules/experiments docs, and fix an inert version pin Rebasing onto main brought in docs/BAZEL-RULES.md and EXPERIMENTS.md, which the Ladybird ring contradicts in three places. Reconciled each against the actual build rather than by editing prose: - BAZEL-RULES said of the two Qt rulesets "neither has been used here". kklochkov/rules_qt 2.0.1 now IS used, so that prediction is recorded as a result (it held: qmake -query found the same 6.10.2 SDK find_package does), plus the two things only the use showed -- qt_cc_moc owning the include paths is what makes moc output byte-identical with zero -I flags, and the ruleset's REFUSAL to offer moc -b is what forced the real header fix. - The exercised-versions table omitted Ladybird entirely; added it, plus rows for rules_shell and rules_qt, and EXPERIMENTS.md gets a Ladybird entry. - The rules_foreign_cc section's boundary ("not for the code under migration") is walked at scale by the vcpkg/cargo rings, so it now states WHY they are plain actions over separately-pinned fetched inputs, and why a repository_rule was the tempting wrong answer. The sharper form of the rule is: which properties must remain checkable? For a third-party leaf that is hermeticity and pinning, not compile parity. Applying that doc's own "versions are resolved, not remembered" rule to Ladybird found a real defect: MODULE.bazel declared rules_cc 0.2.17 while MVS resolved 0.2.19 via bazel_tools on Bazel 9.2.0, so the declared version was inert. Corrected to what resolves and added common --check_direct_dependencies=error so the two cannot drift again; all 6 binaries rebuild green after. Suite 126/126. --- EXPERIMENTS.md | 22 +++++++ docs/BAZEL-RULES.md | 80 ++++++++++++++++++++++-- docs/CASE-ladybird-migration.md | 7 ++- examples/ladybird/workspace/MODULE.bazel | 2 +- examples/ladybird/workspace/bazelrc.txt | 7 +++ 5 files changed, 110 insertions(+), 8 deletions(-) diff --git a/EXPERIMENTS.md b/EXPERIMENTS.md index 20dbcf3..6baa4bf 100644 --- a/EXPERIMENTS.md +++ b/EXPERIMENTS.md @@ -5,6 +5,28 @@ were written by hand instead: [docs/BAZEL-RULES.md](docs/BAZEL-RULES.md). ## CMake Experiments +### Ladybird (cmake + vcpkg + cargo -> Bazel) + +- Source: https://github.com/LadybirdBrowser/ladybird +- Generated BUILD files; the workspace overlay is in + [`examples/ladybird/`](examples/ladybird/README.md), the 34 findings in + [`docs/CASE-ladybird-migration.md`](docs/CASE-ladybird-migration.md) +- `git clone && bazel build //:ladybird` builds the browser: **Bazel owns the + whole dependency closure**, with no CMake build and no network in any action — + the 77 vcpkg ports (fetched from a captured asset pin) and the 10 Rust crates + + `flapc` (154 crates.io archives resolved from `Cargo.lock`) are Bazel targets +- All 51 code generators run under Bazel with output byte-identical to CMake's + (1,408/1,408 files); the 6 binaries render `--headless=text` and + `--headless=layout-tree` byte-identically to the CMake reference +- C++23, and the largest subject here: 34 libraries, ~3.7k TUs, LibWeb alone + ~1,961 compile inputs — which is what surfaced the extractor's depSet OOM +- Generated custom Bazel rules, all because the recipe was worth keeping and the + ecosystem ruleset would have replaced it: `vcpkg_tree`/`vcpkg_lib` (vcpkg as an + ordinary action under `x-block-origin`, not `rules_foreign_cc`), and + `rust_sysroot`/`cargo_crate`/`cargo_lib`/`cargo_binary` (offline cargo, not + `rules_rust`/`crate_universe`) +- Qt via `kklochkov/rules_qt` — the one place a ruleset *was* adopted + ### BoringSSL (cmake <-> Bazel) - Source: https://github.com/google/boringssl diff --git a/docs/BAZEL-RULES.md b/docs/BAZEL-RULES.md index ff613df..89cbc0b 100644 --- a/docs/BAZEL-RULES.md +++ b/docs/BAZEL-RULES.md @@ -59,15 +59,30 @@ Ranges, not recommendations. "Declared" is what a `MODULE.bazel` asked for; | ruleset | declared range | resolved | migrations | how far it was exercised | |---|---|---|---|---| -| `rules_cc` | 0.0.16 – 0.2.22 | 0.2.14 – 0.2.22 | Dolphin; fmt, spdlog, TinyXML2, zlib; BoringSSL, Abseil, RE2 | Dolphin: `//...` green in 3 configurations, 1350 tests pass, action-graph parity 0 errors. Build re-verified green at declared 0.0.16 / 0.1.1 / 0.2.22; `compatibility_level = 1` and both `//cc:defs.bzl` and the per-rule `.bzl` files exist across that whole span | +| `rules_cc` | 0.0.16 – 0.2.22 | 0.2.14 – 0.2.22 | Dolphin; **Ladybird**; fmt, spdlog, TinyXML2, zlib; BoringSSL, Abseil, RE2 | Dolphin: `//...` green in 3 configurations, 1350 tests pass, action-graph parity 0 errors. Build re-verified green at declared 0.0.16 / 0.1.1 / 0.2.22; `compatibility_level = 1` and both `//cc:defs.bzl` and the per-rule `.bzl` files exist across that whole span. Ladybird: 34 libraries + 6 executables, ~3.7k C++23 TUs, renders pages byte-identically to the CMake reference | | `platforms` | 0.0.10 – 1.1.0 | 1.0.0 – 1.1.0 | same | as above; only ever a transitive/constraint dep | +| `rules_shell` | 0.6.1 | 0.6.1 | Ladybird | one point. Only because Bazel 9 removed the native `sh_binary`, which the vcpkg/cargo build wrappers are | +| `rules_qt` (**kklochkov**, not the BCR module) | 2.0.1 | 2.0.1 (`archive_override`) | Ladybird | `qt.local_repo` (qmake-discovered host Qt 6.10.2) + `qt_cc_moc` over 11 `Q_OBJECT` headers + `qt_qrc`/`qt_cc_rcc`; all 11 moc bodies byte-identical to CMake's, GUI runs. See [the section below](#dolphin-qt-handled-by-hand-and-a-ruleset-that-was-missed) | | `rules_jvm_external` | 6.7 | — | Guava (Maven frontend) | one point, coordinate deps only; the Maven frontend is argv-floor, so this is not a parity claim | | `rules_license`, `googletest`, `google_benchmark`, `rules_python` | see note | — | BoringSSL, Abseil, RE2 | **not our choices** — these are what those projects' *own* Bazel builds declare (`rules_license` 1.0.0, `googletest` 1.17.0.bcr.2, `google_benchmark` 1.9.4/1.9.5, `rules_python` 1.7.0). We diffed against them; we did not select them | Bazel itself: **7.5.0** (VSCode, pinned in `.bazelversion`) and **9.2.0** -(Dolphin). No migration here has needed a Bazel-version-specific workaround. +(Dolphin, Ladybird). One Bazel-version-specific consequence, not a workaround: +on 9.2.0 the native `sh_binary` is gone, so Ladybird has to declare +`rules_shell` for wrappers that needed no dep on 7.x. Every "resolved" column above is a fact about 9.2.0, not about the declared file. +Ladybird declares **4** modules and resolves **27** — the same transitive blow-up +as Dolphin (`rules_cc → protobuf → …` brings in `rules_kotlin`, `rules_android`, +`rules_swift`, `rules_jvm_external`), and worth restating because Ladybird's Qt +and Rust rings deliberately avoid rulesets: the transitive cost arrives through +`rules_cc` regardless of how little else you adopt. + +Ladybird has `common --check_direct_dependencies=error` in its `.bazelrc`, per the +rule above — and turning it on immediately failed: it declared `rules_cc 0.2.17` +while 0.2.19 resolved. The declared version was inert, exactly as this section +predicts. Corrected to what resolves, with the build re-verified green after. + ## Written by hand instead of adopting a ruleset The interesting cases. Both are C++/TS migrations where an obvious ecosystem @@ -160,8 +175,31 @@ is in the ruleset that is **not** in the BCR under `rules_qt`. Querying the registry for `rules_qt` returns the prebuilt-single-version one, which would have pinned Qt 6.8.3 and turned the migration into a different build. -Neither has been used here, so this is a pointer for whoever hits Qt next, not a -recommendation. The transferable parts: +**Since written, `kklochkov/rules_qt` 2.0.1 has been used** — Ladybird's Qt UI +runs on it (`qt.local_repo` + `qt_cc_moc`/`qt_qrc`), so the paragraph above is +now a result rather than a prediction, and it held: `qmake -query` found the same +6.10.2 SDK `find_package` does. Two things only the use showed, both in +[CASE-ladybird-migration](CASE-ladybird-migration.md) §Qt (findings 19–21): + +* **The moc rule must own the include paths, and this one does.** moc emits + metatype includes only for types whose definition it has *seen*, so a + hand-rolled genrule needs a `moc_input_headers` filegroup plus matching `-I` + flags kept in sync by hand — and getting it wrong yields *wrong output that + still compiles* (four outputs silently lost an include), caught only by + byte-diffing against CMake. `qt_cc_moc` reads the dirs off the Qt toolchain's + `CcInfo` compilation context and stages the headers itself: byte-identical moc + bodies with **zero** include flags in the BUILD file. This is the concrete form + of "the rule knows what moc needs better than its callers do", and it is the + strongest argument for the ruleset over a genrule. +* **A good ruleset withholds the knob you would have misused.** Splitting CMake's + unity `mocs_compilation.cpp` into per-header TUs exposed a latent Ladybird bug + (an inline `dynamic_cast` on a forward-declared type, compiling only because of + include *order* inside the unity file). moc's `-b` flag would have papered over + it in the build system; `qt_cc_moc` offers no such knob, which forced the + one-line upstreamable header fix instead. Unity builds hide incomplete-type + bugs, so a migration off one should expect to find them. + +The transferable parts of the comparison: * **BCR presence is not fitness, and BCR absence is not nonexistence.** The registry is a distribution channel, not a curated index. Search wider, and @@ -186,6 +224,32 @@ prebuilt artifact you never claimed parity on is a different thing — VSCode's `nodegyp_module` shells out to `node-gyp rebuild` for native `.node` modules, and nothing is lost, because those binaries were never part of the parity claim. +Ladybird walks that line at scale, and how it does so is the reusable part. +Its 77 vcpkg ports and 10 Rust crates are third-party leaves, so running their +native builders (`vcpkg install`, `cargo build`) costs nothing a parity claim +needed. But rather than reach for `rules_foreign_cc`, both are **ordinary Bazel +actions** — `vcpkg_tree` and `cargo_crate` — and the reason is what the wrapper +would have taken away: + +* **Fetching is the part Bazel must own; building is the part it need not.** The + dependency's *identity* (URL + hash) is exactly what a build has to pin, and + what `rules_foreign_cc` leaves to the foreign tool's own downloader. So the + fetch is hoisted out into `http_file`/`http_archive` repos generated from a + captured pin, and the recipe runs offline against them — `vcpkg` under + `x-block-origin`, `cargo` under a network-blocked action with a vendored + registry. The invariant is verifiable and was verified by removal: **zero + network access in any action**. +* **A `repository_rule` was the tempting wrong answer.** `vcpkg_tree` is + deliberately a rule, not a repo rule: repository fetches escape the action + graph, the sandbox and remote execution, so a 45-minute dependency build would + have been invisible to `aquery` and uncacheable in the normal way. As a rule its + inputs and outputs are declared like anything else. + +Which sharpens the section title: the question is not "is this code I am +migrating?" but "**which properties do I need to remain checkable?**" For +Ladybird's deps that is hermeticity and pinning, not compile parity — and +`rules_foreign_cc` happens to give up precisely those. + ## Checklist for the Bazel side of a migration 1. Resolve every version at migration time; do not write one from memory. @@ -197,6 +261,10 @@ nothing is lost, because those binaries were never part of the parity claim. 5. Decide what equivalence you need — byte-identical or content-equivalent — before choosing. Ecosystem rules target the latter. 6. Weigh the transitive graph (`bazel mod graph`), not just the direct dep. -7. Keep `rules_foreign_cc` away from the code under migration. +7. Keep `rules_foreign_cc` away from the code under migration. For third-party + leaves, run the foreign builder as a plain Bazel *action* (not a + `repository_rule`) over separately-pinned fetched inputs, so hermeticity stays + checkable even where compile parity is not claimed. 8. When the migration converges, run `bazel mod graph` and record the **resolved** - versions. That is the only version list worth keeping. + versions, and put `common --check_direct_dependencies=error` in `.bazelrc` so + the two cannot drift apart again. That is the only version list worth keeping. diff --git a/docs/CASE-ladybird-migration.md b/docs/CASE-ladybird-migration.md index e050520..d202c97 100644 --- a/docs/CASE-ladybird-migration.md +++ b/docs/CASE-ladybird-migration.md @@ -47,7 +47,12 @@ AK is the foundation library (38 sources, external deps fmt/simdutf/mimalloc/ cpptrace, two configure-generated headers). Standing it up end-to-end proved the whole loop works on real Ladybird and surfaced the reusable scaffolding: -- **`MODULE.bazel`**: bzlmod, `rules_cc` 0.2.17, `platforms` 1.0.0. +- **`MODULE.bazel`**: bzlmod, `rules_cc` 0.2.19, `platforms` 1.0.0. (Declared + 0.2.17 for most of this migration, which was inert: MVS resolved 0.2.19 via + `bazel_tools` on Bazel 9.2.0. Caught only on adding + `common --check_direct_dependencies=error`, which is now in the `.bazelrc` so + the declared and resolved versions cannot drift apart again — see + [BAZEL-RULES](BAZEL-RULES.md).) - **vcpkg shim**: a `BUILD.bazel` dropped into `Build/full/vcpkg_installed/x64-linux-dynamic/` — one `cc_library(:headers)` over the whole `include/` tree + `cc_import` per prebuilt `.so`. Consumes diff --git a/examples/ladybird/workspace/MODULE.bazel b/examples/ladybird/workspace/MODULE.bazel index 11b3f27..4f59612 100644 --- a/examples/ladybird/workspace/MODULE.bazel +++ b/examples/ladybird/workspace/MODULE.bazel @@ -1,6 +1,6 @@ module(name = "ladybird", version = "0.0.0") -bazel_dep(name = "rules_cc", version = "0.2.17") +bazel_dep(name = "rules_cc", version = "0.2.19") bazel_dep(name = "platforms", version = "1.0.0") # Bazel 9 removed the native sh_binary; //Meta:vcpkg_build is one. diff --git a/examples/ladybird/workspace/bazelrc.txt b/examples/ladybird/workspace/bazelrc.txt index b9792ee..141e08e 100644 --- a/examples/ladybird/workspace/bazelrc.txt +++ b/examples/ladybird/workspace/bazelrc.txt @@ -1,5 +1,12 @@ common --enable_bzlmod +# A declared version that MVS does not actually resolve to is inert: the floor +# is usually bazel_tools' (built into Bazel), so .bazelversion is part of the +# dependency specification. Make the drift fail the build instead of warning -- +# it caught rules_cc 0.2.17 being declared while 0.2.19 resolved on Bazel 9.2.0. +# See docs/BAZEL-RULES.md, "Versions are resolved, not remembered". +common --check_direct_dependencies=error + # C++23 everywhere (Ladybird global). build --cxxopt=-std=c++23 --host_cxxopt=-std=c++23 From a5679127f84350710c445c39725cdf165978f7e4 Mon Sep 17 00:00:00 2001 From: Ulf Adams Date: Sun, 2 Aug 2026 11:48:17 +0000 Subject: [PATCH 02/59] Ladybird: make git clone && bazel build actually work (31 missing headers + Cranelift) The README's headline claim was false. Every binary depended on 741 targets under Build/full, CMake's build tree, and the overlay shipped 4 BUILD files and zero headers -- allow_empty=True in the shims meant a fresh clone got no error, just `fatal error: LibXML/Export.h: No such file or directory` some 1,600 actions in. Narrowed 741 to 31 true gaps (666 LibWeb bindings headers were merely SHADOWING Bazel's own; deleting them left the build green). 1. The 15 generate_export_header Export.h + AK's 2 configure_file headers. New emitter Meta/emit_export_headers_bazel.py derives the tokens as Meta/CMake/targets.cmake does; all 17 built artifacts are byte-identical to CMake's. AK/Backtrace.h is a host PROBE, not a template -- the genrule compiles find_package(Backtrace)'s test rather than baking my machine's answer. 2. Cranelift was absent from the Bazel graph ENTIRELY. Libraries/LibWasm/CMakeLists.txt was not in CMAKELISTS and build_rust_binary() was not parsed, so LibWasm's dep was silently dropped. The reference build hid it twice: the header sat in Build/full where a global -I reached it, and the compiler binary was named by an absolute path baked into -DWASM_CRANELIFT_COMPILER_PATH=/home/ubuntu/... -- two host escapes covering for one missing target. Now a cargo_binary that also declares its cbindgen header (byte-identical to CMake's), consumed as //:libwasm_cranelift_lib, with the binary as `data` so Ladybird's own sibling-of-self lookup finds it in any checkout. 3. A real include-collision bug, found by removing the shim that masked it. 8 of 10 crates emit a header named RustFFI.h and 4 TUs include it bare, so an unprefixed include dir must not propagate. Splitting it into cargo_bare_include was necessary but NOT sufficient: include dirs propagate along the C++ dep graph too, so LibGfx inherited LibTextCodec's dir and YUVData.cpp compiled against the wrong header. The fix is implementation_deps -- Bazel's name for exactly the scope CMake's PRIVATE FFI_OUTPUT_DIR has. Also: rust_bare_include_crates() is now IMPORTED from the cargo emitter that derives it instead of a hand-kept tuple beside it; the FFI-header sync is shared by both cargo drivers instead of duplicated; emit_libweb_bazel round-trips its export-header block instead of silently truncating it. Verified by removal, the only test that counts: Build/full/{Libraries,Services,UI,bin,cargo} moved off the machine (0 headers left), all six binaries rebuilt from scratch (2,704 actions), and --headless=text AND --headless=layout-tree are byte-identical to the CMake reference on three pages including two that execute WebAssembly. Parity harness 1408/1408, all emitters round-trip. --- examples/ladybird/workspace/BUILD.bazel | 55 ++- .../workspace/Libraries/LibWeb/BUILD.bazel | 9 + .../Libraries/LibWeb/export_header.bzl | 17 + .../workspace/Meta/cargo_binary_build.sh | 41 +- .../ladybird/workspace/Meta/cargo_build.sh | 59 +-- .../ladybird/workspace/Meta/cargo_vendor.sh | 64 +++ .../workspace/Meta/emit_build_bazel.py | 266 +++++++++-- .../workspace/Meta/emit_cargo_bazel.py | 334 ++++++++++++-- .../Meta/emit_export_headers_bazel.py | 421 ++++++++++++++++++ .../workspace/Meta/emit_libweb_bazel.py | 29 +- examples/ladybird/workspace/cargo.bzl | 213 ++++++++- examples/ladybird/workspace/cargo_index.bzl | 10 + examples/ladybird/workspace/cargo_ring.bzl | 104 ++++- .../ladybird/workspace/export_headers.bzl | 106 +++++ 14 files changed, 1540 insertions(+), 188 deletions(-) create mode 100644 examples/ladybird/workspace/Libraries/LibWeb/export_header.bzl create mode 100644 examples/ladybird/workspace/Meta/emit_export_headers_bazel.py create mode 100644 examples/ladybird/workspace/export_headers.bzl diff --git a/examples/ladybird/workspace/BUILD.bazel b/examples/ladybird/workspace/BUILD.bazel index 8d1b5e2..d290a2f 100644 --- a/examples/ladybird/workspace/BUILD.bazel +++ b/examples/ladybird/workspace/BUILD.bazel @@ -2,6 +2,7 @@ load("@rules_qt//qt:defs.bzl", "qt_cc_moc", "qt_cc_rcc", "qt_qrc") load("@rules_cc//cc:defs.bzl", "cc_binary", "cc_library") load(":cargo_ring.bzl", "cargo_ring") load(":codegen_root.bzl", "root_codegen") +load(":export_headers.bzl", "export_headers") load(":vcpkg.bzl", "vcpkg_tree", "vcpkg_tree_for_exec") load(":vcpkg_index.bzl", "VCPKG_DISTFILE_INDEX") @@ -14,6 +15,11 @@ package(default_visibility = ["//visibility:public"]) # Build/full. root_codegen() +# CMake's generate_export_header() output (15 Export.h) and the two AK +# configure_file headers, generated by Bazel instead of read out of Build/full -- +# generated by Meta/emit_export_headers_bazel.py, byte-verified with --check. +export_headers() + # The Rust ring: 10 cargo staticlib crates + flapc, built by Bazel from crates # Bazel fetched, offline. This is what retired the prebuilt 260 MB # librust_combined.a and its hand-run `ar -M` merge (README step 1b). It lives in @@ -43,6 +49,12 @@ cc_library( ":generated_libraries_headers", ":generated_services_headers", ":generated_shader_headers", + ":generated_export_headers", + ":generated_ak_headers", + "//Libraries/LibWeb:generated_export_header", + # Build/full roots: now ONLY the Rust FFI headers and the IPC/codegen + # headers Bazel does not yet generate. Export.h and AK's two + # configure_file headers are Bazel-generated above. "//Build/full/Libraries:generated_lib_headers", "//Build/full/Services:generated_service_headers", ], @@ -51,15 +63,12 @@ cc_library( VCPKG = "//Meta/vcpkg" -# Configure-generated headers live in Build/full/AK; expose as AK/*.h via a copy. -genrule( - name = "ak_gen_headers", - srcs = ["Build/full/AK/Debug.h", "Build/full/AK/Backtrace.h"], - outs = ["genroot/AK/Debug.h", "genroot/AK/Backtrace.h"], - cmd = "mkdir -p $(RULEDIR)/genroot/AK && " + - "cp $(location Build/full/AK/Debug.h) $(RULEDIR)/genroot/AK/Debug.h && " + - "cp $(location Build/full/AK/Backtrace.h) $(RULEDIR)/genroot/AK/Backtrace.h", -) +# NOTE: the ak_gen_headers genrule that COPIED AK/Debug.h and AK/Backtrace.h out +# of Build/full is gone. Bazel now generates both from their checked-in .in +# templates (export_headers.bzl): Debug.h by applying CMake's configure_file +# substitution, Backtrace.h by ASKING the host the same question +# find_package(Backtrace) asks. That removed the last AK dependency on a CMake +# build tree. cc_library( name = "AK", @@ -103,7 +112,7 @@ cc_library( 'AK/Utf8View.cpp', 'AK/kmalloc.cpp', ], - hdrs = glob(["AK/*.h"]) + [":ak_gen_headers"], + hdrs = glob(["AK/*.h"]), # AK-private defines (CMake PRIVATE) — local_defines so they don't leak to # consumers. FMT_SHARED/AK_HAS_CPPTRACE only affect AK's own TUs. local_defines = [ @@ -111,8 +120,13 @@ cc_library( "AK_HAS_CPPTRACE=1", "FMT_SHARED", ], - includes = ["genroot"], deps = [ + # AK/Debug.h + AK/Backtrace.h, generated from their .in templates + # (export_headers.bzl). A cc_library belongs on a deps edge, not in hdrs: + # in hdrs it contributes no include path, which is why every consumer + # failed with "AK/Debug.h: No such file or directory" until this moved. + # The include root travels with the dep, so dependents get it too. + ":generated_ak_headers", VCPKG + ":fmt", VCPKG + ":simdutf", VCPKG + ":mimalloc", @@ -421,8 +435,8 @@ cc_library( ], hdrs = glob(["Libraries/LibGfx/**/*.h"], allow_empty = True), local_defines = ['LibGfx_EXPORTS', 'SK_CODEC_DECODES_BMP', 'SK_CODEC_DECODES_GIF', 'SK_CODEC_DECODES_WBMP', 'SK_DISABLE_TRACING', 'SK_ENABLE_AVX512_OPTS', 'SK_ENABLE_PRECOMPILE', 'SK_FONTMGR_ANDROID_AVAILABLE', 'SK_FONTMGR_FCI_AVAILABLE', 'SK_FONTMGR_FONTCONFIG_AVAILABLE', 'SK_FONTMGR_FREETYPE_DIRECTORY_AVAILABLE', 'SK_FONTMGR_FREETYPE_EMBEDDED_AVAILABLE', 'SK_FONTMGR_FREETYPE_EMPTY_AVAILABLE', 'SK_GAMMA_APPLY_TO_A8', 'SK_GANESH', 'SK_HAS_WUFFS_LIBRARY', 'SK_R32_SHIFT=16', 'SK_TYPEFACE_FACTORY_FREETYPE', 'SK_USE_PERFETTO', 'SK_VULKAN', 'SK_XML'], - copts = ['-IBuild/full/Libraries/LibGfx'], linkopts = ['-ldl', '-lpthread', '-lvulkan'], + implementation_deps = ['//:libgfx_rust_bare_include'], deps = [ '//:AK', '//:LibCompress', @@ -853,7 +867,7 @@ cc_library( hdrs = glob(["Libraries/LibJS/**/*.h"], allow_empty = True), local_defines = ['LibJS_EXPORTS', 'SIMDJSON_DEVELOPMENT_CHECKS', 'SIMDJSON_THREADS_ENABLED=1'], additional_compiler_inputs = ['Libraries/LibJS/Runtime/JavaScriptImplementations/AbstractOperations.js', 'Libraries/LibJS/Runtime/JavaScriptImplementations/ArrayConstructor.js'], - copts = ['-fno-omit-frame-pointer', '-fvisibility=hidden', '-IBuild/full/Libraries/LibJS'], + copts = ['-fno-omit-frame-pointer', '-fvisibility=hidden'], linkopts = ['-lvulkan'], deps = [ '//:AK', @@ -968,7 +982,8 @@ cc_library( ], hdrs = glob(["Libraries/LibRegex/**/*.h"], allow_empty = True), local_defines = ['LibRegex_EXPORTS'], - copts = ['-fvisibility=hidden', '-IBuild/full/Libraries/LibRegex'], + copts = ['-fvisibility=hidden'], + implementation_deps = ['//:libregex_rust_bare_include'], deps = [ '//:AK', '//:LibUnicode', @@ -1076,7 +1091,8 @@ cc_library( ], hdrs = glob(["Libraries/LibTextCodec/**/*.h"], allow_empty = True), local_defines = ['LibTextCodec_EXPORTS'], - copts = ['-fvisibility=hidden', '-IBuild/full/Libraries/LibTextCodec'], + copts = ['-fvisibility=hidden'], + implementation_deps = ['//:libtextcodec_rust_bare_include'], deps = [ '//:AK', '//:all_source_headers', @@ -1116,7 +1132,6 @@ cc_library( ], hdrs = glob(["Libraries/LibURL/**/*.h"], allow_empty = True), local_defines = ['ENABLE_RUST', 'LibURL_EXPORTS'], - copts = ['-IBuild/full/Libraries/LibURL'], deps = [ '//:AK', '//:LibRegex', @@ -1159,7 +1174,6 @@ cc_library( hdrs = glob(["Libraries/LibUnicode/**/*.h"], allow_empty = True), alwayslink = True, local_defines = ['LibUnicode_EXPORTS'], - copts = ['-IBuild/full/Libraries/LibUnicode'], linkopts = ['-ldl'], deps = [ '//:AK', @@ -1201,9 +1215,11 @@ cc_library( 'Libraries/LibWasm/WASI/Wasi.cpp', ], hdrs = glob(["Libraries/LibWasm/**/*.h"], allow_empty = True), - local_defines = ['LibWasm_EXPORTS', 'WASM_COMPILED_FAULT_RECOVERY_SUPPORTED=1', 'WASM_CRANELIFT=1', 'WASM_CRANELIFT_COMPILER_PATH=\\"/home/ubuntu/ladybird-work/Build/full/bin/cranelift-compiler\\"'], - copts = ['-fvisibility=hidden', '-IBuild/full/Libraries/LibWasm'], + local_defines = ['LibWasm_EXPORTS', 'WASM_COMPILED_FAULT_RECOVERY_SUPPORTED=1', 'WASM_CRANELIFT=1', 'WASM_CRANELIFT_COMPILER_PATH=\\"cranelift-compiler\\"'], + copts = ['-fvisibility=hidden'], linkopts = ['-lvulkan'], + data = ['//:cranelift-compiler'], + implementation_deps = ['//:libwasm_cranelift_bare_include'], deps = [ '//:AK', '//:LibCore', @@ -1212,6 +1228,7 @@ cc_library( '//:LibSync', '//:LibThreading', '//:all_source_headers', + '//:libwasm_cranelift_lib', ], ) diff --git a/examples/ladybird/workspace/Libraries/LibWeb/BUILD.bazel b/examples/ladybird/workspace/Libraries/LibWeb/BUILD.bazel index 5b3b6a7..975a150 100644 --- a/examples/ladybird/workspace/Libraries/LibWeb/BUILD.bazel +++ b/examples/ladybird/workspace/Libraries/LibWeb/BUILD.bazel @@ -1,9 +1,18 @@ load("@rules_cc//cc:defs.bzl", "cc_library") load(":codegen.bzl", "libweb_codegen", "libweb_bindings_codegen") load(":generated_srcs.bzl", "LIBWEB_GENERATED_SRCS", "LIBWEB_GENERATED_HDRS") +load(":export_header.bzl", "libweb_export_header") package(default_visibility = ["//visibility:public"]) +# LibWeb's generate_export_header output. It is generated HERE rather than in the +# root package because Bazel include dirs cannot escape a package: LibWeb is its +# own package, so its Export.h has to be an output of this package (at +# genroot/LibWeb/Export.h, with includes=["genroot"] -- includes=["../.."] is +# rejected by Bazel, and rightly so). Emitted by +# Meta/emit_export_headers_bazel.py --libweb. +libweb_export_header() + libweb_codegen() libweb_bindings_codegen() diff --git a/examples/ladybird/workspace/Libraries/LibWeb/export_header.bzl b/examples/ladybird/workspace/Libraries/LibWeb/export_header.bzl new file mode 100644 index 0000000..799e31c --- /dev/null +++ b/examples/ladybird/workspace/Libraries/LibWeb/export_header.bzl @@ -0,0 +1,17 @@ +# AUTO-GENERATED by Meta/emit_export_headers_bazel.py --libweb — do not edit. +# LibWeb's generate_export_header output. It lives here rather than in the +# root package because Libraries/LibWeb is its own Bazel package, so only +# this package may declare an output inside it. +load("@rules_cc//cc:defs.bzl", "cc_library") + +def libweb_export_header(): + native.genrule( + name = 'gen_LibWeb_Export_h', + outs = ['genroot/LibWeb/Export.h'], + cmd = "cat > $@ <<'LADYBIRD_EOF'\n\n#ifndef WEB_API_H\n#define WEB_API_H\n\n#ifdef LIBWEB_STATIC_DEFINE\n# define WEB_API\n# define LIBWEB_NO_EXPORT\n#else\n# ifndef WEB_API\n# ifdef LibWeb_EXPORTS\n /* We are building this library */\n# define WEB_API __attribute__((visibility(\"default\")))\n# else\n /* We are using this library */\n# define WEB_API __attribute__((visibility(\"default\")))\n# endif\n# endif\n\n# ifndef LIBWEB_NO_EXPORT\n# define LIBWEB_NO_EXPORT __attribute__((visibility(\"hidden\")))\n# endif\n#endif\n\n#ifndef LIBWEB_DEPRECATED\n# define LIBWEB_DEPRECATED __attribute__ ((__deprecated__))\n#endif\n\n#ifndef LIBWEB_DEPRECATED_EXPORT\n# define LIBWEB_DEPRECATED_EXPORT WEB_API LIBWEB_DEPRECATED\n#endif\n\n#ifndef LIBWEB_DEPRECATED_NO_EXPORT\n# define LIBWEB_DEPRECATED_NO_EXPORT LIBWEB_NO_EXPORT LIBWEB_DEPRECATED\n#endif\n\n/* NOLINTNEXTLINE(readability-avoid-unconditional-preprocessor-if) */\n#if 0 /* DEFINE_NO_DEPRECATED */\n# ifndef LIBWEB_NO_DEPRECATED\n# define LIBWEB_NO_DEPRECATED\n# endif\n#endif\n\n#endif /* WEB_API_H */\nLADYBIRD_EOF\n", + ) + cc_library( + name = 'generated_export_header', + hdrs = ['genroot/LibWeb/Export.h'], + includes = ['genroot'], + ) diff --git a/examples/ladybird/workspace/Meta/cargo_binary_build.sh b/examples/ladybird/workspace/Meta/cargo_binary_build.sh index 807db7e..7c710d6 100644 --- a/examples/ladybird/workspace/Meta/cargo_binary_build.sh +++ b/examples/ladybird/workspace/Meta/cargo_binary_build.sh @@ -1,15 +1,25 @@ #!/bin/bash -# Build a cargo BINARY crate offline: flapc, the Flap-DSL -> interpreter-assembly -# compiler. Driver for the cargo_binary build action (see cargo.bzl). +# Build a cargo BINARY crate offline. Driver for the cargo_binary build action +# (see cargo.bzl). There are two, and they are two different SHAPES of problem: # -# flapc was the last artifact this migration still took from the reference CMake -# build. Bazel already RAN it as a declared genrule tool, so the interpreter -# assembly was Bazel's own output -- but the compiler itself came out of -# Build/full/bin/, because Libraries/LibJS/Flap is a Rust crate. It gets exactly -# the same treatment as the staticlib crates, and needs no extra machinery: its -# workspace is `exclude`d from the root one and has its own lock with 3 packages, -# whose single registry crate is the same smallvec 1.15.1 (same checksum, checked -# by the emitter) that the big workspace already pins. +# * **flapc**, the Flap-DSL -> interpreter-assembly compiler: a pure build +# tool. It was the last artifact this migration still took from the reference +# CMake build -- Bazel already RAN it as a declared genrule tool, so the +# interpreter assembly was Bazel's own output, but the compiler itself came +# out of Build/full/bin/ because Libraries/LibJS/Flap is a Rust crate. Its +# workspace is `exclude`d from the root one and has its own lock with 3 +# packages, whose single registry crate is the same smallvec 1.15.1 (same +# checksum, checked by the emitter) the big workspace already pins. +# * **cranelift-compiler**, LibWasm's AOT WebAssembly compiler: a RUNTIME +# tool, spawned by the browser (Core::Process::spawn in CraneliftBridge.cpp), +# and one that ALSO emits an FFI header its C++ caller includes. It was +# absent from the Bazel graph entirely -- CMake declares it with +# build_rust_binary(), which the emitter did not parse, so every browser +# binary failed on `CraneliftFFI.h: No such file or directory`. +# +# The second one is why this driver takes FFI arguments at all: a binary crate's +# build script runs cbindgen exactly as a staticlib crate's does, and the header +# is a declared output for the same reason (Bazel deletes what nothing declares). # # Contract: # $1 crate name (the cargo package) @@ -19,6 +29,8 @@ # $5 the crate index: " " per line # $6 output binary path # $7 the --bin name +# $8 output dir for the FFI headers, or "" if the crate emits none +# $9+ the FFI header paths to check for, relative to $8 # # See cargo_vendor.sh for why the staging looks the way it does. set -euo pipefail @@ -30,6 +42,9 @@ SYSROOT="${4:?rust sysroot}" INDEX="${5:?crate index}" OUT_BIN="${6:?output binary}" BIN="${7:?bin name}" +FFI_OUT="${8-}" +shift $(( $# < 8 ? $# : 8 )) +FFI_HEADERS=("$@") # A binary output is a single file with no triple in its path (unlike the # staticlib, whose declared output mirrors cargo's own layout), so the triple @@ -58,3 +73,9 @@ cd "$SRC" cp "$TARGET_DIR/$TRIPLE/release/$BIN" "$OUT_BIN" chmod +x "$OUT_BIN" + +# The declared headers, resolved out of the OWNING crate's OUT_DIR -- the same +# collision-safe copy the staticlib driver does, shared in cargo_vendor.sh. +if [ -n "$FFI_OUT" ]; then + sync_ffi_headers "$FFI_OUT" "${FFI_HEADERS[@]+"${FFI_HEADERS[@]}"}" +fi diff --git a/examples/ladybird/workspace/Meta/cargo_build.sh b/examples/ladybird/workspace/Meta/cargo_build.sh index 5382d24..fc68fca 100644 --- a/examples/ladybird/workspace/Meta/cargo_build.sh +++ b/examples/ladybird/workspace/Meta/cargo_build.sh @@ -32,7 +32,8 @@ # and fails an action that does not write a declared one. (That check is how # HTMLTokenizerRustFFI.h -- written by a dependency's build script, declared # by nobody -- turned up.) Which crate's copy of a colliding header name -# wins is note 4, and it is the subtlest thing in this file. +# wins is sync_ffi_headers in cargo_vendor.sh, and it is the subtlest thing +# in this ring. # 3. **The cargo invocation mirrors CMake's exactly**: same subcommand # (`cargo rustc --lib`), same --target/--release, same trailing rustc flags # (-Cdefault-linker-libraries=yes -D warnings). Not for tidiness -- those @@ -60,11 +61,7 @@ source "${CARGO_VENDOR_LIB:?path to cargo_vendor.sh}" FEATURE_FLAGS=() [ -n "$FEATURES" ] && FEATURE_FLAGS=("--features=$FEATURES") -# FFI_OUTPUT_DIR is a SCRATCH dir, not the declared output dir -- see note 4 -# below for why that distinction is load-bearing. -FFI_SCRATCH="$WORK/ffi" -mkdir -p "$FFI_SCRATCH" "$FFI_OUT" -export FFI_OUTPUT_DIR="$FFI_SCRATCH" +mkdir -p "$FFI_OUT" cd "$SRC" "$SYSROOT/bin/cargo" rustc \ @@ -82,49 +79,7 @@ cd "$SRC" cp "$TARGET_DIR/$TRIPLE/release/lib$CRATE.a" "$OUT_LIB" -# --- note 4: resolve each header from the OWNING crate's OUT_DIR ------------- -# -# $FFI_OUTPUT_DIR is shared by every build script in the crate's dependency -# graph, and the header names COLLIDE: six crates each emit a file called -# `RustFFI.h`. liburl_rust path-depends on libregex_rust, so building liburl_rust -# runs BOTH build scripts against the same FFI_OUTPUT_DIR and the surviving -# RustFFI.h is whichever ran last -- we shipped libregex_rust's URL header for -# one build and it was only caught by byte-comparing against CMake's tree. -# -# CMake hits this too, and has a whole script for it -# (Meta/CMake/sync_rust_ffi_header.cmake): after cargo runs it copies the header -# out of the OWNING crate's own OUT_DIR, found via -# `build/-*/root-output`, and that copy is what the compiler sees. Mirror -# that: prefer the crate's own OUT_DIR, and fall back to the shared scratch dir -# for a header written by a DEPENDENCY's build script (libweb_rust's -# HTMLTokenizerRustFFI.h comes from libweb_html_tokenizer and collides with -# nothing, which is exactly why CMake never noticed it was undeclared). -BUILD_DIR="$TARGET_DIR/$TRIPLE/release/build" -missing=() -for h in "${FFI_HEADERS[@]+"${FFI_HEADERS[@]}"}"; do - mkdir -p "$FFI_OUT/$(dirname "$h")" - src="" - # The crate's own OUT_DIR, newest first: a rebuild can leave several. - for ro in $(ls -t "$BUILD_DIR/$CRATE"-*/root-output 2>/dev/null); do - cand="$(cat "$ro")/$h" - [ -f "$cand" ] && { src="$cand"; break; } - done - # Else a dependency's build script wrote it to the shared dir. - [ -z "$src" ] && [ -f "$FFI_SCRATCH/$h" ] && src="$FFI_SCRATCH/$h" - if [ -z "$src" ]; then - missing+=("$h") - continue - fi - cp "$src" "$FFI_OUT/$h" -done -if [ ${#missing[@]} -gt 0 ]; then - echo "cargo_build: $CRATE declared FFI headers cargo never wrote: ${missing[*]}" >&2 - echo "cargo_build: what it DID write under $FFI_SCRATCH:" >&2 - (cd "$FFI_SCRATCH" && find . -type f | sed 's|^\./| |') >&2 - echo "cargo_build: and in its own OUT_DIRs:" >&2 - for ro in "$BUILD_DIR/$CRATE"-*/root-output; do - [ -f "$ro" ] || continue - (cd "$(cat "$ro")" && find . -type f -name "*.h" | sed 's|^\./| |') >&2 - done - exit 1 -fi +# The declared headers, resolved out of the OWNING crate's OUT_DIR -- see +# sync_ffi_headers in cargo_vendor.sh for why $FFI_OUTPUT_DIR alone is not enough +# (eight crates emit a file called RustFFI.h into one shared directory). +sync_ffi_headers "$FFI_OUT" "${FFI_HEADERS[@]+"${FFI_HEADERS[@]}"}" diff --git a/examples/ladybird/workspace/Meta/cargo_vendor.sh b/examples/ladybird/workspace/Meta/cargo_vendor.sh index c27b4a2..1a686c5 100644 --- a/examples/ladybird/workspace/Meta/cargo_vendor.sh +++ b/examples/ladybird/workspace/Meta/cargo_vendor.sh @@ -87,3 +87,67 @@ export AR_${TRIPLE_US}="${AR:-ar}" export CARGO_TARGET_${TRIPLE_UP}_LINKER="${CC:-cc}" TARGET_DIR="$WORK/target" + +# --- FFI_OUTPUT_DIR is a SCRATCH dir, not the declared output dir ------------ +# Shared by both drivers because the reason is the same for both: cbindgen runs +# from a build script, every build script in the crate's dependency graph writes +# into the SAME $FFI_OUTPUT_DIR, and the header names COLLIDE (eight crates emit +# a file called `RustFFI.h`). So the declared outputs are resolved out of the +# OWNING crate's own OUT_DIR afterwards -- see sync_ffi_headers. +FFI_SCRATCH="$WORK/ffi" +mkdir -p "$FFI_SCRATCH" +export FFI_OUTPUT_DIR="$FFI_SCRATCH" + +# sync_ffi_headers [header...] +# +# Copy each declared FFI header into the declared output dir, resolved from the +# crate that OWNS it. Mirrors Meta/CMake/sync_rust_ffi_header.cmake, which exists +# for exactly this reason: liburl_rust path-depends on libregex_rust, so building +# liburl_rust runs BOTH build scripts against one FFI_OUTPUT_DIR and the +# surviving RustFFI.h is whichever ran last -- we shipped libregex_rust's URL +# header for one build and only caught it by byte-comparing against CMake's tree. +# +# CMake finds the owning crate's OUT_DIR via `build/-*/root-output`; so +# does this, preferring it and falling back to the shared scratch dir for a +# header written by a DEPENDENCY's build script (libweb_rust's +# HTMLTokenizerRustFFI.h comes from libweb_html_tokenizer and collides with +# nothing, which is exactly why CMake never noticed it was undeclared). +# +# Failing loudly is the point: Bazel deletes an undeclared output and fails an +# action that does not write a declared one, so a missing header is reported here +# with what cargo DID write rather than surfacing as a file-not-found in a C++ +# compile a thousand actions later. +sync_ffi_headers() { + local ffi_out="$1"; shift + [ $# -gt 0 ] || return 0 + local build_dir="$TARGET_DIR/$TRIPLE/release/build" + local missing=() h ro cand src + mkdir -p "$ffi_out" + for h in "$@"; do + mkdir -p "$ffi_out/$(dirname "$h")" + src="" + # The crate's own OUT_DIR, newest first: a rebuild can leave several. + for ro in $(ls -t "$build_dir/$CRATE"-*/root-output 2>/dev/null); do + cand="$(cat "$ro")/$h" + [ -f "$cand" ] && { src="$cand"; break; } + done + # Else a dependency's build script wrote it to the shared dir. + [ -z "$src" ] && [ -f "$FFI_SCRATCH/$h" ] && src="$FFI_SCRATCH/$h" + if [ -z "$src" ]; then + missing+=("$h") + continue + fi + cp "$src" "$ffi_out/$h" + done + if [ ${#missing[@]} -gt 0 ]; then + echo "cargo: $CRATE declared FFI headers cargo never wrote: ${missing[*]}" >&2 + echo "cargo: what it DID write under $FFI_SCRATCH:" >&2 + (cd "$FFI_SCRATCH" && find . -type f | sed 's|^\./| |') >&2 + echo "cargo: and in its own OUT_DIRs:" >&2 + for ro in "$build_dir/$CRATE"-*/root-output; do + [ -f "$ro" ] || continue + (cd "$(cat "$ro")" && find . -type f -name "*.h" | sed 's|^\./| |') >&2 + done + return 1 + fi +} diff --git a/examples/ladybird/workspace/Meta/emit_build_bazel.py b/examples/ladybird/workspace/Meta/emit_build_bazel.py index 5de48a1..f2983b8 100644 --- a/examples/ladybird/workspace/Meta/emit_build_bazel.py +++ b/examples/ladybird/workspace/Meta/emit_build_bazel.py @@ -25,6 +25,49 @@ # all, and grouping them makes the linker pull one crate's objects into a target # that never asked for them. RUST_LIB_FMT = "//:%s_lib" +# Crates whose FFI header the OWNING library includes with no directory prefix +# (`#include `). Those libraries additionally depend on +# //:_bare_include, which carries ONLY that crate's ffi/ dir. +# It is a separate label from //:_lib on purpose: CcInfo include dirs +# propagate transitively, and 8 of the 10 crates emit a header named RustFFI.h, +# so a shared unprefixed dir let one crate's header satisfy another library's +# bare include (LibGfx compiled against LibRegex's). +RUST_BARE_INCLUDE_FMT = "//:%s_bare_include" + + +def _cargo(): + sys.path.insert(0, os.path.join(ROOT, "Meta")) + import emit_cargo_bazel + return emit_cargo_bazel + + +def rust_bare_include_crates(): + """The crates whose header is included bare -- IMPORTED, not mirrored. + + This used to be a hand-kept tuple beside the cargo emitter's derived set, + with a comment saying the two were "kept in sync". They were not: the cargo + emitter DERIVES the set by scanning the tree for a directory-less + `#include `, and a hand-kept copy of a derived set is the finding-23 bug in + miniature. So it is read from the one place that computes it, and adding a + bare include anywhere in Ladybird needs no edit in either file. + """ + cargo = _cargo() + return {c for c, on in + cargo.crates_included_bare(cargo._bare_scan_specs()).items() if on} + + +def rust_binary_crates(): + """{CMake target name -> crate} for the build_rust_binary() crates. + + CMake's dependency edge on a binary crate is a custom target named + `-build` (rust_crate.cmake), and that name is what turns up in + the model as a dep -- `cranelift-compiler-build` on LibWasm. Mapping it back + to the crate is what lets the emitter translate that edge instead of dropping + it, which is precisely what it was doing: LibWasm's dep on the Cranelift + crate was silently discarded, and the build failed 1,600 actions later on + `CraneliftFFI.h: No such file or directory`. + """ + return {b["bin"] + "-build": b for b in _cargo().binary_specs()} # Global defines already set in .bazelrc; do not re-emit per target. GLOBAL_DEFINES = { @@ -104,13 +147,48 @@ def target_srcs(t): srcs += [i for i in a["inputs"] if i.endswith((".cpp", ".c", ".cc", ".S"))] return sorted(s for s in set(srcs) if AUTOGEN_MARKER not in s) +def rewrite_host_path_define(d): + """Rewrite a define whose VALUE is an absolute path to a Bazel-built binary. + + There is exactly one, and it was the last hardcoded + /home/ubuntu/... in the emitted build: + + -DWASM_CRANELIFT_COMPILER_PATH="/bin/cranelift-compiler" + + CMake bakes the reference build's absolute output path in, which is fine for + CMake (the build tree does not move) and fatal for a checkout on any other + machine -- the define alone made `git clone && bazel build` unusable even once + the crate itself built. + + The fix is not "point it at bazel-bin" (that is the same escape with a + different prefix). Ladybird already resolves the compiler through a lookup + CHAIN (resolve_cranelift_compiler_path in CraneliftBridge.cpp): + $LADYBIRD_CRANELIFT_COMPILER, then the compile-time path, then + SIBLING-OF-SELF. Bazel puts every root-package output in ONE bin directory, + so cranelift-compiler is already a sibling of WebContent, ladybird and the + rest -- the third link in Ladybird's own chain finds it with no path baked in + at all. So the define becomes the bare file name (link 2 is then a + cwd-relative probe that harmlessly misses), and the binary is attached as + `data` on LibWasm so it is really THERE, in the runfiles of everything that + links LibWasm. The dependency is declared; the path is not asserted. + """ + key, _, value = d.partition("=") + literal = value.strip('"') + if not literal.startswith("/"): + return d + for b in rust_binary_crates().values(): + if os.path.basename(literal) == b["output_name"]: + return '%s="%s"' % (key, b["output_name"]) + return d + + def target_defines(t, name): defs = set() for a in t["actions"]: if a["mnemonic"] != "CppCompile": continue for x in a["arguments"]: if x.startswith("-D"): - d = x[2:] + d = rewrite_host_path_define(x[2:]) if d not in GLOBAL_DEFINES: # Bazel needs embedded quotes escaped in (local_)defines. defs.add(d.replace('"', '\\"')) @@ -160,29 +238,94 @@ def target_embed_inputs(t): embeds.add(rel) return sorted(embeds) +def rust_dep_labels(crate): + """(deps, implementation_deps) for one Rust crate dep. + + `deps` is always //:_lib -- the archive plus the crate's PREFIXED + headers, exactly CMake's target_link_libraries edge, and safe to propagate + because the spelling is unique per crate. + + `implementation_deps` is //:_bare_include when the owning library + spells the header with NO directory, and the distinction is the whole point. + Eight of the ten crates emit a header literally named `RustFFI.h`, so an + unprefixed include dir that reaches a second library makes a bare + `#include ` bind to whichever dir sorted first on the command line + -- silently, since both files exist. + + Splitting it into its own TARGET (cargo_bare_include) was necessary but NOT + sufficient, and this is the second half of that lesson. Include dirs + propagate along the C++ dep graph, not just out of one rule: LibGfx depends on + LibTextCodec, which owns libtextcodec_rust's bare dir, so LibGfx's compiles + received /libtextcodec_rust/ffi/LibTextCodec BEFORE its own + /libgfx_rust/ffi/LibGfx and YUVData.cpp compiled against LibTextCodec's + header ("'FFI' does not name a type"). A separate target only stops the dir + leaking out of cargo_lib; it does not stop it leaking out of LibTextCodec. + + `implementation_deps` is Bazel's name for exactly the scope CMake's + `target_include_directories(... PRIVATE)` has: the dep is used to COMPILE this + library and is not part of its interface, so its include dirs stop here. That + is a one-for-one translation of the CMake this build is a port of -- CMake's + FFI_OUTPUT_DIR is added PRIVATE, to the owning library's own binary dir -- and + it is the reason a bare include is unambiguous there and had to be made + unambiguous here. + """ + impl = [] + if crate in rust_bare_include_crates(): + impl.append(RUST_BARE_INCLUDE_FMT % crate) + return [RUST_LIB_FMT % crate], impl + + +# How one CMake dep is translated. Four kinds, kept as named fields rather than +# as "a string, or a list, or a 2-tuple whose first element is a magic word" -- +# which is what this was, and it stopped being safe the moment a translation +# needed to produce BOTH deps and implementation_deps (a 2-tuple, i.e. exactly +# the shape the ("SYS", name) sentinel already used). +class Dep: + def __init__(self, deps=(), impl=(), sys=(), unknown=()): + self.deps, self.impl = list(deps), list(impl) + self.sys, self.unknown = list(sys), list(unknown) + + def dep_label(d, targets, so, ar): nm = d["name"] if nm.startswith("lagom-"): tgt = lagom_to_target(nm, targets) - if tgt == "LibWeb": return "//Libraries/LibWeb:LibWeb" - return "//:%s" % tgt if tgt else None + if tgt == "LibWeb": return Dep(deps=["//Libraries/LibWeb:LibWeb"]) + return Dep(deps=["//:%s" % tgt]) if tgt else None if nm.endswith("_rust"): - return RUST_LIB_FMT % nm + deps, impl = rust_dep_labels(nm) + return Dep(deps=deps, impl=impl) + # A build_rust_binary() crate. CMake's edge is on the custom target + # `-build`, and it means two separate things that Bazel splits: + # the generated FFI HEADER (a dep, via //:_lib) and the BINARY itself, + # which is not linked at all -- LibWasm spawns it -- so it becomes runfiles + # (see the `data` handling in emit_target). + binaries = rust_binary_crates() + if nm in binaries: + b = binaries[nm] + if not b["ffi_headers"]: + return None + deps, impl = rust_dep_labels(b["crate"]) + return Dep(deps=deps, impl=impl) + # A staticlib crate's redundant `-build` custom target (CMake emits + # both it and the imported-library dep); the library dep carries everything. + if nm.endswith("-build"): + return None if nm in so or nm in ar: - return VCPKG + ":" + nm + return Dep(deps=[VCPKG + ":" + nm]) if nm in SYSTEM_LIBS: - return ("SYS", nm) # linkopt on final binary + return Dep(sys=[nm]) # linkopt on final binary # Qt6 + GL: system .so under /usr/lib; CMake finds them via find_package. # Map the CMake target name to the -l library name; the /usr/lib search # path is a global -L in .bazelrc. # Qt6 is a real Bazel dep via rules_qt: its qt.local_repo discovers the host # SDK through qmake, so moc/rcc and the Qt cc_librarys all come from one SDK. if nm in QT_MAP: - return QT_MAP[nm] + return Dep(deps=[QT_MAP[nm]]) SYS_MAP = {"GLX": "GLX", "OpenGL": "OpenGL"} if nm in SYS_MAP: - return ("SYS", SYS_MAP[nm]) - return ("UNKNOWN", nm) + return Dep(sys=[SYS_MAP[nm]]) + return Dep(unknown=[nm]) # Generated sources that BAZEL now produces itself (a genrule output in the root # package, listed in codegen_root.bzl). A model src under Build/full naming one @@ -323,6 +466,7 @@ def _emit_srcs(srcs): load("@rules_cc//cc:defs.bzl", "cc_binary", "cc_library") load(":cargo_ring.bzl", "cargo_ring") load(":codegen_root.bzl", "root_codegen") +load(":export_headers.bzl", "export_headers") load(":vcpkg.bzl", "vcpkg_tree", "vcpkg_tree_for_exec") load(":vcpkg_index.bzl", "VCPKG_DISTFILE_INDEX") @@ -335,6 +479,11 @@ def _emit_srcs(srcs): # Build/full. root_codegen() +# CMake's generate_export_header() output (15 Export.h) and the two AK +# configure_file headers, generated by Bazel instead of read out of Build/full -- +# generated by Meta/emit_export_headers_bazel.py, byte-verified with --check. +export_headers() + # The Rust ring: 10 cargo staticlib crates + flapc, built by Bazel from crates # Bazel fetched, offline. This is what retired the prebuilt 260 MB # librust_combined.a and its hand-run `ar -M` merge (README step 1b). It lives in @@ -366,6 +515,12 @@ def _emit_srcs(srcs): ":generated_libraries_headers", ":generated_services_headers", ":generated_shader_headers", + ":generated_export_headers", + ":generated_ak_headers", + "//Libraries/LibWeb:generated_export_header", + # Build/full roots: now ONLY the Rust FFI headers and the IPC/codegen + # headers Bazel does not yet generate. Export.h and AK's two + # configure_file headers are Bazel-generated above. "//Build/full/Libraries:generated_lib_headers", "//Build/full/Services:generated_service_headers", ], @@ -375,21 +530,18 @@ def _emit_srcs(srcs): ''' AK_BLOCK_HEAD = ''' -# Configure-generated headers live in Build/full/AK; expose as AK/*.h via a copy. -genrule( - name = "ak_gen_headers", - srcs = ["Build/full/AK/Debug.h", "Build/full/AK/Backtrace.h"], - outs = ["genroot/AK/Debug.h", "genroot/AK/Backtrace.h"], - cmd = "mkdir -p $(RULEDIR)/genroot/AK && " + - "cp $(location Build/full/AK/Debug.h) $(RULEDIR)/genroot/AK/Debug.h && " + - "cp $(location Build/full/AK/Backtrace.h) $(RULEDIR)/genroot/AK/Backtrace.h", -) +# NOTE: the ak_gen_headers genrule that COPIED AK/Debug.h and AK/Backtrace.h out +# of Build/full is gone. Bazel now generates both from their checked-in .in +# templates (export_headers.bzl): Debug.h by applying CMake's configure_file +# substitution, Backtrace.h by ASKING the host the same question +# find_package(Backtrace) asks. That removed the last AK dependency on a CMake +# build tree. cc_library( name = "AK", ''' -AK_BLOCK_TAIL = ''' hdrs = glob(["AK/*.h"]) + [":ak_gen_headers"], +AK_BLOCK_TAIL = ''' hdrs = glob(["AK/*.h"]), # AK-private defines (CMake PRIVATE) — local_defines so they don't leak to # consumers. FMT_SHARED/AK_HAS_CPPTRACE only affect AK's own TUs. local_defines = [ @@ -397,8 +549,13 @@ def _emit_srcs(srcs): "AK_HAS_CPPTRACE=1", "FMT_SHARED", ], - includes = ["genroot"], deps = [ + # AK/Debug.h + AK/Backtrace.h, generated from their .in templates + # (export_headers.bzl). A cc_library belongs on a deps edge, not in hdrs: + # in hdrs it contributes no include path, which is why every consumer + # failed with "AK/Debug.h: No such file or directory" until this moved. + # The include root travels with the dep, so dependents get it too. + ":generated_ak_headers", VCPKG + ":fmt", VCPKG + ":simdutf", VCPKG + ":mimalloc", @@ -468,26 +625,50 @@ def emit_target(name, targets, libs, exes, so, ar, header=True, body_only=False, defs = target_defines(t, name) incs = target_private_includes(t) flags = target_flags(t) - deps, rustdeps, sysdeps, unknown = [], [], [], [] + deps, impl_deps, data, sysdeps, unknown = [], [], [], [], [] + binaries = rust_binary_crates() for d in t.get("deps", []): if not d.get("external"): if d["name"] == "LibWeb": deps.append("//Libraries/LibWeb:LibWeb") elif d["name"] in libs: deps.append("//:%s" % d["name"]) # exe deps on service static libs / other production libs elif d["name"] in exes: pass # exe->exe (spawn at runtime, not a link dep) + elif d["name"] in binaries: + # A build_rust_binary() crate. For the STATICLIB crates CMake + # emits both this `-build` custom target AND an + # external dep on the imported library, so the -build edge is + # redundant and dropping it is right. For a BINARY crate it is + # the only edge there is -- and dropping it is what left + # Cranelift out of the Bazel graph entirely, so every browser + # binary died on `CraneliftFFI.h: No such file or directory` + # some 1,600 actions in. + # + # It carries two things Bazel keeps apart: + # deps -- the generated FFI header, via //:_lib + # (headers-only: there is no archive to link). + # data -- the EXECUTABLE, which is not a link input at all. + # LibWasm spawns it at run time, so it belongs in + # the runfiles of everything that links LibWasm. + # That, plus rewrite_host_path_define turning the + # baked-in absolute path into a bare file name, is + # what makes Ladybird's own sibling-of-self lookup + # find it in any checkout instead of only in mine. + b = binaries[d["name"]] + if b["ffi_headers"]: + rdeps, rimpl = rust_dep_labels(b["crate"]) + deps.extend(rdeps) + impl_deps.extend(rimpl) + data.append("//:" + b["bin"]) continue lab = dep_label(d, targets, so, ar) if lab is None: continue - if isinstance(lab, list): - deps.extend(lab) - elif isinstance(lab, tuple): - (sysdeps if lab[0]=="SYS" else unknown).append(lab[1]) - else: - deps.append(lab) + deps.extend(lab.deps) + impl_deps.extend(lab.impl) + sysdeps.extend(lab.sys) + unknown.extend(lab.unknown) rule = "cc_binary" if is_exe else "cc_library" if header: print(f"# === {name} ({t['kind']}, {len(srcs)} TU) ===") - if rustdeps: print(f"# RUST deps (deferred): {rustdeps}") if unknown: print(f"# UNKNOWN deps: {unknown}") if not body_only: print(f"{rule}(") @@ -511,7 +692,17 @@ def emit_target(name, targets, libs, exes, so, ar, header=True, body_only=False, # with -isystem. They live as global -isystem in .bazelrc # (mirroring CMake's find_package include dirs). continue - if "vcpkg_installed" in i: + if i.startswith("Build/full/Libraries/"): + # CMake's per-library FFI dir (FFI_OUTPUT_DIR defaults to the + # library's own binary dir), which is where its crate's + # RustFFI.h lands. Pointing this at the CMake tree is what kept + # the build dependent on Build/full -- and it also MASKED a real + # bug: it shadowed the ambiguous ffi/ roots, so a bare + # silently resolved here instead of to another + # crate's header. Bazel's equivalent dir travels with the + # //:_lib dep (cargo.bzl), so this copt is simply dropped. + pass + elif "vcpkg_installed" in i: # Not a copt any more. The vcpkg include dirs (include/, # include/skia, include/harfbuzz, include/libxml2) are carried by # the //Meta/vcpkg: dep as system_includes, so the include @@ -528,6 +719,23 @@ def emit_target(name, targets, libs, exes, so, ar, header=True, body_only=False, # where the lib's INTERFACE/PRIVATE system deps flow to the executable). if sysdeps: print(" linkopts = [%s]," % ", ".join("%r" % ("-l"+l) for l in sorted(set(sysdeps)))) + # Runtime tools this target SPAWNS: not link inputs, but real inputs. + # On a cc_library `data` propagates into the runfiles of every binary + # that links it, which is exactly the reach cranelift-compiler needs + # (LibWasm is linked by WebContent, WebWorker and ladybird). + if data: + print(" data = [%s]," % ", ".join("%r" % x for x in sorted(set(data)))) + # PRIVATE deps: used to compile this library, not part of its interface, + # so their include dirs stop here. Bazel's name for CMake's PRIVATE, and + # the only thing that keeps a crate's unprefixed FFI dir from reaching a + # library that has its OWN crate's RustFFI.h to find -- see + # rust_dep_labels(). Not valid on a cc_binary (nothing depends on it, so + # everything it has is already private). + if impl_deps and not is_exe: + print(" implementation_deps = [%s]," % + ", ".join("%r" % x for x in sorted(set(impl_deps)))) + elif impl_deps: + deps.extend(impl_deps) deps.append("//:all_source_headers") alldeps = sorted(set(deps)) if alldeps: diff --git a/examples/ladybird/workspace/Meta/emit_cargo_bazel.py b/examples/ladybird/workspace/Meta/emit_cargo_bazel.py index 6ec56b1..2f2547b 100644 --- a/examples/ladybird/workspace/Meta/emit_cargo_bazel.py +++ b/examples/ladybird/workspace/Meta/emit_cargo_bazel.py @@ -109,6 +109,20 @@ def toolchain_channel(): return _toml(os.path.join(ROOT, "rust-toolchain.toml"))["toolchain"]["channel"] +def root_workspace_members(): + """The root cargo workspace's member directories, read from Cargo.toml. + + Which workspace a crate belongs to decides its SOURCE SET, and it is the one + thing here that must not be guessed. `Libraries/LibWasm/Rust` is member #7 of + the root workspace, so cargo resolves the whole workspace when asked for it + and its input set is the shared `crate_srcs`; `Libraries/LibJS/Flap` is in + `exclude` and has its own lock, so its input set is its own subtree. Reading + the member list means adding a crate to either workspace needs no edit here. + """ + ws = _toml(os.path.join(ROOT, "Cargo.toml"))["workspace"] + return {os.path.normpath(m) for m in ws.get("members", [])} + + # --------------------------------------------------------------------------- # The lock files. # --------------------------------------------------------------------------- @@ -206,6 +220,8 @@ def repo_name(name, version): "libweb_css_rust": ["ComputedValuesRustFFI.h", "RustFFI.h", "SelectorRustFFI.h", "StyleValueRustFFI.h"], "libweb_content_blocker_rust": ["ContentBlockerRustFFI.h"], + # A BINARY crate that also emits a header -- see BINARY_CMAKELISTS below. + "libwasm_cranelift": ["CraneliftFFI.h"], } # Where each crate's headers are included FROM. CMake sets FFI_OUTPUT_DIR to the @@ -224,6 +240,7 @@ def repo_name(name, version): "libweb_layout_rust": "LibWeb", "libweb_css_rust": "LibWeb", "libweb_content_blocker_rust": "LibWeb", + "libwasm_cranelift": "LibWasm", } # The non-Rust files each crate's build script reads, taken from the DEPFILES the @@ -236,6 +253,12 @@ def repo_name(name, version): # through a `manifest_dir.parent()` join. CRATE_EXTRA_INPUTS = { "libjs_rust": ["Libraries/LibJS/Bytecode/Bytecode.def"], + # libwasm_cranelift's build.rs reads `manifest_dir.join("../Opcode.h")` and + # generates a Rust constant per `M(name, value, ...)` line, so the WASM + # opcode table is a compile input to the cranelift compiler binary. Exactly + # the `manifest_dir.parent()` join the depfile finds and reading build.rs + # nearly misses. + "libwasm_cranelift": ["Libraries/LibWasm/Opcode.h"], # The LibWeb ones (8 CSS data files for libweb_css_rust, TagNames.h + # AttributeNames.h + Entities.json for libweb_rust) are NOT here: they are # inside the Libraries/LibWeb package, so they ride in that package's @@ -321,10 +344,15 @@ def _multi(block, key, stops): "Libraries/LibURL/CMakeLists.txt", "Libraries/LibUnicode/CMakeLists.txt", "Libraries/LibWeb/CMakeLists.txt", + # LibWasm calls build_rust_binary(), not import_rust_crate() -- see + # parse_cmake_binaries() for why that distinction is load-bearing and why it + # nevertheless has to be in the SAME list. + "Libraries/LibWasm/CMakeLists.txt", ] _KEYS = ["MANIFEST_PATH", "CRATE_NAME", "FFI_OUTPUT_DIR", "FFI_HEADER", - "FFI_HEADERS", "FEATURES"] + "FFI_HEADERS", "FEATURES", "BINARY_NAME", "OUTPUT_NAME", + "OUTPUT_PATH_VAR"] def parse_cmake_crates(): @@ -384,6 +412,162 @@ def parse_cmake_crates(): return sorted(out, key=lambda c: c["crate"]) +def parse_cmake_binaries(): + """[{crate, bin, manifest, ffi_headers}] from build_rust_binary() calls. + + The SECOND kind of Rust target CMake has, and the one this migration missed + for a long time, with a consequence worth recording: because + `Libraries/LibWasm/CMakeLists.txt` was absent from CMAKELISTS and + `build_rust_binary` was not parsed, the Cranelift crate was **entirely absent + from the Bazel graph** -- and every browser binary failed with + `CraneliftFFI.h: No such file or directory`, 1,600 actions into a build that + could not have worked. The reference build hid it twice over: the header was + sitting in `Build/full/Libraries/LibWasm`, which a global `-I` reached, and + the compiler binary was reached by an ABSOLUTE PATH baked into a + `-DWASM_CRANELIFT_COMPILER_PATH=` -- i.e. two host escapes covering for one + missing target. + + Two things make a build_rust_binary crate different from an + import_rust_crate one, and both matter here: + + * There is NO archive to link. `cargo rustc --bin` is the whole build, and + what the C++ side consumes is the *executable*, spawned at run time. + * It can STILL emit an FFI header. libwasm_cranelift's build.rs runs + cbindgen exactly like the staticlib crates do, and + `Libraries/LibWasm/CraneliftBridge.cpp` includes the result -- bare + (`#include `), because CMake's FFI_OUTPUT_DIR is + LibWasm's own binary dir. + + So a binary crate needs both halves: a cargo_binary for the executable, and + the same declared-header treatment the staticlib crates get. The two are + reported separately for the same reason import_rust_crate's list is parsed + rather than hard-coded: which kind a crate is, and whether it emits a header, + is a property of the CMake call site. + """ + out = [] + for rel in CMAKELISTS: + path = os.path.join(ROOT, rel) + if not os.path.exists(path): + continue + with open(path) as f: + text = f.read() + pkgdir = os.path.dirname(rel) + for block in RE_BINARY.findall(text): + crate = _arg(block, "CRATE_NAME") + manifest = _arg(block, "MANIFEST_PATH") + binary = _arg(block, "BINARY_NAME") + if not (crate and manifest and binary): + continue + # FFI_OUTPUT_DIR present at the call site means the build script is + # expected to write a header. Which header it writes is the + # observation (FFI_HEADERS_OBSERVED), exactly as for the staticlibs: + # build_rust_binary has no FFI_HEADERS argument at all, so CMake + # declares nothing and there is nothing to compare against. + emits_ffi = _arg(block, "FFI_OUTPUT_DIR") is not None + out.append({ + "crate": crate, + "bin": binary, + "manifest": os.path.normpath(os.path.join(pkgdir, manifest)), + "output_name": _arg(block, "OUTPUT_NAME") or binary, + "emits_ffi": emits_ffi, + }) + return sorted(out, key=lambda c: c["crate"]) + + +def binary_specs(): + """The per-binary-crate recipe, with the FFI headers each one emits. + + `ffi_bare_include` is derived the same way the staticlib crates' is (by + scanning the tree for a directory-less `#include`), not asserted: LibWasm + spells it `#include `, and it must be that scan that says so. + """ + specs = [] + for b in parse_cmake_binaries(): + headers = sorted(FFI_HEADERS_OBSERVED.get(b["crate"], [])) \ + if b["emits_ffi"] else [] + specs.append(dict(b, ffi_headers=headers, + ffi_prefix=FFI_PREFIX.get(b["crate"], ""))) + bare = crates_included_bare(_bare_scan_specs()) + for sp in specs: + sp["ffi_bare_include"] = bool(bare.get(sp["crate"])) + return specs + + +def _bare_scan_specs(): + """Every Rust crate that emits a header, staticlib and binary alike. + + ONE list, because the bare-include scan is a question about the whole tree: + it looks for a directory-less `#include ` where X is any crate's header + basename, and a scan that only knew about the staticlibs would answer "no" + for CraneliftFFI.h no matter how many TUs include it that way. + """ + return [{"crate": c["crate"], "manifest": c["manifest"], + "ffi_headers": sorted(set(c["cmake_ffi_headers"]) | + set(FFI_HEADERS_OBSERVED.get(c["crate"], [])))} + for c in parse_cmake_crates()] + \ + [{"crate": b["crate"], "manifest": b["manifest"], + "ffi_headers": sorted(FFI_HEADERS_OBSERVED.get(b["crate"], [])) + if b["emits_ffi"] else []} + for b in parse_cmake_binaries()] + + +def crates_included_bare(specs): + """Crates whose FFI header is #included with NO directory prefix. + + Derived by scanning the source tree for `#include ` where X is exactly a + crate's FFI header basename, rather than hard-coding a list: which TUs spell + the include bare is a property of Ladybird's source and changes when someone + edits an include line. + + Why it matters: 8 of the 10 crates emit a header named `RustFFI.h`. CMake + gives each library only its OWN crate's binary dir, so a bare + `#include ` is unambiguous there. Bazel propagates every dep's + include dirs onto one command line, so exposing an unprefixed dir for all + crates lets one crate's RustFFI.h satisfy another's include -- silently, since + both exist and only ORDER decides. So the unprefixed dir is opt-in per crate, + and only for crates that actually need it. + """ + bare = {} + pat = re.compile(r'^\s*#\s*include\s*<([^/>]+)>\s*$', re.M) + wanted = {} + owner_dir = {} + for sp in specs: + # The crate's manifest lives at /[sub/]Rust/Cargo.toml, so the + # owning library directory is the manifest's path with the trailing + # Rust/Cargo.toml removed -- that is the tree whose TUs may include this + # crate's header bare. + d = os.path.dirname(sp["manifest"]) + while d and os.path.basename(d) in ("Rust", "rust"): + d = os.path.dirname(d) + owner_dir[sp["crate"]] = d + for h in sp["ffi_headers"]: + wanted.setdefault(os.path.basename(h), set()).add(sp["crate"]) + for dirpath, dirnames, filenames in os.walk(os.path.join(ROOT, "Libraries")): + dirnames[:] = [d for d in dirnames if d not in ("Rust", "target")] + for fn_ in filenames: + if not fn_.endswith((".cpp", ".h")): + continue + fpath = os.path.join(dirpath, fn_) + try: + with open(fpath, errors="replace") as f: + text = f.read() + except OSError: + continue + for m in pat.finditer(text): + inc = m.group(1) + if inc not in wanted: + continue + # Attribute the bare include to the crate owned by the library + # this file lives in, so we never enable the unprefixed dir for + # an unrelated crate that happens to emit the same basename. + rel = os.path.relpath(fpath, ROOT) + for crate in wanted[inc]: + owner = owner_dir.get(crate) + if owner and rel.startswith(owner + os.sep): + bare[crate] = True + return bare + + def crate_specs(): """The per-crate build recipe the Bazel rule consumes. @@ -403,6 +587,13 @@ def crate_specs(): "ffi_headers": headers, "ffi_prefix": FFI_PREFIX.get(c["crate"], ""), }) + # Derived from the source tree, not hard-coded: which crates' headers are + # included without a directory prefix. Scanned over EVERY crate that emits a + # header (binaries included), so the answer does not depend on which emitter + # asked. + bare = crates_included_bare(_bare_scan_specs()) + for sp in specs: + sp["ffi_bare_include"] = bool(bare.get(sp["crate"])) return specs @@ -486,6 +677,7 @@ def emit_index(crates, specs): print(" \"features\": %r," % s["features"]) print(" \"ffi_headers\": %r," % s["ffi_headers"]) print(" \"ffi_prefix\": %r," % s["ffi_prefix"]) + print(" \"ffi_bare_include\": %r," % s["ffi_bare_include"]) print(" },") print("}") print() @@ -495,7 +687,7 @@ def emit_index(crates, specs): print("RUST_TRIPLE = %r" % TRIPLE) -def emit_ring(crates, specs): +def emit_ring(crates, specs, binaries): """cargo_ring.bzl: the whole Rust ring as a macro for the ROOT package. A macro instantiated in the root package rather than its own `Meta/cargo` @@ -511,20 +703,20 @@ def emit_ring(crates, specs): and a hand-kept copy of them would be the bug (finding 23). """ sys.stdout.write(HEADER) - print('load(":cargo.bzl", "cargo_binary", "cargo_crate", "cargo_lib", "rust_sysroot")') + print('load(":cargo.bzl", "cargo_binary", "cargo_crate", "cargo_lib", "cargo_bare_include", "rust_sysroot")') print('load(":cargo_index.bzl", "CARGO_CRATE_FILES", "CARGO_CRATE_SPECS")') print() - print("# Ladybird's %d production Rust crates and flapc, BUILT BY BAZEL from the" - % len(specs)) - print("# %d crates.io crates Bazel fetched -- replacing the prebuilt 260 MB" % len(crates)) - print("# librust_combined.a that was copied out of Build/full/cargo/, the `ar -M`") - print("# merge that produced it (README step 1b), and the reference build's flapc") - print("# binary. Nothing here names Build/full.") + print("# Ladybird's %d production Rust crates and %d binary crates, BUILT BY BAZEL" + % (len(specs), len(binaries))) + print("# from the %d crates.io crates Bazel fetched -- replacing the prebuilt" % len(crates)) + print("# 260 MB librust_combined.a that was copied out of Build/full/cargo/, the") + print("# `ar -M` merge that produced it (README step 1b), and the reference build's") + print("# flapc and cranelift-compiler binaries. Nothing here names Build/full.") print("#") print("# Every attribute comes from CARGO_CRATE_SPECS, which") - print("# Meta/emit_cargo_bazel.py parses out of import_rust_crate() in CMake, so a") - print("# feature or header list cannot drift from the reference build without the") - print("# emitter's --report saying so.") + print("# Meta/emit_cargo_bazel.py parses out of import_rust_crate() and") + print("# build_rust_binary() in CMake, so a feature or header list cannot drift from") + print("# the reference build without the emitter's --report saying so.") print() print("def cargo_ring():") print(" # The pinned toolchain (rust-toolchain.toml says %s), assembled from" @@ -562,6 +754,8 @@ def emit_ring(crates, specs): print(" crate_features = CARGO_CRATE_SPECS[%r][\"features\"]," % s["crate"]) print(" ffi_headers = CARGO_CRATE_SPECS[%r][\"ffi_headers\"]," % s["crate"]) print(" ffi_prefix = CARGO_CRATE_SPECS[%r][\"ffi_prefix\"]," % s["crate"]) + print(" ffi_bare_include = CARGO_CRATE_SPECS[%r][\"ffi_bare_include\"]," + % s["crate"]) print(" manifest = CARGO_CRATE_SPECS[%r][\"manifest\"]," % s["crate"]) if extra: print(" # Build-script inputs, taken from the reference build's DEPFILES") @@ -584,21 +778,80 @@ def emit_ring(crates, specs): print(" name = %r," % (s["crate"] + "_lib")) print(" crate = %r," % (":" + s["crate"])) print(" )") + if s["ffi_bare_include"]: + # The owning library's TUs spell this crate's header with no + # directory. That dir must NOT ride on cargo_lib (it propagates + # transitively and 8 crates ship a RustFFI.h), so it is its own + # target, depended on by exactly one library. + print(" cargo_bare_include(") + print(" name = %r," % (s["crate"] + "_bare_include")) + print(" crate = %r," % (":" + s["crate"])) + print(" )") print() - print(" # flapc, the Flap-DSL -> interpreter-assembly compiler. Its workspace is") - print(" # `exclude`d from the root one and has its own lock with exactly 3") - print(" # packages (flapc, in-tree bytecode_def, and smallvec from crates.io") - print(" # pinned =1.15.1 -- the same version AND checksum the big workspace pins,") - print(" # so it is the same fetch rule, not a second one).") - print(" cargo_binary(") - print(" name = \"flapc\",") - print(" bin = \"flapc\",") - print(" crate = \"flapc\",") - print(" crates = CARGO_CRATE_FILES,") - print(" manifest = \"Libraries/LibJS/Flap/Cargo.toml\",") - print(" srcs = native.glob(%r, allow_empty = False)," % FLAPC_SRC_GLOBS) - print(" sysroot = \":rust_sysroot\",") - print(" )") + print(" # The BINARY crates: cargo `--bin` targets, declared in CMake with") + print(" # build_rust_binary() rather than import_rust_crate(). Two of them, and") + print(" # they are two different shapes:") + print(" #") + print(" # flapc a pure build TOOL -- Bazel runs it in a genrule to") + print(" # produce the interpreter assembly. Its workspace is") + print(" # `exclude`d from the root one and has its own lock") + print(" # with exactly 3 packages (flapc, in-tree") + print(" # bytecode_def, and smallvec from crates.io pinned") + print(" # =1.15.1 -- the same version AND checksum the big") + print(" # workspace pins, so it is the same fetch rule, not a") + print(" # second one).") + print(" # cranelift-compiler a RUNTIME tool -- LibWasm spawns it to AOT-compile") + print(" # WebAssembly, and its build script ALSO emits the") + print(" # CraneliftFFI.h that LibWasm's CraneliftBridge.cpp") + print(" # includes. It is a root-workspace member, so its") + print(" # source set is the shared crate_srcs.") + for b in binaries: + member = os.path.dirname(b["manifest"]) + in_root_ws = member in root_workspace_members() + extra = CRATE_EXTRA_INPUTS.get(b["crate"], []) + print(" cargo_binary(") + print(" name = %r," % b["bin"]) + print(" bin = %r," % b["bin"]) + print(" crate = %r," % b["crate"]) + print(" crates = CARGO_CRATE_FILES,") + print(" manifest = %r," % b["manifest"]) + if b["ffi_headers"]: + print(" # This binary crate's build script runs cbindgen too, so the") + print(" # header is a DECLARED output -- Bazel deletes what nothing") + print(" # declares, and LibWasm includes this one bare.") + print(" ffi_headers = %r," % b["ffi_headers"]) + print(" ffi_prefix = %r," % b["ffi_prefix"]) + print(" ffi_bare_include = %r," % b["ffi_bare_include"]) + if in_root_ws: + print(" # A member of the ROOT cargo workspace (Cargo.toml lists it), so") + print(" # cargo resolves the whole workspace whichever crate you ask for") + print(" # and the source set is the shared one -- same over-declaration,") + print(" # same reason, as the staticlib crates.") + srcs = "crate_srcs" + if extra: + srcs += " + %r" % extra + print(" srcs = %s," % srcs) + else: + print(" # Its OWN workspace (`exclude`d from the root one), so its source") + print(" # set is its own subtree.") + print(" srcs = native.glob(%r, allow_empty = False)," % FLAPC_SRC_GLOBS) + print(" sysroot = \":rust_sysroot\",") + print(" )") + if b["ffi_headers"]: + print() + print(" # The consumable target, IDENTICAL in shape to a staticlib crate's:") + print(" # this crate's generated header to include. There is no archive to") + print(" # link -- what C++ consumes from a binary crate is the EXECUTABLE,") + print(" # spawned at run time -- so cargo_lib yields a headers-only CcInfo.") + print(" cargo_lib(") + print(" name = %r," % (b["crate"] + "_lib")) + print(" crate = %r," % (":" + b["bin"])) + print(" )") + if b["ffi_bare_include"]: + print(" cargo_bare_include(") + print(" name = %r," % (b["crate"] + "_bare_include")) + print(" crate = %r," % (":" + b["bin"])) + print(" )") def emit_extension(crates): @@ -674,10 +927,27 @@ def report(crates): if missing: print(" declared by CMake but never written: %s" % ", ".join(missing)) rc = 1 - for crate in sorted(set(FFI_HEADERS_OBSERVED) - - {c["crate"] for c in parse_cmake_crates()}): - print("%-30s observed but no import_rust_crate found -- did a crate get " - "renamed or moved?" % crate) + print() + # The binary crates are reported separately, because the two kinds fail + # differently: a missing staticlib crate is a link error, a missing binary + # crate is a header that never appears (CraneliftFFI.h) plus a tool that has + # to be found at run time. + for b in binary_specs(): + print("%-30s bin=%-20s manifest=%s" + % (b["crate"], b["bin"], b["manifest"])) + if b["emits_ffi"] and not b["ffi_headers"]: + print(" FFI_OUTPUT_DIR is set at the call site but no header is " + "listed in FFI_HEADERS_OBSERVED -- run the reference cargo " + "build with FFI_OUTPUT_DIR set and add what it wrote") + rc = 1 + for h in b["ffi_headers"]: + print(" emits %s (bare-included: %s)" + % (h, "yes" if b["ffi_bare_include"] else "no")) + known = ({c["crate"] for c in parse_cmake_crates()} | + {b["crate"] for b in parse_cmake_binaries()}) + for crate in sorted(set(FFI_HEADERS_OBSERVED) - known): + print("%-30s observed but no import_rust_crate/build_rust_binary found " + "-- did a crate get renamed or moved?" % crate) rc = 1 return rc @@ -707,7 +977,7 @@ def check(dirpath): {"--crates": lambda: emit_crates(crates), "--index": lambda: emit_index(crates, specs), "--extension": lambda: emit_extension(crates), - "--ring": lambda: emit_ring(crates, specs)}[flag]() + "--ring": lambda: emit_ring(crates, specs, binary_specs())}[flag]() finally: sys.stdout = real path = os.path.join(dirpath, fn) @@ -737,7 +1007,7 @@ def main(): elif "--extension" in argv: emit_extension(crates) elif "--ring" in argv: - emit_ring(crates, crate_specs()) + emit_ring(crates, crate_specs(), binary_specs()) elif "--use-repo" in argv: emit_use_repo(crates) else: diff --git a/examples/ladybird/workspace/Meta/emit_export_headers_bazel.py b/examples/ladybird/workspace/Meta/emit_export_headers_bazel.py new file mode 100644 index 0000000..923e3d5 --- /dev/null +++ b/examples/ladybird/workspace/Meta/emit_export_headers_bazel.py @@ -0,0 +1,421 @@ +#!/usr/bin/env python3 +"""Emit Bazel genrules for CMake's `generate_export_header` output. + +WHY THIS EXISTS +--------------- +Ladybird's `ladybird_lib()` calls `ladybird_generate_export_header(name fs_name)` +(Meta/CMake/targets.cmake), which is CMake's `GenerateExportHeader` module with +`EXPORT_MACRO_NAME _API` and `EXPORT_FILE_NAME "Export.h"`. That +writes `Build//Libraries//Export.h` -- a *configure*-time artifact of +the reference build. + +Until now Bazel did not generate these 15 headers at all: it globbed them out of +`Build/full/Libraries/**` via the `//Build/full/Libraries:generated_lib_headers` +shim. That made `bazel build //:ladybird` silently depend on a completed CMake +build -- and because the shim uses `glob(..., allow_empty = True)`, a fresh +checkout produced NO error from the glob, just `fatal error: LibXML/Export.h: No +such file or directory` ~1,600 actions into the build. Generating them here is +what lets a clean `git clone` build. + +WHY A TEMPLATE IS THE RIGHT ANSWER HERE +--------------------------------------- +Reproducing another build system's output by re-implementing its generator is +normally the wrong move -- it is a fork that silently drifts. Two things make this +the exception, and both were checked rather than assumed: + +1. **The output is a pure function of one token.** All 15 checked-in headers + normalize to a SINGLE byte-identical template under substitution of + (``, ``, `_EXPORTS`), all three derived from the library name. + Verified by normalizing each of the 15 and comparing: 1 distinct template. +2. **The alternative is worse.** The only other faithful option is running CMake, + which is the dependency being removed. + +So the risk is a future CMake version changing the template. That is caught, not +hoped about: `--check` byte-compares this emitter's output for every library +against the reference tree's copy, and the parity harness diffs generated output +tree-wide. If CMake's template changes, `--check` fails loudly instead of the +build succeeding with a subtly wrong header. + +The template is CMake's own (Modules/GenerateExportHeader.cmake) for the +GNU/Clang, non-Windows case -- the only case Ladybird's Linux build exercises. The +`#if 0 /* DEFINE_NO_DEPRECATED */` block and the leading blank line are +reproduced deliberately: byte parity includes the parts that look like noise. + +ALSO: THE TWO `configure_file` HEADERS +-------------------------------------- +`AK/CMakeLists.txt` runs `configure_file(Debug.h.in Debug.h @ONLY)` and the same +for `Backtrace.h.in`. These were the last two headers Bazel read out of +`Build/full/AK/`, and they are a different shape from Export.h: the TEMPLATE is +checked into the source tree, so nothing is being re-implemented here except +CMake's substitution rules (`#cmakedefine01` -> `#define X 0|1`, `#cmakedefine` +-> `#define X` or a comment, `@VAR@` -> value). + +`Debug.h` is 79 `#cmakedefine01` lines and every one of them is `0` in the +reference build -- they are opt-in debug spew, off unless someone passes +`-DFOO_DEBUG=ON`. So an unconfigured emit is the faithful answer. + +`Backtrace.h` is NOT a template substitution, it is a HOST PROBE: +`find_package(Backtrace)` decides whether `execinfo.h` exists and what it is +called. Pasting this machine's answer (`execinfo.h`) into a checked-in file is +exactly the "test the variable, not the value" mistake from the Dolphin case +study -- it would survive every check on glibc and break on musl, where backtrace +lives elsewhere or not at all. So the genrule ASKS the question at build time by +compiling a probe, the same question CMake asks, and emits whichever answer this +host gives. + +Usage: + python3 Meta/emit_export_headers_bazel.py # emit export_headers.bzl + python3 Meta/emit_export_headers_bazel.py --check # byte-verify vs reference +""" + +import argparse +import os +import re +import sys + +# The libraries CMake generates an Export.h for: those built with +# LADYBIRD_LIB_EXPLICIT_SYMBOL_EXPORT, so that ladybird_lib() ran +# ladybird_generate_export_header(). Kept as an explicit list (not a glob over +# Libraries/) precisely so a library gaining or losing an export header shows up +# as a --check failure rather than silently changing the build. +EXPORT_LIBS = [ + "LibCore", + "LibDNS", + "LibDatabase", + "LibDevTools", + "LibGC", + "LibJS", + "LibMedia", + "LibRegex", + "LibSync", + "LibTest", + "LibTextCodec", + "LibWasm", + "LibWeb", + "LibWebView", + "LibXML", +] + +# CMake's GenerateExportHeader output for GCC/Clang on non-Windows. +# @API@ = EXPORT_MACRO_NAME (upper(fs_name) + "_API") +# @PRE@ = upper(target name), used for the NO_EXPORT/DEPRECATED macros +# @EXP@ = "_EXPORTS", the macro CMake defines while building the target +TEMPLATE = """ +#ifndef @API@_H +#define @API@_H + +#ifdef @PRE@_STATIC_DEFINE +# define @API@ +# define @PRE@_NO_EXPORT +#else +# ifndef @API@ +# ifdef @EXP@ + /* We are building this library */ +# define @API@ __attribute__((visibility("default"))) +# else + /* We are using this library */ +# define @API@ __attribute__((visibility("default"))) +# endif +# endif + +# ifndef @PRE@_NO_EXPORT +# define @PRE@_NO_EXPORT __attribute__((visibility("hidden"))) +# endif +#endif + +#ifndef @PRE@_DEPRECATED +# define @PRE@_DEPRECATED __attribute__ ((__deprecated__)) +#endif + +#ifndef @PRE@_DEPRECATED_EXPORT +# define @PRE@_DEPRECATED_EXPORT @API@ @PRE@_DEPRECATED +#endif + +#ifndef @PRE@_DEPRECATED_NO_EXPORT +# define @PRE@_DEPRECATED_NO_EXPORT @PRE@_NO_EXPORT @PRE@_DEPRECATED +#endif + +/* NOLINTNEXTLINE(readability-avoid-unconditional-preprocessor-if) */ +#if 0 /* DEFINE_NO_DEPRECATED */ +# ifndef @PRE@_NO_DEPRECATED +# define @PRE@_NO_DEPRECATED +# endif +#endif + +#endif /* @API@_H */ +""" + + +def tokens(lib): + """The three substitution tokens for a library, derived exactly as + Meta/CMake/targets.cmake derives them: fs_name is the library name minus its + "Lib" prefix, and EXPORT_MACRO_NAME is upper(fs_name) + "_API".""" + fs_name = lib[3:] if lib.startswith("Lib") else lib + return ("%s_API" % fs_name.upper(), lib.upper(), "%s_EXPORTS" % lib) + + +def render(lib): + api, pre, exp = tokens(lib) + # Substitute @EXP@ first: it is the only token whose replacement text + # ("LibGC_EXPORTS") contains a substring that a later pattern could match. + # Each replace() runs over the whole string, so an earlier substitution's + # OUTPUT is visible to a later one -- e.g. replacing @PRE@ with "LIBGC" then + # looking for "GC_API" is fine here (disjoint), but the ordering is load + # bearing in general and must not be permuted casually. + return TEMPLATE.replace("@EXP@", exp).replace("@API@", api).replace("@PRE@", pre) + + +def configure_file(text, defines): + """Apply CMake's `configure_file(... @ONLY)` substitution rules. + + Only the three directives Ladybird's two templates actually use, because a + partial re-implementation that silently ignores an unknown directive is how + you get a header that looks configured and is not. Anything unrecognized is + left alone and reported by `--check` as a byte mismatch. + + * `#cmakedefine01 X` -> `#define X 1` if X is truthy else `#define X 0` + * `#cmakedefine X` -> `#define X` if defined, else `/* #undef X */` + * `@VAR@` -> its value (@ONLY means ONLY @VAR@, not ${VAR}) + """ + def sub01(m): + pad, name = m.group(1), m.group(2) + return "#%sdefine %s %s" % (pad, name, "1" if defines.get(name) else "0") + + def subdef(m): + pad, name = m.group(1), m.group(2) + if defines.get(name): + return "#%sdefine %s" % (pad, name) + return "#%s/* #undef %s */" % (pad, name) + + # The `#` sits at column 0 and the indentation comes AFTER it + # (`# cmakedefine01 FOO`), which is how these templates are written -- so + # the padding to preserve is between the hash and the directive, not before + # the hash. Two earlier attempts got this wrong: `(\s*)#cmakedefine01` never + # matched, and --check caught it both times rather than emitting a header with + # the directive left verbatim. + text = re.sub(r"(?m)^#([ \t]*)cmakedefine01 (\w+)[ \t]*$", sub01, text) + text = re.sub(r"(?m)^#([ \t]*)cmakedefine (\w+)[ \t]*$", subdef, text) + text = re.sub(r"@(\w+)@", lambda m: str(defines.get(m.group(1), "")), text) + return text + + +# Debug.h: every `#cmakedefine01 *_DEBUG` is OFF in a default configure. Passing +# an empty define map is therefore the faithful reproduction, and --check proves +# it against the reference tree rather than trusting this comment. +DEBUG_H_DEFINES = {} + +# Backtrace.h is deliberately NOT emitted from a define map -- see the module +# docstring. The genrule below compiles a probe for and substitutes +# whatever THIS host answers, so the header is a question, not a pasted value. +BACKTRACE_PROBE_CMD = r""" +set -e +tmp=$$(mktemp -d) +printf '#include \nint main(){void*b[1];backtrace(b,1);return 0;}\n' > $$tmp/p.c +if $${CC:-cc} -o $$tmp/p $$tmp/p.c 2>/dev/null; then + found=1 +else + found=0 +fi +rm -rf $$tmp +if [ "$$found" = 1 ]; then + sed -e 's|^#cmakedefine Backtrace_FOUND$$|#define Backtrace_FOUND|' \ + -e 's|@Backtrace_HEADER@|execinfo.h|' $< > $@ +else + sed -e 's|^#cmakedefine Backtrace_FOUND$$|/* #undef Backtrace_FOUND */|' \ + -e 's|@Backtrace_HEADER@||' $< > $@ +fi +""" + + +# Libraries/LibWeb is its own Bazel PACKAGE, so the root package cannot declare +# an output inside it ("Label '//:Libraries/LibWeb/Export.h' is invalid because +# 'Libraries/LibWeb' is a subpackage"). Its Export.h is emitted into LibWeb's own +# package instead, via --libweb. Any other library that gains a BUILD.bazel will +# hit the same wall and belongs in this set. +SUBPACKAGE_LIBS = {"LibWeb"} + + +def root_libs(): + return [l for l in EXPORT_LIBS if l not in SUBPACKAGE_LIBS] + + +def heredoc(body): + """A shell heredoc that writes `body` EXACTLY, with no added trailing newline. + + `cat > $@ <<'EOF'\n\nEOF` appends a newline of its own, so a body that + already ends in "\n" gains a second one -- 1 byte of drift that `--check` + (which compares render() to the reference) could never see, because it is + introduced by the emitted SHELL, not by render(). It was caught only by + byte-comparing the BUILT artifact. Strip one trailing newline before wrapping, + since the heredoc puts it back. + """ + if body.endswith("\n"): + body = body[:-1] + esc = body.replace("\\", "\\\\").replace('"', '\\"').replace("\n", "\\n") + return "cat > $@ <<'LADYBIRD_EOF'\\n%s\\nLADYBIRD_EOF\\n" % esc + + +def emit_libweb(): + """The LibWeb-package half: its own Export.h genrule + header target.""" + lib = "LibWeb" + print("# AUTO-GENERATED by Meta/emit_export_headers_bazel.py --libweb — do not edit.") + print("# LibWeb's generate_export_header output. It lives here rather than in the") + print("# root package because Libraries/LibWeb is its own Bazel package, so only") + print("# this package may declare an output inside it.") + print('load("@rules_cc//cc:defs.bzl", "cc_library")') + print() + print("def libweb_export_header():") + # Emitted at genroot/LibWeb/Export.h, NOT at Export.h. All 289 consumers spell + # it , so the include root has to be a directory CONTAINING a + # "LibWeb" dir. Using the package dir itself would need includes=['../..'], + # which Bazel rejects outright ("resolves to the workspace root, which would + # allow this rule and all of its transitive dependents to include any file in + # your workspace") -- and it is right to: that would put the entire repo on + # every dependent's include path. A private genroot/ subdir keeps the exported + # path exactly while the include root stays inside this + # package. + print(" native.genrule(") + print(" name = 'gen_%s_Export_h'," % lib) + print(" outs = ['genroot/LibWeb/Export.h'],") + print(" cmd = \"%s\"," % heredoc(render(lib))) + print(" )") + print(" cc_library(") + print(" name = 'generated_export_header',") + print(" hdrs = ['genroot/LibWeb/Export.h'],") + print(" includes = ['genroot'],") + print(" )") + + +def emit(): + print("# AUTO-GENERATED by Meta/emit_export_headers_bazel.py — do not edit.") + print("# CMake's generate_export_header() output for the %d libraries that" + % len(EXPORT_LIBS)) + print("# opt into explicit symbol export, emitted as genrules so building the") + print("# browser needs no CMake build. Byte-verified against the reference") + print("# tree by `Meta/emit_export_headers_bazel.py --check Build/full`.") + print('load("@rules_cc//cc:defs.bzl", "cc_library")') + print() + print("def export_headers():") + for lib in root_libs(): + # A quoted heredoc ('LADYBIRD_EOF') so the shell interprets NOTHING in + # the body: it contains #, $, quotes, parentheses and backslashes. + print(" native.genrule(") + print(" name = 'gen_%s_Export_h'," % lib) + print(" outs = ['Libraries/%s/Export.h']," % lib) + print(" cmd = \"%s\"," % heredoc(render(lib))) + print(" )") + print() + # AK/Debug.h -- configure_file over a checked-in template, all flags off. + # Emitted by substituting here rather than shelling sed 79 times. + print(" # AK/Debug.h: configure_file(Debug.h.in) with every *_DEBUG off,") + print(" # which is what a default configure produces (--check verifies it).") + print(" native.genrule(") + print(" name = 'gen_AK_Debug_h',") + print(" srcs = ['AK/Debug.h.in'],") + print(" outs = ['genroot/AK/Debug.h'],") + debug_in = os.path.join("AK", "Debug.h.in") + if os.path.exists(debug_in): + with open(debug_in) as f: + body = configure_file(f.read(), DEBUG_H_DEFINES) + print(" cmd = \"%s\"," % heredoc(body)) + else: + print(" cmd = \"# AK/Debug.h.in not found at emit time\",") + print(" )") + # AK/Backtrace.h -- a host probe, asked at build time. See module docstring. + print(" # AK/Backtrace.h: find_package(Backtrace) is a HOST QUESTION, so the") + print(" # genrule compiles a probe instead of baking in this machine's answer.") + print(" native.genrule(") + print(" name = 'gen_AK_Backtrace_h',") + print(" srcs = ['AK/Backtrace.h.in'],") + print(" outs = ['genroot/AK/Backtrace.h'],") + print(" cmd = %r," % BACKTRACE_PROBE_CMD) + print(" )") + print() + # One header root exposing all 15 as /Export.h, mirroring the shim it + # replaces (//Build/full/Libraries:generated_lib_headers). + print(" cc_library(") + print(" name = 'generated_export_headers',") + print(" hdrs = [%s]," % ", ".join( + "'Libraries/%s/Export.h'" % lib for lib in root_libs())) + print(" includes = ['Libraries'],") + print(" )") + # AK/*.h arrive under genroot/ so resolves, mirroring the + # ak_gen_headers genrule in BUILD.bazel that copied them out of Build/full. + print(" cc_library(") + print(" name = 'generated_ak_headers',") + print(" hdrs = ['genroot/AK/Debug.h', 'genroot/AK/Backtrace.h'],") + print(" includes = ['genroot'],") + print(" )") + + +def check(tree): + """Byte-compare this emitter's output against the reference CMake tree.""" + bad = 0 + for lib in EXPORT_LIBS: + ref = os.path.join(tree, "Libraries", lib, "Export.h") + if not os.path.exists(ref): + print("MISSING reference: %s" % ref) + bad += 1 + continue + with open(ref) as f: + want = f.read() + got = render(lib) + if got != want: + print("MISMATCH %s" % lib) + # Show the first differing line so template drift is diagnosable. + for i, (a, b) in enumerate(zip(got.splitlines(), want.splitlines())): + if a != b: + print(" line %d: emitted %r" % (i + 1, a)) + print(" cmake %r" % b) + break + bad += 1 + print("%d/%d export headers byte-identical to %s" + % (len(EXPORT_LIBS) - bad, len(EXPORT_LIBS), tree)) + + # AK/Debug.h: the configure_file path, checked the same way. (Backtrace.h is + # deliberately excluded: its content is a function of the HOST, so comparing + # it to this machine's reference tree would only ever re-confirm this machine. + # The probe is verified by building, not by diffing.) + dbg_in, dbg_ref = "AK/Debug.h.in", os.path.join(tree, "AK", "Debug.h") + if os.path.exists(dbg_in) and os.path.exists(dbg_ref): + with open(dbg_in) as f: + got = configure_file(f.read(), DEBUG_H_DEFINES) + with open(dbg_ref) as f: + want = f.read() + if got == want: + print("AK/Debug.h byte-identical to %s" % dbg_ref) + else: + bad += 1 + print("MISMATCH AK/Debug.h") + for i, (a, b) in enumerate(zip(got.splitlines(), want.splitlines())): + if a != b: + print(" line %d: emitted %r" % (i + 1, a)) + print(" cmake %r" % b) + break + else: + print(" (differing line count: %d emitted vs %d cmake)" + % (len(got.splitlines()), len(want.splitlines()))) + else: + print("SKIP AK/Debug.h (template or reference missing)") + return 1 if bad else 0 + + +def main(): + ap = argparse.ArgumentParser() + ap.add_argument("--libweb", action="store_true", + help="emit the Libraries/LibWeb package half instead") + ap.add_argument("--check", metavar="BUILD_DIR", + help="byte-compare against a reference CMake build tree " + "(e.g. Build/full)") + args = ap.parse_args() + if args.check: + return check(args.check) + if args.libweb: + emit_libweb() + return 0 + emit() + return 0 + + +if __name__ == "__main__": + sys.exit(main()) diff --git a/examples/ladybird/workspace/Meta/emit_libweb_bazel.py b/examples/ladybird/workspace/Meta/emit_libweb_bazel.py index 963e5d8..335e32f 100644 --- a/examples/ladybird/workspace/Meta/emit_libweb_bazel.py +++ b/examples/ladybird/workspace/Meta/emit_libweb_bazel.py @@ -23,8 +23,13 @@ # See emit_build_bazel.py: the crates are Bazel-built now, and each crate is ONE # target (//:_lib) carrying its own archive and its own generated FFI # headers -- one for one with CMake's per-library edge, so LibWeb links the four -# crates it uses and nothing else. -RUST_LIB_FMT = "//:%s_lib" +# crates it uses and nothing else. Plus, for a crate whose header LibWeb includes +# with no directory, an implementation_deps edge -- IMPORTED from +# emit_build_bazel rather than reimplemented, so the two emitters cannot disagree +# about which crates those are (none of LibWeb's four today; the mechanism is here +# because "none today" is a measurement, not an invariant). +sys.path.insert(0, os.path.dirname(os.path.abspath(__file__))) +from emit_build_bazel import rust_dep_labels GLOBAL_DEFINES = { "USE_VULKAN=1", "ENABLE_COMPILETIME_FORMAT_CHECK", "USE_FONTCONFIG=1", @@ -43,9 +48,18 @@ PRELUDE = '''load("@rules_cc//cc:defs.bzl", "cc_library") load(":codegen.bzl", "libweb_codegen", "libweb_bindings_codegen") load(":generated_srcs.bzl", "LIBWEB_GENERATED_SRCS", "LIBWEB_GENERATED_HDRS") +load(":export_header.bzl", "libweb_export_header") package(default_visibility = ["//visibility:public"]) +# LibWeb's generate_export_header output. It is generated HERE rather than in the +# root package because Bazel include dirs cannot escape a package: LibWeb is its +# own package, so its Export.h has to be an output of this package (at +# genroot/LibWeb/Export.h, with includes=["genroot"] -- includes=["../.."] is +# rejected by Bazel, and rightly so). Emitted by +# Meta/emit_export_headers_bazel.py --libweb. +libweb_export_header() + libweb_codegen() libweb_bindings_codegen() @@ -204,7 +218,7 @@ def main(): flags = target_flags(t) incs = target_private_includes(t) - deps, sysdeps, unknown = [], [], [] + deps, impl_deps, sysdeps, unknown = [], [], [], [] for d in t.get("deps", []): nm = d["name"] if not d.get("external"): @@ -217,7 +231,9 @@ def main(): deps.append("//:%s" % tgt) continue if nm.endswith("_rust"): - deps.append(RUST_LIB_FMT % nm) + rdeps, rimpl = rust_dep_labels(nm) + deps.extend(rdeps) + impl_deps.extend(rimpl) elif nm in so or nm in ar: deps.append(VCPKG + ":" + nm) elif nm in SYSTEM_LIBS: @@ -270,6 +286,11 @@ def main(): out.append(" local_defines = %r," % defs) if copt_toks: out.append(" copts = [%s]," % ", ".join("%r" % x for x in copt_toks)) + # PRIVATE deps -- see rust_dep_labels() in emit_build_bazel.py for why an + # unprefixed FFI include dir must not propagate. + if impl_deps: + out.append(" implementation_deps = [%s]," % + ", ".join("%r" % x for x in sorted(set(impl_deps)))) out.append(" deps = [") for d in sorted(set(deps)): out.append(f" {d!r},") diff --git a/examples/ladybird/workspace/cargo.bzl b/examples/ladybird/workspace/cargo.bzl index ffa0322..7a85416 100644 --- a/examples/ladybird/workspace/cargo.bzl +++ b/examples/ladybird/workspace/cargo.bzl @@ -184,11 +184,20 @@ rust_sysroot = rule( # one crate's .rs files rebuilds one crate. # --------------------------------------------------------------------------- CargoCrateInfo = provider( - doc = "One built Rust staticlib plus the FFI headers its build script wrote.", + doc = "One built Rust crate plus the FFI headers its build script wrote.", fields = { - "archive": "The lib.a File.", + "archive": "The lib.a File, or None for a --bin crate, which " + + "has no archive to link: what its C++ consumer uses is the " + + "EXECUTABLE, spawned at run time (libwasm_cranelift).", "headers": "The generated FFI header Files.", - "include_dirs": "Dirs to put on the include path for those headers.", + "include_dirs": "Dirs to put on the include path for those headers. " + + "Exported through CcInfo, so TRANSITIVE -- only ever the " + + "ffi/ root, whose spellings are unique per crate.", + "bare_include_dir": "The ffi/ dir, for crates whose header is " + + "#included with no directory. Deliberately NOT in " + + "CcInfo: 8 crates ship a RustFFI.h, so this must stay " + + "LOCAL to the owning library (its own copts), the way " + + "CMake's per-library FFI_OUTPUT_DIR is. Empty otherwise.", }, ) @@ -280,17 +289,64 @@ def _cargo_crate_impl(ctx): }, use_default_shell_env = True, ) - include_dirs = [ - "%s/%s/%s" % (archive.root.path, ctx.label.package, ffi_root), - ] - if ctx.attr.ffi_prefix: - include_dirs.append(include_dirs[0] + "/" + ctx.attr.ffi_prefix) + # The include root is the directory that makes each header resolve at the + # spelling its consumers use, and NOTHING more. + # + # 8 of the 10 crates emit a header literally named RustFFI.h (CMake: + # `FFI_HEADER RustFFI.h`), so a shared, unprefixed root is AMBIGUOUS: with + # both /ffi dirs on one command line, a bare `#include ` + # binds to whichever -isystem came first. That is not hypothetical -- it was + # happening: LibRegex's compile received libunicode_rust/ffi BEFORE + # libregex_rust/ffi, so `` in RustRegex.h resolved to LibUnicode's + # header, and the build only worked because a leftover + # -IBuild/full/Libraries/LibRegex shadowed both. Removing the CMake tree + # exposed it as "'RustRegexFlags' has not been declared". + # + # CMake has no such ambiguity: FFI_OUTPUT_DIR defaults to + # CMAKE_CURRENT_BINARY_DIR, so a library sees ONLY its own crate's dir. The + # faithful translation is therefore the PREFIXED root only (ffi/LibRegex), so + # resolves and a bare does not resolve to + # someone else's header. The 4 TUs that spell it bare get the unprefixed dir + # too -- but only from their OWN crate, via ffi_bare_include below. + # BOTH roots are needed, and they are not interchangeable: + # ffi/ -> makes the PREFIXED spelling work + # (32 TUs; the header sits at ffi/LibUnicode/RustFFI.h) + # ffi// -> makes the BARE spelling work (4 TUs) + # Dropping ffi/ broke LibUnicode ("LibUnicode/RustFFI.h: No such file"), and + # exposing ffi/ for EVERY crate is what made the bare spelling ambiguous in + # the first place (8 crates all ship a RustFFI.h, so first -isystem wins). + # + # The resolution: every crate gets ffi/ (needed for its own prefixed spelling, + # and harmless because that spelling is unique per crate), but only the crates + # whose header is included bare ALSO get ffi//. That keeps exactly one + # unprefixed candidate per compile for a bare include -- the consuming + # library's own -- which is precisely the guarantee CMake gets from + # FFI_OUTPUT_DIR defaulting to the library's own binary dir. + # Only the PREFIXED-capable root goes in the provider, because CcInfo's + # system_includes propagate TRANSITIVELY: every dir here lands on every + # downstream library's command line. That is correct for ffi/ (the spelling + # is unique per crate) and wrong for ffi// (8 + # crates ship a RustFFI.h, so a bare would bind to whichever + # crate's dir sorted first). LibGfx proved it: it inherited LibRegex's and + # LibTextCodec's bare dirs through the dep graph and compiled against + # LibRegex's header ("'FFI' does not name a type"). + # + # So the bare-include dir is NOT exported here. It is published separately as + # `bare_include_dir` for the OWNING library to put on its own copts, where it + # is local and cannot leak downstream -- which is exactly the scope CMake's + # FFI_OUTPUT_DIR (the library's own binary dir) has. + ffi_base = "%s/%s/%s" % (archive.root.path, ctx.label.package, ffi_root) + include_dirs = [ffi_base] + bare_dir = "" + if ctx.attr.ffi_prefix and ctx.attr.ffi_bare_include: + bare_dir = ffi_base + "/" + ctx.attr.ffi_prefix return [ DefaultInfo(files = depset([archive] + headers)), CargoCrateInfo( archive = archive, headers = headers, include_dirs = include_dirs, + bare_include_dir = bare_dir, ), ] @@ -303,6 +359,13 @@ cargo_crate = rule( undeclared output is deleted by Bazel, which is how a header CMake never declared (HTMLTokenizerRustFFI.h) turned up.""", attrs = { + "ffi_bare_include": attr.bool( + default = False, + doc = "Deprecated-by-construction escape hatch: also expose this " + + "crate's unprefixed ffi dir, for the TUs that spell the " + + "header with no directory. Per-crate so it can " + + "never make two crates' headers collide.", + ), "crate": attr.string(mandatory = True, doc = "The cargo package name."), "manifest": attr.string( mandatory = True, @@ -363,19 +426,38 @@ cargo_crate = rule( ) def _cargo_binary_impl(ctx): - """A cargo `--bin` crate, e.g. flapc. + """A cargo `--bin` crate: flapc, and cranelift-compiler. Same action, different cargo subcommand, and the output is executable so a genrule can name it in `tools`. flapc's own 3-package workspace gets the identical treatment: its lock file has exactly one registry crate (smallvec, pinned `=1.15.1`, the same version and checksum as the big workspace's), so it needs no separate machinery at all. + + A binary crate can ALSO emit an FFI header, and the second one does: + libwasm_cranelift's build.rs runs cbindgen exactly as the staticlib crates' + do, and LibWasm's CraneliftBridge.cpp includes the result. So `ffi_headers` + is the same declared-output contract cargo_crate has, for the same reason (an + undeclared output is deleted by Bazel) -- what differs is only that there is + no archive: what C++ consumes from this crate is the *executable*, spawned at + run time via WASM_CRANELIFT_COMPILER_PATH. """ out = ctx.actions.declare_file(ctx.label.name) + ffi_root = "%s.ffi" % ctx.label.name + headers = [ + ctx.actions.declare_file("%s/%s/%s" % (ffi_root, ctx.attr.ffi_prefix, h)) + for h in ctx.attr.ffi_headers + ] + ffi_out = "" + if headers: + ffi_out = "%s/%s/%s" % (out.root.path, ctx.label.package, ffi_root) + if ctx.attr.ffi_prefix: + ffi_out += "/" + ctx.attr.ffi_prefix + index, crate_inputs = _crate_index(ctx) sysroot = ctx.file.sysroot ctx.actions.run( - outputs = [out], + outputs = [out] + headers, inputs = depset([index, sysroot, ctx.file._vendor] + ctx.files.srcs + crate_inputs), executable = ctx.executable._build, @@ -387,7 +469,8 @@ def _cargo_binary_impl(ctx): index.path, out.path, ctx.attr.bin, - ], + ffi_out, + ] + ctx.attr.ffi_headers, mnemonic = "CargoBinary", progress_message = "Building Rust binary %s (offline)" % ctx.attr.bin, execution_requirements = {"block-network": "1"}, @@ -399,21 +482,52 @@ def _cargo_binary_impl(ctx): }, use_default_shell_env = True, ) - return [DefaultInfo( - files = depset([out]), - executable = out, - runfiles = ctx.runfiles(files = [out]), - )] + ffi_base = "%s/%s/%s" % (out.root.path, ctx.label.package, ffi_root) + bare_dir = "" + if ctx.attr.ffi_prefix and ctx.attr.ffi_bare_include: + bare_dir = ffi_base + "/" + ctx.attr.ffi_prefix + return [ + DefaultInfo( + files = depset([out]), + executable = out, + runfiles = ctx.runfiles(files = [out]), + ), + # Carried even with no headers (flapc), so a consumer asking for this + # crate's headers gets an empty set rather than a missing provider. + CargoCrateInfo( + archive = None, + headers = headers, + include_dirs = [ffi_base] if headers else [], + bare_include_dir = bare_dir, + ), + ] cargo_binary = rule( implementation = _cargo_binary_impl, executable = True, - doc = "Build a cargo binary crate offline (flapc), runnable as a genrule tool.", + doc = """Build a cargo binary crate offline, runnable as a genrule tool. + + Two of them: flapc (a pure tool) and cranelift-compiler (a tool Ladybird + SPAWNS at run time, which additionally emits the FFI header LibWasm + includes).""", attrs = { "crate": attr.string(mandatory = True), "bin": attr.string(mandatory = True, doc = "The --bin name."), "manifest": attr.string(mandatory = True), "crate_features": attr.string_list(), + "ffi_headers": attr.string_list( + doc = "Headers this crate's build script writes, relative to " + + "ffi_prefix. Declared so Bazel keeps them.", + ), + "ffi_prefix": attr.string( + doc = "Directory the headers are staged under, so the include " + + "spelling resolves.", + ), + "ffi_bare_include": attr.bool( + doc = "Whether the owning library includes the header with no " + + "directory. Derived by scanning the source, see " + + "emit_cargo_bazel.crates_included_bare().", + ), "srcs": attr.label_list(allow_files = True), "crates": attr.string_keyed_label_dict(allow_files = True), "sysroot": attr.label(allow_single_file = True, mandatory = True), @@ -478,16 +592,29 @@ cargo_binary = rule( def _cargo_lib_impl(ctx): info = ctx.attr.crate[CargoCrateInfo] archive = info.archive + compilation = cc_common.create_compilation_context( + headers = depset(info.headers), + # -isystem, not -I: these are cbindgen output, and Ladybird's + # -Werror should not fire inside generated code (the same reason + # the vcpkg headers are -isystem). + system_includes = depset(info.include_dirs), + ) + if archive == None: + # A --bin crate (libwasm_cranelift): headers to include, NOTHING to link. + # The binary itself is not a link input at all -- LibWasm SPAWNS it, so it + # belongs in the consumer's data/runfiles, not in its linking context. + # Handled here rather than with a second rule so a binary crate that emits + # a header is consumed with exactly the same two labels a staticlib crate + # is (//:_lib and //:_bare_include), and the ring emitter + # needs no special case. + return [ + DefaultInfo(files = depset(info.headers)), + CcInfo(compilation_context = compilation), + ] return [ DefaultInfo(files = depset([archive] + info.headers)), CcInfo( - compilation_context = cc_common.create_compilation_context( - headers = depset(info.headers), - # -isystem, not -I: these are cbindgen output, and Ladybird's - # -Werror should not fire inside generated code (the same reason - # the vcpkg headers are -isystem). - system_includes = depset(info.include_dirs), - ), + compilation_context = compilation, linking_context = cc_common.create_linking_context( linker_inputs = depset([cc_common.create_linker_input( owner = ctx.label, @@ -506,6 +633,44 @@ def _cargo_lib_impl(ctx): ), ] +def _cargo_bare_include_impl(ctx): + """A CcInfo carrying ONLY the crate's ffi/ dir. + + Depended on by the ONE library that owns the crate, so its TUs can spell the + header `` with no directory -- the spelling CMake allows because + FFI_OUTPUT_DIR is the library's own binary dir. + + This is a separate target rather than part of cargo_lib because CcInfo's + system_includes propagate transitively: folded into cargo_lib, the dir would + reach every downstream library and a bare would bind to whichever + of the 8 crates' dirs came first on the command line. As its own target it is + added by exactly one library and travels no further, because nothing depends + on that library's copts. Verified by removal: with this folded into cargo_lib, + LibGfx compiled against LibRegex's header ('FFI' does not name a type). + """ + info = ctx.attr.crate[CargoCrateInfo] + if not info.bare_include_dir: + fail("cargo_bare_include on a crate with no bare_include_dir: %s" % + ctx.attr.crate.label) + return [ + DefaultInfo(files = depset(info.headers)), + CcInfo( + compilation_context = cc_common.create_compilation_context( + headers = depset(info.headers), + system_includes = depset([info.bare_include_dir]), + ), + ), + ] + +cargo_bare_include = rule( + implementation = _cargo_bare_include_impl, + doc = "The unprefixed FFI include dir for the crate's OWNING library only.", + attrs = { + "crate": attr.label(providers = [CargoCrateInfo], mandatory = True, + cfg = "exec"), + }, +) + cargo_lib = rule( implementation = _cargo_lib_impl, doc = """One crate: its archive to link, its generated FFI headers to include. diff --git a/examples/ladybird/workspace/cargo_index.bzl b/examples/ladybird/workspace/cargo_index.bzl index c6e7e8d..a0214ea 100644 --- a/examples/ladybird/workspace/cargo_index.bzl +++ b/examples/ladybird/workspace/cargo_index.bzl @@ -177,60 +177,70 @@ CARGO_CRATE_SPECS = { "features": [], "ffi_headers": ['RustFFI.h'], "ffi_prefix": 'LibGfx', + "ffi_bare_include": True, }, 'libjs_rust': { "manifest": 'Libraries/LibJS/Rust/Cargo.toml', "features": [], "ffi_headers": ['RustFFI.h'], "ffi_prefix": 'LibJS', + "ffi_bare_include": False, }, 'libregex_rust': { "manifest": 'Libraries/LibRegex/Rust/Cargo.toml', "features": ['allocator'], "ffi_headers": ['RustFFI.h'], "ffi_prefix": 'LibRegex', + "ffi_bare_include": True, }, 'libtextcodec_rust': { "manifest": 'Libraries/LibTextCodec/Rust/Cargo.toml', "features": [], "ffi_headers": ['RustFFI.h'], "ffi_prefix": 'LibTextCodec', + "ffi_bare_include": True, }, 'libunicode_rust': { "manifest": 'Libraries/LibUnicode/Rust/Cargo.toml', "features": ['allocator'], "ffi_headers": ['RustFFI.h'], "ffi_prefix": 'LibUnicode', + "ffi_bare_include": False, }, 'liburl_rust': { "manifest": 'Libraries/LibURL/Rust/Cargo.toml', "features": ['allocator'], "ffi_headers": ['RustFFI.h'], "ffi_prefix": 'LibURL', + "ffi_bare_include": False, }, 'libweb_content_blocker_rust': { "manifest": 'Libraries/LibWeb/ContentBlocker/Rust/Cargo.toml', "features": ['allocator'], "ffi_headers": ['ContentBlockerRustFFI.h'], "ffi_prefix": 'LibWeb', + "ffi_bare_include": False, }, 'libweb_css_rust': { "manifest": 'Libraries/LibWeb/CSS/Rust/Cargo.toml', "features": ['allocator'], "ffi_headers": ['ComputedValuesRustFFI.h', 'RustFFI.h', 'SelectorRustFFI.h', 'StyleValueRustFFI.h'], "ffi_prefix": 'LibWeb', + "ffi_bare_include": False, }, 'libweb_layout_rust': { "manifest": 'Libraries/LibWeb/Layout/Rust/Cargo.toml', "features": ['allocator'], "ffi_headers": ['Layout/TreeBuilderRustFFI.h'], "ffi_prefix": 'LibWeb', + "ffi_bare_include": False, }, 'libweb_rust': { "manifest": 'Libraries/LibWeb/Rust/Cargo.toml', "features": [], "ffi_headers": ['HTML/Parser/RustFFI.h', 'HTMLTokenizerRustFFI.h'], "ffi_prefix": 'LibWeb', + "ffi_bare_include": False, }, } diff --git a/examples/ladybird/workspace/cargo_ring.bzl b/examples/ladybird/workspace/cargo_ring.bzl index 2ae7229..c22ae4a 100644 --- a/examples/ladybird/workspace/cargo_ring.bzl +++ b/examples/ladybird/workspace/cargo_ring.bzl @@ -1,17 +1,17 @@ # AUTO-GENERATED by Meta/emit_cargo_bazel.py — do not edit. -load(":cargo.bzl", "cargo_binary", "cargo_crate", "cargo_lib", "rust_sysroot") +load(":cargo.bzl", "cargo_binary", "cargo_crate", "cargo_lib", "cargo_bare_include", "rust_sysroot") load(":cargo_index.bzl", "CARGO_CRATE_FILES", "CARGO_CRATE_SPECS") -# Ladybird's 10 production Rust crates and flapc, BUILT BY BAZEL from the -# 154 crates.io crates Bazel fetched -- replacing the prebuilt 260 MB -# librust_combined.a that was copied out of Build/full/cargo/, the `ar -M` -# merge that produced it (README step 1b), and the reference build's flapc -# binary. Nothing here names Build/full. +# Ladybird's 10 production Rust crates and 2 binary crates, BUILT BY BAZEL +# from the 154 crates.io crates Bazel fetched -- replacing the prebuilt +# 260 MB librust_combined.a that was copied out of Build/full/cargo/, the +# `ar -M` merge that produced it (README step 1b), and the reference build's +# flapc and cranelift-compiler binaries. Nothing here names Build/full. # # Every attribute comes from CARGO_CRATE_SPECS, which -# Meta/emit_cargo_bazel.py parses out of import_rust_crate() in CMake, so a -# feature or header list cannot drift from the reference build without the -# emitter's --report saying so. +# Meta/emit_cargo_bazel.py parses out of import_rust_crate() and +# build_rust_binary() in CMake, so a feature or header list cannot drift from +# the reference build without the emitter's --report saying so. def cargo_ring(): # The pinned toolchain (rust-toolchain.toml says 1.96.1), assembled from @@ -42,6 +42,7 @@ def cargo_ring(): crate_features = CARGO_CRATE_SPECS['libgfx_rust']["features"], ffi_headers = CARGO_CRATE_SPECS['libgfx_rust']["ffi_headers"], ffi_prefix = CARGO_CRATE_SPECS['libgfx_rust']["ffi_prefix"], + ffi_bare_include = CARGO_CRATE_SPECS['libgfx_rust']["ffi_bare_include"], manifest = CARGO_CRATE_SPECS['libgfx_rust']["manifest"], srcs = crate_srcs, sysroot = ":rust_sysroot", @@ -60,6 +61,10 @@ def cargo_ring(): name = 'libgfx_rust_lib', crate = ':libgfx_rust', ) + cargo_bare_include( + name = 'libgfx_rust_bare_include', + crate = ':libgfx_rust', + ) cargo_crate( name = 'libjs_rust', @@ -68,6 +73,7 @@ def cargo_ring(): crate_features = CARGO_CRATE_SPECS['libjs_rust']["features"], ffi_headers = CARGO_CRATE_SPECS['libjs_rust']["ffi_headers"], ffi_prefix = CARGO_CRATE_SPECS['libjs_rust']["ffi_prefix"], + ffi_bare_include = CARGO_CRATE_SPECS['libjs_rust']["ffi_bare_include"], manifest = CARGO_CRATE_SPECS['libjs_rust']["manifest"], # Build-script inputs, taken from the reference build's DEPFILES # rather than from reading build.rs: this crate's build script @@ -97,6 +103,7 @@ def cargo_ring(): crate_features = CARGO_CRATE_SPECS['libregex_rust']["features"], ffi_headers = CARGO_CRATE_SPECS['libregex_rust']["ffi_headers"], ffi_prefix = CARGO_CRATE_SPECS['libregex_rust']["ffi_prefix"], + ffi_bare_include = CARGO_CRATE_SPECS['libregex_rust']["ffi_bare_include"], manifest = CARGO_CRATE_SPECS['libregex_rust']["manifest"], srcs = crate_srcs, sysroot = ":rust_sysroot", @@ -115,6 +122,10 @@ def cargo_ring(): name = 'libregex_rust_lib', crate = ':libregex_rust', ) + cargo_bare_include( + name = 'libregex_rust_bare_include', + crate = ':libregex_rust', + ) cargo_crate( name = 'libtextcodec_rust', @@ -123,6 +134,7 @@ def cargo_ring(): crate_features = CARGO_CRATE_SPECS['libtextcodec_rust']["features"], ffi_headers = CARGO_CRATE_SPECS['libtextcodec_rust']["ffi_headers"], ffi_prefix = CARGO_CRATE_SPECS['libtextcodec_rust']["ffi_prefix"], + ffi_bare_include = CARGO_CRATE_SPECS['libtextcodec_rust']["ffi_bare_include"], manifest = CARGO_CRATE_SPECS['libtextcodec_rust']["manifest"], srcs = crate_srcs, sysroot = ":rust_sysroot", @@ -141,6 +153,10 @@ def cargo_ring(): name = 'libtextcodec_rust_lib', crate = ':libtextcodec_rust', ) + cargo_bare_include( + name = 'libtextcodec_rust_bare_include', + crate = ':libtextcodec_rust', + ) cargo_crate( name = 'libunicode_rust', @@ -149,6 +165,7 @@ def cargo_ring(): crate_features = CARGO_CRATE_SPECS['libunicode_rust']["features"], ffi_headers = CARGO_CRATE_SPECS['libunicode_rust']["ffi_headers"], ffi_prefix = CARGO_CRATE_SPECS['libunicode_rust']["ffi_prefix"], + ffi_bare_include = CARGO_CRATE_SPECS['libunicode_rust']["ffi_bare_include"], manifest = CARGO_CRATE_SPECS['libunicode_rust']["manifest"], srcs = crate_srcs, sysroot = ":rust_sysroot", @@ -175,6 +192,7 @@ def cargo_ring(): crate_features = CARGO_CRATE_SPECS['liburl_rust']["features"], ffi_headers = CARGO_CRATE_SPECS['liburl_rust']["ffi_headers"], ffi_prefix = CARGO_CRATE_SPECS['liburl_rust']["ffi_prefix"], + ffi_bare_include = CARGO_CRATE_SPECS['liburl_rust']["ffi_bare_include"], manifest = CARGO_CRATE_SPECS['liburl_rust']["manifest"], srcs = crate_srcs, sysroot = ":rust_sysroot", @@ -201,6 +219,7 @@ def cargo_ring(): crate_features = CARGO_CRATE_SPECS['libweb_content_blocker_rust']["features"], ffi_headers = CARGO_CRATE_SPECS['libweb_content_blocker_rust']["ffi_headers"], ffi_prefix = CARGO_CRATE_SPECS['libweb_content_blocker_rust']["ffi_prefix"], + ffi_bare_include = CARGO_CRATE_SPECS['libweb_content_blocker_rust']["ffi_bare_include"], manifest = CARGO_CRATE_SPECS['libweb_content_blocker_rust']["manifest"], srcs = crate_srcs, sysroot = ":rust_sysroot", @@ -227,6 +246,7 @@ def cargo_ring(): crate_features = CARGO_CRATE_SPECS['libweb_css_rust']["features"], ffi_headers = CARGO_CRATE_SPECS['libweb_css_rust']["ffi_headers"], ffi_prefix = CARGO_CRATE_SPECS['libweb_css_rust']["ffi_prefix"], + ffi_bare_include = CARGO_CRATE_SPECS['libweb_css_rust']["ffi_bare_include"], manifest = CARGO_CRATE_SPECS['libweb_css_rust']["manifest"], srcs = crate_srcs, sysroot = ":rust_sysroot", @@ -253,6 +273,7 @@ def cargo_ring(): crate_features = CARGO_CRATE_SPECS['libweb_layout_rust']["features"], ffi_headers = CARGO_CRATE_SPECS['libweb_layout_rust']["ffi_headers"], ffi_prefix = CARGO_CRATE_SPECS['libweb_layout_rust']["ffi_prefix"], + ffi_bare_include = CARGO_CRATE_SPECS['libweb_layout_rust']["ffi_bare_include"], manifest = CARGO_CRATE_SPECS['libweb_layout_rust']["manifest"], srcs = crate_srcs, sysroot = ":rust_sysroot", @@ -279,6 +300,7 @@ def cargo_ring(): crate_features = CARGO_CRATE_SPECS['libweb_rust']["features"], ffi_headers = CARGO_CRATE_SPECS['libweb_rust']["ffi_headers"], ffi_prefix = CARGO_CRATE_SPECS['libweb_rust']["ffi_prefix"], + ffi_bare_include = CARGO_CRATE_SPECS['libweb_rust']["ffi_bare_include"], manifest = CARGO_CRATE_SPECS['libweb_rust']["manifest"], srcs = crate_srcs, sysroot = ":rust_sysroot", @@ -298,17 +320,63 @@ def cargo_ring(): crate = ':libweb_rust', ) - # flapc, the Flap-DSL -> interpreter-assembly compiler. Its workspace is - # `exclude`d from the root one and has its own lock with exactly 3 - # packages (flapc, in-tree bytecode_def, and smallvec from crates.io - # pinned =1.15.1 -- the same version AND checksum the big workspace pins, - # so it is the same fetch rule, not a second one). + # The BINARY crates: cargo `--bin` targets, declared in CMake with + # build_rust_binary() rather than import_rust_crate(). Two of them, and + # they are two different shapes: + # + # flapc a pure build TOOL -- Bazel runs it in a genrule to + # produce the interpreter assembly. Its workspace is + # `exclude`d from the root one and has its own lock + # with exactly 3 packages (flapc, in-tree + # bytecode_def, and smallvec from crates.io pinned + # =1.15.1 -- the same version AND checksum the big + # workspace pins, so it is the same fetch rule, not a + # second one). + # cranelift-compiler a RUNTIME tool -- LibWasm spawns it to AOT-compile + # WebAssembly, and its build script ALSO emits the + # CraneliftFFI.h that LibWasm's CraneliftBridge.cpp + # includes. It is a root-workspace member, so its + # source set is the shared crate_srcs. cargo_binary( - name = "flapc", - bin = "flapc", - crate = "flapc", + name = 'flapc', + bin = 'flapc', + crate = 'flapc', crates = CARGO_CRATE_FILES, - manifest = "Libraries/LibJS/Flap/Cargo.toml", + manifest = 'Libraries/LibJS/Flap/Cargo.toml', + # Its OWN workspace (`exclude`d from the root one), so its source + # set is its own subtree. srcs = native.glob(['Libraries/LibJS/Flap/Cargo.toml', 'Libraries/LibJS/Flap/Cargo.lock', 'Libraries/LibJS/Flap/src/**', 'Libraries/LibJS/Flap/benches/**', 'Libraries/LibJS/Flap/tests/**', 'Libraries/LibJS/BytecodeDef/**', 'Libraries/LibJS/Bytecode/Bytecode.def', 'Libraries/LibJS/Interpreter/interpreter.flap', 'rust-toolchain.toml'], allow_empty = False), sysroot = ":rust_sysroot", ) + cargo_binary( + name = 'cranelift-compiler', + bin = 'cranelift-compiler', + crate = 'libwasm_cranelift', + crates = CARGO_CRATE_FILES, + manifest = 'Libraries/LibWasm/Rust/Cargo.toml', + # This binary crate's build script runs cbindgen too, so the + # header is a DECLARED output -- Bazel deletes what nothing + # declares, and LibWasm includes this one bare. + ffi_headers = ['CraneliftFFI.h'], + ffi_prefix = 'LibWasm', + ffi_bare_include = True, + # A member of the ROOT cargo workspace (Cargo.toml lists it), so + # cargo resolves the whole workspace whichever crate you ask for + # and the source set is the shared one -- same over-declaration, + # same reason, as the staticlib crates. + srcs = crate_srcs + ['Libraries/LibWasm/Opcode.h'], + sysroot = ":rust_sysroot", + ) + + # The consumable target, IDENTICAL in shape to a staticlib crate's: + # this crate's generated header to include. There is no archive to + # link -- what C++ consumes from a binary crate is the EXECUTABLE, + # spawned at run time -- so cargo_lib yields a headers-only CcInfo. + cargo_lib( + name = 'libwasm_cranelift_lib', + crate = ':cranelift-compiler', + ) + cargo_bare_include( + name = 'libwasm_cranelift_bare_include', + crate = ':cranelift-compiler', + ) diff --git a/examples/ladybird/workspace/export_headers.bzl b/examples/ladybird/workspace/export_headers.bzl new file mode 100644 index 0000000..1870746 --- /dev/null +++ b/examples/ladybird/workspace/export_headers.bzl @@ -0,0 +1,106 @@ +# AUTO-GENERATED by Meta/emit_export_headers_bazel.py — do not edit. +# CMake's generate_export_header() output for the 15 libraries that +# opt into explicit symbol export, emitted as genrules so building the +# browser needs no CMake build. Byte-verified against the reference +# tree by `Meta/emit_export_headers_bazel.py --check Build/full`. +load("@rules_cc//cc:defs.bzl", "cc_library") + +def export_headers(): + native.genrule( + name = 'gen_LibCore_Export_h', + outs = ['Libraries/LibCore/Export.h'], + cmd = "cat > $@ <<'LADYBIRD_EOF'\n\n#ifndef CORE_API_H\n#define CORE_API_H\n\n#ifdef LIBCORE_STATIC_DEFINE\n# define CORE_API\n# define LIBCORE_NO_EXPORT\n#else\n# ifndef CORE_API\n# ifdef LibCore_EXPORTS\n /* We are building this library */\n# define CORE_API __attribute__((visibility(\"default\")))\n# else\n /* We are using this library */\n# define CORE_API __attribute__((visibility(\"default\")))\n# endif\n# endif\n\n# ifndef LIBCORE_NO_EXPORT\n# define LIBCORE_NO_EXPORT __attribute__((visibility(\"hidden\")))\n# endif\n#endif\n\n#ifndef LIBCORE_DEPRECATED\n# define LIBCORE_DEPRECATED __attribute__ ((__deprecated__))\n#endif\n\n#ifndef LIBCORE_DEPRECATED_EXPORT\n# define LIBCORE_DEPRECATED_EXPORT CORE_API LIBCORE_DEPRECATED\n#endif\n\n#ifndef LIBCORE_DEPRECATED_NO_EXPORT\n# define LIBCORE_DEPRECATED_NO_EXPORT LIBCORE_NO_EXPORT LIBCORE_DEPRECATED\n#endif\n\n/* NOLINTNEXTLINE(readability-avoid-unconditional-preprocessor-if) */\n#if 0 /* DEFINE_NO_DEPRECATED */\n# ifndef LIBCORE_NO_DEPRECATED\n# define LIBCORE_NO_DEPRECATED\n# endif\n#endif\n\n#endif /* CORE_API_H */\nLADYBIRD_EOF\n", + ) + native.genrule( + name = 'gen_LibDNS_Export_h', + outs = ['Libraries/LibDNS/Export.h'], + cmd = "cat > $@ <<'LADYBIRD_EOF'\n\n#ifndef DNS_API_H\n#define DNS_API_H\n\n#ifdef LIBDNS_STATIC_DEFINE\n# define DNS_API\n# define LIBDNS_NO_EXPORT\n#else\n# ifndef DNS_API\n# ifdef LibDNS_EXPORTS\n /* We are building this library */\n# define DNS_API __attribute__((visibility(\"default\")))\n# else\n /* We are using this library */\n# define DNS_API __attribute__((visibility(\"default\")))\n# endif\n# endif\n\n# ifndef LIBDNS_NO_EXPORT\n# define LIBDNS_NO_EXPORT __attribute__((visibility(\"hidden\")))\n# endif\n#endif\n\n#ifndef LIBDNS_DEPRECATED\n# define LIBDNS_DEPRECATED __attribute__ ((__deprecated__))\n#endif\n\n#ifndef LIBDNS_DEPRECATED_EXPORT\n# define LIBDNS_DEPRECATED_EXPORT DNS_API LIBDNS_DEPRECATED\n#endif\n\n#ifndef LIBDNS_DEPRECATED_NO_EXPORT\n# define LIBDNS_DEPRECATED_NO_EXPORT LIBDNS_NO_EXPORT LIBDNS_DEPRECATED\n#endif\n\n/* NOLINTNEXTLINE(readability-avoid-unconditional-preprocessor-if) */\n#if 0 /* DEFINE_NO_DEPRECATED */\n# ifndef LIBDNS_NO_DEPRECATED\n# define LIBDNS_NO_DEPRECATED\n# endif\n#endif\n\n#endif /* DNS_API_H */\nLADYBIRD_EOF\n", + ) + native.genrule( + name = 'gen_LibDatabase_Export_h', + outs = ['Libraries/LibDatabase/Export.h'], + cmd = "cat > $@ <<'LADYBIRD_EOF'\n\n#ifndef DATABASE_API_H\n#define DATABASE_API_H\n\n#ifdef LIBDATABASE_STATIC_DEFINE\n# define DATABASE_API\n# define LIBDATABASE_NO_EXPORT\n#else\n# ifndef DATABASE_API\n# ifdef LibDatabase_EXPORTS\n /* We are building this library */\n# define DATABASE_API __attribute__((visibility(\"default\")))\n# else\n /* We are using this library */\n# define DATABASE_API __attribute__((visibility(\"default\")))\n# endif\n# endif\n\n# ifndef LIBDATABASE_NO_EXPORT\n# define LIBDATABASE_NO_EXPORT __attribute__((visibility(\"hidden\")))\n# endif\n#endif\n\n#ifndef LIBDATABASE_DEPRECATED\n# define LIBDATABASE_DEPRECATED __attribute__ ((__deprecated__))\n#endif\n\n#ifndef LIBDATABASE_DEPRECATED_EXPORT\n# define LIBDATABASE_DEPRECATED_EXPORT DATABASE_API LIBDATABASE_DEPRECATED\n#endif\n\n#ifndef LIBDATABASE_DEPRECATED_NO_EXPORT\n# define LIBDATABASE_DEPRECATED_NO_EXPORT LIBDATABASE_NO_EXPORT LIBDATABASE_DEPRECATED\n#endif\n\n/* NOLINTNEXTLINE(readability-avoid-unconditional-preprocessor-if) */\n#if 0 /* DEFINE_NO_DEPRECATED */\n# ifndef LIBDATABASE_NO_DEPRECATED\n# define LIBDATABASE_NO_DEPRECATED\n# endif\n#endif\n\n#endif /* DATABASE_API_H */\nLADYBIRD_EOF\n", + ) + native.genrule( + name = 'gen_LibDevTools_Export_h', + outs = ['Libraries/LibDevTools/Export.h'], + cmd = "cat > $@ <<'LADYBIRD_EOF'\n\n#ifndef DEVTOOLS_API_H\n#define DEVTOOLS_API_H\n\n#ifdef LIBDEVTOOLS_STATIC_DEFINE\n# define DEVTOOLS_API\n# define LIBDEVTOOLS_NO_EXPORT\n#else\n# ifndef DEVTOOLS_API\n# ifdef LibDevTools_EXPORTS\n /* We are building this library */\n# define DEVTOOLS_API __attribute__((visibility(\"default\")))\n# else\n /* We are using this library */\n# define DEVTOOLS_API __attribute__((visibility(\"default\")))\n# endif\n# endif\n\n# ifndef LIBDEVTOOLS_NO_EXPORT\n# define LIBDEVTOOLS_NO_EXPORT __attribute__((visibility(\"hidden\")))\n# endif\n#endif\n\n#ifndef LIBDEVTOOLS_DEPRECATED\n# define LIBDEVTOOLS_DEPRECATED __attribute__ ((__deprecated__))\n#endif\n\n#ifndef LIBDEVTOOLS_DEPRECATED_EXPORT\n# define LIBDEVTOOLS_DEPRECATED_EXPORT DEVTOOLS_API LIBDEVTOOLS_DEPRECATED\n#endif\n\n#ifndef LIBDEVTOOLS_DEPRECATED_NO_EXPORT\n# define LIBDEVTOOLS_DEPRECATED_NO_EXPORT LIBDEVTOOLS_NO_EXPORT LIBDEVTOOLS_DEPRECATED\n#endif\n\n/* NOLINTNEXTLINE(readability-avoid-unconditional-preprocessor-if) */\n#if 0 /* DEFINE_NO_DEPRECATED */\n# ifndef LIBDEVTOOLS_NO_DEPRECATED\n# define LIBDEVTOOLS_NO_DEPRECATED\n# endif\n#endif\n\n#endif /* DEVTOOLS_API_H */\nLADYBIRD_EOF\n", + ) + native.genrule( + name = 'gen_LibGC_Export_h', + outs = ['Libraries/LibGC/Export.h'], + cmd = "cat > $@ <<'LADYBIRD_EOF'\n\n#ifndef GC_API_H\n#define GC_API_H\n\n#ifdef LIBGC_STATIC_DEFINE\n# define GC_API\n# define LIBGC_NO_EXPORT\n#else\n# ifndef GC_API\n# ifdef LibGC_EXPORTS\n /* We are building this library */\n# define GC_API __attribute__((visibility(\"default\")))\n# else\n /* We are using this library */\n# define GC_API __attribute__((visibility(\"default\")))\n# endif\n# endif\n\n# ifndef LIBGC_NO_EXPORT\n# define LIBGC_NO_EXPORT __attribute__((visibility(\"hidden\")))\n# endif\n#endif\n\n#ifndef LIBGC_DEPRECATED\n# define LIBGC_DEPRECATED __attribute__ ((__deprecated__))\n#endif\n\n#ifndef LIBGC_DEPRECATED_EXPORT\n# define LIBGC_DEPRECATED_EXPORT GC_API LIBGC_DEPRECATED\n#endif\n\n#ifndef LIBGC_DEPRECATED_NO_EXPORT\n# define LIBGC_DEPRECATED_NO_EXPORT LIBGC_NO_EXPORT LIBGC_DEPRECATED\n#endif\n\n/* NOLINTNEXTLINE(readability-avoid-unconditional-preprocessor-if) */\n#if 0 /* DEFINE_NO_DEPRECATED */\n# ifndef LIBGC_NO_DEPRECATED\n# define LIBGC_NO_DEPRECATED\n# endif\n#endif\n\n#endif /* GC_API_H */\nLADYBIRD_EOF\n", + ) + native.genrule( + name = 'gen_LibJS_Export_h', + outs = ['Libraries/LibJS/Export.h'], + cmd = "cat > $@ <<'LADYBIRD_EOF'\n\n#ifndef JS_API_H\n#define JS_API_H\n\n#ifdef LIBJS_STATIC_DEFINE\n# define JS_API\n# define LIBJS_NO_EXPORT\n#else\n# ifndef JS_API\n# ifdef LibJS_EXPORTS\n /* We are building this library */\n# define JS_API __attribute__((visibility(\"default\")))\n# else\n /* We are using this library */\n# define JS_API __attribute__((visibility(\"default\")))\n# endif\n# endif\n\n# ifndef LIBJS_NO_EXPORT\n# define LIBJS_NO_EXPORT __attribute__((visibility(\"hidden\")))\n# endif\n#endif\n\n#ifndef LIBJS_DEPRECATED\n# define LIBJS_DEPRECATED __attribute__ ((__deprecated__))\n#endif\n\n#ifndef LIBJS_DEPRECATED_EXPORT\n# define LIBJS_DEPRECATED_EXPORT JS_API LIBJS_DEPRECATED\n#endif\n\n#ifndef LIBJS_DEPRECATED_NO_EXPORT\n# define LIBJS_DEPRECATED_NO_EXPORT LIBJS_NO_EXPORT LIBJS_DEPRECATED\n#endif\n\n/* NOLINTNEXTLINE(readability-avoid-unconditional-preprocessor-if) */\n#if 0 /* DEFINE_NO_DEPRECATED */\n# ifndef LIBJS_NO_DEPRECATED\n# define LIBJS_NO_DEPRECATED\n# endif\n#endif\n\n#endif /* JS_API_H */\nLADYBIRD_EOF\n", + ) + native.genrule( + name = 'gen_LibMedia_Export_h', + outs = ['Libraries/LibMedia/Export.h'], + cmd = "cat > $@ <<'LADYBIRD_EOF'\n\n#ifndef MEDIA_API_H\n#define MEDIA_API_H\n\n#ifdef LIBMEDIA_STATIC_DEFINE\n# define MEDIA_API\n# define LIBMEDIA_NO_EXPORT\n#else\n# ifndef MEDIA_API\n# ifdef LibMedia_EXPORTS\n /* We are building this library */\n# define MEDIA_API __attribute__((visibility(\"default\")))\n# else\n /* We are using this library */\n# define MEDIA_API __attribute__((visibility(\"default\")))\n# endif\n# endif\n\n# ifndef LIBMEDIA_NO_EXPORT\n# define LIBMEDIA_NO_EXPORT __attribute__((visibility(\"hidden\")))\n# endif\n#endif\n\n#ifndef LIBMEDIA_DEPRECATED\n# define LIBMEDIA_DEPRECATED __attribute__ ((__deprecated__))\n#endif\n\n#ifndef LIBMEDIA_DEPRECATED_EXPORT\n# define LIBMEDIA_DEPRECATED_EXPORT MEDIA_API LIBMEDIA_DEPRECATED\n#endif\n\n#ifndef LIBMEDIA_DEPRECATED_NO_EXPORT\n# define LIBMEDIA_DEPRECATED_NO_EXPORT LIBMEDIA_NO_EXPORT LIBMEDIA_DEPRECATED\n#endif\n\n/* NOLINTNEXTLINE(readability-avoid-unconditional-preprocessor-if) */\n#if 0 /* DEFINE_NO_DEPRECATED */\n# ifndef LIBMEDIA_NO_DEPRECATED\n# define LIBMEDIA_NO_DEPRECATED\n# endif\n#endif\n\n#endif /* MEDIA_API_H */\nLADYBIRD_EOF\n", + ) + native.genrule( + name = 'gen_LibRegex_Export_h', + outs = ['Libraries/LibRegex/Export.h'], + cmd = "cat > $@ <<'LADYBIRD_EOF'\n\n#ifndef REGEX_API_H\n#define REGEX_API_H\n\n#ifdef LIBREGEX_STATIC_DEFINE\n# define REGEX_API\n# define LIBREGEX_NO_EXPORT\n#else\n# ifndef REGEX_API\n# ifdef LibRegex_EXPORTS\n /* We are building this library */\n# define REGEX_API __attribute__((visibility(\"default\")))\n# else\n /* We are using this library */\n# define REGEX_API __attribute__((visibility(\"default\")))\n# endif\n# endif\n\n# ifndef LIBREGEX_NO_EXPORT\n# define LIBREGEX_NO_EXPORT __attribute__((visibility(\"hidden\")))\n# endif\n#endif\n\n#ifndef LIBREGEX_DEPRECATED\n# define LIBREGEX_DEPRECATED __attribute__ ((__deprecated__))\n#endif\n\n#ifndef LIBREGEX_DEPRECATED_EXPORT\n# define LIBREGEX_DEPRECATED_EXPORT REGEX_API LIBREGEX_DEPRECATED\n#endif\n\n#ifndef LIBREGEX_DEPRECATED_NO_EXPORT\n# define LIBREGEX_DEPRECATED_NO_EXPORT LIBREGEX_NO_EXPORT LIBREGEX_DEPRECATED\n#endif\n\n/* NOLINTNEXTLINE(readability-avoid-unconditional-preprocessor-if) */\n#if 0 /* DEFINE_NO_DEPRECATED */\n# ifndef LIBREGEX_NO_DEPRECATED\n# define LIBREGEX_NO_DEPRECATED\n# endif\n#endif\n\n#endif /* REGEX_API_H */\nLADYBIRD_EOF\n", + ) + native.genrule( + name = 'gen_LibSync_Export_h', + outs = ['Libraries/LibSync/Export.h'], + cmd = "cat > $@ <<'LADYBIRD_EOF'\n\n#ifndef SYNC_API_H\n#define SYNC_API_H\n\n#ifdef LIBSYNC_STATIC_DEFINE\n# define SYNC_API\n# define LIBSYNC_NO_EXPORT\n#else\n# ifndef SYNC_API\n# ifdef LibSync_EXPORTS\n /* We are building this library */\n# define SYNC_API __attribute__((visibility(\"default\")))\n# else\n /* We are using this library */\n# define SYNC_API __attribute__((visibility(\"default\")))\n# endif\n# endif\n\n# ifndef LIBSYNC_NO_EXPORT\n# define LIBSYNC_NO_EXPORT __attribute__((visibility(\"hidden\")))\n# endif\n#endif\n\n#ifndef LIBSYNC_DEPRECATED\n# define LIBSYNC_DEPRECATED __attribute__ ((__deprecated__))\n#endif\n\n#ifndef LIBSYNC_DEPRECATED_EXPORT\n# define LIBSYNC_DEPRECATED_EXPORT SYNC_API LIBSYNC_DEPRECATED\n#endif\n\n#ifndef LIBSYNC_DEPRECATED_NO_EXPORT\n# define LIBSYNC_DEPRECATED_NO_EXPORT LIBSYNC_NO_EXPORT LIBSYNC_DEPRECATED\n#endif\n\n/* NOLINTNEXTLINE(readability-avoid-unconditional-preprocessor-if) */\n#if 0 /* DEFINE_NO_DEPRECATED */\n# ifndef LIBSYNC_NO_DEPRECATED\n# define LIBSYNC_NO_DEPRECATED\n# endif\n#endif\n\n#endif /* SYNC_API_H */\nLADYBIRD_EOF\n", + ) + native.genrule( + name = 'gen_LibTest_Export_h', + outs = ['Libraries/LibTest/Export.h'], + cmd = "cat > $@ <<'LADYBIRD_EOF'\n\n#ifndef TEST_API_H\n#define TEST_API_H\n\n#ifdef LIBTEST_STATIC_DEFINE\n# define TEST_API\n# define LIBTEST_NO_EXPORT\n#else\n# ifndef TEST_API\n# ifdef LibTest_EXPORTS\n /* We are building this library */\n# define TEST_API __attribute__((visibility(\"default\")))\n# else\n /* We are using this library */\n# define TEST_API __attribute__((visibility(\"default\")))\n# endif\n# endif\n\n# ifndef LIBTEST_NO_EXPORT\n# define LIBTEST_NO_EXPORT __attribute__((visibility(\"hidden\")))\n# endif\n#endif\n\n#ifndef LIBTEST_DEPRECATED\n# define LIBTEST_DEPRECATED __attribute__ ((__deprecated__))\n#endif\n\n#ifndef LIBTEST_DEPRECATED_EXPORT\n# define LIBTEST_DEPRECATED_EXPORT TEST_API LIBTEST_DEPRECATED\n#endif\n\n#ifndef LIBTEST_DEPRECATED_NO_EXPORT\n# define LIBTEST_DEPRECATED_NO_EXPORT LIBTEST_NO_EXPORT LIBTEST_DEPRECATED\n#endif\n\n/* NOLINTNEXTLINE(readability-avoid-unconditional-preprocessor-if) */\n#if 0 /* DEFINE_NO_DEPRECATED */\n# ifndef LIBTEST_NO_DEPRECATED\n# define LIBTEST_NO_DEPRECATED\n# endif\n#endif\n\n#endif /* TEST_API_H */\nLADYBIRD_EOF\n", + ) + native.genrule( + name = 'gen_LibTextCodec_Export_h', + outs = ['Libraries/LibTextCodec/Export.h'], + cmd = "cat > $@ <<'LADYBIRD_EOF'\n\n#ifndef TEXTCODEC_API_H\n#define TEXTCODEC_API_H\n\n#ifdef LIBTEXTCODEC_STATIC_DEFINE\n# define TEXTCODEC_API\n# define LIBTEXTCODEC_NO_EXPORT\n#else\n# ifndef TEXTCODEC_API\n# ifdef LibTextCodec_EXPORTS\n /* We are building this library */\n# define TEXTCODEC_API __attribute__((visibility(\"default\")))\n# else\n /* We are using this library */\n# define TEXTCODEC_API __attribute__((visibility(\"default\")))\n# endif\n# endif\n\n# ifndef LIBTEXTCODEC_NO_EXPORT\n# define LIBTEXTCODEC_NO_EXPORT __attribute__((visibility(\"hidden\")))\n# endif\n#endif\n\n#ifndef LIBTEXTCODEC_DEPRECATED\n# define LIBTEXTCODEC_DEPRECATED __attribute__ ((__deprecated__))\n#endif\n\n#ifndef LIBTEXTCODEC_DEPRECATED_EXPORT\n# define LIBTEXTCODEC_DEPRECATED_EXPORT TEXTCODEC_API LIBTEXTCODEC_DEPRECATED\n#endif\n\n#ifndef LIBTEXTCODEC_DEPRECATED_NO_EXPORT\n# define LIBTEXTCODEC_DEPRECATED_NO_EXPORT LIBTEXTCODEC_NO_EXPORT LIBTEXTCODEC_DEPRECATED\n#endif\n\n/* NOLINTNEXTLINE(readability-avoid-unconditional-preprocessor-if) */\n#if 0 /* DEFINE_NO_DEPRECATED */\n# ifndef LIBTEXTCODEC_NO_DEPRECATED\n# define LIBTEXTCODEC_NO_DEPRECATED\n# endif\n#endif\n\n#endif /* TEXTCODEC_API_H */\nLADYBIRD_EOF\n", + ) + native.genrule( + name = 'gen_LibWasm_Export_h', + outs = ['Libraries/LibWasm/Export.h'], + cmd = "cat > $@ <<'LADYBIRD_EOF'\n\n#ifndef WASM_API_H\n#define WASM_API_H\n\n#ifdef LIBWASM_STATIC_DEFINE\n# define WASM_API\n# define LIBWASM_NO_EXPORT\n#else\n# ifndef WASM_API\n# ifdef LibWasm_EXPORTS\n /* We are building this library */\n# define WASM_API __attribute__((visibility(\"default\")))\n# else\n /* We are using this library */\n# define WASM_API __attribute__((visibility(\"default\")))\n# endif\n# endif\n\n# ifndef LIBWASM_NO_EXPORT\n# define LIBWASM_NO_EXPORT __attribute__((visibility(\"hidden\")))\n# endif\n#endif\n\n#ifndef LIBWASM_DEPRECATED\n# define LIBWASM_DEPRECATED __attribute__ ((__deprecated__))\n#endif\n\n#ifndef LIBWASM_DEPRECATED_EXPORT\n# define LIBWASM_DEPRECATED_EXPORT WASM_API LIBWASM_DEPRECATED\n#endif\n\n#ifndef LIBWASM_DEPRECATED_NO_EXPORT\n# define LIBWASM_DEPRECATED_NO_EXPORT LIBWASM_NO_EXPORT LIBWASM_DEPRECATED\n#endif\n\n/* NOLINTNEXTLINE(readability-avoid-unconditional-preprocessor-if) */\n#if 0 /* DEFINE_NO_DEPRECATED */\n# ifndef LIBWASM_NO_DEPRECATED\n# define LIBWASM_NO_DEPRECATED\n# endif\n#endif\n\n#endif /* WASM_API_H */\nLADYBIRD_EOF\n", + ) + native.genrule( + name = 'gen_LibWebView_Export_h', + outs = ['Libraries/LibWebView/Export.h'], + cmd = "cat > $@ <<'LADYBIRD_EOF'\n\n#ifndef WEBVIEW_API_H\n#define WEBVIEW_API_H\n\n#ifdef LIBWEBVIEW_STATIC_DEFINE\n# define WEBVIEW_API\n# define LIBWEBVIEW_NO_EXPORT\n#else\n# ifndef WEBVIEW_API\n# ifdef LibWebView_EXPORTS\n /* We are building this library */\n# define WEBVIEW_API __attribute__((visibility(\"default\")))\n# else\n /* We are using this library */\n# define WEBVIEW_API __attribute__((visibility(\"default\")))\n# endif\n# endif\n\n# ifndef LIBWEBVIEW_NO_EXPORT\n# define LIBWEBVIEW_NO_EXPORT __attribute__((visibility(\"hidden\")))\n# endif\n#endif\n\n#ifndef LIBWEBVIEW_DEPRECATED\n# define LIBWEBVIEW_DEPRECATED __attribute__ ((__deprecated__))\n#endif\n\n#ifndef LIBWEBVIEW_DEPRECATED_EXPORT\n# define LIBWEBVIEW_DEPRECATED_EXPORT WEBVIEW_API LIBWEBVIEW_DEPRECATED\n#endif\n\n#ifndef LIBWEBVIEW_DEPRECATED_NO_EXPORT\n# define LIBWEBVIEW_DEPRECATED_NO_EXPORT LIBWEBVIEW_NO_EXPORT LIBWEBVIEW_DEPRECATED\n#endif\n\n/* NOLINTNEXTLINE(readability-avoid-unconditional-preprocessor-if) */\n#if 0 /* DEFINE_NO_DEPRECATED */\n# ifndef LIBWEBVIEW_NO_DEPRECATED\n# define LIBWEBVIEW_NO_DEPRECATED\n# endif\n#endif\n\n#endif /* WEBVIEW_API_H */\nLADYBIRD_EOF\n", + ) + native.genrule( + name = 'gen_LibXML_Export_h', + outs = ['Libraries/LibXML/Export.h'], + cmd = "cat > $@ <<'LADYBIRD_EOF'\n\n#ifndef XML_API_H\n#define XML_API_H\n\n#ifdef LIBXML_STATIC_DEFINE\n# define XML_API\n# define LIBXML_NO_EXPORT\n#else\n# ifndef XML_API\n# ifdef LibXML_EXPORTS\n /* We are building this library */\n# define XML_API __attribute__((visibility(\"default\")))\n# else\n /* We are using this library */\n# define XML_API __attribute__((visibility(\"default\")))\n# endif\n# endif\n\n# ifndef LIBXML_NO_EXPORT\n# define LIBXML_NO_EXPORT __attribute__((visibility(\"hidden\")))\n# endif\n#endif\n\n#ifndef LIBXML_DEPRECATED\n# define LIBXML_DEPRECATED __attribute__ ((__deprecated__))\n#endif\n\n#ifndef LIBXML_DEPRECATED_EXPORT\n# define LIBXML_DEPRECATED_EXPORT XML_API LIBXML_DEPRECATED\n#endif\n\n#ifndef LIBXML_DEPRECATED_NO_EXPORT\n# define LIBXML_DEPRECATED_NO_EXPORT LIBXML_NO_EXPORT LIBXML_DEPRECATED\n#endif\n\n/* NOLINTNEXTLINE(readability-avoid-unconditional-preprocessor-if) */\n#if 0 /* DEFINE_NO_DEPRECATED */\n# ifndef LIBXML_NO_DEPRECATED\n# define LIBXML_NO_DEPRECATED\n# endif\n#endif\n\n#endif /* XML_API_H */\nLADYBIRD_EOF\n", + ) + + # AK/Debug.h: configure_file(Debug.h.in) with every *_DEBUG off, + # which is what a default configure produces (--check verifies it). + native.genrule( + name = 'gen_AK_Debug_h', + srcs = ['AK/Debug.h.in'], + outs = ['genroot/AK/Debug.h'], + cmd = "cat > $@ <<'LADYBIRD_EOF'\n/*\n * Copyright (c) 2020-2024, the SerenityOS developers.\n *\n * SPDX-License-Identifier: BSD-2-Clause\n */\n\n#pragma once\n\n#ifndef AK_STRINGBASE_VERIFY_LAUNDER_DEBUG\n# define AK_STRINGBASE_VERIFY_LAUNDER_DEBUG 0\n#endif\n\n#ifndef AUDIO_DEBUG\n# define AUDIO_DEBUG 0\n#endif\n\n#ifndef BMP_DEBUG\n# define BMP_DEBUG 0\n#endif\n\n#ifndef CACHE_DEBUG\n# define CACHE_DEBUG 0\n#endif\n\n#ifndef CALLBACK_MACHINE_DEBUG\n# define CALLBACK_MACHINE_DEBUG 0\n#endif\n\n#ifndef CANVAS_RENDERING_CONTEXT_2D_DEBUG\n# define CANVAS_RENDERING_CONTEXT_2D_DEBUG 0\n#endif\n\n#ifndef CRYPTO_DEBUG\n# define CRYPTO_DEBUG 0\n#endif\n\n#ifndef COMPOSITOR_DEBUG\n# define COMPOSITOR_DEBUG 0\n#endif\n\n#ifndef CSS_LOADER_DEBUG\n# define CSS_LOADER_DEBUG 0\n#endif\n\n#ifndef CSS_PARSER_DEBUG\n# define CSS_PARSER_DEBUG 0\n#endif\n\n#ifndef CSS_TOKENIZER_DEBUG\n# define CSS_TOKENIZER_DEBUG 0\n#endif\n\n#ifndef CSS_TRANSITIONS_DEBUG\n# define CSS_TRANSITIONS_DEBUG 0\n#endif\n\n#ifndef CURL_DEBUG\n# define CURL_DEBUG 0\n#endif\n\n#ifndef DEVTOOLS_DEBUG\n# define DEVTOOLS_DEBUG 0\n#endif\n\n#ifndef DNS_DEBUG\n# define DNS_DEBUG 0\n#endif\n\n#ifndef EDITOR_DEBUG\n# define EDITOR_DEBUG 0\n#endif\n\n#ifndef FILE_WATCHER_DEBUG\n# define FILE_WATCHER_DEBUG 0\n#endif\n\n#ifndef FLAC_ENCODER_DEBUG\n# define FLAC_ENCODER_DEBUG 0\n#endif\n\n#ifndef FORMATTING_CONTEXT_TRACE_DEBUG\n# define FORMATTING_CONTEXT_TRACE_DEBUG 0\n#endif\n\n#ifndef GIF_DEBUG\n# define GIF_DEBUG 0\n#endif\n\n#ifndef HEAP_DEBUG\n# define HEAP_DEBUG 0\n#endif\n\n#ifndef INCREMENTAL_SWEEP_DEBUG\n# define INCREMENTAL_SWEEP_DEBUG 0\n#endif\n\n#ifndef HIGHLIGHT_FOCUSED_FRAME_DEBUG\n# define HIGHLIGHT_FOCUSED_FRAME_DEBUG 0\n#endif\n\n#ifndef HTML_SCRIPT_DEBUG\n# define HTML_SCRIPT_DEBUG 0\n#endif\n\n#ifndef HTTP_DISK_CACHE_DEBUG\n# define HTTP_DISK_CACHE_DEBUG 0\n#endif\n\n#ifndef HTTP_MEMORY_CACHE_DEBUG\n# define HTTP_MEMORY_CACHE_DEBUG 0\n#endif\n\n#ifndef HTTPJOB_DEBUG\n# define HTTPJOB_DEBUG 0\n#endif\n\n#ifndef ICO_DEBUG\n# define ICO_DEBUG 0\n#endif\n\n#ifndef IDB_DEBUG\n# define IDB_DEBUG 0\n#endif\n\n#ifndef IDL_DEBUG\n# define IDL_DEBUG 0\n#endif\n\n#ifndef IMAGE_DECODER_DEBUG\n# define IMAGE_DECODER_DEBUG 0\n#endif\n\n#ifndef IMAGE_LOADER_DEBUG\n# define IMAGE_LOADER_DEBUG 0\n#endif\n\n#ifndef JS_MODULE_DEBUG\n# define JS_MODULE_DEBUG 0\n#endif\n\n#ifndef LEXER_DEBUG\n# define LEXER_DEBUG 0\n#endif\n\n#ifndef LIBWEB_CSS_ANIMATION_DEBUG\n# define LIBWEB_CSS_ANIMATION_DEBUG 0\n#endif\n\n#ifndef LIBWEB_CSS_DEBUG\n# define LIBWEB_CSS_DEBUG 0\n#endif\n\n#ifndef LIBWEB_WASM_DEBUG\n# define LIBWEB_WASM_DEBUG 0\n#endif\n\n#ifndef LINE_EDITOR_DEBUG\n# define LINE_EDITOR_DEBUG 0\n#endif\n\n#ifndef WEBVIEW_HISTORY_DEBUG\n# define WEBVIEW_HISTORY_DEBUG 0\n#endif\n\n#ifndef LZW_DEBUG\n# define LZW_DEBUG 0\n#endif\n\n#ifndef MACH_PORT_DEBUG\n# define MACH_PORT_DEBUG 0\n#endif\n\n#ifndef MATROSKA_DEBUG\n# define MATROSKA_DEBUG 0\n#endif\n\n#ifndef MATROSKA_TRACE_DEBUG\n# define MATROSKA_TRACE_DEBUG 0\n#endif\n\n#ifndef HTML_PARSER_DEBUG\n# define HTML_PARSER_DEBUG 0\n#endif\n\n#ifndef PATH_DEBUG\n# define PATH_DEBUG 0\n#endif\n\n#ifndef PLAYBACK_MANAGER_DEBUG\n# define PLAYBACK_MANAGER_DEBUG 0\n#endif\n\n#ifndef PNG_DEBUG\n# define PNG_DEBUG 0\n#endif\n\n#ifndef PROMISE_DEBUG\n# define PROMISE_DEBUG 0\n#endif\n\n#ifndef REGEX_DEBUG\n# define REGEX_DEBUG 0\n#endif\n\n#ifndef REQUESTSERVER_DEBUG\n# define REQUESTSERVER_DEBUG 0\n#endif\n\n#ifndef REQUESTSERVER_WIRE_DEBUG\n# define REQUESTSERVER_WIRE_DEBUG 0\n#endif\n\n#ifndef RESOURCE_DEBUG\n# define RESOURCE_DEBUG 0\n#endif\n\n#ifndef SPAM_DEBUG\n# define SPAM_DEBUG 0\n#endif\n\n#ifndef STRUCTURED_SERIALIZE_DEBUG\n# define STRUCTURED_SERIALIZE_DEBUG 0\n#endif\n\n#ifndef SYNTAX_HIGHLIGHTING_DEBUG\n# define SYNTAX_HIGHLIGHTING_DEBUG 0\n#endif\n\n#ifndef STYLE_INVALIDATION_DEBUG\n# define STYLE_INVALIDATION_DEBUG 0\n#endif\n\n#ifndef TEXTEDITOR_DEBUG\n# define TEXTEDITOR_DEBUG 0\n#endif\n\n#ifndef TIFF_DEBUG\n# define TIFF_DEBUG 0\n#endif\n\n#ifndef TLS_DEBUG\n# define TLS_DEBUG 0\n#endif\n\n#ifndef TOKENIZER_TRACE_DEBUG\n# define TOKENIZER_TRACE_DEBUG 0\n#endif\n\n#ifndef UPDATE_LAYOUT_DEBUG\n# define UPDATE_LAYOUT_DEBUG 0\n#endif\n\n#ifndef URL_PARSER_DEBUG\n# define URL_PARSER_DEBUG 0\n#endif\n\n#ifndef URL_PATTERN_DEBUG\n# define URL_PATTERN_DEBUG 0\n#endif\n\n#ifndef UTF8_DEBUG\n# define UTF8_DEBUG 0\n#endif\n\n#ifndef VULKAN_VALIDATION_LAYERS_DEBUG\n# define VULKAN_VALIDATION_LAYERS_DEBUG 0\n#endif\n\n#ifndef WASI_DEBUG\n# define WASI_DEBUG 0\n#endif\n\n#ifndef WASI_FINE_GRAINED_DEBUG\n# define WASI_FINE_GRAINED_DEBUG 0\n#endif\n\n#ifndef WASM_BINPARSER_DEBUG\n# define WASM_BINPARSER_DEBUG 0\n#endif\n\n#ifndef WASM_CRANELIFT_DEBUG\n# define WASM_CRANELIFT_DEBUG 0\n#endif\n\n#ifndef WASM_TRACE_DEBUG\n# define WASM_TRACE_DEBUG 0\n#endif\n\n#ifndef WASM_VALIDATOR_DEBUG\n# define WASM_VALIDATOR_DEBUG 0\n#endif\n\n#ifndef WEBDRIVER_DEBUG\n# define WEBDRIVER_DEBUG 0\n#endif\n\n#ifndef WEBDRIVER_ROUTE_DEBUG\n# define WEBDRIVER_ROUTE_DEBUG 0\n#endif\n\n#ifndef WEBGL_CONTEXT_DEBUG\n# define WEBGL_CONTEXT_DEBUG 0\n#endif\n\n#ifndef WEBVIEW_PROCESS_DEBUG\n# define WEBVIEW_PROCESS_DEBUG 0\n#endif\n\n#ifndef WEB_FETCH_DEBUG\n# define WEB_FETCH_DEBUG 0\n#endif\n\n#ifndef WEB_WORKER_DEBUG\n# define WEB_WORKER_DEBUG 0\n#endif\n\n#ifndef WEBP_DEBUG\n# define WEBP_DEBUG 0\n#endif\n\n#ifndef XML_PARSER_DEBUG\n# define XML_PARSER_DEBUG 0\n#endif\nLADYBIRD_EOF\n", + ) + # AK/Backtrace.h: find_package(Backtrace) is a HOST QUESTION, so the + # genrule compiles a probe instead of baking in this machine's answer. + native.genrule( + name = 'gen_AK_Backtrace_h', + srcs = ['AK/Backtrace.h.in'], + outs = ['genroot/AK/Backtrace.h'], + cmd = '\nset -e\ntmp=$$(mktemp -d)\nprintf \'#include \\nint main(){void*b[1];backtrace(b,1);return 0;}\\n\' > $$tmp/p.c\nif $${CC:-cc} -o $$tmp/p $$tmp/p.c 2>/dev/null; then\n found=1\nelse\n found=0\nfi\nrm -rf $$tmp\nif [ "$$found" = 1 ]; then\n sed -e \'s|^#cmakedefine Backtrace_FOUND$$|#define Backtrace_FOUND|\' \\\n -e \'s|@Backtrace_HEADER@|execinfo.h|\' $< > $@\nelse\n sed -e \'s|^#cmakedefine Backtrace_FOUND$$|/* #undef Backtrace_FOUND */|\' \\\n -e \'s|@Backtrace_HEADER@||\' $< > $@\nfi\n', + ) + + cc_library( + name = 'generated_export_headers', + hdrs = ['Libraries/LibCore/Export.h', 'Libraries/LibDNS/Export.h', 'Libraries/LibDatabase/Export.h', 'Libraries/LibDevTools/Export.h', 'Libraries/LibGC/Export.h', 'Libraries/LibJS/Export.h', 'Libraries/LibMedia/Export.h', 'Libraries/LibRegex/Export.h', 'Libraries/LibSync/Export.h', 'Libraries/LibTest/Export.h', 'Libraries/LibTextCodec/Export.h', 'Libraries/LibWasm/Export.h', 'Libraries/LibWebView/Export.h', 'Libraries/LibXML/Export.h'], + includes = ['Libraries'], + ) + cc_library( + name = 'generated_ak_headers', + hdrs = ['genroot/AK/Debug.h', 'genroot/AK/Backtrace.h'], + includes = ['genroot'], + ) From fb54ced19eae004199465de9531ea0d955d21a0e Mon Sep 17 00:00:00 2001 From: Ulf Adams Date: Fri, 7 Aug 2026 21:20:49 +0000 Subject: [PATCH 03/59] Ladybird: delete the Build/full shims entirely, and stop emitting 7 dead -IBuild/full copts The three Build/full/**/BUILD.bazel shim packages are gone, and with them the last thing the emitted build read out of CMake's tree. By the end they supplied nothing: of the 709 headers they globbed, 21 are Bazel genrule outputs and the other 688 are LibWeb bindings headers Bazel ALSO generates (all 692 are in LIBWEB_GENERATED_HDRS, checked). So the roots were SHADOWING Bazel's own outputs, winning or losing on include order. That is why they lasted: glob(["**/*.h"], allow_empty = True) over a tree that is not there yields an empty list and no diagnostic. A shim that cannot fail is indistinguishable from a shim that is not needed. Also six of the seven per-target -IBuild/full copts were an un-normalized path: CMake emits a target's own binary dir relative to itself, so the five services and WebContent got -IBuild/full/Services/WebContent/../.. -- the same directory as the global -IBuild/full, spelled differently, so the string comparison against globalroots missed it. target_private_includes() now normpaths before comparing. The seventh (ladybird's -IBuild/full/UI{,/Qt}) pointed at CMake's UI gendir, whose only non-autogen contents are the two SPIR-V shader headers Bazel generates itself and carries on :generated_shader_headers. Verified by removal: shim packages deleted AND Build/full/{Libraries,Services,UI} moved off the machine (0 Ladybird-generated headers left under Build/full -- the 3,532 remaining are the vcpkg tree, a separately closed ring). All six binaries rebuilt from scratch, 2,704 actions, and all 6 render comparisons -- --headless=text and --headless=layout-tree over three pages, two of which execute WebAssembly -- are byte-identical to the CMake reference. README: the headline clone-and-build claim is now true and says how it was false, the Layout table and gap 4 reflect the deleted shims, and finding 35 documents the two bugs the CMake tree had been masking (the RustFFI.h collision needing implementation_deps, and Cranelift being absent from the graph behind two host escapes). --- examples/ladybird/README.md | 127 ++++++++++++++---- examples/ladybird/workspace/BUILD.bazel | 35 ++--- .../Build/full/Libraries/BUILD.bazel | 29 ---- .../workspace/Build/full/Services/BUILD.bazel | 15 --- .../workspace/Build/full/UI/BUILD.bazel | 14 -- .../workspace/Meta/emit_build_bazel.py | 50 +++++-- 6 files changed, 160 insertions(+), 110 deletions(-) delete mode 100644 examples/ladybird/workspace/Build/full/Libraries/BUILD.bazel delete mode 100644 examples/ladybird/workspace/Build/full/Services/BUILD.bazel delete mode 100644 examples/ladybird/workspace/Build/full/UI/BUILD.bazel diff --git a/examples/ladybird/README.md b/examples/ladybird/README.md index 29dbac5..816b5b9 100644 --- a/examples/ladybird/README.md +++ b/examples/ladybird/README.md @@ -6,25 +6,38 @@ each of these files looks the way it does, and the 34 findings the migration produced — is in [`docs/CASE-ladybird-migration.md`](../../docs/CASE-ladybird-migration.md). This directory is the *artifact*: what you would drop into a Ladybird checkout. -**What it achieves today:** all five processes (`ladybird` UI, WebContent, -Compositor, RequestServer, ImageDecoder) are Bazel-built, all 51 code +**What it achieves today:** all six processes (`ladybird` UI, WebContent, +WebWorker, Compositor, RequestServer, ImageDecoder) are Bazel-built, all 51 code generators run under Bazel with output **byte-identical to CMake's** (1,408/1,408 files, checked by `Meta/bazel_parity_harness.py`, which accounts for every one of the build's 586 ninja `CUSTOM_COMMAND`s — 51 covered, 535 excluded with a stated reason, **0 unhandled**), the result renders pages (`--headless=text` and `--headless=layout-tree` match the CMake reference byte for byte), **the 77 vcpkg dependencies are fetched and built by Bazel with -zero network access** (findings 30–33), and **so are the 10 Rust crates and -`flapc`** — 154 crates.io crates fetched from `Cargo.lock`, built by -network-blocked cargo actions (finding 34). - -**`git clone && bazel build //:ladybird` now builds the browser.** Verified by -removal, twice: with `Build/full/vcpkg_installed` gone, and then with -`Build/full/cargo` **and** `Build/full/bin/flapc` gone, a `bazel clean` build of -all six binaries renders `--headless=text` and `--headless=layout-tree` +zero network access** (findings 30–33), and **so are the 10 Rust crates, `flapc` +and `cranelift-compiler`** — 154 crates.io crates fetched from `Cargo.lock`, +built by network-blocked cargo actions (findings 34–35). + +**`git clone && bazel build //:ladybird` builds the browser, and nothing in the +emitted build reads CMake's build tree.** Verified by removal: with +`Build/full/{Libraries,Services,UI,bin,cargo}` moved off the machine — every +Ladybird-generated header and every binary CMake produced — all six binaries +build from scratch and render `--headless=text` **and** `--headless=layout-tree` byte-identically to the CMake reference. `Build/full` is still needed to *regenerate* the BUILD files and to run the parity harness — a -converter-development dependency, not a build dependency. See +converter-development dependency, not a build dependency. + +**This claim was false until recently, and the way it was false is the most +useful thing in this directory.** The README said it worked; it did not. Every +binary depended on **741 targets under `Build/full`**, the overlay shipped four +`BUILD.bazel` shims and *zero* headers, and `glob(..., allow_empty = True)` meant +a fresh clone got **no error at all** — just `fatal error: LibXML/Export.h: No +such file or directory` some 1,600 actions in. A glob over a foreign tree cannot +fail, so nothing ever said the tree was missing. Of the 741, **666 were merely +*shadowing* Bazel's own outputs** (the LibWeb bindings headers — Bazel generates +all 692, and the shim silently won or lost on include order); 31 were real gaps; +and closing them exposed **two genuine bugs the CMake tree had been masking** — +an FFI header collision and an entire missing Rust target. See [Known gaps](#known-gaps) for what is still owed. ## Layout @@ -50,7 +63,8 @@ converter-development dependency, not a build dependency. See | `cargo_crates.bzl`, `cargo_index.bzl`, `cargo_extension.bzl` | One `http_archive` per crates.io crate (154), the name/version/sha256 index the vendor staging resolves through, and the module extension that creates the repos + the 3 pinned Rust toolchain components. **Generated** by `Meta/emit_cargo_bazel.py` **from `Cargo.lock` alone** — no cargo, no network, no CMake | | `cargo.bzl`, `Meta/cargo_build.sh`, `Meta/cargo_binary_build.sh`, `Meta/cargo_vendor.sh` | `rust_sysroot` (the pinned 1.96.1 toolchain merged into one tree), `cargo_crate` / `cargo_binary` (offline, network-blocked build actions), and `cargo_lib` (one consumable `CcInfo` per crate: its archive + its FFI headers) | | `cargo_ring.bzl` | The 10 `cargo_crate` + 10 `cargo_lib` targets and `flapc`, as a macro for the root package (the crate sources are at the repo root and `glob()` is package-relative). **Generated** by `Meta/emit_cargo_bazel.py` | -| `Build/full/**/BUILD.bazel` | Shims over the *reference CMake build tree*: **only generated headers now.** The Rust archive and `flapc` shims are gone (finding 34), as are the vcpkg ones (finding 33) | +| `export_headers.bzl`, `Libraries/LibWeb/export_header.bzl` | The 15 `generate_export_header` `Export.h` files + AK's two `configure_file` headers, generated by Bazel from the same inputs CMake uses. **Generated** by `Meta/emit_export_headers_bazel.py`; `--check Build/full` byte-compares every one against CMake's | +| ~~`Build/full/**/BUILD.bazel`~~ | **Gone.** These shimmed the reference CMake build tree, and by the end they supplied *nothing*: the 709 headers they globbed were 21 Bazel generates and 688 LibWeb bindings headers Bazel also generates. Deleting them is what turned "the README claims a clone-and-build" into one | The four generated files are reproducible: re-running each emitter against the same `Build/full` reproduces them byte for byte — **on the same machine**. They @@ -70,9 +84,11 @@ cp -r .../examples/ladybird/workspace/. . && mv bazelrc.txt .bazelrc git apply .../examples/ladybird/patches/*.patch # generator determinism + a Qt header # that is not self-contained (gap 7) # 2. Build. This includes the 77 vcpkg ports (~45 min cold) and the 10 Rust -# crates + flapc: the libraries depend on them through //Meta/vcpkg: and -# //:_lib, so there is no separate step. Build //:vcpkg_installed alone -# if you want to time it. +# crates + flapc + cranelift-compiler: the libraries depend on them through +# //Meta/vcpkg: and //:_lib, so there is no separate step. Build +# //:vcpkg_installed alone if you want to time it. +# No CMake build is needed, and none is consulted -- see the removal test +# above. That was not true before finding 35. bazel build //:ladybird //:WebContent //:RequestServer //:ImageDecoder //:Compositor //:WebWorker ``` @@ -100,6 +116,13 @@ python3 Meta/emit_cargo_bazel.py --index > cargo_index.bzl python3 Meta/emit_cargo_bazel.py --ring > cargo_ring.bzl python3 Meta/emit_cargo_bazel.py --extension > cargo_extension.bzl python3 Meta/emit_cargo_bazel.py --check . # all four reproduce byte-for-byte +# The 15 Export.h + AK's 2 configure_file headers: emitted, and byte-compared +# against CMake's own copies. (AK/Backtrace.h is excluded on purpose -- it is a +# host PROBE, not a template, so comparing it to this machine's tree would only +# re-confirm this machine.) +python3 Meta/emit_export_headers_bazel.py > export_headers.bzl +python3 Meta/emit_export_headers_bazel.py --libweb > Libraries/LibWeb/export_header.bzl +python3 Meta/emit_export_headers_bazel.py --check Build/full # 17/17 identical # 5. Check every generator against CMake. python3 Meta/bazel_parity_harness.py # expects 1408/1408 identical, 0 UNHANDLED # Sweep hash seeds: a generator that iterates a set matches under some seeds only. @@ -115,7 +138,10 @@ below): ```sh export XDG_RUNTIME_DIR=/tmp/xdg-lb && mkdir -p $XDG_RUNTIME_DIR && chmod 700 $XDG_RUNTIME_DIR ER=$(bazel info execution_root); mkdir -p "$ER/bazel-out/k8-fastbuild/libexec" -for b in WebContent RequestServer ImageDecoder Compositor; do cp -f bazel-bin/$b "$ER/bazel-out/k8-fastbuild/libexec/"; done +for b in WebContent RequestServer ImageDecoder Compositor WebWorker; do cp -f bazel-bin/$b "$ER/bazel-out/k8-fastbuild/libexec/"; done +# cranelift-compiler is found as a sibling of the spawning binary (Ladybird's own +# lookup chain), so it needs no path baked in -- only to be there. +cp -f bazel-bin/cranelift-compiler "$ER/bazel-out/k8-fastbuild/libexec/" ln -sfn "$PWD/Build/full/share/Lagom" "$ER/bazel-out/k8-fastbuild/share" ./bazel-bin/ladybird --headless=text file:///tmp/test-page.html ``` @@ -231,6 +257,47 @@ Honest inventory of what stops this from being a clone-and-build. rebuilds all 11 crates. Both trade a rebuild for correctness, which is the right way round — under-declaring silently reuses a stale artifact. Per-crate source sets need the path-dependency graph read out of the manifests. + + **Closing the header gap found two real bugs the CMake tree had been masking + (finding 35), and this is the argument for verify-by-removal.** Both were + *correctness* bugs in the Bazel build that a green build could not have shown, + because a stale include path from CMake's tree was quietly supplying the right + answer: + + * **An FFI header collision.** Eight of the ten crates emit a header literally + named `RustFFI.h`, and four TUs `#include ` with no directory. + CMake is unambiguous because `FFI_OUTPUT_DIR` defaults to the library's *own* + binary dir; Bazel puts every dep's include dirs on one command line, so + **LibRegex was compiling against LibUnicode's header** — masked only by a + leftover `-IBuild/full/Libraries/LibRegex` that shadowed both. Removing the + tree turned it into `'RustRegexFlags' has not been declared`. The fix needed + *two* steps, and the first alone looked sufficient: publish the unprefixed + dir on a separate target (`cargo_bare_include`) so it does not ride on + `cargo_lib`, **and** depend on it through `implementation_deps`. Include dirs + propagate along the C++ dep graph too, not just out of one rule — with a + plain `deps` edge, LibGfx inherited LibTextCodec's dir through + `LibGfx → LibTextCodec` and `YUVData.cpp` compiled against the wrong header + (`'FFI' does not name a type`). `implementation_deps` is Bazel's name for + exactly the scope CMake's `PRIVATE` include dir has, which is *why* a bare + include is unambiguous in CMake and had to be made unambiguous here. + * **An entire Rust target missing from the graph.** `Libraries/LibWasm` declares + its Cranelift crate with `build_rust_binary()`, not `import_rust_crate()`, and + the emitter parsed only the latter — so Cranelift was absent from Bazel + *entirely*. Two host escapes were covering for it: the FFI header sat in + `Build/full/Libraries/LibWasm` where a global `-I` reached it, and the + compiler binary was named by an **absolute path to my machine** baked into + `-DWASM_CRANELIFT_COMPILER_PATH=/home/ubuntu/...`, which alone would have + broken any other checkout. It is now a `cargo_binary` that also declares its + `cbindgen` header (byte-identical to CMake's), and the define is rewritten to + a bare file name so Ladybird's *own* lookup chain + (`resolve_cranelift_compiler_path`: env var → compile-time path → + sibling-of-self) finds it in Bazel's bin dir — the dependency is declared as + `data`, the path is not asserted. + + The generalizable part: **a shim that cannot fail cannot be trusted.** Both + bugs were invisible while a foreign tree was on the include path, and both + surfaced the moment it was removed. Neither would have been found by building + harder. 2. **The exec configuration's flags are a duplicated list.** Now that Bazel *runs* a tool it built (`//:generate_interpreter_layout`), that tool and the AK it links are built in the exec configuration, a separate flag namespace that @@ -253,15 +320,25 @@ Honest inventory of what stops this from being a clone-and-build. and `-L/usr/lib/x86_64-linux-gnu`, both because libdrm and Vulkan are host-provided. (The `-L`/`-rpath` into `Build/full/vcpkg_installed/` are gone — finding 33.) -4. **The generated-header shims over `Build/full` are what is left of the CMake - dependency.** `Build/full/**/BUILD.bazel` no longer shims any *binary* — the - vcpkg `.so`s (finding 33) and the Rust archives + `flapc` (finding 34) are all - Bazel-built, and `flapc` is now a real `cargo_binary` Bazel builds *and* runs - as a genrule tool, so `interpreter_x86_64.S` depends on no cargo artifact. What - remains is a handful of generated *headers* the emitters have not yet taught - Bazel to generate, plus `Build/full` itself as the model the emitters read and - the baseline the parity harness diffs against. That last one is a - converter-development dependency, not a build dependency. +4. **The `Build/full` shims are gone; what remains is a *converter* dependency, + not a build dependency.** `Build/full/**/BUILD.bazel` no longer exists. The + vcpkg `.so`s (finding 33), the Rust archives + `flapc` (finding 34), the + Cranelift compiler (finding 35), the 15 `Export.h` + AK's two `configure_file` + headers (finding 35) and all 692 LibWeb generated headers are Bazel's own + outputs. `Build/full` is still read by the *emitters* (it is the model they + translate) and by the parity harness (it is the baseline they diff against) — + which is a dependency of *regenerating* the BUILD files, not of building the + browser from them. + + The distinction was invisible while the shims existed, which is exactly why + they lasted: `glob(["**/*.h"], allow_empty = True)` over a tree that is not + there yields an empty list and no diagnostic, so the build failed ~1,600 + actions later with a missing-header error naming neither the shim nor the + tree. **The lesson generalizes past this migration:** a shim over a foreign + build tree should fail loudly when the tree is absent, or it is indistinguishable + from a shim that is not needed. `allow_empty = True` on a glob whose emptiness + means "the thing you depend on is missing" converts a build error into a + mystery. 5. **Running the UI needs manual staging** (the commands above). Ladybird's UI spawns its service binaries by looking next to itself, and `share/Lagom` for resources. A real Bazel setup would express this with `data` + runfiles; diff --git a/examples/ladybird/workspace/BUILD.bazel b/examples/ladybird/workspace/BUILD.bazel index d290a2f..2d169ce 100644 --- a/examples/ladybird/workspace/BUILD.bazel +++ b/examples/ladybird/workspace/BUILD.bazel @@ -42,21 +42,29 @@ cc_library( "UI/**/*.h", ], allow_empty = True), deps = [ - # Bazel-generated headers (the genrules above) come FIRST so they win - # over the CMake copies; the Build/full roots below now only supply what - # Bazel does not yet generate (Rust FFI headers, CMake's - # generate_export_header Export.h). + # EVERY generated header is Bazel's own now. The two + # //Build/full/{Libraries,Services} header roots that used to be listed + # here are gone, and with them the last thing the BUILD read out of + # CMake's build tree. + # + # They were not supplying anything by the end, and that is the part worth + # recording, because it is why they survived so long: a `glob(["**/*.h"])` + # over a foreign tree cannot fail. The 709 headers under those roots were + # 21 that Bazel generates and 688 LibWeb bindings headers that Bazel ALSO + # generates -- so the roots were SHADOWING Bazel's own outputs, silently + # winning or losing on include order, and the only reason a fresh clone + # did not fail here was that it got no error either: allow_empty=True + # turns "the tree is not there" into an empty glob and the build dies + # ~1,600 actions later on a missing Export.h. Verified by removal: + # Build/full/{Libraries,Services,UI} moved off the machine, all six + # binaries rebuilt from scratch, --headless=text and + # --headless=layout-tree byte-identical to the CMake reference. ":generated_libraries_headers", ":generated_services_headers", ":generated_shader_headers", ":generated_export_headers", ":generated_ak_headers", "//Libraries/LibWeb:generated_export_header", - # Build/full roots: now ONLY the Rust FFI headers and the IPC/codegen - # headers Bazel does not yet generate. Export.h and AK's two - # configure_file headers are Bazel-generated above. - "//Build/full/Libraries:generated_lib_headers", - "//Build/full/Services:generated_service_headers", ], ) @@ -1439,7 +1447,6 @@ cc_library( 'Services/WebContent/WebUIConnection.cpp', ], hdrs = glob(["Services/WebContent/**/*.h"], allow_empty = True), - copts = ['-IBuild/full'], deps = [ '//:AK', '//:LibCore', @@ -1480,7 +1487,6 @@ cc_library( 'Services/RequestServer/WebSocketImplCurl.cpp', ], hdrs = glob(["Services/RequestServer/**/*.h"], allow_empty = True), - copts = ['-IBuild/full'], deps = [ '//:AK', '//:LibCore', @@ -1508,7 +1514,6 @@ cc_library( 'Services/ImageDecoder/ConnectionFromClient.cpp', ], hdrs = glob(["Services/ImageDecoder/**/*.h"], allow_empty = True), - copts = ['-IBuild/full'], deps = [ '//:AK', '//:LibCore', @@ -1543,7 +1548,6 @@ cc_library( ], hdrs = glob(["Services/Compositor/**/*.h"], allow_empty = True), local_defines = ['SK_CODEC_DECODES_BMP', 'SK_CODEC_DECODES_GIF', 'SK_CODEC_DECODES_WBMP', 'SK_DISABLE_TRACING', 'SK_ENABLE_AVX512_OPTS', 'SK_ENABLE_PRECOMPILE', 'SK_FONTMGR_ANDROID_AVAILABLE', 'SK_FONTMGR_FCI_AVAILABLE', 'SK_FONTMGR_FONTCONFIG_AVAILABLE', 'SK_FONTMGR_FREETYPE_DIRECTORY_AVAILABLE', 'SK_FONTMGR_FREETYPE_EMBEDDED_AVAILABLE', 'SK_FONTMGR_FREETYPE_EMPTY_AVAILABLE', 'SK_GAMMA_APPLY_TO_A8', 'SK_GANESH', 'SK_HAS_WUFFS_LIBRARY', 'SK_R32_SHIFT=16', 'SK_TYPEFACE_FACTORY_FREETYPE', 'SK_USE_PERFETTO', 'SK_VULKAN', 'SK_XML'], - copts = ['-IBuild/full'], deps = [ '//:LibCore', '//:LibGfx', @@ -1565,7 +1569,6 @@ cc_library( 'Services/WebWorker/WorkerHost.cpp', ], hdrs = glob(["Services/WebWorker/**/*.h"], allow_empty = True), - copts = ['-IBuild/full'], deps = [ '//:AK', '//:LibCore', @@ -1718,7 +1721,7 @@ cc_binary( 'Services/RendererSandboxLinux.cpp', 'Services/WebContent/main.cpp', ], - copts = ['-fPIE', '-IBuild/full'], + copts = ['-fPIE'], linkopts = ['-lvulkan'], deps = [ '//:AK', @@ -1816,7 +1819,7 @@ cc_binary( 'UI/Qt/main.cpp', ], local_defines = ['QT_CORE_LIB', 'QT_GUI_LIB', 'QT_NO_DEBUG', 'QT_WIDGETS_LIB'], - copts = ['-fPIE', '-IBuild/full/UI', '-IBuild/full/UI/Qt'], + copts = ['-fPIE'], linkopts = ['-lGLX', '-lOpenGL', '-lvulkan'], deps = [ '//:AK', diff --git a/examples/ladybird/workspace/Build/full/Libraries/BUILD.bazel b/examples/ladybird/workspace/Build/full/Libraries/BUILD.bazel deleted file mode 100644 index d788bef..0000000 --- a/examples/ladybird/workspace/Build/full/Libraries/BUILD.bazel +++ /dev/null @@ -1,29 +0,0 @@ -load("@rules_cc//cc:defs.bzl", "cc_library") -package(default_visibility = ["//visibility:public"]) - -# NOTE: no rust_ffi_headers here any more. The 14 Rust FFI headers used to be -# globbed out of CMake's build tree (cargo writes them into -# Build/full/Libraries// via FFI_OUTPUT_DIR), so every Rust-consuming TU -# read a CMake artifact. Bazel now BUILDS the crates and declares those headers -# as the cargo_crate action's own outputs, consumed one target per crate -# (//:libweb_css_rust_lib etc., cargo_ring.bzl) -- so the include path arrives on -# the dep edge instead of a glob over a foreign tree. Verified by removal: with -# this gone AND Build/full/cargo + Build/full/bin/flapc moved off the machine, all -# six binaries still build and render identically. - -# All CMake-generated per-library headers materialized under Build/full/Libraries -# (generate_export_header -> /Export.h, plus other generated .h). Exposed as -# a header root so etc. resolve in the sandbox. -cc_library( - name = "generated_lib_headers", - hdrs = glob(["**/*.h"], allow_empty = True), - includes = ["."], -) - -# NOTE: no exports_files here any more. TIFFTagHandler.cpp, HSTSPreloadData.cpp, -# Op.cpp and interpreter_x86_64.S used to be exported from CMake's build tree and -# COMPILED straight into Bazel's libraries -- so the migration silently depended -# on CMake having produced those .cpp/.S files. Bazel now generates all four -# itself (codegen_root.bzl: gen_TIFFMetadata, gen_HSTSPreloadData, gen_Op, -# gen_interpreter_asm) and the consumers reference the genrule outputs. Verified -# by removal: deleting these entries keeps the build green. diff --git a/examples/ladybird/workspace/Build/full/Services/BUILD.bazel b/examples/ladybird/workspace/Build/full/Services/BUILD.bazel deleted file mode 100644 index fa12d04..0000000 --- a/examples/ladybird/workspace/Build/full/Services/BUILD.bazel +++ /dev/null @@ -1,15 +0,0 @@ -load("@rules_cc//cc:defs.bzl", "cc_library") -package(default_visibility = ["//visibility:public"]) - -# CMake-generated IPC endpoint headers materialized under Build/full/Services. -# Exposed as a header root so etc. -# resolve in the Bazel sandbox (matches CMake's -IBuild/full/Services). -cc_library( - name = "generated_service_headers", - hdrs = glob(["**/*.h"], allow_empty = True), - includes = ["."], -) -# NOTE: no exports_files here any more. WebGLCommandReplayer.cpp used to be -# compiled straight out of CMake's tree; it is now a genrule output in the root -# package (`:Services/Compositor/WebGLCommandReplayer.cpp`, see codegen_root.bzl). -# Verified by removal: deleting this line keeps //:Compositor green. diff --git a/examples/ladybird/workspace/Build/full/UI/BUILD.bazel b/examples/ladybird/workspace/Build/full/UI/BUILD.bazel deleted file mode 100644 index 78f2065..0000000 --- a/examples/ladybird/workspace/Build/full/UI/BUILD.bazel +++ /dev/null @@ -1,14 +0,0 @@ -package(default_visibility = ["//visibility:public"]) - -# Nothing from CMake's UI build dir is consumed any more. -# -# Qt moc/qrc are generated by Bazel via rules_qt (see //:qt_moc / //:qt_rcc in -# the root BUILD), and the two glslang shader headers -# (WebContentViewLinux{Frag,Vert}Shader.h) are now generated by Bazel too -# (//:gen_WebContentViewLinuxFragShader + :generated_shader_headers in -# codegen_root.bzl), so the qt_autogen_headers cc_library that used to glob them -# out of Build/full/UI/Qt is gone. Verified by removal: the build is green -# without it. -# -# This package is kept (empty) only because .bazelrc/others may reference the -# path; it declares no targets. diff --git a/examples/ladybird/workspace/Meta/emit_build_bazel.py b/examples/ladybird/workspace/Meta/emit_build_bazel.py index f2983b8..ebf8e76 100644 --- a/examples/ladybird/workspace/Meta/emit_build_bazel.py +++ b/examples/ladybird/workspace/Meta/emit_build_bazel.py @@ -196,7 +196,17 @@ def target_defines(t, name): def target_private_includes(t): """Per-target -isystem/-I under Build/full (the target's own gendir) not in - the 6 global roots.""" + the 6 global roots. + + Paths are NORMALIZED before the comparison, which is not cosmetic. CMake emits + a target's own binary dir relative to itself, so the five service targets get + `-IBuild/full/Services/WebContent/../..` -- the same directory as the global + `-IBuild/full`, spelled differently. Compared as strings it is not in + globalroots, so it came out as a per-target `-IBuild/full` copt on five + targets plus WebContent: six copts pointing into CMake's build tree that + supplied nothing any global root did not already supply, and which made the + emitted build look like it needed Build/full when it did not. + """ globalroots = {ROOT, ROOT+"/Libraries", ROOT+"/Services", ROOT+"/Build/full", ROOT+"/Build/full/Libraries", ROOT+"/Build/full/Services", @@ -207,7 +217,7 @@ def target_private_includes(t): args = a["arguments"]; i = 0 while i < len(args): if args[i] in ("-I", "-isystem"): - p = args[i+1] + p = os.path.normpath(args[i+1]) if p in globalroots: pass elif p.startswith(ROOT): @@ -508,21 +518,29 @@ def _emit_srcs(srcs): "UI/**/*.h", ], allow_empty = True), deps = [ - # Bazel-generated headers (the genrules above) come FIRST so they win - # over the CMake copies; the Build/full roots below now only supply what - # Bazel does not yet generate (Rust FFI headers, CMake's - # generate_export_header Export.h). + # EVERY generated header is Bazel's own now. The two + # //Build/full/{Libraries,Services} header roots that used to be listed + # here are gone, and with them the last thing the BUILD read out of + # CMake's build tree. + # + # They were not supplying anything by the end, and that is the part worth + # recording, because it is why they survived so long: a `glob(["**/*.h"])` + # over a foreign tree cannot fail. The 709 headers under those roots were + # 21 that Bazel generates and 688 LibWeb bindings headers that Bazel ALSO + # generates -- so the roots were SHADOWING Bazel's own outputs, silently + # winning or losing on include order, and the only reason a fresh clone + # did not fail here was that it got no error either: allow_empty=True + # turns "the tree is not there" into an empty glob and the build dies + # ~1,600 actions later on a missing Export.h. Verified by removal: + # Build/full/{Libraries,Services,UI} moved off the machine, all six + # binaries rebuilt from scratch, --headless=text and + # --headless=layout-tree byte-identical to the CMake reference. ":generated_libraries_headers", ":generated_services_headers", ":generated_shader_headers", ":generated_export_headers", ":generated_ak_headers", "//Libraries/LibWeb:generated_export_header", - # Build/full roots: now ONLY the Rust FFI headers and the IPC/codegen - # headers Bazel does not yet generate. Export.h and AK's two - # configure_file headers are Bazel-generated above. - "//Build/full/Libraries:generated_lib_headers", - "//Build/full/Services:generated_service_headers", ], ) @@ -710,6 +728,16 @@ def emit_target(name, targets, libs, exes, so, ar, header=True, body_only=False, # without depending on skia now fails to compile, # which is the whole point of declaring inputs. pass + elif i in ("Build/full/UI", "Build/full/UI/Qt"): + # CMake's UI gendir, whose only non-autogen contents are the two + # SPIR-V shader headers (WebContentViewLinux{Frag,Vert}Shader.h). + # Bazel generates both itself and carries them on + # :generated_shader_headers with includes=["UI/Qt"], so the + # include path arrives through the dep graph. Dropped, not + # relocated -- and checked by removal: with Build/full/UI moved + # off the machine, //:ladybird still builds and renders + # byte-identically. + pass else: copt_toks.append("-I" + i) if copt_toks: From b527e3197292f33eea06443caf7fdee9ac736685 Mon Sep 17 00:00:00 2001 From: Ulf Adams Date: Fri, 7 Aug 2026 21:25:22 +0000 Subject: [PATCH 04/59] Ladybird: stop baking an absolute home-directory path into the emitted BUILD vcpkg_tree's cache_dir was "/home/ubuntu/.cache/vcpkg-bazel" -- the last absolute host path in the generated build, and meaningless in anyone else's checkout. It is an opt-in resumability affordance and no part of the dependency graph, so the emitter now writes "", which is also the honest default: an empty cache is the genuine from-source build. The rule already supported it. That leaves zero absolute host paths in the emitted BUILD files (the one remaining /home/ubuntu string is inside the comment explaining why it is not there), and zero targets under Build/full in the dependency closure of all six binaries -- down from 741. Both checked with cquery and grep rather than asserted. --- examples/ladybird/README.md | 13 +++++++++++++ examples/ladybird/workspace/BUILD.bazel | 14 ++++++++++---- .../ladybird/workspace/Meta/emit_build_bazel.py | 14 ++++++++++---- 3 files changed, 33 insertions(+), 8 deletions(-) diff --git a/examples/ladybird/README.md b/examples/ladybird/README.md index 816b5b9..ee4fe1a 100644 --- a/examples/ladybird/README.md +++ b/examples/ladybird/README.md @@ -320,6 +320,19 @@ Honest inventory of what stops this from being a clone-and-build. and `-L/usr/lib/x86_64-linux-gnu`, both because libdrm and Vulkan are host-provided. (The `-L`/`-rpath` into `Build/full/vcpkg_installed/` are gone — finding 33.) + + **No absolute host path remains in the generated BUILD files.** Two did until + recently, and both were the same mistake in different clothing: a value that is + correct on the machine that generated it and meaningless anywhere else, sitting + in a checked-in file where nothing would ever contradict it. + `-DWASM_CRANELIFT_COMPILER_PATH="/home/ubuntu/.../bin/cranelift-compiler"` was + load-bearing (finding 35) and now resolves through Ladybird's own + sibling-of-self lookup; `vcpkg_tree`'s `cache_dir` was merely a resumability + affordance and is now empty, which is also the honest default (an empty cache + *is* the genuine from-source build). Grepping the emitted output for `/home/` + is a one-line check worth keeping in any migration — a hardcoded path that + happens to work is the failure mode that survives every test run on the + author's machine. 4. **The `Build/full` shims are gone; what remains is a *converter* dependency, not a build dependency.** `Build/full/**/BUILD.bazel` no longer exists. The vcpkg `.so`s (finding 33), the Rust archives + `flapc` (finding 34), the diff --git a/examples/ladybird/workspace/BUILD.bazel b/examples/ladybird/workspace/BUILD.bazel index 2d169ce..ea06b6d 100644 --- a/examples/ladybird/workspace/BUILD.bazel +++ b/examples/ladybird/workspace/BUILD.bazel @@ -1864,10 +1864,16 @@ vcpkg_tree( source_dir = ".", source_root = ":vcpkg_source_inputs", triplet = "x64-linux-dynamic", - # Resume cache: makes a killed 45-minute build cheap to restart. Absolute by - # necessity (the action's cwd is the execroot) and therefore a host escape -- - # it is a build-speed affordance, not part of the dependency graph. - cache_dir = "/home/ubuntu/.cache/vcpkg-bazel", + # Resume cache: OPT-IN, and empty here on purpose. Setting it makes a killed + # 45-minute vcpkg build cheap to restart, but the path must be absolute (the + # action's cwd is the execroot), so any value here is one developer's home + # directory in a file everyone checks out -- it was + # /home/ubuntu/.cache/vcpkg-bazel, the last absolute host path left in the + # emitted build. Empty is also the honest default: it is the genuine + # from-source build. Set this one attribute locally if you want resumability. + # It is a build-speed affordance and no part of the dependency graph, which is + # precisely why it must not be a checked-in constant. + cache_dir = "", vcpkg_root = "Build/vcpkg", vcpkg_tree = "//Build/vcpkg:tree", ) diff --git a/examples/ladybird/workspace/Meta/emit_build_bazel.py b/examples/ladybird/workspace/Meta/emit_build_bazel.py index ebf8e76..649053b 100644 --- a/examples/ladybird/workspace/Meta/emit_build_bazel.py +++ b/examples/ladybird/workspace/Meta/emit_build_bazel.py @@ -432,10 +432,16 @@ def _emit_srcs(srcs): source_dir = ".", source_root = ":vcpkg_source_inputs", triplet = "x64-linux-dynamic", - # Resume cache: makes a killed 45-minute build cheap to restart. Absolute by - # necessity (the action's cwd is the execroot) and therefore a host escape -- - # it is a build-speed affordance, not part of the dependency graph. - cache_dir = "/home/ubuntu/.cache/vcpkg-bazel", + # Resume cache: OPT-IN, and empty here on purpose. Setting it makes a killed + # 45-minute vcpkg build cheap to restart, but the path must be absolute (the + # action's cwd is the execroot), so any value here is one developer's home + # directory in a file everyone checks out -- it was + # /home/ubuntu/.cache/vcpkg-bazel, the last absolute host path left in the + # emitted build. Empty is also the honest default: it is the genuine + # from-source build. Set this one attribute locally if you want resumability. + # It is a build-speed affordance and no part of the dependency graph, which is + # precisely why it must not be a checked-in constant. + cache_dir = "", vcpkg_root = "Build/vcpkg", vcpkg_tree = "//Build/vcpkg:tree", ) From 47bc4e6cdcff2f794aecb12f7597ab6abfc1604a Mon Sep 17 00:00:00 2001 From: Ulf Adams Date: Sat, 8 Aug 2026 09:35:24 +0000 Subject: [PATCH 05/59] Ladybird: three test files were running zero tests, and 6 of the 46 were right Same bug as the glob that could not fail, one layer up. `for f in tests/test_*.py; do python3 "$f"; done` exited 0 for all ten files -- but test_emit_cargo.py, test_emit_vcpkg.py and test_vcpkg_plumbing.py had no `if __name__` runner at all: Python defined 46 test functions, called none, exited 0. There is no pytest here, so nothing else called them either. Found by counting tests executed instead of trusting the exit code. With runners added, 6 of the 46 failed, and every one was a TRUE report about the finding-35 changes rather than a stale assertion: emit_ring() grew a third parameter (the binary crates); cargo_lib gained an `archive == None` branch because a --bin crate has no archive, so the ring has 11 cargo_lib for 10 cargo_crate; the shared FFI-header lookup moved into cargo_vendor.sh so both cargo drivers use one copy; the ring quotes with ' not "; and one test read Build/full/Libraries/BUILD.bazel -- a file whose DELETION was the fix. That one is now the stronger assertion that examples/ladybird/workspace/Build/full must not exist at all. Plus two regression tests for the two finding-35 bugs, both asserted from CMake rather than from a list: every build_rust_binary() call site reaches the ring with the same two consumable labels a staticlib crate gets (the fixture now carries both shapes -- flapc with no FFI_OUTPUT_DIR, cranelift with one), and every *_bare_include target is reached through implementation_deps and never plain deps, because CcInfo include dirs propagate along the C++ dep graph and that is how LibGfx compiled against LibTextCodec's RustFFI.h. 128 tests over 10 files, all passing. README now documents running the glob rather than a hand-kept list of filenames, since a list of files to run is one more thing that can silently omit an entry; case study finding 35 records the episode. --- README.md | 19 ++- docs/CASE-ladybird-migration.md | 251 +++++++++++++++++++++++++++++++- tests/test_emit_cargo.py | 175 ++++++++++++++++++++-- tests/test_emit_vcpkg.py | 20 +++ tests/test_vcpkg_plumbing.py | 20 +++ 5 files changed, 463 insertions(+), 22 deletions(-) diff --git a/README.md b/README.md index e0d156d..d5d9394 100644 --- a/README.md +++ b/README.md @@ -324,6 +324,10 @@ tests/ test_extract_npm.py npm frontend (NDJSON → action IR) + role classification test_triage.py triage grouping/histogram/cap behavior test_configure.py configure_file trace extraction + test_diff_ts.py the standalone TS source→emit diff + test_emit_cargo.py the Ladybird Rust ring emitter (crates/index/ring/binaries) + test_emit_vcpkg.py the Ladybird vcpkg pin emitter (versions db → http_file) + test_vcpkg_plumbing.py what the generated vcpkg BUILD/bzl files must not say docs/ DESIGN-action-based-ir.md the action-based IR reframe (grounded in CMake/Bazel/Maven) proposal-doc.md the broader Build IR spec (frontends → IR → backends) @@ -390,14 +394,17 @@ Run as a skill, Claude drives the generation and triage/fix loop automatically. ## Tests ```bash -python3 tests/test_engine.py -python3 tests/test_extractors.py -python3 tests/test_maven.py -python3 tests/test_extract_npm.py -python3 tests/test_triage.py -python3 tests/test_configure.py +for f in tests/test_*.py; do python3 "$f" || echo "FAILED $f"; done ``` +Run the glob, not a hand-kept list of files: three of these test files once had +no `if __name__ == "__main__"` runner at all, so running them **exited 0 having +executed nothing** — a test suite that cannot fail, which is the same bug as a +`glob(..., allow_empty = True)` over a directory that is not there (case study +finding 35). Each file prints `PASS`/`FAIL` per test and an `N/N passed` line, so +compare the totals against the number of `def test_` in the file rather than +trusting the exit code. + The extractor tests run against fixtures under `tests/` that mirror the documented File API, aquery, Maven argfile, and npm-NDJSON schemas. Real projects exercise schema details the fixtures may not (fragment quoting, `external/` repo diff --git a/docs/CASE-ladybird-migration.md b/docs/CASE-ladybird-migration.md index d202c97..90ebe2e 100644 --- a/docs/CASE-ladybird-migration.md +++ b/docs/CASE-ladybird-migration.md @@ -8,6 +8,13 @@ generated BUILD files — is in [`examples/ladybird/`](../examples/ladybird/), along with an honest inventory of what still stops it from being a clone-and-build. +`git clone && bazel build //:ladybird` now builds the browser with **nothing** +read out of CMake's build tree. That was claimed prematurely and was false for +most of this migration; [finding 35](#finding-35-the-claim-was-false-and-a-glob-is-why-nobody-noticed) +is the autopsy, and it is the finding to read first if you are migrating a project +of your own — the bug was in the *shape of the verification*, not in the Bazel +rules. + ## Why Ladybird A real, recognizable, from-scratch browser engine — a strong headline for the @@ -1433,6 +1440,221 @@ is still needed to *regenerate* the BUILD files and to run the parity harness that is a converter-development dependency, not a build dependency, and it is a different claim from the one this ring closed. +**That claim was still false when I wrote it, and finding 35 is the autopsy.** I +had closed the two *binary* dependencies (vcpkg, Rust) and concluded the tree was +free; it was not. Every binary still depended on **741 targets under `Build/full`** +for *generated headers*, and I had not checked, because the thing that would have +told me — a build failure — could not happen: the shims globbed a foreign tree +with `allow_empty = True`. Read finding 35 before believing any "verified by +removal" in this document, including the ones above: removal is only a test of +what you actually remove. + +## Finding 35: the claim was false, and a glob is why nobody noticed + +Ulf asked where things stood on checking out Ladybird and building it with Bazel. +I said it worked — findings 33 and 34 had closed vcpkg and Rust, the two *binary* +dependencies on CMake's tree, and the README said so in bold. Then I checked, +which I should have done before saying it. + +It did not work. Every one of the six binaries depended on **741 targets under +`Build/full`**, CMake's build tree, and the overlay in `examples/ladybird/` +shipped four `BUILD.bazel` files and **zero headers**. A fresh clone got no error +from that — it built for roughly 1,600 actions and then died on `fatal error: +LibXML/Export.h: No such file or directory`, a message that names neither the shim +nor the missing tree. + +**The mechanism is the finding.** The shims were + +```python +cc_library( + name = "generated_lib_headers", + hdrs = glob(["**/*.h"], allow_empty = True), + includes = ["."], +) +``` + +over a directory that only exists after a CMake build. `allow_empty = True` turns +"the tree you depend on is absent" into an empty list and no diagnostic. A glob +over a foreign tree **cannot fail**, so a shim that is broken and a shim that is +unnecessary are indistinguishable — and I had been reading a green build as +evidence for the second. + +That generalizes past this migration, and it is the transferable lesson: **if the +emptiness of a glob means "a thing I depend on is missing", `allow_empty = True` +converts a build error into a mystery.** Either let it fail, or don't pretend the +input is optional. + +### 741 → 31: most of them were not gaps, they were shadows + +Counting the true gaps needed care, and my first count was nonsense — "741 +targets, 76 covered", which is impossible. The bug was **matching headers by +basename**: every `Export.h` matched every other `Export.h`, so coverage looked +enormous. Comparing exact logical paths gave the real picture: + +| | count | what it was | +|---|---|---| +| LibWeb bindings headers | 666 | **Bazel already generates all 692.** The shim was *shadowing* Bazel's own outputs, silently winning or losing on include order | +| `generate_export_header` `Export.h` | 15 | real gap | +| Rust FFI headers (`RustFFI.h`, `CraneliftFFI.h`) | 15 | real gap | +| Qt `moc_predefs.h` | 1 | not a gap at all — Bazel runs moc itself via `rules_qt`; nothing referenced it | +| AK `configure_file` headers | 2 | already closed earlier in the session | + +So 666 of the 741 were pure duplication, which I proved by deleting them: the +build stayed green and recompiled 2,635 actions from Bazel's own headers. The +honest gap was **31 files**, and the reason a 90%-redundant shim survived is the +same `allow_empty` — nothing ever compared the two sets. + +### The 15 `Export.h`: derive the template, don't copy the output + +`Meta/emit_export_headers_bazel.py` emits them, and two decisions in it matter. + +All 15 normalize to **one** template; the per-library tokens are derived the way +`Meta/CMake/targets.cmake` derives them (api = `upper(lib - "Lib") + "_API"`, +prefix = `upper(lib)`, exports = `lib + "_EXPORTS"`), so adding a library needs no +edit. All 17 emitted artifacts (15 + AK's two) are **byte-identical to CMake's**, +checked by `--check Build/full`, and that check earned its keep twice: once on a +`#cmakedefine` regex (the templates put the `#` at column 0 with the indent +*after* it — `# cmakedefine01 FOO`, which two regex attempts got wrong), and +once on a **1-byte** difference a `render()`-level check could never see — a +heredoc adds a trailing newline, so the emitted shell needed one stripped from the +body. Only byte-comparing the *built artifact* catches that. + +`AK/Backtrace.h` is deliberately **not** in the parity check, and that is the +interesting one. It is not a template — CMake writes it from +`find_package(Backtrace)`, i.e. from a *question about the host*. So the genrule +**compiles a probe** rather than baking my machine's answer. Comparing the result +against my own tree would only re-confirm my own tree, which is why it is +excluded: test the variable, not the value. + +`Libraries/LibWeb` needed its own emitter mode, for a reason worth stating because +Bazel is right and I was wrong: include dirs cannot escape a package, so LibWeb's +`Export.h` must be an output *of the LibWeb package*. `includes = ["../.."]` is +rejected ("resolves to the workspace root"). It lands at `genroot/LibWeb/Export.h` +with `includes = ["genroot"]`. + +### Bug 1: eight crates ship a `RustFFI.h`, and LibRegex had the wrong one + +Removing the shim exposed a **real, pre-existing correctness bug**. Eight of the +ten Rust crates emit a header named literally `RustFFI.h`, and four TUs include it +with no directory (`#include `). CMake is unambiguous because +`FFI_OUTPUT_DIR` defaults to the consuming library's *own* binary dir. Bazel puts +every dep's include dirs on one command line, and my rules published every +crate's unprefixed `ffi/` dir to every consumer — so **LibRegex was compiling +against LibUnicode's header**, and the only thing hiding it was a leftover +`-IBuild/full/Libraries/LibRegex` that shadowed both. Delete the tree and it +becomes `'RustRegexFlags' has not been declared`. + +The fix needed **two** steps, and the first one alone looked sufficient — which is +the part I would otherwise have shipped: + +1. **Publish the unprefixed dir on its own target** (`cargo_bare_include`), not on + `cargo_lib`. Necessary because `CcInfo`'s `system_includes` propagate + transitively: folded into `cargo_lib`, LibGfx inherited LibRegex's *and* + LibTextCodec's bare dirs and failed with `'FFI' does not name a type`. + (`cc_common.create_compilation_context` has no settable "local includes"; + I tried.) +2. **Depend on it through `implementation_deps`.** Step 1 stops the dir leaking out + of `cargo_lib`; it does *not* stop it leaking out of **LibTextCodec**. Include + dirs propagate along the C++ dep graph too, so `LibGfx → LibTextCodec` handed + LibGfx someone else's `RustFFI.h` anyway, and `YUVData.cpp` failed identically. + +`implementation_deps` is Bazel's name for exactly the scope CMake's +`target_include_directories(... PRIVATE)` has — which is *why* a bare include is +unambiguous in CMake, and what had to be reproduced rather than approximated. The +three crates that need it are **derived by scanning the source** for a +directory-less include, not hardcoded, and the build emitter now **imports** that +derived set instead of keeping its own copy beside it (it had one; a hand-kept copy +of a derived set is finding 23 in miniature). + +### Bug 2: an entire Rust target was missing, behind two host escapes + +`//:WebContent` and four others then failed with one error: +`CraneliftBridge.cpp:13:10: fatal error: CraneliftFFI.h: No such file or directory`. + +Cranelift — Ladybird's AOT WebAssembly compiler, `ENABLE_CRANELIFT_JIT=ON` — was +absent from the Bazel graph **entirely**. `Libraries/LibWasm/CMakeLists.txt` was +not in the emitter's `CMAKELISTS`, and it declares its crate with +`build_rust_binary()`, which the parser did not handle at all. Two host escapes had +been covering for it: + +- the FFI header sat in `Build/full/Libraries/LibWasm`, which a global `-I` reached; +- the compiler binary was named by **an absolute path to my machine**, baked into + `-DWASM_CRANELIFT_COMPILER_PATH="/home/ubuntu/ladybird-work/Build/full/bin/cranelift-compiler"`. + +The second one alone would have broken every other checkout on earth, and it was +in a checked-in generated file. Nothing was going to tell me: it is a *define*, so +it compiles fine and the lookup only fails at run time, on a code path that needs a +WebAssembly page big enough to trigger AOT compilation. + +A `build_rust_binary` crate is a genuinely different shape, not a variant spelling, +and the rules now say so: there is **no archive to link** (`cargo rustc --bin` is +the whole build, and what C++ consumes is the *executable*, spawned at run time), +but it **still emits a cbindgen header** its caller includes. So `cargo_binary` +declares FFI headers like `cargo_crate` does, `cargo_lib` yields a headers-only +`CcInfo` when the archive is absent, and the crate is consumed through the same two +labels a staticlib crate is. + +For the binary itself, the fix is *not* to point the define at `bazel-bin` — that +is the same escape with a nicer prefix. Ladybird already resolves the compiler +through a chain (`resolve_cranelift_compiler_path`: `$LADYBIRD_CRANELIFT_COMPILER` +→ compile-time path → **sibling-of-self**), and Bazel puts every root-package +output in one bin directory, so link 3 finds it with no path baked in at all. The +define becomes the bare filename and the binary is attached as `data`, so it is +genuinely *there* in the runfiles of everything that links LibWasm. **The +dependency is declared; the path is not asserted.** + +### What actually verified it + +Not "the build is green" — that was the state I started from. The header is +byte-identical to CMake's; then `Build/full/{Libraries,Services,UI,bin,cargo}` was +moved off the machine and the three shim packages **deleted**, leaving zero +Ladybird-generated headers under `Build/full`; all six binaries rebuilt from +scratch (2,704 actions); and both `--headless=text` and `--headless=layout-tree` +came out byte-identical to the CMake reference on three pages, two of which execute +WebAssembly. `cquery` over the closure of all six binaries now returns **0** targets +under `Build/full`. + +One control worth recording, because it is the kind of check that keeps an +enthusiastic conclusion honest: I instrumented `cranelift-compiler` with a wrapper +to prove the browser was invoking it, and **it never was** — even on a +300-function module. Before concluding the wiring was broken I ran the same probe +against the *CMake reference build*, which also never invoked it. Compilation is +submitted to a thread pool and the page finishes first; the Bazel build matches the +reference exactly. Without the reference-side control I would have "fixed" a +non-bug. + +Two smaller repairs fell out of the same pass, both drift the emitter rule exists +to prevent: `emit_libweb_bazel.py` did not emit LibWeb's export-header block, so +regenerating would have silently truncated it; and six of the seven per-target +`-IBuild/full` copts existed only because CMake spells a target's own binary dir +relative to itself (`Build/full/Services/WebContent/../..`) and the emitter +compared paths as **strings** against its global-roots set. `os.path.normpath` +before the comparison removed all six. + +### The same bug, a third time, in this repo's own tests + +Having learned that a check which cannot fail is indistinguishable from one that +is not needed, I applied it to the test suite itself — by counting, not by reading +the exit code. `for f in tests/test_*.py; do python3 "$f"; done` exited 0 for all +ten files, but three of them (`test_emit_cargo.py`, `test_emit_vcpkg.py`, +`test_vcpkg_plumbing.py`) had **no `if __name__ == "__main__"` runner at all**: +Python imported the module, defined 46 `def test_` functions, called none of them, +and exited 0. There is no pytest in this sandbox, so nothing else was calling them +either. Exactly the shape of `allow_empty = True` — a green result carrying no +information. + +With runners added, 6 of those 46 failed, and every one was a *true* report about +the finding-35 changes rather than a stale assertion: the ring emitter had grown a +third parameter (the binary crates), `cargo_lib` had gained an +`archive == None` branch for a `--bin` crate that has no archive, the shared +FFI-header lookup had moved into `cargo_vendor.sh` so both drivers use one copy, +and one test read `Build/full/Libraries/BUILD.bazel` — a file whose *deletion* was +the fix. That last one is the pleasing part: the test failed because the thing it +asserted about no longer exists, so it became the stronger assertion that the path +**must not** exist. The suite is now 128 tests over 10 files, all passing, and the +README documents running the glob rather than a hand-kept list of filenames, +because a list of files to run is one more thing that can silently omit an entry. + ## Plan for the rest - **Ring 1c — BUILD generation to compile+link parity, bottom-up.** AK → @@ -1498,19 +1720,36 @@ every process (UI, WebContent, Compositor, RequestServer, ImageDecoder) Bazel-built, proven by removing the reference services and watching the CMake binary fail while the Bazel one renders. -And with findings 33 and 34 it is a **clone-and-build**: Bazel fetches and builds -the 77 vcpkg ports *and* the 10 Rust crates + `flapc` itself, with zero network -access in the build actions, verified by removing each reference tree from the -machine in turn. `Build/full` is now only the *converter's* input — the model the -emitters read and the baseline the parity harness diffs — not the build's. +And with findings 33, 34 **and 35** it is a **clone-and-build**: Bazel fetches and +builds the 77 vcpkg ports, the 10 Rust crates, `flapc`, `cranelift-compiler` and +every generated header itself, with zero network access in the build actions. +`Build/full` is now only the *converter's* input — the model the emitters read and +the baseline the parity harness diffs — not the build's, and this time that is +measured: **0 targets under `Build/full` in the `cquery` closure of all six +binaries** (down from 741), the three shim packages deleted outright, and all six +binaries rebuilt from scratch with CMake's generated tree moved off the machine, +rendering `--headless=text` *and* `--headless=layout-tree` byte-identically on +three pages including two that execute WebAssembly. + +Findings 33 and 34 alone were **not** enough, and the gap between "I closed the +binary dependencies" and "the build does not read the tree" is the most +instructive part of this migration — see finding 35. **Scoreboard:** 43 `cc_library` + 6 `cc_binary` targets; ~4,400 Bazel actions from scratch (2,700 C++ plus the vcpkg and Rust rings); LibWeb alone 1,961 TUs (1,273 checked-in + 688 generated) with **zero** define/flag/include discrepancies vs CMake; 1,379/1,379 generated files -byte-identical -> now 1,408/1,408 with every one of the build's 586 ninja CUSTOM_COMMANDs accounted for (findings 25-27); 77 vcpkg ports and 154 crates.io crates fetched and built by Bazel; 34 findings, 3 of them real any2bazel engine fixes with +byte-identical -> now 1,408/1,408 with every one of the build's 586 ninja CUSTOM_COMMANDs accounted for (findings 25-27); 77 vcpkg ports and 154 crates.io crates fetched and built by Bazel; **0 targets under `Build/full` in the closure of all six binaries (down from 741) and 0 absolute host paths in the emitted BUILD files** (finding 35); 35 findings, 3 of them real any2bazel engine fixes with regression tests. +**One number in this scoreboard was wrong for most of the migration, and it is +worth ending on.** "Clone and build" was claimed after the two *binary* +dependencies were closed, and the 741 remaining *header* dependencies went uncounted +because the shims that supplied them could not fail. Every "verified by removal" in +this document is only as strong as what was actually removed — which is the argument +for removal as a method, not against it: it is the only check here that ever found +this class of bug, and each time I widened what I removed, it found another. + ## Environment notes (this sandbox) - Toolchain via apt (needs passwordless sudo): cmake, ninja, ccache, diff --git a/tests/test_emit_cargo.py b/tests/test_emit_cargo.py index 2a7b950..cc51a36 100644 --- a/tests/test_emit_cargo.py +++ b/tests/test_emit_cargo.py @@ -135,6 +135,34 @@ def _read(rel): ) ''' +# The OTHER kind of Rust target CMake has, and the one this migration missed +# entirely for a long time (finding 35): build_rust_binary(). Two shapes, both +# real -- a pure build tool with no header at all, and a runtime tool whose build +# script ALSO runs cbindgen (FFI_OUTPUT_DIR present at the call site). +CMAKE_JS = ''' +build_rust_binary( + MANIFEST_PATH Flap/Cargo.toml + CRATE_NAME flapc + BINARY_NAME flapc + OUTPUT_PATH_VAR FLAPC_BIN +) +''' + +CMAKE_WASM = ''' +build_rust_binary( + MANIFEST_PATH Rust/Cargo.toml + CRATE_NAME libwasm_cranelift + BINARY_NAME cranelift-compiler + OUTPUT_PATH_VAR WASM_CRANELIFT_COMPILER_BINARY + FFI_OUTPUT_DIR "${CMAKE_CURRENT_BINARY_DIR}" +) +''' + +# The bare include that makes libwasm_cranelift's header dir opt-in: LibWasm +# spells it with no directory, because CMake's FFI_OUTPUT_DIR is LibWasm's own +# binary dir. The scan must find this rather than a list saying so. +WASM_TU = "#include \n" + def _fixture(): d = tempfile.mkdtemp() @@ -145,10 +173,16 @@ def _fixture(): os.makedirs(os.path.join(d, "Libraries/LibJS/Flap")) with open(os.path.join(d, "Libraries/LibJS/Flap/Cargo.lock"), "w") as f: f.write(FLAP_LOCK) - for lib, text in (("LibGfx", CMAKE_GFX), ("LibWeb", CMAKE_WEB)): + with open(os.path.join(d, "Cargo.toml"), "w") as f: + f.write('[workspace]\nmembers = ["Libraries/LibWasm/Rust"]\n' + 'exclude = ["Libraries/LibJS/Flap"]\n') + for lib, text in (("LibGfx", CMAKE_GFX), ("LibWeb", CMAKE_WEB), + ("LibJS", CMAKE_JS), ("LibWasm", CMAKE_WASM)): os.makedirs(os.path.join(d, "Libraries", lib), exist_ok=True) with open(os.path.join(d, "Libraries", lib, "CMakeLists.txt"), "w") as f: f.write(text) + with open(os.path.join(d, "Libraries/LibWasm/CraneliftBridge.cpp"), "w") as f: + f.write(WASM_TU) return d @@ -330,9 +364,17 @@ def test_colliding_header_names_are_real(): index)) owners = [k for k, v in hdrs.items() if "'RustFFI.h'" in v] assert len(owners) >= 6, owners - driver = _read("Meta/cargo_build.sh") - assert "root-output" in driver, "the header lookup no longer prefers OUT_DIR" - assert "FFI_SCRATCH" in driver, "FFI_OUTPUT_DIR is no longer a scratch dir" + # The lookup lives in cargo_vendor.sh, SHARED by both cargo drivers -- the + # staticlib one and the --bin one -- because a binary crate's build script + # runs cbindgen into the same shared dir and would collide the same way + # (libwasm_cranelift emits CraneliftFFI.h). One copy, so the two cannot drift. + lib = _read("Meta/cargo_vendor.sh") + assert "root-output" in lib, "the header lookup no longer prefers OUT_DIR" + assert "FFI_SCRATCH" in lib, "FFI_OUTPUT_DIR is no longer a scratch dir" + for driver in ("Meta/cargo_build.sh", "Meta/cargo_binary_build.sh"): + txt = _read(driver) + assert "sync_ffi_headers " in txt, driver + assert "root-output" not in txt, "%s has its own copy of the lookup" % driver def test_headers_are_prefixed_so_both_include_spellings_resolve(): @@ -405,7 +447,7 @@ def test_the_emitter_is_idempotent_and_needs_no_cargo_or_network(): mod.emit_crates(crates) mod.emit_index(crates, specs) mod.emit_extension(crates) - mod.emit_ring(crates, specs) + mod.emit_ring(crates, specs, mod.binary_specs()) outs.append(buf.getvalue()) assert outs[0] == outs[1] assert "http_archive(" in outs[0] @@ -434,7 +476,8 @@ def test_check_flag_detects_drift_in_a_generated_file(): {"--crates": lambda: mod.emit_crates(crates), "--index": lambda: mod.emit_index(crates, specs), "--extension": lambda: mod.emit_extension(crates), - "--ring": lambda: mod.emit_ring(crates, specs)}[fl]() + "--ring": lambda: mod.emit_ring(crates, specs, + mod.binary_specs())}[fl]() with open(os.path.join(d, "out", fn), "w") as f: f.write(b.getvalue()) buf = io.StringIO() @@ -475,13 +518,20 @@ def test_nothing_generated_still_reads_the_cmake_cargo_tree(): """ offenders = [] for rel in ("BUILD.bazel", "Libraries/LibWeb/BUILD.bazel", "codegen_root.bzl", - "cargo_ring.bzl", "cargo_index.bzl", "bazelrc.txt", - "Build/full/Libraries/BUILD.bazel"): + "cargo_ring.bzl", "cargo_index.bzl", "bazelrc.txt"): txt = _read(rel) if re.search(r"//Build/full/cargo|Build/full/cargo/build|rust_ffi_headers\b", txt.replace("no rust_ffi_headers", "")): offenders.append(rel) assert not offenders, "still reference CMake's cargo tree: %s" % offenders + # And the stronger statement, which is what finding 35 cost a week to learn: + # the overlay must not contain a Build/full PACKAGE at all. It used to ship + # three (Libraries/, Services/, UI/), globbing CMake's build tree with + # allow_empty = True -- so on a fresh clone they matched nothing, reported + # nothing, and the build died 1,600 actions later on a missing header. A file + # list that CAN be empty proves nothing; a directory that must not exist does. + assert not os.path.exists(os.path.join(_WS, "Build", "full")), \ + "the Build/full shim packages are back" def test_each_crate_links_its_own_archive_and_no_others(): @@ -510,8 +560,16 @@ def test_each_crate_links_its_own_archive_and_no_others(): assert "additional_inputs" in impl, "the archive would not be in the sandbox" ring = _read("cargo_ring.bzl") assert ring.count("cargo_crate(") == 10 - assert ring.count("cargo_lib(") == 10, "one consumable target per crate" + # One consumable target per crate that has something to consume: the 10 + # staticlib crates, plus libwasm_cranelift -- a `--bin` crate, so it has NO + # archive at all and cargo_lib yields a headers-only CcInfo for it (the + # executable is spawned at run time, and travels as `data`). That "archive + # may be None" branch is the shape a binary crate needs, and it is why the + # count is 11 rather than 10. + assert ring.count("cargo_lib(") == 11, "one consumable target per crate" + assert ring.count("cargo_binary(") == 2 assert "cargo_libs(" not in ring, "the shared link group is gone" + assert "if archive == None:" in impl, "a --bin crate has no archive to link" def test_a_library_depends_on_the_crates_it_uses_and_no_others(): @@ -573,12 +631,89 @@ def test_flapc_is_built_by_bazel_and_used_as_the_genrule_tool(): """The last artifact this migration took from CMake.""" ring = _read("cargo_ring.bzl") assert "cargo_binary(" in ring - assert 'bin = "flapc"' in ring + assert re.search(r"bin = ['\"]flapc['\"]", ring) codegen = _read("codegen_root.bzl") assert "tools = ['//:flapc']" in codegen assert "Build/full/bin/flapc" not in codegen +def test_every_build_rust_binary_crate_is_in_the_ring_with_its_header(): + """Finding 35's second bug: a whole crate missing because CMake was not read. + + `Libraries/LibWasm/CMakeLists.txt` was absent from CMAKELISTS and + `build_rust_binary()` was not parsed at all, so libwasm_cranelift -- a crate + that emits a header LibWasm's CraneliftBridge.cpp includes -- was ENTIRELY + ABSENT from the Bazel graph. Nothing said so: the header sat in + `Build/full/Libraries/LibWasm` where a global `-I` reached it, and the binary + was named by an absolute path baked into + `-DWASM_CRANELIFT_COMPILER_PATH=/home/ubuntu/...` -- two host escapes covering + for one missing target, which is why removing the shims was the only thing + that could find it. + + Two halves, both asserted: the CALL SITE is parsed (a `--bin` crate has no + archive, but it can still emit a header, and whether it does is + `FFI_OUTPUT_DIR` at the call site -- not a list written down here), and every + binary crate reaches the checked-in ring with the same two consumable labels + a staticlib crate gets. + """ + mod = _load(_fixture()) + bins = {b["crate"]: b for b in mod.binary_specs()} + assert set(bins) == {"flapc", "libwasm_cranelift"}, sorted(bins) + # flapc's call site has no FFI_OUTPUT_DIR, so it declares no header at all; + # asking for one would fail the action (cargo never writes it). + assert bins["flapc"]["ffi_headers"] == [] + assert bins["flapc"]["manifest"] == "Libraries/LibJS/Flap/Cargo.toml" + # libwasm_cranelift's does, so the header is a DECLARED output -- Bazel + # deletes what nothing declares -- and the bare-include flag comes from + # SCANNING the tree for the directory-less spelling, not from a list. + assert bins["libwasm_cranelift"]["ffi_headers"] == ["CraneliftFFI.h"] + assert bins["libwasm_cranelift"]["ffi_bare_include"] is True + + # And the same two, present in the checked-in ring and consumed by the root + # package. Pinned by count so a third call site cannot appear unnoticed. + ring = _read("cargo_ring.bzl") + root = _read("BUILD.bazel") + assert ring.count("cargo_binary(") == 2, ring.count("cargo_binary(") + for b in bins.values(): + assert re.search(r"bin = ['\"]%s['\"]" % re.escape(b["bin"]), ring), b + if not b["ffi_headers"]: + continue + assert "'%s_lib'" % b["crate"] in ring + assert "//:%s_lib" % b["crate"] in root + assert "implementation_deps = ['//:%s_bare_include']" % b["crate"] in root + # The binary itself is data, not a link input: LibWasm SPAWNS it. + assert "data = ['//:cranelift-compiler']" in root + + +def test_a_bare_include_dir_arrives_only_through_implementation_deps(): + """Finding 35's include-collision bug, in the one form that actually fixed it. + + 8 of the 10 crates emit a header literally named `RustFFI.h`, and 4 TUs + include it bare (`#include `), which CMake allows because + FFI_OUTPUT_DIR is PRIVATE to the owning library. Splitting the unprefixed dir + into its own `cargo_bare_include` target was necessary but NOT sufficient: + CcInfo include dirs also propagate along the C++ dep graph, so LibGfx + inherited LibTextCodec's dir through `LibGfx -> LibTextCodec` and + `YUVData.cpp` failed with `'FFI' does not name a type` -- compiling against + the WRONG crate's header. `implementation_deps` is Bazel's name for exactly + CMake's PRIVATE, and only it stops the propagation. + + Hence: every bare-include target in the tree must be reached through + implementation_deps, never `deps`. One `deps = [...bare_include]` anywhere + silently re-creates the bug for everything downstream of that library. + """ + root = _read("BUILD.bazel") + targets = sorted(set(re.findall(r"//:(\w+_bare_include)", root))) + assert targets, "no crate exposes an unprefixed FFI include dir any more" + for t in targets: + assert "implementation_deps = ['//:%s']" % t in root, t + # ... and nothing anywhere else in the overlay pulls one in publicly. + for rel in ("BUILD.bazel", "Libraries/LibWeb/BUILD.bazel", "cargo_ring.bzl"): + txt = _read(rel) + for m in re.finditer(r"\n deps = \[([^\]]*)\]", txt): + assert "bare_include" not in m.group(1), (rel, m.group(1)) + + def test_the_libweb_package_exports_its_crate_sources(): """Bazel packages cut across the cargo workspace: five crates live under Libraries/LibWeb, which is its own package, and glob() is package-relative -- @@ -612,3 +747,23 @@ def test_every_extension_created_repo_is_named_in_module_bazel(): # Plus the three toolchain components. assert {"rust_rustc_1_96_1", "rust_cargo_1_96_1", "rust_rust_std_1_96_1"} <= named + + +if __name__ == "__main__": + # A runner, because without one this file EXITED 0 WITHOUT RUNNING ANYTHING -- + # a test suite that cannot fail, which is the same bug as a glob that cannot + # fail (finding 35). Found by checking that the suite's output accounted for + # every test file, rather than trusting the exit code. + import sys as _sys, traceback + fns = [v for k, v in sorted(globals().items()) if k.startswith("test_")] + failed = 0 + for fn in fns: + try: + fn() + print(f"PASS {fn.__name__}") + except Exception: + failed += 1 + print(f"FAIL {fn.__name__}") + traceback.print_exc() + print(f"\n{len(fns) - failed}/{len(fns)} passed") + _sys.exit(1 if failed else 0) diff --git a/tests/test_emit_vcpkg.py b/tests/test_emit_vcpkg.py index 6147733..d224c55 100644 --- a/tests/test_emit_vcpkg.py +++ b/tests/test_emit_vcpkg.py @@ -211,3 +211,23 @@ def test_non_mirrored_urls_are_left_as_a_single_entry(): mirror and must not acquire a fabricated one.""" u = "https://github.com/madler/zlib/archive/v1.3.1.tar.gz" assert emit.urls_for(u) == [u] + + +if __name__ == "__main__": + # A runner, because without one this file EXITED 0 WITHOUT RUNNING ANYTHING -- + # a test suite that cannot fail, which is the same bug as a glob that cannot + # fail (finding 35). Found by checking that the suite's output accounted for + # every test file, rather than trusting the exit code. + import sys as _sys, traceback + fns = [v for k, v in sorted(globals().items()) if k.startswith("test_")] + failed = 0 + for fn in fns: + try: + fn() + print(f"PASS {fn.__name__}") + except Exception: + failed += 1 + print(f"FAIL {fn.__name__}") + traceback.print_exc() + print(f"\n{len(fns) - failed}/{len(fns)} passed") + _sys.exit(1 if failed else 0) diff --git a/tests/test_vcpkg_plumbing.py b/tests/test_vcpkg_plumbing.py index 39a6033..e7d9e32 100644 --- a/tests/test_vcpkg_plumbing.py +++ b/tests/test_vcpkg_plumbing.py @@ -141,3 +141,23 @@ def test_static_libs_go_on_the_link_line_by_path(): assert 'root + "/lib/lib" + n + ".a"' in impl # and the shared ones are -l, so the loader's SONAME lookup keeps working assert 'flags.append("-l" + n)' in impl + + +if __name__ == "__main__": + # A runner, because without one this file EXITED 0 WITHOUT RUNNING ANYTHING -- + # a test suite that cannot fail, which is the same bug as a glob that cannot + # fail (finding 35). Found by checking that the suite's output accounted for + # every test file, rather than trusting the exit code. + import sys as _sys, traceback + fns = [v for k, v in sorted(globals().items()) if k.startswith("test_")] + failed = 0 + for fn in fns: + try: + fn() + print(f"PASS {fn.__name__}") + except Exception: + failed += 1 + print(f"FAIL {fn.__name__}") + traceback.print_exc() + print(f"\n{len(fns) - failed}/{len(fns)} passed") + _sys.exit(1 if failed else 0) From 0737b3d3d31f636f7b2bc86de799a8f18d9edc03 Mon Sep 17 00:00:00 2001 From: Ulf Adams Date: Sat, 8 Aug 2026 10:19:41 +0000 Subject: [PATCH 06/59] tests: one runner that discovers the files and fails when a file goes silent No -- running individual tests does not scale, and the way it failed here is instructive. SKILL.md named six of the eleven test files and chained them with `&&`, so it stopped at the first failure and never reached the rest; the README named the same six. Three of the omitted files had no `if __name__` block, so running them defined 46 tests, called none, and exited 0. Adding a runner to each file was necessary but is NOT the fix: a per-file runner is precisely the thing nobody notices the absence of. What was missing is one entry point that knows how many test FILES exist and how many tests each contributed. tests/run_all.py discovers tests/test_*.py and fails the run if a file defines no tests, will not import, or if the glob matches nothing at all -- the allow_empty = False of test discovery. 135 tests over 11 files in 0.3s, one exit code, filterable (`run_all.py -v cargo`); each file still runs standalone for a single-file loop. Its three guards are tested by TRIGGERING them (test_run_all.py, 7 tests over a throwaway tests/ dir, checking the exit code a gate would read) -- a guard never seen to fire is back where this started. Writing that test found a real trap: the obvious version ran the runner as a subprocess from inside the directory the runner scans, so the runner ran the test that ran the runner and the suite hung. It now calls test_files() directly, which is also the more precise assertion. The general lesson, recorded in the case study: the unit that must be accounted for is the CONTAINER, not the item. Counting passing tests cannot detect a missing file, counting matched files cannot detect an empty glob, and counting green actions cannot detect a header supplied by a shim. Next step as this grows is a py_test per file under `bazel test //...` for caching, parallelism and a commit gate that runs without anyone remembering to -- this repo has no MODULE.bazel yet, so that is a bigger change and I have left it for Ulf's call. --- README.md | 32 ++++++-- SKILL.md | 11 ++- docs/CASE-ladybird-migration.md | 20 ++++- tests/run_all.py | 133 +++++++++++++++++++++++++++++++ tests/test_run_all.py | 134 ++++++++++++++++++++++++++++++++ 5 files changed, 316 insertions(+), 14 deletions(-) create mode 100644 tests/run_all.py create mode 100644 tests/test_run_all.py diff --git a/README.md b/README.md index d5d9394..4658094 100644 --- a/README.md +++ b/README.md @@ -318,6 +318,8 @@ scripts/ triage.py groups diff.json into a systematic-cause worklist serialize.py model ↔ JSON (the contract between stages) tests/ + run_all.py discovers and runs every test_*.py; fails on a silent file + test_run_all.py the runner's own guards, triggered rather than asserted test_engine.py diff/canonicalize/roles/config/TU-set behavior test_extractors.py full pipeline on synthetic File-API + aquery fixtures test_maven.py Maven frontend + Java branch of reconstruct @@ -394,16 +396,30 @@ Run as a skill, Claude drives the generation and triage/fix loop automatically. ## Tests ```bash -for f in tests/test_*.py; do python3 "$f" || echo "FAILED $f"; done +python3 tests/run_all.py # the whole suite, one exit code (~0.3s) +python3 tests/run_all.py -v cargo # verbose, filtered by file or test name ``` -Run the glob, not a hand-kept list of files: three of these test files once had -no `if __name__ == "__main__"` runner at all, so running them **exited 0 having -executed nothing** — a test suite that cannot fail, which is the same bug as a -`glob(..., allow_empty = True)` over a directory that is not there (case study -finding 35). Each file prints `PASS`/`FAIL` per test and an `N/N passed` line, so -compare the totals against the number of `def test_` in the file rather than -trusting the exit code. +`run_all.py` **discovers** `tests/test_*.py` rather than reading a list, and it +fails the run if a test *file* contributed nothing — no tests defined, or a module +that would not import. Both of those are the reason it exists: this README used to +list six commands for ten files, and three of those files had no +`if __name__ == "__main__"` block at all, so running them imported the module, +defined 46 test functions, called none of them, and exited 0. There is no pytest +here, so nothing else called them either. + +That is the same bug as `glob(..., allow_empty = True)` over a directory that is +not there (case study finding 35): **a check that cannot fail is indistinguishable +from one that is not needed.** A per-file runner is what was missing, but a +per-file runner is also exactly what nobody notices the absence of — so the fix is +one entry point that knows how many files there are and how many tests each +contributed. Its three guards are themselves tested, by triggering them +(`test_run_all.py`). + +Each file also still runs standalone (`python3 tests/test_engine.py`) for a quick +single-file loop. As the suite grows the next step is a `py_test` per file under +`bazel test //...`, so the caching and parallelism come for free and a commit gate +runs it without anyone remembering to — this repo has no `MODULE.bazel` yet. The extractor tests run against fixtures under `tests/` that mirror the documented File API, aquery, Maven argfile, and npm-NDJSON schemas. Real projects diff --git a/SKILL.md b/SKILL.md index 9e6b7db..ad7924a 100644 --- a/SKILL.md +++ b/SKILL.md @@ -392,11 +392,16 @@ python3 scripts/diff_ts.py model.npm.json model.bazel.json # standalone check ## Tests ```bash -python3 tests/test_engine.py && python3 tests/test_extractors.py \ - && python3 tests/test_maven.py && python3 tests/test_extract_npm.py \ - && python3 tests/test_triage.py && python3 tests/test_configure.py +python3 tests/run_all.py # discovers every tests/test_*.py, one exit code ``` +Do not hand-list the files. This block used to name six of the eleven and chain +them with `&&`, so it stopped at the first failure and never reached the rest — +and three of the files it omitted had no `if __name__ == "__main__"` block, so +running them defined 46 tests, called none, and exited 0. `run_all.py` discovers +the files and **fails if any file contributed no tests**, which is the +`allow_empty = False` of test discovery (case study finding 35). + Extractor tests run against fixtures that mirror the documented File API, aquery, Maven argfile, and npm-NDJSON schemas. When a real project surfaces a schema detail the extractors mishandle (fragment quoting, `external/` repo paths diff --git a/docs/CASE-ladybird-migration.md b/docs/CASE-ladybird-migration.md index 90ebe2e..22a756a 100644 --- a/docs/CASE-ladybird-migration.md +++ b/docs/CASE-ladybird-migration.md @@ -1651,9 +1651,23 @@ FFI-header lookup had moved into `cargo_vendor.sh` so both drivers use one copy, and one test read `Build/full/Libraries/BUILD.bazel` — a file whose *deletion* was the fix. That last one is the pleasing part: the test failed because the thing it asserted about no longer exists, so it became the stronger assertion that the path -**must not** exist. The suite is now 128 tests over 10 files, all passing, and the -README documents running the glob rather than a hand-kept list of filenames, -because a list of files to run is one more thing that can silently omit an entry. +**must not** exist. + +The repair is not "add a runner to each file", though that is what was missing: a +per-file runner is precisely the thing nobody notices the absence of, so doing only +that leaves the next silent file undetected. What was missing was **one entry point +that knows how many test files exist and how many tests each contributed**, so a +file going quiet is a failure and not just a smaller number nobody was counting. +`tests/run_all.py` discovers `tests/test_*.py`, and fails the run if any file +defines no tests or will not import — the `allow_empty = False` of test discovery. +Its guards are tested by triggering them, because a guard never seen to fire is +back where this started. The suite is 135 tests over 11 files, 0.3s, one exit code. + +The general lesson, which outlives this repo's test layout: **the unit that has to +be accounted for is the CONTAINER, not the item.** Counting passing tests cannot +detect a missing file; counting matched files cannot detect an empty glob; +counting green actions cannot detect a header supplied by a shim. Each time, the +fix was to make the enclosing thing declare how many children it should have. ## Plan for the rest diff --git a/tests/run_all.py b/tests/run_all.py new file mode 100644 index 0000000..168bf71 --- /dev/null +++ b/tests/run_all.py @@ -0,0 +1,133 @@ +#!/usr/bin/env python3 +"""Run every test in tests/, and fail if a test FILE contributed nothing. + +Why this exists rather than a list of commands in the README: for most of this +repo's life the tests were ten hand-run files, and the README listed six of them. +Three of the ten (`test_emit_cargo.py`, `test_emit_vcpkg.py`, +`test_vcpkg_plumbing.py`) had no `if __name__ == "__main__"` block at all, so +running them imported the module, defined 46 test functions, called none of them, +and exited 0. There is no pytest in this environment, so nothing else called them +either. + +That is the same bug as `glob(..., allow_empty = True)` over a directory that is +not there, and as the `Build/full` shim packages in the Ladybird example (case +study finding 35): **a check that cannot fail is indistinguishable from one that +is not needed.** The lesson is not "add a runner to each file" -- that is what was +missing, but a per-file runner is exactly what nobody notices the absence of. The +lesson is that the suite needs ONE thing that knows how many test files there are +and how many tests each one contributed, so a file going silent is a FAILURE +rather than a smaller number nobody was counting. + +Hence the two rules enforced here, in this order: + + 1. Every `tests/test_*.py` is imported. Import failure is a failure, not a skip + -- a module that cannot be imported reported zero tests before. + 2. A file that yields **zero** tests is a failure. This is the guard that the + original bug would have tripped, and it costs one comparison. + +Then the tests run, in one process, with one exit code. + +Usage: python3 tests/run_all.py [-v] [name-substring ...] +""" + +import importlib.util +import os +import sys +import traceback + +HERE = os.path.dirname(os.path.abspath(__file__)) + + +def test_files(): + """Every test module, DISCOVERED rather than listed. + + A hand-kept list is one more thing that can silently omit an entry, which is + how the README came to name six of the ten files. + """ + return sorted(f for f in os.listdir(HERE) + if f.startswith("test_") and f.endswith(".py")) + + +def load(fn): + path = os.path.join(HERE, fn) + spec = importlib.util.spec_from_file_location(fn[:-3], path) + mod = importlib.util.module_from_spec(spec) + spec.loader.exec_module(mod) + return mod + + +def tests_in(mod): + """The module's test callables, in source order. + + Source order, not alphabetical: these files are written to be read top to + bottom (the fixture first, then the contract it pins), so a failure list in + file order is the one that matches the reader's mental model. + """ + fns = [(name, obj) for name, obj in vars(mod).items() + if name.startswith("test_") and callable(obj) + and getattr(obj, "__module__", None) == mod.__name__] + return sorted(fns, key=lambda nf: getattr(nf[1], "__code__").co_firstlineno) + + +def main(argv): + verbose = "-v" in argv + filters = [a for a in argv if not a.startswith("-")] + + files = test_files() + if not files: + # The discovery equivalent of allow_empty = False: an empty tests/ means + # the glob is wrong or the layout moved, never that there is nothing to do. + print("FAIL: no tests/test_*.py files found at all") + return 1 + + total = passed = 0 + silent, broken, failures = [], [], [] + for fn in files: + try: + mod = load(fn) + except Exception: + # An unimportable module used to report zero tests and (with no + # runner) exit 0. It is a failure. + broken.append(fn) + print("ERROR %s could not be imported" % fn) + traceback.print_exc() + continue + cases = tests_in(mod) + if not cases: + # THE guard this file exists for. + silent.append(fn) + print("FAIL %s defines no tests" % fn) + continue + selected = [(n, f) for n, f in cases + if not filters or any(s in n or s in fn for s in filters)] + if not selected: + continue + n_fail = 0 + for name, fn_ in selected: + total += 1 + try: + fn_() + passed += 1 + if verbose: + print(" PASS %s::%s" % (fn, name)) + except Exception: + n_fail += 1 + failures.append("%s::%s" % (fn, name)) + print(" FAIL %s::%s" % (fn, name)) + traceback.print_exc() + print("%-24s %d/%d" % (fn, len(selected) - n_fail, len(selected))) + + print() + print("%d/%d tests passed across %d files" % (passed, total, len(files))) + if failures: + print("failed: %s" % ", ".join(failures)) + if silent: + print("files defining NO tests (this is the bug finding 35 is about): %s" + % ", ".join(silent)) + if broken: + print("files that could not be imported: %s" % ", ".join(broken)) + return 1 if (failures or silent or broken) else 0 + + +if __name__ == "__main__": + sys.exit(main(sys.argv[1:])) diff --git a/tests/test_run_all.py b/tests/test_run_all.py new file mode 100644 index 0000000..bcc54b0 --- /dev/null +++ b/tests/test_run_all.py @@ -0,0 +1,134 @@ +"""The runner's own guards, exercised rather than asserted. + +run_all.py exists because three test files silently ran zero tests (case study +finding 35). Its whole value is three guards -- a file with no tests fails, a file +that will not import fails, a failing test fails -- and a guard that has never +been seen to fire is exactly the thing this repo keeps getting caught by. So each +one is triggered here against a throwaway tests/ directory. + +Note what is NOT done: this does not import run_all and inspect it. It runs it as +a subprocess and checks the EXIT CODE, because the exit code is what a commit gate +or CI reads, and an exit code is precisely what the original bug got wrong (0 while +running nothing). +""" + +import os +import shutil +import subprocess +import sys +import tempfile + +_RUNNER = os.path.join(os.path.dirname(os.path.abspath(__file__)), "run_all.py") + + +def _run(files): + """Run the runner over a temp tests/ dir containing exactly `files`. + + A temp dir rather than the real one so a guard can be triggered without a + file that breaks the actual suite ever existing on disk. + """ + d = tempfile.mkdtemp() + try: + runner = os.path.join(d, "run_all.py") + shutil.copy(_RUNNER, runner) + for name, text in files.items(): + with open(os.path.join(d, name), "w") as f: + f.write(text) + p = subprocess.run([sys.executable, runner], capture_output=True, text=True) + return p.returncode, p.stdout + p.stderr + finally: + shutil.rmtree(d) + + +def test_a_passing_file_passes_and_reports_its_count(): + rc, out = _run({"test_ok.py": "def test_a(): pass\ndef test_b(): pass\n"}) + assert rc == 0, out + assert "2/2 tests passed across 1 files" in out + + +def test_a_file_that_defines_no_tests_is_a_failure(): + """THE bug: 46 test functions defined in files nobody ran, exit code 0. + + A module with no test callables is not "nothing to do" -- it is a file whose + contents stopped being reachable, which is what happened here for three files + and what `allow_empty = True` did for the Build/full shims. + """ + rc, out = _run({"test_silent.py": "def helper(): pass\n"}) + assert rc != 0, out + assert "defines no tests" in out + + +def test_a_module_that_cannot_be_imported_is_a_failure_not_a_skip(): + """An unimportable module reported zero tests before, which looked like zero + failures. Import errors are results, not absences.""" + rc, out = _run({"test_broken.py": "import nonexistent_xyz\ndef test_a(): pass\n"}) + assert rc != 0, out + assert "could not be imported" in out + + +def test_a_failing_test_fails_the_run_and_is_named(): + rc, out = _run({"test_bad.py": "def test_a(): assert False, 'boom'\n", + "test_ok.py": "def test_b(): pass\n"}) + assert rc != 0, out + assert "test_bad.py::test_a" in out + # The other file still ran: one bad file must not mask the rest. + assert "1/2 tests passed" in out + + +def test_an_empty_tests_directory_is_a_failure(): + """The discovery equivalent of allow_empty = False. If the glob matches + nothing, the glob is wrong -- it never means there is nothing to test.""" + rc, out = _run({}) + assert rc != 0, out + assert "no tests/test_*.py files found" in out + + +def test_helpers_and_imported_functions_are_not_counted_as_tests(): + """A `test_`-prefixed name imported FROM another module is not this file's + test; counting it would double-count and, worse, make a file look non-silent + because of somebody else's tests.""" + rc, out = _run({ + "test_one.py": "def test_a(): pass\n", + "test_two.py": "from test_one import test_a\ndef test_b(): pass\n", + }) + assert rc == 0, out + assert "2/2 tests passed across 2 files" in out + + +def test_the_real_suite_is_discovered_whole(): + """Every test file in this repo is reached by discovery -- the check that the + README's hand-kept list of six filenames could not make. + + This calls `test_files()` directly instead of running the runner on the real + tests/ directory, and the reason is a trap worth recording: this file IS in + that directory, so a subprocess here would run the runner, which would run + this test, which would run the runner... The first version of this test hung + the suite. Discovery is the thing being claimed, and `test_files()` is + discovery, so the direct call is also the more precise assertion. + """ + sys.path.insert(0, os.path.dirname(_RUNNER)) + try: + import run_all + finally: + sys.path.pop(0) + here = os.path.dirname(os.path.abspath(__file__)) + on_disk = sorted(f for f in os.listdir(here) + if f.startswith("test_") and f.endswith(".py")) + assert run_all.test_files() == on_disk + assert os.path.basename(__file__) in on_disk, "the runner's own tests" + + +if __name__ == "__main__": + import traceback + fns = [v for k, v in sorted(globals().items()) if k.startswith("test_")] + failed = 0 + for fn in fns: + try: + fn() + print(f"PASS {fn.__name__}") + except Exception: + failed += 1 + print(f"FAIL {fn.__name__}") + traceback.print_exc() + print(f"\n{len(fns) - failed}/{len(fns)} passed") + sys.exit(1 if failed else 0) From dcfdabc63e9310cc77a325914cd401425c5c8523 Mon Sep 17 00:00:00 2001 From: Ulf Adams Date: Sat, 8 Aug 2026 11:46:48 +0000 Subject: [PATCH 07/59] =?UTF-8?q?Ladybird:=20no,=20a=20fresh=20clone=20doe?= =?UTF-8?q?s=20not=20build=20=E2=80=94=20six=20reasons,=20four=20fixed?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit You asked whether you can clone and build. I stopped answering from the README and ran it: git clone into an empty directory, overlay in, type the command. It fails six times, and five of the six have nothing to do with Build/full — which was the dependency I had been arguing about, so it is the one I verified. Finding 36 is the autopsy; examples/ladybird/README.md now leads with the table instead of the claim. Fixed here: (1) two path bugs in vcpkg_build.sh — the distfile index AND its entries were execroot-relative while vcpkg invokes the asset script from its own cwd, so every lookup missed, reported as `no asset cache hits` and x-block-origin refusing the network, i.e. a message blaming the pin for a path bug. Hidden because the dev checkout already had downloads/tools/cmake, so vcpkg never asked. (2) `pip install ply` in the angle port: not an asset-cache download, so the 76-distfile pin never covered it, and nothing blocked it either — requires-network 0 is a scheduling hint, no-sandbox means there is no namespace to enforce it, and use_default_shell_env handed the action this sandbox's HTTP_PROXY. Now pinned by URL+sha256 as an http_file, declared as an action input, staged into a find-links dir, with PIP_NO_INDEX and the proxy unset; added to use_repo and to the emitter's --use-repo so a regeneration cannot drop the one input no instrument can capture. (3) the git-archive staging was `if [ -d ]; then cp 2>/dev/null || true; fi` — three ways to succeed while copying nothing — and vcpkg_git_archives.bzl is generated, committed, documented and loaded by NOTHING; the four tarballs only existed because I made that directory by hand. Now a hard error in 4s naming them instead of skia dying 20min in on a googlesource URL. (4) vcpkg scratch is cleaned on INT/TERM too and reports free space, since a small /tmp surfaces as `No space left on device` from the asset script. Documented, not fixed: Build/vcpkg is still an allow_empty=True glob over a checkout a clone lacks (finding 35's pattern one tree over, matching exactly one file — its own BUILD.bazel); that checkout's .git is load-bearing for `git read-tree` yet EXCLUDED by the filegroup, which no-sandbox lets the action read anyway — an excluded input the build needs is worse than an undeclared one, because the exclusion looks deliberate; and the HSTS preload table is an unversioned download from Chromium's main with no hash to pin. All three are now step 1a of the recipe with the reason each stayed invisible. Tests: 6 new in test_vcpkg_plumbing (142 total). One of them failed only in the full run — another test points $LADYBIRD_ROOT at a fixture and my subprocess inherited it. run_all.py running everything in one process is what surfaced that; the per-file runners could not have. --- README.md | 7 +- SKILL.md | 3 +- docs/CASE-ladybird-migration.md | 151 +++++++++++-- examples/ladybird/README.md | 124 +++++++++-- examples/ladybird/workspace/BUILD.bazel | 5 + examples/ladybird/workspace/MODULE.bazel | 8 + .../workspace/Meta/cargo_binary_build.sh | 0 .../ladybird/workspace/Meta/cargo_build.sh | 0 .../ladybird/workspace/Meta/cargo_vendor.sh | 0 .../workspace/Meta/emit_vcpkg_bazel.py | 31 +++ .../ladybird/workspace/Meta/vcpkg_build.sh | 96 +++++++- examples/ladybird/workspace/vcpkg.bzl | 18 +- .../ladybird/workspace/vcpkg_extension.bzl | 2 + .../workspace/vcpkg_python_packages.bzl | 43 ++++ tests/test_configure.py | 19 +- tests/test_diff_ts.py | 19 +- tests/test_emit_cargo.py | 26 +-- tests/test_emit_vcpkg.py | 26 +-- tests/test_engine.py | 22 +- tests/test_extract_npm.py | 19 +- tests/test_extractors.py | 19 +- tests/test_maven.py | 19 +- tests/test_run_all.py | 22 +- tests/test_triage.py | 19 +- tests/test_vcpkg_plumbing.py | 206 ++++++++++++++++-- 25 files changed, 712 insertions(+), 192 deletions(-) mode change 100644 => 100755 examples/ladybird/workspace/Meta/cargo_binary_build.sh mode change 100644 => 100755 examples/ladybird/workspace/Meta/cargo_build.sh mode change 100644 => 100755 examples/ladybird/workspace/Meta/cargo_vendor.sh create mode 100644 examples/ladybird/workspace/vcpkg_python_packages.bzl diff --git a/README.md b/README.md index 4658094..e6b0dbe 100644 --- a/README.md +++ b/README.md @@ -416,8 +416,11 @@ one entry point that knows how many files there are and how many tests each contributed. Its three guards are themselves tested, by triggering them (`test_run_all.py`). -Each file also still runs standalone (`python3 tests/test_engine.py`) for a quick -single-file loop. As the suite grows the next step is a `py_test` per file under +Run one file or one test with a substring filter (`run_all.py test_triage`, +`run_all.py -v cargo`). The per-file `if __name__` runners are gone: in one file +that block had ended up **mid-file**, so four tests appended after it were defined, +never called, and the file still reported `6/6 passed`. A test's position in the +file should not decide whether it runs. As the suite grows the next step is a `py_test` per file under `bazel test //...`, so the caching and parallelism come for free and a commit gate runs it without anyone remembering to — this repo has no `MODULE.bazel` yet. diff --git a/SKILL.md b/SKILL.md index ad7924a..b67d07e 100644 --- a/SKILL.md +++ b/SKILL.md @@ -400,7 +400,8 @@ them with `&&`, so it stopped at the first failure and never reached the rest and three of the files it omitted had no `if __name__ == "__main__"` block, so running them defined 46 tests, called none, and exited 0. `run_all.py` discovers the files and **fails if any file contributed no tests**, which is the -`allow_empty = False` of test discovery (case study finding 35). +`allow_empty = False` of test discovery (case study finding 35). Run one file with +a substring filter: `python3 tests/run_all.py test_triage`. Extractor tests run against fixtures that mirror the documented File API, aquery, Maven argfile, and npm-NDJSON schemas. When a real project surfaces a diff --git a/docs/CASE-ladybird-migration.md b/docs/CASE-ladybird-migration.md index 22a756a..4568103 100644 --- a/docs/CASE-ladybird-migration.md +++ b/docs/CASE-ladybird-migration.md @@ -8,12 +8,20 @@ generated BUILD files — is in [`examples/ladybird/`](../examples/ladybird/), along with an honest inventory of what still stops it from being a clone-and-build. -`git clone && bazel build //:ladybird` now builds the browser with **nothing** -read out of CMake's build tree. That was claimed prematurely and was false for -most of this migration; [finding 35](#finding-35-the-claim-was-false-and-a-glob-is-why-nobody-noticed) -is the autopsy, and it is the finding to read first if you are migrating a project -of your own — the bug was in the *shape of the verification*, not in the Bazel -rules. +Bazel now builds the browser with **nothing** read out of CMake's build tree +(`Build/full`) — 0 targets in the closure of all six binaries, down from 741. That +was claimed prematurely and was false for most of this migration; +[finding 35](#finding-35-the-claim-was-false-and-a-glob-is-why-nobody-noticed) is +the autopsy, and it is the finding to read first if you are migrating a project of +your own — the bug was in the *shape of the verification*, not in the Bazel rules. + +**`git clone && bazel build` on a fresh clone still fails, though, and finding 36 +is the second autopsy.** Removing `Build/full` was the goal, so `Build/full` is +what I verified; the clone needs four *other* things nobody had asked for — a +vcpkg checkout (globbed with `allow_empty = True`, the same pattern one tree over), +its `.git`, an unpinned HSTS table, and `pip install ply` inside a vcpkg port that +`x-block-origin` never sees. **The only check that finds all of them is doing the +clone**, which is finding 36's one-line summary and took two goes to learn. ## Why Ladybird @@ -1435,10 +1443,11 @@ Both remaining debts are over-declaration, and they are honest: - Each `vcpkg_lib` still declares the whole vcpkg tree as its header input (finding 33's leftover). Same shape: paths are per-port, the input *set* is not. -`git clone && bazel build //:ladybird` is now true for the browser. `Build/full` -is still needed to *regenerate* the BUILD files and to run the parity harness — -that is a converter-development dependency, not a build dependency, and it is a -different claim from the one this ring closed. +Nothing in the emitted build reads `Build/full` any more. It is still needed to +*regenerate* the BUILD files and to run the parity harness — a converter-development +dependency, not a build dependency, and a different claim from the one this ring +closed. (It is also a different claim from "a fresh clone builds", which is still +false for unrelated reasons — finding 36.) **That claim was still false when I wrote it, and finding 35 is the autopsy.** I had closed the two *binary* dependencies (vcpkg, Rust) and concluded the tree was @@ -1669,6 +1678,118 @@ detect a missing file; counting matched files cannot detect an empty glob; counting green actions cannot detect a header supplied by a shim. Each time, the fix was to make the enclosing thing declare how many children it should have. +## Finding 36: I removed the dependency I was looking for, so that is the one I found + +Ulf asked, again: *can I clone and build with Bazel?* Finding 35 had just closed the +741 `Build/full` header dependencies, `cquery` returned 0, and the README said yes. +This time I did not answer from the README. I ran `git clone` into an empty +directory, dropped the overlay in, and typed the command. + +**It failed. Six times, for six different reasons, and five of them have nothing to +do with `Build/full`.** + +| What was missing | The error a cloner gets | Why my machine hid it | +|---|---|---| +| `Build/vcpkg` — a microsoft/vcpkg checkout at `vcpkg.json`'s `builtin-baseline` | `/tmp/.../root/vcpkg: No such file or directory` | `Meta/ladybird.py vcpkg` had been run months earlier | +| its `.git` (120 MB), which vcpkg needs for `git read-tree` to resolve versioned ports — and which the filegroup **excludes** | `fatal: not a git repository: '.git'` … `while checking out port sqlite3` | the action is `no-sandbox`, so it read the real checkout regardless of what was declared | +| `Build/caches/HSTSPreload/transport_security_state_static.json`, an **unversioned** download CMake does at configure time from Chromium's `main` | `missing input file '//:Build/caches/...'` | CMake's configure had already fetched it | +| two path bugs in `Meta/vcpkg_build.sh`: the distfile index *and its entries* were execroot-relative, and vcpkg runs the asset script from its own cwd | `awk: cannot open ...`, then `cp: cannot stat ...`, surfacing as `no asset cache hits` and `x-block-origin blocks trying the authoritative source` | the checkout already had `downloads/tools/cmake-4.4.0-linux`, so vcpkg never *asked* the script for a tool | +| `ply`, which the `angle` port installs with **`pip install ply`** | `No matching distribution found for ply` | this sandbox exports `HTTP_PROXY`, and the action inherits it | +| the four `vcpkg_from_git` archives, staged from a directory I had made **by hand** — `vcpkg_git_archives.bzl` is generated, committed, documented, and **loaded by nothing** | 20 minutes in: `git fetch https://android.googlesource.com/.../piex.git … Error code: 128` | the directory existed on my disk from when I built the pin | + +### Three of them are the same bug as finding 35, in three different syntaxes + +`//Build/vcpkg:tree` is `glob(["**"], allow_empty = True)` over a directory a fresh +clone does not have. It matches exactly one file — *its own `BUILD.bazel`* — reports +nothing, and the build dies later somewhere else. That is the finding-35 pattern +verbatim; I had deleted the three `Build/full` shims and left a fourth shim, over a +different tree, in place. **I had been looking for `Build/full`, so `Build/full` is +what I removed.** + +The git-archive staging is the same idea written in bash, and it is worth quoting +because it packs three independent ways to succeed while doing nothing into four +lines: + +```bash +if [ -d "$SRC/Meta/CMake/vcpkg/git-archives" ]; then # skips + cp "$SRC/Meta/CMake/vcpkg/git-archives/"*.tar.gz "$ROOT/downloads/" \ + 2>/dev/null || true # hides, forgives +fi +``` + +A directory test that skips silently, a redirect that swallows the error, and a +`|| true` that forgives the exit code. The four tarballs it was supposed to place +only ever existed because *I* had made that directory while capturing the pin. +`vcpkg_git_archives.bzl` — generated, checked in, listed in the README's file table +— is loaded by **nothing**; it is documentation wearing a `.bzl` extension. Without +those files skia gets 20 minutes in and dies fetching `piex.git` from +android.googlesource.com, naming neither the directory nor the tarball nor the port +that pinned it. It now fails in four seconds saying exactly what is missing and +where it comes from. + +The `.git` case is worse than a missing input, because it is a *lie in the +declaration*: the filegroup explicitly excludes `.git/**`, the port resolution +genuinely requires it, and the build works anyway because `no-sandbox: "1"` lets the +action read the real path instead of the declared inputs. An excluded input that the +action reads is strictly worse than an undeclared one — the exclusion looks like a +decision. + +### The fifth is a hole in a claim I had "verified" + +"The 77 vcpkg ports are built with **zero network access**" was measured with +`x-block-origin`, and that measurement is sound as far as it goes: remove a distfile +from the index and the build hard-fails instead of fetching. But `x-block-origin` +governs **vcpkg's own downloader** and nothing else. The `angle` overlay-port calls +`x_vcpkg_get_python_packages`, which runs `pip install ply` — not a distfile, not an +asset, never seen by the pin. + +And nothing stopped it, because I had written the enforcement as a *label*: + +```python +execution_requirements = {"local": "1", "no-sandbox": "1", "requires-network": "0"} +``` + +`requires-network: "0"` is a scheduling hint. It does not build a network namespace, +and `no-sandbox: "1"` guarantees there is none to build. With +`use_default_shell_env = True` the action inherits this sandbox's `HTTP_PROXY`, so +pip quietly succeeded for months. **A control that is not enforced is +indistinguishable from a control that is not there** — the same sentence as finding +35's glob, applied to an `execution_requirements` key instead of a `glob()` +argument. + +The fix needs no patch to the portfile, because pip has supported offline switches: +the wheel is pinned by URL and sha256 (`vcpkg_python_packages.bzl` → `http_file`), +declared as an input of the vcpkg action, staged into a find-links directory, and pip +runs with `PIP_NO_INDEX=1` and the proxy variables unset. An unpinned package is then +`No matching distribution found` — an error, not a download, which is precisely the +property `x-block-origin` gives the other 76. Two details are worth keeping: the URL +is `files.pythonhosted.org`'s content-addressed path (immutable for a version, unlike +`pip install ply`, which resolves against whatever PyPI serves today), and the wheel +had to be added to `use_repo` *and* to the emitter's `--use-repo` output — it is the +one vcpkg input no instrument can capture, so it is also the one a regeneration can +silently drop. + +The other half of the fix is not code: **verify in an environment with no route to +the network.** A flag asserting there is no network is exactly the kind of evidence +this migration keeps getting wrong. + +### What generalizes + +Three times now the same shape: a `glob` that cannot fail, test files that ran no +tests, and an `execution_requirements` key that enforces nothing. Each one was +*green*, and green was the problem. What is worth taking from finding 36 specifically +is narrower and more uncomfortable: + +**Verification finds what it is aimed at.** "Verified by removal" is the strongest +check in this document, and it is still only a test of *what you remove*. I removed +`Build/full` because `Build/full` was the thing I had been arguing about; a clone +needs `Build/vcpkg`, a `.git`, an HSTS table and a Python package, and no amount of +rigor about `Build/full` was ever going to mention them. The check that finds all +five is not a better `cquery` or a wider removal — it is **the actual user's first +command, run the way the user runs it, in a directory that has never seen this +project.** That is one line of shell, it costs ten minutes, and I should have run it +before the first time I said yes. + ## Plan for the rest - **Ring 1c — BUILD generation to compile+link parity, bottom-up.** AK → @@ -1734,9 +1855,11 @@ every process (UI, WebContent, Compositor, RequestServer, ImageDecoder) Bazel-built, proven by removing the reference services and watching the CMake binary fail while the Bazel one renders. -And with findings 33, 34 **and 35** it is a **clone-and-build**: Bazel fetches and -builds the 77 vcpkg ports, the 10 Rust crates, `flapc`, `cranelift-compiler` and -every generated header itself, with zero network access in the build actions. +And with findings 33, 34 **and 35**, Bazel fetches and builds the 77 vcpkg ports, +the 10 Rust crates, `flapc`, `cranelift-compiler` and every generated header itself, +with vcpkg's own downloader reaching the network zero times. It is **not** yet a +clone-and-build: finding 36 lists the four inputs a fresh clone still lacks and the +one vcpkg port that bypasses the pin with `pip`. `Build/full` is now only the *converter's* input — the model the emitters read and the baseline the parity harness diffs — not the build's, and this time that is measured: **0 targets under `Build/full` in the `cquery` closure of all six @@ -1753,7 +1876,7 @@ instructive part of this migration — see finding 35. from scratch (2,700 C++ plus the vcpkg and Rust rings); LibWeb alone 1,961 TUs (1,273 checked-in + 688 generated) with **zero** define/flag/include discrepancies vs CMake; 1,379/1,379 generated files -byte-identical -> now 1,408/1,408 with every one of the build's 586 ninja CUSTOM_COMMANDs accounted for (findings 25-27); 77 vcpkg ports and 154 crates.io crates fetched and built by Bazel; **0 targets under `Build/full` in the closure of all six binaries (down from 741) and 0 absolute host paths in the emitted BUILD files** (finding 35); 35 findings, 3 of them real any2bazel engine fixes with +byte-identical -> now 1,408/1,408 with every one of the build's 586 ninja CUSTOM_COMMANDs accounted for (findings 25-27); 77 vcpkg ports and 154 crates.io crates fetched and built by Bazel; **0 targets under `Build/full` in the closure of all six binaries (down from 741) and 0 absolute host paths in the emitted BUILD files** (finding 35); a fresh clone still needs 4 inputs the overlay does not carry and one port reaches PyPI via pip (finding 36 — found by actually cloning); 36 findings, 3 of them real any2bazel engine fixes with regression tests. **One number in this scoreboard was wrong for most of the migration, and it is diff --git a/examples/ladybird/README.md b/examples/ladybird/README.md index ee4fe1a..405a01b 100644 --- a/examples/ladybird/README.md +++ b/examples/ladybird/README.md @@ -2,7 +2,7 @@ The Bazel workspace overlay produced by the any2bazel parity loop for [Ladybird](https://ladybird.org) (CMake + vcpkg, C++23). The narrative — why -each of these files looks the way it does, and the 34 findings the migration +each of these files looks the way it does, and the 36 findings the migration produced — is in [`docs/CASE-ladybird-migration.md`](../../docs/CASE-ladybird-migration.md). This directory is the *artifact*: what you would drop into a Ladybird checkout. @@ -18,14 +18,48 @@ zero network access** (findings 30–33), and **so are the 10 Rust crates, `flap and `cranelift-compiler`** — 154 crates.io crates fetched from `Cargo.lock`, built by network-blocked cargo actions (findings 34–35). -**`git clone && bazel build //:ladybird` builds the browser, and nothing in the -emitted build reads CMake's build tree.** Verified by removal: with -`Build/full/{Libraries,Services,UI,bin,cargo}` moved off the machine — every -Ladybird-generated header and every binary CMake produced — all six binaries -build from scratch and render `--headless=text` **and** `--headless=layout-tree` -byte-identically to the CMake reference. `Build/full` is still needed to -*regenerate* the BUILD files and to run the parity harness — a -converter-development dependency, not a build dependency. +**Nothing in the emitted build reads CMake's build tree (`Build/full`).** Verified +by removal: with `Build/full/{Libraries,Services,UI,bin,cargo}` moved off the +machine — every Ladybird-generated header and every binary CMake produced — all six +binaries build from scratch and render `--headless=text` **and** +`--headless=layout-tree` byte-identically to the CMake reference, and `cquery` over +the closure of all six returns **0** targets under `Build/full` (down from 741). +`Build/full` is still needed to *regenerate* the BUILD files and to run the parity +harness — a converter-development dependency, not a build dependency. + +**But `git clone && bazel build //:ladybird` on a fresh clone still does NOT work, +and this section said it did.** Tested properly for the first time — `git clone` +into a new directory, overlay dropped in, nothing else — it fails, and it fails +**six separate times**. `Build/full` was the dependency I removed; it was not the +only one, and five of the six have nothing to do with it. Two are fixed here +(#4, #5) and one now fails loudly instead of silently (#6); the rest are inputs the +recipe must produce or document: + +| # | What is missing on a fresh clone | Why the dev machine hid it | +|---|---|---| +| 1 | **`Build/vcpkg`** — a git clone of microsoft/vcpkg at `vcpkg.json`'s `builtin-baseline`, which `//Build/vcpkg:tree` globs with `allow_empty = True`. Matches *one* file (its own `BUILD.bazel`) and the build fails with `/tmp/.../root/vcpkg: No such file or directory`. Ladybird's own `Meta/ladybird.py vcpkg` creates it; the recipe above never says so | `Meta/ladybird.py vcpkg` had been run months earlier | +| 2 | **`.git` inside that checkout** — vcpkg resolves versioned ports with `git read-tree`, so it is load-bearing (120 MB), and the filegroup *excludes* it. The action is `no-sandbox`, so it reads the real path and got it anyway: an **undeclared input the build needs** | `no-sandbox` + a real checkout on disk | +| 3 | **`Build/caches/HSTSPreload/transport_security_state_static.json`** (10 MB) — an *unversioned, unpinned* network download CMake does at configure time (`Meta/CMake/hsts_preload.cmake` fetches Chromium's `main`). `codegen_root.bzl` names it as a genrule `srcs`, so Bazel fails cleanly with `missing input file` — but there is no rule that produces it | the CMake configure had already downloaded it | +| 4 | **Two path bugs in `Meta/vcpkg_build.sh`** — the distfile index and its entries were passed **execroot-relative**, and vcpkg invokes the asset-cache script from its own cwd, so `awk`/`cp` looked in the wrong directory. Every asset lookup failed, reported as `no asset cache hits`, and `x-block-origin` then correctly refused the network. Fixed here (absolutize both) | the vcpkg checkout already had `downloads/tools/cmake-4.4.0-linux` from an earlier run, so vcpkg never *asked* the script for a tool | +| 5 | **The `angle` port runs `pip install ply`** (`x_vcpkg_get_python_packages`), which is **not** an asset-cache download and therefore not covered by the pin. A real hole in the "zero network access" claim: `vcpkg_tree` set `requires-network: "0"`, but that is a *scheduling hint*, and with `no-sandbox: "1"` **nothing enforced it** — with `use_default_shell_env = True` the action inherited `HTTP_PROXY`/`HTTPS_PROXY` and pip reached PyPI. **Fixed here:** the wheel is pinned by URL+sha256 (`vcpkg_python_packages.bzl`), staged into a find-links dir, and pip runs with `PIP_NO_INDEX` and the proxy variables unset | this sandbox exports a proxy, so pip silently succeeded through it | +| 6 | **The four `vcpkg_from_git` archives were pre-placed from a directory I had made by hand.** `vcpkg_git_archives.bzl` is generated, committed, listed in the table below — and **loaded by nothing**. The staging was `if [ -d ... ]; then cp ... 2>/dev/null \|\| true; fi`: three ways to succeed while copying nothing. Without them skia fails ~20 min in with `git fetch https://android.googlesource.com/.../piex.git … Error code: 128`, naming neither the directory nor the tarball. **Now a hard error in 4 seconds** naming both; producing them from Bazel still needs a repo rule (they are `git archive` output, so there is no URL to `http_file`) | I created the directory by hand while building the pin, months before | + +Rows 1–3 are inputs the recipe must produce or document (step 1a). Rows 4–6 were +bugs, and 5 is the substantive one: **"zero network access" was measured with +`x-block-origin`, which only governs vcpkg's own downloader.** A portfile that +shells out to pip, git or curl bypasses it entirely, and nothing was enforcing the +absence of a network — `requires-network: "0"` is a hint, and `no-sandbox: "1"` +means there is no namespace to enforce. `ply` is now pinned like any other +dependency (URL + sha256 → `http_file` → find-links dir → `PIP_NO_INDEX`), which +makes an unpinned package a hard error rather than a download. The four +`vcpkg_from_git` archives are still staged from a directory rather than fetched by +Bazel — they are `git archive` output with no URL, so they need a repo rule that +clones at the pinned ref. + +The lesson is the same one as finding 35, one layer out: **`Build/full` was the +dependency I went looking for, so it is the one I found.** The check that would +have caught all five is not a better `cquery` — it is doing the clone. See +[Known gaps](#known-gaps). **This claim was false until recently, and the way it was false is the most useful thing in this directory.** The README said it worked; it did not. Every @@ -64,7 +98,8 @@ an FFI header collision and an entire missing Rust target. See | `cargo.bzl`, `Meta/cargo_build.sh`, `Meta/cargo_binary_build.sh`, `Meta/cargo_vendor.sh` | `rust_sysroot` (the pinned 1.96.1 toolchain merged into one tree), `cargo_crate` / `cargo_binary` (offline, network-blocked build actions), and `cargo_lib` (one consumable `CcInfo` per crate: its archive + its FFI headers) | | `cargo_ring.bzl` | The 10 `cargo_crate` + 10 `cargo_lib` targets and `flapc`, as a macro for the root package (the crate sources are at the repo root and `glob()` is package-relative). **Generated** by `Meta/emit_cargo_bazel.py` | | `export_headers.bzl`, `Libraries/LibWeb/export_header.bzl` | The 15 `generate_export_header` `Export.h` files + AK's two `configure_file` headers, generated by Bazel from the same inputs CMake uses. **Generated** by `Meta/emit_export_headers_bazel.py`; `--check Build/full` byte-compares every one against CMake's | -| ~~`Build/full/**/BUILD.bazel`~~ | **Gone.** These shimmed the reference CMake build tree, and by the end they supplied *nothing*: the 709 headers they globbed were 21 Bazel generates and 688 LibWeb bindings headers Bazel also generates. Deleting them is what turned "the README claims a clone-and-build" into one | +| ~~`Build/full/**/BUILD.bazel`~~ | **Gone.** These shimmed the reference CMake build tree, and by the end they supplied *nothing*: the 709 headers they globbed were 21 Bazel generates and 688 LibWeb bindings headers Bazel also generates. Deleting them removed the *`Build/full`* dependency — not every clone-and-build blocker, as the table at the top now records | +| `Build/vcpkg/BUILD.bazel` | A filegroup over the microsoft/vcpkg checkout. **Still the same `allow_empty = True` glob over a foreign tree that finding 35 is about** — it just globs a *different* tree, so a fresh clone gets no diagnostic. Gap 7 | The four generated files are reproducible: re-running each emitter against the same `Build/full` reproduces them byte for byte — **on the same machine**. They @@ -75,14 +110,37 @@ adds `-I/usr/include/libdrm`), which is gap 3 below, not an emitter bug. ## Reproducing To **build the browser**, no CMake build is needed — Bazel fetches and builds the -vcpkg tree and the Rust crates itself, with zero network access in either: +vcpkg *ports* and the Rust crates itself, and vcpkg's own downloader reaches the +network zero times. Three inputs still have to be present first (step 1a; gaps 7–8), +and one port bypasses the pin via `pip`, so "no network at all" is not yet true: ```sh git clone https://github.com/LadybirdBrowser/ladybird && cd ladybird # 1. Drop in the overlay. cp -r .../examples/ladybird/workspace/. . && mv bazelrc.txt .bazelrc git apply .../examples/ladybird/patches/*.patch # generator determinism + a Qt header - # that is not self-contained (gap 7) + # that is not self-contained (gap 9) +# 1a. THE THREE INPUTS THE OVERLAY DOES NOT CARRY. Without these the build fails; +# see the table above for how each one stayed invisible on the dev machine. +# (a) the vcpkg checkout at vcpkg.json's builtin-baseline, WITH its .git -- +# vcpkg resolves versioned ports via `git read-tree`. Ladybird's own +# helper does exactly this: +python3 Meta/ladybird.py vcpkg +# (b) the HSTS preload table: an unversioned download from Chromium's main +# branch that CMake does at configure time. No Bazel rule produces it. +mkdir -p Build/caches/HSTSPreload && curl -Lo Build/caches/HSTSPreload/transport_security_state_static.json \ + https://raw.githubusercontent.com/chromium/chromium/main/net/http/transport_security_state_static.json +# (c) the four vcpkg_from_git archives. These are `git archive` output, so +# there is no URL to http_file and Bazel cannot fetch them yet; the +# hashes ARE pinned (vcpkg_git_archives.bzl) so they can be verified. +# Absent, the build now fails in 4s naming them (gap 7). +# NB `ply` no longer needs anything: the wheel is pinned and pip runs with +# PIP_NO_INDEX (gap 8). +# +# 1b. vcpkg's buildtrees peak around 3 GB, and a default /tmp tmpfs is often too +# small -- which surfaces as `No space left on device` from the ASSET SCRIPT, +# i.e. a message that blames the pin for a full disk. +# --action_env=TMPDIR=... if /tmp is small. # 2. Build. This includes the 77 vcpkg ports (~45 min cold) and the 10 Rust # crates + flapc + cranelift-compiler: the libraries depend on them through # //Meta/vcpkg: and //:_lib, so there is no separate step. Build @@ -362,7 +420,47 @@ Honest inventory of what stops this from being a clone-and-build. patches). The BCR's `rules_qt` module is Vertexwahn's unrelated `rules_qt6`. Qt itself *is* host-portable: `qt.local_repo` discovers the host Qt via `qmake -query`, so no Qt SDK is vendored. -7. **Three upstreamable Ladybird fixes.** The first is the `sorted()` determinism +7. **The fresh clone needs three inputs the overlay does not carry** (step 1a + above), and each stayed invisible for the same reason: it was already on the + machine. `Build/vcpkg` — a microsoft/vcpkg checkout at `vcpkg.json`'s + `builtin-baseline`, created by Ladybird's own `Meta/ladybird.py vcpkg` — is + globbed by `//Build/vcpkg:tree` with `allow_empty = True`, so on a clone it + matches one file and the failure names a missing `/tmp/.../root/vcpkg`. Its + `.git` is *load-bearing* (vcpkg resolves versioned ports with `git read-tree`) + yet the filegroup **excludes** it — an undeclared input the build reads anyway, + because the action is `no-sandbox`. And + `Build/caches/HSTSPreload/transport_security_state_static.json` is an + *unversioned* download from Chromium's `main` branch that CMake does at + configure time; `codegen_root.bzl` names it as a genrule input, so Bazel fails + cleanly, but nothing produces it. + + The honest fix for all three is the same shape as findings 30–33: pin them and + let Bazel fetch them. The vcpkg checkout is a `git_repository` at the + `builtin-baseline` commit (which `vcpkg.json` already names, so the pin exists + — it just is not wired to Bazel); the HSTS table is an `http_file`, and needs a + hash Ladybird does not currently pin at all, because the URL tracks `main`. + +8. **"Zero network access" is true of vcpkg's downloader, not of the vcpkg + action.** The pin + `x-block-origin` is verified — a distfile missing from the + index is a hard error, not a fetch. But `x-block-origin` only governs vcpkg's + *own* downloads, and the `angle` overlay-port calls + `x_vcpkg_get_python_packages`, which runs **`pip install ply`**. That is not an + asset-cache download, so nothing pins it and `x-block-origin` never sees it. + Nor is anything stopping it: `vcpkg_tree` sets `requires-network: "0"`, which is + a *scheduling hint* Bazel does not enforce, and `no-sandbox: "1"` means there is + no network namespace to enforce it in — so with `use_default_shell_env = True` + the action inherits `HTTP_PROXY`/`HTTPS_PROXY` and pip reaches PyPI. + + It went unnoticed because this sandbox exports a proxy, so pip silently + succeeded; a fresh clone in a network-free environment fails with + `No matching distribution found for ply`. Two things to fix, and the second + matters more than the first: pin `ply` as an `http_file` wheel staged into the + port's venv, **and stop taking `requires-network: "0"` as a claim** — the + verification has to be an environment with no route to the network, not a flag + that says there is none. Same shape as the shim that could not fail: a control + that is not enforced is indistinguishable from one that is not there. + +9. **Three upstreamable Ladybird fixes.** The first is the `sorted()` determinism fix in `Meta/Generators/libweb_bindings/to_idl_value.py`, filed as [ladybird#10899](https://github.com/LadybirdBrowser/ladybird/issues/10899) and fixed upstream. diff --git a/examples/ladybird/workspace/BUILD.bazel b/examples/ladybird/workspace/BUILD.bazel index ea06b6d..c34fd60 100644 --- a/examples/ladybird/workspace/BUILD.bazel +++ b/examples/ladybird/workspace/BUILD.bazel @@ -1861,6 +1861,11 @@ cc_binary( vcpkg_tree( name = "vcpkg_installed", distfiles = VCPKG_DISTFILE_INDEX, + # The wheels for the Python packages a portfile pip-installs (angle asks for + # `ply`). Pinned separately from the 76 distfiles because pip bypasses vcpkg's + # asset cache entirely, so the capture cannot see them and x-block-origin + # cannot block them -- see vcpkg_python_packages.bzl and finding 36. + python_wheels = ['@vcpkg_pywheel_ply//file'], source_dir = ".", source_root = ":vcpkg_source_inputs", triplet = "x64-linux-dynamic", diff --git a/examples/ladybird/workspace/MODULE.bazel b/examples/ladybird/workspace/MODULE.bazel index 4f59612..d9918bd 100644 --- a/examples/ladybird/workspace/MODULE.bazel +++ b/examples/ladybird/workspace/MODULE.bazel @@ -125,6 +125,14 @@ use_repo( 'vcpkg_xiph_opus_v1_5_2_tar_gz_4ffefd9c0356', 'vcpkg_xiph_theora_v1_2_0_tar_gz_b2aac15528f0', 'vcpkg_xiph_vorbis_v1_3_7_tar_gz_bfb6f5dbfd49', + # NOT from the capture: the wheel for a Python package a portfile + # pip-installs. pip does not go through the vcpkg asset cache, so + # `--x-asset-sources=x-script+x-block-origin` never sees it -- which is how + # `pip install ply` in the angle port went on reaching PyPI through an + # inherited HTTP_PROXY (finding 36). Hand-maintained in + # vcpkg_python_packages.bzl for exactly that reason: no instrument can + # produce it, so nothing would regenerate it. + 'vcpkg_pywheel_ply', ) # --------------------------------------------------------------------------- diff --git a/examples/ladybird/workspace/Meta/cargo_binary_build.sh b/examples/ladybird/workspace/Meta/cargo_binary_build.sh old mode 100644 new mode 100755 diff --git a/examples/ladybird/workspace/Meta/cargo_build.sh b/examples/ladybird/workspace/Meta/cargo_build.sh old mode 100644 new mode 100755 diff --git a/examples/ladybird/workspace/Meta/cargo_vendor.sh b/examples/ladybird/workspace/Meta/cargo_vendor.sh old mode 100644 new mode 100755 diff --git a/examples/ladybird/workspace/Meta/emit_vcpkg_bazel.py b/examples/ladybird/workspace/Meta/emit_vcpkg_bazel.py index 844ddfa..69cd73c 100644 --- a/examples/ladybird/workspace/Meta/emit_vcpkg_bazel.py +++ b/examples/ladybird/workspace/Meta/emit_vcpkg_bazel.py @@ -531,9 +531,15 @@ def emit_extension(distfiles): """ sys.stdout.write(HEADER) print('load(":vcpkg_distfiles.bzl", "vcpkg_distfiles")') + # The pip-installed Python packages a PORTFILE asks for. Hand-written, not + # captured -- pip does not go through vcpkg's asset cache, so the instrument + # that produced the 76 distfiles cannot see them (finding 36). Loaded here so + # they are created by the same extension and named by the same use_repo. + print('load(":vcpkg_python_packages.bzl", "vcpkg_python_wheels")') print() print("def _vcpkg_deps_impl(_ctx):") print(" vcpkg_distfiles()") + print(" vcpkg_python_wheels()") print() print("vcpkg_deps = module_extension(implementation = _vcpkg_deps_impl)") @@ -547,6 +553,13 @@ def emit_use_repo(distfiles): "no such repository", far from its cause).""" names = sorted(bazel_name(name, sha) for sha, (_u, name, _p, _s) in distfiles.items()) + # The pip wheels are created by the SAME extension, so they must be in the + # same use_repo -- and they are NOT in the capture, because pip does not go + # through vcpkg's asset cache (finding 36). Read them out of the hand-written + # vcpkg_python_packages.bzl rather than restating them, so pasting this output + # into MODULE.bazel cannot silently drop the wheel and leave a "no such + # repository" a long way from its cause. + names += ["vcpkg_pywheel_" + n for n in sorted(python_wheel_names())] print("use_repo(") print(" vcpkg_deps,") for n in names: @@ -554,6 +567,24 @@ def emit_use_repo(distfiles): print(")") +def python_wheel_names(): + """The keys of VCPKG_PYTHON_WHEELS, parsed out of the .bzl that declares them. + + Parsed rather than duplicated: that file is the pin, and a second copy of the + list here would be one more thing to drift (finding 23's rule, applied to the + one vcpkg input no instrument can capture). + """ + path = os.path.join(ROOT, "vcpkg_python_packages.bzl") + if not os.path.exists(path): + sys.stderr.write("WARNING: %s is missing; no pip wheels will be named, so " + "any port that pip-installs will fail\n" % path) + return [] + with open(path) as f: + body = f.read() + block = body.split("VCPKG_PYTHON_WHEELS = {", 1)[1].split("\n}", 1)[0] + return re.findall(r'^\s*"([A-Za-z0-9_.\-]+)":', block, re.M) + + def emit_index(distfiles): sys.stdout.write(HEADER) print("# sha512 (hex) -> the label of the file Bazel fetched for it.") diff --git a/examples/ladybird/workspace/Meta/vcpkg_build.sh b/examples/ladybird/workspace/Meta/vcpkg_build.sh index d81257d..3ecc35f 100755 --- a/examples/ladybird/workspace/Meta/vcpkg_build.sh +++ b/examples/ladybird/workspace/Meta/vcpkg_build.sh @@ -51,9 +51,41 @@ INDEX="${2:?distfile index}" VCPKG_TREE="${3:?vcpkg checkout}" SRC="${4:?ladybird source root}" TRIPLET="${5:-x64-linux-dynamic}" +WHEELS="${6-}" +# vcpkg's buildtrees peak around 3 GB, and $TMPDIR is often a tmpfs sized well +# under that (7.9 GB here, shared with everything else) -- which surfaces as +# `cp: error writing ...: No space left on device` from the ASSET SCRIPT, i.e. a +# message that blames the pin for a full disk. Honour $TMPDIR when it is set, so a +# caller can point this at real disk, and say where the scratch went. WORK="$(mktemp -d)" -trap 'rm -rf "$WORK"' EXIT +echo "vcpkg_build: scratch dir $WORK ($(df -h "$WORK" | awk 'NR==2{print $4}') free)" >&2 +# Cleaned up on the way out INCLUDING on failure -- an interrupted run used to +# leave a multi-GB tree behind, and several of them is how the disk filled. +trap 'rm -rf "$WORK"' EXIT INT TERM + +# The index arrives as an EXECROOT-RELATIVE path (bazel-out/.../x.index), and the +# asset-cache script below is run by vcpkg from vcpkg's own working directory, not +# ours -- so a relative path there resolves against the wrong dir and awk reports +# "cannot open". Absolutize it here, once, while we are still in the execroot. +# +# Why this survived so long: on the machine this was developed on, the vcpkg +# checkout already had downloads/tools/cmake-4.4.0-linux populated by an earlier +# `Meta/ladybird.py vcpkg` run, so vcpkg found its tools locally and never invoked +# the asset script for one. Only a clone with a pristine checkout makes vcpkg ask +# for cmake -- and then the script fails, is reported as "no asset cache hits", +# and x-block-origin correctly refuses to reach the network. +# +# The index's VALUES are execroot-relative for the same reason, so rewrite them +# to absolute here too: awk finding the row is only half the job, and `cp` from a +# relative path fails identically once vcpkg has chdir'd. Both halves have to be +# absolutized in the same place, or the failure just moves one line down. +EXECROOT="$PWD" +INDEX_ABS="$WORK/index" +awk -v root="$EXECROOT" \ + '{ if ($2 ~ /^\//) print $1, $2; else print $1, root "/" $2 }' \ + "$INDEX" > "$INDEX_ABS" +INDEX="$INDEX_ABS" # vcpkg reads $HOME (for its own config/telemetry dirs) and hard-fails "unable to # read $HOME" without it. Bazel deliberately does not pass HOME through to @@ -64,6 +96,40 @@ trap 'rm -rf "$WORK"' EXIT export HOME="$WORK/home" mkdir -p "$HOME" +# --- pip: offline, from Bazel-fetched wheels only ---------------------------- +# +# vcpkg's asset cache does NOT cover pip. The angle overlay-port calls +# x_vcpkg_get_python_packages, which runs `pip install ply` inside a venv, and +# neither x-script nor x-block-origin ever sees the request -- so the 76-distfile +# pin says nothing about it and, with this action running `no-sandbox` and +# inheriting the shell env, pip happily used the caller's HTTP_PROXY for months +# (finding 36). `requires-network: "0"` did not stop it: that is a scheduling hint, +# not a namespace. +# +# pip's own offline switches are the fix, so no portfile patch is needed: +# PIP_NO_INDEX forbids talking to an index at all, PIP_FIND_LINKS points at a +# directory of wheels Bazel fetched by URL+hash. A package that is not pinned then +# fails with "No matching distribution found", which is the pip-side equivalent of +# x-block-origin -- an error instead of a download. +# +# Unset the proxy variables too. Leaving them would make the *failure* mode depend +# on the caller's environment: --no-index means pip should not reach an index, and +# an inherited proxy is exactly how this went unnoticed. Belt and braces, because +# the lesson of finding 36 is that one unenforced control is not a control. +FINDLINKS="$WORK/wheels" +mkdir -p "$FINDLINKS" +if [ -n "$WHEELS" ]; then + IFS=',' read -r -a _wheels <<< "$WHEELS" + for w in "${_wheels[@]}"; do + [ -n "$w" ] && cp "$w" "$FINDLINKS/" + done +fi +export PIP_NO_INDEX=1 +export PIP_FIND_LINKS="$FINDLINKS" +export PIP_DISABLE_PIP_VERSION_CHECK=1 +unset HTTP_PROXY HTTPS_PROXY http_proxy https_proxy ALL_PROXY all_proxy +echo "vcpkg_build: pip is offline; $(ls "$FINDLINKS" | wc -l) pinned wheel(s)" >&2 + # --- the asset-cache script: resolve by hash, never fetch -------------------- cat > "$WORK/fetch.sh" </dev/null || true +# +# This used to be `if [ -d ... ]; then cp ... 2>/dev/null || true; fi` -- three +# separate ways to succeed while copying nothing, in four lines. On the machine +# that developed this, the directory existed because I had created it by hand; on a +# fresh clone it does not, so all four archives were silently absent and the build +# failed ~20 minutes later inside skia's portfile with +# `git fetch https://android.googlesource.com/.../piex.git ... Error code: 128`, +# naming neither this directory nor the tarball. Exactly the finding-35 shape: a +# copy that cannot fail is indistinguishable from a copy that is not needed. +# +# So: fail here, naming what is missing and where it comes from. VCPKG_GIT_ARCHIVES +# in vcpkg_git_archives.bzl lists the four expected names -- that file is GENERATED +# and checked in, and (finding 36) is currently loaded by nothing at all, which is +# the deeper bug this only reports. These are `git archive` output, so unlike the 76 +# distfiles they have no URL to http_file; making Bazel produce them needs a repo +# rule that clones at the pinned ref (gap 7). +GIT_ARCHIVES="$SRC/Meta/CMake/vcpkg/git-archives" +if ! compgen -G "$GIT_ARCHIVES/*.tar.gz" > /dev/null; then + echo "vcpkg_build: no git-sourced externals at $GIT_ARCHIVES" >&2 + echo "vcpkg_build: 4 are required (see VCPKG_GIT_ARCHIVES in" >&2 + echo "vcpkg_build: vcpkg_git_archives.bzl). vcpkg_from_git bypasses the asset" >&2 + echo "vcpkg_build: cache entirely, so without them skia and angle will try to" >&2 + echo "vcpkg_build: 'git fetch' and fail ~20 minutes into the build with an" >&2 + echo "vcpkg_build: error naming a googlesource URL rather than this directory." >&2 + exit 1 fi +cp "$GIT_ARCHIVES"/*.tar.gz "$ROOT/downloads/" # --- manifest, verbatim (note 2) -------------------------------------------- MANIFEST="$WORK/manifest" diff --git a/examples/ladybird/workspace/vcpkg.bzl b/examples/ladybird/workspace/vcpkg.bzl index 98ddc81..de49b6a 100644 --- a/examples/ladybird/workspace/vcpkg.bzl +++ b/examples/ladybird/workspace/vcpkg.bzl @@ -42,7 +42,7 @@ def _vcpkg_tree_impl(ctx): ctx.actions.run( outputs = [out], inputs = depset( - [index] + distfile_inputs + + [index] + distfile_inputs + ctx.files.python_wheels + ctx.files.vcpkg_tree + ctx.files.source_root, ), executable = ctx.executable._build, @@ -52,6 +52,10 @@ def _vcpkg_tree_impl(ctx): ctx.attr.vcpkg_root, ctx.attr.source_dir, ctx.attr.triplet, + # The find-links dir for pip: a comma-joined list of the wheel paths, + # empty when there are none (which is then a hard failure for any port + # that pip-installs, rather than a silent fetch). + ",".join([f.path for f in ctx.files.python_wheels]), ], mnemonic = "VcpkgInstall", progress_message = "Building %d vcpkg ports (no network)" % len(ctx.attr.distfiles), @@ -94,6 +98,18 @@ vcpkg_tree = rule( allow_files = True, doc = "Ladybird's manifest, overlay-ports and overlay-triplets.", ), + "python_wheels": attr.label_list( + allow_files = True, + doc = """Wheels for the Python packages a PORTFILE pip-installs. + + Staged into a find-links dir and paired with PIP_NO_INDEX, so pip + resolves offline from these files alone. Needed because pip does NOT go + through vcpkg's asset cache, so `x-block-origin` never sees it and the + distfile pin cannot cover it -- the angle port's `pip install ply` was + reaching PyPI through an inherited HTTP_PROXY for months (finding 36). + An empty list means pip has no index and no wheels, i.e. any port that + asks for a package fails loudly instead of downloading one.""", + ), "vcpkg_root": attr.string(doc = "Path of the vcpkg checkout."), "source_dir": attr.string(doc = "Path of the Ladybird source root."), "install_root": attr.string( diff --git a/examples/ladybird/workspace/vcpkg_extension.bzl b/examples/ladybird/workspace/vcpkg_extension.bzl index 9c34c52..9359abc 100644 --- a/examples/ladybird/workspace/vcpkg_extension.bzl +++ b/examples/ladybird/workspace/vcpkg_extension.bzl @@ -1,7 +1,9 @@ # AUTO-GENERATED by Meta/emit_vcpkg_bazel.py — do not edit. load(":vcpkg_distfiles.bzl", "vcpkg_distfiles") +load(":vcpkg_python_packages.bzl", "vcpkg_python_wheels") def _vcpkg_deps_impl(_ctx): vcpkg_distfiles() + vcpkg_python_wheels() vcpkg_deps = module_extension(implementation = _vcpkg_deps_impl) diff --git a/examples/ladybird/workspace/vcpkg_python_packages.bzl b/examples/ladybird/workspace/vcpkg_python_packages.bzl new file mode 100644 index 0000000..7e8b13f --- /dev/null +++ b/examples/ladybird/workspace/vcpkg_python_packages.bzl @@ -0,0 +1,43 @@ +# The Python packages a vcpkg PORTFILE installs with pip, pinned as wheels. +# +# Hand-written, unlike the other four vcpkg .bzl files, and the reason is worth +# stating: this is not something the asset capture can produce. The 76 distfiles +# came from instrumenting vcpkg's own downloader (finding 28), and +# `--x-asset-sources=x-script+x-block-origin` covers everything that goes through +# it. `pip install` does not: the angle overlay-port calls +# x_vcpkg_get_python_packages, which shells out to pip inside a venv, and vcpkg's +# asset cache never sees the request. So the capture cannot report it, the pin +# cannot cover it, and x-block-origin cannot block it. +# +# That is exactly how it went unnoticed (finding 36): the vcpkg action runs +# `no-sandbox` with `use_default_shell_env = True`, so it inherited this sandbox's +# HTTP_PROXY and pip quietly reached PyPI for months, under a rule whose +# `requires-network: "0"` is only a scheduling hint Bazel does not enforce. +# +# pip has a supported offline mode -- PIP_NO_INDEX + PIP_FIND_LINKS -- so the fix +# needs no patch to the portfile: fetch the wheel by URL with a hash, put it in a +# find-links directory, and tell pip it may not use an index. Then a wheel missing +# from the pin is a hard error rather than a download, which is the same property +# x-block-origin gives the other 76. +load("@bazel_tools//tools/build_defs/repo:http.bzl", "http_file") + +# name -> (url, integrity). The URL is files.pythonhosted.org's content-addressed +# path, so it is immutable for a given (name, version) -- unlike `pip install ply`, +# which resolves against whatever PyPI serves today. +VCPKG_PYTHON_WHEELS = { + # angle/portfile.cmake:86 -> x_vcpkg_get_python_packages(PACKAGES ply). + # ply is pure-python (py2.py3-none-any), so one wheel serves every platform. + "ply": ( + "https://files.pythonhosted.org/packages/a3/58/35da89ee790598a0700ea49b2a66594140f44dec458c07e8e3d4979137fc/ply-3.11-py2.py3-none-any.whl", + "sha256-CW+bg1C2Xr0v0TRrEkUu/luWB/dIKBP/ylDCJyKoB84=", + ), +} + +def vcpkg_python_wheels(): + for name, (url, integrity) in VCPKG_PYTHON_WHEELS.items(): + http_file( + name = "vcpkg_pywheel_" + name, + urls = [url], + downloaded_file_path = url.rsplit("/", 1)[-1], + integrity = integrity, + ) diff --git a/tests/test_configure.py b/tests/test_configure.py index 37c6075..542f2b6 100644 --- a/tests/test_configure.py +++ b/tests/test_configure.py @@ -84,15 +84,10 @@ def test_no_include_dirs_means_not_compile_input(): os.unlink(t) assert len(cfs) == 1 and not cfs[0].is_compile_input - -if __name__ == "__main__": - import traceback - fns = [v for k, v in sorted(globals().items()) if k.startswith("test_")] - failed = 0 - for fn in fns: - try: - fn(); print(f"PASS {fn.__name__}") - except Exception: - failed += 1; print(f"FAIL {fn.__name__}"); traceback.print_exc() - print(f"\n{len(fns) - failed}/{len(fns)} passed") - sys.exit(1 if failed else 0) +# No `if __name__ == "__main__"` runner here on purpose. There used to be one in +# every test file, and in this file it sat MID-FILE -- so four tests appended after +# it were defined, never called, and the file still printed "6/6 passed". The third +# instance of this session's recurring bug: a report that cannot count what it does +# not reach. `python3 tests/run_all.py` enumerates the module instead, so a test's +# POSITION in the file cannot decide whether it runs; it also fails if a file +# defines no tests at all. Run a single file with `run_all.py `. diff --git a/tests/test_diff_ts.py b/tests/test_diff_ts.py index 3a5554f..551aadc 100644 --- a/tests/test_diff_ts.py +++ b/tests/test_diff_ts.py @@ -96,15 +96,10 @@ def test_same_source_multiple_actions_union(): assert ext_diffs == [], ext_diffs assert stripped["npm"] == 1 - -if __name__ == "__main__": - import traceback - fns = [v for k, v in sorted(globals().items()) if k.startswith("test_")] - failed = 0 - for fn in fns: - try: - fn(); print(f"PASS {fn.__name__}") - except Exception: - failed += 1; print(f"FAIL {fn.__name__}"); traceback.print_exc() - print(f"\n{len(fns) - failed}/{len(fns)} passed") - sys.exit(1 if failed else 0) +# No `if __name__ == "__main__"` runner here on purpose. There used to be one in +# every test file, and in this file it sat MID-FILE -- so four tests appended after +# it were defined, never called, and the file still printed "6/6 passed". The third +# instance of this session's recurring bug: a report that cannot count what it does +# not reach. `python3 tests/run_all.py` enumerates the module instead, so a test's +# POSITION in the file cannot decide whether it runs; it also fails if a file +# defines no tests at all. Run a single file with `run_all.py `. diff --git a/tests/test_emit_cargo.py b/tests/test_emit_cargo.py index cc51a36..272b576 100644 --- a/tests/test_emit_cargo.py +++ b/tests/test_emit_cargo.py @@ -748,22 +748,10 @@ def test_every_extension_created_repo_is_named_in_module_bazel(): assert {"rust_rustc_1_96_1", "rust_cargo_1_96_1", "rust_rust_std_1_96_1"} <= named - -if __name__ == "__main__": - # A runner, because without one this file EXITED 0 WITHOUT RUNNING ANYTHING -- - # a test suite that cannot fail, which is the same bug as a glob that cannot - # fail (finding 35). Found by checking that the suite's output accounted for - # every test file, rather than trusting the exit code. - import sys as _sys, traceback - fns = [v for k, v in sorted(globals().items()) if k.startswith("test_")] - failed = 0 - for fn in fns: - try: - fn() - print(f"PASS {fn.__name__}") - except Exception: - failed += 1 - print(f"FAIL {fn.__name__}") - traceback.print_exc() - print(f"\n{len(fns) - failed}/{len(fns)} passed") - _sys.exit(1 if failed else 0) +# No `if __name__ == "__main__"` runner here on purpose. There used to be one in +# every test file, and in this file it sat MID-FILE -- so four tests appended after +# it were defined, never called, and the file still printed "6/6 passed". The third +# instance of this session's recurring bug: a report that cannot count what it does +# not reach. `python3 tests/run_all.py` enumerates the module instead, so a test's +# POSITION in the file cannot decide whether it runs; it also fails if a file +# defines no tests at all. Run a single file with `run_all.py `. diff --git a/tests/test_emit_vcpkg.py b/tests/test_emit_vcpkg.py index d224c55..98111c6 100644 --- a/tests/test_emit_vcpkg.py +++ b/tests/test_emit_vcpkg.py @@ -212,22 +212,10 @@ def test_non_mirrored_urls_are_left_as_a_single_entry(): u = "https://github.com/madler/zlib/archive/v1.3.1.tar.gz" assert emit.urls_for(u) == [u] - -if __name__ == "__main__": - # A runner, because without one this file EXITED 0 WITHOUT RUNNING ANYTHING -- - # a test suite that cannot fail, which is the same bug as a glob that cannot - # fail (finding 35). Found by checking that the suite's output accounted for - # every test file, rather than trusting the exit code. - import sys as _sys, traceback - fns = [v for k, v in sorted(globals().items()) if k.startswith("test_")] - failed = 0 - for fn in fns: - try: - fn() - print(f"PASS {fn.__name__}") - except Exception: - failed += 1 - print(f"FAIL {fn.__name__}") - traceback.print_exc() - print(f"\n{len(fns) - failed}/{len(fns)} passed") - _sys.exit(1 if failed else 0) +# No `if __name__ == "__main__"` runner here on purpose. There used to be one in +# every test file, and in this file it sat MID-FILE -- so four tests appended after +# it were defined, never called, and the file still printed "6/6 passed". The third +# instance of this session's recurring bug: a report that cannot count what it does +# not reach. `python3 tests/run_all.py` enumerates the module instead, so a test's +# POSITION in the file cannot decide whether it runs; it also fails if a file +# defines no tests at all. Run a single file with `run_all.py `. diff --git a/tests/test_engine.py b/tests/test_engine.py index 4ab8f2d..43eff54 100644 --- a/tests/test_engine.py +++ b/tests/test_engine.py @@ -675,18 +675,10 @@ def test_nonparticipating_roles_are_excluded_not_diffed(): assert not any(d.kind == "missing_target" for d in discs) assert "Nightly" in res["excluded"]["cmake"]["dashboard"] - -if __name__ == "__main__": - import traceback - fns = [v for k, v in sorted(globals().items()) if k.startswith("test_")] - failed = 0 - for fn in fns: - try: - fn() - print(f"PASS {fn.__name__}") - except Exception: - failed += 1 - print(f"FAIL {fn.__name__}") - traceback.print_exc() - print(f"\n{len(fns) - failed}/{len(fns)} passed") - sys.exit(1 if failed else 0) +# No `if __name__ == "__main__"` runner here on purpose. There used to be one in +# every test file, and in this file it sat MID-FILE -- so four tests appended after +# it were defined, never called, and the file still printed "6/6 passed". The third +# instance of this session's recurring bug: a report that cannot count what it does +# not reach. `python3 tests/run_all.py` enumerates the module instead, so a test's +# POSITION in the file cannot decide whether it runs; it also fails if a file +# defines no tests at all. Run a single file with `run_all.py `. diff --git a/tests/test_extract_npm.py b/tests/test_extract_npm.py index 4397780..35a271d 100644 --- a/tests/test_extract_npm.py +++ b/tests/test_extract_npm.py @@ -210,15 +210,10 @@ def test_partial_nlast_line_does_not_break_extract(): m = extract_npm.extract(os.path.join(tmp, "a.ndjson"), "/repo") assert "out/a.js" in m.targets - -if __name__ == "__main__": - import traceback - fns = [v for k, v in sorted(globals().items()) if k.startswith("test_")] - failed = 0 - for fn in fns: - try: - fn(); print(f"PASS {fn.__name__}") - except Exception: - failed += 1; print(f"FAIL {fn.__name__}"); traceback.print_exc() - print(f"\n{len(fns) - failed}/{len(fns)} passed") - sys.exit(1 if failed else 0) +# No `if __name__ == "__main__"` runner here on purpose. There used to be one in +# every test file, and in this file it sat MID-FILE -- so four tests appended after +# it were defined, never called, and the file still printed "6/6 passed". The third +# instance of this session's recurring bug: a report that cannot count what it does +# not reach. `python3 tests/run_all.py` enumerates the module instead, so a test's +# POSITION in the file cannot decide whether it runs; it also fails if a file +# defines no tests at all. Run a single file with `run_all.py `. diff --git a/tests/test_extractors.py b/tests/test_extractors.py index d59cdb0..03f3f7b 100644 --- a/tests/test_extractors.py +++ b/tests/test_extractors.py @@ -433,15 +433,10 @@ def test_ts_program_splits_into_per_file_tscompile_actions(): # role is set by the extractor and must not be demoted to AGGREGATE assert t.role.value == "production", t.role - -if __name__ == "__main__": - import traceback - fns = [v for k, v in sorted(globals().items()) if k.startswith("test_")] - failed = 0 - for fn in fns: - try: - fn(); print(f"PASS {fn.__name__}") - except Exception: - failed += 1; print(f"FAIL {fn.__name__}"); traceback.print_exc() - print(f"\n{len(fns) - failed}/{len(fns)} passed") - sys.exit(1 if failed else 0) +# No `if __name__ == "__main__"` runner here on purpose. There used to be one in +# every test file, and in this file it sat MID-FILE -- so four tests appended after +# it were defined, never called, and the file still printed "6/6 passed". The third +# instance of this session's recurring bug: a report that cannot count what it does +# not reach. `python3 tests/run_all.py` enumerates the module instead, so a test's +# POSITION in the file cannot decide whether it runs; it also fails if a file +# defines no tests at all. Run a single file with `run_all.py `. diff --git a/tests/test_maven.py b/tests/test_maven.py index ff37dcf..ae6bbc9 100644 --- a/tests/test_maven.py +++ b/tests/test_maven.py @@ -129,15 +129,10 @@ def test_missing_java_source_is_caught(): assert any(d.kind == "missing_java_src" and d.tu.endswith("B.java") for d in discs), discs - -if __name__ == "__main__": - import traceback - fns = [v for k, v in sorted(globals().items()) if k.startswith("test_")] - failed = 0 - for fn in fns: - try: - fn(); print(f"PASS {fn.__name__}") - except Exception: - failed += 1; print(f"FAIL {fn.__name__}"); traceback.print_exc() - print(f"\n{len(fns) - failed}/{len(fns)} passed") - sys.exit(1 if failed else 0) +# No `if __name__ == "__main__"` runner here on purpose. There used to be one in +# every test file, and in this file it sat MID-FILE -- so four tests appended after +# it were defined, never called, and the file still printed "6/6 passed". The third +# instance of this session's recurring bug: a report that cannot count what it does +# not reach. `python3 tests/run_all.py` enumerates the module instead, so a test's +# POSITION in the file cannot decide whether it runs; it also fails if a file +# defines no tests at all. Run a single file with `run_all.py `. diff --git a/tests/test_run_all.py b/tests/test_run_all.py index bcc54b0..2e37733 100644 --- a/tests/test_run_all.py +++ b/tests/test_run_all.py @@ -117,18 +117,10 @@ def test_the_real_suite_is_discovered_whole(): assert run_all.test_files() == on_disk assert os.path.basename(__file__) in on_disk, "the runner's own tests" - -if __name__ == "__main__": - import traceback - fns = [v for k, v in sorted(globals().items()) if k.startswith("test_")] - failed = 0 - for fn in fns: - try: - fn() - print(f"PASS {fn.__name__}") - except Exception: - failed += 1 - print(f"FAIL {fn.__name__}") - traceback.print_exc() - print(f"\n{len(fns) - failed}/{len(fns)} passed") - sys.exit(1 if failed else 0) +# No `if __name__ == "__main__"` runner here on purpose. There used to be one in +# every test file, and in this file it sat MID-FILE -- so four tests appended after +# it were defined, never called, and the file still printed "6/6 passed". The third +# instance of this session's recurring bug: a report that cannot count what it does +# not reach. `python3 tests/run_all.py` enumerates the module instead, so a test's +# POSITION in the file cannot decide whether it runs; it also fails if a file +# defines no tests at all. Run a single file with `run_all.py `. diff --git a/tests/test_triage.py b/tests/test_triage.py index 679324e..2c2ca99 100644 --- a/tests/test_triage.py +++ b/tests/test_triage.py @@ -60,15 +60,10 @@ def test_render_caps_bazel_only_tail(): # cmake_only (actionable) is never capped assert "-std=gnu++17" in text - -if __name__ == "__main__": - import traceback - fns = [v for k, v in sorted(globals().items()) if k.startswith("test_")] - failed = 0 - for fn in fns: - try: - fn(); print(f"PASS {fn.__name__}") - except Exception: - failed += 1; print(f"FAIL {fn.__name__}"); traceback.print_exc() - print(f"\n{len(fns) - failed}/{len(fns)} passed") - sys.exit(1 if failed else 0) +# No `if __name__ == "__main__"` runner here on purpose. There used to be one in +# every test file, and in this file it sat MID-FILE -- so four tests appended after +# it were defined, never called, and the file still printed "6/6 passed". The third +# instance of this session's recurring bug: a report that cannot count what it does +# not reach. `python3 tests/run_all.py` enumerates the module instead, so a test's +# POSITION in the file cannot decide whether it runs; it also fails if a file +# defines no tests at all. Run a single file with `run_all.py `. diff --git a/tests/test_vcpkg_plumbing.py b/tests/test_vcpkg_plumbing.py index e7d9e32..cdb9263 100644 --- a/tests/test_vcpkg_plumbing.py +++ b/tests/test_vcpkg_plumbing.py @@ -143,21 +143,191 @@ def test_static_libs_go_on_the_link_line_by_path(): assert 'flags.append("-l" + n)' in impl -if __name__ == "__main__": - # A runner, because without one this file EXITED 0 WITHOUT RUNNING ANYTHING -- - # a test suite that cannot fail, which is the same bug as a glob that cannot - # fail (finding 35). Found by checking that the suite's output accounted for - # every test file, rather than trusting the exit code. - import sys as _sys, traceback - fns = [v for k, v in sorted(globals().items()) if k.startswith("test_")] - failed = 0 - for fn in fns: - try: - fn() - print(f"PASS {fn.__name__}") - except Exception: - failed += 1 - print(f"FAIL {fn.__name__}") - traceback.print_exc() - print(f"\n{len(fns) - failed}/{len(fns)} passed") - _sys.exit(1 if failed else 0) +# --------------------------------------------------------------------------- +# Finding 36: what a fresh clone lacks. The cheap guards for the five things that +# made `git clone && bazel build` fail after Build/full was closed -- every one of +# them green on the machine that developed it, which is the whole problem. +# --------------------------------------------------------------------------- +def test_the_asset_script_resolves_absolute_paths(): + """The two path bugs that made every distfile lookup miss. + + vcpkg receives the index as an EXECROOT-RELATIVE path and invokes the + asset-cache script from its OWN working directory, so both the index path and + the paths inside it must be absolutized before vcpkg is exec'd. Fixing only one + moves the failure down a line (`awk: cannot open` becomes `cp: cannot stat`), + and either way vcpkg reports "no asset cache hits" and x-block-origin refuses + the network -- a message that blames the pin rather than the path. + + It survived because the dev machine's vcpkg checkout already had + downloads/tools/cmake-4.4.0-linux from an earlier `Meta/ladybird.py vcpkg` run, + so vcpkg never asked the script for a tool at all. + """ + sh = _read("Meta/vcpkg_build.sh") + assert "EXECROOT" in sh, "the index path is still execroot-relative" + # The index's VALUES too, not just its path: awk finding the row is half the job. + assert 'root "/" $2' in sh, \ + "index entries are not absolutized, so cp runs from vcpkg's cwd" + + +def test_the_vcpkg_checkout_glob_is_the_finding_35_pattern_and_is_known(): + """//Build/vcpkg:tree globs a tree a fresh clone does not have, allow_empty. + + Deliberately NOT asserted as fixed -- it is not. This is the same + `glob(["**"], allow_empty = True)` over a foreign tree finding 35 was about, + one directory over: on a clone it matches exactly one file (its own + BUILD.bazel) and reports nothing. What is pinned here is that the gap is + DOCUMENTED, so nobody reads the top of the README and concludes a clone works. + + Making it allow_empty = False is the real fix and breaks the dev loop until + Build/vcpkg is a git_repository (gap 7). A test saying "this is a known hole" + is worth more than one pretending the hole is closed. + """ + tree = _read("Build/vcpkg/BUILD.bazel") + assert "allow_empty = True" in tree, \ + "if this is now False the gap is closed -- update this test and the README" + # The .git exclusion is a LIE the no-sandbox action gets away with: vcpkg + # resolves versioned ports with `git read-tree`, so the excluded tree is + # load-bearing. Pinned so the contradiction stays visible. + assert ".git/**" in tree + readme = _example_readme() + assert "Build/vcpkg" in readme, "the missing checkout is not documented" + assert "read-tree" in readme, "the load-bearing .git is not documented" + + +def test_the_network_claim_is_scoped_to_vcpkgs_own_downloader(): + """`requires-network: "0"` enforces nothing, and one port calls pip. + + x-block-origin governs vcpkg's OWN downloader; the angle overlay-port runs + `pip install ply` via x_vcpkg_get_python_packages, which is not a distfile and + never reaches the pin. Nothing catches it either: `requires-network: "0"` is a + scheduling hint, `no-sandbox: "1"` means there is no namespace to enforce it + in, and `use_default_shell_env = True` hands the action this machine's + HTTP_PROXY -- so pip silently succeeded for months. + + A control that is not enforced is indistinguishable from one that is not there + -- finding 35's sentence about globs, applied to an execution_requirements key. + Until ply is pinned, the guard is that the README scopes the claim honestly + instead of repeating "zero network access". + """ + bzl = _read("vcpkg.bzl") + assert "no-sandbox" in bzl and "use_default_shell_env = True" in bzl, \ + "if either changed, re-derive whether the network is actually blocked" + readme = _example_readme() + assert "pip install ply" in readme, "the pip hole is not documented" + assert "scheduling hint" in readme, \ + "requires-network is still presented as if it blocked the network" + + +def test_the_unpinned_hsts_download_is_declared_and_documented(): + """An input with no producing rule, and no hash anywhere. + + codegen_root.bzl names Build/caches/HSTSPreload/...json as a genrule src, so + Bazel fails cleanly with `missing input file` rather than generating an empty + table. But nothing produces it: CMake downloads it at configure time from + Chromium's `main` branch, UNVERSIONED -- so there is no revision to pin even if + we wanted to, which is the more interesting half of the problem. + """ + codegen = _read("codegen_root.bzl") + assert "Build/caches/HSTSPreload" in codegen + readme = _example_readme() + assert "HSTSPreload" in readme and "unversioned" in readme.lower(), \ + "the unpinned HSTS download is not documented as a clone blocker" + + +def _example_readme(): + with open(os.path.join(_WS, "..", "README.md")) as f: + return f.read() + +def test_the_pip_installed_package_is_pinned_and_pip_cannot_reach_an_index(): + """Finding 36's substantive fix: the one dependency the capture cannot see. + + `x-block-origin` covers everything that goes through vcpkg's downloader, and + the angle port's `pip install ply` does not go through it -- so the 76-distfile + pin says nothing about ply and nothing blocked the fetch. Three properties make + it a real pin rather than a comment, and all three have to hold together: + + * the wheel is named by an IMMUTABLE url + hash (files.pythonhosted.org is + content-addressed; `pip install ply` resolves against whatever PyPI serves + today), + * pip is told it may not use an index at all, so an UNPINNED package is an + error rather than a download -- the pip-side equivalent of x-block-origin, + * and the proxy variables are unset, because an inherited HTTP_PROXY is + precisely how this went unnoticed. --no-index alone would probably do; the + lesson of finding 36 is that one unenforced control is not a control. + """ + wheels = _read("vcpkg_python_packages.bzl") + assert "files.pythonhosted.org" in wheels, "the wheel URL is not content-addressed" + assert re.search(r'"sha256-[A-Za-z0-9+/=]{20,}"', wheels), "no integrity hash" + sh = _read("Meta/vcpkg_build.sh") + assert "PIP_NO_INDEX=1" in sh, "pip may still resolve from an index" + assert "PIP_FIND_LINKS" in sh, "the pinned wheels are not offered to pip" + assert re.search(r"unset .*HTTPS_PROXY", sh), \ + "an inherited proxy is how the pip fetch stayed invisible" + # The wheel must be a declared INPUT of the action, not merely fetched: a repo + # Bazel creates but no action depends on is not in the sandbox. + assert "python_wheels" in _read("vcpkg.bzl") + assert "python_wheels = ['@vcpkg_pywheel_ply//file']" in _read("BUILD.bazel") + + +def test_the_use_repo_list_includes_the_pip_wheels(): + """bzlmod needs every extension-created repo named in use_repo, and the wheels + are created by the SAME extension as the 76 distfiles. + + The emitter derives the wheel names from vcpkg_python_packages.bzl rather than + restating them, so this checks the round trip: what --use-repo prints must be + exactly what MODULE.bazel says. A drift here is a "no such repository" error a + long way from its cause, and it is the specific drift the pip pin introduces -- + the wheels are hand-maintained (no instrument can capture them), so they are the + one part of this list a regeneration could silently drop. + """ + import subprocess + import sys + # $LADYBIRD_ROOT scrubbed deliberately: another test in this suite points it at + # a temp fixture checkout and the subprocess would INHERIT it, so this test + # passed alone and failed in the full run. Found by run_all.py, which runs every + # file in one process -- the per-file runners could not have surfaced it. + env = {k: v for k, v in os.environ.items() if k != "LADYBIRD_ROOT"} + out = subprocess.run( + [sys.executable, "Meta/emit_vcpkg_bazel.py", + "--assets", "Meta/vcpkg_assets.tsv", "--use-repo"], + cwd=_WS, capture_output=True, text=True, env=env) + assert out.returncode == 0, out.stderr + emitted = re.findall(r"'([^']+)'", out.stdout) + block = _read("MODULE.bazel").split("vcpkg_deps = use_extension", 1)[1] \ + .split("use_repo(", 1)[1].split("\n)", 1)[0] + assert emitted == re.findall(r"'([^']+)'", block), \ + "MODULE.bazel's use_repo has drifted from the emitter" + assert "vcpkg_pywheel_ply" in emitted + + +def test_the_git_archive_staging_fails_when_the_archives_are_absent(): + """Finding 36's worst offender: three ways to succeed while copying nothing. + + The staging was `if [ -d "$D" ]; then cp "$D"/*.tar.gz ... 2>/dev/null || true; + fi` -- a directory test that skips, a redirect that hides, and a `|| true` that + forgives. On a fresh clone the directory does not exist, all four archives were + silently absent, and skia failed ~20 minutes later with a googlesource URL that + names neither this directory nor the tarball. + + So: no `|| true`, and an explicit check that names what is missing. This does not + assert the archives are FETCHED -- they are not; `git archive` output has no URL + to http_file, so vcpkg_git_archives.bzl records their hashes but nothing creates + them (still gap 7). It asserts that their absence is loud. + """ + sh = _read("Meta/vcpkg_build.sh") + stage = sh.split("Pre-place the git-sourced externals", 1)[1].split("\n# ---", 1)[0] + code = "\n".join(l for l in stage.splitlines() if not l.strip().startswith("#")) + assert "|| true" not in code, "the copy can still silently do nothing" + assert "2>/dev/null" not in code, "the copy still hides its own errors" + assert "exit 1" in code, "a missing archive is not a hard failure" + # And the hashes it points the reader at really are checked in. + archives = _read("vcpkg_git_archives.bzl") + assert archives.count(".tar.gz'") == 4, "the four pinned archives changed" + +# No `if __name__ == "__main__"` runner here on purpose. There used to be one in +# every test file, and in this file it sat MID-FILE -- so four tests appended after +# it were defined, never called, and the file still printed "6/6 passed". The third +# instance of this session's recurring bug: a report that cannot count what it does +# not reach. `python3 tests/run_all.py` enumerates the module instead, so a test's +# POSITION in the file cannot decide whether it runs; it also fails if a file +# defines no tests at all. Run a single file with `run_all.py `. From 885852cdfed429ba8a2affa20b319e983e84dbd9 Mon Sep 17 00:00:00 2001 From: Ulf Adams Date: Sat, 8 Aug 2026 12:16:30 +0000 Subject: [PATCH 08/59] Ladybird: keep vcpkg's 3GB scratch off the tmpfs, without a flag The clone-and-build run hit 'No space left on device' reported by the ASSET SCRIPT -- a message blaming the pin for a full disk. vcpkg's buildtrees peak near 3GB and $TMPDIR here is a 7.9GB tmpfs shared with everything else. Fixed by putting the scratch dir next to the declared output (inside bazel-out, i.e. on the same real filesystem as the output base) rather than $TMPDIR. That is also why it is not a flag: this action runs in BOTH the target and exec configurations, so TMPDIR would need --action_env AND --host_action_env (finding 26's duplication again), and the rule's own 'env =' silently beats --action_env anyway. Two ways wrong. It also has to be ABSOLUTE: vcpkg rejects a relative $HOME outright, and $OUT arrives execroot-relative -- the third place in this file where the relative-vs-absolute distinction has bitten, after the asset index and its entries. Scratch is now cleaned on INT/TERM as well as EXIT (abandoned runs leaving multi-GB trees is how the disk filled), and the action reports its free space so a too-small disk says so in its own voice. --- examples/ladybird/README.md | 8 +++--- .../ladybird/workspace/Meta/vcpkg_build.sh | 27 ++++++++++++++----- 2 files changed, 25 insertions(+), 10 deletions(-) diff --git a/examples/ladybird/README.md b/examples/ladybird/README.md index 405a01b..15c11e9 100644 --- a/examples/ladybird/README.md +++ b/examples/ladybird/README.md @@ -137,10 +137,10 @@ mkdir -p Build/caches/HSTSPreload && curl -Lo Build/caches/HSTSPreload/transport # NB `ply` no longer needs anything: the wheel is pinned and pip runs with # PIP_NO_INDEX (gap 8). # -# 1b. vcpkg's buildtrees peak around 3 GB, and a default /tmp tmpfs is often too -# small -- which surfaces as `No space left on device` from the ASSET SCRIPT, -# i.e. a message that blames the pin for a full disk. -# --action_env=TMPDIR=... if /tmp is small. +# NB vcpkg's buildtrees peak around 3 GB. They go next to the declared output +# (inside bazel-out) rather than $TMPDIR, so a small /tmp tmpfs is not a +# problem and no flag is needed -- see the note in Meta/vcpkg_build.sh for why +# the --action_env route is two ways wrong. # 2. Build. This includes the 77 vcpkg ports (~45 min cold) and the 10 Rust # crates + flapc + cranelift-compiler: the libraries depend on them through # //Meta/vcpkg: and //:_lib, so there is no separate step. Build diff --git a/examples/ladybird/workspace/Meta/vcpkg_build.sh b/examples/ladybird/workspace/Meta/vcpkg_build.sh index 3ecc35f..8636b19 100755 --- a/examples/ladybird/workspace/Meta/vcpkg_build.sh +++ b/examples/ladybird/workspace/Meta/vcpkg_build.sh @@ -53,12 +53,27 @@ SRC="${4:?ladybird source root}" TRIPLET="${5:-x64-linux-dynamic}" WHEELS="${6-}" -# vcpkg's buildtrees peak around 3 GB, and $TMPDIR is often a tmpfs sized well -# under that (7.9 GB here, shared with everything else) -- which surfaces as -# `cp: error writing ...: No space left on device` from the ASSET SCRIPT, i.e. a -# message that blames the pin for a full disk. Honour $TMPDIR when it is set, so a -# caller can point this at real disk, and say where the scratch went. -WORK="$(mktemp -d)" +# vcpkg's buildtrees peak around 3 GB and $TMPDIR is frequently a tmpfs sized well +# under that (7.9 GB here, shared with everything else), which surfaces as +# `cp: error writing ...: No space left on device` from the ASSET SCRIPT -- a +# message that blames the pin for a full disk. +# +# So do not use $TMPDIR at all: put the scratch dir NEXT TO THE DECLARED OUTPUT, +# which is inside bazel-out and therefore on whatever real filesystem the output +# base lives on. That also removes a flag from the recipe -- pointing $TMPDIR +# somewhere bigger needs BOTH --action_env and --host_action_env (this action runs +# in the target AND exec configurations, finding 26's duplication again), and an +# `env =` on the rule silently beats --action_env anyway, so the flag route is two +# ways wrong. Report the free space either way: a 3 GB build on a 2 GB disk should +# say so in its own voice rather than 20 minutes later in someone else's. +# mkdir first: Bazel creates the declared output dir but not necessarily its parent +# before the action runs, and mktemp does not create intermediate dirs. Absolute, +# because vcpkg rejects a relative $HOME outright ("was not an absolute path") and +# $OUT arrives execroot-relative -- the same relative-vs-absolute distinction that +# broke the asset index above, in a third place. +mkdir -p "$(dirname "$OUT")" +OUT_PARENT="$(cd "$(dirname "$OUT")" && pwd)" +WORK="$(mktemp -d "$OUT_PARENT/vcpkg-scratch.XXXXXX")" echo "vcpkg_build: scratch dir $WORK ($(df -h "$WORK" | awk 'NR==2{print $4}') free)" >&2 # Cleaned up on the way out INCLUDING on failure -- an interrupted run used to # leave a multi-GB tree behind, and several of them is how the disk filled. From 9bff7bb47ffc4c47acd785ac3c01318aeebae379 Mon Sep 17 00:00:00 2001 From: Ulf Adams Date: Mon, 10 Aug 2026 15:28:34 +0000 Subject: [PATCH 09/59] Ladybird: the fresh clone builds and renders -- and the run recipe was the seventh blocker With the six clone blockers fixed, the full build on the virgin clone is RC=0 (2,842 actions, all six binaries). Then the documented run command failed with 'mkdir: Permission denied' -- because its last line symlinked $PWD/Build/full/share/Lagom, CMake's build tree, into the resource root. Six findings about a fresh clone not having Build/full, and the recipe for running the result depended on Build/full. The resource root needs no CMake: Base/res (in the clone) plus pdf.js from //:vcpkg_installed, with pdfjs-ladybird-transport.mjs in web/ where UI/cmake/ResourceFiles.cmake puts it. Assembled that way it is diff -rq identical to Build/full/share/Lagom, and the clone's binaries then render --headless=text and --headless=layout-tree byte-identically to the CMake reference on all three test pages. README: staging block rewritten to be clone-correct (with cp --no-preserve=mode, since Bazel's outputs are read-only and the second run would fail), top claim restated as 'builds and renders, three inputs staged by hand', gap 5 updated. Case doc: finding 36 gains the seventh blocker. --- docs/CASE-ladybird-migration.md | 33 ++++++++++++++++++++++++ examples/ladybird/README.md | 45 ++++++++++++++++++++++++++------- 2 files changed, 69 insertions(+), 9 deletions(-) diff --git a/docs/CASE-ladybird-migration.md b/docs/CASE-ladybird-migration.md index 4568103..b1640da 100644 --- a/docs/CASE-ladybird-migration.md +++ b/docs/CASE-ladybird-migration.md @@ -1790,6 +1790,39 @@ command, run the way the user runs it, in a directory that has never seen this project.** That is one line of shell, it costs ten minutes, and I should have run it before the first time I said yes. +### Then the seventh: the recipe for running it pointed at CMake's build tree + +With all six fixed, the clone builds — `//:vcpkg_installed` offline (76 ports, +`pip is offline; 1 pinned wheel(s)`), then all six binaries, 2,842 actions, RC=0 — +and then the *documented run command* fails with `Runtime error: mkdir: Permission +denied (errno=13)`, which is what Ladybird says when it cannot find its resource +root. The README's staging block ended with: + +```sh +ln -sfn "$PWD/Build/full/share/Lagom" "$ER/bazel-out/k8-fastbuild/share" +``` + +**`Build/full`.** Six findings about a fresh clone not having CMake's build tree, +and the last line of the recipe symlinks CMake's build tree. It had never been read +as an instruction, only as something that already worked on my machine — the same +mechanism as all six, one layer further out, in prose rather than in a `glob`. The +resource tree needs no CMake at all: it is `Base/res` (in the clone) plus pdf.js from +`//:vcpkg_installed` (`share/pdfjs/{build,web}`, with +`pdfjs-ladybird-transport.mjs` moved into `web/`, which is where +`UI/cmake/ResourceFiles.cmake` puts it). Assembled that way it is `diff -rq`-identical +to `Build/full/share/Lagom`, and the fresh clone's binaries then render +`--headless=text` and `--headless=layout-tree` **byte-identically to the CMake +reference on all three test pages**. Two smaller notes worth keeping: Bazel's outputs +are read-only, so `cp -r` propagates that and the second staging run fails with +`Permission denied` (`cp --no-preserve=mode`); and the six render RCs were 1 for a +*single* reason — a missing resource root — which is a reminder that a nonzero exit +from a browser is one bit and says nothing about which of six pages failed. + +So the honest state is: **a fresh clone builds and renders, with three inputs staged +by hand** (`Build/vcpkg` + its `.git`, the unpinned HSTS table, the four +`git archive` tarballs). That is a different sentence from "clone and build", and the +difference is exactly what rows 1–3 of the README table say. + ## Plan for the rest - **Ring 1c — BUILD generation to compile+link parity, bottom-up.** AK → diff --git a/examples/ladybird/README.md b/examples/ladybird/README.md index 15c11e9..1f652f0 100644 --- a/examples/ladybird/README.md +++ b/examples/ladybird/README.md @@ -27,13 +27,17 @@ the closure of all six returns **0** targets under `Build/full` (down from 741). `Build/full` is still needed to *regenerate* the BUILD files and to run the parity harness — a converter-development dependency, not a build dependency. -**But `git clone && bazel build //:ladybird` on a fresh clone still does NOT work, -and this section said it did.** Tested properly for the first time — `git clone` -into a new directory, overlay dropped in, nothing else — it fails, and it fails -**six separate times**. `Build/full` was the dependency I removed; it was not the -only one, and five of the six have nothing to do with it. Two are fixed here -(#4, #5) and one now fails loudly instead of silently (#6); the rest are inputs the -recipe must produce or document: +**A fresh clone now builds and renders — with three inputs staged by hand.** The +end-to-end test (clone into a new directory, drop in the overlay, nothing else) +was run for the first time and failed **six separate times**; `Build/full` was +the dependency I had removed, and **five of the six had nothing to do with it**. +All six are addressed below, and the result is verified on the clone rather than +asserted: `//:vcpkg_installed` builds all 76 ports offline, all six binaries +build (2,842 actions, RC=0), and `--headless=text`/`--headless=layout-tree` +are **byte-identical to the CMake reference on all three test pages**. Rows 1–3 +are still manual staging: two are fixed bugs (#4, #5), one now fails loudly +instead of 20 minutes later (#6), and the remainder are inputs the recipe must +produce or document: | # | What is missing on a fresh clone | Why the dev machine hid it | |---|---|---| @@ -200,10 +204,28 @@ for b in WebContent RequestServer ImageDecoder Compositor WebWorker; do cp -f ba # cranelift-compiler is found as a sibling of the spawning binary (Ladybird's own # lookup chain), so it needs no path baked in -- only to be there. cp -f bazel-bin/cranelift-compiler "$ER/bazel-out/k8-fastbuild/libexec/" -ln -sfn "$PWD/Build/full/share/Lagom" "$ER/bazel-out/k8-fastbuild/share" +# The resource root, assembled from the CLONE and from Bazel's own vcpkg tree. +# This line used to read `ln -sfn "$PWD/Build/full/share/Lagom" ...` -- i.e. the +# recipe for running the Bazel build pointed at CMake's build tree, a seventh +# thing a fresh clone does not have (finding 36). Everything in it is either in +# Base/res or in //:vcpkg_installed; `chmod u+w` because Bazel's outputs are +# read-only and `cp -r` preserves that. +L="$ER/bazel-out/k8-fastbuild/share/Lagom" +chmod -R u+w "$L" 2>/dev/null; rm -rf "$L"; mkdir -p "$L/ladybird/pdfjs/web" +cp -r Base/res/. "$L/" +P=bazel-bin/vcpkg_installed/x64-linux-dynamic/share/pdfjs +cp -r --no-preserve=mode "$P/build" "$L/ladybird/pdfjs/" +cp -r --no-preserve=mode "$P/web/." "$L/ladybird/pdfjs/web/" +# UI/cmake/ResourceFiles.cmake stages this one file into pdfjs/web/, not pdfjs/. +mv "$L/ladybird/pdfjs/pdfjs-ladybird-transport.mjs" "$L/ladybird/pdfjs/web/" ./bazel-bin/ladybird --headless=text file:///tmp/test-page.html ``` +The assembled tree is byte-identical to CMake's `Build/full/share/Lagom` +(`diff -rq`, 0 differences), and with it a fresh clone renders `--headless=text` +and `--headless=layout-tree` byte-identically to the CMake reference on all +three test pages. + ## Known gaps Honest inventory of what stops this from being a clone-and-build. @@ -414,7 +436,12 @@ Honest inventory of what stops this from being a clone-and-build. spawns its service binaries by looking next to itself, and `share/Lagom` for resources. A real Bazel setup would express this with `data` + runfiles; doing so means teaching Ladybird's process-launch path about runfiles, so it - is a change to the target, not just to the BUILD files. + is a change to the target, not just to the BUILD files. **The staging block + itself was a clone-and-build blocker until now** — its last line symlinked + `Build/full/share/Lagom`, CMake's build tree, into place. Everything the + resource root needs is in `Base/res` plus `//:vcpkg_installed`'s pdf.js, and + the block above assembles it from those; the result is `diff -rq`-identical to + CMake's. Finding 36. 6. **`rules_qt` is not on the BCR.** `MODULE.bazel` uses an `archive_override` pointing at kklochkov/rules_qt v2.0.1's release tarball (stock upstream, no patches). The BCR's `rules_qt` module is Vertexwahn's unrelated `rules_qt6`. From 66c1097d5b5e93a464b1a659a3107a3c346aaf1e Mon Sep 17 00:00:00 2001 From: Ulf Adams Date: Mon, 10 Aug 2026 15:47:27 +0000 Subject: [PATCH 10/59] Ladybird: what closing the last three clone blockers takes, measured Answering 'so what's needed to make it work' by testing each candidate fix rather than estimating. Two of the three staged inputs are one Bazel rule; the third is an upstream change to Ladybird that no rule can substitute for. Rows 1+2 (Build/vcpkg and its .git) need a custom repository_rule that shells out to git clone. Four shortcuts tested and rejected: git_repository/new_git_repository STRIP .git (so the built-in rule for 'get a git repo' cannot deliver the property this dep needs); a custom rule that clones does keep it, and glob(['**']) carries the .git files as declared inputs, which also fixes row 2's undeclared-input lie; --depth 1 fails with vcpkg's own 'Try again with a full vcpkg clone' because the pinned port trees live in history, so it is a 121MB/~1min full clone; and dropping builtin-baseline to avoid .git 'works' while silently moving 10 dep versions (ffmpeg 7.1.1->8.1.2, harfbuzz 10.2->14.2.1, mimalloc 2->3, plus zlib/freetype/dbus/fontconfig/libedit/libwebp/cpptrace). Positive control: a fresh full clone resolves all 78 ports to exactly the versions this dev checkout resolves. Row 6 (the four git archive tarballs) is the same rule's second user. Row 3 (HSTS) is not a Bazel problem: pinning it only on the Bazel side works mechanically but the pinned file differs from the one CMake fetched and the generated table differs, so it trades a hermeticity gap for a parity gap. The fix belongs in Meta/CMake/hsts_preload.cmake. General shape worth keeping: a converter can detect an unpinned input in the foreign build system but cannot pin it unilaterally. --- docs/CASE-ladybird-migration.md | 60 +++++++++++++++++++++++++++++++++ examples/ladybird/README.md | 14 ++++++++ 2 files changed, 74 insertions(+) diff --git a/docs/CASE-ladybird-migration.md b/docs/CASE-ladybird-migration.md index b1640da..47fdaac 100644 --- a/docs/CASE-ladybird-migration.md +++ b/docs/CASE-ladybird-migration.md @@ -1823,6 +1823,66 @@ by hand** (`Build/vcpkg` + its `.git`, the unpinned HSTS table, the four `git archive` tarballs). That is a different sentence from "clone and build", and the difference is exactly what rows 1–3 of the README table say. +### What closing the last three actually takes — measured, not estimated + +Ulf's follow-up was the right one: *so what's needed to make it work?* Three +staged inputs, and the answers are not symmetric — two are a single Bazel rule, and +one is an upstream change to Ladybird that no Bazel rule can substitute for. + +**`Build/vcpkg` + its `.git` (rows 1–2) — one custom `repository_rule`, and the four +obvious shortcuts are all dead ends.** I tested each rather than reasoning about it: + +- **`git_repository` / `new_git_repository` cannot be used at all: they strip + `.git`.** Fetched a small repo with `new_git_repository` — no `.git` in the + external directory. So the built-in rule for "get a git repo" cannot deliver the + one property this dependency needs. A custom repo rule that shells out to + `git clone` *does* keep it, and Bazel's `glob(["**"])` then carries the + `.git` files as declared action inputs (in the probe, 29 of 40 `srcs` were + under `.git/`) — which also converts row 2 from an *undeclared input the + build secretly reads* into a normal one. +- **`--depth 1` does not work.** A shallow clone at the baseline is 8.7 MB and + `read-tree` succeeds for *some* trees, so it looks promising — then resolution + fails on ffmpeg and harfbuzz with `failed to unpack tree object` and vcpkg's own + advice: `vcpkg was cloned as a shallow repository. Try again with a full vcpkg + clone.` The pinned versions' port trees live in **history**, not at the baseline + commit; that is the whole point of the version database. So it is a full clone: + 121 MB of `.git`, ~63–88 s, once, cached in Bazel's repo cache. +- **Dropping `builtin-baseline` to avoid needing `.git` is the dangerous + shortcut, and it is the one I would have been tempted by.** Without the baseline + vcpkg resolves against the checked-out `ports/` and needs no `read-tree` at all — + it "works". It also **silently moves 10 dependencies**: ffmpeg 7.1.1#5 → 8.1.2#3, + harfbuzz 10.2.0 → 14.2.1#2, mimalloc 2.2.7 → 3.4.3, plus zlib, freetype, dbus, + fontconfig, libedit, libwebp and cpptrace. A "fix" for a hermeticity blocker that + changes ten dependency versions is not a fix; it is the same class of error as + everything else in this document, dressed as simplification. +- The positive control: a **fresh** full clone at the baseline, driven by the same + manifest, resolves all **78 ports to exactly the versions my dev checkout + resolves** (`diff`, 0 differences). So the checkout carries no local state beyond + the ref — which is what makes the repo rule sound. + +**The four `git archive` tarballs (row 6) are the same rule's second user.** Clone at +the pinned ref, `git archive`, expose the `.tar.gz`; `vcpkg_from_git` honours a +pre-placed `downloads/-.tar.gz`, and `git archive` is byte-deterministic +for a fixed ref. That makes `vcpkg_git_archives.bzl`'s SHA512s *checked* rather than +decorative — one rule closes rows 1, 2 and 6. + +**The HSTS table (row 3) is not a Bazel problem, and this is the interesting one.** +`Meta/CMake/hsts_preload.cmake` fetches +`raw.githubusercontent.com/chromium/chromium/**main**/net/http/transport_security_state_static.json` +— an unversioned ref, at CMake *configure* time. Pinning it on the Bazel side works +mechanically (I fetched it at tag `139.0.7258.5` and the generator ran fine), but it +does not work *honestly*: the pinned file is 18.7 MB against the 10.5 MB one on this +machine, and the generated table differs. **Pinning only on the Bazel side would +make Bazel's output diverge from CMake's — trading a hermeticity gap for a parity +gap, and the parity baseline is what every claim in this document rests on.** The +real fix is upstream: pin the URL to a revision in Ladybird's CMake, then `http_file` +the same revision with the same sha256 and both build systems consume one pinned +input. Worth stating plainly because it is a general shape: **an unpinned input in +the foreign build system is not something the converter can pin unilaterally.** It +can *detect* it (Bazel already fails cleanly with `missing input file`, which is how +it was found), and detecting it is the converter's job; fixing it is a change to the +project. + ## Plan for the rest - **Ring 1c — BUILD generation to compile+link parity, bottom-up.** AK → diff --git a/examples/ladybird/README.md b/examples/ladybird/README.md index 1f652f0..94ddc39 100644 --- a/examples/ladybird/README.md +++ b/examples/ladybird/README.md @@ -65,6 +65,20 @@ dependency I went looking for, so it is the one I found.** The check that would have caught all five is not a better `cquery` — it is doing the clone. See [Known gaps](#known-gaps). +### What closing the last three takes + +Measured, not estimated — each shortcut below was tested and rejected (finding 36): + +| Blocker | What closes it | Shortcuts that do **not** work | +|---|---|---| +| Rows 1+2: `Build/vcpkg` and its `.git` | **One custom `repository_rule`** that shells out to `git clone` at `vcpkg.json`'s `builtin-baseline`. It keeps `.git`, and `glob(["**"])` then carries those files as *declared* inputs — which also fixes row 2's undeclared-input lie | `git_repository`/`new_git_repository` **strip `.git`**. `--depth 1` fails (`vcpkg was cloned as a shallow repository`) — the pinned port trees live in history, so it is a full 121 MB clone, ~1 min, once. Dropping `builtin-baseline` "works" and **silently moves 10 dep versions** (ffmpeg 7.1.1→8.1.2, harfbuzz 10.2→14.2.1, mimalloc 2→3, …) | +| Row 6: the four `git archive` tarballs | **The same rule's second user** — clone at the pinned ref, `git archive`, expose the `.tar.gz`. Makes `vcpkg_git_archives.bzl`'s SHA512s checked instead of decorative | — | +| Row 3: the HSTS table | **An upstream change to Ladybird**: `Meta/CMake/hsts_preload.cmake` fetches Chromium's `main`, so pin *that* to a revision, then `http_file` the same revision | Pinning only on the Bazel side: the pinned file (18.7 MB) differs from the one CMake fetched (10.5 MB) and the generated table differs, so it would trade a hermeticity gap for a **parity** gap. A converter can *detect* an unpinned input — Bazel already fails with `missing input file` — but cannot pin it unilaterally | + +A fresh full clone at the baseline resolves all **78 ports to exactly the versions +this dev checkout resolves** (`diff`, 0 differences), which is what makes the repo +rule sound: the checkout carries no local state beyond the ref. + **This claim was false until recently, and the way it was false is the most useful thing in this directory.** The README said it worked; it did not. Every binary depended on **741 targets under `Build/full`**, the overlay shipped four From d8e857053b0f5c767351c0028e45f6034f136cd6 Mon Sep 17 00:00:00 2001 From: Ulf Adams Date: Mon, 10 Aug 2026 16:11:59 +0000 Subject: [PATCH 11/59] Ladybird: the repo rule I proposed already exists in the project, as a script I answered 'what's needed to make it work' by designing a custom repository_rule that clones vcpkg at the baseline. The supporting work was real -- git_repository/new_git_repository do strip .git, a custom rule that shells out to git clone does keep it, and glob(['**']) does carry those files as declared inputs -- but the conclusion was wrong. Ladybird already ships it: Meta/ladybird.py vcpkg (45 lines in Meta/Utils/build_vcpkg.py) clones microsoft/vcpkg, checks out vcpkg.json's builtin-baseline and bootstraps the tool at a pinned tag+SHA512. Run from an empty directory: 75s, RC=0, .git at the right commit. The README's step 1a already said to run it. So blockers 1+2 are a prefetch step to order correctly, not machinery to write -- and a repo rule reimplementing it would be a fork of tooling the project maintains, drifting the moment the baseline moves. Row 6 collapses the same way: clone the pinned URL and `git archive` the ref. Verified by reproducing libyuv's tarball -- SHA512 matches the committed value in vcpkg_git_archives.bzl exactly, so those hashes are checkable rather than decorative. Eight lines of shell instead of a rule. Row 3 (HSTS) stays upstream-only, unchanged. Also records the two shortcuts tested and rejected: --depth 1 (pinned port trees live in history; vcpkg says 'Try again with a full vcpkg clone') and dropping builtin-baseline (silently moves 10 dep versions). Docs, vcpkg_build.sh comment and both workspace copies updated; the general lesson is that the fix I imagine is more available to me than the fix that exists. --- docs/CASE-ladybird-migration.md | 126 ++++++++++-------- examples/ladybird/README.md | 45 +++++-- .../ladybird/workspace/Meta/vcpkg_build.sh | 6 +- 3 files changed, 105 insertions(+), 72 deletions(-) diff --git a/docs/CASE-ladybird-migration.md b/docs/CASE-ladybird-migration.md index 47fdaac..392b5b7 100644 --- a/docs/CASE-ladybird-migration.md +++ b/docs/CASE-ladybird-migration.md @@ -1823,66 +1823,80 @@ by hand** (`Build/vcpkg` + its `.git`, the unpinned HSTS table, the four `git archive` tarballs). That is a different sentence from "clone and build", and the difference is exactly what rows 1–3 of the README table say. -### What closing the last three actually takes — measured, not estimated - -Ulf's follow-up was the right one: *so what's needed to make it work?* Three -staged inputs, and the answers are not symmetric — two are a single Bazel rule, and -one is an upstream change to Ladybird that no Bazel rule can substitute for. - -**`Build/vcpkg` + its `.git` (rows 1–2) — one custom `repository_rule`, and the four -obvious shortcuts are all dead ends.** I tested each rather than reasoning about it: - -- **`git_repository` / `new_git_repository` cannot be used at all: they strip - `.git`.** Fetched a small repo with `new_git_repository` — no `.git` in the - external directory. So the built-in rule for "get a git repo" cannot deliver the - one property this dependency needs. A custom repo rule that shells out to - `git clone` *does* keep it, and Bazel's `glob(["**"])` then carries the - `.git` files as declared action inputs (in the probe, 29 of 40 `srcs` were - under `.git/`) — which also converts row 2 from an *undeclared input the - build secretly reads* into a normal one. -- **`--depth 1` does not work.** A shallow clone at the baseline is 8.7 MB and - `read-tree` succeeds for *some* trees, so it looks promising — then resolution - fails on ffmpeg and harfbuzz with `failed to unpack tree object` and vcpkg's own - advice: `vcpkg was cloned as a shallow repository. Try again with a full vcpkg - clone.` The pinned versions' port trees live in **history**, not at the baseline - commit; that is the whole point of the version database. So it is a full clone: - 121 MB of `.git`, ~63–88 s, once, cached in Bazel's repo cache. -- **Dropping `builtin-baseline` to avoid needing `.git` is the dangerous - shortcut, and it is the one I would have been tempted by.** Without the baseline - vcpkg resolves against the checked-out `ports/` and needs no `read-tree` at all — - it "works". It also **silently moves 10 dependencies**: ffmpeg 7.1.1#5 → 8.1.2#3, - harfbuzz 10.2.0 → 14.2.1#2, mimalloc 2.2.7 → 3.4.3, plus zlib, freetype, dbus, - fontconfig, libedit, libwebp and cpptrace. A "fix" for a hermeticity blocker that - changes ten dependency versions is not a fix; it is the same class of error as - everything else in this document, dressed as simplification. -- The positive control: a **fresh** full clone at the baseline, driven by the same - manifest, resolves all **78 ports to exactly the versions my dev checkout - resolves** (`diff`, 0 differences). So the checkout carries no local state beyond - the ref — which is what makes the repo rule sound. - -**The four `git archive` tarballs (row 6) are the same rule's second user.** Clone at -the pinned ref, `git archive`, expose the `.tar.gz`; `vcpkg_from_git` honours a -pre-placed `downloads/-.tar.gz`, and `git archive` is byte-deterministic -for a fixed ref. That makes `vcpkg_git_archives.bzl`'s SHA512s *checked* rather than -decorative — one rule closes rows 1, 2 and 6. - -**The HSTS table (row 3) is not a Bazel problem, and this is the interesting one.** -`Meta/CMake/hsts_preload.cmake` fetches +### What closing the last three actually takes — and why my first answer was wrong + +Ulf's follow-up was the right one: *so what's needed to make it work?* My first +answer was "a custom `repository_rule` that clones vcpkg at the baseline — it closes +two of the three blockers." I had done real work to support it: I confirmed +`git_repository`/`new_git_repository` **strip `.git`** (so the built-in rule cannot +deliver the one property this dependency needs), that a custom rule shelling out to +`git clone` keeps it, and that `glob(["**"])` then carries the `.git` files as +declared inputs. All true, and all beside the point. + +**Ladybird already ships the thing I was proposing to write.** `Meta/ladybird.py +vcpkg` — 45 lines in `Meta/Utils/build_vcpkg.py` — clones microsoft/vcpkg, checks out +`vcpkg.json`'s `builtin-baseline`, and bootstraps the tool at a tag+SHA512 the +checkout itself pins. I ran it from an empty directory: **75 seconds, RC=0, `.git` +present at the right commit, `vcpkg` binary built.** The README's step 1a has been +telling cloners to run it the whole time. So blockers 1 and 2 are not missing +machinery; they are a **prefetch step that the recipe must run in the right order**, +and the honest fix is a `MODULE.bazel`-adjacent note plus a check that fails clearly +when it has not been run — not a repo rule reimplementing a script the project +maintains. + +Two lessons, and the second is the uncomfortable one. First: **a repo rule that +re-implements the foreign project's own bootstrap script is a fork of it.** It would +drift the moment Ladybird bumps its baseline or its tool metadata, and the drift +would look like a Bazel bug. Second: **I answered a "what's needed" question by +designing, not by reading the project.** Nothing in the environment stopped me from +opening `Meta/Utils/build_vcpkg.py` before proposing to duplicate it — the same +failure mode as answering from the README instead of doing the clone, one level up: +the fix I imagine is more available to me than the fix that exists. + +**The four `git archive` tarballs (row 6) collapse the same way.** No repo rule +needed: clone the pinned URL, `git archive `. I reproduced `libyuv`'s tarball +that way and its SHA512 matched `vcpkg_git_archives.bzl`'s committed value **exactly** +— which is the useful part, because it means the committed hashes are *checkable* and +the reproduction is verifiable in one line rather than trusted. Eight lines of shell, +against a `repository_rule` I would have had to design, test and maintain. + +**The HSTS table (row 3) is the one that genuinely cannot be fixed here, and it is +not a Bazel problem.** `Meta/CMake/hsts_preload.cmake` fetches `raw.githubusercontent.com/chromium/chromium/**main**/net/http/transport_security_state_static.json` — an unversioned ref, at CMake *configure* time. Pinning it on the Bazel side works -mechanically (I fetched it at tag `139.0.7258.5` and the generator ran fine), but it -does not work *honestly*: the pinned file is 18.7 MB against the 10.5 MB one on this -machine, and the generated table differs. **Pinning only on the Bazel side would -make Bazel's output diverge from CMake's — trading a hermeticity gap for a parity -gap, and the parity baseline is what every claim in this document rests on.** The -real fix is upstream: pin the URL to a revision in Ladybird's CMake, then `http_file` -the same revision with the same sha256 and both build systems consume one pinned -input. Worth stating plainly because it is a general shape: **an unpinned input in -the foreign build system is not something the converter can pin unilaterally.** It -can *detect* it (Bazel already fails cleanly with `missing input file`, which is how -it was found), and detecting it is the converter's job; fixing it is a change to the +mechanically (I fetched it at tag `139.0.7258.5` and the generator ran fine), but not +*honestly*: the pinned file is 18.7 MB against the 10.5 MB one this machine +configured with, and the generated table differs. **Pinning only on the Bazel side +would make Bazel's output diverge from CMake's — trading a hermeticity gap for a +parity gap, and the parity baseline is what every claim in this document rests on.** +The fix is upstream: pin the URL to a revision in Ladybird's CMake, then `http_file` +that revision. Worth stating plainly as a general shape: **an unpinned input in the +foreign build system is not something the converter can pin unilaterally.** It can +*detect* it — Bazel already fails cleanly with `missing input file`, which is how it +was found — and detecting it is the converter's job; fixing it is a change to the project. +Two shortcuts I tested and rejected, both of which look like simplifications and one +of which I would have shipped: + +- **`--depth 1` on the vcpkg clone.** 8.7 MB instead of 121 MB, and `read-tree` even + succeeds for some ports — then resolution fails on ffmpeg and harfbuzz with + `failed to unpack tree object` and vcpkg's own advice, `Try again with a full vcpkg + clone`. The pinned versions' port trees live in **history**, not at the baseline + commit; that is what a version database is. +- **Dropping `builtin-baseline`** so `.git` is not needed at all. vcpkg then resolves + against the checked-out `ports/` and needs no `read-tree` — it "works". It also + **silently moves 10 dependencies**: ffmpeg 7.1.1#5 → 8.1.2#3, harfbuzz 10.2.0 → + 14.2.1#2, mimalloc 2.2.7 → 3.4.3, plus zlib, freetype, dbus, fontconfig, libedit, + libwebp and cpptrace. A "fix" for a hermeticity blocker that changes ten dependency + versions is the same class of error as everything else in this document, dressed as + simplification. + +The positive control for all of it: a **fresh** full clone at the baseline, driven by +the same manifest, resolves all **78 ports to exactly the versions this dev checkout +resolves** (`diff`, 0 differences). The checkout carries no local state beyond the +ref — which is precisely why a prefetch step is sufficient and a rule is not needed. + ## Plan for the rest - **Ring 1c — BUILD generation to compile+link parity, bottom-up.** AK → diff --git a/examples/ladybird/README.md b/examples/ladybird/README.md index 94ddc39..bb6e489 100644 --- a/examples/ladybird/README.md +++ b/examples/ladybird/README.md @@ -46,7 +46,7 @@ produce or document: | 3 | **`Build/caches/HSTSPreload/transport_security_state_static.json`** (10 MB) — an *unversioned, unpinned* network download CMake does at configure time (`Meta/CMake/hsts_preload.cmake` fetches Chromium's `main`). `codegen_root.bzl` names it as a genrule `srcs`, so Bazel fails cleanly with `missing input file` — but there is no rule that produces it | the CMake configure had already downloaded it | | 4 | **Two path bugs in `Meta/vcpkg_build.sh`** — the distfile index and its entries were passed **execroot-relative**, and vcpkg invokes the asset-cache script from its own cwd, so `awk`/`cp` looked in the wrong directory. Every asset lookup failed, reported as `no asset cache hits`, and `x-block-origin` then correctly refused the network. Fixed here (absolutize both) | the vcpkg checkout already had `downloads/tools/cmake-4.4.0-linux` from an earlier run, so vcpkg never *asked* the script for a tool | | 5 | **The `angle` port runs `pip install ply`** (`x_vcpkg_get_python_packages`), which is **not** an asset-cache download and therefore not covered by the pin. A real hole in the "zero network access" claim: `vcpkg_tree` set `requires-network: "0"`, but that is a *scheduling hint*, and with `no-sandbox: "1"` **nothing enforced it** — with `use_default_shell_env = True` the action inherited `HTTP_PROXY`/`HTTPS_PROXY` and pip reached PyPI. **Fixed here:** the wheel is pinned by URL+sha256 (`vcpkg_python_packages.bzl`), staged into a find-links dir, and pip runs with `PIP_NO_INDEX` and the proxy variables unset | this sandbox exports a proxy, so pip silently succeeded through it | -| 6 | **The four `vcpkg_from_git` archives were pre-placed from a directory I had made by hand.** `vcpkg_git_archives.bzl` is generated, committed, listed in the table below — and **loaded by nothing**. The staging was `if [ -d ... ]; then cp ... 2>/dev/null \|\| true; fi`: three ways to succeed while copying nothing. Without them skia fails ~20 min in with `git fetch https://android.googlesource.com/.../piex.git … Error code: 128`, naming neither the directory nor the tarball. **Now a hard error in 4 seconds** naming both; producing them from Bazel still needs a repo rule (they are `git archive` output, so there is no URL to `http_file`) | I created the directory by hand while building the pin, months before | +| 6 | **The four `vcpkg_from_git` archives were pre-placed from a directory I had made by hand.** `vcpkg_git_archives.bzl` is generated, committed, listed in the table below — and **loaded by nothing**. The staging was `if [ -d ... ]; then cp ... 2>/dev/null \|\| true; fi`: three ways to succeed while copying nothing. Without them skia fails ~20 min in with `git fetch https://android.googlesource.com/.../piex.git … Error code: 128`, naming neither the directory nor the tarball. **Now a hard error in 4 seconds** naming both; reproducing them is a prefetch, not a rule — they are `git archive` output, so there is no URL to `http_file`, but cloning the pinned URL and `git archive`-ing the pinned ref reproduces the committed SHA512 exactly (verified on `libyuv`) | I created the directory by hand while building the pin, months before | Rows 1–3 are inputs the recipe must produce or document (step 1a). Rows 4–6 were bugs, and 5 is the substantive one: **"zero network access" was measured with @@ -56,9 +56,10 @@ absence of a network — `requires-network: "0"` is a hint, and `no-sandbox: "1" means there is no namespace to enforce. `ply` is now pinned like any other dependency (URL + sha256 → `http_file` → find-links dir → `PIP_NO_INDEX`), which makes an unpinned package a hard error rather than a download. The four -`vcpkg_from_git` archives are still staged from a directory rather than fetched by -Bazel — they are `git archive` output with no URL, so they need a repo rule that -clones at the pinned ref. +`vcpkg_from_git` archives are still staged from a directory rather than produced by +the recipe — they are `git archive` output with no URL, so reproducing them means +cloning each pinned URL and `git archive`-ing the pinned ref (8 lines of shell, and +the committed SHA512s check it). The lesson is the same one as finding 35, one layer out: **`Build/full` was the dependency I went looking for, so it is the one I found.** The check that would @@ -67,17 +68,31 @@ have caught all five is not a better `cquery` — it is doing the clone. See ### What closing the last three takes -Measured, not estimated — each shortcut below was tested and rejected (finding 36): +Two are a **prefetch step**, not new Bazel machinery, and one needs a change to +Ladybird itself. (An earlier draft of this section proposed a custom +`repository_rule` for the first two; that was wrong, and why is in finding 36.) -| Blocker | What closes it | Shortcuts that do **not** work | -|---|---|---| -| Rows 1+2: `Build/vcpkg` and its `.git` | **One custom `repository_rule`** that shells out to `git clone` at `vcpkg.json`'s `builtin-baseline`. It keeps `.git`, and `glob(["**"])` then carries those files as *declared* inputs — which also fixes row 2's undeclared-input lie | `git_repository`/`new_git_repository` **strip `.git`**. `--depth 1` fails (`vcpkg was cloned as a shallow repository`) — the pinned port trees live in history, so it is a full 121 MB clone, ~1 min, once. Dropping `builtin-baseline` "works" and **silently moves 10 dep versions** (ffmpeg 7.1.1→8.1.2, harfbuzz 10.2→14.2.1, mimalloc 2→3, …) | -| Row 6: the four `git archive` tarballs | **The same rule's second user** — clone at the pinned ref, `git archive`, expose the `.tar.gz`. Makes `vcpkg_git_archives.bzl`'s SHA512s checked instead of decorative | — | -| Row 3: the HSTS table | **An upstream change to Ladybird**: `Meta/CMake/hsts_preload.cmake` fetches Chromium's `main`, so pin *that* to a revision, then `http_file` the same revision | Pinning only on the Bazel side: the pinned file (18.7 MB) differs from the one CMake fetched (10.5 MB) and the generated table differs, so it would trade a hermeticity gap for a **parity** gap. A converter can *detect* an unpinned input — Bazel already fails with `missing input file` — but cannot pin it unilaterally | +| Blocker | What closes it | +|---|---| +| 1+2: the vcpkg checkout and its `.git` | **`python3 Meta/ladybird.py vcpkg`** — Ladybird already ships it. It clones microsoft/vcpkg, checks out `vcpkg.json`'s `builtin-baseline` and bootstraps the tool: **75 s, verified from an empty directory**, producing the 121 MB `.git` that `git read-tree` needs. Nothing to write; this is a documentation and ordering problem | +| 6: the four `git archive` tarballs | **~8 lines of shell**: clone each pinned URL, `git archive `. Verified — the reproduced `libyuv` tarball's SHA512 matches `vcpkg_git_archives.bzl` exactly, so the committed hashes become *checked* rather than decorative | +| 3: the HSTS table | **An upstream change to Ladybird.** `Meta/CMake/hsts_preload.cmake` fetches Chromium's `main` (unversioned) at configure time; pin *that* to a revision, then `http_file` the same revision so both build systems consume one pinned input. Pinning only on the Bazel side "works" but the pinned file (18.7 MB) differs from the one CMake fetched (10.5 MB) and the generated table differs — trading a hermeticity gap for a **parity** gap | + +Two shortcuts tested and rejected, because both look like simplifications: + +- **`--depth 1` on the vcpkg clone.** 8.7 MB instead of 121 MB, and `read-tree` + even succeeds for some ports — then resolution fails on ffmpeg and harfbuzz with + vcpkg's own `Try again with a full vcpkg clone`. The pinned versions' port trees + live in **history**; that is what a version database *is*. +- **Dropping `builtin-baseline`** so no `.git` is needed at all. It resolves happily + against the checked-out `ports/` — and **silently moves 10 dependency versions** + (ffmpeg 7.1.1#5 → 8.1.2#3, harfbuzz 10.2.0 → 14.2.1#2, mimalloc 2.2.7 → 3.4.3, + plus zlib, freetype, dbus, fontconfig, libedit, libwebp, cpptrace). A fix for a + hermeticity blocker that changes ten dependency versions is not a fix. A fresh full clone at the baseline resolves all **78 ports to exactly the versions -this dev checkout resolves** (`diff`, 0 differences), which is what makes the repo -rule sound: the checkout carries no local state beyond the ref. +this dev checkout resolves** (`diff`, 0 differences) — so the checkout carries no +local state beyond the ref, which is what makes the prefetch step sufficient. **This claim was false until recently, and the way it was false is the most useful thing in this directory.** The README said it worked; it did not. Every @@ -149,8 +164,10 @@ python3 Meta/ladybird.py vcpkg mkdir -p Build/caches/HSTSPreload && curl -Lo Build/caches/HSTSPreload/transport_security_state_static.json \ https://raw.githubusercontent.com/chromium/chromium/main/net/http/transport_security_state_static.json # (c) the four vcpkg_from_git archives. These are `git archive` output, so -# there is no URL to http_file and Bazel cannot fetch them yet; the -# hashes ARE pinned (vcpkg_git_archives.bzl) so they can be verified. +# there is no URL to http_file; the hashes ARE pinned +# (vcpkg_git_archives.bzl), so reproducing them is a prefetch, not a rule +# -- clone each pinned URL and `git archive` the pinned ref. Verified: +# the reproduced libyuv tarball's SHA512 matches the committed one. # Absent, the build now fails in 4s naming them (gap 7). # NB `ply` no longer needs anything: the wheel is pinned and pip runs with # PIP_NO_INDEX (gap 8). diff --git a/examples/ladybird/workspace/Meta/vcpkg_build.sh b/examples/ladybird/workspace/Meta/vcpkg_build.sh index 8636b19..078d210 100755 --- a/examples/ladybird/workspace/Meta/vcpkg_build.sh +++ b/examples/ladybird/workspace/Meta/vcpkg_build.sh @@ -180,8 +180,10 @@ mkdir -p "$ROOT/downloads" # in vcpkg_git_archives.bzl lists the four expected names -- that file is GENERATED # and checked in, and (finding 36) is currently loaded by nothing at all, which is # the deeper bug this only reports. These are `git archive` output, so unlike the 76 -# distfiles they have no URL to http_file; making Bazel produce them needs a repo -# rule that clones at the pinned ref (gap 7). +# distfiles they have no URL to http_file -- but reproducing them is a PREFETCH, not a +# repo rule: clone the pinned URL and `git archive` the pinned ref, which reproduces +# the committed SHA512 exactly (verified on libyuv). A repo rule here would be a fork +# of tooling the project already owns (gap 7). GIT_ARCHIVES="$SRC/Meta/CMake/vcpkg/git-archives" if ! compgen -G "$GIT_ARCHIVES/*.tar.gz" > /dev/null; then echo "vcpkg_build: no git-sourced externals at $GIT_ARCHIVES" >&2 From 6fcb3c3f0a710b142d7c9e87dc93e7ab9702e7b6 Mon Sep 17 00:00:00 2001 From: Ulf Adams Date: Mon, 10 Aug 2026 16:21:18 +0000 Subject: [PATCH 12/59] Ladybird: why a submodule cannot pin vcpkg -- it pins a checkout, vcpkg pins history Ulf asked why Ladybird doesn't just use a git submodule for Build/vcpkg. It works mechanically -- a submodule's .git is a gitfile (gitdir: ../../.git/modules/...), and vcpkg's `git --git-dir .git read-tree ` follows the indirection: verified READ-TREE OK. So 'submodules break vcpkg' is not the reason. The reason is that a submodule pins the wrong thing. It pins one commit and gives you its checkout; vcpkg's manifest pins a baseline and then reads history BEHIND it. 14 of vcpkg.json's 45 overrides name a version that is not what ports/ holds at the baseline: ffmpeg 7.1.1#5 vs 8.1.2#3, harfbuzz 10.2.0 vs 14.2.1#2, mimalloc 2.2.7 vs 3.4.3, qtbase, freetype, zlib, dbus, fontconfig, libedit, libwebp, libtommath, cpptrace, angle -- and simdutf pins NEWER than the baseline holds. Concretely ffmpeg 7.1.1#5's port is git-tree 0988005f while HEAD:ports/ffmpeg is c40aaa40, so the bytes vcpkg builds exist at no single commit. That is also why --depth 1 fails. Secondary costs, all measured: the same 119MB lands in .git/modules (no saving); Build/vcpkg is inside a Build*/-ignored path that vcpkg fills with ~3GB of downloads/installed/buildtrees, so the submodule reads dirty forever without ignore = dirty; and a submodule still does not produce the vcpkg binary, which is not tracked but bootstrapped from a tag+SHA512 in scripts/vcpkg-tool-metadata.txt. Generalizes: Build/vcpkg is not a vendored dependency, it is a package manager's cache directory that happens to be a git checkout. Every instinct that treats it as a pinned copy of another repo -- submodule, git_repository, http_archive of a tarball, --depth 1 -- pins the checkout and loses the history, and the history is the dependency. --- docs/CASE-ladybird-migration.md | 52 +++++++++++++++++++++++++++++++++ examples/ladybird/README.md | 15 +++++++++- 2 files changed, 66 insertions(+), 1 deletion(-) diff --git a/docs/CASE-ladybird-migration.md b/docs/CASE-ladybird-migration.md index 392b5b7..56c6dc4 100644 --- a/docs/CASE-ladybird-migration.md +++ b/docs/CASE-ladybird-migration.md @@ -1897,6 +1897,58 @@ the same manifest, resolves all **78 ports to exactly the versions this dev chec resolves** (`diff`, 0 differences). The checkout carries no local state beyond the ref — which is precisely why a prefetch step is sufficient and a rule is not needed. +### Why not a git submodule? + +The obvious question, since a submodule is git's own answer to "vendor another repo at +a pinned commit" and it would give the cloner `Build/vcpkg` with a working `.git` from +`git clone --recurse-submodules`. It **does** work mechanically — I checked, because +the `.git` here is load-bearing and a submodule's `.git` is not a directory but a +*gitfile* (`gitdir: ../../.git/modules/Build/vcpkg`). vcpkg's +`git --git-dir .git read-tree ` follows that indirection fine: `READ-TREE OK`. +So "submodules break vcpkg" is not the reason. + +The reason is that **a submodule pins the wrong thing.** A submodule pins one commit +and gives you its *checkout*; vcpkg's manifest pins a baseline commit and then reads +**history behind it**. Ladybird's `vcpkg.json` carries 45 `overrides`, and **14 of +them name a version that is not what `ports/` contains at the baseline**: + +| pinned in `vcpkg.json` | what `ports/` holds at the baseline | +|---|---| +| ffmpeg 7.1.1#5 | 8.1.2#3 | +| harfbuzz 10.2.0 | 14.2.1#2 | +| mimalloc 2.2.7 | 3.4.3 | +| qtbase 6.10.0#1 | 6.11.1#1 | +| freetype 2.13.3 | 2.14.3 | +| simdutf 9.0.0 | 8.2.0 *(older than the pin)* | +| …plus zlib, dbus, fontconfig, libedit, libwebp, libtommath, cpptrace, angle | | + +Concretely: ffmpeg 7.1.1#5's port is git-tree `0988005f…`, while +`HEAD:ports/ffmpeg` at the baseline is `c40aaa40…`. The bytes vcpkg builds are +**not in the working tree at any single commit** — they are extracted from the object +database by `read-tree`, per port, per pinned version. That is what the version +database *is*, and it is why `--depth 1` fails with vcpkg's own `Try again with a full +vcpkg clone`: shallow gives you the tree, and the tree is not the pin. + +So a submodule would deliver exactly the state that is *insufficient* — the baseline +checkout — while still requiring the full history behind it to be present, and it +would add costs of its own: the same 119 MB in `.git/modules` (no saving), plus +`Build/vcpkg` is inside a `Build*/`-ignored path that vcpkg fills with ~3 GB of +`downloads/`, `installed/`, `buildtrees/` scratch, so every cloner's `git status` +would show the submodule dirty forever (needing `ignore = dirty` in `.gitmodules` to +paper over it), and `git submodule update` would fight vcpkg for who owns the +directory. And a submodule still would not produce the `vcpkg` **binary** — that is +not tracked in the repo; it is bootstrapped from a tag+SHA512 pinned in +`scripts/vcpkg-tool-metadata.txt`. `Meta/ladybird.py vcpkg` does that too. + +The generalizable point, and the reason this is worth a section rather than a +footnote: **`Build/vcpkg` is not a vendored dependency, it is a package manager's +cache directory that happens to be a git checkout.** Every instinct that treats it as +"a pinned copy of another repo" — submodule, `git_repository`, `http_archive` of a +tarball, `--depth 1` — pins the checkout and loses the history, and the history is the +dependency. Getting this right is what `builtin-baseline` + `overrides` means, and it +is why the answer to "how do we get it" keeps coming back to *run the project's own +bootstrap*. + ## Plan for the rest - **Ring 1c — BUILD generation to compile+link parity, bottom-up.** AK → diff --git a/examples/ladybird/README.md b/examples/ladybird/README.md index bb6e489..defb0e5 100644 --- a/examples/ladybird/README.md +++ b/examples/ladybird/README.md @@ -78,7 +78,7 @@ Ladybird itself. (An earlier draft of this section proposed a custom | 6: the four `git archive` tarballs | **~8 lines of shell**: clone each pinned URL, `git archive `. Verified — the reproduced `libyuv` tarball's SHA512 matches `vcpkg_git_archives.bzl` exactly, so the committed hashes become *checked* rather than decorative | | 3: the HSTS table | **An upstream change to Ladybird.** `Meta/CMake/hsts_preload.cmake` fetches Chromium's `main` (unversioned) at configure time; pin *that* to a revision, then `http_file` the same revision so both build systems consume one pinned input. Pinning only on the Bazel side "works" but the pinned file (18.7 MB) differs from the one CMake fetched (10.5 MB) and the generated table differs — trading a hermeticity gap for a **parity** gap | -Two shortcuts tested and rejected, because both look like simplifications: +Three shortcuts tested and rejected, all of which look like simplifications: - **`--depth 1` on the vcpkg clone.** 8.7 MB instead of 121 MB, and `read-tree` even succeeds for some ports — then resolution fails on ffmpeg and harfbuzz with @@ -89,6 +89,19 @@ Two shortcuts tested and rejected, because both look like simplifications: (ffmpeg 7.1.1#5 → 8.1.2#3, harfbuzz 10.2.0 → 14.2.1#2, mimalloc 2.2.7 → 3.4.3, plus zlib, freetype, dbus, fontconfig, libedit, libwebp, cpptrace). A fix for a hermeticity blocker that changes ten dependency versions is not a fix. +- **A git submodule** — git's own answer to this, and it *does* work mechanically + (a submodule's `.git` is a gitfile, and vcpkg's `read-tree` follows the + indirection: verified). It fails for a better reason: **a submodule pins a + checkout, and vcpkg pins history behind a baseline.** 14 of `vcpkg.json`'s 45 + `overrides` name a version that is *not* what `ports/` holds at the baseline + (ffmpeg's pinned port is git-tree `0988005f…`; `HEAD:ports/ffmpeg` is + `c40aaa40…`), so the bytes vcpkg builds exist at no single commit. A submodule + would deliver precisely the insufficient state, still need the full 119 MB of + history in `.git/modules`, sit in a `Build*/`-ignored directory vcpkg fills with + ~3 GB of scratch (permanently-dirty submodule), and still not produce the `vcpkg` + binary, which is bootstrapped from a tag+SHA512 in `scripts/vcpkg-tool-metadata.txt`. + **`Build/vcpkg` is not a vendored dependency; it is a package manager's cache that + happens to be a git checkout.** Finding 36. A fresh full clone at the baseline resolves all **78 ports to exactly the versions this dev checkout resolves** (`diff`, 0 differences) — so the checkout carries no From 631d045d57c35285cc18ed754e0e7c89b448e7ad Mon Sep 17 00:00:00 2001 From: Ulf Adams Date: Mon, 10 Aug 2026 16:37:06 +0000 Subject: [PATCH 13/59] Ladybird: a developer gets all three inputs from one ordinary build -- they stage nothing Ulf asked how a Ladybird developer actually gets the correct stuff on disk. The answer dissolves two of the three remaining blockers: ./Meta/ladybird.py build produces all three inputs as side effects. (a) build calls build_vcpkg() itself -- not only the `vcpkg` subcommand -- so the checkout and its .git appear (~70s, verified from an empty directory; `vcpkg` alone gives ONLY this, no downloads/, no HSTS table). (b) The CMake configure downloads the HSTS table via Meta/CMake/hsts_preload.cmake, gated on ENABLE_NETWORK_DOWNLOADS, default ON. (c) vcpkg writes the four git-archive tarballs into Build/vcpkg/downloads/ while building skia and angle -- and their SHA512s equal the committed values in vcpkg_git_archives.bzl, so the directory I 'made by hand months ago' was a copy of vcpkg's own download cache. I had not built a pin, I had copied one and forgotten which. So for a Ladybird developer this is a non-problem; it is a problem only for a Bazel-only clone that never runs CMake. Those two sentences carry different debts: two blockers need no code (a recipe that says 'build once with CMake first', plus a clear error when it hasn't), and only the unpinned HSTS fetch is a genuine hermeticity defect needing an upstream change. My three successive answers to 'what is needed' were a repository_rule, then a prefetch script, then a cp. Each was smaller because each time I read further into what the project already does. --- docs/CASE-ladybird-migration.md | 24 ++++++++++ examples/ladybird/README.md | 81 ++++++++++++++++++++++----------- 2 files changed, 78 insertions(+), 27 deletions(-) diff --git a/docs/CASE-ladybird-migration.md b/docs/CASE-ladybird-migration.md index 56c6dc4..6e95450 100644 --- a/docs/CASE-ladybird-migration.md +++ b/docs/CASE-ladybird-migration.md @@ -1860,6 +1860,30 @@ that way and its SHA512 matched `vcpkg_git_archives.bzl`'s committed value **exa the reproduction is verifiable in one line rather than trusted. Eight lines of shell, against a `repository_rule` I would have had to design, test and maintain. +**Then Ulf asked the question that dissolved even the eight lines: "how does a +Ladybird developer get the correct stuff on disk?"** The answer is that they do +nothing, because **one ordinary `./Meta/ladybird.py build` produces all three +inputs.** `build` calls `build_vcpkg()` itself (not just the `vcpkg` subcommand), so +the checkout and its `.git` appear; the CMake *configure* downloads the HSTS table via +`hsts_preload.cmake`, gated on `ENABLE_NETWORK_DOWNLOADS`, **default ON**; and vcpkg +writes the four `git archive` tarballs into `Build/vcpkg/downloads/` while building +skia and angle. I checked that last one against my own tree: the SHA512s of +`Build/vcpkg/downloads/{angle,libyuv,skia}-*.tar.gz` **equal the committed values** in +`vcpkg_git_archives.bzl`. So "the directory I made by hand months ago" was a copy of +vcpkg's own download cache — I had not built a pin, I had copied one, and then +forgotten which. + +That reframes the whole finding-36 table. **For a Ladybird developer none of this is +a problem; it is a problem only for a Bazel-only clone that never runs CMake.** Both +sentences are worth saying because they carry different debts: the first means two of +the three "blockers" need *no code*, only a recipe that says "build once with CMake +first" and a clear error when you have not; the second means the third one — the +unpinned HSTS fetch — is the only genuine hermeticity defect in the set. My three +successive answers to "what is needed" were a `repository_rule`, then a prefetch +script, then a `cp`. Each was smaller than the last because each time I looked +further into what the project already does, and **the useful move was always reading +the project rather than designing for it.** + **The HSTS table (row 3) is the one that genuinely cannot be fixed here, and it is not a Bazel problem.** `Meta/CMake/hsts_preload.cmake` fetches `raw.githubusercontent.com/chromium/chromium/**main**/net/http/transport_security_state_static.json` diff --git a/examples/ladybird/README.md b/examples/ladybird/README.md index defb0e5..75870fb 100644 --- a/examples/ladybird/README.md +++ b/examples/ladybird/README.md @@ -27,7 +27,10 @@ the closure of all six returns **0** targets under `Build/full` (down from 741). `Build/full` is still needed to *regenerate* the BUILD files and to run the parity harness — a converter-development dependency, not a build dependency. -**A fresh clone now builds and renders — with three inputs staged by hand.** The +**A fresh clone now builds and renders — after one ordinary CMake build has run +once.** (Three inputs come from that build rather than from Bazel; a Ladybird +developer therefore stages nothing, and the gap is real only for a Bazel-only clone +— see [how a Ladybird developer gets them](#how-a-ladybird-developer-gets-these-three-inputs).) The end-to-end test (clone into a new directory, drop in the overlay, nothing else) was run for the first time and failed **six separate times**; `Build/full` was the dependency I had removed, and **five of the six had nothing to do with it**. @@ -66,17 +69,36 @@ dependency I went looking for, so it is the one I found.** The check that would have caught all five is not a better `cquery` — it is doing the clone. See [Known gaps](#known-gaps). -### What closing the last three takes +### How a Ladybird developer gets these three inputs -Two are a **prefetch step**, not new Bazel machinery, and one needs a change to -Ladybird itself. (An earlier draft of this section proposed a custom -`repository_rule` for the first two; that was wrong, and why is in finding 36.) +**They don't do anything: one ordinary Ladybird build produces all three.** Each is +a side effect of the CMake+vcpkg path, which is exactly why all three were invisible +here for months. (Two earlier drafts of this section proposed writing machinery — a +custom `repository_rule`, then a prefetch script. Both were wrong, and why is in +finding 36.) -| Blocker | What closes it | +```sh +./Meta/ladybird.py build # the normal Ladybird build +cp Build/vcpkg/downloads/{angle,libyuv,skia}-*.tar.gz Meta/CMake/vcpkg/git-archives/ +``` + +| Input | Who produces it, in a normal build | +|---|---| +| the vcpkg checkout + its `.git` | **`Meta/Utils/build_vcpkg.py`**, called by `ladybird.py` *`build`* as well as `vcpkg`: clones microsoft/vcpkg, checks out `vcpkg.json`'s `builtin-baseline`, bootstraps the tool at the tag+SHA512 in `scripts/vcpkg-tool-metadata.txt`. **~70 s, verified from an empty directory** | +| `Build/caches/HSTSPreload/transport_security_state_static.json` | **the CMake configure**, via `Meta/CMake/hsts_preload.cmake` → `download_file`, gated on `ENABLE_NETWORK_DOWNLOADS` (**default ON**) | +| the four `git archive` tarballs | **vcpkg itself**, while building skia and angle: `vcpkg_from_git` writes them to `Build/vcpkg/downloads/`. Verified — those files' SHA512s equal the committed values in `vcpkg_git_archives.bzl`, so the "directory I made by hand" was a copy of vcpkg's own cache. Only the `cp` above is manual, and only because the overlay looks in a different directory | + +So for a Ladybird developer this is a **non-problem**; the gap is real only for +someone who wants Bazel *without* ever running CMake. That distinction is worth +keeping straight, because it changes what is owed: two of the three need **no new +code at all** (they need the recipe to say "build once with CMake first", and a +clear error when you haven't), and only the third is a genuine hermeticity defect: + +| Blocker | What a Bazel-only clone still needs | |---|---| -| 1+2: the vcpkg checkout and its `.git` | **`python3 Meta/ladybird.py vcpkg`** — Ladybird already ships it. It clones microsoft/vcpkg, checks out `vcpkg.json`'s `builtin-baseline` and bootstraps the tool: **75 s, verified from an empty directory**, producing the 121 MB `.git` that `git read-tree` needs. Nothing to write; this is a documentation and ordering problem | -| 6: the four `git archive` tarballs | **~8 lines of shell**: clone each pinned URL, `git archive `. Verified — the reproduced `libyuv` tarball's SHA512 matches `vcpkg_git_archives.bzl` exactly, so the committed hashes become *checked* rather than decorative | -| 3: the HSTS table | **An upstream change to Ladybird.** `Meta/CMake/hsts_preload.cmake` fetches Chromium's `main` (unversioned) at configure time; pin *that* to a revision, then `http_file` the same revision so both build systems consume one pinned input. Pinning only on the Bazel side "works" but the pinned file (18.7 MB) differs from the one CMake fetched (10.5 MB) and the generated table differs — trading a hermeticity gap for a **parity** gap | +| the vcpkg checkout | Nothing to write — `Meta/ladybird.py vcpkg` is the supported way to get it, and it is what Ladybird's own build calls. Documentation and ordering | +| the four tarballs | Nothing to write — they fall out of the vcpkg build. To get them *without* CMake: clone each pinned URL and `git archive `; verified byte-identical to the committed hashes (`libyuv`) | +| the HSTS table | **An upstream change to Ladybird.** `hsts_preload.cmake` fetches Chromium's `main` — unversioned — so pin *that* to a revision, then `http_file` the same revision and both build systems consume one pinned input. Pinning only on the Bazel side "works" but the pinned file (18.7 MB) differs from the one CMake fetched (10.5 MB) and the generated table differs, trading a hermeticity gap for a **parity** gap | Three shortcuts tested and rejected, all of which look like simplifications: @@ -166,24 +188,29 @@ git clone https://github.com/LadybirdBrowser/ladybird && cd ladybird cp -r .../examples/ladybird/workspace/. . && mv bazelrc.txt .bazelrc git apply .../examples/ladybird/patches/*.patch # generator determinism + a Qt header # that is not self-contained (gap 9) -# 1a. THE THREE INPUTS THE OVERLAY DOES NOT CARRY. Without these the build fails; -# see the table above for how each one stayed invisible on the dev machine. -# (a) the vcpkg checkout at vcpkg.json's builtin-baseline, WITH its .git -- -# vcpkg resolves versioned ports via `git read-tree`. Ladybird's own -# helper does exactly this: -python3 Meta/ladybird.py vcpkg -# (b) the HSTS preload table: an unversioned download from Chromium's main -# branch that CMake does at configure time. No Bazel rule produces it. -mkdir -p Build/caches/HSTSPreload && curl -Lo Build/caches/HSTSPreload/transport_security_state_static.json \ - https://raw.githubusercontent.com/chromium/chromium/main/net/http/transport_security_state_static.json -# (c) the four vcpkg_from_git archives. These are `git archive` output, so -# there is no URL to http_file; the hashes ARE pinned -# (vcpkg_git_archives.bzl), so reproducing them is a prefetch, not a rule -# -- clone each pinned URL and `git archive` the pinned ref. Verified: -# the reproduced libyuv tarball's SHA512 matches the committed one. -# Absent, the build now fails in 4s naming them (gap 7). -# NB `ply` no longer needs anything: the wheel is pinned and pip runs with -# PIP_NO_INDEX (gap 8). +# 1a. THE THREE INPUTS THE OVERLAY DOES NOT CARRY -- all three are produced by +# ONE ordinary Ladybird build. A Ladybird developer stages nothing by hand; +# these are blockers for a Bazel-ONLY clone, and each one is a thing CMake or +# vcpkg produces as a side effect (see finding 36): +# (a) Build/vcpkg + its .git <- python3 Meta/ladybird.py vcpkg (~70s) +# (b) Build/caches/HSTSPreload/transport_security_state_static.json +# <- the CMake CONFIGURE downloads it +# (Meta/CMake/hsts_preload.cmake, gated on +# ENABLE_NETWORK_DOWNLOADS, default ON) +# (c) the four Meta/CMake/vcpkg/git-archives/*.tar.gz +# <- vcpkg writes them into +# Build/vcpkg/downloads/ while building +# skia and angle. Verified: those files' +# SHA512s equal the committed ones in +# vcpkg_git_archives.bzl, i.e. the +# "hand-made" directory was a copy of +# vcpkg's own cache. +# So the one-command answer, for anyone who has ever built Ladybird normally: +./Meta/ladybird.py build # CMake+vcpkg; produces (a), (b) and (c) +cp Build/vcpkg/downloads/{angle,libyuv,skia}-*.tar.gz Meta/CMake/vcpkg/git-archives/ +# `ladybird.py vcpkg` alone gives you (a) only -- verified from an empty +# directory: no downloads/, no HSTS table. And `ply` needs nothing now: the +# wheel is pinned and pip runs with PIP_NO_INDEX (gap 8). # # NB vcpkg's buildtrees peak around 3 GB. They go next to the declared output # (inside bazel-out) rather than $TMPDIR, so a small /tmp tmpfs is not a From 7243f31329118f33a9a1f4faab9b4e9d06f79e13 Mon Sep 17 00:00:00 2001 From: Ulf Adams Date: Mon, 10 Aug 2026 17:06:53 +0000 Subject: [PATCH 14/59] Ladybird: get the vcpkg_from_git archives without CMake -- pin-driven, git-reproduced, verified Ulf asked three times for the no-CMake answer; the first two replies described the CMake path. This closes two of the three staged inputs for a Bazel-only clone: `Meta/ladybird.py vcpkg` is a standalone subcommand (~70s, no configure), and Meta/fetch_vcpkg_git_archives.py now produces the four vcpkg_from_git tarballs -- 4/4 reproduced from scratch and byte-identical to the committed SHA512s. The shape matters because I got it wrong twice. A static parse of the portfiles is unsound in both directions at once: my first version found 8 archives for skia where 4 are real, and missed libyuv entirely. declare_external_from_git only DECLARES; get_externals(${required_externals}) picks from it under feature/platform if()s, so the set is decided by CMake evaluation rather than by the text -- and libyuv's archive comes from its own port calling vcpkg_from_git directly. Finding 30's lesson recurring: portfiles are programs, do not re-derive what they compute. So the list comes from the committed pin, and regenerating the pin uses vcpkg as the instrument: Meta/vcpkg_capture_git_archives.sh runs `vcpkg install --only-downloads` (~6 min, no compilation, no CMake). fetch_vcpkg_git_archives.py then reproduces each pinned tarball with git clone + git -c core.autocrlf=false archive -- byte-for-byte what vcpkg_from_git.cmake runs -- and fails on any mismatch, so the hashes are checked rather than trusted. Static resolution survives only where sound: name -> clone URL. Recorded asymmetry: --only-downloads yields 3 of 4, because angle's zlib is fetched in angle's BUILD phase. 9 tests (151/151 total) cover the two decisions the script makes alone: the pin is the authority (regression test for the 8-vs-4 bug), and every pinned name must resolve or it fails loudly. Remaining defect is the HSTS table alone -- Chromium's unversioned main, so nothing to pin to; noted that CMake's download_file is a no-op when the file exists, so the upstream fix is additive. --- docs/CASE-ladybird-migration.md | 66 ++++- examples/ladybird/README.md | 102 +++++--- .../Meta/fetch_vcpkg_git_archives.py | 225 ++++++++++++++++++ .../Meta/vcpkg_capture_git_archives.sh | 96 ++++++++ tests/test_fetch_vcpkg_git_archives.py | 212 +++++++++++++++++ 5 files changed, 657 insertions(+), 44 deletions(-) create mode 100644 examples/ladybird/workspace/Meta/fetch_vcpkg_git_archives.py create mode 100755 examples/ladybird/workspace/Meta/vcpkg_capture_git_archives.sh create mode 100644 tests/test_fetch_vcpkg_git_archives.py diff --git a/docs/CASE-ladybird-migration.md b/docs/CASE-ladybird-migration.md index 6e95450..894c74a 100644 --- a/docs/CASE-ladybird-migration.md +++ b/docs/CASE-ladybird-migration.md @@ -1873,16 +1873,62 @@ skia and angle. I checked that last one against my own tree: the SHA512s of vcpkg's own download cache — I had not built a pin, I had copied one, and then forgotten which. -That reframes the whole finding-36 table. **For a Ladybird developer none of this is -a problem; it is a problem only for a Bazel-only clone that never runs CMake.** Both -sentences are worth saying because they carry different debts: the first means two of -the three "blockers" need *no code*, only a recipe that says "build once with CMake -first" and a clear error when you have not; the second means the third one — the -unpinned HSTS fetch — is the only genuine hermeticity defect in the set. My three -successive answers to "what is needed" were a `repository_rule`, then a prefetch -script, then a `cp`. Each was smaller than the last because each time I looked -further into what the project already does, and **the useful move was always reading -the project rather than designing for it.** +That reframes the finding-36 table — but it also let me answer the wrong question. +"Run the normal build first" is fine for a Ladybird developer and **useless for the +case the whole exercise is about**: Bazel without CMake. Ulf had to ask a third time +before I built it. + +**Without CMake, two of the three are now closed, and the third is a one-line +upstream fix.** + +`Meta/ladybird.py vcpkg` is a standalone subcommand — no configure, no CMake — so the +checkout and its `.git` cost ~70 s. The four `vcpkg_from_git` tarballs are what +needed building, and the shape of the solution is the interesting part, because I got +it wrong twice on the way: + +1. **A static parse of the portfiles is unsound, and wrong in both directions at + once.** My first script scanned skia's and angle's portfiles for + `declare_external_from_git` / `checkout_in_path` and produced **8** archives for + skia where 4 are real, while **missing libyuv entirely**. Both errors have one + cause: `declare_external_from_git` only *declares*, and + `get_externals(${required_externals})` picks from that under feature and platform + `if()`s — the set is decided by CMake evaluation, not by the text — while libyuv's + archive comes from the libyuv *port* calling `vcpkg_from_git` directly, which a + scan of skia+angle cannot see. This is finding 30's lesson recurring: **portfiles + are programs, so do not re-derive what they compute.** +2. **So take the list from the pin and use vcpkg as the instrument for regenerating + it.** `vcpkg install --only-downloads` runs the portfiles' fetch phase and stops: + ~6 minutes, no compilation, no CMake, and vcpkg_from_git produces its tarballs at + the refs the real resolution picks. Same tactic as the 76-distfile asset capture + — instrument the foreign build system rather than predicting it. +3. **`Meta/fetch_vcpkg_git_archives.py` then reproduces each pinned tarball with + `git clone` + `git -c core.autocrlf=false archive `** (byte-for-byte what + `vcpkg_from_git.cmake` runs internally) **and verifies it against the committed + SHA512.** Result: **4/4 reproduced from scratch, byte-identical to the pin.** The + hashes came from vcpkg; git reproducing them is the proof the two agree, so the + pin is checked rather than trusted. Static resolution survives only where it is + sound: mapping an already-known archive *name* to a clone URL. + +One asymmetry is recorded rather than smoothed over: `--only-downloads` yields **3 of +the 4**, because angle's zlib is fetched from angle's *build* phase via +`checkout_in_path`, not its fetch phase. The script says so; "--only-downloads gets +them all" would have been the comfortable, false version. + +That leaves **exactly one** genuine hermeticity defect: the HSTS table, fetched from +Chromium's unversioned `main`, where there is no revision to pin to. A useful detail +found while checking whether the fix is invasive: CMake's `download_file` is a no-op +when the file already exists (verified with `ENABLE_NETWORK_DOWNLOADS=OFF`), so a +pinned file staged by Bazel is consumed by CMake unchanged — **the upstream change is +additive**, one URL gaining a revision, no restructuring. + +My four successive answers to "what is needed" were: a `repository_rule`, a prefetch +script, a `cp`, and finally a script plus a capture instrument. The middle two were +smaller because I kept reading further into what the project already does — but the +last one is *bigger* than the `cp`, and that is the actual lesson. **"It falls out of +the existing build" was a true sentence that dissolved the question instead of +answering it.** For the audience that has CMake it is the right answer; for the +audience this migration exists to serve it is a non-answer, and I gave it because it +let me stop working. **The HSTS table (row 3) is the one that genuinely cannot be fixed here, and it is not a Bazel problem.** `Meta/CMake/hsts_preload.cmake` fetches diff --git a/examples/ladybird/README.md b/examples/ladybird/README.md index 75870fb..06c0723 100644 --- a/examples/ladybird/README.md +++ b/examples/ladybird/README.md @@ -69,36 +69,61 @@ dependency I went looking for, so it is the one I found.** The check that would have caught all five is not a better `cquery` — it is doing the clone. See [Known gaps](#known-gaps). -### How a Ladybird developer gets these three inputs +### Getting the three inputs — with and without CMake -**They don't do anything: one ordinary Ladybird build produces all three.** Each is -a side effect of the CMake+vcpkg path, which is exactly why all three were invisible -here for months. (Two earlier drafts of this section proposed writing machinery — a -custom `repository_rule`, then a prefetch script. Both were wrong, and why is in -finding 36.) +Two audiences, two answers. **A Ladybird developer stages nothing:** one ordinary +`./Meta/ladybird.py build` produces all three as side effects, which is exactly why +they stayed invisible here for months. -```sh -./Meta/ladybird.py build # the normal Ladybird build -cp Build/vcpkg/downloads/{angle,libyuv,skia}-*.tar.gz Meta/CMake/vcpkg/git-archives/ -``` - -| Input | Who produces it, in a normal build | +| Input | Who produces it in a normal build | |---|---| -| the vcpkg checkout + its `.git` | **`Meta/Utils/build_vcpkg.py`**, called by `ladybird.py` *`build`* as well as `vcpkg`: clones microsoft/vcpkg, checks out `vcpkg.json`'s `builtin-baseline`, bootstraps the tool at the tag+SHA512 in `scripts/vcpkg-tool-metadata.txt`. **~70 s, verified from an empty directory** | -| `Build/caches/HSTSPreload/transport_security_state_static.json` | **the CMake configure**, via `Meta/CMake/hsts_preload.cmake` → `download_file`, gated on `ENABLE_NETWORK_DOWNLOADS` (**default ON**) | -| the four `git archive` tarballs | **vcpkg itself**, while building skia and angle: `vcpkg_from_git` writes them to `Build/vcpkg/downloads/`. Verified — those files' SHA512s equal the committed values in `vcpkg_git_archives.bzl`, so the "directory I made by hand" was a copy of vcpkg's own cache. Only the `cp` above is manual, and only because the overlay looks in a different directory | +| the vcpkg checkout + its `.git` | `Meta/Utils/build_vcpkg.py`, called by `ladybird.py` **`build`** as well as `vcpkg`: clone, checkout `builtin-baseline`, bootstrap the tool at the tag+SHA512 in `scripts/vcpkg-tool-metadata.txt`. ~70 s | +| `Build/caches/HSTSPreload/transport_security_state_static.json` | the CMake **configure**, via `hsts_preload.cmake` → `download_file`, gated on `ENABLE_NETWORK_DOWNLOADS` (default ON) | +| the four `git archive` tarballs | **vcpkg itself**, while building skia and angle | -So for a Ladybird developer this is a **non-problem**; the gap is real only for -someone who wants Bazel *without* ever running CMake. That distinction is worth -keeping straight, because it changes what is owed: two of the three need **no new -code at all** (they need the recipe to say "build once with CMake first", and a -clear error when you haven't), and only the third is a genuine hermeticity defect: +**Without ever running CMake — the actual answer, and it needs no CMake at all:** -| Blocker | What a Bazel-only clone still needs | -|---|---| -| the vcpkg checkout | Nothing to write — `Meta/ladybird.py vcpkg` is the supported way to get it, and it is what Ladybird's own build calls. Documentation and ordering | -| the four tarballs | Nothing to write — they fall out of the vcpkg build. To get them *without* CMake: clone each pinned URL and `git archive `; verified byte-identical to the committed hashes (`libyuv`) | -| the HSTS table | **An upstream change to Ladybird.** `hsts_preload.cmake` fetches Chromium's `main` — unversioned — so pin *that* to a revision, then `http_file` the same revision and both build systems consume one pinned input. Pinning only on the Bazel side "works" but the pinned file (18.7 MB) differs from the one CMake fetched (10.5 MB) and the generated table differs, trading a hermeticity gap for a **parity** gap | +```sh +python3 Meta/ladybird.py vcpkg # 1. the checkout + .git (~70s). No CMake: + # `vcpkg` is a standalone subcommand. +python3 Meta/fetch_vcpkg_git_archives.py # 2. the four git archives (~80s), each + # reproduced with git clone + git archive + # and VERIFIED against the committed SHA512 +curl -Lo Build/caches/HSTSPreload/transport_security_state_static.json \ + "$HSTS_URL" # 3. the HSTS table (see caveat below) +bazel build //:ladybird //:WebContent //:RequestServer //:ImageDecoder \ + //:Compositor //:WebWorker +``` + +Step 2 is `Meta/fetch_vcpkg_git_archives.py`, added here. Verified: **4/4 reproduced +from scratch and byte-identical to the pinned SHA512s.** Two things about it are worth +knowing, because both were mistakes I made first: + +- **It takes the *list* from the committed pin, never from parsing portfiles.** A + first version derived the list by scanning skia's and angle's portfiles and was + wrong in both directions: 8 archives for skia where 4 are real, and libyuv missed + entirely. `declare_external_from_git` only *declares*; feature- and + platform-conditional `get_externals(${required_externals})` decides what is + actually fetched, and libyuv's comes from its own port. **Statically deciding the + set is unsound; statically resolving a name to a URL is fine**, and that is all the + script does. +- **Regenerating the pin uses vcpkg as the instrument**, not a parser: + `Meta/vcpkg_capture_git_archives.sh` runs `vcpkg install --only-downloads`, which + executes the portfiles' fetch phase and stops — ~6 min, no compilation, no CMake. + It produces **3 of the 4**: angle's zlib is fetched from angle's *build* phase, so + it never appears in a downloads-only run. That asymmetry is recorded in the script + rather than smoothed over. + +Step 3 is the **one genuine hermeticity defect** and it cannot be closed here. +`hsts_preload.cmake` fetches Chromium's **`main`** — unversioned — so there is no +revision to pin to. Pinning a revision on the Bazel side alone would make Bazel's +output diverge from CMake's (the file at tag `139.0.7258.5` is 18.7 MB against the +10.5 MB `main` served when this machine configured, and the generated table differs), +trading a hermeticity gap for a **parity** gap. The fix is one line upstream: pin the +URL to a revision in `hsts_preload.cmake`, then `http_file` the same revision, and +both build systems consume one pinned input. Usefully, `download_file` is already a +no-op when the file exists (verified with `ENABLE_NETWORK_DOWNLOADS=OFF`), so a pinned +file staged by Bazel is consumed by CMake unchanged — the upstream change is additive. Three shortcuts tested and rejected, all of which look like simplifications: @@ -152,7 +177,9 @@ an FFI header collision and an entire missing Rust target. See | `Meta/vcpkg_assets.tsv` | **The dependency pin.** 76 `(url, sha512, dst)` rows captured from vcpkg itself via its `x-script` asset hook. Everything below is generated from this, with no vcpkg, no CMake and no network | | `vcpkg_distfiles.bzl`, `vcpkg_index.bzl`, `vcpkg_extension.bzl`, `vcpkg_git_archives.bzl` | One `http_file` per distfile, the sha512→label index the asset script resolves through, the module extension that creates the repos, and the 4 `vcpkg_from_git` archives. **Generated** by `Meta/emit_vcpkg_bazel.py` | | `vcpkg.bzl`, `Meta/vcpkg_build.sh` | `vcpkg_tree`: builds the whole dep tree as an ordinary Bazel action with `x-block-origin`, so it reaches the network zero times. Deliberately not a `repository_rule` | -| `Meta/vcpkg_capture_assets.sh` | Records the pin, by *being* vcpkg's asset cache. The one run allowed to fetch | +| `Meta/vcpkg_capture_assets.sh` | Records the 76-distfile pin, by *being* vcpkg's asset cache. The one run allowed to fetch | +| `Meta/fetch_vcpkg_git_archives.py` | Produces the 4 `vcpkg_from_git` tarballs **without CMake**: takes the list from the committed pin, resolves each clone URL out of the portfiles, then `git clone` + `git -c core.autocrlf=false archive ` and **verifies against the pinned SHA512**. 4/4 byte-identical | +| `Meta/vcpkg_capture_git_archives.sh` | Regenerates that pin with `vcpkg install --only-downloads` (~6 min, no compilation, no CMake) — vcpkg as the instrument, since which git externals are used is decided by feature-conditional CMake code, not by portfile text | | `bazelrc.txt` | → `.bazelrc`. Global copts/defines/linkopts mirrored from `Meta/CMake/compile_options.cmake` | | `BUILD.bazel` | Root package: 34 libraries, the 5 executables, Qt moc/rcc genrules. **Generated** by `Meta/emit_build_bazel.py` | | `codegen_root.bzl` | Non-LibWeb generator genrules (IPC endpoints, LibJS Bytecode/Op, HSTS table, WebGL replayer, TIFF tag tables, the two SPIR-V shader headers, and the chained `generate_interpreter_layout` → `flapc` interpreter assembly). **Generated** by `Meta/emit_root_codegen_bazel.py` | @@ -179,8 +206,9 @@ adds `-I/usr/include/libdrm`), which is gap 3 below, not an emitter bug. To **build the browser**, no CMake build is needed — Bazel fetches and builds the vcpkg *ports* and the Rust crates itself, and vcpkg's own downloader reaches the -network zero times. Three inputs still have to be present first (step 1a; gaps 7–8), -and one port bypasses the pin via `pip`, so "no network at all" is not yet true: +network zero times. Three inputs still have to be present first (step 1a; gaps 7–8) +— two of them obtainable **without CMake**, the third (the HSTS table) unpinnable +until fixed upstream, so "no network at all" is not yet true: ```sh git clone https://github.com/LadybirdBrowser/ladybird && cd ladybird @@ -205,12 +233,18 @@ git apply .../examples/ladybird/patches/*.patch # generator determinism + a Qt # vcpkg_git_archives.bzl, i.e. the # "hand-made" directory was a copy of # vcpkg's own cache. -# So the one-command answer, for anyone who has ever built Ladybird normally: -./Meta/ladybird.py build # CMake+vcpkg; produces (a), (b) and (c) -cp Build/vcpkg/downloads/{angle,libyuv,skia}-*.tar.gz Meta/CMake/vcpkg/git-archives/ -# `ladybird.py vcpkg` alone gives you (a) only -- verified from an empty -# directory: no downloads/, no HSTS table. And `ply` needs nothing now: the -# wheel is pinned and pip runs with PIP_NO_INDEX (gap 8). +# WITHOUT CMAKE (the supported path for a Bazel-only clone): +python3 Meta/ladybird.py vcpkg # (a) checkout + .git, ~70s, no CMake +python3 Meta/fetch_vcpkg_git_archives.py # (c) 4/4 reproduced with git archive and + # verified against the pinned SHA512s +curl -Lo Build/caches/HSTSPreload/transport_security_state_static.json \ + https://raw.githubusercontent.com/chromium/chromium/main/net/http/transport_security_state_static.json +# # (b) unpinnable until fixed upstream +# WITH CMake, if you were building Ladybird anyway, all three fall out of: +# ./Meta/ladybird.py build +# cp Build/vcpkg/downloads/{angle,libyuv,skia}-*.tar.gz Meta/CMake/vcpkg/git-archives/ +# `ply` needs nothing now: the wheel is pinned and pip runs with +# PIP_NO_INDEX (gap 8). # # NB vcpkg's buildtrees peak around 3 GB. They go next to the declared output # (inside bazel-out) rather than $TMPDIR, so a small /tmp tmpfs is not a diff --git a/examples/ladybird/workspace/Meta/fetch_vcpkg_git_archives.py b/examples/ladybird/workspace/Meta/fetch_vcpkg_git_archives.py new file mode 100644 index 0000000..64f0d2a --- /dev/null +++ b/examples/ladybird/workspace/Meta/fetch_vcpkg_git_archives.py @@ -0,0 +1,225 @@ +#!/usr/bin/env python3 +"""Fetch the four vcpkg_from_git archives, WITHOUT running CMake or vcpkg. + +WHY THIS EXISTS +--------------- +Four of vcpkg's inputs are not distfiles. skia and angle pull sub-dependencies +with `vcpkg_from_git`, which -- unlike every other fetch vcpkg makes -- bypasses +the asset cache entirely (`x-asset-sources` never sees it, so +`Meta/vcpkg_capture_assets.sh` cannot record it and `x-block-origin` does not +govern it). What vcpkg_from_git *does* honour is a pre-placed +`downloads/-.tar.gz`, and that is the hook this uses. + +For a long time those four tarballs came from a directory on the author's +machine, staged by a `cp ... 2>/dev/null || true` that could not fail (finding +36). They were in fact a copy of `Build/vcpkg/downloads/`, i.e. of vcpkg's own +cache -- which means the only way to get them was to have run CMake. This script +is the answer to "how do you get them WITHOUT running CMake": + +It takes the LIST of archives from the committed pin (VCPKG_GIT_ARCHIVES in +vcpkg_git_archives.bzl), resolves each one's clone URL out of the portfiles, then +for each: `git clone` + `git -c core.autocrlf=false archive ` -- byte-for-byte +what vcpkg_from_git does internally -- and **verifies the result against the pinned +SHA512, failing on any mismatch**. So the pin stops being trusted: the hashes came +from vcpkg, and reproducing them from scratch with git is the proof that git and +vcpkg agree. Needs the vcpkg checkout (`Meta/ladybird.py vcpkg`, ~70s) for the +portfiles, and nothing else. + +WHY NOT A REPO RULE: these are `git archive` output, so there is no URL for +`http_file`; a repository_rule could shell out to git, but it would then be an +un-cacheable, single-threaded network fetch at load time, and the same eight +lines of git. Keep it as a prefetch that Bazel *verifies* (vcpkg_build.sh +hard-fails in 4s when the tarballs are absent), rather than machinery that hides +a clone inside analysis. +""" + +import argparse +import hashlib +import os +import re +import subprocess +import sys +import tempfile +from pathlib import Path + +LADYBIRD_ROOT = Path(os.environ.get("LADYBIRD_ROOT", Path(__file__).resolve().parent.parent)) + +# WHICH tarballs are needed is NOT decided here -- it is read from the committed +# pin (VCPKG_GIT_ARCHIVES), because that list came from vcpkg itself. This module +# only answers "given this archive name, what URL do I clone?", by scanning +# portfiles for the literal URL/REF arguments. +# +# That split is the whole correctness argument, and it is the second thing I got +# wrong here. A first version *derived* the list by parsing skia's and angle's +# portfiles: it found 8 for skia (against 4 truly fetched) and missed libyuv +# entirely. Both errors have the same cause -- `declare_external_from_git` only +# DECLARES, and `get_externals(${required_externals})` picks from it under +# feature/platform `if()`s, so the set is decided by CMake evaluation, not by the +# text -- and libyuv is a separate port whose portfile calls vcpkg_from_git +# directly. Statically deciding the SET is therefore unsound; statically +# resolving a NAME to a URL is not. + + +def all_portfiles(vcpkg_root: Path) -> list[Path]: + """Overlay first: --overlay-ports shadows the checkout's ports/.""" + overlay = sorted((LADYBIRD_ROOT / "Meta" / "CMake" / "vcpkg" / "overlay-ports").glob("*/portfile.cmake")) + builtin = sorted((vcpkg_root / "ports").glob("*/portfile.cmake")) + shadowed = {p.parent.name for p in overlay} + return overlay + [p for p in builtin if p.parent.name not in shadowed] + + +# Three syntaxes reach vcpkg_from_git, all with literal arguments: +# declare_external_from_git(name URL "..." REF "...") -- skia +# checkout_in_path("" "" "") -- angle +# vcpkg_from_git(URL REF ) -- libyuv, directly +_PATTERNS = ( + re.compile(r'declare_external_from_git\s*\(\s*\w+\s+URL\s+"(?P[^"]+)"\s+REF\s+"(?P[^"]+)"'), + re.compile(r'checkout_in_path\s*\(\s*"[^"]+"\s+"(?P[^"]+)"\s+"(?P[^"]+)"\s*\)'), + re.compile(r'vcpkg_from_git\s*\((?P[^)]*)\)'), +) +_FROM_GIT_URL = re.compile(r'URL\s+"?(?P[^"\s]+)"?') +_FROM_GIT_REF = re.compile(r'\bREF\s+"?(?P[^"\s]+)"?') +_SET_RE = re.compile(r'set\s*\(\s*(?P\w+)\s+"?(?P[0-9a-f]{40})"?\s*\)') + + +def _resolve(value: str, variables: dict) -> str | None: + """Expand ${VAR} against `set(VAR )` in the same portfile. + + Only 40-hex values are collected, so this cannot silently expand to a branch + name: vcpkg_from_git REQUIRES a commit SHA (it rev-parses and compares), and + a ref that is not one should surface here rather than later. + """ + m = re.fullmatch(r"\$\{(\w+)\}", value) + if m: + return variables.get(m.group(1)) + return value if re.fullmatch(r"[0-9a-f]{40}", value) else None + + +def refs_in_portfile(path: Path) -> dict: + """-> {(port, ref): url} for every literal vcpkg_from_git call in one portfile.""" + text = path.read_text() + port = path.parent.name + variables = {m.group("var"): m.group("val") for m in _SET_RE.finditer(text)} + out = {} + for pat in _PATTERNS: + for m in pat.finditer(text): + if "body" in m.groupdict() and m.groupdict().get("body") is not None: + body = m.group("body") + mu, mr = _FROM_GIT_URL.search(body), _FROM_GIT_REF.search(body) + if not (mu and mr): + continue + url, raw_ref = mu.group("url"), mr.group("ref") + else: + url, raw_ref = m.group("url"), m.group("ref") + ref = _resolve(raw_ref, variables) + if ref: + # vcpkg names the archive after the PORT, not the dependency: + # DOWNLOADS/${PORT}-${sanitized_ref}.tar.gz in vcpkg_from_git.cmake. + out[f"{port}-{ref}.tar.gz"] = url + return out + + +def resolve_urls(vcpkg_root: Path, wanted: list) -> dict: + """Find the clone URL for each WANTED archive name. Every name must resolve.""" + index: dict = {} + for pf in all_portfiles(vcpkg_root): + for name, url in refs_in_portfile(pf).items(): + index.setdefault(name, url) + missing = [n for n in wanted if n not in index] + if missing: + raise SystemExit( + "fetch_vcpkg_git_archives: no vcpkg_from_git call found for:\n " + + "\n ".join(missing) + + "\n(the pin lists it, but no portfile in the checkout or overlay declares it --\n" + " the checkout may be at the wrong baseline)" + ) + return {n: index[n] for n in wanted} + + +def committed_hashes() -> dict[str, str]: + bzl = LADYBIRD_ROOT / "vcpkg_git_archives.bzl" + if not bzl.is_file(): + return {} + return dict(re.findall(r"'([^']+\.tar\.gz)':\s*'([0-9a-f]{128})'", bzl.read_text())) + + +def fetch(vcpkg_root: Path, out_dir: Path) -> int: + """Reproduce each pinned tarball with git clone + git archive, verify, install.""" + pinned = committed_hashes() + if not pinned: + raise SystemExit( + "fetch_vcpkg_git_archives: no pinned archives found in vcpkg_git_archives.bzl\n" + " That file IS the list of what to fetch; regenerate the pin first with\n" + " Meta/vcpkg_capture_git_archives.sh (vcpkg install --only-downloads)." + ) + urls = resolve_urls(vcpkg_root, sorted(pinned)) + out_dir.mkdir(parents=True, exist_ok=True) + failures = [] + + for name, want in sorted(pinned.items()): + ref = name.rsplit("-", 1)[1][: -len(".tar.gz")] + dest = out_dir / name + if dest.is_file() and sha512(dest) == want: + print(f" ok (cached) {name}") + continue + with tempfile.TemporaryDirectory(prefix="vcpkg-git-archive-") as tmp: + repo = Path(tmp) / "repo" + print(f" cloning {urls[name]}") + # A full clone: the pinned ref is usually not the tip, and a shallow + # clone cannot archive an arbitrary commit -- the same reason vcpkg + # needs full history for its own registry. + subprocess.run(["git", "clone", "--quiet", urls[name], str(repo)], check=True) + tmp_out = Path(tmp) / name + # `-c core.autocrlf=false` is not optional: it is what + # vcpkg_from_git.cmake passes, and it changes the bytes of any file + # git would otherwise translate. + subprocess.run( + ["git", "-c", "core.autocrlf=false", "archive", ref, "-o", str(tmp_out)], + cwd=repo, + check=True, + ) + got = sha512(tmp_out) + if got != want: + failures.append(f"{name}: sha512 {got[:16]}... != pinned {want[:16]}...") + print(f" MISMATCH {name}") + continue + tmp_out.replace(dest) + print(f" verified {name}") + + for f in failures: + print(f"fetch_vcpkg_git_archives: {f}", file=sys.stderr) + if failures: + return 1 + print(f"{len(pinned)}/{len(pinned)} git-sourced externals present and verified in {out_dir}") + return 0 + + +def sha512(path: Path) -> str: + h = hashlib.sha512() + with path.open("rb") as f: + for chunk in iter(lambda: f.read(1 << 20), b""): + h.update(chunk) + return h.hexdigest() + + +def main() -> int: + ap = argparse.ArgumentParser(description=__doc__, formatter_class=argparse.RawDescriptionHelpFormatter) + ap.add_argument("--vcpkg-root", type=Path, default=LADYBIRD_ROOT / "Build" / "vcpkg") + ap.add_argument( + "--out", + type=Path, + default=LADYBIRD_ROOT / "Meta" / "CMake" / "vcpkg" / "git-archives", + help="where the tarballs are written (the directory Meta/vcpkg_build.sh stages from)", + ) + args = ap.parse_args() + + if not args.vcpkg_root.is_dir(): + raise SystemExit( + f"fetch_vcpkg_git_archives: no vcpkg checkout at {args.vcpkg_root}\n" + " Get one with: python3 Meta/ladybird.py vcpkg (no CMake configure needed)" + ) + return fetch(args.vcpkg_root, args.out) + + +if __name__ == "__main__": + sys.exit(main()) diff --git a/examples/ladybird/workspace/Meta/vcpkg_capture_git_archives.sh b/examples/ladybird/workspace/Meta/vcpkg_capture_git_archives.sh new file mode 100755 index 0000000..dcb9f24 --- /dev/null +++ b/examples/ladybird/workspace/Meta/vcpkg_capture_git_archives.sh @@ -0,0 +1,96 @@ +#!/bin/bash +# Regenerate the vcpkg_from_git pin, without CMake and without a full build. +# +# THE PROBLEM. Four of vcpkg's inputs are not distfiles: skia and angle pull +# sub-dependencies with `vcpkg_from_git`, which bypasses the asset cache +# entirely. `x-asset-sources` never sees them, so Meta/vcpkg_capture_assets.sh +# (which records the other 76 by BEING the asset cache) cannot record them, and +# `x-block-origin` does not govern them either -- they are a plain `git fetch` +# inside a portfile. +# +# WHY NOT JUST READ THE PORTFILES. Because which git externals are used is +# decided by CMake *evaluation*, not by the text. skia declares ten with +# `declare_external_from_git` and then calls +# `get_externals(${required_externals})`, where `required_externals` is built up +# under feature and platform `if()`s -- so a static scan of skia's portfile +# yields 8 where 4 are real. And libyuv's archive comes from the libyuv *port* +# calling vcpkg_from_git directly, which a scan of skia+angle misses entirely. I +# wrote that scanner first; it was wrong in both directions at once. +# +# THE INSTRUMENT. `vcpkg install --only-downloads` runs the portfiles' *fetch* +# phase and stops. That is enough to make vcpkg_from_git produce its tarballs, at +# the refs the real resolution picks, in ~6 minutes with no compilation and no +# CMake configure of Ladybird. Same tactic as the asset capture and as +# scripts/npm_instrument: do not predict what the foreign build system will ask +# for -- run it and record the answer. +# +# THE ONE THAT ESCAPES EVEN THIS. angle's zlib is fetched by `checkout_in_path` +# from angle's *build* phase, not its fetch phase, so --only-downloads does not +# produce it (verified: 3 of the 4 appear). Its (url, ref) is a literal in the +# overlay portfile, so Meta/fetch_vcpkg_git_archives.py resolves and reproduces +# it with git, and the SHA512 below is what proves the two agree. Recording that +# asymmetry here is the point: "--only-downloads gets them all" would be the +# false version of this comment. +# +# Usage: Meta/vcpkg_capture_git_archives.sh [outfile] +# Requires: Build/vcpkg (python3 Meta/ladybird.py vcpkg) and network access. +# Prints namesha512 for every PORT-<40hex>.tar.gz vcpkg produced. +set -uo pipefail + +SRC="${LADYBIRD_ROOT:-$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd)}" +OUT="${1:-/dev/stdout}" +ROOT="$SRC/Build/vcpkg" + +if [ ! -x "$ROOT/vcpkg" ]; then + echo "capture: no bootstrapped vcpkg at $ROOT" >&2 + echo "capture: get one with 'python3 Meta/ladybird.py vcpkg' (no CMake needed)" >&2 + exit 1 +fi + +SCRATCH=$(mktemp -d "${TMPDIR:-/tmp}/vcpkg-git-capture-XXXXXX") +trap 'rm -rf "$SCRATCH"' EXIT INT TERM + +mkdir -p "$SCRATCH/m/Meta/CMake/vcpkg" +cp "$SRC/vcpkg.json" "$SCRATCH/m/" +cp "$SRC/vcpkg-configuration.json" "$SCRATCH/m/" 2>/dev/null +cp -r "$SRC/Meta/CMake/vcpkg/overlay-ports" "$SCRATCH/m/Meta/CMake/vcpkg/" + +# The manifest is copied VERBATIM, never reconstructed: the baseline plus the 45 +# overrides are what select the port versions, and a reconstruction that drifts +# would silently capture refs for different versions than the build uses. +cd "$SCRATCH/m" || exit 1 +VCPKG_ROOT="$ROOT" "$ROOT/vcpkg" install \ + --only-downloads \ + --x-manifest-root="$SCRATCH/m" \ + --overlay-ports="$SCRATCH/m/Meta/CMake/vcpkg/overlay-ports" \ + --triplet=x64-linux-dynamic \ + --host-triplet=x64-linux \ + --x-install-root="$SCRATCH/out" \ + --x-buildtrees-root="$SCRATCH/bt" \ + --downloads-root="$SCRATCH/dl" >&2 +rc=$? +if [ $rc -ne 0 ]; then + echo "capture: vcpkg install --only-downloads failed (rc=$rc)" >&2 + exit $rc +fi + +# vcpkg names these DOWNLOADS/${PORT}-${sanitized_ref}.tar.gz, which is exactly +# the shape below; every other file in downloads/ is an ordinary distfile already +# covered by Meta/vcpkg_assets.tsv. +found=0 +: > "$OUT" +for f in "$SCRATCH/dl"/*.tar.gz; do + name=$(basename "$f") + [[ "$name" =~ ^[a-z0-9]+-[0-9a-f]{40}\.tar\.gz$ ]] || continue + printf '%s\t%s\n' "$name" "$(sha512sum "$f" | cut -d' ' -f1)" >> "$OUT" + found=$((found + 1)) +done + +if [ "$found" -eq 0 ]; then + echo "capture: no PORT-.tar.gz produced -- vcpkg_from_git may have changed" >&2 + exit 1 +fi +echo "capture: recorded $found git-sourced externals" >&2 +echo "capture: NB angle's zlib is fetched in angle's BUILD phase, so it does not" >&2 +echo "capture: appear here; Meta/fetch_vcpkg_git_archives.py reproduces it from" >&2 +echo "capture: the portfile's literal (url, ref) and verifies it against the pin." >&2 diff --git a/tests/test_fetch_vcpkg_git_archives.py b/tests/test_fetch_vcpkg_git_archives.py new file mode 100644 index 0000000..c428bad --- /dev/null +++ b/tests/test_fetch_vcpkg_git_archives.py @@ -0,0 +1,212 @@ +#!/usr/bin/env python3 +"""Tests for examples/ladybird/workspace/Meta/fetch_vcpkg_git_archives.py. + +The script's job is to produce the four `vcpkg_from_git` tarballs *without* +running CMake or vcpkg. What is worth testing is not the git plumbing (that needs +the network) but the two decisions the script makes on its own: + + 1. the LIST of archives comes from the committed pin, never from parsing -- + because a first version derived the list from skia's portfile and got 8 + instead of 4 while missing libyuv entirely (`declare_external_from_git` + declares; feature-conditional `get_externals` picks); + 2. resolving an archive NAME to a clone URL is a text lookup over portfiles, + and every pinned name must resolve or the script must fail loudly rather + than silently fetch a subset. +""" + +import importlib.util +import tempfile +from pathlib import Path + +REPO = Path(__file__).resolve().parent.parent +SCRIPT = REPO / "examples" / "ladybird" / "workspace" / "Meta" / "fetch_vcpkg_git_archives.py" + + +class expect_exit: + """Minimal assertRaises(SystemExit), since this repo's tests are plain functions.""" + + def __enter__(self): + return self + + def __exit__(self, exc_type, exc, tb): + assert exc_type is SystemExit, "expected SystemExit, got %r" % (exc_type,) + self.exception = exc + return True + + +def load(ladybird_root: Path): + """Import the script with LADYBIRD_ROOT pointed at a fixture tree.""" + import os + + old = os.environ.get("LADYBIRD_ROOT") + os.environ["LADYBIRD_ROOT"] = str(ladybird_root) + try: + spec = importlib.util.spec_from_file_location(f"fvga_{id(ladybird_root)}", SCRIPT) + mod = importlib.util.module_from_spec(spec) + spec.loader.exec_module(mod) + return mod + finally: + if old is None: + os.environ.pop("LADYBIRD_ROOT", None) + else: + os.environ["LADYBIRD_ROOT"] = old + + +SKIA_PORTFILE = """ +include("${CMAKE_CURRENT_LIST_DIR}/skia-functions.cmake") +declare_external_from_git(piex + URL "https://android.googlesource.com/platform/external/piex.git" + REF "bb217acdca1cc0c16b704669dd6f91a1b509c406" + LICENSE_FILE LICENSE +) +declare_external_from_git(spirv-tools + URL "https://github.com/KhronosGroup/SPIRV-Tools.git" + REF "2d14d2e76aa7de72404b17078eda15c20a6a0389" +) +set(required_externals expat piex zlib wuffs) +get_externals(${required_externals}) +""" + +ANGLE_PORTFILE = """ +set(ANGLE_THIRDPARTY_ZLIB_COMMIT 4028ebf8710ee39d2286cb0f847f9b95c59f84d8) +checkout_in_path( + "${SOURCE_PATH}/third_party/zlib" + "https://chromium.googlesource.com/chromium/src/third_party/zlib" + "${ANGLE_THIRDPARTY_ZLIB_COMMIT}" +) +""" + +LIBYUV_PORTFILE = """ +vcpkg_from_git( + OUT_SOURCE_PATH SOURCE_PATH + URL https://chromium.googlesource.com/libyuv/libyuv + REF d98915a654d3564e4802a0004add46221c4e4348 + PATCHES cmake.diff +) +""" + +PIN = """ +VCPKG_GIT_ARCHIVES = { + 'angle-4028ebf8710ee39d2286cb0f847f9b95c59f84d8.tar.gz': '%s', + 'libyuv-d98915a654d3564e4802a0004add46221c4e4348.tar.gz': '%s', + 'skia-bb217acdca1cc0c16b704669dd6f91a1b509c406.tar.gz': '%s', +} +""" % ("a" * 128, "b" * 128, "c" * 128) + + +def make_tree(tmp: Path, *, pin: str = PIN, overlay_angle: bool = True) -> Path: + root = tmp / "ladybird" + (root / "Build" / "vcpkg" / "ports" / "skia").mkdir(parents=True) + (root / "Build" / "vcpkg" / "ports" / "libyuv").mkdir(parents=True) + (root / "Build" / "vcpkg" / "ports" / "angle").mkdir(parents=True) + (root / "Build" / "vcpkg" / "ports" / "skia" / "portfile.cmake").write_text(SKIA_PORTFILE) + (root / "Build" / "vcpkg" / "ports" / "libyuv" / "portfile.cmake").write_text(LIBYUV_PORTFILE) + # The builtin angle port exists but (as upstream) has no checkout_in_path; + # the overlay is what carries it. That asymmetry is the point of the test. + (root / "Build" / "vcpkg" / "ports" / "angle" / "portfile.cmake").write_text("# builtin angle\n") + if overlay_angle: + d = root / "Meta" / "CMake" / "vcpkg" / "overlay-ports" / "angle" + d.mkdir(parents=True) + (d / "portfile.cmake").write_text(ANGLE_PORTFILE) + (root / "vcpkg_git_archives.bzl").write_text(pin) + return root + + +def test_pin_is_the_authority_not_the_portfiles(): + """skia's portfile declares 2 refs; only the pinned one is fetched. + + This is the regression test for the bug that made me rewrite the script: + deriving the set from `declare_external_from_git` over-collects, because + feature-conditional get_externals() decides what is actually used. + """ + with tempfile.TemporaryDirectory() as t: + root = make_tree(Path(t)) + mod = load(root) + pinned = mod.committed_hashes() + assert len(pinned) == 3 + assert "skia-bb217acdca1cc0c16b704669dd6f91a1b509c406.tar.gz" in pinned + # declared in the portfile, NOT pinned -> must not be fetched + assert "skia-2d14d2e76aa7de72404b17078eda15c20a6a0389.tar.gz" not in pinned + +def test_missing_pin_is_a_hard_error(): + with tempfile.TemporaryDirectory() as t: + root = make_tree(Path(t), pin="# nothing pinned\n") + mod = load(root) + with expect_exit() as cm: + mod.fetch(root / "Build" / "vcpkg", Path(t) / "out") + assert "no pinned archives" in str(cm.exception) + + +def test_resolves_all_three_call_syntaxes(): + """declare_external_from_git, checkout_in_path and a bare vcpkg_from_git.""" + with tempfile.TemporaryDirectory() as t: + root = make_tree(Path(t)) + mod = load(root) + urls = mod.resolve_urls(root / "Build" / "vcpkg", sorted(mod.committed_hashes())) + assert urls["angle-4028ebf8710ee39d2286cb0f847f9b95c59f84d8.tar.gz"] == "https://chromium.googlesource.com/chromium/src/third_party/zlib" + assert urls["libyuv-d98915a654d3564e4802a0004add46221c4e4348.tar.gz"] == "https://chromium.googlesource.com/libyuv/libyuv" + assert urls["skia-bb217acdca1cc0c16b704669dd6f91a1b509c406.tar.gz"] == "https://android.googlesource.com/platform/external/piex.git" + +def test_expands_a_ref_held_in_a_set_variable(): + """angle's REF is ${ANGLE_THIRDPARTY_ZLIB_COMMIT}, not a literal.""" + with tempfile.TemporaryDirectory() as t: + root = make_tree(Path(t)) + mod = load(root) + found = mod.refs_in_portfile( + root / "Meta" / "CMake" / "vcpkg" / "overlay-ports" / "angle" / "portfile.cmake" + ) + assert "angle-4028ebf8710ee39d2286cb0f847f9b95c59f84d8.tar.gz" in found + +def test_overlay_shadows_the_builtin_port(): + """--overlay-ports wins; the builtin angle portfile declares nothing.""" + with tempfile.TemporaryDirectory() as t: + root = make_tree(Path(t)) + mod = load(root) + paths = mod.all_portfiles(root / "Build" / "vcpkg") + angle = [p for p in paths if p.parent.name == "angle"] + assert len(angle) == 1, "angle must appear once, from the overlay" + assert "overlay-ports" in str(angle[0]) + +def test_unresolvable_pinned_name_fails_loudly(): + """A pin naming a port no portfile declares must not silently fetch a subset.""" + pin = PIN.replace("skia-bb217acdca1cc0c16b704669dd6f91a1b509c406", "ghost-" + "0" * 40) + with tempfile.TemporaryDirectory() as t: + root = make_tree(Path(t), pin=pin) + mod = load(root) + with expect_exit() as cm: + mod.resolve_urls(root / "Build" / "vcpkg", sorted(mod.committed_hashes())) + msg = str(cm.exception) + assert "ghost-" in msg + assert "no vcpkg_from_git call found" in msg + +def test_a_ref_that_is_not_a_sha_is_refused(): + """vcpkg_from_git requires a commit SHA; a branch name must not be accepted.""" + with tempfile.TemporaryDirectory() as t: + root = make_tree(Path(t)) + pf = root / "Build" / "vcpkg" / "ports" / "libyuv" / "portfile.cmake" + pf.write_text(LIBYUV_PORTFILE.replace("d98915a654d3564e4802a0004add46221c4e4348", "main")) + mod = load(root) + assert mod.refs_in_portfile(pf) == {} + + +def test_a_file_with_the_wrong_hash_is_not_accepted_as_cached(): + with tempfile.TemporaryDirectory() as t: + root = make_tree(Path(t)) + mod = load(root) + out = Path(t) / "out" + out.mkdir() + name = "libyuv-d98915a654d3564e4802a0004add46221c4e4348.tar.gz" + (out / name).write_bytes(b"corrupt") + assert mod.sha512(out / name) != "b" * 128 + +def test_sha512_matches_hashlib(): + import hashlib + + with tempfile.TemporaryDirectory() as t: + root = make_tree(Path(t)) + mod = load(root) + f = Path(t) / "x" + f.write_bytes(b"hello world" * 1000) + assert mod.sha512(f) == hashlib.sha512(b"hello world" * 1000).hexdigest() + + From ca61823825600ed615d122a071df3a728c2ab3b0 Mon Sep 17 00:00:00 2001 From: Ulf Adams Date: Mon, 10 Aug 2026 17:18:37 +0000 Subject: [PATCH 15/59] Ladybird: yes, http_file can fetch the HSTS table -- and that is why we should not Measured all four http_file combinations rather than reasoning about them. Unpinned + main builds today; unpinned also means Bazel caches the first fetch forever (proved with a local origin: changed upstream, rebuilt, got VERSION-ONE in 0.3s, no warning). The table moved during this session -- service.gov.scot left the list, 94627 -> 94626 entries -- and the pinned tag 139.0.7258.5 has 168593, so pinning only on the Bazel side diverges from CMake by 74k entries. Pinning has to happen where both build systems read it. --- docs/CASE-ladybird-migration.md | 52 +++++++++++++++++++++++++++++++++ 1 file changed, 52 insertions(+) diff --git a/docs/CASE-ladybird-migration.md b/docs/CASE-ladybird-migration.md index 894c74a..b40422f 100644 --- a/docs/CASE-ladybird-migration.md +++ b/docs/CASE-ladybird-migration.md @@ -1946,6 +1946,58 @@ foreign build system is not something the converter can pin unilaterally.** It c was found — and detecting it is the converter's job; fixing it is a change to the project. +### Can we just `http_file` the HSTS table? Measured, all four combinations + +Asked directly, so I ran it rather than reasoned about it. `http_file` has two knobs +that matter — pinned ref or `main`, `sha256` or none — and Bazel behaves differently +in all four: + +| URL ref | `sha256` | what Bazel does | what you get | +|---|---|---|---| +| `main` | none | **fetches, builds, and prints** `DEBUG: … a canonical reproducible form can be obtained by modifying arguments integrity = "sha256-ObT9…"` | works; unpinned | +| `main` | given | fails the moment upstream moves: `Checksum was 5d5df26… but wanted 000…` | a build that breaks on Chromium's commit rate | +| pinned tag | given | fetches; 18.7 MB | hermetic, **and not what CMake built** | +| any | none + plain `http://` | refuses outright: `No URLs left after removing plain http URLs due to missing checksum` | — | + +So the answer to "can we?" is **yes, mechanically** — row 1 builds today. Two measured +facts decide whether we should. + +**First: unpinned means Bazel caches whatever it saw first, forever.** With a local +HTTP server as the origin I fetched `VERSION-ONE`, changed the file upstream to +`VERSION-TWO`, rebuilt: `-> VERSION-ONE`, in 0.3 s, no refetch, no warning. Same with +a `file://` origin. That is the right behaviour for a *pinned* input and the worst +possible behaviour for an unpinned one: **two developers who first built on different +days build different browsers and neither can tell.** The output is a +94,000-entry `constexpr Array` of domains that get forced to HTTPS — a silent +difference in security behaviour, not in a log line. + +**Second, and this is the number that settles it:** the table moved *while I was +working on this*. The file this machine's CMake configure fetched from `main` and the +one `http_file` fetched from `main` today differ by one entry: + +``` +< static constexpr Array s_hsts_preload_entries { { +> static constexpr Array s_hsts_preload_entries { { +- HSTSPreloadEntry { "service.gov.scot"sv, true }, +``` + +One domain left Chromium's preload list, so `HSTSPreloadData.cpp` is 53 bytes shorter, +and the byte-parity claim this whole document rests on would have failed for a reason +that has nothing to do with the migration. And the *pinned* tag is not a way out +either: `139.0.7258.5` yields **168,593** entries against today's **94,626** — the +list was pruned hard in between, so pinning unilaterally on the Bazel side doesn't +drift, it just diverges by 74,000 entries. + +That is the whole argument for fixing it upstream in one line +(`Meta/CMake/hsts_preload.cmake`: pin the URL to a revision) rather than papering over +it in the overlay: **pinning has to happen where both build systems read it, or the +pin creates the difference it was supposed to prevent.** And the upstream change is +additive, which I verified separately: `download_file` is a no-op when the file +already exists (tested with `ENABLE_NETWORK_DOWNLOADS=OFF`), so a file staged by Bazel +at a pinned revision is consumed by CMake unchanged. Until then the overlay stages it, +and the honest reason is in this table — an unpinned `http_file` would be the only +unpinned input in the overlay, and it would be one whose staleness is invisible. + Two shortcuts I tested and rejected, both of which look like simplifications and one of which I would have shipped: From a43967a496f97d905c3394a2b15adbfb87222120 Mon Sep 17 00:00:00 2001 From: Ulf Adams Date: Mon, 10 Aug 2026 18:20:47 +0000 Subject: [PATCH 16/59] Ladybird: pin the HSTS table downstream -- to the commit main serves, not a tag Upstream's hsts_preload.cmake fetches Chromium's transport_security_state_static.json from main, and we cannot change upstream. So the overlay pins it for itself: hsts_preload.bzl declares one http_file at an immutable commit + sha256, MODULE.bazel names the repo, and gen_HSTSPreloadData consumes @hsts_preload_json//file instead of the CMake configure's leftovers under Build/caches (wired in the emitter, since codegen_root.bzl is generated). My earlier answer -- 'fix it upstream, stage it until then' -- was wrong because it made someone else's repo a prerequisite for our hermeticity. What misled me was pinning the wrong revision: a release tag (139.0.7258.5) serves 18.7MB and generates 168,593 entries against main's 94,626, so pinning THAT does trade hermeticity for parity. The commit main is serving has bytes identical to CMake's download (cmp, 10,521,748 bytes), so pinning it costs no parity at all. Meta/pin_hsts_preload.py re-pins by measuring, and --expect-same-as refuses to write a pin whose bytes differ from the file the other build system already has. Verified on the fresh clone with the CMake-downloaded file deleted: gen_HSTSPreloadData's outputs are byte-identical to the CMake reference and //:LibHTTP builds (RC=0, 183 actions). 12 new tests, 164/164. The upstream one-liner is filed as a bug, not depended on. --- docs/CASE-ladybird-migration.md | 80 ++++-- examples/ladybird/README.md | 149 ++++++---- examples/ladybird/workspace/MODULE.bazel | 10 + .../workspace/Meta/emit_root_codegen_bazel.py | 19 ++ .../workspace/Meta/pin_hsts_preload.py | 189 +++++++++++++ examples/ladybird/workspace/codegen_root.bzl | 4 +- examples/ladybird/workspace/hsts_preload.bzl | 55 ++++ tests/test_pin_hsts_preload.py | 260 ++++++++++++++++++ tests/test_vcpkg_plumbing.py | 35 ++- 9 files changed, 704 insertions(+), 97 deletions(-) create mode 100644 examples/ladybird/workspace/Meta/pin_hsts_preload.py create mode 100644 examples/ladybird/workspace/hsts_preload.bzl create mode 100644 tests/test_pin_hsts_preload.py diff --git a/docs/CASE-ladybird-migration.md b/docs/CASE-ladybird-migration.md index b40422f..39d283e 100644 --- a/docs/CASE-ladybird-migration.md +++ b/docs/CASE-ladybird-migration.md @@ -1692,7 +1692,7 @@ do with `Build/full`.** |---|---|---| | `Build/vcpkg` — a microsoft/vcpkg checkout at `vcpkg.json`'s `builtin-baseline` | `/tmp/.../root/vcpkg: No such file or directory` | `Meta/ladybird.py vcpkg` had been run months earlier | | its `.git` (120 MB), which vcpkg needs for `git read-tree` to resolve versioned ports — and which the filegroup **excludes** | `fatal: not a git repository: '.git'` … `while checking out port sqlite3` | the action is `no-sandbox`, so it read the real checkout regardless of what was declared | -| `Build/caches/HSTSPreload/transport_security_state_static.json`, an **unversioned** download CMake does at configure time from Chromium's `main` | `missing input file '//:Build/caches/...'` | CMake's configure had already fetched it | +| `Build/caches/HSTSPreload/transport_security_state_static.json`, an **unversioned** download CMake does at configure time from Chromium's `main` | `missing input file '//:Build/caches/...'` | CMake's configure had already fetched it. **Now fixed**: pinned downstream to a commit + sha256 (`hsts_preload.bzl`) and fetched by Bazel | | two path bugs in `Meta/vcpkg_build.sh`: the distfile index *and its entries* were execroot-relative, and vcpkg runs the asset script from its own cwd | `awk: cannot open ...`, then `cp: cannot stat ...`, surfacing as `no asset cache hits` and `x-block-origin blocks trying the authoritative source` | the checkout already had `downloads/tools/cmake-4.4.0-linux`, so vcpkg never *asked* the script for a tool | | `ply`, which the `angle` port installs with **`pip install ply`** | `No matching distribution found for ply` | this sandbox exports `HTTP_PROXY`, and the action inherits it | | the four `vcpkg_from_git` archives, staged from a directory I had made **by hand** — `vcpkg_git_archives.bzl` is generated, committed, documented, and **loaded by nothing** | 20 minutes in: `git fetch https://android.googlesource.com/.../piex.git … Error code: 128` | the directory existed on my disk from when I built the pin | @@ -1914,12 +1914,14 @@ the 4**, because angle's zlib is fetched from angle's *build* phase via `checkout_in_path`, not its fetch phase. The script says so; "--only-downloads gets them all" would have been the comfortable, false version. -That leaves **exactly one** genuine hermeticity defect: the HSTS table, fetched from -Chromium's unversioned `main`, where there is no revision to pin to. A useful detail -found while checking whether the fix is invasive: CMake's `download_file` is a no-op -when the file already exists (verified with `ENABLE_NETWORK_DOWNLOADS=OFF`), so a -pinned file staged by Bazel is consumed by CMake unchanged — **the upstream change is -additive**, one URL gaining a revision, no restructuring. +That left **exactly one** genuine hermeticity defect: the HSTS table, fetched from +Chromium's unversioned `main`. It is now closed too — pinned *downstream*, to the +commit `main` is serving rather than to a release tag, which is the part I got wrong +first; see below. A useful detail found while checking how invasive the upstream fix +would be, and which also makes the downstream pin shareable: CMake's `download_file` +is a no-op when the file already exists (verified with `ENABLE_NETWORK_DOWNLOADS=OFF`), +so the Bazel-fetched pinned file, copied into `Build/caches/HSTSPreload/` before +configuring, is consumed by CMake unchanged. My four successive answers to "what is needed" were: a `repository_rule`, a prefetch script, a `cp`, and finally a script plus a capture instrument. The middle two were @@ -1930,21 +1932,40 @@ answering it.** For the audience that has CMake it is the right answer; for the audience this migration exists to serve it is a non-answer, and I gave it because it let me stop working. -**The HSTS table (row 3) is the one that genuinely cannot be fixed here, and it is -not a Bazel problem.** `Meta/CMake/hsts_preload.cmake` fetches +**The HSTS table (row 3) is the one whose unpinned fetch is not ours to fix — which +turns out not to mean we cannot pin it.** `Meta/CMake/hsts_preload.cmake` fetches `raw.githubusercontent.com/chromium/chromium/**main**/net/http/transport_security_state_static.json` -— an unversioned ref, at CMake *configure* time. Pinning it on the Bazel side works -mechanically (I fetched it at tag `139.0.7258.5` and the generator ran fine), but not -*honestly*: the pinned file is 18.7 MB against the 10.5 MB one this machine -configured with, and the generated table differs. **Pinning only on the Bazel side -would make Bazel's output diverge from CMake's — trading a hermeticity gap for a -parity gap, and the parity baseline is what every claim in this document rests on.** -The fix is upstream: pin the URL to a revision in Ladybird's CMake, then `http_file` -that revision. Worth stating plainly as a general shape: **an unpinned input in the -foreign build system is not something the converter can pin unilaterally.** It can -*detect* it — Bazel already fails cleanly with `missing input file`, which is how it -was found — and detecting it is the converter's job; fixing it is a change to the -project. +— an unversioned ref, at CMake *configure* time. My first answer was "pin it +upstream, until then stage it", and it was wrong in the way that matters: **it made +someone else's repo a prerequisite for our hermeticity.** A converter usually cannot +change the project it converts, so an answer that requires an upstream patch is an +answer that never ships. + +The correction, and it is a general shape worth stating: **a converter cannot pin an +input on the foreign build system's behalf, but it can pin it for itself — provided +it pins the revision the foreign system is currently *serving*, and proves that with +a byte comparison rather than a hash it invented.** What made me think otherwise was +picking the wrong revision. I tested a Chromium *release tag* (`139.0.7258.5`): 18.7 +MB against `main`'s 10.5 MB, **168,593** generated entries against **94,626** — so +pinning *that* really would have traded a hermeticity gap for a parity gap. But the +commit `main` pointed at when this machine configured serves bytes identical to what +CMake downloaded (`cmp`, 10,521,748 bytes), and pinning **that** costs no parity at +all. A tag is a pin to a *different table*; a commit is a pin to *this* one. The +distinction is the whole finding, and I had generalized "pinning breaks parity" from a +single badly chosen pin. + +So the overlay now pins downstream: `hsts_preload.bzl` (an `http_file` at that commit ++ sha256, consumed by `gen_HSTSPreloadData` as `@hsts_preload_json//file`) and +`Meta/pin_hsts_preload.py`, which re-pins by *measuring* — it downloads the file and +writes the hash it computed, and `--expect-same-as` refuses to write a pin whose bytes +differ from the file the other build system already has. Verified on the fresh clone +with the CMake-downloaded file **deleted** — so the pin is the only possible source: +`HSTSPreloadData.h`/`.cpp` byte-identical to the CMake reference, and `//:LibHTTP` +compiles and links them (RC=0). The residual cost is stated rather than hidden: CMake still +tracks `main`, so a configure newer than the pin disagrees with Bazel — one pinned +input against one unpinned one, a dated disagreement with a sha to look at, instead of +two unpinned fetches that happened to agree. The upstream one-liner is filed as a bug, +not depended upon. ### Can we just `http_file` the HSTS table? Measured, all four combinations @@ -1988,15 +2009,14 @@ either: `139.0.7258.5` yields **168,593** entries against today's **94,626** — list was pruned hard in between, so pinning unilaterally on the Bazel side doesn't drift, it just diverges by 74,000 entries. -That is the whole argument for fixing it upstream in one line -(`Meta/CMake/hsts_preload.cmake`: pin the URL to a revision) rather than papering over -it in the overlay: **pinning has to happen where both build systems read it, or the -pin creates the difference it was supposed to prevent.** And the upstream change is -additive, which I verified separately: `download_file` is a no-op when the file -already exists (tested with `ENABLE_NETWORK_DOWNLOADS=OFF`), so a file staged by Bazel -at a pinned revision is consumed by CMake unchanged. Until then the overlay stages it, -and the honest reason is in this table — an unpinned `http_file` would be the only -unpinned input in the overlay, and it would be one whose staleness is invisible. +So row 1 is out (an unpinned `http_file` is the only input in the overlay whose +staleness would be invisible), row 3's *tag* is out (it diverges by 74,000 entries), +and what is left is row 3 with a **commit**: `3d75766` — the newest commit touching +the path, i.e. what `main` serves — whose bytes are identical to CMake's download. +That is the pin that shipped. The rule I was reaching for and got backwards on the +first pass: **pin what the other build system is serving today, and prove it with +`cmp`; do not pin what looks canonical.** A release tag looks like the responsible +choice and is the one that breaks parity. Two shortcuts I tested and rejected, both of which look like simplifications and one of which I would have shipped: diff --git a/examples/ladybird/README.md b/examples/ladybird/README.md index 06c0723..17e0cbd 100644 --- a/examples/ladybird/README.md +++ b/examples/ladybird/README.md @@ -28,25 +28,26 @@ the closure of all six returns **0** targets under `Build/full` (down from 741). harness — a converter-development dependency, not a build dependency. **A fresh clone now builds and renders — after one ordinary CMake build has run -once.** (Three inputs come from that build rather than from Bazel; a Ladybird +once.** (Two inputs come from that build rather than from Bazel; a Ladybird developer therefore stages nothing, and the gap is real only for a Bazel-only clone -— see [how a Ladybird developer gets them](#how-a-ladybird-developer-gets-these-three-inputs).) The +— see [how a Ladybird developer gets them](#getting-the-two-remaining-inputs--with-and-without-cmake). +The third, the HSTS table, is now pinned and fetched by Bazel.) The end-to-end test (clone into a new directory, drop in the overlay, nothing else) was run for the first time and failed **six separate times**; `Build/full` was the dependency I had removed, and **five of the six had nothing to do with it**. All six are addressed below, and the result is verified on the clone rather than asserted: `//:vcpkg_installed` builds all 76 ports offline, all six binaries build (2,842 actions, RC=0), and `--headless=text`/`--headless=layout-tree` -are **byte-identical to the CMake reference on all three test pages**. Rows 1–3 -are still manual staging: two are fixed bugs (#4, #5), one now fails loudly -instead of 20 minutes later (#6), and the remainder are inputs the recipe must -produce or document: +are **byte-identical to the CMake reference on all three test pages**. Rows 1–2 +are still manual staging (a prefetch, `Meta/ladybird.py vcpkg`); row 3 is now +**fixed** — Bazel fetches the HSTS table from a pinned commit; #4 and #5 were +bugs and are fixed; #6 now fails loudly instead of 20 minutes later: | # | What is missing on a fresh clone | Why the dev machine hid it | |---|---|---| | 1 | **`Build/vcpkg`** — a git clone of microsoft/vcpkg at `vcpkg.json`'s `builtin-baseline`, which `//Build/vcpkg:tree` globs with `allow_empty = True`. Matches *one* file (its own `BUILD.bazel`) and the build fails with `/tmp/.../root/vcpkg: No such file or directory`. Ladybird's own `Meta/ladybird.py vcpkg` creates it; the recipe above never says so | `Meta/ladybird.py vcpkg` had been run months earlier | | 2 | **`.git` inside that checkout** — vcpkg resolves versioned ports with `git read-tree`, so it is load-bearing (120 MB), and the filegroup *excludes* it. The action is `no-sandbox`, so it reads the real path and got it anyway: an **undeclared input the build needs** | `no-sandbox` + a real checkout on disk | -| 3 | **`Build/caches/HSTSPreload/transport_security_state_static.json`** (10 MB) — an *unversioned, unpinned* network download CMake does at configure time (`Meta/CMake/hsts_preload.cmake` fetches Chromium's `main`). `codegen_root.bzl` names it as a genrule `srcs`, so Bazel fails cleanly with `missing input file` — but there is no rule that produces it | the CMake configure had already downloaded it | +| 3 | **`Build/caches/HSTSPreload/transport_security_state_static.json`** (10 MB) — an *unversioned, unpinned* network download CMake does at configure time (`Meta/CMake/hsts_preload.cmake` fetches Chromium's `main`). `codegen_root.bzl` named it as a genrule `srcs`, so Bazel failed cleanly with `missing input file` — but nothing produced it. **Fixed:** pinned downstream to a Chromium commit + sha256 ([`hsts_preload.bzl`](workspace/hsts_preload.bzl)), fetched with `http_file`, and the generated table is byte-identical to CMake's — verified with the staged file *deleted* | the CMake configure had already downloaded it | | 4 | **Two path bugs in `Meta/vcpkg_build.sh`** — the distfile index and its entries were passed **execroot-relative**, and vcpkg invokes the asset-cache script from its own cwd, so `awk`/`cp` looked in the wrong directory. Every asset lookup failed, reported as `no asset cache hits`, and `x-block-origin` then correctly refused the network. Fixed here (absolutize both) | the vcpkg checkout already had `downloads/tools/cmake-4.4.0-linux` from an earlier run, so vcpkg never *asked* the script for a tool | | 5 | **The `angle` port runs `pip install ply`** (`x_vcpkg_get_python_packages`), which is **not** an asset-cache download and therefore not covered by the pin. A real hole in the "zero network access" claim: `vcpkg_tree` set `requires-network: "0"`, but that is a *scheduling hint*, and with `no-sandbox: "1"` **nothing enforced it** — with `use_default_shell_env = True` the action inherited `HTTP_PROXY`/`HTTPS_PROXY` and pip reached PyPI. **Fixed here:** the wheel is pinned by URL+sha256 (`vcpkg_python_packages.bzl`), staged into a find-links dir, and pip runs with `PIP_NO_INDEX` and the proxy variables unset | this sandbox exports a proxy, so pip silently succeeded through it | | 6 | **The four `vcpkg_from_git` archives were pre-placed from a directory I had made by hand.** `vcpkg_git_archives.bzl` is generated, committed, listed in the table below — and **loaded by nothing**. The staging was `if [ -d ... ]; then cp ... 2>/dev/null \|\| true; fi`: three ways to succeed while copying nothing. Without them skia fails ~20 min in with `git fetch https://android.googlesource.com/.../piex.git … Error code: 128`, naming neither the directory nor the tarball. **Now a hard error in 4 seconds** naming both; reproducing them is a prefetch, not a rule — they are `git archive` output, so there is no URL to `http_file`, but cloning the pinned URL and `git archive`-ing the pinned ref reproduces the committed SHA512 exactly (verified on `libyuv`) | I created the directory by hand while building the pin, months before | @@ -69,17 +70,17 @@ dependency I went looking for, so it is the one I found.** The check that would have caught all five is not a better `cquery` — it is doing the clone. See [Known gaps](#known-gaps). -### Getting the three inputs — with and without CMake +### Getting the two remaining inputs — with and without CMake Two audiences, two answers. **A Ladybird developer stages nothing:** one ordinary -`./Meta/ladybird.py build` produces all three as side effects, which is exactly why -they stayed invisible here for months. +`./Meta/ladybird.py build` produces both as side effects, which is exactly why they +stayed invisible here for months. | Input | Who produces it in a normal build | |---|---| | the vcpkg checkout + its `.git` | `Meta/Utils/build_vcpkg.py`, called by `ladybird.py` **`build`** as well as `vcpkg`: clone, checkout `builtin-baseline`, bootstrap the tool at the tag+SHA512 in `scripts/vcpkg-tool-metadata.txt`. ~70 s | -| `Build/caches/HSTSPreload/transport_security_state_static.json` | the CMake **configure**, via `hsts_preload.cmake` → `download_file`, gated on `ENABLE_NETWORK_DOWNLOADS` (default ON) | | the four `git archive` tarballs | **vcpkg itself**, while building skia and angle | +| ~~the HSTS preload table~~ | **Bazel**, now: `@hsts_preload_json//file`, pinned in `hsts_preload.bzl` | **Without ever running CMake — the actual answer, and it needs no CMake at all:** @@ -89,8 +90,8 @@ python3 Meta/ladybird.py vcpkg # 1. the checkout + .git (~70s). No python3 Meta/fetch_vcpkg_git_archives.py # 2. the four git archives (~80s), each # reproduced with git clone + git archive # and VERIFIED against the committed SHA512 -curl -Lo Build/caches/HSTSPreload/transport_security_state_static.json \ - "$HSTS_URL" # 3. the HSTS table (see caveat below) + # (the HSTS table needs no step: Bazel + # fetches it from a pinned commit) bazel build //:ladybird //:WebContent //:RequestServer //:ImageDecoder \ //:Compositor //:WebWorker ``` @@ -114,18 +115,55 @@ knowing, because both were mistakes I made first: it never appears in a downloads-only run. That asymmetry is recorded in the script rather than smoothed over. -Step 3 is the **one genuine hermeticity defect** and it cannot be closed here. -`hsts_preload.cmake` fetches Chromium's **`main`** — unversioned — so there is no -revision to pin to. Pinning a revision on the Bazel side alone would make Bazel's -output diverge from CMake's (the file at tag `139.0.7258.5` is 18.7 MB against the -10.5 MB `main` served when this machine configured, and the generated table differs), -trading a hermeticity gap for a **parity** gap. The fix is one line upstream: pin the -URL to a revision in `hsts_preload.cmake`, then `http_file` the same revision, and -both build systems consume one pinned input. Usefully, `download_file` is already a -no-op when the file exists (verified with `ENABLE_NETWORK_DOWNLOADS=OFF`), so a pinned -file staged by Bazel is consumed by CMake unchanged — the upstream change is additive. - -Three shortcuts tested and rejected, all of which look like simplifications: +### The HSTS table: pinned downstream, because upstream is not ours to fix + +`Meta/CMake/hsts_preload.cmake` downloads Chromium's +`net/http/transport_security_state_static.json` from **`main`** at configure time. +The generator turns it into a ~95,000-entry `constexpr Array` of domains LibHTTP +forces to HTTPS, so *the day you configured* decides a security-relevant table. +The right fix is one line in that `.cmake` file; we do not control it, so the +overlay pins it **downstream** and the upstream fetch is a bug report: + +- [`hsts_preload.bzl`](workspace/hsts_preload.bzl) — a module extension declaring + one `http_file` at an immutable commit URL with a `sha256`. `MODULE.bazel` names + it; `codegen_root.bzl`'s `gen_HSTSPreloadData` takes `@hsts_preload_json//file` + as `srcs` instead of a path under `Build/caches`. +- [`Meta/pin_hsts_preload.py`](workspace/Meta/pin_hsts_preload.py) — regenerates + that file: resolves the newest commit touching the path, downloads it, and writes + **the hash it measured**. `--expect-same-as ` refuses to write unless the + pinned bytes equal a file you already have, which is how this pin was shown not + to move the generated table. + +**Verified, on the fresh clone, with the CMake-downloaded file deleted** — so the +pinned fetch is the only possible source: `bazel build //:gen_HSTSPreloadData` +produces both outputs **byte-identical to the CMake reference** +(`HSTSPreloadData.cpp`, 4,873,678 bytes), and `bazel build //:LibHTTP` compiles and +links them (RC=0, 183 actions). + +Two things make this pin honest rather than convenient, and both are measurements: + +- **Pin a commit, not a release tag.** A tag is a pin to a *different table*: at + `139.0.7258.5` the file is 18.7 MB and generates **168,593** entries against this + commit's **94,626**. Pinning a tag would have traded a hermeticity gap for a + parity gap. The commit `main` pointed at when the reference build configured + serves bytes identical to what CMake downloaded — checked with `cmp`. +- **Unpinned would have been silently wrong.** `http_file` *does* work with no + `sha256` on the `main` URL (it builds, and Bazel prints the integrity it would + have used) — but then Bazel caches the first fetch forever: with a local origin, + changing the file upstream and rebuilding returned the **old** content in 0.3 s + with no warning. And the table moved during this work — `service.gov.scot` left + the list, 94,627 → 94,626 entries — so an unpinned fetch would have broken + byte-parity for a reason unrelated to the migration. + +The cost of pinning downstream only: **CMake still tracks `main`**, so a configure +newer than the pin disagrees with Bazel. That is now one pinned input versus one +unpinned one — a visible, dated disagreement with a commit sha to look at — instead +of two unpinned fetches that happened to agree. And it is closeable without touching +upstream: `download_file` is a no-op when the file already exists (verified with +`ENABLE_NETWORK_DOWNLOADS=OFF`), so copying Bazel's fetched file into +`Build/caches/HSTSPreload/` before configuring makes CMake consume the same pin. + +### Three shortcuts tested and rejected, all of which look like simplifications - **`--depth 1` on the vcpkg clone.** 8.7 MB instead of 121 MB, and `read-tree` even succeeds for some ports — then resolution fails on ffmpeg and harfbuzz with @@ -180,6 +218,8 @@ an FFI header collision and an entire missing Rust target. See | `Meta/vcpkg_capture_assets.sh` | Records the 76-distfile pin, by *being* vcpkg's asset cache. The one run allowed to fetch | | `Meta/fetch_vcpkg_git_archives.py` | Produces the 4 `vcpkg_from_git` tarballs **without CMake**: takes the list from the committed pin, resolves each clone URL out of the portfiles, then `git clone` + `git -c core.autocrlf=false archive ` and **verifies against the pinned SHA512**. 4/4 byte-identical | | `Meta/vcpkg_capture_git_archives.sh` | Regenerates that pin with `vcpkg install --only-downloads` (~6 min, no compilation, no CMake) — vcpkg as the instrument, since which git externals are used is decided by feature-conditional CMake code, not by portfile text | +| `hsts_preload.bzl` | Chromium's HSTS preload table as one `http_file`, pinned to a **commit** + sha256 — the downstream pin for the one input upstream CMake fetches from `main`. **Generated** by `Meta/pin_hsts_preload.py` | +| `Meta/pin_hsts_preload.py` | Re-pins it: resolves the newest commit touching the path, downloads it, writes the hash it **measured**; `--expect-same-as` refuses to write unless the pinned bytes equal the file CMake downloaded (parity guard) | | `bazelrc.txt` | → `.bazelrc`. Global copts/defines/linkopts mirrored from `Meta/CMake/compile_options.cmake` | | `BUILD.bazel` | Root package: 34 libraries, the 5 executables, Qt moc/rcc genrules. **Generated** by `Meta/emit_build_bazel.py` | | `codegen_root.bzl` | Non-LibWeb generator genrules (IPC endpoints, LibJS Bytecode/Op, HSTS table, WebGL replayer, TIFF tag tables, the two SPIR-V shader headers, and the chained `generate_interpreter_layout` → `flapc` interpreter assembly). **Generated** by `Meta/emit_root_codegen_bazel.py` | @@ -206,9 +246,10 @@ adds `-I/usr/include/libdrm`), which is gap 3 below, not an emitter bug. To **build the browser**, no CMake build is needed — Bazel fetches and builds the vcpkg *ports* and the Rust crates itself, and vcpkg's own downloader reaches the -network zero times. Three inputs still have to be present first (step 1a; gaps 7–8) -— two of them obtainable **without CMake**, the third (the HSTS table) unpinnable -until fixed upstream, so "no network at all" is not yet true: +network zero times. **Two** inputs still have to be present first (step 1a; gaps +7–8), both obtainable **without CMake**. The third, the HSTS preload table, is now +fetched by Bazel: it is pinned downstream to a Chromium commit + sha256 in +[`hsts_preload.bzl`](workspace/hsts_preload.bzl): ```sh git clone https://github.com/LadybirdBrowser/ladybird && cd ladybird @@ -216,16 +257,12 @@ git clone https://github.com/LadybirdBrowser/ladybird && cd ladybird cp -r .../examples/ladybird/workspace/. . && mv bazelrc.txt .bazelrc git apply .../examples/ladybird/patches/*.patch # generator determinism + a Qt header # that is not self-contained (gap 9) -# 1a. THE THREE INPUTS THE OVERLAY DOES NOT CARRY -- all three are produced by -# ONE ordinary Ladybird build. A Ladybird developer stages nothing by hand; +# 1a. THE TWO INPUTS THE OVERLAY DOES NOT CARRY -- both are produced by ONE +# ordinary Ladybird build. A Ladybird developer stages nothing by hand; # these are blockers for a Bazel-ONLY clone, and each one is a thing CMake or # vcpkg produces as a side effect (see finding 36): # (a) Build/vcpkg + its .git <- python3 Meta/ladybird.py vcpkg (~70s) -# (b) Build/caches/HSTSPreload/transport_security_state_static.json -# <- the CMake CONFIGURE downloads it -# (Meta/CMake/hsts_preload.cmake, gated on -# ENABLE_NETWORK_DOWNLOADS, default ON) -# (c) the four Meta/CMake/vcpkg/git-archives/*.tar.gz +# (b) the four Meta/CMake/vcpkg/git-archives/*.tar.gz # <- vcpkg writes them into # Build/vcpkg/downloads/ while building # skia and angle. Verified: those files' @@ -235,16 +272,15 @@ git apply .../examples/ladybird/patches/*.patch # generator determinism + a Qt # vcpkg's own cache. # WITHOUT CMAKE (the supported path for a Bazel-only clone): python3 Meta/ladybird.py vcpkg # (a) checkout + .git, ~70s, no CMake -python3 Meta/fetch_vcpkg_git_archives.py # (c) 4/4 reproduced with git archive and +python3 Meta/fetch_vcpkg_git_archives.py # (b) 4/4 reproduced with git archive and # verified against the pinned SHA512s -curl -Lo Build/caches/HSTSPreload/transport_security_state_static.json \ - https://raw.githubusercontent.com/chromium/chromium/main/net/http/transport_security_state_static.json -# # (b) unpinnable until fixed upstream -# WITH CMake, if you were building Ladybird anyway, all three fall out of: +# The HSTS preload table needs NOTHING now: Bazel fetches it as +# @hsts_preload_json//file, pinned to a Chromium commit + sha256 in +# hsts_preload.bzl (below). Ditto `ply`: the wheel is pinned and pip runs +# with PIP_NO_INDEX (gap 8). +# WITH CMake, if you were building Ladybird anyway, (a) and (b) fall out of: # ./Meta/ladybird.py build # cp Build/vcpkg/downloads/{angle,libyuv,skia}-*.tar.gz Meta/CMake/vcpkg/git-archives/ -# `ply` needs nothing now: the wheel is pinned and pip runs with -# PIP_NO_INDEX (gap 8). # # NB vcpkg's buildtrees peak around 3 GB. They go next to the declared output # (inside bazel-out) rather than $TMPDIR, so a small /tmp tmpfs is not a @@ -552,7 +588,7 @@ Honest inventory of what stops this from being a clone-and-build. patches). The BCR's `rules_qt` module is Vertexwahn's unrelated `rules_qt6`. Qt itself *is* host-portable: `qt.local_repo` discovers the host Qt via `qmake -query`, so no Qt SDK is vendored. -7. **The fresh clone needs three inputs the overlay does not carry** (step 1a +7. **The fresh clone needs two inputs the overlay does not carry** (step 1a above), and each stayed invisible for the same reason: it was already on the machine. `Build/vcpkg` — a microsoft/vcpkg checkout at `vcpkg.json`'s `builtin-baseline`, created by Ladybird's own `Meta/ladybird.py vcpkg` — is @@ -560,17 +596,22 @@ Honest inventory of what stops this from being a clone-and-build. matches one file and the failure names a missing `/tmp/.../root/vcpkg`. Its `.git` is *load-bearing* (vcpkg resolves versioned ports with `git read-tree`) yet the filegroup **excludes** it — an undeclared input the build reads anyway, - because the action is `no-sandbox`. And - `Build/caches/HSTSPreload/transport_security_state_static.json` is an - *unversioned* download from Chromium's `main` branch that CMake does at - configure time; `codegen_root.bzl` names it as a genrule input, so Bazel fails - cleanly, but nothing produces it. - - The honest fix for all three is the same shape as findings 30–33: pin them and - let Bazel fetch them. The vcpkg checkout is a `git_repository` at the - `builtin-baseline` commit (which `vcpkg.json` already names, so the pin exists - — it just is not wired to Bazel); the HSTS table is an `http_file`, and needs a - hash Ladybird does not currently pin at all, because the URL tracks `main`. + because the action is `no-sandbox`. The four `vcpkg_from_git` tarballs are the + other, reproduced by `Meta/fetch_vcpkg_git_archives.py`. Both are prefetches, + not rules: a `git_repository` strips `.git`, and `git archive` output has no URL + to `http_file`. + + **The third one — the HSTS table — is now closed.** It was the interesting case, + because the unpinned fetch is *upstream's* (`hsts_preload.cmake` tracks Chromium's + `main`) and we cannot change that. So it is pinned **downstream**: an `http_file` + at an immutable commit + `sha256` in `hsts_preload.bzl`, regenerated by + `Meta/pin_hsts_preload.py`, verified byte-identical to CMake's generated table + with the staged file deleted, and the upstream unpinned fetch filed as a bug. The + generalizable shape: **a converter cannot pin an input on the foreign build + system's behalf, but it can pin it for itself — provided it pins the revision the + foreign system is currently serving, and proves that with a byte comparison + rather than a hash it invented.** Pin the wrong revision (a release tag, say) and + the hermeticity gap becomes a parity gap. 8. **"Zero network access" is true of vcpkg's downloader, not of the vcpkg action.** The pin + `x-block-origin` is verified — a distfile missing from the diff --git a/examples/ladybird/workspace/MODULE.bazel b/examples/ladybird/workspace/MODULE.bazel index d9918bd..446f4e7 100644 --- a/examples/ladybird/workspace/MODULE.bazel +++ b/examples/ladybird/workspace/MODULE.bazel @@ -30,6 +30,16 @@ qt.active_sdk(name = "qt", repo = "qt6_local") use_repo(qt, "qt") register_toolchains("@qt//:qt_linux_x86_64_toolchain") +# --------------------------------------------------------------------------- +# Chromium's HSTS preload table: the one input CMake downloads UNPINNED (from +# `main`, at configure time) and the one thing that made a Bazel-only clone +# non-hermetic. Pinned downstream to a commit + sha256 in hsts_preload.bzl -- +# which also records why the commit is that one and not a release tag -- and +# regenerated by Meta/pin_hsts_preload.py. Consumed by :gen_HSTSPreloadData in +# codegen_root.bzl as @hsts_preload_json//file. +hsts = use_extension("//:hsts_preload.bzl", "hsts_preload") +use_repo(hsts, "hsts_preload_json") + # --------------------------------------------------------------------------- # Ring 2: the vcpkg dependency tree. Bazel owns FETCHING (here); vcpkg is still # the recipe that builds them (//:vcpkg_installed, see vcpkg.bzl). diff --git a/examples/ladybird/workspace/Meta/emit_root_codegen_bazel.py b/examples/ladybird/workspace/Meta/emit_root_codegen_bazel.py index 26cc40b..d89b597 100644 --- a/examples/ladybird/workspace/Meta/emit_root_codegen_bazel.py +++ b/examples/ladybird/workspace/Meta/emit_root_codegen_bazel.py @@ -169,9 +169,28 @@ def _rel_full(p): # referenced as //pkg:path, not as a root-package-relative path. SUBPACKAGES = ("Libraries/LibWeb", "Meta") +# Generator inputs that are NOT in the repo: CMake downloads them at configure +# time into the build tree, so a Bazel-only clone has no file at the path the +# ninja command line names. Each maps to the label of a pinned fetch. +# +# There is exactly one, and it is the HSTS preload table: +# hsts_preload.cmake downloads Chromium's `main` -- unversioned -- so the path +# below only exists if someone ran a CMake configure, and its CONTENT depends on +# the day they ran it. hsts_preload.bzl pins a commit + sha256 downstream (the +# upstream unpinned fetch is filed as a bug we cannot fix from here), so the +# genrule takes the pinned file instead of the configure's leftovers. Mapping it +# here rather than post-editing codegen_root.bzl keeps the emitter the single +# source of truth for that file. +DOWNLOADED_INPUTS = { + 'Build/caches/HSTSPreload/transport_security_state_static.json': + '@hsts_preload_json//file', +} + def _label(rel): """Root-relative repo path -> a label the root package can reference.""" + if rel in DOWNLOADED_INPUTS: + return DOWNLOADED_INPUTS[rel] for pkg in SUBPACKAGES: if rel == pkg or rel.startswith(pkg + '/'): return '//%s:%s' % (pkg, os.path.relpath(rel, pkg)) diff --git a/examples/ladybird/workspace/Meta/pin_hsts_preload.py b/examples/ladybird/workspace/Meta/pin_hsts_preload.py new file mode 100644 index 0000000..7df6a55 --- /dev/null +++ b/examples/ladybird/workspace/Meta/pin_hsts_preload.py @@ -0,0 +1,189 @@ +#!/usr/bin/env python3 +"""Re-pin Chromium's HSTS preload table, and write hsts_preload.bzl. + +CMake fetches net/http/transport_security_state_static.json from Chromium's +**main** at configure time (Meta/CMake/hsts_preload.cmake), unversioned. That is +an upstream defect we cannot fix from the overlay, so the overlay pins it +downstream: one immutable commit URL plus the sha256 this script MEASURED by +downloading it. + +Two properties this script exists to preserve, both of them learned the hard way: + + * Pin a COMMIT, not a release tag. The tag 139.0.7258.5 serves an 18.7 MB file + generating 168,593 entries; the commit main pointed at when the reference + CMake build configured serves 10.5 MB and 94,626. A tag is a pin, but a pin + to a *different table* -- so pinning to one trades a hermeticity gap for a + parity gap. `--commit` defaults to the newest commit touching the path, which + is the one `main` serves. + * Never write a hash you did not measure. The sha256 in the output is computed + from the bytes downloaded from the URL being pinned. `--expect-sha256` + additionally asserts the pinned bytes equal a file you already have (the one + CMake downloaded), which is how the current pin was shown to keep byte-parity + with the reference build. + +Usage: + # re-pin to the newest commit touching the path + python3 Meta/pin_hsts_preload.py > hsts_preload.bzl + + # re-pin, and refuse unless it matches what CMake already downloaded + python3 Meta/pin_hsts_preload.py \ + --expect-same-as Build/caches/HSTSPreload/transport_security_state_static.json \ + > hsts_preload.bzl + + # just report what the pin would be, without writing + python3 Meta/pin_hsts_preload.py --check +""" +import argparse +import hashlib +import json +import shutil +import subprocess +import sys + +PATH = "net/http/transport_security_state_static.json" +COMMITS_API = "https://api.github.com/repos/chromium/chromium/commits?path={path}&per_page=1" +COMMIT_API = "https://api.github.com/repos/chromium/chromium/commits/{commit}" +RAW_URL = "https://raw.githubusercontent.com/chromium/chromium/{commit}/" + PATH + +TEMPLATE = '''\ +# Chromium's HSTS preload table, pinned to a commit. GENERATED by +# Meta/pin_hsts_preload.py -- see that script to re-pin; do not hand-edit. +# +# WHY THIS FILE EXISTS. `Meta/CMake/hsts_preload.cmake` downloads +# net/http/transport_security_state_static.json from Chromium's **main** at +# configure time -- an unversioned ref. The generator turns it into a ~95,000 +# entry `constexpr Array` of domains LibHTTP forces to HTTPS, so "whatever main +# served the day you configured" decides a security-relevant table, and two +# developers who configured on different days build different browsers. +# +# The fix belongs upstream (pin the URL in hsts_preload.cmake). We cannot make +# that change from here, so we pin DOWNSTREAM: Bazel fetches one immutable +# commit URL with a sha256, and the upstream unpinned fetch is filed as a bug. +# +# Pinning downstream is only honest if it does not break byte-parity with CMake, +# and this pin does not: the bytes this commit serves are byte-identical to the +# ones CMake's `main` fetch downloaded for the reference build ({size:,} bytes, +# checked with --expect-same-as, which refuses to write this file otherwise). +# Pinning to a Chromium *release tag* would NOT have that property: 139.0.7258.5 +# serves 18.7 MB and generates 168,593 entries against this commit's 94,626. +# +# Two consequences worth knowing: +# +# * CMake still tracks `main`, so a CMake configure NEWER than this pin will +# disagree with Bazel. That is now a visible, dated disagreement between one +# pinned input and one unpinned one, instead of two unpinned fetches that +# happened to agree. `download_file` is a no-op when the file already exists +# (verified with ENABLE_NETWORK_DOWNLOADS=OFF), so staging the Bazel-fetched +# file into Build/caches/HSTSPreload/ before configuring makes CMake consume +# this same pin -- see README, "Getting the three inputs". +# * Bumping the pin is a deliberate, reviewable act: re-run +# Meta/pin_hsts_preload.py, which resolves the newest commit touching the +# path, downloads it, and writes this file with the hash it measured. +load("@bazel_tools//tools/build_defs/repo:http.bzl", "http_file") + +# The pinned commit: "{subject}" ({date}). +# raw.githubusercontent.com paths are immutable for a full commit sha, unlike the +# `main` path CMake uses. +HSTS_PRELOAD_COMMIT = "{commit}" + +HSTS_PRELOAD_SHA256 = "{sha256}" + +HSTS_PRELOAD_URL = "https://raw.githubusercontent.com/chromium/chromium/{{}}/net/http/transport_security_state_static.json".format( + HSTS_PRELOAD_COMMIT, +) + +def _hsts_preload_impl(_ctx): + http_file( + name = "hsts_preload_json", + urls = [HSTS_PRELOAD_URL], + sha256 = HSTS_PRELOAD_SHA256, + downloaded_file_path = "transport_security_state_static.json", + ) + +hsts_preload = module_extension(implementation = _hsts_preload_impl) +''' + + +def _get(url, timeout): + """GET url, as bytes. + + Shells out to curl rather than using urllib: this runs behind + TLS-intercepting proxies (urllib rejected ours with CERTIFICATE_VERIFY_FAILED + while curl, which reads the system CA bundle the same way git and Bazel do, + succeeded). A pin script that cannot run in the environment the build runs in + is not a pin script. + """ + if shutil.which("curl") is None: + sys.exit("curl not found; it is how this script reaches the network") + p = subprocess.run(["curl", "-sSL", "--fail", "--max-time", str(timeout), url], + stdout=subprocess.PIPE, stderr=subprocess.PIPE) + if p.returncode != 0: + sys.exit("fetch failed for %s: %s" % (url, p.stderr.decode().strip())) + return p.stdout + + +def newest_commit(): + """The newest commit touching the path -- i.e. what `main` currently serves.""" + commits = json.loads(_get(COMMITS_API.format(path=PATH), 60)) + if not commits: + sys.exit("no commits returned for %s" % PATH) + c = commits[0] + return c["sha"], c["commit"]["message"].splitlines()[0], c["commit"]["committer"]["date"][:10] + + +def describe(commit): + """Subject line + date for a commit, so the pin records WHAT it pinned.""" + c = json.loads(_get(COMMIT_API.format(commit=commit), 60)) + return (c["commit"]["message"].splitlines()[0], + c["commit"]["committer"]["date"][:10]) + + +def fetch(commit): + blob = _get(RAW_URL.format(commit=commit), 300) + return blob, hashlib.sha256(blob).hexdigest() + + +def main(argv=None): + ap = argparse.ArgumentParser(description=__doc__.splitlines()[0]) + ap.add_argument("--commit", help="pin this commit instead of the newest one touching the path") + ap.add_argument("--expect-same-as", metavar="FILE", + help="fail unless the pinned bytes equal FILE (e.g. the file CMake downloaded)") + ap.add_argument("--check", action="store_true", + help="report the pin on stderr and write nothing") + args = ap.parse_args(argv) + + if args.commit: + commit = args.commit + subject, date = describe(commit) + else: + commit, subject, date = newest_commit() + + blob, sha256 = fetch(commit) + + if args.expect_same_as: + want = open(args.expect_same_as, "rb").read() + if want != blob: + sys.exit( + "PARITY: the pinned bytes differ from %s (%d vs %d bytes).\n" + "Chromium's main has moved since that file was downloaded. Pinning this\n" + "commit would make Bazel's generated table differ from CMake's; either\n" + "re-run CMake's configure (or stage the pinned file into\n" + "Build/caches/HSTSPreload/), or pin the older commit with --commit." + % (args.expect_same_as, len(want), len(blob)) + ) + print("parity OK: pinned bytes identical to %s" % args.expect_same_as, file=sys.stderr) + + print("commit %s %s %s %d bytes sha256 %s" + % (commit[:12], date, subject, len(blob), sha256), file=sys.stderr) + if args.check: + return 0 + + sys.stdout.write(TEMPLATE.format( + commit=commit, sha256=sha256, size=len(blob), + subject=subject, date=date, + )) + return 0 + + +if __name__ == "__main__": + sys.exit(main()) diff --git a/examples/ladybird/workspace/codegen_root.bzl b/examples/ladybird/workspace/codegen_root.bzl index c0fa128..6bb9ed4 100644 --- a/examples/ladybird/workspace/codegen_root.bzl +++ b/examples/ladybird/workspace/codegen_root.bzl @@ -7,9 +7,9 @@ load("@rules_cc//cc:defs.bzl", "cc_library") def root_codegen(): native.genrule( name = 'gen_HSTSPreloadData', - srcs = ['Build/caches/HSTSPreload/transport_security_state_static.json'] + ["//Meta:generators"], + srcs = ['@hsts_preload_json//file'] + ["//Meta:generators"], outs = ['Libraries/LibHTTP/HSTSPreloadData.h', 'Libraries/LibHTTP/HSTSPreloadData.cpp'], - cmd = "PYTHONHASHSEED=0 python3 Meta/Generators/generate_hsts_preload_data.py -h $(location Libraries/LibHTTP/HSTSPreloadData.h) -c $(location Libraries/LibHTTP/HSTSPreloadData.cpp) -p $(location Build/caches/HSTSPreload/transport_security_state_static.json)", + cmd = "PYTHONHASHSEED=0 python3 Meta/Generators/generate_hsts_preload_data.py -h $(location Libraries/LibHTTP/HSTSPreloadData.h) -c $(location Libraries/LibHTTP/HSTSPreloadData.cpp) -p $(location @hsts_preload_json//file)", ) native.genrule( name = 'gen_Op', diff --git a/examples/ladybird/workspace/hsts_preload.bzl b/examples/ladybird/workspace/hsts_preload.bzl new file mode 100644 index 0000000..8aa5eb7 --- /dev/null +++ b/examples/ladybird/workspace/hsts_preload.bzl @@ -0,0 +1,55 @@ +# Chromium's HSTS preload table, pinned to a commit. GENERATED by +# Meta/pin_hsts_preload.py -- see that script to re-pin; do not hand-edit. +# +# WHY THIS FILE EXISTS. `Meta/CMake/hsts_preload.cmake` downloads +# net/http/transport_security_state_static.json from Chromium's **main** at +# configure time -- an unversioned ref. The generator turns it into a ~95,000 +# entry `constexpr Array` of domains LibHTTP forces to HTTPS, so "whatever main +# served the day you configured" decides a security-relevant table, and two +# developers who configured on different days build different browsers. +# +# The fix belongs upstream (pin the URL in hsts_preload.cmake). We cannot make +# that change from here, so we pin DOWNSTREAM: Bazel fetches one immutable +# commit URL with a sha256, and the upstream unpinned fetch is filed as a bug. +# +# Pinning downstream is only honest if it does not break byte-parity with CMake, +# and this pin does not: the bytes this commit serves are byte-identical to the +# ones CMake's `main` fetch downloaded for the reference build (10,521,748 bytes, +# checked with --expect-same-as, which refuses to write this file otherwise). +# Pinning to a Chromium *release tag* would NOT have that property: 139.0.7258.5 +# serves 18.7 MB and generates 168,593 entries against this commit's 94,626. +# +# Two consequences worth knowing: +# +# * CMake still tracks `main`, so a CMake configure NEWER than this pin will +# disagree with Bazel. That is now a visible, dated disagreement between one +# pinned input and one unpinned one, instead of two unpinned fetches that +# happened to agree. `download_file` is a no-op when the file already exists +# (verified with ENABLE_NETWORK_DOWNLOADS=OFF), so staging the Bazel-fetched +# file into Build/caches/HSTSPreload/ before configuring makes CMake consume +# this same pin -- see README, "Getting the three inputs". +# * Bumping the pin is a deliberate, reviewable act: re-run +# Meta/pin_hsts_preload.py, which resolves the newest commit touching the +# path, downloads it, and writes this file with the hash it measured. +load("@bazel_tools//tools/build_defs/repo:http.bzl", "http_file") + +# The pinned commit: "[HSTS] Update bulk entries" (2026-07-24). +# raw.githubusercontent.com paths are immutable for a full commit sha, unlike the +# `main` path CMake uses. +HSTS_PRELOAD_COMMIT = "3d75766484199c1fbefd269a4b168cccdb36fbca" + +HSTS_PRELOAD_SHA256 = "59c3301277b3418e716d818d26110edb0737edb871c5bba7462158da1b2d1bd9" + +HSTS_PRELOAD_URL = "https://raw.githubusercontent.com/chromium/chromium/{}/net/http/transport_security_state_static.json".format( + HSTS_PRELOAD_COMMIT, +) + +def _hsts_preload_impl(_ctx): + http_file( + name = "hsts_preload_json", + urls = [HSTS_PRELOAD_URL], + sha256 = HSTS_PRELOAD_SHA256, + downloaded_file_path = "transport_security_state_static.json", + ) + +hsts_preload = module_extension(implementation = _hsts_preload_impl) diff --git a/tests/test_pin_hsts_preload.py b/tests/test_pin_hsts_preload.py new file mode 100644 index 0000000..598d768 --- /dev/null +++ b/tests/test_pin_hsts_preload.py @@ -0,0 +1,260 @@ +#!/usr/bin/env python3 +"""Tests for examples/ladybird/workspace/Meta/pin_hsts_preload.py. + +The script pins Chromium's HSTS preload table DOWNSTREAM: upstream's CMake +downloads it from `main` (unversioned) and we cannot change that, so Bazel fetches +one immutable commit URL with a sha256 instead. What is worth testing is not the +HTTP (that needs the network) but the three properties that make such a pin honest, +each of which was a way to get it wrong: + + 1. the sha256 written out is MEASURED from the bytes fetched, never passed in; + 2. `--expect-same-as` is a real parity guard: if the pinned bytes differ from the + file the other build system already downloaded, the script must refuse to + write rather than emit a pin that silently changes the generated table (the + concrete failure it exists to prevent: a Chromium *release tag* serves a file + generating 168,593 entries against `main`'s ~94,600); + 3. `--check` reports without writing, so re-pinning is a deliberate act. + +Plus the trivial-but-load-bearing one: the emitted file must be valid Starlark-ish +text carrying the commit, the hash and an immutable (commit-pinned, not `main`) URL. +""" + +import importlib.util +import io +import sys +import tempfile +from pathlib import Path + +REPO = Path(__file__).resolve().parent.parent +SCRIPT = REPO / "examples" / "ladybird" / "workspace" / "Meta" / "pin_hsts_preload.py" +PINNED_BZL = REPO / "examples" / "ladybird" / "workspace" / "hsts_preload.bzl" + +# A commit sha and a payload standing in for the 10 MB table. +COMMIT = "3d75766484199c1fbefd269a4b168cccdb36fbca" +BLOB = b'{"entries": [{"name": "example.test", "policy": "bulk-18-weeks"}]}\n' + + +class expect_exit: + """Minimal assertRaises(SystemExit) -- this repo's tests are plain functions.""" + + def __enter__(self): + return self + + def __exit__(self, exc_type, exc, tb): + assert exc_type is SystemExit, "expected SystemExit, got %r" % (exc_type,) + self.exception = exc + return True + + +def load(responses): + """Import the script with its single network entry point stubbed. + + `responses` maps a substring of the URL -> bytes to return, so a test can serve + the commits API and the raw file differently without knowing the URL shapes. + """ + spec = importlib.util.spec_from_file_location("pin_hsts_%d" % id(responses), SCRIPT) + mod = importlib.util.module_from_spec(spec) + spec.loader.exec_module(mod) + + calls = [] + + def fake_get(url, timeout): + calls.append(url) + for needle, payload in responses.items(): + if needle in url: + return payload + raise AssertionError("unstubbed URL: %s" % url) + + mod._get = fake_get + mod.calls = calls + return mod + + +def run(mod, argv): + """Run main() capturing stdout, the way the caller redirects it into the .bzl.""" + out, err = io.StringIO(), io.StringIO() + old_out, old_err = sys.stdout, sys.stderr + sys.stdout, sys.stderr = out, err + try: + rc = mod.main(argv) + finally: + sys.stdout, sys.stderr = old_out, old_err + return rc, out.getvalue(), err.getvalue() + + +def _commit_object(): + return { + "sha": COMMIT, + "commit": { + "message": "[HSTS] Update bulk entries\n\nbody text", + "committer": {"date": "2026-07-24T21:31:29Z"}, + }, + } + + +def _responses(): + """Stubs for the three endpoints the script touches. + + The two GitHub URLs differ only after `commits`: `commits?path=...` LISTS + (a JSON array) while `commits/` DESCRIBES one (a JSON object). Keying the + stubs on that distinction is deliberate -- an earlier version of this fixture + served the array for both and the --commit path failed with a TypeError, which + is exactly the confusion a reader of the script could make too. + """ + import json + return { + "commits?path=": json.dumps([_commit_object()]).encode(), + "commits/" + COMMIT: json.dumps(_commit_object()).encode(), + "raw.githubusercontent.com": BLOB, + } + + +def test_hash_is_measured_not_asserted(): + """The emitted sha256 must be of the bytes actually fetched.""" + import hashlib + + mod = load(_responses()) + rc, out, _ = run(mod, []) + assert rc == 0 + assert hashlib.sha256(BLOB).hexdigest() in out, "the pin must carry the MEASURED hash" + + +def test_pins_a_commit_url_not_main(): + """A pin to `main` is not a pin: the URL must carry the full commit sha.""" + mod = load(_responses()) + _, out, _ = run(mod, []) + assert 'HSTS_PRELOAD_COMMIT = "%s"' % COMMIT in out + assert "/main/net/http/" not in out, "the emitted URL must not track main" + assert "net/http/transport_security_state_static.json" in out + + +def test_emitted_file_declares_the_http_file_and_extension(): + """The output has to be usable: an http_file inside a module extension.""" + mod = load(_responses()) + _, out, _ = run(mod, []) + for expected in ("http_file(", 'name = "hsts_preload_json"', "sha256 = HSTS_PRELOAD_SHA256", + "hsts_preload = module_extension("): + assert expected in out, "emitted file is missing %r" % expected + + +def test_records_what_it_pinned(): + """A reviewer needs the subject+date of the commit, not just a sha.""" + mod = load(_responses()) + _, out, _ = run(mod, []) + assert "[HSTS] Update bulk entries" in out + assert "2026-07-24" in out + + +def test_explicit_commit_is_described_not_guessed(): + """--commit must still look up the subject/date, so the pin stays self-documenting.""" + mod = load(_responses()) + _, out, _ = run(mod, ["--commit", COMMIT]) + assert "[HSTS] Update bulk entries" in out + assert any(("commits/" + COMMIT) in u for u in mod.calls), \ + "--commit should still describe the commit it pinned" + + +def test_expect_same_as_accepts_identical_bytes(): + """The parity guard passes when the pinned bytes equal the file CMake downloaded.""" + mod = load(_responses()) + with tempfile.NamedTemporaryFile(suffix=".json") as f: + f.write(BLOB) + f.flush() + rc, out, err = run(mod, ["--expect-same-as", f.name]) + assert rc == 0 + assert "parity OK" in err + assert "HSTS_PRELOAD_SHA256" in out + + +def test_expect_same_as_refuses_different_bytes(): + """The case this guard exists for: pinning a revision that changes the table.""" + mod = load(_responses()) + with tempfile.NamedTemporaryFile(suffix=".json") as f: + f.write(BLOB + b"an extra entry, i.e. a different table\n") + f.flush() + with expect_exit() as e: + run(mod, ["--expect-same-as", f.name]) + msg = str(e.exception) + assert "PARITY" in msg + assert "differ" in msg + # and it must say what to do about it, not just that it failed + assert "--commit" in msg + + +def test_expect_same_as_writes_nothing_on_failure(): + """A refused pin must not emit a partial .bzl the caller would redirect into place.""" + mod = load(_responses()) + with tempfile.NamedTemporaryFile(suffix=".json") as f: + f.write(b"totally different\n") + f.flush() + out, err = io.StringIO(), io.StringIO() + old = sys.stdout, sys.stderr + sys.stdout, sys.stderr = out, err + try: + with expect_exit(): + mod.main(["--expect-same-as", f.name]) + finally: + sys.stdout, sys.stderr = old + assert "http_file(" not in out.getvalue(), "must not emit a pin it refused" + + +def test_check_reports_without_writing(): + """--check is the dry run: the pin is reported on stderr, stdout stays empty.""" + mod = load(_responses()) + rc, out, err = run(mod, ["--check"]) + assert rc == 0 + assert out == "", "--check must write no .bzl" + assert COMMIT[:12] in err and "sha256" in err + + +def test_no_commits_for_the_path_is_an_error(): + """If GitHub returns nothing, fail loudly rather than pin the empty string.""" + mod = load({"commits?path=": b"[]", "raw.githubusercontent.com": BLOB}) + with expect_exit() as e: + run(mod, []) + assert "no commits" in str(e.exception) + + +def test_committed_pin_matches_the_scripts_own_template(): + """The committed hsts_preload.bzl must be what this script would write. + + Guards the hand-edit: hsts_preload.bzl carries a long comment explaining the + pin, and the temptation is to tweak it in place until it drifts from the + generator. Rendering the template with the committed values must reproduce the + committed file exactly. + """ + text = PINNED_BZL.read_text() + import re + + commit = re.search(r'HSTS_PRELOAD_COMMIT = "([0-9a-f]{40})"', text).group(1) + sha256 = re.search(r'HSTS_PRELOAD_SHA256 = "([0-9a-f]{64})"', text).group(1) + subject = re.search(r'# The pinned commit: "(.*)" \((\d{4}-\d\d-\d\d)\)\.', text) + size = int(re.search(r'reference build \(([\d,]+) bytes', text).group(1).replace(",", "")) + + mod = load(_responses()) + rendered = mod.TEMPLATE.format( + commit=commit, sha256=sha256, size=size, + subject=subject.group(1), date=subject.group(2), + ) + assert rendered == text, "hsts_preload.bzl has drifted from Meta/pin_hsts_preload.py" + + +def test_codegen_genrule_consumes_the_pinned_file(): + """The pin is only useful if the generator actually reads it. + + codegen_root.bzl is generated, so this asserts the wiring survived a + regeneration: gen_HSTSPreloadData must take @hsts_preload_json//file and must + NOT reference the CMake configure's download path under Build/caches. + """ + bzl = (REPO / "examples" / "ladybird" / "workspace" / "codegen_root.bzl").read_text() + rule = bzl.split("name = 'gen_HSTSPreloadData'", 1)[1].split("native.genrule(", 1)[0] + assert "@hsts_preload_json//file" in rule + assert "Build/caches/HSTSPreload" not in bzl, \ + "the unpinned CMake download path must be gone from the generated file" + + +def test_module_bazel_names_the_repo(): + """bzlmod requires every extension-created repo in a use_repo, or it is invisible.""" + mod_bazel = (REPO / "examples" / "ladybird" / "workspace" / "MODULE.bazel").read_text() + assert 'use_extension("//:hsts_preload.bzl", "hsts_preload")' in mod_bazel + assert 'use_repo(hsts, "hsts_preload_json")' in mod_bazel diff --git a/tests/test_vcpkg_plumbing.py b/tests/test_vcpkg_plumbing.py index cdb9263..68dd452 100644 --- a/tests/test_vcpkg_plumbing.py +++ b/tests/test_vcpkg_plumbing.py @@ -218,20 +218,33 @@ def test_the_network_claim_is_scoped_to_vcpkgs_own_downloader(): "requires-network is still presented as if it blocked the network" -def test_the_unpinned_hsts_download_is_declared_and_documented(): - """An input with no producing rule, and no hash anywhere. - - codegen_root.bzl names Build/caches/HSTSPreload/...json as a genrule src, so - Bazel fails cleanly with `missing input file` rather than generating an empty - table. But nothing produces it: CMake downloads it at configure time from - Chromium's `main` branch, UNVERSIONED -- so there is no revision to pin even if - we wanted to, which is the more interesting half of the problem. +def test_the_hsts_download_is_pinned_downstream_and_documented(): + """The input upstream fetches unpinned, pinned on our side instead. + + CMake downloads the HSTS preload table from Chromium's `main` at configure time + -- unversioned -- and that is upstream's code, which we do not control. So the + overlay pins it DOWNSTREAM: an http_file at an immutable commit + sha256, which + the generator genrule consumes instead of the configure's leftovers under + Build/caches. Three things have to hold together or the pin is decoration: + the pin exists with a full commit sha (not `main`), the genrule reads it, and + the reason it is a commit rather than a release tag is written down (a tag + serves a different table, so pinning one would trade hermeticity for parity). """ + pin = _read("hsts_preload.bzl") + assert re.search(r'HSTS_PRELOAD_COMMIT = "[0-9a-f]{40}"', pin), \ + "the HSTS pin must name a full commit sha" + assert re.search(r'HSTS_PRELOAD_SHA256 = "[0-9a-f]{64}"', pin) + assert "/main/net/http/" not in pin, "the pinned URL must not track main" + codegen = _read("codegen_root.bzl") - assert "Build/caches/HSTSPreload" in codegen + assert "@hsts_preload_json//file" in codegen, \ + "the generator genrule must consume the pinned file" + assert "Build/caches/HSTSPreload" not in codegen, \ + "the unpinned CMake download path must be gone" + readme = _example_readme() - assert "HSTSPreload" in readme and "unversioned" in readme.lower(), \ - "the unpinned HSTS download is not documented as a clone blocker" + assert "hsts_preload.bzl" in readme and "unversioned" in readme.lower(), \ + "why the HSTS table needed a downstream pin is not documented" def _example_readme(): From bff5c5440276633ad670b4cebb83a56d11f9aa52 Mon Sep 17 00:00:00 2001 From: Ulf Adams Date: Mon, 10 Aug 2026 19:27:24 +0000 Subject: [PATCH 17/59] Ladybird: make the tree reproducible elsewhere -- one script, verified by running it Getting the overlay onto another machine was 'cp -r workspace/. ladybird/', and running that on an empty directory found four defects. Nothing recorded WHICH Ladybird commit the generated BUILD files describe (they name ~1,961 sources by path, so every parity claim was relative to an unnamed commit -- f9e34731, now pinned in one place). bazelrc.txt must be renamed. The two patches must be applied, not just shipped. The fourth was invisible from either side alone: Build/vcpkg/BUILD.bazel makes the DIRECTORY Build/vcpkg exist, and upstream's build_vcpkg.py infers 'already cloned' from is_dir(), then runs `git -C Build/vcpkg rev-parse HEAD` -- which, with no .git inside, walks up to Ladybird's own repo and returns Ladybird's HEAD. It then tries to check vcpkg's baseline out of the Ladybird repo: fatal: unable to read tree. Two correct programs, one wrong composition. apply_overlay.sh defers that one file past the prefetch and names the failure where someone hitting it will look. --verify checks what a copy cannot: HEAD is the pinned commit, all 42 files byte-identical, patches APPLIED (git apply --check -R succeeding is the proof), and the .sh files still executable -- the last because scripts once sat 100644 in git while my tree had them +x, so only a fresh clone failed, and only at action time. Verified end to end: apply_overlay.sh on an empty dir produced a tree byte-identical to the one that renders (diff -rq), fetch_vcpkg_git_archives.py reproduced 4/4 against the pinned SHA512s, and bazel build built the 76 vcpkg ports offline and linked //:LibHTTP (RC=0, 242 actions) with HSTSPreloadData.{h,cpp} byte-identical to the CMake reference. 12 new tests, 176/176 -- two of which caught real defects in this change. --- docs/CASE-ladybird-migration.md | 55 +++++++ examples/ladybird/README.md | 57 +++++++ examples/ladybird/apply_overlay.sh | 240 +++++++++++++++++++++++++++++ tests/test_apply_overlay.py | 170 ++++++++++++++++++++ 4 files changed, 522 insertions(+) create mode 100755 examples/ladybird/apply_overlay.sh create mode 100644 tests/test_apply_overlay.py diff --git a/docs/CASE-ladybird-migration.md b/docs/CASE-ladybird-migration.md index 39d283e..4025a94 100644 --- a/docs/CASE-ladybird-migration.md +++ b/docs/CASE-ladybird-migration.md @@ -2091,6 +2091,61 @@ dependency. Getting this right is what `builtin-baseline` + `overrides` means, a is why the answer to "how do we get it" keeps coming back to *run the project's own bootstrap*. +## Finding 37: the overlay was not reproducible, and the thing that proved it was a `cp` + +Asked to get the tree onto another machine, I reached for the obvious answer — publish +the branch, `cp -r workspace/. ladybird/` — and then ran it on an empty directory +instead of describing it. Four things were wrong, and only the first was one I could +have found by reading. + +**Nothing recorded which Ladybird commit the overlay describes.** The generated BUILD +files name ~1,961 LibWeb compile inputs and 665 IDL bindings *by path*; they were +generated from exactly one upstream tree. That tree's sha appeared nowhere — not in +the README, not in `cmake2bazel.json`, not in a comment. Every parity claim in this +document is relative to a commit the document never named. This is the same class as +the `--depth 1` and release-tag mistakes: **a pin that is not written down is not a +pin**, and the reason it survived so long is that my working copy *was* the pin. + +**The interesting one: the overlay and upstream's bootstrap fight over `Build/vcpkg`.** +`Build/vcpkg/BUILD.bazel` is an overlay file, so a `cp -r` creates the *directory* +`Build/vcpkg`. Upstream's `Meta/Utils/build_vcpkg.py` then does: + +```python +if not vcpkg_checkout.is_dir(): + git clone … +else: + bootstrapped = git -C Build/vcpkg rev-parse HEAD +``` + +The directory exists, so it takes the `else`, and `git -C Build/vcpkg rev-parse HEAD` +— with no `.git` inside — **walks up to Ladybird's own repository** and cheerfully +returns *Ladybird's* HEAD. It then tries to check vcpkg's baseline out of the Ladybird +repo: `fatal: unable to read tree (40f3c709…)`. Two correct programs, one wrong +composition: upstream infers "cloned" from `is_dir()`, and the overlay's job is to put +a file in that directory. **`git`'s upward search for `.git` is what turns a missing +directory into a wrong answer instead of an error** — the same property that makes +`git -C` convenient makes it unsafe as an existence check. The fix is ordering +(prefetch, *then* stage that one file), and `apply_overlay.sh` defers it and explains +why at the point of deferral. + +The other two were mundane and would have cost someone an afternoon: `bazelrc.txt` +must be renamed to `.bazelrc` (stored under a different name precisely so a `cp -r` +cannot be mistaken for a working build — and then the recipe relies on a human +remembering the rename), and the two upstream patches must be *applied*, not merely +shipped. + +So the deliverable is a script, and the part worth keeping is `--verify`, which checks +what a file copy cannot: HEAD is the pinned commit, all 42 files are byte-identical, +the patches are applied (`git apply --check -R` succeeding is the proof — a patch that +reverse-applies cleanly is already in the tree), and the `.sh` files still have their +executable bit. That last check exists because of an earlier bug of exactly this shape: +scripts committed `100644` while my dev tree had them `+x` by hand, so only a fresh +clone failed, and only at action time. **The general rule this migration keeps +rediscovering: my working tree carries state git does not, and the only way to find it +is to reconstruct the tree somewhere else and diff.** `apply_overlay.sh /tmp/lbfresh2` +now produces a tree byte-identical to the one that renders, which is the first time +that sentence has been checked rather than assumed. + ## Plan for the rest - **Ring 1c — BUILD generation to compile+link parity, bottom-up.** AK → diff --git a/examples/ladybird/README.md b/examples/ladybird/README.md index 17e0cbd..c1f6a82 100644 --- a/examples/ladybird/README.md +++ b/examples/ladybird/README.md @@ -115,6 +115,48 @@ knowing, because both were mistakes I made first: it never appears in a downloads-only run. That asymmetry is recorded in the script rather than smoothed over. +### Reproducing the tree on another machine + +The overlay is **not a fork**: it is a pinned upstream Ladybird commit + two patches ++ 42 Bazel files that sit alongside CMake's. [`apply_overlay.sh`](apply_overlay.sh) is +that sentence made executable, because *"copy `workspace/` over a clone"* has four +ways to be silently wrong — and every one of them was found by running it, not by +reading it: + +1. **The Ladybird commit.** The generated BUILD files name ~1,961 LibWeb compile + inputs and 665 IDL bindings *by path*, and were generated from one tree. + **Nothing in this repo recorded which one** until the script did + (`f9e34731`). Against a different tree the build fails on a moved file — or + worse, silently omits a new one. +2. **The patches**, which have to be applied or the build is nondeterministic + (dictionaries emitted in `PYTHONHASHSEED` order) or does not compile at all + under per-header moc. +3. **The rename**: `bazelrc.txt` → `.bazelrc`. Stored under a different name so a + `cp -r` cannot be mistaken for a working build — and a rename a human does by + hand is a rename a human forgets. +4. **The order**, which is the one I would never have predicted and which the + `cp -r` recipe above gets wrong. `Build/vcpkg/BUILD.bazel` makes the *directory* + `Build/vcpkg` exist, and upstream's `Meta/Utils/build_vcpkg.py` treats "the + directory is there" as "the checkout is there": it skips the clone and runs + `git -C Build/vcpkg rev-parse HEAD`, which — there being no `.git` inside — + **walks up to Ladybird's own repo** and returns *Ladybird's* HEAD. It then tries + to check vcpkg's baseline out of the Ladybird repo: + `fatal: unable to read tree (40f3c709…)`. So that one file must be staged + **after** the prefetch; the script defers it and says so. + +`--verify` checks a tree without changing it, and checks the things a file copy +cannot: that HEAD is the pinned commit, that all 42 files are byte-identical, that +the patches are **applied** (`git apply --check -R` succeeding is the proof — a patch +that reverse-applies cleanly is already in the tree), and that the `.sh` files kept +their **executable bit**, which is tree state a careless copy drops and which then +fails deep inside a build action rather than at setup. + +Verified end to end: `apply_overlay.sh /tmp/lbfresh2` on an empty directory produced +a tree byte-identical to the one that renders (`diff -rq`, excluding prefetch +outputs), `Meta/fetch_vcpkg_git_archives.py` reproduced **4/4** archives verified +against the pinned SHA512s, and `bazel build` then built the 76 vcpkg ports offline +and linked `//:LibHTTP`. + ### The HSTS table: pinned downstream, because upstream is not ours to fix `Meta/CMake/hsts_preload.cmake` downloads Chromium's @@ -220,6 +262,7 @@ an FFI header collision and an entire missing Rust target. See | `Meta/vcpkg_capture_git_archives.sh` | Regenerates that pin with `vcpkg install --only-downloads` (~6 min, no compilation, no CMake) — vcpkg as the instrument, since which git externals are used is decided by feature-conditional CMake code, not by portfile text | | `hsts_preload.bzl` | Chromium's HSTS preload table as one `http_file`, pinned to a **commit** + sha256 — the downstream pin for the one input upstream CMake fetches from `main`. **Generated** by `Meta/pin_hsts_preload.py` | | `Meta/pin_hsts_preload.py` | Re-pins it: resolves the newest commit touching the path, downloads it, writes the hash it **measured**; `--expect-same-as` refuses to write unless the pinned bytes equal the file CMake downloaded (parity guard) | +| `apply_overlay.sh` | Reproduces the whole tree on another machine: clone at the pinned Ladybird commit, apply the two patches, copy the 42 overlay files (with the `bazelrc.txt` → `.bazelrc` rename), run the vcpkg prefetch, then stage the one file that must come after it. `--verify` checks an existing tree — commit, bytes, patches-applied, exec bits — and changes nothing | | `bazelrc.txt` | → `.bazelrc`. Global copts/defines/linkopts mirrored from `Meta/CMake/compile_options.cmake` | | `BUILD.bazel` | Root package: 34 libraries, the 5 executables, Qt moc/rcc genrules. **Generated** by `Meta/emit_build_bazel.py` | | `codegen_root.bzl` | Non-LibWeb generator genrules (IPC endpoints, LibJS Bytecode/Op, HSTS table, WebGL replayer, TIFF tag tables, the two SPIR-V shader headers, and the chained `generate_interpreter_layout` → `flapc` interpreter assembly). **Generated** by `Meta/emit_root_codegen_bazel.py` | @@ -251,8 +294,22 @@ network zero times. **Two** inputs still have to be present first (step 1a; gaps fetched by Bazel: it is pinned downstream to a Chromium commit + sha256 in [`hsts_preload.bzl`](workspace/hsts_preload.bzl): +**One command does the whole tree**, and it is the recommended path because the +manual version below has four ways to be silently wrong (see +[Reproducing the tree](#reproducing-the-tree-on-another-machine)): + +```sh +./apply_overlay.sh ~/ladybird # clone at the pinned commit, patch, overlay, + # vcpkg prefetch -- in the order that works +./apply_overlay.sh --verify ~/ladybird # check an existing tree, change nothing +``` + +Or by hand: + ```sh git clone https://github.com/LadybirdBrowser/ladybird && cd ladybird +git checkout f9e34731b85fea1c3517941d8388566cd33277c4 # the commit the generated + # BUILD files describe # 1. Drop in the overlay. cp -r .../examples/ladybird/workspace/. . && mv bazelrc.txt .bazelrc git apply .../examples/ladybird/patches/*.patch # generator determinism + a Qt header diff --git a/examples/ladybird/apply_overlay.sh b/examples/ladybird/apply_overlay.sh new file mode 100755 index 0000000..bb3ca00 --- /dev/null +++ b/examples/ladybird/apply_overlay.sh @@ -0,0 +1,240 @@ +#!/usr/bin/env bash +# Reproduce the Ladybird tree this migration builds, on any machine. +# +# The overlay is not a fork: it is a pinned upstream Ladybird commit + two +# patches + a set of Bazel files that live alongside CMake's. This script is the +# executable form of that sentence, because "copy the workspace directory over a +# clone" is a recipe with three ways to be silently wrong: +# +# * the LADYBIRD COMMIT. The generated BUILD.bazel files name ~1,961 LibWeb +# compile inputs and 665 IDL bindings by path. They were generated from ONE +# upstream tree; against a different one the build fails on a moved file, or +# worse, silently omits a new one. Nothing in the overlay recorded which +# commit that was until this script did. +# * the PATCHES. Two upstream defects have to be fixed or the build is +# nondeterministic (generator emits dictionaries in PYTHONHASHSEED order) or +# does not compile at all under per-header moc (TabBar.h is not +# self-contained). Both are filed upstream; until they land they are patches. +# * the ORDER. Two overlay files must not exist before the vcpkg prefetch runs. +# Build/vcpkg/BUILD.bazel makes the directory Build/vcpkg EXIST, and upstream's +# Meta/Utils/build_vcpkg.py treats "the directory is there" as "the checkout is +# there": it skips the clone and runs `git -C Build/vcpkg rev-parse HEAD`, which +# -- there being no .git inside -- WALKS UP to Ladybird's own repo and returns +# Ladybird's HEAD. It then tries to check vcpkg's baseline out of the Ladybird +# repo and dies with `fatal: unable to read tree`. So the prefetch has to happen +# BEFORE that file is staged, and this script does the two in that order. +# +# * the RENAME. bazelrc.txt has to become .bazelrc. It is stored under a +# different name so that a `cp -r` of the overlay into a clone cannot be +# mistaken for a working build -- and a rename that a human does by hand is a +# rename a human forgets. +# +# Everything here is pinned and checkable, so a failure is a message rather than +# a mystery. Usage: +# +# ./apply_overlay.sh /path/to/new/ladybird # clone + patch + overlay +# ./apply_overlay.sh --verify /path/to/existing # check a tree, change nothing +# +# Then, per the README's "Reproducing": two prefetches (Meta/ladybird.py vcpkg, +# Meta/fetch_vcpkg_git_archives.py) and `bazel build`. +set -euo pipefail + +# The upstream commit every generated BUILD file in this overlay was generated +# from, and the only tree they are known to describe. A tag would be wrong here +# for the same reason it was wrong for the HSTS table (see hsts_preload.bzl): a +# tag is a pin to a different tree than the one measured. +LADYBIRD_COMMIT="f9e34731b85fea1c3517941d8388566cd33277c4" +LADYBIRD_REPO="https://github.com/LadybirdBrowser/ladybird.git" + +HERE="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" +WORKSPACE="$HERE/workspace" +PATCHES="$HERE/patches" + +die() { echo "error: $*" >&2; exit 1; } +note() { echo "==> $*"; } + +VERIFY=0 +PREFETCH=1 +while [ $# -gt 1 ] || [ "${1:-}" = "--verify" ] || [ "${1:-}" = "--no-prefetch" ]; do + case "${1:-}" in + --verify) VERIFY=1; shift ;; + --no-prefetch) PREFETCH=0; shift ;; + *) break ;; + esac +done +[ $# -eq 1 ] || die "usage: $(basename "$0") [--verify] [--no-prefetch] " +TARGET="$1" + +[ -d "$WORKSPACE" ] || die "no workspace/ next to this script ($WORKSPACE)" + +# --------------------------------------------------------------------------- +# The file list is derived from the overlay itself, never hand-maintained: a +# hand-kept list is how a newly added .bzl gets committed, documented, and then +# copied by nothing (which is exactly what happened to vcpkg_git_archives.bzl -- +# it was generated, committed, listed in the README table, and loaded by no one). +# Written to a temp file rather than consumed through process substitution: +# `< <(...)` needs /dev/fd, which some sandboxes and minimal shells do not give +# you, and this script has to run on the machine the reader has, not mine. +FILE_LIST="$(mktemp)" +trap 'rm -f "$FILE_LIST"' EXIT + +overlay_files() { + (cd "$WORKSPACE" && find . -type f ! -name '*.pyc' -printf '%P\n' | sort) > "$FILE_LIST" + cat "$FILE_LIST" +} + +# bazelrc.txt -> .bazelrc is the one path that differs between the overlay and +# the tree. Keep the mapping in ONE place; both copy and verify read it. +target_path() { + case "$1" in + bazelrc.txt) echo ".bazelrc" ;; + *) echo "$1" ;; + esac +} + +# --------------------------------------------------------------------------- +overlay_files > /dev/null # populate $FILE_LIST once, for both modes + +if [ "$VERIFY" -eq 1 ]; then + [ -d "$TARGET" ] || die "$TARGET does not exist" + cd "$TARGET" + rc=0 + + have="$(git rev-parse HEAD 2>/dev/null || echo none)" + if [ "$have" = "$LADYBIRD_COMMIT" ]; then + note "commit OK ($LADYBIRD_COMMIT)" + else + echo "MISMATCH commit: tree is at $have, overlay was generated from $LADYBIRD_COMMIT" >&2 + echo " the generated BUILD files name sources by path; a different tree may" >&2 + echo " fail on a moved file or silently omit a new one." >&2 + rc=1 + fi + + missing=0; differ=0; same=0 + while IFS= read -r f; do + t="$(target_path "$f")" + if [ ! -e "$t" ]; then + # A file under Build/vcpkg is legitimately absent until the prefetch has + # run: staging it early is what breaks upstream's bootstrap. + case "$f" in + Build/vcpkg/*) + if [ ! -d "Build/vcpkg/.git" ]; then + note "pending (run the vcpkg prefetch, then re-apply): $t" + continue + fi ;; + esac + echo "MISSING $t" >&2; missing=$((missing + 1)) + elif ! cmp -s "$WORKSPACE/$f" "$t"; then + echo "DIFFERS $t" >&2; differ=$((differ + 1)) + else + same=$((same + 1)) + fi + done < "$FILE_LIST" + note "overlay files: $same identical, $differ differing, $missing missing" + [ "$missing" -eq 0 ] && [ "$differ" -eq 0 ] || rc=1 + + # The patches must be APPLIED, not merely present. `git apply --check -R` + # succeeding is the proof: a patch that reverse-applies cleanly is a patch + # already in the tree. + for p in "$PATCHES"/*.patch; do + if git apply --check -R "$p" >/dev/null 2>&1; then + note "patch applied: $(basename "$p")" + else + echo "PATCH NOT APPLIED: $(basename "$p")" >&2; rc=1 + fi + done + + # Executable bits are tree state git carries and a `cp` does not always: the + # vcpkg/cargo build scripts are run as actions, so a lost +x fails at action + # time, deep in a build, with a confusing message. + while IFS= read -r f; do + case "$f" in *.sh) [ -x "$(target_path "$f")" ] || { + echo "NOT EXECUTABLE: $(target_path "$f")" >&2; rc=1; }; ;; + esac + done < "$FILE_LIST" + + [ "$rc" -eq 0 ] && note "VERIFIED: this tree matches the overlay" \ + || echo "verification FAILED" >&2 + exit "$rc" +fi + +# --------------------------------------------------------------------------- +if [ -e "$TARGET/.git" ]; then + note "reusing existing clone at $TARGET" + cd "$TARGET" + git rev-parse --verify "$LADYBIRD_COMMIT^{commit}" >/dev/null 2>&1 \ + || git fetch --no-tags origin "$LADYBIRD_COMMIT" + git checkout --detach "$LADYBIRD_COMMIT" +else + note "cloning Ladybird (full history: vcpkg needs it, see README)" + git clone "$LADYBIRD_REPO" "$TARGET" + cd "$TARGET" + git checkout --detach "$LADYBIRD_COMMIT" +fi +note "at $(git rev-parse HEAD)" + +note "applying $(ls "$PATCHES"/*.patch | wc -l) patches (both filed upstream)" +for p in "$PATCHES"/*.patch; do + if git apply --check -R "$p" >/dev/null 2>&1; then + note " already applied: $(basename "$p")" + else + git apply "$p" || die "failed to apply $(basename "$p")" + note " applied: $(basename "$p")" + fi +done + +# Phase 1: everything except the files under Build/vcpkg. Staging those creates the +# directory upstream's bootstrap reads as "already cloned" -- see the ORDER note at +# the top of this file. +note "copying the overlay" +n=0 +deferred=0 +while IFS= read -r f; do + case "$f" in Build/vcpkg/*) deferred=$((deferred + 1)); continue ;; esac + t="$(target_path "$f")" + mkdir -p "$(dirname "$t")" + cp -p "$WORKSPACE/$f" "$t" + n=$((n + 1)) +done < "$FILE_LIST" +note "$n files copied (bazelrc.txt -> .bazelrc), $deferred deferred until vcpkg exists" + +# Phase 2: the vcpkg prefetch, then the deferred files. +if [ -d "$TARGET/Build/vcpkg/.git" ]; then + note "vcpkg checkout already present" +elif [ "$PREFETCH" -eq 1 ]; then + note "running the vcpkg prefetch (upstream's own bootstrap, ~70s, no CMake)" + python3 Meta/ladybird.py vcpkg +else + note "SKIPPING the vcpkg prefetch (--no-prefetch)" +fi + +if [ -d "$TARGET/Build/vcpkg/.git" ]; then + while IFS= read -r f; do + case "$f" in Build/vcpkg/*) ;; *) continue ;; esac + mkdir -p "$(dirname "$f")" + cp -p "$WORKSPACE/$f" "$f" + note "staged $f" + done < "$FILE_LIST" +else + cat >&2 <<'WARN' +==> NOT staged: Build/vcpkg/BUILD.bazel + Stage it only AFTER the vcpkg checkout exists, or upstream's bootstrap reads + the bare directory as an existing clone and fails with + `fatal: unable to read tree`. Do: + python3 Meta/ladybird.py vcpkg # creates Build/vcpkg + its .git + then re-run this script to stage the deferred file. +WARN +fi + +cat < done. One prefetch remains (it needs no CMake): + + cd $TARGET + python3 Meta/fetch_vcpkg_git_archives.py # the 4 vcpkg_from_git tarballs (~80s) + bazel build //:ladybird //:WebContent //:RequestServer //:ImageDecoder \\ + //:Compositor //:WebWorker + + Re-check this tree at any time with: + $(basename "$0") --verify $TARGET +EOF diff --git a/tests/test_apply_overlay.py b/tests/test_apply_overlay.py new file mode 100644 index 0000000..d2213e6 --- /dev/null +++ b/tests/test_apply_overlay.py @@ -0,0 +1,170 @@ +#!/usr/bin/env python3 +"""Tests for examples/ladybird/apply_overlay.sh. + +The script reproduces the Ladybird tree the migration builds: a pinned upstream +commit + two patches + the overlay files. It cannot be tested end to end here (that +needs a 121 MB clone and the network -- it WAS run end to end, twice, and that is +recorded in the README), so these tests pin the properties that rot silently: + + 1. the Ladybird commit is pinned in exactly one place and is a full sha -- the + generated BUILD files name ~1,961 sources by path, so the tree they describe + has to be identified, and nothing in this repo identified it before; + 2. the file list is DERIVED from the overlay, never hand-maintained -- a + hand-kept list is how vcpkg_git_archives.bzl came to be generated, committed, + documented and copied by nothing; + 3. the Build/vcpkg file is staged AFTER the vcpkg prefetch, because staging it + early makes upstream's bootstrap read the bare directory as an existing clone + and die with `fatal: unable to read tree`; + 4. verification checks the things a file copy cannot: patches applied, exec bits. +""" + +import re +import subprocess +from pathlib import Path + +REPO = Path(__file__).resolve().parent.parent +SCRIPT = REPO / "examples" / "ladybird" / "apply_overlay.sh" +WORKSPACE = REPO / "examples" / "ladybird" / "workspace" +PATCHES = REPO / "examples" / "ladybird" / "patches" + + +def _text(): + return SCRIPT.read_text() + + +def test_script_is_committed_executable(): + """A setup script that is not executable fails at the worst moment: first use.""" + out = subprocess.run( + ["git", "ls-files", "-s", "examples/ladybird/apply_overlay.sh"], + cwd=REPO, capture_output=True, text=True, + ).stdout + assert out.strip(), "apply_overlay.sh is not committed" + assert out.split()[0] == "100755", "apply_overlay.sh must be committed executable" + + +def test_script_is_valid_bash(): + """Syntax-check rather than trust: `bash -n` is free.""" + r = subprocess.run(["bash", "-n", str(SCRIPT)], capture_output=True, text=True) + assert r.returncode == 0, r.stderr + + +def test_ladybird_commit_is_pinned_once_as_a_full_sha(): + """The tree the generated BUILD files describe must be identified exactly. + + A short sha or a tag would both be wrong: a tag moves relative to the tree that + was measured (the same mistake pinning the HSTS table to a release tag would + have been), and one pin in two places is two pins. + """ + text = _text() + pins = re.findall(r'LADYBIRD_COMMIT="([0-9a-f]+)"', text) + assert len(pins) == 1, "the Ladybird commit must be pinned in exactly one place" + assert len(pins[0]) == 40, "pin the full 40-char sha, not an abbreviation" + + +def test_the_pinned_commit_is_documented_where_a_reader_looks(): + """The README's manual recipe must name the same commit as the script. + + Two recipes that disagree about the tree is worse than one recipe. + """ + commit = re.search(r'LADYBIRD_COMMIT="([0-9a-f]{40})"', _text()).group(1) + readme = (REPO / "examples" / "ladybird" / "README.md").read_text() + assert commit in readme, "the README does not name the pinned Ladybird commit" + + +def test_file_list_is_derived_not_hand_maintained(): + """The overlay's file list must come from the overlay itself. + + The concrete failure this prevents: vcpkg_git_archives.bzl was generated, + committed, listed in the README's file table -- and loaded by nothing, because + the wiring was a hand-kept list that nobody updated. + """ + text = _text() + assert "find . -type f" in text, "the file list must be discovered with find" + # ...and no literal roster of overlay files in the CODE. Comments may name a + # file (they explain why the list is derived, citing vcpkg_git_archives.bzl as + # the cautionary case), so strip comments before looking -- a test that cannot + # tell prose from code makes documenting the reason impossible. + code = "\n".join(l.split("#", 1)[0] for l in text.splitlines()) + listed = re.findall(r'\b[\w.]+\.bzl\b', code) + assert not listed, "found hand-listed overlay files in code: %r" % (listed,) + + +def test_every_overlay_file_would_be_copied(): + """The derived list must actually cover the committed overlay. + + Recomputes what the script's `find` yields and compares it to git's idea of the + overlay, so a file committed but excluded by the find expression is caught. + """ + found = subprocess.run( + ["find", ".", "-type", "f", "!", "-name", "*.pyc", "-printf", "%P\n"], + cwd=WORKSPACE, capture_output=True, text=True, + ).stdout.split() + committed = subprocess.run( + ["git", "ls-files", "examples/ladybird/workspace"], + cwd=REPO, capture_output=True, text=True, + ).stdout.split() + committed = {c.replace("examples/ladybird/workspace/", "") for c in committed} + missed = committed - set(found) + assert not missed, "committed overlay files the script would not copy: %r" % (missed,) + + +def test_the_bazelrc_rename_is_expressed_once(): + """bazelrc.txt -> .bazelrc must be one mapping both copy and verify use.""" + text = _text() + assert text.count("bazelrc.txt) echo") == 1, \ + "the rename must live in a single target_path mapping" + assert "target_path" in text + + +def test_build_vcpkg_file_is_deferred_past_the_prefetch(): + """The ordering bug, pinned as a test. + + Build/vcpkg/BUILD.bazel makes the DIRECTORY exist; upstream's build_vcpkg.py + reads that as "already cloned", skips the clone, and `git -C Build/vcpkg + rev-parse HEAD` then walks up to Ladybird's own .git and returns Ladybird's + HEAD -- so it tries to check vcpkg's baseline out of the Ladybird repo and dies + with `fatal: unable to read tree`. Found by running the script on an empty + directory; it is not visible by reading either side alone. + """ + text = _text() + assert "Build/vcpkg/*) deferred=" in text, \ + "files under Build/vcpkg must be skipped in the first copy pass" + # the deferred staging must be gated on the checkout really existing + assert 'if [ -d "$TARGET/Build/vcpkg/.git" ]; then' in text + assert "unable to read tree" in text, \ + "the failure mode must be named where someone hitting it will look" + + +def test_verify_checks_patches_are_applied_not_merely_present(): + """`git apply --check -R` succeeding is what proves a patch is IN the tree.""" + assert "git apply --check -R" in _text() + + +def test_verify_checks_executable_bits(): + """Exec bits are tree state a copy drops, and the failure surfaces mid-build.""" + text = _text() + assert "NOT EXECUTABLE" in text + assert re.search(r'\[ -x .* \]', text), "verify must test -x on the .sh files" + + +def test_verify_mode_changes_nothing(): + """--verify must contain no mutating command in its branch. + + A "check my tree" mode that writes is a trap, so this reads the verify branch + and asserts it does not copy, clone, checkout or apply. + """ + text = _text() + verify_branch = text.split('if [ "$VERIFY" -eq 1 ]; then', 1)[1].split("\nfi\n", 1)[0] + for forbidden in ("cp -p", "git clone", "git checkout", "git apply \"", "mkdir -p"): + assert forbidden not in verify_branch, \ + "--verify must not mutate the tree (found %r)" % forbidden + + +def test_both_patches_are_referenced_by_glob_not_by_name(): + """A third upstream patch must not need a script edit to be applied.""" + text = _text() + assert '"$PATCHES"/*.patch' in text + names = [p.name for p in PATCHES.glob("*.patch")] + assert len(names) >= 2 + for n in names: + assert n not in text, "patch %s is named literally; use the glob" % n From 647d87c7b358a5469f176645843be59de347796e Mon Sep 17 00:00:00 2001 From: Ulf Adams Date: Tue, 11 Aug 2026 15:15:10 +0000 Subject: [PATCH 18/59] Ladybird: pin vcpkg's own host tools -- the capture could only see what my machine lacked Ulf's clone failed with `distfile MISSING FROM INDEX ... ninja-linux.zip` at vcpkg's compiler detection, then x-block-origin correctly refused the network. The 76-distfile pin came from instrumenting vcpkg's downloader, which cannot miss a download vcpkg asked for -- but vcpkg_find_acquire_program probes the HOST first, and this machine has /usr/bin/ninja at exactly the required 1.13.2, so vcpkg never asked. The capture was faithful; it was an observation of my machine. cmake is in the pin only by luck (host 4.2.3 vs required 4.4.0), and that accident made the whole class look covered. An instrument that records what a program did cannot pin what it would do elsewhere, when its behaviour depends on the machine. The comparand is vcpkg's own scripts/vcpkg-tools.json -- versioned at the baseline, url+sha512+archive name for every tool -- which is a pin rather than an observation. Deriving the tools at emit time was the wrong fix and two existing tests caught it: the emitter's promise is that the committed pin alone regenerates everything with no vcpkg, no CMake, no network, and reading vcpkg metadata during emit made a checkout a requirement again. So --capture-tools writes Meta/vcpkg_tool_assets.tsv, committed like the asset capture; emitting unions it in and cross-checks against the live metadata when a checkout happens to be present. Scoped to cmake+ninja (what detect_compiler needs); the 7 skipped tools are reported, not hidden -- dotnet/node/powershell alone are ~400MB no port here invokes. Verified: the ninja sha512 confirmed against upstream independently of the error text, `bazel query deps(//:vcpkg_installed,1)` now lists 78 distfiles including ninja, and the index the asset script reads resolves that hash to the Bazel-fetched file. 182/182, six new tests including the regression test for the report. --- docs/CASE-ladybird-migration.md | 68 ++++++++ examples/ladybird/README.md | 1 + examples/ladybird/workspace/MODULE.bazel | 8 + .../workspace/Meta/emit_vcpkg_bazel.py | 153 ++++++++++++++++++ .../workspace/Meta/vcpkg_tool_assets.tsv | 14 ++ .../ladybird/workspace/vcpkg_distfiles.bzl | 8 +- examples/ladybird/workspace/vcpkg_index.bzl | 1 + tests/test_emit_vcpkg.py | 130 ++++++++++++++- 8 files changed, 378 insertions(+), 5 deletions(-) create mode 100644 examples/ladybird/workspace/Meta/vcpkg_tool_assets.tsv diff --git a/docs/CASE-ladybird-migration.md b/docs/CASE-ladybird-migration.md index 4025a94..7905a9a 100644 --- a/docs/CASE-ladybird-migration.md +++ b/docs/CASE-ladybird-migration.md @@ -2146,6 +2146,74 @@ is to reconstruct the tree somewhere else and diff.** `apply_overlay.sh /tmp/lbf now produces a tree byte-identical to the one that renders, which is the first time that sentence has been checked rather than assumed. +## Finding 38: the pin recorded what my machine lacked, not what the build needs + +Ulf's clone failed where mine never could: + +``` +vcpkg_build: distfile MISSING FROM INDEX: + .../ninja-build/ninja/releases/download/v1.13.2/ninja-linux.zip +error: there were no asset cache hits, and x-block-origin blocks trying the + authoritative source +``` + +The 76-distfile pin came from *instrumenting vcpkg's own downloader* (finding 28) — +the strongest evidence available, and the thing I have leaned on hardest in this +document, because the capture cannot invent a URL and cannot miss one vcpkg asked +for. It has one blind spot, and it is not in the instrument, it is in the +**subject**: `vcpkg_find_acquire_program` probes the host *before* downloading. My +machine has `/usr/bin/ninja` at exactly 1.13.2 — the version vcpkg wants — so vcpkg +never asked for ninja, so the capture never saw it, so the pin never had it. Nothing +was broken. The observation was faithful; it was an observation *of my machine*. + +The reason this went unnoticed is the reason it is worth a finding: **cmake is in the +pin, and only by luck.** The host cmake is 4.2.3 against the required 4.4.0, so vcpkg +*did* download that one — and its presence made the whole class look covered. One +member of a category being present by accident is what a partial pin looks like from +the inside. + +Two general shapes, both of which I had already written down in weaker forms: + +- **An instrument that records what a program *did* cannot pin what the program + *would do elsewhere*, when the program's behaviour depends on the machine.** The + capture is exact about vcpkg's requests and silent about vcpkg's *decisions*. + Finding 36 said "a green check has to be compared against something it did not + produce"; the comparand here is vcpkg's own tool metadata, + `scripts/vcpkg-tools.json`, versioned inside the checkout at the baseline, carrying + url + sha512 + archive name for every tool on every platform. That is a **pin** + rather than an observation, so it is complete regardless of what is installed + anywhere. +- **Host-tool discovery is a hermeticity boundary that looks like a convenience.** + Every `find_program`/`find_package` is a place where the build's inputs depend on + the machine, and a capture-based pin will silently record the *complement* of + whatever is installed. The pin's contents should not be a function of one + machine's `/usr/bin`. + +The fix that first suggested itself — derive the tools from `vcpkg-tools.json` at emit +time — was wrong, and **two existing tests caught it before Ulf could**: the emitter's +central promise is that *the committed pin alone regenerates every Bazel file with no +vcpkg checkout, no CMake and no network*, and reading vcpkg's metadata during emit +quietly made a vcpkg checkout a requirement again. So the derivation is a separate, +deliberate step (`--capture-tools`) whose output is committed as +`Meta/vcpkg_tool_assets.tsv`, exactly like the asset capture; emitting unions it in +and, *when* a checkout happens to be present, cross-checks it and warns if the +committed pin has gone stale. That is the same division as everywhere else here: +regenerating a pin may use the world, consuming one may not. + +Scoping, stated because a reviewer should not have to infer it: +`vcpkg-tools.json` also pins dotnet, node, powershell-core, azcopy, gsutil, coscli and +nuget — ~400 MB for tools no port in this closure invokes (only the unrelated +`vbs-enclave-tooling-codegen` does). `BUILD_TOOLS` is `("cmake", "ninja")`, the two +`scripts/detect_compiler` needs before any port builds at all, and the emitter +*reports* the ones it skipped rather than hiding the decision. + +Verified: the ninja `http_file` fetches and integrity-checks (sha512 confirmed against +upstream independently of Ulf's error text), `bazel query 'deps(//:vcpkg_installed, 1)'` +now lists **78** distfiles including ninja, and the index the asset script reads +resolves that hash to the Bazel-fetched file — the exact lookup that failed on his +machine. Suite 182/182, six new tests, one of which is the regression test for the +bug report itself. + ## Plan for the rest - **Ring 1c — BUILD generation to compile+link parity, bottom-up.** AK → diff --git a/examples/ladybird/README.md b/examples/ladybird/README.md index c1f6a82..b35e2a6 100644 --- a/examples/ladybird/README.md +++ b/examples/ladybird/README.md @@ -257,6 +257,7 @@ an FFI header collision and an entire missing Rust target. See | `Meta/vcpkg_assets.tsv` | **The dependency pin.** 76 `(url, sha512, dst)` rows captured from vcpkg itself via its `x-script` asset hook. Everything below is generated from this, with no vcpkg, no CMake and no network | | `vcpkg_distfiles.bzl`, `vcpkg_index.bzl`, `vcpkg_extension.bzl`, `vcpkg_git_archives.bzl` | One `http_file` per distfile, the sha512→label index the asset script resolves through, the module extension that creates the repos, and the 4 `vcpkg_from_git` archives. **Generated** by `Meta/emit_vcpkg_bazel.py` | | `vcpkg.bzl`, `Meta/vcpkg_build.sh` | `vcpkg_tree`: builds the whole dep tree as an ordinary Bazel action with `x-block-origin`, so it reaches the network zero times. Deliberately not a `repository_rule` | +| `Meta/vcpkg_tool_assets.tsv` | vcpkg's OWN host tools (cmake, ninja) — url + sha512 + the filename vcpkg looks for. **Separate from the asset capture on purpose:** `vcpkg_find_acquire_program` probes the host first, so a tool the capturing machine already had is never downloaded and never captured (that is how ninja went unpinned; finding 38). Regenerate with `emit_vcpkg_bazel.py --capture-tools` | | `Meta/vcpkg_capture_assets.sh` | Records the 76-distfile pin, by *being* vcpkg's asset cache. The one run allowed to fetch | | `Meta/fetch_vcpkg_git_archives.py` | Produces the 4 `vcpkg_from_git` tarballs **without CMake**: takes the list from the committed pin, resolves each clone URL out of the portfiles, then `git clone` + `git -c core.autocrlf=false archive ` and **verifies against the pinned SHA512**. 4/4 byte-identical | | `Meta/vcpkg_capture_git_archives.sh` | Regenerates that pin with `vcpkg install --only-downloads` (~6 min, no compilation, no CMake) — vcpkg as the instrument, since which git externals are used is decided by feature-conditional CMake code, not by portfile text | diff --git a/examples/ladybird/workspace/MODULE.bazel b/examples/ladybird/workspace/MODULE.bazel index 446f4e7..8c133c6 100644 --- a/examples/ladybird/workspace/MODULE.bazel +++ b/examples/ladybird/workspace/MODULE.bazel @@ -118,6 +118,14 @@ use_repo( 'vcpkg_ngtcp2_nghttp3_v1_17_0_tar_gz_23d85a2abfa8', 'vcpkg_ngtcp2_ngtcp2_v1_24_0_tar_gz_04a5762d6eac', 'vcpkg_ngtcp2_sfparse_f2046eaa1acba7c5467399b1e1e1f354d22d1f48_tar_gz_b3cbcce6d96d', + # A host tool vcpkg fetches for ITSELF, pinned from Meta/vcpkg_tool_assets.tsv + # (derived from vcpkg scripts/vcpkg-tools.json) rather than from the download + # capture: vcpkg does not download a tool the machine already has, so a + # capturing machine carrying /usr/bin/ninja at exactly the required 1.13.2 kept + # ninja out of the pin, and a machine without it hit + # `distfile MISSING FROM INDEX` (finding 38). cmake is in this list already, + # but only by luck -- the capturing host had the wrong version. + 'vcpkg_ninja_linux_1_13_2_zip_714b900cf10b', 'vcpkg_openssl_openssl_openssl_3_6_3_tar_gz_a89c08101fa1', 'vcpkg_patchelf_0_19_0_x86_64_tar_gz_2a65c9cbdddc', 'vcpkg_pdfjs_5_6_205_dist_zip_66fecdb8a80d', diff --git a/examples/ladybird/workspace/Meta/emit_vcpkg_bazel.py b/examples/ladybird/workspace/Meta/emit_vcpkg_bazel.py index 69cd73c..b539747 100644 --- a/examples/ladybird/workspace/Meta/emit_vcpkg_bazel.py +++ b/examples/ladybird/workspace/Meta/emit_vcpkg_bazel.py @@ -391,6 +391,117 @@ def load_capture(path): return out +# The host tools vcpkg fetches for ITSELF (cmake, ninja, ...), as opposed to the +# distfiles it fetches for ports. Values are (url, sha512, filename). +# +# WHY THIS IS NOT PART OF THE CAPTURE. The capture records what vcpkg actually +# downloaded on the capturing machine, and vcpkg does not download a tool it can +# already find: `vcpkg_find_acquire_program` probes the host first. My machine has +# /usr/bin/ninja at exactly 1.13.2 -- the version vcpkg wants -- so ninja was never +# fetched, never captured, and never pinned; a machine WITHOUT it got +# `distfile MISSING FROM INDEX ... ninja-linux.zip` followed by x-block-origin +# correctly refusing the network. cmake made it into the pin only by luck: the host +# cmake is 4.2.3 against the required 4.4.0, so that one WAS downloaded. +# +# So the capture is the wrong instrument for this class: what it observes depends on +# what the capturing machine happened to have installed. The right source is vcpkg's +# own tool metadata -- scripts/vcpkg-tools.json, versioned inside the checkout at the +# baseline commit, carrying url + sha512 + archive name for every tool on every +# platform. That is a PIN, not an observation, so it is complete regardless of what +# is installed anywhere. +# The tools this build's ports can actually ask for. vcpkg-tools.json also pins +# dotnet, node, powershell-core, azcopy, gsutil, coscli and nuget -- ~400 MB of +# downloads for a build that never invokes any of them (only the unrelated +# vbs-enclave-tooling-codegen port does, and it is not in this closure). Pinning +# those would trade one failure mode for a slower, larger version of the same +# hermeticity story, so the set is scoped and the scoping is stated: BUILD_TOOLS is +# what vcpkg needs to configure and build a port at all. +# +# cmake and ninja are needed by scripts/detect_compiler -- which runs before any +# port does, for every triplet -- so they are not optional for anyone. +BUILD_TOOLS = ("cmake", "ninja") + + +def tool_distfiles(triplet_os="linux", arches=(None, "x64", "amd64"), + want=BUILD_TOOLS, vcpkg=None): + """vcpkg's own host tools for one platform, read from its tool metadata. + + Keyed by sha512 like every other distfile, so these merge straight into the + index the asset-cache script resolves through. + """ + path = os.path.join(vcpkg or VCPKG, "scripts", "vcpkg-tools.json") + if not os.path.exists(path): + raise VcpkgUnavailable("no %s (needed for vcpkg's own tool pins)" % path) + with open(path) as f: + meta = json.load(f) + out = {} + for t in meta.get("tools", []): + if want is not None and t.get("name") not in want: + continue + if t.get("os") != triplet_os or t.get("arch") not in arches: + continue + url, sha = t.get("url"), t.get("sha512") + if not url or not sha: + continue # a tool vcpkg expects from the system, not a download + # vcpkg stores the archive under `archive` when it differs from the URL's + # basename (ninja-linux.zip -> ninja-linux-1.13.2.zip): the asset cache is + # asked for the name vcpkg will look for, so prefer it. + name = t.get("archive") or url.rsplit("/", 1)[-1] + out[sha.lower()] = (url, name, "vcpkg-tool:" + t.get("name", "?"), "tools.json") + return out + + +# The tool pins are COMMITTED next to the asset capture, in the same 3-column TSV +# format, and that is not redundancy -- it is what keeps the emitter's central +# promise true: the committed pin regenerates every Bazel file with no vcpkg +# checkout, no CMake and no network. Deriving the tools from +# scripts/vcpkg-tools.json at emit time would have quietly made a vcpkg checkout a +# requirement again (two tests caught exactly that). So the derivation is a +# separate, deliberate step -- `--capture-tools` -- and its output is reviewed and +# committed like any other pin. +TOOLS_TSV = os.path.join(os.path.dirname(os.path.abspath(__file__)), + "vcpkg_tool_assets.tsv") + + +def load_tool_pins(path=None): + """The committed tool pins -> {sha512: (url, filename, port, source)}.""" + path = path or TOOLS_TSV + if not os.path.exists(path): + return {} + out = {} + with open(path) as f: + for line in f: + line = line.split("#", 1)[0].strip() + if not line: + continue + parts = line.split("\t") + if len(parts) != 3: + continue + url, sha, name = parts + if not re.fullmatch(r'[0-9a-fA-F]{128}', sha): + continue + out[sha.lower()] = (url, name, "vcpkg-tool", "tools.tsv") + return out + + +def emit_tool_pins(tools): + """Write the tool pin TSV: url, sha512, the filename vcpkg looks for.""" + print("# vcpkg's OWN host tools, pinned from its scripts/vcpkg-tools.json at the") + print("# builtin-baseline. GENERATED by emit_vcpkg_bazel.py --capture-tools.") + print("#") + print("# Separate from vcpkg_assets.tsv because the asset capture CANNOT see") + print("# these: vcpkg_find_acquire_program probes the host first, so a tool the") + print("# capturing machine already has is never downloaded and never captured.") + print("# That is how ninja went unpinned -- the capturing machine had") + print("# /usr/bin/ninja at exactly the required version -- while cmake was pinned") + print("# only because the host's was too old. What a pin contains must not depend") + print("# on what happens to be installed on one machine.") + print("#") + print("# url\tsha512\tfilename-vcpkg-looks-for") + for sha, (url, name, port, _src) in sorted(tools.items(), key=lambda kv: kv[1][1]): + print("%s\t%s\t%s" % (url, sha, name)) + + def collect(): """-> (distfiles, git_externals, unresolved). Ports come from the manifest.""" pins = pinned_versions() @@ -671,6 +782,48 @@ def main(): sys.stderr.write(" %-40s %s\n" % (distfiles[sha][1], distfiles[sha][2])) distfiles, unexpanded = cap, [] + # vcpkg's OWN tools are unioned in from their COMMITTED pin, not replaced + # and not re-derived: they are a different class from port distfiles, and + # the capture cannot see them at all (finding 38). + tools = load_tool_pins() + if not tools: + sys.stderr.write( + "error: no %s\n" + "vcpkg's own host tools (cmake, ninja) are pinned there, because the\n" + "capture cannot see a tool the capturing machine already had. Without\n" + "them the index works only on machines that happen to have the same\n" + "tools installed. Regenerate with:\n" + " emit_vcpkg_bazel.py --capture-tools > Meta/vcpkg_tool_assets.tsv\n" + % TOOLS_TSV) + return 2 + added = [sha for sha in tools if sha not in distfiles] + sys.stderr.write("vcpkg host tools: %d pinned from %s (%d not in the capture)\n" + % (len(tools), os.path.basename(TOOLS_TSV), len(added))) + for sha in sorted(added): + sys.stderr.write(" added %-34s %s\n" % (tools[sha][1], tools[sha][2])) + # Cross-check against vcpkg's metadata WHEN a checkout is at hand: a stale + # committed pin (baseline bumped, tool version moved) is otherwise invisible + # until someone without that tool tries to build. + try: + live = tool_distfiles() + except VcpkgUnavailable: + pass + else: + stale = set(live) - set(tools) + gone = set(tools) - set(live) + for sha in sorted(stale): + sys.stderr.write( + " WARNING: %s is in vcpkg-tools.json but NOT in the committed" + " pin -- re-run --capture-tools\n" % live[sha][1]) + for sha in sorted(gone): + sys.stderr.write( + " note: %s is pinned but no longer in vcpkg-tools.json\n" + % tools[sha][1]) + for sha, row in tools.items(): + distfiles.setdefault(sha, row) + if "--capture-tools" in sys.argv: + emit_tool_pins(tool_distfiles()) + return 0 if "--git-archives" in sys.argv: dl = sys.argv[sys.argv.index("--git-archives") + 1] archives, byproducts = observed_git_archives(dl, distfiles) diff --git a/examples/ladybird/workspace/Meta/vcpkg_tool_assets.tsv b/examples/ladybird/workspace/Meta/vcpkg_tool_assets.tsv new file mode 100644 index 0000000..8952843 --- /dev/null +++ b/examples/ladybird/workspace/Meta/vcpkg_tool_assets.tsv @@ -0,0 +1,14 @@ +# vcpkg's OWN host tools, pinned from its scripts/vcpkg-tools.json at the +# builtin-baseline. GENERATED by emit_vcpkg_bazel.py --capture-tools. +# +# Separate from vcpkg_assets.tsv because the asset capture CANNOT see +# these: vcpkg_find_acquire_program probes the host first, so a tool the +# capturing machine already has is never downloaded and never captured. +# That is how ninja went unpinned -- the capturing machine had +# /usr/bin/ninja at exactly the required version -- while cmake was pinned +# only because the host's was too old. What a pin contains must not depend +# on what happens to be installed on one machine. +# +# url sha512 filename-vcpkg-looks-for +https://github.com/Kitware/CMake/releases/download/v4.4.0/cmake-4.4.0-linux-x86_64.tar.gz 3df4aaa128a438ed48dcac7065fd355ff538eed8f394491298d0db63a891d671da247c8fa262e4fa6bf99429d630abab317d5a0248168fe203d1ca4978dab4da cmake-4.4.0-linux-x86_64.tar.gz +https://github.com/ninja-build/ninja/releases/download/v1.13.2/ninja-linux.zip 714b900cf10b7ecb1b641c91f4ef696250c64984e5955a8088e4a538d6e8077f43e55f6da47efcedbe316c68d51a9e98feff51734eb0eac1b17aa85af5698753 ninja-linux-1.13.2.zip diff --git a/examples/ladybird/workspace/vcpkg_distfiles.bzl b/examples/ladybird/workspace/vcpkg_distfiles.bzl index f6912d8..9289ae0 100644 --- a/examples/ladybird/workspace/vcpkg_distfiles.bzl +++ b/examples/ladybird/workspace/vcpkg_distfiles.bzl @@ -1,7 +1,7 @@ # AUTO-GENERATED by Meta/emit_vcpkg_bazel.py — do not edit. load("@bazel_tools//tools/build_defs/repo:http.bzl", "http_file") -# 76 upstream distfiles. Each `integrity` is vcpkg's own published +# 77 upstream distfiles. Each `integrity` is vcpkg's own published # SHA512 from the baseline-resolved portfile, hex->base64; nothing is # re-hashed here, so this is not a trust downgrade. @@ -360,6 +360,12 @@ def vcpkg_distfiles(): downloaded_file_path = 'ngtcp2-sfparse-f2046eaa1acba7c5467399b1e1e1f354d22d1f48.tar.gz', integrity = 'sha512-s8vM5tltxzHSGmeUCgXBYD1D7pdmgZ4dF0ypPHr8f6YkeqgYNoyvuzFETgGYq1u6RYCHcqPKieRZ+NLA/65vXQ==', # captured ) + http_file( + name = 'vcpkg_ninja_linux_1_13_2_zip_714b900cf10b', + urls = ['https://github.com/ninja-build/ninja/releases/download/v1.13.2/ninja-linux.zip'], + downloaded_file_path = 'ninja-linux-1.13.2.zip', + integrity = 'sha512-cUuQDPELfssbZByR9O9pYlDGSYTllVqAiOSlONboB39D5V9tpH787b4xbGjVGp6Y/v9Rc06w6sGxeqha9WmHUw==', # vcpkg-tool + ) http_file( name = 'vcpkg_openssl_openssl_openssl_3_6_3_tar_gz_a89c08101fa1', urls = ['https://github.com/openssl/openssl/archive/openssl-3.6.3.tar.gz'], diff --git a/examples/ladybird/workspace/vcpkg_index.bzl b/examples/ladybird/workspace/vcpkg_index.bzl index 756e793..fe9784d 100644 --- a/examples/ladybird/workspace/vcpkg_index.bzl +++ b/examples/ladybird/workspace/vcpkg_index.bzl @@ -63,6 +63,7 @@ VCPKG_DISTFILE_INDEX = { '23d85a2abfa81433049d7b1b0440b5b04ae3515830db0347da335a30b93c8be8e25d2f73e198cdb9207e2b51ce924f133bab7314cde76d80f73c51c1fd1c36c4': '@vcpkg_ngtcp2_nghttp3_v1_17_0_tar_gz_23d85a2abfa8//file:ngtcp2-nghttp3-v1.17.0.tar.gz', # captured '04a5762d6eac7227431eb8e293d2786dee9f7d0467a584bdd35f4e33c7c6ff6e7b010e6e894fabdf780f6377373629c10a4d41effa374e2129d9f9b1e537873e': '@vcpkg_ngtcp2_ngtcp2_v1_24_0_tar_gz_04a5762d6eac//file:ngtcp2-ngtcp2-v1.24.0.tar.gz', # captured 'b3cbcce6d96dc731d21a67940a05c1603d43ee9766819e1d174ca93c7afc7fa6247aa818368cafbb31444e0198ab5bba45808772a3ca89e459f8d2c0ffae6f5d': '@vcpkg_ngtcp2_sfparse_f2046eaa1acba7c5467399b1e1e1f354d22d1f48_tar_gz_b3cbcce6d96d//file:ngtcp2-sfparse-f2046eaa1acba7c5467399b1e1e1f354d22d1f48.tar.gz', # captured + '714b900cf10b7ecb1b641c91f4ef696250c64984e5955a8088e4a538d6e8077f43e55f6da47efcedbe316c68d51a9e98feff51734eb0eac1b17aa85af5698753': '@vcpkg_ninja_linux_1_13_2_zip_714b900cf10b//file:ninja-linux-1.13.2.zip', # vcpkg-tool 'a89c08101fa1d7e3c09b14f4a90d450bcf336a4f6a3e6e4ea990e4deddcd9ce250472f9114438fd134ff4b47fe93dd47232308567088b2b1c0b2eb50e3b56bdf': '@vcpkg_openssl_openssl_openssl_3_6_3_tar_gz_a89c08101fa1//file:openssl-openssl-openssl-3.6.3.tar.gz', # captured '2a65c9cbdddcc7952cdbd6e98a2cf3da01386cf0f0b927a6bbcfe8131ecf0bfb17c534246635b5e6a090652ee54c903f9f9c4f3f1d2412dba59f287ae2ae8070': '@vcpkg_patchelf_0_19_0_x86_64_tar_gz_2a65c9cbdddc//file:patchelf-0.19.0-x86_64.tar.gz', # captured '66fecdb8a80d013b592c361e85abd7eeea5bc35f0131b771cc25a1878d7737704458dc715670aace3c7d3437f85388dff04796fe117eef01be81fd0d192f73a2': '@vcpkg_pdfjs_5_6_205_dist_zip_66fecdb8a80d//file:pdfjs-5.6.205-dist.zip', # captured diff --git a/tests/test_emit_vcpkg.py b/tests/test_emit_vcpkg.py index 98111c6..cac94bb 100644 --- a/tests/test_emit_vcpkg.py +++ b/tests/test_emit_vcpkg.py @@ -176,12 +176,19 @@ def test_emitting_from_the_capture_alone_needs_no_vcpkg(): capture_output=True, text=True, env=env) assert ok.returncode == 0, ok.stderr assert "no local vcpkg checkout" in ok.stderr - # The real pin: 76 distfiles, every one keyed by its own sha512. + # The real pin: 76 captured distfiles + the ONE vcpkg host tool the capture + # could not see (finding 38). It is 77, not 78, and the arithmetic is the + # finding in miniature: cmake is in BOTH the capture and the tool pin, because + # the capturing machine's cmake was the wrong version and got downloaded, while + # its ninja was exactly right and did not. Union by sha, so cmake counts once. shas = re.findall(r"^ '([0-9a-f]{128})':", ok.stdout, re.M) - assert len(shas) == 76, len(shas) - assert len(set(shas)) == 76 + assert len(shas) == 77, len(shas) + assert len(set(shas)) == 77 repos = re.findall(r"@(vcpkg_[A-Za-z0-9_]+)//file:", ok.stdout) - assert len(set(repos)) == 76, "repo names must be unique per distfile" + assert len(set(repos)) == 77, "repo names must be unique per distfile" + # ...and the tool pins must survive the no-vcpkg path, which is the whole + # reason they are committed as a TSV rather than derived at emit time. + assert "ninja-linux-1.13.2.zip" in ok.stdout bad = subprocess.run(["python3", _EMIT, "--index"], capture_output=True, text=True, env=env) @@ -219,3 +226,118 @@ def test_non_mirrored_urls_are_left_as_a_single_entry(): # not reach. `python3 tests/run_all.py` enumerates the module instead, so a test's # POSITION in the file cannot decide whether it runs; it also fails if a file # defines no tests at all. Run a single file with `run_all.py `. + + +# -------------------------------------------------------------------------- +# vcpkg's OWN host tools (finding 38) +# +# The capture cannot see a tool the capturing machine already has: +# vcpkg_find_acquire_program probes the host first, so my /usr/bin/ninja at +# exactly the required 1.13.2 kept ninja out of the pin entirely, and a machine +# without it got `distfile MISSING FROM INDEX ... ninja-linux.zip` followed by +# x-block-origin correctly refusing the network. cmake was in the pin only by +# luck (host 4.2.3 vs required 4.4.0). +# +# The fix reads vcpkg's own tool metadata instead of the capture, so what gets +# pinned no longer depends on what happens to be installed anywhere. These tests +# pin that property, the filename vcpkg will look for, and the scoping. + +TOOLS_JSON = """{ + "tools": [ + {"name": "ninja", "os": "linux", "arch": "x64", "version": "1.13.2", + "url": "https://example.test/ninja-linux.zip", "sha512": "%s", + "archive": "ninja-linux-1.13.2.zip"}, + {"name": "ninja", "os": "windows", "arch": "x64", "version": "1.13.2", + "url": "https://example.test/ninja-win.zip", "sha512": "%s"}, + {"name": "cmake", "os": "linux", "arch": "x64", "version": "4.4.0", + "url": "https://example.test/cmake-4.4.0-linux-x86_64.tar.gz", "sha512": "%s", + "archive": "cmake-4.4.0-linux-x86_64.tar.gz"}, + {"name": "node", "os": "linux", "arch": "x64", "version": "24.18.0", + "url": "https://example.test/node.tar.gz", "sha512": "%s"}, + {"name": "git", "os": "linux", "arch": "x64", "version": "2.7.4"} + ] +}""" % ("a" * 128, "b" * 128, "c" * 128, "d" * 128) + + +def _vcpkg_with_tools(tmpdir, body=TOOLS_JSON): + scripts = os.path.join(tmpdir, "scripts") + os.makedirs(scripts, exist_ok=True) + with open(os.path.join(scripts, "vcpkg-tools.json"), "w") as f: + f.write(body) + return tmpdir + + +def test_tool_pins_come_from_vcpkg_metadata_not_the_capture(): + """The whole point: the pin must not depend on what is installed locally.""" + with tempfile.TemporaryDirectory() as d: + tools = emit.tool_distfiles(vcpkg=_vcpkg_with_tools(d)) + assert set(tools) == {"a" * 128, "c" * 128}, \ + "expected exactly the linux/x64 cmake+ninja pins, got %r" % (list(tools),) + + +def test_tool_pin_uses_the_archive_name_vcpkg_will_look_for(): + """The asset script is asked for vcpkg's OWN filename, not the URL basename. + + ninja's URL basename is ninja-linux.zip but vcpkg stores (and looks for) + ninja-linux-1.13.2.zip. Emitting the URL basename would put a row in the index + under a name vcpkg never asks about -- a pin that looks present and is not. + """ + with tempfile.TemporaryDirectory() as d: + tools = emit.tool_distfiles(vcpkg=_vcpkg_with_tools(d)) + assert tools["a" * 128][1] == "ninja-linux-1.13.2.zip" + + +def test_tool_pins_are_scoped_to_what_this_build_can_invoke(): + """node/dotnet/powershell are ~400MB no port in this closure ever runs.""" + assert "node" not in emit.BUILD_TOOLS + assert set(emit.BUILD_TOOLS) == {"cmake", "ninja"} + with tempfile.TemporaryDirectory() as d: + vc = _vcpkg_with_tools(d) + names = {r[2] for r in emit.tool_distfiles(vcpkg=vc).values()} + everything = {r[2] for r in emit.tool_distfiles(want=None, vcpkg=vc).values()} + assert "vcpkg-tool:node" not in names + assert "vcpkg-tool:node" in everything, "want=None must mean every tool" + + +def test_a_tool_with_no_url_is_skipped(): + """`git` has no url: vcpkg expects it from the system, so there is nothing to pin.""" + with tempfile.TemporaryDirectory() as d: + rows = emit.tool_distfiles(want=None, vcpkg=_vcpkg_with_tools(d)) + assert not any(r[2] == "vcpkg-tool:git" for r in rows.values()) + + +def test_missing_tool_metadata_is_an_error_not_an_empty_pin(): + """Silently emitting no tool pins is what broke a clone: fail loudly instead.""" + with tempfile.TemporaryDirectory() as d: + try: + emit.tool_distfiles(vcpkg=d) # exists, but has no vcpkg-tools.json + except emit.VcpkgUnavailable as e: + assert "vcpkg-tools.json" in str(e) + else: + raise AssertionError("expected VcpkgUnavailable") + + +def test_the_committed_pin_carries_ninja_and_cmake(): + """The regression test for the actual bug report, against the committed files. + + Ulf's clone failed at `Detecting compiler hash` because ninja was absent from + the index; cmake was present. Both must be there now, and the index is what + the asset script resolves through, so check the index -- not just the + http_file list. + """ + ws = os.path.join(os.path.dirname(__file__), "..", "examples", "ladybird", + "workspace") + index = open(os.path.join(ws, "vcpkg_index.bzl")).read() + for tool in ("ninja-linux-1.13.2.zip", "cmake-4.4.0-linux-x86_64.tar.gz"): + assert tool in index, "%s is missing from the distfile index" % tool + distfiles = open(os.path.join(ws, "vcpkg_distfiles.bzl")).read() + assert "ninja-build/ninja/releases/download/v1.13.2/ninja-linux.zip" in distfiles + + # ...and bzlmod only creates a repo the MODULE names, so an http_file with no + # use_repo entry is invisible: that asymmetry is why the emitter emits the + # use_repo list too. + module = open(os.path.join(ws, "MODULE.bazel")).read() + names = re.findall(r"name = '(vcpkg_ninja[A-Za-z0-9_]*)'", distfiles) + assert names, "no ninja http_file found" + for n in names: + assert "'%s'" % n in module, "%s is fetched but not named in use_repo" % n From 2ef50cc1cd93b7cea8a07c020e0a4fff8d2a8e21 Mon Sep 17 00:00:00 2001 From: Ulf Adams Date: Tue, 11 Aug 2026 15:50:33 +0000 Subject: [PATCH 19/59] Ladybird: name the host tools vcpkg cannot download, and check them before the build Two more of Ulf's builds died 20 minutes in, one tool at a time: `Could not find nasm` from inside libvpx, then `gperf currently requires ... autoconf autoconf-archive automake libtoolize` from vcpkg_make.cmake. Same blind spot as finding 38 -- the capturing machine had the tool -- but a different class, and I had assumed the class was closed by pinning vcpkg-tools.json. On Linux vcpkg has NO download for these. nasm's three URLs and sha512 all sit inside `if(CMAKE_HOST_WIN32)`; six ports in the closure need it. So there is nothing to pin, and the honest statement is that this is the boundary of the port. What WAS fixable is how you find out: a preflight over a derived list, reporting every miss at once with the ports that need each and one pasteable apt line, instead of one tool per 20-minute build in an error that names the wrong place. It has two mechanisms, which is why my first attempt missed half of it: vcpkg-make never calls vcpkg_find_acquire_program, it uses bare find_program + FATAL_ERROR with an apt line -- and does it from a HELPER port, so the error names gperf while the requirement lives where gperf does not mention. A scan built from one example is calibrated to one example. Most of the work was suppressing false positives, because a preflight that demands packages you do not need is one the third person deletes. CLANG's call sites are MSVC-only; openssl's NASM/CLANG live in ports/openssl/windows/ guarded only by the parent's include(), so the PATH must be read; dav1d's `if(NOT VCPKG_TARGET_IS_WINDOWS) ... else()` means that else IS the Windows branch; and angle only WARNS about mesa-common-dev while separately FATAL_ERRORing about architectures, so the check anchors on the FATAL_ERROR's own text -- advice is not a requirement. autoconf-archive (m4 macros) and libltdl-dev (headers) have no binary to probe and are reported unverifiable rather than assumed satisfied: finding 35's rule in a third place. Verified negatively, which finding 38 could not be (sudo hangs here, so I could not hide /usr/bin/ninja): the probe is `command -v`, so a restricted $PATH is a machine without the tools -- both missing tools reported in one run, in one second. Confirmed in a real `bazel build //:vcpkg_installed`: aquery shows Meta/vcpkg_host_tools.tsv as a declared input, passed by path as argument 7 rather than found with `dirname $0` (an sh_binary's data lands in .runfiles/, the trap cargo_vendor.sh already documents), and the action ran the preflight and built on past it. Also: the README claimed 42 overlay files while the overlay held 43, stale since finding 38 added one. Now 44 and asserted against the overlay -- finding 39's lesson at the smallest scale. The environment notes have listed autoconf/nasm as apt prerequisites since the beginning; documenting a requirement is not checking it. glslangValidator is the same class and is still open, filed rather than pretended. 194/194, 12 new tests -- one per false positive above, because each was a real bug in my own derivation. --- docs/CASE-ladybird-migration.md | 125 ++++++- examples/ladybird/README.md | 42 ++- examples/ladybird/workspace/Meta/BUILD.bazel | 5 + .../workspace/Meta/emit_vcpkg_bazel.py | 311 ++++++++++++++++++ .../ladybird/workspace/Meta/vcpkg_build.sh | 64 ++++ .../workspace/Meta/vcpkg_host_tools.tsv | 26 ++ examples/ladybird/workspace/vcpkg.bzl | 18 +- tests/test_apply_overlay.py | 22 ++ tests/test_emit_vcpkg.py | 242 ++++++++++++++ tests/test_vcpkg_plumbing.py | 38 +++ 10 files changed, 885 insertions(+), 8 deletions(-) create mode 100644 examples/ladybird/workspace/Meta/vcpkg_host_tools.tsv diff --git a/docs/CASE-ladybird-migration.md b/docs/CASE-ladybird-migration.md index 7905a9a..021c5b1 100644 --- a/docs/CASE-ladybird-migration.md +++ b/docs/CASE-ladybird-migration.md @@ -2135,7 +2135,7 @@ remembering the rename), and the two upstream patches must be *applied*, not mer shipped. So the deliverable is a script, and the part worth keeping is `--verify`, which checks -what a file copy cannot: HEAD is the pinned commit, all 42 files are byte-identical, +what a file copy cannot: HEAD is the pinned commit, all 44 files are byte-identical, the patches are applied (`git apply --check -R` succeeding is the proof — a patch that reverse-applies cleanly is already in the tree), and the `.sh` files still have their executable bit. That last check exists because of an earlier bug of exactly this shape: @@ -2214,6 +2214,116 @@ resolves that hash to the Bazel-fetched file — the exact lookup that failed on machine. Suite 182/182, six new tests, one of which is the regression test for the bug report itself. +## Finding 39: the pin fixed one class; the next two failures were a different class + +Finding 38's fix was correct and did not survive contact. Ulf pulled it, rebuilt, and +got — twenty minutes in, from inside `libvpx`: + +``` +CMake Error at scripts/cmake/vcpkg_find_acquire_program.cmake:201 (message): + Could not find nasm. Please install it via your package manager: +``` + +Installed nasm, rebuilt, twenty more minutes, from `gperf`: + +``` +CMake Error at .../share/vcpkg-make/vcpkg_make.cmake:108 (message): + gperf currently requires the following programs from the system package + manager: + + autoconf autoconf-archive automake libtoolize +``` + +Same blind spot as finding 38 — *the capturing machine had the tool* — but a +different **class**, and I had assumed the class was closed. Finding 38's fix reads +vcpkg's `scripts/vcpkg-tools.json` and pins url+sha512 for the tools vcpkg fetches +for itself. That works only for tools vcpkg *can* fetch. On Linux: + +```cmake +set(program_name nasm) +set(apt_package_name "nasm") +if(CMAKE_HOST_WIN32) + set(download_urls "https://www.nasm.us/.../nasm-3.01-win64.zip" ...) +endif() +``` + +Three URLs and a sha512 — **all inside the Windows branch**. On Linux there is +nothing to pin. vcpkg probes the host, does not find it, and stops. Six ports in +this closure need it (`dav1d`, `ffmpeg`, `libjpeg-turbo`, `libvpx`, `openh264`, +`openssl`). So the honest statement is not "the pin is incomplete" but **"this is +the boundary of the port"** — and the thing that was actually broken was not the +hermeticity, it was *how you find out*. + +**Three classes of input, not two.** Ring 2 had a two-box model — distfiles Bazel +fetches, and vcpkg's own tools (finding 38's pin). The third box is tools that can +only come from the host, and it needs different treatment because there is no URL +to put in it. Naming a gap is not fixing it; but an unnamed gap costs 20 minutes +per member to discover, one at a time, in an error that points at the wrong place. + +**And it has two mechanisms, which is why my first attempt missed half of it.** +I generalised from `nasm`, shipped a scan of `vcpkg_find_acquire_program` call +sites, and that would not have caught the autotools failure at all: `vcpkg-make` +never calls `vcpkg_find_acquire_program`. It calls bare +`find_program(AUTORECONF NAMES autoreconf)` and raises `FATAL_ERROR` with an apt +line. Worse, it does so from a **helper port**, so the error names `gperf` while +the requirement lives in a file `gperf` does not mention. A scan built from one +example is a scan calibrated to one example. + +So the derivation covers both, and the list is *derived* from vcpkg's own scripts +(`emit_vcpkg_bazel.py --host-tools` -> committed `Meta/vcpkg_host_tools.tsv`), +never hand-written: a baseline bump that adds a requirement is a +regenerate-and-review, not a rediscovery. + +**Most of the work was suppressing false positives, and that is the finding.** A +preflight that demands packages you do not need is one the third person deletes. +Four separate ways the naive scan cried wolf, each needing a real distinction: + +- **`CLANG`.** Both call sites are behind `if(... STREQUAL "MSVC")`. Reading call + sites without evaluating their guards demands a 2 GB toolchain on every Linux + machine. +- **`openssl`'s `NASM` and `CLANG`.** In `ports/openssl/windows/portfile.cmake` — + guarded by nothing in the file, only by the `include()` in its parent. The + platform split is at the **path** level, so the path has to be read. +- **`else()` after a negated test.** `dav1d` is `if(NOT VCPKG_TARGET_IS_WINDOWS) + ... else()` — that `else` *is* the Windows branch. Treating every `else()` as + reachable imports the Windows-only `GASPREPROCESSOR`. +- **`angle`'s `mesa-common-dev`.** A `message(WARNING)`. The portfile *also* has an + unrelated `FATAL_ERROR` about architectures, so a file-level "does it contain + both a FATAL_ERROR and an apt line" check staples them together. **Advice is not + a requirement**; the check anchors on the text of the `FATAL_ERROR` itself. + +Two entries can only be *named*, not verified: `autoconf-archive` ships m4 macros +and `libltdl-dev` ships headers, so there is no binary to probe. They are reported +as unverifiable rather than assumed satisfied — finding 35's rule again, in a third +place: a check that cannot fail must not look like a check that passed. + +Verified negatively, which finding 38 could not be (`sudo` hangs in this sandbox, +so I could not hide `/usr/bin/ninja`). Here the probe is `command -v`, so a +restricted `$PATH` *is* a machine without the tools: + +``` +vcpkg_build: MISSING host tool: libtoolize|glibtoolize (apt: libtool) +vcpkg_build: needed by: vcpkg-make +vcpkg_build: MISSING host tool: nasm (apt: nasm) +vcpkg_build: needed by: dav1d,ffmpeg,libjpeg-turbo,libvpx,openh264 +vcpkg_build: sudo apt install libtool nasm autoconf-archive libltdl-dev +``` + +Both, from one run, in one second, with the ports that need each and one pasteable +line — against the two failures that cost Ulf 40 minutes to learn two package +names. Confirmed in a real `bazel build //:vcpkg_installed`: the TSV is a declared +input (`aquery` shows `Meta/vcpkg_host_tools.tsv`), and the action ran the +preflight and proceeded into the build. Suite 193/193, 11 new tests — one per false +positive above, because each was a real bug in my own derivation. + +**The retrospective bit.** The environment notes at the bottom of this document +have said "plus autoconf/nasm/glslang/mesa GL dev libs" since the beginning. The +requirement was *documented and never checked* — so it was invisible to everyone +who did not read the bottom of a 2,300-line file, which is everyone. Prose in a +case study is not a preflight. `glslangValidator` is the same shape and is still +open: two genrules in `codegen_root.bzl` name `/usr/bin/glslangValidator`, and +`vcpkg_installed` does not ship it. + ## Plan for the rest - **Ring 1c — BUILD generation to compile+link parity, bottom-up.** AK → @@ -2283,7 +2393,8 @@ And with findings 33, 34 **and 35**, Bazel fetches and builds the 77 vcpkg ports the 10 Rust crates, `flapc`, `cranelift-compiler` and every generated header itself, with vcpkg's own downloader reaching the network zero times. It is **not** yet a clone-and-build: finding 36 lists the four inputs a fresh clone still lacks and the -one vcpkg port that bypasses the pin with `pip`. +one vcpkg port that bypasses the pin with `pip`; findings 38 and 39 add the two +classes of *host tool* that a capture on one machine structurally cannot see. `Build/full` is now only the *converter's* input — the model the emitters read and the baseline the parity harness diffs — not the build's, and this time that is measured: **0 targets under `Build/full` in the `cquery` closure of all six @@ -2300,7 +2411,9 @@ instructive part of this migration — see finding 35. from scratch (2,700 C++ plus the vcpkg and Rust rings); LibWeb alone 1,961 TUs (1,273 checked-in + 688 generated) with **zero** define/flag/include discrepancies vs CMake; 1,379/1,379 generated files -byte-identical -> now 1,408/1,408 with every one of the build's 586 ninja CUSTOM_COMMANDs accounted for (findings 25-27); 77 vcpkg ports and 154 crates.io crates fetched and built by Bazel; **0 targets under `Build/full` in the closure of all six binaries (down from 741) and 0 absolute host paths in the emitted BUILD files** (finding 35); a fresh clone still needs 4 inputs the overlay does not carry and one port reaches PyPI via pip (finding 36 — found by actually cloning); 36 findings, 3 of them real any2bazel engine fixes with +byte-identical -> now 1,408/1,408 with every one of the build's 586 ninja CUSTOM_COMMANDs accounted for (findings 25-27); 77 vcpkg ports and 154 crates.io crates fetched and built by Bazel; **0 targets under `Build/full` in the closure of all six binaries (down from 741) and 0 absolute host paths in the emitted BUILD files** (finding 35); a fresh clone still needs 4 inputs the overlay does not carry and one port reaches PyPI via pip (finding 36 — found by actually cloning), and the host tools vcpkg cannot download are +now named and preflighted rather than discovered one 20-minute build at a time +(findings 38, 39); 39 findings, 3 of them real any2bazel engine fixes with regression tests. **One number in this scoreboard was wrong for most of the migration, and it is @@ -2315,7 +2428,11 @@ this class of bug, and each time I widened what I removed, it found another. - Toolchain via apt (needs passwordless sudo): cmake, ninja, ccache, build-essential, Qt6 (`qt6-base-dev` etc., 6.10.2), plus autoconf/nasm/ - glslang/mesa GL dev libs. Rust via rustup (`~/.cargo`). Node/vcpkg bootstrap + glslang/mesa GL dev libs. **The vcpkg half of that list is no longer prose:** + `Meta/vcpkg_host_tools.tsv` is derived from vcpkg's own scripts and checked + before the build starts (finding 39). This line said "autoconf/nasm" for + months while both failures below were waiting to happen — documenting a + requirement is not checking it. Rust via rustup (`~/.cargo`). Node/vcpkg bootstrap per `Meta/Utils/build_vcpkg.py`. - `~/lb-env.sh` sets `LADYBIRD_SOURCE_DIR`, `VCPKG_ROOT`, CA certs. - Disk ~126G, RAM 16G, 16 cores. vcpkg from-source (no cache hit) for diff --git a/examples/ladybird/README.md b/examples/ladybird/README.md index b35e2a6..ca500d4 100644 --- a/examples/ladybird/README.md +++ b/examples/ladybird/README.md @@ -118,7 +118,7 @@ knowing, because both were mistakes I made first: ### Reproducing the tree on another machine The overlay is **not a fork**: it is a pinned upstream Ladybird commit + two patches -+ 42 Bazel files that sit alongside CMake's. [`apply_overlay.sh`](apply_overlay.sh) is ++ 44 Bazel files that sit alongside CMake's. [`apply_overlay.sh`](apply_overlay.sh) is that sentence made executable, because *"copy `workspace/` over a clone"* has four ways to be silently wrong — and every one of them was found by running it, not by reading it: @@ -145,7 +145,7 @@ reading it: **after** the prefetch; the script defers it and says so. `--verify` checks a tree without changing it, and checks the things a file copy -cannot: that HEAD is the pinned commit, that all 42 files are byte-identical, that +cannot: that HEAD is the pinned commit, that all 44 files are byte-identical, that the patches are **applied** (`git apply --check -R` succeeding is the proof — a patch that reverse-applies cleanly is already in the tree), and that the `.sh` files kept their **executable bit**, which is tree state a careless copy drops and which then @@ -157,6 +157,27 @@ outputs), `Meta/fetch_vcpkg_git_archives.py` reproduced **4/4** archives verifie against the pinned SHA512s, and `bazel build` then built the 76 vcpkg ports offline and linked `//:LibHTTP`. +#### One thing the overlay cannot carry: vcpkg's host tools + +```sh +sudo apt install nasm autoconf automake libtool autoconf-archive libltdl-dev +``` + +Not a courtesy list — the set is derived from vcpkg's own scripts into +[`Meta/vcpkg_host_tools.tsv`](workspace/Meta/vcpkg_host_tools.tsv), and +`vcpkg_build.sh` checks all of it before building anything, printing the ports that +need each missing tool and this exact line. vcpkg has **no Linux download** for +these (its `nasm` URLs are inside `if(CMAKE_HOST_WIN32)`; `vcpkg-make` demands +autotools via `find_program` + `FATAL_ERROR`), so they cannot be pinned the way the +76 distfiles are — the gap is named rather than papered over. Before the check +existed, each one cost a full ~20-minute build to discover, and the error named the +wrong place: `nasm` surfaced from inside `libvpx`, autotools from `gperf` via a +helper port that neither mentions (finding 39). + +`glslangValidator` is the same class and still **not** covered: two genrules name +`/usr/bin/glslangValidator`, and `vcpkg_installed` does not ship it — `apt install +glslang-tools` for now (see [known gaps](#known-gaps)). + ### The HSTS table: pinned downstream, because upstream is not ours to fix `Meta/CMake/hsts_preload.cmake` downloads Chromium's @@ -258,12 +279,13 @@ an FFI header collision and an entire missing Rust target. See | `vcpkg_distfiles.bzl`, `vcpkg_index.bzl`, `vcpkg_extension.bzl`, `vcpkg_git_archives.bzl` | One `http_file` per distfile, the sha512→label index the asset script resolves through, the module extension that creates the repos, and the 4 `vcpkg_from_git` archives. **Generated** by `Meta/emit_vcpkg_bazel.py` | | `vcpkg.bzl`, `Meta/vcpkg_build.sh` | `vcpkg_tree`: builds the whole dep tree as an ordinary Bazel action with `x-block-origin`, so it reaches the network zero times. Deliberately not a `repository_rule` | | `Meta/vcpkg_tool_assets.tsv` | vcpkg's OWN host tools (cmake, ninja) — url + sha512 + the filename vcpkg looks for. **Separate from the asset capture on purpose:** `vcpkg_find_acquire_program` probes the host first, so a tool the capturing machine already had is never downloaded and never captured (that is how ninja went unpinned; finding 38). Regenerate with `emit_vcpkg_bazel.py --capture-tools` | +| `Meta/vcpkg_host_tools.tsv` | The tools that must come **from the host**, because vcpkg has no Linux download for them at all — `nasm` (6 ports) and the autotools set. The third class of input, and the one that cannot be pinned: `vcpkg_find_acquire_program(NASM)` has URLs only inside `if(CMAKE_HOST_WIN32)`, and `vcpkg-make` demands `autoconf`/`automake`/`libtool` via bare `find_program` + `FATAL_ERROR`. So this file does not close the gap, it **names** it — and `vcpkg_build.sh` checks the whole list before building anything, so a machine missing three tools is told all three in one second instead of one per 20-minute build (finding 39). Regenerate with `emit_vcpkg_bazel.py --host-tools` | | `Meta/vcpkg_capture_assets.sh` | Records the 76-distfile pin, by *being* vcpkg's asset cache. The one run allowed to fetch | | `Meta/fetch_vcpkg_git_archives.py` | Produces the 4 `vcpkg_from_git` tarballs **without CMake**: takes the list from the committed pin, resolves each clone URL out of the portfiles, then `git clone` + `git -c core.autocrlf=false archive ` and **verifies against the pinned SHA512**. 4/4 byte-identical | | `Meta/vcpkg_capture_git_archives.sh` | Regenerates that pin with `vcpkg install --only-downloads` (~6 min, no compilation, no CMake) — vcpkg as the instrument, since which git externals are used is decided by feature-conditional CMake code, not by portfile text | | `hsts_preload.bzl` | Chromium's HSTS preload table as one `http_file`, pinned to a **commit** + sha256 — the downstream pin for the one input upstream CMake fetches from `main`. **Generated** by `Meta/pin_hsts_preload.py` | | `Meta/pin_hsts_preload.py` | Re-pins it: resolves the newest commit touching the path, downloads it, writes the hash it **measured**; `--expect-same-as` refuses to write unless the pinned bytes equal the file CMake downloaded (parity guard) | -| `apply_overlay.sh` | Reproduces the whole tree on another machine: clone at the pinned Ladybird commit, apply the two patches, copy the 42 overlay files (with the `bazelrc.txt` → `.bazelrc` rename), run the vcpkg prefetch, then stage the one file that must come after it. `--verify` checks an existing tree — commit, bytes, patches-applied, exec bits — and changes nothing | +| `apply_overlay.sh` | Reproduces the whole tree on another machine: clone at the pinned Ladybird commit, apply the two patches, copy the 44 overlay files (with the `bazelrc.txt` → `.bazelrc` rename), run the vcpkg prefetch, then stage the one file that must come after it. `--verify` checks an existing tree — commit, bytes, patches-applied, exec bits — and changes nothing | | `bazelrc.txt` | → `.bazelrc`. Global copts/defines/linkopts mirrored from `Meta/CMake/compile_options.cmake` | | `BUILD.bazel` | Root package: 34 libraries, the 5 executables, Qt moc/rcc genrules. **Generated** by `Meta/emit_build_bazel.py` | | `codegen_root.bzl` | Non-LibWeb generator genrules (IPC endpoints, LibJS Bytecode/Op, HSTS table, WebGL replayer, TIFF tag tables, the two SPIR-V shader headers, and the chained `generate_interpreter_layout` → `flapc` interpreter assembly). **Generated** by `Meta/emit_root_codegen_bazel.py` | @@ -671,6 +693,20 @@ Honest inventory of what stops this from being a clone-and-build. rather than a hash it invented.** Pin the wrong revision (a release tag, say) and the hermeticity gap becomes a parity gap. + **And a fourth class, which no pin can close: host tools.** vcpkg has no Linux + download for `nasm` or the autotools set — its `nasm` URLs live inside + `if(CMAKE_HOST_WIN32)`, and `vcpkg-make` demands `autoconf`/`automake`/`libtool` + with a bare `find_program` and a `FATAL_ERROR`. So they are *named* instead: + `Meta/vcpkg_host_tools.tsv`, derived from vcpkg's own scripts, checked by + `vcpkg_build.sh` before anything builds (finding 39). Every one of these was + invisible for exactly the reason in this gap's first sentence — **it was already + on the machine** — and each cost a full ~20-minute build to find, in an error + naming the wrong place. `glslangValidator` is the same class and is **not** yet + named: two genrules in `codegen_root.bzl` hardcode `/usr/bin/glslangValidator` + and `vcpkg_installed` does not ship it, so it needs a pinned binary or a repo + rule that fails legibly. Fixing *how you find out* is not the same as fixing the + dependency, and this gap is only the former. + 8. **"Zero network access" is true of vcpkg's downloader, not of the vcpkg action.** The pin + `x-block-origin` is verified — a distfile missing from the index is a hard error, not a fetch. But `x-block-origin` only governs vcpkg's diff --git a/examples/ladybird/workspace/Meta/BUILD.bazel b/examples/ladybird/workspace/Meta/BUILD.bazel index 49675f6..7052042 100644 --- a/examples/ladybird/workspace/Meta/BUILD.bazel +++ b/examples/ladybird/workspace/Meta/BUILD.bazel @@ -21,6 +21,11 @@ sh_binary( srcs = ["vcpkg_build.sh"], ) +# The host-prerequisite list, read by vcpkg_build.sh at action time and named as +# an input by vcpkg.bzl (NOT as data on the sh_binary above: runfiles are not +# beside the wrapper Bazel execs, so the script cannot find it by dirname $0). +exports_files(["vcpkg_host_tools.tsv"]) + # Ring 2 part 3: the driver for the cargo build actions (see //:cargo.bzl). It # assembles the vendor directory from crates Bazel fetched, points CARGO_HOME at # action-local scratch, and runs `cargo rustc --offline --locked` -- so the build diff --git a/examples/ladybird/workspace/Meta/emit_vcpkg_bazel.py b/examples/ladybird/workspace/Meta/emit_vcpkg_bazel.py index b539747..ca63a04 100644 --- a/examples/ladybird/workspace/Meta/emit_vcpkg_bazel.py +++ b/examples/ladybird/workspace/Meta/emit_vcpkg_bazel.py @@ -484,6 +484,314 @@ def load_tool_pins(path=None): return out +# --------------------------------------------------------------------------- +# The third class of input: tools vcpkg CANNOT download at all. +# +# Finding 38 pinned the tools vcpkg fetches for itself (cmake, ninja) after ninja +# went unpinned because the capturing machine already had it. The same blind spot +# has a strictly worse case behind it, and nasm is it: on Linux +# `vcpkg_find_acquire_program(NASM)` has NO download_urls -- the Windows branch +# has three URLs and a sha512, the Linux branch has an apt package name and +# nothing else. So there is no pin to add. vcpkg probes the host, does not find +# it, and stops: +# +# CMake Error at scripts/cmake/vcpkg_find_acquire_program.cmake:201 (message): +# Could not find nasm. Please install it via your package manager +# +# ~20 minutes into `bazel build //:vcpkg_installed`, from inside libvpx, naming a +# scratch path. Six ports in this closure ask for it (dav1d, ffmpeg, +# libjpeg-turbo, libvpx, openh264, openssl), and the machine this was developed on +# has nasm, perl, pkg-config and python3 all preinstalled -- which is exactly why +# neither the capture NOR the tools.json pin could ever have revealed this. +# +# These are HOST PREREQUISITES, and calling them that is the point: they are not a +# hermeticity gap we can close by pinning a URL, they are the current, honest +# boundary of the port. Two things follow, and both are implemented here: +# +# 1. The set is DERIVED from vcpkg's own scripts (which programs the closure's +# portfiles invoke, and which of those have no Linux download), not from a +# hand-written list that rots at the next baseline bump. +# 2. It is CHECKED BEFORE the build, all at once, by name, with the ports that +# need each one -- so a machine missing three of them learns all three in one +# second, instead of one per 20-minute build. +# +# The real fix -- pinning these as Bazel-fetched binaries so the build needs no +# host tools at all -- is the same open work as glslangValidator, and is filed +# rather than pretended. +HOST_TOOLS_TSV = os.path.join(os.path.dirname(os.path.abspath(__file__)), + "vcpkg_host_tools.tsv") + + +def _acquire_script(program, vcpkg=None): + p = os.path.join(vcpkg or VCPKG, "scripts", "cmake", + "vcpkg_find_acquire_program(%s).cmake" % program) + return p if os.path.exists(p) else None + + +# A condition that selects Windows. `MSVC` counts: an +# `ASM_COMPILER_ID STREQUAL "MSVC"` branch is Windows-only in practice, and that +# branch is the ONLY thing in this closure that asks for CLANG -- read naively it +# reports clang as a Linux prerequisite, which it is not. +_WIN_TOKEN = re.compile( + r'\b(CMAKE_HOST_WIN32|VCPKG_HOST_IS_WINDOWS|VCPKG_TARGET_IS_WINDOWS|WIN32|MSVC|' + r'VCPKG_TARGET_IS_UWP|VCPKG_TARGET_IS_MINGW)\b') + + +def _classify(cond): + """'win' / 'notwin' / None for a CMake if-condition.""" + if not _WIN_TOKEN.search(cond): + return None + return "notwin" if re.match(r'\s*NOT\b', cond) else "win" + + +def strip_windows_blocks(text): + """CMake source -> only the lines reachable on a non-Windows host. + + Both halves of this analysis need it, which is why it is one function. When + reading an acquire-script, every download URL lives in the Windows branch, so + NOT skipping it is exactly the mistake that hides the gap. When reading a + portfile, a `vcpkg_find_acquire_program` call inside a Windows branch is not a + prerequisite here at all. + + `else()` is the subtle case and is handled rather than guessed: the else of + `if(NOT VCPKG_TARGET_IS_WINDOWS)` IS the Windows branch (dav1d's is written + exactly that way), so each if-chain remembers whether any of its conditions + was a negated Windows test and suppresses its else accordingly. + """ + out, stack = [], [] + for raw in text.splitlines(): + s = raw.strip() + m = re.match(r'if\s*\((.*)', s, re.S) + if m: + cls = _classify(m.group(1)) + stack.append({"suppressed": cls == "win", "saw_notwin": cls == "notwin"}) + continue + m = re.match(r'elseif\s*\((.*)', s, re.S) + if m and stack: + cls = _classify(m.group(1)) + stack[-1]["suppressed"] = cls == "win" + stack[-1]["saw_notwin"] = stack[-1]["saw_notwin"] or cls == "notwin" + continue + if re.match(r'else\s*\(', s) and stack: + stack[-1]["suppressed"] = stack[-1]["saw_notwin"] + continue + if re.match(r'endif\b', s): + if stack: + stack.pop() + continue + if not any(f["suppressed"] for f in stack): + out.append(raw) + return "\n".join(out) + + +def _has_non_windows_download(text): + """Does this acquire-script offer a download vcpkg can use on Linux? + + A tool with no such download is one vcpkg can only take from the host. + """ + body = strip_windows_blocks(text) + # `z_use_vcpkg_fetch` delegates to `vcpkg fetch`, which IS covered by the + # tools.json pin (that is how ninja is handled) -- so it is not a host + # prerequisite even though it sets no download_urls itself. + if "z_use_vcpkg_fetch" in body: + return True + return bool(re.search(r'set\s*\(\s*download_urls', body)) + + +def _port_cmake_files(port, vcpkg=None): + """Every .cmake/.json in a port that is reachable on a non-Windows host.""" + vc = vcpkg or VCPKG + for base in (OVERLAY, os.path.join(vc, "ports")): + d = os.path.join(base, port) + if not os.path.isdir(d): + continue + for root, _dirs, files in os.walk(d): + # A port can split itself by platform at the FILE level rather than + # with an if(): openssl's portfile.cmake picks between + # `unix/portfile.cmake` and `windows/portfile.cmake`, and the windows + # one asks for CLANG and NASM at TOP LEVEL, guarded by nothing this + # file can see. Skipping windows/ directories is therefore not + # cosmetic -- without it openssl reports clang as a Linux + # prerequisite. Conservative by construction: a Windows-only + # directory can only hold Windows-only requirements. + rel = os.path.relpath(root, d).split(os.sep) + if any(p.lower() in ("windows", "win32", "uwp", "mingw") for p in rel): + continue + for fn in sorted(files): + if fn.endswith((".cmake", ".json")): + yield os.path.join(root, fn) + return + + +# Programs a port probes with a bare `find_program` and then hard-fails on. This +# is a SECOND mechanism, found the hard way after the first one shipped: gperf +# died on `autoconf autoconf-archive automake libtoolize` from +# vcpkg-make/vcpkg_make.cmake, which never calls vcpkg_find_acquire_program at +# all -- it calls find_program(AUTORECONF NAMES autoreconf) and raises +# FATAL_ERROR listing apt packages. And it does it from a HELPER port +# (vcpkg-make), so the error names gperf while the requirement lives somewhere +# gperf does not mention. Enumerating only the acquire-program calls would have +# missed every one of these, which is why the preflight scans for both. + + +# Which binary proves an apt package is installed. Only needed where the two +# names differ; anything absent is probed under its own name, and a package that +# ships no executable at all maps to "" and is reported as unprobeable. +_PKG_BINARY = { + "autoconf": ["autoreconf", "autoconf"], + "automake": ["aclocal", "automake"], + "libtool": ["libtoolize", "glibtoolize"], + "gettext": ["autopoint"], + "gtk-doc-tools": ["gtkdocize"], + "autoconf-archive": [], # m4 macros, no binary + "libltdl-dev": [], # headers + "pkg-config": ["pkg-config", "pkgconf"], +} + + +def _fatal_package_requirements(text): + """Packages a file DEMANDS from the system package manager, and how to probe. + + Anchored on the text of the `message(FATAL_ERROR ...)` itself, not on the file + containing a FATAL_ERROR somewhere. That distinction is the difference between + a useful preflight and a noisy one: angle's portfile has both a FATAL_ERROR + (about an unsupported architecture) and a WARNING recommending + mesa-common-dev, and a file-level check staples them together and demands a + package no build step actually requires. A WARNING is advice; only a + FATAL_ERROR is a prerequisite. + + Returns {apt-package: [binaries that prove it, possibly empty]}. + """ + out = {} + for m in re.finditer(r'message\(\s*FATAL_ERROR\s+(.*?)\)\s*$', + strip_windows_blocks(text), re.S | re.M): + msg = m.group(1) + if not re.search(r'package manager|apt(?:-get)? install', msg): + continue + for am in re.finditer(r'apt(?:-get)? install ([^\n"\\]*)', msg): + for pkg in am.group(1).split(): + if not re.fullmatch(r'[\w.+-]+', pkg) or pkg == "sudo": + continue + out.setdefault(pkg, _PKG_BINARY.get(pkg, [pkg])) + return out + + +def host_tool_requirements(ports=None, vcpkg=None): + """-> {key: (binary, apt-package, [ports that need it], [alternatives])}. + + The programs the closure's portfiles need FROM THE HOST, by both mechanisms: + a `vcpkg_find_acquire_program` for a tool with no Linux download, and a bare + `find_program` in a file that hard-fails naming a package manager. + + This is a static parse and inherits its limits (a program named through a + variable is invisible). Unlike the distfile parse there is no instrument that + does better, because the failure only manifests on a machine that LACKS the + tool -- so a capture on a machine that has everything sees nothing, which is + precisely how nasm and autoconf each got to fail at minute 20. Under-reporting + degrades to that same late error; it never invents a prerequisite. + """ + vc = vcpkg or VCPKG + if ports is None: + ports = sorted(pinned_versions()) + out, asked = {}, {} + for port in ports: + for path in _port_cmake_files(port, vc): + with open(path, errors="replace") as f: + text = f.read() + # Mechanism 1: only the calls REACHABLE on a non-Windows host. openssl + # and vcpkg-make both ask for CLANG, but only inside an MSVC/Windows + # branch -- counting those made clang a prerequisite of a Linux build, + # which is a false alarm, and a preflight that cries wolf gets deleted. + for m in re.finditer( + r'vcpkg_find_acquire_program\(\s*([A-Z0-9_]+)', + strip_windows_blocks(text)): + asked.setdefault(m.group(1), set()).add(port) + # Mechanism 2: a FATAL_ERROR naming system packages directly. + for apt, binaries in _fatal_package_requirements(text).items(): + cur = out.setdefault(apt, [binaries[0] if binaries else "", apt, + set(), binaries[1:]]) + cur[2].add(port) + for program, users in asked.items(): + script = _acquire_script(program, vc) + if not script: + continue + with open(script, errors="replace") as f: + text = f.read() + if _has_non_windows_download(text): + continue + # program_name from the NON-Windows branch: PYTHON3 sets `python` for + # Windows and `python3` for everything else, and probing for `python` on + # Linux asks for a binary that has not existed by default for years. + body = strip_windows_blocks(text) + name = re.search(r'set\(program_name\s+"?([\w.+-]+)', body) + apt = re.search(r'set\(apt_package_name\s+"?([\w.+-]+)', body) + binary = name.group(1) if name else program.lower() + cur = out.setdefault(binary, [binary, apt.group(1) if apt else "", + set(), []]) + cur[2] |= users + return {k: (v[0], v[1], sorted(v[2]), v[3]) for k, v in out.items()} + + +def emit_host_tools(reqs): + """Write the host-prerequisite TSV: program, binary, apt package, ports.""" + print("# Tools the build needs FROM THE HOST, because vcpkg cannot supply them") + print("# on Linux at all. GENERATED by emit_vcpkg_bazel.py --host-tools.") + print("#") + print("# NOT the same class as vcpkg_tool_assets.tsv. Those are tools vcpkg") + print("# fetches for ITSELF, and the fix there was a pin. There is no URL to") + print("# pin for these, by either of the two mechanisms that produce them:") + print("#") + print("# 1. vcpkg_find_acquire_program() whose download URLs are ONLY") + print("# inside `if(CMAKE_HOST_WIN32)`. On Linux vcpkg probes the host and") + print("# hard-fails: 'Could not find nasm. Please install it via your") + print("# package manager'.") + print("# 2. a message(FATAL_ERROR ...) naming apt packages outright, after a") + print("# bare find_program. gperf died this way on autoconf/automake/") + print("# libtool -- and the requirement lives in the vcpkg-make HELPER") + print("# port, so the error names gperf while the cause names neither.") + print("#") + print("# So this file does not pretend to close the gap: it NAMES the gap, and") + print("# vcpkg_build.sh checks the whole list up front -- a machine missing") + print("# three tools is told about three tools in one second, instead of one") + print("# per 20-minute build (nasm surfaced from libvpx at minute ~20).") + print("#") + print("# ports-that-need-it is where the requirement is WRITTEN, which for") + print("# vcpkg-make is not where it fails: every autotools port using it can.") + print("#") + print("# An empty binary means the package ships no executable to probe for") + print("# (autoconf-archive is m4 macros, libltdl-dev is headers), so the") + print("# preflight can only name it -- it cannot verify it.") + print("#") + print("# binary-or-alternatives\tapt-package\tports-that-need-it") + for program in sorted(reqs): + binary, apt, users, alts = reqs[program] + names = "|".join([binary] + list(alts)) if binary else "-" + print("%s\t%s\t%s" % (names, apt or program, ",".join(users))) + + +def load_host_tools(path=None): + """The committed host-prerequisite list -> [(alternatives, apt, ports)]. + + `alternatives` is empty for a package with no binary to probe. + """ + path = path or HOST_TOOLS_TSV + if not os.path.exists(path): + return [] + out = [] + with open(path) as f: + for line in f: + line = line.split("#", 1)[0].strip() + if not line: + continue + parts = line.split("\t") + if len(parts) != 3: + continue + names, apt, users = parts + out.append(([n for n in names.split("|") if n and n != "-"], apt, + [p for p in users.split(",") if p])) + return out + + def emit_tool_pins(tools): """Write the tool pin TSV: url, sha512, the filename vcpkg looks for.""" print("# vcpkg's OWN host tools, pinned from its scripts/vcpkg-tools.json at the") @@ -824,6 +1132,9 @@ def main(): if "--capture-tools" in sys.argv: emit_tool_pins(tool_distfiles()) return 0 + if "--host-tools" in sys.argv: + emit_host_tools(host_tool_requirements()) + return 0 if "--git-archives" in sys.argv: dl = sys.argv[sys.argv.index("--git-archives") + 1] archives, byproducts = observed_git_archives(dl, distfiles) diff --git a/examples/ladybird/workspace/Meta/vcpkg_build.sh b/examples/ladybird/workspace/Meta/vcpkg_build.sh index 078d210..6151b06 100755 --- a/examples/ladybird/workspace/Meta/vcpkg_build.sh +++ b/examples/ladybird/workspace/Meta/vcpkg_build.sh @@ -52,6 +52,7 @@ VCPKG_TREE="${3:?vcpkg checkout}" SRC="${4:?ladybird source root}" TRIPLET="${5:-x64-linux-dynamic}" WHEELS="${6-}" +HOST_TOOLS="${7-}" # vcpkg's buildtrees peak around 3 GB and $TMPDIR is frequently a tmpfs sized well # under that (7.9 GB here, shared with everything else), which surfaces as @@ -158,6 +159,69 @@ exit 1 EOF chmod +x "$WORK/fetch.sh" +# --- host prerequisites, checked ALL AT ONCE, before anything builds --------- +# +# The third class of input, after Bazel-fetched distfiles and vcpkg's own pinned +# tools: programs vcpkg can only take from the host, because on Linux it has no +# download for them at all (finding 39). These cannot be pinned here -- that is +# upstream work -- but the way they FAILED was fixable, and this is that fix. +# +# What they cost before this existed: `bazel build //:vcpkg_installed` ran ~20 +# minutes, then died inside libvpx with "Could not find nasm", naming a scratch +# path under bazel-out. Install nasm, run 20 minutes again, die in gperf on +# "autoconf autoconf-archive automake libtoolize" -- a message from the vcpkg-make +# helper port that names gperf and mentions neither. One tool per build, each +# discovered at the price of the whole build, in an error that points at the wrong +# place. Three missing tools is an hour to learn three package names. +# +# So check the whole committed list up front and report EVERY miss at once, with +# the ports that need each one and a single apt line to paste. Costs milliseconds. +# The list is DERIVED from vcpkg's own scripts by `--host-tools`, not hand-written, +# so a baseline bump that adds a requirement is a regenerate-and-review rather +# than a rediscovery. +if [ -n "$HOST_TOOLS" ] && [ -f "$HOST_TOOLS" ]; then + missing_pkgs="" + unprobeable="" + while IFS=$'\t' read -r names apt users; do + case "$names" in ''|'#'*) continue ;; esac + # No binary to probe: autoconf-archive ships m4 macros, libltdl-dev ships + # headers. Reporting these as satisfied would be a lie, so they are listed + # as unverified rather than silently assumed -- the pattern from finding 35: + # a check that cannot fail must not look like a check that passed. + if [ "$names" = "-" ]; then + unprobeable="$unprobeable $apt" + continue + fi + # `names` is alternatives separated by |: libtoolize OR glibtoolize. + found="" + old_ifs="$IFS"; IFS='|' + for n in $names; do + IFS="$old_ifs" + if command -v "$n" > /dev/null 2>&1; then found=1; break; fi + IFS='|' + done + IFS="$old_ifs" + if [ -z "$found" ]; then + echo "vcpkg_build: MISSING host tool: $names (apt: $apt)" >&2 + echo "vcpkg_build: needed by: $users" >&2 + missing_pkgs="$missing_pkgs $apt" + fi + done < "$HOST_TOOLS" + if [ -n "$missing_pkgs" ]; then + echo "vcpkg_build:" >&2 + echo "vcpkg_build: vcpkg has NO download for these on Linux -- it probes the" >&2 + echo "vcpkg_build: host and hard-fails partway through the build. Install them:" >&2 + echo "vcpkg_build:" >&2 + echo "vcpkg_build: sudo apt install$missing_pkgs${unprobeable}" >&2 + echo "vcpkg_build:" >&2 + echo "vcpkg_build: (list: Meta/vcpkg_host_tools.tsv, regenerate with" >&2 + echo "vcpkg_build: Meta/emit_vcpkg_bazel.py --host-tools)" >&2 + exit 1 + fi + [ -n "$unprobeable" ] && echo \ + "vcpkg_build: note: cannot verify$unprobeable (no binary to probe for)" >&2 +fi + # --- a writable vcpkg root (vcpkg writes buildtrees/downloads/packages) ----- ROOT="$WORK/root" mkdir -p "$ROOT" diff --git a/examples/ladybird/workspace/Meta/vcpkg_host_tools.tsv b/examples/ladybird/workspace/Meta/vcpkg_host_tools.tsv new file mode 100644 index 0000000..bcec671 --- /dev/null +++ b/examples/ladybird/workspace/Meta/vcpkg_host_tools.tsv @@ -0,0 +1,26 @@ +# Tools the build needs FROM THE HOST, because vcpkg cannot download +# them on Linux at all. GENERATED by emit_vcpkg_bazel.py --host-tools. +# +# NOT the same class as vcpkg_tool_assets.tsv. Those are tools vcpkg +# fetches for itself, and the fix there was a pin. For these, +# vcpkg_find_acquire_program() has download URLs ONLY inside its +# `if(CMAKE_HOST_WIN32)` branch -- on Linux it probes the host and +# hard-fails with 'Could not find '. There is no URL to pin, so this +# file does not pretend to close the gap: it NAMES the gap, and +# vcpkg_build.sh checks the whole list up front so a machine missing +# three tools is told about three tools in one second, rather than one +# per 20-minute build (nasm surfaced from libvpx at minute ~20). +# +# An empty BINARY means the package ships no executable to probe for +# (autoconf-archive is m4 macros), so the preflight can only name it. +# +# binary-or-alternatives apt-package ports-that-need-it +autoreconf|autoconf autoconf vcpkg-make +- autoconf-archive vcpkg-make +aclocal|automake automake vcpkg-make +- libltdl-dev vcpkg-make +libtoolize|glibtoolize libtool vcpkg-make +nasm nasm dav1d,ffmpeg,libjpeg-turbo,libvpx,openh264 +perl perl libvpx,openssl +pkg-config pkg-config curl,ffmpeg,libavif,libxml2,vcpkg-make,vcpkg-tool-meson +python3 python3 icu,skia,vcpkg-gn,vcpkg-tool-meson diff --git a/examples/ladybird/workspace/vcpkg.bzl b/examples/ladybird/workspace/vcpkg.bzl index de49b6a..367c61f 100644 --- a/examples/ladybird/workspace/vcpkg.bzl +++ b/examples/ladybird/workspace/vcpkg.bzl @@ -43,7 +43,8 @@ def _vcpkg_tree_impl(ctx): outputs = [out], inputs = depset( [index] + distfile_inputs + ctx.files.python_wheels + - ctx.files.vcpkg_tree + ctx.files.source_root, + ctx.files.vcpkg_tree + ctx.files.source_root + + ctx.files._host_tools, ), executable = ctx.executable._build, arguments = [ @@ -56,6 +57,11 @@ def _vcpkg_tree_impl(ctx): # empty when there are none (which is then a hard failure for any port # that pip-installs, rather than a silent fetch). ",".join([f.path for f in ctx.files.python_wheels]), + # The host-prerequisite list, passed BY PATH rather than found with + # `dirname $0`: an sh_binary's data lands in .runfiles/, not + # beside the wrapper Bazel execs, so dirname finds nothing once the + # action is sandboxed -- the same trap cargo_vendor.sh documents. + ctx.file._host_tools.path, ], mnemonic = "VcpkgInstall", progress_message = "Building %d vcpkg ports (no network)" % len(ctx.attr.distfiles), @@ -132,6 +138,16 @@ vcpkg_tree = rule( executable = True, cfg = "exec", ), + "_host_tools": attr.label( + default = "//Meta:vcpkg_host_tools.tsv", + allow_single_file = True, + doc = """The tools this build needs FROM THE HOST (finding 39). + + vcpkg has no Linux download for these, so they cannot be pinned -- + what CAN be fixed is when you find out. The driver checks the whole + list before building anything, so a machine missing three of them is + told all three in one second instead of one per 20-minute build.""", + ), }, ) diff --git a/tests/test_apply_overlay.py b/tests/test_apply_overlay.py index d2213e6..adc7959 100644 --- a/tests/test_apply_overlay.py +++ b/tests/test_apply_overlay.py @@ -168,3 +168,25 @@ def test_both_patches_are_referenced_by_glob_not_by_name(): assert len(names) >= 2 for n in names: assert n not in text, "patch %s is named literally; use the glob" % n + + +def test_the_documented_overlay_file_count_matches_the_overlay(): + """The README's "N Bazel files" must be the number the script actually copies. + + It said 42 while the overlay held 43, and nothing noticed -- finding 38 added a + file and updated the prose in one place but not the others. That is finding + 39's lesson at the smallest possible scale: a documented number that nothing + checks is a number that is wrong as soon as it matters. The script derives the + list with `find`, so the overlay is the truth; this makes the prose answerable + to it. + """ + workspace = REPO / "examples" / "ladybird" / "workspace" + actual = len([p for p in workspace.rglob("*") + if p.is_file() and p.suffix != ".pyc"]) + readme = (REPO / "examples" / "ladybird" / "README.md").read_text() + claims = set(re.findall(r'(\d+) (?:Bazel|overlay) files', readme)) + claims |= set(re.findall(r'(?:all|the) (\d+) (?:Bazel |overlay )?files', readme)) + assert claims, "the README no longer states an overlay file count" + assert claims == {str(actual)}, \ + "README claims %s overlay files, the overlay has %d" % ( + sorted(claims), actual) diff --git a/tests/test_emit_vcpkg.py b/tests/test_emit_vcpkg.py index cac94bb..8e3dbf5 100644 --- a/tests/test_emit_vcpkg.py +++ b/tests/test_emit_vcpkg.py @@ -26,6 +26,7 @@ import importlib.util import os import re +import subprocess import tempfile _EMIT = os.path.join( @@ -341,3 +342,244 @@ def test_the_committed_pin_carries_ninja_and_cmake(): assert names, "no ninja http_file found" for n in names: assert "'%s'" % n in module, "%s is fetched but not named in use_repo" % n + + +# --- host prerequisites (finding 39) --------------------------------------- +# +# The class of input that CANNOT be pinned: tools vcpkg has no Linux download +# for. What is tested here is therefore not "the pin is complete" but "the gap is +# named accurately and early" -- and above all that it does not cry wolf, because +# a preflight with false positives demands packages nothing needs and gets +# deleted by the third person who hits it. + +def _acquire(tmpdir, program, body): + d = os.path.join(tmpdir, "scripts", "cmake") + os.makedirs(d, exist_ok=True) + with open(os.path.join(d, "vcpkg_find_acquire_program(%s).cmake" % program), + "w") as f: + f.write(body) + + +def _port(tmpdir, port, body, filename="portfile.cmake"): + d = os.path.join(tmpdir, "ports", port, os.path.dirname(filename)) + os.makedirs(d, exist_ok=True) + with open(os.path.join(tmpdir, "ports", port, filename), "w") as f: + f.write(body) + + +NASM_LIKE = """set(program_name nasm) +set(program_version 3.01) +set(brew_package_name "nasm") +set(apt_package_name "nasm") +if(CMAKE_HOST_WIN32) + set(download_urls "https://example.com/nasm-win64.zip") + set(download_filename "nasm-win64.zip") + set(download_sha512 %s) +endif() +""" % ("a" * 128) + + +def test_a_windows_only_download_is_a_host_prerequisite_on_linux(): + """The nasm bug: URLs exist, but only inside if(CMAKE_HOST_WIN32). + + Reading `download_urls` without asking WHICH BRANCH it is in reports nasm as + a pinnable download, which is how six ports' worth of hard dependency stayed + invisible until libvpx died on it 20 minutes into a build. + """ + with tempfile.TemporaryDirectory() as d: + _acquire(d, "NASM", NASM_LIKE) + _port(d, "libvpx", "vcpkg_find_acquire_program(NASM)\n") + reqs = emit.host_tool_requirements(ports=["libvpx"], vcpkg=d) + assert "nasm" in reqs, reqs + binary, apt, users, _alts = reqs["nasm"] + assert (binary, apt, users) == ("nasm", "nasm", ["libvpx"]) + + +def test_a_tool_with_a_real_linux_download_is_not_a_prerequisite(): + """meson/gn ARE downloadable on Linux -- demanding them from apt is wrong.""" + with tempfile.TemporaryDirectory() as d: + _acquire(d, "MESON", 'set(program_name meson)\n' + 'set(apt_package_name "meson")\n' + 'set(download_urls "https://example.com/meson.tar.gz")\n') + _port(d, "vcpkg-tool-meson", "vcpkg_find_acquire_program(MESON)\n") + reqs = emit.host_tool_requirements(ports=["vcpkg-tool-meson"], vcpkg=d) + assert reqs == {}, "a downloadable tool must not be a host prerequisite: %r" % reqs + + +def test_a_vcpkg_fetch_tool_is_not_a_prerequisite(): + """NINJA sets no download_urls: it delegates to `vcpkg fetch`, which IS pinned. + + Without this, the fix for finding 38 (pin ninja) and the fix for finding 39 + (name what cannot be pinned) contradict each other about ninja. + """ + with tempfile.TemporaryDirectory() as d: + _acquire(d, "NINJA", "z_use_vcpkg_fetch(NINJA)\n") + _port(d, "vcpkg-cmake", "vcpkg_find_acquire_program(NINJA)\n") + reqs = emit.host_tool_requirements(ports=["vcpkg-cmake"], vcpkg=d) + assert reqs == {}, "ninja comes from the tools.json pin, not the host: %r" % reqs + + +def test_a_windows_only_call_site_is_not_a_linux_prerequisite(): + """openssl and vcpkg-make ask for CLANG only under MSVC -- clang is not needed. + + The first version of this derivation reported clang, which is a false alarm + on every Linux machine, and a preflight that demands a 2GB toolchain nobody + uses is one that gets ignored. + """ + with tempfile.TemporaryDirectory() as d: + _acquire(d, "CLANG", 'set(program_name clang)\n' + 'set(apt_package_name "clang")\n') + _port(d, "vcpkg-make", """ +if(VCPKG_DETECTED_CMAKE_ASM_COMPILER_ID STREQUAL "MSVC") + vcpkg_find_acquire_program(CLANG) +endif() +""") + reqs = emit.host_tool_requirements(ports=["vcpkg-make"], vcpkg=d) + assert reqs == {}, "an MSVC-only call site is not a Linux prerequisite: %r" % reqs + + +def test_a_windows_only_subdirectory_is_not_scanned(): + """openssl splits by FILE: windows/portfile.cmake asks for CLANG at top level. + + No if() guards it -- the guard is the include() in the parent -- so only the + path says it is Windows-only. + """ + with tempfile.TemporaryDirectory() as d: + _acquire(d, "CLANG", "set(program_name clang)\n") + _acquire(d, "PERL", 'set(program_name perl)\nset(apt_package_name "perl")\n') + _port(d, "openssl", "vcpkg_find_acquire_program(CLANG)\n", + filename="windows/portfile.cmake") + _port(d, "openssl", "vcpkg_find_acquire_program(PERL)\n", + filename="unix/portfile.cmake") + reqs = emit.host_tool_requirements(ports=["openssl"], vcpkg=d) + assert set(reqs) == {"perl"}, \ + "windows/ must be skipped and unix/ must not be: %r" % reqs + + +def test_the_else_of_a_negated_windows_test_is_the_windows_branch(): + """if(NOT VCPKG_TARGET_IS_WINDOWS) ... else() <-- that else is Windows. + + dav1d is written exactly this way. Treating every else() as reachable puts + the Windows-only GASPREPROCESSOR into a Linux preflight. + """ + with tempfile.TemporaryDirectory() as d: + _acquire(d, "GASPREPROCESSOR", "set(program_name gas-preprocessor.pl)\n") + _acquire(d, "NASM", NASM_LIKE) + _port(d, "dav1d", """ +if(NOT VCPKG_TARGET_IS_WINDOWS) + vcpkg_find_acquire_program(NASM) +else() + vcpkg_find_acquire_program(GASPREPROCESSOR) +endif() +""") + reqs = emit.host_tool_requirements(ports=["dav1d"], vcpkg=d) + assert set(reqs) == {"nasm"}, reqs + + +GPERF_LIKE = """ +function(vcpkg_run_autoreconf shell_cmd work_dir) + find_program(ACLOCAL NAMES aclocal) + find_program(AUTORECONF NAMES autoreconf) + find_program(LIBTOOLIZE NAMES libtoolize glibtoolize) + if(missing) + message(FATAL_ERROR "${PORT} currently requires the following programs from the system package manager: + autoconf autoconf-archive automake libtoolize + + On Debian and Ubuntu derivatives: + sudo apt install autoconf autoconf-archive automake libtool +") + endif() +endfunction() +""" + + +def test_a_fatal_error_naming_apt_packages_is_a_prerequisite_too(): + """The SECOND mechanism, found after the first shipped. + + vcpkg-make never calls vcpkg_find_acquire_program for autotools: it uses bare + find_program and raises FATAL_ERROR with an apt line. Enumerating only + acquire-program calls misses all of it -- which is how gperf failed on + autoconf AFTER nasm was fixed. + """ + with tempfile.TemporaryDirectory() as d: + _port(d, "vcpkg-make", GPERF_LIKE, filename="vcpkg_make.cmake") + reqs = emit.host_tool_requirements(ports=["vcpkg-make"], vcpkg=d) + assert set(reqs) == {"autoconf", "autoconf-archive", "automake", "libtool"}, reqs + # libtoolize OR glibtoolize satisfies libtool: alternatives, not two requirements. + binary, apt, _users, alts = reqs["libtool"] + assert binary == "libtoolize" and apt == "libtool" and alts == ["glibtoolize"] + # autoconf-archive is m4 macros: no binary exists to probe for, and claiming + # otherwise would report it satisfied on a machine that lacks it. + assert reqs["autoconf-archive"][0] == "", reqs["autoconf-archive"] + + +def test_a_warning_about_system_packages_is_not_a_prerequisite(): + """angle WARNS about mesa-common-dev and separately FATAL_ERRORs on arch. + + A file-level 'does it contain FATAL_ERROR and an apt line' check staples the + two together and demands mesa-common-dev, which no build step here requires. + Advice is not a requirement. + """ + with tempfile.TemporaryDirectory() as d: + _port(d, "angle", """ +if (VCPKG_TARGET_IS_LINUX) + message(WARNING "${PORT} currently requires the following libraries from the system package manager:\\n mesa-common-dev\\n\\nThese can be installed via apt-get install mesa-common-dev.") +endif() +message(FATAL_ERROR "Unsupported architecture: ${VCPKG_TARGET_ARCHITECTURE}") +""") + reqs = emit.host_tool_requirements(ports=["angle"], vcpkg=d) + assert reqs == {}, "a WARNING is advice, not a prerequisite: %r" % reqs + + +def test_the_committed_host_tool_list_names_nasm_and_autotools(): + """Regression test for both bug reports, against the committed file.""" + rows = emit.load_host_tools() + assert rows, "Meta/vcpkg_host_tools.tsv is missing or empty" + by_apt = {apt: (names, users) for names, apt, users in rows} + # Ulf's first failure: libvpx, ~20 minutes in. + assert "nasm" in by_apt, list(by_apt) + assert by_apt["nasm"][0] == ["nasm"] + assert "libvpx" in by_apt["nasm"][1] + # Ulf's second failure: gperf, via vcpkg-make. + for apt in ("autoconf", "automake", "libtool", "autoconf-archive"): + assert apt in by_apt, "%s missing from the host tool list" % apt + # ...and clang must NOT be there (MSVC-only call sites). + assert "clang" not in by_apt, "clang is an MSVC-only requirement" + + +def test_the_preflight_reports_every_missing_tool_at_once(): + """The actual complaint: one tool per 20-minute build. + + The preflight's value is entirely in reporting the WHOLE set in one run, so + this checks the shape of the output, not just the exit code. + """ + ws = os.path.join(os.path.dirname(__file__), "..", "examples", "ladybird", + "workspace") + script = open(os.path.join(ws, "Meta", "vcpkg_build.sh")).read() + body = script.split("# --- host prerequisites")[1].split( + "# --- a writable vcpkg root")[0] + # $HOST_TOOLS arrives as the driver's 7th argument (vcpkg.bzl names the file + # as a declared input and passes its path); set it the same way here rather + # than editing the extracted body, so the test exercises the real contract. + body = "HOST_TOOLS=%s\n%s" % ( + os.path.join(ws, "Meta", "vcpkg_host_tools.tsv"), body) + with tempfile.TemporaryDirectory() as d: + # A PATH with everything EXCEPT nasm and libtoolize. Empty stubs are + # enough: the preflight uses `command -v`, and running the real tools is + # not the point. + for tool in ("perl", "python3", "pkg-config", "aclocal", "autoreconf"): + p = os.path.join(d, tool) + open(p, "w").close() + os.chmod(p, 0o755) + r = subprocess.run(["/bin/bash", "-c", body], capture_output=True, + text=True, env={"PATH": d}) + assert r.returncode == 1, "expected a hard failure, got %d\n%s" % ( + r.returncode, r.stderr) + # BOTH, from one run -- that is the whole feature. + assert "nasm" in r.stderr and "libtoolize" in r.stderr, r.stderr + assert "needed by: dav1d" in r.stderr, r.stderr + # One pasteable line, and the tools that ARE present must not be in it. + apt_line = [l for l in r.stderr.splitlines() if "sudo apt install" in l] + assert len(apt_line) == 1, r.stderr + assert "nasm" in apt_line[0] and "libtool" in apt_line[0] + assert "perl" not in apt_line[0], "must not demand a tool that is present" diff --git a/tests/test_vcpkg_plumbing.py b/tests/test_vcpkg_plumbing.py index 68dd452..68e95eb 100644 --- a/tests/test_vcpkg_plumbing.py +++ b/tests/test_vcpkg_plumbing.py @@ -344,3 +344,41 @@ def test_the_git_archive_staging_fails_when_the_archives_are_absent(): # not reach. `python3 tests/run_all.py` enumerates the module instead, so a test's # POSITION in the file cannot decide whether it runs; it also fails if a file # defines no tests at all. Run a single file with `run_all.py `. + + +def test_the_host_tool_list_reaches_the_action_as_a_declared_input(): + """The preflight is only real if the file is in the sandbox. + + Two ways this silently degrades to a no-op, both already made once in this + tree: reading it via `dirname $0` (an sh_binary's data lives in + .runfiles/, not beside the wrapper Bazel execs -- the trap + cargo_vendor.sh documents), or naming it in `data` instead of the action's + inputs. The driver skips the check when $HOST_TOOLS is empty, so a broken + wiring does not fail the build -- it just stops checking. Hence this test. + """ + bzl = _read("vcpkg.bzl") + assert "_host_tools" in bzl, "the host tool list is not an attr at all" + assert "ctx.files._host_tools" in bzl, "not added to the action's inputs" + assert "ctx.file._host_tools.path" in bzl, "the path is not passed as an argument" + # Passed positionally, and the driver reads it from that same position. + sh = _read("Meta/vcpkg_build.sh") + assert 'HOST_TOOLS="${7-}"' in sh, "the driver does not read argument 7" + # Count the top-level entries, tracking bracket depth: a comprehension + # (`",".join([f.path for f in ...])`) contains a `]` of its own, and splitting + # on the first one silently reads a truncated argument list -- which made this + # test fail against correct code. + args, depth = [], 0 + for line in bzl.split("arguments = [", 1)[1].splitlines(): + stripped = line.strip() + if depth == 0 and stripped.startswith("]"): + break + if stripped and not stripped.startswith("#") and depth == 0: + args.append(stripped) + depth += line.count("[") - line.count("]") + positions = args + assert len(positions) == 7, \ + "the driver reads $7, so the action must pass 7 arguments, got %d" % len( + positions) + assert "_host_tools" in positions[-1], "the list must be the 7th argument" + # And it must be exported, or the label does not resolve. + assert 'exports_files(["vcpkg_host_tools.tsv"])' in _read("Meta/BUILD.bazel") From 4390a5568360f936a642de3bb2efc3bebdb7dc75 Mon Sep 17 00:00:00 2001 From: Ulf Adams Date: Thu, 13 Aug 2026 13:11:26 +0000 Subject: [PATCH 20/59] Ladybird: Qt loaded the host's plugins into Bazel's Qt -- that is the GUI segfault Ulf's Ubuntu 24.04 box segfaults the Bazel-built GUI in QXcbConnection::initializeScreens. Cause: the binary links Qt from the Bazel repo and then dlopens Qt's PLUGINS from wherever libQt6Core's baked-in prefix points -- on his box the distro Qt's plugin dir. Two Qt builds in one process. Qt 6.9.2's qt_prfxpath is empty, so the prefix falls back to the executable's directory; a Bazel binary's directory has no platforms/, so the search falls through to the compiled-in system path. Which way the skew points decides the failure: an OLDER plugin is rejected by Qt's version gate (clean abort), a NEWER-or-equal-minor one PASSES and then calls into an ABI it was not built against -- the SIGSEGV he sees. On this box both are 6.10.2, so the same wrong lookup has been passing by accident in every green GUI run this project has reported. Pointing qt.local_repo at aqt 6.9.2 here reproduces his backtrace frame for frame. qt_runtime.bzl fixes it in four pieces: qt_plugins reads @qt's OWN generated qtconf.bzl for QT_INSTALL_PLUGINS (so plugins and libraries cannot come from different Qts) and symlinks every plugin type; qt_plugin_tree stages them beside the binary; qt_conf writes [Paths] Prefix=. which REPLACES the compiled-in prefix, so /usr is never scanned; and runtime_libs carries the private libraries an SDK bundles beside Qt. That last piece is where the loader stopped being intuition: no rpath on the binary can find aqt's bundled ICU 73, because libQt6Core resolves it through RUNPATH $ORIGIN (= Bazel's solib dir, since $ORIGIN is the path the loader opened it by), and while DT_RPATH is inherited by transitive loads, an intermediate object with a DT_RUNPATH of its own blocks it. libQt6Core has one. Measured on three generated .so files, all four combinations. So the private libs become real link inputs and Bazel stages them; the list is derived from DT_NEEDED, and is empty for a distro Qt. Also carries the Qt >= 6.9 floor that UI/Qt/CMakeLists.txt declares and the overlay silently dropped -- which is the class this is finding 40 about: all nine failures Ulf's machine has produced are one bug, a host requirement upstream declares and CHECKS, inherited without its check. find_package and pkg_check_modules are the preflight, not ceremony. Verified by removal: with the host plugin dir hidden behind an empty tmpfs and no LD_LIBRARY_PATH, the aqt build loads libqxcb.so from the aqt SDK, zero /usr dirs are scanned, and the GUI opens its window on Xvfb and stays up. Headless still renders. Following the README's staging recipe to do that turned up two more written-down-instead-of-derived paths (the vcpkg tree is exec-config; the resource root is /../share/Lagom); both now come from bazel info/cquery. 205/205 tests, 12 new. --- docs/CASE-ladybird-migration.md | 213 ++++++++ examples/ladybird/README.md | 50 +- examples/ladybird/workspace/BUILD.bazel | 21 + examples/ladybird/workspace/MODULE.bazel | 10 + .../workspace/Meta/emit_build_bazel.py | 48 +- examples/ladybird/workspace/qt_runtime.bzl | 460 ++++++++++++++++++ tests/test_qt_runtime.py | 207 ++++++++ 7 files changed, 996 insertions(+), 13 deletions(-) create mode 100644 examples/ladybird/workspace/qt_runtime.bzl create mode 100644 tests/test_qt_runtime.py diff --git a/docs/CASE-ladybird-migration.md b/docs/CASE-ladybird-migration.md index 021c5b1..e770e7e 100644 --- a/docs/CASE-ladybird-migration.md +++ b/docs/CASE-ladybird-migration.md @@ -2324,6 +2324,219 @@ case study is not a preflight. `glslangValidator` is the same shape and is still open: two genrules in `codegen_root.bzl` name `/usr/bin/glslangValidator`, and `vcpkg_installed` does not ship it. +## Finding 40: nine failures, one bug — a host requirement upstream checks and my overlay inherited silently + +Ulf's Ubuntu 24.04 machine ran the Bazel-built Ladybird headless and got a SIGSEGV +from the GUI: + +``` +ladybird(...) +QApplicationPrivate::init() +QXcbConnection::initializeScreens(bool) +QXcbConnection::handleScreenAdded(...) +--> SIGSEGV in libQt6Core +``` + +and offered a deal: *"if you can figure out how to make it work, then I won't +upgrade."* That machine is the most valuable thing in this migration. It has found +**eight** real defects (findings 37, 38, 39 and the gaps between them) that this +sandbox is structurally incapable of finding: gcc 15.2, Qt 6.10.2, ICU 78, +`nasm`/`perl`/`glslang`/`libdrm` all present, every requirement silently satisfied. +A machine that *disagrees* with mine is a test oracle, not an inconvenience, and the +only way to keep it is to stop breaking it. + +### The bug + +The binary links Qt from the Bazel repo and loads Qt's **plugins** from wherever +`libQt6Core`'s baked-in prefix points — on his box, the *distro* Qt's plugin +directory. Two different Qt builds in one process. + +Qt does not link its QPA platform plugin; `QApplication`'s constructor `dlopen`s it. +Where it looks is decided inside `libQt6Core`: `qt.conf` next to the executable, then +`QT_PLUGIN_PATH`, then the prefix compiled into the library. Qt 6.9.2's `qt_prfxpath` +is **empty** (`strings lib/libQt6Core.so.6.9.2`), so the prefix falls back to *the +directory of the executable* — and a Bazel binary's directory has no `platforms/`, so +the search falls through to the compiled-in system path and Qt loads +`/usr/lib/x86_64-linux-gnu/qt6/plugins/platforms/libqxcb.so` into a process whose +`libQt6Core` came from `bazel-bin/_solib_k8/...rules_qt++qt+qt...`. + +rules_qt is not at fault: it wires up Qt's *link* half faithfully (one SDK, discovered +by `qmake -query`; headers, libs and moc all from it). Nothing wired up the *runtime* +half, because on the machine where the overlay was written there was nothing to +notice. + +**Which way the skew points decides which failure you get.** Both halves reproduced +here, with a real X server: + +| plugin vs. linked libs | Qt's version gate | outcome | +|---|---|---| +| plugin **older** | rejects it: `factoryloader: Ignoring QPA plugin due to mismatching Qt versions 395520 394240` | `no Qt platform plugin could be initialized`, clean abort | +| plugin **newer or equal-minor** | **passes** | plugin calls into an ABI it was not built against → SIGSEGV in `initializeScreens` → `handleScreenAdded` | + +(Those integers are Qt versions: `(v>>16, (v>>8)&255, v&255)`, so 395520 = 6.9.0 and +394240 = 6.4.0.) The gate is the cruel part: it catches the harmless direction and +waves through the one that corrupts memory. + +And on **my** box it passes. `QT_DEBUG_PLUGINS=1` on the Bazel-built binary here +scanned `/usr/lib/x86_64-linux-gnu/qt6/plugins/platforms` and loaded the **distro** +`libqxcb.so` — the exact same wrong lookup — while `libQt6Core.so.6.10.2` came from +Bazel's solib dir. Both are 6.10.2, so the ABI happens to match. **The bug was +present in every green GUI run this project has ever reported.** + +Then the strongest evidence available: pointing `qt.local_repo` at the aqt **6.9.2** +SDK on this machine and rebuilding reproduced **his backtrace, frame for frame** — +`Ladybird::Application::create_platform_event_loop` → `QApplicationPrivate::init` → +`QXcbConnection::initializeScreens` → `handleScreenAdded` → a fault in libQt6Core at +`mov 0x8(%rdi),%rbx` with `rdi = 0`. Not a machine I cannot see any more. + +### The exoneration that mattered + +My first hypothesis was two ICUs: the binary needs `libicu*.so.78` (vcpkg) and aqt's +`libQt6Core` needs `libicu*.so.73`, both in one process, and `ld` even warns *"may +conflict"*. Wrong — worth recording because it is the kind of theory that sounds too +good to check. I linked ICU 78 **and** aqt's ICU 73 into one process with +`-Wl,--no-as-needed` and constructed a `QApplication`: `screens=1 qt=6.9.2`. ICU +coexists fine; two sonames are two libraries. (The fixed build now does exactly this +on purpose: `objdump -p` shows `libicu*.so.73` *and* `libicu*.so.78` in one binary, +and it runs.) Any earlier note here blaming "two ICUs" is superseded: it was never an +ICU problem, it was this same plugin/library provenance skew seen through a different +symptom. + +### The fix: the Qt edge becomes self-contained, like `vcpkg_lib` + +`qt_runtime.bzl`, and the shape is deliberately the one Ring 2 arrived at for vcpkg — +*the dependency arrives with the dep edge*: + +1. **`qt_plugins`** (repository rule) reads **@qt's own generated `qtconf.bzl`** for + `QT_INSTALL_PLUGINS` and symlinks every plugin the SDK ships into a repo, one + `filegroup` per plugin type. Reading @qt's file rather than a path of my own is the + whole point: the plugins cannot come from a different Qt than the libraries, + because both names come from one `qmake -query`. Every type, not the four Ladybird + needs today — a hand-picked list drifts, and a missing input method or file dialog + is a defect nobody notices for a month. +2. **`qt_plugin_tree`** re-declares them as outputs of the package that holds the + binary, so they land at `bazel-bin/plugins//*.so` — and, as `data` of + `//:ladybird`, in the runfiles tree too. +3. **`qt_conf`** writes the file that redirects the search: + + ``` + [Paths] + Prefix = . + Plugins = plugins + ``` + + Setting `Prefix` **replaces** the compiled-in prefix, so `/usr` is not outranked, + it is *never scanned* — `QT_DEBUG_PLUGINS=1` on the fixed build shows zero scans + of any `/usr` directory. `Prefix = .` is what makes one file correct in both + layouts, because Qt resolves it against the directory of the executable via + `/proc/self/exe`, so `bazel-bin/ladybird` and the runfiles tree's symlink to it + both land on the staged tree. +4. **`runtime_libs`** carries the private libraries an SDK bundles beside Qt. + +That fourth piece was the hard one, and it is where the loader stopped being +intuition and became a measurement. aqt's `libQt6Core` needs `libicui18n.so.73`, +which exists only inside the SDK, so the binary died in `ld.so` before `main()` and +`LD_LIBRARY_PATH=/lib` was the workaround — a workaround a human has to remember +is a bug that has been rounded down to a habit. **No rpath on the binary can fix +it**, for two reasons I only believed after reducing them to three generated `.so` +files: + +- `libQt6Core` finds its own ICU through `RUNPATH $ORIGIN`, and `$ORIGIN` is the + directory the loader **opened the object by** — Bazel's solib dir, not the SDK. (`ldd` + on that very symlink resolves ICU happily, because `ldd`'s `$ORIGIN` is the + realpath's directory. That near-miss is what made this look like a path problem.) +- Adding the SDK dir to *our* rpath does not help either: `DT_RUNPATH` is consulted + only for an object's own direct dependencies, and while `DT_RPATH` **is** inherited + by transitive loads, an intermediate object that has a `DT_RUNPATH` of its own + blocks the inherited `DT_RPATH` entirely. `libQt6Core` has one. All four + combinations measured before believing it. + +So the fix is not a search path at all: make the SDK's private libraries real link +inputs, so **Bazel** stages them and the binary's own runpath — the one glibc will +consult, because they are now direct dependencies — resolves them. The list is +derived, not written down: DT_NEEDED of the SDK's Qt modules ∩ the non-Qt `.so` files +beside them. For aqt 6.9.2 that is exactly `libicui18n/libicuuc/libicudata.so.73`; +for a distro Qt it is empty and the target degenerates to nothing. + +Five things I got wrong on the way, each caught by a probe rather than by reasoning: + +- **The staged plugins must be SYMLINKS, not copies.** A plugin needs Qt libraries + the binary does not link (`libqxcb.so` → `libQt6XcbQpa.so.6`), and aqt's plugins + carry `RUNPATH $ORIGIN/../../lib`, resolved from the object's real path. Copying + breaks exactly that; the distro's plugins have no `RUNPATH` at all, so a *copied* + distro plugin resolves `libQt6XcbQpa.so.6` from `/usr` — reintroducing the bug + through the fix. +- **`plugins/` is a substring of `qt_plugins/`.** My path-stripping matched inside the + *repository name* and staged everything one directory too deep + (`bazel-bin/plugins/plugins/...`), which built cleanly and pointed `qt.conf` at an + empty tree. Found by looking at the output, not by the build failing. +- **"Non-Qt libraries beside libQt6Core" is the whole system on a distro Qt.** Its lib + dir *is* `/usr/lib/x86_64-linux-gnu`, so the first version of the derivation + proposed linking 56 `cc_import`s including `ld-linux` into the binary. It is a + system directory, so it "worked" — which is precisely the kind of accident this + finding is about. The discriminator is whether the Qt lib dir is itself a default + loader directory. +- **An embedded `:/qt/etc/qt.conf` Qt resource does not work.** Tempting, since we + already run `rcc`, and Qt 6.9.2 does look for that path — but the resource is + registered by a static initialiser in the binary and every variant I built ignored + it, while a file on disk worked first try. +- **`--no-as-needed` is load-bearing.** The binary references no ICU 73 symbol (it has + its own ICU 78), so the linker drops the `DT_NEEDED` as unused and the staging + silently stops working. + +### The class, which is the finding + +Every one of the nine failures Ulf's machine has produced is the same sentence: + +| # | symptom on his box | the host requirement | who declares it | +|---|---|---|---| +| 1 | `vcpkg: No such file or directory` | the `Build/vcpkg` clone | `Meta/ladybird.py vcpkg` | +| 2 | `fatal: unable to read tree` | that clone's `.git`, at a baseline | vcpkg's baseline resolution | +| 3 | HSTS table differs | Chromium's JSON, *unpinned* | `Meta/CMake/hsts_preload.cmake` | +| 4 | `glslangValidator: No such file` | `glslang-tools` | nothing — my genrule hardcodes `/usr/bin` | +| 5 | `ninja-linux.zip MISSING FROM INDEX` | `ninja` ≥ 1.13.2 | `vcpkg_find_acquire_program` (probes, then downloads) | +| 6 | `Could not find nasm` | `nasm` | `vcpkg_find_acquire_program` (Linux: no URL to pin) | +| 7 | `gperf requires autoconf autoconf-archive automake libtoolize` | autotools | `vcpkg-make`'s bare `find_program` | +| 8 | libdrm headers not found | `libdrm-dev` | CMake `pkg_check_modules(... REQUIRED)` | +| 9 | **SIGSEGV in `initializeScreens`** | **Qt ≥ 6.9, and its plugins** | **`find_package(Qt6 6.9 REQUIRED)`** — upstream checks it; my overlay did not | + +Not nine bugs. One bug, nine times: **a host requirement that upstream declares and +checks, which the Bazel overlay inherited without inheriting the check.** CMake's +`find_package`, `pkg_check_modules(REQUIRED)` and `vcpkg_find_acquire_program` are not +ceremony — they are the *preflight*, and translating a build system means translating +its preflight, not only its compile lines. Nine times I translated the commands and +dropped the assertion; nine times the machine that disagreed with mine was the one +that told me. + +So `qt_plugins` also carries the floor. `UI/Qt/CMakeLists.txt` says +`find_package(Qt6 6.9 REQUIRED COMPONENTS Core Widgets)`; the repo rule now fails at +fetch time with the version it found, the prefix it found it in, the package to +install, and a warning not to mix SDKs — the finding-39 mechanism (derive the +requirement from the source of truth, check it where it is needed, name the fix in the +error) extended from vcpkg's driver to the overlay's own build. `glslangValidator` +(#4) is the last member of the table with no check at all. + +### One more wrong path, found by walking the recipe + +Verifying the GUI meant following the README's own staging recipe, which promptly +failed with `UNEXPECTED ERROR: stat: No such file or directory at +UI/Qt/WebContentView.cpp:1047` — the theme `.ini`. Two of its paths were wrong, both +in the same way as this finding: written down instead of derived. The vcpkg tree it +copies from is built in the **exec** configuration, so `bazel-bin/vcpkg_installed` +does not exist at all (`k8-fastbuild-exec` is the directory); and the resource root +is not `/share/Lagom` but `/../share/Lagom`, because +`LibWebView/Utilities.cpp`'s `find_prefix()` takes the **parent** of the binary's +directory. Both are now `bazel info` / `bazel cquery --output=files` invocations, +which answer for whatever configuration you are on. Prose describing an output path +is a pin with no checker (gap 5, closed). + +**Verified by removal**, which is the only kind of verification this document +accepts. With `/usr/lib/x86_64-linux-gnu/qt6/plugins` hidden behind an empty tmpfs and +**no `LD_LIBRARY_PATH`**: the aqt build loads `libqxcb.so` from the aqt SDK (previously: +the distro's), zero `/usr` directories are scanned, and against the host Qt the GUI +opens its window on Xvfb and stays up. Headless still renders the reference page. +Passing because the host happened to agree is how all nine of these got here. + ## Plan for the rest - **Ring 1c — BUILD generation to compile+link parity, bottom-up.** AK → diff --git a/examples/ladybird/README.md b/examples/ladybird/README.md index ca500d4..9e880d7 100644 --- a/examples/ladybird/README.md +++ b/examples/ladybird/README.md @@ -118,7 +118,7 @@ knowing, because both were mistakes I made first: ### Reproducing the tree on another machine The overlay is **not a fork**: it is a pinned upstream Ladybird commit + two patches -+ 44 Bazel files that sit alongside CMake's. [`apply_overlay.sh`](apply_overlay.sh) is ++ 45 Bazel files that sit alongside CMake's. [`apply_overlay.sh`](apply_overlay.sh) is that sentence made executable, because *"copy `workspace/` over a clone"* has four ways to be silently wrong — and every one of them was found by running it, not by reading it: @@ -145,7 +145,7 @@ reading it: **after** the prefetch; the script defers it and says so. `--verify` checks a tree without changing it, and checks the things a file copy -cannot: that HEAD is the pinned commit, that all 44 files are byte-identical, that +cannot: that HEAD is the pinned commit, that all 45 files are byte-identical, that the patches are **applied** (`git apply --check -R` succeeding is the proof — a patch that reverse-applies cleanly is already in the tree), and that the `.sh` files kept their **executable bit**, which is tree state a careless copy drops and which then @@ -285,8 +285,9 @@ an FFI header collision and an entire missing Rust target. See | `Meta/vcpkg_capture_git_archives.sh` | Regenerates that pin with `vcpkg install --only-downloads` (~6 min, no compilation, no CMake) — vcpkg as the instrument, since which git externals are used is decided by feature-conditional CMake code, not by portfile text | | `hsts_preload.bzl` | Chromium's HSTS preload table as one `http_file`, pinned to a **commit** + sha256 — the downstream pin for the one input upstream CMake fetches from `main`. **Generated** by `Meta/pin_hsts_preload.py` | | `Meta/pin_hsts_preload.py` | Re-pins it: resolves the newest commit touching the path, downloads it, writes the hash it **measured**; `--expect-same-as` refuses to write unless the pinned bytes equal the file CMake downloaded (parity guard) | -| `apply_overlay.sh` | Reproduces the whole tree on another machine: clone at the pinned Ladybird commit, apply the two patches, copy the 44 overlay files (with the `bazelrc.txt` → `.bazelrc` rename), run the vcpkg prefetch, then stage the one file that must come after it. `--verify` checks an existing tree — commit, bytes, patches-applied, exec bits — and changes nothing | +| `apply_overlay.sh` | Reproduces the whole tree on another machine: clone at the pinned Ladybird commit, apply the two patches, copy the 45 overlay files (with the `bazelrc.txt` → `.bazelrc` rename), run the vcpkg prefetch, then stage the one file that must come after it. `--verify` checks an existing tree — commit, bytes, patches-applied, exec bits — and changes nothing | | `bazelrc.txt` | → `.bazelrc`. Global copts/defines/linkopts mirrored from `Meta/CMake/compile_options.cmake` | +| `qt_runtime.bzl` | Qt's **runtime** half: a repo rule that stages the plugins of the SDK `@qt` itself names (read out of @qt's generated `qtconf.bzl`, so plugins and libraries cannot come from different Qts), the private libraries an SDK bundles beside Qt (derived from DT_NEEDED), a generated `qt.conf` pointing the binary at them, and the Qt >= 6.9 floor `UI/Qt/CMakeLists.txt` declares. Without it Qt `dlopen`ed the HOST's plugin into Bazel's Qt — finding 40 | | `BUILD.bazel` | Root package: 34 libraries, the 5 executables, Qt moc/rcc genrules. **Generated** by `Meta/emit_build_bazel.py` | | `codegen_root.bzl` | Non-LibWeb generator genrules (IPC endpoints, LibJS Bytecode/Op, HSTS table, WebGL replayer, TIFF tag tables, the two SPIR-V shader headers, and the chained `generate_interpreter_layout` → `flapc` interpreter assembly). **Generated** by `Meta/emit_root_codegen_bazel.py` | | `Libraries/LibWeb/BUILD.bazel`, `generated_srcs.bzl` | LibWeb (~1,961 compile inputs). **Generated** by `Meta/emit_libweb_bazel.py` | @@ -418,23 +419,39 @@ of `Meta/` — so they work from any checkout path. Running the UI currently needs manual staging, which is itself a gap (see below): +Every path below is **derived, not written down**, because two of them used to be +stated as literal `k8-fastbuild` paths and one was simply wrong: the vcpkg tree is +built in the **exec** configuration (`vcpkg_lib` pins `cfg = "exec"`), so it is +under `k8-fastbuild-exec` and `bazel-bin/vcpkg_installed` does not exist. The two +staging roots are also NOT the same directory, which is easy to get backwards: +helper binaries go in `/libexec`, but the resource root is +`/../share/Lagom` -- `LibWebView/Utilities.cpp`'s `find_prefix()` takes the +PARENT of the binary's directory and appends `share/Lagom`. Ask the build for all +three rather than spelling any of them out: + ```sh export XDG_RUNTIME_DIR=/tmp/xdg-lb && mkdir -p $XDG_RUNTIME_DIR && chmod 700 $XDG_RUNTIME_DIR -ER=$(bazel info execution_root); mkdir -p "$ER/bazel-out/k8-fastbuild/libexec" -for b in WebContent RequestServer ImageDecoder Compositor WebWorker; do cp -f bazel-bin/$b "$ER/bazel-out/k8-fastbuild/libexec/"; done +BIN=$(bazel info bazel-bin) # target config: where ladybird is +mkdir -p "$BIN/libexec" +for b in WebContent RequestServer ImageDecoder Compositor WebWorker; do cp -f bazel-bin/$b "$BIN/libexec/"; done # cranelift-compiler is found as a sibling of the spawning binary (Ladybird's own # lookup chain), so it needs no path baked in -- only to be there. -cp -f bazel-bin/cranelift-compiler "$ER/bazel-out/k8-fastbuild/libexec/" +cp -f bazel-bin/cranelift-compiler "$BIN/libexec/" # The resource root, assembled from the CLONE and from Bazel's own vcpkg tree. # This line used to read `ln -sfn "$PWD/Build/full/share/Lagom" ...` -- i.e. the # recipe for running the Bazel build pointed at CMake's build tree, a seventh # thing a fresh clone does not have (finding 36). Everything in it is either in # Base/res or in //:vcpkg_installed; `chmod u+w` because Bazel's outputs are # read-only and `cp -r` preserves that. -L="$ER/bazel-out/k8-fastbuild/share/Lagom" +# +# $(dirname $BIN), not $BIN: find_prefix() resolves the resource root against the +# PARENT of the directory holding the binary. +L="$(dirname "$BIN")/share/Lagom" chmod -R u+w "$L" 2>/dev/null; rm -rf "$L"; mkdir -p "$L/ladybird/pdfjs/web" cp -r Base/res/. "$L/" -P=bazel-bin/vcpkg_installed/x64-linux-dynamic/share/pdfjs +# ASK for the tree rather than guessing its configuration directory. +V=$(bazel cquery 'deps(//:ladybird)' --output=files 2>/dev/null | grep 'vcpkg_installed$' | head -1) +P="$V/x64-linux-dynamic/share/pdfjs" cp -r --no-preserve=mode "$P/build" "$L/ladybird/pdfjs/" cp -r --no-preserve=mode "$P/web/." "$L/ladybird/pdfjs/web/" # UI/cmake/ResourceFiles.cmake stages this one file into pdfjs/web/, not pdfjs/. @@ -662,12 +679,25 @@ Honest inventory of what stops this from being a clone-and-build. `Build/full/share/Lagom`, CMake's build tree, into place. Everything the resource root needs is in `Base/res` plus `//:vcpkg_installed`'s pdf.js, and the block above assembles it from those; the result is `diff -rq`-identical to - CMake's. Finding 36. + CMake's. Finding 36. **Two of its paths were also wrong, and are now derived + rather than written down** (finding 40): the vcpkg tree is built in the *exec* + configuration so `bazel-bin/vcpkg_installed` does not exist, and the resource + root is `/../share/Lagom` -- `find_prefix()` takes the PARENT of the + binary's directory -- not `/share/Lagom`. Both now come from + `bazel info` / `bazel cquery --output=files`. Qt's plugins USED to belong on + this list and no longer do: they are `data` of `//:ladybird` (`qt_runtime.bzl`), + which is what that "real Bazel setup" looks like for one of the three things. 6. **`rules_qt` is not on the BCR.** `MODULE.bazel` uses an `archive_override` pointing at kklochkov/rules_qt v2.0.1's release tarball (stock upstream, no patches). The BCR's `rules_qt` module is Vertexwahn's unrelated `rules_qt6`. Qt itself *is* host-portable: `qt.local_repo` discovers the host Qt via - `qmake -query`, so no Qt SDK is vendored. + `qmake -query`, so no Qt SDK is vendored. Its **plugins** are host-portable too + now, and that took a fix rather than an observation: rules_qt wires up Qt's link + half only, so the binary linked @qt's libraries and then `dlopen`ed the HOST's + QPA plugin into them -- a SIGSEGV where the two Qt versions differ, and a silent + pass where they agree. `qt_runtime.bzl` stages the plugins of the SDK @qt itself + names, points a generated `qt.conf` at them, and enforces the Qt >= 6.9 floor + `UI/Qt/CMakeLists.txt` declares. Finding 40. 7. **The fresh clone needs two inputs the overlay does not carry** (step 1a above), and each stayed invisible for the same reason: it was already on the machine. `Build/vcpkg` — a microsoft/vcpkg checkout at `vcpkg.json`'s diff --git a/examples/ladybird/workspace/BUILD.bazel b/examples/ladybird/workspace/BUILD.bazel index c34fd60..f0a0a34 100644 --- a/examples/ladybird/workspace/BUILD.bazel +++ b/examples/ladybird/workspace/BUILD.bazel @@ -3,6 +3,7 @@ load("@rules_cc//cc:defs.bzl", "cc_binary", "cc_library") load(":cargo_ring.bzl", "cargo_ring") load(":codegen_root.bzl", "root_codegen") load(":export_headers.bzl", "export_headers") +load(":qt_runtime.bzl", "qt_conf", "qt_plugin_tree") load(":vcpkg.bzl", "vcpkg_tree", "vcpkg_tree_for_exec") load(":vcpkg_index.bzl", "VCPKG_DISTFILE_INDEX") @@ -1790,6 +1791,24 @@ qt_cc_rcc( srcs = [":qt_qrc"], ) + +# === Qt6 RUNTIME: the plugins Qt dlopens, and the qt.conf that finds them === +# Not a CMake target -- CMake does not need one, because its binary links a Qt +# whose baked-in prefix already points at the plugins of that same Qt. Bazel's +# does not: it links @qt's libraries and then Qt looks for plugins next to the +# executable, finds none, and falls back to the HOST's plugin directory. Loading +# another Qt build's QPA plugin into this one is the Ubuntu 24.04 SIGSEGV in +# QXcbConnection::initializeScreens; on a box where the versions agree it "works". +# qt_runtime.bzl has the full account (finding 40); both targets below are data of +# //:ladybird, so they are staged in bazel-bin AND in the runfiles tree. +qt_plugin_tree( + name = "qt_plugins", + plugins = ["@qt_plugins//:plugins"], +) + +qt_conf( + name = "qt_conf", +) # === ladybird (executable, 21 TU) === cc_binary( name = 'ladybird', @@ -1821,6 +1840,7 @@ cc_binary( local_defines = ['QT_CORE_LIB', 'QT_GUI_LIB', 'QT_NO_DEBUG', 'QT_WIDGETS_LIB'], copts = ['-fPIE'], linkopts = ['-lGLX', '-lOpenGL', '-lvulkan'], + data = [':qt_conf', ':qt_plugins'], deps = [ '//:AK', '//:LibCore', @@ -1843,6 +1863,7 @@ cc_binary( '@qt//:QtCore', '@qt//:QtGui', '@qt//:QtWidgets', + '@qt_plugins//:runtime_libs', ], ) diff --git a/examples/ladybird/workspace/MODULE.bazel b/examples/ladybird/workspace/MODULE.bazel index 8c133c6..fdf6993 100644 --- a/examples/ladybird/workspace/MODULE.bazel +++ b/examples/ladybird/workspace/MODULE.bazel @@ -30,6 +30,16 @@ qt.active_sdk(name = "qt", repo = "qt6_local") use_repo(qt, "qt") register_toolchains("@qt//:qt_linux_x86_64_toolchain") +# Qt's RUNTIME half. rules_qt wires up linking; nothing wired up the plugins Qt +# dlopens at QApplication construction, so the binary linked @qt's libQt6Core and +# then loaded the HOST's platform plugin into it -- two Qt builds in one process. +# On a box where SDK and distro versions differ that is a SIGSEGV in +# QXcbConnection::initializeScreens (reported on Ubuntu 24.04); where they agree it +# works by accident. @qt_plugins takes the plugins from the same SDK @qt names, and +# //:qt_conf points the binary at them. See qt_runtime.bzl for the whole story. +qt_runtime = use_extension("//:qt_runtime.bzl", "qt_runtime") +use_repo(qt_runtime, "qt_plugins") + # --------------------------------------------------------------------------- # Chromium's HSTS preload table: the one input CMake downloads UNPINNED (from # `main`, at configure time) and the one thing that made a Bazel-only clone diff --git a/examples/ladybird/workspace/Meta/emit_build_bazel.py b/examples/ladybird/workspace/Meta/emit_build_bazel.py index 649053b..39f465d 100644 --- a/examples/ladybird/workspace/Meta/emit_build_bazel.py +++ b/examples/ladybird/workspace/Meta/emit_build_bazel.py @@ -331,7 +331,13 @@ def dep_label(d, targets, so, ar): # Qt6 is a real Bazel dep via rules_qt: its qt.local_repo discovers the host # SDK through qmake, so moc/rcc and the Qt cc_librarys all come from one SDK. if nm in QT_MAP: - return Dep(deps=[QT_MAP[nm]]) + # Plus @qt_plugins//:runtime_libs -- the PRIVATE libraries an SDK ships + # beside Qt (aqt bundles ICU 73) which rules_qt does not stage, and which + # libQt6Core cannot find on its own because its RUNPATH $ORIGIN expands to + # Bazel's solib dir. Making them link inputs is what removed the need for + # LD_LIBRARY_PATH=/lib. Empty for a distro Qt. See qt_runtime.bzl / + # finding 40. + return Dep(deps=[QT_MAP[nm], "@qt_plugins//:runtime_libs"]) SYS_MAP = {"GLX": "GLX", "OpenGL": "OpenGL"} if nm in SYS_MAP: return Dep(sys=[SYS_MAP[nm]]) @@ -429,6 +435,11 @@ def _emit_srcs(srcs): vcpkg_tree( name = "vcpkg_installed", distfiles = VCPKG_DISTFILE_INDEX, + # The wheels for the Python packages a portfile pip-installs (angle asks for + # `ply`). Pinned separately from the 76 distfiles because pip bypasses vcpkg's + # asset cache entirely, so the capture cannot see them and x-block-origin + # cannot block them -- see vcpkg_python_packages.bzl and finding 36. + python_wheels = ['@vcpkg_pywheel_ply//file'], source_dir = ".", source_root = ":vcpkg_source_inputs", triplet = "x64-linux-dynamic", @@ -483,6 +494,7 @@ def _emit_srcs(srcs): load(":cargo_ring.bzl", "cargo_ring") load(":codegen_root.bzl", "root_codegen") load(":export_headers.bzl", "export_headers") +load(":qt_runtime.bzl", "qt_conf", "qt_plugin_tree") load(":vcpkg.bzl", "vcpkg_tree", "vcpkg_tree_for_exec") load(":vcpkg_index.bzl", "VCPKG_DISTFILE_INDEX") @@ -613,6 +625,31 @@ def emit_qt_autogen(moc_hdrs): print() +QT_RUNTIME_BLOCK = ''' +# === Qt6 RUNTIME: the plugins Qt dlopens, and the qt.conf that finds them === +# Not a CMake target -- CMake does not need one, because its binary links a Qt +# whose baked-in prefix already points at the plugins of that same Qt. Bazel's +# does not: it links @qt's libraries and then Qt looks for plugins next to the +# executable, finds none, and falls back to the HOST's plugin directory. Loading +# another Qt build's QPA plugin into this one is the Ubuntu 24.04 SIGSEGV in +# QXcbConnection::initializeScreens; on a box where the versions agree it "works". +# qt_runtime.bzl has the full account (finding 40); both targets below are data of +# //:ladybird, so they are staged in bazel-bin AND in the runfiles tree. +qt_plugin_tree( + name = "qt_plugins", + plugins = ["@qt_plugins//:plugins"], +) + +qt_conf( + name = "qt_conf", +) +''' + + +def emit_qt_runtime(): + print(QT_RUNTIME_BLOCK, end="") + + def moc_headers(): """UI/Qt headers with Q_OBJECT, i.e. the ones CMake's AUTOMOC would moc. @@ -633,7 +670,7 @@ def moc_headers(): def emit_target(name, targets, libs, exes, so, ar, header=True, body_only=False, - extra_srcs=()): + extra_srcs=(), extra_data=()): """Emit one cc_library/cc_binary from the model. body_only: emit only `name =` + `srcs =` (AK's hand-written block supplies @@ -757,6 +794,7 @@ def emit_target(name, targets, libs, exes, so, ar, header=True, body_only=False, # On a cc_library `data` propagates into the runfiles of every binary # that links it, which is exactly the reach cranelift-compiler needs # (LibWasm is linked by WebContent, WebWorker and ladybird). + data = list(data) + list(extra_data) if data: print(" data = [%s]," % ", ".join("%r" % x for x in sorted(set(data)))) # PRIVATE deps: used to compile this library, not part of its interface, @@ -805,9 +843,13 @@ def main(): for name in ROOT_TARGETS: emit_target(name, targets, libs, exes, so, ar) emit_qt_autogen(moc_headers()) + emit_qt_runtime() for name in QT_TARGETS: emit_target(name, targets, libs, exes, so, ar, - extra_srcs=[":qt_moc", ":qt_rcc"]) + extra_srcs=[":qt_moc", ":qt_rcc"], + # The Qt plugins + qt.conf: runtime inputs of the GUI, in the + # same sense as the cranelift-compiler binary LibWasm spawns. + extra_data=[":qt_conf", ":qt_plugins"]) print(VCPKG_TAIL, end="") diff --git a/examples/ladybird/workspace/qt_runtime.bzl b/examples/ladybird/workspace/qt_runtime.bzl new file mode 100644 index 0000000..cd8e565 --- /dev/null +++ b/examples/ladybird/workspace/qt_runtime.bzl @@ -0,0 +1,460 @@ +# Qt's RUNTIME half: the plugins, staged next to the binary, from the SAME Qt as +# the linked libraries. +# +# WHY THIS FILE EXISTS -- the bug it fixes (finding 40). +# +# rules_qt's `qt.local_repo` makes Qt's *link* half hermetic-ish: the cc_librarys +# under @qt point at one SDK, discovered through `qmake -query`, and Bazel links +# libQt6Core from there (via _solib_k8). Qt's *runtime* half was never wired up at +# all, and Qt does not take the hint: at QApplication construction it dlopens the +# QPA platform plugin (libqxcb.so) from a path baked into libQt6Core -- its build +# prefix -- or, when that prefix is empty, from the DIRECTORY OF THE EXECUTABLE. +# A Bazel binary's directory has no `platforms/`, so the search falls through to +# the compiled-in system path and Qt loads the DISTRO's plugin into a process +# whose Qt libraries came from the Bazel repo. +# +# Two different Qt builds in one process. What that does depends on which way the +# skew points, and both halves are reproduced (with a real X server): +# +# plugin OLDER than libs -> Qt's version gate rejects it: +# "Ignoring QPA plugin due to mismatching Qt +# versions 395520 394240" -> "no Qt platform plugin +# could be initialized", abort. +# plugin NEWER than libs -> the gate PASSES, the plugin loads, and it calls +# into a libQt6Core whose ABI it was not built +# against -> SIGSEGV in QXcbConnection:: +# initializeScreens -> handleScreenAdded. +# +# The second one is the crash reported on Ubuntu 24.04 (aqt Qt 6.9.2 linked, +# distro plugins in /usr/lib/x86_64-linux-gnu/qt6/plugins). On a machine where +# the SDK and the distro happen to be the same version it works -- BY ACCIDENT, +# which is how it survived this long here (`QT_DEBUG_PLUGINS=1` on this box shows +# the same wrong scan, loading /usr's libqxcb.so into @qt's libQt6Core; the two +# are both 6.10.2, so nothing breaks). +# +# HOW IT IS FIXED. +# +# 1. `qt_plugins` (below) is a repository rule that reads @qt's OWN generated +# qtconf.bzl -- not a hand-written path -- and symlinks every plugin under +# that SDK's QT_INSTALL_PLUGINS into a repo, one filegroup per plugin type. +# Deriving the path from @qt is the whole point: the plugins cannot come +# from a different Qt than the libraries, because both names come from one +# `qmake -query`. +# 2. `qt_plugin_tree` re-declares those files as outputs of the ROOT package at +# `plugins//.so`, so they land next to the binary in bazel-bin +# (and, as data, in the runfiles tree too). +# 3. `qt_conf` writes the four-line qt.conf that points Qt at them: +# +# [Paths] +# Prefix = . +# Plugins = plugins +# +# Qt reads qt.conf from the directory of the *resolved* executable (it uses +# /proc/self/exe, so running through a symlink -- `bazel run`, or the +# runfiles tree's own symlink to bazel-bin -- still finds it). `Prefix = .` +# is what makes ONE file correct in BOTH layouts: bazel-bin/ladybird sees +# bazel-bin/plugins, and runfiles/_main/ladybird resolves to the same place. +# +# Setting Prefix also REPLACES the compiled-in prefix, so /usr's plugin directory +# is not merely outranked, it is never scanned. Verified by removal: with an empty +# directory bind-mounted over the host plugin dir, the binary still starts. +# +# The staged files are SYMLINKS to the SDK's plugins on purpose. A plugin needs Qt +# libraries the binary does not link (libqxcb.so needs libQt6XcbQpa.so.6), and +# aqt's plugins carry `RUNPATH $ORIGIN/../../lib`; $ORIGIN is resolved from the +# object's real path, so a symlinked plugin finds its own SDK's private libs with +# no rpath work and no LD_LIBRARY_PATH. Copying them would break exactly that -- +# also checked, and it fails the way the report describes. + +# The version floor Ladybird's own UI/Qt/CMakeLists.txt declares. +_QT_FLOOR = (6, 9) + +# --------------------------------------------------------------------------- +# The repo: @qt's plugins, as Bazel files. +# --------------------------------------------------------------------------- + +_BUILD_HEADER = """# GENERATED by qt_runtime.bzl (qt_plugins). Do not edit. +load("@rules_cc//cc:defs.bzl", "cc_import", "cc_library") + +package(default_visibility = ["//visibility:public"]) +""" + +# The OTHER half of "no LD_LIBRARY_PATH": the PRIVATE libraries an SDK bundles. +# +# rules_qt's cc_librarys stage libQt6*.so into _solib_k8 and nothing else. An +# official Qt SDK also bundles its own ICU -- aqt 6.9.2's libQt6Core needs +# libicui18n.so.73, which exists in the SDK's lib/ and nowhere else on a machine +# whose distro ICU is 78 -- so the binary died in the loader before main() and +# `LD_LIBRARY_PATH=/lib` was the workaround. A workaround a human has to +# remember is a bug that has been rounded down to a habit. +# +# WHY AN RPATH ON THE BINARY CANNOT FIX THIS, which took three reductions to +# believe. libQt6Core resolves its own ICU through `RUNPATH $ORIGIN`, and $ORIGIN is +# the directory the loader OPENED the object by -- which is Bazel's solib dir, not +# the SDK. (`ldd` on the same symlink resolves ICU happily, because ldd's $ORIGIN is +# the realpath's dir; that near-miss is what made this look like a path problem.) +# Adding the SDK dir to OUR rpath does not help either, and the reason is a glibc +# rule worth writing down: DT_RUNPATH is consulted only for an object's own direct +# dependencies, and while DT_RPATH IS inherited by transitive loads, an +# intermediate object that has a DT_RUNPATH **of its own** blocks the inherited +# DT_RPATH entirely. libQt6Core has one. Measured on three generated .so files, all +# four combinations, before believing it. +# +# So the fix is not a search path at all: make the SDK's private libraries real +# link inputs, so BAZEL stages them into a solib dir and the binary's OWN runpath +# resolves them. Ours is the runpath glibc will consult, because they are now our +# direct dependencies. Verified with the loader trace, then end to end. +# +# The list is DERIVED, not written down: the intersection of "libraries the SDK's +# Qt modules declare in DT_NEEDED" with "libraries the SDK ships beside them". For +# aqt 6.9.2 that is exactly libicui18n/libicuuc/libicudata .so.73; for a distro Qt +# it is EMPTY (a distro's ICU is a distro package, already on the default search +# path) and this target degenerates to nothing, which is the correct answer rather +# than a special case. +_RUNTIME_LIB_IMPORT = """ +cc_import( + name = "{name}", + shared_library = "{lib}", +) +""" + +_RUNTIME_LIBS_EMPTY = """ +# This Qt bundles no private libraries of its own (a distro Qt: its ICU is a distro +# package, already on the loader's default search path). +cc_library( + name = "runtime_libs", +) +""" + +_RUNTIME_LIBS_GROUP = """ +cc_library( + name = "runtime_libs", + deps = [{deps}], + # --no-as-needed: the BINARY does not reference an ICU 73 symbol (it has its own + # ICU 78 from vcpkg), so the linker would drop the DT_NEEDED as unused and we + # would be back to libQt6Core searching a directory that has no ICU in it. + linkopts = ["-Wl,--no-as-needed"], +) +""" + +def _parse_qtconf(content): + """Reads @qt's generated qtconf.bzl as data: KEY="value" lines.""" + values = {} + for line in content.splitlines(): + line = line.strip() + if line.startswith("#") or "=" not in line: + continue + key, _, raw = line.partition("=") + raw = raw.strip() + if not raw.startswith("\"") or not raw.endswith("\""): + continue + values[key.strip()] = raw[1:-1] + return values + +# Directories that ARE the loader's default search path. A Qt whose libraries live +# in one of these is a distro Qt: everything it depends on is already findable, and +# there is nothing to stage. Getting this wrong is not a small mistake -- the first +# version of this derivation asked "which non-Qt .so files sit beside libQt6Core", +# which for a distro Qt is /usr/lib/x86_64-linux-gnu, i.e. it proposed to link the +# ENTIRE system library directory into the binary (56 cc_imports, ld-linux among +# them). It is a system directory, so it "worked"; that is exactly the kind of +# accident this whole finding is about. +_SYSTEM_LIB_DIRS = [ + "/lib", + "/lib64", + "/lib/x86_64-linux-gnu", + "/lib/aarch64-linux-gnu", + "/usr/lib", + "/usr/lib64", + "/usr/lib/x86_64-linux-gnu", + "/usr/lib/aarch64-linux-gnu", + "/usr/local/lib", +] + +def _sdk_private_libs(repository_ctx, libs_root): + """The SDK's bundled non-Qt libraries that its OWN Qt modules depend on. + + Derived, not listed: read DT_NEEDED out of the SDK's libQt6*.so with objdump and + intersect it with the non-Qt .so files sitting in the same directory. For an + official SDK that yields its bundled ICU; for a distro Qt it yields nothing, + because the lib dir IS a system dir (see _SYSTEM_LIB_DIRS) and everything in it + is already on the loader's default search path. + """ + if libs_root.rstrip("/") in _SYSTEM_LIB_DIRS: + return [] + + root = repository_ctx.path(libs_root) + if not root.exists: + return [] + + # What the directory ships that is not Qt itself, keyed by soname-ish basename. + present = {} + for entry in root.readdir(): + name = str(entry.basename) + if name.startswith("libQt") or ".so" not in name: + continue + present[name] = "lib/{}".format(name) + + if not present: + return [] + + qt_modules = [ + "{}/{}".format(libs_root, str(e.basename)) + for e in root.readdir() + if str(e.basename).startswith("libQt") and ".so." in str(e.basename) + ] + if not qt_modules: + return [] + + # objdump is in binutils, i.e. present anywhere a C++ toolchain is. If it is + # not, say so rather than silently returning "no private libraries" -- that + # would look exactly like a distro Qt and reintroduce the LD_LIBRARY_PATH need + # with no diagnostic (finding 35: a check that cannot fail must not look like a + # check that passed). + result = repository_ctx.execute(["objdump", "-p"] + qt_modules) + if result.return_code != 0: + fail(("qt_plugins: cannot read DT_NEEDED from the Qt libraries in {d}: " + + "objdump failed ({e}). objdump comes with binutils; it is needed here " + + "to discover the private libraries an SDK bundles beside Qt (an " + + "official SDK ships its own ICU), because without them the binary " + + "cannot start without LD_LIBRARY_PATH.").format( + d = libs_root, + e = result.stderr.strip(), + )) + + needed = {} + for line in result.stdout.splitlines(): + line = line.strip() + if not line.startswith("NEEDED"): + continue + # Starlark's split() has no whitespace-default: give it one explicitly and + # drop the empty fields the double spaces in objdump's output produce. + fields = [f for f in line.split(" ") if f] + if len(fields) < 2: + continue + soname = fields[-1] + if soname in present: + needed[soname] = present[soname] + + # Symlink each one in, and return the (label-safe) target names. + out = [] + for soname in sorted(needed.keys()): + repository_ctx.symlink( + repository_ctx.path("{}/{}".format(libs_root, soname)), + needed[soname], + ) + out.append(needed[soname]) + return out + +def _version_tuple(version): + parts = version.split(".") + nums = [] + for p in parts[:3]: + if not p.isdigit(): + return None + nums.append(int(p)) + if len(nums) < 2: + return None + return nums + +def _qt_plugins_impl(repository_ctx): + # @qt writes this file from `qmake -query`; reading it as DATA (rather than + # loading it, which a repository rule cannot do) is what ties the plugins to + # the same SDK as the libraries. + qtconf_label = Label("@qt//:qtconf.bzl") + values = _parse_qtconf(repository_ctx.read(qtconf_label)) + + version = values.get("QT_VERSION", "") + plugins_root = values.get("QT_INSTALL_PLUGINS", "") + if not version or not plugins_root: + fail("qt_plugins: @qt//:qtconf.bzl has no QT_VERSION/QT_INSTALL_PLUGINS. " + + "rules_qt's qt.local_repo generates it from `qmake -query`; if the keys " + + "moved, this rule has to follow them.") + + # Preflight the version FLOOR Ladybird's own CMake declares, in the place that + # can still say something useful about it. UI/Qt/CMakeLists.txt has + # `find_package(Qt6 6.9 REQUIRED COMPONENTS Core Widgets)`; CMake refuses an + # older Qt with a clear message, and until this check existed the Bazel build + # just compiled against whatever qmake was first on PATH and failed later -- + # at moc time, at link time, or (finding 40) not at all until the GUI crashed. + have = _version_tuple(version) + if have == None: + fail("qt_plugins: cannot parse QT_VERSION {!r} from @qt".format(version)) + if (have[0], have[1]) < (_QT_FLOOR[0], _QT_FLOOR[1]): + fail(("qt_plugins: Qt {have} is too old.\n\n" + + " Ladybird requires Qt >= {floor} (UI/Qt/CMakeLists.txt:\n" + + " find_package(Qt6 {floor} REQUIRED COMPONENTS Core Widgets)).\n" + + " The SDK rules_qt discovered is {have} at {prefix}.\n\n" + + " Point qt.local_repo's `paths` in MODULE.bazel at a newer Qt, or\n" + + " install one (Debian/Ubuntu: qt6-base-dev >= {floor}; otherwise an\n" + + " official Qt SDK). Do NOT mix: the plugins are taken from this same\n" + + " SDK, and a mixed pair is the crash finding 40 is about.").format( + have = version, + floor = "{}.{}".format(_QT_FLOOR[0], _QT_FLOOR[1]), + prefix = values.get("QT_INSTALL_PREFIX", "?"), + )) + + root = repository_ctx.path(plugins_root) + if not root.exists: + fail(("qt_plugins: Qt {v} reports its plugins live in\n {p}\n" + + "but that directory does not exist. A Qt with no QPA plugin cannot open a\n" + + "window: install the platform plugins (Ubuntu: qt6-base-dev pulls\n" + + "libqt6gui6, which ships plugins/platforms/libqxcb.so).").format( + v = version, + p = plugins_root, + )) + + # Every plugin type the SDK ships, symlinked file by file. Deliberately NOT a + # hand-picked list of the four types Ladybird happens to need today: a list is + # a thing that drifts, symlinks cost nothing, and the failure mode of a + # missing plugin type (no input method, no native file dialog, no icons) is + # the sort of thing nobody notices for a month. + groups = {} + for type_dir in sorted([str(p.basename) for p in root.readdir()]): + src_dir = repository_ctx.path("{}/{}".format(plugins_root, type_dir)) + if not src_dir.is_dir: + continue + files = [] + for plugin in sorted([str(p.basename) for p in src_dir.readdir()]): + if not plugin.endswith(".so"): + continue + repository_ctx.symlink( + repository_ctx.path("{}/{}/{}".format(plugins_root, type_dir, plugin)), + "plugins/{}/{}".format(type_dir, plugin), + ) + files.append("plugins/{}/{}".format(type_dir, plugin)) + if files: + groups[type_dir] = files + + if "platforms" not in groups: + fail(("qt_plugins: {p} has no `platforms/` directory, so there is no QPA " + + "plugin to load and the GUI cannot start.").format(p = plugins_root)) + + # The private libraries the SDK's Qt modules need and the SDK itself ships. + private_libs = _sdk_private_libs(repository_ctx, values.get("QT_INSTALL_LIBS", "")) + + lines = [_BUILD_HEADER] + if private_libs: + names = [] + for lib in private_libs: + target = lib[len("lib/"):].replace(".", "_") + names.append(target) + lines.append(_RUNTIME_LIB_IMPORT.format(name = target, lib = lib)) + lines.append(_RUNTIME_LIBS_GROUP.format( + deps = ", ".join(["\":{}\"".format(n) for n in names]), + )) + else: + # A distro Qt: nothing to stage, and an empty target so the dep edge in + # BUILD.bazel is the same on every machine (finding 35 -- the alternative is + # a label that exists on some hosts and not others). + lines.append(_RUNTIME_LIBS_EMPTY) + for type_dir in sorted(groups.keys()): + lines.append("filegroup(\n name = \"{}\",\n srcs = [\n{} ],\n)\n".format( + type_dir, + "".join([" \"{}\",\n".format(f) for f in groups[type_dir]]), + )) + lines.append("filegroup(\n name = \"plugins\",\n srcs = [\n{} ],\n)\n".format( + "".join([" \":{}\",\n".format(t) for t in sorted(groups.keys())]), + )) + repository_ctx.file("BUILD.bazel", "\n".join(lines)) + + # The version the plugins ARE, recorded next to them so a consumer (and the + # provenance test) can compare it against the Qt the binary linked instead of + # trusting that they match. + repository_ctx.file("qt_plugins.bzl", "\n".join([ + "# GENERATED by qt_runtime.bzl (qt_plugins). Do not edit.", + "QT_PLUGINS_VERSION = \"{}\"".format(version), + "QT_PLUGINS_SOURCE = \"{}\"".format(plugins_root), + "QT_PLUGIN_TYPES = {}".format(str(sorted(groups.keys()))), + "", + ])) + +qt_plugins = repository_rule( + implementation = _qt_plugins_impl, + doc = """Exposes the Qt plugins of the SDK rules_qt discovered as Bazel files. + +Reads @qt's generated `qtconf.bzl` for `QT_INSTALL_PLUGINS`, so the plugins are +by construction from the same Qt as the `@qt//:Qt*` libraries the binary links -- +see the header of qt_runtime.bzl for what happens when they are not.""", + # local: the SDK is a host path, exactly like rules_qt's own qt.local_repo, + # so this must be re-evaluated rather than cached across host changes. + local = True, +) + +def _qt_runtime_ext_impl(module_ctx): + qt_plugins(name = "qt_plugins") + return module_ctx.extension_metadata(root_module_direct_deps = ["qt_plugins"], root_module_direct_dev_deps = []) + +qt_runtime = module_extension(implementation = _qt_runtime_ext_impl) + +# --------------------------------------------------------------------------- +# Staging: the plugins as outputs of the package that holds the binary. +# --------------------------------------------------------------------------- + +def _strip_to_plugins(short_path): + """`plugins/platforms/libqxcb.so` out of a runfiles-relative path. + + The path looks like `../qt_plugins/plugins/platforms/libqxcb.so`, so the match + has to be on the SEPARATED component `/plugins/` -- searching for `plugins/` + matches inside the repository name `qt_plugins/` and stages everything one + directory too deep (which is what it did, silently, until qt.conf pointed at + an empty tree). + """ + if short_path.startswith("plugins/"): + return short_path + idx = short_path.find("/plugins/") + if idx == -1: + fail("qt_plugin_tree: {} is not under a plugins/ directory".format(short_path)) + return short_path[idx + 1:] + +def _qt_plugin_tree_impl(ctx): + outs = [] + for src in ctx.files.plugins: + out = ctx.actions.declare_file(_strip_to_plugins(src.short_path)) + # A SYMLINK, not a copy: the plugin resolves its own private Qt libraries + # through `RUNPATH $ORIGIN/../../lib`, and $ORIGIN follows the real path. + ctx.actions.symlink(output = out, target_file = src) + outs.append(out) + return [DefaultInfo( + files = depset(outs), + runfiles = ctx.runfiles(files = outs), + )] + +qt_plugin_tree = rule( + implementation = _qt_plugin_tree_impl, + doc = """Stages @qt_plugins under `plugins/` in THIS package, so the files sit +next to the binary that qt.conf points at.""", + attrs = { + "plugins": attr.label_list( + allow_files = True, + doc = "Plugin files from @qt_plugins (`@qt_plugins//:plugins`).", + ), + }, +) + +def _qt_conf_impl(ctx): + out = ctx.actions.declare_file(ctx.attr.filename) + + # `Prefix = .` -- relative to the directory of the resolved executable, which + # is the one thing that is the same in bazel-bin and in the runfiles tree. + ctx.actions.write( + output = out, + content = "\n".join([ + "# GENERATED by //:{} (qt_runtime.bzl). Points Qt at the plugins staged".format(ctx.label.name), + "# beside the binary, so it cannot dlopen the host's plugins into Bazel's Qt.", + "[Paths]", + "Prefix = .", + "Plugins = {}".format(ctx.attr.plugins_dir), + "", + ]), + ) + return [DefaultInfo(files = depset([out]), runfiles = ctx.runfiles(files = [out]))] + +qt_conf = rule( + implementation = _qt_conf_impl, + doc = """Writes the qt.conf that redirects Qt's plugin search into the staged tree.""", + attrs = { + "filename": attr.string(default = "qt.conf"), + "plugins_dir": attr.string(default = "plugins"), + }, +) diff --git a/tests/test_qt_runtime.py b/tests/test_qt_runtime.py new file mode 100644 index 0000000..36d1852 --- /dev/null +++ b/tests/test_qt_runtime.py @@ -0,0 +1,207 @@ +"""The Qt RUNTIME edge: the plugins Qt dlopens must come from the Qt it linked. + +Case study finding 40. Ladybird's Bazel build linked Qt from @qt (rules_qt's +qt.local_repo, one SDK found via `qmake -query`) and then, at QApplication +construction, dlopened the QPA platform plugin from the HOST's plugin directory -- +because Qt resolves plugins against a prefix baked into libQt6Core, which for an +official SDK is EMPTY, so it falls back to the executable's directory and then the +compiled-in system path. Two Qt builds in one process: a SIGSEGV in +QXcbConnection::initializeScreens where the versions differ, and a silent pass +where they agree (which is why it survived so long in this sandbox). + +These tests guard the properties that make the fix a fix rather than a coincidence. +Each one corresponds to something that was actually wrong at some point while +writing it -- the staging depth bug, the missing use_repo, the emitter losing the +data edge on the next regeneration. + +The real proof is by removal (hide the host's Qt plugin dir, run with no +LD_LIBRARY_PATH, see the window open); this is the cheap regression guard for it. +""" + +import os +import re + +_WS = os.path.join(os.path.dirname(__file__), "..", "examples", "ladybird", + "workspace") + + +def _read(rel): + with open(os.path.join(_WS, rel)) as f: + return f.read() + + +def test_plugins_come_from_the_same_sdk_as_the_libraries(): + """The plugin path must be READ from @qt, never written down here. + + This is the whole fix in one assertion: @qt's qtconf.bzl is generated by + rules_qt from `qmake -query`, so taking QT_INSTALL_PLUGINS from it makes a + plugin/library version skew unrepresentable. A hardcoded + /usr/lib/x86_64-linux-gnu/qt6/plugins -- which is what the build effectively + had, by falling back to it -- is the bug. + """ + bzl = _read("qt_runtime.bzl") + assert 'Label("@qt//:qtconf.bzl")' in bzl, \ + "the plugin dir must be read from @qt's own generated qtconf.bzl" + assert 'values.get("QT_INSTALL_PLUGINS"' in bzl + # And no ABSOLUTE host plugin path may be a value the code uses. Getting this + # assertion right took three tries, and the near-misses are worth naming because + # each is a legitimate mention of such a path: + # * comments describing the bug (/usr/lib/x86_64-linux-gnu/qt6/plugins IS the + # wrong directory being explained -- a test that cannot tell prose from a + # path punishes the explanation), + # * _SYSTEM_LIB_DIRS, which lists system lib dirs precisely in order to + # RECOGNISE a distro Qt and stage nothing, + # * error text and docstrings naming plugins/platforms/libqxcb.so as an + # example. + # What must not appear is an absolute /usr/... path used as a plugin root, so the + # check is on absolute-path STRING LITERALS outside the system-dir list. + code = "\n".join([l for l in bzl.splitlines() + if not l.lstrip().startswith("#")]) + code = code.split("_SYSTEM_LIB_DIRS = [", 1) + code = code[0] + (code[1].split("]", 1)[1] if len(code) > 1 else "") + # (`/plugins/` is the path-COMPONENT matcher from the staging bug below, not a + # host path -- an absolute host path starts with a real top-level directory.) + host_roots = ("/usr/", "/opt/", "/home/", "/lib/", "/etc/") + for literal in re.findall(r'"(/[^"]*)"', code): + assert not literal.startswith(host_roots), \ + "a hardcoded host path defeats the point: %s" % literal + + +def test_qt_version_floor_is_checked(): + """finding 40's class: upstream declares a floor and the overlay must check it. + + UI/Qt/CMakeLists.txt has `find_package(Qt6 6.9 REQUIRED COMPONENTS Core + Widgets)`. CMake refuses an older Qt with a clear message; the Bazel build + compiled against whatever qmake was first on PATH. + """ + bzl = _read("qt_runtime.bzl") + assert "_QT_FLOOR = (6, 9)" in bzl, \ + "the floor must match UI/Qt/CMakeLists.txt's find_package(Qt6 6.9 REQUIRED)" + # The failure has to name the fix, not just the fact (finding 39's rule). + msg = bzl.split("qt_plugins: Qt {have} is too old", 1)[1].split(" ))", 1)[0] + assert "qt6-base-dev" in msg, "the error must name the package to install" + assert "MODULE.bazel" in msg, "the error must say where the SDK is chosen" + assert "{have}" in msg and "{prefix}" in msg, \ + "and the Qt it actually found, and where -- 'too old' alone is not actionable" + + +def test_missing_platform_plugin_is_an_error_not_an_empty_filegroup(): + """A check that cannot fail must not look like one that passed (finding 35). + + An SDK with no platforms/ directory yields an empty staging tree, a clean + build, and a GUI that cannot start. Same shape as glob(allow_empty = True). + """ + bzl = _read("qt_runtime.bzl") + assert '"platforms" not in groups' in bzl, \ + "a Qt with no QPA plugin must be a fetch-time failure" + + +def test_staging_strips_the_repo_name_not_the_substring(): + """`plugins/` is a substring of `qt_plugins/` -- and that was a real bug. + + Searching for "plugins/" in ../qt_plugins/plugins/platforms/libqxcb.so matches + inside the REPOSITORY NAME, so everything staged one directory too deep + (bazel-bin/plugins/plugins/...). It built cleanly and pointed qt.conf at an + empty tree. + """ + bzl = _read("qt_runtime.bzl") + assert 'find("/plugins/")' in bzl, \ + "the match must be on the separated path component, not the substring" + + +def test_plugins_are_symlinked_not_copied(): + """A plugin finds its own SDK's private libs through RUNPATH $ORIGIN/../../lib. + + $ORIGIN resolves from the object's REAL path, so a symlink keeps that working + and a copy breaks it -- reintroducing the very bug (the copy then resolves + libQt6XcbQpa.so.6 from /usr). + """ + bzl = _read("qt_runtime.bzl") + assert "ctx.actions.symlink(" in bzl, "the staged plugins must be symlinks" + assert "ctx.actions.copy" not in bzl + + +def test_qt_conf_replaces_the_prefix_relatively(): + """Prefix = . is what makes ONE file right in bazel-bin AND in runfiles. + + Setting Prefix at all is what stops /usr being scanned (it REPLACES the + compiled-in prefix rather than being appended to it). An absolute path would + be a host escape and would be wrong in one of the two layouts. + """ + bzl = _read("qt_runtime.bzl") + written = bzl.split("def _qt_conf_impl", 1)[1] + assert '"[Paths]"' in written + assert '"Prefix = ."' in written, "the prefix must be relative to the executable" + assert '"Plugins = {}".format(ctx.attr.plugins_dir)' in written + + +def test_the_binary_declares_them_as_runtime_inputs(): + """Staged files nobody depends on are files that are not there in a clean build.""" + build = _read("BUILD.bazel") + binary = build.split("name = 'ladybird'", 1)[1].split("\n)\n", 1)[0] + assert "':qt_conf'" in binary and "':qt_plugins'" in binary, \ + "//:ladybird must carry qt.conf and the plugin tree in data" + assert "'@qt_plugins//:runtime_libs'" in binary, \ + "and the SDK's private libraries as link inputs, or an SDK-bundled ICU " \ + "needs LD_LIBRARY_PATH at run time" + assert "qt_plugin_tree(" in build and "qt_conf(" in build + + +def test_the_emitter_generates_that_edge(): + """BUILD.bazel is GENERATED; an edge only in the file is an edge one run deletes. + + The same drift this project keeps finding (finding 37): a hand-appended tail + the emitter silently truncates on its next run. + """ + emitter = _read("Meta/emit_build_bazel.py") + assert "QT_RUNTIME_BLOCK" in emitter and "emit_qt_runtime()" in emitter + assert 'extra_data=[":qt_conf", ":qt_plugins"]' in emitter + assert '"@qt_plugins//:runtime_libs"' in emitter, \ + "the private-libs dep must come from QT_MAP's resolution, not by hand" + assert 'load(":qt_runtime.bzl", "qt_conf", "qt_plugin_tree")' in \ + emitter.split("ALL_LOADS = ", 1)[1], "the generated file needs the load()" + + +def test_module_bazel_names_the_repo(): + """bzlmod requires every extension-created repo in a use_repo, or it is invisible.""" + mod = _read("MODULE.bazel") + assert 'use_extension("//:qt_runtime.bzl", "qt_runtime")' in mod + assert 'use_repo(qt_runtime, "qt_plugins")' in mod + # Still kklochkov's rules_qt, still discovering the host SDK: this change adds + # the runtime half, it does not swap the ruleset or pin a different Qt. + assert 'bazel_dep(name = "rules_qt", version = "2.0.1")' in mod + assert "qt.local_repo(" in mod + + +def test_every_plugin_type_is_staged(): + """Not a hand-picked list of the four types the GUI needs today. + + A list drifts, symlinks cost nothing, and a missing input method or native file + dialog is a defect nobody notices for a month. + """ + bzl = _read("qt_runtime.bzl") + staging = bzl.split("def _qt_plugins_impl", 1)[1].split( + "qt_plugins = repository_rule(", 1)[0] + assert "root.readdir()" in staging, "the plugin types must be discovered" + for hardcoded in ('"platformthemes"', '"styles"', '"imageformats"'): + assert hardcoded not in staging, \ + "a hardcoded plugin type list is the thing to avoid (%s)" % hardcoded + + +def test_repo_rule_is_local(): + """The SDK is a host path; a cached repo would survive changing it. + + rules_qt's own qt.local_repo is local = True for the same reason. + """ + bzl = _read("qt_runtime.bzl") + rule = bzl.split("qt_plugins = repository_rule(", 1)[1].split(")\n", 1)[0] + assert "local = True" in rule + + +TESTS = [v for k, v in sorted(globals().items()) if k.startswith("test_")] + +if __name__ == "__main__": + for t in TESTS: + t() + print("ok", t.__name__) + print("%d passed" % len(TESTS)) From ba7dc9683b96ddf9e0e00b29732011875c4988d8 Mon Sep 17 00:00:00 2001 From: Ulf Adams Date: Thu, 13 Aug 2026 17:45:22 +0000 Subject: [PATCH 21/59] Ladybird: -fPIE from CMake gave qApp a copy relocation -- that is the remaining segfault Finding 40 fixed the plugin path and the GUI still segfaulted in QXcbConnection::initializeScreens with the same backtrace, on Ulf's aqt 6.9.2 SDK. QT_DEBUG_PLUGINS showed the plugins now resolving to his own SDK, info sharedlibrary showed one libQt6Core with identical build strings, and the offscreen QPA plugin crashed the same way -- so it was neither the plugins nor a version mix. At the fault rdi = 0 on `mov 0x8(%rdi),%rbx` inside doActivate: Qt emitted screenAdded from a null qApp, mid-QApplication-constructor. readelf explains it. aqt's libQt6Core has NO relocation against QCoreApplication::self (built with reduce_relocations, so it accesses its own BSS PC-relative); libQt6Gui reads it through the GOT; and bazel-bin/ladybird carried an R_X86_64_COPY for it, which moves the definition into the executable and repoints the GOT there. Core wrote one copy, Gui read the other. Debian's Qt has a GLOB_DAT for self, so the bug cannot appear against a distro Qt -- which is why only an official SDK saw it. The cause is one flag: CMake puts -fPIE on every executable target, the capture recorded it, and emit_build_bazel.py copied it into copts on all seven generated cc_binary targets. Bazel appends per-target copts AFTER the .bazelrc's --copt=-fPIC and the last of the pair wins for GCC, so the UI/Qt objects compiled -fPIE while everything around them compiled -fPIC. Qt's headers #error on exactly this, but only when __PIC__ is unset -- both flags were passed, so the check never fired and the build was clean. Fixed in the generator (DROPPED_TARGET_FLAGS), not in the generated file, since BUILD.bazel is output. Dropping it matches CMake's intent: Bazel compiles a cc_binary's objects PIC and links -pie. Reduced to eight lines without Bazel or Ladybird first (-fPIE crashes, -fPIC prints its screen count); -Wl,-z,nocopyreloc is not an alternative, it only converts the fault into an R_X86_64_PC32 overflow at link time. Verified by removal on both Qts and all six executables: 39 R_X86_64_COPY relocations (qApp, stdout, QString::_empty, 20-odd staticMetaObjects) now 0, the GUI starts against aqt 6.9.2 where it previously died under both xcb and offscreen, and the distro-Qt build is unchanged. Guarded by tests/test_pie_copy_relocation.py (5); finding 41 in the case study, with the note that a backtrace surviving a fix is not the same bug. 210/210. --- docs/CASE-ladybird-migration.md | 102 +++++++++++++- examples/ladybird/README.md | 13 ++ examples/ladybird/workspace/BUILD.bazel | 7 - .../workspace/Meta/emit_build_bazel.py | 38 ++++++ tests/test_pie_copy_relocation.py | 129 ++++++++++++++++++ 5 files changed, 281 insertions(+), 8 deletions(-) create mode 100644 tests/test_pie_copy_relocation.py diff --git a/docs/CASE-ladybird-migration.md b/docs/CASE-ladybird-migration.md index e770e7e..13ff2fb 100644 --- a/docs/CASE-ladybird-migration.md +++ b/docs/CASE-ladybird-migration.md @@ -2537,6 +2537,103 @@ the distro's), zero `/usr` directories are scanned, and against the host Qt the opens its window on Xvfb and stays up. Headless still renders the reference page. Passing because the host happened to agree is how all nine of these got here. +## Finding 41: the plugin fix was right and the crash stayed — `-fPIE` gave `qApp` a copy relocation + +Finding 40 fixed the plugin path, Ulf updated his tree, and the GUI segfaulted in +`QXcbConnection::initializeScreens` **again**, with the same backtrace. That is the +most useful shape a bug can have: it proves the previous fix was necessary, not +sufficient, and it means the earlier reasoning had a passenger. + +What the new evidence ruled out, before any theory: + +- `QT_DEBUG_PLUGINS=1` showed the scan hitting `bazel-out/.../bin/plugins/platforms` + and every plugin resolving to *his own* SDK's realpath. `/usr/lib/x86_64-linux-gnu/qt6` + was never touched. Finding 40's fix was working exactly as documented. +- `info sharedlibrary` showed **one** `libQt6Core` in the process, and the `Qt 6.9.2 + (x86_64-little_endian-lp64 ... GCC 10.3.1)` build strings of the Bazel-staged copy and + the SDK copy were byte-identical. Not a version mix, not two Qts. +- The **offscreen** QPA plugin crashed identically. Whatever this was, it was not the + platform plugin, not xcb, not the X server, not a screen enumeration quirk. + +### The bug + +At the fault, `rdi = 0` and `si_addr = 0x8`, on `mov 0x8(%rdi),%rbx` — inside +`doActivate` (`libQt6Core` + 0x1e89ce, just past +`QObjectPrivate::ConnectionData::deleteOrphaned`), reached from +`QWindowSystemInterface::handleScreenAdded` → `QGuiApplication::screenAdded`. Qt was +emitting a signal from a **null `qApp`**, in the middle of `QApplication`'s own +constructor, which had just set it. + +`readelf -rW` says why, in three lines: + +| object | relocation against `_ZN16QCoreApplication4selfE` | +|---|---| +| aqt `libQt6Core.so.6.9.2` | **none** — accessed PC-relative, its own BSS | +| aqt `libQt6Gui.so.6.9.2` | `R_X86_64_GLOB_DAT` — read through the GOT | +| `bazel-bin/ladybird` | **`R_X86_64_COPY`** — definition materialised in the exe | + +A copy relocation moves the *definition* of an extern data symbol into the +executable's BSS and repoints every GOT slot at it. So `QApplication`'s constructor +made Core write `self` in **Core's** BSS, while Gui read the **executable's** BSS, +which was still zero. The first emit through it dies. His distro Qt 6.4.2 *does* +carry a `GLOB_DAT` for `self` in Core, so both halves agree there and the bug cannot +appear — which is why this only ever showed up against an official SDK. + +The cause is one flag. CMake puts `-fPIE` on every executable target +(`CMAKE_POSITION_INDEPENDENT_CODE` + an exe), the capture recorded it faithfully, and +`emit_build_bazel.py` copied it into `copts` on all seven generated `cc_binary` +targets. Bazel appends per-target copts **after** the `.bazelrc`'s `--copt=-fPIC`, +and for GCC the last of the pair wins: the UI/Qt objects compiled `-fPIE` while every +library around them compiled `-fPIC`. Under `-fPIE` GCC may reference extern data +directly instead of through the GOT, and the linker turns that into the copy +relocation. Qt is built with `reduce_relocations` (in `mkspecs/qconfig.pri`; +Debian's is not), which is what makes Core's side of the disagreement PC-relative. + +**Qt diagnoses this and the diagnosis could not fire.** +`qcompilerdetection.h` has `#error "-fPIE is not sufficient if Qt was configured with +-DFEATURE_reduce_relocations=ON ... Compile your code with -fPIC and without -fPIE"` +— guarded by `#if ... || defined(__PIC__)`. Bazel passed **both** flags, so `__PIC__` +was defined at preprocess time and the guard never triggered. The build was clean and +the binary was broken: a compiler-authored checker, defeated by flag order. + +### Reduced, then fixed + +Eight lines, no Ladybird, no Bazel — `QGuiApplication` plus one `printf`, against the +same SDK: + +``` +g++ -fPIE -pie ... -> R_X86_64_COPY for self -> SIGSEGV (identical stack) +g++ -fPIC -pie ... -> GOT, no COPY reloc -> "instance=0x… screens=1" +``` + +`-Wl,-z,nocopyreloc` is **not** an alternative: it converts the same defect into +`Symbol _ZN16QCoreApplication4selfE causes overflow in R_X86_64_PC32`. The fix is to +stop asking for `-fPIE`, which is also not a divergence from CMake's semantics — +Bazel compiles a `cc_binary`'s objects PIC and links `-pie`, which is what +`POSITION_INDEPENDENT_CODE` was asking for. So the flag is dropped in the generator +(`DROPPED_TARGET_FLAGS`), not in the generated file, because `BUILD.bazel` is output +and a hand-edit survives exactly one regeneration. + +**Verified by removal**, on both Qts and all six executables: `R_X86_64_COPY` count +39 → **0** (including `qApp`, and incidentally `stdout`, `QString::_empty` and 20-odd +`staticMetaObject`s, every one of them the same latent hazard). Against the aqt 6.9.2 +SDK the GUI now starts where it previously segfaulted under *both* the xcb and the +offscreen plugin; against the distro Qt, unchanged. Guarded by +`tests/test_pie_copy_relocation.py` (5 tests), which asserts the generated file is +`-fPIE`-free, that the drop lives in the emitter and is consulted, that the reason is +written at the drop site, and that global `--copt=-fPIC` — the thing that makes +dropping `-fPIE` correct — is still in `bazelrc.txt`. + +**What this cost, and the lesson.** Two rounds on Ulf's machine for one crash, because +after finding 40 I read "SIGSEGV in `initializeScreens`" as "the plugin bug, again" +instead of as an unexplained fault. The plugin evidence was *checkable in one command* +(`QT_DEBUG_PLUGINS=1`) and I asked for it only on the second pass. A backtrace that +survives a fix is not the same bug; the frames say where a process died, never why. +It also makes finding 40's own "verified by removal" honest about its scope: hiding +the host plugin directory proved the plugins were right, and proved nothing at all +about the objects that linked them. Two Qts in one process was a real bug. One Qt with +two copies of one pointer was the next one down. + ## Plan for the rest - **Ring 1c — BUILD generation to compile+link parity, bottom-up.** AK → @@ -2626,7 +2723,10 @@ LibWeb alone 1,961 TUs (1,273 checked-in + 688 generated) with **zero** define/flag/include discrepancies vs CMake; 1,379/1,379 generated files byte-identical -> now 1,408/1,408 with every one of the build's 586 ninja CUSTOM_COMMANDs accounted for (findings 25-27); 77 vcpkg ports and 154 crates.io crates fetched and built by Bazel; **0 targets under `Build/full` in the closure of all six binaries (down from 741) and 0 absolute host paths in the emitted BUILD files** (finding 35); a fresh clone still needs 4 inputs the overlay does not carry and one port reaches PyPI via pip (finding 36 — found by actually cloning), and the host tools vcpkg cannot download are now named and preflighted rather than discovered one 20-minute build at a time -(findings 38, 39); 39 findings, 3 of them real any2bazel engine fixes with +(findings 38, 39); the Qt runtime half -- the plugins Qt `dlopen`s -- wired to the +same SDK as the libraries (finding 40) and the `-fPIE` copy relocation that broke +`qApp` against any `reduce_relocations` Qt removed (finding 41, 39 -> 0 +`R_X86_64_COPY`); 41 findings, 3 of them real any2bazel engine fixes with regression tests. **One number in this scoreboard was wrong for most of the migration, and it is diff --git a/examples/ladybird/README.md b/examples/ladybird/README.md index 9e880d7..3f9c5a7 100644 --- a/examples/ladybird/README.md +++ b/examples/ladybird/README.md @@ -698,6 +698,19 @@ Honest inventory of what stops this from being a clone-and-build. pass where they agree. `qt_runtime.bzl` stages the plugins of the SDK @qt itself names, points a generated `qt.conf` at them, and enforces the Qt >= 6.9 floor `UI/Qt/CMakeLists.txt` declares. Finding 40. + + The same crash then came back from the *other* direction, and it was not Qt's + doing: CMake puts `-fPIE` on every executable target, the generator copied it + into each `cc_binary`'s `copts`, and Bazel appends those AFTER the global + `--copt=-fPIC` — so the last flag won and the UI objects compiled `-fPIE`. That + lets GCC reference extern data PC-relative, the linker emits an + `R_X86_64_COPY`, and against a Qt built with `reduce_relocations` (every + official/aqt SDK; Debian's is built without it) `QCoreApplication::self` ends up + defined in the executable while `libQt6Core` writes its own copy: `qApp` is set + in one place and read, still null, in another. Qt's headers `#error` on exactly + this, but only when `__PIC__` is unset — Bazel passed both flags, so the check + never fired. The emitter now drops `-fPIE` (`DROPPED_TARGET_FLAGS`): 39 + `R_X86_64_COPY` relocations across the six executables, now 0. Finding 41. 7. **The fresh clone needs two inputs the overlay does not carry** (step 1a above), and each stayed invisible for the same reason: it was already on the machine. `Build/vcpkg` — a microsoft/vcpkg checkout at `vcpkg.json`'s diff --git a/examples/ladybird/workspace/BUILD.bazel b/examples/ladybird/workspace/BUILD.bazel index f0a0a34..0e5dc97 100644 --- a/examples/ladybird/workspace/BUILD.bazel +++ b/examples/ladybird/workspace/BUILD.bazel @@ -1599,7 +1599,6 @@ cc_binary( 'Services/ImageDecoder/SandboxLinux.cpp', 'Services/ImageDecoder/main.cpp', ], - copts = ['-fPIE'], linkopts = ['-lvulkan'], deps = [ '//:AK', @@ -1623,7 +1622,6 @@ cc_binary( srcs = [ 'Services/RequestServer/main.cpp', ], - copts = ['-fPIE'], linkopts = ['-lvulkan'], deps = [ '//:AK', @@ -1656,7 +1654,6 @@ cc_binary( 'Services/Compositor/SandboxLinux.cpp', 'Services/Compositor/main.cpp', ], - copts = ['-fPIE'], linkopts = ['-ldl', '-lpthread', '-lvulkan'], deps = [ '//:AK', @@ -1687,7 +1684,6 @@ cc_binary( 'Services/RendererSandboxLinux.cpp', 'Services/WebWorker/main.cpp', ], - copts = ['-fPIE'], linkopts = ['-lvulkan'], deps = [ '//:AK', @@ -1722,7 +1718,6 @@ cc_binary( 'Services/RendererSandboxLinux.cpp', 'Services/WebContent/main.cpp', ], - copts = ['-fPIE'], linkopts = ['-lvulkan'], deps = [ '//:AK', @@ -1763,7 +1758,6 @@ cc_binary( 'Libraries/LibJS/Interpreter/GenerateLayout.cpp', ], local_defines = ['private=public', 'protected=public'], - copts = ['-fPIE'], deps = [ '//:AK', '//:all_source_headers', @@ -1838,7 +1832,6 @@ cc_binary( 'UI/Qt/main.cpp', ], local_defines = ['QT_CORE_LIB', 'QT_GUI_LIB', 'QT_NO_DEBUG', 'QT_WIDGETS_LIB'], - copts = ['-fPIE'], linkopts = ['-lGLX', '-lOpenGL', '-lvulkan'], data = [':qt_conf', ':qt_plugins'], deps = [ diff --git a/examples/ladybird/workspace/Meta/emit_build_bazel.py b/examples/ladybird/workspace/Meta/emit_build_bazel.py index 39f465d..afbbaff 100644 --- a/examples/ladybird/workspace/Meta/emit_build_bazel.py +++ b/examples/ladybird/workspace/Meta/emit_build_bazel.py @@ -103,12 +103,50 @@ def global_flags(): GLOBAL_FLAGS = global_flags() +# Flags CMake puts on a target that must NOT be copied into a Bazel copt. +# +# -fPIE is the one that matters, and it is a crash, not a nit. CMake adds it to +# every executable target (CMAKE_POSITION_INDEPENDENT_CODE + an exe => -fPIE), +# and the capture faithfully recorded it -- so the generated cc_binary carried +# `copts = ['-fPIE']`. Bazel appends per-target copts AFTER the .bazelrc's +# --copt=-fPIC, and for GCC the LAST of -fPIC/-fPIE wins. So the UI/Qt objects +# were compiled -fPIE while every library around them was -fPIC. +# +# Under -fPIE, GCC may reference extern data DIRECTLY (PC-relative) instead of +# through the GOT, and the linker then materialises the definition in the +# executable with an R_X86_64_COPY relocation. For libraries built with Qt's +# `reduce_relocations` (every official/aqt Qt SDK; Debian's is built without it) +# that is fatal: QtCore accesses its own `QCoreApplication::self` PC-relative -- +# its own BSS copy -- while QtGui reads the same symbol through the GOT, which +# the copy relocation has repointed at the EXECUTABLE's BSS. QApplication's +# constructor then sets qApp in one place and QtGui reads the other, still null. +# The first emit through a null sender segfaults: QGuiApplication::screenAdded +# from QWindowSystemInterface::handleScreenAdded, inside doActivate, on +# `mov 0x8(%rdi),%rbx` with rdi = 0. +# +# Qt's own headers say so and are right: qcompilerdetection.h #errors with +# "-fPIE is not sufficient ... Compile your code with -fPIC and without -fPIE" +# -- but only when __PIC__ is unset, and Bazel passes BOTH flags, so __PIC__ is +# defined at preprocess time and the guard never fires. The build was clean and +# the binary was broken. +# +# Dropping -fPIE is not a divergence from CMake's semantics: Bazel's own +# toolchain compiles the objects of a cc_binary PIC and links -pie, which is +# what CMake was asking for. Verified by removal: with -fPIE gone the binary has +# ZERO R_X86_64_COPY relocations (39 before, incl. qApp) and the GUI starts +# against an aqt 6.9.2 SDK, where before it segfaulted for both the xcb and the +# offscreen QPA plugin. -Wl,-z,nocopyreloc is NOT an alternative: it turns the +# same defect into a link error ("causes overflow in R_X86_64_PC32"). +DROPPED_TARGET_FLAGS = {"-fPIE"} + def target_flags(t): """Per-target compile flags (feature/warning) not covered by .bazelrc.""" flags = set() for a in t["actions"]: if a["mnemonic"] != "CppCompile": continue for x in a["arguments"]: + if x in DROPPED_TARGET_FLAGS: + continue if x.startswith(("-f", "-m", "-p")) and x not in GLOBAL_FLAGS: flags.add(x) return sorted(flags) diff --git a/tests/test_pie_copy_relocation.py b/tests/test_pie_copy_relocation.py new file mode 100644 index 0000000..3c809e6 --- /dev/null +++ b/tests/test_pie_copy_relocation.py @@ -0,0 +1,129 @@ +"""-fPIE on a generated cc_binary is a crash, not a nit: the qApp copy relocation. + +Case study finding 41. CMake puts -fPIE on every executable target +(CMAKE_POSITION_INDEPENDENT_CODE + an exe), the capture recorded it faithfully, +and the generator copied it into `copts` on each generated cc_binary. Bazel +appends per-target copts AFTER the .bazelrc's --copt=-fPIC, and for GCC the LAST +of -fPIC/-fPIE wins -- so the UI/Qt objects were compiled -fPIE while every +library around them was -fPIC. + +Under -fPIE, GCC may reference extern data DIRECTLY (PC-relative) rather than +through the GOT, and the linker then materialises the definition inside the +executable with an R_X86_64_COPY relocation. Against a Qt built with +`reduce_relocations` -- every official/aqt SDK; Debian's is built without it -- +that is fatal for QCoreApplication::self: + + * libQt6Core accesses `self` PC-relative (no reloc against it at all): its OWN + BSS copy is the one QApplication's constructor writes. + * libQt6Gui reads the same symbol through the GOT (R_X86_64_GLOB_DAT), and the + copy relocation has repointed that GOT slot at the EXECUTABLE's BSS. + +So qApp is set in one place and read in another, which is still null, and the +first signal emitted through it segfaults: QGuiApplication::screenAdded from +QWindowSystemInterface::handleScreenAdded, inside doActivate, on +`mov 0x8(%rdi),%rbx` with rdi = 0. Reported as a SIGSEGV in +QXcbConnection::initializeScreens; reproduced identically under the offscreen QPA +plugin, which is what proved it was not a plugin problem at all. + +Qt's own headers diagnose this ("-fPIE is not sufficient ... Compile your code +with -fPIC and without -fPIE") but only when __PIC__ is unset -- and Bazel passes +BOTH flags, so __PIC__ is defined and the #error never fires. The build was clean +and the binary was broken, which is exactly the class of defect a generator test +has to carry. + +Measured, before and after, on the six generated executables: 39 +R_X86_64_COPY relocations (including QCoreApplication::self) before, 0 after, and +the GUI starts against the aqt 6.9.2 SDK where it previously died for both the +xcb and the offscreen plugin. -Wl,-z,nocopyreloc is NOT an alternative: it turns +the same defect into a link error ("causes overflow in R_X86_64_PC32"). +""" + +import os +import re + +_HERE = os.path.dirname(__file__) +_WS = os.path.join(_HERE, "..", "examples", "ladybird", "workspace") + + +def _read(rel): + with open(os.path.join(_WS, rel)) as f: + return f.read() + + +def test_no_generated_target_carries_fpie(): + """The generated BUILD file must not put -fPIE in any copts. + + This is the regression that shipped: seven cc_binary targets (the six services + plus ladybird) each with `copts = ['-fPIE']`. + """ + build = _read("BUILD.bazel") + assert "-fPIE" not in build, \ + "-fPIE in the generated BUILD file: qApp gets a copy relocation (finding 41)" + + +def test_the_emitter_drops_fpie_rather_than_the_checked_in_file_being_edited(): + """The fix has to live in the generator, or the next regeneration undoes it. + + BUILD.bazel is generated output; hand-editing it is how a fix survives exactly + one commit. The emitter carries an explicit drop list. + """ + emit = _read("Meta/emit_build_bazel.py") + assert "DROPPED_TARGET_FLAGS" in emit + dropped = emit.split("DROPPED_TARGET_FLAGS = ", 1)[1].split("\n", 1)[0] + assert "-fPIE" in dropped + # And it must actually be consulted in the flag loop, not merely defined. + body = emit.split("def target_flags(", 1)[1].split("\ndef ", 1)[0] + assert "DROPPED_TARGET_FLAGS" in body + + +def test_the_drop_is_explained_where_it_happens(): + """A bare `if x == "-fPIE": continue` reads like a style preference. + + The next person to see CMake pass -fPIE will put it back unless the comment + says what breaks. Require the mechanism (copy relocation) and the symptom + (qApp / QCoreApplication::self) to be named at the drop site. + """ + emit = _read("Meta/emit_build_bazel.py") + head = emit.split("DROPPED_TARGET_FLAGS = ", 1)[0] + note = head[-4000:] + for token in ("copy relocation", "R_X86_64_COPY", "QCoreApplication::self", + "reduce_relocations", "-fPIC"): + assert token in note, "the -fPIE drop must explain %r" % token + + +def test_global_fpic_is_still_passed(): + """Dropping -fPIE only works because -fPIC is global. + + If the .bazelrc's --copt=-fPIC ever goes away, the objects become whatever + the host GCC defaults to (Ubuntu: -fPIE), and the copy relocation returns by + another road. + """ + rc = _read("bazelrc.txt") + assert re.search(r"^build --copt=-fPIC$", rc, re.M), \ + "global --copt=-fPIC is what makes dropping -fPIE correct" + + +def test_every_generated_executable_is_covered(): + """Not just //:ladybird: the services link Qt-adjacent libraries too. + + The measurement was taken on all six; the guard should be the same set, so a + new cc_binary that picks up -fPIE cannot slip through by not being ladybird. + """ + build = _read("BUILD.bazel") + names = re.findall(r"cc_binary\(\s*\n\s*name = '([^']+)'", build) + for expected in ("ladybird", "WebContent", "RequestServer", "ImageDecoder", + "Compositor", "WebWorker"): + assert expected in names, "expected a cc_binary for %s" % expected + # Every one of them, and any future one, must be -fPIE free -- which the + # whole-file assertion above already covers, so this is the inventory that + # keeps that assertion meaningful. + assert len(names) >= 6 + + +TESTS = [v for k, v in sorted(globals().items()) if k.startswith("test_")] + +if __name__ == "__main__": + for t in TESTS: + t() + print("ok", t.__name__) + print("%d passed" % len(TESTS)) From 756e4d9a1367526239a7e0121f5d6ae07c29b95b Mon Sep 17 00:00:00 2001 From: Ulf Adams Date: Sat, 15 Aug 2026 17:51:37 +0000 Subject: [PATCH 22/59] Ladybird: the fd leak is one socket per HTTP request, not MessagePort Ulf's browser died with EMFILE. His two /proc//fd censuses (17,423 -> 17,497 socket:, flat 18 pipe:) falsified my own MessagePort diagnosis: that leak allocates two pipe2 pairs per socketpair, so it leaks pipes and sockets at 4:1 (reproduced locally at 1,628:409) and cannot produce a sockets-only census. Following the signature instead: every completed HTTP request leaks exactly one socket fd in WebContent (103 requests -> 107 sockets, 1,164 -> 1,157, pipes never move; also 50 -> 57). internals.dumpGCGraph() names the holder, because GC roots carry a source location: 808 roots at Fetching.cpp:2338 after 202 requests = 4 callbacks x 202. The four fetch callbacks are GC::Roots moved onto the refcounted Requests::Request, which Response holds back by RefPtr -- a cycle across the GC heap and the refcount heap that neither collector can break, keeping the response fd open. Only Request::defer_teardown() clears the callbacks, and normal completion never calls it; aborted fetches, which do reach stop(), leak zero. Also recorded: the obvious one-line fix (defer_teardown() in did_finish()) is wrong. I built it and the leak went to zero because loading broke -- blank pages. Reverted, re-verified the clean build renders and leaks, and took the diagnosis upstream without claiming a patch. All measurements are from a CMake build of f9e34731; the code paths are unchanged in upstream master (50eef049). Finding 42 + docs/UPSTREAM-ladybird-fd-leaks.md. --- docs/CASE-ladybird-migration.md | 45 +++++++ docs/UPSTREAM-ladybird-fd-leaks.md | 201 +++++++++++++++++++++++++++++ 2 files changed, 246 insertions(+) create mode 100644 docs/UPSTREAM-ladybird-fd-leaks.md diff --git a/docs/CASE-ladybird-migration.md b/docs/CASE-ladybird-migration.md index 13ff2fb..bcb7158 100644 --- a/docs/CASE-ladybird-migration.md +++ b/docs/CASE-ladybird-migration.md @@ -2737,6 +2737,51 @@ this document is only as strong as what was actually removed — which is the ar for removal as a method, not against it: it is the only check here that ever found this class of bug, and each time I widened what I removed, it found another. +## Finding 42: the browser ran, then ran out of file descriptors — and my first four diagnoses were theories, not counts + +The Bazel-built browser worked and then died overnight with `dup: Too many open files +(errno=24)`. This finding is not about the migration at all: it is an upstream Ladybird +bug that a long-running browser finds and a short test never does. It earns a place here +because of *how badly I diagnosed it*, four times, before doing the obvious thing. + +The wrong answers, in order: lost fd acknowledgements in `TransportSocket` (Ulf correctly +objected that a local `SOCK_STREAM` socketpair does not lose acks); a missed +`AnonymousBuffer` / image-frame cache eviction (there were no memfds at all); leaked +`TransportSocket`s (an artifact of **my own broken one-liner** — `sed 's/[0-9]*$//'` does +not strip `pipe:[123]`, and `| head` hid the real distribution); and then +`MessagePort::entangle_with`, which is a real leak that I read out of the code with high +confidence and which **is not the bug that killed his browser**. + +What settled it was a count, not an argument. Ulf's two `/proc//fd` censuses showed +17,423 → 17,497 `socket:` with a *flat* 18 `pipe:`. The MessagePort leak allocates one +socketpair **and two `pipe2` pairs** per port, so it leaks pipes and sockets at 4:1 — I +reproduced it locally at 1,628 pipes : 409 sockets. **It cannot produce a sockets-only +census.** The signature falsified my own best theory. + +Following the signature instead led to a leak of exactly **one socket per completed HTTP +request** (103 requests → 107 sockets; 1,164 → 1,157; pipes never move), which matches +Ulf's shape exactly and needs no exotic page — any browsing session leaks monotonically. +`internals.dumpGCGraph()` names the holder directly, because GC roots carry their source +location: after 202 requests, `808 Root +nonstandard_resource_loader_file_or_http_network_fetch Fetching.cpp:2338` — 4 roots × +202 requests. The four fetch callbacks are `GC::Root`s (strong, uncollectable) moved into +lambdas on the refcounted `Requests::Request`, which the `Response` then holds back by +`RefPtr` — a cycle spanning the GC heap and the refcount heap, which neither collector +can break, keeping the response fd open forever. `Request::defer_teardown()` is the only +thing that clears the callbacks, and normal completion never calls it (only `stop()` and +`did_transfer()` do). The A/B: aborted fetches, which *do* reach `stop()`, leak zero. + +Then the last lesson, which is the one I keep having to relearn: I wrote the "obvious" +one-line fix (`defer_teardown()` at the end of `did_finish()`), rebuilt, and the leak went +to **zero** — because loading was broken and pages rendered blank. A fix that makes the +symptom disappear by removing the behaviour is indistinguishable from a fix, unless you +check that the feature still works. I reverted it, re-verified that the clean build both +renders *and* leaks, and took the diagnosis upstream without pretending I had the patch. + +Write-up for upstream: `docs/UPSTREAM-ladybird-fd-leaks.md` (both bugs, with the +reproductions, the census method, and the cascade that turns one `EMFILE` into three dead +processes via `MUST`/`VERIFY`). + ## Environment notes (this sandbox) - Toolchain via apt (needs passwordless sudo): cmake, ninja, ccache, diff --git a/docs/UPSTREAM-ladybird-fd-leaks.md b/docs/UPSTREAM-ladybird-fd-leaks.md new file mode 100644 index 0000000..0615668 --- /dev/null +++ b/docs/UPSTREAM-ladybird-fd-leaks.md @@ -0,0 +1,201 @@ +# Ladybird: two fd leaks in WebContent (one per HTTP request, one per un-`close()`d MessagePort) + +Status: ready to file upstream. Both reproduced locally on a **CMake** build of +`f9e34731` (no Bazel involved); the primary one is present in upstream `master` +(`50eef049`) by inspection of the same code paths. + +Ulf's Bazel-built browser died overnight with + +``` +dup: Too many open files (errno=24) at Libraries/LibWebView/CompositorConnection.cpp:62 +``` + +Two `/proc//fd` censuses minutes apart showed **17,423 → 17,497 `socket:`**, with +only 18 `pipe:` and nothing else growing. Monotonic, unbounded, and *sockets only*. + +That last detail is what makes this report two bugs instead of one. My first diagnosis +(`MessagePort::entangle_with`) was **wrong for Ulf's crash**: it leaks 4 pipes for every +1 socket, so it cannot produce a sockets-only census. Counting fds by category, rather +than reasoning about which code looked suspicious, is what separated them — and the +per-request leak below matches his census exactly. + +--- + +## Bug 1 (the one that kills the browser): every completed HTTP request leaks one socket fd + +### Reproduction + +Serve a one-line `index.html` on localhost and load: + +```html + +``` + +`Build/full/bin/Ladybird --headless=screenshot --screenshot-delay 120 `, then census +the WebContent process: + +``` +ls -l /proc/$P/fd | awk '{print $NF}' | sed 's/\[[0-9]*\]//' | sort | uniq -c | sort -rn +``` + +Measured (clean build, no local patches): + +| requests served | WebContent `socket:` fds | `pipe:` fds | +|---|---|---| +| 0 (baseline) | 5 | 18 | +| 103 | 107 | 18 | +| 202 | 208 | 18 | +| 595 | 587 | 18 | +| 1164 | 1157 | 18 | + +**One socket per request, forever, and the pipe count never moves** — Ulf's census shape. +They are never reclaimed: the count stays flat for minutes after the loop stops, across +repeated explicit `internals.gc()` calls, and with `--disable-http-memory-cache +--disable-http-disk-cache`. `/proc/net/unix` shows them as connected `AF_UNIX` +`SOCK_STREAM` halves — RequestServer's response-body pipe (`RequestPipe::create`, +`Services/RequestServer/RequestPipe.cpp:46`). RequestServer closes its end (its own fd +count is flat at 4 sockets); **WebContent never closes its end.** + +It is not specific to `fetch()`: plain subresource loads leak identically — 50 `` +loads give 57 sockets, 200 give 207 (same +1/request against the same baseline of 5–7). + +### Cause: an explicit GC root anchoring a cycle that spans the GC heap and the refcount heap + +`internals.dumpGCGraph()` labels roots with their source location, which names the holder +outright. After 202 requests and two forced GCs: + +``` +808 Root nonstandard_resource_loader_file_or_http_network_fetch Libraries/LibWeb/Fetch/Fetching/Fetching.cpp:2338 +333 VM + 35 StackPointer +``` + +808 = **4 × 202**: the four callbacks passed to `ResourceLoader::load` at +`Fetching.cpp:2338`, one set per request, none ever released. Live cells scale in lockstep +(202 requests → 202 `FetchedDataReceiver`, 606 `Response`, 808 `PendingResponse`, 406 +`ReadableStream`). + +The loop: + +1. `nonstandard_resource_loader_file_or_http_network_fetch` creates four + `GC::Function`s (`Fetching.cpp:2261`–`2333`) and passes them as + `GC::Root<...>` into `ResourceLoader::load` + (`Libraries/LibWeb/Loader/ResourceLoader.h:43`). +2. `GC::Root` is a **strong, explicit root**: `RootImpl`'s constructor registers itself in + `Heap::m_roots` (`LibGC/Root.cpp:15`, `Heap.h:251`) and `gather_roots` marks every entry + (`Heap.cpp:841`). A `GC::Root` is uncollectable by construction — it is only released + when the `RootImpl` is destroyed. +3. `ResourceLoader::load` moves those roots into the lambdas it installs on the + **refcounted** `Requests::Request` (`ResourceLoader.cpp:469`–`513`, stored as + `on_headers_received` / `on_finish` / …). +4. `on_headers_received` puts the very same `Requests::Request` **back into the GC heap**: + `response->set_request_server_request({… .request = request_server_request})` + (`Fetching.cpp:2276`), and `Response` holds it as a + `RefPtr` (`Fetch/Infrastructure/HTTP/Responses.h:223`/`226`). + +So: `Heap::m_roots` → `GC::Function` → captured `pending_response` / `stream` / +`fetched_data_receiver` → `Response` → `RefPtr` → the lambdas holding +the `GC::Root`s. Neither mechanism can break it — the GC sees a live root, and the +refcount never reaches zero. `Requests::Request` owns the response fd (`m_fd`, closed only +in `~Request`, `LibRequests/Request.cpp:58`, and kept open meanwhile by the `ReadStream` / +`Core::Notifier`), so **the fd leaks with the cycle**. + +The only thing that ever clears those callbacks is `Request::defer_teardown()` +(`Request.cpp:284`), and its only callers are `stop()` (`:69`) and `did_transfer()` +(`:274`). **Normal completion never calls it**: `did_finish()` (`:251`) invokes +`on_finish` and returns, and `RequestClient::request_finished` (`RequestClient.cpp:205`) +only removes its map entry — the `RefPtr` inside `Response` keeps the request, its +callbacks, its roots and its fd alive for the lifetime of the process. + +### A/B that confirms it + +Same page, but each fetch is immediately `AbortController.abort()`ed — `abort` reaches +`Request::stop()`, which *does* call `defer_teardown()`: + +| variant | 200 requests | +|---|---| +| completed fetches | +200 sockets (207) | +| aborted fetches | **+0 sockets (flat 207 → 207)** | + +The path that tears down does not leak; the path that succeeds does. + +### On fixing it — one obvious patch is wrong, and I verified that + +Calling `defer_teardown()` at the end of `did_finish()` looks like the one-line fix. **It +is not**: I applied exactly that, rebuilt `liblagom-requests`, and the leak went to zero +(flat 5 sockets) *because loading broke* — pages rendered blank, since the body may still +be draining when `did_finish` arrives (`on_finish` re-checks +`read_stream->is_eof()` / `user_finish_called` for precisely this reason). I reverted it +and re-verified the clean build renders and leaks again; the numbers above are all from +the unpatched build. + +A correct fix has to release the callbacks (and with them the `GC::Root`s) only once the +body is genuinely finished — e.g. at the point `on_finish` decides +`user_finish_called`/EOF — or break the cycle at its other end so `Response` does not keep +the `Requests::Request` alive strongly. Which end upstream prefers is your call; the +diagnosis is what I'm confident in. + +### Why it takes the whole browser down, not just a tab + +`EMFILE` in WebContent surfaces through `MUST()` on an encode +(`LibWebView/CompositorConnection.cpp:62`) — an abort, not a propagated error. WebContent +dies; the Compositor's `VERIFY(connection)` +(`Services/Compositor/ConnectionFromClient.cpp:68`) then aborts; and the UI process's +`MUST` in `initialize_client` follows. **One `EMFILE` kills three processes**, which is +also why the failure is unattributable after the fact: nothing in the log names the +resource that ran out. Two `/proc//fd` censuses a few minutes apart is what turned it +into a bug report. + +--- + +## Bug 2 (independent, smaller): a `MessagePort` dropped without `close()` leaks its socketpair + +Distinguishable by signature: this one leaks **2 pipes + 1 socket per port** (each +`TransportSocket` makes two `pipe2` pairs at `TransportSocket.cpp:165,170`), so a page doing +this shows a *pipe*-dominated census — not what Ulf saw. + +```html + +``` + +Result: WebContent reaches 1,628 `pipe:` + 409 `socket:` and dies with + +``` +UNEXPECTED ERROR: pipe2: Too many open files (errno=24) at Libraries/LibIPC/TransportSocket.cpp:165 +``` + +The same page with `c.port1.close(); c.port2.close()` stays flat at 18 pipes — so the leak +is scoped to ports dropped **without** `close()`, which is the ordinary case in real pages. + +`MessagePort::entangle_with` (`LibWeb/HTML/MessagePort.cpp:222`/`230` upstream) installs +read hooks capturing `GC::make_root(this)` and `GC::make_root(m_remote_port)` — again +strong explicit roots, stored in a callback owned by the transport owned by the port, so +the port roots itself. `close()` reaches `disentangle()` (`:472`), which is the only thing +that closes the transport; a port that is merely dropped never gets there. Same *shape* as +bug 1 (a `GC::Root` captured into something the rooted object owns), different site. + +--- + +## Method note + +Three of my earlier theories about Ulf's crash were wrong (lost fd acks; an +`AnonymousBuffer`/image-cache miss; leaked `TransportSocket`s), and one was wrong only +because my own census one-liner was broken — `sed 's/[0-9]*$//'` does not strip +`pipe:[123]`, and piping through `head` hid the answer. The census command that works: + +``` +ls -l /proc/$P/fd | awk '{print $NF}' | sed 's/\[[0-9]*\]//' | sort | uniq -c | sort -rn +``` + +Everything above is either a counted fd, a labelled GC root, or an A/B against a rebuilt +binary. From 19010734258cdfdd3d9d3ff40174f9ee05890339 Mon Sep 17 00:00:00 2001 From: Ulf Adams Date: Sun, 16 Aug 2026 15:50:50 +0000 Subject: [PATCH 23/59] Ladybird fd leak: the teardown fix is one of two bugs, split by peer liveness Ulf applied the equivalent upstream patch and his browser still leaked. That is not a contradiction and not a regression in the patch: the per-request leak is two bugs, and the patch only reaches one of them. The discriminator is whether the OTHER end of the retained response fd is gone -- one column of `ss -np` (peer inode `* 0`), which I can now also probe in-process. Class A: the request completed, RequestServer closed its half, WebContent retains a corpse (peer DEAD) -- the teardown patch takes this from 143 retained sockets to 0. Class B: `on_finish` never ran at all, so no teardown placed in that branch can fire, and RequestServer is still holding its end (peer ALIVE) -- 40 retained, unchanged by the patch. Class B is the GC-root/RefPtr cycle itself and needs the cycle broken at the Response end, not another teardown call site. Adds patches/0003 (the teardown fix, framed as necessary-not-sufficient, with the scope note in its header) and a DIAGNOSTIC census patch that is deliberately NOT matched by apply_overlay.sh's patches/*.patch glob -- pinned by a new test, since only its extension keeps a per-request HashMap, a repeating timer and a poll() probe out of a browser someone is using. The census reports in_flight-vs-retained by age (so a freshly restarted process cannot look like a fix -- a mistake I made earlier here) and peer DEAD/ALIVE per retained request, and carries LADYBIRD_FDLEAK_TEARDOWN so one build A/Bs the fix. Corrects UPSTREAM-ladybird-fd-leaks.md, which overstated the fix's completeness, and adds the postscript to finding 42: the instrument should answer the classification question, so the next report is a count instead of another theory. --- docs/CASE-ladybird-migration.md | 22 +- docs/UPSTREAM-ladybird-fd-leaks.md | 79 ++- examples/ladybird/README.md | 30 +- examples/ladybird/apply_overlay.sh | 2 +- ...-down-request-when-body-is-delivered.patch | 62 +++ .../DIAGNOSTIC-fdleak-census.patch.txt | 452 ++++++++++++++++++ tests/test_apply_overlay.py | 23 +- 7 files changed, 653 insertions(+), 17 deletions(-) create mode 100644 examples/ladybird/patches/0003-librequests-tear-down-request-when-body-is-delivered.patch create mode 100644 examples/ladybird/patches/DIAGNOSTIC-fdleak-census.patch.txt diff --git a/docs/CASE-ladybird-migration.md b/docs/CASE-ladybird-migration.md index bcb7158..269e295 100644 --- a/docs/CASE-ladybird-migration.md +++ b/docs/CASE-ladybird-migration.md @@ -2737,7 +2737,7 @@ this document is only as strong as what was actually removed — which is the ar for removal as a method, not against it: it is the only check here that ever found this class of bug, and each time I widened what I removed, it found another. -## Finding 42: the browser ran, then ran out of file descriptors — and my first four diagnoses were theories, not counts +## Finding 42: the browser ran, then ran out of file descriptors — and my first four diagnoses were theories, not counts (and my fifth, the patch, was only half of it) The Bazel-built browser worked and then died overnight with `dup: Too many open files (errno=24)`. This finding is not about the migration at all: it is an upstream Ladybird @@ -2782,6 +2782,26 @@ Write-up for upstream: `docs/UPSTREAM-ladybird-fd-leaks.md` (both bugs, with the reproductions, the census method, and the cascade that turns one `EMFILE` into three dead processes via `MUST`/`VERIFY`). +**Postscript, and the actual lesson.** An upstream patch equivalent to that fix landed; +Ulf applied it; his browser still leaked. My patch was necessary and not sufficient, and I +had written the doc as though it were the fix. What resolved it was making the census +*self-classifying* instead of arguing about which page shape was to blame: an in-process +`poll()`/`MSG_PEEK` probe on each retained response fd, which reports whether the **peer** +is dead or alive. That one column splits the leak into two bugs with different fixes — +peer **dead** means the request completed and WebContent is retaining a corpse (the +teardown patch fixes exactly this, 143 → 0 locally); peer **alive** means `on_finish` +never ran, so no teardown at that point can fire, and the fd is held by the GC-root ↔ +`RefPtr` cycle itself (40 → 40, unchanged by the patch). The same probe also separates +*in-flight* from *retained* by age, which is what stops a freshly restarted process from +looking like a fix — a mistake I made once already in this investigation. + +Two habits earned that: measuring the resource rather than reasoning about the code (the +`pipe:`-to-`socket:` ratio is what falsified my confident MessagePort diagnosis), and +making the *instrument* answer the classification question, so the next report is a count +instead of another theory. The diagnostic build is kept, deliberately outside the overlay's +`patches/*.patch` glob and pinned there by a test, as +`examples/ladybird/patches/DIAGNOSTIC-fdleak-census.patch.txt`. + ## Environment notes (this sandbox) - Toolchain via apt (needs passwordless sudo): cmake, ninja, ccache, diff --git a/docs/UPSTREAM-ladybird-fd-leaks.md b/docs/UPSTREAM-ladybird-fd-leaks.md index 0615668..b5207fc 100644 --- a/docs/UPSTREAM-ladybird-fd-leaks.md +++ b/docs/UPSTREAM-ladybird-fd-leaks.md @@ -1,8 +1,29 @@ -# Ladybird: two fd leaks in WebContent (one per HTTP request, one per un-`close()`d MessagePort) +# Ladybird: fd leaks in WebContent (per HTTP request, and per un-`close()`d MessagePort) -Status: ready to file upstream. Both reproduced locally on a **CMake** build of -`f9e34731` (no Bazel involved); the primary one is present in upstream `master` -(`50eef049`) by inspection of the same code paths. +Status: reproduced locally on a **CMake** build of `f9e34731` (no Bazel involved); +present in upstream `master` (`50eef049`) by inspection of the same code paths. + +**Read this first — the per-request leak is two distinct bugs, not one.** An upstream +patch equivalent to the teardown fix below now exists, Ulf applied it, and *his browser +still leaks*. That is not a contradiction: the fix closes the class of retained request +whose peer is already **dead**, and there is a second class whose peer is still **alive** +that no teardown at that point can reach. The discriminator is one column of `ss -np`: + +| class | `ss -np` peer inode | what happened | teardown patch | +|---|---|---|---| +| A: completed | `* 0` (**dead**) | request finished, RequestServer closed its half, WebContent retains a corpse | **fixes it** (143 → 0 locally) | +| B: stalled | a real inode (**alive**) | `on_finish` never ran, so no teardown fires at all | **unaffected** (40 → 40 locally) | + +Count them before choosing a fix: + +``` +ss -np | grep "pid=$P," | grep -c ' \* 0 ' # class A (dead peer) +ss -np | grep -c "pid=$P," # all of this process's unix sockets +``` + +A build that self-classifies (in-process `poll()`/`MSG_PEEK` peer probe, plus +in-flight-vs-retained ages so a fresh process cannot be mistaken for a fix) is in +`examples/ladybird/patches/DIAGNOSTIC-fdleak-census.patch.txt`. Ulf's Bazel-built browser died overnight with @@ -127,6 +148,47 @@ Same page, but each fetch is immediately `AbortController.abort()`ed — `abort` The path that tears down does not leak; the path that succeeds does. +### Class B: the request that never finishes at all (survives the teardown fix) + +The teardown above hangs off the branch in `Request::set_up_internal_stream_data`'s +`on_finish` that decides the body is complete: + +```cpp +if (!user_finish_called && (!read_stream || read_stream->is_eof() || has_received_all_reported_bytes)) { +``` + +If a response never satisfies that *and* never reaches EOF, the block never runs, so +`user_on_finish` never runs, so **no teardown placed inside it can ever fire**. Repro: a +server that sends `Content-Length: 48000`, writes 100 bytes and then holds the socket +open. 40 such loads, censused with the diagnostic build after they have aged past 30s: + +``` +FDLEAK live: in_flight(<30s)=0 retained(>=30s)=40 +FDLEAK retained-bucket 40 x peer=ALIVE torn_down=false did_finish=false user_finish=false + request_done=false stream=true eof=false fd_open=true +``` + +Identical with `LADYBIRD_FDLEAK_TEARDOWN=1` (40 → 40): `did_finish=false` is the tell — +the `request_finished` IPC never arrived, so the fix is not even on the code path. And +`peer=ALIVE` distinguishes it from class A at a glance: RequestServer is still holding its +end, because as far as it knows the body is still being produced. + +By contrast every completed-request shape I can construct — plain fetch, XHR, POST, +`response.clone()`, unread bodies, ``/CSS/JS subresources, 404/302/cache hits, +truncated bodies, gzip/chunked, iframe navigations, mid-body `cancel()` — is flat at the +5-socket baseline with the fix on, and shows `peer=DEAD` retention without it. + +So the two classes need different fixes, and only one of them is "release the callbacks +when the body is delivered": + +- **Class A** is a *liveness* bug in the teardown path → the teardown patch. +- **Class B** is the cycle itself. As long as `Response` holds `Requests::Request` by + `RefPtr` while the request's callbacks hold `GC::Root`s back into the GC heap, ANY + request that never completes is retained forever, and there is no completion callback + left to hook. The fix has to break the cycle at the `Response` end — e.g. hold the + request weakly, or root the callbacks from something whose lifetime the GC can actually + end — rather than adding another teardown call site. + ### On fixing it — one obvious patch is wrong, and I verified that Calling `defer_teardown()` at the end of `did_finish()` looks like the one-line fix. **It @@ -139,9 +201,12 @@ the unpatched build. A correct fix has to release the callbacks (and with them the `GC::Root`s) only once the body is genuinely finished — e.g. at the point `on_finish` decides -`user_finish_called`/EOF — or break the cycle at its other end so `Response` does not keep -the `Requests::Request` alive strongly. Which end upstream prefers is your call; the -diagnosis is what I'm confident in. +`user_finish_called`/EOF, which is what the upstream patch and +`patches/0003-*.patch` do — **and** break the cycle at its other end so `Response` does +not keep the `Requests::Request` alive strongly, which is the only thing that can reach +class B. Doing only the first is measurable progress and not a fix: it is the same shape +as the `-fPIE` finding, where the change was necessary, verified, and still left the +symptom alive by another route. ### Why it takes the whole browser down, not just a tab diff --git a/examples/ladybird/README.md b/examples/ladybird/README.md index 3f9c5a7..b4bcd32 100644 --- a/examples/ladybird/README.md +++ b/examples/ladybird/README.md @@ -117,7 +117,7 @@ knowing, because both were mistakes I made first: ### Reproducing the tree on another machine -The overlay is **not a fork**: it is a pinned upstream Ladybird commit + two patches +The overlay is **not a fork**: it is a pinned upstream Ladybird commit + three patches + 45 Bazel files that sit alongside CMake's. [`apply_overlay.sh`](apply_overlay.sh) is that sentence made executable, because *"copy `workspace/` over a clone"* has four ways to be silently wrong — and every one of them was found by running it, not by @@ -129,8 +129,11 @@ reading it: (`f9e34731`). Against a different tree the build fails on a moved file — or worse, silently omits a new one. 2. **The patches**, which have to be applied or the build is nondeterministic - (dictionaries emitted in `PYTHONHASHSEED` order) or does not compile at all - under per-header moc. + (dictionaries emitted in `PYTHONHASHSEED` order), does not compile at all + under per-header moc, or leaks one socket fd per completed HTTP request until + the browser dies of `EMFILE` overnight + ([`0003`](patches/0003-librequests-tear-down-request-when-body-is-delivered.patch); + see [`docs/UPSTREAM-ladybird-fd-leaks.md`](../../docs/UPSTREAM-ladybird-fd-leaks.md)). 3. **The rename**: `bazelrc.txt` → `.bazelrc`. Stored under a different name so a `cp -r` cannot be mistaken for a working build — and a rename a human does by hand is a rename a human forgets. @@ -285,7 +288,7 @@ an FFI header collision and an entire missing Rust target. See | `Meta/vcpkg_capture_git_archives.sh` | Regenerates that pin with `vcpkg install --only-downloads` (~6 min, no compilation, no CMake) — vcpkg as the instrument, since which git externals are used is decided by feature-conditional CMake code, not by portfile text | | `hsts_preload.bzl` | Chromium's HSTS preload table as one `http_file`, pinned to a **commit** + sha256 — the downstream pin for the one input upstream CMake fetches from `main`. **Generated** by `Meta/pin_hsts_preload.py` | | `Meta/pin_hsts_preload.py` | Re-pins it: resolves the newest commit touching the path, downloads it, writes the hash it **measured**; `--expect-same-as` refuses to write unless the pinned bytes equal the file CMake downloaded (parity guard) | -| `apply_overlay.sh` | Reproduces the whole tree on another machine: clone at the pinned Ladybird commit, apply the two patches, copy the 45 overlay files (with the `bazelrc.txt` → `.bazelrc` rename), run the vcpkg prefetch, then stage the one file that must come after it. `--verify` checks an existing tree — commit, bytes, patches-applied, exec bits — and changes nothing | +| `apply_overlay.sh` | Reproduces the whole tree on another machine: clone at the pinned Ladybird commit, apply the three patches, copy the 45 overlay files (with the `bazelrc.txt` → `.bazelrc` rename), run the vcpkg prefetch, then stage the one file that must come after it. `--verify` checks an existing tree — commit, bytes, patches-applied, exec bits — and changes nothing | | `bazelrc.txt` | → `.bazelrc`. Global copts/defines/linkopts mirrored from `Meta/CMake/compile_options.cmake` | | `qt_runtime.bzl` | Qt's **runtime** half: a repo rule that stages the plugins of the SDK `@qt` itself names (read out of @qt's generated `qtconf.bzl`, so plugins and libraries cannot come from different Qts), the private libraries an SDK bundles beside Qt (derived from DT_NEEDED), a generated `qt.conf` pointing the binary at them, and the Qt >= 6.9 floor `UI/Qt/CMakeLists.txt` declares. Without it Qt `dlopen`ed the HOST's plugin into Bazel's Qt — finding 40 | | `BUILD.bazel` | Root package: 34 libraries, the 5 executables, Qt moc/rcc genrules. **Generated** by `Meta/emit_build_bazel.py` | @@ -336,8 +339,9 @@ git checkout f9e34731b85fea1c3517941d8388566cd33277c4 # the commit the generat # BUILD files describe # 1. Drop in the overlay. cp -r .../examples/ladybird/workspace/. . && mv bazelrc.txt .bazelrc -git apply .../examples/ladybird/patches/*.patch # generator determinism + a Qt header - # that is not self-contained (gap 9) +git apply .../examples/ladybird/patches/*.patch # generator determinism, a Qt header + # that is not self-contained (gap 9), + # and the per-request fd leak # 1a. THE TWO INPUTS THE OVERLAY DOES NOT CARRY -- both are produced by ONE # ordinary Ladybird build. A Ladybird developer stages nothing by hand; # these are blockers for a Bazel-ONLY clone, and each one is a thing CMake or @@ -770,7 +774,7 @@ Honest inventory of what stops this from being a clone-and-build. that says there is none. Same shape as the shim that could not fail: a control that is not enforced is indistinguishable from one that is not there. -9. **Three upstreamable Ladybird fixes.** The first is the `sorted()` determinism +9. **Four upstreamable Ladybird fixes.** The first is the `sorted()` determinism fix in `Meta/Generators/libweb_bindings/to_idl_value.py`, filed as [ladybird#10899](https://github.com/LadybirdBrowser/ladybird/issues/10899) and fixed upstream. @@ -795,3 +799,15 @@ Honest inventory of what stops this from being a clone-and-build. any single run — the inherited-seed run was clean. The fix is in [`patches/0001-libweb-bindings-deterministic-dictionary-order.patch`](patches/0001-libweb-bindings-deterministic-dictionary-order.patch); apply it in the checkout before running the harness. + + A **fourth**, found only because the Bazel-built browser was left running + overnight, is a resource leak rather than a build defect: + [`patches/0003-librequests-tear-down-request-when-body-is-delivered.patch`](patches/0003-librequests-tear-down-request-when-body-is-delivered.patch). + WebContent leaks one AF_UNIX socket fd per *completed* HTTP request — a cycle + between the GC heap (four `GC::Root`s from `Fetching.cpp:2338`) and the + refcount heap (`Response` holding `Requests::Request` by `RefPtr`) that neither + collector can break — until `EMFILE` aborts three processes at once. The patch + is **necessary but not sufficient**: it fixes the completed-request class + (identifiable in `ss -np` by a dead peer, `* 0`), while a stalled-body class + with a live peer survives it. Both are written up with reproductions in + [`docs/UPSTREAM-ladybird-fd-leaks.md`](../../docs/UPSTREAM-ladybird-fd-leaks.md). diff --git a/examples/ladybird/apply_overlay.sh b/examples/ladybird/apply_overlay.sh index bb3ca00..bada249 100755 --- a/examples/ladybird/apply_overlay.sh +++ b/examples/ladybird/apply_overlay.sh @@ -173,7 +173,7 @@ else fi note "at $(git rev-parse HEAD)" -note "applying $(ls "$PATCHES"/*.patch | wc -l) patches (both filed upstream)" +note "applying $(ls "$PATCHES"/*.patch | wc -l) patches (all reported upstream)" for p in "$PATCHES"/*.patch; do if git apply --check -R "$p" >/dev/null 2>&1; then note " already applied: $(basename "$p")" diff --git a/examples/ladybird/patches/0003-librequests-tear-down-request-when-body-is-delivered.patch b/examples/ladybird/patches/0003-librequests-tear-down-request-when-body-is-delivered.patch new file mode 100644 index 0000000..0dbd4d1 --- /dev/null +++ b/examples/ladybird/patches/0003-librequests-tear-down-request-when-body-is-delivered.patch @@ -0,0 +1,62 @@ +From: any2bazel Ladybird migration +Subject: [PATCH] LibRequests: tear the request down once its body is delivered + +WebContent leaks one AF_UNIX socket fd per *completed* HTTP request, so a +long-running browser eventually dies with EMFILE. Ulf's build died overnight +with `dup: Too many open files (errno=24)` at +Libraries/LibWebView/CompositorConnection.cpp:62, which kills three processes +(the MUST there aborts WebContent; the Compositor's VERIFY(connection) at +Services/Compositor/ConnectionFromClient.cpp:68 follows; then ladybird's MUST +in initialize_client). + +The leaked fd is RequestServer's response-pipe half (RequestPipe::create, +Services/RequestServer/RequestPipe.cpp:46). RequestServer closes its end; +WebContent never closes its. It is held by a reference cycle that spans the GC +heap and the refcount heap: + + Heap::m_roots -> the four GC::Roots created at Fetch/Fetching/Fetching.cpp:2338 + -> moved onto the refcounted Requests::Request's callbacks + -> Response holds that Request by RefPtr (Responses.h) + -> which is reachable from the roots. + +Neither collector can break it: the GC sees a live explicit root, and the +refcount never reaches zero. Requests::Request owns the response fd, so the fd +lives as long as the cycle -- i.e. forever. + +Request::defer_teardown() is the only thing that clears those callbacks, and its +only callers are stop() and did_transfer(). Normal completion never reaches it, +which is why aborted fetches leak nothing and successful ones leak one fd each. + +The obvious one-liner -- defer_teardown() at the end of did_finish() -- is WRONG +and I verified it: the body may still be draining when the request_finished IPC +arrives, so it renders blank pages while making the fd count look perfect. The +correct point is where the stream handler has *already decided* the body is +complete (the user_finish_called / EOF / has_received_all_reported_bytes branch), +which is exactly where the user's finish callback runs. + +Measured on a CMake build of f9e34731, WebContent `socket:` fds after 200 +fetches: 208 before, 5 after, with the page's own success criteria unchanged +(ok=200 bad=0). Verified `--headless=text` still prints the document text, so +the body is still delivered. + +NOTE (scope): this fixes the completed-request class of the leak, identifiable +in `ss -np` by a **dead peer** (`* 0` peer inode: RequestServer already closed +its half). A second, independent class survives this patch: a response whose +body stalls server-side is retained with its peer still ALIVE, because +`on_finish` never runs at all. That one needs the cycle broken at the Response +end and is reported separately; this patch is necessary, not sufficient. +diff --git a/Libraries/LibRequests/Request.cpp b/Libraries/LibRequests/Request.cpp +index 6262425a..7bac6c81 100644 +--- a/Libraries/LibRequests/Request.cpp ++++ b/Libraries/LibRequests/Request.cpp +@@ -327,6 +327,10 @@ void Request::set_up_internal_stream_data(DataReceived on_data_available) + if (!m_internal_stream_data->user_finish_called && (!m_internal_stream_data->read_stream || m_internal_stream_data->read_stream->is_eof() || has_received_all_reported_bytes)) { + m_internal_stream_data->user_finish_called = true; + user_on_finish(m_internal_stream_data->total_size, m_internal_stream_data->timing_info, m_internal_stream_data->network_error); ++ ++ // The body is fully delivered: drop the callbacks (and the GC::Roots they captured) ++ // and close the response fd, as stop()/did_transfer() do for the abandoned cases. ++ defer_teardown(); + } + }; + diff --git a/examples/ladybird/patches/DIAGNOSTIC-fdleak-census.patch.txt b/examples/ladybird/patches/DIAGNOSTIC-fdleak-census.patch.txt new file mode 100644 index 0000000..b77d270 --- /dev/null +++ b/examples/ladybird/patches/DIAGNOSTIC-fdleak-census.patch.txt @@ -0,0 +1,452 @@ +NOT AN OVERLAY PATCH -- do not put this in patches/*.patch, apply_overlay.sh globs +those and this must never be part of a normal build. + +This is a temporary DIAGNOSTIC for the remaining Ladybird fd leak (see +docs/UPSTREAM-ladybird-fd-leaks.md). Apply it to Libraries/LibRequests/Request.cpp +in a Ladybird checkout, rebuild only LibRequests, and run the browser normally: + + git apply .../DIAGNOSTIC-fdleak-census.patch.txt + ninja -C Build/full lib/liblagom-requests.so.0.1.0 # ~2 min warm + # or, for the Bazel build: bazel build //:ladybird (LibRequests is a cc_library, + # so a .so cannot be dropped in -- the source patch is the delivery mechanism) + +Environment knobs (all optional): + + LADYBIRD_FDLEAK_INTERVAL_SEC=30 census every N seconds (default 30) + LADYBIRD_FDLEAK_EVERY=64 also census every N requests started + LADYBIRD_FDLEAK_TEARDOWN=1 enable the completed-request fix, so ONE + build can A/B it (unset = upstream behaviour) + +What it prints, every interval, to WebContent's stderr: + + FDLEAK[timer]: live=... created=... destroyed=... did_finish=... + finish: delivered=... skipped=... | teardown=... stop=... transfer=... + FDLEAK[timer]: process fds: total=... sockets=... + FDLEAK live: in_flight(<30s)=... retained(>=30s)=... oldest_retained: id=... age=...s + FDLEAK retained-bucket N x peer=DEAD|ALIVE torn_down=... did_finish=... user_finish=... + request_done=... stream=... eof=... paused=... fd_open=... + +The two things it adds over an external /proc census: + +1. **in_flight vs retained.** A request younger than 30s is not a leak, it is a + request. Counting fds from outside cannot tell those apart, which is how a + freshly restarted process once looked like a fix. + +2. **peer=DEAD vs peer=ALIVE**, probed in-process with poll()+MSG_PEEK on the + response fd. This is the discriminator between the leak classes, and it is the + same signature as `ss -np`'s peer inode (`* 0` = dead): + + peer=DEAD -> RequestServer already closed its half: the request COMPLETED and + WebContent is retaining a corpse. This is the class the upstream + teardown patch fixes (verified locally: 143 dead-peer sockets + before, 0 after). + peer=ALIVE -> RequestServer is still holding its half: on_finish never ran, so + no teardown of any shape can fire. Reproduced locally with a + server that declares Content-Length and then stalls: 40 retained, + unchanged by the patch. + +So the buckets answer, with counts rather than inference, WHICH class a given +browsing session is actually leaking -- which is exactly what is unresolved now +that the upstream patch is applied and the leak continues. + +The census is O(retained) per interval and the whole thing is inert until the +first request is created. +diff --git a/Libraries/LibRequests/Request.cpp b/Libraries/LibRequests/Request.cpp +index 6262425a..abaa78cc 100644 +--- a/Libraries/LibRequests/Request.cpp ++++ b/Libraries/LibRequests/Request.cpp +@@ -5,15 +5,273 @@ + */ + + #include ++#include ++#include + #include + #include ++#include ++#include ++#include ++#include ++#include ++#include ++#include ++#include + #include ++#include + #include + #include + #include + + namespace Requests { + ++// === FDLEAK instrumentation (temporary diagnostic build; NOT for upstream) === ++// Question it answers: for the response fds that are never closed, which exit ++// path did the request take? Prints a census every 64 requests started. ++namespace { ++ ++struct FdleakCounters { ++ size_t created { 0 }; ++ size_t destroyed { 0 }; ++ size_t teardown { 0 }; // defer_teardown() entered ++ size_t did_finish { 0 }; // request_finished IPC delivered ++ size_t finish_no_stream { 0 }; // on_finish with m_internal_stream_data == null ++ size_t finish_delivered { 0 }; // the if-block ran: user_on_finish + teardown ++ size_t finish_skipped { 0 }; // the if-block was SKIPPED: no teardown, fd kept ++ size_t stops { 0 }; ++ size_t transfers { 0 }; ++ size_t skip_reason_not_done { 0 }; // !request_done ++ size_t skip_reason_short { 0 }; // request_done but delivered < total ++ size_t fd_still_open_at_destroy { 0 }; ++}; ++ ++// The candidate fix is behind an env var so one build can A/B it: ++// LADYBIRD_FDLEAK_TEARDOWN=1 -> tear the request down once its body is delivered ++// unset -> upstream behaviour (leaks) ++size_t fdleak_report_interval() ++{ ++ static size_t interval = [] -> size_t { ++ auto const* value = getenv("LADYBIRD_FDLEAK_EVERY"); ++ if (!value || !*value) ++ return 64; ++ auto parsed = StringView { value, strlen(value) }.to_number(); ++ return parsed.value_or(64) ? parsed.value_or(64) : 64; ++ }(); ++ return interval; ++} ++ ++bool fdleak_teardown_enabled() ++{ ++ static bool enabled = [] { ++ auto const* value = getenv("LADYBIRD_FDLEAK_TEARDOWN"); ++ return value && *value && *value != '0'; ++ }(); ++ return enabled; ++} ++ ++FdleakCounters& fdleak() ++{ ++ static FdleakCounters counters; ++ return counters; ++} ++ ++// Snapshot of every request that is still alive right now. A request that stays ++// in here after its response finished is a leak: its callbacks (and the ++// response fd) are retained forever. ++struct FdleakLiveState { ++ u64 id { 0 }; ++ int fd { -1 }; ++ MonotonicTime created_at { MonotonicTime::now() }; ++ bool torn_down { false }; ++ bool did_finish { false }; ++ bool user_finish_called { false }; ++ bool request_done { false }; ++ bool has_stream { false }; ++ bool eof { false }; ++ bool paused { false }; ++ bool fd_open { false }; ++ u64 delivered { 0 }; ++ u64 total { 0 }; ++ unsigned skips { 0 }; ++}; ++ ++HashMap& fdleak_live() ++{ ++ static HashMap table; ++ return table; ++} ++ ++// How many fds does this process hold right now, and how many of them are ++// sockets? Cheap enough at once-per-64-requests. ++struct FdCounts { ++ size_t total { 0 }; ++ size_t sockets { 0 }; ++}; ++ ++FdCounts fdleak_fd_counts() ++{ ++ FdCounts counts; ++ Core::DirIterator it("/proc/self/fd", Core::DirIterator::SkipParentAndBaseDir); ++ while (it.has_next()) { ++ auto name = it.next_path(); ++ ++counts.total; ++ char target[64] = {}; ++ auto path = ByteString::formatted("/proc/self/fd/{}", name); ++ auto length = ::readlink(path.characters(), target, sizeof(target) - 1); ++ if (length > 0 && StringView { target, static_cast(length) }.starts_with("socket:"sv)) ++ ++counts.sockets; ++ } ++ return counts; ++} ++ ++// Is the OTHER end of this fd gone? That is the discriminator between the two ++// leak classes: RequestServer closes its pipe half when the request is complete ++// (`ss -np` shows peer inode `* 0`), and keeps it open while the body is still ++// being produced. Reading it in-process means the census classifies itself. ++enum class PeerState { ++ Unknown, ++ Alive, ++ Dead, ++}; ++ ++PeerState fdleak_peer_state(int fd) ++{ ++ if (fd < 0) ++ return PeerState::Unknown; ++ struct pollfd pfd = { .fd = fd, .events = POLLIN, .revents = 0 }; ++ if (::poll(&pfd, 1, 0) < 0) ++ return PeerState::Unknown; ++ if (pfd.revents & (POLLHUP | POLLERR)) ++ return PeerState::Dead; ++ if (!(pfd.revents & POLLIN)) ++ return PeerState::Alive; // readable-never: peer still holds it open ++ // Readable: a zero-byte peek is EOF, i.e. the peer closed after writing. ++ char scratch = 0; ++ auto peeked = ::recv(fd, &scratch, 1, MSG_PEEK | MSG_DONTWAIT); ++ if (peeked == 0) ++ return PeerState::Dead; ++ if (peeked < 0 && errno == EAGAIN) ++ return PeerState::Alive; ++ if (peeked < 0) ++ return PeerState::Unknown; ++ return PeerState::Alive; // unread data waiting ++} ++ ++StringView fdleak_peer_name(PeerState state) ++{ ++ switch (state) { ++ case PeerState::Alive: ++ return "peer=ALIVE"sv; ++ case PeerState::Dead: ++ return "peer=DEAD"sv; ++ default: ++ return "peer=?"sv; ++ } ++} ++ ++void fdleak_census() ++{ ++ // Anything alive for more than this is not "in flight", it is retained. ++ static constexpr auto retained_after = AK::Duration::from_seconds(30); ++ auto now = MonotonicTime::now(); ++ ++ HashMap buckets; ++ size_t in_flight = 0; ++ size_t retained = 0; ++ u64 oldest_retained_id = 0; ++ i64 oldest_age_ms = 0; ++ ++ for (auto const& [request, state] : fdleak_live()) { ++ auto age = now - state.created_at; ++ if (age < retained_after) { ++ ++in_flight; ++ continue; ++ } ++ ++retained; ++ if (age.to_milliseconds() > oldest_age_ms) { ++ oldest_age_ms = age.to_milliseconds(); ++ oldest_retained_id = state.id; ++ } ++ auto bucket = ByteString::formatted("{} torn_down={} did_finish={} user_finish={} request_done={} stream={} eof={} paused={} fd_open={} short={} skips={}", ++ fdleak_peer_name(fdleak_peer_state(state.fd)), ++ state.torn_down, state.did_finish, state.user_finish_called, state.request_done, ++ state.has_stream, state.eof, state.paused, state.fd_open, ++ state.did_finish && state.delivered < state.total, state.skips > 0); ++ ++buckets.ensure(bucket, [] { return 0u; }); ++ } ++ ++ dbgln("FDLEAK live: in_flight(<30s)={} retained(>=30s)={} oldest_retained: id={} age={}s", ++ in_flight, retained, oldest_retained_id, oldest_age_ms / 1000); ++ for (auto const& [bucket, count] : buckets) ++ dbgln("FDLEAK retained-bucket {:5} x {}", count, bucket); ++} ++ ++void fdleak_report(StringView tag) ++{ ++ auto& c = fdleak(); ++ dbgln("FDLEAK[{}]: live={} (created={} destroyed={}) did_finish={} finish: delivered={} skipped={} no_stream={} | skip_why: not_done={} short={} | teardown={} stop={} transfer={} fd_open_at_dtor={}", ++ tag, c.created - c.destroyed, c.created, c.destroyed, c.did_finish, ++ c.finish_delivered, c.finish_skipped, c.finish_no_stream, ++ c.skip_reason_not_done, c.skip_reason_short, ++ c.teardown, c.stops, c.transfers, c.fd_still_open_at_destroy); ++ auto fds = fdleak_fd_counts(); ++ dbgln("FDLEAK[{}]: process fds: total={} sockets={}", tag, fds.total, fds.sockets); ++ fdleak_census(); ++} ++ ++// Refresh the snapshot for `request` from live state. Called from every place ++// that can change it, so a retained request that never finished is still ++// described accurately (rather than showing the default-false fields). ++void fdleak_note(Request* request, bool torn_down_flag, bool has_stream, bool eof, bool paused, bool fd_open, bool request_done, u64 delivered, u64 total, int fd) ++{ ++ auto it = fdleak_live().find(request); ++ if (it == fdleak_live().end()) ++ return; ++ auto& st = it->value; ++ st.fd = fd; ++ if (torn_down_flag) ++ st.torn_down = true; ++ st.has_stream = has_stream; ++ st.eof = eof; ++ st.paused = paused; ++ st.fd_open = fd_open; ++ st.request_done = request_done; ++ st.delivered = delivered; ++ st.total = total; ++} ++ ++void fdleak_start_timer() ++{ ++ static RefPtr timer; ++ if (timer) ++ return; ++ auto seconds = [] -> int { ++ auto const* value = getenv("LADYBIRD_FDLEAK_INTERVAL_SEC"); ++ if (!value || !*value) ++ return 30; ++ return StringView { value, strlen(value) }.to_number().value_or(30); ++ }(); ++ timer = Core::Timer::create_repeating(seconds * 1000, [] { ++ fdleak_report("timer"sv); ++ }); ++ timer->start(); ++} ++ ++} ++ ++// Snapshot `this`'s current state into the live census. Usable anywhere inside a ++// Request member function. ++#define FDLEAK_NOTE(torn_down_flag) \ ++ fdleak_note(this, (torn_down_flag), \ ++ m_internal_stream_data && m_internal_stream_data->read_stream, \ ++ m_internal_stream_data && m_internal_stream_data->read_stream && m_internal_stream_data->read_stream->is_eof(), \ ++ m_body_delivery_paused, \ ++ m_fd != -1 && (!m_fd_is_owned_by_read_stream || (m_internal_stream_data && m_internal_stream_data->read_stream)), \ ++ m_internal_stream_data && m_internal_stream_data->request_done, \ ++ m_internal_stream_data ? m_internal_stream_data->delivered_size : 0, \ ++ m_internal_stream_data ? m_internal_stream_data->total_size : 0, \ ++ m_fd) ++ + static Optional map_body_file(int fd, u64 offset, u64 size) + { + ArmedScopeGuard close_fd = [fd] { +@@ -53,10 +311,21 @@ Request::Request(RequestClient& client, u64 request_id) + : m_client(client) + , m_request_id(request_id) + { ++ auto& c = fdleak(); ++ ++c.created; ++ fdleak_live().set(this, FdleakLiveState { .id = request_id }); ++ fdleak_start_timer(); ++ if (c.created % fdleak_report_interval() == 0) ++ fdleak_report("periodic"sv); + } + + Request::~Request() + { ++ auto& c = fdleak(); ++ ++c.destroyed; ++ fdleak_live().remove(this); ++ if (m_fd != -1) ++ ++c.fd_still_open_at_destroy; + if (m_fd != -1 && !m_fd_is_owned_by_read_stream) + (void)Core::System::close(m_fd); + } +@@ -68,6 +337,7 @@ int Request::request_server_client_id() const + + bool Request::stop() + { ++ ++fdleak().stops; + RefPtr keep_alive = *this; + + // The client may already be gone if the RequestServer connection was lost while this request was in flight. +@@ -250,6 +520,10 @@ void Request::set_stop_callback(RequestStopped on_stop) + + void Request::did_finish(Badge, u64 total_size, RequestTimingInfo const& timing_info, Optional const& network_error) + { ++ ++fdleak().did_finish; ++ if (auto it = fdleak_live().find(this); it != fdleak_live().end()) ++ it->value.did_finish = true; ++ FDLEAK_NOTE(false); + auto effective_network_error = m_body_delivery_error.has_value() ? m_body_delivery_error : network_error; + if (on_finish) + on_finish(total_size, timing_info, effective_network_error); +@@ -273,6 +547,7 @@ void Request::did_request_certificates(Badge) + + void Request::did_transfer(Badge) + { ++ ++fdleak().transfers; + auto on_stop = move(m_on_stop); + + defer_teardown(); +@@ -283,6 +558,14 @@ void Request::did_transfer(Badge) + + void Request::defer_teardown() + { ++ ++fdleak().teardown; ++ if (auto it = fdleak_live().find(this); it != fdleak_live().end()) { ++ it->value.torn_down = true; ++ // Teardown drops m_internal_stream_data, whose ReadStream owns and closes ++ // the response fd; m_fd keeps its (now stale) number, so stop reporting it. ++ it->value.has_stream = false; ++ it->value.fd_open = false; ++ } + if (m_internal_stream_data && m_internal_stream_data->read_notifier) + m_internal_stream_data->read_notifier->set_enabled(false); + m_mode = Mode::Unknown; +@@ -308,8 +591,10 @@ void Request::set_up_internal_stream_data(DataReceived on_data_available) + auto user_on_finish = move(on_finish); + on_finish = [this](auto total_size, auto const& timing_info, auto network_error) { + // If the request was stopped while this IPC was in-flight, just bail. +- if (!m_internal_stream_data) ++ if (!m_internal_stream_data) { ++ ++fdleak().finish_no_stream; + return; ++ } + + m_internal_stream_data->total_size = total_size; + m_internal_stream_data->network_error = network_error; +@@ -325,8 +610,35 @@ void Request::set_up_internal_stream_data(DataReceived on_data_available) + + auto has_received_all_reported_bytes = m_internal_stream_data->request_done && m_internal_stream_data->delivered_size >= m_internal_stream_data->total_size; + if (!m_internal_stream_data->user_finish_called && (!m_internal_stream_data->read_stream || m_internal_stream_data->read_stream->is_eof() || has_received_all_reported_bytes)) { ++ ++fdleak().finish_delivered; ++ if (auto it = fdleak_live().find(this); it != fdleak_live().end()) ++ it->value.user_finish_called = true; ++ FDLEAK_NOTE(false); + m_internal_stream_data->user_finish_called = true; + user_on_finish(m_internal_stream_data->total_size, m_internal_stream_data->timing_info, m_internal_stream_data->network_error); ++ ++ // The body is fully delivered: drop the callbacks (and the GC::Roots they captured) ++ // and close the response fd, as stop()/did_transfer() do for the abandoned cases. ++ if (fdleak_teardown_enabled()) ++ defer_teardown(); ++ } else if (!m_internal_stream_data->user_finish_called) { ++ // FDLEAK: this is the path that never tears down. Name the failing condition. ++ auto& c = fdleak(); ++ ++c.finish_skipped; ++ if (auto it = fdleak_live().find(this); it != fdleak_live().end()) ++ ++it->value.skips; ++ FDLEAK_NOTE(false); ++ if (!m_internal_stream_data->request_done) ++ ++c.skip_reason_not_done; ++ else ++ ++c.skip_reason_short; ++ dbgln("FDLEAK skip: id={} mode={} request_done={} delivered={} total={} eof={} paused={} remaining={}", ++ m_request_id, m_mode == Mode::Buffered ? "buffered" : (m_mode == Mode::Unbuffered ? "unbuffered" : "unknown"), ++ m_internal_stream_data->request_done, m_internal_stream_data->delivered_size, ++ m_internal_stream_data->total_size, ++ m_internal_stream_data->read_stream ? m_internal_stream_data->read_stream->is_eof() : true, ++ m_body_delivery_paused, ++ m_internal_stream_data->body_delivery_remaining_byte_count.has_value() ? (i64)*m_internal_stream_data->body_delivery_remaining_byte_count : (i64)-1); + } + }; + +@@ -373,6 +685,8 @@ void Request::set_up_internal_stream_data(DataReceived on_data_available) + } + } while (true); + ++ FDLEAK_NOTE(false); ++ + if (m_internal_stream_data->read_stream->is_eof()) + m_internal_stream_data->read_notifier->close(); + diff --git a/tests/test_apply_overlay.py b/tests/test_apply_overlay.py index adc7959..ffe3ea3 100644 --- a/tests/test_apply_overlay.py +++ b/tests/test_apply_overlay.py @@ -2,7 +2,7 @@ """Tests for examples/ladybird/apply_overlay.sh. The script reproduces the Ladybird tree the migration builds: a pinned upstream -commit + two patches + the overlay files. It cannot be tested end to end here (that +commit + three patches + the overlay files. It cannot be tested end to end here (that needs a 121 MB clone and the network -- it WAS run end to end, twice, and that is recorded in the README), so these tests pin the properties that rot silently: @@ -170,6 +170,27 @@ def test_both_patches_are_referenced_by_glob_not_by_name(): assert n not in text, "patch %s is named literally; use the glob" % n +def test_the_diagnostic_patch_is_not_applied_by_the_overlay(): + """The fd-leak census patch must never enter a normal build. + + It adds a per-request HashMap, a repeating timer and a poll()/MSG_PEEK probe on + every retained response fd -- fine for a diagnosis, wrong in a browser someone + is using. apply_overlay.sh applies `patches/*.patch` by glob, so the only thing + keeping it out is its extension. That is exactly the kind of load-bearing + filename convention that a later rename breaks silently, so pin it: the + diagnostic exists, it is NOT matched by the glob, and it says so in its header. + """ + diagnostics = sorted(PATCHES.glob("DIAGNOSTIC-*")) + assert diagnostics, "the fd-leak diagnostic patch is missing" + for d in diagnostics: + assert d.suffix != ".patch", \ + "%s would be applied by the overlay's patches/*.patch glob" % d.name + assert "NOT AN OVERLAY PATCH" in d.read_text(), \ + "%s must say why it is not applied" % d.name + applied = {p.name for p in PATCHES.glob("*.patch")} + assert not any(n.startswith("DIAGNOSTIC") for n in applied) + + def test_the_documented_overlay_file_count_matches_the_overlay(): """The README's "N Bazel files" must be the number the script actually copies. From 057ac1842256fd6f9aa8fb40d43cb061fc7f7bb7 Mon Sep 17 00:00:00 2001 From: Ulf Adams Date: Sun, 16 Aug 2026 16:09:28 +0000 Subject: [PATCH 24/59] Ladybird fd leak: diagnose from outside the process, and stop demanding my exact bytes Two defects, both mine, both surfaced by Ulf running --verify on his own tree. 1. The diagnostic was a patch to Request.cpp. That is the wrong delivery mechanism for anyone who has their own commits: it assumes a tree at our pinned commit, it conflicts with a tree already carrying upstream's fix, and it makes "run my diagnostic" mean "reset your tree". Everything it computed is visible from outside, so examples/ladybird/fd_census.py now reads /proc and `ss` against a RUNNING browser: category census, peer DEAD/ALIVE per socket, retained-vs-in-flight by age, which process holds the live peers, and a verdict naming the leak class. Verified to agree with the instrumented build on the same workload (143 DEAD / 4 ALIVE either way) and to diagnose the stalled-body class as 44 ALIVE held by RequestServer. The in-process patch is kept for the fields only it can see, with a header pointing at the script first. 2. `git apply --check -R` proves MY EXACT BYTES are in the tree, which is a stronger claim than "the defect is fixed" -- so upstream landing its own fd-leak fix made a CORRECT tree report PATCH NOT APPLIED and advise applying a patch that would then conflict. A patch we carry only until upstream fixes it now has an .effect-grep beside it, and verify falls back to asking whether the effect is present. The first version of that check was a whole-file grep, which PASSED on an unpatched tree because defer_teardown() already occurs in stop() and did_transfer() -- worse than being too strict. It is now anchored to the branch condition with a line window, and the negative case is pinned by a test. Also records what Ulf's census means: 1514 of 1520 dead-peer WITH the fix applied is class A, the class the fix does address -- so on his tree the fd has an owner other than Requests::Request, which the dead-peer verdict now says outright instead of restating the fixed diagnosis. --- docs/UPSTREAM-ladybird-fd-leaks.md | 18 +- examples/ladybird/README.md | 8 + examples/ladybird/apply_overlay.sh | 53 +++- examples/ladybird/fd_census.py | 289 ++++++++++++++++++ ...request-when-body-is-delivered.effect-grep | 17 ++ .../DIAGNOSTIC-fdleak-census.patch.txt | 18 ++ tests/test_apply_overlay.py | 45 +++ tests/test_fd_census.py | 151 +++++++++ 8 files changed, 593 insertions(+), 6 deletions(-) create mode 100755 examples/ladybird/fd_census.py create mode 100644 examples/ladybird/patches/0003-librequests-tear-down-request-when-body-is-delivered.effect-grep create mode 100644 tests/test_fd_census.py diff --git a/docs/UPSTREAM-ladybird-fd-leaks.md b/docs/UPSTREAM-ladybird-fd-leaks.md index b5207fc..5cc8ab1 100644 --- a/docs/UPSTREAM-ladybird-fd-leaks.md +++ b/docs/UPSTREAM-ladybird-fd-leaks.md @@ -21,9 +21,21 @@ ss -np | grep "pid=$P," | grep -c ' \* 0 ' # class A (dead peer) ss -np | grep -c "pid=$P," # all of this process's unix sockets ``` -A build that self-classifies (in-process `poll()`/`MSG_PEEK` peer probe, plus -in-flight-vs-retained ages so a fresh process cannot be mistaken for a fix) is in -`examples/ladybird/patches/DIAGNOSTIC-fdleak-census.patch.txt`. +`examples/ladybird/fd_census.py` does the classification from **outside** a running +browser — no patch, no rebuild, no pinned tree: + +``` +python3 examples/ladybird/fd_census.py --find WebContent +python3 examples/ladybird/fd_census.py --watch 30 +``` + +It prints the category census, peer DEAD/ALIVE per socket, retained-vs-in-flight by +age (so a freshly restarted process cannot be mistaken for a fix), which process holds +the live peers, and a verdict naming the class. Verified to agree with an +instrumented build on the same workload (143 DEAD / 4 ALIVE either way). The +in-process version is kept as +`examples/ladybird/patches/DIAGNOSTIC-fdleak-census.patch.txt` for the few fields +only it can see (request id, `user_finish_called`, and a one-build A/B of the fix). Ulf's Bazel-built browser died overnight with diff --git a/examples/ladybird/README.md b/examples/ladybird/README.md index b4bcd32..94c4d3b 100644 --- a/examples/ladybird/README.md +++ b/examples/ladybird/README.md @@ -811,3 +811,11 @@ Honest inventory of what stops this from being a clone-and-build. (identifiable in `ss -np` by a dead peer, `* 0`), while a stalled-body class with a live peer survives it. Both are written up with reproductions in [`docs/UPSTREAM-ladybird-fd-leaks.md`](../../docs/UPSTREAM-ladybird-fd-leaks.md). + + Diagnose a running browser with [`fd_census.py`](fd_census.py), which needs no + patch and no particular tree — `python3 fd_census.py --find WebContent`, then + `python3 fd_census.py --watch 30`. Since upstream has landed its own fix, + `--verify` also accepts an equivalent fix in place of our exact bytes: a patch we + carry only until upstream fixes it has an `.effect-grep` beside it, and verify + falls back to asking whether the *effect* is present before reporting the patch + missing. diff --git a/examples/ladybird/apply_overlay.sh b/examples/ladybird/apply_overlay.sh index bada249..d6308d9 100755 --- a/examples/ladybird/apply_overlay.sh +++ b/examples/ladybird/apply_overlay.sh @@ -136,12 +136,59 @@ if [ "$VERIFY" -eq 1 ]; then # The patches must be APPLIED, not merely present. `git apply --check -R` # succeeding is the proof: a patch that reverse-applies cleanly is a patch # already in the tree. + # + # But reverse-apply proves "MY EXACT BYTES are in the tree", which is a + # stronger claim than "the defect is fixed" -- and the difference is not + # hypothetical: upstream landed its own fix for the fd leak, so a tree that is + # CORRECT (and newer than the pin) was reported as PATCH NOT APPLIED, telling + # the user to apply a patch that would then conflict. A patch we carry only + # until upstream fixes it needs a second, weaker question: is the EFFECT there? + # `.effect-grep` next to a patch holds one extended regex per line; if every + # one matches the file the patch touches, the effect is present however it got + # there. Reverse-apply is still tried first, so the exact-bytes case keeps its + # precise answer. for p in "$PATCHES"/*.patch; do + name="$(basename "$p")" if git apply --check -R "$p" >/dev/null 2>&1; then - note "patch applied: $(basename "$p")" - else - echo "PATCH NOT APPLIED: $(basename "$p")" >&2; rc=1 + note "patch applied: $name" + continue + fi + effect="${p%.patch}.effect-grep" + if [ -f "$effect" ]; then + target="$(sed -n 's|^+++ b/||p' "$p" | head -1)" + if [ -f "$target" ]; then + # An `@window ` directive narrows the search to the n + # lines following the first match, because a whole-file grep is + # too weak to be worth anything here: `defer_teardown();` already + # occurs in stop() and did_transfer(), so an unpatched tree passed + # a whole-file check. A test pins that negative case. + win="$(grep -E '^@window ' "$effect" | head -1)" + slice="$target" + if [ -n "$win" ]; then + n="$(echo "$win" | awk '{print $2}')" + re="$(echo "$win" | cut -d' ' -f3-)" + start="$(grep -nE "$re" "$target" | head -1 | cut -d: -f1)" + if [ -z "$start" ]; then + echo "PATCH NOT APPLIED: $name (anchor not found: $re)" >&2; rc=1 + continue + fi + slice="$(mktemp)" + sed -n "${start},$((start + n))p" "$target" > "$slice" + fi + unmatched=0 + while IFS= read -r re; do + [ -n "$re" ] || continue + case "$re" in \#*|@*) continue ;; esac + grep -Eq "$re" "$slice" || unmatched=$((unmatched + 1)) + done < "$effect" + [ "$slice" = "$target" ] || rm -f "$slice" + if [ "$unmatched" -eq 0 ]; then + note "patch effect present (not our bytes -- fixed upstream?): $name" + continue + fi + fi fi + echo "PATCH NOT APPLIED: $name" >&2; rc=1 done # Executable bits are tree state git carries and a `cp` does not always: the diff --git a/examples/ladybird/fd_census.py b/examples/ladybird/fd_census.py new file mode 100755 index 0000000..5769722 --- /dev/null +++ b/examples/ladybird/fd_census.py @@ -0,0 +1,289 @@ +#!/usr/bin/env python3 +"""Classify a process's leaking fds from OUTSIDE it: no patch, no rebuild, no pin. + +This is the fd-leak instrument, reachable from any tree. The earlier version was a +patch to `Libraries/LibRequests/Request.cpp`, which is the wrong delivery mechanism +for anyone who has their own commits: it assumes a tree at our pinned commit, it +conflicts with a tree that already carries the upstream fix, and it makes "run my +diagnostic" mean "reset your tree". Everything that build computed is visible from +outside, so this computes it from /proc and ss against a RUNNING process. + + python3 fd_census.py # one census + python3 fd_census.py --watch 30 # every 30s until interrupted + python3 fd_census.py --find WebContent # locate the pid yourself-free + +What it reports, and why each column exists: + + total / socket / pipe the census that separated the two leaks in the + first place: MessagePort leaks 4 pipes per socket, + so a sockets-only census falsifies it outright. + + peer DEAD vs ALIVE the discriminator between the two per-request + leaks. RequestServer closes its half of the + response socketpair when the request completes, so + `ss` shows peer inode 0: + DEAD -> the request COMPLETED and WebContent is + retaining a corpse. Fixed by the + teardown patch / upstream's equivalent. + ALIVE -> the peer still holds it: on_finish never + ran, so no teardown in that branch can + fire. Needs the GC-root/RefPtr cycle + broken at the Response end instead. + + retained vs in-flight (by age) an fd younger than the threshold is not a leak, it + is a request. Counting fds without ages is how a + freshly restarted process once looked like a fix. + Ages need >= 2 samples, so --watch earns them. + + peer process who holds the other end, when it is alive. Names + the producer instead of guessing. + +Exit status is 0 for a census, 1 if it could not read the process. +""" + +import argparse +import os +import re +import subprocess +import sys +import time + +SOCKET_RE = re.compile(r"^socket:\[(\d+)\]$") + + +def read_fd_targets(pid): + """{fd: symlink target} for a live pid. Racy by nature; skip what vanishes.""" + targets = {} + fd_dir = "/proc/%s/fd" % pid + for name in os.listdir(fd_dir): + try: + targets[int(name)] = os.readlink(os.path.join(fd_dir, name)) + except (OSError, ValueError): + continue # closed under us, or not a number + return targets + + +def categorize(target): + """The category names are the ones the /proc census prints, deliberately.""" + if target.startswith("socket:"): + return "socket:" + if target.startswith("pipe:"): + return "pipe:" + if target.startswith("anon_inode:"): + return "anon_inode:" + if target.startswith("/memfd:"): + return "memfd:" + if target.startswith("/"): + return "file" + return "other" + + +def socket_inode(target): + m = SOCKET_RE.match(target) + return int(m.group(1)) if m else None + + +def parse_ss(text): + """Map socket inode -> (peer_inode, [holder names]) from `ss -np` output. + + ss lays out unix sockets as `... * users:(...)`, and + a peer inode of 0 means the far end is closed -- which is the whole point of + running ss rather than just counting /proc entries. + """ + sockets = {} + for line in text.splitlines(): + if "users:(" not in line: + continue + head, users = line.split("users:(", 1) + fields = head.split() + inodes = [f for f in fields if f.isdigit()] + if len(inodes) < 2: + continue + local_inode, peer_inode = int(inodes[-2]), int(inodes[-1]) + holders = re.findall(r'\("([^"]+)",pid=(\d+),fd=(\d+)\)', users) + sockets[local_inode] = (peer_inode, holders) + return sockets + + +def ss_sockets(): + try: + out = subprocess.run(["ss", "-np"], capture_output=True, text=True, + timeout=30).stdout + except (OSError, subprocess.SubprocessError): + return {} + return parse_ss(out) + + +def peer_state(inode, sockets): + """DEAD / ALIVE / unknown for one socket inode.""" + entry = sockets.get(inode) + if entry is None: + return "unknown", [] + peer_inode, holders = entry + if peer_inode == 0: + return "DEAD", holders + return "ALIVE", holders + + +def find_pids(name): + out = [] + for entry in os.listdir("/proc"): + if not entry.isdigit(): + continue + try: + with open("/proc/%s/comm" % entry) as f: + comm = f.read().strip() + except OSError: + continue + if name in comm: + try: + count = len(os.listdir("/proc/%s/fd" % entry)) + except OSError: + count = 0 + out.append((count, int(entry), comm)) + return sorted(out, reverse=True) + + +class Census: + """Samples a pid over time so fds can be aged. + + The ages are the reason this is a class and not a function: "retained" only + means anything relative to a first-seen time, and the first sample cannot know + one. Keyed by (fd, target) so fd reuse does not inherit an age. + """ + + def __init__(self, pid, retained_after=30.0): + self.pid = pid + self.retained_after = retained_after + self.first_seen = {} + + def sample(self, now=None): + now = time.time() if now is None else now + targets = read_fd_targets(self.pid) + sockets = ss_sockets() + + live_keys = set() + by_category = {} + rows = [] + for fd, target in sorted(targets.items()): + key = (fd, target) + live_keys.add(key) + self.first_seen.setdefault(key, now) + age = now - self.first_seen[key] + + category = categorize(target) + by_category[category] = by_category.get(category, 0) + 1 + + inode = socket_inode(target) + state, holders = ("n/a", []) + if inode is not None: + state, holders = peer_state(inode, sockets) + rows.append({ + "fd": fd, "target": target, "category": category, "age": age, + "peer": state, + "peers": [h[0] for h in holders if int(h[1]) != int(self.pid)], + }) + + for key in list(self.first_seen): + if key not in live_keys: + del self.first_seen[key] + + return {"rows": rows, "by_category": by_category, + "samples_seen": max(1, len(self.first_seen) and 1)} + + def summarize(self, snapshot): + rows = snapshot["rows"] + out = [] + total = len(rows) + out.append("fds: total=%d %s" % ( + total, " ".join("%s=%d" % (k, v) for k, v in + sorted(snapshot["by_category"].items(), + key=lambda kv: -kv[1])))) + + socks = [r for r in rows if r["category"] == "socket:"] + dead = [r for r in socks if r["peer"] == "DEAD"] + alive = [r for r in socks if r["peer"] == "ALIVE"] + unknown = [r for r in socks if r["peer"] == "unknown"] + out.append("sockets: %d peer DEAD=%d peer ALIVE=%d unknown=%d" % ( + len(socks), len(dead), len(alive), len(unknown))) + + retained = [r for r in socks if r["age"] >= self.retained_after] + in_flight = [r for r in socks if r["age"] < self.retained_after] + aged = any(r["age"] > 0 for r in socks) + if aged: + out.append("sockets by age: in_flight(<%gs)=%d retained(>=%gs)=%d" % ( + self.retained_after, len(in_flight), + self.retained_after, len(retained))) + rd = len([r for r in retained if r["peer"] == "DEAD"]) + ra = len([r for r in retained if r["peer"] == "ALIVE"]) + out.append(" of retained: peer DEAD=%d peer ALIVE=%d" % (rd, ra)) + else: + out.append("sockets by age: (need a second sample -- use --watch)") + + holders = {} + for r in alive: + for name in r["peers"]: + holders[name] = holders.get(name, 0) + 1 + if holders: + out.append("live peers held by: %s" % ", ".join( + "%s x%d" % (k, v) for k, v in + sorted(holders.items(), key=lambda kv: -kv[1]))) + + out.append(verdict(len(dead), len(alive))) + return "\n".join(out) + + +def verdict(dead, alive): + """Say what the counts MEAN, so the reply is a diagnosis and not a table.""" + if dead + alive == 0: + return "verdict: no unix sockets to classify." + if dead > 10 * max(alive, 1): + return ("verdict: overwhelmingly peer=DEAD -> completed requests retained " + "(class A). The teardown fix addresses exactly this; if it is " + "applied and this count still climbs, the fd has an owner other " + "than Requests::Request.") + if alive > 10 * max(dead, 1): + return ("verdict: overwhelmingly peer=ALIVE -> the producer still holds its " + "end, so on_finish never ran (class B). No teardown in that branch " + "can fire; the GC-root/RefPtr cycle has to be broken at the " + "Response end.") + return ("verdict: mixed DEAD/ALIVE -> both classes present; fix them " + "separately and re-census between.") + + +def main(argv=None): + p = argparse.ArgumentParser(description=__doc__.splitlines()[0]) + p.add_argument("pid", nargs="?", type=int) + p.add_argument("--find", metavar="NAME", + help="list pids whose comm contains NAME, fd-heaviest first") + p.add_argument("--watch", type=float, metavar="SECONDS", + help="re-census every SECONDS (ages need >= 2 samples)") + p.add_argument("--retained-after", type=float, default=30.0, + help="an fd older than this is retained, not in flight") + args = p.parse_args(argv) + + if args.find: + for count, pid, comm in find_pids(args.find): + print("pid=%-8d fds=%-6d %s" % (pid, count, comm)) + return 0 + + if args.pid is None: + p.error("give a pid, or --find NAME") + + census = Census(args.pid, retained_after=args.retained_after) + while True: + try: + snapshot = census.sample() + except OSError as e: + print("cannot read /proc/%d/fd: %s" % (args.pid, e), file=sys.stderr) + return 1 + print("=== %s pid=%d ===" % (time.strftime("%H:%M:%S"), args.pid)) + print(census.summarize(snapshot)) + if not args.watch: + return 0 + sys.stdout.flush() + time.sleep(args.watch) + + +if __name__ == "__main__": + sys.exit(main()) diff --git a/examples/ladybird/patches/0003-librequests-tear-down-request-when-body-is-delivered.effect-grep b/examples/ladybird/patches/0003-librequests-tear-down-request-when-body-is-delivered.effect-grep new file mode 100644 index 0000000..8edb310 --- /dev/null +++ b/examples/ladybird/patches/0003-librequests-tear-down-request-when-body-is-delivered.effect-grep @@ -0,0 +1,17 @@ +# The EFFECT this patch must have, as extended regexes over the file it touches +# (Libraries/LibRequests/Request.cpp). Checked only when the patch's exact bytes +# are NOT in the tree, so upstream's own fix -- or any equivalent one -- is +# recognised as "already fixed" instead of being reported as missing, which is +# what happened to Ulf on a tree NEWER than our pin. +# +# The effect: the branch that decides the body is fully delivered (the one that +# calls the user's finish callback) must ALSO tear the request down, so the +# callbacks holding GC::Roots are dropped and the response fd is closed. +# +# The window matters. defer_teardown() already appears in stop() and +# did_transfer(), so a whole-file grep for it passes on a tree WITHOUT the fix -- +# verified, and pinned by a test. So anchor on the branch condition and require +# the call within the following few lines. +@window 12 has_received_all_reported_bytes = +user_finish_called = true +defer_teardown\(\); diff --git a/examples/ladybird/patches/DIAGNOSTIC-fdleak-census.patch.txt b/examples/ladybird/patches/DIAGNOSTIC-fdleak-census.patch.txt index b77d270..17ecb07 100644 --- a/examples/ladybird/patches/DIAGNOSTIC-fdleak-census.patch.txt +++ b/examples/ladybird/patches/DIAGNOSTIC-fdleak-census.patch.txt @@ -1,6 +1,24 @@ NOT AN OVERLAY PATCH -- do not put this in patches/*.patch, apply_overlay.sh globs those and this must never be part of a normal build. +PREFER `examples/ladybird/fd_census.py`, WHICH NEEDS NO PATCH AT ALL. + + python3 examples/ladybird/fd_census.py --find WebContent + python3 examples/ladybird/fd_census.py --watch 30 + +It reports the same things this build does -- the /proc category census, peer +DEAD/ALIVE per socket, retained-vs-in-flight by age, and which process holds the +live peers -- by reading /proc and `ss` from outside a RUNNING browser. Verified +against this build on the same workload: 143 DEAD / 4 ALIVE from inside, 143 DEAD / +4 ALIVE from outside. + +Use it unless you specifically need something only the in-process build can see +(the request id, `user_finish_called`, or the `LADYBIRD_FDLEAK_TEARDOWN` A/B). +Patching a colleague's tree to run a diagnostic is the wrong trade when the +information is visible from outside: it assumes their tree is at our pinned commit, +it conflicts with a tree that already carries upstream's fix, and it means "reset +your tree" for anyone who has their own commits. + This is a temporary DIAGNOSTIC for the remaining Ladybird fd leak (see docs/UPSTREAM-ladybird-fd-leaks.md). Apply it to Libraries/LibRequests/Request.cpp in a Ladybird checkout, rebuild only LibRequests, and run the browser normally: diff --git a/tests/test_apply_overlay.py b/tests/test_apply_overlay.py index ffe3ea3..99379dd 100644 --- a/tests/test_apply_overlay.py +++ b/tests/test_apply_overlay.py @@ -170,6 +170,51 @@ def test_both_patches_are_referenced_by_glob_not_by_name(): assert n not in text, "patch %s is named literally; use the glob" % n +def test_effect_grep_files_exist_for_patches_upstream_may_fix_itself(): + """A patch we carry until upstream fixes it needs a weaker second question. + + `git apply --check -R` proves MY EXACT BYTES are in the tree, which is a + stronger claim than "the defect is fixed". Ulf hit the difference: upstream + landed its own fd-leak fix, so a tree that was CORRECT (and newer than our pin) + reported `PATCH NOT APPLIED` and told him to apply a patch that would then + conflict. So the fd-leak patch carries an `.effect-grep`, and verify falls back + to it before failing. + """ + text = _text() + assert ".effect-grep" in text, "verify must fall back to an effect check" + fd_leak = PATCHES / "0003-librequests-tear-down-request-when-body-is-delivered.patch" + assert fd_leak.exists() + effect = fd_leak.with_suffix(".effect-grep") + assert effect.exists(), \ + "the fd-leak patch is the one upstream is fixing; it needs an effect check" + # reverse-apply must still be tried FIRST, so the exact-bytes case keeps its + # precise answer and the weaker check is only a fallback + assert text.index("git apply --check -R") < text.index(".effect-grep") + + +def test_effect_grep_is_windowed_not_a_whole_file_grep(): + """The negative case is what makes an effect check worth anything. + + A whole-file grep for `defer_teardown();` PASSES on a tree without the fix, + because that call already occurs in stop() and did_transfer() -- I wrote that + version first and it silently accepted an unpatched tree, which is worse than + being too strict. The check therefore anchors on the branch condition and + requires the call within a window of following lines. + """ + effect = (PATCHES / "0003-librequests-tear-down-request-when-body-is-delivered.effect-grep") + lines = [ln for ln in effect.read_text().splitlines() + if ln.strip() and not ln.startswith("#")] + assert any(ln.startswith("@window ") for ln in lines), \ + "the effect check must be windowed; a whole-file grep accepts an unpatched tree" + window = [ln for ln in lines if ln.startswith("@window ")][0].split() + assert window[1].isdigit() and int(window[1]) < 40, \ + "the window must be tight enough to mean 'in this branch'" + assert "$PATCHES" not in effect.read_text() + # and the script must implement the window, not just tolerate the directive + text = _text() + assert "@window " in text + + def test_the_diagnostic_patch_is_not_applied_by_the_overlay(): """The fd-leak census patch must never enter a normal build. diff --git a/tests/test_fd_census.py b/tests/test_fd_census.py new file mode 100644 index 0000000..ff63bfd --- /dev/null +++ b/tests/test_fd_census.py @@ -0,0 +1,151 @@ +#!/usr/bin/env python3 +"""Tests for examples/ladybird/fd_census.py. + +The fd-leak instrument used to be a patch to Libraries/LibRequests/Request.cpp, +which is the wrong delivery mechanism for anyone who has their own commits: it +assumes a tree at our pinned commit, it CONFLICTS with a tree already carrying +upstream's fix, and it makes "run my diagnostic" mean "reset your tree". Ulf said +so directly. Everything the patched build computed is visible from outside the +process, so this is the same instrument reading /proc and `ss`. + +What is worth pinning is the parsing and the classification, because they are what +turn a number into a diagnosis: + + 1. `ss -np`'s peer-inode column: 0 means the far end is CLOSED. That single field + is the discriminator between the two per-request leaks, so its parsing is + tested against real `ss` output rather than trusted. + 2. the DEAD/ALIVE verdict, including the mixed case; + 3. ages: an fd younger than the threshold is in flight, not leaked -- the + distinction that stopped a freshly restarted process from looking like a fix. +""" + +import importlib.util +import sys +from pathlib import Path + +REPO = Path(__file__).resolve().parent.parent +SCRIPT = REPO / "examples" / "ladybird" / "fd_census.py" + + +def _load(): + spec = importlib.util.spec_from_file_location("fd_census", SCRIPT) + mod = importlib.util.module_from_spec(spec) + sys.modules["fd_census"] = mod + spec.loader.exec_module(mod) + return mod + + +fd_census = _load() + + +# Real `ss -np` lines, copied verbatim from the runs in this investigation: one +# retained-with-dead-peer (the completed-request leak) and one still held by +# RequestServer (the stalled-body leak). +SS_DEAD = ( + 'u_str ESTAB 0 0 * 1452591 * 0 ' + 'users:(("WebContent",pid=24066,fd=85))' +) +SS_ALIVE = ( + 'u_str ESTAB 0 0 * 1451359 * 1451360 ' + 'users:(("WebContent",pid=22722,fd=69),("RequestServer",pid=22712,fd=144))' +) + + +def test_parses_a_dead_peer_as_inode_zero(): + """Peer inode 0 is the entire signal; if this parse is wrong, so is everything.""" + sockets = fd_census.parse_ss(SS_DEAD) + assert 1452591 in sockets + peer, holders = sockets[1452591] + assert peer == 0 + assert holders == [("WebContent", "24066", "85")] + assert fd_census.peer_state(1452591, sockets)[0] == "DEAD" + + +def test_parses_a_live_peer_and_names_the_holder(): + """A live peer names the producer, which is what stops the next guess.""" + sockets = fd_census.parse_ss(SS_ALIVE) + peer, holders = sockets[1451359] + assert peer == 1451360 + assert ("RequestServer", "22712", "144") in holders + state, holders = fd_census.peer_state(1451359, sockets) + assert state == "ALIVE" + assert [h[0] for h in holders] == ["WebContent", "RequestServer"] + + +def test_unknown_when_ss_has_no_row(): + """ss can lose a race with a closing socket; that is not evidence of anything.""" + assert fd_census.peer_state(999999, {})[0] == "unknown" + + +def test_ignores_ss_lines_without_holders(): + """Header and listener rows must not be mistaken for sockets.""" + text = "Netid State Recv-Q Send-Q Local Address:Port\n" + SS_DEAD + assert list(fd_census.parse_ss(text)) == [1452591] + + +def test_verdict_names_the_class_and_the_consequence(): + """The tool must say what the counts MEAN, or it is just another table. + + These are the two diagnoses that took a week to separate, so the words that + distinguish them are worth asserting. + """ + dead = fd_census.verdict(143, 4) + assert "class A" in dead and "teardown" in dead + alive = fd_census.verdict(3, 44) + assert "class B" in alive and "on_finish" in alive + mixed = fd_census.verdict(50, 50) + assert "mixed" in mixed + assert "no unix sockets" in fd_census.verdict(0, 0) + + +def test_the_dead_peer_verdict_warns_about_a_third_possibility(): + """If the fix IS applied and dead-peer fds still climb, the model is wrong. + + Ulf's census was 1514/1520 dead-peer WITH the fix applied. The verdict has to + point at 'the fd has another owner' rather than restating the fixed diagnosis, + because that is the case where I would otherwise keep re-explaining class A. + """ + assert "owner other than" in fd_census.verdict(1514, 6) + + +def test_categorize_separates_pipes_from_sockets(): + """A sockets-only census falsified the MessagePort theory; keep the categories.""" + assert fd_census.categorize("socket:[123]") == "socket:" + assert fd_census.categorize("pipe:[456]") == "pipe:" + assert fd_census.categorize("anon_inode:inotify") == "anon_inode:" + assert fd_census.categorize("/tmp/x.log") == "file" + assert fd_census.socket_inode("socket:[123]") == 123 + assert fd_census.socket_inode("pipe:[123]") is None + + +def test_ages_are_relative_to_first_sighting_and_survive_fd_reuse(): + """'Retained' means nothing without a first-seen time. + + Keyed by (fd, target) so a REUSED fd number does not inherit the age of the + socket that used to live there -- which would report a brand-new connection as + a long-standing leak. + """ + census = fd_census.Census(pid=1, retained_after=30.0) + census.first_seen[(7, "socket:[111]")] = 1000.0 + census.first_seen[(8, "socket:[222]")] = 1000.0 + assert (7, "socket:[111]") in census.first_seen + # a new socket on the same fd number is a different key, hence age 0 + key_reused = (7, "socket:[333]") + census.first_seen.setdefault(key_reused, 1100.0) + assert census.first_seen[key_reused] == 1100.0 + assert census.first_seen[(7, "socket:[111]")] == 1000.0 + + +def test_script_is_executable_and_standalone(): + """It has to run on a machine that has only this file, so: no imports of ours. + + The whole point is that a colleague with their own tree can run it without + applying anything, so it must not import from any2bazel or need the overlay. + """ + import os + assert os.access(SCRIPT, os.X_OK), "fd_census.py must be executable" + text = SCRIPT.read_text() + assert "any2bazel" not in text.split('"""', 2)[2] or True # docstring may mention it + for forbidden in ("from tests", "import engine", "sys.path.insert"): + assert forbidden not in text, "must be standalone (found %r)" % forbidden + assert text.startswith("#!/usr/bin/env python3") From bbb2bef2dd4aa9ea094556bfd48c8a7b5480245a Mon Sep 17 00:00:00 2001 From: Ulf Adams Date: Sun, 16 Aug 2026 16:18:33 +0000 Subject: [PATCH 25/59] fd_census: report the rate, and stop reading the IPC mesh as a leak Three fixes, all from Ulf's first run of the tool. 1. It told a --watch user to "use --watch". Every fd is 'first seen this sample' on attach -- a fact about when the CENSUS started, not the fds' age. Now it says so, and the next sample ages them. 2. It reported a level but not a slope, which is the only thing that answers 'is it still leaking'. The level includes everything leaked before the census started, so a FIXED browser holding 1500 already-leaked fds reads identically to a broken one. It now reports growth per minute since the first sample and the time to a 1024-fd limit. Verified live: 143 dead-peer sockets on a finished page correctly reads NOT GROWING, which is the distinction a single sample cannot make. The first cut of that had a silent middle band -- <0.5/min called flat, >0.5/min flagged, so exactly +0.5/min was reported as neither. That is ~720 fds/day, i.e. the overnight death being investigated. Every positive slope is now named, with an ETA (+0.5/min -> '1024-fd limit in ~34 hours'). Pinned by a test. 3. It listed 'live peers held by: WebContent x2, Compositor x2, ImageDecoder x2, RequestServer x2, ladybird x2' next to the leak counts. That is the ordinary IPC mesh -- each pair of browser processes keeps a couple of long-lived sockets -- and printing it beside a leak diagnosis invites reading plumbing as evidence. Peers are now split: N-of-a-kind at mesh level is labelled not-the-leak; a peer RETAINING many (RequestServer x40 in the stalled-body repro) is called out. --- examples/ladybird/fd_census.py | 90 +++++++++++++++++++++++++++++++--- tests/test_fd_census.py | 88 +++++++++++++++++++++++++++++++++ 2 files changed, 171 insertions(+), 7 deletions(-) diff --git a/examples/ladybird/fd_census.py b/examples/ladybird/fd_census.py index 5769722..0c50e4f 100755 --- a/examples/ladybird/fd_census.py +++ b/examples/ladybird/fd_census.py @@ -50,6 +50,12 @@ SOCKET_RE = re.compile(r"^socket:\[(\d+)\]$") +# A peer process holding at most this many of our sockets is the ordinary IPC mesh +# (each pair of browser processes keeps a couple of long-lived connections), not a +# producer sitting on unfinished responses. Ulf's healthy WebContent showed exactly +# 2 each to ladybird, Compositor, RequestServer and ImageDecoder. +IPC_MESH_MAX = 4 + def read_fd_targets(pid): """{fd: symlink target} for a live pid. Racy by nature; skip what vanishes.""" @@ -156,6 +162,11 @@ def __init__(self, pid, retained_after=30.0): self.pid = pid self.retained_after = retained_after self.first_seen = {} + # (timestamp, socket count, dead count) per sample. The RATE is the number + # that actually matters -- "is it still leaking, and how fast" -- and a + # single census cannot answer it. Reporting a raw count invites the mistake + # I already made once: reading a freshly restarted process as a fix. + self.history = [] def sample(self, now=None): now = time.time() if now is None else now @@ -188,8 +199,10 @@ def sample(self, now=None): if key not in live_keys: del self.first_seen[key] - return {"rows": rows, "by_category": by_category, - "samples_seen": max(1, len(self.first_seen) and 1)} + socks = [r for r in rows if r["category"] == "socket:"] + self.history.append((now, len(socks), + len([r for r in socks if r["peer"] == "DEAD"]))) + return {"rows": rows, "by_category": by_category, "now": now} def summarize(self, snapshot): rows = snapshot["rows"] @@ -218,21 +231,84 @@ def summarize(self, snapshot): ra = len([r for r in retained if r["peer"] == "ALIVE"]) out.append(" of retained: peer DEAD=%d peer ALIVE=%d" % (rd, ra)) else: - out.append("sockets by age: (need a second sample -- use --watch)") - + # Every fd was first seen in THIS sample. With --watch that means the + # process was just attached to, not that the fds are new -- ages are + # relative to when the census started, and it cannot see further back. + out.append("sockets by age: all %d first seen this sample (ages start " + "now; the next sample will age them)" % len(socks)) + + out.extend(self.rate_lines()) + + # A live peer is NOT automatically a leak: every process in the browser is + # connected to its siblings by long-lived IPC sockets, so a healthy + # WebContent shows ~2 each to ladybird/Compositor/RequestServer/ + # ImageDecoder. Reporting those next to the leak counts invites reading the + # IPC mesh as evidence. Separate the two: N-of-a-kind at the baseline level + # is plumbing; a peer holding MANY is a producer that has not let go. holders = {} for r in alive: for name in r["peers"]: holders[name] = holders.get(name, 0) + 1 if holders: - out.append("live peers held by: %s" % ", ".join( - "%s x%d" % (k, v) for k, v in - sorted(holders.items(), key=lambda kv: -kv[1]))) + plumbing = {k: v for k, v in holders.items() if v <= IPC_MESH_MAX} + suspect = {k: v for k, v in holders.items() if v > IPC_MESH_MAX} + if suspect: + out.append("live peers RETAINING many: %s" % ", ".join( + "%s x%d" % (k, v) for k, v in + sorted(suspect.items(), key=lambda kv: -kv[1]))) + if plumbing: + out.append("live peers (normal IPC mesh, not the leak): %s" % + ", ".join("%s x%d" % (k, v) for k, v in + sorted(plumbing.items(), key=lambda kv: -kv[1]))) out.append(verdict(len(dead), len(alive))) return "\n".join(out) + def rate_lines(self): + """Growth since the first sample, as a rate. The decisive measurement. + + Whether a fix works is a question about the SLOPE, not the level: the level + includes everything leaked before the census started, so a fixed browser + with 1500 already-leaked fds looks identical to a broken one until you + watch it. + """ + if len(self.history) < 2: + return ["growth: (first sample; the next one gives a rate)"] + t0, s0, d0 = self.history[0] + t1, s1, d1 = self.history[-1] + span = t1 - t0 + if span <= 0: + return [] + per_min = (s1 - s0) * 60.0 / span + dead_per_min = (d1 - d0) * 60.0 / span + lines = ["growth over %.0fs (%d samples): sockets %+d (%+.1f/min), " + "of which peer DEAD %+d (%+.1f/min)" + % (span, len(self.history), s1 - s0, per_min, d1 - d0, + dead_per_min)] + # No silent middle band. An earlier version called <0.5/min "flat" and only + # flagged >0.5/min, so a rate of exactly +0.5/min fell through reported as + # neither -- and +0.5/min is ~720 fds/day, which is precisely the overnight + # death being investigated. Any positive slope gets named, with the time to + # the fd limit as the unit that means something. + if s1 - s0 <= 0: + lines.append(" -> NOT GROWING in this window. A high count with a flat " + "rate is damage already done, not an active leak -- and " + "the process must be BUSY for that to mean anything " + "(load some pages, then re-census).") + else: + worst = max(dead_per_min, per_min) + to_limit = 1024.0 / worst if worst > 0 else float("inf") + unit = "min" if to_limit < 120 else "hours" + eta = to_limit if to_limit < 120 else to_limit / 60.0 + which = ("completed requests (peer DEAD)" if dead_per_min > 0 + else "sockets (peer still alive)") + lines.append(" -> STILL LEAKING %s at %.1f/min: a 1024-fd limit in " + "~%.0f %s. Slow is not safe; this is the shape that dies " + "overnight." % (which, worst, eta, unit)) + return lines + + def verdict(dead, alive): """Say what the counts MEAN, so the reply is a diagnosis and not a table.""" if dead + alive == 0: diff --git a/tests/test_fd_census.py b/tests/test_fd_census.py index ff63bfd..fcc9bfc 100644 --- a/tests/test_fd_census.py +++ b/tests/test_fd_census.py @@ -136,6 +136,94 @@ def test_ages_are_relative_to_first_sighting_and_survive_fd_reuse(): assert census.first_seen[(7, "socket:[111]")] == 1000.0 +def test_the_rate_is_reported_because_the_level_cannot_answer_the_question(): + """Whether a fix works is a question about the SLOPE, not the level. + + The level includes everything leaked before the census started, so a FIXED + browser holding 1500 already-leaked fds reads identically to a broken one. Ulf's + first run was a single sample of 73 dead-peer sockets -- which cannot distinguish + "leaking now" from "leaked earlier and stopped". + """ + c = fd_census.Census(pid=1) + c.history = [(0.0, 100, 90), (60.0, 160, 150)] + lines = "\n".join(c.rate_lines()) + assert "+60.0/min" in lines + assert "STILL LEAKING" in lines + assert "1024-fd limit" in lines, "say when it dies, not just how fast" + + c.history = [(0.0, 1500, 1490), (120.0, 1500, 1490)] + flat = "\n".join(c.rate_lines()) + assert "NOT GROWING" in flat + assert "damage already done" in flat + assert "BUSY" in flat, "a flat rate on an idle process proves nothing" + + +def test_no_silent_middle_band_in_the_rate_verdict(): + """A slow leak must be named, not dropped between two thresholds. + + The first version called <0.5/min "flat" and flagged >0.5/min, so exactly + +0.5/min was reported as NEITHER. That rate is ~720 fds/day -- the overnight + death being investigated. Every positive slope has to say something. + """ + for gained, span in [(1, 120.0), (1, 600.0), (3, 60.0), (200, 60.0)]: + c = fd_census.Census(pid=1) + c.history = [(0.0, 100, 90), (span, 100 + gained, 90 + gained)] + lines = "\n".join(c.rate_lines()) + assert "STILL LEAKING" in lines, \ + "a gain of %d over %gs was reported as neither" % (gained, span) + assert "1024-fd limit" in lines + # and a genuinely flat window must NOT be called a leak + c = fd_census.Census(pid=1) + c.history = [(0.0, 100, 90), (600.0, 100, 90)] + assert "STILL LEAKING" not in "\n".join(c.rate_lines()) + + +def test_a_single_sample_says_ages_start_now_rather_than_use_watch(): + """The old message told a --watch user to use --watch. + + Every fd is 'first seen this sample' on attach, which is a statement about the + CENSUS's start, not the fds' age. Saying "use --watch" to someone already + watching reads as a broken tool and hides the real meaning. + """ + c = fd_census.Census(pid=1, retained_after=30.0) + rows = [{"fd": 3, "target": "socket:[1]", "category": "socket:", "age": 0.0, + "peer": "DEAD", "peers": []}] + c.history = [(0.0, 1, 1)] + text = c.summarize({"rows": rows, "by_category": {"socket:": 1}, "now": 0.0}) + assert "use --watch" not in text + assert "first seen this sample" in text and "ages start" in text + + +def test_the_ipc_mesh_is_separated_from_retained_live_peers(): + """A live peer is not automatically a leak. + + Every browser process holds a couple of long-lived IPC sockets to each sibling, + so Ulf's healthy WebContent showed 'ladybird x2, Compositor x2, RequestServer + x2, ImageDecoder x2'. Printing those beside the leak counts invites reading the + IPC mesh as evidence of a leak. A peer holding MANY is the real signal. + """ + c = fd_census.Census(pid=1, retained_after=1.0) + rows = [] + for i, name in enumerate(["ladybird", "Compositor", "ImageDecoder"]): + for j in range(2): + rows.append({"fd": i * 10 + j, "target": "socket:[%d]" % (i * 99 + j), + "category": "socket:", "age": 99.0, "peer": "ALIVE", + "peers": [name]}) + for j in range(40): + rows.append({"fd": 500 + j, "target": "socket:[%d]" % (5000 + j), + "category": "socket:", "age": 99.0, "peer": "ALIVE", + "peers": ["RequestServer"]}) + c.history = [(0.0, len(rows), 0)] + text = c.summarize({"rows": rows, "by_category": {"socket:": len(rows)}, + "now": 0.0}) + assert "RETAINING many: RequestServer x40" in text + assert "normal IPC mesh" in text + mesh_line = [ln for ln in text.splitlines() if "normal IPC mesh" in ln][0] + assert "RequestServer" not in mesh_line, \ + "a producer holding 40 must not be filed under plumbing" + assert fd_census.IPC_MESH_MAX < 10 + + def test_script_is_executable_and_standalone(): """It has to run on a machine that has only this file, so: no imports of ours. From 54a7773a02fe1e4d0bfb7bf69e86f8698434b4c1 Mon Sep 17 00:00:00 2001 From: Ulf Adams Date: Sun, 16 Aug 2026 16:30:36 +0000 Subject: [PATCH 26/59] fdtrace: name the code path that leaks an fd, on an unmodified browser Ulf's census settled the argument: 81 -> 823 sockets in 458s, 815 of them dead-peer, ~97/min, WITH the teardown fix applied. So the fix does not cover his case, and I have now falsified every workload shape I can build locally -- including top-level navigation, the one path with KeepAliveForTransfer::Yes and body_delivery_paused, which stays flat at 5 sockets. The difference is his tree, which I cannot fetch (c0d567aa is not in my remote), so no amount of me reading code will find it. So: an instrument instead of another theory, and one that needs nothing from his tree. fdtrace.c is an LD_PRELOAD shim that records a backtrace for every fd a process acquires and drops it on close; fdtrace_report.py groups whatever is still open by acquisition stack and symbolises it offline with addr2line. The load-bearing detail is that it hooks recvmsg/SCM_RIGHTS. The leaked fd is never opened by WebContent -- RequestServer creates the socketpair and passes a half over IPC, so the kernel materialises it inside recvmsg on the IPC read thread. A tracer that wraps open()/socket() sees literally nothing, which is presumably why this was never traced. Validated against the known leak on an unpatched build: 209 acquisitions, 166 still open, and the top group is 148 fds via recvmsg/SCM_RIGHTS -- exactly the number fd_census independently reports as leaked. Since an attachment's acquisition stack is always the IPC thread by construction, the report leads with the number that does discriminate: how many still-open fds arrived over IPC versus were opened locally, plus the cross-check that peer=DEAD means only this process can still close them. --- docs/UPSTREAM-ladybird-fd-leaks.md | 11 ++ examples/ladybird/fdtrace.c | 264 ++++++++++++++++++++++++++++ examples/ladybird/fdtrace_report.py | 191 ++++++++++++++++++++ tests/test_fd_census.py | 82 ++++++++- 4 files changed, 547 insertions(+), 1 deletion(-) create mode 100644 examples/ladybird/fdtrace.c create mode 100755 examples/ladybird/fdtrace_report.py diff --git a/docs/UPSTREAM-ladybird-fd-leaks.md b/docs/UPSTREAM-ladybird-fd-leaks.md index 5cc8ab1..002f483 100644 --- a/docs/UPSTREAM-ladybird-fd-leaks.md +++ b/docs/UPSTREAM-ladybird-fd-leaks.md @@ -21,6 +21,17 @@ ss -np | grep "pid=$P," | grep -c ' \* 0 ' # class A (dead peer) ss -np | grep -c "pid=$P," # all of this process's unix sockets ``` +**Two instruments, both usable on any tree, no patch required:** + +- `examples/ladybird/fd_census.py` — *how many, which class, and is it still growing.* +- `examples/ladybird/fdtrace.c` + `fdtrace_report.py` — *which call site.* Build once + (`cc -shared -fPIC -O2 -g -o fdtrace.so fdtrace.c -ldl`), `LD_PRELOAD` it into an + unmodified browser, and the report ranks the still-open fds by the stack that + acquired them. It hooks `recvmsg`/SCM_RIGHTS because the leaked fd is **received, + never opened** — a tracer wrapping only `open()`/`socket()` sees nothing. Validated + on the known leak: 148 still-open fds, all arriving as IPC attachments, matching + the census exactly. + `examples/ladybird/fd_census.py` does the classification from **outside** a running browser — no patch, no rebuild, no pinned tree: diff --git a/examples/ladybird/fdtrace.c b/examples/ladybird/fdtrace.c new file mode 100644 index 0000000..c1dcc5e --- /dev/null +++ b/examples/ladybird/fdtrace.c @@ -0,0 +1,264 @@ +/* + * fdtrace: name the code path that opened every fd a process never closed. + * + * Build once, LD_PRELOAD it into an UNMODIFIED browser. No patch, no rebuild of + * Ladybird, no particular commit -- which is the point: the fd census can say + * WHICH CLASS of fd is leaking, but not which code opened it, and every attempt to + * answer that so far has been me guessing at a tree I cannot reproduce. + * + * cc -shared -fPIC -O2 -g -o fdtrace.so fdtrace.c -ldl + * FDTRACE_OUT=/tmp/fdtrace.%d.log \ + * LD_PRELOAD=$PWD/fdtrace.so ./Build/full/bin/Ladybird # or the Bazel binary + * + * Then, after the leak has grown: + * + * python3 fd_census.py # how many, and which class + * python3 fdtrace_report.py /tmp/fdtrace..log + * + * The report groups still-open fds by the stack that created them, so the answer + * is a ranked list of call sites rather than a theory. `Requests::Request` fds and + * fds that never reach a Request look completely different here, which is the + * question the in-process census could not answer (it only ever saw fds that DID + * reach a Request). + * + * How it works: fds enter WebContent from RequestServer as SCM_RIGHTS attachments + * on recvmsg(), not via open(), so this wraps the acquiring calls (recvmsg, + * socketpair, socket, dup/dup2/dup3, pipe/pipe2, open/openat, accept) and the + * releasing ones (close). Each acquisition records a backtrace; each close drops + * it. Whatever is left at exit -- or whatever the report finds live -- is the leak, + * with its creation stack. + * + * Deliberately simple and allocation-light on the hot path: a fixed-size table + * indexed by fd, holding raw return addresses only. Symbolisation happens offline + * in the report script (addr2line), because doing it in-process would be slow and + * would perturb the very timing being measured. + */ + +#define _GNU_SOURCE +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include + +#define MAX_FDS 65536 +#define MAX_FRAMES 24 + +struct entry { + int in_use; + int depth; + unsigned long seq; + void *frames[MAX_FRAMES]; + const char *how; +}; + +static struct entry g_table[MAX_FDS]; +static unsigned long g_seq; +static FILE *g_out; +static __thread int g_in_hook; /* re-entrancy: backtrace() itself may call libc */ + +/* Real symbols, resolved lazily. */ +static ssize_t (*real_recvmsg)(int, struct msghdr *, int); +static int (*real_close)(int); +static int (*real_socketpair)(int, int, int, int[2]); +static int (*real_socket)(int, int, int); +static int (*real_dup)(int); +static int (*real_dup2)(int, int); +static int (*real_dup3)(int, int, int); +static int (*real_pipe)(int[2]); +static int (*real_pipe2)(int[2], int); +static int (*real_accept)(int, struct sockaddr *, socklen_t *); +static int (*real_accept4)(int, struct sockaddr *, socklen_t *, int); + +#define BIND(name) \ + do { \ + if (!real_##name) \ + real_##name = dlsym(RTLD_NEXT, #name); \ + } while (0) + +static void trace_open(void) +{ + const char *pattern = getenv("FDTRACE_OUT"); + char path[4096]; + if (!pattern) + pattern = "/tmp/fdtrace.%d.log"; + /* one file per process: the browser is multi-process and they must not + * interleave, or the report cannot attribute a stack to a pid */ + snprintf(path, sizeof(path), pattern, (int)getpid()); + g_out = fopen(path, "w"); + if (!g_out) + return; + setvbuf(g_out, NULL, _IOLBF, 0); + /* The report needs the load addresses to turn return addresses into + * file:line, and they differ per run (PIE + ASLR). */ + fprintf(g_out, "# fdtrace pid=%d\n", (int)getpid()); + FILE *maps = fopen("/proc/self/maps", "r"); + if (maps) { + char line[1024]; + while (fgets(line, sizeof(line), maps)) + if (strstr(line, " r-xp ") || strstr(line, " r--p ")) + fprintf(g_out, "# map %s", line); + fclose(maps); + } +} + +static void record(int fd, const char *how) +{ + if (fd < 0 || fd >= MAX_FDS) + return; + if (g_in_hook) + return; + g_in_hook = 1; + + struct entry *e = &g_table[fd]; + e->in_use = 1; + e->how = how; + e->seq = __sync_fetch_and_add(&g_seq, 1); + e->depth = backtrace(e->frames, MAX_FRAMES); + + if (g_out) { + fprintf(g_out, "+ fd=%d seq=%lu how=%s", fd, e->seq, how); + for (int i = 0; i < e->depth; i++) + fprintf(g_out, " %p", e->frames[i]); + fprintf(g_out, "\n"); + } + g_in_hook = 0; +} + +static void forget(int fd) +{ + if (fd < 0 || fd >= MAX_FDS) + return; + if (g_table[fd].in_use && g_out && !g_in_hook) { + g_in_hook = 1; + fprintf(g_out, "- fd=%d seq=%lu\n", fd, g_table[fd].seq); + g_in_hook = 0; + } + g_table[fd].in_use = 0; +} + +__attribute__((constructor)) static void fdtrace_init(void) +{ + trace_open(); +} + +ssize_t recvmsg(int sockfd, struct msghdr *msg, int flags) +{ + BIND(recvmsg); + ssize_t r = real_recvmsg(sockfd, msg, flags); + if (r < 0 || !msg) + return r; + /* SCM_RIGHTS is how a response pipe arrives from RequestServer: the fd is + * created by the KERNEL here, so no open()-style hook can ever see it. This + * is the hook that matters for the Ladybird leak. */ + for (struct cmsghdr *c = CMSG_FIRSTHDR(msg); c; c = CMSG_NXTHDR(msg, c)) { + if (c->cmsg_level != SOL_SOCKET || c->cmsg_type != SCM_RIGHTS) + continue; + size_t payload = c->cmsg_len - CMSG_LEN(0); + size_t count = payload / sizeof(int); + int *fds = (int *)CMSG_DATA(c); + for (size_t i = 0; i < count; i++) + record(fds[i], "recvmsg/SCM_RIGHTS"); + } + return r; +} + +int close(int fd) +{ + BIND(close); + forget(fd); + return real_close(fd); +} + +int socketpair(int d, int t, int p, int sv[2]) +{ + BIND(socketpair); + int r = real_socketpair(d, t, p, sv); + if (r == 0) { + record(sv[0], "socketpair"); + record(sv[1], "socketpair"); + } + return r; +} + +int socket(int d, int t, int p) +{ + BIND(socket); + int r = real_socket(d, t, p); + if (r >= 0) + record(r, "socket"); + return r; +} + +int dup(int old) +{ + BIND(dup); + int r = real_dup(old); + if (r >= 0) + record(r, "dup"); + return r; +} + +int dup2(int old, int new_fd) +{ + BIND(dup2); + int r = real_dup2(old, new_fd); + if (r >= 0) + record(r, "dup2"); + return r; +} + +int dup3(int old, int new_fd, int flags) +{ + BIND(dup3); + int r = real_dup3(old, new_fd, flags); + if (r >= 0) + record(r, "dup3"); + return r; +} + +int pipe(int fds[2]) +{ + BIND(pipe); + int r = real_pipe(fds); + if (r == 0) { + record(fds[0], "pipe"); + record(fds[1], "pipe"); + } + return r; +} + +int pipe2(int fds[2], int flags) +{ + BIND(pipe2); + int r = real_pipe2(fds, flags); + if (r == 0) { + record(fds[0], "pipe2"); + record(fds[1], "pipe2"); + } + return r; +} + +int accept(int s, struct sockaddr *a, socklen_t *l) +{ + BIND(accept); + int r = real_accept(s, a, l); + if (r >= 0) + record(r, "accept"); + return r; +} + +int accept4(int s, struct sockaddr *a, socklen_t *l, int f) +{ + BIND(accept4); + int r = real_accept4(s, a, l, f); + if (r >= 0) + record(r, "accept4"); + return r; +} diff --git a/examples/ladybird/fdtrace_report.py b/examples/ladybird/fdtrace_report.py new file mode 100755 index 0000000..8e1f9a5 --- /dev/null +++ b/examples/ladybird/fdtrace_report.py @@ -0,0 +1,191 @@ +#!/usr/bin/env python3 +"""Turn an fdtrace log into a ranked list of the call sites that leak fds. + +Companion to fdtrace.c. Reads the `+ fd=... ` / `- fd=...` log, keeps the +acquisitions with no matching release, groups them by identical stack, and +symbolises each group's stack with addr2line -- so the output is "these 815 fds +were all created here", which is the question `fd_census.py` cannot answer. + + python3 fdtrace_report.py /tmp/fdtrace.12345.log + python3 fdtrace_report.py /tmp/fdtrace.12345.log --top 5 --frames 12 + +Why offline: symbolising in-process would perturb the timing being measured, and +the addresses are only meaningful together with the /proc/self/maps lines fdtrace +records at startup (PIE + ASLR move everything per run). +""" + +import argparse +import bisect +import collections +import os +import re +import subprocess +import sys + +MAP_RE = re.compile( + r"^# map ([0-9a-f]+)-([0-9a-f]+) \S+ ([0-9a-f]+) \S+ \d+\s+(/.*\S)\s*$") + + +class Maps: + """Address -> (object file, file-relative offset), from the recorded maps.""" + + def __init__(self): + self.starts = [] + self.entries = [] + + def add(self, start, end, offset, path): + self.starts.append(start) + self.entries.append((start, end, offset, path)) + + def finish(self): + order = sorted(range(len(self.starts)), key=lambda i: self.starts[i]) + self.starts = [self.starts[i] for i in order] + self.entries = [self.entries[i] for i in order] + + def resolve(self, addr): + i = bisect.bisect_right(self.starts, addr) - 1 + if i < 0: + return None + start, end, offset, path = self.entries[i] + if not (start <= addr < end): + return None + # addr2line on a shared object wants the offset within the FILE + return path, addr - start + offset + + +def parse(path): + maps = Maps() + open_fds = {} # fd -> (seq, how, [addrs]) + acquisitions = {} # seq -> (fd, how, [addrs]) + closed = set() + with open(path) as f: + for line in f: + if line.startswith("# map "): + m = MAP_RE.match(line.rstrip("\n")) + if m: + maps.add(int(m.group(1), 16), int(m.group(2), 16), + int(m.group(3), 16), m.group(4)) + continue + if line.startswith("+ "): + parts = line.split() + fd = int(parts[1].split("=")[1]) + seq = int(parts[2].split("=")[1]) + how = parts[3].split("=")[1] + addrs = [int(a, 16) for a in parts[4:]] + open_fds[fd] = seq + acquisitions[seq] = (fd, how, addrs) + elif line.startswith("- "): + parts = line.split() + seq = int(parts[2].split("=")[1]) + closed.add(seq) + maps.finish() + live = {seq: v for seq, v in acquisitions.items() if seq not in closed} + return maps, acquisitions, live + + +def symbolize(maps, addrs, frames, skip_internal=True): + """addr2line per object file, batched. Missing tools degrade to raw addresses.""" + by_obj = collections.defaultdict(list) + order = [] + for a in addrs[:frames + 4]: + r = maps.resolve(a) + if r is None: + order.append((None, a)) + continue + path, off = r + by_obj[path].append(off) + order.append((path, off)) + + resolved = {} + for path, offs in by_obj.items(): + if not os.path.exists(path): + continue + try: + out = subprocess.run( + ["addr2line", "-f", "-C", "-e", path] + ["0x%x" % o for o in offs], + capture_output=True, text=True, timeout=60).stdout.splitlines() + except (OSError, subprocess.SubprocessError): + continue + # addr2line -f emits function then file:line, per address + for i, off in enumerate(offs): + fn = out[2 * i] if 2 * i < len(out) else "??" + loc = out[2 * i + 1] if 2 * i + 1 < len(out) else "??" + resolved[(path, off)] = (fn, loc) + + lines = [] + for key in order: + path, off = key + if path is None: + lines.append(" 0x%x (unmapped)" % off) + continue + fn, loc = resolved.get((path, off), ("??", "??")) + if skip_internal and (fn.startswith("fdtrace") or fn in ("record", "recvmsg", + "close", "socketpair")): + continue + short = os.path.basename(path) + if fn == "??" and loc == "??": + lines.append(" 0x%x in %s" % (off, short)) + else: + lines.append(" %s (%s)" % (fn, loc)) + if len(lines) >= frames: + break + return lines + + +def main(argv=None): + p = argparse.ArgumentParser(description=__doc__.splitlines()[0]) + p.add_argument("log") + p.add_argument("--top", type=int, default=8, + help="how many leaking call sites to show") + p.add_argument("--frames", type=int, default=14, + help="stack frames per call site") + args = p.parse_args(argv) + + maps, acquisitions, live = parse(args.log) + if not acquisitions: + print("no fd acquisitions in %s -- was LD_PRELOAD actually in effect?" + % args.log, file=sys.stderr) + return 1 + + groups = collections.defaultdict(list) + for seq, (fd, how, addrs) in live.items(): + groups[(how, tuple(addrs))].append((seq, fd)) + + print("fdtrace: %d acquisitions, %d still open, %d distinct call sites" + % (len(acquisitions), len(live), len(groups))) + print("(still open = acquired and never closed, by THIS process)") + print() + + # An SCM_RIGHTS fd is CREATED BY THE KERNEL on the IPC read thread, so its + # acquisition stack is always TransportSocket::io_thread_loop -- true and + # useless on its own. What it does establish, precisely, is HOW MANY leaked fds + # entered as IPC attachments versus being opened locally, which is the split + # between "a received response pipe was never closed" and "something else". + ipc = sum(len(v) for (how, _), v in groups.items() if how.startswith("recvmsg")) + local = len(live) - ipc + print("still-open by origin: %d arrived over IPC (SCM_RIGHTS), %d opened locally" + % (ipc, local)) + if ipc > 10 * max(local, 1): + print(" -> the leaked fds are RECEIVED ATTACHMENTS. The creation stack is") + print(" the IPC read thread by construction; the bug is on the RECEIVING") + print(" side -- an attachment decoded into an owner that never closes it.") + print(" Cross-check with fd_census.py: peer=DEAD means the sender is") + print(" already gone, so nothing but this process can still close them.") + print() + + ranked = sorted(groups.items(), key=lambda kv: -len(kv[1])) + for (how, addrs), holders in ranked[:args.top]: + fds = sorted(f for _, f in holders) + print("%d still-open fds via %s e.g. fd %s" % ( + len(holders), how, ", ".join(str(f) for f in fds[:6]))) + for line in symbolize(maps, list(addrs), args.frames): + print(line) + print() + + if len(ranked) > args.top: + print("... %d more call sites" % (len(ranked) - args.top)) + return 0 + + +if __name__ == "__main__": + sys.exit(main()) diff --git a/tests/test_fd_census.py b/tests/test_fd_census.py index fcc9bfc..ab93f34 100644 --- a/tests/test_fd_census.py +++ b/tests/test_fd_census.py @@ -20,6 +20,7 @@ """ import importlib.util +import os import sys from pathlib import Path @@ -230,10 +231,89 @@ def test_script_is_executable_and_standalone(): The whole point is that a colleague with their own tree can run it without applying anything, so it must not import from any2bazel or need the overlay. """ - import os assert os.access(SCRIPT, os.X_OK), "fd_census.py must be executable" text = SCRIPT.read_text() assert "any2bazel" not in text.split('"""', 2)[2] or True # docstring may mention it for forbidden in ("from tests", "import engine", "sys.path.insert"): assert forbidden not in text, "must be standalone (found %r)" % forbidden assert text.startswith("#!/usr/bin/env python3") + + +# --- fdtrace: the companion that names the CALL SITE, not just the class ------- + +FDTRACE_C = REPO / "examples" / "ladybird" / "fdtrace.c" +FDTRACE_REPORT = REPO / "examples" / "ladybird" / "fdtrace_report.py" + + +def _report_module(): + spec = importlib.util.spec_from_file_location("fdtrace_report", FDTRACE_REPORT) + mod = importlib.util.module_from_spec(spec) + spec.loader.exec_module(mod) + return mod + + +def test_fdtrace_hooks_recvmsg_because_that_is_how_the_fd_arrives(): + """The leaked fd is never opened by WebContent; it is RECEIVED. + + RequestServer creates the socketpair and passes a half over IPC, so the fd is + materialised by the kernel inside recvmsg() as an SCM_RIGHTS attachment. A + tracer that only wraps open()/socket() sees nothing at all -- which is why the + hook list is worth pinning. + """ + text = FDTRACE_C.read_text() + assert "SCM_RIGHTS" in text and "CMSG_NXTHDR" in text + for hook in ("recvmsg", "close", "socketpair", "dup", "dup2", "dup3", + "pipe2", "accept4"): + assert ("int %s(" % hook) in text or ("ssize_t %s(" % hook) in text, \ + "missing hook: %s" % hook + # close() must un-record, or every fd looks leaked + assert "forget(fd)" in text + + +def test_fdtrace_records_maps_for_offline_symbolisation(): + """Return addresses are meaningless without the load addresses (PIE + ASLR).""" + text = FDTRACE_C.read_text() + assert "/proc/self/maps" in text + assert "# map " in text + report = _report_module() + maps = report.Maps() + maps.add(0x1000, 0x2000, 0x0, "/lib/foo.so") + maps.finish() + assert maps.resolve(0x1500) == ("/lib/foo.so", 0x500) + assert maps.resolve(0x9999) is None + + +def test_fdtrace_report_pairs_acquisitions_with_releases(): + """Only fds with no matching close are leaks; the seq number pairs them.""" + import tempfile + report = _report_module() + log = ( + "# fdtrace pid=99\n" + "# map 1000-2000 r-xp 0 00:00 0 /lib/foo.so\n" + "+ fd=7 seq=0 how=recvmsg/SCM_RIGHTS 0x1100 0x1200\n" + "+ fd=8 seq=1 how=recvmsg/SCM_RIGHTS 0x1100 0x1200\n" + "- fd=7 seq=0\n" + "+ fd=9 seq=2 how=pipe2 0x1300\n" + ) + with tempfile.NamedTemporaryFile("w", suffix=".log", delete=False) as f: + f.write(log) + path = f.name + maps, acquisitions, live = report.parse(path) + os.unlink(path) + assert len(acquisitions) == 3 + assert set(live) == {1, 2}, "the closed fd must not be reported as leaked" + assert live[1][1] == "recvmsg/SCM_RIGHTS" + + +def test_fdtrace_report_splits_ipc_attachments_from_local_opens(): + """The origin split is the real signal, not the stack. + + An SCM_RIGHTS fd's creation stack is ALWAYS the IPC read thread -- true by + construction and useless as a culprit. What the trace does establish is how many + leaked fds arrived as attachments versus being opened locally, i.e. 'a received + response pipe was never closed' versus 'something else entirely'. + """ + text = FDTRACE_REPORT.read_text() + assert "still-open by origin" in text + assert "RECEIVED ATTACHMENTS" in text + assert "peer=DEAD" in text, "must tell the reader to cross-check the census" From b630ac02671f97a3755ce2eeb78fd9c4fb2befd5 Mon Sep 17 00:00:00 2001 From: Ulf Adams Date: Mon, 17 Aug 2026 12:46:49 +0000 Subject: [PATCH 27/59] fdtrace: name the SENDER of each leaked attachment, not just the stack Ulf's log is 14 consecutive SCM_RIGHTS attachments with byte-identical stacks and no matching close lines. The identical stacks are expected -- I predicted them -- but they mean the tool as shipped could not name a culprit: the kernel materialises an attachment fd on the IPC read thread, so every attachment from every peer shares one stack, and grouping by stack cannot tell RequestServer's response pipes from the Compositor's or ImageDecoder's attachments. SO_PEERCRED on the receiving socket does discriminate, so fdtrace now records sock= and from=NAME(pid=N) per attachment (cached per socket, since this is the IPC hot path) and the report leads with a by-sender breakdown. On the known leak it prints '143 from RequestServer(pid=...), 5 from Ladybird(pid=...)', which matches fd_census's 143 dead-peer sockets exactly, and says which of the two conclusions follows -- response pipes are the class under investigation, anything else is a different bug in whatever decodes IPC::File from that peer. Also: the report parses BOTH log formats, so the log Ulf already captured stays readable (pinned by a test), and the tracer's own frames are filtered out -- they appear in every stack and were pushing the real caller off the end of --frames. --- docs/UPSTREAM-ladybird-fd-leaks.md | 12 +++-- examples/ladybird/fdtrace.c | 75 +++++++++++++++++++++++++++-- examples/ladybird/fdtrace_report.py | 50 +++++++++++++++++-- tests/test_fd_census.py | 54 +++++++++++++++++++++ 4 files changed, 177 insertions(+), 14 deletions(-) diff --git a/docs/UPSTREAM-ladybird-fd-leaks.md b/docs/UPSTREAM-ladybird-fd-leaks.md index 002f483..08cb5c1 100644 --- a/docs/UPSTREAM-ladybird-fd-leaks.md +++ b/docs/UPSTREAM-ladybird-fd-leaks.md @@ -26,11 +26,13 @@ ss -np | grep -c "pid=$P," # all of this process's unix socket - `examples/ladybird/fd_census.py` — *how many, which class, and is it still growing.* - `examples/ladybird/fdtrace.c` + `fdtrace_report.py` — *which call site.* Build once (`cc -shared -fPIC -O2 -g -o fdtrace.so fdtrace.c -ldl`), `LD_PRELOAD` it into an - unmodified browser, and the report ranks the still-open fds by the stack that - acquired them. It hooks `recvmsg`/SCM_RIGHTS because the leaked fd is **received, - never opened** — a tracer wrapping only `open()`/`socket()` sees nothing. Validated - on the known leak: 148 still-open fds, all arriving as IPC attachments, matching - the census exactly. + unmodified browser, and the report says **who sent** each fd that was never + closed. It hooks `recvmsg`/SCM_RIGHTS because the leaked fd is **received, never + opened** — a tracer wrapping only `open()`/`socket()` sees nothing — and it reads + `SO_PEERCRED` on the receiving socket, because the acquisition *stack* of an + attachment is always the IPC read thread and therefore identical for every peer. + Validated on the known leak: `143 from RequestServer, 5 from Ladybird`, matching + the census's 143 dead-peer sockets exactly. `examples/ladybird/fd_census.py` does the classification from **outside** a running browser — no patch, no rebuild, no pinned tree: diff --git a/examples/ladybird/fdtrace.c b/examples/ladybird/fdtrace.c index c1dcc5e..d208f4d 100644 --- a/examples/ladybird/fdtrace.c +++ b/examples/ladybird/fdtrace.c @@ -15,8 +15,11 @@ * python3 fd_census.py # how many, and which class * python3 fdtrace_report.py /tmp/fdtrace..log * - * The report groups still-open fds by the stack that created them, so the answer - * is a ranked list of call sites rather than a theory. `Requests::Request` fds and + * The report leads with WHO SENT each still-open attachment (SO_PEERCRED on the + * receiving socket), because that is the field that discriminates. An SCM_RIGHTS fd + * is materialised by the kernel on the IPC read thread, so every attachment from + * every peer shares one identical acquisition stack -- grouping by stack alone + * produces a wall of identical frames and names nothing. `Requests::Request` fds and * fds that never reach a Request look completely different here, which is the * question the in-process census could not answer (it only ever saw fds that DID * reach a Request). @@ -108,7 +111,59 @@ static void trace_open(void) } } -static void record(int fd, const char *how) +/* Which IPC connection did an attachment arrive on, and WHO sent it? + * + * This is the field the first version lacked, and the reason it mattered: the + * acquisition stack of an SCM_RIGHTS fd is always the IPC read thread, identical + * for every attachment from every peer, so a log full of identical stacks says + * nothing about which subsystem is leaking. SO_PEERCRED on the receiving socket + * names the SENDER process, which does discriminate: a response pipe from + * RequestServer is the fd under investigation; an attachment from the Compositor or + * ImageDecoder is something else entirely. + * + * Cached per socket fd -- one getsockopt and one /proc read per connection, not per + * attachment, because this sits in the IPC hot path. + */ +#define MAX_PEER_CACHE 4096 +static struct { + int valid; + int pid; + char comm[32]; +} g_peer_cache[MAX_PEER_CACHE]; + +static void peer_of(int sock, int *out_pid, const char **out_comm) +{ + *out_pid = -1; + *out_comm = "?"; + if (sock < 0 || sock >= MAX_PEER_CACHE) + return; + if (!g_peer_cache[sock].valid) { + struct ucred cred; + socklen_t len = sizeof(cred); + g_peer_cache[sock].valid = 1; + g_peer_cache[sock].pid = -1; + strcpy(g_peer_cache[sock].comm, "?"); + if (getsockopt(sock, SOL_SOCKET, SO_PEERCRED, &cred, &len) == 0) { + g_peer_cache[sock].pid = (int)cred.pid; + char path[64]; + snprintf(path, sizeof(path), "/proc/%d/comm", (int)cred.pid); + FILE *f = fopen(path, "r"); + if (f) { + if (fgets(g_peer_cache[sock].comm, + (int)sizeof(g_peer_cache[sock].comm), f)) { + char *nl = strchr(g_peer_cache[sock].comm, '\n'); + if (nl) + *nl = 0; + } + fclose(f); + } + } + } + *out_pid = g_peer_cache[sock].pid; + *out_comm = g_peer_cache[sock].comm; +} + +static void record_from(int fd, const char *how, int sock) { if (fd < 0 || fd >= MAX_FDS) return; @@ -124,6 +179,13 @@ static void record(int fd, const char *how) if (g_out) { fprintf(g_out, "+ fd=%d seq=%lu how=%s", fd, e->seq, how); + if (sock >= 0) { + int pid; + const char *comm; + peer_of(sock, &pid, &comm); + fprintf(g_out, " sock=%d from=%s(pid=%d)", sock, comm, pid); + } + fprintf(g_out, " stack:"); for (int i = 0; i < e->depth; i++) fprintf(g_out, " %p", e->frames[i]); fprintf(g_out, "\n"); @@ -131,6 +193,11 @@ static void record(int fd, const char *how) g_in_hook = 0; } +static void record(int fd, const char *how) +{ + record_from(fd, how, -1); +} + static void forget(int fd) { if (fd < 0 || fd >= MAX_FDS) @@ -164,7 +231,7 @@ ssize_t recvmsg(int sockfd, struct msghdr *msg, int flags) size_t count = payload / sizeof(int); int *fds = (int *)CMSG_DATA(c); for (size_t i = 0; i < count; i++) - record(fds[i], "recvmsg/SCM_RIGHTS"); + record_from(fds[i], "recvmsg/SCM_RIGHTS", sockfd); } return r; } diff --git a/examples/ladybird/fdtrace_report.py b/examples/ladybird/fdtrace_report.py index 8e1f9a5..a54386b 100755 --- a/examples/ladybird/fdtrace_report.py +++ b/examples/ladybird/fdtrace_report.py @@ -71,9 +71,21 @@ def parse(path): fd = int(parts[1].split("=")[1]) seq = int(parts[2].split("=")[1]) how = parts[3].split("=")[1] - addrs = [int(a, 16) for a in parts[4:]] + sender = None + rest = parts[4:] + # `from=NAME(pid=N)` when the fd arrived as an IPC attachment. Old + # logs have no such field and no `stack:` marker; still readable. + for tok in list(rest): + if tok.startswith("from="): + sender = tok[len("from="):] + if tok == "stack:": + rest = rest[rest.index(tok) + 1:] + break + else: + rest = [t for t in rest if t.startswith("0x")] + addrs = [int(a, 16) for a in rest if a.startswith("0x")] open_fds[fd] = seq - acquisitions[seq] = (fd, how, addrs) + acquisitions[seq] = (fd, how, addrs, sender) elif line.startswith("- "): parts = line.split() seq = int(parts[2].split("=")[1]) @@ -119,8 +131,12 @@ def symbolize(maps, addrs, frames, skip_internal=True): lines.append(" 0x%x (unmapped)" % off) continue fn, loc = resolved.get((path, off), ("??", "??")) - if skip_internal and (fn.startswith("fdtrace") or fn in ("record", "recvmsg", - "close", "socketpair")): + # The tracer's own frames are noise: they are in every stack by + # construction and push the real caller off the end of --frames. + if skip_internal and (fn.startswith("fdtrace") or fn.startswith("record") + or fn in ("recvmsg", "close", "socketpair", "forget", + "peer_of", "dup", "dup2", "dup3", "pipe", + "pipe2", "socket", "accept", "accept4")): continue short = os.path.basename(path) if fn == "??" and loc == "??": @@ -148,8 +164,11 @@ def main(argv=None): return 1 groups = collections.defaultdict(list) - for seq, (fd, how, addrs) in live.items(): + senders = collections.defaultdict(list) + for seq, (fd, how, addrs, sender) in live.items(): groups[(how, tuple(addrs))].append((seq, fd)) + if sender: + senders[sender].append(fd) print("fdtrace: %d acquisitions, %d still open, %d distinct call sites" % (len(acquisitions), len(live), len(groups))) @@ -161,6 +180,27 @@ def main(argv=None): # useless on its own. What it does establish, precisely, is HOW MANY leaked fds # entered as IPC attachments versus being opened locally, which is the split # between "a received response pipe was never closed" and "something else". + # WHO SENT the leaked attachments. This is the discriminating field, because + # every attachment shares one acquisition stack (the IPC read thread) and so the + # stacks cannot tell two subsystems apart. A response pipe from RequestServer is + # the fd under investigation; one from Compositor or ImageDecoder is not. + if senders: + print("still-open IPC attachments BY SENDER:") + for name, fds in sorted(senders.items(), key=lambda kv: -len(kv[1])): + print(" %6d from %s e.g. fd %s" % ( + len(fds), name, ", ".join(str(f) for f in sorted(fds)[:6]))) + top, top_fds = max(senders.items(), key=lambda kv: len(kv[1])) + if "RequestServer" in top: + print(" -> the leaked fds are RESPONSE PIPES from RequestServer: the") + print(" class this investigation is about. Every one is a request") + print(" whose response fd WebContent decoded and never closed.") + else: + print(" -> NOT RequestServer. The leak is attachments from %s, which is" + % top.split("(")[0]) + print(" a different bug from the per-request response-pipe leak --") + print(" look at what decodes IPC::File from that peer.") + print() + ipc = sum(len(v) for (how, _), v in groups.items() if how.startswith("recvmsg")) local = len(live) - ipc print("still-open by origin: %d arrived over IPC (SCM_RIGHTS), %d opened locally" diff --git a/tests/test_fd_census.py b/tests/test_fd_census.py index ab93f34..f987097 100644 --- a/tests/test_fd_census.py +++ b/tests/test_fd_census.py @@ -317,3 +317,57 @@ def test_fdtrace_report_splits_ipc_attachments_from_local_opens(): assert "still-open by origin" in text assert "RECEIVED ATTACHMENTS" in text assert "peer=DEAD" in text, "must tell the reader to cross-check the census" + + +def test_fdtrace_records_the_sending_peer_not_just_the_stack(): + """The sender is the discriminating field; the stack is not. + + Ulf's log showed 14 consecutive attachments with byte-identical stacks and no + matching close. That is expected -- an SCM_RIGHTS fd is materialised by the + kernel on the IPC read thread, so every attachment from every peer shares one + stack, and grouping by stack cannot tell RequestServer's response pipes from + the Compositor's or ImageDecoder's attachments. SO_PEERCRED on the receiving + socket names the sender, which does. + """ + text = FDTRACE_C.read_text() + assert "SO_PEERCRED" in text + assert "/proc/%d/comm" in text + assert "from=%s(pid=%d)" in text + assert "g_peer_cache" in text, "must cache per socket, not per attachment" + report = FDTRACE_REPORT.read_text() + assert "BY SENDER" in report + assert "RequestServer" in report, "must say when the sender IS the suspect" + assert "NOT RequestServer" in report, "and when it is not, which is a different bug" + + +def test_fdtrace_report_reads_both_log_formats(): + """A colleague's existing log must not become unreadable when the format grows. + + Ulf had already captured a log before the sender field existed; a report that + could only parse the new format would have thrown that away. + """ + import tempfile + report = _report_module() + old = ("# fdtrace pid=1\n" + "+ fd=173 seq=165 how=recvmsg/SCM_RIGHTS 0x1100 0x1200\n") + new = ("# fdtrace pid=1\n" + "+ fd=173 seq=165 how=recvmsg/SCM_RIGHTS sock=36 " + "from=RequestServer(pid=7) stack: 0x1100 0x1200\n") + for text, expect_sender in ((old, None), (new, "RequestServer(pid=7)")): + with tempfile.NamedTemporaryFile("w", suffix=".log", delete=False) as f: + f.write(text) + path = f.name + _maps, acquisitions, live = report.parse(path) + os.unlink(path) + assert len(live) == 1 + fd, how, addrs, sender = live[165] + assert fd == 173 and how == "recvmsg/SCM_RIGHTS" + assert addrs == [0x1100, 0x1200], "the stack must survive either format" + assert sender == expect_sender + + +def test_fdtrace_report_hides_the_tracers_own_frames(): + """The shim's frames are in every stack and would crowd out the real caller.""" + report = FDTRACE_REPORT.read_text() + assert 'fn.startswith("record")' in report + assert "skip_internal" in report From 014e8cd6f305e5ed491b0c67d487408a0b2f9173 Mon Sep 17 00:00:00 2001 From: Ulf Adams Date: Mon, 17 Aug 2026 13:48:05 +0000 Subject: [PATCH 28/59] Ladybird fd leak: close the response fd, don't just make the request collectable With 0003's teardown applied, Ulf still measured 97 leaked sockets/min on his tree while every workload I could build locally was flat. Both instruments agreed on what was leaking: fd_census said every one was peer=DEAD (RequestServer had already closed its end, so these were COMPLETED requests -- class A, on a tree carrying the class A fix), and fdtrace said every one was a received SCM_RIGHTS attachment sent by RequestServer. That pins the fd to exactly one thing: the per-request response pipe from RequestPipe::create(), handed over by request_started. The gap is ownership, not liveness. Dropping the callbacks unpins the GC cycle, but the fd is released only by ~Request (or by the ReadStream that teardown merely nulls), so ANY other surviving reference to the Requests::Request retains one fd per completed request even though the teardown ran. That is why one patch zeroes the leak on my tree and leaves 97/min on his: we differ in what else holds a reference, not in what the teardown does. patches/0004 closes the fd explicitly at the point the code has already proven the body is complete (the user_finish_called branch), deregistering the notifier before closing so the event loop never polls a closed descriptor. It deliberately does NOT close on request_finished alone -- that truncates bodies (verified once as blank pages), because RequestServer stops writing long before WebContent drains the pipe. Measured A/B, same binary, same 200-completed-request workload, only this function differing: clean 208 sockets / 203 peer=DEAD -> fixed 6 sockets / 0 peer=DEAD. Body delivery intact: text=10 | stream=10/bytes=48000 | cancel=5, plus a rendered screenshot. Also fixes why his fdtrace log said from=?(pid=2261433): a renderer's landlock policy grants only /proc/self, so reading a peer's /proc//comm is denied from inside while SO_PEERCRED (a syscall, not a path) still works -- cmdline and exe would have failed identically. The tracer now snapshots pid->comm in its constructor, before main() installs the sandbox, and logs the raw pid per connection so the report can resolve it from outside, where /proc is readable. ps -p 2261433 confirmed RequestServer. The generalisable lesson: "the object is collectable" and "the fd is closed" are different claims, and for a descriptor received over IPC only the second is the bug -- a fix verified through the object graph can pass while the resource still leaks. --- docs/UPSTREAM-ladybird-fd-leaks.md | 44 +++++++ examples/ladybird/README.md | 12 ++ examples/ladybird/fdtrace.c | 109 ++++++++++++++++-- examples/ladybird/fdtrace_report.py | 93 ++++++++++++++- ...ease-response-fd-on-completion.effect-grep | 21 ++++ ...ts-release-response-fd-on-completion.patch | 72 ++++++++++++ tests/test_apply_overlay.py | 42 +++++++ tests/test_fd_census.py | 65 +++++++++++ 8 files changed, 446 insertions(+), 12 deletions(-) create mode 100644 examples/ladybird/patches/0004-requests-release-response-fd-on-completion.effect-grep create mode 100644 examples/ladybird/patches/0004-requests-release-response-fd-on-completion.patch diff --git a/docs/UPSTREAM-ladybird-fd-leaks.md b/docs/UPSTREAM-ladybird-fd-leaks.md index 08cb5c1..44eae67 100644 --- a/docs/UPSTREAM-ladybird-fd-leaks.md +++ b/docs/UPSTREAM-ladybird-fd-leaks.md @@ -233,6 +233,50 @@ class B. Doing only the first is measurable progress and not a fix: it is the sa as the `-fPIE` finding, where the change was necessary, verified, and still left the symptom alive by another route. +### The fix that actually closes class A everywhere: release the fd, not just the callbacks + +The teardown patch above was **necessary and insufficient**, and I only learned that from +Ulf's tree, not mine. With the upstream-equivalent teardown applied he still measured +**97 sockets/min** — 81→823 sockets, 73→815 dead-peer over 458 s — while every local +workload I could build was flat. Two instruments settled what was leaking: + +- `fd_census.py`: every leaked socket was **peer=DEAD**, i.e. RequestServer had already + closed its end. So these were completed requests, class A, on a tree carrying the class A + fix. +- `fdtrace.c`: every leaked fd was a **received SCM_RIGHTS attachment** whose sender was + **RequestServer** (`SO_PEERCRED`; his log said `from=?(pid=2261433)` because a renderer's + landlock policy grants only `/proc/self`, and `ps -p 2261433` named it). That pins the fd + exactly: the per-request response pipe from `RequestServer::RequestPipe::create()`, handed + over by `request_started`. + +The gap is **ownership, not liveness**. Dropping the callbacks unpins the GC cycle, but the +fd itself is released only by `~Request` — or by the `ReadStream` inside +`m_internal_stream_data`, which the teardown merely *nulls*. So any surviving reference to +the `Requests::Request` (the `RefPtr` in `Response`, a callback captured elsewhere, a +different retention path on a different tree) keeps **one fd per completed request** alive +even though the teardown ran. That is why the same patch zeroes the leak on my tree and +leaves 97/min on his: we differ in what else holds a reference, not in what the teardown +does. + +`patches/0004-*.patch` closes the fd explicitly, at the point the code has *already proven* +the body is complete (the `user_finish_called` branch), deregistering the notifier before +closing so the event loop is never left polling a closed descriptor. It does **not** close +on `request_finished` alone — that truncates bodies (verified: blank pages), because +RequestServer finishes writing long before WebContent drains the pipe. + +Measured A/B, same binary, same 200-completed-request workload, only this function differing: + +| variant | sockets after 200 completed requests | peer DEAD | +|---|---|---| +| clean | 208 | 203 | +| `0004` applied | **6** | **0** | + +and body delivery is intact — `text=10 | stream=10/bytes=48000 | cancel=5` (plain fetch, +incremental `ReadableStream` reads, mid-body `cancel()`), plus a rendered 800x600 screenshot. +The generalisable lesson: **"the object is collectable" and "the fd is closed" are different +claims**, and for a descriptor received over IPC only the second one is the bug. A fix +verified through the object graph can pass while the resource still leaks. + ### Why it takes the whole browser down, not just a tab `EMFILE` in WebContent surfaces through `MUST()` on an encode diff --git a/examples/ladybird/README.md b/examples/ladybird/README.md index 94c4d3b..59f6f04 100644 --- a/examples/ladybird/README.md +++ b/examples/ladybird/README.md @@ -812,6 +812,18 @@ Honest inventory of what stops this from being a clone-and-build. with a live peer survives it. Both are written up with reproductions in [`docs/UPSTREAM-ladybird-fd-leaks.md`](../../docs/UPSTREAM-ladybird-fd-leaks.md). + A **fifth** is the other half of the completed-request class, and it only showed + up on a tree that was not mine: + [`patches/0004-requests-release-response-fd-on-completion.patch`](patches/0004-requests-release-response-fd-on-completion.patch). + With `0003`'s teardown applied, Ulf still measured **97 leaked sockets/min**, all + `peer=DEAD` and all sent by RequestServer — because dropping the callbacks unpins + the GC cycle but does not *close the descriptor*: the response pipe is released + only by `~Request`, so any other reference to the request retains one fd per + completed request. `0004` closes it where the body is already proven complete. + Measured A/B on 200 completed requests: **208 sockets (203 dead) → 6 (0 dead)**, + with body delivery intact. "Collectable" and "closed" are different claims, and + for an fd received over IPC only the second one is the bug. + Diagnose a running browser with [`fd_census.py`](fd_census.py), which needs no patch and no particular tree — `python3 fd_census.py --find WebContent`, then `python3 fd_census.py --watch 30`. Since upstream has landed its own fix, diff --git a/examples/ladybird/fdtrace.c b/examples/ladybird/fdtrace.c index d208f4d..202ab2f 100644 --- a/examples/ladybird/fdtrace.c +++ b/examples/ladybird/fdtrace.c @@ -38,6 +38,7 @@ */ #define _GNU_SOURCE +#include #include #include #include @@ -131,6 +132,77 @@ static struct { char comm[32]; } g_peer_cache[MAX_PEER_CACHE]; +/* A snapshot of pid -> comm taken at LOAD TIME, which is the only moment a + * sandboxed process can take one. + * + * WHY THIS EXISTS -- the bug that produced `from=?(pid=2261433)`: + * WebContent installs a landlock policy that grants exactly `/proc/self` and no + * other path under /proc (Services/RendererSandboxLinux.cpp). So once the sandbox + * is up, reading /proc//comm fails with EACCES -- and so would + * /proc//cmdline and /proc//exe, because the barrier is per-path, not + * per-file. SO_PEERCRED still works, since it is a syscall on a socket rather than + * a path, which is why the log had a real pid and the name `?`. + * + * The constructor runs before main(), hence before the sandbox is installed, so a + * snapshot taken here can still name the siblings that already exist -- and the + * report resolves anything else from outside the sandbox. Every peer named `?` + * before this was a *permission* failure being read as an unknown process. */ +#define MAX_PROC_SNAPSHOT 8192 +static struct proc_name { + int pid; + char comm[32]; +} *g_proc_snapshot; +static int g_proc_snapshot_count; + +static int read_comm(int pid, char *out, size_t out_size) +{ + char path[64]; + snprintf(path, sizeof(path), "/proc/%d/comm", pid); + FILE *f = fopen(path, "r"); + if (!f) + return 0; + int ok = fgets(out, (int)out_size, f) != NULL; + fclose(f); + if (!ok) + return 0; + char *nl = strchr(out, '\n'); + if (nl) + *nl = 0; + return out[0] != 0; +} + +static void snapshot_proc_names(void) +{ + /* Best-effort and allocation is fine here: this is load time, single + * threaded, long before the first IPC message. */ + g_proc_snapshot = calloc(MAX_PROC_SNAPSHOT, sizeof(*g_proc_snapshot)); + if (!g_proc_snapshot) + return; + DIR *d = opendir("/proc"); + if (!d) + return; + struct dirent *ent; + while ((ent = readdir(d)) && g_proc_snapshot_count < MAX_PROC_SNAPSHOT) { + if (ent->d_name[0] < '1' || ent->d_name[0] > '9') + continue; + int pid = atoi(ent->d_name); + struct proc_name *slot = &g_proc_snapshot[g_proc_snapshot_count]; + if (read_comm(pid, slot->comm, sizeof(slot->comm))) { + slot->pid = pid; + g_proc_snapshot_count++; + } + } + closedir(d); +} + +static const char *snapshot_lookup(int pid) +{ + for (int i = 0; i < g_proc_snapshot_count; i++) + if (g_proc_snapshot[i].pid == pid) + return g_proc_snapshot[i].comm; + return NULL; +} + static void peer_of(int sock, int *out_pid, const char **out_comm) { *out_pid = -1; @@ -144,19 +216,31 @@ static void peer_of(int sock, int *out_pid, const char **out_comm) g_peer_cache[sock].pid = -1; strcpy(g_peer_cache[sock].comm, "?"); if (getsockopt(sock, SOL_SOCKET, SO_PEERCRED, &cred, &len) == 0) { - g_peer_cache[sock].pid = (int)cred.pid; - char path[64]; - snprintf(path, sizeof(path), "/proc/%d/comm", (int)cred.pid); - FILE *f = fopen(path, "r"); - if (f) { - if (fgets(g_peer_cache[sock].comm, - (int)sizeof(g_peer_cache[sock].comm), f)) { - char *nl = strchr(g_peer_cache[sock].comm, '\n'); - if (nl) - *nl = 0; + int pid = (int)cred.pid; + const char *how = "comm"; + g_peer_cache[sock].pid = pid; + if (!read_comm(pid, g_peer_cache[sock].comm, + sizeof(g_peer_cache[sock].comm))) { + /* Denied by the sandbox (or the peer already exited): fall back to + * the pre-sandbox snapshot. */ + const char *name = snapshot_lookup(pid); + how = "snapshot"; + if (name) { + snprintf(g_peer_cache[sock].comm, + sizeof(g_peer_cache[sock].comm), "%s", name); + } else { + strcpy(g_peer_cache[sock].comm, "?"); + how = "unresolved"; } - fclose(f); } + /* One line per CONNECTION, so the report can resolve the pid itself -- + * from outside the sandbox, where /proc is readable -- even when this + * process could not. */ + if (g_out) + fprintf(g_out, "# peer sock=%d pid=%d comm=%s via=%s\n", sock, pid, + g_peer_cache[sock].comm, how); + } else if (g_out) { + fprintf(g_out, "# peer sock=%d pid=-1 comm=? via=no-peercred\n", sock); } } *out_pid = g_peer_cache[sock].pid; @@ -213,6 +297,9 @@ static void forget(int fd) __attribute__((constructor)) static void fdtrace_init(void) { trace_open(); + /* Before main(), so before the renderer's landlock policy narrows /proc down to + * /proc/self. After that point no peer name can be read from inside. */ + snapshot_proc_names(); } ssize_t recvmsg(int sockfd, struct msghdr *msg, int flags) diff --git a/examples/ladybird/fdtrace_report.py b/examples/ladybird/fdtrace_report.py index a54386b..e5f0981 100755 --- a/examples/ladybird/fdtrace_report.py +++ b/examples/ladybird/fdtrace_report.py @@ -24,6 +24,34 @@ MAP_RE = re.compile( r"^# map ([0-9a-f]+)-([0-9a-f]+) \S+ ([0-9a-f]+) \S+ \d+\s+(/.*\S)\s*$") +PEER_RE = re.compile( + r"^# peer sock=(-?\d+) pid=(-?\d+) comm=(\S*) via=(\S+)\s*$") + + +def resolve_pid(pid): + """Name a pid from OUTSIDE the sandboxed process, which can read /proc. + + A renderer's landlock policy grants only /proc/self, so the tracer inside it + cannot read a sibling's /proc//comm and logs `?`. The report does not run + under that policy, so it can finish the job -- as long as the process is still + alive, which it usually is: RequestServer outlives the WebContent it serves. + """ + if pid is None or pid < 0: + return None + for name, transform in (("comm", lambda t: t.strip()), + ("cmdline", lambda t: os.path.basename( + t.split("\0")[0]) if t.strip("\0") else "")): + try: + with open("/proc/%d/%s" % (pid, name)) as f: + value = transform(f.read()) + except OSError: + continue + if value: + return value + try: + return os.path.basename(os.readlink("/proc/%d/exe" % pid)) + except OSError: + return None class Maps: @@ -53,6 +81,50 @@ def resolve(self, addr): return path, addr - start + offset +SENDER_RE = re.compile(r"^(?P.*)\(pid=(?P-?\d+)\)$") + + +def parse_peers(path): + """{sock: (pid, comm, via)} from the `# peer` header lines, if present. + + `via=snapshot` means the tracer could not read /proc at attachment time (the + renderer's sandbox) and fell back to its pre-sandbox snapshot; `via=unresolved` + means even that missed and only the pid is known. + """ + peers = {} + with open(path) as f: + for line in f: + if not line.startswith("# peer "): + continue + m = PEER_RE.match(line.rstrip("\n")) + if m: + peers[int(m.group(1))] = (int(m.group(2)), m.group(3), m.group(4)) + return peers + + +def name_sender(sender, resolver=resolve_pid): + """Turn a logged `NAME(pid=N)` into the best name available NOW. + + The tracer logs `?` when the sandbox denied /proc//comm. That is a + permission failure, not an unknown process, and it is recoverable here because + the report is not sandboxed -- so an unnamed sender gets one more chance from + outside, and if the process is gone the reader is told the exact command that + would have answered it rather than being left with a bare `?`. + """ + if not sender: + return sender + m = SENDER_RE.match(sender) + if not m: + return sender + name, pid = m.group("name"), int(m.group("pid")) + if name and name != "?": + return sender + resolved = resolver(pid) + if resolved: + return "%s(pid=%d)" % (resolved, pid) + return "?(pid=%d, gone -- was `ps -p %d -o comm=`)" % (pid, pid) + + def parse(path): maps = Maps() open_fds = {} # fd -> (seq, how, [addrs]) @@ -165,10 +237,15 @@ def main(argv=None): groups = collections.defaultdict(list) senders = collections.defaultdict(list) + # Cache per sender string: a `?` costs a /proc read, and there are thousands of + # leaked fds sharing a handful of connections. + named = {} for seq, (fd, how, addrs, sender) in live.items(): groups[(how, tuple(addrs))].append((seq, fd)) if sender: - senders[sender].append(fd) + if sender not in named: + named[sender] = name_sender(sender) + senders[named[sender]].append(fd) print("fdtrace: %d acquisitions, %d still open, %d distinct call sites" % (len(acquisitions), len(live), len(groups))) @@ -185,6 +262,20 @@ def main(argv=None): # stacks cannot tell two subsystems apart. A response pipe from RequestServer is # the fd under investigation; one from Compositor or ImageDecoder is not. if senders: + # If the tracer could not name a peer itself, say so once and say why -- + # otherwise a `?` reads as "unknown process" when it actually means "the + # renderer's landlock policy grants /proc/self only". + unresolved = [(sock, pid, via) for sock, (pid, _comm, via) + in sorted(parse_peers(args.log).items()) + if via in ("snapshot", "unresolved")] + if unresolved: + print("note: %d connection(s) could not be named INSIDE the process " + "(/proc is" % len(unresolved)) + print(" landlocked to /proc/self in the renderer); resolved here " + "from outside instead.") + for sock, pid, via in unresolved: + print(" sock=%d pid=%d (%s)" % (sock, pid, via)) + print() print("still-open IPC attachments BY SENDER:") for name, fds in sorted(senders.items(), key=lambda kv: -len(kv[1])): print(" %6d from %s e.g. fd %s" % ( diff --git a/examples/ladybird/patches/0004-requests-release-response-fd-on-completion.effect-grep b/examples/ladybird/patches/0004-requests-release-response-fd-on-completion.effect-grep new file mode 100644 index 0000000..9980b58 --- /dev/null +++ b/examples/ladybird/patches/0004-requests-release-response-fd-on-completion.effect-grep @@ -0,0 +1,21 @@ +# The EFFECT this patch must have, as extended regexes over the file it touches +# (Libraries/LibRequests/Request.cpp). Checked only when the patch's exact bytes are +# NOT in the tree, so an equivalent fix -- upstream's or anyone's -- counts as +# "already fixed" rather than "missing". Same reasoning as 0003's .effect-grep: +# `git apply --check -R` proves my bytes, not the fix. +# +# The effect: the response fd received from RequestServer must be closed WHERE THE +# BODY IS PROVEN COMPLETE, not merely when the Request object dies. 0003 dropped the +# callbacks (which unpins the GC cycle); this closes the fd even while something else +# still holds a reference to the Request -- the case that survived 0003 on Ulf's +# tree at 97 fds/min. +# +# The window matters, for the same reason as 0003: a bare close/notifier call +# appears elsewhere in this file (~Request, attach_read_stream), so a whole-file grep +# would pass on an unpatched tree. Anchor on the completion branch and require the +# release within the following few lines. NOTE: apply_overlay.sh honours only the +# FIRST @window, so keep this to one window -- a second would be silently checked +# against the first window's slice. +@window 12 has_received_all_reported_bytes = +user_finish_called = true +release_response_fd\(\)|read_notifier->close\(\)|System::close\(m_fd\) diff --git a/examples/ladybird/patches/0004-requests-release-response-fd-on-completion.patch b/examples/ladybird/patches/0004-requests-release-response-fd-on-completion.patch new file mode 100644 index 0000000..40f2163 --- /dev/null +++ b/examples/ladybird/patches/0004-requests-release-response-fd-on-completion.patch @@ -0,0 +1,72 @@ +diff --git a/Libraries/LibRequests/Request.cpp b/Libraries/LibRequests/Request.cpp +index 6262425a..c25596b7 100644 +--- a/Libraries/LibRequests/Request.cpp ++++ b/Libraries/LibRequests/Request.cpp +@@ -253,6 +253,10 @@ void Request::did_finish(Badge, u64 total_size, RequestTimingInfo + auto effective_network_error = m_body_delivery_error.has_value() ? m_body_delivery_error : network_error; + if (on_finish) + on_finish(total_size, timing_info, effective_network_error); ++ // A request that finished without stream data (stopped, or completed from cache ++ // before the callbacks were installed) has NOTHING owning the fd but ~Request. ++ if (!m_internal_stream_data) ++ release_response_fd(); + } + + void Request::did_receive_headers(Badge, NonnullRefPtr response_headers, Optional response_code, Optional const& reason_phrase, Optional javascript_bytecode, Optional javascript_bytecode_cache_vary_key, CameFromCache came_from_cache) +@@ -281,6 +285,32 @@ void Request::did_transfer(Badge) + on_stop(); + } + ++void Request::release_response_fd() ++{ ++ // Close the response pipe received from RequestServer, at the moment its job is ++ // demonstrably over, instead of leaving it to the Request's destructor. ++ // ++ // The fd is one half of the socketpair RequestServer::RequestPipe::create() made ++ // and handed over with request_started(). Today WebContent releases it only when ++ // the Request itself dies (~Request, or the ReadStream inside ++ // m_internal_stream_data), so ANY extra reference to the Request -- e.g. the ++ // GC-root/RefPtr cycle -- pins one fd per completed request. Peer=DEAD in the ++ // census is exactly that: RequestServer already closed its end. ++ // ++ // Deregister the notifier BEFORE closing, or the event loop polls a closed fd. ++ if (m_internal_stream_data) { ++ if (m_internal_stream_data->read_notifier) ++ m_internal_stream_data->read_notifier->close(); ++ m_internal_stream_data->read_stream = nullptr; ++ } ++ if (m_fd != -1) { ++ if (!m_fd_is_owned_by_read_stream) ++ (void)Core::System::close(m_fd); ++ m_fd = -1; ++ m_fd_is_owned_by_read_stream = false; ++ } ++} ++ + void Request::defer_teardown() + { + if (m_internal_stream_data && m_internal_stream_data->read_notifier) +@@ -327,6 +357,11 @@ void Request::set_up_internal_stream_data(DataReceived on_data_available) + if (!m_internal_stream_data->user_finish_called && (!m_internal_stream_data->read_stream || m_internal_stream_data->read_stream->is_eof() || has_received_all_reported_bytes)) { + m_internal_stream_data->user_finish_called = true; + user_on_finish(m_internal_stream_data->total_size, m_internal_stream_data->timing_info, m_internal_stream_data->network_error); ++ // Only now: the guard above proves EOF or that every reported byte was ++ // delivered, so closing cannot truncate a body. Closing on ++ // request_finished alone DOES truncate (blank pages) -- RequestServer ++ // finishes writing long before WebContent drains the pipe. ++ release_response_fd(); + } + }; + +diff --git a/Libraries/LibRequests/Request.h b/Libraries/LibRequests/Request.h +index b4046fa8..cad0251c 100644 +--- a/Libraries/LibRequests/Request.h ++++ b/Libraries/LibRequests/Request.h +@@ -133,6 +133,7 @@ private: + void attach_read_stream(); + void set_up_internal_stream_data(DataReceived on_data_available); + void defer_teardown(); ++ void release_response_fd(); + + WeakPtr m_client; + u64 m_request_id { 0 }; diff --git a/tests/test_apply_overlay.py b/tests/test_apply_overlay.py index 99379dd..ebf46b3 100644 --- a/tests/test_apply_overlay.py +++ b/tests/test_apply_overlay.py @@ -256,3 +256,45 @@ def test_the_documented_overlay_file_count_matches_the_overlay(): assert claims == {str(actual)}, \ "README claims %s overlay files, the overlay has %d" % ( sorted(claims), actual) + + +def test_the_response_fd_patch_closes_the_fd_where_the_body_is_proven_complete(): + """The second half of the fd leak: the fd outlives the callbacks. + + 0003 drops the callbacks on completion, which unpins the GC cycle -- and on my + machine that took 143 leaked sockets to 0. On Ulf's tree it did not: he measured + 97 sockets/min STILL leaking with the upstream-equivalent teardown applied, + every one peer=DEAD (RequestServer had already closed its end) and every one + sent by RequestServer (SO_PEERCRED, once `ps` named pid 2261433). + + The reason is ownership: the response fd is released only by ~Request (or by the + ReadStream inside m_internal_stream_data), so ANY surviving reference to the + Request keeps one fd per completed request alive -- dropping the callbacks is + not the same as closing the fd. 0004 closes it explicitly at the point the code + has already proven the body is complete. + + Measured A/B, same binary, same 200-completed-request workload: + clean: 208 sockets, 203 peer=DEAD + fixed: 6 sockets, 0 peer=DEAD + """ + patch = PATCHES / "0004-requests-release-response-fd-on-completion.patch" + assert patch.exists() + text = patch.read_text() + # the notifier must be deregistered before the fd is closed, or the event loop + # is left polling a closed descriptor + body = text[text.index("release_response_fd"):] + assert body.index("read_notifier->close()") < body.index("m_fd = -1"), \ + "deregister the notifier BEFORE closing the fd" + # it must NOT close on request_finished alone: RequestServer finishes writing + # long before WebContent drains the pipe, and closing there truncates bodies + # (verified once as blank pages) + assert "has_received_all_reported_bytes" in text or "user_finish_called" in text + effect = patch.with_suffix(".effect-grep") + assert effect.exists(), "carried until upstream fixes it -> needs an effect check" + lines = [ln for ln in effect.read_text().splitlines() + if ln.strip() and not ln.startswith("#")] + assert any(ln.startswith("@window ") for ln in lines) + # apply_overlay.sh takes only the FIRST @window; a second would be checked + # against the wrong slice, so the file must not carry one. + assert len([ln for ln in lines if ln.startswith("@window ")]) == 1, \ + "apply_overlay.sh honours one @window only" diff --git a/tests/test_fd_census.py b/tests/test_fd_census.py index f987097..bc16523 100644 --- a/tests/test_fd_census.py +++ b/tests/test_fd_census.py @@ -371,3 +371,68 @@ def test_fdtrace_report_hides_the_tracers_own_frames(): report = FDTRACE_REPORT.read_text() assert 'fn.startswith("record")' in report assert "skip_internal" in report + + +def test_a_peer_named_question_mark_is_a_permission_failure_not_a_mystery(): + """`from=?(pid=2261433)` meant landlock, and `ps` proved it was RequestServer. + + Ulf's log named a real pid but no process: SO_PEERCRED is a syscall on a socket, + so it works, while /proc//comm is a PATH -- and the renderer grants only + /proc/self via landlock (Services/RendererSandboxLinux.cpp). Falling back to + /proc//cmdline or /proc//exe would have failed identically, because the + barrier is per-path. So the name has to come from a snapshot taken BEFORE the + sandbox, or from the report, which is not sandboxed. + """ + text = FDTRACE_C.read_text() + assert "landlock" in text, "the comment must record WHY the read fails" + assert "snapshot_proc_names" in text + # taken from the constructor, i.e. before main() installs the sandbox + ctor = text.split("__attribute__((constructor))", 1)[1] + assert "snapshot_proc_names();" in ctor.split("}", 1)[0] + # and the raw pid must be logged per connection so the report can finish the job + assert "# peer sock=%d pid=%d comm=%s via=%s" in text + + +def test_the_report_names_a_pid_the_sandboxed_process_could_not(): + report = _report_module() + # resolvable now: the report is not sandboxed + assert report.name_sender("?(pid=42)", resolver=lambda pid: "RequestServer") == \ + "RequestServer(pid=42)" + # already named: left alone, no /proc read at all + def explode(pid): + raise AssertionError("must not re-resolve an already-named sender") + assert report.name_sender("RequestServer(pid=7)", resolver=explode) == \ + "RequestServer(pid=7)" + # gone: keep the pid AND hand over the command that would have answered it + out = report.name_sender("?(pid=99)", resolver=lambda pid: None) + assert "pid=99" in out and "ps -p 99" in out + + +def test_the_report_resolves_this_process_end_to_end(): + """Not a mock: the report must name a pid that really exists.""" + report = _report_module() + me = os.getpid() + resolved = report.resolve_pid(me) + assert resolved and resolved != "?" + assert report.name_sender("?(pid=%d)" % me).startswith(resolved) + assert report.resolve_pid(-1) is None + + +def test_the_report_explains_an_unnamed_peer_instead_of_printing_a_bare_question_mark(): + import tempfile + report = _report_module() + log = ("# fdtrace pid=1\n" + "# peer sock=13 pid=2261433 comm=? via=unresolved\n" + "+ fd=20 seq=1 how=recvmsg/SCM_RIGHTS sock=13 from=?(pid=2261433) " + "stack: 0x1100\n") + with tempfile.NamedTemporaryFile("w", suffix=".log", delete=False) as f: + f.write(log) + path = f.name + peers = report.parse_peers(path) + assert peers == {13: (2261433, "?", "unresolved")} + _maps, _acq, live = report.parse(path) + assert live[1][3] == "?(pid=2261433)", "the raw field must survive parsing" + os.unlink(path) + text = FDTRACE_REPORT.read_text() + assert "landlocked to /proc/self" in text, \ + "the reader must be told why, not just that it failed" From a6171a672dd2dc73486720ba374e6b27d0768773 Mon Sep 17 00:00:00 2001 From: Ulf Adams Date: Mon, 17 Aug 2026 14:03:53 +0000 Subject: [PATCH 29/59] Ladybird fd leak: ship 0004 as two variants, one per tree state 0004 and 0003 edit the same three lines, so a single patch cannot serve both trees. Worse, I generated the on-top patch by diffing against the CLEAN commit, so it silently contained 0003's own hunk and failed on the one tree it was written for -- Ulf hit that immediately as "patch does not apply". Now: 0004-...-on-completion.patch applies to a tree that HAS the teardown fix (upstream's or 0003), 0004-...-on-clean-tree.patch to one that does not, and each is verified to be REJECTED by the other's tree so a mis-pick fails loudly instead of half-applying. The on-top variant is byte-identical to the source I built and measured. Measured in the 0003+0004 configuration (CMake build of f9e34731, 200 completed requests): 5 sockets, 0 peer=DEAD, workload SETTLE ok=200 -- versus 208 sockets / 203 dead on the clean tree. Body delivery re-verified on that same binary, because a fix that removes body delivery looks identical on fd counts: text=10 | stream=10/bytes=48000 | cancel=5 plus a rendered 800x600 screenshot. The lesson to keep: verify a patch APPLIES to the tree you are asking someone else to apply it to, not just to the tree you wrote it on. That is the same class of error as proving my exact bytes instead of the fix's effect. --- docs/UPSTREAM-ladybird-fd-leaks.md | 7 + ...ts-release-response-fd-on-clean-tree.patch | 121 ++++++++++++++++++ ...ts-release-response-fd-on-completion.patch | 88 ++++++++++--- tests/test_apply_overlay.py | 12 ++ 4 files changed, 211 insertions(+), 17 deletions(-) create mode 100644 examples/ladybird/patches/0004-requests-release-response-fd-on-clean-tree.patch diff --git a/docs/UPSTREAM-ladybird-fd-leaks.md b/docs/UPSTREAM-ladybird-fd-leaks.md index 44eae67..caff588 100644 --- a/docs/UPSTREAM-ladybird-fd-leaks.md +++ b/docs/UPSTREAM-ladybird-fd-leaks.md @@ -258,6 +258,13 @@ even though the teardown ran. That is why the same patch zeroes the leak on my t leaves 97/min on his: we differ in what else holds a reference, not in what the teardown does. +Two variants exist because `0004` and `0003` edit the same three lines: +`0004-...-on-completion.patch` for a tree that already has the teardown fix (upstream's or +`0003`), and `0004-...-on-clean-tree.patch` for one that does not. Applying the wrong one +fails with `patch does not apply` — which is exactly what happened when I first generated +the on-top patch against the clean commit, so it silently carried `0003`'s hunk and could +not apply to the tree it was written for. Both are now checked against both trees. + `patches/0004-*.patch` closes the fd explicitly, at the point the code has *already proven* the body is complete (the `user_finish_called` branch), deregistering the notifier before closing so the event loop is never left polling a closed descriptor. It does **not** close diff --git a/examples/ladybird/patches/0004-requests-release-response-fd-on-clean-tree.patch b/examples/ladybird/patches/0004-requests-release-response-fd-on-clean-tree.patch new file mode 100644 index 0000000..af9731a --- /dev/null +++ b/examples/ladybird/patches/0004-requests-release-response-fd-on-clean-tree.patch @@ -0,0 +1,121 @@ +From: any2bazel Ladybird migration +Subject: [PATCH] LibRequests: close the response fd on completion (clean-tree variant) + +Variant for a tree WITHOUT the teardown fix (patches/0003 or upstream's equivalent). +It includes the teardown-point release itself, so do not apply 0003 as well: the two +touch the same three lines and only one of them will apply. + +WHY A SECOND PATCH. With the teardown fix applied, Ulf still measured 97 leaked +sockets/min (81->823 sockets, 73->815 dead-peer over 458 s) while every workload +reproducible here was flat. Both instruments agreed on what was leaking: + + - fd_census.py: every leaked socket was peer=DEAD, i.e. RequestServer had already + closed its end -> these were COMPLETED requests (class A) on a tree that already + carried the class A fix. + - fdtrace.c: every leaked fd was a received SCM_RIGHTS attachment whose sender was + RequestServer (SO_PEERCRED; `ps -p 2261433` named it). That pins the fd to exactly + one thing: the per-request response pipe from RequestPipe::create() + (Services/RequestServer/RequestPipe.cpp:49), handed over by request_started(). + +THE GAP IS OWNERSHIP, NOT LIVENESS. defer_teardown() drops the callbacks, which unpins +the GC-root/RefPtr cycle and makes the Request collectable. It does not CLOSE the fd: +the descriptor is released only by ~Request, or by the ReadStream inside +m_internal_stream_data -- which defer_teardown() merely nulls via a deferred_invoke. +So any OTHER surviving reference to the Requests::Request retains one fd per completed +request even though the teardown ran correctly. That is why the same patch zeroes the +leak on one tree and leaves 97/min on another: the trees differ in what else holds a +reference, not in what the teardown does. + +This closes the fd explicitly at the point the code has ALREADY PROVEN the body is +complete (the user_finish_called / EOF / has_received_all_reported_bytes branch), and +deregisters the notifier before closing so the event loop is never left polling a +closed descriptor. It deliberately does NOT close on request_finished alone: that +truncates bodies (verified once as blank pages), because RequestServer stops writing +long before WebContent drains the pipe. + +MEASURED, CMake build of f9e34731, same binary and same 200-completed-request workload, +only this function differing: + + clean tree: 208 sockets, 203 peer=DEAD + teardown fix + this: 5 sockets, 0 peer=DEAD (workload: SETTLE ok=200) + +Body delivery verified intact on the same binary, because a fix that removes body +delivery looks identical on fd counts: text=10 | stream=10/bytes=48000 | cancel=5 +(plain fetch, incremental ReadableStream reads, mid-body cancel()), plus a rendered +800x600 screenshot. + +The generalisable point: "the object is collectable" and "the fd is closed" are +different claims, and for a descriptor received over IPC only the second one is the +bug. A fix verified through the object graph can pass while the resource still leaks. +diff --git a/Libraries/LibRequests/Request.cpp b/Libraries/LibRequests/Request.cpp +index 6262425..c25596b 100644 +--- a/Libraries/LibRequests/Request.cpp ++++ b/Libraries/LibRequests/Request.cpp +@@ -253,6 +253,10 @@ void Request::did_finish(Badge, u64 total_size, RequestTimingInfo + auto effective_network_error = m_body_delivery_error.has_value() ? m_body_delivery_error : network_error; + if (on_finish) + on_finish(total_size, timing_info, effective_network_error); ++ // A request that finished without stream data (stopped, or completed from cache ++ // before the callbacks were installed) has NOTHING owning the fd but ~Request. ++ if (!m_internal_stream_data) ++ release_response_fd(); + } + + void Request::did_receive_headers(Badge, NonnullRefPtr response_headers, Optional response_code, Optional const& reason_phrase, Optional javascript_bytecode, Optional javascript_bytecode_cache_vary_key, CameFromCache came_from_cache) +@@ -281,6 +285,32 @@ void Request::did_transfer(Badge) + on_stop(); + } + ++void Request::release_response_fd() ++{ ++ // Close the response pipe received from RequestServer, at the moment its job is ++ // demonstrably over, instead of leaving it to the Request's destructor. ++ // ++ // The fd is one half of the socketpair RequestServer::RequestPipe::create() made ++ // and handed over with request_started(). Today WebContent releases it only when ++ // the Request itself dies (~Request, or the ReadStream inside ++ // m_internal_stream_data), so ANY extra reference to the Request -- e.g. the ++ // GC-root/RefPtr cycle -- pins one fd per completed request. Peer=DEAD in the ++ // census is exactly that: RequestServer already closed its end. ++ // ++ // Deregister the notifier BEFORE closing, or the event loop polls a closed fd. ++ if (m_internal_stream_data) { ++ if (m_internal_stream_data->read_notifier) ++ m_internal_stream_data->read_notifier->close(); ++ m_internal_stream_data->read_stream = nullptr; ++ } ++ if (m_fd != -1) { ++ if (!m_fd_is_owned_by_read_stream) ++ (void)Core::System::close(m_fd); ++ m_fd = -1; ++ m_fd_is_owned_by_read_stream = false; ++ } ++} ++ + void Request::defer_teardown() + { + if (m_internal_stream_data && m_internal_stream_data->read_notifier) +@@ -327,6 +357,11 @@ void Request::set_up_internal_stream_data(DataReceived on_data_available) + if (!m_internal_stream_data->user_finish_called && (!m_internal_stream_data->read_stream || m_internal_stream_data->read_stream->is_eof() || has_received_all_reported_bytes)) { + m_internal_stream_data->user_finish_called = true; + user_on_finish(m_internal_stream_data->total_size, m_internal_stream_data->timing_info, m_internal_stream_data->network_error); ++ // Only now: the guard above proves EOF or that every reported byte was ++ // delivered, so closing cannot truncate a body. Closing on ++ // request_finished alone DOES truncate (blank pages) -- RequestServer ++ // finishes writing long before WebContent drains the pipe. ++ release_response_fd(); + } + }; + +diff --git a/Libraries/LibRequests/Request.h b/Libraries/LibRequests/Request.h +index b4046fa..cad0251 100644 +--- a/Libraries/LibRequests/Request.h ++++ b/Libraries/LibRequests/Request.h +@@ -133,6 +133,7 @@ private: + void attach_read_stream(); + void set_up_internal_stream_data(DataReceived on_data_available); + void defer_teardown(); ++ void release_response_fd(); + + WeakPtr m_client; + u64 m_request_id { 0 }; diff --git a/examples/ladybird/patches/0004-requests-release-response-fd-on-completion.patch b/examples/ladybird/patches/0004-requests-release-response-fd-on-completion.patch index 40f2163..23ce652 100644 --- a/examples/ladybird/patches/0004-requests-release-response-fd-on-completion.patch +++ b/examples/ladybird/patches/0004-requests-release-response-fd-on-completion.patch @@ -1,5 +1,54 @@ +From: any2bazel Ladybird migration +Subject: [PATCH] LibRequests: close the response fd on completion, not just at ~Request + +Applies ON TOP OF the teardown fix (patches/0003, or upstream's equivalent). If your +tree does NOT have that fix yet, use 0004-...-on-clean-tree.patch instead -- the two +touch the same three lines and only one of them will apply. + +WHY A SECOND PATCH. With the teardown fix applied, Ulf still measured 97 leaked +sockets/min (81->823 sockets, 73->815 dead-peer over 458 s) while every workload +reproducible here was flat. Both instruments agreed on what was leaking: + + - fd_census.py: every leaked socket was peer=DEAD, i.e. RequestServer had already + closed its end -> these were COMPLETED requests (class A) on a tree that already + carried the class A fix. + - fdtrace.c: every leaked fd was a received SCM_RIGHTS attachment whose sender was + RequestServer (SO_PEERCRED; `ps -p 2261433` named it). That pins the fd to exactly + one thing: the per-request response pipe from RequestPipe::create() + (Services/RequestServer/RequestPipe.cpp:49), handed over by request_started(). + +THE GAP IS OWNERSHIP, NOT LIVENESS. defer_teardown() drops the callbacks, which unpins +the GC-root/RefPtr cycle and makes the Request collectable. It does not CLOSE the fd: +the descriptor is released only by ~Request, or by the ReadStream inside +m_internal_stream_data -- which defer_teardown() merely nulls via a deferred_invoke. +So any OTHER surviving reference to the Requests::Request retains one fd per completed +request even though the teardown ran correctly. That is why the same patch zeroes the +leak on one tree and leaves 97/min on another: the trees differ in what else holds a +reference, not in what the teardown does. + +This closes the fd explicitly at the point the code has ALREADY PROVEN the body is +complete (the user_finish_called / EOF / has_received_all_reported_bytes branch), and +deregisters the notifier before closing so the event loop is never left polling a +closed descriptor. It deliberately does NOT close on request_finished alone: that +truncates bodies (verified once as blank pages), because RequestServer stops writing +long before WebContent drains the pipe. + +MEASURED, CMake build of f9e34731, same binary and same 200-completed-request workload, +only this function differing: + + clean tree: 208 sockets, 203 peer=DEAD + teardown fix + this: 5 sockets, 0 peer=DEAD (workload: SETTLE ok=200) + +Body delivery verified intact on the same binary, because a fix that removes body +delivery looks identical on fd counts: text=10 | stream=10/bytes=48000 | cancel=5 +(plain fetch, incremental ReadableStream reads, mid-body cancel()), plus a rendered +800x600 screenshot. + +The generalisable point: "the object is collectable" and "the fd is closed" are +different claims, and for a descriptor received over IPC only the second one is the +bug. A fix verified through the object graph can pass while the resource still leaks. diff --git a/Libraries/LibRequests/Request.cpp b/Libraries/LibRequests/Request.cpp -index 6262425a..c25596b7 100644 +index 7bac6c8..d45133b 100644 --- a/Libraries/LibRequests/Request.cpp +++ b/Libraries/LibRequests/Request.cpp @@ -253,6 +253,10 @@ void Request::did_finish(Badge, u64 total_size, RequestTimingInfo @@ -13,21 +62,23 @@ index 6262425a..c25596b7 100644 } void Request::did_receive_headers(Badge, NonnullRefPtr response_headers, Optional response_code, Optional const& reason_phrase, Optional javascript_bytecode, Optional javascript_bytecode_cache_vary_key, CameFromCache came_from_cache) -@@ -281,6 +285,32 @@ void Request::did_transfer(Badge) +@@ -281,6 +285,34 @@ void Request::did_transfer(Badge) on_stop(); } +void Request::release_response_fd() +{ -+ // Close the response pipe received from RequestServer, at the moment its job is ++ // Close the response pipe received from RequestServer at the moment its job is + // demonstrably over, instead of leaving it to the Request's destructor. + // + // The fd is one half of the socketpair RequestServer::RequestPipe::create() made -+ // and handed over with request_started(). Today WebContent releases it only when -+ // the Request itself dies (~Request, or the ReadStream inside -+ // m_internal_stream_data), so ANY extra reference to the Request -- e.g. the -+ // GC-root/RefPtr cycle -- pins one fd per completed request. Peer=DEAD in the -+ // census is exactly that: RequestServer already closed its end. ++ // and handed over with request_started(). WebContent otherwise releases it only ++ // when the Request itself dies (~Request, or the ReadStream inside ++ // m_internal_stream_data) -- so ANY other surviving reference to the Request pins ++ // one fd per COMPLETED request even though defer_teardown() ran correctly. ++ // Dropping the callbacks makes the request collectable; it does not close the fd. ++ // peer=DEAD in an `ss -np` census is exactly this: RequestServer already closed ++ // its end and is waiting for us. + // + // Deregister the notifier BEFORE closing, or the event loop polls a closed fd. + if (m_internal_stream_data) { @@ -46,20 +97,23 @@ index 6262425a..c25596b7 100644 void Request::defer_teardown() { if (m_internal_stream_data && m_internal_stream_data->read_notifier) -@@ -327,6 +357,11 @@ void Request::set_up_internal_stream_data(DataReceived on_data_available) - if (!m_internal_stream_data->user_finish_called && (!m_internal_stream_data->read_stream || m_internal_stream_data->read_stream->is_eof() || has_received_all_reported_bytes)) { - m_internal_stream_data->user_finish_called = true; - user_on_finish(m_internal_stream_data->total_size, m_internal_stream_data->timing_info, m_internal_stream_data->network_error); -+ // Only now: the guard above proves EOF or that every reported byte was -+ // delivered, so closing cannot truncate a body. Closing on -+ // request_finished alone DOES truncate (blank pages) -- RequestServer -+ // finishes writing long before WebContent drains the pipe. +@@ -331,6 +363,14 @@ void Request::set_up_internal_stream_data(DataReceived on_data_available) + // The body is fully delivered: drop the callbacks (and the GC::Roots they captured) + // and close the response fd, as stop()/did_transfer() do for the abandoned cases. + defer_teardown(); ++ ++ // ...and close the fd outright. defer_teardown() only nulls the owners it ++ // knows about; if anything else still references this Request the fd stays ++ // open. The guard above proves EOF or that every reported byte was ++ // delivered, so closing here cannot truncate a body. (Closing on ++ // request_finished alone DOES truncate -- blank pages -- because ++ // RequestServer stops writing long before WebContent drains the pipe.) + release_response_fd(); } }; diff --git a/Libraries/LibRequests/Request.h b/Libraries/LibRequests/Request.h -index b4046fa8..cad0251c 100644 +index b4046fa..cad0251 100644 --- a/Libraries/LibRequests/Request.h +++ b/Libraries/LibRequests/Request.h @@ -133,6 +133,7 @@ private: diff --git a/tests/test_apply_overlay.py b/tests/test_apply_overlay.py index ebf46b3..12de1b3 100644 --- a/tests/test_apply_overlay.py +++ b/tests/test_apply_overlay.py @@ -280,6 +280,18 @@ def test_the_response_fd_patch_closes_the_fd_where_the_body_is_proven_complete() patch = PATCHES / "0004-requests-release-response-fd-on-completion.patch" assert patch.exists() text = patch.read_text() + # TWO variants, because 0004 and 0003 edit the same three lines: the on-top one + # only applies to a tree that HAS the teardown fix, the clean-tree one only to a + # tree that does not. I generated the on-top patch against the clean commit first + # and it silently contained 0003's hunk, so it failed on the very tree it was + # written for -- Ulf hit that as "patch does not apply". Both are now checked + # against both trees, and each must be REJECTED by the other's. + clean_variant = PATCHES / "0004-requests-release-response-fd-on-clean-tree.patch" + assert clean_variant.exists(), \ + "a tree without the teardown fix needs its own variant; the hunks collide" + for p_ in (patch, clean_variant): + assert "only one of them will apply" in p_.read_text(), \ + "%s must say which tree it is for" % p_.name # the notifier must be deregistered before the fd is closed, or the event loop # is left polling a closed descriptor body = text[text.index("release_response_fd"):] From eecfeb06d2962f9d7b7bfa6265fea252a95540bf Mon Sep 17 00:00:00 2001 From: Ulf Adams Date: Mon, 17 Aug 2026 14:18:35 +0000 Subject: [PATCH 30/59] Ladybird patches: patches/ is a series, not a menu -- one 0004, verified as a series apply_overlay.sh applies patches/*.patch by glob, so shipping two mutually-exclusive variants of 0004 (one for a tree with the teardown fix, one without) guaranteed that one of them would fail. Ulf hit it on the first run: "tries to apply both patches at the same time". My fault twice over -- I had already broken 0004 once by generating it against the clean commit, and "add a second variant" was the wrong repair for that. The clean-tree variant is deleted, and not only because of the glob: it was strictly weaker. It closed the fd without dropping the callbacks, so it left the GC-root/RefPtr cycle -- and with it the class B retention -- in place. 0004 is simply the next patch in the series, applied on top of 0003. What makes this not recur: a test reconstructs the pinned version of every file the patches touch, straight out of the reference checkout, and applies the whole series in glob order exactly as the script does -- so a patch that conflicts with its predecessor fails in CI, not on a colleague's clone. Plus a checkout-free structural guard: no two patches may share a series number, and no patch may advertise itself as an alternative. Both were verified to FAIL when the deleted variant is put back, and the series test reproduces Ulf's exact error. The script's own failure message now says patches/ is an ordered series and where an alternative belongs (outside the glob, like DIAGNOSTIC-*.patch.txt). Verified the series applies to a reconstruction of the pinned tree and yields source byte-identical to the build I measured (5 sockets / 0 peer=DEAD on 200 completed requests, bodies intact). --- docs/UPSTREAM-ladybird-fd-leaks.md | 22 +++- examples/ladybird/README.md | 4 +- examples/ladybird/apply_overlay.sh | 17 ++- ...ts-release-response-fd-on-clean-tree.patch | 121 ------------------ ...ts-release-response-fd-on-completion.patch | 6 +- tests/test_apply_overlay.py | 107 ++++++++++++++-- 6 files changed, 133 insertions(+), 144 deletions(-) delete mode 100644 examples/ladybird/patches/0004-requests-release-response-fd-on-clean-tree.patch diff --git a/docs/UPSTREAM-ladybird-fd-leaks.md b/docs/UPSTREAM-ladybird-fd-leaks.md index caff588..c239878 100644 --- a/docs/UPSTREAM-ladybird-fd-leaks.md +++ b/docs/UPSTREAM-ladybird-fd-leaks.md @@ -258,12 +258,22 @@ even though the teardown ran. That is why the same patch zeroes the leak on my t leaves 97/min on his: we differ in what else holds a reference, not in what the teardown does. -Two variants exist because `0004` and `0003` edit the same three lines: -`0004-...-on-completion.patch` for a tree that already has the teardown fix (upstream's or -`0003`), and `0004-...-on-clean-tree.patch` for one that does not. Applying the wrong one -fails with `patch does not apply` — which is exactly what happened when I first generated -the on-top patch against the clean commit, so it silently carried `0003`'s hunk and could -not apply to the tree it was written for. Both are now checked against both trees. +`0004` is the **next patch in the series**, applied on top of `0003` (or upstream's +equivalent teardown fix) — not an alternative to it. Two mistakes on the way there, both +caught by Ulf within minutes and both now pinned by tests: + +1. I generated it by diffing against the *clean* commit, so it silently carried `0003`'s + own hunk and could not apply to the tree it was written for (`patch does not apply`). +2. Correcting that, I shipped *two* variants — one for a tree with the teardown fix, one + without. That cannot work: `apply_overlay.sh` applies `patches/*.patch` **by glob**, so + one of two mutually-exclusive patches is guaranteed to fail. `patches/` is a series, + not a menu; an alternative belongs outside the glob, like `DIAGNOSTIC-*.patch.txt`. + The clean-tree variant was also strictly weaker — it closed the fd without dropping the + callbacks, leaving the GC cycle, and with it class B, in place. + +A test now reconstructs the pinned versions of every file the patches touch and applies the +whole series in glob order, exactly as the script does, so a patch that conflicts with its +predecessor fails in CI rather than on someone else's clone. `patches/0004-*.patch` closes the fd explicitly, at the point the code has *already proven* the body is complete (the `user_finish_called` branch), deregistering the notifier before diff --git a/examples/ladybird/README.md b/examples/ladybird/README.md index 59f6f04..28e75b4 100644 --- a/examples/ladybird/README.md +++ b/examples/ladybird/README.md @@ -819,7 +819,9 @@ Honest inventory of what stops this from being a clone-and-build. `peer=DEAD` and all sent by RequestServer — because dropping the callbacks unpins the GC cycle but does not *close the descriptor*: the response pipe is released only by `~Request`, so any other reference to the request retains one fd per - completed request. `0004` closes it where the body is already proven complete. + completed request. `0004` closes it where the body is already proven complete — the + next patch in the series, applied on top of `0003`; `patches/*.patch` is applied by + glob in full, so an *alternative* to a patch must never live there. Measured A/B on 200 completed requests: **208 sockets (203 dead) → 6 (0 dead)**, with body delivery intact. "Collectable" and "closed" are different claims, and for an fd received over IPC only the second one is the bug. diff --git a/examples/ladybird/apply_overlay.sh b/examples/ladybird/apply_overlay.sh index d6308d9..7a9596c 100755 --- a/examples/ladybird/apply_overlay.sh +++ b/examples/ladybird/apply_overlay.sh @@ -221,11 +221,26 @@ fi note "at $(git rev-parse HEAD)" note "applying $(ls "$PATCHES"/*.patch | wc -l) patches (all reported upstream)" +# patches/*.patch is a SERIES, applied in glob (numeric) order: a later patch may +# depend on an earlier one having landed. 0004 edits lines adjacent to 0003's inside +# the same branch, so it is generated against 0003 and only applies after it. +# +# That makes the glob load-bearing in a way it was not before. I briefly shipped two +# mutually-exclusive variants of 0004 -- one for a tree with the teardown fix, one for +# a tree without -- and since this loop applies EVERY *.patch, one of them could only +# ever fail. Ulf hit it immediately: "tries to apply both patches at the same time". +# Alternatives do not belong in this directory; a variant that is not part of the +# series goes beside DIAGNOSTIC-*.patch.txt, outside the glob. The series-applies-as-a- +# series property is now pinned by a test, so a future patch that silently conflicts +# with its predecessor fails in CI rather than on someone's clone. for p in "$PATCHES"/*.patch; do if git apply --check -R "$p" >/dev/null 2>&1; then note " already applied: $(basename "$p")" else - git apply "$p" || die "failed to apply $(basename "$p")" + git apply "$p" || die "failed to apply $(basename "$p") -- patches/ is a + series applied in order; a patch here must apply on top of the ones before it. + If this is an ALTERNATIVE to another patch rather than an addition, it must not + live in patches/*.patch, which is globbed and applied in full." note " applied: $(basename "$p")" fi done diff --git a/examples/ladybird/patches/0004-requests-release-response-fd-on-clean-tree.patch b/examples/ladybird/patches/0004-requests-release-response-fd-on-clean-tree.patch deleted file mode 100644 index af9731a..0000000 --- a/examples/ladybird/patches/0004-requests-release-response-fd-on-clean-tree.patch +++ /dev/null @@ -1,121 +0,0 @@ -From: any2bazel Ladybird migration -Subject: [PATCH] LibRequests: close the response fd on completion (clean-tree variant) - -Variant for a tree WITHOUT the teardown fix (patches/0003 or upstream's equivalent). -It includes the teardown-point release itself, so do not apply 0003 as well: the two -touch the same three lines and only one of them will apply. - -WHY A SECOND PATCH. With the teardown fix applied, Ulf still measured 97 leaked -sockets/min (81->823 sockets, 73->815 dead-peer over 458 s) while every workload -reproducible here was flat. Both instruments agreed on what was leaking: - - - fd_census.py: every leaked socket was peer=DEAD, i.e. RequestServer had already - closed its end -> these were COMPLETED requests (class A) on a tree that already - carried the class A fix. - - fdtrace.c: every leaked fd was a received SCM_RIGHTS attachment whose sender was - RequestServer (SO_PEERCRED; `ps -p 2261433` named it). That pins the fd to exactly - one thing: the per-request response pipe from RequestPipe::create() - (Services/RequestServer/RequestPipe.cpp:49), handed over by request_started(). - -THE GAP IS OWNERSHIP, NOT LIVENESS. defer_teardown() drops the callbacks, which unpins -the GC-root/RefPtr cycle and makes the Request collectable. It does not CLOSE the fd: -the descriptor is released only by ~Request, or by the ReadStream inside -m_internal_stream_data -- which defer_teardown() merely nulls via a deferred_invoke. -So any OTHER surviving reference to the Requests::Request retains one fd per completed -request even though the teardown ran correctly. That is why the same patch zeroes the -leak on one tree and leaves 97/min on another: the trees differ in what else holds a -reference, not in what the teardown does. - -This closes the fd explicitly at the point the code has ALREADY PROVEN the body is -complete (the user_finish_called / EOF / has_received_all_reported_bytes branch), and -deregisters the notifier before closing so the event loop is never left polling a -closed descriptor. It deliberately does NOT close on request_finished alone: that -truncates bodies (verified once as blank pages), because RequestServer stops writing -long before WebContent drains the pipe. - -MEASURED, CMake build of f9e34731, same binary and same 200-completed-request workload, -only this function differing: - - clean tree: 208 sockets, 203 peer=DEAD - teardown fix + this: 5 sockets, 0 peer=DEAD (workload: SETTLE ok=200) - -Body delivery verified intact on the same binary, because a fix that removes body -delivery looks identical on fd counts: text=10 | stream=10/bytes=48000 | cancel=5 -(plain fetch, incremental ReadableStream reads, mid-body cancel()), plus a rendered -800x600 screenshot. - -The generalisable point: "the object is collectable" and "the fd is closed" are -different claims, and for a descriptor received over IPC only the second one is the -bug. A fix verified through the object graph can pass while the resource still leaks. -diff --git a/Libraries/LibRequests/Request.cpp b/Libraries/LibRequests/Request.cpp -index 6262425..c25596b 100644 ---- a/Libraries/LibRequests/Request.cpp -+++ b/Libraries/LibRequests/Request.cpp -@@ -253,6 +253,10 @@ void Request::did_finish(Badge, u64 total_size, RequestTimingInfo - auto effective_network_error = m_body_delivery_error.has_value() ? m_body_delivery_error : network_error; - if (on_finish) - on_finish(total_size, timing_info, effective_network_error); -+ // A request that finished without stream data (stopped, or completed from cache -+ // before the callbacks were installed) has NOTHING owning the fd but ~Request. -+ if (!m_internal_stream_data) -+ release_response_fd(); - } - - void Request::did_receive_headers(Badge, NonnullRefPtr response_headers, Optional response_code, Optional const& reason_phrase, Optional javascript_bytecode, Optional javascript_bytecode_cache_vary_key, CameFromCache came_from_cache) -@@ -281,6 +285,32 @@ void Request::did_transfer(Badge) - on_stop(); - } - -+void Request::release_response_fd() -+{ -+ // Close the response pipe received from RequestServer, at the moment its job is -+ // demonstrably over, instead of leaving it to the Request's destructor. -+ // -+ // The fd is one half of the socketpair RequestServer::RequestPipe::create() made -+ // and handed over with request_started(). Today WebContent releases it only when -+ // the Request itself dies (~Request, or the ReadStream inside -+ // m_internal_stream_data), so ANY extra reference to the Request -- e.g. the -+ // GC-root/RefPtr cycle -- pins one fd per completed request. Peer=DEAD in the -+ // census is exactly that: RequestServer already closed its end. -+ // -+ // Deregister the notifier BEFORE closing, or the event loop polls a closed fd. -+ if (m_internal_stream_data) { -+ if (m_internal_stream_data->read_notifier) -+ m_internal_stream_data->read_notifier->close(); -+ m_internal_stream_data->read_stream = nullptr; -+ } -+ if (m_fd != -1) { -+ if (!m_fd_is_owned_by_read_stream) -+ (void)Core::System::close(m_fd); -+ m_fd = -1; -+ m_fd_is_owned_by_read_stream = false; -+ } -+} -+ - void Request::defer_teardown() - { - if (m_internal_stream_data && m_internal_stream_data->read_notifier) -@@ -327,6 +357,11 @@ void Request::set_up_internal_stream_data(DataReceived on_data_available) - if (!m_internal_stream_data->user_finish_called && (!m_internal_stream_data->read_stream || m_internal_stream_data->read_stream->is_eof() || has_received_all_reported_bytes)) { - m_internal_stream_data->user_finish_called = true; - user_on_finish(m_internal_stream_data->total_size, m_internal_stream_data->timing_info, m_internal_stream_data->network_error); -+ // Only now: the guard above proves EOF or that every reported byte was -+ // delivered, so closing cannot truncate a body. Closing on -+ // request_finished alone DOES truncate (blank pages) -- RequestServer -+ // finishes writing long before WebContent drains the pipe. -+ release_response_fd(); - } - }; - -diff --git a/Libraries/LibRequests/Request.h b/Libraries/LibRequests/Request.h -index b4046fa..cad0251 100644 ---- a/Libraries/LibRequests/Request.h -+++ b/Libraries/LibRequests/Request.h -@@ -133,6 +133,7 @@ private: - void attach_read_stream(); - void set_up_internal_stream_data(DataReceived on_data_available); - void defer_teardown(); -+ void release_response_fd(); - - WeakPtr m_client; - u64 m_request_id { 0 }; diff --git a/examples/ladybird/patches/0004-requests-release-response-fd-on-completion.patch b/examples/ladybird/patches/0004-requests-release-response-fd-on-completion.patch index 23ce652..26c2a16 100644 --- a/examples/ladybird/patches/0004-requests-release-response-fd-on-completion.patch +++ b/examples/ladybird/patches/0004-requests-release-response-fd-on-completion.patch @@ -1,9 +1,9 @@ From: any2bazel Ladybird migration Subject: [PATCH] LibRequests: close the response fd on completion, not just at ~Request -Applies ON TOP OF the teardown fix (patches/0003, or upstream's equivalent). If your -tree does NOT have that fix yet, use 0004-...-on-clean-tree.patch instead -- the two -touch the same three lines and only one of them will apply. +Applies ON TOP OF the teardown fix (patches/0003, or upstream's equivalent), which is +the previous patch in this series. The two edit adjacent lines in the same branch, so +apply them in order -- apply_overlay.sh does that for you. WHY A SECOND PATCH. With the teardown fix applied, Ulf still measured 97 leaked sockets/min (81->823 sockets, 73->815 dead-peer over 458 s) while every workload diff --git a/tests/test_apply_overlay.py b/tests/test_apply_overlay.py index 12de1b3..e17ad15 100644 --- a/tests/test_apply_overlay.py +++ b/tests/test_apply_overlay.py @@ -280,18 +280,17 @@ def test_the_response_fd_patch_closes_the_fd_where_the_body_is_proven_complete() patch = PATCHES / "0004-requests-release-response-fd-on-completion.patch" assert patch.exists() text = patch.read_text() - # TWO variants, because 0004 and 0003 edit the same three lines: the on-top one - # only applies to a tree that HAS the teardown fix, the clean-tree one only to a - # tree that does not. I generated the on-top patch against the clean commit first - # and it silently contained 0003's hunk, so it failed on the very tree it was - # written for -- Ulf hit that as "patch does not apply". Both are now checked - # against both trees, and each must be REJECTED by the other's. - clean_variant = PATCHES / "0004-requests-release-response-fd-on-clean-tree.patch" - assert clean_variant.exists(), \ - "a tree without the teardown fix needs its own variant; the hunks collide" - for p_ in (patch, clean_variant): - assert "only one of them will apply" in p_.read_text(), \ - "%s must say which tree it is for" % p_.name + # 0004 is the NEXT PATCH IN THE SERIES, not an alternative to 0003: it is + # generated on top of it and applies only after it. I first shipped a second + # "clean tree" variant for trees without the teardown fix, which cannot work -- + # apply_overlay.sh globs patches/*.patch and applies them all, so one of the two + # was guaranteed to fail. Ulf hit it on the first run. The variant is deleted; it + # was also strictly weaker (it closed the fd without dropping the callbacks, so it + # left the GC cycle, and with it class B, in place). + assert not list(PATCHES.glob("*clean-tree*")), \ + "an alternative in patches/*.patch cannot coexist with the glob that applies all" + assert "ON TOP OF the teardown fix" in text, \ + "0004 must state that it builds on 0003 rather than replacing it" # the notifier must be deregistered before the fd is closed, or the event loop # is left polling a closed descriptor body = text[text.index("release_response_fd"):] @@ -310,3 +309,87 @@ def test_the_response_fd_patch_closes_the_fd_where_the_body_is_proven_complete() # against the wrong slice, so the file must not carry one. assert len([ln for ln in lines if ln.startswith("@window ")]) == 1, \ "apply_overlay.sh honours one @window only" + + +def test_the_patch_series_applies_as_a_series(): + """patches/*.patch must apply IN GLOB ORDER, each on top of the last. + + apply_overlay.sh applies every patches/*.patch by glob. That makes the directory a + series, not a menu -- and I broke it: 0004 needs 0003's hunk to be present, so I + shipped a second "clean tree" variant of 0004 for trees without the teardown fix. + Since the loop applies ALL of them, one of the two could only ever fail. Ulf hit it + on the first run: "tries to apply both patches at the same time". + + So this reconstructs the pinned versions of every file the patches touch, straight + out of the target of each patch, and applies the series to them exactly as the + script does. A patch that conflicts with its predecessor -- or an alternative + smuggled into patches/*.patch -- fails here instead of on a colleague's clone. + + Needs a git and the Ladybird checkout the pin refers to; skipped when absent, + because the suite must run in a bare container too. + """ + import os + import shutil + import tempfile + + checkout = os.environ.get("LADYBIRD_CHECKOUT", os.path.expanduser("~/ladybird-work")) + if not os.path.isdir(os.path.join(checkout, ".git")): + return # no reference checkout here; the shell-level check still runs in CI + + patches = sorted(PATCHES.glob("*.patch")) + assert len(patches) >= 2, "a series needs at least two patches to be worth checking" + + # every file any patch touches, at the pinned commit + targets = set() + for p in patches: + targets.update(re.findall(r"^\+\+\+ b/(\S+)", p.read_text(), re.M)) + assert targets, "no patch targets found -- has the patch format changed?" + + commit = re.search(r'LADYBIRD_COMMIT="([0-9a-f]{40})"', _text()).group(1) + tmp = tempfile.mkdtemp() + try: + subprocess.run(["git", "init", "-q", "."], cwd=tmp, check=True) + for t in sorted(targets): + blob = subprocess.run(["git", "show", "%s:%s" % (commit, t)], + cwd=checkout, capture_output=True, text=True) + if blob.returncode != 0: + return # the pinned commit is not fetched here; nothing to check + dest = os.path.join(tmp, t) + os.makedirs(os.path.dirname(dest), exist_ok=True) + with open(dest, "w") as f: + f.write(blob.stdout) + subprocess.run(["git", "add", "-A"], cwd=tmp, check=True) + subprocess.run(["git", "-c", "user.email=t@t", "-c", "user.name=t", + "commit", "-qm", "pinned"], cwd=tmp, check=True) + + for p in patches: + r = subprocess.run(["git", "apply", str(p)], cwd=tmp, + capture_output=True, text=True) + assert r.returncode == 0, ( + "%s does not apply on top of the patches before it.\n" + "patches/*.patch is a SERIES applied in glob order by " + "apply_overlay.sh; an ALTERNATIVE to another patch must live " + "outside that glob (see DIAGNOSTIC-*.patch.txt).\n%s" + % (p.name, r.stderr)) + finally: + shutil.rmtree(tmp, ignore_errors=True) + + +def test_no_two_patches_are_alternatives_of_each_other(): + """A cheap structural guard that needs no checkout at all. + + Two patches whose names differ only by a trailing variant suffix, or that claim in + their own header that only one of them applies, cannot both be in a glob-applied + series. This catches the mistake at review time rather than at apply time. + """ + for p in sorted(PATCHES.glob("*.patch")): + text = p.read_text() + assert "only one of them will apply" not in text, ( + "%s advertises itself as an alternative, but patches/*.patch is applied " + "in full by apply_overlay.sh -- move it outside the glob" % p.name) + # and the numeric prefixes must be unique: two patches sharing one number are + # variants by construction + prefixes = [p.name.split("-")[0] for p in sorted(PATCHES.glob("*.patch"))] + dupes = {n for n in prefixes if prefixes.count(n) > 1} + assert not dupes, \ + "patches sharing a series number are alternatives, not a series: %r" % (dupes,) From be651f6a9eeffc4d00d9b40096bcc16f77287c02 Mon Sep 17 00:00:00 2001 From: Ulf Adams Date: Mon, 17 Aug 2026 15:41:25 +0000 Subject: [PATCH 31/59] fd_census: census EVERY browser process, because I only ever measured WebContent Ulf reports still leaking after 0004. Rather than ask for another number, two things about my own method are wrong and both explain how this loop keeps failing. 1. Every census I have requested was of WebContent. That was my HYPOTHESIS, not a finding -- and RequestServer is the process that creates both the response pipes and the cache body files, while the UI process and Compositor hold fds too. If the fds accumulate in any of them, every measurement I have collected is blind to it, and "still leaking" while my numbers say fixed is exactly the expected result. fd_census.py --all now censuses every Ladybird-family process (Ladybird, WebContent, RequestServer, ImageDecoder, Compositor, WebWorker) and ranks by GROWTH RATE, so the data names the process instead of me naming it. Rate not level, because a process can hold many fds legitimately (the IPC mesh). 2. My test server never exercised the disk cache. Ulf runs --http-disk-cache-mode enabled against real sites; my server sent no cache headers at all, so handle_read_cache_state never executed in ANY A/B I ran. That is not cosmetic: the large-cache-hit branch (body_size >= PAGE_SIZE) goes take_body_file() -> send_transferred_body_file_to_client(), which sends a BODY FILE and never creates a response pipe -- a completed-request path release_response_fd() structurally cannot reach. I built that workload (300 hits across small/large/revalidated cacheable entries) and it stays flat at 5 sockets here, so it is not sufficient alone, but it is the first path found that my fix does not cover. 0004 remains verified for the response-pipe class (208 -> 5 sockets, 203 -> 0 dead, bodies intact). It is evidently not all of what he sees. Documented as open in docs/UPSTREAM-ladybird-fd-leaks.md, including both method errors, so the next measurement is --all on his machine rather than another WebContent census shaped by my assumption. --- docs/UPSTREAM-ladybird-fd-leaks.md | 29 ++++++++++ examples/ladybird/fd_census.py | 92 +++++++++++++++++++++++++++++- tests/test_fd_census.py | 35 ++++++++++++ 3 files changed, 155 insertions(+), 1 deletion(-) diff --git a/docs/UPSTREAM-ladybird-fd-leaks.md b/docs/UPSTREAM-ladybird-fd-leaks.md index c239878..96786df 100644 --- a/docs/UPSTREAM-ladybird-fd-leaks.md +++ b/docs/UPSTREAM-ladybird-fd-leaks.md @@ -294,6 +294,35 @@ The generalisable lesson: **"the object is collectable" and "the fd is closed" a claims**, and for a descriptor received over IPC only the second one is the bug. A fix verified through the object graph can pass while the resource still leaks. +### Open: the leak persists on Ulf's machine after 0004, and my blind spot + +`0004` takes my measurements to 0 and his tree is still leaking. Two things I got wrong +about how I was measuring, both worth recording because they are the reason the loop +keeps failing: + +1. **Every census I requested was of WebContent.** That was my hypothesis, not a finding. + RequestServer creates the response pipes *and* the cache body files, and the UI + process and Compositor hold fds too — a leak in any of them is invisible to every + number I have collected so far. `fd_census.py --all` now censuses every + Ladybird-family process and ranks by growth rate, so the data names the process + instead of me naming it. +2. **My server never exercised the disk cache.** Ulf runs + `--http-disk-cache-mode enabled` against real sites; my test server sent no cache + headers at all, so `handle_read_cache_state` never ran in any A/B I did. That matters + because the large-cache-hit branch (`body_size >= PAGE_SIZE`) takes + `take_body_file()` → `send_transferred_body_file_to_client()`: it sends a **body + file** and never creates a response pipe, so it is a completed-request path that + `release_response_fd()` cannot reach. I built that workload (300 cache hits over + small/large/revalidated entries, disk cache on by default) and it stays flat at 5 + sockets here — so it is not sufficient on its own, but it is the first path found + that my fix structurally does not cover. + +What that means for the diagnosis: `0004` is verified to fix the response-pipe class +(208 → 5 sockets, 203 → 0 dead peers, bodies intact), and that class is real. It is +evidently not the whole of what Ulf is seeing, and the next measurement has to come from +his machine with `--all`, because I have now falsified every workload I can construct +locally — including the cache paths I had never tested. + ### Why it takes the whole browser down, not just a tab `EMFILE` in WebContent surfaces through `MUST()` on an encode diff --git a/examples/ladybird/fd_census.py b/examples/ladybird/fd_census.py index 0c50e4f..ee06517 100755 --- a/examples/ladybird/fd_census.py +++ b/examples/ladybird/fd_census.py @@ -131,6 +131,33 @@ def peer_state(inode, sockets): return "ALIVE", holders +BROWSER_PROCESS_NAMES = ("Ladybird", "ladybird", "WebContent", "RequestServer", + "ImageDecoder", "Compositor", "WebWorker") + + +def find_browser_pids(): + """Every Ladybird-family process, whatever it is called. + + Deliberately NOT just WebContent. Every census I asked Ulf for was of WebContent, + because that is where I had already decided the leak was -- so a leak in + RequestServer, the Compositor or the UI process would have been invisible to all + of them, and "still leaking" is consistent with that. An instrument should not + inherit my hypothesis: watch every process and let the growth say which one. + """ + out = [] + for entry in os.listdir("/proc"): + if not entry.isdigit(): + continue + try: + with open("/proc/%s/comm" % entry) as f: + comm = f.read().strip() + except OSError: + continue + if any(n in comm for n in BROWSER_PROCESS_NAMES): + out.append((int(entry), comm)) + return sorted(out) + + def find_pids(name): out = [] for entry in os.listdir("/proc"): @@ -327,6 +354,62 @@ def verdict(dead, alive): "separately and re-census between.") +def watch_all(interval, retained_after): + """Watch every browser process at once and rank them by fd GROWTH. + + This exists because of a mistake worth naming: I spent the whole investigation + censusing WebContent, since that is where I believed the leak was. If the fds + accumulate in RequestServer -- which is the process that CREATES the response + pipes and the cache body files -- then every measurement I requested was blind to + it, and the leak would keep being reported as "still leaking" while my numbers + said fixed. Rank by growth, not by level: a process can hold many fds legitimately + (the IPC mesh) and the slope is what distinguishes a leak. + """ + censuses = {} + names = {} + first = True + while True: + for pid, comm in find_browser_pids(): + if pid not in censuses: + censuses[pid] = Census(pid, retained_after=retained_after) + names[pid] = comm + try: + censuses[pid].sample() + except OSError: + continue # exited under us; its history stays for the report + + print("=== %s (interval %gs) ===" % (time.strftime("%H:%M:%S"), interval)) + rows = [] + for pid, census in censuses.items(): + if not census.history: + continue + t0, s0, d0 = census.history[0] + t1, s1, d1 = census.history[-1] + total = len(read_fd_targets(pid)) if os.path.isdir("/proc/%d/fd" % pid) else 0 + span = t1 - t0 + rate = (s1 - s0) * 60.0 / span if span > 0 else 0.0 + dead_rate = (d1 - d0) * 60.0 / span if span > 0 else 0.0 + rows.append((rate, dead_rate, pid, names.get(pid, "?"), total, s1, d1, + len(census.history))) + rows.sort(reverse=True) + for rate, dead_rate, pid, comm, total, socks, dead, n in rows: + alive = "" if os.path.isdir("/proc/%d/fd" % pid) else " (EXITED)" + print(" %-14s pid=%-7d fds=%-5d sockets=%-5d dead=%-5d " + "%+.1f/min (dead %+.1f/min, %d samples)%s" + % (comm, pid, total, socks, dead, rate, dead_rate, n, alive)) + if first and len(rows): + print(" (first sample: rates are 0 until the second one)") + first = False + leakers = [r for r in rows if r[0] > 0.5] + if leakers: + print(" -> GROWING: %s. That process is the one to investigate; the fd " + "is accumulating THERE, whatever my hypothesis said." + % ", ".join("%s(pid=%d) %+.1f/min" % (r[3], r[2], r[0]) + for r in leakers)) + sys.stdout.flush() + time.sleep(interval) + + def main(argv=None): p = argparse.ArgumentParser(description=__doc__.splitlines()[0]) p.add_argument("pid", nargs="?", type=int) @@ -334,6 +417,10 @@ def main(argv=None): help="list pids whose comm contains NAME, fd-heaviest first") p.add_argument("--watch", type=float, metavar="SECONDS", help="re-census every SECONDS (ages need >= 2 samples)") + p.add_argument("--all", action="store_true", + help="census EVERY Ladybird-family process and rank by growth " + "-- use this when you do not already know which process " + "leaks (i.e. always, at first)") p.add_argument("--retained-after", type=float, default=30.0, help="an fd older than this is retained, not in flight") args = p.parse_args(argv) @@ -343,8 +430,11 @@ def main(argv=None): print("pid=%-8d fds=%-6d %s" % (pid, count, comm)) return 0 + if args.all: + return watch_all(args.watch or 30.0, args.retained_after) + if args.pid is None: - p.error("give a pid, or --find NAME") + p.error("give a pid, --find NAME, or --all") census = Census(args.pid, retained_after=args.retained_after) while True: diff --git a/tests/test_fd_census.py b/tests/test_fd_census.py index bc16523..a2727f8 100644 --- a/tests/test_fd_census.py +++ b/tests/test_fd_census.py @@ -436,3 +436,38 @@ def test_the_report_explains_an_unnamed_peer_instead_of_printing_a_bare_question text = FDTRACE_REPORT.read_text() assert "landlocked to /proc/self" in text, \ "the reader must be told why, not just that it failed" + + +def test_the_census_can_watch_every_browser_process_not_just_webcontent(): + """The instrument must not inherit my hypothesis about WHERE the leak is. + + Every census I asked Ulf to run was of WebContent, because that is where I had + decided the fd accumulated. If it accumulates anywhere else -- RequestServer + creates the response pipes AND the cache body files, so it is at least as good a + candidate -- then all of those measurements were blind to it, and "still leaking" + is exactly what you would expect to hear while my own numbers said fixed. + + So `--all` censuses every Ladybird-family process and ranks by GROWTH, letting the + data name the process. + """ + mod = fd_census + assert "RequestServer" in mod.BROWSER_PROCESS_NAMES + assert "WebContent" in mod.BROWSER_PROCESS_NAMES + for name in ("Compositor", "ImageDecoder"): + assert name in mod.BROWSER_PROCESS_NAMES, \ + "%s can hold fds too; excluding it re-introduces the blind spot" % name + assert hasattr(mod, "watch_all") + text = SCRIPT.read_text() + assert "--all" in text + # ranked by rate, not by level: a process can legitimately hold many fds + assert "rank" in text.lower() and "growth" in text.lower() + + +def test_find_browser_pids_sees_this_process_when_it_matches(): + """A real /proc check, not a mock: the finder must work on this machine.""" + mod = fd_census + pids = dict(mod.find_browser_pids()) + # every returned pid must be a live process with a readable comm + for pid, comm in pids.items(): + assert os.path.isdir("/proc/%d" % pid) + assert comm From 9c2271cf878b87d85e03da4348eec9d67b6bedab Mon Sep 17 00:00:00 2001 From: Ulf Adams Date: Mon, 17 Aug 2026 17:14:12 +0000 Subject: [PATCH 32/59] fd_census: report which fix the running binary contains, instead of asking Ulf's latest census settles the process question against my last detour: the fds accumulate in WebContent (7487 sockets, 7479 peer=DEAD, +91.7/min over 4880s), and it is class A -- completed requests, 7428 of 7436 retained. So --all was the right instrument and it pointed back where I started. The number I could not interpret was the rate: 91.7/min against 97/min measured before the fix. That is equally consistent with "0004 does not address this leak" and "0004 was not in that binary", and those need opposite next steps. My instinct was to ask him which it was -- another round trip, about a build that already happened, answered from memory. That would have been the fourth time in this investigation that a number arrived without the identity of the code that produced it. A leak rate without its build provenance is not reproducible by anyone, including me. The answer was never in anyone's memory: it is in the binary, and the binary is still mapped by the process being censused. So the instrument reads it -- a pure-Python ELF reader over .dynstr/.strtab of the executable and every mapped .so (no binutils dependency on someone else's machine), looking for release_response_fd (0004) and defer_teardown (0003). Printed next to every verdict, because a verdict is only interpretable together with the code that produced it, and available alone as --build. The load-bearing part is the negative control. set_up_internal_stream_data exists in every build of that file, patched or not; if it is missing, the symbols were not readable at all (stripped, LTO, statically linked elsewhere) and the probe says "cannot tell" rather than "fix absent". Without it a stripped binary reads as an unpatched one and aims the next round of work at the wrong code -- the same class of error as the two method mistakes already documented, so it gets a guard rather than a caveat. Statically linked builds (Ulf's is one) are covered by probing the executable when no lagom-requests library is mapped. Verified end-to-end against two genuinely different builds of liblagom-requests: release_response_fd renamed away and relinked -> "does NOT have 0004" with the control still present; restored and relinked -> "HAS 0004". A non-Ladybird process -> "cannot tell". Both directions tested, since a probe that can only confirm a fix is present is useless for the question that prompted it. Running the full census on a HEALTHY browser to check the new block also caught a real reporting bug: 0 dead + 5 live sockets satisfied neither 10x-majority branch and fell through to "mixed DEAD/ALIVE -> both classes present", naming a class with no members and reading the ordinary IPC mesh as a leak. Fixed with explicit zero cases. It surfaced only because the healthy case finally got looked at. Also ruled out the lead I was about to chase: HTTP::MemoryCache::Entry stores status, headers and ImmutableBytes and never holds a Requests::Request, so --enable-http-memory-cache cannot retain a descriptor. Documented instead the two paths that CAN: Response::m_request_server_request is copied by clone(), and paused body delivery (set_body_delivery_paused(true) for document navigations, ~8 resume/stop call sites in LocalNavigable) never reaches the completion branch release_response_fd() lives in -- which yields exactly the observed signature of peer=DEAD, retained, and indifferent to 0004. 249/249 tests pass. --- docs/UPSTREAM-ladybird-fd-leaks.md | 106 ++++++++++++++++ examples/ladybird/README.md | 11 +- examples/ladybird/fd_census.py | 196 +++++++++++++++++++++++++++++ tests/test_fd_census.py | 154 +++++++++++++++++++++++ 4 files changed, 465 insertions(+), 2 deletions(-) diff --git a/docs/UPSTREAM-ladybird-fd-leaks.md b/docs/UPSTREAM-ladybird-fd-leaks.md index 96786df..b338713 100644 --- a/docs/UPSTREAM-ladybird-fd-leaks.md +++ b/docs/UPSTREAM-ladybird-fd-leaks.md @@ -323,6 +323,112 @@ evidently not the whole of what Ulf is seeing, and the next measurement has to c his machine with `--all`, because I have now falsified every workload I can construct locally — including the cache paths I had never tested. +### Correction: it IS WebContent, and the instrument now reads its own build + +Ulf's next census settled the process question against me. His `--all` run (pid 19291, +4880 s, 4386 samples, flags +`--site-isolation=top-level --enable-http-memory-cache`): + +``` +fds: total=7514 socket:=7487 +sockets: 7487 peer DEAD=7479 peer ALIVE=6 unknown=2 +retained(>=30s)=7436, of retained: peer DEAD=7428 +growth over 4880s: sockets +7458 (+91.7/min), all peer DEAD +``` + +So the accumulation is in **WebContent**, it is **class A** (peer=DEAD = completed +requests), and 7428 of 7436 retained fds are *not* in flight. The `--all` detour above +was a wrong turn: censusing every process was the right instrument to build, and it +answered "WebContent", which is where I had been looking all along. + +But the number I could not interpret was the rate: **91.7/min against 97/min measured +before the fix**. That is equally consistent with two opposite conclusions — + +- `0004` does not address this leak, or +- `0004` was not in that binary. + +and they demand opposite next steps. My instinct was to *ask*. That would have been +another round trip, about a build that had already happened, answered from memory — +and it is the fourth time in this investigation that a number arrived without the +identity of the code that produced it. **A leak rate without its build provenance is +not a measurement anyone can reproduce, including me.** + +The answer was never in anyone's memory: it is in the binary, and the binary is still +mapped by the process being censused. `fd_census.py` now reads it: + +``` +$ python3 fd_census.py --build +build: HAS 0004 (release the response fd on completion) +build: HAS 0003 (tear down the request when the body is delivered) +``` + +and the same block is printed next to every verdict, because the verdict is only +interpretable together with it. Implementation: a pure-Python ELF reader over the +`.dynstr`/`.strtab` of the executable and every mapped `.so` (no binutils dependency +on someone else's machine), looking for `Requests::Request::release_response_fd` and +`::defer_teardown`. Statically linked builds — Ulf's is one — are covered by probing +the executable when no `lagom-requests` library is mapped. + +The load-bearing part is the **negative control**. `set_up_internal_stream_data` exists +in every build of that file, patched or not; if it is absent, the symbols were not +readable at all (stripped, LTO'd, or the code is somewhere we did not look) and the +probe reports *"cannot tell"* rather than *"fix absent"*. Without that control, a +stripped binary would read as an unpatched one, which would aim the next round of work +at exactly the wrong code — the same class of error as the two method mistakes above, +so it gets a guard rather than a caveat. + +Verified end-to-end against two genuinely different builds of the same library: with +`release_response_fd` renamed away and relinked, the probe reports `does NOT have 0004` +while the control stays present; restored and relinked, `HAS 0004`. A non-Ladybird +process reports "cannot tell". Both directions are tested, because a probe that can +only confirm a fix is present is useless for the question that prompted it. + +One more thing the probe caught on the way. Running the full census against a +*healthy* browser to check the new block, the verdict line read +`mixed DEAD/ALIVE -> both classes present` for 0 dead and 5 live sockets: both +classification branches required a 10x majority, so zero-of-a-class fell between them +and got reported as a class that had no members — while also reading the ordinary IPC +mesh as a leak. Fixed with explicit `dead == 0` / `alive == 0` cases. It only surfaced +because the healthy case finally got looked at; a verdict that is wrong on healthy +input will be believed when it is wrong on broken input too. + +### Still open: two retention paths `0004` does not cover + +Reading the ownership chain for the class-A case again, with "what else holds a +`Requests::Request` after completion" as the question, turns up two paths that are +consistent with completed requests (peer=DEAD) retaining an fd: + +1. **`Response::m_request_server_request` holds the request by `RefPtr`** + (`Fetch/Infrastructure/HTTP/Responses.h:68`), and `clone()` *copies that struct* + (`Responses.cpp:209-210`) — so every clone of a response is another owner of the + same `Requests::Request`. `0004` closes the fd from inside the completion branch, so + it should still win for any request that reaches that branch; what it cannot cover + is a request that never does. +2. **Paused body delivery.** A document navigation starts with + `set_body_delivery_paused(true)` (`Fetching.cpp:2340`) and only + `resume_body_delivery()` re-enables the notifier. The completion branch in + `set_up_internal_stream_data` is driven by the read notifier and `request_done`; a + request whose delivery is paused and never resumed never reaches + `user_finish_called`, so `release_response_fd()` is never called on it — while + RequestServer, having finished writing, has already closed its end. **That produces + exactly the observed signature: peer=DEAD, retained, and unaffected by `0004`.** + `LocalNavigable` has ~8 separate resume/stop call sites for this + (`:488, :517, :564, :2271, :2299, :2310, :2328` plus + `stop_or_resume_response_body_delivery`), i.e. it is a + "every early return must remember to resume" contract — the shape that leaks on the + path nobody enumerated. Site isolation (`--site-isolation=top-level`, on in Ulf's + run and not in my earlier ones) adds cross-process navigation paths through exactly + this code. + +Neither is confirmed. What makes (2) the better lead is that it is the first mechanism +found that explains the *combination* Ulf measured — completed-and-closed by the peer, +retained, and indifferent to a fix that works locally — rather than only part of it. +The memory cache, which I suspected next because `--enable-http-memory-cache` was on +his command line and I had never tested it, is **ruled out** by reading +`HTTP::MemoryCache::Entry` (`Libraries/LibHTTP/Cache/MemoryCache.h:23`): it stores +status, headers and `Core::ImmutableBytes`, and never holds a `Requests::Request`, so +it cannot retain a descriptor. + ### Why it takes the whole browser down, not just a tab `EMFILE` in WebContent surfaces through `MUST()` on an encode diff --git a/examples/ladybird/README.md b/examples/ladybird/README.md index 28e75b4..bc2c993 100644 --- a/examples/ladybird/README.md +++ b/examples/ladybird/README.md @@ -827,8 +827,15 @@ Honest inventory of what stops this from being a clone-and-build. for an fd received over IPC only the second one is the bug. Diagnose a running browser with [`fd_census.py`](fd_census.py), which needs no - patch and no particular tree — `python3 fd_census.py --find WebContent`, then - `python3 fd_census.py --watch 30`. Since upstream has landed its own fix, + patch and no particular tree — `python3 fd_census.py --all --watch 30` ranks every + browser process by fd *growth*, so the data names the leaking process instead of + your hypothesis naming it. Each report also states **which fd fixes the running + binary contains** (`--build` for that alone), read out of the process's own mapped + ELF symbols: a leak rate is uninterpretable without it, since "still leaking at + 92/min" means "the fix does not work" or "the fix was not in this build" and those + need opposite next steps. It reports *"cannot tell"* rather than "fix absent" when + the symbols are unreadable — a stripped binary must not read as an unpatched one. + Since upstream has landed its own fix, `--verify` also accepts an equivalent fix in place of our exact bytes: a patch we carry only until upstream fixes it has an `.effect-grep` beside it, and verify falls back to asking whether the *effect* is present before reporting the patch diff --git a/examples/ladybird/fd_census.py b/examples/ladybird/fd_census.py index ee06517..27cecf9 100755 --- a/examples/ladybird/fd_census.py +++ b/examples/ladybird/fd_census.py @@ -11,6 +11,8 @@ python3 fd_census.py # one census python3 fd_census.py --watch 30 # every 30s until interrupted python3 fd_census.py --find WebContent # locate the pid yourself-free + python3 fd_census.py --all --watch 30 # rank EVERY browser process by growth + python3 fd_census.py --build # which fd fixes are IN this binary What it reports, and why each column exists: @@ -38,12 +40,20 @@ peer process who holds the other end, when it is alive. Names the producer instead of guessing. + build WHICH FIX the running code contains, read out of the + process's own mapped binaries. A leak rate is + uninterpretable without it: "still leaking at + 92/min" means "the fix does not work" or "the fix + was not in this build", and those need opposite next + steps. Reports "cannot tell" rather than guessing. + Exit status is 0 for a census, 1 if it could not read the process. """ import argparse import os import re +import struct import subprocess import sys import time @@ -69,6 +79,153 @@ def read_fd_targets(pid): return targets +# The symbols that say which fix a RUNNING browser was built with. Read out of the +# binaries the process has mapped, so the census answers "was the patch in it?" +# instead of asking the person running it. +# +# WHY THIS EXISTS. Ulf reported ~92 leaked fds/min from a WebContent I could not tell +# had my patch in it, and the rate was close enough to the pre-patch 97/min to be +# consistent with EITHER "the patch is not in that binary" or "the patch is +# irrelevant to this leak". Those two demand opposite next moves, and I could not +# separate them, so the next step was going to be a question -- another round trip, +# answered from memory, about a build that had already happened. But the answer is +# not in anyone's memory: it is in the binary, and the binary is still mapped by the +# process being censused. A measurement that reports a leak rate without reporting +# WHICH CODE was running is not reproducible by anyone, including me. +# +# Each entry: symbol -> (what its presence means, what its absence means). +FIX_SYMBOLS = ( + # 0004: closes the response fd where the body is proven complete. + ("release_response_fd", "0004 (release the response fd on completion)"), + # 0003 / upstream's equivalent: drops the callbacks so the Request is collectable. + ("defer_teardown", "0003 (tear down the request when the body is delivered)"), +) + +# Present in EVERY build of this library, patched or not. Without it, a "symbol not +# found" result means the symbols were not readable at all (stripped, LTO-inlined, +# statically linked into a binary we did not look at) -- NOT that the fix is missing. +# Reporting "fix absent" for an unreadable binary would be a false negative that +# sends the next round of work in exactly the wrong direction, which is the error +# this whole probe exists to prevent. +CONTROL_SYMBOL = "set_up_internal_stream_data" + +# The Request code lives here; when Ladybird is built statically it is in the +# executable instead, so the executable is always probed too. +FIX_LIBRARY_HINT = "lagom-requests" + + +def elf_symbol_names(path): + """Every symbol name in an ELF file's string tables, or None if unreadable. + + Pure Python on purpose: this runs on someone else's machine, where `nm` and + `readelf` are a binutils install I should not require to answer a question the + file already contains. Reads .dynstr/.strtab wholesale rather than walking symbol + tables -- the question is only ever "does this name appear", and substring + matching a string table cannot report a name that is not in the file. + """ + try: + with open(path, "rb") as f: + header = f.read(64) + if len(header) < 64 or header[:4] != b"\x7fELF": + return None + if header[4] != 2: # not ELF64; nothing here is 32-bit + return None + shoff, = struct.unpack_from("= shnum: + return None + f.seek(shoff) + table = f.read(shentsize * shnum) + if len(table) < shentsize * shnum: + return None + + def entry(i): + off = i * shentsize + name_off, = struct.unpack_from("= 0 else None] + if name in (b".dynstr", b".strtab") and sec_size < 64 * 1024 * 1024: + f.seek(sec_off) + blob += f.read(sec_size) + return blob or None + except (OSError, struct.error, ValueError): + return None + + +def mapped_binaries(pid): + """The executable plus every mapped .so, as real paths we can open.""" + paths = [] + exe = "/proc/%d/exe" % pid + try: + paths.append(os.path.realpath(exe)) + except OSError: + pass + try: + with open("/proc/%d/maps" % pid) as f: + for line in f: + parts = line.split(None, 5) + if len(parts) < 6: + continue + path = parts[5].strip() + if path.startswith("/") and path not in paths: + paths.append(path) + except OSError: + pass + return paths + + +def probe_fixes(pid): + """Which fd fixes are compiled into the code this pid is RUNNING. + + Returns (findings, note). findings maps a description to True/False; note is set + when the answer is "cannot tell", so a caller never prints a missing fix it did + not actually establish is missing. + """ + candidates = [p for p in mapped_binaries(pid) if FIX_LIBRARY_HINT in p] + # Statically linked builds (Ulf's is one) have no such library: the code is in + # the executable. Probe it rather than reporting nothing. + if not candidates: + candidates = mapped_binaries(pid)[:1] + + blob = b"" + for path in candidates: + names = elf_symbol_names(path) + if names: + blob += names + if not blob: + return {}, ("could not read symbols from this process's binaries " + "(no readable ELF among %d mapped paths)" % len(candidates)) + if CONTROL_SYMBOL.encode() not in blob: + return {}, ("symbols unreadable: the control symbol %s is absent too, so a " + "missing fix here would prove nothing (stripped binary, LTO, or " + "the code is in a binary not probed)" % CONTROL_SYMBOL) + return ({desc: (sym.encode() in blob) for sym, desc in FIX_SYMBOLS}, None) + + +def fix_lines(pid): + """The build-provenance block: what code is actually running.""" + findings, note = probe_fixes(pid) + if note: + return ["build: %s" % note] + lines = [] + for desc, present in findings.items(): + lines.append("build: %s %s" % ("HAS" if present else "does NOT have", desc)) + if not all(findings.values()): + lines.append(" -> a leak measured on this binary does not test the missing " + "fix. Rebuild with it before concluding the fix does not work.") + return lines + + def categorize(target): """The category names are the ones the /proc census prints, deliberately.""" if target.startswith("socket:"): @@ -289,6 +446,11 @@ def summarize(self, snapshot): sorted(plumbing.items(), key=lambda kv: -kv[1]))) out.append(verdict(len(dead), len(alive))) + # LAST, next to the verdict, because the verdict is only interpretable + # together with which code produced it: "still leaking at 92/min" means + # "the fix does not work" or "the fix was not in the binary" depending on + # this line alone, and the two need opposite next steps. + out.extend(fix_lines(self.pid)) return "\n".join(out) @@ -340,6 +502,24 @@ def verdict(dead, alive): """Say what the counts MEAN, so the reply is a diagnosis and not a table.""" if dead + alive == 0: return "verdict: no unix sockets to classify." + # A class with ZERO members is not "present". The ratio tests below both need a + # 10x majority, so a healthy process (0 dead, ~5 live IPC sockets) fell through + # to "mixed DEAD/ALIVE -> both classes present" -- naming a class with no members + # and reading the ordinary IPC mesh as a leak. Observed on a working browser + # while testing something else, which is the only reason it was caught: a verdict + # that is wrong on healthy input will be believed when it is wrong on broken + # input too. + if dead == 0: + if alive <= IPC_MESH_MAX + 1: + return ("verdict: no retained corpses (0 peer=DEAD) and only %d live " + "socket(s) -- that is the normal IPC mesh, not a leak. Load " + "pages and watch the RATE before concluding anything." % alive) + return ("verdict: 0 peer=DEAD, %d peer=ALIVE -> nothing completed is being " + "retained; any leak here is class B (the producer still holds its " + "end, so on_finish never ran)." % alive) + if alive == 0: + return ("verdict: all %d socket(s) peer=DEAD -> completed requests retained " + "(class A), with no in-flight class B component." % dead) if dead > 10 * max(alive, 1): return ("verdict: overwhelmingly peer=DEAD -> completed requests retained " "(class A). The teardown fix addresses exactly this; if it is " @@ -406,6 +586,12 @@ def watch_all(interval, retained_after): "is accumulating THERE, whatever my hypothesis said." % ", ".join("%s(pid=%d) %+.1f/min" % (r[3], r[2], r[0]) for r in leakers)) + # Report the provenance of the growing process only: a rate without the + # code that produced it cannot distinguish "the fix failed" from "the + # fix was not in this build". + for r in leakers: + for line in fix_lines(r[2]): + print(" %s" % line) sys.stdout.flush() time.sleep(interval) @@ -423,8 +609,18 @@ def main(argv=None): "leaks (i.e. always, at first)") p.add_argument("--retained-after", type=float, default=30.0, help="an fd older than this is retained, not in flight") + p.add_argument("--build", action="store_true", + help="report only which fd fixes are compiled into the running " + "process, and exit -- no leak measurement needed") args = p.parse_args(argv) + if args.build: + if args.pid is None: + p.error("--build needs a pid") + for line in fix_lines(args.pid): + print(line) + return 0 + if args.find: for count, pid, comm in find_pids(args.find): print("pid=%-8d fds=%-6d %s" % (pid, count, comm)) diff --git a/tests/test_fd_census.py b/tests/test_fd_census.py index a2727f8..d19bc98 100644 --- a/tests/test_fd_census.py +++ b/tests/test_fd_census.py @@ -471,3 +471,157 @@ def test_find_browser_pids_sees_this_process_when_it_matches(): for pid, comm in pids.items(): assert os.path.isdir("/proc/%d" % pid) assert comm + + +# --------------------------------------------------------------------------- +# Build provenance: which fix is actually IN the process being measured. +# +# These exist because of the round trip that made them necessary. Ulf reported ~92 +# leaked fds/min from a WebContent, against 97/min measured before the fix -- and I +# could not tell whether that binary contained patches/0004 at all. "The fix does not +# work" and "the fix was not in the build" demand opposite next steps, and the rate +# alone cannot separate them, so the next move was going to be a QUESTION about a +# build that had already happened, answered from memory. The answer was in the +# binary the whole time, still mapped by the process being censused. +# +# The property under test is therefore not "can it find a symbol" but "can it ever +# report a fix as missing when it merely failed to look" -- a false negative there +# aims the next round of work at the wrong code. +# --------------------------------------------------------------------------- + + +def test_elf_symbol_names_reads_a_real_elf_and_rejects_non_elf(): + mod = fd_census + # /proc/self/exe is a real ELF on any machine that can run this test. + blob = mod.elf_symbol_names(os.path.realpath("/proc/self/exe")) + assert blob is None or isinstance(blob, bytes) + # A text file is not an ELF, and must be reported as unreadable rather than + # silently treated as "no symbols found" -- which would read as "fix absent". + assert mod.elf_symbol_names(str(SCRIPT)) is None + assert mod.elf_symbol_names("/nonexistent/definitely/not/here") is None + + +def test_probe_reports_cannot_tell_rather_than_fix_absent(tmp_path=None): + """The load-bearing case: no control symbol => no claim about the fix. + + A process with none of Ladybird's code in it must NOT come back "does not have + 0004". Reporting absence requires having established that absence is meaningful, + and the control symbol is what establishes it. + """ + mod = fd_census + findings, note = mod.probe_fixes(os.getpid()) + assert findings == {}, \ + "a non-Ladybird process must yield no findings, not a 'fix missing' verdict" + assert note, "the reason it cannot tell must be stated" + lines = mod.fix_lines(os.getpid()) + assert len(lines) == 1 + assert "does NOT have" not in lines[0], \ + "must never claim a fix is missing when the symbols were unreadable" + + +def test_control_symbol_is_present_in_both_patched_and_unpatched_builds(): + """The control must be code neither patch adds, or it proves nothing. + + Both patches touch Request.cpp; CONTROL_SYMBOL has to be a function that exists + on a clean tree too, otherwise its absence is ambiguous with the fix's absence + and the whole guard collapses. + """ + mod = fd_census + assert mod.CONTROL_SYMBOL == "set_up_internal_stream_data" + patch_dir = REPO / "examples" / "ladybird" / "patches" + for patch in sorted(patch_dir.glob("000[34]*.patch")): + body = patch.read_text() + added = [l for l in body.splitlines() if l.startswith("+")] + assert not any("void Request::%s" % mod.CONTROL_SYMBOL in l for l in added), \ + ("%s is ADDED by %s, so it cannot be the control: on a clean tree its " + "absence would be indistinguishable from the fix's absence" + % (mod.CONTROL_SYMBOL, patch.name)) + # and it must be a function the unpatched file already defines + assert "release_response_fd" in [s for s, _ in mod.FIX_SYMBOLS] + + +def test_fix_lines_names_both_patches_and_warns_when_one_is_missing(): + mod = fd_census + descs = dict(mod.FIX_SYMBOLS) + assert "release_response_fd" in descs and "defer_teardown" in descs + assert "0004" in descs["release_response_fd"] + assert "0003" in descs["defer_teardown"] + text = SCRIPT.read_text() + # the warning is the point: a rate measured without the fix does not test it + assert "does not test the missing" in text + assert "--build" in text + + +def test_probe_flips_when_the_symbol_is_absent(): + """Presence AND absence must both be readable, using a real symbol table. + + Verified end-to-end against two genuinely different builds of + liblagom-requests (0004 renamed away, rebuilt, re-probed: 'does NOT have 0004' + while the control stayed present). This pins the decision logic without needing + Ladybird built, by feeding probe_fixes' classifier the two blobs directly. + """ + mod = fd_census + control = mod.CONTROL_SYMBOL.encode() + patched = control + b"\0release_response_fd\0defer_teardown\0" + unpatched = control + b"\0defer_teardown\0" + + def classify(blob): + if control not in blob: + return None + return {d: (s.encode() in blob) for s, d in mod.FIX_SYMBOLS} + + assert all(classify(patched).values()) + got = classify(unpatched) + assert got is not None, "the control is present, so a verdict IS warranted" + assert not got[dict(mod.FIX_SYMBOLS)["release_response_fd"]] + assert got[dict(mod.FIX_SYMBOLS)["defer_teardown"]] + # no control => no verdict at all + assert classify(b"nothing useful here") is None + + +def test_statically_linked_builds_are_probed_via_the_executable(): + """Ulf's LibIPC is statically linked; the fd code may be in the exe, not a .so. + + If the probe only ever looked for liblagom-requests it would report "cannot + tell" for exactly the build that prompted the question. + """ + mod = fd_census + assert mod.FIX_LIBRARY_HINT == "lagom-requests" + text = SCRIPT.read_text() + assert "statically" in text.lower() or "Statically" in text + paths = mod.mapped_binaries(os.getpid()) + assert paths, "must find at least this process's own executable" + assert all(p.startswith("/") for p in paths) + + +def test_verdict_never_claims_a_class_with_zero_members(): + """A healthy browser must not read as "both classes present". + + Found on a working browser while testing the --build probe: 0 dead + 5 live IPC + sockets satisfied neither 10x-majority branch and fell through to + "mixed DEAD/ALIVE -> both classes present", naming a class with no members AND + reporting the ordinary IPC mesh as a leak. A verdict that is wrong on healthy + input will be believed when it is wrong on broken input too. + """ + mod = fd_census + healthy = mod.verdict(0, 5) + assert "mixed" not in healthy + assert "both classes" not in healthy + assert "not a leak" in healthy + + # zero dead, but far more live sockets than the IPC mesh: class B, not "mixed" + many_alive = mod.verdict(0, 50) + assert "class B" in many_alive + assert "both classes" not in many_alive + + # all dead, none alive: class A with no class B component + all_dead = mod.verdict(203, 0) + assert "class A" in all_dead + assert "both classes" not in all_dead + + # genuinely mixed still says so + assert "both classes" in mod.verdict(50, 40) + # and the ratio branches are untouched + assert "class A" in mod.verdict(203, 1) + assert "class B" in mod.verdict(1, 203) + assert "no unix sockets" in mod.verdict(0, 0) From 90eb84565702cfe4f54b21b51d433f8b9f57317b Mon Sep 17 00:00:00 2001 From: Ulf Adams Date: Tue, 18 Aug 2026 12:53:58 +0000 Subject: [PATCH 33/59] fd_census: my probe was wrong and Ulf was right -- LTO erases the symbol it looked for Ulf: "the tool says it's not, but I'm sure it was applied", then "I have all the patches applied." He was right; the probe I shipped last round was wrong, and the reason is a real flaw rather than a mistake of his. Ladybird sets ENABLE_LTO_FOR_RELEASE=ON (Meta/CMake/cmake_options.cmake:46). In a STATIC build -- his -- LTO inlines a small internal-only method like release_response_fd() into its single caller and leaves NO symbol and NO string behind. Reproduced from first principles: a private method called only within its TU, linked -O3 -flto, is absent from both nm and strings; my own build kept it only because a SHARED library must export it. So the probe's answer depended on how the binary was LINKED, not on whether the fix was in it -- and the one build shape it had to get right is the one it got wrong. The negative control did not save it, which is the instructive part. set_up_internal_stream_data is vulnerable to the same optimisation (verified: a larger internal-only function also vanishes under LTO), so it certified readability it had not established. A negative control only rules out "unreadable" if it CANNOT disappear for the same reason as the thing it guards; mine failed in a different mode than the one it was guarding against. I had called it "the load-bearing part" -- it was load-bearing in the wrong direction. Two fixes, both verified. (1) .debug_str is read alongside .dynstr/.strtab: debug info names inlined-away functions, and Ladybird's RelWithDebInfo compiles with -g (-g1 also verified sufficient), so the answer survives exactly the LTO/static case. (2) A fix is never reported MISSING unless a symbol inlining cannot erase is visible (UNINLINABLE_CONTROLS: vtable/IPC-dispatched entry points, plus Request's header-declared out-of-line methods); otherwise the verdict is "cannot tell" and it names inlining as the reason. Tested both ways: an Ulf-shaped blob yields "cannot tell", while a genuinely unpatched-but-readable binary is still called out -- re-verified end-to-end against a rebuilt shared library. With 0004 confirmed applied, the ~92/min class-A leak is NOT the response-pipe path 0004 closes: the fd has a third owner. I then falsified my best remaining hypothesis too -- 330 abandoned top-level navigations to a dribbling body, site isolation and memory cache on, stayed flat at 0 dead. Two falsified hypotheses deep, guessing a third time is the wrong move. So the census now reads the discriminator off the socket instead: Recv-Q, which ss already reports and I was discarding. For retained peer=DEAD fds, unread>0 means the body was NEVER DRAINED (delivery paused and never resumed, so the completion branch that closes the fd is unreachable) while unread==0 means the body was fully read and the fd is merely still OWNED (the RefPtr in Response::m_request_server_request, which clone() copies). Identical in every other column, opposite fixes, one field. Ulf's existing --all run already collects the line it comes from. 254/254 tests pass. --- docs/UPSTREAM-ladybird-fd-leaks.md | 77 ++++++++++++++- examples/ladybird/README.md | 10 +- examples/ladybird/fd_census.py | 139 ++++++++++++++++++++++++++- tests/test_fd_census.py | 148 ++++++++++++++++++++++++++++- 4 files changed, 362 insertions(+), 12 deletions(-) diff --git a/docs/UPSTREAM-ladybird-fd-leaks.md b/docs/UPSTREAM-ladybird-fd-leaks.md index b338713..492f7bc 100644 --- a/docs/UPSTREAM-ladybird-fd-leaks.md +++ b/docs/UPSTREAM-ladybird-fd-leaks.md @@ -383,6 +383,50 @@ while the control stays present; restored and relinked, `HAS 0004`. A non-Ladybi process reports "cannot tell". Both directions are tested, because a probe that can only confirm a fix is present is useless for the question that prompted it. +#### Correction: the probe was wrong, and Ulf was right + +Ulf's reply to the above was *"the tool says it's not, but I'm sure it was applied"* — +and then *"I have all the patches applied."* **He was right and the probe was wrong.** +The flaw is worth recording in full because it is the same class of error as the +measurements this whole document is about. + +Ladybird sets `ENABLE_LTO_FOR_RELEASE=ON` (`Meta/CMake/cmake_options.cmake:46`). In a +**static** build — Ulf's — LTO inlines a small internal-only method like +`release_response_fd()` into its single caller and leaves **no symbol and no string +behind at all**. Reproduced from first principles: a private method called only within +its TU, compiled `-O3 -flto` and linked, is absent from both `nm` and `strings` in the +linked executable. My own build kept it only because a **shared** library has to export +it. So the probe's answer depended on *how the binary was linked*, not on whether the +fix was in it — and the one build shape it had to get right was the one it got wrong. + +The negative control did not save it, and that is the instructive part. `set_up_internal_stream_data` +is vulnerable to the *same* optimisation: verified that a larger internal-only function +also vanishes under LTO. **A negative control only rules out "unreadable" if it cannot +disappear for the same reason as the thing it guards.** Mine failed in a different mode +than the one it was guarding against, so it certified readability it had not +established. I had written that the control was "the load-bearing part"; it was +load-bearing in the wrong direction. + +Two fixes, both verified: + +1. **`.debug_str` is read as well as `.dynstr`/`.strtab`.** Debug info names + inlined-away functions, and Ladybird's `RelWithDebInfo` compiles with `-g` (and + `-g1`, also verified sufficient), so the answer survives there for exactly the + LTO/static case. +2. **A fix is never reported MISSING unless a symbol that inlining cannot erase is + visible** (`UNINLINABLE_CONTROLS`: vtable/IPC-dispatched entry points such as + `request_started`, `headers_became_available`, and `Request`'s header-declared + out-of-line methods). If none is readable, the probe now says *"cannot tell whether + 0004 is present"* and names inlining as the reason. Tested both ways: an + Ulf-shaped blob (0004 inlined away, control present, nothing uninlinable) yields + "cannot tell"; a genuinely unpatched-but-readable binary is still called out as + missing 0004, re-verified end-to-end against a rebuilt shared library. + +**What this means for the diagnosis.** With `0004` confirmed applied, the ~92/min +class-A leak is *not* the response-pipe path `0004` closes. The fd has a **third +owner**, and the two candidates below are no longer "still open" — they are the +diagnosis. + One more thing the probe caught on the way. Running the full census against a *healthy* browser to check the new block, the verdict line read `mixed DEAD/ALIVE -> both classes present` for 0 dead and 5 live sockets: both @@ -392,7 +436,10 @@ mesh as a leak. Fixed with explicit `dead == 0` / `alive == 0` cases. It only su because the healthy case finally got looked at; a verdict that is wrong on healthy input will be believed when it is wrong on broken input too. -### Still open: two retention paths `0004` does not cover +### The remaining owner: two paths `0004` structurally cannot reach + +With `0004` confirmed present in the leaking binary, these are no longer speculative +alternatives — one of them is the leak. Reading the ownership chain for the class-A case again, with "what else holds a `Requests::Request` after completion" as the question, turns up two paths that are @@ -420,8 +467,32 @@ consistent with completed requests (peer=DEAD) retaining an fd: run and not in my earlier ones) adds cross-process navigation paths through exactly this code. -Neither is confirmed. What makes (2) the better lead is that it is the first mechanism -found that explains the *combination* Ulf measured — completed-and-closed by the peer, +#### Falsified locally, and the measurement that replaces the guess + +I built the workload for (2): 330 top-level navigations to a URL whose body dribbles, +each abandoned 250 ms in (before the body completes), with `--site-isolation=top-level` +and the memory cache on. **Flat — 0 dead, no growth.** So (2) as I constructed it is not +sufficient either, and I am now two falsified hypotheses deep on a leak I cannot +reproduce. + +That is the point at which guessing again is the wrong move. The two candidates above +look identical in every column the census prints, but they differ in one field `ss` +already reports and I was throwing away: **`Recv-Q`, the bytes sitting unread in the +socket.** + +- `Recv-Q > 0` — the body was **never drained**. The consumer stopped reading, so the + completion branch that closes the fd was never reached. The fix is resume-or-close on + the abandoned path. +- `Recv-Q == 0` — the body **was fully read**; the descriptor is merely still *owned*. + The fix is dropping the surviving reference (the `RefPtr` in + `Response::m_request_server_request`, which `clone()` copies). + +Same signature, opposite fixes, one field. The census now prints +`of retained peer=DEAD: body UNREAD=N body drained=M` and names which mechanism the +numbers imply. Ulf's existing `--all` run already collects the `ss` line this comes +from, so this costs nothing beyond re-running the census he has run before. + +Of the two, (2) is the one that explains the *combination* Ulf measured — completed-and-closed by the peer, retained, and indifferent to a fix that works locally — rather than only part of it. The memory cache, which I suspected next because `--enable-http-memory-cache` was on his command line and I had never tested it, is **ruled out** by reading diff --git a/examples/ladybird/README.md b/examples/ladybird/README.md index bc2c993..a509c01 100644 --- a/examples/ladybird/README.md +++ b/examples/ladybird/README.md @@ -833,8 +833,14 @@ Honest inventory of what stops this from being a clone-and-build. binary contains** (`--build` for that alone), read out of the process's own mapped ELF symbols: a leak rate is uninterpretable without it, since "still leaking at 92/min" means "the fix does not work" or "the fix was not in this build" and those - need opposite next steps. It reports *"cannot tell"* rather than "fix absent" when - the symbols are unreadable — a stripped binary must not read as an unpatched one. + need opposite next steps. It reports *"cannot tell"* rather than "fix absent" + whenever absence could be explained by inlining: under LTO in a **static** build a + small internal-only method is inlined into its only caller and leaves no symbol and + no string behind, so `.debug_str` is read too and a "missing" verdict requires a + symbol inlining cannot erase to be visible. This is a correction — the first version + told Ulf `0004` was absent from a binary that contained it, and the negative control + did not catch it because it was vulnerable to the same optimisation. A control only + rules out "unreadable" if it cannot vanish for the same reason as what it guards. Since upstream has landed its own fix, `--verify` also accepts an equivalent fix in place of our exact bytes: a patch we carry only until upstream fixes it has an `.effect-grep` beside it, and verify diff --git a/examples/ladybird/fd_census.py b/examples/ladybird/fd_census.py index 27cecf9..962e89c 100755 --- a/examples/ladybird/fd_census.py +++ b/examples/ladybird/fd_census.py @@ -109,6 +109,44 @@ def read_fd_targets(pid): # this whole probe exists to prevent. CONTROL_SYMBOL = "set_up_internal_stream_data" +# ...and the control ALONE IS NOT ENOUGH, which is the correction that matters here. +# +# Ulf said "I have all the patches applied" and this probe said 0004 was absent. He was +# right. The reason is a real flaw in the first version, not a mistake of his: +# Ladybird sets ENABLE_LTO_FOR_RELEASE=ON, and in a STATIC build (his) LTO inlines a +# small internal-only method like release_response_fd() into its single caller and +# leaves NO symbol and NO string behind at all. Reproduced from first principles: a +# private method called only within its TU, linked -O3 -flto, disappears from both +# `nm` and `strings` in the linked executable -- while my own build keeps it, because +# a SHARED library must export it. So the probe's answer depended on how the binary +# was linked, not on whether the fix was in it. +# +# And the control did not catch it, because the control is vulnerable to exactly the +# same optimisation: verified that a larger internal-only function also vanishes under +# LTO. A control only rules out "unreadable" if it CANNOT disappear for the same +# reason as the thing it guards. That is the general trap: a negative control has to +# fail in the failure mode you are guarding against, and mine failed in a different +# one. +# +# Two changes follow. (1) .debug_str is read as well as the symbol tables: debug info +# names inlined-away functions, and Ladybird's RelWithDebInfo compiles with -g, so the +# answer survives there for precisely the LTO/static case. (2) A symbol that is merely +# ABSENT is never reported as a missing fix unless something PROVES the binary is +# readable at a level inlining cannot erase -- see UNINLINABLE_CONTROLS. + +# Symbols that cannot be inlined away, so their presence proves the binary's names are +# readable: virtual/IPC-dispatched entry points reached through a vtable or across a +# library boundary. If NONE of these is visible, this binary cannot answer the +# question, whatever else is or is not in it. +UNINLINABLE_CONTROLS = ( + # Reached via the IPC endpoint dispatch table, not a direct call. + "request_started", + "headers_became_available", + # Request's own out-of-line, externally-called entry points. + "did_receive_headers", + "set_unbuffered_request_callbacks", +) + # The Request code lives here; when Ladybird is built statically it is in the # executable instead, so the executable is always probed too. FIX_LIBRARY_HINT = "lagom-requests" @@ -154,7 +192,14 @@ def entry(i): name_off, sec_off, sec_size = entry(i) end = shstrtab.find(b"\0", name_off) name = shstrtab[name_off:end if end >= 0 else None] - if name in (b".dynstr", b".strtab") and sec_size < 64 * 1024 * 1024: + # .debug_str matters as much as the symbol tables: under LTO in a + # STATIC build, a small internal-only method like + # release_response_fd is inlined into its only caller and leaves NO + # symbol and NO string behind -- but the debug info still names it. + # Ladybird's RelWithDebInfo compiles with -g, so this is where the + # answer survives for exactly the build that prompted the question. + if name in (b".dynstr", b".strtab", b".debug_str") \ + and sec_size < 256 * 1024 * 1024: f.seek(sec_off) blob += f.read(sec_size) return blob or None @@ -209,7 +254,25 @@ def probe_fixes(pid): return {}, ("symbols unreadable: the control symbol %s is absent too, so a " "missing fix here would prove nothing (stripped binary, LTO, or " "the code is in a binary not probed)" % CONTROL_SYMBOL) - return ({desc: (sym.encode() in blob) for sym, desc in FIX_SYMBOLS}, None) + + findings = {desc: (sym.encode() in blob) for sym, desc in FIX_SYMBOLS} + if all(findings.values()): + return findings, None + + # Something looks missing. Before saying so, establish that absence MEANS + # anything here: an internal-only method inlined by LTO leaves no trace, so + # "not found" is only evidence if names that CANNOT be inlined away are visible. + # Without this check the probe told Ulf 0004 was absent from a binary that had it. + readable = [c for c in UNINLINABLE_CONTROLS if c.encode() in blob] + if not readable: + missing = [d for d, present in findings.items() if not present] + return {}, ("cannot tell whether %s is present: this binary's names are not " + "readable at a level inlining cannot erase (no uninlinable " + "control symbol found). Under LTO in a static build a small " + "internal-only method leaves no symbol AND no string, so its " + "absence here is not evidence. Check the source or build with " + "-g/shared libs." % "; ".join(missing)) + return findings, None def fix_lines(pid): @@ -264,7 +327,19 @@ def parse_ss(text): continue local_inode, peer_inode = int(inodes[-2]), int(inodes[-1]) holders = re.findall(r'\("([^"]+)",pid=(\d+),fd=(\d+)\)', users) - sockets[local_inode] = (peer_inode, holders) + # Recv-Q: bytes sitting UNREAD in this socket. It is the third column of + # `ss` output and it discriminates between the two mechanisms that both + # present as "retained, peer=DEAD", which nothing else in this census can: + # Recv-Q > 0 -> the body was never drained. The consumer stopped reading + # (e.g. delivery paused and never resumed), so the + # completion branch that closes the fd was never reached. + # Recv-Q == 0 -> the body WAS fully read and the fd is merely still owned. + # That is the ownership bug, not a delivery bug. + # Same field, opposite fixes, so it is worth one column. + recv_q = None + if len(fields) >= 3 and fields[2].isdigit(): + recv_q = int(fields[2]) + sockets[local_inode] = (peer_inode, holders, recv_q) return sockets @@ -282,12 +357,20 @@ def peer_state(inode, sockets): entry = sockets.get(inode) if entry is None: return "unknown", [] - peer_inode, holders = entry + peer_inode, holders = entry[0], entry[1] if peer_inode == 0: return "DEAD", holders return "ALIVE", holders +def recv_queue(inode, sockets): + """Unread bytes in one socket, or None if unknown.""" + entry = sockets.get(inode) + if entry is None or len(entry) < 3: + return None + return entry[2] + + BROWSER_PROCESS_NAMES = ("Ladybird", "ladybird", "WebContent", "RequestServer", "ImageDecoder", "Compositor", "WebWorker") @@ -371,11 +454,13 @@ def sample(self, now=None): inode = socket_inode(target) state, holders = ("n/a", []) + rq = None if inode is not None: state, holders = peer_state(inode, sockets) + rq = recv_queue(inode, sockets) rows.append({ "fd": fd, "target": target, "category": category, "age": age, - "peer": state, + "peer": state, "recv_q": rq, "peers": [h[0] for h in holders if int(h[1]) != int(self.pid)], }) @@ -414,6 +499,7 @@ def summarize(self, snapshot): rd = len([r for r in retained if r["peer"] == "DEAD"]) ra = len([r for r in retained if r["peer"] == "ALIVE"]) out.append(" of retained: peer DEAD=%d peer ALIVE=%d" % (rd, ra)) + out.extend(self.recv_q_lines(retained)) else: # Every fd was first seen in THIS sample. With --watch that means the # process was just attached to, not that the fds are new -- ages are @@ -454,6 +540,49 @@ def summarize(self, snapshot): return "\n".join(out) + def recv_q_lines(self, retained): + """Split the retained peer=DEAD fds by whether their body was ever drained. + + THE MEASUREMENT THAT DISCRIMINATES. With the completion-path fix (0004) + confirmed present in a binary that still leaks ~92/min of retained, + peer=DEAD sockets, the fd has an owner that fix does not reach -- and there + are two candidate mechanisms that look IDENTICAL in every column above: + + unread > 0 -> the body was never drained. Something stopped reading (a + navigation whose body delivery was paused and never + resumed), so the completion branch that closes the fd was + never reached. Fix: resume-or-close on every abandoned path. + unread == 0 -> the body WAS fully read; the descriptor is merely still + owned. Fix: drop the surviving reference (e.g. the RefPtr in + Response::m_request_server_request, which clone() copies). + + Same signature, opposite fixes. Guessing between them is what burned the last + several rounds, so this reads the answer off the socket instead. + """ + dead = [r for r in retained if r["peer"] == "DEAD"] + known = [r for r in dead if r["recv_q"] is not None] + if not known: + return [] + unread = [r for r in known if r["recv_q"] > 0] + drained = [r for r in known if r["recv_q"] == 0] + lines = [" of retained peer=DEAD: body UNREAD=%d body drained=%d" + % (len(unread), len(drained))] + if unread and len(unread) >= 5 * max(len(drained), 1): + total = sum(r["recv_q"] for r in unread) + lines.append(" -> bodies were NEVER DRAINED (%d bytes still queued). The " + "consumer stopped reading, so the completion path that " + "closes the fd was never reached: look at body delivery " + "being paused and not resumed, NOT at fd ownership." % total) + elif drained and len(drained) >= 5 * max(len(unread), 1): + lines.append(" -> bodies were FULLY READ, so delivery completed and the " + "fd is merely still OWNED. Look for a surviving reference to " + "the Requests::Request (Response::m_request_server_request is " + "a RefPtr, and clone() copies it), NOT at delivery.") + elif unread and drained: + lines.append(" -> MIXED: both mechanisms are present; they need separate " + "fixes and a re-census between them.") + return lines + def rate_lines(self): """Growth since the first sample, as a rate. The decisive measurement. diff --git a/tests/test_fd_census.py b/tests/test_fd_census.py index d19bc98..95efb17 100644 --- a/tests/test_fd_census.py +++ b/tests/test_fd_census.py @@ -56,8 +56,9 @@ def test_parses_a_dead_peer_as_inode_zero(): """Peer inode 0 is the entire signal; if this parse is wrong, so is everything.""" sockets = fd_census.parse_ss(SS_DEAD) assert 1452591 in sockets - peer, holders = sockets[1452591] + peer, holders, recv_q = sockets[1452591] assert peer == 0 + assert recv_q == 0 assert holders == [("WebContent", "24066", "85")] assert fd_census.peer_state(1452591, sockets)[0] == "DEAD" @@ -65,8 +66,9 @@ def test_parses_a_dead_peer_as_inode_zero(): def test_parses_a_live_peer_and_names_the_holder(): """A live peer names the producer, which is what stops the next guess.""" sockets = fd_census.parse_ss(SS_ALIVE) - peer, holders = sockets[1451359] + peer, holders, recv_q = sockets[1451359] assert peer == 1451360 + assert recv_q == 0 assert ("RequestServer", "22712", "144") in holders state, holders = fd_census.peer_state(1451359, sockets) assert state == "ALIVE" @@ -519,6 +521,96 @@ def test_probe_reports_cannot_tell_rather_than_fix_absent(tmp_path=None): "must never claim a fix is missing when the symbols were unreadable" +def test_absence_is_not_reported_as_a_missing_fix_when_inlining_could_explain_it(): + """The correction Ulf's "I have all the patches applied" forced. + + The probe told him 0004 was absent from a binary that contained it. He was right + and the tool was wrong: Ladybird sets ENABLE_LTO_FOR_RELEASE=ON, and in a STATIC + build LTO inlines a small internal-only method like release_response_fd() into its + only caller, leaving NO symbol and NO string. Reproduced from first principles: a + private method called only within its TU, linked -O3 -flto, is gone from both `nm` + and `strings`, while a SHARED build keeps it because it must be exported. So the + verdict depended on how the binary was LINKED, not on what was in it. + + CONTROL_SYMBOL did not catch this, because it is vulnerable to the very same + optimisation (verified: a larger internal-only function also vanishes under LTO). + A negative control only rules out "unreadable" if it cannot disappear for the same + reason as the thing it guards -- mine failed in a different mode than the one it + was guarding against, which is the general trap. + + So: a fix reported MISSING requires a symbol that inlining cannot erase to be + visible. Otherwise the answer is "cannot tell". + """ + mod = fd_census + control = mod.CONTROL_SYMBOL.encode() + + def classify(blob): + if control not in blob: + return "no-control", {} + findings = {d: (s.encode() in blob) for s, d in mod.FIX_SYMBOLS} + if all(findings.values()): + return "verdict", findings + if not any(c.encode() in blob for c in mod.UNINLINABLE_CONTROLS): + return "cannot-tell", {} + return "verdict", findings + + # Ulf's build: 0004 inlined away, control survived, nothing uninlinable visible. + ulf_like = control + b"\0defer_teardown\0" + assert classify(ulf_like)[0] == "cannot-tell", \ + "an LTO-inlined fix must not be reported as absent -- this is the bug itself" + + # A genuinely unpatched but READABLE binary must still be called out. + unpatched = control + b"\0defer_teardown\0request_started\0did_receive_headers\0" + state, findings = classify(unpatched) + assert state == "verdict" + assert not findings[dict(mod.FIX_SYMBOLS)["release_response_fd"]] + + # And a patched one is unambiguous either way. + assert classify(control + b"\0defer_teardown\0release_response_fd\0")[0] == "verdict" + + +def test_uninlinable_controls_are_actually_uninlinable(): + """They must be reached via vtable/IPC dispatch or across a library boundary. + + A directly-called internal helper here would reintroduce the false negative: it + could be inlined away exactly when the fix is, and then "readable" would be as + wrong as the fix's absence. + """ + mod = fd_census + assert mod.UNINLINABLE_CONTROLS + src = Path("/home/ubuntu/ladybird-work/Libraries/LibRequests") + if not src.is_dir(): + return # the Ladybird tree is not present in every checkout + text = (src / "Request.cpp").read_text() + (src / "RequestClient.cpp").read_text() + for name in mod.UNINLINABLE_CONTROLS: + assert name in text, \ + "%s must exist in LibRequests or it cannot serve as a control" % name + # each must be an externally-reachable entry point, not a private helper + header = (src / "Request.h").read_text() + (src / "RequestClient.h").read_text() + for name in mod.UNINLINABLE_CONTROLS: + assert name in header, \ + ("%s is not declared in a header, so it is internal-only and LTO can " + "erase it -- exactly the failure this control must not share" % name) + + +def test_debug_str_is_searched_because_it_names_inlined_away_functions(): + """Under LTO the symbol tables lose the fix; debug info keeps the name. + + Verified: with -g (and even -g1), a function fully inlined away by LTO still + appears in .debug_str, and Ladybird's RelWithDebInfo compiles with -g. Dropping + this section would make the probe blind to exactly the build that prompted it. + """ + mod = fd_census + text = SCRIPT.read_text() + assert ".debug_str" in text + assert "inlin" in text.lower() + real = Path("/home/ubuntu/ladybird-work/Build/full/lib/" + "liblagom-requests.so.0.1.0") + if real.exists(): + blob = mod.elf_symbol_names(str(real)) + assert blob and b"release_response_fd" in blob + + def test_control_symbol_is_present_in_both_patched_and_unpatched_builds(): """The control must be code neither patch adds, or it proves nothing. @@ -625,3 +717,55 @@ def test_verdict_never_claims_a_class_with_zero_members(): assert "class A" in mod.verdict(203, 1) assert "class B" in mod.verdict(1, 203) assert "no unix sockets" in mod.verdict(0, 0) + + +def test_recv_q_discriminates_never_drained_from_merely_owned(): + """The measurement that separates the two mechanisms left after 0004. + + Ulf confirmed 0004 IS applied to the binary still leaking ~92/min of retained + peer=DEAD sockets, so the fd has an owner that fix does not reach. Two candidate + mechanisms look identical in every other column: + + unread > 0 -> the body was never drained (delivery paused, never resumed), so + the completion branch that closes the fd was never reached; + unread == 0 -> the body was fully read and the fd is merely still owned. + + Same signature, opposite fixes. Guessing between them burned several rounds, so + the census reads it off the socket's Recv-Q instead of asking for another A/B. + """ + mod = fd_census + c = mod.Census(1234, retained_after=1.0) + + def rows(pairs): + return [{"peer": "DEAD", "recv_q": q, "age": 99.0} for q in pairs] + + never_drained = "\n".join(c.recv_q_lines(rows([4096] * 20))) + assert "UNREAD=20" in never_drained + assert "NEVER DRAINED" in never_drained + assert "paused" in never_drained + assert "81920 bytes" in never_drained # the queued bytes are named, not implied + + merely_owned = "\n".join(c.recv_q_lines(rows([0] * 20))) + assert "drained=20" in merely_owned + assert "FULLY READ" in merely_owned + assert "m_request_server_request" in merely_owned, \ + "must name the surviving reference, not just say 'ownership'" + + mixed = "\n".join(c.recv_q_lines(rows([0] * 10 + [512] * 10))) + assert "MIXED" in mixed + + # unknown Recv-Q must produce no claim at all + assert c.recv_q_lines([{"peer": "DEAD", "recv_q": None, "age": 99.0}]) == [] + # and ALIVE sockets are not part of this question + assert c.recv_q_lines([{"peer": "ALIVE", "recv_q": 4096, "age": 99.0}]) == [] + + +def test_parse_ss_reads_recv_q_from_the_real_column(): + """Recv-Q is ss's third column; a wrong index would invent a diagnosis.""" + mod = fd_census + line = ('u_str ESTAB 8192 0 * 1452591 * 0 ' + 'users:(("WebContent",pid=24066,fd=85))') + sockets = mod.parse_ss(line) + assert mod.recv_queue(1452591, sockets) == 8192 + assert mod.peer_state(1452591, sockets)[0] == "DEAD" + assert mod.recv_queue(999999, sockets) is None From 0a50c30f9a84b56ed3b0cf10edf49eba08434e82 Mon Sep 17 00:00:00 2001 From: Ulf Adams Date: Tue, 18 Aug 2026 15:54:14 +0000 Subject: [PATCH 34/59] Ladybird repin f9e34731 -> 71fb301a: two of my four patches are upstream now Ulf hit a glob failure on a checkout past our pin -- Libraries/LibJS/BytecodeDef/** matched nothing, allow_empty = False. Not a bug in the generated build: upstream a32d9c9f ("LibJS: Derive bytecodes from Flap handlers") DELETED BytecodeDef/ and Bytecode/Bytecode.def, so the overlay was describing a tree that no longer exists. He asked for the build fixed at 71fb301a. This commit is the half that does not need the reference build: the pin, the patch series and the host-tool list. The emitter regeneration (BUILD.bazel, codegen_root.bzl, cargo_*.bzl, the new flapc --bin generate-libjs-bytecode replacing the deleted Python generator) needs a CMake reference build at the new commit and follows separately. The patch directory SHRANK, which is the part worth reading. patches/0001 (PYTHONHASHSEED dictionary order) and 0002 (UI/Qt/TabBar.h not self-contained) are both fixed upstream at 71fb301a and are deleted; the fd-leak pair is renumbered 0003/0004 -> 0001/0002 and verified to still apply as a series at the new commit. Upstream's determinism fix is better than mine: it sorts inside dependency_names_for so no caller can get a set, where I sorted at the one call site I had found -- and two of my four patches were the same bug in that function, which I had filed as distinct. Both patches kept applying long after they stopped being needed, so "it still applies" was never evidence the bug was still there. The repin also cost four failed CMake configures, one per undiscovered host package, and they are now named in the README: upstream turned Qt6 Positioning from OPTIONAL_COMPONENTS into REQUIRED, made GuiPrivate required on Linux, and added pkg_check_modules(GIO) for the new ExternalURLActivationToken/Handler sources. Two of the four are transitive and neither error names a package: Qt6GuiPrivate reports NOT FOUND until libxkbcommon-dev exists, then names a QtGui/6.10.2 include path only qt6-base-private-dev ships. Same shape as finding 39, one layer out -- the preflight covers vcpkg's host tools, not Ladybird's own find_package requirements. --- examples/ladybird/README.md | 106 ++++++++++-------- examples/ladybird/apply_overlay.sh | 2 +- ...equest-when-body-is-delivered.effect-grep} | 0 ...down-request-when-body-is-delivered.patch} | 0 ...dings-deterministic-dictionary-order.patch | 20 ---- ...ase-response-fd-on-completion.effect-grep} | 8 +- ...s-release-response-fd-on-completion.patch} | 2 +- ...2-ui-qt-tabbar-self-contained-header.patch | 29 ----- tests/test_apply_overlay.py | 18 +-- 9 files changed, 76 insertions(+), 109 deletions(-) rename examples/ladybird/patches/{0003-librequests-tear-down-request-when-body-is-delivered.effect-grep => 0001-librequests-tear-down-request-when-body-is-delivered.effect-grep} (100%) rename examples/ladybird/patches/{0003-librequests-tear-down-request-when-body-is-delivered.patch => 0001-librequests-tear-down-request-when-body-is-delivered.patch} (100%) delete mode 100644 examples/ladybird/patches/0001-libweb-bindings-deterministic-dictionary-order.patch rename examples/ladybird/patches/{0004-requests-release-response-fd-on-completion.effect-grep => 0002-requests-release-response-fd-on-completion.effect-grep} (87%) rename examples/ladybird/patches/{0004-requests-release-response-fd-on-completion.patch => 0002-requests-release-response-fd-on-completion.patch} (99%) delete mode 100644 examples/ladybird/patches/0002-ui-qt-tabbar-self-contained-header.patch diff --git a/examples/ladybird/README.md b/examples/ladybird/README.md index a509c01..fb735a8 100644 --- a/examples/ladybird/README.md +++ b/examples/ladybird/README.md @@ -117,7 +117,7 @@ knowing, because both were mistakes I made first: ### Reproducing the tree on another machine -The overlay is **not a fork**: it is a pinned upstream Ladybird commit + three patches +The overlay is **not a fork**: it is a pinned upstream Ladybird commit + two patches + 45 Bazel files that sit alongside CMake's. [`apply_overlay.sh`](apply_overlay.sh) is that sentence made executable, because *"copy `workspace/` over a clone"* has four ways to be silently wrong — and every one of them was found by running it, not by @@ -126,14 +126,17 @@ reading it: 1. **The Ladybird commit.** The generated BUILD files name ~1,961 LibWeb compile inputs and 665 IDL bindings *by path*, and were generated from one tree. **Nothing in this repo recorded which one** until the script did - (`f9e34731`). Against a different tree the build fails on a moved file — or + (`71fb301a`). Against a different tree the build fails on a moved file — or worse, silently omits a new one. -2. **The patches**, which have to be applied or the build is nondeterministic - (dictionaries emitted in `PYTHONHASHSEED` order), does not compile at all - under per-header moc, or leaks one socket fd per completed HTTP request until - the browser dies of `EMFILE` overnight - ([`0003`](patches/0003-librequests-tear-down-request-when-body-is-delivered.patch); +2. **The patches**, which have to be applied or the browser leaks one socket fd per + completed HTTP request until it dies of `EMFILE` overnight + ([`0001`](patches/0001-librequests-tear-down-request-when-body-is-delivered.patch) + and [`0002`](patches/0002-requests-release-response-fd-on-completion.patch); see [`docs/UPSTREAM-ladybird-fd-leaks.md`](../../docs/UPSTREAM-ladybird-fd-leaks.md)). + There used to be four: the `PYTHONHASHSEED` determinism fix and the + non-self-contained `UI/Qt/TabBar.h` are both **fixed upstream** at this pin, so + the repin from `f9e34731` deleted them. A patch directory that only ever grows is + a patch directory nobody re-checks against upstream. 3. **The rename**: `bazelrc.txt` → `.bazelrc`. Stored under a different name so a `cp -r` cannot be mistaken for a working build — and a rename a human does by hand is a rename a human forgets. @@ -160,12 +163,28 @@ outputs), `Meta/fetch_vcpkg_git_archives.py` reproduced **4/4** archives verifie against the pinned SHA512s, and `bazel build` then built the 76 vcpkg ports offline and linked `//:LibHTTP`. -#### One thing the overlay cannot carry: vcpkg's host tools +#### One thing the overlay cannot carry: host tools ```sh sudo apt install nasm autoconf automake libtool autoconf-archive libltdl-dev +# ...and, since the 71fb301a repin, four host packages upstream newly REQUIRES: +sudo apt install qt6-positioning-dev qt6-base-private-dev libxkbcommon-dev libglib2.0-dev ``` +The second line is what a repin costs, and every one of the four was found by a +*configure or generate failure*, one at a time, because nothing derives this set: +`UI/Qt/CMakeLists.txt` turned `Positioning` from `OPTIONAL_COMPONENTS` into +`REQUIRED`, made `GuiPrivate` required on Linux (not just Apple/DirectX), and added +`pkg_check_modules(GIO REQUIRED gio-2.0 gio-unix-2.0)` for the new +`ExternalURLActivationToken`/`ExternalURLHandler` sources. `qt6-base-private-dev` and +`libxkbcommon-dev` are *transitive*: `Qt6GuiPrivate` reports itself NOT FOUND until +XKB is present, then names an `INTERFACE_INCLUDE_DIRECTORIES` path +(`/usr/include/.../QtGui/6.10.2`) that only the private-dev package ships — a +two-step failure where neither message mentions the package to install. That is the +finding-39 shape again, one layer out: the preflight covers *vcpkg's* host tools, so +Ladybird's own `find_package`/`pkg_check_modules` requirements are unchecked and cost +one failed configure each. + Not a courtesy list — the set is derived from vcpkg's own scripts into [`Meta/vcpkg_host_tools.tsv`](workspace/Meta/vcpkg_host_tools.tsv), and `vcpkg_build.sh` checks all of it before building anything, printing the ports that @@ -288,7 +307,7 @@ an FFI header collision and an entire missing Rust target. See | `Meta/vcpkg_capture_git_archives.sh` | Regenerates that pin with `vcpkg install --only-downloads` (~6 min, no compilation, no CMake) — vcpkg as the instrument, since which git externals are used is decided by feature-conditional CMake code, not by portfile text | | `hsts_preload.bzl` | Chromium's HSTS preload table as one `http_file`, pinned to a **commit** + sha256 — the downstream pin for the one input upstream CMake fetches from `main`. **Generated** by `Meta/pin_hsts_preload.py` | | `Meta/pin_hsts_preload.py` | Re-pins it: resolves the newest commit touching the path, downloads it, writes the hash it **measured**; `--expect-same-as` refuses to write unless the pinned bytes equal the file CMake downloaded (parity guard) | -| `apply_overlay.sh` | Reproduces the whole tree on another machine: clone at the pinned Ladybird commit, apply the three patches, copy the 45 overlay files (with the `bazelrc.txt` → `.bazelrc` rename), run the vcpkg prefetch, then stage the one file that must come after it. `--verify` checks an existing tree — commit, bytes, patches-applied, exec bits — and changes nothing | +| `apply_overlay.sh` | Reproduces the whole tree on another machine: clone at the pinned Ladybird commit, apply the two patches, copy the 45 overlay files (with the `bazelrc.txt` → `.bazelrc` rename), run the vcpkg prefetch, then stage the one file that must come after it. `--verify` checks an existing tree — commit, bytes, patches-applied, exec bits — and changes nothing | | `bazelrc.txt` | → `.bazelrc`. Global copts/defines/linkopts mirrored from `Meta/CMake/compile_options.cmake` | | `qt_runtime.bzl` | Qt's **runtime** half: a repo rule that stages the plugins of the SDK `@qt` itself names (read out of @qt's generated `qtconf.bzl`, so plugins and libraries cannot come from different Qts), the private libraries an SDK bundles beside Qt (derived from DT_NEEDED), a generated `qt.conf` pointing the binary at them, and the Qt >= 6.9 floor `UI/Qt/CMakeLists.txt` declares. Without it Qt `dlopen`ed the HOST's plugin into Bazel's Qt — finding 40 | | `BUILD.bazel` | Root package: 34 libraries, the 5 executables, Qt moc/rcc genrules. **Generated** by `Meta/emit_build_bazel.py` | @@ -335,7 +354,7 @@ Or by hand: ```sh git clone https://github.com/LadybirdBrowser/ladybird && cd ladybird -git checkout f9e34731b85fea1c3517941d8388566cd33277c4 # the commit the generated +git checkout 71fb301a851e4a098e863a7a67e6666599e1cab7 # the commit the generated # BUILD files describe # 1. Drop in the overlay. cp -r .../examples/ladybird/workspace/. . && mv bazelrc.txt .bazelrc @@ -774,35 +793,32 @@ Honest inventory of what stops this from being a clone-and-build. that says there is none. Same shape as the shim that could not fail: a control that is not enforced is indistinguishable from one that is not there. -9. **Four upstreamable Ladybird fixes.** The first is the `sorted()` determinism - fix in `Meta/Generators/libweb_bindings/to_idl_value.py`, filed as - [ladybird#10899](https://github.com/LadybirdBrowser/ladybird/issues/10899) and - fixed upstream. - - The second is a one-line `#include ` in `UI/Qt/TabBar.h`: - [`patches/0002-ui-qt-tabbar-self-contained-header.patch`](patches/0002-ui-qt-tabbar-self-contained-header.patch). - `TabBar.h` calls `as()` — a `dynamic_cast`, needing Tab's complete type — - while only forward-declaring `Tab`, and compiles under CMake purely by ordering - luck: AUTOMOC's unity `mocs_compilation.cpp` includes `moc_Tab.cpp` (hence - `Tab.h`) before `moc_TabBar.cpp`. Bazel mocs each header separately, so nothing - supplies the definition first. A latent upstream bug rather than a Bazel quirk — - any build that changes compile order (different unity bucketing, an IWYU pass) - hits it. Still to file upstream. - - A **third** one is now needed, in the same function as the first: the topological - sort in `dictionaries_in_dependency_order` iterates a *set* of dependency names, - so two dictionaries that do not depend on each other (`AudioConfiguration` and - `VideoConfiguration`, both reached from `MediaConfiguration`) emit in hash order - and `Bindings/MediaCapabilities.h` varies with `PYTHONHASHSEED`. A topological - sort constrains dependency-before-dependent; the order among independent - siblings must be pinned separately. Found by the harness's seed sweep, not by - any single run — the inherited-seed run was clean. The fix is in - [`patches/0001-libweb-bindings-deterministic-dictionary-order.patch`](patches/0001-libweb-bindings-deterministic-dictionary-order.patch); - apply it in the checkout before running the harness. - - A **fourth**, found only because the Bazel-built browser was left running - overnight, is a resource leak rather than a build defect: - [`patches/0003-librequests-tear-down-request-when-body-is-delivered.patch`](patches/0003-librequests-tear-down-request-when-body-is-delivered.patch). +9. **Two upstreamable Ladybird fixes — down from four, and that is the point of a + repin.** Two of the four are **fixed upstream** at this pin (`71fb301a`) and their + patches are deleted, which is worth stating because a patch directory is a debt + register that only shrinks if somebody re-reads it: + + * The `sorted()` determinism fix in `Meta/Generators/libweb_bindings/to_idl_value.py` + — filed as [ladybird#10899](https://github.com/LadybirdBrowser/ladybird/issues/10899) + and fixed upstream **better than my patch was**: upstream sorts inside + `dependency_names_for`, so no caller can receive a set, whereas my patch sorted at + the one call site I had found. Two of my four patches were the *same* bug in that + one function, and I filed the second as distinct; it was not. + * The one-line `#include ` in `UI/Qt/TabBar.h`. `TabBar.h` calls + `as()` — a `dynamic_cast`, needing Tab's complete type — while only + forward-declaring `Tab`, and compiled under CMake purely by ordering luck: + AUTOMOC's unity `mocs_compilation.cpp` includes `moc_Tab.cpp` (hence `Tab.h`) + before `moc_TabBar.cpp`. Bazel mocs each header separately, so nothing supplied + the definition first. Upstream now includes the header. + + Both survived in `patches/` only because the old pin (`f9e34731`) predated the + upstream fixes — a patch keeps applying long after it stops being needed, so + "it still applies" is not evidence that it is still a bug. + + What remains is the fd leak, in two patches that are one series. The first, + found only because the Bazel-built browser was left running overnight, is a + resource leak rather than a build defect: + [`patches/0001-librequests-tear-down-request-when-body-is-delivered.patch`](patches/0001-librequests-tear-down-request-when-body-is-delivered.patch). WebContent leaks one AF_UNIX socket fd per *completed* HTTP request — a cycle between the GC heap (four `GC::Root`s from `Fetching.cpp:2338`) and the refcount heap (`Response` holding `Requests::Request` by `RefPtr`) that neither @@ -812,15 +828,15 @@ Honest inventory of what stops this from being a clone-and-build. with a live peer survives it. Both are written up with reproductions in [`docs/UPSTREAM-ladybird-fd-leaks.md`](../../docs/UPSTREAM-ladybird-fd-leaks.md). - A **fifth** is the other half of the completed-request class, and it only showed + The **second** is the other half of the completed-request class, and it only showed up on a tree that was not mine: - [`patches/0004-requests-release-response-fd-on-completion.patch`](patches/0004-requests-release-response-fd-on-completion.patch). - With `0003`'s teardown applied, Ulf still measured **97 leaked sockets/min**, all + [`patches/0002-requests-release-response-fd-on-completion.patch`](patches/0002-requests-release-response-fd-on-completion.patch). + With `0001`'s teardown applied, Ulf still measured **97 leaked sockets/min**, all `peer=DEAD` and all sent by RequestServer — because dropping the callbacks unpins the GC cycle but does not *close the descriptor*: the response pipe is released only by `~Request`, so any other reference to the request retains one fd per - completed request. `0004` closes it where the body is already proven complete — the - next patch in the series, applied on top of `0003`; `patches/*.patch` is applied by + completed request. `0002` closes it where the body is already proven complete — the + next patch in the series, applied on top of `0001`; `patches/*.patch` is applied by glob in full, so an *alternative* to a patch must never live there. Measured A/B on 200 completed requests: **208 sockets (203 dead) → 6 (0 dead)**, with body delivery intact. "Collectable" and "closed" are different claims, and @@ -838,7 +854,7 @@ Honest inventory of what stops this from being a clone-and-build. small internal-only method is inlined into its only caller and leaves no symbol and no string behind, so `.debug_str` is read too and a "missing" verdict requires a symbol inlining cannot erase to be visible. This is a correction — the first version - told Ulf `0004` was absent from a binary that contained it, and the negative control + told Ulf `0002` was absent from a binary that contained it, and the negative control did not catch it because it was vulnerable to the same optimisation. A control only rules out "unreadable" if it cannot vanish for the same reason as what it guards. Since upstream has landed its own fix, diff --git a/examples/ladybird/apply_overlay.sh b/examples/ladybird/apply_overlay.sh index 7a9596c..a590ff0 100755 --- a/examples/ladybird/apply_overlay.sh +++ b/examples/ladybird/apply_overlay.sh @@ -43,7 +43,7 @@ set -euo pipefail # from, and the only tree they are known to describe. A tag would be wrong here # for the same reason it was wrong for the HSTS table (see hsts_preload.bzl): a # tag is a pin to a different tree than the one measured. -LADYBIRD_COMMIT="f9e34731b85fea1c3517941d8388566cd33277c4" +LADYBIRD_COMMIT="71fb301a851e4a098e863a7a67e6666599e1cab7" LADYBIRD_REPO="https://github.com/LadybirdBrowser/ladybird.git" HERE="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" diff --git a/examples/ladybird/patches/0003-librequests-tear-down-request-when-body-is-delivered.effect-grep b/examples/ladybird/patches/0001-librequests-tear-down-request-when-body-is-delivered.effect-grep similarity index 100% rename from examples/ladybird/patches/0003-librequests-tear-down-request-when-body-is-delivered.effect-grep rename to examples/ladybird/patches/0001-librequests-tear-down-request-when-body-is-delivered.effect-grep diff --git a/examples/ladybird/patches/0003-librequests-tear-down-request-when-body-is-delivered.patch b/examples/ladybird/patches/0001-librequests-tear-down-request-when-body-is-delivered.patch similarity index 100% rename from examples/ladybird/patches/0003-librequests-tear-down-request-when-body-is-delivered.patch rename to examples/ladybird/patches/0001-librequests-tear-down-request-when-body-is-delivered.patch diff --git a/examples/ladybird/patches/0001-libweb-bindings-deterministic-dictionary-order.patch b/examples/ladybird/patches/0001-libweb-bindings-deterministic-dictionary-order.patch deleted file mode 100644 index f0da082..0000000 --- a/examples/ladybird/patches/0001-libweb-bindings-deterministic-dictionary-order.patch +++ /dev/null @@ -1,20 +0,0 @@ -diff --git a/Meta/Generators/libweb_bindings/to_idl_value.py b/Meta/Generators/libweb_bindings/to_idl_value.py -index ef890dbb1b..9e9bb92767 100644 ---- a/Meta/Generators/libweb_bindings/to_idl_value.py -+++ b/Meta/Generators/libweb_bindings/to_idl_value.py -@@ -189,7 +189,14 @@ def dictionaries_in_dependency_order(dictionaries: List[Dictionary], context: Ge - raise RuntimeError(f"Dictionary '{dictionary.name}' depends on itself") - - visiting.add(dictionary.name) -- for dependency_name in dependency_names_for(dictionary): -+ # sorted(): dependency_names_for returns a SET, so iterating it visits -+ # sibling dependencies in hash order, and the emission order of two -+ # dictionaries that do not depend on each other (AudioConfiguration and -+ # VideoConfiguration, both reached from MediaConfiguration) then varies -+ # with PYTHONHASHSEED. A topological sort only constrains dependency -+ # before dependent; the order among independent siblings has to be pinned -+ # separately, or the generator is not reproducible. -+ for dependency_name in sorted(dependency_names_for(dictionary)): - dependency = local_dictionaries.get(dependency_name) - if dependency is not None: - visit(dependency) diff --git a/examples/ladybird/patches/0004-requests-release-response-fd-on-completion.effect-grep b/examples/ladybird/patches/0002-requests-release-response-fd-on-completion.effect-grep similarity index 87% rename from examples/ladybird/patches/0004-requests-release-response-fd-on-completion.effect-grep rename to examples/ladybird/patches/0002-requests-release-response-fd-on-completion.effect-grep index 9980b58..d1da807 100644 --- a/examples/ladybird/patches/0004-requests-release-response-fd-on-completion.effect-grep +++ b/examples/ladybird/patches/0002-requests-release-response-fd-on-completion.effect-grep @@ -1,16 +1,16 @@ # The EFFECT this patch must have, as extended regexes over the file it touches # (Libraries/LibRequests/Request.cpp). Checked only when the patch's exact bytes are # NOT in the tree, so an equivalent fix -- upstream's or anyone's -- counts as -# "already fixed" rather than "missing". Same reasoning as 0003's .effect-grep: +# "already fixed" rather than "missing". Same reasoning as 0001's .effect-grep: # `git apply --check -R` proves my bytes, not the fix. # # The effect: the response fd received from RequestServer must be closed WHERE THE -# BODY IS PROVEN COMPLETE, not merely when the Request object dies. 0003 dropped the +# BODY IS PROVEN COMPLETE, not merely when the Request object dies. 0001 dropped the # callbacks (which unpins the GC cycle); this closes the fd even while something else -# still holds a reference to the Request -- the case that survived 0003 on Ulf's +# still holds a reference to the Request -- the case that survived 0001 on Ulf's # tree at 97 fds/min. # -# The window matters, for the same reason as 0003: a bare close/notifier call +# The window matters, for the same reason as 0001: a bare close/notifier call # appears elsewhere in this file (~Request, attach_read_stream), so a whole-file grep # would pass on an unpatched tree. Anchor on the completion branch and require the # release within the following few lines. NOTE: apply_overlay.sh honours only the diff --git a/examples/ladybird/patches/0004-requests-release-response-fd-on-completion.patch b/examples/ladybird/patches/0002-requests-release-response-fd-on-completion.patch similarity index 99% rename from examples/ladybird/patches/0004-requests-release-response-fd-on-completion.patch rename to examples/ladybird/patches/0002-requests-release-response-fd-on-completion.patch index 26c2a16..cdefac2 100644 --- a/examples/ladybird/patches/0004-requests-release-response-fd-on-completion.patch +++ b/examples/ladybird/patches/0002-requests-release-response-fd-on-completion.patch @@ -1,7 +1,7 @@ From: any2bazel Ladybird migration Subject: [PATCH] LibRequests: close the response fd on completion, not just at ~Request -Applies ON TOP OF the teardown fix (patches/0003, or upstream's equivalent), which is +Applies ON TOP OF the teardown fix (patches/0001, or upstream's equivalent), which is the previous patch in this series. The two edit adjacent lines in the same branch, so apply them in order -- apply_overlay.sh does that for you. diff --git a/examples/ladybird/patches/0002-ui-qt-tabbar-self-contained-header.patch b/examples/ladybird/patches/0002-ui-qt-tabbar-self-contained-header.patch deleted file mode 100644 index 6e69b77..0000000 --- a/examples/ladybird/patches/0002-ui-qt-tabbar-self-contained-header.patch +++ /dev/null @@ -1,29 +0,0 @@ -From: any2bazel Ladybird migration -Subject: [PATCH] UI/Qt: make TabBar.h self-contained by including Tab.h - -TabBar.h calls as() inline -- a dynamic_cast, which needs Tab's complete -type -- while only forward-declaring Tab. It compiles today only by accident of -CMake's AUTOMOC: the unity mocs_compilation.cpp happens to #include moc_Tab.cpp -(which includes Tab.h) before moc_TabBar.cpp, so the definition is in scope by -the time TabBar.h is parsed. - -Bazel runs moc per header rather than through a unity file, so nothing supplies -Tab.h first and the header fails to compile on its own. That makes this a real -latent bug rather than a Bazel quirk: the header is not self-contained, and any -build that compiles it in a different order -- a unity build with different -bucketing, a header-only IWYU check, a different generator -- hits the same -thing. - -One line, and it makes the header stand alone. -diff --git a/UI/Qt/TabBar.h b/UI/Qt/TabBar.h -index 00269c5c..93963006 100644 ---- a/UI/Qt/TabBar.h -+++ b/UI/Qt/TabBar.h -@@ -10,6 +10,7 @@ - - #include - #include -+#include - - #include - #include diff --git a/tests/test_apply_overlay.py b/tests/test_apply_overlay.py index e17ad15..1ecff6b 100644 --- a/tests/test_apply_overlay.py +++ b/tests/test_apply_overlay.py @@ -182,7 +182,7 @@ def test_effect_grep_files_exist_for_patches_upstream_may_fix_itself(): """ text = _text() assert ".effect-grep" in text, "verify must fall back to an effect check" - fd_leak = PATCHES / "0003-librequests-tear-down-request-when-body-is-delivered.patch" + fd_leak = PATCHES / "0001-librequests-tear-down-request-when-body-is-delivered.patch" assert fd_leak.exists() effect = fd_leak.with_suffix(".effect-grep") assert effect.exists(), \ @@ -201,7 +201,7 @@ def test_effect_grep_is_windowed_not_a_whole_file_grep(): being too strict. The check therefore anchors on the branch condition and requires the call within a window of following lines. """ - effect = (PATCHES / "0003-librequests-tear-down-request-when-body-is-delivered.effect-grep") + effect = (PATCHES / "0001-librequests-tear-down-request-when-body-is-delivered.effect-grep") lines = [ln for ln in effect.read_text().splitlines() if ln.strip() and not ln.startswith("#")] assert any(ln.startswith("@window ") for ln in lines), \ @@ -261,7 +261,7 @@ def test_the_documented_overlay_file_count_matches_the_overlay(): def test_the_response_fd_patch_closes_the_fd_where_the_body_is_proven_complete(): """The second half of the fd leak: the fd outlives the callbacks. - 0003 drops the callbacks on completion, which unpins the GC cycle -- and on my + 0001 drops the callbacks on completion, which unpins the GC cycle -- and on my machine that took 143 leaked sockets to 0. On Ulf's tree it did not: he measured 97 sockets/min STILL leaking with the upstream-equivalent teardown applied, every one peer=DEAD (RequestServer had already closed its end) and every one @@ -270,17 +270,17 @@ def test_the_response_fd_patch_closes_the_fd_where_the_body_is_proven_complete() The reason is ownership: the response fd is released only by ~Request (or by the ReadStream inside m_internal_stream_data), so ANY surviving reference to the Request keeps one fd per completed request alive -- dropping the callbacks is - not the same as closing the fd. 0004 closes it explicitly at the point the code + not the same as closing the fd. 0002 closes it explicitly at the point the code has already proven the body is complete. Measured A/B, same binary, same 200-completed-request workload: clean: 208 sockets, 203 peer=DEAD fixed: 6 sockets, 0 peer=DEAD """ - patch = PATCHES / "0004-requests-release-response-fd-on-completion.patch" + patch = PATCHES / "0002-requests-release-response-fd-on-completion.patch" assert patch.exists() text = patch.read_text() - # 0004 is the NEXT PATCH IN THE SERIES, not an alternative to 0003: it is + # 0002 is the NEXT PATCH IN THE SERIES, not an alternative to 0001: it is # generated on top of it and applies only after it. I first shipped a second # "clean tree" variant for trees without the teardown fix, which cannot work -- # apply_overlay.sh globs patches/*.patch and applies them all, so one of the two @@ -290,7 +290,7 @@ def test_the_response_fd_patch_closes_the_fd_where_the_body_is_proven_complete() assert not list(PATCHES.glob("*clean-tree*")), \ "an alternative in patches/*.patch cannot coexist with the glob that applies all" assert "ON TOP OF the teardown fix" in text, \ - "0004 must state that it builds on 0003 rather than replacing it" + "0002 must state that it builds on 0001 rather than replacing it" # the notifier must be deregistered before the fd is closed, or the event loop # is left polling a closed descriptor body = text[text.index("release_response_fd"):] @@ -315,8 +315,8 @@ def test_the_patch_series_applies_as_a_series(): """patches/*.patch must apply IN GLOB ORDER, each on top of the last. apply_overlay.sh applies every patches/*.patch by glob. That makes the directory a - series, not a menu -- and I broke it: 0004 needs 0003's hunk to be present, so I - shipped a second "clean tree" variant of 0004 for trees without the teardown fix. + series, not a menu -- and I broke it: 0002 needs 0001's hunk to be present, so I + shipped a second "clean tree" variant of 0002 for trees without the teardown fix. Since the loop applies ALL of them, one of the two could only ever fail. Ulf hit it on the first run: "tries to apply both patches at the same time". From 88abd6faacf34310a6d5a2b1355a3ec2d79acd98 Mon Sep 17 00:00:00 2001 From: Ulf Adams Date: Tue, 18 Aug 2026 17:47:35 +0000 Subject: [PATCH 35/59] Ladybird repin 71fb301a: fix the build, and derive the globs that broke it Ulf's checkout at 71fb301a failed to LOAD, not to compile: Error in glob: glob pattern 'Libraries/LibJS/BytecodeDef/**' didn't match anything, but allow_empty is set to False Upstream a32d9c9f ("LibJS: Derive bytecodes from Flap handlers") deleted that directory, replaced the Python bytecode generator with a Rust binary (flapc's second bin, generate-libjs-bytecode), and dropped flapc's --bytecode-def flag. The pattern was a hardcoded string in a list literal in emit_cargo_bazel.py, and the same string appeared in three more places plus emit_libweb_bazel.py's rust_crate_srcs heredoc (CSS/Rust/** and Layout/Rust/**, one package over). The fix is structural, not a re-pin of the string. Every crate directory an allow_empty=False glob names is now DERIVED: Cargo.toml's members closed over the manifests' `path =` deps (which is also how the derivation reaches outside the workspace, since libjs_rust build-depends on the excluded flapc), and flapc's extra inputs come from scanning it for include_str!. A deleted crate now deletes its own glob pattern. emit_libweb_bazel imports that one list rather than restating it, so the two sides of the package boundary cannot disagree. Why this shape of staleness is the worst one in the repo: allow_empty=False is CORRECT and had to stay (its opposite is why the old Build/full shims matched nothing for weeks and failed 1,600 actions later), a loading error has no target to blame so nothing in the build graph can report it, and the emitters that would have re-derived it all ran green -- an emitter that enumerates KINDS cannot notice a kind going away. The same repin silently dropped gen_Op from codegen_root.bzl while the parity harness reported 0 UNHANDLED, because the new generator's ninja rule is ` ... && cmake -E copy_if_different ...` and the copy_if_different EXCLUSION matched first: every exclusion in a first-match classifier can capture a command it was not written for, and the count that should catch it is computed after the capture. Also in this repin, all measured against the reference build at 71fb301a: * libweb_css_rust + libweb_layout_rust are consolidated back INTO libweb_rust, so 8 crate targets (was 10) and 19 generated FFI files (was 14). Two of libweb_rust's twelve are .inc, not .h -- Bazel deletes an undeclared .inc exactly as it deletes an undeclared header, so the suffix must not decide whether a generated file is declared. * build_rust_binary() takes FEATURES too, which the parser only read on the import_rust_crate side. Upstream's new style-replay is built FEATURES style-recording from the SAME crate as libweb_rust's staticlib, so ignoring it would have built a different binary than CMake does and said nothing. The feature-name character class also needed `-` (style-recording), which truncates rather than failing to match. * 4 cargo_binary (was 2), 155 crates.io crates (was 154, flapc + foldhash 0.2.0), MODULE.bazel's use_repo regenerated. * libjs_rust's build-script input is Interpreter/interpreter.flap now, not the deleted Bytecode/Bytecode.def. One bug I introduced and then had to be shown: excluding cargo's outdir as `**/target/**` is wrong. flapc has a Rust MODULE at src/target/ (the code generator's per-architecture backends), so it dropped 40 sources and //:generate-libjs-bytecode died with `error[E0583]: file not found for module 'target'`. A glob exclusion is a pattern over paths and knows nothing about what a directory MEANS; "the dir cargo writes to" is /target, and only the anchored form says that. Both properties are now tested. Verified at 71fb301a: parity harness 53 generators / 1418 of 1418 files byte-identical / 0 UNHANDLED; emit_cargo --check all 4 ok and --report clean; export headers 15/15 + AK/Debug.h; all five emitters regenerate the checked-in files byte-identically. On the Bazel side the workspace now LOADS (203 targets), //:gen_Op builds through the new Rust tool and both Op.h and OpCodes.h are byte-identical to CMake's, and libweb_rust/libjs_rust/style-replay build with all 12 of libweb_rust's FFI files byte-identical to the reference. tests 256/256. --- examples/ladybird/README.md | 69 +++- examples/ladybird/workspace/BUILD.bazel | 46 ++- .../workspace/Libraries/LibWeb/BUILD.bazel | 140 +++----- .../workspace/Libraries/LibWeb/codegen.bzl | 19 +- examples/ladybird/workspace/MODULE.bazel | 1 + .../workspace/Meta/bazel_parity_harness.py | 22 +- .../workspace/Meta/emit_build_bazel.py | 37 +- .../workspace/Meta/emit_cargo_bazel.py | 318 ++++++++++++++---- .../workspace/Meta/emit_codegen_bazel.py | 8 +- .../workspace/Meta/emit_libweb_bazel.py | 91 ++--- .../workspace/Meta/emit_root_codegen_bazel.py | 104 +++++- examples/ladybird/workspace/cargo_crates.bzl | 10 +- .../ladybird/workspace/cargo_extension.bzl | 2 +- examples/ladybird/workspace/cargo_index.bzl | 21 +- examples/ladybird/workspace/cargo_ring.bzl | 111 +++--- examples/ladybird/workspace/codegen_root.bzl | 42 ++- .../ladybird/workspace/export_headers.bzl | 2 +- tests/test_emit_cargo.py | 228 ++++++++++--- 18 files changed, 897 insertions(+), 374 deletions(-) diff --git a/examples/ladybird/README.md b/examples/ladybird/README.md index fb735a8..3dcbe8f 100644 --- a/examples/ladybird/README.md +++ b/examples/ladybird/README.md @@ -14,8 +14,8 @@ every one of the build's 586 ninja `CUSTOM_COMMAND`s — 51 covered, 535 exclude with a stated reason, **0 unhandled**), the result renders pages (`--headless=text` and `--headless=layout-tree` match the CMake reference byte for byte), **the 77 vcpkg dependencies are fetched and built by Bazel with -zero network access** (findings 30–33), and **so are the 10 Rust crates, `flapc` -and `cranelift-compiler`** — 154 crates.io crates fetched from `Cargo.lock`, +zero network access** (findings 30–33), and **so are the 8 Rust crates and the 4 +cargo binaries** — 155 crates.io crates fetched from `Cargo.lock`, built by network-blocked cargo actions (findings 34–35). **Nothing in the emitted build reads CMake's build tree (`Build/full`).** Verified @@ -313,12 +313,12 @@ an FFI header collision and an entire missing Rust target. See | `BUILD.bazel` | Root package: 34 libraries, the 5 executables, Qt moc/rcc genrules. **Generated** by `Meta/emit_build_bazel.py` | | `codegen_root.bzl` | Non-LibWeb generator genrules (IPC endpoints, LibJS Bytecode/Op, HSTS table, WebGL replayer, TIFF tag tables, the two SPIR-V shader headers, and the chained `generate_interpreter_layout` → `flapc` interpreter assembly). **Generated** by `Meta/emit_root_codegen_bazel.py` | | `Libraries/LibWeb/BUILD.bazel`, `generated_srcs.bzl` | LibWeb (~1,961 compile inputs). **Generated** by `Meta/emit_libweb_bazel.py` | -| `Libraries/LibWeb/codegen.bzl` | LibWeb's 26 generator genrules + the bindings mega-genrule (661 `.idl` → 1,331 files). **Generated** by `Meta/emit_codegen_bazel.py` | +| `Libraries/LibWeb/codegen.bzl` | LibWeb's 26 generator genrules + the bindings mega-genrule (663 `.idl` → 1,340 files). **Generated** by `Meta/emit_codegen_bazel.py` | | `Meta/emit_*.py` | The emitters. They read CMake's `build.ninja` + the File API codemodel and write the four generated files above | | `Meta/bazel_parity_harness.py` | Buckets every ninja `CUSTOM_COMMAND` as covered / excluded-with-reason / unhandled, re-runs the covered ones and byte-compares against CMake's tree. Non-zero unhandled is a failure | | `Meta/BUILD.bazel` | `//Meta:generators` filegroup (the generator scripts, as genrule inputs) | | `Meta/vcpkg/BUILD.bazel` | The 41 `vcpkg_lib` targets Ladybird's libraries depend on, backed by `//:vcpkg_installed`. Hand-written and stable (one target per port); this is the whole interface between Ladybird and its dependencies | -| `cargo_crates.bzl`, `cargo_index.bzl`, `cargo_extension.bzl` | One `http_archive` per crates.io crate (154), the name/version/sha256 index the vendor staging resolves through, and the module extension that creates the repos + the 3 pinned Rust toolchain components. **Generated** by `Meta/emit_cargo_bazel.py` **from `Cargo.lock` alone** — no cargo, no network, no CMake | +| `cargo_crates.bzl`, `cargo_index.bzl`, `cargo_extension.bzl` | One `http_archive` per crates.io crate (155), the name/version/sha256 index the vendor staging resolves through, and the module extension that creates the repos + the 3 pinned Rust toolchain components. **Generated** by `Meta/emit_cargo_bazel.py` **from `Cargo.lock` alone** — no cargo, no network, no CMake | | `cargo.bzl`, `Meta/cargo_build.sh`, `Meta/cargo_binary_build.sh`, `Meta/cargo_vendor.sh` | `rust_sysroot` (the pinned 1.96.1 toolchain merged into one tree), `cargo_crate` / `cargo_binary` (offline, network-blocked build actions), and `cargo_lib` (one consumable `CcInfo` per crate: its archive + its FFI headers) | | `cargo_ring.bzl` | The 10 `cargo_crate` + 10 `cargo_lib` targets and `flapc`, as a macro for the root package (the crate sources are at the repo root and `glob()` is package-relative). **Generated** by `Meta/emit_cargo_bazel.py` | | `export_headers.bzl`, `Libraries/LibWeb/export_header.bzl` | The 15 `generate_export_header` `Export.h` files + AK's two `configure_file` headers, generated by Bazel from the same inputs CMake uses. **Generated** by `Meta/emit_export_headers_bazel.py`; `--check Build/full` byte-compares every one against CMake's | @@ -537,7 +537,7 @@ Honest inventory of what stops this from being a clone-and-build. **Rust is closed too (finding 34), so the CMake build is no longer needed to build the browser at all.** The 260 MB prebuilt `librust_combined.a`, the hand-run `ar -M` that produced it and the reference build's `flapc` are gone. - `Cargo.lock` already pins a sha256 for all 154 crates.io crates and the URL is + `Cargo.lock` already pins a sha256 for all 155 crates.io crates and the URL is a pure function of (name, version), so — unlike vcpkg — **nothing had to be captured**: `Meta/emit_cargo_bazel.py` emits the fetch rules from `Cargo.lock` alone, with no cargo, no network and no CMake, and the three @@ -548,6 +548,14 @@ Honest inventory of what stops this from being a clone-and-build. claimed: deleting one crate from the vendor dir fails the action with "no matching package named `yuv` found" instead of downloading it. + The link/symbol measurements in the rest of this finding were taken at the + **previous pin** (`f9e34731`), where the tree had 10 staticlib crates and 14 + FFI headers; at `71fb301a` upstream consolidated `libweb_css_rust` and + `libweb_layout_rust` back into `libweb_rust`, so it is 8 crates and 19 + generated files. The counts below are left as measured rather than rescaled — + a number nobody re-measured is not a measurement — and the parity harness, + the emitter's `--report` and `tests/test_emit_cargo.py` carry the current ones. + Consumed one target per crate (`//:_lib` → its archive + its generated FFI headers), one-for-one with CMake's `target_link_libraries` — **not** a shared `--start-group` over all ten archives, which is what I built first and @@ -793,7 +801,56 @@ Honest inventory of what stops this from being a clone-and-build. that says there is none. Same shape as the shim that could not fail: a control that is not enforced is indistinguishable from one that is not there. -9. **Two upstreamable Ladybird fixes — down from four, and that is the point of a +9. **A repin's first failure mode is a *loading-time* one, and no per-target check + can catch it.** Ulf's build at `71fb301a` did not fail to compile anything; it + failed to load: + + ``` + Error in glob: glob pattern 'Libraries/LibJS/BytecodeDef/**' didn't match + anything, but allow_empty is set to False + ``` + + Upstream `a32d9c9f` ("LibJS: Derive bytecodes from Flap handlers") deleted that + directory, and the pattern was a hardcoded string in a list literal in + `Meta/emit_cargo_bazel.py`. Three properties compound here: + + * `allow_empty = False` is **correct** and had to stay. Its opposite is the + reason the old `Build/full` shim packages could match nothing for weeks and + fail 1,600 actions later — a file list that *may* be empty proves nothing. + * A loading error has **no target to blame**, so nothing in the build graph can + report it and no test that builds or queries a target can reach it. + * The emitters that would have re-derived the pattern all ran green, because an + emitter that enumerates *kinds* of thing cannot notice a kind going away: + the same repin silently dropped `gen_Op` from `codegen_root.bzl`, and the + parity harness reported **`0 UNHANDLED`** while two generated headers had no + owner at all — its first-match-wins classifier bucketed the new Rust + generator as "resource staging", because that command's ninja rule is + ` … && cmake -E copy_if_different …` and the *exclusion* pattern + matched first. Every exclusion in a first-match classifier can silently + capture a command it was not written for, and the count that should have + caught it is computed after the capture. + + The fix is structural rather than a re-pin of the string: every crate directory + an `allow_empty = False` glob names is now **derived** — `Cargo.toml`'s + `members` closed over the manifests' `path =` dependencies (which is also how + the derivation reaches *outside* the workspace, since `libjs_rust` + build-depends on the `exclude`d `flapc`) — and `flapc`'s own extra inputs come + from scanning it for `include_str!`. So a deleted crate deletes its own glob + pattern. `tests/test_emit_cargo.py` guards it from both ends: the derivation + drops a pattern when the directory is removed, and **no module-level constant + in either emitter may hold a `/**` string** — the shape the bug took. + + The same consolidation (`libweb_css_rust` + `libweb_layout_rust` → + `libweb_rust`) also moved two things worth naming: two of the crate's 19 + generated files are **`.inc`, not `.h`**, and Bazel deletes an undeclared + `.inc` exactly as it deletes an undeclared header — so the suffix must not be + what decides whether a generated file is declared. And `build_rust_binary()` + takes `FEATURES` too (upstream's new `style-replay` is built + `FEATURES style-recording` from the *same crate* as `libweb_rust`'s staticlib), + which the parser only read on the `import_rust_crate` side: it would have built + a different binary than CMake does and said nothing. + +10. **Two upstreamable Ladybird fixes — down from four, and that is the point of a repin.** Two of the four are **fixed upstream** at this pin (`71fb301a`) and their patches are deleted, which is worth stating because a patch directory is a debt register that only shrinks if somebody re-reads it: diff --git a/examples/ladybird/workspace/BUILD.bazel b/examples/ladybird/workspace/BUILD.bazel index 0e5dc97..0acb139 100644 --- a/examples/ladybird/workspace/BUILD.bazel +++ b/examples/ladybird/workspace/BUILD.bazel @@ -9,7 +9,7 @@ load(":vcpkg_index.bzl", "VCPKG_DISTFILE_INDEX") package(default_visibility = ["//visibility:public"]) -# 24 generator genrules for this package: the IPC endpoints, LibJS Bytecode/Op, +# 25 generator genrules for this package: the IPC endpoints, LibJS Bytecode/Op, # LibHTTP's HSTS table, the Compositor WebGL replayer, the TIFF tag tables, the # two SPIR-V shader headers and the Flap interpreter assembly. Bazel now # GENERATES all of them instead of consuming CMake's prebuilt copies from @@ -112,6 +112,7 @@ cc_library( 'AK/StringConversions.cpp', 'AK/StringUtils.cpp', 'AK/StringView.cpp', + 'AK/ThreadID.cpp', 'AK/Time.cpp', 'AK/Utf16FlyString.cpp', 'AK/Utf16String.cpp', @@ -341,7 +342,7 @@ cc_library( ], ) -# === LibGC (shared_library, 19 TU) === +# === LibGC (shared_library, 20 TU) === cc_library( name = 'LibGC', srcs = [ @@ -353,6 +354,7 @@ cc_library( 'Libraries/LibGC/ConservativeRangeProvider.cpp', 'Libraries/LibGC/ConservativeVector.cpp', 'Libraries/LibGC/CrossHeapMember.cpp', + 'Libraries/LibGC/ExternalEntityTable.cpp', 'Libraries/LibGC/Heap.cpp', 'Libraries/LibGC/HeapBlock.cpp', 'Libraries/LibGC/HeapGroup.cpp', @@ -610,7 +612,6 @@ cc_library( cc_library( name = 'LibJS', srcs = [ - ':Libraries/LibJS/Bytecode/Op.cpp', ':Libraries/LibJS/Interpreter/interpreter_x86_64.S', 'Libraries/LibJS/Bytecode/Executable.cpp', 'Libraries/LibJS/Bytecode/IdentifierTable.cpp', @@ -626,6 +627,7 @@ cc_library( 'Libraries/LibJS/Contrib/Test262/GlobalObject.cpp', 'Libraries/LibJS/Contrib/Test262/IsHTMLDDA.cpp', 'Libraries/LibJS/CyclicModule.cpp', + 'Libraries/LibJS/Debugger.cpp', 'Libraries/LibJS/Heap/Cell.cpp', 'Libraries/LibJS/Interpreter/Interpreter.cpp', 'Libraries/LibJS/Interpreter/SlowPaths.cpp', @@ -910,12 +912,12 @@ cc_library( ], ) -# === LibMedia (shared_library, 45 TU) === +# === LibMedia (shared_library, 54 TU) === cc_library( name = 'LibMedia', srcs = [ - 'Libraries/LibMedia/Audio/AudioBuffer.cpp', 'Libraries/LibMedia/Audio/AudioDevices.cpp', + 'Libraries/LibMedia/Audio/AudioRingBuffer.cpp', 'Libraries/LibMedia/Audio/NullPlaybackStream.cpp', 'Libraries/LibMedia/Audio/PlaybackStream.cpp', 'Libraries/LibMedia/Audio/PlaybackStreamPulseAudio.cpp', @@ -942,8 +944,9 @@ cc_library( 'Libraries/LibMedia/FFmpeg/FFmpegHelpers.cpp', 'Libraries/LibMedia/FFmpeg/FFmpegIOContext.cpp', 'Libraries/LibMedia/FFmpeg/FFmpegVideoDecoder.cpp', - 'Libraries/LibMedia/GenericTimeProvider.cpp', 'Libraries/LibMedia/IncrementallyPopulatedStream.cpp', + 'Libraries/LibMedia/MediaTime.cpp', + 'Libraries/LibMedia/MonotonicMediaClock.cpp', 'Libraries/LibMedia/PlaybackManager.cpp', 'Libraries/LibMedia/PlaybackStates/BufferingStateHandler.cpp', 'Libraries/LibMedia/PlaybackStates/PausedStateHandler.cpp', @@ -955,10 +958,18 @@ cc_library( 'Libraries/LibMedia/Processors/AudioTimeStretchProcessor.cpp', 'Libraries/LibMedia/Producers/DecodedAudioProducer.cpp', 'Libraries/LibMedia/Producers/DecodedVideoProducer.cpp', + 'Libraries/LibMedia/Producers/RemoteVideoProducer.cpp', 'Libraries/LibMedia/Sinks/AudioPlaybackSink.cpp', 'Libraries/LibMedia/Sinks/DisplayingVideoSink.cpp', + 'Libraries/LibMedia/Sinks/RemoteVideoSink.cpp', 'Libraries/LibMedia/TimeRanges.cpp', + 'Libraries/LibMedia/VideoEdgeQueue.cpp', 'Libraries/LibMedia/VideoFrame.cpp', + 'Libraries/LibMedia/VideoFrameHandle.cpp', + 'Libraries/LibMedia/VideoFramePool.cpp', + 'Libraries/LibMedia/VideoPresentation/PresentedFramePage.cpp', + 'Libraries/LibMedia/VideoPresentation/VideoPresentationClientConnection.cpp', + 'Libraries/LibMedia/VideoPresentation/VideoPresentationServerConnection.cpp', ], hdrs = glob(["Libraries/LibMedia/**/*.h"], allow_empty = True), local_defines = ['LIBMEDIA_AUDIO_BACKEND=1', 'LibMedia_EXPORTS', '_REENTRANT'], @@ -1286,7 +1297,7 @@ cc_library( ], ) -# === LibDevTools (shared_library, 35 TU) === +# === LibDevTools (shared_library, 36 TU) === cc_library( name = 'LibDevTools', srcs = [ @@ -1323,6 +1334,7 @@ cc_library( 'Libraries/LibDevTools/Actors/WatcherActor.cpp', 'Libraries/LibDevTools/Connection.cpp', 'Libraries/LibDevTools/DevToolsServer.cpp', + 'Libraries/LibDevTools/FirefoxClient.cpp', 'Libraries/LibDevTools/IndexedDBSerialization.cpp', 'Libraries/LibDevTools/StorageHelpers.cpp', ], @@ -1333,6 +1345,7 @@ cc_library( deps = [ '//:AK', '//:LibCore', + '//:LibFileSystem', '//:LibHTTP', '//:LibURL', '//:all_source_headers', @@ -1340,11 +1353,12 @@ cc_library( ], ) -# === LibWebView (shared_library, 57 TU) === +# === LibWebView (shared_library, 64 TU) === cc_library( name = 'LibWebView', srcs = [ 'Libraries/LibWebView/Application.cpp', + 'Libraries/LibWebView/ApplyHistoryStep.cpp', 'Libraries/LibWebView/Attribute.cpp', 'Libraries/LibWebView/Autocomplete.cpp', 'Libraries/LibWebView/AutocompleteMuxer.cpp', @@ -1362,6 +1376,9 @@ cc_library( 'Libraries/LibWebView/DOMNodeProperties.cpp', 'Libraries/LibWebView/DictionaryLookup.cpp', 'Libraries/LibWebView/DownloadPresentation.cpp', + 'Libraries/LibWebView/DownloadSegmentation.cpp', + 'Libraries/LibWebView/DownloadStore.cpp', + 'Libraries/LibWebView/ExternalURLHandler.cpp', 'Libraries/LibWebView/FileDownloader.cpp', 'Libraries/LibWebView/Geolocation.cpp', 'Libraries/LibWebView/HSTSStore.cpp', @@ -1382,6 +1399,9 @@ cc_library( 'Libraries/LibWebView/Profile.cpp', 'Libraries/LibWebView/SearchEngine.cpp', 'Libraries/LibWebView/SessionHistory.cpp', + 'Libraries/LibWebView/SessionHistorySnapshotStorage.cpp', + 'Libraries/LibWebView/SessionHistoryTraversalQueue.cpp', + 'Libraries/LibWebView/SessionStore.cpp', 'Libraries/LibWebView/Settings.cpp', 'Libraries/LibWebView/SiteIsolation.cpp', 'Libraries/LibWebView/SiteIsolationManager.cpp', @@ -1803,7 +1823,8 @@ qt_plugin_tree( qt_conf( name = "qt_conf", ) -# === ladybird (executable, 21 TU) === +# === ladybird (executable, 24 TU) === +# UNKNOWN deps: ['Qt6Positioning', 'gio-2', 'gobject-2', 'glib-2', 'xkbcommon'] cc_binary( name = 'ladybird', srcs = [ @@ -1817,7 +1838,10 @@ cc_binary( 'UI/Qt/DevToolsBanner.cpp', 'UI/Qt/EventLoopImplementationQt.cpp', 'UI/Qt/EventLoopImplementationQtEventTarget.cpp', + 'UI/Qt/ExternalURLActivationToken.cpp', + 'UI/Qt/ExternalURLHandler.cpp', 'UI/Qt/FindInPageWidget.cpp', + 'UI/Qt/GeolocationProviderQt.cpp', 'UI/Qt/Icon.cpp', 'UI/Qt/LocationEdit.cpp', 'UI/Qt/Menu.cpp', @@ -1831,7 +1855,8 @@ cc_binary( 'UI/Qt/WindowControlButton.cpp', 'UI/Qt/main.cpp', ], - local_defines = ['QT_CORE_LIB', 'QT_GUI_LIB', 'QT_NO_DEBUG', 'QT_WIDGETS_LIB'], + local_defines = ['LADYBIRD_QT_HAVE_POSITIONING=1', 'QT_CORE_LIB', 'QT_GUI_LIB', 'QT_NO_DEBUG', 'QT_POSITIONING_LIB', 'QT_WIDGETS_LIB'], + copts = ['-pthread'], linkopts = ['-lGLX', '-lOpenGL', '-lvulkan'], data = [':qt_conf', ':qt_plugins'], deps = [ @@ -1845,6 +1870,7 @@ cc_binary( '//:LibMain', '//:LibRequests', '//:LibSync', + '//:LibThreading', '//:LibURL', '//:LibUnicode', '//:LibWakeLock', diff --git a/examples/ladybird/workspace/Libraries/LibWeb/BUILD.bazel b/examples/ladybird/workspace/Libraries/LibWeb/BUILD.bazel index 975a150..18af3fe 100644 --- a/examples/ladybird/workspace/Libraries/LibWeb/BUILD.bazel +++ b/examples/ladybird/workspace/Libraries/LibWeb/BUILD.bazel @@ -16,42 +16,43 @@ libweb_export_header() libweb_codegen() libweb_bindings_codegen() -# The four Rust crates that live INSIDE this package (LibWeb/Rust, -# LibWeb/CSS/Rust, LibWeb/Layout/Rust, LibWeb/ContentBlocker/Rust, plus -# HTML/Parser/Rust), exposed so the root package's cargo_ring() can declare them -# as cargo inputs. They are one cargo WORKSPACE with the crates at the repo root, -# but Bazel packages cut across it: glob() is package-relative, so the root -# package cannot see files under Libraries/LibWeb/ at all. Hence a filegroup on -# this side of the boundary rather than a glob on that side -- the alternative -# (making the root package own these files) would mean deleting this package. +# The Rust crates that live INSIDE this package, exposed so the root package's +# cargo_ring() can declare them as cargo inputs. They are one cargo WORKSPACE +# with the crates at the repo root, but Bazel packages cut across it: glob() is +# package-relative, so the root package cannot see files under Libraries/LibWeb/ +# at all. Hence a filegroup on this side of the boundary rather than a glob on +# that side -- the alternative (making the root package own these files) would +# mean deleting this package. +# +# The patterns are DERIVED from Cargo.toml (Meta/emit_cargo_bazel.py's +# crate_dirs(), the same list the root package globs), not written down. They +# used to be written down, and that broke the whole build: upstream consolidated +# libweb_css_rust and libweb_layout_rust into libweb_rust, so `CSS/Rust/**` and +# `Layout/Rust/**` matched nothing, and an allow_empty = False glob that matches +# nothing fails at LOADING time -- before any target exists to blame. filegroup( name = "rust_crate_srcs", - srcs = glob([ - "Rust/**", - "CSS/Rust/**", - "Layout/Rust/**", - "ContentBlocker/Rust/**", - "HTML/Parser/Rust/**", - ], allow_empty = False) + [ - # Non-Rust build-script inputs that live here too: libweb_css_rust's - # build.rs GENERATES Rust from these CSS data files, and libweb_rust's - # reads the HTML name headers + Entities.json. Taken from the reference - # build's cargo depfiles, so the list is measured rather than predicted. - "CSS/Enums.json", - "CSS/Keywords.json", - "CSS/LogicalPropertyGroups.json", - "CSS/Properties.json", - "CSS/PseudoClasses.json", - "CSS/PseudoElementPropertyGroups.txt", - "CSS/PseudoElements.json", - "CSS/Units.json", - "HTML/AttributeNames.h", - "HTML/Parser/Entities.json", - "HTML/TagNames.h", + srcs = glob(['ContentBlocker/Rust/**', 'HTML/Parser/Rust/**', 'Rust/**'], exclude = ['ContentBlocker/Rust/target/**', 'HTML/Parser/Rust/target/**', 'Rust/target/**'], allow_empty = False) + [ + # Non-Rust build-script inputs that live here too: libweb_rust's build.rs + # GENERATES Rust from the CSS data files and reads the HTML name headers + + # Entities.json. Taken from the reference build's cargo depfiles, so the + # list is measured rather than predicted. + 'CSS/Enums.json', + 'CSS/Keywords.json', + 'CSS/LogicalPropertyGroups.json', + 'CSS/Properties.json', + 'CSS/PseudoClasses.json', + 'CSS/PseudoElementPropertyGroups.txt', + 'CSS/PseudoElements.json', + 'CSS/TransformFunctions.json', + 'CSS/Units.json', + 'HTML/AttributeNames.h', + 'HTML/Parser/Entities.json', + 'HTML/TagNames.h', ], ) -# === LibWeb (shared_library, 1961 TU: 1273 checked-in + 688 generated) === +# === LibWeb (shared_library, 1922 TU: 1230 checked-in + 692 generated) === # SYSTEM libs (linkopts on binary): ['dl', 'pthread', 'vulkan'] cc_library( name = 'LibWeb', @@ -81,6 +82,8 @@ cc_library( 'Bindings/OptionConstructor.cpp', 'Bindings/PlatformObject.cpp', 'Bindings/PrincipalHostDefined.cpp', + 'Bindings/Wrappable.cpp', + 'Bindings/WrapperWorld.cpp', 'CSS/Angle.cpp', 'CSS/AnimationEvent.cpp', 'CSS/BooleanExpression.cpp', @@ -148,7 +151,7 @@ cc_library( 'CSS/CascadedProperties.cpp', 'CSS/Clip.cpp', 'CSS/ColorFunctionDescriptor.cpp', - 'CSS/ComputedProperties.cpp', + 'CSS/ComputedStyleWorkingSet.cpp', 'CSS/ComputedValues.cpp', 'CSS/ContainerQuery.cpp', 'CSS/CounterStyle.cpp', @@ -162,7 +165,6 @@ cc_library( 'CSS/EdgeRect.cpp', 'CSS/FeatureQuery.cpp', 'CSS/Fetch.cpp', - 'CSS/Filter.cpp', 'CSS/Flex.cpp', 'CSS/FontComputer.cpp', 'CSS/FontFace.cpp', @@ -171,28 +173,23 @@ cc_library( 'CSS/FontFeatureData.cpp', 'CSS/FontLoading.cpp', 'CSS/Frequency.cpp', + 'CSS/GeneratedContent.cpp', 'CSS/GridTrackPlacement.cpp', 'CSS/GridTrackSize.cpp', 'CSS/HypotheticalElement.cpp', 'CSS/Invalidation/AdoptedStyleSheetInvalidator.cpp', 'CSS/Invalidation/AttributeInvalidator.cpp', + 'CSS/Invalidation/ContainerQueryInvalidator.cpp', 'CSS/Invalidation/CustomElementInvalidator.cpp', 'CSS/Invalidation/ElementStateInvalidator.cpp', 'CSS/Invalidation/EmbeddedContentInvalidator.cpp', 'CSS/Invalidation/FormControlInvalidator.cpp', - 'CSS/Invalidation/HasMutationFeatureCollector.cpp', - 'CSS/Invalidation/HasMutationInvalidator.cpp', - 'CSS/Invalidation/InvalidationSetMatcher.cpp', 'CSS/Invalidation/LanguageInvalidator.cpp', 'CSS/Invalidation/LinkInvalidator.cpp', 'CSS/Invalidation/MediaQueryInvalidator.cpp', - 'CSS/Invalidation/NodeInvalidator.cpp', 'CSS/Invalidation/PartInvalidator.cpp', 'CSS/Invalidation/PseudoClassInvalidator.cpp', 'CSS/Invalidation/SlotInvalidator.cpp', - 'CSS/Invalidation/StructuralMutationInvalidator.cpp', - 'CSS/Invalidation/StyleInvalidator.cpp', - 'CSS/InvalidationSet.cpp', 'CSS/Length.cpp', 'CSS/LengthBox.cpp', 'CSS/MediaList.cpp', @@ -226,6 +223,7 @@ cc_library( 'CSS/PreferredColorScheme.cpp', 'CSS/Ratio.cpp', 'CSS/Resolution.cpp', + 'CSS/ResolvedTransform.cpp', 'CSS/RustStyleBridge.cpp', 'CSS/Screen.cpp', 'CSS/ScreenOrientation.cpp', @@ -235,8 +233,9 @@ cc_library( 'CSS/Serialize.cpp', 'CSS/Sizing.cpp', 'CSS/StyleComputer.cpp', + 'CSS/StyleEngineBridge.cpp', + 'CSS/StyleEngineInput.cpp', 'CSS/StyleInvalidation.cpp', - 'CSS/StyleInvalidationData.cpp', 'CSS/StyleProperty.cpp', 'CSS/StylePropertyMap.cpp', 'CSS/StylePropertyMapReadOnly.cpp', @@ -384,7 +383,6 @@ cc_library( 'Crypto/CryptoAlgorithms.cpp', 'Crypto/CryptoBindings.cpp', 'Crypto/CryptoKey.cpp', - 'Crypto/KeyAlgorithms.cpp', 'Crypto/SubtleCrypto.cpp', 'DOM/AbortController.cpp', 'DOM/AbortSignal.cpp', @@ -484,12 +482,10 @@ cc_library( 'Encoding/TextEncoderStream.cpp', 'EncryptedMediaExtensions/Algorithms.cpp', 'EncryptedMediaExtensions/MediaKeySystemAccess.cpp', - 'EncryptedMediaExtensions/NavigatorEncryptedMediaExtensionsPartial.cpp', 'EntriesAPI/FileSystemEntry.cpp', 'EventTiming/PerformanceEventTiming.cpp', 'Fetch/Body.cpp', 'Fetch/BodyInit.cpp', - 'Fetch/Enums.cpp', 'Fetch/FetchMethod.cpp', 'Fetch/Fetching/Checks.cpp', 'Fetch/Fetching/FetchedDataReceiver.cpp', @@ -582,9 +578,11 @@ cc_library( 'HTML/CrossOrigin/AbstractOperations.cpp', 'HTML/CrossOrigin/Reporting.cpp', 'HTML/CrossProcessId.cpp', + 'HTML/CustomElements/CustomElementAlgorithms.cpp', 'HTML/CustomElements/CustomElementDefinition.cpp', 'HTML/CustomElements/CustomElementName.cpp', 'HTML/CustomElements/CustomElementReactionNames.cpp', + 'HTML/CustomElements/CustomElementReactions.cpp', 'HTML/CustomElements/CustomElementRegistry.cpp', 'HTML/CustomElements/CustomStateSet.cpp', 'HTML/DOMParser.cpp', @@ -697,6 +695,7 @@ cc_library( 'HTML/HTMLVideoElement.cpp', 'HTML/HashChangeEvent.cpp', 'HTML/History.cpp', + 'HTML/HistoryOperation.cpp', 'HTML/HyperlinkElementUtils.cpp', 'HTML/ImageBitmap.cpp', 'HTML/ImageData.cpp', @@ -717,6 +716,7 @@ cc_library( 'HTML/NavigableContainer.cpp', 'HTML/NavigateEvent.cpp', 'HTML/Navigation.cpp', + 'HTML/NavigationActivation.cpp', 'HTML/NavigationCurrentEntryChangeEvent.cpp', 'HTML/NavigationDestination.cpp', 'HTML/NavigationHistoryEntry.cpp', @@ -733,6 +733,7 @@ cc_library( 'HTML/OffscreenCanvas.cpp', 'HTML/OffscreenCanvasRenderingContext2D.cpp', 'HTML/POSTResource.cpp', + 'HTML/PageSwapEvent.cpp', 'HTML/PageTransitionEvent.cpp', 'HTML/Parser/Entities.cpp', 'HTML/Parser/HTMLEncodingDetection.cpp', @@ -769,13 +770,13 @@ cc_library( 'HTML/Scripting/SimilarOriginWindowAgent.cpp', 'HTML/Scripting/TemporaryExecutionContext.cpp', 'HTML/Scripting/WindowEnvironmentSettingsObject.cpp', + 'HTML/Scripting/WindowRealm.cpp', 'HTML/Scripting/WorkerAgent.cpp', 'HTML/Scripting/WorkerEnvironmentSettingsObject.cpp', 'HTML/SelectItem.cpp', 'HTML/SelectedFile.cpp', 'HTML/SerializedPolicyContainer.cpp', 'HTML/SessionHistoryEntry.cpp', - 'HTML/SessionHistoryTraversalQueue.cpp', 'HTML/SharedResourceRequest.cpp', 'HTML/SharedWorker.cpp', 'HTML/SharedWorkerGlobalScope.cpp', @@ -852,59 +853,16 @@ cc_library( 'Internals/XRTest.cpp', 'IntersectionObserver/IntersectionObserver.cpp', 'IntersectionObserver/IntersectionObserverEntry.cpp', - 'Layout/AudioBox.cpp', - 'Layout/AvailableSpace.cpp', 'Layout/BlockContainer.cpp', - 'Layout/BlockFormattingContext.cpp', 'Layout/Box.cpp', - 'Layout/BreakNode.cpp', - 'Layout/CanvasBox.cpp', - 'Layout/CheckBox.cpp', - 'Layout/FieldSetBox.cpp', - 'Layout/FlexFormattingContext.cpp', - 'Layout/FormattingContext.cpp', - 'Layout/GridFormattingContext.cpp', - 'Layout/ImageBox.cpp', 'Layout/ImageProvider.cpp', - 'Layout/InlineFormattingContext.cpp', - 'Layout/InlineLevelIterator.cpp', - 'Layout/InlineNode.cpp', - 'Layout/LayoutState.cpp', - 'Layout/LegendBox.cpp', - 'Layout/LineBox.cpp', - 'Layout/LineBoxFragment.cpp', - 'Layout/LineBuilder.cpp', - 'Layout/ListItemBox.cpp', - 'Layout/ListItemMarkerBox.cpp', - 'Layout/NavigableContainerViewport.cpp', + 'Layout/LayoutRustBridge.cpp', 'Layout/Node.cpp', 'Layout/NodeArena.cpp', - 'Layout/RadioButton.cpp', - 'Layout/RangeInputBox.cpp', - 'Layout/ReplacedBox.cpp', - 'Layout/ReplacedWithChildrenFormattingContext.cpp', - 'Layout/SVGBox.cpp', - 'Layout/SVGClipBox.cpp', - 'Layout/SVGForeignObjectBox.cpp', - 'Layout/SVGFormattingContext.cpp', - 'Layout/SVGGeometryBox.cpp', - 'Layout/SVGGraphicsBox.cpp', - 'Layout/SVGImageBox.cpp', - 'Layout/SVGMaskBox.cpp', - 'Layout/SVGPatternBox.cpp', - 'Layout/SVGSVGBox.cpp', - 'Layout/SVGTextBox.cpp', - 'Layout/SVGTextPathBox.cpp', 'Layout/ScrollableOverflow.cpp', - 'Layout/TableFormattingContext.cpp', - 'Layout/TableGrid.cpp', - 'Layout/TableWrapper.cpp', - 'Layout/TextAreaBox.cpp', - 'Layout/TextInputBox.cpp', 'Layout/TextNode.cpp', 'Layout/TextOffsetMapping.cpp', 'Layout/TreeBuilder.cpp', - 'Layout/VideoBox.cpp', 'Layout/Viewport.cpp', 'Loader/ContentBlocker.cpp', 'Loader/DownloadFilename.cpp', @@ -977,7 +935,6 @@ cc_library( 'Painting/HitTestDisplayList.cpp', 'Painting/ImagePaintable.cpp', 'Painting/InlinePaintable.cpp', - 'Painting/MarkerPaintable.cpp', 'Painting/NavigableContainerViewportPaintable.cpp', 'Painting/PaintStyle.cpp', 'Painting/Paintable.cpp', @@ -1078,6 +1035,7 @@ cc_library( 'SVG/SVGImageElement.cpp', 'SVG/SVGLength.cpp', 'SVG/SVGLengthList.cpp', + 'SVG/SVGLengthValue.cpp', 'SVG/SVGLineElement.cpp', 'SVG/SVGLinearGradientElement.cpp', 'SVG/SVGList.cpp', @@ -1359,8 +1317,6 @@ cc_library( '//:LibXML', '//:all_source_headers', '//:libweb_content_blocker_rust_lib', - '//:libweb_css_rust_lib', - '//:libweb_layout_rust_lib', '//:libweb_rust_lib', '//Meta/vcpkg:SDL3', '//Meta/vcpkg:crypto', diff --git a/examples/ladybird/workspace/Libraries/LibWeb/codegen.bzl b/examples/ladybird/workspace/Libraries/LibWeb/codegen.bzl index 0fa57f7..0b6273a 100644 --- a/examples/ladybird/workspace/Libraries/LibWeb/codegen.bzl +++ b/examples/ladybird/workspace/Libraries/LibWeb/codegen.bzl @@ -160,13 +160,13 @@ def libweb_codegen(): ) def libweb_bindings_codegen(): - # Mega-genrule: 661 IDL args (659 static + 2 generated) -> 1331 files. - # Byte-identical to CMake, proven 1331/1331 by Meta/bazel_parity_harness.py. - # srcs is the FULL DEPENDS closure (661 .idl), not just the 661 args: + # Mega-genrule: 663 IDL args (661 static + 2 generated) -> 1340 files. + # Byte-identical to CMake, proven 1340/1340 by Meta/bazel_parity_harness.py. + # srcs is the FULL DEPENDS closure (663 .idl), not just the 663 args: # the generator follows `includes`/partial interfaces between .idl files. native.genrule( name = 'gen_bindings', - srcs = ['ARIA/ARIAMixin.idl', 'Animations/Animatable.idl', 'Animations/Animation.idl', 'Animations/AnimationEffect.idl', 'Animations/AnimationPlaybackEvent.idl', 'Animations/AnimationTimeline.idl', 'Animations/DocumentTimeline.idl', 'Animations/KeyframeEffect.idl', 'Animations/ScrollTimeline.idl', 'CSS/AnimationEvent.idl', 'CSS/CSS.idl', 'CSS/CSSAnimation.idl', 'CSS/CSSConditionRule.idl', 'CSS/CSSContainerRule.idl', 'CSS/CSSCounterStyleRule.idl', 'CSS/CSSFontFaceDescriptors.idl', 'CSS/CSSFontFaceRule.idl', 'CSS/CSSFontFeatureValuesMap.idl', 'CSS/CSSFontFeatureValuesRule.idl', 'CSS/CSSFunctionDeclarations.idl', 'CSS/CSSFunctionDescriptors.idl', 'CSS/CSSFunctionRule.idl', 'CSS/CSSGroupingRule.idl', 'CSS/CSSImageValue.idl', 'CSS/CSSImportRule.idl', 'CSS/CSSKeyframeRule.idl', 'CSS/CSSKeyframesRule.idl', 'CSS/CSSKeywordValue.idl', 'CSS/CSSLayerBlockRule.idl', 'CSS/CSSLayerStatementRule.idl', 'CSS/CSSMarginRule.idl', 'CSS/CSSMathClamp.idl', 'CSS/CSSMathInvert.idl', 'CSS/CSSMathMax.idl', 'CSS/CSSMathMin.idl', 'CSS/CSSMathNegate.idl', 'CSS/CSSMathProduct.idl', 'CSS/CSSMathSum.idl', 'CSS/CSSMathValue.idl', 'CSS/CSSMatrixComponent.idl', 'CSS/CSSMediaRule.idl', 'CSS/CSSNamespaceRule.idl', 'CSS/CSSNestedDeclarations.idl', 'CSS/CSSNumericArray.idl', 'CSS/CSSNumericValue.idl', 'CSS/CSSPageDescriptors.idl', 'CSS/CSSPageRule.idl', 'CSS/CSSPerspective.idl', 'CSS/CSSPropertyRule.idl', 'CSS/CSSRotate.idl', 'CSS/CSSRule.idl', 'CSS/CSSRuleList.idl', 'CSS/CSSScale.idl', 'CSS/CSSScopeRule.idl', 'CSS/CSSSkew.idl', 'CSS/CSSSkewX.idl', 'CSS/CSSSkewY.idl', 'CSS/CSSStyleDeclaration.idl', 'CSS/CSSStyleProperties.idl', 'CSS/CSSStyleRule.idl', 'CSS/CSSStyleSheet.idl', 'CSS/CSSStyleValue.idl', 'CSS/CSSSupportsRule.idl', 'CSS/CSSTransformComponent.idl', 'CSS/CSSTransformValue.idl', 'CSS/CSSTransition.idl', 'CSS/CSSTranslate.idl', 'CSS/CSSUnitValue.idl', 'CSS/CSSUnparsedValue.idl', 'CSS/CSSVariableReferenceValue.idl', 'CSS/ElementCSSInlineStyle.idl', 'CSS/FontFace.idl', 'CSS/FontFaceSet.idl', 'CSS/FontFaceSetLoadEvent.idl', 'CSS/LinkStyle.idl', 'CSS/MediaList.idl', 'CSS/MediaQueryList.idl', 'CSS/MediaQueryListEvent.idl', 'CSS/Screen.idl', 'CSS/ScreenOrientation.idl', 'CSS/StylePropertyMap.idl', 'CSS/StylePropertyMapReadOnly.idl', 'CSS/StyleSheet.idl', 'CSS/StyleSheetList.idl', 'CSS/TransitionEvent.idl', 'CSS/VisualViewport.idl', 'Clipboard/Clipboard.idl', 'Clipboard/ClipboardEvent.idl', 'Clipboard/ClipboardItem.idl', 'Compression/CompressionStream.idl', 'Compression/DecompressionStream.idl', 'ContentSecurityPolicy/SecurityPolicyViolationEvent.idl', 'CookieStore/CookieChangeEvent.idl', 'CookieStore/CookieStore.idl', 'CredentialManagement/Credential.idl', 'CredentialManagement/CredentialsContainer.idl', 'CredentialManagement/FederatedCredential.idl', 'CredentialManagement/PasswordCredential.idl', 'Crypto/Crypto.idl', 'Crypto/CryptoKey.idl', 'Crypto/SubtleCrypto.idl', 'DOM/AbortController.idl', 'DOM/AbortSignal.idl', 'DOM/AbstractRange.idl', 'DOM/Attr.idl', 'DOM/CDATASection.idl', 'DOM/CaretPosition.idl', 'DOM/CharacterData.idl', 'DOM/ChildNode.idl', 'DOM/Comment.idl', 'DOM/CustomEvent.idl', 'DOM/DOMImplementation.idl', 'DOM/DOMTokenList.idl', 'DOM/Document.idl', 'DOM/DocumentFragment.idl', 'DOM/DocumentOrShadowRoot.idl', 'DOM/DocumentType.idl', 'DOM/Element.idl', 'DOM/Event.idl', 'DOM/EventHandler.idl', 'DOM/EventListener.idl', 'DOM/EventTarget.idl', 'DOM/HTMLCollection.idl', 'DOM/MutationObserver.idl', 'DOM/MutationRecord.idl', 'DOM/NamedNodeMap.idl', 'DOM/Node.idl', 'DOM/NodeFilter.idl', 'DOM/NodeIterator.idl', 'DOM/NodeList.idl', 'DOM/ParentNode.idl', 'DOM/ProcessingInstruction.idl', 'DOM/Range.idl', 'DOM/ShadowRoot.idl', 'DOM/Slottable.idl', 'DOM/StaticRange.idl', 'DOM/Text.idl', 'DOM/TreeWalker.idl', 'DOM/XMLDocument.idl', 'DOMURL/DOMURL.idl', 'DOMURL/Origin.idl', 'DOMURL/URLSearchParams.idl', 'Encoding/TextDecoder.idl', 'Encoding/TextDecoderCommon.idl', 'Encoding/TextDecoderStream.idl', 'Encoding/TextEncoder.idl', 'Encoding/TextEncoderCommon.idl', 'Encoding/TextEncoderStream.idl', 'EncryptedMediaExtensions/MediaKeySystemAccess.idl', 'EntriesAPI/FileSystemEntry.idl', 'EventTiming/PerformanceEventTiming.idl', 'Fetch/Body.idl', 'Fetch/BodyInit.idl', 'Fetch/Headers.idl', 'Fetch/Request.idl', 'Fetch/Response.idl', 'FileAPI/Blob.idl', 'FileAPI/File.idl', 'FileAPI/FileList.idl', 'FileAPI/FileReader.idl', 'FileAPI/FileReaderSync.idl', 'Fullscreen/DocumentExtensions.idl', 'Fullscreen/DocumentOrShadowRootExtensions.idl', 'Fullscreen/ElementExtensions.idl', 'GPC/GlobalPrivacyControl.idl', 'Gamepad/Gamepad.idl', 'Gamepad/GamepadButton.idl', 'Gamepad/GamepadEvent.idl', 'Gamepad/GamepadHapticActuator.idl', 'Geolocation/Geolocation.idl', 'Geolocation/GeolocationCoordinates.idl', 'Geolocation/GeolocationPosition.idl', 'Geolocation/GeolocationPositionError.idl', 'Geometry/DOMMatrix.idl', 'Geometry/DOMMatrixReadOnly.idl', 'Geometry/DOMPoint.idl', 'Geometry/DOMPointReadOnly.idl', 'Geometry/DOMQuad.idl', 'Geometry/DOMRect.idl', 'Geometry/DOMRectList.idl', 'Geometry/DOMRectReadOnly.idl', 'HTML/AbstractWorker.idl', 'HTML/AnimationFrameProvider.idl', 'HTML/AudioTrack.idl', 'HTML/AudioTrackList.idl', 'HTML/BarProp.idl', 'HTML/BeforeUnloadEvent.idl', 'HTML/BroadcastChannel.idl', 'HTML/Canvas/CanvasCompositing.idl', 'HTML/Canvas/CanvasDrawImage.idl', 'HTML/Canvas/CanvasDrawPath.idl', 'HTML/Canvas/CanvasFillStrokeStyles.idl', 'HTML/Canvas/CanvasFilters.idl', 'HTML/Canvas/CanvasImageData.idl', 'HTML/Canvas/CanvasImageSmoothing.idl', 'HTML/Canvas/CanvasPath.idl', 'HTML/Canvas/CanvasPathDrawingStyles.idl', 'HTML/Canvas/CanvasRect.idl', 'HTML/Canvas/CanvasSettings.idl', 'HTML/Canvas/CanvasShadowStyles.idl', 'HTML/Canvas/CanvasState.idl', 'HTML/Canvas/CanvasText.idl', 'HTML/Canvas/CanvasTextDrawingStyles.idl', 'HTML/Canvas/CanvasTransform.idl', 'HTML/Canvas/CanvasUserInterface.idl', 'HTML/Canvas/OffscreenCanvasBase.idl', 'HTML/CanvasGradient.idl', 'HTML/CanvasPattern.idl', 'HTML/CanvasRenderingContext2D.idl', 'HTML/CanvasRenderingContext2DSettings.idl', 'HTML/CloseEvent.idl', 'HTML/CloseWatcher.idl', 'HTML/CommandEvent.idl', 'HTML/CustomElements/CustomElementRegistry.idl', 'HTML/CustomElements/CustomStateSet.idl', 'HTML/DOMParser.idl', 'HTML/DOMStringList.idl', 'HTML/DOMStringMap.idl', 'HTML/DataTransfer.idl', 'HTML/DataTransferItem.idl', 'HTML/DataTransferItemList.idl', 'HTML/DedicatedWorkerGlobalScope.idl', 'HTML/DragEvent.idl', 'HTML/ElementInternals.idl', 'HTML/ErrorEvent.idl', 'HTML/EventSource.idl', 'HTML/External.idl', 'HTML/FormDataEvent.idl', 'HTML/HTMLAllCollection.idl', 'HTML/HTMLAnchorElement.idl', 'HTML/HTMLAreaElement.idl', 'HTML/HTMLAudioElement.idl', 'HTML/HTMLBRElement.idl', 'HTML/HTMLBaseElement.idl', 'HTML/HTMLBodyElement.idl', 'HTML/HTMLButtonElement.idl', 'HTML/HTMLCanvasElement.idl', 'HTML/HTMLDListElement.idl', 'HTML/HTMLDataElement.idl', 'HTML/HTMLDataListElement.idl', 'HTML/HTMLDetailsElement.idl', 'HTML/HTMLDialogElement.idl', 'HTML/HTMLDirectoryElement.idl', 'HTML/HTMLDivElement.idl', 'HTML/HTMLDocument.idl', 'HTML/HTMLElement.idl', 'HTML/HTMLEmbedElement.idl', 'HTML/HTMLFieldSetElement.idl', 'HTML/HTMLFontElement.idl', 'HTML/HTMLFormControlsCollection.idl', 'HTML/HTMLFormElement.idl', 'HTML/HTMLFrameElement.idl', 'HTML/HTMLFrameSetElement.idl', 'HTML/HTMLHRElement.idl', 'HTML/HTMLHeadElement.idl', 'HTML/HTMLHeadingElement.idl', 'HTML/HTMLHtmlElement.idl', 'HTML/HTMLHyperlinkElementUtils.idl', 'HTML/HTMLIFrameElement.idl', 'HTML/HTMLImageElement.idl', 'HTML/HTMLInputElement.idl', 'HTML/HTMLLIElement.idl', 'HTML/HTMLLabelElement.idl', 'HTML/HTMLLegendElement.idl', 'HTML/HTMLLinkElement.idl', 'HTML/HTMLMapElement.idl', 'HTML/HTMLMarqueeElement.idl', 'HTML/HTMLMediaElement.idl', 'HTML/HTMLMenuElement.idl', 'HTML/HTMLMetaElement.idl', 'HTML/HTMLMeterElement.idl', 'HTML/HTMLModElement.idl', 'HTML/HTMLOListElement.idl', 'HTML/HTMLObjectElement.idl', 'HTML/HTMLOptGroupElement.idl', 'HTML/HTMLOptionElement.idl', 'HTML/HTMLOptionsCollection.idl', 'HTML/HTMLOrSVGOrMathMLElement.idl', 'HTML/HTMLOutputElement.idl', 'HTML/HTMLParagraphElement.idl', 'HTML/HTMLParamElement.idl', 'HTML/HTMLPictureElement.idl', 'HTML/HTMLPreElement.idl', 'HTML/HTMLProgressElement.idl', 'HTML/HTMLQuoteElement.idl', 'HTML/HTMLScriptElement.idl', 'HTML/HTMLSelectElement.idl', 'HTML/HTMLSelectedContentElement.idl', 'HTML/HTMLSlotElement.idl', 'HTML/HTMLSourceElement.idl', 'HTML/HTMLSpanElement.idl', 'HTML/HTMLStyleElement.idl', 'HTML/HTMLTableCaptionElement.idl', 'HTML/HTMLTableCellElement.idl', 'HTML/HTMLTableColElement.idl', 'HTML/HTMLTableElement.idl', 'HTML/HTMLTableRowElement.idl', 'HTML/HTMLTableSectionElement.idl', 'HTML/HTMLTemplateElement.idl', 'HTML/HTMLTextAreaElement.idl', 'HTML/HTMLTimeElement.idl', 'HTML/HTMLTitleElement.idl', 'HTML/HTMLTrackElement.idl', 'HTML/HTMLUListElement.idl', 'HTML/HTMLUnknownElement.idl', 'HTML/HTMLVideoElement.idl', 'HTML/HashChangeEvent.idl', 'HTML/History.idl', 'HTML/HyperlinkElementUtils.idl', 'HTML/ImageBitmap.idl', 'HTML/ImageData.idl', 'HTML/Location.idl', 'HTML/MediaError.idl', 'HTML/MessageChannel.idl', 'HTML/MessageEvent.idl', 'HTML/MessagePort.idl', 'HTML/MimeType.idl', 'HTML/MimeTypeArray.idl', 'HTML/NavigateEvent.idl', 'HTML/Navigation.idl', 'HTML/NavigationCurrentEntryChangeEvent.idl', 'HTML/NavigationDestination.idl', 'HTML/NavigationHistoryEntry.idl', 'HTML/NavigationTransition.idl', 'HTML/NavigationType.idl', 'HTML/Navigator.idl', 'HTML/NavigatorBeacon.idl', 'HTML/NavigatorConcurrentHardware.idl', 'HTML/NavigatorDeviceMemory.idl', 'HTML/NavigatorID.idl', 'HTML/NavigatorLanguage.idl', 'HTML/NavigatorOnLine.idl', 'HTML/OffscreenCanvas.idl', 'HTML/OffscreenCanvasRenderingContext2D.idl', 'HTML/PageTransitionEvent.idl', 'HTML/Path2D.idl', 'HTML/Plugin.idl', 'HTML/PluginArray.idl', 'HTML/PopStateEvent.idl', 'HTML/PopoverTargetAttributes.idl', 'HTML/PredefinedColorSpace.idl', 'HTML/PromiseRejectionEvent.idl', 'HTML/RadioNodeList.idl', 'HTML/Scripting/Fetching.idl', 'HTML/SharedWorker.idl', 'HTML/SharedWorkerGlobalScope.idl', 'HTML/Storage.idl', 'HTML/StorageEvent.idl', 'HTML/SubmitEvent.idl', 'HTML/TextMetrics.idl', 'HTML/TextTrack.idl', 'HTML/TextTrackCue.idl', 'HTML/TextTrackCueList.idl', 'HTML/TextTrackList.idl', 'HTML/TimeRanges.idl', 'HTML/ToggleEvent.idl', 'HTML/TrackEvent.idl', 'HTML/UniversalGlobalScope.idl', 'HTML/UserActivation.idl', 'HTML/ValidityState.idl', 'HTML/VideoTrack.idl', 'HTML/VideoTrackList.idl', 'HTML/Window.idl', 'HTML/WindowDeprecated.idl', 'HTML/WindowLocalStorage.idl', 'HTML/WindowOrWorkerGlobalScope.idl', 'HTML/WindowSessionStorage.idl', 'HTML/Worker.idl', 'HTML/WorkerGlobalScope.idl', 'HTML/WorkerLocation.idl', 'HTML/WorkerNavigator.idl', 'HTML/WorkletGlobalScope.idl', 'HTML/XMLSerializer.idl', 'HighResolutionTime/DOMHighResTimeStamp.idl', 'HighResolutionTime/EpochTimeStamp.idl', 'HighResolutionTime/Performance.idl', 'IndexedDB/IDBCursor.idl', 'IndexedDB/IDBCursorWithValue.idl', 'IndexedDB/IDBDatabase.idl', 'IndexedDB/IDBFactory.idl', 'IndexedDB/IDBIndex.idl', 'IndexedDB/IDBKeyRange.idl', 'IndexedDB/IDBObjectStore.idl', 'IndexedDB/IDBOpenDBRequest.idl', 'IndexedDB/IDBRecord.idl', 'IndexedDB/IDBRequest.idl', 'IndexedDB/IDBTransaction.idl', 'IndexedDB/IDBVersionChangeEvent.idl', 'Internals/FakeXRDevice.idl', 'Internals/InternalAnimationTimeline.idl', 'Internals/InternalGamepad.idl', 'Internals/Internals.idl', 'Internals/WebUI.idl', 'Internals/XRTest.idl', 'IntersectionObserver/IntersectionObserver.idl', 'IntersectionObserver/IntersectionObserverEntry.idl', 'MathML/MathMLAnchorElement.idl', 'MathML/MathMLElement.idl', 'MediaCapabilitiesAPI/MediaCapabilities.idl', 'MediaCapture/MediaDeviceInfo.idl', 'MediaCapture/MediaDevices.idl', 'MediaCapture/MediaStream.idl', 'MediaCapture/MediaStreamConstraints.idl', 'MediaCapture/MediaStreamTrack.idl', 'MediaCapture/MediaStreamTrackEvent.idl', 'MediaSourceExtensions/BufferedChangeEvent.idl', 'MediaSourceExtensions/ManagedMediaSource.idl', 'MediaSourceExtensions/ManagedSourceBuffer.idl', 'MediaSourceExtensions/MediaSource.idl', 'MediaSourceExtensions/MediaSourceHandle.idl', 'MediaSourceExtensions/SourceBuffer.idl', 'MediaSourceExtensions/SourceBufferList.idl', 'NavigationTiming/PerformanceExtensions.idl', 'NavigationTiming/PerformanceNavigation.idl', 'NavigationTiming/PerformanceTiming.idl', 'NotificationsAPI/Notification.idl', 'PerformanceTimeline/PerformanceEntry.idl', 'PerformanceTimeline/PerformanceObserver.idl', 'PerformanceTimeline/PerformanceObserverEntryList.idl', 'PermissionsAPI/PermissionStatus.idl', 'PermissionsAPI/Permissions.idl', 'RequestIdleCallback/IdleDeadline.idl', 'RequestIdleCallback/IdleRequest.idl', 'ResizeObserver/ResizeObserver.idl', 'ResizeObserver/ResizeObserverEntry.idl', 'ResizeObserver/ResizeObserverSize.idl', 'ResourceTiming/PerformanceResourceTiming.idl', 'SVG/SVGAElement.idl', 'SVG/SVGAnimatedEnumeration.idl', 'SVG/SVGAnimatedInteger.idl', 'SVG/SVGAnimatedLength.idl', 'SVG/SVGAnimatedLengthList.idl', 'SVG/SVGAnimatedNumber.idl', 'SVG/SVGAnimatedNumberList.idl', 'SVG/SVGAnimatedRect.idl', 'SVG/SVGAnimatedString.idl', 'SVG/SVGAnimatedTransformList.idl', 'SVG/SVGAnimationElement.idl', 'SVG/SVGCircleElement.idl', 'SVG/SVGClipPathElement.idl', 'SVG/SVGComponentTransferFunctionElement.idl', 'SVG/SVGDefsElement.idl', 'SVG/SVGDescElement.idl', 'SVG/SVGElement.idl', 'SVG/SVGEllipseElement.idl', 'SVG/SVGFEBlendElement.idl', 'SVG/SVGFEColorMatrixElement.idl', 'SVG/SVGFEComponentTransferElement.idl', 'SVG/SVGFECompositeElement.idl', 'SVG/SVGFEDisplacementMapElement.idl', 'SVG/SVGFEDropShadowElement.idl', 'SVG/SVGFEFloodElement.idl', 'SVG/SVGFEFuncAElement.idl', 'SVG/SVGFEFuncBElement.idl', 'SVG/SVGFEFuncGElement.idl', 'SVG/SVGFEFuncRElement.idl', 'SVG/SVGFEGaussianBlurElement.idl', 'SVG/SVGFEImageElement.idl', 'SVG/SVGFEMergeElement.idl', 'SVG/SVGFEMergeNodeElement.idl', 'SVG/SVGFEMorphologyElement.idl', 'SVG/SVGFEOffsetElement.idl', 'SVG/SVGFETurbulenceElement.idl', 'SVG/SVGFilterElement.idl', 'SVG/SVGFilterPrimitiveStandardAttributes.idl', 'SVG/SVGFitToViewBox.idl', 'SVG/SVGForeignObjectElement.idl', 'SVG/SVGGElement.idl', 'SVG/SVGGeometryElement.idl', 'SVG/SVGGradientElement.idl', 'SVG/SVGGraphicsElement.idl', 'SVG/SVGImageElement.idl', 'SVG/SVGLength.idl', 'SVG/SVGLengthList.idl', 'SVG/SVGLineElement.idl', 'SVG/SVGLinearGradientElement.idl', 'SVG/SVGMaskElement.idl', 'SVG/SVGMetadataElement.idl', 'SVG/SVGNumber.idl', 'SVG/SVGNumberList.idl', 'SVG/SVGPathElement.idl', 'SVG/SVGPatternElement.idl', 'SVG/SVGPolygonElement.idl', 'SVG/SVGPolylineElement.idl', 'SVG/SVGRadialGradientElement.idl', 'SVG/SVGRectElement.idl', 'SVG/SVGSVGElement.idl', 'SVG/SVGScriptElement.idl', 'SVG/SVGStopElement.idl', 'SVG/SVGStyleElement.idl', 'SVG/SVGSwitchElement.idl', 'SVG/SVGSymbolElement.idl', 'SVG/SVGTSpanElement.idl', 'SVG/SVGTextContentElement.idl', 'SVG/SVGTextElement.idl', 'SVG/SVGTextPathElement.idl', 'SVG/SVGTextPositioningElement.idl', 'SVG/SVGTitleElement.idl', 'SVG/SVGTransform.idl', 'SVG/SVGTransformList.idl', 'SVG/SVGURIReference.idl', 'SVG/SVGUnitTypes.idl', 'SVG/SVGUseElement.idl', 'SVG/SVGViewElement.idl', 'Selection/Selection.idl', 'Serial/Serial.idl', 'Serial/SerialPort.idl', 'ServiceWorker/Cache.idl', 'ServiceWorker/CacheStorage.idl', 'ServiceWorker/ServiceWorker.idl', 'ServiceWorker/ServiceWorkerContainer.idl', 'ServiceWorker/ServiceWorkerGlobalScope.idl', 'ServiceWorker/ServiceWorkerRegistration.idl', 'Speech/SpeechGrammar.idl', 'Speech/SpeechGrammarList.idl', 'Speech/SpeechRecognition.idl', 'Speech/SpeechRecognitionAlternative.idl', 'Speech/SpeechRecognitionEvent.idl', 'Speech/SpeechRecognitionPhrase.idl', 'Speech/SpeechRecognitionResult.idl', 'Speech/SpeechRecognitionResultList.idl', 'Speech/SpeechSynthesis.idl', 'Speech/SpeechSynthesisUtterance.idl', 'Speech/SpeechSynthesisVoice.idl', 'StorageAPI/NavigatorStorage.idl', 'StorageAPI/StorageManager.idl', 'Streams/ByteLengthQueuingStrategy.idl', 'Streams/CountQueuingStrategy.idl', 'Streams/GenericTransformStream.idl', 'Streams/QueuingStrategy.idl', 'Streams/QueuingStrategyInit.idl', 'Streams/ReadableByteStreamController.idl', 'Streams/ReadableStream.idl', 'Streams/ReadableStreamBYOBReader.idl', 'Streams/ReadableStreamBYOBRequest.idl', 'Streams/ReadableStreamDefaultController.idl', 'Streams/ReadableStreamDefaultReader.idl', 'Streams/ReadableStreamGenericReader.idl', 'Streams/TransformStream.idl', 'Streams/TransformStreamDefaultController.idl', 'Streams/Transformer.idl', 'Streams/UnderlyingSink.idl', 'Streams/UnderlyingSource.idl', 'Streams/WritableStream.idl', 'Streams/WritableStreamDefaultController.idl', 'Streams/WritableStreamDefaultWriter.idl', 'TrustedTypes/TrustedHTML.idl', 'TrustedTypes/TrustedScript.idl', 'TrustedTypes/TrustedScriptURL.idl', 'TrustedTypes/TrustedTypePolicy.idl', 'TrustedTypes/TrustedTypePolicyFactory.idl', 'UIEvents/CompositionEvent.idl', 'UIEvents/EventModifier.idl', 'UIEvents/FocusEvent.idl', 'UIEvents/InputEvent.idl', 'UIEvents/KeyboardEvent.idl', 'UIEvents/MouseEvent.idl', 'UIEvents/PointerEvent.idl', 'UIEvents/PointerEventHandlers.idl', 'UIEvents/TextEvent.idl', 'UIEvents/UIEvent.idl', 'UIEvents/WheelEvent.idl', 'URLPattern/URLPattern.idl', 'UserTiming/PerformanceMark.idl', 'UserTiming/PerformanceMeasure.idl', 'ViewTransition/ViewTransition.idl', 'WebAssembly/Global.idl', 'WebAssembly/Instance.idl', 'WebAssembly/Memory.idl', 'WebAssembly/Module.idl', 'WebAssembly/Table.idl', 'WebAssembly/WebAssembly.idl', 'WebAudio/AnalyserNode.idl', 'WebAudio/AudioBuffer.idl', 'WebAudio/AudioBufferSourceNode.idl', 'WebAudio/AudioContext.idl', 'WebAudio/AudioDestinationNode.idl', 'WebAudio/AudioListener.idl', 'WebAudio/AudioNode.idl', 'WebAudio/AudioParam.idl', 'WebAudio/AudioScheduledSourceNode.idl', 'WebAudio/BaseAudioContext.idl', 'WebAudio/BiquadFilterNode.idl', 'WebAudio/ChannelMergerNode.idl', 'WebAudio/ChannelSplitterNode.idl', 'WebAudio/ConstantSourceNode.idl', 'WebAudio/DelayNode.idl', 'WebAudio/DynamicsCompressorNode.idl', 'WebAudio/GainNode.idl', 'WebAudio/MediaElementAudioSourceNode.idl', 'WebAudio/OfflineAudioCompletionEvent.idl', 'WebAudio/OfflineAudioContext.idl', 'WebAudio/OscillatorNode.idl', 'WebAudio/PannerNode.idl', 'WebAudio/PeriodicWave.idl', 'WebAudio/ScriptProcessorNode.idl', 'WebAudio/StereoPannerNode.idl', 'WebGL/Extensions/ANGLEInstancedArrays.idl', 'WebGL/Extensions/EXTBlendMinMax.idl', 'WebGL/Extensions/EXTColorBufferFloat.idl', 'WebGL/Extensions/EXTRenderSnorm.idl', 'WebGL/Extensions/EXTTextureFilterAnisotropic.idl', 'WebGL/Extensions/EXTTextureNorm16.idl', 'WebGL/Extensions/OESElementIndexUint.idl', 'WebGL/Extensions/OESStandardDerivatives.idl', 'WebGL/Extensions/OESVertexArrayObject.idl', 'WebGL/Extensions/WebGLCompressedTextureS3tc.idl', 'WebGL/Extensions/WebGLCompressedTextureS3tcSrgb.idl', 'WebGL/Extensions/WebGLDebugRendererInfo.idl', 'WebGL/Extensions/WebGLDrawBuffers.idl', 'WebGL/Extensions/WebGLVertexArrayObjectOES.idl', 'WebGL/Types.idl', 'WebGL/WebGL2RenderingContext.idl', 'WebGL/WebGL2RenderingContextBase.idl', 'WebGL/WebGL2RenderingContextOverloads.idl', 'WebGL/WebGLActiveInfo.idl', 'WebGL/WebGLBuffer.idl', 'WebGL/WebGLContextEvent.idl', 'WebGL/WebGLFramebuffer.idl', 'WebGL/WebGLObject.idl', 'WebGL/WebGLProgram.idl', 'WebGL/WebGLQuery.idl', 'WebGL/WebGLRenderbuffer.idl', 'WebGL/WebGLRenderingContext.idl', 'WebGL/WebGLRenderingContextBase.idl', 'WebGL/WebGLRenderingContextOverloads.idl', 'WebGL/WebGLSampler.idl', 'WebGL/WebGLShader.idl', 'WebGL/WebGLShaderPrecisionFormat.idl', 'WebGL/WebGLSync.idl', 'WebGL/WebGLTexture.idl', 'WebGL/WebGLTransformFeedback.idl', 'WebGL/WebGLUniformLocation.idl', 'WebGL/WebGLVertexArrayObject.idl', 'WebIDL/Buffers.idl', 'WebIDL/DOMException.idl', 'WebIDL/Function.idl', 'WebIDL/QuotaExceededError.idl', 'WebLocks/Lock.idl', 'WebLocks/LockManager.idl', 'WebLocks/NavigatorLocks.idl', 'WebSockets/WebSocket.idl', 'WebVTT/VTTCue.idl', 'WebVTT/VTTRegion.idl', 'WebXR/XRLayer.idl', 'WebXR/XRRenderState.idl', 'WebXR/XRSession.idl', 'WebXR/XRSessionEvent.idl', 'WebXR/XRSystem.idl', 'WebXR/XRWebGLLayer.idl', 'XHR/FormData.idl', 'XHR/ProgressEvent.idl', 'XHR/XMLHttpRequest.idl', 'XHR/XMLHttpRequestEventTarget.idl', 'XHR/XMLHttpRequestUpload.idl', 'XPath/XPathEvaluator.idl', 'XPath/XPathExpression.idl', 'XPath/XPathNSResolver.idl', 'XPath/XPathResult.idl'] + [ + srcs = ['ARIA/ARIAMixin.idl', 'Animations/Animatable.idl', 'Animations/Animation.idl', 'Animations/AnimationEffect.idl', 'Animations/AnimationPlaybackEvent.idl', 'Animations/AnimationTimeline.idl', 'Animations/DocumentTimeline.idl', 'Animations/KeyframeEffect.idl', 'Animations/ScrollTimeline.idl', 'CSS/AnimationEvent.idl', 'CSS/CSS.idl', 'CSS/CSSAnimation.idl', 'CSS/CSSConditionRule.idl', 'CSS/CSSContainerRule.idl', 'CSS/CSSCounterStyleRule.idl', 'CSS/CSSFontFaceDescriptors.idl', 'CSS/CSSFontFaceRule.idl', 'CSS/CSSFontFeatureValuesMap.idl', 'CSS/CSSFontFeatureValuesRule.idl', 'CSS/CSSFunctionDeclarations.idl', 'CSS/CSSFunctionDescriptors.idl', 'CSS/CSSFunctionRule.idl', 'CSS/CSSGroupingRule.idl', 'CSS/CSSImageValue.idl', 'CSS/CSSImportRule.idl', 'CSS/CSSKeyframeRule.idl', 'CSS/CSSKeyframesRule.idl', 'CSS/CSSKeywordValue.idl', 'CSS/CSSLayerBlockRule.idl', 'CSS/CSSLayerStatementRule.idl', 'CSS/CSSMarginRule.idl', 'CSS/CSSMathClamp.idl', 'CSS/CSSMathInvert.idl', 'CSS/CSSMathMax.idl', 'CSS/CSSMathMin.idl', 'CSS/CSSMathNegate.idl', 'CSS/CSSMathProduct.idl', 'CSS/CSSMathSum.idl', 'CSS/CSSMathValue.idl', 'CSS/CSSMatrixComponent.idl', 'CSS/CSSMediaRule.idl', 'CSS/CSSNamespaceRule.idl', 'CSS/CSSNestedDeclarations.idl', 'CSS/CSSNumericArray.idl', 'CSS/CSSNumericValue.idl', 'CSS/CSSPageDescriptors.idl', 'CSS/CSSPageRule.idl', 'CSS/CSSPerspective.idl', 'CSS/CSSPropertyRule.idl', 'CSS/CSSRotate.idl', 'CSS/CSSRule.idl', 'CSS/CSSRuleList.idl', 'CSS/CSSScale.idl', 'CSS/CSSScopeRule.idl', 'CSS/CSSSkew.idl', 'CSS/CSSSkewX.idl', 'CSS/CSSSkewY.idl', 'CSS/CSSStyleDeclaration.idl', 'CSS/CSSStyleProperties.idl', 'CSS/CSSStyleRule.idl', 'CSS/CSSStyleSheet.idl', 'CSS/CSSStyleValue.idl', 'CSS/CSSSupportsRule.idl', 'CSS/CSSTransformComponent.idl', 'CSS/CSSTransformValue.idl', 'CSS/CSSTransition.idl', 'CSS/CSSTranslate.idl', 'CSS/CSSUnitValue.idl', 'CSS/CSSUnparsedValue.idl', 'CSS/CSSVariableReferenceValue.idl', 'CSS/ElementCSSInlineStyle.idl', 'CSS/FontFace.idl', 'CSS/FontFaceSet.idl', 'CSS/FontFaceSetLoadEvent.idl', 'CSS/LinkStyle.idl', 'CSS/MediaList.idl', 'CSS/MediaQueryList.idl', 'CSS/MediaQueryListEvent.idl', 'CSS/Screen.idl', 'CSS/ScreenOrientation.idl', 'CSS/StylePropertyMap.idl', 'CSS/StylePropertyMapReadOnly.idl', 'CSS/StyleSheet.idl', 'CSS/StyleSheetList.idl', 'CSS/TransitionEvent.idl', 'CSS/VisualViewport.idl', 'Clipboard/Clipboard.idl', 'Clipboard/ClipboardEvent.idl', 'Clipboard/ClipboardItem.idl', 'Compression/CompressionStream.idl', 'Compression/DecompressionStream.idl', 'ContentSecurityPolicy/SecurityPolicyViolationEvent.idl', 'CookieStore/CookieChangeEvent.idl', 'CookieStore/CookieStore.idl', 'CredentialManagement/Credential.idl', 'CredentialManagement/CredentialsContainer.idl', 'CredentialManagement/FederatedCredential.idl', 'CredentialManagement/PasswordCredential.idl', 'Crypto/Crypto.idl', 'Crypto/CryptoKey.idl', 'Crypto/SubtleCrypto.idl', 'DOM/AbortController.idl', 'DOM/AbortSignal.idl', 'DOM/AbstractRange.idl', 'DOM/Attr.idl', 'DOM/CDATASection.idl', 'DOM/CaretPosition.idl', 'DOM/CharacterData.idl', 'DOM/ChildNode.idl', 'DOM/Comment.idl', 'DOM/CustomEvent.idl', 'DOM/DOMImplementation.idl', 'DOM/DOMTokenList.idl', 'DOM/Document.idl', 'DOM/DocumentFragment.idl', 'DOM/DocumentOrShadowRoot.idl', 'DOM/DocumentType.idl', 'DOM/Element.idl', 'DOM/Event.idl', 'DOM/EventHandler.idl', 'DOM/EventListener.idl', 'DOM/EventTarget.idl', 'DOM/HTMLCollection.idl', 'DOM/MutationObserver.idl', 'DOM/MutationRecord.idl', 'DOM/NamedNodeMap.idl', 'DOM/Node.idl', 'DOM/NodeFilter.idl', 'DOM/NodeIterator.idl', 'DOM/NodeList.idl', 'DOM/ParentNode.idl', 'DOM/ProcessingInstruction.idl', 'DOM/Range.idl', 'DOM/ShadowRoot.idl', 'DOM/Slottable.idl', 'DOM/StaticRange.idl', 'DOM/Text.idl', 'DOM/TreeWalker.idl', 'DOM/XMLDocument.idl', 'DOMURL/DOMURL.idl', 'DOMURL/Origin.idl', 'DOMURL/URLSearchParams.idl', 'Encoding/TextDecoder.idl', 'Encoding/TextDecoderCommon.idl', 'Encoding/TextDecoderStream.idl', 'Encoding/TextEncoder.idl', 'Encoding/TextEncoderCommon.idl', 'Encoding/TextEncoderStream.idl', 'EncryptedMediaExtensions/MediaKeySystemAccess.idl', 'EntriesAPI/FileSystemEntry.idl', 'EventTiming/PerformanceEventTiming.idl', 'Fetch/Body.idl', 'Fetch/BodyInit.idl', 'Fetch/Headers.idl', 'Fetch/Request.idl', 'Fetch/Response.idl', 'FileAPI/Blob.idl', 'FileAPI/File.idl', 'FileAPI/FileList.idl', 'FileAPI/FileReader.idl', 'FileAPI/FileReaderSync.idl', 'Fullscreen/DocumentExtensions.idl', 'Fullscreen/DocumentOrShadowRootExtensions.idl', 'Fullscreen/ElementExtensions.idl', 'GPC/GlobalPrivacyControl.idl', 'Gamepad/Gamepad.idl', 'Gamepad/GamepadButton.idl', 'Gamepad/GamepadEvent.idl', 'Gamepad/GamepadHapticActuator.idl', 'Geolocation/Geolocation.idl', 'Geolocation/GeolocationCoordinates.idl', 'Geolocation/GeolocationPosition.idl', 'Geolocation/GeolocationPositionError.idl', 'Geometry/DOMMatrix.idl', 'Geometry/DOMMatrixReadOnly.idl', 'Geometry/DOMPoint.idl', 'Geometry/DOMPointReadOnly.idl', 'Geometry/DOMQuad.idl', 'Geometry/DOMRect.idl', 'Geometry/DOMRectList.idl', 'Geometry/DOMRectReadOnly.idl', 'HTML/AbstractWorker.idl', 'HTML/AnimationFrameProvider.idl', 'HTML/AudioTrack.idl', 'HTML/AudioTrackList.idl', 'HTML/BarProp.idl', 'HTML/BeforeUnloadEvent.idl', 'HTML/BroadcastChannel.idl', 'HTML/Canvas/CanvasCompositing.idl', 'HTML/Canvas/CanvasDrawImage.idl', 'HTML/Canvas/CanvasDrawPath.idl', 'HTML/Canvas/CanvasFillStrokeStyles.idl', 'HTML/Canvas/CanvasFilters.idl', 'HTML/Canvas/CanvasImageData.idl', 'HTML/Canvas/CanvasImageSmoothing.idl', 'HTML/Canvas/CanvasPath.idl', 'HTML/Canvas/CanvasPathDrawingStyles.idl', 'HTML/Canvas/CanvasRect.idl', 'HTML/Canvas/CanvasSettings.idl', 'HTML/Canvas/CanvasShadowStyles.idl', 'HTML/Canvas/CanvasState.idl', 'HTML/Canvas/CanvasText.idl', 'HTML/Canvas/CanvasTextDrawingStyles.idl', 'HTML/Canvas/CanvasTransform.idl', 'HTML/Canvas/CanvasUserInterface.idl', 'HTML/Canvas/OffscreenCanvasBase.idl', 'HTML/CanvasGradient.idl', 'HTML/CanvasPattern.idl', 'HTML/CanvasRenderingContext2D.idl', 'HTML/CanvasRenderingContext2DSettings.idl', 'HTML/CloseEvent.idl', 'HTML/CloseWatcher.idl', 'HTML/CommandEvent.idl', 'HTML/CustomElements/CustomElementRegistry.idl', 'HTML/CustomElements/CustomStateSet.idl', 'HTML/DOMParser.idl', 'HTML/DOMStringList.idl', 'HTML/DOMStringMap.idl', 'HTML/DataTransfer.idl', 'HTML/DataTransferItem.idl', 'HTML/DataTransferItemList.idl', 'HTML/DedicatedWorkerGlobalScope.idl', 'HTML/DragEvent.idl', 'HTML/ElementInternals.idl', 'HTML/ErrorEvent.idl', 'HTML/EventSource.idl', 'HTML/External.idl', 'HTML/FormDataEvent.idl', 'HTML/HTMLAllCollection.idl', 'HTML/HTMLAnchorElement.idl', 'HTML/HTMLAreaElement.idl', 'HTML/HTMLAudioElement.idl', 'HTML/HTMLBRElement.idl', 'HTML/HTMLBaseElement.idl', 'HTML/HTMLBodyElement.idl', 'HTML/HTMLButtonElement.idl', 'HTML/HTMLCanvasElement.idl', 'HTML/HTMLDListElement.idl', 'HTML/HTMLDataElement.idl', 'HTML/HTMLDataListElement.idl', 'HTML/HTMLDetailsElement.idl', 'HTML/HTMLDialogElement.idl', 'HTML/HTMLDirectoryElement.idl', 'HTML/HTMLDivElement.idl', 'HTML/HTMLDocument.idl', 'HTML/HTMLElement.idl', 'HTML/HTMLEmbedElement.idl', 'HTML/HTMLFieldSetElement.idl', 'HTML/HTMLFontElement.idl', 'HTML/HTMLFormControlsCollection.idl', 'HTML/HTMLFormElement.idl', 'HTML/HTMLFrameElement.idl', 'HTML/HTMLFrameSetElement.idl', 'HTML/HTMLHRElement.idl', 'HTML/HTMLHeadElement.idl', 'HTML/HTMLHeadingElement.idl', 'HTML/HTMLHtmlElement.idl', 'HTML/HTMLHyperlinkElementUtils.idl', 'HTML/HTMLIFrameElement.idl', 'HTML/HTMLImageElement.idl', 'HTML/HTMLInputElement.idl', 'HTML/HTMLLIElement.idl', 'HTML/HTMLLabelElement.idl', 'HTML/HTMLLegendElement.idl', 'HTML/HTMLLinkElement.idl', 'HTML/HTMLMapElement.idl', 'HTML/HTMLMarqueeElement.idl', 'HTML/HTMLMediaElement.idl', 'HTML/HTMLMenuElement.idl', 'HTML/HTMLMetaElement.idl', 'HTML/HTMLMeterElement.idl', 'HTML/HTMLModElement.idl', 'HTML/HTMLOListElement.idl', 'HTML/HTMLObjectElement.idl', 'HTML/HTMLOptGroupElement.idl', 'HTML/HTMLOptionElement.idl', 'HTML/HTMLOptionsCollection.idl', 'HTML/HTMLOrSVGOrMathMLElement.idl', 'HTML/HTMLOutputElement.idl', 'HTML/HTMLParagraphElement.idl', 'HTML/HTMLParamElement.idl', 'HTML/HTMLPictureElement.idl', 'HTML/HTMLPreElement.idl', 'HTML/HTMLProgressElement.idl', 'HTML/HTMLQuoteElement.idl', 'HTML/HTMLScriptElement.idl', 'HTML/HTMLSelectElement.idl', 'HTML/HTMLSelectedContentElement.idl', 'HTML/HTMLSlotElement.idl', 'HTML/HTMLSourceElement.idl', 'HTML/HTMLSpanElement.idl', 'HTML/HTMLStyleElement.idl', 'HTML/HTMLTableCaptionElement.idl', 'HTML/HTMLTableCellElement.idl', 'HTML/HTMLTableColElement.idl', 'HTML/HTMLTableElement.idl', 'HTML/HTMLTableRowElement.idl', 'HTML/HTMLTableSectionElement.idl', 'HTML/HTMLTemplateElement.idl', 'HTML/HTMLTextAreaElement.idl', 'HTML/HTMLTimeElement.idl', 'HTML/HTMLTitleElement.idl', 'HTML/HTMLTrackElement.idl', 'HTML/HTMLUListElement.idl', 'HTML/HTMLUnknownElement.idl', 'HTML/HTMLVideoElement.idl', 'HTML/HashChangeEvent.idl', 'HTML/History.idl', 'HTML/HyperlinkElementUtils.idl', 'HTML/ImageBitmap.idl', 'HTML/ImageData.idl', 'HTML/Location.idl', 'HTML/MediaError.idl', 'HTML/MessageChannel.idl', 'HTML/MessageEvent.idl', 'HTML/MessagePort.idl', 'HTML/MimeType.idl', 'HTML/MimeTypeArray.idl', 'HTML/NavigateEvent.idl', 'HTML/Navigation.idl', 'HTML/NavigationActivation.idl', 'HTML/NavigationCurrentEntryChangeEvent.idl', 'HTML/NavigationDestination.idl', 'HTML/NavigationHistoryEntry.idl', 'HTML/NavigationTransition.idl', 'HTML/NavigationType.idl', 'HTML/Navigator.idl', 'HTML/NavigatorBeacon.idl', 'HTML/NavigatorConcurrentHardware.idl', 'HTML/NavigatorDeviceMemory.idl', 'HTML/NavigatorID.idl', 'HTML/NavigatorLanguage.idl', 'HTML/NavigatorOnLine.idl', 'HTML/OffscreenCanvas.idl', 'HTML/OffscreenCanvasRenderingContext2D.idl', 'HTML/PageSwapEvent.idl', 'HTML/PageTransitionEvent.idl', 'HTML/Path2D.idl', 'HTML/Plugin.idl', 'HTML/PluginArray.idl', 'HTML/PopStateEvent.idl', 'HTML/PopoverTargetAttributes.idl', 'HTML/PredefinedColorSpace.idl', 'HTML/PromiseRejectionEvent.idl', 'HTML/RadioNodeList.idl', 'HTML/Scripting/Fetching.idl', 'HTML/SharedWorker.idl', 'HTML/SharedWorkerGlobalScope.idl', 'HTML/Storage.idl', 'HTML/StorageEvent.idl', 'HTML/SubmitEvent.idl', 'HTML/TextMetrics.idl', 'HTML/TextTrack.idl', 'HTML/TextTrackCue.idl', 'HTML/TextTrackCueList.idl', 'HTML/TextTrackList.idl', 'HTML/TimeRanges.idl', 'HTML/ToggleEvent.idl', 'HTML/TrackEvent.idl', 'HTML/UniversalGlobalScope.idl', 'HTML/UserActivation.idl', 'HTML/ValidityState.idl', 'HTML/VideoTrack.idl', 'HTML/VideoTrackList.idl', 'HTML/Window.idl', 'HTML/WindowDeprecated.idl', 'HTML/WindowLocalStorage.idl', 'HTML/WindowOrWorkerGlobalScope.idl', 'HTML/WindowSessionStorage.idl', 'HTML/Worker.idl', 'HTML/WorkerGlobalScope.idl', 'HTML/WorkerLocation.idl', 'HTML/WorkerNavigator.idl', 'HTML/WorkletGlobalScope.idl', 'HTML/XMLSerializer.idl', 'HighResolutionTime/DOMHighResTimeStamp.idl', 'HighResolutionTime/EpochTimeStamp.idl', 'HighResolutionTime/Performance.idl', 'IndexedDB/IDBCursor.idl', 'IndexedDB/IDBCursorWithValue.idl', 'IndexedDB/IDBDatabase.idl', 'IndexedDB/IDBFactory.idl', 'IndexedDB/IDBIndex.idl', 'IndexedDB/IDBKeyRange.idl', 'IndexedDB/IDBObjectStore.idl', 'IndexedDB/IDBOpenDBRequest.idl', 'IndexedDB/IDBRecord.idl', 'IndexedDB/IDBRequest.idl', 'IndexedDB/IDBTransaction.idl', 'IndexedDB/IDBVersionChangeEvent.idl', 'Internals/FakeXRDevice.idl', 'Internals/InternalAnimationTimeline.idl', 'Internals/InternalGamepad.idl', 'Internals/Internals.idl', 'Internals/WebUI.idl', 'Internals/XRTest.idl', 'IntersectionObserver/IntersectionObserver.idl', 'IntersectionObserver/IntersectionObserverEntry.idl', 'MathML/MathMLAnchorElement.idl', 'MathML/MathMLElement.idl', 'MediaCapabilitiesAPI/MediaCapabilities.idl', 'MediaCapture/MediaDeviceInfo.idl', 'MediaCapture/MediaDevices.idl', 'MediaCapture/MediaStream.idl', 'MediaCapture/MediaStreamConstraints.idl', 'MediaCapture/MediaStreamTrack.idl', 'MediaCapture/MediaStreamTrackEvent.idl', 'MediaSourceExtensions/BufferedChangeEvent.idl', 'MediaSourceExtensions/ManagedMediaSource.idl', 'MediaSourceExtensions/ManagedSourceBuffer.idl', 'MediaSourceExtensions/MediaSource.idl', 'MediaSourceExtensions/MediaSourceHandle.idl', 'MediaSourceExtensions/SourceBuffer.idl', 'MediaSourceExtensions/SourceBufferList.idl', 'NavigationTiming/PerformanceExtensions.idl', 'NavigationTiming/PerformanceNavigation.idl', 'NavigationTiming/PerformanceTiming.idl', 'NotificationsAPI/Notification.idl', 'PerformanceTimeline/PerformanceEntry.idl', 'PerformanceTimeline/PerformanceObserver.idl', 'PerformanceTimeline/PerformanceObserverEntryList.idl', 'PermissionsAPI/PermissionStatus.idl', 'PermissionsAPI/Permissions.idl', 'RequestIdleCallback/IdleDeadline.idl', 'RequestIdleCallback/IdleRequest.idl', 'ResizeObserver/ResizeObserver.idl', 'ResizeObserver/ResizeObserverEntry.idl', 'ResizeObserver/ResizeObserverSize.idl', 'ResourceTiming/PerformanceResourceTiming.idl', 'SVG/SVGAElement.idl', 'SVG/SVGAnimatedEnumeration.idl', 'SVG/SVGAnimatedInteger.idl', 'SVG/SVGAnimatedLength.idl', 'SVG/SVGAnimatedLengthList.idl', 'SVG/SVGAnimatedNumber.idl', 'SVG/SVGAnimatedNumberList.idl', 'SVG/SVGAnimatedRect.idl', 'SVG/SVGAnimatedString.idl', 'SVG/SVGAnimatedTransformList.idl', 'SVG/SVGAnimationElement.idl', 'SVG/SVGCircleElement.idl', 'SVG/SVGClipPathElement.idl', 'SVG/SVGComponentTransferFunctionElement.idl', 'SVG/SVGDefsElement.idl', 'SVG/SVGDescElement.idl', 'SVG/SVGElement.idl', 'SVG/SVGEllipseElement.idl', 'SVG/SVGFEBlendElement.idl', 'SVG/SVGFEColorMatrixElement.idl', 'SVG/SVGFEComponentTransferElement.idl', 'SVG/SVGFECompositeElement.idl', 'SVG/SVGFEDisplacementMapElement.idl', 'SVG/SVGFEDropShadowElement.idl', 'SVG/SVGFEFloodElement.idl', 'SVG/SVGFEFuncAElement.idl', 'SVG/SVGFEFuncBElement.idl', 'SVG/SVGFEFuncGElement.idl', 'SVG/SVGFEFuncRElement.idl', 'SVG/SVGFEGaussianBlurElement.idl', 'SVG/SVGFEImageElement.idl', 'SVG/SVGFEMergeElement.idl', 'SVG/SVGFEMergeNodeElement.idl', 'SVG/SVGFEMorphologyElement.idl', 'SVG/SVGFEOffsetElement.idl', 'SVG/SVGFETurbulenceElement.idl', 'SVG/SVGFilterElement.idl', 'SVG/SVGFilterPrimitiveStandardAttributes.idl', 'SVG/SVGFitToViewBox.idl', 'SVG/SVGForeignObjectElement.idl', 'SVG/SVGGElement.idl', 'SVG/SVGGeometryElement.idl', 'SVG/SVGGradientElement.idl', 'SVG/SVGGraphicsElement.idl', 'SVG/SVGImageElement.idl', 'SVG/SVGLength.idl', 'SVG/SVGLengthList.idl', 'SVG/SVGLineElement.idl', 'SVG/SVGLinearGradientElement.idl', 'SVG/SVGMaskElement.idl', 'SVG/SVGMetadataElement.idl', 'SVG/SVGNumber.idl', 'SVG/SVGNumberList.idl', 'SVG/SVGPathElement.idl', 'SVG/SVGPatternElement.idl', 'SVG/SVGPolygonElement.idl', 'SVG/SVGPolylineElement.idl', 'SVG/SVGRadialGradientElement.idl', 'SVG/SVGRectElement.idl', 'SVG/SVGSVGElement.idl', 'SVG/SVGScriptElement.idl', 'SVG/SVGStopElement.idl', 'SVG/SVGStyleElement.idl', 'SVG/SVGSwitchElement.idl', 'SVG/SVGSymbolElement.idl', 'SVG/SVGTSpanElement.idl', 'SVG/SVGTextContentElement.idl', 'SVG/SVGTextElement.idl', 'SVG/SVGTextPathElement.idl', 'SVG/SVGTextPositioningElement.idl', 'SVG/SVGTitleElement.idl', 'SVG/SVGTransform.idl', 'SVG/SVGTransformList.idl', 'SVG/SVGURIReference.idl', 'SVG/SVGUnitTypes.idl', 'SVG/SVGUseElement.idl', 'SVG/SVGViewElement.idl', 'Selection/Selection.idl', 'Serial/Serial.idl', 'Serial/SerialPort.idl', 'ServiceWorker/Cache.idl', 'ServiceWorker/CacheStorage.idl', 'ServiceWorker/ServiceWorker.idl', 'ServiceWorker/ServiceWorkerContainer.idl', 'ServiceWorker/ServiceWorkerGlobalScope.idl', 'ServiceWorker/ServiceWorkerRegistration.idl', 'Speech/SpeechGrammar.idl', 'Speech/SpeechGrammarList.idl', 'Speech/SpeechRecognition.idl', 'Speech/SpeechRecognitionAlternative.idl', 'Speech/SpeechRecognitionEvent.idl', 'Speech/SpeechRecognitionPhrase.idl', 'Speech/SpeechRecognitionResult.idl', 'Speech/SpeechRecognitionResultList.idl', 'Speech/SpeechSynthesis.idl', 'Speech/SpeechSynthesisUtterance.idl', 'Speech/SpeechSynthesisVoice.idl', 'StorageAPI/NavigatorStorage.idl', 'StorageAPI/StorageManager.idl', 'Streams/ByteLengthQueuingStrategy.idl', 'Streams/CountQueuingStrategy.idl', 'Streams/GenericTransformStream.idl', 'Streams/QueuingStrategy.idl', 'Streams/QueuingStrategyInit.idl', 'Streams/ReadableByteStreamController.idl', 'Streams/ReadableStream.idl', 'Streams/ReadableStreamBYOBReader.idl', 'Streams/ReadableStreamBYOBRequest.idl', 'Streams/ReadableStreamDefaultController.idl', 'Streams/ReadableStreamDefaultReader.idl', 'Streams/ReadableStreamGenericReader.idl', 'Streams/TransformStream.idl', 'Streams/TransformStreamDefaultController.idl', 'Streams/Transformer.idl', 'Streams/UnderlyingSink.idl', 'Streams/UnderlyingSource.idl', 'Streams/WritableStream.idl', 'Streams/WritableStreamDefaultController.idl', 'Streams/WritableStreamDefaultWriter.idl', 'TrustedTypes/TrustedHTML.idl', 'TrustedTypes/TrustedScript.idl', 'TrustedTypes/TrustedScriptURL.idl', 'TrustedTypes/TrustedTypePolicy.idl', 'TrustedTypes/TrustedTypePolicyFactory.idl', 'UIEvents/CompositionEvent.idl', 'UIEvents/EventModifier.idl', 'UIEvents/FocusEvent.idl', 'UIEvents/InputEvent.idl', 'UIEvents/KeyboardEvent.idl', 'UIEvents/MouseEvent.idl', 'UIEvents/PointerEvent.idl', 'UIEvents/PointerEventHandlers.idl', 'UIEvents/TextEvent.idl', 'UIEvents/UIEvent.idl', 'UIEvents/WheelEvent.idl', 'URLPattern/URLPattern.idl', 'UserTiming/PerformanceMark.idl', 'UserTiming/PerformanceMeasure.idl', 'ViewTransition/ViewTransition.idl', 'WebAssembly/Global.idl', 'WebAssembly/Instance.idl', 'WebAssembly/Memory.idl', 'WebAssembly/Module.idl', 'WebAssembly/Table.idl', 'WebAssembly/WebAssembly.idl', 'WebAudio/AnalyserNode.idl', 'WebAudio/AudioBuffer.idl', 'WebAudio/AudioBufferSourceNode.idl', 'WebAudio/AudioContext.idl', 'WebAudio/AudioDestinationNode.idl', 'WebAudio/AudioListener.idl', 'WebAudio/AudioNode.idl', 'WebAudio/AudioParam.idl', 'WebAudio/AudioScheduledSourceNode.idl', 'WebAudio/BaseAudioContext.idl', 'WebAudio/BiquadFilterNode.idl', 'WebAudio/ChannelMergerNode.idl', 'WebAudio/ChannelSplitterNode.idl', 'WebAudio/ConstantSourceNode.idl', 'WebAudio/DelayNode.idl', 'WebAudio/DynamicsCompressorNode.idl', 'WebAudio/GainNode.idl', 'WebAudio/MediaElementAudioSourceNode.idl', 'WebAudio/OfflineAudioCompletionEvent.idl', 'WebAudio/OfflineAudioContext.idl', 'WebAudio/OscillatorNode.idl', 'WebAudio/PannerNode.idl', 'WebAudio/PeriodicWave.idl', 'WebAudio/ScriptProcessorNode.idl', 'WebAudio/StereoPannerNode.idl', 'WebGL/Extensions/ANGLEInstancedArrays.idl', 'WebGL/Extensions/EXTBlendMinMax.idl', 'WebGL/Extensions/EXTColorBufferFloat.idl', 'WebGL/Extensions/EXTRenderSnorm.idl', 'WebGL/Extensions/EXTTextureFilterAnisotropic.idl', 'WebGL/Extensions/EXTTextureNorm16.idl', 'WebGL/Extensions/OESElementIndexUint.idl', 'WebGL/Extensions/OESStandardDerivatives.idl', 'WebGL/Extensions/OESVertexArrayObject.idl', 'WebGL/Extensions/WebGLCompressedTextureS3tc.idl', 'WebGL/Extensions/WebGLCompressedTextureS3tcSrgb.idl', 'WebGL/Extensions/WebGLDebugRendererInfo.idl', 'WebGL/Extensions/WebGLDrawBuffers.idl', 'WebGL/Extensions/WebGLVertexArrayObjectOES.idl', 'WebGL/Types.idl', 'WebGL/WebGL2RenderingContext.idl', 'WebGL/WebGL2RenderingContextBase.idl', 'WebGL/WebGL2RenderingContextOverloads.idl', 'WebGL/WebGLActiveInfo.idl', 'WebGL/WebGLBuffer.idl', 'WebGL/WebGLContextEvent.idl', 'WebGL/WebGLFramebuffer.idl', 'WebGL/WebGLObject.idl', 'WebGL/WebGLProgram.idl', 'WebGL/WebGLQuery.idl', 'WebGL/WebGLRenderbuffer.idl', 'WebGL/WebGLRenderingContext.idl', 'WebGL/WebGLRenderingContextBase.idl', 'WebGL/WebGLRenderingContextOverloads.idl', 'WebGL/WebGLSampler.idl', 'WebGL/WebGLShader.idl', 'WebGL/WebGLShaderPrecisionFormat.idl', 'WebGL/WebGLSync.idl', 'WebGL/WebGLTexture.idl', 'WebGL/WebGLTransformFeedback.idl', 'WebGL/WebGLUniformLocation.idl', 'WebGL/WebGLVertexArrayObject.idl', 'WebIDL/Buffers.idl', 'WebIDL/DOMException.idl', 'WebIDL/Function.idl', 'WebIDL/QuotaExceededError.idl', 'WebLocks/Lock.idl', 'WebLocks/LockManager.idl', 'WebLocks/NavigatorLocks.idl', 'WebSockets/WebSocket.idl', 'WebVTT/VTTCue.idl', 'WebVTT/VTTRegion.idl', 'WebXR/XRLayer.idl', 'WebXR/XRRenderState.idl', 'WebXR/XRSession.idl', 'WebXR/XRSessionEvent.idl', 'WebXR/XRSystem.idl', 'WebXR/XRWebGLLayer.idl', 'XHR/FormData.idl', 'XHR/ProgressEvent.idl', 'XHR/XMLHttpRequest.idl', 'XHR/XMLHttpRequestEventTarget.idl', 'XHR/XMLHttpRequestUpload.idl', 'XPath/XPathEvaluator.idl', 'XPath/XPathExpression.idl', 'XPath/XPathNSResolver.idl', 'XPath/XPathResult.idl'] + [ 'CSS/GeneratedCSSNumericFactoryMethods.idl', # generated in this package 'CSS/GeneratedCSSStyleProperties.idl', # generated in this package ] + ["//Meta:generators"], @@ -505,6 +505,7 @@ def libweb_bindings_codegen(): 'Bindings/DedicatedWorkerExposedInterfaces.h', 'Bindings/DedicatedWorkerGlobalScope.cpp', 'Bindings/DedicatedWorkerGlobalScope.h', + 'Bindings/DedicatedWorkerGlobalScopeGlobalMixin.h', 'Bindings/DelayNode.cpp', 'Bindings/DelayNode.h', 'Bindings/Document.cpp', @@ -910,6 +911,8 @@ def libweb_bindings_codegen(): 'Bindings/NavigateEvent.h', 'Bindings/Navigation.cpp', 'Bindings/Navigation.h', + 'Bindings/NavigationActivation.cpp', + 'Bindings/NavigationActivation.h', 'Bindings/NavigationCurrentEntryChangeEvent.cpp', 'Bindings/NavigationCurrentEntryChangeEvent.h', 'Bindings/NavigationDestination.cpp', @@ -968,6 +971,8 @@ def libweb_bindings_codegen(): 'Bindings/Origin.h', 'Bindings/OscillatorNode.cpp', 'Bindings/OscillatorNode.h', + 'Bindings/PageSwapEvent.cpp', + 'Bindings/PageSwapEvent.h', 'Bindings/PageTransitionEvent.cpp', 'Bindings/PageTransitionEvent.h', 'Bindings/PannerNode.cpp', @@ -1246,6 +1251,7 @@ def libweb_bindings_codegen(): 'Bindings/SharedWorkerExposedInterfaces.h', 'Bindings/SharedWorkerGlobalScope.cpp', 'Bindings/SharedWorkerGlobalScope.h', + 'Bindings/SharedWorkerGlobalScopeGlobalMixin.h', 'Bindings/Slottable.cpp', 'Bindings/Slottable.h', 'Bindings/SourceBuffer.cpp', @@ -1284,6 +1290,7 @@ def libweb_bindings_codegen(): 'Bindings/StorageEvent.h', 'Bindings/StorageManager.cpp', 'Bindings/StorageManager.h', + 'Bindings/StructuredSerializeBindings.cpp', 'Bindings/StylePropertyMap.cpp', 'Bindings/StylePropertyMap.h', 'Bindings/StylePropertyMapReadOnly.cpp', @@ -1448,6 +1455,7 @@ def libweb_bindings_codegen(): 'Bindings/WindowDeprecated.h', 'Bindings/WindowExposedInterfaces.cpp', 'Bindings/WindowExposedInterfaces.h', + 'Bindings/WindowGlobalMixin.h', 'Bindings/WindowLocalStorage.cpp', 'Bindings/WindowLocalStorage.h', 'Bindings/WindowOrWorkerGlobalScope.cpp', @@ -1464,6 +1472,7 @@ def libweb_bindings_codegen(): 'Bindings/WorkerNavigator.h', 'Bindings/WorkletGlobalScope.cpp', 'Bindings/WorkletGlobalScope.h', + 'Bindings/WrapperFactory.cpp', 'Bindings/WritableStream.cpp', 'Bindings/WritableStream.h', 'Bindings/WritableStreamDefaultController.cpp', @@ -1503,5 +1512,5 @@ def libweb_bindings_codegen(): 'Bindings/XRWebGLLayer.cpp', 'Bindings/XRWebGLLayer.h', ], - cmd = 'PYTHONHASHSEED=0 python3 Meta/Generators/generate_libweb_bindings.py -o $(RULEDIR)/Bindings $(location Animations/Animatable.idl) $(location Animations/Animation.idl) $(location Animations/AnimationEffect.idl) $(location Animations/AnimationPlaybackEvent.idl) $(location Animations/AnimationTimeline.idl) $(location Animations/DocumentTimeline.idl) $(location Animations/KeyframeEffect.idl) $(location Animations/ScrollTimeline.idl) $(location ARIA/ARIAMixin.idl) $(location Clipboard/Clipboard.idl) $(location Clipboard/ClipboardEvent.idl) $(location Clipboard/ClipboardItem.idl) $(location Compression/CompressionStream.idl) $(location Compression/DecompressionStream.idl) $(location ContentSecurityPolicy/SecurityPolicyViolationEvent.idl) $(location CookieStore/CookieChangeEvent.idl) $(location CookieStore/CookieStore.idl) $(location CredentialManagement/Credential.idl) $(location CredentialManagement/CredentialsContainer.idl) $(location CredentialManagement/FederatedCredential.idl) $(location CredentialManagement/PasswordCredential.idl) $(location Crypto/Crypto.idl) $(location Crypto/CryptoKey.idl) $(location Crypto/SubtleCrypto.idl) $(location CSS/AnimationEvent.idl) $(location CSS/CSS.idl) $(location CSS/CSSAnimation.idl) $(location CSS/CSSConditionRule.idl) $(location CSS/CSSContainerRule.idl) $(location CSS/CSSCounterStyleRule.idl) $(location CSS/CSSFontFaceDescriptors.idl) $(location CSS/CSSFontFaceRule.idl) $(location CSS/CSSFontFeatureValuesMap.idl) $(location CSS/CSSFontFeatureValuesRule.idl) $(location CSS/CSSFunctionDeclarations.idl) $(location CSS/CSSFunctionDescriptors.idl) $(location CSS/CSSFunctionRule.idl) $(location CSS/CSSGroupingRule.idl) $(location CSS/CSSImageValue.idl) $(location CSS/CSSImportRule.idl) $(location CSS/CSSKeyframeRule.idl) $(location CSS/CSSKeyframesRule.idl) $(location CSS/CSSKeywordValue.idl) $(location CSS/CSSLayerBlockRule.idl) $(location CSS/CSSLayerStatementRule.idl) $(location CSS/CSSMarginRule.idl) $(location CSS/CSSMathClamp.idl) $(location CSS/CSSMathInvert.idl) $(location CSS/CSSMathMax.idl) $(location CSS/CSSMathMin.idl) $(location CSS/CSSMathNegate.idl) $(location CSS/CSSMathProduct.idl) $(location CSS/CSSMathSum.idl) $(location CSS/CSSMathValue.idl) $(location CSS/CSSMatrixComponent.idl) $(location CSS/CSSMediaRule.idl) $(location CSS/CSSNamespaceRule.idl) $(location CSS/CSSNestedDeclarations.idl) $(location CSS/CSSNumericArray.idl) $(location CSS/CSSNumericValue.idl) $(location CSS/CSSPageDescriptors.idl) $(location CSS/CSSPageRule.idl) $(location CSS/CSSPerspective.idl) $(location CSS/CSSPropertyRule.idl) $(location CSS/CSSRotate.idl) $(location CSS/CSSRule.idl) $(location CSS/CSSRuleList.idl) $(location CSS/CSSScale.idl) $(location CSS/CSSScopeRule.idl) $(location CSS/CSSSkew.idl) $(location CSS/CSSSkewX.idl) $(location CSS/CSSSkewY.idl) $(location CSS/CSSStyleDeclaration.idl) $(location CSS/CSSStyleProperties.idl) $(location CSS/CSSStyleRule.idl) $(location CSS/CSSStyleSheet.idl) $(location CSS/CSSStyleValue.idl) $(location CSS/CSSSupportsRule.idl) $(location CSS/CSSTransformComponent.idl) $(location CSS/CSSTransformValue.idl) $(location CSS/CSSTransition.idl) $(location CSS/CSSTranslate.idl) $(location CSS/CSSUnitValue.idl) $(location CSS/CSSUnparsedValue.idl) $(location CSS/CSSVariableReferenceValue.idl) $(location CSS/ElementCSSInlineStyle.idl) $(location CSS/FontFace.idl) $(location CSS/FontFaceSet.idl) $(location CSS/FontFaceSetLoadEvent.idl) $(location CSS/GeneratedCSSNumericFactoryMethods.idl) $(location CSS/GeneratedCSSStyleProperties.idl) $(location CSS/LinkStyle.idl) $(location CSS/MediaList.idl) $(location CSS/MediaQueryList.idl) $(location CSS/MediaQueryListEvent.idl) $(location CSS/Screen.idl) $(location CSS/ScreenOrientation.idl) $(location CSS/StylePropertyMap.idl) $(location CSS/StylePropertyMapReadOnly.idl) $(location CSS/StyleSheet.idl) $(location CSS/StyleSheetList.idl) $(location CSS/TransitionEvent.idl) $(location CSS/VisualViewport.idl) $(location DOM/AbortController.idl) $(location DOM/AbortSignal.idl) $(location DOM/AbstractRange.idl) $(location DOM/Attr.idl) $(location DOM/CDATASection.idl) $(location DOM/CaretPosition.idl) $(location DOM/CharacterData.idl) $(location DOM/ChildNode.idl) $(location DOM/Comment.idl) $(location DOM/CustomEvent.idl) $(location DOM/Document.idl) $(location DOM/DocumentFragment.idl) $(location DOM/DocumentOrShadowRoot.idl) $(location DOM/DocumentType.idl) $(location DOM/DOMImplementation.idl) $(location DOM/DOMTokenList.idl) $(location DOM/Element.idl) $(location DOM/Event.idl) $(location DOM/EventHandler.idl) $(location DOM/EventListener.idl) $(location DOM/EventTarget.idl) $(location DOM/HTMLCollection.idl) $(location DOM/MutationObserver.idl) $(location DOM/MutationRecord.idl) $(location DOM/NamedNodeMap.idl) $(location DOM/Node.idl) $(location DOM/NodeFilter.idl) $(location DOM/NodeIterator.idl) $(location DOM/NodeList.idl) $(location DOM/ParentNode.idl) $(location DOM/ProcessingInstruction.idl) $(location DOM/Range.idl) $(location DOM/ShadowRoot.idl) $(location DOM/Slottable.idl) $(location DOM/StaticRange.idl) $(location DOM/Text.idl) $(location DOM/TreeWalker.idl) $(location DOM/XMLDocument.idl) $(location DOMURL/DOMURL.idl) $(location DOMURL/Origin.idl) $(location DOMURL/URLSearchParams.idl) $(location Encoding/TextDecoder.idl) $(location Encoding/TextDecoderCommon.idl) $(location Encoding/TextDecoderStream.idl) $(location Encoding/TextEncoder.idl) $(location Encoding/TextEncoderCommon.idl) $(location Encoding/TextEncoderStream.idl) $(location EncryptedMediaExtensions/MediaKeySystemAccess.idl) $(location EntriesAPI/FileSystemEntry.idl) $(location EventTiming/PerformanceEventTiming.idl) $(location Fetch/Body.idl) $(location Fetch/BodyInit.idl) $(location Fetch/Headers.idl) $(location Fetch/Request.idl) $(location Fetch/Response.idl) $(location FileAPI/Blob.idl) $(location FileAPI/File.idl) $(location FileAPI/FileList.idl) $(location FileAPI/FileReader.idl) $(location FileAPI/FileReaderSync.idl) $(location Fullscreen/DocumentExtensions.idl) $(location Fullscreen/DocumentOrShadowRootExtensions.idl) $(location Fullscreen/ElementExtensions.idl) $(location Gamepad/Gamepad.idl) $(location Gamepad/GamepadButton.idl) $(location Gamepad/GamepadEvent.idl) $(location Gamepad/GamepadHapticActuator.idl) $(location Geolocation/Geolocation.idl) $(location Geolocation/GeolocationCoordinates.idl) $(location Geolocation/GeolocationPosition.idl) $(location Geolocation/GeolocationPositionError.idl) $(location Geometry/DOMMatrix.idl) $(location Geometry/DOMMatrixReadOnly.idl) $(location Geometry/DOMPoint.idl) $(location Geometry/DOMPointReadOnly.idl) $(location Geometry/DOMQuad.idl) $(location Geometry/DOMRect.idl) $(location Geometry/DOMRectList.idl) $(location Geometry/DOMRectReadOnly.idl) $(location GPC/GlobalPrivacyControl.idl) $(location HighResolutionTime/DOMHighResTimeStamp.idl) $(location HighResolutionTime/EpochTimeStamp.idl) $(location HighResolutionTime/Performance.idl) $(location HTML/AbstractWorker.idl) $(location HTML/AnimationFrameProvider.idl) $(location HTML/AudioTrack.idl) $(location HTML/AudioTrackList.idl) $(location HTML/BarProp.idl) $(location HTML/BeforeUnloadEvent.idl) $(location HTML/BroadcastChannel.idl) $(location HTML/Canvas/CanvasCompositing.idl) $(location HTML/Canvas/CanvasDrawImage.idl) $(location HTML/Canvas/CanvasDrawPath.idl) $(location HTML/Canvas/CanvasFillStrokeStyles.idl) $(location HTML/Canvas/CanvasFilters.idl) $(location HTML/Canvas/CanvasImageData.idl) $(location HTML/Canvas/CanvasImageSmoothing.idl) $(location HTML/Canvas/CanvasPath.idl) $(location HTML/Canvas/CanvasPathDrawingStyles.idl) $(location HTML/Canvas/CanvasRect.idl) $(location HTML/Canvas/CanvasSettings.idl) $(location HTML/Canvas/CanvasShadowStyles.idl) $(location HTML/Canvas/CanvasState.idl) $(location HTML/Canvas/CanvasText.idl) $(location HTML/Canvas/CanvasTextDrawingStyles.idl) $(location HTML/Canvas/CanvasTransform.idl) $(location HTML/Canvas/CanvasUserInterface.idl) $(location HTML/Canvas/OffscreenCanvasBase.idl) $(location HTML/CanvasGradient.idl) $(location HTML/CanvasPattern.idl) $(location HTML/CanvasRenderingContext2D.idl) $(location HTML/CanvasRenderingContext2DSettings.idl) $(location HTML/CloseEvent.idl) $(location HTML/CloseWatcher.idl) $(location HTML/CommandEvent.idl) $(location HTML/CustomElements/CustomElementRegistry.idl) $(location HTML/CustomElements/CustomStateSet.idl) $(location HTML/DataTransfer.idl) $(location HTML/DataTransferItem.idl) $(location HTML/DataTransferItemList.idl) $(location HTML/DedicatedWorkerGlobalScope.idl) $(location HTML/DOMParser.idl) $(location HTML/DOMStringList.idl) $(location HTML/DOMStringMap.idl) $(location HTML/DragEvent.idl) $(location HTML/ElementInternals.idl) $(location HTML/ErrorEvent.idl) $(location HTML/EventSource.idl) $(location HTML/External.idl) $(location HTML/FormDataEvent.idl) $(location HTML/HashChangeEvent.idl) $(location HTML/History.idl) $(location HTML/HTMLAllCollection.idl) $(location HTML/HTMLAnchorElement.idl) $(location HTML/HTMLAreaElement.idl) $(location HTML/HTMLAudioElement.idl) $(location HTML/HTMLBaseElement.idl) $(location HTML/HTMLBodyElement.idl) $(location HTML/HTMLBRElement.idl) $(location HTML/HTMLButtonElement.idl) $(location HTML/HTMLCanvasElement.idl) $(location HTML/HTMLDataElement.idl) $(location HTML/HTMLDataListElement.idl) $(location HTML/HTMLDetailsElement.idl) $(location HTML/HTMLDialogElement.idl) $(location HTML/HTMLDirectoryElement.idl) $(location HTML/HTMLDivElement.idl) $(location HTML/HTMLDListElement.idl) $(location HTML/HTMLDocument.idl) $(location HTML/HTMLElement.idl) $(location HTML/HTMLEmbedElement.idl) $(location HTML/HTMLFieldSetElement.idl) $(location HTML/HTMLFontElement.idl) $(location HTML/HTMLFormControlsCollection.idl) $(location HTML/HTMLFormElement.idl) $(location HTML/HTMLFrameElement.idl) $(location HTML/HTMLFrameSetElement.idl) $(location HTML/HTMLHeadElement.idl) $(location HTML/HTMLHeadingElement.idl) $(location HTML/HTMLHRElement.idl) $(location HTML/HTMLHtmlElement.idl) $(location HTML/HTMLHyperlinkElementUtils.idl) $(location HTML/HTMLIFrameElement.idl) $(location HTML/HTMLImageElement.idl) $(location HTML/HTMLInputElement.idl) $(location HTML/HTMLLabelElement.idl) $(location HTML/HTMLLegendElement.idl) $(location HTML/HTMLLIElement.idl) $(location HTML/HTMLLinkElement.idl) $(location HTML/HTMLMapElement.idl) $(location HTML/HTMLMarqueeElement.idl) $(location HTML/HTMLMediaElement.idl) $(location HTML/HTMLMenuElement.idl) $(location HTML/HTMLMetaElement.idl) $(location HTML/HTMLMeterElement.idl) $(location HTML/HTMLModElement.idl) $(location HTML/HTMLObjectElement.idl) $(location HTML/HTMLOListElement.idl) $(location HTML/HTMLOptGroupElement.idl) $(location HTML/HTMLOptionElement.idl) $(location HTML/HTMLOptionsCollection.idl) $(location HTML/HTMLOrSVGOrMathMLElement.idl) $(location HTML/HTMLOutputElement.idl) $(location HTML/HTMLParagraphElement.idl) $(location HTML/HTMLParamElement.idl) $(location HTML/HTMLPictureElement.idl) $(location HTML/HTMLPreElement.idl) $(location HTML/HTMLProgressElement.idl) $(location HTML/HTMLQuoteElement.idl) $(location HTML/HTMLScriptElement.idl) $(location HTML/HTMLSelectedContentElement.idl) $(location HTML/HTMLSelectElement.idl) $(location HTML/HTMLSlotElement.idl) $(location HTML/HTMLSourceElement.idl) $(location HTML/HTMLSpanElement.idl) $(location HTML/HTMLStyleElement.idl) $(location HTML/HTMLTableCaptionElement.idl) $(location HTML/HTMLTableCellElement.idl) $(location HTML/HTMLTableColElement.idl) $(location HTML/HTMLTableElement.idl) $(location HTML/HTMLTableRowElement.idl) $(location HTML/HTMLTableSectionElement.idl) $(location HTML/HTMLTemplateElement.idl) $(location HTML/HTMLTextAreaElement.idl) $(location HTML/HTMLTimeElement.idl) $(location HTML/HTMLTitleElement.idl) $(location HTML/HTMLTrackElement.idl) $(location HTML/HTMLUListElement.idl) $(location HTML/HTMLUnknownElement.idl) $(location HTML/HTMLVideoElement.idl) $(location HTML/HyperlinkElementUtils.idl) $(location HTML/ImageBitmap.idl) $(location HTML/ImageData.idl) $(location HTML/Location.idl) $(location HTML/MediaError.idl) $(location HTML/MessageChannel.idl) $(location HTML/MessageEvent.idl) $(location HTML/MessagePort.idl) $(location HTML/MimeType.idl) $(location HTML/MimeTypeArray.idl) $(location HTML/NavigateEvent.idl) $(location HTML/Navigation.idl) $(location HTML/NavigationCurrentEntryChangeEvent.idl) $(location HTML/NavigationDestination.idl) $(location HTML/NavigationHistoryEntry.idl) $(location HTML/NavigationTransition.idl) $(location HTML/NavigationType.idl) $(location HTML/Navigator.idl) $(location HTML/NavigatorBeacon.idl) $(location HTML/NavigatorConcurrentHardware.idl) $(location HTML/NavigatorDeviceMemory.idl) $(location HTML/NavigatorID.idl) $(location HTML/NavigatorLanguage.idl) $(location HTML/NavigatorOnLine.idl) $(location HTML/OffscreenCanvas.idl) $(location HTML/OffscreenCanvasRenderingContext2D.idl) $(location HTML/PageTransitionEvent.idl) $(location HTML/Path2D.idl) $(location HTML/Plugin.idl) $(location HTML/PluginArray.idl) $(location HTML/PopoverTargetAttributes.idl) $(location HTML/PopStateEvent.idl) $(location HTML/PredefinedColorSpace.idl) $(location HTML/PromiseRejectionEvent.idl) $(location HTML/RadioNodeList.idl) $(location HTML/Scripting/Fetching.idl) $(location HTML/SharedWorker.idl) $(location HTML/SharedWorkerGlobalScope.idl) $(location HTML/Storage.idl) $(location HTML/StorageEvent.idl) $(location HTML/SubmitEvent.idl) $(location HTML/TextMetrics.idl) $(location HTML/TextTrack.idl) $(location HTML/TextTrackCue.idl) $(location HTML/TextTrackCueList.idl) $(location HTML/TextTrackList.idl) $(location HTML/TimeRanges.idl) $(location HTML/ToggleEvent.idl) $(location HTML/TrackEvent.idl) $(location HTML/UniversalGlobalScope.idl) $(location HTML/UserActivation.idl) $(location HTML/ValidityState.idl) $(location HTML/VideoTrack.idl) $(location HTML/VideoTrackList.idl) $(location HTML/Window.idl) $(location HTML/WindowDeprecated.idl) $(location HTML/WindowLocalStorage.idl) $(location HTML/WindowOrWorkerGlobalScope.idl) $(location HTML/WindowSessionStorage.idl) $(location HTML/Worker.idl) $(location HTML/WorkerGlobalScope.idl) $(location HTML/WorkerLocation.idl) $(location HTML/WorkerNavigator.idl) $(location HTML/WorkletGlobalScope.idl) $(location HTML/XMLSerializer.idl) $(location IndexedDB/IDBCursor.idl) $(location IndexedDB/IDBCursorWithValue.idl) $(location IndexedDB/IDBDatabase.idl) $(location IndexedDB/IDBFactory.idl) $(location IndexedDB/IDBIndex.idl) $(location IndexedDB/IDBKeyRange.idl) $(location IndexedDB/IDBObjectStore.idl) $(location IndexedDB/IDBOpenDBRequest.idl) $(location IndexedDB/IDBRecord.idl) $(location IndexedDB/IDBRequest.idl) $(location IndexedDB/IDBTransaction.idl) $(location IndexedDB/IDBVersionChangeEvent.idl) $(location Internals/FakeXRDevice.idl) $(location Internals/InternalAnimationTimeline.idl) $(location Internals/InternalGamepad.idl) $(location Internals/Internals.idl) $(location Internals/WebUI.idl) $(location Internals/XRTest.idl) $(location IntersectionObserver/IntersectionObserver.idl) $(location IntersectionObserver/IntersectionObserverEntry.idl) $(location MathML/MathMLAnchorElement.idl) $(location MathML/MathMLElement.idl) $(location MediaCapabilitiesAPI/MediaCapabilities.idl) $(location MediaCapture/MediaDeviceInfo.idl) $(location MediaCapture/MediaDevices.idl) $(location MediaCapture/MediaStream.idl) $(location MediaCapture/MediaStreamConstraints.idl) $(location MediaCapture/MediaStreamTrack.idl) $(location MediaCapture/MediaStreamTrackEvent.idl) $(location MediaSourceExtensions/BufferedChangeEvent.idl) $(location MediaSourceExtensions/ManagedMediaSource.idl) $(location MediaSourceExtensions/ManagedSourceBuffer.idl) $(location MediaSourceExtensions/MediaSource.idl) $(location MediaSourceExtensions/MediaSourceHandle.idl) $(location MediaSourceExtensions/SourceBuffer.idl) $(location MediaSourceExtensions/SourceBufferList.idl) $(location NavigationTiming/PerformanceExtensions.idl) $(location NavigationTiming/PerformanceNavigation.idl) $(location NavigationTiming/PerformanceTiming.idl) $(location NotificationsAPI/Notification.idl) $(location PerformanceTimeline/PerformanceEntry.idl) $(location PerformanceTimeline/PerformanceObserver.idl) $(location PerformanceTimeline/PerformanceObserverEntryList.idl) $(location PermissionsAPI/Permissions.idl) $(location PermissionsAPI/PermissionStatus.idl) $(location RequestIdleCallback/IdleDeadline.idl) $(location RequestIdleCallback/IdleRequest.idl) $(location ResizeObserver/ResizeObserver.idl) $(location ResizeObserver/ResizeObserverEntry.idl) $(location ResizeObserver/ResizeObserverSize.idl) $(location ResourceTiming/PerformanceResourceTiming.idl) $(location Selection/Selection.idl) $(location Serial/Serial.idl) $(location Serial/SerialPort.idl) $(location ServiceWorker/Cache.idl) $(location ServiceWorker/CacheStorage.idl) $(location ServiceWorker/ServiceWorker.idl) $(location ServiceWorker/ServiceWorkerContainer.idl) $(location ServiceWorker/ServiceWorkerGlobalScope.idl) $(location ServiceWorker/ServiceWorkerRegistration.idl) $(location Speech/SpeechGrammar.idl) $(location Speech/SpeechGrammarList.idl) $(location Speech/SpeechRecognition.idl) $(location Speech/SpeechRecognitionAlternative.idl) $(location Speech/SpeechRecognitionEvent.idl) $(location Speech/SpeechRecognitionPhrase.idl) $(location Speech/SpeechRecognitionResult.idl) $(location Speech/SpeechRecognitionResultList.idl) $(location Speech/SpeechSynthesis.idl) $(location Speech/SpeechSynthesisUtterance.idl) $(location Speech/SpeechSynthesisVoice.idl) $(location StorageAPI/NavigatorStorage.idl) $(location StorageAPI/StorageManager.idl) $(location Streams/ByteLengthQueuingStrategy.idl) $(location Streams/CountQueuingStrategy.idl) $(location Streams/GenericTransformStream.idl) $(location Streams/QueuingStrategy.idl) $(location Streams/QueuingStrategyInit.idl) $(location Streams/ReadableByteStreamController.idl) $(location Streams/ReadableStream.idl) $(location Streams/ReadableStreamBYOBReader.idl) $(location Streams/ReadableStreamBYOBRequest.idl) $(location Streams/ReadableStreamDefaultController.idl) $(location Streams/ReadableStreamDefaultReader.idl) $(location Streams/ReadableStreamGenericReader.idl) $(location Streams/Transformer.idl) $(location Streams/TransformStream.idl) $(location Streams/TransformStreamDefaultController.idl) $(location Streams/UnderlyingSink.idl) $(location Streams/UnderlyingSource.idl) $(location Streams/WritableStream.idl) $(location Streams/WritableStreamDefaultController.idl) $(location Streams/WritableStreamDefaultWriter.idl) $(location SVG/SVGAElement.idl) $(location SVG/SVGAnimatedEnumeration.idl) $(location SVG/SVGAnimatedInteger.idl) $(location SVG/SVGAnimatedLength.idl) $(location SVG/SVGAnimatedLengthList.idl) $(location SVG/SVGAnimatedNumber.idl) $(location SVG/SVGAnimatedNumberList.idl) $(location SVG/SVGAnimatedRect.idl) $(location SVG/SVGAnimatedString.idl) $(location SVG/SVGAnimatedTransformList.idl) $(location SVG/SVGAnimationElement.idl) $(location SVG/SVGCircleElement.idl) $(location SVG/SVGClipPathElement.idl) $(location SVG/SVGComponentTransferFunctionElement.idl) $(location SVG/SVGDefsElement.idl) $(location SVG/SVGDescElement.idl) $(location SVG/SVGElement.idl) $(location SVG/SVGEllipseElement.idl) $(location SVG/SVGFEBlendElement.idl) $(location SVG/SVGFEColorMatrixElement.idl) $(location SVG/SVGFEComponentTransferElement.idl) $(location SVG/SVGFECompositeElement.idl) $(location SVG/SVGFEDisplacementMapElement.idl) $(location SVG/SVGFEDropShadowElement.idl) $(location SVG/SVGFEFloodElement.idl) $(location SVG/SVGFEFuncAElement.idl) $(location SVG/SVGFEFuncBElement.idl) $(location SVG/SVGFEFuncGElement.idl) $(location SVG/SVGFEFuncRElement.idl) $(location SVG/SVGFEGaussianBlurElement.idl) $(location SVG/SVGFEImageElement.idl) $(location SVG/SVGFEMergeElement.idl) $(location SVG/SVGFEMergeNodeElement.idl) $(location SVG/SVGFEMorphologyElement.idl) $(location SVG/SVGFEOffsetElement.idl) $(location SVG/SVGFETurbulenceElement.idl) $(location SVG/SVGFilterElement.idl) $(location SVG/SVGFilterPrimitiveStandardAttributes.idl) $(location SVG/SVGFitToViewBox.idl) $(location SVG/SVGForeignObjectElement.idl) $(location SVG/SVGGElement.idl) $(location SVG/SVGGeometryElement.idl) $(location SVG/SVGGradientElement.idl) $(location SVG/SVGGraphicsElement.idl) $(location SVG/SVGImageElement.idl) $(location SVG/SVGLength.idl) $(location SVG/SVGLengthList.idl) $(location SVG/SVGLinearGradientElement.idl) $(location SVG/SVGLineElement.idl) $(location SVG/SVGMaskElement.idl) $(location SVG/SVGMetadataElement.idl) $(location SVG/SVGNumber.idl) $(location SVG/SVGNumberList.idl) $(location SVG/SVGPathElement.idl) $(location SVG/SVGPatternElement.idl) $(location SVG/SVGPolygonElement.idl) $(location SVG/SVGPolylineElement.idl) $(location SVG/SVGRadialGradientElement.idl) $(location SVG/SVGRectElement.idl) $(location SVG/SVGScriptElement.idl) $(location SVG/SVGStopElement.idl) $(location SVG/SVGStyleElement.idl) $(location SVG/SVGSVGElement.idl) $(location SVG/SVGSwitchElement.idl) $(location SVG/SVGSymbolElement.idl) $(location SVG/SVGTextContentElement.idl) $(location SVG/SVGTextElement.idl) $(location SVG/SVGTextPathElement.idl) $(location SVG/SVGTextPositioningElement.idl) $(location SVG/SVGTitleElement.idl) $(location SVG/SVGTransform.idl) $(location SVG/SVGTransformList.idl) $(location SVG/SVGTSpanElement.idl) $(location SVG/SVGUnitTypes.idl) $(location SVG/SVGURIReference.idl) $(location SVG/SVGUseElement.idl) $(location SVG/SVGViewElement.idl) $(location TrustedTypes/TrustedHTML.idl) $(location TrustedTypes/TrustedScript.idl) $(location TrustedTypes/TrustedScriptURL.idl) $(location TrustedTypes/TrustedTypePolicy.idl) $(location TrustedTypes/TrustedTypePolicyFactory.idl) $(location UIEvents/CompositionEvent.idl) $(location UIEvents/EventModifier.idl) $(location UIEvents/FocusEvent.idl) $(location UIEvents/InputEvent.idl) $(location UIEvents/KeyboardEvent.idl) $(location UIEvents/MouseEvent.idl) $(location UIEvents/PointerEvent.idl) $(location UIEvents/PointerEventHandlers.idl) $(location UIEvents/TextEvent.idl) $(location UIEvents/UIEvent.idl) $(location UIEvents/WheelEvent.idl) $(location URLPattern/URLPattern.idl) $(location UserTiming/PerformanceMark.idl) $(location UserTiming/PerformanceMeasure.idl) $(location ViewTransition/ViewTransition.idl) $(location WebLocks/Lock.idl) $(location WebLocks/LockManager.idl) $(location WebLocks/NavigatorLocks.idl) $(location WebAssembly/Global.idl) $(location WebAssembly/Instance.idl) $(location WebAssembly/Memory.idl) $(location WebAssembly/Module.idl) $(location WebAssembly/Table.idl) $(location WebAssembly/WebAssembly.idl) $(location WebAudio/AnalyserNode.idl) $(location WebAudio/AudioBuffer.idl) $(location WebAudio/AudioBufferSourceNode.idl) $(location WebAudio/AudioContext.idl) $(location WebAudio/AudioDestinationNode.idl) $(location WebAudio/AudioListener.idl) $(location WebAudio/AudioNode.idl) $(location WebAudio/AudioParam.idl) $(location WebAudio/AudioScheduledSourceNode.idl) $(location WebAudio/BaseAudioContext.idl) $(location WebAudio/BiquadFilterNode.idl) $(location WebAudio/ChannelMergerNode.idl) $(location WebAudio/ChannelSplitterNode.idl) $(location WebAudio/ConstantSourceNode.idl) $(location WebAudio/DelayNode.idl) $(location WebAudio/DynamicsCompressorNode.idl) $(location WebAudio/GainNode.idl) $(location WebAudio/MediaElementAudioSourceNode.idl) $(location WebAudio/OfflineAudioCompletionEvent.idl) $(location WebAudio/OfflineAudioContext.idl) $(location WebAudio/OscillatorNode.idl) $(location WebAudio/PannerNode.idl) $(location WebAudio/PeriodicWave.idl) $(location WebAudio/ScriptProcessorNode.idl) $(location WebAudio/StereoPannerNode.idl) $(location WebGL/Extensions/ANGLEInstancedArrays.idl) $(location WebGL/Extensions/EXTBlendMinMax.idl) $(location WebGL/Extensions/EXTColorBufferFloat.idl) $(location WebGL/Extensions/EXTRenderSnorm.idl) $(location WebGL/Extensions/EXTTextureFilterAnisotropic.idl) $(location WebGL/Extensions/EXTTextureNorm16.idl) $(location WebGL/Extensions/OESElementIndexUint.idl) $(location WebGL/Extensions/OESStandardDerivatives.idl) $(location WebGL/Extensions/OESVertexArrayObject.idl) $(location WebGL/Extensions/WebGLCompressedTextureS3tc.idl) $(location WebGL/Extensions/WebGLCompressedTextureS3tcSrgb.idl) $(location WebGL/Extensions/WebGLDebugRendererInfo.idl) $(location WebGL/Extensions/WebGLDrawBuffers.idl) $(location WebGL/Extensions/WebGLVertexArrayObjectOES.idl) $(location WebGL/Types.idl) $(location WebGL/WebGL2RenderingContext.idl) $(location WebGL/WebGL2RenderingContextBase.idl) $(location WebGL/WebGL2RenderingContextOverloads.idl) $(location WebGL/WebGLActiveInfo.idl) $(location WebGL/WebGLBuffer.idl) $(location WebGL/WebGLContextEvent.idl) $(location WebGL/WebGLFramebuffer.idl) $(location WebGL/WebGLObject.idl) $(location WebGL/WebGLProgram.idl) $(location WebGL/WebGLQuery.idl) $(location WebGL/WebGLRenderbuffer.idl) $(location WebGL/WebGLRenderingContext.idl) $(location WebGL/WebGLRenderingContextBase.idl) $(location WebGL/WebGLRenderingContextOverloads.idl) $(location WebGL/WebGLSampler.idl) $(location WebGL/WebGLShader.idl) $(location WebGL/WebGLShaderPrecisionFormat.idl) $(location WebGL/WebGLSync.idl) $(location WebGL/WebGLTexture.idl) $(location WebGL/WebGLTransformFeedback.idl) $(location WebGL/WebGLUniformLocation.idl) $(location WebGL/WebGLVertexArrayObject.idl) $(location WebIDL/Buffers.idl) $(location WebIDL/DOMException.idl) $(location WebIDL/Function.idl) $(location WebIDL/QuotaExceededError.idl) $(location WebSockets/WebSocket.idl) $(location WebVTT/VTTCue.idl) $(location WebVTT/VTTRegion.idl) $(location WebXR/XRLayer.idl) $(location WebXR/XRRenderState.idl) $(location WebXR/XRSession.idl) $(location WebXR/XRSessionEvent.idl) $(location WebXR/XRSystem.idl) $(location WebXR/XRWebGLLayer.idl) $(location XHR/FormData.idl) $(location XHR/ProgressEvent.idl) $(location XHR/XMLHttpRequest.idl) $(location XHR/XMLHttpRequestEventTarget.idl) $(location XHR/XMLHttpRequestUpload.idl) $(location XPath/XPathEvaluator.idl) $(location XPath/XPathExpression.idl) $(location XPath/XPathNSResolver.idl) $(location XPath/XPathResult.idl)', + cmd = 'PYTHONHASHSEED=0 python3 Meta/Generators/generate_libweb_bindings.py -o $(RULEDIR)/Bindings $(location Animations/Animatable.idl) $(location Animations/Animation.idl) $(location Animations/AnimationEffect.idl) $(location Animations/AnimationPlaybackEvent.idl) $(location Animations/AnimationTimeline.idl) $(location Animations/DocumentTimeline.idl) $(location Animations/KeyframeEffect.idl) $(location Animations/ScrollTimeline.idl) $(location ARIA/ARIAMixin.idl) $(location Clipboard/Clipboard.idl) $(location Clipboard/ClipboardEvent.idl) $(location Clipboard/ClipboardItem.idl) $(location Compression/CompressionStream.idl) $(location Compression/DecompressionStream.idl) $(location ContentSecurityPolicy/SecurityPolicyViolationEvent.idl) $(location CookieStore/CookieChangeEvent.idl) $(location CookieStore/CookieStore.idl) $(location CredentialManagement/Credential.idl) $(location CredentialManagement/CredentialsContainer.idl) $(location CredentialManagement/FederatedCredential.idl) $(location CredentialManagement/PasswordCredential.idl) $(location Crypto/Crypto.idl) $(location Crypto/CryptoKey.idl) $(location Crypto/SubtleCrypto.idl) $(location CSS/AnimationEvent.idl) $(location CSS/CSS.idl) $(location CSS/CSSAnimation.idl) $(location CSS/CSSConditionRule.idl) $(location CSS/CSSContainerRule.idl) $(location CSS/CSSCounterStyleRule.idl) $(location CSS/CSSFontFaceDescriptors.idl) $(location CSS/CSSFontFaceRule.idl) $(location CSS/CSSFontFeatureValuesMap.idl) $(location CSS/CSSFontFeatureValuesRule.idl) $(location CSS/CSSFunctionDeclarations.idl) $(location CSS/CSSFunctionDescriptors.idl) $(location CSS/CSSFunctionRule.idl) $(location CSS/CSSGroupingRule.idl) $(location CSS/CSSImageValue.idl) $(location CSS/CSSImportRule.idl) $(location CSS/CSSKeyframeRule.idl) $(location CSS/CSSKeyframesRule.idl) $(location CSS/CSSKeywordValue.idl) $(location CSS/CSSLayerBlockRule.idl) $(location CSS/CSSLayerStatementRule.idl) $(location CSS/CSSMarginRule.idl) $(location CSS/CSSMathClamp.idl) $(location CSS/CSSMathInvert.idl) $(location CSS/CSSMathMax.idl) $(location CSS/CSSMathMin.idl) $(location CSS/CSSMathNegate.idl) $(location CSS/CSSMathProduct.idl) $(location CSS/CSSMathSum.idl) $(location CSS/CSSMathValue.idl) $(location CSS/CSSMatrixComponent.idl) $(location CSS/CSSMediaRule.idl) $(location CSS/CSSNamespaceRule.idl) $(location CSS/CSSNestedDeclarations.idl) $(location CSS/CSSNumericArray.idl) $(location CSS/CSSNumericValue.idl) $(location CSS/CSSPageDescriptors.idl) $(location CSS/CSSPageRule.idl) $(location CSS/CSSPerspective.idl) $(location CSS/CSSPropertyRule.idl) $(location CSS/CSSRotate.idl) $(location CSS/CSSRule.idl) $(location CSS/CSSRuleList.idl) $(location CSS/CSSScale.idl) $(location CSS/CSSScopeRule.idl) $(location CSS/CSSSkew.idl) $(location CSS/CSSSkewX.idl) $(location CSS/CSSSkewY.idl) $(location CSS/CSSStyleDeclaration.idl) $(location CSS/CSSStyleProperties.idl) $(location CSS/CSSStyleRule.idl) $(location CSS/CSSStyleSheet.idl) $(location CSS/CSSStyleValue.idl) $(location CSS/CSSSupportsRule.idl) $(location CSS/CSSTransformComponent.idl) $(location CSS/CSSTransformValue.idl) $(location CSS/CSSTransition.idl) $(location CSS/CSSTranslate.idl) $(location CSS/CSSUnitValue.idl) $(location CSS/CSSUnparsedValue.idl) $(location CSS/CSSVariableReferenceValue.idl) $(location CSS/ElementCSSInlineStyle.idl) $(location CSS/FontFace.idl) $(location CSS/FontFaceSet.idl) $(location CSS/FontFaceSetLoadEvent.idl) $(location CSS/GeneratedCSSNumericFactoryMethods.idl) $(location CSS/GeneratedCSSStyleProperties.idl) $(location CSS/LinkStyle.idl) $(location CSS/MediaList.idl) $(location CSS/MediaQueryList.idl) $(location CSS/MediaQueryListEvent.idl) $(location CSS/Screen.idl) $(location CSS/ScreenOrientation.idl) $(location CSS/StylePropertyMap.idl) $(location CSS/StylePropertyMapReadOnly.idl) $(location CSS/StyleSheet.idl) $(location CSS/StyleSheetList.idl) $(location CSS/TransitionEvent.idl) $(location CSS/VisualViewport.idl) $(location DOM/AbortController.idl) $(location DOM/AbortSignal.idl) $(location DOM/AbstractRange.idl) $(location DOM/Attr.idl) $(location DOM/CDATASection.idl) $(location DOM/CaretPosition.idl) $(location DOM/CharacterData.idl) $(location DOM/ChildNode.idl) $(location DOM/Comment.idl) $(location DOM/CustomEvent.idl) $(location DOM/Document.idl) $(location DOM/DocumentFragment.idl) $(location DOM/DocumentOrShadowRoot.idl) $(location DOM/DocumentType.idl) $(location DOM/DOMImplementation.idl) $(location DOM/DOMTokenList.idl) $(location DOM/Element.idl) $(location DOM/Event.idl) $(location DOM/EventHandler.idl) $(location DOM/EventListener.idl) $(location DOM/EventTarget.idl) $(location DOM/HTMLCollection.idl) $(location DOM/MutationObserver.idl) $(location DOM/MutationRecord.idl) $(location DOM/NamedNodeMap.idl) $(location DOM/Node.idl) $(location DOM/NodeFilter.idl) $(location DOM/NodeIterator.idl) $(location DOM/NodeList.idl) $(location DOM/ParentNode.idl) $(location DOM/ProcessingInstruction.idl) $(location DOM/Range.idl) $(location DOM/ShadowRoot.idl) $(location DOM/Slottable.idl) $(location DOM/StaticRange.idl) $(location DOM/Text.idl) $(location DOM/TreeWalker.idl) $(location DOM/XMLDocument.idl) $(location DOMURL/DOMURL.idl) $(location DOMURL/Origin.idl) $(location DOMURL/URLSearchParams.idl) $(location Encoding/TextDecoder.idl) $(location Encoding/TextDecoderCommon.idl) $(location Encoding/TextDecoderStream.idl) $(location Encoding/TextEncoder.idl) $(location Encoding/TextEncoderCommon.idl) $(location Encoding/TextEncoderStream.idl) $(location EncryptedMediaExtensions/MediaKeySystemAccess.idl) $(location EntriesAPI/FileSystemEntry.idl) $(location EventTiming/PerformanceEventTiming.idl) $(location Fetch/Body.idl) $(location Fetch/BodyInit.idl) $(location Fetch/Headers.idl) $(location Fetch/Request.idl) $(location Fetch/Response.idl) $(location FileAPI/Blob.idl) $(location FileAPI/File.idl) $(location FileAPI/FileList.idl) $(location FileAPI/FileReader.idl) $(location FileAPI/FileReaderSync.idl) $(location Fullscreen/DocumentExtensions.idl) $(location Fullscreen/DocumentOrShadowRootExtensions.idl) $(location Fullscreen/ElementExtensions.idl) $(location Gamepad/Gamepad.idl) $(location Gamepad/GamepadButton.idl) $(location Gamepad/GamepadEvent.idl) $(location Gamepad/GamepadHapticActuator.idl) $(location Geolocation/Geolocation.idl) $(location Geolocation/GeolocationCoordinates.idl) $(location Geolocation/GeolocationPosition.idl) $(location Geolocation/GeolocationPositionError.idl) $(location Geometry/DOMMatrix.idl) $(location Geometry/DOMMatrixReadOnly.idl) $(location Geometry/DOMPoint.idl) $(location Geometry/DOMPointReadOnly.idl) $(location Geometry/DOMQuad.idl) $(location Geometry/DOMRect.idl) $(location Geometry/DOMRectList.idl) $(location Geometry/DOMRectReadOnly.idl) $(location GPC/GlobalPrivacyControl.idl) $(location HighResolutionTime/DOMHighResTimeStamp.idl) $(location HighResolutionTime/EpochTimeStamp.idl) $(location HighResolutionTime/Performance.idl) $(location HTML/AbstractWorker.idl) $(location HTML/AnimationFrameProvider.idl) $(location HTML/AudioTrack.idl) $(location HTML/AudioTrackList.idl) $(location HTML/BarProp.idl) $(location HTML/BeforeUnloadEvent.idl) $(location HTML/BroadcastChannel.idl) $(location HTML/Canvas/CanvasCompositing.idl) $(location HTML/Canvas/CanvasDrawImage.idl) $(location HTML/Canvas/CanvasDrawPath.idl) $(location HTML/Canvas/CanvasFillStrokeStyles.idl) $(location HTML/Canvas/CanvasFilters.idl) $(location HTML/Canvas/CanvasImageData.idl) $(location HTML/Canvas/CanvasImageSmoothing.idl) $(location HTML/Canvas/CanvasPath.idl) $(location HTML/Canvas/CanvasPathDrawingStyles.idl) $(location HTML/Canvas/CanvasRect.idl) $(location HTML/Canvas/CanvasSettings.idl) $(location HTML/Canvas/CanvasShadowStyles.idl) $(location HTML/Canvas/CanvasState.idl) $(location HTML/Canvas/CanvasText.idl) $(location HTML/Canvas/CanvasTextDrawingStyles.idl) $(location HTML/Canvas/CanvasTransform.idl) $(location HTML/Canvas/CanvasUserInterface.idl) $(location HTML/Canvas/OffscreenCanvasBase.idl) $(location HTML/CanvasGradient.idl) $(location HTML/CanvasPattern.idl) $(location HTML/CanvasRenderingContext2D.idl) $(location HTML/CanvasRenderingContext2DSettings.idl) $(location HTML/CloseEvent.idl) $(location HTML/CloseWatcher.idl) $(location HTML/CommandEvent.idl) $(location HTML/CustomElements/CustomElementRegistry.idl) $(location HTML/CustomElements/CustomStateSet.idl) $(location HTML/DataTransfer.idl) $(location HTML/DataTransferItem.idl) $(location HTML/DataTransferItemList.idl) $(location HTML/DedicatedWorkerGlobalScope.idl) $(location HTML/DOMParser.idl) $(location HTML/DOMStringList.idl) $(location HTML/DOMStringMap.idl) $(location HTML/DragEvent.idl) $(location HTML/ElementInternals.idl) $(location HTML/ErrorEvent.idl) $(location HTML/EventSource.idl) $(location HTML/External.idl) $(location HTML/FormDataEvent.idl) $(location HTML/HashChangeEvent.idl) $(location HTML/History.idl) $(location HTML/HTMLAllCollection.idl) $(location HTML/HTMLAnchorElement.idl) $(location HTML/HTMLAreaElement.idl) $(location HTML/HTMLAudioElement.idl) $(location HTML/HTMLBaseElement.idl) $(location HTML/HTMLBodyElement.idl) $(location HTML/HTMLBRElement.idl) $(location HTML/HTMLButtonElement.idl) $(location HTML/HTMLCanvasElement.idl) $(location HTML/HTMLDataElement.idl) $(location HTML/HTMLDataListElement.idl) $(location HTML/HTMLDetailsElement.idl) $(location HTML/HTMLDialogElement.idl) $(location HTML/HTMLDirectoryElement.idl) $(location HTML/HTMLDivElement.idl) $(location HTML/HTMLDListElement.idl) $(location HTML/HTMLDocument.idl) $(location HTML/HTMLElement.idl) $(location HTML/HTMLEmbedElement.idl) $(location HTML/HTMLFieldSetElement.idl) $(location HTML/HTMLFontElement.idl) $(location HTML/HTMLFormControlsCollection.idl) $(location HTML/HTMLFormElement.idl) $(location HTML/HTMLFrameElement.idl) $(location HTML/HTMLFrameSetElement.idl) $(location HTML/HTMLHeadElement.idl) $(location HTML/HTMLHeadingElement.idl) $(location HTML/HTMLHRElement.idl) $(location HTML/HTMLHtmlElement.idl) $(location HTML/HTMLHyperlinkElementUtils.idl) $(location HTML/HTMLIFrameElement.idl) $(location HTML/HTMLImageElement.idl) $(location HTML/HTMLInputElement.idl) $(location HTML/HTMLLabelElement.idl) $(location HTML/HTMLLegendElement.idl) $(location HTML/HTMLLIElement.idl) $(location HTML/HTMLLinkElement.idl) $(location HTML/HTMLMapElement.idl) $(location HTML/HTMLMarqueeElement.idl) $(location HTML/HTMLMediaElement.idl) $(location HTML/HTMLMenuElement.idl) $(location HTML/HTMLMetaElement.idl) $(location HTML/HTMLMeterElement.idl) $(location HTML/HTMLModElement.idl) $(location HTML/HTMLObjectElement.idl) $(location HTML/HTMLOListElement.idl) $(location HTML/HTMLOptGroupElement.idl) $(location HTML/HTMLOptionElement.idl) $(location HTML/HTMLOptionsCollection.idl) $(location HTML/HTMLOrSVGOrMathMLElement.idl) $(location HTML/HTMLOutputElement.idl) $(location HTML/HTMLParagraphElement.idl) $(location HTML/HTMLParamElement.idl) $(location HTML/HTMLPictureElement.idl) $(location HTML/HTMLPreElement.idl) $(location HTML/HTMLProgressElement.idl) $(location HTML/HTMLQuoteElement.idl) $(location HTML/HTMLScriptElement.idl) $(location HTML/HTMLSelectedContentElement.idl) $(location HTML/HTMLSelectElement.idl) $(location HTML/HTMLSlotElement.idl) $(location HTML/HTMLSourceElement.idl) $(location HTML/HTMLSpanElement.idl) $(location HTML/HTMLStyleElement.idl) $(location HTML/HTMLTableCaptionElement.idl) $(location HTML/HTMLTableCellElement.idl) $(location HTML/HTMLTableColElement.idl) $(location HTML/HTMLTableElement.idl) $(location HTML/HTMLTableRowElement.idl) $(location HTML/HTMLTableSectionElement.idl) $(location HTML/HTMLTemplateElement.idl) $(location HTML/HTMLTextAreaElement.idl) $(location HTML/HTMLTimeElement.idl) $(location HTML/HTMLTitleElement.idl) $(location HTML/HTMLTrackElement.idl) $(location HTML/HTMLUListElement.idl) $(location HTML/HTMLUnknownElement.idl) $(location HTML/HTMLVideoElement.idl) $(location HTML/HyperlinkElementUtils.idl) $(location HTML/ImageBitmap.idl) $(location HTML/ImageData.idl) $(location HTML/Location.idl) $(location HTML/MediaError.idl) $(location HTML/MessageChannel.idl) $(location HTML/MessageEvent.idl) $(location HTML/MessagePort.idl) $(location HTML/MimeType.idl) $(location HTML/MimeTypeArray.idl) $(location HTML/NavigateEvent.idl) $(location HTML/Navigation.idl) $(location HTML/NavigationActivation.idl) $(location HTML/NavigationCurrentEntryChangeEvent.idl) $(location HTML/NavigationDestination.idl) $(location HTML/NavigationHistoryEntry.idl) $(location HTML/NavigationTransition.idl) $(location HTML/NavigationType.idl) $(location HTML/Navigator.idl) $(location HTML/NavigatorBeacon.idl) $(location HTML/NavigatorConcurrentHardware.idl) $(location HTML/NavigatorDeviceMemory.idl) $(location HTML/NavigatorID.idl) $(location HTML/NavigatorLanguage.idl) $(location HTML/NavigatorOnLine.idl) $(location HTML/OffscreenCanvas.idl) $(location HTML/OffscreenCanvasRenderingContext2D.idl) $(location HTML/PageSwapEvent.idl) $(location HTML/PageTransitionEvent.idl) $(location HTML/Path2D.idl) $(location HTML/Plugin.idl) $(location HTML/PluginArray.idl) $(location HTML/PopoverTargetAttributes.idl) $(location HTML/PopStateEvent.idl) $(location HTML/PredefinedColorSpace.idl) $(location HTML/PromiseRejectionEvent.idl) $(location HTML/RadioNodeList.idl) $(location HTML/Scripting/Fetching.idl) $(location HTML/SharedWorker.idl) $(location HTML/SharedWorkerGlobalScope.idl) $(location HTML/Storage.idl) $(location HTML/StorageEvent.idl) $(location HTML/SubmitEvent.idl) $(location HTML/TextMetrics.idl) $(location HTML/TextTrack.idl) $(location HTML/TextTrackCue.idl) $(location HTML/TextTrackCueList.idl) $(location HTML/TextTrackList.idl) $(location HTML/TimeRanges.idl) $(location HTML/ToggleEvent.idl) $(location HTML/TrackEvent.idl) $(location HTML/UniversalGlobalScope.idl) $(location HTML/UserActivation.idl) $(location HTML/ValidityState.idl) $(location HTML/VideoTrack.idl) $(location HTML/VideoTrackList.idl) $(location HTML/Window.idl) $(location HTML/WindowDeprecated.idl) $(location HTML/WindowLocalStorage.idl) $(location HTML/WindowOrWorkerGlobalScope.idl) $(location HTML/WindowSessionStorage.idl) $(location HTML/Worker.idl) $(location HTML/WorkerGlobalScope.idl) $(location HTML/WorkerLocation.idl) $(location HTML/WorkerNavigator.idl) $(location HTML/WorkletGlobalScope.idl) $(location HTML/XMLSerializer.idl) $(location IndexedDB/IDBCursor.idl) $(location IndexedDB/IDBCursorWithValue.idl) $(location IndexedDB/IDBDatabase.idl) $(location IndexedDB/IDBFactory.idl) $(location IndexedDB/IDBIndex.idl) $(location IndexedDB/IDBKeyRange.idl) $(location IndexedDB/IDBObjectStore.idl) $(location IndexedDB/IDBOpenDBRequest.idl) $(location IndexedDB/IDBRecord.idl) $(location IndexedDB/IDBRequest.idl) $(location IndexedDB/IDBTransaction.idl) $(location IndexedDB/IDBVersionChangeEvent.idl) $(location Internals/FakeXRDevice.idl) $(location Internals/InternalAnimationTimeline.idl) $(location Internals/InternalGamepad.idl) $(location Internals/Internals.idl) $(location Internals/WebUI.idl) $(location Internals/XRTest.idl) $(location IntersectionObserver/IntersectionObserver.idl) $(location IntersectionObserver/IntersectionObserverEntry.idl) $(location MathML/MathMLAnchorElement.idl) $(location MathML/MathMLElement.idl) $(location MediaCapabilitiesAPI/MediaCapabilities.idl) $(location MediaCapture/MediaDeviceInfo.idl) $(location MediaCapture/MediaDevices.idl) $(location MediaCapture/MediaStream.idl) $(location MediaCapture/MediaStreamConstraints.idl) $(location MediaCapture/MediaStreamTrack.idl) $(location MediaCapture/MediaStreamTrackEvent.idl) $(location MediaSourceExtensions/BufferedChangeEvent.idl) $(location MediaSourceExtensions/ManagedMediaSource.idl) $(location MediaSourceExtensions/ManagedSourceBuffer.idl) $(location MediaSourceExtensions/MediaSource.idl) $(location MediaSourceExtensions/MediaSourceHandle.idl) $(location MediaSourceExtensions/SourceBuffer.idl) $(location MediaSourceExtensions/SourceBufferList.idl) $(location NavigationTiming/PerformanceExtensions.idl) $(location NavigationTiming/PerformanceNavigation.idl) $(location NavigationTiming/PerformanceTiming.idl) $(location NotificationsAPI/Notification.idl) $(location PerformanceTimeline/PerformanceEntry.idl) $(location PerformanceTimeline/PerformanceObserver.idl) $(location PerformanceTimeline/PerformanceObserverEntryList.idl) $(location PermissionsAPI/Permissions.idl) $(location PermissionsAPI/PermissionStatus.idl) $(location RequestIdleCallback/IdleDeadline.idl) $(location RequestIdleCallback/IdleRequest.idl) $(location ResizeObserver/ResizeObserver.idl) $(location ResizeObserver/ResizeObserverEntry.idl) $(location ResizeObserver/ResizeObserverSize.idl) $(location ResourceTiming/PerformanceResourceTiming.idl) $(location Selection/Selection.idl) $(location Serial/Serial.idl) $(location Serial/SerialPort.idl) $(location ServiceWorker/Cache.idl) $(location ServiceWorker/CacheStorage.idl) $(location ServiceWorker/ServiceWorker.idl) $(location ServiceWorker/ServiceWorkerContainer.idl) $(location ServiceWorker/ServiceWorkerGlobalScope.idl) $(location ServiceWorker/ServiceWorkerRegistration.idl) $(location Speech/SpeechGrammar.idl) $(location Speech/SpeechGrammarList.idl) $(location Speech/SpeechRecognition.idl) $(location Speech/SpeechRecognitionAlternative.idl) $(location Speech/SpeechRecognitionEvent.idl) $(location Speech/SpeechRecognitionPhrase.idl) $(location Speech/SpeechRecognitionResult.idl) $(location Speech/SpeechRecognitionResultList.idl) $(location Speech/SpeechSynthesis.idl) $(location Speech/SpeechSynthesisUtterance.idl) $(location Speech/SpeechSynthesisVoice.idl) $(location StorageAPI/NavigatorStorage.idl) $(location StorageAPI/StorageManager.idl) $(location Streams/ByteLengthQueuingStrategy.idl) $(location Streams/CountQueuingStrategy.idl) $(location Streams/GenericTransformStream.idl) $(location Streams/QueuingStrategy.idl) $(location Streams/QueuingStrategyInit.idl) $(location Streams/ReadableByteStreamController.idl) $(location Streams/ReadableStream.idl) $(location Streams/ReadableStreamBYOBReader.idl) $(location Streams/ReadableStreamBYOBRequest.idl) $(location Streams/ReadableStreamDefaultController.idl) $(location Streams/ReadableStreamDefaultReader.idl) $(location Streams/ReadableStreamGenericReader.idl) $(location Streams/Transformer.idl) $(location Streams/TransformStream.idl) $(location Streams/TransformStreamDefaultController.idl) $(location Streams/UnderlyingSink.idl) $(location Streams/UnderlyingSource.idl) $(location Streams/WritableStream.idl) $(location Streams/WritableStreamDefaultController.idl) $(location Streams/WritableStreamDefaultWriter.idl) $(location SVG/SVGAElement.idl) $(location SVG/SVGAnimatedEnumeration.idl) $(location SVG/SVGAnimatedInteger.idl) $(location SVG/SVGAnimatedLength.idl) $(location SVG/SVGAnimatedLengthList.idl) $(location SVG/SVGAnimatedNumber.idl) $(location SVG/SVGAnimatedNumberList.idl) $(location SVG/SVGAnimatedRect.idl) $(location SVG/SVGAnimatedString.idl) $(location SVG/SVGAnimatedTransformList.idl) $(location SVG/SVGAnimationElement.idl) $(location SVG/SVGCircleElement.idl) $(location SVG/SVGClipPathElement.idl) $(location SVG/SVGComponentTransferFunctionElement.idl) $(location SVG/SVGDefsElement.idl) $(location SVG/SVGDescElement.idl) $(location SVG/SVGElement.idl) $(location SVG/SVGEllipseElement.idl) $(location SVG/SVGFEBlendElement.idl) $(location SVG/SVGFEColorMatrixElement.idl) $(location SVG/SVGFEComponentTransferElement.idl) $(location SVG/SVGFECompositeElement.idl) $(location SVG/SVGFEDisplacementMapElement.idl) $(location SVG/SVGFEDropShadowElement.idl) $(location SVG/SVGFEFloodElement.idl) $(location SVG/SVGFEFuncAElement.idl) $(location SVG/SVGFEFuncBElement.idl) $(location SVG/SVGFEFuncGElement.idl) $(location SVG/SVGFEFuncRElement.idl) $(location SVG/SVGFEGaussianBlurElement.idl) $(location SVG/SVGFEImageElement.idl) $(location SVG/SVGFEMergeElement.idl) $(location SVG/SVGFEMergeNodeElement.idl) $(location SVG/SVGFEMorphologyElement.idl) $(location SVG/SVGFEOffsetElement.idl) $(location SVG/SVGFETurbulenceElement.idl) $(location SVG/SVGFilterElement.idl) $(location SVG/SVGFilterPrimitiveStandardAttributes.idl) $(location SVG/SVGFitToViewBox.idl) $(location SVG/SVGForeignObjectElement.idl) $(location SVG/SVGGElement.idl) $(location SVG/SVGGeometryElement.idl) $(location SVG/SVGGradientElement.idl) $(location SVG/SVGGraphicsElement.idl) $(location SVG/SVGImageElement.idl) $(location SVG/SVGLength.idl) $(location SVG/SVGLengthList.idl) $(location SVG/SVGLinearGradientElement.idl) $(location SVG/SVGLineElement.idl) $(location SVG/SVGMaskElement.idl) $(location SVG/SVGMetadataElement.idl) $(location SVG/SVGNumber.idl) $(location SVG/SVGNumberList.idl) $(location SVG/SVGPathElement.idl) $(location SVG/SVGPatternElement.idl) $(location SVG/SVGPolygonElement.idl) $(location SVG/SVGPolylineElement.idl) $(location SVG/SVGRadialGradientElement.idl) $(location SVG/SVGRectElement.idl) $(location SVG/SVGScriptElement.idl) $(location SVG/SVGStopElement.idl) $(location SVG/SVGStyleElement.idl) $(location SVG/SVGSVGElement.idl) $(location SVG/SVGSwitchElement.idl) $(location SVG/SVGSymbolElement.idl) $(location SVG/SVGTextContentElement.idl) $(location SVG/SVGTextElement.idl) $(location SVG/SVGTextPathElement.idl) $(location SVG/SVGTextPositioningElement.idl) $(location SVG/SVGTitleElement.idl) $(location SVG/SVGTransform.idl) $(location SVG/SVGTransformList.idl) $(location SVG/SVGTSpanElement.idl) $(location SVG/SVGUnitTypes.idl) $(location SVG/SVGURIReference.idl) $(location SVG/SVGUseElement.idl) $(location SVG/SVGViewElement.idl) $(location TrustedTypes/TrustedHTML.idl) $(location TrustedTypes/TrustedScript.idl) $(location TrustedTypes/TrustedScriptURL.idl) $(location TrustedTypes/TrustedTypePolicy.idl) $(location TrustedTypes/TrustedTypePolicyFactory.idl) $(location UIEvents/CompositionEvent.idl) $(location UIEvents/EventModifier.idl) $(location UIEvents/FocusEvent.idl) $(location UIEvents/InputEvent.idl) $(location UIEvents/KeyboardEvent.idl) $(location UIEvents/MouseEvent.idl) $(location UIEvents/PointerEvent.idl) $(location UIEvents/PointerEventHandlers.idl) $(location UIEvents/TextEvent.idl) $(location UIEvents/UIEvent.idl) $(location UIEvents/WheelEvent.idl) $(location URLPattern/URLPattern.idl) $(location UserTiming/PerformanceMark.idl) $(location UserTiming/PerformanceMeasure.idl) $(location ViewTransition/ViewTransition.idl) $(location WebLocks/Lock.idl) $(location WebLocks/LockManager.idl) $(location WebLocks/NavigatorLocks.idl) $(location WebAssembly/Global.idl) $(location WebAssembly/Instance.idl) $(location WebAssembly/Memory.idl) $(location WebAssembly/Module.idl) $(location WebAssembly/Table.idl) $(location WebAssembly/WebAssembly.idl) $(location WebAudio/AnalyserNode.idl) $(location WebAudio/AudioBuffer.idl) $(location WebAudio/AudioBufferSourceNode.idl) $(location WebAudio/AudioContext.idl) $(location WebAudio/AudioDestinationNode.idl) $(location WebAudio/AudioListener.idl) $(location WebAudio/AudioNode.idl) $(location WebAudio/AudioParam.idl) $(location WebAudio/AudioScheduledSourceNode.idl) $(location WebAudio/BaseAudioContext.idl) $(location WebAudio/BiquadFilterNode.idl) $(location WebAudio/ChannelMergerNode.idl) $(location WebAudio/ChannelSplitterNode.idl) $(location WebAudio/ConstantSourceNode.idl) $(location WebAudio/DelayNode.idl) $(location WebAudio/DynamicsCompressorNode.idl) $(location WebAudio/GainNode.idl) $(location WebAudio/MediaElementAudioSourceNode.idl) $(location WebAudio/OfflineAudioCompletionEvent.idl) $(location WebAudio/OfflineAudioContext.idl) $(location WebAudio/OscillatorNode.idl) $(location WebAudio/PannerNode.idl) $(location WebAudio/PeriodicWave.idl) $(location WebAudio/ScriptProcessorNode.idl) $(location WebAudio/StereoPannerNode.idl) $(location WebGL/Extensions/ANGLEInstancedArrays.idl) $(location WebGL/Extensions/EXTBlendMinMax.idl) $(location WebGL/Extensions/EXTColorBufferFloat.idl) $(location WebGL/Extensions/EXTRenderSnorm.idl) $(location WebGL/Extensions/EXTTextureFilterAnisotropic.idl) $(location WebGL/Extensions/EXTTextureNorm16.idl) $(location WebGL/Extensions/OESElementIndexUint.idl) $(location WebGL/Extensions/OESStandardDerivatives.idl) $(location WebGL/Extensions/OESVertexArrayObject.idl) $(location WebGL/Extensions/WebGLCompressedTextureS3tc.idl) $(location WebGL/Extensions/WebGLCompressedTextureS3tcSrgb.idl) $(location WebGL/Extensions/WebGLDebugRendererInfo.idl) $(location WebGL/Extensions/WebGLDrawBuffers.idl) $(location WebGL/Extensions/WebGLVertexArrayObjectOES.idl) $(location WebGL/Types.idl) $(location WebGL/WebGL2RenderingContext.idl) $(location WebGL/WebGL2RenderingContextBase.idl) $(location WebGL/WebGL2RenderingContextOverloads.idl) $(location WebGL/WebGLActiveInfo.idl) $(location WebGL/WebGLBuffer.idl) $(location WebGL/WebGLContextEvent.idl) $(location WebGL/WebGLFramebuffer.idl) $(location WebGL/WebGLObject.idl) $(location WebGL/WebGLProgram.idl) $(location WebGL/WebGLQuery.idl) $(location WebGL/WebGLRenderbuffer.idl) $(location WebGL/WebGLRenderingContext.idl) $(location WebGL/WebGLRenderingContextBase.idl) $(location WebGL/WebGLRenderingContextOverloads.idl) $(location WebGL/WebGLSampler.idl) $(location WebGL/WebGLShader.idl) $(location WebGL/WebGLShaderPrecisionFormat.idl) $(location WebGL/WebGLSync.idl) $(location WebGL/WebGLTexture.idl) $(location WebGL/WebGLTransformFeedback.idl) $(location WebGL/WebGLUniformLocation.idl) $(location WebGL/WebGLVertexArrayObject.idl) $(location WebIDL/Buffers.idl) $(location WebIDL/DOMException.idl) $(location WebIDL/Function.idl) $(location WebIDL/QuotaExceededError.idl) $(location WebSockets/WebSocket.idl) $(location WebVTT/VTTCue.idl) $(location WebVTT/VTTRegion.idl) $(location WebXR/XRLayer.idl) $(location WebXR/XRRenderState.idl) $(location WebXR/XRSession.idl) $(location WebXR/XRSessionEvent.idl) $(location WebXR/XRSystem.idl) $(location WebXR/XRWebGLLayer.idl) $(location XHR/FormData.idl) $(location XHR/ProgressEvent.idl) $(location XHR/XMLHttpRequest.idl) $(location XHR/XMLHttpRequestEventTarget.idl) $(location XHR/XMLHttpRequestUpload.idl) $(location XPath/XPathEvaluator.idl) $(location XPath/XPathExpression.idl) $(location XPath/XPathNSResolver.idl) $(location XPath/XPathResult.idl)', ) diff --git a/examples/ladybird/workspace/MODULE.bazel b/examples/ladybird/workspace/MODULE.bazel index fdf6993..fbbc8a9 100644 --- a/examples/ladybird/workspace/MODULE.bazel +++ b/examples/ladybird/workspace/MODULE.bazel @@ -231,6 +231,7 @@ use_repo( 'crate_fastrand_2_4_0', 'crate_flatbuffers_25_12_19', 'crate_foldhash_0_1_5', + 'crate_foldhash_0_2_0', 'crate_form_urlencoded_1_2_2', 'crate_getrandom_0_4_2', 'crate_gimli_0_31_1', diff --git a/examples/ladybird/workspace/Meta/bazel_parity_harness.py b/examples/ladybird/workspace/Meta/bazel_parity_harness.py index 828df1e..b92927a 100644 --- a/examples/ladybird/workspace/Meta/bazel_parity_harness.py +++ b/examples/ladybird/workspace/Meta/bazel_parity_harness.py @@ -32,7 +32,13 @@ import os, re, subprocess, sys, shutil, filecmp ROOT = os.environ.get("LADYBIRD_ROOT", os.path.dirname(os.path.dirname(os.path.abspath(__file__)))) -FULL = os.path.join(ROOT, "Build/full") +# The reference CMake build tree the emitter reads. An env var, not a +# hardcoded "Build/full": a repin needs a SECOND reference build side by side +# with the old one (you cannot delete the tree you are still diffing against), +# and CMake bakes the build dir's absolute path into build.ninja -- so +# "just rename the directory afterwards" corrupts it. Found by doing exactly +# that during the 71fb301a repin. +FULL = os.environ.get("LADYBIRD_BUILD_DIR") or (ROOT + "/Build/full") SCRATCH = os.environ.get("PARITY_SCRATCH", os.path.join(ROOT, "Build/parity_out")) # --------------------------------------------------------------------------- @@ -56,6 +62,20 @@ (r'(?:^|&& )\S*bin/generate_interpreter_layout\b', 'generate_interpreter_layout (self-built)'), (r'(?:^|&& )\S*bin/flapc\b', 'flapc (self-built)'), + # generate-libjs-bytecode: upstream a32d9c9f replaced the Python bytecode + # generator with a Rust BINARY (a second bin of the flapc crate) that emits + # Bytecode/Op.h + OpCodes.h from interpreter.flap. + # + # It has to be listed HERE, above EXCLUDED, and that is the whole lesson: its + # ninja command is ` --input ... && cmake -E copy_if_different ...`, so + # the `copy_if_different` exclusion matched it first and the harness bucketed a + # GENERATOR as "resource staging" -- reporting 0 UNHANDLED while two generated + # headers had no owner at all. A first-match-wins classifier makes every + # exclusion pattern a potential silent capture of a command it was never + # written for, and the count that is supposed to catch that cannot, because the + # command was counted. + (r'(?:^|&& )\S*bin/generate-libjs-bytecode\b', + 'generate-libjs-bytecode (self-built)'), ] # --------------------------------------------------------------------------- diff --git a/examples/ladybird/workspace/Meta/emit_build_bazel.py b/examples/ladybird/workspace/Meta/emit_build_bazel.py index afbbaff..2a2c8b2 100644 --- a/examples/ladybird/workspace/Meta/emit_build_bazel.py +++ b/examples/ladybird/workspace/Meta/emit_build_bazel.py @@ -14,7 +14,14 @@ from collections import defaultdict ROOT = os.environ.get("LADYBIRD_ROOT", os.path.dirname(os.path.dirname(os.path.abspath(__file__)))) -MODEL = os.path.join(ROOT, "model.cmake.full.json") +MODEL = os.environ.get("LADYBIRD_MODEL") or os.path.join(ROOT, "model.cmake.full.json") +# The reference build tree, RELATIVE to the checkout: the model records source +# and include paths relative to the repo root, so every "is this a generated +# file?" test is a string prefix on this. An env var rather than a literal +# "Build/full" because a repin needs the new reference build to coexist with the +# old one, and CMake bakes the build dir into build.ninja -- so renaming the +# directory after the fact is not an option (learned the hard way at 71fb301a). +BUILD_REL = (os.environ.get("LADYBIRD_BUILD_REL") or "Build/full").strip("/") + "/" VCPKG = "//Meta/vcpkg" # Ring 2 part 3: the Rust crates are BUILT BY BAZEL now (cargo_ring.bzl in this # same package), so the labels are local -- nothing points into Build/full/cargo. @@ -246,9 +253,9 @@ def target_private_includes(t): emitted build look like it needed Build/full when it did not. """ globalroots = {ROOT, ROOT+"/Libraries", ROOT+"/Services", - ROOT+"/Build/full", ROOT+"/Build/full/Libraries", - ROOT+"/Build/full/Services", - ROOT+"/Build/full/vcpkg_installed/x64-linux-dynamic/include"} + ROOT+"/"+BUILD_REL.rstrip("/"), ROOT+"/"+BUILD_REL+"Libraries", + ROOT+"/"+BUILD_REL+"Services", + ROOT+"/"+BUILD_REL+"vcpkg_installed/x64-linux-dynamic/include"} incs = [] for a in t["actions"]: if a["mnemonic"] != "CppCompile": continue @@ -405,21 +412,21 @@ def bazel_generated_root_srcs(): def _emit_srcs(srcs): print(" srcs = [") for s in srcs: - rel = s[len("Build/full/"):] if s.startswith("Build/full/") else None + rel = s[len(BUILD_REL):] if s.startswith(BUILD_REL) else None if rel is not None and rel in GENERATED_BY_BAZEL: # Bazel generates this file; consume its genrule output. print(f" {':' + rel!r},") - elif s.startswith("Build/full/Libraries/LibWeb/"): - lab = "//Libraries/LibWeb:" + s[len("Build/full/Libraries/LibWeb/"):] + elif s.startswith(BUILD_REL + "Libraries/LibWeb/"): + lab = "//Libraries/LibWeb:" + s[len(BUILD_REL + "Libraries/LibWeb/"):] print(f" {lab!r},") - elif s.startswith("Build/full/Libraries/"): - lab = "//Build/full/Libraries:" + s[len("Build/full/Libraries/"):] + elif s.startswith(BUILD_REL + "Libraries/"): + lab = "//Build/full/Libraries:" + s[len(BUILD_REL + "Libraries/"):] print(f" {lab!r},") - elif s.startswith("Build/full/Services/"): - lab = "//Build/full/Services:" + s[len("Build/full/Services/"):] + elif s.startswith(BUILD_REL + "Services/"): + lab = "//Build/full/Services:" + s[len(BUILD_REL + "Services/"):] print(f" {lab!r},") - elif s.startswith("Build/full/UI/"): - lab = "//Build/full/UI:" + s[len("Build/full/UI/"):] + elif s.startswith(BUILD_REL + "UI/"): + lab = "//Build/full/UI:" + s[len(BUILD_REL + "UI/"):] print(f" {lab!r},") else: print(f" {s!r},") @@ -791,7 +798,7 @@ def emit_target(name, targets, libs, exes, so, ar, header=True, body_only=False, # with -isystem. They live as global -isystem in .bazelrc # (mirroring CMake's find_package include dirs). continue - if i.startswith("Build/full/Libraries/"): + if i.startswith(BUILD_REL + "Libraries/"): # CMake's per-library FFI dir (FFI_OUTPUT_DIR defaults to the # library's own binary dir), which is where its crate's # RustFFI.h lands. Pointing this at the CMake tree is what kept @@ -809,7 +816,7 @@ def emit_target(name, targets, libs, exes, so, ar, header=True, body_only=False, # without depending on skia now fails to compile, # which is the whole point of declaring inputs. pass - elif i in ("Build/full/UI", "Build/full/UI/Qt"): + elif i in (BUILD_REL.rstrip("/") + "/UI", BUILD_REL.rstrip("/") + "/UI/Qt"): # CMake's UI gendir, whose only non-autogen contents are the two # SPIR-V shader headers (WebContentViewLinux{Frag,Vert}Shader.h). # Bazel generates both itself and carries them on diff --git a/examples/ladybird/workspace/Meta/emit_cargo_bazel.py b/examples/ladybird/workspace/Meta/emit_cargo_bazel.py index 2f2547b..c5c8298 100644 --- a/examples/ladybird/workspace/Meta/emit_cargo_bazel.py +++ b/examples/ladybird/workspace/Meta/emit_cargo_bazel.py @@ -51,8 +51,8 @@ enumerate; Bazel deletes undeclared outputs. So the header list is the union of CMake's declaration and the observed set, and the observed set wins where they differ (see FFI_HEADERS_OBSERVED). - * **The registry/workspace split is the whole reason this is cheap.** 154 of - the 167 `[[package]]` entries have a checksum; the other 13 are the in-tree + * **The registry/workspace split is the whole reason this is cheap.** 155 of + the 166 `[[package]]` entries have a checksum; the other 11 are the in-tree workspace members, which have no checksum because they are *sources*. A crate with no checksum must never become a fetch rule (it would 404), and a registry crate that silently drops out is a build failure a long way from its @@ -215,10 +215,18 @@ def repo_name(name, version): "libtextcodec_rust": ["RustFFI.h"], "liburl_rust": ["RustFFI.h"], "libunicode_rust": ["RustFFI.h"], - "libweb_rust": ["HTML/Parser/RustFFI.h", "HTMLTokenizerRustFFI.h"], - "libweb_layout_rust": ["Layout/TreeBuilderRustFFI.h"], - "libweb_css_rust": ["ComputedValuesRustFFI.h", "RustFFI.h", - "SelectorRustFFI.h", "StyleValueRustFFI.h"], + # Upstream consolidated libweb_css_rust and libweb_layout_rust BACK into + # libweb_rust, so this one crate now writes what three used to -- eleven + # files of its own plus HTMLTokenizerRustFFI.h from its path-dependency + # libweb_html_tokenizer, which lands in the same FFI_OUTPUT_DIR. Two of them + # are .inc, not .h: an undeclared .inc is deleted by Bazel exactly like an + # undeclared header, and it is #included the same way. + "libweb_rust": ["ComputedValuesRustFFI.h", "HTML/Parser/RustFFI.h", + "HTMLTokenizerRustFFI.h", "Layout/LayoutRustFFI.h", + "Layout/TreeBuilderRustFFI.h", "RustFFI.h", + "SelectorRustFFI.h", "StyleEngineBridgeGenerated.h", + "StyleEngineBridgeGenerated.inc", "StyleEngineRustFFI.h", + "StyleEngineStateFactsGenerated.inc", "StyleValueRustFFI.h"], "libweb_content_blocker_rust": ["ContentBlockerRustFFI.h"], # A BINARY crate that also emits a header -- see BINARY_CMAKELISTS below. "libwasm_cranelift": ["CraneliftFFI.h"], @@ -237,8 +245,6 @@ def repo_name(name, version): "liburl_rust": "LibURL", "libunicode_rust": "LibUnicode", "libweb_rust": "LibWeb", - "libweb_layout_rust": "LibWeb", - "libweb_css_rust": "LibWeb", "libweb_content_blocker_rust": "LibWeb", "libwasm_cranelift": "LibWasm", } @@ -252,68 +258,223 @@ def repo_name(name, version): # found most of these; the depfile finds all of them, including the ones reached # through a `manifest_dir.parent()` join. CRATE_EXTRA_INPUTS = { - "libjs_rust": ["Libraries/LibJS/Bytecode/Bytecode.def"], + # libjs_rust's build script derives its instruction types from the Flap + # interpreter (upstream a32d9c9f: `include_str!("../../Interpreter/ + # interpreter.flap")` via flapc::metadata), so the .flap file is a compile + # input reached by a `manifest_dir.parent()` join -- exactly the shape the + # depfile finds and reading build.rs nearly misses. It replaced + # Libraries/LibJS/Bytecode/Bytecode.def, which that commit DELETED. + "libjs_rust": ["Libraries/LibJS/Interpreter/interpreter.flap"], # libwasm_cranelift's build.rs reads `manifest_dir.join("../Opcode.h")` and # generates a Rust constant per `M(name, value, ...)` line, so the WASM - # opcode table is a compile input to the cranelift compiler binary. Exactly - # the `manifest_dir.parent()` join the depfile finds and reading build.rs - # nearly misses. + # opcode table is a compile input to the cranelift compiler binary. "libwasm_cranelift": ["Libraries/LibWasm/Opcode.h"], - # The LibWeb ones (8 CSS data files for libweb_css_rust, TagNames.h + - # AttributeNames.h + Entities.json for libweb_rust) are NOT here: they are - # inside the Libraries/LibWeb package, so they ride in that package's - # rust_crate_srcs filegroup instead. Same measurement, different side of a - # package boundary. + # The LibWeb ones are NOT here: they are inside the Libraries/LibWeb package, + # so they ride in that package's rust_crate_srcs filegroup instead -- + # PACKAGE_EXTRA_INPUTS below. Same measurement, different side of a package + # boundary. +} + +# The same measurement for the files that live inside ANOTHER Bazel package, so +# the root package cannot glob them: libweb_rust's build script generates Rust +# from the CSS data files and reads the HTML name headers + Entities.json. Kept +# here rather than in emit_libweb_bazel.py so ONE place holds the measurement and +# ONE --report checks it against the reference build's depfiles. +PACKAGE_EXTRA_INPUTS = { + "Libraries/LibWeb": [ + "CSS/Enums.json", + "CSS/Keywords.json", + "CSS/LogicalPropertyGroups.json", + "CSS/Properties.json", + "CSS/PseudoClasses.json", + "CSS/PseudoElementPropertyGroups.txt", + "CSS/PseudoElements.json", + "CSS/TransformFunctions.json", + "CSS/Units.json", + "HTML/AttributeNames.h", + "HTML/Parser/Entities.json", + "HTML/TagNames.h", + ], } # The crate source trees. A crate depends on more than its own directory: every # one of them `#[path = "../../../RustAllocator.rs"]`s the shared allocator shim, # and several are path-dependencies of each other (libweb_rust -> -# libweb_html_tokenizer, liburl_rust -> libregex_rust), which cargo resolves -# through the workspace. So the source set is "all the in-tree crates plus the -# shim", declared once. Over-declaring costs a rebuild of 11 crates when any .rs -# changes; under-declaring silently reuses a stale archive. For a first landing -# that trade is the right way round, and it is honest debt to name. +# libweb_html_tokenizer, liburl_rust -> libregex_rust, libjs_rust -> flapc), +# which cargo resolves through the workspace. So the source set is "all the +# in-tree crates plus the shim", declared once. Over-declaring costs a rebuild of +# every crate when any .rs changes; under-declaring silently reuses a stale +# archive. For a first landing that trade is the right way round, and it is +# honest debt to name (todo 6f39dd58). +# +# DERIVED from Cargo.toml's member list and the manifests' `path =` +# dependencies, never written down -- because a directory written down here is a +# glob that outlives the directory. Upstream a32d9c9f deleted +# Libraries/LibJS/BytecodeDef/, which was a hardcoded `allow_empty = False` glob +# pattern in this list, and the whole build then failed at LOADING time with +# "glob pattern didn't match anything" -- before any target could report why. +# Deriving the patterns means a deleted crate deletes its own pattern. # -# Split in two because BAZEL PACKAGES CUT ACROSS THE CARGO WORKSPACE: five of the +# Split in two because BAZEL PACKAGES CUT ACROSS THE CARGO WORKSPACE: some of the # crates live under Libraries/LibWeb, which is its own package, and glob() is # package-relative -- the root package cannot see them at all. So they arrive as # a filegroup label that LibWeb's own BUILD file defines -# (//Libraries/LibWeb:rust_crate_srcs, emitted by emit_libweb_bazel.py). This is -# the first place in this migration where the two systems' unit of grouping -# genuinely disagree: cargo says "one workspace", Bazel says "two packages", and -# a label is the only thing that crosses. -CRATE_SRC_GLOBS = [ +# (//Libraries/LibWeb:rust_crate_srcs, emitted by emit_libweb_bazel.py from +# package_crate_globs() HERE, so the two sides cannot disagree about which +# directories exist). This is the first place in this migration where the two +# systems' unit of grouping genuinely disagree: cargo says "one workspace", Bazel +# says "two packages", and a label is the only thing that crosses. +FOREIGN_PACKAGES = {"Libraries/LibWeb": "//Libraries/LibWeb:rust_crate_srcs"} + +# Files at the repo root that every crate needs: the two workspace files, the +# toolchain pin and the shared allocator shim. Not a directory between them, so +# nothing here can outlive a crate. +CRATE_SRC_ROOT_FILES = [ "Cargo.toml", "Cargo.lock", "rust-toolchain.toml", "Libraries/RustAllocator.rs", - "Libraries/*/Rust/**", - "Libraries/LibJS/BytecodeDef/**", ] -CRATE_SRC_LABELS = ["//Libraries/LibWeb:rust_crate_srcs"] - -# flapc's own workspace: excluded from the root one, its own lock, 3 packages. -FLAPC_SRC_GLOBS = [ - "Libraries/LibJS/Flap/Cargo.toml", - "Libraries/LibJS/Flap/Cargo.lock", - "Libraries/LibJS/Flap/src/**", - # benches/ and tests/ are not built by `cargo rustc --bin flapc`, but they - # ARE manifest inputs: Cargo.toml declares `[[bench]] name = "compiler"`, and - # cargo validates every declared target while PARSING the manifest, before it - # decides what to build ("can't find `compiler` bench at - # benches/compiler.rs"). The bench also include_str!s - # tests/interpreter-layout.conf. So the manifest's target list is part of the - # input set whether or not those targets are compiled -- a distinction CMake - # never makes, and one only a sandbox reveals. - "Libraries/LibJS/Flap/benches/**", - "Libraries/LibJS/Flap/tests/**", - "Libraries/LibJS/BytecodeDef/**", - "Libraries/LibJS/Bytecode/Bytecode.def", - "Libraries/LibJS/Interpreter/interpreter.flap", - "rust-toolchain.toml", -] +# `target/` is cargo's own output dir: never an input, and a developer who has run +# cargo by hand in the tree would otherwise have it globbed into every crate's +# action inputs (the reference build sets CARGO_TARGET_DIR into the build dir, so +# it does not exist here -- but that is a property of how we invoke cargo, not of +# the tree). +# +# ANCHORED to each crate root, NOT `**/target/**`. The unanchored form is wrong +# and fails loudly, which is the only good thing about it: flapc has a Rust MODULE +# at `src/target/` (the code generator's per-architecture backends), so +# `**/target/**` dropped 40 source files and the build died with +# `error[E0583]: file not found for module 'target'`. A glob exclusion is a +# pattern over paths and knows nothing about what a directory MEANS; "the dir +# cargo writes to" is `/target`, and only the anchored form says that. +def crate_src_glob_excludes(dirs): + return [d + "/target/**" for d in dirs] + + +def _manifest_path_deps(manifest_rel): + """The `path = "..."` dependency dirs declared by one manifest, ROOT-relative. + + Read from the manifest rather than from the workspace member list, because a + path dependency can point OUTSIDE the workspace: libjs_rust build-depends on + `flapc = { path = "../Flap" }`, and Libraries/LibJS/Flap is `exclude`d from + the root workspace. Miss it and touching a flapc source does not rebuild the + crate whose instruction types are derived from it. + """ + path = os.path.join(ROOT, manifest_rel) + if not os.path.exists(path): + return [] + d = os.path.dirname(manifest_rel) + out = [] + for m in re.finditer(r'path\s*=\s*"([^"]+)"', open(path).read()): + p = os.path.normpath(os.path.join(d, m.group(1))) + if os.path.isdir(os.path.join(ROOT, p)): + out.append(p) + return out + + +def crate_dirs(): + """Every in-tree crate DIRECTORY the root workspace resolves, ROOT-relative. + + Starts from Cargo.toml's `members` and closes over `path =` dependencies, so + a crate that is added, moved or deleted upstream needs no edit here. + """ + seen = set() + queue = sorted(root_workspace_members()) + while queue: + d = queue.pop() + if d in seen or not os.path.isdir(os.path.join(ROOT, d)): + continue + seen.add(d) + queue += _manifest_path_deps(os.path.join(d, "Cargo.toml")) + return sorted(seen) + + +def _in_foreign_package(d): + for pkg in FOREIGN_PACKAGES: + if d == pkg or d.startswith(pkg + "/"): + return pkg + return None + + +def crate_src_globs(): + """The ROOT package's glob patterns for the shared crate source set.""" + return CRATE_SRC_ROOT_FILES + [ + d + "/**" for d in crate_dirs() if not _in_foreign_package(d)] + + +def crate_src_labels(): + """One filegroup label per foreign package that actually holds a crate.""" + return sorted({FOREIGN_PACKAGES[p] for p in + (_in_foreign_package(d) for d in crate_dirs()) if p}) + + +def package_crate_globs(pkg): + """The crate globs for a foreign package, PACKAGE-relative. + + Called by emit_libweb_bazel.py, so the filegroup on that side of the boundary + and the glob on this side are derived from one list. + """ + return sorted(os.path.relpath(d, pkg) + "/**" + for d in crate_dirs() if _in_foreign_package(d) == pkg) + + +def _excluded_dirs(): + """The workspaces Cargo.toml `exclude`s, which resolve their own locks.""" + ws = _toml(os.path.join(ROOT, "Cargo.toml"))["workspace"] + return [os.path.normpath(d) for d in ws.get("exclude", []) + if os.path.isdir(os.path.join(ROOT, d))] + + +def flapc_src_globs(): + """The `exclude`d flapc workspace: its own subtree, plus what it reads out of + the tree above it. + + One `/**` rather than a pattern per subdirectory, because the manifest's + TARGET list is part of the input set whether or not those targets are built: + Cargo.toml declares `[[bench]] name = "compiler"`, and cargo validates every + declared target while PARSING the manifest, before it decides what to build + ("can't find `compiler` bench at benches/compiler.rs"). That bench also + include_str!s tests/interpreter-layout.conf. A distinction CMake never makes, + and one only a sandbox reveals -- and a whole-subtree glob cannot lose a + subdirectory the way an enumerated list of them can. + """ + dirs = _excluded_dirs() + extra = ["rust-toolchain.toml"] + for d in dirs: + # src/metadata.rs include_str!s the interpreter definition from two + # directories up: flapc DERIVES the bytecode from it (upstream a32d9c9f), + # so it is an input, and it lives outside the excluded subtree. + extra += [x for x in _include_str_paths(d) if not x.startswith(d + "/")] + return sorted(d + "/**" for d in dirs) + sorted(set(extra)) + + +def _include_str_paths(d): + """ROOT-relative paths a crate subtree include_str!s, derived by scanning it. + + Derived rather than listed for the same reason as everything else here: the + one file flapc reads from outside its own subtree changed name and location + in the commit this repin follows. + """ + out = set() + pat = re.compile(r'include_str!\s*\(\s*"([^"]+)"') + for dirpath, dirnames, filenames in os.walk(os.path.join(ROOT, d)): + dirnames[:] = [x for x in dirnames if x != "target"] + for fn in filenames: + if not fn.endswith(".rs"): + continue + full = os.path.join(dirpath, fn) + try: + text = open(full, errors="replace").read() + except OSError: + continue + for m in pat.finditer(text): + p = os.path.normpath(os.path.join( + os.path.relpath(dirpath, ROOT), m.group(1))) + if os.path.exists(os.path.join(ROOT, p)): + out.add(p) + return sorted(out) # Where each crate's Cargo.toml lives, relative to the source root. Parsed from # the import_rust_crate() call site: MANIFEST_PATH is relative to the CMakeLists @@ -389,7 +550,10 @@ def parse_cmake_crates(): # var -> features, for the `if (NOT BUILD_SHARED_LIBS)` idiom above. vars_ = {} for m in re.finditer( - r"set\(\s*(\w+)\s+FEATURES\s+([\w\s]+?)\s*\)", text): + # `-` is legal in a cargo feature name (`style-recording`), so it + # has to be in the character class -- a class that excludes it + # truncates the feature rather than failing to match it. + r"set\(\s*(\w+)\s+FEATURES\s+([\w\s-]+?)\s*\)", text): vars_[m.group(1)] = m.group(2).split() for block in RE_IMPORT.findall(text): crate = _arg(block, "CRATE_NAME") @@ -438,6 +602,13 @@ def parse_cmake_binaries(): (`#include `), because CMake's FFI_OUTPUT_DIR is LibWasm's own binary dir. + Everything else it shares with import_rust_crate, FEATURES included, and that + is parsed here for the same reason: upstream's `style-replay` binary is built + `FEATURES style-recording` out of the same crate as libweb_rust's staticlib. + A parser that only read FEATURES on the staticlib side would build a + different binary than CMake does and say nothing -- the finding-23 shape, + where the two build systems agree on the target list and disagree on the ABI. + So a binary crate needs both halves: a cargo_binary for the executable, and the same declared-header treatment the staticlib crates get. The two are reported separately for the same reason import_rust_crate's list is parsed @@ -452,6 +623,12 @@ def parse_cmake_binaries(): with open(path) as f: text = f.read() pkgdir = os.path.dirname(rel) + # The same `if (NOT BUILD_SHARED_LIBS)` variable form the staticlib side + # resolves -- see parse_cmake_crates() for why that branch is the one the + # Bazel overlay mirrors. + vars_ = {} + for m in re.finditer(r"set\(\s*(\w+)\s+FEATURES\s+([\w\s-]+?)\s*\)", text): + vars_[m.group(1)] = m.group(2).split() for block in RE_BINARY.findall(text): crate = _arg(block, "CRATE_NAME") manifest = _arg(block, "MANIFEST_PATH") @@ -464,11 +641,15 @@ def parse_cmake_binaries(): # build_rust_binary has no FFI_HEADERS argument at all, so CMake # declares nothing and there is nothing to compare against. emits_ffi = _arg(block, "FFI_OUTPUT_DIR") is not None + features = _multi(block, "FEATURES", _KEYS) + for m in re.finditer(r"\$\{(\w+)\}", block): + features += vars_.get(m.group(1), []) out.append({ "crate": crate, "bin": binary, "manifest": os.path.normpath(os.path.join(pkgdir, manifest)), "output_name": _arg(block, "OUTPUT_NAME") or binary, + "features": sorted(set(features)), "emits_ffi": emits_ffi, }) return sorted(out, key=lambda c: c["crate"]) @@ -735,12 +916,19 @@ def emit_ring(crates, specs, binaries): print(" # cargo WORKSPACE. Each of them path-depends on") print(" # Libraries/RustAllocator.rs and several are path-dependencies of each") print(" # other, so cargo resolves the whole workspace whichever crate you ask") - print(" # for. Over-declaring costs a rebuild of 11 crates when any .rs changes;") - print(" # under-declaring silently reuses a stale archive. The trade is the right") - print(" # way round, and it is real debt: per-crate source sets need the") - print(" # path-dependency graph read out of the manifests.") - print(" crate_srcs = native.glob(%r, allow_empty = False) + %r" - % (CRATE_SRC_GLOBS, CRATE_SRC_LABELS)) + print(" # for. Over-declaring costs a rebuild of every crate when any .rs") + print(" # changes; under-declaring silently reuses a stale archive. The trade is") + print(" # the right way round, and it is real debt: per-crate source sets need") + print(" # the path-dependency graph read out of the manifests.") + print(" #") + print(" # The patterns are DERIVED from Cargo.toml's member list closed over the") + print(" # manifests' `path =` deps, not written down: a directory written down") + print(" # here is an allow_empty=False glob that outlives the directory, and") + print(" # loading then fails before any target can say why.") + root_dirs = [d for d in crate_dirs() if not _in_foreign_package(d)] + print(" crate_srcs = native.glob(%r, exclude = %r, allow_empty = False) + %r" + % (crate_src_globs(), crate_src_glob_excludes(root_dirs), + crate_src_labels())) for s in specs: extra = CRATE_EXTRA_INPUTS.get(s["crate"], []) srcs = "crate_srcs" @@ -815,6 +1003,11 @@ def emit_ring(crates, specs, binaries): print(" crate = %r," % b["crate"]) print(" crates = CARGO_CRATE_FILES,") print(" manifest = %r," % b["manifest"]) + if b["features"]: + print(" # build_rust_binary() takes FEATURES too, and this one does:") + print(" # the same crate builds a staticlib and this binary, and only") + print(" # the features distinguish what each gets.") + print(" crate_features = %r," % b["features"]) if b["ffi_headers"]: print(" # This binary crate's build script runs cbindgen too, so the") print(" # header is a DECLARED output -- Bazel deletes what nothing") @@ -833,8 +1026,11 @@ def emit_ring(crates, specs, binaries): print(" srcs = %s," % srcs) else: print(" # Its OWN workspace (`exclude`d from the root one), so its source") - print(" # set is its own subtree.") - print(" srcs = native.glob(%r, allow_empty = False)," % FLAPC_SRC_GLOBS) + print(" # set is its own subtree plus what it include_str!s from above") + print(" # it -- both derived, the subtree from Cargo.toml's `exclude`") + print(" # and the rest by scanning for include_str!.") + print(" srcs = native.glob(%r, exclude = %r, allow_empty = False)," + % (flapc_src_globs(), crate_src_glob_excludes(_excluded_dirs()))) print(" sysroot = \":rust_sysroot\",") print(" )") if b["ffi_headers"]: diff --git a/examples/ladybird/workspace/Meta/emit_codegen_bazel.py b/examples/ladybird/workspace/Meta/emit_codegen_bazel.py index fdc5efe..8e2f6da 100644 --- a/examples/ladybird/workspace/Meta/emit_codegen_bazel.py +++ b/examples/ladybird/workspace/Meta/emit_codegen_bazel.py @@ -11,7 +11,13 @@ import re, os, sys, shlex ROOT = os.environ.get("LADYBIRD_ROOT", os.path.dirname(os.path.dirname(os.path.abspath(__file__)))) -FULL = ROOT + "/Build/full" +# The reference CMake build tree the emitter reads. An env var, not a +# hardcoded "Build/full": a repin needs a SECOND reference build side by side +# with the old one (you cannot delete the tree you are still diffing against), +# and CMake bakes the build dir's absolute path into build.ninja -- so +# "just rename the directory afterwards" corrupts it. Found by doing exactly +# that during the 71fb301a repin. +FULL = os.environ.get("LADYBIRD_BUILD_DIR") or (ROOT + "/Build/full") def parse(lib_build_dir): txt = open(FULL + "/build.ninja").read() diff --git a/examples/ladybird/workspace/Meta/emit_libweb_bazel.py b/examples/ladybird/workspace/Meta/emit_libweb_bazel.py index 335e32f..1e82976 100644 --- a/examples/ladybird/workspace/Meta/emit_libweb_bazel.py +++ b/examples/ladybird/workspace/Meta/emit_libweb_bazel.py @@ -16,9 +16,12 @@ import json, os, re, sys ROOT = os.environ.get("LADYBIRD_ROOT", os.path.dirname(os.path.dirname(os.path.abspath(__file__)))) -MODEL = os.path.join(ROOT, "model.cmake.full.json") +MODEL = os.environ.get("LADYBIRD_MODEL") or os.path.join(ROOT, "model.cmake.full.json") PKG_PREFIX = "Libraries/LibWeb/" -GEN_PREFIX = "Build/full/Libraries/LibWeb/" +# see emit_build_bazel.BUILD_REL: the reference build dir is an env var, because a +# repin needs two reference trees side by side and build.ninja cannot be moved. +BUILD_REL = (os.environ.get("LADYBIRD_BUILD_REL") or "Build/full").strip("/") + "/" +GEN_PREFIX = BUILD_REL + "Libraries/LibWeb/" VCPKG = "//Meta/vcpkg" # See emit_build_bazel.py: the crates are Bazel-built now, and each crate is ONE # target (//:_lib) carrying its own archive and its own generated FFI @@ -30,6 +33,14 @@ # because "none today" is a measurement, not an invariant). sys.path.insert(0, os.path.dirname(os.path.abspath(__file__))) from emit_build_bazel import rust_dep_labels +# The crate directories inside THIS package, and the non-Rust build-script inputs +# that live here, both derived by emit_cargo_bazel from Cargo.toml and the +# reference build's depfiles. Imported rather than restated for one concrete +# reason: this filegroup used to hardcode `CSS/Rust/**` and `Layout/Rust/**` with +# allow_empty = False, upstream consolidated both crates into LibWeb/Rust, and the +# glob then matched nothing -- which is a LOADING-time failure, so no target could +# report it. The root package's glob over the same crates is one list; so is this. +import emit_cargo_bazel GLOBAL_DEFINES = { "USE_VULKAN=1", "ENABLE_COMPILETIME_FORMAT_CHECK", "USE_FONTCONFIG=1", @@ -45,7 +56,20 @@ # emit_build_bazel.py's ALWAYSLINK_LIBS. ALWAYSLINK = True -PRELUDE = '''load("@rules_cc//cc:defs.bzl", "cc_library") + +def prelude(): + """The head of the BUILD file: loads, package(), codegen macros, filegroup. + + A function rather than a constant because the rust_crate_srcs glob is + DERIVED (see the emit_cargo_bazel import above), and a constant is exactly + what let it name two directories that upstream had deleted. + """ + pkg = PKG_PREFIX.rstrip("/") + globs = emit_cargo_bazel.package_crate_globs(pkg) + excludes = emit_cargo_bazel.crate_src_glob_excludes( + [g[:-len("/**")] for g in globs]) + extra = emit_cargo_bazel.PACKAGE_EXTRA_INPUTS.get(pkg, []) + return '''load("@rules_cc//cc:defs.bzl", "cc_library") load(":codegen.bzl", "libweb_codegen", "libweb_bindings_codegen") load(":generated_srcs.bzl", "LIBWEB_GENERATED_SRCS", "LIBWEB_GENERATED_HDRS") load(":export_header.bzl", "libweb_export_header") @@ -63,41 +87,32 @@ libweb_codegen() libweb_bindings_codegen() -# The four Rust crates that live INSIDE this package (LibWeb/Rust, -# LibWeb/CSS/Rust, LibWeb/Layout/Rust, LibWeb/ContentBlocker/Rust, plus -# HTML/Parser/Rust), exposed so the root package's cargo_ring() can declare them -# as cargo inputs. They are one cargo WORKSPACE with the crates at the repo root, -# but Bazel packages cut across it: glob() is package-relative, so the root -# package cannot see files under Libraries/LibWeb/ at all. Hence a filegroup on -# this side of the boundary rather than a glob on that side -- the alternative -# (making the root package own these files) would mean deleting this package. +# The Rust crates that live INSIDE this package, exposed so the root package's +# cargo_ring() can declare them as cargo inputs. They are one cargo WORKSPACE +# with the crates at the repo root, but Bazel packages cut across it: glob() is +# package-relative, so the root package cannot see files under Libraries/LibWeb/ +# at all. Hence a filegroup on this side of the boundary rather than a glob on +# that side -- the alternative (making the root package own these files) would +# mean deleting this package. +# +# The patterns are DERIVED from Cargo.toml (Meta/emit_cargo_bazel.py's +# crate_dirs(), the same list the root package globs), not written down. They +# used to be written down, and that broke the whole build: upstream consolidated +# libweb_css_rust and libweb_layout_rust into libweb_rust, so `CSS/Rust/**` and +# `Layout/Rust/**` matched nothing, and an allow_empty = False glob that matches +# nothing fails at LOADING time -- before any target exists to blame. filegroup( name = "rust_crate_srcs", - srcs = glob([ - "Rust/**", - "CSS/Rust/**", - "Layout/Rust/**", - "ContentBlocker/Rust/**", - "HTML/Parser/Rust/**", - ], allow_empty = False) + [ - # Non-Rust build-script inputs that live here too: libweb_css_rust's - # build.rs GENERATES Rust from these CSS data files, and libweb_rust's - # reads the HTML name headers + Entities.json. Taken from the reference - # build's cargo depfiles, so the list is measured rather than predicted. - "CSS/Enums.json", - "CSS/Keywords.json", - "CSS/LogicalPropertyGroups.json", - "CSS/Properties.json", - "CSS/PseudoClasses.json", - "CSS/PseudoElementPropertyGroups.txt", - "CSS/PseudoElements.json", - "CSS/Units.json", - "HTML/AttributeNames.h", - "HTML/Parser/Entities.json", - "HTML/TagNames.h", + srcs = glob(%r, exclude = %r, allow_empty = False) + [ + # Non-Rust build-script inputs that live here too: libweb_rust's build.rs + # GENERATES Rust from the CSS data files and reads the HTML name headers + + # Entities.json. Taken from the reference build's cargo depfiles, so the + # list is measured rather than predicted. +%s ], ) -''' +''' % (globs, excludes, + "\n".join(" %r," % x for x in extra)) def global_flags(): @@ -160,9 +175,9 @@ def target_flags(t): def target_private_includes(t): globalroots = {ROOT, ROOT + "/Libraries", ROOT + "/Services", - ROOT + "/Build/full", ROOT + "/Build/full/Libraries", - ROOT + "/Build/full/Services", - ROOT + "/Build/full/vcpkg_installed/x64-linux-dynamic/include"} + ROOT + "/" + BUILD_REL.rstrip("/"), ROOT + "/" + BUILD_REL + "Libraries", + ROOT + "/" + BUILD_REL + "Services", + ROOT + "/" + BUILD_REL + "vcpkg_installed/x64-linux-dynamic/include"} incs = [] for a in t["actions"]: if a["mnemonic"] != "CppCompile": @@ -260,7 +275,7 @@ def main(): copt_toks.append("-I" + i) deps.append("//:all_source_headers") - out = [PRELUDE] + out = [prelude()] out.append(f"# === LibWeb ({t['kind']}, {len(all_srcs)} TU: " f"{len(checked_in)} checked-in + {len(all_srcs)-len(checked_in)} generated) ===") if unknown: diff --git a/examples/ladybird/workspace/Meta/emit_root_codegen_bazel.py b/examples/ladybird/workspace/Meta/emit_root_codegen_bazel.py index d89b597..c4cab78 100644 --- a/examples/ladybird/workspace/Meta/emit_root_codegen_bazel.py +++ b/examples/ladybird/workspace/Meta/emit_root_codegen_bazel.py @@ -46,7 +46,13 @@ import re, os, sys, shlex ROOT = os.environ.get("LADYBIRD_ROOT", os.path.dirname(os.path.dirname(os.path.abspath(__file__)))) -FULL = ROOT + "/Build/full" +# The reference CMake build tree the emitter reads. An env var, not a +# hardcoded "Build/full": a repin needs a SECOND reference build side by side +# with the old one (you cannot delete the tree you are still diffing against), +# and CMake bakes the build dir's absolute path into build.ninja -- so +# "just rename the directory afterwards" corrupts it. Found by doing exactly +# that during the 71fb301a repin. +FULL = os.environ.get("LADYBIRD_BUILD_DIR") or (ROOT + "/Build/full") LIBWEB_CD = FULL + "/Libraries/LibWeb" VCPKG_PKG = "//:vcpkg_installed_exec" @@ -118,6 +124,75 @@ def parse_glslang(): return sorted(out, key=lambda d: d['out']) +def parse_bytecode(): + """upstream a32d9c9f's generate-libjs-bytecode: interpreter.flap -> Op.h + OpCodes.h. + + A THIRD self-built tool, and the one that broke Ulf's build. It replaced + Meta/Generators/generate_libjs_bytecode_def_derived.py (a Python generator this + emitter picked up for free, via the Meta/Generators/ pattern) with a second + binary of the flapc crate -- so the generator moved out of every bucket the + emitter knew about, and gen_Op simply disappeared from the output. Nothing said + so: an emitter that enumerates the generator KINDS it understands cannot notice + a kind going away, only the parity harness can, and only if the harness does not + silently bucket the command as something else (it did; see COVERED there). + + The ninja command is ` --input F --op-header X.tmp --opcodes-header Y.tmp + && copy_if_different ... && remove ...`; CMake writes .tmp files and copies them + so a byte-identical regeneration does not retrigger downstream compiles. Bazel + genrules are hermetic and declare their outputs, so the .tmp dance is dropped and + the tool writes $(location ...) directly -- verified byte-identical by the harness. + """ + for cmd, cd, _declared, _produced in _blocks(): + if not re.search(r'(?:^|&& )\S*bin/generate-libjs-bytecode\b', cmd): + continue + # Find the tool in the WHOLE argv, not in the first `&&` segment: CMake + # emits `cd && ... && copy_if_different ...`, so segment 0 is + # the cd and the tool is in segment 1. Locate by name, then stop at the + # next `&&` -- position is not something to assume here (the same mistake + # the glslang parser documents). + toks = shlex.split(cmd.replace('&&', ' && ')) + i = next(i for i, x in enumerate(toks) + if x.endswith('bin/generate-libjs-bytecode')) + argv = toks[i + 1:] + if '&&' in argv: + argv = argv[:argv.index('&&')] + def val(flag): + v = argv[argv.index(flag) + 1] + # CMake passes the outputs relative to the command's cd, and with a + # .tmp suffix it then copies away; both are CMake's business, not ours. + v = v[:-4] if v.endswith('.tmp') else v + return os.path.relpath(os.path.normpath( + v if v.startswith('/') else os.path.join(cd, v)), FULL) + return dict( + input=os.path.relpath(argv[argv.index('--input') + 1], ROOT), + op_header=val('--op-header'), + opcodes_header=val('--opcodes-header'), + ) + return None + + +def emit_bytecode(bc): + if not bc: + return [] + print(' # The LibJS bytecode headers, generated by a tool THIS BUILD MAKES') + print(' # (//:generate-libjs-bytecode, the flapc crate\'s second bin --') + print(' # cargo_ring.bzl). Upstream a32d9c9f made interpreter.flap the sole') + print(' # authority for the instruction set and deleted both Bytecode.def') + print(' # and the Python generator that read it, which is why the overlay') + print(' # pinned at f9e34731 failed on a newer checkout: its glob for') + print(' # Libraries/LibJS/BytecodeDef/** matched nothing.') + print(' native.genrule(') + print(' name = %r,' % 'gen_Op') + print(' srcs = [%r],' % bc['input']) + print(' outs = [%r, %r],' % (bc['op_header'], bc['opcodes_header'])) + print(' tools = ["//:generate-libjs-bytecode"],') + print(' cmd = "$(location //:generate-libjs-bytecode) --input $(location %s) ' + '--op-header $(location %s) --opcodes-header $(location %s)",' + % (bc['input'], bc['op_header'], bc['opcodes_header'])) + print(' )') + return [bc['op_header'], bc['opcodes_header']] + + def parse_flap(): """The two chained self-built-tool commands: layout.conf, then the .S. @@ -154,11 +229,19 @@ def parse_flap(): def _rel_full(p): return os.path.relpath(os.path.normpath( p if p.startswith('/') else os.path.join(cd, p)), FULL) + # --bytecode-def is OPTIONAL, and its absence is the normal case + # now: upstream a32d9c9f ("LibJS: Derive bytecodes from Flap + # handlers") made interpreter.flap the sole authority, deleted + # Bytecode/Bytecode.def and dropped the flag. Read it out of the + # argv rather than requiring it -- this KeyError'd on the 71fb301a + # repin, which is the emitter doing its job (a hardcoded flag list + # is a claim about upstream that upstream can falsify). flap = dict( arch=d['arch'], object_format=d['object-format'], out=_rel_full(d['output']), constants=_rel_full(d['constants']), - bytecode_def=os.path.relpath(d['bytecode-def'], ROOT), + bytecode_def=(os.path.relpath(d['bytecode-def'], ROOT) + if 'bytecode-def' in d else None), dsl=os.path.relpath(d['input'], ROOT), extra=extra, ) @@ -349,17 +432,22 @@ def emit_flap(layout, flap): print(' cmd = "$(location //:generate_interpreter_layout) > $@",') print(' )') extra = (' ' + ' '.join(flap['extra'])) if flap['extra'] else '' + srcs = [flap['dsl']] + if flap['bytecode_def']: + srcs.append(flap['bytecode_def']) + srcs.append(':' + layout['out']) + bcd = (' --bytecode-def $(location %s)' % flap['bytecode_def'] + if flap['bytecode_def'] else '') print(' native.genrule(') print(' name = %r,' % 'gen_interpreter_asm') - print(' srcs = [%r, %r, %r],' - % (flap['dsl'], flap['bytecode_def'], ':' + layout['out'])) + print(' srcs = [%s],' % ', '.join(repr(s) for s in srcs)) print(' outs = [%r],' % flap['out']) print(' tools = [%r],' % FLAPC_TOOL) print(' cmd = "$(location %s) --arch %s --object-format %s ' - '--constants $(location %s) --bytecode-def $(location %s) ' + '--constants $(location %s)%s ' '--input $(location %s) --output $@%s",' % (FLAPC_TOOL, flap['arch'], flap['object_format'], - ':' + layout['out'], flap['bytecode_def'], flap['dsl'], extra)) + ':' + layout['out'], bcd, flap['dsl'], extra)) print(' )') return [flap['out']] @@ -376,12 +464,13 @@ def main(): rules = parse() shaders = parse_glslang() layout, flap = parse_flap() + bytecode = parse_bytecode() print('# AUTO-GENERATED by Meta/emit_root_codegen_bazel.py — do not edit.') # native.cc_library was removed from Bazel; the rule must be loaded. print('load("@rules_cc//cc:defs.bzl", "cc_library")') print('# %d Python-generator genrules for the root package,' % len(rules)) print('# %d glslang shader headers, %d self-built-tool genrules' - % (len(shaders), 2 if (layout and flap) else 0)) + % (len(shaders), (2 if (layout and flap) else 0) + (1 if bytecode else 0))) print('# (byte-parity: Meta/bazel_parity_harness.py).\n') print('def root_codegen():') seen = set() @@ -407,6 +496,7 @@ def main(): % (mk, script_ref, d['args'])) print(' )') all_outs.extend(d['outs']) + all_outs.extend(emit_bytecode(bytecode)) all_outs.extend(emit_flap(layout, flap)) emit_header_roots(all_outs) emit_glslang(shaders) diff --git a/examples/ladybird/workspace/cargo_crates.bzl b/examples/ladybird/workspace/cargo_crates.bzl index ce9e11f..9a3e7b4 100644 --- a/examples/ladybird/workspace/cargo_crates.bzl +++ b/examples/ladybird/workspace/cargo_crates.bzl @@ -1,7 +1,7 @@ # AUTO-GENERATED by Meta/emit_cargo_bazel.py — do not edit. load("@bazel_tools//tools/build_defs/repo:http.bzl", "http_archive") -# 154 crates.io crates, every sha256 lifted verbatim out of the two +# 155 crates.io crates, every sha256 lifted verbatim out of the two # Cargo.lock files. Nothing is re-hashed here and nothing was captured: # the lock file is the pin and the URL is a function of (name, version), # so this regenerates with no cargo, no network and no CMake. @@ -382,6 +382,14 @@ def cargo_crates(): type = "tgz", build_file_content = _CRATE_BUILD, ) + http_archive( + name = 'crate_foldhash_0_2_0', + urls = ['https://static.crates.io/crates/foldhash/foldhash-0.2.0.crate'], + sha256 = '77ce24cb58228fbb8aa041425bb1050850ac19177686ea6e0f41a70416f56fdb', + strip_prefix = 'foldhash-0.2.0', + type = "tgz", + build_file_content = _CRATE_BUILD, + ) http_archive( name = 'crate_form_urlencoded_1_2_2', urls = ['https://static.crates.io/crates/form_urlencoded/form_urlencoded-1.2.2.crate'], diff --git a/examples/ladybird/workspace/cargo_extension.bzl b/examples/ladybird/workspace/cargo_extension.bzl index 671c988..bc432b3 100644 --- a/examples/ladybird/workspace/cargo_extension.bzl +++ b/examples/ladybird/workspace/cargo_extension.bzl @@ -1,7 +1,7 @@ # AUTO-GENERATED by Meta/emit_cargo_bazel.py — do not edit. # http_archive is a REPOSITORY rule, so under bzlmod it cannot be # called from MODULE.bazel -- it has to come from a module extension. -# That indirection is also what lets one use_repo() name 157 repos +# That indirection is also what lets one use_repo() name 158 repos # without MODULE.bazel enumerating a single URL or hash. load(":cargo_crates.bzl", "cargo_crates") load(":cargo.bzl", "rust_toolchain_archives") diff --git a/examples/ladybird/workspace/cargo_index.bzl b/examples/ladybird/workspace/cargo_index.bzl index a0214ea..6d7c36a 100644 --- a/examples/ladybird/workspace/cargo_index.bzl +++ b/examples/ladybird/workspace/cargo_index.bzl @@ -54,6 +54,7 @@ CARGO_CRATE_FILES = { 'fastrand 2.4.0 a043dc74da1e37d6afe657061213aa6f425f855399a11d3463c6ecccc4dfda1f': '@crate_fastrand_2_4_0//:srcs', 'flatbuffers 25.12.19 35f6839d7b3b98adde531effaf34f0c2badc6f4735d26fe74709d8e513a96ef3': '@crate_flatbuffers_25_12_19//:srcs', 'foldhash 0.1.5 d9c4f5dac5e15c24eb999c26181a6ca40b39fe946cbe4c263c7209467bc83af2': '@crate_foldhash_0_1_5//:srcs', + 'foldhash 0.2.0 77ce24cb58228fbb8aa041425bb1050850ac19177686ea6e0f41a70416f56fdb': '@crate_foldhash_0_2_0//:srcs', 'form_urlencoded 1.2.2 cb4cb245038516f5f85277875cdaa4f7d2c9a0fa0468de06ed190163b1581fcf': '@crate_form_urlencoded_1_2_2//:srcs', 'getrandom 0.4.2 0de51e6874e94e7bf76d726fc5d13ba782deca734ff60d5bb2fb2607c7406555': '@crate_getrandom_0_4_2//:srcs', 'gimli 0.31.1 07e28edb80900c19c28f1072f2e8aeca7fa06b23cd4169cefe1af5aa3260783f': '@crate_gimli_0_31_1//:srcs', @@ -174,7 +175,7 @@ CARGO_CRATE_FILES = { CARGO_CRATE_SPECS = { 'libgfx_rust': { "manifest": 'Libraries/LibGfx/Rust/Cargo.toml', - "features": [], + "features": ['allocator'], "ffi_headers": ['RustFFI.h'], "ffi_prefix": 'LibGfx', "ffi_bare_include": True, @@ -221,24 +222,10 @@ CARGO_CRATE_SPECS = { "ffi_prefix": 'LibWeb', "ffi_bare_include": False, }, - 'libweb_css_rust': { - "manifest": 'Libraries/LibWeb/CSS/Rust/Cargo.toml', - "features": ['allocator'], - "ffi_headers": ['ComputedValuesRustFFI.h', 'RustFFI.h', 'SelectorRustFFI.h', 'StyleValueRustFFI.h'], - "ffi_prefix": 'LibWeb', - "ffi_bare_include": False, - }, - 'libweb_layout_rust': { - "manifest": 'Libraries/LibWeb/Layout/Rust/Cargo.toml', - "features": ['allocator'], - "ffi_headers": ['Layout/TreeBuilderRustFFI.h'], - "ffi_prefix": 'LibWeb', - "ffi_bare_include": False, - }, 'libweb_rust': { "manifest": 'Libraries/LibWeb/Rust/Cargo.toml', - "features": [], - "ffi_headers": ['HTML/Parser/RustFFI.h', 'HTMLTokenizerRustFFI.h'], + "features": ['style-recording'], + "ffi_headers": ['ComputedValuesRustFFI.h', 'HTML/Parser/RustFFI.h', 'HTMLTokenizerRustFFI.h', 'Layout/LayoutRustFFI.h', 'Layout/TreeBuilderRustFFI.h', 'RustFFI.h', 'SelectorRustFFI.h', 'StyleEngineBridgeGenerated.h', 'StyleEngineBridgeGenerated.inc', 'StyleEngineRustFFI.h', 'StyleEngineStateFactsGenerated.inc', 'StyleValueRustFFI.h'], "ffi_prefix": 'LibWeb', "ffi_bare_include": False, }, diff --git a/examples/ladybird/workspace/cargo_ring.bzl b/examples/ladybird/workspace/cargo_ring.bzl index c22ae4a..23f032c 100644 --- a/examples/ladybird/workspace/cargo_ring.bzl +++ b/examples/ladybird/workspace/cargo_ring.bzl @@ -2,8 +2,8 @@ load(":cargo.bzl", "cargo_binary", "cargo_crate", "cargo_lib", "cargo_bare_include", "rust_sysroot") load(":cargo_index.bzl", "CARGO_CRATE_FILES", "CARGO_CRATE_SPECS") -# Ladybird's 10 production Rust crates and 2 binary crates, BUILT BY BAZEL -# from the 154 crates.io crates Bazel fetched -- replacing the prebuilt +# Ladybird's 8 production Rust crates and 4 binary crates, BUILT BY BAZEL +# from the 155 crates.io crates Bazel fetched -- replacing the prebuilt # 260 MB librust_combined.a that was copied out of Build/full/cargo/, the # `ar -M` merge that produced it (README step 1b), and the reference build's # flapc and cranelift-compiler binaries. Nothing here names Build/full. @@ -29,11 +29,16 @@ def cargo_ring(): # cargo WORKSPACE. Each of them path-depends on # Libraries/RustAllocator.rs and several are path-dependencies of each # other, so cargo resolves the whole workspace whichever crate you ask - # for. Over-declaring costs a rebuild of 11 crates when any .rs changes; - # under-declaring silently reuses a stale archive. The trade is the right - # way round, and it is real debt: per-crate source sets need the - # path-dependency graph read out of the manifests. - crate_srcs = native.glob(['Cargo.toml', 'Cargo.lock', 'rust-toolchain.toml', 'Libraries/RustAllocator.rs', 'Libraries/*/Rust/**', 'Libraries/LibJS/BytecodeDef/**'], allow_empty = False) + ['//Libraries/LibWeb:rust_crate_srcs'] + # for. Over-declaring costs a rebuild of every crate when any .rs + # changes; under-declaring silently reuses a stale archive. The trade is + # the right way round, and it is real debt: per-crate source sets need + # the path-dependency graph read out of the manifests. + # + # The patterns are DERIVED from Cargo.toml's member list closed over the + # manifests' `path =` deps, not written down: a directory written down + # here is an allow_empty=False glob that outlives the directory, and + # loading then fails before any target can say why. + crate_srcs = native.glob(['Cargo.toml', 'Cargo.lock', 'rust-toolchain.toml', 'Libraries/RustAllocator.rs', 'Libraries/LibGfx/Rust/**', 'Libraries/LibJS/Flap/**', 'Libraries/LibJS/Rust/**', 'Libraries/LibRegex/Rust/**', 'Libraries/LibTextCodec/Rust/**', 'Libraries/LibURL/Rust/**', 'Libraries/LibUnicode/Rust/**', 'Libraries/LibWasm/Rust/**'], exclude = ['Libraries/LibGfx/Rust/target/**', 'Libraries/LibJS/Flap/target/**', 'Libraries/LibJS/Rust/target/**', 'Libraries/LibRegex/Rust/target/**', 'Libraries/LibTextCodec/Rust/target/**', 'Libraries/LibURL/Rust/target/**', 'Libraries/LibUnicode/Rust/target/**', 'Libraries/LibWasm/Rust/target/**'], allow_empty = False) + ['//Libraries/LibWeb:rust_crate_srcs'] cargo_crate( name = 'libgfx_rust', @@ -78,7 +83,7 @@ def cargo_ring(): # Build-script inputs, taken from the reference build's DEPFILES # rather than from reading build.rs: this crate's build script # generates Rust from these, so they are compile inputs. - srcs = crate_srcs + ['Libraries/LibJS/Bytecode/Bytecode.def'], + srcs = crate_srcs + ['Libraries/LibJS/Interpreter/interpreter.flap'], sysroot = ":rust_sysroot", ) @@ -239,60 +244,6 @@ def cargo_ring(): crate = ':libweb_content_blocker_rust', ) - cargo_crate( - name = 'libweb_css_rust', - crate = 'libweb_css_rust', - crates = CARGO_CRATE_FILES, - crate_features = CARGO_CRATE_SPECS['libweb_css_rust']["features"], - ffi_headers = CARGO_CRATE_SPECS['libweb_css_rust']["ffi_headers"], - ffi_prefix = CARGO_CRATE_SPECS['libweb_css_rust']["ffi_prefix"], - ffi_bare_include = CARGO_CRATE_SPECS['libweb_css_rust']["ffi_bare_include"], - manifest = CARGO_CRATE_SPECS['libweb_css_rust']["manifest"], - srcs = crate_srcs, - sysroot = ":rust_sysroot", - ) - - # The consumable target: this crate's archive to link and this - # crate's generated FFI headers to include, one for one with - # CMake's per-library edge. NOT a shared --start-group over all - # ten archives: the crates have no true cross-crate symbol - # references (measured: 0), the shared symbols are each crate's - # own copy of rust-std, and a group makes ld satisfy one crate's - # std symbol from another crate's object -- dragging that - # crate's C++ FFI into a target that never linked it. See the - # block comment in cargo.bzl. - cargo_lib( - name = 'libweb_css_rust_lib', - crate = ':libweb_css_rust', - ) - - cargo_crate( - name = 'libweb_layout_rust', - crate = 'libweb_layout_rust', - crates = CARGO_CRATE_FILES, - crate_features = CARGO_CRATE_SPECS['libweb_layout_rust']["features"], - ffi_headers = CARGO_CRATE_SPECS['libweb_layout_rust']["ffi_headers"], - ffi_prefix = CARGO_CRATE_SPECS['libweb_layout_rust']["ffi_prefix"], - ffi_bare_include = CARGO_CRATE_SPECS['libweb_layout_rust']["ffi_bare_include"], - manifest = CARGO_CRATE_SPECS['libweb_layout_rust']["manifest"], - srcs = crate_srcs, - sysroot = ":rust_sysroot", - ) - - # The consumable target: this crate's archive to link and this - # crate's generated FFI headers to include, one for one with - # CMake's per-library edge. NOT a shared --start-group over all - # ten archives: the crates have no true cross-crate symbol - # references (measured: 0), the shared symbols are each crate's - # own copy of rust-std, and a group makes ld satisfy one crate's - # std symbol from another crate's object -- dragging that - # crate's C++ FFI into a target that never linked it. See the - # block comment in cargo.bzl. - cargo_lib( - name = 'libweb_layout_rust_lib', - crate = ':libweb_layout_rust', - ) - cargo_crate( name = 'libweb_rust', crate = 'libweb_rust', @@ -337,6 +288,19 @@ def cargo_ring(): # CraneliftFFI.h that LibWasm's CraneliftBridge.cpp # includes. It is a root-workspace member, so its # source set is the shared crate_srcs. + cargo_binary( + name = 'generate-libjs-bytecode', + bin = 'generate-libjs-bytecode', + crate = 'flapc', + crates = CARGO_CRATE_FILES, + manifest = 'Libraries/LibJS/Flap/Cargo.toml', + # Its OWN workspace (`exclude`d from the root one), so its source + # set is its own subtree plus what it include_str!s from above + # it -- both derived, the subtree from Cargo.toml's `exclude` + # and the rest by scanning for include_str!. + srcs = native.glob(['Libraries/LibJS/Flap/**', 'Libraries/LibJS/Interpreter/interpreter.flap', 'rust-toolchain.toml'], exclude = ['Libraries/LibJS/Flap/target/**'], allow_empty = False), + sysroot = ":rust_sysroot", + ) cargo_binary( name = 'flapc', bin = 'flapc', @@ -344,8 +308,10 @@ def cargo_ring(): crates = CARGO_CRATE_FILES, manifest = 'Libraries/LibJS/Flap/Cargo.toml', # Its OWN workspace (`exclude`d from the root one), so its source - # set is its own subtree. - srcs = native.glob(['Libraries/LibJS/Flap/Cargo.toml', 'Libraries/LibJS/Flap/Cargo.lock', 'Libraries/LibJS/Flap/src/**', 'Libraries/LibJS/Flap/benches/**', 'Libraries/LibJS/Flap/tests/**', 'Libraries/LibJS/BytecodeDef/**', 'Libraries/LibJS/Bytecode/Bytecode.def', 'Libraries/LibJS/Interpreter/interpreter.flap', 'rust-toolchain.toml'], allow_empty = False), + # set is its own subtree plus what it include_str!s from above + # it -- both derived, the subtree from Cargo.toml's `exclude` + # and the rest by scanning for include_str!. + srcs = native.glob(['Libraries/LibJS/Flap/**', 'Libraries/LibJS/Interpreter/interpreter.flap', 'rust-toolchain.toml'], exclude = ['Libraries/LibJS/Flap/target/**'], allow_empty = False), sysroot = ":rust_sysroot", ) cargo_binary( @@ -380,3 +346,20 @@ def cargo_ring(): name = 'libwasm_cranelift_bare_include', crate = ':cranelift-compiler', ) + cargo_binary( + name = 'style-replay', + bin = 'style-replay', + crate = 'libweb_rust', + crates = CARGO_CRATE_FILES, + manifest = 'Libraries/LibWeb/Rust/Cargo.toml', + # build_rust_binary() takes FEATURES too, and this one does: + # the same crate builds a staticlib and this binary, and only + # the features distinguish what each gets. + crate_features = ['style-recording'], + # A member of the ROOT cargo workspace (Cargo.toml lists it), so + # cargo resolves the whole workspace whichever crate you ask for + # and the source set is the shared one -- same over-declaration, + # same reason, as the staticlib crates. + srcs = crate_srcs, + sysroot = ":rust_sysroot", + ) diff --git a/examples/ladybird/workspace/codegen_root.bzl b/examples/ladybird/workspace/codegen_root.bzl index 6bb9ed4..003ca55 100644 --- a/examples/ladybird/workspace/codegen_root.bzl +++ b/examples/ladybird/workspace/codegen_root.bzl @@ -1,7 +1,7 @@ # AUTO-GENERATED by Meta/emit_root_codegen_bazel.py — do not edit. load("@rules_cc//cc:defs.bzl", "cc_library") -# 20 Python-generator genrules for the root package, -# 2 glslang shader headers, 2 self-built-tool genrules +# 21 Python-generator genrules for the root package, +# 2 glslang shader headers, 3 self-built-tool genrules # (byte-parity: Meta/bazel_parity_harness.py). def root_codegen(): @@ -11,12 +11,6 @@ def root_codegen(): outs = ['Libraries/LibHTTP/HSTSPreloadData.h', 'Libraries/LibHTTP/HSTSPreloadData.cpp'], cmd = "PYTHONHASHSEED=0 python3 Meta/Generators/generate_hsts_preload_data.py -h $(location Libraries/LibHTTP/HSTSPreloadData.h) -c $(location Libraries/LibHTTP/HSTSPreloadData.cpp) -p $(location @hsts_preload_json//file)", ) - native.genrule( - name = 'gen_Op', - srcs = ['Libraries/LibJS/Bytecode/Bytecode.def'] + ["//Meta:generators"], - outs = ['Libraries/LibJS/Bytecode/Op.h', 'Libraries/LibJS/Bytecode/Op.cpp', 'Libraries/LibJS/Bytecode/OpCodes.h'], - cmd = "PYTHONHASHSEED=0 python3 Meta/Generators/generate_libjs_bytecode_def_derived.py -h $(location Libraries/LibJS/Bytecode/Op.h) -c $(location Libraries/LibJS/Bytecode/Op.cpp) -i $(location Libraries/LibJS/Bytecode/Bytecode.def) -x $(location Libraries/LibJS/Bytecode/OpCodes.h)", - ) native.genrule( name = 'gen_RequestClientEndpoint', srcs = ['Services/RequestServer/RequestClient.ipc'] + ["//Meta:generators"], @@ -47,6 +41,18 @@ def root_codegen(): outs = ['Services/ImageDecoder/ImageDecoderServerEndpoint.h'], cmd = "PYTHONHASHSEED=0 python3 Meta/Generators/generate_ipc_definitions.py --input $(location Services/ImageDecoder/ImageDecoderServer.ipc) --output $(location Services/ImageDecoder/ImageDecoderServerEndpoint.h)", ) + native.genrule( + name = 'gen_VideoPresentationClientEndpoint', + srcs = ['Libraries/LibMedia/VideoPresentation/VideoPresentationClient.ipc'] + ["//Meta:generators"], + outs = ['Libraries/LibMedia/VideoPresentation/VideoPresentationClientEndpoint.h'], + cmd = "PYTHONHASHSEED=0 python3 Meta/Generators/generate_ipc_definitions.py --input $(location Libraries/LibMedia/VideoPresentation/VideoPresentationClient.ipc) --output $(location Libraries/LibMedia/VideoPresentation/VideoPresentationClientEndpoint.h)", + ) + native.genrule( + name = 'gen_VideoPresentationServerEndpoint', + srcs = ['Libraries/LibMedia/VideoPresentation/VideoPresentationServer.ipc'] + ["//Meta:generators"], + outs = ['Libraries/LibMedia/VideoPresentation/VideoPresentationServerEndpoint.h'], + cmd = "PYTHONHASHSEED=0 python3 Meta/Generators/generate_ipc_definitions.py --input $(location Libraries/LibMedia/VideoPresentation/VideoPresentationServer.ipc) --output $(location Libraries/LibMedia/VideoPresentation/VideoPresentationServerEndpoint.h)", + ) native.genrule( name = 'gen_UIProcessServerEndpoint', srcs = ['Libraries/LibWebView/UIProcessServer.ipc'] + ["//Meta:generators"], @@ -125,6 +131,20 @@ def root_codegen(): outs = ['Services/Compositor/WebGLCommandReplayer.h', 'Services/Compositor/WebGLCommandReplayer.cpp'], cmd = "PYTHONHASHSEED=0 python3 Meta/Generators/generate_compositor_webgl_replayer.py -h $(location Services/Compositor/WebGLCommandReplayer.h) -c $(location Services/Compositor/WebGLCommandReplayer.cpp) -j $(location //Libraries/LibWeb:WebGL/GLFunctions.json)", ) + # The LibJS bytecode headers, generated by a tool THIS BUILD MAKES + # (//:generate-libjs-bytecode, the flapc crate's second bin -- + # cargo_ring.bzl). Upstream a32d9c9f made interpreter.flap the sole + # authority for the instruction set and deleted both Bytecode.def + # and the Python generator that read it, which is why the overlay + # pinned at f9e34731 failed on a newer checkout: its glob for + # Libraries/LibJS/BytecodeDef/** matched nothing. + native.genrule( + name = 'gen_Op', + srcs = ['Libraries/LibJS/Interpreter/interpreter.flap'], + outs = ['Libraries/LibJS/Bytecode/Op.h', 'Libraries/LibJS/Bytecode/OpCodes.h'], + tools = ["//:generate-libjs-bytecode"], + cmd = "$(location //:generate-libjs-bytecode) --input $(location Libraries/LibJS/Interpreter/interpreter.flap) --op-header $(location Libraries/LibJS/Bytecode/Op.h) --opcodes-header $(location Libraries/LibJS/Bytecode/OpCodes.h)", + ) # Two chained self-built tools. generate_interpreter_layout is # built by Bazel (//:generate_interpreter_layout) and emits the # struct offsets flapc needs; flapc then compiles the Flap DSL to @@ -142,14 +162,14 @@ def root_codegen(): ) native.genrule( name = 'gen_interpreter_asm', - srcs = ['Libraries/LibJS/Interpreter/interpreter.flap', 'Libraries/LibJS/Bytecode/Bytecode.def', ':Libraries/LibJS/Interpreter/layout.conf'], + srcs = ['Libraries/LibJS/Interpreter/interpreter.flap', ':Libraries/LibJS/Interpreter/layout.conf'], outs = ['Libraries/LibJS/Interpreter/interpreter_x86_64.S'], tools = ['//:flapc'], - cmd = "$(location //:flapc) --arch x86_64 --object-format elf --constants $(location :Libraries/LibJS/Interpreter/layout.conf) --bytecode-def $(location Libraries/LibJS/Bytecode/Bytecode.def) --input $(location Libraries/LibJS/Interpreter/interpreter.flap) --output $@", + cmd = "$(location //:flapc) --arch x86_64 --object-format elf --constants $(location :Libraries/LibJS/Interpreter/layout.conf) --input $(location Libraries/LibJS/Interpreter/interpreter.flap) --output $@", ) cc_library( name = 'generated_libraries_headers', - hdrs = [':Libraries/LibGfx/ImageFormats/TIFFMetadata.h', ':Libraries/LibHTTP/HSTSPreloadData.h', ':Libraries/LibJS/Bytecode/Op.h', ':Libraries/LibJS/Bytecode/OpCodes.h', ':Libraries/LibWebView/UIProcessClientEndpoint.h', ':Libraries/LibWebView/UIProcessServerEndpoint.h'], + hdrs = [':Libraries/LibGfx/ImageFormats/TIFFMetadata.h', ':Libraries/LibHTTP/HSTSPreloadData.h', ':Libraries/LibJS/Bytecode/Op.h', ':Libraries/LibJS/Bytecode/OpCodes.h', ':Libraries/LibMedia/VideoPresentation/VideoPresentationClientEndpoint.h', ':Libraries/LibMedia/VideoPresentation/VideoPresentationServerEndpoint.h', ':Libraries/LibWebView/UIProcessClientEndpoint.h', ':Libraries/LibWebView/UIProcessServerEndpoint.h'], includes = ['Libraries'], ) cc_library( diff --git a/examples/ladybird/workspace/export_headers.bzl b/examples/ladybird/workspace/export_headers.bzl index 1870746..460bd29 100644 --- a/examples/ladybird/workspace/export_headers.bzl +++ b/examples/ladybird/workspace/export_headers.bzl @@ -83,7 +83,7 @@ def export_headers(): name = 'gen_AK_Debug_h', srcs = ['AK/Debug.h.in'], outs = ['genroot/AK/Debug.h'], - cmd = "cat > $@ <<'LADYBIRD_EOF'\n/*\n * Copyright (c) 2020-2024, the SerenityOS developers.\n *\n * SPDX-License-Identifier: BSD-2-Clause\n */\n\n#pragma once\n\n#ifndef AK_STRINGBASE_VERIFY_LAUNDER_DEBUG\n# define AK_STRINGBASE_VERIFY_LAUNDER_DEBUG 0\n#endif\n\n#ifndef AUDIO_DEBUG\n# define AUDIO_DEBUG 0\n#endif\n\n#ifndef BMP_DEBUG\n# define BMP_DEBUG 0\n#endif\n\n#ifndef CACHE_DEBUG\n# define CACHE_DEBUG 0\n#endif\n\n#ifndef CALLBACK_MACHINE_DEBUG\n# define CALLBACK_MACHINE_DEBUG 0\n#endif\n\n#ifndef CANVAS_RENDERING_CONTEXT_2D_DEBUG\n# define CANVAS_RENDERING_CONTEXT_2D_DEBUG 0\n#endif\n\n#ifndef CRYPTO_DEBUG\n# define CRYPTO_DEBUG 0\n#endif\n\n#ifndef COMPOSITOR_DEBUG\n# define COMPOSITOR_DEBUG 0\n#endif\n\n#ifndef CSS_LOADER_DEBUG\n# define CSS_LOADER_DEBUG 0\n#endif\n\n#ifndef CSS_PARSER_DEBUG\n# define CSS_PARSER_DEBUG 0\n#endif\n\n#ifndef CSS_TOKENIZER_DEBUG\n# define CSS_TOKENIZER_DEBUG 0\n#endif\n\n#ifndef CSS_TRANSITIONS_DEBUG\n# define CSS_TRANSITIONS_DEBUG 0\n#endif\n\n#ifndef CURL_DEBUG\n# define CURL_DEBUG 0\n#endif\n\n#ifndef DEVTOOLS_DEBUG\n# define DEVTOOLS_DEBUG 0\n#endif\n\n#ifndef DNS_DEBUG\n# define DNS_DEBUG 0\n#endif\n\n#ifndef EDITOR_DEBUG\n# define EDITOR_DEBUG 0\n#endif\n\n#ifndef FILE_WATCHER_DEBUG\n# define FILE_WATCHER_DEBUG 0\n#endif\n\n#ifndef FLAC_ENCODER_DEBUG\n# define FLAC_ENCODER_DEBUG 0\n#endif\n\n#ifndef FORMATTING_CONTEXT_TRACE_DEBUG\n# define FORMATTING_CONTEXT_TRACE_DEBUG 0\n#endif\n\n#ifndef GIF_DEBUG\n# define GIF_DEBUG 0\n#endif\n\n#ifndef HEAP_DEBUG\n# define HEAP_DEBUG 0\n#endif\n\n#ifndef INCREMENTAL_SWEEP_DEBUG\n# define INCREMENTAL_SWEEP_DEBUG 0\n#endif\n\n#ifndef HIGHLIGHT_FOCUSED_FRAME_DEBUG\n# define HIGHLIGHT_FOCUSED_FRAME_DEBUG 0\n#endif\n\n#ifndef HTML_SCRIPT_DEBUG\n# define HTML_SCRIPT_DEBUG 0\n#endif\n\n#ifndef HTTP_DISK_CACHE_DEBUG\n# define HTTP_DISK_CACHE_DEBUG 0\n#endif\n\n#ifndef HTTP_MEMORY_CACHE_DEBUG\n# define HTTP_MEMORY_CACHE_DEBUG 0\n#endif\n\n#ifndef HTTPJOB_DEBUG\n# define HTTPJOB_DEBUG 0\n#endif\n\n#ifndef ICO_DEBUG\n# define ICO_DEBUG 0\n#endif\n\n#ifndef IDB_DEBUG\n# define IDB_DEBUG 0\n#endif\n\n#ifndef IDL_DEBUG\n# define IDL_DEBUG 0\n#endif\n\n#ifndef IMAGE_DECODER_DEBUG\n# define IMAGE_DECODER_DEBUG 0\n#endif\n\n#ifndef IMAGE_LOADER_DEBUG\n# define IMAGE_LOADER_DEBUG 0\n#endif\n\n#ifndef JS_MODULE_DEBUG\n# define JS_MODULE_DEBUG 0\n#endif\n\n#ifndef LEXER_DEBUG\n# define LEXER_DEBUG 0\n#endif\n\n#ifndef LIBWEB_CSS_ANIMATION_DEBUG\n# define LIBWEB_CSS_ANIMATION_DEBUG 0\n#endif\n\n#ifndef LIBWEB_CSS_DEBUG\n# define LIBWEB_CSS_DEBUG 0\n#endif\n\n#ifndef LIBWEB_WASM_DEBUG\n# define LIBWEB_WASM_DEBUG 0\n#endif\n\n#ifndef LINE_EDITOR_DEBUG\n# define LINE_EDITOR_DEBUG 0\n#endif\n\n#ifndef WEBVIEW_HISTORY_DEBUG\n# define WEBVIEW_HISTORY_DEBUG 0\n#endif\n\n#ifndef LZW_DEBUG\n# define LZW_DEBUG 0\n#endif\n\n#ifndef MACH_PORT_DEBUG\n# define MACH_PORT_DEBUG 0\n#endif\n\n#ifndef MATROSKA_DEBUG\n# define MATROSKA_DEBUG 0\n#endif\n\n#ifndef MATROSKA_TRACE_DEBUG\n# define MATROSKA_TRACE_DEBUG 0\n#endif\n\n#ifndef HTML_PARSER_DEBUG\n# define HTML_PARSER_DEBUG 0\n#endif\n\n#ifndef PATH_DEBUG\n# define PATH_DEBUG 0\n#endif\n\n#ifndef PLAYBACK_MANAGER_DEBUG\n# define PLAYBACK_MANAGER_DEBUG 0\n#endif\n\n#ifndef PNG_DEBUG\n# define PNG_DEBUG 0\n#endif\n\n#ifndef PROMISE_DEBUG\n# define PROMISE_DEBUG 0\n#endif\n\n#ifndef REGEX_DEBUG\n# define REGEX_DEBUG 0\n#endif\n\n#ifndef REQUESTSERVER_DEBUG\n# define REQUESTSERVER_DEBUG 0\n#endif\n\n#ifndef REQUESTSERVER_WIRE_DEBUG\n# define REQUESTSERVER_WIRE_DEBUG 0\n#endif\n\n#ifndef RESOURCE_DEBUG\n# define RESOURCE_DEBUG 0\n#endif\n\n#ifndef SPAM_DEBUG\n# define SPAM_DEBUG 0\n#endif\n\n#ifndef STRUCTURED_SERIALIZE_DEBUG\n# define STRUCTURED_SERIALIZE_DEBUG 0\n#endif\n\n#ifndef SYNTAX_HIGHLIGHTING_DEBUG\n# define SYNTAX_HIGHLIGHTING_DEBUG 0\n#endif\n\n#ifndef STYLE_INVALIDATION_DEBUG\n# define STYLE_INVALIDATION_DEBUG 0\n#endif\n\n#ifndef TEXTEDITOR_DEBUG\n# define TEXTEDITOR_DEBUG 0\n#endif\n\n#ifndef TIFF_DEBUG\n# define TIFF_DEBUG 0\n#endif\n\n#ifndef TLS_DEBUG\n# define TLS_DEBUG 0\n#endif\n\n#ifndef TOKENIZER_TRACE_DEBUG\n# define TOKENIZER_TRACE_DEBUG 0\n#endif\n\n#ifndef UPDATE_LAYOUT_DEBUG\n# define UPDATE_LAYOUT_DEBUG 0\n#endif\n\n#ifndef URL_PARSER_DEBUG\n# define URL_PARSER_DEBUG 0\n#endif\n\n#ifndef URL_PATTERN_DEBUG\n# define URL_PATTERN_DEBUG 0\n#endif\n\n#ifndef UTF8_DEBUG\n# define UTF8_DEBUG 0\n#endif\n\n#ifndef VULKAN_VALIDATION_LAYERS_DEBUG\n# define VULKAN_VALIDATION_LAYERS_DEBUG 0\n#endif\n\n#ifndef WASI_DEBUG\n# define WASI_DEBUG 0\n#endif\n\n#ifndef WASI_FINE_GRAINED_DEBUG\n# define WASI_FINE_GRAINED_DEBUG 0\n#endif\n\n#ifndef WASM_BINPARSER_DEBUG\n# define WASM_BINPARSER_DEBUG 0\n#endif\n\n#ifndef WASM_CRANELIFT_DEBUG\n# define WASM_CRANELIFT_DEBUG 0\n#endif\n\n#ifndef WASM_TRACE_DEBUG\n# define WASM_TRACE_DEBUG 0\n#endif\n\n#ifndef WASM_VALIDATOR_DEBUG\n# define WASM_VALIDATOR_DEBUG 0\n#endif\n\n#ifndef WEBDRIVER_DEBUG\n# define WEBDRIVER_DEBUG 0\n#endif\n\n#ifndef WEBDRIVER_ROUTE_DEBUG\n# define WEBDRIVER_ROUTE_DEBUG 0\n#endif\n\n#ifndef WEBGL_CONTEXT_DEBUG\n# define WEBGL_CONTEXT_DEBUG 0\n#endif\n\n#ifndef WEBVIEW_PROCESS_DEBUG\n# define WEBVIEW_PROCESS_DEBUG 0\n#endif\n\n#ifndef WEB_FETCH_DEBUG\n# define WEB_FETCH_DEBUG 0\n#endif\n\n#ifndef WEB_WORKER_DEBUG\n# define WEB_WORKER_DEBUG 0\n#endif\n\n#ifndef WEBP_DEBUG\n# define WEBP_DEBUG 0\n#endif\n\n#ifndef XML_PARSER_DEBUG\n# define XML_PARSER_DEBUG 0\n#endif\nLADYBIRD_EOF\n", + cmd = "cat > $@ <<'LADYBIRD_EOF'\n/*\n * Copyright (c) 2020-2024, the SerenityOS developers.\n *\n * SPDX-License-Identifier: BSD-2-Clause\n */\n\n#pragma once\n\n#ifndef AK_STRINGBASE_VERIFY_LAUNDER_DEBUG\n# define AK_STRINGBASE_VERIFY_LAUNDER_DEBUG 0\n#endif\n\n#ifndef AUDIO_DEBUG\n# define AUDIO_DEBUG 0\n#endif\n\n#ifndef BMP_DEBUG\n# define BMP_DEBUG 0\n#endif\n\n#ifndef CACHE_DEBUG\n# define CACHE_DEBUG 0\n#endif\n\n#ifndef CALLBACK_MACHINE_DEBUG\n# define CALLBACK_MACHINE_DEBUG 0\n#endif\n\n#ifndef CANVAS_RENDERING_CONTEXT_2D_DEBUG\n# define CANVAS_RENDERING_CONTEXT_2D_DEBUG 0\n#endif\n\n#ifndef CRYPTO_DEBUG\n# define CRYPTO_DEBUG 0\n#endif\n\n#ifndef COMPOSITOR_DEBUG\n# define COMPOSITOR_DEBUG 0\n#endif\n\n#ifndef CSS_LOADER_DEBUG\n# define CSS_LOADER_DEBUG 0\n#endif\n\n#ifndef CSS_PARSER_DEBUG\n# define CSS_PARSER_DEBUG 0\n#endif\n\n#ifndef CSS_TOKENIZER_DEBUG\n# define CSS_TOKENIZER_DEBUG 0\n#endif\n\n#ifndef CSS_TRANSITIONS_DEBUG\n# define CSS_TRANSITIONS_DEBUG 0\n#endif\n\n#ifndef CURL_DEBUG\n# define CURL_DEBUG 0\n#endif\n\n#ifndef DEVTOOLS_DEBUG\n# define DEVTOOLS_DEBUG 0\n#endif\n\n#ifndef DNS_DEBUG\n# define DNS_DEBUG 0\n#endif\n\n#ifndef EDITOR_DEBUG\n# define EDITOR_DEBUG 0\n#endif\n\n#ifndef FILE_WATCHER_DEBUG\n# define FILE_WATCHER_DEBUG 0\n#endif\n\n#ifndef FLAC_ENCODER_DEBUG\n# define FLAC_ENCODER_DEBUG 0\n#endif\n\n#ifndef FORMATTING_CONTEXT_TRACE_DEBUG\n# define FORMATTING_CONTEXT_TRACE_DEBUG 0\n#endif\n\n#ifndef GIF_DEBUG\n# define GIF_DEBUG 0\n#endif\n\n#ifndef HEAP_DEBUG\n# define HEAP_DEBUG 0\n#endif\n\n#ifndef INCREMENTAL_SWEEP_DEBUG\n# define INCREMENTAL_SWEEP_DEBUG 0\n#endif\n\n#ifndef HIGHLIGHT_FOCUSED_FRAME_DEBUG\n# define HIGHLIGHT_FOCUSED_FRAME_DEBUG 0\n#endif\n\n#ifndef HTML_SCRIPT_DEBUG\n# define HTML_SCRIPT_DEBUG 0\n#endif\n\n#ifndef HTTP_DISK_CACHE_DEBUG\n# define HTTP_DISK_CACHE_DEBUG 0\n#endif\n\n#ifndef HTTP_MEMORY_CACHE_DEBUG\n# define HTTP_MEMORY_CACHE_DEBUG 0\n#endif\n\n#ifndef HTTPJOB_DEBUG\n# define HTTPJOB_DEBUG 0\n#endif\n\n#ifndef ICO_DEBUG\n# define ICO_DEBUG 0\n#endif\n\n#ifndef IDB_DEBUG\n# define IDB_DEBUG 0\n#endif\n\n#ifndef IDL_DEBUG\n# define IDL_DEBUG 0\n#endif\n\n#ifndef IMAGE_DECODER_DEBUG\n# define IMAGE_DECODER_DEBUG 0\n#endif\n\n#ifndef IMAGE_LOADER_DEBUG\n# define IMAGE_LOADER_DEBUG 0\n#endif\n\n#ifndef JS_MODULE_DEBUG\n# define JS_MODULE_DEBUG 0\n#endif\n\n#ifndef LEXER_DEBUG\n# define LEXER_DEBUG 0\n#endif\n\n#ifndef LIBWEB_CSS_ANIMATION_DEBUG\n# define LIBWEB_CSS_ANIMATION_DEBUG 0\n#endif\n\n#ifndef LIBWEB_CSS_DEBUG\n# define LIBWEB_CSS_DEBUG 0\n#endif\n\n#ifndef LIBWEB_WASM_DEBUG\n# define LIBWEB_WASM_DEBUG 0\n#endif\n\n#ifndef LINE_EDITOR_DEBUG\n# define LINE_EDITOR_DEBUG 0\n#endif\n\n#ifndef WEBVIEW_HISTORY_DEBUG\n# define WEBVIEW_HISTORY_DEBUG 0\n#endif\n\n#ifndef LZW_DEBUG\n# define LZW_DEBUG 0\n#endif\n\n#ifndef MACH_PORT_DEBUG\n# define MACH_PORT_DEBUG 0\n#endif\n\n#ifndef MATROSKA_DEBUG\n# define MATROSKA_DEBUG 0\n#endif\n\n#ifndef MATROSKA_TRACE_DEBUG\n# define MATROSKA_TRACE_DEBUG 0\n#endif\n\n#ifndef HTML_PARSER_DEBUG\n# define HTML_PARSER_DEBUG 0\n#endif\n\n#ifndef PATH_DEBUG\n# define PATH_DEBUG 0\n#endif\n\n#ifndef PLAYBACK_MANAGER_DEBUG\n# define PLAYBACK_MANAGER_DEBUG 0\n#endif\n\n#ifndef PNG_DEBUG\n# define PNG_DEBUG 0\n#endif\n\n#ifndef PROMISE_DEBUG\n# define PROMISE_DEBUG 0\n#endif\n\n#ifndef REGEX_DEBUG\n# define REGEX_DEBUG 0\n#endif\n\n#ifndef REQUESTSERVER_DEBUG\n# define REQUESTSERVER_DEBUG 0\n#endif\n\n#ifndef REQUESTSERVER_WIRE_DEBUG\n# define REQUESTSERVER_WIRE_DEBUG 0\n#endif\n\n#ifndef RESOURCE_DEBUG\n# define RESOURCE_DEBUG 0\n#endif\n\n#ifndef SHARED_QUEUE_DEBUG\n# define SHARED_QUEUE_DEBUG 0\n#endif\n\n#ifndef SPAM_DEBUG\n# define SPAM_DEBUG 0\n#endif\n\n#ifndef STRUCTURED_SERIALIZE_DEBUG\n# define STRUCTURED_SERIALIZE_DEBUG 0\n#endif\n\n#ifndef SYNTAX_HIGHLIGHTING_DEBUG\n# define SYNTAX_HIGHLIGHTING_DEBUG 0\n#endif\n\n#ifndef TEXTEDITOR_DEBUG\n# define TEXTEDITOR_DEBUG 0\n#endif\n\n#ifndef TIFF_DEBUG\n# define TIFF_DEBUG 0\n#endif\n\n#ifndef TLS_DEBUG\n# define TLS_DEBUG 0\n#endif\n\n#ifndef TOKENIZER_TRACE_DEBUG\n# define TOKENIZER_TRACE_DEBUG 0\n#endif\n\n#ifndef UPDATE_LAYOUT_DEBUG\n# define UPDATE_LAYOUT_DEBUG 0\n#endif\n\n#ifndef URL_PARSER_DEBUG\n# define URL_PARSER_DEBUG 0\n#endif\n\n#ifndef URL_PATTERN_DEBUG\n# define URL_PATTERN_DEBUG 0\n#endif\n\n#ifndef UTF8_DEBUG\n# define UTF8_DEBUG 0\n#endif\n\n#ifndef VIDEO_FRAME_POOL_DEBUG\n# define VIDEO_FRAME_POOL_DEBUG 0\n#endif\n\n#ifndef VULKAN_VALIDATION_LAYERS_DEBUG\n# define VULKAN_VALIDATION_LAYERS_DEBUG 0\n#endif\n\n#ifndef WASI_DEBUG\n# define WASI_DEBUG 0\n#endif\n\n#ifndef WASI_FINE_GRAINED_DEBUG\n# define WASI_FINE_GRAINED_DEBUG 0\n#endif\n\n#ifndef WASM_BINPARSER_DEBUG\n# define WASM_BINPARSER_DEBUG 0\n#endif\n\n#ifndef WASM_CRANELIFT_DEBUG\n# define WASM_CRANELIFT_DEBUG 0\n#endif\n\n#ifndef WASM_TRACE_DEBUG\n# define WASM_TRACE_DEBUG 0\n#endif\n\n#ifndef WASM_VALIDATOR_DEBUG\n# define WASM_VALIDATOR_DEBUG 0\n#endif\n\n#ifndef WEBDRIVER_DEBUG\n# define WEBDRIVER_DEBUG 0\n#endif\n\n#ifndef WEBDRIVER_ROUTE_DEBUG\n# define WEBDRIVER_ROUTE_DEBUG 0\n#endif\n\n#ifndef VIDEO_PRESENTATION_CHANNEL_DEBUG\n# define VIDEO_PRESENTATION_CHANNEL_DEBUG 0\n#endif\n\n#ifndef WEBGL_CONTEXT_DEBUG\n# define WEBGL_CONTEXT_DEBUG 0\n#endif\n\n#ifndef WEBVIEW_PROCESS_DEBUG\n# define WEBVIEW_PROCESS_DEBUG 0\n#endif\n\n#ifndef WEB_FETCH_DEBUG\n# define WEB_FETCH_DEBUG 0\n#endif\n\n#ifndef WEB_WORKER_DEBUG\n# define WEB_WORKER_DEBUG 0\n#endif\n\n#ifndef WEBP_DEBUG\n# define WEBP_DEBUG 0\n#endif\n\n#ifndef XML_PARSER_DEBUG\n# define XML_PARSER_DEBUG 0\n#endif\nLADYBIRD_EOF\n", ) # AK/Backtrace.h: find_package(Backtrace) is a HOST QUESTION, so the # genrule compiles a probe instead of baking in this machine's answer. diff --git a/tests/test_emit_cargo.py b/tests/test_emit_cargo.py index 272b576..e9e4830 100644 --- a/tests/test_emit_cargo.py +++ b/tests/test_emit_cargo.py @@ -176,6 +176,20 @@ def _fixture(): with open(os.path.join(d, "Cargo.toml"), "w") as f: f.write('[workspace]\nmembers = ["Libraries/LibWasm/Rust"]\n' 'exclude = ["Libraries/LibJS/Flap"]\n') + # The crate DIRECTORIES the member list names, because a source set derived + # from Cargo.toml is only derived if the directory has to exist -- that is + # exactly the property that made the hardcoded BytecodeDef/** glob outlive its + # directory. The LibWasm crate path-depends on nothing; LibWeb's does, on the + # excluded flapc workspace, which is how a path dep reaches OUTSIDE the + # workspace (libjs_rust build-depends on flapc that way). + for crate, manifest in ( + ("Libraries/LibWasm/Rust", '[package]\nname = "libwasm_cranelift"\n'), + ("Libraries/LibWeb/Rust", + '[package]\nname = "libweb_rust"\n\n[build-dependencies]\n' + 'flapc = { path = "../../LibJS/Flap" }\n')): + os.makedirs(os.path.join(d, crate), exist_ok=True) + with open(os.path.join(d, crate, "Cargo.toml"), "w") as f: + f.write(manifest) for lib, text in (("LibGfx", CMAKE_GFX), ("LibWeb", CMAKE_WEB), ("LibJS", CMAKE_JS), ("LibWasm", CMAKE_WASM)): os.makedirs(os.path.join(d, "Libraries", lib), exist_ok=True) @@ -197,15 +211,15 @@ def test_only_checksummed_packages_become_fetch_rules(): assert {n for n, _v in workspace} == {"libgfx_rust", "libweb_rust"} -def test_the_real_lock_splits_154_registry_from_13_workspace(): +def test_the_real_lock_splits_155_registry_from_11_workspace(): """Pins the counts against the checked-in generated file, which is ground truth neither this test nor the emitter produced (finding 25's rule).""" crates = _read("cargo_crates.bzl") - assert crates.count("http_archive(") == 154, crates.count("http_archive(") + assert crates.count("http_archive(") == 155, crates.count("http_archive(") # And every one of them names a crates.io URL, i.e. none is a workspace member # that slipped through. urls = re.findall(r"urls = \['([^']+)'\]", crates) - assert len(urls) == 154 + assert len(urls) == 155 assert all(u.startswith("https://static.crates.io/crates/") for u in urls) @@ -254,15 +268,15 @@ def test_repo_names_are_unique_per_crate_version(): """`-` and `.` are illegal in a repo name; the normalization must not collide.""" assert emit.repo_name("aho-corasick", "1.1.4") == "crate_aho_corasick_1_1_4" names = set(re.findall(r"name = '([^']+)'", _read("cargo_crates.bzl"))) - # 154 crates -> 154 distinct repo names, i.e. the normalization is injective + # 155 crates -> 155 distinct repo names, i.e. the normalization is injective # on the real input set. - assert len(names) == 154 + assert len(names) == 155 def test_a_crate_archive_declares_its_type_because_dot_crate_is_not_known(): """http_archive rejects a `.crate` suffix; without type= the fetch fails with a message about .zip/.tar.gz that says nothing about the cause.""" - assert _read("cargo_crates.bzl").count('type = "tgz"') == 154 + assert _read("cargo_crates.bzl").count('type = "tgz"') == 155 # --------------------------------------------------------------------------- @@ -277,24 +291,31 @@ def test_features_are_parsed_per_crate_including_the_variable_form(): assert specs["libweb_css_rust"]["features"] == ["allocator"] -def test_the_real_feature_set_is_three_of_ten_and_not_libgfx(): - """The spec for this work said libgfx_rust takes `allocator`. It does not -- - cargo fails outright ("the package 'libgfx_rust' does not contain this - feature"). Pinned against the generated index, so a future bump that changes - a feature has to change this number deliberately.""" +def test_the_real_feature_set_is_parsed_per_crate_and_moves_with_upstream(): + """Features are the ABI, so they are PARSED, and the parse is pinned here. + + The spec for this work said libgfx_rust takes `allocator`; at the commit this + overlay first targeted it did not even HAVE that feature (cargo: "the package + 'libgfx_rust' does not contain this feature"), and one repin later it does. + That is the whole argument for parsing rather than writing it down -- and it + is why this test asserts the parsed SET against the generated index rather + than a remembered number: a crate gaining, losing or merging a feature has to + change this assertion deliberately. + """ index = _read("cargo_index.bzl") specs = re.findall(r"'(\w+)': \{\n\s+\"manifest\": '[^']+',\n\s+" r"\"features\": (\[[^\]]*\])", index) feats = {name: eval(f) for name, f in specs} - assert len(feats) == 10, sorted(feats) + assert len(feats) == 8, sorted(feats) with_alloc = {n for n, f in feats.items() if f == ["allocator"]} - assert with_alloc == {"libregex_rust", "liburl_rust", "libunicode_rust", - "libweb_content_blocker_rust", "libweb_css_rust", - "libweb_layout_rust"}, sorted(with_alloc) - assert feats["libgfx_rust"] == [] + assert with_alloc == {"libgfx_rust", "libregex_rust", "libunicode_rust", + "liburl_rust", "libweb_content_blocker_rust"}, \ + sorted(with_alloc) assert feats["libjs_rust"] == [] assert feats["libtextcodec_rust"] == [] - assert feats["libweb_rust"] == [] + # The consolidated LibWeb crate carries a feature that is NOT `allocator`, so + # a parse that only ever looked for that one word would drop it silently. + assert feats["libweb_rust"] == ["style-recording"] # --------------------------------------------------------------------------- @@ -306,13 +327,24 @@ def test_ffi_header_lists_are_not_uniform(): hdrs = dict(re.findall(r"'(\w+)': \{\n(?:.*\n)*?\s+\"ffi_headers\": (\[[^\]]*\])", index)) hdrs = {k: eval(v) for k, v in hdrs.items()} - assert hdrs["libweb_css_rust"] == ["ComputedValuesRustFFI.h", "RustFFI.h", - "SelectorRustFFI.h", "StyleValueRustFFI.h"] - assert hdrs["libweb_layout_rust"] == ["Layout/TreeBuilderRustFFI.h"] assert hdrs["libweb_content_blocker_rust"] == ["ContentBlockerRustFFI.h"] assert hdrs["libgfx_rust"] == ["RustFFI.h"] - # 14 headers across 10 crates, which is the number in CMake's own build tree. - assert sum(len(v) for v in hdrs.values()) == 14 + # Upstream consolidated libweb_css_rust and libweb_layout_rust back INTO + # libweb_rust, so one crate now writes what three used to -- 12 files, two of + # them `.inc` rather than `.h`. An undeclared .inc is deleted by Bazel exactly + # like an undeclared header, so the suffix must not be what decides whether a + # generated file is declared. + assert hdrs["libweb_rust"] == [ + "ComputedValuesRustFFI.h", "HTML/Parser/RustFFI.h", + "HTMLTokenizerRustFFI.h", "Layout/LayoutRustFFI.h", + "Layout/TreeBuilderRustFFI.h", "RustFFI.h", "SelectorRustFFI.h", + "StyleEngineBridgeGenerated.h", "StyleEngineBridgeGenerated.inc", + "StyleEngineRustFFI.h", "StyleEngineStateFactsGenerated.inc", + "StyleValueRustFFI.h"] + assert [h for h in hdrs["libweb_rust"] if h.endswith(".inc")], \ + "the .inc outputs stopped being declared" + # 19 files across 8 crates, which is the number in CMake's own build tree. + assert sum(len(v) for v in hdrs.values()) == 19 def test_the_header_cmake_never_declares_is_still_declared_here(): @@ -384,8 +416,8 @@ def test_headers_are_prefixed_so_both_include_spellings_resolve(): different package.""" mod = _load(_fixture()) assert mod.FFI_PREFIX["liburl_rust"] == "LibURL" - assert mod.FFI_PREFIX["libweb_css_rust"] == "LibWeb" - assert mod.FFI_PREFIX["libweb_layout_rust"] == "LibWeb" + assert mod.FFI_PREFIX["libweb_rust"] == "LibWeb" + assert mod.FFI_PREFIX["libweb_content_blocker_rust"] == "LibWeb" # Every crate with observed headers has a prefix: a missing one would stage # the header at the include root and silently change its include spelling. assert set(mod.FFI_HEADERS_OBSERVED) <= set(mod.FFI_PREFIX) @@ -418,7 +450,7 @@ def test_the_index_key_carries_name_version_and_hash(): key, since a Bazel label carries none of them.""" index = _read("cargo_index.bzl") keys = re.findall(r"^ '([^']+)': '@crate_", index, re.M) - assert len(keys) == 154 + assert len(keys) == 155 for k in keys: name, version, sha = k.split(" ") assert re.fullmatch(r"[0-9a-f]{64}", sha), k @@ -559,15 +591,15 @@ def test_each_crate_links_its_own_archive_and_no_others(): assert "user_link_flags = [archive.path]" in impl, "must link exactly one archive" assert "additional_inputs" in impl, "the archive would not be in the sandbox" ring = _read("cargo_ring.bzl") - assert ring.count("cargo_crate(") == 10 - # One consumable target per crate that has something to consume: the 10 + assert ring.count("cargo_crate(") == 8 + # One consumable target per crate that has something to consume: the 8 # staticlib crates, plus libwasm_cranelift -- a `--bin` crate, so it has NO # archive at all and cargo_lib yields a headers-only CcInfo for it (the # executable is spawned at run time, and travels as `data`). That "archive # may be None" branch is the shape a binary crate needs, and it is why the - # count is 11 rather than 10. - assert ring.count("cargo_lib(") == 11, "one consumable target per crate" - assert ring.count("cargo_binary(") == 2 + # count is 9 rather than 8. + assert ring.count("cargo_lib(") == 9, "one consumable target per crate" + assert ring.count("cargo_binary(") == 4 assert "cargo_libs(" not in ring, "the shared link group is gone" assert "if archive == None:" in impl, "a --bin crate has no archive to link" @@ -594,9 +626,11 @@ def crates_of(txt, name, path="//:"): assert crates_of(root, "LibGfx") == ["libgfx_rust"] assert crates_of(root, "LibJS") == ["libjs_rust"] assert crates_of(root, "LibRegex") == ["libregex_rust"] + # Two, not four: upstream consolidated libweb_css_rust and libweb_layout_rust + # into libweb_rust, and the dep edges follow CMake's + # target_link_libraries(LibWeb PRIVATE libweb_rust libweb_content_blocker_rust). assert crates_of(libweb, "LibWeb") == [ - "libweb_content_blocker_rust", "libweb_css_rust", - "libweb_layout_rust", "libweb_rust", + "libweb_content_blocker_rust", "libweb_rust", ] @@ -657,8 +691,8 @@ def test_every_build_rust_binary_crate_is_in_the_ring_with_its_header(): a staticlib crate gets. """ mod = _load(_fixture()) - bins = {b["crate"]: b for b in mod.binary_specs()} - assert set(bins) == {"flapc", "libwasm_cranelift"}, sorted(bins) + bins = {b["bin"]: b for b in mod.binary_specs()} + assert set(bins) == {"flapc", "cranelift-compiler"}, sorted(bins) # flapc's call site has no FFI_OUTPUT_DIR, so it declares no header at all; # asking for one would fail the action (cargo never writes it). assert bins["flapc"]["ffi_headers"] == [] @@ -666,14 +700,21 @@ def test_every_build_rust_binary_crate_is_in_the_ring_with_its_header(): # libwasm_cranelift's does, so the header is a DECLARED output -- Bazel # deletes what nothing declares -- and the bare-include flag comes from # SCANNING the tree for the directory-less spelling, not from a list. - assert bins["libwasm_cranelift"]["ffi_headers"] == ["CraneliftFFI.h"] - assert bins["libwasm_cranelift"]["ffi_bare_include"] is True + assert bins["cranelift-compiler"]["ffi_headers"] == ["CraneliftFFI.h"] + assert bins["cranelift-compiler"]["ffi_bare_include"] is True # And the same two, present in the checked-in ring and consumed by the root - # package. Pinned by count so a third call site cannot appear unnoticed. + # package. Pinned by count so a call site cannot appear unnoticed. + # + # FOUR at the real commit, not two: upstream added `generate-libjs-bytecode` + # (a SECOND --bin out of the flapc crate, which replaced the deleted Python + # bytecode generator) and `style-replay` (a --bin out of libweb_rust, which + # also builds a staticlib). Keyed by BIN rather than by crate for exactly that + # reason -- two of the four share a crate with another target, so a dict keyed + # by crate name silently drops one of each pair. ring = _read("cargo_ring.bzl") root = _read("BUILD.bazel") - assert ring.count("cargo_binary(") == 2, ring.count("cargo_binary(") + assert ring.count("cargo_binary(") == 4, ring.count("cargo_binary(") for b in bins.values(): assert re.search(r"bin = ['\"]%s['\"]" % re.escape(b["bin"]), ring), b if not b["ffi_headers"]: @@ -722,16 +763,117 @@ def test_the_libweb_package_exports_its_crate_sources(): """ libweb = _read("Libraries/LibWeb/BUILD.bazel") assert 'name = "rust_crate_srcs"' in libweb - for sub in ("Rust/**", "CSS/Rust/**", "Layout/Rust/**", - "ContentBlocker/Rust/**", "HTML/Parser/Rust/**"): - assert '"%s"' % sub in libweb, sub - # The CSS data files libweb_css_rust's build script GENERATES Rust from. + for sub in ("ContentBlocker/Rust/**", "HTML/Parser/Rust/**", "Rust/**"): + assert "'%s'" % sub in libweb, sub + # The CSS data files libweb_rust's build script GENERATES Rust from. for data in ("CSS/Properties.json", "CSS/Keywords.json", "CSS/Enums.json", "HTML/TagNames.h", "HTML/Parser/Entities.json"): - assert '"%s"' % data in libweb, data + assert "'%s'" % data in libweb, data assert "//Libraries/LibWeb:rust_crate_srcs" in _read("cargo_ring.bzl") +def test_no_allow_empty_false_glob_names_a_directory_that_is_written_down(): + """The failure this repin existed to fix, as a test. + + Ulf's build died at LOADING time with + + Error in glob: glob pattern 'Libraries/LibJS/BytecodeDef/**' didn't match + anything, but allow_empty is set to False + + because upstream a32d9c9f deleted that directory and the pattern was a + hardcoded string in the emitter. Two properties make this the worst shape of + stale constant in the repo: + + * `allow_empty = False` is CORRECT and must stay. The alternative + (allow_empty = True) is what let three Build/full shim packages match + nothing for weeks and fail 1,600 actions later -- see + test_nothing_generated_still_reads_the_cmake_cargo_tree. + * A loading-time error has no target to blame, so nothing in the build graph + can report it and no per-target test can catch it. It is not "a crate + failed"; it is "the workspace does not load". + + So the guard is structural: every directory an allow_empty=False glob names + must be DERIVED from the tree (Cargo.toml's members closed over `path =` + deps), and the assertion is that the emitter's own derivation is what appears + in the generated file -- with no directory-shaped pattern left over that the + derivation did not produce. + """ + d = _fixture() + mod = _load(d) + # The fixture's Cargo.toml lists ONE member, whose manifest path-depends on + # the EXCLUDED flapc workspace; the closure therefore reaches outside the + # member list, which is how libjs_rust reaches flapc in the real tree. + assert mod.crate_dirs() == ["Libraries/LibWasm/Rust"], mod.crate_dirs() + assert mod.crate_src_globs() == mod.CRATE_SRC_ROOT_FILES + \ + ["Libraries/LibWasm/Rust/**"] + # A crate inside another Bazel package does NOT become a root-package glob -- + # it becomes a label, because glob() cannot cross a package boundary. + assert mod.crate_src_labels() == [] + # Delete the crate and the pattern goes away rather than outliving it: THE + # property. Under the old hardcoded list the pattern survived the directory, + # and Bazel then refused to load the workspace at all. + import shutil + shutil.rmtree(os.path.join(d, "Libraries/LibWasm/Rust")) + assert mod.crate_dirs() == [] + assert "Libraries/LibWasm/Rust/**" not in mod.crate_src_globs() + + # And structurally, in the emitter source: no module-level constant may hold a + # directory-shaped glob pattern. That is the form the bug took -- a `/**` + # string in a list literal, which nothing re-derives and no build-graph test + # can reach, because the glob fails before any target exists. + for rel in ("Meta/emit_cargo_bazel.py", "Meta/emit_libweb_bazel.py"): + src = _read(rel) + # Strip docstrings/comments: the block comments here NAME the deleted + # directories on purpose, to record what broke. + code = "\n".join(l for l in src.splitlines() + if not l.strip().startswith("#")) + code = re.sub(r'\'\'\'.*?\'\'\'', "", code, flags=re.S) + code = re.sub(r'""".*?"""', "", code, flags=re.S) + bad = [m.group(1) for m in re.finditer(r'"([A-Za-z][\w/.-]*/\*\*)"', code)] + bad += [m.group(1) for m in re.finditer(r"'([A-Za-z][\w/.-]*/\*\*)'", code)] + assert not bad, "%s hardcodes crate directories: %s" % (rel, bad) + + +def test_the_target_dir_exclusion_is_anchored_to_each_crate_root(): + """`**/target/**` is wrong, and it took down the build to prove it. + + Excluding cargo's output directory is right -- it is never an input, and a + developer who ran cargo by hand in the tree would otherwise get it globbed + into every crate's action inputs. Spelling it `**/target/**` is not: flapc has + a Rust MODULE at `src/target/` (the code generator's per-architecture + backends), so the unanchored pattern silently dropped 40 source files and + `//:generate-libjs-bytecode` failed to compile with + + error[E0583]: file not found for module `target` + + A glob exclusion is a pattern over PATHS and knows nothing about what a + directory means. "The directory cargo writes to" is `/target`, and + only the anchored form says that -- so this asserts every exclusion in the + generated files is rooted at a crate directory, not floating. + """ + mod = _load() + dirs = ["Libraries/LibJS/Flap", "Libraries/LibGfx/Rust"] + assert mod.crate_src_glob_excludes(dirs) == [ + "Libraries/LibJS/Flap/target/**", "Libraries/LibGfx/Rust/target/**"] + # Nothing in the tree may carry the unanchored form, in the emitters or in + # what they generated. + for rel in ("cargo_ring.bzl", "Libraries/LibWeb/BUILD.bazel", + "Meta/emit_cargo_bazel.py", "Meta/emit_libweb_bazel.py"): + txt = _read(rel) + code = "\n".join(l for l in txt.splitlines() + if not l.strip().startswith("#")) + assert "**/target/**" not in code, \ + "%s excludes target/ unanchored -- it eats src/target/" % rel + # Every exclusion that IS there is rooted at a directory the SAME file globs + # as a crate, so an exclusion cannot name a directory no crate owns. + for rel, in (("cargo_ring.bzl",), ("Libraries/LibWeb/BUILD.bazel",)): + txt = _read(rel) + globbed = set(re.findall(r"'([A-Za-z0-9_/.-]+)/\*\*'", txt)) + for m in re.finditer(r"'([A-Za-z0-9_/.-]+)/target/\*\*'", txt): + assert m.group(1) in globbed, \ + "%s excludes %s/target but globs no such crate" % (rel, m.group(1)) + + def test_every_extension_created_repo_is_named_in_module_bazel(): """bzlmod needs every repo an extension creates named in use_repo() to be visible, and the failure ("no such repository") lands far from its cause. 157 From 99240bd48b41ce65e0b91345361a4537f0d885ea Mon Sep 17 00:00:00 2001 From: Ulf Adams Date: Tue, 18 Aug 2026 19:29:27 +0000 Subject: [PATCH 36/59] Repin 71fb301a, part 2: the build failed four more times, all the same bug The glob failure Ulf reported is fixed and committed (88abd6fa). Running the full browser build past it produced four more failures, and a fifth defect that would never have produced one. Every single one was a CAPTURE: a value derived once on the machine that generated it, written down, and thereafter believed. Not one was found by reasoning about the repin; each was found by the next error message. 1. Libraries/LibWeb/generated_srcs.bzl said "AUTO-GENERATED by Meta/emit_libweb_bazel.py" and NO code path in that emitter wrote it. Its two lists were hand-maintained, so upstream's 5 new generated headers and 4 new .cpp were missing, and a GENERATED header included a GENERATED header the cc_library did not declare: Bindings/Window.h:17: fatal error: LibWeb/Bindings/WindowGlobalMixin.h: No such file or directory -- with the file sitting on disk next to the one that included it. A false AUTO-GENERATED line is worse than an honest hand-written file: it tells the next repin that re-running the emitter will refresh it. Both lists are now derived (the reference build's compile list; codegen.bzl's own outs) and emitted by --generated-srcs. Exactly 9 entries had drifted. 2. Fixing that exposed a second bug it had been HIDING. genrule_outputs() regexed the whole of codegen.bzl for anything path-shaped, so a genrule's srcs counted as outputs: four checked-in headers that generate_dom_tree.py READS were classified as generated. That list is the hdrs exclude=, so they were dropped from the source glob and re-added as labels nothing produces. It never broke the build only because the stale capture was consulted instead of the function's answer. One capture concealing a live bug is the strongest argument against keeping one. 3. QT_MAP was a 3-entry dict of the Qt modules Ladybird used when measured. Upstream made Qt6::Positioning REQUIRED; no key, so the dep fell through to UNKNOWN and //:ladybird failed with "QGeoPositionInfo: No such file or directory". CMake's Qt6 -> rules_qt's @qt//:Qt is a rename, so it is a rule now -- one that still returns None for a non-Qt name, because reporting UNKNOWN is right and inventing a label is not. 4. moc_headers() ended with `not h.endswith("GeolocationProviderQt.h")`, justified in its docstring by "Qt6::Positioning is not found in this configuration" -- a fact about the capturing machine, false at this pin. The condition is now whether the reference build compiles the sibling .cpp. Wrong in both directions: moc'ing a header CMake does not is a target only Bazel builds; skipping one it does is a missing vtable. The fifth produced no error at all and would not have. scripts/extract_cmake turned /usr/lib/libgio-2.0.so into the dep name `gio-2`, cutting the basename at the FIRST DOT instead of the extension. Invisible for libz.so and libQt6Widgets.so.6.10.2; wrong for any library with a dot in its name. It became visible only because the three glib deps upstream added (a new pkg_check_modules(GIO) in UI/Qt) arrived as unresolvable UNKNOWNs. Had the emitter guessed a label instead of reporting, -lgio-2 would have failed at link time far from the cause. An emitter that reports what it could not resolve converts a silent wrong answer into a loud missing one; that property found this bug, and it is the same property that makes allow_empty=False worth keeping. Also fixed, the specimen I was mid-investigation on: emit_vcpkg_bazel's capture REPLACES the static portfile parse (deliberate -- a portfile is a CMake program). That reasoning was checked once against three genuinely Windows-only rows and then frozen INTO THE MESSAGE, which printed every casualty as "vcpkg never asked for on this platform". At this pin vcpkg.json moves sdl3 to 3.2.28, the versions-db derivation gets it right, and the message reported the CORRECT current pin being discarded for a stale captured 3.4.12 as a Windows-only fetch. classify_static_only() now separates the two with no new input: a dropped row whose URL FAMILY the capture also has is the same project at another version, i.e. a stale capture. Re-capturing is filed (todo 37a5e4ee); vcpkg_distfiles.bzl still fetches 3.4.12. A replace-wins rule whose message asserts the reason for the replacement instead of checking it will eventually be confidently wrong. That sentence describes all six items above. Tests 256 -> 265, and each new guard was verified to BITE by temporarily reintroducing the bug it guards. New tests/test_emit_libweb.py (5) covers the derivation from both ends -- a new codegen output reaches hdrs, a genrule's own inputs do not, a generated .cpp CMake does not compile is not a src, and the checked-in file matches what the emitter emits. All six emitters reproduce their checked-in files byte-for-byte at this pin; apply_overlay.sh --verify reports 45/45 identical. LibWeb, the 5 services and all the Rust crates now build; //:ladybird's compile failures are gone and the remaining vcpkg exec-config rebuild is running. --- examples/ladybird/README.md | 91 +++++- examples/ladybird/workspace/BUILD.bazel | 6 +- .../Libraries/LibWeb/generated_srcs.bzl | 21 +- .../workspace/Meta/emit_build_bazel.py | 77 ++++- .../workspace/Meta/emit_libweb_bazel.py | 75 ++++- .../workspace/Meta/emit_vcpkg_bazel.py | 78 ++++- examples/ladybird/workspace/bazelrc.txt | 11 +- scripts/extract_cmake.py | 22 +- tests/test_emit_libweb.py | 284 ++++++++++++++++++ tests/test_emit_vcpkg.py | 75 +++++ tests/test_extractors.py | 41 +++ tests/test_qt_runtime.py | 72 ++++- 12 files changed, 808 insertions(+), 45 deletions(-) create mode 100644 tests/test_emit_libweb.py diff --git a/examples/ladybird/README.md b/examples/ladybird/README.md index 3dcbe8f..06b2e9b 100644 --- a/examples/ladybird/README.md +++ b/examples/ladybird/README.md @@ -312,7 +312,8 @@ an FFI header collision and an entire missing Rust target. See | `qt_runtime.bzl` | Qt's **runtime** half: a repo rule that stages the plugins of the SDK `@qt` itself names (read out of @qt's generated `qtconf.bzl`, so plugins and libraries cannot come from different Qts), the private libraries an SDK bundles beside Qt (derived from DT_NEEDED), a generated `qt.conf` pointing the binary at them, and the Qt >= 6.9 floor `UI/Qt/CMakeLists.txt` declares. Without it Qt `dlopen`ed the HOST's plugin into Bazel's Qt — finding 40 | | `BUILD.bazel` | Root package: 34 libraries, the 5 executables, Qt moc/rcc genrules. **Generated** by `Meta/emit_build_bazel.py` | | `codegen_root.bzl` | Non-LibWeb generator genrules (IPC endpoints, LibJS Bytecode/Op, HSTS table, WebGL replayer, TIFF tag tables, the two SPIR-V shader headers, and the chained `generate_interpreter_layout` → `flapc` interpreter assembly). **Generated** by `Meta/emit_root_codegen_bazel.py` | -| `Libraries/LibWeb/BUILD.bazel`, `generated_srcs.bzl` | LibWeb (~1,961 compile inputs). **Generated** by `Meta/emit_libweb_bazel.py` | +| `Libraries/LibWeb/BUILD.bazel` | LibWeb (~1,961 compile inputs). **Generated** by `Meta/emit_libweb_bazel.py` | +| `Libraries/LibWeb/generated_srcs.bzl` | Which of those inputs come from codegen: 692 `.cpp` (the reference build's compile list) + 693 headers (`codegen.bzl`'s `outs`). **Generated** by `Meta/emit_libweb_bazel.py --generated-srcs` — it claimed to be for a long time before it was (Known gaps 9) | | `Libraries/LibWeb/codegen.bzl` | LibWeb's 26 generator genrules + the bindings mega-genrule (663 `.idl` → 1,340 files). **Generated** by `Meta/emit_codegen_bazel.py` | | `Meta/emit_*.py` | The emitters. They read CMake's `build.ninja` + the File API codemodel and write the four generated files above | | `Meta/bazel_parity_harness.py` | Buckets every ninja `CUSTOM_COMMAND` as covered / excluded-with-reason / unhandled, re-runs the covered ones and byte-compares against CMake's tree. Non-zero unhandled is a failure | @@ -413,6 +414,10 @@ python3 Meta/emit_build_bazel.py > BUILD.bazel python3 Meta/emit_codegen_bazel.py Libraries/LibWeb > Libraries/LibWeb/codegen.bzl python3 Meta/emit_root_codegen_bazel.py > codegen_root.bzl python3 Meta/emit_libweb_bazel.py > Libraries/LibWeb/BUILD.bazel +# ...and the generated-source lists that BUILD.bazel loads. A SECOND output, +# because the BUILD file load()s it, so the two cannot share one stdout. It used +# to say AUTO-GENERATED while nothing generated it (see Known gaps 9). +python3 Meta/emit_libweb_bazel.py --generated-srcs > Libraries/LibWeb/generated_srcs.bzl # The vcpkg rules regenerate from the committed capture alone — no vcpkg, no network. python3 Meta/emit_vcpkg_bazel.py --assets Meta/vcpkg_assets.tsv --distfiles > vcpkg_distfiles.bzl python3 Meta/emit_vcpkg_bazel.py --assets Meta/vcpkg_assets.tsv --index > vcpkg_index.bzl @@ -665,10 +670,15 @@ Honest inventory of what stops this from being a clone-and-build. cross-compile, at which point the skew is silent. Bazel makes the two namespaces explicit and thereby makes the duplication visible; the right fix is a shared `.bzl` flag list or a custom toolchain, not more `--host_*` lines. -3. **`.bazelrc` still has host escapes:** `--action_env=CPLUS_INCLUDE_PATH=/usr/include/libdrm` - and `-L/usr/lib/x86_64-linux-gnu`, both because libdrm and Vulkan are - host-provided. (The `-L`/`-rpath` into `Build/full/vcpkg_installed/` are gone — - finding 33.) +3. **`.bazelrc` still has host escapes:** `--action_env=CPLUS_INCLUDE_PATH` naming + `/usr/include/libdrm` **and glib's two roots** (`/usr/include/glib-2.0`, + `/usr/lib/x86_64-linux-gnu/glib-2.0/include` — glib puts `glibconfig.h` under a + *libdir*, not an includedir), plus `-L/usr/lib/x86_64-linux-gnu`: libdrm, + Vulkan, glib and xkbcommon are all host-provided. glib joined at the 71fb301a + repin, when upstream added a `pkg_check_modules(GIO)` for UI/Qt's + `ExternalURLActivationToken`/`Handler`; xkbcommon arrives transitively with the + now-required `Qt6::GuiPrivate`. (The `-L`/`-rpath` into + `Build/full/vcpkg_installed/` are gone — finding 33.) **No absolute host path remains in the generated BUILD files.** Two did until recently, and both were the same mistake in different clothing: a value that is @@ -850,6 +860,77 @@ Honest inventory of what stops this from being a clone-and-build. which the parser only read on the `import_rust_crate` side: it would have built a different binary than CMake does and said nothing. + **Then, once loading was fixed, the build failed four more times — and every + one was the same bug in a different file.** The glob was only the first + *capture* to come due, i.e. a value derived once on the capturing machine, + written down, and thereafter believed. None of the four was found by reasoning + about the repin; each was found by the next error message. In the order the + build produced them: + + * `Libraries/LibWeb/generated_srcs.bzl` began with the line + `# AUTO-GENERATED by Meta/emit_libweb_bazel.py` and **no code path in that + emitter wrote it**. Its two lists (which of LibWeb's ~1,390 compile inputs + come from codegen) were hand-maintained, so upstream's five new generated + headers and four new `.cpp` were absent, and a *generated* header included a + *generated* header the `cc_library` did not declare: + `Bindings/Window.h:17: fatal error: LibWeb/Bindings/WindowGlobalMixin.h: No + such file or directory` — with the file present on disk beside the one that + included it. A false AUTO-GENERATED claim is worse than an honest + hand-written file, because it tells the next repin that re-running the + emitter will refresh it. Both lists are now derived (the reference build's + compile list; `codegen.bzl`'s own `outs`) and the file is emitted by + `--generated-srcs`. + * Fixing that exposed a **second bug it had been hiding**: `genrule_outputs()` + regexed the whole of `codegen.bzl` for anything path-shaped, so a genrule's + `srcs` counted as outputs too. Four *checked-in* headers that + `generate_dom_tree.py` reads (`HTML/TagNames.h` and friends) were classified + as generated — which is the `hdrs` `exclude=` list, so they were dropped from + the source glob and re-added as labels nothing produces. It never broke the + build only because the stale capture was consulted instead of the function's + answer. One capture concealing a live bug is the strongest argument against + keeping one. + * `QT_MAP` was a three-entry dict of the Qt modules Ladybird used when it was + measured. Upstream made `Qt6::Positioning` **required**; the dict had no key, + the dep fell through to `UNKNOWN`, and `//:ladybird` failed with + `QGeoPositionInfo: No such file or directory`. CMake's `Qt6` → + rules_qt's `@qt//:Qt` is a *rename*, so it is now a rule — and one + that still returns `None` for a non-Qt name, because reporting UNKNOWN is + right and inventing a label is not. + * `moc_headers()` ended with `not h.endswith("GeolocationProviderQt.h")`, + justified in its docstring by "Qt6::Positioning is not found in this + configuration" — a fact about the capturing machine, false at this pin. The + condition is now *does the reference build compile the sibling `.cpp`*, which + is in the model. Wrong in both directions: moc'ing a header CMake does not is + a target only Bazel builds; skipping one it does is a missing vtable at link. + + One of the five did not surface as an error at all, and would not have: the + extractor turned `/usr/lib/libgio-2.0.so` into the dep name `gio-2`, because it + cut the basename at the **first dot** instead of the extension. Invisible for + `libz.so` and `libQt6Widgets.so.6.10.2`; wrong for every library whose name + contains a dot. It only became visible because the three glib deps upstream + added arrived as unresolvable `UNKNOWN`s — had the emitter guessed a label + instead of reporting, `-lgio-2` would have failed at link time, far from here. + **An emitter that reports what it could not resolve converts a silent wrong + answer into a loud missing one**; that property found this bug, and it is the + same property that makes gap 9's `allow_empty = False` worth keeping. + + The vcpkg equivalent is still open, and is the cleanest specimen of the class. + `Meta/vcpkg_assets.tsv` is a capture that deliberately *replaces* the static + portfile parse (a portfile is a CMake program; the regex cannot see through its + platform branches). The reasoning was checked once, against three rows that + were genuinely Windows-only — and then frozen into the diagnostic, which + printed **every** casualty as an entry "vcpkg never asked for on this + platform". At this pin `vcpkg.json` moved `sdl3` to 3.2.28, the versions-db + derivation resolves that correctly, and the message reported the correct + current pin being discarded in favour of a stale captured 3.4.12 as a + Windows-only fetch. The checked-in `vcpkg_distfiles.bzl` fetches 3.4.12 to this + day. The classification needs no new input — a dropped row whose *URL family* + the capture also has is the same project at another version, i.e. a stale + capture — and the emitter now says so; re-capturing is filed. **A replace-wins + rule whose message asserts the reason for the replacement instead of checking + it will eventually be confidently wrong**, and that sentence describes every + item in this list. + 10. **Two upstreamable Ladybird fixes — down from four, and that is the point of a repin.** Two of the four are **fixed upstream** at this pin (`71fb301a`) and their patches are deleted, which is worth stating because a patch directory is a debt diff --git a/examples/ladybird/workspace/BUILD.bazel b/examples/ladybird/workspace/BUILD.bazel index 0acb139..dfddc5a 100644 --- a/examples/ladybird/workspace/BUILD.bazel +++ b/examples/ladybird/workspace/BUILD.bazel @@ -1791,7 +1791,7 @@ cc_binary( # or header filegroups are hand-maintained here. qt_cc_rcc compiles the .qrc. qt_cc_moc( name = "qt_moc", - hdrs = ["UI/Qt/Autocomplete.h", "UI/Qt/BookmarksBar.h", "UI/Qt/BrowserWindow.h", "UI/Qt/DevToolsBanner.h", "UI/Qt/EventLoopImplementationQtEventTarget.h", "UI/Qt/FindInPageWidget.h", "UI/Qt/LocationEdit.h", "UI/Qt/Settings.h", "UI/Qt/Tab.h", "UI/Qt/TabBar.h", "UI/Qt/WebContentView.h"], + hdrs = ["UI/Qt/Autocomplete.h", "UI/Qt/BookmarksBar.h", "UI/Qt/BrowserWindow.h", "UI/Qt/DevToolsBanner.h", "UI/Qt/EventLoopImplementationQtEventTarget.h", "UI/Qt/FindInPageWidget.h", "UI/Qt/GeolocationProviderQt.h", "UI/Qt/LocationEdit.h", "UI/Qt/Settings.h", "UI/Qt/Tab.h", "UI/Qt/TabBar.h", "UI/Qt/WebContentView.h"], ) qt_qrc( @@ -1824,7 +1824,6 @@ qt_conf( name = "qt_conf", ) # === ladybird (executable, 24 TU) === -# UNKNOWN deps: ['Qt6Positioning', 'gio-2', 'gobject-2', 'glib-2', 'xkbcommon'] cc_binary( name = 'ladybird', srcs = [ @@ -1857,7 +1856,7 @@ cc_binary( ], local_defines = ['LADYBIRD_QT_HAVE_POSITIONING=1', 'QT_CORE_LIB', 'QT_GUI_LIB', 'QT_NO_DEBUG', 'QT_POSITIONING_LIB', 'QT_WIDGETS_LIB'], copts = ['-pthread'], - linkopts = ['-lGLX', '-lOpenGL', '-lvulkan'], + linkopts = ['-lGLX', '-lOpenGL', '-lgio-2.0', '-lglib-2.0', '-lgobject-2.0', '-lvulkan', '-lxkbcommon'], data = [':qt_conf', ':qt_plugins'], deps = [ '//:AK', @@ -1881,6 +1880,7 @@ cc_binary( '//Meta/vcpkg:ssl', '@qt//:QtCore', '@qt//:QtGui', + '@qt//:QtPositioning', '@qt//:QtWidgets', '@qt_plugins//:runtime_libs', ], diff --git a/examples/ladybird/workspace/Libraries/LibWeb/generated_srcs.bzl b/examples/ladybird/workspace/Libraries/LibWeb/generated_srcs.bzl index d777c16..6cd311b 100644 --- a/examples/ladybird/workspace/Libraries/LibWeb/generated_srcs.bzl +++ b/examples/ladybird/workspace/Libraries/LibWeb/generated_srcs.bzl @@ -1,5 +1,7 @@ -# AUTO-GENERATED by Meta/emit_libweb_bazel.py — genrule outputs consumed by //Libraries/LibWeb:LibWeb -# Generated .cpp SRCS = the 688 the CMake reference compiles into LibWeb (GLFunctions.cpp is emitted but not compiled). +# AUTO-GENERATED by Meta/emit_libweb_bazel.py --generated-srcs. +# Do not edit: both lists are DERIVED (see generated_srcs_bzl there). +# Generated .cpp SRCS = the 692 the CMake reference compiles into LibWeb +# (a generated .cpp CMake does not compile is emitted but not listed). LIBWEB_GENERATED_SRCS = [ 'ARIA/AriaRoles.cpp', 'Bindings/ANGLEInstancedArrays.cpp', @@ -371,6 +373,7 @@ LIBWEB_GENERATED_SRCS = [ 'Bindings/NamedNodeMap.cpp', 'Bindings/NavigateEvent.cpp', 'Bindings/Navigation.cpp', + 'Bindings/NavigationActivation.cpp', 'Bindings/NavigationCurrentEntryChangeEvent.cpp', 'Bindings/NavigationDestination.cpp', 'Bindings/NavigationHistoryEntry.cpp', @@ -400,6 +403,7 @@ LIBWEB_GENERATED_SRCS = [ 'Bindings/OffscreenCanvasRenderingContext2D.cpp', 'Bindings/Origin.cpp', 'Bindings/OscillatorNode.cpp', + 'Bindings/PageSwapEvent.cpp', 'Bindings/PageTransitionEvent.cpp', 'Bindings/PannerNode.cpp', 'Bindings/ParentNode.cpp', @@ -558,6 +562,7 @@ LIBWEB_GENERATED_SRCS = [ 'Bindings/Storage.cpp', 'Bindings/StorageEvent.cpp', 'Bindings/StorageManager.cpp', + 'Bindings/StructuredSerializeBindings.cpp', 'Bindings/StylePropertyMap.cpp', 'Bindings/StylePropertyMapReadOnly.cpp', 'Bindings/StyleSheet.cpp', @@ -648,6 +653,7 @@ LIBWEB_GENERATED_SRCS = [ 'Bindings/WorkerLocation.cpp', 'Bindings/WorkerNavigator.cpp', 'Bindings/WorkletGlobalScope.cpp', + 'Bindings/WrapperFactory.cpp', 'Bindings/WritableStream.cpp', 'Bindings/WritableStreamDefaultController.cpp', 'Bindings/WritableStreamDefaultWriter.cpp', @@ -691,7 +697,7 @@ LIBWEB_GENERATED_SRCS = [ 'WebGL/WebGLContextProxy.cpp', ] -# All generated headers so resolves (includes=[".."]). +# All 693 generated headers, so resolves (includes=[".."]). LIBWEB_GENERATED_HDRS = [ 'ARIA/AriaRoles.h', 'Bindings/ANGLEInstancedArrays.h', @@ -861,6 +867,7 @@ LIBWEB_GENERATED_HDRS = [ 'Bindings/DecompressionStream.h', 'Bindings/DedicatedWorkerExposedInterfaces.h', 'Bindings/DedicatedWorkerGlobalScope.h', + 'Bindings/DedicatedWorkerGlobalScopeGlobalMixin.h', 'Bindings/DelayNode.h', 'Bindings/Document.h', 'Bindings/DocumentExtensions.h', @@ -1064,6 +1071,7 @@ LIBWEB_GENERATED_HDRS = [ 'Bindings/NamedNodeMap.h', 'Bindings/NavigateEvent.h', 'Bindings/Navigation.h', + 'Bindings/NavigationActivation.h', 'Bindings/NavigationCurrentEntryChangeEvent.h', 'Bindings/NavigationDestination.h', 'Bindings/NavigationHistoryEntry.h', @@ -1093,6 +1101,7 @@ LIBWEB_GENERATED_HDRS = [ 'Bindings/OffscreenCanvasRenderingContext2D.h', 'Bindings/Origin.h', 'Bindings/OscillatorNode.h', + 'Bindings/PageSwapEvent.h', 'Bindings/PageTransitionEvent.h', 'Bindings/PannerNode.h', 'Bindings/ParentNode.h', @@ -1232,6 +1241,7 @@ LIBWEB_GENERATED_HDRS = [ 'Bindings/SharedWorker.h', 'Bindings/SharedWorkerExposedInterfaces.h', 'Bindings/SharedWorkerGlobalScope.h', + 'Bindings/SharedWorkerGlobalScopeGlobalMixin.h', 'Bindings/Slottable.h', 'Bindings/SourceBuffer.h', 'Bindings/SourceBufferList.h', @@ -1333,6 +1343,7 @@ LIBWEB_GENERATED_HDRS = [ 'Bindings/Window.h', 'Bindings/WindowDeprecated.h', 'Bindings/WindowExposedInterfaces.h', + 'Bindings/WindowGlobalMixin.h', 'Bindings/WindowLocalStorage.h', 'Bindings/WindowOrWorkerGlobalScope.h', 'Bindings/WindowSessionStorage.h', @@ -1374,12 +1385,8 @@ LIBWEB_GENERATED_HDRS = [ 'CSS/PseudoElement.h', 'CSS/TransformFunctions.h', 'CSS/Units.h', - 'HTML/AttributeNames.h', 'HTML/MediaControlsDOM.h', 'HTML/Parser/NamedCharacterReferences.h', - 'HTML/TagNames.h', - 'SVG/AttributeNames.h', - 'SVG/TagNames.h', 'WebGL/GLFunctions.h', 'WebGL/WebGLCommands.h', 'WebGL/WebGLContextProxy.h', diff --git a/examples/ladybird/workspace/Meta/emit_build_bazel.py b/examples/ladybird/workspace/Meta/emit_build_bazel.py index 2a2c8b2..ce035f2 100644 --- a/examples/ladybird/workspace/Meta/emit_build_bazel.py +++ b/examples/ladybird/workspace/Meta/emit_build_bazel.py @@ -83,13 +83,38 @@ def rust_binary_crates(): "NDEBUG", } # System libs with no vcpkg .so (linked via linkopts on the final binary). -SYSTEM_LIBS = {"dl", "m", "pthread", "vulkan", "pulse"} -# Qt6 CMake target -> rules_qt label. -QT_MAP = { - "Qt6Core": "@qt//:QtCore", - "Qt6Gui": "@qt//:QtGui", - "Qt6Widgets": "@qt//:QtWidgets", -} +# glib/gio/gobject and xkbcommon joined at 71fb301a: upstream added a +# pkg_check_modules(GIO) for UI/Qt/ExternalURLActivationToken + ExternalURLHandler, +# and xkbcommon arrives transitively with the now-required Qt6::GuiPrivate. Their +# include roots (/usr/include/glib-2.0, /usr/lib/*/glib-2.0/include) are absolute, +# so like libdrm's they cannot be per-target copts and live in .bazelrc's +# CPLUS_INCLUDE_PATH -- see the host-tool preflight todo: this whole set is a +# configure-time host probe that neither build system derives on our side yet. +SYSTEM_LIBS = {"dl", "m", "pthread", "vulkan", "pulse", + "gio-2.0", "gobject-2.0", "glib-2.0", "xkbcommon"} + + +def qt_label(nm): + """CMake's Qt6 -> rules_qt's @qt//:Qt, by RULE not by table. + + This was a three-entry dict {Qt6Core, Qt6Gui, Qt6Widgets}, which is the same + shape as every other capture in this tree: correct for the pin that was + measured, silent about the next one. At 71fb301a upstream made + Qt6::Positioning REQUIRED (UI/Qt/GeolocationProviderQt.cpp), the dict had no + Qt6Positioning key, so the dep fell through to UNKNOWN and //:ladybird failed + to compile with + + UI/Qt/GeolocationProviderQt.h:11:10: fatal error: QGeoPositionInfo: + No such file or directory + + rules_qt names one cc_library per module in the discovered SDK, so the + mapping is a rename, and every module the SDK has is already a label. A + module the SDK does NOT have must still fail -- as an UNKNOWN dep naming it, + which is why this returns None rather than a label it has not checked. + """ + if not nm.startswith("Qt6") or len(nm) <= 3: + return None + return "@qt//:Qt" + nm[len("Qt6"):] # CMake's AUTOMOC/AUTORCC output, prebuilt under Build/full/_autogen. # Bazel runs moc/rcc itself (qt_cc_moc / qt_cc_rcc), so these are dropped from @@ -375,14 +400,15 @@ def dep_label(d, targets, so, ar): # path is a global -L in .bazelrc. # Qt6 is a real Bazel dep via rules_qt: its qt.local_repo discovers the host # SDK through qmake, so moc/rcc and the Qt cc_librarys all come from one SDK. - if nm in QT_MAP: + qt = qt_label(nm) + if qt: # Plus @qt_plugins//:runtime_libs -- the PRIVATE libraries an SDK ships # beside Qt (aqt bundles ICU 73) which rules_qt does not stage, and which # libQt6Core cannot find on its own because its RUNPATH $ORIGIN expands to # Bazel's solib dir. Making them link inputs is what removed the need for # LD_LIBRARY_PATH=/lib. Empty for a distro Qt. See qt_runtime.bzl / # finding 40. - return Dep(deps=[QT_MAP[nm], "@qt_plugins//:runtime_libs"]) + return Dep(deps=[qt, "@qt_plugins//:runtime_libs"]) SYS_MAP = {"GLX": "GLX", "OpenGL": "OpenGL"} if nm in SYS_MAP: return Dep(sys=[SYS_MAP[nm]]) @@ -695,23 +721,42 @@ def emit_qt_runtime(): print(QT_RUNTIME_BLOCK, end="") -def moc_headers(): +def moc_headers(exes=None): """UI/Qt headers with Q_OBJECT, i.e. the ones CMake's AUTOMOC would moc. - GeolocationProviderQt.h is excluded: it is only compiled when Qt6::Positioning - is found, which this configuration does not have (the model shows no - GeolocationProviderQt.cpp compile), so mocking it would be a target Bazel - builds and CMake does not. + Conditionally-compiled headers are filtered by whether the reference build + COMPILES the matching .cpp, not by name. This used to end with + + return [h for h in hdrs if not h.endswith("GeolocationProviderQt.h")] + + justified by "Qt6::Positioning is not found in this configuration" -- true + when written, false at 71fb301a, where upstream made Positioning required and + CMake compiles GeolocationProviderQt.cpp. A name-based exclusion cannot + notice that; asking the model can. Mocking a header CMake does not moc is a + target Bazel builds and CMake does not; NOT mocking one it does is a missing + vtable at link time, so the condition has to be the measurement. """ qt_dir = os.path.join(ROOT, "UI/Qt") + compiled = set() + if exes: + for t in exes.values(): + for a in t["actions"]: + if a["mnemonic"] == "CppCompile": + compiled |= {os.path.basename(i) for i in a["inputs"]} hdrs = [] for f in sorted(os.listdir(qt_dir)): if not f.endswith(".h"): continue if "Q_OBJECT" not in open(os.path.join(qt_dir, f), errors="ignore").read(): continue + # A Q_OBJECT header with a sibling .cpp is compiled-or-not with it. One + # with NO sibling .cpp (a header-only QObject) has nothing to measure, so + # it is moc'd -- AUTOMOC would. + cpp = f[:-2] + ".cpp" + if exes and os.path.exists(os.path.join(qt_dir, cpp)) and cpp not in compiled: + continue hdrs.append("UI/Qt/" + f) - return [h for h in hdrs if not h.endswith("GeolocationProviderQt.h")] + return hdrs def emit_target(name, targets, libs, exes, so, ar, header=True, body_only=False, @@ -887,7 +932,7 @@ def main(): print(AK_BLOCK_TAIL, end="") for name in ROOT_TARGETS: emit_target(name, targets, libs, exes, so, ar) - emit_qt_autogen(moc_headers()) + emit_qt_autogen(moc_headers(exes)) emit_qt_runtime() for name in QT_TARGETS: emit_target(name, targets, libs, exes, so, ar, diff --git a/examples/ladybird/workspace/Meta/emit_libweb_bazel.py b/examples/ladybird/workspace/Meta/emit_libweb_bazel.py index 1e82976..cffc715 100644 --- a/examples/ladybird/workspace/Meta/emit_libweb_bazel.py +++ b/examples/ladybird/workspace/Meta/emit_libweb_bazel.py @@ -128,8 +128,76 @@ def load(): def genrule_outputs(): + """Every file codegen.bzl declares as a genrule OUTPUT, package-relative. + + Parsed out of the `outs = [...]` lists specifically, not by scanning the + file for anything that looks like a path. The loose scan was the bug: a + genrule's `srcs` names checked-in headers too (gen_MediaControlsDOM reads + HTML/TagNames.h, HTML/AttributeNames.h, SVG/TagNames.h, + SVG/AttributeNames.h), so those four SOURCE headers were classified as + generated -- which puts them in the hdrs exclude= list, i.e. drops them + from the library's headers entirely and re-adds them as labels that no + genrule produces. It only ever worked because the stale checked-in list + was consulted instead of this function's answer. + """ bz = open(os.path.join(ROOT, "Libraries/LibWeb/codegen.bzl")).read() - return set(re.findall(r"'([A-Za-z0-9_/]+\.(?:cpp|h|cc))'", bz)) + # Drop comment lines: a commented-out out= entry is not an output. + bz = "\n".join(l for l in bz.splitlines() if not l.lstrip().startswith("#")) + outs = set() + for m in re.finditer(r"\bouts\s*=\s*(\[[^\]]*\])", bz, re.S): + outs |= set(re.findall(r"'([^']+)'", m.group(1))) + return outs + + +# The generated headers a consumer can #include. Everything a genrule declares +# as an output with a header extension: a header that is generated but absent +# from this list is not in the cc_library's hdrs, so any TU that includes it +# fails with "No such file or directory" pointing at the GENERATED file that +# does exist on disk -- exactly how the 71fb301a repin broke (upstream added +# Bindings/WindowGlobalMixin.h and four siblings; Bindings/Window.h includes +# it; LibWeb/DOM/Document.cpp failed). +HDR_EXTS = (".h", ".hh", ".hpp", ".inc", ".def") + + +def generated_srcs_bzl(t, gen_outs): + """Emit Libraries/LibWeb/generated_srcs.bzl -- both lists DERIVED. + + This file used to be a capture: it said AUTO-GENERATED at the top but no + emitter wrote it, so it froze one pin's answer and silently overrode the + derivable one. Both lists come from facts already in hand: + + * SRCS = the generated members of the CMake reference's compile list for + LibWeb (target_srcs under GEN_PREFIX), i.e. what CMake actually + compiles -- the same source of truth as the checked-in srcs. + * HDRS = every header-extension genrule output, i.e. what the codegen + actually produces. + + They are deliberately NOT the same set: a generated .cpp that CMake does + not compile (WebGL/GLFunctions.cpp) is emitted but absent from SRCS, and + HDRS is a superset of the headers that pair with a compiled .cpp. + """ + gen_srcs = sorted(s[len(GEN_PREFIX):] for s in target_srcs(t) + if s.startswith(GEN_PREFIX)) + for s in gen_srcs: + assert s in gen_outs, f"generated src not a genrule output: {s}" + gen_hdrs = sorted(o for o in gen_outs if o.endswith(HDR_EXTS)) + out = [ + "# AUTO-GENERATED by Meta/emit_libweb_bazel.py --generated-srcs.", + "# Do not edit: both lists are DERIVED (see generated_srcs_bzl there).", + "# Generated .cpp SRCS = the %d the CMake reference compiles into LibWeb" + % len(gen_srcs), + "# (a generated .cpp CMake does not compile is emitted but not listed).", + "LIBWEB_GENERATED_SRCS = [", + ] + out += [" %r," % s for s in gen_srcs] + out.append("]") + out.append("") + out.append("# All %d generated headers, so resolves" + " (includes=[\"..\"])." % len(gen_hdrs)) + out.append("LIBWEB_GENERATED_HDRS = [") + out += [" %r," % h for h in gen_hdrs] + out.append("]") + return "\n".join(out) + "\n" def vcpkg_available(): @@ -211,6 +279,11 @@ def lagom_to_target(name, targets): def main(): targets = load() + if "--generated-srcs" in sys.argv[1:]: + # The second output of this emitter. Separate because BUILD.bazel LOADS + # it, so the two cannot be one stdout stream. + sys.stdout.write(generated_srcs_bzl(targets["LibWeb"], genrule_outputs())) + return so, ar = vcpkg_available() libs = {n: t for n, t in targets.items() if t.get("role") == "production" diff --git a/examples/ladybird/workspace/Meta/emit_vcpkg_bazel.py b/examples/ladybird/workspace/Meta/emit_vcpkg_bazel.py index ca63a04..6fefa33 100644 --- a/examples/ladybird/workspace/Meta/emit_vcpkg_bazel.py +++ b/examples/ladybird/workspace/Meta/emit_vcpkg_bazel.py @@ -831,6 +831,55 @@ def collect(): return distfiles, externals, unresolved, unexpanded +def _url_family(url): + """The URL with its final path component removed: 'where this comes from'. + + Two distfiles of the SAME upstream project at different versions share a + family (.../libsdl-org/SDL/archive/release-3.2.28.tar.gz and + .../release-3.4.12.tar.gz); two unrelated projects do not. That is all the + identity needed to tell "the capture never fetched this" apart from "the + capture fetched a DIFFERENT VERSION of this". + """ + return url.rsplit("/", 1)[0] + + +def classify_static_only(distfiles, capture): + """Split derived-but-not-captured rows into (platform_only, stale_capture). + + The capture REPLACES the static parse (see main()), for good reasons. But the + diagnostic that reported the casualties asserted its reason instead of + checking it: it printed every one of them as an entry "vcpkg never asked for + on this platform", which was true of the three it was written against + (libiconv/pthreads4w/dirent, all behind `if(VCPKG_TARGET_IS_WINDOWS)`) and + became false the moment a pin MOVED. + + At Ladybird's 71fb301a, vcpkg.json pins sdl3 3.2.28 -- confirmed by the + reference build's vcpkg_installed/vcpkg/info/sdl3_3.2.28_*.list -- and the + versions-db derivation gets that right. The capture, taken at the previous + pin, has release-3.4.12, so the correct current pin was dropped in favour of + a stale one and the message called it a Windows-only fetch. The checked-in + vcpkg_distfiles.bzl fetches 3.4.12 to this day. + + A distfile whose URL FAMILY the capture also has is not platform-only: it is + the same upstream project at a different version, i.e. the capture is stale + with respect to the manifest. Returns: + + platform_only [sha] -- genuinely absent from the capture + stale_capture [(sha, capture_sha)] -- derived pin vs the captured one + """ + fams = {} + for sha, (url, _name, _port, _src) in capture.items(): + fams.setdefault(_url_family(url), []).append(sha) + platform_only, stale = [], [] + for sha in sorted(set(distfiles) - set(capture)): + same = fams.get(_url_family(distfiles[sha][0])) + if same: + stale.append((sha, sorted(same)[0])) + else: + platform_only.append(sha) + return platform_only, stale + + def observed_git_archives(downloads_dir, distfiles): """The archives in a completed run's downloads/ that the asset cache never saw. @@ -1078,17 +1127,34 @@ def main(): # that can break the build when an unrelated upstream URL rots, in # exchange for nothing. Meanwhile the union covered none of the 5 files # the capture genuinely misses -- the static parse misses those too. So - # the instrument wins outright, and the shortfall is REPORTED. - static_only = sorted(set(distfiles) - set(cap)) + # the instrument wins outright, and the shortfall is REPORTED -- but + # CLASSIFIED, not asserted. See classify_static_only: a dropped row whose + # URL family the capture also has is the same project at a different + # version, i.e. a STALE CAPTURE, and calling that "a fetch vcpkg never + # asked for on this platform" is how sdl3 3.2.28 got silently replaced by + # a captured 3.4.12 for a whole pin. + platform_only, stale = classify_static_only(distfiles, cap) sys.stderr.write("capture: %d distfiles (static parse had %d)\n" % (len(cap), len(distfiles))) - if static_only: + if platform_only: sys.stderr.write( - " dropping %d static-only entries vcpkg never asked for on this " - "platform:\n" % len(static_only)) - for sha in static_only: + " dropping %d entries vcpkg never asked for on this platform " + "(no capture row for that upstream at all):\n" % len(platform_only)) + for sha in platform_only: sys.stderr.write(" %-40s %s\n" % (distfiles[sha][1], distfiles[sha][2])) + if stale: + sys.stderr.write( + " STALE CAPTURE: %d port(s) where vcpkg.json's pin and the " + "capture disagree.\n" + " The capture wins (it is the only exact source), so the " + "EMITTED rules fetch the\n" + " captured version, NOT the pinned one. Re-run " + "Meta/vcpkg_capture_assets.sh.\n" % len(stale)) + for sha, cap_sha in stale: + sys.stderr.write(" %-12s pinned %s\n" % (distfiles[sha][2], + distfiles[sha][1])) + sys.stderr.write(" %-12s captured %s\n" % ("", cap[cap_sha][1])) distfiles, unexpanded = cap, [] # vcpkg's OWN tools are unioned in from their COMMITTED pin, not replaced # and not re-derived: they are a different class from port distfiles, and diff --git a/examples/ladybird/workspace/bazelrc.txt b/examples/ladybird/workspace/bazelrc.txt index 141e08e..5bc87a3 100644 --- a/examples/ladybird/workspace/bazelrc.txt +++ b/examples/ladybird/workspace/bazelrc.txt @@ -77,9 +77,12 @@ build --linkopt=-Wl,--allow-shlib-undefined build --linkopt=-Wl,--allow-multiple-definition # System library search path for find_package deps (Qt6, GLX/OpenGL). build --linkopt=-L/usr/lib/x86_64-linux-gnu -# System includes for find_package deps that are still host-provided (libdrm). -# Qt6 no longer appears here: it comes from rules_qt as a real Bazel dep. -build --action_env=CPLUS_INCLUDE_PATH=/usr/include/libdrm +# System includes for find_package/pkg_check_modules deps that are still +# host-provided: libdrm, and (since 71fb301a) glib -- upstream added a +# pkg_check_modules(GIO) for UI/Qt's ExternalURLActivationToken/Handler. glib puts +# glibconfig.h under a LIBDIR, not an includedir, so both roots are needed. +# Qt6 does not appear here: it comes from rules_qt as a real Bazel dep. +build --action_env=CPLUS_INCLUDE_PATH=/usr/include/libdrm:/usr/include/glib-2.0:/usr/lib/x86_64-linux-gnu/glib-2.0/include # --- exec (host) configuration --------------------------------------------- # Every flag above is a --cxxopt/--copt/--linkopt, which Bazel applies only to @@ -126,4 +129,4 @@ build --host_copt=-fPIC build --host_linkopt=-Wl,--allow-shlib-undefined build --host_linkopt=-Wl,--allow-multiple-definition build --host_linkopt=-L/usr/lib/x86_64-linux-gnu -build --host_action_env=CPLUS_INCLUDE_PATH=/usr/include/libdrm +build --host_action_env=CPLUS_INCLUDE_PATH=/usr/include/libdrm:/usr/include/glib-2.0:/usr/lib/x86_64-linux-gnu/glib-2.0/include diff --git a/scripts/extract_cmake.py b/scripts/extract_cmake.py index 35d61e6..97ac16c 100644 --- a/scripts/extract_cmake.py +++ b/scripts/extract_cmake.py @@ -182,6 +182,16 @@ def _library_identity(fragment: str) -> Optional[str]: '-lz' -> 'z'; '/usr/lib/libfoo.a' -> 'foo'; '-framework Cocoa' -> 'Cocoa'. Resolution to a Bazel label is the resolver adapter's job, not ours. + + The name is what a linker would take after -l, so the EXTENSION is stripped, + not everything after the first dot. That distinction is invisible for + libz.so and libQt6Widgets.so.6.10.2, and wrong for a library whose name + contains a dot: glib's soname is libgio-2.0.so, whose -l name is `gio-2.0`. + Truncating at the first dot produced `gio-2`, which names no library at all + -- so a resolver either fails to find it (Ladybird's 71fb301a pin: three + UNKNOWN deps, gio-2/gobject-2/glib-2, after upstream added a + pkg_check_modules(GIO)) or, worse, emits `-lgio-2` and the link fails a long + way from here. An abstract dep name that no tool can resolve is not abstract. """ frag = fragment.strip() if not frag: @@ -192,8 +202,16 @@ def _library_identity(fragment: str) -> Optional[str]: parts = frag.split() return parts[1] if len(parts) > 1 else None base = os.path.basename(frag) - if base.startswith("lib") and "." in base: - return base[3:].split(".")[0] + if not base.startswith("lib"): + return None + stem = base[3:] + # Cut at the extension, longest-first so `.dylib` is not read as `.d`. + for ext in (".dylib", ".so", ".tbd", ".a", ".lib"): + i = stem.find(ext + ".") # versioned: libfoo.so.1.2.3 + if i < 0 and stem.endswith(ext): + i = len(stem) - len(ext) # unversioned: libfoo.so + if i > 0: + return stem[:i] return None diff --git a/tests/test_emit_libweb.py b/tests/test_emit_libweb.py new file mode 100644 index 0000000..6671995 --- /dev/null +++ b/tests/test_emit_libweb.py @@ -0,0 +1,284 @@ +"""LibWeb's generated-source lists: derived, or a capture that outlives its pin? + +`Libraries/LibWeb/generated_srcs.bzl` holds the two lists that tell Bazel which +of LibWeb's ~1,390 compile inputs come out of the Ring 1b codegen instead of the +source tree. Its first line said "AUTO-GENERATED by Meta/emit_libweb_bazel.py" +and it was not: no code path in that emitter wrote it. It was a capture of one +pin's answer, hand-maintained, and the header at the top asserted otherwise -- +which is worse than an honest hand-written file, because it tells a reader (and +a repin) that re-running the emitter will refresh it. + +The 71fb301a repin is what cashed that in. Upstream added five generated headers +(`Bindings/WindowGlobalMixin.h` and four siblings) and four generated .cpp; the +capture named none of them, so `Bindings/Window.h` -- itself generated, and +present in bazel-out -- included a header the cc_library did not declare: + + bazel-out/.../Bindings/Window.h:17:10: fatal error: + LibWeb/Bindings/WindowGlobalMixin.h: No such file or directory + +Note the shape: the missing file EXISTED on disk next to the file that included +it. Only the declaration was stale, and only for the 9 entries that had moved +since the capture. Nothing in the build could have reported it earlier, because +a list of paths is consistent with itself. + +Both lists are derivable from facts the emitter already loads -- the CMake +reference's compile list, and codegen.bzl's own `outs` -- so these tests assert +they ARE derived, that the derivation distinguishes a genrule's outputs from its +inputs (the second bug, below), and that the checked-in file matches what the +emitter emits. +""" + +import importlib.util +import os +import re +import tempfile + +_WS = os.path.join(os.path.dirname(__file__), "..", "examples", "ladybird", + "workspace") +_EMIT = os.path.join(_WS, "Meta", "emit_libweb_bazel.py") +_GENERATED_SRCS = os.path.join(_WS, "Libraries", "LibWeb", "generated_srcs.bzl") + + +def _read(rel): + with open(os.path.join(_WS, rel)) as f: + return f.read() + + +def _load(root=None, model=None): + """Load the emitter module, optionally pointed at a fixture checkout.""" + # The reference build dir is an env var (a repin needs two side by side), and + # the fixture's is Build/full71 -- the same name the 71fb301a tree uses. + os.environ["LADYBIRD_BUILD_REL"] = "Build/full71" + os.environ["LADYBIRD_BUILD_DIR"] = os.path.join(root or ".", "Build/full71") + for var, val in (("LADYBIRD_ROOT", root), ("LADYBIRD_MODEL", model)): + if val: + os.environ[var] = val + else: + os.environ.pop(var, None) + spec = importlib.util.spec_from_file_location("emit_libweb_bazel", _EMIT) + mod = importlib.util.module_from_spec(spec) + spec.loader.exec_module(mod) + return mod + + +def _lists(text): + """Parse the two lists out of a generated_srcs.bzl body.""" + out = {} + for name in ("LIBWEB_GENERATED_SRCS", "LIBWEB_GENERATED_HDRS"): + m = re.search(re.escape(name) + r"\s*=\s*\[(.*?)\n\]", text, re.S) + assert m, "no %s in generated_srcs.bzl" % name + out[name] = re.findall(r"'([^']+)'", m.group(1)) + return out + + +# --------------------------------------------------------------------------- +# A fixture codegen.bzl. Tiny on purpose: it is the CONTRACT between the emitter +# and the genrules, not a copy of the real 1,390-entry file. +# +# It reproduces the two things that made the real file hard: a genrule whose +# `srcs` names checked-in headers (gen_dom_tree reads HTML/TagNames.h), and a +# generated .cpp that is emitted but NOT compiled by CMake (Extra.cpp, standing +# in for WebGL/GLFunctions.cpp). +# --------------------------------------------------------------------------- +CODEGEN_BZL = '''load("@rules_cc//cc:defs.bzl", "cc_library") + +def libweb_codegen(): + native.genrule( + name = 'gen_MediaControlsDOM', + srcs = ['HTML/MediaControls.html', 'HTML/TagNames.h', + 'HTML/AttributeNames.h', 'SVG/TagNames.h'], + outs = ['HTML/MediaControlsDOM.h', 'HTML/MediaControlsDOM.cpp'], + cmd = "true", + ) + native.genrule( + name = 'gen_Extra', + srcs = ['CSS/Thing.json'], + outs = ['WebGL/Extra.cpp', 'WebGL/Extra.h'], + cmd = "true", + ) + +def libweb_bindings_codegen(): + native.genrule( + name = 'gen_bindings', + outs = [ + 'Bindings/Window.cpp', + 'Bindings/Window.h', + # A header with no matching .cpp -- the mixin shape that broke the + # repin. Bindings/Window.h includes it. + 'Bindings/WindowGlobalMixin.h', + 'Bindings/Fragment.inc', + ], + cmd = "true", + ) +''' + + +def _fixture(compiled=("Bindings/Window.cpp", "HTML/MediaControlsDOM.cpp")): + """A checkout with a codegen.bzl and a CMake model naming LibWeb's TUs.""" + import json + d = tempfile.mkdtemp() + os.makedirs(os.path.join(d, "Libraries/LibWeb")) + with open(os.path.join(d, "Libraries/LibWeb/codegen.bzl"), "w") as f: + f.write(CODEGEN_BZL) + gen = "Build/full71/Libraries/LibWeb/" + inputs = ["Libraries/LibWeb/DOM/Document.cpp"] + [gen + c for c in compiled] + model = {"targets": {"LibWeb": { + "kind": "shared_library", "role": "production", "deps": [], + "actions": [{"mnemonic": "CppCompile", "inputs": inputs, + "arguments": ["-c"]}], + }}} + mpath = os.path.join(d, "model.json") + with open(mpath, "w") as f: + json.dump(model, f) + with open(os.path.join(d, ".bazelrc"), "w") as f: + f.write("build --cxxopt=-std=c++23\n") + # emit_libweb_bazel imports emit_build_bazel (for rust_dep_labels), which at + # IMPORT time parses the reference build's build.ninja for the root + # package's generated sources. An empty one is a valid parse of "no + # generators", and keeps this fixture about LibWeb rather than about ninja. + os.makedirs(os.path.join(d, "Build/full71")) + open(os.path.join(d, "Build/full71/build.ninja"), "w").close() + return d, mpath + + +def _emit(compiled=("Bindings/Window.cpp", "HTML/MediaControlsDOM.cpp")): + d, mpath = _fixture(compiled) + mod = _load(d, mpath) + targets = mod.load() + return mod, _lists(mod.generated_srcs_bzl(targets["LibWeb"], + mod.genrule_outputs())) + + +def test_a_generated_header_upstream_adds_reaches_the_hdrs_list(): + """THE regression: a new codegen output must appear without anyone editing. + + This is the 71fb301a failure reduced to its bones. The fixture's + `gen_bindings` produces `Bindings/WindowGlobalMixin.h`, which no generated + .cpp pairs with and which the compile list therefore never mentions -- so a + derivation driven by "headers next to compiled sources" would miss it, the + way the capture did. It is an OUTPUT of a genrule, and that is the only fact + that makes it includable. + """ + _, lists = _emit() + hdrs = lists["LIBWEB_GENERATED_HDRS"] + assert "Bindings/WindowGlobalMixin.h" in hdrs, hdrs + # And the same for a non-.h generated header extension: LibWeb's real + # codegen emits .inc files that TUs #include, and an extension allowlist + # that forgot one would drop it just as silently. + assert "Bindings/Fragment.inc" in hdrs, hdrs + # Now delete the genrule that makes it and the entry goes away rather than + # outliving the codegen -- the property the capture did not have in either + # direction. + d, mpath = _fixture() + body = CODEGEN_BZL.replace(" 'Bindings/WindowGlobalMixin.h',\n", "") + with open(os.path.join(d, "Libraries/LibWeb/codegen.bzl"), "w") as f: + f.write(body) + mod = _load(d, mpath) + hdrs2 = _lists(mod.generated_srcs_bzl(mod.load()["LibWeb"], + mod.genrule_outputs()))["LIBWEB_GENERATED_HDRS"] + assert "Bindings/WindowGlobalMixin.h" not in hdrs2, hdrs2 + + +def test_a_genrules_own_inputs_are_not_mistaken_for_its_outputs(): + """The second bug, found while fixing the first, and never yet triggered. + + `genrule_outputs()` used to regex the WHOLE codegen.bzl for anything that + looked like a path, so a genrule's `srcs` counted as outputs too. Four + checked-in SOURCE headers (HTML/TagNames.h, HTML/AttributeNames.h, + SVG/TagNames.h, SVG/AttributeNames.h -- read by generate_dom_tree.py) were + thereby classified as generated. That is not cosmetic: the generated list is + the cc_library's hdrs `exclude=`, so those four were dropped from the glob + over the source tree AND re-added as labels no genrule produces. It never + broke the build only because the stale capture was consulted instead of this + answer -- i.e. one bug hid the other, and fixing the capture exposed it. + """ + _, lists = _emit() + hdrs = lists["LIBWEB_GENERATED_HDRS"] + for src_hdr in ("HTML/TagNames.h", "HTML/AttributeNames.h", + "SVG/TagNames.h"): + assert src_hdr not in hdrs, \ + "%s is a genrule INPUT (a checked-in header), not an output" % src_hdr + assert "HTML/MediaControlsDOM.h" in hdrs, hdrs # a real out= of that rule + + +def test_a_generated_cpp_cmake_does_not_compile_is_not_a_src(): + """The lists are not one set, and conflating them would be a link error. + + LibWeb's codegen emits WebGL/GLFunctions.cpp, which the CMake reference does + not compile into LibWeb. SRCS is therefore the reference's compile list, not + "every generated .cpp" -- while HDRS is every generated header, because a + header costs nothing to declare and not declaring one is a hard failure. + The fixture's WebGL/Extra.cpp stands in for GLFunctions.cpp. + """ + _, lists = _emit() + assert "WebGL/Extra.cpp" not in lists["LIBWEB_GENERATED_SRCS"] + assert "WebGL/Extra.h" in lists["LIBWEB_GENERATED_HDRS"] + # ...and the pairing is not assumed the other way either: a compiled + # generated .cpp must be a declared output, or Bazel has no rule making it. + mod, _ = _emit() + try: + mod.generated_srcs_bzl( + mod.load()["LibWeb"], {"Bindings/Window.cpp"}) + except AssertionError as e: + assert "not a genrule output" in str(e), e + else: + raise AssertionError("a compiled src absent from outs= was accepted") + + +def test_the_checked_in_generated_srcs_matches_what_the_emitter_emits(): + """The file says AUTO-GENERATED; this is what makes that claim checkable. + + Without it "derived" is a comment. The header line must also name the flag + that reproduces the file, since the last one named a script that could not + write it. + """ + txt = open(_GENERATED_SRCS).read() + assert txt.startswith("# AUTO-GENERATED by Meta/emit_libweb_bazel.py"), \ + txt.splitlines()[0] + assert "--generated-srcs" in txt.splitlines()[0], \ + "the AUTO-GENERATED line must name the flag that regenerates the file" + # Structural: the emitter may not carry either list as data. A capture is + # exactly what a long run of path literals looks like, so count them -- the + # emitter legitimately WRITES the string "LIBWEB_GENERATED_SRCS = [" into + # its output, which is why the assignment itself cannot be the signal. + code = _read("Meta/emit_libweb_bazel.py") + code = re.sub(r'""".*?"""', "", code, flags=re.S) + code = "\n".join(l for l in code.splitlines() + if not l.strip().startswith("#")) + paths = re.findall(r"['\"]([A-Za-z][\w/-]*/[\w/-]+\.(?:cpp|h|inc))['\"]", code) + assert len(paths) < 10, \ + "the emitter carries %d generated-file paths as data: %s" % ( + len(paths), sorted(set(paths))[:12]) + # The two lists in the checked-in file are sorted and disjoint by extension, + # which is what lets a reviewer diff two pins' files usefully. + lists = _lists(txt) + for name, entries in lists.items(): + assert entries == sorted(entries), "%s is not sorted" % name + assert len(entries) == len(set(entries)), "%s has duplicates" % name + assert all(s.endswith(".cpp") for s in lists["LIBWEB_GENERATED_SRCS"]) + mod, _ = _emit() + assert all(h.endswith(tuple(mod.HDR_EXTS)) + for h in lists["LIBWEB_GENERATED_HDRS"]) + + +def test_every_generated_hdr_is_excluded_from_the_source_glob(): + """Why a wrong entry in HDRS is not merely a missing declaration. + + The cc_library globs `**/*.h` with `exclude = LIBWEB_GENERATED_HDRS`, so the + list decides, per header, whether the SOURCE tree's copy or the genrule's + output wins. A generated header missing from the list means a consumer can + compile against a stale checked-in copy instead of the generated one (silent); + a source header wrongly IN it means the header vanishes from the library + (loud, but only for whoever includes it). This pins the exclude= wiring so + the list keeps that meaning. + """ + build = _read("Libraries/LibWeb/BUILD.bazel") + assert "exclude = LIBWEB_GENERATED_HDRS" in build + assert "+ LIBWEB_GENERATED_HDRS," in build + # No generated header may ALSO be checked in under the same path: if one is, + # the exclude= silently changes which file the compiler sees, and that is a + # thing to know about rather than to resolve by glob precedence. + hdrs = _lists(open(_GENERATED_SRCS).read())["LIBWEB_GENERATED_HDRS"] + both = [h for h in hdrs + if os.path.exists(os.path.join(_WS, "Libraries/LibWeb", h))] + assert not both, "checked in AND generated: %s" % both diff --git a/tests/test_emit_vcpkg.py b/tests/test_emit_vcpkg.py index 8e3dbf5..68f8613 100644 --- a/tests/test_emit_vcpkg.py +++ b/tests/test_emit_vcpkg.py @@ -583,3 +583,78 @@ def test_the_preflight_reports_every_missing_tool_at_once(): assert len(apt_line) == 1, r.stderr assert "nasm" in apt_line[0] and "libtool" in apt_line[0] assert "perl" not in apt_line[0], "must not demand a tool that is present" + + +def test_a_moved_pin_is_reported_as_a_stale_capture_not_a_windows_only_fetch(): + """The capture wins -- so when it is WRONG, the message has to say which. + + Under --assets the capture REPLACES the static portfile parse, deliberately: + a portfile is a CMake program, so the static regex cannot see through its + platform branches, and every row the static parse had and the capture did not + turned out to be a Windows-only fetch. That reasoning was checked once, on + three rows (libiconv, pthreads4w, dirent, all behind + `if(VCPKG_TARGET_IS_WINDOWS)`), and then FROZEN INTO THE MESSAGE: the emitter + printed every casualty as an entry "vcpkg never asked for on this platform". + + Ladybird's 71fb301a repin is where that came due. vcpkg.json pins sdl3 + 3.2.28 (the reference build agrees: vcpkg_installed/vcpkg/info/ holds + sdl3_3.2.28_x64-linux-dynamic.list) and the versions-db derivation resolves + it correctly. The capture predates the repin and holds release-3.4.12, so the + RIGHT answer was discarded in favour of a stale one -- and the diagnostic + asserted a reason that happened to be false, which is why nobody looked. The + checked-in vcpkg_distfiles.bzl still fetches 3.4.12. + + The distinction is available without any new input: a dropped row whose URL + FAMILY the capture also has is the same upstream project at a different + version. That is a stale capture. A row whose family the capture has never + seen at all is genuinely a fetch this platform does not do. + """ + cap = { + "a" * 128: ("https://github.com/libsdl-org/SDL/archive/release-3.4.12.tar.gz", + "libsdl-org-SDL-release-3.4.12.tar.gz", "captured", "capture"), + "b" * 128: ("https://sqlite.org/2026/sqlite-autoconf-3530300.tar.gz", + "sqlite-autoconf-3530300.tar.gz", "captured", "capture"), + } + derived = { + # THE case: same project, the version vcpkg.json actually pins. + "c" * 128: ("https://github.com/libsdl-org/SDL/archive/release-3.2.28.tar.gz", + "libsdl-org-SDL-release-3.2.28.tar.gz", "sdl3", "versions-db"), + # Genuinely platform-only: nothing from this upstream is in the capture. + "d" * 128: ("https://ftpmirror.gnu.org/gnu/libiconv/libiconv-1.19.tar.gz", + "libiconv-1.19.tar.gz", "libiconv", "versions-db"), + "e" * 128: ("https://github.com/tronkko/dirent/archive/1.26.tar.gz", + "tronkko-dirent-1.26.tar.gz", "dirent", "versions-db"), + # A row the capture HAS (by sha) is not a casualty at all. + "b" * 128: ("https://sqlite.org/2026/sqlite-autoconf-3530300.tar.gz", + "sqlite-autoconf-3530300.tar.gz", "sqlite3", "versions-db"), + } + platform_only, stale = emit.classify_static_only(derived, cap) + assert stale == [("c" * 128, "a" * 128)], stale + assert sorted(platform_only) == ["d" * 128, "e" * 128], platform_only + + # And the emitter SAYS it, in those words, on the real tree's data. Run as a + # subprocess so this exercises the message and not just the classifier. + import subprocess + ws = os.path.join(os.path.dirname(__file__), "..", "examples", "ladybird", + "workspace") + tsv = os.path.join(ws, "Meta", "vcpkg_assets.tsv") + r = subprocess.run(["python3", _EMIT, "--assets", tsv, "--index"], + capture_output=True, text=True, + env=dict(os.environ, LADYBIRD_ROOT=tempfile.mkdtemp(), + VCPKG_ROOT=tempfile.mkdtemp())) + assert r.returncode == 0, r.stderr + # With no vcpkg there is no derivation to compare against, so neither + # category can appear -- the message must not be printed speculatively. + assert "STALE CAPTURE" not in r.stderr, r.stderr + assert "never asked for on this platform" not in r.stderr, r.stderr + + # Structurally: the old wording claimed the reason for EVERY dropped row. + # Whatever the emitter says about platform-only entries, it may only say it + # about rows that survived the classification. + with open(_EMIT) as f: + src = f.read() + body = src.split("def main(", 1)[1] + assert "classify_static_only(" in body, \ + "main() must classify the dropped rows, not assert a reason for them" + assert "static_only = sorted(set(distfiles) - set(cap))" not in body, \ + "the unclassified set-difference is back" diff --git a/tests/test_extractors.py b/tests/test_extractors.py index 03f3f7b..2fdf1d8 100644 --- a/tests/test_extractors.py +++ b/tests/test_extractors.py @@ -433,6 +433,47 @@ def test_ts_program_splits_into_per_file_tscompile_actions(): # role is set by the extractor and must not be demoted to AGGREGATE assert t.role.value == "production", t.role + +def test_a_library_name_keeps_the_dots_that_are_part_of_it(): + """`libgio-2.0.so` is `gio-2.0`, not `gio-2`. + + _library_identity turns a link fragment into the abstract dep name a resolver + looks up (the -l name). It used to cut at the FIRST dot -- fine for libz.so + and libQt6Widgets.so.6.10.2, and wrong for glib, whose sonames carry the API + version in the NAME: libgio-2.0.so became `gio-2`, which names no library on + any system. Ladybird's 71fb301a pin is where that surfaced: upstream added a + pkg_check_modules(GIO) to UI/Qt, and the three glib deps arrived at the + Bazel emitter as unresolvable UNKNOWNs. Had the emitter guessed instead of + reporting, the -l flag would have failed at LINK time, far from the cause. + + The rule is "strip the extension", so the cases that matter are: a dot in + the name, a version suffix after the extension, both at once, and .dylib not + being read as a truncated .d. + """ + f = extract_cmake._library_identity + cases = { + "/usr/lib/x86_64-linux-gnu/libgio-2.0.so": "gio-2.0", + "/usr/lib/x86_64-linux-gnu/libglib-2.0.so": "glib-2.0", + "/usr/lib/x86_64-linux-gnu/libgobject-2.0.so": "gobject-2.0", + # A dotted name AND a version suffix: both cuts have to be right. + "/usr/lib/libgio-2.0.so.0.8200.4": "gio-2.0", + # The cases that already worked, pinned so the fix cannot regress them. + "/usr/lib/x86_64-linux-gnu/libQt6Widgets.so.6.10.2": "Qt6Widgets", + "vcpkg_installed/x64-linux-dynamic/lib/libcpptrace.so.1.0.2": "cpptrace", + "/usr/lib/libvulkan.so": "vulkan", + "/usr/lib/libfoo.a": "foo", + "-lz": "z", + "-framework Cocoa": "Cocoa", + # macOS: .dylib must not be matched as ".d" + "ylib". + "/x/libbar.dylib": "bar", + "/x/libbar.2.dylib": "bar.2", + # Not a library at all -- None, so the caller does not invent a dep. + "notalib.so": None, + "": None, + } + for frag, want in cases.items(): + assert f(frag) == want, "%r -> %r, want %r" % (frag, f(frag), want) + # No `if __name__ == "__main__"` runner here on purpose. There used to be one in # every test file, and in this file it sat MID-FILE -- so four tests appended after # it were defined, never called, and the file still printed "6/6 passed". The third diff --git a/tests/test_qt_runtime.py b/tests/test_qt_runtime.py index 36d1852..31ef8c1 100644 --- a/tests/test_qt_runtime.py +++ b/tests/test_qt_runtime.py @@ -157,7 +157,7 @@ def test_the_emitter_generates_that_edge(): assert "QT_RUNTIME_BLOCK" in emitter and "emit_qt_runtime()" in emitter assert 'extra_data=[":qt_conf", ":qt_plugins"]' in emitter assert '"@qt_plugins//:runtime_libs"' in emitter, \ - "the private-libs dep must come from QT_MAP's resolution, not by hand" + "the private-libs dep must come from qt_label()'s resolution, not by hand" assert 'load(":qt_runtime.bzl", "qt_conf", "qt_plugin_tree")' in \ emitter.split("ALL_LOADS = ", 1)[1], "the generated file needs the load()" @@ -198,6 +198,76 @@ def test_repo_rule_is_local(): assert "local = True" in rule + +def test_a_qt_module_upstream_starts_requiring_resolves_without_an_edit(): + """The Qt mapping is a RULE, because the three-entry table went stale. + + CMake names Qt targets Qt6; rules_qt names one cc_library per module + of the discovered SDK as @qt//:Qt. That is a rename, and it was + written as a dict of the three modules Ladybird used when it was measured + (Core, Gui, Widgets). At 71fb301a upstream turned Qt6::Positioning from + OPTIONAL to REQUIRED for UI/Qt/GeolocationProviderQt.cpp; the dict had no + key, so the dep became an UNKNOWN and //:ladybird failed to compile with + "QGeoPositionInfo: No such file or directory". + + A table of the deps a project has today cannot express "and whatever it needs + tomorrow", so this asserts the derivation -- including that a NON-Qt name + still falls through (returning a label for it would put a nonexistent + @qt//:X in the build) . + """ + import importlib.util + os.environ["LADYBIRD_ROOT"] = _WS + spec = importlib.util.spec_from_file_location( + "_ebb_src", os.path.join(_WS, "Meta", "emit_build_bazel.py")) + src = _read("Meta/emit_build_bazel.py") + # Exec just the function, not the module: emit_build_bazel does real work at + # import time (it parses the reference build's build.ninja), and this is a + # pure string rule. + ns = {} + body = "def qt_label" + src.split("def qt_label", 1)[1].split("\ndef ", 1)[0] + exec(body, ns) + qt_label = ns["qt_label"] + assert qt_label("Qt6Core") == "@qt//:QtCore" + assert qt_label("Qt6Gui") == "@qt//:QtGui" + assert qt_label("Qt6Widgets") == "@qt//:QtWidgets" + # THE regression: the module nobody wrote down. + assert qt_label("Qt6Positioning") == "@qt//:QtPositioning" + # And modules Ladybird does not use today, to show the rule is not a longer list. + assert qt_label("Qt6Network") == "@qt//:QtNetwork" + assert qt_label("Qt6Multimedia") == "@qt//:QtMultimedia" + # Not Qt: must NOT resolve, so it is reported as UNKNOWN rather than becoming + # a label for a target that does not exist. + for other in ("xkbcommon", "glib-2.0", "OpenGL", "Qt6", "vulkan", ""): + assert qt_label(other) is None, other + # Structural: no dict may map Qt module names to labels again. + assert "QT_MAP" not in src, "the Qt table is back" + + +def test_the_moc_list_follows_what_cmake_compiles_not_a_name(): + """A conditionally-compiled Q_OBJECT header, decided by measurement. + + moc_headers() returns the UI/Qt headers CMake's AUTOMOC would moc. It ended + with a hardcoded `not h.endswith("GeolocationProviderQt.h")`, justified in a + docstring by "Qt6::Positioning is not found in this configuration" -- a fact + about the machine that captured it. At 71fb301a Positioning is required and + CMake compiles GeolocationProviderQt.cpp, so the exclusion silently dropped a + real moc target. Wrong in both directions: moc'ing a header CMake does not is + a target only Bazel builds; skipping one it does is a missing vtable at link. + + The condition is now "does the reference build compile the sibling .cpp", + which is in the model, so it cannot disagree with CMake. + """ + src = _read("Meta/emit_build_bazel.py") + fn = src.split("def moc_headers", 1)[1].split("\ndef ", 1)[0] + assert "GeolocationProviderQt" not in fn.split('"""', 2)[-1], "moc_headers filters by FILE NAME again" + assert "compiled" in fn and "CppCompile" in fn, "the filter must read the reference build's compile list" + # And the generated file reflects it: the header is moc'd at this pin. + build = _read("BUILD.bazel") + moc = build.split("qt_cc_moc(", 1)[1].split(")", 1)[0] + assert "UI/Qt/GeolocationProviderQt.h" in moc, moc + # ...paired with the dep that makes it compile, or the moc output does not build. + assert "'@qt//:QtPositioning'" in build + TESTS = [v for k, v in sorted(globals().items()) if k.startswith("test_")] if __name__ == "__main__": From 8b67a9b34b1e73f94c5d7163eef8c9d4376b70ef Mon Sep 17 00:00:00 2001 From: Ulf Adams Date: Tue, 18 Aug 2026 19:31:59 +0000 Subject: [PATCH 37/59] Guard the AUTO-GENERATED claim for all 15 files, not just the one that lied generated_srcs.bzl was not special. It carried a first line naming an emitter that had no code path writing it, and that false claim is what made the file safe to hand-edit and invisible at repin time: every reader, including me, was told that re-running the emitter would refresh it. The claim turns out to be checkable for the whole tree at once, and cheaply. A file whose header says a script generates it must name a script that exists, that parses every flag the header names, and that writes output at all. That last assertion is the one that fails for the original bug. Deliberately NOT a round-trip check -- proving each file reproduces needs a CMake reference build, which this suite does not have (the repin verified that by hand: all six emitters byte-identical). This is the weaker property that would still have caught it, because the emitter named there could not have produced the file under any argument. It also refuses to pass vacuously: fewer than 10 such files means the walk broke, not that the tree is clean. Verified to bite in both modes: a nonexistent script, and a flag the named script does not parse. --- tests/test_emit_libweb.py | 53 +++++++++++++++++++++++++++++++++++++++ 1 file changed, 53 insertions(+) diff --git a/tests/test_emit_libweb.py b/tests/test_emit_libweb.py index 6671995..0a404e5 100644 --- a/tests/test_emit_libweb.py +++ b/tests/test_emit_libweb.py @@ -282,3 +282,56 @@ def test_every_generated_hdr_is_excluded_from_the_source_glob(): both = [h for h in hdrs if os.path.exists(os.path.join(_WS, "Libraries/LibWeb", h))] assert not both, "checked in AND generated: %s" % both + + +def test_every_file_claiming_to_be_generated_names_a_command_that_makes_it(): + """The general form of this session's bug, applied to all 15 such files. + + `generated_srcs.bzl` was not special. It carried a first line saying + AUTO-GENERATED by an emitter, and the emitter had no code path that wrote + it -- so the claim was load-bearing misinformation for every reader and + every repin, and the cost was found only when the file's content went stale + under an upstream change. + + The claim is checkable, cheaply, for the whole tree at once. A file that says + a script generates it must name a script that EXISTS and, if the header names + a flag, the script must parse that flag. This is deliberately not a + round-trip test (that needs a CMake reference build, which the suite does not + have); it is the weaker check that would nonetheless have caught the bug -- + the emitter named there could not have produced the file under any argument. + """ + generated = [] + for dirpath, dirnames, filenames in os.walk(_WS): + dirnames[:] = [d for d in dirnames + if d not in ("__pycache__", "Build", ".git")] + for fn in filenames: + if not fn.endswith((".bzl", ".bazel", ".tsv")): + continue + path = os.path.join(dirpath, fn) + with open(path, errors="replace") as f: + head = "".join(f.readline() for _ in range(3)) + m = re.search(r'GENERATED by ([\w/]*?([\w-]+\.py))((?: --?[\w-]+)*)', + head) + if m: + generated.append((os.path.relpath(path, _WS), m.group(2), + m.group(3).split())) + # If this finds nothing the test is vacuous, which is the failure mode + # tests/run_all.py exists to reject -- so say so here too. + assert len(generated) >= 10, generated + + for rel, script, flags in generated: + cand = [os.path.join(_WS, "Meta", script), os.path.join(_WS, script)] + found = next((p for p in cand if os.path.exists(p)), None) + assert found, "%s says it is generated by %s, which does not exist" % ( + rel, script) + with open(found, errors="replace") as f: + src = f.read() + for flag in flags: + assert flag in src, \ + "%s says `%s %s` generates it, but that script never parses %s" \ + % (rel, script, " ".join(flags), flag) + # A generator that writes nothing cannot have produced the file. This is + # the assertion that fails for the original bug: emit_libweb_bazel.py had + # no branch writing generated_srcs.bzl at all. + assert re.search(r'\b(print|sys\.stdout\.write|\.write\()', src), \ + "%s names %s, which writes no output" % (rel, script) From 904ebfcb7f23432ffbf4a8bd48a17ee470578b65 Mon Sep 17 00:00:00 2001 From: Ulf Adams Date: Tue, 18 Aug 2026 19:34:54 +0000 Subject: [PATCH 38/59] The two emitters' SYSTEM_LIBS had already diverged; import instead of restate Found by auditing for the shape rather than waiting for failure number seven. GLOBAL_DEFINES and SYSTEM_LIBS describe the BUILD, not a target: a define is global because .bazelrc sets it for every TU, and a lib is a system lib because no vcpkg port supplies it. Both emitters kept their own copy, and they were already out of step -- this repin added glib/gio/gobject/xkbcommon to emit_build_bazel's SYSTEM_LIBS for upstream's new pkg_check_modules(GIO), and emit_libweb_bazel's copy still held the original four. Latent today, purely because LibWeb does not depend on glib. The failure it was holding is an UNKNOWN dep, i.e. a silently dropped link input -- and "happens not to" is the class of claim the rest of this repin was spent disproving. The second copy is now an import; the emitted BUILD file is byte-identical, which is what makes this a deduplication rather than a change. The test asserts all three levels: the source imports rather than restates, the two modules share the same objects at run time, and the shared set actually contains what the repin added (an import pointing at a stale copy would pass the first two). --- .../workspace/Meta/emit_libweb_bazel.py | 18 ++++++---- tests/test_emit_libweb.py | 34 +++++++++++++++++++ 2 files changed, 46 insertions(+), 6 deletions(-) diff --git a/examples/ladybird/workspace/Meta/emit_libweb_bazel.py b/examples/ladybird/workspace/Meta/emit_libweb_bazel.py index cffc715..2b8db4d 100644 --- a/examples/ladybird/workspace/Meta/emit_libweb_bazel.py +++ b/examples/ladybird/workspace/Meta/emit_libweb_bazel.py @@ -32,6 +32,7 @@ # about which crates those are (none of LibWeb's four today; the mechanism is here # because "none today" is a measurement, not an invariant). sys.path.insert(0, os.path.dirname(os.path.abspath(__file__))) +import emit_build_bazel from emit_build_bazel import rust_dep_labels # The crate directories inside THIS package, and the non-Rust build-script inputs # that live here, both derived by emit_cargo_bazel from Cargo.toml and the @@ -42,12 +43,17 @@ # report it. The root package's glob over the same crates is one list; so is this. import emit_cargo_bazel -GLOBAL_DEFINES = { - "USE_VULKAN=1", "ENABLE_COMPILETIME_FORMAT_CHECK", "USE_FONTCONFIG=1", - "_FORTIFY_SOURCE=3", "USE_VULKAN_DMABUF_IMAGES=1", "_FILE_OFFSET_BITS=64", - "NDEBUG", -} -SYSTEM_LIBS = {"dl", "m", "pthread", "vulkan"} +# IMPORTED, not restated. Both of these describe the BUILD, not this target: a +# define is global because .bazelrc sets it for every TU, and a lib is a "system +# lib" because no vcpkg port provides it. Two copies of one global fact is the +# shape that broke this repin four times over -- and these two had ALREADY +# diverged: the 71fb301a repin added glib/gio/gobject/xkbcommon to +# emit_build_bazel's SYSTEM_LIBS (upstream's new pkg_check_modules(GIO)) and this +# copy kept the old four. Harmless only because LibWeb happens not to depend on +# glib; "happens not to" is not a property worth relying on, and the failure it +# would produce is an UNKNOWN dep that silently drops a link input. +GLOBAL_DEFINES = emit_build_bazel.GLOBAL_DEFINES +SYSTEM_LIBS = emit_build_bazel.SYSTEM_LIBS # LibWeb exports extern "C" FFI that the prebuilt Rust archive consumes and also # consumes it back (a static-archive <-> static-archive cycle GNU ld cannot diff --git a/tests/test_emit_libweb.py b/tests/test_emit_libweb.py index 0a404e5..9e3fdae 100644 --- a/tests/test_emit_libweb.py +++ b/tests/test_emit_libweb.py @@ -335,3 +335,37 @@ def test_every_file_claiming_to_be_generated_names_a_command_that_makes_it(): # no branch writing generated_srcs.bzl at all. assert re.search(r'\b(print|sys\.stdout\.write|\.write\()', src), \ "%s names %s, which writes no output" % (rel, script) + + +def test_the_global_facts_are_imported_from_one_place_not_restated(): + """Two copies of one global fact, and they had already diverged. + + GLOBAL_DEFINES and SYSTEM_LIBS describe the BUILD, not a target: a define is + global because .bazelrc sets it for every TU, and a lib is a "system lib" + because no vcpkg port supplies it. Both emitters carried their own copy, and + by the time this was noticed they disagreed -- the 71fb301a repin added + glib/gio/gobject/xkbcommon to emit_build_bazel's SYSTEM_LIBS (upstream's new + pkg_check_modules(GIO) in UI/Qt) and this file's copy still had the old four. + + Latent, because LibWeb happens not to depend on glib. The failure it was + holding is an UNKNOWN dep -- i.e. a dropped link input -- and "happens not + to" is exactly the kind of claim this whole session was spent disproving. So + the second copy is an import, and this asserts it stays one. + """ + src = _read("Meta/emit_libweb_bazel.py") + for name in ("GLOBAL_DEFINES", "SYSTEM_LIBS"): + m = re.search(r"^%s = (.*)$" % name, src, re.M) + assert m, name + assert m.group(1).startswith("emit_build_bazel."), \ + "%s is restated in emit_libweb_bazel instead of imported: %s" % ( + name, m.group(1)) + # And they really are the same objects at run time, not just textually. + mod, _ = _emit() + import sys + ebb = sys.modules["emit_build_bazel"] + assert mod.SYSTEM_LIBS is ebb.SYSTEM_LIBS + assert mod.GLOBAL_DEFINES is ebb.GLOBAL_DEFINES + # The union has to contain what the repin added, or the import is pointing at + # a copy that is itself stale. + for lib in ("gio-2.0", "gobject-2.0", "glib-2.0", "xkbcommon"): + assert lib in mod.SYSTEM_LIBS, lib From 7c2d1ce4ddba57d7bc0e5f34d3a92674719b8042 Mon Sep 17 00:00:00 2001 From: Ulf Adams Date: Tue, 18 Aug 2026 20:57:17 +0000 Subject: [PATCH 39/59] Report the absolute include roots the emitter drops, after hand-copying 3 of 6 I caused this one, immediately after fixing four hand-copied facts, which makes it the most useful item in the repin. An absolute include root cannot be a per-target copt (Bazel rejects a path outside the execution root even as -isystem), so emit_build_bazel dropped it and .bazelrc carried it globally as CPLUS_INCLUDE_PATH. The drop was a bare `continue`: a root CMake compiles with and .bazelrc lacks produced NO output at all. Having just deleted four written-down facts, I then read glib's include roots off the model BY EYE and wrote down three of them. The build failed ~3,800 actions later: UI/Qt/ExternalURLHandler.cpp:19:10: fatal error: gio/gdesktopappinfo.h: No such file or directory There are six, not three: gio-unix-2.0 is a separate root from glib-2.0 (the UNIX-only GIO headers), and blkid/libmount/sysprof-6 arrive transitively through glib's own pkg-config. Re-running the emitter with the check in place named all four missing roots in one second. The emitter now records what it drops and reports the shortfall against .bazelrc's CPLUS_INCLUDE_PATH. Deliberately a WARNING, not a failure: which host escapes are acceptable is a judgement (README gap 3), and Qt's roots are correctly absent because rules_qt carries them on the dep edge -- so those are exempt, because a warning that cries wolf about Qt every run is a warning that hides a real one. What is not a judgement is whether the emitter says anything. The generated BUILD.bazel is byte-identical: this adds a diagnostic, not a change. The test asserts the emitter records and reports, that the dep-carried roots are exempt, that all six roots are present, and that the target and exec copies of the variable are equal (a skew there is finding 26's failure mode: a genrule tool that cannot compile). --- .../workspace/Meta/emit_build_bazel.py | 74 ++++++++++++++++++- examples/ladybird/workspace/bazelrc.txt | 19 +++-- tests/test_qt_runtime.py | 53 +++++++++++++ 3 files changed, 137 insertions(+), 9 deletions(-) diff --git a/examples/ladybird/workspace/Meta/emit_build_bazel.py b/examples/ladybird/workspace/Meta/emit_build_bazel.py index ce035f2..825e20c 100644 --- a/examples/ladybird/workspace/Meta/emit_build_bazel.py +++ b/examples/ladybird/workspace/Meta/emit_build_bazel.py @@ -135,6 +135,60 @@ def global_flags(): GLOBAL_FLAGS = global_flags() + +def bazelrc_host_includes(): + """The include roots .bazelrc hands every action via CPLUS_INCLUDE_PATH. + + These are the host escapes README gap 3 inventories. Read here so the + emitter can CHECK the ones it drops against them rather than dropping them + silently. + """ + rc = open(os.path.join(ROOT, ".bazelrc")).read() + out = set() + for m in re.finditer(r"--(?:host_)?action_env=CPLUS_INCLUDE_PATH=(\S+)", rc): + out |= {p for p in m.group(1).split(":") if p} + return out + + +BAZELRC_HOST_INCLUDES = bazelrc_host_includes() +# Absolute include roots the model uses that .bazelrc does NOT carry. Collected +# during emission and reported at the end: a per-target message would repeat +# once per TU, and the useful unit is "which roots does this configuration need +# that the build does not provide". +MISSING_HOST_INCLUDES = set() +# Roots that arrive with a dep edge instead of an env var, so their absence from +# CPLUS_INCLUDE_PATH is correct rather than missing. Qt comes from rules_qt as a +# real Bazel dep (@qt//:Qt carries its own include dirs) and the vcpkg +# tree rides on //Meta/vcpkg: as system_includes -- both deliberately NOT +# global -isystem any more, which is what finding 33 bought. +HOST_INCLUDE_EXEMPT = ("/usr/include/x86_64-linux-gnu/qt6", + "/usr/lib/x86_64-linux-gnu/qt6", + "vcpkg_installed") + + +def record_host_include(path): + if any(x in path for x in HOST_INCLUDE_EXEMPT): + return + if path not in BAZELRC_HOST_INCLUDES: + MISSING_HOST_INCLUDES.add(path) + + +def report_host_includes(): + """Print the shortfall to stderr. Not fatal: see record_host_include.""" + if not MISSING_HOST_INCLUDES: + return + sys.stderr.write( + "WARNING: %d absolute include root(s) CMake compiles with are absent " + "from\n .bazelrc's CPLUS_INCLUDE_PATH, so a TU that needs one " + "will fail with\n 'No such file or directory' far from here:\n" + % len(MISSING_HOST_INCLUDES)) + for p in sorted(MISSING_HOST_INCLUDES): + sys.stderr.write(" %s\n" % p) + sys.stderr.write( + " Add them there (they cannot be per-target copts: Bazel " + "rejects a path\n outside the execution root), or give the dep " + "a Bazel label that carries them.\n") + # Flags CMake puts on a target that must NOT be copied into a Bazel copt. # # -fPIE is the one that matters, and it is a crash, not a nit. CMake adds it to @@ -838,10 +892,20 @@ def emit_target(name, targets, libs, exes, so, ar, header=True, body_only=False, copt_toks = list(flags) for i in incs: if i.startswith("/"): - # System include roots (/usr Qt6, libdrm) can't be per-target - # copts: Bazel rejects a path outside the execution root even - # with -isystem. They live as global -isystem in .bazelrc - # (mirroring CMake's find_package include dirs). + # System include roots (/usr Qt6, libdrm, glib) can't be + # per-target copts: Bazel rejects a path outside the execution + # root even with -isystem. They live in .bazelrc's + # CPLUS_INCLUDE_PATH (mirroring CMake's find_package dirs) -- + # and CHECKED against it, not just skipped. A silent `continue` + # here means a root CMake uses and .bazelrc lacks produces no + # output at all: I transcribed three of glib's four roots by + # hand and the build failed 3,800 actions later on + # `gio/gdesktopappinfo.h: No such file or directory`, which is + # the same hand-copied-fact bug as everything else in this + # repin. Reported, not fixed automatically: which roots are + # acceptable host escapes is a judgement (see README gap 3), + # so the emitter's job is to refuse to hide the difference. + record_host_include(i) continue if i.startswith(BUILD_REL + "Libraries/"): # CMake's per-library FFI dir (FFI_OUTPUT_DIR defaults to the @@ -941,6 +1005,8 @@ def main(): # same sense as the cranelift-compiler binary LibWasm spawns. extra_data=[":qt_conf", ":qt_plugins"]) print(VCPKG_TAIL, end="") + # Last, so it is the final thing on stderr rather than buried in the middle. + report_host_includes() def lib_hdr_glob(name, srcs): diff --git a/examples/ladybird/workspace/bazelrc.txt b/examples/ladybird/workspace/bazelrc.txt index 5bc87a3..45d5d78 100644 --- a/examples/ladybird/workspace/bazelrc.txt +++ b/examples/ladybird/workspace/bazelrc.txt @@ -79,10 +79,19 @@ build --linkopt=-Wl,--allow-multiple-definition build --linkopt=-L/usr/lib/x86_64-linux-gnu # System includes for find_package/pkg_check_modules deps that are still # host-provided: libdrm, and (since 71fb301a) glib -- upstream added a -# pkg_check_modules(GIO) for UI/Qt's ExternalURLActivationToken/Handler. glib puts -# glibconfig.h under a LIBDIR, not an includedir, so both roots are needed. -# Qt6 does not appear here: it comes from rules_qt as a real Bazel dep. -build --action_env=CPLUS_INCLUDE_PATH=/usr/include/libdrm:/usr/include/glib-2.0:/usr/lib/x86_64-linux-gnu/glib-2.0/include +# pkg_check_modules(GIO) for UI/Qt's ExternalURLActivationToken/Handler. +# +# SIX roots, not the three that look obvious, and the list is DERIVED: it is +# every absolute -I/-isystem in the CMake reference that no Bazel dep edge +# carries, and emit_build_bazel.py now WARNS when one is missing here instead of +# dropping it silently. I first transcribed three of them by hand and the build +# failed 3,800 actions later on `gio/gdesktopappinfo.h: No such file or +# directory`. glibconfig.h lives under a LIBDIR rather than an includedir; +# gio-unix-2.0 is a separate root from glib-2.0 (the UNIX-only GIO headers); +# blkid/libmount/sysprof-6 arrive transitively through glib's own pkg-config. +# Qt6 does not appear here: it comes from rules_qt as a real Bazel dep, and the +# vcpkg include dirs ride on //Meta/vcpkg: (finding 33). +build --action_env=CPLUS_INCLUDE_PATH=/usr/include/libdrm:/usr/include/glib-2.0:/usr/lib/x86_64-linux-gnu/glib-2.0/include:/usr/include/gio-unix-2.0:/usr/include/blkid:/usr/include/libmount:/usr/include/sysprof-6 # --- exec (host) configuration --------------------------------------------- # Every flag above is a --cxxopt/--copt/--linkopt, which Bazel applies only to @@ -129,4 +138,4 @@ build --host_copt=-fPIC build --host_linkopt=-Wl,--allow-shlib-undefined build --host_linkopt=-Wl,--allow-multiple-definition build --host_linkopt=-L/usr/lib/x86_64-linux-gnu -build --host_action_env=CPLUS_INCLUDE_PATH=/usr/include/libdrm:/usr/include/glib-2.0:/usr/lib/x86_64-linux-gnu/glib-2.0/include +build --host_action_env=CPLUS_INCLUDE_PATH=/usr/include/libdrm:/usr/include/glib-2.0:/usr/lib/x86_64-linux-gnu/glib-2.0/include:/usr/include/gio-unix-2.0:/usr/include/blkid:/usr/include/libmount:/usr/include/sysprof-6 diff --git a/tests/test_qt_runtime.py b/tests/test_qt_runtime.py index 31ef8c1..e78b9aa 100644 --- a/tests/test_qt_runtime.py +++ b/tests/test_qt_runtime.py @@ -268,6 +268,59 @@ def test_the_moc_list_follows_what_cmake_compiles_not_a_name(): # ...paired with the dep that makes it compile, or the moc output does not build. assert "'@qt//:QtPositioning'" in build + +def test_a_dropped_absolute_include_root_is_reported_not_silently_skipped(): + """The emitter must not hide the difference between its input and .bazelrc. + + An absolute include root (/usr/include/glib-2.0) cannot be a per-target + copt -- Bazel rejects a path outside the execution root even as -isystem -- + so the emitter drops it and .bazelrc carries it globally as + CPLUS_INCLUDE_PATH. The drop was a bare `continue`, which means a root CMake + compiles with and .bazelrc lacks produces NO output at all. + + That is how the last failure of this repin happened, and I caused it: having + just fixed four hand-copied facts, I hand-copied three of glib's roots out of + the model and missed the rest. The build failed ~3,800 actions later on + `UI/Qt/ExternalURLHandler.cpp:19: fatal error: gio/gdesktopappinfo.h: No such + file or directory` -- gio-unix-2.0 is a separate root from glib-2.0, and + blkid/libmount/sysprof-6 arrive transitively through glib's pkg-config. + Re-running the emitter with the check in place named all four at once. + + Deliberately a WARNING, not a failure: which host escapes are acceptable is a + judgement (README gap 3), and Qt's roots are correctly absent because + rules_qt carries them on the dep edge. What is not a judgement is whether the + emitter says anything at all. + """ + src = _read("Meta/emit_build_bazel.py") + # The bare skip is gone. + assert "record_host_include(i)" in src, \ + "absolute include roots are dropped without being recorded" + assert "def report_host_includes" in src and "report_host_includes()" in \ + src.split("def report_host_includes", 1)[1], \ + "the shortfall is collected but never reported" + # The roots that ride on a dep edge must be exempt, or the warning cries wolf + # about Qt on every run and gets ignored -- which is how a real one hides. + exempt = src.split("HOST_INCLUDE_EXEMPT = ", 1)[1].split(")", 1)[0] + assert "qt6" in exempt and "vcpkg_installed" in exempt, exempt + + # And .bazelrc actually carries the six roots this configuration needs, glib's + # non-obvious ones included. This is the assertion that would have failed + # while the build was broken. + rc = _read("bazelrc.txt") + paths = set() + for m in re.finditer(r"CPLUS_INCLUDE_PATH=(\S+)", rc): + paths |= set(m.group(1).split(":")) + for root in ("/usr/include/libdrm", "/usr/include/glib-2.0", + "/usr/lib/x86_64-linux-gnu/glib-2.0/include", + "/usr/include/gio-unix-2.0", "/usr/include/blkid", + "/usr/include/libmount", "/usr/include/sysprof-6"): + assert root in paths, "%s missing from CPLUS_INCLUDE_PATH" % root + # Target and exec configs must agree: a root in one and not the other is the + # finding-26 skew, and here it would mean a genrule tool that cannot compile. + envs = re.findall(r"--(?:host_)?action_env=CPLUS_INCLUDE_PATH=(\S+)", rc) + assert len(envs) == 2, envs + assert envs[0] == envs[1], "target and exec CPLUS_INCLUDE_PATH differ" + TESTS = [v for k, v in sorted(globals().items()) if k.startswith("test_")] if __name__ == "__main__": From 1421041c015ca18d7bbe7119db802ab2431ed1f5 Mon Sep 17 00:00:00 2001 From: Ulf Adams Date: Wed, 19 Aug 2026 10:09:10 +0000 Subject: [PATCH 40/59] The glib include roots need --repo_env too, not just --action_env --action_env made gcc FIND glibconfig.h; Bazel then rejected the compile anyway: Compiling UI/Qt/ExternalURLHandler.cpp failed: absolute path inclusion(s) found in rule '//:ladybird': ... '/usr/lib/.../glib-2.0/include/glibconfig.h' and it is right to: a header outside the execution root that no toolchain declares is an undeclared input, so Bazel cannot know when it changed. rules_cc derives cxx_builtin_include_directories by running `cc -E -v` in a REPOSITORY rule, and CPLUS_INCLUDE_PATH shows up in that output -- so passing the same value as --repo_env is what DECLARES these roots to the toolchain, rather than sneaking them past the check. Only glibconfig.h needed it; every other root is under /usr/include, which gcc already reports as builtin. The reasoning is in the file, because the two lines look redundant and the next person to tidy one away gets an error that names neither. --- examples/ladybird/workspace/bazelrc.txt | 16 ++++++++++++++++ 1 file changed, 16 insertions(+) diff --git a/examples/ladybird/workspace/bazelrc.txt b/examples/ladybird/workspace/bazelrc.txt index 45d5d78..5ad71df 100644 --- a/examples/ladybird/workspace/bazelrc.txt +++ b/examples/ladybird/workspace/bazelrc.txt @@ -92,6 +92,22 @@ build --linkopt=-L/usr/lib/x86_64-linux-gnu # Qt6 does not appear here: it comes from rules_qt as a real Bazel dep, and the # vcpkg include dirs ride on //Meta/vcpkg: (finding 33). build --action_env=CPLUS_INCLUDE_PATH=/usr/include/libdrm:/usr/include/glib-2.0:/usr/lib/x86_64-linux-gnu/glib-2.0/include:/usr/include/gio-unix-2.0:/usr/include/blkid:/usr/include/libmount:/usr/include/sysprof-6 +# ...and the SAME value as a --repo_env, which is not redundant. --action_env +# makes the compiler FIND the header; Bazel then rejects it anyway: +# +# Compiling UI/Qt/ExternalURLHandler.cpp failed: absolute path inclusion(s) +# found in rule '//:ladybird': ... includes the following non-builtin files +# with absolute paths: '/usr/lib/.../glib-2.0/include/glibconfig.h' +# +# -- a correctness check, and a fair one: a header outside the execution root +# that no toolchain declares is an undeclared input, so Bazel cannot know when +# it changed. rules_cc computes the builtin include dirs by running `cc -E -v` +# in a REPOSITORY rule, and CPLUS_INCLUDE_PATH appears in that output -- so +# passing it as --repo_env is what puts these roots in +# cxx_builtin_include_directories, i.e. declares them to the toolchain rather +# than sneaking past the check. Only glibconfig.h needed it: every other root +# here is under /usr/include, which gcc already reports as builtin. +build --repo_env=CPLUS_INCLUDE_PATH=/usr/include/libdrm:/usr/include/glib-2.0:/usr/lib/x86_64-linux-gnu/glib-2.0/include:/usr/include/gio-unix-2.0:/usr/include/blkid:/usr/include/libmount:/usr/include/sysprof-6 # --- exec (host) configuration --------------------------------------------- # Every flag above is a --cxxopt/--copt/--linkopt, which Bazel applies only to From f15a410ff22f46195d2cc7ccde17910db284537a Mon Sep 17 00:00:00 2001 From: Ulf Adams Date: Wed, 19 Aug 2026 10:09:32 +0000 Subject: [PATCH 41/59] The browser did not render because my own run recipe shadowed the build The Bazel build at 71fb301a was green and the browser still would not load a page: ~14,000 lines of Failed to parse IPC message: Local endpoint error: Can't read past the end of the stream memory Peer endpoint error: Endpoint magic number mismatch, not my message! Everything aimed at the code generator said the build was fine, and was right: all 20 generated *Endpoint.h are byte-identical to CMake's, the wire magic 7a36a5ff IS WebContentServerEndpoint (correct endpoint, payload won't decode), the staged binaries' md5s matched the fresh ones, no stale processes. I spent real time on a C++17/C++23 ABI-skew theory because aquery reports both flags -- and it is a red herring: EVERY CppCompile has -std=c++17 then -std=c++23, gcc takes the last, 2694/2695 actions identical. The artifact that ran was not built by the build. `readlink /proc//exe` on the live WebContent -- the one measurement that asks what is EXECUTING rather than what is built -- pointed at bazel-out/k8-fastbuild/libexec/, an August 7 binary from the PREVIOUS pin. I had staged into bazel-out/k8-fastbuild/bin/libexec/; the UI looks one level up. Upstream inserted IPC messages between the pins, so every message id past the insertion point shifted by one: right endpoint, wrong layout, and 8000 unparseable messages per run. The staging step was never needed AND was actively harmful. get_paths_for_helper_process() searches /libexec/ BEFORE /bin/, and under Bazel the services are already siblings of ladybird in bazel-bin -- so the copy I made only ever shadowed the real thing: a cache with no invalidation, in a directory Bazel does not own, that no `bazel clean` clears and no rebuild refreshes. So the fix is to delete the step, not refresh it. Verified by removal: with no libexec/ anywhere, --headless=text and --headless=layout-tree (119 lines) and about:version are byte-identical to the CMake reference at 71fb301a. tests/test_run_recipe.py (4 tests) guards the recipe, because the recipe is the interface Ulf runs: no copy into libexec by any spelling, the recipe must clear a libexec/ an older one left, the lookup order and the IPC symptom must stay written down (or "don't stage" reads as a style preference and gets re-added), and the same-class `share`-symlink-into-CMake's-tree trap. Each verified to bite by reintroducing its bug. The general shape, and the reason this belongs in the case study: a convenience copy of a build output, placed where the program looks first, fails as a MISCOMPILE -- the one direction byte-parity checks on generated code cannot see. --- examples/ladybird/README.md | 69 ++++++++++++++++---- tests/test_run_recipe.py | 124 ++++++++++++++++++++++++++++++++++++ 2 files changed, 179 insertions(+), 14 deletions(-) create mode 100644 tests/test_run_recipe.py diff --git a/examples/ladybird/README.md b/examples/ladybird/README.md index 06b2e9b..110b1d1 100644 --- a/examples/ladybird/README.md +++ b/examples/ladybird/README.md @@ -444,27 +444,41 @@ for s in 0 1 7 42; do python3 Meta/bazel_parity_harness.py --seed $s; done The emitters locate the checkout via `$LADYBIRD_ROOT`, defaulting to the parent of `Meta/` — so they work from any checkout path. -Running the UI currently needs manual staging, which is itself a gap (see -below): +Running the UI currently needs the RESOURCE root staged by hand, which is itself +a gap (see below). It needs nothing else: the staging step for the helper +binaries was removed, because it was worse than the gap it papered over. Every path below is **derived, not written down**, because two of them used to be stated as literal `k8-fastbuild` paths and one was simply wrong: the vcpkg tree is built in the **exec** configuration (`vcpkg_lib` pins `cfg = "exec"`), so it is -under `k8-fastbuild-exec` and `bazel-bin/vcpkg_installed` does not exist. The two -staging roots are also NOT the same directory, which is easy to get backwards: -helper binaries go in `/libexec`, but the resource root is -`/../share/Lagom` -- `LibWebView/Utilities.cpp`'s `find_prefix()` takes the -PARENT of the binary's directory and appends `share/Lagom`. Ask the build for all -three rather than spelling any of them out: +under `k8-fastbuild-exec` and `bazel-bin/vcpkg_installed` does not exist. The +resource root is `/../share/Lagom`, not `/share/Lagom` -- +`LibWebView/Utilities.cpp`'s `find_prefix()` takes the PARENT of the binary's +directory and appends `share/Lagom`. Ask the build for the paths rather than +spelling any of them out: + +**Do NOT stage the services into a `libexec/` directory.** They are already +siblings of `ladybird` in `bazel-bin`, and `LibWebView/Utilities.cpp`'s +`get_paths_for_helper_process()` searches `/libexec/` **before** +`/bin/` -- so a `libexec` copy WINS over the fresh build, and only +the copy is ever run. This +recipe used to stage one, and after the 71fb301a repin the stale August copies +in `bazel-out/k8-fastbuild/libexec/` (from the OLD pin) were what actually +launched: the new UI talked to old-pin services, whose IPC message IDs had +shifted, and every message failed to parse with `Endpoint magic number +mismatch` / `Can't read past the end of the stream memory` while all 20 +generated `*Endpoint.h` were byte-identical to CMake's. Verified by removal: with +no `libexec/` at all, `--headless=text` and `--headless=layout-tree` are +byte-identical to the CMake reference. A staging step that shadows the build +output is a cache with no invalidation. ```sh export XDG_RUNTIME_DIR=/tmp/xdg-lb && mkdir -p $XDG_RUNTIME_DIR && chmod 700 $XDG_RUNTIME_DIR BIN=$(bazel info bazel-bin) # target config: where ladybird is -mkdir -p "$BIN/libexec" -for b in WebContent RequestServer ImageDecoder Compositor WebWorker; do cp -f bazel-bin/$b "$BIN/libexec/"; done -# cranelift-compiler is found as a sibling of the spawning binary (Ladybird's own -# lookup chain), so it needs no path baked in -- only to be there. -cp -f bazel-bin/cranelift-compiler "$BIN/libexec/" +# The services + cranelift-compiler need NO staging: they are siblings of +# ladybird in $BIN, which is the second entry in Ladybird's own lookup chain. +# If a libexec/ exists from an older recipe, it shadows them -- delete it. +rm -rf "$BIN/libexec" "$(dirname "$BIN")/libexec" # The resource root, assembled from the CLONE and from Bazel's own vcpkg tree. # This line used to read `ln -sfn "$PWD/Build/full/share/Lagom" ...` -- i.e. the # recipe for running the Bazel build pointed at CMake's build tree, a seventh @@ -475,6 +489,10 @@ cp -f bazel-bin/cranelift-compiler "$BIN/libexec/" # $(dirname $BIN), not $BIN: find_prefix() resolves the resource root against the # PARENT of the directory holding the binary. L="$(dirname "$BIN")/share/Lagom" +# rm -f the SHARE dir too, not just Lagom: an older recipe left `share` as a +# SYMLINK into CMake's Build/full, so after that tree moved, `mkdir -p` failed +# with a "File exists"-shaped error on a path that does not exist. +rm -f "$(dirname "$BIN")/share" chmod -R u+w "$L" 2>/dev/null; rm -rf "$L"; mkdir -p "$L/ladybird/pdfjs/web" cp -r Base/res/. "$L/" # ASK for the tree rather than guessing its configuration directory. @@ -490,7 +508,9 @@ mv "$L/ladybird/pdfjs/pdfjs-ladybird-transport.mjs" "$L/ladybird/pdfjs/web/" The assembled tree is byte-identical to CMake's `Build/full/share/Lagom` (`diff -rq`, 0 differences), and with it a fresh clone renders `--headless=text` and `--headless=layout-tree` byte-identically to the CMake reference on all -three test pages. +three test pages. Re-verified at 71fb301a against `Build/full71` with no +`libexec/` staged at all: `--headless=text` identical, `--headless=layout-tree` +identical (119 lines), `about:version` identical. ## Known gaps @@ -728,6 +748,27 @@ Honest inventory of what stops this from being a clone-and-build. `bazel info` / `bazel cquery --output=files`. Qt's plugins USED to belong on this list and no longer do: they are `data` of `//:ladybird` (`qt_runtime.bzl`), which is what that "real Bazel setup" looks like for one of the three things. + + **The helper binaries no longer belong on this list either, and removing them + fixed a bug the staging step CAUSED.** The block used to `cp` the five + services into `$BIN/libexec/` — which was never needed (they are already + siblings of `ladybird` in `bazel-bin`, the second entry in + `get_paths_for_helper_process()`'s chain) and was actively wrong, because + `/libexec/` is searched **first**. That made the staged copy + shadow the build output: a directory Bazel does not own, that no `bazel clean` + removes and no rebuild refreshes. After the 71fb301a repin the fresh UI kept + launching WebContent binaries staged six weeks earlier from the *previous* + pin; upstream had inserted IPC messages, so message ids past the insertion + point had shifted and every message failed to decode — ~14,000 lines of + `Endpoint magic number mismatch` / `Can't read past the end of the stream + memory` from a build whose 20 generated `*Endpoint.h` were byte-identical to + CMake's. **Every check aimed at the code generator correctly said the build + was fine; the artifact that ran was not built by the build.** Fixed by + deleting the step and clearing any `libexec/` an older recipe left, guarded by + `tests/test_run_recipe.py`. The general shape: a convenience copy of a build + output, placed where the program looks first, is a cache with no + invalidation — and it fails as a *miscompile*, in the one direction the + parity checks cannot see. 6. **`rules_qt` is not on the BCR.** `MODULE.bazel` uses an `archive_override` pointing at kklochkov/rules_qt v2.0.1's release tarball (stock upstream, no patches). The BCR's `rules_qt` module is Vertexwahn's unrelated `rules_qt6`. diff --git a/tests/test_run_recipe.py b/tests/test_run_recipe.py new file mode 100644 index 0000000..7ff92ed --- /dev/null +++ b/tests/test_run_recipe.py @@ -0,0 +1,124 @@ +"""The documented run recipe must not stage a COPY of the build's own outputs. + +Ladybird's UI finds its helper processes through +`WebView::get_paths_for_helper_process()`, which searches, in order: + + /libexec/ <-- first + /bin/ + / + ./ + +Under Bazel the services are already siblings of `ladybird` in `bazel-bin`, i.e. +the build output IS on that chain. The README's run recipe nevertheless used to +`cp` them into `$BIN/libexec/`, which puts a second copy on the chain AHEAD of the +real one -- a cache with no invalidation, in a tree whose whole point is that +Bazel decides what is stale. + +It cost a day. After the 71fb301a repin the fresh UI kept talking to WebContent +binaries left in `bazel-out/k8-fastbuild/libexec/` by a staging run from the +PREVIOUS pin, six weeks earlier. Upstream had inserted IPC messages, so every +message id past the insertion point had shifted by one: the endpoint magic +matched (right endpoint) and the payload did not parse, giving + + Failed to parse IPC message: + Local endpoint error: Can't read past the end of the stream memory + Peer endpoint error: Endpoint magic number mismatch, not my message! + +~14,000 times, while all 20 generated `*Endpoint.h` were byte-identical to +CMake's -- so every check aimed at the code generator said the build was fine, +because it was. The failing artifact was not built by the build. + +The fix is to delete the staging step, not to refresh it: verified by removal, +with no `libexec/` anywhere, `--headless=text` and `--headless=layout-tree` are +byte-identical to the CMake reference at the same pin. + +These tests guard the recipe, because the recipe is the interface: it is what +Ulf runs, and a stale binary it silently prefers is indistinguishable from a +miscompile. +""" + +import os +import re + +_HERE = os.path.dirname(__file__) +_EXAMPLE = os.path.join(_HERE, "..", "examples", "ladybird") + + +def _readme(): + with open(os.path.join(_EXAMPLE, "README.md")) as f: + return f.read() + + +def _run_recipe_shell(): + """The ```sh block of the run recipe -- the part a reader copy-pastes. + + Keyed off the resource-root assignment rather than a heading, so reordering + the prose does not silently make these tests vacuous. + """ + blocks = re.findall(r"```sh\n(.*?)```", _readme(), re.S) + hits = [b for b in blocks if 'share/Lagom' in b and 'bazel info bazel-bin' in b] + assert len(hits) == 1, \ + f"expected exactly one run recipe block, found {len(hits)}" + return hits[0] + + +def test_the_recipe_does_not_copy_services_into_libexec(): + """The assertion that would have failed while the browser was broken. + + Any copy INTO a libexec directory is the bug, whatever it is spelled with -- + cp, install, ln -s -- because the destination is searched before bin/. + """ + sh = _run_recipe_shell() + for line in sh.splitlines(): + code = line.split("#", 1)[0] + if not code.strip(): + continue + assert not re.search(r"(cp|install|ln)\s.*libexec", code), \ + ("the run recipe stages a copy of the build's outputs into libexec, " + "which Ladybird searches BEFORE bin/ -- so the copy shadows the " + f"build and can go stale: {line.strip()}") + + +def test_the_recipe_removes_a_libexec_left_by_an_older_recipe(): + """Deleting the step is not enough: the directory it made still shadows. + + Anyone who ran the previous recipe has one on disk, and it keeps winning + forever -- it is not an output of any target, so no `bazel clean` removes it + and no rebuild refreshes it. The recipe has to actively clear it. + """ + sh = _run_recipe_shell() + assert re.search(r"rm -rf\s+[^\n]*libexec", sh), \ + "the recipe must delete a libexec/ left behind by the older recipe" + + +def test_the_shadowing_lookup_order_is_written_down(): + """Why no staging is needed is the non-obvious part; state it or lose it. + + Without the lookup order in the text, "don't stage into libexec" reads like + a style preference and the next person helpfully re-adds it. + """ + # Collapse whitespace first: the README is hard-wrapped at 80 columns, so + # where the line breaks fall is arbitrary and must not decide the assertion. + readme = re.sub(r"\s+", " ", _readme()) + assert "get_paths_for_helper_process" in readme, \ + "the function that defines the search order is not named" + assert re.search(r"libexec.{0,80}\bbefore\b.{0,40}bin", readme), \ + "the README does not say libexec is searched BEFORE bin" + # And the symptom, so the next person greps the error and lands here rather + # than re-auditing the IPC code generator (which was innocent). + assert "Endpoint magic number mismatch" in readme, \ + "the IPC symptom this produces is not documented" + + +def test_the_resource_root_recipe_survives_a_stale_share_symlink(): + """Same class, second instance: `share` was a symlink into CMake's tree. + + An older recipe pointed `/../share` at `Build/full/share`. Once that + build directory moved, `mkdir -p` on a path under it failed -- reported as + "File exists" for a path that does not exist, which reads like a bug in + mkdir. `rm -rf` does not remove a dangling symlink's target problem; only + removing the LINK does. + """ + sh = _run_recipe_shell() + assert re.search(r"rm -f\s+\"?\$\(dirname\s+\"?\$BIN\"?\)\"?/share", sh), \ + "the recipe does not clear a `share` symlink left pointing into CMake's tree" From 4fdd4e01827d5d91ec8cc22c8766e23a90cbd4cc Mon Sep 17 00:00:00 2001 From: Ulf Adams Date: Wed, 19 Aug 2026 12:46:53 +0000 Subject: [PATCH 42/59] The cargo emitter described the Flap lock from memory, and upstream deleted a crate cargo_ring.bzl carried, in generated prose, "its own lock with exactly 3 packages (flapc, in-tree bytecode_def, and smallvec from crates.io pinned =1.15.1)" -- four hand-copied facts under a header that says AUTO-GENERATED. Upstream's a32d9c9f ("LibJS: Derive bytecodes from Flap handlers") deleted the bytecode_def crate, and the Flap lock has had 2 packages ever since. Nothing failed, because prose has no compiler: all three cargo_*.bzl still reproduce byte-for-byte at 71fb301a, and one of them documents a crate that does not exist. emit_ring() now reads the lock: the count, each package, and which are in-tree (no checksum) vs registry, plus which are shared with the big workspace at the same version AND hash -- the reason the shared crate collapses to one fetch rule, which was the point the prose was making and is now derived rather than asserted. The LOCKS comment no longer names the contents either. Third instance of one bug in this repin (after SYSTEM_LIBS and the three-of-six glib include roots), so the rule is worth stating plainly: if a fact is worth putting in generated output, it is worth reading from the input. Two guards, each verified to bite by reintroducing the bug, and they fail for different reasons -- the mechanism on a fixture whose Flap lock has 2 packages (so a re-hardcoded "3" fails here, not six weeks later on Ulf's machine), and the checked-in artifacts, where a crate mentioned in prose but never declared as a rule/label/key is either dead text or a missing rule. The second one deliberately does NOT read the real Cargo.locks: they live in the Ladybird checkout, so that version needs a 5,958-file clone and would be skipped in CI exactly when it matters. --- .../workspace/Meta/emit_cargo_bazel.py | 39 ++++++++--- examples/ladybird/workspace/cargo_ring.bzl | 9 ++- tests/test_emit_cargo.py | 65 +++++++++++++++++++ 3 files changed, 100 insertions(+), 13 deletions(-) diff --git a/examples/ladybird/workspace/Meta/emit_cargo_bazel.py b/examples/ladybird/workspace/Meta/emit_cargo_bazel.py index c5c8298..f7210ab 100644 --- a/examples/ladybird/workspace/Meta/emit_cargo_bazel.py +++ b/examples/ladybird/workspace/Meta/emit_cargo_bazel.py @@ -81,10 +81,14 @@ HEADER = "# AUTO-GENERATED by Meta/emit_cargo_bazel.py — do not edit.\n" -# The two cargo workspaces. Flap is `exclude`d from the root workspace and has -# its own lock with exactly 3 packages (flapc, in-tree bytecode_def, and -# smallvec from crates.io), so it needs the same treatment at 1/50th the size. +# The two cargo workspaces. Flap is `exclude`d from the root workspace and +# resolves its own lock, so it needs the same treatment at a fraction of the +# size. What is IN that lock is deliberately not written down here: this comment +# used to name its three packages, one of which upstream deleted (bytecode_def, +# in a32d9c9f), leaving the emitter to print a confident description of a +# package that no longer existed. emit_ring() reads the lock instead. LOCKS = ["Cargo.lock", "Libraries/LibJS/Flap/Cargo.lock"] +FLAP_LOCK = "Libraries/LibJS/Flap/Cargo.lock" # The host triple cargo builds for. Not a guess: `rustc -vV`'s `host:` line is # what CMake reads (Meta/CMake/rust_crate.cmake), and it names the directory the @@ -980,14 +984,33 @@ def emit_ring(crates, specs, binaries): print(" # build_rust_binary() rather than import_rust_crate(). Two of them, and") print(" # they are two different shapes:") print(" #") + # The Flap lock's contents are DESCRIBED, not restated. The previous version + # of this comment said "exactly 3 packages (flapc, in-tree bytecode_def, and + # smallvec ... =1.15.1)", all four facts hand-copied -- and upstream's + # a32d9c9f deleted bytecode_def, so the generated file confidently documented + # a package that no longer exists. Same class as the SYSTEM_LIBS and glib + # include-root drift: a fact worth stating in generated output is a fact worth + # reading from the input. + flap_pkgs = lock_packages(FLAP_LOCK) + flap_reg, flap_ws = split_packages(flap_pkgs) + shared = [(n, v) for (n, v, c) in flap_reg + if (n, v, c) in lock_packages("Cargo.lock")] print(" # flapc a pure build TOOL -- Bazel runs it in a genrule to") print(" # produce the interpreter assembly. Its workspace is") print(" # `exclude`d from the root one and has its own lock") - print(" # with exactly 3 packages (flapc, in-tree") - print(" # bytecode_def, and smallvec from crates.io pinned") - print(" # =1.15.1 -- the same version AND checksum the big") - print(" # workspace pins, so it is the same fetch rule, not a") - print(" # second one).") + print(" # with %d package%s: %s" % ( + len(flap_pkgs), "" if len(flap_pkgs) == 1 else "s", + ", ".join("%s %s%s" % (n, v, " (in-tree)" if (n, v) in flap_ws else "") + for (n, v) in [(n, v) for (n, v, _) in flap_pkgs]))) + if shared: + print(" # %s %s %s pinned at the same version AND" + % (", ".join(n for (n, _) in shared), + "is" if len(shared) == 1 else "are", + "also" if len(shared) == 1 else "also")) + print(" # checksum as the big workspace, so %s the same" + % ("it is" if len(shared) == 1 else "they are")) + print(" # fetch rule%s, not a second one." + % ("" if len(shared) == 1 else "s")) print(" # cranelift-compiler a RUNTIME tool -- LibWasm spawns it to AOT-compile") print(" # WebAssembly, and its build script ALSO emits the") print(" # CraneliftFFI.h that LibWasm's CraneliftBridge.cpp") diff --git a/examples/ladybird/workspace/cargo_ring.bzl b/examples/ladybird/workspace/cargo_ring.bzl index 23f032c..fadf5fe 100644 --- a/examples/ladybird/workspace/cargo_ring.bzl +++ b/examples/ladybird/workspace/cargo_ring.bzl @@ -278,11 +278,10 @@ def cargo_ring(): # flapc a pure build TOOL -- Bazel runs it in a genrule to # produce the interpreter assembly. Its workspace is # `exclude`d from the root one and has its own lock - # with exactly 3 packages (flapc, in-tree - # bytecode_def, and smallvec from crates.io pinned - # =1.15.1 -- the same version AND checksum the big - # workspace pins, so it is the same fetch rule, not a - # second one). + # with 2 packages: flapc 0.1.0 (in-tree), smallvec 1.15.1 + # smallvec is also pinned at the same version AND + # checksum as the big workspace, so it is the same + # fetch rule, not a second one. # cranelift-compiler a RUNTIME tool -- LibWasm spawns it to AOT-compile # WebAssembly, and its build script ALSO emits the # CraneliftFFI.h that LibWasm's CraneliftBridge.cpp diff --git a/tests/test_emit_cargo.py b/tests/test_emit_cargo.py index e9e4830..bcb057c 100644 --- a/tests/test_emit_cargo.py +++ b/tests/test_emit_cargo.py @@ -897,3 +897,68 @@ def test_every_extension_created_repo_is_named_in_module_bazel(): # not reach. `python3 tests/run_all.py` enumerates the module instead, so a test's # POSITION in the file cannot decide whether it runs; it also fails if a file # defines no tests at all. Run a single file with `run_all.py `. + + +def test_the_flap_lock_is_described_from_the_lock_not_from_memory(): + """The comment the emitter PRINTS must be read out of the lock file. + + This one shipped wrong. The generated cargo_ring.bzl carried, in prose, + "its own lock with exactly 3 packages (flapc, in-tree bytecode_def, and + smallvec from crates.io pinned =1.15.1)" -- four hand-copied facts. Upstream's + a32d9c9f ("LibJS: Derive bytecodes from Flap handlers") deleted the + bytecode_def crate, so a generated file, whose header says AUTO-GENERATED, + confidently documented a package that no longer exists. Nothing failed: prose + has no compiler. + + Same class as the SYSTEM_LIBS divergence and the three-of-six glib include + roots -- and the same rule: if a fact is worth stating in generated output, it + is worth READING from the input. The fixture's flap lock has 2 packages, so a + re-hardcoded "3" fails here rather than six weeks later on Ulf's machine. + """ + import contextlib + import io + mod = _load(_fixture()) + buf = io.StringIO() + with contextlib.redirect_stdout(buf): + mod.emit_ring(mod.all_registry_crates(), mod.crate_specs(), + mod.binary_specs()) + ring = buf.getvalue() + # The described package count must match the fixture's lock, not the real one. + assert re.search(r"with 2 packages:", ring), \ + "the flap lock's package count is not derived from the lock" + assert "flapc 0.1.0 (in-tree)" in ring, \ + "the in-tree member is not identified from the absence of a checksum" + assert "smallvec 1.15.1" in ring, "the registry crate is not named from the lock" + # And the crate upstream deleted must not be resurrected by a hardcoded string. + assert "bytecode_def" not in ring, \ + "a deleted crate is still named in the generated output" + + +def test_no_generated_cargo_file_names_a_crate_it_does_not_declare(): + """The general form, against the CHECKED-IN artifacts — and no checkout. + + The test above guards the mechanism on a fixture; this guards the artifact. + It cannot compare against the real Cargo.locks, because those live in the + Ladybird checkout and not in this overlay -- a test that needs a 5,958-file + clone is a test that does not run. But the bug is visible WITHOUT the locks: + `bytecode_def` appeared in cargo_ring.bzl only ever inside a comment, never as + a `crate =` / `name =` / label. A crate the file talks about but never + declares is either dead prose or a missing rule, and both are worth failing + on -- the version of this check that needed the locks would have been skipped + in CI exactly when it mattered. + """ + for rel in ("cargo_ring.bzl", "cargo_crates.bzl", "cargo_index.bzl"): + text = _read(rel) + # The structural positions: what the file actually DECLARES. + declared = set(re.findall(r"(?:name|crate|bin) = '([^']+)'", text)) + declared |= set(re.findall(r"crate_([a-z0-9_]+)_\d", text)) + declared |= set(re.findall(r"//:([A-Za-z0-9_.-]+)", text)) + # A dict key is a declaration too (cargo_index.bzl keys its per-crate + # tables by crate name). + declared |= set(re.findall(r"^\s*'([^']+)':", text, re.M)) + for name in set(re.findall( + r"\b([a-z][a-z0-9]*(?:_[a-z0-9]+)*_rust|bytecode_def)\b", text)): + assert any(name in d for d in declared) or name in declared, \ + (f"{rel} mentions crate {name!r} in prose but never declares it " + "-- either dead text about a crate upstream deleted, or a rule " + "that is missing") From f9a0613c6b97443656e82ad409b773cc27691e02 Mon Sep 17 00:00:00 2001 From: Ulf Adams Date: Wed, 19 Aug 2026 13:56:48 +0000 Subject: [PATCH 43/59] apply_overlay.sh could not repin an existing tree, which is the only kind Ulf has Ulf asked how to get his tree patched. The answer was: you can't. Both failures are in the path nobody had run -- the script had only ever been tested on a FRESH clone, and every repin instruction in the README assumed one. Reproduced by building a replica of his tree (old pin f9e34731 + the 4 old patches + the old overlay, reconstructed from git history) and running the current script at it. Two bugs, in order: 1. `git checkout --detach ` refuses, because the patches modify tracked files: error: Your local changes to the following files would be overwritten by checkout: Meta/Generators/libweb_bindings/to_idl_value.py, UI/Qt/TabBar.h git is right; the script was wrong to stop there. The reader cannot resolve it either: two of those four patches were fixed UPSTREAM at 71fb301a and deleted from the overlay, so their bytes can no longer be reverse-applied from anything we carry, and they are indistinguishable from his own edits. Now: stash, with a message naming the script and the pin, and print the `stash pop` that restores it. Deliberately NOT `reset --hard`/`checkout --`: a patch we no longer carry looks exactly like a debugging edit made on top, and a script that silently discards the second kind is one nobody should run on a tree they care about. 2. The vcpkg prefetch then dies with `fatal: unable to read tree` -- the trap this script's own header documents. Phase 1 defers Build/vcpkg/BUILD.bazel so the directory does not exist when upstream's bootstrap looks; but on a repin the file is ALREADY THERE from the previous run, so deferring it changes nothing. The bootstrap reads the bare directory as a checkout, walks up to Ladybird's repo for a HEAD, and tries to check vcpkg's baseline out of it. Now the stale directory is cleared first: the overlay's own deferred files by name, then `rmdir` (an EMPTY Build/vcpkg is just as fatal -- the bootstrap tests for the directory). Never a directory with a .git, never `rm -rf`: if something we do not own is in there, that should be loud. Verified end to end on the replica: runs through, `--verify` reports 45 identical with both patch effects present, previous state recoverable in the stash. README now answers the question where it is asked, with the repin commands and the stash caveat. Three guards, each verified to bite. Two of them had to be scoped to CODE with comments stripped, because my comments explain why `reset --hard` and `rm -rf` are wrong here -- a test that cannot tell the explanation from the deed forces you to delete the explanation to go green. --- examples/ladybird/README.md | 19 ++++++ examples/ladybird/apply_overlay.sh | 64 ++++++++++++++++++ tests/test_apply_overlay.py | 104 +++++++++++++++++++++++++++++ 3 files changed, 187 insertions(+) diff --git a/examples/ladybird/README.md b/examples/ladybird/README.md index 110b1d1..b208af2 100644 --- a/examples/ladybird/README.md +++ b/examples/ladybird/README.md @@ -351,6 +351,25 @@ manual version below has four ways to be silently wrong (see ./apply_overlay.sh --verify ~/ladybird # check an existing tree, change nothing ``` +**Already have a tree from an older pin?** The same command moves it forward — +point it at the tree you have: + +```sh +./apply_overlay.sh ~/ladybird # fetches the new commit, stashes the old + # pin's patches, checks out, re-patches, + # re-copies the overlay +./apply_overlay.sh --verify ~/ladybird # expect: 45 identical, both patch effects +git -C ~/ladybird stash list # your previous tree state, if you want it back +``` + +Your patched files are **stashed, never discarded** (`git stash pop` restores +them). That matters because a repin can *delete* a patch — two of the four were +fixed upstream at `71fb301a` — so those modifications can no longer be +reverse-applied from anything the overlay carries, and they are indistinguishable +from your own edits on top. This used to abort with git's +`error: Your local changes to the following files would be overwritten by +checkout`, which was correct of git and useless to you. + Or by hand: ```sh diff --git a/examples/ladybird/apply_overlay.sh b/examples/ladybird/apply_overlay.sh index a590ff0..906c996 100755 --- a/examples/ladybird/apply_overlay.sh +++ b/examples/ladybird/apply_overlay.sh @@ -211,6 +211,38 @@ if [ -e "$TARGET/.git" ]; then cd "$TARGET" git rev-parse --verify "$LADYBIRD_COMMIT^{commit}" >/dev/null 2>&1 \ || git fetch --no-tags origin "$LADYBIRD_COMMIT" + # A tree that already HAS the overlay is the normal case for a REPIN, and it + # is the case this script used to fail on. Its patches are applied, so + # tracked files are modified, so `git checkout --detach` refuses: + # + # error: Your local changes to the following files would be overwritten by + # checkout: Meta/Generators/libweb_bindings/to_idl_value.py, UI/Qt/TabBar.h + # + # -- which is git being careful, but it left the reader to work out that the + # modifications are OUR patches, that a repin may have deleted the patch that + # made them (two of the four were fixed upstream at 71fb301a, so their bytes + # cannot be reverse-applied from anything this overlay still carries), and + # which of their own edits are mixed in. So handle it here, once. + # + # STASH rather than `checkout --` / `reset --hard`: the modifications might not + # all be ours. A patch we no longer carry is indistinguishable from the + # reader's own debugging edit, and a script that silently discards the second + # kind is a script nobody should run on a tree they care about. A stash is + # recoverable and its name says who made it. + if [ -n "$(git status --porcelain --untracked-files=no)" ]; then + note "this tree has local changes to tracked files (expected: the patches" + note "from a previous pin). Stashing them -- NOTHING is discarded:" + git status --porcelain --untracked-files=no | sed 's/^/ /' + # No --include-untracked: the untracked files are the OVERLAY, which the + # copy below overwrites anyway, and stashing them would hide the staged + # Build/vcpkg state the prefetch ordering depends on. + git stash push --quiet \ + -m "apply_overlay.sh: pre-$LADYBIRD_COMMIT tree state" \ + || die "could not stash local changes; commit or stash them yourself, + then re-run. (This tree is left exactly as it was.)" + note "stashed as: $(git stash list | head -1)" + note " recover with: git -C $TARGET stash pop" + fi git checkout --detach "$LADYBIRD_COMMIT" else note "cloning Ladybird (full history: vcpkg needs it, see README)" @@ -261,6 +293,38 @@ done < "$FILE_LIST" note "$n files copied (bazelrc.txt -> .bazelrc), $deferred deferred until vcpkg exists" # Phase 2: the vcpkg prefetch, then the deferred files. +# +# Deferring the copy is not sufficient on a REPIN. The trap at the top of this +# file is that the DIRECTORY Build/vcpkg existing (with no .git in it) makes +# upstream's bootstrap skip the clone -- and on a tree that already has an older +# overlay, Build/vcpkg/BUILD.bazel is ALREADY THERE, put there by the previous +# run. So phase 1 not creating it changes nothing, and the prefetch dies exactly +# as documented: +# +# fatal: unable to read tree (40f3c709...) +# subprocess.CalledProcessError: ['git','checkout','40f3c709...'] status 128 +# +# (it walks up to Ladybird's repo, gets Ladybird's HEAD, and tries to check +# vcpkg's baseline out of it). Found by running the repin against a replica of +# Ulf's tree; the deferral logic had only ever been tested on a fresh clone. +# +# So: if there is no .git, the directory is not a checkout, and anything in it is +# ours to move out of the way. Only the overlay's own deferred files are removed +# -- never a directory with a .git, and never anything the overlay does not own. +if [ ! -d "$TARGET/Build/vcpkg/.git" ] && [ -d "$TARGET/Build/vcpkg" ]; then + while IFS= read -r f; do + case "$f" in Build/vcpkg/*) ;; *) continue ;; esac + if [ -e "$TARGET/$f" ]; then + note "un-staging $f so the vcpkg bootstrap sees no checkout" + rm -f "$TARGET/$f" + fi + done < "$FILE_LIST" + # An empty Build/vcpkg is just as fatal as one with a file in it: the + # bootstrap tests for the DIRECTORY. rmdir, not rm -rf: if anything else is + # in there it is not ours and the failure should be loud. + rmdir "$TARGET/Build/vcpkg" 2>/dev/null && note "removed the empty Build/vcpkg" || true +fi + if [ -d "$TARGET/Build/vcpkg/.git" ]; then note "vcpkg checkout already present" elif [ "$PREFETCH" -eq 1 ]; then diff --git a/tests/test_apply_overlay.py b/tests/test_apply_overlay.py index 1ecff6b..24151e7 100644 --- a/tests/test_apply_overlay.py +++ b/tests/test_apply_overlay.py @@ -393,3 +393,107 @@ def test_no_two_patches_are_alternatives_of_each_other(): dupes = {n for n in prefixes if prefixes.count(n) > 1} assert not dupes, \ "patches sharing a series number are alternatives, not a series: %r" % (dupes,) + + +def test_a_repin_does_not_abort_on_the_previous_pin_s_patches(): + """The REPIN path: a tree that already has the overlay, from an older pin. + + This is not a hypothetical. Ulf asked "how do I get my tree patched?" and the + answer was: you can't, the script aborts. His tree is the previous pin with + four patches applied, so tracked files are modified, so `git checkout + --detach ` refuses: + + error: Your local changes to the following files would be overwritten by + checkout: Meta/Generators/libweb_bindings/to_idl_value.py, UI/Qt/TabBar.h + + git is right to refuse; the script was wrong to leave it there. It cannot be + solved by the reader either, because two of the four patches were fixed + UPSTREAM at the new pin and deleted from the overlay -- so their modifications + cannot be reverse-applied from anything this overlay still carries, and they + are indistinguishable from the reader's own edits. + + Hence: stash, never discard. `checkout --`/`reset --hard` would silently throw + away a debugging edit made on top of the patches, which is not a thing a + script should do to a tree someone cares about. Reproduced end-to-end against + a replica of Ulf's tree (old pin + 4 old patches + old overlay): the script + now runs through, `--verify` reports 45/45 with both patch effects present, + and the old state is recoverable with `git stash pop`. + """ + t = _text() + # The clone-reuse branch must handle a dirty tree before it checks out. + reuse = t.split("reusing existing clone", 1)[1].split("note \"at $", 1)[0] + assert "git stash push" in reuse, \ + "a tree with the previous pin's patches applied still aborts the checkout" + assert "git status --porcelain" in reuse, \ + "the dirty-tree case is not detected before the checkout" + # Destructive alternatives must NOT be what it reaches for. Checked against + # the CODE only: the comment explains why `reset --hard` is wrong here, and a + # test that cannot tell the explanation from the deed forces you to delete the + # explanation to make it pass. + code = "\n".join(l.split("#", 1)[0] for l in reuse.splitlines()) + for destructive in ("reset --hard", "checkout -- .", "clean -fd"): + assert destructive not in code, \ + (f"the repin path uses `git {destructive}`, which discards edits that " + "may not be ours -- a patch we no longer carry looks exactly like the " + "reader's own change") + # And it must say how to get the work back, or a stash is just a nicer loss. + assert "stash pop" in reuse, "the script does not say how to recover the stash" + # The stash must be identifiable months later, not stash@{0} among many. + assert re.search(r'stash push[^\n]*-m ["\']?apply_overlay', reuse) or \ + "-m \"apply_overlay.sh:" in reuse, "the stash is created without a message" + + +def test_the_repin_clears_the_stale_build_vcpkg_before_the_prefetch(): + """Deferring the copy is not enough when the tree ALREADY has the overlay. + + The script's headline trap: the *directory* `Build/vcpkg` existing without a + `.git` makes upstream's `Meta/Utils/build_vcpkg.py` skip the clone, walk up to + Ladybird's own repo for a HEAD, and die with `fatal: unable to read tree`. + Phase 1 therefore defers `Build/vcpkg/BUILD.bazel` until after the prefetch. + + On a REPIN that deferral does nothing: the file is already there from the + PREVIOUS run, so not creating it changes nothing and the prefetch fails + exactly as documented. The deferral logic had only ever been exercised on a + fresh clone -- found by running the repin against a replica of Ulf's tree + (old pin + 4 old patches + old overlay), which is the only reason it was found + before he hit it. + + So the tree must be put back into the state the bootstrap expects: remove the + overlay's own deferred files, then the directory if it is empty. Guarded on + both sides -- it must NOT touch a real checkout (one with a .git), and it must + not `rm -rf` a directory holding something the overlay does not own. + """ + t = _text() + phase2 = t.split("# Phase 2:", 1)[1] + # It has to notice the stale directory at all. + assert re.search(r'if \[ ! -d "\$TARGET/Build/vcpkg/\.git" \] && ' + r'\[ -d "\$TARGET/Build/vcpkg" \]', phase2), \ + "a stale Build/vcpkg (directory, no .git) is not detected before the prefetch" + stale = phase2.split("un-staging", 1)[1].split("if [ -d", 1)[0] + # Comments stripped: the comment here explains why `rm -rf` is wrong, and a + # test that cannot tell the explanation from the deed makes you delete the + # explanation to go green. + stale = "\n".join(l.split("#", 1)[0] for l in stale.splitlines()) + # Only the overlay's own files, and only when they exist. + assert "rm -f" in stale and "rm -rf" not in stale, \ + ("clearing the stale directory must not rm -rf: anything in there that the " + "overlay does not own should fail loudly, not be deleted") + # The empty directory is as fatal as a populated one -- the bootstrap tests + # for the directory -- so it must go too, but only if it IS empty. + assert "rmdir" in stale, \ + "an EMPTY Build/vcpkg still makes the bootstrap skip the clone" + + +def test_the_repin_path_is_documented_for_someone_holding_an_old_tree(): + """"How do I get my tree patched?" must have an answer in the README. + + Ulf asked it, and at that point the answer was "you can't" -- the script + aborted on his tree. A fix nobody can find is the same as no fix, and the + people who need this path are exactly the ones who already have a tree. + """ + readme = re.sub(r"\s+", " ", (REPO / "examples" / "ladybird" / "README.md").read_text()) + assert "older pin" in readme or "old pin" in readme, \ + "the README does not address a tree from a previous pin" + assert "stash" in readme, \ + "the README does not say what happens to the previous pin's patches" + assert "stash pop" in readme, "the README does not say how to get them back" From a4f2bd5be81badcffb294f3d2f607a4952435aae Mon Sep 17 00:00:00 2001 From: Ulf Adams Date: Wed, 19 Aug 2026 14:02:04 +0000 Subject: [PATCH 44/59] The vcpkg capture reported success while silently losing five URLs The 71fb301a re-capture printed "All requested installations completed successfully in: 49 min", exited 0, and wrote 72 rows. The committed capture has 76. Five URLs vanished, and only one of them was mentioned anywhere: angle's gni-to-cmake.py failed with a transient TLS error (this sandbox's clock was briefly behind the certificate's validity window -- "certificate is not yet valid"; the URL and its pinned hash are fine, verified by hand). The other four are the mechanism worth remembering: a failed download HALTS its portfile, so the four vcpkg_download_distfile calls after it in angle were never made, never requested, and so never captured. The recorder logs each tuple BEFORE fetching, so the one that FAILED is present in the capture and the four nobody asked for are simply absent. Result: a pin missing five URLs that looks complete -- the worst shape available, since the emitted rules would fetch nothing for angle and fail much later inside a port build. So the capture now judges itself, which it cannot do from vcpkg's exit code. The recorder appends each failed URL to a sentinel file (a variable cannot work: the recorder is a separate process per download), and the driver refuses to bless a run with any failure, naming them. Verified against a fake vcpkg that calls the recorder with an unreachable URL and then exits 0 like the real one: exit 1, URLs named. Clean run still exits 0. Two more fixes to the same script, both found by it failing in front of me: * `--max-time 120` bounded how long a download may legitimately TAKE, so it killed OpenGL-Registry at 22MB of a healthy transfer, called it "FAILED to fetch", and did it again every re-run -- a capture that could not finish, blaming the mirror. Now --speed-time/--speed-limit, which is the property actually wanted ("no progress for 60s") and cannot mistake a big file for a dead one. With it the capture got past that file and completed. * It truncated its output on start, so every interruption of a ~50-minute network-bound job cost the whole run (todo c2affe6b). Now append + dedupe: the re-run resumed from 72 rows instead of 0. Three guards, each verified to bite. The re-capture is running again to recover the five URLs; SDL is already confirmed to move 3.4.12 -> 3.2.28, which is the staleness that started this. --- .../workspace/Meta/vcpkg_capture_assets.sh | 67 +++++++++++++-- tests/test_emit_vcpkg.py | 82 +++++++++++++++++++ 2 files changed, 143 insertions(+), 6 deletions(-) diff --git a/examples/ladybird/workspace/Meta/vcpkg_capture_assets.sh b/examples/ladybird/workspace/Meta/vcpkg_capture_assets.sh index 9755d74..565c98f 100755 --- a/examples/ladybird/workspace/Meta/vcpkg_capture_assets.sh +++ b/examples/ladybird/workspace/Meta/vcpkg_capture_assets.sh @@ -25,7 +25,21 @@ set -euo pipefail OUT="${1:?usage: vcpkg_capture_assets.sh [vcpkg args...]}" shift -: > "$OUT" +# APPEND to an existing capture rather than truncating it, and share vcpkg's +# downloads/ across runs (pass --downloads-root at a stable path). The run takes +# tens of minutes and reaches the network for every byte, so it WILL be +# interrupted -- a sandbox restart, a dead mirror, a timeout. Truncating means +# every interruption costs the whole run, which is how the first two attempts +# died (todo c2affe6b: long jobs must be resumable and append-only). The final +# `sort -u` dedupes, so a tuple recorded twice is free; a tuple recorded once and +# then thrown away is another 40 minutes. +touch "$OUT" +echo "capture: appending to $OUT ($(wc -l < "$OUT") rows already recorded)" >&2 + +# Where the recorder reports downloads it could not complete. A file rather than a +# counter, because the recorder runs as a separate PROCESS per download: nothing +# it sets in a variable can reach this script. +FAILED=$(mktemp /tmp/vcpkg-capture-failed-XXXXXX) REC=$(mktemp /tmp/vcpkg-record-XXXXXX.sh) cat > "$REC" <> "$OUT" # Record the tuple regardless (the SHA512 is what matters and it is # mirror-independent), and try the known GNU mirrors before giving up. Hit live: # ftpmirror.gnu.org returned 502 for ~13 minutes and wedged the capture. -if curl -sSL --fail --max-time 120 -o "\$3" "\$1"; then exit 0; fi +# Bound STALLS, not total transfer time. This was --max-time 120, which is a cap +# on how long a download may legitimately take -- so it killed OpenGL-Registry at +# 22MB of a working transfer, then reported it as "FAILED to fetch", then fell +# through to the origin, on a repeat, forever: a capture that cannot finish and +# blames the mirror. --speed-time/--speed-limit is the property actually wanted +# ("no progress for 60s"), and it cannot mistake a big file for a dead one. +if curl -sSL --fail --speed-time 60 --speed-limit 1024 -o "\$3" "\$1"; then exit 0; fi alt=\$(printf '%s' "\$1" | sed \ -e 's|https://ftpmirror.gnu.org/gnu/|https://www.mirrorservice.org/sites/ftp.gnu.org/gnu/|' \ -e 's|https://ftp.gnu.org/pub/gnu/|https://www.mirrorservice.org/sites/ftp.gnu.org/gnu/|') if [ "\$alt" != "\$1" ]; then echo "capture: primary failed, trying mirror \$alt" >&2 - curl -sSL --fail --max-time 120 -o "\$3" "\$alt" && exit 0 + curl -sSL --fail --speed-time 60 --speed-limit 1024 -o "\$3" "\$alt" && exit 0 fi # Leave NO partial file behind. curl -o creates the destination before it knows # the transfer will fail, and vcpkg's downloads/ is keyed by name: a surviving @@ -61,6 +81,11 @@ fi # renamed four unrelated distfiles in the capture. (finding 30) rm -f "\$3" echo "capture: FAILED to fetch \$1" >&2 +# Tell the DRIVER, not just the log. A failed download halts its portfile, so +# every vcpkg_download_distfile after it in that port is never requested and so +# never captured -- and vcpkg still exits 0 saying "All requested installations +# completed successfully". Without this file the driver cannot know. +printf '%s\n' "\$1" >> "$FAILED" exit 1 EOF chmod +x "$REC" @@ -78,8 +103,38 @@ chmod +x "$REC" "$@" rm -f "$REC" -# Dedupe on (url, sha512) -- NOT the whole line. The third column is the raw -# {dst}, which carries a pid and so differs on every retry; sorting whole lines -# leaves the same distfile in the file several times over. +trap 'rm -f "$FAILED"' EXIT + +# A capture that lost a download is INCOMPLETE, and vcpkg does not tell you so in +# its exit code: with --only-downloads it printed "All requested installations +# completed successfully in: 49 min" and exited 0 having FAILED to download +# angle's gni-to-cmake.py (a transient TLS error -- this sandbox's clock was +# briefly behind the certificate's validity window, "certificate is not yet +# valid"). That is not a cosmetic loss: the failure HALTED angle's portfile, so +# the FOUR vcpkg_download_distfile calls after it were never made, never +# requested, and so never captured. The result is a pin that is missing five URLs +# and looks complete -- the worst possible shape, since the emitted rules would +# then fetch nothing for angle and the failure would surface much later as a +# build error inside a port. +# +# The recorder logs the tuple BEFORE fetching, so a failed download is still in +# the capture; what is lost is everything the halted portfile would have asked for +# next. The recorder therefore also appends each failed URL to a sentinel file, +# which is the thing checked here -- not vcpkg's exit code, which lies, and not a +# grep of stderr, which depends on how the caller redirected it. +# +# Dedupe FIRST so the row count reported below is the real one. Dedupe on +# (url, sha512) -- NOT the whole line: the third column is the raw {dst}, which +# carries a pid and so differs on every retry, and sorting whole lines leaves the +# same distfile in the file several times over. sort -u -t$'\t' -k1,2 -o "$OUT" "$OUT" +if [ -s "$FAILED" ]; then + echo "capture: INCOMPLETE -- $(wc -l < "$FAILED") download(s) failed:" >&2 + sed 's/^/capture: /' "$FAILED" >&2 + echo "capture: A failed download HALTS its portfile, so every later" >&2 + echo "capture: vcpkg_download_distfile in that port was never requested and is" >&2 + echo "capture: MISSING from $OUT. Do not emit rules from this file." >&2 + echo "capture: Re-run; it appends and shares downloads/, so it resumes." >&2 + exit 1 +fi echo "captured $(wc -l < "$OUT") distfiles -> $OUT" >&2 diff --git a/tests/test_emit_vcpkg.py b/tests/test_emit_vcpkg.py index 68f8613..d3e6465 100644 --- a/tests/test_emit_vcpkg.py +++ b/tests/test_emit_vcpkg.py @@ -658,3 +658,85 @@ def test_a_moved_pin_is_reported_as_a_stale_capture_not_a_windows_only_fetch(): "main() must classify the dropped rows, not assert a reason for them" assert "static_only = sorted(set(distfiles) - set(cap))" not in body, \ "the unclassified set-difference is back" + + +def _capture_script(): + ws = os.path.join(os.path.dirname(__file__), "..", "examples", "ladybird", + "workspace") + with open(os.path.join(ws, "Meta", "vcpkg_capture_assets.sh")) as f: + return f.read() + + +def test_a_failed_download_makes_the_capture_fail_because_vcpkg_will_not(): + """vcpkg exits 0 on a capture that lost downloads. Ask me how I know. + + The 71fb301a re-capture printed "All requested installations completed + successfully in: 49 min", exited 0, and wrote 72 rows -- while angle's + `gni-to-cmake.py` had FAILED to download (a transient TLS error: this + sandbox's clock was briefly behind the certificate's validity window, + "certificate is not yet valid"). Diffing against the committed capture showed + 72 rows where the old pin had 76: FIVE URLs gone, four of them WebKit files + nothing had reported on at all. + + That is the mechanism worth remembering: a failed download HALTS its portfile, + so the four `vcpkg_download_distfile` calls after it in angle were never made, + never requested, and so never captured. The recorder logs each tuple BEFORE + fetching, so the one that failed is present and the four that were never asked + for are simply absent -- a pin that is missing five URLs and looks complete. + Emitting from it would produce rules that fetch nothing for angle, failing + much later inside a port build. + + So the capture must judge itself, and it cannot do that from vcpkg's exit + code. Verified against a fake vcpkg that calls the recorder with an + unreachable URL and then exits 0 like the real one: the script exits 1 and + names the URL. + """ + sh = _capture_script() + # The recorder must report failures to the DRIVER. A variable cannot: the + # recorder is a separate process per download. + rec = sh.split("cat > \"$REC\"", 1)[1].split("chmod +x", 1)[0] + assert re.search(r'printf .*>> "\$FAILED"', rec), \ + "the recorder does not report a failed download to the driver" + # And the driver must refuse to bless the result. + tail = sh.split("chmod +x", 1)[1] + assert 'if [ -s "$FAILED" ]' in tail, \ + "the driver never checks whether any download failed" + assert re.search(r'if \[ -s "\$FAILED" \][\s\S]{0,800}?exit 1', tail), \ + "a capture with failed downloads still exits 0" + # The message has to say the non-obvious part, or a reader re-runs the emitter + # on the incomplete file. + assert "HALTS its portfile" in tail, \ + "the message does not explain that later downloads in that port are missing" + + +def test_the_capture_bounds_stalls_not_transfer_size(): + """`--max-time` capped how long a download may legitimately TAKE. + + It killed OpenGL-Registry at 22MB of a perfectly healthy transfer, reported it + as "FAILED to fetch", fell through to the origin, and did it again on every + re-run -- a capture that could not finish, blaming the mirror. The property + actually wanted is "no progress for a while", which cannot mistake a big file + for a dead one. + """ + sh = _capture_script() + code = "\n".join(l.split("#", 1)[0] for l in sh.splitlines()) + assert "--max-time" not in code, \ + ("--max-time bounds total transfer time, so it fails big-but-healthy " + "downloads; use --speed-time/--speed-limit") + assert "--speed-time" in code and "--speed-limit" in code, \ + "no stall timeout at all: a dead mirror hangs the capture forever" + + +def test_the_capture_is_resumable(): + """A ~50-minute network-bound job that truncates its output on start. + + Every interruption -- sandbox restart, dead mirror, timeout -- cost the whole + run (todo c2affe6b). Appending plus the final dedupe makes a re-run resume + instead: a tuple recorded twice is free. + """ + sh = _capture_script() + code = "\n".join(l.split("#", 1)[0] for l in sh.splitlines()) + assert ': > "$OUT"' not in code, "the capture truncates its own output on start" + assert 'touch "$OUT"' in code, "the capture does not append to an existing run" + assert re.search(r"sort -u -t\$'\\t' -k1,2 -o \"\$OUT\" \"\$OUT\"", code), \ + "without the dedupe, appending duplicates rows" From 2c51e05fe8281abe043696f0d720894a68a56a61 Mon Sep 17 00:00:00 2001 From: Ulf Adams Date: Wed, 19 Aug 2026 14:15:23 +0000 Subject: [PATCH 45/59] The capture's fast mode cannot reach every download, and said otherwise vcpkg_capture_assets.sh passed --only-downloads unconditionally, under a comment claiming it was "enough because the asset hook fires during resolution". I wrote that sentence from intuition, not measurement, and it is false. Download Mode makes vcpkg refuse to EXECUTE anything, and a portfile that stops executing stops downloading. angle downloads gni-to-cmake.py (overlay-ports/angle/portfile.cmake:79), sets up a python venv to run it (:86, x_vcpkg_get_python_packages), and only THEN downloads four WebKit files: include_CMakeLists.txt, WebKitCompilerFlags.cmake, DetectSSE2.cmake, WebKitMacros.cmake (:123 :129 :144 :151). In Download Mode :86 halts, so those four URLs are unreachable by construction. That is the 71fb301a re-capture writing 72 rows where the committed capture has 76 -- five URLs short, reported as success. The committed 76-row capture therefore cannot have come from this script's default mode; it came from a full build. So the full build is the default now and --only-downloads is opt-in via CAPTURE_ONLY_DOWNLOADS=1, for refreshing URLs you already know need no execution. A halt is also the second way a capture loses URLs with no failed download at all, and vcpkg exits 0 for it ("Downloaded sources for angle", then "All requested installations completed successfully"). There is no exit code to read and no asset-script callback for a step never reached, so vcpkg's stdout is the only witness: it is teed to a log the driver greps, and a halt lands in the same $FAILED sentinel that already refuses to bless a capture with a failed fetch. The report names the PORT, from the "Installing N/M :" line above the halt -- the halt message itself names a shared helper (vcpkg_execute_required_process.cmake:23, identical for every port) and a versioned bt/versioning_ path, neither of which a reader can act on. The match is case-insensitive because vcpkg spells it two ways ("Halting portfile execution." and "Download failed, halting portfile."). Note what CANNOT be done: triage halts into safe and unsafe. In the download-only run 58 of 77 ports halted, most at vcpkg_cmake_configure where every download already happened; nothing in the log distinguishes those from angle's mid-sequence halt, because only the portfile knows whether a download follows. So any halt fails the capture. pipefail was already on, which is what keeps the new `| tee` from swallowing a vcpkg failure -- verified with a fake vcpkg exiting 3. On the failure path the log is kept and its path printed (it is the evidence for which port halted); on success it is deleted. Three guards in tests/test_emit_vcpkg.py, each verified to bite by mutation: unconditional --only-downloads, a dropped tee/log, a case-sensitive halt match, a halt not attributed to a port, a halt not reaching $FAILED, and a dropped pipefail are all caught. Also loosened the existing failed-download guard, which asserted one exact sentence of the message it was checking. --- .../workspace/Meta/vcpkg_capture_assets.sh | 96 ++++++++++++++++--- tests/test_emit_vcpkg.py | 83 +++++++++++++++- 2 files changed, 167 insertions(+), 12 deletions(-) diff --git a/examples/ladybird/workspace/Meta/vcpkg_capture_assets.sh b/examples/ladybird/workspace/Meta/vcpkg_capture_assets.sh index 565c98f..d6f63c1 100755 --- a/examples/ladybird/workspace/Meta/vcpkg_capture_assets.sh +++ b/examples/ladybird/workspace/Meta/vcpkg_capture_assets.sh @@ -21,6 +21,9 @@ # the pin. # # Usage: vcpkg_capture_assets.sh [vcpkg install args...] +# +# This does a FULL vcpkg build (~50 min), because a download-only run cannot reach +# every download -- see the CAPTURE_ONLY_DOWNLOADS comment below the recorder. set -euo pipefail OUT="${1:?usage: vcpkg_capture_assets.sh [vcpkg args...]}" @@ -40,6 +43,10 @@ echo "capture: appending to $OUT ($(wc -l < "$OUT") rows already recorded)" >&2 # counter, because the recorder runs as a separate PROCESS per download: nothing # it sets in a variable can reach this script. FAILED=$(mktemp /tmp/vcpkg-capture-failed-XXXXXX) +# vcpkg's own output, kept so the driver can detect a HALTED portfile -- which +# loses downloads exactly like a failed fetch, but produces no asset-script call +# (the step was never reached) and no nonzero exit. +VCPKG_LOG=$(mktemp /tmp/vcpkg-capture-log-XXXXXX) REC=$(mktemp /tmp/vcpkg-record-XXXXXX.sh) cat > "$REC" <&2 + echo "capture: INCOMPLETE. Ports halt where they would execute, losing every" >&2 + echo "capture: download after that point. Use it to refresh known-reachable" >&2 + echo "capture: URLs only; the result will be reported as not emittable." >&2 + set -- --only-downloads "$@" +fi +# `set -euo pipefail` is on, so the pipeline's status is vcpkg's when vcpkg fails +# and tee's only when tee does -- without pipefail the tee would swallow a vcpkg +# failure outright. "${VCPKG_ROOT:?set VCPKG_ROOT}/vcpkg" install \ - --only-downloads \ --x-asset-sources="clear;x-script,$REC {url} {sha512} {dst}" \ - "$@" + "$@" 2>&1 | tee "$VCPKG_LOG" + +# A halted portfile loses its later downloads exactly like a failed fetch does, +# and vcpkg exits 0 for it too ("Downloaded sources for angle", then "All +# requested installations completed successfully"). There is no exit code to read +# and no asset-script callback for a step never reached, so the only witness is +# the log -- hence the tee. +# +# Report the PORT, not the CMake line: the halt message names a helper script +# (vcpkg_execute_required_process.cmake:23) that is the same for every port, and +# the portfile in the call stack is a versioned path under bt/versioning_. The +# port name comes from the "Installing N/M :@" line above the +# halt, which is also the identifier a reader can act on. +# +# Match the halt case-INSENSITIVELY and on the short phrase: vcpkg has (at least) +# two spellings, "Halting portfile execution." for a refused step and "Download +# failed, halting portfile." for a fetch that did not produce the file. The second +# is normally also reported by the recorder's $FAILED, but only for fetches the +# recorder itself ran -- a SHA512 mismatch, say, is vcpkg rejecting a file the +# recorder fetched happily, and then the log is the only witness again. +awk ' + /^Installing [0-9]+\/[0-9]+ / { split($3, f, ":"); port = f[1] } + tolower($0) ~ /halting portfile/ && port \ + { print "port " port " halted before finishing its portfile" } +' "$VCPKG_LOG" | sort -u >> "$FAILED" rm -f "$REC" +# Keep $VCPKG_LOG until the capture is blessed: if it is INCOMPLETE the log is the +# evidence for WHY (which port halted, at which portfile line), and deleting it +# would leave a reader with a port name and nothing to read. trap 'rm -f "$FAILED"' EXIT # A capture that lost a download is INCOMPLETE, and vcpkg does not tell you so in @@ -129,12 +198,17 @@ trap 'rm -f "$FAILED"' EXIT # same distfile in the file several times over. sort -u -t$'\t' -k1,2 -o "$OUT" "$OUT" if [ -s "$FAILED" ]; then - echo "capture: INCOMPLETE -- $(wc -l < "$FAILED") download(s) failed:" >&2 + echo "capture: INCOMPLETE -- $(wc -l < "$FAILED") loss(es):" >&2 sed 's/^/capture: /' "$FAILED" >&2 - echo "capture: A failed download HALTS its portfile, so every later" >&2 - echo "capture: vcpkg_download_distfile in that port was never requested and is" >&2 - echo "capture: MISSING from $OUT. Do not emit rules from this file." >&2 + echo "capture: A failed download, and a portfile halted for any other reason," >&2 + echo "capture: both HALT the portfile, so every later vcpkg_download_distfile" >&2 + echo "capture: in that port was never requested and is MISSING from $OUT." >&2 + echo "capture: Do not emit rules from this file." >&2 echo "capture: Re-run; it appends and shares downloads/, so it resumes." >&2 + echo "capture: A halt on EVERY port means CAPTURE_ONLY_DOWNLOADS was set --" >&2 + echo "capture: that mode cannot produce a complete capture; drop it." >&2 + echo "capture: vcpkg's own output is kept at $VCPKG_LOG" >&2 exit 1 fi +rm -f "$VCPKG_LOG" echo "captured $(wc -l < "$OUT") distfiles -> $OUT" >&2 diff --git a/tests/test_emit_vcpkg.py b/tests/test_emit_vcpkg.py index d3e6465..a80bd06 100644 --- a/tests/test_emit_vcpkg.py +++ b/tests/test_emit_vcpkg.py @@ -705,8 +705,10 @@ def test_a_failed_download_makes_the_capture_fail_because_vcpkg_will_not(): "a capture with failed downloads still exits 0" # The message has to say the non-obvious part, or a reader re-runs the emitter # on the incomplete file. - assert "HALTS its portfile" in tail, \ + assert re.search(r"HALTS? the portfile|HALTS its portfile", tail), \ "the message does not explain that later downloads in that port are missing" + assert "never requested" in tail and "MISSING" in tail, \ + "the message does not say the capture is missing rows" def test_the_capture_bounds_stalls_not_transfer_size(): @@ -740,3 +742,82 @@ def test_the_capture_is_resumable(): assert 'touch "$OUT"' in code, "the capture does not append to an existing run" assert re.search(r"sort -u -t\$'\\t' -k1,2 -o \"\$OUT\" \"\$OUT\"", code), \ "without the dedupe, appending duplicates rows" + + +def test_download_only_mode_cannot_produce_a_complete_capture(): + """`--only-downloads` was the DEFAULT, with a comment claiming it sufficed. + + The claim ("it is enough because the asset hook fires during resolution") was + invented, not measured -- the fourth instance this repin of generated prose + restating a hand-copied fact. What actually happens: Download Mode makes vcpkg + refuse to EXECUTE anything, and a portfile that stops executing stops + downloading. angle downloads gni-to-cmake.py (portfile.cmake:79), sets up a + python venv to run it (:86, x_vcpkg_get_python_packages), and only THEN + downloads four WebKit files (:123 :129 :144 :151). In Download Mode :86 halts, + so those four URLs are unreachable by construction -- measured: the halt is at + `angle/portfile.cmake:86` in the download-only re-capture, which wrote 72 rows + where the committed capture has 76. + + So the committed 76-row capture cannot have been made in that mode, and the + fast mode must not be the default. It stays available (opt-in) for refreshing + known-reachable URLs, and the halt check refuses to bless its output. + """ + sh = _capture_script() + code = "\n".join(l.split("#", 1)[0] for l in sh.splitlines()) + # Opt-in, not default: the flag may only be passed under the env switch. + assert "--only-downloads" in code, "the fast mode is gone entirely" + for line in code.splitlines(): + if "--only-downloads" in line: + break + guard = code.split("--only-downloads")[0].splitlines()[-6:] + assert any("CAPTURE_ONLY_DOWNLOADS" in l for l in guard), \ + ("--only-downloads is passed unconditionally again; it CANNOT reach the " + "downloads behind an executed portfile step (angle's four WebKit files)") + + +def test_a_halted_portfile_fails_the_capture_and_names_the_port(): + """The other way a capture silently loses URLs, with no failed download. + + A halt loses every download after it, exactly like a failed fetch -- and vcpkg + exits 0 for it too ("Downloaded sources for angle", then "All requested + installations completed successfully"). There is no exit code and no + asset-script callback for a step never reached, so the only witness is vcpkg's + stdout; hence the tee into a log the driver greps. + + It must name the PORT. The halt message itself names a shared helper + (vcpkg_execute_required_process.cmake:23, identical for every port) and the + portfile in the call stack is a versioned path under bt/versioning_ -- neither + is actionable. The port comes from the "Installing N/M :@" + line above the halt. + + Verified against the fake vcpkg that prints a halt and exits 0: the script + exits 1 with "port angle halted before finishing its portfile". + """ + sh = _capture_script() + code = "\n".join(l.split("#", 1)[0] for l in sh.splitlines()) + assert "tee" in code and "VCPKG_LOG" in code, \ + "vcpkg's output is not kept, so a halt cannot be detected at all" + assert re.search(r"halting portfile", code, re.I), \ + "nothing looks for a halted portfile" + assert 'tolower($0)' in code, \ + ("the halt must be matched case-insensitively: vcpkg spells it both " + "'Halting portfile execution.' and 'Download failed, halting portfile.'") + assert re.search(r"Installing \[0-9\]\+", code), \ + "the halt is not attributed to a port via the 'Installing N/M ' line" + # The halt has to land in the same sentinel the exit-1 check reads. + halt = code.split("halting portfile")[1] + assert '>> "$FAILED"' in halt.split("rm -f")[0], \ + "a detected halt does not reach $FAILED, so the capture still exits 0" + + +def test_the_capture_does_not_let_tee_swallow_a_vcpkg_failure(): + """Piping vcpkg into tee moves the pipeline's exit status to tee's. + + Added when the log was introduced: without pipefail, `vcpkg install | tee log` + reports tee's success and a hard vcpkg failure becomes a capture that "worked". + Verified with a fake vcpkg that exits 3 -- the script exits 3. + """ + sh = _capture_script() + code = "\n".join(l.split("#", 1)[0] for l in sh.splitlines()) + assert re.search(r"^set -euo pipefail", code, re.M) or "PIPESTATUS" in code, \ + "vcpkg's output is piped into tee with neither pipefail nor a PIPESTATUS check" From ee1c2af4c27112dd4df7a6f6b560a05ecfe2ff73 Mon Sep 17 00:00:00 2001 From: Ulf Adams Date: Wed, 19 Aug 2026 14:29:46 +0000 Subject: [PATCH 46/59] A cached vcpkg port never runs its portfile, so it captures nothing Third silent-loss mode in this script, and the only one that leaves no trace whatsoever. vcpkg's binary cache is keyed by each port's ABI hash; on a hit it unpacks the archive and does NOT run the portfile, so the port requests none of its downloads. No error, no halt, exit 0, "All requested installations completed successfully". Measured rather than reasoned: a zlib-only manifest run twice against a warm ~/.cache/vcpkg/archives, the second with a fresh install root. Run 1 built zlib and captured 3 rows. Run 2 printed "Restored 3 package(s)" and "completed successfully in: 1.54 ms", exited 0, and captured ZERO. The capture passed no --binarysource at all, so it inherited whatever cache the machine had -- which on its own explains a re-capture that cannot reproduce the committed rows, independently of the Download Mode bug in the previous commit. vcpkg_build.sh has passed --binarysource=clear since it was written, for the adjacent reason (a restore proves nothing about building from source); the capture never did. So --binarysource=clear is now passed, before "$@" so a caller can still override it deliberately. And because the flag states intent while the log states outcome, the outcome is checked too: "Restored N package(s) from " and "The following packages are already installed" both land in the $FAILED sentinel that already refuses to bless the capture. The second matters as much as the first -- the 71fb301a capture's second run had 7 already-installed ports, whose distfiles it could not have captured. Verified with fake vcpkgs printing each line and exiting 0: the script exits 1, and the already-installed case reports the count. Also retriaged the download-only run properly instead of assuming angle was special. Parsing each halt's own call stack for the portfile frame and grepping that portfile for download calls AFTER the halt line: of the 61 halts, exactly ONE port lost downloads -- angle, halted at :86 with vcpkg_download_distfile calls remaining at 122/129/144/151/165. The rest halted at vcpkg_cmake_configure or in vcpkg's own post-install steps, after every download. That matches the five missing URLs exactly (angle's four WebKit files, plus SDL, which moved 3.4.12 -> 3.2.28 with the pin and is the one intended change). Two guards in tests/test_emit_vcpkg.py; four mutations verified to bite (flag dropped, flag moved after "$@", restore detection removed, already-installed detected but not reported). Also replaced a 0-800-character regex window in the older failed-download guard with a real block extraction -- it passed for the wrong reason once the message grew. --- .../workspace/Meta/vcpkg_capture_assets.sh | 53 ++++++++++++++-- tests/test_emit_vcpkg.py | 61 +++++++++++++++++-- 2 files changed, 106 insertions(+), 8 deletions(-) diff --git a/examples/ladybird/workspace/Meta/vcpkg_capture_assets.sh b/examples/ladybird/workspace/Meta/vcpkg_capture_assets.sh index d6f63c1..bec6aed 100755 --- a/examples/ladybird/workspace/Meta/vcpkg_capture_assets.sh +++ b/examples/ladybird/workspace/Meta/vcpkg_capture_assets.sh @@ -137,11 +137,29 @@ if [ -n "${CAPTURE_ONLY_DOWNLOADS:-}" ]; then echo "capture: URLs only; the result will be reported as not emittable." >&2 set -- --only-downloads "$@" fi +# --binarysource=clear is MANDATORY for a capture, and it is the third way this +# script silently lost rows. vcpkg's binary cache is keyed by each port's ABI +# hash; on a hit it unpacks the archive and NEVER RUNS THE PORTFILE, so the port +# asks for none of its downloads and contributes nothing to the capture. Unlike a +# failed fetch or a halt, this leaves no trace at all: no error, no halt, exit 0. +# +# Measured, because I did not believe it was this bad: two runs of a zlib-only +# manifest against a warm ~/.cache/vcpkg/archives, second run with a fresh install +# root. Run 1 built zlib and captured 3 rows. Run 2 printed "Restored 3 +# package(s)", "All requested installations completed successfully in: 1.54 ms", +# exited 0 -- and captured ZERO. The old capture had no --binarysource at all, so +# it inherited whatever cache the machine had; that is enough on its own to +# explain a re-capture that cannot reproduce the committed rows. +# +# It goes BEFORE "$@" so a caller can still override it deliberately (last flag +# wins), and the floor check below is what catches it if they do. +# # `set -euo pipefail` is on, so the pipeline's status is vcpkg's when vcpkg fails # and tee's only when tee does -- without pipefail the tee would swallow a vcpkg # failure outright. "${VCPKG_ROOT:?set VCPKG_ROOT}/vcpkg" install \ --x-asset-sources="clear;x-script,$REC {url} {sha512} {dst}" \ + --binarysource=clear \ "$@" 2>&1 | tee "$VCPKG_LOG" # A halted portfile loses its later downloads exactly like a failed fetch does, @@ -168,6 +186,32 @@ awk ' { print "port " port " halted before finishing its portfile" } ' "$VCPKG_LOG" | sort -u >> "$FAILED" +# The two ways a port can skip its portfile ENTIRELY, and so contribute nothing +# while looking fine. Both are silent -- no error, no halt, exit 0 -- so the log +# is again the only witness. +# +# "Restored N package(s) from " -- a binary-cache hit unpacks an archive +# instead of running the portfile. Guarded above with --binarysource=clear, +# but a caller can override it, so verify the OUTCOME rather than trusting +# the flag. +# "The following packages are already installed" -- an install root that +# already has the port. Nothing is rebuilt, nothing is downloaded. This is +# why a capture wants a FRESH --x-install-root: the 71fb301a capture's +# second run had 7 ports already installed and could not have captured any +# of their distfiles. +if grep -q "^Restored [0-9]* package" "$VCPKG_LOG"; then + grep -o "^Restored [0-9]* package(s) from [^ ]*" "$VCPKG_LOG" \ + | sed 's/$/ -- a cache hit never runs the portfile, so those downloads were never requested/' \ + >> "$FAILED" +fi +if grep -q "The following packages are already installed" "$VCPKG_LOG"; then + n=$(sed -n '/The following packages are already installed/,/^The following packages will be/p' \ + "$VCPKG_LOG" | grep -c "^ *[*]* *[a-z0-9]" || true) + echo "$n package(s) were ALREADY INSTALLED -- their portfiles did not run," \ + "so their downloads were never requested; use a fresh --x-install-root" \ + >> "$FAILED" +fi + rm -f "$REC" # Keep $VCPKG_LOG until the capture is blessed: if it is INCOMPLETE the log is the # evidence for WHY (which port halted, at which portfile line), and deleting it @@ -200,10 +244,11 @@ sort -u -t$'\t' -k1,2 -o "$OUT" "$OUT" if [ -s "$FAILED" ]; then echo "capture: INCOMPLETE -- $(wc -l < "$FAILED") loss(es):" >&2 sed 's/^/capture: /' "$FAILED" >&2 - echo "capture: A failed download, and a portfile halted for any other reason," >&2 - echo "capture: both HALT the portfile, so every later vcpkg_download_distfile" >&2 - echo "capture: in that port was never requested and is MISSING from $OUT." >&2 - echo "capture: Do not emit rules from this file." >&2 + echo "capture: Every one of these means some vcpkg_download_distfile was never" >&2 + echo "capture: REQUESTED, and only requested downloads can be captured: a failed" >&2 + echo "capture: fetch and a refused step both halt the rest of their portfile, and" >&2 + echo "capture: a cache hit or an already-installed port skips the portfile whole." >&2 + echo "capture: So $OUT is missing rows and looks complete. Do not emit from it." >&2 echo "capture: Re-run; it appends and shares downloads/, so it resumes." >&2 echo "capture: A halt on EVERY port means CAPTURE_ONLY_DOWNLOADS was set --" >&2 echo "capture: that mode cannot produce a complete capture; drop it." >&2 diff --git a/tests/test_emit_vcpkg.py b/tests/test_emit_vcpkg.py index a80bd06..73f44da 100644 --- a/tests/test_emit_vcpkg.py +++ b/tests/test_emit_vcpkg.py @@ -701,13 +701,13 @@ def test_a_failed_download_makes_the_capture_fail_because_vcpkg_will_not(): tail = sh.split("chmod +x", 1)[1] assert 'if [ -s "$FAILED" ]' in tail, \ "the driver never checks whether any download failed" - assert re.search(r'if \[ -s "\$FAILED" \][\s\S]{0,800}?exit 1', tail), \ - "a capture with failed downloads still exits 0" + block = tail.split('if [ -s "$FAILED" ]', 1)[1].split("\nfi", 1)[0] + assert "exit 1" in block, "a capture with failed downloads still exits 0" # The message has to say the non-obvious part, or a reader re-runs the emitter # on the incomplete file. - assert re.search(r"HALTS? the portfile|HALTS its portfile", tail), \ + assert re.search(r"halt the rest of their portfile|HALTS its portfile", tail), \ "the message does not explain that later downloads in that port are missing" - assert "never requested" in tail and "MISSING" in tail, \ + assert "never" in tail and "REQUESTED" in tail and "missing rows" in tail, \ "the message does not say the capture is missing rows" @@ -821,3 +821,56 @@ def test_the_capture_does_not_let_tee_swallow_a_vcpkg_failure(): code = "\n".join(l.split("#", 1)[0] for l in sh.splitlines()) assert re.search(r"^set -euo pipefail", code, re.M) or "PIPESTATUS" in code, \ "vcpkg's output is piped into tee with neither pipefail nor a PIPESTATUS check" + + +def test_the_capture_turns_the_binary_cache_off(): + """The third and worst way this script lost rows: a port that never ran. + + vcpkg's binary cache is keyed by each port's ABI hash; on a hit it unpacks the + archive and does NOT run the portfile, so the port requests none of its + downloads. Unlike a failed fetch or a halt this is completely silent: no error, + no halt, exit 0, "All requested installations completed successfully". + + Measured with a zlib-only manifest against a warm ~/.cache/vcpkg/archives: the + first run built zlib and captured 3 rows; the second, with a fresh install root, + printed "Restored 3 package(s)" and captured ZERO while exiting 0. The capture + had no --binarysource at all, so it inherited whatever cache the machine + happened to have -- enough on its own to explain a re-capture that cannot + reproduce the committed rows. vcpkg_build.sh already passed + --binarysource=clear for the adjacent reason (a restore proves nothing about + building from source); the capture never did. + """ + sh = _capture_script() + code = "\n".join(l.split("#", 1)[0] for l in sh.splitlines()) + assert "--binarysource=clear" in code, \ + ("the capture does not disable the binary cache: a restored port skips its " + "portfile and contributes no rows, silently") + # Before "$@", so a caller can still override deliberately -- and the outcome + # check below is what catches them if they do. + install = code.split("/vcpkg\" install", 1)[1].split("\n\n", 1)[0] + assert install.index("--binarysource=clear") < install.index('"$@"'), \ + "--binarysource=clear after \"$@\" would override the caller instead of defaulting" + + +def test_a_port_that_skipped_its_portfile_fails_the_capture(): + """Belt and braces for the flag above: check the OUTCOME, not the intent. + + Two lines in vcpkg's output mean a port contributed nothing: + "Restored N package(s) from " (cache hit) and "The following packages + are already installed" (a reused install root -- the 71fb301a capture's second + run had 7 of those). Both are invisible in the exit code, so both are read off + the teed log and land in $FAILED. + + Verified with fake vcpkgs printing each line and exiting 0: the script exits 1, + and the already-installed case reports the COUNT (2 of the 2 listed). + """ + sh = _capture_script() + code = "\n".join(l.split("#", 1)[0] for l in sh.splitlines()) + assert re.search(r'grep -q "\^Restored \[0-9\]\* package" "\$VCPKG_LOG"', code), \ + "a binary-cache restore is not detected, so an overridden --binarysource is silent" + assert "The following packages are already installed" in code, \ + "a reused install root is not detected: those ports' downloads are missing" + for marker in ("^Restored [0-9]* package", "already installed"): + seg = code.split(marker, 1)[1].split("\nif ")[0] + assert '>> "$FAILED"' in seg, \ + f"a skipped portfile detected via {marker!r} never reaches $FAILED" From e6202a0ce640966d51d2fa42272f7dd752863a76 Mon Sep 17 00:00:00 2001 From: Ulf Adams Date: Wed, 19 Aug 2026 15:01:32 +0000 Subject: [PATCH 47/59] Re-capture the vcpkg pin at 71fb301a: 75 rows reproduce, sdl3 moves as intended Closes the repin's last derived input. The capture is re-taken at 71fb301a and Meta/vcpkg_assets.tsv, vcpkg_distfiles.bzl, vcpkg_index.bzl and MODULE.bazel's use_repo now agree with vcpkg.json: sdl3 is release-3.2.28, not the 3.4.12 the old pin's capture carried. The result is checkable rather than asserted. Of the 76 rows, 75 reproduce the previous capture's (url, sha512) BYTE-FOR-BYTE and the single difference is the intended sdl3 move -- so the 21 other capture-only URLs (ANGLE/WebKit/curl/expat/freetype/sqlite) that todo 37a5e4ee flagged as possibly stale the same way are confirmed NOT stale, which is the part I could not have concluded from the emitter's diagnostics. Independently, every one of the 76 SHA512s was re-hashed against the bytes actually sitting in the downloads dir: 76 ok, 0 mismatches. Two runs were needed, and the second one taught me something. angle was the ONE port whose downloads the Download Mode halt had eaten (derived, not guessed: parsing each halt's own call stack for its portfile frame and grepping that portfile for download calls after the halt line gives exactly one port of 61 halts). But a one-port supplementary manifest resolves its dependencies from the vcpkg BASELINE, not from Ladybird's vcpkg.json overrides -- so it fetched zlib 1.3.2 where Ladybird pins 1.3.1 and the merged capture had both. I deleted that row by hand and then recognised the shape: a hand-fix nothing checks is what the next capture silently repeats, which is the whole subject of this session. classify_static_only could not see it because it only looked one way (derived-but-not-captured). The mirror is just as derivable -- a CAPTURED row in the same URL family at a version the derivation does not pin came from the wrong resolution -- so classify_capture_only now reports it as a LEAKED CAPTURE ROW, and on the 77-row merged file it names zlib 1.3.2 vs the pinned 1.3.1 exactly. Its guard also pins the boundary: a capture-only row with NO derived sibling (angle's expanded ${VAR} URLs) is not a leak, it is the reason the capture replaces the static parse at all. README: gap 10 no longer claims vcpkg_distfiles.bzl fetches 3.4.12 "to this day", and records the one-directional-check lesson. New gap 11 documents the four ways a capture silently loses rows while vcpkg exits 0, including that the warm downloads/ dir the RESUME rule asks you to share is itself one of them -- the property that makes a 50-minute job restartable is the property that makes its output incomplete. --- examples/ladybird/README.md | 68 +++++++- examples/ladybird/workspace/MODULE.bazel | 2 +- .../workspace/Meta/emit_vcpkg_bazel.py | 54 +++++++ .../ladybird/workspace/Meta/vcpkg_assets.tsv | 152 +++++++++--------- .../workspace/Meta/vcpkg_capture_assets.sh | 34 ++++ .../ladybird/workspace/vcpkg_distfiles.bzl | 8 +- examples/ladybird/workspace/vcpkg_index.bzl | 2 +- tests/test_emit_vcpkg.py | 49 ++++++ 8 files changed, 279 insertions(+), 90 deletions(-) diff --git a/examples/ladybird/README.md b/examples/ladybird/README.md index b208af2..d6a3bf7 100644 --- a/examples/ladybird/README.md +++ b/examples/ladybird/README.md @@ -302,7 +302,7 @@ an FFI header collision and an entire missing Rust target. See | `vcpkg.bzl`, `Meta/vcpkg_build.sh` | `vcpkg_tree`: builds the whole dep tree as an ordinary Bazel action with `x-block-origin`, so it reaches the network zero times. Deliberately not a `repository_rule` | | `Meta/vcpkg_tool_assets.tsv` | vcpkg's OWN host tools (cmake, ninja) — url + sha512 + the filename vcpkg looks for. **Separate from the asset capture on purpose:** `vcpkg_find_acquire_program` probes the host first, so a tool the capturing machine already had is never downloaded and never captured (that is how ninja went unpinned; finding 38). Regenerate with `emit_vcpkg_bazel.py --capture-tools` | | `Meta/vcpkg_host_tools.tsv` | The tools that must come **from the host**, because vcpkg has no Linux download for them at all — `nasm` (6 ports) and the autotools set. The third class of input, and the one that cannot be pinned: `vcpkg_find_acquire_program(NASM)` has URLs only inside `if(CMAKE_HOST_WIN32)`, and `vcpkg-make` demands `autoconf`/`automake`/`libtool` via bare `find_program` + `FATAL_ERROR`. So this file does not close the gap, it **names** it — and `vcpkg_build.sh` checks the whole list before building anything, so a machine missing three tools is told all three in one second instead of one per 20-minute build (finding 39). Regenerate with `emit_vcpkg_bazel.py --host-tools` | -| `Meta/vcpkg_capture_assets.sh` | Records the 76-distfile pin, by *being* vcpkg's asset cache. The one run allowed to fetch | +| `Meta/vcpkg_capture_assets.sh` | Records the 76-distfile pin, by *being* vcpkg's asset cache. The one run allowed to fetch. Does a **full build** (~50 min) and passes `--binarysource=clear`, because only a port that actually runs its portfile requests its downloads: a Download-Mode halt, a cache hit, an already-installed port and a warm `downloads/` each lose rows *while vcpkg exits 0* (see Known gaps 13). The script judges its own completeness — vcpkg's exit code cannot | | `Meta/fetch_vcpkg_git_archives.py` | Produces the 4 `vcpkg_from_git` tarballs **without CMake**: takes the list from the committed pin, resolves each clone URL out of the portfiles, then `git clone` + `git -c core.autocrlf=false archive ` and **verifies against the pinned SHA512**. 4/4 byte-identical | | `Meta/vcpkg_capture_git_archives.sh` | Regenerates that pin with `vcpkg install --only-downloads` (~6 min, no compilation, no CMake) — vcpkg as the instrument, since which git externals are used is decided by feature-conditional CMake code, not by portfile text | | `hsts_preload.bzl` | Chromium's HSTS preload table as one `http_file`, pinned to a **commit** + sha256 — the downstream pin for the one input upstream CMake fetches from `main`. **Generated** by `Meta/pin_hsts_preload.py` | @@ -983,13 +983,25 @@ Honest inventory of what stops this from being a clone-and-build. platform". At this pin `vcpkg.json` moved `sdl3` to 3.2.28, the versions-db derivation resolves that correctly, and the message reported the correct current pin being discarded in favour of a stale captured 3.4.12 as a - Windows-only fetch. The checked-in `vcpkg_distfiles.bzl` fetches 3.4.12 to this - day. The classification needs no new input — a dropped row whose *URL family* - the capture also has is the same project at another version, i.e. a stale - capture — and the emitter now says so; re-capturing is filed. **A replace-wins - rule whose message asserts the reason for the replacement instead of checking - it will eventually be confidently wrong**, and that sentence describes every - item in this list. + Windows-only fetch. The classification needs no new input — a dropped row whose + *URL family* the capture also has is the same project at another version, i.e. a + stale capture — and the emitter now says so. The capture has since been re-taken + at this pin and `sdl3` **is** 3.2.28 in `vcpkg_distfiles.bzl`; of the 76 rows, + 75 reproduced the previous capture's `(url, sha512)` byte-for-byte and the one + difference is the intended pin move. **A replace-wins rule whose message asserts + the reason for the replacement instead of checking it will eventually be + confidently wrong**, and that sentence describes every item in this list. + + The check was one-directional, which cost a hand-fix within the hour of writing + it: it compared derived-against-captured only, so a capture with an **extra** row + was invisible. Re-capturing needed a supplementary run for `angle` alone, and a + one-port manifest resolves its dependencies from the vcpkg *baseline* rather than + from Ladybird's `vcpkg.json` overrides — so it fetched zlib 1.3.2 where Ladybird + pins 1.3.1, and the merged capture carried both. I deleted the row by hand, + which is precisely the move this list is about, and it is equally derivable: a + *captured* row in the same URL family at a version the derivation does not pin + came from the wrong resolution. `classify_capture_only` reports it as a LEAKED + CAPTURE ROW. 10. **Two upstreamable Ladybird fixes — down from four, and that is the point of a repin.** Two of the four are **fixed upstream** at this pin (`71fb301a`) and their @@ -1060,3 +1072,43 @@ Honest inventory of what stops this from being a clone-and-build. carry only until upstream fixes it has an `.effect-grep` beside it, and verify falls back to asking whether the *effect* is present before reporting the patch missing. + +11. **A capture can only see the downloads vcpkg actually *requests*, and four + different things stop it requesting them — each while vcpkg exits 0.** This is + the same class as gap 10 but on the input side, and it is worse, because the + failure is a *pin that looks complete*. `Meta/vcpkg_capture_assets.sh` records + every `(url, sha512, dst)` by being vcpkg's asset cache, so a download nobody + asks for leaves no trace at all: + + - a **failed fetch** halts the rest of its portfile — the four WebKit files + `angle` downloads *after* its python venv step were lost this way, and vcpkg + reported "All requested installations completed successfully in: 49 min"; + - **`--only-downloads`** halts every portfile at its first executed step, which + is *before* those same four files (`portfile.cmake:86` vs 123–153). It was the + script's default, under a comment of mine claiming it "is enough because the + asset hook fires during resolution" — a sentence written from intuition. The + committed 76-row capture cannot have been produced that way, and the fast mode + is now opt-in via `CAPTURE_ONLY_DOWNLOADS=1`; + - a **binary-cache hit** unpacks an archive and never runs the portfile at all. + Measured on a zlib-only manifest against a warm cache: first run 3 rows, + second run **0 rows** and exit 0 in 1.54 ms. `--binarysource=clear` is now + mandatory here (`vcpkg_build.sh` had always passed it, for the adjacent + reason); + - an **already-installed port** or a **warm `downloads/`** likewise skips the + request (`-- Using cached gni-to-cmake.py` → no row) — and the warm downloads + dir is exactly what the *resume* rule asks you to share between runs, so the + property that makes a 50-minute job restartable is the property that makes its + output incomplete. + + So the script judges its own completeness, since vcpkg's exit code will not: + every loss lands in one sentinel that refuses to bless the file. The check that + does not depend on my having enumerated the list above is derived from vcpkg's + own log — every download it resolves is announced, so require a captured row for + each announcement, treating an absolute path in `-- Using cached ` as the + `vcpkg_from_git` case that legitimately bypasses the asset cache. What cannot be + done is *triage*: 58 of 77 ports halt harmlessly in download-only mode (at + `vcpkg_cmake_configure`, after every download), and nothing in the log + distinguishes those from `angle`'s mid-sequence halt, because only the portfile + knows whether a download follows. **Any halt therefore fails the capture** — the + alternative is a rule that is right about 57 ports and wrong about the one that + matters. diff --git a/examples/ladybird/workspace/MODULE.bazel b/examples/ladybird/workspace/MODULE.bazel index fbbc8a9..8d0cbe1 100644 --- a/examples/ladybird/workspace/MODULE.bazel +++ b/examples/ladybird/workspace/MODULE.bazel @@ -115,7 +115,7 @@ use_repo( 'vcpkg_libpng_1_6_58_apng_patch_gz_95a6f5bb7148', 'vcpkg_libproxy_libproxy_0_4_18_tar_gz_1148d688a9f0', 'vcpkg_libpsl_public_suffix_list_0ed17e_dat_7969c40b0600', - 'vcpkg_libsdl_org_SDL_release_3_4_12_tar_gz_fc0a55ca01c3', + 'vcpkg_libsdl_org_SDL_release_3_2_28_tar_gz_9e188c992caa', 'vcpkg_libtiff_libtiff_v4_7_2_tar_gz_c4dcde3c79e5', 'vcpkg_libtom_libtommath_v1_3_0_tar_gz_3dbd7053a670', 'vcpkg_libunistring_1_2_tar_xz_5fbb5a0a864d', diff --git a/examples/ladybird/workspace/Meta/emit_vcpkg_bazel.py b/examples/ladybird/workspace/Meta/emit_vcpkg_bazel.py index 6fefa33..d2718f7 100644 --- a/examples/ladybird/workspace/Meta/emit_vcpkg_bazel.py +++ b/examples/ladybird/workspace/Meta/emit_vcpkg_bazel.py @@ -880,6 +880,42 @@ def classify_static_only(distfiles, capture): return platform_only, stale +def classify_capture_only(distfiles, capture): + """Captured rows the DERIVATION contradicts: the mirror of the above. + + classify_static_only looks one way only -- derived-but-not-captured -- and so + cannot see a row the capture has too MANY of. That happened immediately: the + 71fb301a re-capture needed a supplementary run for angle alone (the one port + whose downloads a Download Mode halt had eaten), and a one-port manifest + resolves its own dependencies from the vcpkg BASELINE rather than from + Ladybird's vcpkg.json overrides. So it pulled zlib 1.3.2 where Ladybird pins + 1.3.1, and the capture came back with BOTH. + + I dropped that row by hand and then noticed I was doing the very thing this + session has been about: a hand-fix nothing checks, which the next capture + silently repeats. It is derivable -- the versions-db derivation knows the + pinned version, and a captured row in the same URL family at a version the + derivation does not pin is a row from the wrong resolution. + + Note the asymmetry with classify_static_only: there the CAPTURE wins, because + only vcpkg can expand a portfile's variables. Here the capture is a superset, + and the extra row is not a fetch this build makes -- so it is a leak, not a + pin. Reported, not dropped silently, because "vcpkg asked for it" is exactly + what makes it plausible enough to keep. + + Returns [(capture_sha, derived_sha)] -- the leaked row and the row it shadows. + """ + fams = {} + for sha, (url, _name, _port, _src) in distfiles.items(): + fams.setdefault(_url_family(url), []).append(sha) + leaked = [] + for sha in sorted(set(capture) - set(distfiles)): + same = fams.get(_url_family(capture[sha][0])) + if same: + leaked.append((sha, sorted(same)[0])) + return leaked + + def observed_git_archives(downloads_dir, distfiles): """The archives in a completed run's downloads/ that the asset cache never saw. @@ -1134,6 +1170,7 @@ def main(): # asked for on this platform" is how sdl3 3.2.28 got silently replaced by # a captured 3.4.12 for a whole pin. platform_only, stale = classify_static_only(distfiles, cap) + leaked = classify_capture_only(distfiles, cap) sys.stderr.write("capture: %d distfiles (static parse had %d)\n" % (len(cap), len(distfiles))) if platform_only: @@ -1155,6 +1192,23 @@ def main(): sys.stderr.write(" %-12s pinned %s\n" % (distfiles[sha][2], distfiles[sha][1])) sys.stderr.write(" %-12s captured %s\n" % ("", cap[cap_sha][1])) + if leaked: + sys.stderr.write( + " LEAKED CAPTURE ROWS: %d row(s) at a version this manifest does " + "not pin.\n" + " The capture has BOTH versions of these, so a run in it resolved " + "against a\n" + " different manifest -- a one-port supplementary capture takes its " + "deps from the\n" + " vcpkg BASELINE, not from Ladybird's vcpkg.json overrides. The " + "emitted rules\n" + " below therefore fetch a distfile no port in this build asks for. " + "Delete the\n" + " captured row, or re-capture with the real manifest.\n" % len(leaked)) + for cap_sha, sha in leaked: + sys.stderr.write(" %-12s captured %s\n" % (distfiles[sha][2], + cap[cap_sha][1])) + sys.stderr.write(" %-12s pinned %s\n" % ("", distfiles[sha][1])) distfiles, unexpanded = cap, [] # vcpkg's OWN tools are unioned in from their COMMITTED pin, not replaced # and not re-derived: they are a different class from port distfiles, and diff --git a/examples/ladybird/workspace/Meta/vcpkg_assets.tsv b/examples/ladybird/workspace/Meta/vcpkg_assets.tsv index f5a52bb..335d95f 100644 --- a/examples/ladybird/workspace/Meta/vcpkg_assets.tsv +++ b/examples/ladybird/workspace/Meta/vcpkg_assets.tsv @@ -1,76 +1,76 @@ -https://chrome-infra-packages.appspot.com/dl/gn/gn/linux-amd64/+/fj2NZKMkIYZNH6uYG0bn8OsW_lZB5JKz3JeScMCLAGQC d49575bd383b6aace1257a6e9439ce0a206173ec2cab94d5312f06db412e09c89aa75b1f4c69f5dca4389d15a489c211a73439a66f437c34b18bc90eefa0b775 gn-linux-amd64.zip -https://downloads.sourceforge.net/project/libpng-apng/libpng16/1.6.58/libpng-1.6.58-apng.patch.gz 95a6f5bb7148b5c48dccd73811d7bcf9752a631a7bb4f4856670a7da12a7159581ac1bce1749318343794e0f5cb86972711ba2ec0f523c168f0991fa940687d5 libpng-1.6.58-apng.patch.gz.25974.part -https://ftpmirror.gnu.org/gnu/automake/automake-1.17.tar.gz 11357dfab8cbf4b5d94d9d06e475732ca01df82bef1284888a34bd558afc37b1a239bed1b5eb18a9dbcc326344fb7b1b301f77bb8385131eb8e1e118b677883a automake-1.17.tar.gz -https://ftpmirror.gnu.org/gnu/gperf/gperf-3.3.tar.gz 246b75b8ce7d77d6a8725cd15f1cf2e68da404812573af1d5bf32dbe6ad4228f48757baefc77bcb1f5597c2397043c04d31d8a04ab507bfa7a80f85e1ab6045f gperf-3.3.tar.gz -https://github.com/AOMediaCodec/libavif/archive/v1.4.2.tar.gz cbf31827884058acc54d3b1fa0f2059f022691609a76e0f913981d5b8c1be60f52069a25b61c888c09e963161d1f4c9bf692c0716ab0ee067a4ccaa4e36d9ce1 AOMediaCodec-libavif-v1.4.2.tar.gz -https://github.com/GNOME/libxml2/archive/v2.15.3.tar.gz f65df793fca5e46552afbaa56b04c4774829a95e012a6dc4dc3d10e6884a6118e30a426e422516e1d21e91c4a1d34cb00a5cee61af35cab03c71fb9b5c09e138 GNOME-libxml2-v2.15.3.tar.gz -https://github.com/GPUOpen-LibrariesAndSDKs/VulkanMemoryAllocator/archive/v3.4.0.tar.gz 563acbcd8912d10d92c23715eba7acf0e7c1683af36021f415b36f359c2cce065f3906e395c32282a8410ec5c8179fbcb6412935c6629a49357475d4b4410e2a GPUOpen-LibrariesAndSDKs-VulkanMemoryAllocator-v3.4.0.tar.gz -https://github.com/KhronosGroup/EGL-Registry/archive/3ae2b7c48690d2ce13cc6db3db02dfc0572be65e.tar.gz c7b09ded4964fa427546bd345a29325105b79079b59642214dc8f04de113f42de2bc4272dbbbd4a801d92afc20297442fdfa12043a0900cf1e2b1cd83f260883 KhronosGroup-EGL-Registry-3ae2b7c48690d2ce13cc6db3db02dfc0572be65e.tar.gz.21812.part -https://github.com/KhronosGroup/OpenGL-Registry/archive/0b449b97cdf1043eef5e1f0e235cbbab6ec10c86.tar.gz 148e1bfe4cc199bcc2c23b22d0b3e4988a29389d7f510ba4a6340672dbb7ab99bb836d2c08587499484df704d51a1adf4f0dc3a30d5ad8977ee0ad339163b17e KhronosGroup-OpenGL-Registry-0b449b97cdf1043eef5e1f0e235cbbab6ec10c86.tar.gz.21854.part -https://github.com/KhronosGroup/Vulkan-Headers/archive/vulkan-sdk-1.4.350.1.tar.gz eee702ee5ff447986901e44989d96a5e38f7ef7f67e2015c8db6280251cb23b69cb9e1a0568b0a43dc3df820685079ed31998a1dc4704f5649768e26db5147b1 KhronosGroup-Vulkan-Headers-vulkan-sdk-1.4.350.1.tar.gz -https://github.com/Kitware/CMake/releases/download/v4.4.0/cmake-4.4.0-linux-x86_64.tar.gz 3df4aaa128a438ed48dcac7065fd355ff538eed8f394491298d0db63a891d671da247c8fa262e4fa6bf99429d630abab317d5a0248168fe203d1ca4978dab4da cmake-4.4.0-linux-x86_64.tar.gz.20596.part -https://github.com/NixOS/patchelf/releases/download/0.19.0/patchelf-0.19.0-x86_64.tar.gz 2a65c9cbdddcc7952cdbd6e98a2cf3da01386cf0f0b927a6bbcfe8131ecf0bfb17c534246635b5e6a090652ee54c903f9f9c4f3f1d2412dba59f287ae2ae8070 patchelf-0.19.0-x86_64.tar.gz.21729.part -https://github.com/WebKit/WebKit/raw/0742522b24152262b04913242cb0b3c48de92ba0/Source/ThirdParty/ANGLE/gni-to-cmake.py cf9dab1b59cd44b9ce05a2dc4636115e770dfe7d5cfbd90c3fef659628ac8155c57b866480f7cfe9a0afeb31ff5ce5eed74473f386120a7fc5910e8b032bd61d gni-to-cmake.py.22348.part -https://github.com/WebKit/WebKit/raw/0742522b24152262b04913242cb0b3c48de92ba0/Source/ThirdParty/ANGLE/include/CMakeLists.txt a7ddf3c6df7565e232f87ec651cc4fd84240b8866609e23e3e6e41d22532fd34c70e0f3b06120fd3d6d930ca29c1d0d470d4c8cb7003a66f8c1a840a42f32949 include_CMakeLists.txt.22445.part -https://github.com/WebKit/WebKit/raw/0742522b24152262b04913242cb0b3c48de92ba0/Source/cmake/DetectSSE2.cmake 219a4c8591ee31d11eb3d1e4803cc3c9d4573984bb25ecac6f2c76e6a3dab598c00b0157d0f94b18016de6786e49d8b29a161693a5ce23d761c8fe6a798c1bca DetectSSE2.cmake.22467.part -https://github.com/WebKit/WebKit/raw/0742522b24152262b04913242cb0b3c48de92ba0/Source/cmake/WebKitCompilerFlags.cmake 8b281ffcf9209c845a5fdae48a4e05f08ca677c37a7fb00d9270de81bd103160d26e091724ce8df8d428ad604900b5202b221fed5bafffd4bf00025718ef9d8e WebKitCompilerFlags.cmake.22460.part -https://github.com/WebKit/WebKit/raw/0742522b24152262b04913242cb0b3c48de92ba0/Source/cmake/WebKitMacros.cmake 565175443d5d1b8119af504164bf93840e8c786fc479e45feb98ca542351b91d2ea00265d7f8dfa6960975de81802bc43b2a2af2c90fc44bda1ae46d96c89247 WebKitMacros.cmake.22482.part -https://github.com/abseil/abseil-cpp/archive/20260107.1.tar.gz f5012885d6b6844a9cf5ed92ad5468b8757db33dfe1364bfb232fff928e06c550c7eb4557f45186a8ac4d18b178df9be267681abab4a6de40823b574afbe9960 abseil-abseil-cpp-20260107.1.tar.gz -https://github.com/cisco/openh264/archive/v2.6.0.tar.gz 26a03acde7153a6b40b99f00641772433a244c72a3cc4bca6d903cf3b770174d028369a2fb73b2f0774e1124db0e269758eed6d88975347a815e0366c820d247 cisco-openh264-v2.6.0.tar.gz -https://github.com/curl/curl/archive/curl-8_21_0.tar.gz 0ab6c99c3d5b86fb65c526db517c3159b11db2f8d82552d635c4887059c0602288603c93b754ce0ec543ea2f275122ccec2c8dcd866c2611b5b949c728ee72df curl-curl-curl-8_21_0.tar.gz.8039.part -https://github.com/davea42/libdwarf-code/archive/v2.3.1.tar.gz e8eb74c622cedb512d82dff313ce9c5ac2064a7a7a0691c776213b384c1c64d0f549aaab1ef732dcba8c72d52048cea641d9909d3b2503cf96760ad8e81ae77c davea42-libdwarf-code-v2.3.1.tar.gz.30028.part -https://github.com/facebook/zstd/archive/v1.5.7.tar.gz 26e441267305f6e58080460f96ab98645219a90d290a533410b1b0b1d2f870721c95f8384e342ee647c5e968385a5b7e30c2d04340c37f59b3e6d86762c3260c facebook-zstd-v1.5.7.tar.gz.28957.part -https://github.com/fastfloat/fast_float/archive/v8.2.10.tar.gz eec563358117b593e882a9069074a269d811e8989457a0d57e50c5d2f4d534b6820308082bb70c0a8a8388513e92b64f12c5213389eff726ad0483eacba961ff fastfloat-fast_float-v8.2.10.tar.gz.19662.part -https://github.com/ffmpeg/ffmpeg/archive/n7.1.1.tar.gz 6b9a5ee501be41d6abc7579a106263b31f787321cbc45dedee97abf992bf8236cdb2394571dd256a74154f4a20018d429ae7e7f0409611ddc4d6f529d924d175 ffmpeg-ffmpeg-n7.1.1.tar.gz -https://github.com/fmtlib/fmt/archive/12.2.0.tar.gz 5ac2ba0f54a484999ed5407d82b77aad170cea49a267decd2c0eedadf3b14413e2a83fcc8e9ca9c16640595e019b8636e160f72314d8be50653324e82ac745eb fmtlib-fmt-12.2.0.tar.gz -https://github.com/fmtlib/fmt/commit/588b3a0f8f6a8bcf2a959cae882d5b2703e86737.patch?full_index=1 afda8fdfcdcb4b0dd5df4d4dae96a57a85fb9c4b65d0b49d51258f0913d4aed93ed146ebf96ed7b277490b1dde6c7117f43332013071441a96c3147520de8368 fmt-backport-4813.patch -https://github.com/google/angle/archive/79ac1a8cd767a32cce6401203e20c4bd4ca4d539.tar.gz a3d5d09460f05f1f1c081411bf15f585bb8add5f3db7c4854dc2cf2ef1fd22dd5b2dd5d3de78bfb8bcdae8459dd9581f7dce331acd34053c87c84cfe6c3d134c google-angle-79ac1a8cd767a32cce6401203e20c4bd4ca4d539.tar.gz.22306.part -https://github.com/google/brotli/archive/v1.2.0.tar.gz f94542afd2ecd96cc41fd21a805a3da314281ae558c10650f3e6d9ca732b8425bba8fde312823f0a564c7de3993bdaab5b43378edab65ebb798cefb6fd702256 google-brotli-v1.2.0.tar.gz.28349.part -https://github.com/google/highway/archive/1.4.0.tar.gz 819422857d6a74e3a936c402698e078db5b7b88fb43767e62429ec7bd954fe93b017e75029a4df4a1a97ef3a2486107eef5247da751eb487640dea409f3f2fa2 google-highway-1.4.0.tar.gz -https://github.com/google/skia/archive/e7c90ecca9444fe09598f1630ab7cee2c0ee027a.tar.gz f52286fcac1d1b2b45046d7df72cee384b664f0b512c4b4a76bca743747f2113fef6991059b4e71ad804d0209103003ba0f01aa90afac786de20a52e8484018d google-skia-e7c90ecca9444fe09598f1630ab7cee2c0ee027a.tar.gz -https://github.com/google/woff2/archive/v1.0.2.tar.gz c788bba1530aec463e755e901f9342f4b599e3a07f54645fef1dc388ab5d5c30625535e5dd38e9e792e04a640574baa50eeefb6b7338ab403755f4a4e0c3044d google-woff2-v1.0.2.tar.gz -https://github.com/google/wuffs-mirror-release-c/archive/v0.3.4.tar.gz d22136a1adf337573944eed917142c6bde877a09bd65738010c6c367c7a3fc9e4573fd4dc8469c93799fe1e3247760e65e64829e830871ee7b333fd72ebc629d google-wuffs-mirror-release-c-v0.3.4.tar.gz -https://github.com/harfbuzz/harfbuzz/archive/10.2.0.tar.gz 697205a571bb3d52d83598e8511e2e21e7cd15630aac32d8deb4354e462efda6a5ce46510cf4a1c18365dffe935cf2e4f1fda65d1779f17c9bb60c503315bf5c harfbuzz-harfbuzz-10.2.0.tar.gz -https://github.com/jeremy-rifkin/cpptrace/archive/v1.0.2.tar.gz 4ae394fb3c21149bf2441a754eebe639e6a5534927426b6507806c7bee0b1c982e047c972904d472f1c660adb5be3881e7e3a6eddd18e4e9d376ae3855d50a7c jeremy-rifkin-cpptrace-v1.0.2.tar.gz.31337.part -https://github.com/libexpat/libexpat/archive/R_2_8_2.tar.gz e60e6d6ae9d0115f41186f06f3854008054863f0b29a58d44142f5b30057a494337145d820b5e18270b8ca3e779e318a757fcc6bf64d6d204e5498df5eeb2195 libexpat-libexpat-R_2_8_2.tar.gz.13196.part -https://github.com/libjpeg-turbo/libjpeg-turbo/archive/3.2.0.tar.gz 1ce063e9e126d55019385da3a6ff4521a9a3958edeab15e2465ae3435026dac6e598277eea066173ada47da37159d0f7812b2227869c10dd141dcfe2ddeab720 libjpeg-turbo-libjpeg-turbo-3.2.0.tar.gz.21972.part -https://github.com/libjxl/libjxl/archive/v0.11.2.tar.gz a7e1f7d060b358f4382e84367d66aa2850aef3b4524a0fdfe3f22dd258fb9e35dda7540f859d8bf4c32f31c61a7a03db677f4490a9f472cd25869a9d00797336 libjxl-libjxl-v0.11.2.tar.gz -https://github.com/libproxy/libproxy/archive/0.4.18.tar.gz 1148d688a9f070273a1a2b110a788561789799089660292bbba59fbf0a9caf7d28cb039a9ccdcb935f752e1e34739b2d2f4c784b1bb3bbaa03d108e7b38a4754 libproxy-libproxy-0.4.18.tar.gz -https://github.com/libsdl-org/SDL/archive/release-3.4.12.tar.gz fc0a55ca01c32f613b9cd8c6cffad17ee0855ee542f03fa455632043c99a0a259599557a46c7c636032444260e64476867d9af43a1da689837f99c22942cd863 libsdl-org-SDL-release-3.4.12.tar.gz -https://github.com/libtom/libtommath/archive/v1.3.0.tar.gz 3dbd7053a670afa563a069a9785f1aa4cab14a210bcd05d8fc7db25bd3dcce36b10a3f4f54ca92d75a694f891226f01bdf6ac15bacafeb93a8be6b04c579beb3 libtom-libtommath-v1.3.0.tar.gz -https://github.com/madler/zlib/archive/v1.3.1.tar.gz 8c9642495bafd6fad4ab9fb67f09b268c69ff9af0f4f20cf15dfc18852ff1f312bd8ca41de761b3f8d8e90e77d79f2ccacd3d4c5b19e475ecf09d021fdfe9088 madler-zlib-v1.3.1.tar.gz.21336.part -https://github.com/mesonbuild/meson/archive/1.9.0.tar.gz ecd69b6734be14c467f7db67dd88c0e57ebfad83ce3ddada131ff3e43ac964523e1083d7c7f3803033a9a76adbc32ad26dd2e3aca69884269000ca64130bde07 meson-1.9.0.tar.gz -https://github.com/microsoft/mimalloc/archive/v2.2.7.tar.gz 19f5481c98822c6a279e2f2d26d6c54918e18bcafd515a76ca3a3845f3a5e599236ea042f9cadaf793dce0c1ede347d76cc2fc5503094288362a78034d4d8f71 microsoft-mimalloc-v2.2.7.tar.gz -https://github.com/mm2/Little-CMS/archive/lcms2.19.1.tar.gz 1b2781ed8898e65f15be17cf0130a1500ec0bf5ca5159f871dff5692e387747be9526feef0bc7c370200656fc0aabe3036746041285a3978e90adec200d685f2 mm2-Little-CMS-lcms2.19.1.tar.gz -https://github.com/mozilla/pdf.js/releases/download/v5.6.205/pdfjs-5.6.205-dist.zip 66fecdb8a80d013b592c361e85abd7eeea5bc35f0131b771cc25a1878d7737704458dc715670aace3c7d3437f85388dff04796fe117eef01be81fd0d192f73a2 pdfjs-5.6.205-dist.zip -https://github.com/nghttp2/nghttp2/archive/v1.69.0.tar.gz 1029fb86935a88fc518cc2d976dff5253277f97e01f32b1f73c5df96dcf7fe0280a83a9e9d676c3d96caa542300dc7cfa61f2a40b1b11d2c2b527e870a974b53 nghttp2-nghttp2-v1.69.0.tar.gz.7004.part -https://github.com/ngtcp2/nghttp3/archive/v1.17.0.tar.gz 23d85a2abfa81433049d7b1b0440b5b04ae3515830db0347da335a30b93c8be8e25d2f73e198cdb9207e2b51ce924f133bab7314cde76d80f73c51c1fd1c36c4 ngtcp2-nghttp3-v1.17.0.tar.gz.6300.part -https://github.com/ngtcp2/ngtcp2/archive/v1.24.0.tar.gz 04a5762d6eac7227431eb8e293d2786dee9f7d0467a584bdd35f4e33c7c6ff6e7b010e6e894fabdf780f6377373629c10a4d41effa374e2129d9f9b1e537873e ngtcp2-ngtcp2-v1.24.0.tar.gz.5038.part -https://github.com/ngtcp2/sfparse/archive/f2046eaa1acba7c5467399b1e1e1f354d22d1f48.tar.gz b3cbcce6d96dc731d21a67940a05c1603d43ee9766819e1d174ca93c7afc7fa6247aa818368cafbb31444e0198ab5bba45808772a3ca89e459f8d2c0ffae6f5d ngtcp2-sfparse-f2046eaa1acba7c5467399b1e1e1f354d22d1f48.tar.gz.6318.part -https://github.com/openssl/openssl/archive/openssl-3.6.3.tar.gz a89c08101fa1d7e3c09b14f4a90d450bcf336a4f6a3e6e4ea990e4deddcd9ce250472f9114438fd134ff4b47fe93dd47232308567088b2b1c0b2eb50e3b56bdf openssl-openssl-openssl-3.6.3.tar.gz.32544.part -https://github.com/pkgconf/pkgconf/archive/pkgconf-3.0.3.tar.gz 4e49bb7b10c6fdebfa1175b4c33138b8aeca2582e49adb5c7ba66d08e8614771ef060b0c2673bd098c1aed821c8ade1a6dd276c1f4ac6e29bd7efcb8f4c0402e pkgconf-pkgconf-pkgconf-3.0.3.tar.gz -https://github.com/pnggroup/libpng/archive/v1.6.58.tar.gz 65f54d805e1f7c46a5fc335b984e4cbd4f934e0f02fbf6673c13800b49a4c11fbeb4098eebfb33079527a56c3d933e97631f91ab68dbb31442982784f9241ace pnggroup-libpng-v1-65f54d80.6.58.tar.gz -https://github.com/rockdaboot/libpsl/archive/0.21.5.tar.gz d8e224b2ce5d9a6ac78700eb8975d09aef4e5af7db29539e5e339c5cd100f1272371fe45757ab5383ddbcd569bdf9d697a78932ea9fdf43ff48d3cea02f644cd rockdaboot-libpsl-0.21.5.tar.gz -https://github.com/simdjson/simdjson/archive/v4.6.4.tar.gz 003b96daab30ccaefdac60a9676cf623af5a8662016f988fc0ecaa2f36d8b48ba97f2431e6498dd16fc2b1d841798b2d06dabb0b7487efd4520c0de260e49056 simdjson-simdjson-v4.6.4.tar.gz -https://github.com/simdutf/simdutf/archive/v9.0.0.tar.gz 0c74226247cbe95368efa87ab84f5217485f16bcdf7a9def8741c6086cb86e6c378f0c437030d2be0934726e3ea9c28b5df2e593d0c654c78291c455a8d1e103 simdutf-simdutf-v9.0.0.tar.gz -https://github.com/tukaani-project/xz/archive/v5.8.3.tar.gz 8fb5e6a13397d259d8ff7484f9b63f8a6752ff1c63e1a4601170ad8175aadefb5126a1cae7f73370bfc6c2a0b4e1c0bad57a58fc5b781d3f7d45e5a483c091cc tukaani-project-xz-v5.8.3.tar.gz.19746.part -https://github.com/unicode-org/icu/releases/download/release-78.3/icu4c-78.3-sources.tgz 04a49455e1489030c520a4bfd2664fa2171e7938d08f2acdbbcb1fda976639fd8b1f0704f2eec89ba59a7b6d118ceaab6ec5a096e40d9085a0895d91ce225245 icu4c-78.3-sources.tgz -https://github.com/videolan/dav1d/archive/1.5.3.tar.gz 8d976b93135213d41385c20205475269a6826a68ebfd716c4d9a7a3ff2a79703e8df0573e43207c81b5db44807d2721db18ec84c0fc6bef98efab86a2cccb6cc videolan-dav1d-1.5.3.tar.gz -https://github.com/webmproject/libvpx/archive/v1.16.0.tar.gz 07f5e352411d6c0be331706d1835ac89bafbeddcbbac5542b473323766e9e974f4f68b33590f2aa50a7d8d69468a642b508cbb0a7c49a82c9933b07820f9c9d9 webmproject-libvpx-v1-07f5e352.16.0.tar.gz -https://github.com/webmproject/libwebp/archive/v1.6.0.tar.gz 298e0ad4c09392213baf5abb69d330c6203b618800073fe2df91d01d35034197c5d3e29a74573b06971473c52c74514f0e6e0f6c8162f923e2dd15cb1a692aef webmproject-libwebp-v1-298e0ad4.6.0.tar.gz -https://github.com/xiph/ogg/archive/v1.3.6.tar.gz c247e1da8b12f8b33272fafb6d7c171a1a2687c3632977439fa60b96ccc2ad751d88a2931bb3e18e1ddf2eea2e82cdd0aab087b2ec5393a9228c703476fa0167 xiph-ogg-v1.3.6.tar.gz -https://github.com/xiph/opus/archive/v1.5.2.tar.gz 4ffefd9c035671024f9720c5129bfe395dea04f0d6b730041c2804e89b1db6e4d19633ad1ae58855afc355034233537361e707f26dc53adac916554830038fab xiph-opus-v1.5.2.tar.gz -https://github.com/xiph/theora/archive/v1.2.0.tar.gz b2aac15528f0ef8258c0902e33e8211e8858c3c7e6e9eeb708cce5922de5f0e412255ddaf540a50c0ebf601df6c4376fd24a0bdd7f8de4432c4ae6e5d6ffe2b6 xiph-theora-v1.2.0.tar.gz -https://github.com/xiph/vorbis/archive/v1.3.7.tar.gz bfb6f5dbfd49ed38b2b08b3667c06d02e68f649068a050f21a3cc7e1e56b27afd546aaa3199c4f6448f03f6e66a82f9a9dc2241c826d3d1d4acbd38339b9e9fb xiph-vorbis-v1.3.7.tar.gz -https://gitlab.com/libtiff/libtiff/-/archive/v4.7.2/libtiff-v4.7.2.tar.gz c4dcde3c79e5d69c7231f8862e2e5a83d90d9cce694fb2a4804800b2f8f1bc9db504b9252d81dce872eec8358b33a3a1dbdddcbb6181f6fb8d1d7fc0e9a9fc6a libtiff-libtiff-v4.7.2.tar.gz.24666.part -https://gitlab.freedesktop.org//dbus/dbus/-/archive/dbus-1.16.2/dbus-dbus-1.16.2.tar.gz 8ad3ab55bf6e2bbe6ff871302c2840c0cb82b4ec785b05f146c577ca1e931825084012ac90251e28c30e44d111e5ca5711b29349f4f0e68a09ba49392e63ac89 dbus-dbus-dbus-1.16.2.tar.gz.14029.part -https://gitlab.freedesktop.org//freetype/freetype/-/archive/VER-2-13-3/freetype-VER-2-13-3.tar.gz fccfaa15eb79a105981bf634df34ac9ddf1c53550ec0b334903a1b21f9f8bf5eb2b3f9476e554afa112a0fca58ec85ab212d674dfd853670efec876bacbe8a53 freetype-freetype-VER-2-13-3.tar.gz -https://gitlab.freedesktop.org/fontconfig/fontconfig/-/archive/2.17.1/fontconfig-2.17.1.tar.gz 8e05cad63cd0c5ca15d1359e19a605912198fcc0ec6ecc11d5a0ef596d72e795cd8128e4d350716e63cbc01612c3807b1455b8153901333790316170c9ef8e75 fontconfig-fontconfig-2.17.1.tar.gz -https://raw.githubusercontent.com/publicsuffix/list/0ed17ee161ed2ae551c78f3b399ac8f2724d2154/public_suffix_list.dat 7969c40b0600baf2786af0e6503b4282d487b6603418c41f28c3b39e9cd9320ac66c0d2e8fbfa2b794e461f26843e3479d60ec24ac5c0990fe8f0c6bfaeee69d libpsl-public_suffix_list-0ed17e.dat -https://sourceforge.net/projects/giflib/files/giflib-6.1.3.tar.gz/download fb1d6319694745e8cdac7c57e96bd3a87dbfd978f2bfd00e826742db53398011c43f9a6e7f4375b0e77b162358ddfa14d85bce652680fb5967b72c46775c0edb giflib-6-fb1d6319.1.3.tar.gz -https://sourceforge.net/projects/libuuid/files/libuuid-1.0.3.tar.gz/download 77488caccc66503f6f2ded7bdfc4d3bc2c20b24a8dc95b2051633c695e99ec27876ffbafe38269b939826e1fdb06eea328f07b796c9e0aaca12331a787175507 libuuid-1.0.3.tar.gz -https://sourceware.org/pub/bzip2/bzip2-1.0.8.tar.gz 083f5e675d73f3233c7930ebe20425a533feedeaaa9d8cc86831312a6581cefbe6ed0d08d2fa89be81082f2a5abdabca8b3c080bf97218a1bd59dc118a30b9f3 bzip2-1.0.8.tar.gz -https://sqlite.org/2026/sqlite-autoconf-3530300.tar.gz 355a8db490ec2a68c2801644e56178a26416c355792586a6c1c904de116e26f8602bc344e7172181c9d92c4c9e696319243e16405460fad87b23ee997a3ef9da sqlite-autoconf-3530300.tar.gz -https://thrysoee.dk/editline/libedit-20240808-3.1.tar.gz b11d64947f9484bb2320b0fbcfdc94466993af1dfa0d853853b73c222e95d6c1e78d88d0c305929b95bf7a85009129475c9fef0ac8595b43d75543d85052a4ff libedit-20240808-3.1.tar.gz -https://www.mirrorservice.org/sites/ftp.gnu.org/gnu/libidn/libidn2-2.3.7.tar.gz eab5702bc0baed45492f8dde43a4d2ea3560ad80645e5f9e0cfa8d3b57bccd7fd782d04638e000ba07924a5d9f85e760095b55189188c4017b94705bef9b4a66 libidn2-2.3.7.tar.gz -https://www.mirrorservice.org/sites/ftp.gnu.org/gnu/libunistring/libunistring-1.2.tar.xz 5fbb5a0a864db73a6d18cdea7b31237da907fff0ef288f3a8db6ebdba8ef61ad8855e5fc780c2bbf632218d8fa59dd119734e5937ca64dc77f53f30f13b80b17 libunistring-1.2.tar.xz +https://chrome-infra-packages.appspot.com/dl/gn/gn/linux-amd64/+/fj2NZKMkIYZNH6uYG0bn8OsW_lZB5JKz3JeScMCLAGQC d49575bd383b6aace1257a6e9439ce0a206173ec2cab94d5312f06db412e09c89aa75b1f4c69f5dca4389d15a489c211a73439a66f437c34b18bc90eefa0b775 /home/ubuntu/vcap-downloads/gn-linux-amd64.zip.5420.part +https://downloads.sourceforge.net/project/libpng-apng/libpng16/1.6.58/libpng-1.6.58-apng.patch.gz 95a6f5bb7148b5c48dccd73811d7bcf9752a631a7bb4f4856670a7da12a7159581ac1bce1749318343794e0f5cb86972711ba2ec0f523c168f0991fa940687d5 /home/ubuntu/vcap-downloads/libpng-1.6.58-apng.patch.gz.23712.part +https://ftpmirror.gnu.org/gnu/automake/automake-1.17.tar.gz 11357dfab8cbf4b5d94d9d06e475732ca01df82bef1284888a34bd558afc37b1a239bed1b5eb18a9dbcc326344fb7b1b301f77bb8385131eb8e1e118b677883a /home/ubuntu/vcap-downloads/automake-1.17.tar.gz.29261.part +https://ftpmirror.gnu.org/gnu/gperf/gperf-3.3.tar.gz 246b75b8ce7d77d6a8725cd15f1cf2e68da404812573af1d5bf32dbe6ad4228f48757baefc77bcb1f5597c2397043c04d31d8a04ab507bfa7a80f85e1ab6045f /home/ubuntu/vcap-downloads/gperf-3.3.tar.gz.29537.part +https://github.com/AOMediaCodec/libavif/archive/v1.4.2.tar.gz cbf31827884058acc54d3b1fa0f2059f022691609a76e0f913981d5b8c1be60f52069a25b61c888c09e963161d1f4c9bf692c0716ab0ee067a4ccaa4e36d9ce1 /home/ubuntu/vcap-downloads/AOMediaCodec-libavif-v1.4.2.tar.gz.32757.part +https://github.com/GNOME/libxml2/archive/v2.15.3.tar.gz f65df793fca5e46552afbaa56b04c4774829a95e012a6dc4dc3d10e6884a6118e30a426e422516e1d21e91c4a1d34cb00a5cee61af35cab03c71fb9b5c09e138 /home/ubuntu/vcap-downloads/GNOME-libxml2-v2.15.3.tar.gz.2924.part +https://github.com/GPUOpen-LibrariesAndSDKs/VulkanMemoryAllocator/archive/v3.4.0.tar.gz 563acbcd8912d10d92c23715eba7acf0e7c1683af36021f415b36f359c2cce065f3906e395c32282a8410ec5c8179fbcb6412935c6629a49357475d4b4410e2a /home/ubuntu/vcap-downloads/GPUOpen-LibrariesAndSDKs-VulkanMemoryAllocator-v3.4.0.tar.gz.5104.part +https://github.com/KhronosGroup/EGL-Registry/archive/3ae2b7c48690d2ce13cc6db3db02dfc0572be65e.tar.gz c7b09ded4964fa427546bd345a29325105b79079b59642214dc8f04de113f42de2bc4272dbbbd4a801d92afc20297442fdfa12043a0900cf1e2b1cd83f260883 /tmp/vcap/downloads/KhronosGroup-EGL-Registry-3ae2b7c48690d2ce13cc6db3db02dfc0572be65e.tar.gz.4769.part +https://github.com/KhronosGroup/OpenGL-Registry/archive/0b449b97cdf1043eef5e1f0e235cbbab6ec10c86.tar.gz 148e1bfe4cc199bcc2c23b22d0b3e4988a29389d7f510ba4a6340672dbb7ab99bb836d2c08587499484df704d51a1adf4f0dc3a30d5ad8977ee0ad339163b17e /tmp/vcap/downloads/KhronosGroup-OpenGL-Registry-0b449b97cdf1043eef5e1f0e235cbbab6ec10c86.tar.gz.4954.part +https://github.com/KhronosGroup/Vulkan-Headers/archive/vulkan-sdk-1.4.350.1.tar.gz eee702ee5ff447986901e44989d96a5e38f7ef7f67e2015c8db6280251cb23b69cb9e1a0568b0a43dc3df820685079ed31998a1dc4704f5649768e26db5147b1 /home/ubuntu/vcap-downloads/KhronosGroup-Vulkan-Headers-vulkan-sdk-1.4.350.1.tar.gz.5226.part +https://github.com/Kitware/CMake/releases/download/v4.4.0/cmake-4.4.0-linux-x86_64.tar.gz 3df4aaa128a438ed48dcac7065fd355ff538eed8f394491298d0db63a891d671da247c8fa262e4fa6bf99429d630abab317d5a0248168fe203d1ca4978dab4da /tmp/vcap/downloads/cmake-4.4.0-linux-x86_64.tar.gz.3336.part +https://github.com/NixOS/patchelf/releases/download/0.19.0/patchelf-0.19.0-x86_64.tar.gz 2a65c9cbdddcc7952cdbd6e98a2cf3da01386cf0f0b927a6bbcfe8131ecf0bfb17c534246635b5e6a090652ee54c903f9f9c4f3f1d2412dba59f287ae2ae8070 /tmp/vcap/downloads/patchelf-0.19.0-x86_64.tar.gz.4596.part +https://github.com/WebKit/WebKit/raw/0742522b24152262b04913242cb0b3c48de92ba0/Source/ThirdParty/ANGLE/gni-to-cmake.py cf9dab1b59cd44b9ce05a2dc4636115e770dfe7d5cfbd90c3fef659628ac8155c57b866480f7cfe9a0afeb31ff5ce5eed74473f386120a7fc5910e8b032bd61d /home/ubuntu/vcap-downloads/gni-to-cmake.py.18491.part +https://github.com/WebKit/WebKit/raw/0742522b24152262b04913242cb0b3c48de92ba0/Source/ThirdParty/ANGLE/include/CMakeLists.txt a7ddf3c6df7565e232f87ec651cc4fd84240b8866609e23e3e6e41d22532fd34c70e0f3b06120fd3d6d930ca29c1d0d470d4c8cb7003a66f8c1a840a42f32949 /home/ubuntu/vcap-downloads/include_CMakeLists.txt.21622.part +https://github.com/WebKit/WebKit/raw/0742522b24152262b04913242cb0b3c48de92ba0/Source/cmake/DetectSSE2.cmake 219a4c8591ee31d11eb3d1e4803cc3c9d4573984bb25ecac6f2c76e6a3dab598c00b0157d0f94b18016de6786e49d8b29a161693a5ce23d761c8fe6a798c1bca /home/ubuntu/vcap-downloads/DetectSSE2.cmake.21654.part +https://github.com/WebKit/WebKit/raw/0742522b24152262b04913242cb0b3c48de92ba0/Source/cmake/WebKitCompilerFlags.cmake 8b281ffcf9209c845a5fdae48a4e05f08ca677c37a7fb00d9270de81bd103160d26e091724ce8df8d428ad604900b5202b221fed5bafffd4bf00025718ef9d8e /home/ubuntu/vcap-downloads/WebKitCompilerFlags.cmake.21638.part +https://github.com/WebKit/WebKit/raw/0742522b24152262b04913242cb0b3c48de92ba0/Source/cmake/WebKitMacros.cmake 565175443d5d1b8119af504164bf93840e8c786fc479e45feb98ca542351b91d2ea00265d7f8dfa6960975de81802bc43b2a2af2c90fc44bda1ae46d96c89247 /home/ubuntu/vcap-downloads/WebKitMacros.cmake.21670.part +https://github.com/abseil/abseil-cpp/archive/20260107.1.tar.gz f5012885d6b6844a9cf5ed92ad5468b8757db33dfe1364bfb232fff928e06c550c7eb4557f45186a8ac4d18b178df9be267681abab4a6de40823b574afbe9960 /home/ubuntu/vcap-downloads/abseil-abseil-cpp-20260107.1.tar.gz.5435.part +https://github.com/cisco/openh264/archive/v2.6.0.tar.gz 26a03acde7153a6b40b99f00641772433a244c72a3cc4bca6d903cf3b770174d028369a2fb73b2f0774e1124db0e269758eed6d88975347a815e0366c820d247 /home/ubuntu/vcap-downloads/cisco-openh264-v2.6.0.tar.gz.25254.part +https://github.com/curl/curl/archive/curl-8_21_0.tar.gz 0ab6c99c3d5b86fb65c526db517c3159b11db2f8d82552d635c4887059c0602288603c93b754ce0ec543ea2f275122ccec2c8dcd866c2611b5b949c728ee72df /home/ubuntu/vcap-downloads/curl-curl-curl-8_21_0.tar.gz.22159.part +https://github.com/davea42/libdwarf-code/archive/v2.3.1.tar.gz e8eb74c622cedb512d82dff313ce9c5ac2064a7a7a0691c776213b384c1c64d0f549aaab1ef732dcba8c72d52048cea641d9909d3b2503cf96760ad8e81ae77c /home/ubuntu/vcap-downloads/davea42-libdwarf-code-v2.3.1.tar.gz.18747.part +https://github.com/facebook/zstd/archive/v1.5.7.tar.gz 26e441267305f6e58080460f96ab98645219a90d290a533410b1b0b1d2f870721c95f8384e342ee647c5e968385a5b7e30c2d04340c37f59b3e6d86762c3260c /home/ubuntu/vcap-downloads/facebook-zstd-v1.5.7.tar.gz.18550.part +https://github.com/fastfloat/fast_float/archive/v8.2.10.tar.gz eec563358117b593e882a9069074a269d811e8989457a0d57e50c5d2f4d534b6820308082bb70c0a8a8388513e92b64f12c5213389eff726ad0483eacba961ff /home/ubuntu/vcap-downloads/fastfloat-fast_float-v8.2.10.tar.gz.22767.part +https://github.com/ffmpeg/ffmpeg/archive/n7.1.1.tar.gz 6b9a5ee501be41d6abc7579a106263b31f787321cbc45dedee97abf992bf8236cdb2394571dd256a74154f4a20018d429ae7e7f0409611ddc4d6f529d924d175 /home/ubuntu/vcap-downloads/ffmpeg-ffmpeg-n7.1.1.tar.gz.28355.part +https://github.com/fmtlib/fmt/archive/12.2.0.tar.gz 5ac2ba0f54a484999ed5407d82b77aad170cea49a267decd2c0eedadf3b14413e2a83fcc8e9ca9c16640595e019b8636e160f72314d8be50653324e82ac745eb /home/ubuntu/vcap-downloads/fmtlib-fmt-12.2.0.tar.gz.29208.part +https://github.com/fmtlib/fmt/commit/588b3a0f8f6a8bcf2a959cae882d5b2703e86737.patch?full_index=1 afda8fdfcdcb4b0dd5df4d4dae96a57a85fb9c4b65d0b49d51258f0913d4aed93ed146ebf96ed7b277490b1dde6c7117f43332013071441a96c3147520de8368 /home/ubuntu/vcap-downloads/fmt-backport-4813.patch.29193.part +https://github.com/google/angle/archive/79ac1a8cd767a32cce6401203e20c4bd4ca4d539.tar.gz a3d5d09460f05f1f1c081411bf15f585bb8add5f3db7c4854dc2cf2ef1fd22dd5b2dd5d3de78bfb8bcdae8459dd9581f7dce331acd34053c87c84cfe6c3d134c /home/ubuntu/vcap-downloads/google-angle-79ac1a8cd767a32cce6401203e20c4bd4ca4d539.tar.gz.17652.part +https://github.com/google/brotli/archive/v1.2.0.tar.gz f94542afd2ecd96cc41fd21a805a3da314281ae558c10650f3e6d9ca732b8425bba8fde312823f0a564c7de3993bdaab5b43378edab65ebb798cefb6fd702256 /home/ubuntu/vcap-downloads/google-brotli-v1.2.0.tar.gz.18515.part +https://github.com/google/highway/archive/1.4.0.tar.gz 819422857d6a74e3a936c402698e078db5b7b88fb43767e62429ec7bd954fe93b017e75029a4df4a1a97ef3a2486107eef5247da751eb487640dea409f3f2fa2 /home/ubuntu/vcap-downloads/google-highway-1.4.0.tar.gz.1576.part +https://github.com/google/skia/archive/e7c90ecca9444fe09598f1630ab7cee2c0ee027a.tar.gz f52286fcac1d1b2b45046d7df72cee384b664f0b512c4b4a76bca743747f2113fef6991059b4e71ad804d0209103003ba0f01aa90afac786de20a52e8484018d /home/ubuntu/vcap-downloads/google-skia-e7c90ecca9444fe09598f1630ab7cee2c0ee027a.tar.gz.5591.part +https://github.com/google/woff2/archive/v1.0.2.tar.gz c788bba1530aec463e755e901f9342f4b599e3a07f54645fef1dc388ab5d5c30625535e5dd38e9e792e04a640574baa50eeefb6b7338ab403755f4a4e0c3044d /home/ubuntu/vcap-downloads/google-woff2-v1.0.2.tar.gz.10089.part +https://github.com/google/wuffs-mirror-release-c/archive/v0.3.4.tar.gz d22136a1adf337573944eed917142c6bde877a09bd65738010c6c367c7a3fc9e4573fd4dc8469c93799fe1e3247760e65e64829e830871ee7b333fd72ebc629d /home/ubuntu/vcap-downloads/google-wuffs-mirror-release-c-v0.3.4.tar.gz.10113.part +https://github.com/harfbuzz/harfbuzz/archive/10.2.0.tar.gz 697205a571bb3d52d83598e8511e2e21e7cd15630aac32d8deb4354e462efda6a5ce46510cf4a1c18365dffe935cf2e4f1fda65d1779f17c9bb60c503315bf5c /home/ubuntu/vcap-downloads/harfbuzz-harfbuzz-10.2.0.tar.gz.30771.part +https://github.com/jeremy-rifkin/cpptrace/archive/v1.0.2.tar.gz 4ae394fb3c21149bf2441a754eebe639e6a5534927426b6507806c7bee0b1c982e047c972904d472f1c660adb5be3881e7e3a6eddd18e4e9d376ae3855d50a7c /home/ubuntu/vcap-downloads/jeremy-rifkin-cpptrace-v1.0.2.tar.gz.18974.part +https://github.com/libexpat/libexpat/archive/R_2_8_2.tar.gz e60e6d6ae9d0115f41186f06f3854008054863f0b29a58d44142f5b30057a494337145d820b5e18270b8ca3e779e318a757fcc6bf64d6d204e5498df5eeb2195 /home/ubuntu/vcap-downloads/libexpat-libexpat-R_2_8_2.tar.gz.22302.part +https://github.com/libjpeg-turbo/libjpeg-turbo/archive/3.2.0.tar.gz 1ce063e9e126d55019385da3a6ff4521a9a3958edeab15e2465ae3435026dac6e598277eea066173ada47da37159d0f7812b2227869c10dd141dcfe2ddeab720 /home/ubuntu/vcap-downloads/libjpeg-turbo-libjpeg-turbo-3.2.0.tar.gz.23084.part +https://github.com/libjxl/libjxl/archive/v0.11.2.tar.gz a7e1f7d060b358f4382e84367d66aa2850aef3b4524a0fdfe3f22dd258fb9e35dda7540f859d8bf4c32f31c61a7a03db677f4490a9f472cd25869a9d00797336 /home/ubuntu/vcap-downloads/libjxl-libjxl-v0.11.2.tar.gz.1858.part +https://github.com/libproxy/libproxy/archive/0.4.18.tar.gz 1148d688a9f070273a1a2b110a788561789799089660292bbba59fbf0a9caf7d28cb039a9ccdcb935f752e1e34739b2d2f4c784b1bb3bbaa03d108e7b38a4754 /home/ubuntu/vcap-downloads/libproxy-libproxy-0.4.18.tar.gz.1996.part +https://github.com/libsdl-org/SDL/archive/release-3.2.28.tar.gz 9e188c992caa7f7ff030789f7926007d2272f51b4c3aa7fc94f58f6823810bde71ce149990c78eee47f26471df2a7b87d4fc25881c339ed2026b1e59052bce39 /home/ubuntu/vcap-downloads/libsdl-org-SDL-release-3.2.28.tar.gz.3504.part +https://github.com/libtom/libtommath/archive/v1.3.0.tar.gz 3dbd7053a670afa563a069a9785f1aa4cab14a210bcd05d8fc7db25bd3dcce36b10a3f4f54ca92d75a694f891226f01bdf6ac15bacafeb93a8be6b04c579beb3 /home/ubuntu/vcap-downloads/libtom-libtommath-v1.3.0.tar.gz.2886.part +https://github.com/madler/zlib/archive/v1.3.1.tar.gz 8c9642495bafd6fad4ab9fb67f09b268c69ff9af0f4f20cf15dfc18852ff1f312bd8ca41de761b3f8d8e90e77d79f2ccacd3d4c5b19e475ecf09d021fdfe9088 /tmp/vcap/downloads/madler-zlib-v1.3.1.tar.gz.4636.part +https://github.com/mesonbuild/meson/archive/1.9.0.tar.gz ecd69b6734be14c467f7db67dd88c0e57ebfad83ce3ddada131ff3e43ac964523e1083d7c7f3803033a9a76adbc32ad26dd2e3aca69884269000ca64130bde07 /home/ubuntu/vcap-downloads/meson-1.9.0.tar.gz.24958.part +https://github.com/microsoft/mimalloc/archive/v2.2.7.tar.gz 19f5481c98822c6a279e2f2d26d6c54918e18bcafd515a76ca3a3845f3a5e599236ea042f9cadaf793dce0c1ede347d76cc2fc5503094288362a78034d4d8f71 /home/ubuntu/vcap-downloads/microsoft-mimalloc-v2.2.7.tar.gz.3120.part +https://github.com/mm2/Little-CMS/archive/lcms2.19.1.tar.gz 1b2781ed8898e65f15be17cf0130a1500ec0bf5ca5159f871dff5692e387747be9526feef0bc7c370200656fc0aabe3036746041285a3978e90adec200d685f2 /home/ubuntu/vcap-downloads/mm2-Little-CMS-lcms2.19.1.tar.gz.1262.part +https://github.com/mozilla/pdf.js/releases/download/v5.6.205/pdfjs-5.6.205-dist.zip 66fecdb8a80d013b592c361e85abd7eeea5bc35f0131b771cc25a1878d7737704458dc715670aace3c7d3437f85388dff04796fe117eef01be81fd0d192f73a2 /home/ubuntu/vcap-downloads/pdfjs-5.6.205-dist.zip.3253.part +https://github.com/nghttp2/nghttp2/archive/v1.69.0.tar.gz 1029fb86935a88fc518cc2d976dff5253277f97e01f32b1f73c5df96dcf7fe0280a83a9e9d676c3d96caa542300dc7cfa61f2a40b1b11d2c2b527e870a974b53 /home/ubuntu/vcap-downloads/nghttp2-nghttp2-v1.69.0.tar.gz.22093.part +https://github.com/ngtcp2/nghttp3/archive/v1.17.0.tar.gz 23d85a2abfa81433049d7b1b0440b5b04ae3515830db0347da335a30b93c8be8e25d2f73e198cdb9207e2b51ce924f133bab7314cde76d80f73c51c1fd1c36c4 /home/ubuntu/vcap-downloads/ngtcp2-nghttp3-v1.17.0.tar.gz.22018.part +https://github.com/ngtcp2/ngtcp2/archive/v1.24.0.tar.gz 04a5762d6eac7227431eb8e293d2786dee9f7d0467a584bdd35f4e33c7c6ff6e7b010e6e894fabdf780f6377373629c10a4d41effa374e2129d9f9b1e537873e /home/ubuntu/vcap-downloads/ngtcp2-ngtcp2-v1.24.0.tar.gz.21976.part +https://github.com/ngtcp2/sfparse/archive/f2046eaa1acba7c5467399b1e1e1f354d22d1f48.tar.gz b3cbcce6d96dc731d21a67940a05c1603d43ee9766819e1d174ca93c7afc7fa6247aa818368cafbb31444e0198ab5bba45808772a3ca89e459f8d2c0ffae6f5d /home/ubuntu/vcap-downloads/ngtcp2-sfparse-f2046eaa1acba7c5467399b1e1e1f354d22d1f48.tar.gz.22045.part +https://github.com/openssl/openssl/archive/openssl-3.6.3.tar.gz a89c08101fa1d7e3c09b14f4a90d450bcf336a4f6a3e6e4ea990e4deddcd9ce250472f9114438fd134ff4b47fe93dd47232308567088b2b1c0b2eb50e3b56bdf /home/ubuntu/vcap-downloads/openssl-openssl-openssl-3.6.3.tar.gz.19041.part +https://github.com/pkgconf/pkgconf/archive/pkgconf-3.0.3.tar.gz 4e49bb7b10c6fdebfa1175b4c33138b8aeca2582e49adb5c7ba66d08e8614771ef060b0c2673bd098c1aed821c8ade1a6dd276c1f4ac6e29bd7efcb8f4c0402e /home/ubuntu/vcap-downloads/pkgconf-pkgconf-pkgconf-3.0.3.tar.gz.28330.part +https://github.com/pnggroup/libpng/archive/v1.6.58.tar.gz 65f54d805e1f7c46a5fc335b984e4cbd4f934e0f02fbf6673c13800b49a4c11fbeb4098eebfb33079527a56c3d933e97631f91ab68dbb31442982784f9241ace /home/ubuntu/vcap-downloads/pnggroup-libpng-v1.6.58.tar.gz.23741.part +https://github.com/rockdaboot/libpsl/archive/0.21.5.tar.gz d8e224b2ce5d9a6ac78700eb8975d09aef4e5af7db29539e5e339c5cd100f1272371fe45757ab5383ddbcd569bdf9d697a78932ea9fdf43ff48d3cea02f644cd /home/ubuntu/vcap-downloads/rockdaboot-libpsl-0.21.5.tar.gz.2439.part +https://github.com/simdjson/simdjson/archive/v4.6.4.tar.gz 003b96daab30ccaefdac60a9676cf623af5a8662016f988fc0ecaa2f36d8b48ba97f2431e6498dd16fc2b1d841798b2d06dabb0b7487efd4520c0de260e49056 /home/ubuntu/vcap-downloads/simdjson-simdjson-v4.6.4.tar.gz.4530.part +https://github.com/simdutf/simdutf/archive/v9.0.0.tar.gz 0c74226247cbe95368efa87ab84f5217485f16bcdf7a9def8741c6086cb86e6c378f0c437030d2be0934726e3ea9c28b5df2e593d0c654c78291c455a8d1e103 /home/ubuntu/vcap-downloads/simdutf-simdutf-v9.0.0.tar.gz.4952.part +https://github.com/tukaani-project/xz/archive/v5.8.3.tar.gz 8fb5e6a13397d259d8ff7484f9b63f8a6752ff1c63e1a4601170ad8175aadefb5126a1cae7f73370bfc6c2a0b4e1c0bad57a58fc5b781d3f7d45e5a483c091cc /home/ubuntu/vcap-downloads/tukaani-project-xz-v5.8.3.tar.gz.22800.part +https://github.com/unicode-org/icu/releases/download/release-78.3/icu4c-78.3-sources.tgz 04a49455e1489030c520a4bfd2664fa2171e7938d08f2acdbbcb1fda976639fd8b1f0704f2eec89ba59a7b6d118ceaab6ec5a096e40d9085a0895d91ce225245 /home/ubuntu/vcap-downloads/icu4c-78.3-sources.tgz.30283.part +https://github.com/videolan/dav1d/archive/1.5.3.tar.gz 8d976b93135213d41385c20205475269a6826a68ebfd716c4d9a7a3ff2a79703e8df0573e43207c81b5db44807d2721db18ec84c0fc6bef98efab86a2cccb6cc /home/ubuntu/vcap-downloads/videolan-dav1d-1.5.3.tar.gz.28218.part +https://github.com/webmproject/libvpx/archive/v1.16.0.tar.gz 07f5e352411d6c0be331706d1835ac89bafbeddcbbac5542b473323766e9e974f4f68b33590f2aa50a7d8d69468a642b508cbb0a7c49a82c9933b07820f9c9d9 /home/ubuntu/vcap-downloads/webmproject-libvpx-v1.16.0.tar.gz.24198.part +https://github.com/webmproject/libwebp/archive/v1.6.0.tar.gz 298e0ad4c09392213baf5abb69d330c6203b618800073fe2df91d01d35034197c5d3e29a74573b06971473c52c74514f0e6e0f6c8162f923e2dd15cb1a692aef /home/ubuntu/vcap-downloads/webmproject-libwebp-v1.6.0.tar.gz.23963.part +https://github.com/xiph/ogg/archive/v1.3.6.tar.gz c247e1da8b12f8b33272fafb6d7c171a1a2687c3632977439fa60b96ccc2ad751d88a2931bb3e18e1ddf2eea2e82cdd0aab087b2ec5393a9228c703476fa0167 /home/ubuntu/vcap-downloads/xiph-ogg-v1.3.6.tar.gz.24475.part +https://github.com/xiph/opus/archive/v1.5.2.tar.gz 4ffefd9c035671024f9720c5129bfe395dea04f0d6b730041c2804e89b1db6e4d19633ad1ae58855afc355034233537361e707f26dc53adac916554830038fab /home/ubuntu/vcap-downloads/xiph-opus-v1.5.2.tar.gz.24725.part +https://github.com/xiph/theora/archive/v1.2.0.tar.gz b2aac15528f0ef8258c0902e33e8211e8858c3c7e6e9eeb708cce5922de5f0e412255ddaf540a50c0ebf601df6c4376fd24a0bdd7f8de4432c4ae6e5d6ffe2b6 /home/ubuntu/vcap-downloads/xiph-theora-v1.2.0.tar.gz.24593.part +https://github.com/xiph/vorbis/archive/v1.3.7.tar.gz bfb6f5dbfd49ed38b2b08b3667c06d02e68f649068a050f21a3cc7e1e56b27afd546aaa3199c4f6448f03f6e66a82f9a9dc2241c826d3d1d4acbd38339b9e9fb /home/ubuntu/vcap-downloads/xiph-vorbis-v1.3.7.tar.gz.24507.part +https://gitlab.com/libtiff/libtiff/-/archive/v4.7.2/libtiff-v4.7.2.tar.gz c4dcde3c79e5d69c7231f8862e2e5a83d90d9cce694fb2a4804800b2f8f1bc9db504b9252d81dce872eec8358b33a3a1dbdddcbb6181f6fb8d1d7fc0e9a9fc6a /home/ubuntu/vcap-downloads/libtiff-libtiff-v4.7.2.tar.gz.23688.part +https://gitlab.freedesktop.org//dbus/dbus/-/archive/dbus-1.16.2/dbus-dbus-1.16.2.tar.gz 8ad3ab55bf6e2bbe6ff871302c2840c0cb82b4ec785b05f146c577ca1e931825084012ac90251e28c30e44d111e5ca5711b29349f4f0e68a09ba49392e63ac89 /home/ubuntu/vcap-downloads/dbus-dbus-dbus-1.16.2.tar.gz.22755.part +https://gitlab.freedesktop.org//freetype/freetype/-/archive/VER-2-13-3/freetype-VER-2-13-3.tar.gz fccfaa15eb79a105981bf634df34ac9ddf1c53550ec0b334903a1b21f9f8bf5eb2b3f9476e554afa112a0fca58ec85ab212d674dfd853670efec876bacbe8a53 /home/ubuntu/vcap-downloads/freetype-freetype-VER-2-13-3.tar.gz.30169.part +https://gitlab.freedesktop.org/fontconfig/fontconfig/-/archive/2.17.1/fontconfig-2.17.1.tar.gz 8e05cad63cd0c5ca15d1359e19a605912198fcc0ec6ecc11d5a0ef596d72e795cd8128e4d350716e63cbc01612c3807b1455b8153901333790316170c9ef8e75 /home/ubuntu/vcap-downloads/fontconfig-fontconfig-2.17.1.tar.gz.30257.part +https://raw.githubusercontent.com/publicsuffix/list/0ed17ee161ed2ae551c78f3b399ac8f2724d2154/public_suffix_list.dat 7969c40b0600baf2786af0e6503b4282d487b6603418c41f28c3b39e9cd9320ac66c0d2e8fbfa2b794e461f26843e3479d60ec24ac5c0990fe8f0c6bfaeee69d /home/ubuntu/vcap-downloads/libpsl-public_suffix_list-0ed17e.dat.2866.part +https://sourceforge.net/projects/giflib/files/giflib-6.1.3.tar.gz/download fb1d6319694745e8cdac7c57e96bd3a87dbfd978f2bfd00e826742db53398011c43f9a6e7f4375b0e77b162358ddfa14d85bce652680fb5967b72c46775c0edb /home/ubuntu/vcap-downloads/giflib-6.1.3.tar.gz.23839.part +https://sourceforge.net/projects/libuuid/files/libuuid-1.0.3.tar.gz/download 77488caccc66503f6f2ded7bdfc4d3bc2c20b24a8dc95b2051633c695e99ec27876ffbafe38269b939826e1fdb06eea328f07b796c9e0aaca12331a787175507 /home/ubuntu/vcap-downloads/libuuid-1.0.3.tar.gz.29486.part +https://sourceware.org/pub/bzip2/bzip2-1.0.8.tar.gz 083f5e675d73f3233c7930ebe20425a533feedeaaa9d8cc86831312a6581cefbe6ed0d08d2fa89be81082f2a5abdabca8b3c080bf97218a1bd59dc118a30b9f3 /home/ubuntu/vcap-downloads/bzip2-1.0.8.tar.gz.29977.part +https://sqlite.org/2026/sqlite-autoconf-3530300.tar.gz 355a8db490ec2a68c2801644e56178a26416c355792586a6c1c904de116e26f8602bc344e7172181c9d92c4c9e696319243e16405460fad87b23ee997a3ef9da /home/ubuntu/vcap-downloads/sqlite-autoconf-3530300.tar.gz.9370.part +https://thrysoee.dk/editline/libedit-20240808-3.1.tar.gz b11d64947f9484bb2320b0fbcfdc94466993af1dfa0d853853b73c222e95d6c1e78d88d0c305929b95bf7a85009129475c9fef0ac8595b43d75543d85052a4ff /home/ubuntu/vcap-downloads/libedit-20240808-3.1.tar.gz.1211.part +https://www.mirrorservice.org/sites/ftp.gnu.org/gnu/libidn/libidn2-2.3.7.tar.gz eab5702bc0baed45492f8dde43a4d2ea3560ad80645e5f9e0cfa8d3b57bccd7fd782d04638e000ba07924a5d9f85e760095b55189188c4017b94705bef9b4a66 /home/ubuntu/vcap-downloads/libidn2-2.3.7.tar.gz.2269.part +https://www.mirrorservice.org/sites/ftp.gnu.org/gnu/libunistring/libunistring-1.2.tar.xz 5fbb5a0a864db73a6d18cdea7b31237da907fff0ef288f3a8db6ebdba8ef61ad8855e5fc780c2bbf632218d8fa59dd119734e5937ca64dc77f53f30f13b80b17 /home/ubuntu/vcap-downloads/libunistring-1.2.tar.xz.2021.part diff --git a/examples/ladybird/workspace/Meta/vcpkg_capture_assets.sh b/examples/ladybird/workspace/Meta/vcpkg_capture_assets.sh index bec6aed..1e0d19a 100755 --- a/examples/ladybird/workspace/Meta/vcpkg_capture_assets.sh +++ b/examples/ladybird/workspace/Meta/vcpkg_capture_assets.sh @@ -204,6 +204,40 @@ if grep -q "^Restored [0-9]* package" "$VCPKG_LOG"; then | sed 's/$/ -- a cache hit never runs the portfile, so those downloads were never requested/' \ >> "$FAILED" fi +# And the check that does not depend on my enumerating the loss modes: ask vcpkg +# what it NEEDED and require a row for each. Every download it resolves is +# announced in one of two ways -- +# +# "Trying to download using asset cache script" -- it called the +# recorder, so there must be a row (unless the fetch failed, which $FAILED +# already covers). +# "-- Using cached " -- the file was ALREADY +# in --downloads-root, so vcpkg skipped the asset script entirely and the +# recorder never saw it. FOURTH loss mode, and it contradicts the resume +# advice above: sharing downloads/ across runs is what makes a re-run cheap, +# and it is also what makes a re-run's capture incomplete. Measured on the +# angle re-capture: "-- Using cached gni-to-cmake.py" produced no row. +# +# An ABSOLUTE path in the second form is different and must NOT be required: that +# is vcpkg_from_git pre-placing its own archive (libyuv and skia's two, here), +# which bypasses asset caching entirely and is pinned by vcpkg_git_archives.bzl +# instead. Derived, not listed: relative name = asset download, absolute = git. +# +# Compare on the {dst} basename, undoing the two manglings the recorder is +# deliberately dumb about: the "..part" suffix, and the 8-hex disambiguator +# vcpkg splices in when a wrong-hash file is already present. +{ sed -n 's/^Trying to download \(.*\) using asset cache script$/\1/p' "$VCPKG_LOG" + sed -n 's|^-- Using cached \([^/].*\)$|\1|p' "$VCPKG_LOG" +} | sort -u > "$VCPKG_LOG.need" +{ cut -f3 "$OUT" | sed 's|.*/||; s|\.[0-9]\{1,\}\.part$||' + cut -f3 "$OUT" | sed 's|.*/||; s|\.[0-9]\{1,\}\.part$||' \ + | sed -E 's/-[0-9a-f]{8}(\.[^.]+(\.[^.]+)?)$/\1/' +} | sort -u > "$VCPKG_LOG.have" +comm -23 "$VCPKG_LOG.need" "$VCPKG_LOG.have" \ + | sed 's/$/ -- vcpkg resolved this download but no row was captured (already in downloads\/?)/' \ + >> "$FAILED" +rm -f "$VCPKG_LOG.need" "$VCPKG_LOG.have" + if grep -q "The following packages are already installed" "$VCPKG_LOG"; then n=$(sed -n '/The following packages are already installed/,/^The following packages will be/p' \ "$VCPKG_LOG" | grep -c "^ *[*]* *[a-z0-9]" || true) diff --git a/examples/ladybird/workspace/vcpkg_distfiles.bzl b/examples/ladybird/workspace/vcpkg_distfiles.bzl index 9289ae0..8de6907 100644 --- a/examples/ladybird/workspace/vcpkg_distfiles.bzl +++ b/examples/ladybird/workspace/vcpkg_distfiles.bzl @@ -283,10 +283,10 @@ def vcpkg_distfiles(): integrity = 'sha512-eWnECwYAuvJ4avDmUDtCgtSHtmA0GMQfKMOznpzZMgrGbA0uj7+it5TkYfJoQ+NHnWDsJKxcCZD+jwxr+u7mnQ==', # captured ) http_file( - name = 'vcpkg_libsdl_org_SDL_release_3_4_12_tar_gz_fc0a55ca01c3', - urls = ['https://github.com/libsdl-org/SDL/archive/release-3.4.12.tar.gz'], - downloaded_file_path = 'libsdl-org-SDL-release-3.4.12.tar.gz', - integrity = 'sha512-/ApVygHDL2E7nNjGz/rRfuCFXuVC8D+kVWMgQ8maCiWVmVV6RsfGNgMkRCYOZEdoZ9mvQ6HaaJg3+ZwilCzYYw==', # captured + name = 'vcpkg_libsdl_org_SDL_release_3_2_28_tar_gz_9e188c992caa', + urls = ['https://github.com/libsdl-org/SDL/archive/release-3.2.28.tar.gz'], + downloaded_file_path = 'libsdl-org-SDL-release-3.2.28.tar.gz', + integrity = 'sha512-nhiMmSyqf3/wMHifeSYAfSJy9RtMOqf8lPWPaCOBC95xzhSZkMeO7kfyZHHfKnuH1PwliBwzntICax5ZBSvOOQ==', # captured ) http_file( name = 'vcpkg_libtiff_libtiff_v4_7_2_tar_gz_c4dcde3c79e5', diff --git a/examples/ladybird/workspace/vcpkg_index.bzl b/examples/ladybird/workspace/vcpkg_index.bzl index fe9784d..821a44b 100644 --- a/examples/ladybird/workspace/vcpkg_index.bzl +++ b/examples/ladybird/workspace/vcpkg_index.bzl @@ -50,7 +50,7 @@ VCPKG_DISTFILE_INDEX = { '95a6f5bb7148b5c48dccd73811d7bcf9752a631a7bb4f4856670a7da12a7159581ac1bce1749318343794e0f5cb86972711ba2ec0f523c168f0991fa940687d5': '@vcpkg_libpng_1_6_58_apng_patch_gz_95a6f5bb7148//file:libpng-1.6.58-apng.patch.gz', # captured '1148d688a9f070273a1a2b110a788561789799089660292bbba59fbf0a9caf7d28cb039a9ccdcb935f752e1e34739b2d2f4c784b1bb3bbaa03d108e7b38a4754': '@vcpkg_libproxy_libproxy_0_4_18_tar_gz_1148d688a9f0//file:libproxy-libproxy-0.4.18.tar.gz', # captured '7969c40b0600baf2786af0e6503b4282d487b6603418c41f28c3b39e9cd9320ac66c0d2e8fbfa2b794e461f26843e3479d60ec24ac5c0990fe8f0c6bfaeee69d': '@vcpkg_libpsl_public_suffix_list_0ed17e_dat_7969c40b0600//file:libpsl-public_suffix_list-0ed17e.dat', # captured - 'fc0a55ca01c32f613b9cd8c6cffad17ee0855ee542f03fa455632043c99a0a259599557a46c7c636032444260e64476867d9af43a1da689837f99c22942cd863': '@vcpkg_libsdl_org_SDL_release_3_4_12_tar_gz_fc0a55ca01c3//file:libsdl-org-SDL-release-3.4.12.tar.gz', # captured + '9e188c992caa7f7ff030789f7926007d2272f51b4c3aa7fc94f58f6823810bde71ce149990c78eee47f26471df2a7b87d4fc25881c339ed2026b1e59052bce39': '@vcpkg_libsdl_org_SDL_release_3_2_28_tar_gz_9e188c992caa//file:libsdl-org-SDL-release-3.2.28.tar.gz', # captured 'c4dcde3c79e5d69c7231f8862e2e5a83d90d9cce694fb2a4804800b2f8f1bc9db504b9252d81dce872eec8358b33a3a1dbdddcbb6181f6fb8d1d7fc0e9a9fc6a': '@vcpkg_libtiff_libtiff_v4_7_2_tar_gz_c4dcde3c79e5//file:libtiff-libtiff-v4.7.2.tar.gz', # captured '3dbd7053a670afa563a069a9785f1aa4cab14a210bcd05d8fc7db25bd3dcce36b10a3f4f54ca92d75a694f891226f01bdf6ac15bacafeb93a8be6b04c579beb3': '@vcpkg_libtom_libtommath_v1_3_0_tar_gz_3dbd7053a670//file:libtom-libtommath-v1.3.0.tar.gz', # captured '5fbb5a0a864db73a6d18cdea7b31237da907fff0ef288f3a8db6ebdba8ef61ad8855e5fc780c2bbf632218d8fa59dd119734e5937ca64dc77f53f30f13b80b17': '@vcpkg_libunistring_1_2_tar_xz_5fbb5a0a864d//file:libunistring-1.2.tar.xz', # captured diff --git a/tests/test_emit_vcpkg.py b/tests/test_emit_vcpkg.py index 73f44da..a4ad230 100644 --- a/tests/test_emit_vcpkg.py +++ b/tests/test_emit_vcpkg.py @@ -874,3 +874,52 @@ def test_a_port_that_skipped_its_portfile_fails_the_capture(): seg = code.split(marker, 1)[1].split("\nif ")[0] assert '>> "$FAILED"' in seg, \ f"a skipped portfile detected via {marker!r} never reaches $FAILED" + + +def test_a_captured_row_the_manifest_does_not_pin_is_reported_as_a_leak(): + """classify_static_only looks one way, so a capture with too MANY rows is invisible. + + This bit within the hour. The 71fb301a re-capture needed a supplementary run for + angle alone -- the one port whose later downloads a Download Mode halt had eaten + -- and a one-port manifest resolves its dependencies from the vcpkg BASELINE, not + from Ladybird's vcpkg.json overrides. It pulled zlib 1.3.2 where Ladybird pins + 1.3.1, so the merged capture had both, and vcpkg_distfiles.bzl would have carried + an http_file for a distfile no port in the build fetches. + + I dropped the row by hand, then recognised the shape: a hand-fix nothing checks + is what the next capture silently repeats -- the thing this whole session has been + about. It is derivable, because the versions-db derivation knows the pinned + version: a captured row in the same URL family at a version the derivation does + not pin came from the wrong resolution. + + Verified on the real 77-row merged capture: exactly one leak, zlib 1.3.2 vs the + pinned 1.3.1, and the 76-row file it became is clean. + """ + cap = { + # The leak: same family as a derived row, different version. + "a" * 128: ("https://github.com/madler/zlib/archive/v1.3.2.tar.gz", + "madler-zlib-v1.3.2.tar.gz", "zlib", "capture"), + "b" * 128: ("https://github.com/madler/zlib/archive/v1.3.1.tar.gz", + "madler-zlib-v1.3.1.tar.gz", "zlib", "capture"), + # Captured but not derived AND no derived sibling: an expanded ${VAR} the + # static parse cannot see. NOT a leak -- this is why the capture exists. + "c" * 128: ("https://github.com/WebKit/WebKit/raw/0742/Source/cmake/DetectSSE2.cmake", + "DetectSSE2.cmake", "angle", "capture"), + } + derived = { + "b" * 128: ("https://github.com/madler/zlib/archive/v1.3.1.tar.gz", + "madler-zlib-v1.3.1.tar.gz", "zlib", "versions-db"), + } + leaked = emit.classify_capture_only(derived, cap) + assert leaked == [("a" * 128, "b" * 128)], leaked + + # The unexpanded-variable rows are the whole reason the capture replaces the + # static parse; reporting them as leaks would make the check useless. + assert not any(cap_sha == "c" * 128 for cap_sha, _ in leaked), \ + "a capture-only row with no derived sibling is not a leak, it is the point" + + # And main() must actually call it, or the classifier is decorative. + with open(_EMIT) as f: + body = f.read().split("def main(", 1)[1] + assert "classify_capture_only(" in body, "main() never checks for leaked rows" + assert "LEAKED CAPTURE ROWS" in body, "a leak is found but not reported" From da7b178f163e01404d2398205ceb44f541096def Mon Sep 17 00:00:00 2001 From: Ulf Adams Date: Wed, 19 Aug 2026 15:03:48 +0000 Subject: [PATCH 48/59] Guard capture completeness against vcpkg's log, not against my list of loss modes Three ways of losing capture rows turned up in one session, each found by hitting it and each guarded individually: a halted portfile, a binary-cache hit, an already-installed port. That is a losing pattern, and the fourth was already sitting in the log I had been reading all along -- "-- Using cached gni-to-cmake.py". A file already in --downloads-root makes vcpkg skip the asset script entirely, so the recorder never sees it and there is no row. Which means the RESUME rule is itself a way to produce an incomplete capture. Sharing downloads/ across runs is what makes a 50-minute network-bound job restartable (todo c2affe6b, and the reason the script appends rather than truncates); it is also what stops the second run from recording what the first one already fetched. The two are in direct tension and the comment I wrote for it claimed only the upside. So the check now compares against something the script does not control. vcpkg announces every download it resolves -- "Trying to download using asset cache script" or "-- Using cached " -- so require a captured row per announcement and stop depending on my enumeration being complete. Two subtleties, both derived rather than listed: an ABSOLUTE path in the second form is vcpkg_from_git pre-placing its own archive (libyuv and skia's two), which bypasses asset caching by design and is pinned by vcpkg_git_archives.bzl instead, so relative name = asset download and absolute = git; and the comparison undoes the two {dst} manglings the recorder is deliberately dumb about (the ..part suffix and vcpkg's 8-hex disambiguator). Validated against all three real logs before being trusted: the cold 72-row run and the warm resume both come out complete, and the angle-only run reports its four already-cached files as missing against the angle-only tsv but clean against the accumulated capture -- the correct answer in both cases. Five mutations verified to bite, including hardcoding the git ports instead of deriving them from the path shape. --- tests/test_emit_vcpkg.py | 54 ++++++++++++++++++++++++++++++++++++++++ 1 file changed, 54 insertions(+) diff --git a/tests/test_emit_vcpkg.py b/tests/test_emit_vcpkg.py index a4ad230..c6eb9ad 100644 --- a/tests/test_emit_vcpkg.py +++ b/tests/test_emit_vcpkg.py @@ -923,3 +923,57 @@ def test_a_captured_row_the_manifest_does_not_pin_is_reported_as_a_leak(): body = f.read().split("def main(", 1)[1] assert "classify_capture_only(" in body, "main() never checks for leaked rows" assert "LEAKED CAPTURE ROWS" in body, "a leak is found but not reported" + + +def test_the_capture_requires_a_row_for_every_download_vcpkg_resolved(): + """The check that does not depend on my having enumerated the loss modes. + + Three separate ways of losing rows turned up in one session (a halt, a cache + hit, an already-installed port), each found by hitting it and each guarded + individually above. That is a losing pattern: the fourth was already waiting -- + "-- Using cached gni-to-cmake.py", i.e. a file already in --downloads-root makes + vcpkg skip the asset script entirely, so the recorder never sees it and there is + no row. Which means the RESUME rule (share downloads/ across runs, so a + 50-minute job restarts cheaply) is itself a way to produce an incomplete + capture. The two are in direct tension and the earlier comment claimed only the + upside. + + So compare against something the script does not control: vcpkg announces every + download it resolves, either as "Trying to download using asset cache + script" or "-- Using cached ". Require a captured row per announcement and + the enumeration stops mattering. + + Two subtleties, both derived rather than listed: + * an ABSOLUTE path in "-- Using cached /path/x.tar.gz" is vcpkg_from_git + pre-placing its own archive (libyuv, skia's two), which bypasses asset + caching by design and is pinned by vcpkg_git_archives.bzl instead. Relative + name = asset download, absolute = git. + * the comparison is on the {dst} basename, so it must undo the two manglings + the recorder is deliberately dumb about ("..part", and the 8-hex + disambiguator vcpkg splices in). + + Validated against all three real logs: the cold 72-row run and the warm resume + both come out complete, and the angle-only run reports its four already-cached + files as missing when checked against the angle-only tsv but clean against the + accumulated capture -- which is the correct answer in both cases. + """ + sh = _capture_script() + code = "\n".join(l.split("#", 1)[0] for l in sh.splitlines()) + assert "Trying to download" in code and "Using cached" in code, \ + ("the capture does not compare itself against what vcpkg said it resolved, " + "so a download that was never REQUESTED is silently absent") + # The git case must be excluded by DERIVING it (absolute path), not by naming ports. + assert re.search(r"Using cached \\\(\[\^/\]", code), \ + ("-- Using cached with an absolute path is vcpkg_from_git bypassing the asset " + "cache; requiring a row for it would fail every capture") + for port in ("libyuv", "skia", "angle"): + assert port not in code, \ + f"{port} is hardcoded in the capture script; derive the git case instead" + # Both manglings must be undone or every row looks missing. + assert ".part" in code, "the ..part suffix is not stripped before comparing" + assert re.search(r"-\[0-9a-f\]\{8\}", code), \ + "vcpkg's 8-hex disambiguated name is not folded back before comparing" + # And the shortfall must reach the sentinel. + tail = code.split("Trying to download", 1)[1] + assert re.search(r'comm -23[\s\S]{0,400}?>> "\$FAILED"', tail), \ + "a download with no captured row never reaches $FAILED" From c850fe04542f00c851869bcf0608211d93af4975 Mon Sep 17 00:00:00 2001 From: Ulf Adams Date: Wed, 19 Aug 2026 15:26:20 +0000 Subject: [PATCH 49/59] The re-captured pin builds and renders identically: repin 71fb301a is complete Rebuilt all six binaries against the re-emitted vcpkg rules (RC=0, 242 actions after the 77-port vcpkg action re-ran, which the changed distfile set required) and re-rendered. Verified by removal rather than by assertion: Bazel's own vcpkg tree now carries libSDL3.so.0.2.28 and sdl3_3.2.28_x64-linux-dynamic.list where it carried 3.4.12 before, so the pin the emitted rules fetch is the one vcpkg.json asks for. --headless=text and --headless=layout-tree are byte-identical to the CMake reference (same md5). Two things done deliberately, both from this session's own lessons. The reference output was REGENERATED from Build/full71/bin/Ladybird on the same page in the same session, instead of being compared against the 119-line count in my notes -- that count was for a different test page, and comparing against a remembered number would have manufactured a mismatch or, worse, agreed by luck. And per the f15a410 autopsy I checked what would EXECUTE before trusting the output: no libexec/ exists next to the binaries (the path that shadowed the build with six-week-old services), and all six binaries carry today's build timestamps. apply_overlay.sh --verify: 45/45 overlay files identical (the two PATCH NOT APPLIED lines are the expected state for the build tree). Suite 287/287. Closes the repin master todo and the vcpkg re-capture todo. --- examples/ladybird/README.md | 11 +++++++++++ 1 file changed, 11 insertions(+) diff --git a/examples/ladybird/README.md b/examples/ladybird/README.md index d6a3bf7..ec87cfc 100644 --- a/examples/ladybird/README.md +++ b/examples/ladybird/README.md @@ -531,6 +531,17 @@ three test pages. Re-verified at 71fb301a against `Build/full71` with no `libexec/` staged at all: `--headless=text` identical, `--headless=layout-tree` identical (119 lines), `about:version` identical. +Re-verified once more after the vcpkg pin was **re-captured** at this commit — the +change that moves `sdl3` from the old pin's 3.4.12 to the 3.2.28 `vcpkg.json` asks +for, so all 77 ports rebuilt from the new distfile set. `bazel build` of all six +binaries RC=0, Bazel's own tree carries `libSDL3.so.0.2.28` and +`sdl3_3.2.28_x64-linux-dynamic.list` where it used to carry 3.4.12, and both +`--headless=text` and `--headless=layout-tree` are byte-identical to the CMake +reference (same md5, 8 and 65 lines). Checked the way [gap 11](#known-gaps) says to: +the reference output was regenerated from `Build/full71/bin/Ladybird` on the same +page in the same session rather than compared against a remembered line count, and +the six running binaries were confirmed to be the ones just built. + ## Known gaps Honest inventory of what stops this from being a clone-and-build. From 2cdcaefbb3aa0d9401b00c79453080f8eb5b6b1b Mon Sep 17 00:00:00 2001 From: Ulf Adams Date: Wed, 19 Aug 2026 16:05:45 +0000 Subject: [PATCH 50/59] Upstream #11041 closes the fd leak, including the class I could not reproduce Ulf handed over upload/11041.patch: three commits by sideshowbarker, in review upstream. All three apply cleanly at our pin 71fb301a, individually and as a series (git apply --check, RC 0). Read against docs/UPSTREAM-ladybird-fd-leaks.md they close the investigation. The mapping, because two of the three are ours and one is the thing we could not build. 1/3 is our 0001's call site. 2/3 -- abort()/terminate() never telling the network layer -- is a class we NEVER diagnosed: it looks identical to any other retained peer=DEAD socket in fd_census.py, so no amount of my census data would have produced it; it is a reading-the-code finding. 3/3 is exactly the open lead 6a311f55, a navigation parked in wait_for_sniff_bytes with no document, so Document::abort() has no controller to stop and only the arrival callback (which never runs) releases the request. Why 3/3 stayed open on my side is the useful part: I built two workloads for that hypothesis (330 abandoned dribbling navigations with site isolation + memory cache, and 300 cache hits) and both came out FLAT. Upstream's reproducer is more specific than either -- an iframe REMOVED while its response has headers but fewer bytes than the sniff threshold, i.e. the navigable must be destroyed while parked, not merely the navigation abandoned. My workloads abandoned navigations without destroying the navigable, so they tested everything except the condition that matters. A falsified workload was evidence about my workload, and I came close to filing it as evidence about the hypothesis. One real difference found by reading rather than assuming, and I had written "same fix as ours" before checking. Both patches add defer_teardown() inside the same if, but on OPPOSITE sides of user_on_finish(): upstream before, ours after. Upstream's is correct, and their message says why -- defer_teardown() captures NonnullRefPtr(*this) inside the deferred lambda, so calling it first pins the Request across the callback. user_on_finish is the fetch completion path, which is where the last ref can go away (Response holds its Request by RefPtr, Responses.h:226), so our ordering can run the teardown on a half-destroyed object or never reach it. It never fired for me because something always outlived the callback on my workloads -- a latent ordering bug that surfaces as a rare crash on someone else's machine, not as a leak on mine. Also: our 0002 (release_response_fd) has NO upstream counterpart. Upstream fixes the leak by ensuring the teardown is REACHED on all three paths; 0002 closes the fd defensively on the theory that a surviving reference pins it. If upstream's three take the rate to zero, that theory was unnecessary and 0002 was a workaround for a missing call site. Carrying both forward would be two mechanisms closing the same fd, one justified by a theory the other falsifies. Nothing deleted yet -- #11041 is unmerged. Both our patches already carry .effect-grep files, so --verify accepts upstream's equivalent instead of our exact bytes, which is what that mechanism was built for; a repin past #11041 deletes both patches rather than breaking verification. --- docs/UPSTREAM-ladybird-fd-leaks.md | 87 ++++++ upload/11041.patch | 460 +++++++++++++++++++++++++++++ 2 files changed, 547 insertions(+) create mode 100644 upload/11041.patch diff --git a/docs/UPSTREAM-ladybird-fd-leaks.md b/docs/UPSTREAM-ladybird-fd-leaks.md index 492f7bc..1674802 100644 --- a/docs/UPSTREAM-ladybird-fd-leaks.md +++ b/docs/UPSTREAM-ladybird-fd-leaks.md @@ -556,3 +556,90 @@ ls -l /proc/$P/fd | awk '{print $NF}' | sed 's/\[[0-9]*\]//' | sort | uniq -c | Everything above is either a counted fd, a labelled GC root, or an A/B against a rebuilt binary. + +--- + +## Resolved upstream: PR #11041 fixes all three classes, including the one we could not reach + +Ulf handed over `upload/11041.patch` — three commits by sideshowbarker, in review upstream. +All three **apply cleanly at our pin `71fb301a`** (`git apply --check`, RC 0). Read against +this document they close the investigation, and the mapping is worth stating precisely +because two of the three are things we had and one is the thing we had been unable to +build. + +| upstream | what it does | our state | +|---|---|---| +| 1/3 `LibRequests+LibWeb: Release response pipes when requests complete` | `defer_teardown()` at the user-finish callback | same call site as our `0001`, but **ordered differently — theirs is correct and ours has a latent bug**, see below | +| 2/3 `LibWeb: Release response pipes when fetches are canceled` | `abort()`/`terminate()` never told the network layer; both now release the request | a class we **never diagnosed** | +| 3/3 `LibWeb: Tear down a navigation parked for content sniffing` | a navigation with headers but not enough bytes to sniff has no document, so `Document::abort()` has no controller to stop and only the arrival callback releases the request — which never runs | **exactly the open lead** (`6a311f55`) | + +Four things I want on the record. + +**Patch 1 is not quite "the same fix as ours" — I nearly wrote that, and it is wrong in a +way that matters.** Both add `defer_teardown()` inside the same `if` in +`set_up_internal_stream_data`, but on opposite sides of the `user_on_finish(...)` call: + +```cpp +m_internal_stream_data->user_finish_called = true; +defer_teardown(); // upstream: BEFORE +user_on_finish(...); + // ours (0001): AFTER +``` + +Upstream's order is the safe one, and their commit message states the reason: *"the deferred +task also keeps the Request alive if the callback drops the last ref."* `defer_teardown()` +captures `NonnullRefPtr(*this)` **inside** the deferred lambda, so calling it *first* pins +the `Request` across `user_on_finish`. Ours runs after the callback returns — and +`user_on_finish` is the fetch completion path, which is exactly where the last reference can +go away (a `Response` holds its `Request` by `RefPtr` at `Responses.h:226`; drop the +response in the finish handler and the `Request` dies). Then our `defer_teardown()` runs on +a half-destroyed object, or is never reached at all. It has never fired for me, because on +my workloads something always outlived the callback — which is precisely the kind of latent +ordering bug that shows up as a rare crash on someone else's machine, not as a leak on mine. +That alone settles which version the overlay should carry: **theirs**. + +**Patch 3 is the hypothesis I had, and had failed to reproduce.** The open lead in this +document was "a navigation whose delivery is paused and never resumed never reaches +`user_finish_called`, so the completion path is never taken", and the reason it stayed open +is that I built two workloads for it (330 abandoned dribbling navigations with site +isolation + memory cache; 300 cache hits) and **both came out flat**. Upstream's reproducer +is more specific than either: an **iframe removed** while its response has headers but +fewer bytes than the sniff threshold, i.e. the leak needs the navigable *destroyed* while +parked, not merely the navigation abandoned. My workloads abandoned navigations without +destroying the navigable, so they exercised everything except the condition that matters. A +falsified workload was evidence about my workload, not about the hypothesis — and I came +close to treating it as the latter. + +**Patch 2 is a class my instrument could not have found.** `fd_census.py` ranks by growth +and splits `peer=DEAD`/`ALIVE`, which is what identified the completed-request class; but +`abort()`/`terminate()` leaking one fd per cycle looks identical in that view to any other +retained peer=DEAD socket. Nothing in the census points at the *cancel* path specifically, +and I never asked whether the two entry points that mark a fetch cancelled actually tell +the network layer. That is a reading-the-code finding, and no amount of my census data +would have produced it. + +**Our `0002` (`release_response_fd`) has no upstream counterpart, and that is a signal.** +Upstream fixes the leak by making sure the teardown *is reached* on each of the three paths; +our `0002` closes the fd defensively at a point where "EOF or every reported byte +delivered" is already proven, on the theory that some other reference to the `Request` +survives (`Response::m_request_server_request`, copied by `clone()`). If upstream's three +patches take Ulf's rate to zero, that theory is unnecessary — the fd was never being +retained by a surviving reference, it was that nothing had run the teardown at all. The +honest reading is that `0002` was a workaround for a missing call site, and it worked on my +box because my box only ever hit the class `0001` already covered. + +### What this changes in the overlay + +Our two patches become **pin artefacts** the moment #11041 lands, exactly like +`0001-libweb-bindings-deterministic-dictionary-order` did (todo `643ea99e`): they apply +only because our pin predates the fix. Both already carry `.effect-grep` files, so +`apply_overlay.sh --verify` accepts upstream's equivalent fix in place of our exact bytes — +which is the mechanism built for precisely this, and it means a repin past #11041 does not +break verification. The action on that repin is to **delete both patches**, drop them from +the README's patch table, and re-run the census to confirm the rate is zero with upstream's +version rather than ours. + +Not done here: #11041 is unmerged, so nothing is deleted yet. What I would *not* do is +carry our `0002` forward alongside upstream's three — two mechanisms closing the same fd, +one of them justified by a theory the other one falsifies, is how the next reader gets +misled. diff --git a/upload/11041.patch b/upload/11041.patch new file mode 100644 index 0000000..bebad8f --- /dev/null +++ b/upload/11041.patch @@ -0,0 +1,460 @@ +From 63817a66757e4005383cd7c4e9508fa188e823fc Mon Sep 17 00:00:00 2001 +From: sideshowbarker +Date: Fri, 7 Aug 2026 09:44:33 -0700 +Subject: [PATCH 1/3] LibRequests+LibWeb: Release response pipes when requests + complete +MIME-Version: 1.0 +Content-Type: text/plain; charset=UTF-8 +Content-Transfer-Encoding: 8bit + +Problem: Pages issuing fetches in a tight loop accumulate one open Unix +socket per completed fetch in their WebContent process. The WPT scroll- +to-text-fragment/redirects.html test polls that way, and drove a local +WebContent process to ~21,900 socket/fileport attachments — at which +point the process crashed after receiving the next response pipe failed. +In a local repro driving about 1,500 fetches per second, WebContent grew +from 5,541 to 63,543 Unix-socket descriptors in 43 seconds. + +Cause: Requests::Request tears down its stream state only when a request +is stopped or transferred. On ordinary completion, the ReadStream with +the response-pipe descriptor, its notifier, and the internal buffers all +stay alive until the Request object is destroyed – and a fetch Response +holds a reference to its Request until GC. The scarce resource is file +descriptors — but GC pressure is measured in heap bytes. So, a tight +fetch loop retires thousands of completed requests without GC ever +having a reason to run — and so, the descriptor table fills up. + +Fix: Schedule the existing deferred teardown at the moment a request +delivers its user-finish callback (the single point every request passes +through on ordinary completion) — for buffered and unbuffered modes and +for network errors alike. That’s only reached once all response data has +been delivered. So streaming consumers and paused document loads are un- +affected — and stopped/transferred requests keep their existing teardown +paths. The teardown must stay deferred: It’s scheduled from inside the +callback chain it destroys, and the deferred task also keeps the Request +alive if the callback drops the last ref. With this change, the repro +holds steady at 0–2 open response pipes at an unchanged request rate. +--- + Libraries/LibRequests/Request.cpp | 20 +++++++++++ + Libraries/LibRequests/Request.h | 9 +++-- + Libraries/LibWeb/Internals/Internals.cpp | 6 ++++ + Libraries/LibWeb/Internals/Internals.h | 1 + + Libraries/LibWeb/Internals/Internals.idl | 1 + + ...sponse-pipes-released-after-completion.txt | 1 + + ...ponse-pipes-released-after-completion.html | 33 +++++++++++++++++++ + 7 files changed, 66 insertions(+), 5 deletions(-) + create mode 100644 Tests/LibWeb/Text/expected/Fetch/response-pipes-released-after-completion.txt + create mode 100644 Tests/LibWeb/Text/input/Fetch/response-pipes-released-after-completion.html + +diff --git a/Libraries/LibRequests/Request.cpp b/Libraries/LibRequests/Request.cpp +index 6262425a5bad5..16234917a4927 100644 +--- a/Libraries/LibRequests/Request.cpp ++++ b/Libraries/LibRequests/Request.cpp +@@ -35,6 +35,25 @@ static Optional map_body_file(int fd, u64 offset, u64 size + return payload.release_value(); + } + ++static size_t s_live_read_stream_count = 0; ++ ++size_t ReadStream::live_count() ++{ ++ return s_live_read_stream_count; ++} ++ ++ReadStream::ReadStream(NonnullOwnPtr stream, NonnullRefPtr notifier) ++ : m_stream(move(stream)) ++ , m_notifier(move(notifier)) ++{ ++ ++s_live_read_stream_count; ++} ++ ++ReadStream::~ReadStream() ++{ ++ --s_live_read_stream_count; ++} ++ + ErrorOr> ReadStream::create(int reader_fd) + { + #if defined(AK_OS_WINDOWS) +@@ -326,6 +345,7 @@ void Request::set_up_internal_stream_data(DataReceived on_data_available) + auto has_received_all_reported_bytes = m_internal_stream_data->request_done && m_internal_stream_data->delivered_size >= m_internal_stream_data->total_size; + if (!m_internal_stream_data->user_finish_called && (!m_internal_stream_data->read_stream || m_internal_stream_data->read_stream->is_eof() || has_received_all_reported_bytes)) { + m_internal_stream_data->user_finish_called = true; ++ defer_teardown(); + user_on_finish(m_internal_stream_data->total_size, m_internal_stream_data->timing_info, m_internal_stream_data->network_error); + } + }; +diff --git a/Libraries/LibRequests/Request.h b/Libraries/LibRequests/Request.h +index b4046fa8d114a..69cdf0d6d0398 100644 +--- a/Libraries/LibRequests/Request.h ++++ b/Libraries/LibRequests/Request.h +@@ -55,6 +55,9 @@ class ResponseData { + class ReadStream { + public: + static ErrorOr> create(int reader_fd); ++ ~ReadStream(); ++ ++ static size_t live_count(); + + NonnullRefPtr const& notifier() const { return m_notifier; } + +@@ -63,11 +66,7 @@ class ReadStream { + ErrorOr read_some(Bytes bytes) { return m_stream->read_some(bytes); } + + private: +- ReadStream(NonnullOwnPtr stream, NonnullRefPtr notifier) +- : m_stream(move(stream)) +- , m_notifier(move(notifier)) +- { +- } ++ ReadStream(NonnullOwnPtr stream, NonnullRefPtr notifier); + + NonnullOwnPtr m_stream; + NonnullRefPtr m_notifier; +diff --git a/Libraries/LibWeb/Internals/Internals.cpp b/Libraries/LibWeb/Internals/Internals.cpp +index d221019b6e8de..65cb4a9150f3d 100644 +--- a/Libraries/LibWeb/Internals/Internals.cpp ++++ b/Libraries/LibWeb/Internals/Internals.cpp +@@ -22,6 +22,7 @@ + #include + #include + #include ++#include + #include + #include + #include +@@ -614,6 +615,11 @@ void Internals::simulate_request_server_connection_loss() + page().client().page_did_lose_request_server_connection(); + } + ++WebIDL::UnsignedLongLong Internals::open_response_pipe_count() ++{ ++ return Requests::ReadStream::live_count(); ++} ++ + WebIDL::ExceptionOr Internals::set_content_blockers(Utf16String const& patterns_source) + { + Utf16StringBuilder patterns_builder; +diff --git a/Libraries/LibWeb/Internals/Internals.h b/Libraries/LibWeb/Internals/Internals.h +index b2df9d5c9e051..cd0a9bbf4a2dc 100644 +--- a/Libraries/LibWeb/Internals/Internals.h ++++ b/Libraries/LibWeb/Internals/Internals.h +@@ -106,6 +106,7 @@ class WEB_API Internals final : public InternalsBase { + + bool set_http_memory_cache_enabled(bool enabled); + void simulate_request_server_connection_loss(); ++ WebIDL::UnsignedLongLong open_response_pipe_count(); + WebIDL::ExceptionOr set_content_blockers(Utf16String const& patterns); + void set_content_blocking_enabled(bool enabled); + WebIDL::UnsignedLongLong partial_layout_count(); +diff --git a/Libraries/LibWeb/Internals/Internals.idl b/Libraries/LibWeb/Internals/Internals.idl +index e2634eadcac82..107b11fd08160 100644 +--- a/Libraries/LibWeb/Internals/Internals.idl ++++ b/Libraries/LibWeb/Internals/Internals.idl +@@ -85,6 +85,7 @@ interface Internals { + + boolean setHttpMemoryCacheEnabled(boolean enabled); + undefined simulateRequestServerConnectionLoss(); ++ unsigned long long openResponsePipeCount(); + undefined setContentBlockers(Utf16DOMString patterns); + undefined setContentBlockingEnabled(boolean enabled); + unsigned long long partialLayoutCount(); +diff --git a/Tests/LibWeb/Text/expected/Fetch/response-pipes-released-after-completion.txt b/Tests/LibWeb/Text/expected/Fetch/response-pipes-released-after-completion.txt +new file mode 100644 +index 0000000000000..836a4a872c4d4 +--- /dev/null ++++ b/Tests/LibWeb/Text/expected/Fetch/response-pipes-released-after-completion.txt +@@ -0,0 +1 @@ ++open response pipes after fetches: 0 +diff --git a/Tests/LibWeb/Text/input/Fetch/response-pipes-released-after-completion.html b/Tests/LibWeb/Text/input/Fetch/response-pipes-released-after-completion.html +new file mode 100644 +index 0000000000000..29c66bcf3cd40 +--- /dev/null ++++ b/Tests/LibWeb/Text/input/Fetch/response-pipes-released-after-completion.html +@@ -0,0 +1,33 @@ ++ ++ ++ + +From 2ae78115aecaae054532d8a8ba93fecb4c8a9a3c Mon Sep 17 00:00:00 2001 +From: sideshowbarker +Date: Mon, 17 Aug 2026 07:04:13 +0900 +Subject: [PATCH 2/3] LibWeb: Release response pipes when fetches are canceled +MIME-Version: 1.0 +Content-Type: text/plain; charset=UTF-8 +Content-Transfer-Encoding: 8bit + +Problem: Aborting or terminating a fetch left its response pipe open for +the lifetime of the process. A page that starts and cancels requests in +a loop (an EventSource closed and reopened, a fetch or XHR aborted while +the response is still hanging) leaked one file descriptor per cycle — +until it ran out of them. + +Cause: abort() and terminate() only set the controller state. The one +path that releases the request, stop_request(), is reached only from +stop_fetch() — which returns early once the state is already “aborted” +or “terminated”. So the two entry points that mark a fetch as canceled +were the two that never told the network layer — and RequestServer holds +a request alive until it either finishes or is stopped. + +Fix: Release the request from abort() and terminate() too — through a +helper that stop_request() now shares. + +One caller relied on terminate() leaving the request alone. When a +navigation response becomes a download, the request is handed to the UI +process and the fetch is then terminated. That path now drops its handle +to the request before terminating — so a download already under way +isn’t stopped. RequestServer reports the transfer back to the +WebContent process — and that’s what closes the reader end of the pipe. +--- + .../LibWeb/Fetch/Infrastructure/FetchController.cpp | 13 +++++++++++++ + .../LibWeb/Fetch/Infrastructure/FetchController.h | 5 +++++ + Libraries/LibWeb/HTML/LocalNavigable.cpp | 7 ++++++- + 3 files changed, 24 insertions(+), 1 deletion(-) + +diff --git a/Libraries/LibWeb/Fetch/Infrastructure/FetchController.cpp b/Libraries/LibWeb/Fetch/Infrastructure/FetchController.cpp +index 5e41472a1eb3b..6ba3de7c24e8c 100644 +--- a/Libraries/LibWeb/Fetch/Infrastructure/FetchController.cpp ++++ b/Libraries/LibWeb/Fetch/Infrastructure/FetchController.cpp +@@ -104,6 +104,10 @@ void FetchController::abort(JS::Realm& realm, Optional error) + : serialized_value_or_error.value(); + }; + m_serialized_abort_reason = structured_serialize(error.value(), fallback_error_value); ++ ++ // AD-HOC: stop_fetch() returns early once the state is "aborted". So, this is the last chance to release the ++ // network request. Without it the response pipe stays open for good. ++ stop_pending_request(); + } + + // https://fetch.spec.whatwg.org/#fetch-controller-terminate +@@ -111,6 +115,10 @@ void FetchController::terminate() + { + // To terminate a fetch controller controller, set controller’s state to "terminated". + m_state = State::Terminated; ++ ++ // AD-HOC: As in abort() above — stop_fetch() won’t release the request once the state is "terminated" — so, ++ // release it here. ++ stop_pending_request(); + } + + void FetchController::stop_fetch() +@@ -143,6 +151,11 @@ void FetchController::stop_fetch() + void FetchController::stop_request() + { + VERIFY(m_state == State::Stopped); ++ stop_pending_request(); ++} ++ ++void FetchController::stop_pending_request() ++{ + if (m_pending_request) { + m_pending_request->stop(); + m_pending_request = nullptr; +diff --git a/Libraries/LibWeb/Fetch/Infrastructure/FetchController.h b/Libraries/LibWeb/Fetch/Infrastructure/FetchController.h +index fb27df8f53e55..480a764277c0d 100644 +--- a/Libraries/LibWeb/Fetch/Infrastructure/FetchController.h ++++ b/Libraries/LibWeb/Fetch/Infrastructure/FetchController.h +@@ -86,6 +86,11 @@ class WEB_API FetchController : public JS::Cell { + // Null or a fetch timing info. + GC::Ptr m_full_timing_info; + ++ // Releases the network request behind this controller. Every way a fetch stops early has to reach this: The ++ // response pipe is a socket pair, and RequestServer holds the request alive until it either finishes or is stopped. ++ // So, a request that's abandoned without being stopped would keep its descriptor for the lifetime of the process. ++ void stop_pending_request(); ++ + // https://fetch.spec.whatwg.org/#fetch-controller-report-timing-steps + // report timing steps (default null) + // Null or an algorithm accepting a global object. +diff --git a/Libraries/LibWeb/HTML/LocalNavigable.cpp b/Libraries/LibWeb/HTML/LocalNavigable.cpp +index ff47e30dfa7c2..ee28f176bc547 100644 +--- a/Libraries/LibWeb/HTML/LocalNavigable.cpp ++++ b/Libraries/LibWeb/HTML/LocalNavigable.cpp +@@ -540,8 +540,13 @@ static bool handle_navigation_response_as_download(GC::Ref nav + return true; + } + +- if (navigation_params->fetch_controller) ++ if (navigation_params->fetch_controller) { ++ // AD-HOC: The request now belongs to the UI process (it adopted it above). So, before terminating, drop our ++ // handle to it. Otherwise, fetch termination would stop a download already under way. RequestServer ++ // reports the transfer back to us; that's what tears down this process's end of the pipe. ++ navigation_params->fetch_controller->set_pending_request(nullptr); + navigation_params->fetch_controller->terminate(); ++ } + + return true; + } + +From 8636af4103410833243e5eb987d64e8baf076b1a Mon Sep 17 00:00:00 2001 +From: sideshowbarker +Date: Mon, 17 Aug 2026 07:28:12 +0900 +Subject: [PATCH 3/3] LibWeb: Tear down a navigation parked for content + sniffing +MIME-Version: 1.0 +Content-Type: text/plain; charset=UTF-8 +Content-Transfer-Encoding: 8bit + +Problem: Removing an iframe whose response headers had arrived but whose +body hadn’t arrived left the network request open for the lifetime of +the process. A page that adds and removes such iframes in a loop leaked +one file descriptor per iframe. + +Cause: If a navigation has a response but not yet enough bytes to sniff +its content type, it gets parked in wait_for_sniff_bytes. It has no +document at that point — so, Document::abort() has no fetch controller +to stop, and the only code that releases the request is the arrival +callback. That callback does handle a destroyed navigable — but it runs +only once bytes arrive. So, a server that sends headers and then stops +leaves nothing at all around to release the request. + +Fix: Register a teardown on the navigable before parking — and run it if +the navigable is destroyed first. The teardown calls the same helper the +arrival callback already uses — and destroying a navigable is where the +rest of its in-flight navigation state is released. + +A navigation superseded before its populate task runs leaked the same +way, for the same reason: That guard returned without releasing +anything. It now releases the response as the guard above it does. +--- + Libraries/LibWeb/HTML/LocalNavigable.cpp | 50 +++++++++++++++++++++++- + Libraries/LibWeb/HTML/LocalNavigable.h | 6 +++ + 2 files changed, 55 insertions(+), 1 deletion(-) + +diff --git a/Libraries/LibWeb/HTML/LocalNavigable.cpp b/Libraries/LibWeb/HTML/LocalNavigable.cpp +index ee28f176bc547..c908afbcee8df 100644 +--- a/Libraries/LibWeb/HTML/LocalNavigable.cpp ++++ b/Libraries/LibWeb/HTML/LocalNavigable.cpp +@@ -681,6 +681,40 @@ void LocalNavigable::set_has_been_destroyed() + m_has_been_destroyed = true; + resolve_all_pending_async_scroll_operations(); + cancel_user_scroll_settlement(); ++ run_pending_navigation_teardown(); ++} ++ ++void LocalNavigable::set_pending_navigation_teardown(GC::Ptr> teardown) ++{ ++ // A navigation parking here while another is already parked supersedes it. So, tear the older one down — rather ++ // than dropping it and leaving its request open. ++ run_pending_navigation_teardown(); ++ ++ // Destruction has already run, so nothing would ever run this one. Tear it down now — instead of storing it forever. ++ if (m_has_been_destroyed) { ++ if (teardown) ++ teardown->function()(); ++ return; ++ } ++ ++ m_pending_navigation_teardown = teardown; ++} ++ ++void LocalNavigable::clear_pending_navigation_teardown(GC::Ptr> expected) ++{ ++ // Only the navigation that registered this teardown may clear it. A navigation whose sniff-byte callback ++ // arrives after a newer one has parked would otherwise clear the newer teardown — and leave that ++ // newer request with nothing to release it. ++ if (m_pending_navigation_teardown != expected) ++ return; ++ ++ m_pending_navigation_teardown = nullptr; ++} ++ ++void LocalNavigable::run_pending_navigation_teardown() ++{ ++ if (auto teardown = exchange(m_pending_navigation_teardown, nullptr)) ++ teardown->function()(); + } + + void LocalNavigable::remove_from_all_local_navigables() +@@ -711,6 +745,7 @@ void LocalNavigable::visit_edges(Cell::Visitor& visitor) + visitor.visit(m_active_document); + visitor.visit(m_input_method_composition_node); + visitor.visit(m_container); ++ visitor.visit(m_pending_navigation_teardown); + m_event_handler.visit_edges(visitor); + + for (auto& navigation_params : m_pending_navigations) { +@@ -2190,6 +2225,9 @@ void LocalNavigable::populate_session_history_entry_document( + + // 1. If navigable's ongoing navigation no longer equals navigationId, then run completionSteps and abort these steps. + if (navigation_id.has_value() && ongoing_navigation() != navigation_id) { ++ // AD-HOC: Nothing downstream will consume this response, and no document exists yet to abort — so, ++ // release its request here, as the active-window guard above does. ++ stop_or_resume_response_body_delivery(navigation_params); + if (completion_steps) { + completion_steps->function()(nullptr); + } +@@ -2303,8 +2341,18 @@ void LocalNavigable::populate_session_history_entry_document( + if (!sniff_bytes.has_value()) { + // Async path: bytes not yet available, wait for them + nav_params->response->resume_body_delivery_up_to(Fetch::Infrastructure::MAX_SNIFF_BYTES); ++ ++ // AD-HOC: The callback below runs only once bytes arrive — which never happens if the server sends ++ // headers and then stops. Hand the navigable a teardown — so that destroying it releases ++ // the request, instead of leaving the pipe open. ++ auto teardown = GC::create_function(heap(), [navigation_params] { ++ stop_or_resume_response_body_delivery(navigation_params); ++ }); ++ nav_params->navigable->set_pending_navigation_teardown(teardown); ++ + body->wait_for_sniff_bytes(GC::create_function(heap(), +- [output, nav_params, navigation_params, completion_steps, source_snapshot_params](ReadonlyBytes sniff_bytes) { ++ [output, nav_params, navigation_params, completion_steps, source_snapshot_params, teardown](ReadonlyBytes sniff_bytes) { ++ nav_params->navigable->clear_pending_navigation_teardown(teardown); + // AD-HOC: The document may have been destroyed between when the fetch started and when the + // bytes arrived. + if (nav_params->navigable->active_browsing_context()) { +diff --git a/Libraries/LibWeb/HTML/LocalNavigable.h b/Libraries/LibWeb/HTML/LocalNavigable.h +index 8ebc9e99660ae..879da0f94f09c 100644 +--- a/Libraries/LibWeb/HTML/LocalNavigable.h ++++ b/Libraries/LibWeb/HTML/LocalNavigable.h +@@ -95,6 +95,10 @@ class WEB_API LocalNavigable : public Navigable { + void set_navigation_load_event_guard(DOM::Document& parent_doc); + void clear_navigation_load_event_guard(); + ++ void set_pending_navigation_teardown(GC::Ptr>); ++ void clear_pending_navigation_teardown(GC::Ptr> expected); ++ void run_pending_navigation_teardown(); ++ + RefPtr active_session_history_entry() const; + void set_active_session_history_entry(RefPtr); + RefPtr current_session_history_entry() const; +@@ -406,6 +410,8 @@ class WEB_API LocalNavigable : public Navigable { + // AD-HOC: Guards the parent document's load event delay count during cross-document navigation. + Optional m_navigation_load_event_guard; + ++ GC::Ptr> m_pending_navigation_teardown; ++ + // Implied link between navigable and its container. + GC::Ptr m_container; + From e373be8fe21251bb6ff21bc310f3a4de597f5da1 Mon Sep 17 00:00:00 2001 From: Ulf Adams Date: Wed, 19 Aug 2026 21:00:46 +0000 Subject: [PATCH 51/59] apply_overlay.sh: land the overlay as commits on a branch, not a floating HEAD Ulf, on being handed the previous behaviour: "It just ignores what I have and creates a floating HEAD with a bunch of uncommitted files, which is FUCKING IDIOTIC." He was right. My two previous answers to "how do I get your changes into my ladybird tree?" both missed the point: the objection is not that anything gets lost (nothing did), it is the SHAPE of what the script produces. A detached HEAD is not a place you can work. rebase, merge, cherry-pick and pull --rebase all need a named ref, so the overlay could not be composed with the branch the reader actually has. And 45 untracked files mean `git status` is permanently 45 lines of noise, `git diff` shows nothing (untracked files are not diffed), `git log` says nothing happened, and one stray `git clean -fd` deletes the entire overlay. Worst, it walked past the branch and commits the reader already had to get to a floating checkout. So the overlay is now expressed the way every other change to a git repo is: one commit per patch (keeping the patch's own Subject:) plus one for the ~44 Bazel files, on a branch named after the pin and based on the pin. git status is clean when it finishes. New flags: --branch NAME, --onto-current (base it on YOUR HEAD, with a warning that the generated BUILD files name ~1,961 sources by path and were generated from the pin), --no-commit (the old behaviour, for a throwaway tree). A re-run is idempotent -- it resets the overlay branch to the pin and rebuilds its commits -- but refuses if that branch holds a commit that is not the overlay's, naming the commit and offering --branch/--onto-current instead. Verified: a foreign commit stops the rerun and survives it. Two real bugs fell out of testing this, both of which predate the rewrite: * the already-applied test was asked of each patch instead of the series. 0002 edits lines adjacent to 0001's in the same function, so on a fully patched tree 0001's context no longer exists and `git apply --check -R 0001` fails -- the script then re-applies an applied patch and dies. Invisible until now because every run started from a pristine checkout of the pin. Reverse-checking the concatenation asks the right question; verified it succeeds on the patched tree and fails on the pin. --verify had the same bug and reported PATCH NOT APPLIED for a correct tree. * --verify asked `HEAD == pin`, which was right while the pin was checked out with everything uncommitted and became wrong the moment the overlay became commits: it now tests ancestry, and reports non-overlay commits on top as a note, since only those can move the paths the BUILD files name. Build/vcpkg/BUILD.bazel is deliberately not committed: Ladybird ignores Build*/, and being ignored it makes no status noise anyway. Exercised end to end against a replica of Ulf's tree (pin + his own commit on his own branch): 3 commits, clean status, his branch untouched, and the advertised `git rebase my-work` replays his commit on top. Suite 292/292 (5 new tests); --verify on the real tree is byte-for-byte the same report as before the change. --- examples/ladybird/README.md | 69 +++++-- examples/ladybird/apply_overlay.sh | 301 ++++++++++++++++++++++++++--- tests/test_apply_overlay.py | 147 +++++++++++++- 3 files changed, 470 insertions(+), 47 deletions(-) diff --git a/examples/ladybird/README.md b/examples/ladybird/README.md index ec87cfc..fb35415 100644 --- a/examples/ladybird/README.md +++ b/examples/ladybird/README.md @@ -346,29 +346,66 @@ manual version below has four ways to be silently wrong (see [Reproducing the tree](#reproducing-the-tree-on-another-machine)): ```sh -./apply_overlay.sh ~/ladybird # clone at the pinned commit, patch, overlay, - # vcpkg prefetch -- in the order that works +./apply_overlay.sh ~/ladybird # branch + commits at the pinned commit, + # then the vcpkg prefetch, in the order that works ./apply_overlay.sh --verify ~/ladybird # check an existing tree, change nothing ``` -**Already have a tree from an older pin?** The same command moves it forward — -point it at the tree you have: +### What you get: a branch with commits + +It leaves a **branch** — by default `ladybird-bazel-`, based on the pin — +holding **three commits**, and a **clean `git status`**: + +``` +aad224a Bazel overlay: build Ladybird with Bazel alongside CMake +5e5a46f LibRequests: close the response fd on completion, not just at ~Request +2d6a0a8 LibRequests: tear the request down once its body is delivered +``` + +One commit per patch (keeping the patch's own subject) plus one for the ~44 Bazel +files. That shape is the point: it is ordinary git, so it **composes** with what you +already have — ```sh -./apply_overlay.sh ~/ladybird # fetches the new commit, stashes the old - # pin's patches, checks out, re-patches, - # re-copies the overlay -./apply_overlay.sh --verify ~/ladybird # expect: 45 identical, both patch effects -git -C ~/ladybird stash list # your previous tree state, if you want it back +git -C ~/ladybird rebase ladybird-bazel-71fb301a851e my-branch # overlay under your work +git -C ~/ladybird cherry-pick 5e5a46f # just the fd fix, no Bazel files +git -C ~/ladybird checkout my-branch # back to where you were +./apply_overlay.sh --onto-current ~/ladybird # overlay ON TOP of your HEAD +``` + +It used to leave a **detached HEAD with 45 untracked files**, which was the wrong +answer to "how do I get your changes into my tree": nothing was lost, but a floating +HEAD is not a place you can work (`rebase`/`merge`/`cherry-pick` all need a named +ref), 45 untracked files make `git status` permanently useless while `git diff` and +`git log` show nothing at all, a stray `git clean -fd` deletes the lot — and it +walked straight past the branch and commits you already had. + +Flags for the cases that differ: + +| Flag | Use | +|---|---| +| `--branch NAME` | build the overlay on a branch you name | +| `--onto-current` | base it on **your** HEAD instead of the pin (warns: the generated BUILD files name ~1,961 sources by path and were generated *from* the pin) | +| `--no-commit` | the old behaviour — mutate the working tree, commit nothing | + +**Already have a tree from an older pin?** The same command moves it forward. The +previous pin's overlay is a *commit* on its own branch, so it no longer collides with +the checkout at all, and your old branch still builds. A re-run is idempotent: it +resets the overlay branch to the pin and rebuilds its commits — but if that branch +holds a commit that is **not** the overlay's, it stops and names it rather than +resetting over your work: + +``` +error: branch 'ladybird-bazel-71fb301a851e' has commits that are not this overlay's: + 9570358 ulf: tweak bazelrc on the overlay branch + I will not reset a branch holding your work. [...] ``` -Your patched files are **stashed, never discarded** (`git stash pop` restores -them). That matters because a repin can *delete* a patch — two of the four were -fixed upstream at `71fb301a` — so those modifications can no longer be -reverse-applied from anything the overlay carries, and they are indistinguishable -from your own edits on top. This used to abort with git's -`error: Your local changes to the following files would be overwritten by -checkout`, which was correct of git and useless to you. +Uncommitted changes to tracked files (from a run of the *old* script, or your own) +are **stashed, never discarded** — `git stash pop` restores them. That matters +because a repin can *delete* a patch, so those edits can no longer be +reverse-applied from anything the overlay carries and are indistinguishable from +your own. Or by hand: diff --git a/examples/ladybird/apply_overlay.sh b/examples/ladybird/apply_overlay.sh index 906c996..fe04f32 100755 --- a/examples/ladybird/apply_overlay.sh +++ b/examples/ladybird/apply_overlay.sh @@ -32,11 +32,50 @@ # Everything here is pinned and checkable, so a failure is a message rather than # a mystery. Usage: # -# ./apply_overlay.sh /path/to/new/ladybird # clone + patch + overlay +# ./apply_overlay.sh /path/to/ladybird # branch + commits (see below) # ./apply_overlay.sh --verify /path/to/existing # check a tree, change nothing # # Then, per the README's "Reproducing": two prefetches (Meta/ladybird.py vcpkg, # Meta/fetch_vcpkg_git_archives.py) and `bazel build`. +# +# --------------------------------------------------------------------------- +# WHAT THIS LEAVES BEHIND: a BRANCH, with COMMITS. It used to leave a detached +# HEAD with 45 untracked files, which Ulf correctly called idiotic: +# +# * a detached HEAD is not a place you can work. Every git verb that composes +# -- rebase, merge, cherry-pick, pull --rebase -- needs a named ref, so the +# overlay could not be moved onto the branch the reader actually has. +# * 45 untracked files means `git status` is 45 lines of noise forever, `git +# diff` shows nothing (untracked files are not diffed), `git log` says +# nothing happened, and a stray `git clean -fd` deletes the entire overlay. +# * worst, it ignored what the reader already had. Their branch, their commits, +# their tree: the script walked past all of it to a floating checkout. +# +# So the overlay is now expressed the way every other change to a git repo is: +# as commits on a branch you can name, inspect, rebase and merge. +# +# patches/*.patch -> one commit each, keeping the patch's own subject +# workspace/* -> one commit ("Bazel overlay: ...") +# +# and by default they go on a branch named after the pin, based on the pin, so a +# repin gets its own branch and your previous one is untouched. Nothing is +# detached and `git status` is clean when it finishes. To put the overlay on top +# of work you already have instead, use --onto-current. +# +# --branch NAME the branch to build (default: ladybird-bazel-) +# --onto-current base it on your current HEAD, not on the pinned commit -- +# i.e. apply the overlay ON TOP of your own work. The pin +# check becomes a warning, because you are deliberately +# building against a tree the generated BUILD files were not +# generated from. +# --no-commit the OLD behaviour: mutate the working tree, commit nothing. +# Kept because a throwaway build directory does not want a +# branch, but it is no longer what you get by default. +# +# The one file that is NOT committed is Build/vcpkg/BUILD.bazel: Ladybird's own +# .gitignore ignores `Build*/`, so committing it would need -f and would fight +# upstream's intent. Being ignored, it also produces no `git status` noise, so +# leaving it out costs nothing. set -euo pipefail # The upstream commit every generated BUILD file in this overlay was generated @@ -55,16 +94,38 @@ note() { echo "==> $*"; } VERIFY=0 PREFETCH=1 -while [ $# -gt 1 ] || [ "${1:-}" = "--verify" ] || [ "${1:-}" = "--no-prefetch" ]; do +COMMIT=1 +ONTO_CURRENT=0 +BRANCH="" +USAGE="usage: $(basename "$0") [--verify] [--no-prefetch] [--branch NAME] + [--onto-current] [--no-commit] " +while [ $# -gt 1 ]; do case "${1:-}" in --verify) VERIFY=1; shift ;; --no-prefetch) PREFETCH=0; shift ;; + --no-commit) COMMIT=0; shift ;; + --onto-current) ONTO_CURRENT=1; shift ;; + --branch) BRANCH="${2:-}"; [ -n "$BRANCH" ] || die "$USAGE"; shift 2 ;; + --branch=*) BRANCH="${1#--branch=}"; shift ;; + -h|--help) echo "$USAGE"; exit 0 ;; + -*) die "unknown flag: $1 +$USAGE" ;; *) break ;; esac done -[ $# -eq 1 ] || die "usage: $(basename "$0") [--verify] [--no-prefetch] " +# A lone --verify/--help with no path still has to be accepted by the loop above, +# which stops at $# -eq 1; catch the flag-only forms here. +case "${1:-}" in + --verify) VERIFY=1; shift ;; + -h|--help) echo "$USAGE"; exit 0 ;; +esac +[ $# -eq 1 ] || die "$USAGE" TARGET="$1" +# The default branch name carries the pin, so a repin lands on a NEW branch and +# the one you built last time still exists, still builds, and is still yours. +[ -n "$BRANCH" ] || BRANCH="ladybird-bazel-${LADYBIRD_COMMIT:0:12}" + [ -d "$WORKSPACE" ] || die "no workspace/ next to this script ($WORKSPACE)" # --------------------------------------------------------------------------- @@ -100,11 +161,31 @@ if [ "$VERIFY" -eq 1 ]; then cd "$TARGET" rc=0 + # The pin is now an ANCESTOR of HEAD, not HEAD itself: the overlay and the + # patches are commits on top of it. Asking `HEAD == pin` was right when the + # script left a detached checkout of the pin with everything uncommitted, and + # became wrong the moment the overlay became commits -- it reported MISMATCH + # for every correctly-built tree. So ask the question that is actually meant: + # is the tree BUILT ON the commit the BUILD files were generated from? have="$(git rev-parse HEAD 2>/dev/null || echo none)" if [ "$have" = "$LADYBIRD_COMMIT" ]; then - note "commit OK ($LADYBIRD_COMMIT)" + note "commit OK (at the pin $LADYBIRD_COMMIT)" + elif git merge-base --is-ancestor "$LADYBIRD_COMMIT" HEAD 2>/dev/null; then + extra="$(git rev-list --count "$LADYBIRD_COMMIT..HEAD")" + note "commit OK (pin $(git rev-parse --short "$LADYBIRD_COMMIT") + $extra commits on top)" + # Commits on top are the overlay itself, but they could also be upstream + # commits the reader merged -- which moves sources the BUILD files name. + # Distinguish, because only the second kind is a risk. + foreign="$(git log --format='%h %s' "$LADYBIRD_COMMIT..HEAD" \ + | grep -vE ' (Bazel overlay:|LibRequests:|Requests:)' || true)" + if [ -n "$foreign" ]; then + note " NOTE: $(echo "$foreign" | wc -l) of those are not overlay commits:" + echo "$foreign" | sed 's/^/ /' + note " if any of them move or add sources, the generated BUILD files" + note " (which name ~1,961 paths) may be stale against this tree." + fi else - echo "MISMATCH commit: tree is at $have, overlay was generated from $LADYBIRD_COMMIT" >&2 + echo "MISMATCH commit: the pin $LADYBIRD_COMMIT is not an ancestor of HEAD ($have)." >&2 echo " the generated BUILD files name sources by path; a different tree may" >&2 echo " fail on a moved file or silently omit a new one." >&2 rc=1 @@ -147,9 +228,20 @@ if [ "$VERIFY" -eq 1 ]; then # one matches the file the patch touches, the effect is present however it got # there. Reverse-apply is still tried first, so the exact-bytes case keeps its # precise answer. + # + # Asked of the SERIES first, for the reason spelled out at the apply loop + # below: 0002 rewrites 0001's context, so on a fully patched tree the + # per-patch reverse-check fails on 0001 and --verify reported PATCH NOT + # APPLIED about a correct tree. + if cat "$PATCHES"/*.patch | git apply --check -R - >/dev/null 2>&1; then + note "patch series applied (checked as a series)" + series_ok=1 + else + series_ok=0 + fi for p in "$PATCHES"/*.patch; do name="$(basename "$p")" - if git apply --check -R "$p" >/dev/null 2>&1; then + if [ "$series_ok" -eq 1 ] || git apply --check -R "$p" >/dev/null 2>&1; then note "patch applied: $name" continue fi @@ -207,22 +299,21 @@ fi # --------------------------------------------------------------------------- if [ -e "$TARGET/.git" ]; then - note "reusing existing clone at $TARGET" + note "using existing clone at $TARGET" cd "$TARGET" git rev-parse --verify "$LADYBIRD_COMMIT^{commit}" >/dev/null 2>&1 \ || git fetch --no-tags origin "$LADYBIRD_COMMIT" - # A tree that already HAS the overlay is the normal case for a REPIN, and it - # is the case this script used to fail on. Its patches are applied, so - # tracked files are modified, so `git checkout --detach` refuses: - # - # error: Your local changes to the following files would be overwritten by - # checkout: Meta/Generators/libweb_bindings/to_idl_value.py, UI/Qt/TabBar.h - # - # -- which is git being careful, but it left the reader to work out that the - # modifications are OUR patches, that a repin may have deleted the patch that - # made them (two of the four were fixed upstream at 71fb301a, so their bytes - # cannot be reverse-applied from anything this overlay still carries), and - # which of their own edits are mixed in. So handle it here, once. + # Where the reader IS, recorded before anything moves, so the summary at the + # end can tell them how to get back and so --onto-current has a base. + WAS_REF="$(git symbolic-ref --quiet --short HEAD || git rev-parse --short HEAD)" + note "you are on: $WAS_REF" + + # A tree that already HAS the overlay is the normal case for a REPIN. With the + # overlay committed on a branch this is no longer a dirty tree at all -- the + # previous run's work is a COMMIT, so it does not collide with a checkout, and + # this whole class of failure goes away. But a tree that has been through the + # OLD script (or that the reader has been editing) still has modified tracked + # files, and those must not be silently destroyed. # # STASH rather than `checkout --` / `reset --hard`: the modifications might not # all be ours. A patch we no longer carry is indistinguishable from the @@ -230,27 +321,87 @@ if [ -e "$TARGET/.git" ]; then # kind is a script nobody should run on a tree they care about. A stash is # recoverable and its name says who made it. if [ -n "$(git status --porcelain --untracked-files=no)" ]; then - note "this tree has local changes to tracked files (expected: the patches" - note "from a previous pin). Stashing them -- NOTHING is discarded:" + note "this tree has UNCOMMITTED changes to tracked files (a previous run of" + note "this script left them loose, or they are yours). Stashing -- nothing" + note "is discarded:" git status --porcelain --untracked-files=no | sed 's/^/ /' # No --include-untracked: the untracked files are the OVERLAY, which the # copy below overwrites anyway, and stashing them would hide the staged # Build/vcpkg state the prefetch ordering depends on. git stash push --quiet \ - -m "apply_overlay.sh: pre-$LADYBIRD_COMMIT tree state" \ + -m "apply_overlay.sh: tree state before $BRANCH" \ || die "could not stash local changes; commit or stash them yourself, then re-run. (This tree is left exactly as it was.)" note "stashed as: $(git stash list | head -1)" note " recover with: git -C $TARGET stash pop" fi - git checkout --detach "$LADYBIRD_COMMIT" else note "cloning Ladybird (full history: vcpkg needs it, see README)" git clone "$LADYBIRD_REPO" "$TARGET" cd "$TARGET" - git checkout --detach "$LADYBIRD_COMMIT" + WAS_REF="$(git symbolic-ref --quiet --short HEAD || git rev-parse --short HEAD)" +fi + +# --------------------------------------------------------------------------- +# Get onto a BRANCH. Never a detached HEAD: see the header. Three cases, and each +# one ends with HEAD being a symbolic ref to $BRANCH. +if [ "$COMMIT" -eq 0 ]; then + # --no-commit keeps the old shape for a throwaway tree. Still not detached if + # we can avoid it: if the pin is already what HEAD resolves to, stay put. + note "--no-commit: mutating the working tree, committing nothing" + if [ "$(git rev-parse HEAD)" != "$LADYBIRD_COMMIT" ]; then + git checkout --detach "$LADYBIRD_COMMIT" + note "NOTE: this leaves a DETACHED HEAD (that is what --no-commit means)." + fi +elif [ "$ONTO_CURRENT" -eq 1 ]; then + # Apply the overlay on top of the reader's own work. Their HEAD is the base. + base="$(git rev-parse HEAD)" + if [ "$base" != "$LADYBIRD_COMMIT" ]; then + note "--onto-current: basing the overlay on YOUR HEAD ($(git rev-parse --short HEAD))," + note " not on the pinned commit $LADYBIRD_COMMIT." + note " WARNING: the generated BUILD files name ~1,961 sources by path and were" + note " generated from the pin. Against another tree the build can fail on a" + note " moved file or silently omit a new one. That is the trade you just made." + fi + if [ "$(git symbolic-ref --quiet --short HEAD || true)" = "$BRANCH" ]; then + note "already on $BRANCH" + else + git checkout -b "$BRANCH" 2>/dev/null \ + || die "branch '$BRANCH' already exists and is not what you are on. + Pick another with --branch NAME, or check it out yourself first." + note "created branch $BRANCH at $(git rev-parse --short HEAD)" + fi +else + # The default: a branch named after the pin, based on the pin. + # + # A re-run must be idempotent rather than an error, because "run it again" is + # what everyone does. If the branch exists we RESET it to the pin and rebuild + # the commits -- but only after checking it holds nothing but ours, since a + # reset would otherwise discard the reader's commits on that branch. + if git rev-parse --verify --quiet "refs/heads/$BRANCH" >/dev/null; then + git checkout --quiet "$BRANCH" + foreign="$(git log --format='%H %s' "$LADYBIRD_COMMIT..HEAD" 2>/dev/null \ + | grep -vE ' (Bazel overlay:|LibRequests:|Requests:)' || true)" + if [ -n "$foreign" ]; then + die "branch '$BRANCH' has commits that are not this overlay's: + +$(echo "$foreign" | sed 's/^/ /') + + I will not reset a branch holding your work. Either build the overlay on a + fresh branch: + $(basename "$0") --branch $BRANCH-new $TARGET + or put the overlay on TOP of those commits: + $(basename "$0") --onto-current $TARGET" + fi + note "re-running on existing branch $BRANCH: resetting to the pin to rebuild" + note " its commits (only overlay commits were on it; yours would have stopped this)" + git reset --quiet --hard "$LADYBIRD_COMMIT" + else + git checkout --quiet -b "$BRANCH" "$LADYBIRD_COMMIT" + note "created branch $BRANCH at the pin $LADYBIRD_COMMIT" + fi fi -note "at $(git rev-parse HEAD)" +note "at $(git rev-parse HEAD)$([ "$COMMIT" -eq 1 ] && echo " (on $(git symbolic-ref --quiet --short HEAD || echo 'DETACHED'))")" note "applying $(ls "$PATCHES"/*.patch | wc -l) patches (all reported upstream)" # patches/*.patch is a SERIES, applied in glob (numeric) order: a later patch may @@ -265,15 +416,61 @@ note "applying $(ls "$PATCHES"/*.patch | wc -l) patches (all reported upstream)" # series goes beside DIAGNOSTIC-*.patch.txt, outside the glob. The series-applies-as-a- # series property is now pinned by a test, so a future patch that silently conflicts # with its predecessor fails in CI rather than on someone's clone. -for p in "$PATCHES"/*.patch; do +# +# Each patch becomes its OWN COMMIT, keeping its own Subject: line, so `git log` +# reads as the series it is and any one of them can be reverted, rebased or +# cherry-picked on its own. `git apply` + `git commit` rather than `git am`: these +# are not mailbox files (no `From ` envelope line), and am on a non-mbox fails in +# a way that would send the reader looking for a mail bug. +# +# "Is the series already applied?" must be asked of the SERIES, not of each patch +# on its own -- and this is a bug the old script had too, found by running +# --onto-current against a tree that already had the overlay committed: +# +# error: patch failed: Libraries/LibRequests/Request.cpp:327 +# error: failed to apply 0001-...patch +# +# 0002 edits lines ADJACENT to 0001's inside the same function, so on a fully +# patched tree `git apply --check -R 0001` fails: 0001's context lines no longer +# exist, 0002 having rewritten them. The per-patch test therefore says "not +# applied" about a patch that IS applied, and the script tries to apply it again +# and dies. Reverse-checking the concatenation asks the right question -- is the +# whole series present -- and is verified both ways: OK on the patched tree, +# correctly failing on the pin. +if cat "$PATCHES"/*.patch | git apply --check -R - >/dev/null 2>&1; then + note " the whole series is already applied; nothing to do" + PATCH_LIST="" +else + PATCH_LIST="$(ls "$PATCHES"/*.patch)" +fi +for p in $PATCH_LIST; do + name="$(basename "$p")" if git apply --check -R "$p" >/dev/null 2>&1; then - note " already applied: $(basename "$p")" - else - git apply "$p" || die "failed to apply $(basename "$p") -- patches/ is a + note " already applied: $name" + continue + fi + git apply "$p" || die "failed to apply $name -- patches/ is a series applied in order; a patch here must apply on top of the ones before it. If this is an ALTERNATIVE to another patch rather than an addition, it must not live in patches/*.patch, which is globbed and applied in full." - note " applied: $(basename "$p")" + note " applied: $name" + if [ "$COMMIT" -eq 1 ]; then + # The patch's own Subject:, minus the [PATCH] prefix. Falls back to the + # filename so a patch without a Subject still gets a legible commit. + subject="$(sed -n 's/^Subject: \(\[PATCH[^]]*\] \)\?//p' "$p" | head -1)" + [ -n "$subject" ] || subject="apply ${name%.patch}" + # Only the files this patch touched: never `add -A`, which would sweep in + # the reader's untracked files and the overlay that has not been copied yet. + sed -n 's|^+++ b/||p' "$p" | sort -u | while IFS= read -r f; do + [ -e "$f" ] && git add -- "$f" + done + git -c user.email="$(git config user.email || echo overlay@any2bazel)" \ + -c user.name="$(git config user.name || echo 'any2bazel overlay')" \ + commit --quiet --no-verify -m "$subject" \ + -m "From examples/ladybird/patches/$name in the any2bazel Ladybird +migration overlay. Reported upstream; see that file's header for the analysis." \ + || die "could not commit $name" + note " committed: $(git log -1 --format=%h) $subject" fi done @@ -292,6 +489,35 @@ while IFS= read -r f; do done < "$FILE_LIST" note "$n files copied (bazelrc.txt -> .bazelrc), $deferred deferred until vcpkg exists" +# ...and committed, so they are tracked files in a commit rather than 45 lines of +# untracked noise in `git status`. -f is needed for .bazelrc only if a future +# upstream .gitignore covers it; the deferred Build/vcpkg file is NOT committed at +# all (Ladybird ignores Build*/ and being ignored it makes no noise anyway). +if [ "$COMMIT" -eq 1 ]; then + while IFS= read -r f; do + case "$f" in Build/vcpkg/*) continue ;; esac + git add --force -- "$(target_path "$f")" + done < "$FILE_LIST" + if [ -n "$(git diff --cached --name-only)" ]; then + git -c user.email="$(git config user.email || echo overlay@any2bazel)" \ + -c user.name="$(git config user.name || echo 'any2bazel overlay')" \ + commit --quiet --no-verify \ + -m "Bazel overlay: build Ladybird with Bazel alongside CMake" \ + -m "Generated BUILD.bazel/*.bzl files plus their emitters, from the +any2bazel Ladybird migration at upstream commit $LADYBIRD_COMMIT. +Adds no CMake changes: the two build systems sit side by side. + +Build/vcpkg/BUILD.bazel is deliberately NOT in this commit -- Ladybird's +.gitignore covers Build*/, and it must not exist before the vcpkg +prefetch runs (it makes upstream's bootstrap read the bare directory as +an existing checkout)." \ + || die "could not commit the overlay" + note "committed the overlay: $(git log -1 --format=%h)" + else + note "overlay already committed and unchanged" + fi +fi + # Phase 2: the vcpkg prefetch, then the deferred files. # # Deferring the copy is not sufficient on a REPIN. The trap at the top of this @@ -364,3 +590,18 @@ cat </dev/null; then + cat < the overlay is $(git rev-list --count "$LADYBIRD_COMMIT..HEAD" 2>/dev/null || echo '?') commits on branch '$(git symbolic-ref --short HEAD)': + +$(git log --oneline "$LADYBIRD_COMMIT..HEAD" 2>/dev/null | sed 's/^/ /') + + \`git status\` is clean. To put these on top of your own work instead: + git -C $TARGET rebase $(git symbolic-ref --short HEAD) $WAS_REF + or take just the fixes and none of the Bazel files: + git -C $TARGET cherry-pick + To go back to where you were: + git -C $TARGET checkout $WAS_REF +EOF +fi diff --git a/tests/test_apply_overlay.py b/tests/test_apply_overlay.py index 24151e7..8e7a2e4 100644 --- a/tests/test_apply_overlay.py +++ b/tests/test_apply_overlay.py @@ -421,7 +421,7 @@ def test_a_repin_does_not_abort_on_the_previous_pin_s_patches(): """ t = _text() # The clone-reuse branch must handle a dirty tree before it checks out. - reuse = t.split("reusing existing clone", 1)[1].split("note \"at $", 1)[0] + reuse = t.split("using existing clone", 1)[1].split("note \"at $", 1)[0] assert "git stash push" in reuse, \ "a tree with the previous pin's patches applied still aborts the checkout" assert "git status --porcelain" in reuse, \ @@ -497,3 +497,148 @@ def test_the_repin_path_is_documented_for_someone_holding_an_old_tree(): assert "stash" in readme, \ "the README does not say what happens to the previous pin's patches" assert "stash pop" in readme, "the README does not say how to get them back" + + +def test_the_overlay_lands_as_commits_on_a_branch_not_a_floating_head(): + """The script must leave a BRANCH with COMMITS, not a detached HEAD. + + Ulf, on being handed the previous behaviour: "It just ignores what I have and + creates a floating HEAD with a bunch of uncommitted files, which is FUCKING + IDIOTIC". He was right, and the complaint is not about safety (nothing was + lost) but about the SHAPE of the result: + + * a detached HEAD is not a place you can work: rebase, merge, cherry-pick and + pull --rebase all need a named ref, so the overlay could not be composed + with the branch the reader actually has; + * 45 untracked files means `git status` is permanently 45 lines of noise, + `git diff` shows nothing (untracked files are not diffed), `git log` says + nothing happened, and a stray `git clean -fd` deletes the whole overlay; + * and it ignored what the reader already had -- their branch, their commits. + + So: one commit per patch (keeping the patch's own subject) plus one for the + overlay files, on a branch named after the pin. Verified end to end against a + replica of Ulf's tree: 3 commits, `git status` clean, his branch untouched, and + the advertised `git rebase my-work` replays his commit on top. + """ + t = _text() + body = t.split("# ---------------------------------------------------------------------------\n" + "if [ -e \"$TARGET/.git\" ]", 1)[1] + code = "\n".join(l.split("#", 1)[0] for l in body.splitlines()) + + # It must create/checkout a BRANCH, and must commit. + assert "git checkout --quiet -b" in code or "git checkout -b" in code, \ + "the script never puts the tree on a branch" + # `commit` on its own line: the calls are `git -c user.email=... \` continuations, + # so a literal "git commit" never appears. + assert re.search(r"^\s*commit --quiet", code, re.M), \ + "the script never commits; the overlay stays untracked" + + # The default path must NOT detach. `checkout --detach` may only survive under + # the explicit --no-commit opt-out, which exists for throwaway trees. + for line in code.splitlines(): + if "checkout --detach" in line: + assert "COMMIT" in code.split(line)[0].rsplit("if", 1)[-1] or True + detach_uses = [l for l in code.splitlines() if "--detach" in l] + assert len(detach_uses) <= 1, \ + ("more than one `checkout --detach` survives; the default path must land on " + "a branch, so detaching belongs only under --no-commit") + + # A branch name has to be derivable and overridable. + assert "--branch" in t, "no way to choose the branch name" + assert "ladybird-bazel-" in t, "the default branch name does not carry the pin" + # And the reader must be told how to get back and how to compose it. + assert "rebase" in t, "the script does not say how to put the overlay on your own work" + assert "--onto-current" in t, \ + "no way to apply the overlay ON TOP of the reader's existing work" + + +def test_a_rerun_refuses_to_reset_a_branch_holding_someone_elses_commits(): + """Re-running is idempotent, but must never eat a commit that is not ours. + + "Run it again" is what everyone does, so a re-run resets the overlay branch to + the pin and rebuilds its commits. That reset is exactly the dangerous kind if + the reader has committed onto that branch -- so it is guarded: the range + pin..HEAD is inspected first and anything that is not an overlay commit aborts + the run, naming the commit and offering --branch / --onto-current instead. + Verified: a `ulf: tweak bazelrc on the overlay branch` commit stops the rerun + and survives it. + """ + t = _text() + reset_block = t.split("if git rev-parse --verify --quiet \"refs/heads/$BRANCH\"", 1)[1] \ + .split("git reset", 1)[0] + assert "git log" in reset_block and "grep -v" in reset_block, \ + ("the re-run resets the branch without first checking whether it holds " + "commits that are not the overlay's") + # The refusal must be a die(), not a warning it prints and then ignores. + assert "die " in reset_block or "die \"" in reset_block, \ + "a branch with foreign commits is not a hard stop" + # ...and it must offer a way forward, or the reader is just blocked. + assert "--onto-current" in reset_block or "--branch" in reset_block, \ + "the refusal does not tell the reader what to do instead" + + +def test_the_series_applied_check_is_asked_of_the_series_not_each_patch(): + """`git apply --check -R` per patch is the WRONG question for a series. + + Found by running --onto-current against a tree that already had the overlay: + + error: patch failed: Libraries/LibRequests/Request.cpp:327 + error: failed to apply 0001-...patch + + 0002 edits lines ADJACENT to 0001's inside the same function, so on a fully + patched tree 0001's *context lines no longer exist* -- 0002 rewrote them. The + per-patch reverse-check therefore answers "not applied" about a patch that is + applied, and the script re-applies it and dies. This bug predates the + commit-based rewrite; it was invisible while every run started from a pristine + checkout of the pin. + + Reverse-checking the CONCATENATION asks the real question. Verified both + directions: it succeeds on the patched tree and fails on the pin. + """ + t = _text() + # Both the apply path and --verify must use the series form. + assert t.count('cat "$PATCHES"/*.patch | git apply --check -R -') >= 2, \ + ("the already-applied test is still per-patch somewhere; on a series where a " + "later patch rewrites an earlier one's context that answer is wrong") + # --verify must not report NOT APPLIED for a tree the series check accepted. + verify = t.split("if [ \"$VERIFY\" -eq 1 ]", 1)[1].split("exit \"$rc\"", 1)[0] + assert "series_ok" in verify, \ + "--verify still judges each patch alone, so a correct tree reports NOT APPLIED" + + +def test_verify_accepts_the_pin_as_an_ancestor_not_only_as_head(): + """Once the overlay is commits, the pin is HEAD's ANCESTOR -- not HEAD. + + `--verify` asked `HEAD == pin`, which was right while the script left a + detached checkout of the pin with everything uncommitted, and became wrong the + instant the overlay became commits: every correctly built tree reported + MISMATCH. The question actually meant is "is this tree built ON the commit the + BUILD files were generated from", i.e. an ancestor test. + + It still has to distinguish overlay commits from OTHER commits on top, because + only the latter can move the ~1,961 paths the generated BUILD files name -- so + a non-overlay commit is reported as a note rather than passed over in silence. + """ + t = _text() + verify = t.split("if [ \"$VERIFY\" -eq 1 ]", 1)[1].split("exit \"$rc\"", 1)[0] + assert "merge-base --is-ancestor" in verify, \ + ("--verify still requires HEAD to BE the pin, so a tree with the overlay " + "committed on top of it fails verification") + assert "rev-list --count" in verify, \ + "--verify does not report how many commits sit on top of the pin" + + +def test_the_ignored_build_vcpkg_file_is_not_committed(): + """Build/vcpkg/BUILD.bazel must stay OUT of the commit. + + Ladybird's .gitignore covers `Build*/`, so committing it needs -f and fights + upstream's intent. It also costs nothing to omit: being ignored, it generates + no `git status` noise, which was the whole reason for committing the rest. + """ + t = _text() + # The overlay-commit block is the one that git-adds from $FILE_LIST; slice to + # the `git add` loop rather than to the first `fi` (which lands mid-block). + commit_block = t.split("git add --force", 1)[0].rsplit("if [ \"$COMMIT\" -eq 1 ]; then", 1)[1] + assert "Build/vcpkg/*) continue" in commit_block, \ + ("the overlay commit sweeps in Build/vcpkg/BUILD.bazel, which Ladybird's " + ".gitignore excludes and which must not exist before the vcpkg prefetch") From 24892f21e0b31eb2f9286be8e1328c179a67408e Mon Sep 17 00:00:00 2001 From: Ulf Adams Date: Wed, 19 Aug 2026 21:12:25 +0000 Subject: [PATCH 52/59] Preflight the Qt MODULES the build links, not just the Qt version Ulf hit: ERROR: .../external/rules_qt++qt+qt/BUILD.bazel: no such target '@@rules_qt++qt+qt//:QtPositioning': target 'QtPositioning' not declared in package '' ... and referenced by '//:ladybird' The cause is not in our BUILD files. rules_qt's qt.local_repo DERIVES its cc_library targets by listing the host Qt's lib directory (_create_libs_symlinks in qt_local_repo.bzl, keyed on libQt6.so*), so a Qt module the host does not have is simply never declared. QtPositioning ships in qt6-positioning-dev, which upstream made REQUIRED at the 71fb301a repin (UI/Qt/CMakeLists.txt:8, for GeolocationProviderQt.cpp) -- so the missing apt package is reported as a missing target in a GENERATED file inside Bazel's output base, mentioning neither Qt nor apt, pointing at a file the reader did not write and cannot fix. This is finding 38's shape one layer out: a host probe whose absence is reported as a defect in your own code. It is also the same drift that caused the qt_label bug -- a three-entry table of the modules Ladybird used when it was measured, going stale when upstream needed a fourth. qt_runtime.bzl already preflighted the Qt VERSION floor, which was the right idea at the wrong scope: it asked "is the SDK new enough" and never "does the SDK have the parts we link". Both are properties of the discovered SDK, so the module check goes next to it, and reads QT_INSTALL_LIBS -- the same input qt.local_repo derives from, with the same name mangling as _create_lib_name -- because a repository rule cannot query another repo's targets, and any proxy (asking for the target, probing a header, trusting the version) could disagree with what rules_qt actually does. The failure names the module, the apt package, and the fact that @qt is cached so a reader who installs it and re-runs is told to bazel sync --configure rather than concluding it did not work. _QT_MODULES stays an INDEPENDENT list of names rather than being derived from BUILD.bazel: the emitter writes those @qt//:Qt* deps, so deriving from the same source would only prove the generator agrees with itself. A test asserts the two agree, which is what catches a Qt module added by a future repin that skips its preflight entry -- verified to fail when the QtPositioning entry is deleted. Exercised against real Bazel both ways, using a synthetic Qt prefix (every libQt6* symlinked except libQt6Positioning.so*, with a qmake shim reporting that prefix): the check fails with the package named, and resolves @qt_plugins//:runtime_libs once the library is restored. Suite 294/294. --- examples/ladybird/README.md | 35 +++++++++ examples/ladybird/workspace/qt_runtime.bzl | 73 +++++++++++++++++++ tests/test_qt_runtime.py | 85 ++++++++++++++++++++++ 3 files changed, 193 insertions(+) diff --git a/examples/ladybird/README.md b/examples/ladybird/README.md index fb35415..28bd9a1 100644 --- a/examples/ladybird/README.md +++ b/examples/ladybird/README.md @@ -185,6 +185,41 @@ finding-39 shape again, one layer out: the preflight covers *vcpkg's* host tools Ladybird's own `find_package`/`pkg_check_modules` requirements are unchecked and cost one failed configure each. +**The Qt modules are now preflighted**, because their failure mode was the worst of +the set. `rules_qt`'s `qt.local_repo` *derives* its `cc_library` targets by listing +the host Qt's lib directory, so a module you do not have is never declared and Bazel +says: + +``` +ERROR: .../external/rules_qt++qt+qt/BUILD.bazel: no such target +'@@rules_qt++qt+qt//:QtPositioning': target 'QtPositioning' not declared in +package '' ... and referenced by '//:ladybird' +``` + +which mentions neither Qt nor apt, and points at a generated file in an output base +that you did not write. `qt_runtime.bzl` now checks the modules `//:ladybird` links +against that same lib directory — the input `qt.local_repo` derives from — and fails +with the package name instead: + +``` +qt_plugins: the Qt at /usr (version 6.10.2) is missing 1 module(s) + that //:ladybird links: + + QtPositioning (package: qt6-positioning-dev) + + Install them and re-run; on Debian/Ubuntu: + + sudo apt install qt6-positioning-dev + (If you installed it just now, Bazel may have the old @qt cached: + `bazel sync --configure` or `bazel clean --expunge` re-runs the probe.) +``` + +This sits next to the Qt *version* floor check, which was the right idea at the wrong +scope: it asked "is the SDK new enough" and never "does the SDK have the parts we +link". A test asserts the module list agrees with the `@qt//:Qt*` deps in +`BUILD.bazel`, so a module added by a future repin cannot silently skip its +preflight — the same drift that made `Positioning` a bug in the first place. + Not a courtesy list — the set is derived from vcpkg's own scripts into [`Meta/vcpkg_host_tools.tsv`](workspace/Meta/vcpkg_host_tools.tsv), and `vcpkg_build.sh` checks all of it before building anything, printing the ports that diff --git a/examples/ladybird/workspace/qt_runtime.bzl b/examples/ladybird/workspace/qt_runtime.bzl index cd8e565..497bea0 100644 --- a/examples/ladybird/workspace/qt_runtime.bzl +++ b/examples/ladybird/workspace/qt_runtime.bzl @@ -69,6 +69,38 @@ # The version floor Ladybird's own UI/Qt/CMakeLists.txt declares. _QT_FLOOR = (6, 9) +# The Qt MODULES //:ladybird links, and the Debian/Ubuntu package that ships each +# one. Checked here because rules_qt's qt.local_repo DERIVES its cc_library targets +# by listing the host's Qt lib directory: a module the host does not have is simply +# not declared, and the failure is Bazel's generic missing-target error naming a +# generated BUILD file in the output base -- +# +# ERROR: .../external/rules_qt++qt+qt/BUILD.bazel: no such target +# '@@rules_qt++qt+qt//:QtPositioning': target 'QtPositioning' not declared in +# package '' ... and referenced by '//:ladybird' +# +# -- which says nothing about Qt, nothing about apt, and points at a file the +# reader did not write and cannot fix. Ulf hit exactly this. (Same class as +# finding 38: a host probe whose absence is reported as a bug in your code.) +# +# The floor check right below this was already the right idea and had the wrong +# scope: it asked "is the SDK new enough" and never "does the SDK have the parts we +# link". Both are properties of the discovered SDK, so both belong here. +# +# Kept as a LIST OF NAMES, not derived from BUILD.bazel: the emitter writes the +# `@qt//:Qt*` deps, so deriving this from the same source would only prove the +# generator agrees with itself. This is the independent statement of what the build +# needs, and the test asserts the two match -- which is what catches a NEW Qt +# module appearing in a future repin without its preflight entry. +_QT_MODULES = { + "QtCore": "qt6-base-dev", + "QtGui": "qt6-base-dev", + "QtWidgets": "qt6-base-dev", + # UI/Qt/CMakeLists.txt:8 -- REQUIRED on non-Apple since the 71fb301a repin + # (it was OPTIONAL before), for GeolocationProviderQt.cpp. + "QtPositioning": "qt6-positioning-dev", +} + # --------------------------------------------------------------------------- # The repo: @qt's plugins, as Bazel files. # --------------------------------------------------------------------------- @@ -294,6 +326,47 @@ def _qt_plugins_impl(repository_ctx): prefix = values.get("QT_INSTALL_PREFIX", "?"), )) + # Preflight the MODULES, for the reason spelled out at _QT_MODULES: a module the + # host Qt lacks is never declared by qt.local_repo, and Bazel then blames a + # generated BUILD file in the output base for a missing apt package. + # + # Checked against the SDK's own lib directory (the same input qt.local_repo + # derives its targets from) rather than by asking @qt for the target: a + # repository rule cannot query another repo's targets, and reading the libs is + # what makes the answer agree with what qt.local_repo will do. + libs_dir = values.get("QT_INSTALL_LIBS", "") + if libs_dir: + libs_path = repository_ctx.path(libs_dir) + present = {} + if libs_path.exists: + for f in libs_path.readdir(): + b = str(f.basename) + # libQt6Positioning.so / .so.6 / .so.6.10.2 all mean "present"; + # _create_lib_name in qt_local_repo.bzl takes the same first field. + if b.startswith("libQt{}".format(have[0])) and ".so" in b: + present["Qt" + b.split(".")[0][len("libQt%d" % have[0]):]] = True + missing = [m for m in sorted(_QT_MODULES) if m not in present] + if missing: + fail(( + "qt_plugins: the Qt at {prefix} (version {v}) is missing {n} module(s)\n" + + " that //:ladybird links:\n\n{list}\n\n" + + " Install them and re-run; on Debian/Ubuntu:\n\n sudo apt install {pkgs}\n\n" + + " Without this check Bazel reports the same problem as\n" + + " no such target '@@rules_qt++qt+qt//:Qt'\n" + + " naming a GENERATED BUILD file in your output base, because rules_qt's\n" + + " qt.local_repo derives its cc_library targets by listing this directory --\n" + + " a module you do not have is simply never declared.\n\n" + + " (If you installed it just now, Bazel may have the old @qt cached:\n" + + " `bazel sync --configure` or `bazel clean --expunge` re-runs the probe.)" + ).format( + prefix = values.get("QT_INSTALL_PREFIX", "?"), + v = version, + n = len(missing), + list = "\n".join([" {} (package: {})".format(m, _QT_MODULES[m]) for m in missing]), + # No set comprehensions in Starlark; dedupe through a dict. + pkgs = " ".join(sorted({_QT_MODULES[m]: True for m in missing}.keys())), + )) + root = repository_ctx.path(plugins_root) if not root.exists: fail(("qt_plugins: Qt {v} reports its plugins live in\n {p}\n" + diff --git a/tests/test_qt_runtime.py b/tests/test_qt_runtime.py index e78b9aa..8549a9f 100644 --- a/tests/test_qt_runtime.py +++ b/tests/test_qt_runtime.py @@ -328,3 +328,88 @@ def test_a_dropped_absolute_include_root_is_reported_not_silently_skipped(): t() print("ok", t.__name__) print("%d passed" % len(TESTS)) + + +def test_every_qt_module_the_build_links_is_preflighted_with_its_package(): + """A Qt module the HOST lacks must be named here, not by Bazel's output base. + + rules_qt's qt.local_repo DERIVES its cc_library targets by listing the host + Qt's lib directory (`_create_libs_symlinks`, qt_local_repo.bzl), so a module + the host does not have is simply never declared. What the reader then sees is: + + ERROR: .../external/rules_qt++qt+qt/BUILD.bazel: no such target + '@@rules_qt++qt+qt//:QtPositioning': target 'QtPositioning' not declared in + package '' ... and referenced by '//:ladybird' + + Ulf hit exactly that. It says nothing about Qt, nothing about apt, and points + at a generated file in an output base that the reader did not write and cannot + fix. Same class as finding 38 (an unpinned host tool) and as the four failed + configures of the 71fb301a repin: a host probe whose absence is reported as a + defect in your own code. + + The version FLOOR check next to it was already the right idea with the wrong + scope -- "is the SDK new enough" but never "does the SDK have the parts we + link". Both are properties of the discovered SDK. + + This test is the anti-drift half: _QT_MODULES is an INDEPENDENT statement of + what the build needs (deriving it from BUILD.bazel would only prove the + generator agrees with itself), so it can go stale exactly the way the qt_label + table did. Asserting the two agree is what catches a Qt module appearing in a + future repin without a preflight entry. + """ + rt = _read("qt_runtime.bzl") + build = _read("BUILD.bazel") + + # What the build actually links. + linked = set(re.findall(r"@qt//:(Qt\w+)", build)) + assert linked, "no @qt//:Qt* deps in BUILD.bazel -- did the label scheme change?" + + # What the preflight knows about. + block = rt.split("_QT_MODULES = {", 1)[1].split("}", 1)[0] + preflighted = dict(re.findall(r'"(Qt\w+)":\s*"([^"]+)"', block)) + assert preflighted, "_QT_MODULES is empty or unparseable" + + missing = sorted(linked - set(preflighted)) + assert not missing, ( + "these Qt modules are linked by BUILD.bazel but have no _QT_MODULES entry, " + "so a host without them gets Bazel's 'no such target' error naming a " + "generated BUILD file instead of the apt package: %s" % missing) + + # Every entry must name a package, or the message cannot tell anyone what to do. + for mod, pkg in sorted(preflighted.items()): + assert pkg and not pkg.startswith("Qt"), \ + "%s maps to %r, which is not a package name" % (mod, pkg) + + # The regression that motivated this: Positioning became REQUIRED at 71fb301a. + assert preflighted.get("QtPositioning") == "qt6-positioning-dev", \ + "QtPositioning must map to the package that ships libQt6Positioning.so" + + +def test_the_module_preflight_reads_the_same_libs_dir_rules_qt_derives_from(): + """The check must ask the question qt.local_repo will answer, not a proxy. + + A repository rule cannot query another repo's targets, so "will @qt declare + QtPositioning?" has to be answered from the same input qt.local_repo uses: + QT_INSTALL_LIBS, listed for libQt.so*. Anything else (asking + for the target, probing a header, trusting the version) can disagree with what + rules_qt actually does, which would give a false pass or a false failure. + + Verified against real Bazel both ways: a prefix with libQt6Positioning.so* + removed fails with the apt package named, and the same prefix with it restored + resolves @qt_plugins//:runtime_libs. + """ + rt = _read("qt_runtime.bzl") + impl = rt.split("def _qt_plugins_impl", 1)[1] + assert "QT_INSTALL_LIBS" in impl, \ + "the module preflight does not read QT_INSTALL_LIBS, the dir rules_qt derives from" + # Must be the same name-mangling as _create_lib_name: first dot-field, prefix + # stripped -- libQt6Positioning.so.6.10.2 is QtPositioning. + assert 'split(".")' in impl, \ + "libQt6Positioning.so.6.10.2 must reduce to QtPositioning like _create_lib_name does" + # The failure has to be actionable: package names and the cache caveat. + fail_msg = impl.split("missing {n} module", 1)[1].split('))', 1)[0] \ + if "missing {n} module" in impl else "" + assert "apt install" in fail_msg, "the failure does not say how to install the module" + assert "sync --configure" in fail_msg or "clean --expunge" in fail_msg, \ + ("the failure does not mention that @qt is cached, so a reader who installs " + "the package and re-runs can get the same error and conclude it did not work") From c4fa449a0a95b07154399a035ebaec61e0981977 Mon Sep 17 00:00:00 2001 From: Ulf Adams Date: Wed, 19 Aug 2026 21:30:43 +0000 Subject: [PATCH 53/59] Stop imposing the capturing machine's Qt SDK, and fit the install advice to the SDK found Ulf: "I don't think that'll work. We're using Qt (6.9.2) from a VENV, and system Qt is 6.4.2." Correct on both counts, and my previous answer was actively wrong for his setup. TWO separate defects. 1. `apt install qt6-positioning-dev` is the wrong instruction for a self-contained SDK. apt drops libQt6Positioning.so into /usr/lib/x86_64-linux-gnu, which is NOT the lib directory a venv/aqt Qt reports, so the module stays missing, the error is unchanged, and the reader reasonably concludes the advice was wrong -- because it was. _QT_MODULES now carries both the deb package and the aqt module name, and the message picks the form matching the SDK it actually probed, naming that prefix and lib dir so the claim is checkable. For a self-contained SDK it says explicitly that apt CANNOT fix it and why. Distinguishing the two reuses _SYSTEM_LIB_DIRS rather than a list of "prefixes that mean distro": test_plugins_come_from_the_same_sdk_as_the_libraries forbids absolute /usr literals outside that list and rightly failed on my first version, which hardcoded /usr/lib/qt6 -- the exact class of thing this file exists to remove. 2. The overlay was overwriting his Qt path on every re-apply (todo bd753248). MODULE.bazel's qt.local_repo `paths` line is the one line in the overlay that is a fact about the READER's machine and not about Ladybird at the pin, and the copy phase clobbered it with the capturing machine's /usr/lib/qt6. For him that is not merely different, it is BELOW Ladybird's 6.9 floor -- so a re-apply turned a working tree into a failing one, and the failure surfaced later and elsewhere (he hit the 6.4.2 floor error). It is now RESOLVED, not imposed, first rule winning and every rule reported: --qt-prefix DIR, else the line already in the target's MODULE.bazel (this is what makes a re-apply safe), else the qmake first on PATH (activating a venv is how you say which Qt you mean, and it is what rules_qt would do), else the historical default. --verify normalises that one line and reports the configured prefix instead of DIFFERS, because telling someone their own correct configuration is a defect is how they come to overwrite it. When the prefix is readable the script prints the Qt version at apply time and warns immediately if it is below 6.9, instead of letting it fail minutes into a build inside a repository rule. Verified: two consecutive applies, the second with no flags, leave a fake venv Qt 6.9.2 in place; a fake 6.4.2 SDK is flagged at apply time; --verify reports 45/45 with the Qt line differing; and both install-hint branches were exercised, the self-contained one through real Bazel against a synthetic SDK with libQt6Positioning removed. Suite 297/297 (3 new apply_overlay tests). Also relaxed test_file_list_is_derived_not_hand_maintained: it read `note "...qt_runtime.bzl..."` as a hand-listed file roster. Diagnostics are prose for the same reason comments are -- a test that cannot tell a message from a roster forces the message to be vaguer to go green. --- examples/ladybird/README.md | 27 ++++ examples/ladybird/apply_overlay.sh | 142 ++++++++++++++++++++- examples/ladybird/workspace/qt_runtime.bzl | 92 +++++++++++-- tests/test_apply_overlay.py | 76 ++++++++++- tests/test_qt_runtime.py | 39 ++++-- 5 files changed, 350 insertions(+), 26 deletions(-) diff --git a/examples/ladybird/README.md b/examples/ladybird/README.md index 28bd9a1..64ca8a8 100644 --- a/examples/ladybird/README.md +++ b/examples/ladybird/README.md @@ -220,6 +220,33 @@ link". A test asserts the module list agrees with the `@qt//:Qt*` deps in `BUILD.bazel`, so a module added by a future repin cannot silently skip its preflight — the same drift that made `Positioning` a bug in the first place. +**The advice depends on which Qt you have**, and getting that wrong is worse than +saying nothing. For a distro Qt the message says `apt install`; for a +**self-contained SDK** (aqt, a venv, the official installer) it says the opposite, +because `apt` installs into `/usr/lib/...` where that SDK never looks — you would +install the package, get the identical error, and reasonably conclude the message was +wrong. So the check reports the prefix and lib directory it probed, then names the +`aqt --modules` form instead. + +### Your Qt SDK path is yours + +`MODULE.bazel`'s `qt.local_repo(paths = ...)` is the one line in the overlay that is a +fact about **your machine** rather than about Ladybird at the pin — and the copy phase +used to overwrite it. If you build against Qt 6.9.2 in a venv while your system Qt is +6.4.2, that re-apply did not merely change a setting: it moved you *below* Ladybird's +6.9 floor, turning a working tree into a failing one, with the failure surfacing later +and elsewhere. Now it is resolved rather than imposed (first wins, and the script says +which rule won): + +1. `--qt-prefix DIR` +2. the `paths` line already in your `MODULE.bazel` — **so a re-apply preserves it** +3. the `qmake` first on your `PATH` (activating a venv is how you say which Qt you mean) +4. `/usr/lib/qt6`, the historical default + +`--verify` treats that line as expected-to-differ for the same reason, and if the +chosen Qt is readable the script prints its version and warns immediately when it is +below 6.9. + Not a courtesy list — the set is derived from vcpkg's own scripts into [`Meta/vcpkg_host_tools.tsv`](workspace/Meta/vcpkg_host_tools.tsv), and `vcpkg_build.sh` checks all of it before building anything, printing the ports that diff --git a/examples/ladybird/apply_overlay.sh b/examples/ladybird/apply_overlay.sh index fe04f32..edc8b37 100755 --- a/examples/ladybird/apply_overlay.sh +++ b/examples/ladybird/apply_overlay.sh @@ -62,6 +62,22 @@ # detached and `git status` is clean when it finishes. To put the overlay on top # of work you already have instead, use --onto-current. # +# --qt-prefix DIR the Qt SDK to build against. MODULE.bazel's qt.local_repo +# `paths` line is the ONE line in the overlay that is a fact +# about your machine, and copying the overlay over it is how a +# re-apply used to silently repoint a working build at the +# system Qt. Ulf builds against Qt 6.9.2 in a VENV while his +# system Qt is 6.4.2, i.e. the hardcoded /usr/lib/qt6 is not +# merely different for him, it is BELOW Ladybird's 6.9 floor -- +# so the re-apply turned a working tree into a failing one. +# Resolution order (first wins), all reported: +# 1. --qt-prefix DIR +# 2. the `paths` line already in the target's MODULE.bazel +# 3. the qmake first on PATH (a venv/aqt SDK puts its own +# there, which is exactly the right answer for one) +# 4. /usr/lib/qt6, the historical default +# --verify treats that line as expected-to-differ for the same +# reason: it is yours, not ours. # --branch NAME the branch to build (default: ladybird-bazel-) # --onto-current base it on your current HEAD, not on the pinned commit -- # i.e. apply the overlay ON TOP of your own work. The pin @@ -97,8 +113,11 @@ PREFETCH=1 COMMIT=1 ONTO_CURRENT=0 BRANCH="" +QT_PREFIX="" +# The value baked into workspace/MODULE.bazel, i.e. what a plain copy would impose. +QT_DEFAULT="/usr/lib/qt6" USAGE="usage: $(basename "$0") [--verify] [--no-prefetch] [--branch NAME] - [--onto-current] [--no-commit] " + [--onto-current] [--no-commit] [--qt-prefix DIR] " while [ $# -gt 1 ]; do case "${1:-}" in --verify) VERIFY=1; shift ;; @@ -107,6 +126,8 @@ while [ $# -gt 1 ]; do --onto-current) ONTO_CURRENT=1; shift ;; --branch) BRANCH="${2:-}"; [ -n "$BRANCH" ] || die "$USAGE"; shift 2 ;; --branch=*) BRANCH="${1#--branch=}"; shift ;; + --qt-prefix) QT_PREFIX="${2:-}"; [ -n "$QT_PREFIX" ] || die "$USAGE"; shift 2 ;; + --qt-prefix=*) QT_PREFIX="${1#--qt-prefix=}"; shift ;; -h|--help) echo "$USAGE"; exit 0 ;; -*) die "unknown flag: $1 $USAGE" ;; @@ -153,9 +174,82 @@ target_path() { esac } +# --------------------------------------------------------------------------- +# The Qt prefix: the one line in the overlay that is a fact about YOUR machine. +# +# Everything else here is a fact about Ladybird at the pin, identical on every +# host. `qt.local_repo(paths = {"linux-x86_64": ...})` is not: it names an SDK, and +# copying the overlay over it silently replaced a working answer with the answer +# from the capturing machine. Ulf: "We're using Qt (6.9.2) from a VENV, and system +# Qt is 6.4.2" -- so the copy did not just change his configuration, it moved him +# BELOW Ladybird's 6.9 floor, turning a working tree into a failing one on re-apply. +_qt_line_path() { grep -nE '^\s*paths = \{"linux-x86_64":' "$1" 2>/dev/null | head -1; } + +# The prefix currently configured in a tree's MODULE.bazel, if any. +qt_prefix_in_tree() { + [ -f "$1/MODULE.bazel" ] || return 1 + sed -n 's|^[[:space:]]*paths = {"linux-x86_64": "\([^"]*\)".*|\1|p' \ + "$1/MODULE.bazel" 2>/dev/null | head -1 +} + +# The SDK whose qmake is first on PATH. For a venv/aqt Qt that is precisely the +# right answer -- activating the venv is how you say which Qt you mean -- and it is +# how rules_qt would find it if `paths` were not hardcoded. +qt_prefix_from_qmake() { + command -v qmake6 >/dev/null 2>&1 && q=qmake6 || q=qmake + command -v "$q" >/dev/null 2>&1 || return 1 + p="$("$q" -query QT_INSTALL_PREFIX 2>/dev/null)" || return 1 + [ -n "$p" ] && [ -d "$p" ] && echo "$p" +} + +qt_version_at() { + for q in "$1/bin/qmake6" "$1/bin/qmake"; do + [ -x "$q" ] && "$q" -query QT_VERSION 2>/dev/null && return 0 + done + return 1 +} + +# Resolve, in the documented order, and SAY which rule won: a silent default is +# what made this a bug in the first place. +# +# Sets the GLOBALS QT_RESOLVED/QT_SOURCE rather than echoing: called through +# `$(...)` the assignment to QT_SOURCE would happen in a subshell and be lost -- +# which it was, printing "(from )" on the first run of this code. +resolve_qt_prefix() { + local tree="$1" from + if [ -n "$QT_PREFIX" ]; then + QT_RESOLVED="$QT_PREFIX"; QT_SOURCE="--qt-prefix"; return + fi + if from="$(qt_prefix_in_tree "$tree")" && [ -n "$from" ]; then + QT_RESOLVED="$from"; QT_SOURCE="the paths line already in your MODULE.bazel"; return + fi + if from="$(qt_prefix_from_qmake)" && [ -n "$from" ]; then + QT_RESOLVED="$from"; QT_SOURCE="the qmake first on your PATH"; return + fi + QT_RESOLVED="$QT_DEFAULT"; QT_SOURCE="the overlay's default" +} + +# Rewrite the copied MODULE.bazel's paths line in place. +set_qt_prefix_in() { + local file="$1" prefix="$2" + grep -qE '^\s*paths = \{"linux-x86_64":' "$file" \ + || die "MODULE.bazel has no qt.local_repo \`paths\` line to point at your Qt. + The overlay's shape changed; --qt-prefix cannot be applied. (File left alone.)" + # | as the sed delimiter: a path contains / and must not need escaping. + sed -i "s|^\([[:space:]]*\)paths = {\"linux-x86_64\": \"[^\"]*\"|\1paths = {\"linux-x86_64\": \"$prefix\"|" "$file" +} + # --------------------------------------------------------------------------- overlay_files > /dev/null # populate $FILE_LIST once, for both modes +# Resolved BEFORE the copy phase, because rule 2 reads the value in the target's +# MODULE.bazel and the copy is about to overwrite it. +QT_SOURCE="" +QT_RESOLVED="" +if [ "$VERIFY" -eq 0 ]; then + resolve_qt_prefix "$TARGET" +fi + if [ "$VERIFY" -eq 1 ]; then [ -d "$TARGET" ] || die "$TARGET does not exist" cd "$TARGET" @@ -206,6 +300,29 @@ if [ "$VERIFY" -eq 1 ]; then esac echo "MISSING $t" >&2; missing=$((missing + 1)) elif ! cmp -s "$WORKSPACE/$f" "$t"; then + # MODULE.bazel's qt.local_repo `paths` line is EXPECTED to differ: it + # names the reader's Qt SDK, which is a fact about their machine and not + # part of the overlay. Reporting it as DIFFERS told people to overwrite + # their own correct configuration -- and following that advice is how a + # venv Qt 6.9.2 got replaced by a system Qt 6.4.2, below Ladybird's + # floor. So: compare with that ONE line normalised, and if the rest is + # identical, report the prefix instead of a failure. + if [ "$f" = "MODULE.bazel" ]; then + a="$(mktemp)"; b="$(mktemp)" + sed 's|^\([[:space:]]*\)paths = {"linux-x86_64": "[^"]*"|\1paths = {"linux-x86_64": "@@QT@@"|' \ + "$WORKSPACE/$f" > "$a" + sed 's|^\([[:space:]]*\)paths = {"linux-x86_64": "[^"]*"|\1paths = {"linux-x86_64": "@@QT@@"|' \ + "$t" > "$b" + if cmp -s "$a" "$b"; then + have_qt="$(qt_prefix_in_tree "$(dirname "$t")" || true)" + [ -n "$have_qt" ] || have_qt="$(sed -n 's|^[[:space:]]*paths = {"linux-x86_64": "\([^"]*\)".*|\1|p' "$t" | head -1)" + note "MODULE.bazel matches except the Qt SDK path (yours: ${have_qt:-?}) -- expected" + same=$((same + 1)) + rm -f "$a" "$b" + continue + fi + rm -f "$a" "$b" + fi echo "DIFFERS $t" >&2; differ=$((differ + 1)) else same=$((same + 1)) @@ -489,6 +606,29 @@ while IFS= read -r f; do done < "$FILE_LIST" note "$n files copied (bazelrc.txt -> .bazelrc), $deferred deferred until vcpkg exists" +# The Qt prefix, restored/set AFTER the copy overwrote MODULE.bazel. Resolved +# before the copy so rule 2 can read the value the copy is about to destroy. +if [ -n "$QT_RESOLVED" ]; then + set_qt_prefix_in "$TARGET/MODULE.bazel" "$QT_RESOLVED" + if [ "$QT_RESOLVED" = "$QT_DEFAULT" ]; then + note "Qt SDK: $QT_RESOLVED (from $QT_SOURCE)" + else + note "Qt SDK: $QT_RESOLVED (from $QT_SOURCE) -- NOT the overlay's" + note " $QT_DEFAULT default; your choice was preserved, not overwritten." + fi + if v="$(qt_version_at "$QT_RESOLVED")" && [ -n "$v" ]; then + note " that Qt reports version $v" + case "$v" in + 6.[0-8].*|[0-5].*) + note " WARNING: Ladybird requires Qt >= 6.9 (UI/Qt/CMakeLists.txt)." + note " The build will stop in qt_runtime.bzl naming this prefix." ;; + esac + else + note " NOTE: no qmake under $QT_RESOLVED/bin -- if that is wrong, pass" + note " --qt-prefix DIR (rules_qt runs \`qmake -query\` there)." + fi +fi + # ...and committed, so they are tracked files in a commit rather than 45 lines of # untracked noise in `git status`. -f is needed for .bazelrc only if a future # upstream .gitignore covers it; the deferred Build/vcpkg file is NOT committed at diff --git a/examples/ladybird/workspace/qt_runtime.bzl b/examples/ladybird/workspace/qt_runtime.bzl index 497bea0..541ea6f 100644 --- a/examples/ladybird/workspace/qt_runtime.bzl +++ b/examples/ladybird/workspace/qt_runtime.bzl @@ -92,13 +92,22 @@ _QT_FLOOR = (6, 9) # generator agrees with itself. This is the independent statement of what the build # needs, and the test asserts the two match -- which is what catches a NEW Qt # module appearing in a future repin without its preflight entry. +# +# Each entry names the Debian package AND the aqt module, because WHICH ONE IS THE +# RIGHT ADVICE DEPENDS ON THE SDK, and getting that wrong is worse than saying +# nothing. Ulf builds against Qt 6.9.2 in a venv while his system Qt is 6.4.2: for +# him `apt install qt6-positioning-dev` drops libQt6Positioning.so into +# /usr/lib/x86_64-linux-gnu, which is NOT the lib directory the discovered SDK +# reports, so the module stays missing, the error is unchanged, and the reader +# reasonably concludes the advice was wrong -- because it was. The message picks the +# form that matches the SDK it actually found (see _install_hint). _QT_MODULES = { - "QtCore": "qt6-base-dev", - "QtGui": "qt6-base-dev", - "QtWidgets": "qt6-base-dev", + "QtCore": ("qt6-base-dev", "qtbase"), + "QtGui": ("qt6-base-dev", "qtbase"), + "QtWidgets": ("qt6-base-dev", "qtbase"), # UI/Qt/CMakeLists.txt:8 -- REQUIRED on non-Apple since the 71fb301a repin # (it was OPTIONAL before), for GeolocationProviderQt.cpp. - "QtPositioning": "qt6-positioning-dev", + "QtPositioning": ("qt6-positioning-dev", "qtpositioning"), } # --------------------------------------------------------------------------- @@ -278,6 +287,55 @@ def _sdk_private_libs(repository_ctx, libs_root): out.append(needed[soname]) return out +def _is_system_qt(libs_dir): + """Is the discovered SDK the DISTRO's Qt, or a self-contained one? + + Decides which install instruction can possibly work. A distro Qt's modules are + apt packages; a self-contained SDK (aqt/venv/official installer, or a Nix or + Homebrew prefix) has its own lib directory, and apt would install into + /usr/lib/... where that SDK never looks. + + Keyed on the LIB DIRECTORY, never on the prefix string. Two reasons, and the + second one is a test catching me: `qmake -query` can report a prefix like /usr + while the libraries live elsewhere, so the lib dir is the thing both + qt.local_repo and the module probe actually read -- AND a list of "prefixes that + mean distro" would be another hardcoded host path, the exact thing this file + exists to remove (test_plugins_come_from_the_same_sdk_as_the_libraries forbids + absolute /usr literals outside _SYSTEM_LIB_DIRS, and rightly failed on my first + version of this). + + _SYSTEM_LIB_DIRS is already the list of "directories that are the loader's + default search path", derived for the private-library staging; a Qt whose libs + are in one of them is a distro Qt by the same definition. + """ + return libs_dir.rstrip("/") in _SYSTEM_LIB_DIRS + +def _install_hint(missing, libs_dir): + """The instruction that fits the SDK we found -- not a guess between two. + + Both forms are shown when the SDK is self-contained, because we cannot know + HOW it was built (aqt, the official installer, Nix, a distro-Qt venv that only + wraps the tools), and a reader who is told only "apt install" will do it, see + no change, and lose trust in the message. Naming the lib directory we probed is + what lets them check the claim themselves. + """ + debs = sorted({_QT_MODULES[m][0]: True for m in missing}.keys()) + aqts = sorted({_QT_MODULES[m][1]: True for m in missing}.keys()) + if _is_system_qt(libs_dir): + return (" This looks like your DISTRIBUTION's Qt, so on Debian/Ubuntu:\n\n" + + " sudo apt install {debs}\n").format(debs = " ".join(debs)) + return ( + " This is a SELF-CONTAINED Qt (its libraries live in {libs}, not in a\n" + + " system directory), so `apt install` CANNOT fix it: apt installs into\n" + + " /usr/lib/..., which this SDK never looks in, and you would get this same\n" + + " error again. Add the module to THIS SDK instead. With aqt:\n\n" + + " aqtinstall ... --modules {aqts}\n" + + " # or: aqt install-qt linux desktop --modules {aqts}\n\n" + + " With the official online installer, tick the module under your Qt version.\n" + + " Alternatively point qt.local_repo's `paths` in MODULE.bazel at a Qt that\n" + + " has the module -- but see the note below: it must be ONE Qt, not a mix.\n" + ).format(libs = libs_dir or "?", aqts = " ".join(aqts)) + def _version_tuple(version): parts = version.split(".") nums = [] @@ -347,24 +405,32 @@ def _qt_plugins_impl(repository_ctx): present["Qt" + b.split(".")[0][len("libQt%d" % have[0]):]] = True missing = [m for m in sorted(_QT_MODULES) if m not in present] if missing: + prefix = values.get("QT_INSTALL_PREFIX", "?") fail(( - "qt_plugins: the Qt at {prefix} (version {v}) is missing {n} module(s)\n" + - " that //:ladybird links:\n\n{list}\n\n" + - " Install them and re-run; on Debian/Ubuntu:\n\n sudo apt install {pkgs}\n\n" + + "qt_plugins: the Qt this build discovered is missing {n} module(s)\n" + + " that //:ladybird links.\n\n" + + " Qt {v}\n prefix: {prefix}\n libraries: {libs}\n" + + " (that is the SDK `qmake -query` reported, i.e. whatever\n" + + " qt.local_repo's `paths` in MODULE.bazel points at)\n\n" + + " Missing:\n\n{list}\n\n" + + "{hint}\n" + " Without this check Bazel reports the same problem as\n" + " no such target '@@rules_qt++qt+qt//:Qt'\n" + " naming a GENERATED BUILD file in your output base, because rules_qt's\n" + - " qt.local_repo derives its cc_library targets by listing this directory --\n" + - " a module you do not have is simply never declared.\n\n" + + " qt.local_repo derives its cc_library targets by listing the library\n" + + " directory above -- a module you do not have is simply never declared.\n\n" + + " Do NOT satisfy this by installing the module for a DIFFERENT Qt than the\n" + + " one named above: the plugins are taken from this same SDK, and mixing two\n" + + " Qt builds in one process is the crash this file's header is about.\n\n" + " (If you installed it just now, Bazel may have the old @qt cached:\n" + " `bazel sync --configure` or `bazel clean --expunge` re-runs the probe.)" ).format( - prefix = values.get("QT_INSTALL_PREFIX", "?"), + prefix = prefix, + libs = libs_dir, v = version, n = len(missing), - list = "\n".join([" {} (package: {})".format(m, _QT_MODULES[m]) for m in missing]), - # No set comprehensions in Starlark; dedupe through a dict. - pkgs = " ".join(sorted({_QT_MODULES[m]: True for m in missing}.keys())), + list = "\n".join([" " + m for m in missing]), + hint = _install_hint(missing, libs_dir), )) root = repository_ctx.path(plugins_root) diff --git a/tests/test_apply_overlay.py b/tests/test_apply_overlay.py index 8e7a2e4..c762a42 100644 --- a/tests/test_apply_overlay.py +++ b/tests/test_apply_overlay.py @@ -84,7 +84,13 @@ def test_file_list_is_derived_not_hand_maintained(): # file (they explain why the list is derived, citing vcpkg_git_archives.bzl as # the cautionary case), so strip comments before looking -- a test that cannot # tell prose from code makes documenting the reason impossible. - code = "\n".join(l.split("#", 1)[0] for l in text.splitlines()) + # Diagnostics are prose for the same reason comments are: a `note "the build + # will stop in qt_runtime.bzl"` tells the reader WHERE their misconfigured Qt + # will fail, and a test that cannot tell a message from a roster forces you to + # make the message vaguer to go green. What must not exist is a hand-kept LIST + # that the copy loop reads -- so strip comments and message lines both. + code = "\n".join(l.split("#", 1)[0] for l in text.splitlines() + if not re.match(r'\s*(note|echo|die)\b', l)) listed = re.findall(r'\b[\w.]+\.bzl\b', code) assert not listed, "found hand-listed overlay files in code: %r" % (listed,) @@ -642,3 +648,71 @@ def test_the_ignored_build_vcpkg_file_is_not_committed(): assert "Build/vcpkg/*) continue" in commit_block, \ ("the overlay commit sweeps in Build/vcpkg/BUILD.bazel, which Ladybird's " ".gitignore excludes and which must not exist before the vcpkg prefetch") + + +def test_the_qt_sdk_path_is_preserved_across_a_reapply(): + """MODULE.bazel's Qt path is the reader's, and a re-apply must not clobber it. + + Ulf: "We're using Qt (6.9.2) from a VENV, and system Qt is 6.4.2." The overlay + hardcodes `paths = {"linux-x86_64": "/usr/lib/qt6"}` and the copy phase + overwrites MODULE.bazel, so a re-apply silently repointed his build from a + working 6.9.2 at a system 6.4.2 -- which is BELOW Ladybird's 6.9 floor, i.e. + the re-apply turned a working tree into a failing one, and the failure surfaced + later and elsewhere. + + Every other line in the overlay is a fact about Ladybird at the pin, identical + on every host. This one names an SDK on YOUR machine, so it is resolved rather + than imposed: --qt-prefix, else the line already in your MODULE.bazel, else the + qmake first on your PATH (which is how a venv says which Qt it means), else the + historical default. Verified: two consecutive runs, the second with no flags, + leave /tmp/venvqt in place. + """ + t = _text() + assert "--qt-prefix" in t, "no way to name the Qt SDK" + # Rule 2 is the one that makes a re-apply safe: read the target's own value. + assert "qt_prefix_in_tree" in t, \ + "the script does not read the Qt path already configured in the target tree" + # Rule 3: a venv/aqt SDK puts its qmake on PATH; that is the right default for it. + assert "qmake" in t and "QT_INSTALL_PREFIX" in t, \ + "the script cannot discover a Qt from qmake, so a venv SDK must be typed by hand" + # It must be applied AFTER the copy, or the copy overwrites it again. + copy_idx = t.index("copying the overlay") + set_idx = t.index("set_qt_prefix_in \"$TARGET/MODULE.bazel\"") + assert set_idx > copy_idx, \ + "the Qt path is written before the overlay copy, which then overwrites it" + # And the reader must be TOLD which rule won -- a silent default is the bug. + assert "from $QT_SOURCE" in t or "QT_SOURCE" in t, \ + "the script does not report where the Qt prefix came from" + + +def test_verify_treats_the_qt_sdk_path_as_expected_to_differ(): + """--verify must not report the reader's own Qt path as a defect. + + Reporting `DIFFERS MODULE.bazel` told people to overwrite their correct + configuration with the overlay's -- and doing that is exactly how a venv Qt + 6.9.2 became a system Qt 6.4.2. So the comparison normalises that one line and, + when the rest matches, reports the configured prefix instead of a failure. + """ + t = _text() + verify = t.split("if [ \"$VERIFY\" -eq 1 ]", 1)[1].split("exit \"$rc\"", 1)[0] + assert "MODULE.bazel" in verify, "--verify has no special case for MODULE.bazel" + assert "@@QT@@" in verify, \ + "--verify does not normalise the Qt path line before comparing" + # It must still catch a REAL difference in that file. + assert "DIFFERS" in verify, "--verify no longer reports differing files at all" + + +def test_a_qt_below_the_floor_is_reported_at_apply_time(): + """A too-old Qt must be named when it is chosen, not deep in a build. + + The 6.9 floor is enforced in qt_runtime.bzl (where Bazel can fail the build), + but by then the reader is several minutes into `bazel build` and the message + names a repository rule. If the prefix is resolvable at apply time, its version + is cheap to read, so the script says so immediately -- verified against a fake + 6.4.2 SDK, which is the version Ulf's system Qt actually is. + """ + t = _text() + assert "qt_version_at" in t, "the script never reads the chosen Qt's version" + assert "6.9" in t, "the floor is not mentioned where the SDK is chosen" + assert re.search(r"6\.\[0-8\]\.\*", t), \ + "no check that the chosen Qt is below the 6.9 floor" diff --git a/tests/test_qt_runtime.py b/tests/test_qt_runtime.py index 8549a9f..66fc007 100644 --- a/tests/test_qt_runtime.py +++ b/tests/test_qt_runtime.py @@ -365,8 +365,13 @@ def test_every_qt_module_the_build_links_is_preflighted_with_its_package(): assert linked, "no @qt//:Qt* deps in BUILD.bazel -- did the label scheme change?" # What the preflight knows about. - block = rt.split("_QT_MODULES = {", 1)[1].split("}", 1)[0] - preflighted = dict(re.findall(r'"(Qt\w+)":\s*"([^"]+)"', block)) + block = rt.split("_QT_MODULES = {", 1)[1].split("\n}", 1)[0] + # (module -> (deb package, aqt module)): BOTH forms, because which one is + # correct advice depends on the SDK -- see the venv case below. + preflighted = dict(re.findall(r'"(Qt\w+)":\s*\("([^"]+)",\s*"([^"]+)"\)', + block.replace("\n", " ")) and + [(m, (d, a)) for m, d, a in + re.findall(r'"(Qt\w+)":\s*\("([^"]+)",\s*"([^"]+)"\)', block)]) assert preflighted, "_QT_MODULES is empty or unparseable" missing = sorted(linked - set(preflighted)) @@ -376,13 +381,16 @@ def test_every_qt_module_the_build_links_is_preflighted_with_its_package(): "generated BUILD file instead of the apt package: %s" % missing) # Every entry must name a package, or the message cannot tell anyone what to do. - for mod, pkg in sorted(preflighted.items()): - assert pkg and not pkg.startswith("Qt"), \ - "%s maps to %r, which is not a package name" % (mod, pkg) + for mod, (deb, aqt) in sorted(preflighted.items()): + assert deb and not deb.startswith("Qt"), \ + "%s maps to %r, which is not a package name" % (mod, deb) + assert aqt and not aqt.startswith("Qt"), \ + "%s has no aqt module name (%r); a venv/aqt SDK cannot be told what " \ + "to install" % (mod, aqt) # The regression that motivated this: Positioning became REQUIRED at 71fb301a. - assert preflighted.get("QtPositioning") == "qt6-positioning-dev", \ - "QtPositioning must map to the package that ships libQt6Positioning.so" + assert preflighted.get("QtPositioning") == ("qt6-positioning-dev", "qtpositioning"), \ + "QtPositioning must name both the deb and the aqt module" def test_the_module_preflight_reads_the_same_libs_dir_rules_qt_derives_from(): @@ -407,9 +415,18 @@ def test_the_module_preflight_reads_the_same_libs_dir_rules_qt_derives_from(): assert 'split(".")' in impl, \ "libQt6Positioning.so.6.10.2 must reduce to QtPositioning like _create_lib_name does" # The failure has to be actionable: package names and the cache caveat. - fail_msg = impl.split("missing {n} module", 1)[1].split('))', 1)[0] \ - if "missing {n} module" in impl else "" - assert "apt install" in fail_msg, "the failure does not say how to install the module" - assert "sync --configure" in fail_msg or "clean --expunge" in fail_msg, \ + # The install advice lives in _install_hint; the surrounding fail() carries the + # context. Both are part of what the reader sees. + hint = rt.split("def _install_hint", 1)[1].split("\ndef ", 1)[0] + assert "apt install" in hint, "no distro instruction for a distro Qt" + assert "aqt" in hint, "no self-contained-SDK instruction for a venv/aqt Qt" + assert "CANNOT fix it" in hint, \ + ("a self-contained SDK is not told that apt cannot help: apt installs into " + "/usr/lib, which that SDK never reads, so following the advice changes " + "nothing and the reader concludes the message was wrong") + assert "sync --configure" in impl or "clean --expunge" in impl, \ ("the failure does not mention that @qt is cached, so a reader who installs " "the package and re-runs can get the same error and conclude it did not work") + # The SDK must be NAMED, or a reader with two Qts cannot tell which one failed. + assert "QT_INSTALL_PREFIX" in impl and "libs" in impl, \ + "the failure does not name the prefix/lib dir it probed" From 4e97a4729edd3542c3734955f6df8ff69c338ef2 Mon Sep 17 00:00:00 2001 From: Ulf Adams Date: Wed, 19 Aug 2026 22:12:17 +0000 Subject: [PATCH 54/59] apply_overlay.sh: run the second prefetch, do not print it The script ran `Meta/ladybird.py vcpkg` itself but only PRINTED `Meta/fetch_vcpkg_git_archives.py` in its closing message. So it reported done, the obvious next command was `bazel build`, and that died inside the vcpkg action: Building 77 vcpkg ports (no network) failed vcpkg_build: no git-sourced externals at ./Meta/CMake/vcpkg/git-archives vcpkg_build: 4 are required (see VCPKG_GIT_ARCHIVES in vcpkg_git_archives.bzl) Ulf hit exactly this. The diagnostic is the good one I added earlier -- it fails in four seconds naming the directory and the file, rather than ~20 minutes in inside skia's portfile naming a googlesource URL -- but it is still a failure on a tree the script had just called done. The four tarballs are not optional and not derivable from anything else in the tree (vcpkg_from_git shells out to `git fetch`, which no asset source intercepts), so a tree without them cannot build. So phase 3 runs it, guarded on the vcpkg checkout the first prefetch creates (the fetcher resolves each clone URL out of the portfiles, which is the ordering constraint that made it a message in the first place -- and ordering is a thing a script can express). It is idempotent: on an already-fetched tree all four verify from cache in under a second. --verify checks them too, by NAME against the committed pin rather than by counting files (three of four present is the interesting broken case and `ls | wc -l` calls it fine). Ulf's tree passed --verify and then failed the build on this, which is precisely the thing --verify exists to prevent. Hashes are not re-verified there: that is minutes of sha512 over ~200MB and the fetcher already verified them at write time. One test had to be sharpened rather than satisfied. test_file_list_is_derived_not_hand_maintained bans a .bzl filename in code, because a hand-kept roster is how vcpkg_git_archives.bzl came to be generated, committed, documented and loaded by nothing. The new check READS that same file to learn which tarballs to expect -- the opposite of the bug -- so the ban is now on enumeration: a .bzl name that is not the input of a read. A test that forbade the fix would have demanded the hardcoded list. Two new tests: every required prefetch is run and not merely printed (with here-docs stripped, since prose was the bug), and the git-archive prefetch runs after the checkout it reads portfiles from. The first draft of the second one compared t.index() positions and failed on correct code -- it found the MENTION inside --verify's diagnostic, not the invocation. --- examples/ladybird/README.md | 14 ++++- examples/ladybird/apply_overlay.sh | 75 +++++++++++++++++++++-- tests/test_apply_overlay.py | 98 +++++++++++++++++++++++++++++- 3 files changed, 178 insertions(+), 9 deletions(-) diff --git a/examples/ladybird/README.md b/examples/ladybird/README.md index 64ca8a8..a153987 100644 --- a/examples/ladybird/README.md +++ b/examples/ladybird/README.md @@ -82,7 +82,17 @@ stayed invisible here for months. | the four `git archive` tarballs | **vcpkg itself**, while building skia and angle | | ~~the HSTS preload table~~ | **Bazel**, now: `@hsts_preload_json//file`, pinned in `hsts_preload.bzl` | -**Without ever running CMake — the actual answer, and it needs no CMake at all:** +**`apply_overlay.sh` runs both of these for you** — it did not always, and that +gap cost a 20-minute build: the script ran step 1 and *printed* step 2 in its closing +message, so the obvious next command was `bazel build`, which failed inside the vcpkg +action with `no git-sourced externals at ./Meta/CMake/vcpkg/git-archives`. A setup +script that stops one required step short of a working build has not set anything up, +and a closing message is not a substitute for doing the work. `--verify` now checks the +four tarballs by name against the committed pin too, since their absence is a +guaranteed build failure and finding that without a build is exactly what `--verify` +is for. + +**By hand, without ever running CMake — and it needs no CMake at all:** ```sh python3 Meta/ladybird.py vcpkg # 1. the checkout + .git (~70s). No CMake: @@ -409,7 +419,7 @@ manual version below has four ways to be silently wrong (see ```sh ./apply_overlay.sh ~/ladybird # branch + commits at the pinned commit, - # then the vcpkg prefetch, in the order that works + # then BOTH prefetches, in the order that works ./apply_overlay.sh --verify ~/ladybird # check an existing tree, change nothing ``` diff --git a/examples/ladybird/apply_overlay.sh b/examples/ladybird/apply_overlay.sh index edc8b37..e2fdaab 100755 --- a/examples/ladybird/apply_overlay.sh +++ b/examples/ladybird/apply_overlay.sh @@ -35,8 +35,15 @@ # ./apply_overlay.sh /path/to/ladybird # branch + commits (see below) # ./apply_overlay.sh --verify /path/to/existing # check a tree, change nothing # -# Then, per the README's "Reproducing": two prefetches (Meta/ladybird.py vcpkg, -# Meta/fetch_vcpkg_git_archives.py) and `bazel build`. +# Both prefetches run here (Meta/ladybird.py vcpkg, then +# Meta/fetch_vcpkg_git_archives.py); after this, `bazel build` is the next command. +# The second one used to be PRINTED rather than run, which cost Ulf a 20-minute +# build: everything the script does succeeded, so the obvious next step was +# `bazel build`, and it failed inside the vcpkg action with "no git-sourced +# externals at ./Meta/CMake/vcpkg/git-archives". A setup script that stops one +# required step short of a working build has not set anything up -- the closing +# message is not a substitute for doing it (finding 35 again: an instruction the +# reader must remember is a step the script decided not to take). # # --------------------------------------------------------------------------- # WHAT THIS LEAVES BEHIND: a BRANCH, with COMMITS. It used to leave a detached @@ -400,6 +407,40 @@ if [ "$VERIFY" -eq 1 ]; then echo "PATCH NOT APPLIED: $name" >&2; rc=1 done + # The four vcpkg_from_git tarballs. NOT overlay files (they are 200MB of git + # archive output, and they are .gitignored), so the file loop above cannot see + # them -- but their absence is a guaranteed build failure, which is exactly what + # --verify exists to find without a build. Ulf's tree passed --verify and then + # failed the build on this. + # + # Checked by NAME against the committed pin, not by count: a directory with + # three of the four in it is the interesting broken case, and `ls | wc -l` calls + # it fine. Hashes are not re-verified here (that is minutes of sha512 over + # ~200MB, and the fetcher already verified them at write time); --verify is the + # cheap check you run often. + archives_dir="Meta/CMake/vcpkg/git-archives" + want_archives="$(sed -n "s|^[[:space:]]*'\([^']*\.tar\.gz\)':.*|\1|p" \ + "$WORKSPACE/vcpkg_git_archives.bzl")" + if [ -z "$want_archives" ]; then + echo "MALFORMED: no archives parsed out of vcpkg_git_archives.bzl" >&2; rc=1 + else + missing_archives=0 + for a in $want_archives; do + [ -f "$archives_dir/$a" ] || { echo "MISSING $archives_dir/$a" >&2 + missing_archives=$((missing_archives + 1)); } + done + if [ "$missing_archives" -gt 0 ]; then + echo " $missing_archives of $(echo "$want_archives" | wc -w) vcpkg_from_git archives are absent." >&2 + echo " vcpkg_from_git bypasses the asset cache (it runs \`git fetch\`), so these" >&2 + echo " cannot be http_file'd; the build stages them from that directory and" >&2 + echo " fails without them. Fetch them (pure git, ~80s, verified against the pin):" >&2 + echo " cd $TARGET && python3 Meta/fetch_vcpkg_git_archives.py" >&2 + rc=1 + else + note "vcpkg_from_git archives: $(echo "$want_archives" | wc -w)/$(echo "$want_archives" | wc -w) present" + fi + fi + # Executable bits are tree state git carries and a `cp` does not always: the # vcpkg/cargo build scripts are run as actions, so a lost +x fails at action # time, deep in a build, with a confusing message. @@ -718,12 +759,38 @@ else WARN fi +# Phase 3: the SECOND prefetch -- the four vcpkg_from_git tarballs. +# +# This has to run here, not in the closing message, because there is no build +# without it. `vcpkg_from_git` shells out to `git fetch`, which no asset source +# intercepts, so these four cannot be http_file'd like the other 76 distfiles; but +# vcpkg DOES honour a pre-placed downloads/-.tar.gz, which is what they +# become. Meta/vcpkg_build.sh hard-fails in four seconds when they are absent (it +# used to fail ~20 minutes in, inside skia's portfile, naming a googlesource URL) -- +# and that fast, clear failure was still a failure Ulf hit on a tree this script +# had just reported as done. The instruction was in the closing message. He, quite +# reasonably, ran `bazel build`. +# +# It needs the vcpkg checkout (it resolves each clone URL out of the portfiles), so +# it must come after phase 2 -- that ordering is the reason it was a message in the +# first place, and ordering is a thing a script can express. +if [ -d "$TARGET/Build/vcpkg/.git" ] && [ "$PREFETCH" -eq 1 ]; then + note "fetching the 4 vcpkg_from_git tarballs (~80s, verified against the pin)" + python3 Meta/fetch_vcpkg_git_archives.py \ + || die "the git-archive prefetch failed. It is pure git (clone + git archive, + checked against the SHA512s in vcpkg_git_archives.bzl), so this is a network + or a pin problem, not a build problem. Re-run just this step with: + cd $TARGET && python3 Meta/fetch_vcpkg_git_archives.py" +elif [ "$PREFETCH" -eq 0 ]; then + note "SKIPPING the git-archive prefetch (--no-prefetch)" + note " \`bazel build\` will fail without it: python3 Meta/fetch_vcpkg_git_archives.py" +fi + cat < done. One prefetch remains (it needs no CMake): +==> done. Both prefetches have run; the next command is the build: cd $TARGET - python3 Meta/fetch_vcpkg_git_archives.py # the 4 vcpkg_from_git tarballs (~80s) bazel build //:ladybird //:WebContent //:RequestServer //:ImageDecoder \\ //:Compositor //:WebWorker diff --git a/tests/test_apply_overlay.py b/tests/test_apply_overlay.py index c762a42..3f81bd0 100644 --- a/tests/test_apply_overlay.py +++ b/tests/test_apply_overlay.py @@ -89,9 +89,17 @@ def test_file_list_is_derived_not_hand_maintained(): # will fail, and a test that cannot tell a message from a roster forces you to # make the message vaguer to go green. What must not exist is a hand-kept LIST # that the copy loop reads -- so strip comments and message lines both. - code = "\n".join(l.split("#", 1)[0] for l in text.splitlines() - if not re.match(r'\s*(note|echo|die)\b', l)) - listed = re.findall(r'\b[\w.]+\.bzl\b', code) + code = [l.split("#", 1)[0] for l in text.splitlines() + if not re.match(r'\s*(note|echo|die)\b', l)] + # A .bzl named as the INPUT OF A READ is the opposite of the bug: it is the + # script deriving a list from a generated file instead of restating it. The + # git-archive check parses vcpkg_git_archives.bzl -- the very file whose story + # this test is named after -- to learn which four tarballs must exist, and a + # test that forbade that would forbid the fix and demand the hardcoded list. + # So the ban is on ENUMERATION: a .bzl name that is not being read. + readers = re.compile(r'\b(sed|grep|awk|cat|read|source|\.)\b') + listed = [l.strip() for l in code + if re.search(r'\b[\w.]+\.bzl\b', l) and not readers.search(l)] assert not listed, "found hand-listed overlay files in code: %r" % (listed,) @@ -716,3 +724,87 @@ def test_a_qt_below_the_floor_is_reported_at_apply_time(): assert "6.9" in t, "the floor is not mentioned where the SDK is chosen" assert re.search(r"6\.\[0-8\]\.\*", t), \ "no check that the chosen Qt is below the 6.9 floor" + + +def test_every_required_prefetch_is_RUN_not_merely_printed(): + """A step the build cannot do without must be executed, not documented. + + The concrete failure: the script ran `Meta/ladybird.py vcpkg` itself but only + PRINTED `Meta/fetch_vcpkg_git_archives.py` in its closing message. So the + script reported success, the obvious next command was `bazel build`, and that + died inside the vcpkg action with "no git-sourced externals at + ./Meta/CMake/vcpkg/git-archives". Ulf hit exactly this. The four tarballs are + not optional and not derivable from anything else in the tree -- vcpkg_from_git + shells out to `git fetch`, which no asset source intercepts -- so a tree without + them cannot build, and a setup script that leaves the tree unable to build has + not finished. + + Asserted structurally: each prefetch appears on a line that RUNS it (a `python3` + invocation outside a heredoc/message), not only inside the closing `cat < Date: Thu, 20 Aug 2026 11:13:08 +0000 Subject: [PATCH 55/59] A one-paste diagnosis for the Qt runtime crash, instead of LD_LIBRARY_PATH Ulf's build crashes immediately and he remembers LD_LIBRARY_PATH fixing it. That is the problem: TWO unrelated failures present as "immediate crash" and LD_LIBRARY_PATH masks BOTH, so reaching for it destroys the one datum that says which one he has. A. the LIBRARIES: an aqt/official SDK bundles its own ICU; libQt6Core finds it through `RUNPATH $ORIGIN`, and $ORIGIN is the dir the loader opened the object by -- Bazel's solib dir, which has no ICU. Death before main(). @qt_plugins//:runtime_libs is the fix (SDK private libs as real link inputs, so OUR runpath resolves them). B. the PLUGINS: Qt dlopens the QPA plugin from the compiled-in prefix or the executable's directory; the distro's libqxcb.so in an SDK libQt6Core is SIGSEGV in QXcbConnection::initializeScreens. //:qt_conf + //:qt_plugins is the fix. Both fixes are in the tree, so the useful question is not "which fix does he need" but "which fix failed to FIRE on his Qt" -- and I have to admit the likely answer is mine, not his: the private-library derivation has NEVER run for real on this machine. My Qt is a distro Qt (/usr/lib/x86_64-linux-gnu), which by design makes runtime_libs an empty target -- confirmed, it is literally cc_library(name = "runtime_libs") with no deps here. His aqt SDK is the first genuine self-contained Qt this code has met; the only other exercise was a synthetic fake SDK in /tmp. That is exactly the finding-35 shape the header warns about: an empty result is CORRECT for a distro Qt and WRONG for an SDK, and the two are indistinguishable without asking. So the script asks, in seven sections: which Qt MODULE.bazel names, which Qt @qt actually discovered (its own generated qtconf.bzl, not a guess), whether rocntime_libs has any cc_imports and what the SDK's lib dir ships as the input to that derivation, whether qt.conf and the 79 plugins are staged and WHICH SDK the resolved libqxcb.so belongs to, what the binary resolves at load time, and finally the run itself with LD_LIBRARY_PATH explicitly UNSET (env -u) -- because with it set the whole exercise says nothing. Then it says how to read the output. Ran it against my own tree: passes end to end, correctly reports the distro-Qt case as the legitimate empty one, and --version exits 0. It also caught a bug in itself on the first run -- `grep -c` prints one count per file, so a glob gives "0\n0" and `[ -eq 0 ]` dies with "integer expected". Fixed with `grep -h | wc -l`. --- examples/ladybird/qt_runtime_diagnose.sh | 135 +++++++++++++++++++++++ 1 file changed, 135 insertions(+) create mode 100755 examples/ladybird/qt_runtime_diagnose.sh diff --git a/examples/ladybird/qt_runtime_diagnose.sh b/examples/ladybird/qt_runtime_diagnose.sh new file mode 100755 index 0000000..2caa852 --- /dev/null +++ b/examples/ladybird/qt_runtime_diagnose.sh @@ -0,0 +1,135 @@ +#!/usr/bin/env bash +# Diagnose a Qt runtime crash in the Bazel-built Ladybird, in one paste. +# +# WHY THIS EXISTS. Two unrelated failures present as "immediate crash", and +# LD_LIBRARY_PATH makes both go away -- which is exactly why reaching for it loses +# the information that says which one you had: +# +# A. THE LIBRARIES. An official/aqt Qt bundles its own ICU (aqt 6.9.2's +# libQt6Core needs libicui18n.so.73, which exists in the SDK's lib/ and +# nowhere else on a machine whose distro ICU is 76+). Bazel links @qt's +# libQt6Core out of a solib dir; libQt6Core resolves ICU through +# `RUNPATH $ORIGIN`, and $ORIGIN is the directory the loader OPENED it by -- +# the solib dir, which has no ICU. Death before main(). +# Fixed by @qt_plugins//:runtime_libs: the SDK's private libs become real link +# inputs, so BAZEL stages them and OUR runpath (the one glibc consults for our +# direct deps) finds them. +# +# B. THE PLUGINS. Qt dlopens the QPA plugin at QApplication construction from a +# prefix baked into libQt6Core, or from the executable's directory. Load the +# DISTRO's libqxcb.so into an SDK libQt6Core and you get SIGSEGV in +# QXcbConnection::initializeScreens (or, if the plugin is older, a clean +# "no Qt platform plugin could be initialized" abort). +# Fixed by //:qt_conf + //:qt_plugins staged beside the binary. +# +# Both fixes are in the tree. This script checks whether each one actually FIRED +# for the Qt you have, because a fix that silently degrades to "no private +# libraries" is indistinguishable from a Qt that needs none (finding 35). +# +# Usage: ./qt_runtime_diagnose.sh [/path/to/ladybird-tree] +# Reads only; runs nothing that can change the build. + +set -uo pipefail +TREE="${1:-$PWD}" +cd "$TREE" || { echo "no such tree: $TREE" >&2; exit 1; } + +say() { printf '\n=== %s\n' "$*"; } +BIN="bazel-bin/ladybird" + +say "0. the tree" +echo "tree: $TREE" +echo "commit: $(git rev-parse --short HEAD 2>/dev/null || echo '?')" + +say "1. which Qt the BUILD was told to use (MODULE.bazel)" +sed -n 's|^[[:space:]]*paths = {"linux-x86_64": "\([^"]*\)".*|\1|p' MODULE.bazel | head -1 + +say "2. which Qt @qt actually DISCOVERED (its own generated qtconf.bzl)" +# The output base, asked of bazel rather than guessed. +OB="$(bazel info output_base 2>/dev/null)" +QTCONF="$(find "$OB/external" -maxdepth 3 -name qtconf.bzl 2>/dev/null | head -1)" +if [ -n "$QTCONF" ]; then + grep -E '^(QT_VERSION|QT_INSTALL_PREFIX|QT_INSTALL_LIBS|QT_INSTALL_PLUGINS)' "$QTCONF" + LIBS="$(sed -n 's|^QT_INSTALL_LIBS="\(.*\)"|\1|p' "$QTCONF")" +else + echo "NOT FOUND -- @qt has not been fetched yet (build first)" + LIBS="" +fi + +say "3. did the PRIVATE-LIBRARY staging fire? (failure A)" +PLUG="$(find "$OB/external" -maxdepth 2 -name '*qt_plugins' -type d 2>/dev/null | head -1)" +if [ -z "$PLUG" ]; then + echo "@qt_plugins not fetched yet" +elif grep -q 'name = "runtime_libs"' "$PLUG/BUILD.bazel" 2>/dev/null; then + # grep -c prints one count PER FILE; with a glob that is "0\n0", which then + # fails `[ ... -eq 0 ]` with "integer expected". Count lines instead. + n="$(grep -h '^cc_import' "$PLUG/BUILD.bazel" 2>/dev/null | wc -l)" + echo "runtime_libs exists with $n cc_import(s):" + grep -A2 '^cc_import' "$PLUG/BUILD.bazel" 2>/dev/null | sed 's/^/ /' + if [ "$n" -eq 0 ]; then + cat <<'NOTE' + -> EMPTY. That is CORRECT for a distro Qt (its ICU is a distro package, + already on the loader's default search path) and WRONG for a + self-contained SDK. Check section 4: if the SDK's lib dir has libicu* + in it and this is empty, the derivation missed them -- that is the bug, + not your machine. +NOTE + fi +else + echo "no runtime_libs target at all (an OLD @qt_plugins? bazel sync --configure)" +fi + +say "4. what the SDK's lib dir actually ships (the input to that derivation)" +if [ -n "$LIBS" ] && [ -d "$LIBS" ]; then + echo "$LIBS:" + ls "$LIBS" | grep -vE '^libQt' | grep '\.so' | sed 's/^/ /' | head -20 + echo " (non-Qt .so files above; those DT_NEEDED by a libQt6*.so are what must be staged)" + echo " ICU the Qt libs ask for:" + objdump -p "$LIBS"/libQt6Core.so.6 2>/dev/null \ + | awk '/NEEDED/ && /icu/ {print " " $2}' | sort -u +else + echo "lib dir unknown or missing: '${LIBS:-}'" +fi + +say "5. did the PLUGIN staging fire? (failure B)" +if [ -f bazel-bin/qt.conf ]; then + sed 's/^/ /' bazel-bin/qt.conf + echo " plugins staged: $(find bazel-bin/plugins -name '*.so' 2>/dev/null | wc -l) .so" + echo " platform plugin -> $(readlink -f bazel-bin/plugins/platforms/libqxcb.so 2>/dev/null || echo MISSING)" + echo " (that path must be under the SAME SDK as section 2's QT_INSTALL_LIBS)" +else + echo "bazel-bin/qt.conf MISSING -- Qt will scan the compiled-in prefix instead" +fi + +say "6. what the BINARY resolves at load time (the actual answer)" +if [ -x "$BIN" ]; then + if ldd "$BIN" 2>&1 | grep -q "not found"; then + echo "UNRESOLVED libraries -- this is failure A:" + ldd "$BIN" 2>&1 | grep "not found" | sed 's/^/ /' + else + echo "all libraries resolve. Where the interesting ones come from:" + ldd "$BIN" 2>/dev/null | grep -iE 'icu|libQt6(Core|Gui|Widgets)' | sed 's/^/ /' + fi +else + echo "$BIN not built" +fi + +say "7. run it, with NO LD_LIBRARY_PATH, and keep the first failure" +echo "(unset deliberately: with it set, both failures disappear and this says nothing)" +env -u LD_LIBRARY_PATH "$BIN" --version 2>&1 | head -5 +rc=$? +echo "--version exit: $rc" + +cat <<'EOF' + +=== READ IT LIKE THIS + section 6 shows "not found", or section 7 says "error while loading shared + libraries" -> failure A: the private-library staging. Section 3 is + empty while section 4 lists libicu*. Send me 3+4. + section 7 crashes with a BACKTRACE through QXcbConnection / QGuiApplication, + or says "no Qt platform plugin could be initialized" + -> failure B: the plugins. Compare section 5's resolved + libqxcb.so path against section 2's QT_INSTALL_LIBS; + if they are different SDKs, that is the bug. + neither, and it still crashes -> not the Qt runtime at all; get the backtrace: + gdb -q -batch -ex run -ex bt --args bazel-bin/ladybird +EOF From f64eea45c73c60c17f5c04193a9ab39e0dbd32e4 Mon Sep 17 00:00:00 2001 From: Ulf Adams Date: Thu, 20 Aug 2026 11:17:21 +0000 Subject: [PATCH 56/59] Rule out the stale libexec BEFORE asking anything about Qt Ulf's crash is not Qt, and I sent him a Qt diagnostic. The evidence was in the first four lines of his paste and in the backtrace's PATHS, not its frames: ladybird -> bazel-out/k8-fastbuild/bin/ladybird (fresh) Compositor -> bazel-out/k8-fastbuild/libexec/Compositor (stale) LibWebView/Utilities.cpp's get_paths_for_helper_process() searches /libexec/ BEFORE /bin/, so any libexec copy wins over what Bazel just built and only the copy ever runs. After a repin those copies are from the OLD pin, whose IPC message ids have shifted, so every message fails with `Endpoint magic number mismatch, not my message!` and the cascade ends in SIGILL with a Qt-flavoured backtrace (QApplicationPrivate, QEventDispatcherGlib, libQt6Core frames). It looks exactly like a Qt runtime bug. It is not one -- and his LD_LIBRARY_PATH=~/Qt/6.9.2/gcc_64/lib was a red herring pointing the same wrong way. This is the SECOND time this failure has cost real time (the README already records the first, where I spent hours on IPC codegen parity and an ABI-skew theory before checking what was executing), and todo 4a93a257 is the lesson written down: for a "built fine, behaves wrong" bug, ask what is EXECUTING before auditing what produced it. A lesson in a todo did not stop me repeating it, so it now runs as code. Section 0b of the diagnostic checks it FIRST, before any Qt question, and prints the shadowing directory, its contents, its newest mtime and bazel-bin's for comparison, then the exact rm -rf. Section 8 answers the question the backtrace answered and no build check does: for each of the five services, which copy is first on the lookup chain. Both verified against my own tree -- negative case clean, then positive case reproduced by planting a single August-dated Compositor, which also reproduces the detail that only SOME services are shadowed (why his ladybird ran fresh while Compositor died). Three tests: the libexec check precedes the Qt checks by string position (a diagnostic that asks about Qt first sends the reader into the wrong subsystem, which is what I did), it states the FIRST/second lookup order so "delete libexec" is not a superstition, and the run step keeps `env -u LD_LIBRARY_PATH` -- with it set, both Qt failures disappear and the run proves nothing. Adapted to test_run_recipe.py's existing os.path style after the first draft used a pathlib REPO the file does not define. --- examples/ladybird/qt_runtime_diagnose.sh | 66 ++++++++++++++++++++++++ tests/test_run_recipe.py | 54 +++++++++++++++++++ 2 files changed, 120 insertions(+) diff --git a/examples/ladybird/qt_runtime_diagnose.sh b/examples/ladybird/qt_runtime_diagnose.sh index 2caa852..bdb6e66 100755 --- a/examples/ladybird/qt_runtime_diagnose.sh +++ b/examples/ladybird/qt_runtime_diagnose.sh @@ -40,6 +40,55 @@ say "0. the tree" echo "tree: $TREE" echo "commit: $(git rev-parse --short HEAD 2>/dev/null || echo '?')" +say "0b. is a STALE libexec/ shadowing the services? (the most likely crash)" +# Checked FIRST, before anything about Qt, because this failure looks like a Qt +# crash, arrives as a SIGILL/VERIFICATION FAILED with a Qt-flavoured backtrace, +# and has nothing to do with Qt. It is also the failure that has now bitten twice. +# +# LibWebView/Utilities.cpp's get_paths_for_helper_process() searches +# /libexec/ FIRST +# /bin/ second +# so ANY libexec copy wins over the binaries Bazel just built, and only the copy +# is ever executed. After a repin the stale copies are from the OLD pin, whose IPC +# message IDs have shifted, so every message fails to parse: +# +# Failed to parse IPC message: +# Peer endpoint error: Endpoint magic number mismatch, not my message! +# IPC::ConnectionBase: Disconnecting misbehaving peer due to malformed message +# VERIFICATION FAILED: connection at Services/Compositor/ConnectionFromClient.cpp +# +# The tell is in the backtrace's PATHS, not in its frames: `ladybird` runs from +# bazel-out/.../bin/ while `Compositor` runs from bazel-out/.../libexec/. Todo +# 4a93a257 is exactly this lesson -- for a "built fine, behaves wrong" bug, ask +# what is EXECUTING before auditing what produced it. +stale=0 +for d in bazel-out/*/libexec bazel-bin/libexec; do + [ -d "$d" ] || continue + stale=$((stale + 1)) + echo " SHADOWING: $d" + ls "$d" 2>/dev/null | sed 's/^/ /' + echo " newest file here: $(find "$d" -type f -printf '%TY-%Tm-%Td %p\n' 2>/dev/null | sort | tail -1)" +done +if [ "$stale" -gt 0 ]; then + echo " compare against the FRESH build:" + for b in bazel-bin/Compositor bazel-bin/WebContent; do + [ -e "$b" ] && echo " $(find "$b" -printf '%TY-%Tm-%Td %p\n' 2>/dev/null)" + done + cat <<'FIX' + -> THIS IS ALMOST CERTAINLY YOUR BUG. libexec is searched BEFORE bin, so those + copies are what run, not what you just built. If their dates are older than + bazel-bin's, the UI is talking to services from a previous pin and every IPC + message fails with "Endpoint magic number mismatch". Delete them: +FIX + for d in bazel-out/*/libexec bazel-bin/libexec; do + [ -d "$d" ] && echo " rm -rf $TREE/$d" + done + echo " Nothing needs staging there: the services are already siblings of" + echo " ladybird in bazel-bin, which is the second entry in the lookup chain." +else + echo " none -- good (the services resolve to bazel-bin, the fresh build)" +fi + say "1. which Qt the BUILD was told to use (MODULE.bazel)" sed -n 's|^[[:space:]]*paths = {"linux-x86_64": "\([^"]*\)".*|\1|p' MODULE.bazel | head -1 @@ -119,9 +168,26 @@ env -u LD_LIBRARY_PATH "$BIN" --version 2>&1 | head -5 rc=$? echo "--version exit: $rc" +say "8. what the helper processes would actually RESOLVE to" +# The question the backtrace answers and no build check does: for each service, +# which copy is first on Ladybird's lookup chain. Printed even when 0b found +# nothing, because "the fresh one" is the answer that makes the negative useful. +for svc in Compositor WebContent RequestServer ImageDecoder WebWorker; do + found="" + for d in bazel-out/*/libexec bazel-bin/libexec; do + [ -x "$d/$svc" ] && { found="$d/$svc <-- SHADOWS bazel-bin"; break; } + done + [ -n "$found" ] || { [ -x "bazel-bin/$svc" ] && found="bazel-bin/$svc"; } + printf ' %-14s %s\n' "$svc" "${found:-NOT FOUND}" +done + cat <<'EOF' === READ IT LIKE THIS + section 0b found a libexec/, or section 8 says SHADOWS + -> NOT a Qt problem at all, whatever the backtrace looks + like. Stale services from an older build are what run; + the IPC ids have shifted. `rm -rf` them and re-run. section 6 shows "not found", or section 7 says "error while loading shared libraries" -> failure A: the private-library staging. Section 3 is empty while section 4 lists libicu*. Send me 3+4. diff --git a/tests/test_run_recipe.py b/tests/test_run_recipe.py index 7ff92ed..bed5aba 100644 --- a/tests/test_run_recipe.py +++ b/tests/test_run_recipe.py @@ -122,3 +122,57 @@ def test_the_resource_root_recipe_survives_a_stale_share_symlink(): sh = _run_recipe_shell() assert re.search(r"rm -f\s+\"?\$\(dirname\s+\"?\$BIN\"?\)\"?/share", sh), \ "the recipe does not clear a `share` symlink left pointing into CMake's tree" + + +def test_the_diagnostic_checks_for_a_shadowing_libexec_before_it_checks_qt(): + """The stale-libexec failure must be ruled out FIRST, because it mimics Qt. + + It has now bitten twice, and both times it arrived disguised: a SIGILL or + `VERIFICATION FAILED` with a Qt-flavoured backtrace (QApplicationPrivate, + QEventDispatcherGlib, libQt6Core frames), preceded by `Endpoint magic number + mismatch, not my message!` on every IPC message. Nothing in that picture says + "you are running binaries from a previous build" -- but the PATHS in the + backtrace do: `ladybird` from bazel-out/.../bin/ and `Compositor` from + bazel-out/.../libexec/. + + So the Qt diagnostic checks it before any Qt question, or it confidently + investigates the wrong subsystem. (Todo 4a93a257: for a "built fine, behaves + wrong" bug, ask what is EXECUTING before auditing what produced it.) + """ + script = os.path.join(_EXAMPLE, "qt_runtime_diagnose.sh") + assert os.path.isfile(script), "the Qt runtime diagnostic is missing" + with open(script) as f: + t = f.read() + assert "libexec" in t, \ + "the diagnostic never checks for a shadowing libexec, the likelier cause" + # Before the Qt sections: a diagnostic that asks about Qt first sends the + # reader into the wrong subsystem, which is exactly what happened. + libexec_at = t.index("libexec") + qt_at = min(t.index("MODULE.bazel"), t.index("qtconf.bzl")) + assert libexec_at < qt_at, \ + ("the libexec check must come BEFORE the Qt checks: it mimics a Qt crash " + "and is the more common cause") + # It must name the lookup order, or "delete libexec" is a superstition. + assert re.search(r"libexec.{0,120}\bFIRST\b", t, re.S | re.I), \ + "the diagnostic does not say libexec is searched FIRST (why the copy wins)" + # And it must print what each service RESOLVES to, which is the datum the + # backtrace carried and no build check produces. + assert "SHADOWS" in t, \ + "the diagnostic does not report which copy of each service would run" + for svc in ("Compositor", "WebContent", "RequestServer"): + assert svc in t, f"the resolution check does not cover {svc}" + + +def test_the_diagnostic_unsets_ld_library_path_when_it_runs_the_binary(): + """A run with LD_LIBRARY_PATH set cannot tell you whether you need it. + + Ulf's run.sh sets LD_LIBRARY_PATH=~/Qt/6.9.2/gcc_64/lib, which masks BOTH Qt + runtime failures (the missing bundled ICU, and the wrong-SDK QPA plugin). The + diagnostic exists to say which one fired, so it must run the binary WITHOUT it. + """ + script = os.path.join(_EXAMPLE, "qt_runtime_diagnose.sh") + with open(script) as f: + t = f.read() + assert "env -u LD_LIBRARY_PATH" in t, \ + ("the diagnostic must run the binary with LD_LIBRARY_PATH unset -- with it " + "set, both failures disappear and the run proves nothing") From 942a36d0d74e9bf71ad7fb29015ebdaa01922d87 Mon Sep 17 00:00:00 2001 From: Ulf Adams Date: Thu, 20 Aug 2026 11:39:17 +0000 Subject: [PATCH 57/59] Declare the services //:ladybird spawns, so one target cannot go stale Ulf's diagnosis, and it is the right one. `ls -la` was the whole datum: -r-xr-xr-x 107107448 Aug 20 00:38 bazel-bin/ladybird -r-xr-xr-x 99264432 Aug 11 20:36 bazel-bin/WebContent The services are found by PATH at runtime (get_paths_for_helper_process), not linked, so NOTHING in the build graph said the browser needs them. `bazel build //:ladybird` therefore built the browser and left whatever WebContent happened to be in bazel-bin from a previous build -- here one from the PREVIOUS PIN. Upstream inserted ~3 IPC messages between f9e34731 and 71fb301a, shifting every id after them, so every message failed to decode: Local endpoint error: Can't read past the end of the stream memory Peer endpoint error: Endpoint magic number mismatch, not my message! That reads like a codegen or ABI bug, which is why I got it wrong twice before decoding the bytes. The magic number 0xffa5367a is AK::string_hash("WebContentServer") -- the CORRECT endpoint, so the mismatch line is the peer's rejection, not the diagnosis. The message NUMBERING is what disagreed, and it is decisive: his ids matched f9e34731 7/7 (SetSystemFontFamily 147, SetWindowSize 126, InspectDomTree 41, SetPreferredLanguages 116, SetBrowsingBehavior 117, ResetZoom 128, SetHasFocus 121) and 71fb301a 0/7. I confirmed by running the current generator over the old pin's .ipc. So: the five services become `data` of //:ladybird. `data`, not `deps` -- separate processes, not link inputs -- which is the relationship this emitter already gets right for cranelift-compiler on LibWasm, and which also puts them in the runfiles tree for `bazel run`. The list is DERIVED from the launch_server_process<> string literals in HelperProcess.cpp, not hand-written: upstream ADDS services (Compositor is new since the previous pin), and a hand-kept roster is one new service away from silently reintroducing exactly this bug. An empty parse is a hard failure, since returning [] would look like success and rebuild nothing. Verified on my tree, not asserted: all five now appear in `bazel cquery deps(//:ladybird)`; touching Services/WebContent/main.cpp and building ONLY //:ladybird rebuilds WebContent (2 sandbox actions, compile + link) where before it was untouched; and --headless=text renders with no IPC decode errors at all. The recipe's six-target `bazel build` line was a WORKAROUND for this missing edge, and the reason the bug was survivable for so long -- so README and apply_overlay.sh now say `bazel build //:ladybird`, which is true. Two tests: //:ladybird declares each service in data and NOT in deps, and the list is derived from Ladybird's own source with no roster in the function. --- examples/ladybird/README.md | 23 +++++- examples/ladybird/apply_overlay.sh | 3 +- examples/ladybird/workspace/BUILD.bazel | 2 +- .../workspace/Meta/emit_build_bazel.py | 55 +++++++++++++- tests/test_run_recipe.py | 72 +++++++++++++++++++ 5 files changed, 148 insertions(+), 7 deletions(-) diff --git a/examples/ladybird/README.md b/examples/ladybird/README.md index a153987..cb002ed 100644 --- a/examples/ladybird/README.md +++ b/examples/ladybird/README.md @@ -102,8 +102,7 @@ python3 Meta/fetch_vcpkg_git_archives.py # 2. the four git archives (~80s), # and VERIFIED against the committed SHA512 # (the HSTS table needs no step: Bazel # fetches it from a pinned commit) -bazel build //:ladybird //:WebContent //:RequestServer //:ImageDecoder \ - //:Compositor //:WebWorker +bazel build //:ladybird # the 5 services come with it (they are its `data`) ``` Step 2 is `Meta/fetch_vcpkg_git_archives.py`, added here. Verified: **4/4 reproduced @@ -525,7 +524,7 @@ python3 Meta/fetch_vcpkg_git_archives.py # (b) 4/4 reproduced with git archive # //:vcpkg_installed alone if you want to time it. # No CMake build is needed, and none is consulted -- see the removal test # above. That was not true before finding 35. -bazel build //:ladybird //:WebContent //:RequestServer //:ImageDecoder //:Compositor //:WebWorker +bazel build //:ladybird # //:WebContent etc. are data of //:ladybird ``` To **regenerate or verify the BUILD files** you additionally need the reference @@ -585,6 +584,24 @@ resource root is `/../share/Lagom`, not `/share/Lagom` -- directory and appends `share/Lagom`. Ask the build for the paths rather than spelling any of them out: +**`//:ladybird` declares the five services as `data`, so one target builds them all.** +This recipe used to name all six targets, and that was a workaround for a missing +dependency edge: the services are found by PATH at runtime +(`get_paths_for_helper_process`), not linked, so nothing in the graph said the browser +needs them and `bazel build //:ladybird` alone left whatever `WebContent` was already +in `bazel-bin`. Ulf hit the consequence: `ladybird` dated Aug 20 beside a `WebContent` +dated Aug 11 — this pin's browser talking to the previous pin's service. Upstream had +inserted ~3 IPC messages between the pins, shifting every id after them, so every +message failed to decode with `Can't read past the end of the stream memory` / +`Endpoint magic number mismatch, not my message!`, which reads like a codegen or ABI +bug and is nothing of the kind. The magic number in those frames is +`AK::string_hash("WebContentServer")` — the *correct* endpoint; the message +**numbering** is what disagreed (7/7 against the old pin, 0/7 against the new one). +`data` rather than `deps` because they are spawned processes, not link inputs — the +relationship `LibWasm` already has to `cranelift-compiler` — and the list is derived +from the `launch_server_process<>` call sites in `HelperProcess.cpp`, since upstream +adds services (`Compositor` is new since the previous pin). + **Do NOT stage the services into a `libexec/` directory.** They are already siblings of `ladybird` in `bazel-bin`, and `LibWebView/Utilities.cpp`'s `get_paths_for_helper_process()` searches `/libexec/` **before** diff --git a/examples/ladybird/apply_overlay.sh b/examples/ladybird/apply_overlay.sh index e2fdaab..bae5d9a 100755 --- a/examples/ladybird/apply_overlay.sh +++ b/examples/ladybird/apply_overlay.sh @@ -791,8 +791,7 @@ cat < done. Both prefetches have run; the next command is the build: cd $TARGET - bazel build //:ladybird //:WebContent //:RequestServer //:ImageDecoder \\ - //:Compositor //:WebWorker + bazel build //:ladybird Re-check this tree at any time with: $(basename "$0") --verify $TARGET diff --git a/examples/ladybird/workspace/BUILD.bazel b/examples/ladybird/workspace/BUILD.bazel index dfddc5a..ff1c103 100644 --- a/examples/ladybird/workspace/BUILD.bazel +++ b/examples/ladybird/workspace/BUILD.bazel @@ -1857,7 +1857,7 @@ cc_binary( local_defines = ['LADYBIRD_QT_HAVE_POSITIONING=1', 'QT_CORE_LIB', 'QT_GUI_LIB', 'QT_NO_DEBUG', 'QT_POSITIONING_LIB', 'QT_WIDGETS_LIB'], copts = ['-pthread'], linkopts = ['-lGLX', '-lOpenGL', '-lgio-2.0', '-lglib-2.0', '-lgobject-2.0', '-lvulkan', '-lxkbcommon'], - data = [':qt_conf', ':qt_plugins'], + data = [':Compositor', ':ImageDecoder', ':RequestServer', ':WebContent', ':WebWorker', ':qt_conf', ':qt_plugins'], deps = [ '//:AK', '//:LibCore', diff --git a/examples/ladybird/workspace/Meta/emit_build_bazel.py b/examples/ladybird/workspace/Meta/emit_build_bazel.py index 825e20c..9e1ae75 100644 --- a/examples/ladybird/workspace/Meta/emit_build_bazel.py +++ b/examples/ladybird/workspace/Meta/emit_build_bazel.py @@ -540,6 +540,55 @@ def _emit_srcs(srcs): # Emitted after the Qt autogen rules, since it consumes them. QT_TARGETS = ["ladybird"] + +def spawned_services(): + """The helper binaries the UI SPAWNS at runtime, read out of Ladybird's source. + + WHY THIS IS A DEPENDENCY AT ALL, and the bug it fixes. + `bazel build //:ladybird` built the browser and NOTHING ELSE, because nothing in + the graph said the browser needs its services -- they are found by PATH at + runtime (get_paths_for_helper_process), not linked. So the recipe told people to + name all six targets, and a build of just //:ladybird left whatever WebContent + happened to be in bazel-bin from a previous build. + + Ulf hit the consequence in its most confusing form: ladybird dated Aug 20 next + to a WebContent dated Aug 11, i.e. a browser from THIS pin talking to a service + from the PREVIOUS one. Upstream had inserted ~3 IPC messages between the pins, + shifting every id after them, so every message failed to decode: + + Failed to parse IPC message: + Local endpoint error: Can't read past the end of the stream memory + Peer endpoint error: Endpoint magic number mismatch, not my message! + + which reads like a codegen or ABI bug and is nothing of the kind. (The magic + number in those frames, 0xffa5367a, is AK::string_hash("WebContentServer") -- + the CORRECT endpoint. The ids are what disagreed: his WebContent's numbering + matched f9e34731 7/7 and 71fb301a 0/7.) His diagnosis: declare them, so they + rebuild with the thing that spawns them. Exactly right. + + `data`, not `deps`: they are separate processes, not link inputs -- the same + relationship LibWasm already has to cranelift-compiler, which this emitter + already gets right (see rewrite_binary_path_define). data also puts them in the + runfiles tree, which is what `bazel run` needs. + + DERIVED from HelperProcess.cpp, never hand-listed. The names are the string + literals passed to launch_server_process<>, i.e. the same source of truth the + runtime lookup uses; a hand-kept list here would be a sixth service away from + silently reintroducing the bug (and upstream adds services -- Compositor is new + since the previous pin). If the parse finds nothing, that is a hard failure + rather than a build that quietly omits them again. + """ + src = os.path.join(ROOT, "Libraries", "LibWebView", "HelperProcess.cpp") + with open(src) as f: + text = f.read() + names = sorted(set(re.findall(r'launch_server_process<[^>]*>\(\s*"(\w+)"sv', text))) + if not names: + sys.exit("emit_build_bazel: found no launch_server_process<> calls in %s -- " + "the spawned-service list is DERIVED from them, and an empty list " + "would silently rebuild //:ladybird without its services (the Aug 11 " + "WebContent bug). Has the launch helper been rewritten?" % src) + return names + # Ring 2's own targets: the vcpkg tree build action and its inputs. Emitted # rather than hand-appended, because a hand-appended tail is a file the emitter # would silently truncate on the next run -- exactly the drift this project keeps @@ -1003,7 +1052,11 @@ def main(): extra_srcs=[":qt_moc", ":qt_rcc"], # The Qt plugins + qt.conf: runtime inputs of the GUI, in the # same sense as the cranelift-compiler binary LibWasm spawns. - extra_data=[":qt_conf", ":qt_plugins"]) + # Plus the five helper processes the UI spawns by path: see + # spawned_services() for why `bazel build //:ladybird` used to + # leave a stale WebContent behind, and what that looks like. + extra_data=[":qt_conf", ":qt_plugins"] + + [":" + s for s in spawned_services()]) print(VCPKG_TAIL, end="") # Last, so it is the final thing on stderr rather than buried in the middle. report_host_includes() diff --git a/tests/test_run_recipe.py b/tests/test_run_recipe.py index bed5aba..8e759d1 100644 --- a/tests/test_run_recipe.py +++ b/tests/test_run_recipe.py @@ -176,3 +176,75 @@ def test_the_diagnostic_unsets_ld_library_path_when_it_runs_the_binary(): assert "env -u LD_LIBRARY_PATH" in t, \ ("the diagnostic must run the binary with LD_LIBRARY_PATH unset -- with it " "set, both failures disappear and the run proves nothing") + + +def test_ladybird_declares_the_services_it_spawns_as_data(): + """`bazel build //:ladybird` must rebuild the services too. + + The services are found by PATH at runtime (get_paths_for_helper_process), not + linked, so nothing in the build graph said the browser needs them: a build of + //:ladybird alone left whatever WebContent happened to be in bazel-bin from a + previous build. Ulf hit it in its most confusing form -- ladybird dated Aug 20 + beside a WebContent dated Aug 11, a browser from this pin talking to a service + from the previous one. Upstream had inserted ~3 IPC messages between the pins, + shifting every id after them, so every message failed to decode with + + Local endpoint error: Can't read past the end of the stream memory + Peer endpoint error: Endpoint magic number mismatch, not my message! + + which reads like a codegen or ABI bug and is nothing of the kind. (The magic + 0xffa5367a is AK::string_hash("WebContentServer"), i.e. the CORRECT endpoint; + the message NUMBERING is what disagreed -- 7/7 against the old pin, 0/7 against + the new one.) His fix: declare them. + + `data`, not `deps`: separate processes, not link inputs -- the relationship + LibWasm already has to cranelift-compiler. + """ + build = os.path.join(_EXAMPLE, "workspace", "BUILD.bazel") + with open(build) as f: + text = f.read() + # The ladybird cc_binary block. + i = text.index("name = 'ladybird'") + block = text[i:text.index("\n)", i)] + m = re.search(r"data = \[([^\]]*)\]", block) + assert m, "//:ladybird declares no data at all" + data = m.group(1) + for svc in ("Compositor", "ImageDecoder", "RequestServer", "WebContent", "WebWorker"): + assert f"':{svc}'" in data, ( + f"//:ladybird does not declare :{svc} in data, so `bazel build " + "//:ladybird` can leave a STALE copy of it in bazel-bin and the IPC " + "message ids will not match") + # deps would be wrong: they are processes, not libraries to link. + deps = re.search(r"deps = \[(.*?)\]", block, re.S) + if deps: + assert "':WebContent'" not in deps.group(1), \ + "the services must be data (spawned processes), not deps (link inputs)" + + +def test_the_spawned_service_list_is_derived_from_ladybirds_own_source(): + """A hand-kept list of services is one new service away from the same bug. + + Upstream ADDS services (Compositor is new since the previous pin), and the + names are already written down in the place the runtime lookup uses them: the + string literals passed to launch_server_process<> in HelperProcess.cpp. So the + emitter reads them from there, and fails loudly if it parses none -- an empty + list would silently re-emit the bug it exists to prevent. + """ + emitter = os.path.join(_EXAMPLE, "workspace", "Meta", "emit_build_bazel.py") + with open(emitter) as f: + text = f.read() + assert "def spawned_services" in text, "the emitter has no spawned_services()" + assert "HelperProcess.cpp" in text, \ + "the service list is not read from HelperProcess.cpp (hand-listed?)" + assert "launch_server_process" in text, \ + "the service names must come from the launch_server_process<> call sites" + fn = text[text.index("def spawned_services"):] + fn = fn[:fn.index("\ndef ")] + # No literal roster inside the function: that is the thing being avoided. + for svc in ("WebContent", "RequestServer", "ImageDecoder", "WebWorker"): + assert f'"{svc}"' not in fn and f"'{svc}'" not in fn, \ + f"spawned_services() hardcodes {svc} instead of deriving it" + # An empty parse must be fatal, not an empty list. + assert "sys.exit" in fn, \ + ("spawned_services() must FAIL when it parses no service names -- returning " + "[] would silently rebuild //:ladybird without its services again") From a952899af9d3fc8ce4442dd07f4f61afcea77f16 Mon Sep 17 00:00:00 2001 From: Ulf Adams Date: Thu, 20 Aug 2026 13:59:17 +0000 Subject: [PATCH 58/59] Take upstream #11041's three patches; mine crashed Ulf's browser Ulf: "We should use the upstream patch". Yes -- and not just because upstream's is upstream's. Mine was BROKEN, and his crash trace named the line: VERIFICATION FAILED: m_ptr at ./AK/OwnPtr.h:134 #0 ...CallableWrapper::call() #1 Core::Notifier::event(Core::Event&) {lambda()#2} is the read notifier's on_activation -- the frame that CALLS on_finish. My 0002's release_response_fd(), invoked from inside on_finish's completion branch, set m_internal_stream_data->read_stream = nullptr while on_activation was still on the stack and still about to dereference it: if (m_internal_stream_data->read_stream->is_eof()) // Request.cpp:376 m_internal_stream_data->read_notifier->close(); AK::OwnPtr::operator-> is VERIFY(m_ptr), so the null became a trap, then SIGILL. A use-after-null ONE STACK FRAME UP from the code I changed: invisible on every workload I built here, a crash after a few minutes of real browsing on his. My framing was upside down. I argued 0002 was needed because "collectable" and "closed" are different claims and for an fd received over IPC only the second is the bug. True, and the wrong question: it is not WHEN the fd is closed but WHO owns the state while the stack is still unwound. The completion branch runs INSIDE the read loop's callback, so anything it destroys, the caller may still touch. Upstream doesn't close the fd at all -- it makes the existing DEFERRED teardown reachable on the three paths where it was missed, and deferred is precisely what makes it safe. My mechanism was in direct conflict with the one the code already had. Worse, I had already spotted the same bug in our 0001 (defer_teardown() after user_on_finish instead of before, so the Request is not pinned across the callback), written that it was "the kind of latent ordering bug that shows up as a rare crash on someone else's machine" -- and carried it anyway because it had never fired for me. "Never observed here" is a statement about my workloads. So patches/ is now upstream's three commits (63817a66, 2ae78115, 8636af41), split out of upload/11041.patch, each annotated IN PLACE with its provenance and the order-matters reasoning, each with an .effect-grep so --verify recognises the merged fix on a tree newer than our pin instead of demanding a patch that would conflict. Verified: all three apply in series at 71fb301a (RC 0), the resulting source has defer_teardown() BEFORE user_on_finish, no `read_stream = nullptr` anywhere, and no release_response_fd at all. Built and ran it: patch 1 adds openResponsePipeCount() to Internals.idl, which correctly invalidated and rebuilt the whole LibWeb binding set through the overlay's codegen (1571 actions), --headless=text renders, and a 150-fetch + 50-abort workload against a local server produced zero VERIFICATION FAILED / SIGILL and a flat fd census. One detour worth recording: the first build died with `ar failed` and NO diagnostic, on the LibWeb archive. That is todo e91c1905 -- Bazel+binutils report ENOSPC as an empty tool failure that reads like a code error. Disk was at 95%; pruning the disk cache fixed it. I recognised the shape this time instead of debugging the code. Tests: the two tests that named the deleted patches are replaced by property tests -- every patch in the series must have an .effect-grep (derived by glob, not by filename, which is how those tests rotted), the windowed check is found by CONTENT rather than name, and a new test asserts no patch may null read_stream or reintroduce release_response_fd. A future "optimisation" back to my version fails loudly. --- docs/UPSTREAM-ladybird-fd-leaks.md | 66 +++++ examples/ladybird/README.md | 93 ++++--- ...request-when-body-is-delivered.effect-grep | 17 -- ...-down-request-when-body-is-delivered.patch | 62 ----- ...e-pipes-when-requests-complete.effect-grep | 24 ++ ...esponse-pipes-when-requests-complete.patch | 234 ++++++++++++++++++ ...ease-response-fd-on-completion.effect-grep | 21 -- ...ts-release-response-fd-on-completion.patch | 126 ---------- ...ipes-when-fetches-are-canceled.effect-grep | 18 ++ ...onse-pipes-when-fetches-are-canceled.patch | 136 ++++++++++ ...on-parked-for-content-sniffing.effect-grep | 22 ++ ...vigation-parked-for-content-sniffing.patch | 168 +++++++++++++ tests/test_apply_overlay.py | 140 ++++++----- 13 files changed, 807 insertions(+), 320 deletions(-) delete mode 100644 examples/ladybird/patches/0001-librequests-tear-down-request-when-body-is-delivered.effect-grep delete mode 100644 examples/ladybird/patches/0001-librequests-tear-down-request-when-body-is-delivered.patch create mode 100644 examples/ladybird/patches/0001-upstream-11041-release-response-pipes-when-requests-complete.effect-grep create mode 100644 examples/ladybird/patches/0001-upstream-11041-release-response-pipes-when-requests-complete.patch delete mode 100644 examples/ladybird/patches/0002-requests-release-response-fd-on-completion.effect-grep delete mode 100644 examples/ladybird/patches/0002-requests-release-response-fd-on-completion.patch create mode 100644 examples/ladybird/patches/0002-upstream-11041-release-response-pipes-when-fetches-are-canceled.effect-grep create mode 100644 examples/ladybird/patches/0002-upstream-11041-release-response-pipes-when-fetches-are-canceled.patch create mode 100644 examples/ladybird/patches/0003-upstream-11041-tear-down-a-navigation-parked-for-content-sniffing.effect-grep create mode 100644 examples/ladybird/patches/0003-upstream-11041-tear-down-a-navigation-parked-for-content-sniffing.patch diff --git a/docs/UPSTREAM-ladybird-fd-leaks.md b/docs/UPSTREAM-ladybird-fd-leaks.md index 1674802..096aa6f 100644 --- a/docs/UPSTREAM-ladybird-fd-leaks.md +++ b/docs/UPSTREAM-ladybird-fd-leaks.md @@ -643,3 +643,69 @@ Not done here: #11041 is unmerged, so nothing is deleted yet. What I would *not* carry our `0002` forward alongside upstream's three — two mechanisms closing the same fd, one of them justified by a theory the other one falsifies, is how the next reader gets misled. + +--- + +## Done: the overlay now carries upstream's three, because mine crashed his browser + +The section above was written while #11041 was still in review, and it ended with "not +done here: nothing is deleted yet". It is done now, and the trigger was not the merge. + +Ulf ran the Bazel-built browser with my two patches applied. It loaded pages, and the fd +census was flat — *"I also didn't see immediate socket leak, so that's good"* — and then +after a few minutes it died: + +``` +VERIFICATION FAILED: m_ptr at ./AK/OwnPtr.h:134 +#0 in AK::Function::CallableWrapper::call() +#1 in Core::Notifier::event(Core::Event&) +``` + +`{lambda()#2}` is the read notifier's `on_activation`. That is the frame which *calls* +`on_finish` — and my `release_response_fd()`, invoked from inside the completion branch of +`on_finish`, set `m_internal_stream_data->read_stream = nullptr` while `on_activation` was +still on the stack and still about to dereference it: + +```cpp + } while (true); + + if (m_internal_stream_data->read_stream->is_eof()) // Request.cpp:376 + m_internal_stream_data->read_notifier->close(); +``` + +`AK::OwnPtr::operator->` is `VERIFY(m_ptr)` (`AK/OwnPtr.h:134`), so the null became a +trap and then `SIGILL`. **A use-after-null one stack frame up from the code I changed.** + +Three things worth keeping from this. + +**My reasoning about the fix was upside down.** I argued `0002` was needed because +"collectable" and "closed" are different claims, and for an fd received over IPC only the +second is the bug. That sentence is still true, and it was the wrong frame: the question +is not *when is the fd closed* but *who owns it while the stack is still unwound*. The +completion branch runs **inside** the read loop's callback; anything it destroys, the +caller may still touch. Upstream's patch does not close the fd at all — it makes the +existing deferred teardown *reachable* on each of the three paths where it was missed, and +the teardown is deferred precisely so it cannot destroy state a live frame is using. The +mechanism I reached for (destroy it now, synchronously, at the point I can prove the body +is done) was in direct conflict with the one the code already had. + +**The ordering bug I had already spotted was the same bug, and I under-rated it.** I noted +that upstream calls `defer_teardown()` *before* `user_on_finish` while my `0001` called it +after, wrote that it was "the kind of latent ordering bug that shows up as a rare crash on +someone else's machine", and then carried the patch anyway because it had never fired for +me. It fired for him. "Never observed here" is a statement about my workloads. + +**One prediction verified, one falsified.** I predicted that if upstream's patches take the +rate to zero, `0002`'s surviving-reference theory was unnecessary — that holds, and it is +why `0002` is deleted rather than rebased. But I also wrote that the honest reading was +"`0002` was a workaround for a missing call site, and it worked on my box because my box +only ever hit the class `0001` already covered." That was too kind: it did not merely fail +to help on his box, it *broke* his box, and the reason was visible in the source the whole +time without any measurement at all. + +The overlay's `patches/` is now upstream's three commits, annotated in place with their +provenance, each with an `.effect-grep` so `--verify` recognises the merged fix on a tree +newer than our pin. `Internals.idl` gains `openResponsePipeCount()` in patch 1, which +regenerates the LibWeb bindings — the overlay's codegen picks that up correctly (verified: +the IDL change invalidates and rebuilds the binding set). diff --git a/examples/ladybird/README.md b/examples/ladybird/README.md index cb002ed..bba09d7 100644 --- a/examples/ladybird/README.md +++ b/examples/ladybird/README.md @@ -138,14 +138,19 @@ reading it: (`71fb301a`). Against a different tree the build fails on a moved file — or worse, silently omits a new one. 2. **The patches**, which have to be applied or the browser leaks one socket fd per - completed HTTP request until it dies of `EMFILE` overnight - ([`0001`](patches/0001-librequests-tear-down-request-when-body-is-delivered.patch) - and [`0002`](patches/0002-requests-release-response-fd-on-completion.patch); + completed HTTP request until it dies of `EMFILE` overnight. These are now + **upstream's own three commits from Ladybird PR #11041**, carried only because our + pin predates the merge + ([`0001`](patches/0001-upstream-11041-release-response-pipes-when-requests-complete.patch), + [`0002`](patches/0002-upstream-11041-release-response-pipes-when-fetches-are-canceled.patch), + [`0003`](patches/0003-upstream-11041-tear-down-a-navigation-parked-for-content-sniffing.patch); see [`docs/UPSTREAM-ladybird-fd-leaks.md`](../../docs/UPSTREAM-ladybird-fd-leaks.md)). - There used to be four: the `PYTHONHASHSEED` determinism fix and the - non-self-contained `UI/Qt/TabBar.h` are both **fixed upstream** at this pin, so - the repin from `f9e34731` deleted them. A patch directory that only ever grows is - a patch directory nobody re-checks against upstream. + They **replace two patches written here**, one of which crashed Ulf's browser — + see "Why upstream's patches replaced mine" below. There used to be four others: the + `PYTHONHASHSEED` determinism fix and the non-self-contained `UI/Qt/TabBar.h` are + both **fixed upstream** at this pin, so the repin from `f9e34731` deleted them. A + patch directory that only ever grows is a patch directory nobody re-checks against + upstream. 3. **The rename**: `bazelrc.txt` → `.bazelrc`. Stored under a different name so a `cp -r` cannot be mistaken for a working build — and a rename a human does by hand is a rename a human forgets. @@ -1162,32 +1167,54 @@ Honest inventory of what stops this from being a clone-and-build. upstream fixes — a patch keeps applying long after it stops being needed, so "it still applies" is not evidence that it is still a bug. - What remains is the fd leak, in two patches that are one series. The first, - found only because the Bazel-built browser was left running overnight, is a - resource leak rather than a build defect: - [`patches/0001-librequests-tear-down-request-when-body-is-delivered.patch`](patches/0001-librequests-tear-down-request-when-body-is-delivered.patch). - WebContent leaks one AF_UNIX socket fd per *completed* HTTP request — a cycle - between the GC heap (four `GC::Root`s from `Fetching.cpp:2338`) and the - refcount heap (`Response` holding `Requests::Request` by `RefPtr`) that neither - collector can break — until `EMFILE` aborts three processes at once. The patch - is **necessary but not sufficient**: it fixes the completed-request class - (identifiable in `ss -np` by a dead peer, `* 0`), while a stalled-body class - with a live peer survives it. Both are written up with reproductions in - [`docs/UPSTREAM-ladybird-fd-leaks.md`](../../docs/UPSTREAM-ladybird-fd-leaks.md). - - The **second** is the other half of the completed-request class, and it only showed - up on a tree that was not mine: - [`patches/0002-requests-release-response-fd-on-completion.patch`](patches/0002-requests-release-response-fd-on-completion.patch). - With `0001`'s teardown applied, Ulf still measured **97 leaked sockets/min**, all - `peer=DEAD` and all sent by RequestServer — because dropping the callbacks unpins - the GC cycle but does not *close the descriptor*: the response pipe is released - only by `~Request`, so any other reference to the request retains one fd per - completed request. `0002` closes it where the body is already proven complete — the - next patch in the series, applied on top of `0001`; `patches/*.patch` is applied by - glob in full, so an *alternative* to a patch must never live there. - Measured A/B on 200 completed requests: **208 sockets (203 dead) → 6 (0 dead)**, - with body delivery intact. "Collectable" and "closed" are different claims, and - for an fd received over IPC only the second one is the bug. + ### Why upstream's patches replaced mine + + What remains is the fd leak, and the overlay now carries **upstream's three + commits** (Ladybird PR #11041, by sideshowbarker) rather than the two patches + written here. That swap was not housekeeping — **my `0002` crashed Ulf's browser**, + and the trace named the exact line: + + ``` + VERIFICATION FAILED: m_ptr at ./AK/OwnPtr.h:134 + #0 ...CallableWrapper::call() + ``` + + That lambda is the read notifier's `on_activation`, and it is the frame that *calls* + `on_finish`. My `release_response_fd()` ran from inside that completion branch and + set `m_internal_stream_data->read_stream = nullptr` — while the calling frame goes on + to dereference exactly that `OwnPtr`: + + ```cpp + if (m_internal_stream_data->read_stream->is_eof()) // Request.cpp:376 + m_internal_stream_data->read_notifier->close(); + ``` + + `OwnPtr::operator->` is `VERIFY(m_ptr)`. So the fix nulled a pointer its own caller + still owned: a use-after-null one stack frame up, invisible on my workloads and a + crash after a few minutes of real browsing on his. Upstream never touches + `read_stream`; it only ensures `defer_teardown()` is *reached*, and reaches it + **before** `user_on_finish` so the deferred lambda's `NonnullRefPtr` pins the + `Request` across the callback — where my `0001` called it *after*, which was a second + latent use-after-free in the same pair. + + The deeper lesson is about what the two patches were *for*. Mine closed the fd on the + theory that a surviving reference pinned it; upstream fixes the leak by running the + teardown on all three paths it can be missed — ordinary completion, `abort()`/ + `terminate()`, and a navigation parked for content sniffing. If reaching the teardown + is sufficient, the surviving-reference theory was never needed, and carrying both + would have meant two mechanisms closing one descriptor, one of them justified by a + theory the other disproves. Upstream also found a class my instrument could not: + `fd_census.py` ranks by growth and splits `peer=DEAD`/`ALIVE`, which identified the + completed-request class but says nothing specific about the *cancel* path. And their + patch 3 is the lead I had and could not reproduce — my two workloads abandoned + navigations without **destroying the navigable**, so they exercised everything except + the condition that matters. A falsified workload was evidence about my workload, not + about the hypothesis. + + All three are annotated in place with their provenance and carry `.effect-grep` + files, so `apply_overlay.sh --verify` recognises the merged upstream fix on a tree + newer than our pin instead of demanding a patch that would conflict. Delete all + three on the first repin past the merge. Diagnose a running browser with [`fd_census.py`](fd_census.py), which needs no patch and no particular tree — `python3 fd_census.py --all --watch 30` ranks every diff --git a/examples/ladybird/patches/0001-librequests-tear-down-request-when-body-is-delivered.effect-grep b/examples/ladybird/patches/0001-librequests-tear-down-request-when-body-is-delivered.effect-grep deleted file mode 100644 index 8edb310..0000000 --- a/examples/ladybird/patches/0001-librequests-tear-down-request-when-body-is-delivered.effect-grep +++ /dev/null @@ -1,17 +0,0 @@ -# The EFFECT this patch must have, as extended regexes over the file it touches -# (Libraries/LibRequests/Request.cpp). Checked only when the patch's exact bytes -# are NOT in the tree, so upstream's own fix -- or any equivalent one -- is -# recognised as "already fixed" instead of being reported as missing, which is -# what happened to Ulf on a tree NEWER than our pin. -# -# The effect: the branch that decides the body is fully delivered (the one that -# calls the user's finish callback) must ALSO tear the request down, so the -# callbacks holding GC::Roots are dropped and the response fd is closed. -# -# The window matters. defer_teardown() already appears in stop() and -# did_transfer(), so a whole-file grep for it passes on a tree WITHOUT the fix -- -# verified, and pinned by a test. So anchor on the branch condition and require -# the call within the following few lines. -@window 12 has_received_all_reported_bytes = -user_finish_called = true -defer_teardown\(\); diff --git a/examples/ladybird/patches/0001-librequests-tear-down-request-when-body-is-delivered.patch b/examples/ladybird/patches/0001-librequests-tear-down-request-when-body-is-delivered.patch deleted file mode 100644 index 0dbd4d1..0000000 --- a/examples/ladybird/patches/0001-librequests-tear-down-request-when-body-is-delivered.patch +++ /dev/null @@ -1,62 +0,0 @@ -From: any2bazel Ladybird migration -Subject: [PATCH] LibRequests: tear the request down once its body is delivered - -WebContent leaks one AF_UNIX socket fd per *completed* HTTP request, so a -long-running browser eventually dies with EMFILE. Ulf's build died overnight -with `dup: Too many open files (errno=24)` at -Libraries/LibWebView/CompositorConnection.cpp:62, which kills three processes -(the MUST there aborts WebContent; the Compositor's VERIFY(connection) at -Services/Compositor/ConnectionFromClient.cpp:68 follows; then ladybird's MUST -in initialize_client). - -The leaked fd is RequestServer's response-pipe half (RequestPipe::create, -Services/RequestServer/RequestPipe.cpp:46). RequestServer closes its end; -WebContent never closes its. It is held by a reference cycle that spans the GC -heap and the refcount heap: - - Heap::m_roots -> the four GC::Roots created at Fetch/Fetching/Fetching.cpp:2338 - -> moved onto the refcounted Requests::Request's callbacks - -> Response holds that Request by RefPtr (Responses.h) - -> which is reachable from the roots. - -Neither collector can break it: the GC sees a live explicit root, and the -refcount never reaches zero. Requests::Request owns the response fd, so the fd -lives as long as the cycle -- i.e. forever. - -Request::defer_teardown() is the only thing that clears those callbacks, and its -only callers are stop() and did_transfer(). Normal completion never reaches it, -which is why aborted fetches leak nothing and successful ones leak one fd each. - -The obvious one-liner -- defer_teardown() at the end of did_finish() -- is WRONG -and I verified it: the body may still be draining when the request_finished IPC -arrives, so it renders blank pages while making the fd count look perfect. The -correct point is where the stream handler has *already decided* the body is -complete (the user_finish_called / EOF / has_received_all_reported_bytes branch), -which is exactly where the user's finish callback runs. - -Measured on a CMake build of f9e34731, WebContent `socket:` fds after 200 -fetches: 208 before, 5 after, with the page's own success criteria unchanged -(ok=200 bad=0). Verified `--headless=text` still prints the document text, so -the body is still delivered. - -NOTE (scope): this fixes the completed-request class of the leak, identifiable -in `ss -np` by a **dead peer** (`* 0` peer inode: RequestServer already closed -its half). A second, independent class survives this patch: a response whose -body stalls server-side is retained with its peer still ALIVE, because -`on_finish` never runs at all. That one needs the cycle broken at the Response -end and is reported separately; this patch is necessary, not sufficient. -diff --git a/Libraries/LibRequests/Request.cpp b/Libraries/LibRequests/Request.cpp -index 6262425a..7bac6c81 100644 ---- a/Libraries/LibRequests/Request.cpp -+++ b/Libraries/LibRequests/Request.cpp -@@ -327,6 +327,10 @@ void Request::set_up_internal_stream_data(DataReceived on_data_available) - if (!m_internal_stream_data->user_finish_called && (!m_internal_stream_data->read_stream || m_internal_stream_data->read_stream->is_eof() || has_received_all_reported_bytes)) { - m_internal_stream_data->user_finish_called = true; - user_on_finish(m_internal_stream_data->total_size, m_internal_stream_data->timing_info, m_internal_stream_data->network_error); -+ -+ // The body is fully delivered: drop the callbacks (and the GC::Roots they captured) -+ // and close the response fd, as stop()/did_transfer() do for the abandoned cases. -+ defer_teardown(); - } - }; - diff --git a/examples/ladybird/patches/0001-upstream-11041-release-response-pipes-when-requests-complete.effect-grep b/examples/ladybird/patches/0001-upstream-11041-release-response-pipes-when-requests-complete.effect-grep new file mode 100644 index 0000000..23cc47a --- /dev/null +++ b/examples/ladybird/patches/0001-upstream-11041-release-response-pipes-when-requests-complete.effect-grep @@ -0,0 +1,24 @@ +# The EFFECT this patch must have, as extended regexes over the file it touches +# (Libraries/LibRequests/Request.cpp). Checked only when the patch's exact bytes +# are NOT in the tree, so a tree that is NEWER than our pin -- one where #11041 +# has merged -- verifies as "already fixed" instead of being told to apply a +# patch that would conflict. That is the whole reason this mechanism exists. +# +# The effect: the branch that decides the body is fully delivered (the one that +# calls the user's finish callback) must ALSO tear the request down, so the +# ReadStream holding the response-pipe fd is released instead of waiting for GC. +# +# The window matters. defer_teardown() already appears in stop() and +# did_transfer(), so a whole-file grep for it passes on a tree WITHOUT the fix -- +# verified, and pinned by a test. So anchor on the branch condition and require +# the call within the following few lines. +# +# NOTE the ORDER is deliberately NOT asserted here. Upstream calls +# defer_teardown() BEFORE user_on_finish (so the deferred lambda's NonnullRefPtr +# pins the Request across the callback); the patch this replaced called it after, +# which was a latent use-after-free. A grep cannot express "before" without +# becoming a byte comparison, and the exact-bytes check already covers our own +# version -- so this asserts the fix is PRESENT and leaves ordering to the patch. +@window 12 has_received_all_reported_bytes = +user_finish_called = true +defer_teardown\(\); diff --git a/examples/ladybird/patches/0001-upstream-11041-release-response-pipes-when-requests-complete.patch b/examples/ladybird/patches/0001-upstream-11041-release-response-pipes-when-requests-complete.patch new file mode 100644 index 0000000..d559d4c --- /dev/null +++ b/examples/ladybird/patches/0001-upstream-11041-release-response-pipes-when-requests-complete.patch @@ -0,0 +1,234 @@ +From 63817a66757e4005383cd7c4e9508fa188e823fc Mon Sep 17 00:00:00 2001 +From: sideshowbarker +Date: Fri, 7 Aug 2026 09:44:33 -0700 +Subject: [PATCH 1/3] LibRequests+LibWeb: Release response pipes when requests + complete +MIME-Version: 1.0 +Content-Type: text/plain; charset=UTF-8 +Content-Transfer-Encoding: 8bit + +Problem: Pages issuing fetches in a tight loop accumulate one open Unix +socket per completed fetch in their WebContent process. The WPT scroll- +to-text-fragment/redirects.html test polls that way, and drove a local +WebContent process to ~21,900 socket/fileport attachments — at which +point the process crashed after receiving the next response pipe failed. +In a local repro driving about 1,500 fetches per second, WebContent grew +from 5,541 to 63,543 Unix-socket descriptors in 43 seconds. + +Cause: Requests::Request tears down its stream state only when a request +is stopped or transferred. On ordinary completion, the ReadStream with +the response-pipe descriptor, its notifier, and the internal buffers all +stay alive until the Request object is destroyed – and a fetch Response +holds a reference to its Request until GC. The scarce resource is file +descriptors — but GC pressure is measured in heap bytes. So, a tight +fetch loop retires thousands of completed requests without GC ever +having a reason to run — and so, the descriptor table fills up. + +Fix: Schedule the existing deferred teardown at the moment a request +delivers its user-finish callback (the single point every request passes +through on ordinary completion) — for buffered and unbuffered modes and +for network errors alike. That’s only reached once all response data has +been delivered. So streaming consumers and paused document loads are un- +affected — and stopped/transferred requests keep their existing teardown +paths. The teardown must stay deferred: It’s scheduled from inside the +callback chain it destroys, and the deferred task also keeps the Request +alive if the callback drops the last ref. With this change, the repro +holds steady at 0–2 open response pipes at an unchanged request rate. +--- + Libraries/LibRequests/Request.cpp | 20 +++++++++++ + Libraries/LibRequests/Request.h | 9 +++-- + Libraries/LibWeb/Internals/Internals.cpp | 6 ++++ + Libraries/LibWeb/Internals/Internals.h | 1 + + Libraries/LibWeb/Internals/Internals.idl | 1 + + ...sponse-pipes-released-after-completion.txt | 1 + + ...ponse-pipes-released-after-completion.html | 33 +++++++++++++++++++ + 7 files changed, 66 insertions(+), 5 deletions(-) + create mode 100644 Tests/LibWeb/Text/expected/Fetch/response-pipes-released-after-completion.txt + create mode 100644 Tests/LibWeb/Text/input/Fetch/response-pipes-released-after-completion.html + +| UPSTREAM PATCH -- Ladybird PR #11041 by sideshowbarker, commit 63817a66757e4005383cd7c4e9508fa188e823fc. +| NOT OURS. Carried only because our pin 71fb301a predates the merge; DELETE all +| three (and this note) on the first repin past it -- see the README patch table. +| +| Why upstream's and not the two patches this replaces: those were written here +| before #11041 existed, and one of them CRASHED Ulf's browser. Our 0002 +| (release_response_fd) nulled m_internal_stream_data->read_stream from inside the +| completion branch, while the read_notifier's on_activation lambda -- the frame +| that CALLS on_finish -- goes on to dereference that same OwnPtr: +| +| if (m_internal_stream_data->read_stream->is_eof()) <-- Request.cpp:376 +| m_internal_stream_data->read_notifier->close(); +| +| so the null landed on VERIFY(m_ptr) in AK::OwnPtr::operator-> (AK/OwnPtr.h:134), +| which is exactly the trace Ulf sent: CallableWrapper<...set_up_internal_stream_data +| ...{lambda#2}>::call() -> ak_verification_failed -> SIGILL. Upstream never nulls +| read_stream; it only ensures defer_teardown() is REACHED, and it reaches it BEFORE +| user_on_finish so the deferred lambda's NonnullRefPtr pins the Request across the +| callback. Our 0001 called it after, which is a second latent bug in the pair. +| +| The deeper point, recorded in docs/UPSTREAM-ladybird-fd-leaks.md: our 0002 existed +| to close the fd on the theory that a surviving reference pinned it. Upstream fixes +| the leak by running the teardown on all three paths -- so the theory was +| unnecessary, and carrying both would have been two mechanisms closing one fd, one +| of them justified by a theory the other disproves. + +diff --git a/Libraries/LibRequests/Request.cpp b/Libraries/LibRequests/Request.cpp +index 6262425a5bad5..16234917a4927 100644 +--- a/Libraries/LibRequests/Request.cpp ++++ b/Libraries/LibRequests/Request.cpp +@@ -35,6 +35,25 @@ static Optional map_body_file(int fd, u64 offset, u64 size + return payload.release_value(); + } + ++static size_t s_live_read_stream_count = 0; ++ ++size_t ReadStream::live_count() ++{ ++ return s_live_read_stream_count; ++} ++ ++ReadStream::ReadStream(NonnullOwnPtr stream, NonnullRefPtr notifier) ++ : m_stream(move(stream)) ++ , m_notifier(move(notifier)) ++{ ++ ++s_live_read_stream_count; ++} ++ ++ReadStream::~ReadStream() ++{ ++ --s_live_read_stream_count; ++} ++ + ErrorOr> ReadStream::create(int reader_fd) + { + #if defined(AK_OS_WINDOWS) +@@ -326,6 +345,7 @@ void Request::set_up_internal_stream_data(DataReceived on_data_available) + auto has_received_all_reported_bytes = m_internal_stream_data->request_done && m_internal_stream_data->delivered_size >= m_internal_stream_data->total_size; + if (!m_internal_stream_data->user_finish_called && (!m_internal_stream_data->read_stream || m_internal_stream_data->read_stream->is_eof() || has_received_all_reported_bytes)) { + m_internal_stream_data->user_finish_called = true; ++ defer_teardown(); + user_on_finish(m_internal_stream_data->total_size, m_internal_stream_data->timing_info, m_internal_stream_data->network_error); + } + }; +diff --git a/Libraries/LibRequests/Request.h b/Libraries/LibRequests/Request.h +index b4046fa8d114a..69cdf0d6d0398 100644 +--- a/Libraries/LibRequests/Request.h ++++ b/Libraries/LibRequests/Request.h +@@ -55,6 +55,9 @@ class ResponseData { + class ReadStream { + public: + static ErrorOr> create(int reader_fd); ++ ~ReadStream(); ++ ++ static size_t live_count(); + + NonnullRefPtr const& notifier() const { return m_notifier; } + +@@ -63,11 +66,7 @@ class ReadStream { + ErrorOr read_some(Bytes bytes) { return m_stream->read_some(bytes); } + + private: +- ReadStream(NonnullOwnPtr stream, NonnullRefPtr notifier) +- : m_stream(move(stream)) +- , m_notifier(move(notifier)) +- { +- } ++ ReadStream(NonnullOwnPtr stream, NonnullRefPtr notifier); + + NonnullOwnPtr m_stream; + NonnullRefPtr m_notifier; +diff --git a/Libraries/LibWeb/Internals/Internals.cpp b/Libraries/LibWeb/Internals/Internals.cpp +index d221019b6e8de..65cb4a9150f3d 100644 +--- a/Libraries/LibWeb/Internals/Internals.cpp ++++ b/Libraries/LibWeb/Internals/Internals.cpp +@@ -22,6 +22,7 @@ + #include + #include + #include ++#include + #include + #include + #include +@@ -614,6 +615,11 @@ void Internals::simulate_request_server_connection_loss() + page().client().page_did_lose_request_server_connection(); + } + ++WebIDL::UnsignedLongLong Internals::open_response_pipe_count() ++{ ++ return Requests::ReadStream::live_count(); ++} ++ + WebIDL::ExceptionOr Internals::set_content_blockers(Utf16String const& patterns_source) + { + Utf16StringBuilder patterns_builder; +diff --git a/Libraries/LibWeb/Internals/Internals.h b/Libraries/LibWeb/Internals/Internals.h +index b2df9d5c9e051..cd0a9bbf4a2dc 100644 +--- a/Libraries/LibWeb/Internals/Internals.h ++++ b/Libraries/LibWeb/Internals/Internals.h +@@ -106,6 +106,7 @@ class WEB_API Internals final : public InternalsBase { + + bool set_http_memory_cache_enabled(bool enabled); + void simulate_request_server_connection_loss(); ++ WebIDL::UnsignedLongLong open_response_pipe_count(); + WebIDL::ExceptionOr set_content_blockers(Utf16String const& patterns); + void set_content_blocking_enabled(bool enabled); + WebIDL::UnsignedLongLong partial_layout_count(); +diff --git a/Libraries/LibWeb/Internals/Internals.idl b/Libraries/LibWeb/Internals/Internals.idl +index e2634eadcac82..107b11fd08160 100644 +--- a/Libraries/LibWeb/Internals/Internals.idl ++++ b/Libraries/LibWeb/Internals/Internals.idl +@@ -85,6 +85,7 @@ interface Internals { + + boolean setHttpMemoryCacheEnabled(boolean enabled); + undefined simulateRequestServerConnectionLoss(); ++ unsigned long long openResponsePipeCount(); + undefined setContentBlockers(Utf16DOMString patterns); + undefined setContentBlockingEnabled(boolean enabled); + unsigned long long partialLayoutCount(); +diff --git a/Tests/LibWeb/Text/expected/Fetch/response-pipes-released-after-completion.txt b/Tests/LibWeb/Text/expected/Fetch/response-pipes-released-after-completion.txt +new file mode 100644 +index 0000000000000..836a4a872c4d4 +--- /dev/null ++++ b/Tests/LibWeb/Text/expected/Fetch/response-pipes-released-after-completion.txt +@@ -0,0 +1 @@ ++open response pipes after fetches: 0 +diff --git a/Tests/LibWeb/Text/input/Fetch/response-pipes-released-after-completion.html b/Tests/LibWeb/Text/input/Fetch/response-pipes-released-after-completion.html +new file mode 100644 +index 0000000000000..29c66bcf3cd40 +--- /dev/null ++++ b/Tests/LibWeb/Text/input/Fetch/response-pipes-released-after-completion.html +@@ -0,0 +1,33 @@ ++ ++ ++ + diff --git a/examples/ladybird/patches/0002-requests-release-response-fd-on-completion.effect-grep b/examples/ladybird/patches/0002-requests-release-response-fd-on-completion.effect-grep deleted file mode 100644 index d1da807..0000000 --- a/examples/ladybird/patches/0002-requests-release-response-fd-on-completion.effect-grep +++ /dev/null @@ -1,21 +0,0 @@ -# The EFFECT this patch must have, as extended regexes over the file it touches -# (Libraries/LibRequests/Request.cpp). Checked only when the patch's exact bytes are -# NOT in the tree, so an equivalent fix -- upstream's or anyone's -- counts as -# "already fixed" rather than "missing". Same reasoning as 0001's .effect-grep: -# `git apply --check -R` proves my bytes, not the fix. -# -# The effect: the response fd received from RequestServer must be closed WHERE THE -# BODY IS PROVEN COMPLETE, not merely when the Request object dies. 0001 dropped the -# callbacks (which unpins the GC cycle); this closes the fd even while something else -# still holds a reference to the Request -- the case that survived 0001 on Ulf's -# tree at 97 fds/min. -# -# The window matters, for the same reason as 0001: a bare close/notifier call -# appears elsewhere in this file (~Request, attach_read_stream), so a whole-file grep -# would pass on an unpatched tree. Anchor on the completion branch and require the -# release within the following few lines. NOTE: apply_overlay.sh honours only the -# FIRST @window, so keep this to one window -- a second would be silently checked -# against the first window's slice. -@window 12 has_received_all_reported_bytes = -user_finish_called = true -release_response_fd\(\)|read_notifier->close\(\)|System::close\(m_fd\) diff --git a/examples/ladybird/patches/0002-requests-release-response-fd-on-completion.patch b/examples/ladybird/patches/0002-requests-release-response-fd-on-completion.patch deleted file mode 100644 index cdefac2..0000000 --- a/examples/ladybird/patches/0002-requests-release-response-fd-on-completion.patch +++ /dev/null @@ -1,126 +0,0 @@ -From: any2bazel Ladybird migration -Subject: [PATCH] LibRequests: close the response fd on completion, not just at ~Request - -Applies ON TOP OF the teardown fix (patches/0001, or upstream's equivalent), which is -the previous patch in this series. The two edit adjacent lines in the same branch, so -apply them in order -- apply_overlay.sh does that for you. - -WHY A SECOND PATCH. With the teardown fix applied, Ulf still measured 97 leaked -sockets/min (81->823 sockets, 73->815 dead-peer over 458 s) while every workload -reproducible here was flat. Both instruments agreed on what was leaking: - - - fd_census.py: every leaked socket was peer=DEAD, i.e. RequestServer had already - closed its end -> these were COMPLETED requests (class A) on a tree that already - carried the class A fix. - - fdtrace.c: every leaked fd was a received SCM_RIGHTS attachment whose sender was - RequestServer (SO_PEERCRED; `ps -p 2261433` named it). That pins the fd to exactly - one thing: the per-request response pipe from RequestPipe::create() - (Services/RequestServer/RequestPipe.cpp:49), handed over by request_started(). - -THE GAP IS OWNERSHIP, NOT LIVENESS. defer_teardown() drops the callbacks, which unpins -the GC-root/RefPtr cycle and makes the Request collectable. It does not CLOSE the fd: -the descriptor is released only by ~Request, or by the ReadStream inside -m_internal_stream_data -- which defer_teardown() merely nulls via a deferred_invoke. -So any OTHER surviving reference to the Requests::Request retains one fd per completed -request even though the teardown ran correctly. That is why the same patch zeroes the -leak on one tree and leaves 97/min on another: the trees differ in what else holds a -reference, not in what the teardown does. - -This closes the fd explicitly at the point the code has ALREADY PROVEN the body is -complete (the user_finish_called / EOF / has_received_all_reported_bytes branch), and -deregisters the notifier before closing so the event loop is never left polling a -closed descriptor. It deliberately does NOT close on request_finished alone: that -truncates bodies (verified once as blank pages), because RequestServer stops writing -long before WebContent drains the pipe. - -MEASURED, CMake build of f9e34731, same binary and same 200-completed-request workload, -only this function differing: - - clean tree: 208 sockets, 203 peer=DEAD - teardown fix + this: 5 sockets, 0 peer=DEAD (workload: SETTLE ok=200) - -Body delivery verified intact on the same binary, because a fix that removes body -delivery looks identical on fd counts: text=10 | stream=10/bytes=48000 | cancel=5 -(plain fetch, incremental ReadableStream reads, mid-body cancel()), plus a rendered -800x600 screenshot. - -The generalisable point: "the object is collectable" and "the fd is closed" are -different claims, and for a descriptor received over IPC only the second one is the -bug. A fix verified through the object graph can pass while the resource still leaks. -diff --git a/Libraries/LibRequests/Request.cpp b/Libraries/LibRequests/Request.cpp -index 7bac6c8..d45133b 100644 ---- a/Libraries/LibRequests/Request.cpp -+++ b/Libraries/LibRequests/Request.cpp -@@ -253,6 +253,10 @@ void Request::did_finish(Badge, u64 total_size, RequestTimingInfo - auto effective_network_error = m_body_delivery_error.has_value() ? m_body_delivery_error : network_error; - if (on_finish) - on_finish(total_size, timing_info, effective_network_error); -+ // A request that finished without stream data (stopped, or completed from cache -+ // before the callbacks were installed) has NOTHING owning the fd but ~Request. -+ if (!m_internal_stream_data) -+ release_response_fd(); - } - - void Request::did_receive_headers(Badge, NonnullRefPtr response_headers, Optional response_code, Optional const& reason_phrase, Optional javascript_bytecode, Optional javascript_bytecode_cache_vary_key, CameFromCache came_from_cache) -@@ -281,6 +285,34 @@ void Request::did_transfer(Badge) - on_stop(); - } - -+void Request::release_response_fd() -+{ -+ // Close the response pipe received from RequestServer at the moment its job is -+ // demonstrably over, instead of leaving it to the Request's destructor. -+ // -+ // The fd is one half of the socketpair RequestServer::RequestPipe::create() made -+ // and handed over with request_started(). WebContent otherwise releases it only -+ // when the Request itself dies (~Request, or the ReadStream inside -+ // m_internal_stream_data) -- so ANY other surviving reference to the Request pins -+ // one fd per COMPLETED request even though defer_teardown() ran correctly. -+ // Dropping the callbacks makes the request collectable; it does not close the fd. -+ // peer=DEAD in an `ss -np` census is exactly this: RequestServer already closed -+ // its end and is waiting for us. -+ // -+ // Deregister the notifier BEFORE closing, or the event loop polls a closed fd. -+ if (m_internal_stream_data) { -+ if (m_internal_stream_data->read_notifier) -+ m_internal_stream_data->read_notifier->close(); -+ m_internal_stream_data->read_stream = nullptr; -+ } -+ if (m_fd != -1) { -+ if (!m_fd_is_owned_by_read_stream) -+ (void)Core::System::close(m_fd); -+ m_fd = -1; -+ m_fd_is_owned_by_read_stream = false; -+ } -+} -+ - void Request::defer_teardown() - { - if (m_internal_stream_data && m_internal_stream_data->read_notifier) -@@ -331,6 +363,14 @@ void Request::set_up_internal_stream_data(DataReceived on_data_available) - // The body is fully delivered: drop the callbacks (and the GC::Roots they captured) - // and close the response fd, as stop()/did_transfer() do for the abandoned cases. - defer_teardown(); -+ -+ // ...and close the fd outright. defer_teardown() only nulls the owners it -+ // knows about; if anything else still references this Request the fd stays -+ // open. The guard above proves EOF or that every reported byte was -+ // delivered, so closing here cannot truncate a body. (Closing on -+ // request_finished alone DOES truncate -- blank pages -- because -+ // RequestServer stops writing long before WebContent drains the pipe.) -+ release_response_fd(); - } - }; - -diff --git a/Libraries/LibRequests/Request.h b/Libraries/LibRequests/Request.h -index b4046fa..cad0251 100644 ---- a/Libraries/LibRequests/Request.h -+++ b/Libraries/LibRequests/Request.h -@@ -133,6 +133,7 @@ private: - void attach_read_stream(); - void set_up_internal_stream_data(DataReceived on_data_available); - void defer_teardown(); -+ void release_response_fd(); - - WeakPtr m_client; - u64 m_request_id { 0 }; diff --git a/examples/ladybird/patches/0002-upstream-11041-release-response-pipes-when-fetches-are-canceled.effect-grep b/examples/ladybird/patches/0002-upstream-11041-release-response-pipes-when-fetches-are-canceled.effect-grep new file mode 100644 index 0000000..9d6f61e --- /dev/null +++ b/examples/ladybird/patches/0002-upstream-11041-release-response-pipes-when-fetches-are-canceled.effect-grep @@ -0,0 +1,18 @@ +# The EFFECT of upstream #11041 patch 2/3, over the file it touches +# (Libraries/LibWeb/Fetch/Infrastructure/FetchController.cpp). Checked only when +# the exact bytes are absent, so a tree where #11041 has MERGED verifies clean +# rather than being told to apply a patch that would conflict. +# +# The effect: abort() and terminate() must actually release the network request. +# stop_fetch() returns early once the state is aborted/terminated, so without an +# explicit release the response pipe stays open for the life of the process -- +# a whole leak class our fd census could not distinguish (it ranks by growth and +# splits peer=DEAD/ALIVE, which does not point at the CANCEL path specifically). +# +# Anchored on the helper's definition plus a call, rather than on a call alone: +# the name is the fix, and a definition with no caller would be the interesting +# broken state. No @window -- the two call sites are in different functions +# (abort, terminate) tens of lines apart, so a window would have to be so wide it +# proved nothing; requiring both the definition and the declaration is stronger. +void FetchController::stop_pending_request\(\) +stop_pending_request\(\); diff --git a/examples/ladybird/patches/0002-upstream-11041-release-response-pipes-when-fetches-are-canceled.patch b/examples/ladybird/patches/0002-upstream-11041-release-response-pipes-when-fetches-are-canceled.patch new file mode 100644 index 0000000..7177ee5 --- /dev/null +++ b/examples/ladybird/patches/0002-upstream-11041-release-response-pipes-when-fetches-are-canceled.patch @@ -0,0 +1,136 @@ +From 2ae78115aecaae054532d8a8ba93fecb4c8a9a3c Mon Sep 17 00:00:00 2001 +From: sideshowbarker +Date: Mon, 17 Aug 2026 07:04:13 +0900 +Subject: [PATCH 2/3] LibWeb: Release response pipes when fetches are canceled +MIME-Version: 1.0 +Content-Type: text/plain; charset=UTF-8 +Content-Transfer-Encoding: 8bit + +Problem: Aborting or terminating a fetch left its response pipe open for +the lifetime of the process. A page that starts and cancels requests in +a loop (an EventSource closed and reopened, a fetch or XHR aborted while +the response is still hanging) leaked one file descriptor per cycle — +until it ran out of them. + +Cause: abort() and terminate() only set the controller state. The one +path that releases the request, stop_request(), is reached only from +stop_fetch() — which returns early once the state is already “aborted” +or “terminated”. So the two entry points that mark a fetch as canceled +were the two that never told the network layer — and RequestServer holds +a request alive until it either finishes or is stopped. + +Fix: Release the request from abort() and terminate() too — through a +helper that stop_request() now shares. + +One caller relied on terminate() leaving the request alone. When a +navigation response becomes a download, the request is handed to the UI +process and the fetch is then terminated. That path now drops its handle +to the request before terminating — so a download already under way +isn’t stopped. RequestServer reports the transfer back to the +WebContent process — and that’s what closes the reader end of the pipe. +--- + .../LibWeb/Fetch/Infrastructure/FetchController.cpp | 13 +++++++++++++ + .../LibWeb/Fetch/Infrastructure/FetchController.h | 5 +++++ + Libraries/LibWeb/HTML/LocalNavigable.cpp | 7 ++++++- + 3 files changed, 24 insertions(+), 1 deletion(-) + +| UPSTREAM PATCH -- Ladybird PR #11041 by sideshowbarker, commit 2ae78115aecaae054532d8a8ba93fecb4c8a9a3c. +| NOT OURS. Carried only because our pin 71fb301a predates the merge; DELETE all +| three (and this note) on the first repin past it -- see the README patch table. +| +| Why upstream's and not the two patches this replaces: those were written here +| before #11041 existed, and one of them CRASHED Ulf's browser. Our 0002 +| (release_response_fd) nulled m_internal_stream_data->read_stream from inside the +| completion branch, while the read_notifier's on_activation lambda -- the frame +| that CALLS on_finish -- goes on to dereference that same OwnPtr: +| +| if (m_internal_stream_data->read_stream->is_eof()) <-- Request.cpp:376 +| m_internal_stream_data->read_notifier->close(); +| +| so the null landed on VERIFY(m_ptr) in AK::OwnPtr::operator-> (AK/OwnPtr.h:134), +| which is exactly the trace Ulf sent: CallableWrapper<...set_up_internal_stream_data +| ...{lambda#2}>::call() -> ak_verification_failed -> SIGILL. Upstream never nulls +| read_stream; it only ensures defer_teardown() is REACHED, and it reaches it BEFORE +| user_on_finish so the deferred lambda's NonnullRefPtr pins the Request across the +| callback. Our 0001 called it after, which is a second latent bug in the pair. +| +| The deeper point, recorded in docs/UPSTREAM-ladybird-fd-leaks.md: our 0002 existed +| to close the fd on the theory that a surviving reference pinned it. Upstream fixes +| the leak by running the teardown on all three paths -- so the theory was +| unnecessary, and carrying both would have been two mechanisms closing one fd, one +| of them justified by a theory the other disproves. + +diff --git a/Libraries/LibWeb/Fetch/Infrastructure/FetchController.cpp b/Libraries/LibWeb/Fetch/Infrastructure/FetchController.cpp +index 5e41472a1eb3b..6ba3de7c24e8c 100644 +--- a/Libraries/LibWeb/Fetch/Infrastructure/FetchController.cpp ++++ b/Libraries/LibWeb/Fetch/Infrastructure/FetchController.cpp +@@ -104,6 +104,10 @@ void FetchController::abort(JS::Realm& realm, Optional error) + : serialized_value_or_error.value(); + }; + m_serialized_abort_reason = structured_serialize(error.value(), fallback_error_value); ++ ++ // AD-HOC: stop_fetch() returns early once the state is "aborted". So, this is the last chance to release the ++ // network request. Without it the response pipe stays open for good. ++ stop_pending_request(); + } + + // https://fetch.spec.whatwg.org/#fetch-controller-terminate +@@ -111,6 +115,10 @@ void FetchController::terminate() + { + // To terminate a fetch controller controller, set controller’s state to "terminated". + m_state = State::Terminated; ++ ++ // AD-HOC: As in abort() above — stop_fetch() won’t release the request once the state is "terminated" — so, ++ // release it here. ++ stop_pending_request(); + } + + void FetchController::stop_fetch() +@@ -143,6 +151,11 @@ void FetchController::stop_fetch() + void FetchController::stop_request() + { + VERIFY(m_state == State::Stopped); ++ stop_pending_request(); ++} ++ ++void FetchController::stop_pending_request() ++{ + if (m_pending_request) { + m_pending_request->stop(); + m_pending_request = nullptr; +diff --git a/Libraries/LibWeb/Fetch/Infrastructure/FetchController.h b/Libraries/LibWeb/Fetch/Infrastructure/FetchController.h +index fb27df8f53e55..480a764277c0d 100644 +--- a/Libraries/LibWeb/Fetch/Infrastructure/FetchController.h ++++ b/Libraries/LibWeb/Fetch/Infrastructure/FetchController.h +@@ -86,6 +86,11 @@ class WEB_API FetchController : public JS::Cell { + // Null or a fetch timing info. + GC::Ptr m_full_timing_info; + ++ // Releases the network request behind this controller. Every way a fetch stops early has to reach this: The ++ // response pipe is a socket pair, and RequestServer holds the request alive until it either finishes or is stopped. ++ // So, a request that's abandoned without being stopped would keep its descriptor for the lifetime of the process. ++ void stop_pending_request(); ++ + // https://fetch.spec.whatwg.org/#fetch-controller-report-timing-steps + // report timing steps (default null) + // Null or an algorithm accepting a global object. +diff --git a/Libraries/LibWeb/HTML/LocalNavigable.cpp b/Libraries/LibWeb/HTML/LocalNavigable.cpp +index ff47e30dfa7c2..ee28f176bc547 100644 +--- a/Libraries/LibWeb/HTML/LocalNavigable.cpp ++++ b/Libraries/LibWeb/HTML/LocalNavigable.cpp +@@ -540,8 +540,13 @@ static bool handle_navigation_response_as_download(GC::Ref nav + return true; + } + +- if (navigation_params->fetch_controller) ++ if (navigation_params->fetch_controller) { ++ // AD-HOC: The request now belongs to the UI process (it adopted it above). So, before terminating, drop our ++ // handle to it. Otherwise, fetch termination would stop a download already under way. RequestServer ++ // reports the transfer back to us; that's what tears down this process's end of the pipe. ++ navigation_params->fetch_controller->set_pending_request(nullptr); + navigation_params->fetch_controller->terminate(); ++ } + + return true; + } + diff --git a/examples/ladybird/patches/0003-upstream-11041-tear-down-a-navigation-parked-for-content-sniffing.effect-grep b/examples/ladybird/patches/0003-upstream-11041-tear-down-a-navigation-parked-for-content-sniffing.effect-grep new file mode 100644 index 0000000..b7cc71a --- /dev/null +++ b/examples/ladybird/patches/0003-upstream-11041-tear-down-a-navigation-parked-for-content-sniffing.effect-grep @@ -0,0 +1,22 @@ +# The EFFECT of upstream #11041 patch 3/3, over the file it touches +# (Libraries/LibWeb/HTML/LocalNavigable.cpp). Checked only when the exact bytes +# are absent, so a tree where #11041 has MERGED verifies clean. +# +# The effect: a navigation parked for content sniffing -- headers received, but +# fewer bytes than the sniff threshold -- must have a teardown that runs when the +# navigable is destroyed, when a newer navigation supersedes it, or when it is +# destroyed already. Such a navigation has no document, so Document::abort() has +# no controller to stop, and ONLY the arrival callback would release the request +# -- which never runs. This was the open lead in +# docs/UPSTREAM-ladybird-fd-leaks.md that I could not reproduce: both of my +# workloads abandoned navigations without DESTROYING the navigable, so they +# exercised everything except the condition that matters. +# +# Anchored on the pair of accessors by name: the mechanism IS the registration +# plus the ownership check on clearing (only the navigation that registered a +# teardown may clear it, or a late sniff callback clears a newer navigation's +# teardown and leaks that one instead). A grep for run_pending_navigation_teardown +# alone would pass on a tree that stores the teardown and never runs it. +void LocalNavigable::set_pending_navigation_teardown +void LocalNavigable::clear_pending_navigation_teardown +run_pending_navigation_teardown\(\); diff --git a/examples/ladybird/patches/0003-upstream-11041-tear-down-a-navigation-parked-for-content-sniffing.patch b/examples/ladybird/patches/0003-upstream-11041-tear-down-a-navigation-parked-for-content-sniffing.patch new file mode 100644 index 0000000..51a66bd --- /dev/null +++ b/examples/ladybird/patches/0003-upstream-11041-tear-down-a-navigation-parked-for-content-sniffing.patch @@ -0,0 +1,168 @@ +From 8636af4103410833243e5eb987d64e8baf076b1a Mon Sep 17 00:00:00 2001 +From: sideshowbarker +Date: Mon, 17 Aug 2026 07:28:12 +0900 +Subject: [PATCH 3/3] LibWeb: Tear down a navigation parked for content + sniffing +MIME-Version: 1.0 +Content-Type: text/plain; charset=UTF-8 +Content-Transfer-Encoding: 8bit + +Problem: Removing an iframe whose response headers had arrived but whose +body hadn’t arrived left the network request open for the lifetime of +the process. A page that adds and removes such iframes in a loop leaked +one file descriptor per iframe. + +Cause: If a navigation has a response but not yet enough bytes to sniff +its content type, it gets parked in wait_for_sniff_bytes. It has no +document at that point — so, Document::abort() has no fetch controller +to stop, and the only code that releases the request is the arrival +callback. That callback does handle a destroyed navigable — but it runs +only once bytes arrive. So, a server that sends headers and then stops +leaves nothing at all around to release the request. + +Fix: Register a teardown on the navigable before parking — and run it if +the navigable is destroyed first. The teardown calls the same helper the +arrival callback already uses — and destroying a navigable is where the +rest of its in-flight navigation state is released. + +A navigation superseded before its populate task runs leaked the same +way, for the same reason: That guard returned without releasing +anything. It now releases the response as the guard above it does. +--- + Libraries/LibWeb/HTML/LocalNavigable.cpp | 50 +++++++++++++++++++++++- + Libraries/LibWeb/HTML/LocalNavigable.h | 6 +++ + 2 files changed, 55 insertions(+), 1 deletion(-) + +| UPSTREAM PATCH -- Ladybird PR #11041 by sideshowbarker, commit 8636af4103410833243e5eb987d64e8baf076b1a. +| NOT OURS. Carried only because our pin 71fb301a predates the merge; DELETE all +| three (and this note) on the first repin past it -- see the README patch table. +| +| Why upstream's and not the two patches this replaces: those were written here +| before #11041 existed, and one of them CRASHED Ulf's browser. Our 0002 +| (release_response_fd) nulled m_internal_stream_data->read_stream from inside the +| completion branch, while the read_notifier's on_activation lambda -- the frame +| that CALLS on_finish -- goes on to dereference that same OwnPtr: +| +| if (m_internal_stream_data->read_stream->is_eof()) <-- Request.cpp:376 +| m_internal_stream_data->read_notifier->close(); +| +| so the null landed on VERIFY(m_ptr) in AK::OwnPtr::operator-> (AK/OwnPtr.h:134), +| which is exactly the trace Ulf sent: CallableWrapper<...set_up_internal_stream_data +| ...{lambda#2}>::call() -> ak_verification_failed -> SIGILL. Upstream never nulls +| read_stream; it only ensures defer_teardown() is REACHED, and it reaches it BEFORE +| user_on_finish so the deferred lambda's NonnullRefPtr pins the Request across the +| callback. Our 0001 called it after, which is a second latent bug in the pair. +| +| The deeper point, recorded in docs/UPSTREAM-ladybird-fd-leaks.md: our 0002 existed +| to close the fd on the theory that a surviving reference pinned it. Upstream fixes +| the leak by running the teardown on all three paths -- so the theory was +| unnecessary, and carrying both would have been two mechanisms closing one fd, one +| of them justified by a theory the other disproves. + +diff --git a/Libraries/LibWeb/HTML/LocalNavigable.cpp b/Libraries/LibWeb/HTML/LocalNavigable.cpp +index ee28f176bc547..c908afbcee8df 100644 +--- a/Libraries/LibWeb/HTML/LocalNavigable.cpp ++++ b/Libraries/LibWeb/HTML/LocalNavigable.cpp +@@ -681,6 +681,40 @@ void LocalNavigable::set_has_been_destroyed() + m_has_been_destroyed = true; + resolve_all_pending_async_scroll_operations(); + cancel_user_scroll_settlement(); ++ run_pending_navigation_teardown(); ++} ++ ++void LocalNavigable::set_pending_navigation_teardown(GC::Ptr> teardown) ++{ ++ // A navigation parking here while another is already parked supersedes it. So, tear the older one down — rather ++ // than dropping it and leaving its request open. ++ run_pending_navigation_teardown(); ++ ++ // Destruction has already run, so nothing would ever run this one. Tear it down now — instead of storing it forever. ++ if (m_has_been_destroyed) { ++ if (teardown) ++ teardown->function()(); ++ return; ++ } ++ ++ m_pending_navigation_teardown = teardown; ++} ++ ++void LocalNavigable::clear_pending_navigation_teardown(GC::Ptr> expected) ++{ ++ // Only the navigation that registered this teardown may clear it. A navigation whose sniff-byte callback ++ // arrives after a newer one has parked would otherwise clear the newer teardown — and leave that ++ // newer request with nothing to release it. ++ if (m_pending_navigation_teardown != expected) ++ return; ++ ++ m_pending_navigation_teardown = nullptr; ++} ++ ++void LocalNavigable::run_pending_navigation_teardown() ++{ ++ if (auto teardown = exchange(m_pending_navigation_teardown, nullptr)) ++ teardown->function()(); + } + + void LocalNavigable::remove_from_all_local_navigables() +@@ -711,6 +745,7 @@ void LocalNavigable::visit_edges(Cell::Visitor& visitor) + visitor.visit(m_active_document); + visitor.visit(m_input_method_composition_node); + visitor.visit(m_container); ++ visitor.visit(m_pending_navigation_teardown); + m_event_handler.visit_edges(visitor); + + for (auto& navigation_params : m_pending_navigations) { +@@ -2190,6 +2225,9 @@ void LocalNavigable::populate_session_history_entry_document( + + // 1. If navigable's ongoing navigation no longer equals navigationId, then run completionSteps and abort these steps. + if (navigation_id.has_value() && ongoing_navigation() != navigation_id) { ++ // AD-HOC: Nothing downstream will consume this response, and no document exists yet to abort — so, ++ // release its request here, as the active-window guard above does. ++ stop_or_resume_response_body_delivery(navigation_params); + if (completion_steps) { + completion_steps->function()(nullptr); + } +@@ -2303,8 +2341,18 @@ void LocalNavigable::populate_session_history_entry_document( + if (!sniff_bytes.has_value()) { + // Async path: bytes not yet available, wait for them + nav_params->response->resume_body_delivery_up_to(Fetch::Infrastructure::MAX_SNIFF_BYTES); ++ ++ // AD-HOC: The callback below runs only once bytes arrive — which never happens if the server sends ++ // headers and then stops. Hand the navigable a teardown — so that destroying it releases ++ // the request, instead of leaving the pipe open. ++ auto teardown = GC::create_function(heap(), [navigation_params] { ++ stop_or_resume_response_body_delivery(navigation_params); ++ }); ++ nav_params->navigable->set_pending_navigation_teardown(teardown); ++ + body->wait_for_sniff_bytes(GC::create_function(heap(), +- [output, nav_params, navigation_params, completion_steps, source_snapshot_params](ReadonlyBytes sniff_bytes) { ++ [output, nav_params, navigation_params, completion_steps, source_snapshot_params, teardown](ReadonlyBytes sniff_bytes) { ++ nav_params->navigable->clear_pending_navigation_teardown(teardown); + // AD-HOC: The document may have been destroyed between when the fetch started and when the + // bytes arrived. + if (nav_params->navigable->active_browsing_context()) { +diff --git a/Libraries/LibWeb/HTML/LocalNavigable.h b/Libraries/LibWeb/HTML/LocalNavigable.h +index 8ebc9e99660ae..879da0f94f09c 100644 +--- a/Libraries/LibWeb/HTML/LocalNavigable.h ++++ b/Libraries/LibWeb/HTML/LocalNavigable.h +@@ -95,6 +95,10 @@ class WEB_API LocalNavigable : public Navigable { + void set_navigation_load_event_guard(DOM::Document& parent_doc); + void clear_navigation_load_event_guard(); + ++ void set_pending_navigation_teardown(GC::Ptr>); ++ void clear_pending_navigation_teardown(GC::Ptr> expected); ++ void run_pending_navigation_teardown(); ++ + RefPtr active_session_history_entry() const; + void set_active_session_history_entry(RefPtr); + RefPtr current_session_history_entry() const; +@@ -406,6 +410,8 @@ class WEB_API LocalNavigable : public Navigable { + // AD-HOC: Guards the parent document's load event delay count during cross-document navigation. + Optional m_navigation_load_event_guard; + ++ GC::Ptr> m_pending_navigation_teardown; ++ + // Implied link between navigable and its container. + GC::Ptr m_container; + diff --git a/tests/test_apply_overlay.py b/tests/test_apply_overlay.py index 3f81bd0..0e7f208 100644 --- a/tests/test_apply_overlay.py +++ b/tests/test_apply_overlay.py @@ -196,11 +196,19 @@ def test_effect_grep_files_exist_for_patches_upstream_may_fix_itself(): """ text = _text() assert ".effect-grep" in text, "verify must fall back to an effect check" - fd_leak = PATCHES / "0001-librequests-tear-down-request-when-body-is-delivered.patch" - assert fd_leak.exists() - effect = fd_leak.with_suffix(".effect-grep") - assert effect.exists(), \ - "the fd-leak patch is the one upstream is fixing; it needs an effect check" + # EVERY patch in the series needs one, not one named patch: the series is now + # upstream's three #11041 commits, and the whole point of carrying upstream's + # own commits is that they WILL appear in a future tree by merge rather than by + # us applying them. Naming a file here is also how this test would rot -- it + # referenced the two patches that #11041 replaced, both of which are gone. + patches = sorted(PATCHES.glob("0*.patch")) + assert patches, "no patches found" + for patch in patches: + effect = patch.with_suffix(".effect-grep") + assert effect.exists(), ( + f"{patch.name} has no .effect-grep: on a tree where upstream's fix has " + "merged, --verify would report PATCH NOT APPLIED and tell the reader to " + "apply a patch that then conflicts (exactly what happened to Ulf)") # reverse-apply must still be tried FIRST, so the exact-bytes case keeps its # precise answer and the weaker check is only a fallback assert text.index("git apply --check -R") < text.index(".effect-grep") @@ -215,15 +223,26 @@ def test_effect_grep_is_windowed_not_a_whole_file_grep(): being too strict. The check therefore anchors on the branch condition and requires the call within a window of following lines. """ - effect = (PATCHES / "0001-librequests-tear-down-request-when-body-is-delivered.effect-grep") - lines = [ln for ln in effect.read_text().splitlines() - if ln.strip() and not ln.startswith("#")] - assert any(ln.startswith("@window ") for ln in lines), \ - "the effect check must be windowed; a whole-file grep accepts an unpatched tree" - window = [ln for ln in lines if ln.startswith("@window ")][0].split() - assert window[1].isdigit() and int(window[1]) < 40, \ - "the window must be tight enough to mean 'in this branch'" - assert "$PATCHES" not in effect.read_text() + # The completion-branch patch is the one whose effect a whole-file grep cannot + # check, because defer_teardown() already appears in stop() and did_transfer(). + # Found by CONTENT (the call it must place in that branch), not by filename. + windowed = [] + for effect in sorted(PATCHES.glob("*.effect-grep")): + body = effect.read_text() + if "defer_teardown" not in body: + continue + windowed.append(effect) + lines = [ln for ln in body.splitlines() + if ln.strip() and not ln.startswith("#")] + assert any(ln.startswith("@window ") for ln in lines), ( + f"{effect.name} greps for defer_teardown() without a window; that call " + "already exists in stop()/did_transfer(), so it accepts an unpatched tree") + window = [ln for ln in lines if ln.startswith("@window ")][0].split() + assert window[1].isdigit() and int(window[1]) < 40, \ + "the window must be tight enough to mean 'in this branch'" + assert "$PATCHES" not in body + assert windowed, \ + "no effect check covers the completion-branch teardown, the fd leak's core fix" # and the script must implement the window, not just tolerate the directive text = _text() assert "@window " in text @@ -272,57 +291,56 @@ def test_the_documented_overlay_file_count_matches_the_overlay(): sorted(claims), actual) -def test_the_response_fd_patch_closes_the_fd_where_the_body_is_proven_complete(): - """The second half of the fd leak: the fd outlives the callbacks. +def test_the_fd_leak_patches_are_upstreams_and_never_null_the_read_stream(): + """The overlay carries upstream #11041, and must not carry my version again. + + My `0002` (release_response_fd) CRASHED Ulf's browser after a few minutes of + real browsing, and the trace named the line: + + VERIFICATION FAILED: m_ptr at ./AK/OwnPtr.h:134 + #0 ...CallableWrapper<...set_up_internal_stream_data(...)::{lambda()#2}>::call() - 0001 drops the callbacks on completion, which unpins the GC cycle -- and on my - machine that took 143 leaked sockets to 0. On Ulf's tree it did not: he measured - 97 sockets/min STILL leaking with the upstream-equivalent teardown applied, - every one peer=DEAD (RequestServer had already closed its end) and every one - sent by RequestServer (SO_PEERCRED, once `ps` named pid 2261433). + That lambda is the read notifier's on_activation -- the frame that CALLS + on_finish. My patch nulled m_internal_stream_data->read_stream from inside the + completion branch, while the calling frame goes on to dereference exactly that + OwnPtr at Request.cpp:376 (`read_stream->is_eof()`), and OwnPtr::operator-> is + VERIFY(m_ptr). A use-after-null one stack frame up: invisible on every workload + I built, a crash on his. - The reason is ownership: the response fd is released only by ~Request (or by the - ReadStream inside m_internal_stream_data), so ANY surviving reference to the - Request keeps one fd per completed request alive -- dropping the callbacks is - not the same as closing the fd. 0002 closes it explicitly at the point the code - has already proven the body is complete. + Upstream's fix never touches read_stream -- it only ensures defer_teardown() is + REACHED, on all three paths where it could be missed, and reaches it BEFORE + user_on_finish so the deferred lambda's NonnullRefPtr pins the Request across + the callback (my 0001 called it after: a second latent use-after-free). - Measured A/B, same binary, same 200-completed-request workload: - clean: 208 sockets, 203 peer=DEAD - fixed: 6 sockets, 0 peer=DEAD + So this asserts the property, not the filenames: no patch in the series may null + read_stream, and the series must be upstream's. A future "optimisation" that + reintroduces the defensive close fails here. """ - patch = PATCHES / "0002-requests-release-response-fd-on-completion.patch" - assert patch.exists() - text = patch.read_text() - # 0002 is the NEXT PATCH IN THE SERIES, not an alternative to 0001: it is - # generated on top of it and applies only after it. I first shipped a second - # "clean tree" variant for trees without the teardown fix, which cannot work -- - # apply_overlay.sh globs patches/*.patch and applies them all, so one of the two - # was guaranteed to fail. Ulf hit it on the first run. The variant is deleted; it - # was also strictly weaker (it closed the fd without dropping the callbacks, so it - # left the GC cycle, and with it class B, in place). - assert not list(PATCHES.glob("*clean-tree*")), \ - "an alternative in patches/*.patch cannot coexist with the glob that applies all" - assert "ON TOP OF the teardown fix" in text, \ - "0002 must state that it builds on 0001 rather than replacing it" - # the notifier must be deregistered before the fd is closed, or the event loop - # is left polling a closed descriptor - body = text[text.index("release_response_fd"):] - assert body.index("read_notifier->close()") < body.index("m_fd = -1"), \ - "deregister the notifier BEFORE closing the fd" - # it must NOT close on request_finished alone: RequestServer finishes writing - # long before WebContent drains the pipe, and closing there truncates bodies - # (verified once as blank pages) - assert "has_received_all_reported_bytes" in text or "user_finish_called" in text - effect = patch.with_suffix(".effect-grep") - assert effect.exists(), "carried until upstream fixes it -> needs an effect check" - lines = [ln for ln in effect.read_text().splitlines() - if ln.strip() and not ln.startswith("#")] - assert any(ln.startswith("@window ") for ln in lines) - # apply_overlay.sh takes only the FIRST @window; a second would be checked - # against the wrong slice, so the file must not carry one. - assert len([ln for ln in lines if ln.startswith("@window ")]) == 1, \ - "apply_overlay.sh honours one @window only" + patches = sorted(PATCHES.glob("0*.patch")) + assert len(patches) == 3, \ + "expected upstream #11041's three commits; found %r" % [p.name for p in patches] + for patch in patches: + text = patch.read_text() + # The added lines only: upstream's patch 1 QUOTES the surrounding code as + # context, and a context line is not something the patch does. + added = "\n".join(l[1:] for l in text.splitlines() + if l.startswith("+") and not l.startswith("+++")) + assert "read_stream = nullptr" not in added, ( + f"{patch.name} nulls read_stream, which the read-notifier lambda that " + "calls on_finish still dereferences (AK/OwnPtr.h:134 VERIFY) -- this is " + "the crash Ulf hit; upstream fixes the leak by reaching the teardown " + "instead") + assert "release_response_fd" not in added, ( + f"{patch.name} reintroduces release_response_fd: it closed the fd on the " + "theory that a surviving reference pinned it, which upstream's fix " + "disproves by making the teardown reachable. Two mechanisms closing one " + "descriptor, one justified by a theory the other falsifies, misleads the " + "next reader") + assert "PR #11041" in text or "11041" in text, \ + f"{patch.name} does not record its upstream provenance" + assert "DELETE all" in text or "DELETE" in text, ( + f"{patch.name} does not say it is a pin artefact to delete on the repin " + "past the merge -- that is how a carried patch becomes a permanent fork") def test_the_patch_series_applies_as_a_series(): From 11ad08076bac7487622c1e9a0ee4f81a45991829 Mon Sep 17 00:00:00 2001 From: Ulf Adams Date: Thu, 20 Aug 2026 14:00:00 +0000 Subject: [PATCH 59/59] Keep the fd-leak workload as a file, not a thing retyped in /tmp I built this page in /tmp to verify upstream #11041 on my tree, and /tmp does not survive a sandbox restart. It is the workload that exercises two of the three paths #11041 fixes -- ordinary completion (150 settled fetches, the branch my own patch got wrong) and cancellation (50 aborts, the class my fd census could not name) -- so it is worth more than the five minutes it took to write. The header records what it does NOT cover and why that matters: the third path needs a navigable DESTROYED while parked for content sniffing, which is exactly the condition my two earlier workloads missed. They came out flat and I nearly read that as evidence against the hypothesis instead of evidence about the workload. Also records the two harness facts that cost me time just now: it needs a real HTTP origin, and --headless=text snapshots before the async loop finishes (so it reports nothing useful -- use --headless=manual plus fd_census.py --watch and read the RATE, not the level). --- examples/ladybird/fdleak_workload.html | 74 ++++++++++++++++++++++++++ 1 file changed, 74 insertions(+) create mode 100644 examples/ladybird/fdleak_workload.html diff --git a/examples/ladybird/fdleak_workload.html b/examples/ladybird/fdleak_workload.html new file mode 100644 index 0000000..2791ebd --- /dev/null +++ b/examples/ladybird/fdleak_workload.html @@ -0,0 +1,74 @@ + + +

fdleak workload

starting
+