diff --git a/.gitignore b/.gitignore index 78015339d..cf89e2fc3 100644 --- a/.gitignore +++ b/.gitignore @@ -23,6 +23,8 @@ local_tests/java/out/ local_tests/java/.gradle/ bottlecap/target bottlecap/proptest-regressions +# scripts/dev/worktree.sh default location +.worktrees integration-tests/cdk.context.json diff --git a/AGENTS.md b/AGENTS.md index 102c32899..d67f6de05 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -10,6 +10,15 @@ The Datadog Lambda Extension ("Bottlecap") is an AWS Lambda Extension written in All Rust code lives under `bottlecap/`. Run these from that directory unless noted. +## Build and test fast + +Opt-in shared compile cache (sccache plus one shared `CARGO_TARGET_DIR` for all worktrees). Details, measurements and the other `scripts/dev/` helpers: `scripts/dev/README.md`. + +- From the repo root, in the same shell call as cargo (agent shells keep no env between calls): `eval "$(scripts/dev/setup-build-cache.sh)" && cd bottlecap && cargo build`. The first run may download sccache. +- Tests that spawn the `bottlecap` binary, and `cargo clippy`, need a per-worktree target dir: use `setup-build-cache.sh --tests` instead. +- Builds from several worktrees queue on cargo's lock (`Blocking waiting for file lock`). Do not kill them. Never `cargo clean` the shared dir. +- Run the tests the way CI does: `cargo nextest run`. + ## Where New Code Goes The `serverless-components` repo (https://github.com/DataDog/serverless-components) holds Rust crates that this extension and the Azure and Google Cloud Platform (GCP) serverless agents share, for example `dogstatsd`, `datadog-fips` and `datadog-agent-config`. diff --git a/scripts/dev/README.md b/scripts/dev/README.md new file mode 100644 index 000000000..7ddf88fff --- /dev/null +++ b/scripts/dev/README.md @@ -0,0 +1,161 @@ +# Developer scripts + +Opt-in helpers to build and test bottlecap fast, for humans and coding agents. +None of them edits shell rc files, `~/.cargo/config.toml` or Docker settings. + +| Script | Purpose | +|--------|---------| +| `setup-build-cache.sh` | Install or detect `sccache`, print the env vars for the shared compile cache | +| `worktree.sh` | Create a git worktree that shares the compile cache | + +## setup-build-cache.sh + +```sh +cd bottlecap +eval "$(../scripts/dev/setup-build-cache.sh)" +cargo build +``` + +The script installs `sccache` 0.19.0 (prebuilt release, sha256 checked; falls +back to `cargo install`) into `~/.cache/bottlecap/tools/bin` if it is not on +`PATH`, then prints five `export` lines: + +| Variable | Default | Why | +|----------|---------|-----| +| `RUSTC_WRAPPER` | path to `sccache` | cache every `rustc` call | +| `SCCACHE_DIR` | `~/.cache/bottlecap/sccache` | object store, shared by every clone and worktree | +| `SCCACHE_CACHE_SIZE` | `20G` | size limit of the store | +| `CARGO_TARGET_DIR` | `~/.cache/bottlecap/target` | one target dir for every clone and worktree | +| `RUSTC_WORKSPACE_WRAPPER` | this worktree's `rustc-workspace-wrapper.sh` | cargo hashes its path into bottlecap's own artifacts, so each worktree keeps its own copy and only dependencies are shared | + +Override any of them except `RUSTC_WORKSPACE_WRAPPER` in the environment before you run the script. An inherited +`CARGO_TARGET_DIR` is kept, with a note. On an error the script prints `false`, +so `eval` fails instead of exporting nothing. + +| Option | Effect | +|--------|--------| +| `--tests` | Same exports, but `CARGO_TARGET_DIR=/bottlecap/target`. Use it for tests that spawn the `bottlecap` binary (`CARGO_BIN_EXE_bottlecap`, the process-level harness): `target/debug/bottlecap` in the shared dir is overwritten by whichever worktree built last. This dir gets no Rust cache hits (see below), so expect one cold build per worktree. | +| `--no-wrapper` | Shared `CARGO_TARGET_DIR` only, `export RUSTC_WRAPPER=` (empty also overrides a `build.rustc-wrapper` in cargo config). | +| `--check` | Print what would be exported. No install, no directories, no server query. | +| `--stats` / `--stop` | `sccache --show-stats` / `sccache --stop-server`. | + +The prebuilt download is verified against sha256 values pinned in the script +for v0.19.0 (four targets). With another `SCCACHE_VERSION` the script uses the +release's own `.sha256` file, which only proves integrity, not origin. + +### Why one shared `CARGO_TARGET_DIR` + +sccache keys a Rust compile on the full `rustc` command line, and cargo passes +`-L dependency=/debug/deps`. A different target dir path is a +different key. + +Measured on a 16-core Apple Silicon Mac with other builds running (load 5 to 17): + +| Case | `cargo build` | then `cargo test --no-run` | +|------|---------------|----------------------------| +| Cold, no cache | 57.6 s | +29.9 s | +| Warm sccache, new target dir path | 65.6 s (0 Rust hits, 463 C/asm hits) | +31.8 s | +| Warm sccache, same path, target dir deleted | 57.3 s (827 of 827 hits) | +22.0 s (151 of 151 hits) | +| Shared target dir, second worktree, first build | 15.5 s | +16.7 s | +| No-op rebuild | 0.8 s | | + +The shared target dir is what saves time. Cargo reuses the 360 dependency +artifacts through its own fingerprints, so a new worktree compiles only the +`bottlecap` crate. sccache alone does not shorten a cold build on a 16-core +machine, because the critical path is work it cannot cache +(`cargo build --timings`, run under load): the `tikv-jemalloc-sys` build script +(`configure` + `make`, 59 s), the `aws-lc-sys` build script (cmake, 21 s; +pulled in by the `reqwest` build-dependency of `libddwaf-sys`, not linked into +the binary), `libddwaf-sys` (10 s) and `zstd-sys` (9 s) build scripts, and then +the `bottlecap` crate itself (42 s + 11 s for the bin). sccache still pays off +when CPU is scarce (CI, the 4-CPU Docker VM, or 12 agents building at once): with +100 % hits the Rust dependency compiles cost almost no CPU. The `cc` crate also +routes C compiles through sccache when `RUSTC_WRAPPER` ends in `sccache`, which +is where the C/C++ hits come from. + +### Lock contention + +Cargo takes a whole-build lock per profile dir (`target/debug/.cargo-lock`). +Two worktrees that build at the same time queue; debug and release builds do +not block each other. + +| Two worktrees touch `src/lib.rs` and build at once | Wall time | +|---|---| +| Shared target dir | 30.1 s (15.3 s + 29.5 s, the second waits on the lock) | +| One target dir per worktree | 9.0 s (8.5 s and 5.0 s in parallel) | +| One incremental build alone, equal load | shared 6.1 s, own 6.5 s | + +Recommendation: share the target dir. The lock costs at most one queued build +(about 15 s), and saves about 60 s and 3 to 5 GB per worktree. Fifteen +per-worktree target dirs held 67 GB; the shared dir with debug +and test artifacts of two worktrees holds 7.5 GB. + +Rules for the shared dir: + +- Never run `cargo clean` there. It deletes every worktree's artifacts. +- `target/debug/bottlecap` is overwritten by whichever worktree built last. + Use `cargo run --bin bottlecap`, or read the path from + `cargo build --message-format=json | jq -r 'select(.executable != null).executable'`. + Tests that spawn that binary (`CARGO_BIN_EXE_bottlecap`) must run with + `setup-build-cache.sh --tests`. Unit test and bench binaries in + `target/debug/deps/` carry a per-worktree hash (from `RUSTC_WORKSPACE_WRAPPER`) + and do not collide. Run the `eval` in the worktree you build in. sccache + still caches dependencies; it runs bottlecap's own compiles uncached. +- `cargo clippy` sets its own `RUSTC_WORKSPACE_WRAPPER`, so in the shared dir it + can replay another worktree's result. Run clippy with `setup-build-cache.sh --tests`. +- `target/criterion/` is shared. Use `cargo bench -- --save-baseline `. +- Builds that wait on the lock print `Blocking waiting for file lock on build directory`. + Give agent shells a long timeout or run builds in the background. +- The sccache wrapper adds about 2 s to an incremental build (6.3 s vs 4.4 s). + For a long edit-compile loop in one worktree, `export RUSTC_WRAPPER=` and keep `CARGO_TARGET_DIR`. +- The sccache server reads `SCCACHE_DIR` and `SCCACHE_CACHE_SIZE` once, at start. + A client with another `SCCACHE_DIR` silently uses the running server. The script warns about that. + +## worktree.sh + +```sh +scripts/dev/worktree.sh jordan.gonzalez/logs/fix # from origin/main (or main) +scripts/dev/worktree.sh jordan.gonzalez/logs/fix v89 # from a tag or commit +scripts/dev/worktree.sh --list +scripts/dev/worktree.sh --remove jordan.gonzalez/logs/fix # keeps the branch +``` + +Worktrees go to `
/.worktrees/` (override with +`WORKTREE_ROOT`). The directory is in `.gitignore`. A branch that exists only +on `origin` is checked out with tracking. The script prints the exact build +commands for the new worktree. All worktrees build into the shared +`CARGO_TARGET_DIR`; see the measurements above for why there is no per-worktree +option, and `setup-build-cache.sh --tests` for the one case that needs one. + +## CI release binaries per commit + +GitLab compiles each flavor in the `bottlecap ()` job +(`.gitlab/templates/pipeline.yaml.tpl`) and keeps the stripped binary +`.binaries/bottlecap-` for one week. On `main` a second job, +`bottlecap symbols ()`, runs for `amd64` and `arm64` +(`keep_symbols: true` in `.gitlab/datasources/flavors.yaml`). It compiles the +same release profile with `CARGO_PROFILE_RELEASE_STRIP=false` and publishes, +for 90 days, under the artifact name `bottlecap--`: + +- `.binaries/bottlecap-` (the shipped stripped binary, copied from the compile job) +- `.binaries/bottlecap--unstripped` (same compile, symbol table kept; never shipped) +- `.binaries/bottlecap-.build-info` (commit, ref, pipeline and job URLs) + +The job also runs on tags and is manual on other branches. Every rule has +`allow_failure: true` and the retry covers only runner and timeout failures, +so a broken symbols build never turns `main` red or blocks a release. The +shipped layer does not change. + +Download the latest `main` build (needs a GitLab token with `read_api`): + +```sh +API=https://gitlab.ddbuild.io/api/v4/projects/DataDog%2Fdatadog-lambda-extension +JOB='bottlecap symbols (arm64)' +curl -fsSL -H "PRIVATE-TOKEN: $GITLAB_TOKEN" -o bottlecap-arm64-unstripped \ + "$API/jobs/artifacts/main/raw/.binaries/bottlecap-arm64-unstripped?job=$(printf %s "$JOB" | jq -sRr @uri)" +``` + +For a given commit, list its pipeline jobs and download the job's artifact zip +(`$API/pipelines?sha=`, then `$API/pipelines//jobs`, then +`$API/jobs//artifacts`). The job log also prints the raw download URL +of the unstripped binary. diff --git a/scripts/dev/rustc-workspace-wrapper.sh b/scripts/dev/rustc-workspace-wrapper.sh new file mode 100755 index 000000000..4664c0844 --- /dev/null +++ b/scripts/dev/rustc-workspace-wrapper.sh @@ -0,0 +1,8 @@ +#!/bin/sh +# Exported as RUSTC_WORKSPACE_WRAPPER by setup-build-cache.sh. Cargo mixes this +# file's path into the artifact hash of workspace members, so each worktree gets +# its own bottlecap artifacts in the shared target dir. Runs rustc unchanged. +case "${1:-}" in + -*) exec "${RUSTC:-rustc}" "$@" ;; # sccache probes the wrapper itself with `-vV` +esac +exec "$@" diff --git a/scripts/dev/setup-build-cache.sh b/scripts/dev/setup-build-cache.sh new file mode 100755 index 000000000..715f3f7a7 --- /dev/null +++ b/scripts/dev/setup-build-cache.sh @@ -0,0 +1,218 @@ +#!/usr/bin/env bash +# Opt-in shared compile cache for bottlecap builds (sccache + one shared target dir). +# +# This script never edits shell rc files or ~/.cargo/config.toml. It prints +# `export` lines on stdout. Load them into the current shell with: +# +# eval "$(scripts/dev/setup-build-cache.sh)" +# +# Other commands: +# scripts/dev/setup-build-cache.sh --tests # same, but a per-worktree target dir (tests that spawn the bottlecap binary) +# scripts/dev/setup-build-cache.sh --no-wrapper # shared CARGO_TARGET_DIR only, no sccache +# scripts/dev/setup-build-cache.sh --check # print what would be exported; no install, no dirs, no server query +# scripts/dev/setup-build-cache.sh --stats # sccache hit/miss stats +# scripts/dev/setup-build-cache.sh --stop # stop the sccache server +# +# Environment (all optional): +# BOTTLECAP_CACHE_ROOT root for cache and tools (default: $HOME/.cache/bottlecap) +# SCCACHE_DIR sccache object store (default: $BOTTLECAP_CACHE_ROOT/sccache) +# SCCACHE_CACHE_SIZE sccache size limit (default: 20G) +# CARGO_TARGET_DIR shared cargo target dir (default: $BOTTLECAP_CACHE_ROOT/target) +# SCCACHE_VERSION version to install (default: 0.19.0, sha256 pinned below) +# SCCACHE_INSTALL prebuilt | cargo | none (default: prebuilt, then cargo) +# +# Why one shared CARGO_TARGET_DIR: sccache hashes the full rustc command line, +# and cargo passes `-L dependency=/debug/deps`. A different +# target dir path gives 0 % Rust cache hits. See scripts/dev/README.md. +# RUSTC_WORKSPACE_WRAPPER points at this worktree's rustc-workspace-wrapper.sh, +# so bottlecap's own artifacts stay per worktree and only dependencies are shared. +set -euo pipefail + +log() { printf '%s\n' "setup-build-cache: $*" >&2; } +die() { log "error: $*"; exit 1; } + +TMP="" +# On any failure, `false` on stdout makes `eval "$(...)"` fail instead of exporting nothing. +# shellcheck disable=SC2317,SC2329 # invoked by the trap +cleanup() { + local rc=$? + [ -z "$TMP" ] || rm -rf "$TMP" + [ "$rc" -eq 0 ] || printf 'false\n' +} +trap cleanup EXIT + +# Exported paths must not depend on the directory cargo later runs from. +abspath() { case "$1" in /*) printf '%s' "$1" ;; *) printf '%s/%s' "$PWD" "$1" ;; esac; } +BOTTLECAP_CACHE_ROOT="$(abspath "${BOTTLECAP_CACHE_ROOT:-$HOME/.cache/bottlecap}")" +DEFAULT_TARGET_DIR="$BOTTLECAP_CACHE_ROOT/target" +SCCACHE_DIR="$(abspath "${SCCACHE_DIR:-$BOTTLECAP_CACHE_ROOT/sccache}")" +SCCACHE_CACHE_SIZE="${SCCACHE_CACHE_SIZE:-20G}" +CARGO_TARGET_DIR="$(abspath "${CARGO_TARGET_DIR:-$DEFAULT_TARGET_DIR}")" +SCCACHE_VERSION="${SCCACHE_VERSION:-0.19.0}" +SCCACHE_INSTALL="${SCCACHE_INSTALL:-prebuilt}" +TOOLS_DIR="$BOTTLECAP_CACHE_ROOT/tools" +WORKSPACE_WRAPPER="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd -P)/rustc-workspace-wrapper.sh" +PINNED_VERSION="0.19.0" +mode="export" +wrapper=1 +target_mode="shared" + +case "${1:-}" in + "") ;; + --tests) + repo_root="$(git -C "$(dirname "${BASH_SOURCE[0]}")" rev-parse --show-toplevel 2>/dev/null)" || die "--tests must run inside the git checkout" + CARGO_TARGET_DIR="$repo_root/bottlecap/target" + target_mode="per-worktree" ;; + --no-wrapper) wrapper=0 ;; + --check) mode="check" ;; + --stats) mode="stats" ;; + --stop) mode="stop" ;; + -h|--help) sed -n '2,28p' "$0" | sed 's/^# \{0,1\}//'; exit 0 ;; + *) die "unknown option: $1 (see --help)" ;; +esac + +if [ "$target_mode" = "shared" ] && [ "$CARGO_TARGET_DIR" != "$DEFAULT_TARGET_DIR" ]; then + log "note: CARGO_TARGET_DIR was already set to $CARGO_TARGET_DIR; keeping it. Other worktrees share this cache only with the same path." +fi + +sha256_file() { + if command -v sha256sum >/dev/null 2>&1; then sha256sum "$1" | cut -d' ' -f1 + else shasum -a 256 "$1" | cut -d' ' -f1; fi +} + +# Release checksums of sccache v0.19.0 (https://github.com/mozilla/sccache/releases/tag/v0.19.0). +pinned_sha256() { + case "$1" in + aarch64-apple-darwin) echo a35a0b7d622618774bc1bb8102d72562dfb21e619f88b153b370d7f89bc286af ;; + x86_64-apple-darwin) echo 2db7344bb96de350df79cd9f0124dd4e9baa47b3de2fac1e63e1346b66b20684 ;; + x86_64-unknown-linux-musl) echo aa73c7270c997894038c0d8f0d297865f57569e1687883322bc431713d4bb9dd ;; + aarch64-unknown-linux-musl) echo 3e47fc4a71b1ddd1b4294a0de9d34498922fdf41e3cd0bda5df3954e3d6deb9d ;; + *) return 1 ;; + esac +} + +prebuilt_triple() { + case "$(uname -s)-$(uname -m)" in + Darwin-arm64) echo aarch64-apple-darwin ;; + Darwin-x86_64) echo x86_64-apple-darwin ;; + Linux-x86_64) echo x86_64-unknown-linux-musl ;; + Linux-aarch64|Linux-arm64) echo aarch64-unknown-linux-musl ;; + *) return 1 ;; + esac +} + +install_prebuilt() { + local triple base url want have + if ! triple="$(prebuilt_triple)"; then log "no prebuilt sccache for $(uname -s)-$(uname -m)"; return 1; fi + base="sccache-v${SCCACHE_VERSION}-${triple}" + url="https://github.com/mozilla/sccache/releases/download/v${SCCACHE_VERSION}/${base}.tar.gz" + TMP="$(mktemp -d)" || return 1 + log "downloading $url" + if ! curl -fsSL --retry 3 -o "$TMP/$base.tar.gz" "$url"; then log "download failed"; return 1; fi + if [ "$SCCACHE_VERSION" = "$PINNED_VERSION" ]; then + want="$(pinned_sha256 "$triple")" + else + log "SCCACHE_VERSION=$SCCACHE_VERSION is not the pinned $PINNED_VERSION; using the release's own .sha256 (integrity only)" + if ! curl -fsSL --retry 3 -o "$TMP/$base.tar.gz.sha256" "$url.sha256"; then log "checksum download failed"; return 1; fi + want="$(cut -d' ' -f1 < "$TMP/$base.tar.gz.sha256")" + fi + have="$(sha256_file "$TMP/$base.tar.gz")" + [ "$want" = "$have" ] || die "sha256 mismatch for $base.tar.gz: expected $want, got $have" + # errexit is off here (called from `if !`), so return on each failure to reach the cargo fallback. + tar -xzf "$TMP/$base.tar.gz" -C "$TMP" || return 1 + mkdir -p "$TOOLS_DIR/bin" || return 1 + install -m 0755 "$TMP/$base/sccache" "$TOOLS_DIR/bin/sccache" || return 1 + log "installed sccache $SCCACHE_VERSION to $TOOLS_DIR/bin/sccache" +} + +install_cargo() { + log "running: cargo install sccache --locked --version $SCCACHE_VERSION --root $TOOLS_DIR (this takes a few minutes)" + cargo install sccache --locked --version "$SCCACHE_VERSION" --root "$TOOLS_DIR" >&2 +} + +find_sccache() { + if [ -n "${SCCACHE_BIN:-}" ] && [ -x "$SCCACHE_BIN" ]; then abspath "$SCCACHE_BIN"; return 0; fi + if [ -x "$TOOLS_DIR/bin/sccache" ]; then echo "$TOOLS_DIR/bin/sccache"; return 0; fi + if command -v sccache >/dev/null 2>&1; then abspath "$(command -v sccache)"; return 0; fi + return 1 +} + +print_exports() { + if [ "$wrapper" = 1 ]; then + printf 'export RUSTC_WRAPPER=%q\n' "$SCCACHE_BIN_PATH" + printf 'export SCCACHE_DIR=%q\n' "$SCCACHE_DIR" + printf 'export SCCACHE_CACHE_SIZE=%q\n' "$SCCACHE_CACHE_SIZE" + if [ "$SCCACHE_BIN_PATH" = "$TOOLS_DIR/bin/sccache" ]; then + # shellcheck disable=SC2016 # $PATH must expand in the caller's shell + case ":$PATH:" in + *":$TOOLS_DIR/bin:"*) ;; + *) printf 'export PATH=%q:"$PATH"\n' "$TOOLS_DIR/bin" ;; + esac + fi + else + # Empty, not unset: an empty RUSTC_WRAPPER also overrides build.rustc-wrapper in cargo config. + printf 'export RUSTC_WRAPPER=\n' + fi + printf 'export RUSTC_WORKSPACE_WRAPPER=%q\n' "$WORKSPACE_WRAPPER" + printf 'export CARGO_TARGET_DIR=%q\n' "$CARGO_TARGET_DIR" +} + +SCCACHE_BIN_PATH="$(find_sccache || true)" + +case "$mode" in + stats) [ -n "$SCCACHE_BIN_PATH" ] || die "sccache not found; run without options to install it"; export SCCACHE_DIR SCCACHE_CACHE_SIZE; exec "$SCCACHE_BIN_PATH" --show-stats ;; + stop) [ -n "$SCCACHE_BIN_PATH" ] || die "sccache not found"; exec "$SCCACHE_BIN_PATH" --stop-server ;; + check) + if [ -z "$SCCACHE_BIN_PATH" ]; then + SCCACHE_BIN_PATH="$TOOLS_DIR/bin/sccache" + log "sccache is not installed; a plain run installs $SCCACHE_VERSION to $SCCACHE_BIN_PATH" + fi + log "would export (no files or servers touched):" + print_exports + exit 0 ;; +esac + +if [ "$wrapper" = 1 ] && [ -z "$SCCACHE_BIN_PATH" ]; then + case "$SCCACHE_INSTALL" in + prebuilt) + if ! install_prebuilt; then + log "prebuilt install failed; falling back to cargo install" + install_cargo + fi ;; + cargo) install_cargo ;; + none) die "sccache not found and SCCACHE_INSTALL=none" ;; + *) die "SCCACHE_INSTALL must be prebuilt, cargo or none" ;; + esac + SCCACHE_BIN_PATH="$(find_sccache)" || die "install succeeded but sccache is still not found" +fi + +mkdir -p "$CARGO_TARGET_DIR" +if [ "$wrapper" = 1 ]; then + mkdir -p "$SCCACHE_DIR" + # The sccache server reads SCCACHE_DIR and SCCACHE_CACHE_SIZE once, when it + # starts. A client with a different SCCACHE_DIR silently uses the running + # server. Detect that and tell the user how to fix it. + realdir() { (cd "$1" 2>/dev/null && pwd -P) || printf '%s' "$1"; } + stats="$(SCCACHE_DIR="$SCCACHE_DIR" SCCACHE_CACHE_SIZE="$SCCACHE_CACHE_SIZE" "$SCCACHE_BIN_PATH" --show-stats 2>&1)" || + die "'$SCCACHE_BIN_PATH --show-stats' failed (a server from another sccache version may be running; stop it with '$SCCACHE_BIN_PATH --stop-server', or use --no-wrapper): +$stats" + running_dir="$(printf '%s\n' "$stats" | sed -n 's/^Cache location *Local disk: "\(.*\)"$/\1/p')" + running_max="$(printf '%s\n' "$stats" | sed -n 's/^Max cache size *//p')" + if [ -n "$running_dir" ] && [ "$(realdir "$running_dir")" != "$(realdir "$SCCACHE_DIR")" ]; then + log "WARNING: an sccache server is already running with SCCACHE_DIR=$running_dir" + log " this shell asked for $SCCACHE_DIR. Run '$SCCACHE_BIN_PATH --stop-server' first," + log " or export SCCACHE_DIR=$running_dir to join the running server." + fi + log "sccache: $SCCACHE_BIN_PATH ($("$SCCACHE_BIN_PATH" --version 2>/dev/null | tr -d '\n'))" + log "SCCACHE_DIR: $SCCACHE_DIR (server limit ${running_max:-unknown}; run with --stop to change it)" +fi +if [ "$target_mode" = "shared" ]; then + log "CARGO_TARGET_DIR: $CARGO_TARGET_DIR (shared by all worktrees; cargo serializes builds on its lock)" +else + log "CARGO_TARGET_DIR: $CARGO_TARGET_DIR (per-worktree; sccache gives no Rust hits here, see README)" +fi +if [ -t 1 ]; then + log "stdout is a terminal. To apply these in your shell run: eval \"\$($0${1:+ $1})\"" +fi +print_exports +exit 0 diff --git a/scripts/dev/worktree.sh b/scripts/dev/worktree.sh new file mode 100755 index 000000000..60f910ac5 --- /dev/null +++ b/scripts/dev/worktree.sh @@ -0,0 +1,91 @@ +#!/usr/bin/env bash +# Create a git worktree that shares the compile cache with every other worktree. +# +# scripts/dev/worktree.sh [base] # create (base default: origin/main, else main) +# scripts/dev/worktree.sh --list # list worktrees +# scripts/dev/worktree.sh --remove # remove the worktree (keeps the branch) +# +# Environment (optional): +# WORKTREE_ROOT where worktrees go (default:
/.worktrees) +# +# The worktree itself holds no build output. All worktrees build into the one +# shared CARGO_TARGET_DIR that scripts/dev/setup-build-cache.sh exports, so a +# new worktree compiles only the bottlecap crate, not its 360 dependencies. +# See scripts/dev/README.md for the measurements. +set -euo pipefail + +log() { printf '%s\n' "worktree: $*" >&2; } +die() { log "error: $*"; exit 1; } + +here="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" +# --path-format=absolute (git >= 2.31): in the main checkout the plain form is the relative ".git". +common_dir="$(git -C "$here" rev-parse --path-format=absolute --git-common-dir)" +main_root="$(dirname "$common_dir")" +WORKTREE_ROOT="${WORKTREE_ROOT:-$main_root/.worktrees}" +case "$WORKTREE_ROOT" in /*) ;; *) WORKTREE_ROOT="$PWD/$WORKTREE_ROOT" ;; esac + +usage() { sed -n '2,14p' "$0" | sed 's/^# \{0,1\}//'; } +slug() { printf '%s' "$1" | tr '/ ' '--'; } +g() { git -C "$main_root" "$@"; } + +case "${1:-}" in + -h|--help|"") usage; exit 0 ;; + --list) exec git -C "$main_root" worktree list ;; + --remove) + [ $# -eq 2 ] || die "--remove needs a branch name" + path="$WORKTREE_ROOT/$(slug "$2")" + [ -d "$path" ] || die "no worktree at $path" + # slug() maps a/b and a-b to the same path; remove only the requested branch. + [ "$(git -C "$path" symbolic-ref --quiet --short HEAD || true)" = "$2" ] || die "$path does not have branch $2 checked out" + g worktree remove "$path" + log "removed $path (branch $2 still exists; delete it with: git branch -D $(printf '%q' "$2"))" + exit 0 ;; +esac + +branch="$1" +base="${2:-}" +path="$WORKTREE_ROOT/$(slug "$branch")" +[ -e "$path" ] && die "$path already exists (use --remove first, or pick another branch name)" +g worktree prune # forget worktrees whose directory was deleted by hand + +if g show-ref --verify --quiet "refs/heads/$branch"; then + [ -z "$base" ] || log "branch $branch exists; ignoring base $base" + mkdir -p "$WORKTREE_ROOT" + g worktree add "$path" "$branch" +elif g show-ref --verify --quiet "refs/remotes/origin/$branch"; then + [ -z "$base" ] || log "branch origin/$branch exists; ignoring base $base" + mkdir -p "$WORKTREE_ROOT" + if ! g worktree add --track -b "$branch" "$path" "origin/$branch" 2>/dev/null; then + # Single-branch clones cannot track refs outside their fetch refspec. + log "could not set up tracking of origin/$branch; creating the branch without upstream" + g worktree add -b "$branch" "$path" "origin/$branch" + fi +else + if [ -z "$base" ]; then + if g rev-parse --verify --quiet origin/main >/dev/null; then base=origin/main; else base=main; fi + log "base not given; using $base ($(g rev-parse --short "$base")). Run 'git fetch origin main' first if that is stale." + fi + g rev-parse --verify --quiet "$base^{commit}" >/dev/null || die "base '$base' is not a commit" + mkdir -p "$WORKTREE_ROOT" + g worktree add -b "$branch" "$path" "$base" +fi + +qpath="$(printf '%q' "$path/bottlecap")" +qbranch="$(printf '%q' "$branch")" +qscript="$(printf '%q' "$here/worktree.sh")" +cat >&2 <