test(release): add a bats suite for the release-proposal scripts - #2507
Draft
iunanua wants to merge 13 commits into
Draft
test(release): add a bats suite for the release-proposal scripts#2507iunanua wants to merge 13 commits into
iunanua wants to merge 13 commits into
Conversation
release-proposal-dispatch.yml can only be exercised end to end by running a real release, so the scripts it calls had no coverage at all. Add 95 tests: - publication-order.sh: which crates a release includes and the order they publish in, including the dev-dependency cycle the topological sort must ignore and the publish = false exclusion. - commits-since-release.sh: commit attribution per crate, annotated-tag dereferencing, the merge-base fallback for tags on squash-merged history, the exclusion rules, and JSON escaping of hostile commit subjects. - semver-level.sh: the bump level fed to `cargo release version -x`, parsed out of cargo-semver-checks and cargo-public-api output. - major-bumps-level.sh: whether a direct libdd-* dependency going major forces a dependent to major, across the dep-kind and wildcard-req cases. - release-proposal-dispatch.yml: structural guards on the job-level `if:` gating, the untrusted-main_start_ref defenses, the branch prefixes shared with release-proposal-test.yml, and the verified-push conditions. These complement actionlint, which checks syntax but not intent. Two helpers keep the suite fast and hermetic. helpers/fixture.bash builds a synthetic six-crate cargo workspace in a throwaway git repository, so cargo metadata, tags and worktrees are real without touching this checkout. helpers/cargo-stub/cargo replays recorded cargo-semver-checks and cargo-public-api output -- the parsing of that output is the logic under test -- and forwards every other subcommand to the real cargo. Runs in ~15s via ./scripts/tests/run.sh; see scripts/tests/README.md. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Adds a workflow that runs the bats suite and lints the standalone shell scripts. actionlint (lint.yml) already shellchecks the `run:` blocks inside workflow files, but nothing covered scripts/*.sh; SC2086 is excluded there to match the actionlint job's options. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…se.sh The script now exports the range its commits were taken from, and release-proposal-dispatch.yml uses that instead of recomputing it. The range reaches a published CHANGELOG through git-cliff, so a wrong start point is not visible until after the release goes out. Covers each way the start point is chosen -- the tag commit, an annotated tag dereferenced to its commit, the merge-base when the tag is not an ancestor of HEAD, and the parent of the oldest commit found when the tag sits on unrelated history -- plus the root-commit fallback, the empty range for a crate with no previous tag, and that every crate's range ends at the same commit. The root-commit case is a regression test: `git rev-parse <root>^` exits non-zero while echoing "<sha>^" on stdout, which previously leaked into the range as an unresolvable ref. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Both fields moved into commits-since-release.sh from the bump loop and both feed decisions that are invisible until after a release ships. latest_tag drives the skip rule: when it differs from the crate's own tag, a newer release already exists and the proposal must not re-release the older version. Covers version-sort ordering (v1.10.0 beats v1.9.0, which plain string ordering gets backwards), the lagging-manifest case the rule exists for, the never-released case, reporting when the crate's own tag is absent, and the sibling-prefix guard -- libdd-alpha-v* must not match libdd-alpha-ffi-v9.9.9. tag_in_local_branch is asserted to be a JSON boolean, not the string "true", because the workflow compares it as one. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The release decision matrix had no coverage at all: which crates get released and at what version, which are deferred to the libdd-* major-bump check, which are skipped because a newer tag exists, and which failures stop the run. 18 tests over a throwaway workspace, one per outcome: - released at the level semver-level.sh reports, with major/minor/patch each asserted against the version cargo-release actually produces - deferred when a tagged crate has no commits -- and verified untouched: no bump, no commit, semver-level.sh never invoked - skipped when the crate's tag is not the latest, and released anyway under --hotfix and --bypass-standard-checks - initial release for an untagged crate at 0.1.0; a hard failure otherwise - semver-level.sh failing, an unresolvable tag range, a malformed input file, and each missing mandatory option - a mixed batch giving four crates four different outcomes in one run cargo-release runs for real rather than stubbed, so the asserted versions are the ones a release would land on; the CI job installs it, and the tests skip rather than fail when it is absent. Only semver-level.sh is doubled, by copying the script under test next to a stub sibling -- it resolves that sibling relative to itself, so no test-only hook is needed in the script. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The audit decides the final shape of a release: which crates are forced to major because a direct libdd-* dependency went to a new major inside the same proposal, and which no-commit candidates get pulled back in or dropped. A mistake here either ships a crate at a compatible-looking version against an incompatible dependency, or removes it from the release without saying so. 15 tests, one per outcome: - a released crate below major promoted, with the version cargo-release lands on and the recorded reason (dependency, previous req, current req) - a crate already at major left alone, and a crate with no libdd-* dependency left alone -- both verified untouched in the manifest, not just in the JSON - dev- and build-dependency major bumps ignored - a pending crate pulled into the release, and one dropped from it entirely - pending_release never surviving into the result - a mixed batch of four crates asserting the output ordering downstream relies on: released crates in order, promoted pending crates appended - the throwaway audit worktree removed, on the success path and on failure - the audit failing, and the argument validation Nothing is stubbed. major-bumps-level.sh only reads cargo metadata, with no compilation, so the whole step runs for real. The simulated previous release is driven through cargo-release rather than by editing manifests: it rewrites dependents' version requirements the way a real release does, and the workspace still resolves afterwards. Hand-editing left libdd-gamma requiring libdd-beta ^0.5 against a 0.6.0 sibling, which cargo refused. That also retires fixture_set_version, now unused. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
CHANGELOG entries are the one release output nothing downstream checks: a wrong or missing entry is published and stays published. 15 tests over the four paths a crate can take -- a git-cliff entry from its commits, a minimal entry when only a dependency went major, an initial-release file, or nothing at all -- plus the compare-link header with and without a previous tag, --remote-url, a missing major_bumps key, an empty release set, and argument handling. git-cliff renders for real against the repository's own cliff.toml, copied into the fixture, so the asserted entries are the ones a release would publish. The test that matters most is "includes only the crate's own commits": git-cliff filters paths on cumulative tree diffs, so an unrelated commit sitting between the oldest and newest of a crate's commits is swept into the entry unless the second pass filters by hash. Verified non-vacuous by removing that filter, which makes the unrelated commit appear and the test fail. Adds two fixture helpers: fixture_add_cliff_config and fixture_commits_json. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The bats suite had no owner. CODEOWNERS has no catch-all and the validator runs the `notowned` experimental check, so unowned files fail CI. A directory pattern rather than a file list: the suite grows a file whenever a release script does, and the existing per-file entries for scripts/ would go stale silently. Owned by libdatadog-core, matching the scripts it exercises. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Regression tests for the process-substitution bug fixed in 4ee9001. The three loops read their rows with `jq -c '.[]' FILE`; wired up as `done < <(jq ...)` that call's exit status was reported by nothing -- neither set -e nor pipefail sees a process substitution -- so a jq dying mid-stream left the loop with no input and the script exited 0 having done nothing. helpers/jq-stub/jq fails only that one argument shape and forwards everything else, so the surrounding script and the assertions keep using the real jq. One test per script, each also checking that nothing happened on the way out: no version bump, no promotion, no CHANGELOG commit. The changelog one carries the reason it matters most -- the caller's `git diff --quiet "$EPHEMERAL_BRANCH"` guard still sees the version-bump commits from the earlier step, so a run that generated no changelogs is indistinguishable from a healthy one. Verified non-vacuous by restoring the process substitution in all three scripts, which makes all three tests fail. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Regression test for d854633. --api-changes was validated in the caller's directory and then passed unchanged into the audit subshell, which runs with `cd "$MAJOR_BUMPS_WT"`, so a relative path failed there with "Not a file" after validating fine a few lines earlier. Also asserts that a relative --out still lands in the caller's directory, since that path deliberately was not resolved: nothing writes it after a cd. Verified non-vacuous by reverting the resolution, which reproduces the original "Not a file: rel-in.json". Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…nd feature changes
iunanua
force-pushed
the
igor/versioning/tests-for-release-proposal-wf
branch
from
September 11, 2026 09:08
729c81a to
6ff6da1
Compare
Contributor
|
✅ All CI checks and tests passed. 🎉 All green!🧪 All tests passed 🎯 Code Coverage (details) 🔗 Commit SHA: 6ff6da1 | Docs | View more details | Give us feedback! |
Contributor
Artifact Size Benchmark Reportaarch64-alpine-linux-musl
aarch64-unknown-linux-gnu
libdatadog-x64-windows
libdatadog-x86-windows
x86_64-alpine-linux-musl
x86_64-unknown-linux-gnu
|
BenchmarksComparisonBenchmark execution time: 2026-09-11 09:44:49 Comparing candidate commit 6ff6da1 in PR branch Found 2 performance improvements and 0 performance regressions! Performance is the same for 164 metrics, 10 unstable metrics.
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What does this PR do?
Adds a bats suite for the scripts that drive the release-proposal workflows, and wires it into CI alongside
shellcheckoverscripts/*.sh(.github/workflows/release-scripts.yml).Coverage, one suite per script — see
scripts/tests/README.mdfor the full table:publication-order.batscommits-since-release.batsrelease-version-bumps.batsrelease-version-major-bumps.batsrelease-generate-changelogs.batssemver-level.batscargo release version -xmajor-bumps-level.batslibdd-*dependency going major forces a dependent to majorrelease-proposal-workflow.batsif:gating, untrusted-main_start_refdefenses, branch prefixes, push path, PR contractThe final commit adds 19 cases and 5 fixture helpers for the dependency-floor and cargo-feature passes introduced in the base PR.
Motivation
scripts/*.shand.github/workflows/release-proposal-dispatch.ymldecide version numbers, changelog contents and publication order for every crate in the workspace, and a real release is otherwise the only way to exercise them. A regression there is discovered by shipping it.Additional Notes
igor/versioning/semver-level-dep-and-feat(#2506), notmain, because the last commit's tests exercise the passes that PR adds. Review #2506 first; this one should be retargeted tomainonce it lands.How the suites work, in case it affects how you read them:
helpers/fixture.bash) — a synthetic cargo workspace in a throwaway git repository, socargo metadata, git tags and worktrees are all real. Its shape is deliberate: a dev-dependency cycle, apublish = falsecrate, a build-dependency edge and a non-libdd-*crate, because those are the cases the scripts have to discriminate. Nothing touches the libdatadog checkout.cargo-releaseand realgit-cliff, not stubs, so the versions and changelog entries asserted are the ones a release would produce. Both are also used to set up state — driving a simulated previous release through cargo-release rewrites dependents' requirements the way a real release does, which a hand-edited manifest does not. Without those tools installed the affected tests skip rather than fail.cargoshim (helpers/cargo-stub/cargo) replays recordedcargo semver-checks/cargo public-apioutput, because those cost minutes per call and their output parsing is whatsemver-level.shgets wrong when it regresses.cargo metadatais deliberately forwarded to the real cargo.run_tool NAME, so the whole suite can be pointed at a reimplementation withRELEASE_TOOL_CMD='target/release/release-tool {name}'. One suite opts out with a stated reason —release-version-bumps.batsdoublessemver-level.shby mirroringscripts/into a temp directory, which does not generalise.helpers/workflow-to-json.py) transcribe the workflow YAML to JSON so tests can assert on it withjq. These are intentional change-detectors: each states why the invariant exists, so changing the workflow means reading the reason first. They complement actionlint, which checks syntax but not intent.How to test the change?
Without root, bats installs with:
CI runs the same suite plus shellcheck via the new
release-scripts.yml. On this PR that runs against the merge of head into base, so the 19 new cases have thesemver-level.shpasses they exercise.🤖 Generated with Claude Code