Skip to content

ci: add a manual Linux watch-soak workflow for the cli_watch flake family (#129, #318, #320) #375

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
dean0x wants to merge 1 commit into
base: main
Choose a base branch
from
Open

ci: add a manual Linux watch-soak workflow for the cli_watch flake family (#129, #318, #320) #375

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
194 changes: 194 additions & 0 deletions .github/workflows/watch-soak.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,194 @@
name: Watch soak (manual)

# MANUAL reproduction instrument for the cli_watch flake family (#129 / #318 / #320).
#
# NOT a gate. `workflow_dispatch` is the only trigger, so this workflow publishes
# zero check-runs on any pull-request head: it cannot enter branch protection and
# cannot appear in scripts/verify-pr-checks.mjs's tally. It touches none of the six
# release-surface paths and does not modify release.yml.
#
# It runs CI's exact command -- `cargo test`, deliberately NOT nextest, because the
# process-per-test model nextest uses is a different execution environment and this
# suite's failures are environment-sensitive. ubuntu-latest only: macOS FSEvents
# cannot reproduce this bug class at all (see .devflow/learning/pitfalls.md PF-026
# and the project-watch-tests-flaky memory -- a green macOS run proves nothing).
#
# Runs every iteration and tallies, rather than aborting on the first red, because
# the quantity of interest is a RATE. "Failed at iteration 3" cannot distinguish
# 1/20 from 20/20, and the before/after control this instrument exists to serve
# (PF-027 resolution 6) needs both numbers.

on:
workflow_dispatch:
inputs:
iterations:
description: 'How many times to run the cli_watch suite (1-200)'
type: string
default: '20'
filter:
description: 'Optional cargo-test name filter (empty = the whole suite)'
type: string
default: ''

permissions:
contents: read

# cancel-in-progress: false is load-bearing. A soak run is a MEASUREMENT; cancelling
# one halfway leaves a partial tally indistinguishable from a clean run with fewer
# iterations. run_id is in the group key so two deliberate dispatches on the same
# ref never evict each other.
concurrency:
group: watch-soak-${{ github.ref }}-${{ github.run_id }}
cancel-in-progress: false

env:
CARGO_TERM_COLOR: always
RUST_BACKTRACE: 1

jobs:
soak:
name: Watch soak (${{ matrix.label }})
runs-on: ubuntu-latest
# The loop is bounded by `iterations`, but a hung child inside cargo test is not.
timeout-minutes: 120
strategy:
fail-fast: false
matrix:
include:
# Default features: the same build CI's `rust` job runs.
- label: default
feature: ''
# The #317 probe widens the publish->arm window to 200ms; N iterations turn
# "6/6 green with the probe on" into a rate.
- label: startup-race-probe
feature: 'startup-race-probe'
steps:
- uses: actions/checkout@v7

- uses: dtolnay/rust-toolchain@4360b52568e2003a75bf9bc1d59f33a8e3fc893c # stable

- uses: Swatinem/rust-cache@v2
with:
# Per-leg key (PF-041): the two legs build different feature sets.
key: watch-soak-${{ matrix.label }}

- name: Validate the iterations input
env:
ITERATIONS: ${{ inputs.iterations }}
run: |
set -euo pipefail
case "$ITERATIONS" in
''|*[!0-9]*)
echo "::error::iterations must be a positive integer; got '$ITERATIONS'"
exit 1
;;
esac
if [ "$ITERATIONS" -lt 1 ] || [ "$ITERATIONS" -gt 200 ]; then
echo "::error::iterations must be between 1 and 200; got $ITERATIONS"
exit 1
fi
echo "iterations validated: $ITERATIONS"

# Compile once so the loop measures the SUITE and not rustc.
- name: Build the test binary once
env:
FEATURE: ${{ matrix.feature }}
run: |
set -euo pipefail
args=(build -p mds-cli --tests)
if [ -n "$FEATURE" ]; then
args+=(--features "$FEATURE")
fi
echo "cargo ${args[*]}"
cargo "${args[@]}"

- name: Soak
id: soak
env:
ITERATIONS: ${{ inputs.iterations }}
FILTER: ${{ inputs.filter }}
FEATURE: ${{ matrix.feature }}
LABEL: ${{ matrix.label }}
run: |
# -e is DELIBERATELY omitted: a non-zero `cargo test` is the DATA this
# step collects, not an error that should abort it. -u and pipefail stay.
set -uo pipefail

mkdir -p soak

args=(test -p mds-cli --test cli_watch)
if [ -n "$FEATURE" ]; then
args+=(--features "$FEATURE")
fi
if [ -n "$FILTER" ]; then
args+=(-- "$FILTER")
fi
echo "command: cargo ${args[*]}"

pass=0
fail=0
failed_iters=""
i=1
while [ "$i" -le "$ITERATIONS" ]; do
log="soak/iter-$(printf '%03d' "$i").log"
if cargo "${args[@]}" > "$log" 2>&1; then
pass=$((pass + 1))
rm -f "$log"
printf 'iter %3d PASS\n' "$i"
else
fail=$((fail + 1))
failed_iters="$failed_iters $i"
printf 'iter %3d FAIL\n' "$i"
echo "::warning title=watch-soak::iteration $i failed on leg $LABEL"
# Surface the discriminator inline (PF-026: the panic's file:line, not
# the test's name, is the diagnosis).
grep -E 'panicked at|\.\.\. FAILED|test result: FAILED' "$log" || true
fi
i=$((i + 1))
done

# The summary file is ALWAYS written, so the artifact is never empty (PF-016).
{
echo "leg: $LABEL"
echo "ref: $GITHUB_REF"
echo "sha: $GITHUB_SHA"
echo "command: cargo ${args[*]}"
echo "iterations: $ITERATIONS"
echo "passed: $pass"
echo "failed: $fail"
echo "failed at: $failed_iters"
} > soak/summary.txt
cat soak/summary.txt

{
echo "### Watch soak - $LABEL"
echo ""
echo "| metric | value |"
echo "| --- | --- |"
echo "| ref | \`$GITHUB_REF\` |"
echo "| sha | \`$GITHUB_SHA\` |"
echo "| command | \`cargo ${args[*]}\` |"
echo "| iterations | $ITERATIONS |"
echo "| passed | $pass |"
echo "| **failed** | **$fail** |"
if [ "$fail" -gt 0 ]; then
echo "| failing iterations |$failed_iters |"
fi
} >> "$GITHUB_STEP_SUMMARY"

if [ "$fail" -gt 0 ]; then
echo "::error::$fail of $ITERATIONS iterations failed on leg $LABEL"
exit 1
fi
echo "clean soak: $pass/$ITERATIONS passed on leg $LABEL"

- name: Upload failing logs and the tally
if: always()
uses: actions/upload-artifact@v7
with:
name: watch-soak-${{ matrix.label }}-${{ github.run_id }}
path: soak/
# `error`, not `ignore`: summary.txt is always written, so an empty upload
# means the glob is wrong, not that the soak was clean (PF-016).
if-no-files-found: error
retention-days: 14
1 change: 1 addition & 0 deletions CHANGELOG.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -60,6 +60,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
- `build-napi` per-leg rust-cache key: adds `key: ${{ matrix.settings.target }}` to the `Swatinem/rust-cache` step so each cross-compile leg's target artifacts stay isolated (PF-041; without the key all four ubuntu legs and both macOS legs restored one shared blob, confirmed live in run 34065573775); `build-python`'s existing `key: matrix.target-matrix.manylinux` (#347) unchanged; spec S20 in `release-auth-probe.spec.mjs` pins both and fails `Version gate` if a key is dropped; spec S3 extended to pin the `-z` CARGO_REG_TOKEN guard in executable code; #345 verified that crates.io `GET /api/v1/me` is `AuthCheck::only_cookie()` (HTTP 403 for any API token) and the only token-accepting read route rejects scoped tokens — non-empty guard is the strongest check available, durable fix tracked in #368; #345 closed won't-fix-as-filed (#345 #352).
- Alpine `node:22-alpine` load tests for both musl napi addons gate `publish-crates`: x64 (`linux-x64-musl`) as the last step of `stage-and-verify-napi` (after the staged artifact upload, so the artifact is never suppressed by an x64 failure), arm64 (`linux-arm64-musl`) in a new unguarded `load-test-musl-arm64` job on a native `ubuntu-24.04-arm` runner using the `napi-staged` artifact; both use `scripts/musl-load-probe.cjs` in a `docker run --network none` step with a positive control; `publish-crates` blocks on both via `needs:` AND its `if:` conjunct (PF-047); spec S21 in `release-auth-probe.spec.mjs` pins job existence, runner, guard shape, wiring, step order, and run-block byte-equality (#340); the first CI run surfaced #371 (string compile fails when the base directory is a filesystem root — `node:22-alpine` has no `WORKDIR` so the default container cwd is `/`); the gate now runs the container from `/w` (`docker run -w /w`) and the probe asserts its cwd so a dropped flag fails loudly.
- Both musl napi legs (`x86_64-unknown-linux-musl`, `aarch64-unknown-linux-musl`) now cross-compile with `napi build … -x` / cargo-zigbuild 0.23.0: the SHA-pinned `taiki-e/install-action` (v2.85.10, `fallback: none`) installs cargo-zigbuild before `Swatinem/rust-cache` (rust-cache deletes `~/.cargo/bin` on save; napi's detector is presence-only and would `cargo install` an unpinned copy mid-build otherwise); the hand-written zig cc wrappers, fake-zig self-check, and both `CARGO_TARGET_*_MUSL_LINKER` exports are deleted; three new steps assert the pinned version (before and after the build) and the no-op detector reads both musl linker vars inside `[ -z ]` guards to confirm none is set; the readelf gate adds `ALLOWED_NEEDED='libc\.so|libgcc_s\.so\.1'` with a planted `libunwind.so.1` control; `mlugg/setup-zig` SHA-pinned (v2.2.1) in the same step; spec S22 in `release-auth-probe.spec.mjs` pins all of the above (#339).
- manual `watch-soak.yml` Linux soak instrument for the cli_watch flake family (#129 #318 #320); `workflow_dispatch` only, not a gate, not a required context, not release-surface

## [0.4.2] — 2026-09-03

Expand Down
Loading