ci: add a manual Linux watch-soak workflow for the cli_watch flake family (#129, #318, #320)

[dean0x]

Summary

This adds an instrument, not a fix. Every cli_watch failure in the last 40 ci.yml runs is one of exactly two mechanisms, both on ubuntu-latest: a fixed-window debounce split at crates/mds-cli/tests/cli_watch.rs:1222, and truncate-then-write tearing at --debounce 0 at crates/mds-cli/tests/cli_watch.rs:4370 (most recently run 34366009518 on main @ 2b91850). macOS FSEvents cannot reproduce this bug class at all, so a green macOS leg proves nothing.

The problem with fixing it blind is that ci.yml runs the suite once per push with no retries. That produces a bit, not a rate — and "failed at iteration 3" cannot distinguish 1-in-20 from 20-in-20. A before/after control (PF-027 resolution 6) needs both numbers.

watch-soak.yml runs cargo test -p mds-cli --test cli_watch N times (1-200, default 20) on ubuntu-latest, across two legs — default and startup-race-probe (the #317 probe widens the publish→arm window to 200 ms) — and tallies pass/fail into a step summary plus an artifact holding only the failing iterations' logs. It deliberately uses cargo test, not nextest: nextest's process-per-test model is a different execution environment, and these failures are environment-sensitive.

Sequencing: this PR lands first so the baseline rate is measured against unmodified main. PR-B (the actual fixes) follows and is dispatched on its own branch for the after-rate.

What this is NOT

• Not a gate. workflow_dispatch is the only trigger.
• Zero check-runs on any PR head — it therefore cannot enter branch protection and cannot appear in scripts/verify-pr-checks.mjs's tally (neither Tier A nor Tier B).
• Not in RELEASE_SURFACE. It touches none of the six release-surface paths and does not modify release.yml.
• Invisible to all 212 gate specs (npm run test:gates) — confirmed green below.
• cancel-in-progress: false is deliberate: a soak is a measurement, and cancelling one halfway leaves a partial tally indistinguishable from a clean run with fewer iterations. github.run_id is in the concurrency group key so two deliberate dispatches on the same ref never evict each other.
• set -uo pipefail without -e in the loop is deliberate: a non-zero cargo test is the DATA this step collects, not an error that should abort it. The step still exits 1 at the end if fail > 0.

Pins

All four mirror the live files byte-for-byte; nothing new was introduced.

Action	Pin used	Source mirrored	Rationale
actions/checkout	@v7	ci.yml:25	first-party actions/* → tag pin (PF-040)
dtolnay/rust-toolchain	@4360b52568e2003a75bf9bc1d59f33a8e3fc893c # stable	ci.yml:26	third-party → SHA-pinned to the commit (PF-040); copied byte-for-byte including the two-space comment gap, verified with diff
Swatinem/rust-cache	@v2	ci.yml:34 (and :62/:80/:177/:239/:280/:332/:410)	ci.yml pins this by tag, not SHA, at all 8 call sites — mirroring the repo's existing convention was chosen over unilaterally introducing a SHA pin in a non-gate workflow. Changing the convention belongs in a dedicated sweep across all 8+ sites, not here.
actions/upload-artifact	@v7	release.yml:450/490/910	first-party actions/* → tag pin (PF-040)

Other pitfalls honoured:

• PF-041 — per-leg rust-cache key watch-soak-${{ matrix.label }}: the two legs build different feature sets, so one shared blob would be a contamination channel.
• PF-045 — no ${{ }} token appears anywhere inside any run: block, not even in a bash comment. Every value crosses the boundary via env:. Scan output below.
• PF-016 — if-no-files-found: error (not ignore) against a soak/summary.txt that is written on every path, so an empty upload means the glob is wrong rather than that the soak was clean.
• PF-026 — on a failing iteration the step greps panicked at / ... FAILED / test result: FAILED inline, because the panic's file:line is the diagnosis, not the test's name.

Verification

Source-hygiene gate:

$ node scripts/verify-no-control-bytes.mjs
✓ source-hygiene gate: Scanned 558 file(s), 6479130 byte(s) from /Users/dean/Sandbox/mdl
EXIT=0

Gate spec suite:

$ npm run test:gates 2>&1 | grep -E '^# (pass|fail|tests)'
# tests 212
# pass 212
# fail 0

PF-045 expression scan — every hit is at a YAML key level; the three run: | blocks span lines 79-90, 97-103, and 113-183, and no hit falls inside any of them:

$ grep -n '\${{' .github/workflows/watch-soak.yml
41:  group: watch-soak-${{ github.ref }}-${{ github.run_id }}
50:    name: Watch soak (${{ matrix.label }})
73:          key: watch-soak-${{ matrix.label }}
77:          ITERATIONS: ${{ inputs.iterations }}
95:          FEATURE: ${{ matrix.feature }}
108:          ITERATIONS: ${{ inputs.iterations }}
109:          FILTER: ${{ inputs.filter }}
110:          FEATURE: ${{ matrix.feature }}
111:          LABEL: ${{ matrix.label }}
189:          name: watch-soak-${{ matrix.label }}-${{ github.run_id }}

$ grep -n '^[[:space:]]*#.*\${{' .github/workflows/watch-soak.yml
(no output; exit 1)

$ grep -n 'run: |' .github/workflows/watch-soak.yml
78:        run: |
96:        run: |
112:        run: |

Pin byte-comparison against ci.yml:

$ diff <(grep -o 'dtolnay/rust-toolchain@.*' .github/workflows/watch-soak.yml) \
       <(sed -n '26p' .github/workflows/ci.yml | grep -o 'dtolnay/rust-toolchain@.*')
TOOLCHAIN LINE: IDENTICAL

YAML parse (actionlint is not installed on this machine and was deliberately not installed):

$ command -v actionlint
actionlint: not installed

$ node -e "const y=require('js-yaml'); ..."
js-yaml: VALID YAML
top keys: name, on, permissions, concurrency, env, jobs
triggers: [ 'workflow_dispatch' ]
jobs: [ 'soak' ]
legs: default,startup-race-probe

Note (PF-045): js-yaml passing is necessary and not sufficient. GitHub's expression pre-processor is a raw-text scanner that js-yaml and actionlint both cannot emulate — that is why the explicit ${{ line-range scan above, not the parser, is the binding check here.

Dispatch probe (D-U5 premise check), run once after the push and before this PR:

$ gh workflow run watch-soak.yml --ref ci/watch-soak-workflow -f iterations=1
EXIT=1
--- STDOUT ---
--- STDERR ---
HTTP 404: workflow watch-soak.yml not found on the default branch (https://api.github.com/repos/dean0x/mdscript/actions/workflows/watch-soak.yml)

Confirmed as expected: a workflow_dispatch workflow is not dispatchable until its file exists on the default branch. This is precisely why the work is split into two PRs — this one lands the instrument so it becomes dispatchable, and PR-B carries the fixes.

Working-tree scope:

$ git diff --cached --name-only
.github/workflows/watch-soak.yml
CHANGELOG.md

(An unrelated local edit to .gitignore was present in the tree throughout and is deliberately not part of this PR.)

Release process note

No gh workflow run release.yml dispatch for this PR: it changes no build input (no crates/**, no bindings, no release-surface path). The user may override at approval.

Related Issues

Refs #129 #318 #320

[dean0x]

CI rerun: pre-existing flaky test, not caused by this PR

Run: 34404318888 on head a8faafbe317335e99c623d5517b86836cc13108a

Attempt 1 failed only in Rust — fmt, clippy, test, with two panics:

thread 'i17_...' panicked at crates/mds-cli/tests/cli_watch.rs:4370:5:
  left: 1
 right: 2

thread 'i18_duplicate_introduced_mid_session_is_reported_on_the_next_rebuild' (...) panicked at crates/mds-cli/tests/cli_watch.rs:4448:5:
  left: 2
 right: 1

This PR touches only .github/workflows/*.yml and CHANGELOG.md — it cannot alter Rust runtime behavior. Both failing tests are in the i16–i20 vars-file family run at --debounce 0, whose test harness writes files via plain std::fs::write (truncate-then-write, PF-027). This is the pre-existing intermittent race class already tracked (see project_watch_tests_flaky memory), not a new regression.

Per plan, exactly one gh run rerun 34404318888 --failed was issued (PF-026: verified by reading the actual panic lines, not assumed a flake).

Attempt 2 (this rerun) is still RED: i18_duplicate_introduced_mid_session_is_reported_on_the_next_rebuild panicked again at the same location, cli_watch.rs:4448 (left: 2, right: 1). i17 passed this time — consistent with an intermittent race, not a deterministic failure. All other 15 jobs are green.

No further reruns issued. Fix belongs to PR-B of the C2 workstream (atomic writes for the test harness).