feat(PMAT-1095): 0.67 CUDA Rust fleet readiness — fix the two wrong-host GPU tests, add scripts/cuda_rust_fleet_check.sh with per-host receipts (gx10 PASS, yoga PASS, lambda-vector blocked on a reboot) - #3068
Open
noahgift wants to merge 13 commits into
Conversation
… / lambda-vector) Pmat-Ticket: PMAT-1095 Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01J9cSQynVPYeUkQ2i7ccvrs
… not a hard-coded 100 GB / RTX 4090 The test allocated a fixed 25e9 f32 (100 GB) and panicked "impossible on RTX 4090!" if it succeeded. On gx10 -- an NVIDIA GB10 with ~128 GB of UNIFIED memory -- a 100 GB allocation legitimately succeeds, so the test FAILED there. It failed under CUDA 13.0 and 13.3 alike (a control run on 2026-09-09), so it was a wrong-host assumption, not a toolkit regression; it passed on the 4090 only by accident of that card being 24 GB. Now: query CudaContext::total_memory() and request 2x the whole device. That exceeds physical memory on every CUDA device, unified or not, and scales with the host. The adversarial intent -- an allocation larger than the device must fail -- is kept; the card name is gone from the assertion and the panic names the requested and total byte counts instead. A_1 on lambda-vector (RTX 4090): test result ok. gx10 re-run follows in the PMAT-1095 receipt. Pmat-Ticket: PMAT-1095 Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01J9cSQynVPYeUkQ2i7ccvrs
…ed memory oversubscribes by design on GB10 Measured on gx10: with the default routing, a 2x-the-device (257 GB vs 128 GB) allocation SUCCEEDED because GpuBuffer::new routes integrated parts to cuMemAllocManaged (PMAT-769), which oversubscribes by design. The falsifiable property is "cuMemAlloc refuses more than the device", so the test now sets MANAGED_MEMORY=0 (under the exclusivity lock that already covers env mutation) and asserts against the device allocator explicitly. Pmat-Ticket: PMAT-1095 Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01J9cSQynVPYeUkQ2i7ccvrs
…ead of asserting a 50 TFLOP/s wall-clock floor Measured 2026-09-09 on gx10: 15.4 TFLOP/s inside the full --features cuda suite, passing in isolation -- a wall-clock assertion failing under load, the class already burned four times (NO wall-clock assertion in a required check). Correctness stays asserted via the two assert_eq! on C[0,0] and C[last]; the speed claim belongs to the beat/bench lane. The measurement itself is still asserted valid (finite, > 0). Pmat-Ticket: PMAT-1095 Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01J9cSQynVPYeUkQ2i7ccvrs
…t for 2x physical — it OOMed and rebooted gx10 The v2 test asked GB10 (128 GB unified) for a 257 GB cuMemAlloc. On a discrete card that is rejected against the VRAM pool before a page is touched; on a unified-memory part the driver tried to BACK it from system RAM. gx10 went global-OOM at 13:36 on 2026-09-09 with this test binary at the top of the OOM table, and rebooted at 13:43. A run that "passed in 34 s" earlier the same afternoon was that thrash, survived by luck. v3 branches on classify_device_memory(): discrete keeps 2x the device; UnifiedMemory asks for 2^60 bytes, which fails validation with nothing to back it from. The same property is asserted -- larger than the device must fail. memory/mod.rs re-exports classify_device_memory + DeviceMemoryClass so tests can reach the classifier. Pmat-Ticket: PMAT-1095 Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01J9cSQynVPYeUkQ2i7ccvrs
…NVIDIA CUDA Rust readiness, with a verdict that can fail
P3 of PMAT-1095. Turns "0.67 works with cuda-core / cutile on gx10, yoga, lambda-vector"
from a sentence into a receipt per host, evidence/cuda-rust-fleet/<host>.json.
Six probes, each PASS | FAIL | SKIP:<reason>:
driver_floor >= R580 (cuda-bindings refuses older at run time; measured)
toolkit_floor >= 13.1 for sm_100+, >= 13.2 for sm_8x (cutile)
clang_resource_dir bindgen needs clang's OWN stddef.h; a bare libclang is not enough
(measured on yoga: "'stddef.h' file not found"); discovered like
cuda-oxide does — clang, then clang-N under /usr/lib/llvm-N/bin
(gx10 has only clang-21, invisible to a non-interactive PATH)
cudacore_probe experiments/cuda-rust-probes/cudacore: PTX load + launch + readback
cutile_probe experiments/cuda-rust-probes/cutile: Tile kernel JIT + run, prints
the REAL compute capability (a copied probe once printed "sm_121" on an
sm_89 host)
aprender_gpu_cuda the tests this ticket and O2/#3066 added or repaired
Verdict has THREE states on purpose: PASS only if every probe PASSed; INCOMPLETE when a
prerequisite was SKIPped (never counted as a pass — "flawless on three hosts" must not be
reachable by skipping the probes that would fail); FAIL on any FAIL, on zero probes
(vacuity), or on a status outside {PASS,FAIL,SKIP} (a blacklist is fail-open on its
complement). --self-test is a case table over that function; it caught a stale row in
its own table on the first run.
RESOURCE GUARD, because the first draft of this runner OOMed the only Blackwell CI host.
On 2026-09-09 a cold `cargo test -p aprender-gpu --features cuda` here, alongside a test
that asked a unified-memory GB10 for 2x physical RAM, put gx10 into global OOM at 13:36
(this test binary at the top of the OOM table) and it rebooted at 13:43. So P6 now:
refuses while a GitHub Actions job runs on the host; refuses below 12 GiB MemAvailable
(32 was first tried and is unreachable on yoga, a ~31 GiB laptop); nice -n 19 with
CARGO_BUILD_JOBS capped; and runs the TEST PROCESS under `systemd-run --scope
-p MemoryMax=48G` so a regression can kill only itself. MemAvailable is in the receipt.
Environment death is classified, not blamed on code: a network failure (gx10 came back
from its reboot with no default route — "Could not resolve host: index.crates.io")
retries --offline once and then records SKIP:env, i.e. INCOMPLETE, never PASS and never
a FAIL that names a CUDA Rust cause. Every probe's full log is persisted next to the
receipt and pulled back from remote hosts; the first draft deleted them and reported
"exit=101" with an empty reason.
Two runner bugs found by running it: cargo prints the test-binary path RELATIVE to the
manifest dir when CARGO_TARGET_DIR is unset (gx10) and absolute when set (lambda-vector,
which hid the bug); and a silent ssh session was measured dropping mid-build
("client_loop: send disconnect: Broken pipe"), so --host mode uses ServerAlive keepalives
and tees to a remote log.
Receipts committed (2026-09-09):
gx10-a5b5 PASS GB10 sm_121, driver 590.48.01, CUDA 13.3 — 6/6
yoga PASS RTX 4060 Laptop sm_89, driver 595.91.07, CUDA 13.3 — 6/6
noah-Lambda-Vector INCOMPLETE RTX 4090 sm_89, driver 570.207 < R580 — cuda-core and
cutile SKIP; the repaired aprender-gpu tests PASS.
580 is staged in Lambda's repo; taking it needs a reboot
the operator owns (installing it live replaced the
userspace libs under the running 570 module and broke
CUDA until rolled back — measured).
The probe crates carry their own [workspace] (verified: cargo metadata --no-deps lists
79 packages, none under experiments/), so --all-features cannot drag cuda-bindings'
build.rs into make coverage / make mutants.
Pmat-Ticket: PMAT-1095
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01J9cSQynVPYeUkQ2i7ccvrs
…le contract claim, set -e abort path, cgroup sizing, TOCTOU, exact pins, truthful lambda-vector acceptance Phase 3 quorum (3 independent agy lanes, all do-not-implement-as-written) found eight defects; every one is addressed here and each was falsifiable against the export. adversarial.rs - MANAGED_MEMORY was set with std::env::set_var and cleared with an unconditional remove_var: a panic between them leaked the override into every later test in the process, and remove_var destroyed any value the suite was launched with. Replaced by EnvVarGuard, which records the PRIOR state and restores it on drop, including unwind. cublas_tests.rs - The doc comment still claimed "FALSIFY-CUBLAS-003: Must achieve > 100 TFLOP/s". No contract under contracts/ carries that id or any TFLOP/s floor (measured by three lanes and by grep). The comment now says what is true: throughput is reported, no floor exists anywhere, and a speed claim belongs to the beat/bench lane. Coverage did not silently drop -- it never existed outside this one wall-clock assert. scripts/cuda_rust_fleet_check.sh - `bin=$(grep … | sed … | head -1)` under set -euo pipefail: an unmatched grep aborted the whole script BEFORE the receipt was written. Now `|| true`, with an explicit not-found branch. - MemoryMax=48G was a magic number that protected nothing on a host with less than 48 GiB (yoga has ~31). Now min(48 GiB, 60% of MemAvailable), floor 4 GiB, for BOTH the build and the test process. - The pgrep Runner.Worker check was TOCTOU: a CI job could start after it. The build now runs under the same cgroup, and the check is repeated after the build, right before the test runs. - --self-test exercised only the verdict function; the python receipt writer was untested. It is now a function, and the self-test drives it with synthetic probes and asserts the JSON (verdict, probe statuses, typed mem field). That new case immediately caught a real bug: bash arrays cannot ride a command's env prefix (they arrive as the literal string "(a b)"). - Receipts carry `blocked_on`: the SKIP reasons, so an INCOMPLETE names its blocker in the artifact itself rather than in a PR comment. - bashrs: 0 errors (the guarded rm -rf was SEC011). experiments/cuda-rust-probes - cuda-core / cutile pinned exactly (=0.3.1); a diagnostic that drifts with a minor release is not a diagnostic. Cargo.lock committed for both. docs/roadmaps/roadmap.yaml - The lambda-vector acceptance line was contradicted by its own receipt. It now says what is true and still falsifiable: PASS, or INCOMPLETE with driver_floor as the ONLY skipped prerequisite and blocked_on naming the reboot that activates the staged 580 driver. An INCOMPLETE that names its blocker is the truthful artifact; a PASS reached by skipping the probe that would fail is not. Verified on the RTX 4090: both repaired tests pass; clippy -D warnings clean under --features cuda; --self-test PASS; check_roadmap_diff_additive.sh PASS. Pmat-Ticket: PMAT-1095 Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01J9cSQynVPYeUkQ2i7ccvrs
…SS, lambda-vector INCOMPLETE (blocked_on: driver 570 < R580, reboot activates staged 580) Re-run with the fixed runner on all three hosts. yoga now PASSes 6/6 with the cgroup sized to 16G on a 28 GiB host (the dynamic MemoryMax from the quorum finding). lambda-vector stays INCOMPLETE and the receipt itself names the blocker in blocked_on. Pmat-Ticket: PMAT-1095 Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01J9cSQynVPYeUkQ2i7ccvrs
…ule doc, blocked_on names the reboot and lists root causes only, memory floor fails CLOSED F2: cublas_tests.rs line 6 cited cublas-gemm-v1.yaml (FALSIFY-CUBLAS-001/-003/-005); neither the file nor the ids exist (delegate-verified). It now names contracts/fp16-cublas-gemm-v1.yaml and states no TFLOP/s floor exists. F8: the driver_floor SKIP reason names the remedy (R580+ driver package AND a reboot to activate it) and blocked_on excludes probes skipped BECAUSE of another prerequisite, so lambda-vector lists exactly one root blocker. N1 (new, delegate-measured): with MemAvailable unreadable, awk printed nothing and exited 0, `|| echo 0` never fired, and the 12 GiB floor fell THROUGH to the build -- fail-open. The floor is now a pure function that refuses empty or non-numeric input, self-tested with "", "x", "0", "5", "11" (refuse) and "12", "20", "115" (allow). Pmat-Ticket: PMAT-1095 Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01J9cSQynVPYeUkQ2i7ccvrs
…e memory floor glob-free and the self-test loops parseable (bashrs 0 errors) The previous commit pushed with three bashrs errors (BRS0010 on a [!0-9] glob it reads as a test bracket, SC2135 on two one-line for/if loops) and one remaining FALSIFY-CUBLAS-005 doc id that no contract carries. All three fixed; --self-test PASS. Pmat-Ticket: PMAT-1095 Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01J9cSQynVPYeUkQ2i7ccvrs
…a-vector INCOMPLETE with one root blocker that names the reboot Re-run on all three hosts with the final runner (sha in each receipt). lambda-vector blocked_on lists exactly one root entry: driver_floor, whose reason states the R580+ package and the reboot that activates it; the two probe skips derived from it are no longer listed as separate blockers. Pmat-Ticket: PMAT-1095 Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01J9cSQynVPYeUkQ2i7ccvrs
noahgift
enabled auto-merge
September 9, 2026 15:10
…ness Identity, routing, dispatch ledger (4 delegate dispatches: ph1 goal partial, ph3 quorum r1 partial / r2 / r3 agreed), claimed-vs-rerun verification table, jidoka log (gx10 OOM+reboot, lambda-vector driver rollback, stray delegate worktree), estimates (K-hat 4 first-run[U], actual ~96), gaps (lambda-vector reboot, pr-review receipts NotRun, vacuous transcript-gate), verdict DONE pending required checks. Pmat-Ticket: PMAT-1095 Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01J9cSQynVPYeUkQ2i7ccvrs
…mpty file The earlier writer used an unquoted heredoc whose prose held a backticked apt command; the substitution hung on sudo and produced 0 lines. Written by python now. Estimates row appended. Pmat-Ticket: PMAT-1095 Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01J9cSQynVPYeUkQ2i7ccvrs
6 tasks
|
§13.11 rung 1 — quorum shadow verdict Shadow mode: this records a verdict and merges nothing. A refusal |
6 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
2>&1 | tail -1
PMAT-1095 · epic #3062 · dark-modules issue #3067 · spec #3061. Run through
paiml-implement(Phase 0 admit, three quorum rounds via the agy delegate, Phase 4 gates). Receipt:docs/audits/impl-PMAT-1095-receipt.md.What it does
test_alloc_oversize_100gbno longer hard-codes a 100 GB / RTX 4090 assumption. It classifies the device: discrete askscuMemAllocfor 2× the device; unified memory asks for 2⁶⁰ bytes, which fails validation with nothing to back it from. The device allocator is pinned with an RAIIEnvVarGuard(restores prior state on unwind).test_cublas_gemm_f16_training_shapereports throughput instead of asserting a 50 TFLOP/s wall-clock floor (15.4 on GB10 under suite load, 165 on a 4090 alone). Parity stays asserted; the staleFALSIFY-CUBLAS-*citations that named a non-existent contract are gone.scripts/cuda_rust_fleet_check.sh— a per-host receipt for cuda-core / cutile / aprender-gpu readiness with a three-state verdict (PASS / INCOMPLETE / FAIL: a skipped prerequisite can never read as a pass; zero probes or an unknown status is FAIL), a--self-testcase table that also drives the receipt writer, env-death classification (network →SKIP:env), persisted per-probe logs, ssh keepalive, and a resource guard (refuses under a running CI job — checked twice — and below 12 GiB fail-closed; nice'd, capped jobs, build and test undersystemd-run MemoryMax=min(48G, 60% of MemAvailable)).Receipts (committed,
evidence/cuda-rust-fleet/)blocked_on= one root entry: driver < R580; an R580+ package is staged in Lambda's repo and needs a reboot the operator ownsWhat went wrong on the way, owned
cargo test --features cudaalongside P1 v2's 2×-physicalcuMemAllocon a unified-memory part put the box into global OOM at 13:36 (this test binary at the top of the OOM table); it rebooted at 13:43 and came back with no default route (its DHCP profile isautoconnect=false; restored withnmcli con up, no config change). Every guard above exists because of this.MemAvailable); round 3 is 3/3 implement-as-written,agreed=true, no dissent.Gates
make gate39/0 · fmt · clippy-D warnings(cuda) · bashrs 0 errors ·--self-testPASS · roadmap additive (+25) · kind-gate code.pv: no contract changed here. Thepresentpr-review receipt is NotRun (not a required check).🤖 Generated with Claude Code
https://claude.ai/code/session_01J9cSQynVPYeUkQ2i7ccvrs