Repository navigation
fix(ptx): emitter was non-deterministic — the cubin cache has NEVER hit (42,880 stale cubins / 498 MB) - #3066
Merged
Merged
Conversation
`RegisterAllocator::emit_declarations` iterated a `HashMap` to emit `.reg` lines.
Rust randomises HashMap iteration order per map instance, so PTX emission was not a
pure function of its inputs.
MEASURED, on gx10, before the fix:
same kernel (VectorizedRmsNormKernel h=4096), 5 emissions in ONE process
-> 5 DISTINCT PTX strings
same kernel, 3 separate processes
-> 3 distinct sha256
reg-declaration order across h=2048/4096/8192
-> [.pred .f32 .u64 .u32] / [.u64 .u32 .f32 .pred] / [.u64 .pred .u32 .f32]
This is not cosmetic. `driver/ptx_cache.rs` keys the cubin disk cache on
`sha256(patched_ptx || jit_target || driver_version)`, and `generate_ptx` is called
per launch path (`aprender-serve/src/cuda/executor/layer_cuda_executor.rs:71`,
`incremental_attention.rs:94`, ...). A fresh key every call means the cache can
never hit: every kernel is JIT-compiled from scratch, every time, and the cache
directory grows without bound.
Fallout found on disk, which is what confirmed the consequence rather than assuming
it:
~/.cache/trueno/ptx/ lambda-vector: 42,880 cubins / 498 MB
gx10 : 25,216 cubins / 274 MB
for a few dozen real kernels. Those are 42,880 DISTINCT sha256 keys, i.e. 42,880
distinct PTX texts emitted for a handful of distinct kernels.
FIX: `HashMap` -> `BTreeMap` for the prefix-grouping map. Declaration order becomes
sorted-by-prefix: arbitrary, but STABLE, which is all `.reg` declarations require.
One line, plus the comment explaining why it must never go back.
AFTER (same script, same host):
5 emissions in one process -> 1 distinct output
3 separate processes -> identical sha256
all three hidden sizes -> identical [.f32 .pred .u32 .u64] order
MUTATION PROOF: reverting BTreeMap -> HashMap turns all three new tests RED, and
restoring turns them GREEN:
ptx::registers::tests::determinism::emit_declarations_is_stable_across_allocator_instances
ptx::registers::tests::determinism::emit_declarations_is_stable_within_one_allocator
ptx::registers::tests::determinism::declarations_are_sorted_by_register_prefix
The second of those is worth noting: even TWO CALLS ON ONE ALLOCATOR disagreed,
because the map is rebuilt per call.
Verified: fmt clean; clippy -D warnings clean on default and --features cuda;
`cargo test -p aprender-gpu --features cuda --lib` = 2604 passed, 0 failed.
CAVEAT, stated because it limits the guard: `pub mod ptx` is `#[cfg(feature =
"cuda")]` (lib.rs:150), so these tests run ONLY with --features cuda -- which today
means the gx10 nightly, not the required check. Filed separately.
Refs #3062
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01J9cSQynVPYeUkQ2i7ccvrs
noahgift
enabled auto-merge
September 9, 2026 13:00
|
§13.11 rung 1 — quorum shadow verdict Shadow mode: this records a verdict and merges nothing. A refusal |
noahgift
added a commit
that referenced
this pull request
Sep 9, 2026
…NVIDIA CUDA Rust readiness, with a verdict that can fail
P3 of PMAT-1095. Turns "0.67 works with cuda-core / cutile on gx10, yoga, lambda-vector"
from a sentence into a receipt per host, evidence/cuda-rust-fleet/<host>.json.
Six probes, each PASS | FAIL | SKIP:<reason>:
driver_floor >= R580 (cuda-bindings refuses older at run time; measured)
toolkit_floor >= 13.1 for sm_100+, >= 13.2 for sm_8x (cutile)
clang_resource_dir bindgen needs clang's OWN stddef.h; a bare libclang is not enough
(measured on yoga: "'stddef.h' file not found"); discovered like
cuda-oxide does — clang, then clang-N under /usr/lib/llvm-N/bin
(gx10 has only clang-21, invisible to a non-interactive PATH)
cudacore_probe experiments/cuda-rust-probes/cudacore: PTX load + launch + readback
cutile_probe experiments/cuda-rust-probes/cutile: Tile kernel JIT + run, prints
the REAL compute capability (a copied probe once printed "sm_121" on an
sm_89 host)
aprender_gpu_cuda the tests this ticket and O2/#3066 added or repaired
Verdict has THREE states on purpose: PASS only if every probe PASSed; INCOMPLETE when a
prerequisite was SKIPped (never counted as a pass — "flawless on three hosts" must not be
reachable by skipping the probes that would fail); FAIL on any FAIL, on zero probes
(vacuity), or on a status outside {PASS,FAIL,SKIP} (a blacklist is fail-open on its
complement). --self-test is a case table over that function; it caught a stale row in
its own table on the first run.
RESOURCE GUARD, because the first draft of this runner OOMed the only Blackwell CI host.
On 2026-09-09 a cold `cargo test -p aprender-gpu --features cuda` here, alongside a test
that asked a unified-memory GB10 for 2x physical RAM, put gx10 into global OOM at 13:36
(this test binary at the top of the OOM table) and it rebooted at 13:43. So P6 now:
refuses while a GitHub Actions job runs on the host; refuses below 12 GiB MemAvailable
(32 was first tried and is unreachable on yoga, a ~31 GiB laptop); nice -n 19 with
CARGO_BUILD_JOBS capped; and runs the TEST PROCESS under `systemd-run --scope
-p MemoryMax=48G` so a regression can kill only itself. MemAvailable is in the receipt.
Environment death is classified, not blamed on code: a network failure (gx10 came back
from its reboot with no default route — "Could not resolve host: index.crates.io")
retries --offline once and then records SKIP:env, i.e. INCOMPLETE, never PASS and never
a FAIL that names a CUDA Rust cause. Every probe's full log is persisted next to the
receipt and pulled back from remote hosts; the first draft deleted them and reported
"exit=101" with an empty reason.
Two runner bugs found by running it: cargo prints the test-binary path RELATIVE to the
manifest dir when CARGO_TARGET_DIR is unset (gx10) and absolute when set (lambda-vector,
which hid the bug); and a silent ssh session was measured dropping mid-build
("client_loop: send disconnect: Broken pipe"), so --host mode uses ServerAlive keepalives
and tees to a remote log.
Receipts committed (2026-09-09):
gx10-a5b5 PASS GB10 sm_121, driver 590.48.01, CUDA 13.3 — 6/6
yoga PASS RTX 4060 Laptop sm_89, driver 595.91.07, CUDA 13.3 — 6/6
noah-Lambda-Vector INCOMPLETE RTX 4090 sm_89, driver 570.207 < R580 — cuda-core and
cutile SKIP; the repaired aprender-gpu tests PASS.
580 is staged in Lambda's repo; taking it needs a reboot
the operator owns (installing it live replaced the
userspace libs under the running 570 module and broke
CUDA until rolled back — measured).
The probe crates carry their own [workspace] (verified: cargo metadata --no-deps lists
79 packages, none under experiments/), so --all-features cannot drag cuda-bindings'
build.rs into make coverage / make mutants.
Pmat-Ticket: PMAT-1095
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01J9cSQynVPYeUkQ2i7ccvrs
6 tasks
github-merge-queue
Bot
removed this pull request from the merge queue due to failed status checks
Sep 9, 2026
6 tasks
Merged
noahgift
added a commit
to guyernest/aprender
that referenced
this pull request
Sep 13, 2026
…ost GPU tests, add scripts/cuda_rust_fleet_check.sh with per-host receipts (gx10 PASS, yoga PASS, lambda-vector blocked on a reboot) (paiml#3068) * roadmap: mint PMAT-1095 — 0.67 CUDA Rust fleet readiness (gx10 / yoga / lambda-vector) Pmat-Ticket: PMAT-1095 Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01J9cSQynVPYeUkQ2i7ccvrs * fix(gpu): test_alloc_oversize_100gb asserts against the DEVICE total, not a hard-coded 100 GB / RTX 4090 The test allocated a fixed 25e9 f32 (100 GB) and panicked "impossible on RTX 4090!" if it succeeded. On gx10 -- an NVIDIA GB10 with ~128 GB of UNIFIED memory -- a 100 GB allocation legitimately succeeds, so the test FAILED there. It failed under CUDA 13.0 and 13.3 alike (a control run on 2026-09-09), so it was a wrong-host assumption, not a toolkit regression; it passed on the 4090 only by accident of that card being 24 GB. Now: query CudaContext::total_memory() and request 2x the whole device. That exceeds physical memory on every CUDA device, unified or not, and scales with the host. The adversarial intent -- an allocation larger than the device must fail -- is kept; the card name is gone from the assertion and the panic names the requested and total byte counts instead. A_1 on lambda-vector (RTX 4090): test result ok. gx10 re-run follows in the PMAT-1095 receipt. Pmat-Ticket: PMAT-1095 Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01J9cSQynVPYeUkQ2i7ccvrs * fix(gpu): test_alloc_oversize_100gb pins the DEVICE allocator — managed memory oversubscribes by design on GB10 Measured on gx10: with the default routing, a 2x-the-device (257 GB vs 128 GB) allocation SUCCEEDED because GpuBuffer::new routes integrated parts to cuMemAllocManaged (PMAT-769), which oversubscribes by design. The falsifiable property is "cuMemAlloc refuses more than the device", so the test now sets MANAGED_MEMORY=0 (under the exclusivity lock that already covers env mutation) and asserts against the device allocator explicitly. Pmat-Ticket: PMAT-1095 Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01J9cSQynVPYeUkQ2i7ccvrs * fix(gpu): test_cublas_gemm_f16_training_shape reports throughput instead of asserting a 50 TFLOP/s wall-clock floor Measured 2026-09-09 on gx10: 15.4 TFLOP/s inside the full --features cuda suite, passing in isolation -- a wall-clock assertion failing under load, the class already burned four times (NO wall-clock assertion in a required check). Correctness stays asserted via the two assert_eq! on C[0,0] and C[last]; the speed claim belongs to the beat/bench lane. The measurement itself is still asserted valid (finite, > 0). Pmat-Ticket: PMAT-1095 Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01J9cSQynVPYeUkQ2i7ccvrs * fix(gpu): test_alloc_oversize_100gb must not ask a unified-memory part for 2x physical — it OOMed and rebooted gx10 The v2 test asked GB10 (128 GB unified) for a 257 GB cuMemAlloc. On a discrete card that is rejected against the VRAM pool before a page is touched; on a unified-memory part the driver tried to BACK it from system RAM. gx10 went global-OOM at 13:36 on 2026-09-09 with this test binary at the top of the OOM table, and rebooted at 13:43. A run that "passed in 34 s" earlier the same afternoon was that thrash, survived by luck. v3 branches on classify_device_memory(): discrete keeps 2x the device; UnifiedMemory asks for 2^60 bytes, which fails validation with nothing to back it from. The same property is asserted -- larger than the device must fail. memory/mod.rs re-exports classify_device_memory + DeviceMemoryClass so tests can reach the classifier. Pmat-Ticket: PMAT-1095 Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01J9cSQynVPYeUkQ2i7ccvrs * feat(fleet): scripts/cuda_rust_fleet_check.sh — per-host receipt for NVIDIA CUDA Rust readiness, with a verdict that can fail P3 of PMAT-1095. Turns "0.67 works with cuda-core / cutile on gx10, yoga, lambda-vector" from a sentence into a receipt per host, evidence/cuda-rust-fleet/<host>.json. Six probes, each PASS | FAIL | SKIP:<reason>: driver_floor >= R580 (cuda-bindings refuses older at run time; measured) toolkit_floor >= 13.1 for sm_100+, >= 13.2 for sm_8x (cutile) clang_resource_dir bindgen needs clang's OWN stddef.h; a bare libclang is not enough (measured on yoga: "'stddef.h' file not found"); discovered like cuda-oxide does — clang, then clang-N under /usr/lib/llvm-N/bin (gx10 has only clang-21, invisible to a non-interactive PATH) cudacore_probe experiments/cuda-rust-probes/cudacore: PTX load + launch + readback cutile_probe experiments/cuda-rust-probes/cutile: Tile kernel JIT + run, prints the REAL compute capability (a copied probe once printed "sm_121" on an sm_89 host) aprender_gpu_cuda the tests this ticket and O2/paiml#3066 added or repaired Verdict has THREE states on purpose: PASS only if every probe PASSed; INCOMPLETE when a prerequisite was SKIPped (never counted as a pass — "flawless on three hosts" must not be reachable by skipping the probes that would fail); FAIL on any FAIL, on zero probes (vacuity), or on a status outside {PASS,FAIL,SKIP} (a blacklist is fail-open on its complement). --self-test is a case table over that function; it caught a stale row in its own table on the first run. RESOURCE GUARD, because the first draft of this runner OOMed the only Blackwell CI host. On 2026-09-09 a cold `cargo test -p aprender-gpu --features cuda` here, alongside a test that asked a unified-memory GB10 for 2x physical RAM, put gx10 into global OOM at 13:36 (this test binary at the top of the OOM table) and it rebooted at 13:43. So P6 now: refuses while a GitHub Actions job runs on the host; refuses below 12 GiB MemAvailable (32 was first tried and is unreachable on yoga, a ~31 GiB laptop); nice -n 19 with CARGO_BUILD_JOBS capped; and runs the TEST PROCESS under `systemd-run --scope -p MemoryMax=48G` so a regression can kill only itself. MemAvailable is in the receipt. Environment death is classified, not blamed on code: a network failure (gx10 came back from its reboot with no default route — "Could not resolve host: index.crates.io") retries --offline once and then records SKIP:env, i.e. INCOMPLETE, never PASS and never a FAIL that names a CUDA Rust cause. Every probe's full log is persisted next to the receipt and pulled back from remote hosts; the first draft deleted them and reported "exit=101" with an empty reason. Two runner bugs found by running it: cargo prints the test-binary path RELATIVE to the manifest dir when CARGO_TARGET_DIR is unset (gx10) and absolute when set (lambda-vector, which hid the bug); and a silent ssh session was measured dropping mid-build ("client_loop: send disconnect: Broken pipe"), so --host mode uses ServerAlive keepalives and tees to a remote log. Receipts committed (2026-09-09): gx10-a5b5 PASS GB10 sm_121, driver 590.48.01, CUDA 13.3 — 6/6 yoga PASS RTX 4060 Laptop sm_89, driver 595.91.07, CUDA 13.3 — 6/6 noah-Lambda-Vector INCOMPLETE RTX 4090 sm_89, driver 570.207 < R580 — cuda-core and cutile SKIP; the repaired aprender-gpu tests PASS. 580 is staged in Lambda's repo; taking it needs a reboot the operator owns (installing it live replaced the userspace libs under the running 570 module and broke CUDA until rolled back — measured). The probe crates carry their own [workspace] (verified: cargo metadata --no-deps lists 79 packages, none under experiments/), so --all-features cannot drag cuda-bindings' build.rs into make coverage / make mutants. Pmat-Ticket: PMAT-1095 Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01J9cSQynVPYeUkQ2i7ccvrs * fix(PMAT-1095): address the 3/3 quorum findings — env RAII guard, stale contract claim, set -e abort path, cgroup sizing, TOCTOU, exact pins, truthful lambda-vector acceptance Phase 3 quorum (3 independent agy lanes, all do-not-implement-as-written) found eight defects; every one is addressed here and each was falsifiable against the export. adversarial.rs - MANAGED_MEMORY was set with std::env::set_var and cleared with an unconditional remove_var: a panic between them leaked the override into every later test in the process, and remove_var destroyed any value the suite was launched with. Replaced by EnvVarGuard, which records the PRIOR state and restores it on drop, including unwind. cublas_tests.rs - The doc comment still claimed "FALSIFY-CUBLAS-003: Must achieve > 100 TFLOP/s". No contract under contracts/ carries that id or any TFLOP/s floor (measured by three lanes and by grep). The comment now says what is true: throughput is reported, no floor exists anywhere, and a speed claim belongs to the beat/bench lane. Coverage did not silently drop -- it never existed outside this one wall-clock assert. scripts/cuda_rust_fleet_check.sh - `bin=$(grep … | sed … | head -1)` under set -euo pipefail: an unmatched grep aborted the whole script BEFORE the receipt was written. Now `|| true`, with an explicit not-found branch. - MemoryMax=48G was a magic number that protected nothing on a host with less than 48 GiB (yoga has ~31). Now min(48 GiB, 60% of MemAvailable), floor 4 GiB, for BOTH the build and the test process. - The pgrep Runner.Worker check was TOCTOU: a CI job could start after it. The build now runs under the same cgroup, and the check is repeated after the build, right before the test runs. - --self-test exercised only the verdict function; the python receipt writer was untested. It is now a function, and the self-test drives it with synthetic probes and asserts the JSON (verdict, probe statuses, typed mem field). That new case immediately caught a real bug: bash arrays cannot ride a command's env prefix (they arrive as the literal string "(a b)"). - Receipts carry `blocked_on`: the SKIP reasons, so an INCOMPLETE names its blocker in the artifact itself rather than in a PR comment. - bashrs: 0 errors (the guarded rm -rf was SEC011). experiments/cuda-rust-probes - cuda-core / cutile pinned exactly (=0.3.1); a diagnostic that drifts with a minor release is not a diagnostic. Cargo.lock committed for both. docs/roadmaps/roadmap.yaml - The lambda-vector acceptance line was contradicted by its own receipt. It now says what is true and still falsifiable: PASS, or INCOMPLETE with driver_floor as the ONLY skipped prerequisite and blocked_on naming the reboot that activates the staged 580 driver. An INCOMPLETE that names its blocker is the truthful artifact; a PASS reached by skipping the probe that would fail is not. Verified on the RTX 4090: both repaired tests pass; clippy -D warnings clean under --features cuda; --self-test PASS; check_roadmap_diff_additive.sh PASS. Pmat-Ticket: PMAT-1095 Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01J9cSQynVPYeUkQ2i7ccvrs * evidence(fleet): receipts after the quorum fixes — gx10 PASS, yoga PASS, lambda-vector INCOMPLETE (blocked_on: driver 570 < R580, reboot activates staged 580) Re-run with the fixed runner on all three hosts. yoga now PASSes 6/6 with the cgroup sized to 16G on a 28 GiB host (the dynamic MemoryMax from the quorum finding). lambda-vector stays INCOMPLETE and the receipt itself names the blocker in blocked_on. Pmat-Ticket: PMAT-1095 Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01J9cSQynVPYeUkQ2i7ccvrs * fix(PMAT-1095): round-2 quorum — real contract name in the cublas module doc, blocked_on names the reboot and lists root causes only, memory floor fails CLOSED F2: cublas_tests.rs line 6 cited cublas-gemm-v1.yaml (FALSIFY-CUBLAS-001/-003/-005); neither the file nor the ids exist (delegate-verified). It now names contracts/fp16-cublas-gemm-v1.yaml and states no TFLOP/s floor exists. F8: the driver_floor SKIP reason names the remedy (R580+ driver package AND a reboot to activate it) and blocked_on excludes probes skipped BECAUSE of another prerequisite, so lambda-vector lists exactly one root blocker. N1 (new, delegate-measured): with MemAvailable unreadable, awk printed nothing and exited 0, `|| echo 0` never fired, and the 12 GiB floor fell THROUGH to the build -- fail-open. The floor is now a pure function that refuses empty or non-numeric input, self-tested with "", "x", "0", "5", "11" (refuse) and "12", "20", "115" (allow). Pmat-Ticket: PMAT-1095 Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01J9cSQynVPYeUkQ2i7ccvrs * fix(PMAT-1095): drop the last non-existent FALSIFY-CUBLAS id; make the memory floor glob-free and the self-test loops parseable (bashrs 0 errors) The previous commit pushed with three bashrs errors (BRS0010 on a [!0-9] glob it reads as a test bracket, SC2135 on two one-line for/if loops) and one remaining FALSIFY-CUBLAS-005 doc id that no contract carries. All three fixed; --self-test PASS. Pmat-Ticket: PMAT-1095 Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01J9cSQynVPYeUkQ2i7ccvrs * evidence(fleet): final receipts — gx10 PASS 6/6, yoga PASS 6/6, lambda-vector INCOMPLETE with one root blocker that names the reboot Re-run on all three hosts with the final runner (sha in each receipt). lambda-vector blocked_on lists exactly one root entry: driver_floor, whose reason states the R580+ package and the reboot that activates it; the two probe skips derived from it are no longer listed as separate blockers. Pmat-Ticket: PMAT-1095 Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01J9cSQynVPYeUkQ2i7ccvrs * docs(audits): impl receipt for PMAT-1095 — 0.67 CUDA Rust fleet readiness Identity, routing, dispatch ledger (4 delegate dispatches: ph1 goal partial, ph3 quorum r1 partial / r2 / r3 agreed), claimed-vs-rerun verification table, jidoka log (gx10 OOM+reboot, lambda-vector driver rollback, stray delegate worktree), estimates (K-hat 4 first-run[U], actual ~96), gaps (lambda-vector reboot, pr-review receipts NotRun, vacuous transcript-gate), verdict DONE pending required checks. Pmat-Ticket: PMAT-1095 Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01J9cSQynVPYeUkQ2i7ccvrs * docs(audits): PMAT-1095 receipt body — the previous commit wrote an empty file The earlier writer used an unquoted heredoc whose prose held a backticked apt command; the substitution hung on sudo and produced 0 lines. Written by python now. Estimates row appended. Pmat-Ticket: PMAT-1095 Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01J9cSQynVPYeUkQ2i7ccvrs * evidence(cuda-rust-fleet): receipts carry ~/ not /home/noah/ — the shipped-path ratchet was +4 on the merged tree (guard-tree red on run 34368743080) Pmat-Ticket: PMAT-1095 Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> * fix(cuda-tests): 67-B1 — capture on a NON-BLOCKING stream (GPU-ORD-5) and a fleet-neutral VRAM floor: gx10 6 FAILED (906/901 stream-capture vs legacy-stream H2D across concurrent tests), yoga 2 FAILED (">20GB VRAM" on an 8 GB 4060) Measured at bb0b32e inside the disposable-runner image with the host toolkit mounted: gx10 2595/6, yoga 2599/2. A capture opened on a blocking stream makes every legacy-stream transfer in the process fail with CUDA_ERROR_STREAM_CAPTURE_IMPLICIT (906) and invalidates the capture (901); CudaStream::new_non_blocking removes the implicit dependency. The VRAM assertions named one host; the invariant is total >= 1 GiB. Lambda (RTX 4090): cuda_graph_tests + driver_and_context + stress_and_advanced + cublas_tests 44/44. Pmat-Ticket: PMAT-1095 Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> * evidence(67-B1): cuda lib suite receipts — gx10 and yoga GREEN at bd05c81 inside the disposable-runner image with the host toolkit mounted Pmat-Ticket: PMAT-1095 Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> * ci(release): the bullseye CUDA build left a root-owned target/ mountpoint in the workspace — pre-create it as the runner (rebuild 34488955316 died at mkdir after a green 1m54s build) Pmat-Ticket: PMAT-1096 Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> * ci(build-pool): BP-1 — the host layout is a property of the box: a first 'Host layout' step exports CI_TARGETS_ROOT / CI_CARGO_ROOT / SCCACHE_HOST_DIR / CI_REGISTRY / IMAGE with the intel clean-room defaults, 33 hardcoded sites read them; byte-identical on intel (paiml#3100) Also: check_runner_labels.sh accepts the `build` pool label. No job changes where it runs yet (BP-3). Pmat-Ticket: PMAT-1098 Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> * ci(build-pool): BP-3 — the containerized jobs and gate run on ANY x86 box carrying `build` (intel clean-room or yoga-eph); gate on any pool box (operator 2026-09-10: requeue to available capacity; paiml#3100) Pmat-Ticket: PMAT-1096 Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> * ci(build-pool): BP-3 — the containerized jobs and gate run on ANY x86 box carrying `build` (intel clean-room or yoga-eph); gate on any pool box (operator 2026-09-10: requeue to available capacity; paiml#3100) Pmat-Ticket: PMAT-1096 Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> * ci(build-pool): the pr-review-* jobs stay on clean-room — a non-required review job held yoga's only pool slot for 15 min while required jobs queued (paiml#3100) Pmat-Ticket: PMAT-1096 Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> * ci(build-pool): vendored-schemas on the any-arch pool — pure scripts, so gx10-build's idle cores may take it (paiml#3100) Pmat-Ticket: PMAT-1098 Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> * ci(build-pool): the reusable sovereign-ci jobs run on the build pool too (runs_on input, paiml/.github#67; operator: move now) Pmat-Ticket: PMAT-1096 Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> * ci(build-pool): measured-only routing — workspace-test + gate on the pool; guard-tree/guard-cargo/mutants/vendored-schemas/pr-review-* and the reusable ci jobs back on clean-room until measured (ci / security refuses to run without pmat, absent on yoga; paiml#3100) Pmat-Ticket: PMAT-1098 Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * ci: revert the ci.yml build-pool edits — routing lives in runner labels (paiml#3100); guard_tree_test.sh rejected BP-1's guard-cargo env move Pmat-Ticket: PMAT-1098 Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * fix(roadmap): PMAT-1095 sorts above PMAT-1096 — the merge of main placed main's PMAT-1096 entry first and check_roadmap_sorted.sh failed guard-tree on intel Pmat-Ticket: PMAT-1095 Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * fix(roadmap): roadmap.yaml is main byte-for-byte plus the PMAT-1095 entry — the reorder re-serialised PMAT-1096 (bytes differ, no field changed) and check_roadmap_diff_additive failed guard-tree Pmat-Ticket: PMAT-1095 Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * roadmap: main's file plus this PR's own entries — the 3-way-by-id driver had dropped ids main added * roadmap: PMAT-1095 after its numeric predecessor among PMAT peers (check_roadmap_sorted) --------- Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com> Co-authored-by: Noah Gift <claude@noahgift.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Found while building T3 (#3065).
RegisterAllocator::emit_declarationsiterated aHashMapto emit.reglines, and Rust randomises HashMap iteration order per map instance — so PTX emission was not a pure function of its inputs.Measured before the fix (gx10)
Why it isn't cosmetic
driver/ptx_cache.rskeys the cubin disk cache onsha256(patched_ptx ‖ jit_target ‖ driver_version), andgenerate_ptxis called per launch path (layer_cuda_executor.rs:71,incremental_attention.rs:94, …). A fresh key every call means the cache can never hit — every kernel is JIT-compiled from scratch, every time, and the cache grows without bound.I checked the predicted consequence on disk rather than assuming it:
~/.cache/trueno/ptx/42,880 distinct sha256 keys for a few dozen distinct kernels.
Fix
HashMap→BTreeMapfor the prefix-grouping map. Declaration order becomes sorted-by-prefix — arbitrary, but stable, which is all.regdeclarations require. One line, plus a comment on why it must not go back.After, same script and host: 5 emissions → 1 distinct output; 3 processes → identical sha256; all three hidden sizes → identical order.
Mutation proof
Reverting
BTreeMap→HashMapturns all three new tests RED; restoring turns them GREEN. Worth noting the second one: two calls on one allocator disagreed, because the map is rebuilt per call.Verification
fmt clean · clippy
-D warningsclean on default and--features cuda·cargo test -p aprender-gpu --features cuda --lib→ 2604 passed, 0 failed.Caveat that limits this guard
pub mod ptxis#[cfg(feature = "cuda")](lib.rs:150), so these tests run only with--features cuda— today that means the gx10 nightly, not the required check. Filed separately.Refs #3062
🤖 Generated with Claude Code
https://claude.ai/code/session_01J9cSQynVPYeUkQ2i7ccvrs