YOGA-NIGHTLY-001 R-3/R-4/R-6: the sm_89 axis, and the assertion that lets it go red - #3060
Open
noahgift wants to merge 9 commits into
Open
YOGA-NIGHTLY-001 R-3/R-4/R-6: the sm_89 axis, and the assertion that lets it go red#3060noahgift wants to merge 9 commits into
noahgift wants to merge 9 commits into
Conversation
added 3 commits
September 9, 2026 12:07
…oes not `--features cuda` appears ZERO times in the whole of silicon-nightly.yml, while the file declares an axis named `aarch64-cuda-sm121` that holds gx10 — the fleet's only Blackwell box — for a 90-minute timeout to run `cargo test -p aprender-compute --lib`. That is a CPU suite. The coverage ledger scores the axis covered; the coverage is of the CPU. .github/silicon-coverage.txt asks whether a RUNNER can serve a selector, and check_silicon_coverage.sh answers that question correctly. Neither asks what the job then RUNS. This guard asks exactly that, for cuda-named axes only. IT FAILS ON COMMITTED REALITY, TODAY: VIOLATION aarch64-cuda-sm121 axis is named `cuda` and invokes CUDA NOWHERE 1 cuda-named axis/axes examined, 1 violation(s) rc=1 which is the strongest demonstration available that it discriminates — a guard whose first act is to pass proves nothing. WHY NOW: yoga (x86_64 + Ada sm_89) is being added to the fleet, and the obvious move is to copy the sm_121 leg's shape. That would produce a SECOND `*-cuda-*` axis running no CUDA and reading green. paiml/infra's YOGA-NIGHTLY-001 §9.3 makes this guard a precondition of that work for that reason. Four exit paths, all measured: real workflow -> 1 (the violation above) --self-test -> 0 (4 fixtures) a compliant workflow -> 0 no cuda-named axis -> 2 NO-GO, never a pass That last one matters: "0 violations" over 0 axes examined is this fleet's signature bug, so a renamed axis makes the guard refuse rather than report a clean bill. The denominator is printed on every run. ONE FIXTURE PER FORM VARIANT, not per form: `--features cuda`, `--features "cuda,serve"`, `--all-features`, and a cuda-named axis with no cuda invocation at all. A detector that has only seen one spelling of a thing has not been shown to detect the thing. Not a YAML parser — same textual posture as check_silicon_coverage.sh, answering one question per job. Refs paiml/infra PMAT-272 (YOGA-NIGHTLY-001 R-6), paiml/infra#361
…ane needs to go red PMAT-272 / YOGA-NIGHTLY-001 R-3. THE PROBLEM. The Blackwell cascade (2026-05-19, 8 PRs / 7 defects / 1 root cause) found a `warm!` macro that hardcoded "silu_forward" as the cache key for EVERY kernel, so eleven-plus "pre-warmed" kernels silently JIT-compiled at runtime under one colliding entry. Lesson 4 records the part that matters here: those bugs existed on sm_89 too, and JIT-on-demand SUCCEEDED there — sm_121's stricter behaviour is what turned them into hard failures. An sm_89 pass/fail lane was therefore GREEN through all seven defects. yoga is an sm_89 box, so a verdict lane on it would inherit exactly that blindness. THE FIX. Count JIT compiles in both kernel caches, expose reset + read, and assert ZERO across a forward pass taken AFTER pre-warm. The reset boundary is the whole assertion: pre-warm legitimately compiles everything it warms, so asserting zero before it would assert that pre-warm did nothing. NOT the post-mortem's recommendation #3 (cross-architecture cache diffing), and the spec argues why: legitimate arch differences change cache keys — cuBLAS on sm_121 versus PTX GEMM on sm_89 IS cascade defect #1804 — yoga and gx10 also differ in CUDA toolkit (12.4 vs 13.0), and a transcript nobody must read rots. This invariant is local: one machine, no confound, and a verdict that fails. MY FIRST VERSION WAS TAUTOLOGICAL AND THE MUTATION CAUGHT IT. It did pre-warm -> reset -> PRE-WARM AGAIN -> assert 0. With the key hardcoded, the second pre-warm writes the same colliding key, finds it cached, reports zero misses, and passes. It compared pre-warm against pre-warm — the same key construction on both sides — so it could not see a defect that lives in the DIFFERENCE between pre-warm keys and runtime keys. That is post-mortem Lesson 5 in miniature: a contract that cannot fail under any execution is a contract bug. Replaced with a real `fused_residual_rmsnorm_forward` pass. MEASURED ON REAL sm_89 HARDWARE, both directions: correct tree -> PASS `warm!` hardcoded key reintroduced -> FAIL, "a real forward pass JIT-compiled 1 kernel(s)" The failure names the kernels: [FWD-CACHE]/[BWD-CACHE] print each one as it compiles, so a red is actionable rather than a bare count. Both caches are instrumented. The backward one is not optional: cascade defects #1810 (pre_warm_lora_backward short-circuiting at lora_rank==0) and #1813 (rms_norm_gamma_reduce stage 2 missing) were BACKWARD kernels, so a forward-only counter would have missed two of the seven. Skips loudly with no CUDA device rather than passing or failing — cuda-nightly selects this test by name, and a CPU box must not report a result it did not measure. Refs paiml/infra PMAT-272 (YOGA-NIGHTLY-001 R-3, acceptance test §9.7)
… to delete RESTORING A DELETION, NOT INVENTING A LANE. #2740 removed the x86_64 GPU leg because it ran on lambda-labs, which must never be a CI host (infra#359). That was correct, and it left a named hole: .github/silicon-coverage.txt has carried `x86_64-cuda-sm89 pending:361` ever since. infra has now stood up `yoga` — RTX 4060 Laptop, AD107M, sm_89, 8188 MiB, CUDA 12.4 — as an org-scoped runner in the `gpu-x86` group, and aprender is in that group's visibility list. Spec: paiml/infra docs/specifications/yoga-nightly-job.md (YOGA-NIGHTLY-001 R-4). A SEPARATE JOB, NOT A MATRIX ARM. `runs-on` in `falsifiers` is literal on purpose — infra#352's guard can only check labels it can SEE, and labels behind a matrix expression are invisible to it. A second architecture therefore needs a second job. Same rule applied here, which is also why `ada` and `yoga` are added to .github/actionlint.yaml rather than the selector being hidden to dodge it. NOT A COPY OF `falsifiers`, and that is the point. Post-mortem Lesson 4: the pre-warm bugs existed on sm_89 too, but JIT-on-demand SUCCEEDED there — sm_121's stricter behaviour is what turned them into hard failures. An sm_89 lane whose only output is pass/fail was GREEN through all seven defects of the Blackwell cascade. So this leg asserts the thing that can go red on sm_89: falsify_cuda_prewarm_covers_runtime_no_jit_001, mutation-proven in both directions on real sm_89 hardware. WHAT IT DELIBERATELY DOES NOT DO: - No QLoRA. 8 GB and a mobile Max-Q part; gx10 owns training. - No performance assertion. SILICON_FLOORS is keyed by compute capability and yoga answers `8.9` — the same key as the 24 GB / ~450 W desktop 4090 the floor was derived from. UNCALIBRATED-SILICON cannot fire because the key matches while the device does not (YOGA-NIGHTLY-001 §7). - No yield-to-training predicate. gx10 has a competing workload; yoga does not. The preflight ASKS THE BOX, never the label: it asserts compute_cap 8.9 and x86_64 before running anything. This gate has already been wrong once about which host it was on (#2835, an sm_89 floor asserted on GB10, red four nights), so routing is not taken on trust. The VRAM step records peak usage rather than asserting throughput. 8188 MiB is the fleet's tightest x86 CUDA budget, and the cascade's two INDEPENDENT bugs were memory bugs (#1804, #1808) — latent on GB10's unified memory and on a 24 GB 4090, immediate at 8 GB. Refs paiml/infra PMAT-272 (YOGA-NIGHTLY-001 R-4), paiml/infra#361, #2740
|
§13.11 rung 1 — quorum shadow verdict Shadow mode: this records a verdict and merges nothing. A refusal |
noahgift
marked this pull request as ready for review
September 9, 2026 10:33
noahgift
enabled auto-merge
September 9, 2026 10:33
…an axis
R-5 of YOGA-NIGHTLY-001 (paiml/infra). The recommendation came out of this
guard's own output, which is the reason to trust it: infra#486 registered
`yoga-gpu` with labels gpu,yoga,cuda,ada and check_silicon_coverage.sh
immediately said
PROMOTE x86_64-cuda-sm89 marked pending:361, but yoga-gpu can serve it NOW
on a fleet where NOT ONE JOB had ever carried that selector. Promoting the ledger
line on that evidence is precisely the blanket exemption the ledger exists to
prevent: the axis would have read `required … ok` for ever while nothing
executed on the box. A machine existing is not a test running.
So the question changes from "could a runner serve this?" to "did a job carrying
it CONCLUDE inside its cadence window?", and the answer has five states instead
of two:
ok a job carried it, inside the window
STALE it ran, then stopped -> red
UNCOVERED a runner CAN serve it and nothing ever has -> red (new)
MISSING no runner and no run -> red
ready deferred, runner exists, nothing has run -> green (new)
`ready` is the behaviour that CHANGED, and it is the point: PROMOTE now requires
a RUN. `x86_64-cuda-sm89` therefore stops failing this guard on yoga's mere
existence and starts failing it the night the ada-yoga leg first concludes —
which is when the ledger line is actually a lie.
AND THE FIRST LIVE RUN OF THE PROBE FOUND THE NEXT ONE DOWN:
ok x86_64-cpu failure 2026-09-09T09:00:26Z 0d ago — Silicon Nightly / summary
`summary` is silicon-nightly's needs-aggregator. It runs on the intel pool and
executes nothing, so it carries `self-hosted,clean-room,intel` perfectly — as
does every one of the 100+ other jobs that touched intel in the same window. A
POOL LABEL SET DOES NOT NAME AN AXIS, and scoring coverage off one means
x86_64-cpu reads green for as long as anything at all touches intel. Same shape
as the defect above, one level down: a probe answering an adjacent question.
Hence the policy's optional 4th column, `job:<glob>`. It is fail-safe by
construction — rename the job and the axis goes UNCOVERED (red), never silently
green — and an axis without one is stamped `[labels only]` in its own verdict
line rather than passing quietly. The real leg declares
`name: ada-yoga (x86_64, sm_89)`, so the column is `job:ada-yoga*`; a fixture
written against the job ID alone would have been green while the live guard
scored the axis UNCOVERED.
MEASURED, on the live fleet, before and after:
before ok x86_64-cpu … / summary (the aggregator)
after ok x86_64-cpu … / x86_64-cpu (the test job)
5 axes, 18 online runners, 125 concluded jobs, 60 runs scanned of 197 listed
TWO RULES BORROWED FROM paiml/infra's dead-man's switch, both load-bearing:
* ASK THE API FOR THE EVENT. `runs?per_page=100` is the newest 100 runs of ALL
events; on this repo that is a few hours of pull_request traffic and can hold
zero scheduled runs. That page once made the switch report a weekly lane DEAD
whose last two scheduled runs had both succeeded.
* CADENCE IS max(3d, 2x the axis's own observed interval). The floor is what
makes a brand-new axis judgeable at all; D-2 (operator: "yoga should test
nightly") makes 3 days right for every axis here.
`success` and `failure` both count as EXERCISED — a red nightly ran the silicon
and its own lane says so. `cancelled`, `skipped` and a null conclusion do not:
nothing executed, which is the state this guard exists to make visible. A job
queued for ever against a purged runner registration is `null`, and that is the
~14-day silent failure the whole lane is built around.
DENOMINATORS AND POSITIVE CONTROLS. The job ledger has its own NO-GO: an empty
ledger would score every axis UNCOVERED, which is indistinguishable from a guard
that read nothing. Zero over zero is this fleet's signature defect, so it exits 2
rather than reporting an absence it never looked for.
SIX FIXTURES AND A MUTATION (§9.2). tests/fixtures/silicon-coverage/ substitutes
both GitHub listings so the whole classification runs offline —
covered / uncovered / wrongjob / stale / ready / promote, one per form variant.
Then the run probe is DELETED between its committed seam markers and both red
cases must go GREEN; if they stay red, something other than the probe decides and
the change is decoration. The mutator refuses a byte-identical copy, because a
mutation that matched nothing would "prove" the probe by re-running it.
The ledger line is NOT promoted here. That is step 9 of §12 and it comes after
the leg has actually run (N-1).
Refs paiml/infra PMAT-272 (YOGA-NIGHTLY-001 R-5, §9.2), paiml/infra#361, #486
…t exist, a
cuda axis with no cuda, and three guards nothing ran
R-3 / R-6 / R-8 of YOGA-NIGHTLY-001 (paiml/infra), which are one commit because
they are one lane and each fix uncovered the next.
R-8 — SILICON-NIGHTLY HAS NEVER BEEN GREEN, and the reason is two characters of
nobody's attention:
error: package ID specification `aprender-primitives` did not match any packages
##[error]Process completed with exit code 101
in BOTH legs, every scheduled run since it merged on 2026-08-29. There is no
`aprender-primitives` in this workspace and there never has been — `git log -S`
finds it introduced by the workflow itself (#2772) and nowhere else. The lane
that exists to prove x86_64 and aarch64 are still tested has proved nothing on
either, while the ledger it guards reported both axes covered. Fixed to
`aprender-core`, the second crate carrying arch-conditional code (10 files with
target_arch/target_feature, behind aprender-compute's 110).
THE CLASS, NOT THE SYMPTOM: a package name in a workflow is a reference into the
workspace with NOTHING checking it. Every other reference in this repo is
compiled or linted; this one is a string, and it fails at runtime, nightly, on a
self-hosted runner, hours after the merge, in a lane nobody reads until someone
asks why it is red. scripts/check_workflow_cargo_packages.sh compares every
static `-p` / `--package` in every workflow against `cargo metadata`. Its first
act on the tree as committed was to fail:
VIOLATION silicon-nightly.yml:75 `-p aprender-primitives` names no workspace member
VIOLATION silicon-nightly.yml:101 `-p aprender-primitives` names no workspace member
15 workflow file(s), 72 static package reference(s), 1 dynamic (skipped), 2 violation(s)
It reads LOGICAL lines, not physical ones, because a `cargo test \` continued
onto the next line puts the `-p` on a line with no `cargo` on it — a detector
that cannot see a form is indistinguishable from a repo that does not contain it.
It strips comments first: three of its first five findings were sentences in
ci.yml quoting `cargo test -p X`. Tokens carrying `$` are counted and printed,
never silently dropped, and 0 references over 15 files is a NO-GO, not a pass.
R-6 — THE CUDA AXIS RAN NO CUDA. `--features cuda` appeared ZERO times in
silicon-nightly.yml while `aarch64-cuda-sm121` held gx10 — the fleet's only
Blackwell box — for a 90-minute timeout to run a CPU test suite. The ledger
scored the axis covered; the coverage was of the CPU. check_silicon_cuda.sh
(shipped last commit, wired here) is the guard; this is the fix. The leg now
asks the box first (`compute_cap == 12.1`, aarch64 — a label says what we
intended, nvidia-smi says what is there) and then asserts `cache_misses == 0`
after pre-warm.
R-3 — THE SAME ASSERTION ON gx10, which the spec asks for in as many words: "on
EVERY GPU runner — gx10 included". It was on yoga only. The cascade's root cause
(`warm!` hardcoding one cache key) SUCCEEDED on sm_89 — slowly — and failed hard
on sm_121, so a pass/fail suite cannot see the cause on the architecture where
it is silent and sees only the consequence on the other. `cache_misses == 0` sees
the cause on both. Cheap: cuda-nightly's falsifiers job has already built
`aprender-train --features cuda --release` two steps earlier.
AND THE REPO'S OWN GUARDS CAUGHT TWO DEFECTS IN LAST COMMIT'S ada-yoga LEG:
* `check_perf_concurrency_groups.sh`: the job declared no job-level
`concurrency.group`, so it serialised against nothing. Now `perf-yoga`,
which is only a valid group because yoga is now declared in
scripts/perf-matrix.yaml — as an NA host with NO performance cell, for the
reason §7/N-4 gives: SILICON_FLOORS is keyed on compute capability and yoga
answers 8.9, the same key as the 24 GB / ~450 W desktop 4090 the floor came
from. UNCALIBRATED-SILICON cannot fire because the key MATCHES while the
device does not, so any number measured there would be compared against a
floor from another machine (R-7).
* `runs-on` did not name `yoga`, while .github/silicon-coverage.txt declares
the axis as `self-hosted,gpu,yoga,cuda,ada`. Under R-5's run probe an axis is
covered only when a job asked for a label set CONTAINING the selector — so
the leg would have run every night and the ledger would have read UNCOVERED
for ever. A job and a policy describing the same machine in two vocabularies
is how this runner got lost in the first place.
WIRING, WHICH IS THE OTHER HALF OF EVERY GUARD HERE. All three offline guards
now run in ci.yml as well as in silicon-nightly's preflight. A guard whose only
home is a red nightly is a guard nobody reads, and this lane is the proof: it
carried check_silicon_coverage.sh for eleven days while failing on a package
name for eight of them. Two of the three were RED on the tree as committed when
they were written, which is the strongest available evidence that they can fail.
MEASURED AFTER, all local, all offline except the last:
check_workflow_cargo_packages.sh 74 refs, 0 violations
check_silicon_cuda.sh 1 cuda-named axis, 0 violations
test_silicon_coverage_run_probe.sh 6 fixtures, 0 failed, 1 mutation, 0 survived
check_perf_concurrency_groups.sh 38 jobs, 6 perf-sensitive, 0 findings
check_perf_matrix_schema.sh 5 hosts, 9 arms, 15 cells, 3 anchors
check_shell_lint_ratchet.sh 178 scripts, 8 errors, unchanged
Refs paiml/infra PMAT-272 (YOGA-NIGHTLY-001 R-3, R-6, R-8), paiml/infra#361, #2772
… files —
seven PRs of one defect, now one CPU test
R-2 of YOGA-NIGHTLY-001 (paiml/infra). The Blackwell cascade — PMAT-698e through
PMAT-698p, plus FALSIFY-CUDA-FUSED-RMSNORM-DEADLOCK-001 — was ONE sentence
restated seven times: `pre_warm_for_model` compiled kernels under keys the
runtime never asked for. The forms are all in the record:
PMAT-698j `warm!` hardcoded the literal "silu_forward" for EVERY kernel, so
eleven-plus modules collided on one HashMap entry and only the
first was ever stored.
PMAT-698k the pre-warm RMSNorm key omitted the `_eps{bits:08x}` suffix the
runtime key carries.
PMAT-698n it then warmed the wrong epsilon — 1e-5 (Llama) while the model in
flight was Qwen2 at 1e-6.
PMAT-698p RoPE was warmed at seq_len=1 while the corpus phase ran at 256.
FALSIFY-… batched_fused_residual_rmsnorm had no pre-warm entry at all.
Each was found on hardware, each cost a PR, and each fix was local. The shared
cause is structural and was never addressed: TWO `format!` STRINGS, IN TWO
FILES, WITH NOTHING TYING THEM TOGETHER. On sm_121 a mid-forward JIT poisons the
stream and fails hard; on sm_89 it simply compiles on demand and SUCCEEDS, which
is why an sm_89 pass/fail lane was green through all seven.
WHAT THIS DOES. `autograd::cuda_forward_keys` is the only place in the crate that
may build a cache key: one constructor per key FORM, plus `prewarm_keys()` and
`forward_runtime_keys()` as sets. All 33 pre-warm sites and all 17 dispatch sites
now call it — there is no `format!` of a cache key left in the module.
The module is deliberately NOT behind `#[cfg(feature = "cuda")]`. That is the
entire point: the property
prewarm_keys(spec) ⊇ forward_runtime_keys(shape, seq_len, eps)
is asserted by a CPU test over four real model shapes x cuBLAS present/absent x
both corpus epsilons — 16 cases, 284 runtime keys — in microseconds, with no GPU,
no CUDA toolkit, no model and no second machine. `cargo nextest run --workspace
--lib` already carries it, so it runs on intel on every PR. R-3's `cache_misses
== 0` falsifier catches the same defect on hardware, after the fact, on two
nightly lanes; this catches it before the merge.
FIVE FALSIFIERS, ONE PER RECORDED DEFECT, each reintroducing the historical bug
against the property and requiring it to be caught:
prewarm_defect_pmat_698j_hardcoded_key_is_caught (one-entry cache)
prewarm_defect_pmat_698k_missing_eps_suffix_is_caught (suffix dropped)
prewarm_defect_pmat_698n_wrong_epsilon_is_caught (1e-5 vs 1e-6)
prewarm_defect_pmat_698p_rope_seq_len_is_caught (warm 1, run 256)
prewarm_defect_missing_fused_residual_rmsnorm_is_caught (no entry at all)
A detector that has only ever seen one spelling of a thing has not been shown to
detect the thing.
AND THE MODEL IS TIED TO REALITY, because a pure list nobody checks is the
SECOND hand-maintained list this change exists to abolish. `pre_warm_for_model`
now records the keys it actually warms and compares them to `prewarm_keys()`
before returning, failing in BOTH directions: modelled-but-not-warmed means the
CPU test is passing on kernels that were never compiled; warmed-but-not-modelled
means a kernel the test has never seen — exactly the blind spot that let five
defects ship. Cost: one BTreeSet of ~40 short strings per cache init, beside ~40
PTX JIT compiles.
MEASURED ON REAL sm_89 HARDWARE (RTX 4090, CUDA 12.8), not argued:
cargo test -p aprender-train --features cuda --lib -- --include-ignored \
--exact …::falsify_cuda_prewarm_covers_runtime_no_jit_001
[CUDA] Pre-warmed 30 forward kernels … ok
and the tie was PROVEN ABLE TO FAIL by deleting one key from the model:
pre-warm key model drift: 0 modelled but not warmed []; 1 warmed but not
modelled ["silu_forward"] FAILED
then reverted. A guard never shown failing is not evidence of anything. The tie
then earned its keep immediately: `pre_warm_for_model` was at cognitive 35
BEFORE this branch, against a threshold of 25, and the tie took it to 38 — so
the NF4 forward and backward blocks are now their own methods (18/less each,
mirroring `keys::nf4_keys`). The hardware run above is the proof that
restructuring 57 lines of `warm!` calls dropped nothing: 30 kernels before, 30
after, and a dropped or duplicated warm is exactly what the tie reports.
DENOMINATORS. The property test's floors are derived, not typed: `cases` is
4x2x2 exactly, and the key count must be >= 15 per case (2 norms + 1 RoPE + 6
attention + 2 FFN + 4 NF4). A refactor that made `forward_runtime_keys` return
almost nothing would keep the property trivially true, and that floor is what
refuses it. The first draft of those two numbers was invented — 300 and 20 — and
both were wrong against the measurement (284 and 18), which is its own small
argument for the rule.
crates/aprender-test-lib/perf-matrix.vendored.yaml is re-synced because
PMAT-958's build script requires it to match byte-for-byte after the previous
commit added yoga as a host.
Refs paiml/infra PMAT-272 (YOGA-NIGHTLY-001 R-2, §5.3), paiml/infra#361
…and its own step name was the first casualty
Wired in the previous commit, it immediately reported two violations that cannot
be fixed:
VIOLATION ci.yml:1196 `-p in` names no workspace member
VIOLATION silicon-nightly.yml:76 `-p in` names no workspace member
Both are the step name the same commit added:
- name: "Every cargo -p in a workflow names a real crate (R-8)"
which contains `cargo` and `-p in`. A `name:`, an `if:` or a comment DESCRIBING a
command is not a command, and a finding nobody can act on is how a gate teaches
people to bypass it — this repo has that lesson recorded twice.
The extractor now reads `run:` BLOCKS only, delimited by indentation, which is
the only thing that delimits a YAML block scalar. The single-line form
(`- run: cargo …`) emits the remainder of the same line; the block form takes
every line indented deeper than the `run:` key. Continuation joining and comment
stripping are unchanged.
TWO NEW FIXTURES, one per newly discovered FORM VARIANT — a `name:` and an `if:`
whose text quotes a cargo invocation — because the rule this repo keeps
relearning is that a detector needs a fixture per variant, not per form, and
this variant was found in production rather than imagined.
MEASURED:
before 15 workflows, 76 static refs, 2 violations (both false)
after 15 workflows, 74 static refs, 0 violations
self-test: 7 forms, 4 references found; skipped `mkdir -p`, a `name:` and an `if:`
AND IT STILL CATCHES THE DEFECT IT WAS BUILT FOR. Reintroducing
`-p aprender-primitives` into silicon-nightly.yml:
VIOLATION silicon-nightly.yml:107 `-p aprender-primitives` names no workspace member
then reverted. A guard that stopped reporting false positives by reporting
nothing would look identical from the denominator alone, which is why the
denominator is printed and why this was checked rather than assumed.
Refs paiml/infra PMAT-272 (YOGA-NIGHTLY-001 R-8), paiml/infra#361
…ards for one class become one #2793 landed on `main` while this branch was open and fixed the SAME root cause from the same evidence: `-p aprender-primitives`, a package that has never existed, on both silicon legs. Independent convergence on a defect is a good sign about the defect. It is a bad state to leave in the tree. WHAT #2793 CONTRIBUTED, AND IS KEPT: * `cargo metadata --locked`. Not cosmetic — WITHOUT it, `cargo metadata` REWRITES Cargo.lock (#2793 measured 44 added `[[patch.unused]]` lines here). A read-only guard has no business mutating a lockfile, and mine was doing exactly that. Now `--locked --manifest-path`, and verified read-only: Cargo.lock is byte-identical across a full run. * `- name: Put the runner's Rust toolchain on PATH` in the `coverage` job. My wiring omitted it, and `check_workflow_cargo_packages.sh` NO-GOes (exit 2) without cargo rather than reporting a clean bill — so my step would have refused to run in the very job I put it in. #2793's step is the reason its guard worked; it is now the reason mine does. * Its prose about WHY the check belongs in the first, blocking, CPU job. WHY ONE GUARD AND NOT TWO. `check_silicon_packages.sh` reads `silicon-nightly.yml` with a flat `grep -oE '-p\s+\S+'` over every line. `check_workflow_cargo_packages.sh` reads `run:` BLOCKS only, across all 15 workflows, joining `\` continuations and stripping comments. That difference is not stylistic, and it is not hypothetical — run against this branch's resolved workflow, #2793's guard says: ::error:: silicon-nightly.yml tests `-p in`, which is not a package 4 package reference(s), 1 unresolvable `-p in` is the step NAME `Every cargo -p in a workflow names a real crate`. The narrower guard goes red on the wider guard's wiring, with a finding nobody can act on — and an unactionable gate is one people learn to bypass, which this repo has recorded twice. Keeping both would mean two implementations of one predicate kept in sync by hand, which is the root cause of bashrs#266 restated. So the narrow one is retired and every assertion it made survives in the wide one, which additionally covers: `--package` / `--package=` / `-p=`, dynamic `${{ }}` tokens (counted, never silently dropped), `mkdir -p` (not cargo's `-p`), and a `name:`/`if:` quoting cargo. Seven form fixtures against its two. MEASURED AFTER THE MERGE, all local: check_workflow_cargo_packages.sh 15 workflows, 74 refs, 0 violations, Cargo.lock unchanged check_silicon_cuda.sh 1 cuda-named axis, 0 violations test_silicon_coverage_run_probe.sh 6 fixtures, 0 failed, 1 mutation, 0 survived check_silicon_coverage.sh --self-test PASS check_perf_concurrency_groups.sh PASS check_workflow_env_defined.sh PASS check_runner_labels.sh PASS check_shell_lint_ratchet.sh 178 scripts, 8 error lines, unchanged The `aarch64-cuda-sm121` CUDA step and preflight (R-6) and the run-probe wiring (R-5) are kept from this branch; `main` had neither. Refs paiml/infra PMAT-272 (YOGA-NIGHTLY-001 R-6, R-8), #2793
6 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Implements R-3, R-4 and R-6 of paiml/infra
docs/specifications/yoga-nightly-job.md(paiml/infra#493). Draft — see Not done at the bottom.infra has stood up
yoga: RTX 4060 Laptop, AD107M, sm_89, 8188 MiB, CUDA 12.4, org-scoped in thegpu-x86group, which aprender is already in. This restores the x86_64 CUDA axis #2740 had to delete when its leg ran on lambda-labs (infra#359).The problem an sm_89 lane has
Post-mortem Lesson 4: the pre-warm bugs existed on sm_89 too, but JIT-on-demand succeeded there — sm_121's stricter behaviour is what turned them into hard failures.
So the leg's value cannot be "did the tests pass".
R-3 — the assertion that can go red
cache_misses == 0after pre-warm, asserted across a real forward pass. Both caches instrumented — the backward one is not optional, since cascade defects #1810 and #1813 were backward kernels.Deliberately not the post-mortem's recommendation #3 (cross-architecture cache diffing): legitimate arch differences change cache keys — cuBLAS on sm_121 vs PTX GEMM on sm_89 is cascade defect #1804 — yoga and gx10 also differ in toolkit (12.4 vs 13.0), and a transcript nobody must read rots. This invariant is local: one machine, no confound, a verdict that fails.
My first version was tautological, and the spec's own mutation test caught it before it shipped. It did pre-warm → reset → pre-warm again → assert 0. With the key hardcoded the second pre-warm writes the same colliding key, finds it cached, and passes. It compared pre-warm against pre-warm — the same key construction on both sides. That is post-mortem Lesson 5 in miniature. Replaced with a real
fused_residual_rmsnorm_forwardpass.Measured on real sm_89 hardware, both directions:
warm!hardcoded key reintroducedR-4 — the
ada-yogalegA separate job, not a matrix arm:
runs-onis literal because infra#352's guard can only check labels it can see. Same reasonada/yogago into.github/actionlint.yamlrather than the selector being hidden to dodge the lint.Preflight asks the box, never the label — asserts
compute_cap 8.9andx86_64before running anything. This gate has already been wrong once about which host it was on (#2835: an sm_89 floor asserted on GB10, red four nights).No QLoRA (8 GB, mobile part — gx10 owns training). No perf assertion:
SILICON_FLOORSis keyed on compute capability and yoga answers8.9, the same key as the 24 GB / 450 W desktop 4090 the floor came from, soUNCALIBRATED-SILICONcannot fire. No yield-to-training predicate — yoga has no competing workload.R-6 — an axis named
cudamust invoke CUDA--features cudaappears ZERO times insilicon-nightly.yml, while it declaresaarch64-cuda-sm121, which holds gx10 for a 90-minute timeout to runcargo test -p aprender-compute --lib— a CPU suite. The ledger scores the axis covered; the coverage is of the CPU.The new guard fails on committed reality today (
rc=1), which is the strongest demonstration available that it discriminates. Four exit paths measured, including NO-GO (rc=2) when no cuda-named axis exists — "0 violations over 0 axes" is the shape it exists to refuse. One fixture per form variant:--features cuda,--features "cuda,serve",--all-features, and a cuda-named axis with none.Not done
pre_warm_keys ⊇ runtime_keysas a CPU-only property test) needs key construction extracted out ofcfg(feature = "cuda")— a real refactor, not attempted here. R-3 covers the same defect class on a GPU box.SILICON_FLOORS) are untouched; R-7 depends on unmerged fix(beat): the ollama decode floor was calibrated on sm_89 and is now asserted on sm_121 (#2835) #2838.cargo fmtclean,clippy --features cuda0 errors,actionlintclean repo-wide.Refs paiml/infra PMAT-272, paiml/infra#493, paiml/infra#361