[bong-water-water-bong]

Root cause

q4nx_dequant_tensor() (npu/forward/src/q4nx_dequant.c:126) decoded every 2-D weight as 5120-byte q4_1 tiles and
only required nbytes >= q4nx_tensor_bytes(rows, cols). A plain bf16 tensor — what
tools/safetensors_to_q4nx.py pack --dtype bf16 writes — is 2/0.625 = 3.2× the tile size for the same shape, so it
passed that test, had two thirds of its buffer ignored, and the rest read as q4_1 tiles. Nothing refused it: the
container converted, verify passed (that compares container-vs-source), the model loaded, served and answered — on
weights that were not the model's. dequant_any() (q4nx_forward.cpp:159-168) routes by bytes-per-tile and both
branches end in the same tile decoder, so there was no bf16 path at all; 1-D norms were unaffected only because
read_bf16() reads those directly.

Reproducer, before and after

Same code both ways, 64×512 tensor of exact bf16 integers (/tmp/repro875.c in the session log):

BEFORE (unpatched decoder): rc=0  32519/32768 values wrong  max|d|=498  max|decoded|=468
AFTER  (patched decoder):   rc=0  0/32768 values wrong      max|d|=0

The rc=0 on the way in is the bug: a whole parity round was measured on this without a single error.

Fix

• new q4nx_dequant_plain_bf16(data, rows, cols, out) — the bf16 sibling of the existing q4nx_dequant_plain_i8()
  (Gemma4's plain int8 + per-32-column scale), so the "plain, no tile header" family has a named entry point;
• q4nx_dequant_tensor() takes that path when nbytes == rows*cols*2, before the 32×256 geometry test, because
  plain bf16 carries no tile grid;
• the tile path's size test becomes exact (nbytes != need → refuse), so a container family whose chunk size nobody
  taught this decoder is refused rather than read as tiles anyway.

Tests

ctest npu_dequant_bf16 (new file tests/npu_dequant_bf16_test.c, plain C, no device) — registered in
CMakeLists.txt and bound to npu_forward_loader_test with add_dependencies because this lane's token cannot write
.github/workflows/ (the same workaround as the other new tests). Nine cases, all pass:

ok   a rows*cols*2-byte tensor decodes
     0 of 32768 values differ from the bf16 source
ok   every value equals the bf16 source (not a q4_1 tile read)
ok   q4nx_dequant_plain_bf16 decodes
ok   plain_bf16 and the q4nx_dequant_tensor path agree
ok   a 3x5 bf16 tensor decodes (no 32x256 alignment needed)
ok   a tile-sized q4_1 grid still decodes
ok   the tile path yields the packed codes (1.0 / 3.0), so bf16 did not hijack it
ok   a tile-plus-2-byte buffer is refused
ok   a short buffer is refused

ctest -R "npu_dequant_bf16|npu_forward_loader|npu_design_grid|npu_q4x|npu_pack|npu_sampling|systemone|decision|npu_plan"
→ 10/10 pass. python3 tools/copyright.py --check rc 0.

On the box (host-GEMM, one npu-hold ticket per run, amdxdna errors 0 before and after)

row	before	after
engine's decoded layers.0.linear_attn.qkv_proj.weight row 0 vs the pack's own row 0, d3-lite-bf16	cos 0.12 at its best row of 6144, mean|·| 8.2× the pack	not re-probed with a temporary dump here; the container-wide rows below are the same statement
d3-lite-bf16 loads (/health)	loads, on garbage weights	loads, and now the parity moves
parity vs the pack's own d3_runtime.py, worst |Δp| (10 contract requests)	0.8382 (3/5 same argmax)	0.7319 (3/5)
the correctly-decoded q4_1 container of the same weights, same rows	0.7300 (3/5)	0.7300 (3/5)
Qwen3-0.6B (q4_1), d3-lite-conv (q4_1) load	ok	ok

So the bf16 container now behaves like a correctly decoded container instead of a mis-decoded one; the residual ~0.73 is
the separate known gap (noncausal mask + #849), not the decoder, and no accuracy claim is made here.

Known limitation, measured rather than assumed

bonsai2-27b-NPU, -b8 and -pb16i fail to load with embedding dtype I8 but dequant failed — on origin/main
before this patch as well (same worktree, patch stashed, identical message), and the temporary diagnostic shows why:

[diag] emb dtype=I8 esz=794624000 en=1271398400 NV=248320 kpad_=0 need=0 rc=-1

kpad_ is 0 when the quantised embedding is dequantized: the assignment moved to q4nx_forward.cpp:730 in #854 while
the reader sits at :559. Filed as #881 with the old/new attribution and the one-line fix, and kept out of this PR
because it is a different defect in a different region. Their own buffer sizes are exact for this rule
(esz == need == 794,624,000 for 248320×5120), so the exact-size test is not what breaks them.

Closes #875.

[github-actions]

PR Reviewer Guide 🔍

(Review updated until commit 68d8457)

Here are some key observations to aid the review process:

🎫 Ticket compliance analysis 🔶 849 - Partially compliant Compliant requirements: Fix the q4_1 zero-point nibbles pairing in npu/pack_import.cpp to match the codes pairing (even/odd instead of halves) Regenerate tests/golden/oga_attn_tile.bin fixture Non-compliant requirements: Add a check that catches the class of issue (fixture whose zero points are not symmetric under halves/even/odd swap) Requires further human verification: Verification of the regenerated golden file and the new check implementation 854 - Partially compliant Compliant requirements: Derive GEMM K and N on the container's and the kernel's grids Pad K with q4nx_pad_k() and N with q4nx_pad_n() Keep real widths beside them (real_k, real_n) Assemble_B() scatters real rows and zeroes the kernel's n-tile pad Move the M=16 twin guard from N % 1024 == 0 to the same 512 grid Fix the container-index walk that made the gate impossible Add tests for npu_design_grid, npu_forward_loader, npu_plan, npu_q4nx, npu_pack_import, npu_pack_import_oga Non-compliant requirements: None Requires further human verification: None 881 - Partially compliant Compliant requirements: Fix kpad_ being read before it is assigned in q4nx_forward.cpp Move the assignment of kpad_/impad_ before the embedding block Non-compliant requirements: None Requires further human verification: None

⏱️ Estimated effort to review: 3 🔵🔵🔵⚪⚪

🧪 PR contains tests

🔒 No security concerns identified

⚡ Recommended focus areas for review Incorrect bf16 tensor decoding The q4nx_dequant_tensor function previously decoded all 2-D weights as 5120-byte q4_1 tiles, even plain bf16 tensors. This caused incorrect decoding where two-thirds of the buffer was ignored and the rest was read as q4_1 tiles, leading to garbage values. The fix correctly identifies plain bf16 tensors by checking for exact byte size (rows * cols * 2) and routes them to the new q4nx_dequant_plain_bf16 function. if (nbytes == (size_t)rows * (size_t)cols * sizeof(uint16_t)) return q4nx_dequant_plain_bf16(data, rows, cols, out); kpad_ read before assignment The variable kpad_ was being read before it was assigned, causing quantized embedding containers to fail to load. The assignment of kpad_ and impad_ has been moved to occur before the embedding block to ensure proper initialization.

[bong-water-water-bong]

Extra row after the PR went up: the exact-size rule does not break the containers that already work — proven at inference, not at /health

Weight loads in this forward are lazy (load_layer() runs on the first step that needs a layer), so the load row
in the description only proves the embedding path. That is exactly the kind of row that can pass while every projection
still refuses, so I checked the stronger one — a real request on each container, which forces load_layer() for every
projection:

container	request	result (patched binary)
d3-lite-conv (q4_1)	POST /v1/systemone, one choice question	200, {"choice":"b","confidence":0.0957,"probabilities":{"a":0.31995,…}}
d3-lite-bf16 (bf16)	same request	200, {"choice":"b","confidence":0.1303,"probabilities":{"a":0.29077,…}}
Qwen3-0.6B (q4_1)	POST /v1/chat/completions, 4 tokens	200, reasoning_content "\nOkay,"
any	serve log	no dequant … failed, no forward init failed, no forward step failed

The two d3 containers now land within 0.03 of each other on the same question — the same weights decoded two ways,
which is the whole point of the change (before it, the bf16 container was 8.2× off in weight magnitude and
uncorrelated).

Also worth recording, because it is what made me re-check: a naive reading of the containers' headers suggests some
tensors are stored at 1.6× the tile grid
(model.layers.0.self_attn.q_proj.weight shape=[256,5120] bytes=1310720, tile grid 819,200). The loader's (rows, cols)
are not the header's shape for those tensors, which is why an exact test derived from headers alone would look
dangerous and an exact test derived from the caller's geometry is not: the three containers above pass it at the
values the loader actually uses. The one family that cannot be validated either way is the 27B int8-embedding one,
which fails on origin/main before this patch (#881) — stated in the description, and its own buffer sizes are exact
(esz == need == 794,624,000 for 248320×5120), so this rule is not what it would trip on.

[bong-water-water-bong]

VERDICT: READY

I reproduced the bug and the fix on a real bf16 container, not a synthetic buffer — ~/models/npu/d3-lite-bf16
(1,506,003,976 B, q4nx_source.converter: tools/safetensors_to_q4nx.py) against the source it was converted from
(~/models/npu/d3-lite-src/model-00001-of-00001.safetensors), decoding model.embed_tokens.weight
[248320,1024] — a tensor the forward really routes through dequant_any() → q4nx_dequant_tensor(). One harness,
compiled twice: once against this PR's q4nx_dequant.c, once against main's.

container bytes=508559360   tile-need=158924800   bf16-need=508559360      (3.2x, exactly as the body says)

main's decoder:    rc=0   254240470/254279680 values differ   max|d|=2.34174  max|ref|=0.230469
this PR:           rc=0            0/254279680 values differ   max|d|=0

So on the real weight, main decoded 254.2 M of 254.3 M values wrong, by up to 10x the true magnitude, and returned 0
— which is the part that matters: the container converted, verified, loaded and answered on those numbers. The fix returns
the tensor exactly.

Also checked:

• the new test is real and its cases bite: npu_dequant_bf16 passes 10 cases, including the two that would catch a
  regression the round trip cannot see — "every value equals the bf16 source (not a q4_1 tile read)", "the tile path
  yields the packed codes (1.0 / 3.0), so bf16 did not hijack it", and the two refusals (tile-plus-2-bytes, short buffer).
  With the tile path's size test now exact, a buffer that is neither rows*cols*2 nor the tile grid is refused instead
  of half-read.
• the neighbouring container tests are unaffected: npu_q4nx, npu_pack_import, npu_forward_loader all still pass
  (4/4 with the new test), so the stricter size test did not start refusing a family that used to decode.
• CI wiring: add_test(NAME npu_dequant_bf16 …) plus add_dependencies(npu_forward_loader_test npu_dequant_bf16_test),
  i.e. bound to a target ci.yml already builds — the documented workaround, with the comment saying when to drop it.

Two notes, neither blocking:

1. dequant_any() still refuses a 2-D tensor that is not on the 32x256 grid before reaching the new bf16 branch, so a
   plain bf16 tensor with, say, 3 rows is refused rather than decoded. A refusal is the right failure mode and the direct
   API handles it (their 3x5 case), but the sentence in docs/npu.md should not promise more than the forward path does.
2. I ran the NPU-lane tests in a configure without the 1bit serve binary, so the repo-wide smoke_serve/recipes_route
   rows are red in my log for that reason alone, not because of this change; the dequant path they exercise is the one
   covered above.

[github-actions]

Persistent review updated to latest commit 933e02b

[bong-water-water-bong]

VERDICT: READY

At 933e02bb2 (merge of main into the head I reviewed; both files blob-identical, so the content is the content I verified).
Build success at this head.

Evidence: reproduced on a real bf16 container (~/models/npu/d3-lite-bf16 vs its d3-lite-src safetensors), model.embed_tokens.weight [248320,1024]: mains decoder returns 254,240,470 of 254,279,680 values wrong, max|d| 2.34 against max|ref| 0.23, and rc=0; this head returns 0 wrong, max|d| 0. The container is 508,559,360 B against a 158,924,800 B tile grid (the 3.2x in the body). 10 new cases pass, the neighbouring container tests still pass under the now-exact size test.

[github-actions]

Persistent review updated to latest commit 4ea19ed

[bong-water-water-bong]

VERDICT: READY

At 4ea19eda3 (merge of main into the head I reviewed; 4 of 4 files blob-identical, so the content is what I verified).
Build success at this head.

Evidence stands: on the real bf16 container (d3-lite-bf16 vs its safetensors source), main decodes 254,240,470 of 254,279,680 embedding values wrong with rc=0; this head gives 0. 10 cases pass and the neighbouring container tests still pass under the now-exact size test.

[github-actions]

Persistent review updated to latest commit 5069770

[github-actions]

Persistent review updated to latest commit 68d8457