[joelteply]

The resident base on the M5 (Qwen3.6-35B-A3B) could not be trained: MUL_MAT_ID had no
backward, so building the gradient graph of any MoE model aborted in
ggml_build_backward_expand ("no backward through MUL_MAT_ID"), inside the serving engine
that /train runs in. A LoRA on attention needs the activations' gradient through every later
expert FFN, with the experts themselves frozen (and usually quantized).

• GGML_OP_MUL_MAT_ID_BACK / ggml_mul_mat_id_back(as, grad, ids):
  dst[:, i, t] = sum_r as[:, r, ids[i, t]] * grad[r, i, t], one row per routed slot, so each
  dst row has exactly one writer. A forward whose b broadcast over the slots sums them back with
  ggml_repeat_back (whose tiling is mul_mat_id's i % n_b).
• CPU kernel: task 0 groups the (slot, token) pairs by expert, as the forward does; each task
  takes whole experts, dequantizes one in GGML_OUT_PROD_KT-row tiles once (the out_prod tiling)
  and adds each tile row, scaled by its gradient, into the slots routed to it. F32, F16 and
  every type with to_float. The gather form (src[3] expert pointers) is refused, as for the
  forward.
• MUL_MAT_ID backward: only the activations' gradient. A gradient for the experts asserts,
  because they train frozen. ids (src[2]) and the gather's pointer table (src[3]) are never
  differentiated.
• GET_ROWS_BACK on batches: a MoE gathers its router weights per token
  ([1, n_expert, n_tokens] by [n_used, n_tokens]); the constructor and the CPU F32 kernel
  scatter-add batch for batch (2-D unchanged). Metal and Vulkan supports_op accept only the 2-D
  form their kernels compute; CUDA already did.

Upstream ggml-org#22704 (closed) was not ported: with quantized experts it ignores src1 too, cutting
the gradient through the MoE (a truncated, biased gradient), and it turns the in-place assert
into warn-and-skip.

Tests (test-backend-ops):

• grad: test_mul_mat_id_grad (per-slot and broadcast b, two KT tiles) and test_get_rows_grad
  (batched, every row gathered), each weighted by a fixed random tensor before the SUM loss.
• eval: test_mul_mat_id_back across F32, F16, Q4_0, Q8_0, Q4_K and Q6_K experts, and batched
  test_get_rows_back.
• Mutations (enrico, CPU, 2026-10-31):
  • dropping the tile offset k0: 8/8 grad lines FAIL;
  • writing every slot to slot 0: 8/8 FAIL;
  • dropping get_rows_back's batch index: FAIL, MAA 0.94.
• Full MUL_MAT_ID and GET_ROWS grad suites on CPU: 0 FAIL.

Found on the way, not changed here: the grad metric (a-b)/(a+b) is NaN where both are 0, and a
NaN mean never exceeds the threshold. So any case with an untouched row passes vacuously,
including every existing GET_ROWS grad case. The tests above gather every row, so they can't.

🤖 Generated with Claude Code

https://claude.ai/code/session_01FRQWzgSfo79JtnwZywHKrE