Fix vocab tiling hidden-state cotangent scaling - #4959
Conversation
There was a problem hiding this comment.
Code Review
This pull request ensures that the hidden-state gradients in vocabulary tiling correctly scale with the outer loss scale by multiplying grad_reshaped_hidden_states by loss_cotangent in the backward passes. It also adds corresponding unit tests for both Linen and NNX implementations to verify this behavior. The review feedback suggests replacing the in-place multiplication operator (*=) with explicit assignment to adhere to JAX's immutable array paradigm and ensure consistency.
|
Could you update the pyink in tests/unit/attention_test.py? see https://github.com/AI-Hypercomputer/maxtext/actions/runs/32472985540/job/96840648989?pr=4959 |
done |
Codecov Report✅ All modified and coverable lines are covered by tests. 📢 Thoughts on this report? Let us know! |
Description
Fix the Linen and NNX vocabulary-tiling custom VJPs so the incoming loss
cotangent scales the hidden-state gradient as well as the output-head gradient.
Each tiled backward pass computes per-tile VJPs with a scalar seed of
1.0andthen applies the incoming
loss_cotangentto the accumulated output-headgradient. Previously, the same chain-rule factor was omitted from the emitted
hidden-state gradient. Directly differentiating the tiled loss supplies an
implicit cotangent of one, so the existing parity tests could not observe the
error. When a caller normalizes or weights the loss, output-head gradients were
scaled correctly while gradients flowing through the transformer body were
not.
Apply the missing factor in both implementations. Add focused Linen and NNX
regressions that differentiate each tiled loss beneath outer scales of
1.0and
2.0, then require the hidden-state gradient to double. These mathematicalchecks now run in the CPU suite; they do not require accelerator hardware.
Tests
uv run --no-sync pytest -q tests/unit/tiling_test.py— 2 passed, 18 skippeduv run --no-sync pre-commit run pyink --files src/maxtext/utils/vocabulary_tiling.py tests/unit/tiling_test.py— passeduv run --no-sync pre-commit run codespell --files src/maxtext/utils/vocabulary_tiling.py tests/unit/tiling_test.py— passeduv pip check --python .venv/bin/python— passedcotangent multiplications are removed and pass when restored.
Checklist
gemini-reviewlabel.