Skip to content

Add an Enzyme rule for cheb_conv (stacked on #708) - #713

Draft
Parvm1102 wants to merge 6 commits into
JuliaGraphs:masterfrom
Parvm1102:enzyme-cheb-conv-rule
Draft

Parvm1102 wants to merge 6 commits into
JuliaGraphs:masterfrom
Parvm1102:enzyme-cheb-conv-rule

Conversation

@Parvm1102

@Parvm1102 Parvm1102 commented Oct 13, 2026 •

Copy link
Copy Markdown
Contributor

Stacked on #708. This branch contains #708's three commits; the commits on top of it are the new part (diff against #708). It targets master because #708's branch lives in my fork, so GitHub can't base a PR on it here. It's a draft for now, mainly to see whether the GNN CI job completes with Enzyme enabled. #708's run in August was killed at minute 77 when the runner ran out of resources.

What changes

  1. An Enzyme reverse rule for cheb_conv (new GNNlibEnzymeCoreExt, GNNlib → 1.5.0-DEV). The array math of cheb_conv moves verbatim into an internal _cheb_core(W, b, X, L), and the extension gives it a hand-written pullback. The scaled Laplacian is treated as a constant, as it already is via GNNGraphsEnzymeCoreExt. Batched annotations fall back to Enzyme's own differentiation.
  2. Enzyme opted out on DCGRUCell and DCGRU. With Enzyme 0.13.216 both fail with an IllegalTypeAnalysisException inside the keyword GNNGraph constructor that d_conv uses for the transposed graph, the same cause that keeps DConv opted out. A keyword-free d_conv (the Fix Enzyme differentiation of GCNConv/SGConv/TAGConv on adjacency-matrix graphs via a keyword-free COO conversion helper #703 recipe) fixes all three; I've tested that locally and left it for a follow-up.

Why

On the current CI stack (Julia 1.13.1, Enzyme 0.13.216), compiling Enzyme gradients is no longer expensive for the message-passing layers (1–13 s per compile). The recurrent layers built on cheb_conv are now the bulk of the cost. GConvLSTMCell calls cheb_conv eight times, and Enzyme differentiated each copy inline. With the rule:

Enzyme compile, one test site before after
GConvLSTM (input / model) 90 s / 203 s 18 s / 10 s
GConvLSTMCell 56 s / 102 s 31 s / 8 s
GConvGRUCell ~31 s / ~71 s 4 s / 4 s

The whole GConvLSTM test item went from 21 min (2.7 GB) to 4 min (1.8 GB). It also makes Enzyme much faster to compile for users of ChebConv, GConvGRU and GConvLSTM.

Testing

Locally on Julia 1.13.1 with Enzyme 0.13.216, one test item per process:

  • ChebConv 37/37, GConvLSTMCell 5/5, GConvLSTM 6/6, GConvGRUCell 4/4 and GConvGRU 6/6 pass. Their test_gradients sites compare Enzyme against finite differences.
  • A new ChebConv case with k = 3, bias = false runs the recursion and the constant-bias branch of the rule.
  • Separately, Enzyme with the rule matches Zygote to ≤ 3.2e-7 on input, model and state gradients (k = 2 and 3).
  • DCGRUCell 4/4 and DCGRU 6/6 pass with Enzyme skipped.

Not yet run: the rest of the suite, and Enzyme on GPU arrays (the GPU tests don't use Enzyme).

🤖 Generated with Claude Code

Parvm1102 and others added 6 commits August 28, 2026 13:59
Signed-off-by: Parvm1102 <parvmittal31757@gmail.com>
Signed-off-by: Parvm1102 <parvmittal31757@gmail.com>
`Flux.destructure`'s `re` writes the gradient into the trainable leaves but
keeps every other array (e.g. BatchNorm's running stats) at its model value.
Zygote returns `nothing` for those fields so `check_equal_leaves` skipped them,
but Enzyme returns a full shadow model with the correct zero gradient there and
the comparison failed on state that is not a gradient at all.

Rebuild the reference from a zeroed model so those leaves compare as zeros.
This enables Enzyme on the GNNChain Parallel/BatchNorm test site for non-dense
graphs; verified neutral for EdgeConv and GNNlib's propagate.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Parvm1102 <parvmittal31757@gmail.com>
Enzyme differentiated the Chebyshev recursion of `cheb_conv` inline at every
call site. `GConvLSTMCell` calls `cheb_conv` eight times and `GConvGRUCell` six,
so compiling their gradients was dominated by these inlined copies (a GConvLSTM
test site took ~5 min to compile on Julia 1.13.1 with Enzyme 0.13.216).

Move the array math of `cheb_conv` into an internal `_cheb_core(W, b, X, L)` and
give it an Enzyme reverse rule in a new `GNNlibEnzymeCoreExt`. The scaled
Laplacian is a constant (it is already marked inactive for Enzyme), and batched
or other annotations fall back to Enzyme's own differentiation.

Add a k = 3, bias = false ChebConv test case, which runs the recursion and the
constant-bias path of the rule.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
With Enzyme 0.13.216 both fail with an IllegalTypeAnalysisException inside the
keyword `GNNGraph` constructor that `d_conv` uses to build the transposed graph,
the same cause that keeps DConv opted out.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@Parvm1102

Parvm1102 commented Oct 13, 2026 •

Copy link
Copy Markdown
Contributor Author

The CI actually passed here. There were lots on changes in Enzyme since #708 and it takes very less time to run now.
There are still some failing layers with Enzyme (which is fixable) and the time in CI is solved for now. How should we proceed from here?

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant