Skip to content

feat(regularize): GroupRegularizeCallback for sparse training before a structured prune - #99

Merged
nathanhubens merged 1 commit into
masterfrom
feat/group-regularize-callback
Sep 24, 2026
Merged

nathanhubens merged 1 commit into
masterfrom
feat/group-regularize-callback

Conversation

@nathanhubens

Copy link
Copy Markdown
Collaborator

Adds GroupRegularizeCallback, the training-loop companion of Pruner.regularize (#96). It implements the sparse-training protocol of Network Slimming / DepGraph: regularize during a fit, prune once after it, then fine-tune.

pruner = Pruner(learn.model, 0.3, 'local', large_final, reg=1e-4, example_inputs=xb)
learn.fit(10, cbs=GroupRegularizeCallback(pruner, schedule=lin))   # group penalty on the gradients
pruner.prune_model()                                                 # once, after the fit
learn.fit(2, reset_opt=True)                                         # the prune replaced Parameters: rebuild the optimizer

Mechanism

  • Order MixedPrecision.order - 2:
    • after GradientAccumulation, which cancels the micro-batches, so the penalty is applied once per real optimizer step;
    • before MixedPrecision unscales the gradients, so the penalty is multiplied by the loss scale of a real GradScaler (1 otherwise).
  • Optional Schedule ramps the penalty. The callback keeps its own copy, so the module-level schedule objects other callbacks rely on are never mutated.
  • Refused, each with a one-line error:
    • a Pruner built with reg == 0;
    • a Pruner built on another model than learn.model;
    • NonNativeMixedPrecision;
    • a PruneCallback in the same fit (mid-fit pruning would invalidate the penalized groups).
  • PruneCallback now refuses reg: before, it was forwarded to the Pruner without ever being applied.
  • Lives in regularize/, next to RegularizeCallback. The RegularizeCallback BatchNorm caveat and the Pruner "Group sparse learning" section both point to it.

Evidence

  • nbs/tests/test_group_regularize_callback.ipynb:
    • hand-computed gradients through a real fit step;
    • a loss-scale stub on CPU (the gradients are scaled, then unscaled);
    • GradientAccumulation applies the penalty once per step;
    • schedule ramps, with the shared schedule left untouched;
    • every refusal;
    • fit, then prune_model(), then fit(reset_opt=True): the optimizer holds exactly the live parameters, while without reset_opt it holds stale ones;
    • the order pinned between GradientAccumulation and MixedPrecision;
    • a #| slow ResNet-18 run ending in learn.save.
  • Three mutations were run: wrong order, loss scale not read, schedule not copied. Each one fails its test.
  • Full suite nbdev-test and the slow tests pass in a clean clone with CI's nbdev 3.3.21. Clean and export leave the tree unchanged. No circular import between prune and regularize.
  • No measured claim: whether sparse training improves accuracy after pruning is not evaluated here.

…ai fit

before_step calls pruner.regularize(), ordered between GradientAccumulation
and MixedPrecision, so the penalty is added once per optimizer step and
multiplied by the loss scale of an active torch.amp.GradScaler. An optional
Schedule ramps it and is copied, leaving the shared module-level schedules
untouched. Pruning stays a separate call after the fit, followed by
learn.fit(n, reset_opt=True). A Pruner with reg=0, a Pruner built on another
model, NonNativeMixedPrecision and a PruneCallback in the same fit are
refused; PruneCallback refuses reg and points to the new callback.
@nathanhubens
nathanhubens merged commit 6ba9ea7 into master Sep 24, 2026
2 checks passed
@nathanhubens
nathanhubens deleted the feat/group-regularize-callback branch September 24, 2026 10:58
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant