emerging_optimizers.weight_update_hooks#
- class emerging_optimizers.weight_update_hooks.WeightUpdateHook(*args, **kwargs)[source]#
Static structural contract for behavior around the final weight update.
- class emerging_optimizers.weight_update_hooks.NoOpWeightUpdateHook[source]#
Default hook that leaves the optimizer update unchanged.
- class emerging_optimizers.weight_update_hooks.HyperballHook(radius, eps=1e-15)[source]#
Normalize update and post-update weights to a fixed Frobenius norm.
This hook mirrors the hyperball-style behavior used by MuonHyperball: before the weight update, normalize the update to the target radius; after the weight update, project the parameter back to that radius.
- class emerging_optimizers.weight_update_hooks.RadialBrakeHook(outward_scale=0.5, inward_scale=1.0, eps=1e-15)[source]#
Dampen radial norm changes after an optimizer update.
The optimizer first applies its usual update
w = w_prev + dw. This hook then rescaleswso that\[\|w_{brake}\| = \|w_{prev}\| + s(\|w\| - \|w_{prev}\|)\]where
sisoutward_scalewhen the update increases the norm, otherwiseinward_scale.
- class emerging_optimizers.weight_update_hooks.RelativeUpdateHook(eps=1e-15)[source]#
Scale the update norm to the current weight norm without projecting the weight.
Before the optimizer applies its learning rate, this hook performs
\[U \leftarrow \frac{\lVert W\rVert_F}{\lVert U\rVert_F} U.\]Therefore the learning rate directly controls the relative update norm. Unlike
HyperballHook, this hook does not project the weight after the update.- Parameters:
eps (float)