emerging_optimizers.weight_update_hooks#

class emerging_optimizers.weight_update_hooks.WeightUpdateHook(*args, **kwargs)[source]#

Static structural contract for behavior around the final weight update.

post_weight_update_inplace(p, pre_update_state)[source]#

Called after the optimizer’s final update and optimizer-specific post-update hook.

Parameters:
  • p (Tensor)

  • pre_update_state (HookStateT)

Return type:

None

pre_weight_update_inplace(p, update)[source]#

Called immediately before p.add_(update, alpha=-lr) and returns pre-update state.

Parameters:
Return type:

HookStateT

class emerging_optimizers.weight_update_hooks.NoOpWeightUpdateHook[source]#

Default hook that leaves the optimizer update unchanged.

class emerging_optimizers.weight_update_hooks.HyperballHook(radius, eps=1e-15)[source]#

Normalize update and post-update weights to a fixed Frobenius norm.

This hook mirrors the hyperball-style behavior used by MuonHyperball: before the weight update, normalize the update to the target radius; after the weight update, project the parameter back to that radius.

Parameters:
class emerging_optimizers.weight_update_hooks.RadialBrakeHook(outward_scale=0.5, inward_scale=1.0, eps=1e-15)[source]#

Dampen radial norm changes after an optimizer update.

The optimizer first applies its usual update w = w_prev + dw. This hook then rescales w so that

\[\|w_{brake}\| = \|w_{prev}\| + s(\|w\| - \|w_{prev}\|)\]

where s is outward_scale when the update increases the norm, otherwise inward_scale.

Parameters:
  • outward_scale (float) – Fraction of an outward norm change to retain.

  • inward_scale (float) – Fraction of an inward norm change to retain.

  • eps (float) – Norm threshold below which values are treated as numerical zero.

class emerging_optimizers.weight_update_hooks.RelativeUpdateHook(eps=1e-15)[source]#

Scale the update norm to the current weight norm without projecting the weight.

Before the optimizer applies its learning rate, this hook performs

\[U \leftarrow \frac{\lVert W\rVert_F}{\lVert U\rVert_F} U.\]

Therefore the learning rate directly controls the relative update norm. Unlike HyperballHook, this hook does not project the weight after the update.

Parameters:

eps (float)