emerging_optimizers.shampoo#

Shampoo#

class emerging_optimizers.shampoo.shampoo.ShampooBase(
params,
lr,
momentum=0.9,
shampoo_beta=0.95,
eps=1e-08,
weight_decay=0.01,
*,
p_root_inv=4,
scale_mode='spectral',
)[source]#

Canonical Shampoo step loop, shared by the Shampoo-family optimizers.

step() is the whole algorithm: update the preconditioner from the gradient, run an inner scalar optimizer in the parameter basis, precondition its update on both sides, and scale it with get_shampoo_scale_factor(). Subclasses customize the two pieces that vary between Shampoo variants and leave the loop alone:

  • PreconditionerCls – how the Kronecker factors and their inverse roots are maintained.

  • _scalar_update() – which scalar optimizer produces the update being preconditioned.

Parameters:
  • params (Iterable[Tensor] | Iterable[dict[str, Any]] | Iterable[tuple[str, Tensor]]) – Iterable of 2D CUDA parameters to optimize or dicts defining parameter groups.

  • lr (float) – Learning rate.

  • momentum (float) – Momentum EMA coefficient.

  • shampoo_beta (float) – Kronecker factor EMA coefficient.

  • eps (float) – Numerical epsilon

  • weight_decay (float) – Decoupled weight decay coefficient.

  • p_root_inv (float) – Inverse root order applied to each Kronecker factor.

  • scale_mode (Literal['shape_scaling', 'spectral', 'unit_rms_norm'] | None) – The type of scale factor to use for the update, see get_shampoo_scale_factor(). None disables it.

Variables:

PreconditionerCls (ClassVar[type[emerging_optimizers.shampoo.precond_base.ShampooPreconditionerProtocol]]) – Preconditioner used for every parameter, and the source of the state layout allocated by _init_group(). Subclasses set it to change how the factors are maintained; its kl_corrected attribute selects the update scale, see get_shampoo_scale_factor().

step(closure: None = None) → None[source]#
step(
closure: Callable[[], float],
) → float

Performs a single optimization step.

Parameters:

closure – Unsupported; must be None.

Raises:

ValueError – If closure is not None.

class emerging_optimizers.shampoo.shampoo.Shampoo(
params,
lr,
momentum=0.9,
shampoo_beta=0.95,
eps=1e-08,
weight_decay=0.01,
*,
p_root_inv=4,
scale_mode='spectral',
)[source]#

Shampoo with EMA momentum as the inner scalar optimizer.

Parameters:
PreconditionerCls#

alias of ShampooPreconditioner

class emerging_optimizers.shampoo.shampoo.KlShampoo(
params,
lr,
momentum=0.9,
shampoo_beta=0.95,
eps=1e-08,
weight_decay=0.01,
*,
p_root_inv=2,
scale_mode='spectral',
)[source]#

Shampoo with KL-corrected Kronecker factors and EMA momentum.

Parameters:
PreconditionerCls#

alias of KlShampooPreconditioner

class emerging_optimizers.shampoo.shampoo.ShampooPreconditioner(state, p_root_inv, eps)[source]#

Per-parameter Shampoo preconditioner holding the Kronecker factors of one 2D parameter.

Parameters:
  • state (dict) – Per-parameter optimizer state holding L and R.

  • p_root_inv (float) – Inverse root order; each factor is applied as A^(-1/p_root_inv).

  • eps (float) – Floor on the eigenvalue magnitudes before inversion.

static init_state(shape, device)[source]#

Creates the Kronecker factors for a parameter shape.

Parameters:
  • shape (tuple[int, ...]) – Shape of the 2D parameter the preconditioner will be attached to.

  • device (device) – Device to allocate the state tensors on.

Returns:

The state entries owned by this preconditioner, keyed as rebind_state() expects them.

Raises:

TypeError – If shape is not 2D.

Return type:

dict[str, Tensor]

init_step(grad, shampoo_beta)[source]#

Performs the first step’s factor update, before any history exists.

Parameters:
  • grad (Tensor) – Gradient of the parameter on the first step.

  • shampoo_beta (float) – EMA coefficient for the Kronecker factor update.

Return type:

None

precondition(x)[source]#

Applies both root inverse to a matrix in the parameter basis.

Parameters:

x (Tensor) – Matrix in the parameter basis.

Returns:

The preconditioned matrix, in the parameter basis.

Return type:

Tensor

rebind_state(state)[source]#

Binds the current preconditioner tensors back into the optimizer state dict.

Parameters:

state (dict) – Per-parameter optimizer state, updated in place.

Raises:

KeyError – If state is missing any of the preconditioner keys.

Return type:

None

step(grad, shampoo_beta)[source]#

Updates the Kronecker factors with the latest gradient.

Parameters:
  • grad (Tensor) – Gradient of the parameter.

  • shampoo_beta (float) – EMA coefficient for the Kronecker factor update.

Return type:

None

update_kronecker_factors(grad, shampoo_beta)[source]#

Accumulates the gradient outer products into the Kronecker factors.

Parameters:
  • grad (Tensor) – Gradient of the parameter.

  • shampoo_beta (float) – EMA coefficient for the Kronecker factor update.

Return type:

None

class emerging_optimizers.shampoo.shampoo.KlShampooPreconditioner(state, p_root_inv, eps)[source]#

Shampoo preconditioner with the KL-corrected Kronecker factor update.

Parameters:
static init_state(shape, device)[source]#

Creates the Kronecker factors for a parameter shape.

Parameters:
  • shape (tuple[int, ...]) – Shape of the 2D parameter the preconditioner will be attached to.

  • device (device) – Device to allocate the state tensors on.

Returns:

The state entries owned by this preconditioner, keyed as rebind_state() expects them.

Raises:

TypeError – If shape is not 2D.

Return type:

dict[str, Tensor]

init_step(grad, shampoo_beta)[source]#

Performs the first step’s factor update, before any history exists.

Parameters:
  • grad (Tensor) – Gradient of the parameter on the first step.

  • shampoo_beta (float) – EMA coefficient for the Kronecker factor update.

Return type:

None

precondition(x)[source]#

Applies both root inverse to a matrix in the parameter basis.

Parameters:

x (Tensor) – Matrix in the parameter basis.

Returns:

The preconditioned matrix, in the parameter basis.

Return type:

Tensor

rebind_state(state)[source]#

Binds the current preconditioner tensors back into the optimizer state dict.

Parameters:

state (dict) – Per-parameter optimizer state, updated in place.

Raises:

KeyError – If state is missing any of the preconditioner keys.

Return type:

None

update_kronecker_factors(grad, shampoo_beta)[source]#

Accumulates the gradient outer products into the Kronecker factors.

Parameters:
  • grad (Tensor) – Gradient of the parameter.

  • shampoo_beta (float) – EMA coefficient for the Kronecker factor update.

Return type:

None

emerging_optimizers.shampoo.shampoo.get_shampoo_scale_factor(
L,
R,
p_root_inv,
mode='spectral',
*,
kl_corrected=False,
eps,
)[source]#

Get the scale for the Shampoo update.

Parameters:
  • L (Tensor) – Left Kronecker factor, size_out x size_out.

  • R (Tensor) – Right Kronecker factor, size_in x size_in.

  • p_root_inv (float) – Inverse root order applied to each Kronecker factor.

  • mode (Literal['shape_scaling', 'spectral', 'unit_rms_norm']) – The mode to use for the scale.

  • kl_corrected (bool) – Whether the Kronecker factors are KL-corrected.

  • eps (float) – Floor on the factor traces; use the preconditioner’s eps.

Returns:

a float when the trace exponent vanishes, otherwise a 0-d tensor.

Return type:

The scale factor for the update

SOAP v3#

class emerging_optimizers.shampoo.soap_v3.SoapBase(
params,
lr,
betas=(0.9, 0.95),
shampoo_beta=0.95,
eps=1e-08,
weight_decay=0.01,
*,
stream_list=None,
)[source]#

Canonical SOAP step loop, shared by the SOAP-family optimizers.

step() is the whole algorithm: update the preconditioner from the gradient, project the gradient into the eigenbasis, run an inner scalar optimizer there, and project the update back out. Subclasses customize the two pieces that actually vary between SOAP variants and leave the loop alone:

  • PreconditionerCls – how the covariance factors and eigenbases are maintained.

  • _scalar_update() – which scalar optimizer runs inside the eigenbasis.

betas lives here because every inner update in the family takes a pair of EMA coefficients. Constants specific to one inner update (MAdam’s scale_log2, for instance) belong on the subclass as class attributes, so that variants do not have to redefine __init__ at all.

Parameters:
  • params (Iterable[Tensor] | Iterable[dict[str, Any]] | Iterable[tuple[str, Tensor]]) – Iterable of parameters to optimize or dicts defining parameter groups

  • lr (float) – The learning rate to use

  • betas (tuple[float, float]) – Inner scalar optimizer’s betas parameters (b1, b2). Per parameter group.

  • shampoo_beta (float) – Beta for the kronecker factor matrices (L and R in paper) moving average

  • eps (float) – Epsilon for the Kronecker factor update, passed to the preconditioner. Whether the inner update also uses it is up to the subclass.

  • weight_decay (float) – Weight decay coefficient

  • max_update_rms – Clip the update RMS to this value (0 means no clipping).

  • stream_list (list[Stream] | None) – Optional list of CUDA streams. When provided, each parameter in the inner loop uses a stream from this list in round-robin fashion.

Variables:

PreconditionerCls (ClassVar[type[emerging_optimizers.shampoo.precond_base.SoapPreconditionerProtocol]]) – Preconditioner used for every parameter. Subclasses set it to change how the covariance factors and eigenbases are maintained; it must satisfy SoapPreconditionerProtocol. It is also what _init_group() allocates state from, so a subclass that swaps it gets that preconditioner’s state layout.

step(closure: None = None) → None[source]#
step(closure: Callable[[], float]) → float

Performs a single optimization step.

Parameters:

closure – Unsupported; must be None.

class emerging_optimizers.shampoo.soap_v3.KlSoapV3(
params,
lr,
betas=(0.9, 0.95),
shampoo_beta=0.95,
eps=1e-08,
weight_decay=0.01,
*,
stream_list=None,
)[source]#

Implements a variant of KLSOAP algorithm.

Parameters:
PreconditionerCls#

alias of KlSoapPreconditioner

class emerging_optimizers.shampoo.soap_v3.ReklsV3(
params,
lr,
betas=(0.9, 0.95),
shampoo_beta=0.95,
eps=1e-08,
weight_decay=0.01,
*,
stream_list=None,
)[source]#

Realtime Eigen KL-Shampoo

Parameters:
PreconditionerCls#

alias of ReklsPreconditioner

class emerging_optimizers.shampoo.soap_v3.KlMSoap(
params,
lr,
betas=(0.9, 0.95),
shampoo_beta=0.95,
eps=1e-08,
weight_decay=0.01,
*,
stream_list=None,
)[source]#

SOAP with the KL-Shampoo kronecker factor update and MAdam as the inner scalar optimizer.

Parameters:
PreconditionerCls#

alias of KlSoapPreconditioner

class emerging_optimizers.shampoo.soap_v3.KlSoapPreconditioner(state, eps)[source]#

Per-parameter SOAP preconditioner holding the Kronecker factors, eigenbases, and eigenvalues.

Parameters:
  • state (dict) – Per-parameter optimizer state holding L/R, Q_L/R, eigvals_L/R, etc.

  • eps (float) – Epsilon for the KL-Shampoo Kronecker factor update.

static init_state(shape, device)[source]#

Creates the Kronecker factors, eigenbases, eigenvalues, and moments for a parameter shape.

Parameters:
  • shape (tuple[int, ...]) – Shape of the 2D parameter the preconditioner will be attached to.

  • device (device) – Device to allocate the state tensors on.

Returns:

The state entries owned by this preconditioner, keyed as rebind_state() expects them.

Raises:

TypeError – If shape is not 2D.

Return type:

dict[str, Tensor]

init_step(grad, shampoo_beta)[source]#

Seeds the kronecker factors and eigenbases from the first gradient with eigh

It calls KL correction in the init step to match legacy Soap behavior.

Parameters:
Return type:

None

project_in(x)[source]#

Projects a tensor into the eigenbasis.

Parameters:

x (Tensor) – Tensor to project.

Returns:

The tensor projected into the eigenbasis.

Return type:

Tensor

project_out(x)[source]#

Projects a tensor out of the eigenbasis, back to the original basis.

Parameters:

x (Tensor) – Tensor to project back.

Returns:

The tensor in the original basis.

Return type:

Tensor

rebind_state(state)[source]#

Binds the current preconditioner tensors back into the optimizer state dict.

Parameters:

state (dict) – Per-parameter optimizer state, updated in place.

Raises:

KeyError – If state is missing any of the preconditioner keys.

Return type:

None

step(grad, shampoo_beta)[source]#

Updates the kronecker factors and eigenbases, re-projecting exp_avg into the new eigenbasis.

Parameters:
  • grad (Tensor) – Gradient of the parameter.

  • shampoo_beta (float) – EMA coefficient for the kronecker factor update.

Return type:

None

update_kronecker_factors(grad, shampoo_beta)[source]#

Accumulates the gradient into the kronecker factors with the KL-Shampoo correction.

Parameters:
  • grad (Tensor) – Gradient of the parameter.

  • shampoo_beta (float) – EMA coefficient for the kronecker factor update.

Return type:

None

class emerging_optimizers.shampoo.soap_v3.ReklsPreconditioner(state, eps)[source]#

KL-Shampoo preconditioner that rebuilds the eigenbases with eigh on every step.

Parameters:
step(grad, shampoo_beta)[source]#

Updates the kronecker factors and eigenbases, re-projecting exp_avg into the new eigenbasis.

Parameters:
  • grad (Tensor) – Gradient of the parameter.

  • shampoo_beta (float) – EMA coefficient for the kronecker factor update.

Return type:

None

emerging_optimizers.shampoo.precond_base#

class emerging_optimizers.shampoo.precond_base.ShampooPreconditionerProtocol(state, /, *args, **kwargs)[source]#

Shampoo preconditioner

Parameters:
precondition(x, /)[source]#

Applies the two-sided preconditioner to a matrix in the parameter basis.

Parameters:

x (Tensor) – Matrix in the parameter basis.

Returns:

The preconditioned matrix, in the parameter basis.

Return type:

Tensor

class emerging_optimizers.shampoo.precond_base.SoapPreconditionerProtocol(state, /, *args, **kwargs)[source]#

Soap preconditioner which projects update from/to eigen bases

Parameters:
project_in(x, /)[source]#

Projects a tensor from the parameter basis into the eigenbasis.

Parameters:

x (Tensor) – Tensor in the parameter basis.

Returns:

The tensor expressed in the eigenbasis.

Return type:

Tensor

project_out(x, /)[source]#

Projects a tensor from the eigenbasis back to the parameter basis.

Parameters:

x (Tensor) – Tensor in the eigenbasis.

Returns:

The tensor expressed in the parameter basis.

Return type:

Tensor

class emerging_optimizers.shampoo.precond_base.TensorPair(L, R)[source]#

A pair of tensors

Parameters:

emerging_optimizers.shampoo.matrix_root_inverse_utils#

emerging_optimizers.shampoo.matrix_root_inverse_utils.mat_4th_root_inv_via_scaled_cans(x, eps=1e-12)[source]#

Compute inverse fourth roots with a scaled coupled CANS-style Newton-Schulz schedule.

Runs the coupled iteration Z <- q(M) Z, M <- q(M)^4 M with per-step degree-one polynomials q(m) = beta + alpha * m, so Z converges to the inverse fourth root of the normalized input. The ten-step schedule was derived by a greedy per-step interval-ratio optimization for normalized eigenvalues in [1e-7, 1]. Like mat_root_inv_via_scaled_cans(), it normalizes with the matrix infinity norm rather than the exact spectral norm. The infinity norm is an inexpensive upper bound, but can conservatively scale matrices whose rows contain substantial cancellation and consequently slow convergence for their smallest eigenvalues.

The tabulated coefficients fold a 1% spectral safety margin into the polynomial. Starting from the designed pairs (beta, alpha), steps zero through eight keep beta and use alpha / 1.01. The final pair additionally absorbs the geometric recentering of the residual interval and the margin’s share of the output normalization and is computed as (beta * rho**-0.125 / 1.01**0.25, alpha * rho**-0.125 / 1.01**1.25) with rho the final interval ratio. Only the data-dependent inf_norm**-0.25 is applied explicitly.

Parameters:
  • x (Tensor) – A 2D symmetric positive-definite FP32 matrix or 3D batch of matrices.

  • eps (float) – Lower bound used when normalizing the matrices.

Returns:

The approximate inverse fourth root as an FP32 tensor with the same shape as x.

Raises:
  • TypeError – If x is not a square FP32 matrix or batch of square FP32 matrices.

  • RuntimeError – If the active FP32 matmul precision is "medium".

Return type:

Tensor

emerging_optimizers.shampoo.matrix_root_inverse_utils.mat_root_inv_via_scaled_cans(x, eps=1e-12)[source]#

Compute inverse square roots with scaled coupled CANS Newton-Schulz.

CANS polynomial-based inverse-root computation from https://arxiv.org/abs/2506.10935.

This implementation applies the CANS orthogonalization polynomials to the coupled Newton-Schulz iteration for a symmetric positive-definite matrix. It uses a fixed ten-step schedule and normalizes with the matrix infinity norm rather than the exact spectral norm. The infinity norm is an inexpensive upper bound, but can conservatively scale matrices whose rows contain substantial cancellation and consequently slow convergence for their smallest eigenvalues.

The tabulated coefficients fold a 1% spectral safety margin into the polynomial. Starting from the unscaled CANS pairs (beta, alpha), steps zero through eight keep beta and use alpha / 1.01. The final pair additionally absorbs the output normalization and is computed as (beta / sqrt(1.01), alpha / 1.01**1.5). This is algebraically equivalent in exact arithmetic to normalizing by 1.01 * inf_norm and applying the original coefficients.

Parameters:
  • x (Tensor) – A 2D symmetric positive-definite FP32 matrix or 3D batch of matrices.

  • eps (float) – Lower bound used when normalizing the matrices.

Returns:

The approximate inverse square root as an FP32 tensor with the same shape as x.

Raises:
  • TypeError – If x is not a square FP32 matrix or batch of square FP32 matrices.

  • RuntimeError – If the active FP32 matmul precision is "medium".

Return type:

Tensor