emerging_optimizers.shampoo#
Shampoo#
- class emerging_optimizers.shampoo.shampoo.ShampooBase(
- params,
- lr,
- momentum=0.9,
- shampoo_beta=0.95,
- eps=1e-08,
- weight_decay=0.01,
- *,
- p_root_inv=4,
- scale_mode='spectral',
Canonical Shampoo step loop, shared by the Shampoo-family optimizers.
step()is the whole algorithm: update the preconditioner from the gradient, run an inner scalar optimizer in the parameter basis, precondition its update on both sides, and scale it withget_shampoo_scale_factor(). Subclasses customize the two pieces that vary between Shampoo variants and leave the loop alone:PreconditionerClsβ how the Kronecker factors and their inverse roots are maintained._scalar_update()β which scalar optimizer produces the update being preconditioned.
- Parameters:
params (Iterable[Tensor] | Iterable[dict[str, Any]] | Iterable[tuple[str, Tensor]]) β Iterable of 2D CUDA parameters to optimize or dicts defining parameter groups.
lr (float) β Learning rate.
momentum (float) β Momentum EMA coefficient.
shampoo_beta (float) β Kronecker factor EMA coefficient.
eps (float) β Numerical epsilon
weight_decay (float) β Decoupled weight decay coefficient.
p_root_inv (float) β Inverse root order applied to each Kronecker factor.
scale_mode (Literal['shape_scaling', 'spectral', 'unit_rms_norm'] | None) β The type of scale factor to use for the update, see
get_shampoo_scale_factor().Nonedisables it.
- Variables:
PreconditionerCls (ClassVar[type[emerging_optimizers.shampoo.precond_base.ShampooPreconditionerProtocol]]) β Preconditioner used for every parameter, and the source of the state layout allocated by
_init_group(). Subclasses set it to change how the factors are maintained; itskl_correctedattribute selects the update scale, seeget_shampoo_scale_factor().
- class emerging_optimizers.shampoo.shampoo.Shampoo(
- params,
- lr,
- momentum=0.9,
- shampoo_beta=0.95,
- eps=1e-08,
- weight_decay=0.01,
- *,
- p_root_inv=4,
- scale_mode='spectral',
Shampoo with EMA momentum as the inner scalar optimizer.
- Parameters:
- PreconditionerCls#
alias of
ShampooPreconditioner
- class emerging_optimizers.shampoo.shampoo.KlShampoo(
- params,
- lr,
- momentum=0.9,
- shampoo_beta=0.95,
- eps=1e-08,
- weight_decay=0.01,
- *,
- p_root_inv=2,
- scale_mode='spectral',
Shampoo with KL-corrected Kronecker factors and EMA momentum.
- Parameters:
- PreconditionerCls#
alias of
KlShampooPreconditioner
- class emerging_optimizers.shampoo.shampoo.ShampooPreconditioner(state, p_root_inv, eps)[source]#
Per-parameter Shampoo preconditioner holding the Kronecker factors of one 2D parameter.
- Parameters:
- static init_state(shape, device)[source]#
Creates the Kronecker factors for a parameter shape.
- Parameters:
- Returns:
The state entries owned by this preconditioner, keyed as
rebind_state()expects them.- Raises:
TypeError β If
shapeis not 2D.- Return type:
- init_step(grad, shampoo_beta)[source]#
Performs the first stepβs factor update, before any history exists.
- rebind_state(state)[source]#
Binds the current preconditioner tensors back into the optimizer state dict.
- class emerging_optimizers.shampoo.shampoo.KlShampooPreconditioner(state, p_root_inv, eps)[source]#
Shampoo preconditioner with the KL-corrected Kronecker factor update.
- static init_state(shape, device)[source]#
Creates the Kronecker factors for a parameter shape.
- Parameters:
- Returns:
The state entries owned by this preconditioner, keyed as
rebind_state()expects them.- Raises:
TypeError β If
shapeis not 2D.- Return type:
- init_step(grad, shampoo_beta)[source]#
Performs the first stepβs factor update, before any history exists.
- emerging_optimizers.shampoo.shampoo.get_shampoo_scale_factor(
- L,
- R,
- p_root_inv,
- mode='spectral',
- *,
- kl_corrected=False,
- eps,
Get the scale for the Shampoo update.
- Parameters:
L (Tensor) β Left Kronecker factor,
size_out x size_out.R (Tensor) β Right Kronecker factor,
size_in x size_in.p_root_inv (float) β Inverse root order applied to each Kronecker factor.
mode (Literal['shape_scaling', 'spectral', 'unit_rms_norm']) β The mode to use for the scale.
kl_corrected (bool) β Whether the Kronecker factors are KL-corrected.
eps (float) β Floor on the factor traces; use the preconditionerβs
eps.
- Returns:
a float when the trace exponent vanishes, otherwise a 0-d tensor.
- Return type:
The scale factor for the update
SOAP v3#
- class emerging_optimizers.shampoo.soap_v3.SoapBase(
- params,
- lr,
- betas=(0.9, 0.95),
- shampoo_beta=0.95,
- eps=1e-08,
- weight_decay=0.01,
- *,
- stream_list=None,
Canonical SOAP step loop, shared by the SOAP-family optimizers.
step()is the whole algorithm: update the preconditioner from the gradient, project the gradient into the eigenbasis, run an inner scalar optimizer there, and project the update back out. Subclasses customize the two pieces that actually vary between SOAP variants and leave the loop alone:PreconditionerClsβ how the covariance factors and eigenbases are maintained._scalar_update()β which scalar optimizer runs inside the eigenbasis.
betaslives here because every inner update in the family takes a pair of EMA coefficients. Constants specific to one inner update (MAdamβsscale_log2, for instance) belong on the subclass as class attributes, so that variants do not have to redefine__init__at all.- Parameters:
params (Iterable[Tensor] | Iterable[dict[str, Any]] | Iterable[tuple[str, Tensor]]) β Iterable of parameters to optimize or dicts defining parameter groups
lr (float) β The learning rate to use
betas (tuple[float, float]) β Inner scalar optimizerβs betas parameters (b1, b2). Per parameter group.
shampoo_beta (float) β Beta for the kronecker factor matrices (L and R in paper) moving average
eps (float) β Epsilon for the Kronecker factor update, passed to the preconditioner. Whether the inner update also uses it is up to the subclass.
weight_decay (float) β Weight decay coefficient
max_update_rms β Clip the update RMS to this value (0 means no clipping).
stream_list (list[Stream] | None) β Optional list of CUDA streams. When provided, each parameter in the inner loop uses a stream from this list in round-robin fashion.
- Variables:
PreconditionerCls (ClassVar[type[emerging_optimizers.shampoo.precond_base.SoapPreconditionerProtocol]]) β Preconditioner used for every parameter. Subclasses set it to change how the covariance factors and eigenbases are maintained; it must satisfy
SoapPreconditionerProtocol. It is also what_init_group()allocates state from, so a subclass that swaps it gets that preconditionerβs state layout.
- class emerging_optimizers.shampoo.soap_v3.KlSoapV3(
- params,
- lr,
- betas=(0.9, 0.95),
- shampoo_beta=0.95,
- eps=1e-08,
- weight_decay=0.01,
- *,
- stream_list=None,
Implements a variant of KLSOAP algorithm.
- Parameters:
- PreconditionerCls#
alias of
KlSoapPreconditioner
- class emerging_optimizers.shampoo.soap_v3.ReklsV3(
- params,
- lr,
- betas=(0.9, 0.95),
- shampoo_beta=0.95,
- eps=1e-08,
- weight_decay=0.01,
- *,
- stream_list=None,
Realtime Eigen KL-Shampoo
- Parameters:
- PreconditionerCls#
alias of
ReklsPreconditioner
- class emerging_optimizers.shampoo.soap_v3.KlMSoap(
- params,
- lr,
- betas=(0.9, 0.95),
- shampoo_beta=0.95,
- eps=1e-08,
- weight_decay=0.01,
- *,
- stream_list=None,
SOAP with the KL-Shampoo kronecker factor update and MAdam as the inner scalar optimizer.
- Parameters:
- PreconditionerCls#
alias of
KlSoapPreconditioner
- class emerging_optimizers.shampoo.soap_v3.KlSoapPreconditioner(state, eps)[source]#
Per-parameter SOAP preconditioner holding the Kronecker factors, eigenbases, and eigenvalues.
- Parameters:
- static init_state(shape, device)[source]#
Creates the Kronecker factors, eigenbases, eigenvalues, and moments for a parameter shape.
- Parameters:
- Returns:
The state entries owned by this preconditioner, keyed as
rebind_state()expects them.- Raises:
TypeError β If
shapeis not 2D.- Return type:
- init_step(grad, shampoo_beta)[source]#
Seeds the kronecker factors and eigenbases from the first gradient with eigh
It calls KL correction in the init step to match legacy Soap behavior.
- rebind_state(state)[source]#
Binds the current preconditioner tensors back into the optimizer state dict.
emerging_optimizers.shampoo.precond_base#
- class emerging_optimizers.shampoo.precond_base.ShampooPreconditionerProtocol(state, /, *args, **kwargs)[source]#
Shampoo preconditioner
emerging_optimizers.shampoo.matrix_root_inverse_utils#
- emerging_optimizers.shampoo.matrix_root_inverse_utils.mat_4th_root_inv_via_scaled_cans(x, eps=1e-12)[source]#
Compute inverse fourth roots with a scaled coupled CANS-style Newton-Schulz schedule.
Runs the coupled iteration
Z <- q(M) Z,M <- q(M)^4 Mwith per-step degree-one polynomialsq(m) = beta + alpha * m, soZconverges to the inverse fourth root of the normalized input. The ten-step schedule was derived by a greedy per-step interval-ratio optimization for normalized eigenvalues in[1e-7, 1]. Likemat_root_inv_via_scaled_cans(), it normalizes with the matrix infinity norm rather than the exact spectral norm. The infinity norm is an inexpensive upper bound, but can conservatively scale matrices whose rows contain substantial cancellation and consequently slow convergence for their smallest eigenvalues.The tabulated coefficients fold a 1% spectral safety margin into the polynomial. Starting from the designed pairs
(beta, alpha), steps zero through eight keepbetaand usealpha / 1.01. The final pair additionally absorbs the geometric recentering of the residual interval and the marginβs share of the output normalization and is computed as(beta * rho**-0.125 / 1.01**0.25, alpha * rho**-0.125 / 1.01**1.25)withrhothe final interval ratio. Only the data-dependentinf_norm**-0.25is applied explicitly.- Parameters:
- Returns:
The approximate inverse fourth root as an FP32 tensor with the same shape as
x.- Raises:
TypeError β If
xis not a square FP32 matrix or batch of square FP32 matrices.RuntimeError β If the active FP32 matmul precision is
"medium".
- Return type:
- emerging_optimizers.shampoo.matrix_root_inverse_utils.mat_root_inv_via_scaled_cans(x, eps=1e-12)[source]#
Compute inverse square roots with scaled coupled CANS Newton-Schulz.
CANS polynomial-based inverse-root computation from https://arxiv.org/abs/2506.10935.
This implementation applies the CANS orthogonalization polynomials to the coupled Newton-Schulz iteration for a symmetric positive-definite matrix. It uses a fixed ten-step schedule and normalizes with the matrix infinity norm rather than the exact spectral norm. The infinity norm is an inexpensive upper bound, but can conservatively scale matrices whose rows contain substantial cancellation and consequently slow convergence for their smallest eigenvalues.
The tabulated coefficients fold a 1% spectral safety margin into the polynomial. Starting from the unscaled CANS pairs
(beta, alpha), steps zero through eight keepbetaand usealpha / 1.01. The final pair additionally absorbs the output normalization and is computed as(beta / sqrt(1.01), alpha / 1.01**1.5). This is algebraically equivalent in exact arithmetic to normalizing by1.01 * inf_normand applying the original coefficients.- Parameters:
- Returns:
The approximate inverse square root as an FP32 tensor with the same shape as
x.- Raises:
TypeError β If
xis not a square FP32 matrix or batch of square FP32 matrices.RuntimeError β If the active FP32 matmul precision is
"medium".
- Return type: