emerging_optimizers.legacy_soap#
SOAP#
- class emerging_optimizers.legacy_soap.soap.SOAP(
- params,
- lr,
- betas=(0.9, 0.95),
- shampoo_beta=0.95,
- eps=1e-08,
- weight_decay=0.1,
- *,
- weight_decay_method='decoupled',
- nesterov=False,
- correct_bias=True,
- fp32_matmul_prec='highest',
- use_eigh=False,
- qr_fp32_matmul_prec='high',
- power_iter_steps=1,
- max_update_rms=0.0,
- use_kl_shampoo=True,
- correct_shampoo_beta_bias=None,
- stream_list=None,
Implements a variant of SOAP (ShampoO with Adam in the Preconditioner eigenbasis) algorithm.
SOAP (https://arxiv.org/abs/2409.11321) is a preconditioned optimizer that combines the benefits of Shampoo’s non-diagonal preconditioning with Adam’s adaptive learning rates. It uses gradient correlation matrix eigenbasis-based preconditioning to adapt to the local geometry of the optimization landscape. Default hyperparameters follow the recipe in SOAP, Muon, and Beyond (https://arxiv.org/abs/2607.20548).
- Parameters:
params (Iterable[Tensor] | Iterable[dict[str, Any]] | Iterable[tuple[str, Tensor]]) – Iterable of parameters to optimize or dicts defining parameter groups
lr (float) – The learning rate to use
betas (tuple[float, float]) – Inner Adam’s betas parameters (b1, b2)
shampoo_beta (float) – Beta for the kronecker factor matrices (L and R in paper) moving average instead of betas[1] if >= 0
eps (float) – Inner Adam’s epsilon for numerical stability
weight_decay (float) – Weight decay coefficient
weight_decay_method (Literal['decoupled', 'independent', 'l2', 'palm']) – Method to apply weight decay, see
WeightDecayMixinfor more details.nesterov (bool) – uses Nesterov momentum in Adam (https://cs229.stanford.edu/proj2015/054_report.pdf)
correct_bias (bool) – Whether to use bias correction in Inner Adam and Kronecker factor matrices EMA
fp32_matmul_prec (Literal['highest', 'high', 'medium']) – Precision of the matmul operations in optimizer states GEMM operations
use_eigh (bool) – Whether to use full symmetric eigendecomposition (eigh) to compute the eigenbasis. If False, use orthogonal iteration to compute the eigenbasis.
qr_fp32_matmul_prec (Literal['highest', 'high', 'medium']) – Precision of the matmul operations in QR decomposition.
power_iter_steps (int) – Number of power iteration steps to perform before QR decomposition. More steps can lead to better convergence but increased computation time.
max_update_rms (float) – Clip the update RMS to this value (0 means no clipping).
use_kl_shampoo (bool) – Whether to use KL-Shampoo correction.
correct_shampoo_beta_bias (bool | None) – Whether to correct shampoo beta bias. Decoupled it from correct_bias for testability because reference implementation of Soap doesn’t bias correct shampoo beta.
stream_list (list[Stream] | None) – Optional list of CUDA streams. When provided, each parameter in the inner loop uses a stream from this list in round-robin fashion.
- emerging_optimizers.legacy_soap.soap.project_in(x, eigenbasis_list)[source]#
Projects a tensor into the eigenbases
Note
For 2D tensors, we can use matmul instead of tensordot for code legibility. However, the code has been using tensordot historically, so does the reference implementation. It is difficult to match matmul and tensordot outputs exactly because of underlying floating point arithmetic differences. Therefore, we decided to keep using tensordot for consistency.
- emerging_optimizers.legacy_soap.soap.project_out(x, eigenbasis_list)[source]#
Projects a tensor out of the eigenbases
Note
Uses
tensordotrather thanmatmulfor the same numerical-consistency reason described inproject_in().
- emerging_optimizers.legacy_soap.soap.update_kronecker_factors(kronecker_factor_list, grad, shampoo_beta)[source]#
Updates the preconditioner matrices using gradient outer products.
This function updates the Kronecker factor matrices (L and R) used for preconditioning by computing and accumulating gradient outer products. kronecker_factor_list is updated in place.
- Parameters:
kronecker_factor_list (list[Tensor]) – List of preconditioner matrices (L and R) to update. Each matrix should be square and match the corresponding dimension of grad.
grad (Tensor) – Gradient tensor of the parameter being optimized
shampoo_beta (float) – Momentum coefficient for updating preconditioners. Controls how much weight to give to new vs old gradient statistics.
- Return type:
None
Example
>>> grad = torch.randn(10, 20) >>> L = torch.zeros(10, 10) >>> R = torch.zeros(20, 20) >>> update_kronecker_factors([L, R], grad, shampoo_beta=0.95)
- emerging_optimizers.legacy_soap.soap.update_kronecker_factors_kl_shampoo(
- kronecker_factor_list,
- grad,
- shampoo_beta,
- eigenbasis_list,
- eigvals_list,
- eps,
- eigval_exp=-1.0,
Updates the kronecker factor matrices in place using KL-Shampoo correction.
Implements the Kullback–Leibler minimization update from https://arxiv.org/pdf/2509.03378.
For a gradient \(G \in \mathbb{R}^{m \times n}\), current kronecker factors \(L_t \in \mathbb{R}^{m \times m}\), \(R_t \in \mathbb{R}^{n \times n}\), their orthonormal eigenbases \(Q_L, Q_R\), and the approximate eigenvalues in those eigenbases
\[\Lambda_L = \mathrm{diag}(Q_L^{\top} L_t Q_L), \quad \Lambda_R = \mathrm{diag}(Q_R^{\top} R_t Q_R) \](passed as
eigvals_list, typically computed and stored at the previous eigenbasis update, seeget_eigenbasis_qr()andget_eigenbasis_eigh()), the EMA update with momentum \(\beta\) (=shampoo_beta) and exponent \(p\) (=eigval_exp, default-1) is\[L_{t+1} = \beta\, L_t + \frac{1-\beta}{n}\, G\, Q_R\, \mathrm{diag}(\Lambda_R^{p})\, Q_R^{\top} G^{\top} \\ R_{t+1} = \beta\, R_t + \frac{1-\beta}{m}\, G^{\top}\, Q_L\, \mathrm{diag}(\Lambda_L^{p})\, Q_L^{\top} G \]Eigenvalues are clamped to
epsfrom below before exponentiation for numerical stability.- Parameters:
kronecker_factor_list (Iterable[Tensor]) – List of preconditioner matrices (L and R) to update.
grad (Tensor) – Gradient tensor of the parameter being optimized
shampoo_beta (float) – Momentum coefficient for updating preconditioners.
eigenbasis_list (Iterable[Tensor]) – List of orthonormal eigenbases of the kronecker factor matrices
eigvals_list (Iterable[Tensor]) – List of approximate eigenvalues of each kronecker factor in its eigenbasis.
eps (float) – Small offset for numerical stability.
eigval_exp (float) – Exponent applied to the (clamped) eigenvalues.
- Return type:
None
- emerging_optimizers.legacy_soap.soap.update_eigenbasis_and_exp_avgs(
- kronecker_factor_list,
- eigenbasis_list,
- exp_avg_sq,
- exp_avg,
- use_eigh=False,
- power_iter_steps=1,
Updates the eigenbases and moving averages.
This function performs an update of the eigenbases (QL and QR) used for preconditioning. It follows these steps:
Projects exp_avg back to the original basis
Updates the eigenbases using QR decomposition and power iteration (orthogonal iteration)
Projects exp_avg back to the new eigenbasis
- Parameters:
kronecker_factor_list (list[Tensor]) – List of preconditioner matrices (L and R) that define the optimization landscape. These are updated with gradient statistics.
eigenbasis_list (list[Tensor]) – List of current eigenbases (QL and QR) used for preconditioning. These will be updated by this function.
exp_avg_sq (Tensor) – Inner Adam’s second moment tensor, used for scaling the preconditioner updates. Permuted along each kronecker-factor axis on the QR path to track the sorted eigenbasis columns; returned unchanged on the eigh path.
exp_avg (Tensor) – Inner Adam’s first moment tensor, used for tracking gradient momentum.
use_eigh (bool) – Whether to use full symmetric eigendecomposition (eigh) to compute the eigenbasis. If False, use orthogonal iteration to compute the eigenbasis.
power_iter_steps (int) – Number of power iteration steps to perform before QR decomposition. More steps can lead to better convergence but increased computation time.
- Returns:
List of (approximate) eigenvalues of each kronecker factor in its updated eigenbasis
Updated list of eigenbases (QL and QR)
Updated exp_avg tensor projected to the new eigenbasis
Updated exp_avg_sq tensor
- Return type:
A tuple containing
Example
>>> L = torch.randn(10, 10) >>> R = torch.randn(20, 20) >>> QL = torch.randn(10, 10) >>> QR = torch.randn(20, 20) >>> exp_avg_sq = torch.randn(10, 20) >>> exp_avg = torch.randn(10, 20) >>> eigvals_list, updated_eigenbasis_list, updated_exp_avg, updated_exp_avg_sq = ( ... update_eigenbasis_and_exp_avgs([L, R], [QL, QR], exp_avg_sq, exp_avg))
REKLS#
- class emerging_optimizers.legacy_soap.rekls.REKLS(
- params,
- lr,
- betas=(0.9, 0.95),
- shampoo_beta=0.95,
- eps=1e-08,
- weight_decay=0.1,
- *,
- weight_decay_method='decoupled',
REKLS (Realtime Eigen Kullback-Leibler Soap) optimizer.
REKLS is a variant of SOAP that uses the up to date eigenbasis calculated by Eigen decomposition. It is “up to date” because current step’s gradient is accumulated to the kronecker factor before eigenbasis update.
Note
Refer to
SOAPfor detailed documentation of arguments.
StackedSoap#
- class emerging_optimizers.legacy_soap.stacked_soap.StackedSoap(
- params,
- lr,
- betas=(0.9, 0.95),
- shampoo_beta=0.95,
- eps=1e-08,
- weight_decay=0.1,
Limited-memory SOAP for batched / 3D parameters via transient 2D stacking.
Optimizes the real parameters directly:
self.param_groups,self.state, and gradients are all keyed by the user’s parameters, so learning-rate schedulers, gradient clipping, andstate_dictbehave exactly as for plainSOAP. Each 3D parameter is flattened to 2D by merging its batch dim into the smaller matrix edge (see_stack_2d()) only for the duration ofstep(): the parameter’sdataandgradare swapped to their 2D views, the inherited SOAP step runs, and the 2D update is unstacked back into the original storage. Because the swap happens before the inherited step, its lazy state initialization sizes the optimizer state to the stacked 2D shape automatically.Stacking on the smaller edge keeps both Kronecker factors small (the larger edge becomes a single shared factor) while reusing the full, unmodified SOAP machinery (KL-Shampoo + QR eigenbasis). The stacking is a storage-sharing view except for the permute branch (
q <= p), which allocates one transient 2D buffer per step. A plain 2D parameter is stacked as itself, so this is exactly stock SOAP.SOAP is configured with the fixed settings appropriate for this use: decoupled weight decay, no Nesterov, bias correction on, the QR eigenbasis path with 1 power-iteration step, KL-Shampoo on, and the default matmul precision.
- Parameters:
params (Iterable[Tensor] | Iterable[dict[str, Any]] | Iterable[tuple[str, Tensor]]) – Iterable of 2D or 3D parameters to optimize or dicts defining parameter groups.
lr (float) – The learning rate.
shampoo_beta (float) – Beta for the kronecker factor moving average.
eps (float) – Inner Adam epsilon.
weight_decay (float) – Decoupled weight decay coefficient.
MOSO#
- class emerging_optimizers.legacy_soap.moso.MOSO(
- params,
- lr=0.0003,
- momentum=0.95,
- rms_beta=0.95,
- shampoo_beta=0.95,
- eps=1e-08,
- weight_decay=0.01,
- *,
- max_update_rms=0.0,
Momentum One-Sided SOAP.
MOSO tracks EMA momentum like Muon, accumulates a SOAP/Shampoo-style covariance of that momentum on the smaller matrix side, and applies an RMSProp update in the covariance eigenbasis. Conceptually, this is one-sided SOAP where
G_t G_t^Tis replaced byM_t M_t^T(orM_t^T M_tfor the right side), and the update is computed by projecting the momentum into the eigenbasis, applying RMSProp there, and projecting back:\[C_t = \beta_s C_{t-1} + (1 - \beta_s) M_t M_t^T,\quad C_t = Q_M \Lambda_M Q_M^T U_t = Q_M \operatorname{RMSprop}(Q_M^T M_t)\]for the left-preconditioned case where
M_t.shape[0] <= M_t.shape[1]; the right-preconditioned case usesC_t = M_t^T M_tand computesU_t = \operatorname{RMSprop}(M_t Q_M) Q_M^T.- Parameters:
params (Iterable[Tensor] | Iterable[dict[str, Any]] | Iterable[tuple[str, Tensor]]) – Iterable of parameters to optimize or dicts defining parameter groups.
lr (float) – Learning rate.
momentum (float) – EMA coefficient for the Muon-style momentum.
rms_beta (float) – EMA coefficient for the second-moment (RMS) normalization in the eigenbasis.
shampoo_beta (float) – EMA coefficient for the one-sided momentum covariance.
eps (float) – RMSProp epsilon for numerical stability.
weight_decay (float) – Weight decay coefficient.
max_update_rms (float) – Clip the update RMS to this value (0 means no clipping).
TpRekls#
- class emerging_optimizers.legacy_soap.tp_rekls.TpRekls(
- params,
- lr,
- betas=(0.9, 0.95),
- shampoo_beta=0.95,
- eps=1e-08,
- weight_decay=0.1,
- *,
- weight_decay_method='decoupled',
- tp_group,
- fp32_matmul_prec='high',
Tensor-parallel variant of
REKLS.Reimplemented from scratch (not inheriting from
SOAP) so the tensor-parallel bookkeeping stays isolated. Eigenbases are not stored in optimizer state; they are recomputed viaget_eigenbasis_eigh()from the kronecker factors. Each step calls eigh twice — once on the pre-update L, R for theupdate_kronecker_factors_kl_shampoo()correction, and once on the post-update L, R for the gradient projection.- State per parameter (one entry per rank):
stepexp_avg,exp_avg_sq: full-size tensors duplicated acrosstp_groupranks.exp_avgis rotated through the basis change between steps (project back via the pre-update eigenbasis, then forward via the post-update eigenbasis), matching SOAP’supdate_eigenbasis_and_exp_avgs().exp_avg_sqis not rotated, matching SOAP’s eigh path.L,R: kronecker factor matrices, sharded along dimension 0 acrosstp_group.
- Parameters:
params (Iterable[Tensor] | Iterable[dict[str, Any]] | Iterable[tuple[str, Tensor]]) – Iterable of parameters to optimize or dicts defining parameter groups.
lr (float) – Learning rate.
shampoo_beta (float) – Beta for the kronecker factor moving average.
eps (float) – Inner Adam epsilon.
weight_decay (float) – Weight decay coefficient.
weight_decay_method (Literal['decoupled', 'independent', 'l2', 'palm']) – See
WeightDecayMixin.tp_group (ProcessGroup) – Process group across which parameters and gradients are sharded.
fp32_matmul_prec (Literal['highest', 'high', 'medium']) – Precision for the optimizer-state GEMM operations.
Note
Sharding is configured per-parameter-group via
partition_dim(an int in{0, 1}, orNonefor replicated parameters). Mixed-layout models should use one group per distinctpartition_dim:optimizer = TpRekls([ {"params": column_parallel_params, "partition_dim": 0}, {"params": row_parallel_params, "partition_dim": 1}, {"params": replicated_params, "partition_dim": None}, ], lr=1e-3, tp_group=tp_group)
Groups without
partition_dimuse the default (None→ replicated, plain non-TP REKLS step on each rank, no collectives, full-sizeL/R).
emerging_optimizers.legacy_soap.soap_utils#
- emerging_optimizers.legacy_soap.soap_utils.get_eigenbasis_eigh(kronecker_factor_list)[source]#
Computes the eigenvalues and eigenbases of the preconditioner using torch.linalg.eigh decomposition.
- emerging_optimizers.legacy_soap.soap_utils.get_eigenbasis_qr(
- kronecker_factor_list,
- eigenbasis_list,
- power_iter_steps=1,
Updates the eigenbases of the preconditioner using power iteration and QR.
- Parameters:
kronecker_factor_list (Iterable[Tensor]) – List of preconditioner matrices (L and R).
eigenbasis_list (Iterable[Tensor]) – List of current eigenbases (QL and QR).
power_iter_steps (int) – Number of power iteration steps to perform before QR decomposition. More steps can lead to better convergence but increased computation time.
- Returns:
Tuple of (list of approximate eigenvalues in descending order, updated list of orthonormal eigenbases (QL and QR) with columns ordered to match).
- Return type:
- emerging_optimizers.legacy_soap.soap_utils.get_eigenbasis_svd(kronecker_factor_list)[source]#
Computes the eigenbases of the preconditioner using torch.linalg.svd decomposition.
The kronecker factors \(L = GG^\top\) and \(R = G^\top G\) are symmetric positive semi-definite, so the left and right singular vectors coincide (up to sign in the presence of repeated singular values); this function returns the left singular vectors \(U\) as the eigenbasis. Singular values from
torch.linalg.svdare returned in descending order.