emerging_optimizers.riemannian_optimizers#
- class emerging_optimizers.riemannian_optimizers.normalized_optimizer.ObliqueAdam(
- params,
- lr=0.001,
- betas=(0.9, 0.99),
- weight_decay=0.0,
- *,
- weight_decay_method='decoupled',
- dim=0,
- eps=1e-08,
- correct_bias=True,
- scale_mode='unit_l2_norm',
Adam optimizer for row- or column-normalized 2D parameters on oblique manifolds.
This optimizer adapts an Adam-like algorithm to work on oblique manifolds, where parameters are constrained to have unit-norm rows or columns. It combines adaptive momentum estimation with Riemannian gradient computation and manifold retraction.
- Parameters:
- class emerging_optimizers.riemannian_optimizers.normalized_optimizer.ObliqueSGD(
- params,
- lr=0.001,
- momentum=0.9,
- weight_decay=0.0,
- *,
- weight_decay_method='decoupled',
- dim=0,
- eps=1e-08,
- scale_mode='unit_l2_norm',
SGD optimizer for row- or column-normalized 2D parameters on oblique manifolds.
This optimizer performs SGD on oblique manifolds, where parameters are constrained to have unit-norm rows or columns. It implements Riemannian SGD with manifold-aware gradient updates and retraction operations.
References
An Introduction to Optimization on Smooth Manifolds (Nicolas Boumal)
Jianlin Su: https://kexue.fm/archives/11196
Raman et al.: https://arxiv.org/abs/1909.06463
Franz Cesista: https://leloykun.github.io/ponder/steepest-descent-stiefel/#6-bonus-a-muon-like-optimizer-for-the-embedding-and-unembedding-layers
- Parameters:
lr (float) β learning rate
momentum (float) β momentum coefficient
weight_decay (float) β weight decay coefficient
weight_decay_method (Literal['decoupled', 'independent', 'l2', 'palm']) β Method to apply weight decay.
dim (int) β The dimension to normalize over
eps (float) β epsilon for numerical stability
scale_mode (Literal['unit_l2_norm', 'unit_rms_norm']) β The type of scale factor to use for the Riemannian gradient and retraction computation. Defaults to βunit_l2_normβ scaling.
- class emerging_optimizers.riemannian_optimizers.normalized_optimizer.ObliqueSteepestAdam(
- params,
- lr=0.001,
- betas=(0.9, 0.99),
- weight_decay=0.0,
- *,
- weight_decay_method='decoupled',
- dim=0,
- eps=1e-08,
- correct_bias=True,
- scale_mode='unit_rms_norm',
Steepest descent Adam optimizer for parameters on Oblique manifolds.
This optimizer blends the adaptive momentum estimation of Adam with the Muon-style normalized update direction. It computes the ambient Adam step, projects it onto the tangent space, and then normalizes the resulting vector.
This guarantees that the final step direction incorporates Adamβs historical second-moment scaling, while the actual step size taken is uniformly bounded by the learning rate via the Linear Minimization Oracle (LMO) constraint.
- class emerging_optimizers.riemannian_optimizers.normalized_optimizer.ObliqueSteepestSGD(
- params,
- lr=0.001,
- momentum=0.9,
- weight_decay=0.0,
- *,
- weight_decay_method='decoupled',
- dim=0,
- eps=1e-08,
- scale_mode='unit_rms_norm',
Muon-style steepest descent optimizer for parameters on Oblique manifolds.
Unlike standard Riemannian SGD, this optimizer computes updates using a Linear Minimization Oracle (LMO) with respect to the \(l_1 \to \text{RMS}\) norm constraint. Mathematically, this is equivalent to finding the Euclidean descent direction, projecting it onto the tangent space, and normalizing the resulting directional vector.
This approach effectively strips the magnitude of the gradient, applying uniform step sizes across all dimensions, mirroring the behavior of the Muon optimizer for Stiefel manifolds when the input features are one-hot encoded.
- class emerging_optimizers.riemannian_optimizers.isospectral.Iso(
- params,
- lr=0.001,
- momentum=0.9,
- retraction='qr',
- *,
- fp32_matmul_prec='highest',
Isospectral optimizer for two-dimensional parameters.
The optimizer factorizes each parameter as
U @ diag(Sigma) @ V.Tand updates both Stiefel factors while keepingSigmafixed. It is designed for reinforcement learning with verifiable rewards (RLVR), particularly for LLM reasoning post-training, but its implementation does not depend on an RL-specific training interface.References
ISO: An RLVR-Native Optimization Stack. arXiv:2607.19331 (2026). [arXiv:2607.19331]
- Parameters:
params (Iterable[Tensor] | Iterable[dict[str, Any]] | Iterable[tuple[str, Tensor]]) β Parameters to optimize.
lr (float) β Learning rate.
momentum (float) β Momentum coefficient.
retraction (Literal['qr', 'polar', 'cayley', 'newton_schulz']) β Retraction used to restore the Stiefel constraints.
fp32_matmul_prec (Literal['highest', 'high', 'medium']) β Precision used for FP32 matrix multiplications.
- emerging_optimizers.riemannian_optimizers.retractions.stiefel.cayley_retraction(point, momentum, step_size)[source]#
Retract a tangent step back to the Stiefel manifold via the Cayley transform.
Constructs a skew-symmetric matrix \(A = D X^T - X D^T\) from the update direction \(D = -\text{momentum}\) and applies the PadΓ© approximation to the matrix exponential: \((I - \frac{\eta}{2} A)^{-1} (I + \frac{\eta}{2} A) X\).
- Parameters:
- Returns:
The retracted orthonormal matrix of shape
(M, K)satisfying \(Q^T Q = I_K\).- Return type:
- emerging_optimizers.riemannian_optimizers.retractions.stiefel.newton_schulz_retraction(
- point,
- momentum,
- step_size,
- coefficient_type='polar_express',
- num_ns_steps=8,
Retract a tangent step back to the Stiefel manifold via iterative Newton-Schulz iterations.
Approximates the polar factor / matrix sign function of \(X = \text{point} - \text{step\_size} \times \text{momentum}\) using iterative matrix polynomial evaluations directly on Tensor Cores, avoiding host-device synchronization and LAPACK factorization overhead.
- Parameters:
point (Tensor) β Current matrix on the Stiefel manifold of shape
(M, K).momentum (Tensor) β Tangent vector or momentum update of shape
(M, K).step_size (float) β Step size scaling the update direction.
coefficient_type (Literal['simple', 'quintic', 'polar_express', 'cans', 'aol', 'deepseekv4', 'custom', 'cubic5']) β Polynomial coefficient scheme used for Newton-Schulz steps. Defaults to
"polar_express".num_ns_steps (int) β Number of iterative polynomial steps to perform. Defaults to 8.
- Returns:
The retracted approximately orthonormal matrix of shape
(M, K)satisfying \(\|Q^T Q - I_K\|_\infty \le \epsilon\).- Return type:
- emerging_optimizers.riemannian_optimizers.retractions.stiefel.polar_retraction(point, momentum, step_size)[source]#
Retract a tangent step back to the Stiefel manifold via the analytical polar factor.
Computes the closest orthonormal matrix in Frobenius norm to \(X = \text{point} - \text{step\_size} \times \text{momentum}\) using the singular value decomposition \(X = U \Sigma V^T \implies \operatorname{polar}(X) = U V^T\).
- Parameters:
- Returns:
The retracted orthonormal matrix of shape
(M, K)satisfying \(Q^T Q = I_K\).- Return type:
- emerging_optimizers.riemannian_optimizers.retractions.stiefel.qr_retraction(point, momentum, step_size)[source]#
Retract a tangent step back to the Stiefel manifold via reduced QR decomposition.
Forms the intermediate matrix \(X = \text{point} - \text{step\_size} \times \text{momentum}\), computes its reduced QR factorization, and resolves column phase ambiguity by ensuring positive diagonal elements in \(R\).
- Parameters:
- Returns:
The retracted orthonormal matrix of shape
(M, K)satisfying \(Q^T Q = I_K\).- Return type: