emerging_optimizers.riemannian_optimizers#

class emerging_optimizers.riemannian_optimizers.normalized_optimizer.ObliqueAdam(
params,
lr=0.001,
betas=(0.9, 0.99),
weight_decay=0.0,
*,
weight_decay_method='decoupled',
dim=0,
eps=1e-08,
correct_bias=True,
scale_mode='unit_l2_norm',
)[source]#

Adam optimizer for row- or column-normalized 2D parameters on oblique manifolds.

This optimizer adapts an Adam-like algorithm to work on oblique manifolds, where parameters are constrained to have unit-norm rows or columns. It combines adaptive momentum estimation with Riemannian gradient computation and manifold retraction.

Parameters:
step(closure: None = None) → None[source]#
step(
closure: Callable[[], float],
) → float

Performs a single optimization step.

Parameters:

closure – Unsupported; must be None.

class emerging_optimizers.riemannian_optimizers.normalized_optimizer.ObliqueSGD(
params,
lr=0.001,
momentum=0.9,
weight_decay=0.0,
*,
weight_decay_method='decoupled',
dim=0,
eps=1e-08,
scale_mode='unit_l2_norm',
)[source]#

SGD optimizer for row- or column-normalized 2D parameters on oblique manifolds.

This optimizer performs SGD on oblique manifolds, where parameters are constrained to have unit-norm rows or columns. It implements Riemannian SGD with manifold-aware gradient updates and retraction operations.

References

Parameters:
  • lr (float) – learning rate

  • momentum (float) – momentum coefficient

  • weight_decay (float) – weight decay coefficient

  • weight_decay_method (Literal['decoupled', 'independent', 'l2', 'palm']) – Method to apply weight decay.

  • dim (int) – The dimension to normalize over

  • eps (float) – epsilon for numerical stability

  • scale_mode (Literal['unit_l2_norm', 'unit_rms_norm']) – The type of scale factor to use for the Riemannian gradient and retraction computation. Defaults to β€œunit_l2_norm” scaling.

  • params (list[Parameter])

step(closure: None = None) → None[source]#
step(
closure: Callable[[], float],
) → float

Performs a single optimization step.

Parameters:

closure – Unsupported; must be None.

class emerging_optimizers.riemannian_optimizers.normalized_optimizer.ObliqueSteepestAdam(
params,
lr=0.001,
betas=(0.9, 0.99),
weight_decay=0.0,
*,
weight_decay_method='decoupled',
dim=0,
eps=1e-08,
correct_bias=True,
scale_mode='unit_rms_norm',
)[source]#

Steepest descent Adam optimizer for parameters on Oblique manifolds.

This optimizer blends the adaptive momentum estimation of Adam with the Muon-style normalized update direction. It computes the ambient Adam step, projects it onto the tangent space, and then normalizes the resulting vector.

This guarantees that the final step direction incorporates Adam’s historical second-moment scaling, while the actual step size taken is uniformly bounded by the learning rate via the Linear Minimization Oracle (LMO) constraint.

Parameters:
class emerging_optimizers.riemannian_optimizers.normalized_optimizer.ObliqueSteepestSGD(
params,
lr=0.001,
momentum=0.9,
weight_decay=0.0,
*,
weight_decay_method='decoupled',
dim=0,
eps=1e-08,
scale_mode='unit_rms_norm',
)[source]#

Muon-style steepest descent optimizer for parameters on Oblique manifolds.

Unlike standard Riemannian SGD, this optimizer computes updates using a Linear Minimization Oracle (LMO) with respect to the \(l_1 \to \text{RMS}\) norm constraint. Mathematically, this is equivalent to finding the Euclidean descent direction, projecting it onto the tangent space, and normalizing the resulting directional vector.

This approach effectively strips the magnitude of the gradient, applying uniform step sizes across all dimensions, mirroring the behavior of the Muon optimizer for Stiefel manifolds when the input features are one-hot encoded.

Parameters:
class emerging_optimizers.riemannian_optimizers.isospectral.Iso(
params,
lr=0.001,
momentum=0.9,
retraction='qr',
*,
fp32_matmul_prec='highest',
)[source]#

Isospectral optimizer for two-dimensional parameters.

The optimizer factorizes each parameter as U @ diag(Sigma) @ V.T and updates both Stiefel factors while keeping Sigma fixed. It is designed for reinforcement learning with verifiable rewards (RLVR), particularly for LLM reasoning post-training, but its implementation does not depend on an RL-specific training interface.

References

  • ISO: An RLVR-Native Optimization Stack. arXiv:2607.19331 (2026). [arXiv:2607.19331]

Parameters:
  • params (Iterable[Tensor] | Iterable[dict[str, Any]] | Iterable[tuple[str, Tensor]]) – Parameters to optimize.

  • lr (float) – Learning rate.

  • momentum (float) – Momentum coefficient.

  • retraction (Literal['qr', 'polar', 'cayley', 'newton_schulz']) – Retraction used to restore the Stiefel constraints.

  • fp32_matmul_prec (Literal['highest', 'high', 'medium']) – Precision used for FP32 matrix multiplications.

step(closure: None = None) → None[source]#
step(closure: Callable[[], float]) → float

Perform a single optimization step.

emerging_optimizers.riemannian_optimizers.retractions.stiefel.cayley_retraction(point, momentum, step_size)[source]#

Retract a tangent step back to the Stiefel manifold via the Cayley transform.

Constructs a skew-symmetric matrix \(A = D X^T - X D^T\) from the update direction \(D = -\text{momentum}\) and applies the PadΓ© approximation to the matrix exponential: \((I - \frac{\eta}{2} A)^{-1} (I + \frac{\eta}{2} A) X\).

Parameters:
  • point (Tensor) – Current matrix on the Stiefel manifold of shape (M, K).

  • momentum (Tensor) – Tangent vector or momentum update of shape (M, K).

  • step_size (float) – Step size scaling the update direction (\(\eta\)).

Returns:

The retracted orthonormal matrix of shape (M, K) satisfying \(Q^T Q = I_K\).

Return type:

Tensor

emerging_optimizers.riemannian_optimizers.retractions.stiefel.newton_schulz_retraction(
point,
momentum,
step_size,
coefficient_type='polar_express',
num_ns_steps=8,
)[source]#

Retract a tangent step back to the Stiefel manifold via iterative Newton-Schulz iterations.

Approximates the polar factor / matrix sign function of \(X = \text{point} - \text{step\_size} \times \text{momentum}\) using iterative matrix polynomial evaluations directly on Tensor Cores, avoiding host-device synchronization and LAPACK factorization overhead.

Parameters:
  • point (Tensor) – Current matrix on the Stiefel manifold of shape (M, K).

  • momentum (Tensor) – Tangent vector or momentum update of shape (M, K).

  • step_size (float) – Step size scaling the update direction.

  • coefficient_type (Literal['simple', 'quintic', 'polar_express', 'cans', 'aol', 'deepseekv4', 'custom', 'cubic5']) – Polynomial coefficient scheme used for Newton-Schulz steps. Defaults to "polar_express".

  • num_ns_steps (int) – Number of iterative polynomial steps to perform. Defaults to 8.

Returns:

The retracted approximately orthonormal matrix of shape (M, K) satisfying \(\|Q^T Q - I_K\|_\infty \le \epsilon\).

Return type:

Tensor

emerging_optimizers.riemannian_optimizers.retractions.stiefel.polar_retraction(point, momentum, step_size)[source]#

Retract a tangent step back to the Stiefel manifold via the analytical polar factor.

Computes the closest orthonormal matrix in Frobenius norm to \(X = \text{point} - \text{step\_size} \times \text{momentum}\) using the singular value decomposition \(X = U \Sigma V^T \implies \operatorname{polar}(X) = U V^T\).

Parameters:
  • point (Tensor) – Current matrix on the Stiefel manifold of shape (M, K).

  • momentum (Tensor) – Tangent vector or momentum update of shape (M, K).

  • step_size (float) – Step size scaling the update direction.

Returns:

The retracted orthonormal matrix of shape (M, K) satisfying \(Q^T Q = I_K\).

Return type:

Tensor

emerging_optimizers.riemannian_optimizers.retractions.stiefel.qr_retraction(point, momentum, step_size)[source]#

Retract a tangent step back to the Stiefel manifold via reduced QR decomposition.

Forms the intermediate matrix \(X = \text{point} - \text{step\_size} \times \text{momentum}\), computes its reduced QR factorization, and resolves column phase ambiguity by ensuring positive diagonal elements in \(R\).

Parameters:
  • point (Tensor) – Current matrix on the Stiefel manifold of shape (M, K).

  • momentum (Tensor) – Tangent vector or momentum update of shape (M, K).

  • step_size (float) – Step size scaling the update direction.

Returns:

The retracted orthonormal matrix of shape (M, K) satisfying \(Q^T Q = I_K\).

Return type:

Tensor