Skip to navigation

nemo_automodel.components.optim.gradient_sketch.row_adam

View as MarkdownOpen in Claude

Adam-style second moments shared across each sketch row; no first moment.

Module Contents

Functions

NameDescription
row_adam_directionNormalize each sketch row by its bias-corrected running RMS.

API

nemo_automodel.components.optim.gradient_sketch.row_adam.row_adam_direction(
sketch: torch.Tensor,
second_moment: torch.Tensor,
step: int,
beta2: float,
epsilon: float,
inplace: bool = False
) -> torch.Tensor

Normalize each sketch row by its bias-corrected running RMS.

The state is indexed by the original weight’s output rows, not by the changing projection coordinates. With inplace the sketch is overwritten with the direction, avoiding a full-size temporary when the sketch is no longer needed.

Parameters:

sketch
Tensor

[rows, rank] accumulated sketch.

second_moment
Tensor

[rows, 1] running second moment, updated in place.

step
int

1-based update count used for bias correction.

beta2
float

Second-moment decay in [0, 1).

epsilon
float

Denominator floor.

inplace
boolDefaults to False

Overwrite sketch with the direction.

Returns: Tensor

The normalized direction, same shape as sketch.