nemo_automodel.components.optim.gradient_sketch.row_adam
nemo_automodel.components.optim.gradient_sketch.row_adam
Adam-style second moments shared across each sketch row; no first moment.
Module Contents
Functions
API
Normalize each sketch row by its bias-corrected running RMS.
The state is indexed by the original weight’s output rows, not by the changing
projection coordinates. With inplace the sketch is overwritten with the
direction, avoiding a full-size temporary when the sketch is no longer needed.
Parameters:
sketch
[rows, rank] accumulated sketch.
second_moment
[rows, 1] running second moment, updated in place.
step
1-based update count used for bias correction.
beta2
Second-moment decay in [0, 1).
epsilon
Denominator floor.
inplace
Overwrite sketch with the direction.
Returns: Tensor
The normalized direction, same shape as sketch.