nemo_automodel.components.optim.gradient_sketch
nemo_automodel.components.optim.gradient_sketch
Gradient sketching: rank-r projected gradients with a compact row-wise optimizer.
Implements the optimizer side of LoGRA (Zhang et al., “Scaling LLM reinforcement
learning with low-rank gradient sketches”, arXiv:2610.06647). Selected linear
weights never form a dense gradient: backward accumulates S = Hᵀ(XAᵀ) into a
shared buffer, RowAdam keeps one second moment per output row, and the update
ΔW = -lr · D · A is merged into the full weight each step.
Submodules
nemo_automodel.components.optim.gradient_sketch.optimizernemo_automodel.components.optim.gradient_sketch.row_adamnemo_automodel.components.optim.gradient_sketch.sketch
Package Contents
Classes
Functions
API
Bases: Optimizer
RowAdam or SGD on gradient sketches, with a native optimizer for everything else.
param_groups[0] holds the sketch targets (plus update_count and the
settings); the remaining groups are the native optimizer’s own group objects (none
when every non-target parameter is frozen), and state is shared with it. state
holds native-parameter state only, so it is empty until the first native step and
torch.distributed.checkpoint initializes it for any native optimizer. Sketch-target
state (flat RowAdam row moments, fp32 masters of low-precision targets) is kept on the
wrapper and exposed through sketch_state_dict / load_sketch_state_dict
(Automodel’s OptimizerState saves it under the "sketch" key). Construct it before
the native optimizer’s first step: target parameters are removed before it can allocate
full-size moment buffers.
Number of completed optimizer updates, stored in param_groups[0] for checkpoints.
Create an fp32 master copy; a DTensor with the weight’s layout under FSDP so DCP shards it.
Restore state produced by sketch_state_dict.
Parameters:
Mapping with the same keys as sketch_state_dict.
Raises:
ValueError: If a key is missing or unexpected, or a shape differs.
Load native state, group settings and update_count from state_dict.
The native half is handed to native.load_state_dict (torch.optim.Optimizer
casts loaded state to the parameter dtype; TE FusedAdam restores fp32 there).
Sketch-target state is not read here; use load_sketch_state_dict.
Parameters:
A dict produced by state_dict.
Raises:
ValueError: If the saved sketch settings differ from the configured ones, or the saved group layout differs from the current one.
Redraw every projection on its device from (seed, update_count, layer name).
Rebase fp32 masters after loading model weights without optimizer state.
Live tensors holding the state of sketch-target parameters.
Returns: dict[str, Tensor]
row_second_moment (the flat [rows, 1] fp32 buffer, identical on every rank, only
Pack param_groups[0] (settings, update_count) with the native optimizer’s own state dict.
The native half comes from native.state_dict() so an optimizer with its own
serialization (TE FusedAdam unscales low-precision moments to fp32) is honoured;
its parameter indices are shifted past the sketch targets so the layout matches
torch.optim.Optimizer’s packing of all groups in order.
Apply the sketch update to targets and the native step to everything else.
An unready call with every group’s lr equal to 0 is the state-initialization probe
used by torch.distributed.checkpoint: only the native optimizer runs, so its state
is created; sketches, projections and update_count are untouched.
Parameters:
Unsupported; must be None.
Multiplies only the sketch-target update, not its weight decay or the native step.
Returns: float | None
None; closures are unsupported, so there is no loss to return.
Raises:
RuntimeError: Ifsynchronize_and_cliphas not run this step (and this is not the zero-lr probe), or a closure is given.
Average sketches across data-parallel ranks, then clip the implied gradient globally.
After the all-reduce every rank holds the full averaged sketch, so the sketch part of the
norm is computed locally over all rows; only the (sharded) native gradient norm is reduced.
process_group must be the FSDP shard group; HSDP replicate groups are unsupported.
Parameters:
FSDP shard group, or None for the default group / a single process.
Clip the global gradient norm to this value, or None to only measure it.
Returns: Tensor
The pre-clip global norm as a 0-d tensor on the sketch device.
Raises:
RuntimeError: If sketches were already synchronized this step, targets are not FSDP-sharded on more than one rank, or projections differ across ranks.FloatingPointError: If the norm is not finite.
Move native state, the flat row moments and the fp32 masters to device.
Clear gradients of every parameter group and zero the sketch buffer.
Parameters:
Set gradients to None instead of zero, as in torch.optim.Optimizer.
Validated settings shared by target selection, hooks, and the optimizer.
RowAdam second-moment decay.
"rademacher" (±1/√r) or "gaussian" (N(0, 1/r)).
RowAdam denominator floor.
Freeze every parameter that is not a sketch target.
"row_adam" or "sgd" rule applied to the sketch.
Sketch rank r (columns of S, rows of A).
Redraw A after every update; False keeps the update-0 draw.
Base seed; each layer and update derives its own seed from it.
Last name segments of nn.Linear modules to sketch.
Names of 3-D fused expert parameters to sketch.
Restrict selection to modules under these prefixes.
Plain-value form stored in optimizer param_groups for checkpoints.
Per-weight sketch; model parameter names and checkpoint keys stay unchanged.
sketch is a [rows, rank] view into one buffer shared by all targets (see
install_sketches), so the optimizer can reduce, scale and zero every sketch
with a single kernel or collective. rows is the product of all but the last
weight dimension, so fused 3-D expert banks are flattened.
Restore the real parameter after the fused expert forward.
Swap the fused expert weight for a view whose gradient is projected.
Project the layer input and register the sketch accumulation for backward.
Return the projection in dtype, reusing one persistent copy.
Mirror a refreshed projection into the low-precision copy.
Overwrite out with a fresh [rank, width] projection drawn on its own device.
Drawing on the device keeps the host out of the per-step refresh: no CPU random stream, no temporaries, no host-to-device copy. Every data-parallel rank draws from the same seed on identical hardware and therefore holds the same matrix; the optimizer verifies that once before the first update.
Parameters:
Preallocated [rank, width] tensor to overwrite in place.
Seed for the device-side generator.
"rademacher" or "gaussian".
Returns: Tensor
The same tensor out, now filled.
Raises:
ValueError: If a dimension is non-positive or the distribution is unknown.
Install sketch hooks on every target and allocate the shared sketch buffer.
Linear targets accumulate sketches directly in backward. Fused experts keep their
forward implementation and project the transient backward gradient. Target weights
get requires_grad=False so autograd never allocates their dense gradient.
Outputs of sketched linear layers must not be modified in place; a violation fails loudly at forward time.
Return the local 2-D weight rows and their global row offset.
Plain tensors are returned whole (3-D expert banks flattened to 2-D). FSDP
row-sharded DTensors return the local shard and its offset; any sharding on
a non-leading dimension (tensor parallelism) is rejected.
Parameters:
A plain or DTensor weight of 2 or more dimensions.
Returns: tuple[Tensor, int]
A (rows, offset) pair: the local 2-D rows and the global index of their first row.
Raises:
ValueError: If the weight is sharded on a dimension other than the leading one.
Allocate a [rank, width] projection and fill it on device.
Parameters:
Number of rows r of the projection.
Number of columns, the weight’s input dimension.
Seed for the device-side generator.
Device to allocate and draw on.
Element type of the projection.
"rademacher" or "gaussian".
Returns: Tensor
The filled projection tensor.
Raises:
ValueError: Ifrankorwidthis non-positive.
Generate a stable per-layer seed independently of module wrappers.
Parameters:
Run-level seed from SketchSettings.seed.
Optimizer update index; changes the seed on every refresh.
Fully qualified module name; FSDP and checkpoint wrapper segments are ignored.
Returns: int
A non-negative 62-bit seed.
Normalize each sketch row by its bias-corrected running RMS.
The state is indexed by the original weight’s output rows, not by the changing
projection coordinates. With inplace the sketch is overwritten with the
direction, avoiding a full-size temporary when the sketch is no longer needed.
Parameters:
[rows, rank] accumulated sketch.
[rows, 1] running second moment, updated in place.
1-based update count used for bias correction.
Second-moment decay in [0, 1).
Denominator floor.
Overwrite sketch with the direction.
Returns: Tensor
The normalized direction, same shape as sketch.
Return (name, module, parameter_name, is_fused_expert) for every sketch target.
Return the shared buffer that every SketchState.sketch is a view of.