nemo_automodel.components.optim.gradient_sketch.optimizer
nemo_automodel.components.optim.gradient_sketch.optimizer
Optimize gradient sketches while delegating ordinary parameters to a native optimizer.
Module Contents
Classes
Functions
Data
API
Bases: Optimizer
RowAdam or SGD on gradient sketches, with a native optimizer for everything else.
param_groups[0] holds the sketch targets (plus update_count and the
settings); the remaining groups are the native optimizer’s own group objects (none
when every non-target parameter is frozen), and state is shared with it. state
holds native-parameter state only, so it is empty until the first native step and
torch.distributed.checkpoint initializes it for any native optimizer. Sketch-target
state (flat RowAdam row moments, fp32 masters of low-precision targets) is kept on the
wrapper and exposed through sketch_state_dict / load_sketch_state_dict
(Automodel’s OptimizerState saves it under the "sketch" key). Construct it before
the native optimizer’s first step: target parameters are removed before it can allocate
full-size moment buffers.
Number of completed optimizer updates, stored in param_groups[0] for checkpoints.
Create an fp32 master copy; a DTensor with the weight’s layout under FSDP so DCP shards it.
Restore state produced by sketch_state_dict.
Parameters:
Mapping with the same keys as sketch_state_dict.
Raises:
ValueError: If a key is missing or unexpected, or a shape differs.
Load native state, group settings and update_count from state_dict.
The native half is handed to native.load_state_dict (torch.optim.Optimizer
casts loaded state to the parameter dtype; TE FusedAdam restores fp32 there).
Sketch-target state is not read here; use load_sketch_state_dict.
Parameters:
A dict produced by state_dict.
Raises:
ValueError: If the saved sketch settings differ from the configured ones, or the saved group layout differs from the current one.
Redraw every projection on its device from (seed, update_count, layer name).
Rebase fp32 masters after loading model weights without optimizer state.
Live tensors holding the state of sketch-target parameters.
Returns: dict[str, Tensor]
row_second_moment (the flat [rows, 1] fp32 buffer, identical on every rank, only
Pack param_groups[0] (settings, update_count) with the native optimizer’s own state dict.
The native half comes from native.state_dict() so an optimizer with its own
serialization (TE FusedAdam unscales low-precision moments to fp32) is honoured;
its parameter indices are shifted past the sketch targets so the layout matches
torch.optim.Optimizer’s packing of all groups in order.
Apply the sketch update to targets and the native step to everything else.
An unready call with every group’s lr equal to 0 is the state-initialization probe
used by torch.distributed.checkpoint: only the native optimizer runs, so its state
is created; sketches, projections and update_count are untouched.
Parameters:
Unsupported; must be None.
Multiplies only the sketch-target update, not its weight decay or the native step.
Returns: float | None
None; closures are unsupported, so there is no loss to return.
Raises:
RuntimeError: Ifsynchronize_and_cliphas not run this step (and this is not the zero-lr probe), or a closure is given.
Average sketches across data-parallel ranks, then clip the implied gradient globally.
After the all-reduce every rank holds the full averaged sketch, so the sketch part of the
norm is computed locally over all rows; only the (sharded) native gradient norm is reduced.
process_group must be the FSDP shard group; HSDP replicate groups are unsupported.
Parameters:
FSDP shard group, or None for the default group / a single process.
Clip the global gradient norm to this value, or None to only measure it.
Returns: Tensor
The pre-clip global norm as a 0-d tensor on the sketch device.
Raises:
RuntimeError: If sketches were already synchronized this step, targets are not FSDP-sharded on more than one rank, or projections differ across ranks.FloatingPointError: If the norm is not finite.
Move native state, the flat row moments and the fp32 masters to device.
Clear gradients of every parameter group and zero the sketch buffer.
Parameters:
Set gradients to None instead of zero, as in torch.optim.Optimizer.
Return the local shard of a DTensor, or the tensor itself.
Canonical parameter FQN of a sketch target (<module>.<param>, no leading dot at the root).
View the local part of a stored tensor as 2-D [rows, width], like local_weight_rows.