Skip to navigation

nemo_automodel.components.optim.gradient_sketch.optimizer

View as MarkdownOpen in Claude

Optimize gradient sketches while delegating ordinary parameters to a native optimizer.

Module Contents

Classes

NameDescription
GradientSketchOptimizerRowAdam or SGD on gradient sketches, with a native optimizer for everything else.

Functions

NameDescription
_localReturn the local shard of a DTensor, or the tensor itself.
_parameter_fqnCanonical parameter FQN of a sketch target (<module>.<param>, no leading dot at the root).
_rowsView the local part of a stored tensor as 2-D [rows, width], like local_weight_rows.

Data

_COUNT_KEY

_SETTINGS_KEY

API

class nemo_automodel.components.optim.gradient_sketch.optimizer.GradientSketchOptimizer(
model: torch.nn.Module,
native: torch.optim.Optimizer,
)

Bases: Optimizer

RowAdam or SGD on gradient sketches, with a native optimizer for everything else.

param_groups[0] holds the sketch targets (plus update_count and the settings); the remaining groups are the native optimizer’s own group objects (none when every non-target parameter is frozen), and state is shared with it. state holds native-parameter state only, so it is empty until the first native step and torch.distributed.checkpoint initializes it for any native optimizer. Sketch-target state (flat RowAdam row moments, fp32 masters of low-precision targets) is kept on the wrapper and exposed through sketch_state_dict / load_sketch_state_dict (Automodel’s OptimizerState saves it under the "sketch" key). Construct it before the native optimizer’s first step: target parameters are removed before it can allocate full-size moment buffers.

anchor_handle
layers
list[SketchState] = install_sketches(model, settings)
local_rows
list[int]
masters
list[Tensor | None]
row_second_moment
Tensor | None = None
sketches
Tensor = sketch_buffer(self.layers)
update_count
int

Number of completed optimizer updates, stored in param_groups[0] for checkpoints.

nemo_automodel.components.optim.gradient_sketch.optimizer.GradientSketchOptimizer._anchor(
module: torch.nn.Module,
inputs: tuple[torch.Tensor, ...],
output: torch.Tensor
) -> torch.Tensor
staticmethod
nemo_automodel.components.optim.gradient_sketch.optimizer.GradientSketchOptimizer._check_sharded_targets() -> None
nemo_automodel.components.optim.gradient_sketch.optimizer.GradientSketchOptimizer._new_master(
weight: torch.Tensor
) -> torch.Tensor
staticmethod

Create an fp32 master copy; a DTensor with the weight’s layout under FSDP so DCP shards it.

nemo_automodel.components.optim.gradient_sketch.optimizer.GradientSketchOptimizer._verify_shared_projections(
process_group: typing.Any
) -> None
nemo_automodel.components.optim.gradient_sketch.optimizer.GradientSketchOptimizer.load_sketch_state_dict(
sketch_state: dict[str, torch.Tensor]
) -> None

Restore state produced by sketch_state_dict.

Parameters:

sketch_state
dict[str, Tensor]

Mapping with the same keys as sketch_state_dict.

Raises:

  • ValueError: If a key is missing or unexpected, or a shape differs.
nemo_automodel.components.optim.gradient_sketch.optimizer.GradientSketchOptimizer.load_state_dict(
state_dict: dict[str, typing.Any]
) -> None

Load native state, group settings and update_count from state_dict.

The native half is handed to native.load_state_dict (torch.optim.Optimizer casts loaded state to the parameter dtype; TE FusedAdam restores fp32 there). Sketch-target state is not read here; use load_sketch_state_dict.

Parameters:

state_dict
dict[str, Any]

A dict produced by state_dict.

Raises:

  • ValueError: If the saved sketch settings differ from the configured ones, or the saved group layout differs from the current one.
nemo_automodel.components.optim.gradient_sketch.optimizer.GradientSketchOptimizer.refresh_projection() -> None

Redraw every projection on its device from (seed, update_count, layer name).

nemo_automodel.components.optim.gradient_sketch.optimizer.GradientSketchOptimizer.reset_master_weights() -> None

Rebase fp32 masters after loading model weights without optimizer state.

nemo_automodel.components.optim.gradient_sketch.optimizer.GradientSketchOptimizer.sketch_state_dict() -> dict[str, torch.Tensor]

Live tensors holding the state of sketch-target parameters.

Returns: dict[str, Tensor]

row_second_moment (the flat [rows, 1] fp32 buffer, identical on every rank, only

nemo_automodel.components.optim.gradient_sketch.optimizer.GradientSketchOptimizer.state_dict() -> dict[str, typing.Any]

Pack param_groups[0] (settings, update_count) with the native optimizer’s own state dict.

The native half comes from native.state_dict() so an optimizer with its own serialization (TE FusedAdam unscales low-precision moments to fp32) is honoured; its parameter indices are shifted past the sketch targets so the layout matches torch.optim.Optimizer’s packing of all groups in order.

nemo_automodel.components.optim.gradient_sketch.optimizer.GradientSketchOptimizer.step(
closure: typing.Callable[[], float] | None = None,
scale: float = 1.0
) -> float | None

Apply the sketch update to targets and the native step to everything else.

An unready call with every group’s lr equal to 0 is the state-initialization probe used by torch.distributed.checkpoint: only the native optimizer runs, so its state is created; sketches, projections and update_count are untouched.

Parameters:

closure
Callable[[], float] | NoneDefaults to None

Unsupported; must be None.

scale
floatDefaults to 1.0

Multiplies only the sketch-target update, not its weight decay or the native step.

Returns: float | None

None; closures are unsupported, so there is no loss to return.

Raises:

  • RuntimeError: If synchronize_and_clip has not run this step (and this is not the zero-lr probe), or a closure is given.
nemo_automodel.components.optim.gradient_sketch.optimizer.GradientSketchOptimizer.synchronize_and_clip(
process_group: typing.Any,
max_norm: float | None
) -> torch.Tensor

Average sketches across data-parallel ranks, then clip the implied gradient globally.

After the all-reduce every rank holds the full averaged sketch, so the sketch part of the norm is computed locally over all rows; only the (sharded) native gradient norm is reduced. process_group must be the FSDP shard group; HSDP replicate groups are unsupported.

Parameters:

process_group
Any

FSDP shard group, or None for the default group / a single process.

max_norm
float | None

Clip the global gradient norm to this value, or None to only measure it.

Returns: Tensor

The pre-clip global norm as a 0-d tensor on the sketch device.

Raises:

  • RuntimeError: If sketches were already synchronized this step, targets are not FSDP-sharded on more than one rank, or projections differ across ranks.
  • FloatingPointError: If the norm is not finite.
nemo_automodel.components.optim.gradient_sketch.optimizer.GradientSketchOptimizer.to(
device: torch.device | str
) -> 'GradientSketchOptimizer'

Move native state, the flat row moments and the fp32 masters to device.

nemo_automodel.components.optim.gradient_sketch.optimizer.GradientSketchOptimizer.zero_grad(
set_to_none: bool = True
) -> None

Clear gradients of every parameter group and zero the sketch buffer.

Parameters:

set_to_none
boolDefaults to True

Set gradients to None instead of zero, as in torch.optim.Optimizer.

nemo_automodel.components.optim.gradient_sketch.optimizer._local(
tensor: torch.Tensor
) -> torch.Tensor

Return the local shard of a DTensor, or the tensor itself.

nemo_automodel.components.optim.gradient_sketch.optimizer._parameter_fqn(
) -> str

Canonical parameter FQN of a sketch target (<module>.<param>, no leading dot at the root).

nemo_automodel.components.optim.gradient_sketch.optimizer._rows(
tensor: torch.Tensor
) -> torch.Tensor

View the local part of a stored tensor as 2-D [rows, width], like local_weight_rows.

nemo_automodel.components.optim.gradient_sketch.optimizer._COUNT_KEY = 'update_count'
nemo_automodel.components.optim.gradient_sketch.optimizer._SETTINGS_KEY = 'sketch_settings'