Skip to navigation

nemo_automodel.components.optim.gradient_sketch

View as MarkdownOpen in Claude

Gradient sketching: rank-r projected gradients with a compact row-wise optimizer.

Implements the optimizer side of LoGRA (Zhang et al., “Scaling LLM reinforcement learning with low-rank gradient sketches”, arXiv:2610.06647). Selected linear weights never form a dense gradient: backward accumulates S = Hᵀ(XAᵀ) into a shared buffer, RowAdam keeps one second moment per output row, and the update ΔW = -lr · D · A is merged into the full weight each step.

Submodules

Package Contents

Classes

NameDescription
GradientSketchOptimizerRowAdam or SGD on gradient sketches, with a native optimizer for everything else.
SketchSettingsValidated settings shared by target selection, hooks, and the optimizer.
SketchStatePer-weight sketch; model parameter names and checkpoint keys stay unchanged.

Functions

NameDescription
fill_projection_Overwrite out with a fresh [rank, width] projection drawn on its own device.
install_sketchesInstall sketch hooks on every target and allocate the shared sketch buffer.
local_weight_rowsReturn the local 2-D weight rows and their global row offset.
make_projectionAllocate a [rank, width] projection and fill it on device.
projection_seedGenerate a stable per-layer seed independently of module wrappers.
row_adam_directionNormalize each sketch row by its bias-corrected running RMS.
select_sketch_targetsReturn (name, module, parameter_name, is_fused_expert) for every sketch target.
sketch_bufferReturn the shared buffer that every SketchState.sketch is a view of.

API

class nemo_automodel.components.optim.gradient_sketch.optimizer.GradientSketchOptimizer(
model: torch.nn.Module,
native: torch.optim.Optimizer,
)

Bases: Optimizer

RowAdam or SGD on gradient sketches, with a native optimizer for everything else.

param_groups[0] holds the sketch targets (plus update_count and the settings); the remaining groups are the native optimizer’s own group objects (none when every non-target parameter is frozen), and state is shared with it. state holds native-parameter state only, so it is empty until the first native step and torch.distributed.checkpoint initializes it for any native optimizer. Sketch-target state (flat RowAdam row moments, fp32 masters of low-precision targets) is kept on the wrapper and exposed through sketch_state_dict / load_sketch_state_dict (Automodel’s OptimizerState saves it under the "sketch" key). Construct it before the native optimizer’s first step: target parameters are removed before it can allocate full-size moment buffers.

anchor_handle
layers
list[SketchState] = install_sketches(model, settings)
local_rows
list[int]
masters
list[Tensor | None]
row_second_moment
Tensor | None = None
sketches
Tensor = sketch_buffer(self.layers)
update_count
int

Number of completed optimizer updates, stored in param_groups[0] for checkpoints.

nemo_automodel.components.optim.gradient_sketch.optimizer.GradientSketchOptimizer._anchor(
module: torch.nn.Module,
inputs: tuple[torch.Tensor, ...],
output: torch.Tensor
) -> torch.Tensor
staticmethod
nemo_automodel.components.optim.gradient_sketch.optimizer.GradientSketchOptimizer._check_sharded_targets() -> None
nemo_automodel.components.optim.gradient_sketch.optimizer.GradientSketchOptimizer._new_master(
weight: torch.Tensor
) -> torch.Tensor
staticmethod

Create an fp32 master copy; a DTensor with the weight’s layout under FSDP so DCP shards it.

nemo_automodel.components.optim.gradient_sketch.optimizer.GradientSketchOptimizer._verify_shared_projections(
process_group: typing.Any
) -> None
nemo_automodel.components.optim.gradient_sketch.optimizer.GradientSketchOptimizer.load_sketch_state_dict(
sketch_state: dict[str, torch.Tensor]
) -> None

Restore state produced by sketch_state_dict.

Parameters:

sketch_state
dict[str, Tensor]

Mapping with the same keys as sketch_state_dict.

Raises:

  • ValueError: If a key is missing or unexpected, or a shape differs.
nemo_automodel.components.optim.gradient_sketch.optimizer.GradientSketchOptimizer.load_state_dict(
state_dict: dict[str, typing.Any]
) -> None

Load native state, group settings and update_count from state_dict.

The native half is handed to native.load_state_dict (torch.optim.Optimizer casts loaded state to the parameter dtype; TE FusedAdam restores fp32 there). Sketch-target state is not read here; use load_sketch_state_dict.

Parameters:

state_dict
dict[str, Any]

A dict produced by state_dict.

Raises:

  • ValueError: If the saved sketch settings differ from the configured ones, or the saved group layout differs from the current one.
nemo_automodel.components.optim.gradient_sketch.optimizer.GradientSketchOptimizer.refresh_projection() -> None

Redraw every projection on its device from (seed, update_count, layer name).

nemo_automodel.components.optim.gradient_sketch.optimizer.GradientSketchOptimizer.reset_master_weights() -> None

Rebase fp32 masters after loading model weights without optimizer state.

nemo_automodel.components.optim.gradient_sketch.optimizer.GradientSketchOptimizer.sketch_state_dict() -> dict[str, torch.Tensor]

Live tensors holding the state of sketch-target parameters.

Returns: dict[str, Tensor]

row_second_moment (the flat [rows, 1] fp32 buffer, identical on every rank, only

nemo_automodel.components.optim.gradient_sketch.optimizer.GradientSketchOptimizer.state_dict() -> dict[str, typing.Any]

Pack param_groups[0] (settings, update_count) with the native optimizer’s own state dict.

The native half comes from native.state_dict() so an optimizer with its own serialization (TE FusedAdam unscales low-precision moments to fp32) is honoured; its parameter indices are shifted past the sketch targets so the layout matches torch.optim.Optimizer’s packing of all groups in order.

nemo_automodel.components.optim.gradient_sketch.optimizer.GradientSketchOptimizer.step(
closure: typing.Callable[[], float] | None = None,
scale: float = 1.0
) -> float | None

Apply the sketch update to targets and the native step to everything else.

An unready call with every group’s lr equal to 0 is the state-initialization probe used by torch.distributed.checkpoint: only the native optimizer runs, so its state is created; sketches, projections and update_count are untouched.

Parameters:

closure
Callable[[], float] | NoneDefaults to None

Unsupported; must be None.

scale
floatDefaults to 1.0

Multiplies only the sketch-target update, not its weight decay or the native step.

Returns: float | None

None; closures are unsupported, so there is no loss to return.

Raises:

  • RuntimeError: If synchronize_and_clip has not run this step (and this is not the zero-lr probe), or a closure is given.
nemo_automodel.components.optim.gradient_sketch.optimizer.GradientSketchOptimizer.synchronize_and_clip(
process_group: typing.Any,
max_norm: float | None
) -> torch.Tensor

Average sketches across data-parallel ranks, then clip the implied gradient globally.

After the all-reduce every rank holds the full averaged sketch, so the sketch part of the norm is computed locally over all rows; only the (sharded) native gradient norm is reduced. process_group must be the FSDP shard group; HSDP replicate groups are unsupported.

Parameters:

process_group
Any

FSDP shard group, or None for the default group / a single process.

max_norm
float | None

Clip the global gradient norm to this value, or None to only measure it.

Returns: Tensor

The pre-clip global norm as a 0-d tensor on the sketch device.

Raises:

  • RuntimeError: If sketches were already synchronized this step, targets are not FSDP-sharded on more than one rank, or projections differ across ranks.
  • FloatingPointError: If the norm is not finite.
nemo_automodel.components.optim.gradient_sketch.optimizer.GradientSketchOptimizer.to(
device: torch.device | str
) -> 'GradientSketchOptimizer'

Move native state, the flat row moments and the fp32 masters to device.

nemo_automodel.components.optim.gradient_sketch.optimizer.GradientSketchOptimizer.zero_grad(
set_to_none: bool = True
) -> None

Clear gradients of every parameter group and zero the sketch buffer.

Parameters:

set_to_none
boolDefaults to True

Set gradients to None instead of zero, as in torch.optim.Optimizer.

class nemo_automodel.components.optim.gradient_sketch.sketch.SketchSettings(
rank: int = 256,
seed: int = 42,
distribution: str = 'rademacher',
refresh: bool = True,
optimizer: str = 'row_adam',
beta2: float = 0.95,
epsilon: float = 1e-08,
target_modules: tuple[str, ...] = _DEFAULT_TARGET_MODULES,
target_parameters: tuple[str, ...] = (),
target_prefixes: tuple[str, ...] = (),
freeze_non_target: bool = False
)
Dataclass

Validated settings shared by target selection, hooks, and the optimizer.

beta2
float = 0.95

RowAdam second-moment decay.

distribution
str = 'rademacher'

"rademacher" (±1/√r) or "gaussian" (N(0, 1/r)).

epsilon
float = 1e-08

RowAdam denominator floor.

freeze_non_target
bool = False

Freeze every parameter that is not a sketch target.

optimizer
str = 'row_adam'

"row_adam" or "sgd" rule applied to the sketch.

rank
int = 256

Sketch rank r (columns of S, rows of A).

refresh
bool = True

Redraw A after every update; False keeps the update-0 draw.

seed
int = 42

Base seed; each layer and update derives its own seed from it.

target_modules
tuple[str, ...] = _DEFAULT_TARGET_MODULES

Last name segments of nn.Linear modules to sketch.

target_parameters
tuple[str, ...] = ()

Names of 3-D fused expert parameters to sketch.

target_prefixes
tuple[str, ...] = ()

Restrict selection to modules under these prefixes.

nemo_automodel.components.optim.gradient_sketch.sketch.SketchSettings.__post_init__() -> None
nemo_automodel.components.optim.gradient_sketch.sketch.SketchSettings.to_dict() -> dict[str, typing.Any]

Plain-value form stored in optimizer param_groups for checkpoints.

class nemo_automodel.components.optim.gradient_sketch.sketch.SketchState(
name: str,
module: torch.nn.Module,
weight_name: str,
projection: torch.Tensor,
sketch: torch.Tensor,
handles: list[torch.utils.hooks.RemovableHandle] = list(),
swapped_weight: torch.Tensor | None = None,
projection_cast: torch.Tensor | None = None
)
Dataclass

Per-weight sketch; model parameter names and checkpoint keys stay unchanged.

sketch is a [rows, rank] view into one buffer shared by all targets (see install_sketches), so the optimizer can reduce, scale and zero every sketch with a single kernel or collective. rows is the product of all but the last weight dimension, so fused 3-D expert banks are flattened.

handles
list[RemovableHandle] = field(default_factory=list)
module
Module
name
str
projection
Tensor
projection_cast
Tensor | None = None
sketch
Tensor
swapped_weight
Tensor | None = None
weight
Parameter
weight_name
str
nemo_automodel.components.optim.gradient_sketch.sketch.SketchState.expert_post_hook(
module: torch.nn.Module,
inputs: tuple[torch.Tensor, ...],
output: typing.Any
) -> None

Restore the real parameter after the fused expert forward.

nemo_automodel.components.optim.gradient_sketch.sketch.SketchState.expert_pre_hook(
module: torch.nn.Module,
inputs: tuple[torch.Tensor, ...]
) -> None

Swap the fused expert weight for a view whose gradient is projected.

nemo_automodel.components.optim.gradient_sketch.sketch.SketchState.forward_hook(
module: torch.nn.Module,
inputs: tuple[torch.Tensor, ...],
output: torch.Tensor
) -> torch.Tensor | None

Project the layer input and register the sketch accumulation for backward.

nemo_automodel.components.optim.gradient_sketch.sketch.SketchState.projection_as(
dtype: torch.dtype
) -> torch.Tensor

Return the projection in dtype, reusing one persistent copy.

nemo_automodel.components.optim.gradient_sketch.sketch.SketchState.refresh_cast() -> None

Mirror a refreshed projection into the low-precision copy.

nemo_automodel.components.optim.gradient_sketch.sketch.fill_projection_(
out: torch.Tensor,
seed: int,
distribution: str
) -> torch.Tensor

Overwrite out with a fresh [rank, width] projection drawn on its own device.

Drawing on the device keeps the host out of the per-step refresh: no CPU random stream, no temporaries, no host-to-device copy. Every data-parallel rank draws from the same seed on identical hardware and therefore holds the same matrix; the optimizer verifies that once before the first update.

Parameters:

out
Tensor

Preallocated [rank, width] tensor to overwrite in place.

seed
int

Seed for the device-side generator.

distribution
str

"rademacher" or "gaussian".

Returns: Tensor

The same tensor out, now filled.

Raises:

  • ValueError: If a dimension is non-positive or the distribution is unknown.
nemo_automodel.components.optim.gradient_sketch.sketch.install_sketches(
model: torch.nn.Module,

Install sketch hooks on every target and allocate the shared sketch buffer.

Linear targets accumulate sketches directly in backward. Fused experts keep their forward implementation and project the transient backward gradient. Target weights get requires_grad=False so autograd never allocates their dense gradient.

Outputs of sketched linear layers must not be modified in place; a violation fails loudly at forward time.

nemo_automodel.components.optim.gradient_sketch.sketch.local_weight_rows(
weight: torch.Tensor
) -> tuple[torch.Tensor, int]

Return the local 2-D weight rows and their global row offset.

Plain tensors are returned whole (3-D expert banks flattened to 2-D). FSDP row-sharded DTensors return the local shard and its offset; any sharding on a non-leading dimension (tensor parallelism) is rejected.

Parameters:

weight
Tensor

A plain or DTensor weight of 2 or more dimensions.

Returns: tuple[Tensor, int]

A (rows, offset) pair: the local 2-D rows and the global index of their first row.

Raises:

  • ValueError: If the weight is sharded on a dimension other than the leading one.
nemo_automodel.components.optim.gradient_sketch.sketch.make_projection(
rank: int,
width: int,
seed: int,
device: torch.device | str,
dtype: torch.dtype,
distribution: str
) -> torch.Tensor

Allocate a [rank, width] projection and fill it on device.

Parameters:

rank
int

Number of rows r of the projection.

width
int

Number of columns, the weight’s input dimension.

seed
int

Seed for the device-side generator.

device
torch.device | str

Device to allocate and draw on.

dtype
torch.dtype

Element type of the projection.

distribution
str

"rademacher" or "gaussian".

Returns: Tensor

The filled projection tensor.

Raises:

  • ValueError: If rank or width is non-positive.
nemo_automodel.components.optim.gradient_sketch.sketch.projection_seed(
base_seed: int,
update: int,
name: str
) -> int

Generate a stable per-layer seed independently of module wrappers.

Parameters:

base_seed
int

Run-level seed from SketchSettings.seed.

update
int

Optimizer update index; changes the seed on every refresh.

name
str

Fully qualified module name; FSDP and checkpoint wrapper segments are ignored.

Returns: int

A non-negative 62-bit seed.

nemo_automodel.components.optim.gradient_sketch.row_adam.row_adam_direction(
sketch: torch.Tensor,
second_moment: torch.Tensor,
step: int,
beta2: float,
epsilon: float,
inplace: bool = False
) -> torch.Tensor

Normalize each sketch row by its bias-corrected running RMS.

The state is indexed by the original weight’s output rows, not by the changing projection coordinates. With inplace the sketch is overwritten with the direction, avoiding a full-size temporary when the sketch is no longer needed.

Parameters:

sketch
Tensor

[rows, rank] accumulated sketch.

second_moment
Tensor

[rows, 1] running second moment, updated in place.

step
int

1-based update count used for bias correction.

beta2
float

Second-moment decay in [0, 1).

epsilon
float

Denominator floor.

inplace
boolDefaults to False

Overwrite sketch with the direction.

Returns: Tensor

The normalized direction, same shape as sketch.

nemo_automodel.components.optim.gradient_sketch.sketch.select_sketch_targets(
model: torch.nn.Module,
) -> list[tuple[str, torch.nn.Module, str, bool]]

Return (name, module, parameter_name, is_fused_expert) for every sketch target.

nemo_automodel.components.optim.gradient_sketch.sketch.sketch_buffer(
) -> torch.Tensor

Return the shared buffer that every SketchState.sketch is a view of.