Skip to navigation

nemo_automodel.components.optim.gradient_sketch.sketch

View as MarkdownOpen in Claude

Sketch targets, projections, and the hooks that accumulate projected gradients.

Module Contents

Classes

NameDescription
SketchSettingsValidated settings shared by target selection, hooks, and the optimizer.
SketchStatePer-weight sketch; model parameter names and checkpoint keys stay unchanged.
_AccumulateSketchKeep projected activations in autograd-managed storage, freed after backward.
_ProjectWeightGradientProject a fused expert gradient before it reaches the stored parameter.

Functions

NameDescription
fill_projection_Overwrite out with a fresh [rank, width] projection drawn on its own device.
install_sketchesInstall sketch hooks on every target and allocate the shared sketch buffer.
local_weight_rowsReturn the local 2-D weight rows and their global row offset.
make_projectionAllocate a [rank, width] projection and fill it on device.
projection_seedGenerate a stable per-layer seed independently of module wrappers.
select_sketch_targetsReturn (name, module, parameter_name, is_fused_expert) for every sketch target.
sketch_bufferReturn the shared buffer that every SketchState.sketch is a view of.

Data

_DEFAULT_TARGET_MODULES

_DISTRIBUTIONS

_RULES

API

class nemo_automodel.components.optim.gradient_sketch.sketch.SketchSettings(
rank: int = 256,
seed: int = 42,
distribution: str = 'rademacher',
refresh: bool = True,
optimizer: str = 'row_adam',
beta2: float = 0.95,
epsilon: float = 1e-08,
target_modules: tuple[str, ...] = _DEFAULT_TARGET_MODULES,
target_parameters: tuple[str, ...] = (),
target_prefixes: tuple[str, ...] = (),
freeze_non_target: bool = False
)
Dataclass

Validated settings shared by target selection, hooks, and the optimizer.

beta2
float = 0.95

RowAdam second-moment decay.

distribution
str = 'rademacher'

"rademacher" (±1/√r) or "gaussian" (N(0, 1/r)).

epsilon
float = 1e-08

RowAdam denominator floor.

freeze_non_target
bool = False

Freeze every parameter that is not a sketch target.

optimizer
str = 'row_adam'

"row_adam" or "sgd" rule applied to the sketch.

rank
int = 256

Sketch rank r (columns of S, rows of A).

refresh
bool = True

Redraw A after every update; False keeps the update-0 draw.

seed
int = 42

Base seed; each layer and update derives its own seed from it.

target_modules
tuple[str, ...] = _DEFAULT_TARGET_MODULES

Last name segments of nn.Linear modules to sketch.

target_parameters
tuple[str, ...] = ()

Names of 3-D fused expert parameters to sketch.

target_prefixes
tuple[str, ...] = ()

Restrict selection to modules under these prefixes.

nemo_automodel.components.optim.gradient_sketch.sketch.SketchSettings.__post_init__() -> None
nemo_automodel.components.optim.gradient_sketch.sketch.SketchSettings.to_dict() -> dict[str, typing.Any]

Plain-value form stored in optimizer param_groups for checkpoints.

class nemo_automodel.components.optim.gradient_sketch.sketch.SketchState(
name: str,
module: torch.nn.Module,
weight_name: str,
projection: torch.Tensor,
sketch: torch.Tensor,
handles: list[torch.utils.hooks.RemovableHandle] = list(),
swapped_weight: torch.Tensor | None = None,
projection_cast: torch.Tensor | None = None
)
Dataclass

Per-weight sketch; model parameter names and checkpoint keys stay unchanged.

sketch is a [rows, rank] view into one buffer shared by all targets (see install_sketches), so the optimizer can reduce, scale and zero every sketch with a single kernel or collective. rows is the product of all but the last weight dimension, so fused 3-D expert banks are flattened.

handles
list[RemovableHandle] = field(default_factory=list)
module
Module
name
str
projection
Tensor
projection_cast
Tensor | None = None
sketch
Tensor
swapped_weight
Tensor | None = None
weight
Parameter
weight_name
str
nemo_automodel.components.optim.gradient_sketch.sketch.SketchState.expert_post_hook(
module: torch.nn.Module,
inputs: tuple[torch.Tensor, ...],
output: typing.Any
) -> None

Restore the real parameter after the fused expert forward.

nemo_automodel.components.optim.gradient_sketch.sketch.SketchState.expert_pre_hook(
module: torch.nn.Module,
inputs: tuple[torch.Tensor, ...]
) -> None

Swap the fused expert weight for a view whose gradient is projected.

nemo_automodel.components.optim.gradient_sketch.sketch.SketchState.forward_hook(
module: torch.nn.Module,
inputs: tuple[torch.Tensor, ...],
output: torch.Tensor
) -> torch.Tensor | None

Project the layer input and register the sketch accumulation for backward.

nemo_automodel.components.optim.gradient_sketch.sketch.SketchState.projection_as(
dtype: torch.dtype
) -> torch.Tensor

Return the projection in dtype, reusing one persistent copy.

nemo_automodel.components.optim.gradient_sketch.sketch.SketchState.refresh_cast() -> None

Mirror a refreshed projection into the low-precision copy.

class nemo_automodel.components.optim.gradient_sketch.sketch._AccumulateSketch()

Bases: Function

Keep projected activations in autograd-managed storage, freed after backward.

The output is returned without a copy, so outputs of sketched linear layers must not be modified in place; a violation fails loudly at forward time.

nemo_automodel.components.optim.gradient_sketch.sketch._AccumulateSketch.backward(
ctx: typing.Any,
grad_outputs: torch.Tensor = ()
) -> tuple[torch.Tensor, None, None]
staticmethod
nemo_automodel.components.optim.gradient_sketch.sketch._AccumulateSketch.forward(
ctx: typing.Any,
output: torch.Tensor,
projected: torch.Tensor,
sketch: torch.Tensor
) -> torch.Tensor
staticmethod
class nemo_automodel.components.optim.gradient_sketch.sketch._ProjectWeightGradient()

Bases: Function

Project a fused expert gradient before it reaches the stored parameter.

Unlike the linear hook, this path materializes one transient dense gradient and keeps the backend’s expert dispatch and matrix multiplication unchanged.

nemo_automodel.components.optim.gradient_sketch.sketch._ProjectWeightGradient.backward(
ctx: typing.Any,
grad_outputs: torch.Tensor = ()
) -> tuple[None, None, None]
staticmethod
nemo_automodel.components.optim.gradient_sketch.sketch._ProjectWeightGradient.forward(
ctx: typing.Any,
weight: torch.Tensor,
state: 'SketchState',
row_offset: int
) -> torch.Tensor
staticmethod
nemo_automodel.components.optim.gradient_sketch.sketch.fill_projection_(
out: torch.Tensor,
seed: int,
distribution: str
) -> torch.Tensor

Overwrite out with a fresh [rank, width] projection drawn on its own device.

Drawing on the device keeps the host out of the per-step refresh: no CPU random stream, no temporaries, no host-to-device copy. Every data-parallel rank draws from the same seed on identical hardware and therefore holds the same matrix; the optimizer verifies that once before the first update.

Parameters:

out
Tensor

Preallocated [rank, width] tensor to overwrite in place.

seed
int

Seed for the device-side generator.

distribution
str

"rademacher" or "gaussian".

Returns: Tensor

The same tensor out, now filled.

Raises:

  • ValueError: If a dimension is non-positive or the distribution is unknown.
nemo_automodel.components.optim.gradient_sketch.sketch.install_sketches(
model: torch.nn.Module,

Install sketch hooks on every target and allocate the shared sketch buffer.

Linear targets accumulate sketches directly in backward. Fused experts keep their forward implementation and project the transient backward gradient. Target weights get requires_grad=False so autograd never allocates their dense gradient.

Outputs of sketched linear layers must not be modified in place; a violation fails loudly at forward time.

nemo_automodel.components.optim.gradient_sketch.sketch.local_weight_rows(
weight: torch.Tensor
) -> tuple[torch.Tensor, int]

Return the local 2-D weight rows and their global row offset.

Plain tensors are returned whole (3-D expert banks flattened to 2-D). FSDP row-sharded DTensors return the local shard and its offset; any sharding on a non-leading dimension (tensor parallelism) is rejected.

Parameters:

weight
Tensor

A plain or DTensor weight of 2 or more dimensions.

Returns: tuple[Tensor, int]

A (rows, offset) pair: the local 2-D rows and the global index of their first row.

Raises:

  • ValueError: If the weight is sharded on a dimension other than the leading one.
nemo_automodel.components.optim.gradient_sketch.sketch.make_projection(
rank: int,
width: int,
seed: int,
device: torch.device | str,
dtype: torch.dtype,
distribution: str
) -> torch.Tensor

Allocate a [rank, width] projection and fill it on device.

Parameters:

rank
int

Number of rows r of the projection.

width
int

Number of columns, the weight’s input dimension.

seed
int

Seed for the device-side generator.

device
torch.device | str

Device to allocate and draw on.

dtype
torch.dtype

Element type of the projection.

distribution
str

"rademacher" or "gaussian".

Returns: Tensor

The filled projection tensor.

Raises:

  • ValueError: If rank or width is non-positive.
nemo_automodel.components.optim.gradient_sketch.sketch.projection_seed(
base_seed: int,
update: int,
name: str
) -> int

Generate a stable per-layer seed independently of module wrappers.

Parameters:

base_seed
int

Run-level seed from SketchSettings.seed.

update
int

Optimizer update index; changes the seed on every refresh.

name
str

Fully qualified module name; FSDP and checkpoint wrapper segments are ignored.

Returns: int

A non-negative 62-bit seed.

nemo_automodel.components.optim.gradient_sketch.sketch.select_sketch_targets(
model: torch.nn.Module,
) -> list[tuple[str, torch.nn.Module, str, bool]]

Return (name, module, parameter_name, is_fused_expert) for every sketch target.

nemo_automodel.components.optim.gradient_sketch.sketch.sketch_buffer(
) -> torch.Tensor

Return the shared buffer that every SketchState.sketch is a view of.

nemo_automodel.components.optim.gradient_sketch.sketch._DEFAULT_TARGET_MODULES = ('q_proj', 'k_proj', 'v_proj', 'o_proj', 'gate_proj', 'up_proj', 'down_proj')
nemo_automodel.components.optim.gradient_sketch.sketch._DISTRIBUTIONS = ('rademacher', 'gaussian')
nemo_automodel.components.optim.gradient_sketch.sketch._RULES = ('row_adam', 'sgd')