nemo_automodel.components.optim.gradient_sketch.sketch
nemo_automodel.components.optim.gradient_sketch.sketch
Sketch targets, projections, and the hooks that accumulate projected gradients.
Module Contents
Classes
Functions
Data
API
Validated settings shared by target selection, hooks, and the optimizer.
RowAdam second-moment decay.
"rademacher" (±1/√r) or "gaussian" (N(0, 1/r)).
RowAdam denominator floor.
Freeze every parameter that is not a sketch target.
"row_adam" or "sgd" rule applied to the sketch.
Sketch rank r (columns of S, rows of A).
Redraw A after every update; False keeps the update-0 draw.
Base seed; each layer and update derives its own seed from it.
Last name segments of nn.Linear modules to sketch.
Names of 3-D fused expert parameters to sketch.
Restrict selection to modules under these prefixes.
Plain-value form stored in optimizer param_groups for checkpoints.
Per-weight sketch; model parameter names and checkpoint keys stay unchanged.
sketch is a [rows, rank] view into one buffer shared by all targets (see
install_sketches), so the optimizer can reduce, scale and zero every sketch
with a single kernel or collective. rows is the product of all but the last
weight dimension, so fused 3-D expert banks are flattened.
Restore the real parameter after the fused expert forward.
Swap the fused expert weight for a view whose gradient is projected.
Project the layer input and register the sketch accumulation for backward.
Return the projection in dtype, reusing one persistent copy.
Mirror a refreshed projection into the low-precision copy.
Bases: Function
Keep projected activations in autograd-managed storage, freed after backward.
The output is returned without a copy, so outputs of sketched linear layers must not be modified in place; a violation fails loudly at forward time.
Bases: Function
Project a fused expert gradient before it reaches the stored parameter.
Unlike the linear hook, this path materializes one transient dense gradient and keeps the backend’s expert dispatch and matrix multiplication unchanged.
Overwrite out with a fresh [rank, width] projection drawn on its own device.
Drawing on the device keeps the host out of the per-step refresh: no CPU random stream, no temporaries, no host-to-device copy. Every data-parallel rank draws from the same seed on identical hardware and therefore holds the same matrix; the optimizer verifies that once before the first update.
Parameters:
Preallocated [rank, width] tensor to overwrite in place.
Seed for the device-side generator.
"rademacher" or "gaussian".
Returns: Tensor
The same tensor out, now filled.
Raises:
ValueError: If a dimension is non-positive or the distribution is unknown.
Install sketch hooks on every target and allocate the shared sketch buffer.
Linear targets accumulate sketches directly in backward. Fused experts keep their
forward implementation and project the transient backward gradient. Target weights
get requires_grad=False so autograd never allocates their dense gradient.
Outputs of sketched linear layers must not be modified in place; a violation fails loudly at forward time.
Return the local 2-D weight rows and their global row offset.
Plain tensors are returned whole (3-D expert banks flattened to 2-D). FSDP
row-sharded DTensors return the local shard and its offset; any sharding on
a non-leading dimension (tensor parallelism) is rejected.
Parameters:
A plain or DTensor weight of 2 or more dimensions.
Returns: tuple[Tensor, int]
A (rows, offset) pair: the local 2-D rows and the global index of their first row.
Raises:
ValueError: If the weight is sharded on a dimension other than the leading one.
Allocate a [rank, width] projection and fill it on device.
Parameters:
Number of rows r of the projection.
Number of columns, the weight’s input dimension.
Seed for the device-side generator.
Device to allocate and draw on.
Element type of the projection.
"rademacher" or "gaussian".
Returns: Tensor
The filled projection tensor.
Raises:
ValueError: Ifrankorwidthis non-positive.
Generate a stable per-layer seed independently of module wrappers.
Parameters:
Run-level seed from SketchSettings.seed.
Optimizer update index; changes the seed on every refresh.
Fully qualified module name; FSDP and checkpoint wrapper segments are ignored.
Returns: int
A non-negative 62-bit seed.
Return (name, module, parameter_name, is_fused_expert) for every sketch target.
Return the shared buffer that every SketchState.sketch is a view of.