> For clean Markdown of any page, append .md to the page URL. > For a complete documentation index, see https://docs.nvidia.com/nemo/automodel/v0.4/nemo-automodel/nemo_automodel/components/optim/gradient_sketch/llms.txt. # nemo_automodel.components.optim.gradient_sketch Gradient sketching: rank-r projected gradients with a compact row-wise optimizer. Implements the optimizer side of LoGRA (Zhang et al., "Scaling LLM reinforcement learning with low-rank gradient sketches", arXiv:2610.06647). Selected linear weights never form a dense gradient: backward accumulates `S = Hᵀ(XAᵀ)` into a shared buffer, RowAdam keeps one second moment per output row, and the update `ΔW = -lr · D · A` is merged into the full weight each step. ## Submodules * **[`nemo_automodel.components.optim.gradient_sketch.optimizer`](/nemo-automodel/nemo_automodel/components/optim/gradient_sketch/optimizer)** * **[`nemo_automodel.components.optim.gradient_sketch.row_adam`](/nemo-automodel/nemo_automodel/components/optim/gradient_sketch/row_adam)** * **[`nemo_automodel.components.optim.gradient_sketch.sketch`](/nemo-automodel/nemo_automodel/components/optim/gradient_sketch/sketch)** ## Package Contents ### Classes | Name | Description | | --------------------------------------------------------------------------------------------------------------- | --------------------------------------------------------------------------------- | | [`GradientSketchOptimizer`](#nemo_automodel-components-optim-gradient_sketch-optimizer-GradientSketchOptimizer) | RowAdam or SGD on gradient sketches, with a native optimizer for everything else. | | [`SketchSettings`](#nemo_automodel-components-optim-gradient_sketch-sketch-SketchSettings) | Validated settings shared by target selection, hooks, and the optimizer. | | [`SketchState`](#nemo_automodel-components-optim-gradient_sketch-sketch-SketchState) | Per-weight sketch; model parameter names and checkpoint keys stay unchanged. | ### Functions | Name | Description | | -------------------------------------------------------------------------------------------------------- | --------------------------------------------------------------------------------- | | [`fill_projection_`](#nemo_automodel-components-optim-gradient_sketch-sketch-fill_projection_) | Overwrite `out` with a fresh `[rank, width]` projection drawn on its own device. | | [`install_sketches`](#nemo_automodel-components-optim-gradient_sketch-sketch-install_sketches) | Install sketch hooks on every target and allocate the shared sketch buffer. | | [`local_weight_rows`](#nemo_automodel-components-optim-gradient_sketch-sketch-local_weight_rows) | Return the local 2-D weight rows and their global row offset. | | [`make_projection`](#nemo_automodel-components-optim-gradient_sketch-sketch-make_projection) | Allocate a `[rank, width]` projection and fill it on `device`. | | [`projection_seed`](#nemo_automodel-components-optim-gradient_sketch-sketch-projection_seed) | Generate a stable per-layer seed independently of module wrappers. | | [`row_adam_direction`](#nemo_automodel-components-optim-gradient_sketch-row_adam-row_adam_direction) | Normalize each sketch row by its bias-corrected running RMS. | | [`select_sketch_targets`](#nemo_automodel-components-optim-gradient_sketch-sketch-select_sketch_targets) | Return `(name, module, parameter_name, is_fused_expert)` for every sketch target. | | [`sketch_buffer`](#nemo_automodel-components-optim-gradient_sketch-sketch-sketch_buffer) | Return the shared buffer that every `SketchState.sketch` is a view of. | ### API ```python class nemo_automodel.components.optim.gradient_sketch.optimizer.GradientSketchOptimizer( model: torch.nn.Module, native: torch.optim.Optimizer, settings: nemo_automodel.components.optim.gradient_sketch.sketch.SketchSettings ) ``` **Bases:** `Optimizer` RowAdam or SGD on gradient sketches, with a native optimizer for everything else. `param_groups[0]` holds the sketch targets (plus `update_count` and the settings); the remaining groups are the native optimizer's own group objects (none when every non-target parameter is frozen), and `state` is shared with it. `state` holds native-parameter state only, so it is empty until the first native step and `torch.distributed.checkpoint` initializes it for any native optimizer. Sketch-target state (flat RowAdam row moments, fp32 masters of low-precision targets) is kept on the wrapper and exposed through `sketch_state_dict` / `load_sketch_state_dict` (Automodel's `OptimizerState` saves it under the `"sketch"` key). Construct it before the native optimizer's first step: target parameters are removed before it can allocate full-size moment buffers. **`anchor_handle`** --- **`layers`** `list[SketchState] = install_sketches(model, settings)` --- **`local_rows`** `list[int]` --- **`masters`** `list[Tensor | None]` --- **`row_second_moment`** `Tensor | None = None` --- **`sketches`** `Tensor = sketch_buffer(self.layers)` --- **`update_count`** `int` Number of completed optimizer updates, stored in `param_groups[0]` for checkpoints. --- ```python nemo_automodel.components.optim.gradient_sketch.optimizer.GradientSketchOptimizer._anchor( module: torch.nn.Module, inputs: tuple[torch.Tensor, ...], output: torch.Tensor ) -> torch.Tensor ``` staticmethod ```python nemo_automodel.components.optim.gradient_sketch.optimizer.GradientSketchOptimizer._check_sharded_targets() -> None ``` ```python nemo_automodel.components.optim.gradient_sketch.optimizer.GradientSketchOptimizer._new_master( weight: torch.Tensor ) -> torch.Tensor ``` staticmethod Create an fp32 master copy; a `DTensor` with the weight's layout under FSDP so DCP shards it. ```python nemo_automodel.components.optim.gradient_sketch.optimizer.GradientSketchOptimizer._verify_shared_projections( process_group: typing.Any ) -> None ``` ```python nemo_automodel.components.optim.gradient_sketch.optimizer.GradientSketchOptimizer.load_sketch_state_dict( sketch_state: dict[str, torch.Tensor] ) -> None ``` Restore state produced by `sketch_state_dict`. **Parameters:** **`sketch_state`** `dict[str, Tensor]` Mapping with the same keys as `sketch_state_dict`. --- **Raises:** * `ValueError`: If a key is missing or unexpected, or a shape differs. ```python nemo_automodel.components.optim.gradient_sketch.optimizer.GradientSketchOptimizer.load_state_dict( state_dict: dict[str, typing.Any] ) -> None ``` Load native state, group settings and `update_count` from `state_dict`. The native half is handed to `native.load_state_dict` (`torch.optim.Optimizer` casts loaded state to the parameter dtype; TE `FusedAdam` restores fp32 there). Sketch-target state is not read here; use `load_sketch_state_dict`. **Parameters:** **`state_dict`** `dict[str, Any]` A dict produced by `state_dict`. --- **Raises:** * `ValueError`: If the saved sketch settings differ from the configured ones, or the saved group layout differs from the current one. ```python nemo_automodel.components.optim.gradient_sketch.optimizer.GradientSketchOptimizer.refresh_projection() -> None ``` Redraw every projection on its device from `(seed, update_count, layer name)`. ```python nemo_automodel.components.optim.gradient_sketch.optimizer.GradientSketchOptimizer.reset_master_weights() -> None ``` Rebase fp32 masters after loading model weights without optimizer state. ```python nemo_automodel.components.optim.gradient_sketch.optimizer.GradientSketchOptimizer.sketch_state_dict() -> dict[str, torch.Tensor] ``` Live tensors holding the state of sketch-target parameters. **Returns:** `dict[str, Tensor]` `row_second_moment` (the flat `[rows, 1]` fp32 buffer, identical on every rank, only ```python nemo_automodel.components.optim.gradient_sketch.optimizer.GradientSketchOptimizer.state_dict() -> dict[str, typing.Any] ``` Pack `param_groups[0]` (settings, `update_count`) with the native optimizer's own state dict. The native half comes from `native.state_dict()` so an optimizer with its own serialization (TE `FusedAdam` unscales low-precision moments to fp32) is honoured; its parameter indices are shifted past the sketch targets so the layout matches `torch.optim.Optimizer`'s packing of all groups in order. ```python nemo_automodel.components.optim.gradient_sketch.optimizer.GradientSketchOptimizer.step( closure: typing.Callable[[], float] | None = None, scale: float = 1.0 ) -> float | None ``` Apply the sketch update to targets and the native step to everything else. An unready call with every group's `lr` equal to 0 is the state-initialization probe used by `torch.distributed.checkpoint`: only the native optimizer runs, so its state is created; sketches, projections and `update_count` are untouched. **Parameters:** **`closure`** `Callable[[], float] | None` — default: None Unsupported; must be `None`. --- **`scale`** `float` — default: 1.0 Multiplies only the sketch-target update, not its weight decay or the native step. --- **Returns:** `float | None` `None`; closures are unsupported, so there is no loss to return. **Raises:** * `RuntimeError`: If `synchronize_and_clip` has not run this step (and this is not the zero-lr probe), or a closure is given. ```python nemo_automodel.components.optim.gradient_sketch.optimizer.GradientSketchOptimizer.synchronize_and_clip( process_group: typing.Any, max_norm: float | None ) -> torch.Tensor ``` Average sketches across data-parallel ranks, then clip the implied gradient globally. After the all-reduce every rank holds the full averaged sketch, so the sketch part of the norm is computed locally over all rows; only the (sharded) native gradient norm is reduced. `process_group` must be the FSDP shard group; HSDP replicate groups are unsupported. **Parameters:** **`process_group`** `Any` FSDP shard group, or `None` for the default group / a single process. --- **`max_norm`** `float | None` Clip the global gradient norm to this value, or `None` to only measure it. --- **Returns:** `Tensor` The pre-clip global norm as a 0-d tensor on the sketch device. **Raises:** * `RuntimeError`: If sketches were already synchronized this step, targets are not FSDP-sharded on more than one rank, or projections differ across ranks. * `FloatingPointError`: If the norm is not finite. ```python nemo_automodel.components.optim.gradient_sketch.optimizer.GradientSketchOptimizer.to( device: torch.device | str ) -> 'GradientSketchOptimizer' ``` Move native state, the flat row moments and the fp32 masters to `device`. ```python nemo_automodel.components.optim.gradient_sketch.optimizer.GradientSketchOptimizer.zero_grad( set_to_none: bool = True ) -> None ``` Clear gradients of every parameter group and zero the sketch buffer. **Parameters:** **`set_to_none`** `bool` — default: True Set gradients to `None` instead of zero, as in `torch.optim.Optimizer`. --- ```python class nemo_automodel.components.optim.gradient_sketch.sketch.SketchSettings( rank: int = 256, seed: int = 42, distribution: str = 'rademacher', refresh: bool = True, optimizer: str = 'row_adam', beta2: float = 0.95, epsilon: float = 1e-08, target_modules: tuple[str, ...] = _DEFAULT_TARGET_MODULES, target_parameters: tuple[str, ...] = (), target_prefixes: tuple[str, ...] = (), freeze_non_target: bool = False ) ``` Dataclass Validated settings shared by target selection, hooks, and the optimizer. **`beta2`** `float = 0.95` RowAdam second-moment decay. --- **`distribution`** `str = 'rademacher'` `"rademacher"` (±1/√r) or `"gaussian"` (N(0, 1/r)). --- **`epsilon`** `float = 1e-08` RowAdam denominator floor. --- **`freeze_non_target`** `bool = False` Freeze every parameter that is not a sketch target. --- **`optimizer`** `str = 'row_adam'` `"row_adam"` or `"sgd"` rule applied to the sketch. --- **`rank`** `int = 256` Sketch rank `r` (columns of `S`, rows of `A`). --- **`refresh`** `bool = True` Redraw `A` after every update; `False` keeps the update-0 draw. --- **`seed`** `int = 42` Base seed; each layer and update derives its own seed from it. --- **`target_modules`** `tuple[str, ...] = _DEFAULT_TARGET_MODULES` Last name segments of `nn.Linear` modules to sketch. --- **`target_parameters`** `tuple[str, ...] = ()` Names of 3-D fused expert parameters to sketch. --- **`target_prefixes`** `tuple[str, ...] = ()` Restrict selection to modules under these prefixes. --- ```python nemo_automodel.components.optim.gradient_sketch.sketch.SketchSettings.__post_init__() -> None ``` ```python nemo_automodel.components.optim.gradient_sketch.sketch.SketchSettings.to_dict() -> dict[str, typing.Any] ``` Plain-value form stored in optimizer `param_groups` for checkpoints. ```python class nemo_automodel.components.optim.gradient_sketch.sketch.SketchState( name: str, module: torch.nn.Module, weight_name: str, projection: torch.Tensor, sketch: torch.Tensor, handles: list[torch.utils.hooks.RemovableHandle] = list(), swapped_weight: torch.Tensor | None = None, projection_cast: torch.Tensor | None = None ) ``` Dataclass Per-weight sketch; model parameter names and checkpoint keys stay unchanged. `sketch` is a `[rows, rank]` view into one buffer shared by all targets (see `install_sketches`), so the optimizer can reduce, scale and zero every sketch with a single kernel or collective. `rows` is the product of all but the last weight dimension, so fused 3-D expert banks are flattened. **`handles`** `list[RemovableHandle] = field(default_factory=list)` --- **`module`** `Module` --- **`name`** `str` --- **`projection`** `Tensor` --- **`projection_cast`** `Tensor | None = None` --- **`sketch`** `Tensor` --- **`swapped_weight`** `Tensor | None = None` --- **`weight`** `Parameter` --- **`weight_name`** `str` --- ```python nemo_automodel.components.optim.gradient_sketch.sketch.SketchState.expert_post_hook( module: torch.nn.Module, inputs: tuple[torch.Tensor, ...], output: typing.Any ) -> None ``` Restore the real parameter after the fused expert forward. ```python nemo_automodel.components.optim.gradient_sketch.sketch.SketchState.expert_pre_hook( module: torch.nn.Module, inputs: tuple[torch.Tensor, ...] ) -> None ``` Swap the fused expert weight for a view whose gradient is projected. ```python nemo_automodel.components.optim.gradient_sketch.sketch.SketchState.forward_hook( module: torch.nn.Module, inputs: tuple[torch.Tensor, ...], output: torch.Tensor ) -> torch.Tensor | None ``` Project the layer input and register the sketch accumulation for backward. ```python nemo_automodel.components.optim.gradient_sketch.sketch.SketchState.projection_as( dtype: torch.dtype ) -> torch.Tensor ``` Return the projection in `dtype`, reusing one persistent copy. ```python nemo_automodel.components.optim.gradient_sketch.sketch.SketchState.refresh_cast() -> None ``` Mirror a refreshed `projection` into the low-precision copy. ```python nemo_automodel.components.optim.gradient_sketch.sketch.fill_projection_( out: torch.Tensor, seed: int, distribution: str ) -> torch.Tensor ``` Overwrite `out` with a fresh `[rank, width]` projection drawn on its own device. Drawing on the device keeps the host out of the per-step refresh: no CPU random stream, no temporaries, no host-to-device copy. Every data-parallel rank draws from the same seed on identical hardware and therefore holds the same matrix; the optimizer verifies that once before the first update. **Parameters:** **`out`** `Tensor` Preallocated `[rank, width]` tensor to overwrite in place. --- **`seed`** `int` Seed for the device-side generator. --- **`distribution`** `str` `"rademacher"` or `"gaussian"`. --- **Returns:** `Tensor` The same tensor `out`, now filled. **Raises:** * `ValueError`: If a dimension is non-positive or the distribution is unknown. ```python nemo_automodel.components.optim.gradient_sketch.sketch.install_sketches( model: torch.nn.Module, settings: nemo_automodel.components.optim.gradient_sketch.sketch.SketchSettings ) -> list[nemo_automodel.components.optim.gradient_sketch.sketch.SketchState] ``` Install sketch hooks on every target and allocate the shared sketch buffer. Linear targets accumulate sketches directly in backward. Fused experts keep their forward implementation and project the transient backward gradient. Target weights get `requires_grad=False` so autograd never allocates their dense gradient. Outputs of sketched linear layers must not be modified in place; a violation fails loudly at forward time. ```python nemo_automodel.components.optim.gradient_sketch.sketch.local_weight_rows( weight: torch.Tensor ) -> tuple[torch.Tensor, int] ``` Return the local 2-D weight rows and their global row offset. Plain tensors are returned whole (3-D expert banks flattened to 2-D). FSDP row-sharded `DTensor`s return the local shard and its offset; any sharding on a non-leading dimension (tensor parallelism) is rejected. **Parameters:** **`weight`** `Tensor` A plain or `DTensor` weight of 2 or more dimensions. --- **Returns:** `tuple[Tensor, int]` A `(rows, offset)` pair: the local 2-D rows and the global index of their first row. **Raises:** * `ValueError`: If the weight is sharded on a dimension other than the leading one. ```python nemo_automodel.components.optim.gradient_sketch.sketch.make_projection( rank: int, width: int, seed: int, device: torch.device | str, dtype: torch.dtype, distribution: str ) -> torch.Tensor ``` Allocate a `[rank, width]` projection and fill it on `device`. **Parameters:** **`rank`** `int` Number of rows `r` of the projection. --- **`width`** `int` Number of columns, the weight's input dimension. --- **`seed`** `int` Seed for the device-side generator. --- **`device`** `torch.device | str` Device to allocate and draw on. --- **`dtype`** `torch.dtype` Element type of the projection. --- **`distribution`** `str` `"rademacher"` or `"gaussian"`. --- **Returns:** `Tensor` The filled projection tensor. **Raises:** * `ValueError`: If `rank` or `width` is non-positive. ```python nemo_automodel.components.optim.gradient_sketch.sketch.projection_seed( base_seed: int, update: int, name: str ) -> int ``` Generate a stable per-layer seed independently of module wrappers. **Parameters:** **`base_seed`** `int` Run-level seed from `SketchSettings.seed`. --- **`update`** `int` Optimizer update index; changes the seed on every refresh. --- **`name`** `str` Fully qualified module name; FSDP and checkpoint wrapper segments are ignored. --- **Returns:** `int` A non-negative 62-bit seed. ```python nemo_automodel.components.optim.gradient_sketch.row_adam.row_adam_direction( sketch: torch.Tensor, second_moment: torch.Tensor, step: int, beta2: float, epsilon: float, inplace: bool = False ) -> torch.Tensor ``` Normalize each sketch row by its bias-corrected running RMS. The state is indexed by the original weight's output rows, not by the changing projection coordinates. With `inplace` the sketch is overwritten with the direction, avoiding a full-size temporary when the sketch is no longer needed. **Parameters:** **`sketch`** `Tensor` `[rows, rank]` accumulated sketch. --- **`second_moment`** `Tensor` `[rows, 1]` running second moment, updated in place. --- **`step`** `int` 1-based update count used for bias correction. --- **`beta2`** `float` Second-moment decay in `[0, 1)`. --- **`epsilon`** `float` Denominator floor. --- **`inplace`** `bool` — default: False Overwrite `sketch` with the direction. --- **Returns:** `Tensor` The normalized direction, same shape as `sketch`. ```python nemo_automodel.components.optim.gradient_sketch.sketch.select_sketch_targets( model: torch.nn.Module, settings: nemo_automodel.components.optim.gradient_sketch.sketch.SketchSettings ) -> list[tuple[str, torch.nn.Module, str, bool]] ``` Return `(name, module, parameter_name, is_fused_expert)` for every sketch target. ```python nemo_automodel.components.optim.gradient_sketch.sketch.sketch_buffer( states: list[nemo_automodel.components.optim.gradient_sketch.sketch.SketchState] ) -> torch.Tensor ``` Return the shared buffer that every `SketchState.sketch` is a view of. ## Docs - [nemo_automodel.components.optim.gradient_sketch.optimizer](https://docs.nvidia.com/nemo/automodel/v0.4/nemo-automodel/nemo_automodel/components/optim/gradient_sketch/optimizer.md) - [nemo_automodel.components.optim.gradient_sketch.row_adam](https://docs.nvidia.com/nemo/automodel/v0.4/nemo-automodel/nemo_automodel/components/optim/gradient_sketch/row_adam.md) - [nemo_automodel.components.optim.gradient_sketch.sketch](https://docs.nvidia.com/nemo/automodel/v0.4/nemo-automodel/nemo_automodel/components/optim/gradient_sketch/sketch.md)