> This page is for version 0.5.0 · 26.06.
> For other versions, use one of these documentation indexes:
> - Nightly (default): https://docs.nvidia.com/nemo/automodel/nightly/llms.txt
> - Latest: https://docs.nvidia.com/nemo/automodel/latest/llms.txt
> - 0.7.0 · 26.10: https://docs.nvidia.com/nemo/automodel/v0.7/llms.txt
> - 0.5.0 · 26.06: https://docs.nvidia.com/nemo/automodel/v0.5/llms.txt
> - 0.4.0 · 26.04: https://docs.nvidia.com/nemo/automodel/v0.4/llms.txt

> For clean Markdown of any page, append .md to the page URL.
> For a complete documentation index, see https://docs.nvidia.com/nemo/automodel/llms.txt.

# nemo_automodel.components.optim.gradient_sketch.sketch

Sketch targets, projections, and the hooks that accumulate projected gradients.

## Module Contents

### Classes

| Name                                                                                                       | Description                                                                   |
| ---------------------------------------------------------------------------------------------------------- | ----------------------------------------------------------------------------- |
| [`SketchSettings`](#nemo_automodel-components-optim-gradient_sketch-sketch-SketchSettings)                 | Validated settings shared by target selection, hooks, and the optimizer.      |
| [`SketchState`](#nemo_automodel-components-optim-gradient_sketch-sketch-SketchState)                       | Per-weight sketch; model parameter names and checkpoint keys stay unchanged.  |
| [`_AccumulateSketch`](#nemo_automodel-components-optim-gradient_sketch-sketch-_AccumulateSketch)           | Keep projected activations in autograd-managed storage, freed after backward. |
| [`_ProjectWeightGradient`](#nemo_automodel-components-optim-gradient_sketch-sketch-_ProjectWeightGradient) | Project a fused expert gradient before it reaches the stored parameter.       |

### Functions

| Name                                                                                                     | Description                                                                       |
| -------------------------------------------------------------------------------------------------------- | --------------------------------------------------------------------------------- |
| [`fill_projection_`](#nemo_automodel-components-optim-gradient_sketch-sketch-fill_projection_)           | Overwrite `out` with a fresh `[rank, width]` projection drawn on its own device.  |
| [`install_sketches`](#nemo_automodel-components-optim-gradient_sketch-sketch-install_sketches)           | Install sketch hooks on every target and allocate the shared sketch buffer.       |
| [`local_weight_rows`](#nemo_automodel-components-optim-gradient_sketch-sketch-local_weight_rows)         | Return the local 2-D weight rows and their global row offset.                     |
| [`make_projection`](#nemo_automodel-components-optim-gradient_sketch-sketch-make_projection)             | Allocate a `[rank, width]` projection and fill it on `device`.                    |
| [`projection_seed`](#nemo_automodel-components-optim-gradient_sketch-sketch-projection_seed)             | Generate a stable per-layer seed independently of module wrappers.                |
| [`select_sketch_targets`](#nemo_automodel-components-optim-gradient_sketch-sketch-select_sketch_targets) | Return `(name, module, parameter_name, is_fused_expert)` for every sketch target. |
| [`sketch_buffer`](#nemo_automodel-components-optim-gradient_sketch-sketch-sketch_buffer)                 | Return the shared buffer that every `SketchState.sketch` is a view of.            |

### Data

[`_DEFAULT_TARGET_MODULES`](#nemo_automodel-components-optim-gradient_sketch-sketch-_DEFAULT_TARGET_MODULES)

[`_DISTRIBUTIONS`](#nemo_automodel-components-optim-gradient_sketch-sketch-_DISTRIBUTIONS)

[`_RULES`](#nemo_automodel-components-optim-gradient_sketch-sketch-_RULES)

### API

```python
class nemo_automodel.components.optim.gradient_sketch.sketch.SketchSettings(
    rank: int = 256,
    seed: int = 42,
    distribution: str = 'rademacher',
    refresh: bool = True,
    optimizer: str = 'row_adam',
    beta2: float = 0.95,
    epsilon: float = 1e-08,
    target_modules: tuple[str, ...] = _DEFAULT_TARGET_MODULES,
    target_parameters: tuple[str, ...] = (),
    target_prefixes: tuple[str, ...] = (),
    freeze_non_target: bool = False
)
```

Dataclass

Validated settings shared by target selection, hooks, and the optimizer.

**`beta2`** `float = 0.95`

RowAdam second-moment decay.

---

**`distribution`** `str = 'rademacher'`

`"rademacher"` (±1/√r) or `"gaussian"` (N(0, 1/r)).

---

**`epsilon`** `float = 1e-08`

RowAdam denominator floor.

---

**`freeze_non_target`** `bool = False`

Freeze every parameter that is not a sketch target.

---

**`optimizer`** `str = 'row_adam'`

`"row_adam"` or `"sgd"` rule applied to the sketch.

---

**`rank`** `int = 256`

Sketch rank `r` (columns of `S`, rows of `A`).

---

**`refresh`** `bool = True`

Redraw `A` after every update; `False` keeps the update-0 draw.

---

**`seed`** `int = 42`

Base seed; each layer and update derives its own seed from it.

---

**`target_modules`** `tuple[str, ...] = _DEFAULT_TARGET_MODULES`

Last name segments of `nn.Linear` modules to sketch.

---

**`target_parameters`** `tuple[str, ...] = ()`

Names of 3-D fused expert parameters to sketch.

---

**`target_prefixes`** `tuple[str, ...] = ()`

Restrict selection to modules under these prefixes.

---

```python
nemo_automodel.components.optim.gradient_sketch.sketch.SketchSettings.__post_init__() -> None
```

```python
nemo_automodel.components.optim.gradient_sketch.sketch.SketchSettings.to_dict() -> dict[str, typing.Any]
```

Plain-value form stored in optimizer `param_groups` for checkpoints.

```python
class nemo_automodel.components.optim.gradient_sketch.sketch.SketchState(
    name: str,
    module: torch.nn.Module,
    weight_name: str,
    projection: torch.Tensor,
    sketch: torch.Tensor,
    handles: list[torch.utils.hooks.RemovableHandle] = list(),
    swapped_weight: torch.Tensor | None = None,
    projection_cast: torch.Tensor | None = None
)
```

Dataclass

Per-weight sketch; model parameter names and checkpoint keys stay unchanged.

`sketch` is a `[rows, rank]` view into one buffer shared by all targets (see
`install_sketches`), so the optimizer can reduce, scale and zero every sketch
with a single kernel or collective. `rows` is the product of all but the last
weight dimension, so fused 3-D expert banks are flattened.

**`handles`** `list[RemovableHandle] = field(default_factory=list)`

---

**`module`** `Module`

---

**`name`** `str`

---

**`projection`** `Tensor`

---

**`projection_cast`** `Tensor | None = None`

---

**`sketch`** `Tensor`

---

**`swapped_weight`** `Tensor | None = None`

---

**`weight`** `Parameter`

---

**`weight_name`** `str`

---

```python
nemo_automodel.components.optim.gradient_sketch.sketch.SketchState.expert_post_hook(
    module: torch.nn.Module,
    inputs: tuple[torch.Tensor, ...],
    output: typing.Any
) -> None
```

Restore the real parameter after the fused expert forward.

```python
nemo_automodel.components.optim.gradient_sketch.sketch.SketchState.expert_pre_hook(
    module: torch.nn.Module,
    inputs: tuple[torch.Tensor, ...]
) -> None
```

Swap the fused expert weight for a view whose gradient is projected.

```python
nemo_automodel.components.optim.gradient_sketch.sketch.SketchState.forward_hook(
    module: torch.nn.Module,
    inputs: tuple[torch.Tensor, ...],
    output: torch.Tensor
) -> torch.Tensor | None
```

Project the layer input and register the sketch accumulation for backward.

```python
nemo_automodel.components.optim.gradient_sketch.sketch.SketchState.projection_as(
    dtype: torch.dtype
) -> torch.Tensor
```

Return the projection in `dtype`, reusing one persistent copy.

```python
nemo_automodel.components.optim.gradient_sketch.sketch.SketchState.refresh_cast() -> None
```

Mirror a refreshed `projection` into the low-precision copy.

```python
class nemo_automodel.components.optim.gradient_sketch.sketch._AccumulateSketch()
```

**Bases:** `Function`

Keep projected activations in autograd-managed storage, freed after backward.

The output is returned without a copy, so outputs of sketched linear layers must not be modified
in place; a violation fails loudly at forward time.

```python
nemo_automodel.components.optim.gradient_sketch.sketch._AccumulateSketch.backward(
    ctx: typing.Any,
    grad_outputs: torch.Tensor = ()
) -> tuple[torch.Tensor, None, None]
```

staticmethod

```python
nemo_automodel.components.optim.gradient_sketch.sketch._AccumulateSketch.forward(
    ctx: typing.Any,
    output: torch.Tensor,
    projected: torch.Tensor,
    sketch: torch.Tensor
) -> torch.Tensor
```

staticmethod

```python
class nemo_automodel.components.optim.gradient_sketch.sketch._ProjectWeightGradient()
```

**Bases:** `Function`

Project a fused expert gradient before it reaches the stored parameter.

Unlike the linear hook, this path materializes one transient dense gradient and
keeps the backend's expert dispatch and matrix multiplication unchanged.

```python
nemo_automodel.components.optim.gradient_sketch.sketch._ProjectWeightGradient.backward(
    ctx: typing.Any,
    grad_outputs: torch.Tensor = ()
) -> tuple[None, None, None]
```

staticmethod

```python
nemo_automodel.components.optim.gradient_sketch.sketch._ProjectWeightGradient.forward(
    ctx: typing.Any,
    weight: torch.Tensor,
    state: 'SketchState',
    row_offset: int
) -> torch.Tensor
```

staticmethod

```python
nemo_automodel.components.optim.gradient_sketch.sketch.fill_projection_(
    out: torch.Tensor,
    seed: int,
    distribution: str
) -> torch.Tensor
```

Overwrite `out` with a fresh `[rank, width]` projection drawn on its own device.

Drawing on the device keeps the host out of the per-step refresh: no CPU random
stream, no temporaries, no host-to-device copy. Every data-parallel rank draws from
the same seed on identical hardware and therefore holds the same matrix; the
optimizer verifies that once before the first update.

**Parameters:**

**`out`** `Tensor`

Preallocated `[rank, width]` tensor to overwrite in place.

---

**`seed`** `int`

Seed for the device-side generator.

---

**`distribution`** `str`

`"rademacher"` or `"gaussian"`.

---

**Returns:** `Tensor`

The same tensor `out`, now filled.

**Raises:**

* `ValueError`: If a dimension is non-positive or the distribution is unknown.

```python
nemo_automodel.components.optim.gradient_sketch.sketch.install_sketches(
    model: torch.nn.Module,
    settings: nemo_automodel.components.optim.gradient_sketch.sketch.SketchSettings
) -> list[nemo_automodel.components.optim.gradient_sketch.sketch.SketchState]
```

Install sketch hooks on every target and allocate the shared sketch buffer.

Linear targets accumulate sketches directly in backward. Fused experts keep their
forward implementation and project the transient backward gradient. Target weights
get `requires_grad=False` so autograd never allocates their dense gradient.

Outputs of sketched linear layers must not be modified in place; a violation fails loudly at forward time.

```python
nemo_automodel.components.optim.gradient_sketch.sketch.local_weight_rows(
    weight: torch.Tensor
) -> tuple[torch.Tensor, int]
```

Return the local 2-D weight rows and their global row offset.

Plain tensors are returned whole (3-D expert banks flattened to 2-D). FSDP
row-sharded `DTensor`s return the local shard and its offset; any sharding on
a non-leading dimension (tensor parallelism) is rejected.

**Parameters:**

**`weight`** `Tensor`

A plain or `DTensor` weight of 2 or more dimensions.

---

**Returns:** `tuple[Tensor, int]`

A `(rows, offset)` pair: the local 2-D rows and the global index of their first row.

**Raises:**

* `ValueError`: If the weight is sharded on a dimension other than the leading one.

```python
nemo_automodel.components.optim.gradient_sketch.sketch.make_projection(
    rank: int,
    width: int,
    seed: int,
    device: torch.device | str,
    dtype: torch.dtype,
    distribution: str
) -> torch.Tensor
```

Allocate a `[rank, width]` projection and fill it on `device`.

**Parameters:**

**`rank`** `int`

Number of rows `r` of the projection.

---

**`width`** `int`

Number of columns, the weight's input dimension.

---

**`seed`** `int`

Seed for the device-side generator.

---

**`device`** `torch.device | str`

Device to allocate and draw on.

---

**`dtype`** `torch.dtype`

Element type of the projection.

---

**`distribution`** `str`

`"rademacher"` or `"gaussian"`.

---

**Returns:** `Tensor`

The filled projection tensor.

**Raises:**

* `ValueError`: If `rank` or `width` is non-positive.

```python
nemo_automodel.components.optim.gradient_sketch.sketch.projection_seed(
    base_seed: int,
    update: int,
    name: str
) -> int
```

Generate a stable per-layer seed independently of module wrappers.

**Parameters:**

**`base_seed`** `int`

Run-level seed from `SketchSettings.seed`.

---

**`update`** `int`

Optimizer update index; changes the seed on every refresh.

---

**`name`** `str`

Fully qualified module name; FSDP and checkpoint wrapper segments are ignored.

---

**Returns:** `int`

A non-negative 62-bit seed.

```python
nemo_automodel.components.optim.gradient_sketch.sketch.select_sketch_targets(
    model: torch.nn.Module,
    settings: nemo_automodel.components.optim.gradient_sketch.sketch.SketchSettings
) -> list[tuple[str, torch.nn.Module, str, bool]]
```

Return `(name, module, parameter_name, is_fused_expert)` for every sketch target.

```python
nemo_automodel.components.optim.gradient_sketch.sketch.sketch_buffer(
    states: list[nemo_automodel.components.optim.gradient_sketch.sketch.SketchState]
) -> torch.Tensor
```

Return the shared buffer that every `SketchState.sketch` is a view of.

```python
nemo_automodel.components.optim.gradient_sketch.sketch._DEFAULT_TARGET_MODULES = ('q_proj', 'k_proj', 'v_proj', 'o_proj', 'gate_proj', 'up_proj', 'down_proj')
```

```python
nemo_automodel.components.optim.gradient_sketch.sketch._DISTRIBUTIONS = ('rademacher', 'gaussian')
```

```python
nemo_automodel.components.optim.gradient_sketch.sketch._RULES = ('row_adam', 'sgd')
```