> This page is for version 0.4.0 · 26.04.
> For other versions, use one of these documentation indexes:
> - Nightly (default): https://docs.nvidia.com/nemo/automodel/nightly/llms.txt
> - Latest: https://docs.nvidia.com/nemo/automodel/latest/llms.txt
> - 0.7.0 · 26.10: https://docs.nvidia.com/nemo/automodel/v0.7/llms.txt
> - 0.5.0 · 26.06: https://docs.nvidia.com/nemo/automodel/v0.5/llms.txt
> - 0.4.0 · 26.04: https://docs.nvidia.com/nemo/automodel/v0.4/llms.txt

> For clean Markdown of any page, append .md to the page URL.
> For a complete documentation index, see https://docs.nvidia.com/nemo/automodel/llms.txt.

# nemo_automodel.components.optim.gradient_sketch.optimizer

Optimize gradient sketches while delegating ordinary parameters to a native optimizer.

## Module Contents

### Classes

| Name                                                                                                            | Description                                                                       |
| --------------------------------------------------------------------------------------------------------------- | --------------------------------------------------------------------------------- |
| [`GradientSketchOptimizer`](#nemo_automodel-components-optim-gradient_sketch-optimizer-GradientSketchOptimizer) | RowAdam or SGD on gradient sketches, with a native optimizer for everything else. |

### Functions

| Name                                                                                          | Description                                                                                              |
| --------------------------------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------- |
| [`_local`](#nemo_automodel-components-optim-gradient_sketch-optimizer-_local)                 | Return the local shard of a `DTensor`, or the tensor itself.                                             |
| [`_parameter_fqn`](#nemo_automodel-components-optim-gradient_sketch-optimizer-_parameter_fqn) | Canonical parameter FQN of a sketch target (`&lt;module&gt;.&lt;param&gt;`, no leading dot at the root). |
| [`_rows`](#nemo_automodel-components-optim-gradient_sketch-optimizer-_rows)                   | View the local part of a stored tensor as 2-D `[rows, width]`, like `local_weight_rows`.                 |

### Data

[`_COUNT_KEY`](#nemo_automodel-components-optim-gradient_sketch-optimizer-_COUNT_KEY)

[`_SETTINGS_KEY`](#nemo_automodel-components-optim-gradient_sketch-optimizer-_SETTINGS_KEY)

### API

```python
class nemo_automodel.components.optim.gradient_sketch.optimizer.GradientSketchOptimizer(
    model: torch.nn.Module,
    native: torch.optim.Optimizer,
    settings: nemo_automodel.components.optim.gradient_sketch.sketch.SketchSettings
)
```

**Bases:** `Optimizer`

RowAdam or SGD on gradient sketches, with a native optimizer for everything else.

`param_groups[0]` holds the sketch targets (plus `update_count` and the
settings); the remaining groups are the native optimizer's own group objects (none
when every non-target parameter is frozen), and `state` is shared with it. `state`
holds native-parameter state only, so it is empty until the first native step and
`torch.distributed.checkpoint` initializes it for any native optimizer. Sketch-target
state (flat RowAdam row moments, fp32 masters of low-precision targets) is kept on the
wrapper and exposed through `sketch_state_dict` / `load_sketch_state_dict`
(Automodel's `OptimizerState` saves it under the `"sketch"` key). Construct it before
the native optimizer's first step: target parameters are removed before it can allocate
full-size moment buffers.

**`anchor_handle`**

---

**`layers`** `list[SketchState] = install_sketches(model, settings)`

---

**`local_rows`** `list[int]`

---

**`masters`** `list[Tensor | None]`

---

**`row_second_moment`** `Tensor | None = None`

---

**`sketches`** `Tensor = sketch_buffer(self.layers)`

---

**`update_count`** `int`

Number of completed optimizer updates, stored in `param_groups[0]` for checkpoints.

---

```python
nemo_automodel.components.optim.gradient_sketch.optimizer.GradientSketchOptimizer._anchor(
    module: torch.nn.Module,
    inputs: tuple[torch.Tensor, ...],
    output: torch.Tensor
) -> torch.Tensor
```

staticmethod

```python
nemo_automodel.components.optim.gradient_sketch.optimizer.GradientSketchOptimizer._check_sharded_targets() -> None
```

```python
nemo_automodel.components.optim.gradient_sketch.optimizer.GradientSketchOptimizer._new_master(
    weight: torch.Tensor
) -> torch.Tensor
```

staticmethod

Create an fp32 master copy; a `DTensor` with the weight's layout under FSDP so DCP shards it.

```python
nemo_automodel.components.optim.gradient_sketch.optimizer.GradientSketchOptimizer._verify_shared_projections(
    process_group: typing.Any
) -> None
```

```python
nemo_automodel.components.optim.gradient_sketch.optimizer.GradientSketchOptimizer.load_sketch_state_dict(
    sketch_state: dict[str, torch.Tensor]
) -> None
```

Restore state produced by `sketch_state_dict`.

**Parameters:**

**`sketch_state`** `dict[str, Tensor]`

Mapping with the same keys as `sketch_state_dict`.

---

**Raises:**

* `ValueError`: If a key is missing or unexpected, or a shape differs.

```python
nemo_automodel.components.optim.gradient_sketch.optimizer.GradientSketchOptimizer.load_state_dict(
    state_dict: dict[str, typing.Any]
) -> None
```

Load native state, group settings and `update_count` from `state_dict`.

The native half is handed to `native.load_state_dict` (`torch.optim.Optimizer`
casts loaded state to the parameter dtype; TE `FusedAdam` restores fp32 there).
Sketch-target state is not read here; use `load_sketch_state_dict`.

**Parameters:**

**`state_dict`** `dict[str, Any]`

A dict produced by `state_dict`.

---

**Raises:**

* `ValueError`: If the saved sketch settings differ from the configured ones, or the
  saved group layout differs from the current one.

```python
nemo_automodel.components.optim.gradient_sketch.optimizer.GradientSketchOptimizer.refresh_projection() -> None
```

Redraw every projection on its device from `(seed, update_count, layer name)`.

```python
nemo_automodel.components.optim.gradient_sketch.optimizer.GradientSketchOptimizer.reset_master_weights() -> None
```

Rebase fp32 masters after loading model weights without optimizer state.

```python
nemo_automodel.components.optim.gradient_sketch.optimizer.GradientSketchOptimizer.sketch_state_dict() -> dict[str, torch.Tensor]
```

Live tensors holding the state of sketch-target parameters.

**Returns:** `dict[str, Tensor]`

`row_second_moment` (the flat `[rows, 1]` fp32 buffer, identical on every rank, only

```python
nemo_automodel.components.optim.gradient_sketch.optimizer.GradientSketchOptimizer.state_dict() -> dict[str, typing.Any]
```

Pack `param_groups[0]` (settings, `update_count`) with the native optimizer's own state dict.

The native half comes from `native.state_dict()` so an optimizer with its own
serialization (TE `FusedAdam` unscales low-precision moments to fp32) is honoured;
its parameter indices are shifted past the sketch targets so the layout matches
`torch.optim.Optimizer`'s packing of all groups in order.

```python
nemo_automodel.components.optim.gradient_sketch.optimizer.GradientSketchOptimizer.step(
    closure: typing.Callable[[], float] | None = None,
    scale: float = 1.0
) -> float | None
```

Apply the sketch update to targets and the native step to everything else.

An unready call with every group's `lr` equal to 0 is the state-initialization probe
used by `torch.distributed.checkpoint`: only the native optimizer runs, so its state
is created; sketches, projections and `update_count` are untouched.

**Parameters:**

**`closure`** `Callable[[], float] | None` — default: None

Unsupported; must be `None`.

---

**`scale`** `float` — default: 1.0

Multiplies only the sketch-target update, not its weight decay or the native step.

---

**Returns:** `float | None`

`None`; closures are unsupported, so there is no loss to return.

**Raises:**

* `RuntimeError`: If `synchronize_and_clip` has not run this step (and this is not the
  zero-lr probe), or a closure is given.

```python
nemo_automodel.components.optim.gradient_sketch.optimizer.GradientSketchOptimizer.synchronize_and_clip(
    process_group: typing.Any,
    max_norm: float | None
) -> torch.Tensor
```

Average sketches across data-parallel ranks, then clip the implied gradient globally.

After the all-reduce every rank holds the full averaged sketch, so the sketch part of the
norm is computed locally over all rows; only the (sharded) native gradient norm is reduced.
`process_group` must be the FSDP shard group; HSDP replicate groups are unsupported.

**Parameters:**

**`process_group`** `Any`

FSDP shard group, or `None` for the default group / a single process.

---

**`max_norm`** `float | None`

Clip the global gradient norm to this value, or `None` to only measure it.

---

**Returns:** `Tensor`

The pre-clip global norm as a 0-d tensor on the sketch device.

**Raises:**

* `RuntimeError`: If sketches were already synchronized this step, targets are not
  FSDP-sharded on more than one rank, or projections differ across ranks.
* `FloatingPointError`: If the norm is not finite.

```python
nemo_automodel.components.optim.gradient_sketch.optimizer.GradientSketchOptimizer.to(
    device: torch.device | str
) -> 'GradientSketchOptimizer'
```

Move native state, the flat row moments and the fp32 masters to `device`.

```python
nemo_automodel.components.optim.gradient_sketch.optimizer.GradientSketchOptimizer.zero_grad(
    set_to_none: bool = True
) -> None
```

Clear gradients of every parameter group and zero the sketch buffer.

**Parameters:**

**`set_to_none`** `bool` — default: True

Set gradients to `None` instead of zero, as in `torch.optim.Optimizer`.

---

```python
nemo_automodel.components.optim.gradient_sketch.optimizer._local(
    tensor: torch.Tensor
) -> torch.Tensor
```

Return the local shard of a `DTensor`, or the tensor itself.

```python
nemo_automodel.components.optim.gradient_sketch.optimizer._parameter_fqn(
    layer: nemo_automodel.components.optim.gradient_sketch.sketch.SketchState
) -> str
```

Canonical parameter FQN of a sketch target (`&lt;module&gt;.&lt;param&gt;`, no leading dot at the root).

```python
nemo_automodel.components.optim.gradient_sketch.optimizer._rows(
    tensor: torch.Tensor
) -> torch.Tensor
```

View the local part of a stored tensor as 2-D `[rows, width]`, like `local_weight_rows`.

```python
nemo_automodel.components.optim.gradient_sketch.optimizer._COUNT_KEY = 'update_count'
```

```python
nemo_automodel.components.optim.gradient_sketch.optimizer._SETTINGS_KEY = 'sketch_settings'
```