> This page is for version 0.5.0 · 26.06.
> For other versions, use one of these documentation indexes:
> - Nightly (default): https://docs.nvidia.com/nemo/automodel/nightly/llms.txt
> - Latest: https://docs.nvidia.com/nemo/automodel/latest/llms.txt
> - 0.5.0 · 26.06: https://docs.nvidia.com/nemo/automodel/v0.5/llms.txt
> - 0.4.0 · 26.04: https://docs.nvidia.com/nemo/automodel/v0.4/llms.txt

> For clean Markdown of any page, append .md to the page URL.
> For a complete documentation index, see https://docs.nvidia.com/nemo/automodel/llms.txt.
> For AI client integration (Claude Code, Cursor, etc.), connect to the MCP server at https://docs.nvidia.com/nemo/automodel/_mcp/server.

# nemo_automodel.components.models.nemotron_omni.state_dict_adapter

State dict adapter for NemotronOmni (NemotronH\_Nano\_Omni\_Reasoning\_V3) models.

Converts between HuggingFace checkpoint format and the custom Automodel format.

HF checkpoint key structure (from model.safetensors.index.json):

# Vision encoder (RADIO) -- loaded as-is into self.vision\_model

vision\_model.radio\_model.model.blocks.\{N}.\{...}
vision\_model.radio\_model.input\_conditioner.norm\_mean
vision\_model.radio\_model.input\_conditioner.norm\_std
vision\_model.radio\_model.model.patch\_generator.\{...}

# Vision projector -- loaded into self.vision\_projector

HF:     mlp1.0.weight  (RMSNorm)
Custom: vision\_projector.norm.weight
HF:     mlp1.1.weight  (Linear1)
Custom: vision\_projector.linear1.weight
HF:     mlp1.3.weight  (Linear2)
Custom: vision\_projector.linear2.weight

# Sound encoder (Parakeet) -- loaded into self.sound\_encoder

HF:     sound\_encoder.encoder.\{...}
Custom: sound\_encoder.\{...}

# Sound projector -- loaded into self.sound\_projection

HF:     sound\_projection.norm.weight
Custom: sound\_projection.norm.weight
HF:     sound\_projection.linear1.weight
Custom: sound\_projection.linear1.weight
HF:     sound\_projection.linear2.weight
Custom: sound\_projection.linear2.weight

# LLM (NemotronH) -- uses nemotron\_v3 state\_dict\_adapter internally

HF:     language\_model.backbone.embeddings.weight
Custom: language\_model.model.embed\_tokens.weight
HF:     language\_model.backbone.layers.\{N}.\{...}
Custom: language\_model.model.layers.\{N}.\{...}
HF:     language\_model.backbone.norm\_f.weight
Custom: language\_model.model.norm.weight
HF:     language\_model.lm\_head.weight
Custom: language\_model.lm\_head.weight

For MoE layers in the LLM:
HF:     language\_model.backbone.layers.\{N}.mixer.experts.\{E}.up\_proj.weight   (split per-expert)
HF:     language\_model.backbone.layers.\{N}.mixer.experts.\{E}.down\_proj.weight
Custom: language\_model.model.layers.\{N}.mixer.experts.gate\_and\_up\_projs       (merged)
Custom: language\_model.model.layers.\{N}.mixer.experts.down\_projs

## Module Contents

### Classes

| Name                                                                                                                              | Description                                                                        |
| --------------------------------------------------------------------------------------------------------------------------------- | ---------------------------------------------------------------------------------- |
| [`NemotronOmniStateDictAdapter`](#nemo_automodel-components-models-nemotron_omni-state_dict_adapter-NemotronOmniStateDictAdapter) | State dict adapter for NemotronOmni (NemotronH\_Nano\_Omni\_Reasoning\_V3) models. |

### Functions

| Name                                                                                                                                  | Description                                                                                 |
| ------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------- |
| [`_map_vision_proj_key_to_custom`](#nemo_automodel-components-models-nemotron_omni-state_dict_adapter-_map_vision_proj_key_to_custom) | Map a HF vision projector parameter or module name to custom format.                        |
| [`_map_vision_proj_key_to_hf`](#nemo_automodel-components-models-nemotron_omni-state_dict_adapter-_map_vision_proj_key_to_hf)         | Map a custom vision projector parameter or module name to HF format.                        |
| [`_radio_legacy_to_native`](#nemo_automodel-components-models-nemotron_omni-state_dict_adapter-_radio_legacy_to_native)               | Rename one legacy ("radio\_model.model.blocks...") RADIO sub-key to the native layout.      |
| [`_radio_native_to_legacy`](#nemo_automodel-components-models-nemotron_omni-state_dict_adapter-_radio_native_to_legacy)               | Inverse of `_radio_legacy_to_native`: native RadioModel sub-keys -> legacy checkpoint keys. |
| [`_rename_radio_key`](#nemo_automodel-components-models-nemotron_omni-state_dict_adapter-_rename_radio_key)                           | Apply every matching rename in order (chained), not just the first match.                   |

### Data

[`_PEFT_PREFIX`](#nemo_automodel-components-models-nemotron_omni-state_dict_adapter-_PEFT_PREFIX)

[`_RADIO_LEGACY_TO_NATIVE_RENAMES`](#nemo_automodel-components-models-nemotron_omni-state_dict_adapter-_RADIO_LEGACY_TO_NATIVE_RENAMES)

[`_RADIO_NATIVE_TO_LEGACY_RENAMES`](#nemo_automodel-components-models-nemotron_omni-state_dict_adapter-_RADIO_NATIVE_TO_LEGACY_RENAMES)

[`_RADIO_QKV_SPLIT_TARGETS`](#nemo_automodel-components-models-nemotron_omni-state_dict_adapter-_RADIO_QKV_SPLIT_TARGETS)

[`_VISION_PROJ_CUSTOM_TO_HF`](#nemo_automodel-components-models-nemotron_omni-state_dict_adapter-_VISION_PROJ_CUSTOM_TO_HF)

[`_VISION_PROJ_HF_TO_CUSTOM`](#nemo_automodel-components-models-nemotron_omni-state_dict_adapter-_VISION_PROJ_HF_TO_CUSTOM)

[`_VISION_PROJ_MODULE_CUSTOM_TO_HF`](#nemo_automodel-components-models-nemotron_omni-state_dict_adapter-_VISION_PROJ_MODULE_CUSTOM_TO_HF)

[`_VISION_PROJ_MODULE_HF_TO_CUSTOM`](#nemo_automodel-components-models-nemotron_omni-state_dict_adapter-_VISION_PROJ_MODULE_HF_TO_CUSTOM)

[`logger`](#nemo_automodel-components-models-nemotron_omni-state_dict_adapter-logger)

### API

```python
class nemo_automodel.components.models.nemotron_omni.state_dict_adapter.NemotronOmniStateDictAdapter(
    config,
    llm_config,
    moe_config: nemo_automodel.components.moe.config.MoEConfig,
    backend: nemo_automodel.components.models.common.BackendConfig,
    dtype: torch.dtype = torch.bfloat16,
    vision_uses_native_radio: bool = False
)
```

**Bases:** [MoESplitExpertsStateDictMixin](/nemo-automodel/nemo_automodel/components/moe/state_dict_mixin#nemo_automodel-components-moe-state_dict_mixin-MoESplitExpertsStateDictMixin), [StateDictAdapter](/nemo-automodel/nemo_automodel/components/checkpoint/state_dict_adapter#nemo_automodel-components-checkpoint-state_dict_adapter-StateDictAdapter)

State dict adapter for NemotronOmni (NemotronH\_Nano\_Omni\_Reasoning\_V3) models.

Handles conversion between HF checkpoint format and custom Automodel format.

The adapter delegates LLM key conversion to NemotronV3StateDictAdapter
(which handles backbone->model renaming, norm\_f->norm, embeddings->embed\_tokens,
and MoE expert merging) and handles vision/audio components directly.

**`_expert_path_segment`** `str`

NemotronOmni uses 'mixer.experts' in its LLM tower.

---

**`_llm_adapter`**

---

**`_v5_peft_target_parameters`** `tuple[str, ...]`

Omni PEFT export targets the remote-code receiver with per-expert target\_modules.

---

**`supports_low_memory_dcp_load`** `bool`

Whether the embedded adapters can load through DCP without model-sized temporaries.

Native RadioModel checkpoints require converting one fused `attn.qkv`
tensor into three model tensors, so that vision layout cannot use the
low-memory path.

---

**`view_loaded_native_keys`** `set[str]`

Language-model keys loaded through model-backed checkpoint views.

---

```python
nemo_automodel.components.models.nemotron_omni.state_dict_adapter.NemotronOmniStateDictAdapter.convert_single_tensor_to_hf(
    fqn: str,
    tensor: typing.Any,
    kwargs = {}
) -> list[tuple[str, typing.Any]]
```

Convert a single tensor from custom format to HF format.

**Parameters:**

**`fqn`** `str`

Fully qualified name of the tensor

---

**`tensor`** `Any`

The tensor to convert

---

**`**kwargs`** — default: \{}

Additional arguments

---

**Returns:** `list[tuple[str, Any]]`

List of (fqn, tensor) tuples in HF format

```python
nemo_automodel.components.models.nemotron_omni.state_dict_adapter.NemotronOmniStateDictAdapter.from_hf(
    hf_state_dict: dict[str, typing.Any],
    device_mesh: typing.Optional[torch.distributed.device_mesh.DeviceMesh] = None,
    kwargs = {}
) -> dict[str, typing.Any]
```

Convert HF checkpoint state dict to custom Automodel format.

Steps:

1. Separate HF state dict into: vision\_model, mlp1, sound\_encoder,
   sound\_projection, language\_model components
2. Convert vision projector keys (mlp1.\* -> vision\_projector.\*)
3. Convert sound encoder keys (sound\_encoder.encoder.\* -> sound\_encoder.\*)
4. Pass language\_model keys through NemotronV3StateDictAdapter
5. Merge everything back

**Parameters:**

**`hf_state_dict`** `dict[str, Any]`

HuggingFace format state dict

---

**`device_mesh`** `Optional[DeviceMesh]` — default: None

Optional device mesh for distributed expert loading

---

**`**kwargs`** — default: \{}

Additional arguments

---

**Returns:** `dict[str, Any]`

Custom format state dict

```python
nemo_automodel.components.models.nemotron_omni.state_dict_adapter.NemotronOmniStateDictAdapter.map_peft_target_module_to_hf(
    name: str,
    v4_compatible: bool = False
) -> str
```

Map native PEFT target modules to the public HF namespace.

```python
nemo_automodel.components.models.nemotron_omni.state_dict_adapter.NemotronOmniStateDictAdapter.to_hf(
    state_dict: dict[str, typing.Any],
    exclude_key_regex: str | None = None,
    kwargs = {}
) -> dict[str, typing.Any]
```

Convert custom Automodel state dict to HF format.

Steps:

1. Separate state dict into components
2. Convert vision projector keys back (vision\_projector.\* -> mlp1.\*)
3. Convert sound encoder keys back (sound\_encoder.\* -> sound\_encoder.encoder.\*)
4. Pass LLM keys through NemotronV3StateDictAdapter.to\_hf
5. Merge everything back

**Parameters:**

**`state_dict`** `dict[str, Any]`

Custom format state dict

---

**`exclude_key_regex`** `str | None` — default: None

Optional regex pattern to exclude keys

---

**`**kwargs`** — default: \{}

Additional arguments

---

**Returns:** `dict[str, Any]`

HuggingFace format state dict

```python
nemo_automodel.components.models.nemotron_omni.state_dict_adapter._map_vision_proj_key_to_custom(
    bare_key: str
) -> str
```

Map a HF vision projector parameter or module name to custom format.

```python
nemo_automodel.components.models.nemotron_omni.state_dict_adapter._map_vision_proj_key_to_hf(
    bare_fqn: str
) -> str
```

Map a custom vision projector parameter or module name to HF format.

```python
nemo_automodel.components.models.nemotron_omni.state_dict_adapter._radio_legacy_to_native(
    sub_key: str,
    value: torch.Tensor
) -> list[tuple[str, torch.Tensor]]
```

Rename one legacy ("radio\_model.model.blocks...") RADIO sub-key to the native layout.

Returns a list because a fused `attn.qkv` tensor splits into three (query/key/value).
The general rename chain (e.g. "radio\_model.model.blocks" -> "encoder.layer") is applied
first so it composes with the "attn.qkv" -> query/key/value split that follows.

```python
nemo_automodel.components.models.nemotron_omni.state_dict_adapter._radio_native_to_legacy(
    vision_sub_dict: dict[str, torch.Tensor]
) -> dict[str, torch.Tensor]
```

Inverse of `_radio_legacy_to_native`: native RadioModel sub-keys -> legacy checkpoint keys.

`attention.attention.{query,key,value}` triplets are concatenated back into a single
fused `attn.qkv` tensor (query, key, value order along dim 0). The qkv-target ->
"attn.qkv" placeholder substitution happens before the general rename chain, so
"encoder.layer" still gets reverse-renamed to "radio\_model.model.blocks" for qkv keys too.

```python
nemo_automodel.components.models.nemotron_omni.state_dict_adapter._rename_radio_key(
    sub_key: str,
    renames: list[tuple[str, str]]
) -> str
```

Apply every matching rename in order (chained), not just the first match.

E.g. "radio\_model.model.blocks.0.attn.proj.weight" needs both the
"radio\_model.model.blocks" -> "encoder.layer" AND "attn.proj" -> "attention.output.dense"
renames applied in sequence.

```python
nemo_automodel.components.models.nemotron_omni.state_dict_adapter._PEFT_PREFIX = 'base_model.model.'
```

```python
nemo_automodel.components.models.nemotron_omni.state_dict_adapter._RADIO_LEGACY_TO_NATIVE_RENAMES: list[tuple[str, str]] = [('radio_model.model.patch_generator.video_embedder', 'embeddings.video_patch_pr...
```

```python
nemo_automodel.components.models.nemotron_omni.state_dict_adapter._RADIO_NATIVE_TO_LEGACY_RENAMES: list[tuple[str, str]] = [(new, old) for old, new in _RADIO_LEGACY_TO_NATIVE_RENAMES]
```

```python
nemo_automodel.components.models.nemotron_omni.state_dict_adapter._RADIO_QKV_SPLIT_TARGETS = ('attention.attention.query', 'attention.attention.key', 'attention.attention.va...
```

```python
nemo_automodel.components.models.nemotron_omni.state_dict_adapter._VISION_PROJ_CUSTOM_TO_HF = {v: k for k, v in (_VISION_PROJ_HF_TO_CUSTOM.items())}
```

```python
nemo_automodel.components.models.nemotron_omni.state_dict_adapter._VISION_PROJ_HF_TO_CUSTOM = {'mlp1.0.weight': 'vision_projector.norm.weight', 'mlp1.1.weight': 'vision_proje...
```

```python
nemo_automodel.components.models.nemotron_omni.state_dict_adapter._VISION_PROJ_MODULE_CUSTOM_TO_HF = {'vision_projector.norm': 'mlp1.0', 'vision_projector.linear1': 'mlp1.1', 'visio...
```

```python
nemo_automodel.components.models.nemotron_omni.state_dict_adapter._VISION_PROJ_MODULE_HF_TO_CUSTOM = {v: k for k, v in (_VISION_PROJ_MODULE_CUSTOM_TO_HF.items())}
```

```python
nemo_automodel.components.models.nemotron_omni.state_dict_adapter.logger = logging.getLogger(__name__)
```