> For clean Markdown of any page, append .md to the page URL.
> For a complete documentation index, see https://docs.nvidia.com/nemo/automodel/llms.txt.
> For AI client integration (Claude Code, Cursor, etc.), connect to the MCP server at https://docs.nvidia.com/nemo/automodel/_mcp/server.

# nemo_automodel.components.models.inkling.multimodal

Native Inkling vision, audio, and multimodal composition modules.

## Module Contents

### Classes

| Name                                                                                                            | Description                                                     |
| --------------------------------------------------------------------------------------------------------------- | --------------------------------------------------------------- |
| [`InklingAudioEmbeddings`](#nemo_automodel-components-models-inkling-multimodal-InklingAudioEmbeddings)         | Embed and sum the independently quantized dMel codebooks.       |
| [`InklingAudioModel`](#nemo_automodel-components-models-inkling-multimodal-InklingAudioModel)                   | Native Inkling audio tower.                                     |
| [`InklingModel`](#nemo_automodel-components-models-inkling-multimodal-InklingModel)                             | Compose native text, vision, and audio Inkling towers.          |
| [`InklingModelOutputWithPast`](#nemo_automodel-components-models-inkling-multimodal-InklingModelOutputWithPast) | Inkling backbone output including projected image features.     |
| [`InklingVisionEncoderLayer`](#nemo_automodel-components-models-inkling-multimodal-InklingVisionEncoderLayer)   | Fold time and space into channels, then project one HMLP stage. |
| [`InklingVisionModel`](#nemo_automodel-components-models-inkling-multimodal-InklingVisionModel)                 | Native hierarchical MLP vision tower.                           |

### Functions

| Name                                                                                                              | Description                                                          |
| ----------------------------------------------------------------------------------------------------------------- | -------------------------------------------------------------------- |
| [`_minimum_cost_scale_indices`](#nemo_automodel-components-models-inkling-multimodal-_minimum_cost_scale_indices) | Select distinct ordered scales with minimum absolute-log cost.       |
| [`_plan_output_scales`](#nemo_automodel-components-models-inkling-multimodal-_plan_output_scales)                 | Plan the time, height, width, and channel sizes of every HMLP stage. |
| [`_prime_factors`](#nemo_automodel-components-models-inkling-multimodal-_prime_factors)                           | Return prime factors in ascending order.                             |

### Data

[`__all__`](#nemo_automodel-components-models-inkling-multimodal-__all__)

### API

```python
class nemo_automodel.components.models.inkling.multimodal.InklingAudioEmbeddings(
    config: nemo_automodel.components.models.inkling.configuration.InklingAudioConfig
)
```

**Bases:** `Module`

Embed and sum the independently quantized dMel codebooks.

**`embed_audio_tokens`**

---

```python
nemo_automodel.components.models.inkling.multimodal.InklingAudioEmbeddings.forward(
    input_ids: torch.Tensor
) -> torch.Tensor
```

Embed discretized mel bins.

**Parameters:**

**`input_ids`** `torch.Tensor`

Long tensor of shape `[frames, mel_bins]`.

---

**Returns:** `torch.Tensor`

Tensor of shape `[frames, hidden]`.

```python
class nemo_automodel.components.models.inkling.multimodal.InklingAudioModel(
    config: nemo_automodel.components.models.inkling.configuration.InklingAudioConfig
)
```

**Bases:** `Module`

Native Inkling audio tower.

**`embed_audio_tokens`** `= InklingAudioEmbeddings(config)`

---

**`norm`**

---

```python
nemo_automodel.components.models.inkling.multimodal.InklingAudioModel.forward(
    audio_input_ids: torch.Tensor
) -> transformers.modeling_outputs.BaseModelOutputWithPooling
```

Encode audio tokens.

**Parameters:**

**`audio_input_ids`** `torch.Tensor`

Long tensor of shape `[frames, mel_bins]`.

---

**Returns:** `BaseModelOutputWithPooling`

An output whose hidden state and pooler output have shape `[frames, hidden]`.

```python
nemo_automodel.components.models.inkling.multimodal.InklingAudioModel.init_weights() -> None
```

Initialize the audio embedding and final normalization.

```python
class nemo_automodel.components.models.inkling.multimodal.InklingModel(
    config: nemo_automodel.components.models.inkling.configuration.InklingConfig,
    backend: nemo_automodel.components.models.common.BackendConfig,
    moe_config: nemo_automodel.components.moe.config.MoEConfig
)
```

**Bases:** `Module`

Compose native text, vision, and audio Inkling towers.

**`audio_tower`** `= InklingAudioModel(config.audio_config)`

---

**`language_model`**

---

**`vision_tower`** `= InklingVisionModel(config.vision_config)`

---

```python
nemo_automodel.components.models.inkling.multimodal.InklingModel._get_placeholder_mask(
    input_ids: torch.Tensor | None,
    inputs_embeds: torch.Tensor,
    features: torch.Tensor,
    token_id: int
) -> torch.Tensor
```

Match multimodal features to placeholder tokens.

**Parameters:**

**`input_ids`** `torch.Tensor | None`

Optional long tensor of shape `[batch, sequence]`.

---

**`inputs_embeds`** `torch.Tensor`

Tensor of shape `[batch, sequence, hidden]`.

---

**`features`** `torch.Tensor`

Tensor of shape `[placeholders, hidden]`.

---

**`token_id`** `int`

Vocabulary ID used as the placeholder.

---

**Returns:** `torch.Tensor`

Boolean tensor of shape `[batch, sequence, hidden]`.

```python
nemo_automodel.components.models.inkling.multimodal.InklingModel.forward(
    input_ids: torch.LongTensor | None = None,
    pixel_values: torch.FloatTensor | None = None,
    audio_input_ids: torch.LongTensor | None = None,
    audio_input_ids_mask: torch.Tensor | None = None,
    attention_mask: torch.Tensor | dict[str, torch.Tensor] | None = None,
    position_ids: torch.LongTensor | None = None,
    past_key_values: nemo_automodel.components.models.inkling.text.InklingDynamicCache | None = None,
    inputs_embeds: torch.FloatTensor | None = None,
    use_cache: bool | None = None,
    kwargs: typing.Any = {}
) -> nemo_automodel.components.models.inkling.multimodal.InklingModelOutputWithPast
```

Merge multimodal features and run the text backbone.

**Parameters:**

**`input_ids`** `torch.LongTensor | None` — default: None

Optional long tensor of shape `[batch, sequence]`.

---

**`pixel_values`** `torch.FloatTensor | None` — default: None

Optional tensor of shape `[patches, time, height, width, channels]`.

---

**`audio_input_ids`** `torch.LongTensor | None` — default: None

Optional long tensor of shape `[audios, frames, mel_bins]`.

---

**`audio_input_ids_mask`** `torch.Tensor | None` — default: None

Optional boolean tensor of shape `[audios, frames]`.

---

**`attention_mask`** `torch.Tensor | dict[str, torch.Tensor] | None` — default: None

Optional padding tensor of shape `[batch, total_sequence]`
or mapping of prepared attention masks.

---

**`position_ids`** `torch.LongTensor | None` — default: None

Optional long tensor of shape `[batch, sequence]`.

---

**`past_key_values`** `InklingDynamicCache | None` — default: None

Optional model-owned decoding cache.

---

**`inputs_embeds`** `torch.FloatTensor | None` — default: None

Optional tensor of shape `[batch, sequence, hidden]`.

---

**`use_cache`** `bool | None` — default: None

Whether to allocate and return a decoding cache.

---

**`**kwargs`** `Any` — default: \{}

Additional text-attention arguments.

---

**Returns:** `InklingModelOutputWithPast`

An output whose last hidden state has shape `[batch, sequence, hidden]`.

```python
nemo_automodel.components.models.inkling.multimodal.InklingModel.get_audio_features(
    audio_input_ids: torch.Tensor,
    audio_input_ids_mask: torch.Tensor | None = None
) -> transformers.modeling_outputs.BaseModelOutputWithPooling
```

Project discretized audio into text hidden space.

**Parameters:**

**`audio_input_ids`** `torch.Tensor`

Long tensor of shape `[audios, frames, mel_bins]`.

---

**`audio_input_ids_mask`** `torch.Tensor | None` — default: None

Optional boolean tensor of shape `[audios, frames]`.

---

**Returns:** `BaseModelOutputWithPooling`

An output whose hidden state has shape `[valid_frames, text_hidden]`.

```python
nemo_automodel.components.models.inkling.multimodal.InklingModel.get_image_features(
    pixel_values: torch.Tensor,
    kwargs: typing.Any = {}
) -> transformers.modeling_outputs.BaseModelOutputWithPooling
```

Project image/video patches into text hidden space.

**Parameters:**

**`pixel_values`** `torch.Tensor`

Tensor of shape `[patches, time, height, width, channels]`.

---

**`**kwargs`** `Any` — default: \{}

Reserved for the common multimodal calling convention.

---

**Returns:** `BaseModelOutputWithPooling`

An output whose pooler output has shape `[patches, text_hidden]`.

```python
nemo_automodel.components.models.inkling.multimodal.InklingModel.get_input_embeddings() -> torch.nn.Module
```

Return the text token-embedding module.

```python
nemo_automodel.components.models.inkling.multimodal.InklingModel.init_weights(
    buffer_device: torch.device
) -> None
```

Initialize every tower for checkpoint-free construction.

```python
class nemo_automodel.components.models.inkling.multimodal.InklingModelOutputWithPast(
    image_hidden_states: torch.FloatTensor | None = None
)
```

Dataclass

**Bases:** `BaseModelOutputWithPast`

Inkling backbone output including projected image features.

**`image_hidden_states`** `FloatTensor | None = None`

---

```python
class nemo_automodel.components.models.inkling.multimodal.InklingVisionEncoderLayer(
    input_dim: int,
    output_dim: int,
    temporal_fold: int,
    spatial_fold: int,
    add_norm: bool,
    dtype: torch.dtype
)
```

**Bases:** `Module`

Fold time and space into channels, then project one HMLP stage.

**`layer_norm`**

---

**`projection`**

---

```python
nemo_automodel.components.models.inkling.multimodal.InklingVisionEncoderLayer._fold_timespace_to_depth(
    hidden_states: torch.Tensor
) -> torch.Tensor
```

Fold time and two spatial axes into the channel dimension.

**Parameters:**

**`hidden_states`** `torch.Tensor`

Tensor of shape `[patches, time, height, width, channels]`.

---

**Returns:** `torch.Tensor`

Tensor of shape \`\`\[patches, time / temporal\_fold, height / spatial\_fold,

```python
nemo_automodel.components.models.inkling.multimodal.InklingVisionEncoderLayer.forward(
    hidden_states: torch.Tensor
) -> torch.Tensor
```

Apply one hierarchical vision projection.

**Parameters:**

**`hidden_states`** `torch.Tensor`

Tensor of shape `[patches, time, height, width, channels]`.

---

**Returns:** `torch.Tensor`

Tensor of shape `[patches, folded_time, folded_height, folded_width, output_channels]`.

```python
nemo_automodel.components.models.inkling.multimodal.InklingVisionEncoderLayer.init_weights(
    init_std: float
) -> None
```

Initialize the projection and optional normalization.

```python
class nemo_automodel.components.models.inkling.multimodal.InklingVisionModel(
    config: nemo_automodel.components.models.inkling.configuration.InklingVisionConfig
)
```

**Bases:** `Module`

Native hierarchical MLP vision tower.

**`encoder_layers`** `= nn.ModuleList()`

---

**`final_norm`**

---

```python
nemo_automodel.components.models.inkling.multimodal.InklingVisionModel.forward(
    pixel_values: torch.Tensor,
    kwargs: typing.Any = {}
) -> transformers.modeling_outputs.BaseModelOutputWithPooling
```

Encode preprocessed image/video patches.

**Parameters:**

**`pixel_values`** `torch.Tensor`

Tensor of shape `[patches, time, height, width, channels]`.

---

**`**kwargs`** `Any` — default: \{}

Reserved for the common multimodal calling convention.

---

**Returns:** `BaseModelOutputWithPooling`

An output whose pooler output has shape `[patches, text_hidden]`.

```python
nemo_automodel.components.models.inkling.multimodal.InklingVisionModel.init_weights() -> None
```

Initialize every vision projection and normalization.

```python
nemo_automodel.components.models.inkling.multimodal._minimum_cost_scale_indices(
    cost_matrix: torch.Tensor
) -> torch.LongTensor
```

Select distinct ordered scales with minimum absolute-log cost.

**Parameters:**

**`cost_matrix`** `torch.Tensor`

Tensor of shape `[layers_plus_one, available_scales]`.

---

**Returns:** `torch.LongTensor`

Long tensor of shape `[layers_plus_one]` indexing distinct scales.

```python
nemo_automodel.components.models.inkling.multimodal._plan_output_scales(
    config: nemo_automodel.components.models.inkling.configuration.InklingVisionConfig
) -> torch.LongTensor
```

Plan the time, height, width, and channel sizes of every HMLP stage.

```python
nemo_automodel.components.models.inkling.multimodal._prime_factors(
    number: int
) -> list[int]
```

Return prime factors in ascending order.

```python
nemo_automodel.components.models.inkling.multimodal.__all__ = ['InklingModel', 'InklingModelOutputWithPast']
```