> This page is for version 0.5.0 · 26.06.
> For other versions, use one of these documentation indexes:
> - Nightly (default): https://docs.nvidia.com/nemo/automodel/nightly/llms.txt
> - Latest: https://docs.nvidia.com/nemo/automodel/latest/llms.txt
> - 0.7.0 · 26.10: https://docs.nvidia.com/nemo/automodel/v0.7/llms.txt
> - 0.5.0 · 26.06: https://docs.nvidia.com/nemo/automodel/v0.5/llms.txt
> - 0.4.0 · 26.04: https://docs.nvidia.com/nemo/automodel/v0.4/llms.txt

> For clean Markdown of any page, append .md to the page URL.
> For a complete documentation index, see https://docs.nvidia.com/nemo/automodel/llms.txt.

# nemo_automodel.components.models.inkling2.multimodal

Image/text entry point sharing Inkling2's native decoder and checkpoint loader.

## Module Contents

### Classes

| Name                                                                                                                         | Description                                                                |
| ---------------------------------------------------------------------------------------------------------------------------- | -------------------------------------------------------------------------- |
| [`Inkling2ForConditionalGeneration`](#nemo_automodel-components-models-inkling2-multimodal-Inkling2ForConditionalGeneration) | Insert unnormalized image features after the text embedding normalization. |

### API

```python
class nemo_automodel.components.models.inkling2.multimodal.Inkling2ForConditionalGeneration(
    config: nemo_automodel.components.models.inkling2.config.Inkling2Config,
    backend: nemo_automodel.components.models.common.BackendConfig | None = None,
    revision: str | None = None
)
```

**Bases:** [Inkling2ForCausalLM](/nemo/automodel/v0.5/nemo-automodel/nemo_automodel/components/models/inkling2/model#nemo_automodel-components-models-inkling2-model-Inkling2ForCausalLM)

Insert unnormalized image features after the text embedding normalization.

**`_vlm_chunk_idx`** `int | None = None`

---

**`_vlm_image_grid_hws_chunks`** `list[Tensor] | None = None`

---

**`_vlm_pixel_values_chunks`** `list[Tensor] | None = None`

---

**`state_dict_adapter`** `= Inkling2StateDictAdapter(config)`

---

**`visual`**

---

```python
nemo_automodel.components.models.inkling2.multimodal.Inkling2ForConditionalGeneration.customize_pipeline_stage_modules(
    module_names_per_stage: list[list[str]],
    layers_prefix: str,
    text_model: torch.nn.Module | None = None
) -> list[list[str]]
```

Keep the vision encoder and projector exclusively on the first stage.

**Parameters:**

**`module_names_per_stage`** `list[list[str]]`

Generated stage FQNs; caller lists are unchanged.

---

**`layers_prefix`** `str`

Empty for the native model's top-level decoder layers.

---

**`text_model`** `nn.Module | None` — default: None

Backbone resolved by the shared splitter.

---

**Returns:** `list[list[str]]`

Copied stage FQNs with one owner for vision, embedding normalization

```python
nemo_automodel.components.models.inkling2.multimodal.Inkling2ForConditionalGeneration.forward(
    input_ids: torch.Tensor,
    pipeline_tensors: torch.Tensor = (),
    inputs_embeds: torch.Tensor | None = None,
    vision_patches_bthwc: torch.Tensor | None = None,
    vision_grid_thw: torch.Tensor | None = None,
    pixel_values: torch.Tensor | None = None,
    image_grid_thw: torch.Tensor | None = None,
    mm_token_type_ids: torch.Tensor | None = None,
    attention_mask: torch.Tensor | None = None,
    padding_mask: torch.Tensor | None = None,
    position_ids: torch.Tensor | None = None,
    logits_to_keep: int | torch.Tensor = 0,
    output_hidden_states: bool = False
) -> torch.Tensor | tuple[torch.Tensor, ...] | transformers.modeling_outputs.CausalLMOutputWithPast
```

Fuse images into unpacked image/text sequences.

**Parameters:**

**`input_ids`** `torch.Tensor`

Integer \[batch, sequence] tokens, including image markers and bodies.

---

**`pipeline_tensors`** `torch.Tensor` — default: ()

Non-first-stage int64 \[batch, sequence, 2] layout,
paired with residual history \[batch, sequence, sources, hidden]
in input_ids. Later stages use only these transported inputs.

---

**`inputs_embeds`** `torch.Tensor | None` — default: None

Optional decoder-ready embeddings \[batch, sequence, hidden];
mutually exclusive with image patches.

---

**`vision_patches_bthwc`** `torch.Tensor | None` — default: None

BF16 \[patches, temporal, side, side, 3] image patches.

---

**`vision_grid_thw`** `torch.Tensor | None` — default: None

Integer \[images, 3] row-major grid metadata.

---

**`pixel_values`** `torch.Tensor | None` — default: None

Shared VLM alias for vision_patches_bthwc, shape
\[patches, temporal, side, side, 3]. Mutually exclusive with it.

---

**`image_grid_thw`** `torch.Tensor | None` — default: None

Shared VLM alias for vision_grid_thw, shape \[images, 3].
Shared PP staging supplies patch/grid chunks on stage zero only.

---

**`mm_token_type_ids`** `torch.Tensor | None` — default: None

Integer/bool \[batch, sequence], 1 only for image-body
positions. Content-image headers share token ID 200005 but must remain 0.

---

**`attention_mask`** `torch.Tensor | None` — default: None

Optional \[batch, sequence] real-token validity.

---

**`padding_mask`** `torch.Tensor | None` — default: None

Optional boolean \[batch, sequence], true for padding.

---

**`position_ids`** `torch.Tensor | None` — default: None

Optional integer \[batch, sequence] consecutive positions.

---

**`logits_to_keep`** `int | torch.Tensor` — default: 0

Tail length or sequence indices for the shared loss path.

---

**`output_hidden_states`** `bool` — default: False

Return final decoder features for external loss.

---

**Returns:** `torch.Tensor | tuple[torch.Tensor, ...] | CausalLMOutputWithPast`

The text decoder's FP32 logits or output object; inputs are not mutated.

```python
nemo_automodel.components.models.inkling2.multimodal.Inkling2ForConditionalGeneration.from_config(
    config: nemo_automodel.components.models.inkling2.config.Inkling2Config | nemo_automodel.components.models.inkling2.config.Inkling2TextConfig,
    backend: nemo_automodel.components.models.common.BackendConfig | None = None
) -> nemo_automodel.components.models.inkling2.multimodal.Inkling2ForConditionalGeneration
```

classmethod

Construct the registered multimodal architecture.

```python
nemo_automodel.components.models.inkling2.multimodal.Inkling2ForConditionalGeneration.initialize_weights(
    buffer_device: torch.device | None = None
) -> None
```

Initialize the decoder and vision encoder after meta materialization.