> This page is for version 0.4.0 · 26.04.
> For other versions, use one of these documentation indexes:
> - Nightly (default): https://docs.nvidia.com/nemo/automodel/nightly/llms.txt
> - Latest: https://docs.nvidia.com/nemo/automodel/latest/llms.txt
> - 0.7.0 · 26.10: https://docs.nvidia.com/nemo/automodel/v0.7/llms.txt
> - 0.5.0 · 26.06: https://docs.nvidia.com/nemo/automodel/v0.5/llms.txt
> - 0.4.0 · 26.04: https://docs.nvidia.com/nemo/automodel/v0.4/llms.txt

> For clean Markdown of any page, append .md to the page URL.
> For a complete documentation index, see https://docs.nvidia.com/nemo/automodel/llms.txt.

# nemo_automodel.components.models.lfm3.vl_model

LFM3-VL: the LFM3 decoder fed by a convolution-and-attention vision encoder through an MLP projector.

## Module Contents

### Classes

| Name                                                                                                               | Description                                                                                         |
| ------------------------------------------------------------------------------------------------------------------ | --------------------------------------------------------------------------------------------------- |
| [`Lfm3VlForConditionalGeneration`](#nemo_automodel-components-models-lfm3-vl_model-Lfm3VlForConditionalGeneration) | LFM3-VL with a vocabulary head, loadable from LiquidAI's released checkpoints.                      |
| [`Lfm3VlModel`](#nemo_automodel-components-models-lfm3-vl_model-Lfm3VlModel)                                       | LFM3 decoder whose image and video placeholder tokens are replaced by projected vision features.    |
| [`Lfm3VlMultiModalProjector`](#nemo_automodel-components-models-lfm3-vl_model-Lfm3VlMultiModalProjector)           | Normalize vision tokens, merge each `spatial_merge_size` square of them, and map to the text width. |
| [`Lfm3VlPreTrainedModel`](#nemo_automodel-components-models-lfm3-vl_model-Lfm3VlPreTrainedModel)                   | Weight initialization and loading shared by the LFM3-VL models.                                     |
| [`Lfm3VlVisionAttention`](#nemo_automodel-components-models-lfm3-vl_model-Lfm3VlVisionAttention)                   | LFM2 grouped-query attention made bidirectional, attending within one frame at a time.              |
| [`Lfm3VlVisionLayer`](#nemo_automodel-components-models-lfm3-vl_model-Lfm3VlVisionLayer)                           | Pre-norm operator (per-frame attention or 2D convolution) followed by a pre-norm MLP.               |
| [`Lfm3VlVisionModel`](#nemo_automodel-components-models-lfm3-vl_model-Lfm3VlVisionModel)                           | LFM3-VL vision encoder over the flattened patches of images and video frames.                       |
| [`Lfm3VlVisionShortConv`](#nemo_automodel-components-models-lfm3-vl_model-Lfm3VlVisionShortConv)                   | Gated depthwise 2D convolution over the patch grid of each frame.                                   |

### Functions

| Name                                                                                           | Description                                                                           |
| ---------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------- |
| [`axial_rotary_factors`](#nemo_automodel-components-models-lfm3-vl_model-axial_rotary_factors) | Build 2D rotary factors in float32: half of each head rotates by row, half by column. |

### Data

[`ModelClass`](#nemo_automodel-components-models-lfm3-vl_model-ModelClass)

### API

```python
class nemo_automodel.components.models.lfm3.vl_model.Lfm3VlForConditionalGeneration(
    config: nemo_automodel.components.models.lfm3.config.Lfm3VlConfig
)
```

**Bases:** [HFCheckpointingMixin](/nemo/automodel/v0.4/nemo-automodel/nemo_automodel/components/models/common/hf_checkpointing_mixin#nemo_automodel-components-models-common-hf_checkpointing_mixin-HFCheckpointingMixin), [Lfm3VlPreTrainedModel](#nemo_automodel-components-models-lfm3-vl_model-Lfm3VlPreTrainedModel), `GenerationMixin`

LFM3-VL with a vocabulary head, loadable from LiquidAI's released checkpoints.

**`_tied_weights_keys`**

---

**`lm_head`**

---

**`model`** `= Lfm3VlModel(config)`

---

**`tie_word_embeddings_support`** `TieSupport = TieSupport.BOTH`

---

```python
nemo_automodel.components.models.lfm3.vl_model.Lfm3VlForConditionalGeneration.forward(
    input_ids: torch.LongTensor | None = None,
    attention_mask: torch.Tensor | dict[str, torch.Tensor | None] | None = None,
    position_ids: torch.LongTensor | None = None,
    past_key_values: transformers.cache_utils.Cache | None = None,
    inputs_embeds: torch.FloatTensor | None = None,
    pixel_values: torch.FloatTensor | None = None,
    image_grid_thw: torch.LongTensor | None = None,
    pixel_values_videos: torch.FloatTensor | None = None,
    video_grid_thw: torch.LongTensor | None = None,
    labels: torch.LongTensor | None = None,
    use_cache: bool | None = None,
    logits_to_keep: int | torch.Tensor = 0,
    mm_encoder_outputs: dict[str, transformers.modeling_outputs.BaseModelOutputWithPooling] | None = None,
    kwargs: transformers.processing_utils.Unpack[transformers.utils.TransformersKwargs] = {}
) -> transformers.modeling_outputs.CausalLMOutputWithPast
```

Predict next-token logits over text interleaved with images and videos.

**Parameters:**

**`input_ids`** `torch.LongTensor | None` — default: None

Long Tensor of shape \[batch, sequence]. Exclusive with `inputs_embeds`.

---

**`attention_mask`** `torch.Tensor | dict[str, torch.Tensor | None] | None` — default: None

Optional padding Tensor of shape \[batch, key_sequence], where
`key_sequence` also counts cached tokens.

---

**`position_ids`** `torch.LongTensor | None` — default: None

Optional long Tensor of shape \[batch, sequence].

---

**`past_key_values`** `Cache | None` — default: None

Optional generation cache, updated in place.

---

**`inputs_embeds`** `torch.FloatTensor | None` — default: None

Tensor of shape \[batch, sequence, hidden].

---

**`pixel_values`** `torch.FloatTensor | None` — default: None

Image patches, Tensor of shape
\[patches, in_channels \* temporal_patch_size \* patch_size\*\*2].

---

**`image_grid_thw`** `torch.LongTensor | None` — default: None

Long Tensor of shape \[images, 3], each patch grid as (frames, height, width).

---

**`pixel_values_videos`** `torch.FloatTensor | None` — default: None

Video patches, laid out like `pixel_values`.

---

**`video_grid_thw`** `torch.LongTensor | None` — default: None

Long Tensor of shape \[videos, 3], laid out like `image_grid_thw`.

---

**`labels`** `torch.LongTensor | None` — default: None

Optional long Tensor of shape \[batch, sequence]; `-100` is ignored.
Shifted internally.

---

**`use_cache`** `bool | None` — default: None

Whether to create and return a generation cache.

---

**`logits_to_keep`** `int | torch.Tensor` — default: 0

Project only the last `logits_to_keep` positions (0 keeps
all), or the positions in a long Tensor of shape \[kept].

---

**`mm_encoder_outputs`** `dict[str, BaseModelOutputWithPooling] | None` — default: None

Precomputed vision features, as generation passes them.

---

**`**kwargs`** `Unpack[TransformersKwargs]` — default: \{}

Hugging Face output-capture, attention and loss arguments.

---

**Returns:** `CausalLMOutputWithPast`

Output whose `logits` is a Tensor of shape \[batch, kept, vocab] and whose

```python
nemo_automodel.components.models.lfm3.vl_model.Lfm3VlForConditionalGeneration.get_input_embeddings() -> torch.nn.Embedding
```

Return the token embedding table.

```python
nemo_automodel.components.models.lfm3.vl_model.Lfm3VlForConditionalGeneration.get_output_embeddings() -> torch.nn.Linear
```

Return the vocabulary head.

```python
nemo_automodel.components.models.lfm3.vl_model.Lfm3VlForConditionalGeneration.set_input_embeddings(
    value: torch.nn.Embedding
) -> None
```

Replace the token embedding table and retie the head.

```python
nemo_automodel.components.models.lfm3.vl_model.Lfm3VlForConditionalGeneration.tie_weights(
    kwargs: bool | set[str] | None = {}
) -> None
```

Share the vocabulary head with the token embeddings when the config asks for it.

```python
class nemo_automodel.components.models.lfm3.vl_model.Lfm3VlModel(
    config: nemo_automodel.components.models.lfm3.config.Lfm3VlConfig
)
```

**Bases:** [Lfm3VlPreTrainedModel](#nemo_automodel-components-models-lfm3-vl_model-Lfm3VlPreTrainedModel)

LFM3 decoder whose image and video placeholder tokens are replaced by projected vision features.

**`language_model`** `= Lfm3Model(config.text_config)`

---

**`multi_modal_projector`** `= Lfm3VlMultiModalProjector(config)`

---

**`vision_tower`** `= Lfm3VlVisionModel(config.vision_config)`

---

```python
nemo_automodel.components.models.lfm3.vl_model.Lfm3VlModel._fill_placeholders(
    inputs_embeds: torch.Tensor,
    input_ids: torch.LongTensor | None,
    token_id: int,
    features: tuple[torch.Tensor, ...]
) -> torch.Tensor
```

Write vision features over the embeddings of one modality's placeholder tokens.

**Parameters:**

**`inputs_embeds`** `torch.Tensor`

Tensor of shape \[batch, sequence, hidden].

---

**`input_ids`** `torch.LongTensor | None`

Long Tensor of shape \[batch, sequence] locating the placeholders.

---

**`token_id`** `int`

The modality's placeholder token.

---

**`features`** `tuple[torch.Tensor, ...]`

One Tensor of shape \[tokens, hidden] per image or video, in the
order their placeholders appear row by row.

---

**Returns:** `torch.Tensor`

New Tensor of shape \[batch, sequence, hidden]; `inputs_embeds` is not modified.

```python
nemo_automodel.components.models.lfm3.vl_model.Lfm3VlModel.forward(
    input_ids: torch.LongTensor | None = None,
    attention_mask: torch.Tensor | dict[str, torch.Tensor | None] | None = None,
    position_ids: torch.LongTensor | None = None,
    past_key_values: transformers.cache_utils.Cache | None = None,
    inputs_embeds: torch.FloatTensor | None = None,
    pixel_values: torch.FloatTensor | None = None,
    image_grid_thw: torch.LongTensor | None = None,
    pixel_values_videos: torch.FloatTensor | None = None,
    video_grid_thw: torch.LongTensor | None = None,
    use_cache: bool | None = None,
    mm_encoder_outputs: dict[str, transformers.modeling_outputs.BaseModelOutputWithPooling] | None = None,
    kwargs: transformers.processing_utils.Unpack[transformers.utils.TransformersKwargs] = {}
) -> transformers.modeling_outputs.BaseModelOutputWithPast
```

Encode any images and videos, splice them into the text, and run the decoder.

**Parameters:**

**`input_ids`** `torch.LongTensor | None` — default: None

Long Tensor of shape \[batch, sequence]. Exclusive with `inputs_embeds`.

---

**`attention_mask`** `torch.Tensor | dict[str, torch.Tensor | None] | None` — default: None

Optional padding Tensor of shape \[batch, key_sequence], where
`key_sequence` also counts cached tokens.

---

**`position_ids`** `torch.LongTensor | None` — default: None

Optional long Tensor of shape \[batch, sequence].

---

**`past_key_values`** `Cache | None` — default: None

Optional generation cache, updated in place.

---

**`inputs_embeds`** `torch.FloatTensor | None` — default: None

Tensor of shape \[batch, sequence, hidden].

---

**`pixel_values`** `torch.FloatTensor | None` — default: None

Image patches, Tensor of shape
\[patches, in_channels \* temporal_patch_size \* patch_size\*\*2].

---

**`image_grid_thw`** `torch.LongTensor | None` — default: None

Long Tensor of shape \[images, 3], each patch grid as (frames, height, width).

---

**`pixel_values_videos`** `torch.FloatTensor | None` — default: None

Video patches, laid out like `pixel_values`.

---

**`video_grid_thw`** `torch.LongTensor | None` — default: None

Long Tensor of shape \[videos, 3], laid out like `image_grid_thw`.

---

**`use_cache`** `bool | None` — default: None

Whether to create and return a generation cache.

---

**`mm_encoder_outputs`** `dict[str, BaseModelOutputWithPooling] | None` — default: None

Results of `get_image_features` and `get_video_features`
under `"image"` and `"video"`, as generation precomputes them. Exclusive
with the pixel inputs.

---

**`**kwargs`** `Unpack[TransformersKwargs]` — default: \{}

Hugging Face output-capture and attention arguments for the decoder.

---

**Returns:** `BaseModelOutputWithPast`

Output whose `last_hidden_state` is a Tensor of shape \[batch, sequence, hidden].

```python
nemo_automodel.components.models.lfm3.vl_model.Lfm3VlModel.get_image_features(
    pixel_values: torch.Tensor,
    image_grid_thw: torch.Tensor
) -> transformers.modeling_outputs.BaseModelOutputWithPooling
```

Encode images into language-model tokens.

**Parameters:**

**`pixel_values`** `torch.Tensor`

Tensor of shape \[patches, in_channels \* temporal_patch_size \* patch_size\*\*2].

---

**`image_grid_thw`** `torch.Tensor`

Long Tensor of shape \[images, 3] giving each patch grid as
(frames, height, width).

---

**Returns:** `BaseModelOutputWithPooling`

Output whose `last_hidden_state` is the vision Tensor of shape

```python
nemo_automodel.components.models.lfm3.vl_model.Lfm3VlModel.get_input_embeddings() -> torch.nn.Embedding
```

Return the token embedding table.

```python
nemo_automodel.components.models.lfm3.vl_model.Lfm3VlModel.get_video_features(
    pixel_values_videos: torch.Tensor,
    video_grid_thw: torch.Tensor
) -> transformers.modeling_outputs.BaseModelOutputWithPooling
```

Encode videos into language-model tokens; same layouts as `get_image_features`, per video.

```python
class nemo_automodel.components.models.lfm3.vl_model.Lfm3VlMultiModalProjector(
    config: nemo_automodel.components.models.lfm3.config.Lfm3VlConfig
)
```

**Bases:** `Module`

Normalize vision tokens, merge each `spatial_merge_size` square of them, and map to the text width.

**`act`** `= ACT2FN[config.projector_hidden_act]`

---

**`layer_norm`** `= nn.LayerNorm(vision.hidden_size, eps=1e-06)`

---

**`linear_1`**

---

**`linear_2`**

---

**`merged_size`**

---

```python
nemo_automodel.components.models.lfm3.vl_model.Lfm3VlMultiModalProjector.forward(
    image_features: torch.Tensor
) -> torch.Tensor
```

Project vision tokens to language tokens.

**Parameters:**

**`image_features`** `torch.Tensor`

Tensor of shape \[patches, vision_hidden], where consecutive runs
of `spatial_merge_size**2` patches form one merge block. Cast to the
projector's dtype: a frozen vision tower may be stored in lower precision.

---

**Returns:** `torch.Tensor`

Tensor of shape \[patches / spatial_merge_size\*\*2, text_hidden].

```python
class nemo_automodel.components.models.lfm3.vl_model.Lfm3VlPreTrainedModel()
```

**Bases:** `PreTrainedModel`

Weight initialization and loading shared by the LFM3-VL models.

**`_no_split_modules`** `= ['Lfm3VlVisionLayer']`

---

**`_skip_keys_device_placement`** `= ['past_key_values']`

---

**`base_model_prefix`** `= 'model'`

---

**`input_modalities`** `= ('image', 'video', 'text')`

---

```python
class nemo_automodel.components.models.lfm3.vl_model.Lfm3VlVisionAttention(
    config: nemo_automodel.components.models.lfm3.config.Lfm3VlVisionConfig,
    layer_idx: int
)
```

**Bases:** `Lfm2Attention`

LFM2 grouped-query attention made bidirectional, attending within one frame at a time.

```python
nemo_automodel.components.models.lfm3.vl_model.Lfm3VlVisionAttention.forward(
    hidden_states: torch.Tensor,
    position_embeddings: tuple[torch.Tensor, torch.Tensor],
    frame_lengths: list[int]
) -> torch.Tensor
```

Attend among the patches of each frame.

**Parameters:**

**`hidden_states`** `torch.Tensor`

Tensor of shape \[patches, hidden], the frames of every image
and video concatenated.

---

**`position_embeddings`** `tuple[torch.Tensor, torch.Tensor]`

Pair of float32 cosine and sine Tensors, each of shape
\[patches, head_dim].

---

**`frame_lengths`** `list[int]`

Patch count of each frame, in order; sums to `patches`.

---

**Returns:** `torch.Tensor`

Tensor of shape \[patches, hidden].

```python
class nemo_automodel.components.models.lfm3.vl_model.Lfm3VlVisionLayer(
    config: nemo_automodel.components.models.lfm3.config.Lfm3VlVisionConfig,
    layer_idx: int
)
```

**Bases:** `GradientCheckpointingLayer`

Pre-norm operator (per-frame attention or 2D convolution) followed by a pre-norm MLP.

**`conv`** `= Lfm3VlVisionShortConv(config)`

---

**`feed_forward`** `= Lfm2MoeMLP(config)`

---

**`ffn_norm`**

---

**`is_attention_layer`** `= config.layer_types[layer_idx] != 'conv'`

---

**`operator_norm`**

---

**`self_attn`** `= Lfm3VlVisionAttention(config, layer_idx)`

---

```python
nemo_automodel.components.models.lfm3.vl_model.Lfm3VlVisionLayer.forward(
    hidden_states: torch.Tensor,
    position_embeddings: tuple[torch.Tensor, torch.Tensor],
    grid_thw: list[list[int]],
    frame_lengths: list[int]
) -> torch.Tensor
```

Run one vision layer.

**Parameters:**

**`hidden_states`** `torch.Tensor`

Tensor of shape \[patches, hidden].

---

**`position_embeddings`** `tuple[torch.Tensor, torch.Tensor]`

Pair of float32 cosine and sine Tensors, each of shape
\[patches, head_dim].

---

**`grid_thw`** `list[list[int]]`

Per image or video, its patch grid as (frames, height, width).

---

**`frame_lengths`** `list[int]`

Patch count of each frame, in order; sums to `patches`.

---

**Returns:** `torch.Tensor`

Tensor of shape \[patches, hidden].

```python
class nemo_automodel.components.models.lfm3.vl_model.Lfm3VlVisionModel(
    config: nemo_automodel.components.models.lfm3.config.Lfm3VlVisionConfig
)
```

**Bases:** [Lfm3VlPreTrainedModel](#nemo_automodel-components-models-lfm3-vl_model-Lfm3VlPreTrainedModel)

LFM3-VL vision encoder over the flattened patches of images and video frames.

**`embedding_norm`**

---

**`input_modalities`** `= ('image', 'video')`

---

**`layers`**

---

**`patchifier`**

---

```python
nemo_automodel.components.models.lfm3.vl_model.Lfm3VlVisionModel.forward(
    pixel_values: torch.Tensor,
    grid_thw: torch.Tensor
) -> torch.Tensor
```

Encode patches.

**Parameters:**

**`pixel_values`** `torch.Tensor`

Tensor of shape \[patches, in_channels \* temporal_patch_size \* patch_size\*\*2],
the flattened patches of every image or video. Within a frame they are
ordered merge block by merge block.

---

**`grid_thw`** `torch.Tensor`

Long Tensor of shape \[items, 3] giving each image's or video's
patch grid as (frames, height, width).

---

**Returns:** `torch.Tensor`

Tensor of shape \[patches, hidden] in the input order.

```python
class nemo_automodel.components.models.lfm3.vl_model.Lfm3VlVisionShortConv(
    config: nemo_automodel.components.models.lfm3.config.Lfm3VlVisionConfig
)
```

**Bases:** `Module`

Gated depthwise 2D convolution over the patch grid of each frame.

**`in_proj`**

---

**`merge_size`** `= config.spatial_merge_size`

---

**`out_proj`**

---

**`spatial_conv`**

---

```python
nemo_automodel.components.models.lfm3.vl_model.Lfm3VlVisionShortConv.forward(
    hidden_states: torch.Tensor,
    grid_thw: list[list[int]]
) -> torch.Tensor
```

Gate, convolve each frame over its patch grid, and gate again.

**Parameters:**

**`hidden_states`** `torch.Tensor`

Tensor of shape \[patches, hidden]. Within a frame the patches
are ordered merge block by merge block, as the processor emits them.

---

**`grid_thw`** `list[list[int]]`

Per image or video, its patch grid as (frames, height, width).

---

**Returns:** `torch.Tensor`

Tensor of shape \[patches, hidden] in the input order.

```python
nemo_automodel.components.models.lfm3.vl_model.axial_rotary_factors(
    position_ids: torch.Tensor,
    theta: float,
    head_dim: int
) -> tuple[torch.Tensor, torch.Tensor]
```

Build 2D rotary factors in float32: half of each head rotates by row, half by column.

**Parameters:**

**`position_ids`** `torch.Tensor`

Long Tensor of shape \[patches, 2] holding each patch's (row, column).

---

**`theta`** `float`

RoPE base.

---

**`head_dim`** `int`

Channels per attention head.

---

**Returns:** `tuple[torch.Tensor, torch.Tensor]`

Pair of float32 cosine and sine Tensors, each of shape \[patches, head_dim].

```python
nemo_automodel.components.models.lfm3.vl_model.ModelClass = Lfm3VlForConditionalGeneration
```