> This page is for version Latest.
> For other versions, use one of these documentation indexes:
> - Nightly (default): https://docs.nvidia.com/nemo/automodel/nightly/llms.txt
> - Latest: https://docs.nvidia.com/nemo/automodel/latest/llms.txt
> - 0.7.0 · 26.10: https://docs.nvidia.com/nemo/automodel/v0.7/llms.txt
> - 0.5.0 · 26.06: https://docs.nvidia.com/nemo/automodel/v0.5/llms.txt
> - 0.4.0 · 26.04: https://docs.nvidia.com/nemo/automodel/v0.4/llms.txt

> For clean Markdown of any page, append .md to the page URL.
> For a complete documentation index, see https://docs.nvidia.com/nemo/automodel/llms.txt.

# nemo_automodel.components.models.lfm3.model

LFM3 causal LM built on the LFM2 and Qwen3 blocks of the pinned Transformers release.

## Module Contents

### Classes

| Name                                                                                      | Description                                                                                          |
| ----------------------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------- |
| [`Lfm3Attention`](#nemo_automodel-components-models-lfm3-model-Lfm3Attention)             | Qwen3 grouped-query attention with a sigmoid gate on each head's output.                             |
| [`Lfm3DecoderLayer`](#nemo_automodel-components-models-lfm3-model-Lfm3DecoderLayer)       | Pre-norm operator (attention or convolution) and MLP, each normalized again before its residual add. |
| [`Lfm3ForCausalLM`](#nemo_automodel-components-models-lfm3-model-Lfm3ForCausalLM)         | LFM3 with a vocabulary head, loadable from LiquidAI's released checkpoints.                          |
| [`Lfm3Model`](#nemo_automodel-components-models-lfm3-model-Lfm3Model)                     | LFM3 decoder: token embeddings, the mixed convolution and attention stack, and a final norm.         |
| [`Lfm3PreTrainedModel`](#nemo_automodel-components-models-lfm3-model-Lfm3PreTrainedModel) | Weight initialization and loading shared by the LFM3 models.                                         |
| [`Lfm3ShortConv`](#nemo_automodel-components-models-lfm3-model-Lfm3ShortConv)             | LFM2 gated short convolution, always run in PyTorch.                                                 |

### Functions

| Name                                                                            | Description                                                                     |
| ------------------------------------------------------------------------------- | ------------------------------------------------------------------------------- |
| [`rotary_factors`](#nemo_automodel-components-models-lfm3-model-rotary_factors) | Build one attention type's rotary factors in float32, straight from the config. |

### Data

[`ModelClass`](#nemo_automodel-components-models-lfm3-model-ModelClass)

[`check_model_inputs`](#nemo_automodel-components-models-lfm3-model-check_model_inputs)

### API

```python
class nemo_automodel.components.models.lfm3.model.Lfm3Attention(
    config: nemo_automodel.components.models.lfm3.config.Lfm3Config,
    layer_idx: int
)
```

**Bases:** `Qwen3Attention`

Qwen3 grouped-query attention with a sigmoid gate on each head's output.

**`gate_proj`**

---

```python
nemo_automodel.components.models.lfm3.model.Lfm3Attention.forward(
    hidden_states: torch.Tensor,
    position_embeddings: tuple[torch.Tensor, torch.Tensor],
    attention_mask: torch.Tensor | None,
    past_key_values: transformers.cache_utils.Cache | None = None,
    kwargs: transformers.processing_utils.Unpack[transformers.utils.TransformersKwargs] = {}
) -> tuple[torch.Tensor, torch.Tensor | None]
```

Attend over the sequence, then scale each head by its gate.

**Parameters:**

**`hidden_states`** `torch.Tensor`

Tensor of shape \[batch, sequence, hidden].

---

**`position_embeddings`** `tuple[torch.Tensor, torch.Tensor]`

Pair of float32 cosine and sine Tensors, each of
shape \[batch, sequence, head_dim].

---

**`attention_mask`** `torch.Tensor | None`

Optional Tensor of shape \[batch, 1, sequence, key_sequence],
additive or boolean as the attention backend expects.

---

**`past_key_values`** `Cache | None` — default: None

Optional generation cache, updated in place.

---

**`**kwargs`** `Unpack[TransformersKwargs]` — default: \{}

Arguments for the Hugging Face attention dispatcher.

---

**Returns:** `torch.Tensor`

Pair of the output Tensor of shape \[batch, sequence, hidden] and, for the

```python
class nemo_automodel.components.models.lfm3.model.Lfm3DecoderLayer(
    config: nemo_automodel.components.models.lfm3.config.Lfm3Config,
    layer_idx: int
)
```

**Bases:** `GradientCheckpointingLayer`

Pre-norm operator (attention or convolution) and MLP, each normalized again before its residual add.

**`conv`** `= Lfm3ShortConv(config, layer_idx)`

---

**`feed_forward`** `= Lfm2MoeMLP(config)`

---

**`ffn_norm`**

---

**`is_attention_layer`** `= config.layer_types[layer_idx] != 'conv'`

---

**`operator_norm`**

---

**`post_ffn_norm`**

---

**`post_operator_norm`**

---

**`self_attn`** `= Lfm3Attention(config, layer_idx)`

---

```python
nemo_automodel.components.models.lfm3.model.Lfm3DecoderLayer.forward(
    hidden_states: torch.Tensor,
    position_embeddings: tuple[torch.Tensor, torch.Tensor] | None = None,
    attention_mask: torch.Tensor | None = None,
    past_key_values: transformers.cache_utils.Cache | None = None,
    kwargs: transformers.processing_utils.Unpack[transformers.utils.TransformersKwargs] = {}
) -> torch.Tensor
```

Run one decoder layer.

**Parameters:**

**`hidden_states`** `torch.Tensor`

Tensor of shape \[batch, sequence, hidden].

---

**`position_embeddings`** `tuple[torch.Tensor, torch.Tensor] | None` — default: None

For attention layers, a pair of float32 cosine and
sine Tensors, each of shape \[batch, sequence, head_dim].

---

**`attention_mask`** `torch.Tensor | None` — default: None

Attention layers take a Tensor of shape
\[batch, 1, sequence, key_sequence]; convolution layers take a padding
Tensor of shape \[batch, sequence]. Either may be `None`.

---

**`past_key_values`** `Cache | None` — default: None

Optional generation cache, updated in place.

---

**`**kwargs`** `Unpack[TransformersKwargs]` — default: \{}

Arguments for the Hugging Face attention dispatcher.

---

**Returns:** `torch.Tensor`

Tensor of shape \[batch, sequence, hidden].

```python
class nemo_automodel.components.models.lfm3.model.Lfm3ForCausalLM(
    config: nemo_automodel.components.models.lfm3.config.Lfm3Config
)
```

**Bases:** [HFCheckpointingMixin](/nemo/automodel/latest/nemo-automodel/nemo_automodel/components/models/common/hf_checkpointing_mixin#nemo_automodel-components-models-common-hf_checkpointing_mixin-HFCheckpointingMixin), [Lfm3PreTrainedModel](#nemo_automodel-components-models-lfm3-model-Lfm3PreTrainedModel), `GenerationMixin`

LFM3 with a vocabulary head, loadable from LiquidAI's released checkpoints.

**`_tied_weights_keys`** `= {'lm_head.weight': 'model.embed_tokens.weight'}`

---

**`lm_head`**

---

**`model`** `= Lfm3Model(config)`

---

**`tie_word_embeddings_support`** `TieSupport = TieSupport.BOTH`

---

```python
nemo_automodel.components.models.lfm3.model.Lfm3ForCausalLM.forward(
    input_ids: torch.LongTensor | None = None,
    attention_mask: torch.Tensor | dict[str, torch.Tensor | None] | None = None,
    position_ids: torch.LongTensor | None = None,
    past_key_values: transformers.cache_utils.Cache | None = None,
    inputs_embeds: torch.FloatTensor | None = None,
    labels: torch.LongTensor | None = None,
    use_cache: bool | None = None,
    logits_to_keep: int | torch.Tensor = 0,
    kwargs: transformers.processing_utils.Unpack[transformers.utils.TransformersKwargs] = {}
) -> transformers.modeling_outputs.CausalLMOutputWithPast
```

Predict next-token logits and, given labels, the causal LM loss.

**Parameters:**

**`input_ids`** `torch.LongTensor | None` — default: None

Long Tensor of shape \[batch, sequence]. Exclusive with `inputs_embeds`.

---

**`attention_mask`** `torch.Tensor | dict[str, torch.Tensor | None] | None` — default: None

Optional padding Tensor of shape \[batch, key_sequence], where
`key_sequence` also counts cached tokens.

---

**`position_ids`** `torch.LongTensor | None` — default: None

Optional long Tensor of shape \[batch, sequence].

---

**`past_key_values`** `Cache | None` — default: None

Optional generation cache, updated in place.

---

**`inputs_embeds`** `torch.FloatTensor | None` — default: None

Tensor of shape \[batch, sequence, hidden].

---

**`labels`** `torch.LongTensor | None` — default: None

Optional long Tensor of shape \[batch, sequence]; `-100` is ignored.
Shifted internally.

---

**`use_cache`** `bool | None` — default: None

Whether to create and return a generation cache.

---

**`logits_to_keep`** `int | torch.Tensor` — default: 0

Project only the last `logits_to_keep` positions (0 keeps
all), or the positions in a long Tensor of shape \[kept].

---

**`**kwargs`** `Unpack[TransformersKwargs]` — default: \{}

Hugging Face output-capture, attention and loss arguments.

---

**Returns:** `CausalLMOutputWithPast`

Output whose `logits` is a Tensor of shape \[batch, kept, vocab] and whose

```python
nemo_automodel.components.models.lfm3.model.Lfm3ForCausalLM.get_input_embeddings() -> torch.nn.Embedding
```

Return the token embedding table.

```python
nemo_automodel.components.models.lfm3.model.Lfm3ForCausalLM.get_output_embeddings() -> torch.nn.Linear
```

Return the vocabulary head.

```python
nemo_automodel.components.models.lfm3.model.Lfm3ForCausalLM.set_input_embeddings(
    value: torch.nn.Embedding
) -> None
```

Replace the token embedding table and retie the head.

```python
nemo_automodel.components.models.lfm3.model.Lfm3ForCausalLM.tie_weights(
    kwargs: bool | set[str] | None = {}
) -> None
```

Share the vocabulary head with the token embeddings when the config asks for it.

```python
class nemo_automodel.components.models.lfm3.model.Lfm3Model(
    config: nemo_automodel.components.models.lfm3.config.Lfm3Config
)
```

**Bases:** [Lfm3PreTrainedModel](#nemo_automodel-components-models-lfm3-model-Lfm3PreTrainedModel)

LFM3 decoder: token embeddings, the mixed convolution and attention stack, and a final norm.

**`embed_tokens`**

---

**`embedding_norm`**

---

**`layers`**

---

```python
nemo_automodel.components.models.lfm3.model.Lfm3Model._layer_masks(
    inputs_embeds: torch.Tensor,
    attention_mask: torch.Tensor | None,
    past_key_values: transformers.cache_utils.Cache | None,
    position_ids: torch.Tensor
) -> dict[str, torch.Tensor | None]
```

Build one mask per layer type present in the model.

Each attention mask is sized from the first cache layer of its own type:
with a leading convolution layer the mask helpers would otherwise pick
whichever attention layer comes first.

**Parameters:**

**`inputs_embeds`** `torch.Tensor`

Tensor of shape \[batch, sequence, hidden].

---

**`attention_mask`** `torch.Tensor | None`

Optional padding Tensor of shape \[batch, key_sequence],
or an already prepared Tensor of shape \[batch, 1, sequence, key_sequence].

---

**`past_key_values`** `Cache | None`

Optional generation cache.

---

**`position_ids`** `torch.Tensor`

Long Tensor of shape \[batch, sequence].

---

**Returns:** `dict[str, torch.Tensor | None]`

Mapping from layer type to its mask. Attention types map to a Tensor of

```python
nemo_automodel.components.models.lfm3.model.Lfm3Model.forward(
    input_ids: torch.LongTensor | None = None,
    attention_mask: torch.Tensor | dict[str, torch.Tensor | None] | None = None,
    position_ids: torch.LongTensor | None = None,
    past_key_values: transformers.cache_utils.Cache | None = None,
    inputs_embeds: torch.FloatTensor | None = None,
    use_cache: bool | None = None,
    kwargs: transformers.processing_utils.Unpack[transformers.utils.TransformersKwargs] = {}
) -> transformers.modeling_outputs.BaseModelOutputWithPast
```

Run the decoder.

**Parameters:**

**`input_ids`** `torch.LongTensor | None` — default: None

Long Tensor of shape \[batch, sequence]. Exclusive with `inputs_embeds`.

---

**`attention_mask`** `torch.Tensor | dict[str, torch.Tensor | None] | None` — default: None

Optional padding Tensor of shape \[batch, key_sequence], where
`key_sequence` also counts cached tokens, or the per-layer-type mapping
that `_layer_masks` returns.

---

**`position_ids`** `torch.LongTensor | None` — default: None

Optional long Tensor of shape \[batch, sequence]. Defaults to
consecutive positions continuing from the cache.

---

**`past_key_values`** `Cache | None` — default: None

Optional generation cache, updated in place.

---

**`inputs_embeds`** `torch.FloatTensor | None` — default: None

Tensor of shape \[batch, sequence, hidden].

---

**`use_cache`** `bool | None` — default: None

Whether to create and return a generation cache.

---

**`**kwargs`** `Unpack[TransformersKwargs]` — default: \{}

Hugging Face output-capture and attention arguments.

---

**Returns:** `BaseModelOutputWithPast`

Output whose `last_hidden_state` is a Tensor of shape \[batch, sequence, hidden].

```python
class nemo_automodel.components.models.lfm3.model.Lfm3PreTrainedModel()
```

**Bases:** `PreTrainedModel`

Weight initialization and loading shared by the LFM3 models.

**`_can_record_outputs`**

---

**`_no_split_modules`** `= ['Lfm3DecoderLayer']`

---

**`_skip_keys_device_placement`** `= ['past_key_values']`

---

**`base_model_prefix`** `= 'model'`

---

```python
class nemo_automodel.components.models.lfm3.model.Lfm3ShortConv()
```

**Bases:** `Lfm2ShortConv`

LFM2 gated short convolution, always run in PyTorch.

The parent dispatches to Hub kernels that only cover widths 2 to 4; LFM3 uses width 7.

```python
nemo_automodel.components.models.lfm3.model.Lfm3ShortConv.forward(
    hidden_states: torch.Tensor,
    past_key_values: transformers.cache_utils.Cache | None = None,
    attention_mask: torch.Tensor | None = None
) -> torch.Tensor
```

Gate, convolve causally along the sequence, and gate again.

**Parameters:**

**`hidden_states`** `torch.Tensor`

Tensor of shape \[batch, sequence, hidden].

---

**`past_key_values`** `Cache | None` — default: None

Optional generation cache. Its window of earlier gated
inputs is prepended as left context and updated in place.

---

**`attention_mask`** `torch.Tensor | None` — default: None

Optional Tensor of shape \[batch, sequence], nonzero for
real tokens and zero for padding.

---

**Returns:** `torch.Tensor`

Tensor of shape \[batch, sequence, hidden].

```python
nemo_automodel.components.models.lfm3.model.rotary_factors(
    position_ids: torch.Tensor,
    rope: dict[str, str | float],
    head_dim: int
) -> tuple[torch.Tensor, torch.Tensor]
```

Build one attention type's rotary factors in float32, straight from the config.

The first `partial_rotary_factor` share of the frequencies follows the
progression of the full head dimension; the rest are zero, so those
channels pass through unrotated. Nothing is cached, so no model-wide dtype
cast can round the frequencies.

**Parameters:**

**`position_ids`** `torch.Tensor`

Long Tensor of shape \[batch, sequence].

---

**`rope`** `dict[str, str | float]`

The attention type's entry of `Lfm3Config.rope_parameters`.

---

**`head_dim`** `int`

Channels per attention head.

---

**Returns:** `tuple[torch.Tensor, torch.Tensor]`

Pair of float32 cosine and sine Tensors, each of shape \[batch, sequence, head_dim].

```python
nemo_automodel.components.models.lfm3.model.ModelClass = Lfm3ForCausalLM
```

```python
nemo_automodel.components.models.lfm3.model.check_model_inputs = get_check_model_inputs_decorator()
```