nemo_automodel.components.models.lfm3.model
nemo_automodel.components.models.lfm3.model
LFM3 causal LM built on the LFM2 and Qwen3 blocks of the pinned Transformers release.
Module Contents
Classes
Functions
Data
API
Bases: Qwen3Attention
Qwen3 grouped-query attention with a sigmoid gate on each head’s output.
Attend over the sequence, then scale each head by its gate.
Parameters:
Tensor of shape [batch, sequence, hidden].
Pair of float32 cosine and sine Tensors, each of shape [batch, sequence, head_dim].
Optional Tensor of shape [batch, 1, sequence, key_sequence], additive or boolean as the attention backend expects.
Optional generation cache, updated in place.
Arguments for the Hugging Face attention dispatcher.
Returns: torch.Tensor
Pair of the output Tensor of shape [batch, sequence, hidden] and, for the
Bases: GradientCheckpointingLayer
Pre-norm operator (attention or convolution) and MLP, each normalized again before its residual add.
Run one decoder layer.
Parameters:
Tensor of shape [batch, sequence, hidden].
For attention layers, a pair of float32 cosine and sine Tensors, each of shape [batch, sequence, head_dim].
Attention layers take a Tensor of shape
[batch, 1, sequence, key_sequence]; convolution layers take a padding
Tensor of shape [batch, sequence]. Either may be None.
Optional generation cache, updated in place.
Arguments for the Hugging Face attention dispatcher.
Returns: torch.Tensor
Tensor of shape [batch, sequence, hidden].
Bases: HFCheckpointingMixin, Lfm3PreTrainedModel, GenerationMixin
LFM3 with a vocabulary head, loadable from LiquidAI’s released checkpoints.
Predict next-token logits and, given labels, the causal LM loss.
Parameters:
Long Tensor of shape [batch, sequence]. Exclusive with inputs_embeds.
Optional padding Tensor of shape [batch, key_sequence], where
key_sequence also counts cached tokens.
Optional long Tensor of shape [batch, sequence].
Optional generation cache, updated in place.
Tensor of shape [batch, sequence, hidden].
Optional long Tensor of shape [batch, sequence]; -100 is ignored.
Shifted internally.
Whether to create and return a generation cache.
Project only the last logits_to_keep positions (0 keeps
all), or the positions in a long Tensor of shape [kept].
Hugging Face output-capture, attention and loss arguments.
Returns: CausalLMOutputWithPast
Output whose logits is a Tensor of shape [batch, kept, vocab] and whose
Return the token embedding table.
Return the vocabulary head.
Replace the token embedding table and retie the head.
Share the vocabulary head with the token embeddings when the config asks for it.
Bases: Lfm3PreTrainedModel
LFM3 decoder: token embeddings, the mixed convolution and attention stack, and a final norm.
Build one mask per layer type present in the model.
Each attention mask is sized from the first cache layer of its own type: with a leading convolution layer the mask helpers would otherwise pick whichever attention layer comes first.
Parameters:
Tensor of shape [batch, sequence, hidden].
Optional padding Tensor of shape [batch, key_sequence], or an already prepared Tensor of shape [batch, 1, sequence, key_sequence].
Optional generation cache.
Long Tensor of shape [batch, sequence].
Returns: dict[str, torch.Tensor | None]
Mapping from layer type to its mask. Attention types map to a Tensor of
Run the decoder.
Parameters:
Long Tensor of shape [batch, sequence]. Exclusive with inputs_embeds.
Optional padding Tensor of shape [batch, key_sequence], where
key_sequence also counts cached tokens, or the per-layer-type mapping
that _layer_masks returns.
Optional long Tensor of shape [batch, sequence]. Defaults to consecutive positions continuing from the cache.
Optional generation cache, updated in place.
Tensor of shape [batch, sequence, hidden].
Whether to create and return a generation cache.
Hugging Face output-capture and attention arguments.
Returns: BaseModelOutputWithPast
Output whose last_hidden_state is a Tensor of shape [batch, sequence, hidden].
Bases: PreTrainedModel
Weight initialization and loading shared by the LFM3 models.
Bases: Lfm2ShortConv
LFM2 gated short convolution, always run in PyTorch.
The parent dispatches to Hub kernels that only cover widths 2 to 4; LFM3 uses width 7.
Gate, convolve causally along the sequence, and gate again.
Parameters:
Tensor of shape [batch, sequence, hidden].
Optional generation cache. Its window of earlier gated inputs is prepended as left context and updated in place.
Optional Tensor of shape [batch, sequence], nonzero for real tokens and zero for padding.
Returns: torch.Tensor
Tensor of shape [batch, sequence, hidden].
Build one attention type’s rotary factors in float32, straight from the config.
The first partial_rotary_factor share of the frequencies follows the
progression of the full head dimension; the rest are zero, so those
channels pass through unrotated. Nothing is cached, so no model-wide dtype
cast can round the frequencies.
Parameters:
Long Tensor of shape [batch, sequence].
The attention type’s entry of Lfm3Config.rope_parameters.
Channels per attention head.
Returns: tuple[torch.Tensor, torch.Tensor]
Pair of float32 cosine and sine Tensors, each of shape [batch, sequence, head_dim].