nemo_automodel.components.models.lfm3.model
nemo_automodel.components.models.lfm3.model
LFM3 causal LM built on the LFM2 and Qwen3 blocks of the pinned Transformers release.
Module Contents
Classes
Functions
Data
API
Bases: Qwen3Attention
Qwen3 grouped-query attention with a sigmoid gate on each head’s output.
Attend over the sequence, then scale each head by its gate.
Parameters:
Tensor of shape [batch, sequence, hidden].
Pair of float32 cosine and sine Tensors, each of shape [batch, sequence, head_dim].
Optional Tensor of shape [batch, 1, sequence, key_sequence], additive or boolean as the attention backend expects.
Optional generation cache, updated in place.
Optional packed-document boundaries. They replace attention_mask,
and under context parallelism add the keys held by earlier ranks.
Arguments for the Hugging Face attention dispatcher.
Returns: torch.Tensor
Pair of the output Tensor of shape [batch, sequence, hidden] and, for the
Bases: GradientCheckpointingLayer
Pre-norm operator (attention or convolution) and MLP, each normalized again before its residual add.
Run one decoder layer.
Parameters:
Tensor of shape [batch, sequence, hidden].
For attention layers, a pair of float32 cosine and sine Tensors, each of shape [batch, sequence, head_dim].
Attention layers take a Tensor of shape
[batch, 1, sequence, key_sequence]; convolution layers take a padding
Tensor of shape [batch, sequence]. Either may be None.
Optional generation cache, updated in place.
Optional packed-document boundaries, used in place of attention_mask.
Arguments for the Hugging Face attention dispatcher.
Returns: torch.Tensor
Tensor of shape [batch, sequence, hidden].
Bases: HFCheckpointingMixin, Lfm3PreTrainedModel, GenerationMixin
LFM3 with a vocabulary head, loadable from LiquidAI’s released checkpoints.
Predict next-token logits and, given labels, the causal LM loss.
Parameters:
Long Tensor of shape [batch, sequence]. Exclusive with inputs_embeds.
Optional padding Tensor of shape [batch, key_sequence], where
key_sequence also counts cached tokens.
Optional long Tensor of shape [batch, sequence].
Optional generation cache, updated in place.
Tensor of shape [batch, sequence, hidden].
Optional long Tensor of shape [batch, sequence]; -100 is ignored.
Shifted internally.
Whether to create and return a generation cache.
Project only the last logits_to_keep positions (0 keeps
all), or the positions in a long Tensor of shape [kept].
Hugging Face output-capture, attention and loss arguments.
Returns: CausalLMOutputWithPast
Output whose logits is a Tensor of shape [batch, kept, vocab] and whose
Return the token embedding table.
Return the vocabulary head.
Return the sharder that keeps packed documents apart and splits the sequence across CP ranks.
Parameters:
The recipe batch with token Tensors of shape [batch, sequence]; it is not read or changed here, the sharder reads it when it shards.
Unused; LFM3 shards whole rows.
Returns: dict[str, ContextParallelSharder]
Mapping with the cp_sharder built on shard_documents.
Replace the token embedding table and retie the head.
Share the vocabulary head with the token embeddings when the config asks for it.
Reject batched packing on backends that give the convolutions no document boundaries.
Bases: Lfm3PreTrainedModel
LFM3 decoder: token embeddings, the mixed convolution and attention stack, and a final norm.
Build one mask per layer type present in the model.
Each attention mask is sized from the first cache layer of its own type: with a leading convolution layer the mask helpers would otherwise pick whichever attention layer comes first.
Parameters:
Tensor of shape [batch, sequence, hidden].
Optional padding Tensor of shape [batch, key_sequence], or an already prepared Tensor of shape [batch, 1, sequence, key_sequence].
Optional generation cache.
Long Tensor of shape [batch, sequence].
Returns: dict[str, torch.Tensor | None]
Mapping from layer type to its mask. Attention types map to a Tensor of
Run the decoder.
Parameters:
Long Tensor of shape [batch, sequence]. Exclusive with inputs_embeds.
Optional padding Tensor of shape [batch, key_sequence], where
key_sequence also counts cached tokens, or the per-layer-type mapping
that _layer_masks returns.
Optional long Tensor of shape [batch, sequence]. Defaults to consecutive positions continuing from the cache.
Optional generation cache, updated in place.
Tensor of shape [batch, sequence, hidden].
Whether to create and return a generation cache. Training mode creates none.
Optional int32 Tensor of shape [documents + 1] with the packed-document
boundaries of the flattened [batch * sequence] tokens. Attention and
convolution then stay inside each document, attention_mask is not used
and no generation cache is created; a supplied past_key_values and the
flex_attention backend are rejected.
Length of the longest packed document.
Under context parallelism, how far the first local document reaches into earlier ranks.
Hugging Face output-capture and attention arguments.
Returns: BaseModelOutputWithPast
Output whose last_hidden_state is a Tensor of shape [batch, sequence, hidden].
Bases: PreTrainedModel
Weight initialization and loading shared by the LFM3 models.
Bases: Lfm2ShortConv
LFM2 gated short convolution, always run in PyTorch.
The parent dispatches to Hub kernels that only cover widths 2 to 4; LFM3 uses width 7.
Gate, convolve causally along the sequence, and gate again.
Parameters:
Tensor of shape [batch, sequence, hidden].
Optional generation cache. Its window of earlier gated inputs is prepended as left context and updated in place.
Optional Tensor of shape [batch, sequence], nonzero for real tokens and zero for padding.
Optional packed-document boundaries; the convolution then restarts at every document.
Returns: torch.Tensor
Tensor of shape [batch, sequence, hidden].
Build one attention type’s rotary factors in float32, straight from the config.
The first partial_rotary_factor share of the frequencies follows the
progression of the full head dimension; the rest are zero, so those
channels pass through unrotated. Nothing is cached, so no model-wide dtype
cast can round the frequencies.
Parameters:
Long Tensor of shape [batch, sequence].
The attention type’s entry of Lfm3Config.rope_parameters.
Channels per attention head.
Returns: tuple[torch.Tensor, torch.Tensor]
Pair of float32 cosine and sine Tensors, each of shape [batch, sequence, head_dim].