Skip to navigation

nemo_automodel.components.models.lfm3.model

View as MarkdownOpen in Claude

LFM3 causal LM built on the LFM2 and Qwen3 blocks of the pinned Transformers release.

Module Contents

Classes

NameDescription
Lfm3AttentionQwen3 grouped-query attention with a sigmoid gate on each head’s output.
Lfm3DecoderLayerPre-norm operator (attention or convolution) and MLP, each normalized again before its residual add.
Lfm3ForCausalLMLFM3 with a vocabulary head, loadable from LiquidAI’s released checkpoints.
Lfm3ModelLFM3 decoder: token embeddings, the mixed convolution and attention stack, and a final norm.
Lfm3PreTrainedModelWeight initialization and loading shared by the LFM3 models.
Lfm3ShortConvLFM2 gated short convolution, always run in PyTorch.

Functions

NameDescription
rotary_factorsBuild one attention type’s rotary factors in float32, straight from the config.

Data

ModelClass

check_model_inputs

API

class nemo_automodel.components.models.lfm3.model.Lfm3Attention(
layer_idx: int
)

Bases: Qwen3Attention

Qwen3 grouped-query attention with a sigmoid gate on each head’s output.

gate_proj
nemo_automodel.components.models.lfm3.model.Lfm3Attention.forward(
hidden_states: torch.Tensor,
position_embeddings: tuple[torch.Tensor, torch.Tensor],
attention_mask: torch.Tensor | None,
past_key_values: transformers.cache_utils.Cache | None = None,
kwargs: transformers.processing_utils.Unpack[transformers.utils.TransformersKwargs] = {}
) -> tuple[torch.Tensor, torch.Tensor | None]

Attend over the sequence, then scale each head by its gate.

Parameters:

hidden_states
torch.Tensor

Tensor of shape [batch, sequence, hidden].

position_embeddings
tuple[torch.Tensor, torch.Tensor]

Pair of float32 cosine and sine Tensors, each of shape [batch, sequence, head_dim].

attention_mask
torch.Tensor | None

Optional Tensor of shape [batch, 1, sequence, key_sequence], additive or boolean as the attention backend expects.

past_key_values
Cache | NoneDefaults to None

Optional generation cache, updated in place.

**kwargs
Unpack[TransformersKwargs]Defaults to {}

Arguments for the Hugging Face attention dispatcher.

Returns: torch.Tensor

Pair of the output Tensor of shape [batch, sequence, hidden] and, for the

class nemo_automodel.components.models.lfm3.model.Lfm3DecoderLayer(
layer_idx: int
)

Bases: GradientCheckpointingLayer

Pre-norm operator (attention or convolution) and MLP, each normalized again before its residual add.

conv
= Lfm3ShortConv(config, layer_idx)
feed_forward
= Lfm2MoeMLP(config)
ffn_norm
is_attention_layer
= config.layer_types[layer_idx] != 'conv'
operator_norm
post_ffn_norm
post_operator_norm
self_attn
= Lfm3Attention(config, layer_idx)
nemo_automodel.components.models.lfm3.model.Lfm3DecoderLayer.forward(
hidden_states: torch.Tensor,
position_embeddings: tuple[torch.Tensor, torch.Tensor] | None = None,
attention_mask: torch.Tensor | None = None,
past_key_values: transformers.cache_utils.Cache | None = None,
kwargs: transformers.processing_utils.Unpack[transformers.utils.TransformersKwargs] = {}
) -> torch.Tensor

Run one decoder layer.

Parameters:

hidden_states
torch.Tensor

Tensor of shape [batch, sequence, hidden].

position_embeddings
tuple[torch.Tensor, torch.Tensor] | NoneDefaults to None

For attention layers, a pair of float32 cosine and sine Tensors, each of shape [batch, sequence, head_dim].

attention_mask
torch.Tensor | NoneDefaults to None

Attention layers take a Tensor of shape [batch, 1, sequence, key_sequence]; convolution layers take a padding Tensor of shape [batch, sequence]. Either may be None.

past_key_values
Cache | NoneDefaults to None

Optional generation cache, updated in place.

**kwargs
Unpack[TransformersKwargs]Defaults to {}

Arguments for the Hugging Face attention dispatcher.

Returns: torch.Tensor

Tensor of shape [batch, sequence, hidden].

class nemo_automodel.components.models.lfm3.model.Lfm3ForCausalLM(
)

Bases: HFCheckpointingMixin, Lfm3PreTrainedModel, GenerationMixin

LFM3 with a vocabulary head, loadable from LiquidAI’s released checkpoints.

_tied_weights_keys
= {'lm_head.weight': 'model.embed_tokens.weight'}
lm_head
model
= Lfm3Model(config)
tie_word_embeddings_support
TieSupport = TieSupport.BOTH
nemo_automodel.components.models.lfm3.model.Lfm3ForCausalLM.forward(
input_ids: torch.LongTensor | None = None,
attention_mask: torch.Tensor | dict[str, torch.Tensor | None] | None = None,
position_ids: torch.LongTensor | None = None,
past_key_values: transformers.cache_utils.Cache | None = None,
inputs_embeds: torch.FloatTensor | None = None,
labels: torch.LongTensor | None = None,
use_cache: bool | None = None,
logits_to_keep: int | torch.Tensor = 0,
kwargs: transformers.processing_utils.Unpack[transformers.utils.TransformersKwargs] = {}
) -> transformers.modeling_outputs.CausalLMOutputWithPast

Predict next-token logits and, given labels, the causal LM loss.

Parameters:

input_ids
torch.LongTensor | NoneDefaults to None

Long Tensor of shape [batch, sequence]. Exclusive with inputs_embeds.

attention_mask
torch.Tensor | dict[str, torch.Tensor | None] | NoneDefaults to None

Optional padding Tensor of shape [batch, key_sequence], where key_sequence also counts cached tokens.

position_ids
torch.LongTensor | NoneDefaults to None

Optional long Tensor of shape [batch, sequence].

past_key_values
Cache | NoneDefaults to None

Optional generation cache, updated in place.

inputs_embeds
torch.FloatTensor | NoneDefaults to None

Tensor of shape [batch, sequence, hidden].

labels
torch.LongTensor | NoneDefaults to None

Optional long Tensor of shape [batch, sequence]; -100 is ignored. Shifted internally.

use_cache
bool | NoneDefaults to None

Whether to create and return a generation cache.

logits_to_keep
int | torch.TensorDefaults to 0

Project only the last logits_to_keep positions (0 keeps all), or the positions in a long Tensor of shape [kept].

**kwargs
Unpack[TransformersKwargs]Defaults to {}

Hugging Face output-capture, attention and loss arguments.

Returns: CausalLMOutputWithPast

Output whose logits is a Tensor of shape [batch, kept, vocab] and whose

nemo_automodel.components.models.lfm3.model.Lfm3ForCausalLM.get_input_embeddings() -> torch.nn.Embedding

Return the token embedding table.

nemo_automodel.components.models.lfm3.model.Lfm3ForCausalLM.get_output_embeddings() -> torch.nn.Linear

Return the vocabulary head.

nemo_automodel.components.models.lfm3.model.Lfm3ForCausalLM.set_input_embeddings(
value: torch.nn.Embedding
) -> None

Replace the token embedding table and retie the head.

nemo_automodel.components.models.lfm3.model.Lfm3ForCausalLM.tie_weights(
kwargs: bool | set[str] | None = {}
) -> None

Share the vocabulary head with the token embeddings when the config asks for it.

class nemo_automodel.components.models.lfm3.model.Lfm3Model(
)

Bases: Lfm3PreTrainedModel

LFM3 decoder: token embeddings, the mixed convolution and attention stack, and a final norm.

embed_tokens
embedding_norm
layers
nemo_automodel.components.models.lfm3.model.Lfm3Model._layer_masks(
inputs_embeds: torch.Tensor,
attention_mask: torch.Tensor | None,
past_key_values: transformers.cache_utils.Cache | None,
position_ids: torch.Tensor
) -> dict[str, torch.Tensor | None]

Build one mask per layer type present in the model.

Each attention mask is sized from the first cache layer of its own type: with a leading convolution layer the mask helpers would otherwise pick whichever attention layer comes first.

Parameters:

inputs_embeds
torch.Tensor

Tensor of shape [batch, sequence, hidden].

attention_mask
torch.Tensor | None

Optional padding Tensor of shape [batch, key_sequence], or an already prepared Tensor of shape [batch, 1, sequence, key_sequence].

past_key_values
Cache | None

Optional generation cache.

position_ids
torch.Tensor

Long Tensor of shape [batch, sequence].

Returns: dict[str, torch.Tensor | None]

Mapping from layer type to its mask. Attention types map to a Tensor of

nemo_automodel.components.models.lfm3.model.Lfm3Model.forward(
input_ids: torch.LongTensor | None = None,
attention_mask: torch.Tensor | dict[str, torch.Tensor | None] | None = None,
position_ids: torch.LongTensor | None = None,
past_key_values: transformers.cache_utils.Cache | None = None,
inputs_embeds: torch.FloatTensor | None = None,
use_cache: bool | None = None,
kwargs: transformers.processing_utils.Unpack[transformers.utils.TransformersKwargs] = {}
) -> transformers.modeling_outputs.BaseModelOutputWithPast

Run the decoder.

Parameters:

input_ids
torch.LongTensor | NoneDefaults to None

Long Tensor of shape [batch, sequence]. Exclusive with inputs_embeds.

attention_mask
torch.Tensor | dict[str, torch.Tensor | None] | NoneDefaults to None

Optional padding Tensor of shape [batch, key_sequence], where key_sequence also counts cached tokens, or the per-layer-type mapping that _layer_masks returns.

position_ids
torch.LongTensor | NoneDefaults to None

Optional long Tensor of shape [batch, sequence]. Defaults to consecutive positions continuing from the cache.

past_key_values
Cache | NoneDefaults to None

Optional generation cache, updated in place.

inputs_embeds
torch.FloatTensor | NoneDefaults to None

Tensor of shape [batch, sequence, hidden].

use_cache
bool | NoneDefaults to None

Whether to create and return a generation cache.

**kwargs
Unpack[TransformersKwargs]Defaults to {}

Hugging Face output-capture and attention arguments.

Returns: BaseModelOutputWithPast

Output whose last_hidden_state is a Tensor of shape [batch, sequence, hidden].

class nemo_automodel.components.models.lfm3.model.Lfm3PreTrainedModel()

Bases: PreTrainedModel

Weight initialization and loading shared by the LFM3 models.

_can_record_outputs
_no_split_modules
= ['Lfm3DecoderLayer']
_skip_keys_device_placement
= ['past_key_values']
base_model_prefix
= 'model'
class nemo_automodel.components.models.lfm3.model.Lfm3ShortConv()

Bases: Lfm2ShortConv

LFM2 gated short convolution, always run in PyTorch.

The parent dispatches to Hub kernels that only cover widths 2 to 4; LFM3 uses width 7.

nemo_automodel.components.models.lfm3.model.Lfm3ShortConv.forward(
hidden_states: torch.Tensor,
past_key_values: transformers.cache_utils.Cache | None = None,
attention_mask: torch.Tensor | None = None
) -> torch.Tensor

Gate, convolve causally along the sequence, and gate again.

Parameters:

hidden_states
torch.Tensor

Tensor of shape [batch, sequence, hidden].

past_key_values
Cache | NoneDefaults to None

Optional generation cache. Its window of earlier gated inputs is prepended as left context and updated in place.

attention_mask
torch.Tensor | NoneDefaults to None

Optional Tensor of shape [batch, sequence], nonzero for real tokens and zero for padding.

Returns: torch.Tensor

Tensor of shape [batch, sequence, hidden].

nemo_automodel.components.models.lfm3.model.rotary_factors(
position_ids: torch.Tensor,
rope: dict[str, str | float],
head_dim: int
) -> tuple[torch.Tensor, torch.Tensor]

Build one attention type’s rotary factors in float32, straight from the config.

The first partial_rotary_factor share of the frequencies follows the progression of the full head dimension; the rest are zero, so those channels pass through unrotated. Nothing is cached, so no model-wide dtype cast can round the frequencies.

Parameters:

position_ids
torch.Tensor

Long Tensor of shape [batch, sequence].

rope
dict[str, str | float]

The attention type’s entry of Lfm3Config.rope_parameters.

head_dim
int

Channels per attention head.

Returns: tuple[torch.Tensor, torch.Tensor]

Pair of float32 cosine and sine Tensors, each of shape [batch, sequence, head_dim].

nemo_automodel.components.models.lfm3.model.ModelClass = Lfm3ForCausalLM
nemo_automodel.components.models.lfm3.model.check_model_inputs = get_check_model_inputs_decorator()