Skip to navigation

nemo_automodel.components.models.lfm3.vl_model

View as MarkdownOpen in Claude

LFM3-VL: the LFM3 decoder fed by a convolution-and-attention vision encoder through an MLP projector.

Module Contents

Classes

NameDescription
Lfm3VlForConditionalGenerationLFM3-VL with a vocabulary head, loadable from LiquidAI’s released checkpoints.
Lfm3VlModelLFM3 decoder whose image and video placeholder tokens are replaced by projected vision features.
Lfm3VlMultiModalProjectorNormalize vision tokens, merge each spatial_merge_size square of them, and map to the text width.
Lfm3VlPreTrainedModelWeight initialization and loading shared by the LFM3-VL models.
Lfm3VlVisionAttentionLFM2 grouped-query attention made bidirectional, attending within one frame at a time.
Lfm3VlVisionLayerPre-norm operator (per-frame attention or 2D convolution) followed by a pre-norm MLP.
Lfm3VlVisionModelLFM3-VL vision encoder over the flattened patches of images and video frames.
Lfm3VlVisionShortConvGated depthwise 2D convolution over the patch grid of each frame.

Functions

NameDescription
axial_rotary_factorsBuild 2D rotary factors in float32: half of each head rotates by row, half by column.

Data

ModelClass

API

class nemo_automodel.components.models.lfm3.vl_model.Lfm3VlForConditionalGeneration(
)

Bases: HFCheckpointingMixin, Lfm3VlPreTrainedModel, GenerationMixin

LFM3-VL with a vocabulary head, loadable from LiquidAI’s released checkpoints.

_tied_weights_keys
lm_head
model
= Lfm3VlModel(config)
tie_word_embeddings_support
TieSupport = TieSupport.BOTH
nemo_automodel.components.models.lfm3.vl_model.Lfm3VlForConditionalGeneration.forward(
input_ids: torch.LongTensor | None = None,
attention_mask: torch.Tensor | dict[str, torch.Tensor | None] | None = None,
position_ids: torch.LongTensor | None = None,
past_key_values: transformers.cache_utils.Cache | None = None,
inputs_embeds: torch.FloatTensor | None = None,
pixel_values: torch.FloatTensor | None = None,
image_grid_thw: torch.LongTensor | None = None,
pixel_values_videos: torch.FloatTensor | None = None,
video_grid_thw: torch.LongTensor | None = None,
labels: torch.LongTensor | None = None,
use_cache: bool | None = None,
logits_to_keep: int | torch.Tensor = 0,
mm_encoder_outputs: dict[str, transformers.modeling_outputs.BaseModelOutputWithPooling] | None = None,
kwargs: transformers.processing_utils.Unpack[transformers.utils.TransformersKwargs] = {}
) -> transformers.modeling_outputs.CausalLMOutputWithPast

Predict next-token logits over text interleaved with images and videos.

Parameters:

input_ids
torch.LongTensor | NoneDefaults to None

Long Tensor of shape [batch, sequence]. Exclusive with inputs_embeds.

attention_mask
torch.Tensor | dict[str, torch.Tensor | None] | NoneDefaults to None

Optional padding Tensor of shape [batch, key_sequence], where key_sequence also counts cached tokens.

position_ids
torch.LongTensor | NoneDefaults to None

Optional long Tensor of shape [batch, sequence].

past_key_values
Cache | NoneDefaults to None

Optional generation cache, updated in place.

inputs_embeds
torch.FloatTensor | NoneDefaults to None

Tensor of shape [batch, sequence, hidden].

pixel_values
torch.FloatTensor | NoneDefaults to None

Image patches, Tensor of shape [patches, in_channels * temporal_patch_size * patch_size**2].

image_grid_thw
torch.LongTensor | NoneDefaults to None

Long Tensor of shape [images, 3], each patch grid as (frames, height, width).

pixel_values_videos
torch.FloatTensor | NoneDefaults to None

Video patches, laid out like pixel_values.

video_grid_thw
torch.LongTensor | NoneDefaults to None

Long Tensor of shape [videos, 3], laid out like image_grid_thw.

labels
torch.LongTensor | NoneDefaults to None

Optional long Tensor of shape [batch, sequence]; -100 is ignored. Shifted internally.

use_cache
bool | NoneDefaults to None

Whether to create and return a generation cache.

logits_to_keep
int | torch.TensorDefaults to 0

Project only the last logits_to_keep positions (0 keeps all), or the positions in a long Tensor of shape [kept].

mm_encoder_outputs
dict[str, BaseModelOutputWithPooling] | NoneDefaults to None

Precomputed vision features, as generation passes them.

**kwargs
Unpack[TransformersKwargs]Defaults to {}

Hugging Face output-capture, attention and loss arguments.

Returns: CausalLMOutputWithPast

Output whose logits is a Tensor of shape [batch, kept, vocab] and whose

nemo_automodel.components.models.lfm3.vl_model.Lfm3VlForConditionalGeneration.get_input_embeddings() -> torch.nn.Embedding

Return the token embedding table.

nemo_automodel.components.models.lfm3.vl_model.Lfm3VlForConditionalGeneration.get_output_embeddings() -> torch.nn.Linear

Return the vocabulary head.

nemo_automodel.components.models.lfm3.vl_model.Lfm3VlForConditionalGeneration.set_input_embeddings(
value: torch.nn.Embedding
) -> None

Replace the token embedding table and retie the head.

nemo_automodel.components.models.lfm3.vl_model.Lfm3VlForConditionalGeneration.tie_weights(
kwargs: bool | set[str] | None = {}
) -> None

Share the vocabulary head with the token embeddings when the config asks for it.

class nemo_automodel.components.models.lfm3.vl_model.Lfm3VlModel(
)

Bases: Lfm3VlPreTrainedModel

LFM3 decoder whose image and video placeholder tokens are replaced by projected vision features.

language_model
= Lfm3Model(config.text_config)
multi_modal_projector
= Lfm3VlMultiModalProjector(config)
vision_tower
= Lfm3VlVisionModel(config.vision_config)
nemo_automodel.components.models.lfm3.vl_model.Lfm3VlModel._fill_placeholders(
inputs_embeds: torch.Tensor,
input_ids: torch.LongTensor | None,
token_id: int,
features: tuple[torch.Tensor, ...]
) -> torch.Tensor

Write vision features over the embeddings of one modality’s placeholder tokens.

Parameters:

inputs_embeds
torch.Tensor

Tensor of shape [batch, sequence, hidden].

input_ids
torch.LongTensor | None

Long Tensor of shape [batch, sequence] locating the placeholders.

token_id
int

The modality’s placeholder token.

features
tuple[torch.Tensor, ...]

One Tensor of shape [tokens, hidden] per image or video, in the order their placeholders appear row by row.

Returns: torch.Tensor

New Tensor of shape [batch, sequence, hidden]; inputs_embeds is not modified.

nemo_automodel.components.models.lfm3.vl_model.Lfm3VlModel.forward(
input_ids: torch.LongTensor | None = None,
attention_mask: torch.Tensor | dict[str, torch.Tensor | None] | None = None,
position_ids: torch.LongTensor | None = None,
past_key_values: transformers.cache_utils.Cache | None = None,
inputs_embeds: torch.FloatTensor | None = None,
pixel_values: torch.FloatTensor | None = None,
image_grid_thw: torch.LongTensor | None = None,
pixel_values_videos: torch.FloatTensor | None = None,
video_grid_thw: torch.LongTensor | None = None,
use_cache: bool | None = None,
mm_encoder_outputs: dict[str, transformers.modeling_outputs.BaseModelOutputWithPooling] | None = None,
kwargs: transformers.processing_utils.Unpack[transformers.utils.TransformersKwargs] = {}
) -> transformers.modeling_outputs.BaseModelOutputWithPast

Encode any images and videos, splice them into the text, and run the decoder.

Parameters:

input_ids
torch.LongTensor | NoneDefaults to None

Long Tensor of shape [batch, sequence]. Exclusive with inputs_embeds.

attention_mask
torch.Tensor | dict[str, torch.Tensor | None] | NoneDefaults to None

Optional padding Tensor of shape [batch, key_sequence], where key_sequence also counts cached tokens.

position_ids
torch.LongTensor | NoneDefaults to None

Optional long Tensor of shape [batch, sequence].

past_key_values
Cache | NoneDefaults to None

Optional generation cache, updated in place.

inputs_embeds
torch.FloatTensor | NoneDefaults to None

Tensor of shape [batch, sequence, hidden].

pixel_values
torch.FloatTensor | NoneDefaults to None

Image patches, Tensor of shape [patches, in_channels * temporal_patch_size * patch_size**2].

image_grid_thw
torch.LongTensor | NoneDefaults to None

Long Tensor of shape [images, 3], each patch grid as (frames, height, width).

pixel_values_videos
torch.FloatTensor | NoneDefaults to None

Video patches, laid out like pixel_values.

video_grid_thw
torch.LongTensor | NoneDefaults to None

Long Tensor of shape [videos, 3], laid out like image_grid_thw.

use_cache
bool | NoneDefaults to None

Whether to create and return a generation cache.

mm_encoder_outputs
dict[str, BaseModelOutputWithPooling] | NoneDefaults to None

Results of get_image_features and get_video_features under "image" and "video", as generation precomputes them. Exclusive with the pixel inputs.

**kwargs
Unpack[TransformersKwargs]Defaults to {}

Hugging Face output-capture and attention arguments for the decoder.

Returns: BaseModelOutputWithPast

Output whose last_hidden_state is a Tensor of shape [batch, sequence, hidden].

nemo_automodel.components.models.lfm3.vl_model.Lfm3VlModel.get_image_features(
pixel_values: torch.Tensor,
image_grid_thw: torch.Tensor
) -> transformers.modeling_outputs.BaseModelOutputWithPooling

Encode images into language-model tokens.

Parameters:

pixel_values
torch.Tensor

Tensor of shape [patches, in_channels * temporal_patch_size * patch_size**2].

image_grid_thw
torch.Tensor

Long Tensor of shape [images, 3] giving each patch grid as (frames, height, width).

Returns: BaseModelOutputWithPooling

Output whose last_hidden_state is the vision Tensor of shape

nemo_automodel.components.models.lfm3.vl_model.Lfm3VlModel.get_input_embeddings() -> torch.nn.Embedding

Return the token embedding table.

nemo_automodel.components.models.lfm3.vl_model.Lfm3VlModel.get_video_features(
pixel_values_videos: torch.Tensor,
video_grid_thw: torch.Tensor
) -> transformers.modeling_outputs.BaseModelOutputWithPooling

Encode videos into language-model tokens; same layouts as get_image_features, per video.

class nemo_automodel.components.models.lfm3.vl_model.Lfm3VlMultiModalProjector(
)

Bases: Module

Normalize vision tokens, merge each spatial_merge_size square of them, and map to the text width.

act
= ACT2FN[config.projector_hidden_act]
layer_norm
= nn.LayerNorm(vision.hidden_size, eps=1e-06)
linear_1
linear_2
merged_size
nemo_automodel.components.models.lfm3.vl_model.Lfm3VlMultiModalProjector.forward(
image_features: torch.Tensor
) -> torch.Tensor

Project vision tokens to language tokens.

Parameters:

image_features
torch.Tensor

Tensor of shape [patches, vision_hidden], where consecutive runs of spatial_merge_size**2 patches form one merge block. Cast to the projector’s dtype: a frozen vision tower may be stored in lower precision.

Returns: torch.Tensor

Tensor of shape [patches / spatial_merge_size**2, text_hidden].

class nemo_automodel.components.models.lfm3.vl_model.Lfm3VlPreTrainedModel()

Bases: PreTrainedModel

Weight initialization and loading shared by the LFM3-VL models.

_no_split_modules
= ['Lfm3VlVisionLayer']
_skip_keys_device_placement
= ['past_key_values']
base_model_prefix
= 'model'
input_modalities
= ('image', 'video', 'text')
class nemo_automodel.components.models.lfm3.vl_model.Lfm3VlVisionAttention(
layer_idx: int
)

Bases: Lfm2Attention

LFM2 grouped-query attention made bidirectional, attending within one frame at a time.

nemo_automodel.components.models.lfm3.vl_model.Lfm3VlVisionAttention.forward(
hidden_states: torch.Tensor,
position_embeddings: tuple[torch.Tensor, torch.Tensor],
frame_lengths: list[int]
) -> torch.Tensor

Attend among the patches of each frame.

Parameters:

hidden_states
torch.Tensor

Tensor of shape [patches, hidden], the frames of every image and video concatenated.

position_embeddings
tuple[torch.Tensor, torch.Tensor]

Pair of float32 cosine and sine Tensors, each of shape [patches, head_dim].

frame_lengths
list[int]

Patch count of each frame, in order; sums to patches.

Returns: torch.Tensor

Tensor of shape [patches, hidden].

class nemo_automodel.components.models.lfm3.vl_model.Lfm3VlVisionLayer(
layer_idx: int
)

Bases: GradientCheckpointingLayer

Pre-norm operator (per-frame attention or 2D convolution) followed by a pre-norm MLP.

conv
= Lfm3VlVisionShortConv(config)
feed_forward
= Lfm2MoeMLP(config)
ffn_norm
is_attention_layer
= config.layer_types[layer_idx] != 'conv'
operator_norm
self_attn
= Lfm3VlVisionAttention(config, layer_idx)
nemo_automodel.components.models.lfm3.vl_model.Lfm3VlVisionLayer.forward(
hidden_states: torch.Tensor,
position_embeddings: tuple[torch.Tensor, torch.Tensor],
grid_thw: list[list[int]],
frame_lengths: list[int]
) -> torch.Tensor

Run one vision layer.

Parameters:

hidden_states
torch.Tensor

Tensor of shape [patches, hidden].

position_embeddings
tuple[torch.Tensor, torch.Tensor]

Pair of float32 cosine and sine Tensors, each of shape [patches, head_dim].

grid_thw
list[list[int]]

Per image or video, its patch grid as (frames, height, width).

frame_lengths
list[int]

Patch count of each frame, in order; sums to patches.

Returns: torch.Tensor

Tensor of shape [patches, hidden].

class nemo_automodel.components.models.lfm3.vl_model.Lfm3VlVisionModel(
)

Bases: Lfm3VlPreTrainedModel

LFM3-VL vision encoder over the flattened patches of images and video frames.

embedding_norm
input_modalities
= ('image', 'video')
layers
patchifier
nemo_automodel.components.models.lfm3.vl_model.Lfm3VlVisionModel.forward(
pixel_values: torch.Tensor,
grid_thw: torch.Tensor
) -> torch.Tensor

Encode patches.

Parameters:

pixel_values
torch.Tensor

Tensor of shape [patches, in_channels * temporal_patch_size * patch_size**2], the flattened patches of every image or video. Within a frame they are ordered merge block by merge block.

grid_thw
torch.Tensor

Long Tensor of shape [items, 3] giving each image’s or video’s patch grid as (frames, height, width).

Returns: torch.Tensor

Tensor of shape [patches, hidden] in the input order.

class nemo_automodel.components.models.lfm3.vl_model.Lfm3VlVisionShortConv(
)

Bases: Module

Gated depthwise 2D convolution over the patch grid of each frame.

in_proj
merge_size
= config.spatial_merge_size
out_proj
spatial_conv
nemo_automodel.components.models.lfm3.vl_model.Lfm3VlVisionShortConv.forward(
hidden_states: torch.Tensor,
grid_thw: list[list[int]]
) -> torch.Tensor

Gate, convolve each frame over its patch grid, and gate again.

Parameters:

hidden_states
torch.Tensor

Tensor of shape [patches, hidden]. Within a frame the patches are ordered merge block by merge block, as the processor emits them.

grid_thw
list[list[int]]

Per image or video, its patch grid as (frames, height, width).

Returns: torch.Tensor

Tensor of shape [patches, hidden] in the input order.

nemo_automodel.components.models.lfm3.vl_model.axial_rotary_factors(
position_ids: torch.Tensor,
theta: float,
head_dim: int
) -> tuple[torch.Tensor, torch.Tensor]

Build 2D rotary factors in float32: half of each head rotates by row, half by column.

Parameters:

position_ids
torch.Tensor

Long Tensor of shape [patches, 2] holding each patch’s (row, column).

theta
float

RoPE base.

head_dim
int

Channels per attention head.

Returns: tuple[torch.Tensor, torch.Tensor]

Pair of float32 cosine and sine Tensors, each of shape [patches, head_dim].

nemo_automodel.components.models.lfm3.vl_model.ModelClass = Lfm3VlForConditionalGeneration