nemo_automodel.components.models.lfm3.vl_model
nemo_automodel.components.models.lfm3.vl_model
LFM3-VL: the LFM3 decoder fed by a convolution-and-attention vision encoder through an MLP projector.
Module Contents
Classes
Functions
Data
API
Bases: HFCheckpointingMixin, Lfm3VlPreTrainedModel, GenerationMixin
LFM3-VL with a vocabulary head, loadable from LiquidAI’s released checkpoints.
Predict next-token logits over text interleaved with images and videos.
Parameters:
Long Tensor of shape [batch, sequence]. Exclusive with inputs_embeds.
Optional padding Tensor of shape [batch, key_sequence], where
key_sequence also counts cached tokens.
Optional long Tensor of shape [batch, sequence].
Optional generation cache, updated in place.
Tensor of shape [batch, sequence, hidden].
Image patches, Tensor of shape [patches, in_channels * temporal_patch_size * patch_size**2].
Long Tensor of shape [images, 3], each patch grid as (frames, height, width).
Video patches, laid out like pixel_values.
Long Tensor of shape [videos, 3], laid out like image_grid_thw.
Optional long Tensor of shape [batch, sequence]; -100 is ignored.
Shifted internally.
Whether to create and return a generation cache.
Project only the last logits_to_keep positions (0 keeps
all), or the positions in a long Tensor of shape [kept].
Precomputed vision features, as generation passes them.
Hugging Face output-capture, attention and loss arguments.
Returns: CausalLMOutputWithPast
Output whose logits is a Tensor of shape [batch, kept, vocab] and whose
Return the token embedding table.
Return the vocabulary head.
Replace the token embedding table and retie the head.
Share the vocabulary head with the token embeddings when the config asks for it.
Bases: Lfm3VlPreTrainedModel
LFM3 decoder whose image and video placeholder tokens are replaced by projected vision features.
Write vision features over the embeddings of one modality’s placeholder tokens.
Parameters:
Tensor of shape [batch, sequence, hidden].
Long Tensor of shape [batch, sequence] locating the placeholders.
The modality’s placeholder token.
One Tensor of shape [tokens, hidden] per image or video, in the order their placeholders appear row by row.
Returns: torch.Tensor
New Tensor of shape [batch, sequence, hidden]; inputs_embeds is not modified.
Encode any images and videos, splice them into the text, and run the decoder.
Parameters:
Long Tensor of shape [batch, sequence]. Exclusive with inputs_embeds.
Optional padding Tensor of shape [batch, key_sequence], where
key_sequence also counts cached tokens.
Optional long Tensor of shape [batch, sequence].
Optional generation cache, updated in place.
Tensor of shape [batch, sequence, hidden].
Image patches, Tensor of shape [patches, in_channels * temporal_patch_size * patch_size**2].
Long Tensor of shape [images, 3], each patch grid as (frames, height, width).
Video patches, laid out like pixel_values.
Long Tensor of shape [videos, 3], laid out like image_grid_thw.
Whether to create and return a generation cache.
Results of get_image_features and get_video_features
under "image" and "video", as generation precomputes them. Exclusive
with the pixel inputs.
Hugging Face output-capture and attention arguments for the decoder.
Returns: BaseModelOutputWithPast
Output whose last_hidden_state is a Tensor of shape [batch, sequence, hidden].
Encode images into language-model tokens.
Parameters:
Tensor of shape [patches, in_channels * temporal_patch_size * patch_size**2].
Long Tensor of shape [images, 3] giving each patch grid as (frames, height, width).
Returns: BaseModelOutputWithPooling
Output whose last_hidden_state is the vision Tensor of shape
Return the token embedding table.
Encode videos into language-model tokens; same layouts as get_image_features, per video.
Bases: Module
Normalize vision tokens, merge each spatial_merge_size square of them, and map to the text width.
Project vision tokens to language tokens.
Parameters:
Tensor of shape [patches, vision_hidden], where consecutive runs
of spatial_merge_size**2 patches form one merge block. Cast to the
projector’s dtype: a frozen vision tower may be stored in lower precision.
Returns: torch.Tensor
Tensor of shape [patches / spatial_merge_size**2, text_hidden].
Bases: PreTrainedModel
Weight initialization and loading shared by the LFM3-VL models.
Bases: Lfm2Attention
LFM2 grouped-query attention made bidirectional, attending within one frame at a time.
Attend among the patches of each frame.
Parameters:
Tensor of shape [patches, hidden], the frames of every image and video concatenated.
Pair of float32 cosine and sine Tensors, each of shape [patches, head_dim].
Patch count of each frame, in order; sums to patches.
Returns: torch.Tensor
Tensor of shape [patches, hidden].
Bases: GradientCheckpointingLayer
Pre-norm operator (per-frame attention or 2D convolution) followed by a pre-norm MLP.
Run one vision layer.
Parameters:
Tensor of shape [patches, hidden].
Pair of float32 cosine and sine Tensors, each of shape [patches, head_dim].
Per image or video, its patch grid as (frames, height, width).
Patch count of each frame, in order; sums to patches.
Returns: torch.Tensor
Tensor of shape [patches, hidden].
Bases: Lfm3VlPreTrainedModel
LFM3-VL vision encoder over the flattened patches of images and video frames.
Encode patches.
Parameters:
Tensor of shape [patches, in_channels * temporal_patch_size * patch_size**2], the flattened patches of every image or video. Within a frame they are ordered merge block by merge block.
Long Tensor of shape [items, 3] giving each image’s or video’s patch grid as (frames, height, width).
Returns: torch.Tensor
Tensor of shape [patches, hidden] in the input order.
Bases: Module
Gated depthwise 2D convolution over the patch grid of each frame.
Gate, convolve each frame over its patch grid, and gate again.
Parameters:
Tensor of shape [patches, hidden]. Within a frame the patches are ordered merge block by merge block, as the processor emits them.
Per image or video, its patch grid as (frames, height, width).
Returns: torch.Tensor
Tensor of shape [patches, hidden] in the input order.
Build 2D rotary factors in float32: half of each head rotates by row, half by column.
Parameters:
Long Tensor of shape [patches, 2] holding each patch’s (row, column).
RoPE base.
Channels per attention head.
Returns: tuple[torch.Tensor, torch.Tensor]
Pair of float32 cosine and sine Tensors, each of shape [patches, head_dim].