Skip to navigation

nemo_automodel.components.models.lfm3.config

View as MarkdownOpen in Claude

LFM3 and LFM3-VL configurations for Transformers releases that predate them.

Module Contents

Classes

NameDescription
Lfm3ConfigConfigure the LFM3 decoder of gated short convolutions and grouped-query attention.
Lfm3VlConfigConfigure LFM3-VL: an LFM3 decoder fed by the LFM3-VL vision encoder through an MLP projector.
Lfm3VlVisionConfigConfigure the LFM3-VL vision encoder of gated 2D convolutions and bidirectional attention.

Data

_JsonValue

API

class nemo_automodel.components.models.lfm3.config.Lfm3Config()

Bases: PreTrainedConfig

Configure the LFM3 decoder of gated short convolutions and grouped-query attention.

layer_types names each layer "conv", "full_attention" or "sliding_attention". rope_parameters holds one entry per attention type. Full-attention layers rotate only the first partial_rotary_factor share of each head and leave the rest unrotated.

Released checkpoints label that partial rotation "default", which Transformers 5.23 reads as a full rotation. Such an entry is relabelled "proportional" here, the name under which Transformers and the publisher’s code both build the trained frequencies, so saved configs load correctly in either.

_rope_type_labels
= ('full_attention', 'sliding_attention')
attention_bias
bool = False
attention_dropout
float = 0.0
attribute_map
= {'rms_norm_eps': 'norm_eps'}
bos_token_id
int | None = 124894
conv_L_cache
int = 7
conv_bias
bool = False
default_theta
= 1000000.0
eos_token_id
int | list[int] | None = 124900
head_dim
int = 128
hidden_size
int = 2048
initializer_range
float = 0.02
intermediate_size
int = 7680
keys_to_ignore_at_inference
= ['past_key_values']
layer_types
list[str] | None = None
max_position_embeddings
int = 128000
model_type
= 'lfm3'
norm_eps
float = 1e-05
num_attention_heads
int = 16
num_hidden_layers
int = 24
num_key_value_heads
int = 4
pad_token_id
int | None = 124893
rope_parameters
dict | None = None
sliding_window
int | None = 256
tie_word_embeddings
bool = True
use_attention_gate
bool = True
use_cache
bool = True
use_sandwich_norm
bool = True
vocab_size
int = 128000
nemo_automodel.components.models.lfm3.config.Lfm3Config.__post_init__(
) -> None
nemo_automodel.components.models.lfm3.config.Lfm3Config.validate_architecture() -> None

Reject RoPE settings this implementation does not run.

Transformers runs this before saving. The config runs it on construction and the model runs it again when built, because loader overrides are written onto an existing config and skip construction.

Raises:

  • ValueError: If an attention layer type has no RoPE entry, uses a RoPE type other than default or proportional, or scales its frequencies with a non-unit factor.
class nemo_automodel.components.models.lfm3.config.Lfm3VlConfig()

Bases: PreTrainedConfig

Configure LFM3-VL: an LFM3 decoder fed by the LFM3-VL vision encoder through an MLP projector.

image_token_id
int = 125030
model_type
= 'lfm3_vl'
projector_bias
bool = True
projector_hidden_act
str = 'gelu'
projector_hidden_size
int = 4096
sub_configs
text_config
dict | PreTrainedConfig | None = None
tie_word_embeddings
bool = True
video_token_id
int = 125029
vision_config
dict | PreTrainedConfig | None = None
vision_end_token_id
int = 125010
vision_start_token_id
int = 125009
nemo_automodel.components.models.lfm3.config.Lfm3VlConfig.__post_init__(
) -> None
class nemo_automodel.components.models.lfm3.config.Lfm3VlVisionConfig()

Bases: PreTrainedConfig

Configure the LFM3-VL vision encoder of gated 2D convolutions and bidirectional attention.

layer_types names each layer "conv" or "full_attention". Patches are patch_size pixels square and temporal_patch_size frames deep; the projector later merges each spatial_merge_size square of them.

base_config_key
= 'vision_config'
conv_kernel_size
int = 3
default_rope_type
= 'axial'
default_theta
= 1000000.0
hidden_size
int = 1024
in_channels
int = 3
initializer_range
float = 0.02
intermediate_size
int = 2560
layer_types
list[str] | None = None
model_type
= 'lfm3_vl_vision'
norm_eps
float = 1e-05
num_attention_heads
int = 16
num_hidden_layers
int = 14
num_key_value_heads
int = 8
patch_size
int = 16
rope_parameters
dict | None = None
spatial_merge_size
int = 2
temporal_patch_size
int = 2
nemo_automodel.components.models.lfm3.config.Lfm3VlVisionConfig.__post_init__(
) -> None
nemo_automodel.components.models.lfm3.config._JsonValue = str | int | float | bool | list | dict | None