Skip to navigation
ReferenceFull Library ReferenceNemo AutomodelNemo AutomodelComponentsModelsnemo_automodel.components.models.minimax_m3_vl

nemo_automodel.components.models.minimax_m3_vl

View as Markdown

Subpackages

Submodules

Package Contents

Classes

NameDescription
MiniMaxM3SparseForCausalLMStandalone M3 text backbone for causal LM (Stage 1 parity target).
MiniMaxM3SparseForConditionalGenerationMiniMax M3 VL: CLIP-style vision tower + projector/merger + M3 text backbone.
MiniMaxM3VLConfigTop-level configuration for MiniMax M3 vision-language checkpoints.
MiniMaxM3VLTextConfigConfiguration for the MiniMax M3 (mixed sparse/dense MoE) text backbone.
MiniMaxM3VLVisionConfigConfiguration for the MiniMax M3 VL CLIP-style vision tower (Conv3d + 3D RoPE).

Data

ModelClass

API

class nemo_automodel.components.models.minimax_m3_vl.model.MiniMaxM3SparseForCausalLM(
    config: typing.Any,
    moe_config: nemo_automodel.components.moe.layers.MoEConfig | None = None,
    backend: nemo_automodel.components.models.common.BackendConfig | None = None,
    kwargs = {}
)

Bases: HFCheckpointingMixin, Module, MoEFSDPSyncMixin

Standalone M3 text backbone for causal LM (Stage 1 parity target).

_keep_in_fp32_modules_strict
_pp_keep_self_forward
bool = True
backend
consumes_packed_seq_ids
bool

Whether the packed loader should hand this model _packed_seq_ids (the compact document map) for every pack, single-document ones included, instead of a [batch, 1, sequence, sequence] mask.

lm_head
model
state_dict_adapter
tie_word_embeddings_support
TieSupport = TieSupport.UNTIED_ONLY
nemo_automodel.components.models.minimax_m3_vl.model.MiniMaxM3SparseForCausalLM.forward(
    input_ids: torch.Tensor,
    position_ids: torch.Tensor | None = None,
    attention_mask: torch.Tensor | None = None,
    padding_mask: torch.Tensor | None = None,
    attn_kwargs: typing.Any = {}
) -> torch.Tensor | nemo_automodel.components.models.minimax_m3_vl.model.MiniMaxM3CausalLMOutput

Map ids/positions/padding[B,S] and mask[B,S] or [B,1,S,S] to logits[B,S,V] or MTP logits; MSA uses BSHD.

nemo_automodel.components.models.minimax_m3_vl.model.MiniMaxM3SparseForCausalLM.from_config(
    config: typing.Any,
    moe_config: nemo_automodel.components.moe.layers.MoEConfig | None = None,
    backend: nemo_automodel.components.models.common.BackendConfig | None = None,
    kwargs = {}
)
classmethod
nemo_automodel.components.models.minimax_m3_vl.model.MiniMaxM3SparseForCausalLM.from_pretrained(
    pretrained_model_name_or_path: str,
    model_args = (),
    kwargs = {}
)
classmethod
nemo_automodel.components.models.minimax_m3_vl.model.MiniMaxM3SparseForCausalLM.get_input_embeddings()
nemo_automodel.components.models.minimax_m3_vl.model.MiniMaxM3SparseForCausalLM.get_output_embeddings()
nemo_automodel.components.models.minimax_m3_vl.model.MiniMaxM3SparseForCausalLM.initialize_weights(
    buffer_device: torch.device | None = None,
    dtype: torch.dtype = torch.bfloat16
) -> None
nemo_automodel.components.models.minimax_m3_vl.model.MiniMaxM3SparseForCausalLM.set_input_embeddings(
    value
)
nemo_automodel.components.models.minimax_m3_vl.model.MiniMaxM3SparseForCausalLM.set_output_embeddings(
    new_embeddings
)
class nemo_automodel.components.models.minimax_m3_vl.model.MiniMaxM3SparseForConditionalGeneration(
    config: typing.Any,
    moe_config: nemo_automodel.components.moe.layers.MoEConfig | None = None,
    backend: nemo_automodel.components.models.common.BackendConfig | None = None,
    kwargs = {}
)

Bases: HFCheckpointingMixin, Module, MoEFSDPSyncMixin

MiniMax M3 VL: CLIP-style vision tower + projector/merger + M3 text backbone.

Vision features (vision_tower(pixel_values, grid_thw)) are spliced into the text embeddings at image_token_index / video_token_index positions, then run through the (sparse/dense MoE) language model + lm_head.

_keep_in_fp32_modules
= ['rotary_emb', 'inv_freq']
_keep_in_fp32_modules_strict
_pp_keep_self_forward
bool = True
backend
consumes_packed_seq_ids
bool

Whether the packed loader should hand this model _packed_seq_ids (the compact document map) for every pack, single-document ones included, instead of a [batch, 1, sequence, sequence] mask.

image_token_index
= config.image_token_index
lm_head
model
state_dict_adapter
tie_word_embeddings_support
TieSupport = TieSupport.UNTIED_ONLY
video_token_index
= config.video_token_index
vision_tower
vocab_size
= text_config.vocab_size
nemo_automodel.components.models.minimax_m3_vl.model.MiniMaxM3SparseForConditionalGeneration._embed_and_splice(
    input_ids: torch.Tensor,
    pixel_values: torch.Tensor | None = None,
    image_grid_thw = None,
    pixel_values_videos: torch.Tensor | None = None,
    video_grid_thw = None
) -> torch.Tensor

Embed token ids and splice vision/video features into the embeddings.

Shared by forward and prepare_model_inputs_for_cp so the splice logic lives in one place.

nemo_automodel.components.models.minimax_m3_vl.model.MiniMaxM3SparseForConditionalGeneration._is_pipeline_parallel_stage() -> bool

True when this is a partial pipeline stage (some text modules nulled).

nemo_automodel.components.models.minimax_m3_vl.model.MiniMaxM3SparseForConditionalGeneration._splice_multimodal(
    inputs_embeds: torch.Tensor,
    input_ids: torch.Tensor,
    pixel_values: torch.Tensor,
    grid_thw,
    token_index: int
) -> torch.Tensor
nemo_automodel.components.models.minimax_m3_vl.model.MiniMaxM3SparseForConditionalGeneration._to_grid_list(
    grid_thw
) -> list[list[int]]
staticmethod
nemo_automodel.components.models.minimax_m3_vl.model.MiniMaxM3SparseForConditionalGeneration.customize_pipeline_stage_modules(
    module_names_per_stage: list[list[str]],
    layers_prefix: str,
    text_model: torch.nn.Module | None = None
) -> list[list[str]]

Rewrite auto-generated pipeline FQNs to M3’s real module paths.

M3’s text stack lives directly under self.model and the vision tower is a top-level sibling (vision_tower). The framework, seeing the language_model property, derives a nested model.language_model. prefix for the text modules and a model. prefix for the multimodal encoders. Map both back to M3’s actual paths so per-stage module nulling keeps/drops the correct submodules.

nemo_automodel.components.models.minimax_m3_vl.model.MiniMaxM3SparseForConditionalGeneration.forward(
    input_ids: torch.Tensor | None = None,
    pixel_values: torch.Tensor | None = None,
    image_grid_thw = None,
    pixel_values_videos: torch.Tensor | None = None,
    video_grid_thw = None,
    position_ids: torch.Tensor | None = None,
    attention_mask: torch.Tensor | None = None,
    inputs_embeds: torch.Tensor | None = None,
    logits_to_keep: int | None = None,
    kwargs: typing.Any = {}
) -> torch.Tensor | nemo_automodel.components.models.minimax_m3_vl.model.MiniMaxM3CausalLMOutput | dict[str, torch.Tensor]

Run one MiniMax-M3 VL microbatch, consuming staged PP media when present.

Parameters:

input_ids
torch.Tensor | NoneDefaults to None

Token IDs of shape [batch, sequence] on the first stage, with one placeholder per merged vision token, or float hidden states of shape [batch, sequence, hidden] from the previous pipeline stage. Under CP they stay full-length until the embeddings are sharded here.

pixel_values
torch.Tensor | NoneDefaults to None

Optional image patch rows of shape [patches, patch_features], where patch_features is channels * temporal_patch_size * patch_size**2 and each image’s time * height * width patches are concatenated in image order. When both media inputs are None, the next staged PP media chunk is used.

image_grid_thw
Defaults to None

Optional tensor of shape [images, 3], or a list of [time, height, width] triples, storing each image’s patch grid.

pixel_values_videos
torch.Tensor | NoneDefaults to None

Optional video patch rows of shape [patches, patch_features], laid out like pixel_values.

video_grid_thw
Defaults to None

Optional tensor of shape [videos, 3], or a list of [time, height, width] triples, storing each video’s patch grid.

position_ids
torch.Tensor | NoneDefaults to None

Optional tensor of shape [batch, sequence]; under CP, sequence is the local shard.

attention_mask
torch.Tensor | NoneDefaults to None

Optional mask of shape [batch, sequence] or [batch, 1, sequence, sequence].

inputs_embeds
torch.Tensor | NoneDefaults to None

Optional tensor of shape [batch, sequence, hidden].

logits_to_keep
int | NoneDefaults to None

When set outside pipeline parallelism, skip the LM head and return hidden states for the fused loss.

**kwargs
AnyDefaults to {}

Attention metadata forwarded to the text model.

Returns: torch.Tensor | MiniMaxM3CausalLMOutput | dict[str, torch.Tensor]

Logits of shape [batch, sequence, vocab], or hidden states of shape

nemo_automodel.components.models.minimax_m3_vl.model.MiniMaxM3SparseForConditionalGeneration.from_config(
    config: typing.Any,
    moe_config: nemo_automodel.components.moe.layers.MoEConfig | None = None,
    backend: nemo_automodel.components.models.common.BackendConfig | None = None,
    kwargs = {}
)
classmethod
nemo_automodel.components.models.minimax_m3_vl.model.MiniMaxM3SparseForConditionalGeneration.from_pretrained(
    pretrained_model_name_or_path: str,
    model_args = (),
    kwargs = {}
)
classmethod
nemo_automodel.components.models.minimax_m3_vl.model.MiniMaxM3SparseForConditionalGeneration.get_dspark_target_feature_modules(
    layer_ids: list[int]
) -> tuple[torch.nn.Module, ...]

Return modules whose inputs are the DSpark target features of layer_ids.

The feature of layer i is the residual stream entering layer i + 1. For the last layer that is the stream entering the final norm, so the draft sees the same unnormalized last-layer state as at serving time instead of the normalized hidden state that feeds lm_head. Each decoder block’s input norm and the final norm take that stream as their first positional argument.

Parameters:

layer_ids
list[int]

Strictly increasing decoder-layer indices in [-1, num_hidden_layers); -1 selects the stream entering layer 0.

Returns: nn.Module

Modules ordered like layer_ids. Each receives a tensor of shape

Raises:

  • ValueError: If the indices are duplicated, unsorted, or outside the decoder depth, or this rank does not hold the final norm.
nemo_automodel.components.models.minimax_m3_vl.model.MiniMaxM3SparseForConditionalGeneration.get_input_embeddings()
nemo_automodel.components.models.minimax_m3_vl.model.MiniMaxM3SparseForConditionalGeneration.get_output_embeddings()
nemo_automodel.components.models.minimax_m3_vl.model.MiniMaxM3SparseForConditionalGeneration.get_pipeline_stage_metas(
    is_first: bool,
    microbatch_size: int,
    seq_len: int,
    dtype: torch.dtype
) -> tuple[tuple[torch.Tensor, ...], tuple[torch.Tensor, ...]]

Per-stage input/output meta tensors for the PP schedule’s shape inference.

First stage consumes the FULL token ids [mb, seq]; later stages consume hidden states. The final stage (owning lm_head) emits logits; earlier stages emit hidden states.

Under context parallelism the first stage embeds the full sequence and shards it to this rank’s round-robin chunk pair inside forward (see shard_sequence_for_cp_round_robin), so every stage output and every later-stage input carries the LOCAL (padded-to-2*cp then //cp) sequence length while the first stage’s input stays full-length. At cp_size == 1 the lengths coincide and the layout is symmetric.

nemo_automodel.components.models.minimax_m3_vl.model.MiniMaxM3SparseForConditionalGeneration.initialize_weights(
    buffer_device: torch.device | None = None,
    dtype: torch.dtype = torch.bfloat16
) -> None
nemo_automodel.components.models.minimax_m3_vl.model.MiniMaxM3SparseForConditionalGeneration.prepare_model_inputs_for_cp(
    batch: dict[str, typing.Any],
    num_chunks: int = 1
) -> dict[str, typing.Any]

Return a sharder-only CP backend; embed + splice + shard happen in forward.

The returned ContextParallelSharder round-robin-shards only the no-grad aux streams (labels/position_ids/loss_mask/padding_mask) via shard_batch_aux_only, leaving input_ids and the multimodal inputs full-length; the forward then embeds + splices and calls shard_sequence_for_cp_round_robin per microbatch, so embeddings and vision stay trainable under CP. Nothing is consumed here. Defining this method is the opt-in signal the recipe checks (hasattr(model, "prepare_model_inputs_for_cp")).

Parameters:

batch
dict[str, Any]

The full-sequence batch; left intact (nothing consumed).

num_chunks
intDefaults to 1

Accepted for hook-signature parity; unused (round-robin CP).

nemo_automodel.components.models.minimax_m3_vl.model.MiniMaxM3SparseForConditionalGeneration.set_input_embeddings(
    value
)
class nemo_automodel.components.models.minimax_m3_vl.config.MiniMaxM3VLConfig(
    vision_config: typing.Union[dict, nemo_automodel.components.models.minimax_m3_vl.config.MiniMaxM3VLVisionConfig] | None = None,
    text_config: typing.Union[dict, nemo_automodel.components.models.minimax_m3_vl.config.MiniMaxM3VLTextConfig] | None = None,
    image_token_index: int = 200025,
    video_token_index: int = 200026,
    image_seq_length: int = 576,
    process_image_mode: str = 'dynamic_res',
    projector_hidden_act: str = 'gelu',
    projector_hidden_size: int = 6144,
    multimodal_projector_bias: bool = True,
    patch_merge_bias: bool = True,
    vision_feature_layer: int = -1,
    vision_feature_select_strategy: str = 'full',
    img_token_compression_config: dict | None = None,
    image_grid_pinpoints: str | None = None,
    quantization_config: nemo_automodel.components.models.minimax_m3_vl.config.MiniMaxM3QuantizationConfig | None = None,
    kwargs = {}
)

Bases: PretrainedConfig

Top-level configuration for MiniMax M3 vision-language checkpoints.

hidden_size
= text_config.hidden_size
img_token_compression_config
= img_token_compression_config or {}
max_position_embeddings
= text_config.max_position_embeddings
model_type
= 'minimax_m3_vl'
sub_configs
nemo_automodel.components.models.minimax_m3_vl.config.MiniMaxM3VLConfig.to_dict()
class nemo_automodel.components.models.minimax_m3_vl.config.MiniMaxM3VLTextConfig(
    hidden_size: int = 6144,
    intermediate_size: int = 3072,
    dense_intermediate_size: int = 12288,
    shared_intermediate_size: int = 3072,
    num_hidden_layers: int = 60,
    num_attention_heads: int = 64,
    num_key_value_heads: int = 4,
    head_dim: int = 128,
    vocab_size: int = 200064,
    max_position_embeddings: int = 524288,
    rms_norm_eps: float = 1e-06,
    use_gemma_norm: bool = True,
    attention_output_gate: bool = False,
    rope_theta: float = 5000000.0,
    rotary_dim: int = 64,
    partial_rotary_factor: float = 0.5,
    hidden_act: str = 'swigluoai',
    use_qk_norm: bool = True,
    qk_norm_type: str = 'per_head',
    tie_word_embeddings: bool = False,
    num_local_experts: int = 128,
    num_experts_per_tok: int = 4,
    n_shared_experts: int = 1,
    scoring_func: str = 'sigmoid',
    use_routing_bias: bool = True,
    routed_scaling_factor: float = 2.0,
    moe_layer_freq: list[int] | None = None,
    swiglu_alpha: float = 1.702,
    swiglu_limit: float = 7.0,
    sparse_attention_config: dict | None = None,
    num_mtp_modules: int = 1,
    quantization_config: nemo_automodel.components.models.minimax_m3_vl.config.MiniMaxM3QuantizationConfig | None = None,
    pad_token_id: int | None = None,
    kwargs = {}
)

Bases: PretrainedConfig

Configuration for the MiniMax M3 (mixed sparse/dense MoE) text backbone.

architectures
= ['MiniMaxM3SparseForCausalLM']
model_type
= 'minimax_m3'
nemo_automodel.components.models.minimax_m3_vl.config.MiniMaxM3VLTextConfig.to_dict()
class nemo_automodel.components.models.minimax_m3_vl.config.MiniMaxM3VLVisionConfig(
    hidden_size: int = 1280,
    num_attention_heads: int = 16,
    num_hidden_layers: int = 32,
    intermediate_size: int = 5120,
    patch_size: int = 14,
    image_size: int = 672,
    projection_dim: int = 6144,
    num_channels: int = 3,
    position_embedding_type: str = 'rope',
    rope_mode: str = '3d',
    rope_theta: float = 10000.0,
    attention_dropout: float = 0.0,
    hidden_act: str = 'gelu',
    layer_norm_eps: float = 1e-05,
    img_token_compression_config: dict | None = None,
    vision_segment_max_frames: int = 4,
    kwargs = {}
)

Bases: PretrainedConfig

Configuration for the MiniMax M3 VL CLIP-style vision tower (Conv3d + 3D RoPE).

img_token_compression_config
model_type
= 'minimax_m3_vision'
nemo_automodel.components.models.minimax_m3_vl.config.MiniMaxM3VLVisionConfig.to_dict()
nemo_automodel.components.models.minimax_m3_vl.ModelClass = MiniMaxM3SparseForConditionalGeneration