nemo_automodel.components.models.gemma4_unified.model
nemo_automodel.components.models.gemma4_unified.model
NeMo wrapper for the Hugging Face Gemma4 Unified model.
Module Contents
Classes
API
Bases: Gemma4ContextParallelMixin, HFCheckpointingMixin, HFGemma4UnifiedForConditionalGeneration
Gemma4 Unified with NeMo checkpoint key conversion and model-owned context parallelism.
The text stack has the same attention layout as dense Gemma4 (sliding-window and full layers, vision-bidirectional sliding layers), so context parallelism reuses Gemma4’s CP hooks, its contiguous per-rank shards and its p2p ring attention. Without CP the forward is Hugging Face’s.
Hugging Face’s forward, or the context-parallel forward once CP is set up.
Image blocks marked by mm_token_type_ids attend bidirectionally on sliding-window layers
only, as in this model’s create_masks_for_generate and in dense Gemma4. Hugging Face’s own
forward, which training and generate() with a dynamic cache go through, also applies that
overlay to the full-attention layers, so without CP the forward passes it this model’s masks;
training with and without CP then matches generation.
Under CP the forward embeds the full input_ids and splices image features as Hugging
Face does, builds the vision-bidirectional block ids on the full sequence, then keeps this
rank’s contiguous slice; the aux-only sharder already sliced position_ids,
padding_mask and _packed_seq_ids to the same layout, and every text self-attention
module runs the ring.
Parameters:
Token ids [batch, sequence]; full-length under CP.
Image patches [images, max_patches, patch_dim] for the vision embedder.
Video patches [videos, frames, max_patches, patch_dim]; not supported
under CP.
Audio features [audios, tokens, audio_embed_dim]; not supported under CP.
Padding mask [batch, sequence] (1 = keep), or a mapping from layer type to
prebuilt masks; under CP the ring builds its own masks.
Valid-audio-token mask [audios, tokens] (True = valid).
Position ids [batch, sequence]; under CP this rank’s slice
[batch, local_sequence].
Patch positions [images, max_patches, 2] (-1 marks padding).
Video patch positions [videos, frames, max_patches, 2].
KV cache; unused under CP.
Modality ids [batch, sequence]; image blocks attend bidirectionally on
sliding-window layers when set.
Input embeddings [batch, sequence, hidden]; not supported under CP.
Labels [batch, sequence] for the Hugging Face loss; under CP the recipe computes the
loss instead.
Whether to return a KV cache; off under CP.
Integer suffix length, or an index tensor [kept_positions] selecting the
returned positions.
Precomputed multimodal encoder outputs; not supported under CP.
Remaining keyword arguments. Under CP, padding_mask (True = pad) and
_packed_seq_ids (0 = pad) carry this rank’s slices to the ring.
Returns: Gemma4UnifiedCausalLMOutputWithPast
Model outputs; under CP, logits for this rank’s sequence shard [batch, local_sequence, vocab].
Raises:
ValueError: Under CP, for inputs the context-parallel forward does not support, or when the image placeholder tokens do not match the image features.
Install the ring attention on every text self-attention module (idempotent).
Parameters:
The context-parallel device submesh.
Raises:
ValueError: If the text model does not use SDPA attention, which the ring replaces, or attends bidirectionally everywhere (use_bidirectional_attention='all'), which the causal ring cannot.
Tie the output head to Gemma4 Unified’s text input embedding.