Skip to navigation

nemo_automodel.components.models.gemma4_unified.model

View as MarkdownOpen in Claude

NeMo wrapper for the Hugging Face Gemma4 Unified model.

Module Contents

Classes

NameDescription
Gemma4UnifiedForConditionalGenerationGemma4 Unified with NeMo checkpoint key conversion and model-owned context parallelism.

API

class nemo_automodel.components.models.gemma4_unified.model.Gemma4UnifiedForConditionalGeneration(
config
)

Bases: Gemma4ContextParallelMixin, HFCheckpointingMixin, HFGemma4UnifiedForConditionalGeneration

Gemma4 Unified with NeMo checkpoint key conversion and model-owned context parallelism.

The text stack has the same attention layout as dense Gemma4 (sliding-window and full layers, vision-bidirectional sliding layers), so context parallelism reuses Gemma4’s CP hooks, its contiguous per-rank shards and its p2p ring attention. Without CP the forward is Hugging Face’s.

state_dict_adapter
= Gemma4UnifiedStateDictAdapter()
tie_word_embeddings_support
TieSupport = TieSupport.TIED_ONLY
nemo_automodel.components.models.gemma4_unified.model.Gemma4UnifiedForConditionalGeneration.forward(
input_ids: torch.Tensor | None = None,
pixel_values: torch.Tensor | None = None,
pixel_values_videos: torch.Tensor | None = None,
input_features: torch.Tensor | None = None,
attention_mask: torch.Tensor | None = None,
input_features_mask: torch.Tensor | None = None,
position_ids: torch.Tensor | None = None,
image_position_ids: torch.Tensor | None = None,
video_position_ids: torch.Tensor | None = None,
past_key_values: typing.Any = None,
mm_token_type_ids: torch.Tensor | None = None,
inputs_embeds: torch.Tensor | None = None,
labels: torch.Tensor | None = None,
use_cache: bool | None = None,
logits_to_keep: int | torch.Tensor = 0,
mm_encoder_outputs: dict[str, typing.Any] | None = None,
kwargs: typing.Any = {}
) -> transformers.models.gemma4_unified.modeling_gemma4_unified.Gemma4UnifiedCausalLMOutputWithPast

Hugging Face’s forward, or the context-parallel forward once CP is set up.

Image blocks marked by mm_token_type_ids attend bidirectionally on sliding-window layers only, as in this model’s create_masks_for_generate and in dense Gemma4. Hugging Face’s own forward, which training and generate() with a dynamic cache go through, also applies that overlay to the full-attention layers, so without CP the forward passes it this model’s masks; training with and without CP then matches generation.

Under CP the forward embeds the full input_ids and splices image features as Hugging Face does, builds the vision-bidirectional block ids on the full sequence, then keeps this rank’s contiguous slice; the aux-only sharder already sliced position_ids, padding_mask and _packed_seq_ids to the same layout, and every text self-attention module runs the ring.

Parameters:

input_ids
torch.Tensor | NoneDefaults to None

Token ids [batch, sequence]; full-length under CP.

pixel_values
torch.Tensor | NoneDefaults to None

Image patches [images, max_patches, patch_dim] for the vision embedder.

pixel_values_videos
torch.Tensor | NoneDefaults to None

Video patches [videos, frames, max_patches, patch_dim]; not supported under CP.

input_features
torch.Tensor | NoneDefaults to None

Audio features [audios, tokens, audio_embed_dim]; not supported under CP.

attention_mask
torch.Tensor | NoneDefaults to None

Padding mask [batch, sequence] (1 = keep), or a mapping from layer type to prebuilt masks; under CP the ring builds its own masks.

input_features_mask
torch.Tensor | NoneDefaults to None

Valid-audio-token mask [audios, tokens] (True = valid).

position_ids
torch.Tensor | NoneDefaults to None

Position ids [batch, sequence]; under CP this rank’s slice [batch, local_sequence].

image_position_ids
torch.Tensor | NoneDefaults to None

Patch positions [images, max_patches, 2] (-1 marks padding).

video_position_ids
torch.Tensor | NoneDefaults to None

Video patch positions [videos, frames, max_patches, 2].

past_key_values
AnyDefaults to None

KV cache; unused under CP.

mm_token_type_ids
torch.Tensor | NoneDefaults to None

Modality ids [batch, sequence]; image blocks attend bidirectionally on sliding-window layers when set.

inputs_embeds
torch.Tensor | NoneDefaults to None

Input embeddings [batch, sequence, hidden]; not supported under CP.

labels
torch.Tensor | NoneDefaults to None

Labels [batch, sequence] for the Hugging Face loss; under CP the recipe computes the loss instead.

use_cache
bool | NoneDefaults to None

Whether to return a KV cache; off under CP.

logits_to_keep
int | torch.TensorDefaults to 0

Integer suffix length, or an index tensor [kept_positions] selecting the returned positions.

mm_encoder_outputs
dict[str, Any] | NoneDefaults to None

Precomputed multimodal encoder outputs; not supported under CP.

**kwargs
AnyDefaults to {}

Remaining keyword arguments. Under CP, padding_mask (True = pad) and _packed_seq_ids (0 = pad) carry this rank’s slices to the ring.

Returns: Gemma4UnifiedCausalLMOutputWithPast

Model outputs; under CP, logits for this rank’s sequence shard [batch, local_sequence, vocab].

Raises:

  • ValueError: Under CP, for inputs the context-parallel forward does not support, or when the image placeholder tokens do not match the image features.
nemo_automodel.components.models.gemma4_unified.model.Gemma4UnifiedForConditionalGeneration.setup_cp_attention(
cp_mesh
) -> None

Install the ring attention on every text self-attention module (idempotent).

Parameters:

cp_mesh

The context-parallel device submesh.

Raises:

  • ValueError: If the text model does not use SDPA attention, which the ring replaces, or attends bidirectionally everywhere (use_bidirectional_attention='all'), which the causal ring cannot.
nemo_automodel.components.models.gemma4_unified.model.Gemma4UnifiedForConditionalGeneration.tie_weights(
_args: set[str] | bool | None = (),
_kwargs: set[str] | bool | None = {}
) -> None

Tie the output head to Gemma4 Unified’s text input embedding.