gemma-4-12B
Gemma-4-12B has checked-in NeMo AutoModel recipes for text generation.
The Hugging Face configuration declares the Gemma4UnifiedForConditionalGeneration architecture.
Fine-Tune Gemma-4-12B
Follow the installation instructions, then run the recipe from the repository root:
Train on Long Sequences with Context Parallelism
Gemma-4-12B supports context parallelism (distributed.cp_size > 1). Each rank keeps a contiguous slice
of the sequence, and the sliding-window and full-attention layers exchange keys and values over a p2p ring:
For 64K-token sequences on a single 8-GPU node, gemma_4_12b_tulu3_cp8_64k.yaml packs Tulu-3
conversations into 64K-token sequences (THD) and shards each over cp_size: 8 (8K tokens per rank):
The CP recipes use FusedLinearCrossEntropy, which computes the loss from the hidden states instead of
materializing the 262K-vocab logits for every token of the shard. Longer sequences take more ranks at
the same 8K-token shard length rather than longer per-rank shards, which exceed 80 GB per GPU with this
model: gemma_4_12b_tulu3_cp16_128k.yaml trains 128K-token sequences over two nodes (cp_size: 16),
launched with torchrun on each node:
Context parallelism needs attn_implementation: sdpa, which the ring attention replaces. The CP recipes
set text_config.cp_full_attn_backend: ffpa, which runs the head_dim=512 full-attention ring chunks
through the FFPA kernel and needs the ffpa extra (uv sync --extra ffpa); remove the key to use Flex
there instead. The 64K and 128K recipes also set text_config.cp_sliding_attn_backend: fa, which runs
the sliding-window layers through FlashAttention and needs the fa extra; without it they use Flex.
The base checkpoint ships no chat template, so the Tulu-3 recipes load the tokenizer of
google/gemma-4-12B-it, which has the same vocabulary and the Gemma 4 chat template.
Context parallelism covers text and image inputs; video and audio inputs are rejected under CP. Image
tokens marked by mm_token_type_ids attend bidirectionally on the sliding-window layers only, with and
without CP.