Skip to navigation

Gemma-4-12B has checked-in NeMo AutoModel recipes for text generation. The Hugging Face configuration declares the Gemma4UnifiedForConditionalGeneration architecture.

Fine-Tune Gemma-4-12B

Follow the installation instructions, then run the recipe from the repository root:

automodel examples/llm_finetune/gemma/gemma_4_12b_hellaswag.yaml --nproc-per-node 8

Train on Long Sequences with Context Parallelism

Gemma-4-12B supports context parallelism (distributed.cp_size > 1). Each rank keeps a contiguous slice of the sequence, and the sliding-window and full-attention layers exchange keys and values over a p2p ring:

automodel examples/llm_finetune/gemma/gemma_4_12b_tulu3_cp2.yaml --nproc-per-node 8

For 64K-token sequences on a single 8-GPU node, gemma_4_12b_tulu3_cp8_64k.yaml packs Tulu-3 conversations into 64K-token sequences (THD) and shards each over cp_size: 8 (8K tokens per rank):

automodel examples/llm_finetune/gemma/gemma_4_12b_tulu3_cp8_64k.yaml --nproc-per-node 8

The CP recipes use FusedLinearCrossEntropy, which computes the loss from the hidden states instead of materializing the 262K-vocab logits for every token of the shard. Longer sequences take more ranks at the same 8K-token shard length rather than longer per-rank shards, which exceed 80 GB per GPU with this model: gemma_4_12b_tulu3_cp16_128k.yaml trains 128K-token sequences over two nodes (cp_size: 16), launched with torchrun on each node:

torchrun --nnodes 2 --node-rank <0|1> --master-addr <node-0 address> --master-port 29500 --nproc-per-node 8 examples/llm_finetune/finetune.py --config examples/llm_finetune/gemma/gemma_4_12b_tulu3_cp16_128k.yaml

Context parallelism needs attn_implementation: sdpa, which the ring attention replaces. The CP recipes set text_config.cp_full_attn_backend: ffpa, which runs the head_dim=512 full-attention ring chunks through the FFPA kernel and needs the ffpa extra (uv sync --extra ffpa); remove the key to use Flex there instead. The 64K and 128K recipes also set text_config.cp_sliding_attn_backend: fa, which runs the sliding-window layers through FlashAttention and needs the fa extra; without it they use Flex. The base checkpoint ships no chat template, so the Tulu-3 recipes load the tokenizer of google/gemma-4-12B-it, which has the same vocabulary and the Gemma 4 chat template. Context parallelism covers text and image inputs; video and audio inputs are rejected under CP. Image tokens marked by mm_token_type_ids attend bidirectionally on the sliding-window layers only, with and without CP.

Choose a Workflow

GoalStart Here
Run the primary recipe for this modelUse the recipe configuration.
Train on long sequences with context parallelismUse the CP recipe configuration.
Train on 64K-token sequences on one nodeUse the 64K CP recipe configuration.
Train on 128K-token sequences on two nodesUse the 128K CP recipe configuration.
Prepare your environmentFollow the installation instructions.

Configuration

SettingConfiguration
Hardware-
StrategyFSDP2; tp_size=1, pp_size=1, cp_size=1
Nodes-
FeaturesCheckpointing (enabled=true)
Advancedbf16

Model Reference

Model Architecture

PropertyValue
TaskText generation
ArchitectureGemma4UnifiedForConditionalGeneration

Available Models