Model CoverageVision Language Models

Gemma 4

View as Markdown

Gemma 4 is Google’s next-generation multimodal Gemma family, supporting image-text inputs with a Mixture-of-Experts (MoE) language backbone at larger scales. NeMo AutoModel replaces the HF-native dense matmul over experts with the NeMo GroupedExperts backend, enabling Expert Parallelism (EP) through the standard MoE parallelizer.

TaskImage-Text-to-Text
ArchitectureGemma4ForConditionalGeneration
Parameters2B – 31B (dense) · 26B-A4B (MoE)
HF Orggoogle

Available Models

  • Gemma 4 E2B IT (VL, dense, kv-shared layers)
  • Gemma 4 E4B IT (VL, dense, kv-shared layers), Gemma 4 E4B IT Assistant (Assistant/drafter model for MTP)
  • Gemma 4 31B IT (VL, dense)
  • Gemma 4 26B-A4B IT (VL, MoE)

Architecture

  • Gemma4ForConditionalGeneration
  • Gemma4AssistantForCausalLM (MTP drafter / assistant head for speculative decoding; co-trainable with the target Gemma 4 base using Gemma4WithDrafter)

Example HF Models

ModelHF ID
Gemma 4 E2B ITgoogle/gemma-4-E2B-it
Gemma 4 E4B ITgoogle/gemma-4-E4B-it
Gemma 4 31B ITgoogle/gemma-4-31B-it
Gemma 4 26B-A4B IT (MoE)google/gemma-4-26B-A4B-it

Example Recipes

RecipeDescription
gemma4_2b.yamlSFT — Gemma 4 E2B on MedPix
gemma4_2b_peft.yamlLoRA — Gemma 4 E2B on MedPix
gemma4_4b.yamlSFT — Gemma 4 E4B on MedPix
gemma4_4b_peft.yamlLoRA — Gemma 4 E4B on MedPix
gemma4_31b.yamlSFT — Gemma 4 31B on MedPix
gemma4_31b_peft.yamlLoRA — Gemma 4 31B on MedPix
gemma4_31b_tp4.yamlSFT — Gemma 4 31B with TP=4
gemma4_31b_tp4_pp2.yamlSFT — Gemma 4 31B with TP=4, PP=2
gemma4_31b_tp4_pp4.yamlSFT — Gemma 4 31B with TP=4, PP=4 (multi-node)
gemma4_26b_a4b_moe.yamlSFT — Gemma 4 26B-A4B MoE on MedPix
gemma4_26b_a4b_moe_peft.yamlLoRA — Gemma 4 26B-A4B MoE on MedPix

Try with NeMo AutoModel

1. Install (full instructions):

$pip install nemo-automodel

2. Clone the repo to get the example recipes:

$git clone https://github.com/NVIDIA-NeMo/Automodel.git
$cd Automodel

3. Run the recipe from inside the repo:

$automodel --nproc-per-node=8 examples/vlm_finetune/gemma4/gemma4_4b.yaml

1. Pull the container and mount a checkpoint directory:

$docker run --gpus all -it --rm \
>  --shm-size=8g \
>  -v $(pwd)/checkpoints:/opt/Automodel/checkpoints \
>  nvcr.io/nvidia/nemo-automodel:26.06.00

2. Navigate to the AutoModel directory (where the recipes are):

$cd /opt/Automodel

3. Run the recipe:

$automodel --nproc-per-node=8 examples/vlm_finetune/gemma4/gemma4_4b.yaml

See the Installation Guide and VLM Fine-Tuning Guide.

Fine-Tuning

See the VLM Fine-Tuning Guide.

Hugging Face Model Cards