Model CoverageDiffusion Language Models

DiffusionGemma

View as Markdown

DiffusionGemma is a block-diffusion language model from Google. Instead of generating tokens left-to-right, it denoises a fixed-length canvas of tokens in parallel: a causal encoder reads the prompt and a bidirectional decoder iteratively refines the response canvas. The released checkpoint is a Mixture-of-Experts model with 26B total parameters and ~4B active per token.

TaskText Generation (Block Diffusion, MoE)
ArchitectureDiffusionGemmaForBlockDiffusion
Parameters26B total / ~4B active
HF Orggoogle

Available Models

  • DiffusionGemma 26B-A4B-it (DiffusionGemmaForBlockDiffusion): instruction-tuned block-diffusion MoE.

Architectures

  • DiffusionGemmaForBlockDiffusion — block-diffusion MoE (causal prompt encoder + bidirectional canvas decoder).

Example HF Models

ModelHF ID
DiffusionGemma 26B-A4B-itgoogle/diffusiongemma-26B-A4B-it

Example Recipes

RecipeDescription
diffusion_gemma_sft.yamlFull SFT — DiffusionGemma 26B-A4B with FSDP2 + Expert Parallelism
diffusion_gemma_lora.yamlLoRA SFT — DiffusionGemma 26B-A4B with FSDP2 + Expert Parallelism

Try with NeMo AutoModel

1. Install (full instructions):

$pip install nemo-automodel

2. Clone the repo to get the example recipes:

$git clone https://github.com/NVIDIA-NeMo/Automodel.git
$cd Automodel

This recipe was validated with Expert Parallelism (EP=8) on a single 8×H100 node. See the Launcher Guide for multi-node setup.

3. Run the recipe from inside the repo:

$automodel --nproc-per-node=8 examples/dllm_sft/diffusion_gemma_sft.yaml

Fine-Tuning

See the DiffusionGemma Fine-Tuning Guide for the block-diffusion training objective, self-conditioning, and the full list of supported features (SFT, LoRA, Expert Parallelism, activation checkpointing).

Hugging Face Model Cards