DeepSeek-V4-Flash | NVIDIA NeMo AutoModel

DeepSeek-V4 Flash is DeepSeek’s latest fine-grained Mixture-of-Experts language model. It uses a 43-layer all-MoE backbone with 256 routed experts plus one shared expert per block, top-6 routing, and a hybrid per-layer attention zoo (SWA / CSA / HCA) selectable through compress_ratios. The first num_hash_layers blocks use a hash-clustering gate, and every block maintains hc_mult=4 Hyper-Connection streams mixed via a learned col-norm-first Sinkhorn router.

Set up NeMo AutoModel with the latest container or follow the installation instructions.

Fine-Tune DeepSeek-V4-Flash

The full 43-layer schedule requires a multi-node run; see the recipe yaml header for ep_size / pp_size guidance. See the Launcher Guide for multi-node setup.

From the repository root, run:

uv run automodel --nproc-per-node=8 examples/llm_finetune/deepseek_v4/deepseek_v4_flash_hellaswag.yaml

Choose a Workflow

Goal	Start Here
Supervised fine-tuning (SFT) - DeepSeek-V4 Flash on HellaSwag with pipeline parallelism	Use `deepseek_v4_flash_hellaswag.yaml`.

Model Reference

Model Architecture

Property	Value
Task	Text Generation (MoE)
Architecture	`DeepseekV4ForCausalLM`
Experts	fine-grained MoE, 256 routed + 1 shared expert
Hugging Face Organization	deepseek-ai

Available Models

Model	HF ID
DeepSeek-V4 Flash	`deepseek-ai/DeepSeek-V4-Flash`