DeepSeek-V4 Pro

DeepSeek-V4 Pro is a fine-grained mixture-of-experts language model supported by NeMo AutoModel’s native DeepSeek-V4 implementation.

Example Model and Recipe

Run the Recipe

This configuration uses PP8 and EP64 across 512 GPUs (64 nodes with 8 H100 GPUs per node), so it is not a single-node example. Launch it through Slurm or an external torchrun environment using the topology declared in the recipe. For example, after setting the rendezvous host and port for your cluster:

$
torchrun --nproc-per-node=8 \
>
  --nnodes=64 \
>
  --rdzv-backend=c10d \
>
  --rdzv-endpoint="${MASTER_ADDR}:${MASTER_PORT}" \
>
  -m nemo_automodel.recipes.llm.train_ft \
>
  -c examples/llm_finetune/deepseek_v4/deepseek_v4_pro_hellaswag_all_tilelang_pp8_ep64_20steps.yaml

See the Launcher Guide for cluster-specific setup and the Large MoE Fine-Tuning Guide for the full multi-node environment setup.