DeepSeek-V4-Pro
DeepSeek-V4 Pro is a fine-grained mixture-of-experts language model supported by NeMo AutoModel’s native DeepSeek-V4 implementation.
Fine-Tune DeepSeek-V4-Pro
This configuration uses PP8 and EP64 across 512 GPUs (64 nodes with 8 H100 GPUs
per node), so it is not a single-node example. Launch it through Slurm or an
external torchrun environment using the topology declared in the recipe. For
example, after setting the rendezvous host and port for your cluster:
See the Launcher Guide for cluster-specific setup and the Large MoE Fine-Tuning Guide for the full multi-node environment setup.
Notes for Cold Starts and Large Rank Counts
- First step is long. On 64 H100 nodes a cold start spends 38-40 minutes before step 0:
each rank JIT-compiles its HybridEP and TileLang kernels, and pipeline parallelism runs those
compiles stage after stage. The recipes therefore set
dist_env.timeout_minutes: 120. PointNEMO_HYBRIDEP_JIT_CACHEat a shared directory to reuse the HybridEP kernels across runs. - Resolve the Hub once, not 512 times. With hundreds of ranks resolving
deepseek-ai/DeepSeek-V4-Proand the dataset online at start-up, the anonymous Hugging Face rate limit (HTTP 429) is hit andhuggingface_hubretries with long back-offs. Pre-download the snapshot and the dataset once (huggingface_hub.snapshot_download,datasets.load_dataset) into a sharedHF_HOMEand run withHF_HUB_OFFLINE=1; the recipe then starts with zero Hub requests.