> For clean Markdown of any page, append .md to the page URL.
> For a complete documentation index, see https://docs.nvidia.com/nemo/automodel/llms.txt.
> For AI client integration (Claude Code, Cursor, etc.), connect to the MCP server at https://docs.nvidia.com/nemo/automodel/_mcp/server.

# DeepSeek-V4 Pro

> Supported DeepSeek-V4 Pro checkpoint and NeMo AutoModel recipes.

[DeepSeek-V4 Pro](https://huggingface.co/deepseek-ai/DeepSeek-V4-Pro) is a fine-grained mixture-of-experts language model supported by NeMo AutoModel's native DeepSeek-V4 implementation.

|                  |                                                   |
| ---------------- | ------------------------------------------------- |
| **Task**         | Text Generation (MoE)                             |
| **Architecture** | `DeepseekV4ForCausalLM`                           |
| **Experts**      | 384 routed, 1 shared; 6 routed experts per token  |
| **HF Org**       | [deepseek-ai](https://huggingface.co/deepseek-ai) |

## Example Model and Recipe

| Model                                                                               | Recipe                                                                                                                                                                           |
| ----------------------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| [`deepseek-ai/DeepSeek-V4-Pro`](https://huggingface.co/deepseek-ai/DeepSeek-V4-Pro) | [TileLang PP8/EP64 HellaSwag](https://github.com/NVIDIA-NeMo/Automodel/blob/main/examples/llm_finetune/deepseek_v4/deepseek_v4_pro_hellaswag_all_tilelang_pp8_ep64_20steps.yaml) |

## Run the Recipe

This configuration uses PP8 and EP64 across 512 GPUs (64 nodes with 8 H100 GPUs
per node), so it is not a single-node example. Launch it through Slurm or an
external `torchrun` environment using the topology declared in the recipe. For
example, after setting the rendezvous host and port for your cluster:

```bash
torchrun --nproc-per-node=8 \
  --nnodes=64 \
  --rdzv-backend=c10d \
  --rdzv-endpoint="${MASTER_ADDR}:${MASTER_PORT}" \
  -m nemo_automodel.recipes.llm.train_ft \
  -c examples/llm_finetune/deepseek_v4/deepseek_v4_pro_hellaswag_all_tilelang_pp8_ep64_20steps.yaml
```

See the [Launcher Guide](/job-launchers/slurm-cluster) for cluster-specific
setup and the [Large MoE Fine-Tuning Guide](/recipes-e2e-examples/large-moe-fine-tuning)
for the full multi-node environment setup.