DeepSeek-V3 | NVIDIA NeMo AutoModel

DeepSeek-V3 is a large-scale Mixture-of-Experts model with 671B total parameters and 37B activated per token. It features Multi-head Latent Attention (MLA), innovative load balancing, and Multi-Token Prediction (MTP). DeepSeek-V3.2 is an updated release with further improvements.

Moonlight by Moonshot AI also uses this architecture with 16B total / 3B activated parameters.

Set up NeMo AutoModel with the latest container or follow the installation instructions.

Model Reference

Model Architecture

Property	Value
Task	Text Generation (MoE)
Architecture	`DeepseekV3ForCausalLM` / `DeepseekV32ForCausalLM`
Parameters	671B total / 37B active
Hugging Face Organization	deepseek-ai

Available Models

Model	HF ID
DeepSeek-V3	`deepseek-ai/DeepSeek-V3`
DeepSeek-V3-Base	`deepseek-ai/DeepSeek-V3-Base`

LLM Fine-Tuning Guide
Large MoE Fine-Tuning Guide

Model Reference

Model Architecture

Available Models

Related Resources