DeepSeek-V3#

DeepSeek-V3 is a large-scale Mixture-of-Experts model with 671B total parameters and 37B activated per token. It features Multi-head Latent Attention (MLA), innovative load balancing, and Multi-Token Prediction (MTP). DeepSeek-V3.2 is an updated release with further improvements.

Moonlight by Moonshot AI also uses this architecture with 16B total / 3B activated parameters.

Task

Text Generation (MoE)

Architecture

DeepseekV3ForCausalLM / DeepseekV32ForCausalLM

Parameters

671B total / 37B active

HF Org

deepseek-ai

Available Models#

  • DeepSeek-V3: 671B total, 37B activated

  • DeepSeek-V3.2 (DeepseekV32ForCausalLM): updated architecture

  • Moonlight-16B-A3B (Moonshot AI): 16B total, 3B activated

Architectures#

  • DeepseekV3ForCausalLM

  • DeepseekV32ForCausalLM

Example HF Models#

Model

HF ID

DeepSeek-V3

deepseek-ai/DeepSeek-V3

DeepSeek-V3-Base

deepseek-ai/DeepSeek-V3-Base

DeepSeek-V3.2

deepseek-ai/DeepSeek-V3.2

Moonlight 16B A3B

moonshotai/Moonlight-16B-A3B

Example Recipes#

Recipe

Description

deepseek_v32_hellaswag_pp.yaml

SFT — DeepSeek-V3.2 on HellaSwag with pipeline parallelism

moonlight_16b_te.yaml

SFT — Moonlight 16B with Transformer Engine

moonlight_16b_te_packed_sequence.yaml

SFT — Moonlight 16B with packed sequences

Try with NeMo AutoModel#

1. Install (full instructions):

pip install nemo-automodel

2. Clone the repo to get the example recipes:

git clone https://github.com/NVIDIA-NeMo/Automodel.git
cd Automodel

Note

This recipe was validated on 32 nodes × 8 GPUs (256 H100s). See the Launcher Guide for multi-node setup.

3. Run the recipe from inside the repo:

automodel --nproc-per-node=8 examples/llm_finetune/deepseek_v32/deepseek_v32_hellaswag_pp.yaml
Run with Docker

1. Pull the container and mount a checkpoint directory:

docker run --gpus all -it --rm \
  --shm-size=8g \
  -v $(pwd)/checkpoints:/opt/Automodel/checkpoints \
  nvcr.io/nvidia/nemo-automodel:26.02.00

2. Navigate to the AutoModel directory (where the recipes are):

cd /opt/Automodel

3. Run the recipe:

automodel --nproc-per-node=8 examples/llm_finetune/deepseek_v32/deepseek_v32_hellaswag_pp.yaml

See the Installation Guide and LLM Fine-Tuning Guide.

Fine-Tuning#

See the LLM Fine-Tuning Guide and the Large MoE Fine-Tuning Guide.

Hugging Face Model Cards#