DeepSeek V3#
DeepSeek-V3 is a large-scale Mixture-of-Experts (MoE) language model with 671B total parameters and 37B activated parameters per token. It features Multi-head Latent Attention (MLA), innovative load balancing strategies, and Multi-Token Prediction (MTP) for improved training efficiency. DeepSeek-V3 achieves state-of-the-art performance while maintaining economical training costs. More information is available in the technical report “DeepSeek-V3 Technical Report”.
DeepSeek V3 models are supported via the Bridge system with auto-detected configuration and weight mapping.
Verified configurations#
Choose an exact recorded configuration to see its command and expected result. These selectors are generated from the authoritative verification cards and never synthesize combinations.
Run a configuration#
Choose a workflow, precision, and exact recorded combination. The command and expected result update below.
Import · CPU
○ UnverifiedExact command
No runnable command is recorded for this status.
Expected result
A CPU import of the pinned blockwise-FP8 Hugging Face checkpoint must dequantize every mapped weight correctly, persist a reloadable Megatron checkpoint, and pass a complete tensor audit. This workflow is deferred by this card.
Import · GPU
✓ VerifiedExact command
./scripts/conversion/convert.sh import --executor slurm --device gpu --nodes 4 --gpus-per-node 8 --hf-model deepseek-ai/DeepSeek-V3 --hf-revision e815299b0bcbac849fa540c768ef21845365c9eb --megatron-path work/model-verification/deepseek-v3/imported-megatron --torch-dtype bfloat16 --low-memory-save --distributed-timeout-minutes 240 --tp 1 --pp 4 --ep 8 --etp 1Expected result
The 32-GPU import completed all 4,583 mapping tasks, blockwise-dequantized the pinned FP8 source into BF16, and persisted a reloadable 32-shard distributed Megatron checkpoint. A strict paired export subsequently matched all 46,183 model tensors and 684,489,845,504 values exactly.
Export · CPU
✓ VerifiedExact command
./scripts/conversion/convert.sh export --executor slurm --device cpu --nodes 4 --cpu-processes-per-node 8 --cpus-per-task 16 --mem 0 --exclusive --hf-model deepseek-ai/DeepSeek-V3 --hf-revision e815299b0bcbac849fa540c768ef21845365c9eb --megatron-path work/model-verification/deepseek-v3/imported-megatron/iter_0000000 --hf-path work/model-verification/deepseek-v3/cpu-hf-export --torch-dtype bfloat16 --export-weight-dtype bfloat16 --tp 1 --pp 4 --ep 8 --etp 1 --distributed-timeout-minutes 240 --distributed-save --save-every-n-ranks 1 --no-progressExpected result
The 32-process distributed CPU export exits successfully and writes all 163 source-shaped shards. The index contains exactly 46,183 BF16 model tensors; an exhaustive audit matches all 684,489,845,504 values against the pinned source after blockwise dequantization of all 45,808 FP8 tensors and 59 other declared dtype normalizations. Transformers strictly reloads the output as DeepseekV3ForCausalLM with all 967 state tensors and no loading discrepancies.
Export · GPU
✓ VerifiedExact command
./scripts/conversion/convert.sh export --executor slurm --device gpu --nodes 4 --gpus-per-node 8 --hf-model deepseek-ai/DeepSeek-V3 --hf-revision e815299b0bcbac849fa540c768ef21845365c9eb --megatron-path work/model-verification/deepseek-v3/imported-megatron/iter_0000000 --hf-path work/model-verification/deepseek-v3/hf-export --torch-dtype bfloat16 --export-weight-dtype bfloat16 --distributed-save --save-every-n-ranks 1 --distributed-timeout-minutes 240 --tp 1 --pp 4 --ep 8 --etp 1Expected result
The command exited successfully and wrote all 163 source-shaped shards. The index contained exactly 46,183 BF16 model tensors; an exhaustive 32-rank audit matched all 684,489,845,504 values against the pinned source after blockwise dequantization of all 45,808 FP8 tensors. The exported configuration instantiated natively as DeepseekV3ForCausalLM.
Pretrain · GB300
○ UnverifiedRecorded metrics
- Initial loss
- None
- Final loss
- None
- Step time · last 10 avg
- None ms
- Model throughput · last 10 avg
- None TFLOP/s/GPU
- Token throughput · last 10 avg
- None tokens/s/GPU
Exact command
No runnable command is recorded for this status.
Expected result
A bounded public-data DeepSeek V3 pretraining run must complete with finite loss, zero skipped or NaN iterations, all five metrics, a saved post-setup configuration, and a reloadable final checkpoint. Functional training is deferred by this card.
SFT · GB300
○ UnverifiedRecorded metrics
- Initial loss
- None
- Final loss
- None
- Step time · last 10 avg
- None ms
- Model throughput · last 10 avg
- None TFLOP/s/GPU
- Token throughput · last 10 avg
- None tokens/s/GPU
Exact command
No runnable command is recorded for this status.
Expected result
A pinned-data full-SFT run must finish with finite loss, zero skipped or NaN iterations, all five metrics, a saved post-setup configuration, and a reloadable final checkpoint. Fine-tuning is deferred by this card.
Long Context · GB300
○ UnverifiedRecorded metrics
- Initial loss
- None
- Final loss
- None
- Step time · last 10 avg
- None ms
- Model throughput · last 10 avg
- None TFLOP/s/GPU
- Token throughput · last 10 avg
- None tokens/s/GPU
Exact command
No runnable command is recorded for this status.
Expected result
A dedicated packed long-context SFT run must complete with context parallelism, finite loss, zero skipped or NaN iterations, all five metrics, and a reloadable checkpoint. This workflow is deferred by this card.
LoRA · GB300
○ UnverifiedRecorded metrics
- Initial loss
- None
- Final loss
- None
- Step time · last 10 avg
- None ms
- Model throughput · last 10 avg
- None TFLOP/s/GPU
- Token throughput · last 10 avg
- None tokens/s/GPU
Exact command
No runnable command is recorded for this status.
Expected result
A DeepSeek V3 PEFT recipe with an audited model-native attention adapter target set must complete with finite loss, zero skipped or NaN iterations, all five metrics, and a reloadable adapter checkpoint. This workflow is deferred by this card.
Benchmark · H100
✓ VerifiedRecorded metrics
- Initial loss
- 11.89496
- Final loss
- 6.815285
- Step time · last 10 avg
- 61,756.130 ms
- Model throughput · last 10 avg
- 276.030 TFLOP/s/GPU
- Token throughput · last 10 avg
- 1,061.206 tokens/s/GPU
Exact command
./scripts/training/train.sh --wait --nodes 128 --gpus-per-node 8 --recipe deepseek_v3_pretrain_1024gpu_h100_bf16_config --mode pretrain --max_steps 50 --seq_length 4096 logger.save_config_filepath=work/model-verification/deepseek-v3/h100-performance/ConfigContainer.yamlExpected result
On exactly 1,024 H100 GPUs, the canonical 50-step mock-data BF16 recipe completed at TP2/PP8/VP4/CP1/EP64/ETP1 and GBS/MBS 16384/1. All 50 keyed rows had finite loss and throughput with zero skipped or NaN iterations, and the resolved configuration was saved. Steps 41-50 averaged 61,756.130 ms and 276.030 model TFLOP/s/GPU, passing gates of at most 62,000 ms and at least 275 TFLOP/s/GPU. Mock data and forced expert balancing make this benchmark-only evidence, not convergence evidence.
Benchmark · GB200
✓ VerifiedRecorded metrics
- Initial loss
- 11.89246
- Final loss
- 6.311604
- Step time · last 10 avg
- 13,841.180 ms
- Model throughput · last 10 avg
- 1,231.040 TFLOP/s/GPU
- Token throughput · last 10 avg
- 4,734.856 tokens/s/GPU
Exact command
./scripts/training/train.sh --wait --nodes 64 --gpus-per-node 4 --recipe deepseek_v3_pretrain_256gpu_gb200_fp8mx_config --mode pretrain --max_steps 50 --seq_length 4096 logger.save_config_filepath=work/model-verification/deepseek-v3/gb200-performance/ConfigContainer.yamlExpected result
On exactly 256 GB200 GPUs, the canonical 50-step mock-data MXFP8 recipe completed at TP1/PP4/VP4/CP1/EP64/ETP1 and GBS/MBS 4096/1. All 50 keyed rows had finite loss and throughput with zero skipped or NaN iterations, and the resolved configuration was saved. Steps 41-50 averaged 13,841.180 ms and 1,231.040 model TFLOP/s/GPU, passing gates of at most 14,000 ms and at least 1,200 TFLOP/s/GPU. Mock data, forced expert balancing, and reduced-precision optimizer moments make this benchmark-only evidence, not convergence evidence.
Benchmark · GB300
✓ VerifiedRecorded metrics
- Initial loss
- 11.89526
- Final loss
- 6.236621
- Step time · last 10 avg
- 10,667.300 ms
- Model throughput · last 10 avg
- 1,597.310 TFLOP/s/GPU
- Token throughput · last 10 avg
- 6,143.635 tokens/s/GPU
Exact command
./scripts/training/train.sh --wait --nodes 64 --gpus-per-node 4 --recipe deepseek_v3_pretrain_256gpu_gb300_fp8mx_config --mode pretrain --max_steps 50 --seq_length 4096 logger.save_config_filepath=work/model-verification/deepseek-v3/gb300-performance/ConfigContainer.yamlExpected result
On exactly 256 GB300 GPUs, the canonical 50-step mock-data MXFP8 recipe completed at TP1/PP2/VP8/CP1/EP32/ETP1 and GBS/MBS 4096/1. All 50 keyed rows had finite loss and throughput with zero skipped or NaN iterations, and the resolved configuration was saved. Steps 41-50 averaged 10,667.300 ms and 1,597.310 model TFLOP/s/GPU, passing gates of at most 11,000 ms and at least 1,550 TFLOP/s/GPU. Mock data, forced expert balancing, and reduced-precision optimizer moments make this benchmark-only evidence, not convergence evidence.
Benchmark · HSDP · GB300
✓ VerifiedRecorded metrics
- Initial loss
- 11.89824
- Final loss
- 5.146079
- Step time · last 10 avg
- 3,197.040 ms
- Model throughput · last 10 avg
- 1,332.420 TFLOP/s/GPU
- Token throughput · last 10 avg
- 5,124.740 tokens/s/GPU
Exact command
./scripts/training/train.sh --wait --nodes 64 --gpus-per-node 4 --recipe deepseek_v3_pretrain_256gpu_gb300_fp8mx_hsdp_config --mode pretrain --max_steps 50 --seq_length 4096 logger.save_config_filepath=work/model-verification/deepseek-v3/gb300-hsdp-performance/ConfigContainer.yamlExpected result
On exactly 256 GB300 GPUs, the 50-step mock-data MXFP8 Megatron Hybrid FSDP recipe completed at TP1/PP1/VP1/CP1/EP64/ETP1 and GBS/MBS 1024/1. Dense parameters use ZeRO-2 optim_grads sharding while expert parameters use optim_grads_params across four distributed optimizer instances, with full-iteration CUDA graphs and MoE paged stashing enabled. All 50 keyed rows had finite loss and throughput with zero skipped or NaN iterations. Steps 41-50 averaged 3,197.040 ms and 1,332.420 model TFLOP/s/GPU. Do not compare this item against pretrain_performance.GB300: that recipe runs a different convergence contract at GBS 4096 with PP2/VP8/EP32, so step time and per-GPU throughput are not commensurable. Mock data, forced expert balancing, and reduced-precision optimizer moments make this benchmark-only evidence, not convergence evidence.