Performance Summary

Training performance measurements for language, vision-language, and image-generation models using NeMo AutoModel with the PyTorch backend. Model names link to pinned YAML recipes.

Fine-Tuning Performance

Full Fine-Tuning: NVIDIA H100 80GB, Precision: BF16

Model	Workload	#GPUs	GBS	MBS	LBS	GA	Seq Length	TP	PP	CP	EP	VP	FSDP	Kernel Optimizations	Time per Global Step (s)	Model TFLOPs/sec/GPU	Tokens/sec	Tokens/sec/GPU
Qwen3.8-27B	MedPix VLM	8	8	1	1	1	Variable	1	1	1	1	-	NR	SDPA + torch	1.732	NR	1,865	233.07
GLM-5.3	Tulu-3 packed	256	256	1	4	1	4096	1	4	1	64	NR	NR	cuDNN + torch + HybridEP	20.584	NR	41,615	162.56
GLM-5.3-Flash	MedPix packed	72	144	1	1	4	2048	1	1	2	72	-	NR	cuDNN + torch + torch_mm + HybridEP	17.094	NR	9,480	131.66
DeepSeek-V4-Flash-Vision-Exp	MedPix VLM	128	128	1	4	1	4096	1	4	1	32	NR	NR	TileLang + torch + torch_mm + HybridEP	11.178	NR	3,299	25.77
MiMo-V2.5-Pro	HellaSwag	144	288	1	36	1	Variable	1	18	1	8	NR	NR	SDPA + torch + torch_mm + DeepEP	6.028	NR	1,816	12.61
MiMo-V2.6-Flash-RL	Tulu-3 packed	64	64	1	1	1	4096	1	1	1	64	-	NR	TE attention + torch + torch_mm + HybridEP	5.221	NR	41,011	640.80
Nemotron-3.5-Lightning-30B-A3B-BF16	Tulu-3 packed	8	8	1	1	1	8192	1	1	1	8	-	NR	TE attention/linear + torch_mm + DeepEP	1.025	NR	55,167	6,895.85
Nemotron-3.5-Super-VL-120B-A12B-BF16	Tulu-3 packed (text only)	64	32	1	1	4	8192	1	1	8	64	-	NR	TE attention/linear	12.355	NR	14,737	230.26
Kimi-Linear-48B-A3B-Instruct	HellaSwag packed	8	8	1	1	1	4096	1	1	1	8	-	NR	eager attention + torch	2.862	NR	11,317	1,414.63
Laguna-XS-2.1	HellaSwag packed	8	8	1	1	1	1024	1	1	1	8	-	NR	SDPA + torch + torch_mm	1.323	NR	6,174	771.73

Qwen3.8-27B freezes vision; GLM-5.3-Flash and Nemotron-3.5-Super-VL freeze vision and embeddings.
Nemotron-3.5-Super-VL uses text-only checkpoint nvidia/NVIDIA-Nemotron-3.5-Super-midtrain-67B-vision-pretrained.
Nemotron-3.5-Lightning uses all 52 layers with MTP disabled.

Full Fine-Tuning: NVIDIA GB200, Precision: BF16

Model	Workload	#GPUs	GBS	MBS	LBS	GA	Seq Length	TP	PP	CP	EP	VP	FSDP	Kernel Optimizations	Time per Global Step (s)	Model TFLOPs/sec/GPU	Tokens/sec	Tokens/sec/GPU
DeepSeek-V4.1-Flash	MedPix VLM	64	64	1	1	1	4096	1	1	1	64	-	NR	TileLang + torch + torch_mm + HybridEP	3.585	NR	5,473	85.52
MiMo-V2.6-Pro-RL	Tulu-3 packed	128	16	1	2	1	65536	1	2	8	64	NR	NR	TE attention/linear + torch_mm + HybridEP	16.973	NR	58,391	456.18

FSDP2; MiMo-V2.6-Pro-RL includes profiler overhead.

LoRA Performance

LoRA: NVIDIA H100 80GB, Precision: BF16

Model	Workload	#GPUs	GBS	MBS	LBS	GA	Seq Length	TP	PP	CP	EP	VP	FSDP	Kernel Optimizations	Time per Global Step (s)	Model TFLOPs/sec/GPU	Tokens/sec	Tokens/sec/GPU
Llama3 8B	Mock	1	32	2	2	16	4096	1	1	1	-	1	1	TE + triton	10.51	402	12,473	12472.87
Qwen2.5 7B	Mock	1	32	2	2	16	4096	1	1	1	-	1	1	TE + triton	9.29	423	14,110	14110.05
Llama3 70B	Mock	8	32	2	2	4	4096	2	1	1	-	1	4	TE + triton + fsdp2_prefetch	15.00	316	8,735	1091.85
Qwen2.5 32B	Mock	8	32	2	2	4	4096	2	1	1	-	1	4	TE + triton + fsdp2_prefetch	7.28	301	18,002	2250.31
Llama3 70B 2-node	Mock	16	32	2	2	2	4096	2	1	1	-	1	8	TE + triton + fsdp2_prefetch	8.32	285	15,758	984.85
Qwen2.5 32B 2-node	Mock	16	32	2	2	2	4096	2	1	1	-	1	8	TE + triton + fsdp2_prefetch	3.95	277	33,166	2072.89

Full-length mock sequences; no padding.

Pre-Training Performance

GPU: NVIDIA H100 80GB, Precision: BF16

Model	Workload	#GPUs	GBS	MBS	LBS	GA	Seq Length	TP	PP	CP	EP	VP	FSDP	Kernel Optimizations	Time per Global Step (s)	Model TFLOPs/sec/GPU	Tokens/sec	Tokens/sec/GPU
Nemotron V3 Super 120B	Mock	64	512	2	2	4	4096	1	1	1	64	-	64	TE + DeepEP + TorchSDPA	7.286	334	287,808	4,497
Nemotron V3 Nano 30B	Mock	8	512	4	4	16	4096	1	1	1	8	-	8	TE + DeepEP + TorchSDPA	15.614	328	134,312	16,789
Nemotron V3 Ultra 550B	Mock	128	256	2	2	1	4096	1	1	1	64	-	128	TE + DeepEP + torch_mm	10.05	293	104,320	815
DeepSeek V3 671B	Mock	1024	8192	1	8	4	4096	1	4	1	64	8	256	TE + DeepEP	37.87	216	885,760	865
DeepSeek V3 671B	Mock	256	512	1	8	1	4096	1	4	1	64	8	64	TE + DeepEP	8.18	250	256,512	1,002
Kimi K2	Mock	256	512	1	8	2	4096	1	8	1	32	4	32	TE + DeepEP	8.86	189	236,544	924
Qwen3 MoE 30B	Mock	8	512	4	4	16	4096	1	1	1	8	-	8	TE + DeepEP	21.773	277	96,320	12,040
GPT-OSS 20B	Mock	8	256	2	2	16	4096	1	1	1	-	-	8	TE + DeepEP + FlexAttn	10.04	279	104,464	13,058
GPT-OSS 120B	Mock	64	512	2	2	4	4096	1	1	1	-	-	64	TE + DeepEP + FlexAttn	4.30	231	488,064	7,626
Llama3 70B	Mock	64	128	1	1	4	8192	1	1	2	-	-	32	TE + fsdp2_prefetch	18.90	389	55,473	866.77
Qwen3.8-Flash-Next	Mock, balanced routing	64	64	1	1	1	8192	1	1	1	64	-	64	FA4 QSA (`cute`) + torch + gmm + HybridEP	3.931	86.85	133,364	2,083.81

Full-length mock sequences; balanced MoE routing.
Qwen3.8-Flash-Next: 20 warmup and 30 measured steps; label-token rate; TFLOPs derived from MFU.

GPU: NVIDIA H100 80GB, Precision: FP8

Model	Workload	#GPUs	GBS	MBS	LBS	GA	Seq Length	TP	PP	CP	EP	VP	FSDP	Kernel Optimizations	Time per Global Step (s)	Model TFLOPs/sec/GPU	Tokens/sec	Tokens/sec/GPU
Moonlight 16B	Mock, balanced routing	8	256	4	4	8	2048	1	1	1	8	-	8	TE FP8 + DeepEP	3.006	324.94	174,414	21,801.73

TE current scaling; 10 warmup and 20 measured steps.

GPU: NVIDIA GB200, Precision: MXFP8

Model	Workload	#GPUs	GBS	MBS	LBS	GA	Seq Length	TP	PP	CP	EP	VP	FSDP	Kernel Optimizations	Time per Global Step (s)	Model TFLOPs/sec/GPU	Tokens/sec	Tokens/sec/GPU
DeepSeek V3 671B	Mock, balanced routing	64	128	2	2	1	4096	1	1	1	64	-	64	TE MXFP8 + HybridEP	2.986	685.45	175,582	2,743.47
DeepSeek V3 671B (compiled MLA)	Mock, balanced routing	64	128	2	2	1	4096	1	1	1	64	-	64	SDPA + compiled MLA + TE MXFP8 + HybridEP	6.044	338.57	86,745	1,355.39
Qwen3 MoE 30B	Mock, balanced routing	8	512	16	16	4	4096	1	1	1	8	-	8	TE MXFP8 + HybridEP	9.164	658.08	228,847	28,605.85
Qwen3 MoE 235B	Mock, balanced routing	32	8192	4	4	64	4096	1	1	1	32	-	32	TE MXFP8 + HybridEP	194.098	799.73	172,874	5,402.30
GPT-OSS 120B	Mock, balanced routing	32	1280	8	8	5	4096	1	1	1	16	-	32	TE MXFP8 + HybridEP	7.840	681.95	668,735	20,897.96

TE MXFP8 block scaling; 10 warmup and 20 measured steps, except Qwen3 MoE 235B (two warmup and three measured steps).
FP8/MXFP8 token rates count full sequences; TFLOPs derive from MFU and the configured peak.

Image Generation Performance

Image Generation, Full Fine-Tuning: NVIDIA H100 80GB, Precision: BF16

Model	Mode	#GPUs	GBS	MBS	LBS	GA	TP	PP	CP	EP	Optimizer	Time per Global Step (s)	Images/sec/GPU
Qwen-Image-2.1	Full fine-tuning	8	16	2	2	1	1	1	1	-	TE FusedAdam, FP32 master weights	~1.38	~1.45

Naruto BLIP, 1024 x 1024, FSDP2. Author-reported 1,000-step run; warmup exclusion unspecified.

Image Generation, LoRA: NVIDIA H100 80GB, Precision: BF16

Model	Mode	#GPUs	GBS	MBS	LBS	GA	TP	PP	CP	EP	Optimizer	Time per Global Step (s)	Images/sec/GPU
HunyuanImage-3.0	LoRA, rank 64	8	8	1	1	1	1	1	1	8	TE FusedAdam, 16-bit remainders	2.226	0.449

FSDP2; 100 steps, averaging steps 10-99. Cached 1024 x 1024 Naruto BLIP inputs; synchronized maximum-rank timing excludes setup and checkpoint I/O.

Glossary

NR: Not recorded or not verified
MFU: Model FLOPs Utilization - ratio of achieved compute to peak hardware capability
TP: Tensor Parallelism - splits individual layers across GPUs
PP: Pipeline Parallelism - splits model layers into stages
CP: Context Parallelism - splits the sequence across GPUs
FSDP: Fully Sharded Data Parallelism - the table column gives the shard degree when verified
EP: Expert Parallelism - distributes MoE experts across GPUs
DP: Data Parallelism - replicates model and splits data
VP: Virtual Pipeline - number of pipeline stages per GPU for interleaving
MBS: Micro-Batch Size - size of one forward pass in pipeline
LBS: Local Batch Size - size of one step per GPU
GBS: Global Batch Size - total batch size across all GPUs
GA: Gradient Accumulation - number of local-batches before optimizer step
TE: Transformer Engine kernel optimizations - RMSNorm, Linear and DotProductAttention
DeepEP: Expert-parallel communication library for MoE token dispatch and combine
FlexAttn: PyTorch’s Flex Attention

Configuration Files

Model names in the tables link directly to their YAML recipes. Mock-data benchmarks live in examples/llm_benchmark/; dataset-based recipes live in examples/llm_finetune/, examples/vlm_finetune/, examples/retrieval/, and examples/diffusion/finetune/. Use the recorded table settings and pinned recipes when reproducing a historical result.

Version Information

Last Updated: 2026-10-14
Default Recipe Snapshot: b203f3f9
Measurements: Historical development and benchmark runs; source commits vary.