Is this page helpful?

Support Matrix for NIMs#

This page lists supported models, deployment profiles, and verified hardware SKUs for NIM LLM and VLM. This page is organized according to model type:

Use the following sections to identify the supported deployment profiles for each model. Each NIM container image embeds one inference engine, vLLM or SGLang. The profile prefix (vllm- or sglang-) identifies the engine, and an image loads only profiles for its own engine. Profile strings follow a naming convention described in Model Profiles and Selection.

Optimized profiles are listed separately in the model sections because multiple configurations can share the same precision and tensor-parallel size. For selection instructions, refer to Model Profiles and Selection.

Important

The list of NIM Certified models is available on the NGC Catalog site.

Use the table below to filter NIM profiles by GPU, tensor parallelism (TP), precision, model type, LoRA support, and model name. Each row is one supported profile; details also appear in the per-model sections that follow.

Optimized profile entries identify their tuning objective and link to the full profile ID in the model section. The filters also apply to rows in the optimized profile tables below.

Note

The GB300-WS GPU is more commonly known as DGX Station.

ModelTPPrecisionLoRA
Apriel-Nemotron-15b-Thinker 1 BF16 No
Codestral-2501 1 BF16 No
gemma-4-26b-a4b-it 1 BF16 No
gemma-4-26b-a4b-it 2 BF16 No
gemma-4-26b-a4b-it 4 BF16 No
gemma-4-26b-a4b-it 1 NVFP4 No
gemma-4-31b-it 1 BF16 No
gemma-4-31b-it 2 BF16 No
gemma-4-31b-it 4 BF16 No
gemma-4-31b-it 2 INT4 No
gemma-4-31b-it 1 NVFP4 No
gemma-4-31b-it 2 NVFP4 No
gemma-4-31b-it 4 NVFP4 No
glm-5.2 4 FP8 No
glm-5.2 4 FP8 Yes
glm-5.2 8 FP8 No
glm-5.2 8 FP8 Yes
glm-5.2 4 NVFP4 No
glm-5.2 4 NVFP4 Yes
glm-5.2 8 NVFP4 No
glm-5.2 8 NVFP4 Yes
GLM-5.3 4 NVFP4 + FP8 No
GLM-5.3 8 NVFP4 + FP8 No
GLM-5.3 8 FP8 No
GLM-5.3-Flash 2 NVFP4 No
GLM-5.3-Flash 4 NVFP4 No
GLM-5.3-Flash 8 NVFP4 No
GLM-5.3-Flash 4 FP8 No
GLM-5.3-Flash 8 FP8 No
GLM-5.3-Flash 4 BF16 No
GLM-5.3-Flash 8 BF16 No
gpt-oss-120b 1 MXFP4 No
gpt-oss-120b 2 MXFP4 No
gpt-oss-120b 4 MXFP4 No
gpt-oss-120b 8 MXFP4 No
gpt-oss-120b 1 MXFP4 Yes
gpt-oss-120b 2 MXFP4 Yes
gpt-oss-120b 4 MXFP4 Yes
gpt-oss-120b 8 MXFP4 Yes
gpt-oss-20b 1 MXFP4 No
gpt-oss-20b 2 MXFP4 No
gpt-oss-20b 4 MXFP4 No
gpt-oss-20b 8 MXFP4 No
gpt-oss-20b 1 MXFP4 Yes
gpt-oss-20b 2 MXFP4 Yes
gpt-oss-20b 4 MXFP4 Yes
gpt-oss-20b 8 MXFP4 Yes
Inkling 4 NVFP4 No
Inkling 4 NVFP4 Yes
Inkling 8 NVFP4 No
Inkling 8 NVFP4 Yes
Inkling 8 NVFP4 No
Inkling 8 NVFP4 Yes
kimi-k2.6 4 NVFP4 No
kimi-k2.6 8 INT4 No
llama-3.1-70b-instruct 1 BF16 No
llama-3.1-70b-instruct 2 BF16 No
llama-3.1-70b-instruct 4 BF16 No
llama-3.1-70b-instruct 8 BF16 No
llama-3.1-70b-instruct 1 BF16 Yes
llama-3.1-70b-instruct 2 BF16 Yes
llama-3.1-70b-instruct 4 BF16 Yes
llama-3.1-70b-instruct 8 BF16 Yes
llama-3.1-70b-instruct 1 FP8 No
llama-3.1-70b-instruct 2 FP8 No
llama-3.1-70b-instruct 4 FP8 No
llama-3.1-70b-instruct 8 FP8 No
llama-3.1-70b-instruct 1 FP8 Yes
llama-3.1-70b-instruct 2 FP8 Yes
llama-3.1-70b-instruct 4 FP8 Yes
llama-3.1-70b-instruct 8 FP8 Yes
llama-3.1-70b-instruct 1 NVFP4 No
llama-3.1-70b-instruct 2 NVFP4 No
llama-3.1-70b-instruct 4 NVFP4 No
llama-3.1-70b-instruct 8 NVFP4 No
llama-3.1-70b-instruct 1 NVFP4 Yes
llama-3.1-70b-instruct 2 NVFP4 Yes
llama-3.1-70b-instruct 4 NVFP4 Yes
llama-3.1-70b-instruct 8 NVFP4 Yes
llama-3.1-8b-instruct 1 BF16 No
llama-3.1-8b-instruct 1 BF16 Yes
llama-3.1-8b-instruct 1 FP8 No
llama-3.1-8b-instruct 1 FP8 Yes
llama-3.1-8b-instruct 1 NVFP4 No
llama-3.1-8b-instruct 1 NVFP4 Yes
llama-3.3-70b-instruct 1 BF16 No
llama-3.3-70b-instruct 2 BF16 No
llama-3.3-70b-instruct 4 BF16 No
llama-3.3-70b-instruct 8 BF16 No
llama-3.3-70b-instruct 1 BF16 Yes
llama-3.3-70b-instruct 2 BF16 Yes
llama-3.3-70b-instruct 4 BF16 Yes
llama-3.3-70b-instruct 8 BF16 Yes
llama-3.3-70b-instruct 1 FP8 No
llama-3.3-70b-instruct 2 FP8 No
llama-3.3-70b-instruct 4 FP8 No
llama-3.3-70b-instruct 8 FP8 No
llama-3.3-70b-instruct 1 FP8 Yes
llama-3.3-70b-instruct 2 FP8 Yes
llama-3.3-70b-instruct 4 FP8 Yes
llama-3.3-70b-instruct 8 FP8 Yes
llama-3.3-70b-instruct 1 NVFP4 No
llama-3.3-70b-instruct 2 NVFP4 No
llama-3.3-70b-instruct 4 NVFP4 No
llama-3.3-70b-instruct 8 NVFP4 No
llama-3.3-70b-instruct 1 NVFP4 Yes
llama-3.3-70b-instruct 2 NVFP4 Yes
llama-3.3-70b-instruct 4 NVFP4 Yes
llama-3.3-70b-instruct 8 NVFP4 Yes
llama-3.3-nemotron-super-49b-v1.5 1 BF16 No
llama-3.3-nemotron-super-49b-v1.5 2 BF16 No
llama-3.3-nemotron-super-49b-v1.5 4 BF16 No
llama-3.3-nemotron-super-49b-v1.5 8 BF16 No
llama-3.3-nemotron-super-49b-v1.5 1 BF16 Yes
llama-3.3-nemotron-super-49b-v1.5 2 BF16 Yes
llama-3.3-nemotron-super-49b-v1.5 4 BF16 Yes
llama-3.3-nemotron-super-49b-v1.5 8 BF16 Yes
llama-3.3-nemotron-super-49b-v1.5 1 FP8 No
llama-3.3-nemotron-super-49b-v1.5 2 FP8 No
llama-3.3-nemotron-super-49b-v1.5 4 FP8 No
llama-3.3-nemotron-super-49b-v1.5 8 FP8 No
llama-3.3-nemotron-super-49b-v1.5 1 FP8 Yes
llama-3.3-nemotron-super-49b-v1.5 2 FP8 Yes
llama-3.3-nemotron-super-49b-v1.5 4 FP8 Yes
llama-3.3-nemotron-super-49b-v1.5 8 FP8 Yes
llama-3.3-nemotron-super-49b-v1.5 1 NVFP4 No
llama-3.3-nemotron-super-49b-v1.5 2 NVFP4 No
llama-3.3-nemotron-super-49b-v1.5 4 NVFP4 No
llama-3.3-nemotron-super-49b-v1.5 8 NVFP4 No
llama-3.3-nemotron-super-49b-v1.5 1 NVFP4 Yes
llama-3.3-nemotron-super-49b-v1.5 2 NVFP4 Yes
llama-3.3-nemotron-super-49b-v1.5 4 NVFP4 Yes
llama-3.3-nemotron-super-49b-v1.5 8 NVFP4 Yes
MiniMax M3.1 Flash 4 FP8 No
MiniMax M3.1 Flash 8 FP8 No
Mistral-Medium-3.5-128B 8 FP8 No
Mistral-Medium-3.5-128B 4 NVFP4 No
mistral-small-4-119b-2603 1 FP8 No
mistral-small-4-119b-2603 2 FP8 No
mistral-small-4-119b-2603 4 FP8 No
mistral-small-4-119b-2603 8 FP8 No
mistral-small-4-119b-2603 1 NVFP4 No
mistral-small-4-119b-2603 2 NVFP4 No
mistral-small-4-119b-2603 4 NVFP4 No
mistral-small-4-119b-2603 8 NVFP4 No
muse-glimmer 1 BF16 No
muse-glimmer 2 BF16 No
muse-glimmer 1 NVFP4 No
muse-glimmer 1 NVFP4 No
muse-glimmer 2 NVFP4 No
Muse Spark Open 8 NVFP4 No
nemotron-3-nano 1 BF16 No
nemotron-3-nano 2 BF16 No
nemotron-3-nano 4 BF16 No
nemotron-3-nano 8 BF16 No
nemotron-3-nano 1 BF16 Yes
nemotron-3-nano 2 BF16 Yes
nemotron-3-nano 4 BF16 Yes
nemotron-3-nano 8 BF16 Yes
nemotron-3-nano 1 FP8 No
nemotron-3-nano 2 FP8 No
nemotron-3-nano 4 FP8 No
nemotron-3-nano 8 FP8 No
nemotron-3-nano 1 NVFP4 No
nemotron-3-nano 2 NVFP4 No
nemotron-3-nano 4 NVFP4 No
nemotron-3-nano 8 NVFP4 No
nemotron-3-nano 1 NVFP4 Yes
nemotron-3-nano-omni-30b-a3b-reasoning 2 BF16 No
nemotron-3-nano-omni-30b-a3b-reasoning 1 FP8 No
nemotron-3-nano-omni-30b-a3b-reasoning 1 NVFP4 No
nemotron-3-super-120b-a12b 1 BF16 No
nemotron-3-super-120b-a12b 2 BF16 No
nemotron-3-super-120b-a12b 4 BF16 No
nemotron-3-super-120b-a12b 8 BF16 No
nemotron-3-super-120b-a12b 2 BF16 Yes
nemotron-3-super-120b-a12b 4 BF16 Yes
nemotron-3-super-120b-a12b 8 BF16 Yes
nemotron-3-super-120b-a12b 1 FP8 No
nemotron-3-super-120b-a12b 2 FP8 No
nemotron-3-super-120b-a12b 4 FP8 No
nemotron-3-super-120b-a12b 8 FP8 No
nemotron-3-super-120b-a12b 1 NVFP4 No
nemotron-3-super-120b-a12b 2 NVFP4 No
nemotron-3-super-120b-a12b 4 NVFP4 No
nemotron-3-super-120b-a12b 8 NVFP4 No
nemotron-3-super-120b-a12b 1 NVFP4 Yes
nemotron-3-super-120b-a12b 2 NVFP4 Yes
nemotron-3-ultra-550b-a55b 8 BF16 No
nemotron-3-ultra-550b-a55b 2 NVFP4 No
nemotron-3-ultra-550b-a55b 4 NVFP4 No
nemotron-3-ultra-550b-a55b 8 NVFP4 No
nemotron-3.5-bolt-e4b 1 BF16 No
nemotron-3.5-lightning 1 BF16 No
nemotron-3.5-lightning 1 BF16 Yes
nemotron-3.5-lightning 2 BF16 No
nemotron-3.5-lightning 2 BF16 Yes
nemotron-3.5-lightning 4 BF16 No
nemotron-3.5-lightning 4 BF16 Yes
nemotron-3.5-lightning 8 BF16 No
nemotron-3.5-lightning 8 BF16 Yes
nemotron-3.5-lightning 1 W4A16 No
nemotron-3.5-lightning 1 W4A16 Yes
nemotron-3.5-lightning 2 W4A16 No
nemotron-3.5-lightning 2 W4A16 Yes
nemotron-3.5-lightning 4 W4A16 No
nemotron-3.5-lightning 4 W4A16 Yes
nemotron-3.5-lightning 1 NVFP4 No
nemotron-3.5-lightning 1 NVFP4 Yes
nemotron-3.5-lightning 2 NVFP4 No
nemotron-3.5-lightning 2 NVFP4 Yes
nemotron-3.5-lightning 4 NVFP4 No
nemotron-3.5-lightning 4 NVFP4 Yes
nemotron-3.5-super-vl-120b-a12b 2 BF16 Yes
nemotron-3.5-super-vl-120b-a12b 4 BF16 Yes
nemotron-3.5-super-vl-120b-a12b 8 BF16 Yes
nemotron-3.5-super-vl-120b-a12b 1 NVFP4 No
nemotron-3.5-super-vl-120b-a12b 2 NVFP4 No
qwen3.5-122b-a10b 1 BF16 No
qwen3.5-122b-a10b 2 BF16 No
qwen3.5-122b-a10b 2 BF16 Yes
qwen3.5-122b-a10b 4 BF16 No
qwen3.5-122b-a10b 4 BF16 Yes
qwen3.5-122b-a10b 8 BF16 No
qwen3.5-122b-a10b 8 BF16 Yes
qwen3.5-122b-a10b 1 FP8 No
qwen3.5-122b-a10b 1 FP8 Yes
qwen3.5-122b-a10b 2 FP8 No
qwen3.5-122b-a10b 2 FP8 Yes
qwen3.5-122b-a10b 4 FP8 No
qwen3.5-122b-a10b 4 FP8 Yes
qwen3.5-122b-a10b 8 FP8 No
qwen3.5-122b-a10b 8 FP8 Yes
qwen3.5-122b-a10b 1 NVFP4 No
qwen3.5-122b-a10b 1 NVFP4 Yes
qwen3.5-122b-a10b 2 NVFP4 No
qwen3.5-122b-a10b 2 NVFP4 Yes
qwen3.5-122b-a10b 4 NVFP4 No
qwen3.5-122b-a10b 4 NVFP4 Yes
qwen3.5-122b-a10b 8 NVFP4 No
qwen3.5-122b-a10b 8 NVFP4 Yes
qwen3.5-397b-a17b 4 BF16 No
qwen3.5-397b-a17b 4 BF16 Yes
qwen3.5-397b-a17b 8 BF16 No
qwen3.5-397b-a17b 8 BF16 Yes
qwen3.5-397b-a17b 2 FP8 No
qwen3.5-397b-a17b 2 FP8 Yes
qwen3.5-397b-a17b 4 FP8 No
qwen3.5-397b-a17b 4 FP8 Yes
qwen3.5-397b-a17b 8 FP8 No
qwen3.5-397b-a17b 8 FP8 Yes
qwen3.5-397b-a17b 1 NVFP4 No
qwen3.5-397b-a17b 2 NVFP4 No
qwen3.5-397b-a17b 2 NVFP4 Yes
qwen3.5-397b-a17b 4 NVFP4 No
qwen3.5-397b-a17b 4 NVFP4 Yes
qwen3.5-397b-a17b 8 NVFP4 No
qwen3.5-397b-a17b 8 NVFP4 Yes
Qwen3.8-2.4T-A95B 16 FP8 No
qwen3.8-27b 1 NVFP4 No
qwen3.8-27b 2 NVFP4 No
qwen3.8-27b 1 FP8 No
qwen3.8-27b 2 FP8 No
qwen3.8-27b 1 BF16 No
qwen3.8-27b 2 BF16 No
qwen3.8-27b 4 BF16 No
Qwen3.8-Flash-Next 1 NVFP4 No
Qwen3.8-Flash-Next 2 NVFP4 No
Qwen3.8-Flash-Next 4 NVFP4 No
Qwen3.8-Flash-Next 2 FP8 No
Qwen3.8-Flash-Next 4 FP8 No
Qwen3.8-Flash-Next 4 BF16 No
Qwen3.8-Flash-Next 8 BF16 No
riva-translate-4b-instruct-v2 1 BF16 No
starcoder2-7b 1 BF16 No
starcoder2-7b 2 BF16 No
Step 5 Preview 8 FP8 No
Step 5 Preview 4 NVFP4 No

LLMs#

The following section lists all of the supported LLMs.

deepseek-v4-flash-0731#

The following table lists the optimized deployment profiles for deepseek-ai/deepseek-v4-flash-0731:

Optimized Deployment Profiles#

Precision

TP

GPU

LoRA

Tuning Objective

Profile ID

FP8

4

NVIDIA-H200

No

Throughput

8a6dfb29fd452f8f1629e46a8d28fc8e0ace03819df3841bf165f12c45ee2531

NVFP4

4

NVIDIA-B200

No

Throughput

7b3dae0653c59f6b2bbcea7276d5a59c5273e65a4c9864423c3a7f0ba5f1cfa2
Verified GPUs

This model has been verified on the following GPUs:

  • NVIDIA-B200

  • NVIDIA-H200

deepseek-v4-pro-0813#

The following table lists the optimized deployment profiles for deepseek-ai/deepseek-v4-pro-0813:

Optimized Deployment Profiles#

Precision

TP

GPU

LoRA

Tuning Objective

Profile ID

FP8

8

NVIDIA-H200

No

Throughput

2cf4354f85b674da421b1f0c0edd95df85ba08d69d7243a7300f5b76d310c917

NVFP4

8

NVIDIA-B200

No

Throughput

f6e709b4126a3f3c6db754c1d5090a5d1006d23b5f0e62c4cec8e423e8eb5de0
Verified GPUs

This model has been verified on the following GPUs:

  • NVIDIA-B200

  • NVIDIA-H200

glm-5.2#

The following table lists the supported profile configurations for zai-org/glm-5.2:

Precision

TP4

TP8

FP8

sglang-fp8-tp4-pp1

sglang-fp8-tp8-pp1

FP8 + LoRA

sglang-fp8-tp4-pp1-lora

sglang-fp8-tp8-pp1-lora

NVFP4

sglang-nvfp4-tp4-pp1

sglang-nvfp4-tp8-pp1

NVFP4 + LoRA

sglang-nvfp4-tp4-pp1-lora

sglang-nvfp4-tp8-pp1-lora
Verified GPUs

This model has been verified on the following GPUs:

  • NVIDIA-B200

  • NVIDIA-B300-SXM6-AC

  • NVIDIA-GB200

  • NVIDIA-GB300

  • NVIDIA-H200

Optimized Deployment Profiles#

Precision

TP

GPU

LoRA

Tuning Objective

Profile ID

FP8

8

NVIDIA-H200

No

Throughput

04fe52ce04491615d3dacb0aa5b5a2bfe0a494143a9997bc56ba487e5accdd37

NVFP4

4

NVIDIA-B200

No

Throughput

aeb98e3f5578a620b09cadb856f7dc017ca4e7704caeb7c4e18c046e151c561b

gpt-oss-120b#

The following table lists the supported profile configurations for openai/gpt-oss-120b:

Precision

TP1

TP2

TP4

TP8

MXFP4

vllm-mxfp4-tp1-pp1

vllm-mxfp4-tp2-pp1

vllm-mxfp4-tp4-pp1

vllm-mxfp4-tp8-pp1

MXFP4 + LoRA

vllm-mxfp4-tp1-pp1-lora

vllm-mxfp4-tp2-pp1-lora

vllm-mxfp4-tp4-pp1-lora

vllm-mxfp4-tp8-pp1-lora
Verified GPUs

This model has been verified on the following GPUs:

  • NVIDIA-A100-SXM4-40GB

  • NVIDIA-A100-SXM4-80GB

  • NVIDIA-A10G

  • NVIDIA-B200

  • NVIDIA-B300-SXM6-AC

  • NVIDIA-GB200

  • NVIDIA-GB300

  • NVIDIA-GB300-WS

  • NVIDIA-GH200-144G-HBM3e

  • NVIDIA-GH200-480GB

  • NVIDIA-H100-80GB-HBM3

  • NVIDIA-H100-NVL

  • NVIDIA-H200

  • NVIDIA-H200-NVL

  • NVIDIA-L40S

  • NVIDIA-RTX-PRO-4500-Blackwell-Server-Edition

  • NVIDIA-RTX-PRO-6000-Blackwell-Server-Edition

Optimized deployment profiles#

Precision

TP

GPU

LoRA

Tuning Objective

Profile ID

MXFP4

1

NVIDIA-B200, NVIDIA-B300-SXM6-AC, NVIDIA-GB200, NVIDIA-GB300, NVIDIA-GB300-WS, NVIDIA-GH200-144G-HBM3e, NVIDIA-GH200-480GB, NVIDIA-H100-80GB-HBM3, NVIDIA-H100-NVL, NVIDIA-H200, NVIDIA-H200-NVL, NVIDIA-RTX-PRO-6000-Blackwell-Server-Edition

No

Throughput

7de9349cde0a225c2a05f727fbabb8192c067a6d3b1440051730dd9f5c078388

gpt-oss-20b#

The following table lists the supported profile configurations for openai/gpt-oss-20b:

Precision

TP1

TP2

TP4

TP8

MXFP4

vllm-mxfp4-tp1-pp1

vllm-mxfp4-tp2-pp1

vllm-mxfp4-tp4-pp1

vllm-mxfp4-tp8-pp1

MXFP4 + LoRA

vllm-mxfp4-tp1-pp1-lora

vllm-mxfp4-tp2-pp1-lora

vllm-mxfp4-tp4-pp1-lora

vllm-mxfp4-tp8-pp1-lora
Optimized Deployment Profiles#

Precision

TP

GPU

LoRA

Tuning Objective

Profile ID

MXFP4

1

NVIDIA-B200

No

Throughput

5fee7c911ea1847f570b4d908cad6ad76595c39a6bcf2b61dca51d42c85cae35

MXFP4

1

NVIDIA-H200

No

Throughput

5f3be1f858456aa63d6bf3b00282b8d290adf290d43c2ee43e9939f219c95eda

These throughput profiles use an FP8 KV cache.

Verified GPUs

This model has been verified on the following GPUs:

  • NVIDIA-A100-SXM4-40GB

  • NVIDIA-A100-SXM4-80GB

  • NVIDIA-A10G

  • NVIDIA-B200

  • NVIDIA-B300-SXM6-AC

  • NVIDIA-GB10

  • NVIDIA-GB200

  • NVIDIA-GB300

  • NVIDIA-GB300-WS

  • NVIDIA-GH200-144G-HBM3e

  • NVIDIA-GH200-480GB

  • NVIDIA-H100-80GB-HBM3

  • NVIDIA-H100-NVL

  • NVIDIA-H200

  • NVIDIA-H200-NVL

  • NVIDIA-L40S

  • NVIDIA-RTX-PRO-4500-Blackwell-Server-Edition

  • NVIDIA-RTX-PRO-6000-Blackwell-Server-Edition

llama-3.1-70b-instruct#

The following table lists the supported profile configurations for meta/llama-3.1-70b-instruct:

Precision

TP1

TP2

TP4

TP8

BF16

vllm-bf16-tp1-pp1

vllm-bf16-tp2-pp1

vllm-bf16-tp4-pp1

vllm-bf16-tp8-pp1

BF16 + LoRA

vllm-bf16-tp1-pp1-lora

vllm-bf16-tp2-pp1-lora

vllm-bf16-tp4-pp1-lora

vllm-bf16-tp8-pp1-lora

FP8

vllm-fp8-tp1-pp1

vllm-fp8-tp2-pp1

vllm-fp8-tp4-pp1

vllm-fp8-tp8-pp1

FP8 + LoRA

vllm-fp8-tp1-pp1-lora

vllm-fp8-tp2-pp1-lora

vllm-fp8-tp4-pp1-lora

vllm-fp8-tp8-pp1-lora

NVFP4

vllm-nvfp4-tp1-pp1

vllm-nvfp4-tp2-pp1

vllm-nvfp4-tp4-pp1

vllm-nvfp4-tp8-pp1

NVFP4 + LoRA

vllm-nvfp4-tp1-pp1-lora

vllm-nvfp4-tp2-pp1-lora

vllm-nvfp4-tp4-pp1-lora

vllm-nvfp4-tp8-pp1-lora
Optimized deployment profiles#

Precision

TP

GPU

LoRA

Tuning Objective

Profile ID

NVFP4

2

NVIDIA-B200

No

Throughput

e1d3ae3b1367b98b9a7a729f3087d8e411e709f436c624129b939f96dcfd02b7

FP8

2

NVIDIA-H200

No

Throughput

421adb20111d74d64424eab4d8e9ed1a4cbd756338f03c52a32c4ee955301044
Verified GPUs

This model has been verified on the following GPUs:

  • NVIDIA-A100-SXM4-40GB

  • NVIDIA-A100-SXM4-80GB

  • NVIDIA-B200

  • NVIDIA-B300-SXM6-AC

  • NVIDIA-GB200

  • NVIDIA-GB300

  • NVIDIA-GB300-WS

  • NVIDIA-GH200-144G-HBM3e

  • NVIDIA-H100-80GB-HBM3

  • NVIDIA-H100-NVL

  • NVIDIA-H200

  • NVIDIA-H200-NVL

  • NVIDIA-L40S

  • NVIDIA-RTX-PRO-4500-Blackwell-Server-Edition

  • NVIDIA-RTX-PRO-6000-Blackwell-Server-Edition

llama-3.1-8b-instruct#

The following table lists the supported profile configurations for meta/llama-3.1-8b-instruct:

Precision

TP1

BF16

vllm-bf16-tp1-pp1

BF16 + LoRA

vllm-bf16-tp1-pp1-lora

FP8

vllm-fp8-tp1-pp1

FP8 + LoRA

vllm-fp8-tp1-pp1-lora

NVFP4

vllm-nvfp4-tp1-pp1

NVFP4 + LoRA

vllm-nvfp4-tp1-pp1-lora
Optimized deployment profiles#

Precision

TP

GPU

LoRA

Tuning Objective

Profile ID

NVFP4

1

NVIDIA-B200

No

Throughput

381790f97c25437119051f987f5b172b00e6e5281545c76e5318caa9880ecd15

FP8

1

NVIDIA-H200

No

Throughput

86e27d488d1a6a6b7fa5fb2a6894bafc7059376021286fab43fcd9c612abd7f4
Verified GPUs

This model has been verified on the following GPUs:

  • NVIDIA-A100-SXM4-40GB

  • NVIDIA-A100-SXM4-80GB

  • NVIDIA-B200

  • NVIDIA-B300-SXM6-AC

  • NVIDIA-GB10

  • NVIDIA-GB200

  • NVIDIA-GB300

  • NVIDIA-GB300-WS

  • NVIDIA-GH200-144G-HBM3e

  • NVIDIA-GH200-480GB

  • NVIDIA-H100-80GB-HBM3

  • NVIDIA-H100-NVL

  • NVIDIA-H200

  • NVIDIA-H200-NVL

  • NVIDIA-L40S

  • NVIDIA-RTX-PRO-4500-Blackwell-Server-Edition

  • NVIDIA-RTX-PRO-6000-Blackwell-Server-Edition

llama-3.3-70b-instruct#

The following table lists the supported profile configurations for meta/llama-3.3-70b-instruct:

Precision

TP1

TP2

TP4

TP8

BF16

vllm-bf16-tp1-pp1

vllm-bf16-tp2-pp1

vllm-bf16-tp4-pp1

vllm-bf16-tp8-pp1

BF16 + LoRA

vllm-bf16-tp1-pp1-lora

vllm-bf16-tp2-pp1-lora

vllm-bf16-tp4-pp1-lora

vllm-bf16-tp8-pp1-lora

FP8

vllm-fp8-tp1-pp1

vllm-fp8-tp2-pp1

vllm-fp8-tp4-pp1

vllm-fp8-tp8-pp1

FP8 + LoRA

vllm-fp8-tp1-pp1-lora

vllm-fp8-tp2-pp1-lora

vllm-fp8-tp4-pp1-lora

vllm-fp8-tp8-pp1-lora

NVFP4

vllm-nvfp4-tp1-pp1

vllm-nvfp4-tp2-pp1

vllm-nvfp4-tp4-pp1

vllm-nvfp4-tp8-pp1

NVFP4 + LoRA

vllm-nvfp4-tp1-pp1-lora

vllm-nvfp4-tp2-pp1-lora

vllm-nvfp4-tp4-pp1-lora

vllm-nvfp4-tp8-pp1-lora
Optimized deployment profiles#

Precision

TP

GPU

LoRA

Tuning Objective

Profile ID

NVFP4

1

NVIDIA-B200

No

Throughput

16d3616335859278752b8ec6bdd3d88657880120895118fcd584c75f6f3249e0

FP8

2

NVIDIA-H200

No

Throughput

f8197e13fcb703fc7df8a9060d3d2be9745b4ff98b0eac3e466779afc127f2a6
Verified GPUs

This model has been verified on the following GPUs:

  • NVIDIA-A100-SXM4-40GB

  • NVIDIA-A100-SXM4-80GB

  • NVIDIA-B200

  • NVIDIA-B300-SXM6-AC

  • NVIDIA-GB200

  • NVIDIA-GB300

  • NVIDIA-GB300-WS

  • NVIDIA-GH200-144G-HBM3e

  • NVIDIA-H100-80GB-HBM3

  • NVIDIA-H100-NVL

  • NVIDIA-H200

  • NVIDIA-H200-NVL

  • NVIDIA-L40S

  • NVIDIA-RTX-PRO-4500-Blackwell-Server-Edition

  • NVIDIA-RTX-PRO-6000-Blackwell-Server-Edition

llama-3.3-nemotron-super-49b-v1.5#

The following table lists the supported profile configurations for nvidia/llama-3.3-nemotron-super-49b-v1.5:

Precision

TP1

TP2

TP4

TP8

BF16

vllm-bf16-tp1-pp1

vllm-bf16-tp2-pp1

vllm-bf16-tp4-pp1

vllm-bf16-tp8-pp1

BF16 + LoRA

vllm-bf16-tp1-pp1-lora

vllm-bf16-tp2-pp1-lora

vllm-bf16-tp4-pp1-lora

vllm-bf16-tp8-pp1-lora

FP8

vllm-fp8-tp1-pp1

vllm-fp8-tp2-pp1

vllm-fp8-tp4-pp1

vllm-fp8-tp8-pp1

FP8 + LoRA

vllm-fp8-tp1-pp1-lora

vllm-fp8-tp2-pp1-lora

vllm-fp8-tp4-pp1-lora

vllm-fp8-tp8-pp1-lora

NVFP4

vllm-nvfp4-tp1-pp1

vllm-nvfp4-tp2-pp1

vllm-nvfp4-tp4-pp1

vllm-nvfp4-tp8-pp1

NVFP4 + LoRA

vllm-nvfp4-tp1-pp1-lora

vllm-nvfp4-tp2-pp1-lora

vllm-nvfp4-tp4-pp1-lora

vllm-nvfp4-tp8-pp1-lora
Optimized deployment profiles#

Precision

TP

GPU

LoRA

Tuning Objective

Profile ID

NVFP4

1

NVIDIA-B200

No

Throughput

e97801692c74dfb4a287f5b05b6d933aa59913368929ed135dc457dea6877ac1

FP8

2

NVIDIA-H200

No

Throughput

9923047c5f78eca58a4a1d586e85cfaaa4c724d5679bb615a9e954b4530033c3
Verified GPUs

This model has been verified on the following GPUs:

  • NVIDIA-A100-SXM4-40GB

  • NVIDIA-A100-SXM4-80GB

  • NVIDIA-A10G

  • NVIDIA-B200

  • NVIDIA-B300-SXM6-AC

  • NVIDIA-GB10

  • NVIDIA-GB200

  • NVIDIA-GB300

  • NVIDIA-GB300-WS

  • NVIDIA-GH200-144G-HBM3e

  • NVIDIA-GH200-480GB

  • NVIDIA-H100-80GB-HBM3

  • NVIDIA-H100-NVL

  • NVIDIA-H200

  • NVIDIA-H200-NVL

  • NVIDIA-L40S

  • NVIDIA-RTX-PRO-4500-Blackwell-Server-Edition

  • NVIDIA-RTX-PRO-6000-Blackwell-Server-Edition

nemotron-3-nano#

The following table lists the supported profile configurations for nvidia/nemotron-3-nano:

Precision

TP1

TP2

TP4

TP8

BF16

vllm-bf16-tp1-pp1

vllm-bf16-tp2-pp1

vllm-bf16-tp4-pp1

vllm-bf16-tp8-pp1

BF16 + LoRA

vllm-bf16-tp1-pp1-lora

vllm-bf16-tp2-pp1-lora

vllm-bf16-tp4-pp1-lora

vllm-bf16-tp8-pp1-lora

FP8

vllm-fp8-tp1-pp1

vllm-fp8-tp2-pp1

vllm-fp8-tp4-pp1

vllm-fp8-tp8-pp1

NVFP4

vllm-nvfp4-tp1-pp1

vllm-nvfp4-tp2-pp1

vllm-nvfp4-tp4-pp1

vllm-nvfp4-tp8-pp1

NVFP4 + LoRA

vllm-nvfp4-tp1-pp1-lora

–

–

–
Optimized deployment profiles#

Precision

TP

GPU

LoRA

Tuning Objective

Profile ID

NVFP4

1

NVIDIA-B200

No

Throughput

669d73c7ab89d5b53b392716cbcd8a0e06cf40d9d5051aae0b9f9a52891933a9

FP8

1

NVIDIA-H200

No

Throughput

8ca069b289f1838b0c63aeb51317f623f390975c099ba57f805c6c8940a81dad
Verified GPUs

This model has been verified on the following GPUs:

  • NVIDIA-A100-SXM4-40GB

  • NVIDIA-A100-SXM4-80GB

  • NVIDIA-A10G

  • NVIDIA-B200

  • NVIDIA-B300-SXM6-AC

  • NVIDIA-GB10

  • NVIDIA-GB200

  • NVIDIA-GB300

  • NVIDIA-GB300-WS

  • NVIDIA-GH200-144G-HBM3e

  • NVIDIA-GH200-480GB

  • NVIDIA-H100-80GB-HBM3

  • NVIDIA-H100-NVL

  • NVIDIA-H200

  • NVIDIA-H200-NVL

  • NVIDIA-L40S

  • NVIDIA-RTX-PRO-4500-Blackwell-Server-Edition

  • NVIDIA-RTX-PRO-6000-Blackwell-Server-Edition

nemotron-3-super-120b-a12b#

The following table lists the supported profile configurations for nvidia/nemotron-3-super-120b-a12b:

Precision

TP1

TP2

TP4

TP8

BF16

vllm-bf16-tp1-pp1

vllm-bf16-tp2-pp1

vllm-bf16-tp4-pp1

vllm-bf16-tp8-pp1

BF16 + LoRA

–

vllm-bf16-tp2-pp1-lora

vllm-bf16-tp4-pp1-lora

vllm-bf16-tp8-pp1-lora

FP8

vllm-fp8-tp1-pp1

vllm-fp8-tp2-pp1

vllm-fp8-tp4-pp1

vllm-fp8-tp8-pp1

NVFP4

vllm-nvfp4-tp1-pp1

vllm-nvfp4-tp2-pp1

vllm-nvfp4-tp4-pp1

vllm-nvfp4-tp8-pp1

NVFP4 + LoRA

vllm-nvfp4-tp1-pp1-lora

vllm-nvfp4-tp2-pp1-lora

–

–
Optimized deployment profiles#

Precision

TP

GPU

LoRA

Tuning Objective

Profile ID

NVFP4

4

NVIDIA-B200

No

Throughput

3c7c3d74b044f7cf4bba1a80f3f596405a17efa35c663ca52963d71aaf381ea3

FP8

4

NVIDIA-H200

No

Throughput

ca31277ebd3b387e1a597a550541ab9b8dde3b31f6b5b5695ec8799c0162573c

Note

This is a large model. Lower-TP profiles require substantially more GPU memory per device, so some verified GPUs support only TP4 or TP8 profiles.

Verified GPUs

This model has been verified on the following GPUs:

  • NVIDIA-A100-SXM4-40GB

  • NVIDIA-A100-SXM4-80GB

  • NVIDIA-B200

  • NVIDIA-B300-SXM6-AC

  • NVIDIA-GB200

  • NVIDIA-GB300

  • NVIDIA-GB300-WS

  • NVIDIA-GH200-144G-HBM3e

  • NVIDIA-H100-80GB-HBM3

  • NVIDIA-H100-NVL

  • NVIDIA-H200

  • NVIDIA-H200-NVL

  • NVIDIA-L40S

  • NVIDIA-RTX-PRO-4500-Blackwell-Server-Edition

  • NVIDIA-RTX-PRO-6000-Blackwell-Server-Edition

nemotron-3-ultra-550b-a55b#

The following table lists the supported profile configurations for nvidia/nemotron-3-ultra-550b-a55b:

Precision

TP1

TP2

TP4

TP8

BF16

–

–

–

vllm-bf16-tp8-pp1

BF16 + LoRA

–

–

–

–

NVFP4

–

vllm-nvfp4-tp2-pp1

vllm-nvfp4-tp4-pp1

vllm-nvfp4-tp8-pp1
Optimized deployment profiles#

Precision

TP

GPU

LoRA

Tuning Objective

Profile ID

NVFP4

4

NVIDIA-B200

No

Throughput

7cdfe4d76f5fd32ad9ae240eef8b5d4af81dcc0fa23c6f20c4bc7da1a389a92c

NVFP4

4

NVIDIA-B300-SXM6-AC

No

Throughput

5b3441c9d0f55e8b4442537a4294304d5401d3a5542870bda16f3f8e18971878

NVFP4

4

NVIDIA-GB200

No

Throughput

a953e969777d28fc50e322d31a1bdfaf6aeb1a3cdff63ca3760cf0d424b9a90a

NVFP4

4

NVIDIA-GB300

No

Throughput

4466f9f08faedae5362260c2552af473240428fd9bf17d7bbaaf5a535abdde1e

Note

The minimum deployment is two GPUs, using vllm-nvfp4-tp2-pp1 on high-memory Blackwell SKUs (NVIDIA-B300-SXM6-AC and NVIDIA-GB300); the other profiles use four or eight GPUs. H200 validation covers one NVFP4 TP4 profile and is less extensive than the Blackwell validation.

FP8 profiles were not released for this NIM because NVFP4 profiles work on all tested Hopper-architecture GPUs (for example, H100 and H200) and deliver better performance than FP8 profiles.

Verified GPUs

This model has been verified on the following GPUs:

  • NVIDIA-B200

  • NVIDIA-B300-SXM6-AC

  • NVIDIA-GB200

  • NVIDIA-GB300

  • NVIDIA-H100-80GB-HBM3

  • NVIDIA-H100-NVL

  • NVIDIA-H200

nemotron-3.5-lightning#

The following table lists the supported profile configurations for nvidia/nemotron-3.5-lightning, along with the minimum GPU VRAM required per device and the GPU architecture that each precision requires. The VRAM values come from the profile disk-sizing metadata reported by list-model-profiles and represent the minimum per-GPU memory needed to load the model weights and the runtime allocations; they do not include additional headroom for large KV caches at high context length or high concurrency.

Nemotron 3.5 Lightning Profile Information#

Precision

TP

PP

Min VRAM per GPU

Min GPU count

Architecture requirement

LoRA Profile

BF16

1

1

66 GB

1

Ampere or newer (SM 8.0+)

Base and LoRA

BF16

2

1

35 GB

2

Ampere or newer (SM 8.0+)

Base and LoRA

BF16

4

1

20 GB

4

Ampere or newer (SM 8.0+)

Base and LoRA

BF16

8

1

12 GB

8

Ampere or newer (SM 8.0+)

Base and LoRA

W4A16

1

1

32 GB

1

Ampere or newer (SM 8.0+)

Base and LoRA

W4A16

2

1

20 GB

2

Ampere or newer (SM 8.0+)

Base and LoRA

W4A16

4

1

14 GB

4

Ampere or newer (SM 8.0+)

Base and LoRA

NVFP4

1

1

30 GB

1

Blackwell or newer (SM 10.0+)

Base and LoRA

NVFP4

2

1

18 GB

2

Blackwell or newer (SM 10.0+)

Base and LoRA

NVFP4

4

1

12 GB

4

Blackwell or newer (SM 10.0+)

Base and LoRA
Optimized deployment profiles#

Precision

TP

GPU

LoRA

Tuning Objective

Profile ID

NVFP4

1

NVIDIA-B200

No

Throughput

4816da181f28c435aaa2efa0e2302309c28f2c08c9534070e89f67854a521d72

NVFP4 (W4A16) + FP8

1

NVIDIA-H200

No

Throughput

1e34732516864e06c8dad271edd1d19b986c4c010b9d7e3adcb8b113df58819b

Notes:

  • Min VRAM per GPU is the floor that list-model-profiles reports for that precision and tensor-parallel size. A GPU below the floor is filtered out during profile selection.

  • Architecture requirement in the generic profile table reflects the kernels those profiles need. The generic NVFP4 profiles require Streaming Multiprocessor (SM) 10.0+ (Blackwell) because they use NVFP4-native tensor cores. BF16 and W4A16 run on any Ampere-class or newer GPU (SM 8.0+).

  • Sharing weights across more GPUs (higher TP) lowers the per-GPU VRAM floor and enables lower-memory GPUs.

  • Model-cache disk sizes range from approximately 19 GB (NVFP4) to approximately 63 GB (BF16). Cache size is separate from GPU VRAM and does not indicate that the model will fit in a given GPU.

Verified GPUs

The following tables list verified GPUs for the generic profiles, grouped by architecture:

Blackwell (SM 10.0) — supports NVFP4, W4A16, BF16:

GPU

VRAM per GPU

Profiles that fit

NVIDIA-B200

180 GB

All profiles

NVIDIA-B300-SXM6-AC

288 GB

All profiles

NVIDIA-GB200

186 GB

All profiles

NVIDIA-GB300

288 GB

All profiles

NVIDIA-GB10

128 GB (unified memory)

All profiles except BF16 TP1 (base and LoRA); set NIM_KVCACHE_PERCENT=0.75 on GB10 (refer to the GPU memory troubleshooting guide)

NVIDIA-RTX-PRO-6000-Blackwell-Server-Edition

96 GB

All profiles

Hopper (SM 9.0) — supports W4A16 and BF16 (no NVFP4):

GPU

VRAM per GPU

Profiles that fit

NVIDIA-H100-80GB-HBM3

80 GB

BF16 TP≥2, W4A16 all sizes

NVIDIA-H200

141 GB

BF16 all sizes, W4A16 all sizes

Ampere (SM 8.0) — supports W4A16 and BF16 (no NVFP4):

GPU

VRAM per GPU

Profiles that fit

NVIDIA-A100-SXM4-80GB

80 GB

BF16 TP≥2

Ada (SM 8.9) — supports W4A16 and BF16 (no NVFP4):

GPU

VRAM per GPU

Profiles that fit

NVIDIA-L40S

48 GB

BF16 TP≥2, W4A16 all sizes

Unverified but potentially compatible. Any GPU that meets the per-profile minimum VRAM floor and the architecture requirement in the profile table can run that profile technically, even if it is not in the preceding Verified GPUs tables. Only the GPUs in those tables have been tested end-to-end for this model at this NIM version; untested systems may exhibit different startup, throughput, or accuracy behavior.

riva-translate-4b-instruct-v2#

The following table lists the supported profile configuration for nvidia/riva-translate-4b-instruct-v2.

The latest release version of this NIM is 2.0.8.

Precision

TP

PP

LoRA

Profile

BF16

1

1

No

vllm-bf16-tp1-pp1
Verified GPUs

This model has been verified on the following GPUs:

  • NVIDIA-A100-SXM4-40GB

  • NVIDIA-A100-SXM4-80GB

  • NVIDIA-A10G

  • NVIDIA-B200

  • NVIDIA-B300-SXM6-AC

  • NVIDIA-GB10

  • NVIDIA-GB200

  • NVIDIA-GB300

  • NVIDIA-GB300-WS

  • NVIDIA-GH200-144G-HBM3e

  • NVIDIA-GH200-480GB

  • NVIDIA-H100-80GB-HBM3

  • NVIDIA-H100-NVL

  • NVIDIA-H200

  • NVIDIA-H200-NVL

  • NVIDIA-L40S

  • NVIDIA-RTX-PRO-4500-Blackwell-Server-Edition

  • NVIDIA-RTX-PRO-6000-Blackwell-Server-Edition

starcoder2-7b#

The following table lists the supported profile configurations for bigcode/starcoder2-7b:

Precision

TP1

TP2

BF16

vllm-bf16-tp1-pp1

vllm-bf16-tp2-pp1
Optimized deployment profiles#

Precision

TP

GPU

LoRA

Tuning Objective

Profile ID

BF16

1

NVIDIA-A100-80GB-PCIe, NVIDIA-A100-SXM4-80GB, NVIDIA-B200, NVIDIA-GB300-WS, NVIDIA-H100-80GB-HBM3, NVIDIA-H200

No

Throughput

65053e7510a772fa5ebc8da386b9b593152c746173b4c2fb85a1739a276e0d2c

BF16

2

NVIDIA-A100-SXM4-80GB, NVIDIA-B200, NVIDIA-H100-80GB-HBM3, NVIDIA-H200

No

Throughput

39725ead141c0eb26d6cbdcce32660a6da469663d88bc7ea7441808ff0134ca5
Verified GPUs

This model has been verified on the following GPUs:

  • NVIDIA-A100-80GB-PCIe

  • NVIDIA-A100-SXM4-80GB

  • NVIDIA-B200

  • NVIDIA-GB300-WS

  • NVIDIA-H100-80GB-HBM3

  • NVIDIA-H200

VLMs#

The following section lists all of the supported VLMs.

gemma-4-26b-a4b-it#

The following table lists the supported profile configurations for google/gemma-4-26b-a4b-it:

Precision

TP1

TP2

TP4

TP8

NVFP4

vllm-nvfp4-tp1-pp1

–

–

–

BF16

vllm-bf16-tp1-pp1

vllm-bf16-tp2-pp1

vllm-bf16-tp4-pp1

–
Optimized deployment profiles#

Precision

TP

GPU

LoRA

Tuning Objective

Profile ID

NVFP4

1

NVIDIA-B200

No

Throughput

234e46324452f6124bf432eaa99cc1221ed917b069ee63ebc0c4d2e636f22f3d

NVFP4

1

NVIDIA-GB200

No

Throughput

04094afda5cf6fc9df04560f322cc677678a0d4caf943d058ff1c9779aaf750c

BF16

4

NVIDIA-A10G

No

Context length

800f3c24e5bdaca88aebd04242be7765c65ac7abbfc9013c404d0ea0ddc9f6fc
Verified GPUs

This model has been verified on the following GPUs:

  • NVIDIA-A100-SXM4-40GB

  • NVIDIA-A100-SXM4-80GB

  • NVIDIA-A10G

  • NVIDIA-B200

  • NVIDIA-B300-SXM6-AC

  • NVIDIA-GB200

  • NVIDIA-GB300

  • NVIDIA-GH200-144G-HBM3e

  • NVIDIA-GH200-480GB

  • NVIDIA-H100-80GB-HBM3

  • NVIDIA-H100-NVL

  • NVIDIA-H200

  • NVIDIA-H200-NVL

  • NVIDIA-L40S

  • NVIDIA-RTX-PRO-4500-Blackwell-Server-Edition

  • NVIDIA-RTX-PRO-6000-Blackwell-Server-Edition

gemma-4-31b-it#

The following table lists the supported profile configurations for google/gemma-4-31b-it:

Precision

TP1

TP2

TP4

TP8

NVFP4

vllm-nvfp4-tp1-pp1

vllm-nvfp4-tp2-pp1

vllm-nvfp4-tp4-pp1

–

BF16

vllm-bf16-tp1-pp1

vllm-bf16-tp2-pp1

vllm-bf16-tp4-pp1

–

INT4

–

vllm-int4-tp2-pp1

–

–
Optimized deployment profiles#

Precision

TP

GPU

LoRA

Tuning Objective

Profile ID

NVFP4

1

NVIDIA-B200, NVIDIA-B300-SXM6-AC, NVIDIA-GB200, NVIDIA-GB300, NVIDIA-RTX-PRO-6000-Blackwell-Server-Edition

No

Throughput

5f2613453901ce34041f9390d1eb035bceabba0af65af73be562fa5cda704a48
Verified GPUs

This model has been verified on the following GPUs:

  • NVIDIA-B200

  • NVIDIA-B300-SXM6-AC

  • NVIDIA-GB200

  • NVIDIA-GB300

  • NVIDIA-H100-80GB-HBM3

  • NVIDIA-H200

  • NVIDIA-L40S

  • NVIDIA-RTX-PRO-6000-Blackwell-Server-Edition

Inkling#

The following table lists the supported profile configurations for thinkingmachines/inkling. This model-specific container uses SGLang as the inference backend.

Inkling Profiles#

Profile

GPUs

Precision

TP

PP

LoRA

Generic NVFP4 TP4

B300, GB300

NVFP4

4

1

No

Generic NVFP4 TP4 + LoRA

B300, GB300

NVFP4

4

1

Yes

Generic NVFP4 TP8

B200, B300

NVFP4

8

1

No

Generic NVFP4 TP8 + LoRA

B200, B300

NVFP4

8

1

Yes

H200 NVFP4 TP8

H200

NVFP4

8

1

No

H200 NVFP4 TP8 + LoRA

H200

NVFP4

8

1

Yes
Optimized Deployment Profiles#

Precision

TP

GPU

LoRA

Tuning Objective

Profile ID

NVFP4

8

NVIDIA-B200

No

Throughput

b06d3d95f3fba80b51ab008fbcce8cc8dd6f0029cf021f13ec85fe612b57db34

NVFP4

8

NVIDIA-H200

No

Throughput

4a30cb634a0f720cc28572ec4592876d255762f2bfe7bf0d2dceff6890f1f5f4

The H200 profiles run the NVFP4 weights with Marlin kernels. Set NIM_ALLOW_NVFP4_EMULATION=1 at runtime on H200.

Verified GPUs

This model has been verified on the following GPUs:

  • NVIDIA-B200

  • NVIDIA-B300-SXM6-AC

  • NVIDIA-GB300

  • NVIDIA-H200

kimi-k2.6#

The following table lists the supported profile configurations for moonshotai/kimi-k2.6:

Precision

TP1

TP2

TP4

TP8

NVFP4

–

–

vllm-nvfp4-tp4-pp1

–

INT4

–

–

–

vllm-int4-tp8-pp1
Optimized deployment profiles#

Precision

TP

GPU

LoRA

Tuning Objective

Profile ID

NVFP4

4

NVIDIA-B300-SXM6-AC

No

Throughput

3f1e102185aade2a7df51fa32bd8e56667db92694f374a51ac450247328bcc0f

NVFP4

4

NVIDIA-GB200

No

Throughput

25868c0d2f0529d801a9e5aea19d16ae15cbed8e7d065dfcdbf8cdc2fd451d86

NVFP4

4

NVIDIA-GB300

No

Throughput

8ca983f7aca5cf99befffea021267b4f01fe023b884b77af747213047a642fe4
Verified GPUs

This model has been verified on the following GPUs:

  • NVIDIA-B200

  • NVIDIA-B300-SXM6-AC

  • NVIDIA-GB200

  • NVIDIA-GB300

  • NVIDIA-H200

mistral-small-4-119b-2603#

The following table lists the supported profile configurations for mistralai/mistral-small-4-119b-2603:

Precision

TP1

TP2

TP4

TP8

FP8

vllm-fp8-tp1-pp1

vllm-fp8-tp2-pp1

vllm-fp8-tp4-pp1

vllm-fp8-tp8-pp1

NVFP4

vllm-nvfp4-tp1-pp1

vllm-nvfp4-tp2-pp1

vllm-nvfp4-tp4-pp1

vllm-nvfp4-tp8-pp1
Optimized Deployment Profiles#

Precision

TP

GPU

LoRA

Tuning Objective

Profile ID

NVFP4

2

NVIDIA-B200

No

Throughput

e7a3b7d62d236b346c92b59465d26554f8fbe8e9ec5dc8e2fbc450ba54ecdf89

This throughput profile uses an FP8 KV cache.

Verified GPUs

This model has been verified on the following GPUs:

  • NVIDIA-B200

  • NVIDIA-B300-SXM6-AC

  • NVIDIA-GB200

  • NVIDIA-GB300

  • NVIDIA-H100-80GB-HBM3

  • NVIDIA-H200

  • NVIDIA-H200-NVL

  • NVIDIA-RTX-PRO-6000-Blackwell-Server-Edition

muse-glimmer#

The following table lists the supported profile configurations for meta/muse-glimmer:

This model reasons before it answers and requires the cache-permission step during deployment. Refer to Muse Glimmer deployment considerations before you send your first request.

Precision

TP1

TP2

TP4

TP8

BF16

sglang-bf16-tp1-pp1

sglang-bf16-tp2-pp1

–

–

NVFP4

sglang-nvfp4-tp1-pp1

sglang-nvfp4-tp2-pp1

–

–
Verified GPUs

This model has been verified on the following GPUs:

  • NVIDIA-A100-SXM4-40GB

  • NVIDIA-A100-SXM4-80GB

  • NVIDIA-B200

  • NVIDIA-B300-SXM6-AC

  • NVIDIA-GB10

  • NVIDIA-GB200

  • NVIDIA-GB300

  • NVIDIA-GH200-144G-HBM3e

  • NVIDIA-GH200-480GB

  • NVIDIA-H100-80GB-HBM3

  • NVIDIA-H100-NVL

  • NVIDIA-H200

  • NVIDIA-H200-NVL

  • NVIDIA-L40S

  • NVIDIA-RTX-PRO-4500-Blackwell-Server-Edition

  • NVIDIA-RTX-PRO-6000-Blackwell-Server-Edition

nemotron-3-nano-omni-30b-a3b-reasoning#

The following table lists the supported profile configurations for nvidia/nemotron-3-nano-omni-30b-a3b-reasoning:

Precision

TP1

TP2

TP4

TP8

BF16

–

vllm-bf16-tp2-pp1

–

–

FP8

vllm-fp8-tp1-pp1

–

–

–

NVFP4

vllm-nvfp4-tp1-pp1

–

–

–
Optimized deployment profiles#

Precision

TP

GPU

LoRA

Tuning Objective

Profile ID

NVFP4

1

NVIDIA-B200

No

Throughput

2883a35a92ca200be25ce99fa4f04c4807a37c3d7b01f2192e00ddd5f9d105b7

NVFP4

1

NVIDIA-B300-SXM6-AC

No

Throughput

7cb9d0358baa2792b09a9907acc032df060acd912ce9c5b99c99589e4f34182c

NVFP4

1

NVIDIA-GB300

No

Throughput

22da479a1490835d7980611878ca91da3dd8bfa8c63c72c814c8867955d9eff8

FP8

1

NVIDIA-H200

No

Throughput

babd305818fe475ed855900b0518a7ef2d6bdf3ccd651584094a706dd010822e

NVFP4

1

NVIDIA-RTX-PRO-4500-Blackwell-Server-Edition

No

Throughput

ffb4dc6902e6bb4c27948ea1700707c664b179ab214072f9f700b0fffdefa1de

NVFP4

1

NVIDIA-RTX-PRO-6000-Blackwell-Server-Edition

No

Throughput

d8e8c64a19e5150344259745878af4dc1eabc270326a8b7bb4516c90ec602e5c
Verified GPUs

This model has been verified on the following GPUs:

  • NVIDIA-A100-SXM4-80GB

  • NVIDIA-B200

  • NVIDIA-B300-SXM6-AC

  • NVIDIA-GB200

  • NVIDIA-GB300

  • NVIDIA-GH200-144G-HBM3e

  • NVIDIA-GH200-480GB

  • NVIDIA-H100-80GB-HBM3

  • NVIDIA-H200

  • NVIDIA-H200-NVL

  • NVIDIA-L40S

  • NVIDIA-RTX-PRO-4500-Blackwell-Server-Edition

  • NVIDIA-RTX-PRO-6000-Blackwell-Server-Edition

nemotron-3.5-bolt-e4b#

The following table lists the supported GPU configurations for nvidia/nemotron-3.5-bolt-e4b-4gb. For model-specific settings, refer to Model-Specific Deployment Considerations.

Nemotron 3.5 Bolt E4B 4GB Profile Information#

GPU

Precision

TP

LoRA

GB10 (DGX Spark)

BF16

1

No

RTX-PRO-6000-Blackwell-Server-Edition

BF16

1

No

RTX-PRO-4500-Blackwell-Server-Edition

BF16

1

No

H100-80GB-HBM3

BF16

1

No
Verified GPUs

This model supports the following GPUs at the precisions and GPU counts listed in the table above:

  • NVIDIA-GB10

  • NVIDIA-H100-80GB-HBM3

  • NVIDIA-RTX-PRO-4500-Blackwell-Server-Edition

  • NVIDIA-RTX-PRO-6000-Blackwell-Server-Edition

nemotron-3.5-content-safety#

The following table lists the supported profile configurations for nvidia/nemotron-3.5-content-safety.

Deployment profiles#

Precision

TP

GPU

LoRA

Tuning Objective

Profile ID

BF16

1

NVIDIA-A100-SXM4-40GB

No

Not specified

ecc0c366dd8591e092b9d521ca26240c8f16119fcc5c3b5f20d1ccb2e08ec843

BF16

1

NVIDIA-A100-SXM4-80GB

No

Not specified

8439d9c09f0dd1e09778da2360522b71018f7cdbd43beb1205f15af1722323d9

BF16

1

NVIDIA-A10G

No

Not specified

79633375948d1847d35dc5cecf6e405059ee8f675db426898ae8646aaa1d3544

BF16

1

NVIDIA-B200

No

Not specified

4bd4088817c26b9ae361d43142053984fd8d9b2d170d332eeb1ee1c7f1256a23

BF16

1

NVIDIA-B200

No

Throughput (FP8 KV cache)

8e1cd95c37e3eb38cf375b1d2405a466772b5126aaa7e0d5eae65d6ac8159971

BF16

1

NVIDIA-B300-SXM6-AC

No

Not specified

116d109f484cac44a0644504c2f752ed7c6a60be907941cad17348396625530f

BF16

1

NVIDIA-GB10

No

Not specified

e67cca9a7077dc626b32ad05fb4ea1f67f35336ee6e123a733a6487461fc3759

BF16

1

NVIDIA-GB200

No

Not specified

32c8887258be660a21ee0413786e83282a5ae84635143b997aa79d5150154c91

BF16

1

NVIDIA-GB300

No

Not specified

df5adf8ebbb6a5f1c787f0eb6d34182ae4409c90fc5f0ea1bdb46352f1423770

BF16

1

NVIDIA-GH200-144G-HBM3e

No

Not specified

6e76fd300bcb71e4094a74a483f34f70b9c7c2d4047db93137b16c33a5c8f413

BF16

1

NVIDIA-GH200-480GB

No

Not specified

dc1503ed2b91153c27cd18a3b9cef86b914727439f99a22eeee83a58d8987138

BF16

1

NVIDIA-H100-80GB-HBM3

No

Not specified

140504fa00018adc1b69c743d02d7efc60dd5bb19e0ea884dbbab8ad565619bc

BF16

1

NVIDIA-H100-NVL

No

Not specified

c099aa54cae529bd0193c055a5f0081fb6eed98cefb699b92e8a7900f61c1047

BF16

1

NVIDIA-H200

No

Not specified

83f5026075c8d36f2ca06265dea5dad380f83cc7b8d0e855f5cc4937b524497f

BF16

1

NVIDIA-H200-NVL

No

Not specified

667cd60dc55899f995767bfa861f4fa6109026776928926b0ba0074ff91ed97e

BF16

1

NVIDIA-L40S

No

Not specified

1163190b38671e02d20347774f7ec9aa0382d0a1a85029a30d6d484416ea881c

BF16

1

NVIDIA-RTX-PRO-4500-Blackwell-Server-Edition

No

Not specified

343b3918e9f57b0c1f4cbf2a1e8ae0577d68314e66b0293408247164f3f23fd2

BF16

1

NVIDIA-RTX-PRO-6000-Blackwell-Server-Edition

No

Not specified

38c75411586262bcd8d4bcff2d84c4941b3375b9e89efcbc1bd21efbfc339140
Verified GPUs

This model has been verified on the following GPUs:

  • NVIDIA-A100-SXM4-40GB

  • NVIDIA-A100-SXM4-80GB

  • NVIDIA-A10G

  • NVIDIA-B200

  • NVIDIA-B300-SXM6-AC

  • NVIDIA-GB10

  • NVIDIA-GB200

  • NVIDIA-GB300

  • NVIDIA-GH200-144G-HBM3e

  • NVIDIA-GH200-480GB

  • NVIDIA-H100-80GB-HBM3

  • NVIDIA-H100-NVL

  • NVIDIA-H200

  • NVIDIA-H200-NVL

  • NVIDIA-L40S

  • NVIDIA-RTX-PRO-4500-Blackwell-Server-Edition

  • NVIDIA-RTX-PRO-6000-Blackwell-Server-Edition

nemotron-3.5-super-vl-120b-a12b#

The following table lists the supported GPU configurations for nvidia/nemotron-3.5-super-vl-120b-a12b. For more information, refer to the dedicated Get Started.

Optimized deployment profiles#

Precision

TP

GPU

LoRA

Tuning Objective

Profile Name

Profile ID

Min VRAM per GPU

BF16

2

NVIDIA-B200

Yes

General

vllm-nvidia-b200-bf16-tp2-pp1-feat_lora-throughput-0.0

f601f9963fcc4bed0469b15df33ec438c2cf55181fb1febdae30fab436369134

119 GB

BF16

2

NVIDIA-B300-SXM6-AC

Yes

General

vllm-nvidia-b300-sxm6-ac-bf16-tp2-pp1-feat_lora-throughput-0.0

12e76812d9e15427ce3a63523778b7bd15bec8dcaefe9817c32fc19a105fffae

119 GB

BF16

2

NVIDIA-GB200

Yes

General

vllm-nvidia-gb200-bf16-tp2-pp1-feat_lora-throughput-0.0

f3dcd36059120231b6fb8a00682af9a36f220dc981ca73a8573bf5aeb5faf85c

119 GB

BF16

2

NVIDIA-GB300

Yes

General

vllm-nvidia-gb300-bf16-tp2-pp1-feat_lora-throughput-0.0

7535e487104e591561705169ece00abad91b1c26255aa844c6456a5b18fc3413

119 GB

BF16

4

NVIDIA-H200

Yes

General

vllm-nvidia-h200-bf16-tp4-pp1-feat_lora-throughput-0.0

29224c376e8bd48f546d1f783a6da93662401524a972c881d3355a00a399da37

61 GB

BF16

8

NVIDIA-H100-80GB-HBM3

Yes

General

vllm-nvidia-h100-80gb-hbm3-bf16-tp8-pp1-feat_lora-throughput-0.0

7f85f2d15f243fa05be912e610b31b19a376f8509dd12eb8a3bc3b367e8b6a9f

32 GB

NVFP4

1

NVIDIA-B200

No

agentX

vllm-nvidia-b200-nvfp4-tp1-pp1-throughput-0.0-agentx

35320c4542dbf3e4ad098dd7669cb3330f728f86a62bdf91eeb4bd34768346f1

72 GB

NVFP4

1

NVIDIA-B300-SXM6-AC

No

agentX

vllm-nvidia-b300-sxm6-ac-nvfp4-tp1-pp1-throughput-0.0-agentx

cc452f7f9da6c7f902014d7e79324a48f8202c12648d42bced69bea9217a81d5

72 GB

NVFP4

1

NVIDIA-GB10

No

General

vllm-nvidia-gb10-nvfp4-tp1-pp1-throughput-0.0

6eb52c8e38c174694d551f314ed8b2021a4b3f767d3616da1754dde0916b855c

72 GB

NVFP4

1

NVIDIA-GB200

No

50k/2k/1k

vllm-nvidia-gb200-nvfp4-tp1-pp1-throughput-0.0-50k-2k-1k

8c840bb323e03589e50b8c97090e059017a35546bbf3b11caec5a1dc5dd334de

72 GB

NVFP4

1

NVIDIA-GB200

No

agentX

vllm-nvidia-gb200-nvfp4-tp1-pp1-throughput-0.0-agentx

d4a164dc936edc781bf577e419a442e4b1712c2d3a17a3ea33b080b162018b2a

72 GB

NVFP4

1

NVIDIA-GB300

No

agentX

vllm-nvidia-gb300-nvfp4-tp1-pp1-throughput-0.0-agentx

26d33ae17bd36e569c9348703f9950a37ffd050b01656292540e2daa35e801b9

72 GB

NVFP4

1

NVIDIA-GB300-WS

No

General

vllm-nvidia-gb300-ws-nvfp4-tp1-pp1-throughput-0.0

5f9194217b5b7856869ae716b6802ec506b6cff1f109a7f1c5e828cb8f105c59

72 GB

NVFP4

1

NVIDIA-H200

No

50k/2k/1k

vllm-nvidia-h200-nvfp4-emulated-tp1-pp1-throughput-0.0-50k-2k-1k

f335b46bb53b1d7a32ed08d093fc93a26145348530dd7aa3b63d578bfd4cfdf9

72 GB

NVFP4

1

NVIDIA-H200

No

agentX

vllm-nvidia-h200-nvfp4-emulated-tp1-pp1-throughput-0.0-agentx

65ec457ca53207b0fa52748a8eeb5e47f58b504f7a005f9ce8f815fec94a8f81

72 GB

NVFP4

2

NVIDIA-A100-SXM4-80GB

No

General

vllm-nvidia-a100-sxm4-80gb-nvfp4-emulated-tp2-pp1-throughput-0.0

737c816b8ac7e1c3c51c871768875600635c9a7939e3e81ed48b6a0a17df29f3

38 GB

NVFP4

2

NVIDIA-H100-80GB-HBM3

No

50k/2k/1k

vllm-nvidia-h100-80gb-hbm3-nvfp4-emulated-tp2-pp1-throughput-0.0-50k-2k-1k

052ee51a97716b2cf070fe268ca0943947e8ff814d331ba52f08d6d8a1e8a20d

38 GB

NVFP4

2

NVIDIA-H100-80GB-HBM3

No

agentX

vllm-nvidia-h100-80gb-hbm3-nvfp4-emulated-tp2-pp1-throughput-0.0-agentx

ab990383587f1f1f9fb3b592895529bc37c34a5a3d6aaf055f07eebe0c865631

38 GB

NVFP4

2

NVIDIA-RTX-PRO-6000-Blackwell-Server-Edition

No

General

vllm-nvidia-rtx-pro-6000-blackwell-server-edition-nvfp4-tp2-pp1-throughput-0.0

da5a5f56a2faa2c54d854046c1374f913bdf829cca30b6406c433f98d472eb21

38 GB

Min VRAM per GPU is the minimum per-GPU memory needed to load the model weights and the runtime allocations at that precision and tensor-parallel size, as computed by NIM LLM and VLM at startup. It does not include headroom for the KV cache at long context length or high concurrency. Each profile is built for the listed GPU model and GPU count, and there is no generic fallback profile. The min_vram_per_device_gb value that list-model-profiles reports for these profiles is not a memory requirement.

Profile Name is the name that list-model-profiles prints next to the profile ID. NIM_MODEL_PROFILE accepts either the name or the ID.

Tuning objectives:

  • 50k/2k/1k: multi-turn conversations that share a 50K-token system prompt, with about 2K new input tokens and 1K output tokens per turn

  • agentX: multi-turn coding-agent sessions with tool calls and long, growing contexts of up to 1M tokens

  • General: the only profile for that GPU and precision

Verified GPUs

This model supports the following GPUs at the precisions and GPU counts listed in the profile table:

  • NVIDIA-A100-SXM4-80GB

  • NVIDIA-B200

  • NVIDIA-B300-SXM6-AC

  • NVIDIA-GB10

  • NVIDIA-GB200

  • NVIDIA-GB300

  • NVIDIA-GB300-WS

  • NVIDIA-H100-80GB-HBM3

  • NVIDIA-H200

  • NVIDIA-RTX-PRO-6000-Blackwell-Server-Edition

nemotron-parse-v2.0#

The following table lists the supported deployment profiles for nvidia/nemotron-parse-v2.0.

Deployment profiles#

Precision

TP

GPU

LoRA

Tuning Objective

Profile ID

BF16

1

NVIDIA-A100-SXM4-40GB

No

Not specified

02ae39e0a84ecb628e3ae2f24f3104e398ac3f02c3eb309ba77430e6aeb57353

BF16

1

NVIDIA-A100-SXM4-80GB

No

Not specified

7b8a6f2aa12b9c06640bdb7653aa4873cbb75096b86fe623e02c52fd8110bef5

BF16

1

NVIDIA-A10G

No

Not specified

5a38a4e1fba6f8b7be13ee2839625d188b1eb0e2a83158dc76a96d786785102b

BF16

1

NVIDIA-B200

No

Not specified

31077528988f89bca85093d60e8e87f03a118db235f33b3c40474e9d46b81c26

BF16

1

NVIDIA-B200

No

Throughput (FP8 KV cache)

98443c5c72f2afd75a056660e8e71799db695e32cdd1ca09fa0e4d5050f29472

BF16

1

NVIDIA-GB200

No

Not specified

e9d12ece6ea1cebd9a2aeb4c1506d6ca2d1e3fb8b3e23274ae84d2e1b91f1fef

BF16

1

NVIDIA-H100-80GB-HBM3

No

Not specified

903be1ea8ffc844bde52490f25ac854752d3144328948367bec6df3f3e7a590a

BF16

1

NVIDIA-H200

No

Not specified

64b7fb5aae787037fe382e5f4a0c8909872b9362e815b70821f70dafd6c8375e

BF16

1

NVIDIA-L40S

No

Not specified

101fd5742d4ce4e0de5ec992bf6cf04da69dda99598984a97e14b2b237e7f869

BF16

1

NVIDIA-RTX-PRO-6000-Blackwell-Server-Edition

No

Not specified

bc69e89091fdf5e99412ab57b32f1b714839f3404746f50b43109821f5e49182

BF16

1

NVIDIA-RTX-PRO-6000-Blackwell-Workstation-Edition

No

Not specified

f633ff89387127be8b44b5d53183d4b5fc471115e87fc1f9e2fd724e66428675

The B200 throughput profile uses an FP8 KV cache while retaining BF16 weights. The remaining profiles do not specify a tuning objective.

Verified GPUs

This model has been verified on the following GPUs:

  • NVIDIA-A100-SXM4-40GB

  • NVIDIA-A100-SXM4-80GB

  • NVIDIA-A10G

  • NVIDIA-B200

  • NVIDIA-GB200

  • NVIDIA-H100-80GB-HBM3

  • NVIDIA-H200

  • NVIDIA-L40S

  • NVIDIA-RTX-PRO-6000-Blackwell-Server-Edition

  • NVIDIA-RTX-PRO-6000-Blackwell-Workstation-Edition

qwen3.5-122b-a10b#

The following table lists the supported profile configurations for qwen/qwen3.5-122b-a10b:

Precision

TP1

TP2

TP4

TP8

BF16

vllm-bf16-tp1-pp1

vllm-bf16-tp2-pp1

vllm-bf16-tp4-pp1

vllm-bf16-tp8-pp1

BF16 + LoRA

–

vllm-bf16-tp2-pp1-lora

vllm-bf16-tp4-pp1-lora

vllm-bf16-tp8-pp1-lora

FP8

vllm-fp8-tp1-pp1

vllm-fp8-tp2-pp1

vllm-fp8-tp4-pp1

vllm-fp8-tp8-pp1

FP8 + LoRA

vllm-fp8-tp1-pp1-lora

vllm-fp8-tp2-pp1-lora

vllm-fp8-tp4-pp1-lora

vllm-fp8-tp8-pp1-lora

NVFP4

vllm-nvfp4-tp1-pp1

vllm-nvfp4-tp2-pp1

vllm-nvfp4-tp4-pp1

vllm-nvfp4-tp8-pp1

NVFP4 + LoRA

vllm-nvfp4-tp1-pp1-lora

vllm-nvfp4-tp2-pp1-lora

vllm-nvfp4-tp4-pp1-lora

vllm-nvfp4-tp8-pp1-lora
Optimized deployment profiles#

Precision

TP

GPU

LoRA

Tuning Objective

Profile ID

NVFP4

1

NVIDIA-B200

No

Throughput

5c60139ad068156cef737d438cd5244742d58c2c0cb312ec61ae8dc0c463d5eb

FP8

2

NVIDIA-H200

No

Latency

c475df816df58cb8d318ea34b7bb05dff51cd2b437c2924eadf10289ab2533c8

FP8

2

NVIDIA-H200

No

Throughput

cbb3cc0300472553b8ae84bfc03f7b1f33e93ac3a9fe2dcb85783ab1981e32b1
Verified GPUs

This model has been verified on the following GPUs:

  • NVIDIA-A100-SXM4-80GB

  • NVIDIA-B200

  • NVIDIA-B300-SXM6-AC

  • NVIDIA-GB200

  • NVIDIA-GB300

  • NVIDIA-GH200-144G-HBM3e

  • NVIDIA-H100-80GB-HBM3

  • NVIDIA-H200

  • NVIDIA-H200-NVL

  • NVIDIA-L40S

  • NVIDIA-RTX-PRO-4500-Blackwell-Server-Edition

  • NVIDIA-RTX-PRO-6000-Blackwell-Server-Edition

qwen3.5-397b-a17b#

The following table lists the supported profile configurations for qwen/qwen3.5-397b-a17b:

Precision

TP1

TP2

TP4

TP8

BF16

–

–

vllm-bf16-tp4-pp1

vllm-bf16-tp8-pp1

BF16 + LoRA

–

–

vllm-bf16-tp4-pp1-lora

vllm-bf16-tp8-pp1-lora

FP8

–

vllm-fp8-tp2-pp1

vllm-fp8-tp4-pp1

vllm-fp8-tp8-pp1

FP8 + LoRA

–

vllm-fp8-tp2-pp1-lora

vllm-fp8-tp4-pp1-lora

vllm-fp8-tp8-pp1-lora

NVFP4

vllm-nvfp4-tp1-pp1

vllm-nvfp4-tp2-pp1

vllm-nvfp4-tp4-pp1

vllm-nvfp4-tp8-pp1

NVFP4 + LoRA

–

vllm-nvfp4-tp2-pp1-lora

vllm-nvfp4-tp4-pp1-lora

vllm-nvfp4-tp8-pp1-lora
Optimized deployment profiles#

Precision

TP

GPU

LoRA

Tuning Objective

Profile ID

FP8

4

NVIDIA-H200

No

Throughput

ed6edfcd8bb730b02a1420640544dcbb13779e732f9de269b25ce1f75bcfb602
Verified GPUs

This model has been verified on the following GPUs:

  • NVIDIA-B200

  • NVIDIA-B300-SXM6-AC

  • NVIDIA-GB200

  • NVIDIA-GB300

  • NVIDIA-H100-80GB-HBM3

  • NVIDIA-H200

  • NVIDIA-H200-NVL

  • NVIDIA-RTX-PRO-6000-Blackwell-Server-Edition

qwen3.8-27b#

The following table lists the supported profile configurations for qwen/qwen3.8-27b.

NIM_MODEL_PROFILE accepts the 64-character profile ID from the tables below or the full profile description printed by list-model-profiles. Every profile in this image is GPU-specific; there are no generic profiles.

Supported profiles#

Precision

TP

GPU

Profile ID

NVFP4

1

NVIDIA-B200

c4dae72230ad2b0c56220d9d801642564445bc21d6cbdd52702300c273a4a8aa

NVFP4

1

NVIDIA-B300-SXM6-AC

5b2f72b5f5796d60ff0741c35f3f070cbbfe07e0766e9a98c94c0edca281e525

NVFP4

1

NVIDIA-GB200

8dbdb556b13a133de592e7e5a93adec5a1d57d89933573a578a24455eb2df439

NVFP4

1

NVIDIA-GB300

fdada0bb416ccf19025bf84c552c01ff51df3ea4009d107ff31e2d75f06565b2

NVFP4

1

NVIDIA-H100-80GB-HBM3

156d81239cba86447af7298f160f3451fd3ff0527179fc437c034648cd8620a7

NVFP4

1

NVIDIA-H100-NVL

7cec39e34ab92e5690b3f4b77c4e43d1824ba819883b7d7ebd22e5a776d8f8e1

NVFP4

1

NVIDIA-H200-NVL

3385c0aff65ee218ca8bf9e972c4295bff35047961f8be02a47dc886da5df852

NVFP4

1

NVIDIA-RTX-PRO-6000-Blackwell-Server-Edition

7a49ca4e43231ac8f0461533709c3861e5901cad434b79c5e0f0c260436321e8

NVFP4

2

NVIDIA-H200

644d1a99bb25c6d9a4436da1eaa94180e6e521d055fdc372b886c84564033593

NVFP4

2

NVIDIA-RTX-PRO-4500-Blackwell-Server-Edition

fd9dd257f0e94f5ea57713f60d4cdb1fa31db0031cbfc7353b0f80b5c9bdc851

FP8

1

NVIDIA-B200

61b19c34a31ab2002566d45689f1bda87dac7342b239b7cb7b534d821c5939ec

FP8

1

NVIDIA-B300-SXM6-AC

284ac649b9186a4ceef13ac152985b91976348dd0fa861dc38def5d28a9c2ded

FP8

1

NVIDIA-GB200

fcbf637e4b7e25a05ddad6af0eab90ecfad4500a1bfea72187e2fba926d886ca

FP8

1

NVIDIA-GB300

00d789bb6488d33ec372d2075a2a7dca50610978ff252cd3eaf8220ef21551e0

FP8

1

NVIDIA-GH200-144G-HBM3e

8ad15a8760e9488f9d2a1320591a7e8832df56ad37559e17c7f84a847e6313ed

FP8

1

NVIDIA-GH200-480GB

c94d2c4756a6d576201a81dc8f29704237a8c68fb4649e09454a339bea4bbbd2

FP8

1

NVIDIA-H100-80GB-HBM3

a2cb834abd156023424dee1418682216380b140fb61f47c7ac8d6eaf540c902a

FP8

1

NVIDIA-H100-NVL

253b217770976f7be094c3279954817f1509d5b0b29dbf5fdbe28e869d7d91a1

FP8

1

NVIDIA-H200

751c271c4b0bf5f107f68358658607fe3414736458c9c5ef34f2b4e94e6f08a7

FP8

1

NVIDIA-H200-NVL

2fea60c1917c259e031ddbc81ec224958a0bbcfd26a873593c681a9400d9fb9a

FP8

1

NVIDIA-RTX-PRO-6000-Blackwell-Server-Edition

17e81574cb3be3731a4de63e69a7be720f7896b3fd8697d1d8e1cc89bed4e9df

FP8

2

NVIDIA-B200

6fd8f83f0ed11f62c0621b724529f82c261ba5caf8a7ec81bcdea36d464a0835

FP8

2

NVIDIA-H200

7e3234d6b041b45af321a2291467a936e514f2a67c1b2f42fe703399c40a567d

FP8

2

NVIDIA-L40S

427e3d5dba49d1cc33db5a031157123153809fdaa8b893706cc9a6c50db27cf2

FP8

2

NVIDIA-RTX-PRO-4500-Blackwell-Server-Edition

c89a3eba918ebbeca80222f1778326a60d1dc101c04d5a7af4f600d935edcaf9

BF16

1

NVIDIA-B200

cd30cd903aa1c612f0122c03ef90d1fe6c0368fdcdeda4915898ca422dbd5bd6

BF16

1

NVIDIA-B300-SXM6-AC

14788d8dee09fc0e70717f8fe126008621e346c767d8200380cadf205d8f8661

BF16

1

NVIDIA-GB200

b7f9282ff34f4e5815d2faa4d68cbbd9ab23c1c0f1322a966d5ae2e89e7d3c8f

BF16

1

NVIDIA-GB300

b071f7097ef2f0544bd6bc3bef2780e16e6c18b90337c34822817f3fac7158ce

BF16

1

NVIDIA-GH200-144G-HBM3e

a6cecb0d6c95b37f262a2f9dece28843ad121e556e1e92ed7cd3dde60911e2c6

BF16

1

NVIDIA-GH200-480GB

565dbe4d1ac756afaa5ab4bcf658cac56248f27b327d45ed0d938fb57b4a2371

BF16

1

NVIDIA-H100-NVL

b7f5b91df660fdf407bdaeb5946f6d2f53f54ae8bff19637cbb7be6da56f1771

BF16

1

NVIDIA-H200-NVL

324d3669a879b87e592ebf2325007071ac9f1ccb1e6eada836e90c04aa409773

BF16

1

NVIDIA-RTX-PRO-6000-Blackwell-Server-Edition

228f3cab54154ca85dd291d0ec67cd0ebe2522f2a0ccdfa3e7c7b76fc9a8329f

BF16

2

NVIDIA-A100-SXM4-80GB

86157c6821e4bfec6a8bd325696d339d0335957670f7a7f791d156d75e7066b0

BF16

2

NVIDIA-B200

ede7c1193e24d624eb40c8e2e8453fe419c44b345a371bf26f165d525d3badd6

BF16

2

NVIDIA-H100-80GB-HBM3

d2823b8224a0e681bd77d42142f15d9d59d8362a67d552e47bd16da06de7dbc6

BF16

2

NVIDIA-H200

e70aced682a875db2e916fc6db718a5456bc1516e332ec40d06b1138dc8e875d

BF16

2

NVIDIA-L40S

6746d14a07b735d835726ed6dbfa08dd828a4a131ded77382ad7186063220bdd

BF16

4

NVIDIA-A100-SXM4-40GB

c0536126549d169e656464e493e01eab3979f862342ae955a61d1f9886d7f396

BF16

4

NVIDIA-RTX-PRO-4500-Blackwell-Server-Edition

9b8040192838c52e3feb06bf585a66c421e3cd0858b56e0cdf477051d07fe5cf
Optimized deployment profiles#

Precision

TP

GPU

LoRA

Tuning Objective

Profile ID

NVFP4

1

NVIDIA-B200

No

Throughput

c4dae72230ad2b0c56220d9d801642564445bc21d6cbdd52702300c273a4a8aa

FP8

2

NVIDIA-B200

No

Throughput

6fd8f83f0ed11f62c0621b724529f82c261ba5caf8a7ec81bcdea36d464a0835

BF16

2

NVIDIA-B200

No

Throughput

ede7c1193e24d624eb40c8e2e8453fe419c44b345a371bf26f165d525d3badd6

NVFP4

2

NVIDIA-H200

No

Throughput

644d1a99bb25c6d9a4436da1eaa94180e6e521d055fdc372b886c84564033593

FP8

2

NVIDIA-H200

No

Throughput

7e3234d6b041b45af321a2291467a936e514f2a67c1b2f42fe703399c40a567d

BF16

2

NVIDIA-H200

No

Throughput

e70aced682a875db2e916fc6db718a5456bc1516e332ec40d06b1138dc8e875d
Verified GPUs

This model has been verified on the following GPUs:

  • NVIDIA-A100-SXM4-40GB

  • NVIDIA-A100-SXM4-80GB

  • NVIDIA-B200

  • NVIDIA-B300-SXM6-AC

  • NVIDIA-GB200

  • NVIDIA-GB300

  • NVIDIA-GH200-144G-HBM3e

  • NVIDIA-GH200-480GB

  • NVIDIA-H100-80GB-HBM3

  • NVIDIA-H100-NVL

  • NVIDIA-H200

  • NVIDIA-H200-NVL

  • NVIDIA-L40S

  • NVIDIA-RTX-PRO-4500-Blackwell-Server-Edition

  • NVIDIA-RTX-PRO-6000-Blackwell-Server-Edition

Model-Free NIM#

vLLM#

The following models are tested and validated for nvidia/vllm-model-free-nim:

Beyond the validated list, the vLLM Model-Free NIM can serve any model that the vLLM version in the image supports. Refer to Model-Free NIM for deployment details.

The following model is tested and validated for nvidia/vllm-model-free-nim-experimental:

Verified GPUs

The vLLM Model-Free NIM has been verified on the following GPUs:

  • NVIDIA-A100-80GB-PCIe

  • NVIDIA-A100-PCIE-40GB

  • NVIDIA-A100-SXM4-40GB

  • NVIDIA-A100-SXM4-80GB

  • NVIDIA-A10G

  • NVIDIA-B200

  • NVIDIA-B300-SXM6-AC

  • NVIDIA-GB10

  • NVIDIA-GB200

  • NVIDIA-GB300

  • NVIDIA-GB300-WS

  • NVIDIA-GH200-144G-HBM3e

  • NVIDIA-GH200-480GB

  • NVIDIA-H100-80GB-HBM3

  • NVIDIA-H100-NVL

  • NVIDIA-H100-PCIe

  • NVIDIA-H200

  • NVIDIA-H200-NVL

  • NVIDIA-L40S

  • NVIDIA-RTX-PRO-4500-Blackwell-Server-Edition

  • NVIDIA-RTX-PRO-6000-Blackwell-Server-Edition

Note

GPU requirements vary by model. Refer to the model-specific tables below for the GPU SKUs and tensor-parallel sizes of the released tuning configurations.

Apriel-Nemotron-15b-Thinker#

Apriel-Nemotron-15b-Thinker is served by the vLLM Model-Free NIM 2.1.4 container, nvcr.io/nim/nvidia/vllm-model-free-nim:2.1.4. The container includes the following throughput tuning configurations:

Apriel-Nemotron-15b-Thinker Supported Configurations#

GPU

Precision

TP

LoRA

Tuning Recipe

NVIDIA-B200

BF16

1

No

apriel-nemotron-15b-thinker.yaml

NVIDIA-H200

BF16

1

No

apriel-nemotron-15b-thinker.yaml

Both configurations use BF16 weights and an FP8 KV cache. The tuning recipe is selected for the matching checkpoint and GPU configuration. Refer to About Model-Free NIM for deployment instructions.

Verified GPUs

This model has been verified on the following GPUs:

  • NVIDIA-B200

  • NVIDIA-H200

Codestral-2501#

Codestral-2501 is served by the vLLM Model-Free NIM 2.1.4 container, nvcr.io/nim/nvidia/vllm-model-free-nim:2.1.4. The container includes the following throughput tuning configurations:

Codestral-2501 Supported Configurations#

GPU

Precision

TP

LoRA

Tuning Recipe

NVIDIA-B200

BF16

1

No

codestral-2501.yaml

NVIDIA-H200

BF16

1

No

codestral-2501.yaml

Both configurations use BF16 weights and an FP8 KV cache. The tuning recipe is selected for the matching checkpoint and GPU configuration. Refer to About Model-Free NIM for deployment instructions.

Verified GPUs

This model has been verified on the following GPUs:

  • NVIDIA-B200

  • NVIDIA-H200

Mistral-Medium-3.5-128B#

Mistral-Medium-3.5-128B is served by the vLLM Model-Free NIM 2.1.4 container, nvcr.io/nim/nvidia/vllm-model-free-nim:2.1.4. The following throughput-tuned configurations have been validated:

Mistral Medium 3.5 128B Validated Configurations#

GPU

Checkpoint

Precision

TP

LoRA

Tuning Recipe

NVIDIA-B200

nvidia/Mistral-Medium-3.5-128B-NVFP4

NVFP4

4

No

mistral-medium-3.5-128b-nvfp4.yaml

NVIDIA-H200

mistralai/Mistral-Medium-3.5-128B

FP8

8

No

mistral-medium-3.5-128b.yaml

Both tuning recipes are included in the container and selected for matching checkpoints and GPU configurations. The H200 recipe uses an FP8 KV cache. For the NVFP4 checkpoint, use --tokenizer-mode hf. For the Mistral-native FP8 checkpoint, use --tokenizer-mode mistral --config-format mistral --load-format mistral. Pass these options through NIM_PASSTHROUGH_ARGS. Refer to About Model-Free NIM for deployment instructions.

Verified GPUs

This model has been verified on the following GPUs:

  • NVIDIA-B200

  • NVIDIA-H200

Qwen3.8-Flash-Next#

Qwen3.8-Flash-Next is served by the vLLM Model-Free NIM container, nvcr.io/nim/nvidia/vllm-model-free-nim:2.1.4-qwen3.8-flash-next. Note the custom tag.

Set NIM_MODEL_PATH to the checkpoint for the precision you want. The checkpoints need about 133 GB (NVFP4), 186 GB (FP8), or 360 GB (BF16) of disk space. For deployment steps, refer to the dedicated Get Started.

The container includes the following throughput tuning configurations:

Qwen3.8-Flash-Next Supported Configurations#

Precision

TP

GPU

Checkpoint (NIM_MODEL_PATH)

Profile ID

NVFP4

1

NVIDIA-B300-SXM6-AC

hf://nvidia/Qwen3.8-Flash-Next-NVFP4

019381de56cbb5b4037542379f7093b1b9534b1e79abd58de69ddc4a51ead79d

NVFP4

1

NVIDIA-GB300

hf://nvidia/Qwen3.8-Flash-Next-NVFP4

2377b57b80450a37dad96a2338cac167f0db4d4cd9720544779ab7a3c1a24116

NVFP4

2

NVIDIA-B200

hf://nvidia/Qwen3.8-Flash-Next-NVFP4

0f131b5aab66ccd189f77633e31821cc6e921eb8ceab9329a79d9405f5d9cede

NVFP4

2

NVIDIA-GB200

hf://nvidia/Qwen3.8-Flash-Next-NVFP4

900d253530249f5324099ec27bebc716685bdfcf2f36800485daa4ab37c005cc

NVFP4

4

NVIDIA-RTX-PRO-6000-Blackwell-Server-Edition

hf://nvidia/Qwen3.8-Flash-Next-NVFP4

64ef57c5f91f95c3c43b8581eadf4e248db5c60acceaf8680fef07351cfc1f5a

FP8

2

NVIDIA-B200

hf://Qwen/Qwen3.8-Flash-Next-FP8

d679efb9225bafe7b7d2c747c08841e8f8e247ec207e3c63049549ae25199b9b

FP8

2

NVIDIA-B300-SXM6-AC

hf://Qwen/Qwen3.8-Flash-Next-FP8

716879e3a114c157a03ad3c7a66d2caeca14e9c7158c321c25e379b7632f2a02

FP8

2

NVIDIA-GB200

hf://Qwen/Qwen3.8-Flash-Next-FP8

b07c1898bef905395d6fd524c59fa9c13d293636231f51226e2bd5142828167f

FP8

2

NVIDIA-GB300

hf://Qwen/Qwen3.8-Flash-Next-FP8

6a943c1622c3d8bf50d2f2e8ec1fd17e4c6bca4f5ac7d07e3e2db5611578724c

FP8

2

NVIDIA-H200

hf://Qwen/Qwen3.8-Flash-Next-FP8

cdf5cf211de2689299a6a27c7d595f5da2aa3dfd8526cb5ac0c46fdea68bc8d3

FP8

4

NVIDIA-H100-NVL

hf://Qwen/Qwen3.8-Flash-Next-FP8

8f14e80d5b91ff1140527a79fae11724791ff3365a9e72b40ae22e8615a1eb51

FP8

4

NVIDIA-H200-NVL

hf://Qwen/Qwen3.8-Flash-Next-FP8

07c78199c6a21fc17771216f3c9a57ac6522e27e0d8d6c3f845f2917f833b158

FP8

4

NVIDIA-RTX-PRO-6000-Blackwell-Server-Edition

hf://Qwen/Qwen3.8-Flash-Next-FP8

dbf0755bbf886934875f66af4b9f7176384c279d8a917b7933146604a8542313

BF16

4

NVIDIA-B300-SXM6-AC

hf://Qwen/Qwen3.8-Flash-Next

9b51e6e835abe95b619d0c52ef16703394231cce7e892161d1cc6952e50b148d

BF16

4

NVIDIA-GB200

hf://Qwen/Qwen3.8-Flash-Next

4dab55bb6c3472ea2f28a25104e40a71378dece59450b45d0546a068d5e43648

BF16

4

NVIDIA-GB300

hf://Qwen/Qwen3.8-Flash-Next

07c00dbc5ce060b63ddffd7f11f651c7f86f29f6f9f64ac19e50a84cecaff447

BF16

8

NVIDIA-B200

hf://Qwen/Qwen3.8-Flash-Next

309fe9b885e4016dc531ef6c2d718a3ee0ffc4d8982f939225527c60c0397755

The FP8 recipes for NVIDIA-B200 and NVIDIA-H200 are throughput-tuned and run expert parallel 2 with MTP speculative decoding. They need --shm-size=16GB, and NIM_TENSOR_PARALLEL_SIZE must stay unset. Refer to Tuned FP8 Recipes on B200 and H200.

Verified GPUs

This model has been verified on the following GPUs:

  • NVIDIA-B200

  • NVIDIA-B300-SXM6-AC

  • NVIDIA-GB200

  • NVIDIA-GB300

  • NVIDIA-H100-NVL

  • NVIDIA-H200

  • NVIDIA-H200-NVL

  • NVIDIA-RTX-PRO-6000-Blackwell-Server-Edition

Step-5-Preview#

Step 5 Preview is served by the vLLM Model-Free NIM (Experimental) 2.1.6-variant container, nvcr.io/nim/nvidia/vllm-model-free-nim-experimental:2.1.6-variant.

The following table lists the supported profile configurations for stepfun-ai/Step-5-Preview. For more information, refer to the dedicated Get Started.

GPU

Precision

GPUs

TP

Expert Parallelism

MTP

LoRA

Recipe File

H20-3e

FP8

8

8

No

3

No

step5-preview-fp8-h20-3e-8gpu-tp8-mtp3.yaml

H200

FP8

8

8

No

3

No

step5-preview-fp8-h200-8gpu-tp8-mtp3.yaml

B200

NVFP4

4

4

Yes

3

No

step5-preview-nvfp4-b200-4gpu-tp4-dp1-ep-mtp3.yaml

GB200

NVFP4

4

4

Yes

3

No

step5-preview-nvfp4-gb200-4gpu-tp4-dp1-ep-mtp3.yaml

All four configurations use multi-token prediction with three speculative tokens (MTP3). The NVFP4 configurations add data parallelism of 1 (DP1) alongside expert parallelism.

Each configuration is delivered as a tuning config file in the Step-5 Preview Serving Recipes resource on NGC.

Verified GPUs

This model has been verified on the following GPUs:

  • NVIDIA-B200

  • NVIDIA-GB200

  • NVIDIA-H20-3e

  • NVIDIA-H200

SGLang#

The following models are tested and validated for nvidia/sglang-model-free-nim:

Beyond the validated list, the SGLang Model-Free NIM can serve any model that the SGLang version in the image supports. Refer to Model-Free NIM for deployment details.

The following models are tested and validated for nvidia/sglang-model-free-nim-experimental:

GLM-5.3#

GLM-5.3 is served by the SGLang Model-Free NIM 2.1.4 container, nvcr.io/nim/nvidia/sglang-model-free-nim:2.1.4. The container includes the following throughput tuning configurations:

GLM-5.3 Supported Configurations#

GPU

Checkpoint

Precision

TP

LoRA

Tuning Recipe

NVIDIA-B200

nvidia/GLM-5.3-NVFP4

NVFP4 + FP8

4

No

glm-5.3-nvfp4.yaml

NVIDIA-B200

nvidia/GLM-5.3-NVFP4

NVFP4 + FP8

8

No

glm-5.3-nvfp4.yaml

NVIDIA-H200

zai-org/GLM-5.3

FP8

8

No

glm-5.3-fp8.yaml

The tuning recipes are selected for matching checkpoints and GPU configurations. The B200 checkpoint uses NVFP4 for MoE experts and FP8 for the dense MLP layers. The H200 recipe uses an FP8 KV cache. Refer to About Model-Free NIM for deployment instructions.

Verified GPUs

This model has been verified on the following GPUs:

  • NVIDIA-B200

  • NVIDIA-H200

GLM-5.3-Flash#

GLM-5.3-Flash is served by the SGLang Model-Free NIM 2.1.4 container, nvcr.io/nim/nvidia/sglang-model-free-nim:2.1.4-variant-glm-5.3-flash.

Set NIM_MODEL_PATH to the checkpoint for the precision you want, including the NVFP4 checkpoint for the throughput-optimized GB200 profile. For deployment steps, refer to the dedicated Get Started.

GLM-5.3-Flash Supported Configurations#

Precision

TP

GPU

Checkpoint (NIM_MODEL_PATH)

Profile ID

NVFP4

4

NVIDIA-B200

hf://nvidia/GLM-5.3-Flash-NVFP4

d0e6e1a4e79b872a77cfa5faf8742f0c06a7ce552420bdba6eaf557278756203

NVFP4

2

NVIDIA-B300-SXM6-AC

hf://nvidia/GLM-5.3-Flash-NVFP4

caeafff184c5ddc019a4dec380841200e5ac8eceebd39fcbd66c58258b794dfb

NVFP4

4

NVIDIA-H200

hf://nvidia/GLM-5.3-Flash-NVFP4

4aca597ab947c56980070dfb62238e742b444d5c47f49a42a384ff88a5247303

NVFP4

4

NVIDIA-H200-NVL

hf://nvidia/GLM-5.3-Flash-NVFP4

03bfe0cfafc7135c482a9e63f4b00d5fab0d0ddd2599aaee89c81fe14ee71b76

NVFP4

8

NVIDIA-H100-80GB-HBM3

hf://nvidia/GLM-5.3-Flash-NVFP4

f47583feff1e90ea79a05f1e511f917c49c3d4f3db45cecce856dbca24d1a547

NVFP4

8

NVIDIA-H100-NVL

hf://nvidia/GLM-5.3-Flash-NVFP4

baf4c2aab0e2f244a3f1f384bee5783e2f6afd3c47b70d0b4ede5c827cd503c4

FP8

4

NVIDIA-B200

hf://zai-org/GLM-5.3-Flash

646d2e5aeba5a8bc21e5f0283aa1d4194763724e7188598d8d75077f7bbc3c34

FP8

4

NVIDIA-B300-SXM6-AC

hf://zai-org/GLM-5.3-Flash

6780254ec7375998309aa95c0fd5aa5ce37b66b5834cabfa5f09e40a3713b876

FP8

4

NVIDIA-GB200

hf://zai-org/GLM-5.3-Flash

dc4413ba1120e2e9390476081494b57bce28e66e4c84ba1abd538f302396a408

FP8

4

NVIDIA-H200

hf://zai-org/GLM-5.3-Flash

d5c2ad86733804c4800af1b06c514ef3a0b37f7f93495a71b6af1298085f6b78

FP8

4

NVIDIA-H200-NVL

hf://zai-org/GLM-5.3-Flash

78083888c84b2207be4e63f97368021754d695a147160b5dfa29887bcad86690

FP8

8

NVIDIA-H100-80GB-HBM3

hf://zai-org/GLM-5.3-Flash

fb32e6152dbf1b1d9df5a5b5b069e90935c929b705173ea5d83e2dbc5d239abe

FP8

8

NVIDIA-H100-NVL

hf://zai-org/GLM-5.3-Flash

72c3835033db4d7970cffc89d4ef936eb14f8f1430fa321e8d425c5172cd8d3c

BF16

4

NVIDIA-B300-SXM6-AC

hf://zai-org/GLM-5.3-Flash-BF16

4bb955cd9a620abfef9439ed2113a386d2a1a9c7c316d77b12807f2b3b9be349

BF16

8

NVIDIA-B200

hf://zai-org/GLM-5.3-Flash-BF16

569079ed4e36c21f44f7e5d77937bf3457055299b0b917f8273651b14ad84e87

BF16

8

NVIDIA-H200

hf://zai-org/GLM-5.3-Flash-BF16

9d87d3c47d7499d35947b9d4f455c5585251b23cb9cd2ed8db9b970ddb09e1ce

BF16

8

NVIDIA-H200-NVL

hf://zai-org/GLM-5.3-Flash-BF16

ca146d5d244b7e07d5aca3ef70fba251ac6d507cf4ea45ec62faf30992f4f255
Optimized deployment profiles#

Precision

TP

GPU

LoRA

Tuning Objective

Checkpoint (NIM_MODEL_PATH)

Profile ID

NVFP4

4

NVIDIA-GB200

No

Throughput

hf://nvidia/GLM-5.3-Flash-NVFP4

f9bd55116d3fdfe1704064ca7c0899c028089ae8f92f140bca6e809457e56670
Verified GPUs

This model has been verified on the following GPUs:

  • NVIDIA-B200

  • NVIDIA-B300-SXM6-AC

  • NVIDIA-GB200

  • NVIDIA-H100-80GB-HBM3

  • NVIDIA-H100-NVL

  • NVIDIA-H200

  • NVIDIA-H200-NVL

Qwen3.8-2.4T-A95B#

Qwen3.8-2.4T-A95B is served by the SGLang Model-Free NIM container. The following table lists the validated multi-node configuration for Qwen/Qwen3.8-2.4T-A95B-FP8, as documented in the Get Started with Qwen3.8-2.4T-A95B guide.

Qwen3.8-2.4T-A95B Supported Configurations#

Precision

TP

PP

Nodes

GPUs per Node

FP8

16

1

4

4

Deployment requires Kubernetes and at least four GB300-NVL72 nodes. The configuration uses 16 GPUs across four nodes, with four GPUs per node. Image and video input are not supported. Refer to the linked guide for the multi-node Helm configuration and deployment requirements.

Verified GPUs

This model has been verified on the following GPUs:

  • NVIDIA-GB300 (GB300-NVL72 only)

Muse Spark Open#

Muse Spark Open is served by the SGLang Model-Free NIM (Experimental) container, nvcr.io/nim/nvidia/sglang-model-free-nim-experimental:2.1.6-variant.

The following table lists the supported profile configurations for meta-models/Muse-Spark-Open-816B-A42B-NVFP4. For deployment instructions, refer to Get Started with Muse Spark Open.

Muse Spark Open Supported Configurations#

Precision

TP8 or EP8

NVFP4

sglang-nvfp4-tp8-pp1

There are no packaged profile IDs for this NIM. The NIM infers the configuration in the table.

Verified GPUs

This model has been verified on the following GPUs:

  • NVIDIA-B200

MiniMax M3.1 Flash#

MiniMax M3.1 Flash is served by the SGLang Model-Free NIM (Experimental) container, nvcr.io/nim/nvidia/sglang-model-free-nim-experimental:2.1.7-variant.

Use the matching 2.1.7-variant serving recipe from the SGLang Model-Free NIM Experimental Collection for your GPU and topology. The container does not include model weights or model-specific recipe settings.

For recipe download, launch, and API examples, refer to Get Started with MiniMax M3.1 Flash.

The following FP8 throughput recipes have been validated:

MiniMax M3.1 Flash Supported Configurations#

GPU

Precision

TP

LoRA

Tuning Recipe

NVIDIA-B200

FP8

4

No

b200_throughput_fp8_tp4.yaml

NVIDIA-B200

FP8

8

No

b200_throughput_fp8_tp8.yaml

NVIDIA-GB200

FP8

4

No

gb200_throughput_fp8_tp4.yaml

NVIDIA-H200

FP8

8

No

h200_throughput_fp8_tp8.yaml

NVIDIA-H20-3e

FP8

8

No

h20-3e_throughput_fp8_tp8.yaml
Verified GPUs

This model has been verified on the following GPUs:

  • NVIDIA-B200

  • NVIDIA-GB200

  • NVIDIA-H20-3e

  • NVIDIA-H200

Select a recipe for the exact hardware and topology you deploy. A verified GPU does not imply support for every precision or tensor-parallel configuration.

1.x NIM LLM Models#

For more information on version 1.x NIMs, refer to the 1.15 version of the NIM LLM Supported Models page.

Show 1.x models

Model (Hardware Requirements)

Organization/Model ID (Catalog Page)

Bielik 11B v2.3 Instruct

speakleash/bielik-11b-v2.3-instruct

Code Llama 13B Instruct

meta/codellama-13b-instruct

Code Llama 34B Instruct

meta/codellama-34b-instruct

Code Llama 70B Instruct

meta/codellama-70b-instruct

DeepSeek Coder V2 Lite Instruct

deepseek-ai/deepseek-coder-v2-lite-instruct

DeepSeek R1

deepseek-ai/deepseek-r1

DeepSeek R1 Distill Llama 8B

deepseek-ai/deepseek-r1-distill-llama-8b

DeepSeek R1 Distill Llama 70B

deepseek-ai/deepseek-r1-distill-llama-70b

DeepSeek R1 Distill Llama 8B RTX

deepseek-ai/deepseek-r1-distill-llama-8b

DeepSeek R1 Distill Qwen 32B

deepseek-ai/deepseek-r1-distill-qwen-32b

DeepSeek-V3.1-Terminus

deepseek-ai/deepseek-v3.1-terminus

DeepSeek-V3.2-Exp

deepseek-ai/deepseek-v32-exp-nim

DeepSeek-V4-Flash

deepseek-ai/deepseek-v4-flash

DeepSeek-V4-Pro

deepseek-ai/deepseek-v4-pro

EuroLLM 9B Instruct

utter-project/eurollm-9b-instruct

Gemma 2 2B

google/gemma-2-2b-instruct

Gemma 2 9B

google/gemma-2-9b-it

Gemma2 9B CPT Sahabat-AI v1 Instruct

gotocompany/gemma2-9b-cpt-sahabatai-v1-instruct

Gemma 3 1B Instruct

google/gemma-3-1b-it

GLM-5

zai-org/glm-5

GLM-5.1

zai-org/glm-51

GPT-OSS-20B

openai/gpt-oss-20b

GPT-OSS-120B

openai/gpt-oss-120b

Granite 3.3 8B Instruct

ibm-granite/granite-3.3-8b-instruct

GreenMind Medium 14B R1

greennode/greenmind-medium-14b-r1

Kanana 1.5 8B Instruct 2505

kakaocorp/kanana-1.5-8b-instruct-2505

(Meta) Llama 2 7B Chat

meta/llama-2-7b-chat

(Meta) Llama 2 13B Chat

meta/llama-2-13b-chat

(Meta) Llama 2 70B Chat

meta/llama-2-70b-chat

Llama 3 SQLCoder 8B

defog/llama-3-sqlcoder-8b

Llama 3 Swallow 70B Instruct V0.1

tokyotech-llm/llama-3-swallow-70b-instruct-v0.1

Llama 3 Taiwan 70B Instruct

yentinglin/llama-3-taiwan-70b-instruct

Llama 3.1 8B Base

meta/llama-3.1-8b-base

Llama 3.1 8B Instruct

meta/llama-3.1-8b-instruct

Llama-3.1-8b-Instruct-DGX-Spark

meta/llama-3.1-8b-instruct-dgx-spark

Llama-3.1-8B-Instruct PB 25h2

meta/llama-3.1-8b-instruct-pb25h2

Llama 3.1 8B Instruct RTX

meta/llama-3.1-8b-instruct

Llama 3.1 70B Instruct

meta/llama-3.1-70b-instruct

Llama-3.1-70B-Instruct PB 25h2

meta/llama-3.1-70b-instruct-pb25h2

Llama 3.1 405B Instruct

meta/llama-3.1-405b-instruct

Llama 3.1 Nemotron Nano 4B V1.1

nvidia/llama3.1-nemotron-nano-4b-v1.1

Llama 3.1 Nemotron Nano 8B V1

nvidia/llama-3.1-nemotron-nano-8b-v1

Llama-3.1-Nemotron-Nano-8B-Healthcare-Text2sql-v1.0

nvidia/llama-3.1-nemotron-nano-8b-healthcare-text2sql-v1.0

Llama 3.1 Nemotron Ultra 253B V1

nvidia/llama-3.1-nemotron-ultra-253b-v1

Llama 3.1 Nemotron 70B Instruct

nvidia/llama-3.1-nemotron-70b-instruct

Llama 3.1 Swallow 8B Instruct v0.1

tokyotech-llm/llama-3.1-swallow-8b-instruct-v0.1

Llama 3.1 Swallow 70B Instruct v0.1

tokyotech-llm/llama-3.1-swallow-70b-instruct-v0.1

Llama 3.1 Typhoon 2 8B Instruct

scb10x/llama3.1-typhoon2-8b-instruct

Llama 3.1 Typhoon 2 70B Instruct

scb10x/llama-3.1-typhoon2-70b-instruct

Llama 3.2 1B Instruct

meta/llama-3.2-1b-instruct

Llama 3.2 3B Instruct

meta/llama-3.2-3b-instruct

Llama 3.3 70B Instruct

meta/llama-3.3-70b-instruct

Llama 3.3 Nemotron Super 49B V1

nvidia/llama-3.3-nemotron-super-49b-v1

Llama-3.3-Nemotron-Super-49B-v1.5

nvidia/llama-3.3-nemotron-super-49b-v1.5

Llama-3.3-Nemotron-Super-49B-Healthcare-Text2sql-v1.0

nvidia/llama-3.3-nemotron-super-49b-healthcare-text2sql-v1.0

Llama-3.3-Nemotron-Super-49B-v1.5 PB 25h2

nvidia/llama-3.3-nemotron-super-49b-v1.5-pb25h2

Meta Llama 3 8B Instruct

meta/llama3-8b-instruct

Meta Llama 3 70B Instruct

meta/llama3-70b-instruct

Muse Glimmer

meta/muse-glimmer

MiMo-V2-Flash

xiaomi/mimo-v2-flash-experimental

MiniMax-M2.5

minimax-ai/minimax-m25

MiniMax-M2.7

minimax-ai/minimax-m27

Mistral 7B Instruct V0.3

mistralai/mistral-7b-instruct-v0.3

Mistral NeMo 12B Instruct RTX

nv-mistralai/mistral-nemo-12b-instruct

Mistral NeMo 12B Instruct

nv-mistralai/mistral-nemo-12b-instruct

Mistral NeMo Minitron 8B 8K Instruct

nv-mistralai/mistral-nemo-minitron-8b-8k-instruct

Mistral Small 24b Instruct 2501

mistralai/mistral-small-24b-instruct-2501

Mixtral 8x7B Instruct V0.1

mistralai/mixtral-8x7b-instruct-v0-1

Mixtral 8x22B Instruct V0.1

mistralai/mixtral-8x22b-instruct-v01

Nemotron 4 340B Instruct

nvidia/nemotron-4-340b-instruct

Nemotron 4 340B Reward

nvidia/nemotron-4-340b-reward

NVIDIA Nemotron 3 Nano

nvidia/nemotron-3-nano

NVIDIA-Nemotron-Nano-9B-v2

nvidia/nvidia-nemotron-nano-9b-v2

NVIDIA-Nemotron-Nano-9B-v2-DGX-Spark

nvidia/nvidia-nemotron-nano-9b-v2-dgx-spark

Nemotron-3-Super-120B-A12B

nvidia/nemotron-3-super-120b-a12b

Phi 3 Mini 4K Instruct

microsoft/phi-3-mini-4k-instruct

Phi 4 Mini Instruct

microsoft/phi-4-mini-instruct

Phind Codellama 34B V2 Instruct

phind/phind-codellama-34b-v2-instruct

Qwen3-Coder-Next

qwen/qwen3-coder-next

Qwen3-Next-80B-A3B-Instruct

qwen/qwen3-next-80b-a3b-instruct

Qwen3 Next 80B A3B Thinking

qwen/qwen3-next-80b-a3b-thinking

Qwen3-32B

qwen/qwen3-32b

Qwen3-32B NIM for DGX Spark

qwen/qwen3-32b-dgx-spark

Qwen2.5 Coder 32B Instruct

qwen/qwen2.5-coder-32b-instruct

Qwen2.5 7B Instruct

qwen/qwen-2.5-7b-instruct

Riva Translate 4B Instruct

nvidia/riva_translate_4b_instruct

Riva-Translate-4b-Instruct-v1.1

nvidia/riva-translate-4b-instruct-v1.1

Sarvam - M

sarvamai/sarvam-m

SILMA 9B Instruct v1.0

silma-ai/silma-9b-instruct-v1.0

StarCoder2 7B

bigcode/starcoder2-7b

StarCoderBase 15.5B

bigcode/starcoderbase-15b

Stockmark-2-100B-Instruct

stockmark/stockmark-2-100b-instruct

Teuken 7B Instruct Commercial v0.4

opengpt-x/teuken-7b-instruct-commercial-v0.4

Additional VLM Models#

VLMs that were released prior to version 2.0.12 are documented on the NVIDIA NIM for Vision Language Models site. Refer to that site for more information on running these VLM models.

Show additional VLM models

Model (Hardware Requirements)

Organization/Model ID (Catalog Page)

Cosmos 3 Reasoner

nvidia/cosmos3-reasoner

Cosmos Reason1 7B

nvidia/cosmos-reason1-7b

Cosmos Reason2

nvidia/cosmos-reason2-8b

DiffusionGemma 26B A4B IT

google/diffusiongemma-26b-a4b-it

Kimi-K2.5

moonshotai/kimi-k2.5

Llama 3.1 Nemotron Nano VL 8B v1

nvidia/llama-3.1-nemotron-nano-vl-8b-v1

Llama 3.2 11B Vision Instruct

meta/llama-3.2-11b-vision-instruct

Llama 3.2 90B Vision Instruct

meta/llama-3.2-90b-vision-instruct

Llama 4 Maverick 17B 128E Instruct

meta/llama-4-maverick-17b-128e-instruct

Llama 4 Scout 17B 16E Instruct

meta/llama-4-scout-17b-16e-instruct

Ministral 3 14B Instruct 2512

mistralai/ministral-3-14b-instruct-2512

Mistral Large 3 675B Instruct 2512

mistralai/mistral-large-3-675b-instruct-2512

Mistral Small 3.2 24B Instruct 2506

mistralai/mistral-small-3.2-24b-instruct-2506

Nemotron Nano 12B v2 VL

nvidia/nemotron-nano-12b-v2-vl

Qwen3.5

qwen/qwen3.5-35b-a3b

Qwen3.6-27B

qwen/qwen3.6-27b

Qwen3.6-35B-A3B

qwen/qwen3.6-35b-a3b

Step 3.7 Flash

stepfun-ai/step-3.7-flash