Support Matrix for NIMs#

This page lists supported models, deployment profiles, and verified hardware SKUs for NIM LLM and VLM. This page is organized according to model type:

Use the following sections to identify the supported deployment profiles for each model. Profile strings follow a naming convention described in Model Profiles and Selection.

Optimized profiles are listed separately in the model sections because multiple configurations can share the same precision and tensor-parallel size. For selection instructions, refer to Model Profiles and Selection.

Important

The list of NIM Certified models is available on the NGC Catalog site.

Use the table below to filter NIM profiles by GPU, tensor parallelism (TP), precision, model type, LoRA support, and model name. Each row is one supported profile; details also appear in the per-model sections that follow.

Optimized profile entries identify their tuning objective and link to the full profile ID in the model section. The filters also apply to rows in the optimized profile tables below.

Note

The GB300-WS GPU is more commonly known as DGX Station.

ModelTPPrecisionLoRA
deepseek-v4-pro-0813 8 FP8 No
deepseek-v4.1-flash 4 FP8 No
deepseek-v4.1-flash 8 FP8 No
gemma-4-26b-a4b-it 1 BF16 No
gemma-4-26b-a4b-it 2 BF16 No
gemma-4-26b-a4b-it 4 BF16 No
gemma-4-26b-a4b-it 1 NVFP4 No
gemma-4-31b-it 1 BF16 No
gemma-4-31b-it 2 BF16 No
gemma-4-31b-it 4 BF16 No
gemma-4-31b-it 2 INT4 No
gemma-4-31b-it 1 NVFP4 No
gemma-4-31b-it 2 NVFP4 No
gemma-4-31b-it 4 NVFP4 No
GLM-5.2 4 FP8 No
GLM-5.2 4 FP8 Yes
GLM-5.2 4 NVFP4 No
GLM-5.2 4 NVFP4 Yes
GLM-5.2 8 FP8 No
GLM-5.2 8 FP8 Yes
GLM-5.2 8 NVFP4 No
GLM-5.2 8 NVFP4 Yes
gpt-oss-120b 1 MXFP4 No
gpt-oss-120b 2 MXFP4 No
gpt-oss-120b 4 MXFP4 No
gpt-oss-120b 8 MXFP4 No
gpt-oss-120b 1 MXFP4 Yes
gpt-oss-120b 2 MXFP4 Yes
gpt-oss-120b 4 MXFP4 Yes
gpt-oss-120b 8 MXFP4 Yes
gpt-oss-20b 1 MXFP4 No
gpt-oss-20b 2 MXFP4 No
gpt-oss-20b 4 MXFP4 No
gpt-oss-20b 8 MXFP4 No
gpt-oss-20b 1 MXFP4 Yes
gpt-oss-20b 2 MXFP4 Yes
gpt-oss-20b 4 MXFP4 Yes
gpt-oss-20b 8 MXFP4 Yes
Inkling 4 NVFP4 No
Inkling 4 NVFP4 Yes
Inkling 8 NVFP4 No
Inkling 8 NVFP4 Yes
Inkling 8 NVFP4 No
kimi-k2.6 4 NVFP4 No
kimi-k2.6 8 INT4 No
llama-3.1-70b-instruct 1 BF16 No
llama-3.1-70b-instruct 2 BF16 No
llama-3.1-70b-instruct 4 BF16 No
llama-3.1-70b-instruct 8 BF16 No
llama-3.1-70b-instruct 1 BF16 Yes
llama-3.1-70b-instruct 2 BF16 Yes
llama-3.1-70b-instruct 4 BF16 Yes
llama-3.1-70b-instruct 8 BF16 Yes
llama-3.1-70b-instruct 1 FP8 No
llama-3.1-70b-instruct 2 FP8 No
llama-3.1-70b-instruct 4 FP8 No
llama-3.1-70b-instruct 8 FP8 No
llama-3.1-70b-instruct 1 FP8 Yes
llama-3.1-70b-instruct 2 FP8 Yes
llama-3.1-70b-instruct 4 FP8 Yes
llama-3.1-70b-instruct 8 FP8 Yes
llama-3.1-70b-instruct 1 NVFP4 No
llama-3.1-70b-instruct 2 NVFP4 No
llama-3.1-70b-instruct 4 NVFP4 No
llama-3.1-70b-instruct 8 NVFP4 No
llama-3.1-70b-instruct 1 NVFP4 Yes
llama-3.1-70b-instruct 2 NVFP4 Yes
llama-3.1-70b-instruct 4 NVFP4 Yes
llama-3.1-70b-instruct 8 NVFP4 Yes
llama-3.1-8b-instruct 1 BF16 No
llama-3.1-8b-instruct 1 BF16 Yes
llama-3.1-8b-instruct 1 FP8 No
llama-3.1-8b-instruct 1 FP8 Yes
llama-3.1-8b-instruct 1 NVFP4 No
llama-3.1-8b-instruct 1 NVFP4 Yes
llama-3.3-70b-instruct 1 BF16 No
llama-3.3-70b-instruct 2 BF16 No
llama-3.3-70b-instruct 4 BF16 No
llama-3.3-70b-instruct 8 BF16 No
llama-3.3-70b-instruct 1 BF16 Yes
llama-3.3-70b-instruct 2 BF16 Yes
llama-3.3-70b-instruct 4 BF16 Yes
llama-3.3-70b-instruct 8 BF16 Yes
llama-3.3-70b-instruct 1 FP8 No
llama-3.3-70b-instruct 2 FP8 No
llama-3.3-70b-instruct 4 FP8 No
llama-3.3-70b-instruct 8 FP8 No
llama-3.3-70b-instruct 1 FP8 Yes
llama-3.3-70b-instruct 2 FP8 Yes
llama-3.3-70b-instruct 4 FP8 Yes
llama-3.3-70b-instruct 8 FP8 Yes
llama-3.3-70b-instruct 1 NVFP4 No
llama-3.3-70b-instruct 2 NVFP4 No
llama-3.3-70b-instruct 4 NVFP4 No
llama-3.3-70b-instruct 8 NVFP4 No
llama-3.3-70b-instruct 1 NVFP4 Yes
llama-3.3-70b-instruct 2 NVFP4 Yes
llama-3.3-70b-instruct 4 NVFP4 Yes
llama-3.3-70b-instruct 8 NVFP4 Yes
llama-3.3-nemotron-super-49b-v1.5 1 BF16 No
llama-3.3-nemotron-super-49b-v1.5 2 BF16 No
llama-3.3-nemotron-super-49b-v1.5 4 BF16 No
llama-3.3-nemotron-super-49b-v1.5 8 BF16 No
llama-3.3-nemotron-super-49b-v1.5 1 BF16 Yes
llama-3.3-nemotron-super-49b-v1.5 2 BF16 Yes
llama-3.3-nemotron-super-49b-v1.5 4 BF16 Yes
llama-3.3-nemotron-super-49b-v1.5 8 BF16 Yes
llama-3.3-nemotron-super-49b-v1.5 1 FP8 No
llama-3.3-nemotron-super-49b-v1.5 2 FP8 No
llama-3.3-nemotron-super-49b-v1.5 4 FP8 No
llama-3.3-nemotron-super-49b-v1.5 8 FP8 No
llama-3.3-nemotron-super-49b-v1.5 1 FP8 Yes
llama-3.3-nemotron-super-49b-v1.5 2 FP8 Yes
llama-3.3-nemotron-super-49b-v1.5 4 FP8 Yes
llama-3.3-nemotron-super-49b-v1.5 8 FP8 Yes
llama-3.3-nemotron-super-49b-v1.5 1 NVFP4 No
llama-3.3-nemotron-super-49b-v1.5 2 NVFP4 No
llama-3.3-nemotron-super-49b-v1.5 4 NVFP4 No
llama-3.3-nemotron-super-49b-v1.5 8 NVFP4 No
llama-3.3-nemotron-super-49b-v1.5 1 NVFP4 Yes
llama-3.3-nemotron-super-49b-v1.5 2 NVFP4 Yes
llama-3.3-nemotron-super-49b-v1.5 4 NVFP4 Yes
llama-3.3-nemotron-super-49b-v1.5 8 NVFP4 Yes
Muse Spark Open 8 NVFP4 No
mistral-small-4-119b-2603 1 FP8 No
mistral-small-4-119b-2603 2 FP8 No
mistral-small-4-119b-2603 4 FP8 No
mistral-small-4-119b-2603 8 FP8 No
mistral-small-4-119b-2603 1 NVFP4 No
mistral-small-4-119b-2603 2 NVFP4 No
mistral-small-4-119b-2603 4 NVFP4 No
mistral-small-4-119b-2603 8 NVFP4 No
nemotron-content-safety-reasoning-4b-experimental 1 BF16 No
nemotron-content-safety-reasoning-4b-experimental 2 BF16 No
nemotron-content-safety-reasoning-4b-experimental 4 BF16 No
nemotron-content-safety-reasoning-4b-experimental 8 BF16 No
nemotron-3-nano 1 BF16 No
nemotron-3-nano 2 BF16 No
nemotron-3-nano 4 BF16 No
nemotron-3-nano 8 BF16 No
nemotron-3-nano 1 BF16 Yes
nemotron-3-nano 2 BF16 Yes
nemotron-3-nano 4 BF16 Yes
nemotron-3-nano 8 BF16 Yes
nemotron-3-nano 1 FP8 No
nemotron-3-nano 2 FP8 No
nemotron-3-nano 4 FP8 No
nemotron-3-nano 8 FP8 No
nemotron-3-nano 1 NVFP4 No
nemotron-3-nano 2 NVFP4 No
nemotron-3-nano 4 NVFP4 No
nemotron-3-nano 8 NVFP4 No
nemotron-3-nano 1 NVFP4 Yes
nemotron-3-nano-omni-30b-a3b-reasoning 2 BF16 No
nemotron-3-nano-omni-30b-a3b-reasoning 1 FP8 No
nemotron-3-nano-omni-30b-a3b-reasoning 1 NVFP4 No
nemotron-3-super-120b-a12b 1 BF16 No
nemotron-3-super-120b-a12b 2 BF16 No
nemotron-3-super-120b-a12b 4 BF16 No
nemotron-3-super-120b-a12b 8 BF16 No
nemotron-3-super-120b-a12b 2 BF16 Yes
nemotron-3-super-120b-a12b 4 BF16 Yes
nemotron-3-super-120b-a12b 8 BF16 Yes
nemotron-3-super-120b-a12b 1 FP8 No
nemotron-3-super-120b-a12b 2 FP8 No
nemotron-3-super-120b-a12b 4 FP8 No
nemotron-3-super-120b-a12b 8 FP8 No
nemotron-3-super-120b-a12b 1 NVFP4 No
nemotron-3-super-120b-a12b 2 NVFP4 No
nemotron-3-super-120b-a12b 4 NVFP4 No
nemotron-3-super-120b-a12b 8 NVFP4 No
nemotron-3-super-120b-a12b 1 NVFP4 Yes
nemotron-3-super-120b-a12b 2 NVFP4 Yes
nemotron-3-ultra-550b-a55b 8 BF16 No
nemotron-3-ultra-550b-a55b 2 NVFP4 No
nemotron-3-ultra-550b-a55b 4 NVFP4 No
nemotron-3-ultra-550b-a55b 8 NVFP4 No
nemotron-3.5-lightning 1 BF16 No
nemotron-3.5-lightning 1 BF16 Yes
nemotron-3.5-lightning 2 BF16 No
nemotron-3.5-lightning 2 BF16 Yes
nemotron-3.5-lightning 4 BF16 No
nemotron-3.5-lightning 4 BF16 Yes
nemotron-3.5-lightning 8 BF16 No
nemotron-3.5-lightning 8 BF16 Yes
nemotron-3.5-lightning 1 W4A16 No
nemotron-3.5-lightning 1 W4A16 Yes
nemotron-3.5-lightning 2 W4A16 No
nemotron-3.5-lightning 2 W4A16 Yes
nemotron-3.5-lightning 4 W4A16 No
nemotron-3.5-lightning 4 W4A16 Yes
nemotron-3.5-lightning 1 NVFP4 No
nemotron-3.5-lightning 1 NVFP4 Yes
nemotron-3.5-lightning 2 NVFP4 No
nemotron-3.5-lightning 2 NVFP4 Yes
nemotron-3.5-lightning 4 NVFP4 No
nemotron-3.5-lightning 4 NVFP4 Yes
nemotron-3.5-super-vl-120b-a12b 2 BF16 Yes
nemotron-3.5-super-vl-120b-a12b 4 BF16 Yes
nemotron-3.5-super-vl-120b-a12b 8 BF16 Yes
nemotron-3.5-super-vl-120b-a12b 1 NVFP4 No
nemotron-3.5-super-vl-120b-a12b 2 NVFP4 No
qwen3.5-122b-a10b 1 BF16 No
qwen3.5-122b-a10b 1 BF16 Yes
qwen3.5-122b-a10b 2 BF16 No
qwen3.5-122b-a10b 2 BF16 Yes
qwen3.5-122b-a10b 4 BF16 No
qwen3.5-122b-a10b 4 BF16 Yes
qwen3.5-122b-a10b 8 BF16 No
qwen3.5-122b-a10b 8 BF16 Yes
qwen3.5-122b-a10b 1 FP8 No
qwen3.5-122b-a10b 1 FP8 Yes
qwen3.5-122b-a10b 2 FP8 No
qwen3.5-122b-a10b 2 FP8 Yes
qwen3.5-122b-a10b 4 FP8 No
qwen3.5-122b-a10b 4 FP8 Yes
qwen3.5-122b-a10b 8 FP8 No
qwen3.5-122b-a10b 8 FP8 Yes
qwen3.5-122b-a10b 1 NVFP4 No
qwen3.5-122b-a10b 1 NVFP4 Yes
qwen3.5-122b-a10b 2 NVFP4 No
qwen3.5-122b-a10b 2 NVFP4 Yes
qwen3.5-122b-a10b 4 NVFP4 No
qwen3.5-122b-a10b 4 NVFP4 Yes
qwen3.5-122b-a10b 8 NVFP4 No
qwen3.5-122b-a10b 8 NVFP4 Yes
qwen3.5-397b-a17b 4 BF16 No
qwen3.5-397b-a17b 4 BF16 Yes
qwen3.5-397b-a17b 8 BF16 No
qwen3.5-397b-a17b 8 BF16 Yes
qwen3.5-397b-a17b 2 FP8 No
qwen3.5-397b-a17b 2 FP8 Yes
qwen3.5-397b-a17b 4 FP8 No
qwen3.5-397b-a17b 4 FP8 Yes
qwen3.5-397b-a17b 8 FP8 No
qwen3.5-397b-a17b 8 FP8 Yes
qwen3.5-397b-a17b 1 NVFP4 No
qwen3.5-397b-a17b 1 NVFP4 Yes
qwen3.5-397b-a17b 2 NVFP4 No
qwen3.5-397b-a17b 2 NVFP4 Yes
qwen3.5-397b-a17b 4 NVFP4 No
qwen3.5-397b-a17b 4 NVFP4 Yes
qwen3.5-397b-a17b 8 NVFP4 No
qwen3.5-397b-a17b 8 NVFP4 Yes
riva-translate-4b-instruct-v2 1 BF16 No
starcoder2-7b 1 BF16 No
starcoder2-7b 2 BF16 No
Step 5 Preview 8 FP8 No
Step 5 Preview 4 NVFP4 No

LLMs#

The following section lists all of the supported LLMs.

deepseek-v4-pro-0813#

The following table lists the supported profile configurations for deepseek-ai/DeepSeek-V4-Pro-0813.

This NIM is an SGLang 2.1.2-variant container. Combined container and model disk space is approximately 831 GB. The fallback profile requires more than 1,024 GB of aggregate GPU memory across eight GPUs. For more information, refer to the dedicated Get Started guide.

GPU

GPU Memory

Precision

TP

PP

LoRA

Profile

Any (fallback)

> 1,024 GB aggregate

FP8

8

1

No

sglang-fp8-tp8-fallback-1024g

H20-3e

141 GB

FP8

8

1

No

sglang-h20-3e-fp8-tp8-throughput-232c:10de-1024g

H200

141 GB

FP8

8

1

No

sglang-h200-fp8-tp8-throughput-2335:10de-1024g

B200

196 GB

FP8

8

1

No

sglang-b200-fp8-tp8-throughput-2901:10de-1024g

Verified GPUs

This model has been verified on the following GPUs:

  • NVIDIA-B200

  • NVIDIA-H20-3e

  • NVIDIA-H200

GLM-5.2#

The following table lists the supported profile configurations for zai-org/glm-5.2. This model-specific container uses SGLang as the inference backend.

GLM-5.2 Generic Profiles#

Profile

GPUs

Precision

TP

PP

LoRA

Generic FP8 TP4

B300, GB300

FP8

4

1

No

Generic FP8 TP4 + LoRA

B300, GB300

FP8

4

1

Yes

Generic FP8 TP8

B200, B300, H200

FP8

8

1

No

Generic FP8 TP8 + LoRA

B200, B300, H200

FP8

8

1

Yes

Generic NVFP4 TP4

B200, B300, GB200, GB300

NVFP4

4

1

No

Generic NVFP4 TP4 + LoRA

B200, B300, GB200, GB300

NVFP4

4

1

Yes

Generic NVFP4 TP8

B200, B300

NVFP4

8

1

No

Generic NVFP4 TP8 + LoRA

B200, B300

NVFP4

8

1

Yes

Optimized deployment profiles#

Precision

TP

GPU

LoRA

Tuning Objective

Profile ID

NVFP4

4

NVIDIA-B200

No

Throughput

aeb98e3f5578a620b09cadb856f7dc017ca4e7704caeb7c4e18c046e151c561b

FP8

8

NVIDIA-H200

No

Throughput

04fe52ce04491615d3dacb0aa5b5a2bfe0a494143a9997bc56ba487e5accdd37

For deployment and runtime settings, refer to Get Started with GLM-5.2.

Verified GPUs

The release validation covers the following GPUs. For model-specific limitations, refer to the deployment guide:

  • NVIDIA-B200

  • NVIDIA-B300-SXM6-AC

  • NVIDIA-GB200

  • NVIDIA-GB300

  • NVIDIA-H200

gpt-oss-120b#

The following table lists the supported profile configurations for openai/gpt-oss-120b:

Precision

TP1

TP2

TP4

TP8

MXFP4

vllm-mxfp4-tp1-pp1

vllm-mxfp4-tp2-pp1

vllm-mxfp4-tp4-pp1

vllm-mxfp4-tp8-pp1

MXFP4 + LoRA

vllm-mxfp4-tp1-pp1-lora

vllm-mxfp4-tp2-pp1-lora

vllm-mxfp4-tp4-pp1-lora

vllm-mxfp4-tp8-pp1-lora

Verified GPUs

This model has been verified on the following GPUs:

  • NVIDIA-A100-SXM4-40GB

  • NVIDIA-A100-SXM4-80GB

  • NVIDIA-A10G

  • NVIDIA-B200

  • NVIDIA-B300-SXM6-AC

  • NVIDIA-GB200

  • NVIDIA-GB300

  • NVIDIA-GB300-WS

  • NVIDIA-GH200-144G-HBM3e

  • NVIDIA-GH200-480GB

  • NVIDIA-H100-80GB-HBM3

  • NVIDIA-H100-NVL

  • NVIDIA-H200

  • NVIDIA-H200-NVL

  • NVIDIA-L40S

  • NVIDIA-RTX-PRO-4500-Blackwell-Server-Edition

  • NVIDIA-RTX-PRO-6000-Blackwell-Server-Edition

Optimized deployment profiles#

Precision

TP

GPU

LoRA

Tuning Objective

Profile ID

MXFP4

1

NVIDIA-B200, NVIDIA-B300-SXM6-AC, NVIDIA-GB200, NVIDIA-GB300, NVIDIA-GB300-WS, NVIDIA-GH200-144G-HBM3e, NVIDIA-GH200-480GB, NVIDIA-H100-80GB-HBM3, NVIDIA-H100-NVL, NVIDIA-H200, NVIDIA-H200-NVL, NVIDIA-RTX-PRO-6000-Blackwell-Server-Edition

No

Throughput

7de9349cde0a225c2a05f727fbabb8192c067a6d3b1440051730dd9f5c078388

gpt-oss-20b#

The following table lists the supported profile configurations for openai/gpt-oss-20b:

Precision

TP1

TP2

TP4

TP8

MXFP4

vllm-mxfp4-tp1-pp1

vllm-mxfp4-tp2-pp1

vllm-mxfp4-tp4-pp1

vllm-mxfp4-tp8-pp1

MXFP4 + LoRA

vllm-mxfp4-tp1-pp1-lora

vllm-mxfp4-tp2-pp1-lora

vllm-mxfp4-tp4-pp1-lora

vllm-mxfp4-tp8-pp1-lora

Verified GPUs

This model has been verified on the following GPUs:

  • NVIDIA-A100-SXM4-40GB

  • NVIDIA-A100-SXM4-80GB

  • NVIDIA-A10G

  • NVIDIA-B200

  • NVIDIA-B300-SXM6-AC

  • NVIDIA-GB10

  • NVIDIA-GB200

  • NVIDIA-GB300

  • NVIDIA-GB300-WS

  • NVIDIA-GH200-144G-HBM3e

  • NVIDIA-GH200-480GB

  • NVIDIA-H100-80GB-HBM3

  • NVIDIA-H100-NVL

  • NVIDIA-H200

  • NVIDIA-H200-NVL

  • NVIDIA-L40S

  • NVIDIA-RTX-PRO-4500-Blackwell-Server-Edition

  • NVIDIA-RTX-PRO-6000-Blackwell-Server-Edition

llama-3.1-70b-instruct#

The following table lists the supported profile configurations for meta/llama-3.1-70b-instruct:

Precision

TP1

TP2

TP4

TP8

BF16

vllm-bf16-tp1-pp1

vllm-bf16-tp2-pp1

vllm-bf16-tp4-pp1

vllm-bf16-tp8-pp1

BF16 + LoRA

vllm-bf16-tp1-pp1-lora

vllm-bf16-tp2-pp1-lora

vllm-bf16-tp4-pp1-lora

vllm-bf16-tp8-pp1-lora

FP8

vllm-fp8-tp1-pp1

vllm-fp8-tp2-pp1

vllm-fp8-tp4-pp1

vllm-fp8-tp8-pp1

FP8 + LoRA

vllm-fp8-tp1-pp1-lora

vllm-fp8-tp2-pp1-lora

vllm-fp8-tp4-pp1-lora

vllm-fp8-tp8-pp1-lora

NVFP4

vllm-nvfp4-tp1-pp1

vllm-nvfp4-tp2-pp1

vllm-nvfp4-tp4-pp1

vllm-nvfp4-tp8-pp1

NVFP4 + LoRA

vllm-nvfp4-tp1-pp1-lora

vllm-nvfp4-tp2-pp1-lora

vllm-nvfp4-tp4-pp1-lora

vllm-nvfp4-tp8-pp1-lora

Optimized deployment profiles#

Precision

TP

GPU

LoRA

Tuning Objective

Profile ID

NVFP4

2

NVIDIA-B200

No

Throughput

e1d3ae3b1367b98b9a7a729f3087d8e411e709f436c624129b939f96dcfd02b7

FP8

2

NVIDIA-H200

No

Throughput

421adb20111d74d64424eab4d8e9ed1a4cbd756338f03c52a32c4ee955301044

Verified GPUs

This model has been verified on the following GPUs:

  • NVIDIA-A100-SXM4-40GB

  • NVIDIA-A100-SXM4-80GB

  • NVIDIA-B200

  • NVIDIA-B300-SXM6-AC

  • NVIDIA-GB200

  • NVIDIA-GB300

  • NVIDIA-GB300-WS

  • NVIDIA-GH200-144G-HBM3e

  • NVIDIA-H100-80GB-HBM3

  • NVIDIA-H100-NVL

  • NVIDIA-H200

  • NVIDIA-H200-NVL

  • NVIDIA-L40S

  • NVIDIA-RTX-PRO-4500-Blackwell-Server-Edition

  • NVIDIA-RTX-PRO-6000-Blackwell-Server-Edition

llama-3.1-8b-instruct#

The following table lists the supported profile configurations for meta/llama-3.1-8b-instruct:

Precision

TP1

BF16

vllm-bf16-tp1-pp1

BF16 + LoRA

vllm-bf16-tp1-pp1-lora

FP8

vllm-fp8-tp1-pp1

FP8 + LoRA

vllm-fp8-tp1-pp1-lora

NVFP4

vllm-nvfp4-tp1-pp1

NVFP4 + LoRA

vllm-nvfp4-tp1-pp1-lora

Optimized deployment profiles#

Precision

TP

GPU

LoRA

Tuning Objective

Profile ID

NVFP4

1

NVIDIA-B200

No

Throughput

381790f97c25437119051f987f5b172b00e6e5281545c76e5318caa9880ecd15

FP8

1

NVIDIA-H200

No

Throughput

86e27d488d1a6a6b7fa5fb2a6894bafc7059376021286fab43fcd9c612abd7f4

Verified GPUs

This model has been verified on the following GPUs:

  • NVIDIA-A100-SXM4-40GB

  • NVIDIA-A100-SXM4-80GB

  • NVIDIA-B200

  • NVIDIA-B300-SXM6-AC

  • NVIDIA-GB10

  • NVIDIA-GB200

  • NVIDIA-GB300

  • NVIDIA-GB300-WS

  • NVIDIA-GH200-144G-HBM3e

  • NVIDIA-GH200-480GB

  • NVIDIA-H100-80GB-HBM3

  • NVIDIA-H100-NVL

  • NVIDIA-H200

  • NVIDIA-H200-NVL

  • NVIDIA-L40S

  • NVIDIA-RTX-PRO-4500-Blackwell-Server-Edition

  • NVIDIA-RTX-PRO-6000-Blackwell-Server-Edition

llama-3.3-70b-instruct#

The following table lists the supported profile configurations for meta/llama-3.3-70b-instruct:

Precision

TP1

TP2

TP4

TP8

BF16

vllm-bf16-tp1-pp1

vllm-bf16-tp2-pp1

vllm-bf16-tp4-pp1

vllm-bf16-tp8-pp1

BF16 + LoRA

vllm-bf16-tp1-pp1-lora

vllm-bf16-tp2-pp1-lora

vllm-bf16-tp4-pp1-lora

vllm-bf16-tp8-pp1-lora

FP8

vllm-fp8-tp1-pp1

vllm-fp8-tp2-pp1

vllm-fp8-tp4-pp1

vllm-fp8-tp8-pp1

FP8 + LoRA

vllm-fp8-tp1-pp1-lora

vllm-fp8-tp2-pp1-lora

vllm-fp8-tp4-pp1-lora

vllm-fp8-tp8-pp1-lora

NVFP4

vllm-nvfp4-tp1-pp1

vllm-nvfp4-tp2-pp1

vllm-nvfp4-tp4-pp1

vllm-nvfp4-tp8-pp1

NVFP4 + LoRA

vllm-nvfp4-tp1-pp1-lora

vllm-nvfp4-tp2-pp1-lora

vllm-nvfp4-tp4-pp1-lora

vllm-nvfp4-tp8-pp1-lora

Optimized deployment profiles#

Precision

TP

GPU

LoRA

Tuning Objective

Profile ID

NVFP4

1

NVIDIA-B200

No

Throughput

16d3616335859278752b8ec6bdd3d88657880120895118fcd584c75f6f3249e0

FP8

2

NVIDIA-H200

No

Throughput

f8197e13fcb703fc7df8a9060d3d2be9745b4ff98b0eac3e466779afc127f2a6

Verified GPUs

This model has been verified on the following GPUs:

  • NVIDIA-A100-SXM4-40GB

  • NVIDIA-A100-SXM4-80GB

  • NVIDIA-B200

  • NVIDIA-B300-SXM6-AC

  • NVIDIA-GB200

  • NVIDIA-GB300

  • NVIDIA-GB300-WS

  • NVIDIA-GH200-144G-HBM3e

  • NVIDIA-H100-80GB-HBM3

  • NVIDIA-H100-NVL

  • NVIDIA-H200

  • NVIDIA-H200-NVL

  • NVIDIA-L40S

  • NVIDIA-RTX-PRO-4500-Blackwell-Server-Edition

  • NVIDIA-RTX-PRO-6000-Blackwell-Server-Edition

llama-3.3-nemotron-super-49b-v1.5#

The following table lists the supported profile configurations for nvidia/llama-3.3-nemotron-super-49b-v1.5:

Precision

TP1

TP2

TP4

TP8

BF16

vllm-bf16-tp1-pp1

vllm-bf16-tp2-pp1

vllm-bf16-tp4-pp1

vllm-bf16-tp8-pp1

BF16 + LoRA

vllm-bf16-tp1-pp1-lora

vllm-bf16-tp2-pp1-lora

vllm-bf16-tp4-pp1-lora

vllm-bf16-tp8-pp1-lora

FP8

vllm-fp8-tp1-pp1

vllm-fp8-tp2-pp1

vllm-fp8-tp4-pp1

vllm-fp8-tp8-pp1

FP8 + LoRA

vllm-fp8-tp1-pp1-lora

vllm-fp8-tp2-pp1-lora

vllm-fp8-tp4-pp1-lora

vllm-fp8-tp8-pp1-lora

NVFP4

vllm-nvfp4-tp1-pp1

vllm-nvfp4-tp2-pp1

vllm-nvfp4-tp4-pp1

vllm-nvfp4-tp8-pp1

NVFP4 + LoRA

vllm-nvfp4-tp1-pp1-lora

vllm-nvfp4-tp2-pp1-lora

vllm-nvfp4-tp4-pp1-lora

vllm-nvfp4-tp8-pp1-lora

Optimized deployment profiles#

Precision

TP

GPU

LoRA

Tuning Objective

Profile ID

NVFP4

1

NVIDIA-B200

No

Throughput

d6eb0398f20918bf0f1ee7703db82d00c5af0580f21facc8ad2a40d1c82c0ba7

FP8

2

NVIDIA-H200

No

Throughput

9923047c5f78eca58a4a1d586e85cfaaa4c724d5679bb615a9e954b4530033c3

Verified GPUs

This model has been verified on the following GPUs:

  • NVIDIA-A100-SXM4-40GB

  • NVIDIA-A100-SXM4-80GB

  • NVIDIA-A10G

  • NVIDIA-B200

  • NVIDIA-B300-SXM6-AC

  • NVIDIA-GB10

  • NVIDIA-GB200

  • NVIDIA-GB300

  • NVIDIA-GB300-WS

  • NVIDIA-GH200-144G-HBM3e

  • NVIDIA-GH200-480GB

  • NVIDIA-H100-80GB-HBM3

  • NVIDIA-H100-NVL

  • NVIDIA-H200

  • NVIDIA-H200-NVL

  • NVIDIA-L40S

  • NVIDIA-RTX-PRO-4500-Blackwell-Server-Edition

  • NVIDIA-RTX-PRO-6000-Blackwell-Server-Edition

nemotron-content-safety-reasoning-4b-experimental#

The following table lists the supported profile configurations for nvidia/nemotron-content-safety-reasoning-4b-experimental. The latest release version of this NIM is 2.0.3. For more information, refer to the dedicated Get Started guide.

Precision

TP1

TP2

TP4

TP8

BF16

vllm-bf16-tp1-pp1

vllm-bf16-tp2-pp1

vllm-bf16-tp4-pp1

vllm-bf16-tp8-pp1

Verified GPUs

This model has been verified on the following GPUs:

  • NVIDIA-B200

  • NVIDIA-GB200

  • NVIDIA-H100-80GB-HBM3

  • NVIDIA-H200

nemotron-3-nano#

The following table lists the supported profile configurations for nvidia/nemotron-3-nano:

Precision

TP1

TP2

TP4

TP8

BF16

vllm-bf16-tp1-pp1

vllm-bf16-tp2-pp1

vllm-bf16-tp4-pp1

vllm-bf16-tp8-pp1

BF16 + LoRA

vllm-bf16-tp1-pp1-lora

vllm-bf16-tp2-pp1-lora

vllm-bf16-tp4-pp1-lora

vllm-bf16-tp8-pp1-lora

FP8

vllm-fp8-tp1-pp1

vllm-fp8-tp2-pp1

vllm-fp8-tp4-pp1

vllm-fp8-tp8-pp1

NVFP4

vllm-nvfp4-tp1-pp1

vllm-nvfp4-tp2-pp1

vllm-nvfp4-tp4-pp1

vllm-nvfp4-tp8-pp1

NVFP4 + LoRA

vllm-nvfp4-tp1-pp1-lora

–

–

–

Optimized deployment profiles#

Precision

TP

GPU

LoRA

Tuning Objective

Profile ID

NVFP4

1

NVIDIA-B200

No

Throughput

669d73c7ab89d5b53b392716cbcd8a0e06cf40d9d5051aae0b9f9a52891933a9

FP8

1

NVIDIA-H200

No

Throughput

8ca069b289f1838b0c63aeb51317f623f390975c099ba57f805c6c8940a81dad

Verified GPUs

This model has been verified on the following GPUs:

  • NVIDIA-A100-SXM4-40GB

  • NVIDIA-A100-SXM4-80GB

  • NVIDIA-A10G

  • NVIDIA-B200

  • NVIDIA-B300-SXM6-AC

  • NVIDIA-GB10

  • NVIDIA-GB200

  • NVIDIA-GB300

  • NVIDIA-GB300-WS

  • NVIDIA-GH200-144G-HBM3e

  • NVIDIA-GH200-480GB

  • NVIDIA-H100-80GB-HBM3

  • NVIDIA-H100-NVL

  • NVIDIA-H200

  • NVIDIA-H200-NVL

  • NVIDIA-L40S

  • NVIDIA-RTX-PRO-4500-Blackwell-Server-Edition

  • NVIDIA-RTX-PRO-6000-Blackwell-Server-Edition

nemotron-3-super-120b-a12b#

The following table lists the supported profile configurations for nvidia/nemotron-3-super-120b-a12b:

Precision

TP1

TP2

TP4

TP8

BF16

vllm-bf16-tp1-pp1

vllm-bf16-tp2-pp1

vllm-bf16-tp4-pp1

vllm-bf16-tp8-pp1

BF16 + LoRA

–

vllm-bf16-tp2-pp1-lora

vllm-bf16-tp4-pp1-lora

vllm-bf16-tp8-pp1-lora

FP8

vllm-fp8-tp1-pp1

vllm-fp8-tp2-pp1

vllm-fp8-tp4-pp1

vllm-fp8-tp8-pp1

NVFP4

vllm-nvfp4-tp1-pp1

vllm-nvfp4-tp2-pp1

vllm-nvfp4-tp4-pp1

vllm-nvfp4-tp8-pp1

NVFP4 + LoRA

vllm-nvfp4-tp1-pp1-lora

vllm-nvfp4-tp2-pp1-lora

–

–

Optimized deployment profiles#

Precision

TP

GPU

LoRA

Tuning Objective

Profile ID

NVFP4

4

NVIDIA-B200

No

Throughput

3c7c3d74b044f7cf4bba1a80f3f596405a17efa35c663ca52963d71aaf381ea3

FP8

4

NVIDIA-H200

No

Throughput

ca31277ebd3b387e1a597a550541ab9b8dde3b31f6b5b5695ec8799c0162573c

Note

This is a large model. Lower-TP profiles require substantially more GPU memory per device, so some verified GPUs support only TP4 or TP8 profiles.

Verified GPUs

This model has been verified on the following GPUs:

  • NVIDIA-A100-SXM4-40GB

  • NVIDIA-A100-SXM4-80GB

  • NVIDIA-B200

  • NVIDIA-B300-SXM6-AC

  • NVIDIA-GB200

  • NVIDIA-GB300

  • NVIDIA-GB300-WS

  • NVIDIA-GH200-144G-HBM3e

  • NVIDIA-H100-80GB-HBM3

  • NVIDIA-H100-NVL

  • NVIDIA-H200

  • NVIDIA-H200-NVL

  • NVIDIA-L40S

  • NVIDIA-RTX-PRO-4500-Blackwell-Server-Edition

  • NVIDIA-RTX-PRO-6000-Blackwell-Server-Edition

nemotron-3-ultra-550b-a55b#

The following table lists the supported profile configurations for nvidia/nemotron-3-ultra-550b-a55b:

Precision

TP1

TP2

TP4

TP8

BF16

–

–

–

vllm-bf16-tp8-pp1

BF16 + LoRA

–

–

–

–

NVFP4

–

vllm-nvfp4-tp2-pp1

vllm-nvfp4-tp4-pp1

vllm-nvfp4-tp8-pp1

Optimized deployment profiles#

Precision

TP

GPU

LoRA

Tuning Objective

Profile ID

NVFP4

4

NVIDIA-B200

No

Throughput

7cdfe4d76f5fd32ad9ae240eef8b5d4af81dcc0fa23c6f20c4bc7da1a389a92c

NVFP4

4

NVIDIA-B300-SXM6-AC

No

Throughput

5b3441c9d0f55e8b4442537a4294304d5401d3a5542870bda16f3f8e18971878

NVFP4

4

NVIDIA-GB200

No

Throughput

a953e969777d28fc50e322d31a1bdfaf6aeb1a3cdff63ca3760cf0d424b9a90a

NVFP4

4

NVIDIA-GB300

No

Throughput

4466f9f08faedae5362260c2552af473240428fd9bf17d7bbaaf5a535abdde1e

Note

The minimum deployment is two GPUs, using vllm-nvfp4-tp2-pp1 on high-memory Blackwell SKUs (NVIDIA-B300-SXM6-AC and NVIDIA-GB300); the other profiles use four or eight GPUs. H200 validation covers one NVFP4 TP4 profile and is less extensive than the Blackwell validation.

FP8 profiles were not released for this NIM because NVFP4 profiles work on all tested Hopper-architecture GPUs (for example, H100 and H200) and deliver better performance than FP8 profiles.

Verified GPUs

This model has been verified on the following GPUs:

  • NVIDIA-B200

  • NVIDIA-B300-SXM6-AC

  • NVIDIA-GB200

  • NVIDIA-GB300

  • NVIDIA-H100-80GB-HBM3

  • NVIDIA-H100-NVL

  • NVIDIA-H200

nemotron-3.5-lightning#

The following table lists the supported profile configurations for nvidia/nemotron-3.5-lightning, along with the minimum GPU VRAM required per device and the GPU architecture that each precision requires. The VRAM values come from the profile disk-sizing metadata reported by list-model-profiles and represent the minimum per-GPU memory needed to load the model weights and the runtime allocations; they do not include additional headroom for large KV caches at high context length or high concurrency.

Nemotron 3.5 Lightning Profile Information#

Precision

TP

PP

Min VRAM per GPU

Min GPU count

Architecture requirement

LoRA Profile

BF16

1

1

66 GB

1

Ampere or newer (SM 8.0+)

Base and LoRA

BF16

2

1

35 GB

2

Ampere or newer (SM 8.0+)

Base and LoRA

BF16

4

1

20 GB

4

Ampere or newer (SM 8.0+)

Base and LoRA

BF16

8

1

12 GB

8

Ampere or newer (SM 8.0+)

Base and LoRA

W4A16

1

1

32 GB

1

Ampere or newer (SM 8.0+)

Base and LoRA

W4A16

2

1

20 GB

2

Ampere or newer (SM 8.0+)

Base and LoRA

W4A16

4

1

14 GB

4

Ampere or newer (SM 8.0+)

Base and LoRA

NVFP4

1

1

30 GB

1

Blackwell or newer (SM 10.0+)

Base and LoRA

NVFP4

2

1

18 GB

2

Blackwell or newer (SM 10.0+)

Base and LoRA

NVFP4

4

1

12 GB

4

Blackwell or newer (SM 10.0+)

Base and LoRA

Optimized deployment profiles#

Precision

TP

GPU

LoRA

Tuning Objective

Profile ID

NVFP4

1

NVIDIA-B200

No

Throughput

4816da181f28c435aaa2efa0e2302309c28f2c08c9534070e89f67854a521d72

NVFP4 (W4A16) + FP8

1

NVIDIA-H200

No

Throughput

1e34732516864e06c8dad271edd1d19b986c4c010b9d7e3adcb8b113df58819b

Notes:

  • Min VRAM per GPU is the floor that list-model-profiles reports for that precision and tensor-parallel size. A GPU below the floor is filtered out during profile selection.

  • Architecture requirement in the generic profile table reflects the kernels those profiles need. The generic NVFP4 profiles require Streaming Multiprocessor (SM) 10.0+ (Blackwell) because they use NVFP4-native tensor cores. BF16 and W4A16 run on any Ampere-class or newer GPU (SM 8.0+).

  • Sharing weights across more GPUs (higher TP) lowers the per-GPU VRAM floor and enables lower-memory GPUs.

  • Model-cache disk sizes range from approximately 19 GB (NVFP4) to approximately 63 GB (BF16). Cache size is separate from GPU VRAM and does not indicate that the model will fit in a given GPU.

Verified GPUs

The following tables list verified GPUs for the generic profiles, grouped by architecture:

Blackwell (SM 10.0) — supports NVFP4, W4A16, BF16:

GPU

VRAM per GPU

Profiles that fit

NVIDIA-B200

180 GB

All profiles

NVIDIA-B300-SXM6-AC

288 GB

All profiles

NVIDIA-GB200

186 GB

All profiles

NVIDIA-GB300

288 GB

All profiles

NVIDIA-GB10

128 GB (unified memory)

All profiles; use --gpu-memory-utilization 0.75 on GB10 (see the getting started guide)

NVIDIA-RTX-PRO-6000-Blackwell-Server-Edition

96 GB

All profiles

Hopper (SM 9.0) — supports W4A16 and BF16 (no NVFP4):

GPU

VRAM per GPU

Profiles that fit

NVIDIA-H100-80GB-HBM3

80 GB

BF16 all sizes (needs --max-num-seqs 512 on TP=1), W4A16 all sizes

NVIDIA-H200

141 GB

BF16 all sizes, W4A16 all sizes

Ampere (SM 8.0) — supports W4A16 and BF16 (no NVFP4):

GPU

VRAM per GPU

Profiles that fit

NVIDIA-A100-SXM4-80GB

80 GB

BF16 TP≥2, W4A16 all sizes

Ada (SM 8.9) — supports W4A16 and BF16 (no NVFP4):

GPU

VRAM per GPU

Profiles that fit

NVIDIA-L40S

48 GB

BF16 TP≥2, W4A16 all sizes

Unverified but potentially compatible. Any GPU that meets the per-profile minimum VRAM floor and the architecture requirement in the profile table can run that profile technically, even if it is not in the preceding Verified GPUs tables. Only the GPUs in those tables have been tested end-to-end for this model at this NIM version; untested systems may exhibit different startup, throughput, or accuracy behavior.

riva-translate-4b-instruct-v2#

The following table lists the supported profile configuration for nvidia/riva-translate-4b-instruct-v2.

The latest release version of this NIM is 2.0.8.

Precision

TP

PP

LoRA

Profile

BF16

1

1

No

vllm-bf16-tp1-pp1

Verified GPUs

This model has been verified on the following GPUs:

  • NVIDIA-A100-SXM4-40GB

  • NVIDIA-A100-SXM4-80GB

  • NVIDIA-A10G

  • NVIDIA-B200

  • NVIDIA-B300-SXM6-AC

  • NVIDIA-GB10

  • NVIDIA-GB200

  • NVIDIA-GB300

  • NVIDIA-GB300-WS

  • NVIDIA-GH200-144G-HBM3e

  • NVIDIA-GH200-480GB

  • NVIDIA-H100-80GB-HBM3

  • NVIDIA-H100-NVL

  • NVIDIA-H200

  • NVIDIA-H200-NVL

  • NVIDIA-L40S

  • NVIDIA-RTX-PRO-4500-Blackwell-Server-Edition

  • NVIDIA-RTX-PRO-6000-Blackwell-Server-Edition

starcoder2-7b#

The following table lists the supported profile configurations for bigcode/starcoder2-7b:

Precision

TP1

TP2

BF16

vllm-bf16-tp1-pp1

vllm-bf16-tp2-pp1

Optimized deployment profiles#

Precision

TP

GPU

LoRA

Tuning Objective

Profile ID

BF16

1

NVIDIA-A100-80GB-PCIe, NVIDIA-A100-SXM4-80GB, NVIDIA-B200, NVIDIA-GB300-WS, NVIDIA-H100-80GB-HBM3, NVIDIA-H200

No

Throughput

65053e7510a772fa5ebc8da386b9b593152c746173b4c2fb85a1739a276e0d2c

BF16

2

NVIDIA-A100-SXM4-80GB, NVIDIA-B200, NVIDIA-H100-80GB-HBM3, NVIDIA-H200

No

Throughput

39725ead141c0eb26d6cbdcce32660a6da469663d88bc7ea7441808ff0134ca5

Verified GPUs

This model has been verified on the following GPUs:

  • NVIDIA-A100-80GB-PCIe

  • NVIDIA-A100-SXM4-80GB

  • NVIDIA-B200

  • NVIDIA-GB300-WS

  • NVIDIA-H100-80GB-HBM3

  • NVIDIA-H200

VLMs#

The following section lists all of the supported VLMs.

deepseek-v4.1-flash#

The following table lists the supported profile configurations for deepseek-ai/deepseek-v4.1-flash:

GPU

GPU Memory

Precision

TP

Profile

B200

196 GB

FP8

4

sglang-b200-fp8-tp4-pp1-throughput-4-174gib (default on B200)

B200

196 GB

FP8

4

sglang-b200-fp8-tp4-pp1-latency-4-178gib (DSpark speculative decoding; select explicitly)

H200

141 GB

FP8

8

sglang-h200-fp8-tp8-pp1-throughput-8-93gib

H20-3e

141 GB

FP8

8

sglang-h20-3e-fp8-tp8-pp1-throughput-8-93gib

Verified GPUs

This model has been verified on the following GPUs:

  • NVIDIA-B200

  • NVIDIA-H200

  • NVIDIA-H20-3e

gemma-4-26b-a4b-it#

The following table lists the supported profile configurations for google/gemma-4-26b-a4b-it:

Precision

TP1

TP2

TP4

TP8

NVFP4

vllm-nvfp4-tp1-pp1

–

–

–

BF16

vllm-bf16-tp1-pp1

vllm-bf16-tp2-pp1

vllm-bf16-tp4-pp1

–

Optimized deployment profiles#

Precision

TP

GPU

LoRA

Tuning Objective

Profile ID

NVFP4

1

NVIDIA-B200

No

Throughput

234e46324452f6124bf432eaa99cc1221ed917b069ee63ebc0c4d2e636f22f3d

NVFP4

1

NVIDIA-GB200

No

Throughput

04094afda5cf6fc9df04560f322cc677678a0d4caf943d058ff1c9779aaf750c

BF16

4

NVIDIA-A10G

No

Context length

800f3c24e5bdaca88aebd04242be7765c65ac7abbfc9013c404d0ea0ddc9f6fc

Verified GPUs

This model has been verified on the following GPUs:

  • NVIDIA-A100-SXM4-40GB

  • NVIDIA-A100-SXM4-80GB

  • NVIDIA-A10G

  • NVIDIA-B200

  • NVIDIA-B300-SXM6-AC

  • NVIDIA-GB200

  • NVIDIA-GB300

  • NVIDIA-GH200-144G-HBM3e

  • NVIDIA-GH200-480GB

  • NVIDIA-H100-80GB-HBM3

  • NVIDIA-H100-NVL

  • NVIDIA-H200

  • NVIDIA-H200-NVL

  • NVIDIA-L40S

  • NVIDIA-RTX-PRO-4500-Blackwell-Server-Edition

  • NVIDIA-RTX-PRO-6000-Blackwell-Server-Edition

gemma-4-31b-it#

The following table lists the supported profile configurations for google/gemma-4-31b-it:

Precision

TP1

TP2

TP4

TP8

NVFP4

vllm-nvfp4-tp1-pp1

vllm-nvfp4-tp2-pp1

vllm-nvfp4-tp4-pp1

–

BF16

vllm-bf16-tp1-pp1

vllm-bf16-tp2-pp1

vllm-bf16-tp4-pp1

–

INT4

–

vllm-int4-tp2-pp1

–

–

Optimized deployment profiles#

Precision

TP

GPU

LoRA

Tuning Objective

Profile ID

NVFP4

1

NVIDIA-B200, NVIDIA-B300-SXM6-AC, NVIDIA-GB200, NVIDIA-GB300, NVIDIA-RTX-PRO-6000-Blackwell-Server-Edition

No

Throughput

5f2613453901ce34041f9390d1eb035bceabba0af65af73be562fa5cda704a48

Verified GPUs

This model has been verified on the following GPUs:

  • NVIDIA-B200

  • NVIDIA-B300-SXM6-AC

  • NVIDIA-GB200

  • NVIDIA-GB300

  • NVIDIA-H100-80GB-HBM3

  • NVIDIA-H200

  • NVIDIA-L40S

  • NVIDIA-RTX-PRO-6000-Blackwell-Server-Edition

Inkling#

The following table lists the supported profile configurations for thinkingmachines/inkling. This model-specific container uses SGLang as the inference backend.

Inkling Generic Profiles#

Profile

GPUs

Precision

TP

PP

LoRA

Generic NVFP4 TP4

B300, GB300

NVFP4

4

1

No

Generic NVFP4 TP4 + LoRA

B300, GB300

NVFP4

4

1

Yes

Generic NVFP4 TP8

B200, B300

NVFP4

8

1

No

Generic NVFP4 TP8 + LoRA

B200, B300

NVFP4

8

1

Yes

Optimized deployment profiles#

Precision

TP

GPU

LoRA

Tuning Objective

Profile ID

NVFP4

8

NVIDIA-B200

No

Throughput

b06d3d95f3fba80b51ab008fbcce8cc8dd6f0029cf021f13ec85fe612b57db34

NVFP4

8

NVIDIA-H200

No

Throughput

4a30cb634a0f720cc28572ec4592876d255762f2bfe7bf0d2dceff6890f1f5f4

For profile selection, deployment, and API examples, refer to Get Started with Inkling.

Verified GPUs

The release validation covers the following GPUs. For model-specific limitations, refer to the deployment guide:

  • NVIDIA-B200

  • NVIDIA-B300-SXM6-AC

  • NVIDIA-GB300

  • NVIDIA-H200

kimi-k2.6#

The following table lists the supported profile configurations for moonshotai/kimi-k2.6:

Precision

TP1

TP2

TP4

TP8

NVFP4

–

–

vllm-nvfp4-tp4-pp1

–

INT4

–

–

–

vllm-int4-tp8-pp1

Optimized deployment profiles#

Precision

TP

GPU

LoRA

Tuning Objective

Profile ID

NVFP4

4

NVIDIA-B200

No

Throughput

918ca5e8acff6936981f6948e7b4d149bcd84eeb59a0eda9b70f0d18c55c8eab

NVFP4

4

NVIDIA-B300-SXM6-AC

No

Throughput

7e73d8d993b39c8bac790c0d2872658ac85d2b86227c1ae7c1393c5e63b3dab7

NVFP4

4

NVIDIA-GB200

No

Throughput

c1d37f672564cec2e2cd881a1da9e3a01ebc9134fdfb274fd09b00c93dfa770d

NVFP4

4

NVIDIA-GB300

No

Throughput

24bfc505637e8cad50ad862666b8e6769a08e9c029528fb1653711de05045987

Verified GPUs

This model has been verified on the following GPUs:

  • NVIDIA-B200

  • NVIDIA-B300-SXM6-AC

  • NVIDIA-GB200

  • NVIDIA-GB300

  • NVIDIA-H200

mistral-small-4-119b-2603#

The following table lists the supported profile configurations for mistralai/mistral-small-4-119b-2603:

Precision

TP1

TP2

TP4

TP8

FP8

vllm-fp8-tp1-pp1

vllm-fp8-tp2-pp1

vllm-fp8-tp4-pp1

vllm-fp8-tp8-pp1

NVFP4

vllm-nvfp4-tp1-pp1

vllm-nvfp4-tp2-pp1

vllm-nvfp4-tp4-pp1

vllm-nvfp4-tp8-pp1

Verified GPUs

This model has been verified on the following GPUs:

  • NVIDIA-B200

  • NVIDIA-B300-SXM6-AC

  • NVIDIA-GB200

  • NVIDIA-GB300

  • NVIDIA-H200-NVL

nemotron-3-nano-omni-30b-a3b-reasoning#

The following table lists the supported profile configurations for nvidia/nemotron-3-nano-omni-30b-a3b-reasoning:

Precision

TP1

TP2

TP4

TP8

BF16

–

vllm-bf16-tp2-pp1

–

–

FP8

vllm-fp8-tp1-pp1

–

–

–

NVFP4

vllm-nvfp4-tp1-pp1

–

–

–

Optimized deployment profiles#

Precision

TP

GPU

LoRA

Tuning Objective

Profile ID

NVFP4

1

NVIDIA-B200

No

Throughput

2883a35a92ca200be25ce99fa4f04c4807a37c3d7b01f2192e00ddd5f9d105b7

NVFP4

1

NVIDIA-B300-SXM6-AC

No

Throughput

7cb9d0358baa2792b09a9907acc032df060acd912ce9c5b99c99589e4f34182c

NVFP4

1

NVIDIA-GB300

No

Throughput

22da479a1490835d7980611878ca91da3dd8bfa8c63c72c814c8867955d9eff8

FP8

1

NVIDIA-H200

No

Throughput

babd305818fe475ed855900b0518a7ef2d6bdf3ccd651584094a706dd010822e

NVFP4

1

NVIDIA-RTX-PRO-4500-Blackwell-Server-Edition

No

Throughput

ffb4dc6902e6bb4c27948ea1700707c664b179ab214072f9f700b0fffdefa1de

NVFP4

1

NVIDIA-RTX-PRO-6000-Blackwell-Server-Edition

No

Throughput

d8e8c64a19e5150344259745878af4dc1eabc270326a8b7bb4516c90ec602e5c

Verified GPUs

This model has been verified on the following GPUs:

  • NVIDIA-A100-SXM4-80GB

  • NVIDIA-B200

  • NVIDIA-B300-SXM6-AC

  • NVIDIA-GB200

  • NVIDIA-GB300

  • NVIDIA-GH200-144G-HBM3e

  • NVIDIA-GH200-480GB

  • NVIDIA-H100-80GB-HBM3

  • NVIDIA-H200

  • NVIDIA-H200-NVL

  • NVIDIA-L40S

  • NVIDIA-RTX-PRO-4500-Blackwell-Server-Edition

  • NVIDIA-RTX-PRO-6000-Blackwell-Server-Edition

nemotron-3.5-super-vl-120b-a12b#

The following table lists the supported GPU configurations for nvidia/nemotron-3.5-super-vl-120b-a12b. For more information, refer to the dedicated Get Started guide.

Optimized deployment profiles#

Precision

TP

GPU

LoRA

Tuning Objective

Profile Name

Profile ID

Min VRAM per GPU

BF16

2

NVIDIA-B200

Yes

General

vllm-nvidia-b200-bf16-tp2-pp1-feat_lora-throughput-0.0

f601f9963fcc4bed0469b15df33ec438c2cf55181fb1febdae30fab436369134

119 GB

BF16

2

NVIDIA-B300-SXM6-AC

Yes

General

vllm-nvidia-b300-sxm6-ac-bf16-tp2-pp1-feat_lora-throughput-0.0

12e76812d9e15427ce3a63523778b7bd15bec8dcaefe9817c32fc19a105fffae

119 GB

BF16

2

NVIDIA-GB200

Yes

General

vllm-nvidia-gb200-bf16-tp2-pp1-feat_lora-throughput-0.0

f3dcd36059120231b6fb8a00682af9a36f220dc981ca73a8573bf5aeb5faf85c

119 GB

BF16

2

NVIDIA-GB300

Yes

General

vllm-nvidia-gb300-bf16-tp2-pp1-feat_lora-throughput-0.0

7535e487104e591561705169ece00abad91b1c26255aa844c6456a5b18fc3413

119 GB

BF16

4

NVIDIA-H200

Yes

General

vllm-nvidia-h200-bf16-tp4-pp1-feat_lora-throughput-0.0

29224c376e8bd48f546d1f783a6da93662401524a972c881d3355a00a399da37

61 GB

BF16

8

NVIDIA-H100-80GB-HBM3

Yes

General

vllm-nvidia-h100-80gb-hbm3-bf16-tp8-pp1-feat_lora-throughput-0.0

7f85f2d15f243fa05be912e610b31b19a376f8509dd12eb8a3bc3b367e8b6a9f

32 GB

NVFP4

1

NVIDIA-B200

No

agentX

vllm-nvidia-b200-nvfp4-tp1-pp1-throughput-0.0-agentx

35320c4542dbf3e4ad098dd7669cb3330f728f86a62bdf91eeb4bd34768346f1

72 GB

NVFP4

1

NVIDIA-B300-SXM6-AC

No

agentX

vllm-nvidia-b300-sxm6-ac-nvfp4-tp1-pp1-throughput-0.0-agentx

cc452f7f9da6c7f902014d7e79324a48f8202c12648d42bced69bea9217a81d5

72 GB

NVFP4

1

NVIDIA-GB10

No

General

vllm-nvidia-gb10-nvfp4-tp1-pp1-throughput-0.0

6eb52c8e38c174694d551f314ed8b2021a4b3f767d3616da1754dde0916b855c

72 GB

NVFP4

1

NVIDIA-GB200

No

50k/2k/1k

vllm-nvidia-gb200-nvfp4-tp1-pp1-throughput-0.0-50k-2k-1k

8c840bb323e03589e50b8c97090e059017a35546bbf3b11caec5a1dc5dd334de

72 GB

NVFP4

1

NVIDIA-GB200

No

agentX

vllm-nvidia-gb200-nvfp4-tp1-pp1-throughput-0.0-agentx

d4a164dc936edc781bf577e419a442e4b1712c2d3a17a3ea33b080b162018b2a

72 GB

NVFP4

1

NVIDIA-GB300

No

agentX

vllm-nvidia-gb300-nvfp4-tp1-pp1-throughput-0.0-agentx

26d33ae17bd36e569c9348703f9950a37ffd050b01656292540e2daa35e801b9

72 GB

NVFP4

1

NVIDIA-GB300-WS

No

General

vllm-nvidia-gb300-ws-nvfp4-tp1-pp1-throughput-0.0

5f9194217b5b7856869ae716b6802ec506b6cff1f109a7f1c5e828cb8f105c59

72 GB

NVFP4

1

NVIDIA-H200

No

50k/2k/1k

vllm-nvidia-h200-nvfp4-emulated-tp1-pp1-throughput-0.0-50k-2k-1k

f335b46bb53b1d7a32ed08d093fc93a26145348530dd7aa3b63d578bfd4cfdf9

72 GB

NVFP4

1

NVIDIA-H200

No

agentX

vllm-nvidia-h200-nvfp4-emulated-tp1-pp1-throughput-0.0-agentx

65ec457ca53207b0fa52748a8eeb5e47f58b504f7a005f9ce8f815fec94a8f81

72 GB

NVFP4

2

NVIDIA-A100-SXM4-80GB

No

General

vllm-nvidia-a100-sxm4-80gb-nvfp4-emulated-tp2-pp1-throughput-0.0

737c816b8ac7e1c3c51c871768875600635c9a7939e3e81ed48b6a0a17df29f3

38 GB

NVFP4

2

NVIDIA-H100-80GB-HBM3

No

50k/2k/1k

vllm-nvidia-h100-80gb-hbm3-nvfp4-emulated-tp2-pp1-throughput-0.0-50k-2k-1k

052ee51a97716b2cf070fe268ca0943947e8ff814d331ba52f08d6d8a1e8a20d

38 GB

NVFP4

2

NVIDIA-H100-80GB-HBM3

No

agentX

vllm-nvidia-h100-80gb-hbm3-nvfp4-emulated-tp2-pp1-throughput-0.0-agentx

ab990383587f1f1f9fb3b592895529bc37c34a5a3d6aaf055f07eebe0c865631

38 GB

NVFP4

2

NVIDIA-RTX-PRO-6000-Blackwell-Server-Edition

No

General

vllm-nvidia-rtx-pro-6000-blackwell-server-edition-nvfp4-tp2-pp1-throughput-0.0

da5a5f56a2faa2c54d854046c1374f913bdf829cca30b6406c433f98d472eb21

38 GB

Min VRAM per GPU is the minimum per-GPU memory needed to load the model weights and the runtime allocations at that precision and tensor-parallel size, as computed by NIM LLM and VLM at startup. It does not include headroom for the KV cache at long context length or high concurrency. Each profile is built for the listed GPU model and GPU count, and there is no generic fallback profile. The min_vram_per_device_gb value that list-model-profiles reports for these profiles is not a memory requirement.

Profile Name is the name that list-model-profiles prints next to the profile ID. NIM_MODEL_PROFILE accepts either the name or the ID.

Tuning objectives:

  • 50k/2k/1k: multi-turn conversations that share a 50K-token system prompt, with about 2K new input tokens and 1K output tokens per turn

  • agentX: multi-turn coding-agent sessions with tool calls and long, growing contexts of up to 1M tokens

  • General: the only profile for that GPU and precision

Verified GPUs

This model supports the following GPUs at the precisions and GPU counts listed in the profile table:

  • NVIDIA-A100-SXM4-80GB

  • NVIDIA-B200

  • NVIDIA-B300-SXM6-AC

  • NVIDIA-GB10

  • NVIDIA-GB200

  • NVIDIA-GB300

  • NVIDIA-GB300-WS

  • NVIDIA-H100-80GB-HBM3

  • NVIDIA-H200

  • NVIDIA-RTX-PRO-6000-Blackwell-Server-Edition

qwen3.5-122b-a10b#

The following table lists the supported profile configurations for qwen/qwen3.5-122b-a10b:

Precision

TP1

TP2

TP4

TP8

BF16

vllm-bf16-tp1-pp1

vllm-bf16-tp2-pp1

vllm-bf16-tp4-pp1

vllm-bf16-tp8-pp1

BF16 + LoRA

vllm-bf16-tp1-pp1-lora

vllm-bf16-tp2-pp1-lora

vllm-bf16-tp4-pp1-lora

vllm-bf16-tp8-pp1-lora

FP8

vllm-fp8-tp1-pp1

vllm-fp8-tp2-pp1

vllm-fp8-tp4-pp1

vllm-fp8-tp8-pp1

FP8 + LoRA

vllm-fp8-tp1-pp1-lora

vllm-fp8-tp2-pp1-lora

vllm-fp8-tp4-pp1-lora

vllm-fp8-tp8-pp1-lora

NVFP4

vllm-nvfp4-tp1-pp1

vllm-nvfp4-tp2-pp1

vllm-nvfp4-tp4-pp1

vllm-nvfp4-tp8-pp1

NVFP4 + LoRA

vllm-nvfp4-tp1-pp1-lora

vllm-nvfp4-tp2-pp1-lora

vllm-nvfp4-tp4-pp1-lora

vllm-nvfp4-tp8-pp1-lora

Optimized deployment profiles#

Precision

TP

GPU

LoRA

Tuning Objective

Profile ID

NVFP4

1

NVIDIA-B200

No

Throughput

5c60139ad068156cef737d438cd5244742d58c2c0cb312ec61ae8dc0c463d5eb

FP8

2

NVIDIA-H200

No

Latency

c475df816df58cb8d318ea34b7bb05dff51cd2b437c2924eadf10289ab2533c8

FP8

2

NVIDIA-H200

No

Throughput

cbb3cc0300472553b8ae84bfc03f7b1f33e93ac3a9fe2dcb85783ab1981e32b1

Verified GPUs

This model has been verified on the following GPUs:

  • NVIDIA-A100-SXM4-80GB

  • NVIDIA-B200

  • NVIDIA-B300-SXM6-AC

  • NVIDIA-GB200

  • NVIDIA-GB300

  • NVIDIA-GH200-144G-HBM3e

  • NVIDIA-H100-80GB-HBM3

  • NVIDIA-H200

  • NVIDIA-H200-NVL

  • NVIDIA-L40S

  • NVIDIA-RTX-PRO-4500-Blackwell-Server-Edition

  • NVIDIA-RTX-PRO-6000-Blackwell-Server-Edition

qwen3.5-397b-a17b#

The following table lists the supported profile configurations for qwen/qwen3.5-397b-a17b:

Precision

TP1

TP2

TP4

TP8

BF16

–

–

vllm-bf16-tp4-pp1

vllm-bf16-tp8-pp1

BF16 + LoRA

–

–

vllm-bf16-tp4-pp1-lora

vllm-bf16-tp8-pp1-lora

FP8

–

vllm-fp8-tp2-pp1

vllm-fp8-tp4-pp1

vllm-fp8-tp8-pp1

FP8 + LoRA

–

vllm-fp8-tp2-pp1-lora

vllm-fp8-tp4-pp1-lora

vllm-fp8-tp8-pp1-lora

NVFP4

vllm-nvfp4-tp1-pp1

vllm-nvfp4-tp2-pp1

vllm-nvfp4-tp4-pp1

vllm-nvfp4-tp8-pp1

NVFP4 + LoRA

vllm-nvfp4-tp1-pp1-lora

vllm-nvfp4-tp2-pp1-lora

vllm-nvfp4-tp4-pp1-lora

vllm-nvfp4-tp8-pp1-lora

Optimized deployment profiles#

Precision

TP

GPU

LoRA

Tuning Objective

Profile ID

NVFP4

2

NVIDIA-B200

No

Throughput

5ce5f8a3d5f0bf67afc04538745d5a9f364da7c3a79a584e84be859c911d7fb5

FP8

4

NVIDIA-H200

No

Throughput

ed6edfcd8bb730b02a1420640544dcbb13779e732f9de269b25ce1f75bcfb602

Verified GPUs

This model has been verified on the following GPUs:

  • NVIDIA-B200

  • NVIDIA-B300-SXM6-AC

  • NVIDIA-GB200

  • NVIDIA-GB300

  • NVIDIA-H100-80GB-HBM3

  • NVIDIA-H200

  • NVIDIA-H200-NVL

  • NVIDIA-RTX-PRO-6000-Blackwell-Server-Edition

Model-Free NIM#

vLLM#

The following models are tested and validated for nvidia/model-free-nim:

  • gpt-oss-20b

  • apriel-nemotron

  • codestral

The following models are tested and validated for nvidia/vllm-model-free-nim-experimental:

While not explicitly validated, the model-free NIM can be used with any model supported by the underlying backend (vLLM) version. Refer to Model-Free NIM for deployment details.

Verified GPUs

The model-free NIM has been verified on the following GPUs:

  • NVIDIA-A100-80GB-PCIe

  • NVIDIA-A100-PCIE-40GB

  • NVIDIA-A100-SXM4-40GB

  • NVIDIA-A100-SXM4-80GB

  • NVIDIA-A10G

  • NVIDIA-B200

  • NVIDIA-B300-SXM6-AC

  • NVIDIA-GB10

  • NVIDIA-GB200

  • NVIDIA-GB300

  • NVIDIA-GB300-WS

  • NVIDIA-GH200-144G-HBM3e

  • NVIDIA-GH200-480GB

  • NVIDIA-H100-80GB-HBM3

  • NVIDIA-H100-NVL

  • NVIDIA-H100-PCIe

  • NVIDIA-H200

  • NVIDIA-H200-NVL

  • NVIDIA-L40S

  • NVIDIA-RTX-PRO-4500-Blackwell-Server-Edition

  • NVIDIA-RTX-PRO-6000-Blackwell-Server-Edition

Note

Model-specific VRAM requirement in 2.0.12: When this NIM serves apriel-nemotron or codestral, use a GPU with at least 80 GB of usable VRAM. These models require approximately 66 GB for weights and runtime allocations and cannot start on smaller GPUs. Other validated models can use the remaining verified GPUs listed above.

Step-5-Preview#

The following table lists the supported profile configurations for stepfun-ai/Step-5-Preview. This model is served by the experimental vLLM Model-Free NIM. For deployment and API examples, refer to the dedicated Get Started guide.

GPU

Precision

GPUs

TP

Expert Parallelism

MTP

LoRA

Recipe File

H20-3e

FP8

8

8

No

3

No

step5-preview-fp8-h20-3e-8gpu-tp8-mtp3.yaml

H200

FP8

8

8

No

3

No

step5-preview-fp8-h200-8gpu-tp8-mtp3.yaml

B200

NVFP4

4

4

Yes

3

No

step5-preview-nvfp4-b200-4gpu-tp4-dp1-ep-mtp3.yaml

GB200

NVFP4

4

4

Yes

3

No

step5-preview-nvfp4-gb200-4gpu-tp4-dp1-ep-mtp3.yaml

All four configurations use multi-token prediction with three speculative tokens (MTP3). The NVFP4 configurations add data parallelism of 1 (DP1) alongside expert parallelism.

Each configuration is delivered as a tuning config file in the Step-5 Preview Serving Recipes resource on NGC. The container does not select these files automatically. Load the one matching your GPU with NIM_CONFIG_FILE, as described in the Get Started guide.

Verified GPUs

This model has been verified on the following GPUs:

  • NVIDIA-B200

  • NVIDIA-GB200

  • NVIDIA-H20-3e

  • NVIDIA-H200

SGLang#

The following models are tested and validated for nvidia/sglang-model-free-nim:

The following models are tested and validated for nvidia/sglang-model-free-nim-experimental:

1.x NIM LLM Models#

For more information on version 1.x NIMs, refer to the 1.15 version of the NIM LLM Supported Models page.

Show 1.x models

Model (Hardware Requirements)

Organization/Model ID (Catalog Page)

Bielik 11B v2.3 Instruct

speakleash/bielik-11b-v2.3-instruct

Code Llama 13B Instruct

meta/codellama-13b-instruct

Code Llama 34B Instruct

meta/codellama-34b-instruct

Code Llama 70B Instruct

meta/codellama-70b-instruct

DeepSeek Coder V2 Lite Instruct

deepseek-ai/deepseek-coder-v2-lite-instruct

DeepSeek R1

deepseek-ai/deepseek-r1

DeepSeek R1 Distill Llama 8B

deepseek-ai/deepseek-r1-distill-llama-8b

DeepSeek R1 Distill Llama 70B

deepseek-ai/deepseek-r1-distill-llama-70b

DeepSeek R1 Distill Llama 8B RTX

deepseek-ai/deepseek-r1-distill-llama-8b

DeepSeek R1 Distill Qwen 32B

deepseek-ai/deepseek-r1-distill-qwen-32b

DeepSeek-V3.1-Terminus

deepseek-ai/deepseek-v3.1-terminus

DeepSeek-V3.2-Exp

deepseek-ai/deepseek-v32-exp-nim

DeepSeek-V4-Flash

deepseek-ai/deepseek-v4-flash

DeepSeek-V4-Flash-0731

deepseek-ai/deepseek-v4-flash-0731

DeepSeek-V4-Pro

deepseek-ai/deepseek-v4-pro

EuroLLM 9B Instruct

utter-project/eurollm-9b-instruct

Gemma 2 2B

google/gemma-2-2b-instruct

Gemma 2 9B

google/gemma-2-9b-it

Gemma2 9B CPT Sahabat-AI v1 Instruct

gotocompany/gemma2-9b-cpt-sahabatai-v1-instruct

Gemma 3 1B Instruct

google/gemma-3-1b-it

GLM-5

zai-org/glm-5

GLM-5.1

zai-org/glm-51

GLM-5.2

zai-org/glm-5.2

GPT-OSS-20B

openai/gpt-oss-20b

GPT-OSS-120B

openai/gpt-oss-120b

Granite 3.3 8B Instruct

ibm-granite/granite-3.3-8b-instruct

GreenMind Medium 14B R1

greennode/greenmind-medium-14b-r1

Kanana 1.5 8B Instruct 2505

kakaocorp/kanana-1.5-8b-instruct-2505

(Meta) Llama 2 7B Chat

meta/llama-2-7b-chat

(Meta) Llama 2 13B Chat

meta/llama-2-13b-chat

(Meta) Llama 2 70B Chat

meta/llama-2-70b-chat

Llama 3 SQLCoder 8B

defog/llama-3-sqlcoder-8b

Llama 3 Swallow 70B Instruct V0.1

tokyotech-llm/llama-3-swallow-70b-instruct-v0.1

Llama 3 Taiwan 70B Instruct

yentinglin/llama-3-taiwan-70b-instruct

Llama 3.1 8B Base

meta/llama-3.1-8b-base

Llama 3.1 8B Instruct

meta/llama-3.1-8b-instruct

Llama-3.1-8b-Instruct-DGX-Spark

meta/llama-3.1-8b-instruct-dgx-spark

Llama-3.1-8B-Instruct PB 25h2

meta/llama-3.1-8b-instruct-pb25h2

Llama 3.1 8B Instruct RTX

meta/llama-3.1-8b-instruct

Llama 3.1 70B Instruct

meta/llama-3.1-70b-instruct

Llama-3.1-70B-Instruct PB 25h2

meta/llama-3.1-70b-instruct-pb25h2

Llama 3.1 405B Instruct

meta/llama-3.1-405b-instruct

Llama 3.1 Nemotron Nano 4B V1.1

nvidia/llama3.1-nemotron-nano-4b-v1.1

Llama 3.1 Nemotron Nano 8B V1

nvidia/llama-3.1-nemotron-nano-8b-v1

Llama-3.1-Nemotron-Nano-8B-Healthcare-Text2sql-v1.0

nvidia/llama-3.1-nemotron-nano-8b-healthcare-text2sql-v1.0

Llama 3.1 Nemotron Ultra 253B V1

nvidia/llama-3.1-nemotron-ultra-253b-v1

Llama 3.1 Nemotron 70B Instruct

nvidia/llama-3.1-nemotron-70b-instruct

Llama 3.1 Swallow 8B Instruct v0.1

tokyotech-llm/llama-3.1-swallow-8b-instruct-v0.1

Llama 3.1 Swallow 70B Instruct v0.1

tokyotech-llm/llama-3.1-swallow-70b-instruct-v0.1

Llama 3.1 Typhoon 2 8B Instruct

scb10x/llama3.1-typhoon2-8b-instruct

Llama 3.1 Typhoon 2 70B Instruct

scb10x/llama-3.1-typhoon2-70b-instruct

Llama 3.2 1B Instruct

meta/llama-3.2-1b-instruct

Llama 3.2 3B Instruct

meta/llama-3.2-3b-instruct

Llama 3.3 70B Instruct

meta/llama-3.3-70b-instruct

Llama 3.3 Nemotron Super 49B V1

nvidia/llama-3.3-nemotron-super-49b-v1

Llama-3.3-Nemotron-Super-49B-v1.5

nvidia/llama-3.3-nemotron-super-49b-v1.5

Llama-3.3-Nemotron-Super-49B-Healthcare-Text2sql-v1.0

nvidia/llama-3.3-nemotron-super-49b-healthcare-text2sql-v1.0

Llama-3.3-Nemotron-Super-49B-v1.5 PB 25h2

nvidia/llama-3.3-nemotron-super-49b-v1.5-pb25h2

Meta Llama 3 8B Instruct

meta/llama3-8b-instruct

Meta Llama 3 70B Instruct

meta/llama3-70b-instruct

Muse Glimmer

meta/muse-glimmer

MiMo-V2-Flash

xiaomi/mimo-v2-flash-experimental

MiniMax-M2.5

minimax-ai/minimax-m25

MiniMax-M2.7

minimax-ai/minimax-m27

Mistral 7B Instruct V0.3

mistralai/mistral-7b-instruct-v0.3

Mistral NeMo 12B Instruct RTX

nv-mistralai/mistral-nemo-12b-instruct

Mistral NeMo 12B Instruct

nv-mistralai/mistral-nemo-12b-instruct

Mistral NeMo Minitron 8B 8K Instruct

nv-mistralai/mistral-nemo-minitron-8b-8k-instruct

Mistral Small 24b Instruct 2501

mistralai/mistral-small-24b-instruct-2501

Mixtral 8x7B Instruct V0.1

mistralai/mixtral-8x7b-instruct-v0-1

Mixtral 8x22B Instruct V0.1

mistralai/mixtral-8x22b-instruct-v01

Nemotron 4 340B Instruct

nvidia/nemotron-4-340b-instruct

Nemotron 4 340B Reward

nvidia/nemotron-4-340b-reward

NVIDIA Nemotron 3 Nano

nvidia/nemotron-3-nano

NVIDIA-Nemotron-Nano-9B-v2

nvidia/nvidia-nemotron-nano-9b-v2

NVIDIA-Nemotron-Nano-9B-v2-DGX-Spark

nvidia/nvidia-nemotron-nano-9b-v2-dgx-spark

Nemotron-3-Super-120B-A12B

nvidia/nemotron-3-super-120b-a12b

Phi 3 Mini 4K Instruct

microsoft/phi-3-mini-4k-instruct

Phi 4 Mini Instruct

microsoft/phi-4-mini-instruct

Phind Codellama 34B V2 Instruct

phind/phind-codellama-34b-v2-instruct

Qwen3-Coder-Next

qwen/qwen3-coder-next

Qwen3-Next-80B-A3B-Instruct

qwen/qwen3-next-80b-a3b-instruct

Qwen3 Next 80B A3B Thinking

qwen/qwen3-next-80b-a3b-thinking

Qwen3-32B

qwen/qwen3-32b

Qwen3-32B NIM for DGX Spark

qwen/qwen3-32b-dgx-spark

Qwen2.5 Coder 32B Instruct

qwen/qwen2.5-coder-32b-instruct

Qwen2.5 7B Instruct

qwen/qwen-2.5-7b-instruct

Riva Translate 4B Instruct

nvidia/riva_translate_4b_instruct

Riva-Translate-4b-Instruct-v1.1

nvidia/riva-translate-4b-instruct-v1.1

Sarvam - M

sarvamai/sarvam-m

SILMA 9B Instruct v1.0

silma-ai/silma-9b-instruct-v1.0

StarCoder2 7B

bigcode/starcoder2-7b

StarCoderBase 15.5B

bigcode/starcoderbase-15b

Stockmark-2-100B-Instruct

stockmark/stockmark-2-100b-instruct

Teuken 7B Instruct Commercial v0.4

opengpt-x/teuken-7b-instruct-commercial-v0.4

Additional VLM Models#

VLMs that were released prior to version 2.0.12 are documented on the NVIDIA NIM for Vision Language Models site. Refer to that site for more information on running these VLM models.

Show additional VLM models

Model (Hardware Requirements)

Organization/Model ID (Catalog Page)

Cosmos 3 Reasoner

nvidia/cosmos3-reasoner

Cosmos Reason1 7B

nvidia/cosmos-reason1-7b

Cosmos Reason2

nvidia/cosmos-reason2-8b

DiffusionGemma 26B A4B IT

google/diffusiongemma-26b-a4b-it

Gemma 4 31B Instruct

google/gemma-4-31b-it

Gemma 4 31B IT

google/gemma-4-31b-it

Gemma-4-26B-A4B-IT

google/gemma-4-26b-a4b-it

GLM-5.3-Flash

zai-org/glm-5.3-flash

Inkling

thinkingmachines/inkling

Kimi-K2.5

moonshotai/kimi-k2.5

Kimi-K2.6 Turbo

moonshotai/kimi-k2.6-turbo

Llama 3.1 Nemotron Nano VL 8B v1

nvidia/llama-3.1-nemotron-nano-vl-8b-v1

Llama 3.2 11B Vision Instruct

meta/llama-3.2-11b-vision-instruct

Llama 3.2 90B Vision Instruct

meta/llama-3.2-90b-vision-instruct

Llama 4 Maverick 17B 128E Instruct

meta/llama-4-maverick-17b-128e-instruct

Llama 4 Scout 17B 16E Instruct

meta/llama-4-scout-17b-16e-instruct

Ministral 3 14B Instruct 2512

mistralai/ministral-3-14b-instruct-2512

Mistral Large 3 675B Instruct 2512

mistralai/mistral-large-3-675b-instruct-2512

Mistral Medium 3.5

mistralai/mistral-medium-3.5-128b

Mistral Medium 3.5 128B

mistralai/mistral-medium-3.5-128b

Mistral Small 3.2 24B Instruct 2506

mistralai/mistral-small-3.2-24b-instruct-2506

Mistral Small 4

mistralai/mistral-small-4-119b-2603

Mistral-Small-4-119B-2603

mistralai/mistral-small-4-119b-2603

Muse Glimmer

meta/muse-glimmer

nemoretriever-parse

nvidia/nemoretriever-parse

Nemotron 3.5 Content Safety

nvidia/nemotron-3.5-content-safety

Nemotron Nano 12B v2 VL

nvidia/nemotron-nano-12b-v2-vl

Nemotron Parse

nvidia/nemotron-parse

Nemotron Parse v2.0

nvidia/nemotron-parse-v2.0

Nemotron-3-Content-Safety

nvidia/nemotron-3-content-safety

Nemotron-3-Nano-Omni-30B-A3B-Reasoning

nvidia/nemotron-3-nano-omni-30b-a3b-reasoning

Nemotron-Parse-v1.2

nvidia/nemotron-parse-v1.2

NVIDIA Nemotron 3 Nano Omni

nvidia/nemotron-3-nano-omni-30b-a3b-reasoning

Qwen3.5

qwen/qwen3.5-35b-a3b

Qwen3.5-122B-A10B

qwen/qwen3.5-122b-a10b

Qwen3.5-397B-A17B

qwen/qwen3.5-397b-a17b

Qwen3.6-27B

qwen/qwen3.6-27b

Qwen3.6-35B-A3B

qwen/qwen3.6-35b-a3b

Qwen3.8-27B

qwen/qwen3.8-27b

Qwen3.8-Flash-Next

nvidia/vllm-model-free-nim

Step 3.7 Flash

stepfun-ai/step-3.7-flash