Is this page helpful?

Support Matrix for NIMs#

This page lists supported models, deployment profiles, and verified hardware SKUs for NIM LLM and VLM. This page is organized according to model type:

LLMs
VLMs
Model-Free NIM

Use the following sections to identify the supported deployment profiles for each model. Each NIM container image embeds one inference engine, vLLM or SGLang. The profile prefix (vllm- or sglang-) identifies the engine, and an image loads only profiles for its own engine. Profile strings follow a naming convention described in Model Profiles and Selection.

Optimized profiles are listed separately in the model sections because multiple configurations can share the same precision and tensor-parallel size. For selection instructions, refer to Model Profiles and Selection.

For programmatic access to published profiles and hardware requirements for a specific container tag, use the NIM Metadata API before pulling the image. Metadata coverage varies by NIM and tag; if metadata is unavailable, use this matrix and the list-model-profiles command described in Model Profiles and Selection.

Important

The list of NIM Certified (NVIDIA AI Enterprise supported) models is available on the NGC Catalog site.

Use the table below to filter NIM profiles by GPU, tensor parallelism (TP), precision, model type, LoRA support, and model name. Each row is one supported profile; details also appear in the per-model sections that follow.

Optimized profile entries identify their tuning objective and link to the full profile ID in the model section. The filters also apply to rows in the optimized profile tables below.

Note

The GB300-WS GPU is more commonly known as DGX Station.

VLM only LoRA only

Model	TP	Precision	LoRA
Apriel-Nemotron-15b-Thinker	1	BF16	No
Codestral-2501	1	BF16	No
deepseek-v4.1-flash	4	FP8	No
deepseek-v4.1-flash	8	FP8	No
deepseek-v4.1-flash	4	NVFP4	No
deepseek-v4.1-flash	8	NVFP4	No
gemma-4-26b-a4b-it	1	BF16	No
gemma-4-26b-a4b-it	2	BF16	No
gemma-4-26b-a4b-it	4	BF16	No
gemma-4-26b-a4b-it	1	NVFP4	No
gemma-4-31b-it	1	BF16	No
gemma-4-31b-it	2	BF16	No
gemma-4-31b-it	4	BF16	No
gemma-4-31b-it	2	INT4	No
gemma-4-31b-it	1	NVFP4	No
gemma-4-31b-it	2	NVFP4	No
gemma-4-31b-it	4	NVFP4	No
glm-5.2	4	FP8	No
glm-5.2	4	FP8	Yes
glm-5.2	8	FP8	No
glm-5.2	8	FP8	Yes
glm-5.2	4	NVFP4	No
glm-5.2	4	NVFP4	Yes
glm-5.2	8	NVFP4	No
glm-5.2	8	NVFP4	Yes
GLM-5.3	4	NVFP4 + FP8	No
GLM-5.3	8	NVFP4 + FP8	No
GLM-5.3	8	FP8	No
GLM-5.3-Flash	2	NVFP4	No
GLM-5.3-Flash	4	NVFP4	No
GLM-5.3-Flash	8	NVFP4	No
GLM-5.3-Flash	4	FP8	No
GLM-5.3-Flash	8	FP8	No
GLM-5.3-Flash	4	BF16	No
GLM-5.3-Flash	8	BF16	No
gpt-oss-120b	1	MXFP4	No
gpt-oss-120b	2	MXFP4	No
gpt-oss-120b	4	MXFP4	No
gpt-oss-120b	8	MXFP4	No
gpt-oss-120b	1	MXFP4	Yes
gpt-oss-120b	2	MXFP4	Yes
gpt-oss-120b	4	MXFP4	Yes
gpt-oss-120b	8	MXFP4	Yes
gpt-oss-20b	1	MXFP4	No
gpt-oss-20b	2	MXFP4	No
gpt-oss-20b	4	MXFP4	No
gpt-oss-20b	8	MXFP4	No
gpt-oss-20b	1	MXFP4	Yes
gpt-oss-20b	2	MXFP4	Yes
gpt-oss-20b	4	MXFP4	Yes
gpt-oss-20b	8	MXFP4	Yes
Inkling	4	NVFP4	No
Inkling	4	NVFP4	Yes
Inkling	8	NVFP4	No
Inkling	8	NVFP4	Yes
Inkling	8	NVFP4	No
Inkling	8	NVFP4	Yes
inkling-small	2	NVFP4	No
inkling-small	4	NVFP4	No
inkling-small	8	BF16	No
inkling-small	8	NVFP4	No
kimi-k2.6	4	NVFP4	No
kimi-k2.6	8	INT4	No
Kimi-K3	8	NVFP4	No
Kimi-K3	8	MXFP4	No
llama-3.1-70b-instruct	1	BF16	No
llama-3.1-70b-instruct	2	BF16	No
llama-3.1-70b-instruct	4	BF16	No
llama-3.1-70b-instruct	8	BF16	No
llama-3.1-70b-instruct	1	BF16	Yes
llama-3.1-70b-instruct	2	BF16	Yes
llama-3.1-70b-instruct	4	BF16	Yes
llama-3.1-70b-instruct	8	BF16	Yes
llama-3.1-70b-instruct	1	FP8	No
llama-3.1-70b-instruct	2	FP8	No
llama-3.1-70b-instruct	4	FP8	No
llama-3.1-70b-instruct	8	FP8	No
llama-3.1-70b-instruct	1	FP8	Yes
llama-3.1-70b-instruct	2	FP8	Yes
llama-3.1-70b-instruct	4	FP8	Yes
llama-3.1-70b-instruct	8	FP8	Yes
llama-3.1-70b-instruct	1	NVFP4	No
llama-3.1-70b-instruct	2	NVFP4	No
llama-3.1-70b-instruct	4	NVFP4	No
llama-3.1-70b-instruct	8	NVFP4	No
llama-3.1-70b-instruct	1	NVFP4	Yes
llama-3.1-70b-instruct	2	NVFP4	Yes
llama-3.1-70b-instruct	4	NVFP4	Yes
llama-3.1-70b-instruct	8	NVFP4	Yes
llama-3.1-8b-instruct	1	BF16	No
llama-3.1-8b-instruct	1	BF16	Yes
llama-3.1-8b-instruct	1	FP8	No
llama-3.1-8b-instruct	1	FP8	Yes
llama-3.1-8b-instruct	1	NVFP4	No
llama-3.1-8b-instruct	1	NVFP4	Yes
llama-3.3-70b-instruct	1	BF16	No
llama-3.3-70b-instruct	2	BF16	No
llama-3.3-70b-instruct	4	BF16	No
llama-3.3-70b-instruct	8	BF16	No
llama-3.3-70b-instruct	1	BF16	Yes
llama-3.3-70b-instruct	2	BF16	Yes
llama-3.3-70b-instruct	4	BF16	Yes
llama-3.3-70b-instruct	8	BF16	Yes
llama-3.3-70b-instruct	1	FP8	No
llama-3.3-70b-instruct	2	FP8	No
llama-3.3-70b-instruct	4	FP8	No
llama-3.3-70b-instruct	8	FP8	No
llama-3.3-70b-instruct	1	FP8	Yes
llama-3.3-70b-instruct	2	FP8	Yes
llama-3.3-70b-instruct	4	FP8	Yes
llama-3.3-70b-instruct	8	FP8	Yes
llama-3.3-70b-instruct	1	NVFP4	No
llama-3.3-70b-instruct	2	NVFP4	No
llama-3.3-70b-instruct	4	NVFP4	No
llama-3.3-70b-instruct	8	NVFP4	No
llama-3.3-70b-instruct	1	NVFP4	Yes
llama-3.3-70b-instruct	2	NVFP4	Yes
llama-3.3-70b-instruct	4	NVFP4	Yes
llama-3.3-70b-instruct	8	NVFP4	Yes
llama-3.3-nemotron-super-49b-v1.5	1	BF16	No
llama-3.3-nemotron-super-49b-v1.5	2	BF16	No
llama-3.3-nemotron-super-49b-v1.5	4	BF16	No
llama-3.3-nemotron-super-49b-v1.5	8	BF16	No
llama-3.3-nemotron-super-49b-v1.5	1	BF16	Yes
llama-3.3-nemotron-super-49b-v1.5	2	BF16	Yes
llama-3.3-nemotron-super-49b-v1.5	4	BF16	Yes
llama-3.3-nemotron-super-49b-v1.5	8	BF16	Yes
llama-3.3-nemotron-super-49b-v1.5	1	FP8	No
llama-3.3-nemotron-super-49b-v1.5	2	FP8	No
llama-3.3-nemotron-super-49b-v1.5	4	FP8	No
llama-3.3-nemotron-super-49b-v1.5	8	FP8	No
llama-3.3-nemotron-super-49b-v1.5	1	FP8	Yes
llama-3.3-nemotron-super-49b-v1.5	2	FP8	Yes
llama-3.3-nemotron-super-49b-v1.5	4	FP8	Yes
llama-3.3-nemotron-super-49b-v1.5	8	FP8	Yes
llama-3.3-nemotron-super-49b-v1.5	1	NVFP4	No
llama-3.3-nemotron-super-49b-v1.5	2	NVFP4	No
llama-3.3-nemotron-super-49b-v1.5	4	NVFP4	No
llama-3.3-nemotron-super-49b-v1.5	8	NVFP4	No
llama-3.3-nemotron-super-49b-v1.5	1	NVFP4	Yes
llama-3.3-nemotron-super-49b-v1.5	2	NVFP4	Yes
llama-3.3-nemotron-super-49b-v1.5	4	NVFP4	Yes
llama-3.3-nemotron-super-49b-v1.5	8	NVFP4	Yes
DeepSeek-V4.1-Flash-1017	4	FP8	No
DeepSeek-V4.1-Flash-1017	8	FP8	No
Kimi K3.1 Pro (2 nodes, PP2)	8	MXFP4	No
Kimi K3.1 Pro (4 nodes, PP4)	8	MXFP4	No
Kimi K3.1 Pro (1 node, PP1)	8	MXFP4	No
MiniMax M3.1 Flash	4	FP8	No
MiniMax M3.1 Flash	8	FP8	No
MiniMax-M3	2	NVFP4	No
MiniMax-M3	4	NVFP4	No
MiniMax-M3	4	NVFP4	No
MiniMax-M3	8	NVFP4	No
MiniMax-M3	4	BF16	No
MiniMax-M3	8	BF16	No
Mistral Large 4	8	NVFP4	No
Mistral Large 4	8	NVFP4	No
Mistral Large 4	8	NVFP4	No
Mistral Large 4	8	FP8	No
Mistral Large 4	8	FP8	No
Mistral Large 4	8	FP8	No
Mistral-Medium-3.5-128B	8	FP8	No
Mistral-Medium-3.5-128B	4	NVFP4	No
mistral-small-4-119b-2603	1	FP8	No
mistral-small-4-119b-2603	2	FP8	No
mistral-small-4-119b-2603	4	FP8	No
mistral-small-4-119b-2603	8	FP8	No
mistral-small-4-119b-2603	1	NVFP4	No
mistral-small-4-119b-2603	2	NVFP4	No
mistral-small-4-119b-2603	4	NVFP4	No
mistral-small-4-119b-2603	8	NVFP4	No
muse-glimmer	1	BF16	No
muse-glimmer	2	BF16	No
muse-glimmer	1	NVFP4	No
muse-glimmer	1	NVFP4	No
muse-glimmer	2	NVFP4	No
Beam-1.0-501B-A23B	8	FP8	No
Beam-1.0-501B-A23B	8	FP8	No
Beam-1.0-501B-A23B	8	FP8	No
Beam-1.0-501B-A23B	4	NVFP4	No
Beam-1.0-501B-A23B	4	NVFP4	No
Muse Spark Open	8	NVFP4	No
nemotron-3-nano	1	BF16	No
nemotron-3-nano	2	BF16	No
nemotron-3-nano	4	BF16	No
nemotron-3-nano	8	BF16	No
nemotron-3-nano	1	BF16	Yes
nemotron-3-nano	2	BF16	Yes
nemotron-3-nano	4	BF16	Yes
nemotron-3-nano	8	BF16	Yes
nemotron-3-nano	1	FP8	No
nemotron-3-nano	2	FP8	No
nemotron-3-nano	4	FP8	No
nemotron-3-nano	8	FP8	No
nemotron-3-nano	1	NVFP4	No
nemotron-3-nano	2	NVFP4	No
nemotron-3-nano	4	NVFP4	No
nemotron-3-nano	8	NVFP4	No
nemotron-3-nano	1	NVFP4	Yes
nemotron-3-nano-omni-30b-a3b-reasoning	2	BF16	No
nemotron-3-nano-omni-30b-a3b-reasoning	1	FP8	No
nemotron-3-nano-omni-30b-a3b-reasoning	1	NVFP4	No
nemotron-3-super-120b-a12b	1	BF16	No
nemotron-3-super-120b-a12b	2	BF16	No
nemotron-3-super-120b-a12b	4	BF16	No
nemotron-3-super-120b-a12b	8	BF16	No
nemotron-3-super-120b-a12b	2	BF16	Yes
nemotron-3-super-120b-a12b	4	BF16	Yes
nemotron-3-super-120b-a12b	8	BF16	Yes
nemotron-3-super-120b-a12b	1	FP8	No
nemotron-3-super-120b-a12b	2	FP8	No
nemotron-3-super-120b-a12b	4	FP8	No
nemotron-3-super-120b-a12b	8	FP8	No
nemotron-3-super-120b-a12b	1	NVFP4	No
nemotron-3-super-120b-a12b	2	NVFP4	No
nemotron-3-super-120b-a12b	4	NVFP4	No
nemotron-3-super-120b-a12b	8	NVFP4	No
nemotron-3-super-120b-a12b	1	NVFP4	Yes
nemotron-3-super-120b-a12b	2	NVFP4	Yes
nemotron-3-ultra-550b-a55b	8	BF16	No
nemotron-3-ultra-550b-a55b	2	NVFP4	No
nemotron-3-ultra-550b-a55b	4	NVFP4	No
nemotron-3-ultra-550b-a55b	8	NVFP4	No
nemotron-3.5-bolt-e4b	1	BF16	No
nemotron-3.5-lightning	1	BF16	No
nemotron-3.5-lightning	1	BF16	Yes
nemotron-3.5-lightning	2	BF16	No
nemotron-3.5-lightning	2	BF16	Yes
nemotron-3.5-lightning	4	BF16	No
nemotron-3.5-lightning	4	BF16	Yes
nemotron-3.5-lightning	8	BF16	No
nemotron-3.5-lightning	8	BF16	Yes
nemotron-3.5-lightning	1	W4A16	No
nemotron-3.5-lightning	1	W4A16	Yes
nemotron-3.5-lightning	2	W4A16	No
nemotron-3.5-lightning	2	W4A16	Yes
nemotron-3.5-lightning	4	W4A16	No
nemotron-3.5-lightning	4	W4A16	Yes
nemotron-3.5-lightning	1	NVFP4	No
nemotron-3.5-lightning	1	NVFP4	Yes
nemotron-3.5-lightning	2	NVFP4	No
nemotron-3.5-lightning	2	NVFP4	Yes
nemotron-3.5-lightning	4	NVFP4	No
nemotron-3.5-lightning	4	NVFP4	Yes
nemotron-3.5-super-vl-120b-a12b	2	BF16	Yes
nemotron-3.5-super-vl-120b-a12b	4	BF16	Yes
nemotron-3.5-super-vl-120b-a12b	8	BF16	Yes
nemotron-3.5-super-vl-120b-a12b	1	NVFP4	No
nemotron-3.5-super-vl-120b-a12b	2	NVFP4	No
qwen3.5-122b-a10b	1	BF16	No
qwen3.5-122b-a10b	2	BF16	No
qwen3.5-122b-a10b	2	BF16	Yes
qwen3.5-122b-a10b	4	BF16	No
qwen3.5-122b-a10b	4	BF16	Yes
qwen3.5-122b-a10b	8	BF16	No
qwen3.5-122b-a10b	8	BF16	Yes
qwen3.5-122b-a10b	1	FP8	No
qwen3.5-122b-a10b	1	FP8	Yes
qwen3.5-122b-a10b	2	FP8	No
qwen3.5-122b-a10b	2	FP8	Yes
qwen3.5-122b-a10b	4	FP8	No
qwen3.5-122b-a10b	4	FP8	Yes
qwen3.5-122b-a10b	8	FP8	No
qwen3.5-122b-a10b	8	FP8	Yes
qwen3.5-122b-a10b	1	NVFP4	No
qwen3.5-122b-a10b	1	NVFP4	Yes
qwen3.5-122b-a10b	2	NVFP4	No
qwen3.5-122b-a10b	2	NVFP4	Yes
qwen3.5-122b-a10b	4	NVFP4	No
qwen3.5-122b-a10b	4	NVFP4	Yes
qwen3.5-122b-a10b	8	NVFP4	No
qwen3.5-122b-a10b	8	NVFP4	Yes
qwen3.5-397b-a17b	4	BF16	No
qwen3.5-397b-a17b	8	BF16	No
qwen3.5-397b-a17b	2	FP8	No
qwen3.5-397b-a17b	4	FP8	No
qwen3.5-397b-a17b	8	FP8	No
qwen3.5-397b-a17b	1	NVFP4	No
qwen3.5-397b-a17b	2	NVFP4	No
qwen3.5-397b-a17b	4	NVFP4	No
qwen3.5-397b-a17b	8	NVFP4	No
Qwen3.8-2.4T-A95B	16	FP8	No
qwen3.8-27b	1	NVFP4	No
qwen3.8-27b	2	NVFP4	No
qwen3.8-27b	1	FP8	No
qwen3.8-27b	2	FP8	No
qwen3.8-27b	1	BF16	No
qwen3.8-27b	2	BF16	No
qwen3.8-27b	4	BF16	No
Qwen3.8-Flash-Next	1	NVFP4	No
Qwen3.8-Flash-Next	2	NVFP4	No
Qwen3.8-Flash-Next	4	NVFP4	No
Qwen3.8-Flash-Next	2	FP8	No
Qwen3.8-Flash-Next	4	FP8	No
Qwen3.8-Flash-Next	4	BF16	No
Qwen3.8-Flash-Next	8	BF16	No
riva-translate-4b-instruct-v2	1	BF16	No
starcoder2-7b	1	BF16	No
starcoder2-7b	2	BF16	No
Step 5 Preview	8	FP8	No
Step 5 Preview	4	NVFP4	No
No matching profiles. This configuration may still be deployable with memory tuning; refer to the memory troubleshooting guide for details.
No matching profiles. This GPU is verified for Model-Free NIM deployment, and this configuration may also be deployable with memory tuning; refer to the memory troubleshooting guide for details.

LLMs#

The following section lists all of the supported LLMs.

deepseek-v4-flash-0731#

The following table lists the optimized deployment profiles for deepseek-ai/deepseek-v4-flash-0731:

Optimized Deployment Profiles#
Precision	TP	GPU	LoRA	Tuning Objective	Profile ID
FP8	4	`NVIDIA-H200`	No	Throughput	`e5199f4aca4d6486c0f2b32afda8a5603300d93b5c70a219f3d2feb5929ee6ef`
NVFP4	4	`NVIDIA-B200`	No	Throughput	`d2371848980d6a4ceae23e40ab40bc6b61a34c4bc672612f2dbacfbac7108704`

deepseek-v4-pro-0813#

The following table lists the optimized deployment profiles for deepseek-ai/deepseek-v4-pro-0813:

Optimized Deployment Profiles#
Precision	TP	GPU	LoRA	Tuning Objective	Profile ID
FP8	8	`NVIDIA-H200`	No	Throughput	`2cf4354f85b674da421b1f0c0edd95df85ba08d69d7243a7300f5b76d310c917`
NVFP4	8	`NVIDIA-B200`	No	Throughput	`f6e709b4126a3f3c6db754c1d5090a5d1006d23b5f0e62c4cec8e423e8eb5de0`

glm-5.2#

The following table lists the supported profile configurations for zai-org/glm-5.2:

Precision	TP4	TP8
FP8	`sglang-fp8-tp4-pp1`	`sglang-fp8-tp8-pp1`
FP8 + LoRA	`sglang-fp8-tp4-pp1-lora`	`sglang-fp8-tp8-pp1-lora`
NVFP4	`sglang-nvfp4-tp4-pp1`	`sglang-nvfp4-tp8-pp1`
NVFP4 + LoRA	`sglang-nvfp4-tp4-pp1-lora`	`sglang-nvfp4-tp8-pp1-lora`

Optimized Deployment Profiles#
Precision	TP	GPU	LoRA	Tuning Objective	Profile ID
FP8	8	NVIDIA-H200	No	Throughput	`28a9b5fd93083449373fbca26192707d1f417d6295e1ec93d2b399f30be6886a`
NVFP4	4	NVIDIA-B200	No	Throughput	`53546dab4fb95002d4ffa193ec8367137140291e3dd847d3a897f185b5a9a1f7`

gpt-oss-120b#

The following table lists the supported profile configurations for openai/gpt-oss-120b:

Precision	TP1	TP2	TP4	TP8
MXFP4	`vllm-mxfp4-tp1-pp1`	`vllm-mxfp4-tp2-pp1`	`vllm-mxfp4-tp4-pp1`	`vllm-mxfp4-tp8-pp1`
MXFP4 + LoRA	`vllm-mxfp4-tp1-pp1-lora`	`vllm-mxfp4-tp2-pp1-lora`	`vllm-mxfp4-tp4-pp1-lora`	`vllm-mxfp4-tp8-pp1-lora`

Optimized deployment profiles#
Precision	TP	GPU	LoRA	Tuning Objective	Profile ID
MXFP4	1	NVIDIA-A100-SXM4-80GB	No	Throughput (32,768-token context)	`bb8ab328c6d04b2418a676dfeec2a7a1dfe1e877918da4d43ec16dab98be7eda`
MXFP4	1	NVIDIA-B200	No	Throughput	`766aa1d86ebcb546aae7635c91d70d90fa1ed68514ddf59a217dce079ad57225`
MXFP4	1	NVIDIA-B300-SXM6-AC	No	Throughput	`4963fa654018a1f933eda4d2d9be699c647c0f92eaf86ada8bb3215bbcf9a93e`
MXFP4	1	NVIDIA-GB200	No	Throughput	`c1b624be93d9a86d0e6b62477b6027ec82097e148d39a488d1127857e14eeb8e`
MXFP4	1	NVIDIA-GB300	No	Throughput	`8a672528744ff2ef4bede3805c4b3adeb30750792ce62fc3e469b46974cbd5d1`
MXFP4	1	NVIDIA-GH200-144G-HBM3e	No	Throughput	`8155b6bc74d36fe73e2b5f9c49b2163da42a8c9ffe37ac2b641cfeedb368a4c5`
MXFP4	1	NVIDIA-GH200-480GB	No	Throughput	`3fbaeeaeeb0585589dbf520321eff6fda207187555b40879a9740076da3adba9`
MXFP4	1	NVIDIA-H100-80GB-HBM3	No	Throughput	`b23bc5c03183519e99d30f279b03cdc5585c39c9ca0151be65e32c968d37f0fe`
MXFP4	1	NVIDIA-H100-NVL	No	Throughput	`9ded975a10abf76e942c3083d855637c8e510e03a1e6ff576f469da7da1bcbf4`
MXFP4	1	NVIDIA-H200	No	Throughput	`659ef6adf3a2beca85f7aa0833ac1d6ac30936e8e0cf9c39c5e0d5cc56f59b8d`
MXFP4	1	NVIDIA-H200-NVL	No	Throughput	`b75074b54ac63427a553c4d6c3986f92ef4e5ade8d22917ab2f74cc4c93a4852`
MXFP4	1	NVIDIA-RTX-PRO-6000-Blackwell-Server-Edition	No	Throughput	`00f7eb9297a36a9a62f72ee3147a2686b30b2f15334a6e5fd1e6126c49c8aa80`

These throughput profiles use an FP8 KV cache, except the NVIDIA A100-SXM4-80GB profile, which uses a BF16 KV cache and supports a maximum sequence length of 32,768 tokens.

gpt-oss-20b#

The following table lists the supported profile configurations for openai/gpt-oss-20b:

Precision	TP1	TP2	TP4	TP8
MXFP4	`vllm-mxfp4-tp1-pp1`	`vllm-mxfp4-tp2-pp1`	`vllm-mxfp4-tp4-pp1`	`vllm-mxfp4-tp8-pp1`
MXFP4 + LoRA	`vllm-mxfp4-tp1-pp1-lora`	`vllm-mxfp4-tp2-pp1-lora`	`vllm-mxfp4-tp4-pp1-lora`	`vllm-mxfp4-tp8-pp1-lora`

Optimized Deployment Profiles#
Precision	TP	GPU	LoRA	Tuning Objective	Profile ID
MXFP4	1	`NVIDIA-B200`	No	Throughput	`e813ebaa3d51d4fce82fc4902a331ff4e14c07bee717afe7345b0c171ef65fd4`
MXFP4	1	`NVIDIA-H200`	No	Throughput	`89a170a9f739172d9579514b50c9da14dc9a41d0c6690da0f4f7551f90bc5abf`

These throughput profiles use an FP8 KV cache.

llama-3.1-70b-instruct#

The following table lists the supported profile configurations for meta/llama-3.1-70b-instruct:

Precision	TP1	TP2	TP4	TP8
BF16	`vllm-bf16-tp1-pp1`	`vllm-bf16-tp2-pp1`	`vllm-bf16-tp4-pp1`	`vllm-bf16-tp8-pp1`
BF16 + LoRA	`vllm-bf16-tp1-pp1-lora`	`vllm-bf16-tp2-pp1-lora`	`vllm-bf16-tp4-pp1-lora`	`vllm-bf16-tp8-pp1-lora`
FP8	`vllm-fp8-tp1-pp1`	`vllm-fp8-tp2-pp1`	`vllm-fp8-tp4-pp1`	`vllm-fp8-tp8-pp1`
FP8 + LoRA	`vllm-fp8-tp1-pp1-lora`	`vllm-fp8-tp2-pp1-lora`	`vllm-fp8-tp4-pp1-lora`	`vllm-fp8-tp8-pp1-lora`
NVFP4	`vllm-nvfp4-tp1-pp1`	`vllm-nvfp4-tp2-pp1`	`vllm-nvfp4-tp4-pp1`	`vllm-nvfp4-tp8-pp1`
NVFP4 + LoRA	`vllm-nvfp4-tp1-pp1-lora`	`vllm-nvfp4-tp2-pp1-lora`	`vllm-nvfp4-tp4-pp1-lora`	`vllm-nvfp4-tp8-pp1-lora`

Optimized deployment profiles#
Precision	TP	GPU	LoRA	Tuning Objective	Profile ID
NVFP4	2	NVIDIA-B200	No	Throughput	`1e245e34c17f9a671b8fc6ea38d1c354310e5c2a4c967cdb4edfa507c714059b`
FP8	2	NVIDIA-H200	No	Throughput	`3d6b44b1c9ec1402fdb54c3f0c08b4d58f1aa8181a53c7cfb20c70399d45ed1b`

llama-3.1-8b-instruct#

The following table lists the supported profile configurations for meta/llama-3.1-8b-instruct:

Precision	TP1
BF16	`vllm-bf16-tp1-pp1`
BF16 + LoRA	`vllm-bf16-tp1-pp1-lora`
FP8	`vllm-fp8-tp1-pp1`
FP8 + LoRA	`vllm-fp8-tp1-pp1-lora`
NVFP4	`vllm-nvfp4-tp1-pp1`
NVFP4 + LoRA	`vllm-nvfp4-tp1-pp1-lora`

Optimized deployment profiles#
Precision	TP	GPU	LoRA	Tuning Objective	Profile ID
NVFP4	1	NVIDIA-B200	No	Throughput	`ed8b932a6346eb9c2c761c9a456c315d30d822d27d016a0fc0848dcde18b49d2`
FP8	1	NVIDIA-H200	No	Throughput	`2c84d95e1636bb8991b37a34ef4e7f46bf26448883ca133e8aaa2015fbcd0d91`

llama-3.3-70b-instruct#

The following table lists the supported profile configurations for meta/llama-3.3-70b-instruct:

Precision	TP1	TP2	TP4	TP8
BF16	`vllm-bf16-tp1-pp1`	`vllm-bf16-tp2-pp1`	`vllm-bf16-tp4-pp1`	`vllm-bf16-tp8-pp1`
BF16 + LoRA	`vllm-bf16-tp1-pp1-lora`	`vllm-bf16-tp2-pp1-lora`	`vllm-bf16-tp4-pp1-lora`	`vllm-bf16-tp8-pp1-lora`
FP8	`vllm-fp8-tp1-pp1`	`vllm-fp8-tp2-pp1`	`vllm-fp8-tp4-pp1`	`vllm-fp8-tp8-pp1`
FP8 + LoRA	`vllm-fp8-tp1-pp1-lora`	`vllm-fp8-tp2-pp1-lora`	`vllm-fp8-tp4-pp1-lora`	`vllm-fp8-tp8-pp1-lora`
NVFP4	`vllm-nvfp4-tp1-pp1`	`vllm-nvfp4-tp2-pp1`	`vllm-nvfp4-tp4-pp1`	`vllm-nvfp4-tp8-pp1`
NVFP4 + LoRA	`vllm-nvfp4-tp1-pp1-lora`	`vllm-nvfp4-tp2-pp1-lora`	`vllm-nvfp4-tp4-pp1-lora`	`vllm-nvfp4-tp8-pp1-lora`

Optimized deployment profiles#
Precision	TP	GPU	LoRA	Tuning Objective	Profile ID
NVFP4	1	`NVIDIA-B200`	No	Throughput	`7c5d3e8017df3f85cdedbf4bdf3bc85c0ae30c44596a4dbf575aa41651f23a2e`
FP8	2	`NVIDIA-H200`	No	Throughput	`fe149b540c81d3efe4d72aeb4c51e7b79a7b75e3fa1b0d6b441fcf020f932990`

llama-3.3-nemotron-super-49b-v1.5#

The following table lists the supported profile configurations for nvidia/llama-3.3-nemotron-super-49b-v1.5:

Precision	TP1	TP2	TP4	TP8
BF16	`vllm-bf16-tp1-pp1`	`vllm-bf16-tp2-pp1`	`vllm-bf16-tp4-pp1`	`vllm-bf16-tp8-pp1`
BF16 + LoRA	`vllm-bf16-tp1-pp1-lora`	`vllm-bf16-tp2-pp1-lora`	`vllm-bf16-tp4-pp1-lora`	`vllm-bf16-tp8-pp1-lora`
FP8	`vllm-fp8-tp1-pp1`	`vllm-fp8-tp2-pp1`	`vllm-fp8-tp4-pp1`	`vllm-fp8-tp8-pp1`
FP8 + LoRA	`vllm-fp8-tp1-pp1-lora`	`vllm-fp8-tp2-pp1-lora`	`vllm-fp8-tp4-pp1-lora`	`vllm-fp8-tp8-pp1-lora`
NVFP4	`vllm-nvfp4-tp1-pp1`	`vllm-nvfp4-tp2-pp1`	`vllm-nvfp4-tp4-pp1`	`vllm-nvfp4-tp8-pp1`
NVFP4 + LoRA	`vllm-nvfp4-tp1-pp1-lora`	`vllm-nvfp4-tp2-pp1-lora`	`vllm-nvfp4-tp4-pp1-lora`	`vllm-nvfp4-tp8-pp1-lora`

Optimized deployment profiles#
Precision	TP	GPU	LoRA	Tuning Objective	Profile ID
NVFP4	1	`NVIDIA-B200`	No	Throughput	`5e6f4b169c3ff1afca1d9ca987bdc9a15f200148c6ac1a2f21193b2931e16a54`
FP8	2	`NVIDIA-H200`	No	Throughput	`c4e38bc38b14bae462b37c1e7a177456e6cba154742bfcbd7307f4781f4672f8`

nemotron-3-nano#

The following table lists the supported profile configurations for nvidia/nemotron-3-nano:

Precision	TP1	TP2	TP4	TP8
BF16	`vllm-bf16-tp1-pp1`	`vllm-bf16-tp2-pp1`	`vllm-bf16-tp4-pp1`	`vllm-bf16-tp8-pp1`
BF16 + LoRA	`vllm-bf16-tp1-pp1-lora`	`vllm-bf16-tp2-pp1-lora`	`vllm-bf16-tp4-pp1-lora`	`vllm-bf16-tp8-pp1-lora`
FP8	`vllm-fp8-tp1-pp1`	`vllm-fp8-tp2-pp1`	`vllm-fp8-tp4-pp1`	`vllm-fp8-tp8-pp1`
NVFP4	`vllm-nvfp4-tp1-pp1`	`vllm-nvfp4-tp2-pp1`	`vllm-nvfp4-tp4-pp1`	`vllm-nvfp4-tp8-pp1`
NVFP4 + LoRA	`vllm-nvfp4-tp1-pp1-lora`	–	–	–

Optimized deployment profiles#
Precision	TP	GPU	LoRA	Tuning Objective	Profile ID
NVFP4	1	`NVIDIA-B200`	No	Throughput	`c63081e9c5f97966c1d1c9609322f12493ff1e4372aa42a133f54a639c9ccf79`
FP8	1	`NVIDIA-H200`	No	Throughput	`e103eae58d844bd3c6e121904b3d57772f3aec0376b4885240448cd354e9d884`

nemotron-3-super-120b-a12b#

The following table lists the supported profile configurations for nvidia/nemotron-3-super-120b-a12b:

Precision	TP1	TP2	TP4	TP8
BF16	`vllm-bf16-tp1-pp1`	`vllm-bf16-tp2-pp1`	`vllm-bf16-tp4-pp1`	`vllm-bf16-tp8-pp1`
BF16 + LoRA	–	`vllm-bf16-tp2-pp1-lora`	`vllm-bf16-tp4-pp1-lora`	`vllm-bf16-tp8-pp1-lora`
FP8	`vllm-fp8-tp1-pp1`	`vllm-fp8-tp2-pp1`	`vllm-fp8-tp4-pp1`	`vllm-fp8-tp8-pp1`
NVFP4	`vllm-nvfp4-tp1-pp1`	`vllm-nvfp4-tp2-pp1`	`vllm-nvfp4-tp4-pp1`	`vllm-nvfp4-tp8-pp1`
NVFP4 + LoRA	`vllm-nvfp4-tp1-pp1-lora`	`vllm-nvfp4-tp2-pp1-lora`	–	–

Optimized deployment profiles#
Precision	TP	GPU	LoRA	Tuning Objective	Profile ID
NVFP4	4	`NVIDIA-B200`	No	Throughput	`6bd9e1c75425eeee3f8e0ebe562409116f3bd728e668c11c9d175a176a2da95e`
FP8	4	`NVIDIA-H200`	No	Throughput	`0e13d08b10121c8c2e41f03f6eb67fdf6f9261ba7e5a30e16f79709ff3622c6f`

Note

This is a large model. Lower-TP profiles require substantially more GPU memory per device, so some verified GPUs support only TP4 or TP8 profiles.

nemotron-3-ultra-550b-a55b#

The following table lists the supported profile configurations for nvidia/nemotron-3-ultra-550b-a55b:

Precision	TP1	TP2	TP4	TP8
BF16	–	–	–	`vllm-bf16-tp8-pp1`
BF16 + LoRA	–	–	–	–
NVFP4	–	`vllm-nvfp4-tp2-pp1`	`vllm-nvfp4-tp4-pp1`	`vllm-nvfp4-tp8-pp1`

Optimized deployment profiles#
Precision	TP	GPU	LoRA	Tuning Objective	Profile ID
NVFP4	4	`NVIDIA-B200`	No	Throughput	`9d06a0e06444481c655310f7521afda72b34916d99427c3264c99c081aa2be5c`
NVFP4	4	`NVIDIA-B300-SXM6-AC`	No	Throughput	`b0ec375f790e252f66961b4825b6a8918a93276e720c131d1ed1e786737523f3`
NVFP4	4	`NVIDIA-GB200`	No	Throughput	`f683567848cba9f9bb5962c35a163dc0935e1477b7ad3e86cf78576e34e9f717`
NVFP4	4	`NVIDIA-GB300`	No	Throughput	`c99f9b66e77ea04239c6888bd82fc4442820d9879f1fa1ebb28cb6a923468410`

Note

The minimum deployment is two GPUs, using vllm-nvfp4-tp2-pp1 on high-memory Blackwell SKUs (NVIDIA-B300-SXM6-AC and NVIDIA-GB300); the other profiles use four or eight GPUs. H200 validation covers one NVFP4 TP4 profile and is less extensive than the Blackwell validation.

FP8 profiles were not released for this NIM because NVFP4 profiles work on all tested Hopper-architecture GPUs (for example, H100 and H200) and deliver better performance than FP8 profiles.

nemotron-3.5-lightning#

The following table lists the supported profile configurations for nvidia/nemotron-3.5-lightning-30b-a3b, along with the minimum GPU VRAM required per device and the GPU architecture that each precision requires. The VRAM values come from the profile disk-sizing metadata reported by list-model-profiles and represent the minimum per-GPU memory needed to load the model weights and the runtime allocations; they do not include additional headroom for large KV caches at high context length or high concurrency.

Nemotron 3.5 Lightning Profile Information#
Precision	TP	PP	Min VRAM per GPU	Min GPU count	Architecture requirement	LoRA Profile
BF16	1	1	66 GB	1	Ampere or newer (SM 8.0+)	Base and LoRA
BF16	2	1	35 GB	2	Ampere or newer (SM 8.0+)	Base and LoRA
BF16	4	1	20 GB	4	Ampere or newer (SM 8.0+)	Base and LoRA
BF16	8	1	12 GB	8	Ampere or newer (SM 8.0+)	Base and LoRA
W4A16	1	1	32 GB	1	Ampere or newer (SM 8.0+)	Base and LoRA
W4A16	2	1	20 GB	2	Ampere or newer (SM 8.0+)	Base and LoRA
W4A16	4	1	14 GB	4	Ampere or newer (SM 8.0+)	Base and LoRA
NVFP4	1	1	30 GB	1	Blackwell or newer (SM 10.0+)	Base and LoRA
NVFP4	2	1	18 GB	2	Blackwell or newer (SM 10.0+)	Base and LoRA
NVFP4	4	1	12 GB	4	Blackwell or newer (SM 10.0+)	Base and LoRA

Optimized deployment profiles#
Precision	TP	GPU	LoRA	Tuning Objective	Profile ID
BF16	1	`NVIDIA-B200`	No	Throughput	`0307da6933369f40342ccb0880d07c7eb00ab0e7c6a0f8a7ea2a39a9b4b8f575`
BF16	1	`NVIDIA-H200`	No	Throughput	`a92b5edd42489d5720b6f479a5c895d36eaccb2610ba8f1ee41c9357f48b0ec7`

Notes:

Min VRAM per GPU is the floor that list-model-profiles reports for that precision and tensor-parallel size. A GPU below the floor is filtered out during profile selection.
Architecture requirement in the generic profile table reflects the kernels those profiles need. The generic NVFP4 profiles require Streaming Multiprocessor (SM) 10.0+ (Blackwell) because they use NVFP4-native tensor cores. BF16 and W4A16 run on any Ampere-class or newer GPU (SM 8.0+).
Sharing weights across more GPUs (higher TP) lowers the per-GPU VRAM floor and enables lower-memory GPUs.
Model-cache disk sizes range from approximately 19 GB (NVFP4) to approximately 63 GB (BF16). Cache size is separate from GPU VRAM and does not indicate that the model will fit in a given GPU.

Verified GPUs

The following tables list verified GPUs for the generic profiles, grouped by architecture:

Blackwell (SM 10.0) — supports NVFP4, W4A16, BF16:

GPU	VRAM per GPU	Profiles that fit
`NVIDIA-B200`	180 GB	All profiles except BF16 TP8
`NVIDIA-B300-SXM6-AC`	288 GB	All profiles except BF16 TP8
`NVIDIA-GB200`	186 GB	All profiles except BF16 TP8
`NVIDIA-GB300`	288 GB	All profiles except BF16 TP8
`NVIDIA-GB10`	128 GB (unified memory)	TP1 profiles (BF16, W4A16, and NVFP4); set `NIM_KVCACHE_PERCENT=0.75` on GB10 (refer to the GPU memory troubleshooting guide)
`NVIDIA-RTX-PRO-6000-Blackwell-Server-Edition`	96 GB	All profiles

Hopper (SM 9.0) — supports W4A16 and BF16 (no NVFP4):

GPU	VRAM per GPU	Profiles that fit
`NVIDIA-H100-80GB-HBM3`	80 GB	BF16 all sizes, W4A16 all sizes
`NVIDIA-H100-NVL`	94 GB	BF16 TP1, TP2, and TP4; W4A16 all sizes
`NVIDIA-H200`	141 GB	BF16 TP1, TP2, and TP4; W4A16 all sizes

Ampere (SM 8.0) — supports W4A16 and BF16 (no NVFP4):

GPU	VRAM per GPU	Profiles that fit
`NVIDIA-A100-SXM4-80GB`	80 GB	BF16 all sizes, W4A16 all sizes

Ada (SM 8.9) — supports W4A16 and BF16 (no NVFP4):

GPU	VRAM per GPU	Profiles that fit
`NVIDIA-L40S`	48 GB	BF16 TP≥2, W4A16 all sizes

Unverified but potentially compatible. Any GPU that meets the per-profile minimum VRAM floor and the architecture requirement in the profile table can run that profile technically, even if it is not in the preceding Verified GPUs tables. Only the GPUs in those tables have been tested end-to-end for this model at this NIM version; untested systems may exhibit different startup, throughput, or accuracy behavior.

riva-translate-4b-instruct-v2#

The following table lists the supported profile configuration for nvidia/riva-translate-4b-instruct-v2.

The latest release version of this NIM is 2.0.8.

Precision	TP	PP	LoRA	Profile
BF16	1	1	No	`vllm-bf16-tp1-pp1`

starcoder2-7b#

The following table lists the supported profile configurations for bigcode/starcoder2-7b:

Precision	TP1	TP2
BF16	`vllm-bf16-tp1-pp1`	`vllm-bf16-tp2-pp1`

Optimized deployment profiles#
Precision	TP	GPU	LoRA	Tuning Objective	Profile ID
BF16	1	`NVIDIA-H200`	No	Throughput	`7be3b5f2958163bd55fcd9da813e83f5a297980c4f06741e69c3231ca321b9a3`

VLMs#

The following section lists all of the supported VLMs.

deepseek-v4.1-flash#

The following table lists the supported profile configurations for deepseek-ai/deepseek-v4.1-flash.

This NIM uses the vLLM 0.31.0 inference backend.

The optimized FP8 profiles on NVIDIA B200, GB200, and H200, which the container selects by default on those GPUs, accept text input only; to send images on those GPUs, select the NVFP4 profile by its full profile ID. The NVFP4 profiles on NVIDIA H100, H100 NVL, H200, and H200 NVL run with weight-only FP4 emulation. Combined container and model disk space is approximately 525 GB for FP8 and 540 GB for NVFP4. TP4 profiles need at least 123 GB (FP8) or 124 GB (NVFP4) of memory per GPU; TP8 profiles need at least 64 GB per GPU. Every profile also needs at least 384 GB of host memory: the model keeps its Engram embedding tables in pinned host memory, and peak host-memory use was about 355 GB in testing. Refer to DeepSeek-V4.1-Flash deployment considerations for startup settings and reasoning controls.

Profiles and supported input by GPU#
GPU	Precision	Input	Profile ID
`NVIDIA-B200`	FP8 (optimized, default)	Text	`06fcf7c1ee59efbe08a000038172357db612b3f00d35a0f293f7691f93cf580f`
`NVIDIA-B200`	NVFP4	Text and image	`b2dd478c2227a88e8135f8b621945650ab5ddaf318e5a13b8844507d2d43acce`
`NVIDIA-GB200`	FP8 (optimized, default)	Text	`33a4f33df4a3e23afca9aa08794e1c096903c164ab12062747edb584842a5aec`
`NVIDIA-GB200`	NVFP4	Text and image	`44fb6adf8c2101e9a51cff52c86c3d5f6888836c549927c9d051bb23778cee28`
`NVIDIA-H200`	FP8 (optimized, default)	Text	`9f34efd50f49c2ff891db2deb4d51a666580bbbf4053d59e0a3ca7a0c33c2cf6`
`NVIDIA-H200`	NVFP4	Text and image	`30d2c978b8a1a2271e38f636de7402172262ef3a30f79d1c09b2991512cde916`
`NVIDIA-B300-SXM6-AC`	FP8	Text and image	`c980256e07d167c33ad03cbc35719d2ac31d17c5b4a836c755b864423d18f148`
`NVIDIA-B300-SXM6-AC`	NVFP4	Text and image	`ec894d10bc186b37ccc51567befa07edd148ef8cf756a1d03d2895d53973143d`
`NVIDIA-GB300`	FP8	Text and image	`4168ffef5b76dbd61d14bb409a21dfb94d169c0442fe725f023fc481e1029f79`
`NVIDIA-GB300`	NVFP4	Text and image	`365b393561330a88be49370ab66f8a151ebeb6ad78e2526fced8a5a70a5eae2b`
`NVIDIA-H200-NVL`	FP8	Text and image	`47fc20c35c9a8230c84a26573a1c0d364214dbfc76e9a3c24501bfc42cb171b0`
`NVIDIA-H200-NVL`	NVFP4	Text and image	`ef1a0084673ceb98d6581aeb89ffa02ca4e0432bae1f058a8472f9c4579daa2f`
`NVIDIA-H100-80GB-HBM3`	FP8	Text and image	`915fa4e844359866d56f88534b6281c70d84f53a7e93dbeea8f09af3690d820e`
`NVIDIA-H100-80GB-HBM3`	NVFP4	Text and image	`765680638787e9b733ec7c139beb68d70ac31cfbe7f5656708754162920e90a7`
`NVIDIA-H100-NVL`	FP8	Text and image	`45266af5d2b4e172a9cd6ba6f3efe1db1a2b04bfac184c414a05180b373da7e6`
`NVIDIA-H100-NVL`	NVFP4	Text and image	`2485f6bf339957f15baa4e60c8d20d3401f9d0222e4c807bb2a7c87706462e95`

Optimized deployment profiles#
Precision	TP	GPU	LoRA	Tuning Objective	Profile ID
FP8	4	`NVIDIA-B200`	No	Throughput	`06fcf7c1ee59efbe08a000038172357db612b3f00d35a0f293f7691f93cf580f`
FP8	4	`NVIDIA-GB200`	No	Throughput	`33a4f33df4a3e23afca9aa08794e1c096903c164ab12062747edb584842a5aec`
FP8	4	`NVIDIA-H200`	No	Throughput	`9f34efd50f49c2ff891db2deb4d51a666580bbbf4053d59e0a3ca7a0c33c2cf6`

gemma-4-26b-a4b-it#

The following table lists the supported profile configurations for google/gemma-4-26b-a4b-it:

Precision	TP1	TP2	TP4	TP8
NVFP4	`vllm-nvfp4-tp1-pp1`	–	–	–
BF16	`vllm-bf16-tp1-pp1`	`vllm-bf16-tp2-pp1`	`vllm-bf16-tp4-pp1`	–

Optimized deployment profiles#
Precision	TP	GPU	LoRA	Tuning Objective	Profile ID
NVFP4	1	NVIDIA-B200	No	Throughput	`935eb76859c7e607a33f300b2b001e019c39e9dc02c4d2549faf143ae594d904`
BF16	1	NVIDIA-H200	No	Throughput	`8d6ff1969c18d7faaf1dc76ecea6589dcd8cd0e62208e040904f8e2c7cb01bc3`
BF16	4	NVIDIA-A10G	No	Context length	`800f3c24e5bdaca88aebd04242be7765c65ac7abbfc9013c404d0ea0ddc9f6fc`

gemma-4-31b-it#

The following table lists the supported profile configurations for google/gemma-4-31b-it:

Precision	TP1	TP2	TP4	TP8
NVFP4	`vllm-nvfp4-tp1-pp1`	`vllm-nvfp4-tp2-pp1`	`vllm-nvfp4-tp4-pp1`	–
BF16	`vllm-bf16-tp1-pp1`	`vllm-bf16-tp2-pp1`	`vllm-bf16-tp4-pp1`	–
INT4	–	`vllm-int4-tp2-pp1`	–	–

Optimized deployment profiles#
Precision	TP	GPU	LoRA	Tuning Objective	Profile ID
NVFP4	1	NVIDIA-B200, NVIDIA-B300-SXM6-AC, NVIDIA-GB200, NVIDIA-GB300, NVIDIA-RTX-PRO-6000-Blackwell-Server-Edition	No	Throughput	`5f2613453901ce34041f9390d1eb035bceabba0af65af73be562fa5cda704a48`

Inkling#

The following table lists the supported profile configurations for thinkingmachines/inkling. This model-specific container uses SGLang as the inference backend.

Inkling Profiles#
Profile	GPUs	Precision	TP	PP	LoRA
Generic NVFP4 TP4	B300, GB300	NVFP4	4	1	No
Generic NVFP4 TP4 + LoRA	B300, GB300	NVFP4	4	1	Yes
Generic NVFP4 TP8	B200, B300	NVFP4	8	1	No
Generic NVFP4 TP8 + LoRA	B200, B300	NVFP4	8	1	Yes
H200 NVFP4 TP8	H200	NVFP4	8	1	No
H200 NVFP4 TP8 + LoRA	H200	NVFP4	8	1	Yes

Optimized Deployment Profiles#
Precision	TP	GPU	LoRA	Tuning Objective	Profile ID
NVFP4	8	`NVIDIA-B200`	No	Throughput	`8502d8a44bc24573a17d5e1b20381cc2c74669280ddb9cc3fde765cf15bbb27a`
NVFP4	8	`NVIDIA-H200`	No	Throughput	`4a30cb634a0f720cc28572ec4592876d255762f2bfe7bf0d2dceff6890f1f5f4`

The H200 profiles run the NVFP4 weights with Marlin kernels. Set NIM_ALLOW_NVFP4_EMULATION=1 at runtime on H200.

inkling-small#

The following table lists the supported profile configurations for thinkingmachines/inkling-small.

For deployment requirements and request behavior, refer to the deployment considerations.

The container sets NIM_ALLOW_NVFP4_EMULATION=1 by default. All NVFP4 profiles on Hopper GPUs (H100, H100 NVL, H200, and H200 NVL) run through NVFP4 emulation because these GPUs do not have native FP4 tensor cores. Do not unset this variable on Hopper GPUs. If you unset it, the NVFP4 profiles are not selected.

Precision	TP1	TP2	TP4	TP8
NVFP4	–	`vllm-nvfp4-tp2-pp1`	`vllm-nvfp4-tp4-pp1`	`vllm-nvfp4-tp8-pp1`
BF16	–	–	–	`vllm-bf16-tp8-pp1`

Optimized deployment profiles#
Precision	TP	GPU	LoRA	Tuning Objective	Profile ID
NVFP4	2	`NVIDIA-B200`	No	Throughput	`f4b90e7699b1168b63324a959f83e5fe10dfaa03ad7dc231e5ae8659b05a0d2e`
NVFP4	4	`NVIDIA-H200`	No	Throughput	`55e3ae57f0e1a5f34e7d737340a6e1f7565678cb0e4481b1f1605ee5cc5021d2`

kimi-k2.6#

The following table lists the supported profile configurations for moonshotai/kimi-k2.6:

Precision	TP1	TP2	TP4	TP8
NVFP4	–	–	`vllm-nvfp4-tp4-pp1`	–
INT4	–	–	–	`vllm-int4-tp8-pp1`

Optimized deployment profiles#
Precision	TP	GPU	LoRA	Tuning Objective	Profile ID
NVFP4	4	`NVIDIA-B200`	No	Throughput	`b197f6a800db21269b7388d2ea7d82cb0e66ed5c66d3278d7049a52c687f86fa`
NVFP4	4	`NVIDIA-B300-SXM6-AC`	No	Throughput	`3f1e102185aade2a7df51fa32bd8e56667db92694f374a51ac450247328bcc0f`
NVFP4	4	`NVIDIA-GB200`	No	Throughput	`25868c0d2f0529d801a9e5aea19d16ae15cbed8e7d065dfcdbf8cdc2fd451d86`
NVFP4	4	`NVIDIA-GB300`	No	Throughput	`8ca983f7aca5cf99befffea021267b4f01fe023b884b77af747213047a642fe4`
INT4	8	`NVIDIA-H200`	No	Throughput	`0efe146262f2491884dd56d8ba5666dada0a27c08706dda4a1812a0292c1b637`

mistral-small-4-119b-2603#

The following table lists the supported profile configurations for mistralai/mistral-small-4-119b-2603:

Precision	TP1	TP2	TP4	TP8
FP8	`vllm-fp8-tp1-pp1`	`vllm-fp8-tp2-pp1`	`vllm-fp8-tp4-pp1`	`vllm-fp8-tp8-pp1`
NVFP4	`vllm-nvfp4-tp1-pp1`	`vllm-nvfp4-tp2-pp1`	`vllm-nvfp4-tp4-pp1`	`vllm-nvfp4-tp8-pp1`

Optimized Deployment Profiles#
Precision	TP	GPU	LoRA	Tuning Objective	Profile ID
NVFP4	2	`NVIDIA-B200`	No	Throughput	`cc3a82b7f3fcacbec1b393a4b30b72b6b92acf5064f8414e0861156f1c3211d6`

This throughput profile uses an FP8 KV cache.

muse-glimmer#

The following table lists the supported profile configurations for meta/muse-glimmer:

This model reasons before it answers and requires the cache-permission step during deployment. Refer to Muse Glimmer deployment considerations before you send your first request.

Precision	TP1	TP2	TP4	TP8
BF16	`sglang-bf16-tp1-pp1`	`sglang-bf16-tp2-pp1`	–	–
NVFP4	`sglang-nvfp4-tp1-pp1`	`sglang-nvfp4-tp2-pp1`	–	–

nemotron-3-nano-omni-30b-a3b-reasoning#

The following table lists the supported profile configurations for nvidia/nemotron-3-nano-omni-30b-a3b-reasoning:

Precision	TP1	TP2	TP4	TP8
BF16	–	`vllm-bf16-tp2-pp1`	–	–
FP8	`vllm-fp8-tp1-pp1`	–	–	–
NVFP4	`vllm-nvfp4-tp1-pp1`	–	–	–

Optimized deployment profiles#
Precision	TP	GPU	LoRA	Tuning Objective	Profile ID
NVFP4	1	`NVIDIA-B200`	No	Throughput	`d76adb5807f1d20f3bed16c4ecad7400db23a9d2d79f589ba7806b909be12ef7`
NVFP4	1	`NVIDIA-B200`	No	Throughput, image-text	`a5484ae50a1d699866f66644e14dc3549b4839ff8d6d30af978267cba2ee0773`
NVFP4	1	`NVIDIA-B200`	No	Throughput, video-text	`48503978c1b961d38cbedb15325ef69bf4a4604108fec270da31d6a92283a9fb`
NVFP4	1	`NVIDIA-B300-SXM6-AC`	No	Throughput	`694ce0ad42092cd93fe5da6c4df45a89e44f1ad10f62ebc611f952fe7a503427`
NVFP4	1	`NVIDIA-GB300`	No	Throughput	`6d8be2b4e5e3085910d47a76efc119a6b9e5404d3d29048a90aa8ab29e5307ed`
NVFP4	1	`NVIDIA-GB300`	No	Throughput, image-text	`6ecb87e0677aac84fee2edbcd01ff2f816c9b071759e524e3c0ce2990e1fde65`
NVFP4	1	`NVIDIA-GB300`	No	Throughput, video-text	`717d9f0408541da3b1fb66eab1d391d018a8dd0e9cdf1a7023d5b0fa9f1ac746`
FP8	1	`NVIDIA-H200`	No	Throughput	`6a4db7e22aaacca0e340d328880d2467d4b96b027943cb3a06bf6ac1bcf55568`
FP8	1	`NVIDIA-H200`	No	Throughput, image-text	`e16c3c9f6c4c472da7c1b9e3c0754851fcd7c50a0948b51c995d7e486d212498`
FP8	1	`NVIDIA-H200`	No	Throughput, video-text	`377d945adf31f0931809c631951408ec1f34cd55b74b711baa494b897f0d864c`
NVFP4	1	`NVIDIA-RTX-PRO-4500-Blackwell-Server-Edition`	No	Throughput	`642f356201892f339146d30c52e92ada94de23ac0914583dae769f3f15b84c51`
NVFP4	1	`NVIDIA-RTX-PRO-6000-Blackwell-Server-Edition`	No	Throughput	`c9a18cd08813cf8c7a390af9364d5084a6bf66fec2fac2e5275187d01871ab2c`

nemotron-3.5-bolt-e4b#

The following table lists the supported GPU configurations for nvidia/nemotron-3.5-bolt-e4b-4gb. For model-specific settings, refer to Model-Specific Deployment Considerations (2.1.4).

Nemotron 3.5 Bolt E4B 4GB Profile Information#
GPU	Precision	TP	LoRA
GB10 (DGX Spark)	BF16	1	No
RTX-PRO-6000-Blackwell-Server-Edition	BF16	1	No
RTX-PRO-4500-Blackwell-Server-Edition	BF16	1	No
H100-80GB-HBM3	BF16	1	No

nemotron-3.5-content-safety#

The following table lists the supported profile configurations for nvidia/nemotron-3.5-content-safety.

Deployment profiles#
Precision	TP	GPU	LoRA	Tuning Objective	Profile ID
BF16	1	`NVIDIA-A100-SXM4-40GB`	No	Not specified	`ecc0c366dd8591e092b9d521ca26240c8f16119fcc5c3b5f20d1ccb2e08ec843`
BF16	1	`NVIDIA-A100-SXM4-80GB`	No	Not specified	`8439d9c09f0dd1e09778da2360522b71018f7cdbd43beb1205f15af1722323d9`
BF16	1	`NVIDIA-A10G`	No	Not specified	`79633375948d1847d35dc5cecf6e405059ee8f675db426898ae8646aaa1d3544`
BF16	1	`NVIDIA-B200`	No	Not specified	`4bd4088817c26b9ae361d43142053984fd8d9b2d170d332eeb1ee1c7f1256a23`
BF16	1	`NVIDIA-B200`	No	Throughput (FP8 KV cache)	`17c838239088a3ca457d492e32bac689d74981810ed17d17ec0611f74cffeb72`
BF16	1	`NVIDIA-B300-SXM6-AC`	No	Not specified	`116d109f484cac44a0644504c2f752ed7c6a60be907941cad17348396625530f`
BF16	1	`NVIDIA-GB10`	No	Not specified	`e67cca9a7077dc626b32ad05fb4ea1f67f35336ee6e123a733a6487461fc3759`
BF16	1	`NVIDIA-GB200`	No	Not specified	`32c8887258be660a21ee0413786e83282a5ae84635143b997aa79d5150154c91`
BF16	1	`NVIDIA-GB300`	No	Not specified	`df5adf8ebbb6a5f1c787f0eb6d34182ae4409c90fc5f0ea1bdb46352f1423770`
BF16	1	`NVIDIA-GH200-144G-HBM3e`	No	Not specified	`6e76fd300bcb71e4094a74a483f34f70b9c7c2d4047db93137b16c33a5c8f413`
BF16	1	`NVIDIA-GH200-480GB`	No	Not specified	`dc1503ed2b91153c27cd18a3b9cef86b914727439f99a22eeee83a58d8987138`
BF16	1	`NVIDIA-H100-80GB-HBM3`	No	Not specified	`140504fa00018adc1b69c743d02d7efc60dd5bb19e0ea884dbbab8ad565619bc`
BF16	1	`NVIDIA-H100-NVL`	No	Not specified	`c099aa54cae529bd0193c055a5f0081fb6eed98cefb699b92e8a7900f61c1047`
BF16	1	`NVIDIA-H200`	No	Not specified	`83f5026075c8d36f2ca06265dea5dad380f83cc7b8d0e855f5cc4937b524497f`
BF16	1	`NVIDIA-H200`	No	Throughput	`8f712379edec107c17214af72fd5ed1cd2ecc03faae268431a1b61fec4503960`
BF16	1	`NVIDIA-H200-NVL`	No	Not specified	`667cd60dc55899f995767bfa861f4fa6109026776928926b0ba0074ff91ed97e`
BF16	1	`NVIDIA-L40S`	No	Not specified	`1163190b38671e02d20347774f7ec9aa0382d0a1a85029a30d6d484416ea881c`
BF16	1	`NVIDIA-RTX-PRO-4500-Blackwell-Server-Edition`	No	Not specified	`343b3918e9f57b0c1f4cbf2a1e8ae0577d68314e66b0293408247164f3f23fd2`
BF16	1	`NVIDIA-RTX-PRO-6000-Blackwell-Server-Edition`	No	Not specified	`38c75411586262bcd8d4bcff2d84c4941b3375b9e89efcbc1bd21efbfc339140`

nemotron-3.5-super-vl-120b-a12b#

The following table lists the supported GPU configurations for nvidia/nemotron-3.5-super-vl-120b-a12b. For more information, refer to the dedicated Get Started.

Optimized deployment profiles#
Precision	TP	GPU	LoRA	Tuning Objective	Profile Name	Profile ID	Min VRAM per GPU
BF16	2	NVIDIA-B200	Yes	General	`vllm-nvidia-b200-bf16-tp2-pp1-feat_lora-throughput-0.0`	`f601f9963fcc4bed0469b15df33ec438c2cf55181fb1febdae30fab436369134`	119 GB
BF16	2	NVIDIA-B300-SXM6-AC	Yes	General	`vllm-nvidia-b300-sxm6-ac-bf16-tp2-pp1-feat_lora-throughput-0.0`	`12e76812d9e15427ce3a63523778b7bd15bec8dcaefe9817c32fc19a105fffae`	119 GB
BF16	2	NVIDIA-GB200	Yes	General	`vllm-nvidia-gb200-bf16-tp2-pp1-feat_lora-throughput-0.0`	`f3dcd36059120231b6fb8a00682af9a36f220dc981ca73a8573bf5aeb5faf85c`	119 GB
BF16	2	NVIDIA-GB300	Yes	General	`vllm-nvidia-gb300-bf16-tp2-pp1-feat_lora-throughput-0.0`	`7535e487104e591561705169ece00abad91b1c26255aa844c6456a5b18fc3413`	119 GB
BF16	4	NVIDIA-H200	Yes	General	`vllm-nvidia-h200-bf16-tp4-pp1-feat_lora-throughput-0.0`	`29224c376e8bd48f546d1f783a6da93662401524a972c881d3355a00a399da37`	61 GB
BF16	8	NVIDIA-H100-80GB-HBM3	Yes	General	`vllm-nvidia-h100-80gb-hbm3-bf16-tp8-pp1-feat_lora-throughput-0.0`	`7f85f2d15f243fa05be912e610b31b19a376f8509dd12eb8a3bc3b367e8b6a9f`	32 GB
NVFP4	1	NVIDIA-B200	No	agentX	`vllm-nvidia-b200-nvfp4-tp1-pp1-throughput-0.0-agentx`	`35320c4542dbf3e4ad098dd7669cb3330f728f86a62bdf91eeb4bd34768346f1`	72 GB
NVFP4	1	NVIDIA-B300-SXM6-AC	No	agentX	`vllm-nvidia-b300-sxm6-ac-nvfp4-tp1-pp1-throughput-0.0-agentx`	`cc452f7f9da6c7f902014d7e79324a48f8202c12648d42bced69bea9217a81d5`	72 GB
NVFP4	1	NVIDIA-GB10	No	General	`vllm-nvidia-gb10-nvfp4-tp1-pp1-throughput-0.0`	`6eb52c8e38c174694d551f314ed8b2021a4b3f767d3616da1754dde0916b855c`	72 GB
NVFP4	1	NVIDIA-GB200	No	50k/2k/1k	`vllm-nvidia-gb200-nvfp4-tp1-pp1-throughput-0.0-50k-2k-1k`	`8c840bb323e03589e50b8c97090e059017a35546bbf3b11caec5a1dc5dd334de`	72 GB
NVFP4	1	NVIDIA-GB200	No	agentX	`vllm-nvidia-gb200-nvfp4-tp1-pp1-throughput-0.0-agentx`	`d4a164dc936edc781bf577e419a442e4b1712c2d3a17a3ea33b080b162018b2a`	72 GB
NVFP4	1	NVIDIA-GB300	No	agentX	`vllm-nvidia-gb300-nvfp4-tp1-pp1-throughput-0.0-agentx`	`26d33ae17bd36e569c9348703f9950a37ffd050b01656292540e2daa35e801b9`	72 GB
NVFP4	1	NVIDIA-GB300-WS	No	General	`vllm-nvidia-gb300-ws-nvfp4-tp1-pp1-throughput-0.0`	`5f9194217b5b7856869ae716b6802ec506b6cff1f109a7f1c5e828cb8f105c59`	72 GB
NVFP4	1	NVIDIA-H200	No	50k/2k/1k	`vllm-nvidia-h200-nvfp4-emulated-tp1-pp1-throughput-0.0-50k-2k-1k`	`f335b46bb53b1d7a32ed08d093fc93a26145348530dd7aa3b63d578bfd4cfdf9`	72 GB
NVFP4	1	NVIDIA-H200	No	agentX	`vllm-nvidia-h200-nvfp4-emulated-tp1-pp1-throughput-0.0-agentx`	`65ec457ca53207b0fa52748a8eeb5e47f58b504f7a005f9ce8f815fec94a8f81`	72 GB
NVFP4	2	NVIDIA-A100-SXM4-80GB	No	General	`vllm-nvidia-a100-sxm4-80gb-nvfp4-emulated-tp2-pp1-throughput-0.0`	`737c816b8ac7e1c3c51c871768875600635c9a7939e3e81ed48b6a0a17df29f3`	38 GB
NVFP4	2	NVIDIA-H100-80GB-HBM3	No	50k/2k/1k	`vllm-nvidia-h100-80gb-hbm3-nvfp4-emulated-tp2-pp1-throughput-0.0-50k-2k-1k`	`052ee51a97716b2cf070fe268ca0943947e8ff814d331ba52f08d6d8a1e8a20d`	38 GB
NVFP4	2	NVIDIA-H100-80GB-HBM3	No	agentX	`vllm-nvidia-h100-80gb-hbm3-nvfp4-emulated-tp2-pp1-throughput-0.0-agentx`	`ab990383587f1f1f9fb3b592895529bc37c34a5a3d6aaf055f07eebe0c865631`	38 GB
NVFP4	2	NVIDIA-RTX-PRO-6000-Blackwell-Server-Edition	No	General	`vllm-nvidia-rtx-pro-6000-blackwell-server-edition-nvfp4-tp2-pp1-throughput-0.0`	`da5a5f56a2faa2c54d854046c1374f913bdf829cca30b6406c433f98d472eb21`	38 GB

Min VRAM per GPU is the minimum per-GPU memory needed to load the model weights and the runtime allocations at that precision and tensor-parallel size, as computed by NIM LLM and VLM at startup. It does not include headroom for the KV cache at long context length or high concurrency. Each profile is built for the listed GPU model and GPU count, and there is no generic fallback profile. The min_vram_per_device_gb value that list-model-profiles reports for these profiles is not a memory requirement.

Profile Name is the name that list-model-profiles prints next to the profile ID. NIM_MODEL_PROFILE accepts either the name or the ID.

Tuning objectives:

50k/2k/1k: multi-turn conversations that share a 50K-token system prompt, with about 2K new input tokens and 1K output tokens per turn
agentX: multi-turn coding-agent sessions with tool calls and long, growing contexts of up to 1M tokens
General: the only profile for that GPU and precision

nemotron-parse-v2.0#

The following table lists the supported deployment profiles for nvidia/nemotron-parse-v2.0.

Deployment profiles#
Precision	TP	GPU	LoRA	Tuning Objective	Profile ID
BF16	1	`NVIDIA-A100-SXM4-40GB`	No	Not specified	`02ae39e0a84ecb628e3ae2f24f3104e398ac3f02c3eb309ba77430e6aeb57353`
BF16	1	`NVIDIA-A100-SXM4-80GB`	No	Not specified	`7b8a6f2aa12b9c06640bdb7653aa4873cbb75096b86fe623e02c52fd8110bef5`
BF16	1	`NVIDIA-A10G`	No	Not specified	`5a38a4e1fba6f8b7be13ee2839625d188b1eb0e2a83158dc76a96d786785102b`
BF16	1	`NVIDIA-B200`	No	Not specified	`31077528988f89bca85093d60e8e87f03a118db235f33b3c40474e9d46b81c26`
BF16	1	`NVIDIA-B200`	No	Throughput (FP8 KV cache)	`df9fbe0520a0f6b97dd45964cdbeae90679bfab547052a494247b9c95de38f98`
BF16	1	`NVIDIA-GB200`	No	Not specified	`e9d12ece6ea1cebd9a2aeb4c1506d6ca2d1e3fb8b3e23274ae84d2e1b91f1fef`
BF16	1	`NVIDIA-H100-80GB-HBM3`	No	Not specified	`903be1ea8ffc844bde52490f25ac854752d3144328948367bec6df3f3e7a590a`
BF16	1	`NVIDIA-H200`	No	Not specified	`64b7fb5aae787037fe382e5f4a0c8909872b9362e815b70821f70dafd6c8375e`
BF16	1	`NVIDIA-L40S`	No	Not specified	`101fd5742d4ce4e0de5ec992bf6cf04da69dda99598984a97e14b2b237e7f869`
BF16	1	`NVIDIA-RTX-PRO-6000-Blackwell-Server-Edition`	No	Not specified	`bc69e89091fdf5e99412ab57b32f1b714839f3404746f50b43109821f5e49182`
BF16	1	`NVIDIA-RTX-PRO-6000-Blackwell-Workstation-Edition`	No	Not specified	`f633ff89387127be8b44b5d53183d4b5fc471115e87fc1f9e2fd724e66428675`

The B200 throughput profile uses an FP8 KV cache while retaining BF16 weights. The remaining profiles do not specify a tuning objective.

qwen3.5-122b-a10b#

The following table lists the supported profile configurations for qwen/qwen3.5-122b-a10b:

Precision	TP1	TP2	TP4	TP8
BF16	`vllm-bf16-tp1-pp1`	`vllm-bf16-tp2-pp1`	`vllm-bf16-tp4-pp1`	`vllm-bf16-tp8-pp1`
BF16 + LoRA	–	`vllm-bf16-tp2-pp1-lora`	`vllm-bf16-tp4-pp1-lora`	`vllm-bf16-tp8-pp1-lora`
FP8	`vllm-fp8-tp1-pp1`	`vllm-fp8-tp2-pp1`	`vllm-fp8-tp4-pp1`	`vllm-fp8-tp8-pp1`
FP8 + LoRA	`vllm-fp8-tp1-pp1-lora`	`vllm-fp8-tp2-pp1-lora`	`vllm-fp8-tp4-pp1-lora`	`vllm-fp8-tp8-pp1-lora`
NVFP4	`vllm-nvfp4-tp1-pp1`	`vllm-nvfp4-tp2-pp1`	`vllm-nvfp4-tp4-pp1`	`vllm-nvfp4-tp8-pp1`
NVFP4 + LoRA	`vllm-nvfp4-tp1-pp1-lora`	`vllm-nvfp4-tp2-pp1-lora`	`vllm-nvfp4-tp4-pp1-lora`	`vllm-nvfp4-tp8-pp1-lora`

Optimized deployment profiles#
Precision	TP	GPU	LoRA	Tuning Objective	Profile ID
NVFP4	1	`NVIDIA-B200`	No	Throughput	`ea6e4daa473a2fdb7b298726fd691c8c818882941983313755316d3e3b58296a`
NVFP4 (W4A16)	1	`NVIDIA-H200`	No	Throughput	`c7c20a4fb98bc4c74dcff7d0c27224498680a356e82f81265e8a6f18c99b1da5`
FP8	2	`NVIDIA-H200`	No	Latency	`cefb12490fa7ed8ac5750081ae434d67200c6a98d82514583746874d62c78fa9`
NVFP4	1	`NVIDIA-B200`	No	Throughput, image-text	`c54e675b81630b3e0b60447ee8bbcdbdf6105eaaee7e518b71b67c687566c339`
NVFP4	1	`NVIDIA-GB300`	No	Throughput, image-text	`92c69c33207923a39e3fef4e8b92a92b5a1a6cffae08c62f89573f42ce83b003`
FP8	2	`NVIDIA-H200`	No	Throughput, image-text	`77a71320695a350bb6692371d82b26ddc48aa13ef4bf1d54d196424ec7e25e18`

qwen3.5-397b-a17b#

The following table lists the supported profile configurations for qwen/qwen3.5-397b-a17b:

Precision	TP1	TP2	TP4	TP8
BF16	–	–	`vllm-bf16-tp4-pp1`	`vllm-bf16-tp8-pp1`
FP8	–	`vllm-fp8-tp2-pp1`	`vllm-fp8-tp4-pp1`	`vllm-fp8-tp8-pp1`
NVFP4	`vllm-nvfp4-tp1-pp1`	`vllm-nvfp4-tp2-pp1`	`vllm-nvfp4-tp4-pp1`	`vllm-nvfp4-tp8-pp1`

Optimized deployment profiles#
Precision	TP	GPU	LoRA	Tuning Objective	Profile ID
NVFP4	2	`NVIDIA-B200`	No	Throughput	`f24752064b688b329aec1f40546ee543ec172acce025a7096cb6f0561e63ff28`
NVFP4 (W4A16)	2	`NVIDIA-H200`	No	Throughput	`9fff2d89d1afd4678a1a831fbe8face8621efd82db9a74c90c807a9cb20828da`
NVFP4	4	`NVIDIA-B200`	No	Throughput, image-text	`15fd4fa281d6b4f00be52efab124589fdfd75752bf39991e5ea3a36a32c864b0`
NVFP4	4	`NVIDIA-GB300`	No	Throughput, image-text	`4fbbecf3030485d80536c875231a200696949b86f93ec0df88ec5542deda0f2f`
FP8	4	`NVIDIA-H200`	No	Throughput, image-text	`dbfb9f610e7f0aed1e6604e400b3580bcc2ee8a49c8423620e291bd335d56996`

qwen3.8-27b#

The following table lists the supported profile configurations for qwen/qwen3.8-27b.

NIM_MODEL_PROFILE accepts the 64-character profile ID from the tables below or the full profile description printed by list-model-profiles. Every profile in this image is GPU-specific; there are no generic profiles.

Supported profiles#
Precision	TP	GPU	Profile ID
NVFP4	1	`NVIDIA-B200`	`c4dae72230ad2b0c56220d9d801642564445bc21d6cbdd52702300c273a4a8aa`
NVFP4	1	`NVIDIA-B300-SXM6-AC`	`5b2f72b5f5796d60ff0741c35f3f070cbbfe07e0766e9a98c94c0edca281e525`
NVFP4	1	`NVIDIA-GB200`	`8dbdb556b13a133de592e7e5a93adec5a1d57d89933573a578a24455eb2df439`
NVFP4	1	`NVIDIA-GB300`	`fdada0bb416ccf19025bf84c552c01ff51df3ea4009d107ff31e2d75f06565b2`
NVFP4	1	`NVIDIA-H100-80GB-HBM3`	`156d81239cba86447af7298f160f3451fd3ff0527179fc437c034648cd8620a7`
NVFP4	1	`NVIDIA-H100-NVL`	`7cec39e34ab92e5690b3f4b77c4e43d1824ba819883b7d7ebd22e5a776d8f8e1`
NVFP4	1	`NVIDIA-H200-NVL`	`3385c0aff65ee218ca8bf9e972c4295bff35047961f8be02a47dc886da5df852`
NVFP4	1	`NVIDIA-RTX-PRO-6000-Blackwell-Server-Edition`	`7a49ca4e43231ac8f0461533709c3861e5901cad434b79c5e0f0c260436321e8`
NVFP4	2	`NVIDIA-H200`	`644d1a99bb25c6d9a4436da1eaa94180e6e521d055fdc372b886c84564033593`
NVFP4	2	`NVIDIA-RTX-PRO-4500-Blackwell-Server-Edition`	`fd9dd257f0e94f5ea57713f60d4cdb1fa31db0031cbfc7353b0f80b5c9bdc851`
FP8	1	`NVIDIA-B200`	`61b19c34a31ab2002566d45689f1bda87dac7342b239b7cb7b534d821c5939ec`
FP8	1	`NVIDIA-B300-SXM6-AC`	`284ac649b9186a4ceef13ac152985b91976348dd0fa861dc38def5d28a9c2ded`
FP8	1	`NVIDIA-GB200`	`fcbf637e4b7e25a05ddad6af0eab90ecfad4500a1bfea72187e2fba926d886ca`
FP8	1	`NVIDIA-GB300`	`00d789bb6488d33ec372d2075a2a7dca50610978ff252cd3eaf8220ef21551e0`
FP8	1	`NVIDIA-GH200-144G-HBM3e`	`8ad15a8760e9488f9d2a1320591a7e8832df56ad37559e17c7f84a847e6313ed`
FP8	1	`NVIDIA-GH200-480GB`	`c94d2c4756a6d576201a81dc8f29704237a8c68fb4649e09454a339bea4bbbd2`
FP8	1	`NVIDIA-H100-80GB-HBM3`	`a2cb834abd156023424dee1418682216380b140fb61f47c7ac8d6eaf540c902a`
FP8	1	`NVIDIA-H100-NVL`	`253b217770976f7be094c3279954817f1509d5b0b29dbf5fdbe28e869d7d91a1`
FP8	1	`NVIDIA-H200`	`751c271c4b0bf5f107f68358658607fe3414736458c9c5ef34f2b4e94e6f08a7`
FP8	1	`NVIDIA-H200-NVL`	`2fea60c1917c259e031ddbc81ec224958a0bbcfd26a873593c681a9400d9fb9a`
FP8	1	`NVIDIA-RTX-PRO-6000-Blackwell-Server-Edition`	`17e81574cb3be3731a4de63e69a7be720f7896b3fd8697d1d8e1cc89bed4e9df`
FP8	2	`NVIDIA-B200`	`6fd8f83f0ed11f62c0621b724529f82c261ba5caf8a7ec81bcdea36d464a0835`
FP8	2	`NVIDIA-H200`	`7e3234d6b041b45af321a2291467a936e514f2a67c1b2f42fe703399c40a567d`
FP8	2	`NVIDIA-L40S`	`427e3d5dba49d1cc33db5a031157123153809fdaa8b893706cc9a6c50db27cf2`
FP8	2	`NVIDIA-RTX-PRO-4500-Blackwell-Server-Edition`	`c89a3eba918ebbeca80222f1778326a60d1dc101c04d5a7af4f600d935edcaf9`
BF16	1	`NVIDIA-B200`	`cd30cd903aa1c612f0122c03ef90d1fe6c0368fdcdeda4915898ca422dbd5bd6`
BF16	1	`NVIDIA-B300-SXM6-AC`	`14788d8dee09fc0e70717f8fe126008621e346c767d8200380cadf205d8f8661`
BF16	1	`NVIDIA-GB200`	`b7f9282ff34f4e5815d2faa4d68cbbd9ab23c1c0f1322a966d5ae2e89e7d3c8f`
BF16	1	`NVIDIA-GB300`	`b071f7097ef2f0544bd6bc3bef2780e16e6c18b90337c34822817f3fac7158ce`
BF16	1	`NVIDIA-GH200-144G-HBM3e`	`a6cecb0d6c95b37f262a2f9dece28843ad121e556e1e92ed7cd3dde60911e2c6`
BF16	1	`NVIDIA-GH200-480GB`	`565dbe4d1ac756afaa5ab4bcf658cac56248f27b327d45ed0d938fb57b4a2371`
BF16	1	`NVIDIA-H100-NVL`	`b7f5b91df660fdf407bdaeb5946f6d2f53f54ae8bff19637cbb7be6da56f1771`
BF16	1	`NVIDIA-H200-NVL`	`324d3669a879b87e592ebf2325007071ac9f1ccb1e6eada836e90c04aa409773`
BF16	1	`NVIDIA-RTX-PRO-6000-Blackwell-Server-Edition`	`228f3cab54154ca85dd291d0ec67cd0ebe2522f2a0ccdfa3e7c7b76fc9a8329f`
BF16	2	`NVIDIA-A100-SXM4-80GB`	`86157c6821e4bfec6a8bd325696d339d0335957670f7a7f791d156d75e7066b0`
BF16	2	`NVIDIA-B200`	`ede7c1193e24d624eb40c8e2e8453fe419c44b345a371bf26f165d525d3badd6`
BF16	2	`NVIDIA-H100-80GB-HBM3`	`d2823b8224a0e681bd77d42142f15d9d59d8362a67d552e47bd16da06de7dbc6`
BF16	2	`NVIDIA-H200`	`e70aced682a875db2e916fc6db718a5456bc1516e332ec40d06b1138dc8e875d`
BF16	2	`NVIDIA-L40S`	`6746d14a07b735d835726ed6dbfa08dd828a4a131ded77382ad7186063220bdd`
BF16	4	`NVIDIA-A100-SXM4-40GB`	`c0536126549d169e656464e493e01eab3979f862342ae955a61d1f9886d7f396`
BF16	4	`NVIDIA-RTX-PRO-4500-Blackwell-Server-Edition`	`9b8040192838c52e3feb06bf585a66c421e3cd0858b56e0cdf477051d07fe5cf`

Optimized deployment profiles#
Precision	TP	GPU	LoRA	Tuning Objective	Profile ID
NVFP4	1	`NVIDIA-B200`	No	Throughput	`c4dae72230ad2b0c56220d9d801642564445bc21d6cbdd52702300c273a4a8aa`
FP8	2	`NVIDIA-B200`	No	Throughput	`6fd8f83f0ed11f62c0621b724529f82c261ba5caf8a7ec81bcdea36d464a0835`
BF16	2	`NVIDIA-B200`	No	Throughput	`ede7c1193e24d624eb40c8e2e8453fe419c44b345a371bf26f165d525d3badd6`
NVFP4	2	`NVIDIA-H200`	No	Throughput	`644d1a99bb25c6d9a4436da1eaa94180e6e521d055fdc372b886c84564033593`
FP8	2	`NVIDIA-H200`	No	Throughput	`7e3234d6b041b45af321a2291467a936e514f2a67c1b2f42fe703399c40a567d`
BF16	2	`NVIDIA-H200`	No	Throughput	`e70aced682a875db2e916fc6db718a5456bc1516e332ec40d06b1138dc8e875d`

Model-Free NIM#

vLLM#

The following models are tested and validated for nvidia/vllm-model-free-nim:

Apriel-Nemotron-15b-Thinker
Codestral-2501
GPT-OSS-20B
Kimi-K3
MiniMax-M3
Mistral-Medium-3.5-128B
Qwen3.8-Flash-Next

Beyond the validated list, the vLLM Model-Free NIM can serve any model that the vLLM version in the image supports. Refer to Model-Free NIM for deployment details.

The following models are tested and validated for nvidia/vllm-model-free-nim-experimental:

Step 5 Preview
Mistral Large 4
Kimi K3.1 Pro

Note

GPU requirements vary by model. Refer to the model-specific tables below for the GPU SKUs and tensor-parallel sizes of the released tuning configurations.

Apriel-Nemotron-15b-Thinker#

Apriel-Nemotron-15b-Thinker is served by the vLLM Model-Free NIM 2.1.5 container, nvcr.io/nim/nvidia/vllm-model-free-nim:2.1.5. The container includes the following throughput tuning configurations:

Apriel-Nemotron-15b-Thinker Supported Configurations#
GPU	Precision	TP	LoRA	Tuning Recipe
`NVIDIA-B200`	BF16	1	No	`apriel-nemotron-15b-thinker.yaml`
`NVIDIA-H200`	BF16	1	No	`apriel-nemotron-15b-thinker.yaml`

Both configurations use BF16 weights and an FP8 KV cache. The tuning recipe is selected for the matching checkpoint and GPU configuration. Refer to About Model-Free NIM for deployment instructions.

Codestral-2501#

Codestral-2501 is served by the vLLM Model-Free NIM 2.1.5 container, nvcr.io/nim/nvidia/vllm-model-free-nim:2.1.5. The container includes the following throughput tuning configurations:

Codestral-2501 Supported Configurations#
GPU	Precision	TP	LoRA	Tuning Recipe
`NVIDIA-B200`	BF16	1	No	`codestral-2501.yaml`
`NVIDIA-H200`	BF16	1	No	`codestral-2501.yaml`

Both configurations use BF16 weights and an FP8 KV cache. The tuning recipe is selected for the matching checkpoint and GPU configuration. Refer to About Model-Free NIM for deployment instructions.

Note

Model-specific VRAM requirement in 2.1.5: When this NIM serves codestral, use a single GPU with at least 80 GB of usable VRAM. Its BF16 weights alone are about 47 GB (44 GiB), which leaves no room for runtime allocations on 40-48 GB GPUs such as NVIDIA-L40S and NVIDIA-A100-SXM4-40GB, so it cannot start on them. In this release, gpt-oss-20b and apriel-nemotron were validated on every GPU tested, including NVIDIA-L40S and NVIDIA-A100-SXM4-40GB.

Kimi-K3#

Kimi-K3 is served by the vLLM Model-Free NIM container, nvcr.io/nim/nvidia/vllm-model-free-nim:2.1.4-kimi-k3, with container-bundled throughput tuning configurations. Refer to Get Started for deployment steps.

Set NIM_MODEL_PATH to the checkpoint for the precision you want.

Kimi-K3 Supported Configurations#
Precision	TP	PP	GPU	GPUs per Node	Nodes	Checkpoint (`NIM_MODEL_PATH`)
NVFP4	8	2	`NVIDIA-B200`	8	2	`hf://nvidia/Kimi-K3-NVFP4`
NVFP4	8	1	`NVIDIA-B300-SXM6-AC`	8	1	`hf://nvidia/Kimi-K3-NVFP4`
NVFP4	8	2	`NVIDIA-GB200`	4	4	`hf://nvidia/Kimi-K3-NVFP4`
NVFP4	8	1	`NVIDIA-GB300`	4	2	`hf://nvidia/Kimi-K3-NVFP4`
MXFP4	8	2	`NVIDIA-B200`	8	2	`hf://moonshotai/Kimi-K3`
MXFP4	8	1	`NVIDIA-B300-SXM6-AC`	8	1	`hf://moonshotai/Kimi-K3`
MXFP4	8	2	`NVIDIA-GB200`	4	4	`hf://moonshotai/Kimi-K3`
MXFP4	8	1	`NVIDIA-GB300`	4	2	`hf://moonshotai/Kimi-K3`

TP is the total tensor-parallel group size across nodes. PP is the pipeline parallel size. For example, GB200 uses 16 GPUs across four nodes with TP8 and PP2; GB300 uses eight GPUs across two nodes with TP8 and PP1. These are multi-node configurations even when PP is 1. Use the node layouts in the table rather than inferring the node count from PP.

The tuning configurations are included in the container; checkpoints are not. To obtain the profile IDs applicable to your checkpoint and hardware, run list-model-profiles as described in Get Started.

Kimi K3.1 Pro#

The following table lists the release configurations and their recipe files for moonshotai/Kimi-K3.1-Pro. This model is served by the experimental vLLM Model-Free NIM container, nvcr.io/nim/nvidia/vllm-model-free-nim-experimental:2.1.6-variant-kimi-k3.1-pro. Refer to Kimi K3.1 Pro for deployment steps.

Kimi K3.1 Pro Deployment Configurations#
GPU	GPU Count	GPUs per Node	Nodes	TP	PP	LoRA	Recipe Filename
`NVIDIA-H20-3e`	16	8	2	8	2	No	`kimi-k3.1-pro-h20-3e-16gpu-nn2-tp8-pp2-<role>.yaml`
`NVIDIA-H20-3e`	32	8	4	8	4	No	`kimi-k3.1-pro-h20-3e-32gpu-nn4-tp8-pp4-<role>.yaml`
`NVIDIA-B300-SXM6-AC`	8	8	1	8	1	No	`kimi-k3.1-pro-mxfp4-b300-8gpu-nn1-tp8-pp1.yaml`

H20 recipes have one file per node role. Replace <role> with head on the first node and worker1 (two nodes) or worker1 to worker3 (four nodes) on the others. B300 uses a single file.

The H20 recipes provide a 1,048,576-token context. All layouts use the Kimi K3.1 Pro checkpoint at revision 6d0a0202956796543364d666ab19c6ad1a494ee1. GB300 is not supported for this release.

Use the checkpoint and recipe for the exact GPU and node layout. H20 requires a working RDMA fabric. Its two-node configuration uses a head recipe and one worker recipe; the four-node configuration uses a head recipe and three worker recipes. TP is the tensor-parallel group size; PP is the pipeline-parallel size. Text and image input are supported. Audio and video input and NIM_DISABLE_MODEL_DOWNLOAD are not supported for this model and release.

MiniMax-M3#

MiniMax-M3 is served by the vLLM Model-Free NIM 2.1.4 container, nvcr.io/nim/nvidia/vllm-model-free-nim:2.1.4-minimax-m3.

Set NIM_MODEL_PATH to the NVFP4 or BF16 checkpoint listed in the table. The tuned GB200 profile uses the NVFP4 checkpoint. For download sizes and startup settings, refer to MiniMax-M3 deployment considerations.

The following configurations have been validated:

MiniMax-M3 Supported Configurations#
GPU	Checkpoint (`NIM_MODEL_PATH`)	Precision	TP	PP	Tuning Recipe	Profile ID
`NVIDIA-B200`	`hf://nvidia/MiniMax-M3-NVFP4`	NVFP4	2	1	`minimax-m3-nvfp4.yaml`	`be37e74d30a3af59dee8c221175d9f6eac4d81c35686ce38ba0c1702b617e323`
`NVIDIA-B300-SXM6-AC`	`hf://nvidia/MiniMax-M3-NVFP4`	NVFP4	2	1	`minimax-m3-nvfp4.yaml`	`ff0a1561b7aac5eb4a9cf282989ac7f6808087878ac58d8a49b6d3802c45078f`
`NVIDIA-GB300`	`hf://nvidia/MiniMax-M3-NVFP4`	NVFP4	2	1	`minimax-m3-nvfp4.yaml`	`479bd6da6fd509fe0ef055e535faeeb859f8baf327ff060e62b819dc76aac220`
`NVIDIA-GB200`	`hf://nvidia/MiniMax-M3-NVFP4`	NVFP4	4	1	`minimax-m3-nvfp4.yaml`	`f7bb51cf7250b303c735dd19446891492c780ec045e8e19dd998fcf52bd8bb34`
`NVIDIA-H200`	`hf://nvidia/MiniMax-M3-NVFP4`	NVFP4	4	1	`minimax-m3-nvfp4.yaml`	`2eb348fa36ccde6d5213252dd6bcbafc4adde76235af90f7b7f2f7250ab1d93c`
`NVIDIA-H200-NVL`	`hf://nvidia/MiniMax-M3-NVFP4`	NVFP4	4	1	`minimax-m3-nvfp4.yaml`	`a4ee5d49dfd99b281c40c1f8a9d96255c4cc051d54fac9a6d267041afd65abc4`
`NVIDIA-H100-NVL`	`hf://nvidia/MiniMax-M3-NVFP4`	NVFP4	4	2	`minimax-m3-nvfp4.yaml`	`6bab84f1c70356ba475f3cae72f7862d4378091ae228ec5155619d162dc6d08c`
`NVIDIA-RTX-PRO-6000-Blackwell-Server-Edition`	`hf://nvidia/MiniMax-M3-NVFP4`	NVFP4	4	2	`minimax-m3-nvfp4.yaml`	`88a3600d8a7948b1aec8c41926d69079cab4170fcb07de656d8cfca8b095c8b1`
`NVIDIA-H100-80GB-HBM3`	`hf://nvidia/MiniMax-M3-NVFP4`	NVFP4	8	1	`minimax-m3-nvfp4.yaml`	`dc49587f6be1302f41a0665f2e0f4d66d07225ac9031d92c2807e57205ba2d9d`
`NVIDIA-B300-SXM6-AC`	`hf://MiniMaxAI/MiniMax-M3`	BF16	4	1	`minimax-m3-bf16.yaml`	`368139f640cd6cfea434d76c3452b3ece12d2a054de11c02a25395aca152a84e`
`NVIDIA-GB300`	`hf://MiniMaxAI/MiniMax-M3`	BF16	4	1	`minimax-m3-bf16.yaml`	`19bd37d5df6c8d54dc556fa388a9ef2054ebd180871c8fb0487cd7ef693ad055`
`NVIDIA-B200`	`hf://MiniMaxAI/MiniMax-M3`	BF16	8	1	`minimax-m3-bf16.yaml`	`beb6dff2fa5301e48a0521622448dfe89dd620efec5249ed1268389ae4aabb60`

The NVFP4 profile on NVIDIA GB200 is a tuned profile with EAGLE3 speculative decoding, an FP8 KV cache, and a 400 GB CPU KV-cache offload tier. All other profiles are support profiles that use vLLM defaults. All validated profiles run on one node. The NVFP4 configurations with tensor parallelism (TP) set to four and pipeline parallelism (PP) set to two require eight GPUs on that node. Refer to MiniMax-M3 deployment considerations for the required startup settings and to About Model-Free NIM for deployment instructions. To select a profile explicitly, set NIM_MODEL_PROFILE to its profile ID. Run list-model-profiles in the container to list the profiles available on your system.

Mistral-Medium-3.5-128B#

Mistral-Medium-3.5-128B is served by the vLLM Model-Free NIM 2.1.5 container, nvcr.io/nim/nvidia/vllm-model-free-nim:2.1.5. The following throughput-tuned configurations have been validated:

Mistral Medium 3.5 128B Validated Configurations#
GPU	Checkpoint	Precision	TP	LoRA	Tuning Recipe
`NVIDIA-B200`	`nvidia/Mistral-Medium-3.5-128B-NVFP4`	NVFP4	4	No	`mistral-medium-3.5-128b-nvfp4.yaml`
`NVIDIA-H200`	`mistralai/Mistral-Medium-3.5-128B`	FP8	8	No	`mistral-medium-3.5-128b.yaml`

Both tuning recipes are included in the container and selected for matching checkpoints and GPU configurations. The H200 recipe uses an FP8 KV cache. For the NVFP4 checkpoint, use --tokenizer-mode hf. For the Mistral-native FP8 checkpoint, use --tokenizer-mode mistral --config-format mistral --load-format mistral. Pass these options through NIM_PASSTHROUGH_ARGS. Refer to About Model-Free NIM for deployment instructions.

Qwen3.8-Flash-Next#

Qwen3.8-Flash-Next is served by the vLLM Model-Free NIM container, nvcr.io/nim/nvidia/vllm-model-free-nim:2.1.4-qwen3.8-flash-next. Note the custom tag.

Set NIM_MODEL_PATH to the checkpoint for the precision you want. The checkpoints need about 133 GB (NVFP4), 186 GB (FP8), or 360 GB (BF16) of disk space. For deployment steps, refer to the dedicated Get Started.

The container includes the following throughput tuning configurations:

Qwen3.8-Flash-Next Supported Configurations#
Precision	TP	GPU	Checkpoint (`NIM_MODEL_PATH`)	Profile ID
NVFP4	1	`NVIDIA-B300-SXM6-AC`	`hf://nvidia/Qwen3.8-Flash-Next-NVFP4`	`019381de56cbb5b4037542379f7093b1b9534b1e79abd58de69ddc4a51ead79d`
NVFP4	1	`NVIDIA-GB300`	`hf://nvidia/Qwen3.8-Flash-Next-NVFP4`	`2377b57b80450a37dad96a2338cac167f0db4d4cd9720544779ab7a3c1a24116`
NVFP4	2	`NVIDIA-B200`	`hf://nvidia/Qwen3.8-Flash-Next-NVFP4`	`0f131b5aab66ccd189f77633e31821cc6e921eb8ceab9329a79d9405f5d9cede`
NVFP4	2	`NVIDIA-GB200`	`hf://nvidia/Qwen3.8-Flash-Next-NVFP4`	`900d253530249f5324099ec27bebc716685bdfcf2f36800485daa4ab37c005cc`
NVFP4	4	`NVIDIA-RTX-PRO-6000-Blackwell-Server-Edition`	`hf://nvidia/Qwen3.8-Flash-Next-NVFP4`	`64ef57c5f91f95c3c43b8581eadf4e248db5c60acceaf8680fef07351cfc1f5a`
FP8	2	`NVIDIA-B200`	`hf://Qwen/Qwen3.8-Flash-Next-FP8`	`d679efb9225bafe7b7d2c747c08841e8f8e247ec207e3c63049549ae25199b9b`
FP8	2	`NVIDIA-B300-SXM6-AC`	`hf://Qwen/Qwen3.8-Flash-Next-FP8`	`716879e3a114c157a03ad3c7a66d2caeca14e9c7158c321c25e379b7632f2a02`
FP8	2	`NVIDIA-GB200`	`hf://Qwen/Qwen3.8-Flash-Next-FP8`	`b07c1898bef905395d6fd524c59fa9c13d293636231f51226e2bd5142828167f`
FP8	2	`NVIDIA-GB300`	`hf://Qwen/Qwen3.8-Flash-Next-FP8`	`6a943c1622c3d8bf50d2f2e8ec1fd17e4c6bca4f5ac7d07e3e2db5611578724c`
FP8	2	`NVIDIA-H200`	`hf://Qwen/Qwen3.8-Flash-Next-FP8`	`cdf5cf211de2689299a6a27c7d595f5da2aa3dfd8526cb5ac0c46fdea68bc8d3`
FP8	4	`NVIDIA-H100-NVL`	`hf://Qwen/Qwen3.8-Flash-Next-FP8`	`8f14e80d5b91ff1140527a79fae11724791ff3365a9e72b40ae22e8615a1eb51`
FP8	4	`NVIDIA-H200-NVL`	`hf://Qwen/Qwen3.8-Flash-Next-FP8`	`07c78199c6a21fc17771216f3c9a57ac6522e27e0d8d6c3f845f2917f833b158`
FP8	4	`NVIDIA-RTX-PRO-6000-Blackwell-Server-Edition`	`hf://Qwen/Qwen3.8-Flash-Next-FP8`	`dbf0755bbf886934875f66af4b9f7176384c279d8a917b7933146604a8542313`
BF16	4	`NVIDIA-B300-SXM6-AC`	`hf://Qwen/Qwen3.8-Flash-Next`	`9b51e6e835abe95b619d0c52ef16703394231cce7e892161d1cc6952e50b148d`
BF16	4	`NVIDIA-GB200`	`hf://Qwen/Qwen3.8-Flash-Next`	`4dab55bb6c3472ea2f28a25104e40a71378dece59450b45d0546a068d5e43648`
BF16	4	`NVIDIA-GB300`	`hf://Qwen/Qwen3.8-Flash-Next`	`07c00dbc5ce060b63ddffd7f11f651c7f86f29f6f9f64ac19e50a84cecaff447`
BF16	8	`NVIDIA-B200`	`hf://Qwen/Qwen3.8-Flash-Next`	`309fe9b885e4016dc531ef6c2d718a3ee0ffc4d8982f939225527c60c0397755`

The FP8 recipes for NVIDIA-B200 and NVIDIA-H200 are throughput-tuned and run expert parallel 2 with MTP speculative decoding. They need --shm-size=16GB, and NIM_TENSOR_PARALLEL_SIZE must stay unset. Refer to Tuned FP8 Recipes on B200 and H200.

Step-5-Preview#

Step 5 Preview is served by the vLLM Model-Free NIM (Experimental) 2.1.6-variant container, nvcr.io/nim/nvidia/vllm-model-free-nim-experimental:2.1.6-variant.

The following table lists the supported profile configurations for stepfun-ai/Step-5-Preview.

The container does not select a configuration automatically. Download the recipe file that matches your GPU. For more information, refer to Run with a Downloaded Serving Recipe.

For more information, refer to the dedicated Get Started.

GPU	Precision	GPUs	TP	Expert Parallelism	MTP	LoRA	Recipe File
H20-3e	FP8	8	8	No	3	No	`step5-preview-fp8-h20-3e-8gpu-tp8-mtp3.yaml`
H200	FP8	8	8	No	3	No	`step5-preview-fp8-h200-8gpu-tp8-mtp3.yaml`
B200	NVFP4	4	4	Yes	3	No	`step5-preview-nvfp4-b200-4gpu-tp4-dp1-ep-mtp3.yaml`
GB200	NVFP4	4	4	Yes	3	No	`step5-preview-nvfp4-gb200-4gpu-tp4-dp1-ep-mtp3.yaml`

All four configurations use multi-token prediction with three speculative tokens (MTP3). The NVFP4 configurations add data parallelism of 1 (DP1) alongside expert parallelism.

Each configuration is delivered as a tuning config file in the Step-5 Preview Serving Recipes resource on NGC.

Mistral Large 4#

The following table lists the supported profile configurations for mistralai/Mistral-Large-4-1T-A52B. This model is served by the experimental vLLM Model-Free NIM. For deployment and API examples, refer to the dedicated Get Started guide.

The following configurations serve the full 1,048,576-token context window. Decode context parallelism (DCP) shards the KV cache across the TP ranks. The recipe files are built into the container and selected automatically; rows marked “default” use the default profile.

GPU	Precision	GPUs	TP	Nodes	DCP	LoRA	Recipe File
H200	NVFP4	8	8	1	8	No	`mistral-large-1051b-a54b-nvfp4.yaml`
B200	NVFP4	8	8	1	-	No	default
GB200	NVFP4	8	8	2	-	No	default
B200	FP8	8	8	1	8	No	`mistral-large-1051b-a54b-fp8.yaml`
GB200	FP8	8	8	2	8	No	`mistral-large-1051b-a54b-fp8.yaml`
VR200	FP8	8	8	2	8	No	`mistral-large-1051b-a54b-fp8.yaml`

The two-node configurations use four GPUs per node and require NIM_NUM_COMPUTE_NODES=2 with nim-serve-ray --head and --head-address. H200 NVFP4 uses Marlin emulation and a recipe that enables allow_nvfp4_emulation and sets GPU memory utilization to 0.90.

SGLang#

The following models are tested and validated for nvidia/sglang-model-free-nim:

GLM-5.3
GLM-5.3-Flash
Qwen3.8-2.4T-A95B

Beyond the validated list, the SGLang Model-Free NIM can serve any model that the SGLang version in the image supports. Refer to Model-Free NIM for deployment details.

The following models are tested and validated for nvidia/sglang-model-free-nim-experimental:

Beam-1.0-501B-A23B
Muse Spark Open
MiniMax M3.1 Flash
DeepSeek-V4.1-Flash-1017

GLM-5.3#

GLM-5.3 is served by the SGLang Model-Free NIM 2.1.5 container, nvcr.io/nim/nvidia/sglang-model-free-nim:2.1.5. The container includes the following throughput tuning configurations:

GLM-5.3 Supported Configurations#
GPU	Checkpoint	Precision	TP	LoRA	Tuning Recipe
`NVIDIA-B200`	`nvidia/GLM-5.3-NVFP4`	NVFP4 + FP8	4	No	`glm-5.3-nvfp4.yaml`
`NVIDIA-B200`	`nvidia/GLM-5.3-NVFP4`	NVFP4 + FP8	8	No	`glm-5.3-nvfp4.yaml`
`NVIDIA-H200`	`zai-org/GLM-5.3`	FP8	8	No	`glm-5.3-fp8.yaml`

The tuning recipes are selected for matching checkpoints and GPU configurations. The B200 checkpoint uses NVFP4 for MoE experts and FP8 for the dense MLP layers. The H200 recipe uses an FP8 KV cache. Refer to About Model-Free NIM for deployment instructions.

GLM-5.3-Flash#

GLM-5.3-Flash is served by the SGLang Model-Free NIM 2.1.4 container, nvcr.io/nim/nvidia/sglang-model-free-nim:2.1.4-variant-glm-5.3-flash.

Set NIM_MODEL_PATH to the checkpoint for the precision you want, including the NVFP4 checkpoint for the throughput-optimized GB200 profile. For deployment steps, refer to the dedicated Get Started.

GLM-5.3-Flash Supported Configurations#
Precision	TP	GPU	Checkpoint (`NIM_MODEL_PATH`)	Profile ID
NVFP4	4	`NVIDIA-B200`	`hf://nvidia/GLM-5.3-Flash-NVFP4`	`d0e6e1a4e79b872a77cfa5faf8742f0c06a7ce552420bdba6eaf557278756203`
NVFP4	2	`NVIDIA-B300-SXM6-AC`	`hf://nvidia/GLM-5.3-Flash-NVFP4`	`caeafff184c5ddc019a4dec380841200e5ac8eceebd39fcbd66c58258b794dfb`
NVFP4	4	`NVIDIA-H200`	`hf://nvidia/GLM-5.3-Flash-NVFP4`	`4aca597ab947c56980070dfb62238e742b444d5c47f49a42a384ff88a5247303`
NVFP4	4	`NVIDIA-H200-NVL`	`hf://nvidia/GLM-5.3-Flash-NVFP4`	`03bfe0cfafc7135c482a9e63f4b00d5fab0d0ddd2599aaee89c81fe14ee71b76`
NVFP4	8	`NVIDIA-H100-80GB-HBM3`	`hf://nvidia/GLM-5.3-Flash-NVFP4`	`f47583feff1e90ea79a05f1e511f917c49c3d4f3db45cecce856dbca24d1a547`
NVFP4	8	`NVIDIA-H100-NVL`	`hf://nvidia/GLM-5.3-Flash-NVFP4`	`baf4c2aab0e2f244a3f1f384bee5783e2f6afd3c47b70d0b4ede5c827cd503c4`
FP8	4	`NVIDIA-B200`	`hf://zai-org/GLM-5.3-Flash`	`646d2e5aeba5a8bc21e5f0283aa1d4194763724e7188598d8d75077f7bbc3c34`
FP8	4	`NVIDIA-B300-SXM6-AC`	`hf://zai-org/GLM-5.3-Flash`	`6780254ec7375998309aa95c0fd5aa5ce37b66b5834cabfa5f09e40a3713b876`
FP8	4	`NVIDIA-GB200`	`hf://zai-org/GLM-5.3-Flash`	`dc4413ba1120e2e9390476081494b57bce28e66e4c84ba1abd538f302396a408`
FP8	4	`NVIDIA-H200`	`hf://zai-org/GLM-5.3-Flash`	`d5c2ad86733804c4800af1b06c514ef3a0b37f7f93495a71b6af1298085f6b78`
FP8	4	`NVIDIA-H200-NVL`	`hf://zai-org/GLM-5.3-Flash`	`78083888c84b2207be4e63f97368021754d695a147160b5dfa29887bcad86690`
FP8	8	`NVIDIA-H100-80GB-HBM3`	`hf://zai-org/GLM-5.3-Flash`	`fb32e6152dbf1b1d9df5a5b5b069e90935c929b705173ea5d83e2dbc5d239abe`
FP8	8	`NVIDIA-H100-NVL`	`hf://zai-org/GLM-5.3-Flash`	`72c3835033db4d7970cffc89d4ef936eb14f8f1430fa321e8d425c5172cd8d3c`
BF16	4	`NVIDIA-B300-SXM6-AC`	`hf://zai-org/GLM-5.3-Flash-BF16`	`4bb955cd9a620abfef9439ed2113a386d2a1a9c7c316d77b12807f2b3b9be349`
BF16	8	`NVIDIA-B200`	`hf://zai-org/GLM-5.3-Flash-BF16`	`569079ed4e36c21f44f7e5d77937bf3457055299b0b917f8273651b14ad84e87`
BF16	8	`NVIDIA-H200`	`hf://zai-org/GLM-5.3-Flash-BF16`	`9d87d3c47d7499d35947b9d4f455c5585251b23cb9cd2ed8db9b970ddb09e1ce`
BF16	8	`NVIDIA-H200-NVL`	`hf://zai-org/GLM-5.3-Flash-BF16`	`ca146d5d244b7e07d5aca3ef70fba251ac6d507cf4ea45ec62faf30992f4f255`

Optimized deployment profiles#
Precision	TP	GPU	LoRA	Tuning Objective	Checkpoint (`NIM_MODEL_PATH`)	Profile ID
NVFP4	4	`NVIDIA-GB200`	No	Throughput	`hf://nvidia/GLM-5.3-Flash-NVFP4`	`f9bd55116d3fdfe1704064ca7c0899c028089ae8f92f140bca6e809457e56670`

Qwen3.8-2.4T-A95B#

Qwen3.8-2.4T-A95B is served by the SGLang Model-Free NIM container. The following table lists the validated multi-node configuration for Qwen/Qwen3.8-2.4T-A95B-FP8, as documented in the Get Started with Qwen3.8-2.4T-A95B guide.

Qwen3.8-2.4T-A95B Supported Configurations#
Precision	TP	PP	Nodes	GPUs per Node
FP8	16	1	4	4

Deployment requires Kubernetes and at least four GB300-NVL72 nodes. The configuration uses 16 GPUs across four nodes, with four GPUs per node. Image and video input are not supported. Refer to the linked guide for the multi-node Helm configuration and deployment requirements.

Beam-1.0-501B-A23B#

Beam-1.0-501B-A23B is served by the experimental SGLang model-free NIM container, nvcr.io/nim/nvidia/sglang-model-free-nim-experimental:2.1.7-variant-beam-1.0-501b, with a matching Beam serving recipe. For deployment instructions and checkpoint selection, refer to Get Started with Beam.

Beam Supported Configurations#
GPU	Precision	Tensor Parallelism	Hugging Face Model Path
NVIDIA H100	FP8	8	`hf://reflection/Beam-1.0-501B-A23B-FP8`
NVIDIA H20-3e	FP8	8	`hf://reflection/Beam-1.0-501B-A23B-FP8`
NVIDIA H200	FP8	8	`hf://reflection/Beam-1.0-501B-A23B-FP8`
NVIDIA B200	NVFP4	4	`hf://reflection/Beam-1.0-501B-A23B-NVFP4`
NVIDIA GB200	NVFP4	4	`hf://reflection/Beam-1.0-501B-A23B-NVFP4`

The container does not select a configuration automatically. Download the recipe file that matches your GPU and desired precision. For more information, refer to Run with a Downloaded Serving Recipe.

Muse Spark Open#

Muse Spark Open is served by the SGLang Model-Free NIM (Experimental) container, nvcr.io/nim/nvidia/sglang-model-free-nim-experimental:2.1.6-variant.

The following table lists the supported profile configurations for meta-models/Muse-Spark-Open-816B-A42B-NVFP4. For deployment instructions, refer to Get Started with Muse Spark Open.

Muse Spark Open Supported Configurations#
Precision	TP8 or EP8
NVFP4	`sglang-nvfp4-tp8-pp1`

There are no packaged profile IDs for this NIM. The NIM infers the configuration in the table.

MiniMax M3.1 Flash#

MiniMax M3.1 Flash is served by the SGLang Model-Free NIM (Experimental) container, nvcr.io/nim/nvidia/sglang-model-free-nim-experimental:2.1.7-variant.

Use the matching 2.1.7-variant serving recipe from the SGLang Model-Free NIM Experimental Collection for your GPU and topology. The container does not include model weights or model-specific recipe settings.

For recipe download, launch, and API examples, refer to Get Started with MiniMax M3.1 Flash.

The following FP8 throughput recipes have been validated:

MiniMax M3.1 Flash Supported Configurations#
GPU	Precision	TP	LoRA	Tuning Recipe
`NVIDIA-B200`	FP8	4	No	`b200_throughput_fp8_tp4.yaml`
`NVIDIA-B200`	FP8	8	No	`b200_throughput_fp8_tp8.yaml`
`NVIDIA-GB200`	FP8	4	No	`gb200_throughput_fp8_tp4.yaml`
`NVIDIA-H200`	FP8	8	No	`h200_throughput_fp8_tp8.yaml`
`NVIDIA-H20-3e`	FP8	8	No	`h20-3e_throughput_fp8_tp8.yaml`

Select a recipe for the exact hardware and topology you deploy. A verified GPU does not imply support for every precision or tensor-parallel configuration.

DeepSeek-V4.1-Flash-1017#

DeepSeek-V4.1-Flash-1017 uses the experimental SGLang Model-Free NIM container nvcr.io/nim/nvidia/sglang-model-free-nim-experimental:2.1.6-variant-deepseek-v4.1-flash-1017 and its matching recipe resource nim/nvidia/deepseek-v4.1-flash-1017-recipe:2.1.6-variant-deepseek-v4.1-flash-1017 from the SGLang Model-Free NIM Experimental Collection. Model weights are supplied separately at runtime.

For recipe download, container launch, and API examples, refer to Get Started with DeepSeek-V4.1-Flash-1017.

All four recipes provide a 1,048,576-token (1M) context with DSpark speculative decoding enabled (block size 5).

The following FP8 configurations use matching tensor and expert parallelism:

DeepSeek-V4.1-Flash-1017 Supported Configurations#
GPU	Precision	GPU Count	TP	EP	LoRA
`NVIDIA-B200`	FP8	4	4	4	No
`NVIDIA-GB200`	FP8	4	4	4	No
`NVIDIA-H200`	FP8	8	8	8	No
`NVIDIA-H20-3e`	FP8	8	8	8	No

Select the recipe for your exact GPU and topology. Use container and recipe version 2.1.6-variant-deepseek-v4.1-flash-1017 together.

1.x NIM LLM Models#

For more information on version 1.x NIMs, refer to the 1.15 version of the NIM LLM Supported Models page.

Show 1.x models

Model (Hardware Requirements)	Organization/Model ID (Catalog Page)
Bielik 11B v2.3 Instruct	`speakleash/bielik-11b-v2.3-instruct`
Code Llama 13B Instruct	`meta/codellama-13b-instruct`
Code Llama 34B Instruct	`meta/codellama-34b-instruct`
Code Llama 70B Instruct	`meta/codellama-70b-instruct`
DeepSeek Coder V2 Lite Instruct	`deepseek-ai/deepseek-coder-v2-lite-instruct`
DeepSeek R1	`deepseek-ai/deepseek-r1`
DeepSeek R1 Distill Llama 8B	`deepseek-ai/deepseek-r1-distill-llama-8b`
DeepSeek R1 Distill Llama 70B	`deepseek-ai/deepseek-r1-distill-llama-70b`
DeepSeek R1 Distill Llama 8B RTX	`deepseek-ai/deepseek-r1-distill-llama-8b`
DeepSeek R1 Distill Qwen 32B	`deepseek-ai/deepseek-r1-distill-qwen-32b`
DeepSeek-V3.1-Terminus	`deepseek-ai/deepseek-v3.1-terminus`
DeepSeek-V3.2-Exp	`deepseek-ai/deepseek-v32-exp-nim`
DeepSeek-V4-Flash	`deepseek-ai/deepseek-v4-flash`
DeepSeek-V4-Pro	`deepseek-ai/deepseek-v4-pro`
EuroLLM 9B Instruct	`utter-project/eurollm-9b-instruct`
Gemma 2 2B	`google/gemma-2-2b-instruct`
Gemma 2 9B	`google/gemma-2-9b-it`
Gemma2 9B CPT Sahabat-AI v1 Instruct	`gotocompany/gemma2-9b-cpt-sahabatai-v1-instruct`
Gemma 3 1B Instruct	`google/gemma-3-1b-it`
GLM-5	`zai-org/glm-5`
GLM-5.1	`zai-org/glm-51`
GPT-OSS-20B	`openai/gpt-oss-20b`
GPT-OSS-120B	`openai/gpt-oss-120b`
Granite 3.3 8B Instruct	`ibm-granite/granite-3.3-8b-instruct`
GreenMind Medium 14B R1	`greennode/greenmind-medium-14b-r1`
Kanana 1.5 8B Instruct 2505	`kakaocorp/kanana-1.5-8b-instruct-2505`
(Meta) Llama 2 7B Chat	`meta/llama-2-7b-chat`
(Meta) Llama 2 13B Chat	`meta/llama-2-13b-chat`
(Meta) Llama 2 70B Chat	`meta/llama-2-70b-chat`
Llama 3 SQLCoder 8B	`defog/llama-3-sqlcoder-8b`
Llama 3 Swallow 70B Instruct V0.1	`tokyotech-llm/llama-3-swallow-70b-instruct-v0.1`
Llama 3 Taiwan 70B Instruct	`yentinglin/llama-3-taiwan-70b-instruct`
Llama 3.1 8B Base	`meta/llama-3.1-8b-base`
Llama 3.1 8B Instruct	`meta/llama-3.1-8b-instruct`
Llama-3.1-8b-Instruct-DGX-Spark	`meta/llama-3.1-8b-instruct-dgx-spark`
Llama-3.1-8B-Instruct PB 25h2	`meta/llama-3.1-8b-instruct-pb25h2`
Llama 3.1 8B Instruct RTX	`meta/llama-3.1-8b-instruct`
Llama 3.1 70B Instruct	`meta/llama-3.1-70b-instruct`
Llama-3.1-70B-Instruct PB 25h2	`meta/llama-3.1-70b-instruct-pb25h2`
Llama 3.1 405B Instruct	`meta/llama-3.1-405b-instruct`
Llama 3.1 Nemotron Nano 4B V1.1	`nvidia/llama3.1-nemotron-nano-4b-v1.1`
Llama 3.1 Nemotron Nano 8B V1	`nvidia/llama-3.1-nemotron-nano-8b-v1`
Llama-3.1-Nemotron-Nano-8B-Healthcare-Text2sql-v1.0	`nvidia/llama-3.1-nemotron-nano-8b-healthcare-text2sql-v1.0`
Llama 3.1 Nemotron Ultra 253B V1	`nvidia/llama-3.1-nemotron-ultra-253b-v1`
Llama 3.1 Nemotron 70B Instruct	`nvidia/llama-3.1-nemotron-70b-instruct`
Llama 3.1 Swallow 8B Instruct v0.1	`tokyotech-llm/llama-3.1-swallow-8b-instruct-v0.1`
Llama 3.1 Swallow 70B Instruct v0.1	`tokyotech-llm/llama-3.1-swallow-70b-instruct-v0.1`
Llama 3.1 Typhoon 2 8B Instruct	`scb10x/llama3.1-typhoon2-8b-instruct`
Llama 3.1 Typhoon 2 70B Instruct	`scb10x/llama-3.1-typhoon2-70b-instruct`
Llama 3.2 1B Instruct	`meta/llama-3.2-1b-instruct`
Llama 3.2 3B Instruct	`meta/llama-3.2-3b-instruct`
Llama 3.3 70B Instruct	`meta/llama-3.3-70b-instruct`
Llama 3.3 Nemotron Super 49B V1	`nvidia/llama-3.3-nemotron-super-49b-v1`
Llama-3.3-Nemotron-Super-49B-v1.5	`nvidia/llama-3.3-nemotron-super-49b-v1.5`
Llama-3.3-Nemotron-Super-49B-Healthcare-Text2sql-v1.0	`nvidia/llama-3.3-nemotron-super-49b-healthcare-text2sql-v1.0`
Llama-3.3-Nemotron-Super-49B-v1.5 PB 25h2	`nvidia/llama-3.3-nemotron-super-49b-v1.5-pb25h2`
Meta Llama 3 8B Instruct	`meta/llama3-8b-instruct`
Meta Llama 3 70B Instruct	`meta/llama3-70b-instruct`
Muse Glimmer	`meta/muse-glimmer`
MiMo-V2-Flash	`xiaomi/mimo-v2-flash-experimental`
MiniMax-M2.5	`minimax-ai/minimax-m25`
MiniMax-M2.7	`minimax-ai/minimax-m27`
Mistral 7B Instruct V0.3	`mistralai/mistral-7b-instruct-v0.3`
Mistral NeMo 12B Instruct RTX	`nv-mistralai/mistral-nemo-12b-instruct`
Mistral NeMo 12B Instruct	`nv-mistralai/mistral-nemo-12b-instruct`
Mistral NeMo Minitron 8B 8K Instruct	`nv-mistralai/mistral-nemo-minitron-8b-8k-instruct`
Mistral Small 24b Instruct 2501	`mistralai/mistral-small-24b-instruct-2501`
Mixtral 8x7B Instruct V0.1	`mistralai/mixtral-8x7b-instruct-v0-1`
Mixtral 8x22B Instruct V0.1	`mistralai/mixtral-8x22b-instruct-v01`
Nemotron 4 340B Instruct	`nvidia/nemotron-4-340b-instruct`
Nemotron 4 340B Reward	`nvidia/nemotron-4-340b-reward`
NVIDIA Nemotron 3 Nano	`nvidia/nemotron-3-nano`
NVIDIA-Nemotron-Nano-9B-v2	`nvidia/nvidia-nemotron-nano-9b-v2`
NVIDIA-Nemotron-Nano-9B-v2-DGX-Spark	`nvidia/nvidia-nemotron-nano-9b-v2-dgx-spark`
Nemotron-3-Super-120B-A12B	`nvidia/nemotron-3-super-120b-a12b`
Phi 3 Mini 4K Instruct	`microsoft/phi-3-mini-4k-instruct`
Phi 4 Mini Instruct	`microsoft/phi-4-mini-instruct`
Phind Codellama 34B V2 Instruct	`phind/phind-codellama-34b-v2-instruct`
Qwen3-Coder-Next	`qwen/qwen3-coder-next`
Qwen3-Next-80B-A3B-Instruct	`qwen/qwen3-next-80b-a3b-instruct`
Qwen3 Next 80B A3B Thinking	`qwen/qwen3-next-80b-a3b-thinking`
Qwen3-32B	`qwen/qwen3-32b`
Qwen3-32B NIM for DGX Spark	`qwen/qwen3-32b-dgx-spark`
Qwen2.5 Coder 32B Instruct	`qwen/qwen2.5-coder-32b-instruct`
Qwen2.5 7B Instruct	`qwen/qwen-2.5-7b-instruct`
Riva Translate 4B Instruct	`nvidia/riva_translate_4b_instruct`
Riva-Translate-4b-Instruct-v1.1	`nvidia/riva-translate-4b-instruct-v1.1`
Sarvam - M	`sarvamai/sarvam-m`
SILMA 9B Instruct v1.0	`silma-ai/silma-9b-instruct-v1.0`
StarCoder2 7B	`bigcode/starcoder2-7b`
StarCoderBase 15.5B	`bigcode/starcoderbase-15b`
Stockmark-2-100B-Instruct	`stockmark/stockmark-2-100b-instruct`
Teuken 7B Instruct Commercial v0.4	`opengpt-x/teuken-7b-instruct-commercial-v0.4`

Additional VLM Models#

VLMs that were released prior to version 2.0.12 are documented on the NVIDIA NIM for Vision Language Models site. Refer to that site for more information on running these VLM models.

Show additional VLM models

Model (Hardware Requirements)	Organization/Model ID (Catalog Page)
Cosmos 3 Reasoner	`nvidia/cosmos3-reasoner`
Cosmos Reason1 7B	`nvidia/cosmos-reason1-7b`
Cosmos Reason2	`nvidia/cosmos-reason2-8b`
DiffusionGemma 26B A4B IT	`google/diffusiongemma-26b-a4b-it`
Kimi-K2.5	`moonshotai/kimi-k2.5`
Llama 3.1 Nemotron Nano VL 8B v1	`nvidia/llama-3.1-nemotron-nano-vl-8b-v1`
Llama 3.2 11B Vision Instruct	`meta/llama-3.2-11b-vision-instruct`
Llama 3.2 90B Vision Instruct	`meta/llama-3.2-90b-vision-instruct`
Llama 4 Maverick 17B 128E Instruct	`meta/llama-4-maverick-17b-128e-instruct`
Llama 4 Scout 17B 16E Instruct	`meta/llama-4-scout-17b-16e-instruct`
Ministral 3 14B Instruct 2512	`mistralai/ministral-3-14b-instruct-2512`
Mistral Large 3 675B Instruct 2512	`mistralai/mistral-large-3-675b-instruct-2512`
Mistral Small 3.2 24B Instruct 2506	`mistralai/mistral-small-3.2-24b-instruct-2506`
Nemotron Nano 12B v2 VL	`nvidia/nemotron-nano-12b-v2-vl`
Qwen3.5	`qwen/qwen3.5-35b-a3b`
Qwen3.6-27B	`qwen/qwen3.6-27b`
Qwen3.6-35B-A3B	`qwen/qwen3.6-35b-a3b`
Step 3.7 Flash	`stepfun-ai/step-3.7-flash`