Support Matrix#

This page lists the supported models, their deployment profiles, and the verified hardware SKUs for NIM LLM.

Supported Models and Profiles#

gpt-oss-120b#

The following profiles are available for this NIM:

BF16 Profiles

FP8 Profiles

Mxfp4 / NVFP4 Profiles

Other Profiles

 

 

  • vllm-mxfp4-tp1-pp1
  • vllm-mxfp4-tp2-pp1
  • vllm-mxfp4-tp4-pp1
  • vllm-mxfp4-tp8-pp1

 

gpt-oss-20b#

The following profiles are available for this NIM:

BF16 Profiles

FP8 Profiles

Mxfp4 / NVFP4 Profiles

Other Profiles

 

 

  • vllm-mxfp4-tp1-pp1
  • vllm-mxfp4-tp2-pp1
  • vllm-mxfp4-tp4-pp1
  • vllm-mxfp4-tp8-pp1

 

llama-3.1-70b-instruct#

The following profiles are available for this NIM:

BF16 Profiles

FP8 Profiles

Mxfp4 / NVFP4 Profiles

Other Profiles

  • vllm-bf16-tp2-pp1
  • vllm-bf16-tp4-pp1
  • vllm-bf16-tp8-pp1
  • vllm-bf16-tp2-pp1-lora
  • vllm-bf16-tp4-pp1-lora
  • vllm-bf16-tp8-pp1-lora

 

 

 

llama-3.1-8b-instruct#

The following profiles are available for this NIM:

BF16 Profiles

FP8 Profiles

Mxfp4 / NVFP4 Profiles

Other Profiles

  • vllm-bf16-tp1-pp1
  • vllm-bf16-tp1-pp1-lora

  • vllm-fp8-tp1-pp1
  • vllm-fp8-tp1-pp1-lora

 

 

llama-3.3-70b-instruct#

The following profiles are available for this NIM:

BF16 Profiles

FP8 Profiles

Mxfp4 / NVFP4 Profiles

Other Profiles

  • vllm-bf16-tp2-pp1
  • vllm-bf16-tp4-pp1
  • vllm-bf16-tp8-pp1
  • vllm-bf16-tp2-pp1-lora
  • vllm-bf16-tp4-pp1-lora
  • vllm-bf16-tp8-pp1-lora

 

 

 

llama-3.3-nemotron-super-49b-v1.5#

The following profiles are available for this NIM:

BF16 Profiles

FP8 Profiles

Mxfp4 / NVFP4 Profiles

Other Profiles

  • vllm-bf16-tp1-pp1-lora
  • vllm-bf16-tp2-pp1-lora
  • vllm-bf16-tp4-pp1-lora
  • vllm-bf16-tp8-pp1-lora
  • vllm-bf16-tp1-pp1
  • vllm-bf16-tp2-pp1
  • vllm-bf16-tp4-pp1
  • vllm-bf16-tp8-pp1

  • vllm-fp8-NVIDIA-GB10-tp1-pp1

  • vllm-nvfp4-NVIDIA-GB10-tp1-pp1

 

nemotron-3-nano#

The following profiles are available for this NIM:

BF16 Profiles

FP8 Profiles

Mxfp4 / NVFP4 Profiles

Other Profiles

  • vllm-bf16-tp1-pp1
  • vllm-bf16-tp2-pp1
  • vllm-bf16-tp4-pp1
  • vllm-bf16-tp8-pp1

  • vllm-fp8-tp1-pp1
  • vllm-fp8-tp2-pp1
  • vllm-fp8-tp4-pp1
  • vllm-fp8-tp8-pp1

 

 

starcoder2-7b#

The following profiles are available for this NIM:

BF16 Profiles

FP8 Profiles

Mxfp4 / NVFP4 Profiles

Other Profiles

  • vllm-bf16-tp1-pp1
  • vllm-bf16-tp2-pp1

 

 

 

Model-Free NIM#

The following profiles are available for this NIM:

BF16 Profiles

FP8 Profiles

Mxfp4 / NVFP4 Profiles

Other Profiles

 

 

 

  • vllm-tp1-pp1

Currently, the following models are tested and validated for model-free NIM:

  • gpt-oss-20b

  • apriel-nemotron

  • codestral

Verified Hardware SKUs#

NIM compatibility and functionality have been validated on the following GPU SKUs:

gpt-oss-120b#

The following GPUs are compatible with this NIM:

  • NVIDIA-A100-SXM4-40GB

  • NVIDIA-A100-SXM4-80GB

  • NVIDIA-B200

  • NVIDIA-A10G

  • NVIDIA-GH200-480GB

  • NVIDIA-GH200-144G-HBM3e

  • NVIDIA-H100-80GB-HBM3

  • NVIDIA-H20

  • NVIDIA-RTX-PRO-6000-Blackwell-Server-Edition

  • NVIDIA-L40S

  • NVIDIA-H200

  • NVIDIA-GB200

  • NVIDIA-B300-SXM6-AC

  • NVIDIA-RTX-PRO-4500-Blackwell-Server-Edition

gpt-oss-20b#

The following GPUs are compatible with this NIM:

  • NVIDIA-A100-SXM4-40GB

  • NVIDIA-A100-SXM4-80GB

  • NVIDIA-B200

  • NVIDIA-A10G

  • NVIDIA-GH200-480GB

  • NVIDIA-GH200-144G-HBM3e

  • NVIDIA-H100-80GB-HBM3

  • NVIDIA-H20

  • NVIDIA-RTX-PRO-6000-Blackwell-Server-Edition

  • NVIDIA-L40S

  • NVIDIA-H200

  • NVIDIA-GB200

  • NVIDIA-B300-SXM6-AC

  • NVIDIA-RTX-PRO-4500-Blackwell-Server-Edition

llama-3.1-70b-instruct#

The following GPUs are compatible with this NIM:

  • NVIDIA-A100-SXM4-40GB

  • NVIDIA-GB200

  • NVIDIA-GH200-144G-HBM3e

  • NVIDIA-H100-80GB-HBM3

  • NVIDIA-H200

  • NVIDIA-RTX-PRO-6000-Blackwell-Server-Edition

  • NVIDIA-A100-SXM4-80GB

  • NVIDIA-B200

  • NVIDIA-H100-NVL

  • NVIDIA-H200-NVL

  • NVIDIA-B300-SXM6-AC

  • NVIDIA-RTX-PRO-4500-Blackwell-Server-Edition

llama-3.1-8b-instruct#

The following GPUs are compatible with this NIM:

  • NVIDIA-A10G

  • NVIDIA-GB200

  • NVIDIA-GH200-144G-HBM3e

  • NVIDIA-H100-80GB-HBM3

  • NVIDIA-H200

  • NVIDIA-L40S

  • NVIDIA-A100-SXM4-80GB

  • NVIDIA-B200

  • NVIDIA-GH200-480GB

  • NVIDIA-H100-NVL

  • NVIDIA-H200-NVL

  • NVIDIA-RTX-PRO-6000-Blackwell-Server-Edition

  • NVIDIA-A100-SXM4-40GB

  • NVIDIA-B300-SXM6-AC

  • NVIDIA-RTX-PRO-4500-Blackwell-Server-Edition

llama-3.3-70b-instruct#

The following GPUs are compatible with this NIM:

  • NVIDIA-A100-SXM4-40GB

  • NVIDIA-A100-SXM4-80GB

  • NVIDIA-B200

  • NVIDIA-GB200

  • NVIDIA-GH200-144G-HBM3e

  • NVIDIA-H100-80GB-HBM3

  • NVIDIA-H200

  • NVIDIA-RTX-PRO-6000-Blackwell-Server-Edition

  • NVIDIA-H100-NVL

  • NVIDIA-H200-NVL

  • NVIDIA-B300-SXM6-AC

  • NVIDIA-RTX-PRO-4500-Blackwell-Server-Edition

llama-3.3-nemotron-super-49b-v1.5#

The following GPUs are compatible with this NIM:

  • NVIDIA-A100-SXM4-40GB

  • NVIDIA-GB200

  • NVIDIA-A10G

  • NVIDIA-GH200-144G-HBM3e

  • NVIDIA-H100-80GB-HBM3

  • NVIDIA-H200

  • NVIDIA-L40S

  • NVIDIA-RTX-PRO-6000-Blackwell-Server-Edition

  • NVIDIA-A100-SXM4-80GB

  • NVIDIA-B200

  • NVIDIA-GB10

  • NVIDIA-GH200-480GB

  • NVIDIA-H100-NVL

  • NVIDIA-H200-NVL

  • NVIDIA-B300-SXM6-AC

  • NVIDIA-RTX-PRO-4500-Blackwell-Server-Edition

  • NVIDIA-A100-80GB-PCIe

  • NVIDIA-A100-PCIE-40GB

  • NVIDIA-H100-PCIe

nemotron-3-nano#

The following GPUs are compatible with this NIM:

  • NVIDIA-A100-SXM4-80GB

  • NVIDIA-B200

  • NVIDIA-GB10

  • NVIDIA-A100-SXM4-40GB

  • NVIDIA-GB200

  • NVIDIA-GH200-144G-HBM3e

  • NVIDIA-H100-80GB-HBM3

  • NVIDIA-H200

  • NVIDIA-L40S

  • NVIDIA-RTX-PRO-6000-Blackwell-Server-Edition

  • NVIDIA-H200-NVL

  • NVIDIA-H100-NVL

  • NVIDIA-B300-SXM6-AC

  • NVIDIA-RTX-PRO-4500-Blackwell-Server-Edition

starcoder2-7b#

The following GPUs are compatible with this NIM:

  • NVIDIA-H100-80GB-HBM3

  • NVIDIA-H200

Model-Free NIM#

The following GPUs are compatible with the model-free NIM:

  • NVIDIA-A100-SXM4-40GB

  • NVIDIA-GB200

  • NVIDIA-A10G

  • NVIDIA-GH200-144G-HBM3e

  • NVIDIA-H100-80GB-HBM3

  • NVIDIA-H200

  • NVIDIA-L40S

  • NVIDIA-RTX-PRO-6000-Blackwell-Server-Edition

  • NVIDIA-A100-SXM4-80GB

  • NVIDIA-B200

  • NVIDIA-GB10

  • NVIDIA-GH200-480GB

  • NVIDIA-H100-NVL

  • NVIDIA-H200-NVL

  • NVIDIA-B300-SXM6-AC

  • NVIDIA-RTX-PRO-4500-Blackwell-Server-Edition

  • NVIDIA-A100-80GB-PCIe

  • NVIDIA-A100-PCIE-40GB

  • NVIDIA-H100-PCIe