Support Matrix#
Hardware#
NVIDIA NIMs for large-language models will run on any NVIDIA GPU, as long as the GPU has sufficient memory, or on multiple, homogeneous NVIDIA GPUs with sufficient aggregate memory and CUDA compute capability > 7.0 (8.0 for bfloat16). Some model/GPU combinations are optimized. See the following Supported Models section for further information.
Software#
Linux operating systems (Ubuntu 20.04 or later recommended)
NVIDIA Driver >= 535
NVIDIA Docker >= 23.0.1
Supported Models#
These models are optimized using TRT-LLM and are available as pre-built, optimized engines on NGC and should use the Chat Completions Endpoint.
Llama 2 7B Chat#
Optimized configurations#
The GPU Memory and Disk Space values are in GB; Disk Space is for both the container and the model; Profile is for what the model is optimized.
GPU |
GPU Memory |
Precision |
Profile |
# of GPUs |
Disk Space |
|---|---|---|---|---|---|
H100 SXM/NVLink |
80 |
8 |
Latency |
2 |
6.66 |
H100 SXM/NVLink |
80 |
16 |
Latency |
2 |
12.93 |
H100 SXM/NVLink |
80 |
8 |
Throughput |
1 |
6.57 |
H100 SXM/NVLink |
80 |
16 |
Throughput |
1 |
12.62 |
H100 SXM/NVLink |
80 |
16 |
Throughput LoRA |
1 |
12.63 |
A100 SXM/NVLink |
80 |
16 |
Latency |
2 |
12.92 |
A100 SXM/NVLink |
80 |
16 |
Throughput |
1 |
15.54 |
A100 SXM/NVLink |
80 |
16 |
Throughput LoRA |
1 |
12.63 |
L40S PCIe |
48 |
8 |
Latency |
2 |
6.64 |
L40S PCIe |
48 |
16 |
Latency |
2 |
12.95 |
L40S PCIe |
48 |
8 |
Throughput |
1 |
6.57 |
L40S PCIe |
48 |
16 |
Throughput |
1 |
12.64 |
L40S PCIe |
48 |
16 |
Throughput LoRA |
1 |
12.65 |
Non-optimized configuration#
Any NVIDIA GPU with sufficient GPU memory or on multiple, homogeneous NVIDIA GPUs with sufficient aggregate memory and compute capability > 7.0 (8.0 for bfloat16)
Llama 2 13B Chat#
Optimized configurations#
The GPU Memory and Disk Space values are in GB; Disk Space is for both the container and the model; Profile is for what the model is optimized.
GPU |
GPU Memory |
Precision |
Profile |
# of GPUs |
Disk Space |
|---|---|---|---|---|---|
H100 SXM/NVLink |
80 |
8 |
Latency |
2 |
12.6 |
H100 SXM/NVLink |
80 |
16 |
Latency |
2 |
24.71 |
H100 SXM/NVLink |
80 |
16 |
Throughput |
1 |
24.33 |
H100 SXM/NVLink |
80 |
16 |
Throughput LoRA |
1 |
24.35 |
A100 SXM/NVLink |
80 |
16 |
Latency |
2 |
24.74 |
A100 SXM/NVLink |
80 |
16 |
Throughput |
2 |
24.34 |
A100 SXM/NVLink |
80 |
16 |
Throughput LoRA |
1 |
24.37 |
L40S PCIe |
48 |
8 |
Latency |
2 |
12.39 |
L40S PCIe |
48 |
16 |
Latency |
2 |
24.7 |
L40S PCIe |
48 |
8 |
Throughput |
1 |
12.49 |
L40S PCIe |
48 |
16 |
Throughput |
1 |
24.33 |
L40S PCIe |
48 |
16 |
Throughput LoRA |
1 |
24.37 |
Non-optimized configuration#
Any NVIDIA GPU with sufficient GPU memory or on multiple, homogeneous NVIDIA GPUs with sufficient aggregate memory and compute capability > 7.0 (8.0 for bfloat16)
Llama 2 70B Chat#
Optimized configurations#
The GPU Memory and Disk Space values are in GB; Disk Space is for both the container and the model; Profile is for what the model is optimized.
GPU |
GPU Memory |
Precision |
Profile |
# of GPUs |
Disk Space |
|---|---|---|---|---|---|
H100 SXM/NVLink |
80 |
16 |
Throughput |
4 |
130.52 |
H100 SXM/NVLink |
80 |
8 |
Latency |
4 |
65.36 |
H100 SXM/NVLink |
80 |
16 |
Latency |
8 |
133.18 |
H100 SXM/NVLink |
80 |
8 |
Throughput |
2 |
65.08 |
H100 SXM/NVLink |
80 |
16 |
Throughput LoRA |
4 |
130.6 |
A100 SXM/NVLink |
80 |
16 |
Latency |
4 |
133.12 |
A100 SXM/NVLink |
80 |
16 |
Throughput |
4 |
130.52 |
A100 SXM/NVLink |
80 |
16 |
Throughput LoRA |
4 |
130.5 |
L40S PCIe |
48 |
8 |
Throughput |
4 |
63.35 |
Non-optimized configuration#
Any NVIDIA GPU with sufficient GPU memory or on multiple, homogeneous NVIDIA GPUs with sufficient aggregate memory and compute capability > 7.0 (8.0 for bfloat16)
Meta-Llama-3-8B-Instruct#
Optimized configurations#
The GPU Memory and Disk Space values are in GB; Disk Space is for both the container and the model; Profile is for what the model is optimized.
GPU |
GPU Memory |
Precision |
Profile |
# of GPUs |
Disk Space |
|---|---|---|---|---|---|
H100 |
80 |
FP16 |
Throughput |
1 |
28 |
H100 |
80 |
FP16 |
Latency |
2 |
28 |
A100 |
80 |
FP16 |
Throughput |
1 |
28 |
A100 |
80 |
FP16 |
Latency |
2 |
28 |
L40S PCIe |
48 |
FP8 |
Throughput |
1 |
20.5 |
L40S PCIe |
48 |
FP8 |
Latency |
2 |
20.5 |
L40S PCIe |
48 |
FP16 |
Throughput |
1 |
28 |
A10G PCIe |
24 |
FP16 |
Throughput |
1 |
28 |
A10G PCIe |
24 |
FP16 |
Latency |
2 |
28 |
Non-optimized configuration#
The GPU Memory and Disk Space values are in GB; Disk Space is for both the container and the model.
GPUs |
GPU Memory |
Precision |
Disk Space |
|---|---|---|---|
Any NVIDIA GPU with sufficient GPU memory or on multiple, homogeneous NVIDIA GPUs with sufficient aggregate memory and compute capability > 7.0 (8.0 for bfloat16) |
24 |
FP16 |
16 |
Meta-Llama-3-70B-Instruct#
Optimized configurations#
The GPU Memory and Disk Space values are in GB; Disk Space is for both the container and the model; Profile is for what the model is optimized.
GPU |
GPU Memory |
Precision |
Profile |
# of GPUs |
Disk Space |
|---|---|---|---|---|---|
H100 |
80 |
FP8 |
Throughput |
4 |
82 |
H100 |
80 |
FP8 |
Latency |
8 |
82 |
H100 |
80 |
FP16 |
Throughput |
4 |
158 |
H100 |
80 |
FP16 |
Latency |
8 |
158 |
A100 |
80 |
FP16 |
Throughput |
4 |
158 |
L40S PCIe |
48 |
FP8 |
Throughput |
4 |
82 |
L40S PCIe |
48 |
FP8 |
Latency |
8 |
82 |
L40S PCIe |
48 |
FP16 |
Throughput |
8 |
158 |
Non-optimized configuration#
The GPU Memory and Disk Space values are in GB; Disk Space is for both the container and the model.
GPUs |
GPU Memory |
Precision |
Disk Space |
|---|---|---|---|
Any NVIDIA GPU with sufficient GPU memory or on multiple, homogeneous NVIDIA GPUs with sufficient aggregate memory and compute capability > 7.0 (8.0 for bfloat16) |
240 |
FP16 |
100 |
Mistral-7B-Instruct-v0.3#
Optimized configurations#
The GPU Memory and Disk Space values are in GB; Disk Space is for both the container and the model; Profile is for what the model is optimized.
GPU |
GPU Memory |
Precision |
Profile |
# of GPUs |
Disk Space |
|---|---|---|---|---|---|
H100 SXM/NVLink |
80 |
FP8 |
Latency |
2 |
7.16 |
H100 SXM/NVLink |
80 |
FP16 |
Latency |
2 |
13.82 |
H100 SXM/NVLink |
80 |
FP8 |
Throughput |
1 |
7.06 |
H100 SXM/NVLink |
80 |
FP16 |
Throughput |
1 |
13.54 |
A100 SXM/NVLink |
80 |
FP16 |
Latency |
2 |
13.82 |
A100 SXM/NVLink |
80 |
FP16 |
Throughput |
1 |
13.54 |
L40S PCIe |
48 |
FP8 |
Latency |
2 |
7.14 |
L40S PCIe |
48 |
FP16 |
Latency |
2 |
13.82 |
L40S PCIe |
48 |
FP8 |
Throughput |
1 |
7.06 |
L40S PCIe |
48 |
FP16 |
Throughput |
1 |
13.54 |
Non-optimized configuration#
The GPU Memory and Disk Space values are in GB; Disk Space is for both the container and the model.
GPUs |
GPU Memory |
Precision |
Disk Space |
|---|---|---|---|
Any NVIDIA GPU with sufficient GPU memory or on multiple, homogeneous NVIDIA GPUs with sufficient aggregate memory and compute capability > 7.0 or higher (8.0 for bfloat16) |
24 |
FP16 |
16 |
Mixtral-8x7B-v0.1#
Optimized configurations#
The GPU Memory and Disk Space values are in GB; Disk Space is for both the container and the model; Profile is for what the model is optimized.
GPU |
GPU Memory |
Precision |
Profile |
# of GPUs |
Disk Space |
|---|---|---|---|---|---|
H100 SXM/NVLink |
80 |
FP8 |
Latency |
4 |
7.16 |
H100 SXM/NVLink |
80 |
FP8 |
Throughput |
2 |
7.06 |
H100 SXM/NVLink |
80 |
FP16 |
Latency |
4 |
13.82 |
H100 SXM/NVLink |
80 |
FP16 |
Throughput |
2 |
13.54 |
A100 SXM/NVLink |
80 |
FP16 |
Throughput |
4 |
13.82 |
A100 SXM/NVLink |
80 |
FP16 |
Throughput |
2 |
13.54 |
L40S PCIe |
48 |
FP16 |
Throughput |
4 |
13.82 |
Non-optimized configuration#
The GPU Memory and Disk Space values are in GB; Disk Space is for both the container and the model.
GPUs |
GPU Memory |
Precision |
Disk Space |
|---|---|---|---|
Any NVIDIA GPU with sufficient GPU memory or on multiple, homogeneous NVIDIA GPUs with sufficient aggregate memory and compute capability > 7.0 or higher (8.0 for bfloat16) |
24 |
FP16 |
16 |
Mixtral-8x22B-v0.1#
Optimized configurations#
The GPU Memory and Disk Space values are in GB; Disk Space is for both the container and the model; Profile is for what the model is optimized.
GPU |
GPU Memory |
Precision |
Profile |
# of GPUs |
Disk Space |
|---|---|---|---|---|---|
H100 SXM/NVLink |
80 |
FP8 |
Throughput |
8 |
132.61 |
H100 SXM/NVLink |
80 |
Int4wo |
Throughput |
8 |
134.82 |
H100 SXM/NVLink |
80 |
FP16 |
Throughput |
8 |
265.59 |
A100 SXM/NVLink |
80 |
FP16 |
Throughput |
8 |
265.7 |
Non-optimized configuration#
The GPU Memory and Disk Space values are in GB; Disk Space is for both the container and the model.
GPUs |
GPU Memory |
Precision |
Disk Space |
|---|---|---|---|
Any NVIDIA GPU with sufficient GPU memory or on multiple, homogeneous NVIDIA GPUs with sufficient aggregate memory |
240 |
FP16 |
100 |
Supported LoRA formats#
The following LoRA formats are supported:
Foundation Model |
HuggingFace Format |
NeMo Format |
|---|---|---|
Meta-Llama3-8b-Instruct |
Yes |
Yes |
Meta-Llama3-70b-Instruct |
Yes |
Yes |