Is this page helpful?

Prerequisites#

Before deploying a NIM LLM container, ensure your environment meets the following requirements:

Hardware Requirements#

Minimum required specifications for supported hardware components.

Requirement

Specification

CPU

AMD64, ARM64

GPU

Any supported NVIDIA GPU

Model GPU Memory Requirements#

Different models require different minimum amounts of GPU memory. For example, Llama 3.1 8B Instruct requires a minimum of 24GB of GPU memory.

Ensure your chosen hardware configuration meets the minimum GPU memory requirement for the model profile you wish to run. For more information, refer to Model Profiles and Support Matrix.

Software Requirements#

Minimum required versions for supported software components.

Requirement

Specification

Operating System

Ubuntu 22.04 LTS or later recommended

Container Toolkit

1.14.0 or later

CUDA SDK

12.9 or later

GPU Driver

580 or later

Docker

24.0 or later

Operating System#

While other Linux distributions can be compatible with NIM LLM, they have not been officially validated.

We recommend using Ubuntu 22.04 LTS or later for the best experience.

CUDA SDK#

Install CUDA SDK by following the CUDA installation guide for Linux.

GPU Drivers#

Install the NVIDIA GPU drivers by following the NVIDIA Driver Installation Guide.

Docker#

Docker is required to run the containerized NIM services.

  1. Install Docker Engine for your Linux distribution by following the Docker Engine installation guide.

  2. Verify that the Docker daemon is running and that your user can execute docker commands without sudo. Add your user to the docker group if needed:

    sudo groupadd docker
sudo usermod -aG docker $USER

  3. Log out and back in for the group change to take effect.

Container Toolkit#

The NVIDIA Container Toolkit enables Docker containers to access the host GPU.

  1. Install the toolkit by following the NVIDIA Container Toolkit installation guide.

  2. Configure Docker to use the NVIDIA runtime by following the Docker configuration steps.

  3. Restart the Docker daemon after configuration:

    sudo systemctl restart docker

Dynamo Mode Requirements#

The preceding host requirements also apply when you run Dynamo mode. The Ubuntu and CUDA versions in a Dynamo-enabled image describe the container userland and do not require the host to run Ubuntu 24.04 or to install the CUDA 13.x SDK.

Published NIM 3.0 images for the models listed in the Dynamo mode support matrix already include the NVIDIA Dynamo runtime. Use the image and profile pinned by the model-specific recipe in the model’s NGC collection.

Scope

Requirement

Specification

Host, GPU deployment

GPU Driver

580 or later

Container image

Backend

vLLM

Container image

Operating System Userland

Ubuntu 24.04

Container image, GPU variant

CUDA Runtime

13.x

Note

The GPU image contains a CUDA 13.x runtime, so the host driver must support that runtime. The host does not need a separate CUDA SDK installation. A deployment on a driver that is too old can fail at startup with RuntimeError: The NVIDIA driver on your system is too old. For the available recipe configurations, deployment modes, platform test paths, and component versions, refer to Dynamo Mode Support.

NIM Container Access#

To download and deploy NIM containers, you need one of the following:

Generate Access Credentials#

NGC Personal API Key

An NGC Personal API key is not required to access most NVIDIA NIM containers and models hosted on NGC. An NGC API key is only required to download Production Branch (PB) models or NIMs released prior to NIM LLM version 2.0.10. This Get Started documentation uses a current feature branch model as an example, so creating an NGC Personal API key is optional.

If you want to create an NGC Personal API key, do the following:

  1. Generate the Personal API Key on the Setup API Keys page.

  2. When creating the Personal API key, select at least NGC Catalog from the Services Included list. You can also include additional services if you want to use the same key for other purposes.

Warning

Legacy API keys are not supported by NIM LLM. Always use a Personal API Key.

Hugging Face Access Token

The required credentials depend on your model source.

  1. Complete the steps in the Model-Specific NIM tab to generate your NGC Personal API key. You will need one to pull the NIM container itself.

  2. To download models, create a Hugging Face access token with read permissions (or higher, only if your org or private repos require it). This token allows model-free NIM to fetch models directly from Hugging Face.

    Note

    If you want to serve a pre-downloaded local model or a private cloud model instead of downloading one from Hugging Face, you do not need a Hugging Face access token. Refer to Model Downloads for your workflow.

Verify NVIDIA Runtime Access#

To ensure that your setup is correct, run the following command:

docker run --rm --runtime=nvidia --gpus all ubuntu nvidia-smi

This command should produce output similar to one of the following, where you can confirm CUDA driver version, and available GPUs.

+-----------------------------------------------------------------------------------------+
| NVIDIA-SMI 580.95.05              Driver Version: 580.95.05      CUDA Version: 12.9     |
|-----------------------------------------+------------------------+----------------------+
| GPU  Name                 Persistence-M | Bus-Id          Disp.A | Volatile Uncorr. ECC |
| Fan  Temp   Perf          Pwr:Usage/Cap |           Memory-Usage | GPU-Util  Compute M. |
|                                         |                        |               MIG M. |
|=========================================+========================+======================|
|   0  NVIDIA H100 80GB HBM3          On  |   00000000:1B:00.0 Off |                    0 |
| N/A   36C    P0            112W /  700W |   78489MiB /  81559MiB |      0%      Default |
|                                         |                        |             Disabled |
+-----------------------------------------+------------------------+----------------------+

+-----------------------------------------------------------------------------------------+
| Processes:                                                                              |
|  GPU   GI   CI        PID   Type   Process name                              GPU Memory |
|        ID   ID                                                               Usage      |
|=========================================================================================|
|  No running processes found                                                             |
+-----------------------------------------------------------------------------------------+