Prerequisites

• Begin with Docker-supported operating system
• Install Docker - minimum version: 23.0.1
• Install NVIDIA Drivers - minimum version: 580
• Install the NVIDIA Container Toolkit - minimum version: 1.13.5
• Verify your container runtime supports NVIDIA GPUs by running

docker run --rm --gpus all ubuntu nvidia-smi

Example output:

+-----------------------------------------------------------------------------+
| NVIDIA-SMI 580.82.07    Driver Version: 580.82.07    CUDA Version: 13.0     |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|                               |                      |               MIG M. |
|===============================+======================+======================|
|   0  NVIDIA GeForce ...  Off  | 00000000:01:00.0 Off |                  N/A |
| 41%   30C    P8     1W / 260W |   2244MiB / 11264MiB |      0%      Default |
|                               |                      |                  N/A |
+-------------------------------+----------------------+----------------------+
+-----------------------------------------------------------------------------+
| Processes:                                                                  |
|  GPU   GI   CI        PID   Type   Process name                  GPU Memory |
|        ID   ID                                                   Usage      |
|=============================================================================|
+-----------------------------------------------------------------------------+

NGC (NVIDIA GPU Cloud) Account

1. Create an account on NGC
2. Generate an API Key
3. Docker log in with your NGC API key using docker login nvcr.io --username='$oauthtoken' --password=${NGC_API_KEY}

NGC CLI Tool

1. Download the NGC CLI tool for your OS.

Use NGC CLI version 3.41.1 or newer. Here is the command to install this on AMD64 Linux in your home directory:

wget --content-disposition https://api.ngc.nvidia.com/v2/resources/nvidia/ngc-apps/ngc_cli/versions/3.41.3/files/ngccli_linux.zip -O ~/ngccli_linux.zip && \
unzip ~/ngccli_linux.zip -d ~/ngc && \
chmod u+x ~/ngc/ngc-cli/ngc && \
echo "export PATH=\"\$PATH:~/ngc/ngc-cli\"" >> ~/.bash_profile && source ~/.bash_profile

2. Set up your NGC CLI Tool locally (You’ll need your API key for this!):

ngc config set

After you enter your API key, you may see multiple options for the org and team. Select as desired or hit enter to accept the default.

3. Log in to NGC

You’ll need log in to NGC via Docker and set the NGC_API_KEY environment variable to pull images:

docker login nvcr.io
Username: $oauthtoken
Password: <Enter your NGC key here>

Then, set the relevant environment variables in your shell. You will need to set the NGC_API_KEY variable:

export NGC_API_KEY=<Enter your NGC key here>

4. Set up your NIM cache

The NIM cache allows you to download models and store previously-downloaded models so that you don’t need to download them again later when you run the NIM again. The NIM cache must be readable and writable by the NIM, so in addition to creating the directory, the permissions on this directory need to be set to globally readable writable. The NIM cache directory can be set up as follows:

## Create the NIM cache directory
mkdir -p ~/.cache/nim
## Set the NIM cache directory permissions to the correct values
chmod -R 777 ~/.cache/nim

Now, you should be able to pull the container and download the model using the environment variables. To get started, see the quickstart guide.