Is this page helpful?

Configuration#

NIM LLM and VLM offers several configuration options using environment variables to control caching, logging, and model profiles. Use this page to export credentials, map a persistent host cache, and optionally set a model-free source.

Export the API Key#

To use your API key when starting the NIM container, you must make it available as an environment variable.

Model-Specific NIM

Export NGC API Key

Export the NGC Personal API key so the container can use it when the image or model artifacts require credentials.

Note

An NGC API key is only required to download Production Branch (PB) models or NIMs released prior to NIM LLM and VLM version 2.0.10. You can skip this section if you did not create an NGC API key.

Export the variable in your shell (temporary), replacing VALUE with your actual API key:
```
export NGC_API_KEY=VALUE
```

Optional: Persist the variable in your shell profile.

If using bash:

echo "export NGC_API_KEY=$NGC_API_KEY" >> ~/.bashrc

If using zsh:

echo "export NGC_API_KEY=$NGC_API_KEY" >> ~/.zshrc

Verify the variable is set:
```
echo "$NGC_API_KEY"
```

Model-Free NIM

Export Hugging Face Access Token

Export a Hugging Face token when the model-free NIM container must download from Hugging Face.

Complete the steps in the Model-Specific NIM tab to export your NGC Personal API key.
Export the variable in your shell (temporary), replacing <token-value> with your actual token:
```
export HF_TOKEN="<token-value>"
```

Optional: Persist the variable in your shell profile for future terminals.

If using bash:

echo 'export HF_TOKEN="<token-value>"' >> ~/.bashrc

If using zsh:

echo 'export HF_TOKEN="<token-value>"' >> ~/.zshrc

Verify the variable is set:
```
echo "$HF_TOKEN"
```

Note

If you want to serve a pre-downloaded local model or a private cloud model instead of downloading one from Hugging Face, you do not need a Hugging Face access token. Refer to Model Downloads for your workflow.

Important

For enhanced security, consider storing your token in a file and retrieving it as needed with cat. Alternatively, consider using a password manager.

Model Cache and Source#

NIM downloads model weights and other assets to a cache directory inside the container. Map a persistent directory on your host machine to that path so files survive restarts and later deployments can reuse them.

Local Cache#

An essential host variable is LOCAL_NIM_CACHE, which maps the host cache directory to the container cache path. Assets (for example, model weights) download to this host directory and persist across container restarts. This guide uses LOCAL_NIM_CACHE=~/.cache/nim in all examples to keep commands consistent and copy-paste safe.

To create and map the local cache, complete the following steps:

Create the cache directory and export LOCAL_NIM_CACHE:

export LOCAL_NIM_CACHE=~/.cache/nim
mkdir -p "$LOCAL_NIM_CACHE"

Optional: Add the sticky bit to avoid issues writing to the cache if the container is running as a different user:
```
chmod -R a+rwxt "$LOCAL_NIM_CACHE"
```

When you start the NIM container, you must map your host machine’s local cache directory ($LOCAL_NIM_CACHE) to the container’s internal cache path (/opt/nim/.cache) using a Docker volume mount, such as -v "$LOCAL_NIM_CACHE:/opt/nim/.cache". This mapping ensures that the large model weights downloaded by the container are saved to your host machine. Because containers are ephemeral, any data stored only inside the container is lost when it stops. By using a volume mount, subsequent container runs detect the existing model files in your local cache and skip the lengthy download process, allowing the NIM to start up faster.

Cache Directory Permissions#

The NIM container runs as a non-root user with GID 0 (root group). The cache directory on your host must be writable by GID 0. To set permissions and mount the cache, complete the following steps:

Set group ownership so GID 0 can write to the cache:

sudo chgrp -R 0 "$LOCAL_NIM_CACHE"
sudo chmod -R g+rwX "$LOCAL_NIM_CACHE"

Run the container with the cache mounted:

docker run --gpus all \
  -v "$LOCAL_NIM_CACHE:/opt/nim/.cache" \
  ...

To run as a custom user (for example, your host user), pass -u <uid>:0:

docker run --gpus all -u $(id -u):0 \
  -v "$LOCAL_NIM_CACHE:/opt/nim/.cache" \
  ...

Important

When using -u <uid>, you must include :0 to set GID 0 (for example, -u $(id -u):0). The container’s writable directories are group-owned by GID 0. Without it, the container fails with PermissionError when writing to cache, config, or log paths.

Tip

To make this setting permanent across terminal sessions, you can add export LOCAL_NIM_CACHE=~/.cache/nim to your ~/.bashrc or ~/.zshrc profile.

Custom Model Source (Model-Free NIM Only)#

You can configure model-free NIM to download a model from Hugging Face or other supported sources. To do this, specify the directory where the model files should be stored and set the appropriate environment variable.

To set a Hugging Face model source, complete the following steps:

Specify a Hugging Face model directly using the hf:// prefix:
```
export NIM_MODEL_PATH="hf://openai/gpt-oss-20b"
```
Map $LOCAL_NIM_CACHE to /opt/nim/.cache when you run the container.

NIM automatically downloads the model from Hugging Face and caches it in your configured LOCAL_NIM_CACHE. When running the container, map your host machine’s local cache directory to a path inside the container using the -v flag. This ensures that the downloaded model weights persist on your host machine across ephemeral container restarts.

For example, using -v "$LOCAL_NIM_CACHE:/opt/nim/.cache" tells Docker: “Take the folder at $LOCAL_NIM_CACHE on my host machine, and make it available inside the container at the path /opt/nim/.cache.”

Note

If you plan to use a model source other than Hugging Face, such as a pre-downloaded local model or private cloud storage, the URI and path are different. Refer to Model Download for instructions.

Advanced Configurations#

For production deployments or specific organizational requirements, you can configure additional environment variables. NIM LLM and VLM supports advanced settings such as:

TLS/SSL Configuration (NIM_SSL_MODE, NIM_SSL_KEY_PATH, NIM_SSL_CERTS_PATH, NIM_SSL_CA_CERTS_PATH): Control whether SSL/TLS is enabled for API connections, and specify the locations of SSL private keys, certificates, and CA certificates.
Unified Structured Logging and Verbosity (NIM_JSONL_LOGGING, NIM_LOG_LEVEL): Enable structured JSONL log output and set the logging verbosity level for debugging or monitoring.
Manual Model Profile Overrides (NIM_MODEL_PROFILE): Manually select or override the model execution profile, allowing control over parameters like precision or hardware acceleration.

For more details on these advanced settings, refer to Advanced Configurations.