Is this page helpful?

Configuration#

NIM LLM offers several configuration options using environment variables to control caching, logging, and model profiles.

Export the API key#

To use your API key when starting the NIM container, you must make it available as an environment variable.

Model-Specific NIM

Export NGC API Key

Export the variable in your shell (temporary), replacing VALUE with your actual API key:
```
export NGC_API_KEY=VALUE
```

Persist the variable (optional):

If using bash:

echo "export NGC_API_KEY=$NGC_API_KEY" >> ~/.bashrc

If using zsh:

echo "export NGC_API_KEY=$NGC_API_KEY" >> ~/.zshrc

Verify the variable is set:
```
echo "$NGC_API_KEY"
```

Model-Free NIM

Export Hugging Face Access Token

Complete the steps in the Model-Specific NIM tab to export your NGC Personal API key.
Export the variable in your shell (temporary), replacing <token-value> with your actual token:
```
export HF_TOKEN="<token-value>"
```

Persist it (optional, for future terminals):

If using bash:

echo 'export HF_TOKEN="<token-value>"' >> ~/.bashrc

If using zsh:

echo 'export HF_TOKEN="<token-value>"' >> ~/.zshrc

Verify the variable is set:
```
echo "$HF_TOKEN"
```

Note

If you want to serve a pre-downloaded local model or a private cloud model instead of downloading one from Hugging Face, you do not need a Hugging Face access token. Refer to Model Downloads for your workflow.

Important

For enhanced security, consider storing your token in a file and retrieving it as needed with cat. Alternatively, consider using a password manager.

Model Cache and Source#

Model-Specific NIM

Local Cache

An essential variable to configure on your host system is the cache path directory. This directory is mapped from the host machine to container; assets (for example, model weights) are downloaded to this host directory and persist across container restarts. Configuring a local cache is highly recommended, as it avoids re-downloading large model files upon subsequent container restarts. You can name the environment variable containing the path to the local cache whatever you want.

Create the cache directory and export an environment variable:

export LOCAL_NIM_CACHE=~/.cache/nim
mkdir -p $LOCAL_NIM_CACHE
# Optionally add sticky bit to avoid issues writing to the cache if the container is running as a different user
chmod -R a+rwxt $LOCAL_NIM_CACHE

When you start the NIM container, you must map this directory to the container’s internal cache path (/opt/nim/.cache) using a Docker volume mount. -v "$LOCAL_NIM_CACHE:/opt/nim/.cache"

Tip

To make this setting permanent across terminal sessions, you can add export NIM_CACHE_PATH=~/.cache/nim to your ~/.bashrc or ~/.zshrc profile.

Model-Free NIM

Custom Model Source

After you have set up your local cache, you can configure model-free NIM to download a model from Hugging Face. To do this, specify the directory where the model files should be stored and set the appropriate environment variable.

Specify a Hugging Face model directly using the hf:// prefix.

export NIM_MODEL_PATH="hf://openai/gpt-oss-20b"

The NIM will automatically download the model from Hugging Face and cache it in your configured NIM_CACHE_PATH. When running the container, you will mount this cache directory to a path inside the container using the -v flag.

For example, using -v "$NIM_CACHE_PATH:/opt/nim/.cache" tells Docker: “Take the folder at $NIM_CACHE_PATH on my host machine, and make it available inside the container at the path /opt/nim/.cache.”

Note

If you plan to use a model source other than Hugging Face, such as a pre-downloaded local model or private cloud storage, the URI and path will be different. Refer to model-download-sources for instructions.

Advanced Configurations#

For production deployments or specific organizational requirements, you can configure additional environment variables. NIM LLM supports advanced settings such as:

TLS/SSL Configuration (NIM_SSL_MODE, NIM_SSL_KEY_PATH, NIM_SSL_CERTS_PATH, NIM_SSL_CA_CERTS_PATH): Control whether SSL/TLS is enabled for API connections, and specify the locations of SSL private keys, certificates, and CA certificates.
Unified Structured Logging and Verbosity (NIM_JSONL_LOGGING, NIM_LOG_LEVEL): Enable structured JSONL log output and set the logging verbosity level for debugging or monitoring.
Manual Model Profile Overrides (NIM_MODEL_PROFILE): Manually select or override the model execution profile, allowing control over parameters like precision or hardware acceleration.

For more details on these advanced settings, refer to Advanced Configurations.