Configuration | Llama 3.1 NemoGuard 8B TopicControl NIM

Llama 3.1 NemoGuard 8B TopicControl NIM is packaged as a container. You can set environment variables and specify command-line arguments to configure the microservice.

GPU Selection

The container is GPU-accelerated and uses NVIDIA Container Toolkit for access to GPUs on the host.

You can specify the --gpus all command-line argument to the docker run command if the host has one or more of the same GPU model. If the host has a combination of GPUs, run the container on compute-capable GPUs only.

Expose specific GPUs to the container by using either of the following methods:

Specify the --gpus argument, such as --gpus="device=1".
Set the NVIDIA_VISIBLE_DEVICES environment variable, such as -e NVIDIA_VISIBLE_DEVICES=1.

Run the nvidia-smi -L command to list the device IDs to specify in the argument or environment variable:

GPU 0: Tesla H100 (UUID: GPU-b404a1a1-d532-5b5c-20bc-b34e37f3ac46)
GPU 1: NVIDIA GeForce RTX 3080 (UUID: GPU-b404a1a1-d532-5b5c-20bc-b34e37f3ac46)

Refer to GPU Enumeration in the NVIDIA Container Toolkit documentation for more information.

Environment Variables

The following table identifies the environment variables that are used in the container. Set environment variables with the -e command-line argument to the docker run command.

Name	Description	Default Value
`NGC_API_KEY`	Set this variable to the value of your personal NGC API key.	None
`NIM_CACHE_PATH`	Specifies the fully qualified path, in the container, for downloaded models.	`/opt/nim/.cache`
`NIM_GRPC_API_PORT`	Specifies the network port number, in the container, for gRPC access to the microservice.	`50051`
`NIM_HTTP_API_PORT`	Specifies the network port number, in the container, for HTTP access to the microservice.Refer to Publishing ports
in the Docker documentation for more information about host and container network ports.	`8000`
`NIM_HTTP_MAX_WORKERS`	Specifies the number of worker threads to start for HTTP requests.	`1`
`NIM_HTTP_TRITON_PORT`	Specifies the network port number, in the container, for NVIDIA Triton Inference Server.	`8080`
`NIM_IGNORE_MODEL_DOWNLOAD_FAIL`	When set to `true` and the microservice fails to download a model from NGC, the microservice continues to run rather than exit.
This environment variable can be useful in an air-gapped environment.	`false`
`NIM_LOG_LEVEL`	Specifies the logging level.
The microservice supports the following values: DEBUG, INFO, WARNING, ERROR, and CRITICAL.	`INFO`
`NIM_LOGGING_JSONL`	When set to `true`, the microservice creates log records in the JSONL format.	`false`
`NIM_MANIFEST_PATH`	Specifies the fully qualified path, in the container, for the model manifest YAML file.	`/opt/nim/etc/default/model_manifest.yaml`
`NIM_MODEL_PROFILE`	Specifies the model profile ID to use with the container.
By default, the container attempts to automatically match the host GPU model and GPU count with the optimal model profile.	None
`NIM_SERVED_MODEL_NAME`	Specifies the model names used in the API.
Specify multiple names in a comma-separated list.
If you specify multiple names, the server responds to any of the names.
The name in the model field of a response is the first name in this list.
By default, the model is inferred from the `model_manifest.yaml`.	None
`NIM_TRITON_GRPC_PORT`	Specifies the gRPC port number, for NVIDIA Triton Inference Server.	`8001`
`NIM_TRITON_LOG_VERBOSE`	When set to `1`, the container starts NVIDIA Triton Inference Server with verbose logging.	`0`

Volumes

The container stores downloaded models in /opt/nim/.cache by default. Set NIM_CACHE_PATH to use a different path inside the container.

Container path or variable	Description
`/opt/nim/.cache` or `NIM_CACHE_PATH`	Container location for downloaded models.

To keep the model cache between runs, create a directory on the host:

mkdir -p ~/.cache/nim

Bind-mount it to the container path when you run the container:

-v ~/.cache/nim:/opt/nim/.cache -u "$(id -u)"

Without a bind mount, the container downloads the model again each time it starts. The -u "$(id -u)" option runs the container with your user ID to avoid file permission issues. If you set NIM_CACHE_PATH to a different container path, use that path as the bind mount destination.