Configure The NIM at Runtime

GPU Selection

On a system with a single NVIDIA GPU, you can use the --gpus device=0 option in docker command to start the service.

In heterogeneous environments with a combination of GPUs, such as L40S and a GeForce display GPU, you should only run workload on the L40S GPU. Use the --gpus device=<GPU ID> option in docker command to start the service.

Use the nvidia-smi -L command to get the device ID(s) to use as input(s). This command should return information similar to the following:

GPU 0: NVIDIA GeForce RTX 3080

GPU 1: L40S

For more information, see NVIDIA Container Toolkit documentation.

Environment Variables

You can pass the following environment variables into a NIM by adding the -e flag to the docker run command.

ENV	Required?	Default	Notes
NGC_API_KEY	Yes	None	Your NGC API key with read access to the model registry for the model profile you are using.
NV_SCRATCH_SPACE	Yes	None	You should provide the same mount path as used in the service launch command. NIM uses this space to store intermediate artifacts.
NIM_LOG_LEVEL	Optional	None	You can pass DEBUG value to enable extra logging.
NIM_TRITON_LOG_VERBOSE	Optional	None	You can pass verbosity level [1,2,3,4,5] to enable extra logging from the triton server backend.
S3_ENDPOINT_URL	Optional	None	S3 endpoint URL required to establish connection.
S3_TEAM_NAME	Optional	None	S3 Team name required to establish connection.
S3_ACCESS_KEY	Optional	None	S3 access key required to establish connection.
S3_REGION_NAME	Optional	None	S3 region name required to establish connection.