Configuring a NIM#

NVIDIA NIM for LLMs (NIM for LLMs) uses Docker containers under the hood. Each NIM is its own Docker container and there are several ways to configure it. Below is a full reference of all the ways to configure a NIM container.

GPU Selection#

Passing --gpus all to docker run is acceptable in homogeneous environments with one or more of the same GPU.

Note

--gpus all only works if your configuration has the same number of GPUs as specified for the model in the Supported Models. Running an inference on a configuration with fewer or more GPUs can result in a runtime error.

In heterogeneous environments with a combination of GPUs (for example: A6000 + a GeForce display GPU), workloads should only run on compute-capable GPUs. Expose specific GPUs inside the container using either:

  • the --gpus flag (ex: --gpus='"device=1"')

  • the environment variable NVIDIA_VISIBLE_DEVICES (ex: -e NVIDIA_VISIBLE_DEVICES=1)

The device ID(s) to use as input(s) are listed in the output of nvidia-smi -L:

GPU 0: Tesla H100 (UUID: GPU-b404a1a1-d532-5b5c-20bc-b34e37f3ac46)
GPU 1: NVIDIA GeForce RTX 3080 (UUID: GPU-b404a1a1-d532-5b5c-20bc-b34e37f3ac46)

Refer to the NVIDIA Container Toolkit documentation for more instructions.

How many GPUs do I need?#

Each Profile will have a TP (Tensor Parallelism) and PP (Pipeline Parallelism), decipherable through their readable name (example: tensorrt_llm-trtllm_buildable-bf16-tp8-pp2).

In most cases, you will need TP * PP amount of GPUs to run a specific profile.

For example, for the profile tensorrt_llm-trtllm_buildable-bf16-tp8-pp2 you will need either 2 nodes with 8 GPUs or 2 * 8 = 16 GPUs on one Node.

Shared memory flag#

Passing --shm-size=16GB to docker run is required when not using NVLink for multi-GPU setups. It is not required on SXM systems or when using profiles using only 1 GPU (e.g NIM_TENSOR_PARALLEL_SIZE=1).

Environment Variables#

Below is a reference for REQUIRED and No environment variables that can be passed into a NIM (-e added to docker run):

ENV

Required?

Default

Notes

NGC_API_KEY

Yes

None

You must set this variable to the value of your personal NGC API key.

NIM_CACHE_PATH

No

/opt/nim/.cache

Location (in container) where the container caches model artifacts.

NIM_DISABLE_LOG_REQUESTS

No

1

Set to 0 to view request logs. By default, logs of request details to v1/completions and v1/chat/completions are disabled. These logs contain sensitive attributes of the request including prompt, sampling_params, and prompt_token_ids. Users should be aware that these attributes will be exposed to container logs when enabling this parameter.

NIM_JSONL_LOGGING

No

0

Set to 1 to enable JSON-formatted logs. Readable text logs are enabled by default.

NIM_LOG_LEVEL

No

DEFAULT

Log level of NIM for LLMs service. Possible values of the variable are DEFAULT, TRACE, DEBUG, INFO, WARNING, ERROR, CRITICAL. Mostly, the effect of DEBUG, INFO, WARNING, ERROR, CRITICAL is described in Python 3 logging docs. TRACE log level enables printing of diagnostic information for debugging purposes in TRT-LLM and in uvicorn. When NIM_LOG_LEVEL is DEFAULT sets all log levels to INFO except for TRT-LLM log level which equals ERROR. When NIM_LOG_LEVEL is CRITICAL TRT-LLM log level is ERROR.

NIM_SERVER_PORT

No

8000

Publish the NIM service to the prescribed port inside the container. Make sure to adjust the port passed to the -p/--publish flag of docker run to reflect that (ex: -p $NIM_SERVER_PORT:$NIM_SERVER_PORT). The left-hand side of this : is your host address:port, and does NOT have to match with $NIM_SERVER_PORT. The right-hand side of the : is the port inside the container which MUST match NIM_SERVER_PORT (or 8000 if not set).

NIM_MODEL_PROFILE

No

None

Override the NIM optimization profile that is automatically selected by specifying a profile ID from the manifest located at /opt/nim/etc/default/model_manifest.yaml. If not specified, NIM will attempt to select an optimal profile compatible with available GPUs. A list of the compatible profiles can be obtained by appending list-model-profiles at the end of the docker run command. Using the profile name default will select a profile that is maximally compatible and may not be optimal for your hardware.

NIM_MANIFEST_ALLOW_UNSAFE

No

0

If set to 1, enable selection of a model profile not included in the original model_manifest.yaml. If set, you must also set NIM_MODEL_NAME to be the path to the model directory or an NGC path.

NIM_MODEL_NAME

No

“Model Name”

Must be set only if NIM_MANIFEST_ALLOW_UNSAFE is set to 1. This must be a path to a model directory or an NGC path of the form ngc://<org>/<team>/<model_name>:<version>. An example: ngc://nim/meta/llama3-8b-instruct:hf.

NIM_PEFT_SOURCE

No

If you want to enable PEFT inference with local PEFT modules, then set a NIM_PEFT_SOURCE environment variable and pass that into the run container command. If your PEFT source is a local directory at LOCAL_PEFT_DIRECTORY, mount your local PEFT directory to the container’s PEFT source set by NIM_PEFT_SOURCE. Make sure that your directory only contains PEFT modules for the base NIM. Also make sure that the PEFT directory and all the contents inside it are readable by NIM.

NIM_MAX_LORA_RANK

No

32

Set the maximum LoRA rank.

NIM_MAX_GPU_LORAS

No

8

Set the number of LoRAs that can fit in GPU PEFT cache. This is the maximum number of LoRAs that can be used in a single batch.

NIM_MAX_CPU_LORAS

No

16

Set the number of LoRAs that can fit in CPU PEFT cache. This should be set >= max concurrency or the number of LoRAs you are serving, whichever is less. If you have more concurrent LoRA requests than NIM_MAX_CPU_LORAS you may see “cache is full” errors. This value must be >= NIM_MAX_GPU_LORAS.

NIM_PEFT_REFRESH_INTERVAL

No

None

How often to check NIM_PEFT_SOURCE for new models in seconds. If not set, PEFT cache will not refresh. If you choose to enable PEFT refreshing by setting this ENV var, we recommend setting the number greater than 30.

NIM_SERVED_MODEL_NAME

No

None

The model name(s) used in the API. If multiple names are provided (comma-separated), the server will respond to any of the provided names. The model name in the model field of a response will be the first name in this list. If not specified, the model name will be inferred from the manifest located at /opt/nim/etc/default/model_manifest.yaml. Note that this name(s) will also be used in model_name tag content of Prometheus metrics, if multiple names provided, metrics tag will take the first one.

NIM_CUSTOM_MODEL_NAME

No

None

The model name given to a locally-built engine. If set, the locally-built engine will be named NIM_CUSTOM_MODEL_NAME and be cached with the same name in the NIM Cache. This name must be non-duplicate to other cached custom engines. This cached engine will also be visible with the same name with the list-model-profiles command and will behave like every other profile. On subsequent docker runs, a locally cached engine will take precedence over every other type of profile. You may also set NIM_MODEL_PROFILE to be a specific custom model name to force NIM LLM to serve that cached engine.

NIM_LOW_MEMORY_MODE

No

0

Set this flag to 1 to enable offloading the locally-built TRTLLM engines to disk. This reduces runtime host memory requirement, but may increase the startup time and disk usage.

NIM_ENABLE_OTEL

No

0

Set this flag to 1 to enable OpenTelemetry instrumentation in NIMs.

NIM_OTEL_TRACES_EXPORTER

No

console

Specifies the OpenTelemetry exporter to use for tracing. Set this flag to otlp to export the traces using the OpenTelemetry Protocol. Set it to console to print the traces to standard output.

NIM_OTEL_METRICS_EXPORTER

No

console

Similar to NIM_OTEL_TRACES_EXPORTER, but for metrics.

NIM_OTEL_EXPORTER_OTLP_ENDPOINT

No

None

The endpoint where the OpenTelemetry Collector is listening for OTLP data. Adjust the URL to match your OpenTelemetry Collector’s configuration.

NIM_OTEL_SERVICE_NAME

No

None

Sets the name of your service to help with identifying and categorizing data.

NIM_TOKENIZER_MODE

No

auto

The tokenizer mode. auto will use the fast tokenizer if available. slow will always use the slow tokenizer.

NIM_ENABLE_KV_CACHE_REUSE

No

0

Set to 1 to enable automatic prefix caching / KV cache reuse. For use cases where large prompts frequently appear and a cache for KV caches across requests would speed up inference.

NIM_RELAX_MEM_CONSTRAINTS

No

0

If set to 1 and NIM_NUM_KV_CACHE_SEQ_LENS not specified then NIM_NUM_KV_CACHE_SEQ_LENS will automatically be set to 1. Otherwise if set to 1 will use value provided from NIM_NUM_KV_CACHE_SEQ_LENS. The recommended default for NIM LLM is for all GPUs to have >= 95% of memory free. Setting this variable to true overrides this default and will run the model regardless of memory constraints. It will also use heuristics to determine if GPU will likely meet or fail memory requirements and will provide a warning if applicable.

NIM_NUM_KV_CACHE_SEQ_LENS

No

None

NIM_RELAX_MEM_CONSTRAINTS must be set to 1 for this environment variable to take effect. Set to a value greater than or equal to 1 to override the default KV cache memory allocation settings for NIM LLM. The value provided will be used to determine how many maximum sequence lengths can fit within the KV cache (for example 2 or 3.75). The maximum sequence length is the context size of the model.

NIM_MAX_MODEL_LEN

No

None

Model context length. If unspecified, will be automatically derived from the model configuration. Note that this setting has an effect on only models running on the vLLM backend and models where the selected profile has trtllm-buildable equal to true. In the case where trtllm-buildable is equal to true the TRT-LLM build parameter max_seq_len will be set to this value.

NIM_REPOSITORY_OVERRIDE

No

None

If set to a non-empty string, the NIM_REPOSITORY_OVERRIDE value will replace the hard-coded location of the repository and the protocol for access to the repository. The structure of the value for this environment variable is as follows: <repository type>://<repository location>. Note that only the protocols ngc://, s3://, and https:// are supported, and only the first component of the URI is replaced. For example:
- If the URI in the manifest is ngc://org/meta/llama3-8b-instruct:hf?file=config.json and NIM_REPOSITORY_OVERRIDE=ngc://myrepo.ai/, the domain name for the API endpoint is set to myrepo.ai.
- If NIM_REPOSITORY_OVERRIDE=s3://mybucket/, the result of the replacement will be s3://mybucket/nim%2Fmeta%2Fllama3-8b-instruct%3Ahf%3Ffile%3Dconfig.json.
- If NIM_REPOSITORY_OVERRIDE=https://mymodel.ai/some_path_optional, the result of the replacement will be https://mymodel.ai/some_path/nim%2Fmeta%2Fllama3-8b-instruct%3Ahf%3Ffile%3Dconfig.json.

This repository override feature supports basic authentication mechanisms:
- https assumes authorization using the Authorization header and the credential value in NIM_HTTPS_CREDENTIAL.
- ngc requires a credential in the NGC_API_KEY environment variable.
- s3 requires the environment variables AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY, and (if using temporary credentials) AWS_SESSION_TOKEN.

NIM_GUIDED_DECODING_BACKEND

No

"outlines"

The guided decoding backend to use. Can be one of "fast_outlines" (custom NIM accelerated version of outlines), "outlines",, "lm-format-enforcer" or a custom guided decoding backend.

NIM_TRUST_CUSTOM_CODE

No

0

Set to 1 to enable custom guided decoding backend. This enables arbitrary Python code execution as part of the custom guided decoding.

NIM_SSL_MODE

No

"DISABLED"

Main switch to enable SSL/TLS or skip environment variables NIM_SSL_KEY_PATH, NIM_SSL_CERTS_PATH, NIM_SSL_CA_CERTS_PATH. Possible values: (1) "DISABLED" - no HTTPS, (2) "TLS" - HTTPS with only server-side TLS (client certificate not required), (3) "MTLS" - HTTPS with mTLS (client certificate required). If "TLS" is used, then NIM_SSL_CERTS_PATH, NIM_SSL_KEY_PATH are required. If "MTLS" is used, then NIM_SSL_CERTS_PATH, NIM_SSL_KEY_PATH, NIM_SSL_CA_CERTS_PATH are required.

NIM_SSL_KEY_PATH

Required if NIM_SSL_MODE is enabled

None

Path to the server’s TLS private key file (required for TLS HTTPS). It’s used to decrypt incoming messages and sign outgoing ones.

NIM_SSL_CERTS_PATH

Required if NIM_SSL_MODE is enabled

None

Path to the server’s certificate file (required for TLS HTTPS). It contains the public key and server identification information.

NIM_SSL_CA_CERTS_PATH

Required if NIM_SSL_MODE="MTLS"

None

Path to a the CA (certificate Authority) certificate.

This file is used to verify client certificates in mutual TLS (mTLS) setups.

Volumes#

Local paths can be mounted to the following container paths.

Container path

Required?

Notes

Docker argument example

/opt/nim/.cache (or NIM_CACHE_PATH if present)

No; however, if this volume is not mounted, the container does a fresh download of the model every time the container starts.

This directory is to where models are downloaded inside the container. You can access this directory from within the container by adding the -u $(id -u) option to the docker run command. For example, to use ~/.cache/nim as the host machine directory for caching models, first run mkdir -p ~/.cache/nim before running the docker run ... command.

-v ~/.cache/nim:/opt/nim/.cache -u $(id -u)