Is this page helpful?

Environment Variables#

This page documents all environment variables supported by NIM LLM. Set variables using -e flags when you run the container:

docker run -d --rm --gpus all \
  -p 8000:8000 \
  -v "$LOCAL_NIM_CACHE:/opt/nim/.cache" \
  -e NIM_MODEL_PATH=hf://meta-llama/Llama-3.1-8B-Instruct \
  -e NIM_SERVER_PORT=8000 \
  -e NIM_LOG_LEVEL=INFO \
  -e NGC_API_KEY \
  -e HF_TOKEN \
  <image>

Logging#

The following variables control log format and verbosity:

NIM_LOG_LEVEL: str | None#

Controls the verbosity of NIM log output. Accepts standard Python logging levels: DEBUG, INFO, WARNING, ERROR, CRITICAL.

Default:

None (uses application default)

Type:

string

Example:

NIM_LOG_LEVEL=DEBUG

NIM_JSONL_LOGGING: bool#

Enables structured JSON Lines (JSONL) log output.

Default:

False

Type:

boolean

Example:

NIM_JSONL_LOGGING=true

For usage details and examples, refer to Logging and Observability.

Model Configuration#

The following variables control model selection, model loading, and related runtime behavior:

NIM_MODEL_PROFILE: string = None#

Selects which model profile to use. Profiles define a validated combination of model variant, precision, and parallelism settings for a given GPU configuration. Run list-model-profiles inside the container to see available profiles and their IDs.

Default:

auto-selected based on detected GPU hardware

Example:

NIM_MODEL_PROFILE=07cd4f2bddd7a14ca84bab0a32602889fd0ae0eb76dc2eb0fc32594d065011a4

NIM_MODEL_PATH: str | None#

Model source URI or local filesystem path. Accepts hf://, ngc://, and modelscope:// prefixes for remote repositories, or a local directory path. When set, a runtime manifest is generated from this URI instead of using the baked-in container manifest.

Default:

None (uses baked-in manifest and NIM_MODEL_PROFILE)

Type:

string

Example:

NIM_MODEL_PATH=hf://meta-llama/Llama-3.1-8B-Instruct

NIM_SERVED_MODEL_NAME: str | None#

Overrides the served model name returned in API responses. When set, the /v1/models endpoint and response metadata use this name instead of the default model identifier.

Default:

None (uses the model’s own identifier)

Type:

string

Example:

NIM_SERVED_MODEL_NAME=my-llama

NIM_MAX_MODEL_LEN: int | None#

Overrides the maximum sequence length (context window) for the model. Values larger than the model’s trained maximum may cause errors.

Default:

None (uses model’s default from config)

Type:

positive integer

Example:

NIM_MAX_MODEL_LEN=4096

NIM_TENSOR_PARALLEL_SIZE: int | None#

Overrides the tensor parallelism degree. Splits model layers across the specified number of GPUs for inference.

Default:

None (auto-detected from profile)

Type:

positive integer

Example:

NIM_TENSOR_PARALLEL_SIZE=2

NIM_PIPELINE_PARALLEL_SIZE: int | None#

Overrides the pipeline parallelism degree. Distributes model stages across the specified number of GPUs for inference.

Default:

None (auto-detected from profile)

Type:

positive integer

Example:

NIM_PIPELINE_PARALLEL_SIZE=2

NIM_NUM_COMPUTE_NODES: integer = None#

Total number of compute nodes for multi-node inference. In multi-node deployments, set this on both the leader and worker nodes to the total node count (leader + workers).

Default:

None (single-node operation)

Example:

NIM_NUM_COMPUTE_NODES=2

NIM_REPOSITORY_OVERRIDE: string = None#

Redirects model downloads to an external repository while preserving the NIM manifest semantics. The container still uses the baked-in manifest for profile selection, but fetches model files from the overridden source.

Default:

None (downloads from the URI specified in the manifest)

Example:

NIM_REPOSITORY_OVERRIDE=s3://my-bucket/models

NIM_DISABLE_MODEL_DOWNLOAD: boolean = None#

Skips model download during container startup. Useful in multi-node deployments where worker nodes use a pre-staged shared filesystem and only the leader node needs to download.

Default:

False

Example:

NIM_DISABLE_MODEL_DOWNLOAD=true

NIM_TRUST_CUSTOM_CODE: bool#

Allows dynamic module loading for custom model code. Required for models that ship custom tokenizer or modeling files.

Default:

False

Type:

boolean

Example:

NIM_TRUST_CUSTOM_CODE=true

Server#

The following variables control server and health-check ports:

NIM_SERVER_PORT: int | None#

Port for the external-facing HTTP API server.

Default:

None (uses container default)

Type:

integer

Example:

NIM_SERVER_PORT=9000

NIM_HEALTH_PORT: int | None#

Port for the proxy health endpoints (/v1/health/live and /v1/health/ready).

In any Dynamo mode this selects a different thing: the worker’s system-status server, which serves /v1/health/ready and the LoRA management routes (/v1/loras). There is no proxy in that mode, the default is 9090 rather than NIM_SERVER_PORT, and the entrypoint copies the resolved value into DYN_SYSTEM_PORT – so this is the only way to move that port.

Default:

None (defaults to NIM_SERVER_PORT; 9090 in Dynamo mode)

Type:

integer

Example:

NIM_HEALTH_PORT=8001

LoRA and PEFT#

The following variables control LoRA and PEFT adapter discovery and refresh behavior:

NIM_PEFT_SOURCE: str | None#

URI for the LoRA adapter source (local path or NGC URI).

Default:

None (LoRA disabled)

Type:

string

Example:

NIM_PEFT_SOURCE=/adapters

NIM_PEFT_REFRESH_INTERVAL: int | None#

Polling interval in seconds for the dynamic LoRA watcher. When set, NIM periodically checks the PEFT source for new or removed adapters.

Default:

None (dynamic reloading disabled)

Type:

positive integer

Example:

NIM_PEFT_REFRESH_INTERVAL=30

NIM_PEFT_API_TIMEOUT_SECS: float | None#

Timeout in seconds for dynamic LoRA adapter API calls.

Default:

30.0

Type:

positive float

Example:

NIM_PEFT_API_TIMEOUT_SECS=60

Model Cache#

The following variable controls the model cache location inside the container:

NIM_CACHE_PATH: str#

Directory path for the model and artifact cache inside the NIM container.

Default:

/opt/nim/.cache

Type:

string

Example:

NIM_CACHE_PATH=/mnt/models/.cache

This one controls the timeout in seconds for the model cache discovery:

NIM_CACHE_PROBE_TIMEOUT: integer = 60#

Deadline in seconds for the initial artifact-cache reachability probe at startup. If NIM_CACHE_PATH is on an unreachable NFS/CIFS/FUSE mount, the container exits within this deadline instead of hanging until the OS TCP timeout.

Default:

60

Example:

NIM_CACHE_PROBE_TIMEOUT=120

Writable Paths#

By default the container writes under /opt/nim. These variables relocate those writes, which is what allows /opt/nim to be mounted read-only (Kubernetes securityContext.readOnlyRootFilesystem: true, or an equivalent immutable-root policy).

They are resolved by the container entrypoint before Python starts, so they are set as container environment variables rather than Python configuration.

NIM_WRITABLE_ROOT: string = /opt/nim#

Umbrella root for runtime writes: nginx state, scratch space (TMPDIR), the $HOME-derived GPU/library caches, and the generated middleware config. Setting this one variable relocates all of them.

It does not relocate NIM_CACHE_PATH or NIM_MANIFEST_PATH. Running with a read-only /opt/nim requires NIM_WRITABLE_ROOT and a writable NIM_CACHE_PATH; Model-Free deployments additionally need NIM_MANIFEST_PATH. Leave NIM_MANIFEST_PATH unset on a model-specific NIM – it is the read location of the baked manifest, and setting it points the loader at a file that does not exist.

Default:

/opt/nim

Example:

NIM_WRITABLE_ROOT=/mnt/rw

NIM_NGINX_DIR: string = ${NIM_WRITABLE_ROOT}/nginx#

Directory holding all nginx runtime state: the generated nginx.conf, nginx.pid, the five *_temp directories, the generated snippet configs, and the access/error logs. Overrides NIM_WRITABLE_ROOT for nginx alone – useful when nginx state belongs on a tmpfs while the caches live on a persistent volume.

Default:

${NIM_WRITABLE_ROOT}/nginx

Example:

NIM_NGINX_DIR=/run/nginx

NIM_MIDDLEWARE_CONFIG_PATH: string = ${NIM_WRITABLE_ROOT}/generated_configs/middleware_config.json#

Path of the generated middleware configuration, written at startup and read back to serve /v1/metadata. Overrides NIM_WRITABLE_ROOT for this file alone.

Default:

${NIM_WRITABLE_ROOT}/generated_configs/middleware_config.json

Example:

NIM_MIDDLEWARE_CONFIG_PATH=/mnt/rw/middleware.json

The nginx access and error logs are written under NIM_NGINX_DIR and can be redirected individually with NIM_NGINX_ACCESS_LOG (default ${NIM_NGINX_DIR}/access.log) and NIM_NGINX_ERROR_LOG (default ${NIM_NGINX_DIR}/error.log).

Payload capture, when enabled, also follows NIM_WRITABLE_ROOT: the path key of NIM_CAPTURE_ARGS defaults to ${NIM_WRITABLE_ROOT}/captures/requests.jsonl. See Payload capture.

Warning

readOnlyRootFilesystem: true freezes /etc/passwd, so the entrypoint cannot map an SCC-assigned arbitrary UID into it. Some libraries on the serve path resolve a cache directory through getpwuid() and fail at import with KeyError: 'getpwuid(): uid not found: <uid>'. Relocating the writable root does not help – the lookup happens before any path is used.

If your platform assigns arbitrary UIDs (OpenShift SCC), either run with a UID that resolves (runAsUser matching the image’s nim user), or leave readOnlyRootFilesystem unset and mount just /opt/nim read-only – the posture these variables are designed for, and the one in which the entrypoint’s /etc/passwd fixup still runs.

Note that a Kubernetes emptyDir cannot be used to make /etc/passwd writable: volumes without a file-typed source mount as directories, and mounting a directory over an existing file is rejected by the runtime, so the container fails to start.

Likewise, update-ca-certificates needs a writable /etc/ssl/certs. On a fully read-only rootfs the entrypoint skips the refresh and logs a warning, so mounted custom CA roots are not picked up; pre-bake them into the image or mount /etc/ssl/certs writable.

Important

The volumes backing $HOME and $TMPDIR must be mounted exec, not merely writable. Triton and vLLM compile kernels at runtime and dlopen() the resulting shared objects from $HOME/.triton and $HOME/.cache/vllm; TorchInductor builds and loads its objects out of $TMPDIR. On a noexec mount the mmap(PROT_EXEC) fails and the model never loads.

Both derive from NIM_WRITABLE_ROOT unless you override them, so in the normal case this is a requirement on that one volume.

Bind mounts and Kubernetes emptyDir volumes allow exec by default. Docker’s --tmpfs does not – pass --tmpfs /mnt/rw:rw,exec. If your platform applies noexec to mounted volumes as a hardening control, it must be relaxed for this volume.

/tmp, /run and /var/tmp may stay noexec: TMPDIR is relocated under NIM_WRITABLE_ROOT, so nothing executes from /tmp.

The entrypoint verifies this at startup and exits with a diagnostic naming the mount (set NIM_SKIP_EXEC_CHECK=1 to bypass). Without it the failure surfaces minutes later inside a worker as OSError: ... failed to map segment from shared object. Note that this error’s own suggested fix – setting TORCHINDUCTOR_CACHE_DIR elsewhere – does not help: the load happens from $TMPDIR, so that is the mount to correct.

Authentication#

The following variables provide credentials for authenticated model downloads:

NGC_API_KEY: string = None#

API key for authenticated model downloads from NGC (NVIDIA GPU Cloud). Only required when downloading production branch (PB) models from ngc:// repositories.

Default:

None

Example:

NGC_API_KEY=nvapi-...

NGC_CLI_API_KEY: string = None#

Backward-compatible NGC credential source. When both NGC_CLI_API_KEY and NGC_API_KEY are set, NGC_CLI_API_KEY takes precedence.

Default:

None

Example:

NGC_CLI_API_KEY=nvapi-...

HF_TOKEN: string = None#

Authentication token for Hugging Face Hub. Required for downloading private or gated models from hf:// repositories.

Default:

None

Example:

HF_TOKEN=hf_...

MODELSCOPE_API_TOKEN: string = None#

Authentication token for ModelScope. Required for authenticated downloads from modelscope:// repositories and to avoid rate limiting.

Default:

None

Example:

MODELSCOPE_API_TOKEN=...

SSL and TLS#

NIM uses TLS in two distinct directions. Inbound TLS secures client connections to the NIM inference API (nginx layer). Outbound TLS secures connections the container makes from itself when downloading model artifacts from NGC, Hugging Face, or a corporate registry such as JFrog Artifactory.

Important

The NIM_SSL_* variables below configure inbound TLS only. They do not affect outbound model downloads. To trust a corporate Certificate Authority (CA) for outbound connections, see Outbound TLS (Model Downloads).

Inbound TLS (NIM API)#

The following variables control TLS termination at the nginx proxy layer:

NIM_SSL_MODE: string = None#

Controls TLS termination at the nginx proxy.

  • DISABLED – plain HTTP (default)

  • TLS – server-side TLS; requires NIM_SSL_KEY_PATH and NIM_SSL_CERTS_PATH

  • MTLS – mutual TLS; additionally requires NIM_SSL_CA_CERTS_PATH

Default:

DISABLED

Example:

NIM_SSL_MODE=TLS

NIM_SSL_KEY_PATH: string = None#

Path to the SSL private key file. Required when NIM_SSL_MODE is TLS or MTLS.

Default:

None

Example:

NIM_SSL_KEY_PATH=/etc/ssl/private/server.key

NIM_SSL_CERTS_PATH: string = None#

Path to the SSL certificate file. Required when NIM_SSL_MODE is TLS or MTLS.

Default:

None

Example:

NIM_SSL_CERTS_PATH=/etc/ssl/certs/server.crt

NIM_SSL_CA_CERTS_PATH: string = None#

Path to the CA certificate file for client verification. Required when NIM_SSL_MODE is MTLS.

Default:

None

Example:

NIM_SSL_CA_CERTS_PATH=/etc/ssl/certs/ca.crt

Outbound TLS (Model Downloads)#

When the NIM container downloads models from a registry that uses a certificate signed by a private or corporate CA, you must provide that CA certificate to the container. This applies to two common scenarios:

  • Corporate registry with private CA — for example, a JFrog Artifactory instance whose TLS certificate is signed by your organization’s internal CA (no proxy involved).

  • TLS-inspecting proxy — a corporate proxy that decrypts and re-encrypts HTTPS traffic using a corporate CA.

In both cases, set SSL_CERT_FILE to a CA bundle that includes the corporate CA so that outbound TLS verification succeeds. A proxy (HTTPS_PROXY) is not required for SSL_CERT_FILE to take effect.

REQUESTS_CA_BUNDLE: string = None#

Same purpose as SSL_CERT_FILE but specific to the Python requests library. Some internal components (such as proxy validation in nimlib) use requests; setting this variable ensures those paths also trust the corporate CA. When in doubt, set both SSL_CERT_FILE and REQUESTS_CA_BUNDLE to the same combined bundle.

Default:

None

Example:

REQUESTS_CA_BUNDLE=/etc/ssl/certs/custom-ca-bundle.pem

SSL_CERT_FILE: string = None#

Path to a PEM-format CA certificate or bundle file inside the container. OpenSSL and the model download pipeline (nim_sdk, reqwest, and native-tls) use this file to verify server certificates during outbound HTTPS connections. Can be used with or without HTTPS_PROXY.

Warning

Setting SSL_CERT_FILE replaces the container’s default trust store. If you point it at a file containing only your corporate CA, connections to public endpoints (such as api.ngc.nvidia.com) will fail because the public CAs are no longer trusted. If you also need to reach public endpoints, use a combined bundle that includes both the default CAs and your corporate CA.

Default:

None (uses /etc/ssl/certs/ca-certificates.crt)

Example:

SSL_CERT_FILE=/etc/ssl/certs/custom-ca-bundle.pem

Create a combined CA bundle (one-time, on the host):

To add your corporate CA without losing trust in public CAs, concatenate the container’s default bundle with your corporate CA certificate:

# Extract the default CA bundle from the container
docker run --rm --entrypoint bash \
  ${NIM_LLM_MODEL_SPECIFIC_IMAGE}:3.0.0 \
  -c 'cat /etc/ssl/certs/ca-certificates.crt' > combined-ca-bundle.pem

# Append your corporate CA
cat /path/to/corporate-ca.pem >> combined-ca-bundle.pem

See also: Air-Gap Deployment: CA Certificate Injection.

CORS#

These variables configure Cross-Origin Resource Sharing (CORS) policy at the nginx proxy layer.

NIM_CORS_ALLOW_ORIGINS: string = None#

Comma-separated list of allowed request origins, or * for any origin.

Default:

*

Example:

NIM_CORS_ALLOW_ORIGINS=https://example.com

NIM_CORS_ALLOW_METHODS: string = None#

Allowed HTTP methods for CORS requests.

Default:

GET, POST, PUT, DELETE, PATCH, OPTIONS

Example:

NIM_CORS_ALLOW_METHODS=GET, POST, OPTIONS

NIM_CORS_ALLOW_HEADERS: string = None#

Allowed request headers for CORS requests.

Default:

Content-Type, Authorization, X-Request-Id, X-Session-Id, X-Correlation-Id

Example:

NIM_CORS_ALLOW_HEADERS=Content-Type, Authorization

NIM_CORS_EXPOSE_HEADERS: string = None#

Response headers that are exposed to the browser in CORS responses.

Default:

X-Request-Id

Example:

NIM_CORS_EXPOSE_HEADERS=X-Request-Id, X-Correlation-Id

NIM_CORS_MAX_AGE: string = None#

Duration in seconds that browsers may cache CORS preflight responses.

Default:

3600

Example:

NIM_CORS_MAX_AGE=7200

AWS SageMaker#

The following variable controls SageMaker BYOC (Bring Your Own Container) compatibility mode. When active, NIM listens on port 8080 and implements the GET /ping health check and POST /invocations inference endpoints required by SageMaker real-time inference.

NIM_SAGEMAKER_MODE: string = None#

Controls AWS SageMaker real-time inference compatibility mode.

  • 1 — Force SageMaker mode on. NIM listens on port 8080 and exposes GET /ping (health) and POST /invocations (inference, proxied to /v1/chat/completions).

  • 0 — Suppress SageMaker mode even when SageMaker environment variables are present. Use this to run NIM on a SageMaker instance without activating the protocol adapter.

  • (unset) — Auto-detect: SageMaker mode is enabled automatically if any of SAGEMAKER_MULTI_MODEL, SAGEMAKER_REGION, or SAGEMAKER_BIND_TO_PORT is present in the environment. These variables are injected by the SageMaker host agent and are not present in other environments.

Default:

(unset) — auto-detect from SageMaker environment signals

Example:

NIM_SAGEMAKER_MODE=1

Advanced#

The following variables control advanced argument handling and runtime behavior:

NIM_PASSTHROUGH_ARGS: str | None#

Passes additional vLLM CLI arguments as a single string. Useful in environments where direct CLI arguments are not available (e.g., container orchestrators). The same name is also accepted as a reserved key inside runtime_config.json, where the string is parsed with the same rules but resolves at runtime-config priority; explicit flat keys in the same file take precedence over it.

Default:

None

Type:

string

Example:

NIM_PASSTHROUGH_ARGS="--enable-prefix-caching --max-num-seqs 128"

NIM_STRICT_ARG_PROCESSING: bool#

Enables strict configuration processing. When true, conflicting configuration overrides (e.g., CLI overwriting an environment variable) raise errors instead of warnings.

Default:

False

Type:

boolean

Example:

NIM_STRICT_ARG_PROCESSING=true

NIM_DISABLE_CUDA_GRAPH: bool#

Disables CUDA graph optimization. May reduce GPU memory usage at the cost of inference throughput.

Default:

False

Type:

boolean

Example:

NIM_DISABLE_CUDA_GRAPH=true

Speculative Decoding#

Speculative decoding (NGRAM / MTP / EAGLE3) is a runtime toggle, not a separate profile. The selected profile decides the default: a profile that ships a spec config in its runtime_config.json serves with speculative decoding, one that doesn’t serves without it. NIM_SPECDEC_ENABLE overrides that default globally (1 forces it on wherever a config exists – profiles without one warn and serve without it – and 0 forces it off everywhere; NIM_SPECDEC_ARGS can override the config). Either way the same profile serves both modes. Refer to the Speculative Decoding guide for recipes, bring-your-own-draft instructions, and benchmarking.

Precedence (highest wins): explicit CLI --speculative-* and NIM_PASSTHROUGH_ARGS override NIM_SPECDEC_ARGS, which overrides the profile’s spec block. The EAGLE3 draft model downloads through the same path as the checkpoint (NGC / HF / S3 / GCS, air-gap and cache aware) from the draft_uri carried in the spec config, or from NIM_DRAFT_MODEL_PATH for the model-free path (NIM_DRAFT_MODEL_PATH takes precedence when both are set).

NIM_SPECDEC_ENABLE: bool | None#

Speculative-decoding override. SpecDec is a runtime toggle, not a profile dimension: the spec config lives in runtime_config.json (the single source of truth) and is gated at launch, so one profile serves both spec-on and spec-off without a dedicated profile. Tri-state:

  • unset (default): the selected profile decides. A profile whose runtime_config.json carries a spec config (a nim_specdec envelope or flat speculative_* keys) serves WITH speculative decoding; a profile without one serves without it.

  • 1: force spec on for profiles that carry (or receive, via NIM_SPECDEC_ARGS) a spec config. A profile with no spec config logs a warning and serves without spec – it never crashes.

  • 0: force spec off for every profile; any spec config is stripped from runtime_config.json before launch.

Default:

unset (the profile decides)

Type:

boolean (tri-state: unset / 1 / 0)

Example:

NIM_SPECDEC_ENABLE=0

NIM_SPECDEC_ARGS: str | None#

Model-level speculative-decoding config, a JSON object of runtime_config.json keys for the active backend. Consumed only when speculative decoding resolves ON (the profile’s default, or forced by NIM_SPECDEC_ENABLE=1); its keys are merged into the workspace runtime_config.json (overriding any profile-shipped spec block). On its own it does not activate spec for a profile that ships no spec config – pair it with NIM_SPECDEC_ENABLE=1 for the model-free path. vLLM uses speculative_config (a JSON string); SGLang uses speculative_algorithm plus the speculative_* flags. An EAGLE3 entry may carry a NIM-internal draft_uri (any NIM_MODEL_PATH scheme) that is materialized through the unified download path and never passed to the backend.

Default:

None

Type:

string (JSON object)

Example (vLLM):

NIM_SPECDEC_ARGS='{"speculative_config": "{...}", "draft_uri": "ngc://org/d:1"}'

Example (SGLang):

NIM_SPECDEC_ARGS='{"speculative_algorithm": "NGRAM"}'

NIM_DRAFT_MODEL_PATH: str | None#

Source URI or local path for a speculative draft model (e.g. an EAGLE3 head). Accepts the same schemes as NIM_MODEL_PATH (hf://, ngc://, s3://, gs://, modelscope://) or an absolute local directory. When set and the assembled speculative config references a draft by a bare name (eagle/eagle3/ draft_model/medusa/standalone), NIM downloads/materializes it into that subdir of the served workspace and rewrites the draft reference to the absolute path. The speculative config itself (method, num tokens, algorithm, …) is supplied separately via CLI args or NIM_PASSTHROUGH_ARGS; reference the draft by a bare subdir name (e.g. draft) there and NIM materializes it at <workspace>/<name>.

Default:

None (no draft download)

Type:

string

Example:

NIM_DRAFT_MODEL_PATH=hf://lmsys/SGLang-EAGLE3-Llama-3.1-8B-Instruct-SpecForge

Payload Capture#

Opt-in capture of inference request payloads as AIPerf-replayable JSONL. Disabled by default; captures exact prompts (PII risk). Refer to the Payload Capture guide for formats, examples, and replay commands.

NIM_CAPTURE_ENABLE: bool#

Opt-in payload capture. When enabled, a NIM-owned ASGI middleware writes each inference request payload as AIPerf-replayable JSONL (see nim_llm/features/payload_capture_middleware.py). DISABLED BY DEFAULT because it captures exact prompts, which may include PII/secrets/customer data – unlike NIM_SPECDEC_ENABLE, no profile ever turns it on. This is the toggle, NIM_CAPTURE_ARGS carries the (optional) tuning.

Default:

False

Type:

boolean

Example:

NIM_CAPTURE_ENABLE=1

NIM_CAPTURE_ARGS: str | None#

Capture configuration as a JSON object, consumed only when NIM_CAPTURE_ENABLE=1. All keys are optional; the middleware validates them and falls back to the default on a bad value (and disables capture if the JSON itself is invalid). Keys:

  • path (str, default $NIM_WRITABLE_ROOT/captures/requests.jsonl, i.e. /opt/nim/captures/requests.jsonl unless the writable root is relocated): destination JSONL, must be writable by the container user (mount a writable dir).

  • format (mooncake_payload default | raw_payload): both record the ACTUAL request payload (never hashed/synthetic prompts). mooncake_payload writes {"timestamp": <ms>, "payload": <request>}; raw_payload writes the request JSON object. Replay with --custom-dataset-type mooncake_trace / raw_payload.

  • max_request_bytes (int, default 1048576): larger requests are forwarded unchanged but not recorded (warned), since a truncated payload is not replayable.

  • sample_rate (float in [0,1], default 1.0): probability a matching request is written; 1.0 = all, 0.1 = ~10%, 0.0 = none. Trims I/O under high QPS.

  • endpoint_pattern (regex, default ^/v1/(chat/completions|completions)$): only matching request paths are captured.

  • max_queue_bytes (int, default 1073741824 = 1 GiB): in-memory ceiling for the off-loop writer queue (bytes of buffered request body). Captures are serialized and written on a background thread; on overload the newest record is dropped with a WARNING (the request path is never blocked). Set >= max_request_bytes.

Default:

None (built-in defaults apply)

Type:

string (JSON object)

Example:

NIM_CAPTURE_ARGS='{"format": "raw_payload", "path": "/captures/t.jsonl"}'

Dynamo#

Dynamo-enabled NIM images can host the vLLM worker under the NVIDIA Dynamo runtime, which adds an HTTP frontend with a KV-aware router in front of one or more workers. Dynamo is opt-in: it is available only in images built with Dynamo support, and within such an image it starts only when one of the mode variables below is set. With none set, the container serves vLLM directly and none of these variables apply.

Warning

Setting a mode variable on an image that does not include the Dynamo runtime is not rejected. Two of the three modes fail silently, so confirm the image is Dynamo-enabled before you set these.

  • NIM_DYNAMO_SINGLE still starts a frontend first, and that frontend exits immediately with ModuleNotFoundError: dynamo — but its output goes to a log file inside the container, not to the container log, so the failure is invisible. The worker then falls back to serving vLLM directly on the internal backend port. Because the entrypoint never reaches the reverse-proxy setup, nothing listens on the published server port at all.

  • NIM_DYNAMO_WORKER launches no frontend. The worker falls back to plain vLLM instead of registering with Dynamo, so it never joins the deployment.

  • NIM_DYNAMO_FRONTEND exits with ModuleNotFoundError: dynamo. It is the only mode that fails loudly.

In every fallback case the reverse proxy and request middleware are skipped, so the management routes are missing too, and the fallback itself is logged only at debug level.

Dynamo mode applies to the vLLM backend only. The entrypoint does not check the backend, so setting a mode variable on an SGLang image still bypasses the reverse proxy while the worker serves SGLang directly.

Selecting a Mode#

Based on your deployment topology, set exactly one of the following Dynamo mode variables:

  • Single-Container Mode (NIM_DYNAMO_SINGLE=1): Runs the frontend and worker in one container using file-based discovery and the ZMQ event plane. This mode requires neither etcd nor NATS and supports single-node Docker deployments.

  • Worker Mode (NIM_DYNAMO_WORKER=1): Runs only the worker in the container while the frontend runs elsewhere. This mode supports multi-pod Kubernetes deployments. Unlike single-container mode it sets no discovery or event-plane defaults, so the deployment supplies them. Under the NVIDIA Dynamo Operator this is handled for you: the operator injects DYN_DISCOVERY_BACKEND=kubernetes, which discovers peers through Kubernetes resources and needs no etcd. Without the operator, Dynamo’s own default is etcd and you must provide the endpoint yourself — NIM exposes no variable for it.

  • Frontend Mode (NIM_DYNAMO_FRONTEND=1): Runs only the frontend and router. It creates a metadata-only workspace without weight shards, so the pod does not require a GPU. That workspace is built with NIM_PASSTHROUGH_ARGS cleared, so express model and profile selection for this container through environment variables — NIM_MODEL_PATH and NIM_MODEL_PROFILE — and keep them identical to the workers’. Otherwise the frontend can resolve different metadata than the model it fronts.

Set the chosen variable to 1 or true, and leave the other two unset. Other values are unsafe: the entrypoint treats anything that is not empty, 0, or false as enabled, but the Python runtime accepts only true, 1, or yes. A value such as on therefore passes the first check and fails the second, leaving the container serving plain vLLM instead of Dynamo, with no error.

If more than one is set, precedence is single-container, then worker, then frontend; the others are ignored without warning.

NIM_DYNAMO_SINGLE: bool#

Runs the Dynamo frontend and the vLLM worker together in a single container. Service discovery is file-based and KV-cache events use the ZMQ peer-to-peer event plane, so no etcd or NATS is required. Intended for single-node Docker deployments without Kubernetes.

Only one of NIM_DYNAMO_SINGLE, NIM_DYNAMO_WORKER, and NIM_DYNAMO_FRONTEND should be set. If more than one is set, the first of that order wins and the others are ignored without warning. With none set, the container serves vLLM directly and Dynamo is not started.

Default:

False

Type:

boolean

Example:

NIM_DYNAMO_SINGLE=1

NIM_DYNAMO_WORKER: bool#

Runs the container as a Dynamo worker only. The frontend runs in a separate container or pod, and the worker registers with it through the discovery backend configured by the deployment.

Default:

False

Type:

boolean

Example:

NIM_DYNAMO_WORKER=1

NIM_DYNAMO_FRONTEND: bool#

Runs the container as a Dynamo frontend only. On startup it materializes a metadata-only workspace (config, tokenizer, chat template, generation config – no weight shards) and then serves dynamo.frontend against it. Because no weights are loaded, the frontend needs no GPU.

Default:

False

Type:

boolean

Example:

NIM_DYNAMO_FRONTEND=1

Routing and Ports#

The following variables configure Dynamo routing and the frontend port. They are read only by the Dynamo frontend, so they take effect in single-container and frontend modes; in worker mode they are computed and discarded — set them on the frontend. DYN_SYSTEM_PORT, listed last, is derived by the entrypoint and shown for reference — the knob is NIM_HEALTH_PORT, documented under Server, which in Dynamo mode selects the worker system-status port and defaults to 9090 rather than to NIM_SERVER_PORT.

DYN_ROUTER_MODE: string#

Routing strategy used by the Dynamo frontend to pick a worker.

kv evaluates KV-cache overlap and decode load per worker and picks the lowest cost, which raises cache hit rates for workloads with shared prefixes. round-robin cycles through the available workers, random picks one per request, and direct takes the target worker from the request’s routing hints instead of selecting one.

KV-aware routing selects among interchangeable worker replicas. It is not a prefill/decode split – every worker runs the full forward pass.

The Dynamo runtime accepts additional routing strategies that are not validated with NIM LLM. Use the values below.

Valid values:

kv, round-robin, random, direct

Default:

kv – set by the container. Note this differs from the Dynamo runtime’s own default of round-robin

Example:

DYN_ROUTER_MODE=round-robin

DYN_HTTP_PORT: integer#

Port the Dynamo frontend listens on for the OpenAI-compatible API. This is the port clients connect to, so it tracks the same port the non-Dynamo path serves on.

Default:

the value of NIM_SERVER_PORT (8000 unless you override it)

Example:

DYN_HTTP_PORT=9000

DYN_FRONTEND_EXTRA_ARGS: string#

Extra command-line arguments appended to the Dynamo frontend invocation. Use this to reach frontend options that NIM does not expose directly, such as the KV-router tuning flags --router-kv-overlap-score-weight, --router-queue-threshold, and --router-queue-policy (these apply only when DYN_ROUTER_MODE=kv).

The value is word-split on whitespace. Quotes are not honoured and survive as literal characters, so an argument that itself contains whitespace cannot be expressed through this variable.

Default:

(unset)

Example:

DYN_FRONTEND_EXTRA_ARGS="--router-queue-threshold 4"

DYN_SYSTEM_PORT: int | None#

Port of the Dynamo worker’s system-status server, which serves /v1/health/ready and the LoRA management routes (/v1/loras). This port must differ from the frontend’s HTTP port.

Derived, not configurable. In every Dynamo mode the entrypoint overwrites this variable unconditionally: an explicit NIM_HEALTH_PORT is used verbatim, otherwise the port is 9090. Setting DYN_SYSTEM_PORT directly has no effect – use NIM_HEALTH_PORT to choose the port.

The 9090 default matches the port the Dynamo Operator hardwires and probes, so operator-managed deployments need no override.

Default:

derived from NIM_HEALTH_PORT, else 9090

Type:

integer

Set by:

the container entrypoint

Tool-Call and Reasoning Parsers#

In Dynamo mode the tool-call and reasoning parsers are selected with environment variables rather than CLI flags.

Important

Remove the API-server parser and chat-template flags from NIM_PASSTHROUGH_ARGS before enabling Dynamo mode. --tool-call-parser, --enable-auto-tool-choice, --tool-parser-plugin and --chat-template are accepted by the container but are not understood by the Dynamo worker, which exits at startup with unrecognized arguments. Set the variables below instead.

--reasoning-parser and --reasoning-parser-plugin are the exception: they are vLLM engine arguments, so the Dynamo worker accepts both. DYN_REASONING_PARSER remains the recommended way to select a reasoning parser, because it applies consistently across every worker in a multi-pod deployment.

A custom tool-parser plugin has no equivalent in Dynamo mode: the available tool parsers are those built into the Dynamo runtime, so a --tool-parser-plugin parser cannot be carried over.

Set Dynamo options with the DYN_* environment variables rather than through NIM_PASSTHROUGH_ARGS. Passthrough arguments are forwarded to the worker verbatim — NIM does not validate them — so a mistyped or misplaced flag surfaces as a worker startup failure rather than a clear error from the container.

DYN_TOOL_CALL_PARSER: string#

Name of the tool-call parser the Dynamo worker uses to extract tool calls from model output. Set it to the parser that matches your model; consult the model’s documentation for the correct name.

Default:

(unset) – no tool-call parsing

Example:

DYN_TOOL_CALL_PARSER=nemotron_nano

DYN_REASONING_PARSER: string#

Name of the reasoning parser the Dynamo worker uses to separate reasoning content from the final answer. Set it to the parser that matches your model.

Default:

(unset) – no reasoning parsing

Example:

DYN_REASONING_PARSER=nemotron_nano

LoRA in Dynamo Mode#

Adapters are managed through Dynamo’s /v1/loras API on the worker system port (NIM_HEALTH_PORT), not on the frontend. The container enables Dynamo-managed LoRA automatically when NIM_PEFT_SOURCE is set or --enable-lora is passed. If LoRA is enabled only by the model profile, neither trigger fires — set NIM_PEFT_SOURCE, or set DYN_LORA_ENABLED=true spelled exactly that way, since 1 and yes are not accepted by the /v1/loras route gate.

Variables That Do Not Apply in Dynamo Mode#

In every Dynamo mode the entrypoint hands off before the reverse proxy is configured, and --middleware is stripped from the worker’s arguments, so neither the proxy nor NIM’s request middleware is started. (In worker mode there is no Dynamo frontend in the container either — it runs elsewhere.) Every variable below is ignored for its documented effect when any Dynamo mode is set, and the container does not warn.

Variables

Why they stop applying

What to do instead

SSL and TLS: NIM_SSL_MODE, NIM_SSL_KEY_PATH, NIM_SSL_CERTS_PATH, NIM_SSL_CA_CERTS_PATH

TLS terminates at the reverse proxy, which is not started

Terminate TLS at an ingress or gateway in front of the frontend

CORS: all NIM_CORS_*

CORS policy is applied by the reverse proxy

Apply CORS policy at the ingress or gateway

Request shaping: all NIM_PROXY_* (body size, buffering, timeouts, keepalive, worker counts)

Same

Use the equivalent ingress settings, or the Dynamo frontend’s own limits

nginx paths and logs: NIM_NGINX_DIR, NIM_NGINX_ACCESS_LOG, NIM_NGINX_ERROR_LOG

No nginx process exists to configure

See the note on frontend logging below

Payload Capture: NIM_CAPTURE_ENABLE, NIM_CAPTURE_ARGS

Capture is a request middleware, and middleware is not installed on the Dynamo worker

Capture at the client or at an ingress

AWS SageMaker: NIM_SAGEMAKER_MODE

Only partly. The port side-effect applies – the serving port moves to 8080 – but GET /ping and POST /invocations are not served, because they come from the nginx front door and the request middleware and neither runs in Dynamo mode. Only the single-container shape is validated on SageMaker; multi-pod DGD is not supported there

Set NIM_SAGEMAKER_MODE=1 explicitly rather than relying on auto-detection. If a SAGEMAKER_* variable is present outside SageMaker, set DYN_HTTP_PORT explicitly

Logging behaves differently rather than stopping: NIM_LOG_LEVEL and NIM_JSONL_LOGGING govern NIM’s own Python processes — the worker in single-container and worker modes, the metadata download step in frontend mode — plus the entrypoint’s startup messages. They do not govern the Dynamo frontend, which has its own logging configuration. A deployment that standardizes on structured logs sees structured worker output alongside the frontend’s own format.

Metrics move as well: the frontend serves Prometheus at /v1/metrics, publishing Dynamo frontend metrics, while vLLM’s engine metrics move to the worker system port (NIM_HEALTH_PORT, default 9090) at /metrics. Refer to Logging and Observability for the two-job scrape setup.

Where the frontend’s own output goes depends on the mode. In frontend mode it is the container’s main process, so its output reaches the container log. In single-container mode the frontend runs in the background with its output redirected to ${TMPDIR}/frontend.log — normally /tmp/frontend.log, see the temporary-directory note below — so it does not appear in the container log. Read it with docker exec <container> cat /tmp/frontend.log when diagnosing frontend startup.

Because the same middleware is absent, two response behaviours also change: X-Request-Id and traceparent are handled by the Dynamo frontend rather than echoed by NIM, and the merged /openapi.json is not served, so the interactive API docs are incomplete.

Finally, the container’s temporary directory moves when a Dynamo mode variable is set: it is placed on /tmp when that path is writable, instead of under NIM_WRITABLE_ROOT. An explicitly set TMPDIR always wins and is never overridden.

Any Dynamo container that gets a noexec /tmp therefore fails startup, whether or not the root filesystem is read-only — --tmpfs /tmp alone is enough, because noexec is Docker’s --tmpfs default. Mount it with exec (--tmpfs /tmp:rw,exec), or set TMPDIR to an exec-capable path. The startup error advises that /tmp may stay noexec because scratch lives under NIM_WRITABLE_ROOT; that guidance does not apply in Dynamo mode.