Configuration

This page lists launch-time environment variables. Start with the model-selection variables and keep automatic defaults unless the workload requires an override. Refer to Deployment for complete Docker commands.

Essential Configuration

Authentication and Cache

Name	Default	Use
NGC_API_KEY	Empty	Download required model and runtime artifacts when they are not already cached
NIM_CACHE_PATH	/opt/nim/.cache	Set the writable in-container artifact cache

Choose the Runtime and Model

Name	Default	Use
NIM_MODEL_TYPE	generator	Select generator or reasoner
NIM_MODEL_VARIANT	nano preference	Select nano or super for either runtime; Reasoner also accepts edge, and Generator also accepts edge-droid, nano-droid, super-t2i, super-t2i-4step, super-i2v, and super-i2v-4step
NIM_PRECISION	Generator: FP8 preference; Reasoner: GPU-derived preference	Optionally pin a precision available in the image; refer to Precision Selection
NIM_PERF_PROFILE	latency	Generator only: choose latency or throughput

Reasoner rejects NIM_PERF_PROFILE, including profile= in NIM_TAGS_SELECTOR, with a startup error. Edge Reasoner has BF16 and FP8 profiles. Edge-DROID and Nano-DROID have BF16 profiles only. There is no general-purpose Edge Generator. The NIM chooses the best compatible profile for these settings and the visible GPUs. A normal deployment does not need a profile ID.

GPU Selection

The GPUs exposed by Docker --gpus are the visible GPUs that automatic selection considers. List host GPUs with their indexes and UUIDs:

nvidia-smi -L

Expose specific GPUs instead of --gpus all, for example --gpus '"device=0,1"' or --gpus '"device=GPU-<uuid>"'. Expose a GPU count listed in the Generator or Reasoner profiles. Extra GPUs sit idle. Expose GPUs of a single model, because mixed GPUs are not supported.

Do not pass -e CUDA_VISIBLE_DEVICES: the NIM sets it for its worker processes, so a passed value can place work outside the intended GPUs.

Bring Your Own Checkpoint

Name	Default	Use
NIM_MODEL_PATH	Empty	Absolute local directory or hf://owner/repository[:revision] for either runtime
NIM_DISABLE_MODEL_DOWNLOAD	false	Disable profile download; set automatically for a Reasoner checkpoint override, incompatible with Reasoner hf://, and rejected for Generator
HF_TOKEN	Empty	Authenticate to a private Hugging Face repository named by NIM_MODEL_PATH

To override the DFlash draft, refer to NIM_DFLASH_MODEL_PATH under Speculative Decoding. Refer to Bring Your Own Checkpoint for layouts, mounts, and compatibility checks.

Server and Logging

Name	Default	Use
NIM_HTTP_API_PORT	8000	Set the container HTTP port
NIM_LOG_LEVEL	INFO	Set the service logging threshold
NIM_LOGGING_JSONL	false	Emit JSON-line logs

Advanced Profile Controls

Use these only when automatic model selection is not sufficient:

Name	Default	Use
NIM_TAGS_SELECTOR	Empty	Filter by comma-separated exact manifest tags as key=value pairs; refer to the valid keys below

NIM_TAGS_SELECTOR lowercases its values and accepts only the following keys:

Key	Meaning	Runtime
model_type, model_variant, precision	Runtime, model, and precision	Both
profile	latency or throughput	Generator
offload, text_guard_offload, video_guard_offload	Offload modes	Generator
n_gpus	GPUs used by one NIM; the product of nim_dp, nim_gp, nim_up, and nim_tp	Both
nim_dp	Data-parallel model replicas; throughput profiles use n_gpus	Generator
nim_gp	CFG (guidance) parallel factor; latency profiles on 2 or more GPUs use 2	Generator
nim_up	Ulysses sequence-parallel factor; latency profiles use n_gpus / 2	Generator
nim_tp	Tensor-parallel factor; Super BF16 also has a 2-GPU nim_tp=2 layout	Reasoner

Selectors filter the image’s existing profiles and cannot create a layout the manifest lacks. A selection that matches no profile fails at startup. Without them, the Generator uses as many visible GPUs as a compatible profile supports (1, 2, 4, or 8). There is no one-GPU throughput profile, and model or layer offload is available only on one GPU. To keep the Generator on fewer GPUs, expose fewer GPUs with --gpus.

On an integrated GPU with unified host/device memory (NVIDIA DGX Spark/GB10, Jetson AGX Thor), the NIM treats 70% of the shared pool as GPU memory for profile floors and as the default Reasoner NIM_GPU_MEMORY_UTILIZATION.

A selector variable, such as NIM_MODEL_VARIANT or NIM_PRECISION, that conflicts with NIM_TAGS_SELECTOR is a startup error. Use the model_variant tag to filter by model. Exact tags and profile IDs are tied to an image release. For the offload, memory-headroom, and profile-pin variables, refer to Advanced Settings.

Generator Configuration

Input and Output

Name	Default	Use
NIM_ALLOW_URL_INPUT	true	Allow HTTP(S) image, video, and Transfer inputs; set false to require encoded input
NIM_TRITON_REQUEST_TIMEOUT	30 minutes (1800000000 microseconds)	Set the queue-plus-execution timeout in microseconds

Client HTTP timeouts do not change this backend ceiling. Refer to Long-Running Requests.

Startup and Execution

Name	Default	Use
NIM_ENABLE_WARMUP	false	Run synthetic inference before readiness at the 256_1_1, 480_16_9, and 720_16_9 resolutions; Edge-DROID and Nano-DROID use only 480_16_9, and 4-step variants use their fixed step count

Guardrails

Name	Default	Use
NIM_ENABLE_TEXT_GUARDRAILS	true	Enable input-prompt policy checks
NIM_ENABLE_VIDEO_GUARDRAILS	true	Enable output image/video face and visual guardrails
NIM_ENABLE_SIGLIP_GUARDRAILS	true	Enable the output-frame safety classifier
NIM_QWEN3GUARD_MAX_INPUT_TOKENS	8192	Set the text guardrail’s maximum input tokens; minimum 512

The selected profile owns guardrail residency. Choose it with the text_guard_offload and video_guard_offload selector keys under Advanced Profile Controls. Refer to Guardrails for what each layer does.

Diffusion Cache, Attention, and VAE Decode

These performance defaults are validated startup settings, not request fields. If you notice a quality issue in a specific workload, try NIM_CACHE_BACKEND=none and NIM_VAE_FAST_PATH=lossless.

The defaults work as follows:

• SeaCache reuses transformer results across denoising steps. It is on for every Generator variant except edge-droid. Set NIM_CACHE_BACKEND to sea_cache or none to change that. NIM_CACHE_CONFIG overrides individual fields as JSON, for example -e NIM_CACHE_CONFIG='{"sea_threshold": 0.20, "sea_max_consecutive_cached": 1}'. The defaults are sea_threshold=0.25, sea_residual_order=1, sea_max_consecutive_cached=2, and sea_power_exp=3.0. Setting sea_max_consecutive_cached=0 means no limit. Startup logs the effective settings.

• SageAttention 2.2 is the default on compute capability 8.0 through 9.x unless the profile uses sequence parallelism. NVIDIA Blackwell GPUs and sequence-parallel profiles use automatic selection. DIFFUSION_ATTENTION_BACKEND overrides the default.

• Variational autoencoder (VAE) decode defaults to NIM_VAE_FAST_PATH=channels_last, the fastest path. The lossless value keeps the reference decoder output, and off uses the reference decoder. On a replica spanning several GPUs, decode is sharded (NIM_ENABLE_VAE_PARALLELISM). Disabling it can raise per-GPU memory. A larger NIM_VAE_SPATIAL_TEMPORAL_CHUNK_SIZE raises peak memory and can change rounding.

Prompt Upsampling

Prompt upsampling is optional and applies only to Generator text-to-image (T2I), text-to-video (T2V), and image-to-video (I2V) requests. It sends the request to an operator-provided OpenAI-compatible endpoint.

Name	Default	Use
NIM_ENABLE_PROMPT_UPSAMPLING	false	Enable prompt upsampling
NIM_PROMPT_UPSAMPLING_ENDPOINT_URL	Empty	Set the OpenAI-compatible endpoint base or Chat Completions route
NIM_PROMPT_UPSAMPLING_MODEL	Empty	Set the external model name
NIM_PROMPT_UPSAMPLING_API_KEY	Empty	Set the external Bearer credential
NIM_PROMPT_UPSAMPLING_TIMEOUT_S	120	Set the external call timeout in seconds
NIM_PROMPT_UPSAMPLING_TEMPLATE_STYLE, NIM_PROMPT_UPSAMPLING_MAX_TOKENS, NIM_PROMPT_UPSAMPLING_TEMPERATURE, NIM_PROMPT_UPSAMPLING_TOP_P, NIM_PROMPT_UPSAMPLING_TOP_K, NIM_PROMPT_UPSAMPLING_EXTRA_BODY	external_api, 8192, omitted, omitted, omitted, empty object	Select external_api or reasoner templates; set the output-token limit, sampling temperature, nucleus sampling, and top-k (when the provider accepts it); merge additional JSON into the external request

Endpoint, model, and key are required when the feature is enabled. A request-time external failure logs a warning and continues with the original prompt.

Reasoner Configuration

Change controls one at a time and measure the effect.

Optimization Presets

Name	Default	Use
NIM_OPTIMIZATION_PRESET	Unset	Reasoner only: apply the experimental TTFT or TPOT starting configuration (case-insensitive); Generator rejects it

The presets configure these variables:

Preset	NIM_USE_DFLASH	NIM_MAX_NUM_BATCHED_TOKENS	NIM_DISABLE_CHUNKED_MM_INPUT
TTFT	0	32768 with at least 70 GiB of GPU memory; otherwise 16384	Unchanged (true)
TPOT	1	4096	0

On profiles with tensor parallelism above 1, both presets also set NIM_MM_ENCODER_TP_MODE=data, which replicates the vision encoder and splits media across TP ranks. An explicitly set variable wins over the preset. On Edge, set NIM_USE_DFLASH=0 with TPOT.

Speculative Decoding

Name	Default	Use
NIM_USE_DFLASH	true for Nano/Super Reasoner; false for Edge	Use the bundled Nano or Super DFlash draft; set false to run the target model without speculative decoding
NIM_DFLASH_MODEL_PATH	Empty	Nano or Super Reasoner: use an independent absolute local DFlash directory containing config.json and model.safetensors; refer to Local Directory for the mount

DFlash is on by default for Nano and Super Reasoner and does not change the request API. Set NIM_USE_DFLASH=0 to disable it (Generator rejects an explicit enable). Edge Reasoner does not support DFlash in this version. It defaults to NIM_USE_DFLASH=0 and rejects 1. For Edge, set NIM_STREAMING_SPEC_METHOD=none when streaming is enabled. NIM_DFLASH_MODEL_PATH accepts only an absolute local path, not hf://, and NIM_DFLASH_CONFIG requires DFlash enabled and cannot set the method or model keys.

Context and Scheduling

Name	Default	Use
NIM_MAX_MODEL_LEN	-1 (auto)	Let the runtime choose a context length, bounded by the checkpoint limit of 262,144 tokens
NIM_GPU_MEMORY_UTILIZATION	Discrete GPU: 0.93, capped to leave NIM_GPU_MEMORY_HEADROOM_GIB free; unified memory: 0.70	Set the Reasoner GPU-memory target in (0,1]; startup warns when it exceeds current free memory but does not reduce it. On DGX Spark/GB10 and Jetson AGX Thor, lower it only from host measurements; refer to Set the Reasoner Memory Share

Caching and Multimodal Processing

Name	Default	Use
NIM_MAX_IMAGES_PER_PROMPT	Unset	Optionally set a nonnegative image limit; when unset, do not override the runtime limit
NIM_MAX_VIDEOS_PER_PROMPT	Unset	Optionally set a nonnegative video limit; when unset, do not override the runtime limit

Prefer request-level media_io_kwargs over the operator-wide NIM_MEDIA_IO_KWARGS. Unchunked multimodal input, the default, can raise peak GPU memory. Set NIM_DISABLE_CHUNKED_MM_INPUT=0 only after measuring memory, latency, and throughput.

Streaming Video Sessions

Set NIM_MODEL_TYPE=reasoner and NIM_ENABLE_STREAMING=true to enable the WebSocket and REST session APIs in standalone NIM. In Dynamo mode, only REST sessions are supported. Changing these settings requires restarting the service with the new environment. Ordinary Chat Completions token streaming uses the request field stream=true and does not require this enable flag.

Name	Default	Use
NIM_ENABLE_STREAMING	false	Enable stateful video-frame sessions and apply the required engine settings
NIM_STREAMING_MAX_SESSIONS	2	Maximum concurrent sessions shared by WebSocket and REST; integer, minimum 1
NIM_STREAMING_MAX_VIDEO_SEGMENTS	8	Default retained frame window per session; integer, minimum 1
NIM_STREAMING_IDLE_TIMEOUT_S	900	Idle timeout for WebSocket sessions, in seconds; must be greater than 0
NIM_STREAMING_REST_IDLE_TIMEOUT_S	120	Idle timeout for REST sessions, in seconds; must be greater than 0

Streaming forces the multimodal processor cache off and disables asynchronous scheduling. It raises the engine’s image budget to at least 2 × NIM_STREAMING_MAX_SESSIONS × NIM_STREAMING_MAX_VIDEO_SEGMENTS, or 32 image slots with the defaults. This floor can override a lower NIM_MAX_IMAGES_PER_PROMPT. Increasing either streaming limit increases encoder-cache memory requirements. Reduce the limits if startup lacks memory, and validate representative requests before increasing concurrency. These engine changes also affect ordinary requests on the same container.

Accepted frames refresh session activity. A frame in flight is not reaped. WebSocket disconnects release the session, while REST clients should explicitly delete it rather than rely on the shorter idle timeout. Increase the relevant timeout for feeds with long pauses. Per-session retention and sampling belong in the creation request. Retention overrides remain subject to shared capacity and model-context admission checks.

Advanced Streaming Controls

Leave these at their defaults unless diagnosing or tuning a measured workload:

Name	Default	Use
NIM_STREAMING_SPEC_METHOD	dflash	Select dflash, GPU n-gram speculation (ngram), or none; while streaming is enabled, it replaces NIM_USE_DFLASH
NIM_STREAMING_SPEC_NUM_TOKENS	3	Number of proposed n-gram tokens; an integer of at least 1, and invalid values fail startup when streaming is enabled
NIM_STREAMING_FORCE_V1_RUNNER	Unset	Set 1 to force the vLLM V1 model runner; otherwise leave runner selection to vLLM

The speculation controls apply only when streaming is enabled. For Edge, which has no DFlash, set NIM_STREAMING_SPEC_METHOD=none explicitly. While streaming is enabled, NIM_USE_DFLASH is ignored, so set NIM_STREAMING_SPEC_METHOD=none to turn speculation off. Both V1 and V2 model runners support the streaming path, so pinning V1 is not required for normal operation.

Inspect GET /v1/streaming/config for the served model and default retention and sampling policy. Standalone NIM and the Dynamo streaming adapter expose this route. Refer to Streaming for session requests and error handling, and Dynamo REST Streaming for the client’s shared-frontend checks, which require no worker URLs or metadata.

API Behavior

Name	Default	Use
NIM_DISABLE_LOG_REQUESTS	true	Keep Reasoner request bodies out of logs
NIM_DISABLE_RESPONSES_ROUTE	false	Remove Responses routes when set to true

Dynamo Configuration

Run NIM as a Dynamo Worker

Use the Cosmos3 NIM image (nvcr.io/nim/nvidia/cosmos3:2.1.0) and start /opt/nim/start_server.sh with NIM_MODEL_TYPE=reasoner and NIM_DYNAMO_WORKER=true. Generator profiles do not support Dynamo worker mode. The NIM selects and loads the model, launches a Dynamo vLLM worker and a private frontend, and exposes the NIM HTTP API. Keep the usual model-selection, cache, and authentication settings from Essential Configuration. The shared frontend is a separate service. It does not need worker enablement or a GPU.

Name	Default	Use
NIM_DYNAMO_WORKER	false	Launch the Reasoner as a Dynamo worker instead of the standalone Reasoner runtime
NIM_DISCOVERY_BACKEND	file	Set worker and private-frontend discovery; use kubernetes for the supplied Kubernetes deployments. Also accepts the legacy JSON argument object shown below
DYN_NAMESPACE_PREFIX	From the nvidia.com/dynamo-namespace pod label in the supplied YAML	Let the private frontend discover operator-suffixed worker namespaces; takes precedence over the exact DYN_NAMESPACE filter
NIM_DYNAMO_ASYNC_SCHEDULING	true	Control worker asynchronous scheduling; streaming forces this off
NIM_DYNAMO_ENABLE_MULTIMODAL	true	Enable Dynamo multimodal handling for image/video requests
NIM_DYNAMO_ARGS	Empty object	Advanced Dynamo worker CLI overrides as a JSON object; use underscore keys such as discovery_backend, namespace, and endpoint
NIM_DYNAMO_PROXY_PORT	8080	Private frontend’s loopback HTTP port; integer from 1 through 65535, different from NIM_HTTP_API_PORT (default 8000)
NIM_DYNAMO_PROXY_REQUEST_TIMEOUT_SECS	120 in the NIM image	NIM proxy request timeout in seconds; persistent session operations use the separate streaming timeout below
NIM_DYNAMO_PROXY_FORWARDED_REQUEST_HEADERS_LIST	Empty additions	Comma- or whitespace-separated extra request headers to forward through the NIM proxy; built-in tracing, affinity, worker-routing, and tenant headers remain enabled

Argument precedence is defaults, then NIM_DISCOVERY_BACKEND (including its legacy JSON fields), then the individual scheduling/multimodal variables, then NIM_DYNAMO_ARGS. Streaming engine constraints are applied last and cannot be overridden by these arguments. The supplied YAML files use the legacy JSON form:

# Regular Dynamo
NIM_DISCOVERY_BACKEND={"async_scheduling":true,"discovery_backend":"kubernetes","enable_multimodal":true}
# Dynamo with REST streaming
NIM_DISCOVERY_BACKEND={"async_scheduling":false,"discovery_backend":"kubernetes","enable_multimodal":true}

Dynamo Streaming

Set NIM_ENABLE_STREAMING=1 alongside NIM_DYNAMO_WORKER=true on workers. Use the same NIM image for the shared frontend, running /opt/nim/.venv/bin/python -m streaming.dynamo.frontend from /opt/nim. The stock python3 -m dynamo.frontend command does not serve video-session routes. The NIM worker selects its private streaming frontend automatically.

Name	Default	Use
NIM_DYNAMO_STREAMING_ENDPOINT	Derived <namespace>.<component>.streaming; standalone adapter fallback dynamo.backend.streaming	Set the same explicit RPC endpoint on the shared frontend and workers. The YAML uses cosmos3-reasoner.backend.streaming; use a distinct endpoint for each model pool
NIM_DYNAMO_STREAMING_REQUEST_TIMEOUT_S	180	Bound each REST session create, frame, or delete operation, including upload and inference, in seconds; positive finite number. Set on the shared frontend and workers
DYN_HTTP_HOST, DYN_HTTP_PORT	0.0.0.0, 8000	Streaming adapter listener defaults; explicit --http-host and --http-port arguments take precedence
DYN_TLS_CERT_PATH, DYN_TLS_KEY_PATH	Unset	Optional certificate and key paths for TLS on the streaming adapter; set both together, or use --tls-cert-path and --tls-key-path

The adapter’s --streaming-endpoint flag overrides its environment endpoint. Keep it aligned with the workers. Use aggregated workers with a local encoder. Disaggregated prefill/decode and remote encoders are unsupported. The streaming capacity, retention, idle-timeout, and engine controls also apply. Streaming forces synchronous scheduling and disables the multimodal processor cache. The request timeout above is separate from NIM_STREAMING_REST_IDLE_TIMEOUT_S, which expires inactive sessions. REST session ownership does not depend on the Chat Completions affinity TTL.

WebSocket (/v1/streaming/ws) is unsupported in Dynamo mode, on both shared frontend and worker NIM endpoints. The streaming adapter and enabled worker NIM proxy expose GET /v1/streaming/config and the REST session routes. The stock dynamo.frontend launcher does not attach these streaming routes. Refer to Dynamo With Streaming for the manifest and REST request examples.

Advanced Settings

Leave these at their defaults unless you have measured a need.

Name	Runtime	Default	Use
NIM_OFFLOAD_MODE	Generator	Automatic	Request none, model, or layer when the model provides it
NIM_GPU_MEMORY_HEADROOM_GIB	Reasoner	4	Free binary GiB reserved per discrete GPU for selection and sizing; lowering it raises OOM risk
NIM_MODEL_PROFILE	Both	Empty	Pin an exact profile ID from the current image
NIM_VIDEO_SAVE_QUALITY	Generator	7	VP9 output quality 1–9; affects encoding, not diffusion
NIM_ENABLE_TORCH_COMPILE	Generator	true	Enable the compilation path
NIM_CACHE_BACKEND	Generator	Edge-DROID: none; other variants: sea_cache	Diffusion cache: sea_cache, cache_dit, or none; refer to Diffusion Cache, Attention, and VAE Decode
NIM_CACHE_CONFIG	Generator	Empty object	Override individual diffusion-cache settings as JSON; rejected when the backend is none
DIFFUSION_ATTENTION_BACKEND	Generator	SAGE_ATTN on compute capability 8.0 through 9.x without sequence parallelism; otherwise automatic	Override the diffusion attention backend
NIM_VAE_FAST_PATH	Generator	channels_last	VAE decode path: channels_last, lossless, or off
NIM_ENABLE_VAE_PARALLELISM	Generator	true when one replica spans more than one GPU	Shard VAE decode spatially across the replica’s GPUs
NIM_VAE_SPATIAL_TEMPORAL_CHUNK_SIZE	Generator	Automatic	Positive number of latent frames per sharded VAE decode chunk
NIM_LINEAR_BACKEND	Both	FP8 on DGX Spark or Jetson AGX Thor: torch; other quantized Generator profiles: cutlass; otherwise unset	auto, cutlass, flashinfer_cutlass, flashinfer_cutedsl, or torch for quantized linear layers; an incompatible backend fails at model load
NIM_TRITON_LOG_VERBOSE	Generator	0	Backend log verbosity for diagnosis
NIM_MAX_SEQUENCE_LENGTH	Generator	5120	Startup prompt-token sequence length
NIM_ALLOW_UNSAFE_TRANSFER	Generator	false	Bypass Transfer’s VRAM admission check for an approved diagnostic; can fail with OOM
NIM_DFLASH_BF16_KV_CACHE	Reasoner	false on NVIDIA Hopper; true otherwise	BF16 DFlash KV cache; can improve acceptance but uses more memory
NIM_DFLASH_CONFIG	Reasoner	Empty object	Extra vLLM DFlash speculative-config fields as JSON
NIM_MAX_NUM_BATCHED_TOKENS	Reasoner	16384 with unchunked multimodal input (24300 on Edge); otherwise 8192	Scheduler token budget; unchunked multimodal input enforces at least 16384 (24300 on Edge)
NIM_MAX_NUM_SEQS	Reasoner	256; 8 on RTX PRO 4500 Blackwell	Maximum scheduled sequences
NIM_ENABLE_KV_CACHE_REUSE	Reasoner	true	Prefix/KV-cache reuse
NIM_ENABLE_CHUNKED_PREFILL	Reasoner	true	Chunked prefill
NIM_DISABLE_CHUNKED_MM_INPUT	Reasoner	true	Require one whole multimodal item per scheduler iteration
NIM_DISABLE_MM_PREPROCESSOR_CACHE	Reasoner	false	Disable the multimodal preprocessor cache
NIM_MEDIA_IO_KWARGS	Reasoner	Video backend pynvvc	Replace the whole operator-level media I/O JSON object
NIM_MM_PROCESSOR_KWARGS	Reasoner	Unset	Operator-level multimodal processor options as a JSON object
NIM_VIDEO_PRUNING_RATE	Reasoner	0.6	Video-token pruning from 0 through 1; 0, none, null, or an empty value disable pruning
NIM_VIDEO_PRUNING_METHOD	Reasoner	vidcom2	vidcom2 or evs when pruning is enabled
NIM_GUIDED_DECODING_BACKEND	Reasoner	xgrammar	Structured-output backend

Secret Handling

Inject NGC_API_KEY, HF_TOKEN, and NIM_PROMPT_UPSAMPLING_API_KEY as separate secrets through the deployment environment, never through source control, model-source URIs, image layers, requests, notebooks, or logs.