Is this page helpful?

Advanced Configuration#

NIM LLM and VLM uses a layered configuration system that resolves values from multiple sources — CLI arguments, environment variables, runtime config files, and model profile tags — with well-defined priorities and provenance tracking. This page describes how the configuration system works and how to use it for advanced deployment scenarios.

For basic configuration such as model path, cache, and logging, refer to Environment Variables. Backend argument names and behavior can change between backends and releases. Use arguments supported by the backend version in your image. Refer to the vLLM CLI documentation or the SGLang server argument documentation.

Each container image includes one inference engine, vLLM or SGLang. NIM’s canonical arguments provide a common interface and are translated to the backend’s native flags. Examples that use native flags identify the backend. To check the engine and resolved arguments for your image, use --dry-run.

Configuration Priority#

Configuration values are resolved from multiple sources with the following priority (highest to lowest):

Priority

Source

Description

1 (highest)

CLI Arguments

Arguments passed directly to the container, such as docker run <image> --tensor-parallel-size 4

2

Passthrough Arguments

NIM_PASSTHROUGH_ARGS environment variable, parsed as CLI-style arguments

3

Environment Variables

NIM-specific NIM_* variables that map to active-backend arguments

4

Runtime Config

runtime_config.json in the model workspace

5

Profile Tags

Values from selected profile metadata, including tp, pp, and nim_json_args

6

Internal Values

Programmatically derived values and memory guardrails

7 (lowest)

NIM Defaults

Built-in defaults applied only when no other source sets the field

Higher-priority sources overwrite lower-priority sources. If the same parameter is set in multiple sources, the highest-priority value wins.

JSON-object values are an exception: they compose across sources instead of replacing one another. For example, if runtime_config.json and the command line both set a JSON-valued backend option, such as vLLM’s --compilation-config, NIM recursively merges the two objects.

For keys defined in both objects, the value from the higher-priority source takes precedence. Keys defined in only one object are preserved. In strict mode, different values for the same key cause a fatal conflict, but values without conflicts can still be merged.

Configuration Sources#

NIM resolves configuration values from the following sources.

CLI Arguments#

Pass backend arguments directly to the container when the deployment interface supports container arguments. Direct arguments resolve at CLI priority and preserve any NIM_PASSTHROUGH_ARGS value provided by the image. When the first container argument is an option, NIM uses nim-serve as the default action. The following example uses NIM canonical arguments, which are accepted on both the vLLM and SGLang images. NIM renders each to the active backend’s flag:

docker run --gpus=all \
  -e NIM_MODEL_PATH=hf://meta-llama/Llama-3.1-8B-Instruct \
  -p 8000:8000 \
  ${NIM_LLM_MODEL_FREE_IMAGE}:2.1.4 \
  --tensor-parallel-size 4 \
  --enable-prefix-caching \
  --gpu-memory-utilization 0.9

For example, --tensor-parallel-size becomes --tp-size and --gpu-memory-utilization becomes --mem-fraction-static on SGLang.

NIM registers its canonical boolean parameters with paired forms on both backends:

  • --enable-prefix-caching sets the value to True

  • --no-enable-prefix-caching sets the value to False

Native boolean flags follow the backend’s argument definitions. Many vLLM flags support paired positive and negative forms. For SGLang-native flags, use the documented spelling rather than adding a --no- prefix.

Direct container arguments take precedence over NIM_PASSTHROUGH_ARGS. In strict mode, NIM exits if the two sources set conflicting values. Arguments with JSON-object values can still compose when their keys do not conflict, as described in Configuration Priority above. For duplicate or contradictory direct arguments, such as --enable-prefix-caching followed by --no-enable-prefix-caching, the last occurrence wins.

Passthrough Arguments (NIM_PASSTHROUGH_ARGS)#

Use NIM_PASSTHROUGH_ARGS as an environment-only fallback when the deployment interface cannot supply container arguments. The value is parsed with shlex.split() and resolves at passthrough priority.

Warning

Some model-specific NIM images define validated backend defaults in the image’s NIM_PASSTHROUGH_ARGS environment variable, and some of those defaults are required for startup. Docker and Kubernetes replace an image environment value when a deployment sets a variable with the same name. They do not append to the image value. Setting NIM_PASSTHROUGH_ARGS to only your custom arguments can remove those defaults and prevent the NIM from starting.

Use the direct-argument field for your deployment interface:

  • For Docker, place the arguments after the image reference.

  • For a Kubernetes Pod, use the container args field.

  • For the NIM Helm chart, use customArgs.

  • For a NIMService that does not set spec.multiNode, use spec.args. Do not use spec.args for a multi-node NIMService; the current LeaderWorkerSet path does not propagate it to the leader or worker NIM containers.

When an environment variable is the only available interface, first pull the exact image tag or digest and check whether the image already defines NIM_PASSTHROUGH_ARGS:

NIM_IMAGE="<exact-image-tag-or-digest>"
docker pull "${NIM_IMAGE}"
docker image inspect "${NIM_IMAGE}" \
  --format '{{range .Config.Env}}{{println .}}{{end}}' | \
  grep '^NIM_PASSTHROUGH_ARGS='

If the command prints a value, copy everything after the first = and append your custom arguments. The following example uses vLLM arguments. If the image provides --block-size 32 and you want to add --max-num-seqs 16, the complete value is:

--block-size 32 --max-num-seqs 16

For an SGLang image, use SGLang flags, for example --max-running-requests 16. A vLLM-only flag in NIM_PASSTHROUGH_ARGS fails startup on an SGLang image, and an SGLang-only flag fails on a vLLM image.

If the command prints nothing, the image does not define the variable and the complete value consists only of your custom arguments. Do not deploy an override if you cannot retrieve an existing image-provided value. Append custom arguments last so that they win when the combined string repeats an option. For Kubernetes or Helm, put the complete value in the manifest or values file before deployment. A Pod specification cannot extend the image’s value by referring to $(NIM_PASSTHROUGH_ARGS) in an env entry with the same name.

When the combined value repeats an option that accepts one value, NIM uses the last occurrence. For a JSON- or object-valued option such as vLLM’s --compilation-config, merge the objects and include the option once. NIM merges JSON objects across configuration sources, not repeated options within one NIM_PASSTHROUGH_ARGS value. Strict mode cannot detect those duplicates because parsing keeps only the last value.

Passthrough arguments use shell-style quoting through shlex.split(); preserve that quoting for JSON-valued arguments. Argument names and supported values must match the backend version in the image.

Custom Middleware (vLLM Backend Only)#

Operator-supplied middleware is supported only on vLLM images. For request correlation on either backend, use the X-Request-Id header as described in Logging and Observability.

For vLLM, --middleware is a repeatable option. NIM preserves each occurrence within one configuration source in argument order. Normal configuration priority still applies between sources: a higher-priority middleware list replaces, rather than appends to, a lower-priority list.

vLLM installs each class with app.add_middleware(). Starlette prepends, so the last --middleware on the argv is outermost.

NIM therefore emits customer classes before its own redirect and request-id middleware. NIM stays outermost, and your class sits innermost. Innermost means the class sees the rewritten /v1/... path and can read any caller-supplied X-Request-Id after NIM records it. Repeat the flag to register more than one class.

The import path must be a plain ASGI middleware class. Do not use VLLM_PLUGINS to replace FastAPI’s middleware list, which can drop Prometheus http_request_* series.

--middleware is not currently supported in NIM with Dynamo deployments (NIM_DYNAMO_SINGLE / NIM_DYNAMO_WORKER / NIM_DYNAMO_FRONTEND, including DynamoGraphDeployment). The flag is stripped before dynamo.vllm starts. The frontend cannot install operator ASGI classes.

For example, set a Kubernetes container’s args field:

args:
  - "--middleware"
  - "mypkg.logging.CallIDMiddleware"
  - "--async-scheduling"

NIM Environment Variables#

NIM defines a small set of environment variables that map to backend arguments. These provide a stable, NIM-specific interface for commonly used parameters.

For an argument that does not have a dedicated NIM environment variable, pass it directly to the container. Use NIM_PASSTHROUGH_ARGS only when direct container arguments are unavailable.

NIM renders each variable to the active backend’s flag:

NIM Environment Variable

Type

Default

vLLM Argument

SGLang Argument

NIM_TENSOR_PARALLEL_SIZE

int

1

--tensor-parallel-size

--tp-size

NIM_PIPELINE_PARALLEL_SIZE

int

1

--pipeline-parallel-size

--pp-size

NIM_EXPERT_PARALLEL_SIZE

int

1

Emits --enable-expert-parallel when value > 1; omitted at 1

--ep-size <N> when value > 1; omitted at 1

NIM_DATA_PARALLEL_SIZE

int

1

--data-parallel-size

--dp-size <N> --enable-dp-attention when value > 1; omitted at 1

NIM_MAX_MODEL_LEN

int

auto

--max-model-len

--context-length

NIM_TRUST_CUSTOM_CODE

bool

false

--trust-remote-code

--trust-remote-code

NIM_DISABLE_CUDA_GRAPH

bool

false

--enforce-eager

--disable-decode-cuda-graph --disable-prefill-cuda-graph

Parallelism Semantics#

Tensor parallelism (TP) splits model computation across GPUs. Expert parallelism (EP) distributes mixture-of-experts (MoE) experts across GPUs. Data parallelism (DP) distributes request batches across replicas. On vLLM, each DP replica has its own TP GPU group. With NIM_EXPERT_PARALLEL_SIZE > 1, NIM requires EP = TP * DP and derives DP from EP and TP if DP is unset.

On SGLang, NIM_DATA_PARALLEL_SIZE=N with N > 1 emits both --dp-size N and --enable-dp-attention. This enables DP for attention within the existing TP GPU group, without creating independent full-model replicas or multiplying the GPU count by DP. SGLang validates the topology for the selected model and communication backend; NIM does not derive DP from EP.

For example, a supported SGLang MoE topology with TP=8, EP=8, DP=8, and pipeline parallelism set to 1 uses eight GPUs. Each GPU handles an attention group, and the experts are distributed across the same eight GPUs. For topology examples and requirements, refer to DP, DPA and SGLang DP Router and Expert Parallelism.

NVFP4 Emulation#

By default, NVFP4 model profiles are only selectable on Blackwell GPUs (compute capability ≥ 10.0), which have native FP4 tensor cores. Set NIM_ALLOW_NVFP4_EMULATION=1 to also make NVFP4 profiles selectable on pre-Blackwell GPUs for GPUs like Ampere and Hopper (compute capability 8.0–9.x) with the vLLM or SGLang backend — these dequantize the FP4 weights to BF16 in-kernel using the Marlin weight-only kernel.

NIM Environment Variable

Description

Default

NIM_ALLOW_NVFP4_EMULATION

Allows NVFP4 profiles to be selected on pre-Blackwell GPUs (Ampere/Ada/Hopper, compute capability 8.0–9.x) with vLLM/SGLang, using Marlin weight-only emulation.

0 (off)

Note

  • Marlin emulation is weight-only (FP4 weights dequantized to BF16 in-kernel), so it may be slower than natively-supported precisions on the same GPU.

  • TensorRT-LLM profiles are unaffected and are supported on Blackwell only (no emulation path).

  • The relaxation is applied per host, not per GPU: it takes effect only when every detected GPU is in the compute-capability 8.0–9.x window. If any GPU falls outside that window, for example Volta/Turing (< 8.0) or Blackwell (≥ 10.0), the flag has no effect and NVFP4 profiles remain Blackwell-only for the entire host. This includes mixed nodes (for example, Blackwell and Ampere/Hopper), which are not relaxed.

  • On A100 (compute capability 8.0), dense (non-MoE) NVFP4 profiles run correctly; FP4 Mixture-of-Experts (MoE) kernels may not be available in all backend versions.

Runtime Config#

You can place a runtime_config.json file in the model workspace directory. NIM reads it automatically if it is present:

{
  "tensor_parallel_size": 2,
  "enable_prefix_caching": true,
  "max_model_len": 8192
}

Unknown keys are passed through to the backend as-is and must be valid arguments for the engine in your image.

Embedded Passthrough Arguments#

The file also accepts a reserved NIM_PASSTHROUGH_ARGS key that holds a full CLI-style string. The string is parsed with the same rules as the NIM_PASSTHROUGH_ARGS environment variable: known arguments receive their registered types, canonical NIM booleans accept --flag and --no-flag, native arguments follow the backend parser, and denied arguments are dropped with a warning.

{
  "NIM_PASSTHROUGH_ARGS": "--max-model-len 8192 --enable-prefix-caching"
}

The embedded string and the flat keys resolve at the same runtime-config priority. Two rules govern how the two in-file forms interact:

  • Explicit keys win. If the same file sets a parameter both inside the embedded string and as a flat key, the flat key takes precedence.

  • Strictness differs by form. When strict passthrough parsing is active (NIM can build the backend’s known-argument list), an unrecognized flag inside the embedded string is a fatal error, matching how the backend treats its own command line. If known-argument parsing is unavailable — the backend is not installed, or its parser cannot be constructed in the environment — unrecognized flags are instead retained leniently for passthrough. An unknown flat key is always retained and passed through to the backend.

To negate a boolean argument with a flat key, set "flag": false or use the equivalent "no-flag": true form. NIM renders known arguments according to the backend’s parser, including negative spellings where supported. For unknown boolean keys, vLLM receives --no-<flag> and SGLang omits the flag. Validate native argument names against the engine in your image.

Profile Tags#

Model profiles include metadata tags that configure parallelism. These values are extracted automatically when a profile is selected:

Profile Tag

NIM Parameter

tp

tensor_parallel_size

pp

pipeline_parallel_size

For example, a profile named vllm-fp8-tp2-pp1 (vLLM image) or sglang-fp8-tp2-pp1 (SGLang image) sets tensor_parallel_size=2 and pipeline_parallel_size=1 at profile priority.

Profile Engine Arguments (nim_json_args)#

A profile can ship engine arguments directly in its manifest through the reserved nim_json_args tag. The tag value is a string that contains a flat-key JSON object. Use canonical engine-argument keys (max_model_len), not the NIM_* environment-variable spellings that runtime_config.json also accepts — the manifest generator rejects those in this tag so the blob has one spelling. Non-canonical keys are backend-native and must be valid for the engine in the image. The following example is from a vLLM profile:

tags:
  tp: '2'
  profile: throughput
  nim_json_args: '{"moe_backend": "flashinfer", "kv_cache_dtype": "fp8", "compilation_config": {"level": 3}}'
  nim_description: 'Turbo-certified H100 throughput profile'

Because the arguments live in the manifest, you can read them from the manifest itself (the NIM ships one at /opt/nim/etc/default/model_manifest.yaml) without downloading a profile’s workspace files. Note that list-model-profiles does not print them: it shows each profile’s synthesized name, and reserved nim_* tags are deliberately excluded from that name so a JSON blob cannot make it unreadable.

Key behavior:

  • Values apply at profile priority, so runtime_config.json, NIM_PASSTHROUGH_ARGS, environment variables, and CLI arguments all override them.

  • Known parameters are type-coerced; unknown keys pass through to the backend. JSON object and array values reach the backend as JSON strings.

  • Secret-bearing keys (for example, api_key and ssl_*) are ignored, matching the CLI and NIM_PASSTHROUGH_ARGS surfaces.

  • A malformed tag logs a warning and is skipped; the profile still serves with its other tags.

  • Dedicated structural tags (tp, pp) win over the same keys inside nim_json_args.

  • Speculative-decoding keys are handled separately — see below.

The nim_description tag holds a human-readable note about the profile. It is never parsed as configuration and is printed at startup and in --dry-run output.

Speculative decoding in nim_json_args#

A profile can carry its speculative-decoding configuration in the tag, in the same two shapes runtime_config.json supports: a nim_specdec envelope, or flat speculative_* keys. The nim_specdec envelope, the draft_uri key, and the NIM_SPECDEC_ENABLE gate are identical on both backends. The speculative-decoding keys inside the envelope use the engine’s native names.

tags:
  nim_json_args: '{"nim_specdec": {"speculative_config": "{\"method\": \"eagle3\", \"model\": \"eagle3_draft\", \"num_speculative_tokens\": 3}", "draft_uri": "ngc://<org>/<team>/<draft-model>:<tag>", "max_model_len": 8192}}'

tags:
  nim_json_args: '{"nim_specdec": {"speculative_algorithm": "EAGLE3", "speculative_draft_model_path": "eagle3_draft", "speculative_num_steps": 3, "speculative_eagle_topk": 1, "speculative_num_draft_tokens": 4, "draft_uri": "ngc://<org>/<team>/<draft-model>:<tag>"}}'

These keys are not applied at profile priority like the rest of the tag. They are resolved through the same NIM_SPECDEC_ENABLE gate as the file, which means:

  • With NIM_SPECDEC_ENABLE unset, a profile whose tag carries a spec config serves with speculative decoding on by default.

  • NIM_SPECDEC_ENABLE=0 turns it off for that profile, exactly as it does for a file-shipped config.

  • draft_uri names the EAGLE3 draft model and is downloaded through the normal path; it never reaches the backend as an argument.

  • Keys inside the nim_specdec envelope — including spec-coupled ones such as max_model_len — apply only when speculative decoding is enabled. NIM removes values from the previous speculative configuration before resolving a new one.

Precedence runs lowest to highest: the tag’s flat keys, the tag’s envelope, the file’s flat keys, the file’s envelope, then NIM_SPECDEC_ARGS. So a workspace runtime_config.json remains a complete override for a tag-shipped spec config.

When a profile supplies arguments in both runtime_config.json and nim_json_args, the file takes precedence. Unknown boolean keys set to false render as --no-<flag> on vLLM and are omitted on SGLang. Use supported argument names to avoid startup failures or silently omitted settings.

NIM Defaults#

The following default values are applied only if no other source sets the field:

Parameter

Default

tensor_parallel_size

1

pipeline_parallel_size

1

Override Warnings and Strict Mode#

NIM reports configuration overrides with warnings, and strict mode changes that behavior.

Override Detection#

When a higher-priority source overwrites a value from a lower-priority source, NIM logs a warning:

WARNING: Config override: 'tensor_parallel_size' changed from 2 (RUNTIME) to 8 (CLI)

Strict Mode#

Set NIM_STRICT_ARG_PROCESSING=true to treat override warnings as errors:

export NIM_STRICT_ARG_PROCESSING=true

In strict mode, the container exits with an error if any configuration override is detected between non-default sources. This setting is useful in the following cases:

  • CI/CD pipelines, where you want to catch configuration conflicts.

  • Production deployments, where configuration should be deterministic.

This behavior also applies when a direct container argument conflicts with an image-provided NIM_PASSTHROUGH_ARGS value. Add disjoint arguments in strict mode. If you intend to replace an image-provided value with a direct argument, leave strict mode disabled for that deployment.

Example:

export NIM_STRICT_ARG_PROCESSING=true
export NIM_TENSOR_PARALLEL_SIZE=2

docker run ... nim-serve --tensor-parallel-size 4

Result:

ERROR: Config override detected in strict mode: 'tensor_parallel_size' changed from 2 (ENV) to 4 (CLI).
Set NIM_STRICT_ARG_PROCESSING=false to allow intentional overrides.

Dry Run#

Use --dry-run to print the fully resolved configuration and the resulting backend arguments without starting the server:

docker run --gpus=all \
  -e NIM_MODEL_PATH=hf://meta-llama/Llama-3.1-8B-Instruct \
  -e NIM_TENSOR_PARALLEL_SIZE=2 \
  ${NIM_LLM_MODEL_FREE_IMAGE}:2.1.4 \
  --max-model-len 4096 \
  --dry-run

The output includes the provenance for each parameter and the final backend argument list. Image-provided passthrough values appear with PASSTHROUGH provenance, and arguments supplied directly to the container appear with CLI provenance.

Denied Arguments#

Certain backend CLI arguments are blocked in NIM containers because the nginx proxy or other system components manage them. If you pass a denied argument, NIM logs a warning and ignores it. The Backend column shows which backend denies each argument.

Denied Argument

Backend

Reason

NIM Alternative

--host

Both

Networking is managed by nginx

—

--port

Both

Port is managed by NIM

NIM_SERVER_PORT (external)

--ssl-keyfile, --ssl-keyfile-password

Both

SSL/TLS is managed by nginx

NIM_SSL_MODE, NIM_SSL_KEY_PATH

--ssl-certfile

Both

SSL/TLS is managed by nginx

NIM_SSL_MODE, NIM_SSL_CERTS_PATH

--ssl-ca-certs

Both

SSL/TLS is managed by nginx

NIM_SSL_CA_CERTS_PATH

--ssl-cert-reqs

Both

SSL/TLS is managed by nginx

NIM_SSL_MODE=MTLS

--api-key, --admin-api-key

Both

Authentication should be managed externally

—

--log-level

SGLang

Logging is managed by NIM

NIM_LOG_LEVEL

--root-path

vLLM

API routing is managed by nginx

—

--uvicorn-log-level

vLLM

Logging is managed by NIM

NIM_LOG_LEVEL

--allowed-origins

vLLM

CORS is managed by nginx

NIM_CORS_ALLOW_ORIGINS

--allowed-methods

vLLM

CORS is managed by nginx

NIM_CORS_ALLOW_METHODS

--allowed-headers

vLLM

CORS is managed by nginx

NIM_CORS_ALLOW_HEADERS

--enable-custom-logit-processor

SGLang

Custom logit processors are disabled in NIM for security reasons

—

Validation and Error Handling#

NIM validates configuration at multiple stages and handles errors differently depending on where validation occurs.

Where Validation Occurs#

Stage

Validated Item

Error Behavior

CLI Parsing

Argument format, type conversion

Warning logged, invalid value skipped; an unknown flag is a fatal error when strict passthrough parsing is active

Env Parsing

Type conversion

Warning logged, invalid value skipped

Runtime Config

JSON syntax, type conversion

Warning logged, invalid key skipped

Config Merge

Override detection

Warning (or error in strict mode)

Backend Startup

Argument validity, model compatibility

The backend exits with an error

Invalid Parameter Handling#

  • Invalid NIM parameters: Logged as warnings; the container continues with the remaining valid values.

  • Unknown parameters: Passed through to the backend as-is when lenient parsing applies; the backend validates them.

  • Invalid backend parameters: The backend validates and reports errors at startup.

Example Error Messages#

The following examples show representative error messages:

# NIM-side type error (warning, non-fatal)
WARNING: Failed to parse CLI arg '--tensor-parallel-size' with value 'abc': invalid literal for int()

# Fatal validation error at startup
ValueError: tensor_parallel_size must be a positive integer

Provenance Tracking#

NIM records where each resolved configuration value came from. At startup, it logs a structured nim_config_resolved event:

{
  "event": "nim_config_resolved",
  "backend": "vllm",
  "backend_version": "<version>",
  "config": {
    "tensor_parallel_size": {
      "value": 4,
      "source": "CLI",
      "previous": {
        "value": 2,
        "source": "PASSTHROUGH"
      }
    },
    "enable_prefix_caching": {
      "value": true,
      "source": "RUNTIME"
    }
  }
}

backend is vllm or sglang, matching the engine built into the image; backend_version is that engine’s version.

When one source overrides another, previous identifies the value and source that were replaced. Run the container with --dry-run to inspect the full provenance report without starting the backend server.

GPU Memory Management#

NIM includes automatic GPU memory management to prevent out-of-memory (OOM) failures on constrained hardware. This process uses two mechanisms: automatic clamping of gpu_memory_utilization and a post-selection memory warning.

Automatic GPU Memory Clamping#

On certain hardware configurations, NIM automatically reduces the gpu_memory_utilization parameter, set by NIM_KVCACHE_PERCENT and rendered to the active backend’s memory-fraction flag, to prevent OOM errors. On vLLM, this fraction (--gpu-memory-utilization) bounds the engine’s total GPU memory. The following caps apply:

Condition

Cap

Reason

vGPU guest (partitioned GPU)

0.75

GPU memory is shared or partitioned across virtual machines

UMA device (DGX Spark, GH200)

0.50

CPU and GPU share the same physical memory

Busy GPU (other processes using memory)

free_ratio - 0.05

Prevents contention with other workloads

The clamped value is applied at a low internal priority, so any explicit setting from environment variables, CLI arguments, or --gpu-memory-utilization takes precedence automatically. The minimum floor is 0.10 (NIM never sets GPU memory utilization below 10%).

On SGLang, NIM_KVCACHE_PERCENT maps to --mem-fraction-static, which reserves memory for model weights and the KV cache. Activations, CUDA graphs, and other runtime allocations need memory outside that fraction. The same clamping rules apply to both backends, but a value that works with one engine can require adjustment for the other. Refer to GPU Memory Troubleshooting.

Post-Selection Memory Warning#

After selecting a profile and resolving all configuration, NIM estimates the total VRAM required for the model (weights, KV cache, activations, and overhead) and compares it to available GPU memory. If the estimate exceeds available memory, NIM logs an advisory warning:

WARNING: Estimated VRAM (45.2 GB) exceeds available GPU memory (39.6 GB).
Consider reducing context length with --max-model-len=4096 (estimated 30.1 GB).

This warning is advisory only. NIM proceeds with startup regardless. The suggested --max-model-len value, if provided, indicates a context length that would fit within available memory.

To apply the suggestion, pass --max-model-len as a CLI argument:

docker run --gpus=all \
  -e NIM_MODEL_PATH=hf://meta-llama/Llama-3.1-8B-Instruct \
  -p 8000:8000 \
  ${NIM_LLM_MODEL_FREE_IMAGE}:2.1.4 \
  --max-model-len 4096

SSL/TLS Configuration#

NIM terminates TLS at the nginx proxy layer. Native backend SSL arguments such as --ssl-keyfile and --ssl-certfile are denied on both vLLM and SGLang. Use the NIM SSL variables instead. Refer to Architecture for more information about the proxy layer.

Variable

Description

Default

NIM_SSL_MODE

SSL mode: DISABLED, TLS, or MTLS

DISABLED

NIM_SSL_KEY_PATH

Path to TLS private key

—

NIM_SSL_CERTS_PATH

Path to TLS certificate

—

NIM_SSL_CA_CERTS_PATH

Path to CA certificate (required for MTLS)

—

Use one of the following NIM_SSL_MODE values to enable SSL:

  • TLS mode: Server presents a certificate; client certificate is not required. To activate, set NIM_SSL_MODE=TLS. Requires NIM_SSL_CERTS_PATH and NIM_SSL_KEY_PATH.

  • Mutual TLS mode: Both server and client present certificates. To activate, set NIM_SSL_MODE=MTLS. Requires NIM_SSL_CERTS_PATH, NIM_SSL_KEY_PATH, and NIM_SSL_CA_CERTS_PATH.

The following example configures NIM to use TLS mode:

docker run --gpus=all \
  -e NIM_MODEL_PATH=hf://meta-llama/Llama-3.1-8B-Instruct \
  -e NIM_SSL_MODE=TLS \
  -e NIM_SSL_KEY_PATH=/certs/server.key \
  -e NIM_SSL_CERTS_PATH=/certs/server.crt \
  -v /path/to/certs:/certs:ro \
  -p 8000:8000 \
  ${NIM_LLM_MODEL_FREE_IMAGE}:2.1.4

CORS Configuration#

CORS is handled by the nginx proxy. The following variables control CORS headers:

Variable

Description

Default

NIM_CORS_ALLOW_ORIGINS

Allowed origins

*

NIM_CORS_ALLOW_METHODS

Allowed HTTP methods

GET, POST, PUT, DELETE, PATCH, OPTIONS

NIM_CORS_ALLOW_HEADERS

Allowed request headers

Content-Type, Authorization, X-Request-Id, X-Session-Id, X-Correlation-Id

NIM_CORS_EXPOSE_HEADERS

Headers exposed to the browser

X-Request-Id

NIM_CORS_MAX_AGE

Preflight cache duration (seconds)

3600

Note

On the vLLM backend, --allowed-origins, --allowed-methods, and --allowed-headers are denied because CORS is managed at the nginx layer. On both backends, configure CORS only through the NIM_CORS_* environment variables.

Examples#

The following examples show how configuration precedence, override warnings, and passthrough arguments work in practice.

CLI Overrides an Environment Variable#

The following example shows how a direct CLI argument overrides an environment variable:

export NIM_TENSOR_PARALLEL_SIZE=2

docker run --gpus=all \
  -e NIM_MODEL_PATH=hf://meta-llama/Llama-3.1-8B-Instruct \
  -e NIM_TENSOR_PARALLEL_SIZE \
  -p 8000:8000 \
  ${NIM_LLM_MODEL_FREE_IMAGE}:2.1.4 \
  nim-serve --tensor-parallel-size 4

This resolves tensor_parallel_size to 4, because the CLI value overrides the environment variable value.

NIM logs the following warning:

WARNING: Config override: 'tensor_parallel_size' changed from 2 (ENV) to 4 (CLI)

Full Priority Chain#

The following example shows how tensor_parallel_size resolves when every source sets a value:

runtime_config.json:        {"tensor_parallel_size": 1, "enable_prefix_caching": true}
Environment:                NIM_TENSOR_PARALLEL_SIZE=2
NIM_PASSTHROUGH_ARGS:       --tensor-parallel-size 3
CLI:                        nim-serve --tensor-parallel-size 4 --no-enable-prefix-caching

The resolved values are:

  • tensor_parallel_size = 4 (CLI overrides passthrough overrides env overrides runtime)

  • enable_prefix_caching = False (CLI --no- overrides runtime true)

Kubernetes with Container Arguments#

The following Kubernetes example passes canonical NIM arguments through the container args field. Each option and value is a separate list item. Do not add NIM_PASSTHROUGH_ARGS to env; leaving it unset in the Pod specification preserves any value provided by the image.

apiVersion: apps/v1
kind: Deployment
spec:
  template:
    spec:
      containers:
        - name: nim-llm
          image: <NIM_LLM_MODEL_FREE_IMAGE>:2.1.4
          args:
            - "--enable-prefix-caching"
            - "--max-model-len"
            - "8192"
          env:
            - name: NGC_API_KEY
              valueFrom:
                secretKeyRef:
                  name: ngc-secret
                  key: api-key
            - name: NIM_MODEL_PATH
              value: "ngc://nim/meta/llama-3.1-8b-instruct"
            - name: NIM_CACHE_PATH
              value: "/opt/nim/.cache"
          ports:
            - containerPort: 8000
          livenessProbe:
            httpGet:
              path: /v1/health/live
              port: 8000
          readinessProbe:
            httpGet:
              path: /v1/health/ready
              port: 8000

Enabling TLS#

The following example enables TLS for a container deployment:

docker run --gpus=all \
  -e NIM_MODEL_PATH=hf://meta-llama/Llama-3.1-8B-Instruct \
  -e NIM_SSL_MODE=TLS \
  -e NIM_SSL_KEY_PATH=/certs/server.key \
  -e NIM_SSL_CERTS_PATH=/certs/server.crt \
  -v /path/to/certs:/certs:ro \
  -p 8000:8000 \
  ${NIM_LLM_MODEL_FREE_IMAGE}:2.1.4

After the container starts, you can validate the TLS configuration with a command such as the following:

curl --cacert /path/to/certs/ca.crt \
  https://localhost:8000/v1/health/ready