Is this page helpful?

Advanced Configuration#

NIM LLM uses a layered configuration system that resolves values from multiple sources — CLI arguments, environment variables, runtime config files, and model profile tags — with well-defined priorities and provenance tracking. This page describes how the configuration system works and how to use it for advanced deployment scenarios.

For basic configuration such as model path, cache, and logging, refer to Environment Variables. Backend argument names and behavior can change between backends and releases. Use arguments supported by the backend version in your image. Refer to the vLLM CLI documentation or the SGLang server argument documentation.

Configuration Priority#

Configuration values are resolved from multiple sources with the following priority (highest to lowest):

Priority

Source

Description

1 (highest)

CLI Arguments

Arguments passed directly to the container, such as docker run <image> --tensor-parallel-size 4

2

Passthrough Arguments

NIM_PASSTHROUGH_ARGS environment variable, parsed as CLI-style arguments

3

Environment Variables

NIM-specific NIM_* variables that map to active-backend arguments

4

Runtime Config

runtime_config.json in the model workspace

5

Profile Tags

Values from selected profile metadata, including tp, pp, and nim_json_args

6

Internal Values

Programmatically derived values and memory guardrails

7 (lowest)

NIM Defaults

Built-in defaults applied only when no other source sets the field

Higher-priority sources overwrite lower-priority sources. If the same parameter is set in multiple sources, the highest-priority value wins.

The exception is JSON-object values, which compose across sources instead of replacing each other. When two sources set the same argument to a JSON object – for example, --compilation-config in runtime_config.json and on the command line – NIM unpacks both objects and merges them recursively. The higher-priority source wins for any key that both objects set; keys that only one source sets always survive. In strict mode, a key set to differing values by both sources is a fatal conflict, while disjoint composition is allowed.

Configuration Sources#

NIM resolves configuration values from the following sources.

CLI Arguments#

Pass backend arguments directly to the container when the deployment interface supports container arguments. Direct arguments resolve at CLI priority and preserve any NIM_PASSTHROUGH_ARGS value provided by the image. When the first container argument is an option, NIM uses nim-serve as the default action. The following example uses vLLM arguments:

docker run --gpus=all \
  -e NIM_MODEL_PATH=hf://meta-llama/Llama-3.1-8B-Instruct \
  -p 8000:8000 \
  ${NIM_LLM_MODEL_FREE_IMAGE}:2.0.11 \
  --tensor-parallel-size 4 \
  --enable-prefix-caching \
  --gpu-memory-utilization 0.9

vLLM uses Python’s argparse.BooleanOptionalAction for boolean flags:

  • --enable-prefix-caching sets the value to True

  • --no-enable-prefix-caching sets the value to False

Direct container arguments take precedence over NIM_PASSTHROUGH_ARGS. In strict mode, NIM exits if the two sources set conflicting values. Arguments with JSON-object values can still compose when their keys do not conflict, as described in Configuration Priority above. For duplicate or contradictory direct arguments, such as --enable-xyz followed by --no-enable-xyz, the last occurrence wins.

Passthrough Arguments (NIM_PASSTHROUGH_ARGS)#

Use NIM_PASSTHROUGH_ARGS as an environment-only fallback when the deployment interface cannot supply container arguments. The value is parsed with shlex.split() and resolves at passthrough priority.

Warning

Some model-specific NIM images define validated backend defaults in the image’s NIM_PASSTHROUGH_ARGS environment variable, and some of those defaults are required for startup. Docker and Kubernetes replace an image environment value when a deployment sets a variable with the same name. They do not append to the image value. Setting NIM_PASSTHROUGH_ARGS to only your custom arguments can remove those defaults and prevent the NIM from starting.

Use the direct-argument field for your deployment interface:

  • For Docker, place the arguments after the image reference.

  • For a Kubernetes Pod, use the container args field.

  • For the NIM Helm chart, use customArgs.

  • For a NIMService that does not set spec.multiNode, use spec.args. Do not use spec.args for a multi-node NIMService; the current LeaderWorkerSet path does not propagate it to the leader or worker NIM containers.

When an environment variable is the only available interface, first pull the exact image tag or digest and check whether the image already defines NIM_PASSTHROUGH_ARGS:

NIM_IMAGE="<exact-image-tag-or-digest>"
docker pull "${NIM_IMAGE}"
docker image inspect "${NIM_IMAGE}" \
  --format '{{range .Config.Env}}{{println .}}{{end}}' | \
  grep '^NIM_PASSTHROUGH_ARGS='

If the command prints a value, copy everything after the first = and append your custom arguments. The following example uses vLLM arguments. If the image provides --block-size 32 and you want to add --max-num-seqs 16, the complete value is:

--block-size 32 --max-num-seqs 16

If the command prints nothing, the image does not define the variable and the complete value consists only of your custom arguments. Do not deploy an override if you cannot retrieve an existing image-provided value. Append custom arguments last so that they win when the combined string repeats an option. For Kubernetes or Helm, put the complete value in the manifest or values file before deployment. A Pod specification cannot extend the image’s value by referring to $(NIM_PASSTHROUGH_ARGS) in an env entry with the same name.

When the combined value repeats an option that accepts one value, NIM uses the last occurrence. For a JSON- or object-valued option such as --compilation-config, merge the objects and include the option once. NIM merges JSON objects across configuration sources, not repeated options within one NIM_PASSTHROUGH_ARGS value. Strict mode cannot detect those duplicates because parsing keeps only the last value.

Passthrough arguments use shell-style quoting through shlex.split(); preserve that quoting for JSON-valued arguments. Argument names and supported values must match the backend version in the image.

vLLM Middleware#

For vLLM, --middleware is a repeatable option. NIM preserves each occurrence within one configuration source in argument order. Normal configuration priority still applies between sources: a higher-priority middleware list replaces, rather than appends to, a lower-priority list.

vLLM installs each class with app.add_middleware(). Starlette prepends, so the last --middleware on the argv is outermost.

NIM therefore emits customer classes before its own redirect and request-id middleware. NIM stays outermost, and your class sits innermost. Innermost means the class sees the rewritten /v1/... path and can read any caller-supplied X-Request-Id after NIM records it. Repeat the flag to register more than one class.

The import path must be a plain ASGI middleware class. Do not use VLLM_PLUGINS to replace FastAPI’s middleware list, which can drop Prometheus http_request_* series.

--middleware is not currently supported in NIM with Dynamo deployments (NIM_DYNAMO_SINGLE / NIM_DYNAMO_WORKER / NIM_DYNAMO_FRONTEND, including DynamoGraphDeployment). The flag is stripped before dynamo.vllm starts. The frontend cannot install operator ASGI classes.

For example, set a Kubernetes container’s args field:

args:
  - "--middleware"
  - "mypkg.logging.CallIDMiddleware"
  - "--async-scheduling"

NIM Environment Variables#

NIM defines a small set of environment variables that map to backend arguments. These provide a stable, NIM-specific interface for commonly used parameters. The following table shows the vLLM mappings:

NIM Environment Variable

vLLM Argument

Type

Default

NIM_TENSOR_PARALLEL_SIZE

--tensor-parallel-size

int

1

NIM_PIPELINE_PARALLEL_SIZE

--pipeline-parallel-size

int

1

NIM_EXPERT_PARALLEL_SIZE

Emits --enable-expert-parallel when value > 1

int

1

NIM_DATA_PARALLEL_SIZE

--data-parallel-size

int

1

NIM_MAX_MODEL_LEN

--max-model-len

int

auto

NIM_TRUST_CUSTOM_CODE

--trust-remote-code

bool

false

NIM_DISABLE_CUDA_GRAPH

--enforce-eager

bool

false

For an argument that does not have a dedicated NIM environment variable, pass it directly to the container. Use NIM_PASSTHROUGH_ARGS only when direct container arguments are unavailable.

When the SGLang backend is in use, the same NIM environment variables map to SGLang’s CLI flags:

NIM Environment Variable

SGLang Argument

NIM_TENSOR_PARALLEL_SIZE

--tp-size

NIM_PIPELINE_PARALLEL_SIZE

--pp-size

NIM_EXPERT_PARALLEL_SIZE

Emits --ep-size <N> when value > 1; omitted at 1. SGLang itself enforces topology (e.g. ep_size * moe_dp_size == tp_size for DeepEP).

NIM_DATA_PARALLEL_SIZE

Emits --dp-size <N> and --enable-dp-attention when value > 1 (DP-attention: dp partitions the tp GPUs, world size stays tp — used with expert parallelism); omitted at 1.

NVFP4 Emulation#

By default, NVFP4 model profiles are only selectable on Blackwell GPUs (compute capability ≥ 10.0), which have native FP4 tensor cores. Set NIM_ALLOW_NVFP4_EMULATION=1 to also make NVFP4 profiles selectable on pre-Blackwell GPUs for GPUs like Ampere and Hopper (compute capability 8.0–9.x) with the vLLM or SGLang backend — these dequantize the FP4 weights to BF16 in-kernel using the Marlin weight-only kernel.

NIM Environment Variable

Description

Default

NIM_ALLOW_NVFP4_EMULATION

Allows NVFP4 profiles to be selected on pre-Blackwell GPUs (Ampere/Ada/Hopper, compute capability 8.0–9.x) with vLLM/SGLang, using Marlin weight-only emulation.

0 (off)

Note

  • Marlin emulation is weight-only (FP4 weights dequantized to BF16 in-kernel), so it may be slower than natively-supported precisions on the same GPU.

  • TensorRT-LLM profiles are unaffected and are supported on Blackwell only (no emulation path).

  • The relaxation is applied per host, not per GPU: it takes effect only when every detected GPU is in the compute-capability 8.0–9.x window. If any GPU falls outside that window, for example Volta/Turing (< 8.0) or Blackwell (≥ 10.0), the flag has no effect and NVFP4 profiles remain Blackwell-only for the entire host. This includes mixed nodes (for example, Blackwell and Ampere/Hopper), which are not relaxed.

  • On A100 (compute capability 8.0), dense (non-MoE) NVFP4 profiles run correctly; FP4 Mixture-of-Experts (MoE) kernels may not be available in all backend versions.

Runtime Config#

You can place a runtime_config.json file in the model workspace directory. NIM reads it automatically if it is present:

{
  "tensor_parallel_size": 2,
  "enable_prefix_caching": true,
  "max_model_len": 8192
}

Unknown keys are passed through to vLLM as-is.

Embedded Passthrough Arguments#

The file also accepts a reserved NIM_PASSTHROUGH_ARGS key that holds a full CLI-style string. The string is parsed with the same rules as the NIM_PASSTHROUGH_ARGS environment variable: known arguments receive their registered types, boolean flags accept the --flag and --no-flag forms, and denied arguments are dropped with a warning.

{
  "NIM_PASSTHROUGH_ARGS": "--max-model-len 8192 --enable-prefix-caching --no-enable-chunked-prefill"
}

The embedded string and the flat keys resolve at the same runtime-config priority. Two rules govern how the two in-file forms interact:

  • Explicit keys win. If the same file sets a parameter both inside the embedded string and as a flat key, the flat key takes precedence.

  • Strictness differs by form. When strict passthrough parsing is active (NIM can build the backend’s known-argument list), an unrecognized flag inside the embedded string is a fatal error, matching how the backend treats its own command line. If known-argument parsing is unavailable — the backend is not installed, or its parser cannot be constructed in the environment — unrecognized flags are instead retained leniently for passthrough. An unknown flat key is always retained and passed through to the backend.

To negate a boolean argument with a flat key, set "flag": false or use the equivalent "no-flag": true form. For arguments the backend registers with negative spellings (vLLM), both resolve to the --no-flag command-line form. SGLang has no --no-* spellings, so a false value for an argument outside the NIM parameter registry results in the flag being omitted.

Profile Tags#

Model profiles include metadata tags that configure parallelism. These values are extracted automatically when a profile is selected:

Profile Tag

vLLM Parameter

tp

tensor_parallel_size

pp

pipeline_parallel_size

For example, a profile named vllm-fp8-tp2-pp1 sets tensor_parallel_size=2 and pipeline_parallel_size=1 at profile priority.

Profile Engine Arguments (nim_json_args)#

A profile can ship engine arguments directly in its manifest through the reserved nim_json_args tag. The tag value is a string that contains a flat-key JSON object. Use canonical engine-argument keys (max_model_len), not the NIM_* environment-variable spellings that runtime_config.json also accepts — the manifest generator rejects those in this tag so the blob has one spelling:

tags:
  tp: '2'
  profile: throughput
  nim_json_args: '{"moe_backend": "flashinfer", "kv_cache_dtype": "fp8", "compilation_config": {"level": 3}}'
  nim_description: 'Turbo-certified H100 throughput profile'

Because the arguments live in the manifest, you can read them from the manifest itself (the NIM ships one at /opt/nim/etc/default/model_manifest.yaml) without downloading a profile’s workspace files. Note that list-model-profiles does not print them: it shows each profile’s synthesized name, and reserved nim_* tags are deliberately excluded from that name so a JSON blob cannot make it unreadable.

Key behavior:

  • Values apply at profile priority, so runtime_config.json, NIM_PASSTHROUGH_ARGS, environment variables, and CLI arguments all override them.

  • Known parameters are type-coerced; unknown keys pass through to the backend. JSON object and array values reach the backend as JSON strings.

  • Secret-bearing keys (for example, api_key and ssl_*) are ignored, matching the CLI and NIM_PASSTHROUGH_ARGS surfaces.

  • A malformed tag logs a warning and is skipped; the profile still serves with its other tags.

  • Dedicated structural tags (tp, pp) win over the same keys inside nim_json_args.

  • Speculative-decoding keys are handled separately — see below.

The nim_description tag holds a human-readable note about the profile. It is never parsed as configuration and is printed at startup and in --dry-run output.

Speculative decoding in nim_json_args#

A profile can carry its speculative-decoding configuration in the tag, in the same two shapes runtime_config.json supports: a nim_specdec envelope, or flat speculative_* keys.

tags:
  nim_json_args: '{"nim_specdec": {"speculative_config": "{\"method\": \"eagle3\", \"num_speculative_tokens\": 3}", "draft_uri": "ngc://org/team/eagle3-draft:1.0", "max_model_len": 8192}}'

These keys are not applied at profile priority like the rest of the tag. They are resolved through the same NIM_SPECDEC_ENABLE gate as the file, which means:

  • With NIM_SPECDEC_ENABLE unset, a profile whose tag carries a spec config serves with speculative decoding on by default.

  • NIM_SPECDEC_ENABLE=0 turns it off for that profile, exactly as it does for a file-shipped config.

  • draft_uri names the EAGLE3 draft model and is downloaded through the normal path; it never reaches the backend as an argument.

  • Keys inside the nim_specdec envelope — including spec-coupled ones such as max_model_len — apply only when speculative decoding resolves on. One caveat on a reused workspace: turning the gate off removes the speculative keys from the realized runtime_config.json, but a coupled base key that an earlier spec-on boot already merged into that file stays behind. A workspace materialized fresh (the normal container start) is unaffected.

Precedence runs lowest to highest: the tag’s flat keys, the tag’s envelope, the file’s flat keys, the file’s envelope, then NIM_SPECDEC_ARGS. So a workspace runtime_config.json remains a complete override for a tag-shipped spec config.

Migration note: profiles that ship a runtime_config.json continue to work unchanged. When both surfaces set the same key, runtime_config.json wins. One rendering change applies on upgrade: an unknown boolean key set to false in runtime_config.json was previously dropped silently and now reaches vLLM as --no-<flag>, so a stale or misspelled flag fails startup loudly instead of being ignored.

NIM Defaults#

The following default values are applied only if no other source sets the field:

Parameter

Default

tensor_parallel_size

1

pipeline_parallel_size

1

Override Warnings and Strict Mode#

NIM reports configuration overrides with warnings, and strict mode changes that behavior.

Override Detection#

When a higher-priority source overwrites a value from a lower-priority source, NIM logs a warning:

WARNING: Config override: 'tensor_parallel_size' changed from 2 (RUNTIME) to 8 (CLI)

Strict Mode#

Set NIM_STRICT_ARG_PROCESSING=true to treat override warnings as errors:

export NIM_STRICT_ARG_PROCESSING=true

In strict mode, the container exits with an error if any configuration override is detected between non-default sources. This setting is useful in the following cases:

  • CI/CD pipelines, where you want to catch configuration conflicts.

  • Production deployments, where configuration should be deterministic.

This behavior also applies when a direct container argument conflicts with an image-provided NIM_PASSTHROUGH_ARGS value. Add disjoint arguments in strict mode. If you intend to replace an image-provided value with a direct argument, leave strict mode disabled for that deployment.

Example:

export NIM_STRICT_ARG_PROCESSING=true
export NIM_TENSOR_PARALLEL_SIZE=2

docker run ... nim-serve --tensor-parallel-size 4

Result:

ERROR: Config override detected in strict mode: 'tensor_parallel_size' changed from 2 (ENV) to 4 (CLI).
Set NIM_STRICT_ARG_PROCESSING=false to allow intentional overrides.

Dry Run#

Use --dry-run to print the fully resolved configuration and the resulting backend arguments without starting the server:

docker run --gpus=all \
  -e NIM_MODEL_PATH=hf://meta-llama/Llama-3.1-8B-Instruct \
  -e NIM_TENSOR_PARALLEL_SIZE=2 \
  ${NIM_LLM_MODEL_FREE_IMAGE}:2.0.11 \
  --max-model-len 4096 \
  --dry-run

The output includes the provenance for each parameter and the final backend argument list. Image-provided passthrough values appear with PASSTHROUGH provenance, and arguments supplied directly to the container appear with CLI provenance.

Denied Arguments#

Certain vLLM CLI arguments are blocked in NIM containers because the nginx proxy or other system components manage them. If you pass a denied argument, NIM logs a warning and ignores it.

Denied Argument

Reason

NIM Alternative

--host

Networking is managed by nginx

—

--port

Port is managed by nginx

NIM_SERVER_PORT (external)

--ssl-keyfile

SSL/TLS is managed by nginx

NIM_SSL_MODE, NIM_SSL_KEY_PATH

--ssl-certfile

SSL/TLS is managed by nginx

NIM_SSL_MODE, NIM_SSL_CERTS_PATH

--ssl-ca-certs

SSL/TLS is managed by nginx

NIM_SSL_CA_CERTS_PATH

--ssl-cert-reqs

SSL/TLS is managed by nginx

NIM_SSL_MODE=MTLS

--root-path

API routing is managed by nginx

—

--api-key

Authentication should be managed externally

—

--uvicorn-log-level

Logging is managed by NIM

NIM_LOG_LEVEL

--allowed-origins

CORS should be configured in nginx

NIM_CORS_ALLOW_ORIGINS

--allowed-methods

CORS should be configured in nginx

NIM_CORS_ALLOW_METHODS

--allowed-headers

CORS should be configured in nginx

NIM_CORS_ALLOW_HEADERS

Validation and Error Handling#

NIM validates configuration at multiple stages and handles errors differently depending on where validation occurs.

Where Validation Occurs#

Stage

Validated Item

Error Behavior

CLI Parsing

Argument format, type conversion

Warning logged, invalid arg skipped

Env Parsing

Type conversion

Warning logged, invalid value skipped

Runtime Config

JSON syntax, type conversion

Warning logged, invalid key skipped

Config Merge

Override detection

Warning (or error in strict mode)

vLLM Startup

Argument validity, model compatibility

vLLM exits with error

Invalid Parameter Handling#

  • Invalid NIM parameters: Logged as warnings; the container continues with the remaining valid values.

  • Unknown parameters: Passed through to vLLM as-is. vLLM handles its own validation.

  • Invalid vLLM parameters: vLLM validates and reports errors at startup.

Example Error Messages#

The following examples show representative error messages:

# NIM-side type error (warning, non-fatal)
WARNING: Failed to parse CLI arg '--tensor-parallel-size' with value 'abc': invalid literal for int()

# vLLM-side validation error (fatal)
ValueError: tensor_parallel_size must be a positive integer

Provenance Tracking#

NIM records where each resolved configuration value came from. At startup, it logs a structured nim_config_resolved event:

{
  "event": "nim_config_resolved",
  "backend": "vllm",
  "backend_version": "<version>",
  "config": {
    "tensor_parallel_size": {
      "value": 4,
      "source": "CLI",
      "previous": {
        "value": 2,
        "source": "PASSTHROUGH"
      }
    },
    "enable_prefix_caching": {
      "value": true,
      "source": "RUNTIME"
    }
  }
}

When one source overrides another, previous identifies the value and source that were replaced. Run the container with --dry-run to inspect the full provenance report without starting the backend server.

GPU Memory Management#

NIM includes automatic GPU memory management to prevent out-of-memory (OOM) failures on constrained hardware. This process uses two mechanisms: automatic clamping of gpu_memory_utilization and a post-selection memory warning.

Automatic GPU Memory Clamping#

On certain hardware configurations, NIM automatically reduces the gpu_memory_utilization parameter, which controls what fraction of GPU memory vLLM can use, to prevent OOM errors:

Condition

Cap

Reason

vGPU guest (partitioned GPU)

0.75

GPU memory is shared or partitioned across virtual machines

UMA device (DGX Spark, GH200)

0.50

CPU and GPU share the same physical memory

Busy GPU (other processes using memory)

free_ratio - 0.05

Prevents contention with other workloads

The clamped value is applied at a low internal priority, so any explicit setting from environment variables, CLI arguments, or --gpu-memory-utilization takes precedence automatically. The minimum floor is 0.10 (NIM never sets GPU memory utilization below 10%).

Post-Selection Memory Warning#

After selecting a profile and resolving all configuration, NIM estimates the total VRAM required for the model (weights, KV cache, activations, and overhead) and compares it to available GPU memory. If the estimate exceeds available memory, NIM logs an advisory warning:

WARNING: Estimated VRAM (45.2 GB) exceeds available GPU memory (39.6 GB).
Consider reducing context length with --max-model-len=4096 (estimated 30.1 GB).

This warning is advisory only. NIM proceeds with startup regardless. The suggested --max-model-len value, if provided, indicates a context length that would fit within available memory.

To apply the suggestion, pass --max-model-len as a CLI argument:

docker run --gpus=all \
  -e NIM_MODEL_PATH=hf://meta-llama/Llama-3.1-8B-Instruct \
  -p 8000:8000 \
  ${NIM_LLM_MODEL_FREE_IMAGE}:2.0.11 \
  --max-model-len 4096

SSL/TLS Configuration#

NIM terminates TLS at the nginx proxy layer. Native vLLM SSL arguments such as --ssl-keyfile and --ssl-certfile are denied. Use the NIM SSL variables instead. Refer to Architecture for more information about the proxy layer.

Variable

Description

Default

NIM_SSL_MODE

SSL mode: DISABLED, TLS, or MTLS

DISABLED

NIM_SSL_KEY_PATH

Path to TLS private key

—

NIM_SSL_CERTS_PATH

Path to TLS certificate

—

NIM_SSL_CA_CERTS_PATH

Path to CA certificate (required for MTLS)

—

Use one of the following NIM_SSL_MODE values to enable SSL:

  • TLS mode: Server presents a certificate; client certificate is not required. To activate, set NIM_SSL_MODE=TLS. Requires NIM_SSL_CERTS_PATH and NIM_SSL_KEY_PATH.

  • Mutual TLS mode: Both server and client present certificates. To activate, set NIM_SSL_MODE=MTLS. Requires NIM_SSL_CERTS_PATH, NIM_SSL_KEY_PATH, and NIM_SSL_CA_CERTS_PATH.

The following example configures NIM to use TLS mode:

docker run --gpus=all \
  -e NIM_MODEL_PATH=hf://meta-llama/Llama-3.1-8B-Instruct \
  -e NIM_SSL_MODE=TLS \
  -e NIM_SSL_KEY_PATH=/certs/server.key \
  -e NIM_SSL_CERTS_PATH=/certs/server.crt \
  -v /path/to/certs:/certs:ro \
  -p 8000:8000 \
  ${NIM_LLM_MODEL_FREE_IMAGE}:2.0.11

CORS Configuration#

CORS is handled by the nginx proxy. The following variables control CORS headers:

Variable

Description

Default

NIM_CORS_ALLOW_ORIGINS

Allowed origins

*

NIM_CORS_ALLOW_METHODS

Allowed HTTP methods

GET, POST, PUT, DELETE, PATCH, OPTIONS

NIM_CORS_ALLOW_HEADERS

Allowed request headers

Content-Type, Authorization, X-Request-Id, X-Session-Id, X-Correlation-Id

NIM_CORS_EXPOSE_HEADERS

Headers exposed to the browser

X-Request-Id

NIM_CORS_MAX_AGE

Preflight cache duration (seconds)

3600

Note

vLLM’s --allowed-origins, --allowed-methods, and --allowed-headers arguments are denied in NIM because CORS is managed at the nginx layer. Use the NIM_CORS_* environment variables instead.

Examples#

The following examples show how configuration precedence, override warnings, and passthrough arguments work in practice.

CLI Overrides an Environment Variable#

The following example shows how a direct CLI argument overrides an environment variable:

export NIM_TENSOR_PARALLEL_SIZE=2

docker run --gpus=all \
  -e NIM_MODEL_PATH=hf://meta-llama/Llama-3.1-8B-Instruct \
  -e NIM_TENSOR_PARALLEL_SIZE \
  -p 8000:8000 \
  ${NIM_LLM_MODEL_FREE_IMAGE}:2.0.11 \
  nim-serve --tensor-parallel-size 4

This resolves tensor_parallel_size to 4, because the CLI value overrides the environment variable value.

NIM logs the following warning:

WARNING: Config override: 'tensor_parallel_size' changed from 2 (ENV) to 4 (CLI)

Full Priority Chain#

The following example shows how tensor_parallel_size resolves when every source sets a value:

runtime_config.json:        {"tensor_parallel_size": 1, "enable_prefix_caching": true}
Environment:                NIM_TENSOR_PARALLEL_SIZE=2
NIM_PASSTHROUGH_ARGS:       --tensor-parallel-size 3
CLI:                        nim-serve --tensor-parallel-size 4 --no-enable-prefix-caching

The resolved values are:

  • tensor_parallel_size = 4 (CLI overrides passthrough overrides env overrides runtime)

  • enable_prefix_caching = False (CLI --no- overrides runtime true)

Kubernetes with Container Arguments#

The following Kubernetes example passes vLLM arguments through the container args field. Each option and value is a separate list item. Do not add NIM_PASSTHROUGH_ARGS to env; leaving it unset in the Pod specification preserves any value provided by the image.

apiVersion: apps/v1
kind: Deployment
spec:
  template:
    spec:
      containers:
        - name: nim-llm
          image: <NIM_LLM_MODEL_FREE_IMAGE>:2.0.11
          args:
            - "--enable-prefix-caching"
            - "--max-num-batched-tokens"
            - "8192"
            - "--enable-chunked-prefill"
          env:
            - name: NGC_API_KEY
              valueFrom:
                secretKeyRef:
                  name: ngc-secret
                  key: api-key
            - name: NIM_MODEL_PATH
              value: "ngc://nim/meta/llama-3.1-8b-instruct"
            - name: NIM_CACHE_PATH
              value: "/opt/nim/.cache"
          ports:
            - containerPort: 8000
          livenessProbe:
            httpGet:
              path: /v1/health/live
              port: 8000
          readinessProbe:
            httpGet:
              path: /v1/health/ready
              port: 8000

Enabling TLS#

The following example enables TLS for a container deployment:

docker run --gpus=all \
  -e NIM_MODEL_PATH=hf://meta-llama/Llama-3.1-8B-Instruct \
  -e NIM_SSL_MODE=TLS \
  -e NIM_SSL_KEY_PATH=/certs/server.key \
  -e NIM_SSL_CERTS_PATH=/certs/server.crt \
  -v /path/to/certs:/certs:ro \
  -p 8000:8000 \
  ${NIM_LLM_MODEL_FREE_IMAGE}:2.0.11

After the container starts, you can validate the TLS configuration with a command such as the following:

curl --cacert /path/to/certs/ca.crt \
  https://localhost:8000/v1/health/ready