Is this page helpful?

Environment Variables#

This page documents all environment variables supported by NIM LLM. Set variables using -e flags when you run the container:

docker run -d --rm --gpus all \
  -p 8000:8000 \
  -v "$LOCAL_NIM_CACHE:/opt/nim/.cache" \
  -e NIM_MODEL_PATH=hf://meta-llama/Llama-3.1-8B-Instruct \
  -e NIM_SERVER_PORT=8000 \
  -e NIM_LOG_LEVEL=INFO \
  -e NGC_API_KEY \
  -e HF_TOKEN \
  <image>

Logging#

The following variables control log format and verbosity:

NIM_LOG_LEVEL: str | None#

Controls the verbosity of NIM log output. Accepts standard Python logging levels: DEBUG, INFO, WARNING, ERROR, CRITICAL.

Default:

None (uses application default)

Type:

string

Example:

NIM_LOG_LEVEL=DEBUG

NIM_JSONL_LOGGING: bool#

Enables structured JSON Lines (JSONL) log output.

Default:

False

Type:

boolean

Example:

NIM_JSONL_LOGGING=true

For usage details and examples, refer to Logging and Observability.

Model Configuration#

The following variables control model selection, model loading, and related runtime behavior:

NIM_MODEL_PROFILE: string = None#

Selects which model profile to use. Profiles define a validated combination of model variant, precision, and parallelism settings for a given GPU configuration. Run list-model-profiles inside the container to see available profiles and their IDs.

Default:

auto-selected based on detected GPU hardware

Example:

NIM_MODEL_PROFILE=07cd4f2bddd7a14ca84bab0a32602889fd0ae0eb76dc2eb0fc32594d065011a4

NIM_MODEL_PATH: str | None#

Model source URI or local filesystem path. Accepts hf://, ngc://, and modelscope:// prefixes for remote repositories, or a local directory path. When set, a runtime manifest is generated from this URI instead of using the baked-in container manifest.

Default:

None (uses baked-in manifest and NIM_MODEL_PROFILE)

Type:

string

Example:

NIM_MODEL_PATH=hf://meta-llama/Llama-3.1-8B-Instruct

NIM_SERVED_MODEL_NAME: str | None#

Overrides the served model name returned in API responses. When set, the /v1/models endpoint and response metadata use this name instead of the default model identifier.

Default:

None (uses the model’s own identifier)

Type:

string

Example:

NIM_SERVED_MODEL_NAME=my-llama

NIM_MAX_MODEL_LEN: int | None#

Overrides the maximum sequence length (context window) for the model. Values larger than the model’s trained maximum may cause errors.

Default:

None (uses model’s default from config)

Type:

positive integer

Example:

NIM_MAX_MODEL_LEN=4096

NIM_TENSOR_PARALLEL_SIZE: int | None#

Overrides the tensor parallelism degree. Splits model layers across the specified number of GPUs for inference.

Default:

None (auto-detected from profile)

Type:

positive integer

Example:

NIM_TENSOR_PARALLEL_SIZE=2

NIM_PIPELINE_PARALLEL_SIZE: int | None#

Overrides the pipeline parallelism degree. Distributes model stages across the specified number of GPUs for inference.

Default:

None (auto-detected from profile)

Type:

positive integer

Example:

NIM_PIPELINE_PARALLEL_SIZE=2

NIM_NUM_COMPUTE_NODES: integer = None#

Total number of compute nodes for multi-node inference. In multi-node deployments, set this on both the leader and worker nodes to the total node count (leader + workers).

Default:

None (single-node operation)

Example:

NIM_NUM_COMPUTE_NODES=2

NIM_REPOSITORY_OVERRIDE: string = None#

Redirects model downloads to an external repository while preserving the NIM manifest semantics. The container still uses the baked-in manifest for profile selection, but fetches model files from the overridden source.

Default:

None (downloads from the URI specified in the manifest)

Example:

NIM_REPOSITORY_OVERRIDE=s3://my-bucket/models

NIM_DISABLE_MODEL_DOWNLOAD: boolean = None#

Skips model download during container startup. Useful in multi-node deployments where worker nodes use a pre-staged shared filesystem and only the leader node needs to download.

Default:

False

Example:

NIM_DISABLE_MODEL_DOWNLOAD=true

NIM_TRUST_CUSTOM_CODE: bool#

Allows dynamic module loading for custom model code. Required for models that ship custom tokenizer or modeling files.

Default:

False

Type:

boolean

Example:

NIM_TRUST_CUSTOM_CODE=true

Server#

The following variables control server and health-check ports:

NIM_SERVER_PORT: int | None#

Port for the external-facing HTTP API server.

Default:

None (uses container default)

Type:

integer

Example:

NIM_SERVER_PORT=9000

NIM_HEALTH_PORT: int | None#

Port for the proxy health endpoints (/v1/health/live and /v1/health/ready).

Default:

None (defaults to NIM_SERVER_PORT)

Type:

integer

Example:

NIM_HEALTH_PORT=8001

LoRA and PEFT#

The following variables control LoRA and PEFT adapter discovery and refresh behavior:

NIM_PEFT_SOURCE: str | None#

URI for the LoRA adapter source (local path or NGC URI).

Default:

None (LoRA disabled)

Type:

string

Example:

NIM_PEFT_SOURCE=/adapters

NIM_PEFT_REFRESH_INTERVAL: int | None#

Polling interval in seconds for the dynamic LoRA watcher. When set, NIM periodically checks the PEFT source for new or removed adapters.

Default:

None (dynamic reloading disabled)

Type:

positive integer

Example:

NIM_PEFT_REFRESH_INTERVAL=30

NIM_PEFT_API_TIMEOUT_SECS: float | None#

Timeout in seconds for dynamic LoRA adapter API calls.

Default:

30.0

Type:

positive float

Example:

NIM_PEFT_API_TIMEOUT_SECS=60

Model Cache#

The following variable controls the model cache location inside the container:

NIM_CACHE_PATH: str#

Directory path for the model and artifact cache inside the NIM container.

Default:

/opt/nim/.cache

Type:

string

Example:

NIM_CACHE_PATH=/mnt/models/.cache

This one controls the timeout in seconds for the model cache discovery:

NIM_CACHE_PROBE_TIMEOUT: integer = 60#

Deadline in seconds for the initial artifact-cache reachability probe at startup. If NIM_CACHE_PATH is on an unreachable NFS/CIFS/FUSE mount, the container exits within this deadline instead of hanging until the OS TCP timeout.

Default:

60

Example:

NIM_CACHE_PROBE_TIMEOUT=120

Writable Paths#

By default the container writes under /opt/nim. These variables relocate those writes, which is what allows /opt/nim to be mounted read-only (Kubernetes securityContext.readOnlyRootFilesystem: true, or an equivalent immutable-root policy).

They are resolved by the container entrypoint before Python starts, so they are set as container environment variables rather than Python configuration.

NIM_WRITABLE_ROOT: string = /opt/nim#

Umbrella root for runtime writes: nginx state, scratch space (TMPDIR), the $HOME-derived GPU/library caches, and the generated middleware config. Setting this one variable relocates all of them.

It does not relocate NIM_CACHE_PATH or NIM_MANIFEST_PATH. Running with a read-only /opt/nim requires NIM_WRITABLE_ROOT and a writable NIM_CACHE_PATH; Model-Free deployments additionally need NIM_MANIFEST_PATH. Leave NIM_MANIFEST_PATH unset on a model-specific NIM – it is the read location of the baked manifest, and setting it points the loader at a file that does not exist.

Default:

/opt/nim

Example:

NIM_WRITABLE_ROOT=/mnt/rw

NIM_NGINX_DIR: string = ${NIM_WRITABLE_ROOT}/nginx#

Directory holding all nginx runtime state: the generated nginx.conf, nginx.pid, the five *_temp directories, the generated snippet configs, and the access/error logs. Overrides NIM_WRITABLE_ROOT for nginx alone – useful when nginx state belongs on a tmpfs while the caches live on a persistent volume.

Default:

${NIM_WRITABLE_ROOT}/nginx

Example:

NIM_NGINX_DIR=/run/nginx

NIM_MIDDLEWARE_CONFIG_PATH: string = ${NIM_WRITABLE_ROOT}/generated_configs/middleware_config.json#

Path of the generated middleware configuration, written at startup and read back to serve /v1/metadata. Overrides NIM_WRITABLE_ROOT for this file alone.

Default:

${NIM_WRITABLE_ROOT}/generated_configs/middleware_config.json

Example:

NIM_MIDDLEWARE_CONFIG_PATH=/mnt/rw/middleware.json

The nginx access and error logs are written under NIM_NGINX_DIR and can be redirected individually with NIM_NGINX_ACCESS_LOG (default ${NIM_NGINX_DIR}/access.log) and NIM_NGINX_ERROR_LOG (default ${NIM_NGINX_DIR}/error.log).

Payload capture, when enabled, also follows NIM_WRITABLE_ROOT: the path key of NIM_CAPTURE_ARGS defaults to ${NIM_WRITABLE_ROOT}/captures/requests.jsonl. See Payload capture.

Warning

readOnlyRootFilesystem: true freezes /etc/passwd, so the entrypoint cannot map an SCC-assigned arbitrary UID into it. Some libraries on the serve path resolve a cache directory through getpwuid() and fail at import with KeyError: 'getpwuid(): uid not found: <uid>'. Relocating the writable root does not help – the lookup happens before any path is used.

If your platform assigns arbitrary UIDs (OpenShift SCC), either run with a UID that resolves (runAsUser matching the image’s nim user), or leave readOnlyRootFilesystem unset and mount just /opt/nim read-only – the posture these variables are designed for, and the one in which the entrypoint’s /etc/passwd fixup still runs.

Note that a Kubernetes emptyDir cannot be used to make /etc/passwd writable: volumes without a file-typed source mount as directories, and mounting a directory over an existing file is rejected by the runtime, so the container fails to start.

Likewise, update-ca-certificates needs a writable /etc/ssl/certs. On a fully read-only rootfs the entrypoint skips the refresh and logs a warning, so mounted custom CA roots are not picked up; pre-bake them into the image or mount /etc/ssl/certs writable.

Important

The volumes backing $HOME and $TMPDIR must be mounted exec, not merely writable. Triton and vLLM compile kernels at runtime and dlopen() the resulting shared objects from $HOME/.triton and $HOME/.cache/vllm; TorchInductor builds and loads its objects out of $TMPDIR. On a noexec mount the mmap(PROT_EXEC) fails and the model never loads.

Both derive from NIM_WRITABLE_ROOT unless you override them, so in the normal case this is a requirement on that one volume.

Bind mounts and Kubernetes emptyDir volumes allow exec by default. Docker’s --tmpfs does not – pass --tmpfs /mnt/rw:rw,exec. If your platform applies noexec to mounted volumes as a hardening control, it must be relaxed for this volume.

/tmp, /run and /var/tmp may stay noexec: TMPDIR is relocated under NIM_WRITABLE_ROOT, so nothing executes from /tmp.

The entrypoint verifies this at startup and exits with a diagnostic naming the mount (set NIM_SKIP_EXEC_CHECK=1 to bypass). Without it the failure surfaces minutes later inside a worker as OSError: ... failed to map segment from shared object. Note that this error’s own suggested fix – setting TORCHINDUCTOR_CACHE_DIR elsewhere – does not help: the load happens from $TMPDIR, so that is the mount to correct.

Authentication#

The following variables provide credentials for authenticated model downloads:

NGC_API_KEY: string = None#

API key for authenticated model downloads from NGC (NVIDIA GPU Cloud). Only required when downloading production branch (PB) models from ngc:// repositories.

Default:

None

Example:

NGC_API_KEY=nvapi-...

NGC_CLI_API_KEY: string = None#

Backward-compatible NGC credential source. When both NGC_CLI_API_KEY and NGC_API_KEY are set, NGC_CLI_API_KEY takes precedence.

Default:

None

Example:

NGC_CLI_API_KEY=nvapi-...

HF_TOKEN: string = None#

Authentication token for Hugging Face Hub. Required for downloading private or gated models from hf:// repositories.

Default:

None

Example:

HF_TOKEN=hf_...

MODELSCOPE_API_TOKEN: string = None#

Authentication token for ModelScope. Required for authenticated downloads from modelscope:// repositories and to avoid rate limiting.

Default:

None

Example:

MODELSCOPE_API_TOKEN=...

SSL and TLS#

NIM uses TLS in two distinct directions. Inbound TLS secures client connections to the NIM inference API (nginx layer). Outbound TLS secures connections the container makes from itself when downloading model artifacts from NGC, Hugging Face, or a corporate registry such as JFrog Artifactory.

Important

The NIM_SSL_* variables below configure inbound TLS only. They do not affect outbound model downloads. To trust a corporate Certificate Authority (CA) for outbound connections, see Outbound TLS (Model Downloads).

Inbound TLS (NIM API)#

The following variables control TLS termination at the nginx proxy layer:

NIM_SSL_MODE: string = None#

Controls TLS termination at the nginx proxy.

  • DISABLED – plain HTTP (default)

  • TLS – server-side TLS; requires NIM_SSL_KEY_PATH and NIM_SSL_CERTS_PATH

  • MTLS – mutual TLS; additionally requires NIM_SSL_CA_CERTS_PATH

Default:

DISABLED

Example:

NIM_SSL_MODE=TLS

NIM_SSL_KEY_PATH: string = None#

Path to the SSL private key file. Required when NIM_SSL_MODE is TLS or MTLS.

Default:

None

Example:

NIM_SSL_KEY_PATH=/etc/ssl/private/server.key

NIM_SSL_CERTS_PATH: string = None#

Path to the SSL certificate file. Required when NIM_SSL_MODE is TLS or MTLS.

Default:

None

Example:

NIM_SSL_CERTS_PATH=/etc/ssl/certs/server.crt

NIM_SSL_CA_CERTS_PATH: string = None#

Path to the CA certificate file for client verification. Required when NIM_SSL_MODE is MTLS.

Default:

None

Example:

NIM_SSL_CA_CERTS_PATH=/etc/ssl/certs/ca.crt

Outbound TLS (Model Downloads)#

When the NIM container downloads models from a registry that uses a certificate signed by a private or corporate CA, you must provide that CA certificate to the container. This applies to two common scenarios:

  • Corporate registry with private CA — for example, a JFrog Artifactory instance whose TLS certificate is signed by your organization’s internal CA (no proxy involved).

  • TLS-inspecting proxy — a corporate proxy that decrypts and re-encrypts HTTPS traffic using a corporate CA.

In both cases, set SSL_CERT_FILE to a CA bundle that includes the corporate CA so that outbound TLS verification succeeds. A proxy (HTTPS_PROXY) is not required for SSL_CERT_FILE to take effect.

REQUESTS_CA_BUNDLE: string = None#

Same purpose as SSL_CERT_FILE but specific to the Python requests library. Some internal components (such as proxy validation in nimlib) use requests; setting this variable ensures those paths also trust the corporate CA. When in doubt, set both SSL_CERT_FILE and REQUESTS_CA_BUNDLE to the same combined bundle.

Default:

None

Example:

REQUESTS_CA_BUNDLE=/etc/ssl/certs/custom-ca-bundle.pem

SSL_CERT_FILE: string = None#

Path to a PEM-format CA certificate or bundle file inside the container. OpenSSL and the model download pipeline (nim_sdk, reqwest, and native-tls) use this file to verify server certificates during outbound HTTPS connections. Can be used with or without HTTPS_PROXY.

Warning

Setting SSL_CERT_FILE replaces the container’s default trust store. If you point it at a file containing only your corporate CA, connections to public endpoints (such as api.ngc.nvidia.com) will fail because the public CAs are no longer trusted. If you also need to reach public endpoints, use a combined bundle that includes both the default CAs and your corporate CA.

Default:

None (uses /etc/ssl/certs/ca-certificates.crt)

Example:

SSL_CERT_FILE=/etc/ssl/certs/custom-ca-bundle.pem

Create a combined CA bundle (one-time, on the host):

To add your corporate CA without losing trust in public CAs, concatenate the container’s default bundle with your corporate CA certificate:

# Extract the default CA bundle from the container
docker run --rm --entrypoint bash \
  ${NIM_LLM_MODEL_SPECIFIC_IMAGE}:2.0.10 \
  -c 'cat /etc/ssl/certs/ca-certificates.crt' > combined-ca-bundle.pem

# Append your corporate CA
cat /path/to/corporate-ca.pem >> combined-ca-bundle.pem

See also: Air-Gap Deployment: CA Certificate Injection.

CORS#

These variables configure Cross-Origin Resource Sharing (CORS) policy at the nginx proxy layer.

NIM_CORS_ALLOW_ORIGINS: string = None#

Comma-separated list of allowed request origins, or * for any origin.

Default:

*

Example:

NIM_CORS_ALLOW_ORIGINS=https://example.com

NIM_CORS_ALLOW_METHODS: string = None#

Allowed HTTP methods for CORS requests.

Default:

GET, POST, PUT, DELETE, PATCH, OPTIONS

Example:

NIM_CORS_ALLOW_METHODS=GET, POST, OPTIONS

NIM_CORS_ALLOW_HEADERS: string = None#

Allowed request headers for CORS requests.

Default:

Content-Type, Authorization, X-Request-Id, X-Session-Id, X-Correlation-Id

Example:

NIM_CORS_ALLOW_HEADERS=Content-Type, Authorization

NIM_CORS_EXPOSE_HEADERS: string = None#

Response headers that are exposed to the browser in CORS responses.

Default:

X-Request-Id

Example:

NIM_CORS_EXPOSE_HEADERS=X-Request-Id, X-Correlation-Id

NIM_CORS_MAX_AGE: string = None#

Duration in seconds that browsers may cache CORS preflight responses.

Default:

3600

Example:

NIM_CORS_MAX_AGE=7200

AWS SageMaker#

The following variable controls SageMaker BYOC (Bring Your Own Container) compatibility mode. When active, NIM listens on port 8080 and implements the GET /ping health check and POST /invocations inference endpoints required by SageMaker real-time inference.

NIM_SAGEMAKER_MODE: string = None#

Controls AWS SageMaker real-time inference compatibility mode.

  • 1 — Force SageMaker mode on. NIM listens on port 8080 and exposes GET /ping (health) and POST /invocations (inference, proxied to /v1/chat/completions).

  • 0 — Suppress SageMaker mode even when SageMaker environment variables are present. Use this to run NIM on a SageMaker instance without activating the protocol adapter.

  • (unset) — Auto-detect: SageMaker mode is enabled automatically if any of SAGEMAKER_MULTI_MODEL, SAGEMAKER_REGION, or SAGEMAKER_BIND_TO_PORT is present in the environment. These variables are injected by the SageMaker host agent and are not present in other environments.

Default:

(unset) — auto-detect from SageMaker environment signals

Example:

NIM_SAGEMAKER_MODE=1

Advanced#

The following variables control advanced argument handling and runtime behavior:

NIM_PASSTHROUGH_ARGS: str | None#

Passes additional vLLM CLI arguments as a single string. Useful in environments where direct CLI arguments are not available (e.g., container orchestrators). The same name is also accepted as a reserved key inside runtime_config.json, where the string is parsed with the same rules but resolves at runtime-config priority; explicit flat keys in the same file take precedence over it.

Default:

None

Type:

string

Example:

NIM_PASSTHROUGH_ARGS="--enable-prefix-caching --max-num-seqs 128"

NIM_STRICT_ARG_PROCESSING: bool#

Enables strict configuration processing. When true, conflicting configuration overrides (e.g., CLI overwriting an environment variable) raise errors instead of warnings.

Default:

False

Type:

boolean

Example:

NIM_STRICT_ARG_PROCESSING=true

NIM_DISABLE_CUDA_GRAPH: bool#

Disables CUDA graph optimization. May reduce GPU memory usage at the cost of inference throughput.

Default:

False

Type:

boolean

Example:

NIM_DISABLE_CUDA_GRAPH=true

Speculative decoding#

Speculative decoding (NGRAM / MTP / EAGLE3) is a runtime toggle, not a separate profile. The selected profile decides the default: a profile that ships a spec config in its runtime_config.json serves with speculative decoding, one that doesn’t serves without it. NIM_SPECDEC_ENABLE overrides that default globally (1 forces it on wherever a config exists – profiles without one warn and serve without it – and 0 forces it off everywhere; NIM_SPECDEC_ARGS can override the config). Either way the same profile serves both modes. See the Speculative Decoding guide for recipes, bring-your-own-draft instructions, and benchmarking.

Precedence (highest wins): explicit CLI --speculative-* and NIM_PASSTHROUGH_ARGS override NIM_SPECDEC_ARGS, which overrides the profile’s spec block. The EAGLE3 draft model downloads through the same path as the checkpoint (NGC / HF / S3 / GCS, air-gap and cache aware) from the draft_uri carried in the spec config, or from NIM_DRAFT_MODEL_PATH for the model-free path (NIM_DRAFT_MODEL_PATH takes precedence when both are set).

NIM_SPECDEC_ENABLE: bool | None#

Speculative-decoding override. SpecDec is a runtime toggle, not a profile dimension: the spec config lives in runtime_config.json (the single source of truth) and is gated at launch, so one profile serves both spec-on and spec-off without a dedicated profile. Tri-state:

  • unset (default): the selected profile decides. A profile whose runtime_config.json carries a spec config (a nim_specdec envelope or flat speculative_* keys) serves WITH speculative decoding; a profile without one serves without it.

  • 1: force spec on for profiles that carry (or receive, via NIM_SPECDEC_ARGS) a spec config. A profile with no spec config logs a warning and serves without spec – it never crashes.

  • 0: force spec off for every profile; any spec config is stripped from runtime_config.json before launch.

Default:

unset (the profile decides)

Type:

boolean (tri-state: unset / 1 / 0)

Example:

NIM_SPECDEC_ENABLE=0

NIM_SPECDEC_ARGS: str | None#

Model-level speculative-decoding config, a JSON object of runtime_config.json keys for the active backend. Consumed only when speculative decoding resolves ON (the profile’s default, or forced by NIM_SPECDEC_ENABLE=1); its keys are merged into the workspace runtime_config.json (overriding any profile-shipped spec block). On its own it does not activate spec for a profile that ships no spec config – pair it with NIM_SPECDEC_ENABLE=1 for the model-free path. vLLM uses speculative_config (a JSON string); SGLang uses speculative_algorithm plus the speculative_* flags. An EAGLE3 entry may carry a NIM-internal draft_uri (any NIM_MODEL_PATH scheme) that is materialized through the unified download path and never passed to the backend.

Default:

None

Type:

string (JSON object)

Example (vLLM):

NIM_SPECDEC_ARGS='{"speculative_config": "{...}", "draft_uri": "ngc://org/d:1"}'

Example (SGLang):

NIM_SPECDEC_ARGS='{"speculative_algorithm": "NGRAM"}'

NIM_DRAFT_MODEL_PATH: str | None#

Source URI or local path for a speculative draft model (e.g. an EAGLE3 head). Accepts the same schemes as NIM_MODEL_PATH (hf://, ngc://, s3://, gs://, modelscope://) or an absolute local directory. When set and the assembled speculative config references a draft by a bare name (eagle/eagle3/ draft_model/medusa/standalone), NIM downloads/materializes it into that subdir of the served workspace and rewrites the draft reference to the absolute path. The speculative config itself (method, num tokens, algorithm, …) is supplied separately via CLI args or NIM_PASSTHROUGH_ARGS; reference the draft by a bare subdir name (e.g. draft) there and NIM materializes it at <workspace>/<name>.

Default:

None (no draft download)

Type:

string

Example:

NIM_DRAFT_MODEL_PATH=hf://lmsys/SGLang-EAGLE3-Llama-3.1-8B-Instruct-SpecForge

Payload capture#

Opt-in capture of inference request payloads as AIPerf-replayable JSONL. Disabled by default; captures exact prompts (PII risk). See the Payload Capture guide for formats, examples, and replay commands.

NIM_CAPTURE_ENABLE: bool#

Opt-in payload capture. When enabled, a NIM-owned ASGI middleware writes each inference request payload as AIPerf-replayable JSONL (see nim_llm/features/payload_capture_middleware.py). DISABLED BY DEFAULT because it captures exact prompts, which may include PII/secrets/customer data – unlike NIM_SPECDEC_ENABLE, no profile ever turns it on. This is the toggle, NIM_CAPTURE_ARGS carries the (optional) tuning.

Default:

False

Type:

boolean

Example:

NIM_CAPTURE_ENABLE=1

NIM_CAPTURE_ARGS: str | None#

Capture configuration as a JSON object, consumed only when NIM_CAPTURE_ENABLE=1. All keys are optional; the middleware validates them and falls back to the default on a bad value (and disables capture if the JSON itself is invalid). Keys:

  • path (str, default $NIM_WRITABLE_ROOT/captures/requests.jsonl, i.e. /opt/nim/captures/requests.jsonl unless the writable root is relocated): destination JSONL, must be writable by the container user (mount a writable dir).

  • format (mooncake_payload default | raw_payload): both record the ACTUAL request payload (never hashed/synthetic prompts). mooncake_payload writes {"timestamp": <ms>, "payload": <request>}; raw_payload writes the request JSON object. Replay with --custom-dataset-type mooncake_trace / raw_payload.

  • max_request_bytes (int, default 1048576): larger requests are forwarded unchanged but not recorded (warned), since a truncated payload is not replayable.

  • sample_rate (float in [0,1], default 1.0): probability a matching request is written; 1.0 = all, 0.1 = ~10%, 0.0 = none. Trims I/O under high QPS.

  • endpoint_pattern (regex, default ^/v1/(chat/completions|completions)$): only matching request paths are captured.

  • max_queue_bytes (int, default 1073741824 = 1 GiB): in-memory ceiling for the off-loop writer queue (bytes of buffered request body). Captures are serialized and written on a background thread; on overload the newest record is dropped with a WARNING (the request path is never blocked). Set >= max_request_bytes.

Default:

None (built-in defaults apply)

Type:

string (JSON object)

Example:

NIM_CAPTURE_ARGS='{"format": "raw_payload", "path": "/captures/t.jsonl"}'