Environment Variables for NVIDIA NeMo Retriever Embedding NIM#

Use this documentation to learn about the environment variables for NVIDIA NeMo Retriever Embedding NIM.

Environment variables set by a container image also override runtime fallbacks. See Batch Size for Nemotron 3.5 Embed.

Binary Environment Variables#

The following table contains the binary environment variables.

Name	Default	Description
`LOG_FORMAT`	Pretty	The log emit format. One of: pretty, json, compact.
`NIM_PERFORMANCE_MODE`	false	Selects latency-oriented (`0`) or throughput-oriented (`1`) defaults for unset NIM environment variables. When set to `1`, unset `NIM_SERVER_MAX_WAIT_MS`, `NIM_ENGINE_COUNT`, and `NIM_PIPELINE_MAX_BATCH_SIZE` default to `10`, `2`, and `64`, respectively. User-provided and image-provided environment variables override these defaults. The Nemotron 3.5 Embed image sets `NIM_PIPELINE_MAX_BATCH_SIZE=16`; setting only this mode does not change that batch-size limit.
`RUST_LOG`	error	The tracing EnvFilter.
`SHOW_CONFIG`	false	True to print env-var help and exit.

Server Environment Variables#

The following table contains the server environment variables.

Name	Default	Description
`NIM_SERVER_BIND_ADDR`	0.0.0.0:8000	The HTTP listen address (host:port).
`NIM_SERVER_GRPC_BIND_ADDR`	-	The optional KServe V2 gRPC listen address (host:port). Empty disables gRPC.
`NIM_SERVER_GRPC_MAX_DECODING_MESSAGE_BYTES`	-	The maximum inbound KServe gRPC message size in bytes. Empty uses the effective HTTP body limit.
`NIM_SERVER_HTTP_BODY_LIMIT_BYTES`	-	The HTTP request body limit for `/v1/embeddings` in bytes (passthrough).
`NIM_SERVER_MAX_IMAGE_BYTES`	5242880	The maximum allowed bytes for an embedded image payload in a single request, for models that support image inputs. The default is 5 MiB.
`NIM_SERVER_MAX_QUEUE_SIZE`	1024	The batcher request queue depth. Must be a positive integer. In version 2.2, setting this value to `0` can cause startup to panic.
`NIM_SERVER_MAX_WAIT_MS`	1	The maximum milliseconds to wait for additional requests before dispatching a batch. The `nemotron-embed` deployment sets `1`. When unset, the runtime uses `0` in latency mode and `10` in throughput mode.
`NIM_SERVER_REQUEST_TIMEOUT_S`	120	The request timeout in seconds.
`NIM_SERVER_SERVED_MODEL_NAME`	-	Comma-separated legacy aliases for the API model name, such as `company/search-embed`. When set, the aliases replace `NIM_ENGINE_MODEL_NAME` as the accepted API names: `/v1/models` lists the aliases, requests must use one of them in the `model` field, and responses return the name from the request. To keep accepting the checkpoint name too, include it in the list. Aliases don’t change which checkpoint the NIM loads, or its precision and dimension limits. For gRPC, aliases are normalized, for example `company/search-embed` becomes `company_search_embed`.
`NIM_SERVER_TLS_CERT_PATH`	-	The path to PEM certificate chain for HTTPS. When set with `NIM_SERVER_TLS_KEY_PATH`, enables TLS.
`NIM_SERVER_TLS_KEY_PATH`	-	The path to PEM private key for HTTPS. Must be set together with `NIM_SERVER_TLS_CERT_PATH`.

Pipeline Environment Variables#

The following table contains the pipeline environment variables.

The engine placement settings describe Nemotron 3.5 Embed in the release 2.4 nemotron-embed container. Earlier models and containers can use different defaults and engine behavior.

Name	Default	Description
`NIM_ENGINE_COUNT`	1	The number of GPUs requested for implicit placement, not the total number of engine replicas. Valid values: `1`–`16`. Defaults to `1` in latency mode and `2` in throughput mode unless explicitly configured. When `NIM_ENGINE_DEVICES` is unset, selects GPU ordinals starting at `0`, limited to a successfully detected positive GPU count.
`NIM_ENGINE_DEVICES`	-	Comma-separated, unique container-visible CUDA device ordinals, such as `0,1`. A nonblank list overrides `NIM_ENGINE_COUNT` for placement; COUNT must still be valid. Duplicate or malformed entries fail validation. Unavailable explicit ordinals fail startup instead of being silently removed.
`NIM_ENGINE_STREAM_COUNT`	4	Engine replicas per selected GPU. The `nemotron-embed` deployment sets `4`; when unset, the runtime uses `0`. Valid values: `0`–`16`. `0` selects the automatic model/GPU default; `1` creates one engine per selected GPU. A positive value overrides automatic selection. See Engine Placement and Replicas.
`NIM_PIPELINE_MAX_BATCH_SIZE`	16	The maximum number of sequences per forward pass. The `nemotron-embed` deployment sets `16`. The runtime fallbacks are `1` in latency mode and `64` in throughput mode when no batch-size setting is present in the server’s process environment. The image value takes precedence over these fallbacks.
`NIM_PIPELINE_MAX_BATCH_TOKENS`	40960	The optional post-tokenization token budget. The `nemotron-embed` deployment sets `40960`; when unset, the runtime uses `0`. Accepts integers from `0` through `2147483647`; `0` disables the additional token cap. For Nemotron 3.5 Embed text inputs, the container raises a positive budget below the configured maximum sequence length to that sequence length. See Text Dispatch Token Budget for the effective bound and NVFP4 Batch Token Budget for the separate CPU-tokenized path.
`NIM_PIPELINE_MAX_SEQ_LEN`	32768	The sequence length limit set by the `nemotron-embed` deployment. When unset in the embedding server’s process environment, the runtime fallback is `512`, not the model’s supported maximum.

The nemotron-embed deployment default changed from 10240 to 32768. For Nemotron 3.5 Embed, you can use a lower limit by adding -e NIM_PIPELINE_MAX_SEQ_LEN=16384 before the image name in your docker run command.

An override does not change the model’s supported maximum. See Sequence Length Limits for model and profile constraints.

Batch Size for Nemotron 3.5 Embed#

The Nemotron 3.5 Embed image sets NIM_PIPELINE_MAX_BATCH_SIZE=16. Omitting this variable from a Docker command or Kubernetes Pod specification preserves the image value in the server’s process environment. Setting only NIM_PERFORMANCE_MODE=1 keeps the maximum batch size at 16; it does not select 64.

The settings take precedence in this order:

A user-provided NIM_PIPELINE_MAX_BATCH_SIZE in Docker -e or Kubernetes env overrides the image value.
The image-provided NIM_PIPELINE_MAX_BATCH_SIZE overrides the performance-mode fallback.
If no batch-size setting is present in the process environment, the runtime uses 1 in latency mode or 64 in throughput mode.

To select throughput mode with a maximum batch size of 64, add both options before the image name in your existing docker run command:

-e NIM_PERFORMANCE_MODE=1 -e NIM_PIPELINE_MAX_BATCH_SIZE=64

For Kubernetes, add these entries to the NIM container’s env list:

env:
  - name: NIM_PERFORMANCE_MODE
    value: "1"
  - name: NIM_PIPELINE_MAX_BATCH_SIZE
    value: "64"

The batch-size limit is an upper bound, not a guaranteed dispatch size. GPU memory requirements depend on the batch size, sequence length, and selected model precision. Do not assume that batch size 64 fits at the maximum sequence length on every supported GPU. Check startup and representative requests for your configuration. If memory is insufficient, reduce the batch size or NIM_PIPELINE_MAX_SEQ_LEN within the supported sequence-length limits.

Text Dispatch Token Budget#

For Nemotron 3.5 Embed text inputs, the container uses the following limits:

B = NIM_PIPELINE_MAX_BATCH_SIZE
S = NIM_PIPELINE_MAX_SEQ_LEN
C = NIM_PIPELINE_MAX_BATCH_TOKENS

T = B * S                   when C = 0
T = min(max(C, S), B * S)    when C > 0

T is the effective text token capacity per GPU dispatch, not necessarily the configured value C. A positive C below S is raised to S so one maximum-length input row still fits. A value above B * S does not increase the capacity. Other CUDA shape and memory limits still apply.

Packed text dispatches must fit the sum of row token counts within T. Padded text dispatches must fit rows * padded sequence length within T, including padding. Both routes also enforce the maximum of B rows per dispatch. The container can split the input rows of a single HTTP request across multiple GPU dispatches. The request is not an indivisible token-budget unit. Splitting preserves the input rows and their output order.

For example, with B = 8 and S = 32768, setting C to 64 or 1 gives T = 32768. Eight input rows with 29 tokens each can therefore run together as one packed dispatch of 232 tokens. Neither value limits that dispatch to 64 or 1 tokens, nor forces one row per dispatch. To avoid this upward adjustment, configure a positive value for C that is at least equal to S.

Engine Placement and Replicas#

The nemotron-embed deployment sets NIM_ENGINE_STREAM_COUNT=4 regardless of GPU model. Omitting a user override keeps that deployment value; it does not select the runtime fallback of 0. Set NIM_ENGINE_STREAM_COUNT=0 explicitly to request automatic selection. For Nemotron 3.5 Embed, automatic selection uses four replicas only when startup identifies every selected GPU as GB200 or RTX PRO 6000 Blackwell. Otherwise, including when GPU identification fails, automatic selection uses one replica per selected GPU.

When startup successfully detects at least one visible GPU, placement follows these rules:

When NIM_ENGINE_DEVICES is unset or blank, the selected GPU count is min(NIM_ENGINE_COUNT, visible_GPU_count).
When NIM_ENGINE_DEVICES is nonblank, its unique entries select the GPUs, regardless of NIM_ENGINE_COUNT.
total_engines = selected_GPU_count * effective_stream_count, where effective_stream_count is the positive setting or the resolved automatic value.

If GPU-count detection fails or returns zero, implicit device selection is not limited. CUDA ordinals refer to GPUs visible inside the container, not host GPU indices. The startup engine_devices list contains one entry per engine, with replicas grouped by GPU. Repeated ordinals in this output represent replicas; do not repeat ordinals in NIM_ENGINE_DEVICES.

The table abbreviates NIM_ENGINE_COUNT, NIM_ENGINE_DEVICES, and NIM_ENGINE_STREAM_COUNT as COUNT, DEVICES, and STREAM_COUNT.

Visible GPUs	COUNT	DEVICES	STREAM_COUNT	`engine_devices`	Total engines
1	1	Unset	4	`[0, 0, 0, 0]`	4
1	2	Unset	4	`[0, 0, 0, 0]`	4
1	2	`0`	1	`[0]`	1
2	2	Unset	4	`[0, 0, 0, 0, 1, 1, 1, 1]`	8
4	1	`1,3`	2	`[1, 1, 3, 3]`	4

Each engine loads a model replica and runtime buffers and uses its own CUDA stream. More replicas require more GPU memory; these settings do not guarantee that the model fits.

Engine Environment Variables#

The following table contains the engine environment variables.

Name	Default	Description
`HF_TOKEN`	-	The Hugging Face token for model download. Passed through to the model downloader. For details, refer to Get Started With NVIDIA NeMo Retriever Embedding NIM.
`NGC_API_KEY`	-	The NGC API key for model download when `NIM_ENGINE_MODEL_DOWNLOAD_PROVIDER=ngc`. Passed through to the model downloader. For details, refer to Get Started With NVIDIA NeMo Retriever Embedding NIM.
`NIM_ENGINE_MODEL_DOWNLOAD_ONLY`	false	Set to `true` to download model weights and exit before CUDA initialization. Use this option to stage weights for air-gapped deployment or on hosts without visible GPUs. Requires `HF_TOKEN` for the Hugging Face provider, or `NGC_API_KEY` when `NIM_ENGINE_MODEL_DOWNLOAD_PROVIDER=ngc`.
`NIM_ENGINE_MODEL_DOWNLOAD_PROVIDER`	auto	The model download provider. One of: `auto`, `ngc`, `hf`, `huggingface`, or `hugging-face`. Use `hf` for Hugging Face; use `ngc` for the NVIDIA NGC Catalog.
`NIM_ENGINE_MODEL_NAME`	-	The checkpoint that the NIM loads. The `nemotron-embed` deployment leaves it unset, so the NIM selects the model based on the GPU. With no model or precision override, Nemotron 3.5 Embed selects `nvidia/Nemotron-3.5-Embed-1B-NVFP4` on GB200, B300, and RTX PRO 6000 Blackwell Server Edition. Other supported GPUs select `nvidia/Nemotron-3.5-Embed-1B-BF16`, executed in FP16. When `NIM_SERVER_SERVED_MODEL_NAME` is unset, the selected model name is also the API name. Check `/v1/models` before constructing requests.
`NIM_ENGINE_MODEL_PATH`	/model/embed	The in-container path for staged model artifacts. The directory must contain artifacts for a supported model. For details, refer to Custom Model Artifact Support in NVIDIA NeMo Retriever Embedding NIM.
`NIM_ENGINE_PRECISION`	-	The inference precision preference. Supported values depend on the model and GPU. For Nemotron 3.5 Embed, use `fp16` for the finalized BF16 checkpoint or `nvfp4` for the NVFP4 checkpoint. Explicit `bf16` execution is not implemented. Other models can support `fp16`, `fp8`, or `nvfp4`. When unset, the runtime selects the default supported precision.
`NIM_ENGINE_PRECOMPILE_ONLY`	-	Compile all CUDA artifacts then exit 0 (passthrough).