Is this page helpful?

Environment Variables for NVIDIA NeMo Retriever Embedding NIM#

Use this documentation to learn about the environment variables for NVIDIA NeMo Retriever Embedding NIM.

Binary Environment Variables#

The following table contains the binary environment variables.

Name	Default	Description
`LOG_FORMAT`	Pretty	The log emit format. One of: pretty, json, compact.
`NIM_PERFORMANCE_MODE`	false	Selects latency-oriented (`0`) or throughput-oriented (`1`) defaults for unset NIM environment variables. When set to `1`, unset `NIM_SERVER_MAX_WAIT_MS`, `NIM_ENGINE_COUNT`, and `NIM_PIPELINE_MAX_BATCH_SIZE` default to `10`, `2`, and `64`, respectively. Explicit environment variables override these defaults.
`RUST_LOG`	error	The tracing EnvFilter.
`SHOW_CONFIG`	false	True to print env-var help and exit.

Server Environment Variables#

The following table contains the server environment variables.

Name	Default	Description
`NIM_SERVER_BIND_ADDR`	0.0.0.0:8000	The HTTP listen address (host:port).
`NIM_SERVER_GRPC_BIND_ADDR`	-	The optional KServe V2 gRPC listen address (host:port). Empty disables gRPC.
`NIM_SERVER_GRPC_MAX_DECODING_MESSAGE_BYTES`	-	The maximum inbound KServe gRPC message size in bytes. Empty uses the effective HTTP body limit.
`NIM_SERVER_HTTP_BODY_LIMIT_BYTES`	-	The HTTP request body limit for `/v1/embeddings` in bytes (passthrough).
`NIM_SERVER_MAX_IMAGE_BYTES`	-	The maximum allowed bytes for an embedded image payload in a single request, for models that support image inputs.
`NIM_SERVER_MAX_QUEUE_SIZE`	1024	The batcher request queue depth. Must be a positive integer. In version 2.2.0, setting this value to `0` can cause startup to panic.
`NIM_SERVER_MAX_WAIT_MS`	0	The maximum milliseconds to wait for additional requests before dispatching a batch. Defaults to `0` in latency mode and `10` in throughput mode unless explicitly configured.
`NIM_SERVER_REQUEST_TIMEOUT_S`	120	The request timeout in seconds.
`NIM_SERVER_SERVED_MODEL_NAME`	-	Comma-separated legacy served model aliases for `/v1/models` and request model matching. If both `NIM_ENGINE_MODEL_NAME` and `NIM_SERVER_SERVED_MODEL_NAME` are configured, `NIM_SERVER_SERVED_MODEL_NAME` currently takes precedence.
`NIM_SERVER_TLS_CERT_PATH`	-	The path to PEM certificate chain for HTTPS. When set with `NIM_SERVER_TLS_KEY_PATH`, enables TLS.
`NIM_SERVER_TLS_KEY_PATH`	-	The path to PEM private key for HTTPS. Must be set together with `NIM_SERVER_TLS_CERT_PATH`.

Pipeline Environment Variables#

The following table contains the pipeline environment variables.

Name	Default	Description
`NIM_ENGINE_COUNT`	1	The number of CudaEngine instances. Defaults to `1` in latency mode and `2` in throughput mode unless explicitly configured.
`NIM_ENGINE_DEVICES`	-	The comma-separated CUDA device ordinals for explicit engine placement (e.g. `NIM_ENGINE_DEVICES`=0,1,2).
`NIM_PIPELINE_MAX_BATCH_SIZE`	1	The number of sequences per forward pass. Defaults to `1` in latency mode and `64` in throughput mode unless explicitly configured.
`NIM_PIPELINE_MAX_BATCH_TOKENS`	0	The optional post-tokenization token budget per embed GPU dispatch. `0` disables token-budget chunking. The runtime splits CPU-tokenized batches on request boundaries when possible; if a single request exceeds the budget, it can still run alone. Values must fit the signed 32-bit CUDA shape ABI.
`NIM_PIPELINE_MAX_SEQ_LEN`	-	The maximum sequence length override. When unset, the model profile default is used.

Engine Environment Variables#

The following table contains the engine environment variables.

Name	Default	Description
`HF_TOKEN`	-	The Hugging Face token for model download. Passed through to the model downloader. For details, refer to Get Started With NVIDIA NeMo Retriever Embedding NIM.
`NGC_API_KEY`	-	The NGC API key for model download when `NIM_ENGINE_MODEL_DOWNLOAD_PROVIDER=ngc`. Passed through to the model downloader. For details, refer to Get Started With NVIDIA NeMo Retriever Embedding NIM.
`NIM_ENGINE_MODEL_DOWNLOAD_ONLY`	false	Set to `true` to download model weights and exit before CUDA initialization. Use this option to stage weights for air-gapped deployment or on hosts without visible GPUs. Requires `HF_TOKEN` for the Hugging Face provider, or `NGC_API_KEY` when `NIM_ENGINE_MODEL_DOWNLOAD_PROVIDER=ngc`.
`NIM_ENGINE_MODEL_DOWNLOAD_PROVIDER`	auto	The model download provider. One of: `auto`, `ngc`, `hf`, `huggingface`, or `hugging-face`. Use `hf` for Hugging Face; use `ngc` for the NVIDIA NGC Catalog.
`NIM_ENGINE_MODEL_NAME`	-	The model name returned in embedding responses. The default model depends on the container and can be one of: `nvidia/nemotron-3-embed-1b`, `nvidia/llama-nemotron-embed-vl-1b-v2`.
`NIM_ENGINE_MODEL_PATH`	/model/embed	The in-container path for staged model artifacts. The directory must contain artifacts for a supported model. For details, refer to Custom Model Artifact Support in NVIDIA NeMo Retriever Embedding NIM.
`NIM_ENGINE_PRECISION`	-	The weight precision. One of: `fp16`, `fp8`, or `nvfp4`. Supported values depend on the model and GPU. When unset, the runtime selects the default supported precision.
`NIM_ENGINE_PRECOMPILE_ONLY`	-	Compile all CUDA artifacts then exit 0 (passthrough).