Environment Variables for NVIDIA NeMo Retriever Reranking NIM

Use this documentation to learn about the environment variables for NVIDIA NeMo Retriever Reranking NIM.

Binary Environment Variables

The following table contains the binary environment variables.

Name	Default	Description
LOG_FORMAT	Pretty	The log emit format. One of: pretty, json, compact.
NIM_PERFORMANCE_MODE	false	Determines whether to optimize for latency or throughput. Selects latency-oriented (false or 0) or throughput-oriented (true or 1) defaults for unset NIM environment variables. In throughput mode, the default NIM_ENGINE_COUNT is 2, and the default NIM_PIPELINE_MAX_BATCH_SIZE is 64. NIM_PERFORMANCE_MODE doesn’t override values that you set explicitly.
RUST_LOG	error	The tracing EnvFilter.
SHOW_CONFIG	false	True to print env-var help and exit.

Server Environment Variables

The following table contains the server environment variables.

Name	Default	Description
NIM_SERVER_BIND_ADDR	0.0.0.0:8000	The HTTP listen address (host:port).
NIM_SERVER_GRPC_BIND_ADDR	-	The optional KServe V2 gRPC listen address (host:port). Empty disables gRPC.
NIM_SERVER_GRPC_MAX_DECODING_MESSAGE_BYTES	-	The maximum inbound KServe gRPC message size in bytes. Empty uses the effective HTTP body limit.
NIM_SERVER_MAX_QUEUE_SIZE	1024	The maximum queued rerank requests waiting for pipeline capacity. Requests beyond this depth are rejected immediately with HTTP 503 on the HTTP path and gRPC RESOURCE_EXHAUSTED on the KServe gRPC path.
NIM_SERVER_REQUEST_TIMEOUT_S	120	The timeout (s) for a single rerank request.
NIM_SERVER_SERVED_MODEL_NAME	-	Comma-separated legacy aliases for the API model name, such as company/search-rerank. When set, the aliases replace NIM_ENGINE_MODEL_NAME as the accepted API names: /v1/models lists the aliases, and requests must use one of them in the model field. To keep accepting the checkpoint name too, include it in the list. Aliases don’t change which checkpoint the NIM loads. For gRPC, aliases are normalized, for example company/search-rerank becomes company_search_rerank.
NIM_SERVER_TLS_CERT_PATH	-	The path to PEM cert chain for HTTPS; empty means no TLS.
NIM_SERVER_TLS_KEY_PATH	-	The path to PEM private key for HTTPS; empty means no TLS.

Pipeline Environment Variables

To limit the load on the NIM, use the following variables:

• NIM_SERVER_MAX_QUEUE_SIZE: limits the number of requests waiting in the queue.

• NIM_PIPELINE_MAX_BATCH_SIZE: limits the number of passages in each GPU batch.

• NIM_PIPELINE_MAX_BATCH_TOKENS: limits the number of tokens in each GPU batch. 0 turns off the limit.

The following table contains the pipeline environment variables.

Name	Default	Description
NIM_ENGINE_COUNT	1	The number of GPUs to use, starting at GPU 0. Ignored when NIM_ENGINE_DEVICES is set. When unset, NIM_PERFORMANCE_MODE=1 changes the default to 2. For details, refer to Engine Count.
NIM_ENGINE_DEVICES	-	A comma-separated list of GPU IDs, such as 0,1. Overrides NIM_ENGINE_COUNT. Every GPU in the list must exist, and the list can’t repeat a GPU ID. NIM_ENGINE_STREAM_COUNT sets the number of engines on each GPU.
NIM_ENGINE_STREAM_COUNT	0	The number of engines on each GPU. 0 runs one engine on each GPU.
NIM_PIPELINE_MAX_BATCH_SIZE	1	The max passages per engine forward pass. When unset, NIM_PERFORMANCE_MODE=1 changes the default to 64. An explicitly set value takes precedence over this default.
NIM_PIPELINE_MAX_SEQ_LEN	10240	The maximum length, in tokens, of each query and passage pair. The container sets 10240, which is lower than the model maximum of 32768. To allow longer pairs, set this variable to a value up to the model maximum. For more information, refer to Support Matrix for NVIDIA NeMo Retriever Reranking NIM.

Engine Environment Variables

The following table contains the engine environment variables.

Name	Default	Description
HF_TOKEN	-	The Hugging Face token for model download. Passed through to the model downloader. For details, refer to Get Started With NVIDIA NeMo Retriever Reranking NIM.
NGC_API_KEY	-	The NGC API key for model download when NIM_ENGINE_MODEL_DOWNLOAD_PROVIDER=ngc. Passed through to the model downloader. For details, refer to Get Started With NVIDIA NeMo Retriever Reranking NIM.
NIM_ENGINE_MODEL_DOWNLOAD_ONLY	false	Set to 1 to download model weights and exit before CUDA initialization. Use this option to stage weights for air-gapped deployment or on hosts without visible GPUs. Requires HF_TOKEN for the default provider, or NGC_API_KEY when NIM_ENGINE_MODEL_DOWNLOAD_PROVIDER=ngc.
NIM_ENGINE_MODEL_DOWNLOAD_PROVIDER	auto	The model download provider. Use auto, hf, huggingface, or hugging-face for Hugging Face; use ngc for the NVIDIA NGC Catalog.
NIM_ENGINE_MODEL_NAME	nvidia/Nemotron-3.5-Rerank-8B-BF16	The checkpoint that the NIM loads. When NIM_SERVER_SERVED_MODEL_NAME is unset, this is also the model name that API requests use and that /v1/models lists.
NIM_ENGINE_MODEL_PATH	/model	The in-container path for staged model artifacts. The directory must contain artifacts for a supported model. For details, refer to Custom Model Artifact Support in NVIDIA NeMo Retriever Reranking NIM.
NIM_ENGINE_PRECISION	fp16	The model precision. Use fp16, auto, or nvfp4. Supported values depend on the GPU. For more information, refer to Precision Override.
NIM_ENGINE_PRECOMPILE_ONLY	false	Compile all CUDA artifacts then exit without starting the server.

Related Topics

• Configure NVIDIA NeMo Retriever Reranking NIM