Environment Variables and Configuration for Harmonizer NIM

This page contains a reference for configuring the Harmonizer NIM container.

GPU Selection

Harmonizer NIM uses one GPU per container. Passing --gpus all is acceptable on a single-GPU host. On a multi-GPU host, expose one target GPU with either --gpus or CUDA_VISIBLE_DEVICES.

For example:

docker run --gpus '"device=0"' ...

or:

docker run -e CUDA_VISIBLE_DEVICES=0 --gpus all ...

Inference Protocol

The container can serve either HTTP or gRPC. NIM_INFERENCE_PROTOCOL defaults to http. Set NIM_INFERENCE_PROTOCOL=grpc to start the gRPC entrypoint. The HTTP health and readiness routes remain available so deployment readiness checks work in either mode. Protocol selection does not restrict processing mode: both non-temporal and temporal requests are available over the selected transport.

Shared Memory

The Docker examples set --shm-size=8g as the recommended launch configuration. The codec workers and GPU coalescing queue use in-process threads and do not exchange request tensors through multiprocessing shared memory.

Model Cache

The container process runs as user ID 1000. The host directory mounted at /opt/nim/.cache (NIM_CACHE_PATH) must therefore be writable by UID 1000, otherwise startup fails while materializing model artifacts with “Permission denied” or “os error 13”.

Create a cache directory that UID 1000 can write to before launching:

mkdir -p /tmp/harmonizer-nim-cache
chmod 777 /tmp/harmonizer-nim-cache

Alternatively, run the container with --user "$(id -u):$(id -g)" so the process matches the owning UID of the mounted cache directory.

start_server.sh places the Hugging Face and Torch asset caches, TorchInductor cache, and Triton cache below NIM_CACHE_PATH. Generated kernels are stored under torchinductor and triton. Persisting this bind mount lets later starts reuse model assets, generated kernels, and autotuning results. Model loading, Dynamo tracing and guard construction, CUDA-graph capture, and warmup still run on each process start. The image does not contain a compilation cache generated at build time.

With version 1.3.0 or later of the nim-wfm Helm chart, set model.nimCache instead of setting NIM_CACHE_PATH in env. The chart sets NIM_CACHE_PATH to this value and mounts its model-store volume at the same path. When persistence.enabled is true, that volume is a persistent volume claim (PVC); with persistence, host-path storage, and NFS disabled, it is an ephemeral emptyDir. The Harmonizer Helm example uses /model-store, overriding the container default of /opt/nim/.cache. Refer to Deploy Harmonizer NIM with Helm for a complete values file.

A cache can be reused across supported GPU SKUs. TorchInductor and Triton cache lookups reuse compatible artifacts rather than blindly selecting an artifact from another SKU. The first start on a new SKU can still compile additional artifacts, take longer, and consume more space in the shared cache. Separating claims by SKU can provide more predictable warm-start time and capacity, but it is not required for cache correctness.

General Environment Variables

The following table describes the runtime configuration variables:

Performance Tuning

The optimized serving runtime ships fully enabled by default: fixed-shape optimized executors, compiled variational autoencoder (VAE) and diffusion transformer (DiT) paths, temporal current-slice decode, the request coalescing queue, and the fast PNG codec. These defaults are tuned per GPU SKU and are covered by the release quality gate, so most deployments do not need to change them.

A large set of advanced HARMONIZER_PKL_* flags expose this tuning (executor specialization matrix, compile and CUDA-graph options, memory formats, cuDNN autotune, and coalescing-queue depth). They are intended for benchmarking and SKU bring-up rather than production configuration. Leave them at their defaults unless you are reproducing a specific benchmark configuration.

Startup Time and Batch Specializations

The NIM reports ready only after model loading, optimization, and warmup finish. Representative cold starts with the default GPU-specific configuration are shown below. These times begin when the inference server starts and end when the service becomes ready; model download and container scheduling are not included.

Startup time varies with host performance, storage, compiler caches, and the configured batch specializations. Persist /opt/nim/.cache to make subsequent starts faster.

HARMONIZER_PKL_OPTIMIZED_SPECIALIZATIONS controls the fixed batch and frame shapes prepared during startup. Reducing the optional batch-size specializations shortens cold startup and reduces GPU memory used by optimized states. The tradeoff is lower peak throughput, primarily on faster GPUs under high concurrency. The impact on low-concurrency serving and slower GPUs is typically small.

The minimum supported value is:

HARMONIZER_PKL_OPTIMIZED_SPECIALIZATIONS=1x1,1x5,4x1

These three specializations are required by the API serving modes. In particular, 4x1 is required for the four-frame prefill used by temporal requests with context_length=4. Omitting 1x1, 1x5, or 4x1 causes the NIM to fail during startup. All other batch-size specializations are optional throughput optimizations.

Related Topics

Continue with these related guides:

• Quickstart Guide
• Deploy with Helm