Skip to navigation

Configuring a NIM

View as Markdown

This page contains a reference for configuring the Alpamayo1.5 NIM container.

GPU Selection

Alpamayo1.5 NIM currently supports one GPU per container. You can pass --gpus all on a single-GPU host. On a multi-GPU host, expose one target GPU with either --gpus or CUDA_VISIBLE_DEVICES.

For example:

docker run --gpus '"device=0"' ...

or:

docker run -e CUDA_VISIBLE_DEVICES=0 --gpus all ...

Shared Memory

Set --shm-size=16g for direct Docker usage because PyTriton and TensorRT-LLM exchange multimodal request payloads through shared memory. Host IPC can also work when allowed by your deployment environment, but an explicit --shm-size=16g keeps the requirement visible in the launch command.

Model Cache

The container process runs as user ID 1000. The host directory mounted at /opt/nim/.cache (NIM_CACHE_PATH) must therefore be writable by UID 1000, otherwise startup fails while materializing model artifacts with Permission denied / os error 13.

NFS-backed home directories are frequently not writable by UID 1000 and can additionally fail with Stale file handle. Prefer a local-disk cache path for direct Docker runs.

Create a cache directory that UID 1000 can write to before launching:

mkdir -p /tmp/alpamayo-nim-cache
chmod 777 /tmp/alpamayo-nim-cache

Alternatively, run the container with --user "$(id -u):$(id -g)" so the process matches the owning UID of the mounted cache directory.

Environment Variables

The server accepts the following customer-facing variables.

VariableDefaultNotes
NGC_API_KEYunsetAPI key used to download assets from NGC.
NIM_CACHE_PATH/opt/nim/.cachePath inside the container where model assets and generated artifacts are cached. Mount this path to persistent storage.
NIM_PRECISIONautoOptional precision selector. Supported values are bf16, fp8, and w4a16. The runtime maps the requested precision to the matching manifest profile and then applies compute-capability and memory gates. Unlisted compatible GPUs default to K=1.
NIM_MODEL_PROFILEautoExact model profile name or hash override. Use only when you need to select a specific manifest profile.
NIM_HTTP_API_PORT8000HTTP API port inside the container.
NIM_FT_CHECKPOINTunsetBYOC Alpamayo checkpoint directory. Refer to Bring Your Own Checkpoint for the supported checkpoint layouts and calibration requirements.
NIM_GRPC_API_PORT50051Port for the gRPC trajectory API.
NIM_ALPAMAYO_TRAJ_SAMPLES1Comma-separated K values warmed and served by the container. Each configured value consumes startup time and retained CUDA graph/cache memory, so configure only the values your deployment needs. Unlisted GPUs may opt into values up to 32, but should start at K=1 and increase gradually; refer to Support Matrix for OOM guidance.
NIM_ALPAMAYO_SEED42Default seed used when a request does not provide a seed value. HTTP and gRPC requests can override this per request. For repeatable K=1 trajectory inference in the same deployment, use a fixed seed with top_k=1 and top_p=1. The seed controls diffusion sampling, and top-1 decoding removes VLM sampling variability.
NIM_ALPAMAYO_STARTUP_WARMUP1Run packaged sample requests before readiness. Set to 0 to skip warmup inference; the backend still loads before readiness.
NIM_ALPAMAYO_CALIBRATION_SAMPLE_LISTunsetComma-separated text manifest files. Each manifest contains one Alpamayo scene directory per line. Used when creating BYOC FP8/W4A16 ModelOpt VLM exports.
NIM_ALPAMAYO_ENABLE_TIMINGSunsetSet to 1 to log detailed request timing payloads and emit NVTX ranges. Refer to Observability for profiling guidance.
NIM_ALPAMAYO_TIMINGS_DIRunsetDirectory for timing artifacts used in offline analysis. The directory must be writable by the container user.

Startup warmup prepares the configured trajectory K values before readiness. The first client request can still populate request-path caches and be slower than steady state, so run one untimed request before collecting latency data.