Support Matrix

Alpamayo1.5 NIM currently supports one GPU per container.

Validated GPUs

The following GPUs have profile-specific validation and published K caps.

GPU	GPU memory	Validated precisions	K configuration
H100 80 GB (SXM or PCIe)	80 GB	bf16, fp8, w4a16	Runtime-selected; maximum K is 15
H100 NVL (94 GB)	94 GB	bf16, fp8, w4a16	Runtime-selected; maximum K is 15
H200	141 GB	bf16, fp8, w4a16	Runtime-selected; maximum K is 20
RTX 4090	24 GB	fp8, w4a16	Runtime-selected; maximum K is 1
RTX 3090	24 GB	w4a16	Runtime-selected; maximum K is 1
RTX 3090 Ti	24 GB	w4a16	Runtime-selected; maximum K is 1

A100 is not profile-specifically validated for this release. It follows the compatible-unvalidated path in the following section: begin with a generic BF16 or W4A16 profile at K=1 and confirm startup and inference on the exact deployment GPU.

Profiles

Profile	Precision	K values	Validated GPUs
bf16	BF16	Default K=1; explicit runtime K list	H100 80 GB, H100 NVL, H200
fp8	ModelOpt FP8 VLM	Default K=1; explicit runtime K list	H100 80 GB, H100 NVL, H200, RTX 4090
w4a16	ModelOpt W4A16/AWQ VLM	Default K=1; explicit runtime K list	H100 80 GB, H100 NVL, H200, RTX 4090, RTX 3090, RTX 3090 Ti

Compatible Unvalidated GPUs

The released precision profiles are generic rather than compiled for one GPU name. A GPU outside the validated table can start when NVML reports a compatible CUDA compute capability and both its total and currently free memory meet the profile minimum:

Profile	Minimum compute capability	Minimum GPU memory
bf16	8.0	30 GB
fp8	8.9	20 GB
w4a16	8.0	20 GB

This compatibility path includes GPUs such as RTX 6000 Ada when they satisfy the table, but it does not imply profile-specific performance validation. Exact GPU-family or SM requirements remain enforced for any internal profile that contains hardware-specific artifacts. Pre-Ampere GPUs with compute capability below 8.0 are not compatible with these profiles.

Profile Selection

The NIM automatically selects a compatible profile for the visible GPU. Use NIM_PRECISION to request a precision:

-e NIM_PRECISION=fp8

If a requested precision is not compatible with the visible GPU, startup fails with a profile-selection error. The manifest exposes generic precision profiles; startup checks map those profiles onto the detected GPU class, compute capability and memory. Auto selection prefers BF16 when at least 30 GB is available, then chooses a compatible quantized profile for smaller GPUs.

The active trajectory count defaults to K=1. Set NIM_ALPAMAYO_TRAJ_SAMPLES to the exact K values that this container should warm and serve:

-e NIM_ALPAMAYO_TRAJ_SAMPLES=1,10

Use a single value, such as NIM_ALPAMAYO_TRAJ_SAMPLES=10, for a fixed high-K deployment, or an explicit list such as 1,2,3 when clients may request multiple K values during serving. Each configured value is warmed during startup and retained for CUDA graph performance, so avoid configuring values that clients do not request. Values above a validated GPU’s published cap fail during profile selection before server startup. Unlisted GPUs default to K=1 and may explicitly request values up to the global experimental limit of 32; successful startup is not a guarantee that every larger K value fits that GPU.

Requests outside the selected K list return HTTP 400 with type="invalid_request_error". For example, num_traj_samples=2 on a container configured with NIM_ALPAMAYO_TRAJ_SAMPLES=1,10 returns an error that the profile supports only those values.

Unlisted GPU and Out-of-Memory Troubleshooting

K is the number of trajectory candidates returned by one request. Every value listed in NIM_ALPAMAYO_TRAJ_SAMPLES is warmed during startup and retains shape-specific CUDA graph/cache state. There is no portable fixed memory cost per K because it depends on the precision, GPU architecture, prompt shape, and configured K list.

For an unlisted GPU, begin with one quantized K=1 profile:

-e NIM_PRECISION=w4a16 \
-e NIM_ALPAMAYO_TRAJ_SAMPLES=1

Use fp8 instead of w4a16 when the GPU has compute capability 8.9 or newer. After the container reaches readiness and completes an untimed request, increase K one value at a time. A list such as 1,2,4 consumes more retained state than a single fixed value such as 4.

If startup or inference reports CUDA out-of-memory:

1. Stop sending requests and check /v1/health/ready.

2. Restart with NIM_ALPAMAYO_TRAJ_SAMPLES=1.

3. Select w4a16 or, on compute capability 8.9 or newer, fp8.

4. Remove unused K values and stop other processes consuming GPU memory.

5. Increase K gradually only after the smaller configuration is stable.

Latency Notes

For customer trajectory serving, prefer direct gRPC on port 50051 when the client can use the binary API. This avoids the HTTP base64 and JSON adapter path and reduces serving overhead. Send JPEG images when possible to use the fastest image decode path.

Multi-GPU deployment is not currently used for Alpamayo1.5 latency profiles. The recommended deployment shape is one NIM container per GPU.