Skip to navigation
Troubleshooting

Troubleshooting Harmonizer NIM

View as Markdown

Use the following guidance to diagnose and resolve Harmonizer NIM deployment and startup issues.

Cache Reports Permission Denied

If startup reports “Permission denied” or “os error 13” below /opt/nim/.cache, verify that the mounted host directory is writable by the container process, which runs as UID 1000 by default:

chmod -R a+rwX "$LOCAL_NIM_CACHE"

NFS-backed home or scratch directories can still reject the container UID because of server-side identity mapping or ACLs. Move the cache to local disk, use a Docker-managed volume, or start the container with --user "$(id -u):$(id -g)". Refer to Environment Variables and Configuration.

Checkpoint Reported as Missing

An inaccessible checkpoint can be reported as “HARMONIZER_PKL_MODEL_PATH does not point to a file” even when the file exists on the host. Verify the default filename and grant read and directory-traversal access before mounting the checkpoint read-only:

test -f "$HARMONIZER_BYOC_PATH/diffusion_harmonizer.pkl"
chmod -R a+rX "$HARMONIZER_BYOC_PATH"

If the checkpoint has another name, set HARMONIZER_PKL_MODEL_PATH to its full path inside the container. Refer to Deploy a Custom Harmonizer Checkpoint.

Container Exits After a Hardware Error

Some hardware failures leave the accelerator context in a state that cannot be recovered safely inside the serving process. The NIM stops accepting work, reports not ready, and exits with status 70 so Docker, Kubernetes, or another supervisor can restart it. Configure an outer restart policy. If the failure repeats after restart, inspect the GPU and node health and move the workload off the affected node.

Readiness Takes Several Minutes

The service reports ready only after model loading, compilation, CUDA-graph capture, and warmup complete. A cold start normally takes about 2 to 9 minutes, depending on the GPU and host. Persist /opt/nim/.cache and wait for GET /v1/health/ready to return 200 before sending inference traffic. Refer to Startup Time and Batch Specializations for representative per-GPU times.

On higher-end GPUs, optional batch specializations account for part of startup compilation and graph capture. Reducing the executor set to the three required API shapes can shorten startup and reduce peak throughput at high concurrency:

HARMONIZER_PKL_OPTIMIZED_SPECIALIZATIONS=1x1,1x5,4x1

Do not remove 1x1, 1x5, or 4x1; the NIM requires them for image, temporal, and temporal-prefill serving.

Unlisted GPU Uses the Fallback Profile

When no exact PCI device profile matches, the NIM selects the generic fallback profile and builds the conservative 1x1,1x5,4x1 executor set. This is expected behavior. It keeps the API-serving shapes available without assuming more than the memory required by the smallest validated policy. An unlisted GPU is not performance-qualified; use a GPU from the Support Matrix for supported deployments.

HARMONIZER_PKL_OPTIMIZED_SPECIALIZATIONS overrides the detected policy on any GPU, including an unlisted GPU using the fallback profile. Keep the required 1x1,1x5,4x1 shapes and add higher-batch shapes only when the GPU has enough memory. Additional shapes can improve peak throughput at high concurrency, but increase startup time and GPU memory use.

Continue with these related guides: