> This page is for version 1.0.0 (Latest) (default).
> For other versions, use one of these documentation indexes:
> - 1.0.0 (Latest) (default): https://docs.nvidia.com/nim/harmonizer/1.0.0/llms.txt

> For clean Markdown of any page, append .md to the page URL.
> For a complete documentation index, see https://docs.nvidia.com/nim/harmonizer/llms.txt.
> For AI client integration (Claude Code, Cursor, etc.), connect to the MCP server at https://docs.nvidia.com/nim/harmonizer/_mcp/server.

# Troubleshooting Harmonizer NIM

> Resolve Harmonizer NIM cache, checkpoint, readiness, GPU profile, and hardware failure symptoms.

Use the following guidance to diagnose and resolve Harmonizer NIM deployment and startup issues.

## Cache Reports Permission Denied

If startup reports “Permission denied” or “os error 13” below
`/opt/nim/.cache`, verify that the mounted host directory is writable by the
container process, which runs as UID `1000` by default:

```bash
chmod -R a+rwX "$LOCAL_NIM_CACHE"
```

NFS-backed home or scratch directories can still reject the container UID
because of server-side identity mapping or ACLs. Move the cache to local disk,
use a Docker-managed volume, or start the container with
`--user "$(id -u):$(id -g)"`. Refer to [Environment Variables and Configuration](/nim/harmonizer/configuration).

## Checkpoint Reported as Missing

An inaccessible checkpoint can be reported as
“HARMONIZER_PKL_MODEL_PATH does not point to a file” even when the file exists
on the host. Verify the default filename and grant read and directory-traversal
access before mounting the checkpoint read-only:

```bash
test -f "$HARMONIZER_BYOC_PATH/diffusion_harmonizer.pkl"
chmod -R a+rX "$HARMONIZER_BYOC_PATH"
```

If the checkpoint has another name, set `HARMONIZER_PKL_MODEL_PATH` to its
full path inside the container. Refer to [Deploy a Custom Harmonizer Checkpoint](/nim/harmonizer/bring-your-own-checkpoint).

## Container Exits After a Hardware Error

Some hardware failures leave the accelerator context in a state that cannot be
recovered safely inside the serving process. The NIM stops accepting work,
reports not ready, and exits with status `70` so Docker, Kubernetes, or
another supervisor can restart it. Configure an outer restart policy. If the
failure repeats after restart, inspect the GPU and node health and move the
workload off the affected node.

## Readiness Takes Several Minutes

The service reports ready only after model loading, compilation, CUDA-graph
capture, and warmup complete. A cold start normally takes about 2 to 9 minutes,
depending on the GPU and host. Persist `/opt/nim/.cache` and wait for
`GET /v1/health/ready` to return `200` before sending inference traffic.
Refer to [Startup Time and Batch Specializations](/nim/harmonizer/configuration#startup-time-and-batch-specializations) for representative per-GPU times.

On higher-end GPUs, optional batch specializations account for part of startup
compilation and graph capture. Reducing the executor set to the three required
API shapes can shorten startup and reduce peak throughput at high concurrency:

```bash
HARMONIZER_PKL_OPTIMIZED_SPECIALIZATIONS=1x1,1x5,4x1
```

Do not remove `1x1`, `1x5`, or `4x1`; the NIM requires them for image,
temporal, and temporal-prefill serving.

## Unlisted GPU Uses the Fallback Profile

When no exact PCI device profile matches, the NIM selects the generic fallback
profile and builds the conservative `1x1,1x5,4x1` executor set. This is
expected behavior. It keeps the API-serving shapes available without assuming
more than the memory required by the smallest validated policy. An unlisted GPU
is not performance-qualified; use a GPU from the [Support Matrix](/nim/harmonizer/support-matrix)
for supported deployments.

`HARMONIZER_PKL_OPTIMIZED_SPECIALIZATIONS` overrides the detected policy on
any GPU, including an unlisted GPU using the fallback profile. Keep the required
`1x1,1x5,4x1` shapes and add higher-batch shapes only when the GPU has enough
memory. Additional shapes can improve peak throughput at high concurrency, but
increase startup time and GPU memory use.

## Related Topics

Continue with these related guides:

* [Deploy with Helm](/nim/harmonizer/helm)
* Release Notes for Harmonizer NIM (the **Release Notes** tab)