Is this page helpful?

Troubleshooting Container Startup Problems#

Use this documentation to troubleshoot startup failures related to the container runtime and writable filesystems. These solutions apply to standard and Dynamo deployments.

The Cache Directory Is Missing#

The entrypoint exits when the configured cache directory does not exist:

ERROR: Cache directory /opt/nim/.cache does not exist.

Mount a host directory or persistent volume at NIM_CACHE_PATH. For Docker setup and cache preparation, refer to Prerequisites and Configuration.

The Cache Directory Is Not Writable#

The container user needs read, write, and search access to the cache mount:

ERROR: Cache directory /opt/nim/.cache is not accessible by the container user

Grant group access to the host directory:

sudo chgrp -R 0 "$LOCAL_NIM_CACHE"
sudo chmod -R g+rwX "$LOCAL_NIM_CACHE"

The GPU Is Not Visible#

This Docker error indicates that the NVIDIA Container Toolkit is missing or not configured:

could not select device driver "" with capabilities: [[gpu]]

Complete the Prerequisites setup, restart Docker, and run the NVIDIA runtime verification command from the prerequisites page.

A Runtime Mount Uses the Noexec Option#

NIM compiles and loads code from TMPDIR and HOME. Startup fails when either directory is on a filesystem mounted with noexec:

ERROR: TMPDIR=/tmp is on a filesystem mounted 'noexec'

Use an executable bind mount or configure a tmpfs with rw,exec. Changing TORCHINDUCTOR_CACHE_DIR does not correct this failure because the load occurs from TMPDIR. Refer to Environment Variables for writable-root and temporary-directory behavior.

In Dynamo mode, the entrypoint selects /tmp when that directory is writable, even when NIM_WRITABLE_ROOT points elsewhere. Docker makes --tmpfs /tmp writable but noexec by default, so a hardened command such as --read-only --tmpfs /tmp can fail the executable-mount check in Dynamo mode while standard mode uses the writable-root temporary directory. Mount it with --tmpfs /tmp:rw,exec, or set TMPDIR explicitly to another writable, executable, pod-local mount.