Model Launcher Troubleshooting and Reference#

Overview#

Start with the message NVIDIA Sync shows.

Open Settings > Models to see the model status and available recovery actions. Fix the reported cause before you try the action again.

Keep the device powered on and connected during recovery.

For a Spark cluster, some checks and recovery actions need every Spark to be reachable.

Wait for the current task to stop before you start another one.

If Sync cannot confirm that a task stopped, use the recovery action it provides.

Setup Does Not Complete#

Read the failed check and fix the reported cause:

  • Docker access. Select Fix Now and follow the prompt.

  • Storage. Free the space Sync reports. See Remote Files and Storage for model download locations and the storage check’s limits.

  • Connection. Reconnect to the device if Sync cannot run a check.

Select Retry after you fix the issue. If setup was interrupted, reopen Settings > Models and select the same device or cluster. Sync shows the progress it can still read.

If Sync cannot tell whether setup finished, it tells you to quit the task. Select Quit Task and wait until cancellation is confirmed before you start setup again.

Quit Task Failed#

Select View Error and follow the recovery step shown for each device. Reboot or reconnect a Spark when Sync tells you to, then select Retry.

Restarting NVIDIA Sync does not reboot the Spark. Setup stays unavailable until cancellation is verified.

The Model Does Not Start#

Starting can take a few minutes. Wait while the model is starting.

If Start fails, read the message and fix the reported cause. You might need to free memory, free port 18765, disable MIG on a Station, or review the container logs. Select Retry after you fix the issue. If Sync says startup appears stalled, stop the model and start it again.

The Model Does Not Stop#

Reconnect to the device and select Stop again. For a Spark cluster, connect to the primary node. Wait for Sync to confirm that the model stopped before you uninstall it.

Model Status Cannot Be Checked#

Unknown means Sync could not confirm the model state.

Check that the device is powered on and connected, then wait for the status check to finish. Open Settings > Models if the system tray tells you to check the model there.

A disconnected device can keep its last shown model status.

That status is the last one Sync confirmed. It does not show whether the model has changed since then. Reconnect and wait for the status check to finish. Disconnecting Sync does not stop the model on the device.

The Endpoint Is Unavailable#

Endpoint Unavailable does not mean the model has stopped.

Sync needs a current, verified URL before it lets you copy the endpoint. A model can keep running while Sync has no usable endpoint to show.

  1. Connect to the device, or the primary node for a cluster.

  2. Wait for Sync to check the model’s status.

  3. Start the model if it is stopped, then wait for Running.

  4. Copy the endpoint again.

With Localhost, disconnecting closes the SSH tunnel and the old URL stops working. With LAN, a client using the device’s network address can still reach the running model. It needs a working network path to the device. For Tailscale connections, the URL Sync shows uses its SSH tunnel.

Stop the model before changing Localhost or LAN access.

You can change the mode while Sync is disconnected. The devices must still be reachable, and Sync must confirm that the model is stopped. If the controls are disabled, wait for the status check and read the hint beside the control. Reconnect if Sync cannot confirm the state. The change takes effect the next time you start the model.

See Configuring Network Access.

The API Key Cannot Be Read or Changed#

Try the action again. If the key remains visible but cannot be copied, copy it manually. To replace a key, stop the model, select Regenerate, and update each client with the new key.

Uninstall Fails#

Check that each device in the model setup is powered on and reachable, then select Try again or Retry Uninstall. The model remains configured until Sync confirms its removal.

If no retry action is available, restart NVIDIA Sync, reopen Settings > Models, and try uninstall again.

Model Status#

The Models tab shows the model’s current state:

  • Not Running: The model is set up and stopped.

  • Starting: The container and model are starting.

  • Running: The model is ready to serve requests.

  • Stopping: Sync is stopping the model.

  • Unknown: Sync cannot confirm the model’s state.

Start or stop failures show a message with details about what went wrong. If Sync cannot check the status, make sure the device is reachable and see Model Status Cannot Be Checked.

Remote Files and Storage#

Model Launcher downloads the model under your user account on the remote device.

The default directory is ~/.config/NVIDIA/Sync/model-launcher/, where ~ is that user’s home directory. If the remote environment sets XDG_CONFIG_HOME to an absolute path, the directory is $XDG_CONFIG_HOME/NVIDIA/Sync/model-launcher/ instead.

The following paths are relative to that Model Launcher directory:

File or Directory

Purpose

cache/hub/

Downloaded model files in Hugging Face cache format.

cache/images/

Container image archives used to transfer images to cluster nodes.

cache/tools/

Cached OpenCode and skills downloads.

cache/quarantine/

Temporary downloads awaiting verification.

deployments/<deployment-id>/

Model setup records, launch settings, and serving scripts.

deployments/<deployment-id>/runtime/<rank>/

Files written by the serving container while the model runs.

deployments/<deployment-id>/opencode/

The OpenCode binary and model connection settings managed by Sync.

operations/<operation-id>/

Task records, progress, and stdout.log and stderr.log files.

bin/nvsync-job

The remote program that runs Model Launcher tasks.

Within cache/hub/, each model has a models--<owner>--<model>/ directory. Model files are stored in blobs/, with links to them under snapshots/<revision>/.

Sync can reuse matching files from ~/.cache/huggingface/hub/ on the remote device.

It copies and verifies the files for the recipe’s model revision into its own cache and leaves the originals in place. This saves downloads but still needs space for the copy. Model Launcher uses its own cache even if you set HF_HOME or HF_HUB_CACHE for other tools.

For a Spark cluster, the primary node downloads missing model files and sends them to the other serving nodes. Each serving node keeps its own model copy under its remote user’s Sync directory. OpenCode is installed on the primary node.

Installed container images are kept in Docker’s image storage.

The cache/images/ directory holds transfer archives, not the installed images. Successful transfers remove their archives; interrupted transfers can leave archives behind.

Sync’s storage check measures free space on the root filesystem, /. If your home directory, Sync directory, or Docker storage is on a separate filesystem, check that filesystem’s free space too.

For custom app scripts and logs, see Remote Files and Working Directory. For the separate OpenCode workspace, see Using OpenCode.

Canceling Setup#

Select Quit Task if you need to stop setup. Wait for Sync to confirm it has stopped before trying again. If it cannot confirm that, see Quit Task Failed.

Uninstalling the Model#

Stop frees memory; Uninstall removes the model files Sync set up.

Stop the model, open Settings > Models, and select Uninstall from the model actions menu. Read the removal preview before you confirm.

Uninstall leaves your OpenCode workspace and sessions in place. Uninstall the model before deleting its device or cluster from Sync; Delete does not remove the model files. If removal fails, see Uninstall Fails.

Uninstall can retain shared or modified downloads and container images, as well as transfer archives left by interrupted tasks. Check the removal preview to see what Sync will remove and retain. Your original files in ~/.cache/huggingface/hub/ are left in place.