Model fine-tuning#

When VISTA-3D NIM is running, by default, it automatically downloads the VISTA-3D model bundle and weights from NGC into the NIM container. However, if you wish to fine-tune VISTA-3D and continue using it within NIM, here is detailed guide to utilize a custom model bundle and weights in VISTA-3D NIM.

Fine-tune the Model#

Please follow this tutorial on Project MONAI to learn about how to fine tune VISTA-3D model.

Use the fine-tuned model in NIM#

Once you have the fine-tuned VISTA-3D bundle at your local storage, you simply need to mount it to /opt/bundle and set the correct model manifest profile for it by setting NIM_MANIFEST_PROFILE environment variable.

-e NIM_MANIFEST_PROFILE=43d0baebee73b7bdc2f9c7edf5c726b2323b4f7283eab92fab893d451592656d \
-v /path/to/fine-tuned/bundle:/opt/bundle \

Here is the full command to run the NIM with fine-tuned model.

docker run --rm -it --name vista3d \
   --runtime=nvidia -e CUDA_VISIBLE_DEVICES=0 \
   --shm-size=8G \
   -p 8000:8000 \
   -e NGC_API_KEY=$NGC_API_KEY \
   -e NIM_MANIFEST_PROFILE=43d0baebee73b7bdc2f9c7edf5c726b2323b4f7283eab92fab893d451592656d \
   -v /path/to/local/bundle:/opt/bundle \
   nvcr.io/nim/nvidia/vista3d:1.0.0

Here is the list of available profiles:

  • 56df3d486bbd1c8ac4a1de4d613d4037927bd462d751bf4fe9cbeb8d14890cb8: (default profile) downloads and use VISTA-3D bundle on NGC.

  • 43d0baebee73b7bdc2f9c7edf5c726b2323b4f7283eab92fab893d451592656d: utilizes the user provided VISTA-3D bundle (fine-tuned model) at /opt/bundle.

Troubleshooting#

If you encounter issues while using the fine-tuned model, here are some common troubleshooting steps:

Common Issues#

  1. Model not loading: Ensure that the path to the fine-tuned bundle is correct and that the bundle is properly mounted to /opt/bundle. You may see this error if the bundle is not mounted under /opt/bundle:

nimlib.exceptions.ManifestDownloadError: Error downloading manifest to cache: /opt/nim/.cache error: Invalid Repo ID: path does not exist: /opt/bundle; repo_id: RepoId { repo_path: "/opt/bundle", revision: None, protocol: Some("local") }

and this one if the local directory (which is mounted to /opt/bundle) does not contain the necessary bundle files:

nimlib.exceptions.ManifestDownloadError: Error downloading manifest to cache: /opt/nim/.cache error: Object not found

  1. Incorrect Profile: Verify that the NIM_MANIFEST_PROFILE environment variable is set to the correct profile ID. The profile ID needs to be one of the above-mentioned IDs. Otherwise, you may end up with some automatic profile selection. For fine-tuned model, you can check in the logs to see if the profile is properly set:

"Matched profile_id in manifest from env NIM_MANIFEST_PROFILE to: 43d0baebee73b7bdc2f9c7edf5c726b2323b4f7283eab92fab893d451592656d"

  1. Docker Permissions: Make sure you have the necessary permissions to run Docker commands and access the specified directories.

Logs and Debugging#

To view logs and debug issues, you can use the following Docker commands:

View Logs:

docker logs vista3d

Access Container Shell:

docker exec -it vista3d /bin/bash
On this page