Model fine-tuning

When MAISI NIM is running, by default, it automatically downloads the MAISI model bundle and weights from NGC into the NIM container. However, if you wish to fine-tune MAISI and continue using it within NIM, here is detailed guide to utilize a custom model bundle and weights in MAISI NIM.

Fine-tune the Model

Please follow this tutorial on Project MONAI to learn about how to train a MAISI model.

Use the fine-tuned model in NIM

Once you have the fine-tuned MAISI bundle at your local storage, you simply need to mount it to /opt/bundle and set the correct model manifest profile for it by setting NIM_MANIFEST_PROFILE environment variable.

-e NIM_MANIFEST_PROFILE=43d0baebee73b7bdc2f9c7edf5c726b2323b4f7283eab92fab893d451592656d \
-v /path/to/fine-tuned/bundle:/opt/bundle \

Here is the full command to run the NIM with fine-tuned model.

docker run --rm -it --name maisi \
   --runtime=nvidia -e CUDA_VISIBLE_DEVICES=0 \
   --shm-size=8G \
   -p 8000:8000 \
   -e NGC_API_KEY=$NGC_API_KEY \
   -e NIM_MANIFEST_PROFILE=43d0baebee73b7bdc2f9c7edf5c726b2323b4f7283eab92fab893d451592656d \
   -v /path/to/local/bundle:/opt/bundle \
   nvcr.io/nim/nvidia/maisi:1.0.1

Here is the list of available profiles:

• 56df3d486bbd1c8ac4a1de4d613d4037927bd462d751bf4fe9cbeb8d14890cb8: (default profile) downloads and use MAISI bundle on NGC.
• 43d0baebee73b7bdc2f9c7edf5c726b2323b4f7283eab92fab893d451592656d: utilizes the user provided MAISI bundle (fine-tuned model) at /opt/bundle.

Troubleshooting

If you encounter issues while using the fine-tuned model, here are some common troubleshooting steps:

Common Issues

1. Model not loading: Ensure that the path to the fine-tuned bundle is correct and that the bundle is properly mounted to /opt/bundle. You may see this error if the bundle is not mounted under /opt/bundle:

   nimlib.exceptions.ManifestDownloadError: Error downloading manifest to cache: /opt/nim/.cache error: Invalid Repo ID: path does not exist: /opt/bundle; repo_id: RepoId { repo_path: "/opt/bundle", revision: None, protocol: Some("local") }

   and this one if the local directory (which is mounted to /opt/bundle) does not contain the necessary bundle files:

   nimlib.exceptions.ManifestDownloadError: Error downloading manifest to cache: /opt/nim/.cache error: Object not found

2. Incorrect Profile: Verify that the NIM_MANIFEST_PROFILE environment variable is set to the correct profile ID. The profile ID needs to be one of the above-mentioned IDs. Otherwise, you may end up with some automatic profile selection. For fine-tuned model, you can check in the logs to see if the profile is properly set:

   "Matched profile_id in manifest from env NIM_MANIFEST_PROFILE to: 43d0baebee73b7bdc2f9c7edf5c726b2323b4f7283eab92fab893d451592656d"

3. Docker Permissions: Make sure you have the necessary permissions to run Docker commands and access the specified directories.

Logs and Debugging

To view logs and debug issues, you can use the following Docker commands:

View Logs:

docker logs maisi

Access Container Shell:

docker exec -it maisi /bin/bash