Frequently Asked Questions
Following the steps below to debug your model:
Check your config.json file:
Check if it is valid JSON format:
import json config = json.load(open('config.json', 'r'))
This code should run without exception.
Check your model file:
If you are using Clara to train models, follows Loading TorchScript Model.
Upload model to AIAA again:
Once you make sure all the pieces are correct, upload your model again to AIAA.
Increase
triton_model_timeout
:
AIAA will poll Triton for this amount of time before AIAA claims the model is not imported correctly. If you are using the Triton engine, you can try using a larger timeout to ensure the model import success. (Modify the TRITON_MODEL_TIMEOUT in “docker-compose.env”)
Check your logs:
If all the above steps do not work, start using flag --debug
and check log files in <AIAA workspace>/logs. You can also go to Nvidia Developer Forums.
Currently, AIAA requires models to have a single input and a single output. Multi-class segmentation can be achieved by having multiple channels in output.
Most of the time, this is caused by the mismatch of data. Make sure your testing data in AIAA have the same characteristics as the data that you used to train your models.
That would include the following:
Resolution/Spacing
Orientation
Contrast/Phase
For example, the pre-trained segmentation models on NGC are using data from Medical Segmentation Decathlon.
We re-scale the image to have a spacing of [1.0, 1.0, 1.0] and make sure the affine matrix of Nifti have all positive values.
MONAI provides some nice transforms to tackle the resolution and orientation problems.
Yes, AIAA server supports 2D models. You can use HTTP requests to directly interacting with the AIAA server API.
If your GPU card is very tight on memory, you can do some of the following points to alleviate this:
Load fewer models in the AIAA server
Reduce
roi
(the size of scanning window) in config_aiaa.jsonTry to reduce your network size
Make sure $AIAA_PORT
is not used by other processes.
To start it all clean, remove the workspace folder and create a new one. Then start the AIAA server with the new workspace.
When AIAA runs with Triton backend, it will put one model instance on every GPU that is visible inside the docker.
Users can modify the “device_ids” section under “deploy” section of “tritonserver” service to change the GPU id that
you want to use.control what GPUs are visible.
For the number of model instances on each GPU, users can modify gpu_instance_count
under triton_model_config
in
their model configs.
When a model instance is loaded in GPU, even if it is not serving any inference requests at that moment, it will occupy some amount of GPU memory. As a result, if users want to free that GPU memory, they will have to either stop the AIAA server or unload some models (using DELETE model API).
Yes. But you have to make sure you are using different ports and they do not overlap.
For full cycle of annotation, training, fine-tuning, please use MONAI Label client/plugins for user interaction.
In the MMAR, the model weights is saved explicitly. And when loading it into the AIAA server, it is saved inside the AIAA workspace. But the admins can limit the access to the machine running the AIAA server.
Regular users only have access to the REST API which abstracts them away from the MMAR.
There is a GET Model command (https://docs.nvidia.com/clara/clara-train-sdk/aiaa/server_apis.html), however it only works for clients with admin rights.
Moreover it would only return the config but not the actual trained model weights.