If you are using Clara-Train to train your model, AIAA provides functionality to let you fine-tune your trained models based on new annotation samples you make.
AIAA will take care of fine-tuning the model and re-load it back to AIAA for serving.
This is an Admin API.
If you have only 1 GPU in your system, then AIAA can’t serve inference through segmentation/annotation APIs during fine-tuning.
You need to make sure you put new samples in
/workspace/samples and have your MMAR in
To see an example of MMAR check nvmidl/mmar:Medical Model Archive (MMAR).
Below is an example directory for clara_ct_seg_spleen_amp model
workspace/ samples/ clara_pt_spleen_ct_segmentation/ options.conf #optional dataset.json #optional images/ spleen_2.nii.gz spleen_3.nii.gz ... labels/ spleen_2.nii.gz spleen_3.nii.gz ... mmars/ clara_pt_spleen_ct_segmentation/ commands/ train_finetune.sh configs/ ...
The uri to call is /admin/finetune/[model]. Note that you can pass all options that nvmidl/mmar:train.sh takes. There are two ways of passing those options, one is via curl, which is demonstrated below.
# basic call curl -X POST "http://127.0.0.1:$AIAA_PORT/admin/finetune/clara_ct_seg_spleen_amp" # fine-tune for 5 epochs, %3D means space curl -X POST "http://127.0.0.1:$AIAA_PORT/admin/finetune/clara_ct_seg_spleen_amp?options=epochs%3D5"
The other is to edit the options.conf file. An example is below. You should put this
file inside your
If you want to pick up specific images for training and validation,
then you can provide your own configuration at:
You can also set automatic fine-tuning by adding flag
--fine_tune true when starting AIAA.
AIAA will run model fine-tune on the
fine_tune_hour every day for all the models.