Model Fine-tune

If you are using Clara-Train to train your model, AIAA provides functionality to let you fine-tune your trained models based on new annotation samples you make.

AIAA will take care of fine-tuning the model and re-load it back to AIAA for serving.

Note

This is an Admin API.

Attention

If you have only 1 GPU in your system, then AIAA can’t serve inference through segmentation/annotation APIs during fine-tuning.

Setup

You need to make sure you put new samples in /workspace/samples and have your MMAR in /workspace/mmars

Hint

To see an example of MMAR check Medical Model Archive (MMAR).

Below is an example directory for clara_ct_seg_spleen_amp model

workspace/
    samples/
        clara_pt_spleen_ct_segmentation/
            options.conf   #optional
            dataset.json   #optional
            images/
                spleen_2.nii.gz
                spleen_3.nii.gz
                ...
            labels/
                spleen_2.nii.gz
                spleen_3.nii.gz
                ...
    mmars/
        clara_pt_spleen_ct_segmentation/
            commands/
                train_finetune.sh
            configs/
            ...

Call fine-tune API

The uri to call is /admin/finetune/[model]. Note that you can pass all options that train.sh takes. There are two ways of passing those options, one is via curl, which is demonstrated below.

# basic call
curl -X POST "http://127.0.0.1:$AIAA_PORT/admin/finetune/clara_ct_seg_spleen_amp"

# fine-tune for 5 epochs, %3D means space
curl -X POST "http://127.0.0.1:$AIAA_PORT/admin/finetune/clara_ct_seg_spleen_amp?options=epochs%3D5"

The other is to edit the options.conf file. An example is below. You should put this file inside your /workspace/samples/[model]/options.conf.

epochs=5
learning_rate=0.00001

Hint

If you want to pick up specific images for training and validation, then you can provide your own configuration at: /workspace/samples/{model}/dataset.json.

Schedule automatic fine-tune

You can also set automatic fine-tuning by adding flag --fine_tune true when starting AIAA. AIAA will run model fine-tune on the fine_tune_hour every day for all the models.