Checkpoint Conversion

NVIDIA provides scripts to convert the external StarCoder2 checkpoints from HuggingFace format to .nemo format. The .nemo checkpoint will be used for SFT, PEFT, and inference. NVIDIA also provides scripts to convert .nemo format back to HuggingFace format.

Run the container using the following command:

            

            
docker run --gpus device=1 --shm-size=2g --net=host --ulimit memlock=-1 --rm -it -v ${PWD}:/workspace -w /workspace -v ${PWD}/results:/results nvcr.io/nvidia/nemo:24.01.starcoder2 bash
        

Convert the HuggingFace StarCoder2 model to .nemo model:

            

            
python3 /opt/NeMo/scripts/checkpoint_converters/convert_starcoder2_hf_to_nemo.py \
--input_name_or_path /path/to/starcoder2/checkpoints/hf \
--output_path /path/to/starcoder2.nemo
        

The generated starcoder2.nemo file uses distributed checkpointing and can be loaded with any tensor parallel (tp) or pipeline parallel (pp) combination without reshaping/splitting.

Convert the Starcoder2 .nemo model to HuggingFace

            

            
python3 /opt/NeMo/scripts/checkpoint_converters/convert_starcoder2_hf_to_nemo.py \
--input_name_or_path /path/to/starcoder2/nemo/checkpoint \
--output_path /path/to/hf/folder
        

The generated HuggingFace checkpoint folder can be loaded using HuggingFace transformers pipeline and upload to the hub.