StarCoder2

Released in 2024, BigCode’s StarCoder2 is a code-generation large language mode based on the transformer decoder framework. The model was trained on a massive code dataset called The Stack v2 which contains over 900B tokens. StarCoder2 is offered in 3B, 7B, and 15B sizes and competes with models well above its size on code-generation tasks. More information can be found in the companion paper “StarCoder 2 and The Stack v2: The Next Generation”.

• StarCoder2
• Data Preparation for SFT and PEFT
  • Step 1: Download dataset
  • Step 2: Data Preprocessing
  • Step 3: Split the data into train, validation and test.
  • Step 4: Prepare Packed Dataset
• Checkpoint Conversion
• Model Evaluation
  • Run Evaluation
    • Configuration
• Supervised Fine-tuning (SFT)
  • Run SFT inside NeMo Container
    • Step 1: Start NeMo Container
    • Step 2: Run SFT
    • Step 3: Run evaluation
  • Run SFT with NeMo Launcher
    • Configuration
• Parameter Efficient Fine-Tuning (PEFT)
  • Run PEFT inside NeMo Container
    • Step 1: Start NeMo Container
    • Step 2: Run PEFT
    • Step 3: Run evaluation
    • Step 4 (Optional): Merge LORA weights
  • Run PEFT with NeMo Launcher
    • Configuration

Feature	Status
Data parallelism	✓
Tensor parallelism	✓
Pipeline parallelism	✓
Interleaved Pipeline Parallelism Sched	N/A
Sequence parallelism	✓
Selective activation checkpointing	✓
Gradient checkpointing	✓
Partial gradient checkpointing	✓
FP32/TF32	✓
AMP/FP16	✗
BF16	✓
TransformerEngine/FP8	✗
Multi-GPU	✓
Multi-Node	✓
Inference	N/A
Slurm	✓
Base Command Manager	✓
Base Command Platform	✓
Distributed data preprcessing	✓
NVfuser	✗
P-Tuning and Prompt Tuning	✓
IA3 and Adapter learning	✓
Distributed Optimizer	✓
Distributed Checkpoint	✓
Fully Shared Data Parallel	N/A