ChatGLM

Released in 2023, ChatGLM3 is the third in a series of pre-trained dialogue models jointly released by Zhipu AI and Tsinghua KEG. Building off the work in the “GLM: General Language Model Pretraining with Autoregressive Blank Infilling” paper, ChatGLM3-6B is an open-source offering in the ChatGLM3 series. Although it is open open-source it retains many excellent features of the first two generations such as smooth dialogue and easy deployment. The provided documentation works for both ChatGLM3-6B and ChatGLM2-6B

• Data Preparation
• Training with Predefined Configurations
• Checkpoint Conversion
• Model Evaluation
• Parameter Efficient Fine-Tuning (PEFT)

Feature	Status
Data parallelism	✓
Tensor parallelism	✓
Pipeline parallelism	✓
Interleaved Pipeline Parallelism Sched	N/A
Sequence parallelism	✓
Selective activation checkpointing	✓
Gradient checkpointing	✓
Partial gradient checkpointing	✓
FP32/TF32	✓
AMP/FP16	✗
BF16	✓
TransformerEngine/FP8	✗
Multi-GPU	✓
Multi-Node	✓
Inference	N/A
Slurm	✓
Base Command Manager	✓
Base Command Platform	✓
Distributed data preprcessing	✓
NVfuser	✗
P-Tuning and Prompt Tuning	✓
IA3 and Adapter learning	✓
Distributed Optimizer	✓
Distributed Checkpoint	✓
Fully Shared Data Parallel	N/A