RETRO

Released in December 2021, the Retrieval-Enhanced Transformer (RETRO) model is an innovative approach to enhancing auto-regressive language models. Developed by researchers at DeepMind, RETRO leverages retrieval from a large text database as a complementary memory to scale reasoning capabilities without significantly increasing computational requirements. More information is available in the companion paper “Improving language models by retrieving from trillions of tokens”.

• Data Preparation
• Training
• Training with Predefined Configurations
• Model Inferencing
• Model Evaluation

Feature	Status
Data parallelism	✓
Tensor parallelism	✗
Pipeline parallelism	✗
Interleaved Pipeline Parallelism Sched	N/A
Sequence parallelism	✗
Selective activation checkpointing	✓
Gradient checkpointing	✓
Partial gradient checkpointing	✓
FP32/TF32	✓
AMP/FP16	✓
BF16	✓
TransformerEngine	✓
TransformerEngine/FP8	✗
Multi-GPU	✓
Multi-Node	✓
Inference	✓
Slurm	✓
Base Command Manager	✓
Kubernetes	✗
Distributed data preprcessing	N/A
NVfuser	✗
P-Tuning and Prompt Tuning	✗
IA3 and Adapter learning	✗
Distributed Optimizer	✓
Distributed Checkpoint	✓
Fully Shared Data Parallel	✗