gpt2

GPT-2 is OpenAI’s foundational decoder-only transformer. NeMo AutoModel uses it as a baseline for the Megatron pretraining smoke test and tutorials - its small footprint makes it a convenient target to validate data pipelines, distributed configs, and logging without needing large compute.

Set up NeMo AutoModel with the latest container or follow the installation instructions.

Pretrain gpt2

From the repository root, run:

uv run automodel --nproc-per-node=8 examples/llm_pretrain/megatron_pretrain_gpt2.yaml

Choose a Workflow

Model Reference

Model Architecture

Available Models

Related Resources

• LLM Pretraining Guide