Model CoverageLarge Language Models

GPT-2

View as Markdown

GPT-2 is OpenAI’s foundational decoder-only transformer. NeMo AutoModel uses it as a baseline for the Megatron pretraining smoke test and tutorials — its small footprint makes it a convenient target to validate data pipelines, distributed configs, and logging without needing large compute.

TaskText Generation (pretraining baseline)
ArchitectureGPT2LMHeadModel
Parameters124M – 1.5B
HF Orgopenai-community

Available Models

  • gpt2 (124M)
  • gpt2-medium (355M)
  • gpt2-large (774M)
  • gpt2-xl (1.5B)

Architecture

  • GPT2LMHeadModel

Example HF Models

ModelHF ID
GPT-2openai-community/gpt2

Example Recipes

RecipeDescription
megatron_pretrain_gpt2.yamlMegatron pretraining smoke test — GPT-2 on FineWeb-Edu

Try with NeMo AutoModel

1. Clone and install from source (full instructions):

$git clone https://github.com/NVIDIA-NeMo/Automodel.git
$cd Automodel
$uv sync --locked --all-groups --all-extras

2. Run the recipe from inside the repo:

$uv run automodel --nproc-per-node=8 examples/llm_pretrain/megatron_pretrain_gpt2.yaml

See the Installation Guide and LLM Pretraining Guide.

Hugging Face Model Cards