GPT-NeoX / Pythia#
GPT-NeoX is EleutherAI’s large-scale language model architecture. The same GPTNeoXForCausalLM architecture is used by the Pythia scaling suite, OpenAssistant, Databricks Dolly, and StableLM models.
Task |
Text Generation |
Architecture |
|
Parameters |
1B – 20B |
HF Org |
Available Models#
GPT-NeoX-20B (EleutherAI)
Pythia suite: 70M, 160M, 410M, 1B, 1.4B, 2.8B, 6.9B, 12B (EleutherAI)
OA-SFT-Pythia-12B (OpenAssistant)
Dolly-v2-12B (Databricks)
StableLM-tuned-alpha-7B (Stability AI)
Architecture#
GPTNeoXForCausalLM
Example HF Models#
Model |
HF ID |
|---|---|
GPT-NeoX 20B |
|
Pythia 12B |
|
OpenAssistant SFT Pythia 12B |
|
Dolly v2 12B |
|
StableLM tuned alpha 7B |
Try with NeMo AutoModel#
Install NeMo AutoModel and follow the fine-tuning guide to configure a recipe for this model.
1. Install (full instructions):
pip install nemo-automodel
2. Clone the repo to get example recipes you can adapt:
git clone https://github.com/NVIDIA-NeMo/Automodel.git
cd Automodel
3. Fine-tune by adapting a base LLM recipe — override the model ID on the CLI:
automodel --nproc-per-node=8 examples/llm_finetune/llama3_2/llama3_2_1b_squad.yaml \
--model.pretrained_model_name_or_path <MODEL_HF_ID>
Replace <MODEL_HF_ID> with the model ID from Example HF Models above.
Run with Docker
1. Pull the container and mount a checkpoint directory:
docker run --gpus all -it --rm \
--shm-size=8g \
-v $(pwd)/checkpoints:/opt/Automodel/checkpoints \
nvcr.io/nvidia/nemo-automodel:26.02.00
2. The recipes are at /opt/Automodel/examples/ — navigate there:
cd /opt/Automodel
3. Fine-tune:
automodel --nproc-per-node=8 examples/llm_finetune/llama3_2/llama3_2_1b_squad.yaml \
--model.pretrained_model_name_or_path <MODEL_HF_ID>
See the Installation Guide and LLM Fine-Tuning Guide.
Fine-Tuning#
See the LLM Fine-Tuning Guide.