Nemotron Training Recipes#
Open and efficient models for agentic AI. Reproducible training pipelines with transparent data, techniques, and weights.
Quick Start#
// Install the Nemotron training recipes
$ git clone https://github.com/NVIDIA-NeMo/Nemotron
$ cd Nemotron && uv sync
// Run a tiny SFT job on your cluster
$ uv run nemotron steps run sft/automodel -c tiny --run YOUR-CLUSTER
// Run the Nano3 pipeline stage by stage
$ uv run nemotron nano3 data prep pretrain --run YOUR-CLUSTER
$ uv run nemotron nano3 pretrain --run YOUR-CLUSTER
$ uv run nemotron nano3 data prep sft --run YOUR-CLUSTER
$ uv run nemotron nano3 sft --run YOUR-CLUSTER
$ uv run nemotron nano3 data prep rl --run YOUR-CLUSTER
$ uv run nemotron nano3 rl --run YOUR-CLUSTER
Note: The
--run YOUR-CLUSTERflag submits jobs to your configured Slurm cluster via NeMo-Run. See Execution through NeMo-Run for setup instructions.
Deploy, Customize, and Use Nemotron#
Deployment guides for Nemotron models: TensorRT-LLM, vLLM, SGLang, NIM, Hugging Face, and agent harnesses.
End-to-end applications: RAG agents, ML agents, and multi-agent systems.
Estimate GPU count, training time, and compute cost for customizing a released Nemotron checkpoint with SFT, LoRA, or GRPO.
Customization Workflows with Nemotron Steps#
How tokenizer extension, curation, continued pretraining, synthetic SFT data, fine-tuning, and evaluation connect into one workflow.
Translate JSONL or Parquet corpora with translate/nemo_curator, NeMo Curator
backends, and optional FAITH quality scoring.
Generate and translate custom multiple-choice benchmarks with byob/mcq.
Generate and evaluate custom function-calling benchmarks from an executable oracle pack with byob/bfcl.
Ingest, profile, filter, audit, decontaminate, and subset multilingual text corpora with the curate steps.
Use sdg/data_designer to produce SFT, tool-use, and preference datasets.
Evaluate hosted endpoints or checkpoints with eval/model_eval.
Add target-language tokens to a base tokenizer and initialize their embeddings with tokenizer_extension/* before continued pretraining.
Training Recipes#
550B total / 55B active parameters, 20T tokens, up to 1M context. Hybrid Mamba-Attention MoE with LatentMoE and MTP.
Stages: Pretraining → SFT → RLVR → MOPD
120.6B total / 12.7B active parameters, up to 1M context. Hybrid Mamba-Transformer with sparse Latent MoE.
Stages: Pretraining → SFT → RL → Quantization → Eval
120B total / 12B active vision-language model with native MTP, post-trained with specialist teachers and multi-teacher on-policy distillation.
Stages: Pretraining → SFT → RLVR → Teachers → MOPD → Eval
31.6B total / 3.6B active parameters, 25T tokens, up to 1M context. Hybrid Mamba-Transformer with sparse MoE.
Stages: Pretraining → SFT → RL
30B total / 3B active parameters. Hybrid Mamba-Transformer with sparse MoE and Multi-Token Prediction, built for the high-volume execution layer of long-running agents.
Stages: Pretraining → SFT → RL → Quantization
34B total / 4.3B active offline speech-and-audio model with contextual transcription, speaker attribution, timestamps, translation, and English-only evaluated audio understanding.
Public workflow: Main-stage SALM construction or released-checkpoint adaptation → target branches (optional DFlash2 draft; optional quantization) → Inference (canonical execution in NeMo Speech)
GA-checkpoint multimodal post-training recipe with stage-local container builds and a three-step RL stack.
Stages: SFT → RL MPO → RL text → RL vision → Eval
Fine-tune Nemotron text and multimodal embedding models with synthetic data generation, retrieval evaluation, and NIM or vLLM deployment.
Stages: SDG → Data Prep → Finetune → Eval → Export → Deploy
Train text and multimodal cross-encoder rerankers. The Llama recipe includes synthetic data generation, evaluation, export, and NIM deployment.
Stages: SDG → Data Prep → Finetune → Eval → Export → Deploy
Recipe Layout#
Nemotron keeps data-producing recipes separate from model-family training recipes:
Path |
Purpose |
Example |
|---|---|---|
|
Filter, dedup, and curate existing corpora |
|
|
Generate synthetic datasets that can feed multiple families |
Long-document SDG feeding Omni3 SFT |
|
Family-specific training, RL, evaluation, and model lifecycle commands |
Training Pipeline#
Nemotron-native recipe families can be tracked through artifact lineage.
Family |
Stage layout |
|---|---|
Pretraining → SFT → RL |
|
Pretraining → SFT → RL → Quantization |
|
SFT → RL MPO → RL text → RL vision → Eval |
|
Pretraining → SFT → RL → Quantization → Eval |
|
Pretraining → SFT → RLVR → Teachers → MOPD → Eval |
|
Pretraining → SFT → RLVR → MOPD |
|
SDG → Data Prep → Finetune → Eval → Export (Llama profile only) → Deploy |
|
SDG → Data Prep → Finetune → Eval → Export (Llama profile only) → Deploy |
Externally maintained workflow maps#
Some model integrations use Nemotron for discovery and model-specific guidance while their executable pipeline remains canonical in another NVIDIA repository. They do not emit Nemotron artifact lineage unless a native recipe is added.
Family |
Public workflow |
Canonical execution |
|---|---|---|
Main-stage SALM construction or released-checkpoint adaptation → target branches (optional DFlash2 draft; optional quantization) → Inference |
Why Nemotron?#
Open Models |
Transparent training data, techniques, and weights for community innovation |
Compute Efficiency |
Model pruning enabling higher throughput via TensorRT-LLM |
High Accuracy |
Built on frontier open models with human-aligned reasoning |
Flexible Deployment |
Deploy anywhere: edge, single GPU, or data center with NIM |
Features#
End-to-end pipelines from raw data to deployment-ready models
Artifact lineage via W&B from data to model
Built on NVIDIA’s NeMo stack (Megatron-Bridge, NeMo-RL)
Reproducible with versioned configs, data blends, and checkpoints
Resources#
Tech Report – Nemotron 3 Nano methodology
Model Weights – pre-trained checkpoints on HuggingFace
Pre-training Datasets – open pre-training data
Post-training Datasets – SFT and RL data
Artifact Lineage – W&B integration guide
Model training steps – SFT, PEFT, RL, and optimization with
nemotron step runTraining FAQ – answers to common questions about training and fine-tuning Nemotron models