NeMo RL Documentation#

Welcome to the NeMo RL documentation. NeMo RL is an open-source post-training library developed by NVIDIA, designed to streamline and scale reinforcement learning methods for multimodal models (LLMs, VLMs, etc.).

This documentation provides comprehensive guides, examples, and references to help you get started with NeMo RL and build powerful post-training pipelines for your models.

Getting Started#

Overview

Learn about NeMo RL’s architecture, design philosophy, and key features that make it ideal for scalable reinforcement learning.

Overview

Quick Start

Get up and running quickly with examples for both Automodel and Megatron Core training backends.

Quick Start

Installation

Step-by-step instructions for installing NeMo RL, including prerequisites, system dependencies, and environment setup.

Installation and Prerequisites

Features

Explore the current features and upcoming enhancements in NeMo RL, including distributed training, advanced parallelism, and more.

Features and Roadmap

Tips and Tricks

Troubleshooting common issues including missing submodules and memory fragmentation.

Tips and Tricks

Training and Generation#

Training Backends

Learn about Automodel and Megatron Core training backends, their capabilities, and how to choose the right one for your use case.

Training and Generation Backends

Algorithms

Discover supported algorithms including GRPO, PPO, SFT, DPO, RM, on-policy distillation, and multi-teacher on-policy distillation (MOPD) with detailed guides and examples.

Algorithms

Evaluation

Measure model and agent capabilities with repeatable benchmarks and custom evaluations.

Evaluation

Cluster Setup

Configure and launch NeMo RL on multi-node Slurm or Kubernetes clusters for distributed computing.

Set Up Clusters

Managed Dynamo Generation

Run a fixed Dynamo vLLM fleet with NCCL refit and W&B telemetry inside a Slurm Ray allocation.

Managed Dynamo generation on Slurm

Guides and Examples#

GRPO DeepscaleR

Reproduce DeepscaleR results with NeMo RL using GRPO on mathematical reasoning tasks.

GRPO on DeepScaler

SFT on OpenMathInstruct2

Step-by-step guide for supervised fine-tuning on the OpenMathInstruct2 dataset.

SFT on OpenMathInstruct-2

Nemotron 3 Ultra

Post-train Nemotron 3 Ultra with RLVR, teacher training, and MOPD stages on GB200 NVL72 hardware.

Nemotron 3 Ultra

Environments

Create custom reward environments and integrate them with NeMo RL training pipelines.

Environments for GRPO Training

Eagle3 Speculative Decoding

Configure offline and online Eagle3 draft-model workflows to accelerate rollout generation with vLLM.

Train with Eagle3 Speculative Decoding

Automodel Speculative Decoding

Co-train DSpark, DFlash, and EAGLE3 drafters alongside the policy on the Automodel backend.

Speculative Decoding Draft Co-Training on Automodel Backend

Audio GRPO on AVQA

Train Qwen2.5-Omni-3B with GRPO on AVQA and evaluate on MMAU, following the R1-AQA approach.

Audio Post-training with NeMo RL

Audio-Visual Intent GRPO

Train Qwen2.5-Omni-7B with GRPO on PhilipC/IntentTrain (audio-visual intent recognition) and evaluate on Daily-Omni, following HumanOmniV2’s joint audio-visual setup.

Audio-Visual GRPO with Qwen2.5-Omni-7B

Two-Stage SWE RL (Qwen3 Thinking)

Train Qwen3-30B-A3B-Thinking into a SWE agent with a pivot stage plus end-to-end agentic RL on SWE-bench.

Two-Stage SWE RL for Qwen3-30B-A3B-Thinking

Adding New Models

Learn how to add support for new model architectures in NeMo RL.

Add New Models

LoRA

Parameter-efficient fine-tuning with LoRA: backend support, Automodel vs Megatron schema comparison, config examples, and recipes.

LoRA (Low-Rank Adaptation)

YaRN Long-Context Training

Extend a model’s context window with YaRN RoPE scaling on the Megatron backend for SFT, GRPO, and other workflows.

YaRN Long-Context Training

Cross-Tokenizer Distillation

Off-policy distillation across mismatched tokenizers — build a (student, teacher) projection matrix and run x-token KD via CUDA-IPC teacher logits.

Cross-Tokenizer (X-Token) Off-Policy Distillation

Weight Refit

Choose among colocated IPC, NCCL, sparse delta, and NIXL refit transports.

Weight Refit: Choosing a Transport

Checkpoint-Engine Refit

Use NIXL checkpoint-engine refit to update non-colocated vLLM generation workers from policy workers.

Checkpoint-Engine Refit

Single-Controller (Async GRPO and PPO)

Run async GRPO or PPO via the SingleController path: TransferQueue data plane, pluggable staleness samplers, and streaming trainer.

Train with Single-Controller (Async GRPO and PPO)

Advanced Topics#

Design and Philosophy

Deep dive into NeMo RL’s architecture, APIs, and design decisions for scalable RL.

Design and Philosophy

Debugging

Tools and techniques for debugging distributed Ray applications and RL training runs.

Debug NeMo RL Applications

Observability

OpenTelemetry traces and rl.* metrics via nemo-lens: span groups, configuration, vLLM tracing, and an OTLP export stack.

Observability

FP8 Quantization

Optimize large language models with FP8 quantization for faster training and inference.

FP8 Quantization in NeMo RL

Quantization-Aware RL

Run quantized GRPO and distillation with NVIDIA ModelOpt.

Quantization-Aware RL (QARL)

End-to-End NVFP4 Training

Run W4A4 policy training and per-token vLLM rollout on NVIDIA Blackwell GPUs.

End-to-End W4A4 NVFP4 with Per-Token vLLM Rollout

Docker Containers

Build and use Docker containers for reproducible NeMo RL environments.

Build Docker Images

API Reference#

Complete API Documentation

Comprehensive reference for all NeMo RL modules, classes, functions, and methods. Browse the complete Python API with detailed docstrings and usage examples.

API Reference