NeMo RL Documentation

Welcome to the NeMo RL documentation. NeMo RL is an open-source post-training library developed by NVIDIA, designed to streamline and scale reinforcement learning methods for multimodal models (LLMs, VLMs, etc.).

This documentation provides comprehensive guides, examples, and references to help you get started with NeMo RL and build powerful post-training pipelines for your models.

Getting Started

Overview

Learn about NeMo RL’s architecture, design philosophy, and key features that make it ideal for scalable reinforcement learning.

Overview

Quick Start

Get up and running quickly with examples for both DTensor and Megatron Core training backends.

Quick Start

Installation

Step-by-step instructions for installing NeMo RL, including prerequisites, system dependencies, and environment setup.

Installation and Prerequisites

Features

Explore the current features and upcoming enhancements in NeMo RL, including distributed training, advanced parallelism, and more.

Features and Roadmap

Tips and Tricks

Troubleshooting common issues including missing submodules and memory fragmentation.

Tips and Tricks

Training and Generation

Training Backends

Learn about DTensor and Megatron Core training backends, their capabilities, and how to choose the right one for your use case.

Training and Generation Backends

Algorithms

Discover supported algorithms including GRPO, PPO, SFT, DPO, RM, on-policy distillation, and multi-teacher on-policy distillation (MOPD) with detailed guides and examples.

Algorithms

Evaluation

Learn how to evaluate your models using built-in evaluation datasets and custom evaluation pipelines.

Evaluation

Cluster Setup

Configure and launch NeMo RL on multi-node Slurm or Kubernetes clusters for distributed computing.

Set Up Clusters

Guides and Examples

GRPO DeepscaleR

Reproduce DeepscaleR results with NeMo RL using GRPO on mathematical reasoning tasks.

GRPO on DeepScaler

SFT on OpenMathInstruct2

Step-by-step guide for supervised fine-tuning on the OpenMathInstruct2 dataset.

SFT on OpenMathInstruct-2

Nemotron 3 Ultra

Post-train Nemotron 3 Ultra with RLVR, teacher training, and MOPD stages on GB200 NVL72 hardware.

Nemotron 3 Ultra

Environments

Create custom reward environments and integrate them with NeMo RL training pipelines.

Environments for GRPO Training

Eagle3 Speculative Decoding

Configure offline and online Eagle3 draft-model workflows to accelerate rollout generation with vLLM.

Train with Eagle3 Speculative Decoding

Audio GRPO on AVQA

Train Qwen2.5-Omni-3B with GRPO on AVQA and evaluate on MMAU, following the R1-AQA approach.

Audio Post-training with NeMo RL

Audio-Visual Intent GRPO

Train Qwen2.5-Omni-7B with GRPO on PhilipC/IntentTrain (audio-visual intent recognition) and evaluate on Daily-Omni, following HumanOmniV2’s joint audio-visual setup.

Audio-Visual GRPO with Qwen2.5-Omni-7B

Two-Stage SWE RL (Qwen3 Thinking)

Train Qwen3-30B-A3B-Thinking into a SWE agent with a pivot stage plus end-to-end agentic RL on SWE-bench.

Two-Stage SWE RL for Qwen3-30B-A3B-Thinking

Adding New Models

Learn how to add support for new model architectures in NeMo RL.

Add New Models

LoRA

Parameter-efficient fine-tuning with LoRA: backend support, DTensor vs Megatron schema comparison, config examples, and recipes.

LoRA (Low-Rank Adaptation)

YaRN Long-Context Training

Extend a model’s context window with YaRN RoPE scaling on the Megatron backend for SFT, GRPO, and other workflows.

YaRN Long-Context Training

Cross-Tokenizer Distillation

Off-policy distillation across mismatched tokenizers — build a (student, teacher) projection matrix and run x-token KD via CUDA-IPC teacher logits.

Cross-Tokenizer (X-Token) Off-Policy Distillation

Weight Refit

Choose among colocated IPC, NCCL, sparse delta, and NIXL refit transports.

Weight Refit: Choosing a Transport

Checkpoint-Engine Refit

Use NIXL checkpoint-engine refit to update non-colocated vLLM generation workers from policy workers.

Checkpoint-Engine Refit

Advanced Topics

Design and Philosophy

Deep dive into NeMo RL’s architecture, APIs, and design decisions for scalable RL.

Design and Philosophy

Debugging

Tools and techniques for debugging distributed Ray applications and RL training runs.

Debug NeMo RL Applications

FP8 Quantization

Optimize large language models with FP8 quantization for faster training and inference.

FP8 Quantization in NeMo RL

Quantization-Aware RL

Run quantization-aware GRPO and distillation using NVIDIA ModelOpt. Includes NVFP4 W4A4 and W4A16 real rollout.

Quantization-Aware RL (QARL)

Docker Containers

Build and use Docker containers for reproducible NeMo RL environments.

Build Docker Images

API Reference

Complete API Documentation

Comprehensive reference for all NeMo RL modules, classes, functions, and methods. Browse the complete Python API with detailed docstrings and usage examples.

API Reference

Full Documentation Index

The complete table of contents below lists all pages, including guide, development, and design-doc pages that are not shown in the cards above.

About

• Overview
  • What You Can Expect
  • Releases
• Performance
  • Nomenclature
  • Performance Metrics
  • Performance Summary for Large Language Models
  • NeMo RL v0.6
    • H100 BF16 Benchmarks
    • H100 FP8 Benchmarks
    • GB200 BF16 Benchmarks
• Model Support
  • Broad coverage for 🤗Hugging Face models via NeMo AutoModel
  • Optimal acceleration for top models via NeMo Megatron Bridge
• Features and Roadmap
  • Available Now
  • Planned / In Progress
• Training and Generation Backends
  • Training Backends
  • Generation Backends
• Quick Start
  • Quick Start Options
    • Smoke Test
• Installation and Prerequisites
  • Docker Quickstart
  • Clone the Repository
  • Install System Dependencies
    • cuDNN (For Megatron Backend)
    • libibverbs (For vLLM Dependencies)
  • Install UV Package Manager
  • Create Virtual Environment
  • Configure cuDNN for Transformer Engine (Bare Metal Only)
  • Using UV to Run Commands
• Algorithms
  • Support Matrix
• Evaluation
  • Convert Model Format (Optional)
  • Run Evaluation
• Set Up Clusters
• Tips and Tricks
  • Missing Submodules Error
  • Memory Fragmentation

Environment Start

• Run on Your Local Workstation
• Set Up Clusters
  • Use Slurm for Batched and Interactive Jobs
    • Batched Job Submission
    • Interactive Launching
    • Slurm Environment Variables
      • Common Environment Configuration
      • Advanced Environment Configuration
    • Topology-Aware Placement for MoE (avoiding cross-rack stalls)
  • Kubernetes
  • Prerequisites
    • 1. Test Cluster Access
    • 2. Build and Push the Docker Container
  • Phase 1: Infrastructure Setup
    • 1. Configure Shared Storage (NFS)
    • 2. Create Registry Secret
  • Phase 2: Ray Cluster Configuration
  • Phase 3: Run Sample NemoRL Workloads
    • 1. Access the Head Node
    • 2. Setup Code on Shared Volume
    • 3. Submit a Job
    • 4. Configuration Adjustments
    • 5. Monitoring
  • Utility: PVC Busybox Helper

E2E Examples

• SFT on OpenMathInstruct-2
  • Train the Model
    • Dataset Splits
    • Training Time
  • Evaluate the Model
  • Results
    • train_1M
    • train

Guides

• Nemotron 3 Nano
  • Download and prepare the data
  • Prepare the code
  • Create a launch script
  • Launch training
• Nemotron 3 Nano Omni
  • AutoModel backend
    • Recipe 1 — CLEVR-CoGenT (single-node)
    • Launch (interactive container)
    • Recipe 2 — MMPR-Tiny (4-node Slurm)
    • Launch (4-node Slurm)
  • Megatron backend
    • Checkpoint compatibility
    • Maintained recipes
• Nemotron 3 Super
  • Container
  • Download and prepare the data
  • Prepare the code
  • Training the model
    • Build sandbox container
    • Launch script
    • Stage 1 - RLVR
      • Stage 1.1 - RLVR 1 (183 H100 GPU nodes)
      • Stage 1.2 - RLVR 2 (183 H100 GPU nodes)
      • Stage 1.3 - RLVR 3 (183 H100 GPU nodes)
    • Stage 2 - SWE
      • Stage 2.1 - SWE 1 (64 H100 GPU nodes)
      • Stage 2.2 - SWE 2 (64 H100 GPU nodes)
    • Stage 3 - RLHF (100 H100 GPU nodes)
• Nemotron 3 Ultra
  • Overview
    • Checkpoint flow
  • Container
  • Download and prepare the data
  • Prepare the code
  • Build the sandbox container
  • Launch script
  • Stage 1 — Student RLVR
    • Phase 1 — 49k context, 128 steps
    • Phase 2 — 65k context, ~50 more steps
  • Stage 2 — Teacher training
    • IFBench Teacher
    • RLHF Teacher
    • Reasoning Teacher
    • SWE Teacher
    • Building the SIF images
  • Stage 3 — MOPD
    • Teacher mapping
    • Launch
• Add New Models
  • Importance of Log Probability Consistency in Training and Inference
  • Understand Discrepancies Between Backends
  • 1. Hugging Face–Based Models
    • Validation Workflow
  • 2. Megatron Models
    • Additional Validation
  • 3. Expected Error Threshold
• Model Diagnostics
  • 1.max_model_len_respected.py
  • 2.long_generation_decode_vs_prefill
  • 3.check_and_reinit_hf_model_embeddings_untrained.py
  • 4.vllm_precision_compilation_test.py
  • 5.prefix_caching_nan.py
  • 6.vllm_routed_experts_completeness.py
• Supervised Fine-Tuning in NeMo RL
  • Launch an SFT Run
  • Example Configuration File
  • Datasets
    • Custom datasets defined outside NeMo RL
    • OpenAI Format Datasets (with Tool Calling Support)
      • Basic Usage
      • Data Format
      • Tool Calling with Heterogeneous Schemas
  • Evaluate the Trained Model
  • LoRA Configuration
  • Optimizations
    • Chunked Linear Cross-Entropy Fusion Loss
• Direct Preference Optimization in NeMo RL
  • Launch a DPO Run
  • Configuration
  • Datasets
    • Custom datasets defined outside NeMo RL
  • DPO-Specific Parameters
  • LoRA Configuration
  • Optimizations
    • Chunked Fused Linear Logprobs
  • Evaluate the Trained Model
• An in-depth Walkthrough of DAPO in NeMo RL
  • Quickstart: Launch a DAPO Run
  • Dynamic Sampling
    • About batch_multiplier
  • Reward Shaping
  • Configuration
  • Example Training Results
  • References
• LoRA (Low-Rank Adaptation)
  • Backend Support
  • Schema Comparison
  • DTensor Configuration
    • DTensor Parameter Details
  • Megatron Configuration
    • Megatron Parameter Details
  • Usage by Algorithm
    • SFT
    • GRPO
    • DPO
  • Example Recipes
  • Exporting a LoRA Checkpoint to Hugging Face Format
• An in-depth Walkthrough of CISPO in NeMo RL
  • Quickstart: Launch a CISPO Run
  • The CISPO Objective
  • Configuration
  • References
• An In-Depth Walkthrough of ProRLv2 in NeMo RL
  • Quickstart: Launch a ProRLv2 Run
  • DAPO: Dynamic Sampling
  • Decoupled local/global advantage normalization
  • Reward Shaping: “Stop properly” Penalty (Truncation Penalty)
  • Loss: Decoupled (Asymmetric) Clipping
  • Loss: Token-level Policy Gradient
  • Importance Sampling Correction and MoE Stability
    • Filtering Strategies
  • Full Example Configs
  • Practical Overrides
  • What to Monitor
  • References
• Two-Stage SWE RL for Qwen3-30B-A3B-Thinking
  • The two stages
  • What makes this a thinking agent
    • Interleaved thinking across turns
    • Three agent harnesses (mixed-prompt training)
  • Async GRPO + MoE setup
  • Results
    • Stage 1 (SWE1, pivot) training reward
    • Stage 2 (SWE2) training reward & agent turns
  • Launch
    • Prerequisites
    • Stage 1 — SWE1 (pivot)
    • Stage 2 — SWE2 (end-to-end agentic)
  • [Alternative] Using TensorRT-LLM generation backend in Stage 2
  • What to monitor
  • References
• An In-depth Walkthrough of GRPO in NeMo RL
  • Quickstart: Launch a GRPO Run
    • Data
      • Dataset
        • Custom datasets defined outside NeMo RL
      • Common Data Format
      • Data Processors
      • Multiple Dataloaders
    • Task–Dataset Mapping
      • Putting It All Together
  • Environments
    • Env–Task Mapping
  • Policy Model
  • Fast Generation
    • Generation Sampling Parameters (temperature, top-p, top-k)
  • Performance Optimizations
    • Chunked Fused Linear Logprobs
  • Loss
    • Improvements to the GRPO Loss Formulation for Stability and Accuracy
      • On-Policy KL Approximation
      • Importance Sampling Correction
      • Overlong Filtering
      • Advantage Clipping
      • Top-p and top-k filtering
  • Metrics
    • Multiplicative Token Probability Error
    • KL Divergence Error
    • Sampling Importance Ratio
    • Entropy
    • GDPO: Group reward-Decoupled Normalization Policy Optimization for Multi-reward RL Optimization
      • Per-reward weights
      • Comparing GDPO against a GRPO baseline
      • What to measure
      • Practical notes
  • LoRA Configuration
  • Evaluate the Trained Model
• An In-Depth Walkthrough of PPO in NeMo RL
  • Quickstart: Launch a PPO Run
    • Data, Environments, Policy, and Generation
  • Value Model (Critic)
    • Deployment Architectures
    • Value Model Configuration
  • Generalized Advantage Estimation (GAE)
    • GAE Configuration
    • VAPO Decoupled GAE
    • Other Advantage Estimators
  • PPO Training Loop
    • Multiple Training Steps per Rollout
    • Critic Warmup
  • Loss
    • Policy Loss
    • Value Loss
  • Configuration
  • Metrics
  • Evaluate the Trained Model
  • References
• GRPO on DeepScaler
  • Train the Model
  • Training Curve
  • Evaluate the Model
  • Evaluation Results
• Solve a Sliding Puzzle Using GRPO
  • Quick Start Guide
    • 1. Install and Set Up NeMo RL with Megatron Backend (Optional)
    • 2. Train a Model
    • 3. Customize Puzzle Configuration
    • 4. Monitor Progress
  • Game Mechanics
    • Puzzle Structure
    • Example Data Sample
    • Movement Rules
  • Data Generation
    • Configuration Parameters
      • Description
      • Generation Algorithm
    • Dataset Size Calculation
    • Data Structure
  • Environment Interface
    • Core Classes
      • SlidingPuzzleEnv
      • SlidingPuzzleGameLogic
      • SlidingPuzzleRunner
    • Processing Pipeline
  • Reward System
    • Reward Structure
    • Reward Calculation Logic
  • Results
• Audio Post-training with NeMo RL
  • 1. Datasets
    • AVQA
    • AudioMCQ-StrongAC-GeminiCoT
  • 2. Train
    • 3B (Qwen2.5-Omni-3B) — audio_grpo_3B_megatron.yaml
    • 7B (Qwen2.5-Omni-7B) — vlm_grpo-qwen2.5-omni-7b-audiomcq-1n8g-megatron.v1.yaml
    • 30B MoE (Qwen3-Omni-30B-A3B-Instruct) — vlm_grpo-qwen3-omni-30ba3b-audiomcq-4n8g-megatron.v1.yaml
  • 3. Convert checkpoint (Megatron → HF)
  • 4. Evaluate on MMAU
  • 5. Results
    • 3B + AVQA (R1-AQA reproduction)
    • 7B + AudioMCQ
• Audio-Visual GRPO with Qwen2.5-Omni-7B
  • 0. Install Audio/Video Dependencies
  • 1. Train the Model
    • 7B training notes
  • 2. Convert Checkpoint (Megatron to HF)
  • 3. Evaluate
  • 4. Results
• Reward Model Training in NeMo RL
  • Launch a Training Job
  • Datasets
    • Custom datasets defined outside NeMo RL
  • Using Reward Models as Environments
    • Reward Model Environment
    • Integration with GRPO
    • Configuration
• Environments for GRPO Training
  • Math Environment
    • Key Features
    • Usage
    • Multi-reward support (GDPO)
  • Code Environment
    • Usage
    • Configuration
  • Code Jaccard Environment
    • How It Works
    • Usage
    • Configuration
    • Sample GRPO Config
  • Reward Model Environment
    • Usage
    • Resource Allocation in GRPO Training
      • Single-Node Setup (num_nodes: 1)
      • Multi-Node Setup (num_nodes > 1)
    • Complete GRPO Training with Reward Model Environments
  • Registering Custom Environments
    • Using the register_env Interface
    • Example: Registering a Custom Environment
• Evaluation
  • Prepare for Evaluation
    • Convert DCP to HF (Optional)
    • Prepare the Evaluation Configuration
  • Run the Evaluation Script
  • Example Evaluation Output
  • List of currently supported benchmarks
• DeepSeek-V3
  • Create BF16 Hugging Face checkpoint
• Model Guides
  • Families
• Qwen
  • Version Guides
• Qwen3.5
  • When to Use This Page
  • Support Status
  • What’s Supported
  • Example Recipes
  • Choose a Recipe
    • Small LLM Smoke Run
    • 35B-A3B GRPO (Megatron or AutoModel)
    • Long-Reasoning GRPO (Ready Out of the Box)
      • 100-Step Long-Run Results
    • Large MoE (397B-A17B)
    • VLM (Geo3K)
  • flash-linear-attention Performance
• Model Quirks
  • Gemma-3
    • vLLM Initialization
    • vLLM V1 runtime
    • Context Parallel with FSDP2
  • DeepScaleR Recipe Convergence Issues
  • vLLM Async Rollout Timeout
  • AutoModel Parameter Freezing (freeze_config)
• Train with Async GRPO
  • Configure Async GRPO
    • Enable Async GRPO
    • Complete Example Config
  • Implementation Structure
    • Core Components
      • 1. Main Training Loop (async_grpo_train in grpo.py)
      • 2. Async Trajectory Collector (AsyncTrajectoryCollector in async_utils/trajectory_collector.py)
      • 3. Replay Buffer (ReplayBuffer in async_utils/replay_buffer.py)
    • Weight Version Tracking
    • Coordination Flow
    • Architecture Diagram
  • Checkpointing
    • What is saved
    • Restore behaviour
    • Gap-filling after restore
    • Disabling replay-buffer restore
  • Usage Tips
  • Why Importance Sampling Correction Is Required for Async
    • The GRPO Objective
• Quantization-Aware RL (QARL)
  • Overview
  • Verified Configurations
  • ModelOpt Layer Spec Toggle
  • Quantization-Aware GRPO (QA-GRPO)
    • Configuration
    • Running QA-GRPO
  • Real-Quant NVFP4 Rollout (W4A4 and W4A16)
    • Minimal Configuration
    • Running the Example
    • Checkpoints and Fresh Starts
    • Log Checks
    • Troubleshooting
  • Quantization-Aware Distillation (On-Policy QAD)
    • Configuration
    • Running QAD
  • Quantization Parameters
  • Megatron Checkpoint Directory
  • Differences from FP8 Training
  • Supported Quantization Formats
  • Exporting Megatron Checkpoints
  • Limitations
• Train with Eagle3 Speculative Decoding
  • Offline vs Online
  • Draft Checkpoint
  • Enablement
    • Generation Only
    • Online Draft Training
  • How It Works
    • Rollout Path
    • Training Path
    • Draft Loss and Time-Step Alignment
  • Important Knobs
  • Notes
• YaRN Long-Context Training
  • Requirements
  • Enablement
    • Required Fields
  • How Conversion Works
  • Forcing a Fresh Conversion
  • Example Recipes
  • Practical Tips
• Cross-Tokenizer (X-Token) Off-Policy Distillation
  • How it works
  • Quickstart — single command
  • Backend and scope
  • Step 1 — Build multi-token mappings
  • Step 2 — Reapply exact-token map
  • Step 3 — Sort and trim to runtime top_k
  • Step 4 — Launch x-token distillation
    • Loss-mode knobs
  • Results — 100-step multi-teacher run
    • Downstream evaluation
    • Throughput and memory
  • Where files live
  • Related
• Weight Refit: Choosing a Transport
  • Pick a Transport
  • Constraints
  • Minimal Configuration
  • Learn More
• Checkpoint-Engine Refit
  • Enable NIXL
  • Runtime Setup
  • Performance Notes
    • FP8 Compatibility
    • DeepSeek-V3 Benchmark
  • Fault Tolerance Boundary
  • Verify
• Router Replay
  • Configuration
  • Validation
    • Validation and Trace Debugging
    • Effectiveness Check
  • Other Notes
    • Fallback for Missing Routes
• Muon Optimizer
  • What is Muon?
  • Requirements
  • Basic Usage
  • Example YAML Configuration
  • Experimental Results
• DTensor Tensor Parallel Accuracy Issue
  • Observed Accuracy Issues Under Tensor Parallelism with DTensor Backend
    • Misalignment between Training and Inference for Policy Models
    • Discrepancies Across TP Configurations in Reward Modeling
    • Overall Performance Degradation Under Tensor Parallelism
  • Batch-Variant Kernels
    • Root Cause
    • Recommended Solutions
  • Row-Wise TP Plan
    • Root Cause
    • Recommended Solutions:
  • Additional Observations and Insights
• Fault Tolerance Launcher Guide
  • Key Arguments
  • Basic Usage
  • FT Config File (examples/ft_launcher/ft_config.yaml)
  • Important Notes

Containers

• Build Docker Images
  • Building the Release Image
  • Skipping vLLM or SGLang Dependencies
  • Custom Setup Commands
    • Override with a custom script
    • Skip custom setup entirely

Development

• CI/CD
  • Test Levels
  • Triggering CI on Pull Requests
  • Required Checks
  • CI Pipeline Architecture
  • Code Review
  • Nightly Runs
  • Release Process
  • Infrastructure
  • Workflow Reference
• Test NeMo RL
  • Unit Tests
    • Experimental: Faster Local Test Iteration with pytest-testmon
    • Refreshing Remote-Selection Artifacts
    • Run Unit Tests in a Hermetic Environment
    • Track Metrics in Unit Tests
  • Functional Tests
    • Run Functional Tests in a Hermetic Environment
  • Bisecting Failing Tests
    • Bisecting Unit/Functional Tests
    • Bisecting Nightlies
• Documentation Development
  • Build the Documentation
  • Checking for Broken Links
  • Live Building
  • Run Tests in Python Docstrings
  • Write Tests in Python Docstrings
  • Documentation Version
• Debug NeMo RL Applications
  • Debug Worker/Actors on SLURM
    • Prerequisites
    • Forward a Port from the Head Node
    • Open the Ray Debugger Extension
    • Add the Ray Cluster
    • Add a Breakpoint and Run Your Program
  • Debug with legacy Ray debugger
• Profile GPU with Nsys
  • Prerequisites
  • Configure the Environment Variables
    • Extra Nsys Options (Optional)
    • Pattern Format
    • Supported Workers
  • Example Usage
    • Profile Only Policy Workers
    • Profile Multiple Worker Types
    • Profile Workers with Exact Names
    • Profile Megatron Workers
    • Profile TensorRT-LLM Generation Workers
  • Profile Output
  • Analyze Profile Files
    • How to Analyze the End-to-End RL Loop All at Once
  • How We Patched Nsight Support in Ray
    • The Problem
    • The Fix
    • Where We Applied the Patch
    • Why We Needed Both Locations
• FP8 Quantization in NeMo RL
  • Supported Features
    • FP8 Generation
    • FP8 Training
    • Recommended recipe
  • Integration with NeMo RL
    • RayDistributedExecutor
    • Quantization Utilities
    • Weight Loading
  • Usage
  • Compatibility Note for DeepSeek-Style FP8 Training
  • Accuracy
• Experiment with Custom vLLM
  • Clone and Build Your Custom vLLM
  • Verify Your Custom vLLM in Isolation
  • Running NeMo RL Apps with Custom vLLM
  • Re-building the NeMo RL Docker Image
    • SSH Setup for Private Repositories
      • Prerequisites
      • Step 1: Verify your SSH key works
      • Step 2: Load your SSH key into the agent
      • Step 3: Run the Docker build with SSH forwarding
  • Running Applications with a Custom vLLM Container
    • Why Not Use uv run or Rebuild Venvs?

Design Docs

• Design and Philosophy
  • Motivation
  • Design
    • RayVirtualCluster
    • RayWorkerGroup
    • Single-Controller and Execution Diagram
    • Configuration Schema: BaseModel, dataclass, and TypedDict
• Padding in NeMo RL
  • Padding Approach
  • Right-Padded Generation Example
  • Verify Right Padding
  • Best Practices
• Logger
  • Requirements
  • Overall Design
  • Supported Logging Backends
    • WandB (Weights & Biases)
    • Swanlab
    • Tensorboard
    • MLflow
      • MLflow Configuration
      • MLflow UI
  • Validation Pretty Logging
    • Example Output
  • GPU Metric Logging
• uv in NeMo RL
  • Overview
  • Why uv?
    • Speed and Efficiency
    • Isolated Environments
    • Dependency Management in Ray Clusters
    • Container-Free Flexibility
  • Implementation in NeMo RL
    • Worker Configuration
    • Supported Python Executables
    • Customization
  • How It Works
  • Conclusion
• Dependency Management
  • Workflows Overview
    • Production Workflow
    • Development Workflow
  • How uv run Works
    • 1. uv lock
    • 2. uv sync
    • 3. source .venv/bin/activate
    • 4. python examples/run_grpo.py
  • Multi-Environment Architecture
    • Worker Groups and Virtual Environments
  • Container Pre-caching
    • When Pre-cached Environments Are Sufficient
  • Handling Dependency Changes
    • Option 1: Force Rebuild Environments
    • Option 2: Rebuild the Container
    • Option 3: Classic Workflow - Mounting Modified Submodules
  • Decision Guide
  • Frozen Environments
    • What Are Frozen Environments?
    • When to Use Frozen Environments
    • Available Python Executables
    • Container Version Checking
  • Summary
• Data Format
  • Hugging Face Chat Datasets
  • Chat Templates
• Generation Interface
  • Generation Interface
    • Key Components
  • Generation Backends
  • VLLM Backend
    • VllmGeneration Class
    • VllmGenerationWorker
    • Custom VLLM Extensions
  • Megatron Backend
    • Key Features
    • MegatronPolicyWorker Generation
    • Configuration
    • Configuration Parameters
  • Usage Examples
    • Using VLLM Backend
    • Using Megatron Backend
  • Extend with New Backends
• Remote Sparse-Delta vLLM Refit
  • Supported scope
  • Architecture
  • Protocol
    • Baseline and ownership
    • Compare and encode
    • Transport and apply
  • Payload and native apply
  • Configuration
  • Metrics and profiling
  • Development and validation
  • Refit bandwidth calculator
  • Failure guide
• Checkpoint Engine Design
  • Goals
  • Control Flow
  • Configuration Contract
  • Backend Interface
  • Worker Integration
  • NIXL Backend
    • Validated Full-Model Layout
  • NIXL Preinit
  • How NIXL Supports Fault Tolerance
  • Adding Another Backend
• Exporting Checkpoints to Hugging Face Format
  • Converting Torch Distributed Checkpoints to Hugging Face Format
  • Converting Megatron Checkpoints to Hugging Face Format
  • Converting Megatron LoRA Adapter Checkpoints to Hugging Face Format
    • Option A — Merged checkpoint
    • Option B — Adapter-only (PEFT format)
    • Arguments
• Loss functions in NeMo RL
• FSDP2 Parallel Plan
  • Fallback Priority
    • 1. Custom Parallel Plan
    • 2. Optimized Parallel Plan
    • 3. Hugging Face Tensor Parallel Plan
  • Custom Parallel Plan Example
  • HSDP (dp_replicate_size)
    • Illustration
    • Configuration
    • When to use it
• Training Backends
  • Available Backends
  • Supported Input Checkpoint Format
  • Backend Selection
    • Megatron Backend
    • DTensor Backend
  • Configuration Examples
  • Megatron Configuration
    • Environment Variable Priority (Highest to Lowest)
    • Configuration Examples
    • Best Practices
    • Force Reconvert
• Sequence Packing and Dynamic Batching
  • Table of Contents
  • Problem
    • Challenge: Variable Sequence Lengths in RL/SFT
  • Sequence Packing and Dynamic Batching
    • Important Notes
  • Sequence Packing
    • Implementation Details
      • 1. Packing Process (BatchedDataDict.shard_by_batch_size)
      • 2. Packing Algorithms (nemo_rl/data/packing/algorithms.py)
        • Concatenative Packer
        • Modified First Fit Decreasing (MFFD)
        • First Fit Decreasing (FFD)
        • First Fit Shuffle
    • Usage with Context Parallelism
      • Understanding CP Load balancing:
    • Nuances
  • Dynamic Batching
    • Architecture
      • Processing Pipeline
      • Implementation Details
    • Interface
    • Nuances
  • Configuration
    • Dynamic Batching Configuration
    • Sequence Packing Configuration
  • Integration with Training Pipeline
    • Loss Function Integration
  • Metrics and Monitoring
    • Packing Efficiency Metrics (nemo_rl/data/packing/metrics.py)
  • Usage
    • Sequence Packing Configuration
    • Dynamic Batching Configuration
    • Framework Compatibility
  • References
• Environment Variable Precedence in NeMo RL
  • Precedence Order
    • 1. Ray Runtime Environment Variables (lowest)
    • 2. System-level Environment Variables (medium)
    • 3. YAML Configuration env_vars (high)
    • 4. Worker-specific configure_worker Method (highest)
• NeMo Gym Integration
  • Overview
  • Configuration
    • Version Requirements
  • Architecture Overview
  • The NemoGym Actor
  • vLLM HTTP Server
  • Initialization Sequence
  • Training Loop Control Flow
    • Key Steps
    • Async Result Processing
    • Async GRPO Collector Invariants
  • Data Format Translation
  • Tokenization and On-Policy Corrections
• ModelOpt Real-Quant Refit Architecture
  • Relationship to the overall NeMo RL design
  • Component responsibilities
  • Quantization modes
  • Refit lifecycle
  • vLLM compatibility adapters
  • Fused-MoE transport
  • CUDA IPC buffer lifetime
  • KV-cache behavior
  • Configuration
  • Current limitations
• NCCL Reshard Refit (Experimental)
  • Enabling It
  • Design Overview
    • Execution Flow: Setup Time
    • Execution Flow: Refit Time
    • The Misc Path
  • Decoupling Backend-Agnostic Parts and Backend-Dependent Parts
  • xferdtensor Transports
  • Expected Performance

API Reference

• API Reference
  • nemo_rl
    • Subpackages
      • nemo_rl.models
        • Subpackages
          • nemo_rl.models.value
            • Subpackages
              • nemo_rl.models.value.workers
                • Submodules
                  • nemo_rl.models.value.workers.dtensor_value_worker_v2
                    • Module Contents
                      • Classes
                      • Functions
                      • API
                        • right_shift_values()
                        • RightShiftLossWrapper
                          • __call__()
                          • __getattr__()
                        • get_train_context()
                        • DTensorValueWorkerV2Impl
                          • __repr__()
                          • train()
                          • get_values()
                          • prepare_for_training()
                          • prepare_for_inference()
                          • finish_training()
                          • finish_inference()
                          • move_optimizer_to_device()
                          • move_to_device()
                          • move_buffer_to_device()
                          • move_to_cuda()
                          • move_to_cpu()
                          • save_checkpoint()
                          • load_checkpoint()
                          • _init_checkpoint_manager()
                        • DTensorValueWorkerV2
                  • nemo_rl.models.value.workers.megatron_value_worker
                    • Module Contents
                      • Classes
                      • Functions
                      • Data
                      • API
                        • TokenizerType
                        • _install_value_head_load_skip()
                        • make_value_head_hook()
                        • _unpack_value_sequences()
                        • _value_loss_prepare_fn()
                        • MegatronValueWorkerImpl
                          • __repr__()
                          • configure_worker()
                          • enable_forward_pre_hook()
                          • disable_forward_pre_hook()
                          • train()
                          • get_values()
                          • prepare_for_training()
                          • prepare_for_inference()
                          • move_model()
                          • move_optimizer()
                          • save_checkpoint()
                          • load_checkpoint()
                          • finish_inference()
                          • finish_training()
                        • MegatronValueWorker
            • Submodules
              • nemo_rl.models.value.interfaces
                • Module Contents
                  • Classes
                  • API
                    • ValueOutputSpec
                      • values
                    • ValueInterface
                      • get_values()
                      • train()
                      • prepare_for_training()
                      • prepare_for_inference()
                      • finish_training()
                      • save_checkpoint()
                      • shutdown()
              • nemo_rl.models.value.config
                • Module Contents
                  • Classes
                  • API
                    • ValueConfig
                      • model_name
                      • tokenizer
                      • train_global_batch_size
                      • train_micro_batch_size
                      • logprob_batch_size
                      • precision
                      • reward_model_cfg
                      • dtensor_cfg
                      • megatron_cfg
                      • hf_config_overrides
                      • dynamic_batching
                      • sequence_packing
                      • make_sequence_length_divisible_by
                      • max_total_sequence_length
                      • max_grad_norm
                      • dequantize_base_checkpoint
                      • optimizer
                      • scheduler
              • nemo_rl.models.value.lm_value
                • Module Contents
                  • Classes
                  • Data
                  • API
                    • PathLike
                    • Value
                      • get_values()
                      • train()
                      • prepare_for_training()
                      • prepare_for_inference()
                      • finish_inference()
                      • finish_training()
                      • save_checkpoint()
                      • shutdown()
                      • __del__()
            • Package Contents
              • Data
              • API
                • __all__
          • nemo_rl.models.automodel
            • Submodules
              • nemo_rl.models.automodel.data
                • Module Contents
                  • Classes
                  • Functions
                  • API
                    • _accepted_forward_kwargs()
                    • _all_image_sizes_equal()
                    • filter_multimodal_kwargs_for_model()
                    • ProcessedInputs
                      • input_ids
                      • seq_len
                      • attention_mask
                      • position_ids
                      • flash_attn_kwargs
                      • vlm_kwargs
                      • cp_buffers
                      • seq_index
                      • has_context_parallel
                      • has_flash_attention
                      • is_multimodal
                    • ProcessedMicrobatch
                      • data_dict
                      • processed_inputs
                      • original_batch_size
                      • original_seq_len
                    • make_processed_microbatch_iterator()
                    • get_microbatch_iterator()
                    • process_microbatch()
                    • process_global_batch()
                    • check_sequence_dim()
              • nemo_rl.models.automodel.checkpoint
                • Module Contents
                  • Classes
                  • Functions
                  • API
                    • _patch_qwen_vl_vision_key_mapping()
                    • AutomodelCheckpointManager
                      • _get_dp_rank()
                      • _get_tp_rank()
                      • init_checkpointer()
                      • update_checkpointer_config()
                      • _rebuild_checkpointer_addons()
                      • save_checkpoint()
                      • load_checkpoint()
                    • detect_checkpoint_format()
                    • _infer_checkpoint_root()
              • nemo_rl.models.automodel.train
                • Module Contents
                  • Classes
                  • Functions
                  • Data
                  • API
                    • PostProcessingFunction
                    • _needs_kv_cache_for_shared_layers()
                    • model_forward()
                    • extract_logits()
                    • apply_temperature_scaling()
                    • apply_top_k_top_p_filtering_for_local_logits()
                    • redistribute_logits_for_cp()
                    • prepare_data_for_cp()
                    • forward_with_post_processing_fn()
                    • automodel_forward_backward()
                    • LossPostProcessor
                      • __call__()
                    • LogprobsPostProcessor
                      • __call__()
                      • _compute_local_logprobs()
                    • TopkLogitsPostProcessor
                      • __call__()
                    • FullLogitsPostProcessor
                      • __call__()
                    • ScorePostProcessor
                      • __call__()
                    • aggregate_training_statistics()
              • nemo_rl.models.automodel.config
                • Module Contents
                  • Classes
                  • API
                    • DistributedContext
                      • device_mesh
                      • moe_mesh
                      • fsdp2_config
                      • moe_config
                      • dp_size
                      • tp_size
                      • cp_size
                    • RuntimeConfig
                      • model_class
                      • model_config
                      • hf_config_overrides
                      • allow_flash_attn_args
                      • attn_impl
                      • dtype
                      • enable_seq_packing
                      • max_grad_norm
                      • cpu_offload
                      • offload_optimizer_for_logprob
                      • is_generation_colocated
                      • sampling_params
                      • is_reward_model
                    • ModelAndOptimizerState
                      • model
                      • optimizer
                      • scheduler
                      • is_hf_model
                      • is_moe_model
                      • is_reward_model
                      • model_class
                      • model_config
                      • peft_config
                      • autocast_enabled
              • nemo_rl.models.automodel.setup
                • Module Contents
                  • Functions
                  • Data
                  • API
                    • STRING_TO_DTYPE
                    • _maybe_set_force_hf()
                    • get_tokenizer()
                    • validate_and_prepare_config()
                    • setup_reference_model_state()
                    • setup_distributed()
                    • _disable_automodel_checkpoint_dtype_restore()
                    • setup_model_and_optimizer()
          • nemo_rl.models.megatron
            • Subpackages
              • nemo_rl.models.megatron.draft
                • Submodules
                  • nemo_rl.models.megatron.draft.eagle
                    • Module Contents
                      • Classes
                      • API
                        • EagleModel
                          • sharded_state_dict()
                          • forward()
                  • nemo_rl.models.megatron.draft.hidden_capture
                    • Module Contents
                      • Classes
                      • Functions
                      • Data
                      • API
                        • get_eagle3_aux_hidden_state_layers()
                        • _DTYPE_TO_CODE
                        • _CODE_TO_DTYPE
                        • CapturedStates
                          • hidden_states
                          • inputs_embeds
                        • HiddenStateCapture
                          • _compute_local_layer_mapping()
                          • _compute_layer_owner_map()
                          • _make_layer_output_hook()
                          • _make_embedding_hook()
                          • register_hooks()
                          • clear_hooks()
                          • capture_context()
                          • _assemble_local_states()
                          • _owner_rank_for_global_layer()
                          • _send_tensor()
                          • _recv_tensor()
                          • _gather_distributed()
                          • get_captured_states()
                        • get_capture_context()
                  • nemo_rl.models.megatron.draft.utils
                    • Module Contents
                      • Classes
                      • Functions
                      • Data
                      • API
                        • StateDict
                        • CheckpointLoader
                        • _CHECKPOINT_CANDIDATE_NAMES
                        • _HF_SNAPSHOT_ALLOW_PATTERNS
                        • _HF_SNAPSHOT_IGNORE_PATTERNS
                        • _MODEL_LAYER_QKV_KEY_PATTERN
                        • _CHECKPOINT_LAYER_KEY_PATTERN
                        • _EagleLayerLayout
                          • layer_index
                          • model_prefix
                          • checkpoint_prefix
                          • hidden_norm_key
                          • input_layernorm_key
                          • post_attention_layernorm_key
                          • qkv_weight_key
                          • proj_weight_key
                          • fc1_weight_key
                          • fc2_weight_key
                        • _resolve_optional_key()
                        • _EagleModelLayout
                          • layers
                          • final_norm_key
                          • lm_head_key
                          • detect()
                          • layer_by_index
                        • _qkv_head_dims()
                        • _interleave_qkv()
                        • _deinterleave_qkv()
                        • _combine_or_shard_weight_parts()
                        • _PendingLayerWeights
                          • qkv_weight
                          • q_weight
                          • k_weight
                          • v_weight
                          • fc1_weight
                          • gate_weight
                          • up_weight
                          • apply_to()
                        • _get_num_aux_hidden_states()
                        • _all_gather_tp_shards()
                        • _gather_tp_qkv_weight()
                        • _gather_tp_gate_up_weight()
                        • _gather_tp_weight_if_needed()
                        • _extract_tensor_state_dict()
                        • _load_safetensors_file()
                        • _load_torch_file()
                        • _merge_checkpoint_shards()
                        • _load_index_checkpoint()
                        • _load_checkpoint_file()
                        • _load_checkpoint_from_directory()
                        • _load_checkpoint_state()
                        • _normalize_hf_key()
                        • _parse_layer_checkpoint_key()
                        • _get_tp_rank()
                        • _build_split_axis_by_parameter()
                        • _shard_to_local_tp()
                        • _assign_optional_layer_weight()
                        • _map_layer_hf_weight()
                        • _map_hf_state_to_eagle_state()
                        • load_hf_weights_to_eagle()
                        • _require_state_tensor()
                        • find_draft_owner_chunk()
                        • get_attached_draft_model()
                        • _export_layer_weights_to_hf()
                        • export_eagle_weights_to_hf()
                        • get_policy_lm_head_weight()
                        • _get_draft_output_layer()
                        • _get_draft_to_target_token_mapping()
                        • copy_policy_lm_head_to_draft()
                        • DRAFT_GRAD_NORM_GROUP
                        • register_draft_grad_norm_group()
                        • build_draft_model()
                • Package Contents
                  • Data
                  • API
                    • __all__
            • Submodules
              • nemo_rl.models.megatron.common
                • Module Contents
                  • Functions
                  • API
                    • _round_up_to_multiple()
                    • broadcast_tensor()
                    • get_moe_metrics()
                    • get_mtp_metrics()
              • nemo_rl.models.megatron.data
                • Module Contents
                  • Classes
                  • Functions
                  • API
                    • ProcessedInputs
                      • input_ids
                      • input_ids_cp_sharded
                      • attention_mask
                      • position_ids
                      • packed_seq_params
                      • cu_seqlens_padded
                      • mtp_loss_mask
                      • routed_experts
                      • routed_experts_cp_sharded
                    • ProcessedMicrobatch
                      • data_dict
                      • input_ids
                      • input_ids_cp_sharded
                      • attention_mask
                      • position_ids
                      • packed_seq_params
                      • cu_seqlens_padded
                      • mtp_loss_mask
                      • routed_experts
                      • routed_experts_cp_sharded
                    • make_processed_microbatch_iterator()
                    • get_microbatch_iterator()
                    • get_ltor_masks_and_position_ids()
                    • process_microbatch()
                    • _make_r3_trace_token_identity()
                    • _verify_r3_trace_cp_token_alignment()
                    • _fill_routed_experts_padding()
                    • process_global_batch()
                    • _prepare_vlm_batch_for_megatron()
                    • _pack_sequences_for_megatron()
                    • _shard_routed_experts_for_cp()
                    • _get_pack_sequence_parameters_for_megatron()
                    • _unpack_sequences_from_megatron()
                    • get_and_validate_seqlen()
              • nemo_rl.models.megatron.community_import
                • Module Contents
                  • Functions
                  • API
                    • to_torch_dtype()
                    • _prefer_nvrx_for_dist_ckpt_save()
                    • import_model_from_hf_name()
                    • export_model_from_megatron()
              • nemo_rl.models.megatron.train
                • Module Contents
                  • Classes
                  • Functions
                  • Data
                  • API
                    • PostProcessingFunction
                    • model_forward()
                    • apply_temperature_scaling()
                    • forward_with_post_processing_fn()
                    • megatron_forward_backward()
                    • LossPostProcessor
                      • __call__()
                    • LogprobsPostProcessor
                      • __call__()
                    • TopkLogitsPostProcessor
                      • __call__()
                    • aggregate_training_statistics()
              • nemo_rl.models.megatron.config
                • Module Contents
                  • Classes
                  • API
                    • RuntimeConfig
                      • megatron_cfg
                      • model_cfg
                      • dtype
                      • optimizer_cpu_offload
                      • offload_optimizer_for_logprob
                      • is_generation_colocated
                      • sampling_params
                      • final_padded_vocab_size
                    • ModelAndOptimizerState
                      • state
                      • model
                      • optimizer
                      • scheduler
                      • checkpointing_context
                      • param_sync_func
                      • draft_model
              • nemo_rl.models.megatron.setup
                • Module Contents
                  • Classes
                  • Functions
                  • Data
                  • API
                    • _HF_CONFIG_PATCHED
                    • _NEMOTRON_OMNI_EXPANDED_SEQUENCE_CONTRACT
                    • _patch_hf_config_double_instantiation()
                    • _force_sync_optimizer_fp32_from_model()
                    • TokenizerType
                    • destroy_parallel_state()
                    • setup_distributed()
                    • validate_and_set_config()
                    • _canonicalize_hf_config_overrides()
                    • _get_hf_config_overrides_hash()
                    • _resolve_iter_dir_from_root()
                    • validate_model_paths()
                    • setup_model_config()
                    • _apply_parallelism_config()
                    • _apply_moe_config()
                    • _apply_mtp_config()
                    • _apply_precision_config()
                    • _apply_performance_config()
                    • _validate_optimizer_config()
                    • _validate_chunking_config()
                    • _create_checkpoint_config()
                    • _validate_training_config()
                    • _validate_dtype_config()
                    • _create_megatron_config()
                    • _create_draft_pre_wrap_hook()
                    • _BRIDGE_SIGNAL_HANDLER_PATCHED
                    • _patch_bridge_signal_handler_for_worker_threads()
                    • setup_model_and_optimizer()
                    • handle_model_import()
                    • setup_reference_model_state()
                    • finalize_megatron_setup()
                    • MoEFloat16Module
                      • re_enable_float32_expert_bias()
                    • make_policy_like_config()
              • nemo_rl.models.megatron.router_replay
                • Module Contents
                  • Functions
                  • Data
                  • API
                    • _ROUTER_REPLAY_VALIDATE_ENV
                    • _MISSING_ROUTE_SENTINEL
                    • _MISSING_ROUTE_FALLBACK_PATCH_ATTR
                    • router_replay_enabled()
                    • configure_vllm_for_router_replay()
                    • validate_router_replay_config()
                    • _iter_model_modules()
                    • _unwrap_model_config()
                    • _global_moe_layer_numbers()
                    • _router_replay_instances_for_model()
                    • _local_layer_numbers_for_model()
                    • _normalize_routed_experts_for_mcore()
                    • _payload_indices_for_moe_layers()
                    • _router_replay_validation_enabled()
                    • _validate_replay_tensor()
                    • _install_missing_route_fallback_patch()
                    • _get_tensor_model_parallel_world_size()
                    • _get_tensor_model_parallel_rank()
                    • _split_for_sequence_parallel()
                    • build_router_replay_tensors()
                    • build_router_replay_assignments()
                    • set_router_replay_forward()
                    • set_router_replay_backward()
                    • clear_router_replay()
                    • clear_global_router_replay_instances()
              • nemo_rl.models.megatron.pipeline_parallel
                • Module Contents
                  • Functions
                  • API
                    • broadcast_obj_from_pp_rank()
                    • broadcast_loss_metrics_from_last_stage()
                    • broadcast_tensors_from_last_stage()
          • nemo_rl.models.huggingface
            • Submodules
              • nemo_rl.models.huggingface.common
                • Module Contents
                  • Classes
                  • Functions
                  • Data
                  • API
                    • Tensor
                    • FlashAttentionKwargs
                      • cu_seqlens_q
                      • cu_seqlens_k
                      • max_seqlen_q
                      • max_seqlen_k
                    • ModelFlag
                      • VLLM_LOAD_FORMAT_AUTO
                      • matches()
                    • is_gemma_model()
                    • is_nano_nemotron_vl_model()
                    • group_and_cat_tensors()
                    • pack_sequences()
                    • unpack_tensor()
                    • get_flash_attention_kwargs()
          • nemo_rl.models.policy
            • Subpackages
              • nemo_rl.models.policy.workers
                • Submodules
                  • nemo_rl.models.policy.workers.megatron_remote_sparse_refit
                    • Module Contents
                      • Classes
                      • API
                        • MegatronRemoteSparseRefit
                          • initialize_baseline()
                          • stream()
                          • finish()
                  • nemo_rl.models.policy.workers.checkpoint_engine
                    • Module Contents
                      • Classes
                      • Functions
                      • API
                        • maybe_preinit_nixl_checkpoint_engine()
                        • DTensorCheckpointEngineSendMixin
                          • model
                          • _prepare_checkpoint_engine_weight_send()
                          • _finalize_checkpoint_engine_weight_send()
                          • _checkpoint_engine_weight_iterator()
                        • MegatronCheckpointEngineSendMixin
                          • _checkpoint_engine_weight_iterator()
                        • PolicyCheckpointEngineMixin
                          • checkpoint_engine
                          • rank
                          • _checkpoint_engine_weight_iterator()
                          • _prepare_checkpoint_engine_weight_send()
                          • _finalize_checkpoint_engine_weight_send()
                          • send_weights_via_checkpoint_engine()
                          • checkpoint_engine_rpc()
                  • nemo_rl.models.policy.workers.dtensor_policy_worker
                    • Module Contents
                      • Classes
                      • Functions
                      • API
                        • dtensor_params_generator()
                        • _attach_context_parallel_hooks()
                        • unshard_fsdp2_model()
                        • get_cpu_state_dict()
                        • DTensorPolicyWorkerImpl
                          • __repr__()
                          • _get_replica_group()
                          • _local_coords()
                          • create_context_parallel_ctx()
                          • _apply_temperature_scaling()
                          • _apply_top_k_top_p_filtering()
                          • train_context()
                          • train()
                          • get_logprobs()
                          • score()
                          • get_topk_logits()
                          • use_reference_model()
                          • _add_noise_to_weights()
                          • return_state_dict()
                          • return_model_config()
                          • prepare_refit_info()
                          • calibrate_qkv_fp8_scales()
                          • stream_weights_via_ipc_zmq()
                          • _checkpoint_engine_params()
                          • broadcast_weights_for_collective()
                          • prepare_for_lp_inference()
                          • prepare_for_training()
                          • offload_before_refit()
                          • offload_after_refit()
                          • move_optimizer_to_device()
                          • move_to_device()
                          • move_buffer_to_device()
                          • move_to_cuda()
                          • move_to_cpu()
                          • save_checkpoint()
                          • load_checkpoint()
                        • DTensorPolicyWorker
                  • nemo_rl.models.policy.workers.dtensor_policy_worker_v2
                    • Module Contents
                      • Classes
                      • Functions
                      • API
                        • dtensor_params_generator()
                        • _maybe_merge_lora_weight()
                        • _maybe_adapt_tensor_to_hf()
                        • get_train_context()
                        • DTensorPolicyWorkerV2Impl
                          • __repr__()
                          • _get_replica_group()
                          • _local_coords()
                          • _update_moe_gate_bias_if_supported()
                          • set_rollout_num_gpus_per_engine()
                          • train()
                          • get_logprobs()
                          • score()
                          • get_topk_logits()
                          • get_full_logits_ipc()
                          • release_ipc_buffer()
                          • use_reference_model()
                          • _add_noise_to_weights()
                          • return_state_dict()
                          • return_model_config()
                          • prepare_refit_info()
                          • calibrate_qkv_fp8_scales()
                          • stream_weights_via_ipc_zmq()
                          • stream_weights_via_http()
                          • _checkpoint_engine_params()
                          • broadcast_weights_for_collective()
                          • prepare_for_lp_inference()
                          • prepare_for_training()
                          • finish_inference()
                          • offload_before_refit()
                          • offload_after_refit()
                          • move_optimizer_to_device()
                          • move_to_device()
                          • move_buffer_to_device()
                          • move_to_cuda()
                          • move_to_cpu()
                          • save_checkpoint()
                          • load_checkpoint()
                          • _init_checkpoint_manager()
                        • DTensorPolicyWorkerV2
                  • nemo_rl.models.policy.workers.megatron_policy_worker
                    • Module Contents
                      • Classes
                      • Functions
                      • Data
                      • API
                        • log
                        • TokenizerType
                        • _should_use_router_replay()
                        • _model_self_packs_for_cp()
                        • _model_self_packs_mtp_loss_mask()
                        • _model_slices_context_parallel_inputs()
                        • _estimate_refit_tensor_size_in_bytes()
                        • _meta_tensor_alloc_context()
                        • MegatronPolicyWorkerImpl
                          • _train_step_state
                          • _remote_sparse_refit
                          • _async_checkpoint_cuda_cache_active
                          • __repr__()
                          • _local_coords()
                          • _get_replica_group()
                          • configure_worker()
                          • enable_forward_pre_hook()
                          • disable_forward_pre_hook()
                          • _forward_pre_hook_enabled()
                          • _disable_forward_pre_hook_until_next_train_step()
                          • _copy_main_params_to_param_buffer()
                          • _uses_mxfp8_overlap_shared_param_buffer()
                          • _get_model_extra_state_dict()
                          • _restore_model_extra_state_dict()
                          • train()
                          • _compute_moe_grad_scale()
                          • _set_moe_grad_scale_func()
                          • get_reference_policy_logprobs()
                          • _split_step_state_init()
                          • _assert_step_open()
                          • _restore_saved_grad_sync_func()
                          • begin_train_step()
                          • train_microbatch()
                          • _train_microbatch_body()
                          • finish_train_step()
                          • _finish_train_step_body()
                          • abort_train_step()
                          • get_logprobs()
                          • _apply_state_dict_to_model()
                          • use_reference_model()
                          • get_topk_logits()
                          • prepare_refit_info()
                          • _collect_mtp_metrics()
                          • _set_mtp_grad_scale_func()
                          • _get_model_config()
                          • init_remote_sparse_delta_baseline()
                          • stream_remote_sparse_weights()
                          • _require_remote_sparse_refit()
                          • finish_remote_sparse_delta_sync()
                          • _is_fp8_export()
                          • _build_refit_conversion_tasks()
                          • _calculate_refit_param_info()
                          • _iter_params_with_optional_kv_scales()
                          • _iter_local_hf_param_shards()
                          • stream_weights_via_ipc_zmq()
                          • broadcast_weights_for_collective()
                          • _build_layer_to_pp_stage()
                          • prepare_nccl_reshard_refit_info()
                          • _build_expert_groups()
                          • _group_experts()
                          • build_hf_to_local_param_map()
                          • nccl_reshard_refit()
                          • _broadcast_misc_params_packed()
                          • prepare_for_lp_inference()
                          • prepare_for_training()
                          • finish_inference()
                          • _clear_fp8_caches()
                          • offload_before_refit()
                          • offload_after_refit()
                          • move_model()
                          • move_optimizer()
                          • save_checkpoint()
                          • _requires_nvrx_cuda_cache_release()
                          • finalize_async_save()
                          • load_checkpoint()
                          • check_tensor_parallel_attributes()
                          • calibrate_qkv_fp8_scales()
                        • MegatronPolicyWorker
                  • nemo_rl.models.policy.workers.base_policy_worker
                    • Module Contents
                      • Classes
                      • API
                        • AbstractPolicyWorker
                          • init_collective()
                          • init_nccl_reshard_comm_group()
                          • prepare_nccl_reshard_refit_info()
                          • nccl_reshard_refit()
                          • is_alive()
                          • reset_peak_memory_stats()
                          • get_gpu_info()
                          • report_device_id()
                          • get_zmq_address()
                          • maybe_init_zmq()
                          • get_free_memory_bytes()
                          • shutdown()
                          • start_gpu_profiling()
                          • stop_gpu_profiling()
                          • report_node_ip_and_gpu_id()
                          • get_reference_policy_logprobs()
                          • finalize_async_save()
                          • finish_training()
                  • nemo_rl.models.policy.workers.patches
                    • Module Contents
                      • Functions
                      • API
                        • _get_transformer_engine_file()
                        • apply_transformer_engine_patch()
            • Submodules
              • nemo_rl.models.policy.interfaces
                • Module Contents
                  • Classes
                  • API
                    • LogprobOutputSpec
                      • logprobs
                    • ReferenceLogprobOutputSpec
                      • reference_logprobs
                    • ScoreOutputSpec
                      • scores
                    • TopkLogitsOutputSpec
                      • topk_logits
                      • topk_indices
                    • PolicyInterface
                      • get_logprobs()
                      • get_reference_policy_logprobs()
                      • get_topk_logits()
                      • train()
                      • calibrate_qkv_fp8_scales()
                      • prepare_for_training()
                      • finish_training()
                      • save_checkpoint()
                      • shutdown()
                    • ColocatablePolicyInterface
                      • init_collective()
                      • offload_before_refit()
                      • offload_after_refit()
                      • offload_to_cpu()
                      • prepare_refit_info()
                      • stream_weights_via_ipc_zmq()
                      • stream_weights_via_http()
                      • set_rollout_num_gpus_per_engine()
                      • broadcast_weights_for_collective()
                      • prepare_nccl_reshard_refit_info()
                      • nccl_reshard_refit()
                      • prepare_for_lp_inference()
              • nemo_rl.models.policy.teacher_worker_group
                • Module Contents
                  • Classes
                  • Functions
                  • API
                    • TeacherConfig
                      • alias
                      • model_name
                      • tensor_model_parallel_size
                      • pipeline_model_parallel_size
                      • context_parallel_size
                      • expert_model_parallel_size
                      • num_nodes
                      • gpus_per_node
                      • precision
                      • micro_batch_size
                      • megatron_cfg_overrides
                    • create_teacher_configs_from_opd_config()
                    • TeacherWorkerGroup
                      • get_logprobs()
                      • shutdown()
                      • __del__()
              • nemo_rl.models.policy.tq_policy
                • Module Contents
                  • Classes
                  • Functions
                  • API
                    • _aggregate_train_results()
                    • TQPolicy
                      • shutdown()
                      • prepare_step()
                      • prepare_val_partition()
                      • discard_samples()
                      • finish_step()
                      • _stamp_pad_seqlen()
                      • read_from_dataplane()
                      • write_to_dataplane()
                      • _packing_args()
                      • _logprob_dispatch()
                      • get_logprobs_from_meta()
                      • get_reference_policy_logprobs_from_meta()
                      • train_from_meta()
                      • begin_train_step()
                      • train_microbatches_from_meta()
                      • finish_train_step()
                      • abort_train_step()
              • nemo_rl.models.policy.lm_policy
                • Module Contents
                  • Classes
                  • Functions
                  • Data
                  • API
                    • PathLike
                    • _aggregate_megatron_flops_metrics()
                    • Policy
                      • data_parallel_size
                      • run_all_workers_single_data()
                      • run_all_workers_multiple_data()
                      • init_collective()
                      • init_collective_mcore_generation()
                      • preinit_nvshmem()
                      • swap_weights_via_reshard()
                      • _shard_for_logprob()
                      • _shard_for_train()
                      • get_logprobs()
                      • get_reference_policy_logprobs()
                      • get_topk_logits()
                      • get_full_logits_ipc()
                      • release_ipc_buffer()
                      • train()
                      • generate()
                      • score()
                      • prepare_for_generation()
                      • finish_generation()
                      • prepare_for_training()
                      • prepare_for_lp_inference()
                      • invalidate_kv_cache()
                      • prepare_refit_info()
                      • finish_inference()
                      • finish_training()
                      • calibrate_qkv_fp8_scales()
                      • get_free_memory_bytes()
                      • stream_weights_via_ipc_zmq()
                      • stream_weights_via_http()
                      • set_rollout_num_gpus_per_engine()
                      • broadcast_weights_for_collective()
                      • init_nccl_reshard_comm_group()
                      • prepare_nccl_reshard_refit_info()
                      • nccl_reshard_refit()
                      • offload_before_refit()
                      • offload_after_refit()
                      • offload_to_cpu()
                      • save_checkpoint()
                      • finalize_async_save()
                      • shutdown()
                      • __del__()
                      • start_gpu_profiling()
                      • stop_gpu_profiling()
                      • print_node_ip_and_gpu_id()
              • nemo_rl.models.policy.utils
                • Module Contents
                  • Classes
                  • Functions
                  • Data
                  • API
                    • AUTOMODEL_FACTORY
                    • IPCProtocol
                      • COMPLETE
                      • ACK
                    • POLICY_WORKER_OVERRIDES
                    • resolve_policy_worker_cls()
                    • resolve_model_class()
                    • is_vllm_v1_engine_enabled()
                    • get_gpu_info()
                    • configure_dynamo_cache()
                    • get_runtime_env_for_policy_worker()
                    • get_megatron_checkpoint_dir()
                    • get_handle_from_tensor()
                    • ensure_teacher_ipc_buffer()
                    • aggregate_per_sample_handles()
                    • calculate_aligned_size()
                    • stream_weights_via_ipc_zmq_impl()
                    • rebuild_cuda_tensor_from_ipc()
                    • _ensure_ipc_topology()
                    • _flush_bucket()
                    • stream_weights_via_http_impl()
            • Package Contents
              • Classes
              • Functions
              • Data
              • API
                • _patch_transformers_tokenizer_class_set()
                • LoRAConfigDisabled
                  • enabled
                • LoRAConfig
                  • enabled
                  • target_modules
                  • exclude_modules
                  • match_all_linear
                  • dim
                  • alpha
                  • dropout
                  • dropout_position
                  • lora_A_init
                  • use_triton
                • AutomodelBackendConfig
                  • _target_
                  • attn
                  • linear
                  • rms_norm
                  • experts
                  • dispatcher
                  • enable_deepep
                  • fake_balanced_gate
                  • enable_hf_state_dict_adapter
                  • enable_fsdp_optimizations
                  • gate_precision
                • AutomodelFreezeConfig
                  • freeze_vision_tower
                  • freeze_audio_tower
                  • freeze_language_model
                • AutomodelKwargs
                  • use_liger_kernel
                  • backend
                  • freeze_config
                  • force_hf
                • DTensorConfigDisabled
                  • enabled
                • MoEParallelizerOptions
                  • ignore_router_for_ac
                  • reshard_after_forward
                  • lm_head_precision
                  • wrap_outer_model
                • DTensorConfig
                  • enabled
                  • env_vars
                  • _v2
                  • tensor_parallel_size
                  • context_parallel_size
                  • expert_parallel_size
                  • dp_replicate_size
                  • sequence_parallel
                  • activation_checkpointing
                  • cpu_offload
                  • custom_parallel_plan
                  • defer_fsdp_grad_sync
                  • moe_parallelizer
                  • lora_cfg
                  • automodel_kwargs
                  • clear_cache_every_n_steps
                • SequencePackingConfigDisabled
                  • enabled
                • SequencePackingConfig
                  • enabled
                  • train_mb_tokens
                  • logprob_mb_tokens
                  • algorithm
                  • microbatch_order
                • RewardModelConfig
                  • enabled
                  • reward_model_type
                • MegatronPeftConfigDisabled
                  • enabled
                • MegatronPeftConfig
                  • enabled
                  • target_modules
                  • exclude_modules
                  • dim
                  • alpha
                  • dropout
                  • dropout_position
                  • lora_A_init_method
                  • lora_B_init_method
                  • a2a_experimental
                  • lora_dtype
                • MegatronOptimizerConfig
                  • optimizer
                  • lr
                  • min_lr
                  • weight_decay
                  • bf16
                  • fp16
                  • params_dtype
                  • adam_beta1
                  • adam_beta2
                  • adam_eps
                  • sgd_momentum
                  • use_distributed_optimizer
                  • use_precision_aware_optimizer
                  • clip_grad
                  • optimizer_cpu_offload
                  • optimizer_offload_fraction
                • MegatronSchedulerConfig
                  • start_weight_decay
                  • end_weight_decay
                  • weight_decay_incr_style
                  • lr_decay_style
                  • lr_decay_iters
                  • lr_warmup_iters
                  • lr_warmup_init
                • MegatronDDPConfig
                  • grad_reduce_in_fp32
                  • overlap_grad_reduce
                  • overlap_param_gather
                  • use_custom_fsdp
                  • data_parallel_sharding_strategy
                • Fp8Config
                  • enabled
                  • fp8
                  • fp8_recipe
                  • fp8_param
                  • force_clear_fp8_caches
                • MegatronConfigDisabled
                  • enabled
                • MegatronCheckpointConfig
                  • async_save
                  • ckpt_assume_constant_structure
                  • ckpt_fully_parallel_save_process_group
                  • ckpt_fully_parallel_load_process_group
                  • ckpt_fully_parallel_load_exchange_algo
                • MegatronConfig
                  • enabled
                  • env_vars
                  • empty_unused_memory_level
                  • activation_checkpointing
                  • recompute_granularity
                  • recompute_modules
                  • tensor_model_parallel_size
                  • pipeline_model_parallel_size
                  • num_layers_in_first_pipeline_stage
                  • num_layers_in_last_pipeline_stage
                  • context_parallel_size
                  • pipeline_dtype
                  • sequence_parallel
                  • freeze_moe_router
                  • expert_tensor_parallel_size
                  • expert_model_parallel_size
                  • defer_fp32_logits
                  • apply_rope_fusion
                  • bias_activation_fusion
                  • force_reconvert_from_hf
                  • attention_backend
                  • moe_per_layer_logging
                  • moe_enable_deepep
                  • moe_token_dispatcher_type
                  • inference_moe_token_dispatcher_type
                  • inference_grouped_gemm_backend
                  • moe_router_num_groups
                  • moe_router_group_topk
                  • transformer_impl
                  • cuda_graph_impl
                  • moe_pad_experts_for_cuda_graph_inference
                  • moe_shared_expert_overlap
                  • use_gloo_process_groups
                  • moe_grouped_gemm
                  • moe_flex_dispatcher_backend
                  • moe_hybridep_num_sms
                  • hybridep_num_ranks_per_nvlink_domain
                  • hybridep_use_mnnvl
                  • peft
                  • optimizer
                  • scheduler
                  • distributed_data_parallel_config
                  • checkpoint
                  • gradient_accumulation_fusion
                  • use_fused_weighted_squared_relu
                  • use_fused_linear_logprobs
                  • fused_linear_logprobs_chunk_size
                  • mtp_num_layers
                  • mtp_loss_scaling_factor
                  • mtp_use_repeated_layer
                  • mtp_detach_heads
                  • clear_memory_caches_before_refit
                  • fp8_cfg
                • DraftConfigDisabled
                  • enabled
                • DraftConfig
                  • enabled
                  • model_name
                  • loss_weight
                  • num_layers
                  • aux_layer_indices
                • TokenizerConfig
                  • name
                  • chat_template
                  • chat_template_kwargs
                  • audio
                  • video
                  • use_processor
                  • use_fastokens
                • PytorchOptimizerConfig
                  • name
                  • kwargs
                • SinglePytorchSchedulerConfig
                  • name
                  • kwargs
                • SinglePytorchMilestonesConfig
                  • milestones
                • SchedulerMilestones
                • DynamicBatchingConfigDisabled
                  • enabled
                • DynamicBatchingConfig
                  • enabled
                  • train_mb_tokens
                  • logprob_mb_tokens
                  • sequence_length_round
                • RouterReplayConfigDisabled
                  • enabled
                • RouterReplayConfig
                  • enabled
                • PolicyConfig
                  • model_name
                  • tokenizer
                  • train_global_batch_size
                  • train_micro_batch_size
                  • logprob_batch_size
                  • logprob_chunk_size
                  • generation
                  • generation_batch_size
                  • precision
                  • reward_model_cfg
                  • dtensor_cfg
                  • megatron_cfg
                  • draft
                  • pretrained_checkpoint
                  • router_replay
                  • hf_config_overrides
                  • dynamic_batching
                  • sequence_packing
                  • make_sequence_length_divisible_by
                  • max_total_sequence_length
                  • max_grad_norm
                  • refit_buffer_size_gb
                  • optimizer
                  • scheduler
                  • quant_cfg
                  • quant_calib_data
                  • quant_calib_size
                  • quant_batch_size
                  • quant_sequence_length
                  • disable_modelopt_layer_spec
                  • is_vlm
          • nemo_rl.models.generation
            • Subpackages
              • nemo_rl.models.generation.trtllm
                • Submodules
                  • nemo_rl.models.generation.trtllm.trtllm_backend
                    • Module Contents
                      • Classes
                      • Functions
                      • API
                        • _call_model_loader_hook_if_available()
                        • NcclExtension
                          • init_collective()
                          • start_gpu_profiling()
                          • stop_gpu_profiling()
                          • prepare_refit_info()
                          • _finalize_weight_update()
                          • update_weights_from_collective()
                          • get_zmq_address()
                          • maybe_init_zmq()
                          • update_weights_via_ipc_zmq()
                          • cleanup_zmq()
                          • report_device_id()
                  • nemo_rl.models.generation.trtllm.trtllm_http_server
                    • Module Contents
                      • Functions
                      • Data
                      • API
                        • logger
                        • _build_reasoning_parser()
                        • create_app()
                        • _resolve_tool_parser_name()
                        • _build_tool_parser()
                        • _make_parse_tool_calls()
                        • _to_int_ids()
                        • _build_prompt_token_ids()
                        • _compute_splice_inputs()
                        • start_server()
                  • nemo_rl.models.generation.trtllm.config
                    • Module Contents
                      • Classes
                      • API
                        • TrtllmSpecificArgs
                          • tensor_parallel_size
                          • model_name
                          • gpu_memory_utilization
                          • max_model_len
                          • precision
                          • max_batch_size
                          • max_num_tokens
                          • expose_http_server
                          • async_engine
                          • moe_tensor_parallel_size
                          • moe_expert_parallel_size
                          • in_flight_weight_updates
                          • recompute_kv_cache_after_weight_updates
                          • default_chat_template_kwargs
                          • tool_parser
                          • reasoning_parser
                        • TrtllmConfig
                          • trtllm_cfg
                          • trtllm_kwargs
                  • nemo_rl.models.generation.trtllm.trtllm_generation
                    • Module Contents
                      • Classes
                      • API
                        • TrtllmGeneration
                          • init_cluster_placement_groups()
                          • _get_tied_worker_bundle_indices()
                          • _report_dp_openai_server_base_urls()
                          • _report_device_id()
                          • init_collective()
                          • generate()
                          • generate_async()
                          • prepare_for_generation()
                          • finish_generation()
                          • prepare_refit_info()
                          • start_gpu_profiling()
                          • stop_gpu_profiling()
                          • update_weights_from_collective()
                          • update_weights_via_ipc_zmq()
                          • invalidate_kv_cache()
                          • clear_logger_metrics()
                          • get_logger_metrics()
                          • shutdown()
                          • __del__()
                  • nemo_rl.models.generation.trtllm.trtllm_worker_async
                    • Module Contents
                      • Classes
                      • API
                        • TrtllmAsyncGenerationWorkerImpl
                          • configure_worker()
                          • __repr__()
                          • is_alive()
                          • post_init_async()
                          • shutdown()
                          • start_http_server()
                          • stop_http_server()
                          • report_dp_openai_server_base_url()
                          • init_collective_async()
                          • prepare_refit_info_async()
                          • update_weights_from_collective_async()
                          • update_weights_via_ipc_zmq_async()
                          • report_device_id_async()
                          • _weights_tags()
                          • _all_sleep_tags()
                          • _resolve_wake_tags()
                          • sleep_async()
                          • wake_up_async()
                          • reset_prefix_cache_async()
                          • start_gpu_profiling_async()
                          • stop_gpu_profiling_async()
                          • generate_async()
                          • _resolve_end_id()
                          • _build_sampling_params()
                        • TrtllmAsyncGenerationWorker
              • nemo_rl.models.generation.megatron
                • Submodules
                  • nemo_rl.models.generation.megatron.megatron_worker
                    • Module Contents
                      • Classes
                      • API
                        • MegatronGenerationMixin
                          • _init_inference_engine_state()
                          • _initialize_inference_engine()
                          • _start_inference_coordinator()
                          • _sleep()
                          • _sleep_engine()
                          • _wake()
                          • _wake_engine()
                          • _start_inference_loop_thread()
                          • _setup_openai_api_server()
                          • _run_async_coordinator_start()
                          • finish_generation()
                          • prepare_for_generation()
                          • report_dp_openai_server_base_url()
                          • _build_sampling_params()
                          • _merge_stop_strings()
                          • _prepare_data_for_generation()
                          • _parse_result_to_batched_data_dict()
                          • generate()
                          • generate_async()
                          • _generate_with_persistent_engine()
                        • MegatronGenerationRefitMixin
                          • init_collective_mcore_generation()
                          • preinit_nvshmem_collective()
                          • swap_weights_via_reshard()
                          • suspend_for_refit()
                          • resume_after_refit()
                  • nemo_rl.models.generation.megatron.config
                    • Module Contents
                      • Classes
                      • API
                        • MCoreGenerationSpecificArgs
                          • async_engine
                          • expose_http_server
                          • parsers
                          • buffer_size_gb
                          • block_size_tokens
                          • max_tokens
                          • max_model_len
                          • num_cuda_graphs
                          • use_cuda_graphs_for_non_decode_steps
                          • cuda_graph_impl
                          • inference_cuda_graph_scope
                          • materialize_only_last_token_logits
                          • enable_chunked_prefill
                          • enable_prefix_caching
                          • refit_backend
                          • num_speculative_tokens
                          • mamba_inference_ssm_states_dtype
                          • mamba_inference_conv_states_dtype
                          • kv_cache_management_mode
                          • logging_step_interval
                        • MCoreGenerationConfig
                          • mcore_generation_config
                  • nemo_rl.models.generation.megatron.megatron_generation
                    • Module Contents
                      • Classes
                      • API
                        • MegatronGeneration
                          • effective_megatron_cfg()
                          • nvlink_domain_span()
                          • init_cluster_placement_groups()
                          • init_collective()
                          • update_weights_from_collective()
                          • generate()
                          • generate_async()
                          • prepare_for_generation()
                          • finish_generation()
                          • preinit_nvshmem_collective()
                          • suspend_for_refit()
                          • resume_after_refit()
                          • prepare_refit_info()
                          • start_gpu_profiling()
                          • stop_gpu_profiling()
                          • shutdown()
                          • __del__()
                  • nemo_rl.models.generation.megatron.utils
                    • Module Contents
                      • Functions
                      • API
                        • resolve_torch_dtype()
                        • log_gpu_memory()
                • Package Contents
                  • Data
                  • API
                    • __all__
              • nemo_rl.models.generation.vllm
                • Submodules
                  • nemo_rl.models.generation.vllm.refit_loader
                    • Module Contents
                      • Classes
                      • API
                        • VllmShardedExpertRefitMixin
                          • _is_sharded_refit_weight()
                          • _sharded_refit_param_names()
                          • _checkpoint_engine_weight_layout()
                          • _validate_expert_storage()
                          • _local_expert_id()
                          • _load_destination_local_expert_group()
                          • _load_sharded_expert_weight_groups()
                          • _load_sharded_expert_weights()
                  • nemo_rl.models.generation.vllm.checkpoint_engine
                    • Module Contents
                      • Classes
                      • Functions
                      • Data
                      • API
                        • NIXL_VLLM_WORKER
                        • _NIXL_CONFIG_KEY
                        • configure_nixl_worker()
                        • preinit_nixl_from_vllm_config()
                        • resolve_rollout_rank()
                        • VllmCheckpointEngineMixin
                          • checkpoint_engine
                          • checkpoint_engine_total_memory_bytes()
                          • _load_hf_weights()
                          • init_checkpoint_engine()
                          • prepare_checkpoint_engine()
                          • init_checkpoint_engine_process_group()
                          • finalize_checkpoint_engine()
                          • _update_weights_from_checkpoint_engine_async()
                          • update_weights_from_checkpoint_engine()
                        • VllmCheckpointEngineRpcMixin
                          • checkpoint_engine_rpc()
                        • VllmAsyncCheckpointEngineRpcMixin
                          • checkpoint_engine_rpc_async()
                  • nemo_rl.models.generation.vllm.config
                    • Module Contents
                      • Classes
                      • Functions
                      • Data
                      • API
                        • VllmRefitTransportName
                        • VllmRefitSelector
                        • VLLM_SPARSE_REFIT_TRANSPORTS
                        • VllmSpecificArgs
                          • tensor_parallel_size
                          • pipeline_parallel_size
                          • expert_parallel_size
                          • gpu_memory_utilization
                          • max_model_len
                          • skip_tokenizer_init
                          • async_engine
                          • load_format
                          • precision
                          • logprobs_mode
                          • cap_max_tokens_to_context
                          • is_mx
                          • kv_cache_dtype
                          • enforce_eager
                          • enable_return_routed_experts
                          • use_tqdm
                          • expose_http_server
                          • http_refit_api_key_env_var
                          • http_refit_server_port
                          • zmq_refit_server_port
                          • http_server_serving_chat_kwargs
                          • tool_parser_plugin
                          • env_vars
                          • reasoning_parser_plugin
                        • VllmDeltaCompressionConfig
                          • encoding
                          • sparse_bucket_size_bytes
                          • export_chunk_bytes
                          • zstd_threads
                        • VllmRefitStorageConfig
                          • s3_bucket
                          • s3_region
                          • s3_prefix
                          • staging_dir
                        • VllmRefitBaselineConfig
                          • in_memory
                          • mmap_dir
                        • VllmRefitTuningConfig
                          • encode_workers
                          • transfer_workers
                          • zmq_retries
                          • zmq_relay_payload_workers
                          • zmq_relay_forward_workers
                          • apply_queue_depth
                          • apply_batch_size
                          • partition_workers
                        • VllmSparseRefitConfig
                          • delta_compression
                          • storage
                          • baseline
                          • tuning
                          • verify_samples_per_payload
                          • request_timeout_s
                        • VllmNixlRefitConfig
                          • update_weights_bucket_memory_ratio
                          • device
                          • backend_name
                          • backend_init_params
                          • release_after_refit
                          • shard_expert_weights
                        • VllmCheckpointEnginePluginConfig
                          • update_weights_bucket_memory_ratio
                          • release_after_refit
                        • VllmRefitConfig
                          • sparse
                          • nixl
                        • VllmConfig
                          • vllm_cfg
                          • vllm_kwargs
                          • refit_transport
                          • refit_cfg
                          • quant_cfg
                          • real_quant
                          • real_quant_export_cpu_offload
                          • real_quant_ignore
                        • normalize_vllm_refit_config()
                  • nemo_rl.models.generation.vllm.vllm_worker_async
                    • Module Contents
                      • Classes
                      • Data
                      • API
                        • LOGGER
                        • VllmAsyncGenerationWorkerImpl
                          • _return_routed_experts_enabled()
                          • _reserve_port()
                          • load_model()
                          • _create_engine()
                          • _install_engine_input_socket_lock()
                          • _start_vllm_metrics_logger()
                          • get_vllm_logger_metrics()
                          • clear_vllm_logger_metrics()
                          • post_init_async()
                          • get_reserved_url()
                          • report_dp_openai_server_base_url()
                          • _setup_vllm_openai_api_server()
                          • _setup_vllm_server()
                          • init_collective_async()
                          • generate_async()
                          • generate_text_async()
                          • report_device_id_async()
                          • prepare_refit_info_async()
                          • update_weights_via_ipc_zmq_async()
                          • update_weights_from_collective_async()
                          • init_nccl_reshard_comm_group_async()
                          • prepare_nccl_reshard_refit_info_async()
                          • nccl_reshard_refit_async()
                          • reset_prefix_cache_async()
                          • sleep_async()
                          • wake_up_async()
                          • shutdown()
                        • VllmAsyncGenerationWorker
                  • nemo_rl.models.generation.vllm.worker_utils
                    • Module Contents
                      • Functions
                      • API
                        • resolve_distributed_executor_backend()
                        • resolve_data_parallel_local_rank()
                  • nemo_rl.models.generation.vllm.vllm_sparse_delta
                    • Module Contents
                      • Classes
                      • Functions
                      • Data
                      • API
                        • _TensorViewKey
                        • _LoaderWeight
                        • _LoaderObservation
                        • _storage_key()
                        • _view_key()
                        • _SparseWeightLoadMode
                          • start()
                          • _observe_xor_copy()
                          • _remember_changed()
                          • __torch_dispatch__()
                          • finish()
                        • VllmSparseDeltaApplier
                          • discover_native_skips()
                          • _source_tensor()
                          • _scatter_values()
                          • _prepare_loader_weight()
                          • _load_weights()
                          • _validate_loader_report()
                          • _apply_decoded_items()
                          • _iter_sparse_payload()
                          • update_weights_from_decoded_sparse_payload()
                          • _load_sparse_payloads()
                          • synchronize_device()
                          • finish_sparse_delta_refit()
                  • nemo_rl.models.generation.vllm.vllm_backend
                    • Module Contents
                      • Classes
                      • Functions
                      • Data
                      • API
                        • logger
                        • WeightUpdateTransport
                        • WeightUpdateFinalizer
                        • _format_refit_key_error()
                        • IPCWeightManifestError
                        • _IPCWeightManifest
                          • validate_batch()
                          • record_loaded()
                          • record_load_failure()
                          • require_complete()
                        • NixlVllmWorker
                          • __new__()
                        • fix_gemma3_vision_weight_name()
                        • _read_mtp_layer_weights_from_checkpoint()
                        • VllmInternalWorkerExtension
                          • _mtp_drafter_from_disk
                          • _sparse_delta_applier
                          • _nrl_named_parameters
                          • _get_named_parameters()
                          • _load_full_hf_weights()
                          • _load_hf_weights()
                          • bind_numa()
                          • init_collective()
                          • init_nccl_reshard_comm_group()
                          • report_device_id()
                          • report_node_hostname()
                          • get_zmq_address()
                          • maybe_init_zmq()
                          • prepare_refit_info()
                          • prepare_sparse_delta_refit_info()
                          • _uses_fp8_kv_cache()
                          • _maybe_process_fp8_kv_cache()
                          • _split_policy_and_draft_weights()
                          • _trim_vocab_padding()
                          • _get_drafter_model()
                          • _load_draft_weights()
                          • _mtp_drafter_refit_enabled()
                          • _maybe_refit_mtp_drafter()
                          • _maybe_process_mtp_drafter_after_loading()
                          • load_mtp_weights_from_disk()
                          • _load_weights()
                          • _get_sparse_delta_applier()
                          • _weight_update_lifecycle()
                          • _weight_update_errors_are_fatal()
                          • _synchronize_before_ipc_data_ack()
                          • update_weights_via_ipc_zmq()
                          • update_weights_from_collective()
                          • update_weights_from_decoded_sparse_payload()
                          • synchronize_device()
                          • finish_sparse_delta_refit()
                          • prepare_nccl_reshard_refit_info()
                          • build_hf_to_local_param_map()
                          • _build_hf_to_gen_backend_mapping()
                          • nccl_reshard_refit()
                          • _receive_and_load_misc_params()
                          • cleanup()
                          • start_gpu_profiling()
                          • stop_gpu_profiling()
                        • VllmInternalWorkerExtensionWithCheckpointEngine
                  • nemo_rl.models.generation.vllm.vllm_worker
                    • Module Contents
                      • Classes
                      • Functions
                      • Data
                      • API
                        • logger
                        • _context_capped_max_new_tokens()
                        • _resolve_enable_prefix_caching()
                        • _merge_fp8_kwargs()
                        • BaseVllmGenerationWorker
                          • __repr__()
                          • configure_worker()
                          • _init_config()
                          • _load_model()
                          • llm()
                          • is_alive()
                          • _merge_stop_strings()
                          • _build_sampling_params()
                          • start_gpu_profiling()
                          • stop_gpu_profiling()
                          • _spec_decode_max_tokens()
                          • _request_max_new_tokens()
                          • _patch_vllm_nsight_config()
                          • _get_raw_spec_counters()
                          • report_refit_server_base_url()
                          • start_zmq_sparse_refit_relay()
                          • configure_zmq_sparse_refit_relay()
                          • stop_zmq_sparse_refit_relay()
                        • VllmGenerationWorkerImpl
                          • _create_engine()
                          • post_init()
                          • init_collective()
                          • generate()
                          • generate_text()
                          • report_device_id()
                          • prepare_refit_info()
                          • update_weights_via_ipc_zmq()
                          • update_weights_from_collective()
                          • init_nccl_reshard_comm_group()
                          • prepare_nccl_reshard_refit_info()
                          • nccl_reshard_refit()
                          • reset_prefix_cache()
                          • sleep()
                          • wake_up()
                          • shutdown()
                        • VllmGenerationWorker
                  • nemo_rl.models.generation.vllm.vllm_generation
                    • Module Contents
                      • Classes
                      • Data
                      • API
                        • logger
                        • VllmGeneration
                          • init_cluster_placement_groups()
                          • _get_tied_worker_bundle_indices()
                          • _report_device_id()
                          • _report_dp_openai_server_base_urls()
                          • _collect_reserved_urls()
                          • load_and_start()
                          • _post_init()
                          • _get_raw_spec_counters()
                          • snapshot_step_metrics()
                          • get_step_metrics()
                          • init_collective()
                          • generate()
                          • generate_text()
                          • _async_generate_base()
                          • generate_text_async()
                          • generate_async()
                          • prepare_for_generation()
                          • finish_generation()
                          • shutdown()
                          • prepare_refit_info()
                          • update_weights_via_ipc_zmq()
                          • update_weights_from_collective()
                          • init_nccl_reshard_comm_group()
                          • prepare_nccl_reshard_refit_info()
                          • nccl_reshard_refit()
                          • start_gpu_profiling()
                          • stop_gpu_profiling()
                          • get_vllm_logger_metrics()
                          • clear_vllm_logger_metrics()
                          • clear_logger_metrics()
                          • get_logger_metrics()
                          • __del__()
                          • invalidate_kv_cache()
                          • requires_kv_scale_sync
                  • nemo_rl.models.generation.vllm.vllm_sparse_refit
                    • Module Contents
                      • Classes
                      • Functions
                      • Data
                      • API
                        • logger
                        • _warn_unauthenticated_refit_server()
                        • _StagedSparsePayload
                          • path
                          • started_at
                          • finished_at
                          • save_s
                        • _stage_sparse_payload()
                        • VllmSparseRefitReceiver
                          • set_worker_hostnames()
                          • start_sync_server()
                          • shutdown()
                          • _enqueue_sparse_payload_apply()
                          • _submit_pending_sparse_payloads()
                          • _notify_refit_apply_waiters()
                          • _collect_refit_apply_results()
                          • _refit_collective_response()
                          • _refit_collective_rpc()
                          • update_weights_from_serialized_sparse_payloads()
                          • update_weights_from_staged_sparse_payloads()
                          • _flush_queued_sparse_payloads()
                          • _prepare_sparse_refit_info()
                          • _apply_s3_manifest_payload()
                          • _apply_zmq_payload()
                          • setup_api_server()
                          • report_refit_server_base_url()
                          • start_zmq_sparse_refit_relay()
                          • configure_zmq_sparse_refit_relay()
                          • stop_zmq_sparse_refit_relay()
                          • flush_zmq_sparse_refit_relay()
                          • _setup_vllm_refit_server()
                  • nemo_rl.models.generation.vllm.refit_layout
                    • Module Contents
                      • Classes
                      • Functions
                      • Data
                      • API
                        • VllmExpertParamLayout
                          • tp_rank
                          • tp_size
                          • local_expert_ids
                        • VllmWeightLayout
                          • expert_params
                          • missing_weight_prefixes
                        • HfExpertWeight
                          • parameter_name
                          • expert_id
                          • shard_id
                          • tp_shard_dim
                        • _HF_EXPERT_WEIGHT_RE
                        • _HF_PROJECTION_SHARDS
                        • parse_hf_expert_weight()
                        • select_hf_weight_for_vllm_target()
                  • nemo_rl.models.generation.vllm.patches
                    • Module Contents
                      • Functions
                      • API
                        • _get_vllm_file()
                        • _locked_file_patch()
                        • _patch_vllm_init_workers_ray()
                        • _patch_vllm_llama_eagle3_own_lm_head()
                        • _patch_vllm_tool_parser_namespace_tool()
                        • _patch_vllm_ray_executor_v2_tcpstore_port()
                        • _patch_vllm_shm_broadcast_bind_retry()
                        • _patch_vllm_radio_layerscale_loader()
                        • ensure_vllm_source_compat()
                        • _apply_vllm_patches()
                  • nemo_rl.models.generation.vllm.utils
                    • Module Contents
                      • Functions
                      • Data
                      • API
                        • R3_MISSING_ROUTE_SENTINEL
                        • G_ROUTED_EXPERTS_RANGE_CHECKED
                        • _as_routed_experts_tensor()
                        • format_prompt_for_vllm_generation()
                        • pad_and_align_routed_expert_indices()
                        • attach_routed_experts_to_chat_response_choices()
                        • model_dump_chat_response_with_routed_experts()
                        • aggregate_spec_decode_counters()
                        • compute_spec_decode_metrics()
                        • GENERATION_WORKER_OVERRIDES
                        • resolve_generation_worker_cls()
                  • nemo_rl.models.generation.vllm.collective_rpc
                    • Module Contents
                      • Functions
                      • API
                        • resolve_collective_rpc_result()
                • Package Contents
                  • Data
                  • API
                    • __all__
            • Submodules
              • nemo_rl.models.generation.interfaces
                • Module Contents
                  • Classes
                  • Functions
                  • Data
                  • API
                    • ROUTED_EXPERTS_FALLBACK_DTYPE
                    • ROUTED_EXPERTS_MISSING_ROUTE_SENTINEL
                    • _ROUTED_EXPERTS_DTYPE_NAMES
                    • get_num_routed_experts()
                    • resolve_routed_experts_dtype()
                    • resolve_routed_experts_dtype_name_for_model()
                    • verify_right_padding()
                    • ResourcesConfig
                      • gpus_per_node
                      • num_nodes
                    • OptionalResourcesConfig
                      • gpus_per_node
                      • num_nodes
                    • ColocationConfig
                      • enabled
                      • resources
                    • CheckpointEngineConfig
                      • backend
                      • update_weights_bucket_memory_ratio
                      • engine_kwargs
                    • GenerationConfig
                      • backend
                      • max_new_tokens
                      • temperature
                      • top_p
                      • top_k
                      • val_temperature
                      • val_top_p
                      • val_top_k
                      • model_name
                      • stop_token_ids
                      • stop_strings
                      • bad_words
                      • colocated
                      • port_range_low
                      • port_range_high
                      • use_async_rollouts
                      • _pad_token_id
                      • _mtp_weights_from_refit
                    • GenerationSamplingParams
                      • temperature
                      • top_p
                      • top_k
                      • from_generation_config()
                    • GenerationDatumSpec
                      • input_ids
                      • input_lengths
                      • stop_strings
                      • __extra__
                    • GenerationOutputSpec
                      • output_ids
                      • generation_lengths
                      • unpadded_sequence_lengths
                      • logprobs
                      • routed_experts
                      • r3_routed_experts_missing_routes
                      • r3_routed_experts_expected_routes
                      • r3_routed_experts_actual_routes
                      • truncated
                      • __extra__
                    • GenerationInterface
                      • init_collective()
                      • generate()
                      • prepare_for_generation()
                      • finish_generation()
                      • requires_kv_scale_sync
                      • prepare_refit_info()
                      • update_weights_via_ipc_zmq()
                      • update_weights_from_collective()
                      • prepare_nccl_reshard_refit_info()
                      • nccl_reshard_refit()
                      • invalidate_kv_cache()
                      • clear_logger_metrics()
                      • get_logger_metrics()
              • nemo_rl.models.generation.openai_server_utils
                • Module Contents
                  • Functions
                  • API
                    • replace_prefix_tokens()
              • nemo_rl.models.generation.constants
                • Module Contents
                  • Data
                  • API
                    • VLLM_BACKEND
                    • SGLANG_BACKEND
                    • MEGATRON_BACKEND
            • Package Contents
              • Functions
              • Data
              • API
                • TokenizerType
                • configure_generation_config()
          • nemo_rl.models.dtensor
            • Submodules
              • nemo_rl.models.dtensor.parallelize
                • Module Contents
                  • Classes
                  • Functions
                  • Data
                  • API
                    • RotaryEmbedParallel
                      • _prepare_input_fn()
                      • _prepare_output_fn()
                    • ColwiseParallelWithGather
                      • _prepare_output_fn()
                      • _apply()
                    • _parallelize_gemma3()
                    • _parallelize_llama()
                    • _parallelize_qwen()
                    • PARALLIZE_FUNCTIONS
                    • translate_parallel_style()
                    • get_hf_tp_plan()
                    • _parallelize_nm5_h()
                    • _parallelize_model()
                    • to_local_if_dtensor()
                    • clip_grad_by_total_norm_()
                    • get_grad_norm()
      • nemo_rl.utils
        • Subpackages
          • nemo_rl.utils.checkpoint_engines
            • Submodules
              • nemo_rl.utils.checkpoint_engines.nixl
                • Module Contents
                  • Classes
                  • Functions
                  • Data
                  • API
                    • NixlAgentMetadata
                    • NIXL_DEFAULT_BACKEND_NAME
                    • NIXL_TRANSFER_BUFFER_COUNT
                    • _source_rank_for_rollout()
                    • _create_nixl_agent()
                    • resolve_nixl_backend_kwargs()
                    • preinit_nixl_agent()
                    • _sync_device()
                    • NixlAgent
                      • get_agent_metadata()
                      • add_remote_agent()
                      • remove_remote_agent()
                      • send_message()
                      • read_message()
                      • wait_notification()
                    • NIXLCheckpointEngine
                      • _allocate_transfer_buffer()
                      • prepare()
                      • get_target_weight_layout()
                      • init_policy_process_group()
                      • init_rollout_process_group()
                      • _disconnect_peers()
                      • _release_transfer_buffers()
                      • finalize()
                      • send_weights()
                      • receive_weight_batches()
                      • _wait_read()
                      • _receive_weight_chunk_batches()
              • nemo_rl.utils.checkpoint_engines.base
                • Module Contents
                  • Classes
                  • Functions
                  • API
                    • TensorMeta
                      • name
                      • shape
                      • dtype
                      • chunk_offset
                      • chunk_size
                      • offset
                      • nbytes
                    • CheckpointEngine
                      • shard_expert_weights
                      • get_target_weight_layout()
                      • prepare()
                      • init_policy_process_group()
                      • init_rollout_process_group()
                      • finalize()
                      • send_weights()
                      • receive_weight_batches()
                    • create_checkpoint_engine()
                    • split_weight_chunks()
                    • merge_weight_chunk_batches()
        • Submodules
          • nemo_rl.utils.nvml
            • Module Contents
              • Functions
              • Data
              • API
                • logger
                • nvml_context()
                • device_id_to_physical_device_id()
                • get_device_uuid()
                • get_free_memory_bytes()
                • _resolve_device_id()
                • log_gpu_memory_diagnostics()
          • nemo_rl.utils.timer
            • Module Contents
              • Classes
              • Functions
              • Data
              • API
                • logger
                • Timer
                  • _REDUCTION_FUNCTIONS
                  • _fmt()
                  • start()
                  • stop()
                  • record()
                  • mark()
                  • get_markers()
                  • time()
                  • get_elapsed()
                  • get_latest_elapsed()
                  • reduce()
                  • get_timing_metrics()
                  • reset()
                • ThreadSafeTimer
                  • start()
                  • stop()
                  • record()
                  • mark()
                  • get_markers()
                  • time()
                  • get_elapsed()
                  • get_latest_elapsed()
                  • reduce()
                  • get_timing_metrics()
                  • reset()
                • convert_to_seconds()
                • TimeoutChecker
                  • check_save()
                  • start_iterations()
                  • mark_iteration()
          • nemo_rl.utils.flops_formulas
            • Module Contents
              • Classes
              • Functions
              • API
                • FLOPSConfig
                  • gbs
                  • enc_seq_len
                  • hs
                  • layers
                  • ffn_hs
                  • attention_heads
                  • head_dim
                  • moe_router_topk
                  • query_groups
                  • img_seq_len
                  • img_h
                  • img_w
                  • in_channels
                  • patch_dim
                  • class_token_len
                  • projector_type
                  • inp_s
                  • model_pattern
                  • vocab_size
                  • model_channels
                  • vec_in_dim
                  • q_lora_rank
                  • kv_lora_rank
                  • qk_head_dim
                  • qk_pos_emb_head_dim
                  • v_head_dim
                  • moe_layer_freq
                  • moe_shared_expert_intermediate_size
                  • moe_ffn_hidden_size
                  • mtp_num_layers
                  • causal_self_attn
                  • is_hybrid_model
                  • hybrid_override_pattern
                  • mamba_state_dim
                  • mamba_head_dim
                  • mamba_num_groups
                  • mamba_num_heads
                • gpt3()
                • llama()
                • nemotron()
                • mixtral()
                • qwen2()
                • qwen3()
                • bert()
                • transformer()
                • flux()
                • deepseekv3()
                • _mlp_layer_flops()
                • _non_mla_attn_layer_flops()
                • _mamba_layer_flops()
                • _hybrid_model_flops()
                • nemotronh()
          • nemo_rl.utils.r3_trace
            • Module Contents
              • Functions
              • Data
              • API
                • _TRACE_ENV
                • _TRACE_STEPS_ENV
                • _TRACE_SAMPLES_ENV
                • _TRACE_DIR_ENV
                • _TRACE_MICROBATCHES_ENV
                • _TRACE_VERIFY_FORWARD_ENV
                • _DEFAULT_TRACE_DIR
                • _DEFAULT_TRACE_STEPS
                • _DEFAULT_TRACE_SAMPLES
                • _DEFAULT_TRACE_MICROBATCHES
                • _write_lock
                • _patch_lock
                • _router_replay_patch_depth
                • _original_get_replay_topk
                • _event_counts
                • _context
                • r3_trace_enabled()
                • r3_trace_verify_forward_enabled()
                • _env_int()
                • _trace_steps()
                • _trace_samples()
                • _trace_microbatches()
                • _next_count()
                • _should_trace_step()
                • _current_context()
                • _torch_rank_info()
                • _trace_path()
                • _write_record()
                • _tensor_sha256()
                • _tensor_preview()
                • _tensor_record()
                • _shape()
                • _valid_sample_record()
                • _length_at()
                • _tensors_equal()
                • _expected_with_missing_route_fallback()
                • _router_replay_action_name()
                • _trace_router_replay_topk_use()
                • _verify_router_replay_forward_context()
                • trace_rollout_payload()
                • trace_tq_fetch_payload()
                • r3_trace_stage()
                • maybe_r3_trace_stage()
                • trace_cp_routed_experts()
                • trace_router_replay_assignment()
                • trace_router_replay_action()
          • nemo_rl.utils.weight_transfer_sparse_codec
            • Module Contents
              • Classes
              • Functions
              • Data
              • API
                • NamedTensor
                • TensorBatch
                • SparseOperation
                • SparseInfo
                • TensorPayload
                • PreparedTensorPayload
                • SparseItem
                • _INTEGER_DTYPE_BY_SIZE
                • _TensorPayloadBuilder
                  • add_locations()
                  • add_values()
                  • finish()
                • integer_dtype_for_element_size()
                • dtype_from_name()
                • sparse_operation()
                • integer_view()
                • encode_sparse_infos()
                • merge_sparse_payloads()
                • sparse_locations_for_item()
                • _encode_explicit_locations()
                • DeltaCompressionTracker
                  • prepare_sparse_delta_payload()
                  • _add_verification_samples()
                  • on_sync_succeeded()
                  • on_sync_failed()
                  • snapshot_baseline()
                  • _wait_for_baseline_commits()
                  • _commit_baseline_updates()
                  • _baseline()
          • nemo_rl.utils.checkpoint
            • Module Contents
              • Classes
              • Functions
              • Data
              • API
                • PathLike
                • _load_megatron_common_state_dict()
                • PretrainedCheckpointConfig
                  • path
                  • format
                • CheckpointingConfig
                  • enabled
                  • checkpoint_dir
                  • metric_name
                  • higher_is_better
                  • save_period
                  • keep_top_k
                  • ft_keep_latest_k
                  • ft_save_period
                  • checkpoint_must_save_by
                  • pretrained_checkpoint
                  • save_optimizer
                  • model_save_format
                  • save_consolidated
                  • model_cache_dir
                  • model_repo_id
                  • is_peft
                  • peft_config
                  • is_async
                • CheckpointManager
                  • get_resume_paths()
                  • init_tmp_checkpoint()
                  • _rename_checkpoint()
                  • finalize_checkpoint()
                  • begin_finalization()
                  • finalize_pending()
                  • shutdown()
                  • _warn_on_delete_failure()
                  • __enter__()
                  • __exit__()
                  • has_pending_finalization
                  • remove_old_checkpoints()
                  • get_best_checkpoint_path()
                  • get_latest_checkpoint_path()
                  • load_training_info()
                • _load_checkpoint_history()
          • nemo_rl.utils.config
            • Module Contents
              • Functions
              • API
                • resolve_path()
                • merge_with_override()
                • load_config_with_inheritance()
                • load_config()
                • OverridesError
                • parse_hydra_overrides()
                • register_omegaconf_resolvers()
          • nemo_rl.utils.nsys
            • Module Contents
              • Classes
              • Functions
              • Data
              • API
                • NRL_NSYS_WORKER_PATTERNS
                • NRL_NSYS_PROFILE_STEP_RANGE
                • _parse_extra_options()
                • NRL_NSYS_EXTRA_OPTIONS
                • ProfilablePolicy
                  • start_gpu_profiling()
                  • stop_gpu_profiling()
                • maybe_gpu_profile_step()
                • wrap_with_nvtx_name()
          • nemo_rl.utils.memory_tracker
            • Module Contents
              • Classes
              • API
                • MemoryTrackerDataPoint
                  • stage
                  • memory_used_before_stage_gb
                  • variables_before_stage
                  • memory_used_after_stage_gb
                  • variables_after_stage
                  • mem_used_diff_gb
                  • new_variables
                  • get_snapshot_str()
                • MemoryTracker
                  • data_points
                  • model_post_init()
                  • snapshot_start_of_stage()
          • nemo_rl.utils.weight_transfer_zmq
            • Module Contents
              • Classes
              • Functions
              • Data
              • API
                • _PROTOCOL
                • _DATA
                • _ACK
                • _NACK
                • _RelayTransfer
                  • checksums
                  • futures
                • _json_bytes()
                • _configure_socket()
                • ZmqSparseRefitClient
                  • send_payload()
                  • close()
                • ZmqSparseRefitServer
                  • configure_tree()
                  • start()
                  • close()
                  • flush()
                  • _fanout()
                  • _forward()
                  • _send_reply()
                  • _parse_data_message()
                  • _run()
                • stream_sparse_delta_payloads_via_zmq()
          • nemo_rl.utils.venvs
            • Module Contents
              • Functions
              • Data
              • API
                • dir_path
                • git_root
                • DEFAULT_VENV_DIR
                • logger
                • create_local_venv()
                • _env_builder()
                • create_local_venv_on_each_node()
                • make_actor_runtime_env()
          • nemo_rl.utils.native_checkpoint
            • Module Contents
              • Classes
              • Functions
              • API
                • ModelState
                  • state_dict()
                  • load_state_dict()
                • OptimizerState
                  • state_dict()
                  • load_state_dict()
                • save_checkpoint()
                • load_checkpoint()
                • convert_dcp_to_hf()
          • nemo_rl.utils.prefetch_venvs
            • Module Contents
              • Functions
              • API
                • prefetch_venvs()
                • create_frozen_environment_symlinks()
          • nemo_rl.utils.weight_transfer_stream
            • Module Contents
              • Classes
              • Functions
              • Data
              • API
                • _STREAM_LOCAL
                • _S3_PART_SIZE
                • _S3_MEMORY_LIMIT
                • SparseRefitTransport
                  • name
                  • transfer_workers
                  • send
                  • cleanup
                • _SparsePayloadBucket
                  • payloads
                  • dense_bytes
                  • encode_s
                  • next_index
                • _s3_client()
                • _S3ObjectStore
                  • put()
                  • get()
                  • delete()
                  • _request()
                • sparse_payload_checksum()
                • decode_sparse_payload()
                • iter_sparse_weight_chunks()
                • _get_manifest_s3_store()
                • sparse_export_chunk_size()
                • _executor()
                • init_sparse_delta_baseline_from_iterator()
                • stream_sparse_delta_payloads()
                • stream_sparse_delta_payloads_via_s3_manifest()
                • download_s3_refit_payload()
                • zstd_compress()
          • nemo_rl.utils.weight_transfer_http
            • Module Contents
              • Functions
              • Data
              • API
                • G_VLLM_REFIT_S3_MANIFEST_PATH
                • G_VLLM_REFIT_PREPARE_PATH
                • G_VLLM_REFIT_FLUSH_PATH
                • G_VLLM_REFIT_ZMQ_FLUSH_PATH
                • G_VLLM_REFIT_API_KEY_HEADER
                • _HTTP_LOCAL
                • _HTTP_ADAPTER
                • vllm_refit_endpoints()
                • vllm_refit_api_key()
                • refit_http_session()
                • _http_executor()
                • post_vllm_refit_endpoints()
                • merge_vllm_refit_metrics()
          • nemo_rl.utils.logger
            • Module Contents
              • Classes
              • Functions
              • Data
              • API
                • _rich_logging_configured
                • WandbConfig
                  • project
                  • name
                  • entity
                  • log_nemo_gym_full_result_tables
                • SwanlabConfig
                  • project
                  • name
                • TensorboardConfig
                  • log_dir
                • MLflowConfig
                  • experiment_name
                  • run_id
                  • run_name
                  • tracking_uri
                  • artifact_location
                • GPUMonitoringConfig
                  • collection_interval
                  • flush_interval
                • LoggerConfig
                  • log_dir
                  • wandb_enabled
                  • swanlab_enabled
                  • tensorboard_enabled
                  • mlflow_enabled
                  • wandb
                  • tensorboard
                  • swanlab
                  • mlflow
                  • monitor_gpus
                  • gpu_monitoring
                  • num_val_samples_to_print
                • should_log_nemo_gym_full_result_tables()
                • LoggerInterface
                  • log_metrics()
                  • log_hyperparams()
                  • log_histogram()
                  • log_plot()
                • TensorboardLogger
                  • _coerce_to_scalar()
                  • log_metrics()
                  • log_histogram()
                  • log_hyperparams()
                  • log_plot()
                • WandbLogger
                  • _log_diffs()
                  • _log_code()
                  • define_metric()
                  • log_metrics()
                  • log_hyperparams()
                  • log_plot()
                  • finish()
                  • log_histogram()
                • SwanlabLogger
                  • log_metrics()
                  • log_hyperparams()
                  • log_plot()
                  • log_histogram()
                • GpuMetricSnapshot
                  • step
                  • metrics
                • RayGpuMonitorLogger
                  • start()
                  • stop()
                  • _collection_loop()
                  • _parse_metric()
                  • _parse_gpu_sku()
                  • _collect_gpu_sku()
                  • _collect_metrics()
                  • _collect()
                  • _fetch_and_parse_metrics()
                  • flush()
                • _merge_generation_logger_workers()
                • _summarize_list()
                • MLflowLogger
                  • log_metrics()
                  • log_hyperparams()
                  • log_plot()
                  • log_histogram()
                  • __del__()
                • Logger
                  • log_metrics()
                  • log_hyperparams()
                  • finish()
                  • log_batched_dict_as_jsonl()
                  • log_string_list_as_jsonl()
                  • log_plot_per_worker_timeline_metrics()
                  • log_histogram()
                  • log_plot()
                  • log_plot_token_mult_prob_error()
                  • __del__()
                • flatten_dict()
                • configure_rich_logging()
                • print_message_log_samples()
                • get_next_experiment_dir()
                • log_container_init_timing()
          • nemo_rl.utils.grad_norm
            • Module Contents
              • Functions
              • Data
              • API
                • _INF_GRAD_NORM_WARNING
                • warn_if_inf_grad_norm()
          • nemo_rl.utils.flops_tracker
            • Module Contents
              • Classes
              • Functions
              • Data
              • API
                • get_default_hf_config()
                • convert_config_to_flops_config()
                • is_using_tf32()
                • THEORETICAL_TFLOPS
                • get_theoretical_tflops()
                • FLOPTracker
                  • from_config()
                  • track()
                  • track_batch()
                  • reset()
          • nemo_rl.utils.fastokens
            • Module Contents
              • Functions
              • Data
              • API
                • logger
                • _patched
                • maybe_patch_fastokens()
          • nemo_rl.utils.packed_tensor
            • Module Contents
              • Functions
              • API
                • get_target_packed_tensor_size()
                • get_num_buffers()
                • packed_broadcast_producer()
                • packed_broadcast_consumer()
          • nemo_rl.utils.routed_experts_codec
            • Module Contents
              • Functions
              • Data
              • API
                • _MAGIC
                • _WIRE_TORCH_DTYPES
                • _TORCH_DTYPE_NAMES
                • encode_routed_experts()
                • decode_routed_experts()
      • nemo_rl.weight_sync
        • Submodules
          • nemo_rl.weight_sync.interfaces
            • Module Contents
              • Classes
              • API
                • WeightSynchronizer
                  • sync_weights()
                  • is_stale
                  • mark_stale()
                  • init_communicator()
                  • shutdown()
          • nemo_rl.weight_sync.nccl_reshard_utils
            • Module Contents
              • Classes
              • Functions
              • Data
              • API
                • MeshInfo
                  • ndim
                • RefitCtx
                  • buf
                  • extra
                • LocalParamSpec
                  • base
                  • pre
                  • post
                • HFToLocalParamMap
                  • specs
                  • get()
                • RefitBuilderInterface
                  • build_hf_to_local_param_map()
                • COLUMN_PARALLEL_SUFFIXES
                • ROW_PARALLEL_SUFFIXES
                • get_tp_shard_dim()
                • is_expert_param()
                • FFN_PROJ_WEIGHT_SUFFIXES
                • FFN_GROUPED_EXPERT_SUFFIXES
                • is_nccl_reshard_param()
                • _get_expert_tp_shard_dim()
                • _STR_TO_DTYPE
                • _restore_placement()
                • restore_refit_info_placements()
                • build_mesh_info()
                • get_placements()
                • _INDIVIDUAL_EXPERT_RE
                • group_expert_params_in_metadata()
                • _LAYER_RE
                • _MODEL_PREFIX_RE
                • _extract_layer_prefix()
                • _extract_layer_name()
                • check_nccl_reshard_refit_support()
                • build_nccl_reshard_refit_info()
          • nemo_rl.weight_sync.collective_weight_synchronizer
            • Module Contents
              • Classes
              • API
                • CollectiveWeightSynchronizer
                  • sync_weights()
                  • is_stale
                  • mark_stale()
                  • init_communicator()
                  • shutdown()
          • nemo_rl.weight_sync.nccl_reshard_weight_synchronizer
            • Module Contents
              • Classes
              • API
                • NcclReshardWeightSynchronizer
                  • _train_parallelism()
                  • _gen_parallelism()
                  • sync_weights()
                  • is_stale
                  • mark_stale()
                  • init_communicator()
                  • shutdown()
          • nemo_rl.weight_sync.http_weight_synchronizer
            • Module Contents
              • Classes
              • API
                • HTTPWeightSynchronizer
                  • sync_weights()
                  • is_stale
                  • mark_stale()
                  • init_communicator()
                  • shutdown()
                  • _compute_buffer_size()
          • nemo_rl.weight_sync.xferdtensor_python
            • Module Contents
              • Functions
              • Data
              • API
                • _mesh_rank_tensor()
                • _mesh_ranks()
                • _mesh_signature()
                • _mesh_coordinates()
                • _normalize_shard_dim()
                • _placement_signature()
                • _validate_layout()
                • _compute_shard_slices()
                • _rank_regions()
                • _intersect()
                • _local_slices()
                • _region_numel()
                • _local_tensor()
                • _tensor_metadata()
                • _destination_groups()
                • _build_exact_plan()
                • _PLAN_CACHE
                • _PLAN_CACHE_MAX_SIZE
                • _plan_geometry()
                • _SUBCOMM_CACHE
                • _INACTIVE_SUBCOMM_CACHE
                • _PROCESS_GROUP_COMM_IDS
                • _PROCESS_GROUP_FINALIZERS
                • _destroy_subcommunicator()
                • _evict_communicators()
                • _finalize_process_group()
                • _parent_communicator_key()
                • _active_replica_signature()
                • _get_replica_subcommunicator()
                • clear_xferdtensor_python_caches()
                • _validate_local_inputs()
                • _stage_rank_operations()
                • _exchange_exact_overlaps()
                • _broadcast_destination()
                • _xferdtensor_python_impl_v1()
                • _STRIPED_PLAN_CACHE
                • _STRIPED_PLAN_CACHE_MAX_SIZE
                • _ordered_replica_members()
                • _build_striped_targets()
                • _build_striped_transfers()
                • _max_p2p_peer_degree()
                • _region_view_is_contiguous()
                • _remote_transfers_avoid_double_staging()
                • _striped_geometry()
                • _destination_buffer()
                • _stage_striped_operations()
                • _allgather_destination()
                • xferdtensor_python_impl()
                • __all__
          • nemo_rl.weight_sync.xferdtensor
            • Module Contents
              • Classes
              • Functions
              • Data
              • API
                • _XFERDTENSOR_PATH_LOGGED
                • DTensorRef
                • _use_golden_api()
                • _use_python_api()
                • xferdtensor()
                • _flatten_mesh_ranks()
                • _get_tensor_meta()
                • _get_mesh_coords()
                • _compute_shard_slices()
                • xferdtensor_golden()
          • nemo_rl.weight_sync.factory
            • Module Contents
              • Functions
              • API
                • create_weight_synchronizer()
          • nemo_rl.weight_sync.checkpoint_engine_weight_synchronizer
            • Module Contents
              • Classes
              • Functions
              • Data
              • API
                • _MEBIBYTE
                • _flatten_metadata()
                • _sort_ranked_metadata()
                • _ordered_generation_metadata()
                • CheckpointEngineWeightSynchronizer
                  • _policy
                  • _generation
                  • _checkpoint_engine_config
                  • _stale
                  • _checkpoint_engine_ready
                  • _bucket_size_bytes
                  • init_communicator()
                  • is_stale
                  • mark_stale()
                  • _release_after_refit()
                  • _run_policy()
                  • _generation_rpc()
                  • _run_generation()
                  • _resolve_bucket_size_bytes()
                  • _ensure_checkpoint_engine_ready()
                  • sync_weights()
                  • shutdown()
          • nemo_rl.weight_sync.checkpoint_engine_config
            • Module Contents
              • Functions
              • API
                • checkpoint_engine_refit_config()
          • nemo_rl.weight_sync.vllm_remote_sparse_weight_synchronizer
            • Module Contents
              • Classes
              • Functions
              • Data
              • API
                • _REMOTE_SPARSE_TRANSPORTS
                • validate_vllm_remote_sparse_refit()
                • VllmRemoteSparseWeightSynchronizer
                  • sync_weights()
                  • is_stale
                  • mark_stale()
                  • _run_policy_workers()
                  • _run_generation_workers()
                  • _request_receivers()
                  • start_baseline()
                  • _merge_refit_info()
                  • init_communicator()
                  • shutdown()
          • nemo_rl.weight_sync.ipc_weight_synchronizer
            • Module Contents
              • Classes
              • API
                • IPCWeightSynchronizer
                  • sync_weights()
                  • is_stale
                  • mark_stale()
                  • init_communicator()
                  • shutdown()
                  • _compute_buffer_size()
        • Package Contents
          • Data
          • API
            • __all__
      • nemo_rl.data_plane
        • Subpackages
          • nemo_rl.data_plane.adapters
            • Submodules
              • nemo_rl.data_plane.adapters.transfer_queue
                • Module Contents
                  • Classes
                  • Functions
                  • Data
                  • API
                    • _get_local_node_ip()
                    • _mooncake_transport_config()
                    • _connect_existing()
                    • _TQ_RUNTIME_ENV_PATCHED
                    • _resolve_tq_pin()
                    • _patch_tq_actor_runtime_env()
                    • _init_tq()
                    • _assert_no_key_loss()
                    • _promote_1d_leaves()
                    • _from_wire()
                    • TQDataPlaneClient
                      • register_partition()
                      • claim_meta()
                      • get_data()
                      • check_consumption_status()
                      • put_samples()
                      • get_samples()
                      • clear_samples()
                      • close()
              • nemo_rl.data_plane.adapters.noop
                • Module Contents
                  • Classes
                  • Functions
                  • API
                    • _reject_non_tensor_leaves()
                    • _Partition
                      • fields
                      • num_samples
                      • consumer_tasks
                      • grpo_group_size
                      • enums
                      • rows
                      • tags
                      • consumed
                    • NoOpDataPlaneClient
                      • register_partition()
                      • claim_meta()
                      • get_data()
                      • check_consumption_status()
                      • put_samples()
                      • get_samples()
                      • clear_samples()
                      • close()
        • Submodules
          • nemo_rl.data_plane.schema
            • Module Contents
              • Functions
              • Data
              • API
                • Layout
                • MICRO_BATCH_INDICES
                • MICRO_BATCH_LENGTHS
                • ELEM_COUNTS_PER_GB
                • GLOBAL_FORWARD_PAD_SEQLEN
                • INPUT_IDS
                • INPUT_LENGTHS
                • SAMPLE_MASK
                • META_IDX
                • DP_TRAIN_FIELDS
                • LP_SEED_FIELDS
                • DP_CALIB_INPUT_FIELDS
                • ROUTED_EXPERTS_FIELD
                • PROMOTE_1D_FIELDS
                • fields_with_optional_routed_experts()
          • nemo_rl.data_plane.interfaces
            • Module Contents
              • Classes
              • API
                • DataPlaneConfig
                  • enabled
                  • impl
                  • backend
                  • storage_capacity
                  • num_storage_units
                  • claim_meta_poll_interval_s
                  • global_segment_size
                  • local_buffer_size
                  • controller_address
                  • ack_timeout_ms
                  • observability
                • ObservabilityConfig
                  • enabled
                  • callback
                • KVBatchMeta
                  • partition_id
                  • task_name
                  • sample_ids
                  • fields
                  • sequence_lengths
                  • extra_info
                  • tags
                  • __post_init__()
                  • size
                  • stamp_tags()
                  • _replace()
                  • subset()
                  • slice()
                  • concat()
                  • drop()
                  • with_fields()
                • DataPlaneClient
                  • register_partition()
                  • claim_meta()
                  • get_data()
                  • check_consumption_status()
                  • put_samples()
                  • get_samples()
                  • clear_samples()
                  • close()
          • nemo_rl.data_plane.column_io
            • Module Contents
              • Functions
              • Data
              • API
                • TOKEN_ALIGNED_FIELDS
                • round_up()
                • read_columns()
                • write_columns()
                • kv_first_write()
          • nemo_rl.data_plane.observability
            • Module Contents
              • Classes
              • Functions
              • Data
              • API
                • EventStatus
                • DataPlaneEvent
                  • op
                  • partition_id
                  • n_keys
                  • n_bytes
                  • wall_ms
                  • status
                • logger
                • _td_bytes()
                • log_event()
                • DataPlaneStats
                  • total_bytes
                  • total_keys
                  • total_ops
                  • bytes_outstanding
                  • peak_bytes_outstanding
                  • max_bytes_per_key_seen
                  • last_put_bytes_per_key
                • MetricsDataPlaneClient
                  • snapshot()
                  • bytes_outstanding_by_partition()
                  • _record_put()
                  • _record_clear()
                  • _run()
                  • _emit()
                  • register_partition()
                  • claim_meta()
                  • get_data()
                  • check_consumption_status()
                  • put_samples()
                  • get_samples()
                  • clear_samples()
                  • close()
          • nemo_rl.data_plane.preshard
            • Module Contents
              • Functions
              • API
                • shard_meta_for_dp()
          • nemo_rl.data_plane.factory
            • Module Contents
              • Functions
              • API
                • build_data_plane_client()
          • nemo_rl.data_plane.codec
            • Module Contents
              • Functions
              • API
                • to_nested_by_length()
                • stack_or_nest()
                • unwrap_wire_stripped_payload()
                • pack_jagged_fields()
                • pack_per_token_field()
                • response_from_nested()
                • materialize()
          • nemo_rl.data_plane.worker_mixin
            • Module Contents
              • Classes
              • Functions
              • Data
              • API
                • FetchPolicy
                • _broadcast_batched_data_dict()
                • TQWorkerMixin
                  • _dp_client
                  • setup_data_plane()
                  • _require_dp_client()
                  • _get_replica_group()
                  • _pad_value_dict()
                  • _forward_pad_seqlen()
                  • _fetch()
                  • _apply_packing_prep()
                  • _attach_or_repack_pack_metadata()
                  • _local_coords()
                  • _is_replica_leader()
                  • _write_back()
                  • _write_back_result_field()
                  • train_presharded()
                  • get_logprobs_presharded()
                  • get_reference_policy_logprobs_presharded()
                  • begin_train_step_presharded()
                  • train_microbatch_presharded()
                  • finish_train_step_presharded()
                  • abort_train_step_presharded()
        • Package Contents
          • Data
          • API
            • __all__
      • nemo_rl.environments
        • Submodules
          • nemo_rl.environments.vlm_environment
            • Module Contents
              • Classes
              • Functions
              • API
                • VLMEnvConfig
                  • num_workers
                  • stop_strings
                  • reward_functions
                • _mute_output()
                • VLMVerifyWorker
                  • verify()
                • VLMEnvironmentMetadata
                  • ground_truth
                • VLMEnvironment
                  • shutdown()
                  • step()
                  • global_post_process_and_metrics()
          • nemo_rl.environments.interfaces
            • Module Contents
              • Classes
              • Data
              • API
                • MetadataT
                • EnvironmentReturn
                  • observations
                  • metadata
                  • next_stop_strings
                  • rewards
                  • terminateds
                  • answers
                • EnvironmentInterface
                  • step()
                  • global_post_process_and_metrics()
          • nemo_rl.environments.nemo_gym
            • Module Contents
              • Classes
              • Functions
              • Data
              • API
                • _ROUTED_EXPERTS_DTYPES
                • DEFAULT_INVALID_TOOL_CALL_PATTERNS
                • DEFAULT_THINKING_TAGS
                • _has_nan_generation_logprobs()
                • get_nemo_gym_uv_cache_dir()
                • get_nemo_gym_venv_dir()
                • NemoGymConfig
                  • model_name
                  • base_urls
                  • initial_global_config_dict
                  • port_range_low
                  • port_range_high
                  • invalid_tool_call_patterns
                  • thinking_tags
                  • require_routed_experts
                  • routed_experts_dtype
                  • use_fastokens
                  • tokenizer_config
                • _detect_invalid_tool_call_and_malformed_thinking()
                • _IMAGE_SRC_PREFIXES
                • _looks_like_image_src()
                • _extract_input_images_from_message()
                • _index_per_turn_images()
                • _attach_multimodal_data_to_user_message()
                • NemoGym
                  • _spinup()
                  • run_rollouts()
                  • _postprocess_nemo_gym_to_nemo_rl_result()
                  • shutdown()
                  • step()
                  • global_post_process_and_metrics()
                • extract_reward_components()
                • build_reward_component_columns()
                • validate_reward_components_match_scalar()
                • setup_nemo_gym_config()
                • spinup_nemo_gym_actor()
          • nemo_rl.environments.metrics
            • Module Contents
              • Functions
              • API
                • calculate_pass_rate_per_prompt()
          • nemo_rl.environments.reward_model_environment
            • Module Contents
              • Classes
              • API
                • RewardModelEnvironmentConfig
                  • enabled
                  • model_name
                  • precision
                  • batch_size
                  • checkpoint_path
                  • logprob_batch_size
                  • resources
                  • dtensor_cfg
                  • dynamic_batching
                  • sequence_packing
                  • max_grad_norm
                  • generation
                • RewardModelEnvironment
                  • DEFAULT_PY_EXECUTABLE
                  • preprocess_data()
                  • step()
                  • global_post_process_and_metrics()
                  • shutdown()
                  • __del__()
          • nemo_rl.environments.code_environment
            • Module Contents
              • Classes
              • API
                • CodeEnvConfig
                  • num_workers
                  • terminate_on_evaluation
                • CodeEnvMetadata
                  • context
                  • working_dir
                • CodeExecutionWorker
                  • sanitize()
                  • format_result()
                  • execute()
                  • chdir()
                  • safe_open()
                  • safe_import()
                • CodeEnvironment
                  • step()
                  • shutdown()
                  • global_post_process_and_metrics()
          • nemo_rl.environments.code_jaccard_environment
            • Module Contents
              • Classes
              • API
                • CodeJaccardEnvConfig
                  • num_workers
                  • stop_strings
                • CodeJaccardEnvironmentMetadata
                  • ground_truth
                • CodeJaccardVerifyWorker
                  • verify()
                  • _calculate_preference_score()
                • CodeJaccardEnvironment
                  • shutdown()
                  • step()
                  • global_post_process_and_metrics()
          • nemo_rl.environments.rewards
            • Module Contents
              • Functions
              • Data
              • API
                • math_verify_func
                • boxed
                • math_expression_reward()
                • format_reward()
                • exact_answer_alphanumeric_reward()
                • bbox_giou_reward()
                • geo3k_reward()
                • combine_reward_functions()
          • nemo_rl.environments.dapo_math_verifier
            • Module Contents
              • Functions
              • Data
              • API
                • last_boxed_only_string()
                • remove_boxed()
                • SUBSTITUTIONS
                • REMOVED_EXPRESSIONS
                • normalize_final_answer()
                • is_correct_minerva()
                • is_correct_strict_box()
                • verify()
                • compute_score()
          • nemo_rl.environments.math_environment
            • Module Contents
              • Classes
              • Functions
              • API
                • MathEnvConfig
                  • num_workers
                  • stop_strings
                  • verifier_type
                  • math_verify_impl
                • _mute_output()
                • HFVerifyWorker
                  • verify()
                • MultilingualMultichoiceVerifyWorker
                  • verify()
                • EnglishMultichoiceVerifyWorker
                  • verify()
                • HFMultiRewardVerifyWorker
                  • REWARD_NAMES
                  • verify()
                • MathEnvironmentMetadata
                  • ground_truth
                  • extracted_answer
                • BaseMathEnvironment
                  • WORKER_CLASS_DICT
                  • shutdown()
                  • global_post_process_and_metrics()
                • MathEnvironment
                  • WORKER_CLASS_DICT
                  • step()
                • MathMultiRewardEnvironment
                  • WORKER_CLASS_DICT
                  • step()
          • nemo_rl.environments.utils
            • Module Contents
              • Classes
              • Functions
              • Data
              • API
                • EnvRegistryEntry
                  • actor_class_fqn
                  • default_processor
                • ENV_REGISTRY
                • chunk_list_to_workers()
                • create_env()
                • register_env()
      • nemo_rl.modelopt
        • Subpackages
          • nemo_rl.modelopt.models
            • Subpackages
              • nemo_rl.modelopt.models.policy
                • Subpackages
                  • nemo_rl.modelopt.models.policy.workers
                    • Submodules
                      • nemo_rl.modelopt.models.policy.workers.megatron_quant_policy_worker
                        • Module Contents
                          • Classes
                          • Functions
                          • API
                            • _quant_checkpoint_cache_suffix()
                            • _find_other_quant_checkpoint_caches()
                            • _warn_if_other_quant_checkpoint_caches()
                            • _set_quantization_model_specs()
                            • MegatronQuantPolicyWorker
                              • maybe_init_zmq()
                              • _quantize()
                              • _patch_validate_model_paths()
                              • _patch_setup_model_and_optimizer()
                              • _restore_modelopt_state_pre_load()
                              • hide_tensor_quantizers()
                              • enable_forward_pre_hook()
                              • disable_forward_pre_hook()
                              • disable_quantization()
                              • _hide_extra_state()
                              • use_reference_model()
                              • without_model_config()
                              • get_quantizer_stats()
                              • generate()
                              • save_checkpoint()
                              • _use_real_quant_refit()
                              • _get_real_quant_mode()
                              • _iter_real_quant_refit_params()
                              • _find_weight_quantizer()
                              • _iter_hf_input_amax_names()
                              • _get_enabled_input_amax()
                              • _iter_input_quantizer_amax_params()
                              • _iter_params_with_optional_kv_scales()
                      • nemo_rl.modelopt.models.policy.workers.dtensor_quant_policy_worker
                        • Module Contents
                          • Classes
                          • API
                            • DTensorQuantPolicyWorker
                      • nemo_rl.modelopt.models.policy.workers.dtensor_quant_policy_worker_v2
                        • Module Contents
                          • Classes
                          • API
                            • DTensorQuantPolicyWorkerV2
                      • nemo_rl.modelopt.models.policy.workers.utils
                        • Module Contents
                          • Classes
                          • Functions
                          • Data
                          • API
                            • MAX_SEQ_LEN
                            • MAX_OUTPUT_LEN
                            • DEFAULT_CALIB_BATCH_SIZE
                            • DEFAULT_CALIB_SAMPLE_LENGTH
                            • symlink_pre_quantized_model()
                            • get_tokenizer()
                            • _DictDataset
                              • __getitem__()
                              • __len__()
                            • get_forward_loop_func()
                            • quantize_model()
                            • get_modelopt_checkpoint_dir()
                            • get_quantization_layer_spec()
                            • get_quantization_mamba_stack_spec()
              • nemo_rl.modelopt.models.generation
                • Submodules
                  • nemo_rl.modelopt.models.generation.vllm_quant_patch
                    • Module Contents
                      • Classes
                      • Functions
                      • API
                        • _tolerate_dummy_weight_nan_amax()
                        • _drop_nonclass_quant_registry_keys()
                        • _register_routed_experts_quant_module()
                        • _fakequant_run_prolog_worker()
                        • FakeQuantWorker
                          • determine_available_memory()
                          • compile_or_warm_up_model()
                  • nemo_rl.modelopt.models.generation.vllm_quant_worker
                    • Module Contents
                      • Classes
                      • Functions
                      • Data
                      • API
                        • _EXTRA_ENV_VARS
                        • _quant_cfg_for_worker_env()
                        • _configure_quant_engine_kwargs()
                        • VllmQuantGenerationWorker
                          • _create_engine()
                          • get_quantizer_stats()
                          • get_weight_snapshot()
                        • VllmQuantAsyncGenerationWorker
                          • _create_engine()
                          • get_quantizer_stats()
                          • get_weight_snapshot()
                  • nemo_rl.modelopt.models.generation.vllm_quant_backend
                    • Module Contents
                      • Classes
                      • Functions
                      • Data
                      • API
                        • _FUSED_MODELOPT_MOE_SUFFIXES
                        • _match_fused_modelopt_moe_weight()
                        • _w13_num_shards_from_state_dict_info()
                        • _batch_fused_modelopt_moe_weights()
                        • _detach_pending_layerwise_weights()
                        • _iter_modelopt_quant_modules()
                        • _modelopt_layerwise_reload_roots()
                        • _require_complete_modelopt_layerwise_reload()
                        • VllmQuantInternalWorkerExtension
                          • _nrl_w13_num_shards_by_prefix
                          • _nrl_modelopt_reload_roots
                          • maybe_init_zmq()
                          • _is_real_quant_model()
                          • _get_modelopt_reload_roots()
                          • _weight_update_lifecycle()
                          • _weight_update_errors_are_fatal()
                          • _synchronize_before_ipc_data_ack()
                          • prepare_refit_info()
                          • _patch_named_parameters_to_include_buffers()
                          • _attach_input_quantizer_amax_loaders()
                          • _load_weights()
                          • get_weight_snapshot()
                          • get_quantizer_stats()
                        • VllmQuantInternalWorkerExtensionWithCheckpointEngine
                  • nemo_rl.modelopt.models.generation.vllm_modelopt
                    • Module Contents
                      • Functions
                      • Data
                      • API
                        • NEMO_MODELOPT_W4A4
                        • NEMO_MODELOPT_W4A16
                        • _W4A4_ALGO
                        • _W4A16_ALGO
                        • _registered
                        • quantization_method_for_mode()
                        • _load_modelopt_moe_input_scale()
                        • _validated_w4a16_config()
                        • _canonicalize_nvfp4_scale_()
                        • register_nemo_modelopt_nvfp4()
        • Submodules
          • nemo_rl.modelopt.registry
            • Module Contents
              • Data
              • API
                • USE_SYSTEM_EXECUTABLE
                • MODELOPT_VLLM_EXECUTABLE
                • MODELOPT_AUTOMODEL_EXECUTABLE
                • MODELOPT_MCORE_EXECUTABLE
                • MODELOPT_ACTOR_REGISTRY
          • nemo_rl.modelopt.utils
            • Module Contents
              • Functions
              • Data
              • API
                • MODELOPT_REAL_QUANT_ZMQ_TIMEOUT_MS
                • _QUANT_IGNORE_NAME_SUFFIXES
                • DEFAULT_NVFP4_IGNORE
                • NVFP4RealQuantMode
                • _NVFP4_REAL_QUANT_MODES
                • _iter_quant_ignore_suffix_variants()
                • iter_quant_ignore_name_candidates()
                • matches_quant_ignore_pattern()
                • build_vllm_modelopt_nvfp4_config()
                • _resolve_effective_quantizer_formats()
                • _is_float_format()
                • _validate_nvfp4_quantizer_format()
                • resolve_nvfp4_real_quant_mode()
                • resolve_quant_cfg()
      • nemo_rl.algorithms
        • Subpackages
          • nemo_rl.algorithms.single_controller_utils
            • Submodules
              • nemo_rl.algorithms.single_controller_utils.config
                • Module Contents
                  • Classes
                  • Functions
                  • API
                    • AsyncRLConfig
                      • sampler
                      • recompute_kv_cache_after_weight_updates
                      • min_groups_for_streaming_train
                      • max_inflight_prompts
                      • max_buffered_rollouts
                      • diagnostics
                    • MasterConfig
                      • policy
                      • loss_fn
                      • env
                      • data
                      • grpo
                      • logger
                      • cluster
                      • checkpointing
                      • data_plane
                      • async_rl
                    • validate_sampler_buffer_capacity()
                    • validate_single_controller_config()
                    • AdvantageConfig
                      • output_field
                      • prompt_ids_field
                      • reward_field
                      • token_mask_field
                      • sample_mask_field
                      • repeated_batch_fields
                      • policy_logprobs_field
                      • reference_logprobs_field
              • nemo_rl.algorithms.single_controller_utils.setup
                • Module Contents
                  • Classes
                  • Functions
                  • API
                    • SingleControllerActorArgs
                      • gen_handle
                      • trainer_handle
                      • env_handles
                      • train_cluster
                      • inference_cluster
                      • dp_client
                      • dataloader
                      • weight_synchronizer
                      • advantage_estimator
                      • loss_fn
                      • rollout_manager
                      • tq_buffer
                      • partition_id
                    • _build_clusters()
                    • _build_generation()
                    • _finish_deferred_generation()
                    • _build_trainer()
                    • _spinup_gym()
                    • _generation_max_seq_len()
                    • _clamp_max_num_steps()
                    • _maybe_inject_megatron_train_iters()
                    • setup_single_controller()
              • nemo_rl.algorithms.single_controller_utils.utils
                • Module Contents
                  • Functions
                  • Data
                  • API
                    • _MB_METRIC_MIN
                    • _MB_METRIC_MAX
                    • _MB_METRIC_MEAN
                    • aggregate_step_metrics()
                    • reduce_advantage_pump_metrics()
                    • tensor_field()
                    • squeeze_trailing_unit_dim()
                    • fields_for_put()
            • Package Contents
              • Data
              • API
                • __all__
          • nemo_rl.algorithms.x_token
            • Submodules
              • nemo_rl.algorithms.x_token.loss_utils
                • Module Contents
                  • Classes
                  • Functions
                  • Data
                  • API
                    • alignment_from_flat_batch()
                    • Fp32SparseMM
                      • forward()
                      • backward()
                    • chunk_log_prob_sums()
                    • chunk_average_finalize()
                    • chunk_average_log_probs()
                    • slice_sparse_projection_rows()
                    • project_student_to_teacher_vocab()
                    • select_teacher_topk_indices()
                    • LocalizedAlignment
                      • sample_mask
                      • student_chunk_id
                      • teacher_chunk_id
                      • pair_valid
                      • pair_is_correct
                      • student_input_ids
                      • student_token_mask
                    • localize_alignment()
                    • student_next_token_ce()
                    • ce_label_mask()
                    • next_token_accuracy()
                    • collect_overlapping_teacher_shards()
                    • assemble_teacher_logits_from_shards()
                    • _try_zero_copy_teacher_logits()
                    • rebuild_teacher_full_logits_from_ipc()
                    • valid_chunk_mask()
                    • parse_projection_file()
                    • _SPARSE_PROJECTION_CACHE
                    • _TOPK_PROJECTION_CACHE
                    • get_sparse_projection_matrix()
                    • get_topk_projection()
                    • _EXACT_TOKEN_MAP_CACHE
                    • build_exact_token_map()
                    • prepare_xtoken_cross_tokenizer_loss_input()
              • nemo_rl.algorithms.x_token.token_aligner
                • Module Contents
                  • Classes
                  • Functions
                  • Data
                  • API
                    • VISUAL_BYTE_MAP
                    • _MULTI_TOKEN_ARTIFACT_FIXES
                    • _UNICODE_FIXES
                    • _SPECIAL_TOKEN_MAP
                    • AlignmentPair
                      • s_tokens
                      • t_tokens
                      • s_start
                      • s_end
                      • t_start
                      • t_end
                      • is_correct
                    • AlignmentBatch
                      • pair_valid
                      • pair_is_correct
                      • student_exact_partition_mask
                      • teacher_exact_partition_mask
                      • student_chunk_id
                      • teacher_chunk_id
                      • num_chunks
                    • TokenAligner
                      • align()
                      • _drop_padding()
                      • _pairs_to_batch()
                      • _align_single()
                      • _remap_pairs_to_original()
                      • _align_with_anchors()
                      • _align_dp()
                      • _shift_pairs()
                      • _alignment_mask()
                      • _post_process_alignment()
                      • _build_pair_strings()
                      • _combine_consecutive_misaligned()
                      • _flatten_chunk()
                    • canonical_token()
                    • _canonicalize_sequence()
                    • _merge_encoding_artifacts()
                    • _get_byte_value()
                    • _merge_consecutive_bytes()
                    • _try_merge_byte_buffer()
                    • _strings_equal_flexible()
              • nemo_rl.algorithms.x_token.utils
                • Module Contents
                  • Functions
                  • API
                    • assert_teacher_student_batch_grid()
                    • assert_xtoken_ipc_node_local()
                    • pad_distillation_val_batch()
            • Package Contents
              • Data
              • API
                • __all__
          • nemo_rl.algorithms.loss
            • Submodules
              • nemo_rl.algorithms.loss.interfaces
                • Module Contents
                  • Classes
                  • API
                    • LossType
                      • TOKEN_LEVEL
                      • SEQUENCE_LEVEL
                    • MetricNormalizer
                      • TOKENS
                      • SEQUENCES
                      • NONE
                    • LossInputType
                      • LOGIT
                      • LOGPROB
                      • DISTILLATION
                      • DISTILLATION_CROSS_TOKENIZER
                      • DRAFT
                    • LossFunction
                      • loss_type
                      • input_type
                      • __call__()
              • nemo_rl.algorithms.loss.wrapper
                • Module Contents
                  • Classes
                  • Functions
                  • Data
                  • API
                    • Tensor
                    • SequencePackingLossWrapper
                      • __call__()
                    • SequencePackingFusionLossWrapper
                      • __call__()
                    • DraftLossWrapper
                      • __call__()
                    • wrap_loss_fn_with_input_preparation()
              • nemo_rl.algorithms.loss.loss_functions
                • Module Contents
                  • Classes
                  • Data
                  • API
                    • Tensor
                    • DraftCrossEntropyLossConfig
                      • vocab_parallel_group
                    • DraftCrossEntropyLossDataDict
                      • teacher_logits
                      • student_logits
                      • token_mask
                      • sample_mask
                      • student_vocab_indices
                    • DraftCrossEntropyLossFn
                      • loss_type
                      • input_type
                      • __call__()
                    • ClippedPGLossConfig
                      • disable_ppo_ratio
                      • token_level_loss
                      • sequence_level_importance_ratios
                      • ratio_clip_min
                      • ratio_clip_max
                      • ratio_clip_c
                      • reference_policy_kl_penalty
                      • reference_policy_kl_type
                      • kl_input_clamp_value
                      • kl_output_clamp_value
                      • use_kl_in_reward
                      • use_importance_sampling_correction
                      • truncated_importance_sampling_type
                      • truncated_importance_sampling_ratio
                      • truncated_importance_sampling_ratio_min
                      • use_on_policy_kl_approximation
                      • force_on_policy_ratio
                      • use_cispo
                      • positive_example_nll_weight
                    • ClippedPGLossDataDict
                      • input_ids
                      • advantages
                      • prev_logprobs
                      • generation_logprobs
                      • reference_policy_logprobs
                      • token_mask
                      • sample_mask
                      • __extra__
                    • ClippedPGLossFn
                      • input_type
                      • __call__()
                    • NLLLossFn
                      • loss_type
                      • input_type
                      • __call__()
                    • PreferenceLossDataDict
                      • input_ids
                      • token_mask
                      • sample_mask
                    • PreferenceLossFn
                      • loss_type
                      • input_type
                      • split_output_tensor()
                      • _preference_loss()
                      • __call__()
                    • DPOLossConfig
                      • reference_policy_kl_penalty
                      • preference_loss_weight
                      • sft_loss_weight
                      • preference_average_log_probs
                      • sft_average_log_probs
                    • DPOLossDataDict
                      • input_ids
                      • reference_policy_logprobs
                      • token_mask
                      • sample_mask
                    • DPOLossFn
                      • loss_type
                      • input_type
                      • _dpo_loss()
                      • __call__()
                    • DistillationLossConfig
                      • kl_type
                      • mixed_kl_weight
                      • zero_outside_topk
                    • DistillationLossDataDict
                      • input_ids
                      • input_lengths
                      • token_mask
                      • sample_mask
                      • teacher_topk_logits
                      • teacher_topk_indices
                    • DistillationLossFn
                      • loss_type
                      • input_type
                      • __call__()
                    • MseValueLossConfig
                      • scale
                      • cliprange
                    • MseValueLossFn
                      • input_type
                      • __call__()
                    • CrossTokenizerDistillationLossConfig
                      • gold_loss
                      • xtoken_loss
                      • temperature
                      • vocab_topk
                      • uncommon_topk
                      • reverse_kl
                      • exact_token_match_only
                      • kl_loss_weight
                      • ce_loss_scale
                      • dynamic_loss_scaling
                      • kd_loss_mode
                      • normalize_teacher_by_vocab
                      • alpha
                      • sum_weights_metric
                      • student_vocab_size
                      • teacher_vocab_sizes
                      • projection_matrix_paths
                      • teacher_weights
                      • teacher_gold_loss
                      • teacher_xtoken_loss
                    • CrossTokenizerDistillationLossDataDict
                      • input_ids
                      • input_lengths
                      • token_mask
                      • sample_mask
                    • CrossTokenizerDistillationLossFn
                      • loss_type
                      • input_type
                      • _teacher_is_same_vocab()
                      • __call__()
                      • _resolve_gold_xtoken()
                      • _compute_teacher_kd()
                      • _compute_same_vocab_kl()
                      • _direct_topk_kl()
                      • _direct_full_vocab_kl()
                      • _same_vocab_masked_kl()
                      • _sum_kd()
                      • _averaged_logits_kd()
                      • _dp_global_masked_mean()
                      • _select_teacher_kd()
                      • _teacher_score_inputs()
                      • _compute_dynamic_weights()
                      • _teacher_weight_score()
                      • _compute_p_kl()
                      • _compute_gold()
                      • _compute_ce()
              • nemo_rl.algorithms.loss.utils
                • Module Contents
                  • Functions
                  • API
                    • prepare_loss_input()
                    • _pack_input_ids()
                    • prepare_packed_loss_input()
            • Package Contents
              • Data
              • API
                • __all__
          • nemo_rl.algorithms.async_utils
            • Submodules
              • nemo_rl.algorithms.async_utils.interfaces
                • Module Contents
                  • Classes
                  • API
                    • ReplayBufferProtocol
                      • add()
                      • sample()
                      • size()
                      • clear()
                      • state_dict()
                      • load_state_dict()
                      • get_trajectories_needed()
                      • has_complete_batch()
              • nemo_rl.algorithms.async_utils.replay_buffer
                • Module Contents
                  • Classes
                  • API
                    • ReplayBufferImpl
                      • _rollout_metrics_turn_count_for_diagnostics()
                      • add()
                      • get_debug_info()
                      • get_last_target_weight_already_generated()
                      • get_existing_target_weights()
                      • _remove_indices()
                      • sample()
                      • size()
                      • clear()
                      • state_dict()
                      • load_state_dict()
                      • _prepare_for_training_step()
                      • _is_valid_for_target()
                      • _remove_stale_trajectories()
                      • _count_for_target()
                      • _truncate_to_max_size()
                      • get_trajectories_needed()
                      • has_complete_batch()
                      • _remove_incomplete_target_steps()
                    • ReplayBuffer
                    • TQReplayBuffer
                      • reserve()
                      • commit()
                      • remove_group()
                      • remove()
                      • size()
                      • __len__()
                      • _call_dp()
              • nemo_rl.algorithms.async_utils.trajectory_collector
                • Module Contents
                  • Classes
                  • Data
                  • API
                    • TokenizerType
                    • _MAX_NEMO_GYM_STREAM_RETRIES
                    • _NEMO_GYM_RETRY_DELAY_BASE_SECONDS
                    • _REPLAY_BUFFER_MAX_BACKOFF_SECONDS
                    • AsyncTrajectoryCollector
                      • _calculate_target_weights()
                      • _get_next_target_for_generation()
                      • set_weight_version()
                      • _should_pause_for_generation_limits()
                      • start_collection()
                      • is_data_exhausted()
                      • get_status()
                      • _collection_loop()
                      • _stamp_nemo_gym_task_indices()
                      • _process_batch()
                      • get_weight_version()
                      • pause()
                      • resume()
                      • prepare_for_refit()
                      • resume_after_refit()
                      • wait_for_pending_generations()
                      • get_dataloader_state()
                      • get_efficiency_metrics()
                      • get_rollouts_state()
                      • _cleanup_finished_threads()
                      • _release_target()
                      • _compute_teacher_logprobs()
                      • _iter_rollout_groups()
                      • _run_rollout_batch_worker()
                      • _build_task_index_map()
                      • _enqueue_rollout_group()
                      • _collect_rollout_batch()
              • nemo_rl.algorithms.async_utils.staleness_sampler
                • Module Contents
                  • Classes
                  • Functions
                  • Data
                  • API
                    • _GATE_POLL_SECONDS
                    • PromptGroupSampler
                      • admit()
                      • select()
                      • evict()
                      • is_on_policy
                      • required_buffer_capacity()
                    • BaseSampler
                      • admit()
                      • select()
                      • evict()
                      • is_on_policy
                      • required_buffer_capacity()
                      • _eviction_window()
                      • _validate_group_bounds()
                      • _finalize_selection()
                    • WindowedSampler
                      • _eviction_window()
                      • admit()
                      • select()
                    • _gated_required_buffer_capacity()
                    • _GatedSampler
                      • _eviction_window()
                      • required_buffer_capacity()
                      • admit()
                      • _stamp()
                    • WeightFifoSampler
                      • select()
                    • InOrderSampler
                      • _stamp()
                      • select()
                      • evict()
                    • WindowedSamplerConfig
                      • name
                      • max_staleness_versions
                      • sample_freshest_first
                    • WeightFifoSamplerConfig
                      • name
                      • max_staleness_versions
                    • InOrderSamplerConfig
                      • name
                      • max_lookahead_versions
                    • CustomSamplerConfig
                      • name
                      • target
                    • SamplerConfig
                    • required_buffer_capacity_for_config()
                    • create_sampler()
            • Package Contents
              • Data
              • API
                • __all__
        • Submodules
          • nemo_rl.algorithms.logits_sampling_utils
            • Module Contents
              • Classes
              • Functions
              • Data
              • API
                • TOP_K_TOP_P_CHUNK_SIZE
                • TrainingSamplingParams
                  • top_k
                  • top_p
                  • temperature
                • _need_top_k_filtering()
                • _need_top_p_filtering()
                • need_top_k_or_top_p_filtering()
                • _apply_top_k_only_fn()
                • _apply_top_k_top_p_fn()
                • _ApplyTopKTopP
                  • forward()
                  • backward()
                • apply_top_k_top_p()
          • nemo_rl.algorithms.metric_utils
            • Module Contents
              • Classes
              • Functions
              • API
                • SetupTimingMetrics
                  • vllm_init_time_s
                  • sglang_init_time_s
                  • trtllm_init_time_s
                  • megatron_generation_init_time_s
                  • generation_init_time_s
                  • generation_init_reserve_time_s
                  • generation_init_load_time_s
                  • policy_init_time_s
                  • nemo_gym_init_time_s
                  • collective_init_time_s
                  • parallel_wall_time_s
                  • parallel_init_enabled
                  • teacher_reservation_time_s
                  • teacher_model_init_time_s
                  • teacher_init_time_s
                  • vllm_checkpoint_engine_init_time_s
                  • total_setup_time_s
                  • worker_setup_time_s
                  • other_setup_time_s
                  • extras
                  • to_metrics_dict()
                • print_setup_timing_summary()
          • nemo_rl.algorithms.rm
            • Module Contents
              • Classes
              • Functions
              • API
                • RMSaveState
                  • epoch
                  • step
                  • total_steps
                  • consumed_samples
                  • total_valid_tokens
                • _initial_rm_save_state()
                • _get_rm_save_state()
                • RMConfig
                  • max_num_steps
                  • max_num_epochs
                  • val_period
                  • val_batches
                  • val_global_batch_size
                  • val_micro_batch_size
                  • val_at_start
                  • val_at_end
                  • seed
                • MasterConfig
                  • policy
                  • data
                  • rm
                  • logger
                  • cluster
                  • checkpointing
                • RMValMetrics
                  • loss
                  • accuracy
                  • rewards_chosen_mean
                  • rewards_rejected_mean
                  • num_valid_samples
                • setup()
                • validate()
                • validate_one_dataset()
                • rm_train()
          • nemo_rl.algorithms.ppo
            • Module Contents
              • Classes
              • Functions
              • Data
              • API
                • TokenizerType
                • AdvEstimatorConfig
                  • name
                  • gae_lambda
                  • gae_gamma
                  • normalize_advantages
                  • gae_lambda_value
                  • gae_lambda_policy
                  • length_adaptive_alpha
                • PPOConfig
                  • num_prompts_per_step
                  • num_generations_per_prompt
                  • max_num_epochs
                  • max_num_steps
                  • max_rollout_turns
                  • val_period
                  • val_batch_size
                  • val_at_start
                  • val_at_end
                  • max_val_samples
                  • skip_reference_policy_logprobs_calculation
                  • seed
                  • overlong_filtering
                  • use_dynamic_sampling
                  • dynamic_sampling_max_gen_batches
                  • batch_multiplier
                  • ppo_epochs
                  • reward_shaping
                  • reward_scaling
                  • calculate_advantages_on_gpu
                  • adv_estimator
                  • policy_training_start_step
                  • seq_logprob_error_threshold
                • PPOSaveState
                  • consumed_samples
                  • current_step
                  • current_epoch
                  • total_steps
                  • total_valid_tokens
                  • val_reward
                • _default_ppo_save_state()
                • _apply_ppo_seq_logprob_error_masking()
                • PPOLoggerConfig
                  • num_val_samples_to_print
                • MasterConfig
                  • policy
                  • value
                  • loss_fn
                  • value_loss_fn
                  • env
                  • data
                  • ppo
                  • logger
                  • cluster
                  • checkpointing
                • setup()
                • dynamic_sampling()
                • _create_advantage_estimator()
                • ppo_train()
                • validate()
          • nemo_rl.algorithms.opd
            • Module Contents
              • Classes
              • Functions
              • API
                • TeacherResourceConfig
                  • tensor_model_parallel_size
                  • pipeline_model_parallel_size
                  • context_parallel_size
                  • expert_model_parallel_size
                  • num_nodes
                  • gpus_per_node
                  • precision
                  • micro_batch_size
                  • megatron_cfg_overrides
                • NonColocatedTeachersConfig
                  • enabled
                  • default_teacher_cfg
                  • teacher_overrides
                • OnPolicyDistillationConfig
                  • enabled
                  • teacher_model_by_agent_name
                  • default_teacher_alias
                  • strict_agent_name_match
                  • deduplicate_shared_teacher_checkpoints
                  • non_colocated_teachers
                • _opd_cfg()
                • is_opd_enabled()
                • is_non_colocated_teachers_enabled()
                • _skip_prev_logprobs()
                • assert_prev_logprobs_available()
                • resolve_reference_aliases()
                • get_teacher_routing_metrics()
                • teacher_seq_pad_multiple()
                • _validate_default_teacher_alias()
                • reserve_teacher_clusters()
                • create_teacher_worker_groups()
          • nemo_rl.algorithms.single_controller
            • Module Contents
              • Classes
              • Data
              • API
                • Generation
                • SingleControllerActor
                  • run()
                  • ping()
                  • _ray_get()
                  • _call_dp()
                  • _rollout_pump()
                  • _train_pump()
                  • _sync_weights()
                  • _advantage_stage()
                  • _advantage_input_fields()
          • nemo_rl.algorithms.sft
            • Module Contents
              • Classes
              • Functions
              • API
                • SFTSaveState
                  • epoch
                  • step
                  • total_steps
                  • consumed_samples
                  • total_valid_tokens
                • _initial_sft_save_state()
                • _get_sft_save_state()
                • SFTConfig
                  • max_num_steps
                  • max_num_epochs
                  • val_period
                  • val_batches
                  • val_global_batch_size
                  • val_micro_batch_size
                  • val_at_start
                  • val_at_end
                  • seed
                  • only_unmask_final
                • MasterConfig
                  • policy
                  • data
                  • sft
                  • logger
                  • cluster
                  • checkpointing
                • setup()
                • validate()
                • sft_train()
          • nemo_rl.algorithms.reward_functions
            • Module Contents
              • Classes
              • Functions
              • Data
              • API
                • Tensor
                • RewardShapingConfig
                  • enabled
                  • overlong_buffer_length
                  • overlong_buffer_penalty
                  • max_response_length
                  • stop_properly_penalty_coef
                • apply_reward_shaping()
          • nemo_rl.algorithms.advantage_estimator
            • Module Contents
              • Classes
              • API
                • AdvEstimatorConfig
                  • name
                  • normalize_rewards
                  • use_leave_one_out_baseline
                  • reward_weights
                  • minus_baseline
                • GRPOAdvantageEstimator
                  • compute_advantage()
                • GDPOAdvantageEstimator
                  • compute_advantage()
                • ReinforcePlusPlusAdvantageEstimator
                  • compute_advantage()
                • RawRewardAdvantageEstimator
                  • compute_advantage()
                • GeneralizedAdvantageEstimator
                  • _reward_whiten()
                  • _build_token_level_rewards()
                  • _resolve_lambda_policy()
                  • _resolve_lambda_value()
                  • compute_advantage()
                  • _compute_gae()
                • OPDAdvantageEstimator
                  • compute_advantage()
                  • _compute_metrics()
          • nemo_rl.algorithms.grpo
            • Module Contents
              • Classes
              • Functions
              • Data
              • API
                • TokenizerType
                • _get_next_nemo_gym_task_index()
                • RewardScalingConfig
                  • enabled
                  • source_min
                  • source_max
                  • target_min
                  • target_max
                • AsyncGRPOConfig
                  • enabled
                  • max_trajectory_age_steps
                  • in_flight_weight_updates
                  • recompute_kv_cache_after_weight_updates
                • RewardPenaltyTokenIdsConfig
                  • unwanted
                  • think_open
                  • think_close
                • RewardPenaltyConfig
                  • penalize_duplicated_reasoning
                  • penalize_empty_final_answer
                  • penalize_unwanted_tokens
                  • penalize_malformed_think_tag
                  • token_ids
                  • _require_unwanted_token_ids_when_penalized()
                • _REWARD_PENALTY_FLAGS
                • GRPOConfig
                  • num_prompts_per_step
                  • num_generations_per_prompt
                  • max_num_epochs
                  • max_num_steps
                  • max_rollout_turns
                  • normalize_rewards
                  • advantage_clip_low
                  • advantage_clip_high
                  • use_leave_one_out_baseline
                  • val_period
                  • val_start_at
                  • val_batch_size
                  • val_at_start
                  • val_at_end
                  • max_val_samples
                  • val_num_generations_per_prompt
                  • stop_at_validation_metric
                  • stop_at_validation_threshold
                  • skip_reference_policy_logprobs_calculation
                  • seed
                  • async_grpo
                  • overlong_filtering
                  • use_dynamic_sampling
                  • dynamic_sampling_max_gen_batches
                  • batch_multiplier
                  • reward_shaping
                  • reward_scaling
                  • calculate_advantages_on_gpu
                  • seq_logprob_error_threshold
                  • invalid_tool_call_advantage
                  • malformed_thinking_advantage
                  • adv_estimator
                • GRPOSaveState
                  • consumed_samples
                  • current_step
                  • current_epoch
                  • total_steps
                  • total_valid_tokens
                  • val_reward
                • _initial_grpo_save_state()
                • _get_grpo_save_state()
                • GRPOLoggerConfig
                  • num_val_samples_to_print
                • MasterConfig
                  • policy
                  • loss_fn
                  • env
                  • data
                  • grpo
                  • logger
                  • cluster
                  • checkpointing
                  • reward_penalties
                  • data_plane
                  • on_policy_distillation
                • setup()
                • dynamic_sampling()
                • scale_rewards()
                • extract_initial_prompt_messages()
                • add_grpo_token_loss_masks_and_generation_logprobs()
                • _resolve_message_level_advantage_penalties()
                • _raise_if_reward_penalties_enabled_without_nemo_gym()
                • _apply_message_level_advantage_penalties()
                • _apply_configured_message_level_advantage_penalties()
                • _should_use_async_rollouts()
                • _preserve_router_replay_routed_experts()
                • _build_async_grpo_train_data()
                • _apply_mask_sample_filter()
                • _should_use_nemo_gym()
                • _should_log_nemo_gym_responses()
                • _write_latest_checkpoint_status()
                • _get_effort_config()
                • _pad_teacher_logprobs()
                • _create_advantage_estimator()
                • _clip_grpo_advantages()
                • refit_policy_generation()
                • _initial_policy_generation_stale()
                • _log_mixed_rewards_and_advantages_information()
                • _placeholder_seq_logprob_error_metrics()
                • _validate_use_kl_in_reward_compat()
                • _resolve_logprob_skip_flags()
                • compute_and_apply_seq_logprob_error_masking()
                • _validation_stop_value()
                • _validation_early_stop_message()
                • grpo_train()
                • validate()
                • aggregate_rollout_metrics()
                • async_grpo_train()
          • nemo_rl.algorithms.distillation
            • Module Contents
              • Classes
              • Functions
              • Data
              • API
                • TokenizerType
                • DistillationConfig
                  • num_prompts_per_step
                  • num_generations_per_prompt
                  • max_rollout_turns
                  • max_num_steps
                  • max_num_epochs
                  • val_batch_size
                  • val_period
                  • val_at_start
                  • val_at_end
                  • max_val_samples
                  • topk_logits_k
                  • seed
                • DistillationSaveState
                  • total_steps
                  • current_epoch
                  • current_step
                  • val_reward
                  • consumed_samples
                  • total_valid_tokens
                • _initial_distillation_save_state()
                • _get_distillation_save_state()
                • MasterConfig
                  • policy
                  • teacher
                  • loss_fn
                  • env
                  • data
                  • distillation
                  • logger
                  • cluster
                  • checkpointing
                • check_vocab_equality()
                • setup()
                • distillation_train()
                • validate()
          • nemo_rl.algorithms.grpo_sync
            • Module Contents
              • Functions
              • API
                • _raise_if_message_level_advantage_penalties_enabled()
                • _train_fields_for_step()
                • _apply_dynamic_sampling()
                • validate_sync()
                • _compute_seq_logprob_error_metrics()
                • grpo_train_sync()
          • nemo_rl.algorithms.xtoken_off_policy_distillation
            • Module Contents
              • Classes
              • Functions
              • API
                • xtoken_non_student_seq_keys()
                • OffPolicyDistillationConfig
                  • num_prompts_per_step
                  • max_num_steps
                  • max_num_epochs
                  • seed
                  • val_period
                  • val_at_start
                  • val_at_end
                • OffPolicyDistillationSaveState
                  • current_epoch
                  • current_step
                  • total_steps
                  • consumed_samples
                  • total_valid_tokens
                  • val_loss
                • _default_off_policy_distillation_save_state()
                • TeacherConfig
                  • projection_matrix_path
                  • weight
                  • gold_loss
                  • xtoken_loss
                  • policy_config()
                • MasterConfig
                  • policy
                  • teachers
                  • loss_fn
                  • data
                  • distillation
                  • logger
                  • cluster
                  • checkpointing
                • setup()
                • export_teacher_logits_and_pack()
                • xtoken_off_policy_distillation_train()
                • validate()
          • nemo_rl.algorithms.dpo
            • Module Contents
              • Classes
              • Functions
              • API
                • DPOSaveState
                  • epoch
                  • step
                  • total_steps
                  • consumed_samples
                  • total_valid_tokens
                • _initial_dpo_save_state()
                • _get_dpo_save_state()
                • DPOConfig
                  • max_num_epochs
                  • max_num_steps
                  • val_period
                  • val_batches
                  • val_global_batch_size
                  • val_micro_batch_size
                  • val_at_start
                  • val_at_end
                  • seed
                  • reference_policy_kl_penalty
                  • preference_average_log_probs
                  • sft_average_log_probs
                  • preference_loss_weight
                  • sft_loss_weight
                • MasterConfig
                  • policy
                  • data
                  • dpo
                  • logger
                  • cluster
                  • checkpointing
                • DPOValMetrics
                  • loss
                  • sft_loss
                  • preference_loss
                  • accuracy
                  • rewards_chosen_mean
                  • rewards_rejected_mean
                  • num_valid_samples
                  • global_valid_seqs
                  • global_valid_toks
                • setup()
                • add_ref_logprobs_to_data()
                • validate()
                • validate_one_dataset()
                • dpo_train()
          • nemo_rl.algorithms.utils
            • Module Contents
              • Functions
              • Data
              • API
                • get_gdpo_reward_component_keys()
                • calculate_kl()
                • calculate_baseline_and_std_per_prompt()
                • surpress_user_warnings()
                • masked_mean()
                • mask_out_neg_inf_logprobs()
                • masked_var()
                • set_seed()
                • get_tokenizer()
                • maybe_pad_last_batch()
                • print_performance_metrics()
                • log_generation_metrics_to_wandb()
                • WALL_CLOCK_EFFICIENCY_CATEGORIES
                • THREAD_ACCUMULATED_EFFICIENCY_CATEGORIES
                • EFFICIENCY_CATEGORIES
                • print_efficiency_summary()
      • nemo_rl.distributed
        • Submodules
          • nemo_rl.distributed.model_utils
            • Module Contents
              • Classes
              • Functions
              • API
                • _compute_distributed_log_softmax_with_grad()
                • _compute_distributed_log_softmax()
                • _compute_distributed_softmax()
                • DistributedLogprob
                  • forward()
                  • backward()
                • DistributedCrossEntropy
                  • forward()
                  • backward()
                • ChunkedDistributedLogprob
                  • forward()
                  • backward()
                • DistributedLogprobWithSampling
                  • forward()
                  • backward()
                • ChunkedDistributedLogprobWithSampling
                  • forward()
                  • backward()
                • ChunkedDistributedGatherLogprob
                  • forward()
                  • backward()
                • dtensor_from_parallel_logits_to_logprobs()
                • from_parallel_logits_to_logprobs()
                • from_parallel_logits_to_logprobs_packed_sequences()
                • _get_tokens_on_this_cp_rank()
                • allgather_cp_sharded_tensor()
                • _AllReduceSum
                  • forward()
                  • backward()
                • group_all_reduce_sum_with_grad()
                • group_all_reduce_sum()
                • AllGatherCPTensor
                  • forward()
                  • backward()
                • cp_load_balanced_to_contiguous()
                • cp_shift_next()
                • vocab_parallel_log_softmax()
                • vocab_parallel_full_log_softmax()
                • vocab_parallel_gather_columns()
                • vocab_parallel_argmax()
                • get_logprobs_from_vocab_parallel_logits()
                • get_next_token_logprobs_from_logits()
                • distributed_vocab_topk()
                • gather_logits_at_global_indices()
                • get_distillation_topk_logprobs_from_logits()
                • ChunkedDistributedEntropy
                  • forward()
                  • backward()
                • from_parallel_hidden_states_to_logprobs()
                • ChunkedDistributedHiddenStatesToLogprobs
                  • forward()
                  • backward()
                • patch_gpt_model_forward_for_linear_ce_fusion()
                • _gpt_forward_with_linear_ce_fusion()
                • all_to_all_vp2sq()
                • all_to_all_sq2vp()
          • nemo_rl.distributed.numa_utils
            • Module Contents
              • Functions
              • Data
              • API
                • logger
                • GPU_CPU_AFFINITY_PATH
                • bind_to_gpu_numa()
                • resolve_visible_gpu_id()
                • _load_libnuma()
                • _get_numa_node()
                • _set_numa_membind()
                • _parse_cpulist()
          • nemo_rl.distributed.virtual_cluster
            • Module Contents
              • Classes
              • Functions
              • Data
              • API
                • logger
                • ClusterConfig
                  • gpus_per_node
                  • num_nodes
                  • master_port_range_low
                  • master_port_range_high
                  • segment_size
                • dir_path
                • git_root
                • PY_EXECUTABLES
                  • SYSTEM
                  • BASE
                  • VLLM
                  • FSDP
                  • AUTOMODEL
                  • MCORE
                  • NEMO_GYM
                  • SGLANG
                  • TRTLLM
                • DEFAULT_GENERATION_PORT_RANGE_LOW
                • DEFAULT_GENERATION_PORT_RANGE_HIGH
                • DEFAULT_GYM_PORT_RANGE_LOW
                • DEFAULT_GYM_PORT_RANGE_HIGH
                • DEFAULT_VLLM_PORT_RANGE_LOW
                • DEFAULT_VLLM_PORTS_PER_ENGINE
                • DEFAULT_SGLANG_ROUTER_PORT_RANGE_LOW
                • DEFAULT_SGLANG_ROUTER_PORT_RANGE_HIGH
                • DEFAULT_SGLANG_PROMETHEUS_PORT_RANGE_LOW
                • DEFAULT_SGLANG_PROMETHEUS_PORT_RANGE_HIGH
                • DEFAULT_MASTER_PORT_RANGE_LOW
                • DEFAULT_MASTER_PORT_RANGE_HIGH
                • NVLINK_DOMAIN_PREFIX
                • TOPO_RANK_KEY
                • NVLINK_DOMAIN_UNKNOWN
                • TOPO_RANK_UNKNOWN
                • _get_node_ip_and_free_port()
                • _get_node_ip_local()
                • _bind_socket_in_range()
                • _get_free_port_local()
                • _get_free_consecutive_ports_local()
                • init_ray()
                • _get_gpu_id_info()
                • get_reordered_bundle()
                • ResourceInsufficientError
                • get_ray_cluster_topology()
                • select_segment_nodes()
                • prepare_segment_topology()
                • _sort_bundle_indices_by_topology()
                • RayVirtualCluster
                  • _init_placement_groups()
                  • _create_placement_groups_internal()
                  • get_placement_groups()
                  • world_size()
                  • node_count()
                  • get_available_address_and_port()
                  • get_available_addresses_and_ports_batch()
                  • get_master_address_and_port()
                  • _get_sorted_bundle_indices()
                  • shutdown()
                  • __del__()
          • nemo_rl.distributed.stateless_process_group
            • Module Contents
              • Classes
              • API
                • StatelessProcessGroup
                  • init_nccl_communicator()
                  • broadcast()
          • nemo_rl.distributed.collectives
            • Module Contents
              • Functions
              • Data
              • API
                • T
                • rebalance_nd_tensor()
                • gather_jagged_object_lists()
          • nemo_rl.distributed.named_sharding
            • Module Contents
              • Classes
              • Data
              • API
                • REPLICATED_AXES
                • NamedSharding
                  • shape
                  • names
                  • ndim
                  • size
                  • layout
                  • get_worker_coords()
                  • is_axis_zero()
                  • get_ranks_by_coord()
                  • get_ranks()
                  • get_axis_index()
                  • get_axis_size()
                  • __repr__()
                  • __eq__()
          • nemo_rl.distributed.worker_groups
            • Module Contents
              • Classes
              • API
                • MultiWorkerFuture
                  • futures
                  • return_from_workers
                  • called_workers
                  • get_results()
                • RayWorkerBuilder
                  • IsolatedWorkerInitializer
                    • create_worker()
                  • create_worker_async()
                  • __call__()
                • RayWorkerGroup
                  • get_dp_leader_worker_idx()
                  • _create_workers_from_bundle_indices()
                  • workers
                  • worker_metadata
                  • dp_size
                  • run_single_worker_single_data()
                  • run_all_workers_multiple_data()
                  • run_all_workers_single_data()
                  • run_all_workers_sharded_data()
                  • get_all_worker_results()
                  • shutdown()
          • nemo_rl.distributed.ray_actor_environment_registry
            • Module Contents
              • Functions
              • Data
              • API
                • USE_SYSTEM_EXECUTABLE
                • VLLM_EXECUTABLE
                • SGLANG_EXECUTABLE
                • MCORE_EXECUTABLE
                • TRTLLM_EXECUTABLE
                • ACTOR_ENVIRONMENT_REGISTRY
                • get_actor_python_env()
          • nemo_rl.distributed.worker_group_utils
            • Module Contents
              • Functions
              • API
                • get_nsight_config_if_pattern_matches()
                • recursive_merge_options()
          • nemo_rl.distributed.batched_data_dict
            • Module Contents
              • Classes
              • Data
              • API
                • DictT
                • SequencePackingArgs
                  • max_tokens_per_microbatch
                  • input_key
                  • input_lengths_key
                  • algorithm
                  • microbatch_order
                  • sequence_length_pad_multiple
                • DynamicBatchingArgs
                  • max_tokens_per_microbatch
                  • sequence_length_round
                  • input_key
                  • input_lengths_key
                • BatchedDataDict
                  • ADDITIONAL_OPTIONAL_KEY_TENSORS
                  • get_multimodal_dict()
                  • from_batches()
                  • all_gather()
                  • chunk()
                  • reorder_data()
                  • shard_by_batch_size()
                  • get_batch()
                  • slice()
                  • repeat_interleave()
                  • truncate_tensors()
                  • make_microbatch_iterator_with_dynamic_shapes()
                  • get_microbatch_iterator_dynamic_shapes_len()
                  • make_microbatch_iterator_for_packable_sequences()
                  • get_microbatch_iterator_for_packable_sequences_len()
                  • make_microbatch_iterator()
                  • size
                  • to()
                  • select_indices()
                  • get_dict()
                • SlicedDataDict
      • nemo_rl.evals
        • Submodules
          • nemo_rl.evals.answer_parsing
            • Module Contents
              • Functions
              • Data
              • API
                • MULTILINGUAL_ANSWER_PATTERN_TEMPLATE
                • MULTILINGUAL_ANSWER_REGEXES
                • normalize_extracted_answer()
                • normalize_response()
          • nemo_rl.evals.eval
            • Module Contents
              • Classes
              • Functions
              • API
                • EvalConfig
                  • metric
                  • num_tests_per_prompt
                  • seed
                  • k_value
                  • save_path
                • _PassThroughEnvConfig
                  • math
                  • mmau
                • MasterConfig
                  • eval
                  • generation
                  • tokenizer
                  • data
                  • env
                  • cluster
                • setup()
                • eval_pass_k()
                • eval_cons_k()
                • run_env_eval()
                • _run_env_eval_impl()
                • _generate_texts()
                • _save_evaluation_data_to_json()
                • _print_results()
      • nemo_rl.experience
        • Submodules
          • nemo_rl.experience.interfaces
            • Module Contents
              • Classes
              • Data
              • API
                • NEMO_GYM_TASK_INDEX_KEY
                • NEXT_NEMO_GYM_TASK_INDEX_KEY
                • Completion
                  • message_log
                  • env_extras
                  • truncated
                  • reward
                • PromptGroupRecord
                  • prompt_idx
                  • prompt
                  • extra_env_info
                  • metadata
                  • completions
                  • rollout_metrics
          • nemo_rl.experience.metric_utils
            • Module Contents
              • Functions
              • API
                • calculate_single_metric()
                • pct()
          • nemo_rl.experience.rollouts
            • Module Contents
              • Classes
              • Functions
              • Data
              • API
                • TokenizerType
                • _add_r3_fallback_metrics()
                • _extract_mask_sample_flags()
                • _attach_routed_experts_to_message_log_prefix()
                • _find_routed_experts_template()
                • _dummy_routed_experts_for_tokens()
                • backfill_missing_routed_experts()
                • EffortLevelsConfig
                  • low_weight
                  • low_penalty
                  • low_ub
                  • low_string
                • _EffortShapingMetrics
                  • length_rewards_low
                  • rewards_low
                  • low_lengths
                  • high_lengths
                • _apply_effort_shaping()
                • generate_responses()
                • generate_responses_async()
                • calculate_rewards()
                • run_multi_turn_rollout()
                • async_generate_response_for_sample_turn()
                • run_sample_multi_turn_rollout()
                • RolloutGroupResult
                  • group_index
                  • final_batch
                  • rollout_metrics
                  • task_index
                • _aggregate_multi_turn_rollout_metrics()
                • _run_multi_turn_rollout_async()
                • run_async_multi_turn_rollout()
                • run_async_multi_turn_rollout_groups()
                • _tensorize_by_key()
                • NemoGymRolloutResult
                  • input_ids
                  • final_batch
                  • rollout_metrics
                  • task_index
                • _CompletedNemoGymGroup
                  • group_index
                  • rows
                  • results
                • _NemoGymStreamAccumulator
                  • is_complete
                  • add()
                  • finish()
                • get_nemo_gym_thinking_tags()
                • should_mask_flagged_samples()
                • _get_reward_penalty_config_value()
                • _get_reward_penalty_token_id()
                • _get_required_reward_penalty_token_id()
                • _get_reward_penalty_token_ids()
                • _get_required_reward_penalty_token_ids()
                • _infer_single_token_id()
                • resolve_reward_penalty_config()
                • apply_reward_penalties()
                • _prepare_nemo_gym_rows()
                • _tensorize_nemo_gym_result()
                • run_async_nemo_gym_rollout()
                • run_nemo_gym_rollout_sync()
                • _postprocess_single_nemo_gym_group()
          • nemo_rl.experience.rollout_manager
            • Module Contents
              • Classes
              • Data
              • API
                • TokenizerType
                • AsyncRolloutImpl
                  • run_rollout()
                  • _run_single_rollout()
                  • _generate_response()
                  • _aggregate_rollout_metrics()
                • AsyncNemoGymRolloutImpl
                  • run_rollout()
                  • _validate_init_params()
                  • _build_inputs()
                  • _run_rollouts()
                  • _result_to_completion()
                  • _compute_rollout_metrics()
                • RolloutManager
                  • set_weight_version()
                  • run_rollout()
                  • generate_and_push()
          • nemo_rl.experience.sync_rollout_actor
            • Module Contents
              • Classes
              • Functions
              • Data
              • API
                • OPT_IN_CARRY_KEYS
                • _flatten_rollout_message_log_for_tq()
                • SyncRolloutActor
                  • rollout_to_tq()
                  • shutdown()
          • nemo_rl.experience.payload
            • Module Contents
              • Functions
              • API
                • record_to_train_batch()
                • pack_payload()
      • nemo_rl.data
        • Subpackages
          • nemo_rl.data.datasets
            • Subpackages
              • nemo_rl.data.datasets.response_datasets
                • Submodules
                  • nemo_rl.data.datasets.response_datasets.refcoco
                    • Module Contents
                      • Classes
                      • Functions
                      • API
                        • download_and_unzip()
                        • format_refcoco_dataset()
                        • RefCOCODataset
                          • format_data()
                  • nemo_rl.data.datasets.response_datasets.oai_format_dataset
                    • Module Contents
                      • Classes
                      • API
                        • PreservingDataset
                          • __len__()
                          • __getitem__()
                          • __iter__()
                          • map()
                        • OpenAIFormatDataset
                          • format_data()
                  • nemo_rl.data.datasets.response_datasets.helpsteer3
                    • Module Contents
                      • Classes
                      • API
                        • HelpSteer3Dataset
                          • format_data()
                  • nemo_rl.data.datasets.response_datasets.squad
                    • Module Contents
                      • Classes
                      • API
                        • SquadDataset
                          • format_data()
                  • nemo_rl.data.datasets.response_datasets.avqa
                    • Module Contents
                      • Classes
                      • Functions
                      • Data
                      • API
                        • DEFAULT_TEMPLATE
                        • _resample_audio()
                        • _parse_question()
                        • AVQADataset
                          • task_name
                          • format_data()
                  • nemo_rl.data.datasets.response_datasets.geometry3k
                    • Module Contents
                      • Classes
                      • Functions
                      • API
                        • format_geometry3k_dataset()
                        • Geometry3KDataset
                  • nemo_rl.data.datasets.response_datasets.mmpr_tiny
                    • Module Contents
                      • Classes
                      • Functions
                      • Data
                      • API
                        • _IMAGE_PLACEHOLDER_RE
                        • format_mmpr_tiny_dataset()
                        • _ensure_mmpr_cached()
                        • _load_mmpr_tiny_from_cache()
                        • MMPRTinyDataset
                  • nemo_rl.data.datasets.response_datasets.tulu3
                    • Module Contents
                      • Classes
                      • API
                        • Tulu3SftMixtureDataset
                          • format_data()
                  • nemo_rl.data.datasets.response_datasets.aime
                    • Module Contents
                      • Classes
                      • Data
                      • API
                        • AIMEVariant
                        • AIMEDataset
                          • format_data()
                  • nemo_rl.data.datasets.response_datasets.audiomcq
                    • Module Contents
                      • Classes
                      • Functions
                      • Data
                      • API
                        • DEFAULT_TEMPLATE
                        • AUDIOMCQ_REPO_ID
                        • AUDIOMCQ_MANIFEST
                        • TARGET_SAMPLE_RATE
                        • STRONG_AC_VALUE
                        • _resample_audio()
                        • _resolve_snapshot_root()
                        • AudioMCQDataset
                          • task_name
                          • _eager_audio_probe()
                          • format_data()
                  • nemo_rl.data.datasets.response_datasets.numinamath
                    • Module Contents
                      • Classes
                      • Data
                      • API
                        • NON_VERIFIABLE_ANSWERS
                        • NuminaMath15Dataset
                          • _has_verifiable_answer()
                          • _is_marked_valid()
                          • format_data()
                  • nemo_rl.data.datasets.response_datasets.openr1_math
                    • Module Contents
                      • Classes
                      • API
                        • OpenR1Math220KDataset
                          • format_data()
                  • nemo_rl.data.datasets.response_datasets.oasst
                    • Module Contents
                      • Classes
                      • Functions
                      • Data
                      • API
                        • SYSTEM_PROMPT
                        • parse_conversations()
                        • get_data_records()
                        • OasstDataset
                  • nemo_rl.data.datasets.response_datasets.response_dataset
                    • Module Contents
                      • Classes
                      • API
                        • ResponseDataset
                          • format_data()
                  • nemo_rl.data.datasets.response_datasets.openmathinstruct2
                    • Module Contents
                      • Classes
                      • API
                        • OpenMathInstruct2Dataset
                          • format_data()
                  • nemo_rl.data.datasets.response_datasets.deepscaler
                    • Module Contents
                      • Classes
                      • API
                        • DeepScalerDataset
                          • format_data()
                  • nemo_rl.data.datasets.response_datasets.daily_omni
                    • Module Contents
                      • Classes
                      • API
                        • DailyOmniDataset
                          • task_name
                          • get_prompt()
                          • format_data()
                  • nemo_rl.data.datasets.response_datasets.arrow_text_dataset
                    • Module Contents
                      • Classes
                      • Functions
                      • API
                        • ArrowTextDataset
                          • format_data()
                        • _pack_generator()
                  • nemo_rl.data.datasets.response_datasets.clevr
                    • Module Contents
                      • Classes
                      • Functions
                      • API
                        • format_answer_fromtags()
                        • format_clevr_cogent_dataset()
                        • CLEVRCoGenTDataset
                  • nemo_rl.data.datasets.response_datasets.nemogym_dataset
                    • Module Contents
                      • Classes
                      • API
                        • NemoGymDataset
                  • nemo_rl.data.datasets.response_datasets.nemotron_cascade2_sft
                    • Module Contents
                      • Classes
                      • API
                        • NemotronCascade2SFTMathDataset
                          • format_data()
                  • nemo_rl.data.datasets.response_datasets.general_conversations_dataset
                    • Module Contents
                      • Classes
                      • Functions
                      • Data
                      • API
                        • conversation_sender_mapping_sample_to_allowed
                        • convert_metadata()
                        • conversation_process_message()
                        • GeneralConversationsJsonlDataset
                          • task_name
                          • process_message_fragment()
                          • _datum_preprocessor()
                  • nemo_rl.data.datasets.response_datasets.dapo_math
                    • Module Contents
                      • Classes
                      • API
                        • DAPOMath17KDataset
                          • format_data()
                        • DAPOMathAIME2024Dataset
                  • nemo_rl.data.datasets.response_datasets.gsm8k
                    • Module Contents
                      • Classes
                      • Functions
                      • API
                        • _extract_hash_answer()
                        • GSM8KDataset
                          • format_data()
                  • nemo_rl.data.datasets.response_datasets.intent
                    • Module Contents
                      • Classes
                      • Functions
                      • Data
                      • API
                        • logger
                        • _TYPE_TEMPLATE
                        • _format_options()
                        • _SPLIT_CONFIG
                        • _EXTRACTION_SENTINEL
                        • _extract_videos_zip_once()
                        • _resolve_video_path()
                        • _read_manifest()
                        • IntentDataset
                          • _download_and_extract()
                          • _load_records()
                          • _filter_records()
                          • format_data()
                        • IntentTrainDataset
                        • IntentBenchDataset
                • Package Contents
                  • Functions
                  • Data
                  • API
                    • DATASET_REGISTRY
                    • load_response_dataset()
                    • __all__
              • nemo_rl.data.datasets.eval_datasets
                • Submodules
                  • nemo_rl.data.datasets.eval_datasets.mmlu
                    • Module Contents
                      • Classes
                      • API
                        • MMLUDataset
                          • _rekey()
                  • nemo_rl.data.datasets.eval_datasets.math
                    • Module Contents
                      • Classes
                      • API
                        • MathDataset
                          • _rekey()
                  • nemo_rl.data.datasets.eval_datasets.mmlu_pro
                    • Module Contents
                      • Classes
                      • API
                        • MMLUProDataset
                          • _rekey()
                  • nemo_rl.data.datasets.eval_datasets.daily_omni
                    • Module Contents
                      • Classes
                      • Data
                      • API
                        • _SINGLE_LETTER_LINE
                        • DailyOmniEvalDataset
                          • _format_for_eval()
                  • nemo_rl.data.datasets.eval_datasets.local_math_dataset
                    • Module Contents
                      • Classes
                      • API
                        • LocalMathDataset
                          • _rekey()
                  • nemo_rl.data.datasets.eval_datasets.gpqa
                    • Module Contents
                      • Classes
                      • API
                        • GPQADataset
                          • _rekey()
                  • nemo_rl.data.datasets.eval_datasets.mmau
                    • Module Contents
                      • Classes
                      • Data
                      • API
                        • DEFAULT_TEMPLATE
                        • MMAUDataset
                          • format_data()
                • Package Contents
                  • Functions
                  • Data
                  • API
                    • MULTIMODAL_DATASETS
                    • _is_multimodal_dataset()
                    • load_eval_dataset()
                    • __all__
              • nemo_rl.data.datasets.preference_datasets
                • Submodules
                  • nemo_rl.data.datasets.preference_datasets.helpsteer3
                    • Module Contents
                      • Classes
                      • API
                        • HelpSteer3Dataset
                          • format_data()
                  • nemo_rl.data.datasets.preference_datasets.tulu3
                    • Module Contents
                      • Classes
                      • API
                        • Tulu3PreferenceDataset
                          • format_data()
                  • nemo_rl.data.datasets.preference_datasets.preference_dataset
                    • Module Contents
                      • Classes
                      • API
                        • PreferenceDataset
                  • nemo_rl.data.datasets.preference_datasets.binary_preference_dataset
                    • Module Contents
                      • Classes
                      • API
                        • BinaryPreferenceDataset
                          • format_data()
                • Package Contents
                  • Functions
                  • Data
                  • API
                    • DATASET_REGISTRY
                    • load_preference_dataset()
                    • __all__
            • Submodules
              • nemo_rl.data.datasets.raw_dataset
                • Module Contents
                  • Classes
                  • API
                    • RawDataset
                      • data_config
                      • dataset
                      • val_dataset
                      • processor
                      • task_spec
                      • preprocessor
                      • split_train_validation()
                      • set_processor()
                      • set_task_spec()
              • nemo_rl.data.datasets.processed_dataset
                • Module Contents
                  • Classes
                  • Data
                  • API
                    • TokenizerType
                    • AllTaskProcessedDataset
                      • __len__()
                      • encode_single()
                      • __getitem__()
              • nemo_rl.data.datasets.utils
                • Module Contents
                  • Functions
                  • Data
                  • API
                    • TokenizerType
                    • load_audio_from_file()
                    • assert_no_double_bos()
                    • pil_to_base64()
                    • load_dataset_from_path()
                    • resolve_external_dataset_class()
                    • _BEHAVIORAL_DATASET_CONFIG_KEYS
                    • warn_on_unsupported_dataset_config_keys()
                    • update_single_dataset_config()
                    • merge_datasets()
                    • extract_necessary_env_names()
                    • get_huggingface_cache_path()
            • Package Contents
              • Data
              • API
                • __all__
          • nemo_rl.data.packing
            • Submodules
              • nemo_rl.data.packing.metrics
                • Module Contents
                  • Classes
                  • API
                    • PackingMetrics
                      • reset()
                      • update()
                      • calculate_stats_only()
                      • get_aggregated_stats()
                      • print_aggregated_stats()
              • nemo_rl.data.packing.algorithms
                • Module Contents
                  • Classes
                  • Functions
                  • API
                    • PackingAlgorithm
                      • CONCATENATIVE
                      • FIRST_FIT_DECREASING
                      • FIRST_FIT_SHUFFLE
                      • MODIFIED_FIRST_FIT_DECREASING
                    • SequencePacker
                      • _pack_implementation()
                      • _adjust_bin_count()
                      • pack()
                      • reset_metrics()
                      • compute_metrics()
                      • get_aggregated_metrics()
                      • print_metrics()
                      • _validate_sequence_lengths()
                      • _create_indexed_lengths()
                      • _estimate_bins_needed()
                    • ConcatenativePacker
                      • max_sequences_per_bin
                      • _pack_implementation()
                    • FirstFitPacker
                      • _prepare_sequences()
                      • _pack_implementation()
                    • FirstFitDecreasingPacker
                      • _prepare_sequences()
                    • FirstFitShufflePacker
                      • _prepare_sequences()
                    • ModifiedFirstFitDecreasingPacker
                      • _classify_items()
                      • _pack_implementation()
                    • get_packer()
            • Package Contents
              • Data
              • API
                • __all__
        • Submodules
          • nemo_rl.data.interfaces
            • Module Contents
              • Classes
              • Data
              • API
                • LLMMessageLogType
                • VLMMessageLogType
                • FlatMessagesType
                • PathLike
                • TokenizerType
                • DatumSpec
                  • message_log
                  • length
                  • extra_env_info
                  • loss_multiplier
                  • idx
                  • task_name
                  • stop_strings
                  • __extra__
                • PreferenceDatumSpec
                  • message_log_chosen
                  • message_log_rejected
                  • length_chosen
                  • length_rejected
                  • loss_multiplier
                  • idx
                • TaskDataSpec
                  • task_name
                  • prompt_file
                  • system_prompt_file
                  • __post_init__()
                  • copy_defaults()
                • TaskDataProcessFnCallable
                  • __call__()
                • TaskDataPreProcessFnCallable
                  • __call__()
          • nemo_rl.data.chat_templates
            • Module Contents
              • Classes
              • API
                • COMMON_CHAT_TEMPLATES
                  • simple_role_header
                  • passthrough_prompt_response
          • nemo_rl.data.processors
            • Module Contents
              • Functions
              • Data
              • API
                • TokenizerType
                • helpsteer3_data_processor()
                • sft_processor()
                • preference_preprocessor()
                • math_data_processor()
                • math_hf_data_processor()
                • vlm_hf_data_processor()
                • _construct_multichoice_prompt()
                • multichoice_qa_processor()
                • nemo_gym_data_processor()
                • kd_data_processor()
                • PROCESSOR_REGISTRY
                • register_processor()
          • nemo_rl.data.collate_fn
            • Module Contents
              • Functions
              • Data
              • API
                • TokenizerType
                • rl_collate_fn()
                • eval_collate_fn()
                • preference_collate_fn()
          • nemo_rl.data.cross_tokenizer_collate
            • Module Contents
              • Classes
              • API
                • CrossTokenizerCollator
                  • __call__()
                  • _tokenize_batch()
          • nemo_rl.data.llm_message_utils
            • Module Contents
              • Functions
              • Data
              • API
                • Tensor
                • TokenizerType
                • message_log_to_flat_messages()
                • get_keys_from_message_log()
                • add_loss_mask_to_message_log()
                • _pad_tensor()
                • _validate_tensor_consistency()
                • batched_message_log_to_flat_message()
                • message_log_shape()
                • get_first_index_that_differs()
                • get_formatted_message_log()
                • remap_dataset_keys()
                • MESSAGE_LOG_BULK_FIELDS
                • decompose_message_log()
                • attach_message_log_view()
                • reconstruct_message_log()
          • nemo_rl.data.dataloader
            • Module Contents
              • Classes
              • API
                • MultipleDataloaderWrapper
                  • _load_custom_dataloader_func()
                  • __iter__()
                  • __next__()
                  • set_records()
          • nemo_rl.data.multimodal_utils
            • Module Contents
              • Classes
              • Functions
              • Data
              • API
                • MEDIA_TAGS
                • MEDIA_TAGS_REVERSED
                • DEFAULT_MEDIA_EXTENSIONS
                • _PLACEHOLDER_STYLE_PROCESSOR_NAMES
                • MEDIA_TAGS_TO_ALLOWED
                • MEDIA_TAG_PATTERN
                • logger
                • uses_image_placeholder()
                • PackedTensor
                  • as_tensor()
                  • __len__()
                  • to()
                  • slice()
                  • empty_like()
                  • concat()
                  • flattened_concat()
                • get_multimodal_keys_from_processor()
                • get_multimodal_default_settings_from_processor()
                • get_dim_to_pack_along()
                • resolve_to_image()
                • image_to_data_url()
                • encode_images_in_examples()
                • get_media_from_message()
                • load_media_from_message()
          • nemo_rl.data.utils
            • Module Contents
              • Functions
              • API
                • get_train_dataset_name()
                • load_dataloader_state()
                • setup_response_data()
                • setup_preference_data()
        • Package Contents
          • Classes
          • Data
          • API
            • ResponseDatasetConfig
              • dataset_name
              • data_path
              • input_key
              • output_key
              • subset
              • split
              • prompt_file
              • system_prompt_file
              • env_name
              • processor
              • download_dir
              • split_validation_size
              • seed
            • PreferenceDatasetConfig
              • dataset_name
              • data_path
              • prompt_key
              • chosen_key
              • rejected_key
              • subset
              • split
              • prompt_file
              • system_prompt_file
            • DataConfig
              • max_input_seq_length
              • add_bos
              • add_eos
              • add_generation_prompt
              • add_system_prompt
              • shuffle
              • num_workers
              • use_multiple_dataloader
              • num_prompts_per_dataloader
              • custom_dataloader
              • train
              • validation
              • default
            • MMLUEvalDataConfig
              • max_input_seq_length
              • dataset_name
              • prompt_file
              • system_prompt_file
            • MMLUProEvalDataConfig
              • max_input_seq_length
              • dataset_name
              • prompt_file
              • system_prompt_file
            • AIMEEvalDataConfig
              • max_input_seq_length
              • dataset_name
              • prompt_file
              • system_prompt_file
              • processor
              • repeat
            • GPQAEvalDataConfig
              • max_input_seq_length
              • dataset_name
              • prompt_file
              • system_prompt_file
            • MathEvalDataConfig
              • max_input_seq_length
              • dataset_name
              • prompt_file
              • system_prompt_file
            • LocalMathEvalDataConfig
              • max_input_seq_length
              • dataset_name
              • problem_key
              • solution_key
              • file_format
              • split
              • prompt_file
              • system_prompt_file
            • MMAUEvalDataConfig
              • max_input_seq_length
              • dataset_name
              • split
              • prompt_file
              • system_prompt_file
              • env_name
            • DailyOmniEvalDataConfig
              • max_input_seq_length
              • dataset_name
              • split
              • prompt_file
              • system_prompt_file
              • env_name
            • EvalDataConfigType
    • Submodules
      • nemo_rl.package_info
        • Module Contents
          • Data
          • API
            • MAJOR
            • MINOR
            • PATCH
            • PRE_RELEASE
            • VERSION
            • __shortversion__
            • __version__
            • __package_name__
            • __contact_names__
            • __contact_emails__
            • __homepage__
            • __repository_url__
            • __download_url__
            • __description__
            • __license__
            • __keywords__
    • Package Contents
      • Functions
      • Data
      • API
        • megatron_path
        • _is_build_isolation()
        • _check_container_fingerprint()
        • _patch_nsight_file()
        • patch_transformers_module_dir()