NeMo RL Documentation

Welcome to the NeMo RL documentation. NeMo RL is an open-source post-training library developed by NVIDIA, designed to streamline and scale reinforcement learning methods for multimodal models (LLMs, VLMs, etc.).

This documentation provides comprehensive guides, examples, and references to help you get started with NeMo RL and build powerful post-training pipelines for your models.

Getting Started

Overview

Learn about NeMo RL’s architecture, design philosophy, and key features that make it ideal for scalable reinforcement learning.

Overview

Quick Start

Get up and running quickly with examples for both DTensor and Megatron Core training backends.

Quick Start

Installation

Step-by-step instructions for installing NeMo RL, including prerequisites, system dependencies, and environment setup.

Installation and Prerequisites

Features

Explore the current features and upcoming enhancements in NeMo RL, including distributed training, advanced parallelism, and more.

Features and Roadmap

Tips and Tricks

Troubleshooting common issues including missing submodules and memory fragmentation.

Tips and Tricks

Training and Generation

Training Backends

Learn about DTensor and Megatron Core training backends, their capabilities, and how to choose the right one for your use case.

Training and Generation Backends

Algorithms

Discover supported algorithms including GRPO, PPO, SFT, DPO, RM, on-policy distillation, and multi-teacher on-policy distillation (MOPD) with detailed guides and examples.

Algorithms

Evaluation

Learn how to evaluate your models using built-in evaluation datasets and custom evaluation pipelines.

Evaluation

Cluster Setup

Configure and launch NeMo RL on multi-node Slurm or Kubernetes clusters for distributed computing.

Set Up Clusters

Guides and Examples

GRPO DeepscaleR

Reproduce DeepscaleR results with NeMo RL using GRPO on mathematical reasoning tasks.

GRPO on DeepScaler

SFT on OpenMathInstruct2

Step-by-step guide for supervised fine-tuning on the OpenMathInstruct2 dataset.

SFT on OpenMathInstruct-2

Environments

Create custom reward environments and integrate them with NeMo RL training pipelines.

Environments for GRPO Training

Eagle3 Speculative Decoding

Configure offline and online Eagle3 draft-model workflows to accelerate rollout generation with vLLM.

Train with Eagle3 Speculative Decoding

Audio GRPO on AVQA

Train Qwen2.5-Omni-3B with GRPO on AVQA and evaluate on MMAU, following the R1-AQA approach.

Audio Post-training with NeMo RL

Audio-Visual Intent GRPO

Train Qwen2.5-Omni-7B with GRPO on PhilipC/IntentTrain (audio-visual intent recognition) and evaluate on Daily-Omni, following HumanOmniV2’s joint audio-visual setup.

Audio-Visual GRPO with Qwen2.5-Omni-7B

Two-Stage SWE RL (Qwen3 Thinking)

Train Qwen3-30B-A3B-Thinking into a SWE agent with a pivot stage plus end-to-end agentic RL on SWE-bench.

Two-Stage SWE RL for Qwen3-30B-A3B-Thinking

Adding New Models

Learn how to add support for new model architectures in NeMo RL.

Add New Models

LoRA

Parameter-efficient fine-tuning with LoRA: backend support, DTensor vs Megatron schema comparison, config examples, and recipes.

LoRA (Low-Rank Adaptation)

YaRN Long-Context Training

Extend a model’s context window with YaRN RoPE scaling on the Megatron backend for SFT, GRPO, and other workflows.

YaRN Long-Context Training

Cross-Tokenizer Distillation

Off-policy distillation across mismatched tokenizers — build a (student, teacher) projection matrix and run x-token KD via CUDA-IPC teacher logits.

Cross-Tokenizer (X-Token) Off-Policy Distillation

Advanced Topics

Design and Philosophy

Deep dive into NeMo RL’s architecture, APIs, and design decisions for scalable RL.

Design and Philosophy

Debugging

Tools and techniques for debugging distributed Ray applications and RL training runs.

Debug NeMo RL Applications

FP8 Quantization

Optimize large language models with FP8 quantization for faster training and inference.

FP8 Quantization in NeMo RL

Quantization-Aware RL

Run quantization-aware GRPO and distillation using NVIDIA ModelOpt. Includes NVFP4 W4A16 real rollout.

Quantization-Aware RL (QARL)

Docker Containers

Build and use Docker containers for reproducible NeMo RL environments.

Build Docker Images

API Reference

Complete API Documentation

Comprehensive reference for all NeMo RL modules, classes, functions, and methods. Browse the complete Python API with detailed docstrings and usage examples.

API Reference

Full Documentation Index

The complete table of contents below lists all pages, including guide, development, and design-doc pages that are not shown in the cards above.

About

• Overview
  • What You Can Expect
  • Releases
• Performance
  • Nomenclature
  • Performance Metrics
  • Performance Summary for Large Language Models
  • NeMo RL v0.6
    • H100 BF16 Benchmarks
    • H100 FP8 Benchmarks
    • GB200 BF16 Benchmarks
• Model Support
  • Broad coverage for 🤗Hugging Face models via NeMo AutoModel
  • Optimal acceleration for top models via NeMo Megatron Bridge
• Features and Roadmap
  • Available Now
  • Planned / In Progress
• Training and Generation Backends
  • Training Backends
  • Generation Backends
• Quick Start
  • Quick Start Options
    • Smoke Test
• Installation and Prerequisites
  • Clone the Repository
  • Install System Dependencies
    • cuDNN (For Megatron Backend)
    • libibverbs (For vLLM Dependencies)
  • Install UV Package Manager
  • Create Virtual Environment
  • Configure cuDNN for Transformer Engine (Bare Metal Only)
  • Using UV to Run Commands
• Algorithms
  • Support Matrix
• Evaluation
  • Convert Model Format (Optional)
  • Run Evaluation
• Set Up Clusters
• Tips and Tricks
  • Missing Submodules Error
  • Memory Fragmentation

Environment Start

• Run on Your Local Workstation
• Set Up Clusters
  • Use Slurm for Batched and Interactive Jobs
    • Batched Job Submission
    • Interactive Launching
    • Slurm Environment Variables
      • Common Environment Configuration
      • Advanced Environment Configuration
    • Topology-Aware Placement for MoE (avoiding cross-rack stalls)
  • Kubernetes
  • Prerequisites
    • 1. Test Cluster Access
    • 2. Build and Push the Docker Container
  • Phase 1: Infrastructure Setup
    • 1. Configure Shared Storage (NFS)
    • 2. Create Registry Secret
  • Phase 2: Ray Cluster Configuration
  • Phase 3: Run Sample NemoRL Workloads
    • 1. Access the Head Node
    • 2. Setup Code on Shared Volume
    • 3. Submit a Job
    • 4. Configuration Adjustments
    • 5. Monitoring
  • Utility: PVC Busybox Helper

E2E Examples

• SFT on OpenMathInstruct-2
  • Train the Model
    • Dataset Splits
    • Training Time
  • Evaluate the Model
  • Results
    • train_1M
    • train

Guides

• Nemotron 3 Nano
  • Download and prepare the data
  • Prepare the code
  • Create a launch script
  • Launch training
• Nemotron 3 Nano Omni
  • Recipe 1 — CLEVR-CoGenT (single-node)
    • Launch (interactive container)
  • Recipe 2 — MMPR-Tiny (4-node Slurm)
    • Launch (4-node Slurm)
• Nemotron 3 Super
  • Container
  • Download and prepare the data
  • Prepare the code
  • Training the model
    • Build sandbox container
    • Launch script
    • Stage 1 - RLVR
      • Stage 1.1 - RLVR 1 (183 H100 GPU nodes)
      • Stage 1.2 - RLVR 2 (183 H100 GPU nodes)
      • Stage 1.3 - RLVR 3 (183 H100 GPU nodes)
    • Stage 2 - SWE
      • Stage 2.1 - SWE 1 (64 H100 GPU nodes)
      • Stage 2.2 - SWE 2 (64 H100 GPU nodes)
    • Stage 3 - RLHF (100 H100 GPU nodes)
• Add New Models
  • Importance of Log Probability Consistency in Training and Inference
  • Understand Discrepancies Between Backends
  • 1. Hugging Face–Based Models
    • Validation Workflow
  • 2. Megatron Models
    • Additional Validation
  • 3. Expected Error Threshold
• Model Diagnostics
  • 1.max_model_len_respected.py
  • 2.long_generation_decode_vs_prefill
  • 3.check_and_reinit_hf_model_embeddings_untrained.py
  • 4.vllm_precision_compilation_test.py
  • 5.prefix_caching_nan.py
  • 6.vllm_routed_experts_completeness.py
• Supervised Fine-Tuning in NeMo RL
  • Launch an SFT Run
  • Example Configuration File
  • Datasets
    • Custom datasets defined outside NeMo RL
    • OpenAI Format Datasets (with Tool Calling Support)
      • Basic Usage
      • Data Format
      • Tool Calling with Heterogeneous Schemas
  • Evaluate the Trained Model
  • LoRA Configuration
  • Optimizations
    • Chunked Linear Cross-Entropy Fusion Loss
• Direct Preference Optimization in NeMo RL
  • Launch a DPO Run
  • Configuration
  • Datasets
    • Custom datasets defined outside NeMo RL
  • DPO-Specific Parameters
  • LoRA Configuration
  • Optimizations
    • Chunked Fused Linear Logprobs
  • Evaluate the Trained Model
• An in-depth Walkthrough of DAPO in NeMo RL
  • Quickstart: Launch a DAPO Run
  • Dynamic Sampling
    • About batch_multiplier
  • Reward Shaping
  • Configuration
  • Example Training Results
  • References
• LoRA (Low-Rank Adaptation)
  • Backend Support
  • Schema Comparison
  • DTensor Configuration
    • DTensor Parameter Details
  • Megatron Configuration
    • Megatron Parameter Details
  • Usage by Algorithm
    • SFT
    • GRPO
    • DPO
  • Example Recipes
  • Exporting a LoRA Checkpoint to Hugging Face Format
• An in-depth Walkthrough of CISPO in NeMo RL
  • Quickstart: Launch a CISPO Run
  • The CISPO Objective
  • Configuration
  • References
• An In-Depth Walkthrough of ProRLv2 in NeMo RL
  • Quickstart: Launch a ProRLv2 Run
  • DAPO: Dynamic Sampling
  • Decoupled local/global advantage normalization
  • Reward Shaping: “Stop properly” Penalty (Truncation Penalty)
  • Loss: Decoupled (Asymmetric) Clipping
  • Loss: Token-level Policy Gradient
  • Importance Sampling Correction and MoE Stability
    • Filtering Strategies
  • Full Example Configs
  • Practical Overrides
  • What to Monitor
  • References
• Two-Stage SWE RL for Qwen3-30B-A3B-Thinking
  • The two stages
  • What makes this a thinking agent
    • Interleaved thinking across turns
    • Three agent harnesses (mixed-prompt training)
  • Async GRPO + MoE setup
  • Results
    • Stage 1 (SWE1, pivot) training reward
    • Stage 2 (SWE2) training reward & agent turns
  • Launch
    • Prerequisites
    • Stage 1 — SWE1 (pivot)
    • Stage 2 — SWE2 (end-to-end agentic)
  • What to monitor
  • References
• An In-depth Walkthrough of GRPO in NeMo RL
  • Quickstart: Launch a GRPO Run
    • Data
      • Dataset
        • Custom datasets defined outside NeMo RL
      • Common Data Format
      • Data Processors
      • Multiple Dataloaders
    • Task–Dataset Mapping
      • Putting It All Together
  • Environments
    • Env–Task Mapping
  • Policy Model
  • Fast Generation
    • Generation Sampling Parameters (temperature, top-p, top-k)
  • Performance Optimizations
    • Chunked Fused Linear Logprobs
  • Loss
    • Improvements to the GRPO Loss Formulation for Stability and Accuracy
      • On-Policy KL Approximation
      • Importance Sampling Correction
      • Overlong Filtering
      • Advantage Clipping
      • Top-p and top-k filtering
  • Metrics
    • Multiplicative Token Probability Error
    • KL Divergence Error
    • Sampling Importance Ratio
    • Entropy
    • GDPO: Group reward-Decoupled Normalization Policy Optimization for Multi-reward RL Optimization
  • LoRA Configuration
  • Evaluate the Trained Model
• An In-Depth Walkthrough of PPO in NeMo RL
  • Quickstart: Launch a PPO Run
    • Data, Environments, Policy, and Generation
  • Value Model (Critic)
    • Colocated Architecture
    • Value Model Configuration
  • Generalized Advantage Estimation (GAE)
    • GAE Configuration
    • VAPO Decoupled GAE
    • Other Advantage Estimators
  • PPO Training Loop
    • Multiple Training Steps per Rollout
    • Critic Warmup
  • Loss
    • Policy Loss
    • Value Loss
  • Configuration
  • Metrics
  • Evaluate the Trained Model
  • References
• GRPO on DeepScaler
  • Train the Model
  • Training Curve
  • Evaluate the Model
  • Evaluation Results
• Solve a Sliding Puzzle Using GRPO
  • Quick Start Guide
    • 1. Install and Set Up NeMo RL with Megatron Backend (Optional)
    • 2. Train a Model
    • 3. Customize Puzzle Configuration
    • 4. Monitor Progress
  • Game Mechanics
    • Puzzle Structure
    • Example Data Sample
    • Movement Rules
  • Data Generation
    • Configuration Parameters
      • Description
      • Generation Algorithm
    • Dataset Size Calculation
    • Data Structure
  • Environment Interface
    • Core Classes
      • SlidingPuzzleEnv
      • SlidingPuzzleGameLogic
      • SlidingPuzzleRunner
    • Processing Pipeline
  • Reward System
    • Reward Structure
    • Reward Calculation Logic
  • Results
• Audio Post-training with NeMo RL
  • 1. Datasets
    • AVQA
    • AudioMCQ-StrongAC-GeminiCoT
  • 2. Train
    • 3B (Qwen2.5-Omni-3B) — audio_grpo_3B_megatron.yaml
    • 7B (Qwen2.5-Omni-7B) — vlm_grpo-qwen2.5-omni-7b-audiomcq-1n8g-megatron.v1.yaml
    • 30B MoE (Qwen3-Omni-30B-A3B-Instruct) — vlm_grpo-qwen3-omni-30ba3b-audiomcq-4n8g-megatron.v1.yaml
  • 3. Convert checkpoint (Megatron → HF)
  • 4. Evaluate on MMAU
  • 5. Results
    • 3B + AVQA (R1-AQA reproduction)
    • 7B + AudioMCQ
• Audio-Visual GRPO with Qwen2.5-Omni-7B
  • 1. Train the Model
    • 7B training notes
  • 2. Convert Checkpoint (Megatron to HF)
  • 3. Evaluate
  • 4. Results
• Reward Model Training in NeMo RL
  • Launch a Training Job
  • Datasets
    • Custom datasets defined outside NeMo RL
  • Using Reward Models as Environments
    • Reward Model Environment
    • Integration with GRPO
    • Configuration
• Environments for GRPO Training
  • Math Environment
    • Key Features
    • Usage
    • Multi-reward support (GDPO)
  • Code Environment
    • Usage
    • Configuration
  • Code Jaccard Environment
    • How It Works
    • Usage
    • Configuration
    • Sample GRPO Config
  • Reward Model Environment
    • Usage
    • Resource Allocation in GRPO Training
      • Single-Node Setup (num_nodes: 1)
      • Multi-Node Setup (num_nodes > 1)
    • Complete GRPO Training with Reward Model Environments
  • Registering Custom Environments
    • Using the register_env Interface
    • Example: Registering a Custom Environment
• Evaluation
  • Prepare for Evaluation
    • Convert DCP to HF (Optional)
    • Prepare the Evaluation Configuration
  • Run the Evaluation Script
  • Example Evaluation Output
  • List of currently supported benchmarks
• DeepSeek-V3
  • Create BF16 Hugging Face checkpoint
• Model Quirks
  • Gemma-3
    • vLLM Initialization
    • vLLM V1 runtime
    • Context Parallel with FSDP2
  • DeepScaleR Recipe Convergence Issues
  • vLLM Async Rollout Timeout
  • AutoModel Parameter Freezing (freeze_config)
• Train with Async GRPO
  • Configure Async GRPO
    • Enable Async GRPO
    • Complete Example Config
  • Implementation Structure
    • Core Components
      • 1. Main Training Loop (async_grpo_train in grpo.py)
      • 2. Async Trajectory Collector (AsyncTrajectoryCollector in async_utils/trajectory_collector.py)
      • 3. Replay Buffer (ReplayBuffer in async_utils/replay_buffer.py)
    • Weight Version Tracking
    • Coordination Flow
    • Architecture Diagram
  • Checkpointing
    • What is saved
    • Restore behaviour
    • Gap-filling after restore
    • Disabling replay-buffer restore
  • Usage Tips
  • Why Importance Sampling Correction Is Required for Async
    • The GRPO Objective
• Quantization-Aware RL (QARL)
  • Overview
  • Verified Configurations
  • ModelOpt Layer Spec Toggle
  • Quantization-Aware GRPO (QA-GRPO)
    • Configuration
    • Running QA-GRPO
  • Real-Quant NVFP4 Rollout (W4A16)
    • Minimal Configuration
    • Running the Example
    • Checkpoints and Fresh Starts
    • Log Checks
    • Troubleshooting
  • Quantization-Aware Distillation (On-Policy QAD)
    • Configuration
    • Running QAD
  • Quantization Parameters
  • Megatron Checkpoint Directory
  • Differences from FP8 Training
  • Supported Quantization Formats
  • Exporting Checkpoints
  • Limitations
• Train with Eagle3 Speculative Decoding
  • Offline vs Online
  • Draft Checkpoint
  • Enablement
    • Generation Only
    • Online Draft Training
  • How It Works
    • Rollout Path
    • Training Path
    • Draft Loss and Time-Step Alignment
  • Important Knobs
  • Notes
• YaRN Long-Context Training
  • Requirements
  • Enablement
    • Required Fields
  • How Conversion Works
  • Forcing a Fresh Conversion
  • Example Recipes
  • Practical Tips
• Cross-Tokenizer (X-Token) Off-Policy Distillation
  • How it works
  • Quickstart — single command
  • Backend and scope
  • Step 1 — Build multi-token mappings
  • Step 2 — Reapply exact-token map
  • Step 3 — Sort and trim to runtime top_k
  • Step 4 — Launch x-token distillation
    • Loss-mode knobs
  • Results — 100-step multi-teacher run
    • Downstream evaluation
    • Throughput and memory
  • Where files live
  • Related
• Router Replay
  • Configuration
  • Validation
    • Trace Debugging
    • Effectiveness Check
  • Other Notes
    • Fallback for Missing Routes
• Muon Optimizer
  • What is Muon?
  • Requirements
  • Basic Usage
  • Example YAML Configuration
  • Experimental Results
• DTensor Tensor Parallel Accuracy Issue
  • Observed Accuracy Issues Under Tensor Parallelism with DTensor Backend
    • Misalignment between Training and Inference for Policy Models
    • Discrepancies Across TP Configurations in Reward Modeling
    • Overall Performance Degradation Under Tensor Parallelism
  • Batch-Variant Kernels
    • Root Cause
    • Recommended Solutions
  • Row-Wise TP Plan
    • Root Cause
    • Recommended Solutions:
  • Additional Observations and Insights
• Fault Tolerance Launcher Guide
  • Key Arguments
  • Basic Usage
  • FT Config File (examples/ft_launcher/ft_config.yaml)
  • Important Notes

Containers

• Build Docker Images
  • Building the Release Image
  • Skipping vLLM or SGLang Dependencies
  • Custom Setup Commands
    • Override with a custom script
    • Skip custom setup entirely

Development

• CI/CD
  • Test Levels
  • Triggering CI on Pull Requests
  • Required Checks
  • CI Pipeline Architecture
  • Code Review
  • Nightly Runs
  • Release Process
  • Infrastructure
  • Workflow Reference
• Test NeMo RL
  • Unit Tests
    • Experimental: Faster Local Test Iteration with pytest-testmon
    • Refreshing Remote-Selection Artifacts
    • Run Unit Tests in a Hermetic Environment
    • Track Metrics in Unit Tests
  • Functional Tests
    • Run Functional Tests in a Hermetic Environment
  • Bisecting Failing Tests
    • Bisecting Unit/Functional Tests
    • Bisecting Nightlies
• Documentation Development
  • Build the Documentation
  • Checking for Broken Links
  • Live Building
  • Run Tests in Python Docstrings
  • Write Tests in Python Docstrings
  • Documentation Version
• Debug NeMo RL Applications
  • Debug Worker/Actors on SLURM
    • Prerequisites
    • Forward a Port from the Head Node
    • Open the Ray Debugger Extension
    • Add the Ray Cluster
    • Add a Breakpoint and Run Your Program
  • Debug with legacy Ray debugger
• Profile GPU with Nsys
  • Prerequisites
  • Configure the Environment Variables
    • Extra Nsys Options (Optional)
    • Pattern Format
    • Supported Workers
  • Example Usage
    • Profile Only Policy Workers
    • Profile Multiple Worker Types
    • Profile Workers with Exact Names
    • Profile Megatron Workers
  • Profile Output
  • Analyze Profile Files
    • How to Analyze the End-to-End RL Loop All at Once
  • How We Patched Nsight Support in Ray
    • The Problem
    • The Fix
    • Where We Applied the Patch
    • Why We Needed Both Locations
• FP8 Quantization in NeMo RL
  • Supported Features
    • FP8 Generation
    • FP8 Training
    • Recommended recipe
  • Integration with NeMo RL
    • RayDistributedExecutor
    • Quantization Utilities
    • Weight Loading
  • Usage
  • Compatibility Note for DeepSeek-Style FP8 Training
  • Accuracy
• Experiment with Custom vLLM
  • Clone and Build Your Custom vLLM
  • Verify Your Custom vLLM in Isolation
  • Running NeMo RL Apps with Custom vLLM
  • Re-building the NeMo RL Docker Image
    • SSH Setup for Private Repositories
      • Prerequisites
      • Step 1: Verify your SSH key works
      • Step 2: Load your SSH key into the agent
      • Step 3: Run the Docker build with SSH forwarding
  • Running Applications with a Custom vLLM Container
    • Why Not Use uv run or Rebuild Venvs?

Design Docs

• Design and Philosophy
  • Motivation
  • Design
    • RayVirtualCluster
    • RayWorkerGroup
    • Single-Controller and Execution Diagram
    • Configuration Schema: BaseModel, dataclass, and TypedDict
• Padding in NeMo RL
  • Padding Approach
  • Right-Padded Generation Example
  • Verify Right Padding
  • Best Practices
• Logger
  • Requirements
  • Overall Design
  • Supported Logging Backends
    • WandB (Weights & Biases)
    • Swanlab
    • Tensorboard
    • MLflow
      • MLflow Configuration
      • MLflow UI
  • Validation Pretty Logging
    • Example Output
  • GPU Metric Logging
• uv in NeMo RL
  • Overview
  • Why uv?
    • Speed and Efficiency
    • Isolated Environments
    • Dependency Management in Ray Clusters
    • Container-Free Flexibility
  • Implementation in NeMo RL
    • Worker Configuration
    • Supported Python Executables
    • Customization
  • How It Works
  • Conclusion
• Dependency Management
  • Workflows Overview
    • Production Workflow
    • Development Workflow
  • How uv run Works
    • 1. uv lock
    • 2. uv sync
    • 3. source .venv/bin/activate
    • 4. python examples/run_grpo.py
  • Multi-Environment Architecture
    • Worker Groups and Virtual Environments
  • Container Pre-caching
    • When Pre-cached Environments Are Sufficient
  • Handling Dependency Changes
    • Option 1: Force Rebuild Environments
    • Option 2: Rebuild the Container
    • Option 3: Classic Workflow - Mounting Modified Submodules
  • Decision Guide
  • Frozen Environments
    • What Are Frozen Environments?
    • When to Use Frozen Environments
    • Available Python Executables
    • Container Version Checking
  • Summary
• Data Format
  • Hugging Face Chat Datasets
  • Chat Templates
• Generation Interface
  • Generation Interface
    • Key Components
  • Generation Backends
  • VLLM Backend
    • VllmGeneration Class
    • VllmGenerationWorker
    • Custom VLLM Extensions
  • Megatron Backend
    • Key Features
    • MegatronPolicyWorker Generation
    • Configuration
    • Configuration Parameters
  • Usage Examples
    • Using VLLM Backend
    • Using Megatron Backend
  • Extend with New Backends
• Exporting Checkpoints to Hugging Face Format
  • Converting Torch Distributed Checkpoints to Hugging Face Format
  • Converting Megatron Checkpoints to Hugging Face Format
  • Converting Megatron LoRA Adapter Checkpoints to Hugging Face Format
    • Option A — Merged checkpoint
    • Option B — Adapter-only (PEFT format)
    • Arguments
• Loss functions in NeMo RL
• FSDP2 Parallel Plan
  • Fallback Priority
    • 1. Custom Parallel Plan
    • 2. Optimized Parallel Plan
    • 3. Hugging Face Tensor Parallel Plan
  • Custom Parallel Plan Example
  • HSDP (dp_replicate_size)
    • Illustration
    • Configuration
    • When to use it
• Training Backends
  • Available Backends
  • Supported Input Checkpoint Format
  • Backend Selection
    • Megatron Backend
    • DTensor Backend
  • Configuration Examples
  • Megatron Configuration
    • Environment Variable Priority (Highest to Lowest)
    • Configuration Examples
    • Best Practices
    • Force Reconvert
• Sequence Packing and Dynamic Batching
  • Table of Contents
  • Problem
    • Challenge: Variable Sequence Lengths in RL/SFT
  • Sequence Packing and Dynamic Batching
    • Important Notes
  • Sequence Packing
    • Implementation Details
      • 1. Packing Process (BatchedDataDict.shard_by_batch_size)
      • 2. Packing Algorithms (nemo_rl/data/packing/algorithms.py)
        • Concatenative Packer
        • Modified First Fit Decreasing (MFFD)
        • First Fit Decreasing (FFD)
        • First Fit Shuffle
    • Usage with Context Parallelism
      • Understanding CP Load balancing:
    • Nuances
  • Dynamic Batching
    • Architecture
      • Processing Pipeline
      • Implementation Details
    • Interface
    • Nuances
  • Configuration
    • Dynamic Batching Configuration
    • Sequence Packing Configuration
  • Integration with Training Pipeline
    • Loss Function Integration
  • Metrics and Monitoring
    • Packing Efficiency Metrics (nemo_rl/data/packing/metrics.py)
  • Usage
    • Sequence Packing Configuration
    • Dynamic Batching Configuration
    • Framework Compatibility
  • References
• Environment Variable Precedence in NeMo RL
  • Precedence Order
    • 1. Ray Runtime Environment Variables (lowest)
    • 2. System-level Environment Variables (medium)
    • 3. YAML Configuration env_vars (high)
    • 4. Worker-specific configure_worker Method (highest)
• NeMo Gym Integration
  • Overview
  • Configuration
    • Version Requirements
  • Architecture Overview
  • The NemoGym Actor
  • vLLM HTTP Server
  • Initialization Sequence
  • Training Loop Control Flow
    • Key Steps
    • Async Result Processing
  • Data Format Translation
  • Tokenization and On-Policy Corrections

API Reference

• API Reference
  • nemo_rl
    • Subpackages
      • nemo_rl.environments
        • Submodules
          • nemo_rl.environments.code_environment
            • Module Contents
              • Classes
              • API
                • CodeEnvConfig
                  • num_workers
                  • terminate_on_evaluation
                • CodeEnvMetadata
                  • context
                  • working_dir
                • CodeExecutionWorker
                  • sanitize()
                  • format_result()
                  • execute()
                  • chdir()
                  • safe_open()
                  • safe_import()
                • CodeEnvironment
                  • step()
                  • shutdown()
                  • global_post_process_and_metrics()
          • nemo_rl.environments.vlm_environment
            • Module Contents
              • Classes
              • Functions
              • API
                • VLMEnvConfig
                  • num_workers
                  • stop_strings
                  • reward_functions
                • _mute_output()
                • VLMVerifyWorker
                  • verify()
                • VLMEnvironmentMetadata
                  • ground_truth
                • VLMEnvironment
                  • shutdown()
                  • step()
                  • global_post_process_and_metrics()
          • nemo_rl.environments.interfaces
            • Module Contents
              • Classes
              • Data
              • API
                • MetadataT
                • EnvironmentReturn
                  • observations
                  • metadata
                  • next_stop_strings
                  • rewards
                  • terminateds
                  • answers
                • EnvironmentInterface
                  • step()
                  • global_post_process_and_metrics()
          • nemo_rl.environments.nemo_gym
            • Module Contents
              • Classes
              • Functions
              • Data
              • API
                • DEFAULT_INVALID_TOOL_CALL_PATTERNS
                • DEFAULT_THINKING_TAGS
                • get_nemo_gym_uv_cache_dir()
                • get_nemo_gym_venv_dir()
                • NemoGymConfig
                  • model_name
                  • base_urls
                  • initial_global_config_dict
                  • port_range_low
                  • port_range_high
                  • invalid_tool_call_patterns
                  • thinking_tags
                  • require_routed_experts
                • _detect_invalid_tool_call_and_malformed_thinking()
                • NemoGym
                  • _spinup()
                  • run_rollouts()
                  • _postprocess_nemo_gym_to_nemo_rl_result()
                  • shutdown()
                  • step()
                  • global_post_process_and_metrics()
                • setup_nemo_gym_config()
          • nemo_rl.environments.utils
            • Module Contents
              • Classes
              • Functions
              • Data
              • API
                • EnvRegistryEntry
                  • actor_class_fqn
                  • default_processor
                • ENV_REGISTRY
                • chunk_list_to_workers()
                • create_env()
                • register_env()
          • nemo_rl.environments.dapo_math_verifier
            • Module Contents
              • Functions
              • Data
              • API
                • last_boxed_only_string()
                • remove_boxed()
                • SUBSTITUTIONS
                • REMOVED_EXPRESSIONS
                • normalize_final_answer()
                • is_correct_minerva()
                • is_correct_strict_box()
                • verify()
                • compute_score()
          • nemo_rl.environments.math_environment
            • Module Contents
              • Classes
              • Functions
              • API
                • MathEnvConfig
                  • num_workers
                  • stop_strings
                  • verifier_type
                  • math_verify_impl
                • _mute_output()
                • HFVerifyWorker
                  • verify()
                • MultilingualMultichoiceVerifyWorker
                  • verify()
                • EnglishMultichoiceVerifyWorker
                  • verify()
                • HFMultiRewardVerifyWorker
                  • REWARD_NAMES
                  • verify()
                • MathEnvironmentMetadata
                  • ground_truth
                  • extracted_answer
                • BaseMathEnvironment
                  • WORKER_CLASS_DICT
                  • shutdown()
                  • global_post_process_and_metrics()
                • MathEnvironment
                  • WORKER_CLASS_DICT
                  • step()
                • MathMultiRewardEnvironment
                  • WORKER_CLASS_DICT
                  • step()
          • nemo_rl.environments.reward_model_environment
            • Module Contents
              • Classes
              • API
                • RewardModelEnvironmentConfig
                  • enabled
                  • model_name
                  • precision
                  • batch_size
                  • checkpoint_path
                  • logprob_batch_size
                  • resources
                  • dtensor_cfg
                  • dynamic_batching
                  • sequence_packing
                  • max_grad_norm
                  • generation
                • RewardModelEnvironment
                  • DEFAULT_PY_EXECUTABLE
                  • preprocess_data()
                  • step()
                  • global_post_process_and_metrics()
                  • shutdown()
                  • __del__()
          • nemo_rl.environments.code_jaccard_environment
            • Module Contents
              • Classes
              • API
                • CodeJaccardEnvConfig
                  • num_workers
                  • stop_strings
                • CodeJaccardEnvironmentMetadata
                  • ground_truth
                • CodeJaccardVerifyWorker
                  • verify()
                  • _calculate_preference_score()
                • CodeJaccardEnvironment
                  • shutdown()
                  • step()
                  • global_post_process_and_metrics()
          • nemo_rl.environments.metrics
            • Module Contents
              • Functions
              • API
                • calculate_pass_rate_per_prompt()
          • nemo_rl.environments.rewards
            • Module Contents
              • Functions
              • Data
              • API
                • math_verify_func
                • boxed
                • math_expression_reward()
                • format_reward()
                • exact_answer_alphanumeric_reward()
                • bbox_giou_reward()
                • geo3k_reward()
                • combine_reward_functions()
      • nemo_rl.experience
        • Submodules
          • nemo_rl.experience.interfaces
            • Module Contents
              • Classes
              • API
                • Completion
                  • message_log
                  • env_extras
                  • truncated
                  • reward
                • PromptGroupRecord
                  • prompt_idx
                  • prompt
                  • extra_env_info
                  • metadata
                  • completions
                  • rollout_metrics
          • nemo_rl.experience.rollout_manager
            • Module Contents
              • Classes
              • Data
              • API
                • TokenizerType
                • AsyncRolloutImpl
                  • run_rollout()
                  • _run_single_rollout()
                  • _generate_response()
                  • _aggregate_rollout_metrics()
                • AsyncNemoGymRolloutImpl
                  • run_rollout()
                  • _validate_init_params()
                  • _build_inputs()
                  • _run_rollouts()
                  • _result_to_completion()
                  • _compute_rollout_metrics()
                • RolloutManager
                  • run_rollout()
          • nemo_rl.experience.rollouts
            • Module Contents
              • Classes
              • Functions
              • Data
              • API
                • TokenizerType
                • _add_r3_fallback_metrics()
                • _extract_mask_sample_flags()
                • _attach_routed_experts_to_message_log_prefix()
                • _find_routed_experts_template()
                • _dummy_routed_experts_for_tokens()
                • EffortLevelsConfig
                  • low_weight
                  • low_penalty
                  • low_ub
                  • low_string
                • _EffortShapingMetrics
                  • length_rewards_low
                  • rewards_low
                  • low_lengths
                  • high_lengths
                • _apply_effort_shaping()
                • generate_responses()
                • generate_responses_async()
                • calculate_rewards()
                • run_multi_turn_rollout()
                • async_generate_response_for_sample_turn()
                • run_sample_multi_turn_rollout()
                • run_async_multi_turn_rollout()
                • _tensorize_by_key()
                • AsyncNemoGymRolloutResult
                  • input_ids
                  • final_batch
                  • rollout_metrics
                • _calculate_single_metric()
                • get_nemo_gym_thinking_tags()
                • _get_reward_penalty_config_value()
                • _get_reward_penalty_token_id()
                • _get_required_reward_penalty_token_id()
                • _get_reward_penalty_token_ids()
                • _get_required_reward_penalty_token_ids()
                • _infer_single_token_id()
                • resolve_reward_penalty_config()
                • apply_reward_penalties()
                • run_async_nemo_gym_rollout()
          • nemo_rl.experience.sync_rollout_actor
            • Module Contents
              • Classes
              • Functions
              • Data
              • API
                • OPT_IN_CARRY_KEYS
                • _flatten_rollout_message_log_for_tq()
                • SyncRolloutActor
                  • rollout_to_tq()
                  • shutdown()
      • nemo_rl.utils
        • Submodules
          • nemo_rl.utils.checkpoint
            • Module Contents
              • Classes
              • Functions
              • Data
              • API
                • PathLike
                • PretrainedCheckpointConfig
                  • path
                  • format
                • CheckpointingConfig
                  • enabled
                  • checkpoint_dir
                  • metric_name
                  • higher_is_better
                  • save_period
                  • keep_top_k
                  • checkpoint_must_save_by
                  • pretrained_checkpoint
                  • save_optimizer
                  • model_save_format
                  • save_consolidated
                  • model_cache_dir
                  • model_repo_id
                  • is_peft
                  • peft_config
                  • is_async
                • CheckpointManager
                  • get_resume_paths()
                  • init_tmp_checkpoint()
                  • _rename_checkpoint()
                  • finalize_checkpoint()
                  • begin_finalization()
                  • finalize_pending()
                  • shutdown()
                  • _warn_on_delete_failure()
                  • __enter__()
                  • __exit__()
                  • has_pending_finalization
                  • remove_old_checkpoints()
                  • get_best_checkpoint_path()
                  • get_latest_checkpoint_path()
                  • load_training_info()
                • _load_checkpoint_history()
          • nemo_rl.utils.grad_norm
            • Module Contents
              • Functions
              • Data
              • API
                • _INF_GRAD_NORM_WARNING
                • warn_if_inf_grad_norm()
          • nemo_rl.utils.memory_tracker
            • Module Contents
              • Classes
              • API
                • MemoryTrackerDataPoint
                  • stage
                  • memory_used_before_stage_gb
                  • variables_before_stage
                  • memory_used_after_stage_gb
                  • variables_after_stage
                  • mem_used_diff_gb
                  • new_variables
                  • get_snapshot_str()
                • MemoryTracker
                  • data_points
                  • model_post_init()
                  • snapshot_start_of_stage()
          • nemo_rl.utils.prefetch_venvs
            • Module Contents
              • Functions
              • API
                • prefetch_venvs()
                • create_frozen_environment_symlinks()
          • nemo_rl.utils.flops_formulas
            • Module Contents
              • Classes
              • Functions
              • API
                • FLOPSConfig
                  • gbs
                  • enc_seq_len
                  • hs
                  • layers
                  • ffn_hs
                  • attention_heads
                  • head_dim
                  • moe_router_topk
                  • query_groups
                  • img_seq_len
                  • img_h
                  • img_w
                  • in_channels
                  • patch_dim
                  • class_token_len
                  • projector_type
                  • inp_s
                  • model_pattern
                  • vocab_size
                  • model_channels
                  • vec_in_dim
                  • q_lora_rank
                  • kv_lora_rank
                  • qk_head_dim
                  • qk_pos_emb_head_dim
                  • v_head_dim
                  • moe_layer_freq
                  • moe_shared_expert_intermediate_size
                  • moe_ffn_hidden_size
                  • mtp_num_layers
                  • causal_self_attn
                  • is_hybrid_model
                  • hybrid_override_pattern
                  • mamba_state_dim
                  • mamba_head_dim
                  • mamba_num_groups
                  • mamba_num_heads
                • gpt3()
                • llama()
                • nemotron()
                • mixtral()
                • qwen2()
                • qwen3()
                • bert()
                • transformer()
                • flux()
                • deepseekv3()
                • _mlp_layer_flops()
                • _non_mla_attn_layer_flops()
                • _mamba_layer_flops()
                • _hybrid_model_flops()
                • nemotronh()
          • nemo_rl.utils.config
            • Module Contents
              • Functions
              • API
                • resolve_path()
                • merge_with_override()
                • load_config_with_inheritance()
                • load_config()
                • OverridesError
                • parse_hydra_overrides()
                • register_omegaconf_resolvers()
          • nemo_rl.utils.timer
            • Module Contents
              • Classes
              • Functions
              • Data
              • API
                • logger
                • Timer
                  • _REDUCTION_FUNCTIONS
                  • _fmt()
                  • start()
                  • stop()
                  • record()
                  • mark()
                  • get_markers()
                  • time()
                  • get_elapsed()
                  • get_latest_elapsed()
                  • reduce()
                  • get_timing_metrics()
                  • reset()
                • ThreadSafeTimer
                  • start()
                  • stop()
                  • record()
                  • mark()
                  • get_markers()
                  • time()
                  • get_elapsed()
                  • get_latest_elapsed()
                  • reduce()
                  • get_timing_metrics()
                  • reset()
                • convert_to_seconds()
                • TimeoutChecker
                  • check_save()
                  • start_iterations()
                  • mark_iteration()
          • nemo_rl.utils.flops_tracker
            • Module Contents
              • Classes
              • Functions
              • Data
              • API
                • get_default_hf_config()
                • convert_config_to_flops_config()
                • is_using_tf32()
                • THEORETICAL_TFLOPS
                • get_theoretical_tflops()
                • FLOPTracker
                  • from_config()
                  • track()
                  • track_batch()
                  • reset()
          • nemo_rl.utils.native_checkpoint
            • Module Contents
              • Classes
              • Functions
              • API
                • ModelState
                  • state_dict()
                  • load_state_dict()
                • OptimizerState
                  • state_dict()
                  • load_state_dict()
                • save_checkpoint()
                • load_checkpoint()
                • convert_dcp_to_hf()
          • nemo_rl.utils.nvml
            • Module Contents
              • Functions
              • Data
              • API
                • logger
                • nvml_context()
                • device_id_to_physical_device_id()
                • get_device_uuid()
                • get_free_memory_bytes()
                • _resolve_device_id()
                • log_gpu_memory_diagnostics()
          • nemo_rl.utils.logger
            • Module Contents
              • Classes
              • Functions
              • Data
              • API
                • _rich_logging_configured
                • WandbConfig
                  • project
                  • name
                  • entity
                • SwanlabConfig
                  • project
                  • name
                • TensorboardConfig
                  • log_dir
                • MLflowConfig
                  • experiment_name
                  • run_id
                  • run_name
                  • tracking_uri
                  • artifact_location
                • GPUMonitoringConfig
                  • collection_interval
                  • flush_interval
                • LoggerConfig
                  • log_dir
                  • wandb_enabled
                  • swanlab_enabled
                  • tensorboard_enabled
                  • mlflow_enabled
                  • wandb
                  • tensorboard
                  • swanlab
                  • mlflow
                  • monitor_gpus
                  • gpu_monitoring
                  • num_val_samples_to_print
                • LoggerInterface
                  • log_metrics()
                  • log_hyperparams()
                  • log_histogram()
                  • log_plot()
                • TensorboardLogger
                  • _coerce_to_scalar()
                  • log_metrics()
                  • log_histogram()
                  • log_hyperparams()
                  • log_plot()
                • WandbLogger
                  • _log_diffs()
                  • _log_code()
                  • define_metric()
                  • log_metrics()
                  • log_hyperparams()
                  • log_plot()
                  • log_histogram()
                • SwanlabLogger
                  • log_metrics()
                  • log_hyperparams()
                  • log_plot()
                  • log_histogram()
                • GpuMetricSnapshot
                  • step
                  • metrics
                • RayGpuMonitorLogger
                  • start()
                  • stop()
                  • _collection_loop()
                  • _parse_metric()
                  • _parse_gpu_sku()
                  • _collect_gpu_sku()
                  • _collect_metrics()
                  • _collect()
                  • _fetch_and_parse_metrics()
                  • flush()
                • _merge_generation_logger_workers()
                • _summarize_list()
                • MLflowLogger
                  • log_metrics()
                  • log_hyperparams()
                  • log_plot()
                  • log_histogram()
                  • __del__()
                • Logger
                  • log_metrics()
                  • log_hyperparams()
                  • log_batched_dict_as_jsonl()
                  • log_string_list_as_jsonl()
                  • log_plot_per_worker_timeline_metrics()
                  • log_histogram()
                  • log_plot()
                  • log_plot_token_mult_prob_error()
                  • __del__()
                • flatten_dict()
                • configure_rich_logging()
                • print_message_log_samples()
                • get_next_experiment_dir()
                • log_container_init_timing()
          • nemo_rl.utils.nsys
            • Module Contents
              • Classes
              • Functions
              • Data
              • API
                • NRL_NSYS_WORKER_PATTERNS
                • NRL_NSYS_PROFILE_STEP_RANGE
                • _parse_extra_options()
                • NRL_NSYS_EXTRA_OPTIONS
                • ProfilablePolicy
                  • start_gpu_profiling()
                  • stop_gpu_profiling()
                • maybe_gpu_profile_step()
                • wrap_with_nvtx_name()
          • nemo_rl.utils.r3_trace
            • Module Contents
              • Functions
              • Data
              • API
                • _TRACE_ENV
                • _TRACE_STEPS_ENV
                • _TRACE_SAMPLES_ENV
                • _TRACE_DIR_ENV
                • _TRACE_MICROBATCHES_ENV
                • _TRACE_VERIFY_FORWARD_ENV
                • _DEFAULT_TRACE_DIR
                • _DEFAULT_TRACE_STEPS
                • _DEFAULT_TRACE_SAMPLES
                • _DEFAULT_TRACE_MICROBATCHES
                • _write_lock
                • _patch_lock
                • _router_replay_patch_depth
                • _original_get_replay_topk
                • _event_counts
                • _context
                • r3_trace_enabled()
                • r3_trace_verify_forward_enabled()
                • _env_int()
                • _trace_steps()
                • _trace_samples()
                • _trace_microbatches()
                • _next_count()
                • _should_trace_step()
                • _current_context()
                • _torch_rank_info()
                • _trace_path()
                • _write_record()
                • _tensor_sha256()
                • _tensor_preview()
                • _tensor_record()
                • _shape()
                • _valid_sample_record()
                • _length_at()
                • _tensors_equal()
                • _expected_with_missing_route_fallback()
                • _router_replay_action_name()
                • _trace_router_replay_topk_use()
                • _verify_router_replay_forward_context()
                • trace_rollout_payload()
                • trace_tq_fetch_payload()
                • r3_trace_stage()
                • maybe_r3_trace_stage()
                • trace_cp_routed_experts()
                • trace_router_replay_assignment()
                • trace_router_replay_action()
          • nemo_rl.utils.packed_tensor
            • Module Contents
              • Functions
              • API
                • get_target_packed_tensor_size()
                • get_num_buffers()
                • packed_broadcast_producer()
                • packed_broadcast_consumer()
          • nemo_rl.utils.venvs
            • Module Contents
              • Functions
              • Data
              • API
                • dir_path
                • git_root
                • DEFAULT_VENV_DIR
                • logger
                • create_local_venv()
                • _env_builder()
                • create_local_venv_on_each_node()
                • make_actor_runtime_env()
      • nemo_rl.distributed
        • Submodules
          • nemo_rl.distributed.ray_actor_environment_registry
            • Module Contents
              • Functions
              • Data
              • API
                • USE_SYSTEM_EXECUTABLE
                • VLLM_EXECUTABLE
                • SGLANG_EXECUTABLE
                • MCORE_EXECUTABLE
                • ACTOR_ENVIRONMENT_REGISTRY
                • get_actor_python_env()
          • nemo_rl.distributed.named_sharding
            • Module Contents
              • Classes
              • Data
              • API
                • REPLICATED_AXES
                • NamedSharding
                  • shape
                  • names
                  • ndim
                  • size
                  • layout
                  • get_worker_coords()
                  • is_axis_zero()
                  • get_ranks_by_coord()
                  • get_ranks()
                  • get_axis_index()
                  • get_axis_size()
                  • __repr__()
                  • __eq__()
          • nemo_rl.distributed.stateless_process_group
            • Module Contents
              • Classes
              • API
                • StatelessProcessGroup
                  • init_nccl_communicator()
                  • broadcast()
          • nemo_rl.distributed.worker_groups
            • Module Contents
              • Classes
              • API
                • MultiWorkerFuture
                  • futures
                  • return_from_workers
                  • called_workers
                  • get_results()
                • RayWorkerBuilder
                  • IsolatedWorkerInitializer
                    • create_worker()
                  • create_worker_async()
                  • __call__()
                • RayWorkerGroup
                  • get_dp_leader_worker_idx()
                  • _create_workers_from_bundle_indices()
                  • workers
                  • worker_metadata
                  • dp_size
                  • run_single_worker_single_data()
                  • run_all_workers_multiple_data()
                  • run_all_workers_single_data()
                  • run_all_workers_sharded_data()
                  • get_all_worker_results()
                  • shutdown()
          • nemo_rl.distributed.model_utils
            • Module Contents
              • Classes
              • Functions
              • API
                • _compute_distributed_log_softmax_with_grad()
                • _compute_distributed_log_softmax()
                • _compute_distributed_softmax()
                • DistributedLogprob
                  • forward()
                  • backward()
                • DistributedCrossEntropy
                  • forward()
                  • backward()
                • ChunkedDistributedLogprob
                  • forward()
                  • backward()
                • DistributedLogprobWithSampling
                  • forward()
                  • backward()
                • ChunkedDistributedLogprobWithSampling
                  • forward()
                  • backward()
                • ChunkedDistributedGatherLogprob
                  • forward()
                  • backward()
                • dtensor_from_parallel_logits_to_logprobs()
                • from_parallel_logits_to_logprobs()
                • from_parallel_logits_to_logprobs_packed_sequences()
                • _get_tokens_on_this_cp_rank()
                • allgather_cp_sharded_tensor()
                • _AllReduceSum
                  • forward()
                  • backward()
                • group_all_reduce_sum_with_grad()
                • group_all_reduce_sum()
                • AllGatherCPTensor
                  • forward()
                  • backward()
                • cp_load_balanced_to_contiguous()
                • cp_shift_next()
                • vocab_parallel_log_softmax()
                • vocab_parallel_full_log_softmax()
                • vocab_parallel_argmax()
                • get_logprobs_from_vocab_parallel_logits()
                • get_next_token_logprobs_from_logits()
                • distributed_vocab_topk()
                • gather_logits_at_global_indices()
                • get_distillation_topk_logprobs_from_logits()
                • ChunkedDistributedEntropy
                  • forward()
                  • backward()
                • from_parallel_hidden_states_to_logprobs()
                • ChunkedDistributedHiddenStatesToLogprobs
                  • forward()
                  • backward()
                • patch_gpt_model_forward_for_linear_ce_fusion()
                • _gpt_forward_with_linear_ce_fusion()
                • all_to_all_vp2sq()
                • all_to_all_sq2vp()
          • nemo_rl.distributed.virtual_cluster
            • Module Contents
              • Classes
              • Functions
              • Data
              • API
                • logger
                • ClusterConfig
                  • gpus_per_node
                  • num_nodes
                  • master_port_range_low
                  • master_port_range_high
                  • segment_size
                • dir_path
                • git_root
                • PY_EXECUTABLES
                  • SYSTEM
                  • BASE
                  • VLLM
                  • FSDP
                  • AUTOMODEL
                  • MCORE
                  • NEMO_GYM
                  • SGLANG
                • DEFAULT_GENERATION_PORT_RANGE_LOW
                • DEFAULT_GENERATION_PORT_RANGE_HIGH
                • DEFAULT_GYM_PORT_RANGE_LOW
                • DEFAULT_GYM_PORT_RANGE_HIGH
                • DEFAULT_VLLM_PORT_RANGE_LOW
                • DEFAULT_VLLM_PORTS_PER_ENGINE
                • DEFAULT_MASTER_PORT_RANGE_LOW
                • DEFAULT_MASTER_PORT_RANGE_HIGH
                • NVLINK_DOMAIN_PREFIX
                • TOPO_RANK_KEY
                • NVLINK_DOMAIN_UNKNOWN
                • TOPO_RANK_UNKNOWN
                • _get_node_ip_and_free_port()
                • _get_node_ip_local()
                • _bind_socket_in_range()
                • _get_free_port_local()
                • init_ray()
                • _get_gpu_id_info()
                • get_reordered_bundle()
                • ResourceInsufficientError
                • get_ray_cluster_topology()
                • select_segment_nodes()
                • prepare_segment_topology()
                • _sort_bundle_indices_by_topology()
                • RayVirtualCluster
                  • _init_placement_groups()
                  • _create_placement_groups_internal()
                  • get_placement_groups()
                  • world_size()
                  • node_count()
                  • get_available_address_and_port()
                  • get_available_addresses_and_ports_batch()
                  • get_master_address_and_port()
                  • _get_sorted_bundle_indices()
                  • shutdown()
                  • __del__()
          • nemo_rl.distributed.collectives
            • Module Contents
              • Functions
              • Data
              • API
                • T
                • rebalance_nd_tensor()
                • gather_jagged_object_lists()
          • nemo_rl.distributed.worker_group_utils
            • Module Contents
              • Functions
              • API
                • get_nsight_config_if_pattern_matches()
                • recursive_merge_options()
          • nemo_rl.distributed.numa_utils
            • Module Contents
              • Functions
              • Data
              • API
                • logger
                • GPU_CPU_AFFINITY_PATH
                • bind_to_gpu_numa()
                • resolve_visible_gpu_id()
                • _load_libnuma()
                • _get_numa_node()
                • _set_numa_membind()
                • _parse_cpulist()
          • nemo_rl.distributed.batched_data_dict
            • Module Contents
              • Classes
              • Data
              • API
                • DictT
                • SequencePackingArgs
                  • max_tokens_per_microbatch
                  • input_key
                  • input_lengths_key
                  • algorithm
                  • sequence_length_pad_multiple
                • DynamicBatchingArgs
                  • max_tokens_per_microbatch
                  • sequence_length_round
                  • input_key
                  • input_lengths_key
                • BatchedDataDict
                  • ADDITIONAL_OPTIONAL_KEY_TENSORS
                  • get_multimodal_dict()
                  • from_batches()
                  • all_gather()
                  • chunk()
                  • reorder_data()
                  • shard_by_batch_size()
                  • get_batch()
                  • slice()
                  • repeat_interleave()
                  • truncate_tensors()
                  • make_microbatch_iterator_with_dynamic_shapes()
                  • get_microbatch_iterator_dynamic_shapes_len()
                  • make_microbatch_iterator_for_packable_sequences()
                  • get_microbatch_iterator_for_packable_sequences_len()
                  • make_microbatch_iterator()
                  • size
                  • to()
                  • select_indices()
                  • get_dict()
                • SlicedDataDict
      • nemo_rl.models
        • Subpackages
          • nemo_rl.models.automodel
            • Submodules
              • nemo_rl.models.automodel.data
                • Module Contents
                  • Classes
                  • Functions
                  • API
                    • ProcessedInputs
                      • input_ids
                      • seq_len
                      • attention_mask
                      • position_ids
                      • flash_attn_kwargs
                      • vlm_kwargs
                      • cp_buffers
                      • seq_index
                      • has_context_parallel
                      • has_flash_attention
                      • is_multimodal
                    • ProcessedMicrobatch
                      • data_dict
                      • processed_inputs
                      • original_batch_size
                      • original_seq_len
                    • make_processed_microbatch_iterator()
                    • get_microbatch_iterator()
                    • process_microbatch()
                    • process_global_batch()
                    • check_sequence_dim()
              • nemo_rl.models.automodel.checkpoint
                • Module Contents
                  • Classes
                  • Functions
                  • API
                    • _patch_qwen_vl_vision_key_mapping()
                    • AutomodelCheckpointManager
                      • _get_dp_rank()
                      • _get_tp_rank()
                      • init_checkpointer()
                      • update_checkpointer_config()
                      • _rebuild_checkpointer_addons()
                      • save_checkpoint()
                      • load_checkpoint()
                    • detect_checkpoint_format()
                    • _infer_checkpoint_root()
              • nemo_rl.models.automodel.config
                • Module Contents
                  • Classes
                  • API
                    • DistributedContext
                      • device_mesh
                      • moe_mesh
                      • fsdp2_config
                      • moe_config
                      • dp_size
                      • tp_size
                      • cp_size
                    • RuntimeConfig
                      • model_class
                      • model_config
                      • hf_config_overrides
                      • allow_flash_attn_args
                      • attn_impl
                      • dtype
                      • enable_seq_packing
                      • max_grad_norm
                      • cpu_offload
                      • offload_optimizer_for_logprob
                      • is_generation_colocated
                      • sampling_params
                      • is_reward_model
                    • ModelAndOptimizerState
                      • model
                      • optimizer
                      • scheduler
                      • is_hf_model
                      • is_moe_model
                      • is_reward_model
                      • model_class
                      • model_config
                      • peft_config
                      • autocast_enabled
              • nemo_rl.models.automodel.setup
                • Module Contents
                  • Functions
                  • Data
                  • API
                    • STRING_TO_DTYPE
                    • _maybe_set_force_hf()
                    • get_tokenizer()
                    • validate_and_prepare_config()
                    • setup_reference_model_state()
                    • setup_distributed()
                    • _disable_automodel_checkpoint_dtype_restore()
                    • setup_model_and_optimizer()
              • nemo_rl.models.automodel.train
                • Module Contents
                  • Classes
                  • Functions
                  • Data
                  • API
                    • PostProcessingFunction
                    • _needs_kv_cache_for_shared_layers()
                    • model_forward()
                    • extract_logits()
                    • apply_temperature_scaling()
                    • apply_top_k_top_p_filtering_for_local_logits()
                    • redistribute_logits_for_cp()
                    • prepare_data_for_cp()
                    • forward_with_post_processing_fn()
                    • automodel_forward_backward()
                    • LossPostProcessor
                      • __call__()
                    • LogprobsPostProcessor
                      • __call__()
                      • _compute_local_logprobs()
                    • TopkLogitsPostProcessor
                      • __call__()
                    • FullLogitsPostProcessor
                      • __call__()
                    • ScorePostProcessor
                      • __call__()
                    • aggregate_training_statistics()
          • nemo_rl.models.generation
            • Subpackages
              • nemo_rl.models.generation.vllm
                • Submodules
                  • nemo_rl.models.generation.vllm.vllm_worker_async
                    • Module Contents
                      • Classes
                      • Functions
                      • Data
                      • API
                        • LOGGER
                        • _replace_prefix_tokens()
                        • VllmAsyncGenerationWorkerImpl
                          • _return_routed_experts_enabled()
                          • _reserve_port()
                          • load_model()
                          • _create_engine()
                          • _install_engine_input_socket_lock()
                          • _start_vllm_metrics_logger()
                          • get_vllm_logger_metrics()
                          • clear_vllm_logger_metrics()
                          • post_init_async()
                          • get_reserved_url()
                          • report_dp_openai_server_base_url()
                          • _setup_vllm_openai_api_server()
                          • _setup_vllm_server()
                          • init_collective_async()
                          • generate_async()
                          • generate_text_async()
                          • report_device_id_async()
                          • prepare_refit_info_async()
                          • update_weights_via_ipc_zmq_async()
                          • update_weights_from_collective_async()
                          • reset_prefix_cache_async()
                          • sleep_async()
                          • wake_up_async()
                          • shutdown()
                        • VllmAsyncGenerationWorker
                  • nemo_rl.models.generation.vllm.vllm_worker
                    • Module Contents
                      • Classes
                      • Functions
                      • Data
                      • API
                        • logger
                        • _resolve_enable_prefix_caching()
                        • _merge_fp8_kwargs()
                        • BaseVllmGenerationWorker
                          • __repr__()
                          • configure_worker()
                          • _init_config()
                          • _load_model()
                          • llm()
                          • is_alive()
                          • _merge_stop_strings()
                          • _build_sampling_params()
                          • start_gpu_profiling()
                          • stop_gpu_profiling()
                          • _patch_vllm_nsight_config()
                          • _get_raw_spec_counters()
                        • VllmGenerationWorkerImpl
                          • _create_engine()
                          • post_init()
                          • init_collective()
                          • generate()
                          • generate_text()
                          • report_device_id()
                          • prepare_refit_info()
                          • update_weights_via_ipc_zmq()
                          • update_weights_from_collective()
                          • reset_prefix_cache()
                          • sleep()
                          • wake_up()
                          • shutdown()
                        • VllmGenerationWorker
                  • nemo_rl.models.generation.vllm.patches
                    • Module Contents
                      • Functions
                      • API
                        • _get_vllm_file()
                        • _locked_file_patch()
                        • _patch_vllm_init_workers_ray()
                        • _patch_vllm_llama_eagle3_own_lm_head()
                        • _patch_vllm_hermes_tool_parser_thread_safety()
                        • _apply_vllm_patches()
                  • nemo_rl.models.generation.vllm.vllm_backend
                    • Module Contents
                      • Classes
                      • Functions
                      • API
                        • fix_gemma3_vision_weight_name()
                        • _read_mtp_layer_weights_from_checkpoint()
                        • VllmInternalWorkerExtension
                          • bind_numa()
                          • init_collective()
                          • report_device_id()
                          • get_zmq_address()
                          • maybe_init_zmq()
                          • prepare_refit_info()
                          • _maybe_process_fp8_kv_cache()
                          • _split_policy_and_draft_weights()
                          • _trim_vocab_padding()
                          • _load_draft_weights()
                          • load_mtp_weights_from_disk()
                          • _load_weights()
                          • update_weights_via_ipc_zmq()
                          • update_weights_from_collective()
                          • cleanup()
                          • start_gpu_profiling()
                          • stop_gpu_profiling()
                  • nemo_rl.models.generation.vllm.config
                    • Module Contents
                      • Classes
                      • API
                        • VllmSpecificArgs
                          • tensor_parallel_size
                          • pipeline_parallel_size
                          • expert_parallel_size
                          • gpu_memory_utilization
                          • max_model_len
                          • skip_tokenizer_init
                          • async_engine
                          • load_format
                          • precision
                          • is_mx
                          • kv_cache_dtype
                          • enforce_eager
                          • enable_return_routed_experts
                          • use_tqdm
                          • expose_http_server
                          • http_server_serving_chat_kwargs
                          • tool_parser_plugin
                          • env_vars
                          • reasoning_parser_plugin
                        • VllmConfig
                          • vllm_cfg
                          • vllm_kwargs
                          • quant_cfg
                          • real_quant
                          • real_quant_ignore
                  • nemo_rl.models.generation.vllm.utils
                    • Module Contents
                      • Functions
                      • Data
                      • API
                        • R3_MISSING_ROUTE_SENTINEL
                        • format_prompt_for_vllm_generation()
                        • pad_and_align_routed_expert_indices()
                        • attach_routed_experts_to_chat_response_choices()
                        • model_dump_chat_response_with_routed_experts()
                        • aggregate_spec_decode_counters()
                        • compute_spec_decode_metrics()
                        • GENERATION_WORKER_OVERRIDES
                        • resolve_generation_worker_cls()
                  • nemo_rl.models.generation.vllm.vllm_generation
                    • Module Contents
                      • Classes
                      • Data
                      • API
                        • logger
                        • VllmGeneration
                          • init_cluster_placement_groups()
                          • _get_tied_worker_bundle_indices()
                          • _report_device_id()
                          • _report_dp_openai_server_base_urls()
                          • _collect_reserved_urls()
                          • load_and_start()
                          • _post_init()
                          • _get_raw_spec_counters()
                          • snapshot_step_metrics()
                          • get_step_metrics()
                          • init_collective()
                          • generate()
                          • generate_text()
                          • _async_generate_base()
                          • generate_text_async()
                          • generate_async()
                          • prepare_for_generation()
                          • finish_generation()
                          • shutdown()
                          • prepare_refit_info()
                          • update_weights_via_ipc_zmq()
                          • update_weights_from_collective()
                          • start_gpu_profiling()
                          • stop_gpu_profiling()
                          • get_vllm_logger_metrics()
                          • clear_vllm_logger_metrics()
                          • clear_logger_metrics()
                          • get_logger_metrics()
                          • __del__()
                          • invalidate_kv_cache()
                          • requires_kv_scale_sync
                • Package Contents
                  • Data
                  • API
                    • __all__
              • nemo_rl.models.generation.megatron
                • Submodules
                  • nemo_rl.models.generation.megatron.megatron_generation
                    • Module Contents
                      • Classes
                      • API
                        • MegatronGeneration
                          • init_cluster_placement_groups()
                          • init_collective()
                          • update_weights_from_collective()
                          • generate()
                          • generate_async()
                          • prepare_for_generation()
                          • finish_generation()
                          • preinit_nvshmem_collective()
                          • suspend_for_refit()
                          • resume_after_refit()
                          • prepare_refit_info()
                          • start_gpu_profiling()
                          • stop_gpu_profiling()
                          • shutdown()
                          • __del__()
                  • nemo_rl.models.generation.megatron.megatron_worker
                    • Module Contents
                      • Classes
                      • API
                        • MegatronGenerationMixin
                          • _init_inference_engine_state()
                          • _initialize_inference_engine()
                          • _start_inference_coordinator()
                          • _sleep()
                          • _sleep_engine()
                          • _wake()
                          • _wake_engine()
                          • _start_inference_loop_thread()
                          • _setup_openai_api_server()
                          • _run_async_coordinator_start()
                          • finish_generation()
                          • prepare_for_generation()
                          • report_dp_openai_server_base_url()
                          • _build_sampling_params()
                          • _merge_stop_strings()
                          • _prepare_data_for_generation()
                          • _parse_result_to_batched_data_dict()
                          • generate()
                          • generate_async()
                          • _generate_with_persistent_engine()
                        • MegatronGenerationRefitMixin
                          • init_collective_mcore_generation()
                          • preinit_nvshmem_collective()
                          • swap_weights_via_reshard()
                          • suspend_for_refit()
                          • resume_after_refit()
                  • nemo_rl.models.generation.megatron.config
                    • Module Contents
                      • Classes
                      • API
                        • MCoreGenerationSpecificArgs
                          • async_engine
                          • expose_http_server
                          • parsers
                          • buffer_size_gb
                          • block_size_tokens
                          • max_tokens
                          • max_model_len
                          • num_cuda_graphs
                          • use_cuda_graphs_for_non_decode_steps
                          • cuda_graph_impl
                          • inference_cuda_graph_scope
                          • materialize_only_last_token_logits
                          • enable_chunked_prefill
                          • enable_prefix_caching
                          • refit_backend
                          • num_speculative_tokens
                          • mamba_inference_ssm_states_dtype
                          • mamba_inference_conv_states_dtype
                          • kv_cache_management_mode
                          • logging_step_interval
                        • MCoreGenerationConfig
                          • mcore_generation_config
                  • nemo_rl.models.generation.megatron.utils
                    • Module Contents
                      • Functions
                      • API
                        • resolve_torch_dtype()
                        • log_gpu_memory()
                • Package Contents
                  • Data
                  • API
                    • __all__
            • Submodules
              • nemo_rl.models.generation.interfaces
                • Module Contents
                  • Classes
                  • Functions
                  • API
                    • verify_right_padding()
                    • ResourcesConfig
                      • gpus_per_node
                      • num_nodes
                    • OptionalResourcesConfig
                      • gpus_per_node
                      • num_nodes
                    • ColocationConfig
                      • enabled
                      • resources
                    • GenerationConfig
                      • backend
                      • max_new_tokens
                      • temperature
                      • top_p
                      • top_k
                      • model_name
                      • stop_token_ids
                      • stop_strings
                      • colocated
                      • port_range_low
                      • port_range_high
                      • use_async_rollouts
                      • _pad_token_id
                      • _mtp_weights_from_refit
                    • GenerationDatumSpec
                      • input_ids
                      • input_lengths
                      • stop_strings
                      • __extra__
                    • GenerationOutputSpec
                      • output_ids
                      • generation_lengths
                      • unpadded_sequence_lengths
                      • logprobs
                      • routed_experts
                      • r3_routed_experts_missing_routes
                      • r3_routed_experts_expected_routes
                      • r3_routed_experts_actual_routes
                      • truncated
                      • __extra__
                    • GenerationInterface
                      • init_collective()
                      • generate()
                      • prepare_for_generation()
                      • finish_generation()
                      • requires_kv_scale_sync
                      • prepare_refit_info()
                      • update_weights_via_ipc_zmq()
                      • update_weights_from_collective()
                      • invalidate_kv_cache()
                      • clear_logger_metrics()
                      • get_logger_metrics()
              • nemo_rl.models.generation.constants
                • Module Contents
                  • Data
                  • API
                    • VLLM_BACKEND
                    • SGLANG_BACKEND
                    • MEGATRON_BACKEND
            • Package Contents
              • Functions
              • Data
              • API
                • TokenizerType
                • configure_generation_config()
          • nemo_rl.models.policy
            • Subpackages
              • nemo_rl.models.policy.workers
                • Submodules
                  • nemo_rl.models.policy.workers.patches
                    • Module Contents
                      • Functions
                      • API
                        • _get_transformer_engine_file()
                        • apply_transformer_engine_patch()
                  • nemo_rl.models.policy.workers.dtensor_policy_worker_v2
                    • Module Contents
                      • Classes
                      • Functions
                      • API
                        • dtensor_params_generator()
                        • _maybe_merge_lora_weight()
                        • _maybe_adapt_tensor_to_hf()
                        • get_train_context()
                        • DTensorPolicyWorkerV2Impl
                          • __repr__()
                          • _get_replica_group()
                          • _local_coords()
                          • _update_moe_gate_bias_if_supported()
                          • set_rollout_num_gpus_per_engine()
                          • train()
                          • get_logprobs()
                          • score()
                          • get_topk_logits()
                          • get_full_logits_ipc()
                          • release_ipc_buffer()
                          • use_reference_model()
                          • _add_noise_to_weights()
                          • return_state_dict()
                          • return_model_config()
                          • prepare_refit_info()
                          • calibrate_qkv_fp8_scales()
                          • stream_weights_via_ipc_zmq()
                          • stream_weights_via_http()
                          • broadcast_weights_for_collective()
                          • prepare_for_lp_inference()
                          • prepare_for_training()
                          • finish_inference()
                          • offload_before_refit()
                          • offload_after_refit()
                          • move_optimizer_to_device()
                          • move_to_device()
                          • move_buffer_to_device()
                          • move_to_cuda()
                          • move_to_cpu()
                          • save_checkpoint()
                          • load_checkpoint()
                          • _init_checkpoint_manager()
                        • DTensorPolicyWorkerV2
                  • nemo_rl.models.policy.workers.dtensor_policy_worker
                    • Module Contents
                      • Classes
                      • Functions
                      • API
                        • _attach_context_parallel_hooks()
                        • unshard_fsdp2_model()
                        • get_cpu_state_dict()
                        • DTensorPolicyWorkerImpl
                          • __repr__()
                          • _get_replica_group()
                          • _local_coords()
                          • create_context_parallel_ctx()
                          • _apply_temperature_scaling()
                          • _apply_top_k_top_p_filtering()
                          • train_context()
                          • train()
                          • get_logprobs()
                          • score()
                          • get_topk_logits()
                          • use_reference_model()
                          • _add_noise_to_weights()
                          • return_state_dict()
                          • return_model_config()
                          • prepare_refit_info()
                          • calibrate_qkv_fp8_scales()
                          • stream_weights_via_ipc_zmq()
                          • broadcast_weights_for_collective()
                          • prepare_for_lp_inference()
                          • prepare_for_training()
                          • offload_before_refit()
                          • offload_after_refit()
                          • move_optimizer_to_device()
                          • move_to_device()
                          • move_buffer_to_device()
                          • move_to_cuda()
                          • move_to_cpu()
                          • save_checkpoint()
                          • load_checkpoint()
                        • DTensorPolicyWorker
                  • nemo_rl.models.policy.workers.megatron_policy_worker
                    • Module Contents
                      • Classes
                      • Functions
                      • Data
                      • API
                        • log
                        • TokenizerType
                        • _should_use_router_replay()
                        • _model_self_packs_for_cp()
                        • MegatronPolicyWorkerImpl
                          • _train_step_state
                          • __repr__()
                          • _local_coords()
                          • _get_replica_group()
                          • configure_worker()
                          • enable_forward_pre_hook()
                          • disable_forward_pre_hook()
                          • _forward_pre_hook_enabled()
                          • _disable_forward_pre_hook_until_next_train_step()
                          • _copy_main_params_to_param_buffer()
                          • _uses_mxfp8_overlap_shared_param_buffer()
                          • _get_model_extra_state_dict()
                          • _restore_model_extra_state_dict()
                          • train()
                          • _compute_moe_grad_scale()
                          • _set_moe_grad_scale_func()
                          • get_reference_policy_logprobs()
                          • _split_step_state_init()
                          • _assert_step_open()
                          • _restore_saved_grad_sync_func()
                          • begin_train_step()
                          • train_microbatch()
                          • _train_microbatch_body()
                          • finish_train_step()
                          • _finish_train_step_body()
                          • abort_train_step()
                          • get_logprobs()
                          • _apply_state_dict_to_model()
                          • use_reference_model()
                          • get_topk_logits()
                          • prepare_refit_info()
                          • _collect_mtp_metrics()
                          • _set_mtp_grad_scale_func()
                          • _get_model_config()
                          • _calculate_refit_param_info()
                          • _iter_params_with_optional_kv_scales()
                          • stream_weights_via_ipc_zmq()
                          • broadcast_weights_for_collective()
                          • _use_real_quant_refit()
                          • prepare_for_lp_inference()
                          • prepare_for_training()
                          • finish_inference()
                          • _clear_fp8_caches()
                          • offload_before_refit()
                          • offload_after_refit()
                          • move_model()
                          • move_optimizer()
                          • save_checkpoint()
                          • finalize_async_save()
                          • load_checkpoint()
                          • check_tensor_parallel_attributes()
                          • calibrate_qkv_fp8_scales()
                        • MegatronPolicyWorker
                  • nemo_rl.models.policy.workers.base_policy_worker
                    • Module Contents
                      • Classes
                      • API
                        • AbstractPolicyWorker
                          • init_collective()
                          • is_alive()
                          • reset_peak_memory_stats()
                          • get_gpu_info()
                          • report_device_id()
                          • get_zmq_address()
                          • maybe_init_zmq()
                          • get_free_memory_bytes()
                          • shutdown()
                          • start_gpu_profiling()
                          • stop_gpu_profiling()
                          • report_node_ip_and_gpu_id()
                          • get_reference_policy_logprobs()
                          • finalize_async_save()
                          • finish_training()
            • Submodules
              • nemo_rl.models.policy.interfaces
                • Module Contents
                  • Classes
                  • API
                    • LogprobOutputSpec
                      • logprobs
                    • ReferenceLogprobOutputSpec
                      • reference_logprobs
                    • ScoreOutputSpec
                      • scores
                    • TopkLogitsOutputSpec
                      • topk_logits
                      • topk_indices
                    • PolicyInterface
                      • get_logprobs()
                      • get_reference_policy_logprobs()
                      • get_topk_logits()
                      • train()
                      • calibrate_qkv_fp8_scales()
                      • prepare_for_training()
                      • finish_training()
                      • save_checkpoint()
                      • shutdown()
                    • ColocatablePolicyInterface
                      • init_collective()
                      • offload_before_refit()
                      • offload_after_refit()
                      • offload_to_cpu()
                      • prepare_refit_info()
                      • stream_weights_via_ipc_zmq()
                      • stream_weights_via_http()
                      • set_rollout_num_gpus_per_engine()
                      • broadcast_weights_for_collective()
                      • prepare_for_lp_inference()
              • nemo_rl.models.policy.utils
                • Module Contents
                  • Classes
                  • Functions
                  • Data
                  • API
                    • AUTOMODEL_FACTORY
                    • IPCProtocol
                      • COMPLETE
                      • ACK
                    • POLICY_WORKER_OVERRIDES
                    • resolve_policy_worker_cls()
                    • resolve_model_class()
                    • is_vllm_v1_engine_enabled()
                    • get_gpu_info()
                    • configure_dynamo_cache()
                    • get_runtime_env_for_policy_worker()
                    • get_megatron_checkpoint_dir()
                    • get_handle_from_tensor()
                    • ensure_teacher_ipc_buffer()
                    • aggregate_per_sample_handles()
                    • calculate_aligned_size()
                    • stream_weights_via_ipc_zmq_impl()
                    • rebuild_cuda_tensor_from_ipc()
                    • _ensure_ipc_topology()
                    • _flush_bucket()
                    • stream_weights_via_http_impl()
              • nemo_rl.models.policy.tq_policy
                • Module Contents
                  • Classes
                  • Functions
                  • API
                    • _aggregate_train_results()
                    • TQPolicy
                      • shutdown()
                      • prepare_step()
                      • prepare_val_partition()
                      • discard_samples()
                      • finish_step()
                      • _stamp_pad_seqlen()
                      • read_from_dataplane()
                      • write_to_dataplane()
                      • _packing_args()
                      • _logprob_dispatch()
                      • get_logprobs_from_meta()
                      • get_reference_policy_logprobs_from_meta()
                      • train_from_meta()
                      • begin_train_step()
                      • train_microbatches_from_meta()
                      • finish_train_step()
                      • abort_train_step()
              • nemo_rl.models.policy.lm_policy
                • Module Contents
                  • Classes
                  • Functions
                  • Data
                  • API
                    • PathLike
                    • _aggregate_megatron_flops_metrics()
                    • Policy
                      • data_parallel_size
                      • run_all_workers_single_data()
                      • run_all_workers_multiple_data()
                      • init_collective()
                      • init_collective_mcore_generation()
                      • preinit_nvshmem()
                      • swap_weights_via_reshard()
                      • _shard_for_logprob()
                      • _shard_for_train()
                      • get_logprobs()
                      • get_reference_policy_logprobs()
                      • get_topk_logits()
                      • get_full_logits_ipc()
                      • release_ipc_buffer()
                      • train()
                      • generate()
                      • score()
                      • prepare_for_generation()
                      • finish_generation()
                      • prepare_for_training()
                      • prepare_for_lp_inference()
                      • invalidate_kv_cache()
                      • prepare_refit_info()
                      • finish_inference()
                      • finish_training()
                      • calibrate_qkv_fp8_scales()
                      • get_free_memory_bytes()
                      • stream_weights_via_ipc_zmq()
                      • stream_weights_via_http()
                      • set_rollout_num_gpus_per_engine()
                      • broadcast_weights_for_collective()
                      • offload_before_refit()
                      • offload_after_refit()
                      • offload_to_cpu()
                      • save_checkpoint()
                      • finalize_async_save()
                      • shutdown()
                      • __del__()
                      • start_gpu_profiling()
                      • stop_gpu_profiling()
                      • print_node_ip_and_gpu_id()
              • nemo_rl.models.policy.teacher_worker_group
                • Module Contents
                  • Classes
                  • Functions
                  • API
                    • TeacherConfig
                      • alias
                      • model_name
                      • tensor_model_parallel_size
                      • pipeline_model_parallel_size
                      • context_parallel_size
                      • expert_model_parallel_size
                      • num_nodes
                      • gpus_per_node
                      • precision
                      • micro_batch_size
                      • megatron_cfg_overrides
                    • create_teacher_configs_from_opd_config()
                    • TeacherWorkerGroup
                      • get_logprobs()
                      • shutdown()
                      • __del__()
            • Package Contents
              • Classes
              • Functions
              • Data
              • API
                • _patch_transformers_tokenizer_class_set()
                • LoRAConfigDisabled
                  • enabled
                • LoRAConfig
                  • enabled
                  • target_modules
                  • exclude_modules
                  • match_all_linear
                  • dim
                  • alpha
                  • dropout
                  • dropout_position
                  • lora_A_init
                  • use_triton
                • AutomodelBackendConfig
                  • _target_
                  • attn
                  • linear
                  • rms_norm
                  • experts
                  • dispatcher
                  • enable_deepep
                  • fake_balanced_gate
                  • enable_hf_state_dict_adapter
                  • enable_fsdp_optimizations
                  • gate_precision
                • AutomodelFreezeConfig
                  • freeze_vision_tower
                  • freeze_audio_tower
                  • freeze_language_model
                • AutomodelKwargs
                  • use_liger_kernel
                  • backend
                  • freeze_config
                  • force_hf
                • DTensorConfigDisabled
                  • enabled
                • MoEParallelizerOptions
                  • ignore_router_for_ac
                  • reshard_after_forward
                  • lm_head_precision
                  • wrap_outer_model
                • DTensorConfig
                  • enabled
                  • env_vars
                  • _v2
                  • tensor_parallel_size
                  • context_parallel_size
                  • expert_parallel_size
                  • dp_replicate_size
                  • sequence_parallel
                  • activation_checkpointing
                  • cpu_offload
                  • custom_parallel_plan
                  • defer_fsdp_grad_sync
                  • moe_parallelizer
                  • lora_cfg
                  • automodel_kwargs
                  • clear_cache_every_n_steps
                • SequencePackingConfigDisabled
                  • enabled
                • SequencePackingConfig
                  • enabled
                  • train_mb_tokens
                  • logprob_mb_tokens
                  • algorithm
                • RewardModelConfig
                  • enabled
                  • reward_model_type
                • MegatronPeftConfigDisabled
                  • enabled
                • MegatronPeftConfig
                  • enabled
                  • target_modules
                  • exclude_modules
                  • dim
                  • alpha
                  • dropout
                  • dropout_position
                  • lora_A_init_method
                  • lora_B_init_method
                  • a2a_experimental
                  • lora_dtype
                • MegatronOptimizerConfig
                  • optimizer
                  • lr
                  • min_lr
                  • weight_decay
                  • bf16
                  • fp16
                  • params_dtype
                  • adam_beta1
                  • adam_beta2
                  • adam_eps
                  • sgd_momentum
                  • use_distributed_optimizer
                  • use_precision_aware_optimizer
                  • clip_grad
                  • optimizer_cpu_offload
                  • optimizer_offload_fraction
                • MegatronSchedulerConfig
                  • start_weight_decay
                  • end_weight_decay
                  • weight_decay_incr_style
                  • lr_decay_style
                  • lr_decay_iters
                  • lr_warmup_iters
                  • lr_warmup_init
                • MegatronDDPConfig
                  • grad_reduce_in_fp32
                  • overlap_grad_reduce
                  • overlap_param_gather
                  • use_custom_fsdp
                  • data_parallel_sharding_strategy
                • Fp8Config
                  • enabled
                  • fp8
                  • fp8_recipe
                  • fp8_param
                  • force_clear_fp8_caches
                • MegatronConfigDisabled
                  • enabled
                • MegatronCheckpointConfig
                  • async_save
                  • ckpt_assume_constant_structure
                  • ckpt_fully_parallel_save_process_group
                  • ckpt_fully_parallel_load_process_group
                  • ckpt_fully_parallel_load_exchange_algo
                • MegatronConfig
                  • enabled
                  • env_vars
                  • empty_unused_memory_level
                  • activation_checkpointing
                  • recompute_granularity
                  • recompute_modules
                  • tensor_model_parallel_size
                  • pipeline_model_parallel_size
                  • num_layers_in_first_pipeline_stage
                  • num_layers_in_last_pipeline_stage
                  • context_parallel_size
                  • pipeline_dtype
                  • sequence_parallel
                  • freeze_moe_router
                  • expert_tensor_parallel_size
                  • expert_model_parallel_size
                  • defer_fp32_logits
                  • apply_rope_fusion
                  • bias_activation_fusion
                  • force_reconvert_from_hf
                  • attention_backend
                  • moe_per_layer_logging
                  • moe_enable_deepep
                  • moe_token_dispatcher_type
                  • inference_moe_token_dispatcher_type
                  • inference_grouped_gemm_backend
                  • moe_router_num_groups
                  • moe_router_group_topk
                  • transformer_impl
                  • cuda_graph_impl
                  • moe_pad_experts_for_cuda_graph_inference
                  • moe_shared_expert_overlap
                  • use_gloo_process_groups
                  • moe_grouped_gemm
                  • moe_flex_dispatcher_backend
                  • moe_hybridep_num_sms
                  • hybridep_num_ranks_per_nvlink_domain
                  • hybridep_use_mnnvl
                  • peft
                  • optimizer
                  • scheduler
                  • distributed_data_parallel_config
                  • checkpoint
                  • gradient_accumulation_fusion
                  • use_fused_weighted_squared_relu
                  • use_fused_linear_logprobs
                  • fused_linear_logprobs_chunk_size
                  • mtp_num_layers
                  • mtp_loss_scaling_factor
                  • mtp_use_repeated_layer
                  • mtp_detach_heads
                  • clear_memory_caches_before_refit
                  • fp8_cfg
                • DraftConfigDisabled
                  • enabled
                • DraftConfig
                  • enabled
                  • model_name
                  • loss_weight
                  • num_layers
                  • aux_layer_indices
                • TokenizerConfig
                  • name
                  • chat_template
                  • chat_template_kwargs
                  • audio
                  • video
                  • use_processor
                • PytorchOptimizerConfig
                  • name
                  • kwargs
                • SinglePytorchSchedulerConfig
                  • name
                  • kwargs
                • SinglePytorchMilestonesConfig
                  • milestones
                • SchedulerMilestones
                • DynamicBatchingConfigDisabled
                  • enabled
                • DynamicBatchingConfig
                  • enabled
                  • train_mb_tokens
                  • logprob_mb_tokens
                  • sequence_length_round
                • RouterReplayConfigDisabled
                  • enabled
                • RouterReplayConfig
                  • enabled
                • PolicyConfig
                  • model_name
                  • tokenizer
                  • train_global_batch_size
                  • train_micro_batch_size
                  • logprob_batch_size
                  • logprob_chunk_size
                  • generation
                  • generation_batch_size
                  • precision
                  • reward_model_cfg
                  • dtensor_cfg
                  • megatron_cfg
                  • draft
                  • pretrained_checkpoint
                  • router_replay
                  • hf_config_overrides
                  • dynamic_batching
                  • sequence_packing
                  • make_sequence_length_divisible_by
                  • max_total_sequence_length
                  • max_grad_norm
                  • refit_buffer_size_gb
                  • optimizer
                  • scheduler
                  • quant_cfg
                  • quant_calib_data
                  • quant_calib_size
                  • quant_batch_size
                  • quant_sequence_length
                  • disable_modelopt_layer_spec
          • nemo_rl.models.value
            • Subpackages
              • nemo_rl.models.value.workers
                • Submodules
                  • nemo_rl.models.value.workers.megatron_value_worker
                    • Module Contents
                      • Classes
                      • Functions
                      • Data
                      • API
                        • TokenizerType
                        • _install_value_head_load_skip()
                        • make_value_head_hook()
                        • _unpack_value_sequences()
                        • _value_loss_prepare_fn()
                        • MegatronValueWorkerImpl
                          • __repr__()
                          • configure_worker()
                          • enable_forward_pre_hook()
                          • disable_forward_pre_hook()
                          • train()
                          • get_values()
                          • prepare_for_training()
                          • prepare_for_inference()
                          • move_model()
                          • move_optimizer()
                          • save_checkpoint()
                          • load_checkpoint()
                          • finish_inference()
                          • finish_training()
                        • MegatronValueWorker
                  • nemo_rl.models.value.workers.dtensor_value_worker_v2
                    • Module Contents
                      • Classes
                      • Functions
                      • API
                        • right_shift_values()
                        • RightShiftLossWrapper
                          • __call__()
                          • __getattr__()
                        • get_train_context()
                        • DTensorValueWorkerV2Impl
                          • __repr__()
                          • train()
                          • get_values()
                          • prepare_for_training()
                          • prepare_for_inference()
                          • finish_training()
                          • finish_inference()
                          • move_optimizer_to_device()
                          • move_to_device()
                          • move_buffer_to_device()
                          • move_to_cuda()
                          • move_to_cpu()
                          • save_checkpoint()
                          • load_checkpoint()
                          • _init_checkpoint_manager()
                        • DTensorValueWorkerV2
            • Submodules
              • nemo_rl.models.value.interfaces
                • Module Contents
                  • Classes
                  • API
                    • ValueOutputSpec
                      • values
                    • ValueInterface
                      • get_values()
                      • train()
                      • prepare_for_training()
                      • prepare_for_inference()
                      • finish_training()
                      • save_checkpoint()
                      • shutdown()
              • nemo_rl.models.value.config
                • Module Contents
                  • Classes
                  • API
                    • ValueConfig
                      • model_name
                      • tokenizer
                      • train_global_batch_size
                      • train_micro_batch_size
                      • logprob_batch_size
                      • precision
                      • reward_model_cfg
                      • dtensor_cfg
                      • megatron_cfg
                      • hf_config_overrides
                      • dynamic_batching
                      • sequence_packing
                      • make_sequence_length_divisible_by
                      • max_total_sequence_length
                      • max_grad_norm
                      • dequantize_base_checkpoint
                      • optimizer
                      • scheduler
              • nemo_rl.models.value.lm_value
                • Module Contents
                  • Classes
                  • Data
                  • API
                    • PathLike
                    • Value
                      • get_values()
                      • train()
                      • prepare_for_training()
                      • prepare_for_inference()
                      • finish_inference()
                      • finish_training()
                      • save_checkpoint()
                      • shutdown()
                      • __del__()
            • Package Contents
              • Data
              • API
                • __all__
          • nemo_rl.models.dtensor
            • Submodules
              • nemo_rl.models.dtensor.parallelize
                • Module Contents
                  • Classes
                  • Functions
                  • Data
                  • API
                    • RotaryEmbedParallel
                      • _prepare_input_fn()
                      • _prepare_output_fn()
                    • ColwiseParallelWithGather
                      • _prepare_output_fn()
                      • _apply()
                    • _parallelize_gemma3()
                    • _parallelize_llama()
                    • _parallelize_qwen()
                    • PARALLIZE_FUNCTIONS
                    • translate_parallel_style()
                    • get_hf_tp_plan()
                    • _parallelize_nm5_h()
                    • _parallelize_model()
                    • to_local_if_dtensor()
                    • clip_grad_by_total_norm_()
                    • get_grad_norm()
          • nemo_rl.models.huggingface
            • Submodules
              • nemo_rl.models.huggingface.common
                • Module Contents
                  • Classes
                  • Functions
                  • Data
                  • API
                    • Tensor
                    • FlashAttentionKwargs
                      • cu_seqlens_q
                      • cu_seqlens_k
                      • max_seqlen_q
                      • max_seqlen_k
                    • ModelFlag
                      • VLLM_LOAD_FORMAT_AUTO
                      • matches()
                    • is_gemma_model()
                    • is_nano_nemotron_vl_model()
                    • group_and_cat_tensors()
                    • pack_sequences()
                    • unpack_tensor()
                    • get_flash_attention_kwargs()
          • nemo_rl.models.megatron
            • Subpackages
              • nemo_rl.models.megatron.draft
                • Submodules
                  • nemo_rl.models.megatron.draft.hidden_capture
                    • Module Contents
                      • Classes
                      • Functions
                      • Data
                      • API
                        • get_eagle3_aux_hidden_state_layers()
                        • _DTYPE_TO_CODE
                        • _CODE_TO_DTYPE
                        • CapturedStates
                          • hidden_states
                          • inputs_embeds
                        • HiddenStateCapture
                          • _compute_local_layer_mapping()
                          • _compute_layer_owner_map()
                          • _make_layer_output_hook()
                          • _make_embedding_hook()
                          • register_hooks()
                          • clear_hooks()
                          • capture_context()
                          • _assemble_local_states()
                          • _owner_rank_for_global_layer()
                          • _send_tensor()
                          • _recv_tensor()
                          • _gather_distributed()
                          • get_captured_states()
                        • get_capture_context()
                  • nemo_rl.models.megatron.draft.utils
                    • Module Contents
                      • Classes
                      • Functions
                      • Data
                      • API
                        • StateDict
                        • CheckpointLoader
                        • _CHECKPOINT_CANDIDATE_NAMES
                        • _HF_SNAPSHOT_ALLOW_PATTERNS
                        • _HF_SNAPSHOT_IGNORE_PATTERNS
                        • _MODEL_LAYER_QKV_KEY_PATTERN
                        • _CHECKPOINT_LAYER_KEY_PATTERN
                        • _EagleLayerLayout
                          • layer_index
                          • model_prefix
                          • checkpoint_prefix
                          • hidden_norm_key
                          • input_layernorm_key
                          • post_attention_layernorm_key
                          • qkv_weight_key
                          • proj_weight_key
                          • fc1_weight_key
                          • fc2_weight_key
                        • _resolve_optional_key()
                        • _EagleModelLayout
                          • layers
                          • final_norm_key
                          • lm_head_key
                          • detect()
                          • layer_by_index
                        • _qkv_head_dims()
                        • _interleave_qkv()
                        • _deinterleave_qkv()
                        • _combine_or_shard_weight_parts()
                        • _PendingLayerWeights
                          • qkv_weight
                          • q_weight
                          • k_weight
                          • v_weight
                          • fc1_weight
                          • gate_weight
                          • up_weight
                          • apply_to()
                        • _get_num_aux_hidden_states()
                        • _all_gather_tp_shards()
                        • _gather_tp_qkv_weight()
                        • _gather_tp_gate_up_weight()
                        • _gather_tp_weight_if_needed()
                        • _extract_tensor_state_dict()
                        • _load_safetensors_file()
                        • _load_torch_file()
                        • _merge_checkpoint_shards()
                        • _load_index_checkpoint()
                        • _load_checkpoint_file()
                        • _load_checkpoint_from_directory()
                        • _load_checkpoint_state()
                        • _normalize_hf_key()
                        • _parse_layer_checkpoint_key()
                        • _get_tp_rank()
                        • _build_split_axis_by_parameter()
                        • _shard_to_local_tp()
                        • _assign_optional_layer_weight()
                        • _map_layer_hf_weight()
                        • _map_hf_state_to_eagle_state()
                        • load_hf_weights_to_eagle()
                        • _require_state_tensor()
                        • find_draft_owner_chunk()
                        • get_attached_draft_model()
                        • _export_layer_weights_to_hf()
                        • export_eagle_weights_to_hf()
                        • get_policy_lm_head_weight()
                        • _get_draft_output_layer()
                        • _get_draft_to_target_token_mapping()
                        • copy_policy_lm_head_to_draft()
                        • build_draft_model()
                  • nemo_rl.models.megatron.draft.eagle
                    • Module Contents
                      • Classes
                      • API
                        • EagleModel
                          • sharded_state_dict()
                          • forward()
                • Package Contents
                  • Data
                  • API
                    • __all__
            • Submodules
              • nemo_rl.models.megatron.data
                • Module Contents
                  • Classes
                  • Functions
                  • API
                    • ProcessedInputs
                      • input_ids
                      • input_ids_cp_sharded
                      • attention_mask
                      • position_ids
                      • packed_seq_params
                      • cu_seqlens_padded
                      • mtp_loss_mask
                      • routed_experts
                      • routed_experts_cp_sharded
                    • ProcessedMicrobatch
                      • data_dict
                      • input_ids
                      • input_ids_cp_sharded
                      • attention_mask
                      • position_ids
                      • packed_seq_params
                      • cu_seqlens_padded
                      • mtp_loss_mask
                      • routed_experts
                      • routed_experts_cp_sharded
                    • make_processed_microbatch_iterator()
                    • get_microbatch_iterator()
                    • get_ltor_masks_and_position_ids()
                    • process_microbatch()
                    • _make_r3_trace_token_identity()
                    • _verify_r3_trace_cp_token_alignment()
                    • _fill_routed_experts_padding()
                    • process_global_batch()
                    • _prepare_vlm_batch_for_megatron()
                    • _pack_sequences_for_megatron()
                    • _shard_routed_experts_for_cp()
                    • _get_pack_sequence_parameters_for_megatron()
                    • _unpack_sequences_from_megatron()
                    • get_and_validate_seqlen()
              • nemo_rl.models.megatron.router_replay
                • Module Contents
                  • Functions
                  • Data
                  • API
                    • _ROUTER_REPLAY_VALIDATE_ENV
                    • _MISSING_ROUTE_SENTINEL
                    • _MISSING_ROUTE_FALLBACK_PATCH_ATTR
                    • router_replay_enabled()
                    • configure_vllm_for_router_replay()
                    • validate_router_replay_config()
                    • _iter_model_modules()
                    • _unwrap_model_config()
                    • _global_moe_layer_numbers()
                    • _router_replay_instances_for_model()
                    • _local_layer_numbers_for_model()
                    • _normalize_routed_experts_for_mcore()
                    • _payload_indices_for_moe_layers()
                    • _router_replay_validation_enabled()
                    • _validate_replay_tensor()
                    • _install_missing_route_fallback_patch()
                    • _get_tensor_model_parallel_world_size()
                    • _get_tensor_model_parallel_rank()
                    • _split_for_sequence_parallel()
                    • build_router_replay_tensors()
                    • build_router_replay_assignments()
                    • set_router_replay_forward()
                    • set_router_replay_backward()
                    • clear_router_replay()
                    • clear_global_router_replay_instances()
              • nemo_rl.models.megatron.config
                • Module Contents
                  • Classes
                  • API
                    • RuntimeConfig
                      • megatron_cfg
                      • model_cfg
                      • dtype
                      • optimizer_cpu_offload
                      • offload_optimizer_for_logprob
                      • is_generation_colocated
                      • sampling_params
                      • final_padded_vocab_size
                    • ModelAndOptimizerState
                      • state
                      • model
                      • optimizer
                      • scheduler
                      • checkpointing_context
                      • param_sync_func
                      • draft_model
              • nemo_rl.models.megatron.setup
                • Module Contents
                  • Classes
                  • Functions
                  • Data
                  • API
                    • _HF_CONFIG_PATCHED
                    • _patch_hf_config_double_instantiation()
                    • TokenizerType
                    • destroy_parallel_state()
                    • setup_distributed()
                    • validate_and_set_config()
                    • _canonicalize_hf_config_overrides()
                    • _get_hf_config_overrides_hash()
                    • _resolve_iter_dir_from_root()
                    • validate_model_paths()
                    • setup_model_config()
                    • _apply_parallelism_config()
                    • _apply_moe_config()
                    • _apply_mtp_config()
                    • _apply_precision_config()
                    • _apply_performance_config()
                    • _validate_optimizer_config()
                    • _validate_chunking_config()
                    • _create_checkpoint_config()
                    • _validate_training_config()
                    • _validate_dtype_config()
                    • _create_megatron_config()
                    • _create_draft_pre_wrap_hook()
                    • _BRIDGE_SIGNAL_HANDLER_PATCHED
                    • _patch_bridge_signal_handler_for_worker_threads()
                    • setup_model_and_optimizer()
                    • handle_model_import()
                    • setup_reference_model_state()
                    • finalize_megatron_setup()
                    • MoEFloat16Module
                      • re_enable_float32_expert_bias()
                    • make_policy_like_config()
              • nemo_rl.models.megatron.pipeline_parallel
                • Module Contents
                  • Functions
                  • API
                    • broadcast_obj_from_pp_rank()
                    • broadcast_loss_metrics_from_last_stage()
                    • broadcast_tensors_from_last_stage()
              • nemo_rl.models.megatron.train
                • Module Contents
                  • Classes
                  • Functions
                  • Data
                  • API
                    • PostProcessingFunction
                    • model_forward()
                    • apply_temperature_scaling()
                    • forward_with_post_processing_fn()
                    • megatron_forward_backward()
                    • LossPostProcessor
                      • __call__()
                    • LogprobsPostProcessor
                      • __call__()
                    • TopkLogitsPostProcessor
                      • __call__()
                    • aggregate_training_statistics()
              • nemo_rl.models.megatron.community_import
                • Module Contents
                  • Functions
                  • API
                    • to_torch_dtype()
                    • _prefer_nvrx_for_dist_ckpt_save()
                    • import_model_from_hf_name()
                    • export_model_from_megatron()
              • nemo_rl.models.megatron.common
                • Module Contents
                  • Functions
                  • API
                    • _round_up_to_multiple()
                    • broadcast_tensor()
                    • get_moe_metrics()
                    • get_mtp_metrics()
      • nemo_rl.data_plane
        • Subpackages
          • nemo_rl.data_plane.adapters
            • Submodules
              • nemo_rl.data_plane.adapters.noop
                • Module Contents
                  • Classes
                  • Functions
                  • API
                    • _reject_non_tensor_leaves()
                    • _Partition
                      • fields
                      • num_samples
                      • consumer_tasks
                      • grpo_group_size
                      • enums
                      • rows
                      • tags
                      • consumed
                    • NoOpDataPlaneClient
                      • register_partition()
                      • claim_meta()
                      • get_data()
                      • check_consumption_status()
                      • put_samples()
                      • get_samples()
                      • clear_samples()
                      • close()
              • nemo_rl.data_plane.adapters.transfer_queue
                • Module Contents
                  • Classes
                  • Functions
                  • Data
                  • API
                    • _get_local_node_ip()
                    • _mooncake_transport_config()
                    • _connect_existing()
                    • _TQ_RUNTIME_ENV_PATCHED
                    • _resolve_tq_pin()
                    • _patch_tq_actor_runtime_env()
                    • _init_tq()
                    • _assert_no_key_loss()
                    • _promote_1d_leaves()
                    • _from_wire()
                    • TQDataPlaneClient
                      • register_partition()
                      • claim_meta()
                      • get_data()
                      • check_consumption_status()
                      • put_samples()
                      • get_samples()
                      • clear_samples()
                      • close()
        • Submodules
          • nemo_rl.data_plane.factory
            • Module Contents
              • Functions
              • API
                • build_data_plane_client()
          • nemo_rl.data_plane.interfaces
            • Module Contents
              • Classes
              • API
                • DataPlaneConfig
                  • enabled
                  • impl
                  • backend
                  • storage_capacity
                  • num_storage_units
                  • claim_meta_poll_interval_s
                  • global_segment_size
                  • local_buffer_size
                  • controller_address
                  • ack_timeout_ms
                  • observability
                • ObservabilityConfig
                  • enabled
                  • callback
                • KVBatchMeta
                  • partition_id
                  • task_name
                  • sample_ids
                  • fields
                  • sequence_lengths
                  • extra_info
                  • tags
                  • __post_init__()
                  • size
                  • stamp_tags()
                  • _replace()
                  • subset()
                  • slice()
                  • concat()
                • DataPlaneClient
                  • register_partition()
                  • claim_meta()
                  • get_data()
                  • check_consumption_status()
                  • put_samples()
                  • get_samples()
                  • clear_samples()
                  • close()
          • nemo_rl.data_plane.schema
            • Module Contents
              • Functions
              • Data
              • API
                • Layout
                • MICRO_BATCH_INDICES
                • MICRO_BATCH_LENGTHS
                • ELEM_COUNTS_PER_GB
                • GLOBAL_FORWARD_PAD_SEQLEN
                • INPUT_IDS
                • INPUT_LENGTHS
                • SAMPLE_MASK
                • META_IDX
                • DP_TRAIN_FIELDS
                • LP_SEED_FIELDS
                • DP_CALIB_INPUT_FIELDS
                • ROUTED_EXPERTS_FIELD
                • fields_with_optional_routed_experts()
          • nemo_rl.data_plane.column_io
            • Module Contents
              • Functions
              • Data
              • API
                • TOKEN_ALIGNED_FIELDS
                • round_up()
                • read_columns()
                • write_columns()
                • kv_first_write()
          • nemo_rl.data_plane.preshard
            • Module Contents
              • Functions
              • API
                • shard_meta_for_dp()
          • nemo_rl.data_plane.codec
            • Module Contents
              • Functions
              • API
                • to_nested_by_length()
                • stack_or_nest()
                • unwrap_wire_stripped_payload()
                • pack_jagged_fields()
                • pack_per_token_field()
                • response_from_nested()
                • materialize()
          • nemo_rl.data_plane.worker_mixin
            • Module Contents
              • Classes
              • Functions
              • Data
              • API
                • FetchPolicy
                • _broadcast_batched_data_dict()
                • TQWorkerMixin
                  • _dp_client
                  • setup_data_plane()
                  • _require_dp_client()
                  • _get_replica_group()
                  • _pad_value_dict()
                  • _forward_pad_seqlen()
                  • _fetch()
                  • _apply_packing_prep()
                  • _attach_or_repack_pack_metadata()
                  • _local_coords()
                  • _is_replica_leader()
                  • _write_back()
                  • _write_back_result_field()
                  • train_presharded()
                  • get_logprobs_presharded()
                  • get_reference_policy_logprobs_presharded()
                  • begin_train_step_presharded()
                  • train_microbatch_presharded()
                  • finish_train_step_presharded()
                  • abort_train_step_presharded()
          • nemo_rl.data_plane.observability
            • Module Contents
              • Classes
              • Functions
              • Data
              • API
                • EventStatus
                • DataPlaneEvent
                  • op
                  • partition_id
                  • n_keys
                  • n_bytes
                  • wall_ms
                  • status
                • logger
                • _td_bytes()
                • log_event()
                • DataPlaneStats
                  • total_bytes
                  • total_keys
                  • total_ops
                  • bytes_outstanding
                  • peak_bytes_outstanding
                  • max_bytes_per_key_seen
                  • last_put_bytes_per_key
                • MetricsDataPlaneClient
                  • snapshot()
                  • bytes_outstanding_by_partition()
                  • _record_put()
                  • _record_clear()
                  • _run()
                  • _emit()
                  • register_partition()
                  • claim_meta()
                  • get_data()
                  • check_consumption_status()
                  • put_samples()
                  • get_samples()
                  • clear_samples()
                  • close()
        • Package Contents
          • Data
          • API
            • __all__
      • nemo_rl.modelopt
        • Subpackages
          • nemo_rl.modelopt.models
            • Subpackages
              • nemo_rl.modelopt.models.generation
                • Submodules
                  • nemo_rl.modelopt.models.generation.vllm_modelopt_patch
                    • Module Contents
                      • Functions
                      • Data
                      • API
                        • _DENSE_HF_PARAMS
                        • _MODELOPT_W4A16_QUANT_MODES
                        • _MODELOPT_W4A16_ATTR
                        • _ORIGINAL_NVFP4_CONFIG_FROM_CONFIG_ATTR
                        • _ORIGINAL_LINEAR_APPLY_ATTR
                        • _unwrap_vllm_model()
                        • _canonicalize_nvfp4_weight_scale()
                        • _requests_w4a16_modelopt_config()
                        • _is_w4a16_modelopt_quant_config()
                        • _modelopt_nvfp4_config_from_config()
                        • _convert_nvfp4_linear_kernel_format()
                        • _convert_w4a16_linear_kernel_format()
                        • _capture_modelopt_dense_param_reload_meta()
                        • _modelopt_dense_process_w4a16_weights()
                        • _modelopt_dense_process_weights()
                        • _modelopt_dense_apply()
                        • prepare_modelopt_for_weight_reload()
                        • modelopt_process_weights_after_loading()
                        • _patched
                        • apply_modelopt_nvfp4_patches()
                  • nemo_rl.modelopt.models.generation.vllm_quant_patch
                    • Module Contents
                      • Classes
                      • Functions
                      • API
                        • _tolerate_dummy_weight_nan_amax()
                        • _fakequant_run_prolog_worker()
                        • FakeQuantWorker
                          • determine_available_memory()
                          • compile_or_warm_up_model()
                  • nemo_rl.modelopt.models.generation.vllm_quant_backend
                    • Module Contents
                      • Classes
                      • API
                        • VllmQuantInternalWorkerExtension
                          • _is_real_quant_model()
                          • _patch_named_parameters_to_include_buffers()
                          • _load_weights()
                          • update_weights_via_ipc_zmq()
                          • update_weights_from_collective()
                          • get_weight_snapshot()
                          • get_quantizer_stats()
                  • nemo_rl.modelopt.models.generation.vllm_quant_worker
                    • Module Contents
                      • Classes
                      • Functions
                      • Data
                      • API
                        • _EXTRA_ENV_VARS
                        • _configure_quant_engine_kwargs()
                        • VllmQuantGenerationWorker
                          • _create_engine()
                          • get_quantizer_stats()
                          • get_weight_snapshot()
                        • VllmQuantAsyncGenerationWorker
                          • _create_engine()
                          • get_quantizer_stats()
                          • get_weight_snapshot()
              • nemo_rl.modelopt.models.policy
                • Subpackages
                  • nemo_rl.modelopt.models.policy.workers
                    • Submodules
                      • nemo_rl.modelopt.models.policy.workers.utils
                        • Module Contents
                          • Classes
                          • Functions
                          • Data
                          • API
                            • MAX_SEQ_LEN
                            • MAX_OUTPUT_LEN
                            • DEFAULT_CALIB_BATCH_SIZE
                            • DEFAULT_CALIB_SAMPLE_LENGTH
                            • symlink_pre_quantized_model()
                            • get_tokenizer()
                            • _DictDataset
                              • __getitem__()
                              • __len__()
                            • get_forward_loop_func()
                            • quantize_model()
                            • get_modelopt_checkpoint_dir()
                            • get_quantization_layer_spec()
                            • get_quantization_mamba_stack_spec()
                      • nemo_rl.modelopt.models.policy.workers.dtensor_quant_policy_worker_v2
                        • Module Contents
                          • Classes
                          • API
                            • DTensorQuantPolicyWorkerV2
                      • nemo_rl.modelopt.models.policy.workers.dtensor_quant_policy_worker
                        • Module Contents
                          • Classes
                          • API
                            • DTensorQuantPolicyWorker
                      • nemo_rl.modelopt.models.policy.workers.megatron_quant_policy_worker
                        • Module Contents
                          • Classes
                          • Functions
                          • API
                            • _w4a16_modelopt_exporter()
                            • MegatronQuantPolicyWorker
                              • _quantize()
                              • _patch_validate_model_paths()
                              • _patch_setup_model_and_optimizer()
                              • _restore_modelopt_state_pre_load()
                              • hide_tensor_quantizers()
                              • enable_forward_pre_hook()
                              • disable_forward_pre_hook()
                              • disable_quantization()
                              • _hide_extra_state()
                              • use_reference_model()
                              • without_model_config()
                              • get_quantizer_stats()
                              • generate()
                              • save_checkpoint()
                              • _use_real_quant_refit()
                              • _iter_real_quant_refit_params()
                              • _find_weight_quantizer()
                              • _iter_hf_input_amax_names()
                              • _get_enabled_input_amax()
                              • _iter_input_quantizer_amax_params()
                              • _iter_params_with_optional_kv_scales()
        • Submodules
          • nemo_rl.modelopt.utils
            • Module Contents
              • Functions
              • Data
              • API
                • _QUANT_IGNORE_NAME_SUFFIXES
                • DEFAULT_NVFP4_IGNORE
                • _iter_quant_ignore_suffix_variants()
                • iter_quant_ignore_name_candidates()
                • matches_quant_ignore_pattern()
                • build_vllm_modelopt_nvfp4_config()
                • resolve_quant_cfg()
          • nemo_rl.modelopt.registry
            • Module Contents
              • Data
              • API
                • USE_SYSTEM_EXECUTABLE
                • MODELOPT_VLLM_EXECUTABLE
                • MODELOPT_AUTOMODEL_EXECUTABLE
                • MODELOPT_MCORE_EXECUTABLE
                • MODELOPT_ACTOR_REGISTRY
      • nemo_rl.data
        • Subpackages
          • nemo_rl.data.packing
            • Submodules
              • nemo_rl.data.packing.algorithms
                • Module Contents
                  • Classes
                  • Functions
                  • API
                    • PackingAlgorithm
                      • CONCATENATIVE
                      • FIRST_FIT_DECREASING
                      • FIRST_FIT_SHUFFLE
                      • MODIFIED_FIRST_FIT_DECREASING
                    • SequencePacker
                      • _pack_implementation()
                      • _adjust_bin_count()
                      • pack()
                      • reset_metrics()
                      • compute_metrics()
                      • get_aggregated_metrics()
                      • print_metrics()
                      • _validate_sequence_lengths()
                      • _create_indexed_lengths()
                      • _estimate_bins_needed()
                    • ConcatenativePacker
                      • max_sequences_per_bin
                      • _pack_implementation()
                    • FirstFitPacker
                      • _prepare_sequences()
                      • _pack_implementation()
                    • FirstFitDecreasingPacker
                      • _prepare_sequences()
                    • FirstFitShufflePacker
                      • _prepare_sequences()
                    • ModifiedFirstFitDecreasingPacker
                      • _classify_items()
                      • _pack_implementation()
                    • get_packer()
              • nemo_rl.data.packing.metrics
                • Module Contents
                  • Classes
                  • API
                    • PackingMetrics
                      • reset()
                      • update()
                      • calculate_stats_only()
                      • get_aggregated_stats()
                      • print_aggregated_stats()
            • Package Contents
              • Data
              • API
                • __all__
          • nemo_rl.data.datasets
            • Subpackages
              • nemo_rl.data.datasets.preference_datasets
                • Submodules
                  • nemo_rl.data.datasets.preference_datasets.tulu3
                    • Module Contents
                      • Classes
                      • API
                        • Tulu3PreferenceDataset
                          • format_data()
                  • nemo_rl.data.datasets.preference_datasets.binary_preference_dataset
                    • Module Contents
                      • Classes
                      • API
                        • BinaryPreferenceDataset
                          • format_data()
                  • nemo_rl.data.datasets.preference_datasets.helpsteer3
                    • Module Contents
                      • Classes
                      • API
                        • HelpSteer3Dataset
                          • format_data()
                  • nemo_rl.data.datasets.preference_datasets.preference_dataset
                    • Module Contents
                      • Classes
                      • API
                        • PreferenceDataset
                • Package Contents
                  • Functions
                  • Data
                  • API
                    • DATASET_REGISTRY
                    • load_preference_dataset()
                    • __all__
              • nemo_rl.data.datasets.response_datasets
                • Submodules
                  • nemo_rl.data.datasets.response_datasets.audiomcq
                    • Module Contents
                      • Classes
                      • Functions
                      • Data
                      • API
                        • DEFAULT_TEMPLATE
                        • AUDIOMCQ_REPO_ID
                        • AUDIOMCQ_MANIFEST
                        • TARGET_SAMPLE_RATE
                        • STRONG_AC_VALUE
                        • _resample_audio()
                        • _resolve_snapshot_root()
                        • AudioMCQDataset
                          • task_name
                          • _eager_audio_probe()
                          • format_data()
                  • nemo_rl.data.datasets.response_datasets.response_dataset
                    • Module Contents
                      • Classes
                      • API
                        • ResponseDataset
                          • format_data()
                  • nemo_rl.data.datasets.response_datasets.clevr
                    • Module Contents
                      • Classes
                      • Functions
                      • API
                        • format_answer_fromtags()
                        • format_clevr_cogent_dataset()
                        • CLEVRCoGenTDataset
                  • nemo_rl.data.datasets.response_datasets.nemogym_dataset
                    • Module Contents
                      • Classes
                      • API
                        • NemoGymDataset
                  • nemo_rl.data.datasets.response_datasets.squad
                    • Module Contents
                      • Classes
                      • API
                        • SquadDataset
                          • format_data()
                  • nemo_rl.data.datasets.response_datasets.openmathinstruct2
                    • Module Contents
                      • Classes
                      • API
                        • OpenMathInstruct2Dataset
                          • format_data()
                  • nemo_rl.data.datasets.response_datasets.tulu3
                    • Module Contents
                      • Classes
                      • API
                        • Tulu3SftMixtureDataset
                          • format_data()
                  • nemo_rl.data.datasets.response_datasets.refcoco
                    • Module Contents
                      • Classes
                      • Functions
                      • API
                        • download_and_unzip()
                        • format_refcoco_dataset()
                        • RefCOCODataset
                          • format_data()
                  • nemo_rl.data.datasets.response_datasets.aime
                    • Module Contents
                      • Classes
                      • Data
                      • API
                        • AIMEVariant
                        • AIMEDataset
                          • format_data()
                  • nemo_rl.data.datasets.response_datasets.gsm8k
                    • Module Contents
                      • Classes
                      • Functions
                      • API
                        • _extract_hash_answer()
                        • GSM8KDataset
                          • format_data()
                  • nemo_rl.data.datasets.response_datasets.general_conversations_dataset
                    • Module Contents
                      • Classes
                      • Functions
                      • Data
                      • API
                        • conversation_sender_mapping_sample_to_allowed
                        • convert_metadata()
                        • conversation_process_message()
                        • GeneralConversationsJsonlDataset
                          • task_name
                          • process_message_fragment()
                          • _datum_preprocessor()
                  • nemo_rl.data.datasets.response_datasets.mmpr_tiny
                    • Module Contents
                      • Classes
                      • Functions
                      • Data
                      • API
                        • _IMAGE_PLACEHOLDER_RE
                        • format_mmpr_tiny_dataset()
                        • _ensure_mmpr_cached()
                        • _load_mmpr_tiny_from_cache()
                        • MMPRTinyDataset
                  • nemo_rl.data.datasets.response_datasets.helpsteer3
                    • Module Contents
                      • Classes
                      • API
                        • HelpSteer3Dataset
                          • format_data()
                  • nemo_rl.data.datasets.response_datasets.arrow_text_dataset
                    • Module Contents
                      • Classes
                      • Functions
                      • API
                        • ArrowTextDataset
                          • format_data()
                        • _pack_generator()
                  • nemo_rl.data.datasets.response_datasets.oai_format_dataset
                    • Module Contents
                      • Classes
                      • API
                        • PreservingDataset
                          • __len__()
                          • __getitem__()
                          • __iter__()
                          • map()
                        • OpenAIFormatDataset
                          • format_data()
                  • nemo_rl.data.datasets.response_datasets.daily_omni
                    • Module Contents
                      • Classes
                      • API
                        • DailyOmniDataset
                          • task_name
                          • get_prompt()
                          • format_data()
                  • nemo_rl.data.datasets.response_datasets.intent
                    • Module Contents
                      • Classes
                      • Functions
                      • Data
                      • API
                        • logger
                        • _TYPE_TEMPLATE
                        • _format_options()
                        • _SPLIT_CONFIG
                        • _EXTRACTION_SENTINEL
                        • _extract_videos_zip_once()
                        • _resolve_video_path()
                        • _read_manifest()
                        • IntentDataset
                          • _download_and_extract()
                          • _load_records()
                          • _filter_records()
                          • format_data()
                        • IntentTrainDataset
                        • IntentBenchDataset
                  • nemo_rl.data.datasets.response_datasets.avqa
                    • Module Contents
                      • Classes
                      • Functions
                      • Data
                      • API
                        • DEFAULT_TEMPLATE
                        • _resample_audio()
                        • _parse_question()
                        • AVQADataset
                          • task_name
                          • format_data()
                  • nemo_rl.data.datasets.response_datasets.deepscaler
                    • Module Contents
                      • Classes
                      • API
                        • DeepScalerDataset
                          • format_data()
                  • nemo_rl.data.datasets.response_datasets.dapo_math
                    • Module Contents
                      • Classes
                      • API
                        • DAPOMath17KDataset
                          • format_data()
                        • DAPOMathAIME2024Dataset
                  • nemo_rl.data.datasets.response_datasets.oasst
                    • Module Contents
                      • Classes
                      • Functions
                      • Data
                      • API
                        • SYSTEM_PROMPT
                        • parse_conversations()
                        • get_data_records()
                        • OasstDataset
                  • nemo_rl.data.datasets.response_datasets.geometry3k
                    • Module Contents
                      • Classes
                      • Functions
                      • API
                        • format_geometry3k_dataset()
                        • Geometry3KDataset
                  • nemo_rl.data.datasets.response_datasets.nemotron_cascade2_sft
                    • Module Contents
                      • Classes
                      • API
                        • NemotronCascade2SFTMathDataset
                          • format_data()
                • Package Contents
                  • Functions
                  • Data
                  • API
                    • DATASET_REGISTRY
                    • load_response_dataset()
                    • __all__
              • nemo_rl.data.datasets.eval_datasets
                • Submodules
                  • nemo_rl.data.datasets.eval_datasets.gpqa
                    • Module Contents
                      • Classes
                      • API
                        • GPQADataset
                          • _rekey()
                  • nemo_rl.data.datasets.eval_datasets.mmau
                    • Module Contents
                      • Classes
                      • Data
                      • API
                        • DEFAULT_TEMPLATE
                        • MMAUDataset
                          • format_data()
                  • nemo_rl.data.datasets.eval_datasets.daily_omni
                    • Module Contents
                      • Classes
                      • Data
                      • API
                        • _SINGLE_LETTER_LINE
                        • DailyOmniEvalDataset
                          • _format_for_eval()
                  • nemo_rl.data.datasets.eval_datasets.mmlu_pro
                    • Module Contents
                      • Classes
                      • API
                        • MMLUProDataset
                          • _rekey()
                  • nemo_rl.data.datasets.eval_datasets.local_math_dataset
                    • Module Contents
                      • Classes
                      • API
                        • LocalMathDataset
                          • _rekey()
                  • nemo_rl.data.datasets.eval_datasets.math
                    • Module Contents
                      • Classes
                      • API
                        • MathDataset
                          • _rekey()
                  • nemo_rl.data.datasets.eval_datasets.mmlu
                    • Module Contents
                      • Classes
                      • API
                        • MMLUDataset
                          • _rekey()
                • Package Contents
                  • Functions
                  • Data
                  • API
                    • MULTIMODAL_DATASETS
                    • _is_multimodal_dataset()
                    • load_eval_dataset()
                    • __all__
            • Submodules
              • nemo_rl.data.datasets.utils
                • Module Contents
                  • Functions
                  • Data
                  • API
                    • TokenizerType
                    • load_audio_from_file()
                    • assert_no_double_bos()
                    • pil_to_base64()
                    • load_dataset_from_path()
                    • resolve_external_dataset_class()
                    • update_single_dataset_config()
                    • merge_datasets()
                    • extract_necessary_env_names()
                    • get_huggingface_cache_path()
              • nemo_rl.data.datasets.processed_dataset
                • Module Contents
                  • Classes
                  • Data
                  • API
                    • TokenizerType
                    • AllTaskProcessedDataset
                      • __len__()
                      • encode_single()
                      • __getitem__()
              • nemo_rl.data.datasets.raw_dataset
                • Module Contents
                  • Classes
                  • API
                    • RawDataset
                      • data_config
                      • dataset
                      • val_dataset
                      • processor
                      • task_spec
                      • preprocessor
                      • split_train_validation()
                      • set_processor()
                      • set_task_spec()
            • Package Contents
              • Data
              • API
                • __all__
        • Submodules
          • nemo_rl.data.collate_fn
            • Module Contents
              • Functions
              • Data
              • API
                • TokenizerType
                • rl_collate_fn()
                • eval_collate_fn()
                • preference_collate_fn()
          • nemo_rl.data.cross_tokenizer_collate
            • Module Contents
              • Classes
              • API
                • CrossTokenizerCollator
                  • __call__()
                  • _tokenize_batch()
          • nemo_rl.data.interfaces
            • Module Contents
              • Classes
              • Data
              • API
                • LLMMessageLogType
                • VLMMessageLogType
                • FlatMessagesType
                • PathLike
                • TokenizerType
                • DatumSpec
                  • message_log
                  • length
                  • extra_env_info
                  • loss_multiplier
                  • idx
                  • task_name
                  • stop_strings
                  • __extra__
                • PreferenceDatumSpec
                  • message_log_chosen
                  • message_log_rejected
                  • length_chosen
                  • length_rejected
                  • loss_multiplier
                  • idx
                • TaskDataSpec
                  • task_name
                  • prompt_file
                  • system_prompt_file
                  • __post_init__()
                  • copy_defaults()
                • TaskDataProcessFnCallable
                  • __call__()
                • TaskDataPreProcessFnCallable
                  • __call__()
          • nemo_rl.data.utils
            • Module Contents
              • Functions
              • API
                • get_train_dataset_name()
                • load_dataloader_state()
                • setup_response_data()
                • setup_preference_data()
          • nemo_rl.data.dataloader
            • Module Contents
              • Classes
              • API
                • MultipleDataloaderWrapper
                  • _load_custom_dataloader_func()
                  • __iter__()
                  • __next__()
                  • set_records()
          • nemo_rl.data.llm_message_utils
            • Module Contents
              • Functions
              • Data
              • API
                • Tensor
                • TokenizerType
                • message_log_to_flat_messages()
                • get_keys_from_message_log()
                • add_loss_mask_to_message_log()
                • _pad_tensor()
                • _validate_tensor_consistency()
                • batched_message_log_to_flat_message()
                • message_log_shape()
                • get_first_index_that_differs()
                • get_formatted_message_log()
                • remap_dataset_keys()
                • MESSAGE_LOG_BULK_FIELDS
                • decompose_message_log()
                • attach_message_log_view()
                • reconstruct_message_log()
          • nemo_rl.data.chat_templates
            • Module Contents
              • Classes
              • API
                • COMMON_CHAT_TEMPLATES
                  • simple_role_header
                  • passthrough_prompt_response
          • nemo_rl.data.processors
            • Module Contents
              • Functions
              • Data
              • API
                • TokenizerType
                • helpsteer3_data_processor()
                • sft_processor()
                • preference_preprocessor()
                • math_data_processor()
                • math_hf_data_processor()
                • vlm_hf_data_processor()
                • _construct_multichoice_prompt()
                • multichoice_qa_processor()
                • nemo_gym_data_processor()
                • kd_data_processor()
                • PROCESSOR_REGISTRY
                • register_processor()
          • nemo_rl.data.multimodal_utils
            • Module Contents
              • Classes
              • Functions
              • Data
              • API
                • MEDIA_TAGS
                • MEDIA_TAGS_REVERSED
                • DEFAULT_MEDIA_EXTENSIONS
                • MEDIA_TAGS_TO_ALLOWED
                • MEDIA_TAG_PATTERN
                • logger
                • PackedTensor
                  • as_tensor()
                  • __len__()
                  • to()
                  • slice()
                  • empty_like()
                  • concat()
                  • flattened_concat()
                • get_multimodal_keys_from_processor()
                • get_multimodal_default_settings_from_processor()
                • get_dim_to_pack_along()
                • resolve_to_image()
                • get_media_from_message()
                • load_media_from_message()
        • Package Contents
          • Classes
          • Data
          • API
            • ResponseDatasetConfig
              • dataset_name
              • data_path
              • input_key
              • output_key
              • subset
              • split
              • prompt_file
              • system_prompt_file
              • env_name
              • processor
              • download_dir
              • split_validation_size
              • seed
            • PreferenceDatasetConfig
              • dataset_name
              • data_path
              • prompt_key
              • chosen_key
              • rejected_key
              • subset
              • split
              • prompt_file
              • system_prompt_file
            • DataConfig
              • max_input_seq_length
              • add_bos
              • add_eos
              • add_generation_prompt
              • add_system_prompt
              • shuffle
              • num_workers
              • use_multiple_dataloader
              • num_prompts_per_dataloader
              • custom_dataloader
              • train
              • validation
              • default
            • MMLUEvalDataConfig
              • max_input_seq_length
              • dataset_name
              • prompt_file
              • system_prompt_file
            • MMLUProEvalDataConfig
              • max_input_seq_length
              • dataset_name
              • prompt_file
              • system_prompt_file
            • AIMEEvalDataConfig
              • max_input_seq_length
              • dataset_name
              • prompt_file
              • system_prompt_file
              • processor
              • repeat
            • GPQAEvalDataConfig
              • max_input_seq_length
              • dataset_name
              • prompt_file
              • system_prompt_file
            • MathEvalDataConfig
              • max_input_seq_length
              • dataset_name
              • prompt_file
              • system_prompt_file
            • LocalMathEvalDataConfig
              • max_input_seq_length
              • dataset_name
              • problem_key
              • solution_key
              • file_format
              • split
              • prompt_file
              • system_prompt_file
            • MMAUEvalDataConfig
              • max_input_seq_length
              • dataset_name
              • split
              • prompt_file
              • system_prompt_file
              • env_name
            • DailyOmniEvalDataConfig
              • max_input_seq_length
              • dataset_name
              • split
              • prompt_file
              • system_prompt_file
              • env_name
            • EvalDataConfigType
      • nemo_rl.weight_sync
        • Submodules
          • nemo_rl.weight_sync.factory
            • Module Contents
              • Functions
              • API
                • create_weight_synchronizer()
          • nemo_rl.weight_sync.interfaces
            • Module Contents
              • Classes
              • API
                • WeightSynchronizer
                  • sync_weights()
                  • is_stale
                  • mark_stale()
                  • init_communicator()
                  • shutdown()
          • nemo_rl.weight_sync.collective_weight_synchronizer
            • Module Contents
              • Classes
              • API
                • CollectiveWeightSynchronizer
                  • sync_weights()
                  • is_stale
                  • mark_stale()
                  • init_communicator()
                  • shutdown()
          • nemo_rl.weight_sync.http_weight_synchronizer
            • Module Contents
              • Classes
              • API
                • HTTPWeightSynchronizer
                  • sync_weights()
                  • is_stale
                  • mark_stale()
                  • init_communicator()
                  • shutdown()
                  • _compute_buffer_size()
          • nemo_rl.weight_sync.ipc_weight_synchronizer
            • Module Contents
              • Classes
              • API
                • IPCWeightSynchronizer
                  • sync_weights()
                  • is_stale
                  • mark_stale()
                  • init_communicator()
                  • shutdown()
                  • _compute_buffer_size()
        • Package Contents
          • Data
          • API
            • __all__
      • nemo_rl.evals
        • Submodules
          • nemo_rl.evals.answer_parsing
            • Module Contents
              • Functions
              • Data
              • API
                • MULTILINGUAL_ANSWER_PATTERN_TEMPLATE
                • MULTILINGUAL_ANSWER_REGEXES
                • normalize_extracted_answer()
                • normalize_response()
          • nemo_rl.evals.eval
            • Module Contents
              • Classes
              • Functions
              • API
                • EvalConfig
                  • metric
                  • num_tests_per_prompt
                  • seed
                  • k_value
                  • save_path
                • _PassThroughEnvConfig
                  • math
                  • mmau
                • MasterConfig
                  • eval
                  • generation
                  • tokenizer
                  • data
                  • env
                  • cluster
                • setup()
                • eval_pass_k()
                • eval_cons_k()
                • run_env_eval()
                • _run_env_eval_impl()
                • _generate_texts()
                • _save_evaluation_data_to_json()
                • _print_results()
      • nemo_rl.algorithms
        • Subpackages
          • nemo_rl.algorithms.x_token
            • Submodules
              • nemo_rl.algorithms.x_token.utils
                • Module Contents
                  • Functions
                  • API
                    • assert_teacher_student_batch_grid()
                    • assert_xtoken_ipc_node_local()
                    • pad_distillation_val_batch()
              • nemo_rl.algorithms.x_token.loss_utils
                • Module Contents
                  • Classes
                  • Functions
                  • Data
                  • API
                    • alignment_from_flat_batch()
                    • Fp32SparseMM
                      • forward()
                      • backward()
                    • chunk_log_prob_sums()
                    • chunk_average_finalize()
                    • chunk_average_log_probs()
                    • slice_sparse_projection_rows()
                    • project_student_to_teacher_vocab()
                    • select_teacher_topk_indices()
                    • LocalizedAlignment
                      • sample_mask
                      • student_chunk_id
                      • teacher_chunk_id
                      • pair_valid
                      • pair_is_correct
                      • student_input_ids
                      • student_token_mask
                    • localize_alignment()
                    • student_next_token_ce()
                    • ce_label_mask()
                    • next_token_accuracy()
                    • collect_overlapping_teacher_shards()
                    • assemble_teacher_logits_from_shards()
                    • _try_zero_copy_teacher_logits()
                    • rebuild_teacher_full_logits_from_ipc()
                    • valid_chunk_mask()
                    • parse_projection_file()
                    • _SPARSE_PROJECTION_CACHE
                    • _TOPK_PROJECTION_CACHE
                    • get_sparse_projection_matrix()
                    • get_topk_projection()
                    • _EXACT_TOKEN_MAP_CACHE
                    • build_exact_token_map()
                    • prepare_xtoken_cross_tokenizer_loss_input()
              • nemo_rl.algorithms.x_token.token_aligner
                • Module Contents
                  • Classes
                  • Functions
                  • Data
                  • API
                    • VISUAL_BYTE_MAP
                    • _MULTI_TOKEN_ARTIFACT_FIXES
                    • _UNICODE_FIXES
                    • _SPECIAL_TOKEN_MAP
                    • AlignmentPair
                      • s_tokens
                      • t_tokens
                      • s_start
                      • s_end
                      • t_start
                      • t_end
                      • is_correct
                    • AlignmentBatch
                      • pair_valid
                      • pair_is_correct
                      • student_exact_partition_mask
                      • teacher_exact_partition_mask
                      • student_chunk_id
                      • teacher_chunk_id
                      • num_chunks
                    • TokenAligner
                      • align()
                      • _drop_padding()
                      • _pairs_to_batch()
                      • _align_single()
                      • _remap_pairs_to_original()
                      • _align_with_anchors()
                      • _align_dp()
                      • _shift_pairs()
                      • _alignment_mask()
                      • _post_process_alignment()
                      • _build_pair_strings()
                      • _combine_consecutive_misaligned()
                      • _flatten_chunk()
                    • canonical_token()
                    • _canonicalize_sequence()
                    • _merge_encoding_artifacts()
                    • _get_byte_value()
                    • _merge_consecutive_bytes()
                    • _try_merge_byte_buffer()
                    • _strings_equal_flexible()
            • Package Contents
              • Data
              • API
                • __all__
          • nemo_rl.algorithms.async_utils
            • Submodules
              • nemo_rl.algorithms.async_utils.interfaces
                • Module Contents
                  • Classes
                  • API
                    • ReplayBufferProtocol
                      • add()
                      • sample()
                      • size()
                      • clear()
                      • state_dict()
                      • load_state_dict()
                      • get_trajectories_needed()
                      • has_complete_batch()
              • nemo_rl.algorithms.async_utils.replay_buffer
                • Module Contents
                  • Classes
                  • API
                    • ReplayBufferImpl
                      • _rollout_metrics_turn_count_for_diagnostics()
                      • add()
                      • get_debug_info()
                      • get_last_target_weight_already_generated()
                      • get_existing_target_weights()
                      • _remove_indices()
                      • sample()
                      • size()
                      • clear()
                      • state_dict()
                      • load_state_dict()
                      • _prepare_for_training_step()
                      • _is_valid_for_target()
                      • _remove_stale_trajectories()
                      • _count_for_target()
                      • _truncate_to_max_size()
                      • get_trajectories_needed()
                      • has_complete_batch()
                      • _remove_incomplete_target_steps()
                    • ReplayBuffer
                    • ReplayBufferNew
                      • _evict()
                      • sample()
              • nemo_rl.algorithms.async_utils.trajectory_collector
                • Module Contents
                  • Classes
                  • Data
                  • API
                    • TokenizerType
                    • AsyncTrajectoryCollector
                      • _calculate_target_weights()
                      • _get_next_target_for_generation()
                      • set_weight_version()
                      • _should_pause_for_generation_limits()
                      • start_collection()
                      • is_data_exhausted()
                      • get_status()
                      • _collection_loop()
                      • _process_batch()
                      • get_weight_version()
                      • pause()
                      • resume()
                      • prepare_for_refit()
                      • resume_after_refit()
                      • wait_for_pending_generations()
                      • get_dataloader_state()
                      • get_efficiency_metrics()
                      • _cleanup_finished_threads()
                      • _maybe_release_target()
                      • _compute_teacher_logprobs()
                      • _run_prompt_group_worker()
            • Package Contents
              • Data
              • API
                • __all__
          • nemo_rl.algorithms.loss
            • Submodules
              • nemo_rl.algorithms.loss.interfaces
                • Module Contents
                  • Classes
                  • API
                    • LossType
                      • TOKEN_LEVEL
                      • SEQUENCE_LEVEL
                    • MetricNormalizer
                      • TOKENS
                      • SEQUENCES
                      • NONE
                    • LossInputType
                      • LOGIT
                      • LOGPROB
                      • DISTILLATION
                      • DISTILLATION_CROSS_TOKENIZER
                      • DRAFT
                    • LossFunction
                      • loss_type
                      • input_type
                      • __call__()
              • nemo_rl.algorithms.loss.utils
                • Module Contents
                  • Functions
                  • API
                    • prepare_loss_input()
                    • _pack_input_ids()
                    • prepare_packed_loss_input()
              • nemo_rl.algorithms.loss.loss_functions
                • Module Contents
                  • Classes
                  • Data
                  • API
                    • Tensor
                    • DraftCrossEntropyLossConfig
                      • vocab_parallel_group
                    • DraftCrossEntropyLossDataDict
                      • teacher_logits
                      • student_logits
                      • token_mask
                      • sample_mask
                      • student_vocab_indices
                    • DraftCrossEntropyLossFn
                      • loss_type
                      • input_type
                      • __call__()
                    • ClippedPGLossConfig
                      • disable_ppo_ratio
                      • token_level_loss
                      • sequence_level_importance_ratios
                      • ratio_clip_min
                      • ratio_clip_max
                      • ratio_clip_c
                      • reference_policy_kl_penalty
                      • reference_policy_kl_type
                      • kl_input_clamp_value
                      • kl_output_clamp_value
                      • use_kl_in_reward
                      • use_importance_sampling_correction
                      • truncated_importance_sampling_type
                      • truncated_importance_sampling_ratio
                      • truncated_importance_sampling_ratio_min
                      • use_on_policy_kl_approximation
                      • force_on_policy_ratio
                      • use_cispo
                      • positive_example_nll_weight
                    • ClippedPGLossDataDict
                      • input_ids
                      • advantages
                      • prev_logprobs
                      • generation_logprobs
                      • reference_policy_logprobs
                      • token_mask
                      • sample_mask
                      • __extra__
                    • ClippedPGLossFn
                      • input_type
                      • __call__()
                    • NLLLossFn
                      • loss_type
                      • input_type
                      • __call__()
                    • PreferenceLossDataDict
                      • input_ids
                      • token_mask
                      • sample_mask
                    • PreferenceLossFn
                      • loss_type
                      • input_type
                      • split_output_tensor()
                      • _preference_loss()
                      • __call__()
                    • DPOLossConfig
                      • reference_policy_kl_penalty
                      • preference_loss_weight
                      • sft_loss_weight
                      • preference_average_log_probs
                      • sft_average_log_probs
                    • DPOLossDataDict
                      • input_ids
                      • reference_policy_logprobs
                      • token_mask
                      • sample_mask
                    • DPOLossFn
                      • loss_type
                      • input_type
                      • _dpo_loss()
                      • __call__()
                    • DistillationLossConfig
                      • kl_type
                      • mixed_kl_weight
                      • zero_outside_topk
                    • DistillationLossDataDict
                      • input_ids
                      • input_lengths
                      • token_mask
                      • sample_mask
                      • teacher_topk_logits
                      • teacher_topk_indices
                    • DistillationLossFn
                      • loss_type
                      • input_type
                      • __call__()
                    • MseValueLossConfig
                      • scale
                      • cliprange
                    • MseValueLossFn
                      • input_type
                      • __call__()
                    • CrossTokenizerDistillationLossConfig
                      • gold_loss
                      • xtoken_loss
                      • temperature
                      • vocab_topk
                      • uncommon_topk
                      • reverse_kl
                      • exact_token_match_only
                      • kl_loss_weight
                      • ce_loss_scale
                      • dynamic_loss_scaling
                      • kd_loss_mode
                      • normalize_teacher_by_vocab
                      • alpha
                      • sum_weights_metric
                      • student_vocab_size
                      • teacher_vocab_sizes
                      • projection_matrix_paths
                      • teacher_weights
                      • teacher_gold_loss
                      • teacher_xtoken_loss
                    • CrossTokenizerDistillationLossDataDict
                      • input_ids
                      • input_lengths
                      • token_mask
                      • sample_mask
                    • CrossTokenizerDistillationLossFn
                      • loss_type
                      • input_type
                      • _teacher_is_same_vocab()
                      • __call__()
                      • _resolve_gold_xtoken()
                      • _compute_teacher_kd()
                      • _compute_same_vocab_kl()
                      • _direct_topk_kl()
                      • _direct_full_vocab_kl()
                      • _same_vocab_masked_kl()
                      • _sum_kd()
                      • _averaged_logits_kd()
                      • _dp_global_masked_mean()
                      • _select_teacher_kd()
                      • _teacher_score_inputs()
                      • _compute_dynamic_weights()
                      • _teacher_weight_score()
                      • _compute_p_kl()
                      • _compute_gold()
                      • _compute_ce()
              • nemo_rl.algorithms.loss.wrapper
                • Module Contents
                  • Classes
                  • Functions
                  • Data
                  • API
                    • Tensor
                    • SequencePackingLossWrapper
                      • __call__()
                    • SequencePackingFusionLossWrapper
                      • __call__()
                    • DraftLossWrapper
                      • __call__()
                    • wrap_loss_fn_with_input_preparation()
            • Package Contents
              • Data
              • API
                • __all__
        • Submodules
          • nemo_rl.algorithms.dpo
            • Module Contents
              • Classes
              • Functions
              • API
                • DPOSaveState
                  • epoch
                  • step
                  • total_steps
                  • consumed_samples
                  • total_valid_tokens
                • _initial_dpo_save_state()
                • DPOConfig
                  • max_num_epochs
                  • max_num_steps
                  • val_period
                  • val_batches
                  • val_global_batch_size
                  • val_micro_batch_size
                  • val_at_start
                  • val_at_end
                  • seed
                  • reference_policy_kl_penalty
                  • preference_average_log_probs
                  • sft_average_log_probs
                  • preference_loss_weight
                  • sft_loss_weight
                • MasterConfig
                  • policy
                  • data
                  • dpo
                  • logger
                  • cluster
                  • checkpointing
                • DPOValMetrics
                  • loss
                  • sft_loss
                  • preference_loss
                  • accuracy
                  • rewards_chosen_mean
                  • rewards_rejected_mean
                  • num_valid_samples
                  • global_valid_seqs
                  • global_valid_toks
                • setup()
                • add_ref_logprobs_to_data()
                • validate()
                • validate_one_dataset()
                • dpo_train()
          • nemo_rl.algorithms.distillation
            • Module Contents
              • Classes
              • Functions
              • Data
              • API
                • TokenizerType
                • DistillationConfig
                  • num_prompts_per_step
                  • num_generations_per_prompt
                  • max_rollout_turns
                  • max_num_steps
                  • max_num_epochs
                  • val_batch_size
                  • val_period
                  • val_at_start
                  • val_at_end
                  • max_val_samples
                  • topk_logits_k
                  • seed
                • DistillationSaveState
                  • total_steps
                  • current_epoch
                  • current_step
                  • val_reward
                  • consumed_samples
                  • total_valid_tokens
                • _default_distillation_save_state()
                • MasterConfig
                  • policy
                  • teacher
                  • loss_fn
                  • env
                  • data
                  • distillation
                  • logger
                  • cluster
                  • checkpointing
                • check_vocab_equality()
                • setup()
                • distillation_train()
                • validate()
          • nemo_rl.algorithms.advantage_estimator
            • Module Contents
              • Classes
              • API
                • GRPOAdvantageEstimator
                  • compute_advantage()
                • GDPOAdvantageEstimator
                  • compute_advantage()
                • ReinforcePlusPlusAdvantageEstimator
                  • compute_advantage()
                • RawRewardAdvantageEstimator
                  • compute_advantage()
                • GeneralizedAdvantageEstimator
                  • _reward_whiten()
                  • _build_token_level_rewards()
                  • _resolve_lambda_policy()
                  • _resolve_lambda_value()
                  • compute_advantage()
                  • _compute_gae()
                • OPDAdvantageEstimator
                  • compute_advantage()
                  • _compute_metrics()
          • nemo_rl.algorithms.opd
            • Module Contents
              • Classes
              • Functions
              • API
                • TeacherResourceConfig
                  • tensor_model_parallel_size
                  • pipeline_model_parallel_size
                  • context_parallel_size
                  • expert_model_parallel_size
                  • num_nodes
                  • gpus_per_node
                  • precision
                  • micro_batch_size
                  • megatron_cfg_overrides
                • NonColocatedTeachersConfig
                  • enabled
                  • default_teacher_cfg
                  • teacher_overrides
                • OnPolicyDistillationConfig
                  • enabled
                  • teacher_model_by_agent_name
                  • default_teacher_alias
                  • strict_agent_name_match
                  • deduplicate_shared_teacher_checkpoints
                  • non_colocated_teachers
                • _opd_cfg()
                • is_opd_enabled()
                • is_non_colocated_teachers_enabled()
                • _skip_prev_logprobs()
                • assert_prev_logprobs_available()
                • resolve_reference_aliases()
                • get_teacher_routing_metrics()
                • teacher_seq_pad_multiple()
                • create_teacher_worker_groups()
          • nemo_rl.algorithms.reward_functions
            • Module Contents
              • Classes
              • Functions
              • Data
              • API
                • Tensor
                • RewardShapingConfig
                  • enabled
                  • overlong_buffer_length
                  • overlong_buffer_penalty
                  • max_response_length
                  • stop_properly_penalty_coef
                • apply_reward_shaping()
          • nemo_rl.algorithms.grpo
            • Module Contents
              • Classes
              • Functions
              • Data
              • API
                • TokenizerType
                • RewardScalingConfig
                  • enabled
                  • source_min
                  • source_max
                  • target_min
                  • target_max
                • AsyncGRPOConfig
                  • enabled
                  • max_trajectory_age_steps
                  • in_flight_weight_updates
                  • recompute_kv_cache_after_weight_updates
                • AdvEstimatorConfig
                  • name
                  • normalize_rewards
                  • use_leave_one_out_baseline
                  • minus_baseline
                • RewardPenaltyTokenIdsConfig
                  • unwanted
                  • think_open
                  • think_close
                • RewardPenaltyConfig
                  • penalize_duplicated_reasoning
                  • penalize_empty_final_answer
                  • penalize_unwanted_tokens
                  • penalize_malformed_think_tag
                  • token_ids
                  • _require_unwanted_token_ids_when_penalized()
                • _REWARD_PENALTY_FLAGS
                • GRPOConfig
                  • num_prompts_per_step
                  • num_generations_per_prompt
                  • max_num_epochs
                  • max_num_steps
                  • max_rollout_turns
                  • normalize_rewards
                  • advantage_clip_low
                  • advantage_clip_high
                  • use_leave_one_out_baseline
                  • val_period
                  • val_batch_size
                  • val_at_start
                  • val_at_end
                  • max_val_samples
                  • skip_reference_policy_logprobs_calculation
                  • seed
                  • async_grpo
                  • overlong_filtering
                  • use_dynamic_sampling
                  • dynamic_sampling_max_gen_batches
                  • batch_multiplier
                  • reward_shaping
                  • reward_scaling
                  • calculate_advantages_on_gpu
                  • seq_logprob_error_threshold
                  • invalid_tool_call_advantage
                  • malformed_thinking_advantage
                  • adv_estimator
                • GRPOSaveState
                  • consumed_samples
                  • current_step
                  • current_epoch
                  • total_steps
                  • total_valid_tokens
                  • val_reward
                • _default_grpo_save_state()
                • GRPOLoggerConfig
                  • num_val_samples_to_print
                • MasterConfig
                  • policy
                  • loss_fn
                  • env
                  • data
                  • grpo
                  • logger
                  • cluster
                  • checkpointing
                  • reward_penalties
                  • data_plane
                  • on_policy_distillation
                • setup()
                • dynamic_sampling()
                • scale_rewards()
                • extract_initial_prompt_messages()
                • add_grpo_token_loss_masks_and_generation_logprobs()
                • _resolve_message_level_advantage_penalties()
                • _raise_if_reward_penalties_enabled_without_nemo_gym()
                • _apply_message_level_advantage_penalties()
                • _apply_configured_message_level_advantage_penalties()
                • _should_use_async_rollouts()
                • _preserve_router_replay_routed_experts()
                • _build_async_grpo_train_data()
                • _apply_mask_sample_filter()
                • _should_use_nemo_gym()
                • _should_log_nemo_gym_responses()
                • _write_latest_checkpoint_status()
                • _get_effort_config()
                • _pad_teacher_logprobs()
                • _create_advantage_estimator()
                • _clip_grpo_advantages()
                • refit_policy_generation()
                • _log_mixed_rewards_and_advantages_information()
                • compute_and_apply_seq_logprob_error_masking()
                • grpo_train()
                • validate()
                • aggregate_rollout_metrics()
                • async_grpo_train()
          • nemo_rl.algorithms.utils
            • Module Contents
              • Functions
              • Data
              • API
                • get_gdpo_reward_component_keys()
                • calculate_kl()
                • calculate_baseline_and_std_per_prompt()
                • surpress_user_warnings()
                • masked_mean()
                • mask_out_neg_inf_logprobs()
                • masked_var()
                • set_seed()
                • get_tokenizer()
                • maybe_pad_last_batch()
                • print_performance_metrics()
                • log_generation_metrics_to_wandb()
                • WALL_CLOCK_EFFICIENCY_CATEGORIES
                • THREAD_ACCUMULATED_EFFICIENCY_CATEGORIES
                • EFFICIENCY_CATEGORIES
                • print_efficiency_summary()
          • nemo_rl.algorithms.grpo_sync
            • Module Contents
              • Functions
              • API
                • _raise_if_message_level_advantage_penalties_enabled()
                • _apply_dynamic_sampling()
                • validate_sync()
                • _compute_seq_logprob_error_metrics()
                • grpo_train_sync()
          • nemo_rl.algorithms.logits_sampling_utils
            • Module Contents
              • Classes
              • Functions
              • Data
              • API
                • TOP_K_TOP_P_CHUNK_SIZE
                • TrainingSamplingParams
                  • top_k
                  • top_p
                  • temperature
                • _need_top_k_filtering()
                • _need_top_p_filtering()
                • need_top_k_or_top_p_filtering()
                • _apply_top_k_only_fn()
                • _apply_top_k_top_p_fn()
                • _ApplyTopKTopP
                  • forward()
                  • backward()
                • apply_top_k_top_p()
          • nemo_rl.algorithms.rm
            • Module Contents
              • Classes
              • Functions
              • API
                • RMSaveState
                  • epoch
                  • step
                  • total_steps
                  • consumed_samples
                  • total_valid_tokens
                • _initial_rm_save_state()
                • RMConfig
                  • max_num_steps
                  • max_num_epochs
                  • val_period
                  • val_batches
                  • val_global_batch_size
                  • val_micro_batch_size
                  • val_at_start
                  • val_at_end
                  • seed
                • MasterConfig
                  • policy
                  • data
                  • rm
                  • logger
                  • cluster
                  • checkpointing
                • RMValMetrics
                  • loss
                  • accuracy
                  • rewards_chosen_mean
                  • rewards_rejected_mean
                  • num_valid_samples
                • setup()
                • validate()
                • validate_one_dataset()
                • rm_train()
          • nemo_rl.algorithms.ppo
            • Module Contents
              • Classes
              • Functions
              • Data
              • API
                • TokenizerType
                • AdvEstimatorConfig
                  • name
                  • gae_lambda
                  • gae_gamma
                  • normalize_advantages
                  • gae_lambda_value
                  • gae_lambda_policy
                  • length_adaptive_alpha
                • PPOConfig
                  • num_prompts_per_step
                  • num_generations_per_prompt
                  • max_num_epochs
                  • max_num_steps
                  • max_rollout_turns
                  • val_period
                  • val_batch_size
                  • val_at_start
                  • val_at_end
                  • max_val_samples
                  • skip_reference_policy_logprobs_calculation
                  • seed
                  • overlong_filtering
                  • use_dynamic_sampling
                  • dynamic_sampling_max_gen_batches
                  • batch_multiplier
                  • ppo_epochs
                  • reward_shaping
                  • reward_scaling
                  • calculate_advantages_on_gpu
                  • adv_estimator
                  • policy_training_start_step
                • PPOSaveState
                  • consumed_samples
                  • current_step
                  • current_epoch
                  • total_steps
                  • total_valid_tokens
                  • val_reward
                • _default_ppo_save_state()
                • PPOLoggerConfig
                  • num_val_samples_to_print
                • MasterConfig
                  • policy
                  • value
                  • loss_fn
                  • value_loss_fn
                  • env
                  • data
                  • ppo
                  • logger
                  • cluster
                  • checkpointing
                • setup()
                • dynamic_sampling()
                • _create_advantage_estimator()
                • ppo_train()
                • validate()
          • nemo_rl.algorithms.xtoken_off_policy_distillation
            • Module Contents
              • Classes
              • Functions
              • API
                • xtoken_non_student_seq_keys()
                • OffPolicyDistillationConfig
                  • num_prompts_per_step
                  • max_num_steps
                  • max_num_epochs
                  • seed
                  • val_period
                  • val_at_start
                  • val_at_end
                • OffPolicyDistillationSaveState
                  • current_epoch
                  • current_step
                  • total_steps
                  • consumed_samples
                  • total_valid_tokens
                  • val_loss
                • _default_off_policy_distillation_save_state()
                • TeacherConfig
                  • projection_matrix_path
                  • weight
                  • gold_loss
                  • xtoken_loss
                  • policy_config()
                • MasterConfig
                  • policy
                  • teachers
                  • loss_fn
                  • data
                  • distillation
                  • logger
                  • cluster
                  • checkpointing
                • setup()
                • export_teacher_logits_and_pack()
                • xtoken_off_policy_distillation_train()
                • validate()
          • nemo_rl.algorithms.sft
            • Module Contents
              • Classes
              • Functions
              • API
                • SFTSaveState
                  • epoch
                  • step
                  • total_steps
                  • consumed_samples
                  • total_valid_tokens
                • _initial_sft_save_state()
                • SFTConfig
                  • max_num_steps
                  • max_num_epochs
                  • val_period
                  • val_batches
                  • val_global_batch_size
                  • val_micro_batch_size
                  • val_at_start
                  • val_at_end
                  • seed
                  • only_unmask_final
                • MasterConfig
                  • policy
                  • data
                  • sft
                  • logger
                  • cluster
                  • checkpointing
                • setup()
                • validate()
                • sft_train()
    • Submodules
      • nemo_rl.package_info
        • Module Contents
          • Data
          • API
            • MAJOR
            • MINOR
            • PATCH
            • PRE_RELEASE
            • VERSION
            • __shortversion__
            • __version__
            • __package_name__
            • __contact_names__
            • __contact_emails__
            • __homepage__
            • __repository_url__
            • __download_url__
            • __description__
            • __license__
            • __keywords__
    • Package Contents
      • Functions
      • Data
      • API
        • megatron_path
        • _is_build_isolation()
        • _check_container_fingerprint()
        • _patch_nsight_file()
        • patch_transformers_module_dir()