Skip to main content
Ctrl+K
NeMo-RL - Home NeMo-RL - Home

NeMo-RL

  • GitHub
NeMo-RL - Home NeMo-RL - Home

NeMo-RL
  • GitHub

Table of Contents

About

  • Overview
  • Performance
  • Model Support
  • Features and Roadmap
  • Training and Generation Backends
  • Quick Start
  • Installation and Prerequisites
  • Algorithms
    • GRPO
    • DAPO
    • CISPO
    • PPO
    • On-policy Distillation
    • Multi-Teacher On-Policy Distillation (MOPD)
    • Supervised Fine-Tuning (SFT)
    • DPO
    • RM
  • Evaluation
  • Set Up Clusters
  • Tips and Tricks

Environment Start

  • Run on Your Local Workstation
  • Set Up Clusters

E2E Examples

  • SFT on OpenMathInstruct-2

Guides

  • Nemotron 3 Nano
  • Nemotron 3 Nano Omni
  • Nemotron 3 Super
  • Nemotron 3 Ultra
  • Add New Models
  • Supervised Fine-Tuning in NeMo RL
  • Direct Preference Optimization in NeMo RL
  • An in-depth Walkthrough of DAPO in NeMo RL
  • LoRA (Low-Rank Adaptation)
  • An in-depth Walkthrough of CISPO in NeMo RL
  • An In-Depth Walkthrough of ProRLv2 in NeMo RL
  • Two-Stage SWE RL for Qwen3-30B-A3B-Thinking
  • An In-depth Walkthrough of GRPO in NeMo RL
  • An In-Depth Walkthrough of PPO in NeMo RL
  • GRPO on DeepScaler
  • Solve a Sliding Puzzle Using GRPO
  • Audio Post-training with NeMo RL
  • Audio-Visual GRPO with Qwen2.5-Omni-7B
  • Reward Model Training in NeMo RL
  • Environments for GRPO Training
  • Evaluation
  • DeepSeek-V3
  • Model Guides
  • Qwen
  • Qwen3.5
  • Model Quirks
  • Train with Async GRPO
  • Quantization-Aware RL (QARL)
  • Train with Eagle3 Speculative Decoding
  • YaRN Long-Context Training
  • Cross-Tokenizer (X-Token) Off-Policy Distillation
  • Weight Refit: Choosing a Transport
  • Checkpoint-Engine Refit
  • Router Replay
  • Muon Optimizer
  • DTensor Tensor Parallel Accuracy Issue
  • Fault Tolerance Launcher Guide

Containers

  • Build Docker Images

Development

  • CI/CD
  • Test NeMo RL
  • Documentation Development
  • Debug NeMo RL Applications
  • Profile GPU with Nsys
  • FP8 Quantization in NeMo RL
  • Experiment with Custom vLLM

Design Docs

  • Design and Philosophy
  • Padding in NeMo RL
  • Logger
  • uv in NeMo RL
  • Dependency Management
  • Data Format
  • Generation Interface
  • Remote Sparse-Delta vLLM Refit
  • Checkpoint Engine Design
  • Exporting Checkpoints to Hugging Face Format
  • Loss functions in NeMo RL
  • FSDP2 Parallel Plan
  • Training Backends
  • Sequence Packing and Dynamic Batching
  • Environment Variable Precedence in NeMo RL
  • NeMo Gym Integration
  • ModelOpt Real-Quant Refit Architecture
  • NCCL Reshard Refit (Experimental)

API Reference

  • API Reference
    • nemo_rl
      • nemo_rl.models
        • nemo_rl.models.value
        • nemo_rl.models.automodel
        • nemo_rl.models.megatron
        • nemo_rl.models.huggingface
        • nemo_rl.models.policy
        • nemo_rl.models.generation
        • nemo_rl.models.dtensor
      • nemo_rl.utils
        • nemo_rl.utils.checkpoint_engines
        • nemo_rl.utils.nvml
        • nemo_rl.utils.timer
        • nemo_rl.utils.flops_formulas
        • nemo_rl.utils.r3_trace
        • nemo_rl.utils.weight_transfer_sparse_codec
        • nemo_rl.utils.checkpoint
        • nemo_rl.utils.config
        • nemo_rl.utils.nsys
        • nemo_rl.utils.memory_tracker
        • nemo_rl.utils.weight_transfer_zmq
        • nemo_rl.utils.venvs
        • nemo_rl.utils.native_checkpoint
        • nemo_rl.utils.prefetch_venvs
        • nemo_rl.utils.weight_transfer_stream
        • nemo_rl.utils.weight_transfer_http
        • nemo_rl.utils.logger
        • nemo_rl.utils.grad_norm
        • nemo_rl.utils.flops_tracker
        • nemo_rl.utils.fastokens
        • nemo_rl.utils.packed_tensor
        • nemo_rl.utils.routed_experts_codec
      • nemo_rl.weight_sync
        • nemo_rl.weight_sync.interfaces
        • nemo_rl.weight_sync.nccl_reshard_utils
        • nemo_rl.weight_sync.collective_weight_synchronizer
        • nemo_rl.weight_sync.nccl_reshard_weight_synchronizer
        • nemo_rl.weight_sync.http_weight_synchronizer
        • nemo_rl.weight_sync.xferdtensor_python
        • nemo_rl.weight_sync.xferdtensor
        • nemo_rl.weight_sync.factory
        • nemo_rl.weight_sync.checkpoint_engine_weight_synchronizer
        • nemo_rl.weight_sync.checkpoint_engine_config
        • nemo_rl.weight_sync.vllm_remote_sparse_weight_synchronizer
        • nemo_rl.weight_sync.ipc_weight_synchronizer
      • nemo_rl.data_plane
        • nemo_rl.data_plane.adapters
        • nemo_rl.data_plane.schema
        • nemo_rl.data_plane.interfaces
        • nemo_rl.data_plane.column_io
        • nemo_rl.data_plane.observability
        • nemo_rl.data_plane.preshard
        • nemo_rl.data_plane.factory
        • nemo_rl.data_plane.codec
        • nemo_rl.data_plane.worker_mixin
      • nemo_rl.environments
        • nemo_rl.environments.vlm_environment
        • nemo_rl.environments.interfaces
        • nemo_rl.environments.nemo_gym
        • nemo_rl.environments.metrics
        • nemo_rl.environments.reward_model_environment
        • nemo_rl.environments.code_environment
        • nemo_rl.environments.code_jaccard_environment
        • nemo_rl.environments.rewards
        • nemo_rl.environments.dapo_math_verifier
        • nemo_rl.environments.math_environment
        • nemo_rl.environments.utils
      • nemo_rl.modelopt
        • nemo_rl.modelopt.models
        • nemo_rl.modelopt.registry
        • nemo_rl.modelopt.utils
      • nemo_rl.algorithms
        • nemo_rl.algorithms.single_controller_utils
        • nemo_rl.algorithms.x_token
        • nemo_rl.algorithms.loss
        • nemo_rl.algorithms.async_utils
        • nemo_rl.algorithms.logits_sampling_utils
        • nemo_rl.algorithms.metric_utils
        • nemo_rl.algorithms.rm
        • nemo_rl.algorithms.ppo
        • nemo_rl.algorithms.opd
        • nemo_rl.algorithms.single_controller
        • nemo_rl.algorithms.sft
        • nemo_rl.algorithms.reward_functions
        • nemo_rl.algorithms.advantage_estimator
        • nemo_rl.algorithms.grpo
        • nemo_rl.algorithms.distillation
        • nemo_rl.algorithms.grpo_sync
        • nemo_rl.algorithms.xtoken_off_policy_distillation
        • nemo_rl.algorithms.dpo
        • nemo_rl.algorithms.utils
      • nemo_rl.distributed
        • nemo_rl.distributed.model_utils
        • nemo_rl.distributed.numa_utils
        • nemo_rl.distributed.virtual_cluster
        • nemo_rl.distributed.stateless_process_group
        • nemo_rl.distributed.collectives
        • nemo_rl.distributed.named_sharding
        • nemo_rl.distributed.worker_groups
        • nemo_rl.distributed.ray_actor_environment_registry
        • nemo_rl.distributed.worker_group_utils
        • nemo_rl.distributed.batched_data_dict
      • nemo_rl.evals
        • nemo_rl.evals.answer_parsing
        • nemo_rl.evals.eval
      • nemo_rl.experience
        • nemo_rl.experience.interfaces
        • nemo_rl.experience.metric_utils
        • nemo_rl.experience.rollouts
        • nemo_rl.experience.rollout_manager
        • nemo_rl.experience.sync_rollout_actor
        • nemo_rl.experience.payload
      • nemo_rl.data
        • nemo_rl.data.datasets
        • nemo_rl.data.packing
        • nemo_rl.data.interfaces
        • nemo_rl.data.chat_templates
        • nemo_rl.data.processors
        • nemo_rl.data.collate_fn
        • nemo_rl.data.cross_tokenizer_collate
        • nemo_rl.data.llm_message_utils
        • nemo_rl.data.dataloader
        • nemo_rl.data.multimodal_utils
        • nemo_rl.data.utils
      • nemo_rl.package_info
  • API Reference
  • nemo_rl
  • nemo_rl.modelopt
  • nemo_rl.modelopt.models
  • nemo_rl.modelopt.models.policy
  • nemo_rl.modelopt.models.policy.workers
  • nemo_rl.modelopt.models.policy.workers.dtensor_quant_policy_worker_v2

nemo_rl.modelopt.models.policy.workers.dtensor_quant_policy_worker_v2#

Module Contents#

Classes#

DTensorQuantPolicyWorkerV2

API#

class nemo_rl.modelopt.models.policy.workers.dtensor_quant_policy_worker_v2.DTensorQuantPolicyWorkerV2(*args, **kwargs)#

Bases: nemo_rl.models.policy.workers.dtensor_policy_worker_v2.DTensorPolicyWorkerV2Impl

previous

nemo_rl.modelopt.models.policy.workers.dtensor_quant_policy_worker

next

nemo_rl.modelopt.models.policy.workers.utils
On this page
  • Module Contents
    • Classes
    • API
      • DTensorQuantPolicyWorkerV2
NVIDIA NVIDIA
Privacy Policy | Your Privacy Choices | Terms of Service | Accessibility | Corporate Policies | Product Security | Contact

Copyright © 2026, NVIDIA Corporation.