Skip to main content
Ctrl+K
NeMo-RL - Home NeMo-RL - Home

NeMo-RL

  • GitHub
NeMo-RL - Home NeMo-RL - Home

NeMo-RL
  • GitHub

Table of Contents

About

  • Overview
  • Performance
  • Model Support
  • Features and Roadmap
  • Training and Generation Backends
  • Quick Start
  • Installation and Prerequisites
  • Algorithms
    • GRPO
    • DAPO
    • CISPO
    • PPO
    • On-policy Distillation
    • Multi-Teacher On-Policy Distillation (MOPD)
    • Supervised Fine-Tuning (SFT)
    • DPO
    • RM
  • Evaluation
  • Set Up Clusters
  • Tips and Tricks

Environment Start

  • Run on Your Local Workstation
  • Set Up Clusters

E2E Examples

  • SFT on OpenMathInstruct-2

Guides

  • Add New Models
  • Supervised Fine-Tuning in NeMo RL
  • Direct Preference Optimization in NeMo RL
  • Nemotron Omni MPO
  • An in-depth Walkthrough of DAPO in NeMo RL
  • LoRA (Low-Rank Adaptation)
  • An in-depth Walkthrough of CISPO in NeMo RL
  • An In-Depth Walkthrough of ProRLv2 in NeMo RL
  • Two-Stage SWE RL for Qwen3-30B-A3B-Thinking
  • An In-depth Walkthrough of GRPO in NeMo RL
  • Length Penalties for GRPO (NeMo-Gym)
  • An In-Depth Walkthrough of PPO in NeMo RL
  • GRPO on DeepScaler
  • Solve a Sliding Puzzle Using GRPO
  • Audio Post-training with NeMo RL
  • Audio-Visual GRPO with Qwen2.5-Omni-7B
  • Reward Model Training in NeMo RL
  • Environments for GRPO Training
  • DeepSeek-V3
  • Model Guides
    • DeepSeek
      • DeepSeek V4 Flash
    • GLM
      • GLM-5.1 and GLM-5.2
    • Gemma
      • Gemma 4
    • MiniMax
      • MiniMax-M3
    • Nemotron
      • Nemotron 3 Nano
      • Nemotron 3 Nano Omni
      • Nemotron 3.5 Lightning
      • Nemotron 3.5 Super VL
      • Nemotron 3 Super
      • Nemotron 3 Super Omni Image MOPD
      • Nemotron 3 Ultra
    • Qwen
      • Qwen3.5
      • Qwen3.8
  • Model Quirks
  • Train with Async GRPO
  • Train with Single-Controller (Async GRPO and PPO)
  • Quantization-Aware RL (QARL)
  • Train with Eagle3 Speculative Decoding
  • Speculative Decoding Draft Co-Training on Automodel Backend
  • YaRN Long-Context Training
  • Cross-Tokenizer (X-Token) Off-Policy Distillation
  • Weight Refit: Choosing a Transport
  • Checkpoint-Engine Refit
  • Managed Dynamo generation on Slurm
  • Router Replay
  • Muon Optimizer
  • Automodel Tensor Parallel Accuracy Issue
  • Fault Tolerance Launcher Guide

Containers

  • Build Docker Images

Development

  • CI/CD
  • Test NeMo RL
  • Documentation Development
  • Debug NeMo RL Applications
  • Profile GPU with Nsys
  • FP8 Quantization in NeMo RL
  • Experiment with Custom vLLM

Observability

  • Observability
    • Configuration
    • Span Groups
    • Metrics
    • vLLM Tracing
    • Exporting to an OTLP backend
    • Extending Instrumentation

Design Docs

  • Design and Philosophy
  • Padding in NeMo RL
  • Logger
  • uv in NeMo RL
  • Dependency Management
  • Data Format
  • Generation Interface
  • Colocated PPO streaming
  • NemoRL<>TRTLLM Disaggregation Rollout (Experimental)
  • Managed Dynamo generation design
  • Remote Sparse-Delta vLLM Refit
  • Checkpoint Engine Design
  • Exporting Checkpoints to Hugging Face Format
  • Loss functions in NeMo RL
  • Automodel Parallel Plan
  • Training Backends
  • Sequence Packing and Dynamic Batching
  • Environment Variable Precedence in NeMo RL
  • NeMo Gym Integration
  • ModelOpt Real-Quant Refit Architecture
  • End-to-End W4A4 NVFP4 with Per-Token vLLM Rollout
  • NCCL Reshard Refit (Experimental)
  • DSpark Auxiliary Heads
  • Training on text that contains <image>
  • Automodel v0.6.0 and Context-Parallel Integration
  • Token Capture Lineage Ledger

API Reference

  • API Reference
    • nemo_rl
      • nemo_rl.environments
        • nemo_rl.environments.vlm_environment
        • nemo_rl.environments.nemo_gym_multimodal
        • nemo_rl.environments.math_environment
        • nemo_rl.environments.code_jaccard_environment
        • nemo_rl.environments.nemotron_utils
        • nemo_rl.environments.code_environment
        • nemo_rl.environments.dapo_math_verifier
        • nemo_rl.environments.nemo_gym_request
        • nemo_rl.environments.rewards
        • nemo_rl.environments.nemo_gym_shards
        • nemo_rl.environments.nemo_gym
        • nemo_rl.environments.utils
        • nemo_rl.environments.metrics
        • nemo_rl.environments.reward_model_environment
        • nemo_rl.environments.interfaces
      • nemo_rl.evals
        • nemo_rl.evals.eval
        • nemo_rl.evals.answer_parsing
      • nemo_rl.algorithms
        • nemo_rl.algorithms.x_token
        • nemo_rl.algorithms.async_utils
        • nemo_rl.algorithms.single_controller_utils
        • nemo_rl.algorithms.loss
        • nemo_rl.algorithms.grpo_sync
        • nemo_rl.algorithms.dpo
        • nemo_rl.algorithms.metric_utils
        • nemo_rl.algorithms.sft
        • nemo_rl.algorithms.single_controller
        • nemo_rl.algorithms.grpo
        • nemo_rl.algorithms.logits_sampling_utils
        • nemo_rl.algorithms.xtoken_off_policy_distillation
        • nemo_rl.algorithms.rm
        • nemo_rl.algorithms.opd
        • nemo_rl.algorithms.reward_functions
        • nemo_rl.algorithms.mpo
        • nemo_rl.algorithms.distillation
        • nemo_rl.algorithms.sft_v2
        • nemo_rl.algorithms.advantage_estimator
        • nemo_rl.algorithms.utils
        • nemo_rl.algorithms.ppo
        • nemo_rl.algorithms.advantage_actor
      • nemo_rl.modelopt
        • nemo_rl.modelopt.models
        • nemo_rl.modelopt.utils
      • nemo_rl.weight_sync
        • nemo_rl.weight_sync.megatron_weight_synchronizer
        • nemo_rl.weight_sync.nccl_reshard_weight_synchronizer
        • nemo_rl.weight_sync.factory
        • nemo_rl.weight_sync.membership
        • nemo_rl.weight_sync.sglang_weight_synchronizer
        • nemo_rl.weight_sync.collective_weight_synchronizer
        • nemo_rl.weight_sync.checkpoint_engine_weight_synchronizer
        • nemo_rl.weight_sync.checkpoint_engine_config
        • nemo_rl.weight_sync.nccl_reshard_utils
        • nemo_rl.weight_sync.vllm_remote_sparse_weight_synchronizer
        • nemo_rl.weight_sync.xferdtensor
        • nemo_rl.weight_sync.ipc_weight_synchronizer
        • nemo_rl.weight_sync.xferdtensor_python
        • nemo_rl.weight_sync.interfaces
      • nemo_rl.utils
        • nemo_rl.utils.checkpoint_engines
        • nemo_rl.utils.packed_tensor
        • nemo_rl.utils.nsys
        • nemo_rl.utils.weight_transfer_http
        • nemo_rl.utils.rpc_guard
        • nemo_rl.utils.config
        • nemo_rl.utils.multimodal_payload_metrics
        • nemo_rl.utils.sequence_lengths
        • nemo_rl.utils.cuda_ipc
        • nemo_rl.utils.flops_formulas
        • nemo_rl.utils.fastokens
        • nemo_rl.utils.routed_experts_codec
        • nemo_rl.utils.checkpoint
        • nemo_rl.utils.r3_trace
        • nemo_rl.utils.weight_transfer_sparse_codec
        • nemo_rl.utils.logger
        • nemo_rl.utils.timer
        • nemo_rl.utils.flops_tracker
        • nemo_rl.utils.train_data_dump
        • nemo_rl.utils.length_penalty
        • nemo_rl.utils.outdated_config_checks
        • nemo_rl.utils.nvml
        • nemo_rl.utils.tensor_ops
        • nemo_rl.utils.weight_transfer_zmq
        • nemo_rl.utils.memory_tracker
        • nemo_rl.utils.grad_norm
        • nemo_rl.utils.weight_transfer_stream
        • nemo_rl.utils.prefetch_venvs
        • nemo_rl.utils.venvs
      • nemo_rl.experience
        • nemo_rl.experience.reward_penalties
        • nemo_rl.experience.rollouts
        • nemo_rl.experience.metric_utils
        • nemo_rl.experience.sync_rollout_actor
        • nemo_rl.experience.rollout_reassembler
        • nemo_rl.experience.route_assembly
        • nemo_rl.experience.route_plan
        • nemo_rl.experience.rollout_manager
        • nemo_rl.experience.payload
        • nemo_rl.experience.failures
        • nemo_rl.experience.rollout_reassembler_actor
        • nemo_rl.experience.rollout_recovery
        • nemo_rl.experience.interfaces
      • nemo_rl.data
        • nemo_rl.data.xtoken
        • nemo_rl.data.energon
        • nemo_rl.data.packing
        • nemo_rl.data.datasets
        • nemo_rl.data.dataloader
        • nemo_rl.data.chat_templates
        • nemo_rl.data.processors
        • nemo_rl.data.captured_media
        • nemo_rl.data.llm_message_utils
        • nemo_rl.data.deepseek_v4_encoding
        • nemo_rl.data.megatron_sft_packed
        • nemo_rl.data.chat_utils
        • nemo_rl.data.utils
        • nemo_rl.data.deepseek_v4_tokenizer
        • nemo_rl.data.multimodal_utils
        • nemo_rl.data.collate_fn
        • nemo_rl.data.interfaces
      • nemo_rl.models
        • nemo_rl.models.automodel
        • nemo_rl.models.generation
        • nemo_rl.models.megatron
        • nemo_rl.models.policy
        • nemo_rl.models.value
        • nemo_rl.models.huggingface
      • nemo_rl.distributed
        • nemo_rl.distributed.virtual_cluster
        • nemo_rl.distributed.model_utils
        • nemo_rl.distributed.refit_watchdog
        • nemo_rl.distributed.held_port
        • nemo_rl.distributed.collectives
        • nemo_rl.distributed.actor_environments
        • nemo_rl.distributed.named_sharding
        • nemo_rl.distributed.ray_actor_environment_registry
        • nemo_rl.distributed.batched_data_dict
        • nemo_rl.distributed.numa_utils
        • nemo_rl.distributed.worker_groups
        • nemo_rl.distributed.ray_worker_setup
        • nemo_rl.distributed.stateless_process_group
        • nemo_rl.distributed.worker_group_utils
      • nemo_rl.data_plane
        • nemo_rl.data_plane.adapters
        • nemo_rl.data_plane.worker_mixin
        • nemo_rl.data_plane.codec
        • nemo_rl.data_plane.grouping
        • nemo_rl.data_plane.factory
        • nemo_rl.data_plane.driver_mixin
        • nemo_rl.data_plane.column_io
        • nemo_rl.data_plane.schema
        • nemo_rl.data_plane.tq_token_sink
        • nemo_rl.data_plane.async_utils
        • nemo_rl.data_plane.mooncake_storage_unit
        • nemo_rl.data_plane.observability
        • nemo_rl.data_plane.preshard
        • nemo_rl.data_plane.interfaces
      • nemo_rl.telemetry
        • nemo_rl.telemetry.instrumentation
        • nemo_rl.telemetry.host_memory
        • nemo_rl.telemetry.config
        • nemo_rl.telemetry.span_groups
        • nemo_rl.telemetry.vocabulary
        • nemo_rl.telemetry.setup
        • nemo_rl.telemetry.metrics
      • nemo_rl.package_info
      • nemo_rl.transformers_compat
  • API Reference
  • nemo_rl
  • nemo_rl.modelopt
  • nemo_rl.modelopt.models
  • nemo_rl.modelopt.models.policy
  • nemo_rl.modelopt.models.policy.workers
  • nemo_rl.modelopt.models.policy.workers.automodel_quant_policy_worker

nemo_rl.modelopt.models.policy.workers.automodel_quant_policy_worker#

Module Contents#

Classes#

AutomodelQuantPolicyWorker

API#

class nemo_rl.modelopt.models.policy.workers.automodel_quant_policy_worker.AutomodelQuantPolicyWorker(*args, **kwargs)#

Bases: nemo_rl.models.policy.workers.automodel_policy_worker.AutomodelPolicyWorkerImpl

previous

nemo_rl.modelopt.models.policy.workers

next

nemo_rl.modelopt.models.policy.workers.megatron_quant_policy_worker
On this page
  • Module Contents
    • Classes
    • API
      • AutomodelQuantPolicyWorker
NVIDIA NVIDIA
Privacy Policy | Your Privacy Choices | Terms of Service | Accessibility | Corporate Policies | Product Security | Contact

Copyright © 2026, NVIDIA Corporation.