For AI agents: a documentation index is available at the root level at /llms.txt and /llms-full.txt. Append /llms.txt to any URL for a page-level index, or .md for the markdown version of any page.
DocumentationAPI Reference
DocumentationAPI Reference
  • About
    • Concepts
    • Architecture
    • Ecosystem
    • Release Notes
  • Get Started
    • Prerequisites
    • Installation
    • Quickstart
  • Agent Server
  • Model Server
    • vLLM
  • Resources Server
  • Data
    • Prepare and Validate
    • Download from Hugging Face
    • Prompt Config
  • Environment Tutorials
    • Single-Step Environment
    • Multi-Step Environment
    • Stateful Environment
    • Real-World Environment
    • Integrate external libraries
    • Add a benchmark
    • Verification Patterns
    • Aggregate Metrics
  • Training Tutorials
    • NeMo RL
    • Unsloth
    • Multi-Environment Training
    • Training with VeRL
    • Offline Training (SFT/DPO)
  • Model Recipes
    • Nemotron 3 Nano
    • Nemotron 3 Super
  • Infrastructure
    • Deployment Topology
    • Engineering Notes
  • Reference
    • Configuration
    • RL Framework Compatibility
    • CLI Commands
    • FAQ
  • Troubleshooting
    • Configuration Errors
  • Contribute
    • Development Setup
    • Environments
    • Integrate RL Frameworks
      • Generation Backend
      • Integration Footprint
      • On-Policy Corrections
      • Success Criteria
NVIDIANVIDIA
Developer-friendly docs for your API
Privacy Policy | Your Privacy Choices | Terms of Service | Accessibility | Corporate Policies | Product Security | Contact

Copyright © 2026, NVIDIA Corporation.

LogoLogoNeMo Gym
On this page
  • OpenAI-Compatible Server Requirements
  • Generation in RL Training
  • Existing Framework Implementations
  • Integration Guidelines
  • Frameworks Using vLLM or SGLang
  • Frameworks Using Other Backends
  • Related Topics
ContributeIntegrate RL Frameworks

Generation Backend

||View as Markdown|
Previous

Integrate RL Frameworks

Next

Integration Footprint

Gym requires an OpenAI-compatible HTTP server to handle model generations during training. This page covers the server requirements and existing implementations across popular RL frameworks.

OpenAI-Compatible Server Requirements

Gym communicates with generation backends using the OpenAI HTTP API specification. Your generation server must implement endpoints compatible with one of these reference implementations:

ProviderDocumentation
OpenAI APIResponses API Reference
GeminiOpenAI Compatibility
vLLMOpenAI-Compatible Server
SGLangOpenAI-Compatible APIs
TGIOpenAI Messages API

Generation in RL Training

Most RL frameworks that support policy optimization algorithms (PPO, GRPO) require online on-policy model generations. Integrating generation backends into the RL training loop introduces several challenges:

  • Refit: Synchronizing model weights between training and generation
  • Off-policyness: Ensuring generations reflect the current policy state
  • Latency: Minimizing generation overhead during training iterations

Existing Framework Implementations

The following table shows how popular RL frameworks implement generation backends.

If your framework uses vLLM or SGLang, you can reference these implementations when adding OpenAI HTTP server support.

FrameworkGeneration BackendReference Implementation
NeMo RLvLLMvllm_generation.py
VeRLHF, vLLM, SGLanghf_rollout.py, vLLM rollout, SGLang rollout
TRLvLLM, HFgrpo_trainer.py (vLLM), grpo_trainer.py (HF)
SlimeSGLangsglang_engine.py
OpenPIPE ARTvLLMvLLM module

NeMo RL, VeRL, Slime, and OpenPIPE ART all expose OpenAI-compatible HTTP server endpoints.

Integration Guidelines

Frameworks Using vLLM or SGLang

If your training framework already uses vLLM or SGLang but does not expose an OpenAI-compatible HTTP server:

  1. Reference the implementations listed above
  2. Add server endpoints that follow the OpenAI API specification
  3. Test your implementation using the vLLM HTTP server tests from NeMo RL

Frameworks Using Other Backends

If your training framework does not use vLLM or SGLang as a generation backend, you may need significant refactoring to achieve proper Gym integration. Consider:

  • Migrating to vLLM or SGLang for generation
  • Implementing an adapter layer that exposes OpenAI-compatible endpoints
  • Evaluating the complexity of maintaining a custom generation backend

Related Topics

After setting up your generation backend, proceed to:

  • On-Policy Corrections - Required fixes for multi-step and multi-turn scenarios
  • Gym Integration Footprint And Form Factor - Full integration component breakdown