Integrate RL Frameworks

These guides cover how to integrate NeMo Gym into a new RL training framework. Use them if you are:

A training framework maintainer adding NeMo Gym support
Contributing NeMo Gym integration for a training framework that does not have one yet

Just want to train models? See Training Tutorials for supported frameworks.

Prerequisites

Before integrating Gym into your training framework, ensure you have:

An RL training framework with policy optimization support (PPO, GRPO, or similar)
A generation backend (vLLM, SGLang, or equivalent)
Familiarity with OpenAI-compatible HTTP server APIs

Gym integration requires implementing the following components in your training framework:

OpenAI-compatible HTTP server requirements and existing implementations across RL frameworks.

Fixes for on-policy training in multi-step and multi-turn scenarios to prevent train-generation mismatch.

Implementation components, form factor, and reference implementations from NeMo RL.

Validation criteria and benchmarks to verify correct Gym integration.

The typical integration workflow follows this sequence:

Step	Component	Description
1	Generation backend	Expose your generation engine, such as vLLM or SGLang, as an OpenAI-compatible HTTP server
2	On-policy corrections	Implement token ID fixes to prevent re-tokenization and re-templating issues
3	Gym integration	Connect Gym to your training loop using the rollout orchestration APIs
4	Validation	Verify integration using the success criteria benchmarks