Reinforcement Learning

Use Dynamo as the rollout-serving layer for RL workloads

View as Markdown

Dynamo provides routing, worker management, weight-update controls, and serving telemetry for reinforcement learning (RL) rollouts. Your RL framework continues to own training, rewards, environments, trajectory semantics, checkpoints, and sample acceptance.

Use Dynamo when rollout serving has become a distributed-systems problem: many workers serve bursty traffic, samples share large prompt prefixes, policies must refresh without rebuilding the serving stack, or you need to diagnose and replay the serving workload independently of the trainer.

Proven on Real RL Workloads

In production, Cognition used Dynamo while training SWE-1.7 to manage inference-engine lifecycles and route inference across a multi-cluster RL system. When a replica failed, Dynamo rerouted inference to another worker and rescheduled the replica so the rollout pipeline could remain available while the latest policy state was restored.

Dynamo also connects to a growing RL framework ecosystem. verl publishes a Dynamo rollout recipe, NeMo RL includes a managed Dynamo generation backend, and Prime-RL supports the Dynamo router as a drop-in routing option. Integration work with SLIME extends the same serving capabilities to SGLang-based rollout stacks. Together, these paths demonstrate that Dynamo can support different training frameworks and deployment models while leaving RL semantics with the trainer.

Keep Rollouts Running Through Failures

Dynamo detects worker loss and routes new work to healthy capacity. It can also migrate supported in-flight generations, reject new work explicitly when every eligible worker is overloaded, propagate client cancellation, and drain workers during planned shutdown. The RL framework still decides when to retry, discard, or accept a sample.

These serving behaviors were central to Cognition’s deployment above: failed replicas could be replaced without restarting the training system’s serving plane. Request migration and overload rejection are optional frontend behaviors; cancellation is built into supported request paths, while graceful shutdown and engine failover depend on the deployment. See Fault Tolerance for the supported behaviors, defaults, and limitations.

Decide Whether to Add Dynamo

Current setupRecommended path
One rollout worker with no routing or live-update problemKeep the framework’s direct backend path.
Multiple workers serving repeated or bursty promptsEvaluate Dynamo routing against the current direct-backend baseline.
A colocated framework already owns policy transferUse Dynamo for generation and routing while keeping the proven framework update path.
An external rollout fleet needs discovery and direct controlIntegrate Dynamo’s request, discovery, and worker-administration surfaces explicitly.

Start with a specific bottleneck and a baseline metric. Measure cache reuse, worker imbalance, policy-refresh time, or time to diagnose a failure before adding Dynamo.

Where Dynamo Fits

The reinforcement learning loop from prompts through rollout generation, rewards, and optimization. Dynamo owns rollout-serving orchestration, inference backends execute generation and apply weights, and the RL framework owns every training decision.

Dynamo does not decide whether a trajectory is on-policy, accepted, or fresh enough for training. The framework must gate requests around synchronous updates or enforce its own bounded-staleness policy.

Framework Integrations

FrameworkStatusStart here
Custom or external trainerExperimental contractExternal Trainer Integration for generation, discovery, administration, and policy updates.
verlExperimentalverl Integration for the public colocated Dynamo/vLLM recipe.
NeMo RLExperimentalNeMo RL Integration for the managed Slurm/Ray Dynamo backend.
SLIMEIntegration in progressReview the current boundary in Framework Compatibility.
Prime-RLRouting available; integration in progressSee Prime-RL’s routing overview and the current boundary in Framework Compatibility.

Experimental guides have runnable upstream artifacts but do not make a general compatibility promise. Integrations in progress remain in the compatibility table until a maintained path lands.

Kubernetes is optional for these RL integrations. Use it only when the selected framework or deployment environment requires it.

Next Steps