Reinforcement Learning

verl Integration

Run the public verl-recipe Dynamo rollout backend

View as Markdown

Experimental. verl-recipe provides an asynchronous Dynamo rollout backend with a shared frontend, vLLM workers, routing, and colocated policy updates. Use the upstream recipe as the implementation source of truth; this page covers the shortest Dynamo workflow and its support boundary.

Integration Shape

ConcernCurrent path
Training and rollout orchestrationverl and verl-recipe
GenerationDynamo vLLM workers behind one shared frontend
RoutingNative Dynamo router when ThunderAgent is disabled; ThunderAgent when explicitly enabled
Policy updateRecipe-owned Ray/ZMQ control with colocated CUDA IPC
Service discoveryRecipe-managed etcd and NATS
DeploymentLocal or multi-node Ray processes; Kubernetes is not required

The native router and ThunderAgent are different scheduling paths. Choose one before building the environment and do not compare their results as if only a router flag changed.

Prerequisites

  • A Linux GPU environment that satisfies the selected verl, Dynamo, vLLM, CUDA, and PyTorch versions
  • git, Python, etcd, and nats-server
  • Model and dataset paths visible on every participating node
  • Enough GPUs for the trainer and rollout layout; the validation smoke is not a full training run

Prepare the Source

The reviewed recipe snapshot is 461b830c, and its REQUIRED_VERL.txt selects the matching core verl commit. The core commit records an older recipe submodule, so use the installer and then set the nested recipe checkout to the same reviewed snapshot:

$git clone https://github.com/verl-project/verl-recipe.git
$git -C verl-recipe checkout 461b830cfee4f5a67c21edc300c24373230babc7
$
$cd verl-recipe
$./install_verl.sh --recipe dynamo --method git --dest ../verl
$
$git -C ../verl/recipe fetch origin 461b830cfee4f5a67c21edc300c24373230babc7
$git -C ../verl/recipe checkout 461b830cfee4f5a67c21edc300c24373230babc7
$test -f ../verl/recipe/dynamo/main_dynamo.py

The recipe does not pin a complete Dynamo/vLLM image for the native-router path. Build one clean environment, record its immutable image and package versions, and keep the Dynamo and nested recipe checkouts clean during validation.

Run the Validation Smoke

Run the upstream validation-only smoke from the resulting verl checkout:

$cd ../verl
$
$MODEL_PATH=/models/Qwen2.5-0.5B-Instruct \
>TRAIN_FILE=/data/dapo-math-17k.parquet \
>TEST_FILE=/data/aime-2024.parquet \
>RAY_DATA_HOME=/data/verl \
>bash recipe/dynamo/smoke_dynamo_v1.sh

The smoke starts recipe-managed etcd, NATS, a Dynamo vLLM worker, and the shared frontend. PASS: Dynamo validation smoke completed confirms the validation command completed; it does not prove an optimizer step or policy refresh.

Run a Training Iteration

After the smoke passes, run at least one optimizer step with the same environment. Use the pinned upstream Dynamo trainer configuration as the baseline, then make the model, data, resource, and routing overrides required for your environment. This example selects the native Dynamo router explicitly:

$export VERL_USE_EXTERNAL_MODULES=recipe.dynamo.register
$export VERL_DYNAMO_WORKER_METRICS_DIR=/tmp/verl-dynamo/workers
$mkdir -p "$VERL_DYNAMO_WORKER_METRICS_DIR"
$
$python3 recipe/dynamo/metrics_sidecar.py \
>  --endpoints-glob "$VERL_DYNAMO_WORKER_METRICS_DIR/*.endpoints" \
>  --output /tmp/verl-dynamo/kv-metrics.jsonl \
>  --label dynamo-kv \
>  --interval 30 &
$METRICS_SIDECAR_PID=$!
$trap 'kill "$METRICS_SIDECAR_PID" 2>/dev/null || true' EXIT
$
$python3 -m recipe.dynamo.main_dynamo \
>  algorithm.adv_estimator=grpo \
>  data.train_files=/data/gsm8k/train.parquet \
>  data.val_files=/data/gsm8k/test.parquet \
>  actor_rollout_ref.model.path=Qwen/Qwen2.5-0.5B-Instruct \
>  actor_rollout_ref.rollout.name=dynamo \
>  actor_rollout_ref.rollout.mode=async \
>  actor_rollout_ref.rollout.engine_kwargs.dynamo.router_mode=kv \
>  ++actor_rollout_ref.rollout.engine_kwargs.dynamo.thunderagent.enabled=false \
>  ++actor_rollout_ref.rollout.engine_kwargs.dynamo.enable_worker_system_metrics=true \
>  trainer.n_gpus_per_node=2 \
>  trainer.nnodes=1 \
>  trainer.total_training_steps=2

Adjust model, data, and resource values for your environment. A passing run must include rollout generation, reward or advantage computation, an actor update, policy synchronization, and generation after the update.

Verify the Run

Check three boundaries before scaling:

  1. Generation correctness: The completion token IDs and selected log probabilities consumed by verl match in length and order. Record terminal and canceled attempts separately.
  2. Policy update: Every intended rollout shard receives the same trainer step through the recipe’s CUDA IPC path, stale KV state is handled, and post-update generation succeeds.
  3. Routing: With ThunderAgent disabled, compare round-robin and kv using the same prompts, concurrency, cache state, and update cadence. Report useful framework output, not only request throughput.

Set request_completion_token_ids=true when the framework must score the exact engine tokens. Use RL Integration Reference for the shared response checks and KV-Aware Load Balancing for RL Rollouts for the routing experiment.

Observe and Recover

The training command above enables worker system metrics and starts the provided sidecar before workers come online. The sidecar rediscovers endpoint files throughout the run and writes per-worker snapshots to /tmp/verl-dynamo/kv-metrics.jsonl; starting it only after training leaves no live workers to scrape.

SymptomCheck first
Frontend never becomes readyetcd/NATS health, worker logs, port reachability, and expected registration count
Trainer token scores differCompletion token IDs, logprob alignment, tokenizer/model version, and response adaptation
One shard remains staleRay actor, ZMQ bridge, CUDA rank mapping, and per-shard update acknowledgements
Cache reuse drops after updateCache reset, worker wake-up, and the warm-up boundary
Shutdown leaves processesRecipe watchdog and frontend, worker, NATS, then etcd teardown

See Profile and Simulate RL Rollouts for request tracing and Distribute and Update Rollout Weights for the policy-update boundary.

Current Limitations

  • The recipe does not provide one complete native-path Dynamo/vLLM image pin.
  • The documented policy update is the recipe’s colocated CUDA IPC path, not public Dynamo worker discovery or ModelExpress.
  • Multi-node and large-model layouts require separate topology and failure validation.
  • Supported status requires an independent run with token correctness, policy refresh, post-update generation, and request, worker, and update recovery.

Upstream Resources