RL Implementation Guide

Experimental. This guide is for engineers using Dynamo to serve RL rollouts during a training loop. It covers how to return token IDs and log probabilities, expose routing or engine metadata, discover live workers, and push weight updates without restarting the serving stack. For SGLang rollouts, use the SGLang-compatible /generate API as the primary inference path. Discover vLLM workers through the read-only RL discovery API. Send lifecycle or weight updates directly to the selected worker system URL.

Dynamo can add value to RL rollout serving when the rollout plane needs more than a static inference endpoint. Advanced routing can steer rollout traffic across heterogeneous workers, weight synchronization with Model Express can help move updated checkpoints into serving quickly, fault tolerance can keep rollout generation available across worker failures, and autoscaling can match serving capacity to changing rollout demand during training.

What This Page Covers

Backend Support

Experimental means the RL integration surfaces are still converging around real framework usage. For SGLang, build new token-in/token-out integrations on /generate first. This API preserves the request fields and streaming response objects from the installed SGLang version. Use the OpenAI-compatible routes when an integration needs a cross-backend schema or nvext.metadata_upload. Treat engine_data, uploaded meta_info, direct /engine/ route bodies, and custom route names as backend-specific interfaces. Their contracts can change independently.

Choose an Interface

The discovery API runs on a dedicated frontend listener. It is not mounted on the main frontend port, and it does not proxy /engine/ calls. The request-plane URL in its response is used internally to query route descriptors; an RL orchestrator should use the returned system_url for administration.

SGLang Happy Path

Use /generate for SGLang reinforcement learning clients that send token IDs and consume SGLang streaming responses. Aggregated and prefill/decode deployments accept the same request.

$
DYN_SGLANG_ENABLE_GENERATE=1 python -m dynamo.frontend

$
python -m dynamo.sglang \
>
  --model-path Qwen/Qwen3-0.6B

$
curl -N http://localhost:8000/generate \
>
  -H 'Content-Type: application/json' \
>
  -d '{
>
    "input_ids": [151644, 8948, 198],
>
    "sampling_params": {
>
      "max_new_tokens": 32,
>
      "temperature": 0,
>
      "n": 1
>
    },
>
    "stream": true,
>
    "return_logprob": true,
>
    "top_logprobs_num": 0
>
  }'

The current API has these limits:

• Provide one non-empty input_ids sequence.
• Set stream to true.
• Set sampling_params.n to 1.
• Use token input. The API does not support text, batched, multimodal, or non-streaming requests.
• Use an aggregated deployment for prompt log probabilities. They are not parity-complete in prefill/decode deployments.

Dynamo preserves other public SGLang request fields. The worker validates them against the installed SGLang version. The frontend rejects Dynamo-owned bootstrap and routing fields. Dynamo injects those fields after it selects a worker.

vLLM Happy Path

This is the shortest path for wiring an RL orchestrator to a vLLM rollout worker: start the frontend discovery listener, start an RL-enabled worker, discover its system URL, run a token-in rollout, then pause, update weights, and resume generation.

$
DYN_ENABLE_RL=true DYN_RL_PORT=8001 python -m dynamo.frontend

$
DYN_SYSTEM_PORT=8081 python -m dynamo.vllm \
>
  --model Qwen/Qwen3-0.6B \
>
  --enable-rl

$
curl http://localhost:8001/v1/rl/workers

$
curl http://localhost:8000/v1/completions \
>
  -H 'Content-Type: application/json' \
>
  -d '{
>
    "model": "Qwen/Qwen3-0.6B",
>
    "prompt": "",
>
    "max_tokens": 32,
>
    "temperature": 0,
>
    "logprobs": 5,
>
    "prompt_logprobs": 5,
>
    "nvext": {
>
      "token_data": [151644, 8948, 198],
>
      "extra_fields": ["completion_token_ids", "prompt_logprobs"]
>
    }
>
  }'

$
worker_url=http://10.0.0.12:8081
$
$
curl --fail-with-body "$worker_url/engine/pause_generation" \
>
  -H 'Content-Type: application/json' \
>
  -d '{"mode": "keep", "clear_cache": false}'
$
$
curl --fail-with-body "$worker_url/engine/update_weights_from_disk" \
>
  -H 'Content-Type: application/json' \
>
  -d '{"model_path": "/models/checkpoint-42", "weight_version": "42"}'
$
$
curl --fail-with-body "$worker_url/engine/resume_generation" \
>
  -H 'Content-Type: application/json' \
>
  -d '{}'

Frontend Feature Support

The OpenAI-compatible completion routes provide a cross-backend token-in/token-out interface. For SGLang clients that use native request and response shapes, prefer /generate.

See NVIDIA Request Extensions for the complete nvext reference.

Backend RL flags also select engine-specific behavior:

• On vLLM, --enable-rl registers the discovery and administration routes, selects processed log probabilities, and prevents model generation_config.json sampling overrides from changing RL token-in requests marked with nvext.token_data. Explicit request sampling values still apply.
• On SGLang, --enable-rl enables out-of-band meta_info upload and the /engine/call_tokenizer_manager passthrough route. SGLang also exposes control routes such as /engine/control/update_weights_from_disk, but SGLang workers do not currently register with /v1/rl/workers.

Token-In/Token-Out Example

Send pre-tokenized input and request token IDs plus prompt and completion log probabilities:

$
curl http://localhost:8000/v1/completions \
>
  -H 'Content-Type: application/json' \
>
  -d '{
>
    "model": "Qwen/Qwen3-0.6B",
>
    "prompt": "",
>
    "max_tokens": 32,
>
    "temperature": 0,
>
    "logprobs": 5,
>
    "prompt_logprobs": 5,
>
    "nvext": {
>
      "token_data": [151644, 8948, 198],
>
      "extra_fields": ["completion_token_ids", "prompt_logprobs"]
>
    }
>
  }'

The relevant response fields have this shape:

1
{
2
  "choices": [
3
    {
4
      "index": 0,
5
      "text": "...",
6
      "logprobs": {}
7
    }
8
  ],
9
  "nvext": {
10
    "completion_token_ids": [9707, 11],
11
    "prompt_logprobs": [
12
      null,
13
      {"8948": {"logprob": -0.42}},
14
      {"198": {"logprob": -0.31}}
15
    ]
16
  }
17
}

To use the chat route while retaining pre-tokenized input, provide the normal messages field and set nvext.token_data to the complete token sequence that the engine should receive. The frontend uses token_data instead of rendering and tokenizing messages.

Routed Expert Data

Opt into routed-expert data per request:

1
{
2
  "model": "my-moe-model",
3
  "prompt": [1, 2, 3],
4
  "max_tokens": 16,
5
  "nvext": {
6
    "extra_fields": ["completion_token_ids", "routed_experts"]
7
  }
8
}

The payload format is backend-specific:

• vLLM-compatible builds return an object containing base64 data, shape, dtype, and start. The start offset identifies the first returned routing row in the full prompt-plus-completion sequence.
• SGLang builds whose async_generate API supports return_routed_experts return the engine’s base64 string. Start SGLang with --enable-return-routed-experts to request capture. Builds that omit this engine argument do not return the field.
• TensorRT-LLM does not currently return routed-expert data through this frontend extension.

Use nvext.engine_data only when the orchestrator must consume other backend-specific data. The named completion_token_ids, prompt_logprobs, and routed_experts fields provide more stable contracts.

Upload SGLang Metadata

The SGLang metadata upload feature uses the OpenAI-compatible route and nvext. When the rollout pipeline must store large metadata objects outside the HTTP response, use this feature. SGLang can upload the final cumulative meta_info for each choice to any filesystem supported by the installed fsspec backend.

Start the worker with RL support. Add the routed-expert flag only when the SGLang engine build supports it:

$
DYN_SYSTEM_PORT=8081 python -m dynamo.sglang \
>
  --model-path Qwen/Qwen3-0.6B \
>
  --enable-rl \
>
  --enable-return-routed-experts

Set a unique upload directory for each request:

1
{
2
  "model": "Qwen/Qwen3-0.6B",
3
  "prompt": [151644, 8948, 198],
4
  "max_tokens": 32,
5
  "logprobs": 5,
6
  "nvext": {
7
    "metadata_upload": {
8
      "url": "s3://rl-rollouts/run-42/request-7"
9
    }
10
  }
11
}

The worker writes choice_0.msgpack.zst, choice_1.msgpack.zst, and so on beneath the URL. Each file contains a Zstandard-compressed MessagePack object:

1
{
2
  "schema_version": 1,
3
  "metadata": {
4
    "id": "...",
5
    "finish_reason": {"type": "stop"},
6
    "output_token_logprobs": [],
7
    "output_top_logprobs": [],
8
    "routed_experts": "..."
9
  }
10
}

Install the matching fsspec extra for remote storage, such as fsspec[s3] for S3. The worker also requires msgspec and zstandard. Local URLs such as file:///tmp/rollouts/request-7 are useful for development.

When metadata_upload is present, SGLang uploads only the final cumulative meta_info for each choice and omits inline log probabilities and routed-expert metadata. The final response waits for the upload. An upload failure fails the request, so use a durable destination and a unique URL to avoid overwriting another request’s choice_<index>.msgpack.zst objects.

Discover RL Workers

RL discovery currently covers RL-enabled vLLM workers. Start the frontend discovery listener and a vLLM worker with its system server enabled:

$
DYN_ENABLE_RL=true DYN_RL_PORT=8001 python -m dynamo.frontend

$
DYN_SYSTEM_PORT=8081 python -m dynamo.vllm \
>
  --model Qwen/Qwen3-0.6B \
>
  --enable-rl

Query the dedicated discovery port:

$
curl http://localhost:8001/v1/rl/workers

The response describes each live worker and probes it for the routes available in that process:

1
{
2
  "namespace": "dynamo",
3
  "workers": [
4
    {
5
      "namespace": "dynamo",
6
      "component": "backend",
7
      "endpoint": "rl",
8
      "instance_id": 12345,
9
      "transport": {"tcp": "10.0.0.12:1234/..."},
10
      "request_plane_url": "dyn://dynamo.backend.rl",
11
      "system_url": "http://10.0.0.12:8081",
12
      "model": "Qwen/Qwen3-0.6B",
13
      "routes": [
14
        "get_weight_version",
15
        "liveness_probe",
16
        "pause_generation",
17
        "resume_generation",
18
        "update_weights_from_disk"
19
      ]
20
    }
21
  ]
22
}

Treat routes as the capability list for that worker instead of assuming every engine exposes the same methods. Optional features such as Low-Rank Adaptation (LoRA) add routes only when enabled. Discovery can also return a worker with an error and an empty route list when its descriptor probe times out or fails.

The discovery listener uses these environment variables:

Call vLLM Engine Routes Directly

Read system_url and routes from the discovery response, then call the selected vLLM worker. Send a JSON object even when the route does not require arguments:

$
curl http://10.0.0.12:8081/engine/liveness_probe \
>
  -H 'Content-Type: application/json' \
>
  -d '{}'

For a vLLM weight update cycle, pause the selected worker, call one of its advertised weight-update routes, validate the result, and resume generation. Install jq before running this example. The exit trap attempts to resume the worker if the update or another command fails:

$
set -euo pipefail
$
$
worker_url=http://10.0.0.12:8081
$
$
resume_generation() {
>
  response=$(curl --fail-with-body "$worker_url/engine/resume_generation" \
>
    -H 'Content-Type: application/json' \
>
    -d '{}')
>
  jq -e '.status == "ok"' <<<"$response" >/dev/null
>
}
$
$
pause_response=$(curl --fail-with-body "$worker_url/engine/pause_generation" \
>
  -H 'Content-Type: application/json' \
>
  -d '{"mode": "keep", "clear_cache": false}')
$
jq -e '.status == "ok"' <<<"$pause_response" >/dev/null
$
trap resume_generation EXIT
$
$
update_response=$(curl --fail-with-body "$worker_url/engine/update_weights_from_disk" \
>
  -H 'Content-Type: application/json' \
>
  -d '{"model_path": "/models/checkpoint-42", "weight_version": "42"}')
$
jq -e '.status == "ok"' <<<"$update_response" >/dev/null
$
$
resume_generation
$
trap - EXIT

Route request bodies and response fields are engine-specific. A callback can return {"status":"error"} with HTTP 200, while an exception in the callback produces HTTP 500. Check both the HTTP status and the JSON result before advancing the rollout state.

SGLang direct administration uses SGLang-specific routes and response shapes. For example, call /engine/control/update_weights_from_disk directly when you already know the worker URL, or call /engine/call_tokenizer_manager with {"method":"update_weights_from_disk", ...} for tokenizer-manager passthrough behavior.

The /v1/rl/workers endpoint is read-only. It intentionally does not expose /v1/rl/engine or /v1/rl/engines proxy routes, which prevents an accidental frontend fan-out of a mutating engine operation.

Register a Custom Engine Route

Register an asynchronous Python callback on the worker’s DistributedRuntime. The route name is appended to /engine/ on the system server:

1
from typing import Any
2
3
4
async def set_rollout_state(body: dict[str, Any]) -> dict[str, Any]:
5
    rollout_id = body.get("rollout_id")
6
    if not isinstance(rollout_id, str) or not rollout_id:
7
        return {"status": "error", "message": "rollout_id is required"}
8
9
    await engine.set_rollout_state(rollout_id)
10
    return {"status": "ok", "rollout_id": rollout_id}
11
12
13
runtime.register_engine_route("rl/set_rollout_state", set_rollout_state)

Enable the system server with DYN_SYSTEM_PORT, then call the route directly:

$
curl http://worker-host:8081/engine/rl/set_rollout_state \
>
  -H 'Content-Type: application/json' \
>
  -d '{"rollout_id": "run-42-step-7"}'

register_engine_route makes the HTTP route callable but does not automatically advertise it in the RL discovery response. When extending an RL-enabled vLLM worker, register and advertise the route together with the shared helper:

1
from dynamo.common.rl import register_rl_routes
2
3
4
register_rl_routes(
5
    runtime,
6
    handler.rl_route_registry,
7
    {"rl/set_rollout_state": set_rollout_state},
8
    enable_dispatch=handler.config.enable_rl,
9
)

The next GET /v1/rl/workers probe includes rl/set_rollout_state in that worker’s routes list.