RL Integration Reference

Use this reference when implementing or reviewing a Dynamo rollout adapter. It defines the shared serving contract once; framework guides describe only what differs for that integration.

Separate the Three Planes

Send rollout inference through the shared frontend. Send mutating operations only to selected workers. The discovery endpoint is read-only and does not create a fleet-wide transaction.

Choose a Request Interface

Use the native SGLang route when the framework already speaks that schema. Use an OpenAI-compatible route when the adapter needs one request envelope across backends or named nvext response fields.

Preserve Token Authority

The engine token sequence used for training is authoritative. Do not reconstruct it by tokenizing generated text; chat templates, normalization, special tokens, and tokenizer versions can change the result.

For OpenAI-compatible completions, send token IDs and request generated token IDs explicitly:

$
curl http://localhost:8000/v1/completions \
>
  -H 'Content-Type: application/json' \
>
  -d '{
>
    "model": "Qwen/Qwen3-0.6B",
>
    "prompt": [151644, 8948, 198],
>
    "max_tokens": 32,
>
    "temperature": 0,
>
    "logprobs": 0,
>
    "nvext": {"extra_fields": ["completion_token_ids"]}
>
  }'

Before admitting a sample to training, verify:

1. Generated token IDs are present and contain the expected number of choices.
2. Selected log probabilities match those token IDs in length and order.
3. Prompt log probabilities preserve undefined positions instead of shifting alignment.
4. The response has a terminal state recognized by the framework.
5. The tokenizer and model identity match the selected rollout worker.

POST /v1/responses/input_tokens estimates input size for preflight decisions. It does not return authoritative token IDs, verify the model or tokenizer, or prove that a worker is ready.

Know What Returns to the Trainer

See NVIDIA Request Extensions for the complete nvext request and response shapes.

Handle Streaming and Retries

Treat a streaming request as a state machine: accepted, streaming, terminal success, canceled, or failed. Admit only a verified terminal result. Preserve attempt identity when retrying, because a timed-out request may have completed after the client stopped waiting and generation is not idempotent.

The framework owns retry policy, duplicate suppression, partial-rollout handling, and sample acceptance. Dynamo propagates supported cancellations and serving failures but does not decide whether a partial attempt is scoreable.

Discover Workers Safely

Start the RL listener and an RL-enabled vLLM worker on the trusted control network:

$
DYN_ENABLE_RL=true DYN_RL_PORT=8001 python -m dynamo.frontend

$
DYN_SYSTEM_PORT=8081 python -m dynamo.vllm \
>
  --model Qwen/Qwen3-0.6B \
>
  --enable-rl

$
curl http://localhost:8001/v1/rl/workers

A protocol version 1 response has this shape. Optional fields such as model, system_url, admin_base_url, and world_size appear only when the worker provides them:

1
{
2
  "protocol_version": 1,
3
  "namespace": "dynamo",
4
  "workers": [
5
    {
6
      "namespace": "dynamo",
7
      "component": "backend",
8
      "endpoint": "rl",
9
      "instance_id": 12345,
10
      "transport": {"tcp": "10.0.0.12:1234/..."},
11
      "request_plane_url": "dyn://dynamo.backend.rl",
12
      "system_url": "http://10.0.0.12:8081",
13
      "model": "Qwen/Qwen3-0.6B",
14
      "routes": [
15
        "get_weight_version",
16
        "liveness_probe",
17
        "pause_generation",
18
        "resume_generation",
19
        "update_weights_from_disk"
20
      ]
21
    }
22
  ]
23
}

Check protocol_version before reading the worker list. In protocol version 1:

Refresh discovery before each control phase and require the complete capability set for the selected update path. Do not cache membership indefinitely or interpret list position as worker identity.

Coordinate Policy Refresh

The framework owns the fleet-level lifecycle:

1. Gate new rollout work for the target workers.
2. Resolve the current worker set and required capabilities.
3. Pause or drain workers when the backend requires it.
4. Transfer and apply one target policy.
5. Invalidate stale KV state.
6. Verify every worker and run post-update generation before reopening the fleet.

Per-worker success is not fleet-wide atomicity. Keep generation gated when membership changes, an update fails, cache reset fails, or post-update generation does not pass. See Distribute and Update Rollout Weights for the supported paths and recovery rules.

Compare vLLM and SGLang Administration

The backends do not share one administration schema. Use the exact route returned by discovery or configured by the deployment, and validate request bodies against the installed backend version.

Even two vLLM deployments can expose different route families because the Python worker and native sidecar adapt different backend control APIs. Never prepend, remove, or rename route segments returned in routes.

Framework Compatibility

A status reflects the documented integration, not whether the framework can call a generic HTTP endpoint.

Backend Compatibility

Backend support does not imply every topology or framework combination is validated. Record aggregated or prefill/decode serving, placement, model parallelism, model class, cancellation, discovery, weight layout, and cache behavior for the path you test.

Integration Checklist

Before publishing a runnable integration, verify:

• Generation reaches the intended Dynamo frontend and backend.
• Token IDs, log probabilities, masks, and terminal states match the framework contract.
• Retries and cancellations cannot admit the same sample twice.
• Discovery is refreshed and every administration route is negotiated.
• Mutating calls use direct worker URLs on a trusted network.
• One complete training iteration includes policy refresh and post-update generation.
• Request, worker, and update failures have a tested recovery path.
• Framework identity can be correlated with Dynamo telemetry without high-cardinality metric labels.
• The tested versions, environment, and topology are recorded with the integration.