vLLM

vLLM is a popular LLM inference engine. The NeMo Gym VLLMModel server wraps vLLM’s Chat Completions endpoint and converts requests and responses to NeMo Gym’s native format, the OpenAI Responses API schema.

Most open-source models use Chat Completions format, while NeMo Gym uses the Responses API natively. VLLMModel bridges this gap by converting between the two formats automatically. For background on why NeMo Gym chose the Responses API and how the two schemas differ, see Responses API.

VLLMModel provides a Responses API to Chat Completions mapping middleware layer via responses_api_models/vllm_model. It assumes you are pointing to a vLLM instance since it relies on vLLM-specific endpoints like /tokenize and vLLM-specific arguments like return_tokens_as_token_ids.

Two upstream backends

VLLMModel can drive either of vLLM’s OpenAI-compatible endpoints, selected by the use_completions_api config flag. Both backends keep the same external Gym surface — /v1/responses and /v1/chat/completions continue to work identically; only the call to vLLM swaps.

To use VLLMModel, just change the responses_api_models/openai_model/configs/openai_model.yaml in your config paths to responses_api_models/vllm_model/configs/vllm_model.yaml!

gym env start \
    --resources-server example_multi_step \
    --model-type vllm_model

Use VLLMModel

Below is an e2e example of how to spin up a NeMo Gym compatible vLLM Chat Completions OpenAI server and run rollout collection with it. This section walks through starting a vLLM server manually and connecting NeMo Gym to it through responses_api_models/vllm_model. If you want NeMo Gym to manage the vLLM server lifecycle for you instead, see LocalVLLMModel.

Install vLLM

Please run the steps below in a separate terminal than your NeMo Gym terminal! The installation will take a few minutes.

uv venv --python 3.13.14 --seed .venv
source .venv/bin/activate
# hf_transfer for faster model download
uv pip install hf_transfer vllm --torch-backend=auto

# Discover the flashinfer-python version that vLLM installed
FI_VER=$(python -c "import flashinfer; print(flashinfer.__version__)")
echo "flashinfer-python: $FI_VER"
# Discover your torch CUDA tag (e.g. cu128, cu129, cu130)
TORCH_CU=$(python -c "import torch; print('cu' + torch.version.cuda.replace('.', ''))")
echo "torch CUDA tag: $TORCH_CU"
# Pre-compiled CUBINs (CUTLASS / TRTLLM-gen kernels)
uv pip install "flashinfer-cubin==${FI_VER}"
# Pre-built JIT object cache for your CUDA build
uv pip install "flashinfer-jit-cache==${FI_VER}" --index-url "https://flashinfer.ai/whl/${TORCH_CU}"

RuntimeError: Could not find nvcc and default cuda_home='/usr/local/cuda' doesn't exist

RuntimeError: flashinfer-jit-cache version (X.Y.Z+cuNNN) does not match flashinfer version (A.B.C)

Download the model

This download will take a few minutes.

# Qwen/Qwen3-4B-Thinking-2507, usable in Nemo RL!
HF_HOME=.cache/ \
HF_HUB_ENABLE_HF_TRANSFER=1 \
    hf download Qwen/Qwen3-4B-Thinking-2507

HF_TOKEN=... \
HF_HOME=.cache/ \
HF_HUB_ENABLE_HF_TRANSFER=1 \
    hf download Qwen/Qwen3-4B-Thinking-2507

Spin up a vLLM server

vLLM server configuration

• If you want to use tools, find the appropriate vLLM arguments regarding the tool call parser to use. In this example, we use Qwen/Qwen3-4B-Thinking-2507, which is suggested to use the hermes tool call parser.
• If you are using a reasoning model, find the appropriate vLLM arguments regarding reasoning parser to use. In this example, we use Qwen/Qwen3-4B-Thinking-2507, which is suggested to use the deepseek_r1 reasoning parser.
• The example below uses --tensor-parallel-size 1 which requires 1 GPU.

The spinup step will take a few minutes.

HF_HOME=.cache/ \
HOME=. \
vllm serve \
    Qwen/Qwen3-4B-Thinking-2507 \
    --tensor-parallel-size 4 \
    --enable-auto-tool-choice --tool-call-parser hermes \
    --reasoning-parser deepseek_r1 \
    --host 0.0.0.0 \
    --port 10240

Configure NeMo Gym to use the local vLLM server

In a second terminal on the same GPU node that was used to spin up the vLLM server, enter the NeMo Gym Python environment, and start the NeMo Gym servers.

gym env start \
    --resources-server example_multi_step \
    --model-type vllm_model \
    --model-url http://0.0.0.0:10240/v1 \
    --model Qwen/Qwen3-4B-Thinking-2507 \
    --model-api-key dummy_key

hostname -i

Run rollout collection

In a third terminal on the same GPU node that was used to spin up the vLLM server, enter the NeMo Gym Python environment, and run rollout collection.

gym eval run --no-serve \
    --agent example_multi_step_simple_agent \
    --input resources_servers/example_multi_step/data/example.jsonl \
    --output results/example_multi_step_rollouts.jsonl

The /v1/completions backend

Set use_completions_api: true to drive vLLM’s text-completions endpoint instead of chat completions. A ready-to-use config ships at responses_api_models/vllm_model/configs/vllm_model_completions.yaml:

policy_model:
  responses_api_models:
    vllm_model:
      entrypoint: app.py
      base_url: ${policy_base_url}
      api_key: ${policy_api_key}
      model: ${policy_model_name}
      return_token_id_information: false
      uses_reasoning_parser: false   # /v1/completions does not run vLLM's reasoning parser
      uses_interleaved_reasoning: false
      use_completions_api: true      # drives vLLM /v1/completions

Render modes

A second flag, render_chat_template, picks how the caller’s messages list becomes the prompt string:

• render_chat_template: false (default) — raw render. Forwards bytes verbatim. Required input: a single user message, optionally preceded by a single system message; their content is joined with \n\n and sent as prompt. tools, multi-turn turns, and non-text content blocks are rejected. This is the cheapest path and the most common base-model setup.
• render_chat_template: true — chat-template render. Renders the messages list to a prompt string client-side via HF AutoTokenizer.apply_chat_template(messages, tokenize=False, add_generation_prompt=True, tools=..., **chat_template_kwargs). Multi-turn assistant / tool turns and tools are allowed.

For Gym’s /v1/responses endpoint, a string input is forwarded as the prompt directly under raw mode; under chat-template mode the converter still wraps it as a single user message before rendering.

tokenizer config (chat-template mode)

When render_chat_template: true, the HF tokenizer is loaded once at server startup. By default it’s loaded from the same identifier as model. Override with tokenizer: for base-model setups where the model checkpoint has no chat template in its tokenizer config — point tokenizer: at a different model whose template you want to inherit:

model: Qwen/Qwen2.5-7B           # base model, no chat template
use_completions_api: true
render_chat_template: true
tokenizer: Qwen/Qwen2.5-7B-Instruct  # borrow the instruct chat template

If the loaded tokenizer has no chat_template, the server fails at startup — silent fallback to raw mode would mask a config bug. The transformers package is required at runtime for chat-template mode (it’s a Gym dep, but if it’s missing the startup error message points at it explicitly).

Constraint matrix

Tools in chat-template mode. The chat template renders the tool definitions into the prompt (Hermes / Qwen / etc. all do this), but /v1/completions doesn’t run vLLM’s tool-call parser regardless of how the prompt is built — so any tool-call output text the model emits is not parsed by Gym. The caller is responsible for parsing tool calls out of the assistant text. If you need first-class tool-call routing, use use_completions_api: false.

<think>...</think> blocks emitted inline by the model are extracted downstream by ChatCompletionsConverter._extract_reasoning_from_content when results are converted back to a Response, exactly as on the chat path.

Token-ID extraction (RL training)

When return_token_id_information: true, the completions backend automatically adds logprobs: 0, return_token_ids: true, and return_tokens_as_token_ids: true to outbound requests. It reads prompt and generation token IDs directly from each choice. For older vLLM versions that omit inline IDs, it falls back to /tokenize for the prompt and the "token_id:<int>" entries in logprobs.tokens for the generation.

Use it

The end-to-end flow is identical to the Use VLLMModel walkthrough above; just swap the model-server config:

gym env start \
    --config resources_servers/example_multi_step/configs/example_multi_step.yaml \
    --config responses_api_models/vllm_model/configs/vllm_model_completions.yaml \
    --model-url http://0.0.0.0:10240/v1 \
    --model <your-model> \
    --model-api-key dummy_key
gym eval run --no-serve \
    --agent <your_agent> \
    --input <your_data.jsonl> \
    --output results/rollouts_completions.jsonl

Make sure your input JSONL respects the selected mode in the Constraint matrix above.

VLLMModel configuration reference

Advanced: chat_template_kwargs

Override chat template behavior for specific models:

chat_template_kwargs:
  enable_thinking: false  # Model-specific

Per-request values can come from two places, merged over the configured value in this order (later wins):

1. A request’s top-level chat_template_kwargs (for example, OpenAI SDK clients that pass it in extra_body). Applied only when forward_request_chat_template_kwargs: true; otherwise it is dropped and a warning is logged once per server. Forwarding changes generation for agents that send it on every call, such as Stirrup’s enable_thinking, so re-run baselines when you enable it.
2. metadata.chat_template_kwargs, a JSON-encoded object on the request.

With use_completions_api: true, only the configured value and metadata.chat_template_kwargs apply. Streaming (stream: true) requests never forward the top-level field.

Advanced: extra_body

Pass vLLM-specific parameters not in the standard OpenAI API:

extra_body:
  guided_json: '{"type": "object", "properties": {...}}'
  min_tokens: 10
  repetition_penalty: 1.1

The shipped vllm_model configs leave chat_template_kwargs, sampling_overrides, and extra_body unset, so a value a benchmark or environment config sets for policy_model survives --model-type vllm_model, whose config merges after the benchmark’s. A null there would replace it.

Bound retries to a dead model server

By default a model call retries a refused or dropped connection without a time limit, so a training run can ride out an engine restart. For evaluation, where a dead engine should fail the run, set a cap:

endpoint_connection_retry_timeout_s: 300

The value must be positive. The window starts at the call’s first connection error and closes when the server answers, so it bounds only an unbroken run of connection failures. A call that has connected and is generating is not cut short, and HTTP-status retries (for example a 503 with Retry-After) keep their own limits. Each model call gets its own window, so a caller that retries the failed call (for example the OpenAI SDK’s max_retries in an agent’s policy client) waits up to one window per retry. Set the cap above the model server’s startup time if rollouts can start before the engine accepts connections. null adds no time cap, but does not lift the attempt cap that endpoint_connection_retries sets when endpoint_file or route_around_failing_endpoints is used.

Use VLLMModel with multiple replicas of a model endpoint

The vLLM model server supports multiple endpoints for horizontal scaling:

base_url:
  - http://gpu-node-1:8000/v1
  - http://gpu-node-2:8000/v1
  - http://gpu-node-3:8000/v1

How it works:

1. Initial assignment: New sessions are assigned to endpoints using round-robin (session 1 → endpoint 1, session 2 → endpoint 2, etc.)
2. Session affinity: Once assigned, a session always uses the same endpoint (tracked via HTTP session cookies)
3. Why affinity? Multi-turn conversations and agentic workflows that call the model multiple times in one trajectory need to hit the same model endpoint in order to hit the prefix cache, which significantly speeds up the prefill phase of model inference.

Context Length Exceeded Handling

When a conversation exceeds the vLLM model’s maximum context length (max_seq_length), VLLMModel handles the error gracefully instead of crashing the entire rollout collection.

How it works

1. vLLM rejects the request: vLLM returns an HTTP 400 error with a message like "This model's maximum context length is 32768 tokens. However, you requested 32818 tokens...".
2. VLLMModel catches the error: Instead of propagating the exception, VLLMModel returns an empty response with finish_reason: "length".
3. Responses API mapping: The finish_reason: "length" is converted to incomplete_details: { reason: "max_output_tokens" } in the Responses API response returned to the agent.

This is particularly important for multi-turn agentic rollouts where conversation length can grow unpredictably across tool-call turns.

How to detect truncated responses

Downstream consumers (agents, RL training frameworks) can check the incomplete_details field on the response:

response = await model_server.responses(request, body)
if response.incomplete_details and response.incomplete_details["reason"] == "max_output_tokens":
    # The conversation exceeded the model's context window.
    # The response output will be empty — handle accordingly.
    pass

Implications for training

When using NeMo Gym with NeMo RL or another training framework, responses with incomplete_details.reason == "max_output_tokens" indicate that the full conversation (prompt + prior generations) exceeded max_seq_length. Training frameworks should filter or handle these responses appropriately since they contain no generated tokens.

Training vs Offline Inference

By default, VLLMModel will not track any token IDs explicitly. However, token IDs are necessary when using NeMo Gym in conjunction with a training framework in order to train a model. For training workflows, use the training-dedicated config which enables token ID tracking:

# Use vllm_model_for_training.yaml
return_token_id_information: true

This enables:

• prompt_token_ids: Token IDs for the input prompt
• generation_token_ids: Token IDs for generated text
• generation_log_probs: Log probabilities for each generated token