vLLM

vLLM is a popular LLM inference engine. The NeMo Gym VLLMModel server wraps vLLM’s Chat Completions endpoint and converts requests and responses to NeMo Gym’s native format, the OpenAI Responses API schema.

Most open-source models use Chat Completions format, while NeMo Gym uses the Responses API natively. VLLMModel bridges this gap by converting between the two formats automatically. For background on why NeMo Gym chose the Responses API and how the two schemas differ, see responses-api-evolution.

VLLMModel provides a Responses API to Chat Completions mapping middleware layer via responses_api_models/vllm_model. It assumes you are pointing to a vLLM instance since it relies on vLLM-specific endpoints like /tokenize and vLLM-specific arguments like return_tokens_as_token_ids.

Two upstream backends

VLLMModel can drive either of vLLM’s OpenAI-compatible endpoints, selected by the use_completions_api config flag. Both backends keep the same external Gym surface — /v1/responses and /v1/chat/completions continue to work identically; only the call to vLLM swaps.

To use VLLMModel, just change the responses_api_models/openai_model/configs/openai_model.yaml in your config paths to responses_api_models/vllm_model/configs/vllm_model.yaml!

$
gym env start \
>
    --resources-server example_multi_step \
>
    --model-type vllm_model

Use VLLMModel

Below is an e2e example of how to spin up a NeMo Gym compatible vLLM Chat Completions OpenAI server and run rollout collection with it. This section walks through starting a vLLM server manually and connecting NeMo Gym to it through responses_api_models/vllm_model. If you want NeMo Gym to manage the vLLM server lifecycle for you instead, see LocalVLLMModel.

Install vLLM

Please run the steps below in a separate terminal than your NeMo Gym terminal! The installation will take a few minutes.

$
uv venv --python 3.12 --seed .venv
$
source .venv/bin/activate
$
# hf_transfer for faster model download
$
uv pip install hf_transfer vllm --torch-backend=auto

$
# Discover the flashinfer-python version that vLLM installed
$
FI_VER=$(python -c "import flashinfer; print(flashinfer.__version__)")
$
echo "flashinfer-python: $FI_VER"
$
$
# Discover your torch CUDA tag (e.g. cu128, cu129, cu130)
$
TORCH_CU=$(python -c "import torch; print('cu' + torch.version.cuda.replace('.', ''))")
$
echo "torch CUDA tag: $TORCH_CU"
$
$
# Pre-compiled CUBINs (CUTLASS / TRTLLM-gen kernels)
$
uv pip install "flashinfer-cubin==${FI_VER}"
$
$
# Pre-built JIT object cache for your CUDA build
$
uv pip install "flashinfer-jit-cache==${FI_VER}" --index-url "https://flashinfer.ai/whl/${TORCH_CU}"

RuntimeError: Could not find nvcc and default cuda_home='/usr/local/cuda' doesn't exist

RuntimeError: flashinfer-jit-cache version (X.Y.Z+cuNNN) does not match flashinfer version (A.B.C)

Download the model

This download will take a few minutes.

$
# Qwen/Qwen3-4B-Thinking-2507, usable in Nemo RL!
$
HF_HOME=.cache/ \
>
HF_HUB_ENABLE_HF_TRANSFER=1 \
>
    hf download Qwen/Qwen3-4B-Thinking-2507

$
HF_TOKEN=... \
>
HF_HOME=.cache/ \
>
HF_HUB_ENABLE_HF_TRANSFER=1 \
>
    hf download Qwen/Qwen3-4B-Thinking-2507

Spin up a vLLM server

vLLM server configuration

• If you want to use tools, find the appropriate vLLM arguments regarding the tool call parser to use. In this example, we use Qwen/Qwen3-4B-Thinking-2507, which is suggested to use the hermes tool call parser.
• If you are using a reasoning model, find the appropriate vLLM arguments regarding reasoning parser to use. In this example, we use Qwen/Qwen3-4B-Thinking-2507, which is suggested to use the deepseek_r1 reasoning parser.
• The example below uses --tensor-parallel-size 1 which requires 1 GPU.

The spinup step will take a few minutes.

$
HF_HOME=.cache/ \
>
HOME=. \
>
vllm serve \
>
    Qwen/Qwen3-4B-Thinking-2507 \
>
    --tensor-parallel-size 4 \
>
    --enable-auto-tool-choice --tool-call-parser hermes \
>
    --reasoning-parser deepseek_r1 \
>
    --host 0.0.0.0 \
>
    --port 10240

Configure NeMo Gym to use the local vLLM server

In a second terminal on the same GPU node that was used to spin up the vLLM server, enter the NeMo Gym Python environment, and start the NeMo Gym servers.

$
gym env start \
>
    --resources-server example_multi_step \
>
    --model-type vllm_model \
>
    --model-url http://0.0.0.0:10240/v1 \
>
    --model Qwen/Qwen3-4B-Thinking-2507 \
>
    --model-api-key dummy_key

$
hostname -i

Run rollout collection

In a third terminal on the same GPU node that was used to spin up the vLLM server, enter the NeMo Gym Python environment, and run rollout collection.

$
gym eval run --no-serve \
>
    --agent example_multi_step_simple_agent \
>
    --input resources_servers/example_multi_step/data/example.jsonl \
>
    --output results/example_multi_step_rollouts.jsonl

The /v1/completions backend

Set use_completions_api: true to drive vLLM’s text-completions endpoint instead of chat completions. A ready-to-use config ships at responses_api_models/vllm_model/configs/vllm_model_completions.yaml:

1
policy_model:
2
  responses_api_models:
3
    vllm_model:
4
      entrypoint: app.py
5
      base_url: ${policy_base_url}
6
      api_key: ${policy_api_key}
7
      model: ${policy_model_name}
8
      return_token_id_information: false
9
      uses_reasoning_parser: false   # /v1/completions does not run vLLM's reasoning parser
10
      uses_interleaved_reasoning: false
11
      use_completions_api: true      # drives vLLM /v1/completions
12
      chat_template_kwargs: null
13
      extra_body: null

Render modes

A second flag, render_chat_template, picks how the caller’s messages list becomes the prompt string:

• render_chat_template: false (default) — raw render. Forwards bytes verbatim. Required input: a single user message, optionally preceded by a single system message; their content is joined with \n\n and sent as prompt. tools, multi-turn turns, and non-text content blocks are rejected. This is the cheapest path and the most common base-model setup.
• render_chat_template: true — chat-template render. Renders the messages list to a prompt string client-side via HF AutoTokenizer.apply_chat_template(messages, tokenize=False, add_generation_prompt=True, tools=..., **chat_template_kwargs). Multi-turn assistant / tool turns and tools are allowed.

For Gym’s /v1/responses endpoint, a string input is forwarded as the prompt directly under raw mode; under chat-template mode the converter still wraps it as a single user message before rendering.

tokenizer config (chat-template mode)

When render_chat_template: true, the HF tokenizer is loaded once at server startup. By default it’s loaded from the same identifier as model. Override with tokenizer: for base-model setups where the model checkpoint has no chat template in its tokenizer config — point tokenizer: at a different model whose template you want to inherit:

1
model: Qwen/Qwen2.5-7B           # base model, no chat template
2
use_completions_api: true
3
render_chat_template: true
4
tokenizer: Qwen/Qwen2.5-7B-Instruct  # borrow the instruct chat template

If the loaded tokenizer has no chat_template, the server fails at startup — silent fallback to raw mode would mask a config bug. The transformers package is required at runtime for chat-template mode (it’s a Gym dep, but if it’s missing the startup error message points at it explicitly).

Constraint matrix

Tools in chat-template mode. The chat template renders the tool definitions into the prompt (Hermes / Qwen / etc. all do this), but /v1/completions doesn’t run vLLM’s tool-call parser regardless of how the prompt is built — so any tool-call output text the model emits is not parsed by Gym. The caller is responsible for parsing tool calls out of the assistant text. If you need first-class tool-call routing, use use_completions_api: false.

<think>...</think> blocks emitted inline by the model are extracted downstream by VLLMConverter._extract_reasoning_from_content when results are converted back to a Response, exactly as on the chat path.

Token-ID extraction (RL training)

When return_token_id_information: true, the completions backend automatically adds logprobs: 0, return_token_ids: true, and return_tokens_as_token_ids: true to outbound requests. It reads prompt and generation token IDs directly from each choice. For older vLLM versions that omit inline IDs, it falls back to /tokenize for the prompt and the "token_id:<int>" entries in logprobs.tokens for the generation.

Use it

The end-to-end flow is identical to the Use VLLMModel walkthrough above; just swap the model-server config:

$
gym env start \
>
    --config resources_servers/example_multi_step/configs/example_multi_step.yaml \
>
    --config responses_api_models/vllm_model/configs/vllm_model_completions.yaml \
>
    --model-url http://0.0.0.0:10240/v1 \
>
    --model <your-model> \
>
    --model-api-key dummy_key
$
$
gym eval run --no-serve \
>
    --agent <your_agent> \
>
    --input <your_data.jsonl> \
>
    --output results/rollouts_completions.jsonl

Make sure your input JSONL respects the selected mode in the Constraint matrix above.

VLLMModel configuration reference

Advanced: chat_template_kwargs

Override chat template behavior for specific models:

1
chat_template_kwargs:
2
  enable_thinking: false  # Model-specific

Advanced: extra_body

Pass vLLM-specific parameters not in the standard OpenAI API:

1
extra_body:
2
  guided_json: '{"type": "object", "properties": {...}}'
3
  min_tokens: 10
4
  repetition_penalty: 1.1

Use VLLMModel with multiple replicas of a model endpoint

The vLLM model server supports multiple endpoints for horizontal scaling:

1
base_url:
2
  - http://gpu-node-1:8000/v1
3
  - http://gpu-node-2:8000/v1
4
  - http://gpu-node-3:8000/v1

How it works:

1. Initial assignment: New sessions are assigned to endpoints using round-robin (session 1 → endpoint 1, session 2 → endpoint 2, etc.)
2. Session affinity: Once assigned, a session always uses the same endpoint (tracked via HTTP session cookies)
3. Why affinity? Multi-turn conversations and agentic workflows that call the model multiple times in one trajectory need to hit the same model endpoint in order to hit the prefix cache, which significantly speeds up the prefill phase of model inference.

Context Length Exceeded Handling

When a conversation exceeds the vLLM model’s maximum context length (max_seq_length), VLLMModel handles the error gracefully instead of crashing the entire rollout collection.

How it works

1. vLLM rejects the request: vLLM returns an HTTP 400 error with a message like "This model's maximum context length is 32768 tokens. However, you requested 32818 tokens...".
2. VLLMModel catches the error: Instead of propagating the exception, VLLMModel returns an empty response with finish_reason: "length".
3. Responses API mapping: The finish_reason: "length" is converted to incomplete_details: { reason: "max_output_tokens" } in the Responses API response returned to the agent.

This is particularly important for multi-turn agentic rollouts where conversation length can grow unpredictably across tool-call turns.

How to detect truncated responses

Downstream consumers (agents, RL training frameworks) can check the incomplete_details field on the response:

1
response = await model_server.responses(request, body)
2
3
if response.incomplete_details and response.incomplete_details["reason"] == "max_output_tokens":
4
    # The conversation exceeded the model's context window.
5
    # The response output will be empty — handle accordingly.
6
    pass

Implications for training

When using NeMo Gym with NeMo RL or another training framework, responses with incomplete_details.reason == "max_output_tokens" indicate that the full conversation (prompt + prior generations) exceeded max_seq_length. Training frameworks should filter or handle these responses appropriately since they contain no generated tokens.

Training vs Offline Inference

By default, VLLMModel will not track any token IDs explicitly. However, token IDs are necessary when using NeMo Gym in conjunction with a training framework in order to train a model. For training workflows, use the training-dedicated config which enables token ID tracking:

1
# Use vllm_model_for_training.yaml
2
return_token_id_information: true

This enables:

• prompt_token_ids: Token IDs for the input prompt
• generation_token_ids: Token IDs for generated text
• generation_log_probs: Log probabilities for each generated token