> For clean Markdown of any page, append .md to the page URL.
> For a complete documentation index, see https://docs.nvidia.com/nemo/labs-voice-agent/llms.txt.
> For AI client integration (Claude Code, Cursor, etc.), connect to the MCP server at https://docs.nvidia.com/nemo/labs-voice-agent/_mcp/server.

# Serving with vLLM

With `llm.type: vllm`, NeMo Labs Voice Agent does not load the large language model (LLM) in-process. It
connects to an
OpenAI-compatible vLLM server over `llm.base_url`. This page covers who starts that server, which flags it
needs, and how to point the agent at one that is already running.

## Prerequisites

Before you start a vLLM deployment, complete the following preparation:

1. Install NeMo Labs Voice Agent by following the [Installation](/nemo/labs-voice-agent/get-started/installation) guide.
2. Activate the repository's Python environment.
3. Read the model sub-YAML you plan to use so the served model ID and flags match the agent configuration.

## The Model Sub-YAML Is the Source of Truth

Serving flags live in the model config that `llm.model_config` points at — not in `default.yaml`, and not in
this page. The sub-YAML **overrides** `default.yaml` for every `llm.*` key it sets, which is why the shipped
`llm.type: auto` ends up as `vllm`. Read
`examples/generic_voice_agent/server/server_configs/llm_configs/nemotron_nano_v3.yaml` before copying any
command from here. If the two disagree, the YAML wins.

Keys that matter for vLLM:

| Key | Purpose |
| --- | --- |
| `llm.type` | `auto`, `hf`, `vllm`, or `nvidia`. `auto` probes vLLM support for the model and falls back to Hugging Face. |
| `llm.model` | Sent as the `model` field on every request, so it must match the id vLLM serves. |
| `llm.base_url` | Where the agent sends completions. Shipped default: `http://localhost:8000/v1`. |
| `llm.api_key` | vLLM ignores it unless it was started with an API key; the configs ship `"EMPTY"`. |
| `llm.start_vllm_on_init` | `true` makes the agent spawn `vllm serve` at startup. `false` means you start it. |
| `llm.vllm_server_params` | A single string of CLI flags appended to `vllm serve <model>`. Only read when spawning. |
| `llm.vllm_generation_params` | Per-request sampling settings (OpenAI fields plus an `extra` dict). |
| `llm.dtype` | Prepended as `--dtype <value>` when spawning, unless the flag string already contains `dtype`. |
| `llm.device` | Read only by the `hf` backend. It is not passed to `VLLMService`, so it has no effect on the spawned vLLM process, as described below. |

## Who Starts vLLM

The `start_vllm_on_init` setting determines whether the voice-agent process owns the vLLM server lifecycle.

| `start_vllm_on_init` | Behavior | Ships with |
| --- | --- | --- |
| `false` | You run `vllm serve` in a separate terminal. The agent connects to `base_url`. | `nemotron_nano_v3.yaml`, `nemotron_nano_v3_think.yaml`, both `*_omni*.yaml`, and the eval configs under `evaluation/server_configs/` |
| `true` | `VLLMService.__init__` spawns `vllm serve` and blocks until the server reports the model. | `nemotron_nano_v2.yaml`, `qwen3-8B.yaml`, `qwen2.5-7B.yaml`, `llama3.1-8B-instruct.yaml`, `hf_llm_generic.yaml` |

Because the shipped default sets `false`, running `python server.py` by itself does not produce a working
agent. Start vLLM first.

## Start vLLM Yourself (Shipped Default)

These flags are the `vllm_server_params` string from `nemotron_nano_v3.yaml`, one flag per line:

```bash
source .venv/bin/activate
vllm serve nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B-NVFP4 \
    --trust-remote-code \
    --tensor-parallel-size 1 \
    --enable-prefix-caching \
    --max-num-seqs 1 \
    --gpu-memory-utilization 0.8 \
    --enable-auto-tool-choice \
    --tool-call-parser qwen3_coder \
    --reasoning-parser nemotron_v3
```

Wait until vLLM reports it is serving, then confirm the model ID that the agent requests:

```bash
curl -s http://localhost:8000/v1/models
```

Only then start the agent from `examples/generic_voice_agent/server/`.

Notes on individual flags:

- `--max-num-seqs 1` — the agent serves one client at a time. A second WebSocket connection is rejected with
  close code 1013, and the incumbent is kept. Extra sequence slots only cost memory.
- `--enable-auto-tool-choice --tool-call-parser qwen3_coder` — required for [tool calling](/nemo/labs-voice-agent/build-voice-agents/tools/tool-calling).
  Without them the model's tool calls arrive as plain text. Some models need a parser plugin file instead. Refer to
  [vLLM Plugins](/nemo/labs-voice-agent/build-voice-agents/model-serving/v-llm-plugins).
- `--reasoning-parser nemotron_v3` — strips reasoning content out of the response server-side. Refer to
  [Reasoning Mode](/nemo/labs-voice-agent/about/core-concepts/language-models/reasoning-mode).
- `--gpu-memory-utilization 0.8` leaves headroom on the same GPU for ASR, diarization, and TTS. Lower it if
  those models fail to allocate.

`nemotron_nano_v3_think.yaml` uses the **same** server flags, so one running server serves both configs. The
differences are all request-side: `enable_thinking: True` under
`vllm_generation_params.extra.extra_body.chat_template_kwargs`, `max_new_tokens` raised from 1024 to 4096, and a
new `thinking_budget: 2048` forwarded as `thinking_token_budget`.

## Let the Agent Start vLLM

Set `start_vllm_on_init: true` in the model sub-YAML. At startup `VLLMService` (in
`nemo_voice_agent/pipecat/services/nemo/llm.py`) does the following:

1. Takes the port from `base_url`. An explicit `--port` inside `vllm_server_params` overrides it with a warning.
2. Probes `/v1/models` on that port. If a server is already answering there with the **same** model ID, it
   reuses it and skips the spawn entirely.
3. Otherwise scans upward from that port for a free one, rewrites the base URL to
   `http://localhost:<port>/v1`, and runs `vllm serve <llm.model> <vllm_server_params>`.
4. Polls `/v1/models` every few seconds until the model appears, and raises if the child process dies or the
   wait exceeds its (non-configurable) one-hour cap.
5. Terminates the child process when the pipeline stops or is cancelled.

Two constraints follow from the implementation:

- **Local only.** The resolved URL is always `localhost`, so a remote `base_url` is discarded in this mode.
  Use `start_vllm_on_init: false` for a remote server.
- **No shell quoting.** `vllm_server_params` is split on whitespace, so any flag whose value contains a space
  (such as the omni `--limit-mm-per-prompt` JSON) is mangled. Start those models manually.

`llm.device` does **not** reach the spawned vLLM process: `get_llm_service_from_config` passes `device` only on
the `hf` path, so `VLLMService` always uses its built-in `cuda` default and never sets `CUDA_VISIBLE_DEVICES`.
To pin the spawned server to a specific GPU, export `CUDA_VISIBLE_DEVICES` in the shell that starts the agent
server, or start vLLM yourself with `start_vllm_on_init: false`.

## Point at an Already-Running vLLM

Set `base_url` to the server and leave `start_vllm_on_init: false`:

```yaml
# llm_configs/<your-model>.yaml
type: vllm
base_url: "http://gpu-host.internal:8000/v1"
api_key: "EMPTY"
start_vllm_on_init: false
```

Set `llm.model` (top level, in `default.yaml`) to exactly the ID returned by that server's `/v1/models`,
otherwise every request fails with a 404 from vLLM. Start the remote server with `--host 0.0.0.0` so it
accepts connections from outside its machine. For a hosted endpoint rather than your own vLLM, use
`llm.type: nvidia` instead. Refer to [NVIDIA NIM and Riva](/nemo/labs-voice-agent/build-voice-agents/model-serving/nvidia-nim-riva).

## Tensor Parallelism and VRAM

The shipped default is NVFP4 and fits one GPU with FP4 support at `--tensor-parallel-size 1`. For rough sizing,
allow about 21 GB for a 9B LLM and 13 GB for a 4B LLM. Also reserve a few GB for speech models that share the
GPU. `nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B-BF16` needs more than 60 GB, so raise
`--tensor-parallel-size` to 2 (and set `--max-model-len 8192` to keep the KV cache in budget).

Tuning order when you hit an out-of-memory error:

1. Lower `--gpu-memory-utilization`.
2. Lower `--max-model-len`.
3. Raise `--tensor-parallel-size` to spread across GPUs, or move ASR/diarization/TTS to a different GPU with
   their own `device:` keys.

## Omni (Audio-In) Recipe

`nemotron_nano_v3_omni.yaml` sets `is_omni_model: true`, which inserts the user-audio buffer into the
pipeline so raw audio reaches the LLM. Its `vllm_server_params` add the multimodal flags:

```bash
vllm serve nvidia/Nemotron-3-Nano-Omni-30B-A3B-Reasoning-NVFP4 \
    --trust-remote-code \
    --tensor-parallel-size 1 \
    --enable-prefix-caching \
    --max-num-seqs 1 \
    --gpu-memory-utilization 0.8 \
    --max-model-len 131072 \
    --video-pruning-rate 0.5 \
    --allowed-local-media-path / \
    --limit-mm-per-prompt '{"video": 999, "image": 999, "audio": 999}' \
    --media-io-kwargs '{"video": {"fps": 2, "num_frames": 256}}' \
    --enable-auto-tool-choice \
    --tool-call-parser qwen3_coder \
    --reasoning-parser nemotron_v3
```

Keep `start_vllm_on_init: false` here: the quoted JSON values would not survive the whitespace split
described above. Set `keep_only_last_audio_turn: false` against a self-hosted vLLM, which accepts multiple
audio turns. For the remaining omni keys, refer to
[Multimodal / Omni](/nemo/labs-voice-agent/about/core-concepts/language-models/multimodal-omni).

## Related Pages

Use these pages to configure the backend and understand the model behavior behind it:

- [LLM Backends](/nemo/labs-voice-agent/about/core-concepts/language-models/llm-backends) — choosing between `auto`, `hf`, `vllm`, and `nvidia`.
- [Server Configuration](/nemo/labs-voice-agent/build-voice-agents/configure/server-config) — how the sub-YAML merge works.
- [Troubleshooting](/nemo/labs-voice-agent/troubleshooting/troubleshooting) — connection and startup failures.