> This page is for version Nightly (default). > For other versions, use one of these documentation indexes: > - Nightly (default): https://docs.nvidia.com/nemo/labs-voice-agent/nightly/llms.txt > For clean Markdown of any page, append .md to the page URL. > For a complete documentation index, see https://docs.nvidia.com/nemo/labs-voice-agent/llms.txt. > For AI client integration (Claude Code, Cursor, etc.), connect to the MCP server at https://docs.nvidia.com/nemo/labs-voice-agent/_mcp/server. # Serving with vLLM With `llm.type: vllm`, NeMo Labs Voice Agent does not load the large language model (LLM) in-process. It connects to an OpenAI-compatible vLLM server over `llm.base_url`. This page covers who starts that server, which flags it needs, and how to point the agent at one that is already running. ## Prerequisites Before you start a vLLM deployment, complete the following preparation: 1. Install NeMo Labs Voice Agent by following the [Installation](/nemo/labs-voice-agent/get-started/installation) guide. 2. Activate the repository's Python environment. 3. Read the model sub-YAML you plan to use so the served model ID and flags match the agent configuration. ## The Model Sub-YAML Is the Source of Truth Serving flags live in the model config that `llm.model_config` points at — not in `default.yaml`, and not in this page. The sub-YAML **overrides** `default.yaml` for every `llm.*` key it sets, which is why the shipped `llm.type: auto` ends up as `vllm`. Read `examples/generic_voice_agent/server/server_configs/llm_configs/nemotron_3.5_lightning.yaml` before copying any command from here. If the two disagree, the YAML wins. Keys that matter for vLLM: | Key | Purpose | | --- | --- | | `llm.type` | `auto`, `hf`, `vllm`, or `nvidia`. `auto` probes vLLM support for the model and falls back to Hugging Face. | | `llm.model` | Sent as the `model` field on every request, so it must match the id vLLM serves. | | `llm.base_url` | Where the agent sends completions. Shipped default: `http://localhost:8000/v1`. | | `llm.api_key` | vLLM ignores it unless it was started with an API key; the configs ship `"EMPTY"`. | | `llm.start_vllm_on_init` | `true` makes the agent spawn `vllm serve` at startup. `false` means you start it. | | `llm.vllm_server_params` | A single string of CLI flags appended to `vllm serve `. Only read when spawning. | | `llm.vllm_generation_params` | Per-request sampling settings (OpenAI fields plus an `extra` dict). | | `llm.dtype` | Prepended as `--dtype ` when spawning, unless the flag string already contains `dtype`. | | `llm.device` | Read only by the `hf` backend. It is not passed to `VLLMService`, so it has no effect on the spawned vLLM process, as described below. | ## Who Starts vLLM The `start_vllm_on_init` setting determines whether the voice-agent process owns the vLLM server lifecycle. | `start_vllm_on_init` | Behavior | Ships with | | --- | --- | --- | | `false` | You run `vllm serve` in a separate terminal. The agent connects to `base_url`. | `nemotron_3.5_lightning.yaml`, `nemotron_3.5_lightning_think.yaml`, `nemotron_nano_v3.yaml`, `nemotron_nano_v3_think.yaml`, both `*_omni*.yaml`, and the eval configs under `evaluation/server_configs/` | | `true` | `VLLMService.__init__` spawns `vllm serve` and blocks until the server reports the model. | `nemotron_nano_v2.yaml`, `qwen3-8B.yaml`, `qwen2.5-7B.yaml`, `llama3.1-8B-instruct.yaml`, `hf_llm_generic.yaml` | Because the shipped default sets `false`, running `python server.py` by itself does not produce a working agent. Start vLLM first. ## Start vLLM Yourself (Shipped Default) These flags are the `vllm_server_params` string from `nemotron_3.5_lightning.yaml`, one flag per line: ```bash source .venv/bin/activate vllm serve nvidia/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-NVFP4 \ --trust-remote-code \ --tensor-parallel-size 1 \ --enable-prefix-caching \ --max-num-seqs 1 \ --gpu-memory-utilization 0.8 \ --enable-auto-tool-choice \ --tool-call-parser qwen3_coder \ --reasoning-parser nemotron_v3 ``` Wait until vLLM reports it is serving, then confirm the model ID that the agent requests: ```bash curl -s http://localhost:8000/v1/models ``` Only then start the agent from `examples/generic_voice_agent/server/`. Notes on individual flags: - `--max-num-seqs 1` — the agent serves one client at a time. A second WebSocket connection is rejected with close code 1013, and the incumbent is kept. Extra sequence slots only cost memory. - `--enable-auto-tool-choice --tool-call-parser qwen3_coder` — required for [tool calling](/nemo/labs-voice-agent/build-voice-agents/tools/tool-calling). Without them the model's tool calls arrive as plain text. Some models need a parser plugin file instead. Refer to [vLLM Plugins](/nemo/labs-voice-agent/build-voice-agents/model-serving/v-llm-plugins). - `--reasoning-parser nemotron_v3` — strips reasoning content out of the response server-side, and supplies the reasoning delimiters that a thinking budget forces once it runs out. Refer to [Reasoning Mode](/nemo/labs-voice-agent/about/core-concepts/language-models/reasoning-mode). - `--gpu-memory-utilization 0.8` leaves headroom on the same GPU for ASR, diarization, and TTS. Lower it if those models fail to allocate. vLLM derives the reasoning delimiters from the parser, so no shipped config sets them. Override both with `--reasoning-config` when a checkpoint closes its reasoning span with a non-standard string: ```bash vllm serve nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B-BF16 \ --reasoning-parser nemotron_v3 \ --reasoning-config '{"reasoning_start_str": "", "reasoning_end_str": "\nI have to finalize the answer now."}' ``` The Nemotron-3 model cards do not ask for this flag. Add it only when a manual run leaves reasoning text in the spoken response or ignores the thinking budget. `nemotron_3.5_lightning_think.yaml` (and, for the older model family, `nemotron_nano_v3_think.yaml`) uses the **same** server flags, so one running server serves both configs. The differences are all request-side: `enable_thinking: True` under `vllm_generation_params.extra.extra_body.chat_template_kwargs`, `max_new_tokens` raised from 1024 to 4096, and a new `thinking_budget: 2048` forwarded as `thinking_token_budget`. ## Let the Agent Start vLLM Set `start_vllm_on_init: true` in the model sub-YAML. At startup `VLLMService` (in `nemo_voice_agent/pipecat/services/nemo/llm.py`) does the following: 1. Takes the port from `base_url`. An explicit `--port` inside `vllm_server_params` overrides it with a warning. 2. Probes `/v1/models` on that port. If a server is already answering there with the **same** model ID, it reuses it and skips the spawn entirely. 3. Otherwise scans upward from that port for a free one, rewrites the base URL to `http://localhost:/v1`, and runs `vllm serve `. 4. Polls `/v1/models` every few seconds until the model appears, and raises if the child process dies or the wait exceeds its (non-configurable) one-hour cap. 5. Terminates the child process when the pipeline stops or is cancelled. Two constraints follow from the implementation: - **Local only.** The resolved URL is always `localhost`, so a remote `base_url` is discarded in this mode. Use `start_vllm_on_init: false` for a remote server. - **No shell quoting.** `vllm_server_params` is split on whitespace, so any flag whose value contains a space (such as the omni `--limit-mm-per-prompt` JSON) is mangled. Start those models manually. `llm.device` does **not** reach the spawned vLLM process: `get_llm_service_from_config` passes `device` only on the `hf` path, so `VLLMService` always uses its built-in `cuda` default and never sets `CUDA_VISIBLE_DEVICES`. To pin the spawned server to a specific GPU, export `CUDA_VISIBLE_DEVICES` in the shell that starts the agent server, or start vLLM yourself with `start_vllm_on_init: false`. ## Point at an Already-Running vLLM Set `base_url` to the server and leave `start_vllm_on_init: false`: ```yaml # llm_configs/.yaml type: vllm base_url: "http://gpu-host.internal:8000/v1" api_key: "EMPTY" start_vllm_on_init: false ``` Set `llm.model` (top level, in `default.yaml`) to exactly the ID returned by that server's `/v1/models`, otherwise every request fails with a 404 from vLLM. Start the remote server with `--host 0.0.0.0` so it accepts connections from outside its machine. For a hosted endpoint rather than your own vLLM, use `llm.type: nvidia` instead. Refer to [NVIDIA NIM and Riva](/nemo/labs-voice-agent/build-voice-agents/model-serving/nvidia-nim-riva). ## Tensor Parallelism and VRAM The shipped default is NVFP4 and fits one GPU with FP4 support at `--tensor-parallel-size 1`. For rough sizing, allow about 21 GB for a 9B LLM and 13 GB for a 4B LLM. Also reserve a few GB for speech models that share the GPU. `nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B-BF16` needs more than 60 GB, so raise `--tensor-parallel-size` to 2 (and set `--max-model-len 8192` to keep the KV cache in budget). On a GPU that supports FP8 but not FP4, serve [`nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B-FP8`](https://huggingface.co/nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B-FP8) instead of the BF16 weights: point `llm.model` at that repository, keep the rest of the config as shipped, and the VRAM requirement drops well below the BF16 figure. Tuning order when you hit an out-of-memory error: 1. Lower `--gpu-memory-utilization`. 2. Lower `--max-model-len`. 3. Raise `--tensor-parallel-size` to spread across GPUs, or move ASR/diarization/TTS to a different GPU with their own `device:` keys. ## Omni (Audio-In) Recipe `nemotron_nano_v3_omni.yaml` sets `is_omni_model: true`, which inserts the user-audio buffer into the pipeline so raw audio reaches the LLM. Its `vllm_server_params` add the multimodal flags: ```bash vllm serve nvidia/Nemotron-3-Nano-Omni-30B-A3B-Reasoning-NVFP4 \ --trust-remote-code \ --tensor-parallel-size 1 \ --enable-prefix-caching \ --max-num-seqs 1 \ --gpu-memory-utilization 0.8 \ --max-model-len 131072 \ --video-pruning-rate 0.5 \ --allowed-local-media-path / \ --limit-mm-per-prompt '{"video": 999, "image": 999, "audio": 999}' \ --media-io-kwargs '{"video": {"fps": 2, "num_frames": 256}}' \ --enable-auto-tool-choice \ --tool-call-parser qwen3_coder \ --reasoning-parser nemotron_v3 ``` Keep `start_vllm_on_init: false` here: the quoted JSON values would not survive the whitespace split described above. Because you type this command yourself, you can also add `--kv-cache-dtype fp8`, which the [model card](https://huggingface.co/nvidia/Nemotron-3-Nano-Omni-30B-A3B-Reasoning-BF16) lists for the quantized checkpoints to shrink the KV cache. Omit that flag for the BF16 checkpoint. Set `keep_only_last_audio_turn: false` against a self-hosted vLLM, which accepts multiple audio turns. For the remaining omni keys, refer to [Multimodal / Omni](/nemo/labs-voice-agent/about/core-concepts/language-models/multimodal-omni). ## Related Pages Use these pages to configure the backend and understand the model behavior behind it: - [LLM Backends](/nemo/labs-voice-agent/about/core-concepts/language-models/llm-backends) — choosing between `auto`, `hf`, `vllm`, and `nvidia`. - [Server Configuration](/nemo/labs-voice-agent/build-voice-agents/configure/server-config) — how the sub-YAML merge works. - [Troubleshooting](/nemo/labs-voice-agent/troubleshooting/troubleshooting) — connection and startup failures.