Build Voice AgentsModel Serving

Serving with vLLM

View as Markdown

With llm.type: vllm, NeMo Labs Voice Agent does not load the large language model (LLM) in-process. It connects to an OpenAI-compatible vLLM server over llm.base_url. This page covers who starts that server, which flags it needs, and how to point the agent at one that is already running.

Prerequisites

Before you start a vLLM deployment, complete the following preparation:

  1. Install NeMo Labs Voice Agent by following the Installation guide.
  2. Activate the repository’s Python environment.
  3. Read the model sub-YAML you plan to use so the served model ID and flags match the agent configuration.

The Model Sub-YAML Is the Source of Truth

Serving flags live in the model config that llm.model_config points at — not in default.yaml, and not in this page. The sub-YAML overrides default.yaml for every llm.* key it sets, which is why the shipped llm.type: auto ends up as vllm. Read examples/generic_voice_agent/server/server_configs/llm_configs/nemotron_3.5_lightning.yaml before copying any command from here. If the two disagree, the YAML wins.

Keys that matter for vLLM:

KeyPurpose
llm.typeauto, hf, vllm, or nvidia. auto probes vLLM support for the model and falls back to Hugging Face.
llm.modelSent as the model field on every request, so it must match the id vLLM serves.
llm.base_urlWhere the agent sends completions. Shipped default: http://localhost:8000/v1.
llm.api_keyvLLM ignores it unless it was started with an API key; the configs ship "EMPTY".
llm.start_vllm_on_inittrue makes the agent spawn vllm serve at startup. false means you start it.
llm.vllm_server_paramsA single string of CLI flags appended to vllm serve <model>. Only read when spawning.
llm.vllm_generation_paramsPer-request sampling settings (OpenAI fields plus an extra dict).
llm.dtypePrepended as --dtype <value> when spawning, unless the flag string already contains dtype.
llm.deviceRead only by the hf backend. It is not passed to VLLMService, so it has no effect on the spawned vLLM process, as described below.

Who Starts vLLM

The start_vllm_on_init setting determines whether the voice-agent process owns the vLLM server lifecycle.

start_vllm_on_initBehaviorShips with
falseYou run vllm serve in a separate terminal. The agent connects to base_url.nemotron_3.5_lightning.yaml, nemotron_3.5_lightning_think.yaml, nemotron_nano_v3.yaml, nemotron_nano_v3_think.yaml, both *_omni*.yaml, and the eval configs under evaluation/server_configs/
trueVLLMService.__init__ spawns vllm serve and blocks until the server reports the model.nemotron_nano_v2.yaml, qwen3-8B.yaml, qwen2.5-7B.yaml, llama3.1-8B-instruct.yaml, hf_llm_generic.yaml

Because the shipped default sets false, running python server.py by itself does not produce a working agent. Start vLLM first.

Start vLLM Yourself (Shipped Default)

These flags are the vllm_server_params string from nemotron_3.5_lightning.yaml, one flag per line:

$source .venv/bin/activate
$vllm serve nvidia/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-NVFP4 \
>    --trust-remote-code \
>    --tensor-parallel-size 1 \
>    --enable-prefix-caching \
>    --max-num-seqs 1 \
>    --gpu-memory-utilization 0.8 \
>    --enable-auto-tool-choice \
>    --tool-call-parser qwen3_coder \
>    --reasoning-parser nemotron_v3

Wait until vLLM reports it is serving, then confirm the model ID that the agent requests:

$curl -s http://localhost:8000/v1/models

Only then start the agent from examples/generic_voice_agent/server/.

Notes on individual flags:

  • --max-num-seqs 1 — the agent serves one client at a time. A second WebSocket connection is rejected with close code 1013, and the incumbent is kept. Extra sequence slots only cost memory.
  • --enable-auto-tool-choice --tool-call-parser qwen3_coder — required for tool calling. Without them the model’s tool calls arrive as plain text. Some models need a parser plugin file instead. Refer to vLLM Plugins.
  • --reasoning-parser nemotron_v3 — strips reasoning content out of the response server-side, and supplies the reasoning delimiters that a thinking budget forces once it runs out. Refer to Reasoning Mode.
  • --gpu-memory-utilization 0.8 leaves headroom on the same GPU for ASR, diarization, and TTS. Lower it if those models fail to allocate.

vLLM derives the reasoning delimiters from the parser, so no shipped config sets them. Override both with --reasoning-config when a checkpoint closes its reasoning span with a non-standard string:

$vllm serve nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B-BF16 \
>    --reasoning-parser nemotron_v3 \
>    --reasoning-config '{"reasoning_start_str": "<think>", "reasoning_end_str": "\nI have to finalize the answer now.</think>"}'

The Nemotron-3 model cards do not ask for this flag. Add it only when a manual run leaves reasoning text in the spoken response or ignores the thinking budget.

nemotron_3.5_lightning_think.yaml (and, for the older model family, nemotron_nano_v3_think.yaml) uses the same server flags, so one running server serves both configs. The differences are all request-side: enable_thinking: True under vllm_generation_params.extra.extra_body.chat_template_kwargs, max_new_tokens raised from 1024 to 4096, and a new thinking_budget: 2048 forwarded as thinking_token_budget.

Let the Agent Start vLLM

Set start_vllm_on_init: true in the model sub-YAML. At startup VLLMService (in nemo_voice_agent/pipecat/services/nemo/llm.py) does the following:

  1. Takes the port from base_url. An explicit --port inside vllm_server_params overrides it with a warning.
  2. Probes /v1/models on that port. If a server is already answering there with the same model ID, it reuses it and skips the spawn entirely.
  3. Otherwise scans upward from that port for a free one, rewrites the base URL to http://localhost:<port>/v1, and runs vllm serve <llm.model> <vllm_server_params>.
  4. Polls /v1/models every few seconds until the model appears, and raises if the child process dies or the wait exceeds its (non-configurable) one-hour cap.
  5. Terminates the child process when the pipeline stops or is cancelled.

Two constraints follow from the implementation:

  • Local only. The resolved URL is always localhost, so a remote base_url is discarded in this mode. Use start_vllm_on_init: false for a remote server.
  • No shell quoting. vllm_server_params is split on whitespace, so any flag whose value contains a space (such as the omni --limit-mm-per-prompt JSON) is mangled. Start those models manually.

llm.device does not reach the spawned vLLM process: get_llm_service_from_config passes device only on the hf path, so VLLMService always uses its built-in cuda default and never sets CUDA_VISIBLE_DEVICES. To pin the spawned server to a specific GPU, export CUDA_VISIBLE_DEVICES in the shell that starts the agent server, or start vLLM yourself with start_vllm_on_init: false.

Point at an Already-Running vLLM

Set base_url to the server and leave start_vllm_on_init: false:

1# llm_configs/<your-model>.yaml
2type: vllm
3base_url: "http://gpu-host.internal:8000/v1"
4api_key: "EMPTY"
5start_vllm_on_init: false

Set llm.model (top level, in default.yaml) to exactly the ID returned by that server’s /v1/models, otherwise every request fails with a 404 from vLLM. Start the remote server with --host 0.0.0.0 so it accepts connections from outside its machine. For a hosted endpoint rather than your own vLLM, use llm.type: nvidia instead. Refer to NVIDIA NIM and Riva.

Tensor Parallelism and VRAM

The shipped default is NVFP4 and fits one GPU with FP4 support at --tensor-parallel-size 1. For rough sizing, allow about 21 GB for a 9B LLM and 13 GB for a 4B LLM. Also reserve a few GB for speech models that share the GPU. nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B-BF16 needs more than 60 GB, so raise --tensor-parallel-size to 2 (and set --max-model-len 8192 to keep the KV cache in budget). On a GPU that supports FP8 but not FP4, serve nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B-FP8 instead of the BF16 weights: point llm.model at that repository, keep the rest of the config as shipped, and the VRAM requirement drops well below the BF16 figure.

Tuning order when you hit an out-of-memory error:

  1. Lower --gpu-memory-utilization.
  2. Lower --max-model-len.
  3. Raise --tensor-parallel-size to spread across GPUs, or move ASR/diarization/TTS to a different GPU with their own device: keys.

Omni (Audio-In) Recipe

nemotron_nano_v3_omni.yaml sets is_omni_model: true, which inserts the user-audio buffer into the pipeline so raw audio reaches the LLM. Its vllm_server_params add the multimodal flags:

$vllm serve nvidia/Nemotron-3-Nano-Omni-30B-A3B-Reasoning-NVFP4 \
>    --trust-remote-code \
>    --tensor-parallel-size 1 \
>    --enable-prefix-caching \
>    --max-num-seqs 1 \
>    --gpu-memory-utilization 0.8 \
>    --max-model-len 131072 \
>    --video-pruning-rate 0.5 \
>    --allowed-local-media-path / \
>    --limit-mm-per-prompt '{"video": 999, "image": 999, "audio": 999}' \
>    --media-io-kwargs '{"video": {"fps": 2, "num_frames": 256}}' \
>    --enable-auto-tool-choice \
>    --tool-call-parser qwen3_coder \
>    --reasoning-parser nemotron_v3

Keep start_vllm_on_init: false here: the quoted JSON values would not survive the whitespace split described above. Because you type this command yourself, you can also add --kv-cache-dtype fp8, which the model card lists for the quantized checkpoints to shrink the KV cache. Omit that flag for the BF16 checkpoint. Set keep_only_last_audio_turn: false against a self-hosted vLLM, which accepts multiple audio turns. For the remaining omni keys, refer to Multimodal / Omni.

Use these pages to configure the backend and understand the model behavior behind it: