vLLM Plugins

NeMo Labs Voice Agent ships three files that run inside the vLLM process, not inside the voice agent process: a tool-call parser, a reasoning parser, and a logits processor. The vllm serve command-line flags load them, so their logging, imports, and Python environment belong to vLLM — nothing in nemo_voice_agent/pipecat/ can import or configure them at runtime.

Most deployments need none of them. Current vLLM releases provide reasoning parsing and native thinking-budget control for Nemotron-3 and newer. The shipped nemotron_nano_v3* configs therefore use only built-in functionality. The custom plugins predate that support and remain for the one model that still needs a plugin and for older vLLM releases.

What Ships

The repository includes parser plugins for tool calls and reasoning plus a deprecated logits processor.

Which Do You Need?

Choose plugins according to the model’s output format and whether it exposes reasoning or tool calls.

The two deprecated files are kept, not deleted, so that a deployment pinned to an older vLLM — one without nemotron_v3 or thinking_token_budget — still has a working path. Prefer the built-ins on any current vLLM release. The plugins receive no further work.

Tool-Call Parser: nemotron_json

nemotron_toolcall_parser_streaming.py defines NemotronToolParser and registers it with vLLM’s ToolParserManager under the name nemotron_json. It is a streaming parser for Nemotron-Nano-9B-v2, which emits tool calls wrapped in <TOOLCALL> control tokens. The parser intercepts those tokens and buffers ambiguous partial tags so they do not reach the user as spoken text. It reconstructs valid objects from incomplete JSON with partial_json_parser and emits monotonic deltas. This buffering matters for voice because a leaked partial tag goes directly to text-to-speech (TTS).

It is the only plugin any shipped config loads. nemotron_nano_v2.yaml sets, inside vllm_server_params:

--enable-auto-tool-choice
--tool-parser-plugin server/parsers/nemotron_toolcall_parser_streaming.py
--tool-call-parser nemotron_json

All three flags are required together: the plugin flag imports the module (which runs the registration decorator), and --tool-call-parser selects the now-registered name.

The Plugin Path Is Relative

server/parsers/... is written relative to examples/generic_voice_agent/, and vLLM resolves it against its own working directory. nemotron_nano_v2.yaml also sets start_vllm_on_init: true, and the spawn in nemo_voice_agent/pipecat/services/nemo/llm.py calls subprocess.Popen without a cwd override — so vLLM inherits the voice agent server’s working directory.

Launch the server from examples/generic_voice_agent/, not from server/:

$
cd examples/generic_voice_agent
$
# export SERVER_CONFIG_PATH=server/server_configs/my_config.yaml  # optional; defaults to server_configs/default.yaml
$
python server/server.py

Config files still resolve correctly from there, because server.py anchors ConfigManager to its own directory rather than to the working directory. If you prefer to run from elsewhere, edit vllm_server_params and give --tool-parser-plugin an absolute path.

If you start vLLM yourself in a separate terminal (the pattern used by the default Nemotron-3 config), the same rule applies to that shell’s working directory:

$
cd examples/generic_voice_agent
$
vllm serve nvidia/NVIDIA-Nemotron-Nano-9B-v2 \
>
    --trust-remote-code --enable-prefix-caching --max-num-seqs 1 \
>
    --gpu-memory-utilization 0.85 --max-model-len 8192 \
>
    --mamba_ssm_cache_dtype float32 \
>
    --enable-auto-tool-choice \
>
    --tool-parser-plugin server/parsers/nemotron_toolcall_parser_streaming.py \
>
    --tool-call-parser nemotron_json

Reasoning Parsers

The shipped Nemotron-3 configs do not use a plugin for reasoning. Every nemotron_nano_v3*.yaml passes --reasoning-parser nemotron_v3, which is a parser built into vLLM. The evaluation configs under evaluation/server_configs/ pass --reasoning-parser deepseek_r1, also built in. Configs for other models set no reasoning parser at all.

examples/generic_voice_agent/server/parsers/nano_v3_reasoning_parser.py is deprecated. It subclasses vLLM’s DeepSeek-R1 parser and registers itself as nano_v3, and it predates vLLM shipping nemotron_v3 natively. It is superseded: use --reasoning-parser nemotron_v3 for Nemotron-3 and newer.

No shipped config loads it — none passes --reasoning-parser nano_v3, and none passes --reasoning-parser-plugin to import the file, so its registration never runs. Do not mistake it for the mechanism behind the default model’s reasoning support. That is the built-in nemotron_v3. It is retained only for deployments pinned to a vLLM release old enough to lack nemotron_v3, where both flags are needed:

$
cd examples/generic_voice_agent
$
vllm serve <model> \
>
    --reasoning-parser-plugin server/parsers/nano_v3_reasoning_parser.py \
>
    --reasoning-parser nano_v3

A reasoning parser strips the thinking span out of the OpenAI response entirely, so the voice agent does not receive it. That is a different mechanism from tts.think_tokens, which lets the text reach the pipeline but keeps it out of TTS. Refer to Reasoning.

ReasoningBudgetLogitsProcessor (Deprecated)

Use thinking_token_budget instead. Current vLLM releases cap thinking natively for Nemotron-3 and newer, and that is what the shipped configs use. llm_configs/nemotron_nano_v3_think.yaml sets a top-level thinking_budget and passes it through as a request parameter:

1
vllm_generation_params:
2
  extra:
3
    extra_body:
4
      chat_template_kwargs:
5
        enable_thinking: True
6
      thinking_token_budget: ${llm.thinking_budget}

No plugin, no --logits-processors flag, and it works over the plain OpenAI-compatible API. The logits processor below is retained only for vLLM releases that predate thinking_token_budget.

nemo_voice_agent/vllm/v1/sample/logits_processor/ holds a vLLM V1 LogitsProcessor that caps how many tokens a request can spend inside a thinking block. Long reasoning is the dominant latency cost in a voice turn, so a hard budget bounds time-to-first-audio even when the model wants to keep thinking.

It tracks <think> and </think> boundaries in each request’s output stream. As the count approaches the budget, it boosts the logits of newline and end tokens. At the hard limit, it masks every other token so the forced end sequence is emitted. It reports is_argmax_invariant() as False because forcing tokens changes the argmax outcome.

No shipped config loads it. Enable it per the module docstring by adding --logits-processors to vllm_server_params (or to your manual vllm serve command), pointing at the importable path:

--logits-processors '["nemo_voice_agent.vllm.v1.sample.logits_processor.reasoning_budget_logits_processor:ReasoningBudgetLogitsProcessor"]'

vLLM parses each entry as a fully qualified class name in module:Type form and imports it with importlib. The nemo_voice_agent package must be installed in the vLLM environment. It is installed when both processes share this repository’s virtual environment.

The processor remains inactive until a request supplies a budget. Use vllm_xargs over the OpenAI API. For offline use with vllm.LLM, set SamplingParams(extra_args=...).

Values are validated when the request is admitted: a negative or non-integer thinking_budget, a negative grace period, or a non-string delimiter raises ValueError.

To send vllm_xargs from the voice agent, put it under llm.vllm_generation_params.extra in your model config — Pipecat merges that dict into the chat-completion request body verbatim.

Two suites cover the processor: tests/unit/test_reasoning_budget_processor_coverage.py (CPU-only, runs in the default unit pass) and tests/functional/vllm/test_reasoning_budget_logits_processor.py.

$
uv run pytest tests/unit/test_reasoning_budget_processor_coverage.py

Note that the llm.reasoning_budget config key is unrelated to this plugin: it is read only on the Hugging Face backend path in nemo_voice_agent/pipecat/services/nemo/llm.py and is never forwarded to vLLM.

Related Pages

Use these pages for the server configuration and runtime behavior that load the plugins:

• vLLM Backend — starting the server, vllm_server_params, start_vllm_on_init.
• Reasoning — thinking mode, tts.think_tokens, the *_think.yaml configs.
• Tool Calling — what the parser feeds into.
• Testing — markers and how to run the functional suites.