> This page is for version Latest (default). > For other versions, use one of these documentation indexes: > - Latest (default): https://docs.nvidia.com/nemo-helix/latest/llms.txt > - 0.9: https://docs.nvidia.com/nemo-helix/v0.9/llms.txt > - 0.8: https://docs.nvidia.com/nemo-helix/v0.8/llms.txt > - 0.7: https://docs.nvidia.com/nemo-helix/v0.7/llms.txt > - 0.6: https://docs.nvidia.com/nemo-helix/v0.6/llms.txt > - 0.5: https://docs.nvidia.com/nemo-helix/v0.5/llms.txt > For clean Markdown of any page, append .md to the page URL. > For a complete documentation index, see https://docs.nvidia.com/nemo-helix/llms.txt. > For AI client integration (Claude Code, Cursor, etc.), connect to the MCP server at https://docs.nvidia.com/nemo-helix/_mcp/server. # Virtual Models > Route, guard, and transform inference with Inference Gateway virtual models and middleware. A virtual model is a named inference route in the Inference Gateway. Clients and agents call it like any other model, and the virtual model decides what happens to each request: which backend model serves it, and which middleware runs before and after. Every model the gateway serves is reached through a virtual model. The Models controller creates a passthrough virtual model for each model entity; you create your own to add routing or guardrails without changing the clients that call them. ```mermaid flowchart LR Client --> VM["Virtual model"] VM --> Req["Request middleware"] Req --> Backend["Backend model"] Backend --> Res["Response middleware"] Res --> Client ``` The CLI and Python SDK connect to the deployment in your current CLI context. To connect to a local or remote deployment, see [Connect with CLI Contexts](/nemo-helix/latest/documentation/reference/cli-reference/connect-to-deployments). #### CLI ```bash # Confirm the deployment and workspace in the current context nemo config view ``` #### Python SDK ```python from nemo_helix_plugin.client.client import NemoClient client = NemoClient.from_config() ``` > **Note** > > These examples use the `default` workspace. If your current CLI context uses a different workspace, run `nemo config set --workspace default` first. To check, run `nemo config view`. ## Anatomy | Field | Purpose | | -------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------- | | `default_model_entity` | Backend model entity (`workspace/name`) written into each request before middleware runs. Omit it when a request middleware picks the backend. | | `models` | Model entities the virtual model uses, each with an optional `backend_format`. | | `request_middleware` | Ordered middleware that runs before the request reaches the backend. | | `response_middleware` | Ordered middleware that runs on the response before it returns to the caller. | | `post_response_middleware` | Middleware that runs after the response is returned, for logging and analytics. | Each middleware entry names a plugin, a `config_type`, and either an inline `config` or the `config_id` of a stored configuration: ```json {"name": "nemo-guardrails", "config_type": "guardrail_config", "config_id": "default/my-rails"} ``` NeMo Helix ships three middleware plugins: `nemo-switchyard` for [routing](#route-across-models-with-switchyard), `nemo-guardrails` for [safety rails](#add-guardrails), and `nemo-evals-closed-book` for [closed-book evaluations](#keep-evaluations-closed-book). To write your own, see [Virtual Model Plugins](/nemo-helix/latest/documentation/reference/plugin-architecture#virtual-model-plugins). ## Call a Virtual Model Call a virtual model by name through the gateway's OpenAI-compatible route: #### CLI ```bash nemo chat -m routed-model "What is the capital of France?" ``` #### curl ```bash curl -s "${NHX_BASE_URL:-http://localhost:8080}/apis/inference-gateway/v2/workspaces/default/openai/-/v1/chat/completions" \ -H "Content-Type: application/json" \ -d '{"model": "default/routed-model", "messages": [{"role": "user", "content": "What is the capital of France?"}]}' ``` The `curl` example defaults to a local platform at `http://localhost:8080`. If your CLI context points at a remote deployment, export its URL as `NHX_BASE_URL` first, and for an authenticated deployment add `-H "Authorization: Bearer $(nemo auth token)"`. The gateway resolves names in the workspace from the request path, so a virtual model must be in the same workspace as its callers. ### Point an Agent at a Virtual Model Set the agent's model to the virtual model, and leave `base_url` unset so the platform routes it through the gateway: ```yaml models: default: provider: nvidia model: default/routed-model ``` Register the updated config or [redeploy](/nemo-helix/latest/documentation/agents/deploy-agents#update-a-deployed-agent) the agent. Its code is unchanged and unaware of the routing or rails behind the name. ## Route Across Models with Switchyard The `nemo-switchyard` middleware routes each request to one of several models, for example to send easy requests to a cheaper model. It runs in `request_middleware` and supports OpenAI Chat Completions requests and backends. | `config_type` | Routes | Required configuration | | ---------------- | --------------------------------------------------- | ---------------------------------------------------------------------- | | `random_routing` | A weighted random choice between two models | `strong`, `weak`, `strong_probability` | | `stage_router` | Between capable and efficient models, by confidence | `confidence_threshold`, `models.capable`, `models.efficient` | | `llm_classifier` | By a judge model's assessment of the request | `base_threshold`, `models.judge`, `models.capable`, `models.efficient` | This virtual model sends 80% of requests to the strong model and 20% to the weak one: #### CLI ```bash nemo inference virtual-models create routed-model \ --models '[{"model": "default/"}, {"model": "default/"}]' \ --request-middleware '[{ "name": "nemo-switchyard", "config_type": "random_routing", "config": { "strong": {"model": "default/"}, "weak": {"model": "default/"}, "strong_probability": 0.8 } }]' ``` #### Python SDK ```python from nemo_helix_plugin.client.client import NemoClient from nemo_helix_plugin.virtual_models.client import VirtualModelsClient from nemo_helix_plugin.virtual_models.types import ( CreateVirtualModelRequest, MiddlewareCall, VirtualModelInferenceConfig, ) client = NemoClient.from_config() VirtualModelsClient.from_client(client).create_virtual_model( body=CreateVirtualModelRequest( name="routed-model", models=[ VirtualModelInferenceConfig(model="default/"), VirtualModelInferenceConfig(model="default/"), ], request_middleware=[ MiddlewareCall( name="nemo-switchyard", config_type="random_routing", config={ "strong": {"model": "default/"}, "weak": {"model": "default/"}, "strong_probability": 0.8, }, ) ], ) ) ``` The other routers take a list of models per role: ```json { "name": "nemo-switchyard", "config_type": "stage_router", "config": { "confidence_threshold": 0.5, "models": {"capable": ["default/"], "efficient": ["default/"]} } } ``` `llm_classifier` adds `"mode": "capability"`, a `base_threshold`, and a `models.judge` list. Judge calls go directly to the judge's provider. To find a good split for an agent automatically, run the [`switchyard` optimization strategy](/nemo-helix/latest/documentation/agents/optimize-agents#switchyard), which creates one virtual model per model pair and routing strategy and evaluates them as candidate agents. ## Add Guardrails The `nemo-guardrails` middleware runs a [guardrail configuration](/nemo-helix/latest/documentation/guardrail-models) on every call: input rails in `request_middleware`, output rails in `response_middleware`. Pointing an agent at a guarded virtual model secures its model path without changing the agent. You need a guardrail configuration and a backend model entity. Common guardrail models in the catalog include `nvidia-llama-3-1-nemoguard-8b-content-safety`, `nvidia-llama-3-1-nemoguard-8b-topic-control`, and `nvidia-llama-3-1-nemotron-safety-guard-8b-v3`. #### CLI ```bash nemo inference virtual-models create guarded-model \ --default-model-entity default/ \ --request-middleware '[{"name": "nemo-guardrails", "config_type": "guardrail_config", "config_id": "default/"}]' \ --response-middleware '[{"name": "nemo-guardrails", "config_type": "guardrail_config", "config_id": "default/"}]' ``` #### Python SDK ```python from nemo_helix_plugin.client.client import NemoClient from nemo_helix_plugin.virtual_models.client import VirtualModelsClient from nemo_helix_plugin.virtual_models.types import CreateVirtualModelRequest, MiddlewareCall client = NemoClient.from_config() rails = MiddlewareCall( name="nemo-guardrails", config_type="guardrail_config", config_id="default/", ) VirtualModelsClient.from_client(client).create_virtual_model( body=CreateVirtualModelRequest( name="guarded-model", default_model_entity="default/", request_middleware=[rails], response_middleware=[rails], ) ) ``` Omit either side when the configuration has no rails for it. Then [point the agent at `guarded-model`](#point-an-agent-at-a-virtual-model). Rails cover the agent's model calls only. They do not cover tool misuse or instructions injected through tool output; to test an agent against those and generate fixes, see [Agent Governance](/nemo-helix/latest/documentation/agents/governance). For configuration structure, streaming, and the `guardrails` request options, see [Guardrails Architecture](/nemo-helix/latest/documentation/guardrail-models/core-concepts/architecture) and [Running Inference](/nemo-helix/latest/documentation/guardrail-models/core-concepts/running-inference). ## Keep Evaluations Closed-Book A closed-book evaluation must not let the model look answers up on the web. Blocking the agent's network is not enough when the model provider can search for it: hosted tools such as OpenAI `web_search`, Anthropic `web_search_20250305`, and MCP servers run on the provider's side, and some models search on every call. The `nemo-evals-closed-book` middleware closes that path for the request the gateway forwards: * It removes every tool the provider would execute, and keeps client-executed tools such as functions, custom tools, a `shell` tool without a provider container, computer use, and Anthropic's `bash` and `text_editor`. A tool type it does not recognize counts as provider-executed and is removed. * It removes provider-side features such as `web_search_options` and `mcp_servers`, and resets a `tool_choice` that pointed at a removed tool to `auto`. When it removes every tool, it also drops `tool_choice` and `parallel_tool_calls`. * It refuses with `403` models that search on their own, such as Perplexity Sonar and `*-search-preview` models, and any model whose provider adds hosted tools or search features to requests through its `default_extra_body` or `required_extra_body`. It also refuses a model it cannot resolve to a provider, since it cannot check what that model is. * It refuses with `403` a Responses request that continues provider-stored history through `previous_response_id` or `conversation`, since that history may hold search results it cannot see. Send the conversation history in the request instead. It takes no settings. Add it last in `request_middleware`, so it sees the model a routing middleware selected: ```bash nemo inference virtual-models create closed-book-model \ --default-model-entity default/ \ --request-middleware '[{"name": "nemo-evals-closed-book", "config_type": "closed_book", "config": {}}]' ``` Point the model or agent under evaluation at `closed-book-model`; for each kind of evaluation target, see [Closed-book evaluations](/nemo-helix/latest/documentation/evaluate-models/agent-eval/targets-and-runners#closed-book-evaluations). The middleware covers Chat Completions, Responses, and Anthropic Messages requests, streaming or not. It does not cover model calls that another middleware makes on its own, such as the judge calls of the `nemo-switchyard` `llm_classifier` router. ## Manage Virtual Models ```bash nemo inference virtual-models list nemo inference virtual-models get routed-model nemo inference virtual-models patch routed-model --request-middleware '[...]' nemo inference virtual-models delete routed-model ``` To browse virtual models and chat through them, see [NeMo Studio Virtual Models](/nemo-helix/latest/documentation/studio/virtual-models).