Virtual Models

A virtual model is a named inference route in the Inference Gateway. Clients and agents call it like any other model, and the virtual model decides what happens to each request: which backend model serves it, and which middleware runs before and after.

Every model the gateway serves is reached through a virtual model. The Models controller creates a passthrough virtual model for each model entity; you create your own to add routing or guardrails without changing the clients that call them.

The CLI and Python SDK connect to the deployment in your current CLI context. To connect to a local or remote deployment, see Connect with CLI Contexts.

# Confirm the deployment and workspace in the current context
nemo config view

Anatomy

Each middleware entry names a plugin, a config_type, and either an inline config or the config_id of a stored configuration:

{"name": "nemo-guardrails", "config_type": "guardrail_config", "config_id": "default/my-rails"}

NeMo Helix ships three middleware plugins: nemo-switchyard for routing, nemo-guardrails for safety rails, and nemo-evals-closed-book for closed-book evaluations. To write your own, see Virtual Model Plugins.

Call a Virtual Model

Call a virtual model by name through the gateway’s OpenAI-compatible route:

nemo chat -m routed-model "What is the capital of France?"

The curl example defaults to a local platform at http://localhost:8080. If your CLI context points at a remote deployment, export its URL as NHX_BASE_URL first, and for an authenticated deployment add -H "Authorization: Bearer $(nemo auth token)".

The gateway resolves names in the workspace from the request path, so a virtual model must be in the same workspace as its callers.

Point an Agent at a Virtual Model

Set the agent’s model to the virtual model, and leave base_url unset so the platform routes it through the gateway:

models:
  default:
    provider: nvidia
    model: default/routed-model

Register the updated config or redeploy the agent. Its code is unchanged and unaware of the routing or rails behind the name.

Route Across Models with Switchyard

The nemo-switchyard middleware routes each request to one of several models, for example to send easy requests to a cheaper model. It runs in request_middleware and supports OpenAI Chat Completions requests and backends.

This virtual model sends 80% of requests to the strong model and 20% to the weak one:

nemo inference virtual-models create routed-model \
  --models '[{"model": "default/<strong-model>"}, {"model": "default/<weak-model>"}]' \
  --request-middleware '[{
    "name": "nemo-switchyard",
    "config_type": "random_routing",
    "config": {
      "strong": {"model": "default/<strong-model>"},
      "weak": {"model": "default/<weak-model>"},
      "strong_probability": 0.8
    }
  }]'

The other routers take a list of models per role:

{
  "name": "nemo-switchyard",
  "config_type": "stage_router",
  "config": {
    "confidence_threshold": 0.5,
    "models": {"capable": ["default/<strong-model>"], "efficient": ["default/<weak-model>"]}
  }
}

llm_classifier adds "mode": "capability", a base_threshold, and a models.judge list. Judge calls go directly to the judge’s provider.

To find a good split for an agent automatically, run the switchyard optimization strategy, which creates one virtual model per model pair and routing strategy and evaluates them as candidate agents.

Add Guardrails

The nemo-guardrails middleware runs a guardrail configuration on every call: input rails in request_middleware, output rails in response_middleware. Pointing an agent at a guarded virtual model secures its model path without changing the agent.

You need a guardrail configuration and a backend model entity. Common guardrail models in the catalog include nvidia-llama-3-1-nemoguard-8b-content-safety, nvidia-llama-3-1-nemoguard-8b-topic-control, and nvidia-llama-3-1-nemotron-safety-guard-8b-v3.

nemo inference virtual-models create guarded-model \
  --default-model-entity default/<model-entity> \
  --request-middleware '[{"name": "nemo-guardrails", "config_type": "guardrail_config", "config_id": "default/<guardrail-config>"}]' \
  --response-middleware '[{"name": "nemo-guardrails", "config_type": "guardrail_config", "config_id": "default/<guardrail-config>"}]'

Omit either side when the configuration has no rails for it. Then point the agent at guarded-model.

Rails cover the agent’s model calls only. They do not cover tool misuse or instructions injected through tool output; to test an agent against those and generate fixes, see Agent Governance.

For configuration structure, streaming, and the guardrails request options, see Guardrails Architecture and Running Inference.

Keep Evaluations Closed-Book

A closed-book evaluation must not let the model look answers up on the web. Blocking the agent’s network is not enough when the model provider can search for it: hosted tools such as OpenAI web_search, Anthropic web_search_20250305, and MCP servers run on the provider’s side, and some models search on every call.

The nemo-evals-closed-book middleware closes that path for the request the gateway forwards:

• It removes every tool the provider would execute, and keeps client-executed tools such as functions, custom tools, a shell tool without a provider container, computer use, and Anthropic’s bash and text_editor. A tool type it does not recognize counts as provider-executed and is removed.
• It removes provider-side features such as web_search_options and mcp_servers, and resets a tool_choice that pointed at a removed tool to auto. When it removes every tool, it also drops tool_choice and parallel_tool_calls.
• It refuses with 403 models that search on their own, such as Perplexity Sonar and *-search-preview models, and any model whose provider adds hosted tools or search features to requests through its default_extra_body or required_extra_body. It also refuses a model it cannot resolve to a provider, since it cannot check what that model is.
• It refuses with 403 a Responses request that continues provider-stored history through previous_response_id or conversation, since that history may hold search results it cannot see. Send the conversation history in the request instead.

It takes no settings. Add it last in request_middleware, so it sees the model a routing middleware selected:

nemo inference virtual-models create closed-book-model \
  --default-model-entity default/<model-entity> \
  --request-middleware '[{"name": "nemo-evals-closed-book", "config_type": "closed_book", "config": {}}]'

Point the model or agent under evaluation at closed-book-model; for each kind of evaluation target, see Closed-book evaluations. The middleware covers Chat Completions, Responses, and Anthropic Messages requests, streaming or not. It does not cover model calls that another middleware makes on its own, such as the judge calls of the nemo-switchyard llm_classifier router.

Manage Virtual Models

nemo inference virtual-models list
nemo inference virtual-models get routed-model
nemo inference virtual-models patch routed-model --request-middleware '[...]'
nemo inference virtual-models delete routed-model

To browse virtual models and chat through them, see NeMo Studio Virtual Models.