Virtual Models

A virtual model is a named inference route in the Inference Gateway. Clients and agents call it like any other model, and the virtual model decides what happens to each request: which backend model serves it, and which middleware runs before and after.

Every model the gateway serves is reached through a virtual model. The Models controller creates a passthrough virtual model for each model entity; you create your own to add routing or guardrails without changing the clients that call them.

The CLI and Python SDK connect to the deployment in your current CLI context. To connect to a local or remote deployment, see Connect with CLI Contexts.

# Confirm the deployment and workspace in the current context
nemo config view

Anatomy

Each middleware entry names a plugin, a config_type, and either an inline config or the config_id of a stored configuration:

{"name": "nemo-guardrails", "config_type": "guardrail_config", "config_id": "default/my-rails"}

NeMo Helix ships two middleware plugins: nemo-switchyard for routing and nemo-guardrails for safety rails. To write your own, see Virtual Model Plugins.

Call a Virtual Model

Call a virtual model by name through the gateway’s OpenAI-compatible route:

nemo chat -m routed-model "What is the capital of France?"

The curl example defaults to a local platform at http://localhost:8080. If your CLI context points at a remote deployment, export its URL as NHX_BASE_URL first, and for an authenticated deployment add -H "Authorization: Bearer $(nemo auth token)".

The gateway resolves names in the workspace from the request path, so a virtual model must be in the same workspace as its callers.

Point an Agent at a Virtual Model

Set the agent’s model to the virtual model, and leave base_url unset so the platform routes it through the gateway:

models:
  default:
    provider: nvidia
    model: default/routed-model

Register the updated config or redeploy the agent. Its code is unchanged and unaware of the routing or rails behind the name.

Route Across Models with Switchyard

The nemo-switchyard middleware routes each request to one of several models, for example to send easy requests to a cheaper model. It runs in request_middleware and supports OpenAI Chat Completions requests and backends.

This virtual model sends 80% of requests to the strong model and 20% to the weak one:

nemo inference virtual-models create routed-model \
  --models '[{"model": "default/<strong-model>"}, {"model": "default/<weak-model>"}]' \
  --request-middleware '[{
    "name": "nemo-switchyard",
    "config_type": "random_routing",
    "config": {
      "strong": {"model": "default/<strong-model>"},
      "weak": {"model": "default/<weak-model>"},
      "strong_probability": 0.8
    }
  }]'

The other routers take a list of models per role:

{
  "name": "nemo-switchyard",
  "config_type": "stage_router",
  "config": {
    "confidence_threshold": 0.5,
    "models": {"capable": ["default/<strong-model>"], "efficient": ["default/<weak-model>"]}
  }
}

llm_classifier adds "mode": "capability", a base_threshold, and a models.judge list. Judge calls go directly to the judge’s provider.

To find a good split for an agent automatically, run the switchyard optimization strategy, which creates one virtual model per model pair and routing strategy and evaluates them as candidate agents.

Add Guardrails

The nemo-guardrails middleware runs a guardrail configuration on every call: input rails in request_middleware, output rails in response_middleware. Pointing an agent at a guarded virtual model secures its model path without changing the agent.

You need a guardrail configuration and a backend model entity. Common guardrail models in the catalog include nvidia-llama-3-1-nemoguard-8b-content-safety, nvidia-llama-3-1-nemoguard-8b-topic-control, and nvidia-llama-3-1-nemotron-safety-guard-8b-v3.

nemo inference virtual-models create guarded-model \
  --default-model-entity default/<model-entity> \
  --request-middleware '[{"name": "nemo-guardrails", "config_type": "guardrail_config", "config_id": "default/<guardrail-config>"}]' \
  --response-middleware '[{"name": "nemo-guardrails", "config_type": "guardrail_config", "config_id": "default/<guardrail-config>"}]'

Omit either side when the configuration has no rails for it. Then point the agent at guarded-model.

Rails cover the agent’s model calls only. They do not cover tool misuse or instructions injected through tool output; to test an agent against those and generate fixes, see Agent Governance.

For configuration structure, streaming, and the guardrails request options, see Guardrails Architecture and Running Inference.

Manage Virtual Models

nemo inference virtual-models list
nemo inference virtual-models get routed-model
nemo inference virtual-models patch routed-model --request-middleware '[...]'
nemo inference virtual-models delete routed-model

To browse virtual models and chat through them, see NeMo Studio Virtual Models.