Virtual Models

A virtual model is a named inference route in the Inference Gateway. Clients and agents call it like any other model, and the virtual model decides what happens to each request: which backend model serves it, and which middleware runs before and after.

Every model the gateway serves is reached through a virtual model. The Models controller creates a passthrough virtual model for each model entity; you create your own to add routing or guardrails without changing the clients that call them.

The CLI and Python SDK connect to the deployment in your current CLI context. To connect to a local or remote deployment, see Connect with CLI Contexts.

# Confirm the deployment and workspace in the current context
nemo config view

Anatomy

Each middleware entry names a plugin, a config_type, and either an inline config or the config_id of a stored configuration:

{"name": "nemo-guardrails", "config_type": "guardrail_config", "config_id": "default/my-rails"}

NeMo Helix ships four middleware plugins: nemo-switchyard for routing, nemo-guardrails for safety rails, nemo-evals-closed-book for closed-book evaluations, and nemo-evals-trial-recording for recording evaluation trials. To write your own, see Virtual Model Plugins.

Call a Virtual Model

Call a virtual model by name through the gateway’s OpenAI-compatible route:

nemo chat -m routed-model "What is the capital of France?"

The curl example defaults to a local platform at http://localhost:8080. If your CLI context points at a remote deployment, export its URL as NHX_BASE_URL first, and for an authenticated deployment add -H "Authorization: Bearer $(nemo auth token)".

The gateway resolves names in the workspace from the request path, so a virtual model must be in the same workspace as its callers.

Point an Agent at a Virtual Model

Set the agent’s model to the virtual model, and leave base_url unset so the platform routes it through the gateway:

models:
  default:
    provider: nvidia
    model: default/routed-model

Register the updated config or redeploy the agent. Its code is unchanged and unaware of the routing or rails behind the name.

Route Across Models with Switchyard

The nemo-switchyard middleware routes each request to one of several models, for example to send easy requests to a cheaper model. It runs in request_middleware and supports OpenAI Chat Completions requests and backends.

This virtual model sends 80% of requests to the strong model and 20% to the weak one:

nemo inference virtual-models create routed-model \
  --models '[{"model": "default/<strong-model>"}, {"model": "default/<weak-model>"}]' \
  --request-middleware '[{
    "name": "nemo-switchyard",
    "config_type": "random_routing",
    "config": {
      "strong": {"model": "default/<strong-model>"},
      "weak": {"model": "default/<weak-model>"},
      "strong_probability": 0.8
    }
  }]'

The other routers take a list of models per role:

{
  "name": "nemo-switchyard",
  "config_type": "stage_router",
  "config": {
    "confidence_threshold": 0.5,
    "models": {"capable": ["default/<strong-model>"], "efficient": ["default/<weak-model>"]}
  }
}

llm_classifier adds "mode": "capability", a base_threshold, and a models.judge list. Judge calls go directly to the judge’s provider.

Classifier Output Token Limits

max_output_tokens controls the classifier’s judge output, not the routed backend model’s response. It is optional in both classifier configurations:

Omit the field or set it to JSON null to use the native Switchyard default. Both forms leave the native constructor argument unset; null does not mean an unlimited output budget. An explicit integer, such as 64, overrides the native default.

For a top-level llm_classifier, this middleware entry uses the native default:

{
  "name": "nemo-switchyard",
  "config_type": "llm_classifier",
  "config": {
    "mode": "capability",
    "base_threshold": 0.5,
    "max_output_tokens": null,
    "models": {
      "judge": ["default/<judge-model>"],
      "capable": ["default/<strong-model>"],
      "efficient": ["default/<weak-model>"]
    }
  }
}

For a stage_router, put the limit inside the optional classifier object. Enabling the classifier also requires a models.judge list. This example overrides the limit with 64; omitting only max_output_tokens or replacing 64 with null uses the native default while keeping the classifier enabled:

{
  "name": "nemo-switchyard",
  "config_type": "stage_router",
  "config": {
    "confidence_threshold": 0.5,
    "classifier": {"base_threshold": 0.5, "max_output_tokens": 64},
    "models": {
      "judge": ["default/<judge-model>"],
      "capable": ["default/<strong-model>"],
      "efficient": ["default/<weak-model>"]
    }
  }
}

These are virtual-model middleware settings, not fields in the optimization strategy’s --spec.

To find a good split for an agent automatically, run the switchyard optimization strategy, which creates one virtual model per model pair and routing strategy and evaluates them as candidate agents.

Add Guardrails

The nemo-guardrails middleware runs a guardrail configuration on every call: input rails in request_middleware, output rails in response_middleware. Pointing an agent at a guarded virtual model secures its model path without changing the agent.

You need a guardrail configuration and a backend model entity. Common guardrail models in the catalog include nvidia-llama-3-1-nemoguard-8b-content-safety, nvidia-llama-3-1-nemoguard-8b-topic-control, and nvidia-llama-3-1-nemotron-safety-guard-8b-v3.

nemo inference virtual-models create guarded-model \
  --default-model-entity default/<model-entity> \
  --request-middleware '[{"name": "nemo-guardrails", "config_type": "guardrail_config", "config_id": "default/<guardrail-config>"}]' \
  --response-middleware '[{"name": "nemo-guardrails", "config_type": "guardrail_config", "config_id": "default/<guardrail-config>"}]'

Omit either side when the configuration has no rails for it. Then point the agent at guarded-model.

Rails cover the agent’s model calls only. They do not cover tool misuse or instructions injected through tool output; to test an agent against those and generate fixes, see Agent Governance.

For configuration structure, streaming, and the guardrails request options, see Guardrails Architecture and Running Inference.

Keep Evaluations Closed-Book

A closed-book evaluation must not let the model look answers up on the web. Blocking the agent’s network is not enough when the model provider can search for it: hosted tools such as OpenAI web_search, Anthropic web_search_20250305, and MCP servers run on the provider’s side, and some models search on every call.

The nemo-evals-closed-book middleware closes that path for the request the gateway forwards:

• It removes every tool the provider would execute, and keeps client-executed tools such as functions, custom tools, a shell tool without a provider container, computer use, and Anthropic’s bash and text_editor. A tool type it does not recognize counts as provider-executed and is removed.
• It removes provider-side features such as web_search_options and mcp_servers, and resets a tool_choice that pointed at a removed tool to auto. When it removes every tool, it also drops tool_choice and parallel_tool_calls.
• It refuses with 403 models that search on their own, such as Perplexity Sonar and *-search-preview models, and any model whose provider adds hosted tools or search features to requests through its default_extra_body or required_extra_body. It also refuses a model it cannot resolve to a provider, since it cannot check what that model is.
• It refuses with 403 a Responses request that continues provider-stored history through previous_response_id, conversation, or item_reference input items, since that history may hold search results it cannot see. Send the conversation history in the request instead.
• It refuses with 403 a model whose provider calls models back through the gateway with a platform credential, such as a router registered as a model provider with a platform access key, since it can’t see the calls that provider makes. A model provider’s own API key isn’t a platform credential and passes. A router registered to call back as its caller is served only to closed evaluation trials; see Route Through a Router That Calls Back.

Add it last in request_middleware, so it sees the model a routing middleware selected:

nemo inference virtual-models create closed-book-model \
  --default-model-entity default/<model-entity> \
  --request-middleware '[{"name": "nemo-evals-closed-book", "config_type": "closed_book", "config": {}}]'

By default it applies to every call. With "config": {"apply": "closed_credential"} it applies only to calls made with a closed evaluation trial token, and passes every other call through unchanged, so one virtual model serves open and closed-book evaluations alike. A trial token whose signature can’t be checked right now gets 503 rather than an open call.

Point the model or agent under evaluation at closed-book-model; for each kind of evaluation target, see Closed-book evaluations. The middleware covers Chat Completions, Responses, and Anthropic Messages requests, streaming or not. It does not cover model calls that another middleware makes on its own, such as the judge calls of the nemo-switchyard llm_classifier router.

Route Through a Router That Calls Back

A router registered as a model provider, such as a Switchyard server, picks a model for each call and calls it back through the gateway. Register it so that it calls back as its caller: every policy that applied to the caller then applies to the router’s own calls, and the router needs no closed-book or evaluation settings of its own.

The router never holds the caller’s credential:

1. The gateway sends the router the provider’s stored key and a caller context, an x-nemo-caller-context header holding the caller’s credential sealed with the gateway’s key. Only the gateway can read it. It is bound to that provider and expires after 15 minutes.
2. The router returns the context with each call it makes back through the gateway with that key. Switchyard does this for a backend with forward_auth, which forwards the caller’s headers.
3. The gateway opens the context and puts the caller’s credential back before platform auth runs, so the call is authenticated and authorized as the caller.

nemo inference providers create switchyard --host-url http://switchyard:4002 \
  --api-key-secret-name switchyard-key --calls-back-as-caller

The stored key needs no platform permissions of its own. Through the router a caller reaches only what its own credential allows, and only while that credential is valid. The gateway refuses with 403 any call made with the stored key that doesn’t carry a valid caller context, so the key is no use on its own.

The gateway seals contexts with inference_gateway.caller_context_key, a base64-encoded 32-byte key set through NHX_INFERENCE_GATEWAY_CALLER_CONTEXT_KEY. A Helm install whose chart generates the API environment Secret generates this key too. Set it yourself on an existing install, or where you provide that Secret. Every gateway process must share it. Without one, each process uses a key of its own, which works only while the router’s calls back reach the process that called it.

On the virtual model that serves a router’s route, list the virtual models the router calls back, such as its targets and judge, in its trial-recording middleware’s routes_to. An evaluation trial held to that virtual model may also call those. List the trial-recording middleware in both request_middleware and response_middleware, so the router’s calls back are recorded beneath the trial’s call to the router. In request_middleware, list it after any middleware that changes the model:

nemo inference virtual-models create routed \
  --default-model-entity default/classified \
  --request-middleware '[{"name": "nemo-evals-trial-recording", "config_type": "trial_recording", "config": {"routes_to": ["default/strong", "default/weak", "default/judge"]}}]' \
  --response-middleware '[{"name": "nemo-evals-trial-recording", "config_type": "trial_recording", "config": {}}]'

To serve open and closed-book evaluations through the same targets, give each target the closed-book middleware with "config": {"apply": "closed_credential"}: a closed evaluation’s calls through the router are closed-book, and every other call passes through unchanged.

• Switchyard’s fallback_client forwards any request it doesn’t route, with the caller’s headers, to its own upstream. Give a router registered this way no fallback_client, or one that points at the gateway.
• Closed-book refuses a model served by such a provider unless the caller is a closed evaluation trial, since only then are the router’s own calls held closed-book.

Record Evaluation Trials

An agent evaluation gives each Harbor trial that calls a gateway model its own trial token, a credential that names the trial and the models it may call. That covers Harbor’s built-in agents given a gateway model and registered Fabric agents whose default model goes through the gateway; such an agent gets the token in the key variable of every model it calls through the gateway. The nemo-evals-trial-recording middleware reads the token to record each of the trial’s model calls in Intake, as a span in the trial’s trace with the token usage the provider reported. The span names the model the call asked for (gen_ai.request.model) and the model the response says served it (gen_ai.response.model), which differ when a routing provider picks the model. It doesn’t change, delay or fail the call, and calls made with any other credential aren’t recorded. With platform auth on, it trusts only trial tokens the platform signed.

When a call carries a valid W3C traceparent, its span goes into the caller’s trace beneath the caller’s span instead. The session, evaluation and test case still come from the trial token, never from headers.

Intake counts each call’s usage once. In a trace that holds gateway call spans, their usage replaces the agent’s own spans for the same calls, as long as every gateway span in the trace recorded usage; if one didn’t, the trace counts the agent’s own numbers instead. When the agent’s trace is separate, the evaluation marks the published trace as superseded, so the session and trace metrics count the gateway’s usage. The agent’s own totals stay on its trace for inspection, and a metric the gateway doesn’t report, such as cost, still counts from it. The evaluation marks the trace only when every model the agent calls is a virtual model that records trials. If one isn’t, the session counts both the agent’s trace and the gateway’s spans.

Add it to request_middleware, so a call’s span starts when the call arrives, and to response_middleware, where each call is recorded. Its only setting, routes_to, is for a virtual model served by a router; see Route Through a Router That Calls Back.

nemo inference virtual-models create recorded-model \
  --default-model-entity default/<model-entity> \
  --request-middleware '[{"name": "nemo-evals-trial-recording", "config_type": "trial_recording", "config": {}}]' \
  --response-middleware '[{"name": "nemo-evals-trial-recording", "config_type": "trial_recording", "config": {}}]'

To add recording to an existing virtual model, update it with its current middleware lists plus these entries. Each list you pass replaces the old one. To combine it with closed-book, put both entries in request_middleware, closed-book last.

When the evaluation’s gateway model doesn’t list the middleware, the job logs a warning and runs without recording. With platform auth on, each trial calls the gateway with a Scoped Access Key minted for it, so Scoped Access Keys must be enabled. If a key can’t be minted, the job fails before any trial runs. Each key lives up to 24 hours from the start of the job, so a trial still queued after that can’t call the gateway. Calls the model provider rejects aren’t recorded either, nor are calls the closed-book middleware refuses: the gateway stops those before the response middleware that records them runs.

Hold Trials to Their Models

A trial token lists the virtual models the trial’s agent was configured with: the one a built-in agent’s model names, or every gateway model a registered agent’s config names, and the virtual models each of those routes to, as its trial-recording middleware’s routes_to lists them. The token is closed when the evaluation is labelled closed_book. The nemo-evals-access middleware, which comes with the evals plugin and which no virtual model lists, checks every call made with a trial token, on every route and before any virtual model’s middleware runs. It refuses:

A refused trial call is recorded in the trial as a failed call with zero tokens, with the reason in nemo.evals.refusal.reason. Calls made with any other credential are left alone. With platform auth off, tokens are unsigned, so an agent that writes its own token, or calls a model that doesn’t record trials with no token, isn’t stopped; run evaluations that must hold against a misbehaving agent with auth on.

Manage Virtual Models

nemo inference virtual-models list
nemo inference virtual-models get routed-model
nemo inference virtual-models patch routed-model --request-middleware '[...]'
nemo inference virtual-models delete routed-model

To browse virtual models and chat through them, see NeMo Studio Virtual Models.