> This page is for version Latest (default). > For other versions, use one of these documentation indexes: > - Latest (default): https://docs.nvidia.com/nemo-helix/latest/llms.txt > - 0.8.0: https://docs.nvidia.com/nemo-helix/v0.8.0/llms.txt > - 0.7.3: https://docs.nvidia.com/nemo-helix/v0.7.3/llms.txt > - 0.7.2: https://docs.nvidia.com/nemo-helix/v0.7.2/llms.txt > - 0.7.1: https://docs.nvidia.com/nemo-helix/v0.7.1/llms.txt > - 0.7.0: https://docs.nvidia.com/nemo-helix/v0.7.0/llms.txt > - 0.6.0: https://docs.nvidia.com/nemo-helix/v0.6.0/llms.txt > - 0.5.1: https://docs.nvidia.com/nemo-helix/v0.5.1/llms.txt > - 0.5.0: https://docs.nvidia.com/nemo-helix/v0.5.0/llms.txt > - 0.4.0: https://docs.nvidia.com/nemo-helix/v0.4.0/llms.txt > - 0.3.0: https://docs.nvidia.com/nemo-helix/v0.3.0/llms.txt > - 0.2.0: https://docs.nvidia.com/nemo-helix/v0.2.0/llms.txt > For clean Markdown of any page, append .md to the page URL. > For a complete documentation index, see https://docs.nvidia.com/nemo-helix/llms.txt. > For AI client integration (Claude Code, Cursor, etc.), connect to the MCP server at https://docs.nvidia.com/nemo-helix/_mcp/server. # Virtual Models > Route, guard, and transform inference with Inference Gateway virtual models and middleware. A virtual model is a named inference route in the Inference Gateway. Clients and agents call it like any other model, and the virtual model decides what happens to each request: which backend model serves it, and which middleware runs before and after. Every model the gateway serves is reached through a virtual model. The Models controller creates a passthrough virtual model for each model entity; you create your own to add routing or guardrails without changing the clients that call them. ```mermaid flowchart LR Client --> VM["Virtual model"] VM --> Req["Request middleware"] Req --> Backend["Backend model"] Backend --> Res["Response middleware"] Res --> Client ``` The CLI and Python SDK connect to the deployment in your current CLI context. To connect to a local or remote deployment, see [Connect with CLI Contexts](/documentation/reference/cli-reference/connect-to-deployments). #### CLI ```bash # Confirm the deployment and workspace in the current context nemo config view ``` #### Python SDK ```python from nemo_helix_plugin.client.client import NemoClient client = NemoClient.from_config() ``` > **Note** > > These examples use the `default` workspace. If your current CLI context uses a different workspace, run `nemo config set --workspace default` first. To check, run `nemo config view`. ## Anatomy | Field | Purpose | | -------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------- | | `default_model_entity` | Backend model entity (`workspace/name`) written into each request before middleware runs. Omit it when a request middleware picks the backend. | | `models` | Model entities the virtual model uses, each with an optional `backend_format`. | | `request_middleware` | Ordered middleware that runs before the request reaches the backend. | | `response_middleware` | Ordered middleware that runs on the response before it returns to the caller. | | `post_response_middleware` | Middleware that runs after the response is returned, for logging and analytics. | Each middleware entry names a plugin, a `config_type`, and either an inline `config` or the `config_id` of a stored configuration: ```json {"name": "nemo-guardrails", "config_type": "guardrail_config", "config_id": "default/my-rails"} ``` NeMo Helix ships two middleware plugins: `nemo-switchyard` for [routing](#route-across-models-with-switchyard) and `nemo-guardrails` for [safety rails](#add-guardrails). To write your own, see [Virtual Model Plugins](/documentation/reference/plugin-architecture#virtual-model-plugins). ## Call a Virtual Model Call a virtual model by name through the gateway's OpenAI-compatible route: #### CLI ```bash nemo chat -m routed-model "What is the capital of France?" ``` #### curl ```bash curl -s "${NHX_BASE_URL:-http://localhost:8080}/apis/inference-gateway/v2/workspaces/default/openai/-/v1/chat/completions" \ -H "Content-Type: application/json" \ -d '{"model": "default/routed-model", "messages": [{"role": "user", "content": "What is the capital of France?"}]}' ``` The `curl` example defaults to a local platform at `http://localhost:8080`. If your CLI context points at a remote deployment, export its URL as `NHX_BASE_URL` first, and for an authenticated deployment add `-H "Authorization: Bearer $(nemo auth token)"`. The gateway resolves names in the workspace from the request path, so a virtual model must be in the same workspace as its callers. ### Point an Agent at a Virtual Model Set the agent's model to the virtual model, and leave `base_url` unset so the platform routes it through the gateway: ```yaml models: default: provider: nvidia model: default/routed-model ``` Register the updated config or [redeploy](/documentation/agents/deploy-agents#update-a-deployed-agent) the agent. Its code is unchanged and unaware of the routing or rails behind the name. ## Route Across Models with Switchyard The `nemo-switchyard` middleware routes each request to one of several models, for example to send easy requests to a cheaper model. It runs in `request_middleware` and supports OpenAI Chat Completions requests and backends. | `config_type` | Routes | Required configuration | | ---------------- | --------------------------------------------------- | ---------------------------------------------------------------------- | | `random_routing` | A weighted random choice between two models | `strong`, `weak`, `strong_probability` | | `stage_router` | Between capable and efficient models, by confidence | `confidence_threshold`, `models.capable`, `models.efficient` | | `llm_classifier` | By a judge model's assessment of the request | `base_threshold`, `models.judge`, `models.capable`, `models.efficient` | This virtual model sends 80% of requests to the strong model and 20% to the weak one: #### CLI ```bash nemo inference virtual-models create routed-model \ --models '[{"model": "default/"}, {"model": "default/"}]' \ --request-middleware '[{ "name": "nemo-switchyard", "config_type": "random_routing", "config": { "strong": {"model": "default/"}, "weak": {"model": "default/"}, "strong_probability": 0.8 } }]' ``` #### Python SDK ```python from nemo_helix_plugin.client.client import NemoClient from nemo_helix_plugin.virtual_models.client import VirtualModelsClient from nemo_helix_plugin.virtual_models.types import ( CreateVirtualModelRequest, MiddlewareCall, VirtualModelInferenceConfig, ) client = NemoClient.from_config() VirtualModelsClient.from_client(client).create_virtual_model( body=CreateVirtualModelRequest( name="routed-model", models=[ VirtualModelInferenceConfig(model="default/"), VirtualModelInferenceConfig(model="default/"), ], request_middleware=[ MiddlewareCall( name="nemo-switchyard", config_type="random_routing", config={ "strong": {"model": "default/"}, "weak": {"model": "default/"}, "strong_probability": 0.8, }, ) ], ) ) ``` The other routers take a list of models per role: ```json { "name": "nemo-switchyard", "config_type": "stage_router", "config": { "confidence_threshold": 0.5, "models": {"capable": ["default/"], "efficient": ["default/"]} } } ``` `llm_classifier` adds `"mode": "capability"`, a `base_threshold`, and a `models.judge` list. Judge calls go directly to the judge's provider. To find a good split for an agent automatically, run the [`switchyard` optimization strategy](/documentation/agents/optimize-agents#switchyard), which creates one virtual model per model pair and routing strategy and evaluates them as candidate agents. ## Add Guardrails The `nemo-guardrails` middleware runs a [guardrail configuration](/documentation/guardrail-models) on every call: input rails in `request_middleware`, output rails in `response_middleware`. Pointing an agent at a guarded virtual model secures its model path without changing the agent. You need a guardrail configuration and a backend model entity. Common guardrail models in the catalog include `nvidia-llama-3-1-nemoguard-8b-content-safety`, `nvidia-llama-3-1-nemoguard-8b-topic-control`, and `nvidia-llama-3-1-nemotron-safety-guard-8b-v3`. #### CLI ```bash nemo inference virtual-models create guarded-model \ --default-model-entity default/ \ --request-middleware '[{"name": "nemo-guardrails", "config_type": "guardrail_config", "config_id": "default/"}]' \ --response-middleware '[{"name": "nemo-guardrails", "config_type": "guardrail_config", "config_id": "default/"}]' ``` #### Python SDK ```python from nemo_helix_plugin.client.client import NemoClient from nemo_helix_plugin.virtual_models.client import VirtualModelsClient from nemo_helix_plugin.virtual_models.types import CreateVirtualModelRequest, MiddlewareCall client = NemoClient.from_config() rails = MiddlewareCall( name="nemo-guardrails", config_type="guardrail_config", config_id="default/", ) VirtualModelsClient.from_client(client).create_virtual_model( body=CreateVirtualModelRequest( name="guarded-model", default_model_entity="default/", request_middleware=[rails], response_middleware=[rails], ) ) ``` Omit either side when the configuration has no rails for it. Then [point the agent at `guarded-model`](#point-an-agent-at-a-virtual-model). Rails cover the agent's model calls only. They do not cover tool misuse or instructions injected through tool output; to test an agent against those and generate fixes, see [Agent Governance](/documentation/agents/governance). For configuration structure, streaming, and the `guardrails` request options, see [Guardrails Architecture](/documentation/guardrail-models/core-concepts/architecture) and [Running Inference](/documentation/guardrail-models/core-concepts/running-inference). ## Manage Virtual Models ```bash nemo inference virtual-models list nemo inference virtual-models get routed-model nemo inference virtual-models patch routed-model --request-middleware '[...]' nemo inference virtual-models delete routed-model ``` To browse virtual models and chat through them, see [NeMo Studio Virtual Models](/documentation/studio/virtual-models).