Routing Models with NeMo Switchyard
Running every agent request on your most capable model can waste budget and add latency. NeMo Switchyard chooses a model for each request or step, so routine work can use efficient models while demanding work uses capable ones.
With NVIDIA NeMo Relay, Switchyard runs as a first-party native plugin. Add routing to an existing supported agent setup without changing the agent or operating a separate routing service. Relay records model choices, fallbacks, latency, and token use so you can assess the tradeoff.
Use this guide to evaluate a routing policy on your workload: compare it with fixed-model baselines and check that task quality meets your requirements.
Use Switchyard with NeMo Relay explains the request flow, available routing algorithms, supported APIs, credential handling, and routing telemetry. It is the reference for how the integration works; this guide focuses on choosing and evaluating a policy.
Run the complete example to compare fixed and routed coding tasks using Claude Code or Codex.
What is Brev?
NVIDIA Brev provides cloud development machines you can access from your local tools, so you can try the example without installing the agent stack on your workstation.
1. Choose representative tasks and models
Choose tasks with clear pass/fail checks, including work that needs a capable model. Keep task inputs, tools, output limits, and acceptance checks the same across these three policies:
Choose models that support your tools, context, and output requirements. Switchyard’s model selection guide explains how to select targets and a judge for policies that need one.
2. Configure a routing policy
Use Relay’s first-party Switchyard plugin guide as the definitive source for Relay and plugin version requirements. Follow it to install the plugin, configure trust and provider access, and enable and validate it in your existing Relay setup.
A Switchyard deployment defines provider connections as clients, model
choices as targets, and selection policies as routes. Configure a fixed
route for each baseline and a candidate route for the comparison. Point your
agent at the candidate’s route ID, such as switchyard/routed, so Switchyard
selects the model behind that name.
For an LLM classifier policy, choose the judge, decision threshold, and when to classify. Keeping a selection for a session avoids changing models at each tool continuation; give each independent task a distinct session identity. Refer to classifier routing for these settings and other policy choices.
3. Run comparable work through each policy
Run the same tasks through the capable, efficient, and routed policies. Apply your acceptance checks to the resulting work, even when the agent exits successfully. Record task results, elapsed time, model choices, and available usage, including failed attempts, retries, and judge calls.
Repeat the comparison to distinguish consistent behavior from a single good or bad run. Keep failed results so you can identify which tasks each policy misses.
4. Interpret task quality and routing together
Compare the routed policy with both fixed-model baselines on your workload.
A capable answer after an invalid judge verdict does not demonstrate successful classification. Inspect the recovery evidence alongside the accepted task.
Add cost when prices are known
Price input and output usage separately for each answer and judge model, using your validated pricing catalog. Keep missing usage or pricing marked unknown.
Compare the result with the fixed-capable baseline. Apply your task-quality requirement before accepting a cost improvement.
5. Refine and validate the policy
If routing misses required tasks, inspect the chosen target and judge verdict. If task quality holds but latency or usage grows, check judge frequency and retries. If the efficient baseline already meets your requirements, consider whether routing adds useful value.
Change one setting at a time and repeat the comparison. Adopt a policy when it meets your task-quality and resource requirements across repeated trials. Re-evaluate after changes to models, providers, or the harness.
Run the Brev example
Follow this section from a fresh VM through the evidence checks. The workload uses your choice of Claude Code or Codex with NVIDIA-hosted inference and the Switchyard native plugin. Use the same harness for all three policies in a comparison. Other supported model-handling processes remain deployment choices.
Codex sends OpenAI Responses requests; Claude Code sends Anthropic Messages. Switchyard translates either format to the configured NVIDIA Chat Completions targets. Repeat acceptance when changing a harness, model, or plugin version.
1. Provision and connect to a VM
In the Brev Console, create a fresh instance using these recommended starting resources. Refer to Brev provisioning for instance selection; available CPU types depend on your organization and provider.
These are starting resources for the example, not Relay sizing requirements. Use a fresh VM without other system-wide Relay policy: system configuration still applies when you select an explicit plugin file.
The acceptance checker executes generated Python outside the harness sandbox. Its environment excludes provider and installation credentials, but it still runs with your VM user’s file and network access. Keep only this example’s credentials and synthetic data on the VM.
On your local computer, install the Brev CLI if needed. On macOS:
For other operating systems, use the Brev CLI installation instructions. Connect to the instance you created:
Run the remaining setup in Bash on the VM host.
2. Install the runtimes
Use uv to install Python 3.12 on either Ubuntu image:
Install Node.js 24.13.1 system-wide so login shells can find the same runtime:
Confirm the required platform before downloading the plugin:
Expect x86_64, a glibc version, Python 3.12+, and Node.js 24.13.1.
3. Choose a harness and install Relay and the plugin
Choose one harness. For its Relay integration details, refer to Claude Code or Codex.
Claude Code
Codex
Switchyard uses NVIDIA_API_KEY to call the efficient model, capable model,
and routing judge on NVIDIA’s inference endpoint. Claude Code or Codex supplies
the agent interface; the runner connects it to Relay automatically. This example
requires no ChatGPT or Claude subscription and no OpenAI or Anthropic API key.
This example was tested with Relay 0.11.1 and Switchyard plugin 0.3.2. For current version requirements, refer to Relay’s first-party Switchyard plugin guide.
Install the pinned Relay version and obtain the example files:
Authenticate the GitHub CLI with gh auth login, or
supply its supported GH_TOKEN environment variable. Use Relay’s plugin installer
with a separate configuration directory so this example does not alter your
normal user policy:
The Relay plugin installer selects and verifies the platform bundle from plugin release 0.3.2. Keep it disabled in the installation directory; the runner enables it only in each comparison’s configuration.
The runner uses integrity_only for this unsigned native bundle. A checksum
verifies artifact integrity, not independent publisher authenticity. Only load
a bundle from a publisher you trust; refer to
plugin trust configuration for deployment options.
The runner writes isolated route, plugin, and harness configuration, enables and probes the native plugin, and starts/stops its task gateways. You supply the plugin manifest, model IDs, and provider credentials.
4. Configure inference and run one comparison
Choose models from build.nvidia.com and use the NVIDIA inference API. The models below are suggestions; use any compatible models you prefer. The efficient and capable models need tool calling, and the judge needs JSON verdict support. This example reuses the efficient model as the judge.
Set your chosen model IDs and API credentials in the same shell you use to run the comparison:
Prepare configuration and task scaffolds without making model calls:
This checks configuration generation and TOML syntax. It does not load the plugin, validate provider access, or test model behavior. Inspect the generated files, then start with one task across three policies in a different output directory:
The first check runs incident-ID validation across all three policies. Omit
--task ticket-id to include retry delays and dependency planning. The latter includes cycles, transitive blocking, lexical
topological order, and a 10,000-node chain. It tests a broader capability boundary;
observe the judge’s actual selection rather than assuming it chooses the capable model.
The full workload runs three tasks across three policies, for nine attempts.
For a repeated full evaluation, choose a new output directory and use --repeat 3
without a task filter (27 attempts); policy order rotates between trials.
External acceptance runs outside the agent’s working directory. Every arm has
the selected harness’s tools and the same 16,384-token provider completion limit,
including reasoning. That limit also applies to the judge. Failures remain in results.json;
the overall command returns nonzero if any arm fails.
Compare Claude Code and Codex in separate output directories; their tools and prompts differ, so their results are separate experiments.
The Codex path uses its workspace-write sandbox. The Claude Code path allows
file tools and python3 test commands for the supplied synthetic tasks. Review
any Relay hook trust prompts; keep the run confined to the disposable example.
5. Inspect the evidence
Judge HTTP success does not establish a valid verdict. Review final JSON and recovery reasons alongside the task’s acceptance log.
Use the interpretation guidance above to compare task acceptance, served models, usage, and elapsed time. A completed comparison can contain failed tasks; retain those results. It does not need to demonstrate savings or favor the routed policy. If setup fails before tasks start, check the plugin probe, model access, and the selected runtime versions before repeating in a new output directory.
6. Clean up
The runner stops each task gateway. Retain the scorecard and logs, then remove
only your chosen output directories and the directory recorded in
SWITCHYARD_INSTALL_ROOT, which contains this example’s plugin installation.
Leave other users’ harness settings and plugin installations in place. Stop or
delete the Brev VM when done.