Skip to navigation

Routing Models with NeMo Switchyard

View as MarkdownOpen in Claude

Running every agent request on your most capable model can waste budget and add latency. NeMo Switchyard chooses a model for each request or step, so routine work can use efficient models while demanding work uses capable ones.

With NVIDIA NeMo Relay, Switchyard runs as a first-party native plugin. Add routing to an existing supported agent setup without changing the agent or operating a separate routing service. Relay records model choices, fallbacks, latency, and token use so you can assess the tradeoff.

Use this guide to evaluate a routing policy on your workload: compare it with fixed-model baselines and check that task quality meets your requirements.

Understand the Switchyard integration

Use Switchyard with NeMo Relay explains the request flow, available routing algorithms, supported APIs, credential handling, and routing telemetry. It is the reference for how the integration works; this guide focuses on choosing and evaluating a policy.

Try the complete example on Brev

Run the complete example to compare fixed and routed coding tasks using Claude Code or Codex.

What is Brev?

NVIDIA Brev provides cloud development machines you can access from your local tools, so you can try the example without installing the agent stack on your workstation.

1. Choose representative tasks and models

Choose tasks with clear pass/fail checks, including work that needs a capable model. Keep task inputs, tools, output limits, and acceptance checks the same across these three policies:

PolicyWhat it tells you
Fixed capableThe task quality, latency, and usage of your capable-model baseline.
Fixed efficientWhether the efficient model alone meets your requirements.
RoutedWhether model selection preserves task quality with useful changes in latency and usage.

Choose models that support your tools, context, and output requirements. Switchyard’s model selection guide explains how to select targets and a judge for policies that need one.

2. Configure a routing policy

Use Relay’s first-party Switchyard plugin guide as the definitive source for Relay and plugin version requirements. Follow it to install the plugin, configure trust and provider access, and enable and validate it in your existing Relay setup.

A Switchyard deployment defines provider connections as clients, model choices as targets, and selection policies as routes. Configure a fixed route for each baseline and a candidate route for the comparison. Point your agent at the candidate’s route ID, such as switchyard/routed, so Switchyard selects the model behind that name.

For an LLM classifier policy, choose the judge, decision threshold, and when to classify. Keeping a selection for a session avoids changing models at each tool continuation; give each independent task a distinct session identity. Refer to classifier routing for these settings and other policy choices.

3. Run comparable work through each policy

Run the same tasks through the capable, efficient, and routed policies. Apply your acceptance checks to the resulting work, even when the agent exits successfully. Record task results, elapsed time, model choices, and available usage, including failed attempts, retries, and judge calls.

Repeat the comparison to distinguish consistent behavior from a single good or bad run. Keep failed results so you can identify which tasks each policy misses.

4. Interpret task quality and routing together

Compare the routed policy with both fixed-model baselines on your workload.

SignalHow to interpret it
Task acceptanceSimilar totals can hide failures on different tasks. Inspect which requirements each policy missed.
Elapsed timeCompare repeated task durations and slow cases, including judge calls, retries, and timeouts.
Answer usageCompare input and output tokens by model. Lower token totals alone do not establish lower cost.
Judge overheadInclude classification usage and latency when evaluating the whole workflow.
Routing mixCheck whether selections match task needs. If a policy always chooses one target, compare it with that fixed-model baseline.
Recoveries and truncationInspect fallback reasons and incomplete calls before attributing results to the routing policy.

A capable answer after an invalid judge verdict does not demonstrate successful classification. Inspect the recovery evidence alongside the accepted task.

Add cost when prices are known

Price input and output usage separately for each answer and judge model, using your validated pricing catalog. Keep missing usage or pricing marked unknown.

Workload cost = answer-model cost + judge cost
Cost per accepted task = workload cost / accepted tasks

Compare the result with the fixed-capable baseline. Apply your task-quality requirement before accepting a cost improvement.

5. Refine and validate the policy

If routing misses required tasks, inspect the chosen target and judge verdict. If task quality holds but latency or usage grows, check judge frequency and retries. If the efficient baseline already meets your requirements, consider whether routing adds useful value.

Change one setting at a time and repeat the comparison. Adopt a policy when it meets your task-quality and resource requirements across repeated trials. Re-evaluate after changes to models, providers, or the harness.


Run the Brev example

ExampleClaude CodeOpenAI CodexNeMo Switchyard

Follow this section from a fresh VM through the evidence checks. The workload uses your choice of Claude Code or Codex with NVIDIA-hosted inference and the Switchyard native plugin. Use the same harness for all three policies in a comparison. Other supported model-handling processes remain deployment choices.

Codex sends OpenAI Responses requests; Claude Code sends Anthropic Messages. Switchyard translates either format to the configured NVIDIA Chat Completions targets. Repeat acceptance when changing a harness, model, or plugin version.

1. Provision and connect to a VM

In the Brev Console, create a fresh instance using these recommended starting resources. Refer to Brev provisioning for instance selection; available CPU types depend on your organization and provider.

SettingRecommendation
Operating systemUbuntu 22.04 or 24.04 LTS, x86_64
CPU and memory4 vCPUs and 16 GB RAM
StorageAt least 40 GB
GPUNone; inference runs on the model provider
AccessAdministrator account with sudo and outbound HTTPS

These are starting resources for the example, not Relay sizing requirements. Use a fresh VM without other system-wide Relay policy: system configuration still applies when you select an explicit plugin file.

Use a disposable VM

The acceptance checker executes generated Python outside the harness sandbox. Its environment excludes provider and installation credentials, but it still runs with your VM user’s file and network access. Keep only this example’s credentials and synthetic data on the VM.

On your local computer, install the Brev CLI if needed. On macOS:

brew install brevdev/homebrew-brev/brev

For other operating systems, use the Brev CLI installation instructions. Connect to the instance you created:

brev login
brev refresh
brev list
brev shell YOUR_INSTANCE

Run the remaining setup in Bash on the VM host.

2. Install the runtimes

Use uv to install Python 3.12 on either Ubuntu image:

sudo apt-get update
sudo apt-get install -y ca-certificates curl gh git python3 xz-utils
curl -LsSf https://astral.sh/uv/install.sh | sh
export PATH="$HOME/.local/bin:$PATH"
uv python install 3.12
python3.12 --version
git --version
gh --version

Install Node.js 24.13.1 system-wide so login shells can find the same runtime:

curl -fsS https://nodejs.org/dist/v24.13.1/node-v24.13.1-linux-x64.tar.xz -o /tmp/node-v24.13.1-linux-x64.tar.xz
curl -fsS https://nodejs.org/dist/v24.13.1/SHASUMS256.txt -o /tmp/node-24.13.1-SHASUMS256.txt
(cd /tmp && awk '$2 == "node-v24.13.1-linux-x64.tar.xz"' node-24.13.1-SHASUMS256.txt | sha256sum --check)
sudo tar -xJf /tmp/node-v24.13.1-linux-x64.tar.xz -C /usr/local --strip-components=1
node --version
npm --version

Confirm the required platform before downloading the plugin:

uname -m
getconf GNU_LIBC_VERSION
python3.12 -c 'import sys; assert sys.version_info >= (3, 12); print("Python ready")'
node --version

Expect x86_64, a glibc version, Python 3.12+, and Node.js 24.13.1.

3. Choose a harness and install Relay and the plugin

Choose one harness. For its Relay integration details, refer to Claude Code or Codex.

sudo npm install --global --prefix /usr/local @anthropic-ai/claude-code@2.1.309
claude --version
export RELAY_EXAMPLE_HARNESS=claude
One key for the models

Switchyard uses NVIDIA_API_KEY to call the efficient model, capable model, and routing judge on NVIDIA’s inference endpoint. Claude Code or Codex supplies the agent interface; the runner connects it to Relay automatically. This example requires no ChatGPT or Claude subscription and no OpenAI or Anthropic API key.

Example version pins

This example was tested with Relay 0.11.1 and Switchyard plugin 0.3.2. For current version requirements, refer to Relay’s first-party Switchyard plugin guide.

Install the pinned Relay version and obtain the example files:

curl -fsSL https://raw.githubusercontent.com/NVIDIA/NeMo-Relay/0.11.1/install.sh -o /tmp/install-relay.sh
NEMO_RELAY_VERSION=0.11.1 sh /tmp/install-relay.sh
export PATH="$HOME/.local/bin:$PATH"
git clone --depth 1 https://github.com/NVIDIA/NeMo-Relay.git relay-routing
cd relay-routing/docs/user-guides/_examples/model-routing

Authenticate the GitHub CLI with gh auth login, or supply its supported GH_TOKEN environment variable. Use Relay’s plugin installer with a separate configuration directory so this example does not alter your normal user policy:

SWITCHYARD_INSTALL_ROOT=$(mktemp -d /tmp/relay-switchyard-install.XXXXXX)
GH_CONFIG_DIR="${GH_CONFIG_DIR:-${XDG_CONFIG_HOME:-$HOME/.config}/gh}" \
XDG_CONFIG_HOME="$SWITCHYARD_INSTALL_ROOT" nemo-relay plugins install --user \
github:NVIDIA/NeMo-Relay-Plugins@switchyard-plugin-0.3.2 --no-enable
SWITCHYARD_MANIFEST=$(python3.12 -c '
import sys, tomllib
with open(sys.argv[1], "rb") as source:
print(tomllib.load(source)["plugins"]["dynamic"][0]["manifest"])
' "$SWITCHYARD_INSTALL_ROOT/nemo-relay/plugins.toml")
export SWITCHYARD_MANIFEST

The Relay plugin installer selects and verifies the platform bundle from plugin release 0.3.2. Keep it disabled in the installation directory; the runner enables it only in each comparison’s configuration.

Trust the plugin publisher

The runner uses integrity_only for this unsigned native bundle. A checksum verifies artifact integrity, not independent publisher authenticity. Only load a bundle from a publisher you trust; refer to plugin trust configuration for deployment options.

The runner manages the example configuration

The runner writes isolated route, plugin, and harness configuration, enables and probes the native plugin, and starts/stops its task gateways. You supply the plugin manifest, model IDs, and provider credentials.

4. Configure inference and run one comparison

Choose models from build.nvidia.com and use the NVIDIA inference API. The models below are suggestions; use any compatible models you prefer. The efficient and capable models need tool calling, and the judge needs JSON verdict support. This example reuses the efficient model as the judge.

RoleExample model
Efficient target and judgeNVIDIA Nemotron 3.5 Lightning 30B A3B
Capable targetNVIDIA Nemotron 3 Ultra 550B A55B

Set your chosen model IDs and API credentials in the same shell you use to run the comparison:

export NVIDIA_BASE_URL=https://integrate.api.nvidia.com/v1
export EFFICIENT_MODEL=nvidia/nemotron-3.5-lightning-30b-a3b
export CAPABLE_MODEL=nvidia/nemotron-3-ultra-550b-a55b
export JUDGE_MODEL="$EFFICIENT_MODEL"
IFS= read -r -s -p 'NVIDIA API key: ' NVIDIA_API_KEY; printf '\n'
export NVIDIA_API_KEY

Prepare configuration and task scaffolds without making model calls:

python3.12 run.py --harness "$RELAY_EXAMPLE_HARNESS" --output /tmp/relay-routing-prepared --prepare-only

This checks configuration generation and TOML syntax. It does not load the plugin, validate provider access, or test model behavior. Inspect the generated files, then start with one task across three policies in a different output directory:

python3.12 run.py --harness "$RELAY_EXAMPLE_HARNESS" --output /tmp/relay-routing --task ticket-id --repeat 1
python3.12 summarize.py /tmp/relay-routing

The first check runs incident-ID validation across all three policies. Omit --task ticket-id to include retry delays and dependency planning. The latter includes cycles, transitive blocking, lexical topological order, and a 10,000-node chain. It tests a broader capability boundary; observe the judge’s actual selection rather than assuming it chooses the capable model.

The full workload runs three tasks across three policies, for nine attempts. For a repeated full evaluation, choose a new output directory and use --repeat 3 without a task filter (27 attempts); policy order rotates between trials. External acceptance runs outside the agent’s working directory. Every arm has the selected harness’s tools and the same 16,384-token provider completion limit, including reasoning. That limit also applies to the judge. Failures remain in results.json; the overall command returns nonzero if any arm fails.

Compare Claude Code and Codex in separate output directories; their tools and prompts differ, so their results are separate experiments.

The Codex path uses its workspace-write sandbox. The Claude Code path allows file tools and python3 test commands for the supplied synthetic tasks. Review any Relay hook trust prompts; keep the run confined to the disposable example.

5. Inspect the evidence

EvidenceInspect
Plugin setupEach policy’s setup.log, and startup.log for the initial gateway probe
Harness and task selectioncomparison.json
Task acceptanceresults.json, generated files, and acceptance.log
Selection and deliveryswitchyard.routing.decision: selected_model, served_model, fallback_used
Judge recoveryDecision evidence.source and reason_code
Provider attemptsswitchyard.routing.llm_call: call_role, model, and outcome
Usage and truncationsummary.json: tokens_by_model groups answer/routing usage by target model and token type; missing usage is null. Inspect incomplete model calls alongside usage.

Judge HTTP success does not establish a valid verdict. Review final JSON and recovery reasons alongside the task’s acceptance log.

Use the interpretation guidance above to compare task acceptance, served models, usage, and elapsed time. A completed comparison can contain failed tasks; retain those results. It does not need to demonstrate savings or favor the routed policy. If setup fails before tasks start, check the plugin probe, model access, and the selected runtime versions before repeating in a new output directory.

6. Clean up

The runner stops each task gateway. Retain the scorecard and logs, then remove only your chosen output directories and the directory recorded in SWITCHYARD_INSTALL_ROOT, which contains this example’s plugin installation. Leave other users’ harness settings and plugin installations in place. Stop or delete the Brev VM when done.