Switchyard

Switchyard is a routing proxy: it decides, per request, which model should carry the work. The switchyard_model server (in responses_api_models/switchyard_model) puts that decision behind a NeMo Gym model server, so any benchmark Gym already supports can be run against a router without changing the agent harness.

Two ways to run the proxy

Both are fully supported; pick whichever fits how you work.

The mode is inferred from whichever field you set — there is no separate switch. Setting neither is a startup error naming both.

Let Gym host the proxy

Nothing to install. Switchyard is a dependency of this model server, so NeMo Gym installs its published wheel into the server’s virtual environment when the server starts. It is scoped to this server rather than to Gym’s core dependencies — Gym itself never imports Switchyard on Gym’s side of the boundary, and runs that do not route through it are unaffected.

A deployment is a TOML file declaring the llm_clients, targets, and routes the proxy serves. A minimal one with a fixed baseline route and a strong/weak classifier route looks like this (see the Switchyard documentation for the full schema and the other routing algorithms):

1
schema_version = 1
2
3
[llm_clients.nvidia]
4
format = "openai_chat"
5
base_url = "https://integrate.api.nvidia.com/v1"
6
api_key_env = "NVIDIA_API_KEY" # pragma: allowlist secret
7
8
[targets.strong]
9
id = "meta/llama-3.3-70b-instruct"
10
llm_client = "nvidia"
11
12
[targets.weak]
13
id = "meta/llama-3.1-8b-instruct"
14
llm_client = "nvidia"
15
16
[routes.strong-only]
17
id = "strong-only"
18
type = "passthrough"
19
target = "strong"
20
21
[routes.policy-model]
22
id = "policy-model"
23
type = "llm_classifier"
24
classifier_target = "weak"
25
strong_target = "strong"
26
weak_target = "weak"
27
base_threshold = 0.5

Each route is a routing condition an eval can run under, and --model selects one — so running the same benchmark twice, once with --model strong-only and once with --model policy-model, compares the fixed baseline against the router on identical tasks.

Point deployment at your TOML file and run an eval:

$
gym eval run \
>
    --benchmark <name> \
>
    --model-type switchyard_model \
>
    --model <route-name> \
>
    ++policy_model.responses_api_models.switchyard_model.deployment=/abs/path/routes.toml

Or start a persistent environment:

$
gym env start \
>
    --resources-server example_single_tool_call \
>
    --model-type switchyard_model \
>
    --model <route-name> \
>
    ++policy_model.responses_api_models.switchyard_model.deployment=/abs/path/routes.toml

deployment can also come from the SWITCHYARD_DEPLOYMENT environment variable, in which case the override can be dropped.

Hosted mode is single-process by design: num_workers greater than 1 is rejected at startup, because each worker process would host its own proxy — splitting session affinity and statistics, and colliding on a fixed proxy_port. To parallelize workers, run one proxy yourself and attach every worker to it.

Under the hood, Gym hosts Switchyard’s native Rust server inside the model-server process (switchyard_rust.server.Server) on a free loopback port that does not collide with Gym’s own servers, and points its client at http://127.0.0.1:<port>/v1. The deployment is loaded and validated when the server starts, so a bad routing config fails immediately with Switchyard’s validation error rather than as a timeout mid-run. Because the proxy lives in the server’s process, it also cannot outlive it — there is no subprocess to leak or reap.

The deployment file plus the selected route is the routing condition the eval runs under — an explicit, diffable artifact. Comparing routing conditions means running the same benchmark per condition and comparing the results: either two routes from one deployment file (as in the example above) or two deployment files.

--model names a route, not a model

For other model servers, --model is the model to serve. Here it is the route name Switchyard resolves — it must match a route in your deployment. Which concrete model actually served a call is Switchyard’s decision, made per request, and comes back on the response.

Manage the proxy yourself

Start Switchyard however you normally would — the standalone switchyard-server binary (built from the Switchyard repo with cargo build --release; it is not part of the nemo-switchyard wheel), a container, a shared service on another host — then set switchyard_base_url instead of deployment. Gym will use that proxy and never start one of its own.

$
switchyard-server --config routes.toml --port 4000
$
$
gym eval run \
>
    --benchmark <name> \
>
    --model-type switchyard_model \
>
    --model <route-name> \
>
    ++policy_model.responses_api_models.switchyard_model.switchyard_base_url=http://127.0.0.1:4000/v1

switchyard_base_url can also come from the SWITCHYARD_BASE_URL environment variable, and switchyard_api_key from SWITCHYARD_API_KEY.

Two reasons to prefer this mode:

• Pinning. Running the proxy yourself lets an eval target a specific Switchyard build, so a Gym run can be compared against another harness’s run of the same commit.
• Shared state. Routing strategies that use session or agent affinity are stateful. If several model servers each host their own proxy, calls that belong together can land on different instances, and routing will not behave the way a single deployed proxy does. One proxy shared by all of them avoids this. Stateless strategies — task classification, for example — are unaffected. A hosted proxy also binds loopback only, so anything that must reach the proxy from another machine needs this mode.

Correlating routing decisions with rollouts

Gym sends its rollout-attempt id to Switchyard as an opaque session id, using the x-switchyard-session-id header — the name the native server parses into its routing metadata, where it reaches request logs, OpenTelemetry spans, and session-affinity routing. Set forward_session_id: false to disable it, or session_id_headers to change the names sent.

On the Switchyard side, aggregate routing statistics are on /v1/stats. Per-session decision snapshots on /v1/routing/session-stats require a durable routing log, which at Switchyard 0.2.0 only the standalone binary can enable — run switchyard-server --routing-log-file <path>, attach with switchyard_base_url, and add proxy_x_session_id to session_id_headers, the name that log keys sessions on. Add header names knowingly: Switchyard forwards headers it does not recognize (and, at 0.2.0, the session header itself) through to the upstream provider.

To see which model served each call from Gym’s side, enable model-call capture. Each captured exchange records the response, so the routed model is visible per call.

Compare routing conditions

A routing condition is a deployment file plus a selected route. To measure what routing does to a benchmark, run it once per condition and compare — the example deployment above defines two conditions ready for exactly that: strong-only (a fixed baseline) and policy-model (the strong/weak router).

Run each condition with its own output file and condition_dir:

$
gym eval run --benchmark <name> --model-type switchyard_model \
>
    --model strong-only \
>
    -o results/strong-only/rollouts.jsonl \
>
    ++policy_model.responses_api_models.switchyard_model.deployment=/abs/path/routes.toml \
>
    ++policy_model.responses_api_models.switchyard_model.condition_dir=results/strong-only
$
$
gym eval run --benchmark <name> --model-type switchyard_model \
>
    --model policy-model \
>
    -o results/policy-model/rollouts.jsonl \
>
    ++policy_model.responses_api_models.switchyard_model.deployment=/abs/path/routes.toml \
>
    ++policy_model.responses_api_models.switchyard_model.condition_dir=results/policy-model

Each results directory now identifies its condition. switchyard-condition.json is the provenance manifest — the route, the deployment’s SHA-256 and archived contents (inline api_key and all extra_headers values redacted), and the nemo-switchyard version — written when the proxy starts. switchyard-stats.json wraps the proxy’s /v1/stats at shutdown: per-target requests, errors, tokens, and latency, plus the classifier-side usage of LLM-classifier routes, which for a hosted proxy would otherwise die with the process. Two runs are fairly comparable when their manifests differ only in route; a hosted run is reproducible from the archived deployment alone.

Comparing scores and cost means aligning the two runs’ rollout rows first — a rollout that failed in one condition must not be counted in the other — and remembering that a routed call can cost more than its selected model: an llm_classifier route also spends classifier tokens, which appear in the stats snapshot rather than the rollout’s own usage. Rollout rows carry _ng_task_index and _ng_rollout_index for exactly this kind of join:

$
python3 - <<'EOF'
$
import json
$
$
def rollouts(condition):
$
    with open(f"results/{condition}/rollouts.jsonl") as lines:
$
        rows = [json.loads(line) for line in lines]
$
    return {(row["_ng_task_index"], row["_ng_rollout_index"]): row for row in rows}
$
$
conditions = {name: rollouts(name) for name in ("strong-only", "policy-model")}
$
shared = set.intersection(*(set(rows) for rows in conditions.values()))
$
for name, rows in conditions.items():
$
    if len(rows) != len(shared):
$
        print(f"{name}: {len(rows) - len(shared)} rollouts have no counterpart; comparing the {len(shared)} shared")
$
$
for name, rows in conditions.items():
$
    rewards = [rows[key]["reward"] for key in shared]
$
    selected_tokens = sum(rows[key]["response"]["usage"]["total_tokens"] for key in shared)
$
    stats = json.load(open(f"results/{name}/switchyard-stats.json"))["stats"]
$
    classifier_tokens = stats["classifier"]["total_tokens"]["total"]
$
    print(
>
        f"{name}: mean reward {sum(rewards) / len(rewards):.3f}, "
>
        f"selected-model tokens {selected_tokens}, classifier tokens {classifier_tokens}"
>
    )
$
EOF

Configuration reference