Skip to navigation
Use PluginsFirst Party Plugins

NeMo Switchyard

View as Markdown

About

Route one OpenAI-compatible request through Switchyard inside NeMo Relay, then confirm the response and routing event in a local Agent Trajectory Observability Format (ATOF) file.

The NeMo Relay Plugins repository builds and releases plugins that extend Relay, including the plugin for NeMo Switchyard, which routes LLM requests across models and providers.

Prerequisites

Before you start, make sure you have:

This guide pins Switchyard plugin 0.3.2 and NeMo Relay 0.11.0 separately. Relay selects the bundle for your operating system, CPU, and CLI libc target. The plugin is a native library loaded into the Relay process; install it only from a source you trust.

On Unix, the verification step also uses jq; PowerShell uses its built-in JSON parser. Use a fresh project directory and reserve local port 4040 for this test.

Relay does not require the model used below. The routing example selects a public build.nvidia.com model. For another provider, update the target model, the plugin and gateway endpoint URLs, and the credential environment variable together.

1

Install

Install Relay

Install the documented Relay release with the current installer script. The script comes from main; NEMO_RELAY_VERSION selects the CLI release.

curl -fsSL https://raw.githubusercontent.com/NVIDIA/NeMo-Relay/main/install.sh | NEMO_RELAY_VERSION=0.11.0 sh
export PATH="$HOME/.local/bin:$PATH"

Install the Switchyard Plugin

Install the published bundle for your user account. Keep it disabled until you have configured routing and trust:

nemo-relay plugins install --user github:NVIDIA/NeMo-Relay-Plugins@switchyard-plugin-0.3.2 --no-enable
nemo-relay plugins list --user

Relay downloads and verifies the platform bundle, registers nvidia.switchyard, and retains it across runs. The list includes its scope, enabled state, validation and policy status, and release source. Before configuration, the plugin is disabled and may report configuration or policy errors.

For a system-wide deployment, use --global instead of --user for installation, listing, and uninstallation, and configure the system plugins.toml below. Run global management commands with an account that can write that directory and authenticate to GitHub. Install in one scope to avoid duplicate plugin IDs.

2

Configure

Edit the plugins.toml in the scope you selected:

ScopeConfiguration file
User$XDG_CONFIG_HOME/nemo-relay/plugins.toml, or ~/.config/nemo-relay/plugins.toml when XDG_CONFIG_HOME is unset
Global on Unix/etc/nemo-relay/plugins.toml
Global on Windows%ProgramData%\nemo-relay\plugins.toml

Keep the installer’s top-level settings and [[plugins.dynamic]] entry, including its managed manifest path. Replace any generated empty [plugins.dynamic.config] table with the settings below. Add the following configuration to that entry and merge the observability and trust settings into the same file. Do not create a second [[plugins.dynamic]] entry for Switchyard or overwrite other plugin settings. The [plugins.dynamic.config] tables belong to the installer’s Switchyard entry; place them after its manifest line and before any other [[plugins.dynamic]] entry.

plugins.toml settings
# SPDX-FileCopyrightText: Copyright (c) 2026, NVIDIA CORPORATION & AFFILIATES. All rights reserved.
# SPDX-License-Identifier: Apache-2.0
# Add these tables to the installed Switchyard entry; retain its manifest path.
[plugins.dynamic.config]
priority = 0
[plugins.dynamic.config.switchyard_config]
schema_version = 1
[plugins.dynamic.config.switchyard_config.llm_clients.nvidia]
format = "openai_chat"
base_url = "https://integrate.api.nvidia.com/v1"
api_key_env = "NVIDIA_API_KEY"
[plugins.dynamic.config.switchyard_config.targets.default]
id = "nvidia/nemotron-3-nano-omni-30b-a3b-reasoning"
llm_client = "nvidia"
[plugins.dynamic.config.switchyard_config.routes.default]
id = "switchyard/quickstart"
type = "passthrough"
target = "default"
[plugins.policy.overrides."nvidia.switchyard"]
attestation = "integrity_only"
[[components]]
kind = "observability"
enabled = true
[components.config]
version = 4
enable_full_payloads = false
[components.config.atof]
enabled = true
[[components.config.atof.sinks]]
type = "file"
output_directory = "logs"
filename = "events.jsonl"
mode = "overwrite"

The request uses the route ID switchyard/quickstart. The plugin configuration maps that route to the target model nvidia/nemotron-3-nano-omni-30b-a3b-reasoning through the NVIDIA client. This single-target passthrough route demonstrates model resolution; it does not choose dynamically between multiple candidate models.

The installer checksums and integrity_only policy check artifact integrity, not independent publisher authenticity: the archive and checksum come from the same source. Only load a bundle whose publisher you trust. The example uses a server-owned API key: the native plugin does not support forwarding each caller’s credential to the provider. Protect logs/events.jsonl; Relay events can contain request or response data even with full payload export disabled.

3

Run

Create a working directory for the response and local event log:

mkdir relay-switchyard
cd relay-switchyard

In a gateway terminal in this directory, set your API key, enable the configured plugin, and validate it. Start Relay without --config or --plugin-config-path so it discovers the user and system plugin files. An explicit configuration path replaces user plugin discovery; see Configure.

export NVIDIA_API_KEY="your-api-key"
export NEMO_RELAY_OPENAI_BASE_URL="https://integrate.api.nvidia.com/v1"

Enable and validate the plugin, then start Relay:

rm -f -- response.json logs/events.jsonl &&
nemo-relay plugins enable nvidia.switchyard &&
nemo-relay plugins validate nvidia.switchyard &&
nemo-relay --bind 127.0.0.1:4040

Leave Relay running and wait until it reports that it is listening on 127.0.0.1:4040. In a second terminal, from the same directory, send one request to the route ID switchyard/quickstart and save the response:

curl -fsS --max-time 120 --output response.json http://127.0.0.1:4040/v1/chat/completions \
  -H 'Content-Type: application/json' \
  -d '{"model":"switchyard/quickstart","messages":[{"role":"user","content":"Reply with RELAY_QUICKSTART_RESPONSE_OK."}],"stream":false}'
4

Verify

The verification command prints RELAY_QUICKSTART_RESPONSE_OK and RELAY_QUICKSTART_EVENTS_OK.

Run the verification checks

After the request finishes, stop Relay with Ctrl+C in its terminal to flush the event file. Then check the assistant response and confirm that Relay recorded a mark named switchyard.routing.decision with the configured target in data.selected_model:

jq -e '.choices[0].message | .role == "assistant" and
  (.content | contains("RELAY_QUICKSTART_RESPONSE_OK"))' response.json > /dev/null &&
jq -se 'length > 0 and
  all(.[]; .atof_version == "0.1" and (.kind == "scope" or .kind == "mark")) and
  any(.[]; .kind == "mark" and .name == "switchyard.routing.decision" and
    .data.selected_model == "nvidia/nemotron-3-nano-omni-30b-a3b-reasoning")' \
  logs/events.jsonl > /dev/null &&
printf 'RELAY_QUICKSTART_RESPONSE_OK\nRELAY_QUICKSTART_EVENTS_OK\n'

The verification command prints the following output when its checks pass:

RELAY_QUICKSTART_RESPONSE_OK
RELAY_QUICKSTART_EVENTS_OK

This is a live provider test; it does not assert identical model wording, timestamps, token counts, or event UUIDs.

Inspect the Routing Decision

The caller requested switchyard/quickstart, and the verification above requires Switchyard’s decision to name the configured target model. Inspect the decision to see what served the request:

jq 'select(.kind == "mark" and .name == "switchyard.routing.decision") |
  {selected_model: .data.selected_model, served_model: .data.served_model,
   algorithm: .data.algorithm, fallback_used: .data.fallback_used}' logs/events.jsonl

selected_model is Switchyard’s chosen target. served_model is the model reported by the provider, when available; fallback_used compares those model identities. algorithm identifies the routing method. Use these values to explain the routing result rather than relying only on an OK marker.

Uninstall

After stopping Relay, remove the managed plugin using its ID:

nemo-relay plugins uninstall --user nvidia.switchyard
nemo-relay plugins list --user

For a system-wide installation, use --global in both commands. Uninstall removes registration and the managed bundle. Review any remaining observability and trust settings in plugins.toml before reusing the configuration.

For More Details

For more details, refer to: