Switchyard Configuration

Experimental NeMo Relay 0.6.0 integration: This release calls a separately running Switchyard Decision API.

Start with the Switchyard 0.6.0 setup and validation guide before using this page as the complete configuration reference.

The Switchyard component connects Relay’s CLI gateway to a separately running Switchyard Decision API. Relay owns provider credentials, backend endpoints, dispatch, retries, and trusted fallbacks. The Switchyard service selects a backend. Relay validates the decision, translates requests and responses in process through the switchyard-translation library, and dispatches directly to the selected Relay-owned target.

How the Configuration Fits Together

Configure the integration in four core steps. Profiles that use ATOF history add the fifth requirement:

Set decision_api_url and authentication so Relay can call the Switchyard Decision API.
Select the Switchyard routing profile with decision_profile_id.
Add a Relay-owned targets binding for every Switchyard backend_id that the profile can return. Relay validates the returned backend ID, model, protocol, and endpoint exactly before dispatch.
Choose a trusted same-protocol target for each inbound protocol under default_targets. Relay uses these defaults for observe-only traffic and fail-open dispatch.
Configure profiles that depend on ATOF history, including StageRouter, with context_mode = "atof_required" and atof_endpoint_name, then configure the matching observability stream sink. Relay fails startup validation if the named sink is missing or invalid.

The following is a complete component configuration. The referenced environment variables must be set before Relay starts.

1 version = 1
2 
3 [[components]]
4 kind = "switchyard"
5 enabled = true
6 
7 [components.config]
8 version = 1
9 mode = "enforce"
10 priority = 0
11 decision_api_url = "http://127.0.0.1:4000/v1/routing/decision"
12 decision_profile_id = "stage-router"
13 request_materialization = "recent_message_window"
14 context_mode = "atof_required"
15 atof_endpoint_name = "switchyard"
16 decision_timeout_millis = 25
17 max_retries = 3
18 recent_message_count = 8
19 enabled_inbound_profiles = ["openai_chat", "openai_responses", "anthropic_messages"]
20 
21 [components.config.decision_header_env]
22 authorization = "SWITCHYARD_AUTHORIZATION"
23 
24 [components.config.default_targets]
25 openai_chat = "openai-default"
26 openai_responses = "responses-default"
27 anthropic_messages = "anthropic-default"
28 
29 [components.config.targets.openai-default]
30 model = "provider/efficient-chat"
31 protocol = "openai_chat"
32 endpoint = "/v1/chat/completions"
33 base_url = "https://provider.example.com"
34 
35 [components.config.targets.openai-default.header_env]
36 authorization = "OPENAI_PROVIDER_AUTHORIZATION"
37 
38 [components.config.targets.responses-default]
39 model = "provider/efficient-responses"
40 protocol = "openai_responses"
41 endpoint = "/v1/responses"
42 base_url = "https://provider.example.com"
43 
44 [components.config.targets.responses-default.header_env]
45 authorization = "OPENAI_PROVIDER_AUTHORIZATION"
46 
47 [components.config.targets.anthropic-default]
48 model = "provider/efficient-messages"
49 protocol = "anthropic_messages"
50 endpoint = "/v1/messages"
51 base_url = "https://provider.example.com"
52 
53 [components.config.targets.anthropic-default.header_env]
54 x-api-key = "ANTHROPIC_PROVIDER_API_KEY"
55 
56 [components.config.targets.capable-chat]
57 model = "provider/capable-chat"
58 protocol = "openai_chat"
59 endpoint = "/v1/chat/completions"
60 base_url = "https://provider.example.com"
61 
62 [components.config.targets.capable-chat.header_env]
63 authorization = "OPENAI_PROVIDER_AUTHORIZATION"
64 
65 [[components]]
66 kind = "observability"
67 enabled = true
68 
69 [components.config]
70 version = 2
71 
72 [components.config.atof]
73 enabled = true
74 
75 [[components.config.atof.sinks]]
76 type = "stream"
77 name = "switchyard"
78 url = "http://127.0.0.1:4000/v1/atof/events"
79 transport = "http_post"
80 field_name_policy = "preserve"
81 
82 [components.config.atof.sinks.header_env]
83 authorization = "SWITCHYARD_AUTHORIZATION"

Decision API and Rollout Options

The following table describes the Decision API and rollout settings:

Option	Default	Description and constraints
`version`	`1`	Component configuration version.
`mode`	`"enforce"`	`enforce` applies valid decisions. `observe_only` records the hypothetical decision but dispatches the trusted same-protocol default once.
`priority`	`0`	Component execution priority.
`decision_api_url`	Required	HTTP or HTTPS URL for the Switchyard Decision API.
`decision_profile_id`	Required	Non-empty Switchyard profile identifier sent with each routing request.
`decision_timeout_millis`	`25`	Decision API timeout in milliseconds; must be greater than zero.
`decision_headers`	Empty	Optional non-sensitive static Decision API headers.
`decision_header_env`	Empty	Map of header names to environment-variable names for sensitive Decision API headers. Referenced values must be present and non-blank.

Do not define the same header, including case variants, in both a static header map and its environment map. Keep bearer tokens, API keys, cookies, and other secrets in environment variables rather than tracked configuration.

Request Materialization

Every Decision API request contains routing identity, inbound protocol, a request summary, and the attempt number. request_materialization controls how much additional request content is included.

Mode	Material sent to Switchyard
`none`	No `current_request`; use identity, protocol, summary, and attempt metadata only.
`summary_only`	No `current_request`; explicitly selects summary-based routing.
`latest_user_prompt`	A provider-valid body containing the latest user material plus an explicit prompt field. The request fails open if no user message is available.
`recent_message_window`	The most recent normalized messages, bounded by `recent_message_count`, re-encoded for the inbound protocol.
`annotated_request`	The final intercepted provider body plus Relay’s normalized request representation.
`full_body`	The final intercepted provider request body.

Choose the least materialization needed by the selected profile. In particular, none and summary_only intentionally share the same current wire behavior but express different configuration intent.

Context, Retries, and Inbound Protocols

The following table describes context, retry, and inbound protocol settings:

Option	Default	Description and constraints
`context_mode`	Required	`payload_only` or `atof_required`. See below.
`max_retries`	`3`	Provider retries after the initial attempt; maximum `10`. Each retry obtains a new Switchyard decision.
`recent_message_count`	`8`	Message limit for `recent_message_window`; must be greater than zero.
`enabled_inbound_profiles`	All three supported protocols	Non-empty list containing `openai_chat`, `openai_responses`, and/or `anthropic_messages`.
`atof_endpoint_name`	Required for `atof_required`	Name of exactly one stream sink under the observability component’s `atof.sinks` list.

payload_only supports request-aware routing without an ATOF dependency. Relay can create request-scoped synthetic identity when stable identity is not available.

atof_required is for profiles that classify from accumulated ATOF context. It requires stable session and request identity plus an enabled, authenticated Relay HTTP ATOF stream sink whose name matches atof_endpoint_name. The sink specifies the Switchyard ingestion URL, must use transport = "http_post", and must set field_name_policy = "preserve". Relay rejects missing or duplicate named sinks during startup validation. A local ATOF file sink does not satisfy this delivery requirement by itself.

Target Bindings and Trusted Defaults

The following table describes target bindings and trusted defaults:

Option	Required fields	Purpose
`targets.<backend_id>`	`model`, `protocol`, `endpoint`, `base_url`	Binds an exact Switchyard backend ID to a Relay-owned provider destination.
`targets.<backend_id>.headers`	None	Optional non-sensitive static provider headers.
`targets.<backend_id>.header_env`	None	Optional provider header names mapped to environment-variable names. Referenced values must be present and non-blank.
`default_targets.openai_chat`	Target using `openai_chat`	Trusted fallback for OpenAI Chat Completions requests.
`default_targets.openai_responses`	Target using `openai_responses`	Trusted fallback for OpenAI Responses requests.
`default_targets.anthropic_messages`	Target using `anthropic_messages`	Trusted fallback for Anthropic Messages requests.

Do not define the same target header, including case variants, in both headers and header_env.

Protocol and endpoint pairs are fixed:

Protocol	Endpoint
`openai_chat`	`/v1/chat/completions`
`openai_responses`	`/v1/responses`
`anthropic_messages`	`/v1/messages`

Each protocol in enabled_inbound_profiles requires a corresponding default_targets entry. Defaults for disabled protocols can be omitted. Each configured default must refer to a target with the matching protocol. Duplicate exact bindings for model, protocol, endpoint, and base URL are rejected.

Recommended Recipes

Observe-Only Rollout

Use observe-only mode to verify decisions and routing marks without sending traffic to the selected target:

1 [components.config]
2 mode = "observe_only"
3 context_mode = "payload_only"
4 request_materialization = "summary_only"

Relay calls the Decision API and records the hypothetical selection, but sends the request to the inbound protocol’s trusted default once. Routing retries are not applied.

Payload-Only Request-Aware Routing

Use recent request content for routing without operating the ATOF accumulator:

1 [components.config]
2 mode = "enforce"
3 context_mode = "payload_only"
4 request_materialization = "recent_message_window"
5 recent_message_count = 8

ATOF-Backed StageRouter Routing

Use ATOF context when StageRouter should classify from accumulated session signals:

1 [components.config]
2 mode = "enforce"
3 decision_profile_id = "stage-router"
4 context_mode = "atof_required"
5 request_materialization = "summary_only"
6 atof_endpoint_name = "switchyard"

Configure the matching observability component with a named stream sink:

1 [[components]]
2 kind = "observability"
3 enabled = true
4 
5 [components.config]
6 version = 2
7 
8 [components.config.atof]
9 enabled = true
10 
11 [[components.config.atof.sinks]]
12 type = "stream"
13 name = "switchyard"
14 url = "http://127.0.0.1:4000/v1/atof/events"
15 transport = "http_post"
16 field_name_policy = "preserve"
17 
18 [components.config.atof.sinks.header_env]
19 authorization = "SWITCHYARD_AUTHORIZATION"

Runtime and Fail-Open Behavior

The integration handles routing and failures as follows:

Relay validates the decision’s backend ID, model, protocol, and endpoint against its target binding before dispatch. Unknown backends, malformed decisions, and target drift fail open.
Decision API errors are not recursively retried. Relay immediately dispatches the trusted same-protocol default.
Retryable provider connection, timeout, status, context-window, and model-unavailable failures can obtain a new decision, bounded by max_retries.
Non-retryable provider failures use the trusted fallback immediately. After retry exhaustion, Relay dispatches that fallback once.
A streaming request can retry only before receiving its first upstream item. After the stream commits, Relay propagates later failures without another dispatch.
Relay forwards provider-specific extensions unchanged when every configured target uses the inbound protocol. If any target uses another protocol, nonportable extensions fail open to the trusted same-protocol default before the Decision API call. Request translation failures before provider dispatch and buffered response translation failures also fail open. After a streaming response emits its first item, later provider or translation failures propagate without another dispatch.
Routing-mark delivery is best-effort and does not change provider results.

Experimental Limitations

Consider the following experimental limitations before adopting the plugin:

The plugin is published as nemo-relay-switchyard but is excluded from default Relay CLI builds. Build the CLI with cargo build -p nemo-relay-cli --features switchyard.
The routing boundary is currently service-based. Relay calls Switchyard’s HTTP Decision API but does not start or supervise the service. Relay derives /health from decision_api_url and fails plugin activation unless the service returns {"status":"ok"}. Decision API failures after activation fail open to the trusted same-protocol default.
ATOF-backed profiles additionally depend on Switchyard’s HTTP ingestion and accumulator runtime, stable request and session identity, and one named, authenticated Relay ATOF HTTP stream sink. A local file sink does not populate the Switchyard accumulator.
Provider-protocol translation runs in Relay’s process through the switchyard-translation Rust library. Routing decisions and ATOF accumulation remain out of process.
The Rust Switchyard component is currently registered only by Relay’s CLI gateway. Hermes native plugins and the LangChain, LangGraph, and Deep Agents integrations do not load it in process. They receive Switchyard routing only when their provider traffic is explicitly sent through the Relay gateway.
The Switchyard examples are manual compatibility and trajectory workflows, not production deployment orchestration.
Compatibility is pinned to Switchyard commit 8f9db9a6 on topic/nemo-relay-integration. Testing another revision is deliberate and can expose contract drift.
The Decision API and service/library boundary are experimental.
Nonportable provider extensions in configurations that can translate across protocols, malformed decisions, service failures, pre-dispatch translation failures, buffered-response translation failures, and target drift fail open to Relay’s trusted same-protocol default. Failures after a streaming response commits propagate without another dispatch.
Secrets must remain environment-referenced. The examples generate ephemeral local credentials and are not credential-management guidance.

Run nemo-relay doctor to validate component configuration before startup. For the pinned local service and manual compatibility workflow, refer to the Switchyard integration examples.