Provider Codecs

Use this guide when a framework integration needs NeMo Relay middleware, intercepts, or subscribers to reason about provider-specific LLM payloads through a stable annotated shape.

What You Build

You will attach request and response codecs to a managed LLM wrapper so that:

• Request intercepts can work with normalized instructions, messages, model names, tools, generation parameters, and tagged provider-specific fields
• The provider callback still receives the provider payload that the framework expects
• Response subscribers can receive normalized response annotations without changing the caller-visible provider response

Before You Start

You need:

• A framework LLM boundary that can call llm.execute, llm.stream_execute, llmCallExecute, typedLlmExecute, or llm_call_execute.
• A provider payload that is JSON-compatible.
• A matching built-in provider codec, or a custom codec that can preserve unmodeled provider fields.
• Request intercepts or subscribers that benefit from normalized request or response data.

What Provider Codecs Are

A provider codec is a pure data translator at the NeMo Relay LLM boundary.

• An LLM request codec converts a raw provider request into a normalized annotated request, then encodes any annotated edits back into the original provider request.
• An LLM response codec converts a raw provider response into a normalized response annotation for lifecycle events.

Provider codecs let framework code keep using provider-native payloads while NeMo Relay middleware works against a shared annotated model. For application-facing type conversion, use Using Codecs.

How Provider Codecs Work

When a managed LLM call has a request codec:

1. NeMo Relay calls decode before LLM request intercepts run.
2. Request intercepts receive both the raw request and the annotated request.
3. Intercepts edit provider-body fields through the annotated request and may edit transport headers through the raw request. Raw request.content is read-only while the codec is active.
4. NeMo Relay calls encode to merge the annotated request back into the original raw request.
5. Execution intercepts and the provider callback receive the encoded provider request.

If a codec-aware intercept changes raw request.content or omits the returned annotation, Relay rejects the outcome before creating the LLM lifecycle. When no request codec is active, the raw request remains fully writable and is the provider-visible source of truth.

Gateway interceptors must preserve the request’s effective stream mode. The gateway chooses buffered or streaming response handling before request intercepts run and rejects changes between those modes before contacting the provider.

When a managed LLM call has a response codec, NeMo Relay decodes the raw provider response for observability and attaches the result to the emitted LLM end event. The response codec does not rewrite the value returned to the application. Use Provider Response Codecs for response-only behavior and custom response codec examples.

The built-in codecs guarantee encode(decode(original), original) == original at the JSON-value level. Their encoders compare an edited annotation with a decoded baseline and patch only changed fields. As a result, unchanged string-versus-block content, explicit nulls, legacy field names, ordered input items, metadata, and unknown fields remain untouched. Removing an unknown top-level key from extra removes it from the provider payload; adding or changing a key overlays it.

Use the annotated request surfaces according to their ownership:

• instructions for Anthropic system or OpenAI Responses instructions.
• Portable messages, content parts, function calls and results, tools, and tool_choice when the component has shared semantics.
• Tagged api_specific fields for modeled controls that belong to Anthropic Messages, OpenAI Chat Completions, or OpenAI Responses.
• { provider, kind, value } native components for provider-only input items, blocks, tools, tool choices, and future union members. value is the exact provider JSON.
• Top-level extra only for unknown future fields.

Provider-native components are intentionally surface-bound. If an intercept places an Anthropic native block in an OpenAI request, or makes a portable edit that the target API cannot encode, Relay returns an explicit error before the provider callback.

Built-in Provider Codecs

Use the built-in provider codecs when the framework payload already matches a supported provider API:

• OpenAIChatCodec: OpenAI Chat Completions-compatible requests and responses.
• OpenAIResponsesCodec: OpenAI Responses-compatible requests and responses.
• AnthropicMessagesCodec: Anthropic Messages-compatible requests and responses.

Provider Codec Roles

Provider codecs have separate request and response roles:

• LlmCodec decodes provider-specific requests into an annotated request form and encodes edits back into the provider request.
• LlmResponseCodec decodes raw provider responses into annotated response data for lifecycle events.

The built-in provider codecs expose the same core methods:

Choose the provider codec that matches the payload shape the framework already sends to the provider. Do not translate to a different provider shape only to make the codec fit.

The nemo-relay gateway selects these request codecs automatically for /v1/messages, /v1/chat/completions, and /v1/responses, for both buffered and streaming calls. Count-token, model, probe, and non-LLM passthrough routes do not use request codecs.

Example: Add a System Message with a Provider Codec

This example uses a request intercept to edit the normalized request. The codec writes the edited messages back into the provider payload before the provider callback runs.

1
import nemo_relay
2
from nemo_relay import LLMRequest
3
from nemo_relay.codecs import OpenAIChatCodec
4
5
def add_system_message(_name, request, annotated):
6
    if annotated is None:
7
        return nemo_relay.LLMRequestInterceptOutcome(request)
8
9
    # Attributes of the annotated request can be re-assigned, but cannot be modified in-place.
10
    # For example `annotated.messages.append(...)` would not work, but re-assigning
11
    # `annotated.messages = annotated.messages + [...]` does work.
12
    annotated.messages = [
13
        {"role": "system", "content": "Answer with concise technical detail."},
14
        *annotated.messages,
15
    ]
16
    return nemo_relay.LLMRequestInterceptOutcome(request, annotated)
17
18
nemo_relay.intercepts.register_llm_request(
19
    "framework.add_system_message",
20
    10,
21
    False,
22
    add_system_message,
23
)
24
25
async def invoke_provider(request: LLMRequest):
26
    return {
27
        "id": "chatcmpl-demo",
28
        "model": request.content["model"],
29
        "choices": [
30
            {"message": {"role": "assistant", "content": "Codec-enabled response."}},
31
        ],
32
    }
33
34
codec = OpenAIChatCodec()
35
request = LLMRequest(
36
    {},
37
    {
38
        "model": "gpt-4o-mini",
39
        "messages": [{"role": "user", "content": "Explain scopes."}],
40
        "temperature": 0.2,
41
    },
42
)
43
44
response = await nemo_relay.llm.execute(
45
    "openai-chat",
46
    request,
47
    invoke_provider,
48
    model_name="gpt-4o-mini",
49
    codec=codec,
50
    response_codec=codec,
51
)

Example: Edit Provider-Specific and Native Data

The tagged api_specific object is mutable. Python getters return JSON values, so retrieve, edit, and reassign the value. The same reassignment pattern works for a provider-native component inside messages.

1
def tune_responses_request(_name, request, annotated):
2
    if annotated is None:
3
        return nemo_relay.LLMRequestInterceptOutcome(request)
4
5
    api_specific = dict(annotated.api_specific or {})
6
    if api_specific.get("api") == "openai_responses":
7
        api_specific["background"] = True
8
        api_specific["stream_options"] = {"include_obfuscation": False}
9
        annotated.api_specific = api_specific
10
11
    messages = list(annotated.messages)
12
    for index, item in enumerate(messages):
13
        if (
14
            item.get("role") == "provider_native"
15
            and item.get("provider") == "openai_responses"
16
            and item.get("kind") == "reasoning"
17
        ):
18
            native_value = dict(item["value"])
19
            native_value["encrypted_content"] = "updated-ciphertext"
20
            messages[index] = {**item, "value": native_value}
21
    annotated.messages = messages
22
23
    # request.headers remains mutable; request.content is read-only here.
24
    request.headers["x-request-policy"] = "codec-aware"
25
    return nemo_relay.LLMRequestInterceptOutcome(request, annotated)

Native values are not normalized across providers. Keep the provider tag unchanged unless you also replace the entire component with a representation that the target codec owns.

Example: Write a Custom Framework Codec

Use a custom codec when a framework uses a payload shape that does not directly match a built-in provider format. The codec decodes the framework shape into AnnotatedLLMRequest, and encodes edits back into the framework shape.

1
from nemo_relay import AnnotatedLLMRequest, LLMRequest
2
from nemo_relay.codecs import LlmCodec
3
4
class FrameworkChatCodec(LlmCodec):
5
    def decode(self, request: LLMRequest) -> AnnotatedLLMRequest:
6
        content = request.content
7
        params = {}
8
        if "temperature" in content:
9
            params["temperature"] = content["temperature"]
10
11
        return AnnotatedLLMRequest(
12
            content.get("turns", []),
13
            model=content.get("model_name"),
14
            params=params or None,
15
            tools=content.get("tool_specs"),
16
            extra={"tenant_id": content.get("tenant_id")},
17
        )
18
19
    def encode(self, annotated: AnnotatedLLMRequest, original: LLMRequest) -> LLMRequest:
20
        content = {
21
            **original.content,
22
            "turns": annotated.messages,
23
        }
24
        if annotated.model is not None:
25
            content["model_name"] = annotated.model
26
        if annotated.params:
27
            content.update(annotated.params)
28
        if annotated.tools is not None:
29
            content["tool_specs"] = annotated.tools
30
        return LLMRequest(original.headers, content)

Validation Checklist

Use this checklist to confirm the implementation preserves the expected runtime contract.

• Intercepts receive annotated only when the managed call supplies a request codec.
• An unchanged annotation encodes to the exact original JSON value.
• Portable edits preserve adjacent provider-native components and unknown fields.
• Provider-specific edits use api_specific; extra is limited to unknown future top-level fields.
• Provider-mismatched native components fail before the provider callback.
• Response codecs are used only for event annotations, not caller-visible response rewriting.
• Codec implementations are pure data transforms and do not perform provider I/O.
• Framework-owned clients, sockets, streams, callbacks, and file handles stay outside codec results.

Migrate Raw Gateway Body Mutation

The 0.6 request annotation expansion uses nemo.relay.AnnotatedLlmRequest@2 and nemo.relay.LlmRequestInterceptOutcome@2 at dynamic plugin boundaries. Rebuild Rust native plugins and Rust grpc-v1 workers against NeMo Relay 0.6, and update Python workers to the 0.6 worker SDK. A dynamic plugin that registers an LLM request intercept must set compat.relay = ">=0.6,<1.0", or another range that excludes Relay 0.5. Relay enforces this floor when the intercept is registered.

Gateway interceptors that mutate provider request bodies must now edit the annotation on the three generation routes. For example, replace request.content["messages"] = messages with annotated.messages = messages, and return the edited annotation with the raw request. Keep transport-only changes in request.headers. Relay rejects a raw body edit while the route codec is active, so the provider never receives an ambiguous mix of raw and annotated changes.

Next Steps

Use these links to continue from this workflow into the next related task.

• Use Using Codecs for typed value codecs.
• Use Provider Response Codecs for response-only annotations and subscriber examples.
• Use Add Middleware before adding request transforms.
• Use Handle Non-Serializable Data when the framework boundary includes SDK objects or streams that cannot pass through JSON payloads.