Provider Codecs

Use this guide when a framework integration needs NeMo Relay middleware, intercepts, or subscribers to reason about provider-specific LLM payloads through a stable annotated shape.

What You Build

You will attach request and response codecs to a managed LLM wrapper so that:

• Request intercepts can work with normalized messages, model names, tools, generation parameters, and provider-specific extras
• The provider callback still receives the provider payload that the framework expects
• Response subscribers can receive normalized response annotations without changing the caller-visible provider response

Before You Start

You need:

• A framework LLM boundary that can call llm.execute, llm.stream_execute, llmCallExecute, typedLlmExecute, or llm_call_execute.
• A provider payload that is JSON-compatible.
• A matching built-in provider codec, or a custom codec that can preserve unmodeled provider fields.
• Request intercepts or subscribers that benefit from normalized request or response data.

What Provider Codecs Are

A provider codec is a pure data translator at the NeMo Relay LLM boundary.

• An LLM request codec converts a raw provider request into a normalized annotated request, then encodes any annotated edits back into the original provider request.
• An LLM response codec converts a raw provider response into a normalized response annotation for lifecycle events.

Provider codecs let framework code keep using provider-native payloads while NeMo Relay middleware works against a shared annotated model. For application-facing type conversion, use Using Codecs.

How Provider Codecs Work

When a managed LLM call has a request codec:

1. NeMo Relay calls decode before LLM request intercepts run.
2. Request intercepts receive both the raw request and the annotated request.
3. Intercepts may edit the raw request, the annotated request, or both.
4. NeMo Relay calls encode to merge the annotated request back into the original raw request.
5. Execution intercepts and the provider callback receive the encoded provider request.

When a managed LLM call has a response codec, NeMo Relay decodes the raw provider response for observability and attaches the result to the emitted LLM end event. The response codec does not rewrite the value returned to the application. Use Provider Response Codecs for response-only behavior and custom response codec examples.

Codec implementations should preserve fields they do not understand. Treat encode as a merge operation over the original provider payload, not as a full replacement.

Built-in Provider Codecs

Use the built-in provider codecs when the framework payload already matches a supported provider API:

• OpenAIChatCodec: OpenAI Chat Completions-compatible requests and responses.
• OpenAIResponsesCodec: OpenAI Responses-compatible requests and responses.
• AnthropicMessagesCodec: Anthropic Messages-compatible requests and responses.

Provider Codec Roles

Provider codecs have separate request and response roles:

• LlmCodec decodes provider-specific requests into an annotated request form and encodes edits back into the provider request.
• LlmResponseCodec decodes raw provider responses into annotated response data for lifecycle events.

The built-in provider codecs expose the same core methods:

Choose the provider codec that matches the payload shape the framework already sends to the provider. Do not translate to a different provider shape only to make the codec fit.

Example: Add a System Message with a Provider Codec

This example uses a request intercept to edit the normalized request. The codec writes the edited messages back into the provider payload before the provider callback runs.

1
import nemo_relay
2
from nemo_relay import LLMRequest
3
from nemo_relay.codecs import OpenAIChatCodec
4
5
def add_system_message(_name, request, annotated):
6
    if annotated is None:
7
        return request, annotated
8
9
    # Attributes of the annotated request can be re-assigned, but cannot be modified in-place.
10
    # For example `annotated.messages.append(...)` would not work, but re-assigning
11
    # `annotated.messages = annotated.messages + [...]` does work.
12
    annotated.messages = [
13
        {"role": "system", "content": "Answer with concise technical detail."},
14
        *annotated.messages,
15
    ]
16
    return request, annotated
17
18
nemo_relay.intercepts.register_llm_request(
19
    "framework.add_system_message",
20
    10,
21
    False,
22
    add_system_message,
23
)
24
25
async def invoke_provider(request: LLMRequest):
26
    return {
27
        "id": "chatcmpl-demo",
28
        "model": request.content["model"],
29
        "choices": [
30
            {"message": {"role": "assistant", "content": "Codec-enabled response."}},
31
        ],
32
    }
33
34
codec = OpenAIChatCodec()
35
request = LLMRequest(
36
    {},
37
    {
38
        "model": "gpt-4o-mini",
39
        "messages": [{"role": "user", "content": "Explain scopes."}],
40
        "temperature": 0.2,
41
    },
42
)
43
44
response = await nemo_relay.llm.execute(
45
    "openai-chat",
46
    request,
47
    invoke_provider,
48
    model_name="gpt-4o-mini",
49
    codec=codec,
50
    response_codec=codec,
51
)

Example: Write a Custom Framework Codec

Use a custom codec when a framework uses a payload shape that does not directly match a built-in provider format. The codec decodes the framework shape into AnnotatedLLMRequest, and encodes edits back into the framework shape.

1
from nemo_relay import AnnotatedLLMRequest, LLMRequest
2
from nemo_relay.codecs import LlmCodec
3
4
class FrameworkChatCodec(LlmCodec):
5
    def decode(self, request: LLMRequest) -> AnnotatedLLMRequest:
6
        content = request.content
7
        params = {}
8
        if "temperature" in content:
9
            params["temperature"] = content["temperature"]
10
11
        return AnnotatedLLMRequest(
12
            content.get("turns", []),
13
            model=content.get("model_name"),
14
            params=params or None,
15
            tools=content.get("tool_specs"),
16
            extra={"tenant_id": content.get("tenant_id")},
17
        )
18
19
    def encode(self, annotated: AnnotatedLLMRequest, original: LLMRequest) -> LLMRequest:
20
        content = {
21
            **original.content,
22
            "turns": annotated.messages,
23
        }
24
        if annotated.model is not None:
25
            content["model_name"] = annotated.model
26
        if annotated.params:
27
            content.update(annotated.params)
28
        if annotated.tools is not None:
29
            content["tool_specs"] = annotated.tools
30
        return LLMRequest(original.headers, content)

Validation Checklist

Use this checklist to confirm the implementation preserves the expected runtime contract.

• Intercepts receive annotated only when the managed call supplies a request codec.
• encode preserves provider fields that the annotated model does not represent.
• Response codecs are used only for event annotations, not caller-visible response rewriting.
• Codec implementations are pure data transforms and do not perform provider I/O.
• Framework-owned clients, sockets, streams, callbacks, and file handles stay outside codec results.

Next Steps

Use these links to continue from this workflow into the next related task.

• Use Using Codecs for typed value codecs.
• Use Provider Response Codecs for response-only annotations and subscriber examples.
• Use Add Middleware before adding request transforms.
• Use Handle Non-Serializable Data when the framework boundary includes SDK objects or streams that cannot pass through JSON payloads.