OpenInference

Use the openinference section when you want NeMo Relay lifecycle events exported as OTLP trace spans with OpenInference-oriented semantics.

OpenInference export maps model-centric payloads directly into trace attributes. Scope, tool, and LLM start inputs become input.value. End outputs become output.value. LLM usage metadata maps to token-count attributes when provider responses include usage information. For LLM spans, NeMo Relay emits flattened request and response message attributes from typed codec annotations and supported replay request/response payloads. Typed codec annotations also provide tool schema, finish-reason, and invocation-parameter attributes when those fields are available.

plugins.toml Example

Add the following OpenInference exporter configuration to plugins.toml:

1
version = 1
2
3
[[components]]
4
kind = "observability"
5
enabled = true
6
7
[components.config]
8
version = 1
9
10
[components.config.openinference]
11
enabled = true
12
transport = "http_binary"
13
endpoint = "http://localhost:6006/v1/traces"
14
service_name = "agent-service"
15
service_namespace = "nemo"
16
service_version = "1.0.0"
17
instrumentation_scope = "nemo-relay-openinference"
18
timeout_millis = 3000
19
20
[components.config.openinference.headers]
21
authorization = "Bearer <token>"
22
23
[components.config.openinference.resource_attributes]
24
"deployment.environment" = "dev"

This configuration registers a plugin-owned OpenInference subscriber and sends OpenInference-style OTLP spans to Phoenix or another compatible backend.

Fields

OpenInference uses the same OTLP section shape as OpenTelemetry:

Expected Output

The backend should show OpenInference-oriented spans for scopes, tools, and LLM calls grouped by root scope. LLM usage metadata appears as token counters when provider responses include usage information. LLM request and response messages, system prompts, and model-emitted tool calls are emitted as flattened OpenInference attributes when available from codec annotations or supported replay request/response payloads. Tool schemas, finish reasons, and invocation parameters are emitted when typed codec annotations supply them. Exported LLM attributes exclude request headers and other non-observable transport metadata.

Each lifecycle span includes nemo_relay.uuid and nemo_relay.parent_uuid attributes. These values match ATIF step.extra.ancestry.function_id and step.extra.ancestry.parent_id for the same events. For plugin-managed ATIF, the trajectory-root span’s nemo_relay.uuid also matches the ATIF session_id. Backend-native trace_id and span_id values are not written into ATIF.

LLM token counts appear as llm.token_count.prompt, llm.token_count.completion, llm.token_count.total, and llm.token_count.prompt_details.cache_read/cache_write. Cost appears as USD-denominated llm.cost.total. Refer to Token and Cost Field Semantics for the full mapping.

Redact sensitive event payloads with sanitize guardrails before production export.

Plugin Configuration

Use plugin configuration when the application should let NeMo Relay own the OpenInference subscriber lifecycle. The following examples configure and activate the OpenInference exporter through each supported language binding.

validate() checks only the supplied in-memory object. initialize() also layers discovered plugins.toml configuration. For effective file-backed validation, refer to Plugin Configuration Files and run the gateway with the same configuration path that production uses.

1
import asyncio
2
3
from nemo_relay import plugin
4
from nemo_relay.observability import ComponentSpec, ObservabilityConfig, OtlpConfig
5
6
config = plugin.PluginConfig(
7
    components=[
8
        ComponentSpec(
9
            ObservabilityConfig(
10
                openinference=OtlpConfig(
11
                    enabled=True,
12
                    transport="http_binary",
13
                    endpoint="http://localhost:6006/v1/traces",
14
                    service_name="agent-service",
15
                    service_namespace="nemo",
16
                    service_version="1.0.0",
17
                    instrumentation_scope="nemo-relay-openinference",
18
                    resource_attributes={"deployment.environment": "dev"},
19
                    headers={"authorization": "Bearer <token>"},
20
                )
21
            )
22
        )
23
    ]
24
)
25
26
report = plugin.validate(config)
27
if any(diagnostic["level"] == "error" for diagnostic in report["diagnostics"]):
28
    raise RuntimeError(report["diagnostics"])
29
30
async def main():
31
    await plugin.initialize(config)
32
    try:
33
        # Run instrumented application work here.
34
        pass
35
    finally:
36
        plugin.clear()
37
38
asyncio.run(main())

Manual API

Use the manual subscriber API when you need an explicit subscriber name or direct force_flush control.

1
from nemo_relay import OpenInferenceConfig, OpenInferenceSubscriber
2
3
config = OpenInferenceConfig()
4
config.transport = "http_binary"
5
config.endpoint = "http://localhost:6006/v1/traces"
6
config.service_name = "agent-service"
7
config.set_resource_attribute("deployment.environment", "dev")
8
9
subscriber = OpenInferenceSubscriber(config)
10
subscriber.register("openinference-exporter")
11
12
# Run instrumented application work here.
13
14
subscriber.force_flush()
15
subscriber.deregister("openinference-exporter")
16
subscriber.shutdown()

Common Configuration and Runtime Issues

• transport is not http_binary or grpc.
• Headers or resource attributes are not string-to-string maps.
• The OpenInference feature is unavailable in the current build or target.
• Tool and LLM calls do not use managed helpers, so spans contain only scope lifecycle data.