OpenInference

Use the openinference section when you want NeMo Relay lifecycle events exported as OTLP trace spans with OpenInference-oriented semantics.

OpenInference export maps model-centric payloads directly into trace attributes. Scope, tool, and LLM start inputs become input.value. End outputs become output.value. LLM usage metadata maps to token-count attributes when provider responses include usage information. For LLM spans, NeMo Relay emits flattened request and response message attributes from typed codec annotations and supported replay request/response payloads. Typed codec annotations also provide tool schema, finish-reason, and invocation-parameter attributes when those fields are available.

plugins.toml Example

Add the following OpenInference exporter configuration to plugins.toml:

1
version = 1
2
3
[[components]]
4
kind = "observability"
5
enabled = true
6
7
[components.config]
8
version = 2
9
10
[components.config.openinference]
11
enabled = true
12
transport = "http_binary"
13
endpoint = "http://localhost:6006/v1/traces"
14
service_name = "agent-service"
15
service_namespace = "nemo"
16
service_version = "1.0.0"
17
instrumentation_scope = "nemo-relay-openinference"
18
timeout_millis = 3000
19
20
[components.config.openinference.headers]
21
authorization = "Bearer <token>"
22
23
[components.config.openinference.resource_attributes]
24
"deployment.environment" = "dev"

This configuration registers a plugin-owned OpenInference subscriber and sends OpenInference-style OTLP spans to Phoenix or another compatible backend.

Fields

OpenInference uses the same OTLP section shape as OpenTelemetry:

Expected Output

The backend should show OpenInference-oriented spans for scopes, tools, and LLM calls grouped by root scope. LLM usage metadata appears as token counters when provider responses include usage information. LLM request and response messages, system prompts, and model-emitted tool calls are emitted as flattened OpenInference attributes when available from codec annotations or supported replay request/response payloads. Tool schemas, finish reasons, and invocation parameters are emitted when typed codec annotations supply them. Exported LLM attributes exclude request headers and other non-observable transport metadata.

The default inherit projection preserves exporter-native mark handling: a mark with an active parent span is a span event, while an orphan mark is a standalone zero-duration mark span. mark_projection = "event" explicitly selects that representation. With mark_projection = "tool", each eligible mark becomes a zero-duration span (a child span when parent context is available) with openinference.span.kind = "TOOL" and the original mark payload, category, profile, timestamp, and parent identifiers retained as attributes. High-volume llm.chunk marks retain exporter-native handling by default and when excluded from tool projection.

Add other event names to mark_exclude_names for a backend to keep those marks in exporter-native form rather than visible tool children. The exclusion list affects only tool projection; it does not remove mark payload or metadata. Set mark_exclude_names = [] to disable the default llm.chunk exclusion.

Each lifecycle span includes nemo_relay.uuid and nemo_relay.parent_uuid attributes. These values match ATIF step.extra.ancestry.function_id and step.extra.ancestry.parent_id for the same events. For plugin-managed ATIF, the trajectory-root span’s nemo_relay.uuid also matches the ATIF session_id. Backend-native trace_id and span_id values are not written into ATIF.

Coding-agent trace roots also carry session.id, optional user.id, and nemo_relay.session.instance_id. The first value is the logical harness session ID, while the Relay instance ID is the existing root scope UUID shared by trace roots from one runtime session instance. These fields are root-only: filter matching roots first, then use the OpenInference backend’s trace_id to select child rows. OpenInference and generic OpenTelemetry export generate independent trace and span IDs, but retain the same Relay instance ID and nemo_relay.uuid values. The session.start mark carries the same correlation fields whether represented as a span event or an orphan zero-duration span.

LLM token counts appear as llm.token_count.prompt, llm.token_count.completion, llm.token_count.total, and llm.token_count.prompt_details.cache_read/cache_write. Cost appears as USD-denominated llm.cost.total. Refer to Token and Cost Field Semantics for the full mapping.

NeMo Relay projects top-level lifecycle payload fields to typed OTLP attributes with dotted names. Non-LLM start metadata and all end metadata use the openinference.metadata prefix, so metadata = { tenant = "acme" } becomes openinference.metadata.tenant = "acme". Non-LLM start events use nemo_relay.start.data, nemo_relay.start.input, and nemo_relay.handle_attributes. End events use nemo_relay.end.data and nemo_relay.end.output. LLM start events use OpenInference semantic input attributes instead of those generic start projections, and their final metadata comes from the end event. Mark data, metadata, attributes, and category-profile fields use the corresponding nemo_relay.mark.* prefixes.

Scalar strings, booleans, and numbers that fit an OTLP numeric type keep their types. NeMo Relay emits larger unsigned integers as strings. When a top-level field contains an object or array, NeMo Relay emits its value as a JSON string at that field’s dotted name. Nested null values remain in that string, but a top-level field whose value is null is omitted. NeMo Relay no longer emits the old aggregate *_json payload attributes. The exporter still keeps the OpenInference metadata JSON-string attribute for backend compatibility.

Configure an alias when a backend expects a different attribute name:

1
[components.config.openinference]
2
attribute_mappings = [
3
  { key = "openinference.metadata.tenant", alias = "tenant.id" },
4
]

The source attribute remains in the span. If the span already contains tenant.id, NeMo Relay keeps the existing value instead of replacing it with the alias.

Redact sensitive event payloads with sanitize guardrails before production export.

Plugin Configuration

Use plugin configuration when the application should let NeMo Relay own the OpenInference subscriber lifecycle. The following examples configure and activate the OpenInference exporter through each supported language binding.

validate() checks only the supplied in-memory object. initialize() also layers discovered plugins.toml configuration. For effective file-backed validation, refer to Plugin Configuration Files and run the gateway with the same configuration path that production uses.

1
import asyncio
2
3
from nemo_relay import plugin
4
from nemo_relay.observability import ComponentSpec, ObservabilityConfig, OtlpConfig
5
6
config = plugin.PluginConfig(
7
    components=[
8
        ComponentSpec(
9
            ObservabilityConfig(
10
                openinference=OtlpConfig(
11
                    enabled=True,
12
                    transport="http_binary",
13
                    endpoint="http://localhost:6006/v1/traces",
14
                    service_name="agent-service",
15
                    service_namespace="nemo",
16
                    service_version="1.0.0",
17
                    instrumentation_scope="nemo-relay-openinference",
18
                    resource_attributes={"deployment.environment": "dev"},
19
                    headers={"authorization": "Bearer <token>"},
20
                )
21
            )
22
        )
23
    ]
24
)
25
26
report = plugin.validate(config)
27
if any(diagnostic["level"] == "error" for diagnostic in report["diagnostics"]):
28
    raise RuntimeError(report["diagnostics"])
29
30
async def main():
31
    await plugin.initialize(config)
32
    try:
33
        # Run instrumented application work here.
34
        pass
35
    finally:
36
        plugin.clear()
37
38
asyncio.run(main())

Manual API

Use the manual subscriber API when you need an explicit subscriber name or direct force_flush control.

1
from nemo_relay import OpenInferenceConfig, OpenInferenceSubscriber
2
3
config = OpenInferenceConfig()
4
config.transport = "http_binary"
5
config.endpoint = "http://localhost:6006/v1/traces"
6
config.service_name = "agent-service"
7
config.set_resource_attribute("deployment.environment", "dev")
8
9
subscriber = OpenInferenceSubscriber(config)
10
subscriber.register("openinference-exporter")
11
12
# Run instrumented application work here.
13
14
subscriber.force_flush()
15
subscriber.deregister("openinference-exporter")
16
subscriber.shutdown()

Common Configuration and Runtime Issues

• transport is not http_binary or grpc.
• Headers or resource attributes are not string-to-string maps.
• The OpenInference feature is unavailable in the current build or target.
• Tool and LLM calls do not use managed helpers, so spans contain only scope lifecycle data.