OpenTelemetry

Use the opentelemetry section to export traces to one or more OpenTelemetry Protocol (OTLP) destinations. OpenTelemetry support is always included; no Cargo feature enables or disables it.

Each endpoint selects one fixed semantic projection:

You can repeat a type or combine types. Each endpoint owns an independent exporter and can use a different endpoint.

The gen_ai projection targets the OpenTelemetry GenAI semantic-conventions v1.42-era snapshot. Use that pinned snapshot when comparing emitted operation and attribute names with collector or backend schemas.

NeMo Relay uses OpenTelemetry Rust 0.32. It deterministically derives compliant trace and span IDs from Relay lifecycle UUIDs, so endpoints that receive the same event stream use the same identifiers and parentage. Different endpoint types must therefore use independent OTLP destinations; configuring them with the same endpoint and transport is rejected to prevent identifier collisions at the receiver. Duplicate detection compares canonical destinations: HTTP and HTTPS default ports are realized, repeated and trailing path slashes are normalized, and standardized loopback hosts such as localhost, names under .localhost, 127.0.0.0/8, and ::1 are equivalent. Relay does not use DNS resolution for this comparison, and query strings remain significant. Rooted Relay propagation continues the Relay-derived trace across the import boundary. Rootless propagation retains Relay event parentage but starts a new OpenTelemetry trace from the first local event. Carry W3C traceparent and tracestate alongside Relay propagation when an integration also needs to preserve upstream OpenTelemetry sampling or vendor state.

plugins.toml Example

The following configuration exports the gen_ai projection through OTLP/HTTP.

1
version = 1
2
3
[[components]]
4
kind = "observability"
5
enabled = true
6
7
[components.config]
8
version = 3
9
10
[components.config.opentelemetry]
11
enabled = true
12
13
[[components.config.opentelemetry.endpoints]]
14
type = "gen_ai"
15
endpoint = "http://localhost:4318/v1/traces"
16
transport = "http_binary"
17
service_name = "agent-service"
18
19
[components.config.opentelemetry.endpoints.header_env]
20
authorization = "OTEL_AUTHORIZATION"
21
22
[[components.config.opentelemetry.endpoints]]
23
type = "openinference"
24
endpoint = "http://localhost:6006/v1/traces"
25
service_name = "agent-service"

When enabled = true, configure at least one endpoint. NeMo Relay constructs every endpoint before registering the fan-out subscriber. An invalid endpoint prevents activation, and a delivery failure from one exporter does not stop application work or delivery to the other exporters.

Endpoint Fields

Batch Processor Environment Variables

OpenTelemetry’s standard batch processor settings apply process-wide to every configured endpoint. Set them before the plugin activates; endpoint-specific batch sizing is not supported.

Use positive integer values. The SDK falls back to its defaults for malformed values but accepts zero; do not use zero values. Queue capacity counts spans, not bytes.

Increasing OTEL_BSP_MAX_QUEUE_SIZE can reduce the risk for a known burst size, but a finite queue does not guarantee lossless telemetry. Always clear the plugin during graceful shutdown so NeMo Relay can record the final drop count and the SDK can attempt to export queued spans.

Endpoint Capacity and Sizing

NeMo Relay does not impose a maximum number of OpenTelemetry endpoints. Each endpoint owns an exporter, tracer provider, batch processor, batch queue, and exporter runtime resources. Nothing in that export stack is shared between endpoints. Queue capacity and memory are per endpoint, and total export traffic is approximately the trace payload size multiplied by the endpoint count.

Typical deployments need one to three endpoints. Validate configurations with tens or hundreds of endpoints against the process limits for threads, memory, and network egress before deploying them.

Use header_env for secrets so configuration files contain only environment variable names. Each variable contains the complete header value. NeMo Relay validates variable names without reading their values, then resolves and snapshots the values when the plugin activates. Every referenced variable name must be nonblank and have no surrounding whitespace. Its value must be set and nonblank, with no surrounding whitespace, when the component activates. A header name cannot appear in both headers and header_env, including names that differ only by ASCII case. Reactivate the plugin to pick up a changed environment value.

Process-global OTEL_EXPORTER_OTLP_HEADERS and OTEL_EXPORTER_OTLP_TRACES_HEADERS are rejected because they cannot be isolated between endpoints. Put non-secret values in each endpoint’s headers map and secret variable references in header_env.

full and openinference endpoints retain the legacy mark and attribute-alias controls shown above. gen_ai is standards-only: it ignores those controls and does not emit Relay-private attributes. semantic_selector and capture_content are unsupported.

GenAI Projection

The gen_ai endpoint uses these operation names:

Marks are omitted. Relay scope types without GenAI semantics are emitted as minimal internal spans so that the original span parentage is preserved. This projection never emits nemo_relay.* fields. LLM spans include the gen_ai.system_instructions, gen_ai.input.messages, and gen_ai.output.messages attributes as JSON strings that follow the OpenTelemetry GenAI schemas. Each attribute is emitted only when its normalized instructions, messages, or response content is present. Redact sensitive content with an LLM or event sanitizer. Tool and retrieval payloads are not exported. Set top-level enable_full_payloads = true to retain complete sanitized LLM request history on every start span.

Error Type Mapping

For managed LLM, tool, and stream failures, NeMo Relay maps structured FlowError values to the OpenTelemetry error.type attribute:

External application and callback exceptions that do not have a more specific FlowError classification emit internal_error. Python and JavaScript callback boundaries also preserve the exception class separately, and both the full and gen_ai projections emit an exception span event with exception.type. NeMo Relay does not inspect error messages to recover exception class names. When an errored parent span has no useful classification of its own, it inherits the failed descendant’s error.type and exception type. When no structured FlowError is available, such as a cancellation or dropped execution, the projection emits _OTHER. Caller-provided error.type and exception.type metadata take precedence over values derived from FlowError.

FlowError is an exhaustive Rust enum. Rust callers upgrading to this release must handle the new CallbackException variant in exhaustive matches. It maps to the same internal status as Internal, while retaining exception_type for observability projection.

Direct Subscriber

1
from nemo_relay import OpenTelemetryConfig, OpenTelemetrySubscriber
2
3
config = OpenTelemetryConfig(
4
    "gen_ai",
5
    "http://localhost:4318/v1/traces",
6
)
7
config.service_name = "agent-service"
8
subscriber = OpenTelemetrySubscriber(config)

Direct construction creates one independently managed exporter. Register the subscriber before instrumented work. During graceful teardown, deregister it, call the binding’s force-flush method (force_flush() or forceFlush()), and then call shutdown().

Version 2 to Version 3

Version 3 replaces the separate version-2 sections:

• Move the old opentelemetry fields into one endpoint with type = "full".
• Move the old openinference fields into the same section with type = "openinference".
• Use type = "gen_ai" for standardized GenAI-only output.

Version-2 OTLP section shapes are rejected when version = 3; NeMo Relay does not silently normalize them. For complete before-and-after configuration and binding API changes, refer to Migration Guides.