OpenTelemetry

Use the opentelemetry section when you want NeMo Relay lifecycle events exported as generic OpenTelemetry Protocol (OTLP) trace spans.

OpenTelemetry export is a good fit when your tracing backend already expects OTLP spans and you want NeMo Relay scopes, tool calls, LLM calls, and marks to appear in the same tracing pipeline as the rest of the application.

plugins.toml Example

Add the following OpenTelemetry exporter configuration to plugins.toml:

1
version = 1
2
3
[[components]]
4
kind = "observability"
5
enabled = true
6
7
[components.config]
8
version = 2
9
10
[components.config.opentelemetry]
11
enabled = true
12
transport = "http_binary"
13
endpoint = "http://localhost:4318/v1/traces"
14
service_name = "agent-service"
15
service_namespace = "nemo"
16
service_version = "1.0.0"
17
instrumentation_scope = "nemo-relay-otel"
18
timeout_millis = 3000
19
20
[components.config.opentelemetry.headers]
21
authorization = "Bearer <token>"
22
23
[components.config.opentelemetry.resource_attributes]
24
"deployment.environment" = "dev"

This configuration registers a plugin-owned OpenTelemetry subscriber and sends NeMo Relay trace spans to the configured OTLP endpoint.

Fields

The following table describes OpenTelemetry exporter settings:

Expected Output

The collector should receive OTLP trace export requests. The tracing backend should show spans for NeMo Relay scopes, tools, LLM calls, and marks grouped by root scope.

The default inherit projection follows exporter-native handling: a mark with an active parent span is a span event, while an orphan mark is a standalone zero-duration mark:<name> span. mark_projection = "event" explicitly selects that representation. Set mark_projection = "tool" only when the target backend needs eligible marks represented as visible child spans. Projected marks use SpanKind::Internal, carry nemo_relay.mark.projection = "tool", and retain mark UUID, parentage, category/profile, and payload attributes. High-volume llm.chunk marks retain exporter-native handling by default and when excluded from tool projection.

Add other event names to mark_exclude_names for a backend to keep those marks in exporter-native form rather than visible tool children. The exclusion list affects only tool projection; it does not remove mark payload or metadata. Set mark_exclude_names = [] to disable the default llm.chunk exclusion.

Each lifecycle span includes nemo_relay.uuid and nemo_relay.parent_uuid attributes. These values match ATIF step.extra.ancestry.function_id and step.extra.ancestry.parent_id for the same events. For plugin-managed ATIF, the trajectory-root span’s nemo_relay.uuid also matches the ATIF session_id. Backend-native trace_id and span_id values are not written into ATIF.

NeMo Relay projects top-level lifecycle payload fields to typed OTLP attributes with dotted names. For example, metadata = { tenant = "acme" } becomes nemo_relay.start.metadata.tenant = "acme".

• Start events use the nemo_relay.start.data, nemo_relay.start.metadata, and nemo_relay.start.input prefixes.
• End events use nemo_relay.end.data, nemo_relay.end.metadata, and nemo_relay.end.output.
• Handle attributes use nemo_relay.handle_attributes. Mark data, metadata, attributes, and category-profile fields use the corresponding nemo_relay.mark.* prefixes.

Scalar strings, booleans, and numbers that fit an OTLP numeric type keep their types. NeMo Relay emits larger unsigned integers as strings. When a top-level field contains an object or array, NeMo Relay emits its value as a JSON string at that field’s dotted name. Nested null values remain in that string, but a top-level field whose value is null is omitted. NeMo Relay no longer emits the old aggregate *_json payload attributes.

Configure an alias when a backend expects a different attribute name:

1
[components.config.opentelemetry]
2
attribute_mappings = [
3
  { key = "nemo_relay.start.metadata.tenant", alias = "tenant.id" },
4
]

The source attribute remains in the span. If the span already contains tenant.id, NeMo Relay keeps the existing value instead of replacing it with the alias.

Coding-agent trace roots also carry these correlation attributes:

These canonical fields are emitted on trace roots rather than repeated on every child span. Filter roots by session.id, user.id, or nemo_relay.session.instance_id, then use the backend trace_id to select the remaining rows in each trace. A session.start mark carries the same fields: it is a span event when a parent is active and a zero-duration root span when it is orphaned.

For LLM end spans, cost is emitted as nemo_relay.llm.cost.total and nemo_relay.llm.cost.currency (any currency). Token counts are not emitted as discrete attributes. Refer to Token and Cost Field Semantics for the full mapping.

Register the plugin before the first instrumented request, use stable service identity fields, keep credentials outside source code, and flush during graceful shutdown.

Plugin Configuration

Use plugin configuration when the application should let NeMo Relay own the OpenTelemetry subscriber lifecycle. The following examples configure and activate the OpenTelemetry exporter through each supported language binding.

validate() checks only the supplied in-memory object. initialize() also layers discovered plugins.toml configuration. For effective file-backed validation, refer to Plugin Configuration Files and run the gateway with the same configuration path that production uses.

1
import asyncio
2
3
from nemo_relay import plugin
4
from nemo_relay.observability import ComponentSpec, ObservabilityConfig, OtlpConfig
5
6
config = plugin.PluginConfig(
7
    components=[
8
        ComponentSpec(
9
            ObservabilityConfig(
10
                opentelemetry=OtlpConfig(
11
                    enabled=True,
12
                    transport="http_binary",
13
                    endpoint="http://localhost:4318/v1/traces",
14
                    service_name="agent-service",
15
                    service_namespace="nemo",
16
                    service_version="1.0.0",
17
                    instrumentation_scope="nemo-relay-otel",
18
                    resource_attributes={"deployment.environment": "dev"},
19
                    headers={"authorization": "Bearer <token>"},
20
                )
21
            )
22
        )
23
    ]
24
)
25
26
report = plugin.validate(config)
27
if any(diagnostic["level"] == "error" for diagnostic in report["diagnostics"]):
28
    raise RuntimeError(report["diagnostics"])
29
30
async def main():
31
    await plugin.initialize(config)
32
    try:
33
        # Run instrumented application work here.
34
        pass
35
    finally:
36
        plugin.clear()
37
38
asyncio.run(main())

Manual API

Use the manual subscriber API when you need an explicit subscriber name or direct force_flush control.

1
from nemo_relay import OpenTelemetryConfig, OpenTelemetrySubscriber
2
3
config = OpenTelemetryConfig()
4
config.transport = "http_binary"
5
config.endpoint = "http://localhost:4318/v1/traces"
6
config.service_name = "agent-service"
7
config.set_resource_attribute("deployment.environment", "dev")
8
9
subscriber = OpenTelemetrySubscriber(config)
10
subscriber.register("otel-exporter")
11
12
# Run instrumented application work here.
13
14
subscriber.force_flush()
15
subscriber.deregister("otel-exporter")
16
subscriber.shutdown()

Common Configuration and Runtime Issues

• transport is not http_binary or grpc.
• Headers or resource attributes are not string-to-string maps.
• The exporter feature is unavailable in the current build or target.
• The endpoint is unreachable at runtime.