OpenTelemetry

Use the opentelemetry section to configure independent OpenTelemetry Protocol (OTLP) trace, log, and metric pipelines. OpenTelemetry support is always included; no Cargo feature enables or disables it.

Relay classifies each sanitized event before OTLP signal fan-out:

Scope events maintain lineage for log correlation, but Relay does not export them as log records or metric measurements. Metric marks do not fall back to logs or traces when metric export is disabled. A valid mark with the reserved metric schema routes exclusively to metrics. An invalid reserved metric mark reaches no OTLP signal and produces a rate-limited operational diagnostic. Other marks retain the existing projection-specific trace behavior and can be exported as logs.

Trace Projections

Each endpoint selects one fixed semantic projection:

You can repeat a type or combine types. Each endpoint owns an independent exporter and can use a different endpoint.

The gen_ai projection targets the OpenTelemetry GenAI semantic-conventions v1.42-era snapshot. Use that pinned snapshot when comparing emitted operation and attribute names with collector or backend schemas.

NeMo Relay uses the currently vendored OpenTelemetry Rust SDK (0.32). It deterministically derives compliant trace and span IDs from Relay lifecycle UUIDs, so endpoints that receive the same event stream use the same identifiers and parentage. Different endpoint types must therefore use independent OTLP destinations; configuring them with the same endpoint and transport is rejected to prevent identifier collisions at the receiver. Duplicate detection compares canonical destinations: HTTP and HTTPS default ports are realized, repeated and trailing path slashes are normalized, and standardized loopback hosts such as localhost, names under .localhost, 127.0.0.0/8, and ::1 are equivalent. Relay does not use DNS resolution for this comparison, and query strings remain significant. Relay propagation continues the Relay-derived trace across the import boundary by default. Use capture_rootless_propagation_context() only when a receiver must start a new OpenTelemetry trace. Carry W3C traceparent and tracestate alongside Relay propagation when an integration also needs to preserve upstream OpenTelemetry sampling or vendor state.

plugins.toml Example

The following version-4 configuration exports a gen_ai trace and derives log and metric destinations from the trace destination.

1
version = 1
2
3
[[components]]
4
kind = "observability"
5
enabled = true
6
7
[components.config]
8
version = 4
9
10
[components.config.opentelemetry]
11
enabled = true
12
13
[[components.config.opentelemetry.endpoints]]
14
type = "gen_ai"
15
endpoint = "http://localhost:4318/v1/traces"
16
transport = "http_binary"
17
service_name = "agent-service"
18
promote_metadata_prefixes = ["app."]
19
promote_resource_metadata_prefixes = ["deployment."]
20
max_queue_size = 4096
21
max_export_batch_size = 512
22
scheduled_delay_millis = 1000
23
completed_span_context_ttl_millis = 60000
24
25
[components.config.opentelemetry.endpoints.header_env]
26
authorization = "OTEL_AUTHORIZATION"
27
28
[components.config.opentelemetry.endpoints.resource_attributes]
29
"nv.project" = "observability-dev"
30
31
[components.config.opentelemetry.logs]
32
enabled = true
33
completed_span_context_ttl_millis = 60000
34
minimum_severity = "info"
35
36
[components.config.opentelemetry.metrics]
37
enabled = true
38
temporality = "cumulative"

Relay replaces the terminal /v1/traces with /v1/logs and /v1/metrics. Derived destinations copy the trace endpoint’s transport, authentication, resource attributes, service identity, instrumentation scope, and timeout. In this example, all three signals use the same authorization and nv.project routing value.

When enabled = true, configure at least one trace endpoint or an enabled signal with explicit endpoints. NeMo Relay constructs every endpoint before registering subscribers. An invalid endpoint is skipped with an activation warning while valid endpoints register. Activation fails when no trace, log, or metric endpoint can be registered. A delivery failure from one exporter does not stop application work or delivery to the other exporters.

Trace Endpoint Fields

Event Metadata Promotion

Set promote_metadata_prefixes on a trace endpoint to copy selected keys from the final sanitized Event metadata into that endpoint’s OpenTelemetry output. The setting defaults to an empty list, so Relay does not promote metadata unless you configure at least one prefix.

For example, "app.", "app_", and "app-" are valid literal prefixes. Leading or repeated dots, whitespace, other punctuation, and glob expressions such as "app.*" are rejected.

Matching is case-sensitive and compares the beginning of each key literally. Relay does not infer a dot or metadata-key segment boundary. For example, "app." selects app.name and app.version, but not app_name. The broader "app" prefix selects all three keys. Configure the narrowest prefix that selects the metadata you intend to export.

Scope-start and Scope-end are separate Event records. When Scope-end completes the span, a metadata key present on that Event replaces the corresponding promoted Scope-start value. Mark metadata is promoted to the attributes of the projected span event or tool span. The gen_ai projection continues to omit Marks.

Promotion supports strings, booleans, signed 64-bit integers, floating-point numbers, empty arrays, and homogeneous arrays of those primitive types. Relay omits rejected values and records one bounded runtime diagnostic per rejected key. The diagnostic code is otel.metadata_promotion_value_unsupported.<metadata-key>, its message contains the key and rejection reason, and its count is the number of occurrences for that key. Relay does not record the rejected value or stop trace export. Match the otel.metadata_promotion_value_unsupported. prefix to monitor rejected span metadata keys.

Projection-owned attributes take precedence over promoted metadata with the same key. For full and openinference, configured attribute-mapping aliases also take precedence. Relay also omits selected keys in namespaces owned by Relay or supported semantic projections: nemo_relay., gen_ai., error., exception., input., output., llm., openinference., server., service., session., tool., tool_call., and user.. Relay omits the bare metadata key as well. Rejected keys produce a rate-limited operational diagnostic without dropping the Event or span.

Promotion does not modify the Event or ATOF payload. In OTLP trace output only, Relay removes successfully promoted keys from serialized Relay metadata attributes. This applies to every Scope-start, Scope-end, and Mark event, and also to OpenInference’s metadata JSON attribute. Resource-promotion prefixes participate in that filtering for every event even though only a trace root’s Scope-start metadata can create resource attributes. Keys that cannot be promoted, including values overridden by configured resource attributes, remain in serialized metadata. Use resource_attributes instead for static values that must be attached to every span from an endpoint.

Root Resource Metadata Promotion

Set promote_resource_metadata_prefixes to derive resource attributes from the sanitized metadata on a trace root’s Scope-start Event. Every span and scoped mark in that trace uses the same resource; child metadata and later root metadata changes do not modify it. The prefixes and supported value types match promote_metadata_prefixes.

Configured service_name, service_namespace, service_version, and explicit resource_attributes take precedence over promoted metadata with the same key. Each distinct effective resource creates a retained OTLP trace pipeline, so use only controlled, low-cardinality values such as deployment, region, or cluster identity. Do not promote request, tenant, or user identifiers.

Relay records two bounded runtime diagnostics for this setting. A rejected value produces otel.resource_metadata_promotion_value_unsupported.<metadata-key>. A failed resource-pipeline construction produces otel.resource_metadata_pipeline_build_failed; Relay then exports the trace through the endpoint’s configured resource instead. Match both the otel.metadata_promotion_value_unsupported. and otel.resource_metadata_promotion_value_unsupported. prefixes to monitor all rejected metadata keys.

Log and Metric Endpoint Resolution

An enabled logs or metrics section can omit endpoints. Relay then derives one signal endpoint from every trace endpoint:

• A bare HTTP authority, with or without a root trailing /, gains /v1/logs or /v1/metrics.
• A terminal /v1/traces, including one below a path prefix, is replaced with the signal path. Query parameters are preserved.
• A gRPC endpoint reuses its authority without path rewriting.
• A trace endpoint with any other custom path cannot be derived. Configure signal endpoints explicitly in that case.

An explicit nonempty signal endpoint list replaces derivation. Relay preserves an explicit custom signal path exactly, but rejects an obvious standard path for another signal, such as /v1/traces in a log endpoint. An explicit empty list is invalid when the signal is enabled.

The following example sends logs to a custom intake path while metrics continue to derive from the trace endpoint:

1
[components.config.opentelemetry.logs]
2
enabled = true
3
4
[[components.config.opentelemetry.logs.endpoints]]
5
endpoint = "https://collector.example/custom/log-intake"
6
transport = "http_binary"
7
service_name = "agent-service"
8
9
[components.config.opentelemetry.metrics]
10
enabled = true

The signal endpoint fields are endpoint, transport, headers, header_env, resource_attributes, service_name, service_namespace, service_version, instrumentation_scope, and timeout_millis. Their defaults match the corresponding trace fields. Each signal rejects duplicate destinations within that signal. Logs, metrics, and traces can share the same authority because OTLP treats them as different signals.

Trace Batch Processor Configuration

Configure batch processing independently on each endpoint with max_queue_size, max_export_batch_size, and scheduled_delay_millis. When an endpoint omits a field, the corresponding standard OpenTelemetry environment variable applies process-wide. If neither is set, the SDK default applies.

The precedence for each setting is endpoint value, then environment variable, then SDK default. Set environment variables before the plugin activates.

Endpoint values must be positive integers. If both endpoint size fields are set, max_export_batch_size must not exceed max_queue_size. When one size is inherited, the SDK caps the effective batch size at the effective queue size. The SDK falls back to its default for malformed environment values. Queue and batch sizes count spans, not bytes.

Completed Scope Lineage Retention

Trace endpoints retain a completed scope’s trace and parent span context for completed_span_context_ttl_millis after its scope-end event. A late mark in that window remains attached to the original trace and parent span. When the TTL expires, Relay emits subsequent marks as orphan spans and records the otel.completed_span_context_expired runtime diagnostic when it purges expired contexts.

Keep closed scope handles only for short deferred follow-up work. Prefer emitting an event before the scope closes, or create a new active scope for later work. Increasing the TTL retains more completed contexts: memory grows with the completed-scope rate, TTL, and number of configured trace endpoints.

OTLP logs use the same TTL-based completed-scope lineage behavior. Configure opentelemetry.logs.completed_span_context_ttl_millis independently when log export is enabled. Metrics do not retain completed-scope parent context. Do not rely on a closed scope handle for long-running follow-up work; emit the mark before closing the scope or use an active/new scope instead.

If an endpoint’s explicit batch settings are invalid, Relay skips that endpoint and records an observability.invalid_otel_endpoint configuration warning with its opentelemetry.endpoints[N] field. Other valid endpoints continue to activate. Activation still fails when no trace, log, or metric endpoint can be registered.

Relay also skips and logs any trace endpoint that fails during exporter construction. This includes malformed collector destinations that cannot be detected during configuration validation.

Relay’s thread-based batch processor exports serially, so it does not expose the SDK’s concurrent-export setting. It also does not expose a separate batch processor export timeout; use the endpoint’s timeout_millis to bound each OTLP request.

Increasing an endpoint’s max_queue_size, or the process-wide OTEL_BSP_MAX_QUEUE_SIZE fallback, can reduce the risk for a known burst size, but a finite queue does not guarantee lossless telemetry. Always clear the plugin during graceful shutdown so NeMo Relay can record the final drop count and the SDK can attempt to export queued spans.

Endpoint Capacity and Sizing

NeMo Relay does not impose a maximum number of OpenTelemetry endpoints. Each trace, log, or metric endpoint owns an exporter, signal provider, processor or reader, and exporter runtime resources. Nothing in those export stacks is shared between endpoints. Queue capacity and memory are per endpoint, and total export traffic grows with the endpoint count.

Typical deployments need one to three endpoints. Validate configurations with tens or hundreds of endpoints against the process limits for threads, memory, and network egress before deploying them.

Use header_env for secrets so configuration files contain only environment variable names. Each variable contains the complete header value. NeMo Relay validates variable names without reading their values, then resolves and snapshots the values when the plugin activates. Every referenced variable name must be nonblank and have no surrounding whitespace. Its value must be set and nonblank, with no surrounding whitespace, when the component activates. A header name cannot appear in both headers and header_env, including names that differ only by ASCII case. Reactivate the plugin to pick up a changed environment value.

Process-global OTEL_EXPORTER_OTLP_HEADERS, OTEL_EXPORTER_OTLP_TRACES_HEADERS, OTEL_EXPORTER_OTLP_LOGS_HEADERS, and OTEL_EXPORTER_OTLP_METRICS_HEADERS are rejected because they cannot be isolated between endpoints. Put non-secret values in each endpoint’s headers map and secret variable references in header_env.

full and openinference endpoints retain the legacy mark and attribute-alias controls shown above. gen_ai is standards-only: it ignores those controls and does not emit Relay-private attributes. semantic_selector and capture_content are unsupported.

On a successful tool end span with a present, non-null annotation, the full and openinference projections emit the opaque value as one JSON string attribute named nemo_relay.tool.result.annotation. Relay does not flatten the annotation’s application-defined keys. The gen_ai projection omits this Relay-private attribute.

Emit Log and Metric Marks

Rust, Python, and Node.js generic mark APIs accept optional data_schema and severity values. Prefer the typed metric helper rather than constructing the reserved schema by hand: metric in Rust, Python, and Node.js. The helper validates the complete measurement group before publishing the mark. The exporter validates the sanitized payload again before recording any measurement.

The following examples emit one warning log mark and one metric mark:

1
from nemo_relay import LogSeverity, MetricKind, MetricMeasurement, MetricValueType
2
from nemo_relay import scope
3
4
scope.event(
5
    "budget-nearly-exhausted",
6
    data={"remaining_tokens": 128},
7
    severity=LogSeverity.Warn,
8
)
9
10
scope.metric(
11
    "tokenomics",
12
    [
13
        MetricMeasurement(
14
            "example.tokens.saved",
15
            MetricKind.Counter,
16
            MetricValueType.U64,
17
            42,
18
            unit="{token}",
19
            description="Tokens avoided",
20
            attributes={"model": "example-model"},
21
        )
22
    ],
23
)

Mark sanitizers run for both calls. Routing uses the immutable data_schema after sanitization, and a metric mark never falls back to the log pipeline.

Log Export

The log pipeline exports one OTLP LogRecord for each sanitized non-metric mark. Marks with data_schema = null and marks with an application-defined schema are logs. Any mark that uses the reserved nemo.relay.metric_measurements schema name is routed away from logs, including unsupported schema versions and invalid metric payloads. Scope start and end events update the lineage used for correlation but do not become log records.

Use the typed severity argument on the generic mark API. Relay stores it in sanitizer-visible metadata as nemo_relay.log.severity. The typed argument overrides that metadata key and requires metadata to be an object. After mark sanitizers run, Relay parses the remaining key, defaults an absent key to info, and drops a log record with an invalid value. Supported values are trace, debug, info, warn, and error; warning is accepted as an alias for warn.

The logs section applies these processing settings to every log endpoint:

Relay maps the event timestamp to the log timestamp and post-sanitization processing time to the observed timestamp. Sanitized data becomes the structured body. An absent or top-level JSON null payload has no body; a nested JSON null becomes the string "null" because OTLP AnyValue has no null variant.

The log attributes preserve the mark name, UUID, optional parent UUID, category and category profile, schema, sanitized metadata, and ATOF version under nemo_relay.* keys. A mark in a resolvable active or completed scope receives trace and span context. An orphan mark receives no invented trace context. Relay leaves OTLP event_name unset with the currently vendored OpenTelemetry SDK (0.32) and retains the dynamic name in nemo_relay.mark.name.

Metric Export

The metric pipeline consumes only sanitized marks with this exact schema:

1
{
2
  "data_schema": {
3
    "name": "nemo.relay.metric_measurements",
4
    "version": "1"
5
  },
6
  "data": {
7
    "measurements": [
8
      {
9
        "name": "example.tokens.saved",
10
        "kind": "counter",
11
        "value_type": "u64",
12
        "value": 42,
13
        "unit": "{token}",
14
        "description": "Tokens avoided",
15
        "attributes": {"model": "example-model"},
16
        "boundaries": null
17
      }
18
    ]
19
  }
20
}

A mark with the reserved schema name and an unsupported version or invalid payload is dropped from both logs and metrics. Relay emits a rate-limited operational diagnostic without creating another ATOF or OTLP event.

The measurements array is required and nonempty, and unknown fields are rejected. Each measurement is an SDK recording operation, not a pre-aggregated OTLP point:

Unsigned values must not exceed i64::MAX, which prevents loss in the pinned OTLP conversion. Metric names must be 1 to 255 ASCII bytes, start with a letter, and contain only letters, digits, _, ., -, or /. Units must be ASCII and at most 63 bytes. Optional histogram boundaries can include negative values, but every boundary must be finite, strictly increasing, unique, and the list can contain at most 64 entries.

Attributes can contain strings, Booleans, signed integers, finite doubles, and homogeneous nonempty arrays of those primitive types. Blank keys, nulls, nested objects, mixed arrays, and unsigned integers above i64::MAX are invalid. Do not use event UUIDs, timestamps, metadata, trace IDs, or other high-cardinality values as metric attributes.

Relay treats the complete mark atomically. It records no measurements from the mark when a measurement is invalid, an instrument descriptor conflicts, or an instrument limit would be exceeded. Instrument names compare case-insensitively, and a name must retain its kind, numeric type, unit, description, and histogram boundaries for the lifetime of that destination.

The metrics section applies these settings to every metric endpoint:

Metric points use SDK collection timestamps. The currently vendored OpenTelemetry SDK (0.32) cannot preserve the source mark timestamp or attach a trace-linked exemplar through this path. Relay does not emulate correlation with high-cardinality attributes.

GenAI Projection

The gen_ai endpoint uses these operation names:

Marks are omitted. Relay scope types without GenAI semantics are emitted as minimal internal spans so that the original span parentage is preserved. This projection never emits nemo_relay.* fields. LLM spans include the gen_ai.system_instructions, gen_ai.input.messages, and gen_ai.output.messages attributes as JSON strings that follow the OpenTelemetry GenAI schemas. Each attribute is emitted only when its normalized instructions, messages, or response content is present. Redact sensitive content with an LLM or event sanitizer. Tool and retrieval payloads are not exported. Set top-level enable_full_payloads = true to retain complete sanitized LLM request history on every start span.

Error Type Mapping

For managed LLM, tool, and stream failures, NeMo Relay maps structured FlowError values to the OpenTelemetry error.type attribute:

External application and callback exceptions that do not have a more specific FlowError classification emit internal_error. Python and JavaScript callback boundaries also preserve the exception class separately, and both the full and gen_ai projections emit an exception span event with exception.type. NeMo Relay does not inspect error messages to recover exception class names. When an errored parent span has no useful classification of its own, it inherits the failed descendant’s error.type and exception type. When no structured FlowError is available, such as a cancellation or dropped execution, the projection emits _OTHER. Caller-provided error.type and exception.type metadata take precedence over values derived from FlowError.

FlowError is an exhaustive Rust enum. Rust callers upgrading to this release must handle the new CallbackException variant in exhaustive matches. It maps to the same internal status as Internal, while retaining exception_type for observability projection.

Direct Subscribers

1
from nemo_relay import OpenTelemetryConfig, OpenTelemetrySubscriber
2
3
config = OpenTelemetryConfig(
4
    "gen_ai",
5
    "http://localhost:4318/v1/traces",
6
)
7
config.service_name = "agent-service"
8
config.header_env = {"authorization": "OTEL_AUTHORIZATION"}
9
subscriber = OpenTelemetrySubscriber(config)

Set each referenced environment variable before constructing the subscriber. Direct trace, log, and metric configs resolve header_env when the subscriber is constructed and retain that value for the subscriber’s activation. Changing the process environment affects only a subsequently constructed subscriber. Static headers remain unchanged. A header name cannot appear in both maps, including names that differ only by ASCII case, and names within header_env must also be unique ignoring ASCII case.

Each header_env reference must be nonblank, have no surrounding whitespace, and contain neither = nor NUL. Its environment value must be set, nonblank, contain no leading or trailing whitespace, be valid Unicode, and be a valid HTTP header value. Validation errors name the header and environment variable but do not include the resolved value. Relay supplies resolved values only as outbound OTLP request headers; it does not copy them into Event data, OpenTelemetry payloads, resource attributes, or runtime diagnostics.

The log and metric equivalents are OpenTelemetryLogConfig with OpenTelemetryLogSubscriber, and OpenTelemetryMetricConfig with OpenTelemetryMetricSubscriber. Each config takes one required endpoint and exposes the signal settings documented above. Bare OTLP/HTTP authorities gain the corresponding standard signal path. Rust, Python, and Node.js expose the same three independently managed subscriber kinds. The C FFI remains experimental and source-first.

Direct construction creates one independently managed exporter. Register the subscriber before instrumented work. During graceful teardown, deregister it, call the binding’s force-flush method (force_flush() or forceFlush()), and then call shutdown(). Force flush first crosses Relay’s subscriber barrier and then flushes the provider. Log shutdown drains the batch queue. Metric shutdown performs the reader’s final collection; it does not add a second metric flush. For direct trace and log subscribers, a successful force flush updates runtime_diagnostics() with any batch queue drops observed so far; the diagnostic count remains cumulative through later flushes and shutdown.

Log and Metric Subscriber Lifecycle

The following examples create and register direct log and metric subscribers, inspect runtime diagnostics, and perform graceful teardown.

1
from nemo_relay import (
2
    OpenTelemetryLogConfig,
3
    OpenTelemetryLogSubscriber,
4
    OpenTelemetryMetricConfig,
5
    OpenTelemetryMetricSubscriber,
6
)
7
8
logs = OpenTelemetryLogSubscriber(
9
    OpenTelemetryLogConfig("http://localhost:4318/v1/logs")
10
)
11
metrics = OpenTelemetryMetricSubscriber(
12
    OpenTelemetryMetricConfig("http://localhost:4318/v1/metrics")
13
)
14
logs.register("otlp-logs")
15
metrics.register("otlp-metrics")
16
try:
17
    # Run instrumented work here.
18
    for diagnostic in logs.runtime_diagnostics().entries:
19
        print(diagnostic.code, diagnostic.message)
20
finally:
21
    logs.deregister("otlp-logs")
22
    logs.force_flush()
23
    logs.shutdown()
24
    metrics.deregister("otlp-metrics")
25
    metrics.force_flush()
26
    metrics.shutdown()

Every direct trace, log, and metric subscriber exposes a bounded runtime diagnostics snapshot: runtime_diagnostics() in Rust and Python, runtimeDiagnostics() in Node.js. It reports each runtime condition’s stable code, occurrence count, and most recent message. C FFI callers use nemo_relay_otel_subscriber_runtime_diagnostics_json, nemo_relay_otel_log_subscriber_runtime_diagnostics_json, or nemo_relay_otel_metric_subscriber_runtime_diagnostics_json. Each writes a caller-owned bounded JSON array of diagnostic entries that the caller must release with nemo_relay_string_free. Use diagnostics to monitor rejected metric marks, capacity limits, and delivery failures without configuring the observability plugin. The plugin continues to include the same conditions in its runtime report.

Migrating from Version 3 to Version 4

Version 4 adds the sibling opentelemetry.logs and opentelemetry.metrics sections. For the complete upgrade path and programmatic configuration, see Migrating from Version 3 to Version 4.

Version 2 to Version 3

Version 3 replaces the separate version-2 sections:

• Move the old opentelemetry fields into one endpoint with type = "full".
• Move the old openinference fields into the same section with type = "openinference".
• Use type = "gen_ai" for standardized GenAI-only output.

Version-2 OTLP section shapes are rejected when version = 3; NeMo Relay does not silently normalize them. For complete before-and-after configuration and binding API changes, refer to Observability Configuration.