OpenTelemetry

Use the opentelemetry section to configure independent OpenTelemetry Protocol (OTLP) trace, log, and metric pipelines. OpenTelemetry support is always included; no Cargo feature enables or disables it.

Timestamp Range

Trace and log timestamps must fit in OTLP’s unsigned 64-bit count of nanoseconds since the Unix epoch: 1970-01-01T00:00:00Z through 2554-07-21T23:34:33.709551615Z, inclusive. This applies to network exporters and local OTLP file sinks, including all full, gen_ai, and openinference trace projections. On Windows, trace and log timestamps have 100-nanosecond precision.

The trace and log subscribers drop events outside this range and report otel.timestamp_out_of_range in runtime diagnostics. They do not replace invalid timestamps with different times.

A dropped scope start creates no span. A dropped scope end leaves an existing span open for a later valid end. Dropped marks produce no span event, standalone span, or log record. Agent Trajectory Observability Format (ATOF) events keep the original timestamp, and scope operations still complete. Metric points use SDK collection timestamps instead of source event timestamps.

Automatic OpenTelemetry Configuration

When at least one nonblank OpenTelemetry OTLP endpoint environment variable is present, Relay automatically adds the matching gen_ai trace pipeline and log and metric pipelines. This automatic exporter is additive: explicitly configured Relay endpoints can export independently as long as process-global OTLP header variables are unset. Global OTLP headers cannot be isolated from configured endpoints, so Relay skips a configured endpoint that is initialized with those variables. Use that endpoint’s headers, header_env, or header_file configuration instead. Relay delegates endpoint, protocol, timeout, compression, temporality, header parsing, and signal-specific precedence to its vendored OpenTelemetry Rust SDK.

An explicit opentelemetry section with enabled: false is an opt-out: Relay does not create automatic OTLP exporters, even when endpoint environment variables are present. OTEL_SDK_DISABLED=true also disables every Relay OTLP exporter. An enabled opentelemetry section without configured endpoints is compatible with automatic export.

Set OTEL_EXPORTER_OTLP_ENDPOINT to enable all three signals, or set one or more signal-specific endpoint variables to enable only their matching signals:

• OTEL_EXPORTER_OTLP_TRACES_ENDPOINT
• OTEL_EXPORTER_OTLP_LOGS_ENDPOINT
• OTEL_EXPORTER_OTLP_METRICS_ENDPOINT

Resource, sampler, batch-processor, protocol, timeout, compression, header, and temporality settings refine an automatic exporter after an endpoint has enabled it; they never activate export by themselves. For example:

export OTEL_EXPORTER_OTLP_ENDPOINT=http://localhost:4318
export OTEL_SERVICE_NAME=relay-agent
export OTEL_EXPORTER_OTLP_HEADERS='authorization=Bearer%20token'

Environment-variable support

Automatic means Relay creates the standard gen_ai trace pipeline plus log and metric pipelines after an endpoint activates that signal and delegates parsing to the OpenTelemetry Rust SDK/OTLP exporter. Relay endpoints means a structured Relay endpoint keeps its existing explicit behavior; it does not inherit the setting.

Relay classifies each sanitized event before OTLP signal fan-out:

Scope events maintain lineage for log correlation, but Relay does not export them as log records or metric measurements. Metric marks do not fall back to logs or traces when metric export is disabled. A valid mark with the reserved metric schema routes exclusively to metrics. An invalid reserved metric mark reaches no OTLP signal and produces a rate-limited operational diagnostic. Other marks retain the existing projection-specific trace behavior and can be exported as logs.

When the resource_metrics component and polling are enabled, each poll emits marks with the reserved metric schema. An enabled metric pipeline exports available numerical measurements. It omits unavailable measurements instead of recording them as zero. Refer to Resource Metrics for snapshot and polling details.

Trace Projections

Each endpoint selects one fixed semantic projection:

You can repeat a type or combine types. Each endpoint owns an independent exporter and can use a different endpoint.

The gen_ai projection keeps the OpenTelemetry GenAI semantic-conventions v1.42-era snapshot as its core compatibility baseline. Relay also implements selected newer Development attributes. Refer to GenAI Projection Attribute Support for the exact current inventory and its newer registry baseline.

NeMo Relay uses the currently vendored OpenTelemetry Rust SDK (0.32). It deterministically derives compliant trace and span IDs from Relay lifecycle UUIDs, so endpoints that receive the same event stream use the same identifiers and parentage. Different endpoint types must therefore use independent OTLP destinations; configuring them with the same endpoint and transport is rejected to prevent identifier collisions at the receiver. Duplicate detection compares canonical destinations: HTTP and HTTPS default ports are realized, repeated and trailing path slashes are normalized, and standardized loopback hosts such as localhost, names under .localhost, 127.0.0.0/8, and ::1 are equivalent. Relay does not use DNS resolution for this comparison, and query strings remain significant. Log lineage follows the same root selection as traces. Relay captures parent trace context when it queues an event, so a subscriber registered after a local parent starts can still place later children in the parent’s trace. When neither the parent lifecycle nor captured context is available, the observed scope starts a trace using its own UUID; a propagation-root UUID alone does not establish an imported parent. Marks under that scope, including late marks within the completed-context retention window, use its trace identity. An explicit imported parent continues the propagated trace. Relay propagation continues the Relay-derived trace across the import boundary by default. Use capture_rootless_propagation_context() only when a receiver must start a new Relay-derived OpenTelemetry trace when no W3C parent is available. Valid W3C headers are always emitted and take precedence for OpenTelemetry parentage. PropagationContext JSON also carries optional W3C traceparent and tracestate fields, so an integration transports one payload when it must preserve upstream OpenTelemetry sampling or vendor state. Relay validates the pair through the W3C propagator; malformed values are warned about and ignored while Relay propagation continues.

plugins.toml Example

The following version-4 configuration exports a gen_ai trace and derives log and metric destinations from the trace destination.

version = 1
[[components]]
kind = "observability"
enabled = true
[components.config]
version = 4
[components.config.opentelemetry]
enabled = true
[[components.config.opentelemetry.endpoints]]
type = "gen_ai"
endpoint = "http://localhost:4318/v1/traces"
transport = "http_binary"
service_name = "agent-service"
promote_metadata_prefixes = ["app."]
promote_resource_metadata_prefixes = ["deployment."]
max_queue_size = 4096
max_export_batch_size = 512
scheduled_delay_millis = 1000
completed_span_context_ttl_millis = 60000
[components.config.opentelemetry.endpoints.header_env]
authorization = "OTEL_AUTHORIZATION"
[components.config.opentelemetry.endpoints.resource_attributes]
"nv.project" = "observability-dev"
[components.config.opentelemetry.logs]
enabled = true
completed_span_context_ttl_millis = 60000
minimum_severity = "info"
[components.config.opentelemetry.metrics]
enabled = true
temporality = "cumulative"

Relay replaces the terminal /v1/traces with /v1/logs and /v1/metrics. Derived destinations copy the trace endpoint’s transport, authentication, resource attributes, resource-metadata promotion prefixes, service identity, instrumentation scope, and timeout. In this example, all three signals use the same authorization and nv.project routing value.

When enabled = true, configure at least one trace endpoint or an enabled signal with explicit endpoints. NeMo Relay constructs every endpoint before registering subscribers. An invalid endpoint is skipped with an activation warning while valid endpoints register. Activation fails when no trace, log, or metric endpoint can be registered. A delivery failure from one exporter does not stop application work or delivery to the other exporters.

Trace Endpoint Fields

All trace projections (full, gen_ai, and openinference) require HTTPS for remote collectors, with either HTTP or gRPC transport. HTTP is permitted only for localhost and loopback IP addresses such as 127.0.0.1 and [::1]. Redaction does not disable this requirement. Existing remote plaintext trace endpoints must migrate to HTTPS or use a loopback collector.

Relay-configured OTLP/HTTP exporters do not follow redirects, even to another HTTPS endpoint. This keeps an export from bypassing transport checks or sending its payload and credentials to another destination. Use the final collector URL. This applies to trace, log, and metric exporters whether or not headers are configured. gRPC does not use HTTP redirects.

Trace File Sinks

A file sink writes spans to a local file instead of exporting them to a collector. File sinks and endpoints can be configured together, and each destination receives the same projected spans.

[[components.config.opentelemetry.file_sinks]]
type = "full"
output_directory = "/var/log/nemo-relay"
format = "json_lines"

Two formats are available. json_lines implements the OpenTelemetry Protocol File Exporter specification and is the default; use it for interoperability with other OTLP tooling. proto writes the same records as length-delimited protobuf, matching the OpenTelemetry Collector file exporter’s format: proto; use it when file size or parse cost matters.

Each export is flushed before it is reported as delivered, so a run that exits between batches still leaves a readable prefix. Output files are created with owner-only permissions and confined to output_directory, the same as the ATOF and ATIF file sinks.

Two file sinks cannot write the same path; the configuration is rejected at activation.

Event Metadata Promotion

Wrapped CLI events can include trusted agent_version metadata. To export this generic field as a span attribute, include "agent_version" in promote_metadata_prefixes. Refer to Coding-Agent Identity Metadata for its production and lifecycle semantics.

Set promote_metadata_prefixes on a trace endpoint to copy selected keys from the final sanitized Event metadata into that endpoint’s OpenTelemetry output. The setting defaults to an empty list, so Relay does not promote metadata unless you configure at least one prefix.

For example, "app.", "app_", and "app-" are valid literal prefixes. Leading or repeated dots, whitespace, other punctuation, and glob expressions such as "app.*" are rejected.

Matching is case-sensitive and compares the beginning of each key literally. Relay does not infer a dot or metadata-key segment boundary. For example, "app." selects app.name and app.version, but not app_name. The broader "app" prefix selects all three keys. Configure the narrowest prefix that selects the metadata you intend to export.

Scope-start and Scope-end are separate Event records. When Scope-end completes the span, a metadata key present on that Event replaces the corresponding promoted Scope-start value. Mark metadata is promoted to the attributes of the projected span event or tool span. The gen_ai projection continues to omit Marks.

Promotion supports strings, booleans, signed 64-bit integers, floating-point numbers, empty arrays, and homogeneous arrays of those primitive types. Relay omits rejected values and records one bounded runtime diagnostic per rejected key. The diagnostic code is otel.metadata_promotion_value_unsupported.<metadata-key>, its message contains the key and rejection reason, and its count is the number of occurrences for that key. Relay does not record the rejected value or stop trace export. Match the otel.metadata_promotion_value_unsupported. prefix to monitor rejected span metadata keys.

Projection-owned attributes take precedence over promoted metadata with the same key. For full and openinference, configured attribute-mapping aliases also take precedence. Relay also omits selected keys in namespaces owned by Relay or supported semantic projections: nemo_relay., gen_ai., error., exception., input., output., llm., openinference., server., service., session., tool., tool_call., and user.. Relay omits the bare metadata key as well. Rejected keys produce a rate-limited operational diagnostic without dropping the Event or span.

Promotion does not modify the Event or ATOF payload. In OTLP trace output only, Relay removes successfully promoted keys from serialized Relay metadata attributes. This applies to every Scope-start, Scope-end, and Mark event, and also to OpenInference’s metadata JSON attribute. Resource-promotion prefixes participate in that filtering for every event even though only a trace root’s Scope-start metadata can create resource attributes. Keys that cannot be promoted, including values overridden by configured resource attributes, remain in serialized metadata. Use resource_attributes instead for static values that must be attached to every span from an endpoint.

Root Resource Metadata Promotion

Set promote_resource_metadata_prefixes on a trace, log, or metric endpoint to derive resource attributes from the sanitized metadata on a root Scope-start Event. Every span, scoped log, and scoped metric measurement in that scope tree uses the same resource for its configured endpoint; child metadata and later root metadata changes do not modify it. Derived log and metric endpoints inherit the trace endpoint’s prefixes. The prefixes and supported value types match promote_metadata_prefixes.

For explicit log and metric endpoints, the Python plugin helper OpenTelemetrySignalEndpointConfig and the Node.js plugin helper openTelemetrySignalEndpoint accept promote_resource_metadata_prefixes. The Go plugin helper ObservabilityOpenTelemetrySignalEndpointConfig exposes PromoteResourceMetadataPrefixes. The Go trace endpoint helper ObservabilityOpenTelemetryEndpointConfig exposes the same field, which derived log and metric endpoints inherit. Omit the setting to keep promotion disabled. These plugin settings also apply when endpoints are configured through TOML or JSON. The Python, Node.js, and Go direct log and metric subscriber APIs do not expose this option.

Configured service_name, service_namespace, service_version, and explicit resource_attributes take precedence over promoted metadata with the same key. Each distinct effective resource creates a retained OTLP provider pipeline for each enabled signal, so use only controlled, low-cardinality values such as deployment, region, client version, or environment identity. Do not promote request, tenant, or user identifiers.

Each log or metric endpoint retains at most 16 dynamic resource pipelines, plus its configured base provider. Existing resource keys continue to reuse their pipelines at the limit; pipelines are not evicted. Additional resource keys use the base resource and produce otel.resource_metadata_pipeline_limit. This preserves delivery but omits the promoted resource distinction, so metric measurements that differ only by that resource can be aggregated together. The limit is fixed, applies independently to each log and metric endpoint, and does not cap trace pipelines or total process memory.

Each log or metric endpoint tracks at most 4,096 active resource scopes, including nested scopes. Once full, it keeps existing routes and declines new scope starts with otel.resource_metadata_active_scope_limit; marks attached to those untracked scopes use the base resource. Ending a tracked scope frees a slot for a subsequent scope start. Active routes are not evicted by age, so long-running scopes retain their identity. Missing end events can occupy slots until the subscriber is replaced, but cannot grow this active-route map beyond the limit. This limit does not bound completed-route caches, trace correlation state, or total process memory.

Completed resource routes expire before processing an event beyond the TTL; events exactly at the boundary remain linked. Logs use opentelemetry.logs.completed_span_context_ttl_millis (default: 60 seconds). Metrics use a fixed 60-second resource-route TTL. Expiration is driven by event timestamps and runs when another event arrives, rather than on a background timer.

Relay records bounded runtime diagnostics for this setting. A rejected value produces otel.resource_metadata_promotion_value_unsupported.<metadata-key>. A failed resource-pipeline construction produces otel.resource_metadata_pipeline_build_failed; Relay then exports the affected signal through the endpoint’s configured resource instead. Match both the otel.metadata_promotion_value_unsupported. and otel.resource_metadata_promotion_value_unsupported. prefixes to monitor all rejected metadata keys.

Log and Metric Endpoint Resolution

An enabled logs or metrics section can omit endpoints. Relay then derives one signal endpoint from every trace endpoint:

• A bare HTTP authority, with or without a root trailing /, gains /v1/logs or /v1/metrics.
• A terminal /v1/traces, including one below a path prefix, is replaced with the signal path. Query parameters are preserved.
• A gRPC endpoint reuses its authority without path rewriting.
• A trace endpoint with any other custom path cannot be derived. Configure signal endpoints explicitly in that case.

An explicit nonempty signal endpoint list replaces derivation. Relay preserves an explicit custom signal path exactly, but rejects an obvious standard path for another signal, such as /v1/traces in a log endpoint. An explicit empty list is invalid when the signal is enabled.

The following example sends logs to a custom intake path while metrics continue to derive from the trace endpoint:

[components.config.opentelemetry.logs]
enabled = true
[[components.config.opentelemetry.logs.endpoints]]
endpoint = "https://collector.example/custom/log-intake"
transport = "http_binary"
service_name = "agent-service"
[components.config.opentelemetry.metrics]
enabled = true

The signal endpoint fields are endpoint, transport, headers, header_env, header_file, resource_attributes, promote_resource_metadata_prefixes, service_name, service_namespace, service_version, instrumentation_scope, and timeout_millis. Their defaults match the corresponding trace fields. Each signal rejects duplicate destinations within that signal. Logs, metrics, and traces can share the same authority because OTLP treats them as different signals. All three signals automatically include the reserved telemetry.sdk.name, telemetry.sdk.language, and telemetry.sdk.version resource attributes; configuring those keys in resource_attributes rejects the endpoint.

Trace Batch Processor Configuration

Configure batch processing independently on each endpoint with max_queue_size, max_export_batch_size, and scheduled_delay_millis. When an endpoint omits a field, the corresponding standard OpenTelemetry environment variable applies process-wide. If neither is set, the SDK default applies.

The precedence for each setting is endpoint value, then environment variable, then SDK default. Set environment variables before the plugin activates.

Endpoint values must be positive integers. If both endpoint size fields are set, max_export_batch_size must not exceed max_queue_size. When one size is inherited, the SDK caps the effective batch size at the effective queue size. The SDK falls back to its default for malformed environment values. Queue and batch sizes count spans, not bytes.

Completed Scope Lineage Retention

Trace endpoints retain a completed scope’s trace and parent span context for completed_span_context_ttl_millis after its scope-end event. A late mark in that window remains attached to the original trace and parent span. When the TTL expires, Relay emits subsequent marks as orphan spans and records the otel.completed_span_context_expired runtime diagnostic when it purges expired contexts.

Keep closed scope handles only for short deferred follow-up work. Prefer emitting an event before the scope closes, or create a new active scope for later work. Increasing the TTL retains more completed contexts: memory grows with the completed-scope rate, TTL, and number of configured trace endpoints.

OTLP logs use the same TTL-based completed-scope lineage behavior. Configure opentelemetry.logs.completed_span_context_ttl_millis independently when log export is enabled. Metrics retain completed resource routes for 60 seconds when resource promotion is enabled, but do not retain trace-correlation context. Do not rely on a closed scope handle for long-running follow-up work; emit the mark before closing the scope or use an active/new scope instead.

If an endpoint’s explicit batch settings are invalid, Relay skips that endpoint and records an observability.invalid_otel_endpoint configuration warning with its opentelemetry.endpoints[N] field. Other valid endpoints continue to activate. Activation still fails when no trace, log, or metric endpoint can be registered.

Relay also skips and logs any trace endpoint that fails during exporter construction. This includes malformed collector destinations that cannot be detected during configuration validation.

Relay also skips a file sink that fails during exporter construction and adds an observability.invalid_otel_file_sink warning to the activation report’s configuration diagnostics. The diagnostic identifies the affected sink and includes the filesystem error. If another destination activates, Relay continues with that destination. If no trace, log, or metric destination can be registered, activation fails and includes the file sink failure instead of reporting only a missing collector endpoint.

Relay’s thread-based batch processor exports serially, so it does not expose the SDK’s concurrent-export setting. It also does not expose a separate batch processor export timeout; use the endpoint’s timeout_millis to bound each OTLP request.

Increasing an endpoint’s max_queue_size, or the process-wide OTEL_BSP_MAX_QUEUE_SIZE fallback, can reduce the risk for a known burst size, but a finite queue does not guarantee lossless telemetry. Always clear the plugin during graceful shutdown so NeMo Relay can record the final drop count and the SDK can attempt to export queued spans.

Endpoint Capacity and Sizing

NeMo Relay does not impose a maximum number of OpenTelemetry endpoints. Each trace, log, or metric endpoint owns an exporter, signal provider, processor or reader, and exporter runtime resources. Nothing in those export stacks is shared between endpoints. Queue capacity and memory are per endpoint, and total export traffic grows with the endpoint count.

Typical deployments need one to three endpoints. Validate configurations with tens or hundreds of endpoints against the process limits for threads, memory, and network egress before deploying them.

Use header_env for secrets so configuration files contain only environment variable names. Each variable contains the complete header value. NeMo Relay validates variable names without reading their values, then resolves and snapshots the values when the plugin activates. Every referenced variable name must be nonblank and have no surrounding whitespace. Its value must be set and nonblank, with no surrounding whitespace, when the component activates. A header name cannot appear in both headers and header_env, including names that differ only by ASCII case. Reactivate the plugin to pick up a changed environment value.

Use header_file when another process updates a credential in place, such as a projected token file. Each file must be regular and exist when the plugin activates. Relay reads the file only when it exports. Relay removes trailing whitespace. A missing, unreadable, blank, or invalid value fails only that export, and diagnostics do not show the value. A header can use only one of headers, header_env, and header_file. Header names are case-insensitive. This applies to trace, log, and metric endpoints.

Remote trace endpoints must use HTTPS. When you set any header source (headers, header_env, or header_file), remote log and metric endpoints must also use HTTPS. HTTP is allowed only for localhost and loopback IP addresses. OTLP/HTTP exporters never follow redirects. For example, an OIDC token-file writer must write the complete value Bearer eyJ... to the file. Relay does not add the authentication scheme for you.

For Relay-configured endpoints, process-global OTEL_EXPORTER_OTLP_HEADERS, OTEL_EXPORTER_OTLP_TRACES_HEADERS, OTEL_EXPORTER_OTLP_LOGS_HEADERS, and OTEL_EXPORTER_OTLP_METRICS_HEADERS are rejected because they cannot be isolated between endpoints. Put non-secret values in each endpoint’s headers map and secret variable references in header_env.

Those variables are accepted by automatic OpenTelemetry configuration, where the OTLP Rust exporter parses the header values itself. Relay validates percent escapes before building each automatic exporter. If the selected header list contains an invalid escape, such as %ZZ or a trailing %, Relay skips that signal’s automatic exporter without logging header values. Signal-specific header lists take precedence over the generic list, including empty lists.

A configured endpoint skipped because of process-global headers does not prevent activation when at least one automatic exporter is active. The automatic exporters retain their environment headers; skipped configured endpoints receive no requests.

full and openinference endpoints retain the legacy mark and attribute-alias controls shown above. gen_ai is standards-only: it ignores those controls and does not emit Relay-private attributes. semantic_selector and capture_content are unsupported.

On a successful tool end span with a present, non-null annotation, the full and openinference projections emit the opaque value as one JSON string attribute named nemo_relay.tool.result.annotation. Relay does not flatten the annotation’s application-defined keys. The gen_ai projection omits this Relay-private attribute.

Emit Log and Metric Marks

Rust, Python, and Node.js generic mark APIs accept optional data_schema and severity values. Prefer the typed metric helper rather than constructing the reserved schema by hand: metric in Rust, Python, and Node.js. The helper validates the complete measurement group before publishing the mark. The exporter validates the sanitized payload again before recording any measurement.

The following examples emit one warning log mark and one metric mark:

from nemo_relay import LogSeverity, MetricKind, MetricMeasurement, MetricValueType
from nemo_relay import scope
scope.event(
    "budget-nearly-exhausted",
    data={"remaining_tokens": 128},
    severity=LogSeverity.Warn,
)
scope.metric(
    "tokenomics",
    [
        MetricMeasurement(
            "example.tokens.saved",
            MetricKind.Counter,
            MetricValueType.U64,
            42,
            unit="{token}",
            description="Tokens avoided",
            attributes={"model": "example-model"},
        )
    ],
)

Mark sanitizers run for both calls. Routing uses the immutable data_schema after sanitization, and a metric mark never falls back to the log pipeline.

Log Export

The log pipeline exports one OTLP LogRecord for each sanitized non-metric mark. Marks with data_schema = null and marks with an application-defined schema are logs. Any mark that uses the reserved nemo.relay.metric_measurements schema name is routed away from logs, including unsupported schema versions and invalid metric payloads. Scope start and end events update the lineage used for correlation but do not become log records.

Use the typed severity argument on the generic mark API. Relay stores it in sanitizer-visible metadata as nemo_relay.log.severity. The typed argument overrides that metadata key and requires metadata to be an object. After mark sanitizers run, Relay parses the remaining key, defaults an absent key to info, and drops a log record with an invalid value. Supported values are trace, debug, info, warn, and error; warning is accepted as an alias for warn.

The logs section applies these processing settings to every log endpoint:

Relay maps the event timestamp to the log timestamp and post-sanitization processing time to the observed timestamp. Sanitized data becomes the structured body. An absent or top-level JSON null payload has no body; a nested JSON null becomes the string "null" because OTLP AnyValue has no null variant.

The log attributes preserve the mark name, UUID, optional parent UUID, category and category profile, schema, sanitized metadata, and ATOF version under nemo_relay.* keys. A mark in a resolvable active or completed scope receives trace and span context. An orphan mark receives no invented trace context. Relay leaves OTLP event_name unset with the currently vendored OpenTelemetry SDK (0.32) and retains the dynamic name in nemo_relay.mark.name.

Metric Export

The metric pipeline consumes only sanitized marks with this exact schema:

{
  "data_schema": {
    "name": "nemo.relay.metric_measurements",
    "version": "1"
  },
  "data": {
    "measurements": [
      {
        "name": "example.tokens.saved",
        "kind": "counter",
        "value_type": "u64",
        "value": 42,
        "unit": "{token}",
        "description": "Tokens avoided",
        "attributes": {"model": "example-model"},
        "boundaries": null
      }
    ]
  }
}

A mark with the reserved schema name and an unsupported version or invalid payload is dropped from both logs and metrics. Relay emits a rate-limited operational diagnostic without creating another ATOF or OTLP event.

The measurements array is required and nonempty, and unknown fields are rejected. Each measurement is an SDK recording operation, not a pre-aggregated OTLP point:

Unsigned values must not exceed i64::MAX, which prevents loss in the pinned OTLP conversion. Metric names must be 1 to 255 ASCII bytes, start with a letter, and contain only letters, digits, _, ., -, or /. Units must be ASCII and at most 63 bytes. Optional histogram boundaries can include negative values, but every boundary must be finite, strictly increasing, unique, and the list can contain at most 64 entries.

Attributes can contain strings, Booleans, signed integers, finite doubles, and homogeneous nonempty arrays of those primitive types. Blank keys, nulls, nested objects, mixed arrays, and unsigned integers above i64::MAX are invalid. Do not use event UUIDs, timestamps, metadata, trace IDs, or other high-cardinality values as metric attributes.

Relay treats the complete mark atomically. It records no measurements from the mark when a measurement is invalid, an instrument descriptor conflicts, or an instrument limit would be exceeded. Instrument names compare case-insensitively, and a name must retain its kind, numeric type, unit, description, and histogram boundaries for the lifetime of that destination.

The metrics section applies these settings to every metric endpoint:

Metric points use SDK collection timestamps. The currently vendored OpenTelemetry SDK (0.32) cannot preserve the source mark timestamp or attach a trace-linked exemplar through this path. Relay does not emulate correlation with high-cardinality attributes.

GenAI Projection

Set an OpenTelemetry endpoint’s type to gen_ai to select this projection:

[[components.config.opentelemetry.endpoints]]
type = "gen_ai"
endpoint = "http://localhost:4318/v1/traces"

The gen_ai endpoint uses these operation names:

Marks are omitted. Relay scope types without GenAI semantics are emitted as minimal internal spans so that the original span parentage is preserved. This projection never emits nemo_relay.* fields. LLM spans include the gen_ai.system_instructions, gen_ai.input.messages, and gen_ai.output.messages attributes as JSON strings that follow the OpenTelemetry GenAI schemas. Each attribute is emitted only when its normalized instructions, messages, or response content is present. Redact sensitive content with an LLM or event sanitizer. Tool content is exported by default; retrieval payloads are not exported. Set top-level enable_full_payloads = true to retain complete sanitized LLM request history on every start span.

GenAI Projection Attribute Support

This inventory follows the 72-attribute OpenTelemetry GenAI registry snapshot used for the current Relay audit. Every listed upstream attribute is marked Development. This newer inventory includes attributes added or renamed after Relay’s v1.42-era compatibility baseline.

Supported means Relay’s native GenAI projection emits the exact attribute on an applicable signal when an authoritative normalized or instrumentation source exists. It does not mean every integration supplies that optional source. Partial means Relay emits the attribute but cannot represent the complete upstream cardinality or value-shape contract. Not supported means the native projection does not emit it. The gen_ai.* namespace is reserved, so generic metadata promotion cannot be used to add an unsupported GenAI attribute.

Usage is recorded per model request. Relay’s normalized response usage retains an optional uncached_input_tokens count only when the provider’s response contract proves it. The standard GenAI export deliberately does not add a nonstandard uncached field: consumers can use the total gen_ai.usage.input_tokens with the separately reported cache-read and cache-write counts. An omitted cache or uncached count means unavailable, not zero.

For managed streaming LLM calls, the end span also includes gen_ai.response.time_to_first_chunk in seconds. Relay measures from the start of managed stream execution until the first provider protocol chunk is received; this is not a first-text-token measurement. The attribute is omitted for non-streaming calls and streams that never yield a chunk.

The metric endpoint records the same sample as the standard gen_ai.client.operation.time_to_first_chunk f64 histogram with unit s when Relay can determine the provider dimension.

The gen_ai projection includes sanitized gen_ai.tool.call.arguments at tool start, gen_ai.tool.call.result on successful completion, and gen_ai.tool.definitions on inference spans by default. These attributes can contain sensitive information. Use the PII redaction trajectory_context preset to remove opaque payloads while retaining analytical structure and trace parentage. Credential removal and event sanitizers run before projection. enable_full_payloads controls LLM request-history retention independently.

Arguments and results are emitted as canonical JSON strings. Relay parses serialized JSON before projection, then preserves every non-null sanitized JSON value, including objects, arrays, strings, booleans, and numbers. Failed tool calls do not emit a result attribute. Result annotations remain separate and are never included in this projection. Tool definitions include only required type and name properties. Known OpenAI Responses built-ins use their native type as identity; recognized Gemini native tool groups emit an identity for each tool in the group. Explicit native names are preserved. Unknown unnamed native definitions are omitted rather than assigned invented identities.

Tool Identity

Tool identity comes from instrumentation metadata and the typed tool-call ID, never from similarly named fields in tool arguments or results. Nonblank string metadata supports gen_ai.tool.name, gen_ai.tool.type / tool_type, gen_ai.tool.call.id / tool_call_id, gen_ai.tool.description / tool_description / description, and gen_ai.agent.name / agent_name. The typed call ID takes precedence. Absent tool names use the scope name; unknown descriptions, types, and agent names are omitted.

Claude Code, Codex, and Pi tool hooks default gen_ai.tool.type to function because these harnesses execute tools locally. They default gen_ai.agent.name to claude-code, codex, or pi, or subagent:<id> when ownership is known. These defaults apply to paired hooks and post-only hooks that synthesize a start, without replacing explicitly supplied nonblank string metadata, including the tool_type and agent_name aliases. Generic gateway sessions do not infer a tool type or root executing-agent name. Invocation arguments are never used to infer tool descriptions.

Harness defaults carry internal provenance. Explicit completion metadata may refine inferred tool type and agent name, including through their aliases. Explicit start metadata and typed tool-call IDs remain authoritative; later inferred values cannot replace them. This provenance is not exported as a GenAI attribute.

Claude Code and Codex MCP tool hooks with a qualified mcp__<server>__<tool> name receive mcp.method.name = "tools/call" in event metadata, including post-only hooks. Relay preserves existing method metadata and does not infer MCP identity for ordinary tools, Pi, or generic gateway events. The original qualified tool name remains unchanged. To include the method in OTLP span attributes, configure promote_metadata_prefixes to include "mcp.method.name".

This identifies a requested MCP tool operation, not proof that a request reached the server or succeeded. Permission-denied calls retain their denial and error metadata. The server segment is a configured alias; Relay does not derive server.address, mcp.session.id, or connection status from it. Hook duration measures the harness tool scope, not necessarily MCP transport latency.

Error Type Mapping

For managed LLM, tool, and stream failures, NeMo Relay maps structured FlowError values to the OpenTelemetry error.type attribute:

External application and callback exceptions that do not have a more specific FlowError classification emit internal_error. Python and JavaScript callback boundaries also preserve the exception class separately, and both the full and gen_ai projections emit an exception span event with exception.type. NeMo Relay does not inspect error messages to recover exception class names. When an errored parent span has no useful classification of its own, it inherits the failed descendant’s error.type and exception type. When no structured FlowError is available, such as a cancellation or dropped execution, the projection emits _OTHER. Caller-provided error.type and exception.type metadata take precedence over values derived from FlowError. An authenticated Claude Code or Codex permission rejection closes the matched active tool span with error.type = "guardrail_rejected". The corresponding permission guardrail events carry the canonical gen_ai.tool.call.id in event metadata so subscribers can correlate the decision with that tool call.

FlowError is an exhaustive Rust enum. Rust callers upgrading to this release must handle the new CallbackException variant in exhaustive matches. It maps to the same internal status as Internal, while retaining exception_type for observability projection.

Direct Subscribers

from nemo_relay import OpenTelemetryConfig, OpenTelemetrySubscriber
config = OpenTelemetryConfig(
    "gen_ai",
    "http://localhost:4318/v1/traces",
)
config.service_name = "agent-service"
config.header_env = {"authorization": "OTEL_AUTHORIZATION"}
subscriber = OpenTelemetrySubscriber(config)

Set each referenced environment variable before constructing the subscriber. Direct trace, log, and metric configs resolve header_env when the subscriber is constructed and retain that value for the subscriber’s activation. Changing the process environment affects only a subsequently constructed subscriber. Static headers remain unchanged. A header name cannot appear in both maps, including names that differ only by ASCII case, and names within header_env must also be unique ignoring ASCII case.

Each header_env reference must be nonblank, have no surrounding whitespace, and contain neither = nor NUL. Its environment value must be set, nonblank, contain no leading or trailing whitespace, be valid Unicode, and be a valid HTTP header value. Validation errors name the header and environment variable but do not include the resolved value. Relay supplies resolved values only as outbound OTLP request headers; it does not copy them into Event data, OpenTelemetry payloads, resource attributes, or runtime diagnostics.

The log and metric equivalents are OpenTelemetryLogConfig with OpenTelemetryLogSubscriber, and OpenTelemetryMetricConfig with OpenTelemetryMetricSubscriber. Each config takes one required endpoint and exposes the signal settings documented above. Bare OTLP/HTTP authorities gain the corresponding standard signal path. Rust, Python, and Node.js expose the same three independently managed subscriber kinds. The C FFI remains experimental and source-first.

Direct construction creates one independently managed exporter. Register the subscriber before instrumented work. During graceful teardown, deregister it, call the binding’s force-flush method (force_flush() or forceFlush()), and then call shutdown(). Force flush first crosses Relay’s subscriber barrier and then flushes the provider. Log shutdown drains the batch queue. Metric shutdown performs the reader’s final collection; it does not add a second metric flush. For direct trace and log subscribers, a successful force flush updates runtime_diagnostics() with any batch queue drops observed so far; the diagnostic count remains cumulative through later flushes and shutdown.

Log and Metric Subscriber Lifecycle

The following examples create and register direct log and metric subscribers, inspect runtime diagnostics, and perform graceful teardown.

from nemo_relay import (
    OpenTelemetryLogConfig,
    OpenTelemetryLogSubscriber,
    OpenTelemetryMetricConfig,
    OpenTelemetryMetricSubscriber,
)
logs = OpenTelemetryLogSubscriber(
    OpenTelemetryLogConfig("http://localhost:4318/v1/logs")
)
metrics = OpenTelemetryMetricSubscriber(
    OpenTelemetryMetricConfig("http://localhost:4318/v1/metrics")
)
logs.register("otlp-logs")
metrics.register("otlp-metrics")
try:
    # Run instrumented work here.
    for diagnostic in logs.runtime_diagnostics().entries:
        print(diagnostic.code, diagnostic.message)
finally:
    logs.deregister("otlp-logs")
    logs.force_flush()
    logs.shutdown()
    metrics.deregister("otlp-metrics")
    metrics.force_flush()
    metrics.shutdown()

Every direct trace, log, and metric subscriber exposes a bounded runtime diagnostics snapshot: runtime_diagnostics() in Rust and Python, runtimeDiagnostics() in Node.js. It reports each runtime condition’s stable code, occurrence count, and most recent message. C FFI callers use nemo_relay_otel_subscriber_runtime_diagnostics_json, nemo_relay_otel_log_subscriber_runtime_diagnostics_json, or nemo_relay_otel_metric_subscriber_runtime_diagnostics_json. Each writes a caller-owned bounded JSON array of diagnostic entries that the caller must release with nemo_relay_string_free. Use diagnostics to monitor rejected metric marks, capacity limits, and delivery failures without configuring the observability plugin. The plugin continues to include the same conditions in its runtime report.