> For clean Markdown of any page, append .md to the page URL. > For a complete documentation index, see https://docs.nvidia.com/nemo/relay/llms.txt. > For AI client integration (Claude Code, Cursor, etc.), connect to the MCP server at https://docs.nvidia.com/nemo/relay/_mcp/server. # OpenTelemetry > Export NeMo Relay traces through typed OpenTelemetry endpoints. Use the `opentelemetry` section to export traces to one or more OpenTelemetry Protocol (OTLP) destinations. OpenTelemetry support is always included; no Cargo feature enables or disables it. Each endpoint selects one fixed semantic projection: | Type | Projection | | --------------- | --------------------------------------------------------------------------------------------- | | `full` | Complete NeMo Relay projection, including `nemo_relay.*` attributes and native mark handling. | | `gen_ai` | OpenTelemetry GenAI semantic conventions only. | | `openinference` | OpenInference-compatible spans with the existing default mark handling. | You can repeat a type or combine types. Each endpoint owns an independent exporter and can use a different endpoint. The `gen_ai` projection targets the [OpenTelemetry GenAI semantic-conventions v1.42-era snapshot](https://github.com/open-telemetry/semantic-conventions-genai/tree/43633a68ef8f8ed87a1d5eb205990311ca708bf1). Use that pinned snapshot when comparing emitted operation and attribute names with collector or backend schemas. NeMo Relay uses OpenTelemetry Rust `0.32`. It deterministically derives compliant trace and span IDs from Relay lifecycle UUIDs, so endpoints that receive the same event stream use the same identifiers and parentage. Different endpoint types must therefore use independent OTLP destinations; configuring them with the same endpoint and transport is rejected to prevent identifier collisions at the receiver. Duplicate detection compares canonical destinations: HTTP and HTTPS default ports are realized, repeated and trailing path slashes are normalized, and standardized loopback hosts such as `localhost`, names under `.localhost`, `127.0.0.0/8`, and `::1` are equivalent. Relay does not use DNS resolution for this comparison, and query strings remain significant. Rooted Relay propagation continues the Relay-derived trace across the import boundary. Rootless propagation retains Relay event parentage but starts a new OpenTelemetry trace from the first local event. Carry W3C `traceparent` and `tracestate` alongside Relay propagation when an integration also needs to preserve upstream OpenTelemetry sampling or vendor state. ## `plugins.toml` Example The following configuration exports the `gen_ai` projection through OTLP/HTTP. ```toml version = 1 [[components]] kind = "observability" enabled = true [components.config] version = 3 [components.config.opentelemetry] enabled = true [[components.config.opentelemetry.endpoints]] type = "gen_ai" endpoint = "http://localhost:4318/v1/traces" transport = "http_binary" service_name = "agent-service" [components.config.opentelemetry.endpoints.header_env] authorization = "OTEL_AUTHORIZATION" [[components.config.opentelemetry.endpoints]] type = "openinference" endpoint = "http://localhost:6006/v1/traces" service_name = "agent-service" ``` When `enabled = true`, configure at least one endpoint. NeMo Relay constructs every endpoint before registering the fan-out subscriber. An invalid endpoint prevents activation, and a delivery failure from one exporter does not stop application work or delivery to the other exporters. ## Endpoint Fields | Field | Default | Notes | | ----------------------- | ----------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | | `type` | Required | `full`, `gen_ai`, or `openinference`. | | `endpoint` | Required | Nonblank OTLP endpoint. For OTLP/HTTP, Relay appends `/v1/traces` when the endpoint contains only a scheme, host, and optional port with no explicit path. Add a trailing `/` to export to the root path. Any other explicit path is preserved. gRPC endpoints are always preserved. | | `transport` | `http_binary` | `http_binary` or `grpc`. | | `service_name` | `unknown_service` | `service.name` resource attribute. | | `service_namespace` | Omitted | Optional `service.namespace`. | | `service_version` | Omitted | Optional `service.version`. | | `instrumentation_scope` | `opentelemetry` | Instrumentation scope name. | | `timeout_millis` | `3000` | Export timeout. | | `headers` | `{}` | String-to-string exporter headers. | | `header_env` | `{}` | Header names mapped to environment variable names containing secret values. | | `resource_attributes` | `{}` | String-to-string resource attributes. | | `mark_projection` | `inherit` | Mark representation for `full` and `openinference`: `inherit`, `event`, or `tool`. | | `mark_exclude_names` | `["llm.chunk"]` | Mark names excluded from `full` and `openinference` projection. | | `attribute_mappings` | `[]` | `{ key, alias }` copies applied by `full` and `openinference` projection. | ## Batch Processor Environment Variables OpenTelemetry's standard batch processor settings apply process-wide to every configured endpoint. Set them before the plugin activates; endpoint-specific batch sizing is not supported. | Variable | Default | Notes | | -------------------------------- | --------: | -------------------------------------------------------------- | | `OTEL_BSP_MAX_QUEUE_SIZE` | `2048` | Maximum completed spans buffered per endpoint. | | `OTEL_BSP_MAX_EXPORT_BATCH_SIZE` | `512` | Maximum spans exported in one batch; capped at the queue size. | | `OTEL_BSP_SCHEDULE_DELAY` | `5000` ms | Maximum delay before exporting a non-full batch. | Use positive integer values. The SDK falls back to its defaults for malformed values but accepts zero; do not use zero values. Queue capacity counts spans, not bytes. A full queue drops completed spans instead of applying backpressure to application work. Bursts can therefore drop spans that finish late, including an enclosing root span, and leave an incomplete trace in the backend. The OpenTelemetry SDK logs `BatchSpanProcessor.SpanDroppingStarted` at warning level when each endpoint first drops a span. It suppresses additional first-drop warnings for that endpoint to avoid a log storm. During graceful shutdown, it logs `BatchSpanProcessor.SpansDropped` with the endpoint processor's exact `dropped_span_count` and `max_queue_size`. For plugin-managed exporters, NeMo Relay also records `otel.spans_dropped` in the active plugin report's `runtime_diagnostics`. Its `count` is the exact number of dropped spans, `field` identifies the affected `opentelemetry.endpoints[N].endpoint`, and `message` includes the configured endpoint URL. If spans were dropped, clearing the plugin returns a delivery failure error and retains the diagnostic for inspection. This error does not disable later plugin configuration. Increasing `OTEL_BSP_MAX_QUEUE_SIZE` can reduce the risk for a known burst size, but a finite queue does not guarantee lossless telemetry. Always clear the plugin during graceful shutdown so NeMo Relay can record the final drop count and the SDK can attempt to export queued spans. ## Endpoint Capacity and Sizing NeMo Relay does not impose a maximum number of OpenTelemetry endpoints. Each endpoint owns an exporter, tracer provider, batch processor, batch queue, and exporter runtime resources. Nothing in that export stack is shared between endpoints. Queue capacity and memory are per endpoint, and total export traffic is approximately the trace payload size multiplied by the endpoint count. Typical deployments need one to three endpoints. Validate configurations with tens or hundreds of endpoints against the process limits for threads, memory, and network egress before deploying them. Use `header_env` for secrets so configuration files contain only environment variable names. Each variable contains the complete header value. NeMo Relay validates variable names without reading their values, then resolves and snapshots the values when the plugin activates. Every referenced variable name must be nonblank and have no surrounding whitespace. Its value must be set and nonblank, with no surrounding whitespace, when the component activates. A header name cannot appear in both `headers` and `header_env`, including names that differ only by ASCII case. Reactivate the plugin to pick up a changed environment value. Process-global `OTEL_EXPORTER_OTLP_HEADERS` and `OTEL_EXPORTER_OTLP_TRACES_HEADERS` are rejected because they cannot be isolated between endpoints. Put non-secret values in each endpoint's `headers` map and secret variable references in `header_env`. `full` and `openinference` endpoints retain the legacy mark and attribute-alias controls shown above. `gen_ai` is standards-only: it ignores those controls and does not emit Relay-private attributes. `semantic_selector` and `capture_content` are unsupported. ## GenAI Projection The `gen_ai` endpoint uses these operation names: | Relay scope | OpenTelemetry operation | | ----------- | ------------------------------------------------ | | Agent | `invoke_agent` | | LLM | `chat`, `generate_content`, or `text_completion` | | Tool | `execute_tool` | | Embedder | `embeddings` | | Retriever | `retrieval` | Marks are omitted. Relay scope types without GenAI semantics are emitted as minimal internal spans so that the original span parentage is preserved. This projection never emits `nemo_relay.*` fields. LLM spans include the `gen_ai.system_instructions`, `gen_ai.input.messages`, and `gen_ai.output.messages` attributes as JSON strings that follow the OpenTelemetry GenAI schemas. Each attribute is emitted only when its normalized instructions, messages, or response content is present. Redact sensitive content with an LLM or event sanitizer. Tool and retrieval payloads are not exported. Set top-level `enable_full_payloads = true` to retain complete sanitized LLM request history on every start span. ### Error Type Mapping For managed LLM, tool, and stream failures, NeMo Relay maps structured `FlowError` values to the OpenTelemetry `error.type` attribute: | Relay error | `error.type` | | ------------------------------- | -------------------- | | `AlreadyExists` | `already_exists` | | `NotFound` | `not_found` | | `InvalidArgument` | `invalid_argument` | | `ScopeStackEmpty` | `scope_stack_empty` | | `GuardrailRejected` | `guardrail_rejected` | | Upstream connection failure | `connection_error` | | Upstream timeout | `timeout` | | Upstream retryable status | `retryable_status` | | Upstream context-window failure | `context_window` | | Upstream model unavailable | `model_unavailable` | | Upstream authentication failure | `authentication` | | Upstream invalid request | `invalid_request` | | Other upstream failure | `upstream_error` | | `Internal` | `internal_error` | | Binding callback exception | `internal_error` | External application and callback exceptions that do not have a more specific `FlowError` classification emit `internal_error`. Python and JavaScript callback boundaries also preserve the exception class separately, and both the `full` and `gen_ai` projections emit an `exception` span event with `exception.type`. NeMo Relay does not inspect error messages to recover exception class names. When an errored parent span has no useful classification of its own, it inherits the failed descendant's `error.type` and exception type. When no structured `FlowError` is available, such as a cancellation or dropped execution, the projection emits `_OTHER`. Caller-provided `error.type` and `exception.type` metadata take precedence over values derived from `FlowError`. `FlowError` is an exhaustive Rust enum. Rust callers upgrading to this release must handle the new `CallbackException` variant in exhaustive matches. It maps to the same internal status as `Internal`, while retaining `exception_type` for observability projection. ## Direct Subscriber #### Python ```python from nemo_relay import OpenTelemetryConfig, OpenTelemetrySubscriber config = OpenTelemetryConfig( "gen_ai", "http://localhost:4318/v1/traces", ) config.service_name = "agent-service" subscriber = OpenTelemetrySubscriber(config) ``` #### Node.js ```javascript const { OpenTelemetrySubscriber } = require("nemo-relay-node"); const subscriber = new OpenTelemetrySubscriber({ type: "gen_ai", endpoint: "http://localhost:4318/v1/traces", serviceName: "agent-service", }); ``` #### Rust ```rust use nemo_relay::observability::OpenTelemetryType; use nemo_relay::observability::otel::{OpenTelemetryConfig, OpenTelemetrySubscriber}; let config = OpenTelemetryConfig::new( OpenTelemetryType::GenAi, "http://localhost:4318/v1/traces", ) .with_service_name("agent-service"); let subscriber = OpenTelemetrySubscriber::new(config)?; ``` Direct construction creates one independently managed exporter. Register the subscriber before instrumented work. During graceful teardown, deregister it, call the binding's force-flush method (`force_flush()` or `forceFlush()`), and then call `shutdown()`. ## Version 2 to Version 3 Version 3 replaces the separate version-2 sections: * Move the old `opentelemetry` fields into one endpoint with `type = "full"`. * Move the old `openinference` fields into the same section with `type = "openinference"`. * Use `type = "gen_ai"` for standardized GenAI-only output. Version-2 OTLP section shapes are rejected when `version = 3`; NeMo Relay does not silently normalize them. For complete before-and-after configuration and binding API changes, refer to [Migration Guides](/reference/migration-guides#observability-configuration-version-3).