Skip to navigation

Use the shared capture and verification path to make runs captured by NVIDIA NeMo Relay available to a team investigating traces in Datadog. The example routes Relay OTLP traces to an organization-operated OpenTelemetry Collector; that Collector forwards them to your Datadog site.

Use the same companion workload

Select --destination datadog in the parent guide’s example. Supply your running Collector’s trace URL. Its Datadog exporter and credentials stay in that collector or tracing backend. Find service relay-support-triage and apply this page’s destination checks.

The example uses OpenInference to keep the same agent/model/tool roles as the Arize Phoenix and LangSmith recipes. Datadog accepts both OpenInference and OpenTelemetry GenAI conventions; use gen_ai when your pipeline standardizes on those attributes. This recipe verifies APM trace arrival. Enabling Datadog’s Agent/LLM Observability product requires its additional OpenTelemetry configuration.

Choose the collector or tracing backend

Datadog recommends an Agent or OpenTelemetry Collector for production OTLP traffic. The collector or tracing backend can add host metadata, process spans, and apply central sampling. Choose one that the Relay worker or application can reach and that your platform team already operates. A direct Datadog OTLP trace intake is available when an Agent or Collector is not feasible; it uses a site-specific HTTPS endpoint and an API-key header.

The following plugins.toml example sends OpenInference traces to a managed Collector. Replace the illustrative host with its actual HTTPS trace URL:

version = 1
[[components]]
kind = "observability"
enabled = true
[components.config]
version = 4
[components.config.opentelemetry]
enabled = true
[[components.config.opentelemetry.endpoints]]
type = "openinference"
endpoint = "https://otel-gateway.example.com/v1/traces"
transport = "http_binary"
service_name = "relay-support-triage"

If the Collector requires authentication from Relay, add its approved header with header_env; keep the secret outside plugins.toml. The Collector owns the Datadog site and API key in this topology. Follow Datadog’s Collector setup or Agent OTLP ingestion for the chosen collector or tracing backend.

Check Datadog APM

Verification is manual in Datadog; the companion’s --verify option supports Phoenix only. Confirm intake in Datadog as well as successful forwarding by your collector.

Find a recent trace under the service relay-support-triage. Confirm that its agent span has the observed model and tool work beneath it, then inspect model identity and sanitized payload fields.

Collector processing can change operation names. Compare the resulting APM trace with Relay’s local events, accounting for Collector mapping when matching agent, model, and tool activity.

Missing evidenceFirst place to check
No traceRelay exporter delivery, Collector or Agent ingestion, and the Datadog site.
Partial traceRelay capture coverage and any sampling or span processing in the collector or tracing backend.
Unexpected operation namesThe selected trace projection and Datadog’s OTLP span mapping.

Direct intake does not automatically compute APM trace metrics; plan that separately if the enterprise dashboard needs them. Datadog’s direct intake guide explains that option and its required headers.