Collect Agent Traces
Observability on NeMo Helix is provided by NeMo Intake, the trace ingestion and query service for agent telemetry, and optionally by NeMo Studio, the web user interface (UI) for browsing that telemetry. Observability helps humans and agents understand what happened in an agent system: which models were called, which tools were used, what inputs and outputs were produced, where failures occurred, and what feedback or evaluation scores were attached.
Before You Start
Who This Is For
NeMo Intake is for anyone who owns, operates, evaluates, or optimizes an agent. It is written for engineers wiring telemetry into an agent runtime, and for human and agent reviewers who need to diagnose failures from traces, in a production system or in offline evaluation flows.
Requirements
- A reachable ClickHouse database.
- A NeMo Helix environment with the
intakeservice running. - A telemetry source, such as an OpenTelemetry or OpenInference exporter, NeMo Agent Toolkit (NAT), NeMo Flow, NeMo Helix Evals runs, captured OpenAI-compatible chat-completion payloads, or Agent Trajectory Interchange Format (ATIF) trajectories.
- Optional: NeMo Studio for the UI review flow.
You do not need to deploy your agents on NeMo Helix. To start, run the platform, post one representative interaction, and confirm that it appears.
NeMo Helix wires trace export for nemo-agents-spec-v1 agents that it deploys or runs as jobs: their
ATIF trajectories go to Intake under the agent’s registered name. An explicit telemetry.agent_name
in agent.yaml takes precedence, so keep it equal to the registered name or name-based features such
as Insights will not find the traces. This applies to the DeepAgents, Claude Code,
Codex, Hermes, and NOOA harnesses, and only when the agent.yaml leaves the destination open:
telemetry.enabled: falseortelemetry.atif.enabled: falseopts out.- A
telemetry.atif.storagedestination is used as written instead.
Legacy NAT workflows, remote agents, and agents that run elsewhere send their own traces. Tag every
span with the agent’s name (gen_ai.agent.name for OTLP) so the traces can be
registered and
analyzed.
For the Fabric 0.5 upgrade, distinguish automatic persisted Relay observability v3-to-v4 compatibility from execution-time ATIF/OTLP gateway rewriting. Platform Intake endpoints may use a different origin in a container than in the source config; independent collectors keep their own destination. See gateway telemetry routing and troubleshooting.
How It Works
What It Does
Intake normalizes agent telemetry across multiple ingestion paths into queryable spans and traces. It preserves captured inputs and outputs, stores semantic fields such as model, provider, tool name, status, token counts, cost, and errors, and keeps unhandled source attributes available for debugging.
Studio reads the same Intake APIs to provide trace lists, span lists, trace detail, span detail, and annotation review pages. The UI is a human review surface; Intake is the ingestion and storage service.
Native Relay Full Captures
Intake interprets native nemo_relay.* fields from Relay full OTLP exports, verified against
nemo-relay==0.9.3. This includes recorded scope kinds, input/output, LangChain response messages,
model identity, usage, tools, and errors. Explicit OpenInference and GenAI semantic attributes take
precedence. Usage is selected as a group: any standard usage measurement, including zero,
suppresses all native usage fallback. Model/provider identity is also selected as a group, so a
standard model or provider does not get paired with the other field from Relay. Missing fields
remain unknown rather than mixing source tuples. Raw native attributes, span IDs, timestamps, and
parent relationships remain available. Missing providers, usage, or reasoning are not inferred.
Evaluator OTLP publication fills missing root input/output from the recorded task instruction and
retained trial final answer. Descendant outputs are never promoted into final answers. Without a
recorded task instruction, input fallback requires an explicit gen_ai.operation.name=invoke_agent
wrapper and agreeing recorded inputs at its topmost agent layer; nested internal agents do not supply
the task boundary. Otherwise nemo.evaluation.boundary.input or .output records an unknown-boundary
diagnostic. Empty task instructions and retained answers are missing; explicitly recorded root values,
including empty strings, still win. Root span payloads win over resource-level aliases, which are
used only when that root recorded no payload. Nested identities are preserved. Republishing retained
captures does not require rerunning agents or judges.
Usage Estimates
For usage not covered by the Inference Gateway ledger, OTLP rollups are interim estimates, not verified reconciliation. For each trace and each metric independently, Intake takes the maximum of summed model-call counters and the largest recorded non-model aggregate (including parent, agent, tool, or chain measurements), then sums the trace estimates. It never adds parent and child representations of the same usage or sums nested aggregate representations. Gateway call coverage and superseded-trace handling still take precedence so the same gateway-served calls are not counted again from agent captures. ATIF retains its ingest-reconciled ledger.
Explicit zero is a measurement; a metric with no reported measurements remains unknown. Partial captures can undercount usage, and conflicting sources can produce inconsistent input/output/total metrics because each metric is estimated independently. Separate sibling aggregates without model-call counters are not reconciled into an additive ledger. Inspect the original span measurements to assess coverage and conflicts; these rollups do not certify completeness or billing accuracy. This query-time policy applies to existing stored non-gateway OTLP traces too: larger parent/tool measurements can replace smaller call totals, and sibling aggregates of 100 and 200 without call counters yield 200, not their former sum of 300.
Avoid duplicate model-call instrumentation. Relay and a second OpenInference/GenAI exporter can record the same logical call under distinct span IDs. Those spans are counted as distinct model calls; Intake does not guess cross-exporter call equivalence from matching names, times, or counters. Dual exporters can therefore double call usage even when a parent aggregate is present. Publish one usage representation per call, or reconcile duplicate instrumentation upstream; the parent/child estimate policy does not deduplicate separate exporter call spans.
When to Use It
Use Intake when you need to:
- Feed the optimization loop with production traces, staged replays, seed interactions, or synthetic cases.
- Diagnose specific failures from concrete runs instead of reconstructing behavior from logs.
- Establish baselines before changing a prompt, model, tool, guardrail, or agent workflow.
- Standardize telemetry across teams so everyone shares one vocabulary, data model, and review surface.
Production traffic provides useful optimization evidence. New agents can start with staged traffic or generated cases, then improve coverage as production traffic arrives.
Intake accepts three standardized ingest formats depending on your instrumentation:
Core Concepts and Data Model
Telemetry uses a three-level hierarchy:
- A span is one timed operation, such as a large language model (LLM) call, tool invocation, retrieval, guardrail, evaluator, or chain step.
- A trace is one end-to-end agent run, made of spans that share a trace ID.
- A session groups related traces, such as a multi-turn conversation or a multi-system evaluation.
Additional records attach signal to the same session or span:
- Annotations store post-hoc feedback, labels, notes, and metadata.
- Evaluator results store numeric, boolean, categorical, or text scores for a span.
- Experiments and Evaluations organize evaluation runs into leaderboard rollups for comparison. An Evaluation is a named run whose sessions are individual test cases; an Experiment rolls related Evaluations into one leaderboard. Refer to Experiments.
The most useful instrumentation logs granular steps: every model call, tool call, final response, and error. A practical test: the trace can answer where did the system go wrong, and why?
Get Started
Setup
The examples call the Intake API directly with curl, which can’t read your CLI context. They use
$NHX_BASE_URL, which defaults to a local platform from nemo setup or nemo quickstart up. If your
current context points at a deployed platform, export its URL as NHX_BASE_URL first. For details,
refer to When to use NHX_BASE_URL.
Set NHX_STUDIO_URL explicitly if Studio uses a different origin or path.
The examples use bearer-token authentication and stop on HTTP errors. For a local development
deployment with authentication disabled, omit the Authorization header and the
NHX_ACCESS_TOKEN command. Also omit OTEL_EXPORTER_OTLP_HEADERS from the OTLP example.
Running from source (local development)
If you are bringing the pieces up yourself from a repository checkout, make sure Docker Desktop or
the Docker daemon is running, then start the backend. Intake automatically provisions and reuses a
local ClickHouse container for the resolved NeMo data directory unless NHX_INTAKE_CLICKHOUSE_URL
explicitly points to an external instance.
-
Backend services:
intakeplus itsauth(access checks) andentities(entity store) dependencies.--portdefaults to8080; omituv runif you installed thenemocommand-line interface (CLI): -
Studio (optional, for the UI review flow). From the
web/workspace, with the intake feature flag on and pointed at the backend:
Confirm the Intake read path can reach ClickHouse:
A 200 response with an empty list is healthy. A 503 response means the Intake service is running
but cannot reach ClickHouse.
First Workflow
Send one captured chat-completion interaction. This basic verification test does not require an OpenTelemetry exporter:
Confirm It Worked
Read the span back from Intake:
Then open $NHX_STUDIO_URL/workspaces/$WORKSPACE/intake/traces.
A successful result has the following characteristics:
- The API call returns your interaction: one span with
session_iddemo-session-001, modelexample-model, and the prompt and response you sent as its input and output. - Studio, pointed at the same workspace, lists that trace, and opening it shows the same request and response in the span tree.
The interaction is now queryable through the application programming interface (API) and reviewable in the UI. The captured fields include session, trace, span, request, response, status, error, feedback, and evaluator results. The remaining documentation uses this vocabulary.
Common Workflows
Send OTLP Traces from an Instrumented Agent
Use OTLP when your framework or collector already emits OpenTelemetry traces:
Then run the instrumented agent. Intake maps OpenInference and OpenTelemetry generative AI semantic attributes onto span fields. You can then query model, tool, status, token, and error data.
Send ATIF Trajectories
Use ATIF when your source system exports complete agent trajectories, including steps, agent metadata, or final metrics:
Each trajectory is stored as a structured session of spans. If the trajectory’s top-level
extra.verifier_result.rewards is populated, Intake writes those as evaluator results automatically.
Stock Harbor output does not populate it. Harbor rewards are stored in a separate reward.json, so
you must enrich the output first. Refer to
Capture Evaluator Results for the automatic and explicit paths.
Add Feedback and Labels
Use annotations to attach review signal after a trace lands:
Use kind: "note" for reviewer notes, kind: "label" for categorical or numeric labels, and
kind: "metadata" for structured key/value context.
Capture Evaluator Results
Evaluator results attach a judge rating, pass or fail check, or similarity score to a span. You can send them to Intake in two ways.
Automatically, from ATIF. Put a verifier_result.rewards object in the trajectory’s top-level
extra, keyed by criterion ({criterion: score}). On ingest, Intake synthesizes an evaluator span
named harbor.verifier and writes one evaluator result per key onto it; a bare scalar
extra.verifier_result.score instead becomes a single reward result. Nothing else to send.
A stock Harbor trajectory.json does not carry this value. Harbor writes rewards to a separate
reward.json, so you must copy them into the ATIF extra before ingestion or use the explicit path
below. For example:
Explicitly, through the API. Post a result against a span and session. Unlike the ATIF automatic
path, which lands on the synthesized harbor.verifier span, this attaches to the exact span_id you
post. data_type picks which field carries the score:
For example, a numeric score:
Read them back for one span, or list and filter across the workspace by evaluator name, data type, or value range:
Find Recurring Failures
Query traces or spans with filters, then group spans by session or trace to find repeated errors:
In Studio, start from the trace list, filter to negative feedback or error status, then open trace detail to inspect the span tree.
Turn Evaluation Telemetry into a Leaderboard
Once you are ingesting evaluation runs, group them into an Experiment to compare them side by
side. Refer to Experiments, or use the agent-assisted
nemo-experiments-upload skill to go from zero to a populated leaderboard.
To compare versions of one agent, give each candidate its own Evaluation and select its traces by Evaluation name. Refer to Evaluate Agent Candidates.
Operations
Limits
- OTLP request bodies are capped at 5 MiB by default.
- Read endpoints default to a 30-day lookback when no time filter is supplied.
- Page size is capped at 1,000 records.
- Span data and trace-index data expire after 90 days.
Raise the OTLP body cap with NHX_INTAKE_OTLP_MAX_BODY_BYTES when larger batches are required. For
high-volume producers, prefer smaller export batches over very large single requests.
Security and Access Control
All Intake endpoints are workspace-scoped under /apis/intake/v2/workspaces/{workspace}/. The service
depends on platform auth and checks workspace access before ingesting or reading telemetry.
Do not send secrets, credentials, raw personally identifiable information (PII), or regulated data unless your deployment, retention, and access-control policy permits it. Intake preserves request and response payloads so reviewers can diagnose behavior. This data is useful for debugging and important for data governance.
Retention and Storage
ClickHouse is the telemetry datastore. Intake owns the ClickHouse schema and lazily initializes the tables on first use.
Span and trace-index tables have a 90-day time-to-live (TTL) setting. Annotations and evaluator results are retained in their ClickHouse tables without that 90-day span TTL.
For production storage sizing, estimate span volume across the retained window, then size ClickHouse for interactive reads over recent traces and periodic aggregate queries.
Related Topics
- Experiments: roll evaluation runs up into a comparable leaderboard built from this telemetry.
- Evaluate Agent Candidates: keep each candidate’s traces apart by Evaluation.
- Insights from Traces: find recurring failures in these traces.
- Optimize Agents: use captured traces as the baseline for optimization.