DocumentationAgents

Observe Agents

View as Markdown

Observability on NeMo Platform is provided by NeMo Intake, the trace ingestion and query service for agent telemetry, and optionally by NeMo Studio, the web UI for browsing that telemetry. Observability helps humans and agents understand what happened in an agent system: which models were called, which tools were used, what inputs and outputs were produced, where failures occurred, and what feedback or evaluation scores were attached.

Before You Start

Who This Is For

NeMo Intake is for anyone who owns, operates, evaluates, or optimizes an agent. It is written for engineers wiring telemetry into an agent runtime, and for human and agent reviewers who need to diagnose failures from traces, in a production system or in offline evaluation flows.

Requirements

  • A reachable ClickHouse database.
  • A NeMo Platform environment with the intake service running.
  • A telemetry source: an OpenTelemetry/OpenInference exporter, NeMo Agent Toolkit (NAT), NeMo Flow, NeMo Evaluator runs, captured OpenAI-compatible chat-completion payloads, or ATIF trajectories.
  • (Optional) NeMo Studio for the UI review flow.

You do not need to deploy your agents on NeMo Platform. The lightweight path is to run the platform, post one representative interaction, and confirm it appears.

How It Works

What It Does

Intake normalizes agent telemetry across multiple ingestion paths into queryable spans and traces. It preserves captured inputs and outputs, stores semantic fields such as model, provider, tool name, status, token counts, cost, and errors, and keeps unhandled source attributes available for debugging.

Studio reads the same Intake APIs to provide trace lists, span lists, trace detail, span detail, and annotation review pages. The UI is a human review surface; Intake is the ingestion and storage service.

When to Use It

Use Intake when you need to:

  • Feed the optimization loop with production traces, staged replays, seed interactions, or synthetic cases.
  • Diagnose specific failures from concrete runs instead of reconstructing behavior from logs.
  • Establish baselines before changing a prompt, model, tool, guardrail, or agent workflow.
  • Standardize telemetry across teams so everyone shares one vocabulary, data model, and review surface.

Production traffic is the best fuel for optimization. New agents can start with staged traffic or generated cases, then improve coverage as real traffic arrives.

Intake accepts three standardized ingest formats depending on your instrumentation:

FormatExample use casesEndpoint
OTLP/HTTP protobuf (OTel GenAI or OpenInference semantic conventions)NeMo Relay, LangChain Deep Agents/apis/intake/v2/workspaces/{workspace}/ingest/otlp/v1/traces
Chat completionsImporting raw logs, instrumenting a proxy server, custom logging/apis/intake/v2/workspaces/{workspace}/ingest/chat-completions
ATIFRunning Harbor evaluations/apis/intake/v2/workspaces/{workspace}/ingest/atif

Core Concepts and Data Model

Telemetry uses a three-level hierarchy:

  • A span is one timed operation: an LLM call, tool invocation, retrieval, guardrail, evaluator, or chain step.
  • A trace is one end-to-end agent run, made of spans that share a trace ID.
  • A session groups related traces, such as a multi-turn conversation or a multi-system evaluation.

Additional records attach signal to the same session or span:

  • Annotations store post-hoc feedback, labels, notes, and metadata.
  • Evaluator results store numeric, boolean, categorical, or text scores for a span.
  • Experiments and Evaluations organize evaluation runs into leaderboard rollups for comparison. An Evaluation is a named run whose sessions are individual test cases; an Experiment rolls related Evaluations into one leaderboard. See Experiments.

The most useful instrumentation logs granular steps: every model call, tool call, final response, and error. A practical test: the trace can answer where did the system go wrong, and why?

Get Started

Setup

The examples assume a running NeMo Platform reachable at $NMP_BASE_URL. Point at whatever you have, whether a deployed platform or a local one from nemo setup or nemo quickstart up:

$export NMP_BASE_URL=http://127.0.0.1:8080
$export WORKSPACE=default

If you are bringing the pieces up yourself from a repository checkout, start them in this order.

  1. ClickHouse (the telemetry datastore):

    $services/intake/scripts/spans/run_clickhouse.sh

  2. Backend services: intake plus its auth (access checks) and entities (entity store) dependencies. --port defaults to 8080; drop uv run if you installed the nemo CLI:

    $uv run nemo services run --services auth,entities,intake --host 127.0.0.1 --port 8080

  3. Studio (optional, for the UI review flow). From the web/ workspace, with the intake feature flag on and pointed at the backend:

    $VITE_FF_INTAKE_ENABLED=true VITE_PLATFORM_BASE_URL=http://127.0.0.1:8080 \
    >  pnpm --filter nemo-studio-ui start -- --host 127.0.0.1

Confirm the Intake read path can reach ClickHouse:

$curl -i "$NMP_BASE_URL/apis/intake/v2/workspaces/$WORKSPACE/spans?page=1&page_size=1"

A 200 response with an empty list is healthy. A 503 response means the Intake service is running but cannot reach ClickHouse.

First Workflow

Send one captured chat-completion interaction. This is the lowest-friction smoke test — it needs no OpenTelemetry exporter:

$export SESSION_ID="demo-session-001"
$curl -X POST "$NMP_BASE_URL/apis/intake/v2/workspaces/$WORKSPACE/ingest/chat-completions" \
>  -H "Content-Type: application/json" \
>  -d '{
>        "session_id": "'"$SESSION_ID"'",
>        "provider": "example",
>        "request": {
>          "model": "example-model",
>          "messages": [
>            { "role": "user", "content": "Summarize this alert and identify the likely next action." }
>          ]
>        },
>        "response": {
>          "choices": [
>            { "message": { "role": "assistant",
>              "content": "This appears to be a developing incident. Verify the source, check related signals, then decide whether to escalate." } }
>          ]
>        }
>      }'

Confirm It Worked

Read the span back from Intake:

$curl "$NMP_BASE_URL/apis/intake/v2/workspaces/$WORKSPACE/spans?filter[session_id]=$SESSION_ID&page=1&page_size=10"

Then open Studio and navigate to /workspaces/default/intake/traces.

What good looks like:

  • The API call returns your interaction: one span with session_id demo-session-001, model example-model, and the prompt and response you sent as its input and output.
  • Studio, pointed at the same workspace, lists that trace, and opening it shows the same request and response in the span tree.

That is the full loop: something you posted is now queryable through the API and reviewable in the UI. The fields Intake captured here — session, trace, span, request, response, status, error, plus any feedback or evaluator results — are the vocabulary the rest of these docs build on.

Common Workflows

Send OTLP Traces from an Instrumented Agent

Use OTLP when your framework or collector already emits OpenTelemetry traces:

$export OTEL_EXPORTER_OTLP_TRACES_ENDPOINT="$NMP_BASE_URL/apis/intake/v2/workspaces/$WORKSPACE/ingest/otlp/v1/traces"
$export OTEL_EXPORTER_OTLP_PROTOCOL=http/protobuf

Then run the instrumented agent. Intake maps OpenInference and OTel GenAI semantic attributes onto span fields so model, tool, status, token, and error data are queryable.

Send ATIF Trajectories

Use ATIF when your source system exports complete agent trajectories — steps, agent metadata, or final metrics:

$curl -X POST "$NMP_BASE_URL/apis/intake/v2/workspaces/$WORKSPACE/ingest/atif" \
>  -H "Content-Type: application/json" \
>  --data-binary @trajectory.json

Each trajectory is stored as a structured session of spans. If the trajectory’s top-level extra.verifier_result.rewards is populated, Intake writes those as evaluator results automatically. Stock Harbor output does not populate it — Harbor’s rewards live in a separate reward.json — so it must be enriched first. See Capture Evaluator Results for the automatic and explicit paths.

Add Feedback and Labels

Use annotations to attach review signal after a trace lands:

$curl -X POST "$NMP_BASE_URL/apis/intake/v2/workspaces/$WORKSPACE/annotations" \
>  -H "Content-Type: application/json" \
>  -d '{ "kind": "feedback", "session_id": "'"$SESSION_ID"'", "value": "negative" }'

Use kind: "note" for reviewer notes, kind: "label" for categorical or numeric labels, and kind: "metadata" for structured key/value context.

Capture Evaluator Results

Evaluator results — a judge’s rating, a pass/fail check, a similarity score — attach to a span. There are two ways to get them into Intake.

Automatically, from ATIF. Put a verifier_result.rewards object in the trajectory’s top-level extra, keyed by criterion ({criterion: score}). On ingest, Intake synthesizes an evaluator span named harbor.verifier and writes one evaluator result per key onto it; a bare scalar extra.verifier_result.score instead becomes a single reward result. Nothing else to send.

A stock Harbor trajectory.json does not carry this — Harbor writes rewards to a separate reward.json — so you must copy them into the ATIF extra before ingest, or use the explicit path below. For example:

1"extra": { "verifier_result": { "rewards": { "solved": 1, "groundedness": 0.8 } } }

Explicitly, through the API. Post a result against a span and session. Unlike the ATIF automatic path, which lands on the synthesized harbor.verifier span, this attaches to the exact span_id you post. data_type picks which field carries the score:

data_typeWhere the score goes
NUMERICA number in value
BOOLEANvalue, either 0 or 1
CATEGORICALA label in string_value
TEXTFree text in string_value

For example, a numeric score:

$curl -X POST "$NMP_BASE_URL/apis/intake/v2/workspaces/$WORKSPACE/evaluator-results" \
>  -H "Content-Type: application/json" \
>  -d '{
>        "span_id": "<span-id>",
>        "session_id": "'"$SESSION_ID"'",
>        "name": "faithfulness/v1",
>        "data_type": "NUMERIC",
>        "value": 0.82
>      }'

Read them back for one span, or list and filter across the workspace by evaluator name, data type, or value range:

$curl "$NMP_BASE_URL/apis/intake/v2/workspaces/$WORKSPACE/spans/<span-id>/evaluator-results"
$curl -g "$NMP_BASE_URL/apis/intake/v2/workspaces/$WORKSPACE/evaluator-results?filter[name]=faithfulness/v1&filter[value][\$gte]=0.8"

Find Recurring Failures

Query traces or spans with filters, then group spans by session or trace to find repeated errors:

$curl -g "$NMP_BASE_URL/apis/intake/v2/workspaces/$WORKSPACE/spans?filter[status]=ERROR&page=1&page_size=20"
$curl -g "$NMP_BASE_URL/apis/intake/v2/workspaces/$WORKSPACE/spans/groups?by=session_id&filter[status]=ERROR"

In Studio, start from the trace list, filter to negative feedback or error status, then open trace detail to inspect the span tree.

Turn Evaluation Telemetry into a Leaderboard

Once you are ingesting evaluation runs, group them into an Experiment to compare them side by side. See Experiments, or use the agent-assisted nemo-experiments-upload skill to go from zero to a populated leaderboard.

Operations

Limits

  • OTLP request bodies are capped at 5 MiB by default.
  • Read endpoints default to a 30-day lookback when no time filter is supplied.
  • Page size is capped at 1000 records.
  • Span data and trace-index data expire after 90 days.

Raise the OTLP body cap with NMP_INTAKE_OTLP_MAX_BODY_BYTES when larger batches are required. For high-volume producers, prefer smaller export batches over very large single requests.

Security and Access Control

All Intake endpoints are workspace-scoped under /apis/intake/v2/workspaces/{workspace}/. The service depends on platform auth and checks workspace access before ingesting or reading telemetry.

Do not send secrets, credentials, raw PII, or regulated data unless your deployment, retention, and access-control policy permits it. Intake preserves request and response payloads so reviewers can diagnose behavior — useful for debugging, and important for data governance.

Retention and Storage

ClickHouse is the telemetry datastore. Intake owns the ClickHouse schema and lazily initializes the tables on first use.

Span and trace-index tables have a 90-day TTL. Annotations and evaluator results are retained in their ClickHouse tables without that 90-day span TTL.

For production storage sizing, estimate span volume across the retained window, then size ClickHouse for interactive reads over recent traces and periodic aggregate queries.

  • Experiments: roll evaluation runs up into a comparable leaderboard built from this telemetry.
  • Optimize Agents: use captured traces as the baseline for optimization.
  • Secure Agents: scan recent telemetry for sensitive data.