DocumentationAgents

Observe Agents

View as Markdown

Observability on NeMo Platform is provided by NeMo Intake, the trace ingestion and query service for agent telemetry, and optionally by NeMo Studio, the web user interface (UI) for browsing that telemetry. Observability helps humans and agents understand what happened in an agent system: which models were called, which tools were used, what inputs and outputs were produced, where failures occurred, and what feedback or evaluation scores were attached.

Before You Start

Who This Is For

NeMo Intake is for anyone who owns, operates, evaluates, or optimizes an agent. It is written for engineers wiring telemetry into an agent runtime, and for human and agent reviewers who need to diagnose failures from traces, in a production system or in offline evaluation flows.

Requirements

  • A reachable ClickHouse database.
  • A NeMo Platform environment with the intake service running.
  • A telemetry source, such as an OpenTelemetry or OpenInference exporter, NeMo Agent Toolkit (NAT), NeMo Flow, NeMo Evaluator runs, captured OpenAI-compatible chat-completion payloads, or Agent Trajectory Interchange Format (ATIF) trajectories.
  • Optional: NeMo Studio for the UI review flow.

You do not need to deploy your agents on NeMo Platform. To start, run the platform, post one representative interaction, and confirm that it appears.

How It Works

What It Does

Intake normalizes agent telemetry across multiple ingestion paths into queryable spans and traces. It preserves captured inputs and outputs, stores semantic fields such as model, provider, tool name, status, token counts, cost, and errors, and keeps unhandled source attributes available for debugging.

Studio reads the same Intake APIs to provide trace lists, span lists, trace detail, span detail, and annotation review pages. The UI is a human review surface; Intake is the ingestion and storage service.

When to Use It

Use Intake when you need to:

  • Feed the optimization loop with production traces, staged replays, seed interactions, or synthetic cases.
  • Diagnose specific failures from concrete runs instead of reconstructing behavior from logs.
  • Establish baselines before changing a prompt, model, tool, guardrail, or agent workflow.
  • Standardize telemetry across teams so everyone shares one vocabulary, data model, and review surface.

Production traffic provides useful optimization evidence. New agents can start with staged traffic or generated cases, then improve coverage as production traffic arrives.

Intake accepts three standardized ingest formats depending on your instrumentation:

FormatExample use casesEndpoint
OpenTelemetry Protocol (OTLP) over HTTP using Protocol Buffers (OpenTelemetry generative AI or OpenInference semantic conventions)NeMo Relay, LangChain Deep Agents/apis/intake/v2/workspaces/{workspace}/ingest/otlp/v1/traces
Chat completionsImporting raw logs, instrumenting a proxy server, custom logging/apis/intake/v2/workspaces/{workspace}/ingest/chat-completions
ATIFRunning Harbor evaluations/apis/intake/v2/workspaces/{workspace}/ingest/atif

Core Concepts and Data Model

Telemetry uses a three-level hierarchy:

  • A span is one timed operation, such as a large language model (LLM) call, tool invocation, retrieval, guardrail, evaluator, or chain step.
  • A trace is one end-to-end agent run, made of spans that share a trace ID.
  • A session groups related traces, such as a multi-turn conversation or a multi-system evaluation.

Additional records attach signal to the same session or span:

  • Annotations store post-hoc feedback, labels, notes, and metadata.
  • Evaluator results store numeric, boolean, categorical, or text scores for a span.
  • Experiments and Evaluations organize evaluation runs into leaderboard rollups for comparison. An Evaluation is a named run whose sessions are individual test cases; an Experiment rolls related Evaluations into one leaderboard. Refer to Experiments.

The most useful instrumentation logs granular steps: every model call, tool call, final response, and error. A practical test: the trace can answer where did the system go wrong, and why?

Get Started

Setup

The examples assume a running NeMo Platform reachable at $NMP_BASE_URL. Point at whatever you have, whether a deployed platform or a local one from nemo setup or nemo quickstart up:

$export NMP_BASE_URL=http://127.0.0.1:8080
$export WORKSPACE=default
$export NMP_STUDIO_URL="$NMP_BASE_URL/studio"
$export NMP_ACCESS_TOKEN="$(nemo auth token)"

Set NMP_STUDIO_URL explicitly if Studio uses a different origin or path. The examples use bearer-token authentication and stop on HTTP errors. For a local development deployment with authentication disabled, omit the Authorization header and the NMP_ACCESS_TOKEN command. Also omit OTEL_EXPORTER_OTLP_HEADERS from the OTLP example.

If you are bringing the pieces up yourself from a repository checkout, make sure Docker Desktop or the Docker daemon is running, then start the backend. Intake automatically provisions and reuses a local ClickHouse container for the resolved NeMo data directory unless NMP_INTAKE_CLICKHOUSE_URL explicitly points to an external instance.

  1. Backend services: intake plus its auth (access checks) and entities (entity store) dependencies. --port defaults to 8080; omit uv run if you installed the nemo command-line interface (CLI):

    $uv run nemo services run --services auth,entities,intake --host 127.0.0.1 --port 8080

  2. Studio (optional, for the UI review flow). From the web/ workspace, with the intake feature flag on and pointed at the backend:

    $VITE_FF_INTAKE_ENABLED=true VITE_PLATFORM_BASE_URL=http://127.0.0.1:8080 \
    >  pnpm --filter nemo-studio-ui start -- --host 127.0.0.1

Confirm the Intake read path can reach ClickHouse:

$curl --fail-with-body -i \
>  -H "Authorization: Bearer $NMP_ACCESS_TOKEN" \
>  "$NMP_BASE_URL/apis/intake/v2/workspaces/$WORKSPACE/spans?page=1&page_size=1"

A 200 response with an empty list is healthy. A 503 response means the Intake service is running but cannot reach ClickHouse.

First Workflow

Send one captured chat-completion interaction. This basic verification test does not require an OpenTelemetry exporter:

$export SESSION_ID="demo-session-001"
$curl --fail-with-body -X POST \
>  "$NMP_BASE_URL/apis/intake/v2/workspaces/$WORKSPACE/ingest/chat-completions" \
>  -H "Authorization: Bearer $NMP_ACCESS_TOKEN" \
>  -H "Content-Type: application/json" \
>  -d '{
>        "session_id": "'"$SESSION_ID"'",
>        "provider": "example",
>        "request": {
>          "model": "example-model",
>          "messages": [
>            { "role": "user", "content": "Summarize this alert and identify the likely next action." }
>          ]
>        },
>        "response": {
>          "choices": [
>            { "message": { "role": "assistant",
>              "content": "This appears to be a developing incident. Verify the source, check related signals, then decide whether to escalate." } }
>          ]
>        }
>      }'

Confirm It Worked

Read the span back from Intake:

$curl -g --fail-with-body \
>  -H "Authorization: Bearer $NMP_ACCESS_TOKEN" \
>  "$NMP_BASE_URL/apis/intake/v2/workspaces/$WORKSPACE/spans?filter[session_id]=$SESSION_ID&page=1&page_size=10"

Then open $NMP_STUDIO_URL/workspaces/$WORKSPACE/intake/traces.

A successful result has the following characteristics:

  • The API call returns your interaction: one span with session_id demo-session-001, model example-model, and the prompt and response you sent as its input and output.
  • Studio, pointed at the same workspace, lists that trace, and opening it shows the same request and response in the span tree.

The interaction is now queryable through the application programming interface (API) and reviewable in the UI. The captured fields include session, trace, span, request, response, status, error, feedback, and evaluator results. The remaining documentation uses this vocabulary.

Common Workflows

Send OTLP Traces from an Instrumented Agent

Use OTLP when your framework or collector already emits OpenTelemetry traces:

$export OTEL_EXPORTER_OTLP_TRACES_ENDPOINT="$NMP_BASE_URL/apis/intake/v2/workspaces/$WORKSPACE/ingest/otlp/v1/traces"
$export OTEL_EXPORTER_OTLP_PROTOCOL=http/protobuf
$export OTEL_EXPORTER_OTLP_HEADERS="Authorization=Bearer%20$NMP_ACCESS_TOKEN"

Then run the instrumented agent. Intake maps OpenInference and OpenTelemetry generative AI semantic attributes onto span fields. You can then query model, tool, status, token, and error data.

Send ATIF Trajectories

Use ATIF when your source system exports complete agent trajectories, including steps, agent metadata, or final metrics:

$curl --fail-with-body -X POST \
>  "$NMP_BASE_URL/apis/intake/v2/workspaces/$WORKSPACE/ingest/atif" \
>  -H "Authorization: Bearer $NMP_ACCESS_TOKEN" \
>  -H "Content-Type: application/json" \
>  --data-binary @trajectory.json

Each trajectory is stored as a structured session of spans. If the trajectory’s top-level extra.verifier_result.rewards is populated, Intake writes those as evaluator results automatically. Stock Harbor output does not populate it. Harbor rewards are stored in a separate reward.json, so you must enrich the output first. Refer to Capture Evaluator Results for the automatic and explicit paths.

Add Feedback and Labels

Use annotations to attach review signal after a trace lands:

$curl --fail-with-body -X POST \
>  "$NMP_BASE_URL/apis/intake/v2/workspaces/$WORKSPACE/annotations" \
>  -H "Authorization: Bearer $NMP_ACCESS_TOKEN" \
>  -H "Content-Type: application/json" \
>  -d '{ "kind": "feedback", "session_id": "'"$SESSION_ID"'", "value": "negative" }'

Use kind: "note" for reviewer notes, kind: "label" for categorical or numeric labels, and kind: "metadata" for structured key/value context.

Capture Evaluator Results

Evaluator results attach a judge rating, pass or fail check, or similarity score to a span. You can send them to Intake in two ways.

Automatically, from ATIF. Put a verifier_result.rewards object in the trajectory’s top-level extra, keyed by criterion ({criterion: score}). On ingest, Intake synthesizes an evaluator span named harbor.verifier and writes one evaluator result per key onto it; a bare scalar extra.verifier_result.score instead becomes a single reward result. Nothing else to send.

A stock Harbor trajectory.json does not carry this value. Harbor writes rewards to a separate reward.json, so you must copy them into the ATIF extra before ingestion or use the explicit path below. For example:

1"extra": { "verifier_result": { "rewards": { "solved": 1, "groundedness": 0.8 } } }

Explicitly, through the API. Post a result against a span and session. Unlike the ATIF automatic path, which lands on the synthesized harbor.verifier span, this attaches to the exact span_id you post. data_type picks which field carries the score:

data_typeWhere the score goes
NUMERICA number in value
BOOLEANvalue, either 0 or 1
CATEGORICALA label in string_value
TEXTFree text in string_value

For example, a numeric score:

$curl --fail-with-body -X POST \
>  "$NMP_BASE_URL/apis/intake/v2/workspaces/$WORKSPACE/evaluator-results" \
>  -H "Authorization: Bearer $NMP_ACCESS_TOKEN" \
>  -H "Content-Type: application/json" \
>  -d '{
>        "span_id": "<span-id>",
>        "session_id": "'"$SESSION_ID"'",
>        "name": "faithfulness/v1",
>        "data_type": "NUMERIC",
>        "value": 0.82
>      }'

Read them back for one span, or list and filter across the workspace by evaluator name, data type, or value range:

$curl --fail-with-body \
>  -H "Authorization: Bearer $NMP_ACCESS_TOKEN" \
>  "$NMP_BASE_URL/apis/intake/v2/workspaces/$WORKSPACE/spans/<span-id>/evaluator-results"
$curl -g --fail-with-body \
>  -H "Authorization: Bearer $NMP_ACCESS_TOKEN" \
>  "$NMP_BASE_URL/apis/intake/v2/workspaces/$WORKSPACE/evaluator-results?filter[name]=faithfulness/v1&filter[value][\$gte]=0.8"

Find Recurring Failures

Query traces or spans with filters, then group spans by session or trace to find repeated errors:

$curl -g --fail-with-body \
>  -H "Authorization: Bearer $NMP_ACCESS_TOKEN" \
>  "$NMP_BASE_URL/apis/intake/v2/workspaces/$WORKSPACE/spans?filter[status]=error&page=1&page_size=20"
$curl -g --fail-with-body \
>  -H "Authorization: Bearer $NMP_ACCESS_TOKEN" \
>  "$NMP_BASE_URL/apis/intake/v2/workspaces/$WORKSPACE/spans/groups?by=session_id&filter[status]=error"

In Studio, start from the trace list, filter to negative feedback or error status, then open trace detail to inspect the span tree.

Turn Evaluation Telemetry into a Leaderboard

Once you are ingesting evaluation runs, group them into an Experiment to compare them side by side. Refer to Experiments, or use the agent-assisted nemo-experiments-upload skill to go from zero to a populated leaderboard.

Operations

Limits

  • OTLP request bodies are capped at 5 MiB by default.
  • Read endpoints default to a 30-day lookback when no time filter is supplied.
  • Page size is capped at 1,000 records.
  • Span data and trace-index data expire after 90 days.

Raise the OTLP body cap with NMP_INTAKE_OTLP_MAX_BODY_BYTES when larger batches are required. For high-volume producers, prefer smaller export batches over very large single requests.

Security and Access Control

All Intake endpoints are workspace-scoped under /apis/intake/v2/workspaces/{workspace}/. The service depends on platform auth and checks workspace access before ingesting or reading telemetry.

Do not send secrets, credentials, raw personally identifiable information (PII), or regulated data unless your deployment, retention, and access-control policy permits it. Intake preserves request and response payloads so reviewers can diagnose behavior. This data is useful for debugging and important for data governance.

Retention and Storage

ClickHouse is the telemetry datastore. Intake owns the ClickHouse schema and lazily initializes the tables on first use.

Span and trace-index tables have a 90-day time-to-live (TTL) setting. Annotations and evaluator results are retained in their ClickHouse tables without that 90-day span TTL.

For production storage sizing, estimate span volume across the retained window, then size ClickHouse for interactive reads over recent traces and periodic aggregate queries.

  • Experiments: roll evaluation runs up into a comparable leaderboard built from this telemetry.
  • Optimize Agents: use captured traces as the baseline for optimization.
  • Secure Agents: scan recent telemetry for sensitive data.