> For clean Markdown of any page, append .md to the page URL.
> For a complete documentation index, see https://docs.nvidia.com/nemo-platform/llms.txt.
> For AI client integration (Claude Code, Cursor, etc.), connect to the MCP server at https://docs.nvidia.com/nemo-platform/_mcp/server.

# Observe Agents

> Ingest, store, and query agent telemetry with NeMo Intake — OTLP, chat-completion, and ATIF ingest paths, annotations, and evaluator results, reviewable in NeMo Studio.

<a id="agents-observability" />

Observability on NeMo Platform is provided by **NeMo Intake**, the trace ingestion and query service
for agent telemetry, and optionally by [NeMo Studio](/documentation/studio), the web UI for browsing
that telemetry. Observability helps humans and agents understand what happened in an agent system:
which models were called, which tools were used, what inputs and outputs were produced, where
failures occurred, and what feedback or evaluation scores were attached.

## Before You Start

### Who This Is For

NeMo Intake is for anyone who owns, operates, evaluates, or optimizes an agent. It is written for
engineers wiring telemetry into an agent runtime, and for human and agent reviewers who need to
diagnose failures from traces, in a production system or in offline evaluation flows.

### Requirements

* A reachable ClickHouse database.
* A NeMo Platform environment with the `intake` service running.
* A telemetry source: an OpenTelemetry/OpenInference exporter, NeMo Agent Toolkit (NAT), NeMo Flow,
  NeMo Evaluator runs, captured OpenAI-compatible chat-completion payloads, or ATIF trajectories.
* (Optional) [NeMo Studio](/documentation/studio) for the UI review flow.

You do not need to deploy your agents on NeMo Platform. The lightweight path is to run the platform,
post one representative interaction, and confirm it appears.

## How It Works

### What It Does

Intake normalizes agent telemetry across multiple ingestion paths into queryable spans and traces. It
preserves captured inputs and outputs, stores semantic fields such as model, provider, tool name,
status, token counts, cost, and errors, and keeps unhandled source attributes available for
debugging.

Studio reads the same Intake APIs to provide trace lists, span lists, trace detail, span detail, and
annotation review pages. The UI is a human review surface; Intake is the ingestion and storage
service.

### When to Use It

Use Intake when you need to:

* Feed the optimization loop with production traces, staged replays, seed interactions, or synthetic
  cases.
* Diagnose specific failures from concrete runs instead of reconstructing behavior from logs.
* Establish baselines before changing a prompt, model, tool, guardrail, or agent workflow.
* Standardize telemetry across teams so everyone shares one vocabulary, data model, and review
  surface.

Production traffic is the best fuel for optimization. New agents can start with staged traffic or
generated cases, then improve coverage as real traffic arrives.

Intake accepts three standardized ingest formats depending on your instrumentation:

| Format                                                                | Example use cases                                                | Endpoint                                                         |
| --------------------------------------------------------------------- | ---------------------------------------------------------------- | ---------------------------------------------------------------- |
| OTLP/HTTP protobuf (OTel GenAI or OpenInference semantic conventions) | NeMo Relay, LangChain Deep Agents                                | `/apis/intake/v2/workspaces/{workspace}/ingest/otlp/v1/traces`   |
| Chat completions                                                      | Importing raw logs, instrumenting a proxy server, custom logging | `/apis/intake/v2/workspaces/{workspace}/ingest/chat-completions` |
| ATIF                                                                  | Running Harbor evaluations                                       | `/apis/intake/v2/workspaces/{workspace}/ingest/atif`             |

### Core Concepts and Data Model

Telemetry uses a three-level hierarchy:

* A **span** is one timed operation: an LLM call, tool invocation, retrieval, guardrail, evaluator, or
  chain step.
* A **trace** is one end-to-end agent run, made of spans that share a trace ID.
* A **session** groups related traces, such as a multi-turn conversation or a multi-system evaluation.

Additional records attach signal to the same session or span:

* **Annotations** store post-hoc feedback, labels, notes, and metadata.
* **Evaluator results** store numeric, boolean, categorical, or text scores for a span.
* **Experiments** and **Evaluations** organize evaluation runs into leaderboard rollups for
  comparison. An Evaluation is a named run whose sessions are individual test cases; an Experiment
  rolls related Evaluations into one leaderboard. See
  [Experiments](/documentation/evaluate-models/experiments).

The most useful instrumentation logs granular steps: every model call, tool call, final response, and
error. A practical test: the trace can answer *where did the system go wrong, and why?*

## Get Started

### Setup

The examples assume a running NeMo Platform reachable at `$NMP_BASE_URL`. Point at whatever you have,
whether a deployed platform or a local one from `nemo setup` or `nemo quickstart up`:

```shell
export NMP_BASE_URL=http://127.0.0.1:8080
export WORKSPACE=default
```

#### Running from source (local development)

If you are bringing the pieces up yourself from a repository checkout, start them in this order.

1. **ClickHouse** (the telemetry datastore):

   ```shell
   services/intake/scripts/spans/run_clickhouse.sh
   ```

2. **Backend services**: `intake` plus its `auth` (access checks) and `entities` (entity store)
   dependencies. `--port` defaults to `8080`; drop `uv run` if you installed the `nemo` CLI:

   ```shell
   uv run nemo services run --services auth,entities,intake --host 127.0.0.1 --port 8080
   ```

3. **Studio** (optional, for the UI review flow). From the `web/` workspace, with the intake feature
   flag on and pointed at the backend:

   ```shell
   VITE_FF_INTAKE_ENABLED=true VITE_PLATFORM_BASE_URL=http://127.0.0.1:8080 \
     pnpm --filter nemo-studio-ui start -- --host 127.0.0.1
   ```

Confirm the Intake read path can reach ClickHouse:

```shell
curl -i "$NMP_BASE_URL/apis/intake/v2/workspaces/$WORKSPACE/spans?page=1&page_size=1"
```

A `200` response with an empty list is healthy. A `503` response means the Intake service is running
but cannot reach ClickHouse.

### First Workflow

Send one captured chat-completion interaction. This is the lowest-friction smoke test — it needs no
OpenTelemetry exporter:

```shell
export SESSION_ID="demo-session-001"
curl -X POST "$NMP_BASE_URL/apis/intake/v2/workspaces/$WORKSPACE/ingest/chat-completions" \
  -H "Content-Type: application/json" \
  -d '{
        "session_id": "'"$SESSION_ID"'",
        "provider": "example",
        "request": {
          "model": "example-model",
          "messages": [
            { "role": "user", "content": "Summarize this alert and identify the likely next action." }
          ]
        },
        "response": {
          "choices": [
            { "message": { "role": "assistant",
              "content": "This appears to be a developing incident. Verify the source, check related signals, then decide whether to escalate." } }
          ]
        }
      }'
```

### Confirm It Worked

Read the span back from Intake:

```shell
curl "$NMP_BASE_URL/apis/intake/v2/workspaces/$WORKSPACE/spans?filter[session_id]=$SESSION_ID&page=1&page_size=10"
```

Then open Studio and navigate to `/workspaces/default/intake/traces`.

**What good looks like:**

* The API call returns your interaction: one span with `session_id` `demo-session-001`, model
  `example-model`, and the prompt and response you sent as its input and output.
* Studio, pointed at the same workspace, lists that trace, and opening it shows the same request and
  response in the span tree.

That is the full loop: something you posted is now queryable through the API and reviewable in the UI.
The fields Intake captured here — session, trace, span, request, response, status, error, plus any
feedback or evaluator results — are the vocabulary the rest of these docs build on.

## Common Workflows

### Send OTLP Traces from an Instrumented Agent

Use OTLP when your framework or collector already emits OpenTelemetry traces:

```shell
export OTEL_EXPORTER_OTLP_TRACES_ENDPOINT="$NMP_BASE_URL/apis/intake/v2/workspaces/$WORKSPACE/ingest/otlp/v1/traces"
export OTEL_EXPORTER_OTLP_PROTOCOL=http/protobuf
```

Then run the instrumented agent. Intake maps OpenInference and OTel GenAI semantic attributes onto
span fields so model, tool, status, token, and error data are queryable.

### Send ATIF Trajectories

Use ATIF when your source system exports complete agent trajectories — steps, agent metadata, or final
metrics:

```shell
curl -X POST "$NMP_BASE_URL/apis/intake/v2/workspaces/$WORKSPACE/ingest/atif" \
  -H "Content-Type: application/json" \
  --data-binary @trajectory.json
```

Each trajectory is stored as a structured session of spans. If the trajectory's top-level
`extra.verifier_result.rewards` is populated, Intake writes those as evaluator results automatically.
Stock Harbor output does not populate it — Harbor's rewards live in a separate `reward.json` — so it
must be enriched first. See [Capture Evaluator Results](#capture-evaluator-results) for the automatic
and explicit paths.

### Add Feedback and Labels

Use annotations to attach review signal after a trace lands:

```shell
curl -X POST "$NMP_BASE_URL/apis/intake/v2/workspaces/$WORKSPACE/annotations" \
  -H "Content-Type: application/json" \
  -d '{ "kind": "feedback", "session_id": "'"$SESSION_ID"'", "value": "negative" }'
```

Use `kind: "note"` for reviewer notes, `kind: "label"` for categorical or numeric labels, and
`kind: "metadata"` for structured key/value context.

### Capture Evaluator Results

<a id="capture-evaluator-results" />

Evaluator results — a judge's rating, a pass/fail check, a similarity score — attach to a span. There
are two ways to get them into Intake.

**Automatically, from ATIF.** Put a `verifier_result.rewards` object in the trajectory's **top-level**
`extra`, keyed by criterion (`{criterion: score}`). On ingest, Intake synthesizes an evaluator span
named `harbor.verifier` and writes one evaluator result per key onto it; a bare scalar
`extra.verifier_result.score` instead becomes a single `reward` result. Nothing else to send.

A stock Harbor `trajectory.json` does **not** carry this — Harbor writes rewards to a separate
`reward.json` — so you must copy them into the ATIF `extra` before ingest, or use the explicit path
below. For example:

```json
"extra": { "verifier_result": { "rewards": { "solved": 1, "groundedness": 0.8 } } }
```

**Explicitly, through the API.** Post a result against a span and session. Unlike the ATIF automatic
path, which lands on the synthesized `harbor.verifier` span, this attaches to the exact `span_id` you
post. `data_type` picks which field carries the score:

| `data_type`   | Where the score goes        |
| ------------- | --------------------------- |
| `NUMERIC`     | A number in `value`         |
| `BOOLEAN`     | `value`, either `0` or `1`  |
| `CATEGORICAL` | A label in `string_value`   |
| `TEXT`        | Free text in `string_value` |

For example, a numeric score:

```shell
curl -X POST "$NMP_BASE_URL/apis/intake/v2/workspaces/$WORKSPACE/evaluator-results" \
  -H "Content-Type: application/json" \
  -d '{
        "span_id": "<span-id>",
        "session_id": "'"$SESSION_ID"'",
        "name": "faithfulness/v1",
        "data_type": "NUMERIC",
        "value": 0.82
      }'
```

Read them back for one span, or list and filter across the workspace by evaluator name, data type, or
value range:

```shell
curl "$NMP_BASE_URL/apis/intake/v2/workspaces/$WORKSPACE/spans/<span-id>/evaluator-results"
curl -g "$NMP_BASE_URL/apis/intake/v2/workspaces/$WORKSPACE/evaluator-results?filter[name]=faithfulness/v1&filter[value][\$gte]=0.8"
```

### Find Recurring Failures

Query traces or spans with filters, then group spans by session or trace to find repeated errors:

```shell
curl -g "$NMP_BASE_URL/apis/intake/v2/workspaces/$WORKSPACE/spans?filter[status]=ERROR&page=1&page_size=20"
curl -g "$NMP_BASE_URL/apis/intake/v2/workspaces/$WORKSPACE/spans/groups?by=session_id&filter[status]=ERROR"
```

In Studio, start from the trace list, filter to negative feedback or error status, then open trace
detail to inspect the span tree.

### Turn Evaluation Telemetry into a Leaderboard

Once you are ingesting evaluation runs, group them into an **Experiment** to compare them side by
side. See [Experiments](/documentation/evaluate-models/experiments), or use the agent-assisted
`nemo-experiments-upload` skill to go from zero to a populated leaderboard.

## Operations

### Limits

* OTLP request bodies are capped at 5 MiB by default.
* Read endpoints default to a 30-day lookback when no time filter is supplied.
* Page size is capped at 1000 records.
* Span data and trace-index data expire after 90 days.

Raise the OTLP body cap with `NMP_INTAKE_OTLP_MAX_BODY_BYTES` when larger batches are required. For
high-volume producers, prefer smaller export batches over very large single requests.

### Security and Access Control

All Intake endpoints are workspace-scoped under `/apis/intake/v2/workspaces/{workspace}/`. The service
depends on platform auth and checks workspace access before ingesting or reading telemetry.

Do not send secrets, credentials, raw PII, or regulated data unless your deployment, retention, and
access-control policy permits it. Intake preserves request and response payloads so reviewers can
diagnose behavior — useful for debugging, and important for data governance.

### Retention and Storage

ClickHouse is the telemetry datastore. Intake owns the ClickHouse schema and lazily initializes the
tables on first use.

Span and trace-index tables have a 90-day TTL. Annotations and evaluator results are retained in their
ClickHouse tables without that 90-day span TTL.

For production storage sizing, estimate span volume across the retained window, then size ClickHouse
for interactive reads over recent traces and periodic aggregate queries.

## Related Topics

* [Experiments](/documentation/evaluate-models/experiments): roll evaluation runs up into a comparable
  leaderboard built from this telemetry.
* [Optimize Agents](/documentation/agents/optimize-agents): use captured traces as the baseline for
  optimization.
* [Secure Agents](/documentation/agents/secure-agents): scan recent telemetry for sensitive data.