DocumentationEvaluate Models & Agents

Experiments

View as Markdown

NeMo Experiments is the comparison layer for agent optimization tasks, such as evaluating the impact of changes to a harness, infrastructure, tools, or agent code. It gives you one place to visualize and compare evaluation results from the runner of your choice — Harbor, the NeMo Optimizer, NeMo Evaluator, or your own — ranked on the metrics that matter, such as cost, latency, and evaluator scores.

Those metrics are derived from NeMo Intake observability data, the same traces and evaluator results your runs already produce, so any runner that lands telemetry in Intake feeds the same comparison.

Before You Start

Who This Is For

NeMo Experiments is for anyone running evaluations on their agent who needs a way to compare, analyze, and promote the results. For example: engineers iterating on prompts, models, tools, or routing; teams benchmarking many models or configurations at once; and reviewers who need a shared leaderboard to decide what to promote.

Requirements

  • A NeMo Platform environment with the intake service running.
  • The platform entity store (Postgres).
  • A reachable ClickHouse database.
  • A producer that creates Evaluations and sends their telemetry: the NeMo Optimizer, a benchmark or evaluation framework such as Harbor, or a direct API integration.
  • (Recommended) NeMo Studio for the full UI — trace comparison views, a customizable leaderboard, and Pareto charts. Everything is also available through the API, but Studio is where the experience really lives. The Experiments UI is gated by the VITE_FF_EXPERIMENT feature flag, which is off by default.

How It Works

What It Does

Experiments computes comparable, always-current metrics for your evaluation runs. For each run it rolls up cost, latency, tokens, and per-evaluator scores from the underlying telemetry in ClickHouse and returns them as ranked rows. Rollups are computed at read time, so a leaderboard always reflects current telemetry, with no denormalized score table to maintain.

You group runs however makes sense — an insight to investigate, a model bake-off, a benchmark leaderboard, or the top runs promoted from several groups — and a single run can belong to more than one group. Sorting, filtering, and pinning within a group surface the runs that matter.

When to Use It

Use Experiments when you need to:

  • Compare candidates from the optimizer, a prompt or model change, or a routing strategy against a baseline.
  • Run a benchmark leaderboard: import many runs into one Experiment and rank them by the metric that matters.
  • Track iteration over time: keep one Experiment per project so successive attempts stay side by side.
  • Standardize comparison across a team: everyone reads the same metrics, in the same place, with the same vocabulary.

Experiments are most useful once real evaluation telemetry exists. A new group can start empty and fill in as runs land.

Core Concepts and Data Model

Experiments sit on top of Intake’s telemetry hierarchy (span → trace → session):

  • An Experiment is a named container of Evaluations. It holds durable metadata (description, summary, free-form metadata, optional insight_id) and view configuration: a default_sort and a pareto (default X/Y metrics for the Pareto view).
  • An Evaluation is one run and one leaderboard row. It records producer-supplied fields (dataset_name, dataset_version, source_link, metadata, description, status, root_cause), the experiment_ids it belongs to (at least one — an Evaluation can live in more than one Experiment), and an optional parent_evaluation_id linking a variant back to the run it was derived from.
  • A Session is one test-case execution within an Evaluation, a single ingested run made of spans. Sessions carry per-case status, latency, token and cost totals, and evaluator scores.

At read time, each Evaluation is enriched with rollups derived from its sessions in ClickHouse:

RollupMeaning
test_case_countNumber of distinct test cases (distinct non-empty test_case_id values). Sessions with no test_case_id don’t count toward it or the rollups.
cost_usdCost aggregate across the Evaluation’s sessions.
latency_msLatency aggregate across the Evaluation’s sessions.
tokensAverage total tokens (input + output) per test case.
evaluators.<name>Aggregate of a named evaluator’s session scores.
model_names, agent_names, agent_versionsDistinct models, agents, and versions observed in the telemetry.

Metric aggregates expose these statistics: sum, mean, median, p90, p95, p99, count. A metric path is therefore test_case_count, cost_usd.<stat>, latency_ms.<stat>, tokens.<stat>, or evaluators.<name>.<stat> — the same grammar used for sorting and filtering below.

Two things about evaluator rollups specifically:

  • Response shape. evaluators.<name>.<stat> is the sort/filter query grammar. In the Evaluation JSON the same scores come back under aggregate_scores — a map keyed by evaluator name, each value carrying the stats above — alongside an evaluator_names list. There is no evaluators field in the response; evaluators.<name>.<stat> (query) reads aggregate_scores[<name>].<stat> (response).
  • Missing-value semantics. Evaluator rollups are test-case-weighted: each stat is computed over test_case_count, the full set of test cases, with a test case that didn’t report a given evaluator counted as 0 rather than dropped.

Two Experiment-level behaviors are worth knowing:

  • Default sort. An Experiment stores a default_sort (a sort-param string such as -evaluators.solved.mean) so its leaderboard opens ordered by the metric the team cares about. It defaults to -created_at (newest first).
  • Pinning. Any Evaluation can be pinned to the top of its Experiment — for example, the current baseline. Pins are workspace-shared: everyone with access sees the same pinned set, regardless of the active sort.

Get Started

Setup

Experiments are part of the intake service, so any running NeMo Platform already serves them. Point at whatever you have, whether a deployed platform or a local one from nemo setup or nemo quickstart up:

$export NMP_BASE_URL=http://127.0.0.1:8080
$export WORKSPACE=default

ClickHouse must be reachable either way, since the leaderboard rollups are computed from it at read time.

If you are bringing the pieces up yourself from a repository checkout, start them in this order.

  1. ClickHouse (required for the rollups):

    $services/intake/scripts/spans/run_clickhouse.sh

  2. Backend services: intake plus its auth and entities dependencies. --port defaults to 8080; drop uv run if you installed the nemo CLI:

    $uv run nemo services run --services auth,entities,intake --host 127.0.0.1 --port 8080

  3. Studio (optional, for the leaderboard and drill-down UI). From the web/ workspace, with the Experiments feature flag on and intake enabled for trace drill-down:

    $VITE_FF_EXPERIMENT=true VITE_FF_INTAKE_ENABLED=true VITE_PLATFORM_BASE_URL=http://127.0.0.1:8080 \
    >  pnpm --filter nemo-studio-ui start -- --host 127.0.0.1

Confirm the Experiments read path is reachable:

$curl -i "$NMP_BASE_URL/apis/intake/v2/workspaces/$WORKSPACE/experiments?page=1&page_size=1"

A 200 with a (possibly empty) list is healthy. A 503 means the service is running but cannot reach ClickHouse; reads that need metric rollups will fail until it recovers.

First Workflow

Create an Experiment, add an Evaluation, send it telemetry, then see it in Studio and through the API.

1. Create the Experiment (the leaderboard container):

$curl -X POST "$NMP_BASE_URL/apis/intake/v2/workspaces/$WORKSPACE/experiments" \
>  -H "Content-Type: application/json" \
>  -d '{
>        "name": "reranker-prompt-iteration",
>        "description": "Iterating on the support-bench RAG agent'\''s reranker and system prompt."
>      }'

Capture its id for the next step:

$export EXPERIMENT_ID=$(curl -sf \
>  "$NMP_BASE_URL/apis/intake/v2/workspaces/$WORKSPACE/experiments/reranker-prompt-iteration" \
>  | python3 -c 'import sys,json; print(json.load(sys.stdin)["id"])')
$echo "experiment id: $EXPERIMENT_ID"

2. Add an Evaluation to it. experiment_ids is a list, so an Evaluation belongs to one or more existing Experiments:

$curl -X POST "$NMP_BASE_URL/apis/intake/v2/workspaces/$WORKSPACE/evaluations" \
>  -H "Content-Type: application/json" \
>  -d '{
>        "name": "reranker-add-cross-encoder",
>        "experiment_ids": ["'"$EXPERIMENT_ID"'"],
>        "dataset_name": "support-bench",
>        "dataset_version": "v3",
>        "metadata": { "reranker": "cross-encoder" }
>      }'

An Evaluation is a durable record on its own; its leaderboard metrics appear once evaluation telemetry for it lands in Intake.

3. Send it telemetry. Send the run’s telemetry to Intake just as you would for any agent run, whether it comes from the optimizer, a benchmark framework like Harbor, or your own code. Tag each session with the Evaluation’s identity:

  • For ATIF and chat-completions, add a top-level evaluation_context object to the ingest payload carrying evaluation_id (the Evaluation’s name) and test_case_id.
  • For OTLP, set the nemo.experiment.id and nemo.test_case.id root-span attributes.

The per-evaluator scores on the leaderboard come from evaluator results captured on those sessions, either automatically from ATIF verifier rewards or explicitly through the evaluator-results endpoint. See Capture Evaluator Results for both paths, and Observe Agents for the ingestion paths themselves. The nemo-experiments-upload skill walks this through end to end.

test_case_id is required for a populated leaderboard. A session tagged with only evaluation_id still ingests and appears in the Evaluation’s session list, but it doesn’t count toward test_case_count or any rollup — so the row reads as all zeros, with tokens, model, and agent blank too. Always send test_case_id alongside evaluation_id.

Confirm It Worked

List the Experiment’s Evaluations and confirm the new row is present:

$curl -g "$NMP_BASE_URL/apis/intake/v2/workspaces/$WORKSPACE/evaluations?filter[experiment_id]=$EXPERIMENT_ID&page=1&page_size=20"

Then open Studio and navigate to /workspaces/default/experiment, then open reranker-prompt-iteration.

What good looks like: the Experiment opens in Studio, your Evaluation appears as a row, and once its sessions are ingested the row shows non-zero test_case_count, cost, latency, tokens, and evaluator scores. Sorting by a metric reorders the table, and opening the Evaluation lists its individual test cases.

Common Workflows

Add an Evaluation to an Existing Experiment

This is the everyday operation: an Experiment already exists and you want to record another run in it. Create the Evaluation with the Experiment’s id in experiment_ids:

$curl -X POST "$NMP_BASE_URL/apis/intake/v2/workspaces/$WORKSPACE/evaluations" \
>  -H "Content-Type: application/json" \
>  -d '{
>        "name": "reranker-bge-large",
>        "experiment_ids": ["'"$EXPERIMENT_ID"'"],
>        "dataset_name": "support-bench",
>        "dataset_version": "v3",
>        "metadata": { "reranker": "bge-large" }
>      }'

To move or re-scope an existing Evaluation’s membership, PATCH its experiment_ids, which must stay non-empty:

$curl -X PATCH "$NMP_BASE_URL/apis/intake/v2/workspaces/$WORKSPACE/evaluations/reranker-bge-large" \
>  -H "Content-Type: application/json" \
>  -d '{ "experiment_ids": ["'"$EXPERIMENT_ID"'", "'"$OTHER_EXPERIMENT_ID"'"] }'

An Evaluation can belong to several Experiments at once, which is useful when the same run should appear on both a per-project board and a cross-project benchmark.

Update an Evaluation

Use PATCH for partial updates: only the fields you send change. name, dataset_name, and dataset_version are immutable:

$curl -X PATCH "$NMP_BASE_URL/apis/intake/v2/workspaces/$WORKSPACE/evaluations/reranker-bge-large" \
>  -H "Content-Type: application/json" \
>  -d '{ "status": "winner", "root_cause": "Best groundedness at acceptable cost." }'

PUT does a full replace of the mutable fields — omitted fields reset — so prefer PATCH for one-off edits.

Rank an Experiment by the Metric That Matters

List an Experiment’s Evaluations sorted by a metric. Prefix the field with - for descending:

$curl -g "$NMP_BASE_URL/apis/intake/v2/workspaces/$WORKSPACE/evaluations?filter[experiment_id]=$EXPERIMENT_ID&sort=-evaluators.solved.mean&page=1&page_size=20"

You can sort by an entity column (name, created_at) or any rollup metric (test_case_count, cost_usd.<stat>, latency_ms.<stat>, tokens.<stat>, evaluators.<name>.<stat>). In Studio, click a column header. If the Experiment has a default_sort, the leaderboard opens already ordered that way.

Filter to the Evaluations You Care About

Filter by a metric range to narrow the leaderboard — for example, only Evaluations whose average cost is under $0.50:

$curl -g "$NMP_BASE_URL/apis/intake/v2/workspaces/$WORKSPACE/evaluations?filter[experiment_id]=$EXPERIMENT_ID&filter[cost_usd.mean][\$lte]=0.5"

Metric filters use the same grammar as sort, such as filter[test_case_count][\$gte]=5 or filter[evaluators.groundedness.mean][\$gte]=0.8. You can also filter by filter[metadata.<key>]=<value>, filter[status]=…, filter[is_pinned]=true, and created_at/updated_at ranges. In Studio, use the column filters on cost, latency, test-case count, and evaluator columns.

Set an Experiment’s Default Sort

Store the ordering the team should see first so no one has to re-sort each visit:

$curl -X PUT "$NMP_BASE_URL/apis/intake/v2/workspaces/$WORKSPACE/experiments/reranker-prompt-iteration" \
>  -H "Content-Type: application/json" \
>  -d '{ "name": "reranker-prompt-iteration", "default_sort": "-evaluators.solved.mean" }'

In Studio, set it from the Experiment’s Edit dialog.

Pin a Baseline to the Top

Keep the current baseline, or any reference run, at the top of the Experiment for everyone:

$curl -X POST   "$NMP_BASE_URL/apis/intake/v2/workspaces/$WORKSPACE/evaluations/reranker-main-baseline/pin"
$
$# Unpin:
$curl -X DELETE "$NMP_BASE_URL/apis/intake/v2/workspaces/$WORKSPACE/evaluations/reranker-main-baseline/pin"

Pinned Evaluations float to the top of the leaderboard regardless of the active sort. List just the pinned set with filter[is_pinned]=true.

Drill into an Evaluation’s Test Cases

Open one Evaluation’s sessions to see per-test-case behavior: status, latency, cost, and evaluator scores:

$curl "$NMP_BASE_URL/apis/intake/v2/workspaces/$WORKSPACE/evaluations/reranker-add-cross-encoder/sessions?page=1&page_size=20"

Use this to move from “this candidate scores lower” to “these specific cases regressed.” In Studio, open the Evaluation row to reach its sessions, then follow a session into its trace in Intake.

Configure the Pareto View

Each Experiment stores a pareto config: the default X/Y metrics for its Pareto (trade-off) chart in Studio, defaulting to cost vs. latency. The axes are x_metric and y_metric, and each takes a base metric id — cost_usd, latency_ms, or evaluators.<name>, with no .<stat> suffix. Set it on create or update:

$curl -X PUT "$NMP_BASE_URL/apis/intake/v2/workspaces/$WORKSPACE/experiments/reranker-prompt-iteration" \
>  -H "Content-Type: application/json" \
>  -d '{ "name": "reranker-prompt-iteration",
>        "pareto": { "x_metric": "cost_usd", "y_metric": "evaluators.solved" } }'

Delete an Experiment or Evaluation

Deleting an Experiment soft-deletes it and cascades to its Evaluations; deleting an Evaluation soft-deletes just that row. Soft-deleted records are hidden from list and get operations unless explicitly requested with filter[is_deleted]=true:

$curl -X DELETE "$NMP_BASE_URL/apis/intake/v2/workspaces/$WORKSPACE/evaluations/reranker-bge-large"
$curl -X DELETE "$NMP_BASE_URL/apis/intake/v2/workspaces/$WORKSPACE/experiments/reranker-prompt-iteration"

Where Evaluations Come From

You usually don’t create Evaluations by hand. Common producers:

  • Optimizer: records each run as an Experiment and its candidates as Evaluations, so results land automatically. See Optimize Agents.
  • Evaluation framework: a framework like Harbor sends complete trajectories through Intake’s ATIF ingest, and each run becomes an Evaluation with its final metrics as evaluator results. See Observe Agents.
  • Direct API: create the Experiment and Evaluations, then ingest their sessions, for a custom evaluation pipeline.

Do It with an Agent

If you are working with a coding agent, the nemo-experiments-upload skill automates this whole path: create an Experiment, add an Evaluation, log traces and scores to an ingest endpoint, and verify the rollups. It ships with reference guides for the ATIF/Harbor, chat-completions, and OTLP ingest formats and a troubleshooting matrix. Invoke it when you want to upload, log, ingest, or publish evaluation runs to NeMo Experiments.

Operations

Limits

  • An Experiment is sorted and filtered in memory over the full set of its Evaluations, bounded to 1,000 Evaluations per Experiment. A request that would select more is rejected with 413 rather than returned partially sorted. Narrow it with filters.
  • Page size is capped at 1,000 records.
  • Metric sorting and filtering require rollups. If ClickHouse is unavailable, a metric sort or filter returns 503 rather than a silently unsorted result. Sorting and filtering by entity columns (name, created_at) still works.

Security and Access Control

All Experiments endpoints are workspace-scoped under /apis/intake/v2/workspaces/{workspace}/, and the service checks workspace access before reading or writing.

Evaluation metadata, source_link, and related fields are producer-supplied and surfaced to reviewers, so treat them like any other telemetry payload. Do not store secrets, credentials, or regulated data unless your deployment’s policy permits it.

Retention and Storage

Experiment and Evaluation metadata lives in the platform entity store and persists until you delete it; it does not expire on the telemetry TTL. Rollups are derived from ClickHouse at read time, so leaderboard metrics reflect whatever telemetry is still retained: span and trace-index data expire after 90 days, while evaluator results are retained without that span TTL. An Experiment older than the span window keeps its records and evaluator-based scores, but cost, latency, and run-count rollups reflect only the retained telemetry window.

Troubleshooting

SymptomCause and fix
Rows show zero metricsThree causes: no sessions have been ingested for that Evaluation yet; the sessions were ingested without test_case_id, so they don’t count toward test_case_count or the rollups; or ClickHouse is unreachable. Confirm ingestion in Intake, that sessions carry test_case_id, and that the read path returns 200 rather than 503.
A metric sort or filter returns 503Rollups can’t be computed because ClickHouse is down. Retry once the read path is healthy, or fall back to an entity-column sort.
A list returns 413The Experiment selected more than 1,000 Evaluations for an in-memory sort. Add filters to narrow the set.
An Evaluation isn’t in the ExperimentConfirm you created it with the correct Experiment id in experiment_ids and are querying the right workspace (filter[experiment_id]=<experiment-id>).
422 on create EvaluationAn Evaluation must belong to at least one Experiment: provide experiment_ids. Required fields are name, experiment_ids, and dataset_name; metadata values must be strings.
  • Observe Agents: the telemetry these leaderboards are computed from, and how to ingest it.
  • Agent Evaluation: the task-driven evaluation model that produces these runs.
  • Optimize Agents: the optimizer that creates Experiments and Evaluations automatically.