Experiments | NVIDIA NeMo Platform

NeMo Experiments is the comparison layer for agent optimization tasks, such as evaluating the impact of changes to a harness, infrastructure, tools, or agent code. It provides one place to visualize and compare evaluation results from Harbor, the NeMo Optimizer, NeMo Evaluator, or your own runner. You can rank these results by cost, latency, evaluator scores, and other metrics.

Those metrics are derived from NeMo Intake observability data, the same traces and evaluator results your runs already produce, so any runner that lands telemetry in Intake feeds the same comparison.

Before You Start

Who This Is For

NeMo Experiments is for anyone running evaluations on their agent who needs a way to compare, analyze, and promote the results. For example: engineers iterating on prompts, models, tools, or routing; teams benchmarking many models or configurations at once; and reviewers who need a shared leaderboard to decide what to promote.

Requirements

A NeMo Platform environment with the intake service running.
The platform entity store (Postgres).
A reachable ClickHouse database.
A producer that creates Evaluations and sends their telemetry: the NeMo Optimizer, a benchmark or evaluation framework such as Harbor, or a direct application programming interface (API) integration.
Recommended: NeMo Studio for the complete web user interface (UI), including trace comparison views, a customizable leaderboard, and Pareto charts. The API provides the same data. The Experiments UI is gated by the VITE_FF_EXPERIMENT feature flag, which is off by default.

How It Works

What It Does

Experiments computes comparable, always-current metrics for your evaluation runs. For each run it rolls up cost, latency, tokens, and per-evaluator scores from the underlying telemetry in ClickHouse and returns them as ranked rows. Rollups are computed at read time, so a leaderboard always reflects current telemetry, with no denormalized score table to maintain.

You can group runs by an insight to investigate, a model comparison, a benchmark leaderboard, or promoted runs from several groups. A single run can belong to more than one group. Sorting, filtering, and pinning within a group surface the relevant runs.

When to Use It

Use Experiments when you need to:

Compare candidates from the optimizer, a prompt or model change, or a routing strategy against a baseline.
Run a benchmark leaderboard: import many runs into one Experiment and rank them by the metric that matters.
Track iteration over time: keep one Experiment per project so successive attempts stay side by side.
Standardize comparison across a team: everyone reads the same metrics, in the same place, with the same vocabulary.

Experiments are most useful after real evaluation telemetry exists. A new group can start empty and fill in as runs land.

Core Concepts and Data Model

Experiments sit on top of Intake’s telemetry hierarchy (span → trace → session):

An Experiment is a named container of Evaluations. It holds durable metadata (description, summary, free-form metadata, optional insight_id) and view configuration: a default_sort and a pareto (default X/Y metrics for the Pareto view).
An Evaluation is one run and one leaderboard row. It records producer-supplied fields (dataset_name, dataset_version, source_link, metadata, description, status, root_cause), the experiment_ids it belongs to (at least one; an Evaluation can live in more than one Experiment), and an optional parent_evaluation_id linking a variant back to the run it was derived from.
A Session is one test-case execution within an Evaluation, a single ingested run made of spans. Sessions carry per-case status, latency, token and cost totals, and evaluator scores.

At read time, each Evaluation is enriched with rollups derived from its sessions in ClickHouse:

Rollup	Meaning
`test_case_count`	Number of distinct test cases (distinct non-empty `test_case_name` values). Sessions with no `test_case_name` do not count toward it or the rollups.
`cost_usd`	Cost aggregate across the Evaluation’s sessions.
`latency_ms`	Latency aggregate across the Evaluation’s sessions.
`tokens`	Average total tokens (input + output) per test case.
`evaluators.<name>`	Aggregate of a named evaluator’s session scores.
`model_names`, `agent_names`, `agent_versions`	Distinct models, agents, and versions observed in the telemetry.

Metric aggregates expose these statistics: sum, mean, median, p90, p95, p99, count. A metric path is therefore test_case_count, cost_usd.<stat>, latency_ms.<stat>, tokens.<stat>, or evaluators.<name>.<stat>. Sorting and filtering use the same grammar.

Two things about evaluator rollups specifically:

Response shape. evaluators.<name>.<stat> is the sort/filter query grammar. The Evaluation JSON response returns the same scores under aggregate_scores, a map keyed by evaluator name. Each value contains the statistics above, and an evaluator_names list appears alongside it. There is no evaluators field in the response; evaluators.<name>.<stat> (query) reads aggregate_scores[<name>].<stat> (response).
Missing-value semantics. Evaluator rollups are test-case-weighted: each stat is computed over test_case_count, the full set of test cases, with a test case that did not report a given evaluator counted as 0 rather than dropped.

Two Experiment-level behaviors are worth knowing:

Default sort. An Experiment stores a default_sort (a sort-param string such as -evaluators.solved.mean) so its leaderboard opens ordered by the metric the team cares about. It defaults to -created_at (newest first).
Pinning. Any Evaluation can be pinned to the top of its Experiment, such as the current baseline. Pins are workspace-shared: everyone with access sees the same pinned set, regardless of the active sort.

Get Started

Setup

Experiments are part of the intake service, so any running NeMo Platform already serves them. Point at whatever you have, whether a deployed platform or a local one from nemo setup or nemo quickstart up:

$ export NMP_BASE_URL=http://127.0.0.1:8080
$ export WORKSPACE=default
$ export NMP_STUDIO_URL="$NMP_BASE_URL/studio"
$ export NMP_ACCESS_TOKEN="$(nemo auth token)"

Set NMP_STUDIO_URL explicitly if Studio uses a different origin or path. The examples use bearer-token authentication and stop on HTTP errors. For a local development deployment with authentication disabled, omit the Authorization header and the NMP_ACCESS_TOKEN command.

ClickHouse must be reachable either way, since the leaderboard rollups are computed from it at read time.

Running from source (local development)

If you are bringing the pieces up yourself from a repository checkout, make sure Docker Desktop or the Docker daemon is running. Intake automatically provisions and reuses a local ClickHouse container for the resolved NeMo data directory unless NMP_INTAKE_CLICKHOUSE_URL explicitly points to an external instance.

Backend services: intake plus its auth and entities dependencies. --port defaults to 8080; omit uv run if you installed the nemo command-line interface (CLI):
```
$ uv run nemo services run --services auth,entities,intake --host 127.0.0.1 --port 8080
```

Studio (optional, for the leaderboard and drill-down UI). From the web/ workspace, with the Experiments feature flag on and intake enabled for trace drill-down:

$ VITE_FF_EXPERIMENT=true VITE_FF_INTAKE_ENABLED=true VITE_PLATFORM_BASE_URL=http://127.0.0.1:8080 \
>   pnpm --filter nemo-studio-ui start -- --host 127.0.0.1

Confirm that the entity read path and ClickHouse rollups are reachable:

$ curl --fail-with-body -i \
>   -H "Authorization: Bearer $NMP_ACCESS_TOKEN" \
>   "$NMP_BASE_URL/apis/intake/v2/workspaces/$WORKSPACE/experiments?sort=-created_at&page=1&page_size=1"
$ curl --fail-with-body -i \
>   -H "Authorization: Bearer $NMP_ACCESS_TOKEN" \
>   "$NMP_BASE_URL/apis/intake/v2/workspaces/$WORKSPACE/evaluations?sort=-cost_usd.mean&page=1&page_size=1"

The first request checks the entity store and can return 200 while ClickHouse is unavailable. The second request explicitly sorts by a rollup metric. A 200 response from both requests is healthy. A 503 response from the second request means that Intake cannot compute ClickHouse rollups.

First Workflow

Create an Experiment, add an Evaluation, send it telemetry, then see it in Studio and through the API.

1. Create the Experiment (the leaderboard container):

$ curl --fail-with-body -X POST \
>   "$NMP_BASE_URL/apis/intake/v2/workspaces/$WORKSPACE/experiments" \
>   -H "Authorization: Bearer $NMP_ACCESS_TOKEN" \
>   -H "Content-Type: application/json" \
>   -d '{
>         "name": "reranker-prompt-iteration",
>         "description": "Iterating on the support-bench retrieval-augmented generation agent'\''s reranker and system prompt."
>       }'

Capture its id for the next step:

$ export EXPERIMENT_ID=$(curl --fail-with-body --silent \
>   -H "Authorization: Bearer $NMP_ACCESS_TOKEN" \
>   "$NMP_BASE_URL/apis/intake/v2/workspaces/$WORKSPACE/experiments/reranker-prompt-iteration" \
>   | python3 -c 'import sys,json; print(json.load(sys.stdin)["id"])')
$ echo "experiment id: $EXPERIMENT_ID"

2. Add an Evaluation to it. experiment_ids is a list, so an Evaluation belongs to one or more existing Experiments:

$ curl --fail-with-body -X POST \
>   "$NMP_BASE_URL/apis/intake/v2/workspaces/$WORKSPACE/evaluations" \
>   -H "Authorization: Bearer $NMP_ACCESS_TOKEN" \
>   -H "Content-Type: application/json" \
>   -d '{
>         "name": "reranker-add-cross-encoder",
>         "experiment_ids": ["'"$EXPERIMENT_ID"'"],
>         "dataset_name": "support-bench",
>         "dataset_version": "v3",
>         "metadata": { "reranker": "cross-encoder" }
>       }'

An Evaluation is a durable record. Its leaderboard metrics appear after its evaluation telemetry lands in Intake.

3. Send it telemetry. Send the run’s telemetry to Intake in the same way as other agent runs, whether it comes from the optimizer, a benchmark framework like Harbor, or your own code. Tag each session with the Evaluation’s identity:

For Agent Trajectory Interchange Format (ATIF) and chat-completions, add a top-level evaluation_context object to the ingest payload carrying evaluation_name and test_case_name.
For OpenTelemetry Protocol (OTLP), set nemo.evaluation.name and nemo.test_case.name on each trace’s root span.

The per-evaluator scores on the leaderboard come from evaluator results captured on those sessions. Intake can derive them from ATIF verifier rewards. OTLP traces carry session telemetry, so the framework or caller must publish their scores through the evaluator-results endpoint. Refer to Capture Evaluator Results for both paths, and Observe Agents for the ingestion paths themselves. The nemo-experiments-upload skill walks this through end to end.

test_case_name is required for a populated leaderboard. A session tagged with only evaluation_name still ingests and appears in the Evaluation’s session list, but it does not count toward test_case_count or any rollup. The row therefore reads as all zeros, with tokens, model, and agent blank too. Always send test_case_name alongside evaluation_name.

Confirm It Worked

List the Experiment’s Evaluations and confirm the new row is present:

$ curl -g --fail-with-body \
>   -H "Authorization: Bearer $NMP_ACCESS_TOKEN" \
>   "$NMP_BASE_URL/apis/intake/v2/workspaces/$WORKSPACE/evaluations?filter[experiment_id]=$EXPERIMENT_ID&page=1&page_size=20"

Then open $NMP_STUDIO_URL/workspaces/$WORKSPACE/experiment, then open reranker-prompt-iteration.

A successful result opens the Experiment in Studio and shows your Evaluation as a row. After its sessions are ingested the row shows non-zero test_case_count, cost, latency, tokens, and evaluator scores. Sorting by a metric reorders the table, and opening the Evaluation lists its individual test cases.

Common Workflows

Add an Evaluation to an Existing Experiment

This is the everyday operation: an Experiment already exists and you want to record another run in it. Create the Evaluation with the Experiment’s id in experiment_ids:

$ curl --fail-with-body -X POST \
>   "$NMP_BASE_URL/apis/intake/v2/workspaces/$WORKSPACE/evaluations" \
>   -H "Authorization: Bearer $NMP_ACCESS_TOKEN" \
>   -H "Content-Type: application/json" \
>   -d '{
>         "name": "reranker-bge-large",
>         "experiment_ids": ["'"$EXPERIMENT_ID"'"],
>         "dataset_name": "support-bench",
>         "dataset_version": "v3",
>         "metadata": { "reranker": "bge-large" }
>       }'

Associate an Evaluation with Another Experiment

An Evaluation can belong to several Experiments simultaneously, which is useful when the same run should appear on both a per-project board and a cross-project benchmark. To demonstrate adding a second membership, first create the second Experiment and capture its ID:

$ curl --fail-with-body -X POST \
>   "$NMP_BASE_URL/apis/intake/v2/workspaces/$WORKSPACE/experiments" \
>   -H "Authorization: Bearer $NMP_ACCESS_TOKEN" \
>   -H "Content-Type: application/json" \
>   -d '{
>         "name": "reranker-cross-project-benchmark",
>         "description": "Compare reranker candidates across projects."
>       }'
$ 
$ export OTHER_EXPERIMENT_ID=$(curl --fail-with-body --silent \
>   -H "Authorization: Bearer $NMP_ACCESS_TOKEN" \
>   "$NMP_BASE_URL/apis/intake/v2/workspaces/$WORKSPACE/experiments/reranker-cross-project-benchmark" \
>   | python3 -c 'import sys,json; print(json.load(sys.stdin)["id"])')

Then PATCH the existing Evaluation’s experiment_ids with both IDs. The list must stay non-empty:

$ curl --fail-with-body -X PATCH \
>   "$NMP_BASE_URL/apis/intake/v2/workspaces/$WORKSPACE/evaluations/reranker-bge-large" \
>   -H "Authorization: Bearer $NMP_ACCESS_TOKEN" \
>   -H "Content-Type: application/json" \
>   -d '{ "experiment_ids": ["'"$EXPERIMENT_ID"'", "'"$OTHER_EXPERIMENT_ID"'"] }'

Update an Evaluation

Use PATCH for partial updates: only the fields you send change. name, dataset_name, and dataset_version are immutable:

$ curl --fail-with-body -X PATCH \
>   "$NMP_BASE_URL/apis/intake/v2/workspaces/$WORKSPACE/evaluations/reranker-bge-large" \
>   -H "Authorization: Bearer $NMP_ACCESS_TOKEN" \
>   -H "Content-Type: application/json" \
>   -d '{ "status": "winner", "root_cause": "Best groundedness at acceptable cost." }'

PUT does a full replacement of the mutable fields. Omitted fields reset, so prefer PATCH for individual edits.

Rank an Experiment by the Metric That Matters

List an Experiment’s Evaluations sorted by a metric. Prefix the field with - for descending:

$ curl -g --fail-with-body \
>   -H "Authorization: Bearer $NMP_ACCESS_TOKEN" \
>   "$NMP_BASE_URL/apis/intake/v2/workspaces/$WORKSPACE/evaluations?filter[experiment_id]=$EXPERIMENT_ID&sort=-evaluators.solved.mean&page=1&page_size=20"

You can sort by an entity column (name, created_at) or any rollup metric (test_case_count, cost_usd.<stat>, latency_ms.<stat>, tokens.<stat>, evaluators.<name>.<stat>). In Studio, click a column header. If the Experiment has a default_sort, the leaderboard opens already ordered that way.

Filter to the Evaluations You Care About

Filter by a metric range to narrow the leaderboard. For example, select only Evaluations whose average cost is under $0.50:

$ curl -g --fail-with-body \
>   -H "Authorization: Bearer $NMP_ACCESS_TOKEN" \
>   "$NMP_BASE_URL/apis/intake/v2/workspaces/$WORKSPACE/evaluations?filter[experiment_id]=$EXPERIMENT_ID&filter[cost_usd.mean][\$lte]=0.5"

Metric filters use the same grammar as sort, such as filter[test_case_count][\$gte]=5 or filter[evaluators.groundedness.mean][\$gte]=0.8. You can also filter by filter[metadata.<key>]=<value>, filter[status]=<value>, filter[is_pinned]=true, and created_at/updated_at ranges. In Studio, use the column filters on cost, latency, test-case count, and evaluator columns.

Set an Experiment’s Default Sort

Store the ordering the team should see first so no one has to re-sort each visit:

$ curl --fail-with-body -X PUT \
>   "$NMP_BASE_URL/apis/intake/v2/workspaces/$WORKSPACE/experiments/reranker-prompt-iteration" \
>   -H "Authorization: Bearer $NMP_ACCESS_TOKEN" \
>   -H "Content-Type: application/json" \
>   -d '{ "name": "reranker-prompt-iteration", "default_sort": "-evaluators.solved.mean" }'

In Studio, set it from the Experiment’s Edit dialog.

Pin a Baseline to the Top

Keep the current baseline, or any reference run, at the top of the Experiment for everyone:

$ curl --fail-with-body -X POST \
>   "$NMP_BASE_URL/apis/intake/v2/workspaces/$WORKSPACE/evaluations" \
>   -H "Authorization: Bearer $NMP_ACCESS_TOKEN" \
>   -H "Content-Type: application/json" \
>   -d '{
>         "name": "reranker-main-baseline",
>         "experiment_ids": ["'"$EXPERIMENT_ID"'"],
>         "dataset_name": "support-bench",
>         "dataset_version": "v3",
>         "metadata": { "reranker": "main-baseline" }
>       }'
$ 
$ curl --fail-with-body -X POST \
>   -H "Authorization: Bearer $NMP_ACCESS_TOKEN" \
>   "$NMP_BASE_URL/apis/intake/v2/workspaces/$WORKSPACE/evaluations/reranker-main-baseline/pin"
$ 
$ # Unpin:
$ curl --fail-with-body -X DELETE \
>   -H "Authorization: Bearer $NMP_ACCESS_TOKEN" \
>   "$NMP_BASE_URL/apis/intake/v2/workspaces/$WORKSPACE/evaluations/reranker-main-baseline/pin"

Pinned Evaluations float to the top of the leaderboard regardless of the active sort. List only the pinned set with filter[is_pinned]=true.

Drill into an Evaluation’s Test Cases

Open one Evaluation’s sessions to see per-test-case behavior: status, latency, cost, and evaluator scores:

$ curl --fail-with-body \
>   -H "Authorization: Bearer $NMP_ACCESS_TOKEN" \
>   "$NMP_BASE_URL/apis/intake/v2/workspaces/$WORKSPACE/evaluations/reranker-add-cross-encoder/sessions?page=1&page_size=20"

Use this to move from “this candidate scores lower” to “these specific cases regressed.” In Studio, open the Evaluation row to reach its sessions, then follow a session into its trace in Intake.

Configure the Pareto View

Each Experiment stores a pareto configuration: the default X/Y metrics for its Pareto chart in Studio, defaulting to cost vs. latency. The axes are x_metric and y_metric, and each takes a base metric ID: cost_usd, latency_ms, or evaluators.<name>, with no .<stat> suffix. Set it on create or update:

$ curl --fail-with-body -X PUT \
>   "$NMP_BASE_URL/apis/intake/v2/workspaces/$WORKSPACE/experiments/reranker-prompt-iteration" \
>   -H "Authorization: Bearer $NMP_ACCESS_TOKEN" \
>   -H "Content-Type: application/json" \
>   -d '{ "name": "reranker-prompt-iteration",
>         "pareto": { "x_metric": "cost_usd", "y_metric": "evaluators.solved" } }'

Delete an Experiment or Evaluation

Deleting an Experiment soft-deletes it. An Evaluation that belongs only to that Experiment is also soft-deleted. A shared Evaluation remains active, and the deleted Experiment is removed from its experiment_ids membership. Deleting an Evaluation directly soft-deletes only that Evaluation.

$ curl --fail-with-body -X DELETE \
>   -H "Authorization: Bearer $NMP_ACCESS_TOKEN" \
>   "$NMP_BASE_URL/apis/intake/v2/workspaces/$WORKSPACE/evaluations/reranker-bge-large"
$ curl --fail-with-body -X DELETE \
>   -H "Authorization: Bearer $NMP_ACCESS_TOKEN" \
>   "$NMP_BASE_URL/apis/intake/v2/workspaces/$WORKSPACE/experiments/reranker-cross-project-benchmark"
$ curl --fail-with-body -X DELETE \
>   -H "Authorization: Bearer $NMP_ACCESS_TOKEN" \
>   "$NMP_BASE_URL/apis/intake/v2/workspaces/$WORKSPACE/experiments/reranker-prompt-iteration"

Soft deletion renames the stored record. A GET request that uses the original name does not return the deleted record. To audit deleted records, use filter[is_deleted]=true on a list operation:

$ curl -g --fail-with-body \
>   -H "Authorization: Bearer $NMP_ACCESS_TOKEN" \
>   "$NMP_BASE_URL/apis/intake/v2/workspaces/$WORKSPACE/evaluations?filter[is_deleted]=true&page=1&page_size=20"

Where Evaluations Come From

You typically do not create Evaluations manually. Common producers include:

Optimizer: records each run as an Experiment and its candidates as Evaluations, so results land automatically. Refer to Optimize Agents.
Evaluation framework: a framework like Harbor sends trajectories through Intake’s ATIF or OTLP ingest, publishes its scores as evaluator results, and records each run as an Evaluation. Refer to Observe Agents.
Direct API: create the Experiment and Evaluations, then ingest their sessions, for a custom evaluation pipeline.

Do It with an Agent

If you are working with a coding agent, the nemo-experiments-upload skill automates this whole path: create an Experiment, add an Evaluation, log traces and scores to an ingest endpoint, and verify the rollups. It ships with reference guides for the ATIF/Harbor, chat-completions, and OTLP ingest formats and a troubleshooting matrix. Invoke it when you want to upload, log, ingest, or publish evaluation runs to NeMo Experiments.

Operations

Limits

An Experiment is sorted and filtered in memory over the full set of its Evaluations, bounded to 1,000 Evaluations per Experiment. A request that would select more is rejected with 413 rather than returned partially sorted. Narrow it with filters.
Page size is capped at 1,000 records.
Metric sorting and filtering require rollups. If ClickHouse is unavailable, a metric sort or filter returns 503 rather than a silently unsorted result. Sorting and filtering by entity columns (name, created_at) still works.

Security and Access Control

All Experiments endpoints are workspace-scoped under /apis/intake/v2/workspaces/{workspace}/, and the service checks workspace access before reading or writing.

Evaluation metadata, source_link, and related fields are producer-supplied and surfaced to reviewers, so treat them like any other telemetry payload. Do not store secrets, credentials, or regulated data unless your deployment’s policy permits it.

Retention and Storage

Experiment and Evaluation metadata lives in the platform entity store and persists until you delete it. This metadata does not expire with telemetry. Span and trace-index records in ClickHouse expire after 90 days. Evaluator-result rows do not use that 90-day time-to-live (TTL) setting.

All leaderboard rollups depend on the retained trace index. Evaluator score rollups join the retained trace index to evaluator-result rows at read time. After a trace-index entry expires, its evaluator scores no longer appear on the leaderboard even if the evaluator-result rows remain. Cost, latency, token, test-case count, and evaluator score rollups therefore reflect the retained 90-day trace window.

Troubleshooting

Symptom	Cause and fix
Rows show zero metrics	Three causes: no sessions have been ingested for that Evaluation yet; the sessions were ingested without `test_case_name`, so they do not count toward `test_case_count` or the rollups; or ClickHouse is unreachable. Confirm ingestion in Intake, confirm that sessions carry `test_case_name`, and run the explicit metric-sort health check.
A metric sort or filter returns `503`	Rollups cannot be computed because ClickHouse is down. Retry after the read path is healthy, or use an entity-column sort.
A list returns `413`	The Experiment selected more than 1,000 Evaluations for an in-memory sort. Add filters to narrow the set.
An Evaluation is not in the Experiment	Confirm that you created it with the correct Experiment `id` in `experiment_ids` and that you are querying the correct workspace (`filter[experiment_id]=<experiment-id>`).
`422` on create Evaluation	An Evaluation must belong to at least one Experiment: provide `experiment_ids`. Required fields are `name`, `experiment_ids`, and `dataset_name`; `metadata` values must be strings.

Observe Agents: the telemetry these leaderboards are computed from, and how to ingest it.
Agent Evaluation: the task-driven evaluation model that produces these runs.
Optimize Agents: the optimizer that creates Experiments and Evaluations automatically.