> For clean Markdown of any page, append .md to the page URL. > For a complete documentation index, see https://docs.nvidia.com/nemo-platform/llms.txt. > For AI client integration (Claude Code, Cursor, etc.), connect to the MCP server at https://docs.nvidia.com/nemo-platform/_mcp/server. # Experiments > Compare evaluation runs on one leaderboard with NeMo Experiments — cost, latency, token, and evaluator rollups computed at read time from NeMo Intake telemetry. NeMo Experiments is the comparison layer for agent optimization tasks, such as evaluating the impact of changes to a harness, infrastructure, tools, or agent code. It gives you one place to visualize and compare evaluation results from the runner of your choice — Harbor, the NeMo Optimizer, NeMo Evaluator, or your own — ranked on the metrics that matter, such as cost, latency, and evaluator scores. Those metrics are derived from [NeMo Intake](/documentation/agents/observe-agents) observability data, the same traces and evaluator results your runs already produce, so any runner that lands telemetry in Intake feeds the same comparison. ## Before You Start ### Who This Is For NeMo Experiments is for anyone running evaluations on their agent who needs a way to compare, analyze, and promote the results. For example: engineers iterating on prompts, models, tools, or routing; teams benchmarking many models or configurations at once; and reviewers who need a shared leaderboard to decide what to promote. ### Requirements * A NeMo Platform environment with the `intake` service running. * The platform entity store (Postgres). * A reachable ClickHouse database. * A producer that creates Evaluations and sends their telemetry: the NeMo Optimizer, a benchmark or evaluation framework such as Harbor, or a direct API integration. * (Recommended) [NeMo Studio](/documentation/studio) for the full UI — trace comparison views, a customizable leaderboard, and Pareto charts. Everything is also available through the API, but Studio is where the experience really lives. The Experiments UI is gated by the `VITE_FF_EXPERIMENT` feature flag, which is off by default. ## How It Works ### What It Does Experiments computes comparable, always-current metrics for your evaluation runs. For each run it rolls up cost, latency, tokens, and per-evaluator scores from the underlying telemetry in ClickHouse and returns them as ranked rows. Rollups are computed at read time, so a leaderboard always reflects current telemetry, with no denormalized score table to maintain. You group runs however makes sense — an insight to investigate, a model bake-off, a benchmark leaderboard, or the top runs promoted from several groups — and a single run can belong to more than one group. Sorting, filtering, and pinning within a group surface the runs that matter. ### When to Use It Use Experiments when you need to: * **Compare candidates** from the optimizer, a prompt or model change, or a routing strategy against a baseline. * **Run a benchmark leaderboard**: import many runs into one Experiment and rank them by the metric that matters. * **Track iteration over time**: keep one Experiment per project so successive attempts stay side by side. * **Standardize comparison across a team**: everyone reads the same metrics, in the same place, with the same vocabulary. Experiments are most useful once real evaluation telemetry exists. A new group can start empty and fill in as runs land. ### Core Concepts and Data Model Experiments sit on top of Intake's telemetry hierarchy (span → trace → session): * An **Experiment** is a named container of Evaluations. It holds durable metadata (`description`, `summary`, free-form `metadata`, optional `insight_id`) and view configuration: a `default_sort` and a `pareto` (default X/Y metrics for the Pareto view). * An **Evaluation** is one run and one leaderboard row. It records producer-supplied fields (`dataset_name`, `dataset_version`, `source_link`, `metadata`, `description`, `status`, `root_cause`), the `experiment_ids` it belongs to (at least one — an Evaluation can live in more than one Experiment), and an optional `parent_evaluation_id` linking a variant back to the run it was derived from. * A **Session** is one test-case execution within an Evaluation, a single ingested run made of spans. Sessions carry per-case status, latency, token and cost totals, and evaluator scores. At read time, each Evaluation is enriched with rollups derived from its sessions in ClickHouse: | Rollup | Meaning | | ---------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------- | | `test_case_count` | Number of distinct test cases (distinct non-empty `test_case_id` values). Sessions with no `test_case_id` don't count toward it or the rollups. | | `cost_usd` | Cost aggregate across the Evaluation's sessions. | | `latency_ms` | Latency aggregate across the Evaluation's sessions. | | `tokens` | Average total tokens (input + output) per test case. | | `evaluators.` | Aggregate of a named evaluator's session scores. | | `model_names`, `agent_names`, `agent_versions` | Distinct models, agents, and versions observed in the telemetry. | Metric aggregates expose these statistics: `sum`, `mean`, `median`, `p90`, `p95`, `p99`, `count`. A **metric path** is therefore `test_case_count`, `cost_usd.`, `latency_ms.`, `tokens.`, or `evaluators..` — the same grammar used for sorting and filtering below. Two things about evaluator rollups specifically: * **Response shape.** `evaluators..` is the sort/filter *query* grammar. In the Evaluation JSON the same scores come back under `aggregate_scores` — a map keyed by evaluator name, each value carrying the stats above — alongside an `evaluator_names` list. There is no `evaluators` field in the response; `evaluators..` (query) reads `aggregate_scores[].` (response). * **Missing-value semantics.** Evaluator rollups are test-case-weighted: each stat is computed over `test_case_count`, the full set of test cases, with a test case that didn't report a given evaluator counted as `0` rather than dropped. Two Experiment-level behaviors are worth knowing: * **Default sort.** An Experiment stores a `default_sort` (a sort-param string such as `-evaluators.solved.mean`) so its leaderboard opens ordered by the metric the team cares about. It defaults to `-created_at` (newest first). * **Pinning.** Any Evaluation can be pinned to the top of its Experiment — for example, the current baseline. Pins are workspace-shared: everyone with access sees the same pinned set, regardless of the active sort. ## Get Started ### Setup Experiments are part of the `intake` service, so any running NeMo Platform already serves them. Point at whatever you have, whether a deployed platform or a local one from `nemo setup` or `nemo quickstart up`: ```shell export NMP_BASE_URL=http://127.0.0.1:8080 export WORKSPACE=default ``` ClickHouse must be reachable either way, since the leaderboard rollups are computed from it at read time. #### Running from source (local development) If you are bringing the pieces up yourself from a repository checkout, start them in this order. 1. **ClickHouse** (required for the rollups): ```shell services/intake/scripts/spans/run_clickhouse.sh ``` 2. **Backend services**: `intake` plus its `auth` and `entities` dependencies. `--port` defaults to `8080`; drop `uv run` if you installed the `nemo` CLI: ```shell uv run nemo services run --services auth,entities,intake --host 127.0.0.1 --port 8080 ``` 3. **Studio** (optional, for the leaderboard and drill-down UI). From the `web/` workspace, with the Experiments feature flag on and intake enabled for trace drill-down: ```shell VITE_FF_EXPERIMENT=true VITE_FF_INTAKE_ENABLED=true VITE_PLATFORM_BASE_URL=http://127.0.0.1:8080 \ pnpm --filter nemo-studio-ui start -- --host 127.0.0.1 ``` Confirm the Experiments read path is reachable: ```shell curl -i "$NMP_BASE_URL/apis/intake/v2/workspaces/$WORKSPACE/experiments?page=1&page_size=1" ``` A `200` with a (possibly empty) list is healthy. A `503` means the service is running but cannot reach ClickHouse; reads that need metric rollups will fail until it recovers. ### First Workflow Create an Experiment, add an Evaluation, send it telemetry, then see it in Studio and through the API. **1. Create the Experiment** (the leaderboard container): ```shell curl -X POST "$NMP_BASE_URL/apis/intake/v2/workspaces/$WORKSPACE/experiments" \ -H "Content-Type: application/json" \ -d '{ "name": "reranker-prompt-iteration", "description": "Iterating on the support-bench RAG agent'\''s reranker and system prompt." }' ``` Capture its `id` for the next step: ```shell export EXPERIMENT_ID=$(curl -sf \ "$NMP_BASE_URL/apis/intake/v2/workspaces/$WORKSPACE/experiments/reranker-prompt-iteration" \ | python3 -c 'import sys,json; print(json.load(sys.stdin)["id"])') echo "experiment id: $EXPERIMENT_ID" ``` **2. Add an Evaluation** to it. `experiment_ids` is a list, so an Evaluation belongs to one or more existing Experiments: ```shell curl -X POST "$NMP_BASE_URL/apis/intake/v2/workspaces/$WORKSPACE/evaluations" \ -H "Content-Type: application/json" \ -d '{ "name": "reranker-add-cross-encoder", "experiment_ids": ["'"$EXPERIMENT_ID"'"], "dataset_name": "support-bench", "dataset_version": "v3", "metadata": { "reranker": "cross-encoder" } }' ``` An Evaluation is a durable record on its own; its leaderboard metrics appear once evaluation telemetry for it lands in Intake. **3. Send it telemetry.** Send the run's telemetry to Intake just as you would for any agent run, whether it comes from the optimizer, a benchmark framework like Harbor, or your own code. Tag each session with the Evaluation's identity: * For ATIF and chat-completions, add a **top-level** `evaluation_context` object to the ingest payload carrying `evaluation_id` (the Evaluation's **name**) and `test_case_id`. * For OTLP, set the `nemo.experiment.id` and `nemo.test_case.id` root-span attributes. The per-evaluator scores on the leaderboard come from **evaluator results** captured on those sessions, either automatically from ATIF verifier rewards or explicitly through the evaluator-results endpoint. See [Capture Evaluator Results](/documentation/agents/observe-agents#capture-evaluator-results) for both paths, and [Observe Agents](/documentation/agents/observe-agents) for the ingestion paths themselves. The `nemo-experiments-upload` skill walks this through end to end. **`test_case_id` is required for a populated leaderboard.** A session tagged with only `evaluation_id` still ingests and appears in the Evaluation's session list, but it doesn't count toward `test_case_count` or any rollup — so the row reads as all zeros, with tokens, model, and agent blank too. Always send `test_case_id` alongside `evaluation_id`. ### Confirm It Worked List the Experiment's Evaluations and confirm the new row is present: ```shell curl -g "$NMP_BASE_URL/apis/intake/v2/workspaces/$WORKSPACE/evaluations?filter[experiment_id]=$EXPERIMENT_ID&page=1&page_size=20" ``` Then open Studio and navigate to `/workspaces/default/experiment`, then open **reranker-prompt-iteration**. **What good looks like:** the Experiment opens in Studio, your Evaluation appears as a row, and once its sessions are ingested the row shows non-zero `test_case_count`, cost, latency, tokens, and evaluator scores. Sorting by a metric reorders the table, and opening the Evaluation lists its individual test cases. ## Common Workflows ### Add an Evaluation to an Existing Experiment This is the everyday operation: an Experiment already exists and you want to record another run in it. Create the Evaluation with the Experiment's `id` in `experiment_ids`: ```shell curl -X POST "$NMP_BASE_URL/apis/intake/v2/workspaces/$WORKSPACE/evaluations" \ -H "Content-Type: application/json" \ -d '{ "name": "reranker-bge-large", "experiment_ids": ["'"$EXPERIMENT_ID"'"], "dataset_name": "support-bench", "dataset_version": "v3", "metadata": { "reranker": "bge-large" } }' ``` To **move or re-scope** an existing Evaluation's membership, `PATCH` its `experiment_ids`, which must stay non-empty: ```shell curl -X PATCH "$NMP_BASE_URL/apis/intake/v2/workspaces/$WORKSPACE/evaluations/reranker-bge-large" \ -H "Content-Type: application/json" \ -d '{ "experiment_ids": ["'"$EXPERIMENT_ID"'", "'"$OTHER_EXPERIMENT_ID"'"] }' ``` An Evaluation can belong to several Experiments at once, which is useful when the same run should appear on both a per-project board and a cross-project benchmark. ### Update an Evaluation Use `PATCH` for partial updates: only the fields you send change. `name`, `dataset_name`, and `dataset_version` are immutable: ```shell curl -X PATCH "$NMP_BASE_URL/apis/intake/v2/workspaces/$WORKSPACE/evaluations/reranker-bge-large" \ -H "Content-Type: application/json" \ -d '{ "status": "winner", "root_cause": "Best groundedness at acceptable cost." }' ``` `PUT` does a full replace of the mutable fields — omitted fields reset — so prefer `PATCH` for one-off edits. ### Rank an Experiment by the Metric That Matters List an Experiment's Evaluations sorted by a metric. Prefix the field with `-` for descending: ```shell curl -g "$NMP_BASE_URL/apis/intake/v2/workspaces/$WORKSPACE/evaluations?filter[experiment_id]=$EXPERIMENT_ID&sort=-evaluators.solved.mean&page=1&page_size=20" ``` You can sort by an entity column (`name`, `created_at`) or any rollup metric (`test_case_count`, `cost_usd.`, `latency_ms.`, `tokens.`, `evaluators..`). In Studio, click a column header. If the Experiment has a `default_sort`, the leaderboard opens already ordered that way. ### Filter to the Evaluations You Care About Filter by a metric range to narrow the leaderboard — for example, only Evaluations whose average cost is under \$0.50: ```shell curl -g "$NMP_BASE_URL/apis/intake/v2/workspaces/$WORKSPACE/evaluations?filter[experiment_id]=$EXPERIMENT_ID&filter[cost_usd.mean][\$lte]=0.5" ``` Metric filters use the same grammar as sort, such as `filter[test_case_count][\$gte]=5` or `filter[evaluators.groundedness.mean][\$gte]=0.8`. You can also filter by `filter[metadata.]=`, `filter[status]=…`, `filter[is_pinned]=true`, and `created_at`/`updated_at` ranges. In Studio, use the column filters on cost, latency, test-case count, and evaluator columns. ### Set an Experiment's Default Sort Store the ordering the team should see first so no one has to re-sort each visit: ```shell curl -X PUT "$NMP_BASE_URL/apis/intake/v2/workspaces/$WORKSPACE/experiments/reranker-prompt-iteration" \ -H "Content-Type: application/json" \ -d '{ "name": "reranker-prompt-iteration", "default_sort": "-evaluators.solved.mean" }' ``` In Studio, set it from the Experiment's Edit dialog. ### Pin a Baseline to the Top Keep the current baseline, or any reference run, at the top of the Experiment for everyone: ```shell curl -X POST "$NMP_BASE_URL/apis/intake/v2/workspaces/$WORKSPACE/evaluations/reranker-main-baseline/pin" # Unpin: curl -X DELETE "$NMP_BASE_URL/apis/intake/v2/workspaces/$WORKSPACE/evaluations/reranker-main-baseline/pin" ``` Pinned Evaluations float to the top of the leaderboard regardless of the active sort. List just the pinned set with `filter[is_pinned]=true`. ### Drill into an Evaluation's Test Cases Open one Evaluation's sessions to see per-test-case behavior: status, latency, cost, and evaluator scores: ```shell curl "$NMP_BASE_URL/apis/intake/v2/workspaces/$WORKSPACE/evaluations/reranker-add-cross-encoder/sessions?page=1&page_size=20" ``` Use this to move from "this candidate scores lower" to "these specific cases regressed." In Studio, open the Evaluation row to reach its sessions, then follow a session into its trace in Intake. ### Configure the Pareto View Each Experiment stores a `pareto` config: the default X/Y metrics for its Pareto (trade-off) chart in Studio, defaulting to cost vs. latency. The axes are `x_metric` and `y_metric`, and each takes a base metric id — `cost_usd`, `latency_ms`, or `evaluators.`, with no `.` suffix. Set it on create or update: ```shell curl -X PUT "$NMP_BASE_URL/apis/intake/v2/workspaces/$WORKSPACE/experiments/reranker-prompt-iteration" \ -H "Content-Type: application/json" \ -d '{ "name": "reranker-prompt-iteration", "pareto": { "x_metric": "cost_usd", "y_metric": "evaluators.solved" } }' ``` ### Delete an Experiment or Evaluation Deleting an Experiment soft-deletes it and cascades to its Evaluations; deleting an Evaluation soft-deletes just that row. Soft-deleted records are hidden from list and get operations unless explicitly requested with `filter[is_deleted]=true`: ```shell curl -X DELETE "$NMP_BASE_URL/apis/intake/v2/workspaces/$WORKSPACE/evaluations/reranker-bge-large" curl -X DELETE "$NMP_BASE_URL/apis/intake/v2/workspaces/$WORKSPACE/experiments/reranker-prompt-iteration" ``` ### Where Evaluations Come From You usually don't create Evaluations by hand. Common producers: * **Optimizer**: records each run as an Experiment and its candidates as Evaluations, so results land automatically. See [Optimize Agents](/documentation/agents/optimize-agents). * **Evaluation framework**: a framework like Harbor sends complete trajectories through Intake's ATIF ingest, and each run becomes an Evaluation with its final metrics as evaluator results. See [Observe Agents](/documentation/agents/observe-agents). * **Direct API**: create the Experiment and Evaluations, then ingest their sessions, for a custom evaluation pipeline. ### Do It with an Agent If you are working with a coding agent, the `nemo-experiments-upload` skill automates this whole path: create an Experiment, add an Evaluation, log traces and scores to an ingest endpoint, and verify the rollups. It ships with reference guides for the ATIF/Harbor, chat-completions, and OTLP ingest formats and a troubleshooting matrix. Invoke it when you want to upload, log, ingest, or publish evaluation runs to NeMo Experiments. ## Operations ### Limits * An Experiment is sorted and filtered in memory over the full set of its Evaluations, bounded to **1,000 Evaluations** per Experiment. A request that would select more is rejected with `413` rather than returned partially sorted. Narrow it with filters. * Page size is capped at 1,000 records. * Metric sorting and filtering require rollups. If ClickHouse is unavailable, a metric sort or filter returns `503` rather than a silently unsorted result. Sorting and filtering by entity columns (`name`, `created_at`) still works. ### Security and Access Control All Experiments endpoints are workspace-scoped under `/apis/intake/v2/workspaces/{workspace}/`, and the service checks workspace access before reading or writing. Evaluation `metadata`, `source_link`, and related fields are producer-supplied and surfaced to reviewers, so treat them like any other telemetry payload. Do not store secrets, credentials, or regulated data unless your deployment's policy permits it. ### Retention and Storage Experiment and Evaluation metadata lives in the platform entity store and persists until you delete it; it does not expire on the telemetry TTL. Rollups are derived from ClickHouse at read time, so leaderboard metrics reflect whatever telemetry is still retained: span and trace-index data expire after 90 days, while evaluator results are retained without that span TTL. An Experiment older than the span window keeps its records and evaluator-based scores, but cost, latency, and run-count rollups reflect only the retained telemetry window. ### Troubleshooting | Symptom | Cause and fix | | ----------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | **Rows show zero metrics** | Three causes: no sessions have been ingested for that Evaluation yet; the sessions were ingested without `test_case_id`, so they don't count toward `test_case_count` or the rollups; or ClickHouse is unreachable. Confirm ingestion in Intake, that sessions carry `test_case_id`, and that the read path returns `200` rather than `503`. | | **A metric sort or filter returns `503`** | Rollups can't be computed because ClickHouse is down. Retry once the read path is healthy, or fall back to an entity-column sort. | | **A list returns `413`** | The Experiment selected more than 1,000 Evaluations for an in-memory sort. Add filters to narrow the set. | | **An Evaluation isn't in the Experiment** | Confirm you created it with the correct Experiment `id` in `experiment_ids` and are querying the right workspace (`filter[experiment_id]=`). | | **`422` on create Evaluation** | An Evaluation must belong to at least one Experiment: provide `experiment_ids`. Required fields are `name`, `experiment_ids`, and `dataset_name`; `metadata` values must be strings. | ## Related Topics * [Observe Agents](/documentation/agents/observe-agents): the telemetry these leaderboards are computed from, and how to ingest it. * [Agent Evaluation](/documentation/evaluate-models/agent-eval): the task-driven evaluation model that produces these runs. * [Optimize Agents](/documentation/agents/optimize-agents): the optimizer that creates Experiments and Evaluations automatically.