> For clean Markdown of any page, append .md to the page URL. > For a complete documentation index, see https://docs.nvidia.com/nemo-platform/llms.txt. > For AI client integration (Claude Code, Cursor, etc.), connect to the MCP server at https://docs.nvidia.com/nemo-platform/_mcp/server. # Experiments > Compare evaluation runs on a NeMo Experiments leaderboard with cost, latency, token, and evaluator rollups computed from NeMo Intake telemetry. NeMo Experiments is the comparison layer for agent optimization tasks, such as evaluating the impact of changes to a harness, infrastructure, tools, or agent code. It provides one place to visualize and compare evaluation results from Harbor, the NeMo Optimizer, NeMo Evaluator, or your own runner. You can rank these results by cost, latency, evaluator scores, and other metrics. Those metrics are derived from [NeMo Intake](/documentation/agents/observe-agents) observability data, the same traces and evaluator results your runs already produce, so any runner that lands telemetry in Intake feeds the same comparison. ## Before You Start ### Who This Is For NeMo Experiments is for anyone running evaluations on their agent who needs a way to compare, analyze, and promote the results. For example: engineers iterating on prompts, models, tools, or routing; teams benchmarking many models or configurations at once; and reviewers who need a shared leaderboard to decide what to promote. ### Requirements * A NeMo Platform environment with the `intake` service running. * The platform entity store (Postgres). * A reachable ClickHouse database. * A producer that creates Evaluations and sends their telemetry: the NeMo Optimizer, a benchmark or evaluation framework such as Harbor, or a direct application programming interface (API) integration. * Recommended: [NeMo Studio](/documentation/studio) for the complete web user interface (UI), including trace comparison views, a customizable leaderboard, and Pareto charts. The API provides the same data. The Experiments UI is gated by the `VITE_FF_EXPERIMENT` feature flag, which is off by default. ## How It Works ### What It Does Experiments computes comparable, always-current metrics for your evaluation runs. For each run it rolls up cost, latency, tokens, and per-evaluator scores from the underlying telemetry in ClickHouse and returns them as ranked rows. Rollups are computed at read time, so a leaderboard always reflects current telemetry, with no denormalized score table to maintain. You can group runs by an insight to investigate, a model comparison, a benchmark leaderboard, or promoted runs from several groups. A single run can belong to more than one group. Sorting, filtering, and pinning within a group surface the relevant runs. ### When to Use It Use Experiments when you need to: * **Compare candidates** from the optimizer, a prompt or model change, or a routing strategy against a baseline. * **Run a benchmark leaderboard**: import many runs into one Experiment and rank them by the metric that matters. * **Track iteration over time**: keep one Experiment per project so successive attempts stay side by side. * **Standardize comparison across a team**: everyone reads the same metrics, in the same place, with the same vocabulary. Experiments are most useful after real evaluation telemetry exists. A new group can start empty and fill in as runs land. ### Core Concepts and Data Model Experiments sit on top of Intake's telemetry hierarchy (span → trace → session): * An **Experiment** is a named container of Evaluations. It holds durable metadata (`description`, `summary`, free-form `metadata`, optional `insight_id`) and view configuration: a `default_sort` and a `pareto` (default X/Y metrics for the Pareto view). * An **Evaluation** is one run and one leaderboard row. It records producer-supplied fields (`dataset_name`, `dataset_version`, `source_link`, `metadata`, `description`, `status`, `root_cause`), the `experiment_ids` it belongs to (at least one; an Evaluation can live in more than one Experiment), and an optional `parent_evaluation_id` linking a variant back to the run it was derived from. * A **Session** is one test-case execution within an Evaluation, a single ingested run made of spans. Sessions carry per-case status, latency, token and cost totals, and evaluator scores. At read time, each Evaluation is enriched with rollups derived from its sessions in ClickHouse: | Rollup | Meaning | | ---------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------- | | `test_case_count` | Number of distinct test cases (distinct non-empty `test_case_name` values). Sessions with no `test_case_name` do not count toward it or the rollups. | | `cost_usd` | Cost aggregate across the Evaluation's sessions. | | `latency_ms` | Latency aggregate across the Evaluation's sessions. | | `tokens` | Average total tokens (input + output) per test case. | | `evaluators.` | Aggregate of a named evaluator's session scores. | | `model_names`, `agent_names`, `agent_versions` | Distinct models, agents, and versions observed in the telemetry. | Metric aggregates expose these statistics: `sum`, `mean`, `median`, `p90`, `p95`, `p99`, `count`. A **metric path** is therefore `test_case_count`, `cost_usd.`, `latency_ms.`, `tokens.`, or `evaluators..`. Sorting and filtering use the same grammar. Two things about evaluator rollups specifically: * **Response shape.** `evaluators..` is the sort/filter *query* grammar. The Evaluation JSON response returns the same scores under `aggregate_scores`, a map keyed by evaluator name. Each value contains the statistics above, and an `evaluator_names` list appears alongside it. There is no `evaluators` field in the response; `evaluators..` (query) reads `aggregate_scores[].` (response). * **Missing-value semantics.** Evaluator rollups are test-case-weighted: each stat is computed over `test_case_count`, the full set of test cases, with a test case that did not report a given evaluator counted as `0` rather than dropped. Two Experiment-level behaviors are worth knowing: * **Default sort.** An Experiment stores a `default_sort` (a sort-param string such as `-evaluators.solved.mean`) so its leaderboard opens ordered by the metric the team cares about. It defaults to `-created_at` (newest first). * **Pinning.** Any Evaluation can be pinned to the top of its Experiment, such as the current baseline. Pins are workspace-shared: everyone with access sees the same pinned set, regardless of the active sort. ## Get Started ### Setup Experiments are part of the `intake` service, so any running NeMo Platform already serves them. Point at whatever you have, whether a deployed platform or a local one from `nemo setup` or `nemo quickstart up`: ```bash export NMP_BASE_URL=http://127.0.0.1:8080 export WORKSPACE=default export NMP_STUDIO_URL="$NMP_BASE_URL/studio" export NMP_ACCESS_TOKEN="$(nemo auth token)" ``` Set `NMP_STUDIO_URL` explicitly if Studio uses a different origin or path. The examples use bearer-token authentication and stop on HTTP errors. For a local development deployment with authentication disabled, omit the `Authorization` header and the `NMP_ACCESS_TOKEN` command. ClickHouse must be reachable either way, since the leaderboard rollups are computed from it at read time. #### Running from source (local development) If you are bringing the pieces up yourself from a repository checkout, make sure Docker Desktop or the Docker daemon is running. Intake automatically provisions and reuses a local ClickHouse container for the resolved NeMo data directory unless `NMP_INTAKE_CLICKHOUSE_URL` explicitly points to an external instance. 1. **Backend services**: `intake` plus its `auth` and `entities` dependencies. `--port` defaults to `8080`; omit `uv run` if you installed the `nemo` command-line interface (CLI): ```bash uv run nemo services run --services auth,entities,intake --host 127.0.0.1 --port 8080 ``` 2. **Studio** (optional, for the leaderboard and drill-down UI). From the `web/` workspace, with the Experiments feature flag on and intake enabled for trace drill-down: ```bash VITE_FF_EXPERIMENT=true VITE_FF_INTAKE_ENABLED=true VITE_PLATFORM_BASE_URL=http://127.0.0.1:8080 \ pnpm --filter nemo-studio-ui start -- --host 127.0.0.1 ``` Confirm that the entity read path and ClickHouse rollups are reachable: ```bash curl --fail-with-body -i \ -H "Authorization: Bearer $NMP_ACCESS_TOKEN" \ "$NMP_BASE_URL/apis/intake/v2/workspaces/$WORKSPACE/experiments?sort=-created_at&page=1&page_size=1" curl --fail-with-body -i \ -H "Authorization: Bearer $NMP_ACCESS_TOKEN" \ "$NMP_BASE_URL/apis/intake/v2/workspaces/$WORKSPACE/evaluations?sort=-cost_usd.mean&page=1&page_size=1" ``` The first request checks the entity store and can return `200` while ClickHouse is unavailable. The second request explicitly sorts by a rollup metric. A `200` response from both requests is healthy. A `503` response from the second request means that Intake cannot compute ClickHouse rollups. ### First Workflow Create an Experiment, add an Evaluation, send it telemetry, then see it in Studio and through the API. **1. Create the Experiment** (the leaderboard container): ```bash curl --fail-with-body -X POST \ "$NMP_BASE_URL/apis/intake/v2/workspaces/$WORKSPACE/experiments" \ -H "Authorization: Bearer $NMP_ACCESS_TOKEN" \ -H "Content-Type: application/json" \ -d '{ "name": "reranker-prompt-iteration", "description": "Iterating on the support-bench retrieval-augmented generation agent'\''s reranker and system prompt." }' ``` Capture its `id` for the next step: ```bash export EXPERIMENT_ID=$(curl --fail-with-body --silent \ -H "Authorization: Bearer $NMP_ACCESS_TOKEN" \ "$NMP_BASE_URL/apis/intake/v2/workspaces/$WORKSPACE/experiments/reranker-prompt-iteration" \ | python3 -c 'import sys,json; print(json.load(sys.stdin)["id"])') echo "experiment id: $EXPERIMENT_ID" ``` **2. Add an Evaluation** to it. `experiment_ids` is a list, so an Evaluation belongs to one or more existing Experiments: ```bash curl --fail-with-body -X POST \ "$NMP_BASE_URL/apis/intake/v2/workspaces/$WORKSPACE/evaluations" \ -H "Authorization: Bearer $NMP_ACCESS_TOKEN" \ -H "Content-Type: application/json" \ -d '{ "name": "reranker-add-cross-encoder", "experiment_ids": ["'"$EXPERIMENT_ID"'"], "dataset_name": "support-bench", "dataset_version": "v3", "metadata": { "reranker": "cross-encoder" } }' ``` An Evaluation is a durable record. Its leaderboard metrics appear after its evaluation telemetry lands in Intake. **3. Send it telemetry.** Send the run's telemetry to Intake in the same way as other agent runs, whether it comes from the optimizer, a benchmark framework like Harbor, or your own code. Tag each session with the Evaluation's identity: * For Agent Trajectory Interchange Format (ATIF) and chat-completions, add a **top-level** `evaluation_context` object to the ingest payload carrying `evaluation_name` and `test_case_name`. * For OpenTelemetry Protocol (OTLP), set `nemo.evaluation.name` and `nemo.test_case.name` on each trace's root span. The per-evaluator scores on the leaderboard come from **evaluator results** captured on those sessions. Intake can derive them from ATIF verifier rewards. OTLP traces carry session telemetry, so the framework or caller must publish their scores through the evaluator-results endpoint. Refer to [Capture Evaluator Results](/documentation/agents/observe-agents#capture-evaluator-results) for both paths, and [Observe Agents](/documentation/agents/observe-agents) for the ingestion paths themselves. The `nemo-experiments-upload` skill walks this through end to end. **`test_case_name` is required for a populated leaderboard.** A session tagged with only `evaluation_name` still ingests and appears in the Evaluation's session list, but it does not count toward `test_case_count` or any rollup. The row therefore reads as all zeros, with tokens, model, and agent blank too. Always send `test_case_name` alongside `evaluation_name`. ### Confirm It Worked List the Experiment's Evaluations and confirm the new row is present: ```bash curl -g --fail-with-body \ -H "Authorization: Bearer $NMP_ACCESS_TOKEN" \ "$NMP_BASE_URL/apis/intake/v2/workspaces/$WORKSPACE/evaluations?filter[experiment_id]=$EXPERIMENT_ID&page=1&page_size=20" ``` Then open `$NMP_STUDIO_URL/workspaces/$WORKSPACE/experiment`, then open **reranker-prompt-iteration**. A successful result opens the Experiment in Studio and shows your Evaluation as a row. After its sessions are ingested the row shows non-zero `test_case_count`, cost, latency, tokens, and evaluator scores. Sorting by a metric reorders the table, and opening the Evaluation lists its individual test cases. ## Common Workflows ### Add an Evaluation to an Existing Experiment This is the everyday operation: an Experiment already exists and you want to record another run in it. Create the Evaluation with the Experiment's `id` in `experiment_ids`: ```bash curl --fail-with-body -X POST \ "$NMP_BASE_URL/apis/intake/v2/workspaces/$WORKSPACE/evaluations" \ -H "Authorization: Bearer $NMP_ACCESS_TOKEN" \ -H "Content-Type: application/json" \ -d '{ "name": "reranker-bge-large", "experiment_ids": ["'"$EXPERIMENT_ID"'"], "dataset_name": "support-bench", "dataset_version": "v3", "metadata": { "reranker": "bge-large" } }' ``` ### Associate an Evaluation with Another Experiment An Evaluation can belong to several Experiments simultaneously, which is useful when the same run should appear on both a per-project board and a cross-project benchmark. To demonstrate adding a second membership, first create the second Experiment and capture its ID: ```bash curl --fail-with-body -X POST \ "$NMP_BASE_URL/apis/intake/v2/workspaces/$WORKSPACE/experiments" \ -H "Authorization: Bearer $NMP_ACCESS_TOKEN" \ -H "Content-Type: application/json" \ -d '{ "name": "reranker-cross-project-benchmark", "description": "Compare reranker candidates across projects." }' export OTHER_EXPERIMENT_ID=$(curl --fail-with-body --silent \ -H "Authorization: Bearer $NMP_ACCESS_TOKEN" \ "$NMP_BASE_URL/apis/intake/v2/workspaces/$WORKSPACE/experiments/reranker-cross-project-benchmark" \ | python3 -c 'import sys,json; print(json.load(sys.stdin)["id"])') ``` Then `PATCH` the existing Evaluation's `experiment_ids` with both IDs. The list must stay non-empty: ```bash curl --fail-with-body -X PATCH \ "$NMP_BASE_URL/apis/intake/v2/workspaces/$WORKSPACE/evaluations/reranker-bge-large" \ -H "Authorization: Bearer $NMP_ACCESS_TOKEN" \ -H "Content-Type: application/json" \ -d '{ "experiment_ids": ["'"$EXPERIMENT_ID"'", "'"$OTHER_EXPERIMENT_ID"'"] }' ``` ### Update an Evaluation Use `PATCH` for partial updates: only the fields you send change. `name`, `dataset_name`, and `dataset_version` are immutable: ```bash curl --fail-with-body -X PATCH \ "$NMP_BASE_URL/apis/intake/v2/workspaces/$WORKSPACE/evaluations/reranker-bge-large" \ -H "Authorization: Bearer $NMP_ACCESS_TOKEN" \ -H "Content-Type: application/json" \ -d '{ "status": "winner", "root_cause": "Best groundedness at acceptable cost." }' ``` `PUT` does a full replacement of the mutable fields. Omitted fields reset, so prefer `PATCH` for individual edits. ### Rank an Experiment by the Metric That Matters List an Experiment's Evaluations sorted by a metric. Prefix the field with `-` for descending: ```bash curl -g --fail-with-body \ -H "Authorization: Bearer $NMP_ACCESS_TOKEN" \ "$NMP_BASE_URL/apis/intake/v2/workspaces/$WORKSPACE/evaluations?filter[experiment_id]=$EXPERIMENT_ID&sort=-evaluators.solved.mean&page=1&page_size=20" ``` You can sort by an entity column (`name`, `created_at`) or any rollup metric (`test_case_count`, `cost_usd.`, `latency_ms.`, `tokens.`, `evaluators..`). In Studio, click a column header. If the Experiment has a `default_sort`, the leaderboard opens already ordered that way. ### Filter to the Evaluations You Care About Filter by a metric range to narrow the leaderboard. For example, select only Evaluations whose average cost is under \$0.50: ```bash curl -g --fail-with-body \ -H "Authorization: Bearer $NMP_ACCESS_TOKEN" \ "$NMP_BASE_URL/apis/intake/v2/workspaces/$WORKSPACE/evaluations?filter[experiment_id]=$EXPERIMENT_ID&filter[cost_usd.mean][\$lte]=0.5" ``` Metric filters use the same grammar as sort, such as `filter[test_case_count][\$gte]=5` or `filter[evaluators.groundedness.mean][\$gte]=0.8`. You can also filter by `filter[metadata.]=`, `filter[status]=`, `filter[is_pinned]=true`, and `created_at`/`updated_at` ranges. In Studio, use the column filters on cost, latency, test-case count, and evaluator columns. ### Set an Experiment's Default Sort Store the ordering the team should see first so no one has to re-sort each visit: ```bash curl --fail-with-body -X PUT \ "$NMP_BASE_URL/apis/intake/v2/workspaces/$WORKSPACE/experiments/reranker-prompt-iteration" \ -H "Authorization: Bearer $NMP_ACCESS_TOKEN" \ -H "Content-Type: application/json" \ -d '{ "name": "reranker-prompt-iteration", "default_sort": "-evaluators.solved.mean" }' ``` In Studio, set it from the Experiment's Edit dialog. ### Pin a Baseline to the Top Keep the current baseline, or any reference run, at the top of the Experiment for everyone: ```bash curl --fail-with-body -X POST \ "$NMP_BASE_URL/apis/intake/v2/workspaces/$WORKSPACE/evaluations" \ -H "Authorization: Bearer $NMP_ACCESS_TOKEN" \ -H "Content-Type: application/json" \ -d '{ "name": "reranker-main-baseline", "experiment_ids": ["'"$EXPERIMENT_ID"'"], "dataset_name": "support-bench", "dataset_version": "v3", "metadata": { "reranker": "main-baseline" } }' curl --fail-with-body -X POST \ -H "Authorization: Bearer $NMP_ACCESS_TOKEN" \ "$NMP_BASE_URL/apis/intake/v2/workspaces/$WORKSPACE/evaluations/reranker-main-baseline/pin" # Unpin: curl --fail-with-body -X DELETE \ -H "Authorization: Bearer $NMP_ACCESS_TOKEN" \ "$NMP_BASE_URL/apis/intake/v2/workspaces/$WORKSPACE/evaluations/reranker-main-baseline/pin" ``` Pinned Evaluations float to the top of the leaderboard regardless of the active sort. List only the pinned set with `filter[is_pinned]=true`. ### Drill into an Evaluation's Test Cases Open one Evaluation's sessions to see per-test-case behavior: status, latency, cost, and evaluator scores: ```bash curl --fail-with-body \ -H "Authorization: Bearer $NMP_ACCESS_TOKEN" \ "$NMP_BASE_URL/apis/intake/v2/workspaces/$WORKSPACE/evaluations/reranker-add-cross-encoder/sessions?page=1&page_size=20" ``` Use this to move from "this candidate scores lower" to "these specific cases regressed." In Studio, open the Evaluation row to reach its sessions, then follow a session into its trace in Intake. ### Configure the Pareto View Each Experiment stores a `pareto` configuration: the default X/Y metrics for its Pareto chart in Studio, defaulting to cost vs. latency. The axes are `x_metric` and `y_metric`, and each takes a base metric ID: `cost_usd`, `latency_ms`, or `evaluators.`, with no `.` suffix. Set it on create or update: ```bash curl --fail-with-body -X PUT \ "$NMP_BASE_URL/apis/intake/v2/workspaces/$WORKSPACE/experiments/reranker-prompt-iteration" \ -H "Authorization: Bearer $NMP_ACCESS_TOKEN" \ -H "Content-Type: application/json" \ -d '{ "name": "reranker-prompt-iteration", "pareto": { "x_metric": "cost_usd", "y_metric": "evaluators.solved" } }' ``` ### Delete an Experiment or Evaluation Deleting an Experiment soft-deletes it. An Evaluation that belongs only to that Experiment is also soft-deleted. A shared Evaluation remains active, and the deleted Experiment is removed from its `experiment_ids` membership. Deleting an Evaluation directly soft-deletes only that Evaluation. ```bash curl --fail-with-body -X DELETE \ -H "Authorization: Bearer $NMP_ACCESS_TOKEN" \ "$NMP_BASE_URL/apis/intake/v2/workspaces/$WORKSPACE/evaluations/reranker-bge-large" curl --fail-with-body -X DELETE \ -H "Authorization: Bearer $NMP_ACCESS_TOKEN" \ "$NMP_BASE_URL/apis/intake/v2/workspaces/$WORKSPACE/experiments/reranker-cross-project-benchmark" curl --fail-with-body -X DELETE \ -H "Authorization: Bearer $NMP_ACCESS_TOKEN" \ "$NMP_BASE_URL/apis/intake/v2/workspaces/$WORKSPACE/experiments/reranker-prompt-iteration" ``` Soft deletion renames the stored record. A `GET` request that uses the original name does not return the deleted record. To audit deleted records, use `filter[is_deleted]=true` on a list operation: ```bash curl -g --fail-with-body \ -H "Authorization: Bearer $NMP_ACCESS_TOKEN" \ "$NMP_BASE_URL/apis/intake/v2/workspaces/$WORKSPACE/evaluations?filter[is_deleted]=true&page=1&page_size=20" ``` ### Where Evaluations Come From You typically do not create Evaluations manually. Common producers include: * **Optimizer**: records each run as an Experiment and its candidates as Evaluations, so results land automatically. Refer to [Optimize Agents](/documentation/agents/optimize-agents). * **Evaluation framework**: a framework like Harbor sends trajectories through Intake's ATIF or OTLP ingest, publishes its scores as evaluator results, and records each run as an Evaluation. Refer to [Observe Agents](/documentation/agents/observe-agents). * **Direct API**: create the Experiment and Evaluations, then ingest their sessions, for a custom evaluation pipeline. ### Do It with an Agent If you are working with a coding agent, the `nemo-experiments-upload` skill automates this whole path: create an Experiment, add an Evaluation, log traces and scores to an ingest endpoint, and verify the rollups. It ships with reference guides for the ATIF/Harbor, chat-completions, and OTLP ingest formats and a troubleshooting matrix. Invoke it when you want to upload, log, ingest, or publish evaluation runs to NeMo Experiments. ## Operations ### Limits * An Experiment is sorted and filtered in memory over the full set of its Evaluations, bounded to **1,000 Evaluations** per Experiment. A request that would select more is rejected with `413` rather than returned partially sorted. Narrow it with filters. * Page size is capped at 1,000 records. * Metric sorting and filtering require rollups. If ClickHouse is unavailable, a metric sort or filter returns `503` rather than a silently unsorted result. Sorting and filtering by entity columns (`name`, `created_at`) still works. ### Security and Access Control All Experiments endpoints are workspace-scoped under `/apis/intake/v2/workspaces/{workspace}/`, and the service checks workspace access before reading or writing. Evaluation `metadata`, `source_link`, and related fields are producer-supplied and surfaced to reviewers, so treat them like any other telemetry payload. Do not store secrets, credentials, or regulated data unless your deployment's policy permits it. ### Retention and Storage Experiment and Evaluation metadata lives in the platform entity store and persists until you delete it. This metadata does not expire with telemetry. Span and trace-index records in ClickHouse expire after 90 days. Evaluator-result rows do not use that 90-day time-to-live (TTL) setting. All leaderboard rollups depend on the retained trace index. Evaluator score rollups join the retained trace index to evaluator-result rows at read time. After a trace-index entry expires, its evaluator scores no longer appear on the leaderboard even if the evaluator-result rows remain. Cost, latency, token, test-case count, and evaluator score rollups therefore reflect the retained 90-day trace window. ### Troubleshooting | Symptom | Cause and fix | | ------------------------------------------ | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | | **Rows show zero metrics** | Three causes: no sessions have been ingested for that Evaluation yet; the sessions were ingested without `test_case_name`, so they do not count toward `test_case_count` or the rollups; or ClickHouse is unreachable. Confirm ingestion in Intake, confirm that sessions carry `test_case_name`, and run the explicit metric-sort health check. | | **A metric sort or filter returns `503`** | Rollups cannot be computed because ClickHouse is down. Retry after the read path is healthy, or use an entity-column sort. | | **A list returns `413`** | The Experiment selected more than 1,000 Evaluations for an in-memory sort. Add filters to narrow the set. | | **An Evaluation is not in the Experiment** | Confirm that you created it with the correct Experiment `id` in `experiment_ids` and that you are querying the correct workspace (`filter[experiment_id]=`). | | **`422` on create Evaluation** | An Evaluation must belong to at least one Experiment: provide `experiment_ids`. Required fields are `name`, `experiment_ids`, and `dataset_name`; `metadata` values must be strings. | ## Related Topics * [Observe Agents](/documentation/agents/observe-agents): the telemetry these leaderboards are computed from, and how to ingest it. * [Agent Evaluation](/documentation/evaluate-models/agent-eval): the task-driven evaluation model that produces these runs. * [Optimize Agents](/documentation/agents/optimize-agents): the optimizer that creates Experiments and Evaluations automatically.