> This page is for version Latest (default). > For other versions, use one of these documentation indexes: > - Latest (default): https://docs.nvidia.com/nemo-helix/latest/llms.txt > - 0.9: https://docs.nvidia.com/nemo-helix/v0.9/llms.txt > - 0.8: https://docs.nvidia.com/nemo-helix/v0.8/llms.txt > - 0.7: https://docs.nvidia.com/nemo-helix/v0.7/llms.txt > - 0.6: https://docs.nvidia.com/nemo-helix/v0.6/llms.txt > - 0.5: https://docs.nvidia.com/nemo-helix/v0.5/llms.txt > For clean Markdown of any page, append .md to the page URL. > For a complete documentation index, see https://docs.nvidia.com/nemo-helix/llms.txt. > For AI client integration (Claude Code, Cursor, etc.), connect to the MCP server at https://docs.nvidia.com/nemo-helix/_mcp/server. # Evaluate Agent Candidates > Keep each agent candidate's traces apart with one Evaluation per candidate, then score, analyze, and compare the candidates by Evaluation. When you try changes to an agent, such as a new prompt, model, or tool, give each candidate its own **Evaluation** and tag every run with it. NeMo Intake, NeMo Compass, and NeMo Studio can then select one candidate's traces by Evaluation name, even when candidates run at the same time. Do not tell candidates apart by time window or by a temporary agent name: * Candidates that run concurrently overlap in time, so a time window mixes their runs. A discarded candidate's runs also stay in every window that covers them. * Compass stores Insights under the agent name. A per-candidate agent name scatters findings about one agent across several names. ## Before You Start You need: * A running NeMo Helix with Intake. Compass and NeMo Studio are needed for the analysis and comparison steps. * An Experiment to hold the candidates. See the [Experiments First Workflow](/nemo-helix/documentation/evaluate-models/experiments#first-workflow). * One agent name for every candidate. Every trace must carry the same `agent_name`, the one you pass to Compass as `--agent`. See [Telemetry Requirement](/nemo-helix/documentation/agents/traces-and-insights/insights-from-traces#telemetry-requirement). The examples use the agent `support-agent` and two candidates, `support-agent-baseline` and `support-agent-short-prompt`. ## Steps ### Create One Evaluation per Candidate Create an Evaluation for each candidate in the same Experiment **before** you send its runs. Intake rejects runs that name an Evaluation that does not exist. Use a new name for each candidate, and record what changed in the Evaluation's `metadata`. Do not reuse a deleted Evaluation's name: traces keep the name they were ingested with, so the new Evaluation would also select the deleted one's runs. See [Add an Evaluation to an Existing Experiment](/nemo-helix/documentation/evaluate-models/experiments#add-an-evaluation-to-an-existing-experiment). If you run candidates as agent evaluation jobs, the job publishes each run and its scores to the Evaluation named by `publication.intake.evaluation_id`, which covers the next two steps. The job derives the recorded agent name from the target. When candidates are separate targets and the job publishes an ATIF trajectory because the trial has no OTLP trace, set `publication.intake.agent_name` to the shared agent name. That field is written onto the ATIF trajectory only. When the job publishes the trial's OTLP trace, the agent name stays whatever the trace already records as `gen_ai.agent.name`. The producer must record the shared name. See [Telemetry Requirement](/nemo-helix/documentation/agents/traces-and-insights/insights-from-traces#telemetry-requirement). The **Run Evaluation** wizard in [NeMo Studio Agents](/nemo-helix/documentation/studio/agents#evaluate-an-agent) also creates the Experiment and Evaluation. See [Where Evaluations Come From](/nemo-helix/documentation/evaluate-models/experiments#where-evaluations-come-from). ### Tag Every Run with Its Evaluation and Test Case Tag each run at ingest with the candidate's Evaluation name and the test case it ran: * For chat-completions, add a top-level `evaluation_context` object to the ingest payload. * For ATIF, put the same object in the trajectory's `extra.evaluation_context`. The ingest payload also accepts that object as a top-level `evaluation_context`. When both are present, they must match. * For OTLP, set `nemo.evaluation.name` and `nemo.test_case.name` on each trace's root span. ```json { "evaluation_context": { "evaluation_name": "support-agent-short-prompt", "test_case_name": "refund-request-001" } } ``` Send `test_case_name` on every run, and use the same test case name for the same task in every candidate. A run without `test_case_name` does not count toward the Evaluation's leaderboard metrics. Shared test case names let Studio compare two candidates' runs of one task, and let Compass count repeated attempts at one task as a single case. For full ingest examples, see step 3 of the [Experiments First Workflow](/nemo-helix/documentation/evaluate-models/experiments#first-workflow). ### Attach Evaluator Results Attach each run's scores as evaluator results. ATIF trajectories can carry them in `extra.verifier_result.rewards`; for other formats, post them to the evaluator-results endpoint. See [Capture Evaluator Results](/nemo-helix/documentation/agents/traces-and-insights/collect-agent-traces#capture-evaluator-results). Use the same evaluator names in every candidate, such as `solved` or `faithfulness/v1`, so the candidates' scores line up in one leaderboard column. Evaluator results also enable the Compass `grader-failure-patterns` stream, which is skipped for traces without them. ### List One Candidate's Traces Filter Intake's trace list by Evaluation name: ```bash nemo intake traces list \ --filter.agent-name support-agent \ --filter.evaluation-name support-agent-short-prompt ``` Add `--filter.test-case-name refund-request-001` to list one test case. The traces API takes the same filter as `filter[evaluation_name]=support-agent-short-prompt`. ### Analyze One Candidate with Compass Pass the Evaluation name to `--evaluation-id`: ```bash nemo compass analysis-runs create \ --agent support-agent \ --evaluation-id support-agent-short-prompt \ --wait ``` `--evaluation-id` restricts every trace and span the run reads to that Evaluation. `--agent` is still required and must match the traces' `agent_name`. Do not use `--since` to isolate a candidate. Two things stay true when the run reads only one Evaluation: * Compass still applies its recurrence thresholds to the traces it reads. It counts independent cases by test case name, so a tool issue is recorded only after it shows up in 3 distinct test cases in that Evaluation. Repeating one test case does not add a case, so a smaller Evaluation can leave a real problem `below threshold`. See [Recurrence Thresholds](/nemo-helix/documentation/agents/traces-and-insights/insights-from-traces#recurrence-thresholds). * Insights are stored on the agent, not on the Evaluation. The run's `analysis-report` records which Insights that run created or updated. See [Read the Analysis Report](/nemo-helix/documentation/agents/traces-and-insights/insights-from-traces#read-the-analysis-report). ### Compare Candidates In NeMo Studio, open the Experiment and toggle **Scorecard**. It lists every Evaluation side by side with each evaluator's score and its change against a baseline Evaluation. Pick the baseline from **Baseline**. To compare two candidates on one task, open an Evaluation, open a session, and select **Compare against evaluation run**. The list offers that test case's other runs, grouped by Evaluation, from every Evaluation in the Experiment. See [NeMo Studio Experiments](/nemo-helix/documentation/studio/experiments). To rank candidates through the API instead, see [Rank an Experiment by the Metric That Matters](/nemo-helix/documentation/evaluate-models/experiments#rank-an-experiment-by-the-metric-that-matters). ## Troubleshooting | Symptom | Cause and fix | | ----------------------------------------------------------------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | ATIF or chat-completions ingest returns `400` with `Evaluation '' must be created before it can be logged.` | The Evaluation does not exist in this workspace, or it was deleted. Create it first, or create a new Evaluation with a new name. | | OTLP traces for a candidate are missing | OTLP ingest drops spans whose `nemo.evaluation.name` names an Evaluation that does not exist, and still returns success. The response body's `errors` list carries the same message. Create the Evaluation, then send the traces again. | | A candidate's leaderboard row shows zero metrics | Its runs were ingested without `test_case_name`. See [Experiments Troubleshooting](/nemo-helix/documentation/evaluate-models/experiments#troubleshooting). | | Compass reports `Analyzed 0 traces` | `--agent` does not match the traces' `agent_name`, or `--evaluation-id` is not the Evaluation name used at ingest. | ## Related Topics * [Experiments](/nemo-helix/documentation/evaluate-models/experiments) * [Collect Agent Traces](/nemo-helix/documentation/agents/traces-and-insights/collect-agent-traces) * [Insights from Traces](/nemo-helix/documentation/agents/traces-and-insights/insights-from-traces) * [NeMo Studio Experiments](/nemo-helix/documentation/studio/experiments) * [Optimize Agents](/nemo-helix/documentation/agents/optimize-agents)