Evaluate Agent Candidates

When you try changes to an agent, such as a new prompt, model, or tool, give each candidate its own Evaluation and tag every run with it. NeMo Intake, NeMo Compass, and NeMo Studio can then select one candidate’s traces by Evaluation name, even when candidates run at the same time.

Do not tell candidates apart by time window or by a temporary agent name:

• Candidates that run concurrently overlap in time, so a time window mixes their runs. A discarded candidate’s runs also stay in every window that covers them.
• Compass stores Insights under the agent name. A per-candidate agent name scatters findings about one agent across several names.

Before You Start

You need:

• A running NeMo Helix with Intake. Compass and NeMo Studio are needed for the analysis and comparison steps.
• An Experiment to hold the candidates. See the Experiments First Workflow.
• One agent name for every candidate. Every trace must carry the same agent_name, the one you pass to Compass as --agent. See Telemetry Requirement.

The examples use the agent support-agent and two candidates, support-agent-baseline and support-agent-short-prompt.

Steps

Create One Evaluation per Candidate

Create an Evaluation for each candidate in the same Experiment before you send its runs. Intake rejects runs that name an Evaluation that does not exist. Use a new name for each candidate, and record what changed in the Evaluation’s metadata. Do not reuse a deleted Evaluation’s name: traces keep the name they were ingested with, so the new Evaluation would also select the deleted one’s runs. See Add an Evaluation to an Existing Experiment.

If you run candidates as agent evaluation jobs, the job publishes each run and its scores to the Evaluation named by publication.intake.evaluation_id, which covers the next two steps. The job derives the recorded agent name from the target. When candidates are separate targets and the job publishes an ATIF trajectory because the trial has no OTLP trace, set publication.intake.agent_name to the shared agent name. That field is written onto the ATIF trajectory only. When the job publishes the trial’s OTLP trace, the agent name stays whatever the trace already records as gen_ai.agent.name. The producer must record the shared name. See Telemetry Requirement. The Run Evaluation wizard in NeMo Studio Agents also creates the Experiment and Evaluation. See Where Evaluations Come From.

Tag Every Run with Its Evaluation and Test Case

Tag each run at ingest with the candidate’s Evaluation name and the test case it ran:

• For chat-completions, add a top-level evaluation_context object to the ingest payload.
• For ATIF, put the same object in the trajectory’s extra.evaluation_context. The ingest payload also accepts that object as a top-level evaluation_context. When both are present, they must match.
• For OTLP, set nemo.evaluation.name and nemo.test_case.name on each trace’s root span.

{
  "evaluation_context": {
    "evaluation_name": "support-agent-short-prompt",
    "test_case_name": "refund-request-001"
  }
}

Send test_case_name on every run, and use the same test case name for the same task in every candidate. A run without test_case_name does not count toward the Evaluation’s leaderboard metrics. Shared test case names let Studio compare two candidates’ runs of one task, and let Compass count repeated attempts at one task as a single case. For full ingest examples, see step 3 of the Experiments First Workflow.

Attach Evaluator Results

Attach each run’s scores as evaluator results. ATIF trajectories can carry them in extra.verifier_result.rewards; for other formats, post them to the evaluator-results endpoint. See Capture Evaluator Results.

Use the same evaluator names in every candidate, such as solved or faithfulness/v1, so the candidates’ scores line up in one leaderboard column. Evaluator results also enable the Compass grader-failure-patterns stream, which is skipped for traces without them.

List One Candidate’s Traces

Filter Intake’s trace list by Evaluation name:

nemo intake traces list \
  --filter.agent-name support-agent \
  --filter.evaluation-name support-agent-short-prompt

Add --filter.test-case-name refund-request-001 to list one test case. The traces API takes the same filter as filter[evaluation_name]=support-agent-short-prompt.

Analyze One Candidate with Compass

Pass the Evaluation name to --evaluation-id:

nemo compass analysis-runs create \
  --agent support-agent \
  --evaluation-id support-agent-short-prompt \
  --wait

--evaluation-id restricts every trace and span the run reads to that Evaluation. --agent is still required and must match the traces’ agent_name. Do not use --since to isolate a candidate.

Two things stay true when the run reads only one Evaluation:

• Compass still applies its recurrence thresholds to the traces it reads. It counts independent cases by test case name, so a tool issue is recorded only after it shows up in 3 distinct test cases in that Evaluation. Repeating one test case does not add a case, so a smaller Evaluation can leave a real problem below threshold. See Recurrence Thresholds.
• Insights are stored on the agent, not on the Evaluation. The run’s analysis-report records which Insights that run created or updated. See Read the Analysis Report.

Compare Candidates

In NeMo Studio, open the Experiment and toggle Scorecard. It lists every Evaluation side by side with each evaluator’s score and its change against a baseline Evaluation. Pick the baseline from Baseline.

To compare two candidates on one task, open an Evaluation, open a session, and select Compare against evaluation run. The list offers that test case’s other runs, grouped by Evaluation, from every Evaluation in the Experiment. See NeMo Studio Experiments.

To rank candidates through the API instead, see Rank an Experiment by the Metric That Matters.

Troubleshooting

Related Topics

• Experiments
• Collect Agent Traces
• Insights from Traces
• NeMo Studio Experiments
• Optimize Agents