Aggregate Metrics

After rollout collection, NeMo Gym computes aggregate metrics for each agent by calling the /aggregate_metrics endpoint on the agent server. The results are written to a single _aggregate_metrics.json file.

To compare these saved metrics between runs, or aggregate benchmark scores across models with confidence intervals, see Compare Results. Run aggregation produces the input artifacts; suite comparison combines their selected repeat scores.

How It Works

1. Rollouts complete — gym eval run --no-serve gathers verify responses (reward + custom fields) for every task/rollout pair.
2. Group by agent — responses are partitioned by agent name.
3. Call /aggregate_metrics — for each agent, the stripped verify responses are POSTed to the agent’s /aggregate_metrics endpoint.
4. Compute stats — per-task and overall statistics (mean, max, min, median, std) are computed for every numeric field. If the resources server overrides compute_metrics() or get_key_metrics(), those are called to add additional metrics.
5. Compute variability stats — when an agent has two or more repeats, eligible numeric per-repeat metrics are summarized across repeats. See Repeat-Level Metrics.
6. Write results — all per-agent metrics are written to <output>_aggregate_metrics.json beside the resolved output. For a file symlink, this is normally beside its target; an unambiguous legacy companion layout beside the link is preserved.

Explicit failure scoring

Failures are excluded from scores by default. count_failure_classes_as_zero explicitly includes selected failures as zero-valued metric inputs without changing saved outcomes. It matches either the canonical _ng_failure_class or the original producer name saved in _ng_failure_legacy_class, when present. For example, a GDPVal failure with kind gdpval:reference_missing still matches an existing reference_missing selection. Selecting both names counts the task once. Collection and offline aggregation use the same rule; see evaluation resume.

Output Format

The output file is a JSON array with one entry per agent:

[
  {
    "agent_ref": {"name": "my_agent"},
    "agent_metrics": {
      "mean/reward": 0.75,
      "max/reward": 1.0,
      "min/reward": 0.0,
      "median/reward": 1.0,
      "std/reward": 0.433,
      "num_repeats": 2,
      "mean_across_repeats/mean/reward": 0.75,
      "median_across_repeats/mean/reward": 0.75,
      "se_across_repeats/mean/reward": 0.042,
      "ci_low_95_across_repeats/mean/reward": 0.66,
      "ci_high_95_across_repeats/mean/reward": 0.84
    },
    "key_metrics": {
      "mean/reward": 0.75
    },
    "group_level_metrics": [
      {"mean/reward": 1.0, "sample": {"...": "..."}},
      {"mean/reward": 0.5, "sample": {"...": "..."}}
    ],
    "repeat_level_metrics": [
      {"_ng_rollout_index": 0, "mean/reward": 0.72, "sem/reward": 0.09, "...": "..."},
      {"_ng_rollout_index": 1, "mean/reward": 0.78, "sem/reward": 0.09, "...": "..."}
    ]
  }
]

Repeat-Level Metrics

A single run is a point estimate. Collecting repeats (--num-repeats) lets you separate a real score difference from sampling noise, and NeMo Gym reports that variability at three levels.

Within a repeat, across tasks

repeat_level_metrics holds one entry per (agent, repeat). It is produced only for agents with two or more repeats — an agent with a single repeat has nothing to compare against and is skipped. When no agent qualifies, the list is empty.

Each entry carries:

plus, for every numeric field (reward, token usage, and any numeric field your verifier returns):

Across repeats

Eligible numeric point estimates in repeat_level_metrics are summarized across repeats and merged into agent_metrics. Each repeat counts as one observation. This includes mean/{field} and custom benchmark metrics (for example, subtask_accuracy) supplied by compute_repeat_metrics(). Existing statistics and uncertainty estimates, including suffixes such as /p95 and /ci_low_95, remain in repeat_level_metrics without being aggregated again.

Per task, across repeats

group_level_metrics reports num_rollouts, mean, median, and std per task. It deliberately carries no confidence interval: a CI here would require assuming a distribution for a single task’s repeated outcomes, which are frequently binomial rather than normal, and the Central Limit Theorem does not rescue it because these are raw outcomes rather than averages.

Two cases worth knowing about

Both emit a UserWarning, so you will see them in your terminal:

• Unequal sample counts across repeats. If a task is missing from some repeats (a crashed rollout, an interrupted run), each repeat’s statistics are computed over a different task set, so they are not directly comparable — and agent_metrics skews toward whichever tasks happened to complete. Collect the missing rollouts before drawing conclusions. Note this fires off missing_count, so it inherits the blind spot above: tasks that failed in every repeat do not trigger it.
• Zero standard error. If every value in a sample is identical, the confidence interval collapses to the single point (mean, mean). This is reported rather than left null, because SciPy’s t.interval computes an indeterminate ±inf * 0 at scale=0 and returns NaN.

Where it lands on disk

The same statistics are laid out differently depending on which command produced them:

gym eval profile always writes its file, containing [] when there is only one repeat. Cross-repeat aggregates (*_across_repeats/{metric}) live in agent_metrics.

Custom Metrics

Override these hooks on your resources server to add custom metrics.

compute_metrics(tasks)

Receives all verify responses grouped by task. Use this for metrics that need the full dataset — pass@k, confidence intervals, cross-task statistics.

compute_repeat_metrics(tasks)

Receives one repeat grouped by task. Return eligible finite numeric point estimates to include them in cross-repeat aggregates.

get_key_metrics(agent_metrics)

Selects headline numbers from full-run agent_metrics. Default returns all mean/* entries.

Example: pass@k

from nemo_gym.server_runtime import (
    BaseVerifyRequest,
    BaseVerifyResponse,
    SimpleResourcesServer,
)
class MathServer(SimpleResourcesServer):
    async def verify(self, body: BaseVerifyRequest) -> BaseVerifyResponse:
        # ... verification logic ...
        pass
    def compute_metrics(self, tasks):
        n_tasks = len(tasks)
        # pass@k: fraction of tasks where at least one rollout got reward=1
        pass_at_k = sum(
            1 for rollouts in tasks if any(r["reward"] >= 1.0 for r in rollouts)
        ) / n_tasks
        # pass@1 (average of per-task mean rewards)
        pass_at_1 = sum(
            sum(r["reward"] for r in rollouts) / len(rollouts)
            for rollouts in tasks
        ) / n_tasks
        return {"pass@k": pass_at_k, "pass@1": pass_at_1}
    def get_key_metrics(self, agent_metrics):
        return {
            k: agent_metrics[k]
            for k in ("pass@k", "pass@1")
            if k in agent_metrics
        }

Given 3 tasks with 4 rollouts each (task 0: all correct, task 1: all wrong, task 2: half correct), this produces:

[
  {
    "agent_ref": {"name": "math_simple_agent"},
    "agent_metrics": {
      "mean/reward": 0.5,
      "max/reward": 1.0,
      "min/reward": 0.0,
      "median/reward": 0.5,
      "std/reward": 0.522,
      "pass@k": 0.667,
      "pass@1": 0.5
    },
    "key_metrics": {
      "pass@k": 0.667,
      "pass@1": 0.5
    },
    "group_level_metrics": [
      {
        "mean/reward": 1.0,
        "max/reward": 1.0,
        "min/reward": 1.0,
        "median/reward": 1.0,
        "std/reward": 0.0
      },
      {
        "mean/reward": 0.0,
        "max/reward": 0.0,
        "min/reward": 0.0,
        "median/reward": 0.0,
        "std/reward": 0.0
      },
      {
        "mean/reward": 0.5,
        "max/reward": 1.0,
        "min/reward": 0.0,
        "median/reward": 0.5,
        "std/reward": 0.577
      }
    ]
  }
]