Aggregate Metrics

After rollout collection, NeMo Gym computes aggregate metrics for each agent by calling the /aggregate_metrics endpoint on the agent server. The results are written to a single _aggregate_metrics.json file.

How It Works

1. Rollouts complete — gym eval run --no-serve gathers verify responses (reward + custom fields) for every task/rollout pair.
2. Group by agent — responses are partitioned by agent name.
3. Call /aggregate_metrics — for each agent, the stripped verify responses are POSTed to the agent’s /aggregate_metrics endpoint.
4. Compute stats — per-task and overall statistics (mean, max, min, median, std) are computed for every numeric field. If the resources server overrides compute_metrics() or get_key_metrics(), those are called to add additional metrics.
5. Compute variability stats — when an agent has two or more repeats, per-repeat statistics are computed for every numeric field and summarized across repeats. See Repeat-Level Metrics.
6. Write results — all per-agent metrics are written to <output>_aggregate_metrics.json.

Output Format

The output file is a JSON array with one entry per agent:

1
[
2
  {
3
    "agent_ref": {"name": "my_agent"},
4
    "agent_metrics": {
5
      "mean/reward": 0.75,
6
      "max/reward": 1.0,
7
      "min/reward": 0.0,
8
      "median/reward": 1.0,
9
      "std/reward": 0.433,
10
11
      "mean_across_repeats/mean/reward": 0.75,
12
      "median_across_repeats/mean/reward": 0.75,
13
      "se_across_repeats/mean/reward": 0.042,
14
      "ci_low_95_across_repeats/mean/reward": 0.66,
15
      "ci_high_95_across_repeats/mean/reward": 0.84
16
    },
17
    "key_metrics": {
18
      "mean/reward": 0.75
19
    },
20
    "group_level_metrics": [
21
      {"mean/reward": 1.0, "sample": {"...": "..."}},
22
      {"mean/reward": 0.5, "sample": {"...": "..."}}
23
    ],
24
    "repeat_level_metrics": [
25
      {"_ng_rollout_index": 0, "mean/reward": 0.72, "sem/reward": 0.09, "...": "..."},
26
      {"_ng_rollout_index": 1, "mean/reward": 0.78, "sem/reward": 0.09, "...": "..."}
27
    ]
28
  }
29
]

Repeat-Level Metrics

A single run is a point estimate. Collecting repeats (--num-repeats) lets you separate a real score difference from sampling noise, and NeMo Gym reports that variability at three levels.

Within a repeat, across tasks

repeat_level_metrics holds one entry per (agent, repeat). It is produced only for agents with two or more repeats — an agent with a single repeat has nothing to compare against and is skipped. When no agent qualifies, the list is empty.

Each entry carries:

plus, for every numeric field (reward, token usage, and any numeric field your verifier returns):

Across repeats

The per-repeat mean/{field} values are themselves summarized and merged into agent_metrics, treating each repeat as one observation. This answers a different question than the keys above: not “how much do tasks vary within a repeat” but “how much does the headline score move if I run the whole benchmark again.”

Per task, across repeats

group_level_metrics reports num_rollouts, mean, median, and std per task. It deliberately carries no confidence interval: a CI here would require assuming a distribution for a single task’s repeated outcomes, which are frequently binomial rather than normal, and the Central Limit Theorem does not rescue it because these are raw outcomes rather than averages.

Two cases worth knowing about

Both emit a UserWarning, so you will see them in your terminal:

• Unequal sample counts across repeats. If a task is missing from some repeats (a crashed rollout, an interrupted run), each repeat’s statistics are computed over a different task set, so they are not directly comparable — and agent_metrics skews toward whichever tasks happened to complete. Collect the missing rollouts before drawing conclusions. Note this fires off missing_count, so it inherits the blind spot above: tasks that failed in every repeat do not trigger it.
• Zero standard error. If every value in a sample is identical, the confidence interval collapses to the single point (mean, mean). This is reported rather than left null, because SciPy’s t.interval computes an indeterminate ±inf * 0 at scale=0 and returns NaN.

Where it lands on disk

The same statistics are laid out differently depending on which command produced them:

gym eval profile always writes its file, containing [] when there is only one repeat. Either way, the cross-repeat aggregates (mean_across_repeats/mean/*, se_across_repeats/mean/*, and the CI bounds) live in agent_metrics.

Custom Metrics

Override two hooks on your resources server to add custom metrics.

compute_metrics(tasks)

Receives all verify responses grouped by task. Use this for metrics that need the full dataset — pass@k, confidence intervals, cross-task statistics.

get_key_metrics(agent_metrics)

Selects headline numbers from the final agent_metrics dict. Default returns all mean/* entries.

Example: pass@k

1
from nemo_gym.base_resources_server import (
2
    BaseVerifyRequest,
3
    BaseVerifyResponse,
4
    SimpleResourcesServer,
5
)
6
7
class MathServer(SimpleResourcesServer):
8
    async def verify(self, body: BaseVerifyRequest) -> BaseVerifyResponse:
9
        # ... verification logic ...
10
        pass
11
12
    def compute_metrics(self, tasks):
13
        n_tasks = len(tasks)
14
        # pass@k: fraction of tasks where at least one rollout got reward=1
15
        pass_at_k = sum(
16
            1 for rollouts in tasks if any(r["reward"] >= 1.0 for r in rollouts)
17
        ) / n_tasks
18
19
        # pass@1 (average of per-task mean rewards)
20
        pass_at_1 = sum(
21
            sum(r["reward"] for r in rollouts) / len(rollouts)
22
            for rollouts in tasks
23
        ) / n_tasks
24
25
        return {"pass@k": pass_at_k, "pass@1": pass_at_1}
26
27
    def get_key_metrics(self, agent_metrics):
28
        return {
29
            k: agent_metrics[k]
30
            for k in ("pass@k", "pass@1")
31
            if k in agent_metrics
32
        }

Given 3 tasks with 4 rollouts each (task 0: all correct, task 1: all wrong, task 2: half correct), this produces:

1
[
2
  {
3
    "agent_ref": {"name": "math_simple_agent"},
4
    "agent_metrics": {
5
      "mean/reward": 0.5,
6
      "max/reward": 1.0,
7
      "min/reward": 0.0,
8
      "median/reward": 0.5,
9
      "std/reward": 0.522,
10
      "pass@k": 0.667,
11
      "pass@1": 0.5
12
    },
13
    "key_metrics": {
14
      "pass@k": 0.667,
15
      "pass@1": 0.5
16
    },
17
    "group_level_metrics": [
18
      {
19
        "mean/reward": 1.0,
20
        "max/reward": 1.0,
21
        "min/reward": 1.0,
22
        "median/reward": 1.0,
23
        "std/reward": 0.0
24
      },
25
      {
26
        "mean/reward": 0.0,
27
        "max/reward": 0.0,
28
        "min/reward": 0.0,
29
        "median/reward": 0.0,
30
        "std/reward": 0.0
31
      },
32
      {
33
        "mean/reward": 0.5,
34
        "max/reward": 1.0,
35
        "min/reward": 0.0,
36
        "median/reward": 0.5,
37
        "std/reward": 0.577
38
      }
39
    ]
40
  }
41
]