> For clean Markdown of any page, append .md to the page URL.
> For a complete documentation index, see https://docs.nvidia.com/nemo/gym/llms.txt.
> For full documentation content, see https://docs.nvidia.com/nemo/gym/llms-full.txt.
> For AI client integration (Claude Code, Cursor, etc.), connect to the MCP server at https://docs.nvidia.com/nemo/gym/_mcp/server.

# nemo_gym.reward_profile

## Module Contents

### Classes

| Name                                                                      | Description                                                                                   |
| ------------------------------------------------------------------------- | --------------------------------------------------------------------------------------------- |
| [`AggregateMetricsMixin`](#nemo_gym-reward_profile-AggregateMetricsMixin) | Mixin providing compute\_metrics/get\_key\_metrics hooks and the aggregate\_metrics endpoint. |
| [`RewardProfileConfig`](#nemo_gym-reward_profile-RewardProfileConfig)     | -                                                                                             |
| [`RewardProfiler`](#nemo_gym-reward_profile-RewardProfiler)               | -                                                                                             |

### Functions

| Name                                                                                      | Description                                                                                  |
| ----------------------------------------------------------------------------------------- | -------------------------------------------------------------------------------------------- |
| [`_group_by_task`](#nemo_gym-reward_profile-_group_by_task)                               | Group verify responses by task index, returning a list of per-task rollout lists.            |
| [`_rollout_key`](#nemo_gym-reward_profile-_rollout_key)                                   | -                                                                                            |
| [`add_avg_sample_std_dev`](#nemo_gym-reward_profile-add_avg_sample_std_dev)               | Add avg\_sample\_std\_dev statistics to an existing metrics dict.                            |
| [`compute_aggregate_metrics`](#nemo_gym-reward_profile-compute_aggregate_metrics)         | Shared aggregation logic for /aggregate\_metrics.                                            |
| [`compute_pass_majority_metrics`](#nemo_gym-reward_profile-compute_pass_majority_metrics) | Compute pass\@k, majority\@k, no\_answer, and variance statistics from grouped task results. |
| [`compute_subset_metrics`](#nemo_gym-reward_profile-compute_subset_metrics)               | Group tasks by a field and compute pass\@k metrics per subset.                               |
| [`highest_k_metrics`](#nemo_gym-reward_profile-highest_k_metrics)                         | Select the highest-k entries matching a metric pattern.                                      |
| [`reward_profile`](#nemo_gym-reward_profile-reward_profile)                               | -                                                                                            |

### API

```python
class nemo_gym.reward_profile.AggregateMetricsMixin()
```

Mixin providing compute\_metrics/get\_key\_metrics hooks and the aggregate\_metrics endpoint.

Inherited by both SimpleResourcesServer and SimpleResponsesAPIAgent so that
benchmark-specific metric logic can live on either server type.

```python
nemo_gym.reward_profile.AggregateMetricsMixin.compute_metrics(
    tasks: typing.List[typing.List[typing.Dict[str, typing.Any]]]
) -> typing.Dict[str, typing.Any]
```

Override to compute custom metrics from all verify responses.

Receives verify responses grouped by task: tasks\[i] is a list of rollout
dicts for task i. Each dict has at minimum reward, plus any custom fields
from the verify response (e.g. symbolic\_correct, judgement-gen-base).

Use for metrics that need the full dataset at once:

* Confidence intervals (ArenaMetrics)
* Cross-task statistics (std\_dev\_across\_runs)
* pass\@k with proper combinatorial computation

The returned dict is merged into agent\_metrics.
Default: empty dict (no additional metrics).

```python
nemo_gym.reward_profile.AggregateMetricsMixin.get_key_metrics(
    agent_metrics: typing.Dict[str, typing.Any]
) -> typing.Dict[str, typing.Any]
```

Override to select headline metrics for this benchmark.

Default: all mean/\* entries from agent\_metrics.

```python
class nemo_gym.reward_profile.RewardProfileConfig()
```

**Bases:** [BaseNeMoGymCLIConfig](/nemo-gym/nemo_gym/config_types#nemo_gym-config_types-BaseNeMoGymCLIConfig)

```python
class nemo_gym.reward_profile.RewardProfiler()
```

```python
nemo_gym.reward_profile.RewardProfiler._index_by_rollout_key(
    rows: typing.List[typing.Dict[str, typing.Any]],
    name: str
) -> typing.Dict[typing.Tuple[int, int], typing.Dict[str, typing.Any]]
```

```python
nemo_gym.reward_profile.RewardProfiler.align_rows_and_results(
    rows: typing.List[typing.Dict[str, typing.Any]],
    results: typing.List[typing.Dict[str, typing.Any]],
    allow_partial_rollouts: bool = False
) -> typing.List[typing.Tuple[typing.Dict[str, typing.Any], typing.Dict[str, typing.Any]]]
```

```python
nemo_gym.reward_profile.RewardProfiler.calculate_metrics_single_df(
    grouped_df: pandas.core.groupby.generic.DataFrameGroupBy
) -> typing.List[typing.Dict[str, typing.Any]]
```

```python
nemo_gym.reward_profile.RewardProfiler.describe_dataframe(
    df: pandas.DataFrame
) -> pandas.DataFrame
```

```python
nemo_gym.reward_profile.RewardProfiler.histogram(
    data: pandas.Series
) -> typing.Optional[wandb.Histogram]
```

```python
nemo_gym.reward_profile.RewardProfiler.prepare_for_serialization(
    metrics: typing.List[typing.Dict]
) -> typing.List[typing.Dict]
```

Non-destructively cleans metrics output by RewardProfiler for downstream serialization.

```python
nemo_gym.reward_profile.RewardProfiler.profile_completion_summary(
    rows: typing.List[typing.Dict[str, typing.Any]],
    results: typing.List[typing.Dict[str, typing.Any]]
) -> typing.Dict[str, typing.Any]
```

```python
nemo_gym.reward_profile.RewardProfiler.profile_from_data(
    rows: typing.List[typing.Dict[str, typing.Any]],
    results: typing.List[typing.Dict[str, typing.Any]],
    allow_partial_rollouts: bool = False
) -> typing.Tuple[typing.List[typing.Dict[str, typing.Any]], typing.List[typing.Dict[str, typing.Any]]]
```

```python
nemo_gym.reward_profile.RewardProfiler.rollout_info_from_result(
    result: typing.Dict[str, typing.Any]
) -> typing.Dict[str, typing.Any]
```

```python
nemo_gym.reward_profile.RewardProfiler.write_to_disk(
    group_level_metrics: typing.List[typing.Dict[str, typing.Any]],
    agent_level_metrics: typing.List[typing.Dict[str, typing.Any]],
    base_output_fpath: pathlib.Path
) -> typing.Tuple[pathlib.Path, pathlib.Path]
```

```python
nemo_gym.reward_profile._group_by_task(
    verify_responses: typing.List[typing.Dict[str, typing.Any]]
) -> typing.List[typing.List[typing.Dict[str, typing.Any]]]
```

Group verify responses by task index, returning a list of per-task rollout lists.

```python
nemo_gym.reward_profile._rollout_key(
    row: typing.Dict[str, typing.Any]
) -> typing.Tuple[int, int]
```

```python
nemo_gym.reward_profile.add_avg_sample_std_dev(
    metrics: typing.Dict[str, typing.Any],
    all_score_dicts: typing.List[typing.List[typing.Dict[str, float]]],
    score_names: list,
    max_k: int
) -> None
```

Add avg\_sample\_std\_dev statistics to an existing metrics dict.

Computes the average of per-task standard deviations across k rollouts — a measure of
within-task variance that complements the across-run variance (std\_dev\_across\_runs).

Modifies `metrics` in place.

```python
nemo_gym.reward_profile.compute_aggregate_metrics(
    verify_responses: typing.List[typing.Dict[str, typing.Any]],
    compute_metrics_fn = None,
    get_key_metrics_fn = None
) -> nemo_gym.config_types.AggregateMetrics
```

Shared aggregation logic for /aggregate\_metrics.

RewardProfiler runs with defaults to produce baseline stats (mean/max/min/median/std)
for both group-level (per-task) and agent-level metrics.

```python
nemo_gym.reward_profile.compute_pass_majority_metrics(
    tasks: typing.List[typing.List[typing.Dict[str, typing.Any]]],
    score_fn: typing.Optional[typing.Any] = None,
    answer_key: typing.Optional[str] = None
) -> typing.Tuple[typing.Dict[str, typing.Any], typing.List[typing.List[typing.Dict[str, float]]], typing.List[str], int]
```

Compute pass\@k, majority\@k, no\_answer, and variance statistics from grouped task results.

Shared utility for any resource server's compute\_metrics() override.

**Parameters:**

tasks\[i] is a list of rollout dicts for task i.

Callable(result\_dict) -> Dict\[str, float|bool] returning named scores.
Defaults to `lambda r: {"accuracy": r["reward"]}`.

Field name for extracted answer (enables majority\@k and no\_answer).
If None, majority\@k and no\_answer are skipped.

**Returns:** `Dict[str, Any]`

Metrics, all\_score\_dicts, score\_names, max\_k

```python
nemo_gym.reward_profile.compute_subset_metrics(
    tasks: typing.List[typing.List[typing.Dict[str, typing.Any]]],
    subset_key: str,
    score_fn: typing.Optional[typing.Any] = None,
    answer_key: typing.Optional[str] = None
) -> typing.Dict[str, typing.Any]
```

Group tasks by a field and compute pass\@k metrics per subset.

Returns flat dict with subset-prefixed keys, e.g. `"easy/pass@1/accuracy"`.
Skips the `per_sample_aggregate` key from each subset's metrics.

**Parameters:**

tasks\[i] is a list of rollout dicts for task i.

Field name in rollout dicts to group by (e.g. `"difficulty"`).

Passed through to `compute_pass_majority_metrics`.

Passed through to `compute_pass_majority_metrics`.

```python
nemo_gym.reward_profile.highest_k_metrics(
    agent_metrics: typing.Dict[str, typing.Any],
    pattern: str,
    score_names: typing.Optional[typing.List[str]] = None,
    exclude_names: typing.Optional[typing.List[str]] = None
) -> typing.Dict[str, typing.Any]
```

Select the highest-k entries matching a metric pattern.

Finds all keys matching `pattern` (with `{k}` as the k placeholder), determines the
highest k value, and returns all entries at that k.

Example::

# Get highest-k pass\@k for accuracy only

highest\_k\_metrics(am, "pass@\{k}", score\_names=\["accuracy"])

# → \{"pass\@32/accuracy": 95.0}

# Get highest-k pass\@1\[avg-of-k] for all scores except no\_answer, without stats

highest\_k\_metrics(am, "pass\@1\[avg-of-\{k}]", exclude\_names=\["no\_answer"])

# → \{"pass\@1\[avg-of-32]/accuracy": 94.5, "pass\@1\[avg-of-32]/symbolic\_accuracy": 93.2}

**Parameters:**

Full agent metrics dict.

Pattern with `{k}` placeholder, e.g. `"pass@{k}"` or `"pass@1[avg-of-{k}]"`.

If provided, only return entries whose score name (after the last `/`)
is in this list. Stat suffixes (std\_dev, std\_err, avg\_sample) are always excluded.

Score names to exclude (e.g. `["no_answer"]`). Applied after score\_names.

**Returns:** `Dict[str, Any]`

Dict of matching metrics at the highest k, e.g. `{"pass@32/accuracy": 95.0}`.

```python
nemo_gym.reward_profile.reward_profile()
```