> This page is for version Nightly (default).
> For other versions, use one of these documentation indexes:
> - Nightly (default): https://docs.nvidia.com/nemo/automodel/nightly/llms.txt
> - Latest: https://docs.nvidia.com/nemo/automodel/latest/llms.txt
> - 0.7.0 · 26.10: https://docs.nvidia.com/nemo/automodel/v0.7/llms.txt
> - 0.5.0 · 26.06: https://docs.nvidia.com/nemo/automodel/v0.5/llms.txt
> - 0.4.0 · 26.04: https://docs.nvidia.com/nemo/automodel/v0.4/llms.txt

> For clean Markdown of any page, append .md to the page URL.
> For a complete documentation index, see https://docs.nvidia.com/nemo/automodel/llms.txt.
> For AI client integration (Claude Code, Cursor, etc.), connect to the MCP server at https://docs.nvidia.com/nemo/automodel/_mcp/server.

# nemo_automodel.components.eval.decision.metrics

Evaluation metrics for decision models: accuracy, joint exact match, decision reward, and calibration.

## Module Contents

### Classes

| Name                                                                                                    | Description                                               |
| ------------------------------------------------------------------------------------------------------- | --------------------------------------------------------- |
| [`DecisionEvaluator`](#nemo_automodel-components-eval-decision-metrics-DecisionEvaluator)               | Accumulates per-question predictions and summarises them. |
| [`EvaluationTargetProtocol`](#nemo_automodel-components-eval-decision-metrics-EvaluationTargetProtocol) | Read-only supervision required by this component.         |

### Functions

| Name                                                                            | Description                                             |
| ------------------------------------------------------------------------------- | ------------------------------------------------------- |
| [`save_summary`](#nemo_automodel-components-eval-decision-metrics-save_summary) | Write a `DecisionEvaluator.summary` dictionary as JSON. |

### API

```python
class nemo_automodel.components.eval.decision.metrics.DecisionEvaluator(
    slice_keys: typing.Sequence[str] = (),
    num_bins: int = 10
)
```

Accumulates per-question predictions and summarises them.

Call `add` once per (record, question) with the model's probabilities in the model's option order.

**`rows`** `list[dict[str, Any]] = []`

---

**`slice_keys`** `= tuple(slice_keys)`

---

```python
nemo_automodel.components.eval.decision.metrics.DecisionEvaluator._ece(
    rows: list[dict]
) -> float
```

```python
nemo_automodel.components.eval.decision.metrics.DecisionEvaluator._exact_match(
    rows: list[dict]
) -> float
```

```python
nemo_automodel.components.eval.decision.metrics.DecisionEvaluator._question_summary(
    rows: list[dict]
) -> dict[str, float]
```

```python
nemo_automodel.components.eval.decision.metrics.DecisionEvaluator.add(
    record_id: str,
    question_id: str,
    option_ids: typing.Sequence[str],
    probabilities: typing.Sequence[float],
    label: int,
    rewards: typing.Sequence[float] | None = None,
    meta: dict[str, typing.Any] | None = None
) -> None
```

Add one identified decision with probabilities in option_ids order.

**Parameters:**

**`record_id`** `str`

Input record identity, shared by its questions.

---

**`question_id`** `str`

Question identity within that record.

---

**`option_ids`** `Sequence[str]`

Allowed options in model-output order.

---

**`probabilities`** `Sequence[float]`

Probability for each allowed option, in the same order.

---

**`label`** `int`

Correct option index, or a negative value for unlabelled data.

---

**`rewards`** `Sequence[float] | None` — default: None

Optional utility for each option in the same order.

---

**`meta`** `dict[str, Any] | None` — default: None

Optional record metadata used to group evaluation slices.

---

```python
nemo_automodel.components.eval.decision.metrics.DecisionEvaluator.add_batch(
    records: typing.Sequence[dict[str, typing.Any]],
    targets_per_record: typing.Sequence[typing.Sequence[nemo_automodel.components.eval.decision.metrics.EvaluationTargetProtocol]],
    probs_per_record: typing.Sequence[typing.Sequence[typing.Sequence[float]]]
) -> None
```

Accumulate records with structurally typed targets and aligned probabilities.

**Parameters:**

**`records`** `Sequence[dict[str, Any]]`

Input records with IDs and optional slicing metadata.

---

**`targets_per_record`** `Sequence[Sequence[EvaluationTargetProtocol]]`

Targets grouped by record, in model-output order.

---

**`probs_per_record`** `Sequence[Sequence[Sequence[float]]]`

Option probabilities grouped by record and question,
in each target's option order.

---

```python
nemo_automodel.components.eval.decision.metrics.DecisionEvaluator.predictions() -> list[dict[str, typing.Any]]
```

Return the accumulated identified decisions and their aligned probabilities.

```python
nemo_automodel.components.eval.decision.metrics.DecisionEvaluator.summary() -> dict[str, typing.Any]
```

Return `{"overall": {...}, "questions": {qid: {...}}, "slices": {key: {value: {...}}}}`.

```python
class nemo_automodel.components.eval.decision.metrics.EvaluationTargetProtocol()
```

Protocol

Read-only supervision required by this component.

**`label`** `int`

Gold option index, or -100 when unlabelled.

---

**`option_ids`** `tuple[str, ...]`

Option identities in model-output order.

---

**`question_id`** `str`

Question identity within its record.

---

**`rewards`** `tuple[float, ...] | None`

Optional utilities aligned with the option identities.

---

```python
nemo_automodel.components.eval.decision.metrics.save_summary(
    summary: dict[str, typing.Any],
    path: str | pathlib.Path
) -> None
```

Write a `DecisionEvaluator.summary` dictionary as JSON.