> This page is for version Nightly (default).
> For other versions, use one of these documentation indexes:
> - Nightly (default): https://docs.nvidia.com/nemo/automodel/nightly/llms.txt
> - Latest: https://docs.nvidia.com/nemo/automodel/latest/llms.txt
> - 0.7.0 · 26.10: https://docs.nvidia.com/nemo/automodel/v0.7/llms.txt
> - 0.5.0 · 26.06: https://docs.nvidia.com/nemo/automodel/v0.5/llms.txt
> - 0.4.0 · 26.04: https://docs.nvidia.com/nemo/automodel/v0.4/llms.txt

> For clean Markdown of any page, append .md to the page URL.
> For a complete documentation index, see https://docs.nvidia.com/nemo/automodel/llms.txt.
> For AI client integration (Claude Code, Cursor, etc.), connect to the MCP server at https://docs.nvidia.com/nemo/automodel/_mcp/server.

# nemo_automodel.components.eval.decision

Evaluation metrics for typed decision predictions.

## Submodules

* **[`nemo_automodel.components.eval.decision.metrics`](/nemo/automodel/nemo-automodel/nemo_automodel/components/eval/decision/metrics)**

## Package Contents

### Classes

| Name                                                                                      | Description                                               |
| ----------------------------------------------------------------------------------------- | --------------------------------------------------------- |
| [`DecisionEvaluator`](#nemo_automodel-components-eval-decision-metrics-DecisionEvaluator) | Accumulates per-question predictions and summarises them. |

### API

```python
class nemo_automodel.components.eval.decision.metrics.DecisionEvaluator(
    slice_keys: typing.Sequence[str] = (),
    num_bins: int = 10
)
```

Accumulates per-question predictions and summarises them.

Call `add` once per (record, question) with the model's probabilities in the model's option order.

**`rows`** `list[dict[str, Any]] = []`

---

**`slice_keys`** `= tuple(slice_keys)`

---

```python
nemo_automodel.components.eval.decision.metrics.DecisionEvaluator._ece(
    rows: list[dict]
) -> float
```

```python
nemo_automodel.components.eval.decision.metrics.DecisionEvaluator._exact_match(
    rows: list[dict]
) -> float
```

```python
nemo_automodel.components.eval.decision.metrics.DecisionEvaluator._question_summary(
    rows: list[dict]
) -> dict[str, float]
```

```python
nemo_automodel.components.eval.decision.metrics.DecisionEvaluator.add(
    record_id: str,
    question_id: str,
    option_ids: typing.Sequence[str],
    probabilities: typing.Sequence[float],
    label: int,
    rewards: typing.Sequence[float] | None = None,
    meta: dict[str, typing.Any] | None = None
) -> None
```

Add one identified decision with probabilities in option_ids order.

**Parameters:**

**`record_id`** `str`

Input record identity, shared by its questions.

---

**`question_id`** `str`

Question identity within that record.

---

**`option_ids`** `Sequence[str]`

Allowed options in model-output order.

---

**`probabilities`** `Sequence[float]`

Probability for each allowed option, in the same order.

---

**`label`** `int`

Correct option index, or a negative value for unlabelled data.

---

**`rewards`** `Sequence[float] | None` — default: None

Optional utility for each option in the same order.

---

**`meta`** `dict[str, Any] | None` — default: None

Optional record metadata used to group evaluation slices.

---

```python
nemo_automodel.components.eval.decision.metrics.DecisionEvaluator.add_batch(
    records: typing.Sequence[dict[str, typing.Any]],
    targets_per_record: typing.Sequence[typing.Sequence[nemo_automodel.components.eval.decision.metrics.EvaluationTargetProtocol]],
    probs_per_record: typing.Sequence[typing.Sequence[typing.Sequence[float]]]
) -> None
```

Accumulate records with structurally typed targets and aligned probabilities.

**Parameters:**

**`records`** `Sequence[dict[str, Any]]`

Input records with IDs and optional slicing metadata.

---

**`targets_per_record`** `Sequence[Sequence[EvaluationTargetProtocol]]`

Targets grouped by record, in model-output order.

---

**`probs_per_record`** `Sequence[Sequence[Sequence[float]]]`

Option probabilities grouped by record and question,
in each target's option order.

---

```python
nemo_automodel.components.eval.decision.metrics.DecisionEvaluator.predictions() -> list[dict[str, typing.Any]]
```

Return the accumulated identified decisions and their aligned probabilities.

```python
nemo_automodel.components.eval.decision.metrics.DecisionEvaluator.summary() -> dict[str, typing.Any]
```

Return `{"overall": {...}, "questions": {qid: {...}}, "slices": {key: {value: {...}}}}`.