nemo_automodel.components.eval.decision.metrics

Evaluation metrics for decision models: accuracy, joint exact match, decision reward, and calibration.

Module Contents

Classes

Name	Description
`DecisionEvaluator`	Accumulates per-question predictions and summarises them.
`EvaluationTargetProtocol`	Read-only supervision required by this component.

Functions

Name	Description
`save_summary`	Write a `DecisionEvaluator.summary` dictionary as JSON.

API

class nemo_automodel.components.eval.decision.metrics.DecisionEvaluator(
    slice_keys: typing.Sequence[str] = (),
    num_bins: int = 10
)

Accumulates per-question predictions and summarises them.

Call add once per (record, question) with the model’s probabilities in the model’s option order.

rows

list[dict[str, Any]] = []

slice_keys

= tuple(slice_keys)

nemo_automodel.components.eval.decision.metrics.DecisionEvaluator._ece(
    rows: list[dict]
) -> float

nemo_automodel.components.eval.decision.metrics.DecisionEvaluator._exact_match(
    rows: list[dict]
) -> float

nemo_automodel.components.eval.decision.metrics.DecisionEvaluator._question_summary(
    rows: list[dict]
) -> dict[str, float]

nemo_automodel.components.eval.decision.metrics.DecisionEvaluator.add(
    record_id: str,
    question_id: str,
    option_ids: typing.Sequence[str],
    probabilities: typing.Sequence[float],
    label: int,
    rewards: typing.Sequence[float] | None = None,
    meta: dict[str, typing.Any] | None = None
) -> None

Add one identified decision with probabilities in option_ids order.

Parameters:

record_id

str

Input record identity, shared by its questions.

question_id

str

Question identity within that record.

option_ids

Sequence[str]

Allowed options in model-output order.

probabilities

Sequence[float]

Probability for each allowed option, in the same order.

label

int

Correct option index, or a negative value for unlabelled data.

rewards

Sequence[float] | NoneDefaults to None

Optional utility for each option in the same order.

meta

dict[str, Any] | NoneDefaults to None

Optional record metadata used to group evaluation slices.

nemo_automodel.components.eval.decision.metrics.DecisionEvaluator.add_batch(
    records: typing.Sequence[dict[str, typing.Any]],
    targets_per_record: typing.Sequence[typing.Sequence[nemo_automodel.components.eval.decision.metrics.EvaluationTargetProtocol]],
    probs_per_record: typing.Sequence[typing.Sequence[typing.Sequence[float]]]
) -> None

Accumulate records with structurally typed targets and aligned probabilities.

Parameters:

records

Sequence[dict[str, Any]]

Input records with IDs and optional slicing metadata.

targets_per_record

Sequence[Sequence[EvaluationTargetProtocol]]

Targets grouped by record, in model-output order.

probs_per_record

Sequence[Sequence[Sequence[float]]]

Option probabilities grouped by record and question, in each target’s option order.

nemo_automodel.components.eval.decision.metrics.DecisionEvaluator.predictions() -> list[dict[str, typing.Any]]

Return the accumulated identified decisions and their aligned probabilities.

nemo_automodel.components.eval.decision.metrics.DecisionEvaluator.summary() -> dict[str, typing.Any]

Return {"overall": {...}, "questions": {qid: {...}}, "slices": {key: {value: {...}}}}.

class nemo_automodel.components.eval.decision.metrics.EvaluationTargetProtocol()

Protocol

Read-only supervision required by this component.

label

int

Gold option index, or -100 when unlabelled.

option_ids

tuple[str, ...]

Option identities in model-output order.

question_id

str

Question identity within its record.

rewards

tuple[float, ...] | None

Optional utilities aligned with the option identities.

nemo_automodel.components.eval.decision.metrics.save_summary(
    summary: dict[str, typing.Any],
    path: str | pathlib.Path
) -> None

Write a DecisionEvaluator.summary dictionary as JSON.