nat.plugins.eval.evaluator.atif_evaluator#

ATIF-native evaluator protocol definitions.

Attributes#

AtifEvalSampleList

Classes#

AtifEvalSample

ATIF-native evaluation sample used by ATIF-backed evaluators.

AtifEvaluator

Protocol for evaluators that consume ATIF-native samples.

LegacyEvaluator

Protocol for evaluators that consume legacy EvalInput payloads.

Module Contents#

class AtifEvalSample(/, **data: Any)#

Bases: pydantic.BaseModel

ATIF-native evaluation sample used by ATIF-backed evaluators.

Create a new model by parsing and validating input data from keyword arguments.

Raises [ValidationError][pydantic_core.ValidationError] if the input data cannot be validated to form a valid model.

self is explicitly positional-only to allow self as a field name.

item_id: Any = None#
trajectory: nat.atif.ATIFTrajectory = None#
expected_output_obj: Any = None#
output_obj: Any = None#
metadata: dict[str, Any] = None#
AtifEvalSampleList#
class AtifEvaluator#

Bases: Protocol

Protocol for evaluators that consume ATIF-native samples.

async evaluate_atif_fn(
atif_samples: AtifEvalSampleList,
) → nat.plugins.eval.data_models.evaluator_io.EvalOutput#

Evaluate using ATIF-native sample payloads.

class LegacyEvaluator#

Bases: Protocol

Protocol for evaluators that consume legacy EvalInput payloads.

async evaluate_fn(
eval_input,
) → nat.plugins.eval.data_models.evaluator_io.EvalOutput#

Evaluate using legacy eval input payloads.