nat.plugins.eval.data_models.evaluator_io

Evaluation output models owned by the eval plugin.

Classes

EvalOutputItem	A single output item from evaluation.
EvalOutput	Container for evaluation output items.

Module Contents

class EvalOutputItem(/, **data: Any)

Bases: pydantic.BaseModel

A single output item from evaluation.

Create a new model by parsing and validating input data from keyword arguments.

Raises [ValidationError][pydantic_core.ValidationError] if the input data cannot be validated to form a valid model.

self is explicitly positional-only to allow self as a field name.

model_config

Configuration for the model, should be a dictionary conforming to [ConfigDict][pydantic.config.ConfigDict].

id: Any = None

score: Any = None

reasoning: Any = None

error: str | None = None

class EvalOutput(/, **data: Any)

Bases: pydantic.BaseModel

Container for evaluation output items.

Create a new model by parsing and validating input data from keyword arguments.

Raises [ValidationError][pydantic_core.ValidationError] if the input data cannot be validated to form a valid model.

self is explicitly positional-only to allow self as a field name.

average_score: Any = None

eval_output_items: list[pydantic.SerializeAsAny[EvalOutputItem]] = None