nat.data_models.evaluator

Evaluation input data models and evaluator configs.

Attributes

EvaluatorBaseConfigT

Classes

EvaluatorBaseConfig	Subclass of Pydantic BaseModel that allows for specifying the object type. Use in Pydantic discriminated unions.
EvaluatorLLMConfig	Base config for evaluators that use an LLM as a judge.
EvalInputItem	A single input item for evaluation.
EvalInput	Container for evaluation input items.
EvalOutputItemLike	Structural contract for a single evaluation output item.
EvalOutputLike	Structural contract for a collection of evaluation output items.

Module Contents

class EvaluatorBaseConfig(/, **data: Any)

Bases: nat.data_models.common.TypedBaseModel, nat.data_models.common.BaseModelRegistryTag

Subclass of Pydantic BaseModel that allows for specifying the object type. Use in Pydantic discriminated unions.

Create a new model by parsing and validating input data from keyword arguments.

Raises [ValidationError][pydantic_core.ValidationError] if the input data cannot be validated to form a valid model.

self is explicitly positional-only to allow self as a field name.

class EvaluatorLLMConfig(/, **data: Any)

Bases: EvaluatorBaseConfig, nat.data_models.retry_mixin.RetryMixin

Base config for evaluators that use an LLM as a judge.

Create a new model by parsing and validating input data from keyword arguments.

Raises [ValidationError][pydantic_core.ValidationError] if the input data cannot be validated to form a valid model.

self is explicitly positional-only to allow self as a field name.

llm_name: str = None

EvaluatorBaseConfigT

class EvalInputItem(/, **data: Any)

Bases: pydantic.BaseModel

A single input item for evaluation.

Create a new model by parsing and validating input data from keyword arguments.

Raises [ValidationError][pydantic_core.ValidationError] if the input data cannot be validated to form a valid model.

self is explicitly positional-only to allow self as a field name.

id: Any = None

input_obj: Any = None

expected_output_obj: Any = None

output_obj: Any = None

expected_trajectory: list[nat.data_models.intermediate_step.IntermediateStep] = None

trajectory: list[nat.data_models.intermediate_step.IntermediateStep] = None

full_dataset_entry: Any = None

copy_with_updates(**updates) → EvalInputItem

Copy EvalInputItem with optional field updates.

class EvalInput(/, **data: Any)

Bases: pydantic.BaseModel

Container for evaluation input items.

Create a new model by parsing and validating input data from keyword arguments.

Raises [ValidationError][pydantic_core.ValidationError] if the input data cannot be validated to form a valid model.

self is explicitly positional-only to allow self as a field name.

eval_input_items: list[EvalInputItem] = None

class EvalOutputItemLike

Bases: Protocol

Structural contract for a single evaluation output item.

id: Any

score: Any

reasoning: Any

error: str | None

class EvalOutputLike

Bases: Protocol

Structural contract for a collection of evaluation output items.

average_score: Any

eval_output_items: collections.abc.Sequence[EvalOutputItemLike]