nat.data_models.evaluator

Evaluation input/output data models shared across core and eval.

Attributes

EvaluatorBaseConfigT

Classes

EvaluatorBaseConfig
EvaluatorLLMConfig	Base config for evaluators that use an LLM as a judge.
EvalInputItem	A single input item for evaluation.
EvalInput	Container for evaluation input items.
EvalOutputItem	A single output item from evaluation.
EvalOutput	Container for evaluation output items.

Module Contents

class EvaluatorBaseConfig

Bases: nat.data_models.common.TypedBaseModel, nat.data_models.common.BaseModelRegistryTag

class EvaluatorLLMConfig

Bases: EvaluatorBaseConfig, nat.data_models.retry_mixin.RetryMixin

Base config for evaluators that use an LLM as a judge.

llm_name: str = None

EvaluatorBaseConfigT

class EvalInputItem(/, **data: Any)

Bases: pydantic.BaseModel

A single input item for evaluation.

Create a new model by parsing and validating input data from keyword arguments.

Raises [ValidationError][pydantic_core.ValidationError] if the input data cannot be validated to form a valid model.

self is explicitly positional-only to allow self as a field name.

id: Any = None

input_obj: Any = None

expected_output_obj: Any = None

output_obj: Any = None

expected_trajectory: list[nat.data_models.intermediate_step.IntermediateStep] = None

trajectory: list[nat.data_models.intermediate_step.IntermediateStep] = None

full_dataset_entry: Any = None

copy_with_updates(**updates) → EvalInputItem

Copy EvalInputItem with optional field updates.

class EvalInput(/, **data: Any)

Bases: pydantic.BaseModel

Container for evaluation input items.

Create a new model by parsing and validating input data from keyword arguments.

Raises [ValidationError][pydantic_core.ValidationError] if the input data cannot be validated to form a valid model.

self is explicitly positional-only to allow self as a field name.

eval_input_items: list[EvalInputItem] = None

class EvalOutputItem(/, **data: Any)

Bases: pydantic.BaseModel

A single output item from evaluation.

Create a new model by parsing and validating input data from keyword arguments.

Raises [ValidationError][pydantic_core.ValidationError] if the input data cannot be validated to form a valid model.

self is explicitly positional-only to allow self as a field name.

model_config

Configuration for the model, should be a dictionary conforming to [ConfigDict][pydantic.config.ConfigDict].

id: Any = None

score: Any = None

reasoning: Any = None

error: str | None = None

class EvalOutput(/, **data: Any)

Bases: pydantic.BaseModel

Container for evaluation output items.

Create a new model by parsing and validating input data from keyword arguments.

Raises [ValidationError][pydantic_core.ValidationError] if the input data cannot be validated to form a valid model.

self is explicitly positional-only to allow self as a field name.

average_score: Any = None

eval_output_items: list[pydantic.SerializeAsAny[EvalOutputItem]] = None