aiq.eval.config

Classes

EvaluationRunConfig	Parameters used for a single evaluation run.
EvaluationRunOutput	Output of a single evaluation run.

Module Contents

class EvaluationRunConfig(/, **data: Any)

Bases: pydantic.BaseModel

Parameters used for a single evaluation run.

Create a new model by parsing and validating input data from keyword arguments.

Raises [ValidationError][pydantic_core.ValidationError] if the input data cannot be validated to form a valid model.

self is explicitly positional-only to allow self as a field name.

config_file: pathlib.Path

dataset: str | None

result_json_path: str

skip_workflow: bool

skip_completed_entries: bool

endpoint: str | None

endpoint_timeout: int

reps: int

class EvaluationRunOutput(/, **data: Any)

Bases: pydantic.BaseModel

Output of a single evaluation run.

Create a new model by parsing and validating input data from keyword arguments.

Raises [ValidationError][pydantic_core.ValidationError] if the input data cannot be validated to form a valid model.

self is explicitly positional-only to allow self as a field name.

workflow_output_file: pathlib.Path | None

evaluator_output_files: list[pathlib.Path]

workflow_interrupted: bool