aiq.eval.config

Classes

EvaluationRunConfig	Parameters used for a single evaluation run.
EvaluationRunOutput	Output of a single evaluation run.

Module Contents

class EvaluationRunConfig(/, **data: Any)

Bases: pydantic.BaseModel

Parameters used for a single evaluation run.

Create a new model by parsing and validating input data from keyword arguments.

Raises [ValidationError][pydantic_core.ValidationError] if the input data cannot be validated to form a valid model.

self is explicitly positional-only to allow self as a field name.

config_file: pathlib.Path

dataset: str | None

result_json_path: str = '$'

skip_workflow: bool = False

skip_completed_entries: bool = False

endpoint: str | None = None

endpoint_timeout: int = 300

reps: int = 1

override: tuple[tuple[str, str], Ellipsis] = ()

class EvaluationRunOutput(/, **data: Any)

Bases: pydantic.BaseModel

Output of a single evaluation run.

Create a new model by parsing and validating input data from keyword arguments.

Raises [ValidationError][pydantic_core.ValidationError] if the input data cannot be validated to form a valid model.

self is explicitly positional-only to allow self as a field name.

workflow_output_file: pathlib.Path | None

evaluator_output_files: list[pathlib.Path]

workflow_interrupted: bool