nat.eval.runners.config

Classes

MultiEvaluationRunConfig	Parameters used for a multi-evaluation run.
MultiEvaluationRunOutput	Output of a multi-evaluation run.

Module Contents

class MultiEvaluationRunConfig(/, **data: Any)

Bases: pydantic.BaseModel

Parameters used for a multi-evaluation run. This includes a dict of configs. The key is an id of any type. Each pass loads the config, applies the overrides and runs to completion before the next pass starts.

Create a new model by parsing and validating input data from keyword arguments.

Raises [ValidationError][pydantic_core.ValidationError] if the input data cannot be validated to form a valid model.

self is explicitly positional-only to allow self as a field name.

configs: dict[Any, nat.eval.config.EvaluationRunConfig]

class MultiEvaluationRunOutput(/, **data: Any)

Bases: pydantic.BaseModel

Output of a multi-evaluation run. The results per-pass are accumulated in the evaluation_run_outputs dict.

Create a new model by parsing and validating input data from keyword arguments.

Raises [ValidationError][pydantic_core.ValidationError] if the input data cannot be validated to form a valid model.

self is explicitly positional-only to allow self as a field name.

evaluation_run_outputs: dict[Any, nat.eval.config.EvaluationRunOutput]