nat.plugins.eval.runners.multi_eval_runner#

Classes#

MultiEvaluationRunner

Run a multi-evaluation run.

Module Contents#

class MultiEvaluationRunner(
config: nat.plugins.eval.runners.config.MultiEvaluationRunConfig,
)#

Run a multi-evaluation run.

Initialize a multi-evaluation run.

config#
evaluation_run_outputs: dict[Any, nat.data_models.evaluate_runtime.EvaluationRunOutput]#
async run_all()#

Run all evaluations defined by the overrides.

async run_single_evaluation(
id: Any,
config: nat.data_models.evaluate_runtime.EvaluationRunConfig,
) → nat.data_models.evaluate_runtime.EvaluationRunOutput#

Run a single evaluation and return the output.