nat.eval.runners.multi_eval_runner

Classes

MultiEvaluationRunner

Run a multi-evaluation run.

Module Contents

class MultiEvaluationRunner(

config: nat.eval.runners.config.MultiEvaluationRunConfig,

)

Run a multi-evaluation run.

Initialize a multi-evaluation run.

config

evaluation_run_outputs: dict[Any, nat.eval.config.EvaluationRunOutput]

async run_all()

Run all evaluations defined by the overrides.

async run_single_evaluation(

id: Any,

config: nat.eval.config.EvaluationRunConfig,

) → nat.eval.config.EvaluationRunOutput

Run a single evaluation and return the output.