nat.plugins.eval.cli.evaluate#

Attributes#

`eval_command`(→ None)	Evaluate datasets with the specified mechanism
`write_tabular_output`(eval_run_output)	Write evaluation results in a tabular format.
`_build_eval_callback_manager`(config)	Build callback manager from registered eval callbacks matching the tracing config.
`run_and_evaluate`(config)
`process_nat_eval`(processors, *, config_file, dataset, ...)	Process the eval command and execute the evaluation. Here the config_file, if provided, is checked for its existence

eval_command(ctx, \*\*kwargs) → None#: Evaluate datasets with the specified mechanism

write_tabular_output( eval_run_output: nat.data_models.evaluate_runtime.EvaluationRunOutput, )#: Write evaluation results in a tabular format.

_build_eval_callback_manager( config: nat.data_models.evaluate_runtime.EvaluationRunConfig, )#: Build callback manager from registered eval callbacks matching the tracing config.

async run_and_evaluate( config: nat.data_models.evaluate_runtime.EvaluationRunConfig, )#

process_nat_eval( processors, *, config_file: pathlib.Path, dataset: pathlib.Path, result_json_path: str, skip_workflow: bool, skip_completed_entries: bool, endpoint: str, endpoint_timeout: int, reps: int, override: tuple[tuple[str, str], Ellipsis], user_id: str | None, )#: Process the eval command and execute the evaluation. Here the config_file, if provided, is checked for its existence on disk.