nat.cli.commands.evaluate

Attributes

logger

Functions

eval_command(→ None)	Evaluate datasets with the specified mechanism
write_tabular_output(eval_run_output)	Write evaluation results in a tabular format.
run_and_evaluate(config)
process_nat_eval(processors, *, config_file, dataset, ...)	Process the eval command and execute the evaluation. Here the config_file, if provided, is checked for its existence

Module Contents

logger

eval_command(ctx, \*\*kwargs) → None

Evaluate datasets with the specified mechanism

write_tabular_output(

eval_run_output: nat.eval.config.EvaluationRunOutput,

)

Write evaluation results in a tabular format.

async run_and_evaluate(config: nat.eval.evaluate.EvaluationRunConfig)

process_nat_eval(

processors,

*,

config_file: pathlib.Path,

dataset: pathlib.Path,

result_json_path: str,

skip_workflow: bool,

skip_completed_entries: bool,

endpoint: str,

endpoint_timeout: int,

reps: int,

override: tuple[tuple[str, str], Ellipsis],

user_id: str | None,

)

Process the eval command and execute the evaluation. Here the config_file, if provided, is checked for its existence on disk.