aiq.cli.commands.evaluate#

Attributes#

logger

Functions#

eval_command(→ None)

Evaluate datasets with the specified mechanism

run_and_evaluate(config)

process_aiq_eval(processors, *, config_file, dataset, ...)

Process the eval command and execute the evaluation. Here the config_file, if provided, is checked for its existence

Module Contents#

logger#
eval_command(ctx, **kwargs) → None#

Evaluate datasets with the specified mechanism

async run_and_evaluate(config: aiq.eval.evaluate.EvaluationRunConfig)#
process_aiq_eval(
processors,
*,
config_file: pathlib.Path,
dataset: pathlib.Path,
result_json_path: str,
skip_workflow: bool,
skip_completed_entries: bool,
endpoint: str,
endpoint_timeout: int,
reps: int,
)#

Process the eval command and execute the evaluation. Here the config_file, if provided, is checked for its existence on disk.