nat.eval.evaluator.base_evaluator

Classes

BaseEvaluator

Base class for custom evaluators.

Module Contents

class BaseEvaluator(max_concurrency: int = 4, tqdm_desc: str = 'Evaluating')

Bases: abc.ABC

Base class for custom evaluators.

Warning

Experimental Feature: The Evaluation API is experimental and may change in future releases. Future versions may introduce breaking changes without notice.

Each custom evaluator must implement the evaluate_item method which is used to evaluate a single EvalInputItem.

max_concurrency = 4

semaphore

tqdm_desc = 'Evaluating'

abstractmethod evaluate_item(

item: nat.eval.evaluator.evaluator_model.EvalInputItem,

) → nat.eval.evaluator.evaluator_model.EvalOutputItem

Async:

Each evaluator must implement this for item-level evaluation

async evaluate(

eval_input: nat.eval.evaluator.evaluator_model.EvalInput,

) → nat.eval.evaluator.evaluator_model.EvalOutput