nat.plugins.eval.evaluator.base_evaluator#

Attributes#

`_tqdm`
`logger`

Classes#

`_NoOpProgressBar`
`BaseEvaluator`	Base class for custom evaluators.

Functions#

_make_progress_bar(→ tuple[Any, int | None])

Module Contents#

_tqdm = None#

logger#

class _NoOpProgressBar#

update(_: int) → None#

close() → None#

_make_progress_bar(total: int, desc: str) → tuple[Any, int | None]#

class BaseEvaluator(max_concurrency: int = 4, tqdm_desc: str = 'Evaluating')#

Bases: abc.ABC

Base class for custom evaluators.

Warning

Experimental Feature: The Evaluation API is experimental and may change in future releases. Future versions may introduce breaking changes without notice.

Each custom evaluator must implement the evaluate_item method which is used to evaluate a single EvalInputItem.

max_concurrency = 4#

semaphore#

tqdm_desc = 'Evaluating'#

abstractmethod evaluate_item( item: nat.data_models.evaluator.EvalInputItem, ) → nat.plugins.eval.data_models.evaluator_io.EvalOutputItem#

Async:

Each evaluator must implement this for item-level evaluation

async evaluate( eval_input: nat.data_models.evaluator.EvalInput, ) → nat.plugins.eval.data_models.evaluator_io.EvalOutput#