nemo_evaluator.api.base#

Module Contents#

Classes#

EvalFrameworkBase

When subclassing, register with @register_framework decorator

API#

class nemo_evaluator.api.base.EvalFrameworkBase[source]#

Bases: abc.ABC

When subclassing, register with @register_framework decorator

register_framework is imported from nemo_evaluator.api.run

abstractmethod static framework_def() → pathlib.Path[source]#

Path to the yml definition of the tasks.

The file framework.yml contains the definition of tasks, default parameters, and other crucial properties of the harness. It’s documented in the contributing guide and, for quick bootstrapping, can be created by nemo_evaluator_example command that is installed along with nemo_evaluator.

abstractmethod static parse_output(
output_dir: str,
) → nemo_evaluator.api.api_dataclasses.EvaluationResult[source]#

Parser of the harness output into Nvidia Eval result type.

Defining this method ensures that Nvidia Eval Commons can translate the harness’s output into the dataclass understandable by the library.