nemo_evaluator.core.registry#

Module Contents#

Classes#

EvalFramworkRegistry

Singleton registry for storing framework entry classes.

Functions#

get_global_registry

register_framework

Decorator to register an entry class for evaluation.

API#

class nemo_evaluator.core.registry.EvalFramworkRegistry[source]#

Singleton registry for storing framework entry classes.

get_entries() → list[Type[nemo_evaluator.api.base.EvalFrameworkBase]][source]#
register(
framework_entry_class: Type[nemo_evaluator.api.base.EvalFrameworkBase],
) → Type[nemo_evaluator.api.base.EvalFrameworkBase][source]#
nemo_evaluator.core.registry.get_global_registry() → nemo_evaluator.core.registry.EvalFramworkRegistry[source]#
nemo_evaluator.core.registry.register_framework(
user_framework: Type[nemo_evaluator.api.base.EvalFrameworkBase],
) → Type[nemo_evaluator.api.base.EvalFrameworkBase][source]#

Decorator to register an entry class for evaluation.

Usage: @register_framework class ExampleEvalFramwork(EvalFrameworkBase): @staticmethod def parse_output(output_dir: str) -> EvaluationResult: # COMPLETE: add the parsing logic pass

    @staticmethod
    def framework_def() -> Path:
        # COMPLETE: add the path to your `framework.yml`
        pass