nat.plugins.ragas.rag_evaluator.evaluate#

Attributes#

logger

Classes#

RAGEvaluator

Base class for custom evaluators.

Module Contents#

logger#

class RAGEvaluator( metric: ragas.metrics.base.SimpleBaseMetric, max_concurrency: int = 8, input_obj_field: str | None = None, )#

Bases: nat.plugins.eval.evaluator.base_evaluator.BaseEvaluator

Base class for custom evaluators.

Warning

Experimental Feature: The Evaluation API is experimental and may change in future releases. Future versions may introduce breaking changes without notice.

Each custom evaluator must implement the evaluate_item method which is used to evaluate a single EvalInputItem.

Initialize evaluator with a single RAGAS metric.

metric#

input_obj_field = None#

_extract_input_obj( item: nat.data_models.evaluator.EvalInputItem, ) → str#: Extracts the input object from EvalInputItem based on the configured input_obj_field.

_eval_input_item_to_ragas( item: nat.data_models.evaluator.EvalInputItem, )#: Convert one EvalInputItem into a ragas SingleTurnSample.

async evaluate_item( item: nat.data_models.evaluator.EvalInputItem, ) → nat.plugins.eval.data_models.evaluator_io.EvalOutputItem#: Run configured ragas metric for one eval item and return one output item.