nat.plugins.ragas.rag_evaluator.atif_evaluate

Classes

RAGAtifEvaluator

Base class for ATIF-native custom evaluators.

Functions

_observation_result_to_text(→ str)
_trajectory_to_retrieved_contexts(→ list[str])

Module Contents

_observation_result_to_text(

result: nat.atif.ATIFObservationResult,

) → str

_trajectory_to_retrieved_contexts(

trajectory: nat.atif.ATIFTrajectory,

) → list[str]

class RAGAtifEvaluator(

metric: ragas.metrics.base.SimpleBaseMetric,

max_concurrency: int = 8,

)

Bases: nat.plugins.eval.evaluator.atif_base_evaluator.AtifBaseEvaluator

Base class for ATIF-native custom evaluators.

Implementers provide item-level scoring via evaluate_atif_item. This base handles bounded concurrency, gathers all items asynchronously, and computes EvalOutput.average_score from numeric per-item scores.

metric

static _atif_sample_to_ragas(

sample: nat.plugins.eval.evaluator.atif_evaluator.AtifEvalSample,

) → ragas.SingleTurnSample

Converts one ATIF sample into a ragas SingleTurnSample.

async evaluate_atif_item(

sample: nat.plugins.eval.evaluator.atif_evaluator.AtifEvalSample,

) → nat.plugins.eval.data_models.evaluator_io.EvalOutputItem

Run configured ragas metric for one ATIF sample and return one output item.