nat.eval.trajectory_evaluator.evaluate

Attributes

logger

Classes

TrajectoryEvaluator

Base class for custom evaluators.

Module Contents

logger

class TrajectoryEvaluator(

llm: langchain_core.language_models.BaseChatModel,

tools: list[langchain_core.tools.BaseTool] | None = None,

max_concurrency: int = 8,

)

Bases: nat.eval.evaluator.base_evaluator.BaseEvaluator

Base class for custom evaluators.

Warning

Experimental Feature: The Evaluation API is experimental and may change in future releases. Future versions may introduce breaking changes without notice.

Each custom evaluator must implement the evaluate_item method which is used to evaluate a single EvalInputItem.

llm

tools = None

traj_eval_chain

async evaluate_item(

item: nat.eval.evaluator.evaluator_model.EvalInputItem,

) → nat.eval.evaluator.evaluator_model.EvalOutputItem

Evaluate a single EvalInputItem and return an EvalOutputItem.