aiq.eval.tunable_rag_evaluator.evaluate

Attributes

logger

Classes

TunableRagEvaluator

Tunable RAG evaluator class with customizable LLM prompt for scoring.

Functions

evaluation_prompt(judge_llm_prompt, question, ...)

This function generates a prompt for the judge LLM to evaluate the generated answer.

Module Contents

logger

evaluation_prompt(

judge_llm_prompt: str,

question: str,

answer_description: str,

generated_answer: str,

format_instructions: str,

default_scoring: bool,

)

This function generates a prompt for the judge LLM to evaluate the generated answer.

class TunableRagEvaluator(

llm: langchain_core.language_models.BaseChatModel,

judge_llm_prompt: str,

max_concurrency: int,

default_scoring: bool,

default_score_weights: dict,

)

Tunable RAG evaluator class with customizable LLM prompt for scoring.

llm

max_concurrency

judge_llm_prompt

semaphore

default_scoring

default_score_weights

async evaluate(

eval_input: aiq.eval.evaluator.evaluator_model.EvalInput,

) → aiq.eval.evaluator.evaluator_model.EvalOutput

Evaluate function