nemoguardrails.eval.check

Module Contents

Classes

Name	Description
`LLMJudgeComplianceChecker`	LLM Judge compliance checker.

Data

executor

API

class nemoguardrails.eval.check.LLMJudgeComplianceChecker(
    eval_config_path: str,
    output_paths: typing.List[str],
    llm_judge_model: typing.Optional[str] = None,
    policy_ids: typing.Optional[typing.List[str]] = None,
    verbose: bool = False,
    force: bool = False,
    reset: bool = False,
    parallel: int = 1
)

LLM Judge compliance checker.

eval_config

= EvalConfig.from_path(eval_config_path)

eval_config_path

= os.path.abspath(self.eval_config_path)

eval_data

llm

= model_cls(**kwargs)

llm_task_manager

= LLMTaskManager(config=_config)

policies

= self.eval_config.policies

policy_by_id

= {}

progress_idx

= 0

nemoguardrails.eval.check.LLMJudgeComplianceChecker.check_interaction_compliance(
    interaction_output: nemoguardrails.eval.models.InteractionOutput,
    interaction_log: nemoguardrails.eval.models.InteractionLog,
    interaction_set: nemoguardrails.eval.models.InteractionSet,
    progress_idx: int
) -> bool

async

Check the compliance for the provided interaction.

The interaction output and log are updated in accordance with the check.

Parameters:

interaction_output

InteractionOutput

The output from the interaction.

interaction_log

InteractionLog

The detailed log for the interaction.

interaction_set

InteractionSet

The corresponding interaction set.

progress_idx

int

The index in the processing

Returns: bool

True if there were any changes.

nemoguardrails.eval.check.LLMJudgeComplianceChecker.print_completion(
    completion: str
)

Helper to print a completion.

nemoguardrails.eval.check.LLMJudgeComplianceChecker.print_progress_detail(
    text: typing.Union[str, rich.text.Text]
)

Helper to print detailed progress.

nemoguardrails.eval.check.LLMJudgeComplianceChecker.print_prompt(
    prompt: str
)

Helper for printing a prompt.

nemoguardrails.eval.check.LLMJudgeComplianceChecker.run()

async

Run the compliance check.

nemoguardrails.eval.check.executor = ThreadPoolExecutor(max_workers=1)