nemoguardrails.evaluate.evaluate_hallucination

Module Contents

Classes

Name	Description
`HallucinationRailsEvaluation`	Helper class for running the hallucination rails evaluation for a Guardrails app.

Functions

Name	Description
`main`	Main function to run the hallucination rails evaluation.

API

class nemoguardrails.evaluate.evaluate_hallucination.HallucinationRailsEvaluation(
    config: str,
    dataset_path: str = 'data/hallucination/sample....,
    num_samples: int = 50,
    output_dir: str = 'outputs/hallucination',
    write_outputs: bool = True
)

Helper class for running the hallucination rails evaluation for a Guardrails app. It contains all the configuration parameters required to run the evaluation.

dataset

llm

= self.rails.llm

llm_task_manager

= LLMTaskManager(self.rails_config)

rails

= LLMRails(self.rails_config)

rails_config

= RailsConfig.from_path(self.config_path)

nemoguardrails.evaluate.evaluate_hallucination.HallucinationRailsEvaluation.get_extra_responses(
    prompt,
    num_responses = 2
)

Sample extra responses with temperature=1.0 from the LLM for hallucination check.

Parameters:

prompt

str

The prompt to generate extra responses for.

num_responses

intDefaults to 2

Number of extra responses to generate.

Returns:

List[str]: The list of extra responses.

nemoguardrails.evaluate.evaluate_hallucination.HallucinationRailsEvaluation.get_response_with_retries(
    prompt,
    max_tries = 1,
    llm_params = None
)

nemoguardrails.evaluate.evaluate_hallucination.HallucinationRailsEvaluation.run()

Run and print the hallucination rail evaluation.

nemoguardrails.evaluate.evaluate_hallucination.HallucinationRailsEvaluation.self_check_hallucination()

Run the hallucination rail evaluation. For each prompt, generate 2 extra responses from the LLM and check consistency with the bot response. If inconsistency is detected, flag the prompt as hallucination.

Returns:

Tuple[List[HallucinationPrediction], int]: Tuple containing hallucination predictions and the number flagged.

nemoguardrails.evaluate.evaluate_hallucination.main(
    config: str,
    data_path: str = typer.Option('data/hallucin...,
    num_samples: int = typer.Option(50, help='Numb...,
    output_dir: str = typer.Option('outputs/hallu...,
    write_outputs: bool = typer.Option(True, help='Wr...
)

Main function to run the hallucination rails evaluation.

Parameters:

config

str

The path to the config folder.

data_path

strDefaults to typer.Option('data/hallucination/sample.txt', help='Dataset path')

Dataset path.

num_samples

intDefaults to typer.Option(50, help='Number of samples to evaluate')

Number of samples to evaluate.

output_dir

strDefaults to typer.Option('outputs/hallucination', help='Output directory')

Output directory for predictions.

write_outputs

boolDefaults to typer.Option(True, help='Write outputs to file')

Whether to write the predictions to a file.