nemoguardrails.evaluate.evaluate_factcheck

Module Contents

Classes

Name	Description
`FactCheckEvaluation`	Helper class for running the fact checking evaluation for a Guardrails app.

Functions

Name	Description
`main`	-

API

class nemoguardrails.evaluate.evaluate_factcheck.FactCheckEvaluation(
    config: str,
    dataset_path: str = 'data/factchecking/sample.j...,
    num_samples: int = 50,
    create_negatives: bool = True,
    output_dir: str = 'outputs/factchecking',
    write_outputs: bool = True
)

Helper class for running the fact checking evaluation for a Guardrails app. It contains all the configuration parameters required to run the evaluation.

dataset

llm

= self.rails.llm

llm_task_manager

= LLMTaskManager(self.rails_config)

rails

= LLMRails(self.rails_config)

rails_config

= RailsConfig.from_path(self.config_path)

nemoguardrails.evaluate.evaluate_factcheck.FactCheckEvaluation.check_facts(
    split = 'positive'
)

Check facts using the fact checking rail. The fact checking rail is a binary classifier that takes in evidence and a response and predicts whether the response is grounded in the evidence or not.

Parameters:

split

strDefaults to 'positive'

The split type for checking facts. Either “positive” or “negative”.

Returns:

Tuple[List[FactCheckPrediction], int, float]: Tuple containing fact check predictions,

nemoguardrails.evaluate.evaluate_factcheck.FactCheckEvaluation.create_negative_samples(
    dataset
)

async

Create synthetic negative samples for fact checking. The negative samples are created by an LLM that acts as an adversary and modifies the answer to make it incorrect.

Parameters:

dataset

List[Dict]

The dataset to create negative samples for.

Returns:

List[Dict]: The dataset with synthetic negative samples.

nemoguardrails.evaluate.evaluate_factcheck.FactCheckEvaluation.run()

Run the fact checking evaluation and print the results.

nemoguardrails.evaluate.evaluate_factcheck.main(
    config: str,
    dataset_path: str = typer.Option('./data/factch...,
    num_samples: int = typer.Option(50, help='Numb...,
    create_negatives: bool = typer.Option(True, help='cr...,
    output_dir: str = typer.Option('eval_outputs/...,
    write_outputs: bool = typer.Option(True, help='Wr...
)