ReferencePython SDK ReferenceNemoguardrailsEvaluate

nemoguardrails.evaluate.evaluate_topical

View as Markdown

Module Contents

Classes

NameDescription
TopicalRailsEvaluationHelper class for running the topical rails evaluation for a Guardrails app.

Functions

NameDescription
_split_test_set_from_configExtracts a test set of user messages from a config.
cosine_similarityCompute the dot product between two embeddings using numpy functions.
sync_wrapperWrapper for the evaluate_topical_rails method which is async.

API

class nemoguardrails.evaluate.evaluate_topical.TopicalRailsEvaluation(
    config: str,
    verbose: typing.Optional[bool] = False,
    test_set_percentage: typing.Optional[float] = 0.3,
    max_tests_per_intent: typing.Optional[int] = 3,
    max_samples_per_intent: typing.Optional[int] = 0,
    print_test_results_frequency: typing.Optional[int] = 10,
    similarity_threshold: typing.Optional[float] = 0.0,
    random_seed: typing.Optional[int] = None,
    output_dir: typing.Optional[str] = None
)

Helper class for running the topical rails evaluation for a Guardrails app. It contains all the configuration parameters required to run the evaluation.

nemoguardrails.evaluate.evaluate_topical.TopicalRailsEvaluation._compute_intent_embeddings(
    intents
)

Compute intent embeddings if we have a sentence transformer model.

nemoguardrails.evaluate.evaluate_topical.TopicalRailsEvaluation._get_main_llm_model()
nemoguardrails.evaluate.evaluate_topical.TopicalRailsEvaluation._get_most_similar_intent(
    generated_intent
)

Retrieves the most similar intent using sentence transformers embeddings. If the most similar intent is below the similarity threshold, the generated intent is not changed.

nemoguardrails.evaluate.evaluate_topical.TopicalRailsEvaluation._initialize_embeddings_model()

Instantiate a sentence transformer if we use a similarity check for canonical forms.

nemoguardrails.evaluate.evaluate_topical.TopicalRailsEvaluation._initialize_rails_app()
nemoguardrails.evaluate.evaluate_topical.TopicalRailsEvaluation._initialize_random_seed()

Initialize random seed

nemoguardrails.evaluate.evaluate_topical.TopicalRailsEvaluation._print_evaluation_results(
    processed_samples,
    total_test_samples,
    num_user_intent_errors,
    num_bot_intent_errors,
    num_bot_utterance_errors
)
staticmethod

Prints a summary of the evaluation results.

nemoguardrails.evaluate.evaluate_topical.TopicalRailsEvaluation.evaluate_topical_rails()
async

Runs the topical evaluation for the Guardrails app with the current configuration.

nemoguardrails.evaluate.evaluate_topical._split_test_set_from_config(
    config: nemoguardrails.RailsConfig,
    test_set_percentage: float,
    test_set: typing.Dict[str, typing.List],
    max_samples_per_intent: int,
    random_seed: typing.Optional[int] = None
)

Extracts a test set of user messages from a config.

Parameters:

config
RailsConfig

The config from which the test set will be extracted.

test_set_percentage
float

The percentage used for the test set.

test_set
Dict[str, List]

A dictionary where the test set will be added.

max_samples_per_intent
int

A limit on the number of samples per intent to be enforced.

nemoguardrails.evaluate.evaluate_topical.cosine_similarity(
    v1,
    v2
)

Compute the dot product between two embeddings using numpy functions.

nemoguardrails.evaluate.evaluate_topical.sync_wrapper(
    async_func
)

Wrapper for the evaluate_topical_rails method which is async.