ReferencePython SDK ReferenceNemoguardrailsEval

nemoguardrails.eval.eval

View as Markdown

Module Contents

Functions

NameDescription
_extract_interaction_logExtracts an InteractionLog object from an GenerationLog object.
_extract_interaction_outputsExtract the list of interactions that should be run.
_extract_spansExtract a simplified span view from the log of activated rails.
_load_eval_outputLoads the existing evaluation output from the provided path.
run_evalRun a guardrail evaluation.

Data

ROOT

API

nemoguardrails.eval.eval._extract_interaction_log(
    interaction_output: nemoguardrails.eval.models.InteractionOutput,
    generation_log: nemoguardrails.rails.llm.options.GenerationLog
) -> nemoguardrails.eval.models.InteractionLog

Extracts an InteractionLog object from an GenerationLog object.

nemoguardrails.eval.eval._extract_interaction_outputs(
    eval_config: nemoguardrails.eval.models.EvalConfig
) -> typing.List[nemoguardrails.eval.models.InteractionOutput]

Extract the list of interactions that should be run.

Creates the output objects with no data.

nemoguardrails.eval.eval._extract_spans(
    activated_rails: typing.List[nemoguardrails.rails.llm.options.ActivatedRail]
) -> typing.List[nemoguardrails.eval.models.Span]

Extract a simplified span view from the log of activated rails.

nemoguardrails.eval.eval._load_eval_output(
    output_path: str,
    eval_config: nemoguardrails.eval.models.EvalConfig
) -> nemoguardrails.eval.models.EvalOutput

Loads the existing evaluation output from the provided path.

If existing data is found, it is automatically loaded. If the evaluation config has changed meanwhile, it will try to reuse as much as possible. For example, if a new interaction is added, the output will keep all the data from the previous interactions.

nemoguardrails.eval.eval.run_eval(
    eval_config_path: str,
    guardrail_config_path: str,
    output_path: str,
    output_format: str = 'json',
    parallel: int = 1
)
async

Run a guardrail evaluation.

Parameters:

eval_config_path
str

Path to a directory containing eval configuration files.

guardrail_config_path
str

Path to a directory containing the guardrail configuration.

output_path
str

Output directory for predictions. Defaults to None.

output_format
strDefaults to 'json'

Output format. Supported values are “yaml” and “json”. Defaults to “yaml”.

parallel
intDefaults to 1

Number degree of parallelism to use. Defaults to 1.

nemoguardrails.eval.eval.ROOT = os.path.join(os.path.dirname(os.path.abspath(__file__)), '../..')