ReferencePython SDK ReferenceNemoguardrailsEval

nemoguardrails.eval.models

View as Markdown

Module Contents

Classes

NameDescription
ComplianceCheckLogDetailed log about a compliance check.
ComplianceCheckResultInformation about a compliance check.
EvalConfigAn evaluation configuration for an evaluation dataset.
EvalOutputThe output of the evaluation.
ExpectedOutputAn expected output from the system, as dictated by a policy.
GenericOutput-
InteractionLogDetailed log about the execution of an interaction.
InteractionOutputThe output for running and evaluating an interaction.
InteractionSetAn interaction set description that is part of an evaluation dataset.
PolicyA policy for the evaluation of a guardrail configuration.
RefusalOutput-
SimilarMessageOutput-
SpanA generic span object

API

class nemoguardrails.eval.models.ComplianceCheckLog()

Bases: BaseModel

Detailed log about a compliance check.

id
str
llm_calls
List[LLMCallInfo]
class nemoguardrails.eval.models.ComplianceCheckResult()

Bases: BaseModel

Information about a compliance check.

compliance
Dict[str, Optional[Union[bool, str]]]
created_at
str
details
str
id
str
interaction_id
Optional[str] = Field(description='The id of the interaction.')
method
str
class nemoguardrails.eval.models.EvalConfig()

Bases: BaseModel

An evaluation configuration for an evaluation dataset.

expected_latencies
Dict[str, float]
interactions
List[InteractionSet]
models
List[Model]
policies
List[Policy]
prompts
Optional[List[TaskPrompt]]
classmethod

Loads an eval configuration from a given path.

It supports YAML files (*.yml and .yaml) and .json files (.json).

nemoguardrails.eval.models.EvalConfig.validate_policy_ids(
    values: typing.Any
)

Validates the policy ids used in the interactions.

class nemoguardrails.eval.models.EvalOutput()

Bases: BaseModel

The output of the evaluation.

logs
List[InteractionLog]
results
List[InteractionOutput]
nemoguardrails.eval.models.EvalOutput.compute_compliance(
    eval_config: nemoguardrails.eval.models.EvalConfig
) -> typing.Dict[str, dict]

Helper to compute the compliance rate per policy.

classmethod

Loads an eval output from a given path.

It supports YAML files (*.yml and .yaml) and .json files (.json).

class nemoguardrails.eval.models.ExpectedOutput()

Bases: BaseModel

An expected output from the system, as dictated by a policy.

policy
str
type
str
class nemoguardrails.eval.models.GenericOutput()

Bases: ExpectedOutput

description
str
type
str = 'generic'
nemoguardrails.eval.models.GenericOutput.__str__()
class nemoguardrails.eval.models.InteractionLog()

Bases: BaseModel

Detailed log about the execution of an interaction.

activated_rails
List[ActivatedRail]
compliance_checks
List[ComplianceCheckLog]
events
List[dict]
id
str
trace
List[Span]
class nemoguardrails.eval.models.InteractionOutput()

Bases: BaseModel

The output for running and evaluating an interaction.

compliance
Dict[str, Optional[Union[bool, str]]]
compliance_checks
List[ComplianceCheckResult]
expected_latencies
Dict[str, Union[float]]
id
str
input
Union[str, dict] = Field(description='The input of the interaction.')
latencies
Dict[str, Union[float]]
output
Optional[Union[str, List[dict]]]
resource_usage
Dict[str, Union[int, float]]
class nemoguardrails.eval.models.InteractionSet()

Bases: BaseModel

An interaction set description that is part of an evaluation dataset.

An interaction set groups multiple interactions with the same expected output.

evaluation_context
Dict[str, Any]
exclude_policies
List[str]
expected_output
List[ExpectedOutput]
id
str
include_policies
List[str]
inputs
List[Union[str, dict]]
tags
List[str]
nemoguardrails.eval.models.InteractionSet.instantiate_expected_output(
    values: typing.Any
)

Creates the right instance of the expected output.

class nemoguardrails.eval.models.Policy()

Bases: BaseModel

A policy for the evaluation of a guardrail configuration.

apply_to_all
bool
description
str
id
str
weight
int
class nemoguardrails.eval.models.RefusalOutput()

Bases: ExpectedOutput

type
str = 'refusal'
nemoguardrails.eval.models.RefusalOutput.__str__()
class nemoguardrails.eval.models.SimilarMessageOutput()

Bases: ExpectedOutput

message
str
type
str = 'similar_message'
nemoguardrails.eval.models.SimilarMessageOutput.__str__()
class nemoguardrails.eval.models.Span()

Bases: BaseModel

A generic span object

duration
float
end_time
float = Field(description='The end time of the span.')
metrics
Dict[str, Union[int, float]]
name
str
parent_id
Optional[str]
resource_id
Optional[str]
span_id
str = Field(description='The id of the span.')
start_time
float = Field(description='The start time of the span.')