nemoguardrails.eval.ui.utils

Module Contents

Classes

Name	Description
`EvalData`	Data relation to an evaluation, relevant for the UI.

Functions

Name	Description
`collect_interaction_metrics`	Collects and aggregates the metrics from all the interactions.
`collect_interaction_metrics_with_expected_latencies`	Similar to collect_interaction_metrics but with expected latencies.

API

class nemoguardrails.eval.ui.utils.EvalData()

Bases: BaseModel

Data relation to an evaluation, relevant for the UI.

eval_config

EvalConfig

eval_config_path

str

eval_outputs

Dict[str, EvalOutput]

output_paths

List[str]

selected_output_path

Optional[str] = None

nemoguardrails.eval.ui.utils.EvalData.update_config_latencies()

Update back the expected latencies.

nemoguardrails.eval.ui.utils.EvalData.update_results()

Updates back the evaluation results.

nemoguardrails.eval.ui.utils.EvalData.update_results_and_logs(
    output_path: str
)

Update back the results and the logs.

nemoguardrails.eval.ui.utils.collect_interaction_metrics(
    interaction_outputs: typing.List[nemoguardrails.eval.models.InteractionOutput]
) -> typing.Dict[str, typing.Union[int, float]]

Collects and aggregates the metrics from all the interactions.

nemoguardrails.eval.ui.utils.collect_interaction_metrics_with_expected_latencies(
    interaction_outputs: typing.List[nemoguardrails.eval.models.InteractionOutput],
    interaction_logs: typing.List[nemoguardrails.eval.models.InteractionLog],
    expected_latencies: typing.Dict[str, float]
)

Similar to collect_interaction_metrics but with expected latencies.