nemo_voice_agent.evaluation.utils | NeMo Labs Voice Agent

Module Contents

Classes

Name	Description
`LLMJudge`	LLM-based judge for evaluating voice agent responses.

Functions

Name	Description
`_clean_label`	Deterministic local text normalizer used by evaluator matching.
`_compact_context_history_for_judge`	Keep judge input useful without sending full prompt/history dumps.
`_compact_context_value_for_judge`	-
`_failed_nl_assertion_verdicts`	-
`_filtered_args`	Filter a tool-call arguments dict by tau2’s `compare_args` semantics.
`_is_system_message`	-
`_match_action`	Deterministic action-record match honoring tau2’s `compare_args` field.
`_positive_float`	-
`_positive_int`	-
`_remove_punctuations`	Deterministic local punctuation normalizer used by evaluator matching.
`_truncate_for_judge`	-
`_validate_threshold`	-
`check_if_task_success`	Check if the prediction is matches with the reference answer.
`match_dict`	Check if pred_dict contains all keys and matching values from ref_dict.
`match_item`	Recursively match a reference value against a prediction value.
`match_list`	Check if each item in ref_list has a matching item in pred_list (order-independent).
`match_str_and_float`	Match the reference and prediction value.
`normalize_scenario_payload`	Normalize a scenario payload (reference or prediction) to a canonical shape.
`validate_judge_numeric_options`	Validate judge-related numeric options shared by CLI and library tests.

Data

_JUDGE_CONTEXT_MESSAGE_LIMIT

_JUDGE_CONTEXT_STRING_LIMIT

_JUDGE_CONTEXT_SYSTEM_STRING_LIMIT

API

class nemo_voice_agent.evaluation.utils.LLMJudge(
    url: str,
    model: str,
    api_key: typing.Optional[str] = None,
    api_key_name: str = 'API_KEY',
    default_prompt: typing.Optional[str] = None,
    timeout: typing.Optional[float] = 120.0,
    compact_context: bool = False,
    context_message_limit: typing.Optional[int] = None,
    context_system_string_limit: typing.Optional[int] = None,
    context_string_limit: typing.Optional[int] = None,
    kwargs = {}
)

LLM-based judge for evaluating voice agent responses.

Uses an OpenAI-compatible chat completions API to score how well a prediction matches a reference answer. Returns a float score between 0 and 1.

Parameters:

url

str

The URL of the OpenAI-compatible chat completions endpoint.

model

str

The model name to use for judging.

api_key

Optional[str]" default="None

The API key. If None, will be loaded from environment variable.

api_key_name

str" default="'API_KEY'

The environment variable name for the API key (default: “API_KEY”).

default_prompt

Optional[str]" default="None

Custom default system prompt. If None, uses DEFAULT_PROMPT.

**kwargs

Defaults to {}

Additional keyword arguments passed to the API payload (e.g., temperature, max_tokens).

DEFAULT_PROMPT

SCENARIO_PROMPT

context_message_limit

context_string_limit

context_system_string_limit

default_prompt

= default_prompt or self.DEFAULT_PROMPT

headers

timeout

nemo_voice_agent.evaluation.utils.LLMJudge._get_payload(
    user_content: str,
    prompt: typing.Optional[str] = None
) -> dict

nemo_voice_agent.evaluation.utils.LLMJudge._parse_response(
    response: requests.Response
) -> dict

Parse the LLM response and extract the judgement JSON.

Returns: A dict with “score” (float) and optionally “reason” (str). Raises: ValueError: If the response cannot be parsed.

Parameters:

response

requests.Response

The HTTP response from the API.

nemo_voice_agent.evaluation.utils.LLMJudge.judge(
    reference: str,
    prediction: str,
    prompt: typing.Optional[str] = None
) -> dict

Judge the similarity between a reference and a prediction.

Returns: A dict with “score” (float between 0 and 1) and “reason” (str). On error, returns {“score”: 0.0, “reason”: “<error message>”}.

Parameters:

reference

str

The reference answer string.

prediction

str

The prediction answer string.

prompt

Optional[str]" default="None

Optional custom system prompt. Uses default_prompt if not provided.

nemo_voice_agent.evaluation.utils.LLMJudge.judge_file(
    reference: str,
    prediction: str,
    prompt: typing.Optional[str] = None
) -> dict

Judge the similarity between a reference file and a prediction file.

Returns: A dict with “score” (float between 0 and 1) and “reason” (str).

Parameters:

reference

str

Path to the reference JSON file.

prediction

str

Path to the prediction JSON file.

prompt

Optional[str]" default="None

Optional custom system prompt.

nemo_voice_agent.evaluation.utils.LLMJudge.judge_scenario(
    reference: typing.Optional[str] = None,
    prediction: typing.Optional[str] = None,
    conversation: typing.Optional[list] = None,
    agent_context_history: typing.Optional[list] = None,
    user_context_history: typing.Optional[list] = None,
    context_history: typing.Optional[list] = None,
    nl_assertions: typing.Optional[typing.List[str]] = None,
    prompt: typing.Optional[str] = None
) -> dict

Judge agent performance with full scenario context including conversation history.

Returns: A dict with “score” (float between 0 and 1) and “reason” (str). When nl_assertions is non-empty, also includes nl_assertion_verdicts (list of {index, passed, reason}) and nl_assertion_pass_rate (float).

Parameters:

reference

Optional[str]" default="None

Optional reference answer string (or JSON string).

prediction

Optional[str]" default="None

Optional prediction answer string (or JSON string).

conversation

Optional[list]" default="None

List of conversation turns, each a dict with “role” and “text” keys.

agent_context_history

Optional[list]" default="None

Agent’s LLM context messages (from bot_logs_agent/llm_context.json). Contains the agent’s tool calls + results. Rendered as <agent_context_history>.

user_context_history

Optional[list]" default="None

Simulated user’s LLM context messages (from bot_logs_user/llm_context.json). Contains the user-sim’s own tool calls — essential for dual-side domains like telecom where reference actions with side="user" are executed by the user-sim, not the agent. Rendered as <user_context_history>.

context_history

Optional[list]" default="None

Backward-compatible alias for agent_context_history.

nl_assertions

Optional[List[str]]" default="None

Optional natural-language assertions (tau2 retail). When provided, each assertion is appended to the prompt and the LLM is instructed to emit a per-assertion verdict list. The returned dict gains a nl_assertion_verdicts field (one entry per assertion, {index, passed, reason}`) plus `nl_assertion_pass_rate. Missing/malformed verdicts are filled with passed=False so the runner can still aggregate cleanly. When None (or empty), the returned shape is the basic {score, reason} dict — no per-assertion fields.

prompt

Optional[str]" default="None

Optional custom system prompt. Uses SCENARIO_PROMPT if not provided.

nemo_voice_agent.evaluation.utils._clean_label(
    text: str,
    lowercase: bool = False
) -> str

Deterministic local text normalizer used by evaluator matching.

nemo_voice_agent.evaluation.utils._compact_context_history_for_judge(
    history: typing.Optional[list],
    message_limit: int,
    system_string_limit: int,
    string_limit: int
) -> typing.Optional[list]

Keep judge input useful without sending full prompt/history dumps.

nemo_voice_agent.evaluation.utils._compact_context_value_for_judge(
    value: typing.Any,
    message_limit: int,
    system_string_limit: int,
    string_limit: int
) -> typing.Any

nemo_voice_agent.evaluation.utils._failed_nl_assertion_verdicts(
    nl_assertions: typing.List[str],
    reason: str
) -> typing.List[dict]

nemo_voice_agent.evaluation.utils._filtered_args(
    args: dict,
    compare_args: typing.Optional[typing.List[str]]
) -> dict

Filter a tool-call arguments dict by tau2’s compare_args semantics.

compare_args=None → return args verbatim (compare all). compare_args=[] → return {} (name-only match — empty == empty). compare_args=[k...]→ return {k: args.get(k) for k in compare_args}.

Pitfall: do NOT write compare_args or "all"-style fallbacks — [] is falsy and would silently collapse name-only matches into compare-all. Tau2 stores compare_args: None explicitly (the key is present), so ref.get("compare_args", "all") also doesn’t fire the default. The explicit is None check below is the only safe sentinel.

nemo_voice_agent.evaluation.utils._is_system_message(
    value: typing.Any
) -> bool

nemo_voice_agent.evaluation.utils._match_action(
    ref: dict,
    pred: dict
) -> bool

Deterministic action-record match honoring tau2’s compare_args field.

Name must always match. Argument comparison is delegated to _filtered_args which applies the compare_args filter to both sides before equality.

Reference records that omit compare_args (e.g. eva_airline records) get the default None from ref.get("compare_args"), which means “compare all arguments” — preserving existing behavior for non-tau2 domains.

Adapted from https://github.com/sierra-research/tau2-bench/tree/voice-user-sim-v1.0 src/tau2/data_model/tasks.py:175-182

nemo_voice_agent.evaluation.utils._positive_float(
    name: str,
    value: float
) -> float

nemo_voice_agent.evaluation.utils._positive_int(
    name: str,
    value: int
) -> int

nemo_voice_agent.evaluation.utils._remove_punctuations(
    text: str
) -> str

Deterministic local punctuation normalizer used by evaluator matching.

nemo_voice_agent.evaluation.utils._truncate_for_judge(
    value: str,
    limit: int
) -> str

nemo_voice_agent.evaluation.utils._validate_threshold(
    name: str,
    value: float
) -> float

nemo_voice_agent.evaluation.utils.check_if_task_success(
    reference: str,
    prediction: str,
    ignore_capitalization: bool = False,
    ignore_punctuation: bool = False,
    clean_text: bool = False,
    disallow_extra_items: bool = False
) -> bool

Check if the prediction is matches with the reference answer.

Situations:

If the reference is a dictionary, and the prediction is a dictionary:

The prediction should have the same keys and values as the reference.
Additional keys in prediction are allowed.

If the reference is a dictionary, and the prediction is a list of dictionaries:

the last dictionary in the prediction would be matched with the reference.

If the reference is a list of dictionaries, and the prediction is a list of dictionaries:

For each dictionary in the reference, there should be a dictionary in the prediction that matches it according to the criteria in Situation 1.
The order of the dictionaries in the reference/prediction is not important.
All dictionaries in the reference should be matched with a dictionary in the prediction to be considered as a success.
If disallow_extra_items is True, the lengths must also match exactly (exact bijection — no extra prediction items tolerated).

Both inputs are first passed through normalize_scenario_payload to collapse the list-of-1-dict / single-dict shape difference. The remaining situations 1-3 then handle the post-normalization shapes cleanly.

Returns: True if the task is considered as successful, False otherwise.

Parameters:

reference

str

The path to the reference json file.

prediction

str

The path to the prediction json file.

ignore_capitalization

bool" default="False

Whether to ignore case when comparing strings.

ignore_punctuation

bool" default="False

Whether to ignore punctuation when comparing strings.

clean_text

bool" default="False

Whether to clean the text before comparing.

disallow_extra_items

bool" default="False

For list-of-dicts comparisons (Situation 3), require len(reference) == len(prediction). Default False preserves the lenient behavior where agent extras pass. Note: Situation 2 (single dict reference, list-of-dicts prediction) is unaffected — the last prediction dict is still picked and matched.

nemo_voice_agent.evaluation.utils.match_dict(
    ref_dict: dict,
    pred_dict: dict,
    ignore_capitalization: bool = False,
    ignore_punctuation: bool = False,
    clean_text: bool = False
) -> bool

Check if pred_dict contains all keys and matching values from ref_dict. Additional keys in pred_dict are allowed.

nemo_voice_agent.evaluation.utils.match_item(
    ref_value,
    pred_value,
    ignore_capitalization: bool = False,
    ignore_punctuation: bool = False,
    clean_text: bool = False
) -> bool

Recursively match a reference value against a prediction value. Handles dicts, lists, strings, and numbers.

nemo_voice_agent.evaluation.utils.match_list(
    ref_list: list,
    pred_list: list,
    ignore_capitalization: bool = False,
    ignore_punctuation: bool = False,
    clean_text: bool = False
) -> bool

Check if each item in ref_list has a matching item in pred_list (order-independent). Each prediction item can only be matched once.

nemo_voice_agent.evaluation.utils.match_str_and_float(
    ref_value: typing.Union[str, float],
    pred_value: typing.Union[str, float],
    ignore_capitalization: bool = False,
    ignore_punctuation: bool = False,
    clean_text: bool = False
) -> bool

Match the reference and prediction value.

Returns: True if the reference and prediction value match, False otherwise.

Parameters:

ref_value

Union[str, float]

The reference value, can be a string or a float.

pred_value

Union[str, float]

The prediction value, can be a string or a float.

ignore_capitalization

bool" default="False

Whether to ignore capitalization when comparing strings.

ignore_punctuation

bool" default="False

Whether to ignore punctuation when comparing strings.

clean_text

bool" default="False

Whether to clean the text by replacing special characters before comparing.

nemo_voice_agent.evaluation.utils.normalize_scenario_payload(
    payload
)

Normalize a scenario payload (reference or prediction) to a canonical shape.

Used by both the deterministic comparator (check_if_task_success) and the LLM judge prep path (runner.run_dynamic_evaluation) so they apply the same shape-equivalence rule. Without this, the LLM judge reads the raw file text and deducts for cosmetic {...} vs. [{...}] differences that the deterministic comparator already treats as equivalent (its old “Situation 2” logic). One source of truth, applied to both scoring paths.

nemo_voice_agent.evaluation.utils.validate_judge_numeric_options(
    judge_threshold: typing.Optional[float] = None,
    judge_timeout: typing.Optional[float] = None,
    judge_thinking_token_budget: typing.Optional[int] = None,
    judge_context_message_limit: typing.Optional[int] = None,
    judge_context_system_string_limit: typing.Optional[int] = None,
    judge_context_string_limit: typing.Optional[int] = None,
    judge_max_tokens: typing.Optional[int] = None,
    judge_top_p: typing.Optional[float] = None
) -> None

Validate judge-related numeric options shared by CLI and library tests.

nemo_voice_agent.evaluation.utils._JUDGE_CONTEXT_MESSAGE_LIMIT = 40

nemo_voice_agent.evaluation.utils._JUDGE_CONTEXT_STRING_LIMIT = 6000

nemo_voice_agent.evaluation.utils._JUDGE_CONTEXT_SYSTEM_STRING_LIMIT = 2500