> For clean Markdown of any page, append .md to the page URL. > For a complete documentation index, see https://docs.nvidia.com/nemo/labs-voice-agent/llms.txt. > For AI client integration (Claude Code, Cursor, etc.), connect to the MCP server at https://docs.nvidia.com/nemo/labs-voice-agent/_mcp/server. # nemo_voice_agent.evaluation.utils ## Module Contents ### Classes | Name | Description | | --------------------------------------------------------- | ----------------------------------------------------- | | [`LLMJudge`](#nemo_voice_agent-evaluation-utils-LLMJudge) | LLM-based judge for evaluating voice agent responses. | ### Functions | Name | Description | | ------------------------------------------------------------------------------------------------------------- | -------------------------------------------------------------------------------------- | | [`_clean_label`](#nemo_voice_agent-evaluation-utils-_clean_label) | Deterministic local text normalizer used by evaluator matching. | | [`_compact_context_history_for_judge`](#nemo_voice_agent-evaluation-utils-_compact_context_history_for_judge) | Keep judge input useful without sending full prompt/history dumps. | | [`_compact_context_value_for_judge`](#nemo_voice_agent-evaluation-utils-_compact_context_value_for_judge) | - | | [`_failed_nl_assertion_verdicts`](#nemo_voice_agent-evaluation-utils-_failed_nl_assertion_verdicts) | - | | [`_filtered_args`](#nemo_voice_agent-evaluation-utils-_filtered_args) | Filter a tool-call arguments dict by tau2's `compare_args` semantics. | | [`_is_system_message`](#nemo_voice_agent-evaluation-utils-_is_system_message) | - | | [`_match_action`](#nemo_voice_agent-evaluation-utils-_match_action) | Deterministic action-record match honoring tau2's `compare_args` field. | | [`_positive_float`](#nemo_voice_agent-evaluation-utils-_positive_float) | - | | [`_positive_int`](#nemo_voice_agent-evaluation-utils-_positive_int) | - | | [`_remove_punctuations`](#nemo_voice_agent-evaluation-utils-_remove_punctuations) | Deterministic local punctuation normalizer used by evaluator matching. | | [`_truncate_for_judge`](#nemo_voice_agent-evaluation-utils-_truncate_for_judge) | - | | [`_validate_threshold`](#nemo_voice_agent-evaluation-utils-_validate_threshold) | - | | [`check_if_task_success`](#nemo_voice_agent-evaluation-utils-check_if_task_success) | Check if the prediction is matches with the reference answer. | | [`match_dict`](#nemo_voice_agent-evaluation-utils-match_dict) | Check if pred\_dict contains all keys and matching values from ref\_dict. | | [`match_item`](#nemo_voice_agent-evaluation-utils-match_item) | Recursively match a reference value against a prediction value. | | [`match_list`](#nemo_voice_agent-evaluation-utils-match_list) | Check if each item in ref\_list has a matching item in pred\_list (order-independent). | | [`match_str_and_float`](#nemo_voice_agent-evaluation-utils-match_str_and_float) | Match the reference and prediction value. | | [`normalize_scenario_payload`](#nemo_voice_agent-evaluation-utils-normalize_scenario_payload) | Normalize a scenario payload (reference or prediction) to a canonical shape. | | [`validate_judge_numeric_options`](#nemo_voice_agent-evaluation-utils-validate_judge_numeric_options) | Validate judge-related numeric options shared by CLI and library tests. | ### Data [`_JUDGE_CONTEXT_MESSAGE_LIMIT`](#nemo_voice_agent-evaluation-utils-_JUDGE_CONTEXT_MESSAGE_LIMIT) [`_JUDGE_CONTEXT_STRING_LIMIT`](#nemo_voice_agent-evaluation-utils-_JUDGE_CONTEXT_STRING_LIMIT) [`_JUDGE_CONTEXT_SYSTEM_STRING_LIMIT`](#nemo_voice_agent-evaluation-utils-_JUDGE_CONTEXT_SYSTEM_STRING_LIMIT) ### API ```python class nemo_voice_agent.evaluation.utils.LLMJudge( url: str, model: str, api_key: typing.Optional[str] = None, api_key_name: str = 'API_KEY', default_prompt: typing.Optional[str] = None, timeout: typing.Optional[float] = 120.0, compact_context: bool = False, context_message_limit: typing.Optional[int] = None, context_system_string_limit: typing.Optional[int] = None, context_string_limit: typing.Optional[int] = None, kwargs = {} ) ``` LLM-based judge for evaluating voice agent responses. Uses an OpenAI-compatible chat completions API to score how well a prediction matches a reference answer. Returns a float score between 0 and 1. **Parameters:** **`url`** The URL of the OpenAI-compatible chat completions endpoint. --- **`model`** The model name to use for judging. --- **`api_key`** The API key. If None, will be loaded from environment variable. --- **`api_key_name`** The environment variable name for the API key (default: "API\_KEY"). --- **`default_prompt`** Custom default system prompt. If None, uses DEFAULT\_PROMPT. --- **`**kwargs`** — default: \{} Additional keyword arguments passed to the API payload (e.g., temperature, max\_tokens). --- **`DEFAULT_PROMPT`** --- **`SCENARIO_PROMPT`** --- **`context_message_limit`** --- **`context_string_limit`** --- **`context_system_string_limit`** --- **`default_prompt`** --- **`headers`** --- **`timeout`** --- ```python nemo_voice_agent.evaluation.utils.LLMJudge._get_payload( user_content: str, prompt: typing.Optional[str] = None ) -> dict ``` ```python nemo_voice_agent.evaluation.utils.LLMJudge._parse_response( response: requests.Response ) -> dict ``` Parse the LLM response and extract the judgement JSON. Returns: A dict with "score" (float) and optionally "reason" (str). Raises: ValueError: If the response cannot be parsed. **Parameters:** **`response`** The HTTP response from the API. --- ```python nemo_voice_agent.evaluation.utils.LLMJudge.judge( reference: str, prediction: str, prompt: typing.Optional[str] = None ) -> dict ``` Judge the similarity between a reference and a prediction. Returns: A dict with "score" (float between 0 and 1) and "reason" (str). On error, returns \{"score": 0.0, "reason": "\"}. **Parameters:** **`reference`** The reference answer string. --- **`prediction`** The prediction answer string. --- **`prompt`** Optional custom system prompt. Uses default\_prompt if not provided. --- ```python nemo_voice_agent.evaluation.utils.LLMJudge.judge_file( reference: str, prediction: str, prompt: typing.Optional[str] = None ) -> dict ``` Judge the similarity between a reference file and a prediction file. Returns: A dict with "score" (float between 0 and 1) and "reason" (str). **Parameters:** **`reference`** Path to the reference JSON file. --- **`prediction`** Path to the prediction JSON file. --- **`prompt`** Optional custom system prompt. --- ```python nemo_voice_agent.evaluation.utils.LLMJudge.judge_scenario( reference: typing.Optional[str] = None, prediction: typing.Optional[str] = None, conversation: typing.Optional[list] = None, agent_context_history: typing.Optional[list] = None, user_context_history: typing.Optional[list] = None, context_history: typing.Optional[list] = None, nl_assertions: typing.Optional[typing.List[str]] = None, prompt: typing.Optional[str] = None ) -> dict ``` Judge agent performance with full scenario context including conversation history. Returns: A dict with "score" (float between 0 and 1) and "reason" (str). When `nl_assertions` is non-empty, also includes `nl_assertion_verdicts` (list of `{index, passed, reason}`) and `nl_assertion_pass_rate` (float). **Parameters:** **`reference`** Optional reference answer string (or JSON string). --- **`prediction`** Optional prediction answer string (or JSON string). --- **`conversation`** List of conversation turns, each a dict with "role" and "text" keys. --- **`agent_context_history`** Agent's LLM context messages (from `bot_logs_agent/llm_context.json`). Contains the agent's tool calls + results. Rendered as `<agent_context_history>`. --- **`user_context_history`** Simulated user's LLM context messages (from `bot_logs_user/llm_context.json`). Contains the user-sim's own tool calls — essential for dual-side domains like telecom where reference actions with `side="user"` are executed by the user-sim, not the agent. Rendered as `<user_context_history>`. --- **`context_history`** Backward-compatible alias for agent\_context\_history. --- **`nl_assertions`** Optional natural-language assertions (tau2 retail). When provided, each assertion is appended to the prompt and the LLM is instructed to emit a per-assertion verdict list. The returned dict gains a `nl_assertion_verdicts` field (one entry per assertion, ``{index, passed, reason}`) plus `nl_assertion_pass_rate``. Missing/malformed verdicts are filled with `passed=False` so the runner can still aggregate cleanly. When `None` (or empty), the returned shape is the basic `{score, reason}` dict — no per-assertion fields. --- **`prompt`** Optional custom system prompt. Uses SCENARIO\_PROMPT if not provided. --- ```python nemo_voice_agent.evaluation.utils._clean_label( text: str, lowercase: bool = False ) -> str ``` Deterministic local text normalizer used by evaluator matching. ```python nemo_voice_agent.evaluation.utils._compact_context_history_for_judge( history: typing.Optional[list], message_limit: int, system_string_limit: int, string_limit: int ) -> typing.Optional[list] ``` Keep judge input useful without sending full prompt/history dumps. ```python nemo_voice_agent.evaluation.utils._compact_context_value_for_judge( value: typing.Any, message_limit: int, system_string_limit: int, string_limit: int ) -> typing.Any ``` ```python nemo_voice_agent.evaluation.utils._failed_nl_assertion_verdicts( nl_assertions: typing.List[str], reason: str ) -> typing.List[dict] ``` ```python nemo_voice_agent.evaluation.utils._filtered_args( args: dict, compare_args: typing.Optional[typing.List[str]] ) -> dict ``` Filter a tool-call arguments dict by tau2's `compare_args` semantics. `compare_args=None` → return `args` verbatim (compare all). `compare_args=[]` → return `{}` (name-only match — empty == empty). `compare_args=[k...]`→ return `{k: args.get(k) for k in compare_args}`. Pitfall: do NOT write `compare_args or "all"`-style fallbacks — `[]` is falsy and would silently collapse name-only matches into compare-all. Tau2 stores `compare_args: None` explicitly (the key is present), so `ref.get("compare_args", "all")` also doesn't fire the default. The explicit `is None` check below is the only safe sentinel. ```python nemo_voice_agent.evaluation.utils._is_system_message( value: typing.Any ) -> bool ``` ```python nemo_voice_agent.evaluation.utils._match_action( ref: dict, pred: dict ) -> bool ``` Deterministic action-record match honoring tau2's `compare_args` field. Name must always match. Argument comparison is delegated to `_filtered_args` which applies the `compare_args` filter to both sides before equality. Reference records that omit `compare_args` (e.g. eva\_airline records) get the default `None` from `ref.get("compare_args")`, which means "compare all arguments" — preserving existing behavior for non-tau2 domains. Adapted from [https://github.com/sierra-research/tau2-bench/tree/voice-user-sim-v1.0](https://github.com/sierra-research/tau2-bench/tree/voice-user-sim-v1.0) src/tau2/data\_model/tasks.py:175-182 ```python nemo_voice_agent.evaluation.utils._positive_float( name: str, value: float ) -> float ``` ```python nemo_voice_agent.evaluation.utils._positive_int( name: str, value: int ) -> int ``` ```python nemo_voice_agent.evaluation.utils._remove_punctuations( text: str ) -> str ``` Deterministic local punctuation normalizer used by evaluator matching. ```python nemo_voice_agent.evaluation.utils._truncate_for_judge( value: str, limit: int ) -> str ``` ```python nemo_voice_agent.evaluation.utils._validate_threshold( name: str, value: float ) -> float ``` ```python nemo_voice_agent.evaluation.utils.check_if_task_success( reference: str, prediction: str, ignore_capitalization: bool = False, ignore_punctuation: bool = False, clean_text: bool = False, disallow_extra_items: bool = False ) -> bool ``` Check if the prediction is matches with the reference answer. Situations: 1. If the reference is a dictionary, and the prediction is a dictionary: * The prediction should have the same keys and values as the reference. * Additional keys in prediction are allowed. 2. If the reference is a dictionary, and the prediction is a list of dictionaries: * the last dictionary in the prediction would be matched with the reference. 3. If the reference is a list of dictionaries, and the prediction is a list of dictionaries: * For each dictionary in the reference, there should be a dictionary in the prediction that matches it according to the criteria in Situation 1. * The order of the dictionaries in the reference/prediction is not important. * All dictionaries in the reference should be matched with a dictionary in the prediction to be considered as a success. * If `disallow_extra_items` is True, the lengths must also match exactly (exact bijection — no extra prediction items tolerated). Both inputs are first passed through `normalize_scenario_payload` to collapse the list-of-1-dict / single-dict shape difference. The remaining situations 1-3 then handle the post-normalization shapes cleanly. Returns: True if the task is considered as successful, False otherwise. **Parameters:** **`reference`** The path to the reference json file. --- **`prediction`** The path to the prediction json file. --- **`ignore_capitalization`** Whether to ignore case when comparing strings. --- **`ignore_punctuation`** Whether to ignore punctuation when comparing strings. --- **`clean_text`** Whether to clean the text before comparing. --- **`disallow_extra_items`** For list-of-dicts comparisons (Situation 3), require `len(reference) == len(prediction)`. Default False preserves the lenient behavior where agent extras pass. Note: Situation 2 (single dict reference, list-of-dicts prediction) is unaffected — the last prediction dict is still picked and matched. --- ```python nemo_voice_agent.evaluation.utils.match_dict( ref_dict: dict, pred_dict: dict, ignore_capitalization: bool = False, ignore_punctuation: bool = False, clean_text: bool = False ) -> bool ``` Check if pred\_dict contains all keys and matching values from ref\_dict. Additional keys in pred\_dict are allowed. ```python nemo_voice_agent.evaluation.utils.match_item( ref_value, pred_value, ignore_capitalization: bool = False, ignore_punctuation: bool = False, clean_text: bool = False ) -> bool ``` Recursively match a reference value against a prediction value. Handles dicts, lists, strings, and numbers. ```python nemo_voice_agent.evaluation.utils.match_list( ref_list: list, pred_list: list, ignore_capitalization: bool = False, ignore_punctuation: bool = False, clean_text: bool = False ) -> bool ``` Check if each item in ref\_list has a matching item in pred\_list (order-independent). Each prediction item can only be matched once. ```python nemo_voice_agent.evaluation.utils.match_str_and_float( ref_value: typing.Union[str, float], pred_value: typing.Union[str, float], ignore_capitalization: bool = False, ignore_punctuation: bool = False, clean_text: bool = False ) -> bool ``` Match the reference and prediction value. Returns: True if the reference and prediction value match, False otherwise. **Parameters:** **`ref_value`** The reference value, can be a string or a float. --- **`pred_value`** The prediction value, can be a string or a float. --- **`ignore_capitalization`** Whether to ignore capitalization when comparing strings. --- **`ignore_punctuation`** Whether to ignore punctuation when comparing strings. --- **`clean_text`** Whether to clean the text by replacing special characters before comparing. --- ```python nemo_voice_agent.evaluation.utils.normalize_scenario_payload( payload ) ``` Normalize a scenario payload (reference or prediction) to a canonical shape. Used by both the deterministic comparator (`check_if_task_success`) and the LLM judge prep path (`runner.run_dynamic_evaluation`) so they apply the same shape-equivalence rule. Without this, the LLM judge reads the raw file text and deducts for cosmetic `{...}` vs. `[{...}]` differences that the deterministic comparator already treats as equivalent (its old "Situation 2" logic). One source of truth, applied to both scoring paths. ```python nemo_voice_agent.evaluation.utils.validate_judge_numeric_options( judge_threshold: typing.Optional[float] = None, judge_timeout: typing.Optional[float] = None, judge_thinking_token_budget: typing.Optional[int] = None, judge_context_message_limit: typing.Optional[int] = None, judge_context_system_string_limit: typing.Optional[int] = None, judge_context_string_limit: typing.Optional[int] = None, judge_max_tokens: typing.Optional[int] = None, judge_top_p: typing.Optional[float] = None ) -> None ``` Validate judge-related numeric options shared by CLI and library tests. ```python nemo_voice_agent.evaluation.utils._JUDGE_CONTEXT_MESSAGE_LIMIT = 40 ``` ```python nemo_voice_agent.evaluation.utils._JUDGE_CONTEXT_STRING_LIMIT = 6000 ``` ```python nemo_voice_agent.evaluation.utils._JUDGE_CONTEXT_SYSTEM_STRING_LIMIT = 2500 ```