nemo_voice_agent.evaluation.scenarios.data.tau2_retail.base

Tau2-retail scenario base.

Tau2RetailBaseScenario mirrors the airline base, plus exposes nl_assertions from evaluation_criteria.nl_assertions for the LLM judge. Each scenario subclass only needs to set tau2_id (e.g. "0").

Module Contents

Classes

Name	Description
`Tau2RetailBaseScenario`	Base class for scenarios ported from tau2-bench/retail (voice-user-sim-v1.0).

API

class nemo_voice_agent.evaluation.scenarios.data.tau2_retail.base.Tau2RetailBaseScenario()

Bases: Tau2BaseScenario

Base class for scenarios ported from tau2-bench/retail (voice-user-sim-v1.0).

Subclasses must set tau2_id (the key from tasks.json). For example::

@register_eval_scenario class Tau2Retail0(Tau2RetailBaseScenario): name = “tau2_retail__0” tau2_id = “0”

Adds one signal beyond airline: nl_assertions. 40 of 114 retail tasks carry natural-language assertions in evaluation_criteria.nl_assertions (e.g. “Agent should tell the user that there are 10 t-shirt options available.”). The runner threads these into LLMJudge.judge_scenario, which returns per-assertion verdicts and a per-domain nl_assertion_success_rate.

agent_resources

Resources

Full tau2_retail tool surface (all 16 ported tools) + per-scenario info.

OVERRIDES Tau2BaseScenario.agent_resources (which returns an empty stub). Tool keys are the Tool class names (registry lookup); each maps to an empty kwargs dict because tau2 tools take only shared_state which the bot server injects automatically.

EndConversationTool resolves via the registry’s “default” namespace fallback (it lives in basic_tools.py, not tau2_retail_tools.py). The paired prompt-side guideline lives in nemo_voice_agent.evaluation.scenarios.END_CONVERSATION_GUIDELINE.

domain

str = 'tau2_retail'

has_user_state

bool = False

nl_assertions

Optional[List[str]]

Per-task natural-language assertions for the LLM judge.

Pulled directly from tau2’s evaluation_criteria.nl_assertions — a list of strings like “Agent should tell the user that there are 10 t-shirt options available.”. None (not []) when the task has no assertions, so the runner’s if scenario_nl_assertions: guard in runner.py correctly skips verdict aggregation for action-only tasks. Empty list from upstream is also normalized to None for the same reason.

success_signals

tuple

DB-state hash + (when present) per-assertion NL judge verdicts.

40 of 114 retail tasks carry nl_assertions; the rest don’t. The rule is mechanical — derive from self.nl_assertions so it cannot drift from the per-task opt-in. JUDGE_PASSED (overall judge score) is intentionally NOT gating: DB_STATE_MATCH is deterministic, and NL_ASSERTION is already per-claim judge verdicts when present — the overall judge becomes informational.

nemo_voice_agent.evaluation.scenarios.data.tau2_retail.base.Tau2RetailBaseScenario._build_tool_map(
    state: dict
) -> typing.Dict[str, typing.Any]

Instantiate one of each ported retail tool, bound to the given state.

Used by Tau2BaseScenario._gold_replay to dispatch reference actions.