nemo_voice_agent.evaluation.db_hash | NeMo Labs Voice Agent

Utilities for computing hashes and structured diffs of scenario DB states.

Used by the eva-style “DB-state hash matching” scoring mode (see evaluation/README.md “eva_airline domain notes”): hash the post-run DB and the expected DB; if they differ, compute_db_diff produces a tables → records → fields diff for human debugging.

Path-independent scoring: any sequence of agent actions that lands in the right end state passes, regardless of how the agent got there.

Module Contents

Functions

Name	Description
`_compute_record_diff`	Recursively diff two record values; returns None if identical.
`_compute_table_diff`	Diff between two dict-valued tables; returns None if identical.
`compute_db_diff`	Compute a structured diff between expected and actual DB states.
`get_dict_hash`	Compute SHA-256 hash of a dict (canonical JSON serialization).
`normalize_for_comparison`	Recursively normalize values for consistent comparison and hashing.

Data

HASH_EXCLUDED_KEYS

ORDER_INDEPENDENT_LIST_FIELDS

API

nemo_voice_agent.evaluation.db_hash._compute_record_diff(
    expected_record: typing.Any,
    actual_record: typing.Any,
    path: str = '',
    field_name: str = ''
) -> dict | None

Recursively diff two record values; returns None if identical.

nemo_voice_agent.evaluation.db_hash._compute_table_diff(
    expected_table: dict,
    actual_table: dict
) -> dict | None

Diff between two dict-valued tables; returns None if identical.

nemo_voice_agent.evaluation.db_hash.compute_db_diff(
    expected_db: dict,
    actual_db: dict
) -> dict

Compute a structured diff between expected and actual DB states.

Adapted from https://github.com/ServiceNow/eva/tree/0.1.3.

Returned dict has shape::

{ “tables_added”: [name, …], # in actual but not expected “tables_removed”: [name, …], # in expected but not actual “tables_modified”: {table_name: {records_added, records_removed, records_modified}}, }

Used for human debugging when get_dict_hash(expected) != get_dict_hash(actual). Not used for scoring itself (the hash comparison is the verdict).

nemo_voice_agent.evaluation.db_hash.get_dict_hash(
    obj: dict
) -> str

Compute SHA-256 hash of a dict (canonical JSON serialization).

Adapted from https://github.com/ServiceNow/eva/tree/0.1.3.

Follows tau-2 bench’s approach:

Drop keys in HASH_EXCLUDED_KEYS (e.g. session) from the top level.
Normalize via normalize_for_comparison so float/int and “none”/None don’t cause false mismatches.
Serialize with sort_keys=True, no whitespace, default=str for non-JSON-serializable types.
SHA-256 of the resulting string.

nemo_voice_agent.evaluation.db_hash.normalize_for_comparison(
    obj: typing.Any
) -> typing.Any

Recursively normalize values for consistent comparison and hashing.

Adapted from https://github.com/ServiceNow/eva/tree/0.1.3 — verbatim.

Eliminates common false-mismatches between the agent’s DB and the expected DB (e.g. one side stores 1 and the other 1.0).

nemo_voice_agent.evaluation.db_hash.HASH_EXCLUDED_KEYS: set[str] = {'session'}

nemo_voice_agent.evaluation.db_hash.ORDER_INDEPENDENT_LIST_FIELDS: set[str] = {'standby_list', 'notifications', 'bookings', 'system_accounts', 'group_membersh...