nemo_voice_agent.evaluation.resume | NeMo Labs Voice Agent

Resume-state classification utilities shared between the evaluation runner and the check_resume.py dry-run script.

Module Contents

Functions

Name	Description
`classify_scenario_resume_state`	Classify a scenario directory for resume purposes.
`count_agent_llm_messages`	Return the number of assistant-role messages in the agent’s saved LLM context.
`count_agent_responses`	Return how many LLM responses the agent produced for a scenario.

API

nemo_voice_agent.evaluation.resume.classify_scenario_resume_state(
    scenario_dir: str,
    min_agent_turns: int = 0
) -> typing.Tuple[str, str]

Classify a scenario directory for resume purposes.

Returns a (state, reason) tuple where state is one of:

"completed" — has a valid metrics.json and passes all stall checks; the runner will skip it and load metrics from disk.
"in_flight" — started but not cleanly finished (missing/unreadable metrics.json, 0 turns, or fewer agent LLM messages than min_agent_turns); the runner will move it aside and re-run it.
"fresh" — no subdir exists yet; the runner will run it normally.

reason is a short human-readable string explaining the classification, useful for logging and the dry-run check script.

nemo_voice_agent.evaluation.resume.count_agent_llm_messages(
    scenario_dir: str
) -> typing.Optional[int]

Return the number of assistant-role messages in the agent’s saved LLM context.

Returns None when the file is absent (old runs, crashed before write) or unreadable — callers treat None as “unknown, skip the check.”

NOTE: this file is unreliable — the agent’s bot_logs_agent/llm_context.json is frequently saved empty even for scenarios where the agent ran fine (the context is lost during end-of-scenario retrieval). Prefer count_agent_responses for stall detection; this remains only as a last-resort fallback for runs whose metrics.json predates token_usage.

nemo_voice_agent.evaluation.resume.count_agent_responses(
    scenario_dir: str,
    metrics: typing.Optional[dict] = None
) -> typing.Optional[int]

Return how many LLM responses the agent produced for a scenario.

This is the robust stall-detection signal. It prefers the live-recorded token_usage.agent.n_calls from metrics.json — that counter is accumulated by the bridge during the run and survives the agent-context save bug (see count_agent_llm_messages). Resolution order:

metrics["token_usage"]["agent"]["n_calls"] from the passed-in dict (the live runner path hands us the in-memory metrics directly).
The same field read from <scenario_dir>/metrics.json on disk (resume / load-from-disk path).
Fallback to count_agent_llm_messages for old runs whose metrics.json predates the token_usage field.

Returns None when no signal is available — callers treat None as “unknown, skip the check.”