Skip to navigation
TutorialsEvaluation TutorialsRun Multi-Turn User Simulation with NeMo User Sim

Inspect NeMo User Sim Rollouts

View as Markdown

Use this guide after completing Run Multi-Turn User Simulation with NeMo User Sim. The examples assume:

task_index=0
rollout_index=0
rollouts=results/nemo_user_sim/rollouts.jsonl
inputs=results/nemo_user_sim/rollouts_materialized_inputs.jsonl
capture_dir="$(pwd)/results/nemo_user_sim/model-calls"

To inspect the full validation run, point rollouts, inputs, and capture_dir at the same files under results/nemo_user_sim/all-probes/. Keep all three on the same run: capture files are named by task and repeat index, so a rollout from one run would otherwise open another run’s capture.

Inspect the scenario and persona

The materialized-input file is the authoritative record of the sampled inputs:

jq --argjson task "${task_index}" --argjson repeat "${rollout_index}" \
  'select(._ng_task_index == $task and ._ng_rollout_index == $repeat)
   | .task_input.resolved_row
   | {
       probe_type,
       persona: {
         id: .persona_uuid,
         name: ([.persona.first_name, .persona.last_name] | map(select(. != null)) | join(" "))
       },
       trajectory_id,
       theme
     }' \
  "${inputs}"

Inspect the conversation and tools

NeMo User Sim keeps User messages, Assistant messages, function calls, and tool results in their original order:

jq --argjson task "${task_index}" --argjson repeat "${rollout_index}" \
  'select(._ng_task_index == $task and ._ng_rollout_index == $repeat)
   | {
       probe_type: .verification.verifier_data.probe_type,
       num_turns: .usersim_result.num_turns,
       num_tool_calls: .usersim_result.num_tool_calls,
       conversation: .usersim_result.conversation_messages
     }' \
  "${rollouts}"

Read consecutive User and Assistant messages as one conversation. Do not remove intervening tool messages when diagnosing behavior.

To display only function calls and tool results:

jq --argjson task "${task_index}" --argjson repeat "${rollout_index}" \
  'select(._ng_task_index == $task and ._ng_rollout_index == $repeat)
   | .usersim_result.conversation_messages
   | map(select(.role == "tool" or ((.tool_calls // []) | length > 0)))' \
  "${rollouts}"

Interpret the score

The Resources Server calls NeMo User Sim’s TrajectoryEvaluatorGenerator.agenerate() and the scorer selected for the prepared probe. The integration evaluates Assistant behavior, not the simulated User.

Two related fields expose quality:

  • verification.verifier_data.normalized_axis_scores is the full NeMo User Sim quality map.
  • reward_components is Gym’s flat metrics projection. It repeats quality axes with a quality. prefix and adds completion/scoring flags plus assistant_quality, so aggregate metrics can consume them directly.

The duplication is intentional: the first field preserves NeMo User Sim’s structured verifier evidence, while the second uses Gym’s standard scalar metrics contract.

jq --argjson task "${task_index}" --argjson repeat "${rollout_index}" \
  'select(._ng_task_index == $task and ._ng_rollout_index == $repeat)
   | {
       conversation_status: .usersim_result.conversation_status,
       scenario_completed: .verification.scenario_completed,
       reward,
       mask_sample,
       reward_components,
       quality_scores: .verification.verifier_data.normalized_axis_scores,
       task_check: {
         name: .verification.verifier_data.scorer_name,
         result: .verification.verifier_data.scorer_result
       },
       judge_axes: .verification.verifier_data.assistant_eval.axes,
       failure_kind,
       failure_reason
     }' \
  "${rollouts}"

Interpret the fields as follows:

  • scenario_completed requires a completed conversation, both participants, and a passing applicable probe check.
  • reward is the mean of normalized helpfulness, accuracy, and coherence when scenario_completed is true and all three axes exist; otherwise it is 0.0.
  • mask_sample: false includes the reward in aggregate metrics. true preserves the row but excludes its reward.
  • scorer_result contains probe-specific evidence. A failed applicable check is a measured policy result.
  • assistant_eval.axes contains the judge’s original scores and explanations.

Inspect Environment Server invocations

The top-level invocations ledger records User and Assistant Agent activations and support-model calls made by the Environment Server:

jq --argjson task "${task_index}" --argjson repeat "${rollout_index}" \
  'select(._ng_task_index == $task and ._ng_rollout_index == $repeat)
   | .invocations[]
   | {
       sequence,
       role,
       request,
       response,
       observations,
       state_after,
       termination_reason
     }' \
  "${rollouts}"

Roles can include user, assistant, judge, summary, and tool_simulation. Use invocations to inspect support-model calls. Function calls and tool results remain in usersim_result.conversation_messages.

Inspect captured participant model calls

ng_model_call_capture and the capture JSONL files contain provider-level calls made through the User and Assistant Agents. They do not include plain support-model or Resources Server verifier calls; inspect those through invocations and verification.

jq --argjson task "${task_index}" --argjson repeat "${rollout_index}" \
  'select(._ng_task_index == $task and ._ng_rollout_index == $repeat)
   | .ng_model_call_capture
   | {rollout_id, metrics, calls}' \
  "${rollouts}"
find "${capture_dir}" -maxdepth 3 -type f -print

To inspect a capture file:

capture_rollout_id=$(jq -r --argjson task "${task_index}" --argjson repeat "${rollout_index}" \
  'select(._ng_task_index == $task and ._ng_rollout_index == $repeat)
   | .ng_model_call_capture.rollout_id' \
  "${rollouts}")
capture_file="${capture_dir}/${capture_rollout_id}.capture.jsonl"
jq -s '.[0] | {model_call_id, model_ref, status_code, latency_ms, request, response}' "${capture_file}"

For execution records normalized across Agent implementations, inspect:

jq --argjson task "${task_index}" --argjson repeat "${rollout_index}" \
  'select(._ng_task_index == $task and ._ng_rollout_index == $repeat)
   | .ng_trajectory' \
  "${rollouts}"

ng_trajectory.gaps identifies information that the participating Agent did not expose. The complete NeMo User Sim conversation remains available in usersim_result.conversation_messages.