Troubleshoot NeMo User Sim Rollouts

Start with the output files:

• results/nemo_user_sim/rollouts.jsonl contains completed episode responses, including measured failures.
• results/nemo_user_sim/rollouts_failures.jsonl contains attempts that failed before Gym could save a completed episode.
• results/nemo_user_sim/quality_summary.json summarizes rollout health.

For the full validation run, use the same files under results/nemo_user_sim/all-probes/.

A zero reward does not by itself mean that the Assistant performed poorly. Check the failure stage and mask_sample before interpreting it.

Collection and Environment Server failures

Exceptions that escape the Environment Server runtime do not produce usersim_result. Gym records them in the failures sidecar. This includes participant transport failures, provider failures, session errors, and unexpected runtime exceptions:

jq '{
  rollout_id: "\(._ng_task_index):\(._ng_rollout_index)",
  class: ._ng_failure_class,
  stage: ._ng_failure_stage,
  message: ._ng_failure_message
}' results/nemo_user_sim/rollouts_failures.jsonl

_ng_failure_stage is admission, seed, agent, verification, or cleanup, and is absent when the collector could not reach the Environment Server. _ng_failure_class is environment_server_failed for every failure the Environment Server reports, so use the stage and _ng_failure_message to diagnose it. These are infrastructure or execution failures, not measured Assistant-quality results.

Guarded health rows are rejected when the Resources Server seeds the session, so they appear here with _ng_failure_stage: "seed".

Simulation failures recorded in the episode

Failures that NeMo User Sim handles internally, and Assistant activation errors that the Environment Server classifies as Assistant mistakes, still produce a completed episode response. Inspect:

jq '{
  simulation_outcome: .usersim_result.simulation_outcome,
  reward,
  mask_sample,
  failure_kind,
  failure_reason
}' results/nemo_user_sim/rollouts.jsonl

Interpret attribution carefully:

• A simulation failure attributed to assistant_model is a measured Assistant failure: reward is 0.0 and mask_sample is false.
• A simulation failure attributed elsewhere is not evidence about the Assistant: reward is 0.0 and mask_sample is true.

failure_kind is derived from failure_class. A class that is already a registered Gym failure kind or already contains a namespace is kept as is. Any other class is lowercased, characters other than letters, digits, and underscores become underscores, and usersim: is prepended, for example usersim:assistant_activation_error.

Assistant-caused invalid responses and non-retryable Assistant 4xx responses follow the measured-failure path. Participant timeouts, provider/transport failures, retryable status codes, and session-plumbing failures remain failures outside the scored episode.

Grading failures

The Resources Server evaluates a completed conversation and runs an applicable probe scorer. Inspect its evidence:

jq '{
  reward,
  mask_sample,
  failure_kind,
  failure_reason,
  scorer_state: .verification.verifier_data.scorer_state,
  scorer_result: .verification.verifier_data.scorer_result,
  assistant_eval: .verification.verifier_data.assistant_eval
}' results/nemo_user_sim/rollouts.jsonl

The main cases are:

• A probe returns status_proposal: false: measured policy failure, unmasked zero reward.
• A probe returns status_proposal: null: inconclusive result, masked as usersim:scorer_inconclusive.
• A scorer or trajectory evaluator raises or returns an error: masked as verifier_error.
• A completed scenario has no Assistant quality score: masked as judge_failed.
• A verifier timeout: masked; failure_reason reports the configured evaluation timeout.
• The Resources Server session is gone at verification: masked as session_lost, or session_expired when the seed outlived session_ttl_seconds. See Session cleanup failures.

NeMo User Sim scorer results classified as not_applicable, including its documented skipped sentinel, do not gate the episode.

Support-model schema errors

All configured upstream URLs must expose an OpenAI-compatible Chat Completions API. The support endpoint must accept JSON Schema response formats. Either kind of schema rejection produces a masked result rather than a measured Assistant result:

• A rejected probe-scorer schema is masked as verifier_error.
• A rejected judge schema leaves NeMo User Sim with no quality axes, so a completed scenario is masked as judge_failed.

Verify the support_model_base_url, credentials, model name, and structured-output support in env.yaml.

Missing or unexpected captures

ng_model_call_capture contains calls routed through the participating User and Assistant Agents. Support calls and final verifier calls use plain model routes and are not part of that per-rollout capture.

Use these sources instead:

• Top-level invocations: User, Assistant, judge, summary, and tool-simulation calls made by the Environment Server.
• verification.verifier_data: final evaluator and probe-scorer evidence.
• Server logs: provider request failures and schema errors.

Prepared-row mismatch

At seed time, the Resources Server validates required fields, the pinned NeMo User Sim revision, and the Nemotron-Personas version ("0.0.2", or "synthetic" for the committed example rows), and rejects guarded health rows. It stores a canonical digest of the resolved row (resolved_row). At verification, a changed resolved row or mismatched task, episode, or trajectory identity is rejected.

The digest proves that the resolved row remained unchanged during the episode; it does not independently prove that the row equals an earlier preparation artifact. Regenerate the validation rows with:

gym eval prepare --config environments/nemo_user_sim/config.yaml

Session cleanup failures

The Environment Server registers cleanup before seeding and closes participant and Resources Server sessions on every outcome. Cleanup is idempotent. Verifying an unknown or closed Resources Server session returns a masked zero reward with failure_kind session_lost; a session whose seed outlived session_ttl_seconds returns session_expired.

When diagnosing session errors, preserve the failure stage and HTTP status. Session-related 404 and 409 responses are infrastructure failures, not Assistant-policy failures.