> This page is for version Main (default).
> For other versions, use one of these documentation indexes:
> - Main (default): https://docs.nvidia.com/nemo/gym/main/llms.txt
> - 0.7.0: https://docs.nvidia.com/nemo/gym/v0.7.0/llms.txt
> - 0.6.0: https://docs.nvidia.com/nemo/gym/v0.6.0/llms.txt
> - 0.5.1: https://docs.nvidia.com/nemo/gym/v0.5.1/llms.txt
> - 0.5.0: https://docs.nvidia.com/nemo/gym/v0.5.0/llms.txt
> - 0.4.0: https://docs.nvidia.com/nemo/gym/v0.4.0/llms.txt
> - 0.3.0: https://docs.nvidia.com/nemo/gym/v0.3.0/llms.txt
> - 0.2.1: https://docs.nvidia.com/nemo/gym/v0.2.1/llms.txt

> For clean Markdown of any page, append .md to the page URL.
> For a complete documentation index, see https://docs.nvidia.com/nemo/gym/llms.txt.
> For AI client integration (Claude Code, Cursor, etc.), connect to the MCP server at https://docs.nvidia.com/nemo/gym/_mcp/server.

# Troubleshoot NeMo User Sim Rollouts

> Diagnose collection, simulation, and grading failures in NeMo User Sim evaluations.

Start with the output files:

* `results/nemo_user_sim/rollouts.jsonl` contains completed episode responses, including measured failures.
* `results/nemo_user_sim/rollouts_failures.jsonl` contains attempts that failed before Gym could save a completed episode.
* `results/nemo_user_sim/quality_summary.json` summarizes rollout health.

For the full validation run, use the same files under `results/nemo_user_sim/all-probes/`.

A zero reward does not by itself mean that the Assistant performed poorly. Check the failure stage and
`mask_sample` before interpreting it.

## Collection and Environment Server failures

Exceptions that escape the Environment Server runtime do not produce `usersim_result`. Gym records them in the
failures sidecar. This includes participant transport failures, provider failures, session errors, and unexpected
runtime exceptions:

```bash
jq '{
  rollout_id: "\(._ng_task_index):\(._ng_rollout_index)",
  class: ._ng_failure_class,
  stage: ._ng_failure_stage,
  message: ._ng_failure_message
}' results/nemo_user_sim/rollouts_failures.jsonl
```

`_ng_failure_stage` is `admission`, `seed`, `agent`, `verification`, or `cleanup`, and is absent when the collector
could not reach the Environment Server. `_ng_failure_class` is `environment_server_failed` for every failure the
Environment Server reports, so use the stage and `_ng_failure_message` to diagnose it. These are infrastructure or
execution failures, not measured Assistant-quality results.

Guarded health rows are rejected when the Resources Server seeds the session, so they appear here with
`_ng_failure_stage: "seed"`.

## Simulation failures recorded in the episode

Failures that NeMo User Sim handles internally, and Assistant activation errors that the Environment Server
classifies as Assistant mistakes, still produce a completed episode response. Inspect:

```bash
jq '{
  simulation_outcome: .usersim_result.simulation_outcome,
  reward,
  mask_sample,
  failure_kind,
  failure_reason
}' results/nemo_user_sim/rollouts.jsonl
```

Interpret attribution carefully:

* A simulation failure attributed to `assistant_model` is a measured Assistant failure: `reward` is `0.0` and
  `mask_sample` is `false`.
* A simulation failure attributed elsewhere is not evidence about the Assistant: `reward` is `0.0` and
  `mask_sample` is `true`.

`failure_kind` is derived from `failure_class`. A class that is already a registered Gym failure kind or already
contains a namespace is kept as is. Any other class is lowercased, characters other than letters, digits, and
underscores become underscores, and `usersim:` is prepended, for example `usersim:assistant_activation_error`.

Assistant-caused invalid responses and non-retryable Assistant 4xx responses follow the measured-failure path.
Participant timeouts, provider/transport failures, retryable status codes, and session-plumbing failures remain failures
outside the scored episode.

## Grading failures

The Resources Server evaluates a completed conversation and runs an applicable probe scorer. Inspect its evidence:

```bash
jq '{
  reward,
  mask_sample,
  failure_kind,
  failure_reason,
  scorer_state: .verification.verifier_data.scorer_state,
  scorer_result: .verification.verifier_data.scorer_result,
  assistant_eval: .verification.verifier_data.assistant_eval
}' results/nemo_user_sim/rollouts.jsonl
```

The main cases are:

* A probe returns `status_proposal: false`: measured policy failure, unmasked zero reward.
* A probe returns `status_proposal: null`: inconclusive result, masked as `usersim:scorer_inconclusive`.
* A scorer or trajectory evaluator raises or returns an error: masked as `verifier_error`.
* A completed scenario has no Assistant quality score: masked as `judge_failed`.
* A verifier timeout: masked; `failure_reason` reports the configured evaluation timeout.
* The Resources Server session is gone at verification: masked as `session_lost`, or `session_expired` when the seed
  outlived `session_ttl_seconds`. See [Session cleanup failures](#session-cleanup-failures).

NeMo User Sim scorer results classified as `not_applicable`, including its documented skipped sentinel, do not
gate the episode.

## Support-model schema errors

All configured upstream URLs must expose an OpenAI-compatible Chat Completions API. The support endpoint must
accept JSON Schema response formats. Either kind of schema rejection produces a masked result rather than a measured
Assistant result:

* A rejected probe-scorer schema is masked as `verifier_error`.
* A rejected judge schema leaves NeMo User Sim with no quality axes, so a completed scenario is masked as
  `judge_failed`.

Verify the `support_model_base_url`, credentials, model name, and structured-output support in `env.yaml`.

## Missing or unexpected captures

`ng_model_call_capture` contains calls routed through the participating User and Assistant Agents. Support calls
and final verifier calls use plain model routes and are not part of that per-rollout capture.

Use these sources instead:

* Top-level `invocations`: User, Assistant, judge, summary, and tool-simulation calls made by the Environment Server.
* `verification.verifier_data`: final evaluator and probe-scorer evidence.
* Server logs: provider request failures and schema errors.

## Prepared-row mismatch

At seed time, the Resources Server validates required fields, the pinned NeMo User Sim revision, and the
Nemotron-Personas version (`"0.0.2"`, or `"synthetic"` for the committed example rows), and rejects guarded health
rows. It stores a canonical digest of the resolved row (`resolved_row`). At verification, a changed resolved row or
mismatched task, episode, or trajectory identity is rejected.

The digest proves that the resolved row remained unchanged during the episode; it does not independently prove that the
row equals an earlier preparation artifact. Regenerate the validation rows with:

```bash
gym eval prepare --config environments/nemo_user_sim/config.yaml
```

## Session cleanup failures

The Environment Server registers cleanup before seeding and closes participant and Resources Server sessions on every
outcome. Cleanup is idempotent. Verifying an unknown or closed Resources Server session returns a masked zero reward with
`failure_kind` `session_lost`; a session whose seed outlived `session_ttl_seconds` returns `session_expired`.

When diagnosing session errors, preserve the failure stage and HTTP status. Session-related 404 and 409 responses
are infrastructure failures, not Assistant-policy failures.