> For clean Markdown of any page, append .md to the page URL.
> For a complete documentation index, see https://docs.nvidia.com/nemo/gym/main/tutorials/evaluation-tutorials/nemo-user-sim-rollout-collection/llms.txt.
> For AI client integration (Claude Code, Cursor, etc.), connect to the MCP server at https://docs.nvidia.com/nemo/gym/_mcp/server.

# Run Multi-Turn User Simulation with NeMo User Sim

> Generate NeMo User Sim validation tasks, evaluate an assistant, and inspect one rollout.

> **Info**
>
> **Goal:** Generate the 14-task NeMo User Sim validation set, run one conversation with the assistant you want
> to evaluate, and confirm that the rollout contains a conversation and score.

[NeMo User Sim](https://github.com/NVIDIA-NeMo/UserSim) supplies simulated users and conversation scenarios.
NeMo Gym runs each scenario through your configured assistant and uses NeMo User Sim's evaluation components
inside the Resources Server to score the assistant's responses and tool choices.

Each task exercises one NeMo User Sim [probe](https://github.com/NVIDIA-NeMo/UserSim/blob/a5f676bf6dc5a73914c8a0860f97c10dd2c214ee/docs/probes.md):
a scenario family, such as `tool_calling` or `financial_services`, that decides what the simulated User does and
how the Assistant is scored. The task's `probe_type` names its probe.

The rollout has two participants:

* The **User Agent** generates messages for the sampled user persona.
* The **Assistant Agent** invokes the assistant you are evaluating through `policy_model`.

A third endpoint, `support_model`, handles structured internal work such as judging response quality, producing
summaries, simulating selected tool results, and running probe-specific checks.

```mermaid
sequenceDiagram
    participant P as Offline preparation
    participant C as Gym rollout collector
    participant E as Environment Server
    participant R as Resources Server
    participant U as User Agent
    participant A as Assistant Agent
    participant S as Support model

    P-->>C: Prepared task row
    C->>E: Dispatch episode request
    E->>R: Seed session with resolved row
    R-->>E: Validated row
    loop Multi-turn conversation
        E->>U: Request next user message
        U-->>E: User message
        E->>A: Request assistant response
        A-->>E: Message or function call
        E->>S: Optional judge, summary, or tool simulation
        S-->>E: Structured response
    end
    E->>R: Verify completed episode
    R->>S: Judge quality and run probe check
    S-->>R: Scores and evidence
    R-->>E: Reward and verification
    E-->>C: Episode response
```

## Prerequisites

Install NeMo Gym as described in [Installation](/get-started/installation), activate its environment, and
run commands from the repository root:

```bash
source .venv/bin/activate
```

NeMo User Sim samples personas from Nemotron-Personas version 0.0.2 on NGC. Preparation reads `en_US.parquet` from
Data Designer's managed-assets directory and rejects any file whose SHA-256 differs from the pinned version. The
default directory, `~/.data-designer/managed-assets/`, is also where `data-designer download personas` stores its
own release, so point preparation at a dedicated directory:

```bash
export DATA_DESIGNER_MANAGED_ASSETS_PATH="$HOME/.cache/nemo-user-sim/managed-assets"
```

Preparation then reads `${DATA_DESIGNER_MANAGED_ASSETS_PATH}/datasets/en_US.parquet`. Setting `DATA_DESIGNER_HOME`
instead also works; the file then goes in `<path>/managed-assets/datasets/en_US.parquet`. If the pinned file is not
there yet, export an NGC API key:

```bash
export NGC_CLI_API_KEY="<your-ngc-key>"
```

Preparation then installs the NGC CLI, downloads the pinned persona resource, verifies its checksum, and caches
`en_US.parquet` for later runs. See the
[API key guide](https://docs.nvidia.com/ngc/latest/ngc-catalog-user-guide.html#ngc-api-keys) to create a key. If
organization discovery fails, also export `NGC_CLI_ORG`.

To download the file yourself, install the
[NGC CLI](https://docs.nvidia.com/ngc/latest/ngc-catalog-user-guide.html#installing-ngc-catalog-cli), then run:

```bash
ngc registry resource download-version \
  "nvidia/nemotron-personas/nemotron-personas-dataset-en_us:0.0.2" --dest /tmp/nemotron-personas-0.0.2
mkdir -p "${DATA_DESIGNER_MANAGED_ASSETS_PATH}/datasets"
find /tmp/nemotron-personas-0.0.2 -name en_US.parquet \
  -exec cp {} "${DATA_DESIGNER_MANAGED_ASSETS_PATH}/datasets/en_US.parquet" \;
```

A persona file from another release, such as the latest `data-designer download personas` download, fails the
checksum. Preparation does not download over a mismatched file. Set `DATA_DESIGNER_MANAGED_ASSETS_PATH` to a
directory without that file and rerun preparation.

Create or update `env.yaml` in the repository root:

```yaml
user_model_base_url: https://your-user-model-endpoint.example.com/v1
user_model_api_key: ${oc.env:USER_MODEL_API_KEY}
user_model_name: your-user-simulation-model

policy_base_url: https://your-policy-endpoint.example.com/v1
policy_api_key: ${oc.env:POLICY_API_KEY}
policy_model_name: your-assistant-model

support_model_base_url: https://your-support-endpoint.example.com/v1
support_model_api_key: ${oc.env:SUPPORT_MODEL_API_KEY}
support_model_name: your-structured-output-model
```

The `user_model_*` keys are required: the simulated User runs on its own model, separate from the Assistant you are
evaluating, and Gym stops with an error that names any unset key. `support_model_*` defaults to the `policy_*`
values; set it only when the support role uses a different endpoint.

All three URLs must expose an OpenAI-compatible Chat Completions API. The support endpoint must also accept
`json_schema` response formats. If it rejects the probe-scorer schema, verification is masked as `verifier_error`.
If it rejects the judge schema, NeMo User Sim returns no quality axes, and a completed scenario is masked as
`judge_failed`.

The `financial_services` probe also retrieves documents through an embedding endpoint. `embedding_model_base_url`,
`embedding_model_api_key`, and `embedding_model_name` default to the `support_model_*` values, and Gym sends
embedding requests to `<embedding_model_base_url>/embeddings`. If that endpoint does not serve embeddings, retrieval
falls back to lexical search without failing the episode, so set these three keys when the support endpoint is
chat-only.

Export the referenced secrets:

```bash
export USER_MODEL_API_KEY="<your-api-key>"
export POLICY_API_KEY="<your-api-key>"
export SUPPORT_MODEL_API_KEY="<your-api-key>"
```

## Generate the validation tasks

Ask Gym to run the validation dataset's declared preparation script:

```bash
gym eval prepare --config environments/nemo_user_sim/config.yaml
```

The script runs the pinned NeMo User Sim materializer in an isolated dependency environment and writes
`environments/nemo_user_sim/data/nemo_user_sim.jsonl`. Its 14 deterministic rows contain one task for each
built-in probe. NeMo User Sim resolves the persona, scenario, tools, seed, trajectory identity, and provenance
before rollout collection begins.

Confirm that all probe tasks were generated and inspect them:

```bash
wc -l environments/nemo_user_sim/data/nemo_user_sim.jsonl
jq '.resolved_row | {
  probe_type, trajectory_id, usersim_provenance, usersim_config
}' environments/nemo_user_sim/data/nemo_user_sim.jsonl
```

When it seeds a session, the Resources Server validates required fields, the pinned NeMo User Sim revision, and
the Nemotron-Personas version (`"0.0.2"`, or `"synthetic"` for the committed example rows), and rejects guarded
health rows. It stores a digest of the resolved row (`resolved_row`) and compares it during verification, proving
that the resolved row did not change during the episode.

## Collect one rollout

Start with one task and one repeat:

```bash
mkdir -p results/nemo_user_sim/model-calls
capture_dir="$(pwd)/results/nemo_user_sim/model-calls"

gym eval run \
  --environment nemo_user_sim \
  --split validation \
  --output results/nemo_user_sim/rollouts.jsonl \
  --limit 1 \
  --num-repeats 1 \
  +user_model_uses_reasoning_parser=false \
  +policy_uses_reasoning_parser=false \
  +support_model_uses_reasoning_parser=false \
  ++observability_enabled=true \
  ++model_call_capture_dir="${capture_dir}"
```

The parser flags above use the configuration defaults. Set one to `true` only when that endpoint returns a
separate reasoning field that Gym must parse.

The first generated task is `financial_services`, so this smoke run calls the embedding endpoint described above.
If that endpoint does not serve embeddings, the run still completes, with lexical retrieval.

A successful run writes:

* `results/nemo_user_sim/rollouts.jsonl`: completed episodes and scores.
* `results/nemo_user_sim/rollouts_materialized_inputs.jsonl`: the exact input used for each attempt.
* `results/nemo_user_sim/rollouts_failures.jsonl`: attempts that failed before a completed episode was saved.
* `results/nemo_user_sim/model-calls/`: captured participant model calls.

## Inspect the result

Read the ordered conversation:

```bash
jq '{
  probe_type: .verification.verifier_data.probe_type,
  conversation: .usersim_result.conversation_messages,
  reward,
  mask_sample
}' results/nemo_user_sim/rollouts.jsonl
```

`reward` is the mean of normalized helpfulness, accuracy, and coherence when `scenario_completed` is `true`
and all three axes are available; otherwise it is `0.0`. `mask_sample: false` includes the reward in aggregate
metrics. A masked rollout remains available for diagnosis but is excluded from aggregation.

Inspect the exact participant requests and responses:

```bash
jq '.invocations[]
  | select(.role == "user" or .role == "assistant")
  | {sequence, role, request, response, termination_reason}' \
  results/nemo_user_sim/rollouts.jsonl
```

The smoke run is complete when the rollout contains both User and Assistant messages,
`verification.verifier_data.assistant_eval` has no `error`, and `mask_sample` is `false`.

## Run all validation tasks

After the smoke rollout succeeds, remove `--limit 1` to run all 14 generated probe tasks. Give the full run its
own directory: capture files are keyed by task and repeat, and the health report is written next to the output
file, so a shared directory would replace the smoke run's captures and `quality_summary.json`.

```bash
mkdir -p results/nemo_user_sim/all-probes/model-calls

gym eval run \
  --environment nemo_user_sim \
  --split validation \
  --output results/nemo_user_sim/all-probes/rollouts.jsonl \
  --num-repeats 1 \
  +user_model_uses_reasoning_parser=false \
  +policy_uses_reasoning_parser=false \
  +support_model_uses_reasoning_parser=false \
  ++observability_enabled=true \
  ++model_call_capture_dir="$(pwd)/results/nemo_user_sim/all-probes/model-calls"
```

To inspect or troubleshoot the full run with the guides below, replace `results/nemo_user_sim/` with
`results/nemo_user_sim/all-probes/` in their paths.

## Next steps

* [Inspect NeMo User Sim rollouts](/tutorials/evaluation-tutorials/nemo-user-sim-rollout-collection/rollout-inspection) for
  conversations, tools, scores, invocations, and model-call captures.
* [Troubleshoot NeMo User Sim rollouts](/tutorials/evaluation-tutorials/nemo-user-sim-rollout-collection/rollout-troubleshooting) for
  collection, simulation, and grading failures.
* [NeMo User Sim integration reference](/tutorials/evaluation-tutorials/nemo-user-sim-rollout-collection/integration-reference) for
  architecture, ownership, contracts, and field semantics.

> Generate NeMo User Sim validation tasks, evaluate an assistant, and inspect one rollout.

## Docs

- [Inspect NeMo User Sim Rollouts](https://docs.nvidia.com/nemo/gym/main/tutorials/evaluation-tutorials/nemo-user-sim-rollout-collection/rollout-inspection.md): Inspect conversations, tools, scores, invocations, and captured participant model calls.
- [Troubleshoot NeMo User Sim Rollouts](https://docs.nvidia.com/nemo/gym/main/tutorials/evaluation-tutorials/nemo-user-sim-rollout-collection/rollout-troubleshooting.md): Diagnose collection, simulation, and grading failures in NeMo User Sim evaluations.
- [NeMo User Sim Integration Reference](https://docs.nvidia.com/nemo/gym/main/tutorials/evaluation-tutorials/nemo-user-sim-rollout-collection/integration-reference.md): Architecture, ownership, request contracts, and saved fields for the NeMo User Sim integration.