Skip to navigation
TutorialsEvaluation Tutorials

Run Multi-Turn User Simulation with NeMo User Sim

View as Markdown

Goal: Generate the 14-task NeMo User Sim validation set, run one conversation with the assistant you want to evaluate, and confirm that the rollout contains a conversation and score.

NeMo User Sim supplies simulated users and conversation scenarios. NeMo Gym runs each scenario through your configured assistant and uses NeMo User Sim’s evaluation components inside the Resources Server to score the assistant’s responses and tool choices.

Each task exercises one NeMo User Sim probe: a scenario family, such as tool_calling or financial_services, that decides what the simulated User does and how the Assistant is scored. The task’s probe_type names its probe.

The rollout has two participants:

  • The User Agent generates messages for the sampled user persona.
  • The Assistant Agent invokes the assistant you are evaluating through policy_model.

A third endpoint, support_model, handles structured internal work such as judging response quality, producing summaries, simulating selected tool results, and running probe-specific checks.

loop [Multi-turn conversation] Prepared task row Dispatch episode request Seed session with resolved row Validated row Request next user message User message Request assistant response Message or function call Optional judge, summary, or tool simulation Structured response Verify completed episode Judge quality and run probe check Scores and evidence Reward and verification Episode response Offline preparation Gym rollout collector Environment Server Resources Server User Agent Assistant Agent Support model

Prerequisites

Install NeMo Gym as described in Installation, activate its environment, and run commands from the repository root:

source .venv/bin/activate

NeMo User Sim samples personas from Nemotron-Personas version 0.0.2 on NGC. Preparation reads en_US.parquet from Data Designer’s managed-assets directory and rejects any file whose SHA-256 differs from the pinned version. The default directory, ~/.data-designer/managed-assets/, is also where data-designer download personas stores its own release, so point preparation at a dedicated directory:

export DATA_DESIGNER_MANAGED_ASSETS_PATH="$HOME/.cache/nemo-user-sim/managed-assets"

Preparation then reads ${DATA_DESIGNER_MANAGED_ASSETS_PATH}/datasets/en_US.parquet. Setting DATA_DESIGNER_HOME instead also works; the file then goes in <path>/managed-assets/datasets/en_US.parquet. If the pinned file is not there yet, export an NGC API key:

export NGC_CLI_API_KEY="<your-ngc-key>"

Preparation then installs the NGC CLI, downloads the pinned persona resource, verifies its checksum, and caches en_US.parquet for later runs. See the API key guide to create a key. If organization discovery fails, also export NGC_CLI_ORG.

To download the file yourself, install the NGC CLI, then run:

ngc registry resource download-version \
  "nvidia/nemotron-personas/nemotron-personas-dataset-en_us:0.0.2" --dest /tmp/nemotron-personas-0.0.2
mkdir -p "${DATA_DESIGNER_MANAGED_ASSETS_PATH}/datasets"
find /tmp/nemotron-personas-0.0.2 -name en_US.parquet \
  -exec cp {} "${DATA_DESIGNER_MANAGED_ASSETS_PATH}/datasets/en_US.parquet" \;

A persona file from another release, such as the latest data-designer download personas download, fails the checksum. Preparation does not download over a mismatched file. Set DATA_DESIGNER_MANAGED_ASSETS_PATH to a directory without that file and rerun preparation.

Create or update env.yaml in the repository root:

user_model_base_url: https://your-user-model-endpoint.example.com/v1
user_model_api_key: ${oc.env:USER_MODEL_API_KEY}
user_model_name: your-user-simulation-model
policy_base_url: https://your-policy-endpoint.example.com/v1
policy_api_key: ${oc.env:POLICY_API_KEY}
policy_model_name: your-assistant-model
support_model_base_url: https://your-support-endpoint.example.com/v1
support_model_api_key: ${oc.env:SUPPORT_MODEL_API_KEY}
support_model_name: your-structured-output-model

The user_model_* keys are required: the simulated User runs on its own model, separate from the Assistant you are evaluating, and Gym stops with an error that names any unset key. support_model_* defaults to the policy_* values; set it only when the support role uses a different endpoint.

All three URLs must expose an OpenAI-compatible Chat Completions API. The support endpoint must also accept json_schema response formats. If it rejects the probe-scorer schema, verification is masked as verifier_error. If it rejects the judge schema, NeMo User Sim returns no quality axes, and a completed scenario is masked as judge_failed.

The financial_services probe also retrieves documents through an embedding endpoint. embedding_model_base_url, embedding_model_api_key, and embedding_model_name default to the support_model_* values, and Gym sends embedding requests to <embedding_model_base_url>/embeddings. If that endpoint does not serve embeddings, retrieval falls back to lexical search without failing the episode, so set these three keys when the support endpoint is chat-only.

Export the referenced secrets:

export USER_MODEL_API_KEY="<your-api-key>"
export POLICY_API_KEY="<your-api-key>"
export SUPPORT_MODEL_API_KEY="<your-api-key>"

Generate the validation tasks

Ask Gym to run the validation dataset’s declared preparation script:

gym eval prepare --config environments/nemo_user_sim/config.yaml

The script runs the pinned NeMo User Sim materializer in an isolated dependency environment and writes environments/nemo_user_sim/data/nemo_user_sim.jsonl. Its 14 deterministic rows contain one task for each built-in probe. NeMo User Sim resolves the persona, scenario, tools, seed, trajectory identity, and provenance before rollout collection begins.

Confirm that all probe tasks were generated and inspect them:

wc -l environments/nemo_user_sim/data/nemo_user_sim.jsonl
jq '.resolved_row | {
  probe_type, trajectory_id, usersim_provenance, usersim_config
}' environments/nemo_user_sim/data/nemo_user_sim.jsonl

When it seeds a session, the Resources Server validates required fields, the pinned NeMo User Sim revision, and the Nemotron-Personas version ("0.0.2", or "synthetic" for the committed example rows), and rejects guarded health rows. It stores a digest of the resolved row (resolved_row) and compares it during verification, proving that the resolved row did not change during the episode.

Collect one rollout

Start with one task and one repeat:

mkdir -p results/nemo_user_sim/model-calls
capture_dir="$(pwd)/results/nemo_user_sim/model-calls"
gym eval run \
  --environment nemo_user_sim \
  --split validation \
  --output results/nemo_user_sim/rollouts.jsonl \
  --limit 1 \
  --num-repeats 1 \
  +user_model_uses_reasoning_parser=false \
  +policy_uses_reasoning_parser=false \
  +support_model_uses_reasoning_parser=false \
  ++observability_enabled=true \
  ++model_call_capture_dir="${capture_dir}"

The parser flags above use the configuration defaults. Set one to true only when that endpoint returns a separate reasoning field that Gym must parse.

The first generated task is financial_services, so this smoke run calls the embedding endpoint described above. If that endpoint does not serve embeddings, the run still completes, with lexical retrieval.

A successful run writes:

  • results/nemo_user_sim/rollouts.jsonl: completed episodes and scores.
  • results/nemo_user_sim/rollouts_materialized_inputs.jsonl: the exact input used for each attempt.
  • results/nemo_user_sim/rollouts_failures.jsonl: attempts that failed before a completed episode was saved.
  • results/nemo_user_sim/model-calls/: captured participant model calls.

Inspect the result

Read the ordered conversation:

jq '{
  probe_type: .verification.verifier_data.probe_type,
  conversation: .usersim_result.conversation_messages,
  reward,
  mask_sample
}' results/nemo_user_sim/rollouts.jsonl

reward is the mean of normalized helpfulness, accuracy, and coherence when scenario_completed is true and all three axes are available; otherwise it is 0.0. mask_sample: false includes the reward in aggregate metrics. A masked rollout remains available for diagnosis but is excluded from aggregation.

Inspect the exact participant requests and responses:

jq '.invocations[]
  | select(.role == "user" or .role == "assistant")
  | {sequence, role, request, response, termination_reason}' \
  results/nemo_user_sim/rollouts.jsonl

The smoke run is complete when the rollout contains both User and Assistant messages, verification.verifier_data.assistant_eval has no error, and mask_sample is false.

Run all validation tasks

After the smoke rollout succeeds, remove --limit 1 to run all 14 generated probe tasks. Give the full run its own directory: capture files are keyed by task and repeat, and the health report is written next to the output file, so a shared directory would replace the smoke run’s captures and quality_summary.json.

mkdir -p results/nemo_user_sim/all-probes/model-calls
gym eval run \
  --environment nemo_user_sim \
  --split validation \
  --output results/nemo_user_sim/all-probes/rollouts.jsonl \
  --num-repeats 1 \
  +user_model_uses_reasoning_parser=false \
  +policy_uses_reasoning_parser=false \
  +support_model_uses_reasoning_parser=false \
  ++observability_enabled=true \
  ++model_call_capture_dir="$(pwd)/results/nemo_user_sim/all-probes/model-calls"

To inspect or troubleshoot the full run with the guides below, replace results/nemo_user_sim/ with results/nemo_user_sim/all-probes/ in their paths.

Next steps