> This page is for version 0.4.0. > For other versions, use one of these documentation indexes: > - Main (default): https://docs.nvidia.com/nemo/gym/main/llms.txt > - 0.7.0: https://docs.nvidia.com/nemo/gym/v0.7.0/llms.txt > - 0.6.0: https://docs.nvidia.com/nemo/gym/v0.6.0/llms.txt > - 0.5.1: https://docs.nvidia.com/nemo/gym/v0.5.1/llms.txt > - 0.5.0: https://docs.nvidia.com/nemo/gym/v0.5.0/llms.txt > - 0.4.0: https://docs.nvidia.com/nemo/gym/v0.4.0/llms.txt > - 0.3.0: https://docs.nvidia.com/nemo/gym/v0.3.0/llms.txt > - 0.2.1: https://docs.nvidia.com/nemo/gym/v0.2.1/llms.txt > For clean Markdown of any page, append .md to the page URL. > For a complete documentation index, see https://docs.nvidia.com/nemo/gym/llms.txt. > For AI client integration (Claude Code, Cursor, etc.), connect to the MCP server at https://docs.nvidia.com/nemo/gym/_mcp/server. # nemo_gym.rollout_compare Per-task parity report between two rollout JSONL files, the gym dev compare command. A migration aid: before an old resources server is deleted in favour of its Harbor-path replacement, both are run on the same tasks and this report is the gate. It joins the rows by task, classifies every task as identical, flipped, masked or missing, and prints the losing side's verifier output for every flip. Rows from either path are accepted, and masked failures from the failures sidecar are folded back in. The package re-exports the public names so that nemo_gym.cli.dev and the tests import them from nemo_gym.rollout_compare unchanged. ## Submodules * **[`nemo_gym.rollout_compare.command`](/nemo/gym/nemo-gym/nemo_gym/rollout_compare/command)** * **[`nemo_gym.rollout_compare.compare`](/nemo/gym/nemo-gym/nemo_gym/rollout_compare/compare)** * **[`nemo_gym.rollout_compare.join`](/nemo/gym/nemo-gym/nemo_gym/rollout_compare/join)** * **[`nemo_gym.rollout_compare.report`](/nemo/gym/nemo-gym/nemo_gym/rollout_compare/report)** * **[`nemo_gym.rollout_compare.rows`](/nemo/gym/nemo-gym/nemo_gym/rollout_compare/rows)** * **[`nemo_gym.rollout_compare.schema`](/nemo/gym/nemo-gym/nemo_gym/rollout_compare/schema)** ## Package Contents ### Classes | Name | Description | | -------------------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------- | | [`CompareInputError`](#nemo_gym-rollout_compare-schema-CompareInputError) | An input file could not be read or does not look like rollouts. | | [`CompareResult`](#nemo_gym-rollout_compare-schema-CompareResult) | - | | [`RolloutCompareConfig`](#nemo_gym-rollout_compare-command-RolloutCompareConfig) | Compare two rollout JSONL files task by task: count identical, flipped, masked and missing tasks and print the | | [`TaskComparison`](#nemo_gym-rollout_compare-schema-TaskComparison) | - | | [`TaskSide`](#nemo_gym-rollout_compare-schema-TaskSide) | One side's rows for a task, reduced to what the comparison needs. | ### Functions | Name | Description | | ------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------- | | [`agent_error`](#nemo_gym-rollout_compare-rows-agent_error) | An agent-level error the row reports without being masked: the legacy agent's `error` field or an | | [`compare_rollouts`](#nemo_gym-rollout_compare-compare-compare_rollouts) | - | | [`current_failures`](#nemo_gym-rollout_compare-rows-current_failures) | The sidecar records that still stand: the collector writes one record per attempt and a retried rollout | | [`detect_task_key`](#nemo_gym-rollout_compare-join-detect_task_key) | The first candidate field every row carries. Sidecar failure rows that name no task at all are left out | | [`group_by_task`](#nemo_gym-rollout_compare-join-group_by_task) | (rows grouped by task label, anonymous rows). An anonymous row is a sidecar failure that names no task: | | [`load_rollouts`](#nemo_gym-rollout_compare-rows-load_rollouts) | Read a rollouts JSONL plus, when present, its sibling `<name>_failures.jsonl` of masked failures. A failure | | [`mask_reason`](#nemo_gym-rollout_compare-rows-mask_reason) | Why a row does not count: `mask_sample`, no reward, or a NaN/infinite reward. | | [`measured_failure_kind`](#nemo_gym-rollout_compare-rows-measured_failure_kind) | The `failure_kind` of a row that still counts (not masked): a scored failure the report labels but does | | [`render_report`](#nemo_gym-rollout_compare-report-render_report) | - | | [`run_compare`](#nemo_gym-rollout_compare-command-run_compare) | Load both sides, print the report (or the JSON summary with `json_stdout`) and write the JSON file when | | [`summarize_side`](#nemo_gym-rollout_compare-rows-summarize_side) | - | | [`summary_dict`](#nemo_gym-rollout_compare-report-summary_dict) | - | | [`tail_lines`](#nemo_gym-rollout_compare-report-tail_lines) | - | | [`task_id_of`](#nemo_gym-rollout_compare-join-task_id_of) | The label a row is joined and listed under: the plain task id, or `taskset/task_id` with `with_taskset` | | [`tasksets_of`](#nemo_gym-rollout_compare-join-tasksets_of) | - | | [`verifier_output`](#nemo_gym-rollout_compare-report-verifier_output) | (text, source) for a row's verifier stdout: the file under `verifier_logs_dir` when it exists, else the | ### API ```python class nemo_gym.rollout_compare.CompareInputError() ``` Exception **Bases:** `Exception` An input file could not be read or does not look like rollouts. ```python class nemo_gym.rollout_compare.CompareResult( tasks: list[nemo_gym.rollout_compare.schema.TaskComparison], old_key: str, new_key: str, old_rows: int, new_rows: int, old_masked_rows: int = 0, new_masked_rows: int = 0, old_prefix_stripped: typing.Optional[str] = None, new_prefix_stripped: typing.Optional[str] = None, keyed_by_taskset: bool = False, notes: list[str] = list() ) ``` Dataclass **`keyed_by_taskset`** `bool = False` --- **`new_key`** `str` --- **`new_masked_rows`** `int = 0` --- **`new_prefix_stripped`** `Optional[str] = None` --- **`new_rows`** `int` --- **`notes`** `list[str] = field(default_factory=list)` --- **`old_key`** `str` --- **`old_masked_rows`** `int = 0` --- **`old_prefix_stripped`** `Optional[str] = None` --- **`old_rows`** `int` --- **`tasks`** `list[TaskComparison]` --- ```python nemo_gym.rollout_compare.CompareResult.by_category( category: nemo_gym.rollout_compare.schema.Category ) -> list[nemo_gym.rollout_compare.schema.TaskComparison] ``` ```python nemo_gym.rollout_compare.CompareResult.counts() -> dict[str, int] ``` ```python class nemo_gym.rollout_compare.RolloutCompareConfig() ``` **Bases:** [BaseNeMoGymCLIConfig](/nemo/gym/nemo-gym/nemo_gym/config_types#nemo_gym-config_types-BaseNeMoGymCLIConfig) Compare two rollout JSONL files task by task: count identical, flipped, masked and missing tasks and print the losing side's verifier output for every flip. A report, not a gate: the exit code is 0 for a report and 1 when an input cannot be read or the `--json` summary cannot be written. Examples: ```python # Old server vs Harbor path on the same tasks gym dev compare old/rollouts.jsonl new/rollouts.jsonl # Resolve relative `verifier_logs_dir` paths against the Harbor server directory, keep 30 log lines per flip, # and also write a machine-readable summary gym dev compare old.jsonl new.jsonl --logs-root environment_servers/harbor --tail 30 --json compare.json # Join on an explicit field when auto-detection picks the wrong one gym dev compare old.jsonl new.jsonl --key verifier_metadata.instance_id ``` **`json_output`** `Optional[str]` --- **`json_stdout`** `bool` --- **`key`** `Optional[str]` --- **`logs_root`** `Optional[str]` --- **`new_rollouts`** `str` --- **`old_rollouts`** `str` --- **`tail`** `int` --- ```python class nemo_gym.rollout_compare.TaskComparison( task: str, category: nemo_gym.rollout_compare.schema.Category, old: typing.Optional[nemo_gym.rollout_compare.schema.TaskSide] = None, new: typing.Optional[nemo_gym.rollout_compare.schema.TaskSide] = None, winner: typing.Optional[nemo_gym.rollout_compare.schema.Side] = None ) ``` Dataclass **`category`** `Category` --- **`loser`** `Optional[Side]` --- **`new`** `Optional[TaskSide] = None` --- **`old`** `Optional[TaskSide] = None` --- **`task`** `str` --- **`winner`** `Optional[Side] = None` --- ```python class nemo_gym.rollout_compare.TaskSide( rows: list[dict[str, typing.Any]], reward: typing.Optional[float], masked: typing.Optional[str], agent_error: typing.Optional[str], reward_rows: int = 0, failure_kind: typing.Optional[str] = None ) ``` Dataclass One side's rows for a task, reduced to what the comparison needs. **`agent_error`** `Optional[str]` --- **`failure_kind`** `Optional[str] = None` --- **`masked`** `Optional[str]` --- **`repeated`** `bool` --- **`reward`** `Optional[float]` --- **`reward_rows`** `int = 0` --- **`rows`** `list[dict[str, Any]]` --- ```python nemo_gym.rollout_compare.TaskSide.reward_text() -> str ``` ```python nemo_gym.rollout_compare.agent_error( row: dict[str, typing.Any] ) -> typing.Optional[str] ``` An agent-level error the row reports without being masked: the legacy agent's `error` field or an `*error_type` entry in the response metadata (e.g. Terminus's `terminus2_error_type`). ```python nemo_gym.rollout_compare.compare_rollouts( old_rows: list[dict[str, typing.Any]], new_rows: list[dict[str, typing.Any]], key: typing.Optional[str] = None ) -> nemo_gym.rollout_compare.schema.CompareResult ``` ```python nemo_gym.rollout_compare.current_failures( failures: list[dict[str, typing.Any]], scored: list[dict[str, typing.Any]] ) -> list[dict[str, typing.Any]] ``` The sidecar records that still stand: the collector writes one record per attempt and a retried rollout can end as a scored row in the main file, so only the last record per (task index, rollout index) counts and none whose rollout was scored after all. Records without the indices are kept as they are. ```python nemo_gym.rollout_compare.detect_task_key( rows: list[dict[str, typing.Any]] ) -> str ``` The first candidate field every row carries. Sidecar failure rows that name no task at all are left out of the vote; they cannot be joined and are listed as masked under an `unknown (<side>) #<n>` label. ```python nemo_gym.rollout_compare.group_by_task( rows: list[dict[str, typing.Any]], key: str, with_taskset: bool = False ) -> tuple[dict[str, list[dict[str, typing.Any]]], list[dict[str, typing.Any]]] ``` (rows grouped by task label, anonymous rows). An anonymous row is a sidecar failure that names no task: it is kept out of the join entirely and listed as masked on its own side. ```python nemo_gym.rollout_compare.load_rollouts( path: str | pathlib.Path ) -> list[dict[str, typing.Any]] ``` Read a rollouts JSONL plus, when present, its sibling `<name>_failures.jsonl` of masked failures. A failure superseded by a later attempt (another sidecar record or a scored row) is left out. ```python nemo_gym.rollout_compare.mask_reason( row: dict[str, typing.Any] ) -> typing.Optional[str] ``` Why a row does not count: `mask_sample`, no reward, or a NaN/infinite reward. A `failure_kind` alone does not mask: with `mask_sample` false it labels a measured reward (e.g. the Harbor server's scored zero for a verifier timeout), which counts like any other reward; see `measured_failure_kind`. ```python nemo_gym.rollout_compare.measured_failure_kind( row: dict[str, typing.Any] ) -> typing.Optional[str] ``` The `failure_kind` of a row that still counts (not masked): a scored failure the report labels but does not set aside, so a flip it causes is a real flip. ```python nemo_gym.rollout_compare.render_report( result: nemo_gym.rollout_compare.schema.CompareResult, roots: nemo_gym.rollout_compare.schema.Roots, tail: int = 15 ) -> str ``` ```python nemo_gym.rollout_compare.run_compare( config: nemo_gym.rollout_compare.command.RolloutCompareConfig, out = None ) -> int ``` Load both sides, print the report (or the JSON summary with `json_stdout`) and write the JSON file when asked. Exit code: 0 for a report, 1 for unreadable input or unwritable output. ```python nemo_gym.rollout_compare.summarize_side( rows: list[dict[str, typing.Any]] ) -> nemo_gym.rollout_compare.schema.TaskSide ``` ```python nemo_gym.rollout_compare.summary_dict( result: nemo_gym.rollout_compare.schema.CompareResult ) -> dict[str, typing.Any] ``` ```python nemo_gym.rollout_compare.tail_lines( text: str, count: int ) -> list[str] ``` ```python nemo_gym.rollout_compare.task_id_of( row: dict[str, typing.Any], key: str, with_taskset: bool = False ) -> str ``` The label a row is joined and listed under: the plain task id, or `taskset/task_id` with `with_taskset` (used when a side spans several tasksets, where the same task id may recur). ```python nemo_gym.rollout_compare.tasksets_of( rows: list[dict[str, typing.Any]], key: str ) -> set[str] ``` ```python nemo_gym.rollout_compare.verifier_output( row: dict[str, typing.Any], roots: list[pathlib.Path] ) -> tuple[typing.Optional[str], str] ``` (text, source) for a row's verifier stdout: the file under `verifier_logs_dir` when it exists, else the text embedded in the row; `(None, reason)` when neither is available.