nemo_gym.rollout_compare

Per-task parity report between two rollout JSONL files, the gym dev compare command.

A migration aid: before an old resources server is deleted in favour of its Harbor-path replacement, both are run on the same tasks and this report is the gate. It joins the rows by task, classifies every task as identical, flipped, masked or missing, and prints the losing side’s verifier output for every flip. Rows from either path are accepted, and masked failures from the failures sidecar are folded back in. The package re-exports the public names so that nemo_gym.cli.dev and the tests import them from nemo_gym.rollout_compare unchanged.

Submodules

Package Contents

Classes

Name	Description
`CompareInputError`	An input file could not be read or does not look like rollouts.
`CompareResult`	-
`RolloutCompareConfig`	Compare two rollout JSONL files task by task: count identical, flipped, masked and missing tasks and print the
`TaskComparison`	-
`TaskSide`	One side’s rows for a task, reduced to what the comparison needs.

Functions

Name	Description
`agent_error`	An agent-level error the row reports without being masked: the legacy agent’s `error` field or an
`compare_rollouts`	-
`current_failures`	The sidecar records that still stand: the collector writes one record per attempt and a retried rollout
`detect_task_key`	The first candidate field every row carries. Sidecar failure rows that name no task at all are left out
`group_by_task`	(rows grouped by task label, anonymous rows). An anonymous row is a sidecar failure that names no task:
`load_rollouts`	Read a rollouts JSONL plus, when present, its sibling `<name>_failures.jsonl` of masked failures. A failure
`mask_reason`	Why a row does not count: `mask_sample`, no reward, or a NaN/infinite reward.
`measured_failure_kind`	The `failure_kind` of a row that still counts (not masked): a scored failure the report labels but does
`render_report`	-
`run_compare`	Load both sides, print the report (or the JSON summary with `json_stdout`) and write the JSON file when
`summarize_side`	-
`summary_dict`	-
`tail_lines`	-
`task_id_of`	The label a row is joined and listed under: the plain task id, or `taskset/task_id` with `with_taskset`
`tasksets_of`	-
`verifier_output`	(text, source) for a row’s verifier stdout: the file under `verifier_logs_dir` when it exists, else the

API

class nemo_gym.rollout_compare.CompareInputError()

Exception

Bases: Exception

An input file could not be read or does not look like rollouts.

class nemo_gym.rollout_compare.CompareResult(
    tasks: list[nemo_gym.rollout_compare.schema.TaskComparison],
    old_key: str,
    new_key: str,
    old_rows: int,
    new_rows: int,
    old_masked_rows: int = 0,
    new_masked_rows: int = 0,
    old_prefix_stripped: typing.Optional[str] = None,
    new_prefix_stripped: typing.Optional[str] = None,
    keyed_by_taskset: bool = False,
    notes: list[str] = list()
)

Dataclass

keyed_by_taskset

bool = False

new_key

str

new_masked_rows

int = 0

new_prefix_stripped

Optional[str] = None

new_rows

int

notes

list[str] = field(default_factory=list)

old_key

str

old_masked_rows

int = 0

old_prefix_stripped

Optional[str] = None

old_rows

int

tasks

list[TaskComparison]

nemo_gym.rollout_compare.CompareResult.by_category(
    category: nemo_gym.rollout_compare.schema.Category
) -> list[nemo_gym.rollout_compare.schema.TaskComparison]

nemo_gym.rollout_compare.CompareResult.counts() -> dict[str, int]

class nemo_gym.rollout_compare.RolloutCompareConfig()

Bases: BaseNeMoGymCLIConfig

Compare two rollout JSONL files task by task: count identical, flipped, masked and missing tasks and print the losing side’s verifier output for every flip. A report, not a gate: the exit code is 0 for a report and 1 when an input cannot be read or the --json summary cannot be written.

Examples:

# Old server vs Harbor path on the same tasks
gym dev compare old/rollouts.jsonl new/rollouts.jsonl
# Resolve relative `verifier_logs_dir` paths against the Harbor server directory, keep 30 log lines per flip,
# and also write a machine-readable summary
gym dev compare old.jsonl new.jsonl --logs-root environment_servers/harbor --tail 30 --json compare.json
# Join on an explicit field when auto-detection picks the wrong one
gym dev compare old.jsonl new.jsonl --key verifier_metadata.instance_id

json_output

Optional[str]

json_stdout

bool

key

Optional[str]

logs_root

Optional[str]

new_rollouts

str

old_rollouts

str

tail

int

class nemo_gym.rollout_compare.TaskComparison(
    task: str,
    category: nemo_gym.rollout_compare.schema.Category,
    old: typing.Optional[nemo_gym.rollout_compare.schema.TaskSide] = None,
    new: typing.Optional[nemo_gym.rollout_compare.schema.TaskSide] = None,
    winner: typing.Optional[nemo_gym.rollout_compare.schema.Side] = None
)

Dataclass

Category

loser

Optional[Side]

new

Optional[TaskSide] = None

old

Optional[TaskSide] = None

task

str

winner

Optional[Side] = None

class nemo_gym.rollout_compare.TaskSide(
    rows: list[dict[str, typing.Any]],
    reward: typing.Optional[float],
    masked: typing.Optional[str],
    agent_error: typing.Optional[str],
    reward_rows: int = 0,
    failure_kind: typing.Optional[str] = None
)

Dataclass

One side’s rows for a task, reduced to what the comparison needs.

agent_error

Optional[str]

failure_kind

Optional[str] = None

masked

Optional[str]

repeated

bool

reward

Optional[float]

reward_rows

int = 0

rows

list[dict[str, Any]]

nemo_gym.rollout_compare.TaskSide.reward_text() -> str

nemo_gym.rollout_compare.agent_error(
    row: dict[str, typing.Any]
) -> typing.Optional[str]

An agent-level error the row reports without being masked: the legacy agent’s error field or an *error_type entry in the response metadata (e.g. Terminus’s terminus2_error_type).

nemo_gym.rollout_compare.compare_rollouts(
    old_rows: list[dict[str, typing.Any]],
    new_rows: list[dict[str, typing.Any]],
    key: typing.Optional[str] = None
) -> nemo_gym.rollout_compare.schema.CompareResult

nemo_gym.rollout_compare.current_failures(
    failures: list[dict[str, typing.Any]],
    scored: list[dict[str, typing.Any]]
) -> list[dict[str, typing.Any]]

The sidecar records that still stand: the collector writes one record per attempt and a retried rollout can end as a scored row in the main file, so only the last record per (task index, rollout index) counts and none whose rollout was scored after all. Records without the indices are kept as they are.

nemo_gym.rollout_compare.detect_task_key(
    rows: list[dict[str, typing.Any]]
) -> str

The first candidate field every row carries. Sidecar failure rows that name no task at all are left out of the vote; they cannot be joined and are listed as masked under an unknown (<side>) #<n> label.

nemo_gym.rollout_compare.group_by_task(
    rows: list[dict[str, typing.Any]],
    key: str,
    with_taskset: bool = False
) -> tuple[dict[str, list[dict[str, typing.Any]]], list[dict[str, typing.Any]]]

(rows grouped by task label, anonymous rows). An anonymous row is a sidecar failure that names no task: it is kept out of the join entirely and listed as masked on its own side.

nemo_gym.rollout_compare.load_rollouts(
    path: str | pathlib.Path
) -> list[dict[str, typing.Any]]

Read a rollouts JSONL plus, when present, its sibling <name>_failures.jsonl of masked failures. A failure superseded by a later attempt (another sidecar record or a scored row) is left out.

nemo_gym.rollout_compare.mask_reason(
    row: dict[str, typing.Any]
) -> typing.Optional[str]

Why a row does not count: mask_sample, no reward, or a NaN/infinite reward.

A failure_kind alone does not mask: with mask_sample false it labels a measured reward (e.g. the Harbor server’s scored zero for a verifier timeout), which counts like any other reward; see measured_failure_kind.

nemo_gym.rollout_compare.measured_failure_kind(
    row: dict[str, typing.Any]
) -> typing.Optional[str]

The failure_kind of a row that still counts (not masked): a scored failure the report labels but does not set aside, so a flip it causes is a real flip.

nemo_gym.rollout_compare.render_report(
    result: nemo_gym.rollout_compare.schema.CompareResult,
    roots: nemo_gym.rollout_compare.schema.Roots,
    tail: int = 15
) -> str

nemo_gym.rollout_compare.run_compare(
    config: nemo_gym.rollout_compare.command.RolloutCompareConfig,
    out = None
) -> int

Load both sides, print the report (or the JSON summary with json_stdout) and write the JSON file when asked. Exit code: 0 for a report, 1 for unreadable input or unwritable output.

nemo_gym.rollout_compare.summarize_side(
    rows: list[dict[str, typing.Any]]
) -> nemo_gym.rollout_compare.schema.TaskSide

nemo_gym.rollout_compare.summary_dict(
    result: nemo_gym.rollout_compare.schema.CompareResult
) -> dict[str, typing.Any]

nemo_gym.rollout_compare.tail_lines(
    text: str,
    count: int
) -> list[str]

nemo_gym.rollout_compare.task_id_of(
    row: dict[str, typing.Any],
    key: str,
    with_taskset: bool = False
) -> str

The label a row is joined and listed under: the plain task id, or taskset/task_id with with_taskset (used when a side spans several tasksets, where the same task id may recur).

nemo_gym.rollout_compare.tasksets_of(
    rows: list[dict[str, typing.Any]],
    key: str
) -> set[str]

nemo_gym.rollout_compare.verifier_output(
    row: dict[str, typing.Any],
    roots: list[pathlib.Path]
) -> tuple[typing.Optional[str], str]

(text, source) for a row’s verifier stdout: the file under verifier_logs_dir when it exists, else the text embedded in the row; (None, reason) when neither is available.