nemo_gym.data.tasks.harbor.validation_report

Summarise a gym dataset validate run from its rollout and failure files.

Reads the rows a validation run wrote, keys them by task, reports which tasks passed, which scored below one and which were never run, and returns the text and the overall verdict as a ValidationReport. The task digest lookup the resume checks share lives here as well. nemo_gym.cli.eval imports it for validate_target and for the resume conflict check of gym eval run.

Module Contents

Classes

Name	Description
`ValidationReport`	-

Functions

Name	Description
`_dig`	-
`_iter_validation_rows`	The records of a rollouts, failures or materialized inputs file, in order (none when there is no file).
`_task_digests`	`key(row) -> task digest` for materialized rows; shared by the eval and validation resume checks.
`_task_name`	-
`_validation_key`	-
`_validation_rows`	`_iter_validation_rows` as a list, for the readers that need every record at once.
`summarize_validation`	Report every selected task/repeat, reconciling results with failed attempts.

API

class nemo_gym.data.tasks.harbor.validation_report.ValidationReport(
    text: str,
    ok: bool
)

Dataclass

bool

text

str

nemo_gym.data.tasks.harbor.validation_report._dig(
    mapping: typing.Any,
    keys: str = ()
) -> typing.Any

nemo_gym.data.tasks.harbor.validation_report._iter_validation_rows(
    path: pathlib.Path
) -> collections.abc.Iterator[dict[str, object]]

The records of a rollouts, failures or materialized inputs file, in order (none when there is no file).

One record is read and parsed at a time, so a reader that keeps a projection of each row (the run summary) never holds a long run’s responses together. A run killed while appending leaves a final line without a newline that does not parse (the cut may land inside a multibyte character, so it may not decode either). The collector drops that line from the rollouts and failures files when it resumes (_drop_truncated_tail), so it is skipped here with the same tolerance, and a resume that reads the cache first still reaches that repair. A complete final record without a newline is read like any other. A line that does not parse anywhere else is damage the collector does not repair, and is reported by file and line rather than resumed over.

nemo_gym.data.tasks.harbor.validation_report._task_digests(
    rows: list[dict[str, object]],
    key: typing.Any
) -> dict[typing.Any, object]

key(row) -> task digest for materialized rows; shared by the eval and validation resume checks.

nemo_gym.data.tasks.harbor.validation_report._task_name(
    row: dict[str, typing.Any]
) -> str

nemo_gym.data.tasks.harbor.validation_report._validation_key(
    row: dict[str, object]
) -> tuple[str, str, object, object]

nemo_gym.data.tasks.harbor.validation_report._validation_rows(
    path: pathlib.Path
) -> list[dict[str, object]]

_iter_validation_rows as a list, for the readers that need every record at once.

nemo_gym.data.tasks.harbor.validation_report.summarize_validation(
    rollouts: pathlib.Path,
    unvalidated: list[str],
    expected: list[dict[str, object]] | None = None,
    skipped: dict[str, str] | None = None
) -> nemo_gym.data.tasks.harbor.validation_report.ValidationReport

Report every selected task/repeat, reconciling results with failed attempts.

The collector stores verify responses directly and keeps historical failed attempts in a sidecar. A result supersedes failures for the same task/repeat, but never another repeat. expected contains the collector’s selected, expanded input rows (after limit/repeats).