> This page is for version 0.5.1. > For other versions, use one of these documentation indexes: > - Main (default): https://docs.nvidia.com/nemo/gym/main/llms.txt > - 0.7.0: https://docs.nvidia.com/nemo/gym/v0.7.0/llms.txt > - 0.6.0: https://docs.nvidia.com/nemo/gym/v0.6.0/llms.txt > - 0.5.1: https://docs.nvidia.com/nemo/gym/v0.5.1/llms.txt > - 0.5.0: https://docs.nvidia.com/nemo/gym/v0.5.0/llms.txt > - 0.4.0: https://docs.nvidia.com/nemo/gym/v0.4.0/llms.txt > - 0.3.0: https://docs.nvidia.com/nemo/gym/v0.3.0/llms.txt > - 0.2.1: https://docs.nvidia.com/nemo/gym/v0.2.1/llms.txt > For clean Markdown of any page, append .md to the page URL. > For a complete documentation index, see https://docs.nvidia.com/nemo/gym/llms.txt. > For AI client integration (Claude Code, Cursor, etc.), connect to the MCP server at https://docs.nvidia.com/nemo/gym/_mcp/server. # nemo_gym.data.tasks.harbor.run_summary The per-run summary of a Harbor run: one JSON file beside the rollouts that says where everything is. A run of `gym dataset validate` or `gym eval run ` leaves its evidence in several places: the rollouts JSONL and its sidecars (failures, materialized inputs, aggregate metrics), the prepared rows with the data processor's `_prepare.jsonl` and `_metrics.json` beside them, the collated collector input under `/preprocessed_datasets/`, the run config, the health check's `quality_summary.json` and `rollout_verdicts.jsonl`, and one verifier log folder per verification under the server's `artifacts_dir` (with `test-stdout.txt`, the reward file, the ATIF `trajectory.json` and, in separate mode, the collected `artifacts/`). The rows name the log folders by absolute path, one row at a time. `write_run_summary` joins them into `_run_summary.json` (`run_summary_path_for`) next to the rollouts: the run's identity (target, taskset, agent, dataset source and digests, the dataset label the rows were materialized with as `task_source` and the one the result rows carry), every file of the run and, per selected task and repeat, its outcome, the paths of its verifier logs, artifacts and trajectory, and the input the row carried (its digests and the `responses_create_params` metadata, per turn for a multi-step task). Every path is relative to the summary's own folder, so a results folder moved as a whole still resolves, and a path a row names that is not on disk is listed under `missing_paths` rather than silently dropped. The files that follow a completed collection are listed only when they are this run's: the aggregate metrics when at least as new as the rollouts file (a run that stops early writes none, and the one left by an earlier run under the same name is reported as `null`), and the health check's two reports when the quality summary records this rollouts file as the one it was computed from (`_health_reports`: the reports' names are not scoped to the `--output` stem, so a folder with several stems' rollouts holds whichever run's completed last, newer than every one of them). A fresh start (`resume_from_cache` off, or nothing to resume) clears the rollouts and failures files and rewrites the materialized inputs only once its servers are up, so the CLI records what sits under those names before it dispatches (`output_signatures`), and a file still carrying that signature afterwards was left by an earlier run the collector never reached: it is `null` under `files`, its rows are not this run's (the selected tasks are `missing`), and it is named under `stale_files` so the folder still explains itself. A resumed run's cached rows are its own and are read as before. The two `_metrics.json` are never an earlier run's: prepare removes the one beside the rows when it rewrites them, and a fresh start removes the collated `_metrics.json` (with any `_conflict.json`) before it dispatches, so the data processor rewrites both for this run's selection instead of refusing it; a resumed run, or one that reuses the prepared input, keeps the collated one. A run that reuses the prepared input has no rows sidecar of its own (`files.rows_metrics` is `null`): prepare removed the earlier one and the processor that would rewrite it did not run. The file is written whatever the run did: a run that wrote no rollouts, or a validate that dispatched nothing because no selected task has a `solution/`, gets a summary that says so (`files.rollouts` is `null` and every task is `missing`), beside the rollouts path the command asked for. The tasks a validate left out for want of a `solution/` are not scored and have no task entry; they are named under `unvalidated` with the reason, as the tasks that did not load are under `skipped`, so an all-unsolvable selection explains why nothing ran. Nothing here depends on the run having succeeded, and the writer never raises into the run: the CLI logs a summary it could not write and keeps the run's own outcome. ## Module Contents ### Functions | Name | Description | | ------------------------------------------------------------------------------------ | ------------------------------------------------------------------------------------------------------ | | [`_aggregation_shards`](#nemo_gym-data-tasks-harbor-run_summary-_aggregation_shards) | The shards `gym eval aggregate` merged into `rollouts`, as its `<stem>_sources.json` lists them, | | [`_existing`](#nemo_gym-data-tasks-harbor-run_summary-_existing) | The relative path when it exists on disk, else `None`. | | [`_failure_entry`](#nemo_gym-data-tasks-harbor-run_summary-_failure_entry) | - | | [`_health_reports`](#nemo_gym-data-tasks-harbor-run_summary-_health_reports) | The health check's two reports in the rollouts' folder when they are this run's, else `None` each. | | [`_identity`](#nemo_gym-data-tasks-harbor-run_summary-_identity) | - | | [`_input_echo`](#nemo_gym-data-tasks-harbor-run_summary-_input_echo) | What the materialized row carried for the task: its digests and the request metadata, per turn. | | [`_located`](#nemo_gym-data-tasks-harbor-run_summary-_located) | `path` with its directories resolved and its own name kept, as the health check locates the files it | | [`_mapping`](#nemo_gym-data-tasks-harbor-run_summary-_mapping) | - | | [`_missing_entry`](#nemo_gym-data-tasks-harbor-run_summary-_missing_entry) | - | | [`_number`](#nemo_gym-data-tasks-harbor-run_summary-_number) | - | | [`_recorded_sources`](#nemo_gym-data-tasks-harbor-run_summary-_recorded_sources) | The rollouts files the quality summary records as the ones it was computed from (`run.rollouts`, which | | [`_relative`](#nemo_gym-data-tasks-harbor-run_summary-_relative) | `path` relative to `base` (`..` segments allowed), or absolute when no relative form exists. | | [`_result_entry`](#nemo_gym-data-tasks-harbor-run_summary-_result_entry) | - | | [`_rows_task_source`](#nemo_gym-data-tasks-harbor-run_summary-_rows_task_source) | The `task_source` the run's result and failure rows carry, when they agree (`None` without rows). | | [`_run_outputs`](#nemo_gym-data-tasks-harbor-run_summary-_run_outputs) | The files collection writes under the run's own name, by the key `files` lists them under. | | [`_signature`](#nemo_gym-data-tasks-harbor-run_summary-_signature) | - | | [`_sort_key`](#nemo_gym-data-tasks-harbor-run_summary-_sort_key) | - | | [`_turn_entries`](#nemo_gym-data-tasks-harbor-run_summary-_turn_entries) | - | | [`_verifier_paths`](#nemo_gym-data-tasks-harbor-run_summary-_verifier_paths) | The relative paths of one verification's log folder and what it holds, plus the ones not on disk. | | [`_written_after`](#nemo_gym-data-tasks-harbor-run_summary-_written_after) | The relative path when `path` exists and is at least as new as `rollouts`, else `None`. | | [`build_run_summary`](#nemo_gym-data-tasks-harbor-run_summary-build_run_summary) | The summary document for `rollouts` (see the module docstring); paths are relative to its folder. | | [`output_signatures`](#nemo_gym-data-tasks-harbor-run_summary-output_signatures) | What sits under the run's output names now, taken by a fresh start before it dispatches. | | [`write_run_summary`](#nemo_gym-data-tasks-harbor-run_summary-write_run_summary) | Write `build_run_summary`'s document to `run_summary_path_for` `(rollouts)` and return it. | ### Data [`ARTIFACTS_SUBDIR`](#nemo_gym-data-tasks-harbor-run_summary-ARTIFACTS_SUBDIR) [`FileSignature`](#nemo_gym-data-tasks-harbor-run_summary-FileSignature) [`NO_REFERENCE_SOLUTION`](#nemo_gym-data-tasks-harbor-run_summary-NO_REFERENCE_SOLUTION) [`PREPROCESSED_SUBDIR`](#nemo_gym-data-tasks-harbor-run_summary-PREPROCESSED_SUBDIR) [`RUN_SUMMARY_SCHEMA_VERSION`](#nemo_gym-data-tasks-harbor-run_summary-RUN_SUMMARY_SCHEMA_VERSION) [`STATUS_FAILED`](#nemo_gym-data-tasks-harbor-run_summary-STATUS_FAILED) [`STATUS_MISSING`](#nemo_gym-data-tasks-harbor-run_summary-STATUS_MISSING) [`STATUS_RESULT`](#nemo_gym-data-tasks-harbor-run_summary-STATUS_RESULT) [`VERIFIER_LOG_FILES`](#nemo_gym-data-tasks-harbor-run_summary-VERIFIER_LOG_FILES) ### API ```python nemo_gym.data.tasks.harbor.run_summary._aggregation_shards( rollouts: pathlib.Path ) -> set[pathlib.Path] ``` The shards `gym eval aggregate` merged into `rollouts`, as its `_sources.json` lists them, located; empty without that sidecar. The health check of a merged file reads the shards (with their failures and inventories) and records them as its sources, so the merged file's run owns such a report. ```python nemo_gym.data.tasks.harbor.run_summary._existing( path: pathlib.Path | None, base: pathlib.Path ) -> str | None ``` The relative path when it exists on disk, else `None`. ```python nemo_gym.data.tasks.harbor.run_summary._failure_entry( key: tuple[str, str, typing.Any, typing.Any], row: collections.abc.Mapping[str, typing.Any], base: pathlib.Path ) -> dict[str, typing.Any] ``` ```python nemo_gym.data.tasks.harbor.run_summary._health_reports( rollouts: pathlib.Path | None, base: pathlib.Path ) -> dict[str, str | None] ``` The health check's two reports in the rollouts' folder when they are this run's, else `None` each. The reports keep one name per folder whatever the `--output` stem, so a folder with several stems' rollouts holds the reports of whichever run completed last, newer than every rollouts file in it: timestamps alone do not tell whose they are (a resumed run that failed before the collector's cleanup would link another stem's). The quality summary records the rollouts it was computed from (`_recorded_sources`), so it is this run's when that record names this rollouts file (or the shards of a merged one, `_aggregation_shards`) and it is at least as new as the rollouts; the verdicts are written with the summary and removed with it (the provisional summary of a stopped run writes none), so they follow it. A report without the record is not linked; `rollouts` is `None` when the run has no rollouts of its own. ```python nemo_gym.data.tasks.harbor.run_summary._identity( key: tuple[str, str, typing.Any, typing.Any] ) -> dict[str, typing.Any] ``` ```python nemo_gym.data.tasks.harbor.run_summary._input_echo( task_input: typing.Any ) -> dict[str, typing.Any] ``` What the materialized row carried for the task: its digests and the request metadata, per turn. ```python nemo_gym.data.tasks.harbor.run_summary._located( path: pathlib.Path ) -> pathlib.Path ``` `path` with its directories resolved and its own name kept, as the health check locates the files it records (a shard that is a link keeps its name). ```python nemo_gym.data.tasks.harbor.run_summary._mapping( value: typing.Any ) -> dict[str, typing.Any] ``` ```python nemo_gym.data.tasks.harbor.run_summary._missing_entry( key: tuple[str, str, typing.Any, typing.Any] ) -> dict[str, typing.Any] ``` ```python nemo_gym.data.tasks.harbor.run_summary._number( value: typing.Any ) -> float | int | None ``` ```python nemo_gym.data.tasks.harbor.run_summary._recorded_sources( quality_summary: pathlib.Path ) -> set[pathlib.Path] | None ``` The rollouts files the quality summary records as the ones it was computed from (`run.rollouts`, which the health check writes relative to the report's folder), located; `None` when it records none (a report an older check wrote, or one that does not parse). ```python nemo_gym.data.tasks.harbor.run_summary._relative( path: pathlib.Path, base: pathlib.Path ) -> str ``` `path` relative to `base` (`..` segments allowed), or absolute when no relative form exists. ```python nemo_gym.data.tasks.harbor.run_summary._result_entry( key: tuple[str, str, typing.Any, typing.Any], row: collections.abc.Mapping[str, typing.Any], base: pathlib.Path ) -> dict[str, typing.Any] ``` ```python nemo_gym.data.tasks.harbor.run_summary._rows_task_source( sources: collections.abc.Iterable[str] ) -> str | None ``` The `task_source` the run's result and failure rows carry, when they agree (`None` without rows). Materialization stamps the dataset label on every row and collection preserves it, so this cross-checks `task_source` (the label, known before dispatch): a results file another run wrote shows up as a different value here. ```python nemo_gym.data.tasks.harbor.run_summary._run_outputs( rollouts: pathlib.Path ) -> dict[str, pathlib.Path] ``` The files collection writes under the run's own name, by the key `files` lists them under. ```python nemo_gym.data.tasks.harbor.run_summary._signature( path: pathlib.Path ) -> nemo_gym.data.tasks.harbor.run_summary.FileSignature | None ``` ```python nemo_gym.data.tasks.harbor.run_summary._sort_key( key: tuple[str, str, typing.Any, typing.Any] ) -> tuple[typing.Any, ...] ``` ```python nemo_gym.data.tasks.harbor.run_summary._turn_entries( turns: typing.Any, base: pathlib.Path ) -> list[dict[str, typing.Any]] ``` ```python nemo_gym.data.tasks.harbor.run_summary._verifier_paths( logs_dir: str | None, base: pathlib.Path ) -> tuple[dict[str, str | None], list[str]] ``` The relative paths of one verification's log folder and what it holds, plus the ones not on disk. `logs_dir` is the row's absolute `verifier_logs_dir`. A row that names none (`test.sh` never ran) reports every path as `None` and nothing missing: there was nothing to collect. A row that names a folder that is gone lists `verifier_logs` as missing; a folder without an optional file (no `artifacts/` in shared mode, no trajectory when the export was skipped) lists nothing, those are not promised. ```python nemo_gym.data.tasks.harbor.run_summary._written_after( path: pathlib.Path, rollouts: pathlib.Path | None, base: pathlib.Path ) -> str | None ``` The relative path when `path` exists and is at least as new as `rollouts`, else `None`. For the files written once the rollouts are complete: one older than the rollouts file, or present without one (`rollouts` is `None` when the run has no rollouts of its own), was left by an earlier run (a run that stopped early writes no new one). ```python nemo_gym.data.tasks.harbor.run_summary.build_run_summary( rollouts: pathlib.Path, command: str, target: str, prepared: nemo_gym.data.tasks.harbor.prepare.PreparedTaskset, agent: nemo_gym.data.tasks.harbor.prepare.AgentSelection, run_config: pathlib.Path | None, expected: list[dict[str, typing.Any]] | None, split: str | None = None, before_dispatch: collections.abc.Mapping[str, nemo_gym.data.tasks.harbor.run_summary.FileSignature | None] | None = None, unvalidated: collections.abc.Mapping[str, str] | None = None ) -> dict[str, typing.Any] ``` The summary document for `rollouts` (see the module docstring); paths are relative to its folder. `expected` is the collector's selected, expanded input rows (after `limit` and repeats), which name every task and repeat the run was to produce: for a resumed run, the saved materialized inputs it resumed (its schedule is that run's, whatever `num_repeats` the current config says). Without it (the run stopped before they were known) the selected set is whatever the rollouts and failures files hold. `split` is the collector's split (`validation` on this path), which names the collated input under `preprocessed_datasets/`. `before_dispatch` is `output_signatures` taken before a fresh start dispatched (or before a validate that dispatched nothing wrote its summary): an output file unchanged since was left by an earlier run and is reported under `stale_files`, not `files`, and its rows are not read. `None` (a resumed run) reads whatever is on disk. `unvalidated` names, with the reason, the selected tasks reference validation left out without scoring them (a task without a `solution/`, `NO_REFERENCE_SOLUTION`); they have no row and no task entry, so a validate that ran nothing explains itself here. A model run validates none. ```python nemo_gym.data.tasks.harbor.run_summary.output_signatures( rollouts: pathlib.Path ) -> dict[str, nemo_gym.data.tasks.harbor.run_summary.FileSignature | None] ``` What sits under the run's output names now, taken by a fresh start before it dispatches. A fresh start (`resume_from_cache` off, or nothing to resume) unlinks the rollouts and failures files and rewrites the materialized inputs only once its servers are up, so a run that dies before that leaves the files an earlier run put under the same names. Handed to `build_run_summary` as `before_dispatch`, these signatures let the summary tell such a file (unchanged since) from one this run wrote. `None` marks a name with no file; a resumed run passes no signatures, its cached rows are its own. ```python nemo_gym.data.tasks.harbor.run_summary.write_run_summary( rollouts: pathlib.Path, kwargs: typing.Any = {} ) -> pathlib.Path ``` Write `build_run_summary`'s document to `run_summary_path_for` `(rollouts)` and return it. The file is replaced atomically, so a reader never sees a half-written summary; the folder is created when the run wrote nothing there. ```python nemo_gym.data.tasks.harbor.run_summary.ARTIFACTS_SUBDIR = 'artifacts' ``` ```python nemo_gym.data.tasks.harbor.run_summary.FileSignature = tuple[int, int, int] ``` ```python nemo_gym.data.tasks.harbor.run_summary.NO_REFERENCE_SOLUTION = 'no reference solution' ``` ```python nemo_gym.data.tasks.harbor.run_summary.PREPROCESSED_SUBDIR = 'preprocessed_datasets' ``` ```python nemo_gym.data.tasks.harbor.run_summary.RUN_SUMMARY_SCHEMA_VERSION = 1 ``` ```python nemo_gym.data.tasks.harbor.run_summary.STATUS_FAILED = 'failed' ``` ```python nemo_gym.data.tasks.harbor.run_summary.STATUS_MISSING = 'missing' ``` ```python nemo_gym.data.tasks.harbor.run_summary.STATUS_RESULT = 'result' ``` ```python nemo_gym.data.tasks.harbor.run_summary.VERIFIER_LOG_FILES = {'test_stdout': 'test-stdout.txt', 'trajectory': 'trajectory.json', 'entrypoint_... ```