nemo_gym.data.tasks.harbor.run_summary
nemo_gym.data.tasks.harbor.run_summary
The per-run summary of a Harbor run: one JSON file beside the rollouts that says where everything is.
A run of gym dataset validate or gym eval run <task target> leaves its evidence in several places:
the rollouts JSONL and its sidecars (failures, materialized inputs, aggregate metrics), the prepared rows with
the data processor’s _prepare.jsonl and _metrics.json beside them, the collated collector input under
<rollouts stem>/preprocessed_datasets/, the run config, the health check’s quality_summary.json and
rollout_verdicts.jsonl, and one verifier log folder per verification under the server’s artifacts_dir
(with test-stdout.txt, the reward file, the ATIF trajectory.json and, in separate mode, the collected
artifacts/). The rows name the log folders by absolute path, one row at a time.
write_run_summary joins them into <rollouts stem>_run_summary.json (run_summary_path_for)
next to the rollouts: the run’s identity (target, taskset, agent, dataset source and digests, the dataset label
the rows were materialized with as task_source and the one the result rows carry), every file of the run
and, per selected task and repeat, its outcome, the paths of its verifier logs, artifacts and trajectory, and
the input the row carried (its digests and the responses_create_params metadata, per turn for a multi-step
task). Every path is relative to the summary’s own folder, so a results folder moved as a whole still resolves,
and a path a row names that is not on disk is listed under missing_paths rather than silently dropped. The
files that follow a completed collection are listed only when they are this run’s: the aggregate metrics when
at least as new as the rollouts file (a run that stops early writes none, and the one left by an earlier run
under the same name is reported as null), and the health check’s two reports when the quality summary
records this rollouts file as the one it was computed from (_health_reports: the reports’ names are not
scoped to the --output stem, so a folder with several stems’ rollouts holds whichever run’s completed last,
newer than every one of them).
A fresh start (resume_from_cache off, or nothing to resume) clears the rollouts and failures files and
rewrites the materialized inputs only once its servers are up, so the CLI records what sits under those names
before it dispatches (output_signatures), and a file still carrying that signature afterwards was left
by an earlier run the collector never reached: it is null under files, its rows are not this run’s (the
selected tasks are missing), and it is named under stale_files so the folder still explains itself. A
resumed run’s cached rows are its own and are read as before. The two _metrics.json are never an earlier
run’s: prepare removes the one beside the rows when it rewrites them, and a fresh start removes the collated
<split>_metrics.json (with any _conflict.json) before it dispatches, so the data processor rewrites both
for this run’s selection instead of refusing it; a resumed run, or one that reuses the prepared input, keeps the
collated one. A run that reuses the prepared input has no rows sidecar of its own (files.rows_metrics is
null): prepare removed the earlier one and the processor that would rewrite it did not run. The file is
written whatever the run did: a run
that wrote no rollouts, or a validate that dispatched nothing because no selected task has a solution/,
gets a summary that says so (files.rollouts is null and every task is missing), beside the rollouts
path the command asked for. The tasks a validate left out for want of a solution/ are not scored and have
no task entry; they are named under unvalidated with the reason, as the tasks that did not load are under
skipped, so an all-unsolvable selection explains why nothing ran.
Nothing here depends on the run having succeeded, and the writer never raises into the run: the CLI logs a summary it could not write and keeps the run’s own outcome.
Module Contents
Functions
Data
API
The shards gym eval aggregate merged into rollouts, as its <stem>_sources.json lists them,
located; empty without that sidecar. The health check of a merged file reads the shards (with their failures
and inventories) and records them as its sources, so the merged file’s run owns such a report.
The relative path when it exists on disk, else None.
The health check’s two reports in the rollouts’ folder when they are this run’s, else None each.
The reports keep one name per folder whatever the --output stem, so a folder with several stems’ rollouts
holds the reports of whichever run completed last, newer than every rollouts file in it: timestamps alone do
not tell whose they are (a resumed run that failed before the collector’s cleanup would link another stem’s).
The quality summary records the rollouts it was computed from (_recorded_sources), so it is this
run’s when that record names this rollouts file (or the shards of a merged one, _aggregation_shards)
and it is at least as new as the rollouts; the verdicts are written with the summary and removed with it
(the provisional summary of a stopped run writes none), so they follow it. A report without the record is not
linked; rollouts is None when the run has no rollouts of its own.
What the materialized row carried for the task: its digests and the request metadata, per turn.
path with its directories resolved and its own name kept, as the health check locates the files it
records (a shard that is a link keeps its name).
The rollouts files the quality summary records as the ones it was computed from (run.rollouts, which
the health check writes relative to the report’s folder), located; None when it records none (a report an
older check wrote, or one that does not parse).
path relative to base (.. segments allowed), or absolute when no relative form exists.
The task_source the run’s result and failure rows carry, when they agree (None without rows).
Materialization stamps the dataset label on every row and collection preserves it, so this cross-checks
task_source (the label, known before dispatch): a results file another run wrote shows up as a different
value here.
The files collection writes under the run’s own name, by the key files lists them under.
The relative paths of one verification’s log folder and what it holds, plus the ones not on disk.
logs_dir is the row’s absolute verifier_logs_dir. A row that names none (test.sh never ran)
reports every path as None and nothing missing: there was nothing to collect. A row that names a
folder that is gone lists verifier_logs as missing; a folder without an optional file (no
artifacts/ in shared mode, no trajectory when the export was skipped) lists nothing, those are not
promised.
The relative path when path exists and is at least as new as rollouts, else None.
For the files written once the rollouts are complete: one older than the rollouts file, or present without
one (rollouts is None when the run has no rollouts of its own), was left by an earlier run (a run that
stopped early writes no new one).
The summary document for rollouts (see the module docstring); paths are relative to its folder.
expected is the collector’s selected, expanded input rows (after limit and repeats), which name
every task and repeat the run was to produce: for a resumed run, the saved materialized inputs it resumed
(its schedule is that run’s, whatever num_repeats the current config says). Without it (the run stopped
before they were known) the selected set is whatever the rollouts and failures files hold. split is the
collector’s split (validation on this path), which names the collated input under
preprocessed_datasets/.
before_dispatch is output_signatures taken before a fresh start dispatched (or before a validate
that dispatched nothing wrote its summary): an output file unchanged since was left by an earlier run and
is reported under stale_files, not files, and its rows are not read. None (a resumed run) reads
whatever is on disk. unvalidated names, with the reason, the selected tasks reference validation left out
without scoring them (a task without a solution/, NO_REFERENCE_SOLUTION); they have no row and
no task entry, so a validate that ran nothing explains itself here. A model run validates none.
What sits under the run’s output names now, taken by a fresh start before it dispatches.
A fresh start (resume_from_cache off, or nothing to resume) unlinks the rollouts and failures files and
rewrites the materialized inputs only once its servers are up, so a run that dies before that leaves the
files an earlier run put under the same names. Handed to build_run_summary as before_dispatch,
these signatures let the summary tell such a file (unchanged since) from one this run wrote. None marks
a name with no file; a resumed run passes no signatures, its cached rows are its own.
Write build_run_summary’s document to run_summary_path_for (rollouts) and return it.
The file is replaced atomically, so a reader never sees a half-written summary; the folder is created when the run wrote nothing there.