nemo_gym.data.tasks.harbor.materialize
nemo_gym.data.tasks.harbor.materialize
Turn loaded tasks into episode rows and the run config that serves them.
Module Contents
Classes
Functions
Data
API
Bases: BaseModel
A Harbor row’s input: one responses_create_params for a single-turn task, or one per turn.
task_data carries the folder digest and the task’s own fields; the server reads everything else
from the pinned folder.
Fold flat task fields into task_data, keeping both request-envelope fields where they are.
The flat single-turn input (the task fields beside responses_create_params, as
SingleAgentTurnTaskInput accepts them) is the canonical row once validated, so a client that sends
{"responses_create_params": ..., "_ng_digest": ...} runs as the row does. Only the keys the input set
are passed on, so a row that names no responses_create_params (a multi-step row) keeps the field unset
and its serialized form (exclude_unset) carries no null.
Bases: BaseModel
One turn of a multi-step task: the Harbor step’s name and the request the agent receives.
Return an override-compatible agent instance name for the taskset.
The datasets: entry that declares a taskset’s rows, from the dataset’s [gym] table.
type is the split the rows enter (GymDatasetConfig.effective_type: the declared one, else
validation with a license, else benchmark) and license the declared one, so a hub dataset
without a license is a benchmark that evaluates and never trains, and a train or validation
dataset carries the license Gym requires of one. verified is the dataset’s declaration (false
unless it says otherwise, as the pre-commit hook starts a new resources server), and domain and
allowed_agents are carried when declared. The entry sets no num_repeats: collate would duplicate
the rows by it and the collector’s run-level count would multiply on top, and the metrics sidecar beside
the rows records the entry, so a count that moved with the command line would conflict run to run.
[gym.defaults].num_repeats is the run config’s top-level num_repeats instead (see
nemo_gym.data.tasks.harbor.prepare.build_run). A benchmark entry names no prepare_script: the rows are
written by the preparation that generates this entry.
Return an override-compatible environment instance name for the taskset.
One row: the instruction as a user message, the folder digest and the task’s fields in task_data.
A multi-step task gets one turn per step, each with that step’s instruction and agent budget. The
top-level instruction.md is never sent for such a task.
task_data holds the task’s own field values (task_data.json in its folder, validated against
the dataset’s task_data.py before any row is written) under the reserved _ng_* keys. The values
are the file’s as written, not the validated model’s dump: the model validates strictly, as JSON, so an
accepted value already has its field’s type (validate),
and the same fields are what the seed compares with the folder’s file and what the in-sandbox verifier
reads as task_data.json. A dump would drift from both (defaults filled in, aliases and AliasPath
shapes rewritten) without changing what any verifier sees.
With dataset_digest, the row also carries the dataset’s digest (manifest.toml’s
[dataset].digest). Rollout collection copies the _ng_* keys of task_data onto every
results row it writes, so each row then says which task folder and which dataset it ran.
The data processor’s per-dataset metrics beside the rows and the conflict file it writes on a mismatch:
<rows stem>_metrics.json and <rows stem>_metrics_conflict.json (nemo_gym.data.dataset_preparation).
The run-config blocks that bind one taskset to the harbor environment server.
The environment server block inherits the shipped harbor_environment_server config and
names the taskset’s agent. The agent block inherits the selected harness config and declares
the taskset’s rows as its dataset (dataset_entry, from the [gym] table of
dataset.toml: split, license, repeats and catalog metadata), which is where rollout collection
looks for datasets. Rows route to the environment server by taskset. The taskset mapping records the
dataset digest and the dataset.toml digest, so the run config says which dataset ran and
whether the Gym layer or upstream moved between two runs.
Where a taskset’s folders live and which digest each task had at materialization.
Servers resolve a task as <folder>/<task_id>, or <folder>/<paths[task_id]> for a task
found below the first level (see nemo_gym.data.tasks.harbor.task.discover_tasks). For a
folder of task folders folder is the folder itself; for a single task folder it is the
task’s parent, so folder is always the root the tasks were discovered under.
Write one materialized row per task as JSONL and return the rows.
Beside the task’s identity and input, every row carries task_source: the dataset folder’s name,
task_source when given (a single task folder target inside a dataset passes its parent’s name; a
standalone task folder, whose parent holds no dataset.toml or manifest.toml, passes its own) and
the taskset otherwise (a dataset folder target’s taskset is that name). It is the dataset label a results row and
a trainer read, and nothing the collector routes by: a materialized row reaches its environment server
by its taskset, agent_map and fan_out do not apply to it, and a per-dataset num_repeats dict
keys on the taskset.
The metrics sidecars the data processor leaves beside the rows (rows_metrics_sidecars) go with the
rows they described: they are derived from one selection of the task folder, and the processor would refuse
a changed selection against them (“Found conflicting aggregate metrics”), a check meant for a committed
source dataset, not for rows regenerated on every command. Removed here, they are rewritten by the run.