nemo_gym.data.tasks.harbor.publish

gym dataset publish --harbor: a Gym-authored dataset as plain Harbor task folders.

Expands a parameterized template into its task folders, writes the tests/test.sh shim for every task graded by a Python verifier (nemo_gym.data.tasks.harbor.python_verifier) and a solution/solve.sh for every solution/answer.md, writes dataset.toml without its [gym] table (Harbor’s own tables byte for byte, as the RFC has it: the published file is Harbor’s artifact, Gym’s license, defaults and overrides stay with the source), and runs Harbor’s own format check on each staged folder when the harbor package is importable (it is not a Gym dependency; the report says how to install it otherwise). Gym-only files (tasks.jsonl, task_data.py, a dataset-level verifier.py) stay behind: what is published is what Harbor reads.

Before anything is written, every task’s fields pass the dataset’s TaskData (task_data.py), the check prepare runs before a row is written, so Gym and the published dataset agree on which rows exist and no shim carries a value the model refuses. The folders are published into a staging folder beside output, Harbor’s check runs over them there, and they are moved into output only once every task published and passed the check: a publish refused by Gym or by Harbor leaves nothing behind (output stays absent) and a retry into the same folder needs no cleanup.

A published task holds one grader, and the same one Gym runs. A multi-step task ([[steps]]) is never graded by a Python verifier in Gym (the loader grades each step by a test.sh, its own or the task’s), so publish copies such a task as it is and writes no shim for it, even under a dataset-level verifier.py: a shim tests/test.sh would become Harbor’s fallback grader for every step without its own script, and the published copy would grade differently from the source. Two published folders that share a [task].name are refused: Harbor keys a dataset on that name (harbor add keeps one folder per name), so the published dataset would silently lose a task. A task’s own tests/verifier.py is copied where it is, with the fixtures and helper modules beside it, and the shim’s test.sh runs it there (/tests/verifier.py, as Gym’s server does), so a module that reads a sibling fixture by Path(__file__) grades the published task as it graded the source. The published tests/ is then the hub’s helper shape, test.sh beside verifier.py, which the loader grades by test.sh: the folder loads back into Gym as a plain test.sh task that gym dataset validate plays the way Harbor would. Only a dataset-level verifier.py travels in the shim, under tests/.ng/verifier.py beside the runner (a task it grades has no tests/ of its own). A source task that already has tests/test.sh beside a tests/verifier.py (the hub’s helper shape) is a test.sh task to the loader and is copied as it is, with no shim.

A verifier whose verify declares response is refused: Harbor’s test.sh sees the container only, never the agent’s response, so such a task stays Gym-only. So is a Python verifier whose task grades in Harbor’s separate verifier mode ([verifier] environment_mode = "separate", or a [verifier.environment] table, which implies it): Harbor uploads nothing into a separate verifier container (skip_tests_upload in its trial), so the published shim would never reach it and whatever /tests/test.sh the verifier image ships would grade instead. Gym’s own server uploads the shim in both modes, which is why such a task validates under Gym and still cannot be published. A published answer.md becomes a solve.sh that writes the answer to answer.md in the working directory (answer_write_command), which is the only place Harbor’s oracle can put it. Gym’s oracle copies the file there before it submits the text as its response (a command carrying the text would exceed the exec argument limit for a long answer), so a verifier that reads workspace / "answer.md" grades identically under gym dataset validate and under Harbor: that is the publishable answer shape.

What is published is the task folders (of each, what the digest covers, the top-level task files and the environment/, tests/, solution/ and steps/ folders, plus a root .gitignore: together the set Harbor’s own packager reads, the .gitignore being its ignore rules at harbor add and harbor publish, so a task published as it is, with no shim and no solve.sh written for an answer.md, packages to the source’s hub content hash, unless the source holds a byte-code cache or editor leftover that its own .gitignore does not list: publish leaves those out at every depth, where Harbor’s packager reads the .gitignore in place of its defaults and would package them (the one harbor task init writes names __pycache__/ and *.pyc but no *~ or *.swp); a root .env, .git/ or .venv/ is the author’s and stays behind, while a template’s root .gitignore is rendered into every instance, the one hidden root entry the renderer keeps) and dataset.toml ([gym] stripped); everything else at the dataset root (tasks.jsonl, task_data.py, a dataset-level verifier.py, manifest.toml and the image records) is Gym’s and stays behind. The published dataset.toml has to be a manifest Harbor reads, since harbor add takes the dataset branch as soon as the file exists: a [dataset] table with an org/name name is required of what remains (a PublishError otherwise, with or without the harbor package), Harbor’s own DatasetManifest reads the text as part of the format check when the package is installed, and a dataset.toml that held Gym’s [gym] table alone is not published at all (the report says so): the folder is then a plain folder of task folders, which Harbor scans. A task folder holding a symbolic link among the published paths is refused naming the link: copytree would follow it and publish the target’s bytes, host files outside the dataset included, where the loader refuses a file that escapes the task folder, the renderer refuses links and the hub fetcher refuses them.

Module Contents

Classes

Name	Description
`PublishError`	The dataset cannot be published as Harbor tasks.
`PublishReport`	-

Functions

Name	Description
`_check_distinct_names`	Refuse a publish in which two task folders share a `[task].name`: Harbor would keep one of them.
`_check_harbor_manifest_shape`	Refuse a published `dataset.toml` Harbor’s `DatasetManifest` would: no `[dataset]` table, or a `name`
`_check_no_links`	Refuse a task folder that is a symbolic link, or has one anywhere among the paths publish copies.
`_check_task_data`	Every task’s fields against the dataset’s `TaskData`, the check prepare runs before a row is written.
`_harbor_problem`	One line for a refusal by Harbor: a pydantic error’s fields, else the message’s first line.
`_harbor_task_loader`	Harbor’s `Task` and `DatasetManifest` classes and its version, or `None` when the `harbor` package is
`_harbor_task_name`	The `[task].name` Harbor keys the task on, or `None` when the file has no `[task]` table.
`_is_published`	Whether the task path at `relative` (posix, under the task folder) is one publish copies: what the digest
`_publish_task`	Write `task_folder` as a Harbor task at `destination` and return its `[task].name` (`None` without one).
`_read_task_toml`	-
`_task_workdir`	-
`answer_solve_script`	A `solve.sh` that leaves the reference answer as `answer.md` in the working directory.
`answer_write_command`	The shell command that leaves `answer` as `answer.md` in the current directory, byte for byte.
`harbor_format_check`	Harbor’s own loader over each folder and, when `dataset_toml` is given, its `DatasetManifest` over that
`publish_harbor`	Write `target` (a dataset folder, a task folder or a template folder) as Harbor tasks under `output`.
`published_dataset_toml`	The text of `dataset.toml` as it is published, or `None` when nothing of Harbor’s is in it.
`separate_verifier_mode`	Why the task’s verifier runs in a separate environment under Harbor, or `None` in shared mode.
`strip_gym_table`	`dataset.toml`’s text without its `[gym]` table: `[gym]`, every `[gym.]` and `[[gym.]]` table with

Data

API

class nemo_gym.data.tasks.harbor.publish.PublishError()

Bases: HarborTaskError

The dataset cannot be published as Harbor tasks.

class nemo_gym.data.tasks.harbor.publish.PublishReport(
    output: pathlib.Path,
    tasks: list[str] = list(),
    wrapped_verifiers: list[str] = list(),
    wrapped_answers: list[str] = list(),
    harbor_check: dict[str, str] | None = None,
    harbor_note: str | None = None,
    dataset_toml_note: str | None = None
)

Dataclass

dataset_toml_note

str | None = None

harbor_check

dict[str, str] | None = None

harbor_note

str | None = None

bool

Whether the folders were published: Harbor’s check refused none (or did not run).

output

Path

tasks

list[str] = field(default_factory=list)

wrapped_answers

list[str] = field(default_factory=list)

wrapped_verifiers

list[str] = field(default_factory=list)

nemo_gym.data.tasks.harbor.publish.PublishReport.text() -> str

nemo_gym.data.tasks.harbor.publish._check_distinct_names(
    names: typing.Mapping[str, list[str]]
) -> None

Refuse a publish in which two task folders share a [task].name: Harbor would keep one of them.

nemo_gym.data.tasks.harbor.publish._check_harbor_manifest_shape(
    dataset_file: pathlib.Path,
    manifest: typing.Mapping[str, typing.Any]
) -> None

Refuse a published dataset.toml Harbor’s DatasetManifest would: no [dataset] table, or a name that is not org/name. Gym’s own rule, so the refusal does not depend on the harbor package.

nemo_gym.data.tasks.harbor.publish._check_no_links(
    task_folder: pathlib.Path,
    name: str
) -> None

Refuse a task folder that is a symbolic link, or has one anywhere among the paths publish copies.

The copy follows links, so a link would publish its target’s bytes as a regular file, host files outside the dataset included, where the loader refuses a task file that escapes the folder and the renderer refuses every link. The folder itself first: the folder walk lists a linked task folder (so that the loader can refuse it, which it does), and copytree would otherwise follow the link and publish a whole task from outside the dataset. A link under a path publish leaves behind (a root .venv’s interpreter) is not published, so it is not refused either; a linked root .gitignore is, since that file travels.

nemo_gym.data.tasks.harbor.publish._check_task_data(
    task_folders: list[pathlib.Path],
    dataset_root: pathlib.Path
) -> None

Every task’s fields against the dataset’s TaskData, the check prepare runs before a row is written.

A task_data.py that does not import, a malformed task_data.json or a task whose fields the model refuses is a PublishError naming the task, before anything is written: the published shim would otherwise carry a value that gym eval run and gym dataset validate refuse at prepare, and grade against it under Harbor.

nemo_gym.data.tasks.harbor.publish._harbor_problem(
    exc: Exception
) -> str

One line for a refusal by Harbor: a pydantic error’s fields, else the message’s first line.

nemo_gym.data.tasks.harbor.publish._harbor_task_loader() -> tuple[typing.Any, typing.Any, str] | None

Harbor’s Task and DatasetManifest classes and its version, or None when the harbor package is not installed.

Only a missing package is a skip. A module named harbor that is not Harbor (a folder of that name earlier on sys.path, which happens when environment_servers/ is on it) or a Harbor whose own imports fail is a PublishError naming the import error and, for a shadow, the file import harbor resolved to: the check must not be skipped silently with a hint to install a package that is installed.

nemo_gym.data.tasks.harbor.publish._harbor_task_name(
    config: typing.Mapping[str, typing.Any]
) -> str | None

The [task].name Harbor keys the task on, or None when the file has no [task] table.

nemo_gym.data.tasks.harbor.publish._is_published(
    relative: str
) -> bool

Whether the task path at relative (posix, under the task folder) is one publish copies: what the digest covers (is_covered), and the task root’s .gitignore, which Harbor’s packager reads beyond those paths as its ignore rules at harbor add and harbor publish (harbor task init writes one into every task, leaving node_modules/, .venv/ and .env behind at any depth), so that a task published as it is, with no shim and no solve.sh written for an answer.md, packages to the source’s hub content hash (a shim adds tests/test.sh and tests/.ng/, and an answer.md task solution/solve.sh, which Harbor packages too), unless the source holds a byte-code cache or editor leftover that its own .gitignore does not list: publish leaves those out (is_covered), where Harbor’s packager reads the .gitignore in place of its defaults and would package them. The root task_data.json is covered but left behind by the copy itself.

nemo_gym.data.tasks.harbor.publish._publish_task(
    task_folder: pathlib.Path,
    dataset_root: pathlib.Path,
    destination: pathlib.Path,
    report: nemo_gym.data.tasks.harbor.publish.PublishReport
) -> str | None

Write task_folder as a Harbor task at destination and return its [task].name (None without one).

nemo_gym.data.tasks.harbor.publish._read_task_toml(
    task_folder: pathlib.Path
) -> dict

nemo_gym.data.tasks.harbor.publish._task_workdir(
    data: typing.Mapping[str, typing.Any]
) -> str | None

nemo_gym.data.tasks.harbor.publish.answer_solve_script(
    answer: str
) -> str

A solve.sh that leaves the reference answer as answer.md in the working directory.

nemo_gym.data.tasks.harbor.publish.answer_write_command(
    answer: str
) -> str

The shell command that leaves answer as answer.md in the current directory, byte for byte.

The published solve.sh runs it in the task’s working directory under Harbor’s oracle, from a file, where the quoted answer’s length is no concern. Gym’s oracle copies /solution/answer.md there instead (an exec’s command is one argument to the sandbox’s shell, which Linux caps at 128 KiB on 4 KiB pages), so both leave the same bytes.

nemo_gym.data.tasks.harbor.publish.harbor_format_check(
    task_folders: list[pathlib.Path],
    dataset_toml: str | None = None
) -> tuple[dict[str, str] | None, str]

Harbor’s own loader over each folder and, when dataset_toml is given, its DatasetManifest over that text: (problems by task, note), the manifest’s problem under dataset.toml; problems None without the package.

nemo_gym.data.tasks.harbor.publish.publish_harbor(
    target: pathlib.Path,
    output: pathlib.Path
) -> nemo_gym.data.tasks.harbor.publish.PublishReport

Write target (a dataset folder, a task folder or a template folder) as Harbor tasks under output.

Every task’s fields pass the dataset’s TaskData first (_check_task_data); the folders are written beside output, Harbor’s format check runs over them there, and they land in output at once, after every task published and passed; two folders sharing a [task].name are a PublishError first (_check_distinct_names), as is a task folder holding a symbolic link (_check_no_links). A publish refused by Gym (a PublishError) or by Harbor (a report with ok false, the refused folders named) leaves no output behind. Harbor’s check is a step of every publish, not an option: it is skipped (and said so, with the install hint) only when the harbor package is not installed; a harbor that does not import is an error, and nothing is published. The check covers the published dataset.toml too (Harbor’s DatasetManifest over the stripped text, reported under its file name), after Gym’s own rule on it (published_dataset_toml: a [dataset] table with an org/name name, or nothing of Harbor’s left, in which case no dataset.toml is published and the report says so).

nemo_gym.data.tasks.harbor.publish.published_dataset_toml(
    dataset_file: pathlib.Path
) -> str | None

The text of dataset.toml as it is published, or None when nothing of Harbor’s is in it.

Unchanged without a [gym] table, else with the table stripped (strip_gym_table) and checked: the result must parse to the original document minus gym, so a file the line-based strip cannot handle (a [gym] header inside a multi-line string) is a PublishError rather than a mangled manifest. What remains has to be a manifest Harbor reads, because harbor add takes the dataset branch whenever the file exists: a [dataset] table whose name is org/name (is_harbor_name, Harbor’s own pattern) is required, whether or not the harbor package is installed to check the rest; a file that held the [gym] table alone is None, not published, so the output is a plain folder of task folders.

nemo_gym.data.tasks.harbor.publish.separate_verifier_mode(
    data: typing.Mapping[str, typing.Any]
) -> str | None

Why the task’s verifier runs in a separate environment under Harbor, or None in shared mode.

Harbor 0.22.0’s resolve_task_verifier_mode: an explicit [verifier] environment_mode wins; without one, a [verifier.environment] table implies "separate"; otherwise the verifier is shared.

nemo_gym.data.tasks.harbor.publish.strip_gym_table(
    text: str
) -> str

dataset.toml’s text without its [gym] table: [gym], every [gym.*] and [[gym.*]] table with the lines under it, and a gym.<key> or gym = {...} entry of the root table (before any header; a key named gym under [dataset] or another table is Harbor’s and stays). Every other line is kept as it is, so Harbor’s own tables are untouched byte for byte. published_dataset_toml checks the result parses to the original without gym before it is written.

nemo_gym.data.tasks.harbor.publish.ANSWER_FILE = 'answer.md'

nemo_gym.data.tasks.harbor.publish.INSTALL_HINT = "install the `harbor` package (for example `uv pip install harbor`) to run Harbo...

nemo_gym.data.tasks.harbor.publish.SOLVE_SCRIPT = 'solve.sh'

nemo_gym.data.tasks.harbor.publish._GYM_KEY = re.compile('^gym\\s*[.=]')

nemo_gym.data.tasks.harbor.publish._TABLE_HEADER = re.compile('^\\[\\[?\\s*([^\\]]*?)\\s*\\]\\]?\\s*(?:#.*)?$')