Skip to navigation
API ReferenceFull Library ReferenceNemo GymNemo GymTasksHarbor

nemo_gym.tasks.harbor.hub

View as Markdown

Resolve harbor:<dataset>[@<version>] through the Harbor registry.

The registry is one JSON file listing datasets; each task entry names a git repository, a commit and a path. Fetching copies the task folders into Gym’s shared datasets folder so that every server reads the same bytes and nothing downloads at run time. A HEAD pin is resolved to a commit before anything is copied.

Module Contents

Classes

NameDescription
HubErrorThe reference cannot be resolved or fetched.
HubRef-
RegistryDataset-
RegistryTask-

Functions

NameDescription
_git-
_numeric_version-
_parse_registry-
_read_manifest_pinsThe (git_url, commit, path) pins a previous fetch recorded, or {}.
_validate_task_path-
_validate_task_treeRefuse links that could import host files, before any task bytes are copied.
_version_keyTotal order over versions: numeric versions first (by component), then the rest by text.
_write_manifest-
datasets_dirGym’s shared datasets folder: $NEMO_GYM_DATASETS_DIR or ./datasets.
fetch_datasetFetch tasks, retaining the recorded source pins of existing folders.
fetch_refResolve and fetch a harbor: reference; return the dataset folder.
is_hub_ref-
load_registryRead the registry, validating downloads before atomically replacing the cache.
resolve_commitPin HEAD (or a branch name) to the commit it points at right now.
resolve_dataset-
validate_git_urlReturn git_url if it is an https, ssh or git URL (or user@host:path); else raise.
validate_task_nameRequire one folder component, reserving the dataset manifest’s name.

Data

DATASETS_DIR_ENV

MANIFEST_FILE

REF_PREFIX

REGISTRY_URL

_GIT_URL_SCHEME

_GIT_URL_SCP

_NAME

_SHA

_UNSAFE

logger

API

class nemo_gym.tasks.harbor.hub.HubError()

Bases: ValueError

The reference cannot be resolved or fetched.

class nemo_gym.tasks.harbor.hub.HubRef(
    name: str,
    version: str | None
)
Dataclass
name
str
version
str | None
classmethod
class nemo_gym.tasks.harbor.hub.RegistryDataset(
    name: str,
    version: str,
    description: str,
    tasks: tuple[nemo_gym.tasks.harbor.hub.RegistryTask, ...]
)
Dataclass
description
str
folder_name
str

<name>-<version>, so two versions of one dataset never share a folder.

name
str
tasks
tuple[RegistryTask, ...]
version
str
class nemo_gym.tasks.harbor.hub.RegistryTask(
    name: str,
    git_url: str,
    git_commit_id: str,
    path: str
)
Dataclass
git_commit_id
str
git_url
str
name
str
path
str
nemo_gym.tasks.harbor.hub._git(
    args: str = (),
    cwd: pathlib.Path | None = None
) -> str
 
nemo_gym.tasks.harbor.hub._numeric_version(
    version: str
) -> tuple[int, ...] | None
 
nemo_gym.tasks.harbor.hub._parse_registry(
    payload: bytes
) -> list[nemo_gym.tasks.harbor.hub.RegistryDataset]
 
nemo_gym.tasks.harbor.hub._read_manifest_pins(
    folder: pathlib.Path
) -> dict[str, tuple[str, str, str]]

The (git_url, commit, path) pins a previous fetch recorded, or {}.

nemo_gym.tasks.harbor.hub._validate_task_path(
    path: str
) -> None
 
nemo_gym.tasks.harbor.hub._validate_task_tree(
    source: pathlib.Path
) -> None

Refuse links that could import host files, before any task bytes are copied.

nemo_gym.tasks.harbor.hub._version_key(
    version: str
) -> tuple

Total order over versions: numeric versions first (by component), then the rest by text.

nemo_gym.tasks.harbor.hub._write_manifest(
    folder: pathlib.Path,
    dataset: nemo_gym.tasks.harbor.hub.RegistryDataset,
    pins: dict[str, tuple[str, str, str]]
) -> None
 
nemo_gym.tasks.harbor.hub.datasets_dir() -> pathlib.Path

Gym’s shared datasets folder: $NEMO_GYM_DATASETS_DIR or ./datasets.

nemo_gym.tasks.harbor.hub.fetch_dataset(
    dataset: nemo_gym.tasks.harbor.hub.RegistryDataset,
    root: pathlib.Path
) -> pathlib.Path

Fetch tasks, retaining the recorded source pins of existing folders.

Publish a pin after each task, so a later failure leaves completed tasks reusable. A folder without a trustworthy pin (including interruption before pin publication) is refused; the user can move it aside and retry without losing local edits.

nemo_gym.tasks.harbor.hub.fetch_ref(
    target: str,
    root: pathlib.Path | None = None,
    refresh_registry: bool = False
) -> pathlib.Path

Resolve and fetch a harbor: reference; return the dataset folder.

nemo_gym.tasks.harbor.hub.is_hub_ref(
    target: str
) -> bool
 
nemo_gym.tasks.harbor.hub.load_registry(
    cache_dir: pathlib.Path,
    refresh: bool = False
) -> list[nemo_gym.tasks.harbor.hub.RegistryDataset]

Read the registry, validating downloads before atomically replacing the cache.

A damaged cache raises HubError with its path and recovery instructions. Failed downloads or updates leave any previous cache intact.

nemo_gym.tasks.harbor.hub.resolve_commit(
    git_url: str,
    git_commit_id: str,
    cache: dict[tuple[str, str], str] | None = None
) -> str

Pin HEAD (or a branch name) to the commit it points at right now.

cache memoizes the answer per (git_url, ref) so one fetch asks each remote once.

 
nemo_gym.tasks.harbor.hub.validate_git_url(
    git_url: str
) -> str

Return git_url if it is an https, ssh or git URL (or user@host:path); else raise.

nemo_gym.tasks.harbor.hub.validate_task_name(
    name: str
) -> str

Require one folder component, reserving the dataset manifest’s name.

nemo_gym.tasks.harbor.hub.DATASETS_DIR_ENV = 'NEMO_GYM_DATASETS_DIR'
 
nemo_gym.tasks.harbor.hub.MANIFEST_FILE = 'manifest.toml'
 
nemo_gym.tasks.harbor.hub.REF_PREFIX = 'harbor:'
 
nemo_gym.tasks.harbor.hub.REGISTRY_URL = 'https://raw.githubusercontent.com/harbor-framework/harbor/main/registry.json'
 
nemo_gym.tasks.harbor.hub._GIT_URL_SCHEME = re.compile('(https|ssh|git)://[A-Za-z0-9]', re.IGNORECASE)
 
nemo_gym.tasks.harbor.hub._GIT_URL_SCP = re.compile('[A-Za-z0-9][A-Za-z0-9._-]*@[A-Za-z0-9][A-Za-z0-9.-]*:(?!-)[^:]*')
 
nemo_gym.tasks.harbor.hub._NAME = re.compile('[A-Za-z0-9][A-Za-z0-9._-]*')
 
nemo_gym.tasks.harbor.hub._SHA = re.compile('[0-9a-f]{40}')
 
nemo_gym.tasks.harbor.hub._UNSAFE = re.compile('[^A-Za-z0-9._-]')
 
nemo_gym.tasks.harbor.hub.logger = logging.getLogger(__name__)