> This page is for version 0.4.0.
> For other versions, use one of these documentation indexes:
> - Main (default): https://docs.nvidia.com/nemo/gym/main/llms.txt
> - 0.7.0: https://docs.nvidia.com/nemo/gym/v0.7.0/llms.txt
> - 0.6.0: https://docs.nvidia.com/nemo/gym/v0.6.0/llms.txt
> - 0.5.1: https://docs.nvidia.com/nemo/gym/v0.5.1/llms.txt
> - 0.5.0: https://docs.nvidia.com/nemo/gym/v0.5.0/llms.txt
> - 0.4.0: https://docs.nvidia.com/nemo/gym/v0.4.0/llms.txt
> - 0.3.0: https://docs.nvidia.com/nemo/gym/v0.3.0/llms.txt
> - 0.2.1: https://docs.nvidia.com/nemo/gym/v0.2.1/llms.txt

> For clean Markdown of any page, append .md to the page URL.
> For a complete documentation index, see https://docs.nvidia.com/nemo/gym/llms.txt.
> For AI client integration (Claude Code, Cursor, etc.), connect to the MCP server at https://docs.nvidia.com/nemo/gym/_mcp/server.

# nemo_gym.tasks.harbor.hub

Resolve `harbor:<dataset>[@<version>]` through the Harbor registry.

The registry is one JSON file listing datasets; each task entry names a git
repository, a commit and a path. Fetching copies the task folders into Gym's shared
datasets folder so that every server reads the same bytes and nothing downloads at
run time. A `HEAD` pin is resolved to a commit before anything is copied.

## Module Contents

### Classes

| Name                                                            | Description                                  |
| --------------------------------------------------------------- | -------------------------------------------- |
| [`HubError`](#nemo_gym-tasks-harbor-hub-HubError)               | The reference cannot be resolved or fetched. |
| [`HubRef`](#nemo_gym-tasks-harbor-hub-HubRef)                   | -                                            |
| [`RegistryDataset`](#nemo_gym-tasks-harbor-hub-RegistryDataset) | -                                            |
| [`RegistryTask`](#nemo_gym-tasks-harbor-hub-RegistryTask)       | -                                            |

### Functions

| Name                                                                    | Description                                                                              |
| ----------------------------------------------------------------------- | ---------------------------------------------------------------------------------------- |
| [`_git`](#nemo_gym-tasks-harbor-hub-_git)                               | -                                                                                        |
| [`_numeric_version`](#nemo_gym-tasks-harbor-hub-_numeric_version)       | -                                                                                        |
| [`_parse_registry`](#nemo_gym-tasks-harbor-hub-_parse_registry)         | -                                                                                        |
| [`_read_manifest_pins`](#nemo_gym-tasks-harbor-hub-_read_manifest_pins) | The `(git_url, commit, path)` pins a previous fetch recorded, or `{}`.         |
| [`_validate_task_path`](#nemo_gym-tasks-harbor-hub-_validate_task_path) | -                                                                                        |
| [`_validate_task_tree`](#nemo_gym-tasks-harbor-hub-_validate_task_tree) | Refuse links that could import host files, before any task bytes are copied.             |
| [`_version_key`](#nemo_gym-tasks-harbor-hub-_version_key)               | Total order over versions: numeric versions first (by component), then the rest by text. |
| [`_write_manifest`](#nemo_gym-tasks-harbor-hub-_write_manifest)         | -                                                                                        |
| [`datasets_dir`](#nemo_gym-tasks-harbor-hub-datasets_dir)               | Gym's shared datasets folder: `$NEMO_GYM_DATASETS_DIR` or `./datasets`.                  |
| [`fetch_dataset`](#nemo_gym-tasks-harbor-hub-fetch_dataset)             | Fetch tasks, retaining the recorded source pins of existing folders.                     |
| [`fetch_ref`](#nemo_gym-tasks-harbor-hub-fetch_ref)                     | Resolve and fetch a `harbor:` reference; return the dataset folder.                      |
| [`is_hub_ref`](#nemo_gym-tasks-harbor-hub-is_hub_ref)                   | -                                                                                        |
| [`load_registry`](#nemo_gym-tasks-harbor-hub-load_registry)             | Read the registry, validating downloads before atomically replacing the cache.           |
| [`resolve_commit`](#nemo_gym-tasks-harbor-hub-resolve_commit)           | Pin `HEAD` (or a branch name) to the commit it points at right now.                      |
| [`resolve_dataset`](#nemo_gym-tasks-harbor-hub-resolve_dataset)         | -                                                                                        |
| [`validate_git_url`](#nemo_gym-tasks-harbor-hub-validate_git_url)       | Return `git_url` if it is an https, ssh or git URL (or `user@host:path`); else raise.    |
| [`validate_task_name`](#nemo_gym-tasks-harbor-hub-validate_task_name)   | Require one folder component, reserving the dataset manifest's name.                     |

### Data

[`DATASETS_DIR_ENV`](#nemo_gym-tasks-harbor-hub-DATASETS_DIR_ENV)

[`MANIFEST_FILE`](#nemo_gym-tasks-harbor-hub-MANIFEST_FILE)

[`REF_PREFIX`](#nemo_gym-tasks-harbor-hub-REF_PREFIX)

[`REGISTRY_URL`](#nemo_gym-tasks-harbor-hub-REGISTRY_URL)

[`_GIT_URL_SCHEME`](#nemo_gym-tasks-harbor-hub-_GIT_URL_SCHEME)

[`_GIT_URL_SCP`](#nemo_gym-tasks-harbor-hub-_GIT_URL_SCP)

[`_NAME`](#nemo_gym-tasks-harbor-hub-_NAME)

[`_SHA`](#nemo_gym-tasks-harbor-hub-_SHA)

[`_UNSAFE`](#nemo_gym-tasks-harbor-hub-_UNSAFE)

[`logger`](#nemo_gym-tasks-harbor-hub-logger)

### API

```python
class nemo_gym.tasks.harbor.hub.HubError()
```

**Bases:** `ValueError`

The reference cannot be resolved or fetched.

```python
class nemo_gym.tasks.harbor.hub.HubRef(
    name: str,
    version: str | None
)
```

Dataclass

**`name`** `str`

---

**`version`** `str | None`

---

```python
nemo_gym.tasks.harbor.hub.HubRef.parse(
    target: str
) -> nemo_gym.tasks.harbor.hub.HubRef
```

classmethod

```python
class nemo_gym.tasks.harbor.hub.RegistryDataset(
    name: str,
    version: str,
    description: str,
    tasks: tuple[nemo_gym.tasks.harbor.hub.RegistryTask, ...]
)
```

Dataclass

**`description`** `str`

---

**`folder_name`** `str`

`<name>-<version>`, so two versions of one dataset never share a folder.

---

**`name`** `str`

---

**`tasks`** `tuple[RegistryTask, ...]`

---

**`version`** `str`

---

```python
class nemo_gym.tasks.harbor.hub.RegistryTask(
    name: str,
    git_url: str,
    git_commit_id: str,
    path: str
)
```

Dataclass

**`git_commit_id`** `str`

---

**`git_url`** `str`

---

**`name`** `str`

---

**`path`** `str`

---

```python
nemo_gym.tasks.harbor.hub._git(
    args: str = (),
    cwd: pathlib.Path | None = None
) -> str
```

```python
nemo_gym.tasks.harbor.hub._numeric_version(
    version: str
) -> tuple[int, ...] | None
```

```python
nemo_gym.tasks.harbor.hub._parse_registry(
    payload: bytes
) -> list[nemo_gym.tasks.harbor.hub.RegistryDataset]
```

```python
nemo_gym.tasks.harbor.hub._read_manifest_pins(
    folder: pathlib.Path
) -> dict[str, tuple[str, str, str]]
```

The `(git_url, commit, path)` pins a previous fetch recorded, or `{}`.

```python
nemo_gym.tasks.harbor.hub._validate_task_path(
    path: str
) -> None
```

```python
nemo_gym.tasks.harbor.hub._validate_task_tree(
    source: pathlib.Path
) -> None
```

Refuse links that could import host files, before any task bytes are copied.

```python
nemo_gym.tasks.harbor.hub._version_key(
    version: str
) -> tuple
```

Total order over versions: numeric versions first (by component), then the rest by text.

```python
nemo_gym.tasks.harbor.hub._write_manifest(
    folder: pathlib.Path,
    dataset: nemo_gym.tasks.harbor.hub.RegistryDataset,
    pins: dict[str, tuple[str, str, str]]
) -> None
```

```python
nemo_gym.tasks.harbor.hub.datasets_dir() -> pathlib.Path
```

Gym's shared datasets folder: `$NEMO_GYM_DATASETS_DIR` or `./datasets`.

```python
nemo_gym.tasks.harbor.hub.fetch_dataset(
    dataset: nemo_gym.tasks.harbor.hub.RegistryDataset,
    root: pathlib.Path
) -> pathlib.Path
```

Fetch tasks, retaining the recorded source pins of existing folders.

Publish a pin after each task, so a later failure leaves completed tasks reusable.
A folder without a trustworthy pin (including interruption before pin publication)
is refused; the user can move it aside and retry without losing local edits.

```python
nemo_gym.tasks.harbor.hub.fetch_ref(
    target: str,
    root: pathlib.Path | None = None,
    refresh_registry: bool = False
) -> pathlib.Path
```

Resolve and fetch a `harbor:` reference; return the dataset folder.

```python
nemo_gym.tasks.harbor.hub.is_hub_ref(
    target: str
) -> bool
```

```python
nemo_gym.tasks.harbor.hub.load_registry(
    cache_dir: pathlib.Path,
    refresh: bool = False
) -> list[nemo_gym.tasks.harbor.hub.RegistryDataset]
```

Read the registry, validating downloads before atomically replacing the cache.

A damaged cache raises `HubError` with its path and recovery instructions.
Failed downloads or updates leave any previous cache intact.

```python
nemo_gym.tasks.harbor.hub.resolve_commit(
    git_url: str,
    git_commit_id: str,
    cache: dict[tuple[str, str], str] | None = None
) -> str
```

Pin `HEAD` (or a branch name) to the commit it points at right now.

`cache` memoizes the answer per `(git_url, ref)` so one fetch asks each remote once.

```python
nemo_gym.tasks.harbor.hub.resolve_dataset(
    ref: nemo_gym.tasks.harbor.hub.HubRef,
    registry: list[nemo_gym.tasks.harbor.hub.RegistryDataset]
) -> nemo_gym.tasks.harbor.hub.RegistryDataset
```

```python
nemo_gym.tasks.harbor.hub.validate_git_url(
    git_url: str
) -> str
```

Return `git_url` if it is an https, ssh or git URL (or `user@host:path`); else raise.

```python
nemo_gym.tasks.harbor.hub.validate_task_name(
    name: str
) -> str
```

Require one folder component, reserving the dataset manifest's name.

```python
nemo_gym.tasks.harbor.hub.DATASETS_DIR_ENV = 'NEMO_GYM_DATASETS_DIR'
```

```python
nemo_gym.tasks.harbor.hub.MANIFEST_FILE = 'manifest.toml'
```

```python
nemo_gym.tasks.harbor.hub.REF_PREFIX = 'harbor:'
```

```python
nemo_gym.tasks.harbor.hub.REGISTRY_URL = 'https://raw.githubusercontent.com/harbor-framework/harbor/main/registry.json'
```

```python
nemo_gym.tasks.harbor.hub._GIT_URL_SCHEME = re.compile('(https|ssh|git)://[A-Za-z0-9]', re.IGNORECASE)
```

```python
nemo_gym.tasks.harbor.hub._GIT_URL_SCP = re.compile('[A-Za-z0-9][A-Za-z0-9._-]*@[A-Za-z0-9][A-Za-z0-9.-]*:(?!-)[^:]*')
```

```python
nemo_gym.tasks.harbor.hub._NAME = re.compile('[A-Za-z0-9][A-Za-z0-9._-]*')
```

```python
nemo_gym.tasks.harbor.hub._SHA = re.compile('[0-9a-f]{40}')
```

```python
nemo_gym.tasks.harbor.hub._UNSAFE = re.compile('[^A-Za-z0-9._-]')
```

```python
nemo_gym.tasks.harbor.hub.logger = logging.getLogger(__name__)
```