> For clean Markdown of any page, append .md to the page URL.
> For a complete documentation index, see https://docs.nvidia.com/nemo/gym/llms.txt.
> For AI client integration (Claude Code, Cursor, etc.), connect to the MCP server at https://docs.nvidia.com/nemo/gym/_mcp/server.

# nemo_gym.orchestration.jobs

The record a submission leaves behind.

`api.py` is the input schema (what to submit); this is the output schema (what
was submitted).

This module is executor-agnostic on purpose, and that is a rule rather than a
coincidence: it must describe a submission made by ANY executor -- Slurm today,
k8s or a local runner later. Two consequences, both enforced by
`test_jobs_module_is_executor_agnostic`:

* Nothing here imports from `executors/`, so a reader -- today EFB's collect,
  tomorrow `gym eval status` -- can load a record without pulling Slurm, SSH and
  the sbatch templates into its import path.
* No field is typed or named after one executor's vocabulary. Where a docstring
  explains a value by example, it says which executor the example comes from.

Compatibility. A record outlives the Gym that wrote it, so `schema_version` is
read in one direction only: a reader accepts anything at or below its own
version and refuses only what is newer than it understands. Upgrading nemo-gym
therefore never strands records already on disk.

That is only sound if the schema keeps its side of the bargain, so within one
schema version a field may only be ADDED WITH A DEFAULT, or removed. A field
must never be repurposed or have its meaning changed -- an older record would
then be read as if it meant the new thing. Anything of that kind needs a
`SCHEMA_VERSION` bump and an explicit migration, not a silent reinterpretation.

## Module Contents

### Classes

| Name                                                                | Description                                      |
| ------------------------------------------------------------------- | ------------------------------------------------ |
| [`BenchmarkJob`](#nemo_gym-orchestration-jobs-BenchmarkJob)         | One benchmark's submission.                      |
| [`SubmissionRecord`](#nemo_gym-orchestration-jobs-SubmissionRecord) | Everything needed to find a submitted run again. |

### Functions

| Name                                                                              | Description                                                                                     |
| --------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------- |
| [`_git`](#nemo_gym-orchestration-jobs-_git)                                       | -                                                                                               |
| [`_git_install_commit`](#nemo_gym-orchestration-jobs-_git_install_commit)         | The commit recorded in the distribution's PEP 610 `direct_url.json`, or None.                   |
| [`_source_checkout_commit`](#nemo_gym-orchestration-jobs-_source_checkout_commit) | HEAD of the git checkout `package_dir` is tracked in, `-dirty` when it has uncommitted changes. |
| [`_unknown_commit`](#nemo_gym-orchestration-jobs-_unknown_commit)                 | -                                                                                               |
| [`installed_gym_commit`](#nemo_gym-orchestration-jobs-installed_gym_commit)       | Return the git commit of the nemo-gym that is running, or None when it cannot be determined.    |
| [`local_index_dir`](#nemo_gym-orchestration-jobs-local_index_dir)                 | Where this machine remembers its own submissions.                                               |
| [`new_gym_job_id`](#nemo_gym-orchestration-jobs-new_gym_job_id)                   | The submission's primary key, and the run directory's name.                                     |
| [`utc_now`](#nemo_gym-orchestration-jobs-utc_now)                                 | The clock the run directory is named from.                                                      |
| [`utc_timestamp`](#nemo_gym-orchestration-jobs-utc_timestamp)                     | ISO 8601, seconds, explicit Z. A run directory is read on a cluster whose                       |

### Data

[`MANIFEST_NAME`](#nemo_gym-orchestration-jobs-MANIFEST_NAME)

[`RESOLVED_CONFIG_NAME`](#nemo_gym-orchestration-jobs-RESOLVED_CONFIG_NAME)

[`SCHEMA_VERSION`](#nemo_gym-orchestration-jobs-SCHEMA_VERSION)

[`_PACKAGE_DIR`](#nemo_gym-orchestration-jobs-_PACKAGE_DIR)

[`logger`](#nemo_gym-orchestration-jobs-logger)

### API

```python
class nemo_gym.orchestration.jobs.BenchmarkJob()
```

**Bases:** `BaseModel`

One benchmark's submission.

`job_id` is None exactly when the executor failed to enqueue this benchmark
(for the Slurm executor, a failed `sbatch`), in which case `error` says why.
The other benchmarks in the same submission are unaffected -- one failure
does not discard the record for the ones that did start.

**`benchmark`** `str`

---

**`error`** `str | None = None`

---

**`job_dir`** `str`

---

**`job_id`** `str | None = None`

---

```python
class nemo_gym.orchestration.jobs.SubmissionRecord()
```

**Bases:** `BaseModel`

Everything needed to find a submitted run again.

`executor` names the executor that produced this record, and is the field a
reader branches on before interpreting the executor-shaped parts of the rest
\-- `job_id`, for instance, is a Slurm job ID under the Slurm executor and
need not be numeric under another. It is deliberately a plain `str` rather
than an enum of the executors that happen to exist today, so that adding one
does not require a schema version bump on every reader.

`hostname` None means the submission was made from the machine that runs the
workload manager's client directly, rather than reaching it over SSH -- not
that the host is unknown. Executors with no remote-submission concept at all
leave it None.

`gym_commit` is the commit of the Gym that wrote the record, when its install
records one (see `installed_gym_commit`). A `-dirty` suffix marks a checkout
with uncommitted changes: the commit alone does not reproduce that Gym.

**`benchmarks`** `list[BenchmarkJob]`

---

**`cluster`** `str`

---

**`executor`** `str`

---

**`executor_metadata`** `dict[str, str] = {}`

---

**`failed`** `list[BenchmarkJob]`

---

**`gym_commit`** `str | None = None`

---

**`gym_job_id`** `str`

---

**`gym_version`** `str`

---

**`hostname`** `str | None = None`

---

**`run_dir`** `str`

---

**`schema_version`** `int = SCHEMA_VERSION`

---

**`submitted_at`** `str`

---

**`submitted_by`** `str`

---

```python
nemo_gym.orchestration.jobs.SubmissionRecord.dumps() -> str
```

The manifest's on-disk bytes; one spelling, so every store matches.

```python
nemo_gym.orchestration.jobs.SubmissionRecord.load(
    payload: dict
) -> nemo_gym.orchestration.jobs.SubmissionRecord
```

classmethod

Parse a record, refusing only one written by a NEWER Gym.

Older records stay readable: fields added since take their defaults, which
the module docstring's compatibility rule is there to guarantee. Refusing
them instead would mean an upgrade silently orphaned every job already
submitted.

A newer record is the one case that cannot be read safely -- it may carry
fields whose meaning this version does not know -- so it fails, and says
which way round the mismatch is.

```python
nemo_gym.orchestration.jobs.SubmissionRecord.write_local_index() -> pathlib.Path | None
```

Record the submission on this machine, or report why not and carry on.

Best-effort by design: this runs after the jobs are queued, so raising
here would report a failure for work that is really running. The durable
copy is the manifest in the run directory.

```python
nemo_gym.orchestration.jobs._git(
    directory: pathlib.Path,
    args: str = ()
) -> subprocess.CompletedProcess
```

```python
nemo_gym.orchestration.jobs._git_install_commit() -> str | None
```

The commit recorded in the distribution's PEP 610 `direct_url.json`, or None.

```python
nemo_gym.orchestration.jobs._source_checkout_commit(
    package_dir: pathlib.Path
) -> str | None
```

HEAD of the git checkout `package_dir` is tracked in, `-dirty` when it has uncommitted changes.

```python
nemo_gym.orchestration.jobs._unknown_commit(
    reason: str,
    args: object = ()
) -> None
```

```python
nemo_gym.orchestration.jobs.installed_gym_commit() -> str | None
```

Return the git commit of the nemo-gym that is running, or None when it cannot be determined.

A git install (`pip install git+...`) records its commit in the distribution
metadata. Otherwise the running package's own source tree is asked: a git
checkout reports its HEAD, suffixed `-dirty` when it has uncommitted changes.

```python
nemo_gym.orchestration.jobs.local_index_dir() -> pathlib.Path
```

Where this machine remembers its own submissions.

```python
nemo_gym.orchestration.jobs.new_gym_job_id(
    now: datetime.datetime
) -> str
```

The submission's primary key, and the run directory's name.

The random suffix is what keeps two submits in the same second against the
same `job.output_path` from sharing a directory. Sharing one is not merely
untidy: an executor that stages by mirroring a directory will delete what it
does not recognise (the Slurm executor copies with `rsync --delete`), so the
second submit silently erases the first's staged scripts.

```python
nemo_gym.orchestration.jobs.utc_now() -> datetime.datetime
```

The clock the run directory is named from.

Here rather than in an executor because run-directory identity is this
module's concern, and every executor needs the same answer. A seam: tests
freeze it to prove two submits in the same second still get distinct
directories.

```python
nemo_gym.orchestration.jobs.utc_timestamp(
    now: datetime.datetime
) -> str
```

ISO 8601, seconds, explicit Z. A run directory is read on a cluster whose
timezone need not match the submitter's.

```python
nemo_gym.orchestration.jobs.MANIFEST_NAME = 'gym-job.json'
```

```python
nemo_gym.orchestration.jobs.RESOLVED_CONFIG_NAME = 'resolved-config.yaml'
```

```python
nemo_gym.orchestration.jobs.SCHEMA_VERSION = 1
```

```python
nemo_gym.orchestration.jobs._PACKAGE_DIR = Path(__file__).resolve().parents[1]
```

```python
nemo_gym.orchestration.jobs.logger = logging.getLogger(__name__)
```