> For clean Markdown of any page, append .md to the page URL.
> For a complete documentation index, see https://docs.nvidia.com/nemo/gym/llms.txt.
> For AI client integration (Claude Code, Cursor, etc.), connect to the MCP server at https://docs.nvidia.com/nemo/gym/_mcp/server.

# nemo_gym.orchestration.jobs

The record a submission leaves behind.

`api.py` is the input schema (what to submit); this is the output schema (what
was submitted).

This module is executor-agnostic on purpose, and that is a rule rather than a
coincidence: it must describe a submission made by ANY executor -- Slurm today,
k8s or a local runner later. Two consequences, both enforced by
`test_jobs_module_is_executor_agnostic`:

* Nothing here imports from `executors/`, so a reader -- today EFB's collect,
  tomorrow `gym eval status` -- can load a record without pulling Slurm, SSH and
  the sbatch templates into its import path.
* No field is typed or named after one executor's vocabulary. Where a docstring
  explains a value by example, it says which executor the example comes from.

Compatibility. A record outlives the Gym that wrote it, so `schema_version` is
read in one direction only: a reader accepts anything at or below its own
version and refuses only what is newer than it understands. Upgrading nemo-gym
therefore never strands records already on disk.

That is only sound if the schema keeps its side of the bargain, so within one
schema version a field may only be ADDED WITH A DEFAULT, or removed. A field
must never be repurposed or have its meaning changed -- an older record would
then be read as if it meant the new thing. Anything of that kind needs a
`SCHEMA_VERSION` bump and an explicit migration, not a silent reinterpretation.

## Module Contents

### Classes

| Name                                                                | Description                                      |
| ------------------------------------------------------------------- | ------------------------------------------------ |
| [`BenchmarkJob`](#nemo_gym-orchestration-jobs-BenchmarkJob)         | One benchmark's submission.                      |
| [`SubmissionRecord`](#nemo_gym-orchestration-jobs-SubmissionRecord) | Everything needed to find a submitted run again. |

### Functions

| Name                                                              | Description                                                               |
| ----------------------------------------------------------------- | ------------------------------------------------------------------------- |
| [`local_index_dir`](#nemo_gym-orchestration-jobs-local_index_dir) | Where this machine remembers its own submissions.                         |
| [`new_gym_job_id`](#nemo_gym-orchestration-jobs-new_gym_job_id)   | The submission's primary key, and the run directory's name.               |
| [`utc_now`](#nemo_gym-orchestration-jobs-utc_now)                 | The clock the run directory is named from.                                |
| [`utc_timestamp`](#nemo_gym-orchestration-jobs-utc_timestamp)     | ISO 8601, seconds, explicit Z. A run directory is read on a cluster whose |

### Data

[`MANIFEST_NAME`](#nemo_gym-orchestration-jobs-MANIFEST_NAME)

[`RESOLVED_CONFIG_NAME`](#nemo_gym-orchestration-jobs-RESOLVED_CONFIG_NAME)

[`SCHEMA_VERSION`](#nemo_gym-orchestration-jobs-SCHEMA_VERSION)

### API

```python
class nemo_gym.orchestration.jobs.BenchmarkJob()
```

**Bases:** `BaseModel`

One benchmark's submission.

`job_id` is None exactly when the executor failed to enqueue this benchmark
(for the Slurm executor, a failed `sbatch`), in which case `error` says why.
The other benchmarks in the same submission are unaffected -- one failure
does not discard the record for the ones that did start.

**`benchmark`** `str`

---

**`error`** `str | None = None`

---

**`job_dir`** `str`

---

**`job_id`** `str | None = None`

---

```python
class nemo_gym.orchestration.jobs.SubmissionRecord()
```

**Bases:** `BaseModel`

Everything needed to find a submitted run again.

`executor` names the executor that produced this record, and is the field a
reader branches on before interpreting the executor-shaped parts of the rest
\-- `job_id`, for instance, is a Slurm job ID under the Slurm executor and
need not be numeric under another. It is deliberately a plain `str` rather
than an enum of the executors that happen to exist today, so that adding one
does not require a schema version bump on every reader.

`hostname` None means the submission was made from the machine that runs the
workload manager's client directly, rather than reaching it over SSH -- not
that the host is unknown. Executors with no remote-submission concept at all
leave it None.

**`benchmarks`** `list[BenchmarkJob]`

---

**`cluster`** `str`

---

**`executor`** `str`

---

**`executor_metadata`** `dict[str, str] = {}`

---

**`failed`** `list[BenchmarkJob]`

---

**`gym_job_id`** `str`

---

**`gym_version`** `str`

---

**`hostname`** `str | None = None`

---

**`run_dir`** `str`

---

**`schema_version`** `int = SCHEMA_VERSION`

---

**`submitted_at`** `str`

---

**`submitted_by`** `str`

---

```python
nemo_gym.orchestration.jobs.SubmissionRecord.dumps() -> str
```

The manifest's on-disk bytes; one spelling, so every store matches.

```python
nemo_gym.orchestration.jobs.SubmissionRecord.load(
    payload: dict
) -> nemo_gym.orchestration.jobs.SubmissionRecord
```

classmethod

Parse a record, refusing only one written by a NEWER Gym.

Older records stay readable: fields added since take their defaults, which
the module docstring's compatibility rule is there to guarantee. Refusing
them instead would mean an upgrade silently orphaned every job already
submitted.

A newer record is the one case that cannot be read safely -- it may carry
fields whose meaning this version does not know -- so it fails, and says
which way round the mismatch is.

```python
nemo_gym.orchestration.jobs.SubmissionRecord.write_local_index() -> pathlib.Path | None
```

Record the submission on this machine, or report why not and carry on.

Best-effort by design: this runs after the jobs are queued, so raising
here would report a failure for work that is really running. The durable
copy is the manifest in the run directory.

```python
nemo_gym.orchestration.jobs.local_index_dir() -> pathlib.Path
```

Where this machine remembers its own submissions.

```python
nemo_gym.orchestration.jobs.new_gym_job_id(
    now: datetime.datetime
) -> str
```

The submission's primary key, and the run directory's name.

The random suffix is what keeps two submits in the same second against the
same `job.output_path` from sharing a directory. Sharing one is not merely
untidy: an executor that stages by mirroring a directory will delete what it
does not recognise (the Slurm executor copies with `rsync --delete`), so the
second submit silently erases the first's staged scripts.

```python
nemo_gym.orchestration.jobs.utc_now() -> datetime.datetime
```

The clock the run directory is named from.

Here rather than in an executor because run-directory identity is this
module's concern, and every executor needs the same answer. A seam: tests
freeze it to prove two submits in the same second still get distinct
directories.

```python
nemo_gym.orchestration.jobs.utc_timestamp(
    now: datetime.datetime
) -> str
```

ISO 8601, seconds, explicit Z. A run directory is read on a cluster whose
timezone need not match the submitter's.

```python
nemo_gym.orchestration.jobs.MANIFEST_NAME = 'gym-job.json'
```

```python
nemo_gym.orchestration.jobs.RESOLVED_CONFIG_NAME = 'resolved-config.yaml'
```

```python
nemo_gym.orchestration.jobs.SCHEMA_VERSION = 1
```