nemo_gym.orchestration.jobs

The record a submission leaves behind.

api.py is the input schema (what to submit); this is the output schema (what was submitted).

This module is executor-agnostic on purpose, and that is a rule rather than a coincidence: it must describe a submission made by ANY executor — Slurm today, k8s or a local runner later. Two consequences, both enforced by test_jobs_module_is_executor_agnostic:

Nothing here imports from executors/, so a reader — today EFB’s collect, tomorrow gym eval status — can load a record without pulling Slurm, SSH and the sbatch templates into its import path.
No field is typed or named after one executor’s vocabulary. Where a docstring explains a value by example, it says which executor the example comes from.

Compatibility. A record outlives the Gym that wrote it, so schema_version is read in one direction only: a reader accepts anything at or below its own version and refuses only what is newer than it understands. Upgrading nemo-gym therefore never strands records already on disk.

That is only sound if the schema keeps its side of the bargain, so within one schema version a field may only be ADDED WITH A DEFAULT, or removed. A field must never be repurposed or have its meaning changed — an older record would then be read as if it meant the new thing. Anything of that kind needs a SCHEMA_VERSION bump and an explicit migration, not a silent reinterpretation.

Module Contents

Classes

Name	Description
`BenchmarkJob`	One benchmark’s submission.
`SubmissionRecord`	Everything needed to find a submitted run again.

Functions

Name	Description
`local_index_dir`	Where this machine remembers its own submissions.
`new_gym_job_id`	The submission’s primary key, and the run directory’s name.
`utc_now`	The clock the run directory is named from.
`utc_timestamp`	ISO 8601, seconds, explicit Z. A run directory is read on a cluster whose

Data

MANIFEST_NAME

RESOLVED_CONFIG_NAME

SCHEMA_VERSION

API

class nemo_gym.orchestration.jobs.BenchmarkJob()

Bases: BaseModel

One benchmark’s submission.

job_id is None exactly when the executor failed to enqueue this benchmark (for the Slurm executor, a failed sbatch), in which case error says why. The other benchmarks in the same submission are unaffected — one failure does not discard the record for the ones that did start.

benchmark

str

error

str | None = None

job_dir

str

job_id

str | None = None

class nemo_gym.orchestration.jobs.SubmissionRecord()

Bases: BaseModel

Everything needed to find a submitted run again.

executor names the executor that produced this record, and is the field a reader branches on before interpreting the executor-shaped parts of the rest — job_id, for instance, is a Slurm job ID under the Slurm executor and need not be numeric under another. It is deliberately a plain str rather than an enum of the executors that happen to exist today, so that adding one does not require a schema version bump on every reader.

hostname None means the submission was made from the machine that runs the workload manager’s client directly, rather than reaching it over SSH — not that the host is unknown. Executors with no remote-submission concept at all leave it None.

benchmarks

list[BenchmarkJob]

cluster

str

executor

str

executor_metadata

dict[str, str] = {}

failed

list[BenchmarkJob]

gym_job_id

str

gym_version

str

hostname

str | None = None

run_dir

str

schema_version

int = SCHEMA_VERSION

submitted_at

str

submitted_by

str

nemo_gym.orchestration.jobs.SubmissionRecord.dumps() -> str

The manifest’s on-disk bytes; one spelling, so every store matches.

nemo_gym.orchestration.jobs.SubmissionRecord.load(
    payload: dict
) -> nemo_gym.orchestration.jobs.SubmissionRecord

classmethod

Parse a record, refusing only one written by a NEWER Gym.

Older records stay readable: fields added since take their defaults, which the module docstring’s compatibility rule is there to guarantee. Refusing them instead would mean an upgrade silently orphaned every job already submitted.

A newer record is the one case that cannot be read safely — it may carry fields whose meaning this version does not know — so it fails, and says which way round the mismatch is.

nemo_gym.orchestration.jobs.SubmissionRecord.write_local_index() -> pathlib.Path | None

Record the submission on this machine, or report why not and carry on.

Best-effort by design: this runs after the jobs are queued, so raising here would report a failure for work that is really running. The durable copy is the manifest in the run directory.

nemo_gym.orchestration.jobs.local_index_dir() -> pathlib.Path

Where this machine remembers its own submissions.

nemo_gym.orchestration.jobs.new_gym_job_id(
    now: datetime.datetime
) -> str

The submission’s primary key, and the run directory’s name.

The random suffix is what keeps two submits in the same second against the same job.output_path from sharing a directory. Sharing one is not merely untidy: an executor that stages by mirroring a directory will delete what it does not recognise (the Slurm executor copies with rsync --delete), so the second submit silently erases the first’s staged scripts.

nemo_gym.orchestration.jobs.utc_now() -> datetime.datetime

The clock the run directory is named from.

Here rather than in an executor because run-directory identity is this module’s concern, and every executor needs the same answer. A seam: tests freeze it to prove two submits in the same second still get distinct directories.

nemo_gym.orchestration.jobs.utc_timestamp(
    now: datetime.datetime
) -> str

ISO 8601, seconds, explicit Z. A run directory is read on a cluster whose timezone need not match the submitter’s.

nemo_gym.orchestration.jobs.MANIFEST_NAME = 'gym-job.json'

nemo_gym.orchestration.jobs.RESOLVED_CONFIG_NAME = 'resolved-config.yaml'

nemo_gym.orchestration.jobs.SCHEMA_VERSION = 1