nemo_gym.orchestration.jobs

The record a submission leaves behind.

api.py is the input schema (what to submit); this is the output schema (what was submitted).

This module is executor-agnostic on purpose, and that is a rule rather than a coincidence: it must describe a submission made by ANY executor — Slurm today, k8s or a local runner later. Two consequences, both enforced by test_jobs_module_is_executor_agnostic:

Nothing here imports from executors/, so a reader — today EFB’s collect, gym eval status — can load a record without pulling Slurm, SSH and the sbatch templates into its import path.
No field is typed or named after one executor’s vocabulary. Where a docstring explains a value by example, it says which executor the example comes from.

Compatibility. A record outlives the Gym that wrote it, so schema_version is read in one direction only: a reader accepts anything at or below its own version and refuses only what is newer than it understands. Upgrading nemo-gym therefore never strands records already on disk.

That is only sound if the schema keeps its side of the bargain, so within one schema version a field may only be ADDED WITH A DEFAULT, or removed. A field must never be repurposed or have its meaning changed — an older record would then be read as if it meant the new thing. Anything of that kind needs a SCHEMA_VERSION bump and an explicit migration, not a silent reinterpretation.

Module Contents

Classes

Name	Description
`BenchmarkJob`	One benchmark’s submission.
`SubmissionEnvironment`	Names observed at submission, never values.
`SubmissionRecord`	Everything needed to find a submitted run again.
`SubmissionTelemetry`	Resolved submission-time telemetry controls, not credentials or a delivery claim.

Functions

Name	Description
`_git`	-
`_git_install_commit`	The commit recorded in the distribution’s PEP 610 `direct_url.json`, or None.
`_source_checkout_commit`	HEAD of the git checkout `package_dir` is tracked in, `-dirty` when it has uncommitted changes.
`_unknown_commit`	-
`installed_gym_commit`	Return the git commit of the nemo-gym that is running, or None when it cannot be determined.
`load_submission_record`	The record of submission `gym_job_id`: the run’s manifest when its run dir is on this machine, else
`local_index_dir`	Where this machine remembers its own submissions.
`new_gym_job_id`	The submission’s primary key, and the run directory’s name.
`restore_index_entry`	Rebuild this machine’s job index entry for `gym_job_id` from the run directory’s own record.
`utc_now`	The clock the run directory is named from.
`utc_timestamp`	ISO 8601, seconds, explicit Z. A run directory is read on a cluster whose

Data

ARTIFACTS_DIRNAME

EVAL_EXIT_FILENAME

EVAL_SUCCEEDED_FILENAME

EXPORTS_DIRNAME

EXPORT_RESULT_SUFFIX

INDEX_RESTORE_JOURNAL_NAME

LOGS_DIRNAME

MANIFEST_NAME

RESOLVED_CONFIG_NAME

API

class nemo_gym.orchestration.jobs.BenchmarkJob()

Bases: BaseModel

One benchmark’s submission.

job_id is None exactly when the executor failed to enqueue this benchmark (for the Slurm executor, a failed sbatch), in which case error says why. The other benchmarks in the same submission are unaffected — one failure does not discard the record for the ones that did start.

benchmark

str

error

str | None = None

gym_commit

str | None = None

job_dir

str

job_id

str | None = None

walltime

str | None = None

class nemo_gym.orchestration.jobs.SubmissionEnvironment()

Bases: BaseModel

Names observed at submission, never values.

required_env names submit-shell inputs (host: references and required forwarding). forwarded_env names variables actually exported by the submission transport. A runtime: alias is not a submit-shell requirement unless explicitly forwarded. None on the containing record means this observation was not recorded.

forwarded_env

list[str]

model_config

= ConfigDict(extra='forbid')

required_env

list[str]

telemetry

SubmissionTelemetry

nemo_gym.orchestration.jobs.SubmissionEnvironment.validate_names(
    names: list[str]
) -> list[str]

classmethod

class nemo_gym.orchestration.jobs.SubmissionRecord()

Bases: BaseModel

Everything needed to find a submitted run again.

executor names the executor that produced this record, and is the field a reader branches on before interpreting the executor-shaped parts of the rest — job_id, for instance, is a Slurm job ID under the Slurm executor and need not be numeric under another. It is deliberately a plain str rather than an enum of the executors that happen to exist today, so that adding one does not require a schema version bump on every reader.

hostname None means the submission was made from the machine that runs the workload manager’s client directly, rather than reaching it over SSH — not that the host is unknown. Executors with no remote-submission concept at all leave it None.

gym_commit is the commit of the Gym that wrote the record, when its install records one (see installed_gym_commit). A -dirty suffix marks a checkout with uncommitted changes: the commit alone does not reproduce that Gym.

benchmarks

list[BenchmarkJob]

cluster

str

environment

SubmissionEnvironment | None

executor

str

executor_metadata

dict[str, str] = {}

failed

list[BenchmarkJob]

gym_commit

str | None = None

gym_job_id

str

gym_version

str

hostname

str | None = None

run_dir

str

schema_version

int = SCHEMA_VERSION

submitted_at

str

submitted_by

str

nemo_gym.orchestration.jobs.SubmissionRecord.benchmark_job(
    benchmark: str
) -> nemo_gym.orchestration.jobs.BenchmarkJob

This record’s entry for benchmark; an unknown name is an error that lists the known ones.

nemo_gym.orchestration.jobs.SubmissionRecord.dumps() -> str

The manifest’s on-disk bytes; one spelling, so every store matches.

nemo_gym.orchestration.jobs.SubmissionRecord.load(
    payload: dict
) -> nemo_gym.orchestration.jobs.SubmissionRecord

classmethod

Parse a record, refusing only one written by a NEWER Gym.

Older records stay readable: fields added since take their defaults, which the module docstring’s compatibility rule is there to guarantee. Refusing them instead would mean an upgrade silently orphaned every job already submitted.

A newer record is the one case that cannot be read safely — it may carry fields whose meaning this version does not know — so it fails, and says which way round the mismatch is.

nemo_gym.orchestration.jobs.SubmissionRecord.with_benchmark(
    job: nemo_gym.orchestration.jobs.BenchmarkJob
) -> nemo_gym.orchestration.jobs.SubmissionRecord

A copy of this record with the entry for job.benchmark replaced by job.

nemo_gym.orchestration.jobs.SubmissionRecord.write_local_index() -> pathlib.Path | None

Record the submission on this machine, or report why not and carry on.

Best-effort by design: this runs after the jobs are queued, so raising here would report a failure for work that is really running. The durable copy is the manifest in the run directory.

class nemo_gym.orchestration.jobs.SubmissionTelemetry()

Bases: BaseModel

Resolved submission-time telemetry controls, not credentials or a delivery claim.

enabled

StrictBool

model_config

= ConfigDict(extra='forbid')

remote_export

StrictBool

token_env

str

nemo_gym.orchestration.jobs.SubmissionTelemetry.validate_name(
    name: str
) -> str

classmethod

nemo_gym.orchestration.jobs._git(
    directory: pathlib.Path,
    args: str = ()
) -> subprocess.CompletedProcess

nemo_gym.orchestration.jobs._git_install_commit() -> str | None

The commit recorded in the distribution’s PEP 610 direct_url.json, or None.

nemo_gym.orchestration.jobs._source_checkout_commit(
    package_dir: pathlib.Path
) -> str | None

HEAD of the git checkout package_dir is tracked in, -dirty when it has uncommitted changes.

nemo_gym.orchestration.jobs._unknown_commit(
    reason: str,
    args: object = ()
) -> None

nemo_gym.orchestration.jobs.installed_gym_commit() -> str | None

Return the git commit of the nemo-gym that is running, or None when it cannot be determined.

A git install (pip install git+...) records its commit in the distribution metadata. Otherwise the running package’s own source tree is asked: a git checkout reports its HEAD, suffixed -dirty when it has uncommitted changes.

nemo_gym.orchestration.jobs.load_submission_record(
    gym_job_id: str
) -> nemo_gym.orchestration.jobs.SubmissionRecord

The record of submission gym_job_id: the run’s manifest when its run dir is on this machine, else this machine’s job index. Both are written by BaseExecutor.persist().

nemo_gym.orchestration.jobs.local_index_dir() -> pathlib.Path

Where this machine remembers its own submissions.

nemo_gym.orchestration.jobs.new_gym_job_id(
    now: datetime.datetime
) -> str

The submission’s primary key, and the run directory’s name.

The random suffix is what keeps two submits in the same second against the same job.output_path from sharing a directory. Sharing one is not merely untidy: an executor that stages by mirroring a directory will delete what it does not recognise (the Slurm executor copies with rsync --delete), so the second submit silently erases the first’s staged scripts.

nemo_gym.orchestration.jobs.restore_index_entry(
    gym_job_id: str,
    run_dir: str | pathlib.Path
) -> nemo_gym.orchestration.jobs.SubmissionRecord

Rebuild this machine’s job index entry for gym_job_id from the run directory’s own record.

A recycled machine loses its index while the run directory keeps the gym-job.json that persist() wrote there. That record is taken only as this submission’s and this directory’s: its gym_job_id is the one asked for and names the directory, its run_dir is the directory, and every benchmark’s job_dir lies inside it. An existing entry is kept as it is, with a warning when it names another run directory. Each restore is journaled in the index (INDEX_RESTORE_JOURNAL_NAME) with the source record’s SHA-256, the user and the host.

nemo_gym.orchestration.jobs.utc_now() -> datetime.datetime

The clock the run directory is named from.

Here rather than in an executor because run-directory identity is this module’s concern, and every executor needs the same answer. A seam: tests freeze it to prove two submits in the same second still get distinct directories.

nemo_gym.orchestration.jobs.utc_timestamp(
    now: datetime.datetime
) -> str

ISO 8601, seconds, explicit Z. A run directory is read on a cluster whose timezone need not match the submitter’s.

nemo_gym.orchestration.jobs.ARTIFACTS_DIRNAME = 'artifacts'

nemo_gym.orchestration.jobs.EVAL_EXIT_FILENAME = 'eval.exit'

nemo_gym.orchestration.jobs.EVAL_SUCCEEDED_FILENAME = 'eval.succeeded'

nemo_gym.orchestration.jobs.EXPORTS_DIRNAME = 'exports'

nemo_gym.orchestration.jobs.EXPORT_RESULT_SUFFIX = '.result.json'

nemo_gym.orchestration.jobs.INDEX_RESTORE_JOURNAL_NAME = 'index-restores.jsonl'

nemo_gym.orchestration.jobs.LOGS_DIRNAME = 'logs'

nemo_gym.orchestration.jobs.MANIFEST_NAME = 'gym-job.json'

nemo_gym.orchestration.jobs.RESOLVED_CONFIG_NAME = 'resolved-config.yaml'

nemo_gym.orchestration.jobs.ROLLOUTS_FILENAME = 'rollouts.jsonl'

nemo_gym.orchestration.jobs.SCHEMA_VERSION = 1

nemo_gym.orchestration.jobs._PACKAGE_DIR = Path(__file__).resolve().parents[1]

nemo_gym.orchestration.jobs.logger = logging.getLogger(__name__)