Skip to navigation

nemo_gym.orchestration.jobs

View as MarkdownOpen in Claude

The record a submission leaves behind.

api.py is the input schema (what to submit); this is the output schema (what was submitted).

This module is executor-agnostic on purpose, and that is a rule rather than a coincidence: it must describe a submission made by ANY executor — Slurm today, k8s or a local runner later. Two consequences, both enforced by test_jobs_module_is_executor_agnostic:

  • Nothing here imports from executors/, so a reader — today EFB’s collect, gym eval status — can load a record without pulling Slurm, SSH and the sbatch templates into its import path.
  • No field is typed or named after one executor’s vocabulary. Where a docstring explains a value by example, it says which executor the example comes from.

Compatibility. A record outlives the Gym that wrote it, so schema_version is read in one direction only: a reader accepts anything at or below its own version and refuses only what is newer than it understands. Upgrading nemo-gym therefore never strands records already on disk.

That is only sound if the schema keeps its side of the bargain, so within one schema version a field may only be ADDED WITH A DEFAULT, or removed. A field must never be repurposed or have its meaning changed — an older record would then be read as if it meant the new thing. Anything of that kind needs a SCHEMA_VERSION bump and an explicit migration, not a silent reinterpretation.

Module Contents

Classes

NameDescription
BenchmarkInfoWhere one benchmark’s job lives and what it writes: its directory, the job ids observed for it
BenchmarkJobOne benchmark’s submission.
FailureWhy a benchmark’s job failed, with where the reason came from.
FailureCategoryWhich kind of thing went wrong, so a reader knows how far to trust a Failure.
JobInfoA description of one submission and where its parts are, with no live state.
JobRefA record paired with one of its benchmark jobs, so one call can span many submissions.
JobStateWhat is happening to one benchmark’s job, in terms every executor shares.
JobStatusThe status of one benchmark’s job as shown to a user.
LogSourceOne log a benchmark’s job writes, as the executor planned it at submit time.
ObservedStatusWhat an executor saw of one benchmark’s job: the primitive behind every status.
ProgressHow far a run got, in counts only: rollouts done of total, and how many failed.
SubmissionEnvironmentNames observed at submission, never values.
SubmissionRecordEverything needed to find a submitted run again.
SubmissionStatusThe status of every selected benchmark of one submission.
SubmissionTelemetryResolved submission-time telemetry controls, not credentials or a delivery claim.

Functions

NameDescription
benchmark_artifact_dirsThe artifacts directory of each benchmark of the submission target names, or None when no record is found.
list_submissionsThe submissions in this machine’s job index, newest first, at most limit of them.
load_submission_recordThe record of submission gym_job_id: the run’s manifest when its run dir is on this machine, else
local_index_dirWhere this machine remembers its own submissions.
new_gym_job_idThe submission’s primary key, and the run directory’s name.
restore_index_entryRebuild this machine’s job index entry for gym_job_id from the run directory’s own record.
utc_nowThe clock the run directory is named from.
utc_timestampISO 8601, seconds, explicit Z. A run directory is read on a cluster whose

Data

ARTIFACTS_DIRNAME

CHAIN_INCOMPLETE_FILE

EVAL_EXIT_FILE

EVAL_EXIT_FILENAME

EVAL_SUCCEEDED_FILENAME

EXPORTS_DIRNAME

EXPORT_RESULT_SUFFIX

FINISHED_STATES

INDEX_RESTORE_JOURNAL_NAME

JobKey

LOGS_DIRNAME

MANIFEST_NAME

RESOLVED_CONFIG_NAME

ROLLOUTS_FILENAME

SCHEMA_VERSION

__getattr__

logger

API

class nemo_gym.orchestration.jobs.BenchmarkInfo()

Bases: BaseModel

Where one benchmark’s job lives and what it writes: its directory, the job ids observed for it (the whole chain when a run was resumed), and the logs it was set up to write.

benchmark
str
job_dir
str
job_ids
list[str] = []
log_sources
list[LogSource] = []
class nemo_gym.orchestration.jobs.BenchmarkJob()

Bases: BaseModel

One benchmark’s submission.

job_id is None exactly when the executor failed to enqueue this benchmark (for the Slurm executor, a failed sbatch), in which case error says why. The other benchmarks in the same submission are unaffected — one failure does not discard the record for the ones that did start.

log_sources is the list of logs the job was set up to write, computed at submit time by the same code that renders the job’s log flags, so the plan and the job cannot disagree. A plan is intent, not proof that a file exists. A record written before this field existed has none.

benchmark
str
error
str | None = None
gym_commit
str | None = None
job_dir
str
job_id
str | None = None
log_sources
list[LogSource] = []
walltime
str | None = None
class nemo_gym.orchestration.jobs.Failure()

Bases: BaseModel

Why a benchmark’s job failed, with where the reason came from.

source says what produced message (the scheduler, the record, a file the job wrote), and logs is a command that shows the job’s logs. The logs themselves are never parsed to guess a cause.

category
FailureCategory
exit_code
int | None = None
logs
str | None = None
message
str
source
str
class nemo_gym.orchestration.jobs.FailureCategory

Bases: enum.Enum

Which kind of thing went wrong, so a reader knows how far to trust a Failure.

CANCELLED
= 'CANCELLED'
ENQUEUE
= 'ENQUEUE'
EXPORT
= 'EXPORT'
INCOMPLETE
= 'INCOMPLETE'
RUN
= 'RUN'
SCHEDULER
= 'SCHEDULER'
class nemo_gym.orchestration.jobs.JobInfo()

Bases: BaseModel

A description of one submission and where its parts are, with no live state.

config is the resolved config the submission was made with, or None when the executor cannot read it, in which case config_unavailable says why.

benchmarks
list[BenchmarkInfo]
config
dict[str, object] | None = None
config_unavailable
str | None = None
record
SubmissionRecord
class nemo_gym.orchestration.jobs.JobRef()

Bases: NamedTuple

A record paired with one of its benchmark jobs, so one call can span many submissions.

job
BenchmarkJob
key
JobKey
record
SubmissionRecord
class nemo_gym.orchestration.jobs.JobState

Bases: enum.Enum

What is happening to one benchmark’s job, in terms every executor shares.

Each executor maps its own scheduler’s vocabulary onto this; the native string survives only as a status’s detail. UNKNOWN means the executor could not tell (a scheduler that is unreachable, or no longer remembers the job).

CANCELLED
= 'CANCELLED'
FAILED
= 'FAILED'
QUEUED
= 'QUEUED'
RUNNING
= 'RUNNING'
SUCCEEDED
= 'SUCCEEDED'
UNKNOWN
= 'UNKNOWN'
class nemo_gym.orchestration.jobs.JobStatus()

Bases: BaseModel

The status of one benchmark’s job as shown to a user.

Built from an ObservedStatus and the files the job wrote. progress is None until an executor reports it.

benchmark
str
detail
str | None = None
exit_code
int | None = None
failure
Failure | None = None
job_id
str | None = None
progress
Progress | None = None
state
JobState
class nemo_gym.orchestration.jobs.LogSource()

Bases: BaseModel

One log a benchmark’s job writes, as the executor planned it at submit time.

kind is driver for the process that runs the benchmark and service for every other step of the job. instance names one instance of a service that runs several; it is None for a source that is not split by instance. location is where the log is, in the executor’s terms and opaque to everyone else: for the Slurm executor a path relative to the job directory with a {job_id} placeholder, because the scheduler’s id is not known at submit time and changes when a job is resumed; for a container-based executor, a container name. A reader fills in the placeholder with the id of the job it wants.

instance
str | None = None
kind
Literal['driver', 'service']
location
str
name
str
class nemo_gym.orchestration.jobs.ObservedStatus()

Bases: BaseModel

What an executor saw of one benchmark’s job: the primitive behind every status.

state is set by the executor from its own vocabulary, and detail keeps the native string (a Slurm TIMEOUT, say). job_id is the id actually observed, which can differ from the record’s when a job was resumed or started outside Gym, and job_ids is every job id seen for the benchmark, oldest first (the whole chain when a run was resumed). files holds the state files the job wrote into its directory, by name, read in the same call as the scheduler’s answer; a file that is missing or unreadable is absent.

detail
str | None = None
exit_code
int | None = None
files
dict[str, str] = {}
job_id
str | None = None
job_ids
list[str] = []
reason
str | None = None
state
JobState
class nemo_gym.orchestration.jobs.Progress()

Bases: BaseModel

How far a run got, in counts only: rollouts done of total, and how many failed.

done
int
failed
int
total
int
class nemo_gym.orchestration.jobs.SubmissionEnvironment()

Bases: BaseModel

Names observed at submission, never values.

required_env names submit-shell inputs (host: references and required forwarding). forwarded_env names variables actually exported by the submission transport. A runtime: alias is not a submit-shell requirement unless explicitly forwarded. None on the containing record means this observation was not recorded.

forwarded_env
list[str]
model_config
= ConfigDict(extra='forbid')
required_env
list[str]
telemetry
SubmissionTelemetry
nemo_gym.orchestration.jobs.SubmissionEnvironment.validate_names(
names: list[str]
) -> list[str]
classmethod
class nemo_gym.orchestration.jobs.SubmissionRecord()

Bases: BaseModel

Everything needed to find a submitted run again.

executor names the executor that produced this record, and is the field a reader branches on before interpreting the executor-shaped parts of the rest — job_id, for instance, is a Slurm job ID under the Slurm executor and need not be numeric under another. It is deliberately a plain str rather than an enum of the executors that happen to exist today, so that adding one does not require a schema version bump on every reader.

hostname None means the submission was made from the machine that runs the workload manager’s client directly, rather than reaching it over SSH — not that the host is unknown. Executors with no remote-submission concept at all leave it None.

gym_commit is the commit of the Gym that wrote the record, when its install records one (see nemo_gym.utils.gym_commit.installed_gym_commit). A -dirty suffix marks a checkout with uncommitted changes: the commit alone does not reproduce that Gym.

benchmarks
list[BenchmarkJob]
cluster
str
environment
SubmissionEnvironment | None
executor
str
executor_metadata
dict[str, str] = {}
failed
list[BenchmarkJob]
gym_commit
str | None = None
gym_job_id
str
gym_version
str
hostname
str | None = None
provenance
dict[str, JsonValue] = Field(default_factory=dict)
run_dir
str
schema_version
int = SCHEMA_VERSION
submitted_at
str
submitted_by
str
nemo_gym.orchestration.jobs.SubmissionRecord.benchmark_job(
benchmark: str

This record’s entry for benchmark; an unknown name is an error that lists the known ones.

nemo_gym.orchestration.jobs.SubmissionRecord.dumps() -> str

The manifest’s on-disk bytes; one spelling, so every store matches.

classmethod

Parse a record, refusing only one written by a NEWER Gym.

Older records stay readable: fields added since take their defaults, which the module docstring’s compatibility rule is there to guarantee. Refusing them instead would mean an upgrade silently orphaned every job already submitted.

A newer record is the one case that cannot be read safely — it may carry fields whose meaning this version does not know — so it fails, and says which way round the mismatch is.

A copy of this record with the entry for job.benchmark replaced by job.

nemo_gym.orchestration.jobs.SubmissionRecord.write_local_index() -> pathlib.Path | None

Record the submission on this machine, or report why not and carry on.

Best-effort by design: this runs after the jobs are queued, so raising here would report a failure for work that is really running. The durable copy is the manifest in the run directory.

class nemo_gym.orchestration.jobs.SubmissionStatus()

Bases: BaseModel

The status of every selected benchmark of one submission.

state rolls the benchmarks up into one value and counts says how many are in each state, so a mixed result is not hidden behind one word. Both describe only the benchmarks in jobs.

counts
dict[JobState, int]
jobs
list[JobStatus]
record
SubmissionRecord
state
JobState
class nemo_gym.orchestration.jobs.SubmissionTelemetry()

Bases: BaseModel

Resolved submission-time telemetry controls, not credentials or a delivery claim.

enabled
StrictBool
model_config
= ConfigDict(extra='forbid')
remote_export
StrictBool
token_env
str
nemo_gym.orchestration.jobs.SubmissionTelemetry.validate_name(
name: str
) -> str
classmethod
nemo_gym.orchestration.jobs.benchmark_artifact_dirs(
target: str
) -> dict[str, pathlib.Path] | None

The artifacts directory of each benchmark of the submission target names, or None when no record is found.

target is a run directory holding the submission’s manifest, or a Gym job ID in this machine’s job index.

nemo_gym.orchestration.jobs.list_submissions(
limit: int | None = None

The submissions in this machine’s job index, newest first, at most limit of them.

Reads the index only, so it needs no executor and no network. A file that cannot be read as a record is skipped with a warning: one bad file must not hide the rest.

nemo_gym.orchestration.jobs.load_submission_record(
gym_job_id: str

The record of submission gym_job_id: the run’s manifest when its run dir is on this machine, else this machine’s job index. Both are written by BaseExecutor.persist().

nemo_gym.orchestration.jobs.local_index_dir() -> pathlib.Path

Where this machine remembers its own submissions.

nemo_gym.orchestration.jobs.new_gym_job_id(
now: datetime.datetime
) -> str

The submission’s primary key, and the run directory’s name.

The random suffix is what keeps two submits in the same second against the same job.output_path from sharing a directory. Sharing one is not merely untidy: an executor that stages by mirroring a directory will delete what it does not recognise (the Slurm executor copies with rsync --delete), so the second submit silently erases the first’s staged scripts.

nemo_gym.orchestration.jobs.restore_index_entry(
gym_job_id: str,
run_dir: str | pathlib.Path

Rebuild this machine’s job index entry for gym_job_id from the run directory’s own record.

A recycled machine loses its index while the run directory keeps the gym-job.json that persist() wrote there. That record is taken only as this submission’s and this directory’s: its gym_job_id is the one asked for and names the directory, its run_dir is the directory, and every benchmark’s job_dir lies inside it. An existing entry is kept as it is, with a warning when it names another run directory. Each restore is journaled in the index (INDEX_RESTORE_JOURNAL_NAME) with the source record’s SHA-256, the user and the host.

nemo_gym.orchestration.jobs.utc_now() -> datetime.datetime

The clock the run directory is named from.

Here rather than in an executor because run-directory identity is this module’s concern, and every executor needs the same answer. A seam: tests freeze it to prove two submits in the same second still get distinct directories.

nemo_gym.orchestration.jobs.utc_timestamp(
now: datetime.datetime
) -> str

ISO 8601, seconds, explicit Z. A run directory is read on a cluster whose timezone need not match the submitter’s.

nemo_gym.orchestration.jobs.ARTIFACTS_DIRNAME = 'artifacts'
nemo_gym.orchestration.jobs.CHAIN_INCOMPLETE_FILE = '.gym_chain_incomplete'
nemo_gym.orchestration.jobs.EVAL_EXIT_FILE = 'eval.exit'
nemo_gym.orchestration.jobs.EVAL_EXIT_FILENAME = 'eval.exit'
nemo_gym.orchestration.jobs.EVAL_SUCCEEDED_FILENAME = 'eval.succeeded'
nemo_gym.orchestration.jobs.EXPORTS_DIRNAME = 'exports'
nemo_gym.orchestration.jobs.EXPORT_RESULT_SUFFIX = '.result.json'
nemo_gym.orchestration.jobs.FINISHED_STATES = frozenset({JobState.SUCCEEDED, JobState.FAILED, JobState.CANCELLED})
nemo_gym.orchestration.jobs.INDEX_RESTORE_JOURNAL_NAME = 'index-restores.jsonl'
nemo_gym.orchestration.jobs.JobKey = tuple[str, str]
nemo_gym.orchestration.jobs.LOGS_DIRNAME = 'logs'
nemo_gym.orchestration.jobs.MANIFEST_NAME = 'gym-job.json'
nemo_gym.orchestration.jobs.RESOLVED_CONFIG_NAME = 'resolved-config.yaml'
nemo_gym.orchestration.jobs.ROLLOUTS_FILENAME = 'rollouts.jsonl'
nemo_gym.orchestration.jobs.SCHEMA_VERSION = 1
nemo_gym.orchestration.jobs.__getattr__ = moved_names_getattr(__name__)
nemo_gym.orchestration.jobs.logger = logging.getLogger(__name__)