nemo_gym.orchestration.executors.capabilities
nemo_gym.orchestration.executors.capabilities
The opt-in interfaces an executor adds to BaseExecutor.
BaseExecutor needs only run. Each feature beyond submitting is an interface here, and an executor
lists the ones it supports in its bases:
class SlurmExecutor(BaseExecutor, Resubmittable, Terminable, LogReadable): …
Every method of a capability is abstract, so a capability cannot be half implemented; the only
exceptions are the optional hooks, which carry a default and say so. Terminable and LogReadable
extend Inspectable because both need query_states.
Module Contents
Classes
Functions
Data
API
An executor that can say what is happening to the jobs it queued.
Which connection serves record; records with the same key can be queried over one connection.
Records are grouped by this key so that asking about many submissions costs one connection per scheduler, not one per submission.
What the scheduler says about each of refs: one ObservedStatus per JobKey.
Every ref shares one connection_key, so an implementation opens one connection and asks the
scheduler once for all of them. The record locates a job and the scheduler decides: look the job
up by where it runs, not by the recorded id, so a job started by hand or resubmitted from another
machine is still seen. When several jobs belong to one benchmark, aggregate them into one status
whose job_id is the one observed. state is the executor’s own mapping onto JobState. A scheduler
that answers with an error yields UNKNOWN with the reason in detail; a connection that cannot be
opened may raise, and the caller then reports UNKNOWN for every job of the group.
Optional. How far job’s run got, or None when this executor cannot tell.
Optional. The contents of files in the run directory, None for one that is missing or unreadable.
The default reads nothing, which is right for an executor that cannot reach the run directory.
Bases: Inspectable
An executor that can read the logs of the jobs it queued.
Yield the lines of source, a resolved LogSource whose location already names the job.
tail keeps only the last lines. follow keeps yielding as the log grows.
An executor that can queue a benchmark of an existing submission again.
Recovery also passes keyword options beyond resubmit’s two arguments (retry budget, walltime, repin and others) that
only the Slurm executor accepts today; they are not part of this contract.
Queue benchmark of record again, in place, and return its updated entry.
Refuses while a job of that benchmark is still active, so two jobs never write one run. Persists the updated record before returning.
Bases: Inspectable
An executor that can stop the jobs it queued.
Stop job, and everything that would start it again (a queued follow-up, say).
Does only that: it neither checks the job’s state nor touches the record. Raises when the scheduler refuses, so the caller can report it for this benchmark and go on with the others.
executor, typed as capability, or UnsupportedOperationError naming what it cannot do.