Skip to navigation

nemo_gym.orchestration.executors.capabilities

View as MarkdownOpen in Claude

The opt-in interfaces an executor adds to BaseExecutor.

BaseExecutor needs only run. Each feature beyond submitting is an interface here, and an executor lists the ones it supports in its bases:

class SlurmExecutor(BaseExecutor, Resubmittable, Terminable, LogReadable): …

Every method of a capability is abstract, so a capability cannot be half implemented; the only exceptions are the optional hooks, which carry a default and say so. Terminable and LogReadable extend Inspectable because both need query_states.

Module Contents

Classes

NameDescription
InspectableAn executor that can say what is happening to the jobs it queued.
LogReadableAn executor that can read the logs of the jobs it queued.
ResubmittableAn executor that can queue a benchmark of an existing submission again.
TerminableAn executor that can stop the jobs it queued.

Functions

NameDescription
requireexecutor, typed as capability, or UnsupportedOperationError naming what it cannot do.

Data

_Capability

API

class nemo_gym.orchestration.executors.capabilities.Inspectable()
Abstract

An executor that can say what is happening to the jobs it queued.

nemo_gym.orchestration.executors.capabilities.Inspectable.connection_key(
) -> str
abstract

Which connection serves record; records with the same key can be queried over one connection.

Records are grouped by this key so that asking about many submissions costs one connection per scheduler, not one per submission.

nemo_gym.orchestration.executors.capabilities.Inspectable.query_states(
abstract

What the scheduler says about each of refs: one ObservedStatus per JobKey.

Every ref shares one connection_key, so an implementation opens one connection and asks the scheduler once for all of them. The record locates a job and the scheduler decides: look the job up by where it runs, not by the recorded id, so a job started by hand or resubmitted from another machine is still seen. When several jobs belong to one benchmark, aggregate them into one status whose job_id is the one observed. state is the executor’s own mapping onto JobState. A scheduler that answers with an error yields UNKNOWN with the reason in detail; a connection that cannot be opened may raise, and the caller then reports UNKNOWN for every job of the group.

nemo_gym.orchestration.executors.capabilities.Inspectable.read_progress(

Optional. How far job’s run got, or None when this executor cannot tell.

nemo_gym.orchestration.executors.capabilities.Inspectable.read_run_files(
paths: list[pathlib.Path]
) -> dict[pathlib.Path, str | None]

Optional. The contents of files in the run directory, None for one that is missing or unreadable.

The default reads nothing, which is right for an executor that cannot reach the run directory.

class nemo_gym.orchestration.executors.capabilities.LogReadable()

Bases: Inspectable

An executor that can read the logs of the jobs it queued.

nemo_gym.orchestration.executors.capabilities.LogReadable.stream_logs(
tail: int | None,
follow: bool
) -> collections.abc.Iterator[str]
abstract

Yield the lines of source, a resolved LogSource whose location already names the job.

tail keeps only the last lines. follow keeps yielding as the log grows.

class nemo_gym.orchestration.executors.capabilities.Resubmittable()
Abstract

An executor that can queue a benchmark of an existing submission again.

Recovery also passes keyword options beyond resubmit’s two arguments (retry budget, walltime, repin and others) that only the Slurm executor accepts today; they are not part of this contract.

nemo_gym.orchestration.executors.capabilities.Resubmittable.resubmit(
benchmark: str
abstract

Queue benchmark of record again, in place, and return its updated entry.

Refuses while a job of that benchmark is still active, so two jobs never write one run. Persists the updated record before returning.

class nemo_gym.orchestration.executors.capabilities.Terminable()

Bases: Inspectable

An executor that can stop the jobs it queued.

nemo_gym.orchestration.executors.capabilities.Terminable.terminate(
) -> None
abstract

Stop job, and everything that would start it again (a queued follow-up, say).

Does only that: it neither checks the job’s state nor touches the record. Raises when the scheduler refuses, so the caller can report it for this benchmark and go on with the others.

nemo_gym.orchestration.executors.capabilities.require(
executor: object,
operation: str

executor, typed as capability, or UnsupportedOperationError naming what it cannot do.

nemo_gym.orchestration.executors.capabilities._Capability = TypeVar('_Capability')