Skip to navigation

nemo_gym.orchestration.executors.slurm.executor

View as MarkdownOpen in Claude

SlurmExecutor: render a run’s job scripts, submit them with sbatch and record the submission.

Its recovery of a run’s chain (chain) and its judge-only jobs (judge_job) are mixed in.

Module Contents

Classes

NameDescription
SlurmExecutorSlurm executor for Pyxis-enabled clusters (https://github.com/NVIDIA/pyxis).

Functions

NameDescription
_validate_benchmark_namesFail before anything is staged or copied.
_validate_mounts-

Data

_EXPORT_TIMEOUT_S

API

class nemo_gym.orchestration.executors.slurm.SlurmExecutor()

Bases: ChainRecovery, JudgeJobs, BaseExecutor, Resubmittable, Terminable, LogReadable, Inspectable

Slurm executor for Pyxis-enabled clusters (https://github.com/NVIDIA/pyxis).

Every service and the driver are launched via srun --container-image so they run inside the container specified in their config. Health checks run as plain bash inside the sbatch script (no container needed — they just poll HTTP).

name
= 'slurm'
nemo_gym.orchestration.executors.slurm.SlurmExecutor._build_record(
cluster: str,
gym_job_id: str,
now: datetime.datetime,
remote_run_dir: pathlib.Path,
benchmark_names: list[str],
output: str,
nemo_gym.orchestration.executors.slurm.SlurmExecutor._dry_run(
remote_run_dir: pathlib.Path
) -> None
nemo_gym.orchestration.executors.slurm.SlurmExecutor._saved_config(
run_dir: pathlib.Path
staticmethod

The resolved config persist() saved for this run, so a resubmit repeats what submit did.

nemo_gym.orchestration.executors.slurm.SlurmExecutor._stage(
remote_run_dir: pathlib.Path,
staging: pathlib.Path
) -> pathlib.Path
nemo_gym.orchestration.executors.slurm.SlurmExecutor.connection_key(
) -> str
nemo_gym.orchestration.executors.slurm.SlurmExecutor.export_tracking(
benchmark: str,
) -> dict[str, typing.Any]

Run gym eval export mlflow on the run’s saved artifacts from the login host, and report its result.

The export runs where the remote Store reader runs, with the run’s saved pinned Gym install (and its MLflow extra) in a temporary checkout: it only reads the saved artifacts and talks to the tracking server, so it needs no allocation. It takes the benchmark’s saved mlflow_* and upload_rollouts run settings, and its credentials as the driver did: the submission’s forwarded environment and the driver’s env. Gym’s export takes the run lock, prepares the health metrics and writes the receipt bound to the artifact digests, and skips a run that already has one. The resubmit lock is held and no job of the chain may be live meanwhile.

nemo_gym.orchestration.executors.slurm.SlurmExecutor.gym_pin(
) -> str | None

The full Gym commit the run’s saved config pins its driver to; None when there is none or it is unreadable.

nemo_gym.orchestration.executors.slurm.SlurmExecutor.installed_commit() -> str | None

The Gym commit this launcher runs; the same value the judge-only job and a repin are checked against.

nemo_gym.orchestration.executors.slurm.SlurmExecutor.query_states(

One script on one connection: a single squeue, a single sacct and each job directory’s state files.

nemo_gym.orchestration.executors.slurm.SlurmExecutor.read_run_files(
paths: list[pathlib.Path]
) -> dict[pathlib.Path, str | None]
nemo_gym.orchestration.executors.slurm.SlurmExecutor.run(
dry_run: bool = False
nemo_gym.orchestration.executors.slurm.SlurmExecutor.stream_logs(
tail: int | None,
follow: bool
) -> collections.abc.Iterator[str]

The lines of source, read in one call; the location is relative to the job directory.

nemo_gym.orchestration.executors.slurm.SlurmExecutor.terminate(
) -> None

Cancel everything of job that is queued or running, the pending jobs first, in one script.

nemo_gym.orchestration.executors.slurm.executor._validate_benchmark_names(
benchmarks: list[str]
) -> None

Fail before anything is staged or copied.

_sbatch_command raises this same check, but only once submission is already underway — after staging, connecting, mount validation and the rsync. Checking here first means a bad name fails before any of that, and fails on --dry-run too, instead of the dry run printing a clean script listing for a benchmark that could never actually be submitted.

nemo_gym.orchestration.executors.slurm.executor._validate_mounts(
) -> None
nemo_gym.orchestration.executors.slurm.executor._EXPORT_TIMEOUT_S = 3600