> For clean Markdown of any page, append .md to the page URL. > For a complete documentation index, see https://docs.nvidia.com/nemo/gym/v0.4.0/nemo-gym/nemo_gym/orchestration/executors/slurm/llms.txt. # nemo_gym.orchestration.executors.slurm The Slurm executor for Pyxis-enabled clusters: `SlurmExecutor`. Its parts, each in its own module: * `executor`: render a run's job scripts, submit them with sbatch and record the submission; * `chain`: recovery of a run's auto-resume chain (`recovery_evidence`, `resubmit`), mixed into the executor; * `judge_job`: judge-only jobs that grade a run's saved answers again (`reverify_judge_failures`), mixed in too; * `reader`: the remote Store reader both of them read a run through; * `scheduler`: the Slurm commands they share; * `script`: render a benchmark's sbatch script, and `resume_script`: the auto-resume prologue and epilogue it includes for a resumable job; * `lustre_md_stats`: read the per-phase Lustre metadata counters a job records with `lustre_md_stats: true`. Only Slurm implements resubmit today, so these parts are Slurm code with no scheduler-agnostic interface between them; the executor capabilities give resubmit its seam when a second scheduler needs one. ## Submodules * **[`nemo_gym.orchestration.executors.slurm.chain`](/nemo/gym/nemo-gym/nemo_gym/orchestration/executors/slurm/chain)** * **[`nemo_gym.orchestration.executors.slurm.executor`](/nemo/gym/nemo-gym/nemo_gym/orchestration/executors/slurm/executor)** * **[`nemo_gym.orchestration.executors.slurm.judge_job`](/nemo/gym/nemo-gym/nemo_gym/orchestration/executors/slurm/judge_job)** * **[`nemo_gym.orchestration.executors.slurm.judge_profile`](/nemo/gym/nemo-gym/nemo_gym/orchestration/executors/slurm/judge_profile)** * **[`nemo_gym.orchestration.executors.slurm.lustre_md_stats`](/nemo/gym/nemo-gym/nemo_gym/orchestration/executors/slurm/lustre_md_stats)** * **[`nemo_gym.orchestration.executors.slurm.reader`](/nemo/gym/nemo-gym/nemo_gym/orchestration/executors/slurm/reader)** * **[`nemo_gym.orchestration.executors.slurm.resume_script`](/nemo/gym/nemo-gym/nemo_gym/orchestration/executors/slurm/resume_script)** * **[`nemo_gym.orchestration.executors.slurm.scheduler`](/nemo/gym/nemo-gym/nemo_gym/orchestration/executors/slurm/scheduler)** * **[`nemo_gym.orchestration.executors.slurm.script`](/nemo/gym/nemo-gym/nemo_gym/orchestration/executors/slurm/script)** ## Package Contents ### Classes | Name | Description | | --------------------------------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------- | | [`SlurmExecutor`](#nemo_gym-orchestration-executors-slurm-executor-SlurmExecutor) | Slurm executor for Pyxis-enabled clusters ([https://github.com/NVIDIA/pyxis](https://github.com/NVIDIA/pyxis)). | ### API ```python class nemo_gym.orchestration.executors.slurm.SlurmExecutor() ``` **Bases:** [ChainRecovery](/nemo/gym/nemo-gym/nemo_gym/orchestration/executors/slurm/chain#nemo_gym-orchestration-executors-slurm-chain-ChainRecovery), [JudgeJobs](/nemo/gym/nemo-gym/nemo_gym/orchestration/executors/slurm/judge_job#nemo_gym-orchestration-executors-slurm-judge_job-JudgeJobs), [BaseExecutor](/nemo/gym/nemo-gym/nemo_gym/orchestration/executors/base#nemo_gym-orchestration-executors-base-BaseExecutor), [Resubmittable](/nemo/gym/nemo-gym/nemo_gym/orchestration/executors/capabilities#nemo_gym-orchestration-executors-capabilities-Resubmittable), [Terminable](/nemo/gym/nemo-gym/nemo_gym/orchestration/executors/capabilities#nemo_gym-orchestration-executors-capabilities-Terminable), [LogReadable](/nemo/gym/nemo-gym/nemo_gym/orchestration/executors/capabilities#nemo_gym-orchestration-executors-capabilities-LogReadable), [Inspectable](/nemo/gym/nemo-gym/nemo_gym/orchestration/executors/capabilities#nemo_gym-orchestration-executors-capabilities-Inspectable) Slurm executor for Pyxis-enabled clusters ([https://github.com/NVIDIA/pyxis](https://github.com/NVIDIA/pyxis)). Every service and the driver are launched via `srun --container-image` so they run inside the container specified in their config. Health checks run as plain bash inside the sbatch script (no container needed — they just poll HTTP). **`name`** `= 'slurm'` --- ```python nemo_gym.orchestration.executors.slurm.SlurmExecutor._build_record( cluster: str, compute: nemo_gym.orchestration.api.SlurmComputeConfig, gym_job_id: str, now: datetime.datetime, remote_run_dir: pathlib.Path, benchmark_names: list[str], output: str, log_sources: list[nemo_gym.orchestration.jobs.LogSource] ) -> nemo_gym.orchestration.jobs.SubmissionRecord ``` ```python nemo_gym.orchestration.executors.slurm.SlurmExecutor._dry_run( config: nemo_gym.orchestration.api.SubmitConfig, compute: nemo_gym.orchestration.api.SlurmComputeConfig, remote_run_dir: pathlib.Path ) -> None ``` ```python nemo_gym.orchestration.executors.slurm.SlurmExecutor._saved_config( conn: nemo_gym.orchestration.executors.connection.Connection, run_dir: pathlib.Path ) -> nemo_gym.orchestration.api.SubmitConfig ``` staticmethod The resolved config `persist()` saved for this run, so a resubmit repeats what submit did. ```python nemo_gym.orchestration.executors.slurm.SlurmExecutor._stage( config: nemo_gym.orchestration.api.SubmitConfig, compute: nemo_gym.orchestration.api.SlurmComputeConfig, remote_run_dir: pathlib.Path, staging: pathlib.Path ) -> pathlib.Path ``` ```python nemo_gym.orchestration.executors.slurm.SlurmExecutor.connection_key( record: nemo_gym.orchestration.jobs.SubmissionRecord ) -> str ``` ```python nemo_gym.orchestration.executors.slurm.SlurmExecutor.export_tracking( record: nemo_gym.orchestration.jobs.SubmissionRecord, benchmark: str, evidence: nemo_gym.eval.recovery.evidence.CollectedEvidence | None ) -> dict[str, typing.Any] ``` Run `gym eval export mlflow` on the run's saved artifacts from the login host, and report its result. The export runs where the remote Store reader runs, with the run's saved pinned Gym install (and its MLflow extra) in a temporary checkout: it only reads the saved artifacts and talks to the tracking server, so it needs no allocation. It takes the benchmark's saved `mlflow_*` and `upload_rollouts` run settings, and its credentials as the driver did: the submission's forwarded environment and the driver's env. Gym's export takes the run lock, prepares the health metrics and writes the receipt bound to the artifact digests, and skips a run that already has one. The resubmit lock is held and no job of the chain may be live meanwhile. ```python nemo_gym.orchestration.executors.slurm.SlurmExecutor.gym_pin( record: nemo_gym.orchestration.jobs.SubmissionRecord ) -> str | None ``` The full Gym commit the run's saved config pins its driver to; None when there is none or it is unreadable. ```python nemo_gym.orchestration.executors.slurm.SlurmExecutor.installed_commit() -> str | None ``` The Gym commit this launcher runs; the same value the judge-only job and a repin are checked against. ```python nemo_gym.orchestration.executors.slurm.SlurmExecutor.query_states( refs: list[nemo_gym.orchestration.jobs.JobRef] ) -> dict[nemo_gym.orchestration.jobs.JobKey, nemo_gym.orchestration.jobs.ObservedStatus] ``` One script on one connection: a single squeue, a single sacct and each job directory's state files. ```python nemo_gym.orchestration.executors.slurm.SlurmExecutor.read_run_files( record: nemo_gym.orchestration.jobs.SubmissionRecord, paths: list[pathlib.Path] ) -> dict[pathlib.Path, str | None] ``` ```python nemo_gym.orchestration.executors.slurm.SlurmExecutor.run( config: nemo_gym.orchestration.api.SubmitConfig, dry_run: bool = False ) -> nemo_gym.orchestration.jobs.SubmissionRecord | None ``` ```python nemo_gym.orchestration.executors.slurm.SlurmExecutor.stream_logs( record: nemo_gym.orchestration.jobs.SubmissionRecord, job: nemo_gym.orchestration.jobs.BenchmarkJob, source: nemo_gym.orchestration.jobs.LogSource, tail: int | None, follow: bool ) -> collections.abc.Iterator[str] ``` The lines of `source`, read in one call; the location is relative to the job directory. ```python nemo_gym.orchestration.executors.slurm.SlurmExecutor.terminate( record: nemo_gym.orchestration.jobs.SubmissionRecord, job: nemo_gym.orchestration.jobs.BenchmarkJob ) -> None ``` Cancel everything of `job` that is queued or running, the pending jobs first, in one script. ## Docs - [nemo_gym.orchestration.executors.slurm.chain](https://docs.nvidia.com/nemo/gym/v0.4.0/nemo-gym/nemo_gym/orchestration/executors/slurm/chain.md) - [nemo_gym.orchestration.executors.slurm.executor](https://docs.nvidia.com/nemo/gym/v0.4.0/nemo-gym/nemo_gym/orchestration/executors/slurm/executor.md) - [nemo_gym.orchestration.executors.slurm.judge_job](https://docs.nvidia.com/nemo/gym/v0.4.0/nemo-gym/nemo_gym/orchestration/executors/slurm/judge_job.md) - [nemo_gym.orchestration.executors.slurm.judge_profile](https://docs.nvidia.com/nemo/gym/v0.4.0/nemo-gym/nemo_gym/orchestration/executors/slurm/judge_profile.md) - [nemo_gym.orchestration.executors.slurm.lustre_md_stats](https://docs.nvidia.com/nemo/gym/v0.4.0/nemo-gym/nemo_gym/orchestration/executors/slurm/lustre_md_stats.md) - [nemo_gym.orchestration.executors.slurm.reader](https://docs.nvidia.com/nemo/gym/v0.4.0/nemo-gym/nemo_gym/orchestration/executors/slurm/reader.md) - [nemo_gym.orchestration.executors.slurm.resume_script](https://docs.nvidia.com/nemo/gym/v0.4.0/nemo-gym/nemo_gym/orchestration/executors/slurm/resume_script.md) - [nemo_gym.orchestration.executors.slurm.scheduler](https://docs.nvidia.com/nemo/gym/v0.4.0/nemo-gym/nemo_gym/orchestration/executors/slurm/scheduler.md) - [nemo_gym.orchestration.executors.slurm.script](https://docs.nvidia.com/nemo/gym/v0.4.0/nemo-gym/nemo_gym/orchestration/executors/slurm/script.md)