> This page is for version 0.7.0.
> For other versions, use one of these documentation indexes:
> - Main (default): https://docs.nvidia.com/nemo/gym/main/llms.txt
> - 0.7.0: https://docs.nvidia.com/nemo/gym/v0.7.0/llms.txt
> - 0.6.0: https://docs.nvidia.com/nemo/gym/v0.6.0/llms.txt
> - 0.5.1: https://docs.nvidia.com/nemo/gym/v0.5.1/llms.txt
> - 0.5.0: https://docs.nvidia.com/nemo/gym/v0.5.0/llms.txt
> - 0.4.0: https://docs.nvidia.com/nemo/gym/v0.4.0/llms.txt
> - 0.3.0: https://docs.nvidia.com/nemo/gym/v0.3.0/llms.txt
> - 0.2.1: https://docs.nvidia.com/nemo/gym/v0.2.1/llms.txt

> For clean Markdown of any page, append .md to the page URL.
> For a complete documentation index, see https://docs.nvidia.com/nemo/gym/llms.txt.

# nemo_gym.orchestration.executors.slurm.executor

`SlurmExecutor`: render a run's job scripts, submit them with sbatch and record the submission.

Its recovery of a run's chain (`chain`) and its judge-only jobs (`judge_job`) are mixed in.

## Module Contents

### Classes

| Name                                                                              | Description                                                                                                     |
| --------------------------------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------- |
| [`SlurmExecutor`](#nemo_gym-orchestration-executors-slurm-executor-SlurmExecutor) | Slurm executor for Pyxis-enabled clusters ([https://github.com/NVIDIA/pyxis](https://github.com/NVIDIA/pyxis)). |

### Functions

| Name                                                                                                      | Description                               |
| --------------------------------------------------------------------------------------------------------- | ----------------------------------------- |
| [`_validate_benchmark_names`](#nemo_gym-orchestration-executors-slurm-executor-_validate_benchmark_names) | Fail before anything is staged or copied. |
| [`_validate_mounts`](#nemo_gym-orchestration-executors-slurm-executor-_validate_mounts)                   | -                                         |

### Data

[`_EXPORT_TIMEOUT_S`](#nemo_gym-orchestration-executors-slurm-executor-_EXPORT_TIMEOUT_S)

### API

```python
class nemo_gym.orchestration.executors.slurm.SlurmExecutor()
```

**Bases:** [ChainRecovery](/nemo/gym/nemo-gym/nemo_gym/orchestration/executors/slurm/chain#nemo_gym-orchestration-executors-slurm-chain-ChainRecovery), [JudgeJobs](/nemo/gym/nemo-gym/nemo_gym/orchestration/executors/slurm/judge_job#nemo_gym-orchestration-executors-slurm-judge_job-JudgeJobs), [BaseExecutor](/nemo/gym/nemo-gym/nemo_gym/orchestration/executors/base#nemo_gym-orchestration-executors-base-BaseExecutor), [Resubmittable](/nemo/gym/nemo-gym/nemo_gym/orchestration/executors/capabilities#nemo_gym-orchestration-executors-capabilities-Resubmittable), [Terminable](/nemo/gym/nemo-gym/nemo_gym/orchestration/executors/capabilities#nemo_gym-orchestration-executors-capabilities-Terminable), [LogReadable](/nemo/gym/nemo-gym/nemo_gym/orchestration/executors/capabilities#nemo_gym-orchestration-executors-capabilities-LogReadable), [Inspectable](/nemo/gym/nemo-gym/nemo_gym/orchestration/executors/capabilities#nemo_gym-orchestration-executors-capabilities-Inspectable)

Slurm executor for Pyxis-enabled clusters ([https://github.com/NVIDIA/pyxis](https://github.com/NVIDIA/pyxis)).

Every service and the driver are launched via `srun --container-image` so they
run inside the container specified in their config. Health checks run as plain
bash inside the sbatch script (no container needed — they just poll HTTP).

**`name`** `= 'slurm'`

---

```python
nemo_gym.orchestration.executors.slurm.SlurmExecutor._build_record(
    cluster: str,
    compute: nemo_gym.orchestration.api.SlurmComputeConfig,
    gym_job_id: str,
    now: datetime.datetime,
    remote_run_dir: pathlib.Path,
    benchmark_names: list[str],
    output: str,
    log_sources: list[nemo_gym.orchestration.jobs.LogSource]
) -> nemo_gym.orchestration.jobs.SubmissionRecord
```

```python
nemo_gym.orchestration.executors.slurm.SlurmExecutor._dry_run(
    config: nemo_gym.orchestration.api.SubmitConfig,
    compute: nemo_gym.orchestration.api.SlurmComputeConfig,
    remote_run_dir: pathlib.Path
) -> None
```

```python
nemo_gym.orchestration.executors.slurm.SlurmExecutor._saved_config(
    conn: nemo_gym.orchestration.executors.connection.Connection,
    run_dir: pathlib.Path
) -> nemo_gym.orchestration.api.SubmitConfig
```

staticmethod

The resolved config `persist()` saved for this run, so a resubmit repeats what submit did.

```python
nemo_gym.orchestration.executors.slurm.SlurmExecutor._stage(
    config: nemo_gym.orchestration.api.SubmitConfig,
    compute: nemo_gym.orchestration.api.SlurmComputeConfig,
    remote_run_dir: pathlib.Path,
    staging: pathlib.Path
) -> pathlib.Path
```

```python
nemo_gym.orchestration.executors.slurm.SlurmExecutor.connection_key(
    record: nemo_gym.orchestration.jobs.SubmissionRecord
) -> str
```

```python
nemo_gym.orchestration.executors.slurm.SlurmExecutor.export_tracking(
    record: nemo_gym.orchestration.jobs.SubmissionRecord,
    benchmark: str,
    evidence: nemo_gym.eval.recovery.evidence.CollectedEvidence | None
) -> dict[str, typing.Any]
```

Run `gym eval export mlflow` on the run's saved artifacts from the login host, and report its result.

The export runs where the remote Store reader runs, with the run's saved pinned Gym install (and its MLflow
extra) in a temporary checkout: it only reads the saved artifacts and talks to the tracking server, so it needs
no allocation. It takes the benchmark's saved `mlflow_*` and `upload_rollouts` run settings, and its
credentials as the driver did: the submission's forwarded environment and the driver's env. Gym's export takes
the run lock, prepares the health metrics and writes the receipt bound to the artifact digests, and skips a run
that already has one. The resubmit lock is held and no job of the chain may be live meanwhile.

```python
nemo_gym.orchestration.executors.slurm.SlurmExecutor.gym_pin(
    record: nemo_gym.orchestration.jobs.SubmissionRecord
) -> str | None
```

The full Gym commit the run's saved config pins its driver to; None when there is none or it is unreadable.

```python
nemo_gym.orchestration.executors.slurm.SlurmExecutor.installed_commit() -> str | None
```

The Gym commit this launcher runs; the same value the judge-only job and a repin are checked against.

```python
nemo_gym.orchestration.executors.slurm.SlurmExecutor.query_states(
    refs: list[nemo_gym.orchestration.jobs.JobRef]
) -> dict[nemo_gym.orchestration.jobs.JobKey, nemo_gym.orchestration.jobs.ObservedStatus]
```

One script on one connection: a single squeue, a single sacct and each job directory's state files.

```python
nemo_gym.orchestration.executors.slurm.SlurmExecutor.read_run_files(
    record: nemo_gym.orchestration.jobs.SubmissionRecord,
    paths: list[pathlib.Path]
) -> dict[pathlib.Path, str | None]
```

```python
nemo_gym.orchestration.executors.slurm.SlurmExecutor.run(
    config: nemo_gym.orchestration.api.SubmitConfig,
    dry_run: bool = False
) -> nemo_gym.orchestration.jobs.SubmissionRecord | None
```

```python
nemo_gym.orchestration.executors.slurm.SlurmExecutor.stream_logs(
    record: nemo_gym.orchestration.jobs.SubmissionRecord,
    job: nemo_gym.orchestration.jobs.BenchmarkJob,
    source: nemo_gym.orchestration.jobs.LogSource,
    tail: int | None,
    follow: bool
) -> collections.abc.Iterator[str]
```

The lines of `source`, read in one call; the location is relative to the job directory.

```python
nemo_gym.orchestration.executors.slurm.SlurmExecutor.terminate(
    record: nemo_gym.orchestration.jobs.SubmissionRecord,
    job: nemo_gym.orchestration.jobs.BenchmarkJob
) -> None
```

Cancel everything of `job` that is queued or running, the pending jobs first, in one script.

```python
nemo_gym.orchestration.executors.slurm.executor._validate_benchmark_names(
    benchmarks: list[str]
) -> None
```

Fail before anything is staged or copied.

`_sbatch_command` raises this same check, but only once submission is
already underway — after staging, connecting, mount validation and the
rsync. Checking here first means a bad name fails before any of that, and
fails on `--dry-run` too, instead of the dry run printing a clean script
listing for a benchmark that could never actually be submitted.

```python
nemo_gym.orchestration.executors.slurm.executor._validate_mounts(
    config: nemo_gym.orchestration.api.SubmitConfig,
    conn: nemo_gym.orchestration.executors.connection.Connection
) -> None
```

```python
nemo_gym.orchestration.executors.slurm.executor._EXPORT_TIMEOUT_S = 3600
```