> This page is for version 0.4.0.
> For other versions, use one of these documentation indexes:
> - Main (default): https://docs.nvidia.com/nemo/gym/main/llms.txt
> - 0.7.0: https://docs.nvidia.com/nemo/gym/v0.7.0/llms.txt
> - 0.6.0: https://docs.nvidia.com/nemo/gym/v0.6.0/llms.txt
> - 0.5.1: https://docs.nvidia.com/nemo/gym/v0.5.1/llms.txt
> - 0.5.0: https://docs.nvidia.com/nemo/gym/v0.5.0/llms.txt
> - 0.4.0: https://docs.nvidia.com/nemo/gym/v0.4.0/llms.txt
> - 0.3.0: https://docs.nvidia.com/nemo/gym/v0.3.0/llms.txt
> - 0.2.1: https://docs.nvidia.com/nemo/gym/v0.2.1/llms.txt

> For clean Markdown of any page, append .md to the page URL.
> For a complete documentation index, see https://docs.nvidia.com/nemo/gym/llms.txt.

# nemo_gym.orchestration.executors.slurm.scheduler

The Slurm commands the executor's parts share: sbatch and its reply, partition limits, live checks.

## Module Contents

### Functions

| Name                                                                                                       | Description                                                                                         |
| ---------------------------------------------------------------------------------------------------------- | --------------------------------------------------------------------------------------------------- |
| [`_chain_live_check`](#nemo_gym-orchestration-executors-slurm-scheduler-_chain_live_check)                 | Shell lines that fail unless the scheduler confirms that no job of `chain` waits or runs.           |
| [`_parse_sbatch_results`](#nemo_gym-orchestration-executors-slurm-scheduler-_parse_sbatch_results)         | Benchmark name to `(job_id, error)`, exactly one of which is set.                                   |
| [`_partition_time_limit`](#nemo_gym-orchestration-executors-slurm-scheduler-_partition_time_limit)         | The partition a job of `compute` queues on and its MaxTime, read with sinfo over `conn`.            |
| [`_sbatch_command`](#nemo_gym-orchestration-executors-slurm-scheduler-_sbatch_command)                     | One `sbatch` that reports its own benchmark, exit status and output.                                |
| [`_slurm_duration`](#nemo_gym-orchestration-executors-slurm-scheduler-_slurm_duration)                     | `seconds` as a Slurm time limit, `HH:MM:SS`.                                                        |
| [`_slurm_time_limit_seconds`](#nemo_gym-orchestration-executors-slurm-scheduler-_slurm_time_limit_seconds) | A partition time limit as sinfo prints it (`%l`) in seconds; None for `UNLIMITED` or `infinite`.    |
| [`_submission_env_exports`](#nemo_gym-orchestration-executors-slurm-scheduler-_submission_env_exports)     | Forward explicitly selected values over SSH stdin, never into artifacts.                            |
| [`_submission_exports`](#nemo_gym-orchestration-executors-slurm-scheduler-_submission_exports)             | What `run()` exports ahead of sbatch: the forwarded values, then the telemetry token when it is on. |

### Data

[`_JOB_ID_RE`](#nemo_gym-orchestration-executors-slurm-scheduler-_JOB_ID_RE)

[`_MARKER`](#nemo_gym-orchestration-executors-slurm-scheduler-_MARKER)

[`_RESUBMIT_LOCK`](#nemo_gym-orchestration-executors-slurm-scheduler-_RESUBMIT_LOCK)

[`_VALID_BENCHMARK_NAME`](#nemo_gym-orchestration-executors-slurm-scheduler-_VALID_BENCHMARK_NAME)

### API

```python
nemo_gym.orchestration.executors.slurm.scheduler._chain_live_check(
    chain: list[str],
    live_error: str
) -> list[str]
```

Shell lines that fail unless the scheduler confirms that no job of `chain` waits or runs.

Slurm drops a finished job from the controller after MinJobAge, and `squeue -j` then exits 1 with "Invalid job
id specified" once it knows none of the ids. That is no answer about the jobs, so sacct, which keeps finished
jobs, decides: a job whose latest incarnation (highest Restarts; sacct -D lists every requeue) is in one of the
`LIVE_JOB_STATES` is live, and one it does not know is not. Any other squeue failure, or a failing sacct, leaves
the scheduler unknown.

```python
nemo_gym.orchestration.executors.slurm.scheduler._parse_sbatch_results(
    output: str
) -> dict[str, tuple[str | None, str | None]]
```

Benchmark name to `(job_id, error)`, exactly one of which is set.

```python
nemo_gym.orchestration.executors.slurm.scheduler._partition_time_limit(
    conn: nemo_gym.orchestration.executors.connection.Connection,
    compute: nemo_gym.orchestration.api.SlurmComputeConfig
) -> dict[str, typing.Any]
```

The partition a job of `compute` queues on and its MaxTime, read with sinfo over `conn`.

The partitions are the node pools' ones, else `extra_args.partition` (a list `a,b` allowed), else the
cluster's default one. Every one must be listed, and the lowest limit applies: `partition` names it,
`max_time` is sinfo's text and `max_time_s` its seconds (None when unlimited); `error` says why they are
unknown.

```python
nemo_gym.orchestration.executors.slurm.scheduler._sbatch_command(
    benchmark: str,
    script: pathlib.Path
) -> str
```

One `sbatch` that reports its own benchmark, exit status and output.

`rc` is captured immediately: `$?` after the `tr` pipeline would be *tr's*
status, which is zero however badly sbatch failed. `tr` flattens sbatch's
multi-line error messages so the whole result stays on one marker line.

```python
nemo_gym.orchestration.executors.slurm.scheduler._slurm_duration(
    seconds: int
) -> str
```

`seconds` as a Slurm time limit, `HH:MM:SS`.

```python
nemo_gym.orchestration.executors.slurm.scheduler._slurm_time_limit_seconds(
    text: str
) -> int | None
```

A partition time limit as sinfo prints it (`%l`) in seconds; None for `UNLIMITED` or `infinite`.

Takes Slurm's duration spellings (`MM`, `MM:SS`, `HH:MM:SS`, `D-HH`, `D-HH:MM`, `D-HH:MM:SS`) and
raises ValueError for anything else, such as `n/a`.

```python
nemo_gym.orchestration.executors.slurm.scheduler._submission_env_exports(
    compute: nemo_gym.orchestration.api.SlurmComputeConfig
) -> list[str]
```

Forward explicitly selected values over SSH stdin, never into artifacts.

```python
nemo_gym.orchestration.executors.slurm.scheduler._submission_exports(
    config: nemo_gym.orchestration.api.SubmitConfig
) -> list[str]
```

What `run()` exports ahead of sbatch: the forwarded values, then the telemetry token when it is on.

```python
nemo_gym.orchestration.executors.slurm.scheduler._JOB_ID_RE = re.compile('^\\d+(;\\S+)?$')
```

```python
nemo_gym.orchestration.executors.slurm.scheduler._MARKER = '__GYM_JOB:'
```

```python
nemo_gym.orchestration.executors.slurm.scheduler._RESUBMIT_LOCK = RESUBMIT_LOCK_NAME
```

```python
nemo_gym.orchestration.executors.slurm.scheduler._VALID_BENCHMARK_NAME = re.compile('^[A-Za-z0-9._-]+$')
```