> This page is for version Main (default).
> For other versions, use one of these documentation indexes:
> - Main (default): https://docs.nvidia.com/nemo/gym/main/llms.txt
> - 0.7.0: https://docs.nvidia.com/nemo/gym/v0.7.0/llms.txt
> - 0.6.0: https://docs.nvidia.com/nemo/gym/v0.6.0/llms.txt
> - 0.5.1: https://docs.nvidia.com/nemo/gym/v0.5.1/llms.txt
> - 0.5.0: https://docs.nvidia.com/nemo/gym/v0.5.0/llms.txt
> - 0.4.0: https://docs.nvidia.com/nemo/gym/v0.4.0/llms.txt
> - 0.3.0: https://docs.nvidia.com/nemo/gym/v0.3.0/llms.txt
> - 0.2.1: https://docs.nvidia.com/nemo/gym/v0.2.1/llms.txt

> For clean Markdown of any page, append .md to the page URL.
> For a complete documentation index, see https://docs.nvidia.com/nemo/gym/llms.txt.

# nemo_gym.orchestration.executors.slurm_control

How the Slurm executor stops the jobs it queued.

Stopping a benchmark is more than `scancel` on its latest job. Under auto-resume every job queues its own
follow-up, so a follow-up that is still pending would start after its parent is cancelled. The script finds
everything of the benchmark that is queued or running -- by the directory it runs in, and by the job ids
the chain and the follow-up claim files name -- and cancels the pending jobs before the running ones, so no
follow-up can start in between.

A job that is being cancelled can still queue its follow-up in the moments before it dies, after the queue
was read. So the queue is read again until nothing of the benchmark is left in it, and whatever appeared in
between is cancelled too.

## Module Contents

### Functions

| Name                                                                                                 | Description                                                                                           |
| ---------------------------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------- |
| [`_line`](#nemo_gym-orchestration-executors-slurm_control-_line)                                     | -                                                                                                     |
| [`parse_terminate_output`](#nemo_gym-orchestration-executors-slurm_control-parse_terminate_output)   | The job ids `render_terminate_script` cancelled, and what went wrong, if anything.                    |
| [`render_terminate_script`](#nemo_gym-orchestration-executors-slurm_control-render_terminate_script) | The commands that cancel every queued or running job of `job`: the pending ones first, then the rest. |

### Data

[`FOLLOWUP_CLAIM_PREFIX`](#nemo_gym-orchestration-executors-slurm_control-FOLLOWUP_CLAIM_PREFIX)

[`_DRAIN_PASSES`](#nemo_gym-orchestration-executors-slurm_control-_DRAIN_PASSES)

[`_DRAIN_WAIT_S`](#nemo_gym-orchestration-executors-slurm_control-_DRAIN_WAIT_S)

[`_FLAT_GROUP`](#nemo_gym-orchestration-executors-slurm_control-_FLAT_GROUP)

[`_FLAT_OUT`](#nemo_gym-orchestration-executors-slurm_control-_FLAT_OUT)

[`_FLAT_QUEUE`](#nemo_gym-orchestration-executors-slurm_control-_FLAT_QUEUE)

[`_KIND_CANCELLED`](#nemo_gym-orchestration-executors-slurm_control-_KIND_CANCELLED)

[`_KIND_ERROR`](#nemo_gym-orchestration-executors-slurm_control-_KIND_ERROR)

[`_KIND_NONE`](#nemo_gym-orchestration-executors-slurm_control-_KIND_NONE)

[`_MARK`](#nemo_gym-orchestration-executors-slurm_control-_MARK)

[`_SEP`](#nemo_gym-orchestration-executors-slurm_control-_SEP)

### API

```python
nemo_gym.orchestration.executors.slurm_control._line(
    kind: str,
    payload: str
) -> str
```

```python
nemo_gym.orchestration.executors.slurm_control.parse_terminate_output(
    output: str
) -> tuple[list[str], list[str]]
```

The job ids `render_terminate_script` cancelled, and what went wrong, if anything.

```python
nemo_gym.orchestration.executors.slurm_control.render_terminate_script(
    job: nemo_gym.orchestration.jobs.BenchmarkJob
) -> list[str]
```

The commands that cancel every queued or running job of `job`: the pending ones first, then the rest.

```python
nemo_gym.orchestration.executors.slurm_control.FOLLOWUP_CLAIM_PREFIX = '.gym_followup_queued.'
```

```python
nemo_gym.orchestration.executors.slurm_control._DRAIN_PASSES = 6
```

```python
nemo_gym.orchestration.executors.slurm_control._DRAIN_WAIT_S = 5
```

```python
nemo_gym.orchestration.executors.slurm_control._FLAT_GROUP = '$(echo $_group | tr " " ",")'
```

```python
nemo_gym.orchestration.executors.slurm_control._FLAT_OUT = '$(echo "$out" | tr "\\n" " ")'
```

```python
nemo_gym.orchestration.executors.slurm_control._FLAT_QUEUE = '$(echo "$squeue_out" | tr "\\n" " ")'
```

```python
nemo_gym.orchestration.executors.slurm_control._KIND_CANCELLED = 'cancelled'
```

```python
nemo_gym.orchestration.executors.slurm_control._KIND_ERROR = 'error'
```

```python
nemo_gym.orchestration.executors.slurm_control._KIND_NONE = 'none'
```

```python
nemo_gym.orchestration.executors.slurm_control._MARK = '__GYM_CONTROL'
```

```python
nemo_gym.orchestration.executors.slurm_control._SEP = ':'
```