> This page is for version 0.5.1.
> For other versions, use one of these documentation indexes:
> - Main (default): https://docs.nvidia.com/nemo/gym/main/llms.txt
> - 0.7.0: https://docs.nvidia.com/nemo/gym/v0.7.0/llms.txt
> - 0.6.0: https://docs.nvidia.com/nemo/gym/v0.6.0/llms.txt
> - 0.5.1: https://docs.nvidia.com/nemo/gym/v0.5.1/llms.txt
> - 0.5.0: https://docs.nvidia.com/nemo/gym/v0.5.0/llms.txt
> - 0.4.0: https://docs.nvidia.com/nemo/gym/v0.4.0/llms.txt
> - 0.3.0: https://docs.nvidia.com/nemo/gym/v0.3.0/llms.txt
> - 0.2.1: https://docs.nvidia.com/nemo/gym/v0.2.1/llms.txt

> For clean Markdown of any page, append .md to the page URL.
> For a complete documentation index, see https://docs.nvidia.com/nemo/gym/llms.txt.

# nemo_gym.orchestration.executors.slurm_status

How the Slurm executor looks at the jobs it queued.

One script per host asks Slurm about every job directory of every record at once: a single `squeue` for
the whole batch, a single `sacct` over every job id any of them has had, and the state files each job
directory holds. The cost grows with the number of hosts, not with the number of submissions. Jobs are
found by where they run, not by the recorded id, so a job started by hand, a resumed one, or one
resubmitted from another machine is still seen.

## Module Contents

### Functions

| Name                                                                                                  | Description                                                                                       |
| ----------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------- |
| [`_exit_status`](#nemo_gym-orchestration-executors-slurm_status-_exit_status)                         | The exit code of sacct's `code:signal` pair.                                                      |
| [`_observe`](#nemo_gym-orchestration-executors-slurm_status-_observe)                                 | -                                                                                                 |
| [`parse_read_files_output`](#nemo_gym-orchestration-executors-slurm_status-parse_read_files_output)   | The contents `render_read_files_script` printed, `None` for a file it did not find.               |
| [`parse_status_output`](#nemo_gym-orchestration-executors-slurm_status-parse_status_output)           | What `render_status_script` reported, as one `ObservedStatus` per job in `refs`.                  |
| [`render_read_files_script`](#nemo_gym-orchestration-executors-slurm_status-render_read_files_script) | The commands that print each readable file of `paths`, encoded so a file cannot break the output. |
| [`render_status_script`](#nemo_gym-orchestration-executors-slurm_status-render_status_script)         | The commands that report the state of every job in `refs`, on one connection.                     |
| [`state_from_sacct`](#nemo_gym-orchestration-executors-slurm_status-state_from_sacct)                 | Slurm's own state name as a `JobState`; a name this does not know is `UNKNOWN`.                   |

### Data

[`STATE_FILES`](#nemo_gym-orchestration-executors-slurm_status-STATE_FILES)

[`_ACTIVE_STATES`](#nemo_gym-orchestration-executors-slurm_status-_ACTIVE_STATES)

[`_FAILED_STATES`](#nemo_gym-orchestration-executors-slurm_status-_FAILED_STATES)

[`_KIND_ACCT`](#nemo_gym-orchestration-executors-slurm_status-_KIND_ACCT)

[`_KIND_ACCT_ERROR`](#nemo_gym-orchestration-executors-slurm_status-_KIND_ACCT_ERROR)

[`_KIND_ACTIVE`](#nemo_gym-orchestration-executors-slurm_status-_KIND_ACTIVE)

[`_KIND_CHAIN`](#nemo_gym-orchestration-executors-slurm_status-_KIND_CHAIN)

[`_KIND_FILE`](#nemo_gym-orchestration-executors-slurm_status-_KIND_FILE)

[`_KIND_SQUEUE_ERROR`](#nemo_gym-orchestration-executors-slurm_status-_KIND_SQUEUE_ERROR)

[`_MARK`](#nemo_gym-orchestration-executors-slurm_status-_MARK)

[`_PENDING_STATES`](#nemo_gym-orchestration-executors-slurm_status-_PENDING_STATES)

[`_SEP`](#nemo_gym-orchestration-executors-slurm_status-_SEP)

[`_STATE_FILE_BYTES`](#nemo_gym-orchestration-executors-slurm_status-_STATE_FILE_BYTES)

### API

```python
nemo_gym.orchestration.executors.slurm_status._exit_status(
    text: str
) -> int | None
```

The exit code of sacct's `code:signal` pair.

```python
nemo_gym.orchestration.executors.slurm_status._observe(
    ref: nemo_gym.orchestration.jobs.JobRef,
    chain: list[str],
    active: list[tuple[str, str]],
    files: dict[str, str],
    accounting: dict[str, tuple[str, int | None]],
    squeue_error: str | None,
    acct_error: str | None
) -> nemo_gym.orchestration.jobs.ObservedStatus
```

```python
nemo_gym.orchestration.executors.slurm_status.parse_read_files_output(
    output: str,
    paths: list[pathlib.Path]
) -> dict[pathlib.Path, str | None]
```

The contents `render_read_files_script` printed, `None` for a file it did not find.

```python
nemo_gym.orchestration.executors.slurm_status.parse_status_output(
    output: str,
    refs: list[nemo_gym.orchestration.jobs.JobRef]
) -> dict[nemo_gym.orchestration.jobs.JobKey, nemo_gym.orchestration.jobs.ObservedStatus]
```

What `render_status_script` reported, as one `ObservedStatus` per job in `refs`.

```python
nemo_gym.orchestration.executors.slurm_status.render_read_files_script(
    paths: collections.abc.Iterable[pathlib.Path]
) -> list[str]
```

The commands that print each readable file of `paths`, encoded so a file cannot break the output.

```python
nemo_gym.orchestration.executors.slurm_status.render_status_script(
    refs: collections.abc.Iterable[nemo_gym.orchestration.jobs.JobRef]
) -> list[str]
```

The commands that report the state of every job in `refs`, on one connection.

```python
nemo_gym.orchestration.executors.slurm_status.state_from_sacct(
    native: str
) -> nemo_gym.orchestration.jobs.JobState
```

Slurm's own state name as a `JobState`; a name this does not know is `UNKNOWN`.

```python
nemo_gym.orchestration.executors.slurm_status.STATE_FILES = (CHAIN_INCOMPLETE_FILE, EVAL_EXIT_FILE)
```

```python
nemo_gym.orchestration.executors.slurm_status._ACTIVE_STATES = {'RUNNING', 'COMPLETING', 'CONFIGURING', 'SUSPENDED', 'RESIZING', 'SIGNALING', '...
```

```python
nemo_gym.orchestration.executors.slurm_status._FAILED_STATES = {'FAILED', 'TIMEOUT', 'OUT_OF_MEMORY', 'NODE_FAIL', 'PREEMPTED', 'BOOT_FAIL', 'D...
```

```python
nemo_gym.orchestration.executors.slurm_status._KIND_ACCT = 'acct'
```

```python
nemo_gym.orchestration.executors.slurm_status._KIND_ACCT_ERROR = 'acct-error'
```

```python
nemo_gym.orchestration.executors.slurm_status._KIND_ACTIVE = 'active'
```

```python
nemo_gym.orchestration.executors.slurm_status._KIND_CHAIN = 'chain'
```

```python
nemo_gym.orchestration.executors.slurm_status._KIND_FILE = 'file'
```

```python
nemo_gym.orchestration.executors.slurm_status._KIND_SQUEUE_ERROR = 'squeue-error'
```

```python
nemo_gym.orchestration.executors.slurm_status._MARK = '__GYM_STATE'
```

```python
nemo_gym.orchestration.executors.slurm_status._PENDING_STATES = {'PENDING', 'REQUEUED', 'REQUEUE_HOLD', 'REQUEUE_FED', 'RESV_DEL_HOLD'}
```

```python
nemo_gym.orchestration.executors.slurm_status._SEP = ':'
```

```python
nemo_gym.orchestration.executors.slurm_status._STATE_FILE_BYTES = 4096
```