> This page is for version 0.3.0.
> For other versions, use one of these documentation indexes:
> - Main (default): https://docs.nvidia.com/nemo/gym/main/llms.txt
> - 0.7.0: https://docs.nvidia.com/nemo/gym/v0.7.0/llms.txt
> - 0.6.0: https://docs.nvidia.com/nemo/gym/v0.6.0/llms.txt
> - 0.5.1: https://docs.nvidia.com/nemo/gym/v0.5.1/llms.txt
> - 0.5.0: https://docs.nvidia.com/nemo/gym/v0.5.0/llms.txt
> - 0.4.0: https://docs.nvidia.com/nemo/gym/v0.4.0/llms.txt
> - 0.3.0: https://docs.nvidia.com/nemo/gym/v0.3.0/llms.txt
> - 0.2.1: https://docs.nvidia.com/nemo/gym/v0.2.1/llms.txt

> For clean Markdown of any page, append .md to the page URL.
> For a complete documentation index, see https://docs.nvidia.com/nemo/gym/llms.txt.

# nemo_gym.orchestration.executors.slurm.reader

The remote Store reader: runs the evidence collector and the run's own Store selection on the cluster, in a
Slurm step when one is configured, else on the login host, and bounds, watches and stops it.

## Module Contents

### Classes

| Name                                                                                                | Description                                                                                                    |
| --------------------------------------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------- |
| [`ReaderStep`](#nemo_gym-orchestration-executors-slurm-reader-ReaderStep)                           | How a heavy reader runs in a Slurm step: the caller's srun arguments (each quoted), how long the step may take |
| [`RemoteReaderInterrupted`](#nemo_gym-orchestration-executors-slurm-reader-RemoteReaderInterrupted) | A signal stopped recovery while the login-host Store reader ran; the reader was stopped by its token.          |

### Functions

| Name                                                                                                            | Description                                                                                                   |
| --------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------- |
| [`_bound_lines`](#nemo_gym-orchestration-executors-slurm-reader-_bound_lines)                                   | -                                                                                                             |
| [`_env_count`](#nemo_gym-orchestration-executors-slurm-reader-_env_count)                                       | -                                                                                                             |
| [`_env_seconds`](#nemo_gym-orchestration-executors-slurm-reader-_env_seconds)                                   | A nonnegative number of seconds from the environment; the default for an unset or invalid value.              |
| [`_kill_reader_command`](#nemo_gym-orchestration-executors-slurm-reader-_kill_reader_command)                   | Stop this user's login-host processes whose environment or command line carries `token`: TERM, then KILL      |
| [`_reader_command`](#nemo_gym-orchestration-executors-slurm-reader-_reader_command)                             | The login-host steps: size the bound from the rollout files, run the reader, report its exit code and stderr. |
| [`_reader_command_in_step`](#nemo_gym-orchestration-executors-slurm-reader-_reader_command_in_step)             | `_reader_command` with the reader in a Slurm step: the payload is saved once, sent on the step's stdin, and   |
| [`_reader_env_lines`](#nemo_gym-orchestration-executors-slurm-reader-_reader_env_lines)                         | The settings a remote reader takes from its caller: the reader's command starts from a cleared environment on |
| [`_reader_failure`](#nemo_gym-orchestration-executors-slurm-reader-_reader_failure)                             | The reader failure named from its report with its exit code and stderr, and whether it reached its deadline   |
| [`_reader_step_lines`](#nemo_gym-orchestration-executors-slurm-reader-_reader_step_lines)                       | Bash that runs `script` in a Slurm step, else leaves it to the login host.                                    |
| [`_remote_managed_evidence`](#nemo_gym-orchestration-executors-slurm-reader-_remote_managed_evidence)           | Use the saved pinned install over the existing transport; never import Gym into plain host Python.            |
| [`_remote_managed_reader_script`](#nemo_gym-orchestration-executors-slurm-reader-_remote_managed_reader_script) | The reader with the caller's metadata helper defined after its imports, sent as one script.                   |
| [`_remote_reader_bound_s`](#nemo_gym-orchestration-executors-slurm-reader-_remote_reader_bound_s)               | The explicit reader bound from `NEMO_GYM_REMOTE_READER_TIMEOUT_S`, clamped to \[300, 14400], else None.       |
| [`_remote_reader_error_text`](#nemo_gym-orchestration-executors-slurm-reader-_remote_reader_error_text)         | Name the reader failure with its cause, not only the exception type: the end of the message, where it is.     |
| [`_report_placement_lines`](#nemo_gym-orchestration-executors-slurm-reader-_report_placement_lines)             | -                                                                                                             |
| [`_run_reader`](#nemo_gym-orchestration-executors-slurm-reader-_run_reader)                                     | Run the reader. A signal on the client (raised as `KeyboardInterrupt`) leaves the remote reader running,      |
| [`_watch_lines`](#nemo_gym-orchestration-executors-slurm-reader-_watch_lines)                                   | Bash that waits for the background reader `$_gym_reader`, started in its own session, and sets `_gym_rc`.     |
| [`collector_command`](#nemo_gym-orchestration-executors-slurm-reader-collector_command)                         | The collector in a reader step, else on the login host; its output keeps only the collector's own lines after |
| [`reader_placement`](#nemo_gym-orchestration-executors-slurm-reader-reader_placement)                           | Where a reader ran ("login" or "cpu:\<job id>") and why it fell back to the login host, from its report; both |
| [`reader_step`](#nemo_gym-orchestration-executors-slurm-reader-reader_step)                                     | The reader step from `NEMO_GYM_REMOTE_READER_SRUN` (extra srun arguments, for example `-p cpu -c 4`),         |
| [`reader_token`](#nemo_gym-orchestration-executors-slurm-reader-reader_token)                                   | The token this process's login-host Store readers carry in their environment as `GYM_READER_TOKEN`: the       |
| [`without_placement`](#nemo_gym-orchestration-executors-slurm-reader-without_placement)                         | The output without the reader placement report.                                                               |

### Data

[`COLLECTOR_NOT_RUN`](#nemo_gym-orchestration-executors-slurm-reader-COLLECTOR_NOT_RUN)

[`READER_LOGIN_FALLBACK_ENV`](#nemo_gym-orchestration-executors-slurm-reader-READER_LOGIN_FALLBACK_ENV)

[`READER_MAX_ENV`](#nemo_gym-orchestration-executors-slurm-reader-READER_MAX_ENV)

[`READER_NOT_RUN_HOST`](#nemo_gym-orchestration-executors-slurm-reader-READER_NOT_RUN_HOST)

[`READER_QUEUE_WAIT_ENV`](#nemo_gym-orchestration-executors-slurm-reader-READER_QUEUE_WAIT_ENV)

[`READER_SRUN_ENV`](#nemo_gym-orchestration-executors-slurm-reader-READER_SRUN_ENV)

[`READER_SRUN_WAIT_ENV`](#nemo_gym-orchestration-executors-slurm-reader-READER_SRUN_WAIT_ENV)

[`READER_TOKEN_ENV`](#nemo_gym-orchestration-executors-slurm-reader-READER_TOKEN_ENV)

[`_KILL_READER_SCRIPT`](#nemo_gym-orchestration-executors-slurm-reader-_KILL_READER_SCRIPT)

[`_READER_CANCEL_WAIT_S`](#nemo_gym-orchestration-executors-slurm-reader-_READER_CANCEL_WAIT_S)

[`_READER_CEILING_S`](#nemo_gym-orchestration-executors-slurm-reader-_READER_CEILING_S)

[`_READER_FALLBACK_PREFIX`](#nemo_gym-orchestration-executors-slurm-reader-_READER_FALLBACK_PREFIX)

[`_READER_FLOOR_S`](#nemo_gym-orchestration-executors-slurm-reader-_READER_FLOOR_S)

[`_READER_HOST_PREFIX`](#nemo_gym-orchestration-executors-slurm-reader-_READER_HOST_PREFIX)

[`_READER_PROGRESS_INTERVAL_S`](#nemo_gym-orchestration-executors-slurm-reader-_READER_PROGRESS_INTERVAL_S)

[`_READER_PROGRESS_SCRIPT`](#nemo_gym-orchestration-executors-slurm-reader-_READER_PROGRESS_SCRIPT)

[`_READER_QUEUE_POLL_S`](#nemo_gym-orchestration-executors-slurm-reader-_READER_QUEUE_POLL_S)

[`_READER_QUEUE_WAIT_DEFAULT_S`](#nemo_gym-orchestration-executors-slurm-reader-_READER_QUEUE_WAIT_DEFAULT_S)

[`_READER_RETRY_BACKOFF_S`](#nemo_gym-orchestration-executors-slurm-reader-_READER_RETRY_BACKOFF_S)

[`_READER_SRUN_RETRY_MAX_S`](#nemo_gym-orchestration-executors-slurm-reader-_READER_SRUN_RETRY_MAX_S)

[`_READER_SRUN_RETRY_S`](#nemo_gym-orchestration-executors-slurm-reader-_READER_SRUN_RETRY_S)

[`_READER_S_PER_GB`](#nemo_gym-orchestration-executors-slurm-reader-_READER_S_PER_GB)

[`_READER_UV_LOCK_TIMEOUT_DEFAULT_S`](#nemo_gym-orchestration-executors-slurm-reader-_READER_UV_LOCK_TIMEOUT_DEFAULT_S)

[`_REMOTE_MANAGED_READER`](#nemo_gym-orchestration-executors-slurm-reader-_REMOTE_MANAGED_READER)

[`_SRUN_REFUSAL`](#nemo_gym-orchestration-executors-slurm-reader-_SRUN_REFUSAL)

[`_reader_token`](#nemo_gym-orchestration-executors-slurm-reader-_reader_token)

### API

```python
class nemo_gym.orchestration.executors.slurm.reader.ReaderStep(
    args: str,
    wait_s: int,
    max_readers: int,
    login_fallback: bool = False,
    queue_wait_s: int = 0
)
```

Dataclass

How a heavy reader runs in a Slurm step: the caller's srun arguments (each quoted), how long the step may take
to start, how many reader steps the user may have running or pending at once, how long a reader waits for one of
them to end when they are at that cap, and whether a reader that cannot run in its step may run on the login host
instead.

**`args`** `str`

---

**`login_fallback`** `bool = False`

---

**`max_readers`** `int`

---

**`queue_wait_s`** `int = 0`

---

**`wait_s`** `int`

---

```python
class nemo_gym.orchestration.executors.slurm.reader.RemoteReaderInterrupted()
```

**Bases:** `KeyboardInterrupt`

A signal stopped recovery while the login-host Store reader ran; the reader was stopped by its token.

A `KeyboardInterrupt`, so the reader's ordinary error handling does not turn it into evidence and the open
connection unwinds and closes its master.

```python
nemo_gym.orchestration.executors.slurm.reader._bound_lines(
    job_dir: pathlib.Path,
    bound_s: int | None,
    factor: int
) -> list[str]
```

```python
nemo_gym.orchestration.executors.slurm.reader._env_count(
    name: str,
    default: int
) -> int
```

```python
nemo_gym.orchestration.executors.slurm.reader._env_seconds(
    name: str,
    default: int
) -> int
```

A nonnegative number of seconds from the environment; the default for an unset or invalid value.

```python
nemo_gym.orchestration.executors.slurm.reader._kill_reader_command(
    token: str
) -> list[str]
```

Stop this user's login-host processes whose environment or command line carries `token`: TERM, then KILL
for any still running after 10 s. It prints `__GYM_READER_KILL:term=<n> kill=<n>`.

```python
nemo_gym.orchestration.executors.slurm.reader._reader_command(
    entrypoint: str,
    job_dir: pathlib.Path,
    payload: str,
    bound_s: int | None,
    factor: int,
    token: str | None = None,
    watchdog: bool = False,
    progress_interval_s: int | None = None,
    step: nemo_gym.orchestration.executors.slurm.reader.ReaderStep | None = None
) -> list[str]
```

The login-host steps: size the bound from the rollout files, run the reader, report its exit code and stderr.

The reader's own failure is reported on stdout (`__GYM_READER_*` lines) so a timeout, a MemoryError or a
failed bootstrap reaches the decision with its exit code and the end of its stderr; only SSH itself fails the
command. With `token`, the reader runs with it as `GYM_READER_TOKEN` in its environment. With `step`, it
runs in a Slurm step when it can (`_reader_step_lines`) and reports where it ran.

With `watchdog`, for a connection that holds the shell's stdin open while it runs: the reader runs in its own
session (`setsid`), and a watcher in its own session waits for EOF on the shell's stdin, which comes when the
client process or its SSH connection ends, then kills the reader's process group (TERM, KILL after 10 s). It is
an EOF loop, not a hangup: without a terminal nothing sends SIGHUP. The steps run as one function that exits the
shell, so the watcher never reads a line of the script itself. In a step, the watched process is srun, and the
watcher also cancels the step by name. On the login host, a progress loop in its own session prints
`gym-reader progress: pass=<n> bytes=<b> elapsed_s=<s>` on stderr every `progress_interval_s` (60 s by
default) while the reader runs; a reader in a step has none. Stdout and its `__GYM_READER_*` lines are
unchanged.

```python
nemo_gym.orchestration.executors.slurm.reader._reader_command_in_step(
    entrypoint: str,
    job_dir: pathlib.Path,
    payload: str,
    bound_s: int | None,
    factor: int,
    token: str | None,
    step: nemo_gym.orchestration.executors.slurm.reader.ReaderStep,
    watchdog: bool = False
) -> list[str]
```

`_reader_command` with the reader in a Slurm step: the payload is saved once, sent on the step's stdin, and
read again from the file by a login-host reader that runs instead. With `watchdog`, srun and that login-host
reader each run under the EOF watcher, as the reader of `_reader_command(..., watchdog=True)` does.

```python
nemo_gym.orchestration.executors.slurm.reader._reader_env_lines() -> list[str]
```

The settings a remote reader takes from its caller: the reader's command starts from a cleared environment on
another host, so without this a cold install there waits uv's own default for the lock and gives up early.

```python
nemo_gym.orchestration.executors.slurm.reader._reader_failure(
    output: str,
    elapsed_s: float
) -> tuple[str | None, bool]
```

The reader failure named from its report with its exit code and stderr, and whether it reached its deadline
(the only failure retried with twice the bound); (None, False) when it exited 0.

```python
nemo_gym.orchestration.executors.slurm.reader._reader_step_lines(
    step: nemo_gym.orchestration.executors.slurm.reader.ReaderStep,
    token: str,
    script: str,
    stdin: str,
    watchdog: bool = False
) -> list[str]
```

Bash that runs `script` in a Slurm step, else leaves it to the login host.

It sets `_gym_host` ("login" or "cpu:\<job id>") and `_gym_fallback` (why the reader stays on the login host,
empty when it ran in the step); after a step that ran, `_gym_rc` is its exit code and `$_gym_step_out` holds
its output. Inside the step the input on stdin becomes FD 3, because srun forwards stdin but no other descriptor,
and `script` reads the reader bound as `$1`. The step is named gym-reader-\<token>, and its time limit is the
reader bound plus 60 s, so Slurm ends an orphan. A step that has not started after the wait is cancelled by name,
and the reader runs on the login host. With `watchdog` (FD 4 is the shell's stdin), srun runs under the EOF
watcher (`_watch_lines`), which also cancels the step by name. Without `step.login_fallback`, a reader that
would fall back runs nowhere: `_gym_host` is `none`, `_gym_rc` 1, and `_gym_fallback` says why.

When the user's reader steps are at `step.max_readers`, the reader queues: it counts them again every
`_READER_QUEUE_POLL_S` seconds for up to `step.queue_wait_s`, saying on stderr how many it is queued behind
(a progress line), and runs in its step once one ends. Only a cap still reached after that wait keeps it out.

A step srun refused outright (`_SRUN_REFUSAL`: a submit or QOS limit, or no node free to start it) was never
queued, so the reader submits it again after `_READER_SRUN_RETRY_S` seconds, doubling up to
`_READER_SRUN_RETRY_MAX_S`, with a progress line each time, for as long as `step.queue_wait_s` allows. Any
other srun failure, or a refusal past that wait, falls through to the cancellation below as before.

```python
nemo_gym.orchestration.executors.slurm.reader._remote_managed_evidence(
    conn: nemo_gym.orchestration.executors.connection.Connection,
    collected: nemo_gym.eval.recovery.evidence.CollectedEvidence,
    job_dir: pathlib.Path,
    config: nemo_gym.orchestration.api.SubmitConfig
) -> nemo_gym.eval.recovery.evidence.CollectedEvidence
```

Use the saved pinned install over the existing transport; never import Gym into plain host Python.

```python
nemo_gym.orchestration.executors.slurm.reader._remote_managed_reader_script() -> str
```

The reader with the caller's metadata helper defined after its imports, sent as one script.

```python
nemo_gym.orchestration.executors.slurm.reader._remote_reader_bound_s() -> int | None
```

The explicit reader bound from `NEMO_GYM_REMOTE_READER_TIMEOUT_S`, clamped to \[300, 14400], else None.

Without it the login host sizes the bound from the run's rollout files (`_reader_command`).

```python
nemo_gym.orchestration.executors.slurm.reader._remote_reader_error_text(
    error: BaseException
) -> str
```

Name the reader failure with its cause, not only the exception type: the end of the message, where it is.

```python
nemo_gym.orchestration.executors.slurm.reader._report_placement_lines() -> list[str]
```

```python
nemo_gym.orchestration.executors.slurm.reader._run_reader(
    conn: nemo_gym.orchestration.executors.connection.Connection,
    command: list[str],
    token: str
) -> str
```

Run the reader. A signal on the client (raised as `KeyboardInterrupt`) leaves the remote reader running,
since `ssh bash -s` has no terminal to hang up, so it is stopped by its token before the error unwinds.

A connection that `holds_input` keeps the remote shell's stdin open while the reader runs, and the command's
watcher (`_reader_command(..., watchdog=True)`) stops the reader when that stdin ends.

```python
nemo_gym.orchestration.executors.slurm.reader._watch_lines(
    cancel: str | None = None,
    progress: list[str] | None = None
) -> list[str]
```

Bash that waits for the background reader `$_gym_reader`, started in its own session, and sets `_gym_rc`.

A watcher in its own session reads the shell's stdin through FD 4 and at EOF kills the reader's process group:
TERM, then KILL after 10 s. With `cancel`, a quoted Slurm job name, it cancels that job before the TERM: the step
runs on another host, and srun may exit at the TERM, which ends the wait and stops the watcher. `progress`
starts a loop that sets `$_gym_progress`; it is stopped after the wait.

```python
nemo_gym.orchestration.executors.slurm.reader.collector_command(
    collect: str,
    job_dir: pathlib.Path,
    step: nemo_gym.orchestration.executors.slurm.reader.ReaderStep,
    token: str
) -> list[str]
```

The collector in a reader step, else on the login host; its output keeps only the collector's own lines after
the placement report is taken off (`reader_placement`, `without_placement`). Its exit code is the
collector's.

```python
nemo_gym.orchestration.executors.slurm.reader.reader_placement(
    output: str
) -> tuple[str | None, str | None]
```

Where a reader ran ("login" or "cpu:\<job id>") and why it fell back to the login host, from its report; both
None for a reader that ran without a reader step.

```python
nemo_gym.orchestration.executors.slurm.reader.reader_step() -> nemo_gym.orchestration.executors.slurm.reader.ReaderStep | None
```

The reader step from `NEMO_GYM_REMOTE_READER_SRUN` (extra srun arguments, for example `-p cpu -c 4`),
`NEMO_GYM_REMOTE_READER_SRUN_WAIT_S` (default 120), `NEMO_GYM_REMOTE_READER_MAX` (default 8),
`NEMO_GYM_REMOTE_READER_QUEUE_WAIT_S` (default 600; 0 does not wait) and
`NEMO_GYM_REMOTE_READER_LOGIN_FALLBACK` (`allow`; anything else forbids it); None when the arguments are unset
or empty.

```python
nemo_gym.orchestration.executors.slurm.reader.reader_token() -> str
```

The token this process's login-host Store readers carry in their environment as `GYM_READER_TOKEN`: the
caller's own value when it sets one, else one generated uuid for the process. Recovery reports it as
`reader_token`, so whoever stops a recovery can find its reader.

```python
nemo_gym.orchestration.executors.slurm.reader.without_placement(
    output: str
) -> str
```

The output without the reader placement report.

```python
nemo_gym.orchestration.executors.slurm.reader.COLLECTOR_NOT_RUN = 'the evidence collector could not run in its Slurm step and the login-host fallb...
```

```python
nemo_gym.orchestration.executors.slurm.reader.READER_LOGIN_FALLBACK_ENV = 'NEMO_GYM_REMOTE_READER_LOGIN_FALLBACK'
```

```python
nemo_gym.orchestration.executors.slurm.reader.READER_MAX_ENV = 'NEMO_GYM_REMOTE_READER_MAX'
```

```python
nemo_gym.orchestration.executors.slurm.reader.READER_NOT_RUN_HOST = 'none'
```

```python
nemo_gym.orchestration.executors.slurm.reader.READER_QUEUE_WAIT_ENV = 'NEMO_GYM_REMOTE_READER_QUEUE_WAIT_S'
```

```python
nemo_gym.orchestration.executors.slurm.reader.READER_SRUN_ENV = 'NEMO_GYM_REMOTE_READER_SRUN'
```

```python
nemo_gym.orchestration.executors.slurm.reader.READER_SRUN_WAIT_ENV = 'NEMO_GYM_REMOTE_READER_SRUN_WAIT_S'
```

```python
nemo_gym.orchestration.executors.slurm.reader.READER_TOKEN_ENV = 'GYM_READER_TOKEN'
```

```python
nemo_gym.orchestration.executors.slurm.reader._KILL_READER_SCRIPT = '_gym_token=@TOKEN@\n_gym_matches() {\n    local proc pid environ cmdline\n    f...
```

```python
nemo_gym.orchestration.executors.slurm.reader._READER_CANCEL_WAIT_S = 60
```

```python
nemo_gym.orchestration.executors.slurm.reader._READER_CEILING_S = 14400
```

```python
nemo_gym.orchestration.executors.slurm.reader._READER_FALLBACK_PREFIX = '__GYM_READER_FALLBACK:'
```

```python
nemo_gym.orchestration.executors.slurm.reader._READER_FLOOR_S = 900
```

```python
nemo_gym.orchestration.executors.slurm.reader._READER_HOST_PREFIX = '__GYM_READER_HOST:'
```

```python
nemo_gym.orchestration.executors.slurm.reader._READER_PROGRESS_INTERVAL_S = 60
```

```python
nemo_gym.orchestration.executors.slurm.reader._READER_PROGRESS_SCRIPT = 'while sleep "$2" >/dev/null 2>&1; kill -0 "$1" 2>/dev/null; do\n    b=0\n    fo...
```

```python
nemo_gym.orchestration.executors.slurm.reader._READER_QUEUE_POLL_S = 10
```

```python
nemo_gym.orchestration.executors.slurm.reader._READER_QUEUE_WAIT_DEFAULT_S = 600
```

```python
nemo_gym.orchestration.executors.slurm.reader._READER_RETRY_BACKOFF_S = 10
```

```python
nemo_gym.orchestration.executors.slurm.reader._READER_SRUN_RETRY_MAX_S = 60
```

```python
nemo_gym.orchestration.executors.slurm.reader._READER_SRUN_RETRY_S = 15
```

```python
nemo_gym.orchestration.executors.slurm.reader._READER_S_PER_GB = 100
```

```python
nemo_gym.orchestration.executors.slurm.reader._READER_UV_LOCK_TIMEOUT_DEFAULT_S = 1800
```

```python
nemo_gym.orchestration.executors.slurm.reader._REMOTE_MANAGED_READER = '\nimport json, subprocess, sys\nfrom pathlib import Path\ntry:\n    import nemo...
```

```python
nemo_gym.orchestration.executors.slurm.reader._SRUN_REFUSAL = 'Job violates accounting/QOS policy|Requested nodes are busy|Immediate execution...
```

```python
nemo_gym.orchestration.executors.slurm.reader._reader_token: str | None = None
```