Skip to navigation

nemo_gym.orchestration.executors.slurm.reader

View as MarkdownOpen in Claude

The remote Store reader: runs the evidence collector and the run’s own Store selection on the cluster, in a Slurm step when one is configured, else on the login host, and bounds, watches and stops it.

Module Contents

Classes

NameDescription
ReaderStepHow a heavy reader runs in a Slurm step: the caller’s srun arguments (each quoted), how long the step may take
RemoteReaderInterruptedA signal stopped recovery while the login-host Store reader ran; the reader was stopped by its token.

Functions

NameDescription
_bound_lines-
_env_count-
_env_secondsA nonnegative number of seconds from the environment; the default for an unset or invalid value.
_kill_reader_commandStop this user’s login-host processes whose environment or command line carries token: TERM, then KILL
_reader_commandThe login-host steps: size the bound from the rollout files, run the reader, report its exit code and stderr.
_reader_command_in_step_reader_command with the reader in a Slurm step: the payload is saved once, sent on the step’s stdin, and
_reader_env_linesThe settings a remote reader takes from its caller: the reader’s command starts from a cleared environment on
_reader_failureThe reader failure named from its report with its exit code and stderr, and whether it reached its deadline
_reader_step_linesBash that runs script in a Slurm step, else leaves it to the login host.
_remote_managed_evidenceUse the saved pinned install over the existing transport; never import Gym into plain host Python.
_remote_managed_reader_scriptThe reader with the caller’s metadata helper defined after its imports, sent as one script.
_remote_reader_bound_sThe explicit reader bound from NEMO_GYM_REMOTE_READER_TIMEOUT_S, clamped to [300, 14400], else None.
_remote_reader_error_textName the reader failure with its cause, not only the exception type: the end of the message, where it is.
_report_placement_lines-
_run_readerRun the reader. A signal on the client (raised as KeyboardInterrupt) leaves the remote reader running,
_watch_linesBash that waits for the background reader $_gym_reader, started in its own session, and sets _gym_rc.
collector_commandThe collector in a reader step, else on the login host; its output keeps only the collector’s own lines after
reader_placementWhere a reader ran (“login” or “cpu:<job id>”) and why it fell back to the login host, from its report; both
reader_stepThe reader step from NEMO_GYM_REMOTE_READER_SRUN (extra srun arguments, for example -p cpu -c 4),
reader_tokenThe token this process’s login-host Store readers carry in their environment as GYM_READER_TOKEN: the
without_placementThe output without the reader placement report.

Data

COLLECTOR_NOT_RUN

READER_LOGIN_FALLBACK_ENV

READER_MAX_ENV

READER_NOT_RUN_HOST

READER_QUEUE_WAIT_ENV

READER_SRUN_ENV

READER_SRUN_WAIT_ENV

READER_TOKEN_ENV

_KILL_READER_SCRIPT

_READER_CANCEL_WAIT_S

_READER_CEILING_S

_READER_FALLBACK_PREFIX

_READER_FLOOR_S

_READER_HOST_PREFIX

_READER_PROGRESS_INTERVAL_S

_READER_PROGRESS_SCRIPT

_READER_QUEUE_POLL_S

_READER_QUEUE_WAIT_DEFAULT_S

_READER_RETRY_BACKOFF_S

_READER_SRUN_RETRY_MAX_S

_READER_SRUN_RETRY_S

_READER_S_PER_GB

_READER_UV_LOCK_TIMEOUT_DEFAULT_S

_REMOTE_MANAGED_READER

_SRUN_REFUSAL

_reader_token

API

class nemo_gym.orchestration.executors.slurm.reader.ReaderStep(
args: str,
wait_s: int,
max_readers: int,
login_fallback: bool = False,
queue_wait_s: int = 0
)
Dataclass

How a heavy reader runs in a Slurm step: the caller’s srun arguments (each quoted), how long the step may take to start, how many reader steps the user may have running or pending at once, how long a reader waits for one of them to end when they are at that cap, and whether a reader that cannot run in its step may run on the login host instead.

args
str
login_fallback
bool = False
max_readers
int
queue_wait_s
int = 0
wait_s
int
class nemo_gym.orchestration.executors.slurm.reader.RemoteReaderInterrupted()

Bases: KeyboardInterrupt

A signal stopped recovery while the login-host Store reader ran; the reader was stopped by its token.

A KeyboardInterrupt, so the reader’s ordinary error handling does not turn it into evidence and the open connection unwinds and closes its master.

nemo_gym.orchestration.executors.slurm.reader._bound_lines(
job_dir: pathlib.Path,
bound_s: int | None,
factor: int
) -> list[str]
nemo_gym.orchestration.executors.slurm.reader._env_count(
name: str,
default: int
) -> int
nemo_gym.orchestration.executors.slurm.reader._env_seconds(
name: str,
default: int
) -> int

A nonnegative number of seconds from the environment; the default for an unset or invalid value.

nemo_gym.orchestration.executors.slurm.reader._kill_reader_command(
token: str
) -> list[str]

Stop this user’s login-host processes whose environment or command line carries token: TERM, then KILL for any still running after 10 s. It prints __GYM_READER_KILL:term=<n> kill=<n>.

nemo_gym.orchestration.executors.slurm.reader._reader_command(
entrypoint: str,
job_dir: pathlib.Path,
payload: str,
bound_s: int | None,
factor: int,
token: str | None = None,
watchdog: bool = False,
progress_interval_s: int | None = None,
) -> list[str]

The login-host steps: size the bound from the rollout files, run the reader, report its exit code and stderr.

The reader’s own failure is reported on stdout (__GYM_READER_* lines) so a timeout, a MemoryError or a failed bootstrap reaches the decision with its exit code and the end of its stderr; only SSH itself fails the command. With token, the reader runs with it as GYM_READER_TOKEN in its environment. With step, it runs in a Slurm step when it can (_reader_step_lines) and reports where it ran.

With watchdog, for a connection that holds the shell’s stdin open while it runs: the reader runs in its own session (setsid), and a watcher in its own session waits for EOF on the shell’s stdin, which comes when the client process or its SSH connection ends, then kills the reader’s process group (TERM, KILL after 10 s). It is an EOF loop, not a hangup: without a terminal nothing sends SIGHUP. The steps run as one function that exits the shell, so the watcher never reads a line of the script itself. In a step, the watched process is srun, and the watcher also cancels the step by name. On the login host, a progress loop in its own session prints gym-reader progress: pass=<n> bytes=<b> elapsed_s=<s> on stderr every progress_interval_s (60 s by default) while the reader runs; a reader in a step has none. Stdout and its __GYM_READER_* lines are unchanged.

nemo_gym.orchestration.executors.slurm.reader._reader_command_in_step(
entrypoint: str,
job_dir: pathlib.Path,
payload: str,
bound_s: int | None,
factor: int,
token: str | None,
watchdog: bool = False
) -> list[str]

_reader_command with the reader in a Slurm step: the payload is saved once, sent on the step’s stdin, and read again from the file by a login-host reader that runs instead. With watchdog, srun and that login-host reader each run under the EOF watcher, as the reader of _reader_command(..., watchdog=True) does.

nemo_gym.orchestration.executors.slurm.reader._reader_env_lines() -> list[str]

The settings a remote reader takes from its caller: the reader’s command starts from a cleared environment on another host, so without this a cold install there waits uv’s own default for the lock and gives up early.

nemo_gym.orchestration.executors.slurm.reader._reader_failure(
output: str,
elapsed_s: float
) -> tuple[str | None, bool]

The reader failure named from its report with its exit code and stderr, and whether it reached its deadline (the only failure retried with twice the bound); (None, False) when it exited 0.

nemo_gym.orchestration.executors.slurm.reader._reader_step_lines(
token: str,
script: str,
stdin: str,
watchdog: bool = False
) -> list[str]

Bash that runs script in a Slurm step, else leaves it to the login host.

It sets _gym_host (“login” or “cpu:<job id>”) and _gym_fallback (why the reader stays on the login host, empty when it ran in the step); after a step that ran, _gym_rc is its exit code and $_gym_step_out holds its output. Inside the step the input on stdin becomes FD 3, because srun forwards stdin but no other descriptor, and script reads the reader bound as $1. The step is named gym-reader-<token>, and its time limit is the reader bound plus 60 s, so Slurm ends an orphan. A step that has not started after the wait is cancelled by name, and the reader runs on the login host. With watchdog (FD 4 is the shell’s stdin), srun runs under the EOF watcher (_watch_lines), which also cancels the step by name. Without step.login_fallback, a reader that would fall back runs nowhere: _gym_host is none, _gym_rc 1, and _gym_fallback says why.

When the user’s reader steps are at step.max_readers, the reader queues: it counts them again every _READER_QUEUE_POLL_S seconds for up to step.queue_wait_s, saying on stderr how many it is queued behind (a progress line), and runs in its step once one ends. Only a cap still reached after that wait keeps it out.

A step srun refused outright (_SRUN_REFUSAL: a submit or QOS limit, or no node free to start it) was never queued, so the reader submits it again after _READER_SRUN_RETRY_S seconds, doubling up to _READER_SRUN_RETRY_MAX_S, with a progress line each time, for as long as step.queue_wait_s allows. Any other srun failure, or a refusal past that wait, falls through to the cancellation below as before.

nemo_gym.orchestration.executors.slurm.reader._remote_managed_evidence(
job_dir: pathlib.Path,

Use the saved pinned install over the existing transport; never import Gym into plain host Python.

nemo_gym.orchestration.executors.slurm.reader._remote_managed_reader_script() -> str

The reader with the caller’s metadata helper defined after its imports, sent as one script.

nemo_gym.orchestration.executors.slurm.reader._remote_reader_bound_s() -> int | None

The explicit reader bound from NEMO_GYM_REMOTE_READER_TIMEOUT_S, clamped to [300, 14400], else None.

Without it the login host sizes the bound from the run’s rollout files (_reader_command).

nemo_gym.orchestration.executors.slurm.reader._remote_reader_error_text(
error: BaseException
) -> str

Name the reader failure with its cause, not only the exception type: the end of the message, where it is.

nemo_gym.orchestration.executors.slurm.reader._report_placement_lines() -> list[str]
nemo_gym.orchestration.executors.slurm.reader._run_reader(
command: list[str],
token: str
) -> str

Run the reader. A signal on the client (raised as KeyboardInterrupt) leaves the remote reader running, since ssh bash -s has no terminal to hang up, so it is stopped by its token before the error unwinds.

A connection that holds_input keeps the remote shell’s stdin open while the reader runs, and the command’s watcher (_reader_command(..., watchdog=True)) stops the reader when that stdin ends.

nemo_gym.orchestration.executors.slurm.reader._watch_lines(
cancel: str | None = None,
progress: list[str] | None = None
) -> list[str]

Bash that waits for the background reader $_gym_reader, started in its own session, and sets _gym_rc.

A watcher in its own session reads the shell’s stdin through FD 4 and at EOF kills the reader’s process group: TERM, then KILL after 10 s. With cancel, a quoted Slurm job name, it cancels that job before the TERM: the step runs on another host, and srun may exit at the TERM, which ends the wait and stops the watcher. progress starts a loop that sets $_gym_progress; it is stopped after the wait.

nemo_gym.orchestration.executors.slurm.reader.collector_command(
collect: str,
job_dir: pathlib.Path,
token: str
) -> list[str]

The collector in a reader step, else on the login host; its output keeps only the collector’s own lines after the placement report is taken off (reader_placement, without_placement). Its exit code is the collector’s.

nemo_gym.orchestration.executors.slurm.reader.reader_placement(
output: str
) -> tuple[str | None, str | None]

Where a reader ran (“login” or “cpu:<job id>”) and why it fell back to the login host, from its report; both None for a reader that ran without a reader step.

nemo_gym.orchestration.executors.slurm.reader.reader_step() -> nemo_gym.orchestration.executors.slurm.reader.ReaderStep | None

The reader step from NEMO_GYM_REMOTE_READER_SRUN (extra srun arguments, for example -p cpu -c 4), NEMO_GYM_REMOTE_READER_SRUN_WAIT_S (default 120), NEMO_GYM_REMOTE_READER_MAX (default 8), NEMO_GYM_REMOTE_READER_QUEUE_WAIT_S (default 600; 0 does not wait) and NEMO_GYM_REMOTE_READER_LOGIN_FALLBACK (allow; anything else forbids it); None when the arguments are unset or empty.

nemo_gym.orchestration.executors.slurm.reader.reader_token() -> str

The token this process’s login-host Store readers carry in their environment as GYM_READER_TOKEN: the caller’s own value when it sets one, else one generated uuid for the process. Recovery reports it as reader_token, so whoever stops a recovery can find its reader.

nemo_gym.orchestration.executors.slurm.reader.without_placement(
output: str
) -> str

The output without the reader placement report.

nemo_gym.orchestration.executors.slurm.reader.COLLECTOR_NOT_RUN = 'the evidence collector could not run in its Slurm step and the login-host fallb...
nemo_gym.orchestration.executors.slurm.reader.READER_LOGIN_FALLBACK_ENV = 'NEMO_GYM_REMOTE_READER_LOGIN_FALLBACK'
nemo_gym.orchestration.executors.slurm.reader.READER_MAX_ENV = 'NEMO_GYM_REMOTE_READER_MAX'
nemo_gym.orchestration.executors.slurm.reader.READER_NOT_RUN_HOST = 'none'
nemo_gym.orchestration.executors.slurm.reader.READER_QUEUE_WAIT_ENV = 'NEMO_GYM_REMOTE_READER_QUEUE_WAIT_S'
nemo_gym.orchestration.executors.slurm.reader.READER_SRUN_ENV = 'NEMO_GYM_REMOTE_READER_SRUN'
nemo_gym.orchestration.executors.slurm.reader.READER_SRUN_WAIT_ENV = 'NEMO_GYM_REMOTE_READER_SRUN_WAIT_S'
nemo_gym.orchestration.executors.slurm.reader.READER_TOKEN_ENV = 'GYM_READER_TOKEN'
nemo_gym.orchestration.executors.slurm.reader._KILL_READER_SCRIPT = '_gym_token=@TOKEN@\n_gym_matches() {\n local proc pid environ cmdline\n f...
nemo_gym.orchestration.executors.slurm.reader._READER_CANCEL_WAIT_S = 60
nemo_gym.orchestration.executors.slurm.reader._READER_CEILING_S = 14400
nemo_gym.orchestration.executors.slurm.reader._READER_FALLBACK_PREFIX = '__GYM_READER_FALLBACK:'
nemo_gym.orchestration.executors.slurm.reader._READER_FLOOR_S = 900
nemo_gym.orchestration.executors.slurm.reader._READER_HOST_PREFIX = '__GYM_READER_HOST:'
nemo_gym.orchestration.executors.slurm.reader._READER_PROGRESS_INTERVAL_S = 60
nemo_gym.orchestration.executors.slurm.reader._READER_PROGRESS_SCRIPT = 'while sleep "$2" >/dev/null 2>&1; kill -0 "$1" 2>/dev/null; do\n b=0\n fo...
nemo_gym.orchestration.executors.slurm.reader._READER_QUEUE_POLL_S = 10
nemo_gym.orchestration.executors.slurm.reader._READER_QUEUE_WAIT_DEFAULT_S = 600
nemo_gym.orchestration.executors.slurm.reader._READER_RETRY_BACKOFF_S = 10
nemo_gym.orchestration.executors.slurm.reader._READER_SRUN_RETRY_MAX_S = 60
nemo_gym.orchestration.executors.slurm.reader._READER_SRUN_RETRY_S = 15
nemo_gym.orchestration.executors.slurm.reader._READER_S_PER_GB = 100
nemo_gym.orchestration.executors.slurm.reader._READER_UV_LOCK_TIMEOUT_DEFAULT_S = 1800
nemo_gym.orchestration.executors.slurm.reader._REMOTE_MANAGED_READER = '\nimport json, subprocess, sys\nfrom pathlib import Path\ntry:\n import nemo...
nemo_gym.orchestration.executors.slurm.reader._SRUN_REFUSAL = 'Job violates accounting/QOS policy|Requested nodes are busy|Immediate execution...
nemo_gym.orchestration.executors.slurm.reader._reader_token: str | None = None