nemo_gym.orchestration.executors.slurm.reader
nemo_gym.orchestration.executors.slurm.reader
The remote Store reader: runs the evidence collector and the run’s own Store selection on the cluster, in a Slurm step when one is configured, else on the login host, and bounds, watches and stops it.
Module Contents
Classes
Functions
Data
_READER_UV_LOCK_TIMEOUT_DEFAULT_S
API
How a heavy reader runs in a Slurm step: the caller’s srun arguments (each quoted), how long the step may take to start, how many reader steps the user may have running or pending at once, how long a reader waits for one of them to end when they are at that cap, and whether a reader that cannot run in its step may run on the login host instead.
Bases: KeyboardInterrupt
A signal stopped recovery while the login-host Store reader ran; the reader was stopped by its token.
A KeyboardInterrupt, so the reader’s ordinary error handling does not turn it into evidence and the open
connection unwinds and closes its master.
A nonnegative number of seconds from the environment; the default for an unset or invalid value.
Stop this user’s login-host processes whose environment or command line carries token: TERM, then KILL
for any still running after 10 s. It prints __GYM_READER_KILL:term=<n> kill=<n>.
The login-host steps: size the bound from the rollout files, run the reader, report its exit code and stderr.
The reader’s own failure is reported on stdout (__GYM_READER_* lines) so a timeout, a MemoryError or a
failed bootstrap reaches the decision with its exit code and the end of its stderr; only SSH itself fails the
command. With token, the reader runs with it as GYM_READER_TOKEN in its environment. With step, it
runs in a Slurm step when it can (_reader_step_lines) and reports where it ran.
With watchdog, for a connection that holds the shell’s stdin open while it runs: the reader runs in its own
session (setsid), and a watcher in its own session waits for EOF on the shell’s stdin, which comes when the
client process or its SSH connection ends, then kills the reader’s process group (TERM, KILL after 10 s). It is
an EOF loop, not a hangup: without a terminal nothing sends SIGHUP. The steps run as one function that exits the
shell, so the watcher never reads a line of the script itself. In a step, the watched process is srun, and the
watcher also cancels the step by name. On the login host, a progress loop in its own session prints
gym-reader progress: pass=<n> bytes=<b> elapsed_s=<s> on stderr every progress_interval_s (60 s by
default) while the reader runs; a reader in a step has none. Stdout and its __GYM_READER_* lines are
unchanged.
_reader_command with the reader in a Slurm step: the payload is saved once, sent on the step’s stdin, and
read again from the file by a login-host reader that runs instead. With watchdog, srun and that login-host
reader each run under the EOF watcher, as the reader of _reader_command(..., watchdog=True) does.
The settings a remote reader takes from its caller: the reader’s command starts from a cleared environment on another host, so without this a cold install there waits uv’s own default for the lock and gives up early.
The reader failure named from its report with its exit code and stderr, and whether it reached its deadline (the only failure retried with twice the bound); (None, False) when it exited 0.
Bash that runs script in a Slurm step, else leaves it to the login host.
It sets _gym_host (“login” or “cpu:<job id>”) and _gym_fallback (why the reader stays on the login host,
empty when it ran in the step); after a step that ran, _gym_rc is its exit code and $_gym_step_out holds
its output. Inside the step the input on stdin becomes FD 3, because srun forwards stdin but no other descriptor,
and script reads the reader bound as $1. The step is named gym-reader-<token>, and its time limit is the
reader bound plus 60 s, so Slurm ends an orphan. A step that has not started after the wait is cancelled by name,
and the reader runs on the login host. With watchdog (FD 4 is the shell’s stdin), srun runs under the EOF
watcher (_watch_lines), which also cancels the step by name. Without step.login_fallback, a reader that
would fall back runs nowhere: _gym_host is none, _gym_rc 1, and _gym_fallback says why.
When the user’s reader steps are at step.max_readers, the reader queues: it counts them again every
_READER_QUEUE_POLL_S seconds for up to step.queue_wait_s, saying on stderr how many it is queued behind
(a progress line), and runs in its step once one ends. Only a cap still reached after that wait keeps it out.
A step srun refused outright (_SRUN_REFUSAL: a submit or QOS limit, or no node free to start it) was never
queued, so the reader submits it again after _READER_SRUN_RETRY_S seconds, doubling up to
_READER_SRUN_RETRY_MAX_S, with a progress line each time, for as long as step.queue_wait_s allows. Any
other srun failure, or a refusal past that wait, falls through to the cancellation below as before.
Use the saved pinned install over the existing transport; never import Gym into plain host Python.
The reader with the caller’s metadata helper defined after its imports, sent as one script.
The explicit reader bound from NEMO_GYM_REMOTE_READER_TIMEOUT_S, clamped to [300, 14400], else None.
Without it the login host sizes the bound from the run’s rollout files (_reader_command).
Name the reader failure with its cause, not only the exception type: the end of the message, where it is.
Run the reader. A signal on the client (raised as KeyboardInterrupt) leaves the remote reader running,
since ssh bash -s has no terminal to hang up, so it is stopped by its token before the error unwinds.
A connection that holds_input keeps the remote shell’s stdin open while the reader runs, and the command’s
watcher (_reader_command(..., watchdog=True)) stops the reader when that stdin ends.
Bash that waits for the background reader $_gym_reader, started in its own session, and sets _gym_rc.
A watcher in its own session reads the shell’s stdin through FD 4 and at EOF kills the reader’s process group:
TERM, then KILL after 10 s. With cancel, a quoted Slurm job name, it cancels that job before the TERM: the step
runs on another host, and srun may exit at the TERM, which ends the wait and stops the watcher. progress
starts a loop that sets $_gym_progress; it is stopped after the wait.
The collector in a reader step, else on the login host; its output keeps only the collector’s own lines after
the placement report is taken off (reader_placement, without_placement). Its exit code is the
collector’s.
Where a reader ran (“login” or “cpu:<job id>”) and why it fell back to the login host, from its report; both None for a reader that ran without a reader step.
The reader step from NEMO_GYM_REMOTE_READER_SRUN (extra srun arguments, for example -p cpu -c 4),
NEMO_GYM_REMOTE_READER_SRUN_WAIT_S (default 120), NEMO_GYM_REMOTE_READER_MAX (default 8),
NEMO_GYM_REMOTE_READER_QUEUE_WAIT_S (default 600; 0 does not wait) and
NEMO_GYM_REMOTE_READER_LOGIN_FALLBACK (allow; anything else forbids it); None when the arguments are unset
or empty.
The token this process’s login-host Store readers carry in their environment as GYM_READER_TOKEN: the
caller’s own value when it sets one, else one generated uuid for the process. Recovery reports it as
reader_token, so whoever stops a recovery can find its reader.
The output without the reader placement report.