nemo_gym.orchestration.executors.slurm.chain
nemo_gym.orchestration.executors.slurm.chain
Recovery of a run’s auto-resume chain: its evidence and decision, and the resubmit that continues it with the journaled extensions, raises, repins and operator continuations it records.
Module Contents
Classes
Functions
Data
API
The recovery half of SlurmExecutor: read a run’s saved evidence, decide, and continue its chain.
Mixed into SlurmExecutor, whose _saved_config and installed_commit its methods use.
The locked steps that record extend_attempts more attempts for every rollout, restored unless queued.
The Store reads the record (artifacts/rollouts_attempt_extension) as an addition to the run’s saved
max_rollout_attempts; a rollout that ended with a model failure is never run again.
The locked steps that record extend_retries more infra retries, restored unless the job is queued.
The chain marker a fresh restart writes: why, the recorded walltime and the new one.
Ask the scheduler about the run’s judge-only jobs that have no receipt (CollectedEvidence.judge_jobs,
state queued), set each one’s state and scheduler_state from its record, and name the newest live
one as judge-only job <id> (STATE); None when none is live, or when the query failed or knows none.
The locked steps of an operator continuation: fill the counters it names, replace the unknown marker and journal the continuation with its epoch before sbatch, so a queued continuation always has its epoch.
Each file is kept aside first and put back unless the job is queued. A readable counter is never named, except
the retry extension an explicit --extend-retries raises when the spent count is readable.
The locked shell steps that render one benchmark’s job.sh and saved config again, with a repin, a walltime
raise or both, and the journal entries to append once its job is queued. A fresh restart raises the walltime
above the recorded one without keep_budget.
The locked steps that journal the approval to run masked results again, taken back unless the job is queued.
The line goes to the rerun-masked journal beside the rollouts file, where the Store reads it when the job starts: its time, user, reason and each approved identity with the attempt it covers and the attempts counted for it.
The decision of recovery_evidence without a score key.
Collect the job dir’s identities, failures, quality summary, named score and chain budget over SSH.
reader_ref runs the Store reader at that full commit instead of the saved one: a repinned run resumes there.
max_walltime is a walltime raise the resubmit will render: the plan checks the chain cap it renders.
server_spinup_timeout_seconds (with spinup_timeout_reason) is a spin-up limit raise the resubmit
will render: the decision names it.
rerun_masked_without_artifact plans the masked results that kept a saved answer but have no gradeable
artifact and no grade as retries; the resubmit journals them (see resubmit).
operator_budget lets a run whose chain accounting is unknown continue (see decide_from_collected); with
it, a failed scheduler query is a stop that names it instead of an error.
Queue benchmark’s job.sh again from its job dir and record the new job id.
With keep_budget (--recover) the job keeps the spent retry count, walltime and per-rollout attempts;
a job.sh rendered before that option would reset them, so it is refused. accounting_snapshot is the
collected evidence’s accounting_snapshot; it is required with keep_budget and rechecked under the
resubmit lock before anything in the job dir changes, so a decision made on stale accounting is refused.
accounting_reconciled (with keep_budget) admits the requeue marker that the collected evidence
reconciled from terminal scheduler records; it is replaced by a note just before the job is queued.
repin (with keep_budget and repin_reason) continues the run on that full Gym commit, which must be
this launcher’s: under the lock, after the live-job and snapshot checks, job.sh and resolved-config.yaml are
rendered again with the new ref (and the MLflow pin tags that named the old one), the old files are kept
beside them, and both are restored if the job is not queued. Rows, attempts and budget files stay as they are;
the queued job’s record carries the new commit and the run dir’s repin journal records the move.
extend_retries (with keep_budget) adds that many infra retries to the recorded extension that the
job’s prologue adds to the configured cap; it is written just before sbatch, restored if the job is not
queued, and journaled once queued. The spent retry count is never reset.
max_walltime (with keep_budget and walltime_reason) raises the Slurm walltime the same way a repin
moves the commit, in the same render: compute.<cluster>.walltime becomes it, the chain cap becomes at least
it, and a value below the recorded walltime is refused. The run dir’s walltime journal records the raise.
segment_walltime (from walltime_plan) is what each Slurm segment requests instead, when the partition’s
MaxTime is below the raise: the raise stays the chain cap.
rerun_masked (with keep_budget) is the operator’s approval to run again the masked results that kept a
saved answer but have no gradeable artifact and no grade: its reason and identities (identity, attempt
id and attempts counted) are appended as one line to the run’s rerun-masked journal in artifacts/, which the
Store reads. The line is written just before sbatch and taken back if the job is not queued.
A job the scheduler refuses raises SchedulerRefusedError with sbatch’s text and the restored job dir.
unsafe_resume_reason (with keep_budget) renders the job with allow_unsafe_resume the same way, so
its resume proceeds past a refused run identity check; the run dir’s unsafe-resume journal records the reason
and the user once the job is queued.
server_spinup_timeout_seconds (with spinup_timeout_reason) renders the job with that server
spin-up limit the same way; it only raises the recorded limit (0 removes it), and the job dir’s
spin-up timeout journal records both limits, the reason and the user once the job is queued.
operator_continuation (with keep_budget) is a decision’s OPERATOR-CONTINUATION record for a run whose
accounting is unknown: under the lock it fills the counters the record names, replaces the unknown-accounting
marker with a note and keeps the readable counters; each step is undone unless the job is queued, and the job
dir’s operator journal records the queued continuation.
fresh_restart (with max_walltime, walltime_reason and accounting_snapshot, without
keep_budget) queues a run that has no task inventory again as a fresh run: a fresh retry and walltime
budget, a walltime longer than the recorded one, the snapshot rechecked under the lock. Its walltime journal
entry, the receipt, carries plan FRESH-RESTART and why; .gym_chain_incomplete names the restart
and both walltimes until the queued job continues the chain.
Refuses while any of the caller’s jobs runs in that job dir, so two jobs never write one run, and
refuses a run with rollouts whose gym eval run job.sh predates resubmit: it would start the run
fresh and delete them. A command benchmark’s job.sh runs its own command, so it is not refused.
Check a --max-walltime raise against the partition’s MaxTime, read with sinfo on the cluster, before
any evidence is read (see _plan_walltime). A failed query keeps the whole raise per segment and says so.
Bases: RuntimeError
A job script would be rendered again by a launcher of another Gym revision than the one the job runs.
The launcher’s job.sh templates may need modules the pinned revision does not have, and the capability checks run later inspect only the newly rendered script, so the mismatch is refused before anything changes.
The used walltime of a chain after an operator continuation: its baseline plus every allocation since.
The continuation filled or kept the counter as the baseline, so the allocations before it are not accounted again. None when the records since then are incomplete or the counter does not match them.
How the raise max_walltime fits the partition limit: each segment requests the smaller of the raise
and the partition’s MaxTime, and the raise is the chain cap the segments continue under. Refused when no segment
fits: a run that does not chain segments needs the whole raise in one job, and a chained one at least its
recorded segment walltime.
The chain cap after a raise to walltime: at least the raise, and the raise itself when there was none.
The benchmark’s server spin-up limit in seconds as the run starts its servers; 0 or less is no limit.
config with the driver’s Gym install at ref, and the benchmark’s MLflow pin tags that named the old one.
config with the benchmark’s server spin-up limit seconds; every other field as saved.
Why seconds is not a raise of the recorded limit (0 is no limit), or None when it is.
The sentence a decision carries for a spin-up timeout raise.
config with the benchmark’s job resuming past a refused run identity check; every other field as saved.
config with a chain cap of at least walltime and each Slurm segment requesting segment (by default
walltime itself); every other field as saved. A chain without a cap takes walltime as its cap: a raise
always bounds the total.