Skip to navigation

nemo_gym.orchestration.executors.resume_script

View as MarkdownOpen in Claude

The auto-resume prologue: a self-chaining Slurm job.

Rendered at the top of a benchmark’s sbatch script, ahead of every service/driver command, when BenchmarkRunConfig.resumable is set. Each job in the chain inspects the PREVIOUS job (passed as $1) via sacct, decides whether to continue, and unconditionally queues its own successor with an afternotok dependency before doing any real work, and a sentinel that continues the chain if that successor dies before its prologue. A job that COMPLETEs leaves its queued successor with a dependency that can never be satisfied, so Slurm would cancel it on its own, but the sentinel would then start on a full allocation only to exit. So the epilogue, rendered after the driver step, cancels both when the job finished the run.

Module Contents

Functions

NameDescription
_walltime_to_secondsConvert a Slurm-style duration string to seconds.
fresh_restart_commandThe command that queues a run with no task inventory again, as a fresh run with a longer walltime.
record_failure_kindLeave the registered kind error carries in the failure file the driver step names, for the next guard.
render_resume_epilogueRender the step after the driver of a resumable job: a job that finished the run ends its own chain.
render_resume_prologueRender the auto-resume prologue for one benchmark’s sbatch script.

Data

ACCOUNTED_WALLTIME_PREFIX

ACCOUNTING_UNKNOWN_PREFIX

ALLOCATION_ACCOUNTING_MARKER

BOUNDED_BUDGET_MARKER

CANCELLED_FOLLOWUP_MARKER

CHAIN_ACCOUNTING_ENV

CHAIN_ACCOUNTING_FORMAT

CHAIN_END_JOURNAL_NAME

CONTINUATION_REASON_PREFIX

CREDENTIALS_REJECTED_REASON

DRIVER_FAILURE_NAME

EXPORT_RECEIPT_MISSING

EXPORT_RECEIPT_MISSING_REASON

FOLLOW_UP_SUBMISSION_REFUSED

FRESH_RESTART_MARKER_NAME

GUARDED_DRIVER_FAILURE_KINDS

INCOMPLETE_MARKER_NAME

INSTALL_ERROR_MAX_CHARS

JOB_ALLOCATIONS_BUDGET_S

JOB_ALLOCATIONS_FORMAT

JOB_ALLOCATIONS_LOCK_NAME

JOB_ALLOCATIONS_NAME

JOB_ALLOCATIONS_RETRY_S

JOB_CHAIN_NAME

KEEP_BUDGET_ENV

NO_ELIGIBLE_RETRY

NO_ELIGIBLE_RETRY_EXIT

NO_ELIGIBLE_RETRY_REASON

NO_PROGRESS_SEGMENT_REASON

PREPARE_FAILED_REASON

PRE_INVENTORY_REASON

PROLOGUE_MARKER

PROTOCOL_VIOLATION_REASON

PROVISIONAL_MARKER_PREFIX

QUOTA_EXHAUSTED_REASON

REQUEUE_RECONCILED_NOTE

REQUEUE_RECONCILE_STATE

REQUEUE_SETTLE_BOUND_S

REQUEUE_SETTLE_POLL_S

REQUEUE_UNKNOWN_REASON

RESUBMIT_ENV

RESUBMIT_LOCK_NAME

RESUME_REFUSED_REASON

RETRY_EXTENSION_MARKER

RETRY_EXTENSION_NAME

RETRY_FILE_NAME

ROWS_AT_START_NAME

RUNTIME_ADMISSION_MARKER

SCHEDULER_CAUSE_MARKER

SPINUP_TIMEOUT_REASON

_EPILOGUE_TEMPLATE

_JOB_SCRIPT_NAME

_MAX_WALLTIME_CHECK_TEMPLATE

_PROLOGUE_TEMPLATE

_RECEIPT_ARTIFACTS

logger

API

nemo_gym.orchestration.executors.resume_script._walltime_to_seconds(
duration: str
) -> int

Convert a Slurm-style duration string to seconds.

Covers the subset of Slurm’s --time grammar that max_walltime accepts: MM, MM:SS, HH:MM:SS, D-HH, D-HH:MM, D-HH:MM:SS — the same spellings Slurm itself takes, so a value can be copied straight from compute.walltime.

nemo_gym.orchestration.executors.resume_script.fresh_restart_command(
gym_job_id: str,
benchmark: str
) -> str

The command that queues a run with no task inventory again, as a fresh run with a longer walltime.

Submission ids and benchmark names are shell-safe by construction (see new_gym_job_id and the executor’s benchmark name check), so the command needs no quoting, and the prologue renders it with shell expansions of both.

nemo_gym.orchestration.executors.resume_script.record_failure_kind(
error: BaseException
) -> None

Leave the registered kind error carries in the failure file the driver step names, for the next guard.

The driver step of a resumable job names the file in DRIVER_FAILURE_ENV; elsewhere this does nothing.

nemo_gym.orchestration.executors.resume_script.render_resume_epilogue() -> str

Render the step after the driver of a resumable job: a job that finished the run ends its own chain.

When the driver step succeeded and the run is complete, it writes the record the follow-up or its sentinel would have written, cancels the sentinel and then the follow-up it queued, and journals both in CHAIN_END_JOURNAL_NAME, so no allocation starts for a finished run. A driver that exits NO_ELIGIBLE_RETRY_EXIT (nothing is eligible for another attempt) ends its chain the same way, with the NO_ELIGIBLE_RETRY_REASON stop, unless it left in DRIVER_FAILURE_NAME a failure kind the follow-up’s prologue turns into its own stop (spin-up timeout, install failed, resume refused, credentials rejected). The job keeps the driver step’s status.

nemo_gym.orchestration.executors.resume_script.render_resume_prologue(
admission_command: str = 'false'
) -> str

Render the auto-resume prologue for one benchmark’s sbatch script.

nemo_gym.orchestration.executors.resume_script.ACCOUNTED_WALLTIME_PREFIX = '__GYM_ACCOUNTED_WALLTIME:'
nemo_gym.orchestration.executors.resume_script.ACCOUNTING_UNKNOWN_PREFIX = 'GYM_ACCOUNTING_UNKNOWN:'
nemo_gym.orchestration.executors.resume_script.ALLOCATION_ACCOUNTING_MARKER = '# --- Durable allocation accounting v1 ---'
nemo_gym.orchestration.executors.resume_script.BOUNDED_BUDGET_MARKER = '# --- Bounded resume budgets v1 ---'
nemo_gym.orchestration.executors.resume_script.CANCELLED_FOLLOWUP_MARKER = 'was cancelled before it started'
nemo_gym.orchestration.executors.resume_script.CHAIN_ACCOUNTING_ENV = 'GYM_CHAIN_ACCOUNTING'
nemo_gym.orchestration.executors.resume_script.CHAIN_ACCOUNTING_FORMAT = 'JobIDRaw,State,ElapsedRaw,Restarts'
nemo_gym.orchestration.executors.resume_script.CHAIN_END_JOURNAL_NAME = 'chain-end-journal.jsonl'
nemo_gym.orchestration.executors.resume_script.CONTINUATION_REASON_PREFIX = '__GYM_CONTINUATION_REASON:'
nemo_gym.orchestration.executors.resume_script.CREDENTIALS_REJECTED_REASON = f'{PROVIDER_CREDENTIALS_REJECTED}: the provider refused the credentials; the cha...
nemo_gym.orchestration.executors.resume_script.DRIVER_FAILURE_NAME = '.gym_driver_failure'
nemo_gym.orchestration.executors.resume_script.EXPORT_RECEIPT_MISSING = 'run_complete_export_receipt_missing'
nemo_gym.orchestration.executors.resume_script.EXPORT_RECEIPT_MISSING_REASON = f'{EXPORT_RECEIPT_MISSING}: the run is complete, but its MLflow export receipt (...
nemo_gym.orchestration.executors.resume_script.FOLLOW_UP_SUBMISSION_REFUSED = 'follow_up_submission_refused'
nemo_gym.orchestration.executors.resume_script.FRESH_RESTART_MARKER_NAME = '.gym_fresh_restart'
nemo_gym.orchestration.executors.resume_script.GUARDED_DRIVER_FAILURE_KINDS = (SERVER_SPINUP_TIMEOUT, DRIVER_INSTALL_FAILED, COLLECTION_RESUME_REFUSED, PROVID...
nemo_gym.orchestration.executors.resume_script.INCOMPLETE_MARKER_NAME = '.gym_chain_incomplete'
nemo_gym.orchestration.executors.resume_script.INSTALL_ERROR_MAX_CHARS = 200
nemo_gym.orchestration.executors.resume_script.JOB_ALLOCATIONS_BUDGET_S = 30
nemo_gym.orchestration.executors.resume_script.JOB_ALLOCATIONS_FORMAT = 'JobIDRaw,State,Start,End,ElapsedRaw,NNodes,NodeList,AllocTRES,Restarts'
nemo_gym.orchestration.executors.resume_script.JOB_ALLOCATIONS_LOCK_NAME = '.gym_job_allocations.lock'
nemo_gym.orchestration.executors.resume_script.JOB_ALLOCATIONS_NAME = 'job-allocations.jsonl'
nemo_gym.orchestration.executors.resume_script.JOB_ALLOCATIONS_RETRY_S = 5
nemo_gym.orchestration.executors.resume_script.JOB_CHAIN_NAME = '.gym_job_chain'
nemo_gym.orchestration.executors.resume_script.KEEP_BUDGET_ENV = 'GYM_RESUBMIT_KEEP_BUDGET'
nemo_gym.orchestration.executors.resume_script.NO_ELIGIBLE_RETRY = 'no_eligible_retry'
nemo_gym.orchestration.executors.resume_script.NO_ELIGIBLE_RETRY_EXIT = 76
nemo_gym.orchestration.executors.resume_script.NO_ELIGIBLE_RETRY_REASON = 'No tasks are eligible for retry: the job ended its chain itself, with nothing q...
nemo_gym.orchestration.executors.resume_script.NO_PROGRESS_SEGMENT_REASON = "the segment saved no rollout: its walltime is shorter than this run's setup (Gy...
nemo_gym.orchestration.executors.resume_script.PREPARE_FAILED_REASON = f'{ENVIRONMENT_PREPARE_FAILED}: EVAL STOPPED: data preparation failed; the chain...
nemo_gym.orchestration.executors.resume_script.PRE_INVENTORY_REASON = 'the segment ended before the task inventory was written; nothing can be resumed...
nemo_gym.orchestration.executors.resume_script.PROLOGUE_MARKER = '# --- Auto-resume chain ---'
nemo_gym.orchestration.executors.resume_script.PROTOCOL_VIOLATION_REASON = f'{ENVIRONMENT_PROTOCOL_VIOLATION}: an environment server answered with a reply ...
nemo_gym.orchestration.executors.resume_script.PROVISIONAL_MARKER_PREFIX = 'Continuation '
nemo_gym.orchestration.executors.resume_script.QUOTA_EXHAUSTED_REASON = f'{PROVIDER_QUOTA_EXHAUSTED}: the grading provider's quota is exhausted; the cha...
nemo_gym.orchestration.executors.resume_script.REQUEUE_RECONCILED_NOTE = 'Requeued allocation accounting was reconciled from terminal scheduler records; ...
nemo_gym.orchestration.executors.resume_script.REQUEUE_RECONCILE_STATE = 'REQUEUED'
nemo_gym.orchestration.executors.resume_script.REQUEUE_SETTLE_BOUND_S = 300
nemo_gym.orchestration.executors.resume_script.REQUEUE_SETTLE_POLL_S = 30
nemo_gym.orchestration.executors.resume_script.REQUEUE_UNKNOWN_REASON = 'Requeued allocation elapsed time is unknown; recover only after complete termin...
nemo_gym.orchestration.executors.resume_script.RESUBMIT_ENV = 'GYM_RESUBMIT'
nemo_gym.orchestration.executors.resume_script.RESUBMIT_LOCK_NAME = '.gym_resubmit.lock'
nemo_gym.orchestration.executors.resume_script.RESUME_REFUSED_REASON = f'{COLLECTION_RESUME_REFUSED}: the saved rollouts do not verify against this run...
nemo_gym.orchestration.executors.resume_script.RETRY_EXTENSION_MARKER = '# --- Recorded retry extension v1 ---'
nemo_gym.orchestration.executors.resume_script.RETRY_EXTENSION_NAME = '.gym_infra_retry_extension'
nemo_gym.orchestration.executors.resume_script.RETRY_FILE_NAME = '.gym_infra_retries'
nemo_gym.orchestration.executors.resume_script.ROWS_AT_START_NAME = '.gym_rows_at_start'
nemo_gym.orchestration.executors.resume_script.RUNTIME_ADMISSION_MARKER = '# --- Installed-runtime continuation admission v1 ---'
nemo_gym.orchestration.executors.resume_script.SCHEDULER_CAUSE_MARKER = '# --- Scheduler-cause retry admission v1 ---'
nemo_gym.orchestration.executors.resume_script.SPINUP_TIMEOUT_REASON = f'{SERVER_SPINUP_TIMEOUT}: the Gym servers did not become ready within server_sp...
nemo_gym.orchestration.executors.resume_script._EPILOGUE_TEMPLATE = '# --- A job that finishes the run ends its own chain ---\n_gym_driver_rc=$?\n_g...
nemo_gym.orchestration.executors.resume_script._JOB_SCRIPT_NAME = 'job.sh'
nemo_gym.orchestration.executors.resume_script._MAX_WALLTIME_CHECK_TEMPLATE = 'if (( _gym_accumulated >= {seconds} )); then\n _gym_stop_incomplete "Accumul...
nemo_gym.orchestration.executors.resume_script._PROLOGUE_TEMPLATE = '# --- Auto-resume chain ---\nOUTPUT_DIR="$PWD"\n_this_script="$OUTPUT_DIR/{job_...
nemo_gym.orchestration.executors.resume_script._RECEIPT_ARTIFACTS = (('rollouts_jsonl_fpath', 'rollouts.jsonl'), ('resolved_config_yaml_fpath', 'rol...
nemo_gym.orchestration.executors.resume_script.logger = logging.getLogger(__name__)