Skip to navigation

nemo_gym.eval.recovery.markers

View as MarkdownOpen in Claude

The chain markers: the strings the auto-resume chain scripts write into a job directory and the recovery evidence reader parses back.

They are the whole contract between the chain (nemo_gym.orchestration.executors.slurm.resume_script, which renders the scripts) and the recovery planner (nemo_gym.eval.recovery), so they live below both: the chain imports them from here, and the planner never imports the chain.

Module Contents

Functions

NameDescription
record_failure_kindLeave the registered kind error carries in the failure file the driver step names, for the next guard.

Data

ACCOUNTED_WALLTIME_PREFIX

ACCOUNTING_UNKNOWN_PREFIX

ACCUMULATED_WALLTIME_NAME

CANCELLED_FOLLOWUP_MARKER

CHAIN_ACCOUNTING_ENV

CONTINUATION_REASON_PREFIX

DRIVER_FAILURE_ENV

FOLLOW_UP_SUBMISSION_REFUSED

GUARDED_DRIVER_FAILURE_KINDS

NO_ELIGIBLE_RETRY_REASON

PROVISIONAL_MARKER_PREFIX

REQUEUE_RECONCILE_STATE

REQUEUE_UNKNOWN_REASON

RETRY_EXTENSION_NAME

RETRY_FILE_NAME

logger

API

nemo_gym.eval.recovery.markers.record_failure_kind(
error: BaseException
) -> None

Leave the registered kind error carries in the failure file the driver step names, for the next guard.

The driver step of a resumable job names the file in DRIVER_FAILURE_ENV; elsewhere this does nothing.

nemo_gym.eval.recovery.markers.ACCOUNTED_WALLTIME_PREFIX = '__GYM_ACCOUNTED_WALLTIME:'
nemo_gym.eval.recovery.markers.ACCOUNTING_UNKNOWN_PREFIX = 'GYM_ACCOUNTING_UNKNOWN:'
nemo_gym.eval.recovery.markers.ACCUMULATED_WALLTIME_NAME = '.gym_accumulated_walltime'
nemo_gym.eval.recovery.markers.CANCELLED_FOLLOWUP_MARKER = 'was cancelled before it started'
nemo_gym.eval.recovery.markers.CHAIN_ACCOUNTING_ENV = 'GYM_CHAIN_ACCOUNTING'
nemo_gym.eval.recovery.markers.CONTINUATION_REASON_PREFIX = '__GYM_CONTINUATION_REASON:'
nemo_gym.eval.recovery.markers.DRIVER_FAILURE_ENV = 'GYM_DRIVER_FAILURE_FILE'
nemo_gym.eval.recovery.markers.FOLLOW_UP_SUBMISSION_REFUSED = 'follow_up_submission_refused'
nemo_gym.eval.recovery.markers.GUARDED_DRIVER_FAILURE_KINDS = (SERVER_SPINUP_TIMEOUT, DRIVER_INSTALL_FAILED, COLLECTION_RESUME_REFUSED, PROVID...
nemo_gym.eval.recovery.markers.NO_ELIGIBLE_RETRY_REASON = 'No tasks are eligible for retry: the job ended its chain itself, with nothing q...
nemo_gym.eval.recovery.markers.PROVISIONAL_MARKER_PREFIX = 'Continuation '
nemo_gym.eval.recovery.markers.REQUEUE_RECONCILE_STATE = 'REQUEUED'
nemo_gym.eval.recovery.markers.REQUEUE_UNKNOWN_REASON = 'Requeued allocation elapsed time is unknown; recover only after complete termin...
nemo_gym.eval.recovery.markers.RETRY_EXTENSION_NAME = '.gym_infra_retry_extension'
nemo_gym.eval.recovery.markers.RETRY_FILE_NAME = '.gym_infra_retries'
nemo_gym.eval.recovery.markers.logger = logging.getLogger(__name__)