Skip to navigation

nemo_gym.eval.recovery.decision

View as MarkdownOpen in Claude

The recovery decision: identity states, run-level actions, and the decision and outcome records.

Module Contents

Classes

NameDescription
IdentityEvidence-
IdentityState-
RecoveryAction-
RecoveryDecision-
RecoveryOutcomeWhat recover did: dispatched names the one scheduler or judge action taken, None when nothing ran.
RerunMaskedApprovalOne masked result the operator approves to run again: its identity, the attempts counted for it before the

Functions

NameDescription
_events-
_judge_job_unsettled-
_known_kind-
failed_judge_jobs_blockingThe ids of the last JUDGE_JOB_FAILURE_BOUND judge-only jobs when all ended without settling a saved answer and no
failed_judge_jobs_stopThe stop text of failed_judge_jobs_blocking: the jobs, where to read them, and the explicit way on.

Data

ACKNOWLEDGE_JUDGE_JOBS_FLAGS

ALLOCATION_ENDED_STATES

COLLECTED_CAUSE

COUNTS_AS_ZERO

EXTEND_ATTEMPTS_HINT

FAILURE_CLASS

FAILURE_EVENTS

FAILURE_ID

FRESH_RESTART_REASON

JUDGE_JOB_FAILURE_BOUND

LIVE_JOB_STATES

MAX_ATTEMPTS

MAX_EXTEND_ATTEMPTS

NO_INVENTORY

OPERATOR_CONTINUATION

OPERATOR_CONTINUATION_NOTE

RERUN_MASKED_FLAGS

RERUN_MASKED_REASON

RESERVATION_WITHOUT_OUTCOME

SCORED_UNPARSEABLE_REASON

STORE_DOES_NOT_REPLAY

TERMINAL

UNKNOWN_BUDGET_HINT

UNKNOWN_BUDGET_REASON

UNKNOWN_WALLTIME_REASON

_FAILED_JUDGE_JOBS

API

class nemo_gym.eval.recovery.decision.IdentityEvidence()

Bases: BaseModel

attempts
int = 0
events
list[tuple[str | None, str | None, str | None, str | None, str | None]] = Field(default_factory=list)
failure_id
str | None = None
failure_kind
str | None = None
http_status
int | None = None
identity
Identity
max_attempts
int | None = None
reason
str | None = None
stage
FailureStage | None = None
state
IdentityState
unscored
bool = False
class nemo_gym.eval.recovery.decision.IdentityState

Bases: enum.Enum

EXHAUSTED
= 'exhausted'
JUDGE_RECOVERABLE
= 'judge_recoverable'
MASKED
= 'masked'
MISSING
= 'missing'
MODEL_FINAL
= 'model_final'
OMITTED
= 'omitted'
RETRYABLE
= 'retryable'
SCORED
= 'scored'
SKIPPED
= 'skipped'
TERMINAL
= 'terminal'
UNKNOWN
= 'unknown'
class nemo_gym.eval.recovery.decision.RecoveryAction

Bases: enum.Enum

FRESH_RESTART
= 'fresh-restart'
NOOP
= 'noop'
RESUBMIT
= 'resubmit'
REVERIFY
= 'reverify'
STOP
= 'stop'
UNKNOWN
= 'unknown'
class nemo_gym.eval.recovery.decision.RecoveryDecision()

Bases: BaseModel

accounting_unknown
Literal['budget', 'walltime'] | None = Field(default=None, exclude=True)
action
RecoveryAction
chain_stop_kind
str | None = Field(default=None, exclude=True)
complete
bool
continuation_refused
Literal['scheduler_unreachable', 'job_live', 'claim_unidentified', 'counters_changed', 'no_retry_cap'] | None = Field(default=None, exclude=True)
counts
dict[str, int]
duplicate_results
list[Identity] = Field(default_factory=list)
evidence
list[IdentityEvidence]
expected
int
identities
dict[str, list[Identity]]
judge_refusal
Literal['cpu_partition_unknown', 'cpu_placement', 'served_placement', 'unsupported_profile', 'refused', 'concurrent', 'other'] | None = Field(default=None, exclude=True)
masked_without_artifact
list[Identity] = Field(default_factory=list)
operator_continuation
dict[str, Any] | None = None
outcome_policy
dict[str, Any] | None = None
reason
str
refused_option
str | None = Field(default=None, exclude=True)
refused_over
RecoveryAction | None = Field(default=None, exclude=True)
requested
list[str] = Field(default_factory=list, exclude=True)
rerun_masked
list[RerunMaskedApproval] = Field(default_factory=list)
reverify_first
list[Identity] = Field(default_factory=list)
scheduler_live
bool = False
target
str
unverified
bool

Whether results of the run were deliberately not verified (judge_skipped), so it is UNVERIFIED.

class nemo_gym.eval.recovery.decision.RecoveryOutcome()

Bases: BaseModel

What recover did: dispatched names the one scheduler or judge action taken, None when nothing ran.

decision
RecoveryDecision
dispatched
str | None
gate_task
dict[str, Any] | None = None
job_id
str | None = None
judge_job
dict[str, Any] | None = None
mlflow
dict[str, Any] | None = None
ok
bool
operator_verdict
dict[str, str] | None = None
pin_plan
dict[str, Any] | None = None
reader_token
str | None = None
reverify
str | None = None
scheduler_refusal
dict[str, Any] | None = None
unsupported
str | None = None
walltime_plan
dict[str, Any] | None = None
class nemo_gym.eval.recovery.decision.RerunMaskedApproval()

Bases: BaseModel

One masked result the operator approves to run again: its identity, the attempts counted for it before the approval, and the attempt id of the masked row the approval covers (a later attempt is never covered).

attempt
int = Field(ge=0)
attempts
int = Field(ge=0)
identity
Identity
nemo_gym.eval.recovery.decision._events(
row: collections.abc.Mapping[str, typing.Any]
) -> list[tuple[str | None, str | None, str | None, str | None, str | None]]
nemo_gym.eval.recovery.decision._judge_job_unsettled(
entry: collections.abc.Mapping[str, typing.Any]
) -> bool
nemo_gym.eval.recovery.decision._known_kind(
kind: typing.Any
) -> bool
nemo_gym.eval.recovery.decision.failed_judge_jobs_blocking(
judge_jobs: collections.abc.Sequence[collections.abc.Mapping[str, typing.Any]],
acknowledgements: collections.abc.Sequence[collections.abc.Mapping[str, typing.Any]] = ()
) -> list[str | None] | None

The ids of the last JUDGE_JOB_FAILURE_BOUND judge-only jobs when all ended without settling a saved answer and no acknowledgement names exactly them; else None. An acknowledgement covers the failed jobs it names, so a later pair of failures (the one more job it let queue, and the one before it) needs an acknowledgement of its own.

nemo_gym.eval.recovery.decision.failed_judge_jobs_stop(
ids: collections.abc.Sequence[str | None],
where: str
) -> str

The stop text of failed_judge_jobs_blocking: the jobs, where to read them, and the explicit way on.

nemo_gym.eval.recovery.decision.ACKNOWLEDGE_JUDGE_JOBS_FLAGS = '--acknowledge-failed-judge-jobs --acknowledge-reason'
nemo_gym.eval.recovery.decision.ALLOCATION_ENDED_STATES = frozenset({'TIMEOUT', 'PREEMPTED', 'NODE_FAIL', 'BOOT_FAIL', 'CANCELLED', 'FAILE...
nemo_gym.eval.recovery.decision.COLLECTED_CAUSE = '_ng_collected_cause'
nemo_gym.eval.recovery.decision.COUNTS_AS_ZERO = NG_FAILURE_COUNTS_AS_ZERO_KEY
nemo_gym.eval.recovery.decision.EXTEND_ATTEMPTS_HINT = 'an exhausted identity whose attempts went to infrastructure failures or allocat...
nemo_gym.eval.recovery.decision.FAILURE_CLASS = NG_FAILURE_CLASS_KEY
nemo_gym.eval.recovery.decision.FAILURE_EVENTS = FailureReportRow.EVENTS
nemo_gym.eval.recovery.decision.FAILURE_ID = FailureReportRow.FAILURE_ID
nemo_gym.eval.recovery.decision.FRESH_RESTART_REASON = f'nothing to recover: {NO_INVENTORY}; resubmit it as a fresh run with a longer w...
nemo_gym.eval.recovery.decision.JUDGE_JOB_FAILURE_BOUND = 2
nemo_gym.eval.recovery.decision.LIVE_JOB_STATES = frozenset({'PENDING', 'RUNNING', 'CONFIGURING', 'COMPLETING', 'SUSPENDED', 'REQU...
nemo_gym.eval.recovery.decision.MAX_ATTEMPTS = NG_FAILURE_MAX_ATTEMPTS_KEY
nemo_gym.eval.recovery.decision.MAX_EXTEND_ATTEMPTS = 100
nemo_gym.eval.recovery.decision.NO_INVENTORY = 'the run ended before the task inventory was written'
nemo_gym.eval.recovery.decision.OPERATOR_CONTINUATION = 'OPERATOR-CONTINUATION'
nemo_gym.eval.recovery.decision.OPERATOR_CONTINUATION_NOTE = f'Operator continuation: an explicit budget filled the unknown accounting; see {...
nemo_gym.eval.recovery.decision.RERUN_MASKED_FLAGS = '--rerun-masked-without-artifact --rerun-reason'
nemo_gym.eval.recovery.decision.RERUN_MASKED_REASON = f'retained, never run again, not measured: it reruns only with {RERUN_MASKED_FLA...
nemo_gym.eval.recovery.decision.RESERVATION_WITHOUT_OUTCOME = 'reservation_without_outcome'
nemo_gym.eval.recovery.decision.SCORED_UNPARSEABLE_REASON = 'judge gave no verdict (unparseable); re-judge planned'
nemo_gym.eval.recovery.decision.STORE_DOES_NOT_REPLAY = "the policy retries it but the run's Store does not plan it again"
nemo_gym.eval.recovery.decision.TERMINAL = NG_TERMINAL_KEY
nemo_gym.eval.recovery.decision.UNKNOWN_BUDGET_HINT = 'An operator may continue it with an explicit budget: gym eval resubmit --recove...
nemo_gym.eval.recovery.decision.UNKNOWN_BUDGET_REASON = 'the job chain or scheduler state is unavailable, so the budget is unknown'
nemo_gym.eval.recovery.decision.UNKNOWN_WALLTIME_REASON = 'the run caps its walltime but the used walltime is unavailable'
nemo_gym.eval.recovery.decision._FAILED_JUDGE_JOBS = re.compile(f'the last {JUDGE_JOB_FAILURE_BOUND} judge-only job\(s\) \((?P<ids>.*...