Skip to navigation

nemo_gym.eval.recovery.rows

View as MarkdownOpen in Claude

Whether gym eval resubmit may queue a run again, decided from the run’s persisted evidence only.

Every expected identity (_ng_task_index, _ng_rollout_index) of the materialized inventory gets one state:

  • scored: a result row that is not a judge failure. Kept as saved; never re-run.
  • model_final: a failures-sidecar row marked _ng_failure_counts_as_zero: the evaluated model’s terminal failure, a scored zero that no infra retry may replace; or an agent_timeout row that kept the agent’s answer (is_answered_agent_deadline), the model’s result under its own deadline, which no resume runs again.
  • judge_recoverable: a result row of class judge_failed. Its saved answer is re-judged by gym eval reverify --judge-failed-only; queueing the policy again is never needed for it.
  • retryable: the last sidecar row is non-terminal and under its recorded attempt cap: a Gym resume retries it.
  • exhausted: the last sidecar row is non-terminal and at its attempt cap: only a resubmit gives it a fresh budget.
  • terminal: a terminal sidecar row that is not a scored zero; excluded, and the run cannot become complete.
  • missing: no row at all (interrupted or never started); a resume restarts its input.
  • masked: a result row the producer masked or did not evaluate. Collection keeps it as completed and never runs it again, so it is retained, but it is not measured and the run is not complete.
  • omitted: a planned row that a staged collection’s accepted partial stage omitted. It is retained, never planned again and not measured, and the run is complete without it.
  • unknown: a failure whose kind is unregistered or absent, or a row whose identity is invalid; nobody owns it.

The run-level action follows: stop for a spent or unknown resubmit budget, or for an unknown identity that a resume would run again; noop when nothing is left to do, reverify when only judge failures remain, else resubmit. A run that ended before its inventory was written has nothing to recover: fresh-restart queues it again as a fresh run with a longer walltime. An unknown identity that a resume never runs again (the Store or the saved result holds it) does not block the missing and retryable ones: the decision reports it apart from what is retried and what is retained. A decision never reports a run as complete while an identity is terminal, missing or unknown.

Module Contents

Functions

NameDescription
_chain_blockers_noteFor a stop on the identities’ own states (exhausted, masked): what in the chain’s budget would stop the resume
_live_reason-
_populationsWhat a resume runs, what nobody owns (by kind, largest first) and the saved rows kept as they are.
_unknown_labelThe kind of an unknown identity with the rule that refused it, without naming the kind twice.
decide_recoveryClassify every materialized identity and choose the run-level action.
omission_summaryThe omitted identities by stage and policy, e.g. “1 omitted by an accepted partial stage (stage_index 1, tolerate_unresolved)”.

Data

_ATTEMPTS_USED_TOKEN

_CHAIN_BLOCKERS_LEAD

_EXTEND_RETRIES_SCOPE

_INFRA_BUDGET_USED_PREFIX

_MASKED_STOP_TOKEN

_NOT_RECORDED_TOKEN

_RESERVATIONS_UNKNOWN

_WALLTIME_USED_TOKEN

API

nemo_gym.eval.recovery.rows._chain_blockers_note(
) -> str

For a stop on the identities’ own states (exhausted, masked): what in the chain’s budget would stop the resume an extension or a rerun allows, as the stop texts name it, so one call can lift every blocker; else "".

nemo_gym.eval.recovery.rows._live_reason(
live_job_name: str | None
) -> str
nemo_gym.eval.recovery.rows._populations(
counts: collections.abc.Mapping[str, int],
) -> str

What a resume runs, what nobody owns (by kind, largest first) and the saved rows kept as they are.

nemo_gym.eval.recovery.rows._unknown_label(
) -> str

The kind of an unknown identity with the rule that refused it, without naming the kind twice.

nemo_gym.eval.recovery.rows.decide_recovery(
materialized: collections.abc.Iterable[collections.abc.Mapping[str, typing.Any]],
results: collections.abc.Iterable[collections.abc.Mapping[str, typing.Any]],
failures: collections.abc.Iterable[collections.abc.Mapping[str, typing.Any]],
default_max_attempts: int = 3,
resubmit_offset: int = 0,
managed_selection: list[nemo_gym.eval.recovery.evidence.ManagedRecoveryFact] | None = None,
planned_missing_continuation: bool = False,
allocation_cut: bool = False,
rerun_masked_without_artifact: bool = False

Classify every materialized identity and choose the run-level action.

failures are every failures-sidecar row in file order (_line is the row’s line number when known). The standing row of a rollout is chosen as rollout collection chooses it (_latest_failure_rows): rows that is_non_superseding_rollout_row names are skipped, and an interrupted omitted row means the rollout is missing. Only rows at or after resubmit_offset count towards the attempt cap. The rows are only read.

allocation_cut says that the allocation that last wrote the Store has ended, by any path (one of ALLOCATION_ENDED_STATES), and no allocation of the run is live: a reservation with no saved row was in flight when it ended and saved nothing, so it is not started for recovery purposes and the Store runs it again.

A masked result with a saved answer, no gradeable artifact and no recorded grade (masked_without_artifact_kind) stays MASKED, named in masked_without_artifact, unless the Store reports an approval for it (ManagedRecoveryFact.rerun_approved) or rerun_masked_without_artifact asks for it now: it is then planned like an infrastructure-masked result, one failed attempt within its cap, and rerun_masked lists the retries this call would have journaled.

nemo_gym.eval.recovery.rows.omission_summary(
evidence: collections.abc.Iterable[nemo_gym.eval.recovery.decision.IdentityEvidence]
) -> str

The omitted identities by stage and policy, e.g. “1 omitted by an accepted partial stage (stage_index 1, tolerate_unresolved)”.

nemo_gym.eval.recovery.rows._ATTEMPTS_USED_TOKEN = 'used every attempt since the last resubmit'
nemo_gym.eval.recovery.rows._CHAIN_BLOCKERS_LEAD = '; a resume of them would also stop, as '
nemo_gym.eval.recovery.rows._EXTEND_RETRIES_SCOPE = '--extend-retries raises infra retries only; it does not raise the per-identity ...
nemo_gym.eval.recovery.rows._INFRA_BUDGET_USED_PREFIX = 'the stored infra-retry budget is used'
nemo_gym.eval.recovery.rows._MASKED_STOP_TOKEN = 'completed result(s) are masked and not measured'
nemo_gym.eval.recovery.rows._NOT_RECORDED_TOKEN = 'asked in this call is not recorded, as nothing was queued: ask it again in the ...
nemo_gym.eval.recovery.rows._RESERVATIONS_UNKNOWN = 'the allocation that made them is not known to have ended, so whether they ran i...
nemo_gym.eval.recovery.rows._WALLTIME_USED_TOKEN = 'the stored walltime budget is used'