nemo_gym.eval.recovery.rows
nemo_gym.eval.recovery.rows
Whether gym eval resubmit may queue a run again, decided from the run’s persisted evidence only.
Every expected identity (_ng_task_index, _ng_rollout_index) of the materialized inventory gets one state:
scored: a result row that is not a judge failure. Kept as saved; never re-run.model_final: a failures-sidecar row marked_ng_failure_counts_as_zero: the evaluated model’s terminal failure, a scored zero that no infra retry may replace; or anagent_timeoutrow that kept the agent’s answer (is_answered_agent_deadline), the model’s result under its own deadline, which no resume runs again.judge_recoverable: a result row of classjudge_failed. Its saved answer is re-judged bygym eval reverify --judge-failed-only; queueing the policy again is never needed for it.retryable: the last sidecar row is non-terminal and under its recorded attempt cap: a Gym resume retries it.exhausted: the last sidecar row is non-terminal and at its attempt cap: only a resubmit gives it a fresh budget.terminal: a terminal sidecar row that is not a scored zero; excluded, and the run cannot become complete.missing: no row at all (interrupted or never started); a resume restarts its input.masked: a result row the producer masked or did not evaluate. Collection keeps it as completed and never runs it again, so it is retained, but it is not measured and the run is not complete.omitted: a planned row that a staged collection’s accepted partial stage omitted. It is retained, never planned again and not measured, and the run is complete without it.unknown: a failure whose kind is unregistered or absent, or a row whose identity is invalid; nobody owns it.
The run-level action follows: stop for a spent or unknown resubmit budget, or for an unknown identity that a resume
would run again; noop when nothing is left to do, reverify when only judge failures remain, else resubmit.
A run that ended before its inventory was written has nothing to recover: fresh-restart queues it again as a fresh
run with a longer walltime.
An unknown identity that a resume never runs again (the Store or the saved result holds it) does not block the
missing and retryable ones: the decision reports it apart from what is retried and what is retained. A decision never
reports a run as complete while an identity is terminal, missing or unknown.
Module Contents
Functions
Data
API
For a stop on the identities’ own states (exhausted, masked): what in the chain’s budget would stop the resume an extension or a rerun allows, as the stop texts name it, so one call can lift every blocker; else "".
What a resume runs, what nobody owns (by kind, largest first) and the saved rows kept as they are.
The kind of an unknown identity with the rule that refused it, without naming the kind twice.
Classify every materialized identity and choose the run-level action.
failures are every failures-sidecar row in file order (_line is the row’s line number when known). The
standing row of a rollout is chosen as rollout collection chooses it (_latest_failure_rows): rows that
is_non_superseding_rollout_row names are skipped, and an interrupted omitted row means the rollout is missing.
Only rows at or after resubmit_offset count towards the attempt cap. The rows are only read.
allocation_cut says that the allocation that last wrote the Store has ended, by any path (one of
ALLOCATION_ENDED_STATES), and no allocation of the run is live: a reservation with no saved row was in flight
when it ended and saved nothing, so it is not started for recovery purposes and the Store runs it again.
A masked result with a saved answer, no gradeable artifact and no recorded grade (masked_without_artifact_kind)
stays MASKED, named in masked_without_artifact, unless the Store reports an approval for it
(ManagedRecoveryFact.rerun_approved) or rerun_masked_without_artifact asks for it now: it is then planned
like an infrastructure-masked result, one failed attempt within its cap, and rerun_masked lists the retries
this call would have journaled.
The omitted identities by stage and policy, e.g. “1 omitted by an accepted partial stage (stage_index 1, tolerate_unresolved)”.