nemo_gym.orchestration.submit

Recover benchmark of submission gym_job_id from its persisted evidence (gym eval resubmit --recover).

The executor collects identities, results, failures, the quality summary, the score_key metric and the job chain’s stored budget; decide_recovery picks one action, which runs here: noop dispatches nothing; reverify re-judges saved answers through the executor (never the policy) or reports that the executor cannot; resubmit queues the run with the existing resubmit keeping its spent budget (retries, walltime and per-rollout attempts), and its Gym resume runs only missing and retryable identities; fresh-restart (no task inventory) queues the run again as a fresh run with the explicit, longer max_walltime, and without one names the command; stop dispatches nothing and says why. Every outcome carries the acceptance gate task entry: per-rollout health is judged against max_unhealthy_fraction and reported in its warnings; response/trial health is evaluated separately, and MLflow delivery is checked by mlflow_errors. Plain resubmit still starts a fresh retry and walltime budget.

The outcome of a resubmit the scheduler refused: nothing queued, its text verbatim, the job dir restored.

How a RESUBMIT re-verifies the saved answers it names before the policy work, or None when it names none.

The executor keywords of a spin-up timeout raise, none without one, so executors that predate it work.

Whether a complete run lacks the MLflow export receipt its saved config requires.

Only a run whose saved config configures MLflow (tracking URI, experiment and run name) and does not set mlflow_tracking_required to false, with no post-run receipt at all and no live job, is exported. A run that configured no exporter is never exported, and a receipt, current or not, is never replaced.

Inspect the same saved evidence and acceptance gate as recovery, without dispatching any action.

It plans as recovery would, on the installed Gym, and reports the same pin decision (pin_plan).

With repin it reports the plan a recovery repinned to that commit would follow: the run’s Store is read at the target commit, as the repinned resume would read it, and nothing is rendered, repinned or queued.

The masked results that kept a saved answer but have no gradeable artifact and no grade are named in the decision (masked_without_artifact) and stay masked, with or without rerun_masked_without_artifact: inspect never journals an approval.

Read an original submission file and its content digest without indexing or rewriting it.

Recover a saved submission, preserving its recorded retry and walltime budgets.

A run pinned at another Gym commit than this launcher’s is planned with the installed Gym’s rules. When that plan needs the installed revision to run (a re-judge, whose job runs the installed recovery source, or a resubmit the pinned revision plans differently), the run moves to the installed commit, journaled as an automatic repin with the reason; otherwise its pin stays. auto_repin=False refuses that case instead, naming repin.

repin (with repin_reason) continues the run on another full Gym commit, which must be this launcher’s; a judge-only job of the run runs at it too. judge_partition names the CPU partition of the judge-only job that grades a model-served run’s saved answers again, when its saved config declares no CPU node pool and the launcher’s reader placement names none, and judge_qos that job’s QOS (else the saved CPU pool’s, the reader placement’s or the partition’s default); neither is used for a run that serves its judge on a node pool of its own, whose job runs on that saved pool. extend_retries records that many more infra retries for the run: explicit and journaled, never a reset. extend_attempts records that many more attempts for every rollout the Store would run again (not a model failure); explicit, journaled with the infra extension, and named in the decision. max_walltime (with walltime_reason) raises the run’s Slurm walltime, never below the recorded one. unsafe_resume_reason resumes the job with allow_unsafe_resume, past a refused run identity check; the reason and the user are journaled. server_spinup_timeout_seconds (with spinup_timeout_reason) raises how long the run’s Gym servers may take to become ready, never below the recorded limit; 0 removes it. The raise is journaled in the job dir. For a run whose chain accounting is unknown, the two are also an operator’s explicit budget: the run continues as an OPERATOR-CONTINUATION with extend_retries (one when not given) infra retries remaining, recorded in the decision and the job dir’s operator journal (see decide_from_collected). rerun_masked_without_artifact (with a nonempty rerun_reason) runs again the masked results that kept a saved answer but have neither a gradeable artifact nor a recorded grade, which a resume otherwise keeps and re-verification cannot grade: the identities the plan finds qualifying are journaled with the reason in the run’s artifacts when the job is queued, and each counts one failed attempt within its cap. acknowledge_failed_judge_jobs (with a nonempty acknowledge_reason) lets one more judge-only job queue when the last two ended without settling a saved answer; the acknowledgement, naming those jobs, is journaled with the reason when the new job is queued, and covers only them. The outcome names the reader_token its login-host Store readers carry.

Queue benchmark of submission gym_job_id again, in place, and return its updated record entry.