nemo_gym.orchestration.executors.base

The ordering and failure handling persist() describes, for both of its callers.

Send a complete run without an MLflow export receipt to MLflow with Gym’s post-run export, and report what happened: state (succeeded, skipped, failed or unsupported), run_id and error.

The full Gym commit the run’s saved config pins its driver to; None when there is none or it is unknown.

The Gym commit this launcher runs, as installed_gym_commit reports it.

Store the record and the resolved config, in the order that survives a partial failure.

Shared by every executor because the ordering and the failure handling are policy rather than transport. The machine-local index goes first precisely because it cannot fail the submit, so if the manifest write does fail there is still a parseable record for the by-hand recovery the error asks for.

The resolved config is written here, not by each executor’s staging step, so that every executor — today Slurm, tomorrow k8s or a local runner — gets it for free from the one place all of them already call.

Only the files’ transport differs per executor, so it arrives as write_manifest — Connection.write_text already has this signature — rather than this class owning a connection it cannot know how to open. Call it while that transport is still open: reopening one here would pay a second connection on every submit.

The jobs are queued by the time this runs, so a failure has to name them or they are stranded with no record anywhere.

Store an updated record in the same two places persist() does; the resolved config is unchanged.

Decide from the run’s persisted evidence whether resubmit may queue it; without a collector, stop.

Decide from the run’s persisted evidence, and return what was collected; without a collector, stop.

Takes every control a recovery passes, so an executor without a collector reports that it cannot decide.

Queue benchmark of record again, in place, and return its updated entry.

keep_budget (used by recovery) keeps the run’s spent retry, walltime and attempt budget instead of starting a fresh one. fresh_restart (used by recovery of a run with no task inventory) starts a fresh one with the longer max_walltime. rerun_masked (with keep_budget) is the approval to run again the masked results it names (reason and identities), journaled with the run’s artifacts before the job is queued and undone if it is not.

Refuses while a job of that benchmark is still active, so two jobs never write one run. Persists the updated record before returning. An executor that cannot do this raises UnsupportedOperationError.

Re-judge the run’s saved judge-failed answers; executors without a saved-answer path report that.

judge_partition (with judge_qos, its QOS) names the CPU partition of a judge-only job for a run that served its policy on GPUs (not used for one that serves its judge on a node pool of its own), and repin (with repin_reason) the Gym commit that job runs; an executor without such a job ignores them. acknowledge_failed_judge_jobs (with acknowledge_reason) lets one more judge-only job queue after earlier ones ended without settling a saved answer.

Submit config and return the record describing it.

Returns None on a dry run, which renders the scripts and stops before anything is submitted; every other path either returns a record or raises.

Capture known names and resolved controls without inspecting environment values.

Executors pass the names their actual transport required/forwarded. Host references are already retained by config validation, including nested service configurations.

How a --max-walltime raise fits the scheduler’s limits, checked before any evidence is read; None when the executor has no such limit to check.