nemo_gym.orchestration.executors.base
nemo_gym.orchestration.executors.base
Module Contents
Classes
Data
API
What a new executor extends: implement run and it can submit.
Everything else is opt-in. executors/capabilities.py has an interface for each further feature
(Resubmittable, Inspectable, Terminable, LogReadable); an executor lists the ones it
supports in its bases and implements every abstract method of each.
The ordering and failure handling persist() describes, for both of its callers.
Send a complete run without an MLflow export receipt to MLflow with Gym’s post-run export, and report
what happened: state (succeeded, skipped, failed or unsupported), run_id and error.
The full Gym commit the run’s saved config pins its driver to; None when there is none or it is unknown.
The Gym commit this launcher runs, as installed_gym_commit reports it.
Store the record and the resolved config, in the order that survives a partial failure.
Shared by every executor because the ordering and the failure handling are policy rather than transport. The machine-local index goes first precisely because it cannot fail the submit, so if the manifest write does fail there is still a parseable record for the by-hand recovery the error asks for.
The resolved config is written here, not by each executor’s staging step, so that every executor — today Slurm, tomorrow k8s or a local runner — gets it for free from the one place all of them already call.
Only the files’ transport differs per executor, so it arrives as
write_manifest — Connection.write_text already has this signature —
rather than this class owning a connection it cannot know how to open.
Call it while that transport is still open: reopening one here would pay
a second connection on every submit.
The jobs are queued by the time this runs, so a failure has to name them or they are stranded with no record anywhere.
Store an updated record in the same two places persist() does; the resolved config is unchanged.
Decide from the run’s persisted evidence whether resubmit may queue it; without a collector, stop.
Decide from the run’s persisted evidence, and return what was collected; without a collector, stop.
Takes every control a recovery passes, so an executor without a collector reports that it cannot decide.
Re-judge the run’s saved judge-failed answers; executors without a saved-answer path report that.
judge_partition (with judge_qos, its QOS) names the CPU partition of a judge-only job for a run that
served its policy on GPUs (not used for one that serves its judge on a node pool of its own), and
repin (with repin_reason) the Gym commit that job runs; an executor without such a job ignores them.
acknowledge_failed_judge_jobs (with acknowledge_reason) lets one more judge-only job queue after earlier ones
ended without settling a saved answer.
Submit config and return the record describing it.
Returns None on a dry run, which renders the scripts and stops before anything is submitted; every other path either returns a record or raises.
Capture known names and resolved controls without inspecting environment values.
Executors pass the names their actual transport required/forwarded. Host references are already retained by config validation, including nested service configurations.
How a --max-walltime raise fits the scheduler’s limits, checked before any evidence is read; None
when the executor has no such limit to check.
Bases: RuntimeError
The scheduler refused to queue a resubmitted job; detail holds its text verbatim and the job dir’s state.
Bases: RuntimeError
The executor does not implement this operation.