Skip to navigation

nemo_gym.orchestration.executors.base

View as MarkdownOpen in Claude

Module Contents

Classes

NameDescription
BaseExecutorWhat a new executor extends: implement run and it can submit.
SchedulerRefusedErrorThe scheduler refused to queue a resubmitted job; detail holds its text verbatim and the job dir’s state.
UnsupportedOperationErrorThe executor does not implement this operation.

Data

logger

API

class nemo_gym.orchestration.executors.base.BaseExecutor()
Abstract

What a new executor extends: implement run and it can submit.

Everything else is opt-in. executors/capabilities.py has an interface for each further feature (Resubmittable, Inspectable, Terminable, LogReadable); an executor lists the ones it supports in its bases and implements every abstract method of each.

name
str
supports_resumable
bool = False
nemo_gym.orchestration.executors.base.BaseExecutor._store(
write_manifest: collections.abc.Callable[..., None],
action: str,
write_config: collections.abc.Callable[[], None] | None = None
) -> None
staticmethod

The ordering and failure handling persist() describes, for both of its callers.

nemo_gym.orchestration.executors.base.BaseExecutor.export_tracking(
benchmark: str,
) -> dict[str, typing.Any]

Send a complete run without an MLflow export receipt to MLflow with Gym’s post-run export, and report what happened: state (succeeded, skipped, failed or unsupported), run_id and error.

nemo_gym.orchestration.executors.base.BaseExecutor.gym_pin(
) -> str | None

The full Gym commit the run’s saved config pins its driver to; None when there is none or it is unknown.

nemo_gym.orchestration.executors.base.BaseExecutor.installed_commit() -> str | None

The Gym commit this launcher runs, as installed_gym_commit reports it.

nemo_gym.orchestration.executors.base.BaseExecutor.persist(
write_manifest: collections.abc.Callable[..., None]
) -> None

Store the record and the resolved config, in the order that survives a partial failure.

Shared by every executor because the ordering and the failure handling are policy rather than transport. The machine-local index goes first precisely because it cannot fail the submit, so if the manifest write does fail there is still a parseable record for the by-hand recovery the error asks for.

The resolved config is written here, not by each executor’s staging step, so that every executor — today Slurm, tomorrow k8s or a local runner — gets it for free from the one place all of them already call.

Only the files’ transport differs per executor, so it arrives as write_manifest — Connection.write_text already has this signature — rather than this class owning a connection it cannot know how to open. Call it while that transport is still open: reopening one here would pay a second connection on every submit.

The jobs are queued by the time this runs, so a failure has to name them or they are stranded with no record anywhere.

nemo_gym.orchestration.executors.base.BaseExecutor.persist_record(
write_manifest: collections.abc.Callable[..., None]
) -> None

Store an updated record in the same two places persist() does; the resolved config is unchanged.

nemo_gym.orchestration.executors.base.BaseExecutor.recovery_decision(
benchmark: str

Decide from the run’s persisted evidence whether resubmit may queue it; without a collector, stop.

nemo_gym.orchestration.executors.base.BaseExecutor.recovery_evidence(
benchmark: str,
score_key: list[str | int] | None = None,
score_agent: str | None = None,
score_environment: str | None = None,
reader_ref: str | None = None,
extend_retries: int = 0,
extend_attempts: int = 0,
max_walltime: str | None = None,
operator_budget: nemo_gym.eval.recovery.budgets.OperatorBudget | None = None,
server_spinup_timeout_seconds: int | None = None,
spinup_timeout_reason: str | None = None,
rerun_masked_without_artifact: bool = False

Decide from the run’s persisted evidence, and return what was collected; without a collector, stop.

Takes every control a recovery passes, so an executor without a collector reports that it cannot decide.

nemo_gym.orchestration.executors.base.BaseExecutor.reverify_judge_failures(
benchmark: str,
judge_partition: str | None = None,
judge_qos: str | None = None,
repin: str | None = None,
repin_reason: str | None = None,
acknowledge_failed_judge_jobs: bool = False,
acknowledge_reason: str | None = None

Re-judge the run’s saved judge-failed answers; executors without a saved-answer path report that.

judge_partition (with judge_qos, its QOS) names the CPU partition of a judge-only job for a run that served its policy on GPUs (not used for one that serves its judge on a node pool of its own), and repin (with repin_reason) the Gym commit that job runs; an executor without such a job ignores them. acknowledge_failed_judge_jobs (with acknowledge_reason) lets one more judge-only job queue after earlier ones ended without settling a saved answer.

nemo_gym.orchestration.executors.base.BaseExecutor.run(
dry_run: bool = False
abstract

Submit config and return the record describing it.

Returns None on a dry run, which renders the scripts and stops before anything is submitted; every other path either returns a record or raises.

nemo_gym.orchestration.executors.base.BaseExecutor.submission_environment(
required_env: list[str],
forwarded_env: list[str]
staticmethod

Capture known names and resolved controls without inspecting environment values.

Executors pass the names their actual transport required/forwarded. Host references are already retained by config validation, including nested service configurations.

nemo_gym.orchestration.executors.base.BaseExecutor.walltime_plan(
benchmark: str,
max_walltime: str
) -> dict[str, typing.Any] | None

How a --max-walltime raise fits the scheduler’s limits, checked before any evidence is read; None when the executor has no such limit to check.

class nemo_gym.orchestration.executors.base.SchedulerRefusedError(
message: str,
detail: dict[str, typing.Any]
)

Bases: RuntimeError

The scheduler refused to queue a resubmitted job; detail holds its text verbatim and the job dir’s state.

class nemo_gym.orchestration.executors.base.UnsupportedOperationError()

Bases: RuntimeError

The executor does not implement this operation.

nemo_gym.orchestration.executors.base.logger = logging.getLogger(__name__)