Skip to navigation

nemo_gym.orchestration.operations

View as MarkdownOpen in Claude

What you do with a submission: status, info, cancel and logs.

This is where the policy lives — guards, ordering, partial-failure handling, grouping and the mapping to JobState — so that an executor only answers questions about its own scheduler. Each function takes an executor typed as the capability it needs (see executors/capabilities.py) and calls only that capability’s methods, so nothing here knows which executor it is talking to. This module imports no executor: submit.py, which knows which executors exist, resolves one from a record and calls in.

Module Contents

Functions

NameDescription
_as_int-
_read_configThe resolved config of record and None, or None and why it is not available.
_reconcileThe status shown for job: the executor’s state, corrected by what the job wrote.
_resolve_log_sourceThe one planned log that source and instance name, or ValueError listing what exists.
_roll_upOne state for a submission: running wins, then queued, failed, cancelled, and only all-succeeded succeeds.
_select_benchmarksThe jobs to ask about, per record; a record that lacks a named benchmark is an error for the whole call.
cancelStop benchmark’s job and return its status as observed afterwards.
cancel_submissionStop every live benchmark of record, and return each one’s status as observed afterwards.
infoDescribe record, or only benchmark of it: its directories, the job ids observed, its log
log_sourcesThe logs benchmark’s job was set up to write, as planned at submit time.
logsYield the lines of one of benchmark’s logs.
statusThe live status of records, one SubmissionStatus each, in the order given.

Data

_DECIDED_BY_FILES

_MAX_CONCURRENT_CONNECTIONS

_NOT_REPORTED

logger

API

nemo_gym.orchestration.operations._as_int(
text: str
) -> int | None
nemo_gym.orchestration.operations._read_config(
) -> tuple[dict[str, object] | None, str | None]

The resolved config of record and None, or None and why it is not available.

The status shown for job: the executor’s state, corrected by what the job wrote.

Active states stand. For a job that finished, or that the executor could not place, a chain that stopped early makes it FAILED, then the exit code of the run decides. CANCELLED is never overridden: a file left by an earlier attempt would otherwise mask a cancel.

nemo_gym.orchestration.operations._resolve_log_source(
source: str,
instance: str | None

The one planned log that source and instance name, or ValueError listing what exists.

nemo_gym.orchestration.operations._roll_up(

One state for a submission: running wins, then queued, failed, cancelled, and only all-succeeded succeeds.

nemo_gym.orchestration.operations._select_benchmarks(
benchmarks: list[str] | None

The jobs to ask about, per record; a record that lacks a named benchmark is an error for the whole call.

Stop benchmark’s job and return its status as observed afterwards.

Re-queries the scheduler first, so it acts on the jobs that exist now and not on a stale record, and leaves a job that is already terminal alone. Calls terminate and writes nothing to the record. A refusal from the scheduler (permission denied, say) is raised for the caller to report against this benchmark. The scheduler can take a moment to report a stopped job, so the status returned may still read RUNNING.

nemo_gym.orchestration.operations.cancel_submission(
confirm: str

Stop every live benchmark of record, and return each one’s status as observed afterwards.

confirm has no default and must equal record.gym_job_id, or ValueError is raised before anything is asked: cancelling a whole submission by accident costs more than typing its id. A benchmark that cannot be stopped is reported on its own, with a CANCELLED failure saying why, and the others proceed.

nemo_gym.orchestration.operations.info(
benchmark: str | None = None

Describe record, or only benchmark of it: its directories, the job ids observed, its log sources and, when read_run_files can read it, the resolved config. Read-only.

The observed job ids are best effort: when the scheduler cannot be reached the recorded ids are shown.

nemo_gym.orchestration.operations.log_sources(
benchmark: str

The logs benchmark’s job was set up to write, as planned at submit time.

Read from the record, so it needs no executor and costs no remote call. A record written before sources were planned has none. Raises ValueError naming the known benchmarks for one the record lacks.

nemo_gym.orchestration.operations.logs(
benchmark: str,
source: str = 'driver',
instance: str | None = None,
tail: int | None = None,
follow: bool = False
) -> collections.abc.Iterator[str]

Yield the lines of one of benchmark’s logs.

source and instance are resolved against log_sources, so stream_logs receives one LogSource and executors never validate names. A name that matches nothing, or a source with several instances and none chosen, raises ValueError listing what exists; so does a submission that recorded no log sources. The current job is found with query_states, and its id fills the location’s {job_id}. Resolution happens when this is called, not when the lines are first read, so a wrong name fails before any output.

nemo_gym.orchestration.operations.status(
benchmarks: list[str] | None = None

The live status of records, one SubmissionStatus each, in the order given.

benchmarks limits every submission to those benchmarks; a submission that lacks one is an error, raised before anything is asked, listing each offender with the benchmarks it does have. Records are grouped by connection_key and query_states is called once per group, so the cost grows with the number of connections, not submissions; the groups run concurrently. A group whose connection fails yields UNKNOWN for its jobs and does not hide the others. The executor’s state is reconciled with the files the job wrote, and CANCELLED is never overridden by a file. Read-only: it never writes the record.

nemo_gym.orchestration.operations._DECIDED_BY_FILES = {JobState.FAILED, JobState.SUCCEEDED, JobState.UNKNOWN}
nemo_gym.orchestration.operations._MAX_CONCURRENT_CONNECTIONS = 8
nemo_gym.orchestration.operations._NOT_REPORTED = ObservedStatus(state=(JobState.UNKNOWN), detail='the executor did not report thi...
nemo_gym.orchestration.operations.logger = logging.getLogger(__name__)