nemo_gym.orchestration.operations
nemo_gym.orchestration.operations
What you do with a submission: status, info, cancel and logs.
This is where the policy lives — guards, ordering, partial-failure handling, grouping and the mapping
to JobState — so that an executor only answers questions about its own scheduler. Each function
takes an executor typed as the capability it needs (see executors/capabilities.py) and calls only
that capability’s methods, so nothing here knows which executor it is talking to. This module imports
no executor: submit.py, which knows which executors exist, resolves one from a record and calls in.
Module Contents
Functions
Data
API
The resolved config of record and None, or None and why it is not available.
The status shown for job: the executor’s state, corrected by what the job wrote.
Active states stand. For a job that finished, or that the executor could not place, a chain that
stopped early makes it FAILED, then the exit code of the run decides. CANCELLED is never
overridden: a file left by an earlier attempt would otherwise mask a cancel.
The one planned log that source and instance name, or ValueError listing what exists.
One state for a submission: running wins, then queued, failed, cancelled, and only all-succeeded succeeds.
The jobs to ask about, per record; a record that lacks a named benchmark is an error for the whole call.
Stop benchmark’s job and return its status as observed afterwards.
Re-queries the scheduler first, so it acts on the jobs that exist now and not on a stale record, and
leaves a job that is already terminal alone. Calls terminate and writes nothing to the record. A
refusal from the scheduler (permission denied, say) is raised for the caller to report against this
benchmark. The scheduler can take a moment to report a stopped job, so the status returned may still
read RUNNING.
Stop every live benchmark of record, and return each one’s status as observed afterwards.
confirm has no default and must equal record.gym_job_id, or ValueError is raised before anything is
asked: cancelling a whole submission by accident costs more than typing its id. A benchmark that cannot
be stopped is reported on its own, with a CANCELLED failure saying why, and the others proceed.
Describe record, or only benchmark of it: its directories, the job ids observed, its log
sources and, when read_run_files can read it, the resolved config. Read-only.
The observed job ids are best effort: when the scheduler cannot be reached the recorded ids are shown.
The logs benchmark’s job was set up to write, as planned at submit time.
Read from the record, so it needs no executor and costs no remote call. A record written before
sources were planned has none. Raises ValueError naming the known benchmarks for one the record lacks.
Yield the lines of one of benchmark’s logs.
source and instance are resolved against log_sources, so stream_logs receives one LogSource
and executors never validate names. A name that matches nothing, or a source with several instances
and none chosen, raises ValueError listing what exists; so does a submission that recorded no log
sources. The current job is found with query_states, and its id fills the location’s {job_id}.
Resolution happens when this is called, not when the lines are first read, so a wrong name fails before
any output.
The live status of records, one SubmissionStatus each, in the order given.
benchmarks limits every submission to those benchmarks; a submission that lacks one is an error,
raised before anything is asked, listing each offender with the benchmarks it does have. Records are
grouped by connection_key and query_states is called once per group, so the cost grows with the
number of connections, not submissions; the groups run concurrently. A group whose connection fails
yields UNKNOWN for its jobs and does not hide the others. The executor’s state is reconciled with
the files the job wrote, and CANCELLED is never overridden by a file. Read-only: it never writes the
record.