> This page is for version 0.3.0.
> For other versions, use one of these documentation indexes:
> - Main (default): https://docs.nvidia.com/nemo/gym/main/llms.txt
> - 0.7.0: https://docs.nvidia.com/nemo/gym/v0.7.0/llms.txt
> - 0.6.0: https://docs.nvidia.com/nemo/gym/v0.6.0/llms.txt
> - 0.5.1: https://docs.nvidia.com/nemo/gym/v0.5.1/llms.txt
> - 0.5.0: https://docs.nvidia.com/nemo/gym/v0.5.0/llms.txt
> - 0.4.0: https://docs.nvidia.com/nemo/gym/v0.4.0/llms.txt
> - 0.3.0: https://docs.nvidia.com/nemo/gym/v0.3.0/llms.txt
> - 0.2.1: https://docs.nvidia.com/nemo/gym/v0.2.1/llms.txt

> For clean Markdown of any page, append .md to the page URL.
> For a complete documentation index, see https://docs.nvidia.com/nemo/gym/llms.txt.

# nemo_gym.orchestration.operations

What you do with a submission: status, info, cancel and logs.

This is where the policy lives -- guards, ordering, partial-failure handling, grouping and the mapping
to `JobState` -- so that an executor only answers questions about its own scheduler. Each function
takes an executor typed as the capability it needs (see `executors/capabilities.py`) and calls only
that capability's methods, so nothing here knows which executor it is talking to. This module imports
no executor: `submit.py`, which knows which executors exist, resolves one from a record and calls in.

## Module Contents

### Functions

| Name                                                                            | Description                                                                                                |
| ------------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------- |
| [`_as_int`](#nemo_gym-orchestration-operations-_as_int)                         | -                                                                                                          |
| [`_read_config`](#nemo_gym-orchestration-operations-_read_config)               | The resolved config of `record` and None, or None and why it is not available.                             |
| [`_reconcile`](#nemo_gym-orchestration-operations-_reconcile)                   | The status shown for `job`: the executor's state, corrected by what the job wrote.                         |
| [`_resolve_log_source`](#nemo_gym-orchestration-operations-_resolve_log_source) | The one planned log that `source` and `instance` name, or `ValueError` listing what exists.                |
| [`_roll_up`](#nemo_gym-orchestration-operations-_roll_up)                       | One state for a submission: running wins, then queued, failed, cancelled, and only all-succeeded succeeds. |
| [`_select_benchmarks`](#nemo_gym-orchestration-operations-_select_benchmarks)   | The jobs to ask about, per record; a record that lacks a named benchmark is an error for the whole call.   |
| [`cancel`](#nemo_gym-orchestration-operations-cancel)                           | Stop `benchmark`'s job and return its status as observed afterwards.                                       |
| [`cancel_submission`](#nemo_gym-orchestration-operations-cancel_submission)     | Stop every live benchmark of `record`, and return each one's status as observed afterwards.                |
| [`info`](#nemo_gym-orchestration-operations-info)                               | Describe `record`, or only `benchmark` of it: its directories, the job ids observed, its log               |
| [`log_sources`](#nemo_gym-orchestration-operations-log_sources)                 | The logs `benchmark`'s job was set up to write, as planned at submit time.                                 |
| [`logs`](#nemo_gym-orchestration-operations-logs)                               | Yield the lines of one of `benchmark`'s logs.                                                              |
| [`status`](#nemo_gym-orchestration-operations-status)                           | The live status of `records`, one `SubmissionStatus` each, in the order given.                             |

### Data

[`_DECIDED_BY_FILES`](#nemo_gym-orchestration-operations-_DECIDED_BY_FILES)

[`_MAX_CONCURRENT_CONNECTIONS`](#nemo_gym-orchestration-operations-_MAX_CONCURRENT_CONNECTIONS)

[`_NOT_REPORTED`](#nemo_gym-orchestration-operations-_NOT_REPORTED)

[`logger`](#nemo_gym-orchestration-operations-logger)

### API

```python
nemo_gym.orchestration.operations._as_int(
    text: str
) -> int | None
```

```python
nemo_gym.orchestration.operations._read_config(
    executor: nemo_gym.orchestration.executors.capabilities.Inspectable,
    record: nemo_gym.orchestration.jobs.SubmissionRecord
) -> tuple[dict[str, object] | None, str | None]
```

The resolved config of `record` and None, or None and why it is not available.

```python
nemo_gym.orchestration.operations._reconcile(
    record: nemo_gym.orchestration.jobs.SubmissionRecord,
    job: nemo_gym.orchestration.jobs.BenchmarkJob,
    observed: nemo_gym.orchestration.jobs.ObservedStatus
) -> nemo_gym.orchestration.jobs.JobStatus
```

The status shown for `job`: the executor's state, corrected by what the job wrote.

Active states stand. For a job that finished, or that the executor could not place, a chain that
stopped early makes it `FAILED`, then the exit code of the run decides. `CANCELLED` is never
overridden: a file left by an earlier attempt would otherwise mask a cancel.

```python
nemo_gym.orchestration.operations._resolve_log_source(
    job: nemo_gym.orchestration.jobs.BenchmarkJob,
    source: str,
    instance: str | None
) -> nemo_gym.orchestration.jobs.LogSource
```

The one planned log that `source` and `instance` name, or `ValueError` listing what exists.

```python
nemo_gym.orchestration.operations._roll_up(
    states: list[nemo_gym.orchestration.jobs.JobState]
) -> nemo_gym.orchestration.jobs.JobState
```

One state for a submission: running wins, then queued, failed, cancelled, and only all-succeeded succeeds.

```python
nemo_gym.orchestration.operations._select_benchmarks(
    records: list[nemo_gym.orchestration.jobs.SubmissionRecord],
    benchmarks: list[str] | None
) -> list[list[nemo_gym.orchestration.jobs.BenchmarkJob]]
```

The jobs to ask about, per record; a record that lacks a named benchmark is an error for the whole call.

```python
nemo_gym.orchestration.operations.cancel(
    executor: nemo_gym.orchestration.executors.capabilities.Terminable,
    record: nemo_gym.orchestration.jobs.SubmissionRecord,
    benchmark: str
) -> nemo_gym.orchestration.jobs.JobStatus
```

Stop `benchmark`'s job and return its status as observed afterwards.

Re-queries the scheduler first, so it acts on the jobs that exist now and not on a stale record, and
leaves a job that is already terminal alone. Calls `terminate` and writes nothing to the record. A
refusal from the scheduler (permission denied, say) is raised for the caller to report against this
benchmark. The scheduler can take a moment to report a stopped job, so the status returned may still
read `RUNNING`.

```python
nemo_gym.orchestration.operations.cancel_submission(
    executor: nemo_gym.orchestration.executors.capabilities.Terminable,
    record: nemo_gym.orchestration.jobs.SubmissionRecord,
    confirm: str
) -> list[nemo_gym.orchestration.jobs.JobStatus]
```

Stop every live benchmark of `record`, and return each one's status as observed afterwards.

`confirm` has no default and must equal `record.gym_job_id`, or `ValueError` is raised before anything is
asked: cancelling a whole submission by accident costs more than typing its id. A benchmark that cannot
be stopped is reported on its own, with a `CANCELLED` failure saying why, and the others proceed.

```python
nemo_gym.orchestration.operations.info(
    executor: nemo_gym.orchestration.executors.capabilities.Inspectable,
    record: nemo_gym.orchestration.jobs.SubmissionRecord,
    benchmark: str | None = None
) -> nemo_gym.orchestration.jobs.JobInfo
```

Describe `record`, or only `benchmark` of it: its directories, the job ids observed, its log
sources and, when `read_run_files` can read it, the resolved config. Read-only.

The observed job ids are best effort: when the scheduler cannot be reached the recorded ids are shown.

```python
nemo_gym.orchestration.operations.log_sources(
    record: nemo_gym.orchestration.jobs.SubmissionRecord,
    benchmark: str
) -> list[nemo_gym.orchestration.jobs.LogSource]
```

The logs `benchmark`'s job was set up to write, as planned at submit time.

Read from the record, so it needs no executor and costs no remote call. A record written before
sources were planned has none. Raises `ValueError` naming the known benchmarks for one the record lacks.

```python
nemo_gym.orchestration.operations.logs(
    executor: nemo_gym.orchestration.executors.capabilities.LogReadable,
    record: nemo_gym.orchestration.jobs.SubmissionRecord,
    benchmark: str,
    source: str = 'driver',
    instance: str | None = None,
    tail: int | None = None,
    follow: bool = False
) -> collections.abc.Iterator[str]
```

Yield the lines of one of `benchmark`'s logs.

`source` and `instance` are resolved against `log_sources`, so `stream_logs` receives one `LogSource`
and executors never validate names. A name that matches nothing, or a source with several instances
and none chosen, raises `ValueError` listing what exists; so does a submission that recorded no log
sources. The current job is found with `query_states`, and its id fills the location's `{job_id}`.
Resolution happens when this is called, not when the lines are first read, so a wrong name fails before
any output.

```python
nemo_gym.orchestration.operations.status(
    executor: nemo_gym.orchestration.executors.capabilities.Inspectable,
    records: list[nemo_gym.orchestration.jobs.SubmissionRecord],
    benchmarks: list[str] | None = None
) -> list[nemo_gym.orchestration.jobs.SubmissionStatus]
```

The live status of `records`, one `SubmissionStatus` each, in the order given.

`benchmarks` limits every submission to those benchmarks; a submission that lacks one is an error,
raised before anything is asked, listing each offender with the benchmarks it does have. Records are
grouped by `connection_key` and `query_states` is called once per group, so the cost grows with the
number of connections, not submissions; the groups run concurrently. A group whose connection fails
yields `UNKNOWN` for its jobs and does not hide the others. The executor's `state` is reconciled with
the files the job wrote, and `CANCELLED` is never overridden by a file. Read-only: it never writes the
record.

```python
nemo_gym.orchestration.operations._DECIDED_BY_FILES = {JobState.FAILED, JobState.SUCCEEDED, JobState.UNKNOWN}
```

```python
nemo_gym.orchestration.operations._MAX_CONCURRENT_CONNECTIONS = 8
```

```python
nemo_gym.orchestration.operations._NOT_REPORTED = ObservedStatus(state=(JobState.UNKNOWN), detail='the executor did not report thi...
```

```python
nemo_gym.orchestration.operations.logger = logging.getLogger(__name__)
```