> This page is for version 0.5.0.
> For other versions, use one of these documentation indexes:
> - Main (default): https://docs.nvidia.com/nemo/gym/main/llms.txt
> - 0.7.0: https://docs.nvidia.com/nemo/gym/v0.7.0/llms.txt
> - 0.6.0: https://docs.nvidia.com/nemo/gym/v0.6.0/llms.txt
> - 0.5.1: https://docs.nvidia.com/nemo/gym/v0.5.1/llms.txt
> - 0.5.0: https://docs.nvidia.com/nemo/gym/v0.5.0/llms.txt
> - 0.4.0: https://docs.nvidia.com/nemo/gym/v0.4.0/llms.txt
> - 0.3.0: https://docs.nvidia.com/nemo/gym/v0.3.0/llms.txt
> - 0.2.1: https://docs.nvidia.com/nemo/gym/v0.2.1/llms.txt

> For clean Markdown of any page, append .md to the page URL.
> For a complete documentation index, see https://docs.nvidia.com/nemo/gym/llms.txt.

# nemo_gym.orchestration.executors.capabilities

The opt-in interfaces an executor adds to `BaseExecutor`.

`BaseExecutor` needs only `run`. Each feature beyond submitting is an interface here, and an executor
lists the ones it supports in its bases:

class SlurmExecutor(BaseExecutor, Resubmittable, Terminable, LogReadable): ...

Every method of a capability is abstract, so a capability cannot be half implemented; the only
exceptions are the optional hooks, which carry a default and say so. `Terminable` and `LogReadable`
extend `Inspectable` because both need `query_states`.

## Module Contents

### Classes

| Name                                                                            | Description                                                             |
| ------------------------------------------------------------------------------- | ----------------------------------------------------------------------- |
| [`Inspectable`](#nemo_gym-orchestration-executors-capabilities-Inspectable)     | An executor that can say what is happening to the jobs it queued.       |
| [`LogReadable`](#nemo_gym-orchestration-executors-capabilities-LogReadable)     | An executor that can read the logs of the jobs it queued.               |
| [`Resubmittable`](#nemo_gym-orchestration-executors-capabilities-Resubmittable) | An executor that can queue a benchmark of an existing submission again. |
| [`Terminable`](#nemo_gym-orchestration-executors-capabilities-Terminable)       | An executor that can stop the jobs it queued.                           |

### Functions

| Name                                                                | Description                                                                                 |
| ------------------------------------------------------------------- | ------------------------------------------------------------------------------------------- |
| [`require`](#nemo_gym-orchestration-executors-capabilities-require) | `executor`, typed as `capability`, or `UnsupportedOperationError` naming what it cannot do. |

### Data

[`_Capability`](#nemo_gym-orchestration-executors-capabilities-_Capability)

### API

```python
class nemo_gym.orchestration.executors.capabilities.Inspectable()
```

Abstract

An executor that can say what is happening to the jobs it queued.

```python
nemo_gym.orchestration.executors.capabilities.Inspectable.connection_key(
    record: nemo_gym.orchestration.jobs.SubmissionRecord
) -> str
```

abstract

Which connection serves `record`; records with the same key can be queried over one connection.

Records are grouped by this key so that asking about many submissions costs one connection per
scheduler, not one per submission.

```python
nemo_gym.orchestration.executors.capabilities.Inspectable.query_states(
    refs: list[nemo_gym.orchestration.jobs.JobRef]
) -> dict[nemo_gym.orchestration.jobs.JobKey, nemo_gym.orchestration.jobs.ObservedStatus]
```

abstract

What the scheduler says about each of `refs`: one `ObservedStatus` per `JobKey`.

Every ref shares one `connection_key`, so an implementation opens one connection and asks the
scheduler once for all of them. The record locates a job and the scheduler decides: look the job
up by where it runs, not by the recorded id, so a job started by hand or resubmitted from another
machine is still seen. When several jobs belong to one benchmark, aggregate them into one status
whose `job_id` is the one observed. `state` is the executor's own mapping onto `JobState`. A scheduler
that answers with an error yields `UNKNOWN` with the reason in `detail`; a connection that cannot be
opened may raise, and the caller then reports `UNKNOWN` for every job of the group.

```python
nemo_gym.orchestration.executors.capabilities.Inspectable.read_progress(
    record: nemo_gym.orchestration.jobs.SubmissionRecord,
    job: nemo_gym.orchestration.jobs.BenchmarkJob
) -> nemo_gym.orchestration.jobs.Progress | None
```

Optional. How far `job`'s run got, or `None` when this executor cannot tell.

```python
nemo_gym.orchestration.executors.capabilities.Inspectable.read_run_files(
    record: nemo_gym.orchestration.jobs.SubmissionRecord,
    paths: list[pathlib.Path]
) -> dict[pathlib.Path, str | None]
```

Optional. The contents of files in the run directory, `None` for one that is missing or unreadable.

The default reads nothing, which is right for an executor that cannot reach the run directory.

```python
class nemo_gym.orchestration.executors.capabilities.LogReadable()
```

**Bases:** [Inspectable](#nemo_gym-orchestration-executors-capabilities-Inspectable)

An executor that can read the logs of the jobs it queued.

```python
nemo_gym.orchestration.executors.capabilities.LogReadable.stream_logs(
    record: nemo_gym.orchestration.jobs.SubmissionRecord,
    job: nemo_gym.orchestration.jobs.BenchmarkJob,
    source: nemo_gym.orchestration.jobs.LogSource,
    tail: int | None,
    follow: bool
) -> collections.abc.Iterator[str]
```

abstract

Yield the lines of `source`, a resolved `LogSource` whose location already names the job.

`tail` keeps only the last lines. `follow` keeps yielding as the log grows.

```python
class nemo_gym.orchestration.executors.capabilities.Resubmittable()
```

Abstract

An executor that can queue a benchmark of an existing submission again.

Recovery also passes keyword options beyond `resubmit`'s two arguments (retry budget, walltime, repin and others) that
only the Slurm executor accepts today; they are not part of this contract.

```python
nemo_gym.orchestration.executors.capabilities.Resubmittable.resubmit(
    record: nemo_gym.orchestration.jobs.SubmissionRecord,
    benchmark: str
) -> nemo_gym.orchestration.jobs.BenchmarkJob
```

abstract

Queue `benchmark` of `record` again, in place, and return its updated entry.

Refuses while a job of that benchmark is still active, so two jobs never write one run.
Persists the updated record before returning.

```python
class nemo_gym.orchestration.executors.capabilities.Terminable()
```

**Bases:** [Inspectable](#nemo_gym-orchestration-executors-capabilities-Inspectable)

An executor that can stop the jobs it queued.

```python
nemo_gym.orchestration.executors.capabilities.Terminable.terminate(
    record: nemo_gym.orchestration.jobs.SubmissionRecord,
    job: nemo_gym.orchestration.jobs.BenchmarkJob
) -> None
```

abstract

Stop `job`, and everything that would start it again (a queued follow-up, say).

Does only that: it neither checks the job's state nor touches the record. Raises when the
scheduler refuses, so the caller can report it for this benchmark and go on with the others.

```python
nemo_gym.orchestration.executors.capabilities.require(
    executor: object,
    capability: type[nemo_gym.orchestration.executors.capabilities._Capability],
    operation: str
) -> nemo_gym.orchestration.executors.capabilities._Capability
```

`executor`, typed as `capability`, or `UnsupportedOperationError` naming what it cannot do.

```python
nemo_gym.orchestration.executors.capabilities._Capability = TypeVar('_Capability')
```