> This page is for version 0.2.1.
> For other versions, use one of these documentation indexes:
> - Main (default): https://docs.nvidia.com/nemo/gym/main/llms.txt
> - 0.7.0: https://docs.nvidia.com/nemo/gym/v0.7.0/llms.txt
> - 0.6.0: https://docs.nvidia.com/nemo/gym/v0.6.0/llms.txt
> - 0.5.1: https://docs.nvidia.com/nemo/gym/v0.5.1/llms.txt
> - 0.5.0: https://docs.nvidia.com/nemo/gym/v0.5.0/llms.txt
> - 0.4.0: https://docs.nvidia.com/nemo/gym/v0.4.0/llms.txt
> - 0.3.0: https://docs.nvidia.com/nemo/gym/v0.3.0/llms.txt
> - 0.2.1: https://docs.nvidia.com/nemo/gym/v0.2.1/llms.txt

> For clean Markdown of any page, append .md to the page URL.
> For a complete documentation index, see https://docs.nvidia.com/nemo/gym/llms.txt.

# nemo_gym.orchestration.executors.slurm.lustre_md_stats

Lustre metadata request counts of a submitted Slurm job, per phase (the compute's `lustre_md_stats`).

The job script appends one TSV row per node, MDT and operation at each phase to
`<benchmark dir>/logs/lustre-md-stats-<slurm job id>.tsv`:
`phase<TAB>epoch<TAB>node<TAB>mdt<TAB>op<TAB>count`. A count is the node's cumulative MDC request counter
(`lctl get_param mdc.*.md_stats`), so only the difference between two phases means anything.

## Module Contents

### Classes

| Name                                                                                     | Description                                                                                |
| ---------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------ |
| [`IntervalStats`](#nemo_gym-orchestration-executors-slurm-lustre_md_stats-IntervalStats) | Requests made between two phases, summed over the nodes that recorded both.                |
| [`NodeSnapshot`](#nemo_gym-orchestration-executors-slurm-lustre_md_stats-NodeSnapshot)   | One node's counters at one phase, by (MDT, operation). An operation lctl omitted counts 0. |

### Functions

| Name                                                                                                     | Description                                                                                              |
| -------------------------------------------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------- |
| [`_ranked`](#nemo_gym-orchestration-executors-slurm-lustre_md_stats-_ranked)                             | -                                                                                                        |
| [`_rate`](#nemo_gym-orchestration-executors-slurm-lustre_md_stats-_rate)                                 | -                                                                                                        |
| [`_top`](#nemo_gym-orchestration-executors-slurm-lustre_md_stats-_top)                                   | -                                                                                                        |
| [`analyze`](#nemo_gym-orchestration-executors-slurm-lustre_md_stats-analyze)                             | Every phase interval of a stats file, then `total` from its first recorded phase to its last.            |
| [`format_report`](#nemo_gym-orchestration-executors-slurm-lustre_md_stats-format_report)                 | A text table of one stats file's report: one row per interval.                                           |
| [`interval_stats`](#nemo_gym-orchestration-executors-slurm-lustre_md_stats-interval_stats)               | The requests between phases `first` and `last`, or None when no node recorded both.                      |
| [`load_snapshots`](#nemo_gym-orchestration-executors-slurm-lustre_md_stats-load_snapshots)               | Read a stats file into phase -> node -> snapshot.                                                        |
| [`print_lustre_md_stats`](#nemo_gym-orchestration-executors-slurm-lustre_md_stats-print_lustre_md_stats) | Print the report of every stats file under `target`; with `json_output`, a mapping from file to report.  |
| [`stats_files`](#nemo_gym-orchestration-executors-slurm-lustre_md_stats-stats_files)                     | The stats files of a benchmark directory (one per Slurm job of the run), or `target` itself when a file. |

### Data

[`DRIVER_END`](#nemo_gym-orchestration-executors-slurm-lustre_md_stats-DRIVER_END)

[`END`](#nemo_gym-orchestration-executors-slurm-lustre_md_stats-END)

[`INTERVALS`](#nemo_gym-orchestration-executors-slurm-lustre_md_stats-INTERVALS)

[`PHASES`](#nemo_gym-orchestration-executors-slurm-lustre_md_stats-PHASES)

[`SERVICES_READY`](#nemo_gym-orchestration-executors-slurm-lustre_md_stats-SERVICES_READY)

[`START`](#nemo_gym-orchestration-executors-slurm-lustre_md_stats-START)

[`STATS_FILE_PREFIX`](#nemo_gym-orchestration-executors-slurm-lustre_md_stats-STATS_FILE_PREFIX)

[`STATS_FILE_SUFFIX`](#nemo_gym-orchestration-executors-slurm-lustre_md_stats-STATS_FILE_SUFFIX)

[`_TOP`](#nemo_gym-orchestration-executors-slurm-lustre_md_stats-_TOP)

### API

```python
class nemo_gym.orchestration.executors.slurm.lustre_md_stats.IntervalStats(
    name: str,
    first: str,
    last: str,
    seconds: int,
    nodes: int,
    requests: int,
    by_mdt: dict[str, int],
    by_op: dict[str, int],
    peak_node: str | None,
    peak_node_requests_per_second: float | None
)
```

Dataclass

Requests made between two phases, summed over the nodes that recorded both.

**`by_mdt`** `dict[str, int]`

---

**`by_op`** `dict[str, int]`

---

**`first`** `str`

---

**`last`** `str`

---

**`name`** `str`

---

**`nodes`** `int`

---

**`peak_node`** `str | None`

---

**`peak_node_requests_per_second`** `float | None`

---

**`requests`** `int`

---

**`requests_per_second`** `float | None`

---

**`seconds`** `int`

---

```python
nemo_gym.orchestration.executors.slurm.lustre_md_stats.IntervalStats.to_json() -> dict[str, typing.Any]
```

```python
class nemo_gym.orchestration.executors.slurm.lustre_md_stats.NodeSnapshot(
    epoch: int,
    counts: dict[tuple[str, str], int] = dict()
)
```

Dataclass

One node's counters at one phase, by (MDT, operation). An operation lctl omitted counts 0.

**`counts`** `dict[tuple[str, str], int] = field(default_factory=dict)`

---

**`epoch`** `int`

---

```python
nemo_gym.orchestration.executors.slurm.lustre_md_stats._ranked(
    counter: collections.Counter
) -> dict[str, int]
```

```python
nemo_gym.orchestration.executors.slurm.lustre_md_stats._rate(
    value: float | None
) -> str
```

```python
nemo_gym.orchestration.executors.slurm.lustre_md_stats._top(
    counts: dict[str, int],
    seconds: int
) -> str
```

```python
nemo_gym.orchestration.executors.slurm.lustre_md_stats.analyze(
    path: pathlib.Path
) -> dict[str, typing.Any]
```

Every phase interval of a stats file, then `total` from its first recorded phase to its last.

```python
nemo_gym.orchestration.executors.slurm.lustre_md_stats.format_report(
    path: pathlib.Path,
    report: dict[str, typing.Any]
) -> str
```

A text table of one stats file's report: one row per interval.

```python
nemo_gym.orchestration.executors.slurm.lustre_md_stats.interval_stats(
    snapshots: dict[str, dict[str, nemo_gym.orchestration.executors.slurm.lustre_md_stats.NodeSnapshot]],
    first: str,
    last: str,
    name: str | None = None
) -> nemo_gym.orchestration.executors.slurm.lustre_md_stats.IntervalStats | None
```

The requests between phases `first` and `last`, or None when no node recorded both.

`seconds` is the longest per-node interval; the peak is the node with the highest requests per second.

```python
nemo_gym.orchestration.executors.slurm.lustre_md_stats.load_snapshots(
    path: pathlib.Path
) -> dict[str, dict[str, nemo_gym.orchestration.executors.slurm.lustre_md_stats.NodeSnapshot]]
```

Read a stats file into phase -> node -> snapshot.

A line that is not a whole row (one cut short when the job was killed) is skipped. A `start` row after rows of
a later phase begins the file again: Slurm requeued the job under the same ID, and only its last run counts.

```python
nemo_gym.orchestration.executors.slurm.lustre_md_stats.print_lustre_md_stats(
    target: pathlib.Path,
    json_output: bool = False
) -> None
```

Print the report of every stats file under `target`; with `json_output`, a mapping from file to report.

```python
nemo_gym.orchestration.executors.slurm.lustre_md_stats.stats_files(
    target: pathlib.Path
) -> list[pathlib.Path]
```

The stats files of a benchmark directory (one per Slurm job of the run), or `target` itself when a file.

```python
nemo_gym.orchestration.executors.slurm.lustre_md_stats.DRIVER_END = 'driver_end'
```

```python
nemo_gym.orchestration.executors.slurm.lustre_md_stats.END = 'end'
```

```python
nemo_gym.orchestration.executors.slurm.lustre_md_stats.INTERVALS = ((START, SERVICES_READY), (SERVICES_READY, DRIVER_END), (DRIVER_END, END))
```

```python
nemo_gym.orchestration.executors.slurm.lustre_md_stats.PHASES = (START, SERVICES_READY, DRIVER_END, END)
```

```python
nemo_gym.orchestration.executors.slurm.lustre_md_stats.SERVICES_READY = 'services_ready'
```

```python
nemo_gym.orchestration.executors.slurm.lustre_md_stats.START = 'start'
```

```python
nemo_gym.orchestration.executors.slurm.lustre_md_stats.STATS_FILE_PREFIX = 'lustre-md-stats-'
```

```python
nemo_gym.orchestration.executors.slurm.lustre_md_stats.STATS_FILE_SUFFIX = '.tsv'
```

```python
nemo_gym.orchestration.executors.slurm.lustre_md_stats._TOP = 3
```