Skip to navigation

nemo_gym.orchestration.executors.slurm.lustre_md_stats

View as MarkdownOpen in Claude

Lustre metadata request counts of a submitted Slurm job, per phase (the compute’s lustre_md_stats).

The job script appends one TSV row per node, MDT and operation at each phase to <benchmark dir>/logs/lustre-md-stats-<slurm job id>.tsv: phase<TAB>epoch<TAB>node<TAB>mdt<TAB>op<TAB>count. A count is the node’s cumulative MDC request counter (lctl get_param mdc.*.md_stats), so only the difference between two phases means anything.

Module Contents

Classes

NameDescription
IntervalStatsRequests made between two phases, summed over the nodes that recorded both.
NodeSnapshotOne node’s counters at one phase, by (MDT, operation). An operation lctl omitted counts 0.

Functions

NameDescription
_ranked-
_rate-
_top-
analyzeEvery phase interval of a stats file, then total from its first recorded phase to its last.
format_reportA text table of one stats file’s report: one row per interval.
interval_statsThe requests between phases first and last, or None when no node recorded both.
load_snapshotsRead a stats file into phase -> node -> snapshot.
print_lustre_md_statsPrint the report of every stats file under target; with json_output, a mapping from file to report.
stats_filesThe stats files of a benchmark directory (one per Slurm job of the run), or target itself when a file.

Data

DRIVER_END

END

INTERVALS

PHASES

SERVICES_READY

START

STATS_FILE_PREFIX

STATS_FILE_SUFFIX

_TOP

API

class nemo_gym.orchestration.executors.slurm.lustre_md_stats.IntervalStats(
name: str,
first: str,
last: str,
seconds: int,
nodes: int,
requests: int,
by_mdt: dict[str, int],
by_op: dict[str, int],
peak_node: str | None,
peak_node_requests_per_second: float | None
)
Dataclass

Requests made between two phases, summed over the nodes that recorded both.

by_mdt
dict[str, int]
by_op
dict[str, int]
first
str
last
str
name
str
nodes
int
peak_node
str | None
peak_node_requests_per_second
float | None
requests
int
requests_per_second
float | None
seconds
int
nemo_gym.orchestration.executors.slurm.lustre_md_stats.IntervalStats.to_json() -> dict[str, typing.Any]
class nemo_gym.orchestration.executors.slurm.lustre_md_stats.NodeSnapshot(
epoch: int,
counts: dict[tuple[str, str], int] = dict()
)
Dataclass

One node’s counters at one phase, by (MDT, operation). An operation lctl omitted counts 0.

counts
dict[tuple[str, str], int] = field(default_factory=dict)
epoch
int
nemo_gym.orchestration.executors.slurm.lustre_md_stats._ranked(
counter: collections.Counter
) -> dict[str, int]
nemo_gym.orchestration.executors.slurm.lustre_md_stats._rate(
value: float | None
) -> str
nemo_gym.orchestration.executors.slurm.lustre_md_stats._top(
counts: dict[str, int],
seconds: int
) -> str
nemo_gym.orchestration.executors.slurm.lustre_md_stats.analyze(
path: pathlib.Path
) -> dict[str, typing.Any]

Every phase interval of a stats file, then total from its first recorded phase to its last.

nemo_gym.orchestration.executors.slurm.lustre_md_stats.format_report(
path: pathlib.Path,
report: dict[str, typing.Any]
) -> str

A text table of one stats file’s report: one row per interval.

nemo_gym.orchestration.executors.slurm.lustre_md_stats.interval_stats(
first: str,
last: str,
name: str | None = None

The requests between phases first and last, or None when no node recorded both.

seconds is the longest per-node interval; the peak is the node with the highest requests per second.

nemo_gym.orchestration.executors.slurm.lustre_md_stats.load_snapshots(
path: pathlib.Path

Read a stats file into phase -> node -> snapshot.

A line that is not a whole row (one cut short when the job was killed) is skipped. A start row after rows of a later phase begins the file again: Slurm requeued the job under the same ID, and only its last run counts.

nemo_gym.orchestration.executors.slurm.lustre_md_stats.print_lustre_md_stats(
target: pathlib.Path,
json_output: bool = False
) -> None

Print the report of every stats file under target; with json_output, a mapping from file to report.

nemo_gym.orchestration.executors.slurm.lustre_md_stats.stats_files(
target: pathlib.Path
) -> list[pathlib.Path]

The stats files of a benchmark directory (one per Slurm job of the run), or target itself when a file.

nemo_gym.orchestration.executors.slurm.lustre_md_stats.DRIVER_END = 'driver_end'
nemo_gym.orchestration.executors.slurm.lustre_md_stats.END = 'end'
nemo_gym.orchestration.executors.slurm.lustre_md_stats.INTERVALS = ((START, SERVICES_READY), (SERVICES_READY, DRIVER_END), (DRIVER_END, END))
nemo_gym.orchestration.executors.slurm.lustre_md_stats.PHASES = (START, SERVICES_READY, DRIVER_END, END)
nemo_gym.orchestration.executors.slurm.lustre_md_stats.SERVICES_READY = 'services_ready'
nemo_gym.orchestration.executors.slurm.lustre_md_stats.START = 'start'
nemo_gym.orchestration.executors.slurm.lustre_md_stats.STATS_FILE_PREFIX = 'lustre-md-stats-'
nemo_gym.orchestration.executors.slurm.lustre_md_stats.STATS_FILE_SUFFIX = '.tsv'
nemo_gym.orchestration.executors.slurm.lustre_md_stats._TOP = 3