nemo_gym.orchestration.executors.slurm.lustre_md_stats
nemo_gym.orchestration.executors.slurm.lustre_md_stats
Lustre metadata request counts of a submitted Slurm job, per phase (the compute’s lustre_md_stats).
The job script appends one TSV row per node, MDT and operation at each phase to
<benchmark dir>/logs/lustre-md-stats-<slurm job id>.tsv:
phase<TAB>epoch<TAB>node<TAB>mdt<TAB>op<TAB>count. A count is the node’s cumulative MDC request counter
(lctl get_param mdc.*.md_stats), so only the difference between two phases means anything.
Module Contents
Classes
Functions
Data
API
Requests made between two phases, summed over the nodes that recorded both.
One node’s counters at one phase, by (MDT, operation). An operation lctl omitted counts 0.
Every phase interval of a stats file, then total from its first recorded phase to its last.
A text table of one stats file’s report: one row per interval.
The requests between phases first and last, or None when no node recorded both.
seconds is the longest per-node interval; the peak is the node with the highest requests per second.
Read a stats file into phase -> node -> snapshot.
A line that is not a whole row (one cut short when the job was killed) is skipped. A start row after rows of
a later phase begins the file again: Slurm requeued the job under the same ID, and only its last run counts.
Print the report of every stats file under target; with json_output, a mapping from file to report.
The stats files of a benchmark directory (one per Slurm job of the run), or target itself when a file.