Metrics, Health Checks, Enummetrics, And Actions
This appendix describes the metrics (section G.1), health checks (section G.2), enummetrics (section 12.2.2), and actions (section G.4), along with their parameters, in a newly-installed cluster. Metrics, health checks, enummetrics, and actions can each be standalone scripts, or they can be built-ins. Standalone scripts can be those supplied with the system, or they can be custom scripts built by the administrator. Scripts often require environment variables (as described in section 3.3.1 of the Developer Manual). On success scripts must exit with a status of 0, as is the normal practice.
G.1 Metrics And Their Parameters
A list of metric names can be viewed, for example, for the head node, using cmsh as follows (section 12.5.3.1):
The metrics listed in this section are classed into 10 kinds:
-
regular metrics (section G.1.1)
-
NFS metrics (section G.1.2)
-
InfiniBand metrics (section G.1.3)
-
monitoring system metrics (section G.1.4)
-
GPU metrics (section G.1.6)
-
Job metrics (section G.1.8)
-
IPMI metrics (section G.1.9)
-
Redfish metrics (section G.1.10)
-
SMART metrics (section G.1.11)
-
Prometheus metrics (section G.1.12)
G.1.1 Regular Metrics
* Cumulative metric. I.e. the metric is derived from cumulative raw measurements taken at two different times, according to:
metric[time[2]] = ((measurement[2] - measurement[1]) / (time[2] - time[1]))
The metric is a “per second” measurement.
** Standalone scripts, not built-ins.
If sampling from a head node, the script is in directory: /cm/local/apps/cmd/scripts/metrics/
For regular nodes, the script is in directory: /cm/images/default-image/cm/local/apps/cmd/scripts/metrics/
† Takes block device name (sda, sdc, nvme0n1 and so on) as parameter
‡ Takes interface device name (eth0, eth1, en01, docker0, ib0 and so on) as parameter
§ Takes mount point (for example: /, or /var) as parameter
¶ Takes a process, eg: cmd as a parameter
G.1.2 NFS Metrics
The NFS metrics are all cumulative. They correspond to nfsstat output, and are shown in table G.1.2.
G.1.3 InfiniBand Metrics
The available InfiniBand metrics are displayed in table G.1.3.
G.1.4 Monitoring System Metrics
Internal metrics for the monitoring system itself are produced by the MonitoringSystem data producer.
The data producer does not run by default on any node, as is seen by running the nodes command (here) for it:
However, for convenience, some monitoring system metrics values are displayed on demand in an organized way, by running the monitoringinfo command for a device:
Example
The internal monitoring system metrics can be activated so that their data values are saved. This is not recommended, except temporarily for debugging purposes. To save the values, the Introspect flag for the monitoring system data producer should be set to yes:
Example
After a time defined by interval, the metrics can be viewed:
Example
The monitoring system metrics that are saved are shown in table G.1.4.
G.1.5 CPU Metrics Sampled By The CPUSampler And GPUSampler
Some CPU metrics are sampled by the data producers CPUSampler and GPUSampler (table G.1.5):
* The number 0 in the parameter column can be replaced by the number associated with the core used. Thus:
cpu_core0 can be replaced by cpu_core1, cup_core2…
and
cpu0 can be replaced by cpu1, cpu2…
G.1.6 GPU Metrics
The data producer (section 12.2.10) for the GPU metrics of this section is GPUSampler.
There were GPU metrics described earlier on in table G.1.1. These were cluster overview metrics about GPUs, and were provided by the ClusterTotal data producer.
However, the NVIDIA GPU metrics of this section, as the GPUSampler data producer name suggests, is about gathering the sampled GPU data from the devices themselves.
There is also a separate section about job GPU metrics (section G.1.8.4, here), which uses the JobSampler data producer, and is about gathering the sampled job data from the GPU devices themselves.
The device parameter for the GPU metrics in this section, unless otherwise noted, specifies the device slot number that the GPU uses. The parameter takes the form gpu0, gpu1, and so on. It is appended to the metric with a colon character. For example, the gpu_ecc_dbe_agg metric, if used with gpu1, is specified as:
Example
Available GPU metrics for V100 and A100 GPUs are listed in table G.1.6. The available GPU health checks for V100 and A100 GPUs are listed in table G.2.2.
If the cluster has been configured with AMD GPUs (section 7.4 of the Installation Manual) then AMD GPU metrics become available. Metrics in the table that are also valid AMD GPU metrics are noted. Some metrics have been added and noted that are only valid for AMD GPUs.
* Specified as average, or specified for a GPU.
For example, for the gpu_dec_utilization metric:
gpu_dec_utilization:average or gpu_dec_utilization:GPU0
** Specified as a total, or specified for a GPU.
For example, for the gpu_mem_total metric:
gpu_mem_total:total or gpu_mem_total:GPU0
G.1.7 GPU Profiling Metrics
The data producer (section 12.2.10) for the GPU profiling metrics of this section is GPUSampler.
The device parameter for the GPU profiling metrics in this section, unless otherwise noted, specifies the device slot number that the GPU uses. The parameter takes the form gpu0, gpu1, and so on. It is appended to the metric with a colon character. For example, the gpu_profiling_fp64_active metric, if used with gpu1, is specified as:
Example
Available GPU profiling metrics for V100 and A100 GPUs are listed in table G.1.7.
G.1.8 Job Metrics
Job metrics are introduced in section 13.1.
G.1.8.1 Basic Job Metrics
The following table lists some of the most useful job metrics that BCM can monitor and visualize. In the table, the text <device> denotes a block device name, such as sda.
On virtual machines, block device metrics may be unavailable because of virtualization.
The third column in the table shows the precise source file name that is used when the value is retrieved. These files are all virtual files, and are created as the cgroup controllers are mounted to the cgroup directory. In this case several controllers are mounted to the same directory, which means that all the virtual files will show up in that directory, and in its associated subdirectories—job cgroup directories—when the job runs.
G.1.8.2 Advanced Job Metrics
The metrics in the preceding table are enabled by default. There are also over 40 other advanced metrics that can be enabled via the Enable Advanced Metrics property of the jobmetricsettings object:
The advanced job metrics are:
G.1.8.3 Job Queue Metrics
Job queue metrics become available after workload manager jobs are run. They are produced by the JobQueueSampler data producer.
G.1.8.4 Job GPU Metrics
The GPU metrics of section G.1.6 have a GPUSampler data producer (section 12.2.10). As the name suggests, those metrics are about gathering the sampled GPU data from the devices themselves.
The job GPU metrics of this section have a JobSampler data producer. As the name suggests, these metrics are about gathering the sampled job data from the GPU devices themselves.
In other words, the job GPU metrics are essentially the same metrics as in the GPU metrics section, but are valid only for the job. This is reflected in their names, which are identical, except for the job_ prefix. Other differences of job GPU metrics in comparison with GPU metrics are that
-
they are sampled only on the node that hosts the GPUs on which the job ran, and do not take specific GPUs as a parameter.
-
the metrics are listed in the
monitoring measurablemode ofcmshonly after jobs have run on the GPUs.
Available job GPU metrics for V100 and A100 GPUs are listed in table G.1.8.4.
If the cluster has been configured with AMD GPUs (section 7.4 of the Installation Manual) then AMD job GPU metrics become available. Metrics in the table that are also valid AMD job GPU metrics are noted. Some metrics have been added and noted that are only valid for AMD GPUs.
G.1.9 IPMI Metrics
The IPMI metrics correspond to metrics provided by the BMC devices. The metrics available depend on the manufacturer, and are detected by BCM. The metrics listed in the following table are a limited list of what may be detected on a system.
The data producer (section 12.2.10) for the IPMI metrics is the ipmi data producer.
** Standalone scripts, not built-ins.
If sampling from a head node, the script is in directory: /cm/local/apps/cmd/scripts/metrics/
For regular nodes, the script is in directory: /cm/images/default-image/cm/local/apps/cmd/scripts/metrics/
G.1.10 Redfish Metrics
By default Redfish metrics are sampled only when the BMC interface (section 3.7) starts with rf (e.g. rf0, rf1 etc). The Redfish metric sampler is enabled by setting userdefinedresources as follows:
Example
G.1.10.1 Redfish Standalone Script Metrics
An example script that samples Redfish metrics is the standalone script /cm/images/default-image/cm/local/apps/cmd/scripts/metrics/sample_redfish:
G.1.10.2 Redfish Metrics From CMDaemon Built-in Sampling
CMDaemon built-in sampling also samples Redfish metrics from the hardware:
Example
The preceding list is a truncated excerpt from an NVIDIA DGX GH200 system. The Redfish metrics that are available depend on the vendor and hardware.
G.1.11 SMART Metrics
The SMART metrics correspond to metrics provided by SMART hard drive implementation. The metrics available depend on the manufacturer, and are detected by BCM. The metrics listed in the following table are a limited list of what may be detected on a system.
A SMART metric takes a block device name (sda, sdc, nvme0n1 and so on) as a parameter. Some reported values, such as “spin up time” and “start/stop count” are not relevant on non-rotational (solid-state) drives.
The data producer (section 12.2.10) for SMART metrics is the smart data producer.
G.1.12 Prometheus Metrics
Prometheus metrics are introduced in section 14.2.
The data producers for Prometheus metrics are JobSampler, JobMetadataSampler, and some others.
* total in the past 7x24x60x60 seconds, as measured at the time of sampling
G.1.13 NetQ Metrics
NetQ metrics (table G.1.13) are metrics sourced from NetQ. Configuring NetQ with BCM is described in section 3.11.
The data producer (section 12.2.10) for NetQ metrics is the netq data producer.
NetQ health checks are covered in section G.2.4.
G.1.14 Kubernetes Metrics
Configuring Kubernetes with BCM is described in Chapter 4 of the Containerization Manual.
There were cluster Kubernetes metrics described earlier on in table G.1.1. Those were overview metrics about Kubernetes.
The Kubernetes metrics in the following table G.1.14 are state metrics sourced from Kubernetes itself. The data producer (section 12.2.10) for the Kubernetes metrics of this section is kubestatemetrics.
The metrics that are not tagged as [STABLE] are experimental, and may change in behavior during Kubernetes updates.
G.1.15 Parameters For Metrics
Metrics have the parameters indicated by the left column in the following example:
Example
If the value is inherited from the producer, then it is shown in parentheses next to the value. An inherited value can be overwritten by setting it directly for the parameter of a measurable.
The meanings of the parameters are:
Class: A choice assigned to a metric. It can be an internal type, or it can be a standalone class type. A slash (/) is used to separate class levels. A partial list of the class values is:
-
CPU: CPU-related -
Disk: Disk-related -
Disk/Smart: SMART Disk-related -
Fabric: Fabric-related -
GPU: GPU-related -
Internal: An internal metric -
Job: Job metric -
License: License-related -
Memory: Memory-related -
Network: Network-related -
OS: Operating-system-related -
Process: Process-related -
Prometheus: Prometheus-related -
Total: Total cluster-wide-related -
Workload: Workload-related -
Environmental: Environmental-related
Consolidator: This is described in detail in sections 12.4.3 and 12.5.2
Cumulative: If set to no, then the raw value is treated as not cumulative (for example, CoresUp), and the raw value is presented as the metric value.
If set to yes, then the metric is treated as being cumulative, which means that a rate (per second) value is presented.
More explicitly: When set to yes, it means that the raw sample used to calculate the metric is expected to be cumulative, like, for example, the bytes-received counter for an Ethernet interface. This in turn means that the metric is calculated from the raw value by taking the difference in raw sample measurement values, and dividing it by the time period over which the raw values are sampled. Thus, for example:
-
The bytes-received raw measurements, which accumulate as the packets are received, and are in bytes, and have
Cumulativeset to yes, and then have a corresponding metric,BytesRecv, with a value in bytes/second. -
The system uptime raw measurements, which accumulate at the rate of 1 second per second, and are in seconds, have
Cumulativeset to yes, and have a corresponding metric,Uptime, with a value that uses no units. Ideally, the metric has a value of 1, but in practice the measured value varies a little due to jitter.
Description: Description of the raw measurement used by the metric. Empty by default.
Disabled: If set to no (default) then the metric runs.
Gap: The number of samples that are allowed to be missed before a value of NaN is set for the value of the metric.
Maximal age: the maximum age of RLE samples that are kept. If Maximal age is set to 0 then the sample age is not considered. Units can be w, d, h, m, s (weeks, days, hours, minutes, seconds), with s as the default.
Maximal samples: the maximum number of RLE samples that are kept. If Maximal samples is set to 0 then the number of samples is not considered.
Maximum: the value that the y-axis maximum takes in graphs plotted in Base View by default. If the maximum of the y-values is more than the default y-axis maximum value, then the maximum of the y-values becomes the y-axis maximum.
Minimum: the value that the y-axis minimum takes in graphs plotted in Base View by default. If the minimum of the y-values is less than the default y-axis minimum value, then the minimum of the y-values becomes the y-axis minimum.
NOTE: To clarify the Maximum and Minimum concepts, a case can be considered where minimum=0, maximum=3 are set. If a data point with a y-value of 2 is plotted on a graph, then the y-axis spans the range from 0 to 3 by default. However
-
if the data point has a y-value of 4 instead, then it means the y-axis maximum of 3 is re-sized from its default of 3 to the value of 4, so that the y-axis now spans from 0 to 4.
-
if the data point has a y-value of -1 instead, then it means the y-axis minimum of 0 is re-sized from its default of 0 to the value of -1, so that the y-axis now spans from -1 to 3.
Name: The name given to the metric.
Parameter: Parameter used for this metric. For example, eth0 with the metric BytesRecv
Producer: The data producer that produces the metric
Revision: User-definable revision number for the object
Type: This can be one of metric, healthcheck, or enummetric
Unit: A unit for the metric. For example: B/s (bytes/second) for BytesRecv metric, or unit-less for the Uptime metric. A percent is indicated with %
G.2 Health Checks And Their Parameters
A list of health checks can be viewed, for example, for the head node, using cmsh as follows (section 12.5.3.1):
The health checks listed in this section are classed into 4 kinds:
-
Regular health checks (section 12.2.4) are listed and described in section G.2.1.
-
GPU health checks (section G.2.2)
-
Redfish health checks (section G.2.3)
-
NetQ health checks (section G.2.4)
G.2.1 Regular Health Checks
* built-ins, not standalone scripts.
If sampling from a head node, a standalone script is in directory:
/cm/local/apps/cmd/scripts/healthchecks/
If sampling from a regular node, a standalone script is in directory:
/cm/images/default-image/cm/local/apps/cmd/scripts/healthchecks/
G.2.2 GPU Health Checks
The data producer (section 12.2.10) for the GPU health checks of this section is GPUSampler.
The NVIDIA GPU health checks of this section, as the GPUSampler data producer name suggests, is about gathering the sampled GPU data from the devices themselves.
The device parameter for the GPU health checks in this section, unless otherwise noted, requires as a parameter the device slot number that the GPU uses. The parameter takes the form gpu0, gpu1, and so on. It is appended to the metric with a colon character. For example, the gpu_health_inforom health check, if used with gpu1, is specified as:
Example
Available GPU health checks for V100 and A100 GPUs are listed in table G.2.2. The available GPU metrics are displayed in table G.1.6.
* Specified without a GPU because the check is a check for all GPUs.
** If specified without a GPU, then the check is a check for all GPUs.
For example, for the gpu_health_overall health check:
gpu_health_overall is for all the GPUs
gpu_health_overall:gpu0 is just for GPU0
G.2.3 Redfish Health Checks
The available Redfish health checks are displayed in table G.2.3.
G.2.4 NetQ Health Checks
NetQ health checks (table G.2.4) are healthcheck measurables sourced from NetQ. Configuring NetQ with BCM is described in section 3.11.
The data producer (section 12.2.10) for NetQ measurables is the netq data producer.
G.2.5 Parameters For Health Checks
Health checks have the parameters indicated by the left column in the example below:
Example
If the value is inherited from the producer, then it is shown in parentheses next to the value. An inherited value can be overwritten by setting it directly for the parameter of a measurable.
The parameters are a subset of the parameters for metrics described in section G.1.15.
G.3 Enummetrics
G.4 Actions And Their Parameters
G.4.1 Actions
* standalone scripts, not built-ins.
If running from a head node, the script is in directory: /cm/local/apps/cmd/scripts/actions/
If running from a regular node, the script is in directory: /cm/images/default-image/cm/local/apps/
cmd/scripts/actions/
G.4.2 Parameters For A Monitoring Action
The default monitoring actions are listed in section 12.4.4.
All actions have in common the parameters shown by the left column, illustrated by the example below for the drain action:
Example
Out of the full list of default actions, the actions with only the common parameter settings are:
-
Poweron: Powers off the node -
PowerOff: Powers off the node -
PowerReset: Hard resets the node -
Drain: Drains the node (does not allow new jobs on that node) -
Undrain: Undrains the node (allows new jobs on that node) -
Reboot: Reboots node via the operating system. -
Shutdown: Shuts the node down via the operating system. -
ImageUpdate: Updates the node from the software image -
Event: Sends an event to users connected withcmshor Base View
G.4.2.1 Extra Parameters For Some Actions
The following actions have extra parameters:
Action of the type ScriptAction:
-
killprocess: A script that kills a specified process -
testaction: A test script -
remount: A script to remount all devices
The extra parameters for an action of type ScriptAction are:
-
-
Arguments: List of arguments that are taken by the script -
Node environment: Does the script run in the node environment? -
Script: The script path -
timeout: Time within which the script must run before giving up
-
Action of the type EmailAction:
Send e-mail to administrators: Sends an e-mail out, by default to the administrators
The extra parameters for an action of type EmailAction are:
-
-
All administrators: sends the e-mail to the list of users in theAdministrator e-mailsetting inpartition[base]mode -
Info: the body of the e-mail message -
Recipients: a list of recipients
-