> This page is for version 11 (default).
> For other versions, use one of these documentation indexes:
> - 11 (default): https://docs.nvidia.com/base-command-manager/product-manuals/11/llms.txt
> - 10: https://docs.nvidia.com/base-command-manager/product-manuals/10/llms.txt

> For clean Markdown of any page, append .md to the page URL.
> For a complete documentation index, see https://docs.nvidia.com/base-command-manager/product-manuals/llms.txt.
> For AI client integration (Claude Code, Cursor, etc.), connect to the MCP server at https://docs.nvidia.com/base-command-manager/product-manuals/_mcp/server.

# Metrics, Health Checks, Enummetrics, And Actions

This appendix describes the metrics (section [G.1](#g1-metrics-and-their-parameters)), health checks (section [G.2](#g2-health-checks-and-their-parameters)), enummetrics (section [10.2.2](/admin/monitoring#1022-enummetrics)), and actions (section [G.4](#g4-actions-and-their-parameters)), along with their parameters, in a newly-installed cluster. Metrics, health checks, enummetrics, and actions can each be standalone scripts, or they can be built-ins. Standalone scripts can be those supplied with the system, or they can be custom scripts built by the administrator. Scripts often require environment variables (as described in [section 3.3.1 of the *Developer Manual*](/developer/monitoring-actions#331-standard-environment-variables-available-in-action-scripts)). On success scripts must exit with a status of 0, as is the normal practice.

## G.1  Metrics And Their Parameters

A list of metric names can be viewed, for example, for the head node, using `cmsh` as follows (section [10.5.3.1](/admin/monitoring#10531-the-measurable-submode-list-show-and-get)):

```bash
[basecm11 ~]# cmsh -c "monitoring measurable; list metric"
```

The metrics listed in this section are classed into 10 kinds:

1. regular metrics (section [G.1.1](#g11-regular-metrics))

2. NFS metrics (section [G.1.2](#g12-nfs-metrics))

3. InfiniBand metrics (section [G.1.3](#g13-infiniband-metrics))

4. monitoring system metrics (section [G.1.4](#g14-monitoring-system-metrics))

5. GPU metrics (section [G.1.6](#g16-gpu-metrics))

6. Job metrics (section [G.1.8](#g18-job-metrics))

7. IPMI metrics (section [G.1.9](#g19-ipmi-metrics))

8. Redfish metrics (section [G.1.10](#g110-redfish-metrics))

9. SMART metrics (section [G.1.11](#g111-smart-metrics))

10. Prometheus metrics (section [G.1.12](#g112-prometheus-metrics))

### G.1.1  Regular Metrics

<table>
  <thead>
    <tr>
      <th colspan="2">*Table  [G.1.1](#g11-regular-metrics): List Of Metrics*</th>
    </tr>
  </thead>

  <tbody>
    <tr>
      <td>
        **Metric**
      </td>

      <td>
        **Description**
      </td>
    </tr>

    <tr>
      <td>
        `AlertLevel`
      </td>

      <td>
        Indicates the healthiness of a device based on severity of events (section 

        [10.2.8](/admin/monitoring#1028-alertlevel)

        ). The lower it is, the better. There are 3 parameters it can take: 

        \


        • count: the number of active triggers

        \


        • maximum: the maximum alert level of all active triggers

        \


        • sum: the summed alert level of all active triggers
      </td>
    </tr>

    <tr>
      <td>
        `BlockedProcesses`
      </td>

      <td>
        Blocked processes waiting for I/O
      </td>
    </tr>

    <tr>
      <td>
        `BufferMemory`
      </td>

      <td>
        System memory used for buffering
      </td>
    </tr>

    <tr>
      <td>
        `BytesRecv`

        *,‡
      </td>

      <td>
        Bytes/s received
      </td>
    </tr>

    <tr>
      <td>
        `BytesSent`

        *,‡
      </td>

      <td>
        Bytes/s sent
      </td>
    </tr>

    <tr>
      <td>
        `CPUGuest`

        \*
      </td>

      <td>
        CPU time spent in guest mode (Jiffies/s)
      </td>
    </tr>

    <tr>
      <td>
        `CPUIdle`

         

        \*
      </td>

      <td>
        CPU time spent in idle mode (Jiffies/s)
      </td>
    </tr>

    <tr>
      <td>
        `CPUIrq`

         

        \*
      </td>

      <td>
        CPU time spent in servicing IRQ (Jiffies/s)
      </td>
    </tr>

    <tr>
      <td>
        `CPUNice`

         

        \*
      </td>

      <td>
        CPU time spent in nice mode (Jiffies/s)
      </td>
    </tr>

    <tr>
      <td>
        `CPUSoftIrq`

         

        \*
      </td>

      <td>
        CPU time spent in servicing soft IRQ (Jiffies/s)
      </td>
    </tr>

    <tr>
      <td>
        `CPUSteal`

         

        \*
      </td>

      <td>
        CPU time spent in steal mode (Jiffies/s)
      </td>
    </tr>

    <tr>
      <td>
        `CPUSystem`

         

        \*
      </td>

      <td>
        CPU time spent in system mode (Jiffies/s)
      </td>
    </tr>

    <tr>
      <td>
        `CPUUser`

         

        \*
      </td>

      <td>
        CPU time spent in user mode (Jiffies/s)
      </td>
    </tr>

    <tr>
      <td>
        `CPUUsage`

         

        \*
      </td>

      <td>
        Percent of time not spent in idle mode (sum of non-idling percentages) (%/s)
      </td>
    </tr>

    <tr>
      <td>
        `CPUWait`

         

        \*
      </td>

      <td>
        CPU time spent in I/O wait mode (Jiffies/s).
      </td>
    </tr>

    <tr>
      <td>
        `CacheMemory`
      </td>

      <td>
        System memory used for caching.
      </td>
    </tr>

    <tr>
      <td>
        `Cores`
      </td>

      <td>
        Number of cores for a node
      </td>
    </tr>

    <tr>
      <td>
        `CoresDown`
      </td>

      <td>
        Number of cores for all nodes marked as DOWN
      </td>
    </tr>

    <tr>
      <td>
        `CoresTotal`
      </td>

      <td>
        Total number of known cores for all nodes
      </td>
    </tr>

    <tr>
      <td>
        `CoresUp`
      </td>

      <td>
        Number of cores for all nodes marked as UP
      </td>
    </tr>

    <tr>
      <td>
        `CtxtSwitches`

        \*
      </td>

      <td>
        Context switches/s
      </td>
    </tr>

    <tr>
      <td>
        `DPUNodesClosed`
      </td>

      <td>
        Number of DPUs not marked as UP or DOWN
      </td>
    </tr>

    <tr>
      <td>
        `DPUNodesDown`
      </td>

      <td>
        Number of DPUs marked as DOWN
      </td>
    </tr>

    <tr>
      <td>
        `DPUNodesTotal`
      </td>

      <td>
        Total number of DPUs
      </td>
    </tr>

    <tr>
      <td>
        `DPUNodesUp`
      </td>

      <td>
        Number of DPUs not marked as UP or DOWN
      </td>
    </tr>

    <tr>
      <td>
        `DevicesClosed`
      </td>

      <td>
        Number of devices not marked as UP or DOWN
      </td>
    </tr>

    <tr>
      <td>
        `DevicesDown`
      </td>

      <td>
        Number of devices marked as DOWN
      </td>
    </tr>

    <tr>
      <td>
        `DevicesTotal`
      </td>

      <td>
        Total number of devices
      </td>
    </tr>

    <tr>
      <td>
        `DevicesUp`
      </td>

      <td>
        Number of devices in status UP. A node (head, regular, virtual, cloud) or GPU Unit is not classed as a device. A device can be an item such as a switch, PDU, chassis, or rack, if the item is enabled and configured for management.
      </td>
    </tr>

    <tr>
      <td>
        `DropRecv`

        *,‡
      </td>

      <td>
        Packets/s received and dropped
      </td>
    </tr>

    <tr>
      <td>
        `DropSent`

        *,‡
      </td>

      <td>
        Packets/s sent and dropped
      </td>
    </tr>

    <tr>
      <td>
        `EC2SpotPrice`
      </td>

      <td>
        Amazon EC2 price for spot instances
      </td>
    </tr>

    <tr>
      <td>
        `EccDBitGPU`

        \*

        \*
      </td>

      <td>
        Total number of double bit ECC errors/s (file: 

        `sample_gpu`

        )
      </td>
    </tr>

    <tr>
      <td>
        `EccSBitGPU`

        \*

        \*
      </td>

      <td>
        Total number of single bit ECC errors/s (file: 

        `sample_gpu`

        )
      </td>
    </tr>

    <tr>
      <td>
        `ErrorsRecv`

        *,‡
      </td>

      <td>
        Packets/s received with error
      </td>
    </tr>

    <tr>
      <td>
        `ErrorsSent`

        *,‡
      </td>

      <td>
        Packets/s sent with error
      </td>
    </tr>

    <tr>
      <td>
        `FPGAsDown`
      </td>

      <td>
        Number of FPGAs for all nodes marked as DOWN
      </td>
    </tr>

    <tr>
      <td>
        `FPGAsTotal`
      </td>

      <td>
        Total number of known FPGAs for all nodes
      </td>
    </tr>

    <tr>
      <td>
        `FPGAsUp`
      </td>

      <td>
        Number of FPGAs for all nodes marked as UP
      </td>
    </tr>

    <tr>
      <td>
        `FabricTopologies`
      </td>

      <td>
        Number of fabric topologies
      </td>
    </tr>

    <tr>
      <td>
        `FabricTopologyHostUsage`
      </td>

      <td>
        Average usage of all topology hosts
      </td>
    </tr>

    <tr>
      <td>
        `FabricTopologyResourceBoxUsage`
      </td>

      <td>
        Average usage of all topology resource boxes
      </td>
    </tr>

    <tr>
      <td>
        `Forks`

        \*
      </td>

      <td>
        Forked processes/s
      </td>
    </tr>

    <tr>
      <td>
        `FrameErrors`

        *,‡
      </td>

      <td>
        Packet framing errors/s
      </td>
    </tr>

    <tr>
      <td>
        `FreeFiles`

        \[§]
      </td>

      <td>
        Free file inodes on the specified mount point
      </td>
    </tr>

    <tr>
      <td>
        `FreeSpace`

        \[§]
      </td>

      <td>
        Free space for non-root user. Takes mount point as a parameter
      </td>
    </tr>

    <tr>
      <td>
        `GPUUnitsClosed`
      </td>

      <td>
        Number of GPU units not marked as UP or DOWN
      </td>
    </tr>

    <tr>
      <td>
        `GPUUnitsDown`
      </td>

      <td>
        Number of GPU units marked as DOWN
      </td>
    </tr>

    <tr>
      <td>
        `GPUUnitsTotal`
      </td>

      <td>
        Total number of GPU Units
      </td>
    </tr>

    <tr>
      <td>
        `GPUUnitsUp`
      </td>

      <td>
        Number of GPU units marked as UP
      </td>
    </tr>

    <tr>
      <td>
        `GPUsTotal`
      </td>

      <td>
        Total number of known GPUs for all nodes
      </td>
    </tr>

    <tr>
      <td>
        `GPUsUp`
      </td>

      <td>
        Number of GPUs for all nodes marked as UP
      </td>
    </tr>

    <tr>
      <td>
        `GPUsDown`
      </td>

      <td>
        Number of GPUs for all nodes marked as DOWN
      </td>
    </tr>

    <tr>
      <td>
        `HardwareCorruptedMemory`
      </td>

      <td>
        Hardware corrupted memory detected by ECC
      </td>
    </tr>

    <tr>
      <td>
        `IOInProgress`

        \[†]
      </td>

      <td>
        I/O operations in progress
      </td>
    </tr>

    <tr>
      <td>
        `IOTime`

        *,†
      </td>

      <td>
        I/O operations time in milliseconds/s
      </td>
    </tr>

    <tr>
      <td>
        `InterfaceState`
      </td>

      <td>
        Interface operation state
      </td>
    </tr>

    <tr>
      <td>
        `IpForwDatagrams`

        \*
      </td>

      <td>
        Input IP datagrams/s to be forwarded/s
      </td>
    </tr>

    <tr>
      <td>
        `IpFragCreates`

        \*
      </td>

      <td>
        IP datagram fragments/s generated/s
      </td>
    </tr>

    <tr>
      <td>
        `IpFragFails`

        \*
      </td>

      <td>
        IP datagrams/s which needed to be fragmented but could not
      </td>
    </tr>

    <tr>
      <td>
        `IpFragOKs`

        \*
      </td>

      <td>
        IP datagrams/s successfully fragmented
      </td>
    </tr>

    <tr>
      <td>
        `IpInAddrErrors`

        \*
      </td>

      <td>
        Input datagrams/s discarded because the IP address in their header was not a valid address
      </td>
    </tr>

    <tr>
      <td>
        `IpInDelivers`

        \*
      </td>

      <td>
        Input IP datagrams/s successfully delivered
      </td>
    </tr>

    <tr>
      <td>
        `IpInDiscards`

        \*
      </td>

      <td>
        Input IP datagrams/s discarded
      </td>
    </tr>

    <tr>
      <td>
        `IpInHdrErrors`

        \*
      </td>

      <td>
        Input IP datagrams/s discarded due to errors in their IP headers
      </td>
    </tr>

    <tr>
      <td>
        `IpInReceives`

        \*
      </td>

      <td>
        Input IP datagrams/s, including ones with errors, received from all interfaces
      </td>
    </tr>

    <tr>
      <td>
        `IpInUnknownProtos`

        \*
      </td>

      <td>
        Input IP datagrams/s received but discarded due to an unknown or unsupported protocol
      </td>
    </tr>

    <tr>
      <td>
        `IpOutDiscards`

        \*
      </td>

      <td>
        Output IP datagrams/s discarded
      </td>
    </tr>

    <tr>
      <td>
        `IpOutNoRoutes`

        \*
      </td>

      <td>
        Output IP datagrams/s discarded because no route could be found
      </td>
    </tr>

    <tr>
      <td>
        `IpOutRequests`

        \*
      </td>

      <td>
        Output IP datagrams/s supplied to IP in requests for transmission
      </td>
    </tr>

    <tr>
      <td>
        `IpReasmOKs`

        \*
      </td>

      <td>
        IP datagrams/s successfully re-assembled
      </td>
    </tr>

    <tr>
      <td>
        `IpReasmReqds`

        \*
      </td>

      <td>
        IP fragments/s received needing re-assembly
      </td>
    </tr>

    <tr>
      <td>
        `JobsRunning`
      </td>

      <td>
        Jobs running on the node
      </td>
    </tr>

    <tr>
      <td>
        `LiteNodesClosed`
      </td>

      <td>
        Number of lite nodes not marked as UP or DOWN
      </td>
    </tr>

    <tr>
      <td>
        `LiteNodesDown`
      </td>

      <td>
        Number of lite nodes marked as DOWN
      </td>
    </tr>

    <tr>
      <td>
        `LiteNodesTotal`
      </td>

      <td>
        Total number of lite nodes
      </td>
    </tr>

    <tr>
      <td>
        `LiteNodesUp`
      </td>

      <td>
        Number of lite nodes marked as UP
      </td>
    </tr>

    <tr>
      <td>
        `LoadFifteen`
      </td>

      <td>
        Load average on 15 minutes
      </td>
    </tr>

    <tr>
      <td>
        `LoadFive`
      </td>

      <td>
        Load average on 5 minutes
      </td>
    </tr>

    <tr>
      <td>
        `LoadOne`
      </td>

      <td>
        Load average on 1 minute 
      </td>
    </tr>

    <tr>
      <td>
        `MajorPageFaults`

        \*
      </td>

      <td>
        Page faults/s that require I/O
      </td>
    </tr>

    <tr>
      <td>
        `ManagedServicesOk`
      </td>

      <td>
        This metric uses the 

        `ManagedServicesOk`

         health check (

        [here](#g21-regular-health-checks)

        ) for a grouping of nodes, such as the nodes in a category, or the nodes in a cluster. A parameter is specified for the metric, and a value is returned, as follows: 

        \


        • fail: Total number of health checks in a FAIL state

        \


        • good: Percentage that are in the PASS state

        \


        • pass: Total number that are in the PASS state

        \


        • total: Total number of the ManagedServicesOk metrics being sampled, regardless of state

        \


        • unknown: Total number that are in the UNKNOWN state
      </td>
    </tr>

    <tr>
      <td>
        `ManagedSwitchesClosed`
      </td>

      <td>
        Number of managed switches not marked as UP or DOWN
      </td>
    </tr>

    <tr>
      <td>
        `ManagedSwitchesDown`
      </td>

      <td>
        Number of managed switches marked as DOWN
      </td>
    </tr>

    <tr>
      <td>
        `ManagedSwitchesTotal`
      </td>

      <td>
        Number of managed switches not marked as UP or DOWN
      </td>
    </tr>

    <tr>
      <td>
        `ManagedSwitchesUp`
      </td>

      <td>
        Total number of managed switches
      </td>
    </tr>

    <tr>
      <td>
        `MemoryAvailable`
      </td>

      <td>
        Available system memory
      </td>
    </tr>

    <tr>
      <td>
        `MemoryFree`
      </td>

      <td>
        Free system memory
      </td>
    </tr>

    <tr>
      <td>
        `MemoryTotal`
      </td>

      <td>
        Total system memory
      </td>
    </tr>

    <tr>
      <td>
        `MemoryUsed`

        \[¶]
      </td>

      <td>
        Used system memory for a process specified as a parameter. Processes can be anything that is always running: 

        \


        **Example**

         

        \


        • cm-lite-daemon

        \


        • cm-mqtt

        \


        • cmd

        \


        • mysqld

        \


        • slurmctld

        \


        • slurmd
      </td>
    </tr>

    <tr>
      <td>
        `MemoryUtilization`
      </td>

      <td>
        Memory utilization
      </td>
    </tr>

    <tr>
      <td>
        `MergedReads`

        *,†
      </td>

      <td>
        Merged reads/s
      </td>
    </tr>

    <tr>
      <td>
        `MergedWrites`

        *,†
      </td>

      <td>
        Merged writes/s
      </td>
    </tr>

    <tr>
      <td>
        `NodesClosed`
      </td>

      <td>
        Number of nodes not marked as UP or DOWN
      </td>
    </tr>

    <tr>
      <td>
        `NodesDown`
      </td>

      <td>
        Number of nodes marked as DOWN
      </td>
    </tr>

    <tr>
      <td>
        `NodesTotal`
      </td>

      <td>
        Total number of nodes
      </td>
    </tr>

    <tr>
      <td>
        `NodesUp`
      </td>

      <td>
        Number of nodes in status UP
      </td>
    </tr>

    <tr>
      <td rowspan="3">
        `OccupationRate`
      </td>

      <td>
        Cluster occupation rate—a normalized cluster load percentage. 100% means all cores on all nodes are fully loaded.
      </td>
    </tr>

    <tr>
      <td>
        The calculation is done as follows: 

        `LoadOne`

         on each node is mapped to a value, calibrated so that 

        `LoadOne`

        \=1 corresponds to 100% per node. The maximum allowed for a node in the mapping is 100%. The average of these mappings taken over all nodes is the 

        `OccupationRate`

        .
      </td>
    </tr>

    <tr>
      <td>
        A high value can indicate the cluster is being used optimally. However, a value that is 100% most of the time suggests the cluster may need to be expanded.
      </td>
    </tr>

    <tr>
      <td>
        `OOM kill`
      </td>

      <td>
        Number of processes killed by OOM killer since boot
      </td>
    </tr>

    <tr>
      <td>
        `PacketsRecv`

        *,‡
      </td>

      <td>
        Packets/s received
      </td>
    </tr>

    <tr>
      <td>
        `PacketsSent`

        *,‡
      </td>

      <td>
        Packets/s sent
      </td>
    </tr>

    <tr>
      <td>
        `PageFaults`

        \*
      </td>

      <td>
        Page faults/s
      </td>
    </tr>

    <tr>
      <td>
        `PageIn`

         

        \*
      </td>

      <td>
        Number of bytes the system has paged in from disk/s
      </td>
    </tr>

    <tr>
      <td>
        `PageOut`

         

        \*
      </td>

      <td>
        Number of bytes the system has paged out to disk/s
      </td>
    </tr>

    <tr>
      <td>
        `PageSwapIn`

         

        \*
      </td>

      <td>
        Number of bytes the system has swapped in from disk/s
      </td>
    </tr>

    <tr>
      <td>
        `PageSwapOut`

         

        \*
      </td>

      <td>
        Number of bytes the system has swapped out to disk/s
      </td>
    </tr>

    <tr>
      <td>
        `PDUBankLoad`
      </td>

      <td>
        Total PDU bank load, in amps
      </td>
    </tr>

    <tr>
      <td>
        `PDULoad`
      </td>

      <td>
        Total PDU phase load, in amps
      </td>
    </tr>

    <tr>
      <td>
        `PDUUptime`

        \*
      </td>

      <td>
        PDU uptime per second. I.e. ideally=1, but in practice has jitter effects.
      </td>
    </tr>

    <tr>
      <td>
        `PhaseLoad`
      </td>

      <td>
        Sum of PDULoad over all power distribution units
      </td>
    </tr>

    <tr>
      <td>
        `ProcessCount`
      </td>

      <td>
        [Total number of all processes in the OS. These are running processes (`RunningProcesses`) and blocked processes(`BlockedProcesses`).](https://linux.die.net/lkmpg/x1052.html)
      </td>
    </tr>

    <tr>
      <td>
        `ReadOnly`

        \[§]
      </td>

      <td>
        Indicates if the specified mount point was mounted as read-only
      </td>
    </tr>

    <tr>
      <td>
        `ReadTime`

        *,†
      </td>

      <td>
        Read time in milliseconds/s
      </td>
    </tr>

    <tr>
      <td>
        `Reads`

        *,†
      </td>

      <td>
        Reads/s completed successfully
      </td>
    </tr>

    <tr>
      <td>
        `RunningProcesses`
      </td>

      <td>
        Running processes
      </td>
    </tr>

    <tr>
      <td>
        `ReportedSpeed`
      </td>

      <td>
        Speed reported by the port of the switch. The parameter name used is the hardware port name, as specified by the output of the 

        `switchoverview`

         command. For example, for the Cumulus switch these can be 

        `swp0`

        , 

        `swp1`

        ...
      </td>
    </tr>

    <tr>
      <td>
        `SectorsRead`

        *,†
      </td>

      <td>
        Sectors/s read successfully/s
      </td>
    </tr>

    <tr>
      <td>
        `SectorsWritten`

        *,†
      </td>

      <td>
        Sectors/s written successfully
      </td>
    </tr>

    <tr>
      <td>
        `SwapCached`
      </td>

      <td>
        Cached swap memory
      </td>
    </tr>

    <tr>
      <td>
        `SwapFree`
      </td>

      <td>
        Free swap memory
      </td>
    </tr>

    <tr>
      <td>
        `SwapTotal`
      </td>

      <td>
        Total swap memory
      </td>
    </tr>

    <tr>
      <td>
        `SwapUsed`
      </td>

      <td>
        Used swap memory
      </td>
    </tr>

    <tr>
      <td>
        `SwapUtilization`
      </td>

      <td>
        Swap memory utilization
      </td>
    </tr>

    <tr>
      <td>
        `SystemTime`

        \[¶]
      </td>

      <td>
        System time used by process specified by a parameter, for example: 

        `cmd`
      </td>
    </tr>

    <tr>
      <td>
        `SwitchBroadcastPackets`

        \*
      </td>

      <td>
        Total number of good packets received and directed to the broadcast address/s
      </td>
    </tr>

    <tr>
      <td>
        `SwitchCollisions`

        \*
      </td>

      <td>
        Collisions/s on this network segment
      </td>
    </tr>

    <tr>
      <td>
        `SwitchCPUUsage`
      </td>

      <td>
        Switch CPU utilization estimation (%)
      </td>
    </tr>

    <tr>
      <td>
        `SwitchDelayDiscardFrames`

        \*
      </td>

      <td>
        Frames discarded/s due to excessive transit delay through the bridge
      </td>
    </tr>

    <tr>
      <td>
        `SwitchFilterDiscardFrames`

        \*
      </td>

      <td>
        Valid frames received/s but discarded by the forwarding process
      </td>
    </tr>

    <tr>
      <td>
        `SwitchMTUDiscardFrames`

        \*
      </td>

      <td>
        Number of frames discarded/s due to an excessive size
      </td>
    </tr>

    <tr>
      <td>
        `SwitchMulticastPackets`

        \*
      </td>

      <td>
        Total number of good packets/s received and directed to a multicast address
      </td>
    </tr>

    <tr>
      <td>
        `SwitchOverSizedPackets`

        \*
      </td>

      <td>
        Well-received packets/s longer than 1518 octets
      </td>
    </tr>

    <tr>
      <td>
        `SwitchUnderSizedPackets`

        \*
      </td>

      <td>
        Packets/s received which are less than 64 octets long
      </td>
    </tr>

    <tr>
      <td>
        `SwitchUptime`

        \*
      </td>

      <td>
        Switch uptime per second. Ie, ideally=1, but in practice has jitter effects
      </td>
    </tr>

    <tr>
      <td>
        `TcpCurrEstab`
      </td>

      <td>
        TCP connections that are either ESTABLISHED or CLOSE-WAIT
      </td>
    </tr>

    <tr>
      <td>
        `TcpInErrs`

        \*
      </td>

      <td>
        Input IP segments/s received in error
      </td>
    </tr>

    <tr>
      <td>
        `TcpRetransSegs`

        \*
      </td>

      <td>
        Total number of IP segments/s re-transmitted
      </td>
    </tr>

    <tr>
      <td>
        `ThreadsUsed`

        \[¶]
      </td>

      <td>
        Threads used by process. For example: 

        `cmd`
      </td>
    </tr>

    <tr>
      <td>
        `TotalBytesRecv`
      </td>

      <td>
        Total bytes received on swp interfaces over all managed switches
      </td>
    </tr>

    <tr>
      <td>
        `TotalBytesSent`
      </td>

      <td>
        Total bytes sent on swp interfaces over all managed switches
      </td>
    </tr>

    <tr>
      <td>
        `TotalCPUIdle`
      </td>

      <td>
        Cluster-wide core usage in idle tasks (sum of all CPUIdle metric percentages)
      </td>
    </tr>

    <tr>
      <td>
        `TotalCPUPowerUsage`
      </td>

      <td>
        Total CPU power usage over all nodes
      </td>
    </tr>

    <tr>
      <td>
        `TotalCPUSystem`
      </td>

      <td>
        Cluster-wide core usage in system mode (sum of all CPUSystem metric percentages)
      </td>
    </tr>

    <tr>
      <td>
        `TotalCPUTemperature`
      </td>

      <td>
        Average CPU temperature over all nodes
      </td>
    </tr>

    <tr>
      <td>
        `TotalCPUUser`
      </td>

      <td>
        Cluster-wide core usage in user mode (sum of all CPUUser metric percentages)
      </td>
    </tr>

    <tr>
      <td>
        `TotalCPUUtilization`
      </td>

      <td>
        Sum of CPUUsage over all nodes
      </td>
    </tr>

    <tr>
      <td>
        `TotalGPUMemoryUtilization`
      </td>

      <td>
        Average of GPU memory utilization percentage 

        `gpu_mem_utilization:average`

         (table 

        [G.1.6](#g16-gpu-metrics)

        ) over all nodes
      </td>
    </tr>

    <tr>
      <td>
        `TotalGPUNvlinkBandwidth`
      </td>

      <td>
        Total GPU Nvlink bandwidth using 

        `gpu_nvlink_total_bandwidth:total`

         (table 

        [G.1.6](#g16-gpu-metrics)

        ) over all nodes
      </td>
    </tr>

    <tr>
      <td>
        `TotalGPUPowerUsage`
      </td>

      <td>
        Total GPU power usage, using 

        `gpu_power_usage:total`

         (table 

        [G.1.6](#g16-gpu-metrics)

        ) over all nodes
      </td>
    </tr>

    <tr>
      <td>
        `TotalGPUTemperature`
      </td>

      <td>
        Average of average GPU temperature using 

        `gpu_temperature:average`

         (table 

        [G.1.6](#g16-gpu-metrics)

        ) over all nodes
      </td>
    </tr>

    <tr>
      <td>
        `TotalGPUUtilization`
      </td>

      <td>
        Average of GPU utilization using 

        `gpu_utilization:average`

         (table 

        [G.1.6](#g16-gpu-metrics)

        ) over all nodes
      </td>
    </tr>

    <tr>
      <td>
        `TotalMemory`
      </td>

      <td>
        Sum of 

        `MemoryTotal`

         over all nodes
      </td>
    </tr>

    <tr>
      <td>
        `TotalMemoryFree`
      </td>

      <td>
        Cluster-wide total of memory free
      </td>
    </tr>

    <tr>
      <td>
        `TotalMemoryUsed`
      </td>

      <td>
        Cluster-wide total of memory used
      </td>
    </tr>

    <tr>
      <td>
        `TotalMemoryUtilization`
      </td>

      <td>
        Sum of MemoryUtilization over all nodes
      </td>
    </tr>

    <tr>
      <td>
        `TotalNodePowerUsage`
      </td>

      <td>
        Total power usage over all nodes
      </td>
    </tr>

    <tr>
      <td>
        `TotalSwap`
      </td>

      <td>
        Sum of 

        `SwapTotal`

         over all nodes
      </td>
    </tr>

    <tr>
      <td>
        `TotalSwapFree`
      </td>

      <td>
        Cluster-wide total swap free
      </td>
    </tr>

    <tr>
      <td>
        `TotalSwapUsed`
      </td>

      <td>
        Cluster-wide total swap used
      </td>
    </tr>

    <tr>
      <td>
        `TotalUser`
      </td>

      <td>
        Total number of known users
      </td>
    </tr>

    <tr>
      <td>
        `TotalUserLogin`
      </td>

      <td>
        Total number of logged in users
      </td>
    </tr>

    <tr>
      <td>
        `UdpInDatagrams`

        \*
      </td>

      <td>
        Input UDP datagrams/s delivered to UDP users
      </td>
    </tr>

    <tr>
      <td>
        `UdpInErrors`

        \*
      </td>

      <td>
        Input UDP datagrams/s received that could not be delivered/s for other reasons (no port excl.)
      </td>
    </tr>

    <tr>
      <td>
        `UdpNoPorts`

        \*
      </td>

      <td>
        Received UDP datagrams/s for which there was no application at the destination port
      </td>
    </tr>

    <tr>
      <td>
        `UniqueUserLogin`
      </td>

      <td>
        Number of unique users logged in
      </td>
    </tr>

    <tr>
      <td>
        `Uptime`

        \*
      </td>

      <td>
        System uptime per second. Ie, ideally=1, but in practice has jitter effects
      </td>
    </tr>

    <tr>
      <td>
        `UsedFiles`

        \[§]
      </td>

      <td>
        Used file inodes on the specified parameter as mount point
      </td>
    </tr>

    <tr>
      <td>
        `UsedSpace`

        \[§]
      </td>

      <td>
        Used space on the specified parameter as mount point
      </td>
    </tr>

    <tr>
      <td>
        `UserTime`

        \[¶]
      </td>

      <td>
        User time used by specified process. For example: 

        `cmd`
      </td>
    </tr>

    <tr>
      <td>
        `Utilization`

        \[¶]
      </td>

      <td>
        Utilization by specified process. For example; 

        `cmd`
      </td>
    </tr>

    <tr>
      <td>
        `VirtualMemoryUsed`

        \[¶]
      </td>

      <td>
        Virtual memory used by specified process. For example 

        `cmd`
      </td>
    </tr>

    <tr>
      <td>
        `WlmSlotsFree`
      </td>

      <td>
        The number of WLM slots free
      </td>
    </tr>

    <tr>
      <td>
        `WlmSlotsTotal`
      </td>

      <td>
        The total number of WLM slots
      </td>
    </tr>

    <tr>
      <td>
        `WlmSlotsUsed`
      </td>

      <td>
        The number of WLM slots in use
      </td>
    </tr>

    <tr>
      <td>
        `WlmSlotsUtilization`
      </td>

      <td>
        The percentage of WLM slots in use
      </td>
    </tr>

    <tr>
      <td>
        `wlm_slurm_state_count`
      </td>

      <td>
        Number of nodes in the state for a parameter. The possible parameters are: 

        \


        • allocated

        \


        • completing

        \


        • down

        \


        • drain

        \


        • draining

        \


        • fail

        \


        • failing

        \


        • idle

        \


        • maint

        \


        • mixed
      </td>
    </tr>

    <tr>
      <td>
        `WriteTime`

        *,†
      </td>

      <td>
        Write time in milliseconds/s (this is a per mille), for the parameter as device
      </td>
    </tr>

    <tr>
      <td>
        `Writes`

        *,†
      </td>

      <td>
        Writes/s completed successfully for the parameter as device
      </td>
    </tr>

    <tr>
      <td>
        `isilon_node_disk_access_latency`
      </td>

      <td>
        Isilon access latency for node disk
      </td>
    </tr>

    <tr>
      <td>
        `isilon_node_disk_iosched_queue`
      </td>

      <td>
        Isilon iosched queue for node disk
      </td>
    </tr>

    <tr>
      <td>
        `isilon_node_disk_xfer_size_in`
      </td>

      <td>
        Transfer size in for node disk
      </td>
    </tr>

    <tr>
      <td>
        `isilon_node_disk_xfer_size_out`
      </td>

      <td>
        Transfer size out for node disk
      </td>
    </tr>

    <tr>
      <td>
        `isilon_node_ip`
      </td>

      <td>
        IP address of isilon node
      </td>
    </tr>

    <tr>
      <td>
        `isilon_in_rate`
      </td>

      <td>
        Bytes written to NFS client from Isilon
      </td>
    </tr>

    <tr>
      <td>
        `isilon_out_rate`
      </td>

      <td>
        Bytes read from Isilon to NFS client
      </td>
    </tr>

    <tr>
      <td>
        `isilon_op_rate`
      </td>

      <td>
        I/O rate for NFS client to/from Isilon
      </td>
    </tr>

    <tr>
      <td>
        `node_network_carrier_changes_total`
      </td>

      <td>
        Samples total network carrier changes
      </td>
    </tr>

    <tr>
      <td>
        `nvidia_licensed_compute_resources`
      </td>

      <td>
        Total licensed compute resources
      </td>
    </tr>

    <tr>
      <td>
        `nvidia_used_compute_resources`
      </td>

      <td>
        Used compute resources, (maximum of nodes used and GPUs used, but value can only be up to the total licensed compute resources)
      </td>
    </tr>

    <tr>
      <td>
        `nvidia_used_gpu_resources`
      </td>

      <td>
        Used GPU resources
      </td>
    </tr>

    <tr>
      <td>
        `nvidia_used_node_resources`
      </td>

      <td>
        Used node resources
      </td>
    </tr>

    <tr>
      <td>
        `nvidia_used_node_resources`
      </td>

      <td>
        Used node resources
      </td>
    </tr>
  </tbody>
</table>

\* Cumulative metric. I.e. the metric is derived from cumulative raw measurements taken at two different times, according to:

metric\[time\[2]] = ((measurement\[2] - measurement\[1]) / (time\[2] - time\[1]))

The metric is a “per second” measurement.

\*\* Standalone scripts, not built-ins.

If sampling from a head node, the script is in directory: `/cm/local/apps/cmd/scripts/metrics/`

For regular nodes, the script is in directory: `/cm/images/default-image/cm/local/apps/cmd/scripts/metrics/`

† Takes block device name (`sda`, `sdc`, `nvme0n1` and so on) as parameter

‡ Takes interface device name (`eth0`, `eth1`, `en01`, `docker0`, `ib0` and so on) as parameter

§ Takes mount point (for example: `/`, or `/var`) as parameter

¶ Takes a process, eg: `cmd` as a parameter

### G.1.2  NFS Metrics

The NFS metrics are all cumulative. They correspond to `nfsstat` output, and are shown in table [G.1.2](#g12-nfs-metrics).

<table>
  <tbody>
    <tr>
      <td>
        `nfs_client_packet_packets`
      </td>

      <td>
        NFS client packets statistics: packets
      </td>
    </tr>

    <tr>
      <td>
        `nfs_client_packet_tcp`
      </td>

      <td>
        NFS client package statistics: TCP/IP packets
      </td>
    </tr>

    <tr>
      <td>
        `nfs_client_packet_tcpconn`
      </td>

      <td>
        NFS client package statistics: TCP/IP connections
      </td>
    </tr>

    <tr>
      <td>
        `nfs_client_packet_udp`
      </td>

      <td>
        NFS client package statistics: UDP packets
      </td>
    </tr>

    <tr>
      <td>
        `nfs_client_rpc_authrefrsh`
      </td>

      <td>
        NFS Client RPC statistics: authenticated refreshes to RPC server
      </td>
    </tr>

    <tr>
      <td>
        `nfs_client_rpc_calls`
      </td>

      <td>
        NFS Client RPC statistics: calls
      </td>
    </tr>

    <tr>
      <td>
        `nfs_client_rpc_retrans`
      </td>

      <td>
        NFS Client RPC statistics: re-transmissions
      </td>
    </tr>

    <tr>
      <td>
        `nfs_server_file_anon`
      </td>

      <td>
        NFS Server file statistics: anonymous access
      </td>
    </tr>

    <tr>
      <td>
        `nfs_server_file_lookup`
      </td>

      <td>
        NFS Server file statistics: look-ups
      </td>
    </tr>

    <tr>
      <td>
        `nfs_server_file_ncachedir`
      </td>

      <td>
        NFS Server file statistics: ncachedir
      </td>
    </tr>

    <tr>
      <td>
        `nfs_server_file_stale`
      </td>

      <td>
        NFS Server file statistics: stale files
      </td>
    </tr>

    <tr>
      <td>
        `nfs_server_packet_packets`
      </td>

      <td>
        NFS Server packet statistics: packets
      </td>
    </tr>

    <tr>
      <td>
        `nfs_server_packet_tcp`
      </td>

      <td>
        NFS Server packet statistics: TCP/IP packets
      </td>
    </tr>

    <tr>
      <td>
        `nfs_server_packet_tcpconn`
      </td>

      <td>
        NFS Server packet statistics: TCP/IP connections
      </td>
    </tr>

    <tr>
      <td>
        `nfs_server_packet_udp`
      </td>

      <td>
        NFS Server packet statistics: UDP packets
      </td>
    </tr>

    <tr>
      <td>
        `nfs_server_reply_hits`
      </td>

      <td>
        NFS Server reply statistics: hits
      </td>
    </tr>

    <tr>
      <td>
        `nfs_server_reply_misses`
      </td>

      <td>
        NFS Server reply statistics: misses
      </td>
    </tr>

    <tr>
      <td>
        `nfs_server_reply_nocache`
      </td>

      <td>
        NFS Server reply statistics: no cache
      </td>
    </tr>

    <tr>
      <td>
        `nfs_server_rpc_badauth`
      </td>

      <td>
        NFS Server RPC statistics: bad authentication
      </td>
    </tr>

    <tr>
      <td>
        `nfs_server_rpc_badcalls`
      </td>

      <td>
        NFS Server RPC statistics:bad RPC requests
      </td>
    </tr>

    <tr>
      <td>
        `nfs_server_rpc_badclnt`
      </td>

      <td>
        NFS Server RPC statistics: badclnt
      </td>
    </tr>

    <tr>
      <td>
        `nfs_server_rpc_calls`
      </td>

      <td>
        NFS Server RPC statistics: all calls to NFS and NLM
      </td>
    </tr>

    <tr>
      <td>
        `nfs_v3_client_access`
      </td>

      <td>
        NFSv3 client statistics: access
      </td>
    </tr>

    <tr>
      <td>
        `nfs_v3_client_create`
      </td>

      <td>
        NFSv3 client statistics: create
      </td>
    </tr>

    <tr>
      <td>
        `nfs_v3_client_fsinfo`
      </td>

      <td>
        NFSv3 client statistics: static file system information
      </td>
    </tr>

    <tr>
      <td>
        `nfs_v3_client_fsstat`
      </td>

      <td>
        NFSv3 client statistics: dynamic file system status
      </td>
    </tr>

    <tr>
      <td>
        `nfs_v3_client_getattr`
      </td>

      <td>
        NFSv3 client statistics: file system attributes
      </td>
    </tr>

    <tr>
      <td>
        `nfs_v3_client_lookup`
      </td>

      <td>
        NFSv3 client statistics: lookup
      </td>
    </tr>

    <tr>
      <td>
        `nfs_v3_client_pathconf`
      </td>

      <td>
        NFSv3 client statistics: configuration path
      </td>
    </tr>

    <tr>
      <td>
        `nfs_v3_client_read`
      </td>

      <td>
        NFSv3 client statistics: reads
      </td>
    </tr>

    <tr>
      <td>
        `nfs_v3_client_read_readdirplus`
      </td>

      <td>
        NFSv3 client statistics: readdirplus
      </td>
    </tr>

    <tr>
      <td>
        `nfs_v3_client_remove`
      </td>

      <td>
        NFSv3 client statistics: removes
      </td>
    </tr>

    <tr>
      <td>
        `nfs_v3_client_settattr`
      </td>

      <td>
        NFSv3 client statistics: setattr
      </td>
    </tr>

    <tr>
      <td>
        `nfs_v3_client_total`
      </td>

      <td>
        NFSv3 client statistics: total
      </td>
    </tr>

    <tr>
      <td>
        `nfs_v3_client_write`
      </td>

      <td>
        NFSv3 client statistics: writes
      </td>
    </tr>

    <tr>
      <td>
        `nfs_v3_server_access`
      </td>

      <td>
        NFSv3 server statistics: access
      </td>
    </tr>

    <tr>
      <td>
        `nfs_v3_server_create`
      </td>

      <td>
        NFSv3 server statistics: create
      </td>
    </tr>

    <tr>
      <td>
        `nfs_v3_server_fsinfo`
      </td>

      <td>
        NFSv3 server statistics: static file system information
      </td>
    </tr>

    <tr>
      <td>
        `nfs_v3_server_fsstat`
      </td>

      <td>
        NFSv3 server statistics: dynamic file system information
      </td>
    </tr>

    <tr>
      <td>
        `nfs_v3_server_getattr`
      </td>

      <td>
        NFSv3 server statistics: file system attributes gets
      </td>
    </tr>

    <tr>
      <td>
        `nfs_v3_server_lookup`
      </td>

      <td>
        NFSv3 server statistics: file name look-ups
      </td>
    </tr>

    <tr>
      <td>
        `nfs_v3_server_mkdir`
      </td>

      <td>
        NFSv3 server statistics: directory creation
      </td>
    </tr>

    <tr>
      <td>
        `nfs_v3_server_null`
      </td>

      <td>
        NFSv3 server statistics: null operations
      </td>
    </tr>

    <tr>
      <td>
        `nfs_v3_server_pathconf`
      </td>

      <td>
        NFSv3 server statistics: retrieve POSIX information
      </td>
    </tr>

    <tr>
      <td>
        `nfs_v3_server_read`
      </td>

      <td>
        NFSv3 server statistics: reads
      </td>
    </tr>

    <tr>
      <td>
        `nfs_v3_server_readdirplus`
      </td>

      <td>
        NFSv3 server statistics: READDIRPLUS procedures
      </td>
    </tr>

    <tr>
      <td>
        `nfs_v3_server_readlink`
      </td>

      <td>
        NFSv3 server statistics: Symbolic link reads
      </td>
    </tr>

    <tr>
      <td>
        `nfs_v3_server_setattr`
      </td>

      <td>
        NFSv3 server statistics: file system attribute sets
      </td>
    </tr>

    <tr>
      <td>
        `nfs_v3_server_total`
      </td>

      <td>
        NFSv3 server statistics: total
      </td>
    </tr>

    <tr>
      <td>
        `nfs_v3_server_write`
      </td>

      <td>
        NFSv3 server statistics: writes
      </td>
    </tr>

    <tr>
      <td>
        `nfs_v4_server_compound`
      </td>

      <td>
        NFSv4 server statistics: compound operations
      </td>
    </tr>

    <tr>
      <td>
        `nfs_v4_server_null`
      </td>

      <td>
        NFSv4 server statistics: null operations
      </td>
    </tr>

    <tr>
      <td>
        `nfs_v4_server_total`
      </td>

      <td>
        NFSv4 server statistics: total
      </td>
    </tr>

    <tr>
      <td>
        `nfs_v4_servop_access`
      </td>

      <td>
        NFSv4 server statistics: access
      </td>
    </tr>

    <tr>
      <td>
        `nfs_v4_servop_close`
      </td>

      <td>
        NFSv4 server statistics: close
      </td>
    </tr>

    <tr>
      <td>
        `nfs_v4_servop_create`
      </td>

      <td>
        NFSv4 server statistics: create
      </td>
    </tr>

    <tr>
      <td>
        `nfs_v4_servop_getattr`
      </td>

      <td>
        NFSv4 server statistics: file system attributes gets
      </td>
    </tr>

    <tr>
      <td>
        `nfs_v4_servop_getfh`
      </td>

      <td>
        NFSv4 server statistics: filehandle gets
      </td>
    </tr>

    <tr>
      <td>
        `nfs_v4_servop_lookup`
      </td>

      <td>
        NFSv4 server statistics: file name look-ups
      </td>
    </tr>

    <tr>
      <td>
        `nfs_v4_servop_open`
      </td>

      <td>
        NFSv4 server statistics: opens
      </td>
    </tr>

    <tr>
      <td>
        `nfs_v4_servop_putfh`
      </td>

      <td>
        NFSv4 server statistics: filehandle puts
      </td>
    </tr>

    <tr>
      <td>
        `nfs_v4_servop_putrootfh`
      </td>

      <td>
        NFSv4 server statistics: filehandle puts to root
      </td>
    </tr>

    <tr>
      <td>
        `nfs_v4_servop_read`
      </td>

      <td>
        NFSv4 server statistics: reads
      </td>
    </tr>

    <tr>
      <td>
        `nfs_v4_servop_readdirplus`
      </td>

      <td>
        NFSv4 server statistics: READDIRPLUS procedures
      </td>
    </tr>

    <tr>
      <td>
        `nfs_v4_servop_readlink`
      </td>

      <td>
        NFSv4 server statistics: Symbolic link reads
      </td>
    </tr>

    <tr>
      <td>
        `nfs_v4_servop_rename`
      </td>

      <td>
        NFSv4 server statistics: renames
      </td>
    </tr>

    <tr>
      <td>
        `nfs_v4_servop_savefh`
      </td>

      <td>
        NFSv4 server statistics: filehandle saves
      </td>
    </tr>

    <tr>
      <td>
        `nfs_v4_servop_setattr`
      </td>

      <td>
        NFSv4 server statistics: file system attribute sets
      </td>
    </tr>

    <tr>
      <td>
        `nfs_v4_servop_total`
      </td>

      <td>
        NFSv4 server statistics: total
      </td>
    </tr>

    <tr>
      <td>
        `nfs_v4_servop_write`
      </td>

      <td>
        NFSv4 server statistics: writes
      </td>
    </tr>
  </tbody>
</table>

### G.1.3  InfiniBand Metrics

The available InfiniBand metrics are displayed in table [G.1.3](#g13-infiniband-metrics).

<table>
  <thead>
    <tr>
      <th colspan="2">*Table  [G.1.3](#g13-infiniband-metrics): InfiniBand Metrics*</th>
    </tr>
  </thead>

  <tbody>
    <tr>
      <td>
        **InfiniBand Metric**
      </td>

      <td>
        **Description**
      </td>
    </tr>

    <tr>
      <td>
        `SymbolErrorCount`
      </td>

      <td>
        Total number of minor link errors detected on one or more physical lanes.
      </td>
    </tr>

    <tr>
      <td>
        `LinkErrorRecoveryCount`
      </td>

      <td>
        Total number of times the Port Training state machine has successfully completed the link error recovery process.
      </td>
    </tr>

    <tr>
      <td>
        `LinkDownedCounter`
      </td>

      <td>
        Total number of times the Port Training state machine has failed the link error recovery process and downed the link.
      </td>
    </tr>

    <tr>
      <td>
        `PortRcvErrors`
      </td>

      <td>
        Total number of packets containing an error that were received on the port.
      </td>
    </tr>

    <tr>
      <td>
        `PortRcvRemotePhysicalErrors`
      </td>

      <td>
        Total number of packets marked with the EBP delimiter received on the port.
      </td>
    </tr>

    <tr>
      <td>
        `PortRcvSwitchRelayErrors`
      </td>

      <td>
        Total number of packets received on the port that were discarded because they could not be forwarded by the switch relay.
      </td>
    </tr>

    <tr>
      <td>
        `PortXmitDiscards`
      </td>

      <td>
        Total number of outbound packets discarded by the port because the port is down or congested.
      </td>
    </tr>

    <tr>
      <td>
        `PortXmitConstrainErrors`
      </td>

      <td>
        Total number of packets not transmitted from the switch physical port.
      </td>
    </tr>

    <tr>
      <td>
        `PortRcvConstraintErrors`
      </td>

      <td>
        Total number of packets received on the switch physical port that are discarded.
      </td>
    </tr>

    <tr>
      <td>
        `LocalLinkIntegrityErrors`
      </td>

      <td>
        The number of times that the count of local physical errors exceeded the threshold specified by LocalPhyErrors.
      </td>
    </tr>

    <tr>
      <td>
        `ExcessiveBufferOverrunError`
      </td>

      <td>
        The number of times that OverrunErrors consecutive flow control update periods occurred, each having at least one overrun error
      </td>
    </tr>

    <tr>
      <td>
        `QP1Dropped`
      </td>

      <td>
        Drops on the lower priority QP1 interconnect
      </td>
    </tr>

    <tr>
      <td>
        `VL15Dropped`
      </td>

      <td>
        Drops in the highest priority virtual lane 15
      </td>
    </tr>

    <tr>
      <td>
        `PortXmitData`
      </td>

      <td>
        Total number of data octets, divided by 4 (lanes), transmitted on all VLs. This is a 64-bit counter.
      </td>
    </tr>

    <tr>
      <td>
        `PortRcvData`
      </td>

      <td>
        Total number of data octets, divided by 4 (lanes), received on all VLs. This is a 64-bit counter.
      </td>
    </tr>

    <tr>
      <td>
        `PortXmitPkts`
      </td>

      <td>
        Total number of packets transmitted on all VLs from this port, including packets with errors, and excluding link packets.
      </td>
    </tr>

    <tr>
      <td>
        `PortRcvPkts`
      </td>

      <td>
        Total number of packets, including packets containing errors, and excluding link packets, received from all VLs on this port. This is a 64-bit counter.
      </td>
    </tr>

    <tr>
      <td>
        `PortXmitWait`
      </td>

      <td>
        The number of ticks during which the port had data to transmit but no data was sent during the entire tick (either because of insufficient credits or because of lack of arbitration).
      </td>
    </tr>
  </tbody>
</table>

### G.1.4  Monitoring System Metrics

Internal metrics for the monitoring system itself are produced by the `MonitoringSystem` data producer.

The data producer does not run by default on any node, as is seen by running the `nodes` command ([here](/admin/monitoring#10545-the-setup-submode-deeper-submodes)) for it:

```bash
[basecm11->monitoring->setup[MonitoringSystem]]% nodes
Not used
```

However, for convenience, some monitoring system metrics values are displayed on demand in an organized way, by running the `monitoringinfo` command for a device:

**Example**

```bash
[basecm11->device[basecm11]]% monitoringinfo
Service                       Queued   Handled    Cache miss Stopped   Suspended  Last operation
----------------------------- -------- ---------- ---------- --------- ---------- ---------------
Mon::CacheGather              4        910        4          no        no         Fri Oct  4 ...
Mon::DataConverter            0        0          0          no        no         -
Mon::DataProcessorEngine      0        155,470    0          no        no         Fri Oct  4 ...
Mon::DataProcessorKeepLatest  0        155,470    0          no        no         Fri Oct  4 ...
Mon::DataProcessorKeepRecent  0        0          0          no        no         -
...
```

The internal monitoring system metrics can be activated so that their data values are saved. This is not recommended, except temporarily for debugging purposes. To save the values, the `Introspect` flag for the monitoring system data producer should be set to `yes`:

**Example**

```bash
[basecm11->monitoring->setup[MonitoringSystem]]% set --extra Introspect yes
[basecm11->monitoring->setup*[MonitoringSystem*]]% commit
[basecm11->monitoring->setup[MonitoringSystem]]% nodes
basecm11
[basecm11->monitoring->setup[MonitoringSystem]]% get interval
900
```

After a time defined by `interval`, the metrics can be viewed:

**Example**

```bash
[basecm11->device[basecm11]]% latestmetricdata | grep Mon::Storage
Mon::Storage::Engine::elements         Internal/Monitoring/Storage  999,549      44.3s
Mon::Storage::Engine::size             Internal/Monitoring/Storage  1.00 GiB     44.3s
Mon::Storage::Engine::usage            Internal/Monitoring/Storage  9.63%        44.3s
Mon::Storage::Message::elements        Internal/Monitoring/Storage  28           44.3s
...
```

The monitoring system metrics that are saved are shown in table [G.1.4](#g14-monitoring-system-metrics).

<table>
  <thead>
    <tr>
      <th colspan="2">*Table  [G.1.4](#g14-monitoring-system-metrics): Monitoring System Metrics*</th>
    </tr>
  </thead>

  <tbody>
    <tr>
      <td>
        **Monitoring System Metric**
      </td>

      <td>
        **Description**
      </td>
    </tr>

    <tr>
      <td>
        `Mon::CacheGather::handled`
      </td>

      <td>
        Cache gathers handled/s
      </td>
    </tr>

    <tr>
      <td>
        `Mon::CacheGather::miss`
      </td>

      <td>
        Cache gathers missed/s
      </td>
    </tr>

    <tr>
      <td>
        `Mon::DataProcessor::handled`
      </td>

      <td>
        Data processors handled/s
      </td>
    </tr>

    <tr>
      <td>
        `Mon::DataProcessor::miss`
      </td>

      <td>
        Data processors missed /s
      </td>
    </tr>

    <tr>
      <td>
        `Mon::DataTranslator::handled`
      </td>

      <td>
        Data translators handled/s
      </td>
    </tr>

    <tr>
      <td>
        `Mon::DataTranslator::miss`
      </td>

      <td>
        Data translators missed/s
      </td>
    </tr>

    <tr>
      <td>
        `Mon::EntityMeasurableCache::handled`
      </td>

      <td>
        Measurable cache handled/s
      </td>
    </tr>

    <tr>
      <td>
        `Mon::EntityMeasurableCache::miss`
      </td>

      <td>
        Measurable cache missed/s
      </td>
    </tr>

    <tr>
      <td>
        `Mon::MeasurableBroker::handled`
      </td>

      <td>
        Measurable broker handled/s
      </td>
    </tr>

    <tr>
      <td>
        `Mon::MeasurableBroker::miss`
      </td>

      <td>
        Measurable broker missed/s
      </td>
    </tr>

    <tr>
      <td>
        `Mon::OOB::TaskService::handled`
      </td>

      <td>
        Out-of-band task service handled/s
      </td>
    </tr>

    <tr>
      <td>
        `Mon::OOB::TaskService::miss`
      </td>

      <td>
        Out-of-band task service missed/s
      </td>
    </tr>

    <tr>
      <td>
        `Mon::Replicate::Collector::handled`
      </td>

      <td>
        Replication collection handled/s
      </td>
    </tr>

    <tr>
      <td>
        `Mon::Replicate::Collector::miss`
      </td>

      <td>
        Replication collection missed/s
      </td>
    </tr>

    <tr>
      <td>
        `Mon::Replicate::Combiner::handled`
      </td>

      <td>
        Replication combiner handled/s
      </td>
    </tr>

    <tr>
      <td>
        `Mon::Replicate::Combiner::miss`
      </td>

      <td>
        Replication combiner missed/s
      </td>
    </tr>

    <tr>
      <td>
        `Mon::RepositoryAllocator::handled`
      </td>

      <td>
        Repository allocator handled/s
      </td>
    </tr>

    <tr>
      <td>
        `Mon::RepositoryAllocator::miss`
      </td>

      <td>
        Repository allocator missed/s
      </td>
    </tr>

    <tr>
      <td>
        `Mon::RepositoryTrim::handled`
      </td>

      <td>
        Repository trim handled/s
      </td>
    </tr>

    <tr>
      <td>
        `Mon::RepositoryTrim::miss`
      </td>

      <td>
        Repository trim missed/s
      </td>
    </tr>

    <tr>
      <td>
        `Mon::Storage::Engine::elements`
      </td>

      <td>
        Storage engine data elements, in total
      </td>
    </tr>

    <tr>
      <td>
        `Mon::Storage::Engine::size`
      </td>

      <td>
        Storage engine size, in bytes
      </td>
    </tr>

    <tr>
      <td>
        `Mon::Storage::Engine::usage`
      </td>

      <td>
        Storage engine usage
      </td>
    </tr>

    <tr>
      <td>
        `Mon::Storage::Message::elements `
      </td>

      <td>
        Storage message data elements, in total
      </td>
    </tr>

    <tr>
      <td>
        `Mon::Storage::Message::size`
      </td>

      <td>
        Storage message size in bytes
      </td>
    </tr>

    <tr>
      <td>
        `Mon::Storage::Message::usage`
      </td>

      <td>
        Storage message usage
      </td>
    </tr>

    <tr>
      <td>
        `Mon::Storage::RepositoryId::elements`
      </td>

      <td>
        Storage repository ID data elements, in total
      </td>
    </tr>

    <tr>
      <td>
        `Mon::Storage::RepositoryId::size`
      </td>

      <td>
        Storage repository ID, size, in bytes
      </td>
    </tr>

    <tr>
      <td>
        `Mon::Storage::RepositoryId::usage`
      </td>

      <td>
        Repository ID usage
      </td>
    </tr>

    <tr>
      <td>
        `Mon::TaskInitializer::handled`
      </td>

      <td>
        Task initializer handled/s
      </td>
    </tr>

    <tr>
      <td>
        `Mon::TaskInitializer::miss`
      </td>

      <td>
        Task initializer missed/s
      </td>
    </tr>

    <tr>
      <td>
        `Mon::TaskSampler::handled`
      </td>

      <td>
        Task sampler handled/s
      </td>
    </tr>

    <tr>
      <td>
        `Mon::TaskSampler::miss`
      </td>

      <td>
        Task sampler missed/s
      </td>
    </tr>

    <tr>
      <td>
        `Mon::Trigger::Actuator::handled`
      </td>

      <td>
        Trigger actuators handled/s
      </td>
    </tr>

    <tr>
      <td>
        `Mon::Trigger::Actuator::miss`
      </td>

      <td>
        Trigger actuators missed/s
      </td>
    </tr>

    <tr>
      <td>
        `Mon::Trigger::Dispatcher::handled`
      </td>

      <td>
        Trigger dispatchers handled/s
      </td>
    </tr>

    <tr>
      <td>
        `Mon::Trigger::Dispatcher::miss`
      </td>

      <td>
        Trigger dispatchers missed/s
      </td>
    </tr>

    <tr>
      <td>
        `Prometheus::DataTranslator::handled`
      </td>

      <td>
        DataTranslator queries handled. The DataTranslator is a translation layer between PromQL format and BCM format sampling.
      </td>
    </tr>

    <tr>
      <td>
        `Prometheus::DataTranslator::miss`
      </td>

      <td>
        DataTranslator misses
      </td>
    </tr>
  </tbody>
</table>

### G.1.5  CPU Metrics Sampled By The CPUSampler And GPUSampler

Some CPU metrics are sampled by the data producers CPUSampler and GPUSampler (table [G.1.5](#g15-cpu-metrics-sampled-by-the-cpusampler-and-gpusampler)):

<table>
  <colgroup>
    <col />

    <col />

    <col />
  </colgroup>

  <thead>
    <tr>
      <th colspan="3">*Table [G.1.5](#g15-cpu-metrics-sampled-by-the-cpusampler-and-gpusampler): CPU Metrics Sampled By The CPUSampler And GPUSampler*</th>
    </tr>
  </thead>

  <tbody>
    <tr>
      <td>
        **CPU Metric**
      </td>

      <td>
        **Parameter**
      </td>

      <td>
        **Description**
      </td>
    </tr>

    <tr>
      <td>
        `cpu_core_clock_speed`

        \*
      </td>

      <td>
        `cpu_core0`
      </td>

      <td>
        CPU0 core clock speed
      </td>
    </tr>

    <tr>
      <td>
        `cpu_power_cap_enabled`

        \*
      </td>

      <td>
        `cpu0`
      </td>

      <td>
        CPU0 power capping enabled
      </td>
    </tr>

    <tr>
      <td>
        `cpu_power_cap_long_term_max_limit`

        \*
      </td>

      <td>
        `cpu0`
      </td>

      <td>
        Long term CPU0 power constraint
      </td>
    </tr>

    <tr>
      <td>
        `cpu_power_cap_long_term_time_window`

        \*
      </td>

      <td>
        `cpu0`
      </td>

      <td>
        Long term CPU0 power constraint time window
      </td>
    </tr>

    <tr>
      <td>
        `cpu_power_cap_short_term_max_limit`

        \*
      </td>

      <td>
        `cpu0`
      </td>

      <td>
        Short term CPU0 power constraint
      </td>
    </tr>

    <tr>
      <td>
        `cpu_power_cap_short_term_time_window`

        \*
      </td>

      <td>
        `cpu0`
      </td>

      <td>
        Short term CPU0 power constraint time window
      </td>
    </tr>

    <tr>
      <td>
        `cpu_power_limit`

        \*
      </td>

      <td>
        `cpu0`
      </td>

      <td>
        CPU0 power limit
      </td>
    </tr>

    <tr>
      <td>
        `cpu_power_usage`

        \*
      </td>

      <td>
        `cpu0`
      </td>

      <td>
        CPU0 power usage
      </td>
    </tr>

    <tr>
      <td>
        `cpu_power_usage`
      </td>

      <td>
        `total`
      </td>

      <td>
        Total CPU power usage
      </td>
    </tr>

    <tr>
      <td>
        `cpu_temperature`
      </td>

      <td>
        `average`
      </td>

      <td>
        Average CPU temperature
      </td>
    </tr>

    <tr>
      <td>
        `cpu_temperature`

        \*
      </td>

      <td>
        `cpu0`
      </td>

      <td>
        CPU0 temperature
      </td>
    </tr>

    <tr>
      <td>
        `cpu_utilization`
      </td>

      <td>
        `average`
      </td>

      <td>
        Average CPU utilization
      </td>
    </tr>
  </tbody>
</table>

\* The number 0 in the parameter column can be replaced by the number associated with the core used. Thus:

`cpu_core0` can be replaced by `cpu_core1`, `cup_core2`...

and

`cpu0` can be replaced by `cpu1`, `cpu2`...

### G.1.6  GPU Metrics

The data producer (section [10.2.10](/admin/monitoring#10210-data-producer)) for the GPU metrics of this section is `GPUSampler`.

There were GPU metrics described earlier on in table [G.1.1](#g11-regular-metrics). These were cluster overview metrics about GPUs, and were provided by the `ClusterTotal` data producer.

However, the NVIDIA GPU metrics of this section, as the `GPUSampler` data producer name suggests, is about gathering the sampled GPU data from the devices themselves.

There is also a separate section about job GPU metrics (section [G.1.8.4](#g184-job-gpu-metrics), [here](#g184-job-gpu-metrics)), which uses the `JobSampler` data producer, and is about gathering the sampled job data from the GPU devices themselves.

The device parameter for the GPU metrics in this section, unless otherwise noted, specifies the device slot number that the GPU uses. The parameter takes the form `gpu0`, `gpu1`, and so on. It is appended to the metric with a colon character. For example, the `gpu_ecc_dbe_agg` metric, if used with `gpu1`, is specified as:

**Example**

```bash
gpu_ecc_dbe_agg:gpu1
```

Available GPU metrics for V100 and A100 GPUs are listed in table [G.1.6](#g16-gpu-metrics). The available GPU health checks for V100 and A100 GPUs are listed in table [G.2.2](#g22-gpu-health-checks).

If the cluster has been configured with [AMD GPUs (section 7.4 of the *Installation Manual*)](/installation/thirdparty#74-amd-gpu-driver-installation) then AMD GPU metrics become available. Metrics in the table that are also valid AMD GPU metrics are noted. Some metrics have been added and noted that are only valid for AMD GPUs.

Extra GPU measurables for the GB200 GPUs are listed in [section 12.6 of the *NVIDIA Mission Control Manual*](/nvidia-mission-control/mcmetricshealthchecksactions#126-gpu-measurables-for-the-dgx-gb200).

<table>
  <thead>
    <tr>
      <th colspan="2">*Table [G.1.6](#g16-gpu-metrics): GPU Metrics*</th>
    </tr>
  </thead>

  <tbody>
    <tr>
      <td>
        **GPU Metric**
      </td>

      <td>
        **Description**
      </td>
    </tr>

    <tr>
      <td>
        `gpu_dec_utilization`

        \*
      </td>

      <td>
        GPU decoding usage
      </td>
    </tr>

    <tr>
      <td>
        `gpu_ecc_dbe_agg`
      </td>

      <td>
        Total double bit aggregate ECC errors
      </td>
    </tr>

    <tr>
      <td>
        `gpu_ecc_dbe_vol`
      </td>

      <td>
        Total double bit volatile ECC errors
      </td>
    </tr>

    <tr>
      <td>
        `gpu_ecc_sbe_agg`
      </td>

      <td>
        Total single bit aggregate ECC errors
      </td>
    </tr>

    <tr>
      <td>
        `gpu_ecc_sbe_vol`
      </td>

      <td>
        Total single bit volatile ECC errors
      </td>
    </tr>

    <tr>
      <td>
        `gpu_enc_utilization`

        \*
      </td>

      <td>
        GPU encoding usage
      </td>
    </tr>

    <tr>
      <td>
        `gpu_enforced_power_limit`
      </td>

      <td>
        GPU-enforced power limit
      </td>
    </tr>

    <tr>
      <td>
        `gpu_mem_clock`
      </td>

      <td>
        GPU memory clock (also for AMD GPU)
      </td>
    </tr>

    <tr>
      <td>
        `gpu_mem_copy_utilization`

        \*
      </td>

      <td>
        Percentage of GPU memory copy used
      </td>
    </tr>

    <tr>
      <td>
        `gpu_mem_free`
      </td>

      <td>
        Amount of GPU free memory
      </td>
    </tr>

    <tr>
      <td>
        `gpu_mem_total`

        \**
      </td>

      <td>
        GPU framebuffer size (also for AMD GPU)
      </td>
    </tr>

    <tr>
      <td>
        `gpu_mem_used`

        \**
      </td>

      <td>
        Amount of GPU memory used (also for AMD GPU)
      </td>
    </tr>

    <tr>
      <td>
        `gpu_mem_utilization`

        \*
      </td>

      <td>
        GPU memory used percentage
      </td>
    </tr>

    <tr>
      <td>
        `gpu_memory_temp`

        \*
      </td>

      <td>
        GPU memory temperature (Celsius)
      </td>
    </tr>

    <tr>
      <td>
        `gpu_nvlink_total_bandwidth`

        \**
      </td>

      <td>
        Total NVLink bandwidth used
      </td>
    </tr>

    <tr>
      <td>
        `gpu_power_management_limit`
      </td>

      <td>
        GPU power management limit
      </td>
    </tr>

    <tr>
      <td>
        `gpu_power_usage`

        \**
      </td>

      <td>
        GPU power usage (also for AMD GPU)
      </td>
    </tr>

    <tr>
      <td>
        `gpu_power_violation`

        \**
      </td>

      <td>
        Throttling duration due to power constraints
      </td>
    </tr>

    <tr>
      <td>
        `gpu_shutdown_temp`
      </td>

      <td>
        GPU shutdown temperature (Celsius)
      </td>
    </tr>

    <tr>
      <td>
        `gpu_slowdown_temp`
      </td>

      <td>
        GPU slowdown temperature (Celsius)
      </td>
    </tr>

    <tr>
      <td>
        `gpu_sm_clock`
      </td>

      <td>
        GPU shader multiprocessor clock (also for AMD GPU)
      </td>
    </tr>

    <tr>
      <td>
        `gpu_temperature`

        \*
      </td>

      <td>
        GPU temperature (Celsius)
      </td>
    </tr>

    <tr>
      <td>
        `gpu_thermal_violation`

        \**
      </td>

      <td>
        Throttling duration due to thermal constraints
      </td>
    </tr>

    <tr>
      <td>
        `gpu_utilization`

        \[*]
      </td>

      <td>
        Average GPU utilization percentage
      </td>
    </tr>

    <tr>
      <td>
        `gpu_xid_error`
      </td>

      <td>
        The value is the specific XID error
      </td>
    </tr>
  </tbody>
</table>

\* Specified as average, or specified for a GPU.

For example, for the `gpu_dec_utilization` metric:

`gpu_dec_utilization:average` or `gpu_dec_utilization:GPU0`

\*\* Specified as a total, or specified for a GPU.

For example, for the `gpu_mem_total` metric:

`gpu_mem_total:total` or `gpu_mem_total:GPU0`

### G.1.7  GPU Profiling Metrics

The data producer (section [10.2.10](/admin/monitoring#10210-data-producer)) for the GPU profiling metrics of this section is `GPUSampler`.

The device parameter for the GPU profiling metrics in this section, unless otherwise noted, specifies the device slot number that the GPU uses. The parameter takes the form `gpu0`, `gpu1`, and so on. It is appended to the metric with a colon character. For example, the `gpu_profiling_fp64_active` metric, if used with `gpu1`, is specified as:

**Example**

```bash
gpu_profiling_fp64_active:gpu1
```

Available GPU profiling metrics for V100 and A100 GPUs are listed in table [G.1.7](#g17-gpu-profiling-metrics).

<table>
  <thead>
    <tr>
      <th colspan="2">*Table [G.1.7](#g17-gpu-profiling-metrics): GPU Profiling Metrics*</th>
    </tr>
  </thead>

  <tbody>
    <tr>
      <td>
        **GPU Profiling Metric**
      </td>

      <td>
        **Description**
      </td>
    </tr>

    <tr>
      <td>
        `gpu_profiling_dram_active`
      </td>

      <td>
        The ratio of cycles the device memory interface is active sending or receiving data
      </td>
    </tr>

    <tr>
      <td>
        `gpu_profiling_fp16_active`
      </td>

      <td>
        Ratio of cycles the fp16 pipe is active
      </td>
    </tr>

    <tr>
      <td>
        `gpu_profiling_fp32_active`
      </td>

      <td>
        Ratio of cycles the fp32 pipe is active
      </td>
    </tr>

    <tr>
      <td>
        `gpu_profiling_fp64_active`
      </td>

      <td>
        Ratio of cycles the fp64 pipe is active
      </td>
    </tr>

    <tr>
      <td>
        `gpu_profiling_graphics_engine_active`
      </td>

      <td>
        Ratio of time the graphics engine is active. The graphics engine is active if a graphics/compute context is bound and the graphics pipe or compute pipe is busy
      </td>
    </tr>

    <tr>
      <td>
        `gpu_profiling_nvlink_read`
      </td>

      <td>
        The number of bytes of active NvLink rx (read) data including both header and payload
      </td>
    </tr>

    <tr>
      <td>
        `gpu_profiling_nvlink_transmit`
      </td>

      <td>
        The number of bytes of active NvLink tx (transmit) data including both header and payload
      </td>
    </tr>

    <tr>
      <td>
        `gpu_profiling_pcie_read`
      </td>

      <td>
        The number of bytes of active PCIe rx (read) data including both header and payload
      </td>
    </tr>

    <tr>
      <td>
        `gpu_profiling_pcie_transmit`
      </td>

      <td>
        The number of bytes of active PCIe tx (transmit) data including both header and payload
      </td>
    </tr>

    <tr>
      <td>
        `gpu_profiling_pipe_tensor_active`
      </td>

      <td>
        The ratio of cycles the tensor (HMMA) pipe is active
      </td>
    </tr>

    <tr>
      <td>
        `gpu_profiling_sm_active`
      </td>

      <td>
        The ratio of cycles an SM has at least 1 warp assigned
      </td>
    </tr>

    <tr>
      <td>
        `gpu_profiling_sm_occupancy`
      </td>

      <td>
        The ratio of cycles the tensor (HMMA) pipe is active
      </td>
    </tr>
  </tbody>
</table>

### G.1.8  Job Metrics

Job metrics are introduced in section [11.1](/admin/monitoringjob#111-job-metrics-introduction).

#### G.1.8.1  Basic Job Metrics

The following table lists some of the most useful job metrics that BCM can monitor and visualize. In the table, the text \<*device*> denotes a block device name, such as `sda`.

On virtual machines, block device metrics may be unavailable because of virtualization.

<table>
  <thead>
    <tr>
      <th colspan="3">*Table [G.1.8](#g18-job-metrics).1: Basic Job Metrics*</th>
    </tr>
  </thead>

  <tbody>
    <tr>
      <td>
        **Job Metric**
      </td>

      <td>
        **Description**
      </td>

      <td>
        **Cgroup Source File**
      </td>
    </tr>

    <tr>
      <td>
        `blkio.time:`

        `<`

        *device*

        `>`
      </td>

      <td>
        Time job had I/O access to device
      </td>

      <td>
        `blkio.time_recursive`
      </td>
    </tr>

    <tr>
      <td>
        `blkio.sectors:`

        `<`

        *device*

        `>`
      </td>

      <td>
        Sectors transferred to or from specific devices by a cgroup
      </td>

      <td>
        `blkio.sectors_recursive`
      </td>
    </tr>

    <tr>
      <td>
        `blkio.io_service_read:`

        `<`

        *device*

        `>`
      </td>

      <td>
        Bytes read
      </td>

      <td>
        `blkio.io_service_bytes_recursive`
      </td>
    </tr>

    <tr>
      <td>
        `blkio.io_service_write:`

        `<`

        *device*

        `>`
      </td>

      <td>
        Bytes written
      </td>

      <td>
        `blkio.io_service_bytes_recursive`
      </td>
    </tr>

    <tr>
      <td>
        `blkio.io_service_sync:`

        `<`

        *device*

        `>`
      </td>

      <td>
        Bytes transferred synchronously
      </td>

      <td>
        `blkio.io_service_bytes_recursive`
      </td>
    </tr>

    <tr>
      <td>
        `blkio.io_service_async:`

        `<`

        *device*

        `>`
      </td>

      <td>
        Bytes transferred asynchronously
      </td>

      <td>
        `blkio.io_service_bytes_recursive`
      </td>
    </tr>

    <tr>
      <td>
        `blkio.io_wait_time_read:`

        `<`

        *device*

        `>`
      </td>

      <td>
        Total time spent waiting for service in the scheduler queues for I/O read operations
      </td>

      <td>
        `blkio.io_wait_time_recursive`
      </td>
    </tr>

    <tr>
      <td>
        `blkio.io_wait_time_write:`

        `<`

        *device*

        `>`
      </td>

      <td>
        Total time spent waiting for service in the scheduler queues for I/O write operations
      </td>

      <td>
        `blkio.io_wait_time_recursive`
      </td>
    </tr>

    <tr>
      <td>
        `blkio.io_wait_time_sync:`

        `<`

        *device*

        `>`
      </td>

      <td>
        Total time spent waiting for service in the scheduler queues for I/O synchronous operations
      </td>

      <td>
        `blkio.io_wait_time_recursive`
      </td>
    </tr>

    <tr>
      <td>
        `blkio.io_wait_time_async:`

        `<`

        *device*

        `>`
      </td>

      <td>
        Total time spent waiting for service in the scheduler queues for I/O asynchronous operations
      </td>

      <td>
        `blkio.io_wait_time_recursive`
      </td>
    </tr>

    <tr>
      <td>
        `cpuacct.usage`
      </td>

      <td>
        Total CPU time consumed by all job processes
      </td>

      <td>
        `cpuacct.usage`
      </td>
    </tr>

    <tr>
      <td>
        `cpuacct.stat.user`
      </td>

      <td>
        User CPU time consumed by all job processes
      </td>

      <td>
        `cpuacct.stat`
      </td>
    </tr>

    <tr>
      <td>
        `cpuacct.stat.system`
      </td>

      <td>
        System CPU time consumed by all job processes
      </td>

      <td>
        `cpuacct.stat`
      </td>
    </tr>

    <tr>
      <td>
        `memory.usage`
      </td>

      <td>
        Total current memory usage
      </td>

      <td>
        `memory.usage_in_bytes`
      </td>
    </tr>

    <tr>
      <td>
        `memory.memsw.usage`
      </td>

      <td>
        Sum of current memory plus swap space usage
      </td>

      <td>
        `memory.memsw.usage_in_bytes`
      </td>
    </tr>

    <tr>
      <td>
        `memory.memsw.max_usage`
      </td>

      <td>
        Maximum amount of memory and swap space used
      </td>

      <td>
        `memory.memsw.max_usage_in_bytes`
      </td>
    </tr>

    <tr>
      <td>
        `memory.failcnt`
      </td>

      <td>
        How often the memory limit has reached the value set in 

        `memory.limit_in_bytes`
      </td>

      <td>
        `memory.failcnt`
      </td>
    </tr>

    <tr>
      <td>
        `memory.memsw.failcnt`
      </td>

      <td>
        How often the memory plus swap space limit has reached the value set in 

        `memory.memsw.limit_in_bytes`
      </td>

      <td>
        `memory.memsw.failcnt`
      </td>
    </tr>

    <tr>
      <td>
        `memory.swap`
      </td>

      <td>
        Total swap usage
      </td>

      <td>
        `memory`
      </td>
    </tr>

    <tr>
      <td>
        `memory.cache`
      </td>

      <td>
        Total page cache, including tmpfs (shmem)
      </td>

      <td>
        `memory`
      </td>
    </tr>

    <tr>
      <td>
        `memory.mapped_file`
      </td>

      <td>
        Size of memory-mapped mapped files, including tmpfs (shmem)
      </td>

      <td>
        `memory`
      </td>
    </tr>

    <tr>
      <td>
        `memory.unevictable`
      </td>

      <td>
        Memory that cannot be reclaimed
      </td>

      <td>
        `memory`
      </td>
    </tr>
  </tbody>
</table>

The third column in the table shows the precise source file name that is used when the value is retrieved. These files are all virtual files, and are created as the cgroup controllers are mounted to the cgroup directory. In this case several controllers are mounted to the same directory, which means that all the virtual files will show up in that directory, and in its associated subdirectories—job cgroup directories—when the job runs.

#### G.1.8.2  Advanced Job Metrics

The metrics in the preceding table are enabled by default. There are also over 40 other advanced metrics that can be enabled via the `Enable Advanced Metrics` property of the `jobmetricsettings` object:

```bash
[basecm11->monitoring->setup[JobSampler]->jobmetricsettings]% show
Parameter                        Value
-------------------------------- ------------------------------------------------
Enable Advanced Metrics          no
Exclude Devices                  loop,sr
Exclude Metrics
Include Devices
Include Metrics
Revision
Sampling Type                    Both
[basecm11->monitoring->setup[JobSampler]->jobmetricsettings]% set enableadvancedmetrics yes
[basecm11->monitoring->setup*[JobSampler*]->jobmetricsettings*]% commit
```

The advanced job metrics are:

<table>
  <thead>
    <tr>
      <th colspan="3">*Table [G.1.8](#g18-job-metrics).2: Advanced Job Metrics*</th>
    </tr>
  </thead>

  <tbody>
    <tr>
      <td>
        **Advanced Job Metric**
      </td>

      <td>
        **Description**
      </td>

      <td>
        **Cgroup Source File**
      </td>
    </tr>

    <tr>
      <td>
        `blkio.io_service_time_read:`

        `<`

        *device*

        `>`
      </td>

      <td>
        Total time between request dispatch and request completion according to CFQ scheduler for I/O read operations
      </td>

      <td>
        `blkio.io_service_time_recursive`
      </td>
    </tr>

    <tr>
      <td>
        `blkio.io_service_time_write:`

        `<`

        *device*

        `>`
      </td>

      <td>
        Total time between request dispatch and request completion according to CFQ scheduler for I/O write operations
      </td>

      <td>
        `blkio.io_service_time_recursive`
      </td>
    </tr>

    <tr>
      <td>
        `blkio.io_service_time_sync:`

        `<`

        *device*

        `>`
      </td>

      <td>
        Total time between request dispatch and request completion according to CFQ scheduler for I/O synchronous operations
      </td>

      <td>
        `blkio.io_service_time_recursive`
      </td>
    </tr>

    <tr>
      <td>
        `blkio.io_service_time_async:`

        `<`

        *device*

        `>`
      </td>

      <td>
        Total time between request dispatch and request completion according to CFQ scheduler for I/O asynchronous operations
      </td>

      <td>
        `blkio.io_service_time_recursive`
      </td>
    </tr>

    <tr>
      <td>
        `blkio.io_serviced_read:`

        `<`

        *device*

        `>`
      </td>

      <td>
        Read I/O operations
      </td>

      <td>
        `blkio.io_serviced_recursive`
      </td>
    </tr>

    <tr>
      <td>
        `blkio.io_serviced_write:`

        `<`

        *device*

        `>`
      </td>

      <td>
        Write I/O operations
      </td>

      <td>
        `blkio.io_serviced_recursive`
      </td>
    </tr>

    <tr>
      <td>
        `blkio.io_serviced_sync:`

        `<`

        *device*

        `>`
      </td>

      <td>
        Synchronous I/O operations
      </td>

      <td>
        `blkio.io_serviced_recursive`
      </td>
    </tr>

    <tr>
      <td>
        `blkio.io_serviced_async:`

        `<`

        *device*

        `>`
      </td>

      <td>
        Asynchronous I/O operations
      </td>

      <td>
        `blkio.io_serviced_recursive`
      </td>
    </tr>

    <tr>
      <td>
        `blkio.io_merged_read:`

        `<`

        *device*

        `>`
      </td>

      <td>
        Number of block I/Os (requests) merged into requests for I/O read operations
      </td>

      <td>
        `blkio.io_merged_recursive`
      </td>
    </tr>

    <tr>
      <td>
        `blkio.io_merged_write:`

        `<`

        *device*

        `>`
      </td>

      <td>
        Number of block I/Os (requests) merged into requests for I/O write operations
      </td>

      <td>
        `blkio.io_merged_recursive`
      </td>
    </tr>

    <tr>
      <td>
        `blkio.io_merged_sync:`

        `<`

        *device*

        `>`
      </td>

      <td>
        Number of block I/Os (requests) merged into requests for I/O synchronous operations
      </td>

      <td>
        `blkio.io_merged_recursive`
      </td>
    </tr>

    <tr>
      <td>
        `blkio.io_merged_async:`

        `<`

        *device*

        `>`
      </td>

      <td>
        Number of block I/Os (requests) merged into requests for I/O asynchronous operations
      </td>

      <td>
        `blkio.io_merged_recursive`
      </td>
    </tr>

    <tr>
      <td>
        `blkio.io_queued_read:`

        `<`

        *device*

        `>`
      </td>

      <td>
        Number of requests queued for I/O read operations
      </td>

      <td>
        `blkio.io_queued_recursive`
      </td>
    </tr>

    <tr>
      <td>
        `blkio.io_queued_write:`

        `<`

        *device*

        `>`
      </td>

      <td>
        Number of requests queued for I/O write operations
      </td>

      <td>
        `blkio.io_queued_recursive`
      </td>
    </tr>

    <tr>
      <td>
        `blkio.io_queued_sync:`

        `<`

        *device*

        `>`
      </td>

      <td>
        Number of requests queued for I/O synchronous operations
      </td>

      <td>
        `blkio.io_queued_recursive`
      </td>
    </tr>

    <tr>
      <td>
        `blkio.io_queued_async:`

        `<`

        *device*

        `>`
      </td>

      <td>
        Number of requests queued for I/O asynchronous operations
      </td>

      <td>
        `blkio.io_queued_recursive`
      </td>
    </tr>

    <tr>
      <td>
        `memory.rss`
      </td>

      <td>
        Anonymous and swap cache, not including tmpfs (shmem)
      </td>

      <td>
        `memory`
      </td>
    </tr>

    <tr>
      <td>
        `memory.pgpgin`
      </td>

      <td>
        Number of pages paged into memory
      </td>

      <td>
        `memory`
      </td>
    </tr>

    <tr>
      <td>
        `memory.pgpgout`
      </td>

      <td>
        Number of pages paged out of memory
      </td>

      <td>
        `memory`
      </td>
    </tr>

    <tr>
      <td>
        `memory.active_anon`
      </td>

      <td>
        Anonymous and swap cache on active least-recently-used (LRU) list, including tmpfs (shmem)
      </td>

      <td>
        `memory`
      </td>
    </tr>

    <tr>
      <td>
        `memory.inactive_anon`
      </td>

      <td>
        Anonymous and swap cache on inactive LRU list, including tmpfs (shmem)
      </td>

      <td>
        `memory`
      </td>
    </tr>

    <tr>
      <td>
        `memory.active_file`
      </td>

      <td>
        File-backed memory on active LRU list
      </td>

      <td>
        `memory`
      </td>
    </tr>

    <tr>
      <td>
        `memory.inactive_file`
      </td>

      <td>
        File-backed memory on inactive LRU list
      </td>

      <td>
        `memory`
      </td>
    </tr>

    <tr>
      <td>
        `memory.hierarchical`

        \
      </td>

      <td>
        Memory limit for the
      </td>

      <td>
        `memory`
      </td>
    </tr>

    <tr>
      <td>
        `_memory_limit`
      </td>

      <td>
        Hierarchy that contains the memory cgroup of job
      </td>

      <td />
    </tr>

    <tr>
      <td>
        `memory.hierarchical`

        \
      </td>

      <td>
        Memory plus swap limit for
      </td>

      <td>
        `memory`
      </td>
    </tr>

    <tr>
      <td>
        `_memsw_limit`
      </td>

      <td>
        Hierarchy that contains the memory cgroup of job
      </td>

      <td />
    </tr>
  </tbody>
</table>

#### G.1.8.3  Job Queue Metrics

Job queue metrics become available after workload manager jobs are run. They are produced by the `JobQueueSampler` data producer.

<table>
  <colgroup>
    <col />

    <col />
  </colgroup>

  <thead>
    <tr>
      <th colspan="2">*Table  [G.1.8.3](#g183-job-queue-metrics): List Of Job Queue Metrics*</th>
    </tr>
  </thead>

  <tbody>
    <tr>
      <td>
        **Metric**
      </td>

      <td>
        **Description**
      </td>
    </tr>

    <tr>
      <td>
        `AvgJobDuration`
      </td>

      <td>
        Average job duration of current jobs
      </td>
    </tr>

    <tr>
      <td>
        `AvgJobStartDelay`
      </td>

      <td>
        Average job start delay of current jobs
      </td>
    </tr>

    <tr>
      <td>
        `CompletedJobs`
      </td>

      <td>
        Successfully completed jobs
      </td>
    </tr>

    <tr>
      <td>
        `CoresInQueue`
      </td>

      <td>
        Cores in queue
      </td>
    </tr>

    <tr>
      <td>
        `CoresUPInQueue`
      </td>

      <td>
        Active cores in queue
      </td>
    </tr>

    <tr>
      <td>
        `EstimatedDelay`
      </td>

      <td>
        Estimated delay time to execute jobs
      </td>
    </tr>

    <tr>
      <td>
        `FailedJobs`
      </td>

      <td>
        Failed completed jobs
      </td>
    </tr>

    <tr>
      <td>
        `GPUsInQueue`
      </td>

      <td>
        GPUs in queue
      </td>
    </tr>

    <tr>
      <td>
        `GPUsUPInQueue`
      </td>

      <td>
        Active GPUs in queue
      </td>
    </tr>

    <tr>
      <td>
        `JobThroughput`
      </td>

      <td>
        Average number of Jobs finished
      </td>
    </tr>

    <tr>
      <td>
        `NodesInQueue`
      </td>

      <td>
        Number of nodes in the queue
      </td>
    </tr>

    <tr>
      <td>
        `NodesUPInQueue`
      </td>

      <td>
        Active nodes in queue
      </td>
    </tr>

    <tr>
      <td>
        `QueuedJobs`
      </td>

      <td>
        Queued jobs
      </td>
    </tr>

    <tr>
      <td>
        `RunningJobs`
      </td>

      <td>
        Running jobs
      </td>
    </tr>

    <tr>
      <td>
        `RunningJobsMax`
      </td>

      <td>
        Maximal running jobs
      </td>
    </tr>

    <tr>
      <td>
        `RunningJobsUtilization`
      </td>

      <td>
        Running jobs utilization
      </td>
    </tr>
  </tbody>
</table>

#### G.1.8.4  Job GPU Metrics

The GPU metrics of section [G.1.6](#g16-gpu-metrics) have a `GPUSampler` data producer (section [10.2.10](/admin/monitoring#10210-data-producer)). As the name suggests, those metrics are about gathering the sampled GPU data from the devices themselves.

The job GPU metrics of this section have a `JobSampler` data producer. As the name suggests, these metrics are about gathering the sampled job data from the GPU devices themselves.

In other words, the job GPU metrics are essentially the same metrics as in the GPU metrics section, but are valid only for the job. This is reflected in their names, which are identical, except for the `job_` prefix. Other differences of job GPU metrics in comparison with GPU metrics are that

* they are sampled only on the node that hosts the GPUs on which the job ran, and do not take specific GPUs as a parameter.

* the metrics are listed in the `monitoring measurable` mode of `cmsh` only after jobs have run on the GPUs.

Available job GPU metrics for V100 and A100 GPUs are listed in table [G.1.8.4](#g184-job-gpu-metrics).

If the cluster has been configured with [AMD GPUs (section 7.4 of the *Installation Manual*)](/installation/thirdparty#74-amd-gpu-driver-installation) then AMD job GPU metrics become available. Metrics in the table that are also valid AMD job GPU metrics are noted. Some metrics have been added and noted that are only valid for AMD GPUs.

<table>
  <thead>
    <tr>
      <th colspan="2">*Table [G.1.8.4](#g184-job-gpu-metrics): Job GPU Metrics For Node*</th>
    </tr>
  </thead>

  <tbody>
    <tr>
      <td>
        **Job GPU Metric**
      </td>

      <td>
        **Description For Job On Node**
      </td>
    </tr>

    <tr>
      <td>
        `job_gpu_dec_utilization`
      </td>

      <td>
        GPU decoding usage
      </td>
    </tr>

    <tr>
      <td>
        `job_gpu_ecc_dbe_agg`
      </td>

      <td>
        Total double bit aggregate ECC errors
      </td>
    </tr>

    <tr>
      <td>
        `job_gpu_ecc_dbe_vol`
      </td>

      <td>
        Total double bit volatile ECC errors
      </td>
    </tr>

    <tr>
      <td>
        `job_gpu_ecc_sbe_agg`
      </td>

      <td>
        Total single bit aggregate ECC errors
      </td>
    </tr>

    <tr>
      <td>
        `job_gpu_ecc_sbe_vol`
      </td>

      <td>
        Total single bit volatile ECC errors
      </td>
    </tr>

    <tr>
      <td>
        `job_gpu_enc_utilization`
      </td>

      <td>
        GPU encoding usage
      </td>
    </tr>

    <tr>
      <td>
        `job_gpu_enforced_power_limit`
      </td>

      <td>
        GPU-enforced power limit
      </td>
    </tr>

    <tr>
      <td>
        `job_gpu_mem_clock`
      </td>

      <td>
        GPU memory clock (also for AMD GPU)
      </td>
    </tr>

    <tr>
      <td>
        `job_gpu_mem_copy_utilization`
      </td>

      <td>
        Percentage of GPU memory copy used
      </td>
    </tr>

    <tr>
      <td>
        `job_gpu_mem_free`
      </td>

      <td>
        Amount of GPU free memory
      </td>
    </tr>

    <tr>
      <td>
        `job_gpu_mem_total`
      </td>

      <td>
        GPU framebuffer size (also for AMD GPU)
      </td>
    </tr>

    <tr>
      <td>
        `job_gpu_mem_used`
      </td>

      <td>
        Amount of GPU memory used (also for AMD GPU)
      </td>
    </tr>

    <tr>
      <td>
        `job_gpu_mem_utilization`
      </td>

      <td>
        GPU memory used percentage
      </td>
    </tr>

    <tr>
      <td>
        `job_gpu_memory_temp`
      </td>

      <td>
        GPU memory temperature
      </td>
    </tr>

    <tr>
      <td>
        `job_gpu_nvlink_total_bandwidth`
      </td>

      <td>
        Total NVLink bandwidth used
      </td>
    </tr>

    <tr>
      <td>
        `job_gpu_power_management_limit`
      </td>

      <td>
        GPU power management limit
      </td>
    </tr>

    <tr>
      <td>
        `job_gpu_power_usage`
      </td>

      <td>
        GPU power usage (also for AMD GPU)
      </td>
    </tr>

    <tr>
      <td>
        `job_gpu_power_violation`
      </td>

      <td>
        Throttling duration due to power constraints
      </td>
    </tr>

    <tr>
      <td>
        `job_gpu_shutdown_temp`
      </td>

      <td>
        GPU shutdown temperature
      </td>
    </tr>

    <tr>
      <td>
        `job_gpu_slowdown_temp`
      </td>

      <td>
        GPU slowdown temperature
      </td>
    </tr>

    <tr>
      <td>
        `job_gpu_sm_clock`
      </td>

      <td>
        GPU shader multiprocessor clock (also for AMD GPU)
      </td>
    </tr>

    <tr>
      <td>
        `job_gpu_temperature`
      </td>

      <td>
        GPU temperature
      </td>
    </tr>

    <tr>
      <td>
        `job_gpu_thermal_violation`
      </td>

      <td>
        Throttling duration due to thermal constraints
      </td>
    </tr>

    <tr>
      <td>
        `job_gpu_utilization`
      </td>

      <td>
        GPU utilization (section 

        [12.4.1](/admin/monitoringaccounting#1241-two-job-gpu-metrics-used-in-promql-queries)

        )
      </td>
    </tr>

    <tr>
      <td>
        `job_gpu_wasted`
      </td>

      <td>
        GPU wasted (section 

        [12.4.1](/admin/monitoringaccounting#1241-two-job-gpu-metrics-used-in-promql-queries)

        )
      </td>
    </tr>

    <tr>
      <td>
        `job_gpu_xid_error`
      </td>

      <td>
        The value is the specific XID error
      </td>
    </tr>
  </tbody>
</table>

### G.1.9  IPMI Metrics

The IPMI metrics correspond to metrics provided by the BMC devices. The metrics available depend on the manufacturer, and are detected by BCM. The metrics listed in the following table are a limited list of what may be detected on a system.

The data producer (section [10.2.10](/admin/monitoring#10210-data-producer)) for the IPMI metrics is the `ipmi` data producer.

<table>
  <thead>
    <tr>
      <th colspan="2">*Table  [G.1.9](#g19-ipmi-metrics): IPMI Metrics*</th>
    </tr>
  </thead>

  <tbody>
    <tr>
      <td>
        **IPMI Metric**
      </td>

      <td>
        **Description**
      </td>
    </tr>

    <tr>
      <td>
        `Current_<`

        *number*

        `>`

        \**
      </td>

      <td>
        Current seen by BMC sensor <

        *number*

        \>, in amps (file: 

        `sample_ipmi`

        )
      </td>
    </tr>

    <tr>
      <td>
        `Exhaust_Temp`
      </td>

      <td>
        Exhaust temperature, in Celsius
      </td>
    </tr>

    <tr>
      <td>
        `FETDRV_PG`
      </td>

      <td>
        PowerEdge voltage sensor on IPMI board
      </td>
    </tr>

    <tr>
      <td>
        `Fan<`

        *number*

        `>_RPM`
      </td>

      <td>
        RPM of Fan<

        *number*

        \> as seen by BMC (file: 

        `sample_ipmi`

        )
      </td>
    </tr>

    <tr>
      <td>
        `Fan_Redundancy`
      </td>

      <td>
        [fan redundancy status](https://www.dell.com/support/manuals/nl-nl/dell-opnmang-sw-v8.0.1/eemi_13g-v1/rdu-event-messages?guid=guid-0a608eff-2318-4c32-9395-499fa2b6a15e\&lang=en-us)
      </td>
    </tr>

    <tr>
      <td>
        `Inlet_Temp`

        \**
      </td>

      <td>
        Inlet Temperature, in Celsius (file: 

        `sample_ipmi`

        )
      </td>
    </tr>

    <tr>
      <td>
        `M01_VDDQ_PG`
      </td>

      <td>
        PowerEdge CPU voltage sensor M01_VDDQ_PG
      </td>
    </tr>

    <tr>
      <td>
        `M01_VTT_PG`
      </td>

      <td>
        PowerEdge CPU voltage sensor
      </td>
    </tr>

    <tr>
      <td>
        `M23_VDDQ_PG`
      </td>

      <td>
        PowerEdge CPU voltage sensor
      </td>
    </tr>

    <tr>
      <td>
        `M23_VTT_PG `
      </td>

      <td>
        PowerEdge CPU voltage sensor
      </td>
    </tr>

    <tr>
      <td>
        `NDC_PG`
      </td>

      <td>
        PowerEdge system board voltage sensor
      </td>
    </tr>

    <tr>
      <td>
        `PFault_Fail_Safe`
      </td>

      <td>
        PowerEdge sensor
      </td>
    </tr>

    <tr>
      <td>
        `PLL_PG`
      </td>

      <td>
        PowerEdge sensor
      </td>
    </tr>

    <tr>
      <td>
        `PS1_PG_Fail`
      </td>

      <td>
        PowerEdge sensor
      </td>
    </tr>

    <tr>
      <td>
        `PS2_PG_Fail`
      </td>

      <td>
        PowerEdge sensor
      </td>
    </tr>

    <tr>
      <td>
        `Pwr_Consumption`
      </td>

      <td>
        Power consumed by BMC, in watts (file: 

        `sample_ipmi`

        )
      </td>
    </tr>

    <tr>
      <td>
        `Temp`
      </td>

      <td>
        Temperature, in Celsius
      </td>
    </tr>

    <tr>
      <td>
        `VSA_PG`
      </td>

      <td>
        PowerEdge voltage sensor
      </td>
    </tr>

    <tr>
      <td>
        `VTT_PG`
      </td>

      <td>
        PowerEdge voltage sensor
      </td>
    </tr>

    <tr>
      <td>
        `Voltage_<`

        *number*

        `>`

        \**
      </td>

      <td>
        Voltage seen by BMC sensor 

        *number*

        , in Volts (file: 

        `sample_ipmi`

        )
      </td>
    </tr>
  </tbody>
</table>

\*\* Standalone scripts, not built-ins.

If sampling from a head node, the script is in directory: `/cm/local/apps/cmd/scripts/metrics/`

For regular nodes, the script is in directory: `/cm/images/default-image/cm/local/apps/cmd/scripts/metrics/`

### G.1.10  Redfish Metrics

By default Redfish metrics are sampled only when the BMC interface (section [3.7](/admin/configuring#37-configuring-bmc-ipmiilodraccimcredfish-interfaces)) starts with `rf` (e.g. `rf0`, `rf1` etc). The Redfish metric sampler is enabled by setting `userdefinedresources` as follows:

**Example**

```bash
[basecm11->device[node001]]% set userdefinedresources redfish
[basecm11->device[node001]]% commit
```

#### G.1.10.1  Redfish Standalone Script Metrics

An example script that samples Redfish metrics is the standalone script `/cm/images/default-image/cm/local/apps/cmd/scripts/metrics/sample_redfish`:

<table>
  <thead>
    <tr>
      <th colspan="2">*Table  [G.1.10](#g110-redfish-metrics): Redfish Standalone Script Metrics*</th>
    </tr>
  </thead>

  <tbody>
    <tr>
      <td>
        **Redfish Standalone Script Metrics**
      </td>

      <td>
        **Description**
      </td>
    </tr>

    <tr>
      <td>
        `fan_speed`
      </td>

      <td>
        Speed of fan (RPM)
      </td>
    </tr>

    <tr>
      <td>
        `psu_power`
      </td>

      <td>
        The average power supply unit power (W)
      </td>
    </tr>

    <tr>
      <td>
        `sensor_reading`
      </td>

      <td>
        Temperature sensor reading (C)
      </td>
    </tr>
  </tbody>
</table>

#### G.1.10.2  Redfish Metrics From CMDaemon Built-in Sampling

CMDaemon built-in sampling also samples Redfish metrics from the hardware:

**Example**

```bash
[basecm11->device[node001]]% latestmetricdata | head -2; latestmetricdata | grep ^RF
Measurable                                       Parameter Type                                    Value    ...
------------------------------------------------ --------- --------------------------------------- ---------
RF_Baseboard_0_PCB_0_Temp_0                      reading   Environmental/Redfish/Sensor reading    34 C
RF_Baseboard_0_PCB_1_Temp_0                      reading   Environmental/Redfish/Sensor reading    33.5 C
RF_Baseboard_0_PCB_2_Temp_0                      reading   Environmental/Redfish/Sensor reading    33 C
RF_Baseboard_0_StandbyHSC_0_Power_0              reading   Environmental/Redfish/Sensor reading    34.746 W
RF_Baseboard_0_StandbyHSC_0_Temp_0               reading   Environmental/Redfish/Sensor reading    33.4375 C
RF_C2C_0_Resource_MaxSpeed                                 Environmental/Redfish/Port              0 B/s
RF_DIMM_Slot_AllowedSpeeds                                 Environmental/Redfish/Memory            no data
RF_DIMM_Slot_Capacity                                      Environmental/Redfish/Memory            0 B
RF_DIMM_Slot_CapacityMiB                                   Environmental/Redfish/Memory            0 B
RF_DIMM_Slot_Nvidia_RowRemappingFailed                     Environmental/Redfish/Memory            0
RF_GPU_0_DRAM_0_Memory_Metrics_Bandwidth                   Environmental/Redfish/Memorymetrics     0.0%
RF_GPU_0_DRAM_0_Memory_Metrics_CapacityUtilizat+           Environmental/Redfish/Memorymetrics     0.0%
RF_GPU_0_Processor_Metrics_Bandwidth                       Environmental/Redfish/Processormetrics  0.0%
RF_GPU_0_Processor_Metrics_LifeTime_Correctable+           Environmental/Redfish/Processormetrics  0
RF_GPU_0_Processor_Metrics_LifeTime_Uncorrectab+           Environmental/Redfish/Processormetrics  0
RF_GPU_0_Processor_Metrics_Nvidia_AccumulatedGP+           Environmental/Redfish/Processormetrics  no data
RF_GPU_0_Processor_Metrics_Nvidia_AccumulatedSM+           Environmental/Redfish/Processormetrics  no data
RF_GPU_0_Processor_Metrics_Nvidia_DMMAUtilizatio           Environmental/Redfish/Processormetrics  0.0%
RF_GPU_0_Processor_Metrics_Nvidia_FP16Activity             Environmental/Redfish/Processormetrics  0.0%
```

The preceding list is a truncated excerpt from an NVIDIA DGX system. The Redfish metrics that are available depend on the vendor and hardware.

Further examples of Redfish measurables on an NVIDIA DGX system can be seen in [section 12.8 (metrics)](/nvidia-mission-control/mcmetricshealthchecksactions#128-redfish-metrics-for-the-dgx-gb200), [section 12.9 (enums)](/nvidia-mission-control/mcmetricshealthchecksactions#129-redfish-enums-for-the-dgx-gb200), and [section 12.10 (health checks)](/nvidia-mission-control/mcmetricshealthchecksactions#1210-health-checks-for-the-dgx-gb200) of the *NVIDIA Mission Control Manual*.

### G.1.11  SMART Metrics

The SMART metrics correspond to metrics provided by SMART hard drive implementation. The metrics available depend on the manufacturer, and are detected by BCM. The metrics listed in the following table are a limited list of what may be detected on a system.

A SMART metric takes a block device name (`sda`, `sdc`, `nvme0n1` and so on) as a parameter. Some reported values, such as “spin up time” and “start/stop count” are not relevant on non-rotational (solid-state) drives.

The data producer (section [10.2.10](/admin/monitoring#10210-data-producer)) for SMART metrics is the `smart` data producer.

<table>
  <thead>
    <tr>
      <th colspan="2">*Table  [G.1.11](#g111-smart-metrics): SMART Metrics*</th>
    </tr>
  </thead>

  <tbody>
    <tr>
      <td>
        **SMART Metric**
      </td>

      <td>
        **Description**
      </td>
    </tr>

    <tr>
      <td>
        `Command_Timeout`
      </td>

      <td>
        Command timeout
      </td>
    </tr>

    <tr>
      <td>
        `Current_Pending_Sector`
      </td>

      <td>
        Current pending sectors
      </td>
    </tr>

    <tr>
      <td>
        `Hardware_ECC_Recovered`
      </td>

      <td>
        Hardware ECC recovered
      </td>
    </tr>

    <tr>
      <td>
        `Offline_Uncorrectable`
      </td>

      <td>
        Uncorrectable sectors
      </td>
    </tr>

    <tr>
      <td>
        `Raw_Read_Error_Rate`
      </td>

      <td>
        Raw read error rate
      </td>
    </tr>

    <tr>
      <td>
        `Reallocated_Sector_Ct`
      </td>

      <td>
        Reallocated sectors count
      </td>
    </tr>

    <tr>
      <td>
        `Reported_Uncorrect`
      </td>

      <td>
        Reported uncorrectable errors
      </td>
    </tr>

    <tr>
      <td>
        `UDMA_CRC_Error_Count`
      </td>

      <td>
        UDMA CRC errors
      </td>
    </tr>
  </tbody>
</table>

### G.1.12  Prometheus Metrics

Prometheus metrics are introduced in section [12.2](/admin/monitoringaccounting#122-labeled-entities).

The data producers for Prometheus metrics are JobSampler, JobMetadataSampler, and some others.

<table>
  <thead>
    <tr>
      <th colspan="2">*Table  [G.1.12](#g112-prometheus-metrics): Prometheus Metrics*</th>
    </tr>
  </thead>

  <tbody>
    <tr>
      <td>
        **Prometheus Metric**
      </td>

      <td>
        **Description (for a job, unless asterisked)**
      </td>
    </tr>

    <tr>
      <td>
        `job_blkio_io_merged`
      </td>

      <td>
        Number of block I/Os (requests) merged into requests for I/O operations by a cgroup
      </td>
    </tr>

    <tr>
      <td>
        `job_blkio_io_queued`
      </td>

      <td>
        Number of requests queued for I/O operations by a cgroup
      </td>
    </tr>

    <tr>
      <td>
        `job_blkio_io_service_bytes`
      </td>

      <td>
        Reports the number of bytes transferred to or from specific devices by a cgroup as seen by the CFQ scheduler
      </td>
    </tr>

    <tr>
      <td>
        `job_blkio_io_service_bytes_total`
      </td>

      <td>
        Reports the number of bytes transferred to or from specific devices by a cgroup as seen by the CFQ scheduler (for all jobs)

        \*
      </td>
    </tr>

    <tr>
      <td>
        `job_blkio_io_service_time_seconds`
      </td>

      <td>
        Reports the total time in seconds between request dispatch and request completion for I/O operations on specific devices by a cgroup as seen by the CFQ scheduler
      </td>
    </tr>

    <tr>
      <td>
        `job_blkio_io_serviced`
      </td>

      <td>
        Reports the number of I/O operations performed on specific devices by a cgroup as seen by the CFQ scheduler
      </td>
    </tr>

    <tr>
      <td>
        `job_blkio_io_wait_time_seconds`
      </td>

      <td>
        Reports the total time I/O operations on specific devices by a cgroup spent waiting for service in the scheduler queues
      </td>
    </tr>

    <tr>
      <td>
        `job_blkio_sectors`
      </td>

      <td>
        Reports the number of sectors transferred to or from specific devices by a cgroup
      </td>
    </tr>

    <tr>
      <td>
        `job_blkio_time_seconds`
      </td>

      <td>
        Reports the time that a cgroup had I/O access to specific devices
      </td>
    </tr>

    <tr>
      <td>
        `job_cpuacct_stat_system`
      </td>

      <td>
        System CPU time consumed by processes
      </td>
    </tr>

    <tr>
      <td>
        `job_cpuacct_stat_user`
      </td>

      <td>
        User CPU time consumed by processes
      </td>
    </tr>

    <tr>
      <td>
        `job_cpuacct_usage_seconds`
      </td>

      <td>
        CPU usage time consumed
      </td>
    </tr>

    <tr>
      <td>
        `job_memory_active_anon_bytes`
      </td>

      <td>
        Anonymous and swap cache on active least-recently-used (LRU) list, including tmpfs (shmem), in bytes
      </td>
    </tr>

    <tr>
      <td>
        `job_memory_active_file_bytes`
      </td>

      <td>
        File-backed memory on active LRU list, in bytes
      </td>
    </tr>

    <tr>
      <td>
        `job_memory_cache_bytes`
      </td>

      <td>
        Page cache, including tmpfs (shmem), in bytes
      </td>
    </tr>

    <tr>
      <td>
        `job_memory_failcnt`
      </td>

      <td>
        Reports the number of times that the memory limit has reached the value set in memory.limit_in_bytes
      </td>
    </tr>

    <tr>
      <td>
        `job_memory_hierarchical_memory_limit_bytes`
      </td>

      <td>
        Memory limit for the hierarchy that contains the memory cgroup, in bytes
      </td>
    </tr>

    <tr>
      <td>
        `job_memory_hierarchical_memsw_limit_bytes`
      </td>

      <td>
        Memory plus swap limit for the hierarchy that contains the memory cgroup, in bytes
      </td>
    </tr>

    <tr>
      <td>
        `job_memory_inactive_anon_bytes`
      </td>

      <td>
        Anonymous and swap cache on inactive LRU list, including tmpfs (shmem), in bytes
      </td>
    </tr>

    <tr>
      <td>
        `job_memory_inactive_file_bytes`
      </td>

      <td>
        File-backed memory on inactive LRU list, in bytes
      </td>
    </tr>

    <tr>
      <td>
        `job_memory_mapped_file_bytes`
      </td>

      <td>
        Size of memory-mapped mapped files, including tmpfs (shmem), in bytes
      </td>
    </tr>

    <tr>
      <td>
        `job_memory_memsw_failcnt`
      </td>

      <td>
        Reports the number of times that the memory plus swap space limit has reached the value set in 

        `memory.memsw.limit_in_bytes`
      </td>
    </tr>

    <tr>
      <td>
        `job_memory_memsw_max_usage_bytes`
      </td>

      <td>
        Reports the maximum amount of memory and swap space used by processes in the cgroup, in bytes
      </td>
    </tr>

    <tr>
      <td>
        `job_memory_memsw_usage_bytes`
      </td>

      <td>
        Reports the sum of current memory usage plus swap space used by processes in the cgroup, in bytes
      </td>
    </tr>

    <tr>
      <td>
        `job_memory_pgpgin_bytes`
      </td>

      <td>
        Number of pages paged into memory
      </td>
    </tr>

    <tr>
      <td>
        `job_memory_pgpgout_bytes`
      </td>

      <td>
        Number of pages paged out of memory
      </td>
    </tr>

    <tr>
      <td>
        `job_memory_rss_bytes`
      </td>

      <td>
        Anonymous and swap cache, not including tmpfs (shmem), in bytes
      </td>
    </tr>

    <tr>
      <td>
        `job_memory_swap_bytes`
      </td>

      <td>
        Swap usage, in bytes
      </td>
    </tr>

    <tr>
      <td>
        `job_memory_unevictable_bytes`
      </td>

      <td>
        Memory that cannot be reclaimed, in bytes,
      </td>
    </tr>

    <tr>
      <td>
        `job_memory_usage_bytes`
      </td>

      <td>
        Reports the total current memory usage by processes in the cgroup, in bytes
      </td>
    </tr>

    <tr>
      <td>
        `job_memory_usage_bytes_total`
      </td>

      <td>
        Reports the total current memory usage by processes in the cgroup, in bytes (for all jobs)

        \*
      </td>
    </tr>

    <tr>
      <td>
        `job_metadata_allocated_cpu_cores`
      </td>

      <td>
        CPU cores used by a job by the user
      </td>
    </tr>

    <tr>
      <td>
        `job_metadata_allocated_gpus`
      </td>

      <td>
        GPUs used by a job by the user
      </td>
    </tr>

    <tr>
      <td>
        `job_metadata_is_running`
      </td>

      <td>
        Returns 1 if the job metadata sampler is running a job
      </td>
    </tr>

    <tr>
      <td>
        `job_metadata_is_waiting`
      </td>

      <td>
        Returns 1 if the job metadata sampler is waiting
      </td>
    </tr>

    <tr>
      <td>
        `job_metadata_num_cpus`
      </td>

      <td>
        Number of CPUs that the job runs on
      </td>
    </tr>

    <tr>
      <td>
        `job_metadata_num_nodes`
      </td>

      <td>
        Number of nodes that the job runs on
      </td>
    </tr>

    <tr>
      <td>
        `job_metadata_pending_jobs`
      </td>

      <td>
        Number of pending jobs for the user
      </td>
    </tr>

    <tr>
      <td>
        `job_metadata_running_jobs`
      </td>

      <td>
        Number of running jobs for the user
      </td>
    </tr>

    <tr>
      <td>
        `job_metadata_running_seconds`
      </td>

      <td>
        Time the job has run
      </td>
    </tr>

    <tr>
      <td>
        `job_metadata_waiting_seconds`
      </td>

      <td>
        Time the job has been waiting to run
      </td>
    </tr>

    <tr>
      <td>
        `users_job_effective_cpu_seconds:1w`
      </td>

      <td>
        CPU seconds used by users over the past week

        \*
      </td>
    </tr>

    <tr>
      <td>
        `users_job_running_count:1w`
      </td>

      <td>
        Number of jobs run by users over the past week

        \*
      </td>
    </tr>

    <tr>
      <td>
        `users_job_waiting_seconds:1w`
      </td>

      <td>
        Time users have been waiting for jobs to run over the past week

        \*
      </td>
    </tr>

    <tr>
      <td>
        `users_job_wall_clock_seconds:1w`
      </td>

      <td>
        Time the jobs runs for users according to wall time over the past week

        \*
      </td>
    </tr>

    <tr>
      <td>
        `users_job_wasted_cpu_seconds:1w`
      </td>

      <td>
        CPU time wasted during users jobs over the past week

        \*
      </td>
    </tr>
  </tbody>
</table>

\* total in the past 7x24x60x60 seconds, as measured at the time of sampling

### G.1.13  NetQ Metrics

NetQ metrics (table [G.1.13](#g113-netq-metrics)) are metrics sourced from NetQ. Configuring NetQ with BCM is described in section [3.11](/admin/configuring#311-configuring-the-netq-network-management-system).

The data producer (section [10.2.10](/admin/monitoring#10210-data-producer)) for NetQ metrics is the `netq` data producer.

<table>
  <thead>
    <tr>
      <th colspan="2">*Table  [G.1.13](#g113-netq-metrics): NetQ Metrics*</th>
    </tr>
  </thead>

  <tbody>
    <tr>
      <td>
        **NetQ Metric**
      </td>

      <td>
        **Description**
      </td>
    </tr>

    <tr>
      <td>
        `NetQ_node_fan_speed`
      </td>

      <td>
        Fan speed (RPM)
      </td>
    </tr>

    <tr>
      <td>
        `NetQ_node_nvlink_rx_all_flits`
      </td>

      <td>
        NVLink packets received, control and data (

        [FLITs/s](https://handwiki.org/wiki/FLITs)

        )
      </td>
    </tr>

    <tr>
      <td>
        `NetQ_node_nvlink_rx_data_flits`
      </td>

      <td>
        NVLink packets received, data (FLITs/s)
      </td>
    </tr>

    <tr>
      <td>
        `NetQ_node_nvlink_tx_all_flits`
      </td>

      <td>
        NVLink packets sent, control and data (FLITs/s)
      </td>
    </tr>

    <tr>
      <td>
        `NetQ_node_nvlink_tx_data_flits`
      </td>

      <td>
        NVLink packets sent, data (FLITs/s)
      </td>
    </tr>

    <tr>
      <td>
        `NetQ_node_PSU_power_input`
      </td>

      <td>
        Power supply power input (W)
      </td>
    </tr>

    <tr>
      <td>
        `NetQ_node_PSU_power_output`
      </td>

      <td>
        Power supply power output (W)
      </td>
    </tr>

    <tr>
      <td>
        `NetQ_node_PSU_voltage_input`
      </td>

      <td>
        Power supply voltage input (V)
      </td>
    </tr>

    <tr>
      <td>
        `NetQ_node_PSU_voltage_output`
      </td>

      <td>
        Power supply voltage output (V)
      </td>
    </tr>

    <tr>
      <td>
        `NetQ_node_Temp_sensor`
      </td>

      <td>
        Temperature sensor value (C)
      </td>
    </tr>
  </tbody>
</table>

NetQ health checks are covered in section [G.2.4](#g24-netq-health-checks).

### G.1.14  Kubernetes Metrics

Configuring Kubernetes with BCM is described in [Chapter 4 of the *Containerization Manual*](/containerization/kubernetes).

There were cluster Kubernetes metrics described earlier on in table [G.1.1](#g11-regular-metrics). Those were overview metrics about Kubernetes.

The Kubernetes metrics in the following table [G.1.14](#g114-kubernetes-metrics) are state metrics sourced from Kubernetes itself. The data producer (section [10.2.10](/admin/monitoring#10210-data-producer)) for the Kubernetes metrics of this section is `kubestatemetrics`.

The metrics that are not tagged as \[STABLE] are experimental, and may change in behavior during Kubernetes updates.

<table>
  <thead>
    <tr>
      <th colspan="2">*Table [G.1.14](#g114-kubernetes-metrics): Kubernetes Metrics*</th>
    </tr>
  </thead>

  <tbody>
    <tr>
      <td>
        **Kubernetes Metric**
      </td>

      <td>
        **Description**
      </td>
    </tr>

    <tr>
      <td>
        `kube_configmap_annotations`
      </td>

      <td>
        Kubernetes annotations converted to Prometheus labels.
      </td>
    </tr>

    <tr>
      <td>
        `kube_configmap_created`
      </td>

      <td>
        \[STABLE] Unix creation timestamp
      </td>
    </tr>

    <tr>
      <td>
        `kube_configmap_info`
      </td>

      <td>
        \[STABLE] Information about configmap.
      </td>
    </tr>

    <tr>
      <td>
        `kube_configmap_labels`
      </td>

      <td>
        \[STABLE] Kubernetes labels converted to Prometheus labels.
      </td>
    </tr>

    <tr>
      <td>
        `kube_configmap_metadata_resource_version`
      </td>

      <td>
        Resource version representing a specific version of the configmap.
      </td>
    </tr>

    <tr>
      <td>
        `kube_cronjob_annotations`
      </td>

      <td>
        Kubernetes annotations converted to Prometheus labels.
      </td>
    </tr>

    <tr>
      <td>
        `kube_cronjob_created`
      </td>

      <td>
        \[STABLE] Unix creation timestamp
      </td>
    </tr>

    <tr>
      <td>
        `kube_cronjob_info`
      </td>

      <td>
        \[STABLE] Info about cronjob.
      </td>
    </tr>

    <tr>
      <td>
        `kube_cronjob_labels`
      </td>

      <td>
        \[STABLE] Kubernetes labels converted to Prometheus labels.
      </td>
    </tr>

    <tr>
      <td>
        `kube_cronjob_metadata_resource_version`
      </td>

      <td>
        \[STABLE] Resource version representing a specific version of the cronjob.
      </td>
    </tr>

    <tr>
      <td>
        `kube_cronjob_next_schedule_time`
      </td>

      <td>
        \[STABLE] Next time the cronjob should be scheduled. The time after lastScheduleTime, or after the cron job’s creation time if it’s never been scheduled. Use this to determine if the job is delayed.
      </td>
    </tr>

    <tr>
      <td>
        `kube_cronjob_spec_failed_job_history_limit`
      </td>

      <td>
        Failed job history limit tells the controller how many failed jobs should be preserved.
      </td>
    </tr>

    <tr>
      <td>
        `kube_cronjob_spec_successful_job_history_ limit`
      </td>

      <td>
        Successful job history limit tells the controller how many completed jobs should be preserved.
      </td>
    </tr>

    <tr>
      <td>
        `kube_cronjob_spec_suspend`
      </td>

      <td>
        \[STABLE] Suspend flag tells the controller to suspend subsequent executions.
      </td>
    </tr>

    <tr>
      <td>
        `kube_cronjob_status_active`
      </td>

      <td>
        \[STABLE] Active holds pointers to currently running jobs.
      </td>
    </tr>

    <tr>
      <td>
        `kube_cronjob_status_last_schedule_time`
      </td>

      <td>
        \[STABLE] LastScheduleTime keeps information of when was the last time the job was successfully scheduled.
      </td>
    </tr>

    <tr>
      <td>
        `kube_cronjob_status_last_successful_time`
      </td>

      <td>
        LastSuccessfulTime keeps information of when was the last time the job was completed successfully.
      </td>
    </tr>

    <tr>
      <td>
        `kube_daemonset_annotations`
      </td>

      <td>
        Kubernetes annotations converted to Prometheus labels.
      </td>
    </tr>

    <tr>
      <td>
        `kube_daemonset_created`
      </td>

      <td>
        \[STABLE] Unix creation timestamp
      </td>
    </tr>

    <tr>
      <td>
        `kube_daemonset_labels`
      </td>

      <td>
        \[STABLE] Kubernetes labels converted to Prometheus labels.
      </td>
    </tr>

    <tr>
      <td>
        `kube_daemonset_metadata_generation`
      </td>

      <td>
        \[STABLE] Sequence number representing a specific generation of the desired state.
      </td>
    </tr>

    <tr>
      <td>
        `kube_daemonset_status_current_number_ scheduled`
      </td>

      <td>
        \[STABLE] The number of nodes running at least one daemon pod and are supposed to.
      </td>
    </tr>

    <tr>
      <td>
        `kube_daemonset_status_desired_number_ scheduled`
      </td>

      <td>
        \[STABLE] The number of nodes that should be running the daemon pod.
      </td>
    </tr>

    <tr>
      <td>
        `kube_daemonset_status_number_available`
      </td>

      <td>
        \[STABLE] The number of nodes that should be running the daemon pod and have one or more of the daemon pod running and available
      </td>
    </tr>

    <tr>
      <td>
        `kube_daemonset_status_number_misscheduled`
      </td>

      <td>
        \[STABLE] The number of nodes running a daemon pod but are not supposed to.
      </td>
    </tr>

    <tr>
      <td>
        `kube_daemonset_status_number_ready`
      </td>

      <td>
        \[STABLE] The number of nodes that should be running the daemon pod and have one or more of the daemon pod running and ready.
      </td>
    </tr>

    <tr>
      <td>
        `kube_daemonset_status_number_unavailable`
      </td>

      <td>
        \[STABLE] The number of nodes that should be running the daemon pod and have none of the daemon pod running and available
      </td>
    </tr>

    <tr>
      <td>
        `kube_daemonset_status_observed_generation`
      </td>

      <td>
        \[STABLE] The most recent generation observed by the daemon set controller.
      </td>
    </tr>

    <tr>
      <td>
        `kube_daemonset_status_updated_number_ scheduled`
      </td>

      <td>
        \[STABLE] The total number of nodes that are running updated daemon pod
      </td>
    </tr>

    <tr>
      <td>
        `kube_deployment_annotations`
      </td>

      <td>
        Kubernetes annotations converted to Prometheus labels.
      </td>
    </tr>

    <tr>
      <td>
        `kube_deployment_created`
      </td>

      <td>
        \[STABLE] Unix creation timestamp
      </td>
    </tr>

    <tr>
      <td>
        `kube_deployment_labels`
      </td>

      <td>
        \[STABLE] Kubernetes labels converted to Prometheus labels.
      </td>
    </tr>

    <tr>
      <td>
        `kube_deployment_metadata_generation`
      </td>

      <td>
        \[STABLE] Sequence number representing a specific generation of the desired state.
      </td>
    </tr>

    <tr>
      <td>
        `kube_deployment_spec_paused`
      </td>

      <td>
        \[STABLE] Whether the deployment is paused and will not be processed by the deployment controller.
      </td>
    </tr>

    <tr>
      <td>
        `kube_deployment_spec_replicas`
      </td>

      <td>
        \[STABLE] Number of desired pods for a deployment.
      </td>
    </tr>

    <tr>
      <td>
        `kube_deployment_spec_strategy_ rollingupdate_max_surge`
      </td>

      <td>
        \[STABLE] Maximum number of replicas that can be scheduled above the desired number of replicas during a rolling update of a deployment.
      </td>
    </tr>

    <tr>
      <td>
        `kube_deployment_spec_strategy_ rollingupdate_max_unavailable`
      </td>

      <td>
        \[STABLE] Maximum number of unavailable replicas during a rolling update of a deployment.
      </td>
    </tr>

    <tr>
      <td>
        `kube_deployment_status_condition`
      </td>

      <td>
        \[STABLE] The current status conditions of a deployment.
      </td>
    </tr>

    <tr>
      <td>
        `kube_deployment_status_observed_generation`
      </td>

      <td>
        \[STABLE] The generation observed by the deployment controller.
      </td>
    </tr>

    <tr>
      <td>
        `kube_deployment_status_replicas`
      </td>

      <td>
        \[STABLE] The number of replicas per deployment.
      </td>
    </tr>

    <tr>
      <td>
        `kube_deployment_status_replicas_available`
      </td>

      <td>
        \[STABLE] The number of available replicas per deployment.
      </td>
    </tr>

    <tr>
      <td>
        `kube_deployment_status_replicas_ready`
      </td>

      <td>
        \[STABLE] The number of ready replicas per deployment.
      </td>
    </tr>

    <tr>
      <td>
        `kube_deployment_status_replicas_unavailable`
      </td>

      <td>
        \[STABLE] The number of unavailable replicas per deployment.
      </td>
    </tr>

    <tr>
      <td>
        `kube_deployment_status_replicas_updated`
      </td>

      <td>
        \[STABLE] The number of updated replicas per deployment.
      </td>
    </tr>

    <tr>
      <td>
        `kube_endpoint_address`
      </td>

      <td>
        \[STABLE] Information about Endpoint available and non available addresses.
      </td>
    </tr>

    <tr>
      <td>
        `kube_endpoint_address_available`
      </td>

      <td>
        (Deprecated since v2.6.0) Number of addresses available in endpoint.
      </td>
    </tr>

    <tr>
      <td>
        `kube_endpoint_address_not_ready`
      </td>

      <td>
        (Deprecated since v2.6.0) Number of addresses not ready in endpoint.
      </td>
    </tr>

    <tr>
      <td>
        `kube_endpoint_annotations`
      </td>

      <td>
        Kubernetes annotations converted to Prometheus labels.
      </td>
    </tr>

    <tr>
      <td>
        `kube_endpoint_created`
      </td>

      <td>
        \[STABLE] Unix creation timestamp
      </td>
    </tr>

    <tr>
      <td>
        `kube_endpoint_info`
      </td>

      <td>
        \[STABLE] Information about endpoint.
      </td>
    </tr>

    <tr>
      <td>
        `kube_endpoint_labels`
      </td>

      <td>
        \[STABLE] Kubernetes labels converted to Prometheus labels.
      </td>
    </tr>

    <tr>
      <td>
        `kube_endpoint_ports`
      </td>

      <td>
        \[STABLE] Information about the Endpoint ports.
      </td>
    </tr>

    <tr>
      <td>
        `kube_ingress_annotations`
      </td>

      <td>
        Kubernetes annotations converted to Prometheus labels.
      </td>
    </tr>

    <tr>
      <td>
        `kube_ingress_created`
      </td>

      <td>
        \[STABLE] Unix creation timestamp
      </td>
    </tr>

    <tr>
      <td>
        `kube_ingress_info`
      </td>

      <td>
        \[STABLE] Information about ingress.
      </td>
    </tr>

    <tr>
      <td>
        `kube_ingress_labels`
      </td>

      <td>
        \[STABLE] Kubernetes labels converted to Prometheus labels.
      </td>
    </tr>

    <tr>
      <td>
        `kube_ingress_metadata_resource_version`
      </td>

      <td>
        Resource version representing a specific version of ingress.
      </td>
    </tr>

    <tr>
      <td>
        `kube_ingress_path`
      </td>

      <td>
        \[STABLE] Ingress host, paths and backend service information.
      </td>
    </tr>

    <tr>
      <td>
        `kube_job_annotations`
      </td>

      <td>
        Kubernetes annotations converted to Prometheus labels.
      </td>
    </tr>

    <tr>
      <td>
        `kube_job_complete`
      </td>

      <td>
        \[STABLE] The job has completed its execution.
      </td>
    </tr>

    <tr>
      <td>
        `kube_job_created`
      </td>

      <td>
        \[STABLE] Unix creation timestamp
      </td>
    </tr>

    <tr>
      <td>
        `kube_job_info`
      </td>

      <td>
        \[STABLE] Information about job.
      </td>
    </tr>

    <tr>
      <td>
        `kube_job_labels`
      </td>

      <td>
        \[STABLE] Kubernetes labels converted to Prometheus labels.
      </td>
    </tr>

    <tr>
      <td>
        `kube_job_owner`
      </td>

      <td>
        \[STABLE] Information about the Job’s owner.
      </td>
    </tr>

    <tr>
      <td>
        `kube_job_spec_completions`
      </td>

      <td>
        \[STABLE] The desired number of successfully finished pods the job should be run with.
      </td>
    </tr>

    <tr>
      <td>
        `kube_job_spec_parallelism`
      </td>

      <td>
        \[STABLE] The maximum desired number of pods the job should run at any given time.
      </td>
    </tr>

    <tr>
      <td>
        `kube_job_status_active`
      </td>

      <td>
        \[STABLE] The number of actively running pods.
      </td>
    </tr>

    <tr>
      <td>
        `kube_job_status_completion_time`
      </td>

      <td>
        \[STABLE] CompletionTime represents time when the job was completed.
      </td>
    </tr>

    <tr>
      <td>
        `kube_job_status_failed`
      </td>

      <td>
        \[STABLE] The number of pods which reached Phase Failed and the reason for failure.
      </td>
    </tr>

    <tr>
      <td>
        `kube_job_status_start_time`
      </td>

      <td>
        \[STABLE] StartTime represents time when the job was acknowledged by the Job Manager.
      </td>
    </tr>

    <tr>
      <td>
        `kube_job_status_succeeded`
      </td>

      <td>
        \[STABLE] The number of pods which reached Phase Succeeded.
      </td>
    </tr>

    <tr>
      <td>
        `kube_lease_owner`
      </td>

      <td>
        Information about the Lease’s owner.
      </td>
    </tr>

    <tr>
      <td>
        `kube_lease_renew_time`
      </td>

      <td>
        Kube lease renew time.
      </td>
    </tr>

    <tr>
      <td>
        `kube_mutatingwebhookconfiguration_created`
      </td>

      <td>
        Unix creation timestamp.
      </td>
    </tr>

    <tr>
      <td>
        `kube_mutatingwebhookconfiguration_info`
      </td>

      <td>
        Information about the MutatingWebhookConfiguration.
      </td>
    </tr>

    <tr>
      <td>
        `kube_mutatingwebhookconfiguration_ metadata_resource_version`
      </td>

      <td>
        Resource version representing a specific version of the MutatingWebhookConfiguration.
      </td>
    </tr>

    <tr>
      <td>
        `kube_mutatingwebhookconfiguration_ webhook_clientconfig_service`
      </td>

      <td>
        Service used by the apiserver to connect to a mutating webhook.
      </td>
    </tr>

    <tr>
      <td>
        `kube_namespace_annotations`
      </td>

      <td>
        Kubernetes annotations converted to Prometheus labels.
      </td>
    </tr>

    <tr>
      <td>
        `kube_namespace_created`
      </td>

      <td>
        \[STABLE] Unix creation timestamp
      </td>
    </tr>

    <tr>
      <td>
        `kube_namespace_labels`
      </td>

      <td>
        \[STABLE] Kubernetes labels converted to Prometheus labels.
      </td>
    </tr>

    <tr>
      <td>
        `kube_namespace_status_condition`
      </td>

      <td>
        The condition of a namespace.
      </td>
    </tr>

    <tr>
      <td>
        `kube_namespace_status_phase`
      </td>

      <td>
        \[STABLE] kubernetes namespace status phase.
      </td>
    </tr>

    <tr>
      <td>
        `kube_node_annotations`
      </td>

      <td>
        Kubernetes annotations converted to Prometheus labels.
      </td>
    </tr>

    <tr>
      <td>
        `kube_node_created`
      </td>

      <td>
        \[STABLE] Unix creation timestamp
      </td>
    </tr>

    <tr>
      <td>
        `kube_node_info`
      </td>

      <td>
        \[STABLE] Information about a cluster node.
      </td>
    </tr>

    <tr>
      <td>
        `kube_node_labels`
      </td>

      <td>
        \[STABLE] Kubernetes labels converted to Prometheus labels.
      </td>
    </tr>

    <tr>
      <td>
        `kube_node_role`
      </td>

      <td>
        The role of a cluster node.
      </td>
    </tr>

    <tr>
      <td>
        `kube_node_spec_taint`
      </td>

      <td>
        \[STABLE] The taint of a cluster node.
      </td>
    </tr>

    <tr>
      <td>
        `kube_node_spec_unschedulable`
      </td>

      <td>
        \[STABLE] Whether a node can schedule new pods.
      </td>
    </tr>

    <tr>
      <td>
        `kube_node_status_allocatable`
      </td>

      <td>
        \[STABLE] The allocatable for different resources of a node that are available for scheduling.
      </td>
    </tr>

    <tr>
      <td>
        `kube_node_status_capacity`
      </td>

      <td>
        \[STABLE] The capacity for different resources of a node.
      </td>
    </tr>

    <tr>
      <td>
        `kube_node_status_condition`
      </td>

      <td>
        \[STABLE] The condition of a cluster node.
      </td>
    </tr>

    <tr>
      <td>
        `kube_pod_completion_time`
      </td>

      <td>
        \[STABLE] Completion time in unix timestamp for a pod.
      </td>
    </tr>

    <tr>
      <td>
        `kube_pod_container_info`
      </td>

      <td>
        \[STABLE] Information about a container in a pod.
      </td>
    </tr>

    <tr>
      <td>
        `kube_pod_container_resource_limits`
      </td>

      <td>
        The number of requested limit resource by a container. It is recommended to use the kube_pod_resource_limits metric exposed by kube-scheduler instead, as it is more precise.
      </td>
    </tr>

    <tr>
      <td>
        `kube_pod_container_resource_requests`
      </td>

      <td>
        The number of requested request resource by a container. It is recommended to use the kube_pod_resource_requests metric exposed by kube-scheduler instead, as it is more precise.
      </td>
    </tr>

    <tr>
      <td>
        `kube_pod_container_state_started`
      </td>

      <td>
        \[STABLE] Start time in unix timestamp for a pod container.
      </td>
    </tr>

    <tr>
      <td>
        `kube_pod_container_status_last_terminated_ exitcode`
      </td>

      <td>
        Describes the exit code for the last container in terminated state.
      </td>
    </tr>

    <tr>
      <td>
        `kube_pod_container_status_last_terminated_ reason`
      </td>

      <td>
        Describes the last reason the container was in terminated state.
      </td>
    </tr>

    <tr>
      <td>
        `kube_pod_container_status_ready`
      </td>

      <td>
        \[STABLE] Describes whether the containers readiness check succeeded.
      </td>
    </tr>

    <tr>
      <td>
        `kube_pod_container_status_restarts_total`
      </td>

      <td>
        \[STABLE] The number of container restarts per container.
      </td>
    </tr>

    <tr>
      <td>
        `kube_pod_container_status_running`
      </td>

      <td>
        \[STABLE] Describes whether the container is currently in running state.
      </td>
    </tr>

    <tr>
      <td>
        `kube_pod_container_status_terminated`
      </td>

      <td>
        \[STABLE] Describes whether the container is currently in terminated state.
      </td>
    </tr>

    <tr>
      <td>
        `kube_pod_container_status_terminated_ reason`
      </td>

      <td>
        Describes the reason the container is currently in terminated state.
      </td>
    </tr>

    <tr>
      <td>
        `kube_pod_container_status_waiting`
      </td>

      <td>
        \[STABLE] Describes whether the container is currently in waiting state.
      </td>
    </tr>

    <tr>
      <td>
        `kube_pod_container_status_waiting_ reason`
      </td>

      <td>
        \[STABLE] Describes the reason the container is currently in waiting state.
      </td>
    </tr>

    <tr>
      <td>
        `kube_pod_created`
      </td>

      <td>
        \[STABLE] Unix creation timestamp
      </td>
    </tr>

    <tr>
      <td>
        `kube_pod_deletion_timestamp`
      </td>

      <td>
        Unix deletion timestamp
      </td>
    </tr>

    <tr>
      <td>
        `kube_pod_info`
      </td>

      <td>
        \[STABLE] Information about pod.
      </td>
    </tr>

    <tr>
      <td>
        `kube_pod_init_container_info`
      </td>

      <td>
        \[STABLE] Information about an init container in a pod.
      </td>
    </tr>

    <tr>
      <td>
        `kube_pod_init_container_resource_limits`
      </td>

      <td>
        The number of requested limit resource by an init container.
      </td>
    </tr>

    <tr>
      <td>
        `kube_pod_init_container_resource_requests`
      </td>

      <td>
        The number of requested request resource by an init container.
      </td>
    </tr>

    <tr>
      <td>
        `kube_pod_init_container_status_last_ terminated_reason`
      </td>

      <td>
        Describes the last reason the init container was in terminated state.
      </td>
    </tr>

    <tr>
      <td>
        `kube_pod_init_container_status_ready`
      </td>

      <td>
        \[STABLE] Describes whether the init containers readiness check succeeded.
      </td>
    </tr>

    <tr>
      <td>
        `kube_pod_init_container_status_restarts_ total`
      </td>

      <td>
        \[STABLE] The number of restarts for the init container.
      </td>
    </tr>

    <tr>
      <td>
        `kube_pod_init_container_status_running`
      </td>

      <td>
        \[STABLE] Describes whether the init container is currently in running state.
      </td>
    </tr>

    <tr>
      <td>
        `kube_pod_init_container_status_terminated`
      </td>

      <td>
        \[STABLE] Describes whether the init container is currently in terminated state.
      </td>
    </tr>

    <tr>
      <td>
        `kube_pod_init_container_status_terminated_ reason`
      </td>

      <td>
        Describes the reason the init container is currently in terminated state.
      </td>
    </tr>

    <tr>
      <td>
        `kube_pod_init_container_status_waiting`
      </td>

      <td>
        \[STABLE] Describes whether the init container is currently in waiting state.
      </td>
    </tr>

    <tr>
      <td>
        `kube_pod_init_container_status_waiting_ reason`
      </td>

      <td>
        Describes the reason the init container is currently in waiting state.
      </td>
    </tr>

    <tr>
      <td>
        `kube_pod_ips`
      </td>

      <td>
        Pod IP addresses
      </td>
    </tr>

    <tr>
      <td>
        `kube_pod_owner`
      </td>

      <td>
        \[STABLE] Information about the Pod’s owner.
      </td>
    </tr>

    <tr>
      <td>
        `kube_pod_restart_policy`
      </td>

      <td>
        \[STABLE] Describes the restart policy in use by this pod.
      </td>
    </tr>

    <tr>
      <td>
        `kube_pod_service_account`
      </td>

      <td>
        The service account for a pod.
      </td>
    </tr>

    <tr>
      <td>
        `kube_pod_start_time`
      </td>

      <td>
        \[STABLE] Start time in unix timestamp for a pod.
      </td>
    </tr>

    <tr>
      <td>
        `kube_pod_status_container_ready_time`
      </td>

      <td>
        Readiness achieved time in unix timestamp for a pod containers.
      </td>
    </tr>

    <tr>
      <td>
        `kube_pod_status_initialized_time`
      </td>

      <td>
        Initialized time in unix timestamp for a pod.
      </td>
    </tr>

    <tr>
      <td>
        `kube_pod_status_phase`
      </td>

      <td>
        \[STABLE] The pods current phase.
      </td>
    </tr>

    <tr>
      <td>
        `kube_pod_status_qos_class`
      </td>

      <td>
        The pods current qosClass.
      </td>
    </tr>

    <tr>
      <td>
        `kube_pod_status_ready`
      </td>

      <td>
        \[STABLE] Describes whether the pod is ready to serve requests.
      </td>
    </tr>

    <tr>
      <td>
        `kube_pod_status_ready_time`
      </td>

      <td>
        Readiness achieved time in unix timestamp for a pod.
      </td>
    </tr>

    <tr>
      <td>
        `kube_pod_status_reason`
      </td>

      <td>
        The pod status reasons
      </td>
    </tr>

    <tr>
      <td>
        `kube_pod_status_scheduled`
      </td>

      <td>
        \[STABLE] Describes the status of the scheduling process for the pod.
      </td>
    </tr>

    <tr>
      <td>
        `kube_pod_status_scheduled_time`
      </td>

      <td>
        \[STABLE] Unix timestamp when pod moved into scheduled status
      </td>
    </tr>

    <tr>
      <td>
        `kube_pod_tolerations`
      </td>

      <td>
        Information about the pod tolerations
      </td>
    </tr>

    <tr>
      <td>
        `kube_poddisruptionbudget_annotations`
      </td>

      <td>
        Kubernetes annotations converted to Prometheus labels.
      </td>
    </tr>

    <tr>
      <td>
        `kube_poddisruptionbudget_created`
      </td>

      <td>
        \[STABLE] Unix creation timestamp
      </td>
    </tr>

    <tr>
      <td>
        `kube_poddisruptionbudget_labels`
      </td>

      <td>
        Kubernetes labels converted to Prometheus labels.
      </td>
    </tr>

    <tr>
      <td>
        `kube_poddisruptionbudget_status_current_ healthy`
      </td>

      <td>
        \[STABLE] Current number of healthy pods
      </td>
    </tr>

    <tr>
      <td>
        `kube_poddisruptionbudget_status_desired_ healthy`
      </td>

      <td>
        \[STABLE] Minimum desired number of healthy pods
      </td>
    </tr>

    <tr>
      <td>
        `kube_poddisruptionbudget_status_expected_ pods`
      </td>

      <td>
        \[STABLE] Total number of pods counted by this disruption budget
      </td>
    </tr>

    <tr>
      <td>
        `kube_poddisruptionbudget_status_observed_ generation`
      </td>

      <td>
        \[STABLE] Most recent generation observed when updating this PDB status
      </td>
    </tr>

    <tr>
      <td>
        `kube_poddisruptionbudget_status_pod_ disruptions_allowed`
      </td>

      <td>
        \[STABLE] Number of pod disruptions that are currently allowed
      </td>
    </tr>

    <tr>
      <td>
        `kube_replicaset_created`
      </td>

      <td>
        \[STABLE] Unix creation timestamp
      </td>
    </tr>

    <tr>
      <td>
        `kube_replicaset_metadata_generation`
      </td>

      <td>
        \[STABLE] Sequence number representing a specific generation of the desired state.
      </td>
    </tr>

    <tr>
      <td>
        `kube_replicaset_owner`
      </td>

      <td>
        \[STABLE] Information about the ReplicaSet’s owner.
      </td>
    </tr>

    <tr>
      <td>
        `kube_replicaset_spec_replicas`
      </td>

      <td>
        \[STABLE] Number of desired pods for a ReplicaSet.
      </td>
    </tr>

    <tr>
      <td>
        `kube_replicaset_status_fully_labeled_ replicas`
      </td>

      <td>
        \[STABLE] The number of fully labeled replicas per ReplicaSet.
      </td>
    </tr>

    <tr>
      <td>
        `kube_replicaset_status_observed_generation`
      </td>

      <td>
        \[STABLE] The generation observed by the ReplicaSet controller.
      </td>
    </tr>

    <tr>
      <td>
        `kube_replicaset_status_ready_replicas`
      </td>

      <td>
        \[STABLE] The number of ready replicas per ReplicaSet.
      </td>
    </tr>

    <tr>
      <td>
        `kube_replicaset_status_replicas`
      </td>

      <td>
        \[STABLE] The number of replicas per ReplicaSet.
      </td>
    </tr>

    <tr>
      <td>
        `kube_replicationcontroller_created`
      </td>

      <td>
        \[STABLE] Unix creation timestamp
      </td>
    </tr>

    <tr>
      <td>
        `kube_replicationcontroller_metadata_ generation`
      </td>

      <td>
        \[STABLE] Sequence number representing a specific generation of the desired state.
      </td>
    </tr>

    <tr>
      <td>
        `kube_replicationcontroller_owner`
      </td>

      <td>
        Information about the ReplicationController’s owner.
      </td>
    </tr>

    <tr>
      <td>
        `kube_replicationcontroller_spec_replicas`
      </td>

      <td>
        \[STABLE] Number of desired pods for a ReplicationController.
      </td>
    </tr>

    <tr>
      <td>
        `kube_replicationcontroller_status_ available_replicas`
      </td>

      <td>
        \[STABLE] The number of available replicas per ReplicationController.
      </td>
    </tr>

    <tr>
      <td>
        `kube_replicationcontroller_status_fully_ labeled_replicas`
      </td>

      <td>
        \[STABLE] The number of fully labeled replicas per ReplicationController.
      </td>
    </tr>

    <tr>
      <td>
        `kube_replicationcontroller_status_ observed_generation`
      </td>

      <td>
        \[STABLE] The generation observed by the ReplicationController controller.
      </td>
    </tr>

    <tr>
      <td>
        `kube_replicationcontroller_status_ready_ replicas`
      </td>

      <td>
        \[STABLE] The number of ready replicas per ReplicationController.
      </td>
    </tr>

    <tr>
      <td>
        `kube_replicationcontroller_status_replicas`
      </td>

      <td>
        \[STABLE] The number of replicas per ReplicationController.
      </td>
    </tr>

    <tr>
      <td>
        `kube_resourcequota`
      </td>

      <td>
        \[STABLE] Information about resource quota.
      </td>
    </tr>

    <tr>
      <td>
        `kube_resourcequota_created`
      </td>

      <td>
        \[STABLE] Unix creation timestamp
      </td>
    </tr>

    <tr>
      <td>
        `kube_secret_created`
      </td>

      <td>
        \[STABLE] Unix creation timestamp
      </td>
    </tr>

    <tr>
      <td>
        `kube_secret_info`
      </td>

      <td>
        \[STABLE] Information about secret.
      </td>
    </tr>

    <tr>
      <td>
        `kube_secret_metadata_resource_version`
      </td>

      <td>
        Resource version representing a specific version of secret.
      </td>
    </tr>

    <tr>
      <td>
        `kube_secret_type`
      </td>

      <td>
        \[STABLE] Type about secret.
      </td>
    </tr>

    <tr>
      <td>
        `kube_service_created`
      </td>

      <td>
        \[STABLE] Unix creation timestamp
      </td>
    </tr>

    <tr>
      <td>
        `kube_service_info`
      </td>

      <td>
        \[STABLE] Information about service.
      </td>
    </tr>

    <tr>
      <td>
        `kube_service_spec_type`
      </td>

      <td>
        \[STABLE] Type about service.
      </td>
    </tr>

    <tr>
      <td>
        `kube_statefulset_created`
      </td>

      <td>
        \[STABLE] Unix creation timestamp
      </td>
    </tr>

    <tr>
      <td>
        `kube_statefulset_metadata_generation`
      </td>

      <td>
        \[STABLE] Sequence number representing a specific generation of the desired state for the StatefulSet.
      </td>
    </tr>

    <tr>
      <td>
        `kube_statefulset_persistentvolumeclaim_ retention_policy`
      </td>

      <td>
        Count of retention policy for StatefulSet template PVCs
      </td>
    </tr>

    <tr>
      <td>
        `kube_statefulset_replicas`
      </td>

      <td>
        \[STABLE] Number of desired pods for a StatefulSet.
      </td>
    </tr>

    <tr>
      <td>
        `kube_statefulset_status_current_revision`
      </td>

      <td>
        \[STABLE] Indicates the version of the StatefulSet used to generate Pods in the sequence [0,currentReplicas).
      </td>
    </tr>

    <tr>
      <td>
        `kube_statefulset_status_observed_ generation`
      </td>

      <td>
        \[STABLE] The generation observed by the StatefulSet controller.
      </td>
    </tr>

    <tr>
      <td>
        `kube_statefulset_status_replicas`
      </td>

      <td>
        \[STABLE] The number of replicas per StatefulSet.
      </td>
    </tr>

    <tr>
      <td>
        `kube_statefulset_status_replicas_available`
      </td>

      <td>
        The number of available replicas per StatefulSet.
      </td>
    </tr>

    <tr>
      <td>
        `kube_statefulset_status_replicas_current`
      </td>

      <td>
        \[STABLE] The number of current replicas per StatefulSet.
      </td>
    </tr>

    <tr>
      <td>
        `kube_statefulset_status_replicas_ready`
      </td>

      <td>
        \[STABLE] The number of ready replicas per StatefulSet.
      </td>
    </tr>

    <tr>
      <td>
        `kube_statefulset_status_replicas_updated`
      </td>

      <td>
        \[STABLE] The number of updated replicas per StatefulSet.
      </td>
    </tr>

    <tr>
      <td>
        `kube_statefulset_status_update_revision`
      </td>

      <td>
        \[STABLE] Indicates the version of the StatefulSet used to generate Pods in the sequence [replicas-updatedReplicas,replicas)
      </td>
    </tr>

    <tr>
      <td>
        `kube_storageclass_created`
      </td>

      <td>
        \[STABLE] Unix creation timestamp
      </td>
    </tr>

    <tr>
      <td>
        `kube_storageclass_info`
      </td>

      <td>
        \[STABLE] Information about storageclass.
      </td>
    </tr>

    <tr>
      <td>
        `kube_validatingwebhookconfiguration_created`
      </td>

      <td>
        Unix creation timestamp.
      </td>
    </tr>

    <tr>
      <td>
        `kube_validatingwebhookconfiguration_info`
      </td>

      <td>
        Information about the ValidatingWebhookConfiguration.
      </td>
    </tr>

    <tr>
      <td>
        `kube_validatingwebhookconfiguration_ metadata_resource_version`
      </td>

      <td>
        Resource version representing a specific version of the ValidatingWebhookConfiguration.
      </td>
    </tr>

    <tr>
      <td>
        `kube_validatingwebhookconfiguration_ webhook_clientconfig_service`
      </td>

      <td>
        Service used by the apiserver to connect to a validating webhook.
      </td>
    </tr>
  </tbody>
</table>

### G.1.15  Parameters For Metrics

Metrics have the parameters indicated by the left column in the following example:

**Example**

```bash
[basecm11->monitoring->measurable[CPUUser]]% show
Parameter               Value
----------------------- ----------------------------
Class                   CPU
Consolidator            default (ProcStat)
Cumulative              yes
Description             CPU time spent in user mode
Disabled                no (ProcStat)
Gap                     0 (ProcStat)
Maximal age             0s (ProcStat)
Maximal samples         4,096 (ProcStat)
Maximum                 0
Minimum                 0
Name                    CPUUser
Parameter
Producer                ProcStat
Revision
Type                    Metric
Unit                    Jiffies/s
```

If the value is inherited from the producer, then it is shown in parentheses next to the value. An inherited value can be overwritten by setting it directly for the parameter of a measurable.

The meanings of the parameters are:

`Class`: A choice assigned to a metric. It can be an internal type, or it can be a standalone class type. A slash (`/`) is used to separate class levels. A partial list of the class values is:

* `CPU`: CPU-related

* `Disk`: Disk-related

* `Disk/Smart`: SMART Disk-related

* `Fabric`: Fabric-related

* `GPU`: GPU-related

* `Internal`: An internal metric

* `Job`: Job metric

* `License`: License-related

* `Memory`: Memory-related

* `Network`: Network-related

* `OS`: Operating-system-related

* `Process`: Process-related

* `Prometheus`: Prometheus-related

* `Total`: Total cluster-wide-related

* `Workload`: Workload-related

* `Environmental`: Environmental-related

`Consolidator`: This is described in detail in sections [10.4.3](/admin/monitoring#1043-monitoring-configuration-consolidators) and [10.5.2](/admin/monitoring#1052-the-consolidator-submode)

`Cumulative`: If set to `no`, then the raw value is treated as not cumulative (for example, `CoresUp`), and the raw value is presented as the metric value.

If set to `yes`, then the metric is treated as being cumulative, which means that a rate (per second) value is presented.

More explicitly: When set to `yes`, it means that the raw sample used to calculate the metric is expected to be cumulative, like, for example, the bytes-received counter for an Ethernet interface. This in turn means that the metric is calculated from the raw value by taking the difference in raw sample measurement values, and dividing it by the time period over which the raw values are sampled. Thus, for example:

* The bytes-received raw measurements, which accumulate as the packets are received, and are in bytes, and have `Cumulative` set to yes, and then have a corresponding metric, `BytesRecv`, with a value in bytes/second.

* The system uptime raw measurements, which accumulate at the rate of 1 second per second, and are in seconds, have `Cumulative` set to yes, and have a corresponding metric, `Uptime`, with a value that uses no units. Ideally, the metric has a value of 1, but in practice the measured value varies a little due to jitter.

`Description`: Description of the raw measurement used by the metric. Empty by default.

`Disabled`: If set to `no` (default) then the metric runs.

`Gap`: The number of samples that are allowed to be missed before a value of `NaN` is set for the value of the metric.

`Maximal age`: the maximum age of RLE samples that are kept. If `Maximal age` is set to `0` then the sample age is not considered. Units can be `w`, `d`, `h`, `m`, `s` (weeks, days, hours, minutes, seconds), with `s` as the default.

`Maximal samples`: the maximum number of RLE samples that are kept. If `Maximal samples` is set to `0` then the number of samples is not considered.

`Maximum`: the value that the y-axis maximum takes in graphs plotted in Base View by default. If the maximum of the y-values is more than the default y-axis maximum value, then the maximum of the y-values becomes the y-axis maximum.

`Minimum`: the value that the y-axis minimum takes in graphs plotted in Base View by default. If the minimum of the y-values is less than the default y-axis minimum value, then the minimum of the y-values becomes the y-axis minimum.

**NOTE:** To clarify the `Maximum` and `Minimum` concepts, a case can be considered where `minimum`=0, `maximum`=3 are set. If a data point with a y-value of 2 is plotted on a graph, then the y-axis spans the range from 0 to 3 by default. However

* if the data point has a y-value of 4 instead, then it means the y-axis maximum of 3 is re-sized from its default of 3 to the value of 4, so that the y-axis now spans from 0 to 4.

* if the data point has a y-value of -1 instead, then it means the y-axis minimum of 0 is re-sized from its default of 0 to the value of -1, so that the y-axis now spans from -1 to 3.

`Name`: The name given to the metric.

`Parameter`: Parameter used for this metric. For example, `eth0` with the metric `BytesRecv`

`Producer`: The data producer that produces the metric

`Revision`: User-definable revision number for the object

`Type`: This can be one of `metric`, `healthcheck`, or `enummetric`

`Unit`: A unit for the metric. For example: B/s (bytes/second) for `BytesRecv` metric, or unit-less for the `Uptime` metric. A percent is indicated with %

## G.2  Health Checks And Their Parameters

A list of health checks can be viewed, for example, for the head node, using `cmsh` as follows (section [10.5.3.1](/admin/monitoring#10531-the-measurable-submode-list-show-and-get)):

```bash
[basecm11 ~]# cmsh -c "monitoring measurable; list healthcheck"
```

The health checks listed in this section are classed into 4 kinds:

1. Regular health checks (section [10.2.4](/admin/monitoring#1024-health-check)) are listed and described in section [G.2.1](#g21-regular-health-checks).

2. GPU health checks (section [G.2.2](#g22-gpu-health-checks))

3. Redfish health checks (section [G.2.3](#g23-redfish-health-checks))

4. NetQ health checks (section [G.2.4](#g24-netq-health-checks))

### G.2.1  Regular Health Checks

<table>
  <thead>
    <tr>
      <th colspan="2">*Table  [G.2.1](#g21-regular-health-checks): List Of Health Checks*</th>
    </tr>
  </thead>

  <tbody>
    <tr>
      <td>
        **Name**
      </td>

      <td>
        **Query**

         (script response is 

        `PASS`

        /

        `FAIL`

        )
      </td>
    </tr>

    <tr>
      <td>
        `ManagedServicesOk`

        \*
      </td>

      <td>
        Are CMDaemon-monitored services all OK? If the response is 

        `FAIL`

        , then at least one of the services being monitored is failing. The 

        `latesthealthdata -v`

         command (section 

        [10.6.3](/admin/monitoring#1063-the-latest-data-and-counter-valuesthe-latestdata-and-latestmetriccounters-commands)

        ) should show which one(s). After correcting the problem with the service, a reset of the service is normally carried out (section 

        [3.15](/admin/configuring#315-managing-and-configuring-services)

        , 

        [here](/admin/configuring#31526-monitoring-a-service-with-cmsh-and-base-view)

        ). There is also a related 

        `ManagedServicesOk`

         metric 

        [here](#g11-regular-metrics)

        .
      </td>
    </tr>

    <tr>
      <td>
        `Mon::Storage`
      </td>

      <td>
        Is space available for the monitoring system metrics (section 

        [G.1.4](#g14-monitoring-system-metrics)

        )?
      </td>
    </tr>

    <tr>
      <td>
        `chrootprocess`
      </td>

      <td>
        Are there daemon processes running using chroot in software images? Here: yes = 

        `FAIL`

        . On failure, kill cron daemon processes running in the software images.
      </td>
    </tr>

    <tr>
      <td>
        `cm-chroot-sw-img`
      </td>

      <td>
        Are there dangling mounts left behind of images created by cm-chroot-sw-img? Here: yes = 

        `FAIL`

        . The cluster administrator is expected to inspect and unmount the dangling mounts.
      </td>
    </tr>

    <tr>
      <td>
        `cmha-status`
      </td>

      <td>
        Are both head nodes up and running?
      </td>
    </tr>

    <tr>
      <td>
        `cmsh `

        \*
      </td>

      <td>
        Is 

        `cmsh`

         available?
      </td>
    </tr>

    <tr>
      <td>
        `cuda-dcgm`
      </td>

      <td>
        Is 

        `cuda-dcgm `

         available?
      </td>
    </tr>

    <tr>
      <td>
        `defaultgateway`
      </td>

      <td>
        Is there a default gateway available?
      </td>
    </tr>

    <tr>
      <td>
        `dellnss`
      </td>

      <td>
        If running, is the Dell NFS Storage Solution healthy?
      </td>
    </tr>

    <tr>
      <td rowspan="2">
        `diskspace`
      </td>

      <td>
        Is there less local disk space available to non-root users than any of the space parameters specified? *The space parameters can be specified as MB, GB, TB, or as percentages with %. The default severity of notices from this check is 10, when one space parameter is used. For more than one space parameter, the severity decreases by 10 for each space parameter, sequentially, down to 10 for the last space parameter. By default a space parameter of 10% is assumed. Another, also optional, non-space parameter, the filesystem mount point parameter, can be specified after the last space parameter to track filesystem space, instead of disk space. A metric-based alternative to tracking filesystem space changes is to use the built-in metric 

        `freespace`

         (

        [here](#g11-regular-metrics)

        ) instead.*
      </td>
    </tr>

    <tr>
      <td>
        Examples: 

        \


        • diskspace 10% less than 10% space = FAIL, severity 10

        \


        • diskspace 10% 20% 30% less than 30% space = FAIL, with severity levels as indicated:

        \


        • diskspace 10GB 20GB less than 20GB space = FAIL, severity 10 less than 10GB space = FAIL, severity 20

        \


        • diskspace 10% 20% /var For the filesystem /var: less than 20% space = FAIL, severity 10 less than 10% space = FAIL, severity 20
      </td>
    </tr>

    <tr>
      <td>
        `dmesg`
      </td>

      <td>
        Is 

        `dmesg`

         output OK? *Regexes to parse the output can be constructed in the configuration file at 

        `/cm/local/apps/cmd/scripts/healthchecks/configfiles/dmesg.py`

        \*
      </td>
    </tr>

    <tr>
      <td>
        `docker`
      </td>

      <td>
        Is Docker running OK? Checks for Docker server availability and corruption, dead containers, proper endpoints
      </td>
    </tr>

    <tr>
      <td>
        `dockerregistry`
      </td>

      <td>
        Is the Docker registry running OK? Checks registry endpoint and registry availability
      </td>
    </tr>

    <tr>
      <td>
        `exports`
      </td>

      <td>
        Are all filesystems as defined by the cluster management system exported?
      </td>
    </tr>

    <tr>
      <td>
        `etcd`
      </td>

      <td>
        Are the core etcd processes of Kubernetes running OK? Checks endpoints and interfaces
      </td>
    </tr>

    <tr>
      <td>
        `failedprejob`
      </td>

      <td>
        Are there failed prejob health checks (section 

        [7.8.2](/admin/workloadmanagement#782-setting-up-a-prejob-or-postjob-check)

        )? Here: yes = FAIL. By default, the job ID is saved under 

        `/cm/local/apps/`

        \<

        *scheduler*

        \>

        `/var/`

        : 

        \


        • On FAIL, in failedprejobs.

        \


        • On PASS, in allprejobs The maximum number of IDs stored is 1000 by default. The maximum period for which the IDs are stored is 30 days by default. Both these maxima can be set with the 

        `failedprejob`

         health check script.
      </td>
    </tr>

    <tr>
      <td>
        `hpraid`
      </td>

      <td>
        Are the HP Smart Array controllers OK?
      </td>
    </tr>

    <tr>
      <td>
        `ib`
      </td>

      <td>
        Is the InfiniBand Host Channel Adapter working properly? *A configuration file for this health check is at 

        `/cm/local/apps/cmd/scripts/healthchecks/configfiles/ib.py`

        \*
      </td>
    </tr>

    <tr>
      <td>
        `interfaces`
      </td>

      <td>
        Are the interfaces up and running at full speed?
      </td>
    </tr>

    <tr>
      <td>
        `ipmihealth`
      </td>

      <td>
        Is the BMC (IPMI or iLO) health OK? Uses the script 

        `sample_ipmi`

        .
      </td>
    </tr>

    <tr>
      <td>
        `kubernetescertsexpiration`
      </td>

      <td>
        Are Kubernetes certificates valid for at least the next 30 days?
      </td>
    </tr>

    <tr>
      <td>
        `kuberneteschildnode`
      </td>

      <td>
        Are all Kubernetes child nodes up?
      </td>
    </tr>

    <tr>
      <td>
        `kubernetescomponentsstatus`
      </td>

      <td>
        Are all expected agents and services up and running for active nodes?
      </td>
    </tr>

    <tr>
      <td>
        `kubernetesnodesstatus`
      </td>

      <td>
        Is the status for all Kubernetes nodes OK?
      </td>
    </tr>

    <tr>
      <td>
        `kubernetespodsstatus`
      </td>

      <td>
        Is the status for all pods OK?
      </td>
    </tr>

    <tr>
      <td>
        `ldap`
      </td>

      <td>
        Can the ID of the user be looked up with LDAP?
      </td>
    </tr>

    <tr>
      <td>
        `lustre`
      </td>

      <td>
        Is the Lustre filesystem running OK?
      </td>
    </tr>

    <tr>
      <td>
        `megaraid`
      </td>

      <td>
        Are the MegaRAID controllers OK? *Either the proprietary MegaCLI software, or its successor, the proprietary StorCLI software is needed for this health check. The MegaCLI software was originally provided by LSI Logic, but LSI is now part of Broadcom. Both the MegaCLI software and the StorCLI software are now available from the Broadcom website (

        [http://www.broadcom.com](http://www.broadcom.com)

        ). For BCM 10 and onwards, the healthcheck first checks for StorCLI, and then for MegaCLI, and uses the first binary that is detected. For BCM versions prior to version 10, the healthcheck first checks for MegaCLI, and then StorCLI, and uses the first binary that is detected.*
      </td>
    </tr>

    <tr>
      <td>
        `mounts`
      </td>

      <td>
        Are all mounts defined in the fstab OK?
      </td>
    </tr>

    <tr>
      <td>
        `mysql`
      </td>

      <td>
        Is the status and configuration of MySQL correct?
      </td>
    </tr>

    <tr>
      <td rowspan="2">
        `node-hardware-profile`
      </td>

      <td>
        Is the specified node’s hardware configuration during health check use unchanged?
      </td>
    </tr>

    <tr>
      <td>
        *The options to this script are described using the “-h” help option. Before this script is used for health checks, the specified hardware profile is usually first saved with the 

        `-s`

         option. Eg: “

        `node-hardware-profile -n node001 -s hardwarenode001`

        ”*
      </td>
    </tr>

    <tr>
      <td>
        `ntp`

        \*
      </td>

      <td>
        Is NTP synchronization happening?
      </td>
    </tr>

    <tr>
      <td>
        `oomkiller`
      </td>

      <td>
        Has the oomkiller process run? Yes=FAIL. The 

        `oomkiller`

         health check checks if the oomkiller process has run. The configuration file 

        `/cm/local/apps/cmd/scripts/healthchecks/configfiles/oomkiller.conf`

         for the 

        `oomkiller`

         health check can be configured to reset the response to PASS after one FAIL is logged, until the next oomkiller process runs. The processes killed by the oomkiller process are logged in 

        `/var/spool/cmd/save-oomkilleraction`

        . 

        *A consideration of the causes and consequences of the killed processes is strongly recommended. A reset of the node is generally recommended.*
      </td>
    </tr>

    <tr>
      <td>
        `opalinkhealth`
      </td>

      <td>
        Are the quality and the integrity of the Intel OPA HFI link OK?
      </td>
    </tr>

    <tr>
      <td>
        `Overall_Health`

        `:<`

        *sda*

        `>`
      </td>

      <td>
        Overall disk health status (SMART response) for specified device, in this case 

        `<`

        *sda*

        `>`

        .
      </td>
    </tr>

    <tr>
      <td>
        `SMART_Health`

        `:<`

        *sda*

        `>`
      </td>

      <td>
        Overall SMART health as reported by the exit code, for a specified device, in this case 

        `<`

        *sda*

        `>`

        .
      </td>
    </tr>

    <tr>
      <td rowspan="2">
        `rogueprocess`
      </td>

      <td>
        Are the processes that are running legitimate (ie, not ’rogue’)? Besides the 

        `FAIL`

        /

        `PASS`

        /

        `UNKNOWN`

         response to CMDaemon, also returns a list of rogue process IDs to file descriptor 3 (InfoMessages), which the 

        `killprocess`

         action (

        [here](#g41-actions)

        ) can then go ahead and kill. Illegitimate processes are processes that should not be running on the node. An illegitimate process is at least one of the following, by default: 

        \


        • not part of the workload manager service or its jobs

        \


        • not a root- or system-owned process

        \


        • in the state Z, T, W, or X. States are described in the ps man pages in the section on “PROCESS STATE CODES”
      </td>
    </tr>

    <tr>
      <td>
        Rogue process criteria can be configured in the file 

        `/cm/local/apps/cmd/scripts/healthchecks/configfiles/rogueprocess.py`

         within the software image. To implement a changed criteria configuration, the software image used by systems on which the health check is run should be updated (section 

        [5.6](/admin/provisioning#56-updating-running-nodes)

        ). For example, using: 

        `cmsh -c "device; imageupdate -c default -w"`

         for the 

        `default`

         category of nodes.
      </td>
    </tr>

    <tr>
      <td>
        `schedulers`
      </td>

      <td>
        Are the queue instances of all schedulers on a node healthy ?
      </td>
    </tr>

    <tr>
      <td>
        `smart`
      </td>

      <td>
        Is the SMART response healthy? The severities can be configured in the file 

        `/cm/local/apps/cmd/scripts/healthchecks/configfiles/smart.conf`

        . By default, if a drive does not support the SMART commands and results in a "Smart command failed" info message for that drive, then the healthcheck is configured to give a PASS response. This is because the mere fact that the drive is a non-SMART drive should not be a reason to conclude that the drive is unhealthy. The info messages can be suppressed by setting an allowed list of the disks to be checked within 

        `/cm/local/apps/cmd/scripts/healthchecks/smart`

        .
      </td>
    </tr>

    <tr>
      <td>
        `ssh2node`
      </td>

      <td>
        Is passwordless ssh root login, from head to a node that is up, working? Some details of its behavior are: 

        \


        • The health check fails on the head node if root ssh login to the head node has been disabled.

        \


        • The health check fails if ssh certificate-based access, to a non-head node that is in the UP state, fails, even if login (key-based) access is still available. The UP state is determined by whether CMDaemon is running on that node.

        \


        • If the regular node is in a DOWN state—which could be due to CMDaemon being down, or the node having been powered off gracefully, or the node suffering a sudden power failure—then the health check responds with a PASS. The idea here is to check key or certificate access, and decouple it from the node state.
      </td>
    </tr>

    <tr>
      <td>
        `swraid`
      </td>

      <td>
        Are the software RAID arrays healthy?
      </td>
    </tr>

    <tr>
      <td>
        `testhealthcheck`
      </td>

      <td>
        *A health check script example for creating scripts, or setting a mix of PASS/FAIL/UNKNOWN responses. The source includes examples of environment variables that can be used, as well as configuration suggestions.*
      </td>
    </tr>
  </tbody>
</table>

\* built-ins, not standalone scripts.

If sampling from a head node, a standalone script is in directory:

`/cm/local/apps/cmd/scripts/healthchecks/`

If sampling from a regular node, a standalone script is in directory:

`/cm/images/default-image/cm/local/apps/cmd/scripts/healthchecks/`

### G.2.2  GPU Health Checks

The data producer (section [10.2.10](/admin/monitoring#10210-data-producer)) for the GPU health checks of this section is `GPUSampler`.

The NVIDIA GPU health checks of this section, as the `GPUSampler` data producer name suggests, is about gathering the sampled GPU data from the devices themselves.

The device parameter for the GPU health checks in this section, unless otherwise noted, requires as a parameter the device slot number that the GPU uses. The parameter takes the form `gpu0`, `gpu1`, and so on. It is appended to the metric with a colon character. For example, the `gpu_health_inforom` health check, if used with `gpu1`, is specified as:

**Example**

```bash
gpu_health_inforom:gpu1
```

Available GPU health checks for V100 and A100 GPUs are listed in table [G.2.2](#g22-gpu-health-checks). The available GPU metrics are displayed in table [G.1.6](#g16-gpu-metrics).

A DCGM `WARN` result does not add a fourth state to these health checks — the value reported for a `gpu_health_*` check remains one of `PASS`, `UNKNOWN`, or `FAIL`. By default, a DCGM `WARN` is reported as `PASS`, with the reason recorded in the check’s `Info` field rather than changing its `Value`:

**Example**

```bash
[basecm11->device[dgx2-1]]% latesthealthdata -m gpu_health_overall:gpu0
Measurable          Parameter  Type  Value  Age     State  Info
------------------  ---------  ----  -----  ------  -----  -------------------
gpu_health_overall  gpu0       GPU   PASS   1m 21s         DCGM health warning
```

This default can be overridden per cluster by enabling the `Warn as fail` extra parameter on the `GPUSampler` data producer, which makes a `WARN` result show as `FAIL` instead of `PASS`:

**Example**

```bash
[basecm11]% monitoring setup
[basecm11->monitoring->setup]% use gpusampler
[basecm11->monitoring->setup[GPUSampler]]% set -e "Warn as fail" true
[basecm11->monitoring->setup*[GPUSampler*]]% commit
```

The `Warn as fail` extra parameter is `false` by default. Setting it to `true` applies to all `gpu_health_*` checks from the `GPUSampler` producer, whether specified with a GPU parameter (for example `gpu_health_overall:gpu0`) or without one.

<table>
  <thead>
    <tr>
      <th colspan="2">*Table  [G.2.2](#g22-gpu-health-checks): List Of GPU Health Checks*</th>
    </tr>
  </thead>

  <tbody>
    <tr>
      <td>
        **Name**
      </td>

      <td>
        **Query**

         (script response is 

        `PASS`

        /

        `FAIL`

        )
      </td>
    </tr>

    <tr>
      <td>
        `gpu_health_driver`
      </td>

      <td>
        Is the driver-related subsystem OK?
      </td>
    </tr>

    <tr>
      <td>
        `gpu_health_hostengine`

        \*
      </td>

      <td>
        Is the host engine status, for all GPU devices on that node, OK?
      </td>
    </tr>

    <tr>
      <td>
        `gpu_health_inforom`
      </td>

      <td>
        Is the Inforom OK?
      </td>
    </tr>

    <tr>
      <td>
        `gpu_health_mcu`
      </td>

      <td>
        Is the microcontroller unit OK?
      </td>
    </tr>

    <tr>
      <td>
        `gpu_health_mem`
      </td>

      <td>
        Is the memory subsystem OK?
      </td>
    </tr>

    <tr>
      <td>
        `gpu_health_nvlink`
      </td>

      <td>
        Is the NVLINK system OK?
      </td>
    </tr>

    <tr>
      <td>
        `gpu_health_nvswitch_fatal`
      </td>

      <td>
        Is the NVSwitch showing no fatal errors?
      </td>
    </tr>

    <tr>
      <td>
        `gpu_health_nvswitch_non_fatal`
      </td>

      <td>
        Is the NVSwitch showing no non-fatal errors?
      </td>
    </tr>

    <tr>
      <td>
        `gpu_health_overall`

        \**
      </td>

      <td>
        Is the overall GPU health OK?
      </td>
    </tr>

    <tr>
      <td>
        `gpu_health_pcie`
      </td>

      <td>
        Is the PCIe system OK?
      </td>
    </tr>

    <tr>
      <td>
        `gpu_health_pmu`
      </td>

      <td>
        Is the power management unit OK?
      </td>
    </tr>

    <tr>
      <td>
        `gpu_health_power`
      </td>

      <td>
        Is the power OK?
      </td>
    </tr>

    <tr>
      <td>
        `gpu_health_sm`
      </td>

      <td>
        Is the streaming multiprocessor OK?
      </td>
    </tr>

    <tr>
      <td>
        `gpu_health_thermal`
      </td>

      <td>
        Is the temperature OK?
      </td>
    </tr>
  </tbody>
</table>

\* Specified without a GPU because the check is a check for all GPUs.

\*\* If specified without a GPU, then the check is a check for all GPUs.

For example, for the `gpu_health_overall` health check:

`gpu_health_overall` is for all the GPUs

`gpu_health_overall:gpu0` is just for GPU0

### G.2.3  Redfish Health Checks

The available Redfish health checks are displayed in table [G.2.3](#g23-redfish-health-checks).

<table>
  <thead>
    <tr>
      <th colspan="2">*Table  [G.2.3](#g23-redfish-health-checks): Redfish health checks*</th>
    </tr>
  </thead>

  <tbody>
    <tr>
      <td>
        **Redfish Health Check**
      </td>

      <td>
        **Description**
      </td>
    </tr>

    <tr>
      <td>
        `chassis_health`
      </td>

      <td>
        Health status of chassis
      </td>
    </tr>

    <tr>
      <td>
        `cpu_health`
      </td>

      <td>
        Health status of processor
      </td>
    </tr>

    <tr>
      <td>
        `memory_health`
      </td>

      <td>
        Health status of memory
      </td>
    </tr>

    <tr>
      <td>
        `storage_health`
      </td>

      <td>
        Health status of storage
      </td>
    </tr>

    <tr>
      <td>
        `device_health`
      </td>

      <td>
        Health status of storage device
      </td>
    </tr>

    <tr>
      <td>
        `drive_health`
      </td>

      <td>
        Health status of storage drive
      </td>
    </tr>

    <tr>
      <td>
        `volume_health`
      </td>

      <td>
        Health status of storage volume
      </td>
    </tr>

    <tr>
      <td>
        `psu_health`
      </td>

      <td>
        Health status of power supply
      </td>
    </tr>

    <tr>
      <td>
        `fan_health`
      </td>

      <td>
        Health status of a fan
      </td>
    </tr>

    <tr>
      <td>
        `sensor_health`
      </td>

      <td>
        Health status of a sensor
      </td>
    </tr>

    <tr>
      <td>
        `pcie_health`
      </td>

      <td>
        Health status of PCIe device
      </td>
    </tr>

    <tr>
      <td>
        `manager_health`
      </td>

      <td>
        Health status of manager (e.g.: HPE iLO)
      </td>
    </tr>
  </tbody>
</table>

### G.2.4  NetQ Health Checks

NetQ health checks (table [G.2.4](#g24-netq-health-checks)) are healthcheck measurables sourced from NetQ. Configuring NetQ with BCM is described in section [3.11](/admin/configuring#311-configuring-the-netq-network-management-system).

The data producer (section [10.2.10](/admin/monitoring#10210-data-producer)) for NetQ measurables is the `netq` data producer.

<table>
  <thead>
    <tr>
      <th colspan="2">*Table  [G.2.4](#g24-netq-health-checks): NetQ Health Checks*</th>
    </tr>
  </thead>

  <tbody>
    <tr>
      <td>
        **NetQ Health Check**
      </td>

      <td>
        **Query**

         (Response is 

        `PASS`

        /

        `FAIL`

        )
      </td>
    </tr>

    <tr>
      <td>
        `NetQ_node_fan_status`
      </td>

      <td>
        Is the fan working?
      </td>
    </tr>

    <tr>
      <td>
        `NetQ_node_NVLink_status`
      </td>

      <td>
        Is NVLink up?
      </td>
    </tr>

    <tr>
      <td>
        `NetQ_node_PSU_status`
      </td>

      <td>
        Is the power supply unit working?
      </td>
    </tr>

    <tr>
      <td>
        `NetQ_node_Temp_status`
      </td>

      <td>
        Is the temperature sensor working?
      </td>
    </tr>
  </tbody>
</table>

### G.2.5  Parameters For Health Checks

Health checks have the parameters indicated by the left column in the example below:

**Example**

```bash
[myheadnode->monitoring->measurable]% show cmsh
Parameter                        Value
-------------------------------- ----------------------------------------------
Class                            Internal
Consolidator                     - (cmsh)
Description                      Checks whether cmsh is available, i.e. can we
                                 use cmsh for the default cluster?
Disabled                         no (cmsh)
Gap                              0 (cmsh)
Maximal age                      0s (cmsh)
Maximal samples                  4,096 (cmsh)
Name                             cmsh
Parameter
Producer                         cmsh
Revision
Type                             HealthCheck
```

If the value is inherited from the producer, then it is shown in parentheses next to the value. An inherited value can be overwritten by setting it directly for the parameter of a measurable.

The parameters are a subset of the parameters for metrics described in section [G.1.15](#g115-parameters-for-metrics).

## G.3  Enummetrics

<table>
  <colgroup>
    <col />

    <col />
  </colgroup>

  <thead>
    <tr>
      <th colspan="2">*Table  [G.3](#g3-enummetrics): List Of Enummetrics*</th>
    </tr>
  </thead>

  <tbody>
    <tr>
      <td>
        **Name**
      </td>

      <td>
        **Query**
      </td>
    </tr>

    <tr>
      <td>
        `DeviceStatus`
      </td>

      <td>
        What is the status of the device? Possible values are: 

        \


        • up

        \


        • down

        \


        • pending

        \


        • installing

        \


        • installer_failed

        \


        • installer_rebooting

        \


        • installer_calling_init

        \


        • installer_unreachable

        \


        • going_down

        \


        • booting

        \


        • undefined

        \


        • closed

        \


        • muted

        \


        • terminated

        \


        • burning

        \


        • unknown

        \


        • opening

        \


        • no data
      </td>
    </tr>

    <tr>
      <td>
        `wlm_slurm_state`
      </td>

      <td>
        What is the status of the device allocated to Slurm? Possible values are: 

        \


        • allocated

        \


        • completing

        \


        • down

        \


        • drain

        \


        • draining

        \


        • fail

        \


        • failing

        \


        • maint

        \


        • idle

        \


        • mixed
      </td>
    </tr>
  </tbody>
</table>

## G.4  Actions And Their Parameters

### G.4.1  Actions

<table>
  <thead>
    <tr>
      <th colspan="2">*Table  [G.4.1](#g41-actions): List Of Actions*</th>
    </tr>
  </thead>

  <tbody>
    <tr>
      <td>
        **Name**
      </td>

      <td>
        **Description**
      </td>
    </tr>

    <tr>
      <td>
        `Drain`
      </td>

      <td>
        Allows no new processes on a compute node from the workload manager. This means that already running jobs are permitted to complete. Usage Tip: Plan for undrain from another node becoming active
      </td>
    </tr>

    <tr>
      <td>
        `Send e-mail to administrators`
      </td>

      <td>
        Sends mail using the mailserver that was set up during server configuration. Default destination is 

        `root@localhost`

        . The e-mail address that it is otherwise sent to is specified by the 

        `recipient`

         parameter for this action.
      </td>
    </tr>

    <tr>
      <td>
        `Event`
      </td>

      <td>
        Send an event to users with a connected client
      </td>
    </tr>

    <tr>
      <td>
        `ImageUpdate`
      </td>

      <td>
        Update the image on the node
      </td>
    </tr>

    <tr>
      <td>
        `PowerOff`
      </td>

      <td>
        Powers off, hard
      </td>
    </tr>

    <tr>
      <td>
        `PowerOn`
      </td>

      <td>
        Powers on, hard
      </td>
    </tr>

    <tr>
      <td>
        `PowerReset`
      </td>

      <td>
        Power reset, hard
      </td>
    </tr>

    <tr>
      <td>
        `Reboot `
      </td>

      <td>
        Reboot via the system, trying to shut everything down cleanly, and then start up again
      </td>
    </tr>

    <tr>
      <td>
        `killprocess`

        \*
      </td>

      <td>
        Kills processes seen by CMDaemon with the KILL (-9) signal. The PIDs are passed via the 

        `CMD_INFO_MESSAGE`

         environmental variable. Syntax: 

        `killprocess `

        \<

        *PID1*

        \[,<

        *PID2*

        \>,...]> This action is designed to work with 

        `rogueprocess`

         (

        [here](#g21-regular-health-checks)

        )
      </td>
    </tr>

    <tr>
      <td>
        `remount`

        \*
      </td>

      <td>
        remounts all defined mounts
      </td>
    </tr>

    <tr>
      <td>
        `testaction`

        \*
      </td>

      <td>
        An action script example for users who would like to create their own scripts. The source has helpful remarks about the environment variables that can be used as well as tips on configuring it generally
      </td>
    </tr>

    <tr>
      <td>
        `Shutdown `
      </td>

      <td>
        Power off via system, trying to shut everything down cleanly
      </td>
    </tr>

    <tr>
      <td>
        `Undrain node`
      </td>

      <td>
        Allow processes to run on the node from the workload manager
      </td>
    </tr>
  </tbody>
</table>

\* standalone scripts, not built-ins.

If running from a head node, the script is in directory: `/cm/local/apps/cmd/scripts/actions/`

If running from a regular node, the script is in directory: `/cm/images/default-image/cm/local/apps/`

`cmd/scripts/actions/`

### G.4.2  Parameters For A Monitoring Action

The default monitoring actions are listed in section [10.4.4](/admin/monitoring#1044-monitoring-configuration-actions).

All actions have in common the parameters shown by the left column, illustrated by the example below for the `drain` action:

**Example**

```bash
[myheadnode->monitoring->action]% show drain
Parameter          Value
------------------ -----------------------
Action             Drain node from all WLM
Allowed time
Disable            no
Name               Drain
Revision
Run on             Active
Type               DrainAction
```

Out of the full list of default actions, the actions with only the common parameter settings are:

* `Poweron`: Powers off the node

* `PowerOff`: Powers off the node

* `PowerReset`: Hard resets the node

* `Drain`: Drains the node (does not allow new jobs on that node)

* `Undrain`: Undrains the node (allows new jobs on that node)

* `Reboot`: Reboots node via the operating system.

* `Shutdown`: Shuts the node down via the operating system.

* `ImageUpdate`: Updates the node from the software image

* `Event`: Sends an event to users connected with `cmsh` or Base View

#### G.4.2.1  Extra Parameters For Some Actions

The following actions have extra parameters:

**Action of the type `ScriptAction`:**

* `killprocess`: A script that kills a specified process

* `testaction`: A test script

* `remount`: A script to remount all devices

The extra parameters for an action of type `ScriptAction` are:

* * `Arguments`: List of arguments that are taken by the script

  * `Node environment`: Does the script run in the node environment?

  * `Script`: The script path

  * `timeout`: Time within which the script must run before giving up

**Action of the type `EmailAction`:**

* `Send e-mail to administrators`: Sends an e-mail out, by default to the administrators

The extra parameters for an action of type `EmailAction` are:

* * `All administrators`: sends the e-mail to the list of users in the `Administrator e-mail` setting in `partition[base]` mode

  * `Info`: the body of the e-mail message

  * `Recipients`: a list of recipients