Rack Management And Management Policies
3.1 Introduction
Rack management is the process of using tools and software to organize and manage the infrastructure components and racks in a data center.
A DGX GB200 system can be thought of as a “disaggregated supercomputer in a rack”. The cluster administrator can manage the individual nodes in the traditional way, but BCM rack management features introduced in BCM version 11 allow the GB200 nodes to be managed more conveniently as unified NVL domains.
3.2 Basic Rack Management
Basic rack management is covered in section 3.16 of the Administrator Manual. The commands in rack mode that are discussed there include:
-
The
listcommand can give useful information on how the devices in racks are organized, if the attributes of racks inrackmode have been set up in a meaningful way. An example is given in section 4.9.1. -
The
displaycommand is useful for seeing the position of where devices are located in the rack, if the cluster administrator has recorded the device positions in the rack. An example is given in section 4.9.3. -
The
addrackpositioncommand (here) was introduced in BCM version 11. It allows rack positions in a cluster to be filled more easily than with a looping script. -
The
rackoverviewcommand (here) was also introduced in BCM version 11. It allows rack positions in a cluster to be overviewed conveniently. It shows information about the types of entities in a rack. It also then lists some more detailed information about some of the entity types.
For a DGX GB200 platform that has been configured, the output of the rackoverview command looks similar to the following for a specified rack (here it is a rack named b05 that has 18 nodes, 9 switches, and 8 power shelves):
Example
3.3 Introduction To Advanced Features In Rack Management
Over time, clusters have become more powerful and their cooling requirements have become more demanding. Liquid cooling technology has therefore become more common over time.
The monitoring associated with liquid cooling has also developed. Monitoring of the electrical supply, fans, component temperatures, and coolant values such as flow and pressure has become popular. Coolant monitoring in BCM includes an ability to detect leaks and then take actions. An NVIDIA-conforming BMS (section 3.3.1) provides some of these monitoring and managing features, including features associated with liquid cooling and electrical isolation.
The cluster administrator can monitor these advanced features using mostly rack mode commands (section 3.3.2.1).
3.3.1 How BMS And BCM Work Together
A Building Management System (BMS) is a software used to manage IT assets in a building. A BMS software that can integrate with BCM, is known as a conforming BMS. The conformance is to a reference design for data center planning (NVIDIA Controls and Monitoring Reference Design), available to some NVIDIA product users.
Since BCM version 11, BCM integrates with a conforming BMS for leak detection and control in the data center (DC). The NVIDIA Mission Control Software Installation Guide describes the implementation of the integration between BCM and a conforming BMS at:
The guide is also meant to serve as a guide to NVIDIA customers who may use different BMS solutions.
BCM communicates with a conforming BMS. Together they monitor the DC. The integration works with DGX GB200 systems that are liquid-cooled, as well as other systems.
In a DC, a conforming BMS is able to detect most of the detected leak-related events. The remaining detected leak-related events, from the in-tray BMCs (compute tray and switch tray), are dealt with by BCM.
BCM uses the MQTT protocol for two-way communication with the conforming BMS, using a BCM service cm-mqtt.
The BMS sends BCM information about the inventory of the DC, and about the leak-related events it knows about (non-tray events). MQTT topics are created, and structured in a way that BCM can parse.
BCM in turn sends the BMS instructions on liquid management or power management, according to defined policies on events such as leaks or power outages (section 4.8).
3.3.1.1 Building Management System Configuration
BMS integration can be configured in BCM in partition mode in cmsh and Base View.
The type of BMS can be configured in partition mode:
Example
In the preceding example, the BMS is set to an NVIDIA-conforming BMS. The possible types of BMS configuration can be:
-
NVIDIA-conforming BMS -
pipe -
file -
URL
If bms is set to the URL of a BMS server, for example http://bmserver/path, then credentials for it can then be set as follows:
Example
In the following example, an mqtt role is assigned in the mqtt configuration overlay. The overlay specifies where the cm-mqtt service should run. In this case it runs on all the head nodes:
Example
Within the role assignment, the administrator specifies the hostname or IP address of where the BMS server runs, and where the cm-mqtt service pulls data from:
A username, password, and service certificate can be set. These allow BCM to communicate with the BMS securely:
If the cluster manager has correctly established communication with the BMS, then the status of the BMS (metrics for power circuit, CDUs, electrical, and liquid cooling) can be seen from within cmsh (section 4.8).
Example
3.3.2 Commands For Monitoring Advanced Features In Rack Management
3.3.2.1 The electricaloverview Command
The electricaloverview command within rack mode gives a convenient overview of the electrical supply to the cluster.
The power used and the power budget of a specified rack (B05 in the following example) can be seen:
Example
If no rack is specified, then the command displays the information for all racks.
3.3.2.2 The liquidcoolingoverview Command
The liquidcoolingoverview command can be run from within rack mode, and conveniently displays coolant metrics:
If no rack is specified, then the command displays the information for all racks.
3.3.2.3 The leakdetectionoverview Command
The leakdetectionoverview command within rack mode gives a convenient overview about sensor readings related to leak detection in BCM and the electrical isolation status.
The output from the command applied to a specific rack looks similar to the following:
If no rack is specified, then the command displays the information for all racks.
3.3.2.4 Leak Detection Within device Mode: The leakinfo Command
The leakinfo command, is related to leakage detection, but it is not a rack mode command. It is a device mode command. It is noted in this section for completeness. It can be run at device mode level to list all leaks detected in the cluster.
Typically, and ideally, running the command shows something similar to:
If there are issues on the devices, then the output may look similar to:
Running help leakinfo lists further command options.
3.3.2.5 Leak Detection Within device Mode: The latestmonitoringdata Command For NVL8 Clusters
The latestmonitoringdata command (section 10.6.3 of the Administrator Manual) is also a device mode command rather than a rack mode command. It, too, is given an extra mention in this leak detection section, because, for NVL8 hardware, it can be used to inspect raw Redfish leak detector states.
For example, on NVL8 hardware, in device mode, filtered by leak, it might display something like:
The output includes:
-
RF_LD_LeakDetection: The status of the dedicated hardware leak detection system on the device (LeakDetector/Status). This reflects whether the leak detection hardware itself has reported a leak event, and is independent of the individual moisture sensor readings, described next. -
RF_LeakDetector: Per-detector Redfish moisture readings (LeakDetector/Moisture). ADegradedstatus indicates that the sensor has detected elevated moisture, which may or may not have crossed the threshold to trigger a leak event inRF_LD_LeakDetection. The parameter column identifies the detector location:-
HGX_Chassis_0_LeakDetector_N: detectors on the HGX chassis -
LeakDetector_GB_N: detectors on the Grace-Blackwell compute modules -
LeakDetector_SWB_N: detectors on the NVLink switchboard
-
A status of OK indicates no moisture is detected. A status of Degraded indicates that the detector has detected moisture. The leakinfo command (section 3.3.2.4) and leakdetectionoverview command (section 3.3.2.3) provide higher-level views of cluster-wide leak status and isolation state.
3.3.2.6 The isolationrequestinfo Command
If a leak is detected, then BCM receives the report. BCM then:
-
powers off the node that is leaking
-
powers off the rack if multiple nodes are leaking
-
isolates the rack to prevent other nodes suffering the same fate
Isolation means that power no longer runs through the leaking racks.
The isolation status of a rack can be viewed with the isolationrequestinfo command, which normally shows something similar to:
Example
During a leakage event, the output may show something similar to:
Example
3.3.3 Managing Leaks With rules In leakactionpolicies
The cluster administrator has the freedom to construct actions based on triggers based on leak detection measurables. While that is powerful, that can be quite labor-intensive for cluster administrators.
BCM provides a simpler way to deal with leaks, based on leak action policies. Leak action policies are a collection of rules that carry out actions to be followed when leaks are detected. Racks play an important part in these policies and rules.
The policies use predefined scopes, or groupings. The names of these groupings are rules. The groupings are listed next, in order from largest to smallest, along with their associated descriptions:
-
Building: a building is made up of rooms -
Room: A room has rows of racks in it. -
Row: A row of racks has individual racks in it. -
Rack: An individual rack has devices in it. -
Device: The smallest possible grouping
The policies can be listed in the leakactionpolicies submode, under partition mode.
Example
The rules used by each policy are also shown, under the Rules column in the preceding example.
Each rule in a policy is a copy of a template. By default the rules per policy are identical.
Just the rules used by a specific policy can also be viewed:
Example
Within a policy, the value of the settings for these rules can be listed:
Example
The rules for each policy can be modified independently for each policy, so that, for example the Rack rule values may differ between Default row policy and Default building policy. However, a sensible cluster administrator is not expected to carry out such a modification.
Another way of phrasing this is: The values of the rules can be changed from the defaults. In that case, a sensible administrator is expected to keep the values synchronized across the rules in the leak action policies.
In the preceding example:
If the minimal devices criterion is met for leaking for each rule in a policy, then the values for the column header are applied to the rules of that policy.
So, for the policy rules in the preceding example, when leaking is detected:
-
If only one or two devices are detected as leaking, then only the devices involved are powered off.
-
If, in one rack, more than 3 devices, but less than 5 devices, are detected as leaking, then in addition to the device rules being applied, the entire rack is powered off. Isolation is also carried out for the electrical power and liquid coolant.
-
If more devices meet the conditions, so a larger scope is affected by leaking, then the rules for the larger scope are also applied.
The rules for a specific policy can be removed. The row rule for Default building policy can be removed with, for example:
3.3.3.1 Custom Rules For leakactionpolicies
The names for the rules (Device, Rack, Row …) are really just default labels, and a sensible cluster administrator is expected to have the description associated with the name match the physical reality. Creating a custom rule may help with this.
Custom rules can be added by the cluster administrator, by first adding another policy:
By default, there are no rules in the new policy:
The cluster administrator may have a testing room in which to keep a rack. In that case, the following might be added:
Adding a rule adds a default with the preceding rack-based rule values.
The rule values can be modified using the set command.
Running set without any options provides a help text explaining what can be set: