> This page is for version 11 (default). > For other versions, use one of these documentation indexes: > - 11 (default): https://docs.nvidia.com/base-command-manager/product-manuals/11/llms.txt > - 10: https://docs.nvidia.com/base-command-manager/product-manuals/10/llms.txt > For clean Markdown of any page, append .md to the page URL. > For a complete documentation index, see https://docs.nvidia.com/base-command-manager/product-manuals/llms.txt. > For AI client integration (Claude Code, Cursor, etc.), connect to the MCP server at https://docs.nvidia.com/base-command-manager/product-manuals/_mcp/server. # Rack Management And Management Policies ## 3.1 Introduction Rack management is the process of using tools and software to organize and manage the infrastructure components and racks in a data center. A DGX GB200 system can be thought of as a “disaggregated supercomputer in a rack”. The cluster administrator can manage the individual nodes in the traditional way, but BCM rack management features introduced in BCM version 11 allow the GB200 nodes to be managed more conveniently as unified NVL domains. ## 3.2 Basic Rack Management Basic rack management is covered in [section 3.16 of the *Administrator Manual*](/admin/configuring#316-managing-and-configuring-a-rack). The commands in `rack` mode that are discussed there include: * The `list` command can give useful information on how the devices in racks are organized, if the attributes of racks in `rack` mode have been set up in a meaningful way. An example is given in section [4.9.1](/nvidia-mission-control/powershelf#491-the-list-command). * The `display` command is useful for seeing the position of where devices are located in the rack, if the cluster administrator has recorded the device positions in the rack. An example is given in section [4.9.3](/nvidia-mission-control/powershelf#493-the-display-command). * The `addrackposition` command ([here](/admin/configuring#31622-assigning-devices-to-a-rack-using-cmsh)) was introduced in BCM version 11. It allows rack positions in a cluster to be filled more easily than with a looping script. * The `rackoverview` command ([here](/admin/configuring#rackoverview)) was also introduced in BCM version 11. It allows rack positions in a cluster to be overviewed conveniently. It shows information about the types of entities in a rack. It also then lists some more detailed information about some of the entity types. For a DGX GB200 platform that has been configured, the output of the `rackoverview` command looks similar to the following for a specified rack (here it is a rack named `b05` that has 18 nodes, 9 switches, and 8 power shelves): **Example** ```bash [a03-p1-head-01->rack]% rackoverview b05 Type Up Down Closed Total ----------------- ---------------- ---------------- ---------------- ----------- Nodes 18 0 0 18 DPU nodes 0 0 0 0 Managed switches 0 0 0 0 NVLink switches 9 0 0 9 Power shelves 8 0 0 8 Devices 0 0 0 0 Cores 2,592 - - 2,592 GPUs 72 - - 72 Name Value --------------------------- ----------------- User CPU 0.03% System CPU 0.07% Idle CPU 99.9% Other CPU 0.0% Memory used 1.09 TiB (3.75%) Memory unused 28.4 TiB (97.9%) Memory total 29.0 TiB Total GPU utilization 0 W Total GPU power usage 11.6 KW Total GPU NVlink bandwidth 0 W Average GPU temperature 30.9028 C Total CPU power usage 1.60 KW Average CPU temperature 48.6361 C Node GPU Utilization Temperature Power usage Memory used... ------------------ -------- ------------ ------------ ------------ -----------... b05-p1-dgx-05-c01 gpu0 0.0% 30 C 167.184 W 0 B ... b05-p1-dgx-05-c01 gpu1 0.0% 31 C 154.726 W 0 B ... b05-p1-dgx-05-c01 gpu2 0.0% 32 C 154.894 W 0 B ... b05-p1-dgx-05-c01 gpu3 0.0% 31 C 170.649 W 0 B ... b05-p1-dgx-05-c02 gpu0 0.0% 31 C 162.994 W 0 B ... b05-p1-dgx-05-c02 gpu1 0.0% 31 C 151.328 W 0 B ... b05-p1-dgx-05-c02 gpu2 0.0% 31 C 151.149 W 0 B ... b05-p1-dgx-05-c02 gpu3 0.0% 31 C 170.537 W 0 B ... b05-p1-dgx-05-c03 gpu0 0.0% 31 C 166.664 W 0 B ... b05-p1-dgx-05-c03 gpu1 0.0% 31 C 154.725 W 0 B ... b05-p1-dgx-05-c03 gpu2 0.0% 31 C 151.047 W 0 B ... b05-p1-dgx-05-c03 gpu3 0.0% 31 C 170.125 W 0 B ... b05-p1-dgx-05-c04 gpu0 0.0% 31 C 170.642 W 0 B ... b05-p1-dgx-05-c04 gpu1 0.0% 31 C 155.034 W 0 B ... b05-p1-dgx-05-c04 gpu2 0.0% 31 C 152.078 W 0 B ... b05-p1-dgx-05-c04 gpu3 0.0% 31 C 166.551 W 0 B ... b05-p1-dgx-05-c05 gpu0 0.0% 31 C 170.516 W 0 B ... b05-p1-dgx-05-c05 gpu1 0.0% 31 C 158.7 W 0 B ... b05-p1-dgx-05-c05 gpu2 0.0% 31 C 158.7 W 0 B ... b05-p1-dgx-05-c05 gpu3 0.0% 31 C 174.413 W 0 B ... b05-p1-dgx-05-c06 gpu0 0.0% 30 C 166.44 W 0 B ... b05-p1-dgx-05-c06 gpu1 0.0% 30 C 154.628 W 0 B ... b05-p1-dgx-05-c06 gpu2 0.0% 31 C 154.865 W 0 B ... b05-p1-dgx-05-c06 gpu3 0.0% 31 C 170.423 W 0 B ... b05-p1-dgx-05-c07 gpu0 0.0% 31 C 164.078 W 0 B ... b05-p1-dgx-05-c07 gpu1 0.0% 31 C 158.303 W 0 B ... b05-p1-dgx-05-c07 gpu2 0.0% 31 C 150.958 W 0 B ... b05-p1-dgx-05-c07 gpu3 0.0% 31 C 170.537 W 0 B ... b05-p1-dgx-05-c08 gpu0 0.0% 31 C 166.661 W 0 B ... b05-p1-dgx-05-c08 gpu1 0.0% 31 C 155.875 W 0 B ... b05-p1-dgx-05-c08 gpu2 0.0% 31 C 154.726 W 0 B ... b05-p1-dgx-05-c08 gpu3 0.0% 32 C 170.311 W 0 B ... b05-p1-dgx-05-c09 gpu0 0.0% 31 C 166.331 W 0 B ... b05-p1-dgx-05-c09 gpu1 0.0% 31 C 155.469 W 0 B ... b05-p1-dgx-05-c09 gpu2 0.0% 30 C 154.751 W 0 B ... b05-p1-dgx-05-c09 gpu3 0.0% 32 C 175.429 W 0 B ... b05-p1-dgx-05-c10 gpu0 0.0% 30 C 174.181 W 0 B ... b05-p1-dgx-05-c10 gpu1 0.0% 31 C 158.523 W 0 B ... b05-p1-dgx-05-c10 gpu2 0.0% 31 C 152.808 W 0 B ... b05-p1-dgx-05-c10 gpu3 0.0% 31 C 166.551 W 0 B ... b05-p1-dgx-05-c11 gpu0 0.0% 30 C 166.704 W 0 B ... b05-p1-dgx-05-c11 gpu1 0.0% 31 C 154.726 W 0 B ... b05-p1-dgx-05-c11 gpu2 0.0% 31 C 155.034 W 0 B ... b05-p1-dgx-05-c11 gpu3 0.0% 30 C 170.679 W 0 B ... b05-p1-dgx-05-c12 gpu0 0.0% 31 C 170.424 W 0 B ... b05-p1-dgx-05-c12 gpu1 0.0% 31 C 155.601 W 0 B ... b05-p1-dgx-05-c12 gpu2 0.0% 31 C 150.892 W 0 B ... b05-p1-dgx-05-c12 gpu3 0.0% 31 C 166.806 W 0 B ... b05-p1-dgx-05-c13 gpu0 0.0% 31 C 166.766 W 0 B ... b05-p1-dgx-05-c13 gpu1 0.0% 30 C 151.894 W 0 B ... b05-p1-dgx-05-c13 gpu2 0.0% 31 C 151.22 W 0 B ... b05-p1-dgx-05-c13 gpu3 0.0% 31 C 170.205 W 0 B ... b05-p1-dgx-05-c14 gpu0 0.0% 31 C 172.778 W 0 B ... b05-p1-dgx-05-c14 gpu1 0.0% 31 C 162.152 W 0 B ... b05-p1-dgx-05-c14 gpu2 0.0% 31 C 150.857 W 0 B ... b05-p1-dgx-05-c14 gpu3 0.0% 31 C 166.33 W 0 B ... b05-p1-dgx-05-c15 gpu0 0.0% 31 C 166.518 W 0 B ... b05-p1-dgx-05-c15 gpu1 0.0% 31 C 154.729 W 0 B ... b05-p1-dgx-05-c15 gpu2 0.0% 31 C 154.1 W 0 B ... b05-p1-dgx-05-c15 gpu3 0.0% 31 C 170.537 W 0 B ... b05-p1-dgx-05-c16 gpu0 0.0% 30 C 174.067 W 0 B ... b05-p1-dgx-05-c16 gpu1 0.0% 30 C 158.385 W 0 B ... b05-p1-dgx-05-c16 gpu2 0.0% 31 C 151.157 W 0 B ... b05-p1-dgx-05-c16 gpu3 0.0% 31 C 166.77 W 0 B ... b05-p1-dgx-05-c17 gpu0 0.0% 31 C 166.161 W 0 B ... b05-p1-dgx-05-c17 gpu1 0.0% 31 C 158.62 W 0 B ... b05-p1-dgx-05-c17 gpu2 0.0% 31 C 151.058 W 0 B ... b05-p1-dgx-05-c17 gpu3 0.0% 31 C 169.704 W 0 B ... b05-p1-dgx-05-c18 gpu0 0.0% 31 C 166.441 W 0 B ... b05-p1-dgx-05-c18 gpu1 0.0% 31 C 155.164 W 0 B ... b05-p1-dgx-05-c18 gpu2 0.0% 31 C 150.758 W 0 B ... b05-p1-dgx-05-c18 gpu3 0.0% 31 C 169.752 W 0 B ... Switch Utilization Temperature Power usage Fan speed Links active L... --------------- ------------ ----------- ------------ --------- ------------ -... B05-P1-NVSW-01 30.2% 0 C 0 W 0 RPM 0 0 B05-P1-NVSW-02 27.5% 0 C 0 W 0 RPM 0 0 B05-P1-NVSW-03 0.0% 0 C 0 W 0 RPM 0 0 B05-P1-NVSW-04 27.6% 0 C 0 W 0 RPM 0 0 B05-P1-NVSW-05 27.6% 0 C 0 W 0 RPM 0 0 B05-P1-NVSW-06 27.5% 0 C 0 W 0 RPM 0 0 B05-P1-NVSW-07 27.5% 0 C 0 W 0 RPM 0 0 B05-P1-NVSW-08 27.5% 0 C 0 W 0 RPM 0 0 B05-P1-NVSW-09 27.6% 0 C 0 W 0 RPM 0 0 Power shelf Input power Output power Temperature Fan speed Active PSU To... -------------- ------------ ------------ ------------ --------- ----------- --... B05-P1-PWR-01 3.7 KW 3.5 KW 43.3333 C 6.7 KRPM 6 6 B05-P1-PWR-02 3.11 KW 2.95 KW 42.4 C 6.7 KRPM 6 6 B05-P1-PWR-03 3.7 KW 3.5 KW 44.1667 C 6.7 KRPM 6 6 B05-P1-PWR-04 3.8 KW 3.6 KW 44 C 6.7 KRPM 6 6 B05-P1-PWR-05 3.6 KW 3.4 KW 45.5 C 6.7 KRPM 6 6 B05-P1-PWR-06 3.8 KW 3.6 KW 47.1667 C 6.7 KRPM 6 6 B05-P1-PWR-07 3.5 KW 3.4 KW 46 C 6.7 KRPM 6 6 B05-P1-PWR-08 3.6 KW 3.5 KW 47 C 6.7 KRPM 6 6 [a03-p1-head-01->rack]% ``` ## 3.3 Introduction To Advanced Features In Rack Management Over time, clusters have become more powerful and their cooling requirements have become more demanding. Liquid cooling technology has therefore become more common over time. The monitoring associated with liquid cooling has also developed. Monitoring of the electrical supply, fans, component temperatures, and coolant values such as flow and pressure has become popular. Coolant monitoring in BCM includes an ability to detect leaks and then take actions. An NVIDIA-conforming BMS (section [3.3.1](#331-how-bms-and-bcm-work-together)) provides some of these monitoring and managing features, including features associated with liquid cooling and electrical isolation. The cluster administrator can monitor these advanced features using mostly `rack` mode commands (section [3.3.2.1](#3321-the-electricaloverview-command)). ### 3.3.1 How BMS And BCM Work Together A Building Management System (BMS) is a software used to manage IT assets in a building. A BMS software that can integrate with BCM, is known as a *conforming BMS*. The conformance is to a reference design for data center planning ([NVIDIA Controls and Monitoring Reference Design](https://apps.nvidia.com/pid/contentlibraries/detail?id=1118640)), available to some NVIDIA product users. Since BCM version 11, BCM integrates with a conforming BMS for leak detection and control in the data center (DC). The [*NVIDIA Mission Control Software Installation Guide*](https://docs.nvidia.com/mission-control/docs/nmc-software-installation-guide/2.0.0/) describes the implementation of the integration between BCM and a conforming BMS at: [https://docs.nvidia.com/mission-control/docs/nmc-software-installation-guide/2.0.0/integration-of-bms-with-bcm.html](https://docs.nvidia.com/mission-control/docs/nmc-software-installation-guide/2.0.0/integration-of-bms-with-bcm.html) The guide is also meant to serve as a guide to NVIDIA customers who may use different BMS solutions. BCM communicates with a conforming BMS. Together they monitor the DC. The integration works with DGX GB200 systems that are liquid-cooled, as well as other systems. In a DC, a conforming BMS is able to detect most of the detected leak-related events. The remaining detected leak-related events, from the in-tray BMCs (compute tray and switch tray), are dealt with by BCM. BCM uses the MQTT protocol for two-way communication with the conforming BMS, using a BCM service `cm-mqtt`. The BMS sends BCM information about the inventory of the DC, and about the leak-related events it knows about (non-tray events). MQTT topics are created, and structured in a way that BCM can parse. BCM in turn sends the BMS instructions on liquid management or power management, according to defined policies on events such as leaks or power outages (section [4.8](/nvidia-mission-control/powershelf#48-power-shelf-circuit-monitoring)). #### 3.3.1.1 Building Management System Configuration BMS integration can be configured in BCM in partition mode in `cmsh` and Base View. The type of BMS can be configured in `partition` mode: **Example** ```bash root@basecm11:~# cmsh [basecm11]% partition [basecm11->partition[base]]% set bms nvidia-conforming bms; commit ``` In the preceding example, the BMS is set to an NVIDIA-conforming BMS. The possible types of BMS configuration can be: * `NVIDIA-conforming BMS` * `pipe` * `file` * `URL` If `bms` is set to the URL of a BMS server, for example `http://bmserver/path`, then credentials for it can then be set as follows: **Example** ```bash root@basecm11:~# cmsh [basecm11]% partition [basecm11->partition[base]]% set bms url [basecm11->partition*[base*]]% set bmscertificate [basecm11->partition*[base*]]% set bmspath http://bmsserver/path [basecm11->partition*[base*]]% set bmsprivatekey [basecm11->partition*[base*]]% commit ``` In the following example, an `mqtt` role is assigned in the `mqtt` configuration overlay. The overlay specifies where the `cm-mqtt` service should run. In this case it runs on all the head nodes: **Example** ```bash root@basecm11:~# cmsh [basecm11->partition[base]]% configurationoverlay [basecm11->configurationoverlay]% add mqtt [basecm11->configurationoverlay*[mqtt*]]% set allheadnodes yes ``` Within the role assignment, the administrator specifies the hostname or IP address of where the BMS server runs, and where the `cm-mqtt` service pulls data from: ```bash [basecm11->configurationoverlay*[mqtt*]]% roles [basecm11->configurationoverlay*[mqtt*]->roles]% assign mqtt [basecm11->configurationoverlay*[mqtt*]->roles*[mqtt*]]% show Parameter Value -------------------------------- ---------------------------------------------------------- Name mqtt Revision Type MQTTRole Add services yes Servers <0 in submode> CA certificate path /cm/local/apps/cmd/pythoncm/lib/python3.12/site-packages/\ pythoncm/etc/cacert.pem Private key path /cm/local/apps/cmd/cm-mqtt/etc/mqtt.key Certificate path /cm/local/apps/cmd/cm-mqtt/etc/mqtt.pem Write named pipe path /var/spool/cmd/mqtt.pipe [basecm11->configurationoverlay*[mqtt*]->roles*[mqtt*]]% servers [basecm11->configurationoverlay*[mqtt*]->roles*[mqtt*]->servers]% list Server (key) Port Disabled ---------------- ---------- ---------- [basecm11->configurationoverlay*[mqtt*]->roles*[mqtt*]->servers]% add mqhost [basecm11->configurationoverlay*[mqtt*]->roles*[mqtt*]->servers*[mqhost*]]% show Parameter Value -------------------------------- ------------------------------------------------ Revision Server mqhost Port 1883 Topic BCM/# Disabled no Username Password < not set > Transport tcp Protocol v3.1.1 Certificate required yes CA certificate Certificate Private key [basecm11->configurationoverlay*[mqtt*]->roles*[mqtt*]->servers*[mqhost*]]% set certificate ... ``` A username, password, and service certificate can be set. These allow BCM to communicate with the BMS securely: ```bash [basecm11->configurationoverlay[mqtt]->roles[mqtt]->servers[mqhost]]% show Parameter Value -------------------------------- ------------------------------------------------ Revision Server mqhost Port 1883 Topic BCM/# Disabled no Username cronus Password ******** Transport tcp Protocol v3.1.1 Certificate required yes CA certificate Certificate Private key [basecm11->configurationoverlay[mqtt]->roles[mqtt]->servers[mqhost]]% ``` If the cluster manager has correctly established communication with the BMS, then the status of the BMS (metrics for power circuit, CDUs, electrical, and liquid cooling) can be seen from within `cmsh` (section [4.8](/nvidia-mission-control/powershelf#48-power-shelf-circuit-monitoring)). **Example** ```bash [basecm11]% powercircuit overview Power circuit Power Current Current limit Current phase 1 Current phase 2 ... -------------- --------- ----------- --------------- ---------------- ----------------... RPP-B12-3 14.0 KW 26.1118 A 0 A 17.7822 A 26.1118 A ... RPP-B14-3 15.3 KW 29.6178 A 0 A 17.7419 A 29.6178 A ... RPP-B21-5 7.5 KW 11.2255 A 0 A 10.4872 A 10.5412 A ... ... ``` ### 3.3.2 Commands For Monitoring Advanced Features In Rack Management #### 3.3.2.1 The `electricaloverview` Command The `electricaloverview` command within `rack` mode gives a convenient overview of the electrical supply to the cluster. The power used and the power budget of a specified rack (`B05` in the following example) can be seen: **Example** ```bash [basecm11->rack[B05]]% electricaloverview Rack Power used Power budget Isolation status ---------------- ---------------- ---------------- ---------------- B05 28.5 kW 135 kW 0 ``` If no rack is specified, then the command displays the information for all racks. #### 3.3.2.2 The `liquidcoolingoverview` Command The `liquidcoolingoverview` command can be run from within `rack` mode, and conveniently displays coolant metrics: ```bash [basecm11->rack]% liquidcoolingoverview a05 Supply Return Flow Differential Isolation Rack temperature temperature rate pressure status ----- ------------- ----------- ----------- ------------- ----------- A05 26.03 C 30.78 C 70.938 LPM 31 KPa 0 ``` If no rack is specified, then the command displays the information for all racks. #### 3.3.2.3 The `leakdetectionoverview` Command The `leakdetectionoverview` command within `rack` mode gives a convenient overview about sensor readings related to leak detection in BCM and the electrical isolation status. The output from the command applied to a specific rack looks similar to the following: ```bash [basecm11->rack]% leakdetectionoverview a05 Liquid Liquid Electrical Electrical Leak Leak isolation isolation Liquid isolation isolation Electrical Leak tray sensor request request isolation request request isolation Rack detected detected fault (BCM) (BMS) status (BCM) (BMS) status ---- -------- -------- ------ --------- --------- --------- ---------- --------- ---------- A05 0 0 0 0 0 0 0 0 0 [basecm11->rack]% ``` If no rack is specified, then the command displays the information for all racks. #### 3.3.2.4 Leak Detection Within `device` Mode: The `leakinfo` Command The `leakinfo` command, is related to leakage detection, but it is not a `rack` mode command. It is a `device` mode command. It is noted in this section for completeness. It can be run at `device` mode level to list all leaks detected in the cluster. Typically, and ideally, running the command shows something similar to: ```bash [basecm11->device]% leakinfo Hostname Timestamp Severity Info ---------------- --------- --------- ---------------- [basecm11->device]% ``` If there are issues on the devices, then the output may look similar to: ```bash [basecm11->device]% leakinfo Hostname Timestamp Severity Info --------------- --------- --------- -------------------------------------------- a05-dgx-01-c09 15:09:23 large The state of resource `Chassis_0_LeakDete... a07-dgx-03-c16 15:38:48 large The state of resource `Chassis_0_LeakDete... b05-dgx-05-c08 09:11:06 large The state of resource `Chassis_0_LeakDete... b05-dgx-05-c10 09:11:05 large The state of resource `Chassis_0_LeakDete... b06-dgx-06-c17 11:49:30 large The state of resource `Chassis_0_LeakDete... [basecm11->device]% ``` Running `help leakinfo` lists further command options. #### 3.3.2.5 Leak Detection Within `device` Mode: The `latestmonitoringdata` Command For NVL8 Clusters The `latestmonitoringdata` command ([section 10.6.3 of the *Administrator Manual*](/admin/monitoring#1063-the-latest-data-and-counter-valuesthe-latestdata-and-latestmetriccounters-commands)) is also a `device` mode command rather than a `rack` mode command. It, too, is given an extra mention in this leak detection section, because, for NVL8 hardware, it can be used to inspect raw Redfish leak detector states. For example, on NVL8 hardware, in `device` mode, filtered by leak, it might display something like: ```bash [basecm11->device[a10-u25-p01-fractal-002]]% latestmonitoringdata | grep -i leak RF_LD_LeakDetection LeakDetector/Status OK 2m 2s RF_LeakDetector HGX_Chassis_0_LeakDetector_0 LeakDetector/Moisture OK 2m 2s RF_LeakDetector HGX_Chassis_0_LeakDetector_1 LeakDetector/Moisture OK 2m 2s RF_LeakDetector LeakDetector_GB_0 LeakDetector/Moisture Degraded 2m 2s RF_LeakDetector LeakDetector_GB_1 LeakDetector/Moisture Degraded 2m 2s RF_LeakDetector LeakDetector_SWB_0 LeakDetector/Moisture Degraded 2m 2s ``` The output includes: * `RF_LD_LeakDetection`: The status of the dedicated hardware leak detection system on the device (`LeakDetector/Status`). This reflects whether the leak detection hardware itself has reported a leak event, and is independent of the individual moisture sensor readings, described next. * `RF_LeakDetector`: Per-detector Redfish moisture readings (`LeakDetector/Moisture`). A `Degraded` status indicates that the sensor has detected elevated moisture, which may or may not have crossed the threshold to trigger a leak event in `RF_LD_LeakDetection`. The parameter column identifies the detector location: * `HGX_Chassis_0_LeakDetector_N`: detectors on the HGX chassis * `LeakDetector_GB_N`: detectors on the Grace-Blackwell compute modules * `LeakDetector_SWB_N`: detectors on the NVLink switchboard A status of `OK` indicates no moisture is detected. A status of `Degraded` indicates that the detector has detected moisture. The `leakinfo` command (section [3.3.2.4](#3324-leak-detection-within-device-mode-the-leakinfo-command)) and `leakdetectionoverview` command (section [3.3.2.3](#3323-the-leakdetectionoverview-command)) provide higher-level views of cluster-wide leak status and isolation state. #### 3.3.2.6 The `isolationrequestinfo` Command If a leak is detected, then BCM receives the report. BCM then: * powers off the node that is leaking * powers off the rack if multiple nodes are leaking * isolates the rack to prevent other nodes suffering the same fate Isolation means that power no longer runs through the leaking racks. The isolation status of a rack can be viewed with the `isolationrequestinfo` command, which normally shows something similar to: **Example** ```bash [basecm11->rack]% isolationrequestinfo Rack Timestamp Kind Value Info ---------------- ------------ ------------ -------- ---------------- [basecm11->rack]% ``` During a leakage event, the output may show something similar to: **Example** ```bash [basecm11->rack]% isolationrequestinfo Rack Timestamp Kind Value Info ----- --------- ---- ----- ---------------------------- B05 03:40:01 4 Default building policy/Rack [basecm11->rack]% ``` ### 3.3.3 Managing Leaks With `rules` In `leakactionpolicies` The cluster administrator has the freedom to construct actions based on triggers based on leak detection measurables. While that is powerful, that can be quite labor-intensive for cluster administrators. BCM provides a simpler way to deal with leaks, based on leak action policies. Leak action policies are a collection of rules that carry out actions to be followed when leaks are detected. Racks play an important part in these policies and rules. The policies use predefined scopes, or groupings. The names of these groupings are rules. The groupings are listed next, in order from largest to smallest, along with their associated descriptions: * `Building`: a building is made up of rooms * `Room`: A room has rows of racks in it. * `Row`: A row of racks has individual racks in it. * `Rack`: An individual rack has devices in it. * `Device`: The smallest possible grouping The policies can be listed in the `leakactionpolicies` submode, under `partition` mode. **Example** ```bash [basecm11->partition[base]->leakactionpolicies]% list Name (key) Rules ------------------------ ---------------------------------- Default building policy Building, Device, Rack, Room, Row Default device policy Device Default rack policy Device, Rack Default room policy Device, Rack, Room, Row Default row policy Device, Rack, Row ``` The rules used by each policy are also shown, under the `Rules` column in the preceding example. Each rule in a policy is a copy of a template. By default the rules per policy are identical. Just the rules used by a specific policy can also be viewed: **Example** ```bash [basecm11->partition[base]->leakactionpolicies]% use default building policy [basecm11->partition[base]->leakactionpolicies[Default building policy]]% show Parameter Value -------------------------------- ------------------------------------------------ Name Default building policy Revision Rules Building, Device, Rack, Room, Row ``` Within a policy, the value of the settings for these rules can be listed: **Example** ```bash [basecm11->...leakactionpolicies[Default building policy]->rules]% list Power Power Electrical Liquid Minimal Maximal Name (key) Scope off off rack isolation isolation devices devices ----------- --------- ------ -------- ---------- ---------- ------- ------- Building building yes yes no no 11 100000 Device device yes no no no Rack rack yes yes yes yes 3 100 Room room yes yes no no 7 100000 Row row yes yes no no 5 100 ``` The rules for each policy can be modified independently for each policy, so that, for example the `Rack` rule values may differ between `Default row policy` and `Default building policy`. However, a sensible cluster administrator is not expected to carry out such a modification. Another way of phrasing this is: The values of the rules can be changed from the defaults. In that case, a sensible administrator is expected to keep the values synchronized across the rules in the leak action policies. In the preceding example: If the minimal devices criterion is met for leaking for each rule in a policy, then the values for the column header are applied to the rules of that policy. So, for the policy rules in the preceding example, when leaking is detected: * If only one or two devices are detected as leaking, then only the devices involved are powered off. * If, in one rack, more than 3 devices, but less than 5 devices, are detected as leaking, then in addition to the device rules being applied, the entire rack is powered off. Isolation is also carried out for the electrical power and liquid coolant. * If more devices meet the conditions, so a larger scope is affected by leaking, then the rules for the larger scope are also applied. The rules for a specific policy can be removed. The `row` rule for `Default building policy` can be removed with, for example: ```bash [basecm11->partition[base]->leakactionpolicies[Default building policy]]% rules [basecm11->...leakactionpolicies[Default building policy]->rules]% remove building device rack room row [basecm11->...leakactionpolicies[Default building policy]->rules]% remove row; commit ``` #### 3.3.3.1 Custom Rules For `leakactionpolicies` The names for the rules (`Device`, `Rack`, `Row` ...) are really just default labels, and a sensible cluster administrator is expected to have the description associated with the name match the physical reality. Creating a custom rule may help with this. Custom rules can be added by the cluster administrator, by first adding another policy: ```bash [basecm11->partition[base]->leakactionpolicies[Default building policy]]% add mypolicy [basecm11->partition*[base*]->leakactionpolicies*[mypolicy*]]% list Name (key) Rules ------------------------ ---------------------------------- Default building policy Building, Device, Rack, Room, Row Default device policy Device Default rack policy Device, Rack Default room policy Device, Rack, Room, Row Default row policy Device, Rack, Row mypolicy none [basecm11->partition*[base*]->leakactionpolicies*[mypolicy*]]% show Parameter Value -------------------------------- ------------------------------------------------ Name mypolicy Revision Rules none ``` By default, there are no rules in the new policy: ```bash [basecm11->partition*[base*]->leakactionpolicies*[mypolicy*]]% rules [basecm11->partition*[base*]->leakactionpolicies*[mypolicy*]->rules]% list Electrical Liquid Minimal Maximal Name (key) Scope Power off Power off rack isolation isolation devices devices ----------- --------- ---------- --------------- ---------- ----------- -------- ------ ``` The cluster administrator may have a testing room in which to keep a rack. In that case, the following might be added: ```bash [basecm11->partition*[base*]->leakactionpolicies*[mypolicy*]->rules]% add testroom [basecm11->partition*[base*]->leakactionpolicies*[mypolicy*]->rules*[testroom*]]% show Parameter Value -------------------------------- ------------------------------------------------ Name testroom Revision Scope rack Disabled no Minimal devices 1 Maximal devices 100 Grace period 0s Correlation window 5m Power off yes Power off rack no Electrical isolation no Liquid isolation no Minimal severity 1 Maximal severity 100 ``` Adding a rule adds a default with the preceding rack-based rule values. The rule values can be modified using the `set` command. Running `set` without any options provides a help text explaining what can be set: ```bash [basecm11->partition*[base*]->leakactionpolicies*[mypolicy*]->rules]% set ```