BCM Power Shelf Integration
The NVIDIA GB200 server platforms use power shelves to distribute power in racks. Several power shelves are used per rack. Each power shelf takes up one rack unit. Each power shelf comprises several PSUs (Power Supply Units).
The reference design for an NVIDIA GB200 rack has 8 power shelves, and uses 6 PSUs in each power shelf. Each PSU is rated up to 5.5kW, so that each power shelf is rated up to 33kW.
This chapter primarily considers an NVIDIA GB200 server platform that has been configured to work with BCM, though BCM’s power shelf support also extends to other platform generations, as described in section 4.1.
4.1 The Vera Rubin Power Shelf
BCM also supports racks using the reference architecture (RA) Vera Rubin generation power shelf. From BCM’s perspective, the RA power shelf is discovered, monitored, and controlled identically to the GB200 power shelf described in this chapter: it appears as the same powershelf device kind, uses the same list -t powershelf, powershelfoverview, and rack-mode commands, and follows the same power sequencing. PSU counts are always discovered dynamically over Redfish rather than hardcoded, so the command examples shown throughout this chapter apply to RA racks without modification. Examples showing shelf counts or physical rack layout are GB200-specific, since those differ for RA (see the following list).
Physically, the RA power shelf differs from the GB200 power shelf as follows:
-
4 power shelves per rack, compared to 8 for GB200 — each RA power shelf delivers substantially more power, so fewer are needed.
-
6 PSUs per shelf, the same count as GB200.
-
3 rack units (3RU) in height, compared to 1RU for GB200.
-
Rated up to 110kW output per shelf at 415VAC or higher input, compared to 33kW for GB200.
-
54V output voltage, compared to 50V for GB200.
-
3-phase PSU topology, compared to single phase for GB200.
The input voltage range (415-480VAC) and cooling method (air) are unchanged from GB200.
4.2 Power Shelf Listing And Overview
The list -t powershelf command in device mode can be used to list the power shelves of a configured cluster
Power shelves are configured as devices on an IPMI network, as indicated by the following example (some rows ellipsized):
Example
The powershelfoverview command displays an overview of a particular power shelf, showing the measurements from its PSUs (6 in the following example):
Example
4.3 Power Shelves Networking Configuration
The network interface settings for a specific power shelf can be managed with its Redfish interface (section 3.7.1 of the Administrator Manual). For example, a power shelf, here named B05-P1-PWR-01, can have the following values set with the set command:
Example
Redfish metrics, which are the ones prefixed by RF_ (as seen in section 12.8), are sampled only when the BMC interface starts with rf0, rf1, …
4.4 Access Configuration For The Power Management Controller
Access to the Power Management Controller (PMC) is configured in the BMC settings:
Example
Authentication allows NVIDIA Mission Control to monitor the power shelves using Redfish sampling.
4.5 Power Shelf Settings Overview
An overview of the settings for a particular power shelf can be viewed with the show command:
Example
4.6 Power Shelf Metrics Overview
A listing of the power shelf metrics can be seen with some grepping (the output for PSUs 2 to 5 have been ellipsized for readability in this manual):
The descriptions for these metrics are as indicated in table 12.4.
4.7 Power Shelf Firmware
Power shelf firmware is managed by the firmware command options as described in Section 14.5.3 of the Administrator Manual.
BCM currently supports power shelf firmware updates for two vendors, Delta and Lite-On. Each power shelf, regardless of vendor, has two kinds of firmware components: a Power Management Controller (PMC), and several Power Supply Units (PSUs). Unlike DGX compute trays and NVLink switch trays, the device type PowerShelf on its own selects the correct firmware handling for a power shelf, so no separate firmwaremanagemode setting is needed.
Firmware files for both supported vendors are uploaded to the same directory on the head node:
/cm/local/apps/cmd/etc/htdocs/bios/firmware/powershelf/
Once uploaded, the firmware info command, run from device mode against a power shelf, lists the available firmware files and the components they apply to:
Example
The firmware status command shows the current version of each firmware component on the power shelf:
Example
Running firmware flash with a firmware file updates only the components that file applies to. The following example flashes a PMC firmware file, which updates only the PMC components, and leaves the power shelf’s PSUs alone:
Example
The --targets option of firmware flash is not supported for power shelves, and updates are always carried out with the effect of the --force option, so the components in the uploaded file are always installed.
While flashing is in progress, firmware status shows the components being updated with a flashing state and a progress percentage. Only the PMC components in the following example are being updated by the flash just carried out; the PSUs are unaffected and remain current:
Example
Unlike DGX compute trays, power shelves activate new firmware automatically, without requiring a manual AC power cycle. However, firmware status does not reliably report progress once flashing passes approximately 60%, since the power shelf resets during activation. A firmware status command run during this window may briefly show a failed result with no further detail; this does not necessarily mean the update failed. Running firmware status again after a short wait confirms the new firmware version is shown as current.
Lite-On power shelves currently have a more limited firmware workflow than Delta power shelves. For Lite-On power shelves:
-
PSUs must be updated one at a time rather than in a single
firmware flashcall. -
There is no indication of progress in the
Progresscolumn for firmware updates. What can be observed is that:-
by default, the power shelf auto-activates new firmware immediately, and stops responding while activating
-
firmware statusgives a response after the firmware has been updated
Progress monitoring is expected to be addressed in a future release.
-
Updating firmware on multiple power shelves in parallel:
firmware status and firmware flash can also be run against several power shelves at once, from device mode, by combining the rack (--rack, or -r) and device type (--type, or -t) options with the intersection option (--intersection, or -i). This starts one background firmware process per device, rather than updating the shelves one at a time. The following example checks firmware status on every power shelf in rack d03:
Example
A --dry-run flash can be run the same way, across the same rack-wide selection, before committing to a real flash:
Example
This pattern is not specific to power shelves: any device grouping of the same hardware make, model, and part number can be targeted this way. For compute nodes and NVSwitches, category (--category, or -c) and node lists (--nodes, or -n) are typically used instead of rack and type.
4.8 Power Shelf Circuit Monitoring
A power circuit typically supplies several power shelves, based on the rated power of the circuit and power shelves. Administration and wiring convenience may also play a role in allocating a circuit to a power shelf.
So, for example, the administrator may configure a rack with 6 power shelves so that it is supplied power from one power circuit.
The power circuit breakers used by each power circuit, typically provided by a remote power panel (RPP), can have their locations noted for administrative convenience.
Example
The cluster may have been set up to monitor the power circuit breaker for the power reaching BCM (section 3.3.1.1). In that case, the overview command typically displays the power consumption and current per phase.
Example
A value of 0 A means that the current limit is not reported.
4.9 Power Shelves And Rack Mode Commands
This section discusses some of the commands that can be useful for setting up and viewing devices from within the rack mode of cmsh (section 3.16 of the Administrator Manual), but with a focus on power shelf management.
4.9.1 The list Command
If the attributes of racks in rack mode have been set up in a meaningful way, then the list command can give useful information on how the power shelves in racks are organized:
Example
The help set command describes how these values can be set.
In the preceding example, the administrator has noted that a rack, for example A05, is in a room T, and is at an (x,y) coordinate of (1,5) in that room.
The power shelf devices have also been given meaningful names: a05-p1-pwr-01, a05-p1-pwr-02…, which here has a prefix that is the rack it is in, and uses pwr to indicate it is a power shelf.
4.9.2 The rackoverview Command
The rackoverview command displays information about the types of entities in a rack. It also then lists some more detailed information about some of the entity types.
The power shelf is one of the types that has more detailed information listed (some output truncated and ellipsized, because this section is focussing on the power shelves):
Example
In the preceding, the A05 rack is seen to have some active power shelves, which show some power-related values.
4.9.3 The display Command
If the cluster administrator has recorded the device positions in the rack, then the display command in rack mode is useful for seeing the position of where power shelves are located in the rack:
Example
4.9.4 Power Management Of Racks
For devices in general, power operation commands are described in Section 4.2 of the Administrator Manual. Power status responses are described here.
Racks can take a while to power up through the configured sequence of components (power shelves, switches, nodes). The BUSY status can therefore be visible for a while. In this status, the power up operation can be cancelled.
A session showing power commands running on the racks of a DGX OS cluster may show output similar to the following:
Example
More details on power commands use in the rack mode of cmsh can be seen with:
Example