> This page is for version 11 (default). > For other versions, use one of these documentation indexes: > - 11 (default): https://docs.nvidia.com/base-command-manager/product-manuals/11/llms.txt > - 10: https://docs.nvidia.com/base-command-manager/product-manuals/10/llms.txt > For clean Markdown of any page, append .md to the page URL. > For a complete documentation index, see https://docs.nvidia.com/base-command-manager/product-manuals/llms.txt. > For AI client integration (Claude Code, Cursor, etc.), connect to the MCP server at https://docs.nvidia.com/base-command-manager/product-manuals/_mcp/server. # Power Reservation Steering ## 10.1 Introduction Power reservation steering (PRS) is about automated speculative planning of power across groups of nodes used by Slurm. PRS is typically done for larger clusters, where minimising the number of idling nodes typically results in significant savings. The planning is done over a period of time spent measuring existing resource consumption and observing the power consumed. The patterns that are seen in that period of time suggest a direction for the power allocation on the cluster (hence the word *steering*) for the next period of time. Based on the planned steering, a power cap is set for some groups of nodes (power domains) in the cluster. Setting the power cap for those power domains, means that a *reservation* of the remaining power is made for the rest of the cluster during the upcoming period of time. PRS works using a PRS daemon. This tracks the power availability and manages power planning. A component is the PRS controller that connects to CMDaemon, and which during the time loop period controls the power domains. DPS (Dynamic Power Software, Chapter [11](/nvidia-mission-control/dps)) is a datacenter level software intended to optimally manage power allocation across the entire datacenter. By default, if a Slurm topology has PRS enabled, then DPS configures PRS with a power budget for nodes. ## 10.2 Deployment Of PRS After a Slurm cluster has been configured and deployed ([section 7.3.2 of the *Administrator Manual*](/admin/workloadmanagement#732-running-cm-wlm-setup-as-a-tui)), PRS can be deployed. Deployment of PRS is started in BCM by running the `cm-prs-setup` script. This brings up a TUI wizard (figure [10.1](#fig-cm-mission-control-setup-cm-prs-setup-deploy-start)): ![Starting deploying with cm-prs-setup](/base-command-manager/product-manuals/_fern-img/0646e6adbaa4f7c797bb46eb0ff365e7147fbeb94540ca5370cb49e25e98a6d8.webp) *Figure 10.1: Starting deploying with cm-prs-setup* Categories can be selected for PRS (figure [10.2](#fig-cm-mission-control-setup-cm-prs-setup-select-category)): ![Selecting a category with cm-prs-setup](/base-command-manager/product-manuals/_fern-img/d505ea44e1d2c3348a9587f0a639053daed6ff23ccd39ff14548c6e7747a2dfa.webp) *Figure 10.2: Selecting a category with cm-prs-setup* If selecting categories is too inclusive, then individual nodes can be selected instead in a following screen. After categories or individual nodes have been selected, one or more power domains can be added (figure [10.3](#fig-cm-mission-control-setup-cm-prs-setup-set-power-domain)): ![Setting a power domain with cm-prs-setup](/base-command-manager/product-manuals/_fern-img/b7c7687e12d4f17bae64f876beae3507596d54f66c7b7d99e6d0e313b876ffec.webp) *Figure 10.3: Setting a power domain with cm-prs-setup* Each power domain can be given an arbitrary name, and a domain grouping (figure [10.4](#fig-cm-mission-control-setup-cm-prs-setup-domain-power-rack)): ![Configuring the power domain budget and grouping with cm-prs-setup](/base-command-manager/product-manuals/_fern-files/nvidia-bcm-manuals.docs.buildwithfern.com/f88a6834ae11854de151a49ee4383ea1fe9b81d8c43d7c9ea63519dc498177d3/_dot_dot_/_dot_dot_/_dot_dot_/_dot_dot_/_dot_dot_/_dot_dot_/tmp/tmp-465-wPoXzKKBLZez/latextofern/fern/assets/figures/cm-mission-control-setup-cm-prs-setup-domain-power-rack.png) *Figure 10.4: Configuring the power domain budget and grouping with cm-prs-setup* The default domain group of `all` is simply one big power domain. Groups such as rack, row, room, and so on become new domains, each with the specified power budget. The power budgets for the domains are set automatically if the nodes have been powered up. Otherwise the budgets must be specified manually. The maximum power consumption per node is: * 400W for A100 nodes * 700W for H100 nodes * 1000W for B200 nodes * 2700W for GB200 nodes The `Static power` is the power consumption when the node is shut down, and only the BMC is standing by. Typically it is about 20W. The power draw factor can be set in the range from 0.0-1.0, in steps of 0.1. The power consumption limits can then be set for the domain. ![Setting power consumption limits with cm-prs-setup](/base-command-manager/product-manuals/_fern-files/nvidia-bcm-manuals.docs.buildwithfern.com/d729e891586d5db596a4c2aa1d185a111b9ff8e0efbc6e11398319393059b639/_dot_dot_/_dot_dot_/_dot_dot_/_dot_dot_/_dot_dot_/_dot_dot_/tmp/tmp-465-wPoXzKKBLZez/latextofern/fern/assets/figures/cm-mission-control-setup-cm-prs-setup-set-power-consumption.png) *Figure 10.5: Setting power consumption limits with cm-prs-setup* After the configuration steps are completed, the configuration can be saved and PRS deployed (figure [10.6](#fig-cm-mission-control-setup-cm-prs-setup-save-deploy)): ![Saving the configuration and deploying it with cm-prs-setup](/base-command-manager/product-manuals/_fern-files/nvidia-bcm-manuals.docs.buildwithfern.com/ecfac82dd9d2713a6cd19b91735a8d662434cebf6e715201dd755c1479906ec4/_dot_dot_/_dot_dot_/_dot_dot_/_dot_dot_/_dot_dot_/_dot_dot_/tmp/tmp-465-wPoXzKKBLZez/latextofern/fern/assets/figures/cm-mission-control-setup-cm-prs-setup-save-deploy.png) *Figure 10.6: Saving the configuration and deploying it with cm-prs-setup* ## 10.3 Changes Made On Deployment, And Managing Changes Post-Deployment With `cmsh` ### 10.3.1 PRS Values PRS deployment creates the following configuration overlays: * `prs-server`: this has the `PRSServerRole` with PRSDomain configurations * `prs-client`: this has the `PRSClientRole` with power usage configuration The PRS server configuration overlay is always assigned to the head nodes, and the client overlay is assigned to the nodes and categories configured in the wizard. PRS server configuration values can be managed in `cmsh` with their role under the configuration overlay: **Example** ```bash [basecm11->configurationoverlay[prs-server]->roles[PRS::Server]]% show Parameter Value -------------------------------- ----------------------------------------------------------------- Name PRS::Server Revision Type PRSServerRole Add services yes Config server port 8880 Job scheduler server port 8881 Timeout 10s Interval 30s Window 5 PRS server certificate path /cm/local/apps/prs/etc/server.pem PRS server private key path /cm/local/apps/prs/etc/server.key PRS server CA certificate path /cm/local/apps/prs/etc/ca.pem PRS server CA private key path /cm/local/apps/prs/etc/ca.key CMD certificate path /cm/local/apps/prs/etc/cmd.pem CMD private key path /cm/local/apps/prs/etc/cmd.key PRS client certificate path /cm/local/apps/prs/etc/prs.pem PRS client private key path /cm/local/apps/prs/etc/prs.key PRS client CA certificate path /cm/local/apps/cmd/pythoncm/lib/python3.12/site-packages/ pythoncm/etc/cacert.pem Domains <2 in submode> [basecm11->configurationoverlay[prs-server]->roles[PRS::Server]]% domains [basecm11->configurationoverlay[prs-server]->roles[PRS::Server]->domains]% list Name (key) Power budget Power budget model Power draw model Power draw factor Group by -------------- ------------ ------------------ ---------------- ----------------- --------- anotherdom 30.0KW scalar linear 1 all mypowerdomain 10.0KW scalar linear 1 all [basecm11->configurationoverlay[prs-server]->roles[PRS::Server]->domains]% ``` Similarly, the PRS client role values can be managed under their role: **Example** ```bash [basecm11->configurationoverlay[prs-client]->roles[PRS::Client]]% show Parameter Value -------------------------------- ------------------------------------------------ Name PRS::Client Revision Type PRSClientRole Add services yes Static power usage 800W Static power usage down 30W Min CPU power limit 0W Max CPU power limit 0W Min GPU power limit 0W Max GPU power limit 0W ``` ### 10.3.2 Slurm Values On deployment of PRS, the selectType of the Slurm instance is changed from: `select/cons_tres` to `select/gnl_cons_tres` **Example** ```bash [basecm11->wlm[slurm]]% get selecttype select/gnl_cons_tres ``` The certificate paths for PRS for the Slurm instances are managed within the PRS settings ([here](/admin/workloadmanagement#7517-slurm-prs-settings)) of the Slurm instances: **Example** ```bash [basecm11->wlm[slurm]->prssettings]% show Parameter Value -------------------------------- ------------------------------------------------ Certificate path /cm/local/apps/prs/etc/slurm-slurm.pem Revision Private key path /cm/local/apps/prs/etc/slurm-slurm.key ``` ### 10.3.3 PRS Removal The `cm-prs-setup` script can be used to remove existing PRS domains. ## 10.4 Changes Made On Deployment, And Managing Changes Post-Deployment With Base View ### 10.4.1 Base View Power Reservation Steering Wizard The Base View GUI PRS wizard takes the same inputs as the `cm-prs-setup` TUI PRS wizard of section [10.3](#103-changes-made-on-deployment-and-managing-changes-post-deployment-with-cmsh). The Base View wizard can be accessed using the navigation path: `Mission Control > Power Reservation Steering > Start Wizard` The navigation path is only available if Slurm has previously been installed. Slurm can be installed using the navigation path: `HPC > Workload Management Wizard` The Base View PRS wizard, on starting up, displays an introduction screen as in figure [10.7](#fig-cm-mission-control-setup-bv-prs-installation-start): ![Running The PRS Installation Wizard From Base View](/base-command-manager/product-manuals/_fern-img/287d7e16857b85b6af39c1caece4d5dd9c5b7246649566c391f4b492f63f8254.webp) *Figure 10.7: Running The PRS Installation Wizard From Base View* The steps that the PRS wizard goes through can be followed in the breadcrumb trail. The steps are: * Introduction: a custom YAML configuration can be uploaded here, and the existing YAML configuration can be viewed * Clusters Selection: Slurm instances can be selected here for PRS * Categories and Nodes: regular and head nodes can be chosen for PRS * Configure Power Domain: a power domain can be set, with settings as in the TUI version (figure [10.4](#fig-cm-mission-control-setup-cm-prs-setup-domain-power-rack)) * Node Power Consumption: power consumption for the domain can be set * Summary: a summary of the settings is displayed * Deploy: the deployment stages are carried out and shown ### 10.4.2 Base View Power Reservation Steering Client And Server Configuration Overlays With Base View, the values set for the PRS client and PRS server can be accessed using their roles at configuration overlay, category, and device level. For example, for the PRS client as configured in the Base View PRS configuration overlay, the navigation path to the configuration pane (figure [10.8](#fig-cm-mission-control-bv-prs-client-role-from-overlay)) is: Configuration Overlay > prs-client \> `Settings` \> Roles \> PRS-client ![PRS Client Values Using The Configuration Role In Base View](/base-command-manager/product-manuals/_fern-img/5d860851aa61ae1e883d746487fb9b79cf3db4d2e4f7541c91e77be39c5ec3e3.webp) *Figure 10.8: PRS Client Values Using The Configuration Role In Base View*