> This page is for version 11 (default). > For other versions, use one of these documentation indexes: > - 11 (default): https://docs.nvidia.com/base-command-manager/product-manuals/11/llms.txt > - 10: https://docs.nvidia.com/base-command-manager/product-manuals/10/llms.txt > For clean Markdown of any page, append .md to the page URL. > For a complete documentation index, see https://docs.nvidia.com/base-command-manager/product-manuals/llms.txt. > For AI client integration (Claude Code, Cursor, etc.), connect to the MCP server at https://docs.nvidia.com/base-command-manager/product-manuals/_mcp/server. # Dynamic Power Software ## 11.1 Introduction NVIDIA Dynamic Power Software (DPS), formerly Domain Power Service, is a datacenter level software ([https://docs.nvidia.com/datacenter/dps/versions/latest/](https://docs.nvidia.com/datacenter/dps/versions/latest/)) intended to optimally manage power allocation across the entire datacenter. By default, if a Slurm topology has PRS (Chapter [10](/nvidia-mission-control/prs)) enabled, then DPS configures PRS with a power budget for nodes. ## 11.2 Deployment Of DPS DPS runs within a Kubernetes cluster. A Kubernetes cluster can be set up as described in [section 4.2 of the *Containerization Manual*](/containerization/kubernetes#42-kubernetes-setup). DPS works with Slurm power management. After a Slurm cluster has been configured and deployed ([section 7.3.2 of the *Administrator Manual*](/admin/workloadmanagement#732-running-cm-wlm-setup-as-a-tui)), DPS can be deployed. Deployment of DPS is started in BCM by running the `cm-mission-control-setup` script. This brings up a TUI wizard (figure [11.1](#fig-cm-mission-control-setup-cm-dps-setup-deploy-start)): ![Starting DPS deployment with cm-mission-control-setup](/base-command-manager/product-manuals/_fern-files/nvidia-bcm-manuals.docs.buildwithfern.com/dc4a2f25e15422e2c87c091d2e88c3dd5c8d7c314e72a4f8ad45e3285f502b57/_dot_dot_/_dot_dot_/_dot_dot_/_dot_dot_/_dot_dot_/_dot_dot_/tmp/tmp-465-wPoXzKKBLZez/latextofern/fern/assets/figures/cm-mission-control-setup-cm-dps-setup-deploy-start.png) *Figure 11.1: Starting DPS deployment with cm-mission-control-setup* The menu item can still show the earlier name of DPS, Domain Power Service, as in figure [11.1](#fig-cm-mission-control-setup-cm-dps-setup-deploy-start). Selecting DPS deployment brings up a screen that allows a password to be set for the DPS user (figure [11.2](#fig-cm-mission-control-setup-cm-dps-setup-deploy-password)): ![Setting a password for DPS with cm-mission-control-setup](/base-command-manager/product-manuals/_fern-files/nvidia-bcm-manuals.docs.buildwithfern.com/70a4ed65110693e860bdd1312bf12c5cb133cafd65c0ecb708567590a550af0e/_dot_dot_/_dot_dot_/_dot_dot_/_dot_dot_/_dot_dot_/_dot_dot_/tmp/tmp-465-wPoXzKKBLZez/latextofern/fern/assets/figures/cm-mission-control-setup-cm-dps-setup-deploy-password.png) *Figure 11.2: Setting a password for DPS with cm-mission-control-setup* A DPS version can then be set (figure [11.3](#fig-cm-mission-control-setup-cm-dps-setup-deploy-version)): ![Selecting a DPS version with cm-mission-control-setup](/base-command-manager/product-manuals/_fern-files/nvidia-bcm-manuals.docs.buildwithfern.com/e1b5b4468efcbdd71025d57bb35dbb6cdcb0204691fa56c13fdad714909325ac/_dot_dot_/_dot_dot_/_dot_dot_/_dot_dot_/_dot_dot_/_dot_dot_/tmp/tmp-465-wPoXzKKBLZez/latextofern/fern/assets/figures/cm-mission-control-setup-cm-dps-setup-deploy-version.png) *Figure 11.3: Selecting a DPS version with cm-mission-control-setup* Deployment can be carried out by the next screen. DPS can be undeployed using the `cm-mission-control-setup` TUI. ## 11.3 Known Issues For BCM 11.31.0 ### 11.3.1 Known Issue When Undeploying DPS If there are DNS zones other than DPS in the file `/etc/bind/named.conf.include` and DPS is undeployed using `cm-mission-control-setup`, then the `named` service enters a broken state. To avoid this issue, DPS should not be undeployed through `cm-mission-control-setup` in BCM version 11.31.0. Instead, DPS can be undeployed as follows: * The `epilogslurmctld` and `prologslurmctld` parameters of the Slurm instance that DPS is running are cleaned up. For example for a Slurm instance DPSslurm: ```bash root@basecm11:~# cmsh [basecm11]% wlm use DPSslurm [basecm11->wlm[DPSslurm]]% clear epilogslurmctld [basecm11->wlm*[DPSslurm*]]% clear prologslurmctld [basecm11->wlm*[DPSslurm*]]% commit [basecm11->wlm[DPSslurm]]% ``` * The Kubernetes module for the instance of Kubernetes where DPS is deployed is loaded. ```bash root@basecm11:~# module load kubernetes ``` * DPS is uninstalled with Helm. Also any PVC in the `dps` namespace is deleted, and the `dps` namespace is deleted: ```bash root@basecm11:~# helm uninstall -n dps dps root@basecm11:~# kubectl delete pvc --all -n dps --ignore-not-found=true root@basecm11:~# kubectl delete namespace dps ``` * if the DPS LDAP user, for example \<*dps-ldap-username*>, was created for DPS-usage only, then the user can be removed with: **Example** ```bash root@basecm11:~# cmsh -c "user; remove ; commit" ``` * It is a good idea to make a backup of `named.conf.include` for reference: **Example** ```bash root@basecm11:~# cd /etc/bind/ root@basecm11:/etc/bind# cp named.conf.include named.conf.include-$(date +%Y-%m-%d_%H-%M-%S) ``` * The DPS zone, which is a zone defined for DPS in the file `/etc/bind/named.conf.include`, must be removed. The zone in the file looks like this: ```bash zone "dps" in { type master; file "dps.zone"; }; ``` The other zones in the file must not be changed. * It is a good idea to make a backup of `/etc/bind/dps.zone` for reference: ```bash root@basecm11:~# cd /etc/bind/ root@basecm11:/etc/bind# cp -v dps.zone dps.zone-$(date +%Y-%m-%d_%H-%M-%S) ``` * A reload of `named` can then be carried out with: ```bash root@basecm11:~# systemctl reload named ``` and the DNS server should run properly again. ### 11.3.2 Verify Deployment Failure Due To Mismatching LDAP User Password If DPS is deployed using a pre-existing user and if the password in LDAP for that user does not match the password used for DPS setup, then the `Verify Deployment` stage may fail with an error. The deployment halts, prompting for further input: **Example** ```bash ## Progress: 94 #### stage: dps: Verify Deployment ### ERROR FOUND ### Failed to login: CalledProcessError('Command \'. /etc/profile.d/modules.sh && module load cm-dpsctl && dpsctl --host=api.dps --port=443 --insecure-tls-skip-verify --credentials-path /cm/local/apps/dps/etc/creds.yaml login\' returned exit code EX_ERROR<1(0x01)> while expected (,)\n\tSTDOUT:\n{\n "status": {\n "diag_msg": "Command execution failed with an error",\n...\n "ok": false\n }\n}\n\tSTDERR:\n') Undo/Abort/Skip/Retry/Info/Debug/Remote debug: u/a/s/r/i/d/D: ``` The issue can be worked around by modifying the password (\<*dps-user-password*>) used by the LDAP DPS user (\<*dps-user*>) to match the password of the DPS configuration: **Example** ```bash root@basecm11:~# cmsh [basecm11]% user [basecm11->user]% use [basecm11->user[]]% set password enter new password: retype new password: [basecm11->user[]]% commit ``` A retry (typing r at the prompt) of the `Deployment Verification` stage should then succeed: **Example** ```bash ## Progress: 89 #### stage: dps: Setup Slurm ## Progress: 94 #### stage: dps: Verify Deployment ### ERROR FOUND ### Failed to login: CalledProcessError('Command \'. /etc/profile.d/modules.sh && module load cm-dpsctl && dpsctl --host=api.dps --port=443 --insecure-tls-skip-verify --credentials-path /cm/local/apps/dps/etc/creds.yaml login\' returned exit code EX_ERROR<1(0x01)> while expected (,)\n\tSTDOUT:\n{\n "status": {\n "diag_msg": "Command execution failed with an error",\n...\n "ok": false\n }\n}\n\tSTDERR:\n') Undo/Abort/Skip/Retry/Info/Debug/Remote debug: u/a/s/r/i/d/D:r Retrying the stage 'Verify Deployment'. #### stage: dps: Verify Deployment ## Progress: 100 Took: 01:18 min. Progress: 100/100 ################### Finished execution for 'CM NVIDIA Mission Control Setup', status: completed CM NVIDIA Mission Control Setup finished! ```