Dynamic Power Software

11.1 Introduction

NVIDIA Dynamic Power Software (DPS), formerly Domain Power Service, is a datacenter level software (https://docs.nvidia.com/datacenter/dps/versions/latest/) intended to optimally manage power allocation across the entire datacenter. By default, if a Slurm topology has PRS (Chapter 10) enabled, then DPS configures PRS with a power budget for nodes.

11.2 Deployment Of DPS

DPS runs within a Kubernetes cluster. A Kubernetes cluster can be set up as described in section 4.2 of the Containerization Manual.

DPS works with Slurm power management. After a Slurm cluster has been configured and deployed (section 7.3.2 of the Administrator Manual), DPS can be deployed.

Deployment of DPS is started in BCM by running the cm-mission-control-setup script. This brings up a TUI wizard (figure 11.1):

Figure 11.1: Starting DPS deployment with cm-mission-control-setup

The menu item can still show the earlier name of DPS, Domain Power Service, as in figure 11.1.

Selecting DPS deployment brings up a screen that allows a password to be set for the DPS user (figure 11.2):

Figure 11.2: Setting a password for DPS with cm-mission-control-setup

A DPS version can then be set (figure 11.3):

Figure 11.3: Selecting a DPS version with cm-mission-control-setup

Deployment can be carried out by the next screen.

DPS can be undeployed using the cm-mission-control-setup TUI.

11.3 Known Issues For BCM 11.31.0

11.3.1 Known Issue When Undeploying DPS

If there are DNS zones other than DPS in the file /etc/bind/named.conf.include and DPS is undeployed using cm-mission-control-setup, then the named service enters a broken state.

To avoid this issue, DPS should not be undeployed through cm-mission-control-setup in BCM version 11.31.0.

Instead, DPS can be undeployed as follows:

• The epilogslurmctld and prologslurmctld parameters of the Slurm instance that DPS is running are cleaned up. For example for a Slurm instance DPSslurm:

  root@basecm11:~# cmsh
  [basecm11]% wlm use DPSslurm
  [basecm11->wlm[DPSslurm]]% clear epilogslurmctld
  [basecm11->wlm*[DPSslurm*]]% clear prologslurmctld
  [basecm11->wlm*[DPSslurm*]]% commit
  [basecm11->wlm[DPSslurm]]%

• The Kubernetes module for the instance of Kubernetes where DPS is deployed is loaded.

  root@basecm11:~# module load kubernetes

• DPS is uninstalled with Helm. Also any PVC in the dps namespace is deleted, and the dps namespace is deleted:

  root@basecm11:~# helm uninstall -n dps dps
  root@basecm11:~# kubectl delete pvc --all -n dps --ignore-not-found=true
  root@basecm11:~# kubectl delete namespace dps

• if the DPS LDAP user, for example <dps-ldap-username>, was created for DPS-usage only, then the user can be removed with:

  Example

  root@basecm11:~# cmsh -c "user; remove <dps-ldap-username>; commit"

• It is a good idea to make a backup of named.conf.include for reference:

  Example

  root@basecm11:~# cd /etc/bind/
  root@basecm11:/etc/bind# cp named.conf.include named.conf.include-$(date +%Y-%m-%d_%H-%M-%S)

• The DPS zone, which is a zone defined for DPS in the file /etc/bind/named.conf.include, must be removed. The zone in the file looks like this:

  zone "dps" in {
    type master;
    file "dps.zone";
  };

  The other zones in the file must not be changed.

• It is a good idea to make a backup of /etc/bind/dps.zone for reference:

  root@basecm11:~# cd /etc/bind/
  root@basecm11:/etc/bind# cp -v dps.zone dps.zone-$(date +%Y-%m-%d_%H-%M-%S)

• A reload of named can then be carried out with:

  root@basecm11:~# systemctl reload named

and the DNS server should run properly again.

11.3.2 Verify Deployment Failure Due To Mismatching LDAP User Password

If DPS is deployed using a pre-existing user and if the password in LDAP for that user does not match the password used for DPS setup, then the Verify Deployment stage may fail with an error. The deployment halts, prompting for further input:

Example

## Progress: 94
#### stage: dps: Verify Deployment
### ERROR FOUND ###
Failed to login: CalledProcessError('Command \'. /etc/profile.d/modules.sh && module load cm-dpsctl && dpsctl
--host=api.dps --port=443 --insecure-tls-skip-verify --credentials-path /cm/local/apps/dps/etc/creds.yaml
login\' returned exit code EX_ERROR<1(0x01)> while expected (<ExitCodes.EX_OK: 0>,)\n\tSTDOUT:\n{\n  "status":
{\n    "diag_msg": "Command execution failed with an error",\n...\n    "ok": false\n  }\n}\n\tSTDERR:\n')
Undo/Abort/Skip/Retry/Info/Debug/Remote debug: u/a/s/r/i/d/D:

The issue can be worked around by modifying the password (<dps-user-password>) used by the LDAP DPS user (<dps-user>) to match the password of the DPS configuration:

Example

root@basecm11:~# cmsh
[basecm11]% user
[basecm11->user]% use <dps-user>
[basecm11->user[<dps-user>]]% set password
enter new password:
retype new password:
[basecm11->user[<dps-user>]]% commit

A retry (typing r at the prompt) of the Deployment Verification stage should then succeed:

Example

## Progress: 89
#### stage: dps: Setup Slurm
## Progress: 94
#### stage: dps: Verify Deployment
### ERROR FOUND ###
Failed to login: CalledProcessError('Command \'. /etc/profile.d/modules.sh && module load cm-dpsctl && dpsctl
--host=api.dps --port=443 --insecure-tls-skip-verify --credentials-path /cm/local/apps/dps/etc/creds.yaml
login\' returned exit code EX_ERROR<1(0x01)> while expected (<ExitCodes.EX_OK: 0>,)\n\tSTDOUT:\n{\n  "status":
{\n    "diag_msg": "Command execution failed with an error",\n...\n    "ok": false\n  }\n}\n\tSTDERR:\n')
Undo/Abort/Skip/Retry/Info/Debug/Remote debug: u/a/s/r/i/d/D:r
Retrying the stage 'Verify Deployment'.
#### stage: dps: Verify Deployment
## Progress: 100
Took:     01:18 min.
Progress: 100/100
################### Finished execution for 'CM NVIDIA Mission Control Setup', status: completed
CM NVIDIA Mission Control Setup finished!