Integrating with Slurm#

Overview#

Integration with HPC workload managers like Slurm can be achieved through a scheduler prolog/epilog configuration. Each HPC job will be represented in DPS as a resource group. The prolog handles creation of the resource group with the selected DPS policy bundle and resource-group settings such as WPPS. The epilog handles the deletion of the resource group and restoring any default policies. If DPS detects insufficient power to start the job at the requested settings it may make adjustments or return a failure.

Example Slurm Integration#

You can use the PrologSlurmctld and EpilogSlurmctld parameters in the slurm.conf file.

With this configuration a job will be re-queued if the DPS prolog fails (returns a non-zero exit code). While there can be various reasons for this failure, in general DPS was unable to configure a corresponding resource group with the requested power settings and the job should not run.

#slurm.conf
PrologSlurmctld=/usr/share/dps/prolog.sh
EpilogSlurmctld=/usr/share/dps/epilog.sh

Example Prolog script#

The prolog script runs before job execution and:

  1. Creates a resource group with the job ID as the external identifier

  2. Adds allocated compute nodes to the resource group

  3. Activates the effective policies from the selected bundle

#!/bin/bash
# Create DPS resource group for power management
JOB_NAME=${SLURM_JOB_NAME}
JOB_ID=${SLURM_JOB_ID}
NODES=${SLURM_JOB_NODELIST}

# Convert Slurm nodelist to comma-separated format
NODE_LIST=$(scontrol show hostname ${NODES} | tr '\n' ',' | sed 's/,$//')

# Create resource group
dpsctl resource-group create \
  --resource-group "${JOB_NAME}" \
  --external-id ${JOB_ID} \
  --policy-bundle MaxP

# Add allocated nodes
dpsctl resource-group add \
  --resource-group "${JOB_NAME}" \
  --entities "${NODE_LIST}"

# Activate power policies
dpsctl resource-group activate \
  --resource-group "${JOB_NAME}" \
  --sync

Example Epilog Script#

The epilog script runs after job completion and deletes the resource group, returning devices to their original power configuration set in the topology.

#!/bin/bash
# Slurm Epilog - Clean up resource group

JOB_NAME=${SLURM_JOB_NAME}

# Delete resource group (automatically deactivates)
dpsctl resource-group delete \
  --resource-group "${JOB_NAME}"

Job Comment Integration#

The following dps_* comment keys are a site-defined convention for a custom prolog; they are not built-in Slurm fields or dpsctl flags. Keep dps_policy only when existing submitters and the prolog already share that schema. The prolog must map its value to an installed policy bundle and pass that name through dpsctl resource-group create --policy-bundle.

A site-specific prolog can parse DPS settings from job comments:

# Submit job with DPS settings in comment
sbatch --comment="dps_policy:MaxP,dps_prs:false,dps_dpm:true" job_script.sh

The prolog can map these site-defined values:

  • dps_policy:<string> - Site-defined policy-bundle selection consumed by the prolog

  • dps_prs:<bool> - Disable Power Reservation Steering

  • dps_dpm:<bool> - Enables Dynamic Power Management

  • dps_wpps:<comma-separated ints> - Sets workload profile IDs

Troubleshooting#

See the Slurm documentation for information on Prolog and Epilog.

The DPS prolog/epilog scripts log to STDOUT by default.

Ensure that dpsctl has been properly configured for authentication and authorization.

Integration with Workload Managers#

Slurm Integration Compliance#

When integrated with Slurm, DPS compliance works seamlessly:

  1. Job Submission: User submits job with power requirements

  2. Prolog Execution: Slurm prolog script calls DPS to create resource group

  3. Compliance Check: DPS validates power compliance for allocated nodes

  4. Allocation Adjustment: If allowed, DPS reduces the allocation within the effective bundle limits

  5. Job Execution: Job runs with compliant power configuration

  6. Epilog Cleanup: Slurm epilog script removes resource group, freeing power budget

This integration ensures that computational workloads never compromise power infrastructure integrity while maximizing available performance within compliance constraints.