Run the MaxLPS Power Pilot#

Start Here#

Use this procedure after reading MaxLPS Power Pilot. Run the same mix of at least two representative inference jobs—one low-power-utilization and one high-power-utilization—on GB200 NVL72 or GB300 NVL72 racks first without, then with, DPS fleet power management. Keep fleet-wide demand and client behavior the same for both runs.

Pilot goal: determine whether four managed racks deliver more aggregate inference throughput inside the same facility power envelope as a three-rack baseline, while maintaining power compliance.

For the MaxLPS operating model and the scheduler-independent configuration used in this runbook, refer to MaxLPS Power Management.

Detailed Procedure#

Complete prerequisites and deploy DPS — Confirm the pilot site, deploy DPS, install and authenticate dpsctl, then verify the deployment (Step 1).
Set up metrics collection — Configure and validate workload, power, GPU-limit, and BMC telemetry before creating the topology or running the baseline. Retain raw data for both comparisons (Metrics setup).
Create a topology and run a BMC health check — Import devices if needed, create and activate the 4-rack topology with a single PowerDomain budget, check every BMC, then deactivate the topology for the baseline (Step 3).
Prepare the workload and comparison controls — Stage the low- and high-power inference jobs, fix fleet-wide QPS or concurrency and client behavior, then define the steady-state window, KPIs, and evidence package (Step 4).
Run the baseline — Execute the workload mix without DPS in the control path and retain the baseline evidence (Step 5).
Enable MaxLPS — Reactivate the topology, select the default MaxLPS policy bundle, then create and activate the resource group with dynamic power management, Power Steering, and Shared GPU (Step 6).
Run the MaxLPS test — Repeat the same workload mix and fleet-wide QPS or concurrency across 4 racks under the same envelope with DPS enforcing the topology budget (Step 7).
Compare results — Align metrics windows, quantify throughput uplift, and check for envelope breaches or throttling (Step 8).

Step 1: Complete Prerequisites and Deploy DPS#

Back to the five-step pilot overview.

Confirm Prerequisites#

Scheduler-independent configuration#

Here scheduling and provisioning means workload schedulers (Slurm, Kubernetes, and similar job or pod placement layers) and provisioning systems (cluster installers, fleet managers, automation that registers, scales, or re-images nodes)—everything that normally owns where nodes run and which hardware is in service—not DPS’s power plane.

DPS is designed to follow those layers when you want it to: workload schedulers and provisioning systems can drive resource group lifecycle and policy through the nvidia.dcpower.v1 resource group API and the matching dpsctl resource-group commands so membership stays aligned with jobs, reservations, or capacity changes. Managing Resource Groups documents that integrated style of operation step by step.

You can still pursue the MaxLPS pattern without that coupling by defining one or more resource groups whose members you manage outside both workload schedulers and provisioning systems—fixed topology entities, dpsctl resource-group create / add / activate only—while inference continues to run through your existing scheduling and provisioning unchanged. Scheduler-independent configuration, in this runbook, is that decoupled path: the smallest setup that enforces a shared GPU power budget across the fleet while scheduling and provisioning never call DPS to add or remove nodes for you.

That configuration consists of:

A simplified topology that models the 4 racks and the operating power domain above them.
A single resource group containing every compute node in the topology, created with three capabilities enabled together:
DPM (Dynamic Power Management) — DPS actively enforces per-node power policy.
PRS (Power Reservation Steering) — DPS’s mechanism that redistributes headroom across GPUs based on live telemetry. Refer to Power Reservation Steering.
Shared GPU — the resource group’s GPU budget is enforced as a group sum rather than per-node, which is what makes operating more racks within one envelope possible. Refer to Shared GPU Budget.

If any one of these three is missing, the pilot configuration described here is incomplete and the comparative results in this runbook will not be reproducible. Creating these conditions is covered in the following sections.

What you will do#

Validate the deployment — Authenticate, run dpsctl verify, and interpret failures so you are not debugging topology on a broken server (Part 1).
Model the site and gate end-to-end BMC health — Import devices if needed, build the 4-rack topology with a single PowerDomain budget, run the deep BMC health check against every node, and activate the topology once as a temporary DPS-to-BMC end-to-end gate (Part 2).
Deactivate before the baseline — After the BMC health and activation gates in Part 2 pass, deactivate the topology so the unmanaged baseline runs with DPS out of the control path (Part 3).
Prepare the baseline run — Stage inference consistently, define the steady-state window and KPIs, and document the 3-rack footprint and power context (Part 4).
Run the unmanaged baseline — Execute the workload without DPS in the control path and capture baseline metrics (Part 5).
Enable fleet power policy — Re-activate the topology, then create and activate the resource group with the shipped MaxLPS bundle and DPM, PRS, and Shared GPU (Part 6).
Run the managed comparison — Repeat the same inference profile across 4 racks under the same envelope with DPS enforcing the topology budget (Part 7).
Compare and decide — Align metrics windows, quantify throughput uplift, and watch for envelope breaches or throttling (Part 8).

Prerequisites#

Before starting this runbook, confirm each of the following:

Hardware: Four GB200 NVL72 or GB300 NVL72 racks identified for the pilot. All 18 compute nodes per rack must be powered on and BMC-reachable.
BMC preparation for managed DPS: Before you import the topology, complete the Redfish compatibility, firmware, client-capacity, access, and credential prerequisites in BMC Readiness and Health Guide. Step 3 runs the BMC health gate after import and before the managed run.
Inference workload and dataset staged on every node in all 4 racks, launchable from your job scheduler or equivalent orchestration.
PowerDomain budget for the pilot is known and documented — this runbook uses the same 3-rack operating envelope as the baseline.

Deploy DPS and Install `dpsctl`#

Deploy the DPS 0.9.x server to a Kubernetes cluster with the Deployment and Configuration Guide, then install a matching 0.9 dpsctl client from DPS on NGC. For this MaxLPS pilot, deploy a DPS Agent with mTLS before you import the topology. The agent must be ready and able to reach every assigned BMC or Redfish proxy. Set up the BMC access and credentials as described in BMC Readiness and Health Guide.

Deploy the Pilot Agent#

Issue an agent certificate whose common name is maxlps-pilot from a CA that the DPS server trusts. Create the Secret in the DPS namespace with the CA, certificate, and private-key files. Do not put private keys in the Helm values file or topology JSON.

kubectl create secret generic dps-agent-mtls \
  --namespace dps \
  --from-file=ca.crt=<path-to-agent-ca-certificate> \
  --from-file=tls.crt=<path-to-maxlps-pilot-certificate> \
  --from-file=tls.key=<path-to-maxlps-pilot-private-key>

Configure the DPS server with TLS or mTLS before enabling the agent, and ensure that DPS trusts the agent certificate chain. Then add the following settings to the Helm values file used for the DPS deployment. This pilot uses maxlps-pilot as its agent ID. The certificate common name is allowed separately in dps.grpcServiceMTLSCN.

dps:
  agent:
    allowHandshakeFallback: false
  grpcServiceMTLSCN:
    nvidia.dcpower.v1.DPSAgentService:
      - "maxlps-pilot"

agent:
  enabled: true
  replicas: 1
  agentID: "maxlps-pilot"
  tls:
    transportSecurity: "mtls"
    existingSecret: "dps-agent-mtls"

Set DPS_RELEASE to the selected 0.9 chart version. The following example uses the dps release name and namespace.

Do not continue until the Agent Pod is Ready. If it does not become Ready, inspect the certificate chain, certificate common name, DPS server trust configuration, and Agent-to-BMC or proxy network path.

Install and authenticate dpsctl as described in Installing dpsctl. The client must be able to reach the DPS server.

Note: If you do not have the physical hardware to run this pilot but still want to try DPS, you can deploy the DPS SDK Simulator to a VM to explore DPS in a simulated environment.

Verify DPS Deployment#

Before you configure a topology, confirm the deployment is healthy end-to-end. Treat this as a gate. If any component is unhealthy, stop and fix it before moving to Step 3.

Note: If you have not already, export a few dpsctl environment variables to keep the commands that follow concise:
export DPSCTL_HOST="api.dps.your-domain.com"
export DPSCTL_PORT="443"

For a production pilot, use a certificate that the client trusts. When the endpoint uses a private CA, set its certificate path:

export DPSCTL_CA_CERT_PATH="/path/to/ca-certificate.pem"

Do not set DPSCTL_INSECURE_TLS_SKIP_VERIFY for a production endpoint.

Use DPSCTL_INSECURE=true only for local bring-up against a deliberately plaintext endpoint. DPSCTL_INSECURE_TLS_SKIP_VERIFY permits an encrypted connection without certificate verification; it does not configure a plaintext connection.

Authenticate with `dpsctl login`#

Before any other dpsctl command will succeed, authenticate against the DPS server. The token is cached under ~/.dpsctl/credentials.yaml and reused by subsequent commands until it expires.

dpsctl login -u <username>

dpsctl login prompts for the password interactively, or reads DPSCTL_USERNAME and DPSCTL_PASSWORD from the environment if set. See Install dpsctl for the one-time install and initial configuration.

Confirm Server and Client Versions#

Confirm authentication succeeded and capture the server version:

dpsctl server-version

A successful call returns the server’s version payload. Confirm the server version matches the latest DPS release available on NGC.

Confirm the local dpsctl client matches the server version:

dpsctl --version

Confirm this reports the same version as the server. An AUTHENTICATION_FAILED or missing token error from dpsctl server-version means the previous dpsctl login did not complete — rerun it before continuing.

Run the Verify Gate#

Proceed to Step 3 only when status.ok is true and no component reports healthy: false. A skipped: true component is acceptable — it means that component is not part of this deployment.

If BCM is intentionally not part of the deployment, its result does not block this pilot. Confirm the other components are healthy and continue. Refer to Deployment Component Unhealthy for the no-BCM output and to Troubleshoot the MaxLPS power pilot for the full component list.

Run dpsctl verify with no component flags. It checks the DPS server, database, authentication service, UI ingress, and BCM connectivity in a single call:

dpsctl verify

Each component (dps_server, database, auth, ui, bcm) returns healthy, message, and skipped. The details field is informational and is mainly useful for troubleshooting.

Troubleshooting: If deployment verification fails, see Why Does DPS Deployment Verification Fail?.

Step 2: Set Up Metrics Collection#

Configure and validate metrics before creating the topology or starting either comparison run. This avoids discovering after the baseline that a required signal is missing or clocks cannot be aligned.

Collect the following signals with timestamps from the same time source:

workload demand, throughput, token rate, latency percentiles, and errors;
per-GPU power, utilization, and observed power limits; use Collect Pilot Telemetry: 1-Second GPU and CPU Data with DCGM Exporter to set up and validate the DCGM capture;
BMC health and latency, plus rack or facility power for the same boundary as the pilot envelope; and
the DPS Prometheus metrics used to verify GPU-cap movement during the managed run. Configure the scrape before the managed run as described in Enable DPS Prometheus Metrics.

Run a short test capture before creating the topology. Confirm every required signal has a timestamp, expected units, and a known retention location. Record the collector versions, sampling intervals, clock source, and any offsets in the evidence package.

Step 3: Create a Topology and Run a BMC Health Check#

Back to the five-step pilot overview.

To manage power effectively, DPS needs to know the structure of the data center in which it is operating. We call this the data center topology. DPS also needs to know of any custom devices it will encounter in that topology.

Import Device Specifications (Optional)#

DPS ships with a set of supported device specifications out of the box. The supported devices include, at time of writing:

Compute: DGX_GB200, DGX_GB300, DGX_B200, DGX_B300, DGX_H100
GPUs: GB200, GB300, B200, B300, H100
CPUs: Grace, Grace_GB300
Power infrastructure: FloorPDU95 (Generic Floor PDU with 95% efficiency factor) , RackPDU95_57500W (Generic Rack PDU with 95% efficiency factor and max load of 57,000 W), PowerSupply95_3300W (Generic Power Supply with 95% efficiency factor and max load of 3,300 W)

GB300 GPU minimum power (optional). minLoadWatts in the GB300 GPU entry of devices.yaml is the lower power bound that Dynamic Power Management can request. Do not derive it from a workload average. Change it only when the BMC-supported minimum differs from the shipped model, and confirm that minimum on a representative system before dpsctl device upsert. A value below the supported minimum can make DPS request a limit that the BMC rejects. A value above the supported minimum can prevent expected power reductions. Keep maxLoadWatts above minLoadWatts, record any approved override, and re-import it per Managing Devices.

If your data center uses PDUs, PSUs, or other equipment whose model does not match the list above, you must register a specification for it before it can appear in a topology. Follow the full procedure in Managing Devices, then import your additions:

dpsctl device upsert devices.yaml

Verify Device Specifications#

Confirm that every model you reference in your topology appears in the output:

dpsctl device list

Define the Topology and Static Load#

The topology is how you describe your power-distribution network to DPS. For this runbook, the topology must contain at minimum:

Entity type	Count	Purpose
`PowerDomain`	1	Top-level 3-rack operating envelope applied across all 4 racks.
`PowerDistribution`	4 (one per rack)	Rack PDU.
`ComputerSystem`	72 (18 per rack)	GB200 or GB300 compute nodes, parented under their rack PDU.

The PowerDomain’s PowerValue is the top-level operating envelope DPS enforces across the 4-rack pilot topology. Set it to the same 3-rack envelope used by the baseline: 375 kW for GB200 NVL72 or 405 kW for GB300 NVL72.

Each GB300 NVL72 rack contains 9 NVSwitch trays. Their draw is not managed by DPS, but it must still be reflected in the rack PDU and PowerDomain budget.

The runbook represents the per-rack NVSwitch aggregate with the StaticLoad field on each FloorPDU-Rack0X (PowerDistribution) entity. StaticLoad declares fixed power consumption from unmanaged devices and is propagated up the power-distribution network automatically—see Entities → Static Load.

Use 7,416 W per rack for a standard GB300 NVL72 9-switch aggregate (replace with measured or vendor-specified values when available). The example topology in this step applies this value on every rack PDU. For a standard GB200 NVL72 rack, we recommend a 9-switch aggregate power value of 11,160 W per rack.

Prepare the Topology File#

Download the reference topology that matches your hardware:

GB300 topology: maxlps-pilot-gb300.json — 4 GB300 NVL72 racks, 72 DGX_GB300 nodes, 4 FloorPDU95 rack PDUs at 135 kW (StaticLoad 7,416 W), and a PDU-Root PowerDomain at the 3-rack envelope of 405 kW.
GB200 topology: maxlps-pilot-gb200.json — 4 GB200 NVL72 racks, 72 DGX_GB200 nodes, 4 FloorPDU95 rack PDUs at 125 kW (StaticLoad 11,160 W), and a PDU-Root PowerDomain at the 3-rack envelope of 375 kW.

These DPS 0.9 topology files intentionally contain no Policies section. DPS 0.9 rejects legacy topology policies; bind the default MaxLPS policy bundle when you create the resource group in Step 6.

Replace the BMC URL and SecretName values with those for your fleet before importing. The downloadable GB200 and GB300 examples assign the pilot agent at PDU-Root, so all child devices inherit AgentId: maxlps-pilot. Replace this value only if it matches the deployed agent’s agent.agentID. Configure the certificate common name separately in dps.grpcServiceMTLSCN. Refer to BMC Readiness and Health Guide for BMC credential mapping and proxy guidance.

Set TOPOLOGY_FILE to the file you downloaded:

TOPOLOGY_FILE=maxlps-pilot-gb300.json # Use maxlps-pilot-gb200.json for GB200.

For additional topology examples, refer to Import a Topology and Managing Topologies.

Validate and Import the Topology#

The topology JSON cannot carry a typed bundle selection and must not contain a top-level Policies array. During import, the DPS 0.9 server resolves the current global_policy_bundle and persists that selection on the new topology. The later resource-group step binds the shipped MaxLPS bundle with --policy-bundle; it does not read the topology’s stored bundle. Refer to Import a Topology.

If you want the imported topology’s policy_bundle to be MaxLPS, confirm settings.global_policy_bundle with dpsctl settings list before import. Do not change the global default unless it is empty or points at a different bundle. If you do change it for import, restore the recorded value immediately after the import command returns, even if import failed.

dpsctl topology validate "$TOPOLOGY_FILE"
dpsctl topology import "$TOPOLOGY_FILE"

These commands only register configuration, topology, and inventory. No policy is applied to a BMC until topology activation.

Troubleshooting: If topology validation or import fails, see Why Does Topology Validation or Import Fail?.

Activate the Topology Before Checking BMCs#

Before activating, run the deep BMC prerequisite health check against every BMC in the imported pilot topology. This check is broader than a reachability ping: it validates Redfish endpoint reachability, firmware inventory, power-limit write and read-back behavior, telemetry sampling, GPU power-policy state, and in-band/out-of-band power drift across every node. Passing --topology runs the check against every node in that topology in a single call. The topology only needs to be imported — not active — for this check to run. Complete the write-safety preflight in Run a BMC Health Check before you use --force-writes; write mode changes power limits temporarily and can disable active WPPS profiles.

On large clusters, the probe can take hours. dpsctl verify bmc-health start starts the check asynchronously and returns a task_id. The server task continues after dpsctl exits, so retain the ID.

First, run and retain the full read-only baseline report. Resolve any failures before you continue:

dpsctl verify bmc-health start --topology maxlps-pilot --skip-writes --expected-edpp-pct 100 --samples-per-telemetry 2500 --telemetry-interval 500ms --per-node-timeout 1h
dpsctl verify bmc-health report <read-only-task-id> --wait --output json > bmc-health-maxlps-pilot-read-only.json

Next, validate write behavior on one representative node with one in-flight node probe. Confirm that the report passes before you continue:

dpsctl verify bmc-health start --topology maxlps-pilot --nodes <node-name> --force-writes --concurrency 1 --expected-edpp-pct 100 --samples-per-telemetry 2500 --telemetry-interval 500ms --per-node-timeout 1h
dpsctl verify bmc-health report <node-write-task-id> --wait --output json > bmc-health-write-probe.json

After both checks pass, start the fleet-wide write-enabled check and query its status:

dpsctl verify bmc-health start --topology maxlps-pilot --force-writes --expected-edpp-pct 100 --samples-per-telemetry 2500 --telemetry-interval 500ms --per-node-timeout 1h
# Query the status until the health check is done
dpsctl verify bmc-health status <fleet-write-task-id>
# Once done, generate a report
dpsctl verify bmc-health report <fleet-write-task-id> --wait --output json > bmc-health-maxlps-pilot.json
dpsctl verify bmc-health report <fleet-write-task-id> --summary-only

Step 7: Review BMC health results#

This run is also where you collect BMC latency metrics for the pilot. From bmc-health-maxlps-pilot.json, capture the following latency summaries for every node:

endpoint_latency_ms
telemetry_latency_ms
power_get_latency_ms
power_set_latency_ms
edpp_get_latency_ms
wpps_get_latency_ms

For MaxLPS readiness, target p50 below 0.5 seconds, p95 below 0.7 seconds, p99 below 0.8 seconds, and max near 1 second for read and write paths. If p99 is greater than 1 to 2 seconds, document the latency measurements and assess cluster-wide BMC client connection pressure before activation.

Prepare other BMC clients before you run this check. Confirm the BMC client connection requirements from the prerequisites are still in place:

For GB200 and GB300 systems, the NVIDIA BMC and DPS teams recommend no more than four BMC client connections, including two DPS client connections.
Use session token authentication instead of basic authentication.
Use keep-alive to maintain the connection.

In the summary report generated in Step 6, confirm:

status.ok is true.
cluster_summary.total_nodes is 72, matching the node count in maxlps-pilot.
cluster_summary.failed is 0.
cluster_summary.nodes_unreachable is 0.
issues contains no SEVERITY_ERROR entries.

Known B200/B300 firmware-validation exception: on systems with B200 or B300 GPUs, DPS 0.8.x can report a FIRMWARE_VALIDATION_FAILED issue with message: firmware validation failed even when the BMC is usable. If this is the only SEVERITY_ERROR for the affected B200 or B300 node, cluster_summary.nodes_unreachable is still 0, and the rest of the health report is clean, document the exception and continue with the runbook.

If any of these do not hold, other than the known B200/B300 firmware-validation exception described here, the issues list identifies what to fix. Each entry includes the affected node, the resource within that node, an issue code, the observed value, the threshold it violated, and a human-readable message. Fix the affected nodes and re-run the health check until the report is clean before activating the topology.

Troubleshooting: For issue-specific remediation, see BMC Health Check.

Step 8: Activate the topology#

Activate the topology as a temporary end-to-end gate. Activation exercises the full DPS-to-BMC set-policy path against every node so any device-model or policy-application issue that the BMC health review in Step 7 could not catch surfaces now, at setup time, instead of immediately before the 4-rack managed run in Part 7. The topology will be deactivated in Part 3 so the unmanaged baseline in Part 5 runs with DPS out of the control path, and re-activated later in this runbook.

dpsctl topology activate --topology maxlps-pilot # Must match Topology.Name in the imported file.

Troubleshooting: If topology activation fails, see What If Topology Activation Fails?.

Step 9: Verify topology activation#

dpsctl topology activate --topology maxlps-pilot
dpsctl tp list --active

Confirm the topology appears, leaf_node_names lists all 72 compute nodes, and policy_bundle matches the global_policy_bundle that was in effect at import.

dpsctl check connection --topology maxlps-pilot

The check must succeed for every pilot node; otherwise stop and correct the agent ID assignment, agent-to-BMC network path, or BMC credentials before the health gate. For activation failures, refer to What If Topology Activation Fails?.

Complete the BMC Health Gate#

Run the full BMC health check against the active topology, review its latency profile, and resolve every error before continuing. Keep the topology active until the check finishes and you have saved its report. Refer to BMC Readiness and Health Guide for the procedure, slow-BMC guidance, and issue-specific recovery.

Deactivate the Topology for the Baseline#

After saving the health report, deactivate the topology to remove DPS from the baseline power-control path:

dpsctl topology deactivate --topology maxlps-pilot
dpsctl tp list --active

Confirm that no topology is active. Deactivation cleanup is best effort. On all 54 baseline compute nodes, check every GPU:

nvidia-smi --query-gpu=index,power.default_limit,power.limit --format=csv

Every GPU must report its default limit before you start the baseline. For limits that remain changed, follow the reset procedure in Step 3: Verify per-GPU caps returned to default, then recheck every affected GPU. Do not run either BMC command while the topology is inactive in 0.9. If you need another check, repeat the activation, check, and deactivation sequence before the baseline.

Step 4: Prepare the Workload and Comparison Controls#

Workload Requirements#

What “baseline” means here. The baseline run is the unmanaged reference. DPS is installed and running, but the imported topology and resource group are not active, so DPS does not enforce power policy on any node. The topology is deactivated after the BMC health check. Confirm that dpsctl tp list --active returns no rows and dpsctl rg list returns no resource groups before you launch the workload.

For the 3-rack baseline and the 4-rack DPS-managed run to be directly comparable, the workload mix must meet these requirements:

Each job is an inference workload. Training workloads, synthetic power stressors (gpu-burn, etc.), and pure benchmarks do not exhibit the bursty, non-sustained power profile that envelope-based inference pilots target and will produce misleading results.
It includes at least two complementary power profiles. Run at least one low-power-utilization and one high-power-utilization representative inference job.
Each job is representative of the production workload. Ideally it uses the actual model, framework, and request pattern (batch size, concurrency, input length distribution) you plan to run once the pilot succeeds.
It runs long enough to reach a steady-state comparison window. The “steady-state window” is the contiguous slice of the run that you and the 4-rack managed run will be compared on in Step 8. Identify it as follows, and apply the same procedure on every run:
- Run the workload mix for at least one hour total so the run is not dominated by ramp-up.
- Trim ramp-up and cool-down from the start and end of the run.
- Within the trimmed interval, require that both per-rack power draw and aggregate tokens/sec have visibly flattened on your time-series plots — no monotonic climb, no large step changes, only normal short-term workload jitter.
- The remaining window must be at least 30 minutes long and contiguous; if it is shorter or broken up, extend the run rather than relying on a noisy window.
- Record the start and end timestamps of this window and use exactly those timestamps when capturing every metric in What to Collect. The same procedure applies to the 4-rack managed run in Step 7 and to the comparison in Step 8.
The 3 pilot racks are under load in isolation. No other workloads should be sharing these racks or competing for the same facility power budget during the baseline window; otherwise the “3-rack baseline” no longer describes what the 3 racks alone can do.

Comparison Controls#

Fix the client-side controls before starting the baseline, and retain them for the managed run:

Fleet-wide demand. For open-loop testing, use the same aggregate QPS schedule across all load generators for each inference job. Do not change per-node QPS just because the managed run has more nodes. For closed-loop testing, use the same aggregate configured concurrency and record the configured concurrency for each generator and the total.
Client behavior. Keep request timeouts, retry and backoff behavior, cancellation policy, warm-up, total duration, and stop conditions the same.
Workload placement. Record the job-to-node placement and the nodes under load. Do not add unrelated workload to the pilot racks during either steady-state window.

What to Collect#

Before you run the baseline inference test, capture the following for the duration of the baseline run. You will compare these signals against the 4-rack managed run in Step 7 and interpret them using Step 8:

Aggregate workload throughput — requests/sec, queries/sec, or whichever KPI matches the workload.
Aggregate token throughput (tokens per second across the cluster) and latency percentiles (refer to Step 8 for the KPI set).
Workload metadata and configuration - application, model, ISL/OSL, and any other relevant configuration parameters.
Per-rack and aggregate power draw, sampled at a consistent interval. Keep both the time series and a steady-state average.
Wall-clock duration of the run and of the steady-state window within it.
Per-GPU power draw distribution across the 3 racks, if available — useful for showing how headroom is being left on the table by static provisioning.
EDPp settings per node. The same EDPp setting should be used across the entire fleet.
BMC health report and latency profile — retain the full JSON report from Step 3, including node-level read, telemetry, and power-write latencies.

Retain an Evidence Package for Every Run#

For both baseline and managed runs, retain an immutable package that includes:

the workload manifest or launch command, image and model version, dataset or prompt profile, batching, aggregate QPS or configured concurrency, load generator allocation, and client timeout, retry, and stop settings;
raw workload, power, GPU-limit, and BMC telemetry series; their UTC run and steady-state bounds; and the clock source and any known offsets;
topology and resource-group state, active node lists, DPS and dpsctl versions, deployment or Helm values relevant to Power Steering, and the BMC health report; and
the source used for every aggregate in the comparison.

Do not put BMC credentials, tokens, or other secrets in this package. Redact hostnames and endpoint details where required by your site’s retention policy.

Use whatever stack you already trust for time-series capture (BMC or rack telemetry, DCGM or vendor exporters, inference server metrics, facility BMS). The important part is consistent sampling, aligned clocks, and the same definitions on baseline and managed runs.

The baseline run establishes the reference numbers the 4-rack DPS-managed run will be compared against. It is run without DPS in the control path, after deactivating the topology used for health checks and before creating a resource group.

This runbook does not prescribe how you launch the workload. Whatever you already use to orchestrate AI inference on these racks — Slurm, a Kubernetes inference operator, a Triton deployment, your own job runner — is exactly what you should use here. DPS cares about the power side of the run, not the scheduling side.

Step 5: Run the Baseline#

Launch the workload mix across 3 of the 4 pilot racks (54 compute nodes) using your existing orchestration tooling. DPS should not be in the control path during this run.

Record the number of compute nodes that are actually under load during the baseline run. You will compare that footprint with the 4-rack managed run in Part 8.

The topology remains imported but inactive, and no resource group exists yet, so DPS is not in the control path during this baseline run.

Step 6: Enable MaxLPS#

Back to the five-step pilot overview.

Confirm Power Steering Settings#

Before enabling MaxLPS, confirm that the Helm values used for the deployment enable dynamic power management and Power Steering:

dps:
  prs:
    enabled: true
power-steering:
  enabled: true

Both values default to true in DPS 0.9. Keep them explicit in the pilot overlay so a chart customization cannot omit the Power Steering controller.

Reactivate the Topology#

The topology passed BMC health checks and was deactivated for the baseline. Reactivate it now before creating the resource group:

dpsctl topology activate --topology maxlps-pilot

Troubleshooting: If topology activation fails, see What If Topology Activation Fails?.

Verify the Topology Is Active#

dpsctl tp list --active

Confirm maxlps-pilot appears as the active topology and its policy_bundle matches the global_policy_bundle that was in effect at import.

Select the Default MaxLPS Policy Bundle#

Use the default MaxLPS policy bundle. Do not create a pilot-specific policy bundle. The bundle contains model-scoped limits for dynamic power management and Power Steering on the supported systems; its GPU values are aggregate watts for all GPUs in one ComputerSystem, not per-GPU watt values.

Confirm the default bundle is available before creating the resource group:

dpsctl policy bundle list

Confirm that the output lists MaxLPS. If it does not, stop and reconcile the DPS installation before continuing. Do not replace the default bundle with a pilot-specific bundle.

Bind the shipped MaxLPS bundle on the resource group. Do not create a custom percentage bundle for this pilot. Confirm the name with dpsctl policy bundle list before you create the group.

MaxLPS does not cap each node at a fixed fraction of MaxP. It sets a per-model operating band and an aggregate GPU target below the GPU ceiling: for DGX_GB300, a 100% node ceiling with a 4,108 W GPU target against a 5,600 W GPU ceiling, and for DGX_GB200, a 100% node ceiling with a 3,648 W GPU target against a 4,800 W ceiling.

The 3-rack envelope is held by the topology instead. The PDU-Root PowerDomain limit set in Part 2 bounds all four racks, and DPM, PRS, and the shared-GPU budget redistribute power under that bound as demand moves. That is what lets 72 managed nodes run inside an envelope sized for 54: DPS admits nodes at their effective policy floor when the target does not fit, then steers power toward the nodes actually under load. Expect activation to report some nodes below their policy target — that is the mechanism working, not a failure. Refer to What If Resource Group Activation Fails?.

Create the Resource Group#

Create one resource group containing every compute node in the topology, with dynamic power management, Power Steering, and Shared GPU all enabled. The --shared-gpu-enable flag is what binds the resource group’s GPU power consumption to a shared, group-wide budget rather than per-node caps. Set --policy-bundle to the default MaxLPS bundle selected in Step 3.

dpsctl resource-group create \
  --resource-group maxlps-pilot \
  --external-id 1 \
  --dpm-enable \
  --prs-enabled \
  --shared-gpu-enable \
  --policy-bundle MaxLPS

Use MaxLPS for both GB200 and GB300 pilots.

Troubleshooting: If resource group creation fails, see What If Resource Group Creation Fails?.

--external-id is required and must be a 64-bit integer. It is the handle a workload scheduler or provisioning system would normally use to correlate this resource group with one of its own records. In the scheduler-independent configuration this runbook describes, no external system owns the resource group — pick any positive integer and keep a note of it for your own records. DPS does not enforce uniqueness on this value, but using a distinct ID per resource group makes dpsctl rg list output easier to read once you have more than one.

Add Every Compute Node#

Add every compute node in the topology to the resource group. Substitute your real node names for gb300-nvl-001-compute01,gb300-nvl-001-compute02,...:

dpsctl resource-group add \
  --resource-group maxlps-pilot \
  --entities "gb300-nvl-001-compute01,gb300-nvl-001-compute02,gb300-nvl-001-compute03..."

See dpsctl resource-group add for additional flags (partial-update behavior, strict policy, etc.).

Troubleshooting: If adding resources fails, see What If I Cannot Add Nodes to the Resource Group?.

Activate the Resource Group#

Activate the resource group. Use --sync so the command blocks until every node has its policy applied, --strict-policy so an infeasible target does not fall back to its effective floor, and --allow-reprovision=false to prevent the pilot from changing any other resource group:

dpsctl resource-group activate \
  --resource-group maxlps-pilot \
  --strict-policy \
  --allow-reprovision=false \
  --sync

See dpsctl resource-group activate for the full activation flag set, including strict-policy and partial-timeout options.

Verify Resource Group Activation#

dpsctl rg list --active

Confirm the maxlps-pilot resource group appears with:

activation_status: active
dpm_enable: true
prs_enabled: true
shared_gpu_enable: true
resource_names listing all 72 compute nodes

Troubleshooting: If resource group activation fails, see What If Resource Group Activation Fails?.

Verify Power Steering Readiness#

After the resource group is active, confirm the Power Steering controller is Ready and has no reconciliation errors. The 0.9 Power Steering 2.1 controller does not use older controller log text as a stable interface, so controller readiness and reconciliation health are the acceptance evidence for this step.

kubectl get pods -n dps \
  -l app.kubernetes.io/component=controller,app.kubernetes.io/part-of=power-steering
kubectl logs -n dps \
  -l app.kubernetes.io/component=controller,app.kubernetes.io/part-of=power-steering \
  -c controller --since=10m

Troubleshooting: If the Power Steering controller is not Ready or reports errors, refer to How Do I Check Power Steering?.

Step 7: Run the MaxLPS Test#

Back to the five-step pilot overview.

The only intended changes between Step 5 and this step are the number of racks under load and that DPS is now actively managing their power. Workload mix, model, framework, batching, fleet-wide demand, client behavior, controller timing, and the operating envelope are unchanged. If anything else differs, the comparison in Step 8 is invalid.

The two runs differ on exactly two dimensions:

Dimension	Baseline (Step 5)	Managed (Step 7)
Racks Under Load	3 racks (54 compute nodes)	4 racks (72 compute nodes)
DPS Control Path	Not in control path; no topology and no resource group active	In control path; topology `maxlps-pilot` active and resource group `maxlps-pilot` active
Operating Power Envelope	Same	Same
Workload, Model, Framework, and Batching	Same	Same
Fleet-wide Demand	Same aggregate QPS schedule (open loop) or configured concurrency (closed loop)	Same
Client Behavior	Same timeout, retry, cancellation, warm-up, duration, and stop rules	Same
Power Steering Timing	Not controlling power	Same Power Steering scheduler interval recorded for the comparison

The Operating Power Envelope row is held by the topology in the managed run: the active PDU-Root PowerDomain already sets the same 3-rack operating limit used by the baseline, and DPS enforces that limit across all 4 racks through the resource group’s shared-GPU policy.

With the topology active and the pilot resource group (maxlps-pilot) in place—created and maintained without integration to your workload schedulers or provisioning systems—re-run the same workload mix you used for the 3-rack baseline—same jobs, model, framework, and request pattern—but now across all 4 racks (72 nodes), with DPS in the control path.

As in Step 5, this runbook does not prescribe how you launch the workload mix. Use the same orchestration tooling (Slurm, Kubernetes inference operator, Triton, your own job runner) you used for the baseline. The only thing that changes between the two runs is the number of racks under load and whether DPS is enforcing power policy. Everything else must stay the same so the comparison is meaningful.

Workload Requirements#

The 4-rack run must be directly comparable to the 3-rack baseline. In addition to the baseline’s requirements:

It is the same workload mix as the baseline. Run the same low-power-utilization and high-power-utilization inference jobs with the same model, framework, batch size, and input distribution. If anything about the workload mix changes, the throughput numbers are no longer comparable.
It uses the same fleet-wide demand and client behavior. Use the aggregate QPS schedule or aggregate configured concurrency, timeout, retry, cancellation, warm-up, duration, and stop rules recorded under Comparison controls. Do not compensate for the larger fleet by changing per-node QPS.
All 4 pilot racks are under load in isolation. No other workloads should be sharing these racks or competing for the same facility power budget during the run. The same operating envelope that constrained the 3-rack baseline should constrain the 4-rack run.
DPS is in the control path. Confirm the pilot topology from Step 3 and the active resource group from Step 6 are in place (dpsctl tp list, dpsctl rg list) and that a read-only BMC health recheck still passes (dpsctl verify bmc-health start --topology maxlps-pilot --skip-writes --wait --summary-only) before launching the workload.
It produces a comparable steady-state window. DPS’s per-GPU caps are set dynamically from telemetry, so the run needs enough duration for the control loop to settle into steady-state redistribution, not just the workload itself. Apply the same steady-state window procedure defined in Step 4 under “Workload Requirements”, and make the resulting window at least as long as the one used for the 3-rack baseline.

Enable DPS Prometheus Metrics#

In DPS 0.9, dps-server exposes Prometheus text metrics at /metrics on the dedicated <release>-server-metrics Service (for example, dps-server-metrics) at port 9090, with service port name dps-metrics. The direct listener uses HTTP and is separate from the main HTTP API Service. Enable scraping for the pilot so the DPS-side signals called out in What to Record — per-GPU cap movement and group-sum allocation versus observed aggregate draw — land in the same time-series store as the workload and BMC telemetry you captured in Step 4, and can be cross-checked against the steady-state window with PromQL. The chart-managed ServiceMonitor (dps.serviceMonitor.enabled: true) is the recommended path for in-cluster Prometheus and is applied through the Helm upgrade flow in the Deployment and Configuration Guide; refer to Prometheus Monitoring for the full setup, including external Prometheus and protected-metrics configurations. If you enable dps.metrics.authentication, it requires dps.auth.oidc.enabled=true and a complete OAuth2 or authorization profile for the ServiceMonitor with the required dps:metrics:read scope.

Note: Land this Helm change once, before the 4-rack managed run starts so the steady-state window in What to record is fully covered by Prometheus samples.

Run the Workload#

Launch the workload mix across all 4 pilot racks (72 compute nodes) using the same orchestration tooling and submission flow you used for the 3-rack baseline. DPS is now in the control path and will actively redistribute per-GPU power headroom across the resource group in response to live BMC telemetry.

Verify GPU Limit Movement#

After the workload reaches steady state, confirm that the BMC or DCGM limit time series shows at least one GPU limit change while aggregate power remains within the operating envelope. Do not mark the pilot healthy solely because a pod is Running or a legacy log phrase appears. If limits do not move, refer to How Do I Check Power Steering?.

What to Record#

Capture the same metrics you captured for the 3-rack baseline, over a comparable steady-state window, so the two runs can be compared directly. Additionally, capture the DPS-managed run observations below.

Additionally, specific to the DPS-managed run:

Per-GPU cap movement during the run — the per-GPU power_max that DPS is actively setting, sampled over time. This demonstrates DPS redistributing budget live, rather than statically provisioning.
GPU power allocations vs. observed aggregate draw — confirms the group sum is being enforced rather than per-node caps leaving headroom on the table.
DPS-provided Prometheus metrics — once scraping is enabled per Enable DPS Prometheus metrics, capture the dedicated DPS metrics service’s /metrics series for the same steady-state window so per-GPU cap movement and group-sum enforcement can be cross-checked against PromQL.

To sample per-GPU caps and utilization over time, we recommend using DCGM on a representative node set on a fixed interval (for example every 60 seconds via a simple loop or your automation), or export the same fields from your existing BMC/telemetry pipeline. Store set_limit / max_limit alongside observed GPU power so cap movement can be plotted against workload phase.

Step 8: Compare Results#

Compare the raw captures from Step 4, Step 5, and Step 7. Compare runs only on their steady-state windows — at minimum 1 hour total run time, ramp-up and cool-down trimmed, both per-rack power and aggregate tokens/sec visibly flattened, contiguous window of at least 30 minutes, and the exact start/end timestamps recorded. The full procedure is defined in Step 4 under “Workload Requirements.” Do not compare headline numbers unless the steady-state windows, workload mix, fleet-wide demand, client behavior, and controller timing match between the 3-rack baseline and the 4-rack managed run.

Pilot success quick-check#

Before reading the KPI tables, confirm the run satisfies every pass condition below. A miss on any row means the pilot did not meet its goals on this configuration.

Check	Pass condition	Source KPI
Compliance	Zero new compliance events in the managed run’s steady-state window	Compliance events (Data center power and compliance KPIs)
Throughput uplift	Aggregate output tokens/sec strictly greater than baseline at matched load	Aggregate output tokens/sec (Inference service KPIs)
Service stability	Error rate matches baseline within ±0.1 percentage points	Success / error rate (Inference service KPIs)
Envelope adherence	Overall observed power ≤ binding data center constraint for ~100% of steady-state samples	Overall observed power (Data center power and compliance KPIs)
Experiment validity	Workload mix, fleet-wide demand, client behavior, controller timing, and steady-state selection match; both evidence packages are complete	Comparison controls and evidence package

Time alignment#

Clock authority. Synchronize every sampling agent — BMC/BMS exporter, DCGM, inference server, load generator — to the cluster’s NTP/chrony source (typically the cluster head node or site-wide NTP server) before either run. All timestamps in this step are interpreted relative to that authority. If a sampling agent cannot sync to the cluster source, document the offset and treat its data as advisory, not authoritative.
Use the same wall-clock sampling interval (or finer on the managed run) for power and service metrics.
If you change anything about the client (QPS, concurrency, prompt distribution, batching, timeout, retry, or stop behavior), treat the run as a new experiment—do not attribute deltas to DPS.

Interpret offered load correctly#

Open-loop tests: The offered request rate is the configured aggregate QPS schedule, independent of completion rate. Report offered QPS, completed QPS, success and timeout counts, queueing, and TTFT. A capacity shortfall can appear as a growing queue and worse latency even when offered QPS is fixed.
Closed-loop tests: New requests replace completed requests, so observed QPS changes with system capacity. Report configured and observed aggregate concurrency, completion rate, success and timeout counts, and latency.

Do not compare an open-loop run with a closed-loop run, or compare results from different aggregate demand settings as a MaxLPS result.

Data center power and compliance KPIs#

The definitions below assume you can obtain an overall facility or pilot-segment power budget (the data center constraint), provisioned electrical or thermal capacity for the GPUs or systems under test (from nameplate TDP, derated caps, or your capacity model), and metered or telemetry-derived actual draw for the same boundary. Map each to your monitoring source (BMS, rack PDU, BCM, or aggregated BMC).

KPI	Definition	What to expect in a successful pilot
Compliance events	Distinct count of intervals (or events) where overall observed power exceeds any applicable data center constraint (contract limit, breaker headroom, pilot `PowerDomain` budget—use the tightest binding constraint you track).	Target: zero new compliance events in the managed run’s steady-state window. The baseline may already be clean; the managed run must not introduce excursions.
Overall provisioned power	Total system power (provisioned) — sum (or budgeted envelope) of provisioned capacity for the systems in scope (for example aggregate GPU or node TDP limits you could enforce statically).	Between runs: provisioned total for the 4-rack fleet can be higher than for the 3-rack baseline because more hardware sits in the topology, while the same pilot envelope caps observed aggregate draw. You are measuring whether you provisioned more compute than naive sizing would have allowed under that envelope.
Overall observed power	Total system power (observed utilization) — metered or telemetry-summed draw for the same boundary as provisioned and constraints.	Must remain ≤ the binding data center constraint for essentially 100% of steady-state samples on the managed run (allow rare spikes only if your constraint definition already includes agreed crest factor). Expect observed at or below the pilot envelope you set in topology.
Capacity efficiency	Power budget minus total system power (provisioned).	Interpret as headroom between envelope and what you provisioned on paper. With MaxLPS-style sizing, this number is often smaller than in conservative MaxP-only planning because you deliberately provision closer to the envelope—that is expected if compliance stays clean.
Provisioning efficiency	Total system power (provisioned) minus total system power (observed utilization).	Interpret as stranded provisioned headroom not appearing as draw. You often want this gap to shrink when inference loads the fleet more evenly; a large persistent gap under load suggests under-utilization or caps binding unevenly.
Resource group efficiency	Aggregated average power (W) divided by aggregated provisioned TDP limit (W) for all members of the pilot resource group, using the same time window.	Ratios in roughly 0.35–0.85 are common for bursty inference once warm; values sustained below 0.25 under declared steady load suggest the workload or caps are not exercising the fleet, while values sustained above 0.95 leave little margin for spikes—watch compliance.
Node efficiency	Node average power (W) divided by node provisioned TDP limit (W) per compute node, then compare distributions across racks.	Expect a similar spread to baseline at matched load, with managed runs sometimes showing higher averages on more nodes because Power Steering moves limits toward demand. Flag nodes stuck below 0.2 while peers approach limits—possible mapping, cooling, or telemetry issues.

If your tooling cannot yet compute one of the aggregates exactly as defined, document the substitute signal (for example sum of BMC inlet metrics vs. rack PDU) and apply it consistently on both runs.

Inference service KPIs#

These metrics come from the inference stack (vLLM, Triton, TensorRT-LLM, proprietary routers, etc.), not from DPS. Prefer server-reported token and latency counters over client-only estimates.

KPI	Definition	What to expect in a successful pilot
Aggregate output tokens/sec	Total output tokens completed per second across all replicas in scope (cluster-wide), measured over the steady-state window. Exclude failed generations if your stack exposes goodput separately.	Strictly greater than the 3-rack baseline steady-state value when load generator settings are unchanged—this is the headline uplift. Relative uplift depends on how much headroom the baseline left; single-digit to mid-double-digit percent improvements are plausible when the fleet was envelope-starved, but your measured baseline is the reference, not a fixed percentage from this table.
Aggregate total tokens/sec	Input + output tokens per second if your economics or SLAs track both.	Should move roughly in proportion to output tokens/sec unless the prompt length distribution changed.
Tokens/sec per concurrent user (or per active session)	Aggregate output tokens/sec divided by a defined concurrency measure (open sessions, active clients, or in-flight requests—pick one and keep it fixed). Proxy for interactive saturation.	At fixed concurrency, expect within ~±15% of the baseline value if power is not the dominant bottleneck; a large sustained drop warrants checking for cap-induced queueing or straggler GPUs.
Time to first token (TTFT)	Latency from request accept to first output token. Report p50/p95/p99 over the window.	Expect p95 within ~1.2× of baseline p95 and p99 not more than ~1.5× baseline p99 when configuration is healthy—Power Steering can redistribute headroom and introduce modest tail movement. Investigate if p99 regresses beyond ~2× baseline.
Inter-token latency (ITL) / decode token interval	Spacing between successive output tokens (stream quality).	Similar tolerance band as TTFT: p95 ~1.2× baseline, p99 ~1.5× baseline as a first-pass gate; tighten to your product SLO if stricter.
End-to-end request latency	Wall time for full response.	Same percentile bands as TTFT/ITL for streaming workloads; for non-streaming, this is often the primary user-facing metric—use the same ~1.2× / ~1.5× p95/p99 gates unless your SLO says otherwise.
Success / error rate	Fraction of requests completing without server-side error.	Match baseline within ±0.1 percentage points on error rate; regressions here invalidate throughput comparisons.

The ±15%, 1.2×, and 1.5× bands above are starting heuristics for a first pilot read—not substitute contractual SLOs. Replace them with your own targets where you have historical production data.

How to document the outcome#

Compliance table — List each binding constraint, observed max, and count of excursions (baseline vs. managed).
Throughput and latency table — Baseline vs. managed for aggregate tokens/sec, tokens/sec per concurrent user, and p50/p95/p99 for TTFT (or TTFT + E2E).
Efficiency plots — Time series for overall observed vs. constraint, resource group efficiency, and (optional) per-rack node efficiency histograms.
DPS overlay — Example plots: per-GPU set_limit vs. observed GPU power for a sample of nodes, keyed to the same timestamps as inference metrics.

Recommended report layout#

Use the following one-page layout for the final comparison output. Each section should be a single paragraph, table, or plot — anything longer belongs in an appendix.

Run metadata — date, operator, topology name, resource group name, policy, workload identifier, model, framework, fleet-wide demand, client controls, and Power Steering scheduler interval.
Steady-state windows — start/end timestamps for the baseline run and the managed run, plus total run duration.
Clock authority — name of the NTP/chrony source used and any sampling agents whose offset was outside it.
Pilot success quick-check — a copy of the Pilot success quick-check table from this step with the actual pass/fail per row.
Compliance summary — binding constraints, observed maxima, and excursion counts (baseline vs. managed).
Power KPI table — baseline vs. managed values for each row of Data center power and compliance KPIs.
Service KPI table — baseline vs. managed values for each row of Inference service KPIs.
Plots — overall observed power vs. constraint over time; per-GPU set_limit vs. observed GPU power for a sample of nodes; (optional) per-rack node-efficiency histograms.
Evidence package — location and retention reference for the raw telemetry, BMC health report, configuration, and workload records.
Verdict and next action — pass / fail / partial pass, plus the single next change to make if not a pass (per the discipline in Next Steps).

If compliance stays clean, aggregate output tokens/sec rises materially, and latency percentiles remain inside your agreed bands, you have a documented case that the pilot configuration met its goals. If not, capture the same package of telemetry before changing topology, policy, or workload shape so the next iteration is evidence-driven.

Cleanup#

When the pilot is complete, reverse the configuration in the opposite order you applied it.

Step 1: Delete the resource group#

DPS does not expose a standalone resource-group deactivate subcommand — deleting an active resource group automatically deactivates it first. See dpsctl resource-group delete.

dpsctl resource-group delete --resource-group maxlps-pilot

Step 2: Deactivate the topology#

dpsctl topology deactivate --topology maxlps-pilot

See dpsctl topology deactivate for details.

At this point DPS is no longer in the power-control path for the pilot racks.

Step 3: Verify per-GPU caps returned to default#

During topology deactivation, DPS asks each managed node to unset its DPS-managed policy. The GB200 and GB300 device plugins apply a configured idle policy or restore BMC default limits. Device cleanup is best effort, so an unreachable BMC, a manual nvidia-smi -pl ... command after deactivation, or another management tool can leave a nondefault limit. Confirm the result on a representative compute node from each pilot rack:

nvidia-smi --query-gpu=index,power.default_limit,power.limit --format=csv

Every GPU should report power.limit equal to power.default_limit (within rounding). If any GPU shows a lower power.limit, reset it explicitly before moving on:

sudo nvidia-smi -i <gpu-index> -pl <default-limit-watts>

Repeat on each affected GPU. Do not proceed to Next Steps or to the optional uninstall below until every sampled GPU reads back its default limit.

Step 4: (Optional) Uninstall DPS#

If you have more tuning passes planned (different policies or additional workloads from Next Steps), skip this section. DPS can remain installed indefinitely; with the resource group deleted and the topology deactivated, it is not in the power-control path and is safe to leave in place between runs.

Only if you are done with the pilot and want to remove DPS entirely:

helm uninstall dps -n dps

Optionally remove the namespace and any BMC secrets created for the pilot:

kubectl delete namespace dps

Next Steps#

The pilot topology and decoupled resource group are intentionally small so you can prove behavior end-to-end. After Step 8 shows the outcome you want, work the following three priorities in order, re-running the compliance and service KPI captures from Step 8 after each change.

1. Model a larger, more complex topology#

Expand beyond the 4-rack sketch to full rows, feeds, and redundancy paths your facility actually runs: additional PowerDomain boundaries, floor and rack PDUs, diversity groups, and any devices that were simplified or omitted here. The goal is to practice the same import → activate → verify loop on data that resembles a future data center rollout, so surprises surface in the lab—not the first production change window.

Refer to Topologies for the entity model, Import a Topology for validation and import, and Managing Topologies for activation and other lifecycle operations. Rerun the deployment-health gate from Part 1 and the BMC health gate (dpsctl verify bmc-health start --topology <name> --wait --summary-only) from Part 2 after each major topology change.

2. Tune power budgets and overprovisioning carefully#

Many pilots still show observed draw below 100% of the binding budget during steady inference: that is normal when burstiness or conservative caps leave slack. Tune the topology model before you expand the pilot:

Topology PowerDomain — The operating envelope is expressed as the domain’s OperatingLimit (the cap DPS enforces against).
Topology overprovisioning — Overprovisioning is not a Helm chart setting: it is the optional numeric field OverProvisioningPercentage on the PowerDomain entity in your topology JSON. Per the Topology Schema Reference (PowerDomain definition), provisioning headroom scales with the operating limit as provisioning limit = operating limit × (1 + OverProvisioningPercentage / 100). The schema defines the exact type and requires OperatingLimit when OverProvisioningPercentage is present. After editing the topology, give the replacement a unique name and import it by following Import a Topology. Deactivate the previous topology and activate the replacement by following Managing Topologies, then revalidate metrics before comparing runs.

Use small steps on OperatingLimit and OverProvisioningPercentage so DPS can hand out a little more effective GPU budget—only while aggregate telemetry and facility constraints prove you remain safe.

Treat this as an experiment matrix, not a single knob turn:

Change one family of settings at a time (PowerDomain’s OperatingLimit or PowerDomain’s OverProvisioningPercentage), apply it through the topology workflow, and re-run a comparable workload window.
Stay within roughly 10% of the true electrical or contractual limit on the binding constraint when you first push outward—enough to learn sensitivity without jumping past derated margins you have not modeled here.
Observe heavily: the same compliance, efficiency, and inference tables from Step 8 should improve or at least not regress; any increase in compliance events or tail latency is a signal to roll back that change before stacking the next one.

3. Experiment with different workloads#

Repeat measurement passes with other inference models, batch sizes, concurrency levels, and prompt length distributions that matter for your road map—including bursty or seasonal patterns if you can simulate them. The goal is to learn where token throughput and latency stay acceptable as power policy moves, not to optimize a single demo model forever. Non-inference loads (training, stress tools) remain poor substitutes for the envelope story this runbook targets.

Once those three are stable, common production-readiness follow-ons are: integrating scheduling and provisioning (Managing Resource Groups); promoting Step 8 plots into dashboards and SLOs; running multi-hour or multi-day soaks; validating the topology model against as-built power; defining change control for PowerDomain OperatingLimit and OverProvisioningPercentage; and rehearsing failure and maintenance drills with DPS active. Sequence them by risk — topology fidelity and compliance first, then throughput tuning, then integration and operations.

Run the MaxLPS Power Pilot#

Start Here#

Detailed Procedure#

Step 1: Complete Prerequisites and Deploy DPS#

Confirm Prerequisites#

Scheduler-independent configuration#

What you will do#

Prerequisites#

Deploy DPS and Install dpsctl#

Deploy the Pilot Agent#

Verify DPS Deployment#

Authenticate with dpsctl login#

Confirm Server and Client Versions#

Run the Verify Gate#

Step 2: Set Up Metrics Collection#

Step 3: Create a Topology and Run a BMC Health Check#

Import Device Specifications (Optional)#

Verify Device Specifications#

Define the Topology and Static Load#

Prepare the Topology File#

Validate and Import the Topology#

Activate the Topology Before Checking BMCs#

Step 7: Review BMC health results#

Step 8: Activate the topology#

Step 9: Verify topology activation#

Complete the BMC Health Gate#

Deactivate the Topology for the Baseline#

Step 4: Prepare the Workload and Comparison Controls#

Workload Requirements#

Comparison Controls#

What to Collect#

Retain an Evidence Package for Every Run#

Step 5: Run the Baseline#

Step 6: Enable MaxLPS#

Confirm Power Steering Settings#

Reactivate the Topology#

Verify the Topology Is Active#

Select the Default MaxLPS Policy Bundle#

Create the Resource Group#

Add Every Compute Node#

Activate the Resource Group#

Verify Resource Group Activation#

Verify Power Steering Readiness#

Step 7: Run the MaxLPS Test#

Workload Requirements#

Enable DPS Prometheus Metrics#

Run the Workload#

Verify GPU Limit Movement#

What to Record#

Step 8: Compare Results#

Pilot success quick-check#

Time alignment#

Interpret offered load correctly#

Data center power and compliance KPIs#

Inference service KPIs#

How to document the outcome#

Recommended report layout#

Cleanup#

Step 1: Delete the resource group#

Step 2: Deactivate the topology#

Step 3: Verify per-GPU caps returned to default#

Step 4: (Optional) Uninstall DPS#

Next Steps#

1. Model a larger, more complex topology#

2. Tune power budgets and overprovisioning carefully#

3. Experiment with different workloads#

Deploy DPS and Install `dpsctl`#

Authenticate with `dpsctl login`#