layout-3 layout-6 layout-1 layout-top
layout-top layout-6 layout-1

Inference power pilot: NVL72 fleet power management

Overview

This runbook tests whether a fixed facility power budget can support more aggregate inference throughput when DPS (Dynamic Power Software) manages a larger GPU fleet inside that envelope.

You will compare a 3-rack unmanaged baseline with a 4-rack managed run. The workload launch path stays unchanged; DPS controls infrastructure power without requiring workload-scheduler or provisioning-system integration.

Pilot goal Successful result
Increase useful capacity Aggregate output tokens/sec increases at a load point that constrains the baseline, or a controlled capacity sweep demonstrates a higher sustainable load.
Protect service quality Latency remains within the agreed p95 and p99 bands, with no material error-rate regression.
Respect the power envelope Aggregate draw remains within the binding constraint for the full steady-state window.

If both runs complete all offered work, equal throughput is expected. That load point does not demonstrate additional managed capacity; use a controlled QPS or concurrency sweep instead.

Pilot at a glance

Part Action Racks under load DPS state Continue when
1 Verify deployment and set MaxLPS runtime values None Installed; no pilot group active Deployment is healthy and effective MaxLPS settings are recorded.
2 Import and validate the topology; run BMC health gate None Topology temporarily active All 72 nodes pass the required BMC checks or have a documented permitted exception, and topology activation succeeds.
3 Remove DPS from the control path None No active topology or resource group dpsctl tp list --active returns no active topology.
4 Stage workload, telemetry, logs, and run metadata None Out of control path Every evidence stream is queryable and the workload configuration is archived.
5 Run the unmanaged baseline 3 (54 nodes) Out of control path A valid steady-state window and complete evidence package exist.
6 Activate the topology and shared-GPU resource group None In control path DPM, MaxLPS power steering, and Shared GPU are active for all 72 nodes.
7 Run the managed comparison 4 (72 nodes) In control path A comparable steady-state window and complete evidence package exist.
8 Compare compliance, throughput, latency, and efficiency N/A No change The result is recorded as pass, partial pass, or fail.

What does MaxLPS mean?

Modern AI inference rarely sustains peak GPU power draw. NVIDIA MaxLPS (Maximum Land Power Shell) is the data center pattern of deploying more compute than static MaxP sizing would allow while a control plane keeps aggregate draw within the agreed operating envelope.

Some pilots evaluate roughly 40% more compute against the same envelope. Treat that only as an experiment starting point. Validate the hardware, workload, redundancy model, and facility limits before choosing a ratio.

The operating envelope is an upper-bound guarantee, not a promise that the fleet will draw the exact budget or that every node will receive or consume equal power. The MaxLPS control loop redistributes GPU limits according to observed demand, so per-node limits and draw can differ even when nodes run the same workload. Evaluate aggregate compliance together with the distribution of applied limits, utilization, throughput, and latency across nodes.

This runbook validates that pattern on NVL72: reproduce a fixed envelope with more GPUs, measure inference throughput and latency, and confirm power compliance.

Scheduler-independent configuration

This runbook uses the scheduler-independent path: workload schedulers and provisioning systems continue to own placement and fleet membership, while operators manage a fixed DPS topology and resource group directly with dpsctl.

Required element Role in this pilot
4-rack topology Models the pilot fleet and its shared operating envelope.
DPM Enforces power policy on the member nodes.
MaxLPS power steering Redistributes GPU headroom using live telemetry. See Power-steering behavior.
Shared GPU Enforces the resource-group GPU budget as a group sum. See Resource Groups.

All four elements are required. For scheduler-integrated lifecycle management, see Managing Resource Groups and the nvidia.dcpower.v1 API.

Prerequisites

Do not start Part 1 until every row is ready.

Requirement Ready when
Hardware Four GB200 NVL72 or GB300 NVL72 racks are identified; all 18 compute nodes per rack are powered on and BMC-reachable.
BMC compatibility The Day 0 Redfish assessment is complete for all 72 nodes and any gaps are documented.
DPS The server is deployed to Kubernetes, BMC credential secrets exist for every node, and server and dpsctl versions match.
MaxLPS controls Both Helm enablement flags are true; -30 is configured as the initial headroom value.
Workload The inference application, model, and dataset are staged on all four racks and can be launched through the existing orchestrator.
Power envelope The 3-rack operating envelope is approved and recorded for use in both runs.
▼ BMC compatibility checklist

Deploy DPS by following the Deployment Guide. Use the latest 0.8.x release and matching dpsctl client available from DPS on NGC.

Configure the MaxLPS components in the Helm values.yaml used for install or upgrade. The prs keys below are the chart’s implementation identifiers:

prs:
  enabled: true

dps:
  prs:
    enabled: true
    headroomPercent: -30.0

The top-level prs.enabled deploys the MaxLPS controller. dps.prs.enabled connects the DPS server to that controller. Both must be true. dps.prs.headroomPercent supplies the deployment default; Part 1, Step 4 verifies and controls the effective runtime value. See Power-steering behavior for the canonical semantics.

Install and authenticate dpsctl using Install dpsctl.

No physical hardware? Use the DPS SDK Simulator to explore the product. Simulator results do not validate the physical MaxLPS pilot described here.

Part 1: Verify DPS Deployment

Outcome: Authentication works, server and client versions match, dpsctl verify passes, and the original MaxLPS settings are saved before the pilot values are applied.

Stop if: Any required component reports healthy: false, the client and server versions differ, or the original MaxLPS settings cannot be saved.

Note: If you haven’t already, export a few dpsctl environment variables to keep the commands that follow concise:

export DPSCTL_HOST="api.dps.your-domain.com"
export DPSCTL_PORT="443"
export DPSCTL_INSECURE_TLS_SKIP_VERIFY=true # Needed if TLS was not configured

Otherwise, you must add --host, --port, and --insecure-tls-skip-verify to every dpsctl command.

Step 1: Authenticate with dpsctl login

Before any other dpsctl command will succeed, authenticate against the DPS server. The token is cached under ~/.dpsctl/credentials.yaml and reused by subsequent commands until it expires.

dpsctl login -u <username>

dpsctl login prompts for the password interactively, or reads DPSCTL_USERNAME and DPSCTL_PASSWORD from the environment if set. See Install dpsctl for the one-time install and initial configuration.

Step 2: Confirm server and client versions

Confirm authentication succeeded and capture the server version:

dpsctl server-version

A successful call returns the server’s version payload. Confirm the server version matches the latest 0.8.x DPS release available on NGC.

Confirm the local dpsctl client matches the server version:

dpsctl --version

Confirm this reports the same version as the server. An AUTHENTICATION_FAILED or missing token error from dpsctl server-version means the previous dpsctl login did not complete — rerun it before continuing.

Step 3: Run the verify gate

Proceed to Part 2 only when status.ok is true and no component reports healthy: false. A skipped: true component is acceptable — it means that component is not part of this deployment.

Run dpsctl verify with no component flags. It checks the DPS server, database, authentication service, UI ingress, and BCM connectivity in a single call:

dpsctl verify

Each component (dps_server, database, auth, ui, bcm) returns healthy, message, and skipped. The details field is informational and is mainly useful for troubleshooting.

▼ Example healthy output

Troubleshooting: If deployment verification fails, see Deployment Verification.

Step 4: Record and set the effective MaxLPS settings

The API exposes the MaxLPS settings through the implementation fields prs_enabled and prs_headroom_percent. Their effective values can differ from Helm when values already stored in the database override the chart defaults. Read and save both values before changing them so Cleanup can restore the deployment exactly. These are deployment-wide settings: use a dedicated pilot deployment or coordinate a change window if any other MaxLPS-managed resource groups are active.

dpsctl settings list | tee maxlps-pilot-original-settings.json
jq -e '
  .settings.prs_enabled != null and
  .settings.prs_headroom_percent != null
' maxlps-pilot-original-settings.json

Keep maxlps-pilot-original-settings.json with the pilot run metadata. Enable the MaxLPS runtime controls, set the first managed run to -30, and verify both effective values:

dpsctl settings update \
  --set prs_enabled=true \
  --set prs_headroom_percent=-30
dpsctl settings list \
  | jq -e '
      .settings.prs_enabled == "true" and
      .settings.prs_headroom_percent == "-30"
    '

The runtime values are persisted and override the Helm defaults across server restarts until they are updated again. When enabling MaxLPS power steering or changing the effective headroom value, DPS asynchronously refreshes active control domains; it does not require a server restart. A write that leaves both effective values unchanged is a no-op and produces no new re-upsert log. At this point no pilot resource group is active, so the verified values will be used when the group is activated in Part 6. See dpsctl settings list and dpsctl settings update for the settings behavior and accepted range.

Part 2: DPS Configuration

Outcome: The 4-rack topology is imported, every node passes the BMC health gate or has a documented permitted exception, and one test activation proves the complete DPS-to-BMC path.

Stop if: A required device model is missing, the topology is invalid, any blocking BMC issue remains, or activation fails.

DPS needs device specifications and a topology that represents the pilot’s power-distribution structure.

Step Action Gate
1–2 Import any custom device specifications and verify every referenced model. Every topology model appears in dpsctl device list.
3–5 Define, validate, and import the 4-rack topology. Validation passes and all 72 compute nodes are present.
6–7 Run and review the BMC health check. Blocking issue count is zero; any permitted exception is documented.
8–9 Activate once and verify the topology. Activation succeeds and leaf_node_names contains all 72 nodes.

Step 1: Import Device Specifications (if needed)

DPS ships with a set of supported device specifications out of the box. The supported devices include, at time of writing:

  • Compute: DGX_GB200, DGX_GB300, DGX_B200, DGX_B200EM, DGX_B300, DGX_B300EM, DGX_H100
  • GPUs: GB200, GB300, B200, B300, H100
  • CPUs: Grace, Grace_GB300
  • Power infrastructure: FloorPDU95 (Generic Floor PDU with 95% efficiency factor) , RackPDU95_57500W (Generic Rack PDU with 95% efficiency factor and max load of 57,000 W), PowerSupply95_3300W (Generic Power Supply with 95% efficiency factor and max load of 3,300 W)

GB300 GPU minimum power (optional, second pass). If you already have a representative baseline, you can refine the GB300 GPU entry in your device-model YAML (devices.yaml) before dpsctl device upsert. Set each GPU’s minLoadWatts (see Device Specifications) to roughly 100 W below the average GPU power you measured during the baseline inference run, using the same steady-state window you plan to compare in Part 8. That lowers the floor DPM assumes for each GPU without changing the upper cap, provided maxLoadWatts stays above minLoadWatts and still reflects hardware limits.

For a first-time pilot, leave the shipped minimum unchanged through the baseline in Part 5. If you then tune minLoadWatts, treat the result as a second experiment: document the before/after values, re-import the device definition, repeat the BMC health and activation gates, and rerun both comparison windows. Do not compare a baseline collected with one device floor against a managed run collected with another.

If your data center uses PDUs, PSUs, or other equipment whose model does not match the list above, you must register a specification for it before it can appear in a topology. Follow the full procedure in Managing Devices, then import your additions:

dpsctl device upsert devices.yaml

Step 2: Verify device specifications

Confirm that every model you reference in your topology appears in the output:

dpsctl device list

Step 3: Define the topology and static load

The topology is how you describe your power-distribution network to DPS. For this runbook, the topology must contain at minimum:

Entity type Count Purpose
PowerDomain 1 Top-level 3-rack operating envelope applied across all 4 racks.
PowerDistribution 4 (one per rack) Rack PDU.
ComputerSystem 72 (18 per rack) GB200 or GB300 compute nodes, parented under their rack PDU.

The PowerDomain’s PowerValue is the top-level operating envelope DPS enforces across the 4-rack pilot topology. Set it to the same 3-rack envelope used by the baseline: 375 kW for GB200 NVL72 or 405 kW for GB300 NVL72.

Each GB300 NVL72 rack contains 9 NVSwitch trays. Their draw is not managed by DPS, but it must still be reflected in the rack PDU and PowerDomain budget.

The runbook represents the per-rack NVSwitch aggregate with the StaticLoad field on each FloorPDU-Rack0X (PowerDistribution) entity. StaticLoad declares fixed power consumption from unmanaged devices and is propagated up the power-distribution network automatically—see Entities → Static Load.

Use 7,416 W per rack for a standard GB300 NVL72 9-switch aggregate (replace with measured or vendor-specified values when available). The example topology in Step 4 applies this value on every rack PDU. For a standard GB200 NVL72 rack, we recommend a 9-switch aggregate power value of 11,160 W per rack.

Step 4: Prepare the topology file

Download the reference topology that matches your hardware:

  • maxlps-pilot-gb300.json — 4 GB300 NVL72 racks, 72 DGX_GB300 nodes, 4 FloorPDU95 rack PDUs at 135 kW (StaticLoad 7,416 W), PDU-Root PowerDomain at the 3-rack envelope of 405 kW. Ships with the GB300-* policy presets the runbook references in later steps.
  • maxlps-pilot-gb200.json — 4 GB200 NVL72 racks, 72 DGX_GB200 nodes, 4 FloorPDU95 rack PDUs at 125 kW (StaticLoad 11,160 W), PDU-Root PowerDomain at the 3-rack envelope of 375 kW. Ships with the GB200-Per-* policy presets derived from a live GB200 environment.
  • generic-2-node-topology.json — Smaller, generic two-node topology for B200 OEM models, with two DGX_B200EM nodes under one PowerDomain at 20 kW. Use this example as a reference when your deployment uses B200 or B300 models instead of GB200 or GB300 NVL systems.

Replace the BMC URL and SecretName values with those for your fleet before importing.

▼ Preview: topology root, one rack PDU, and the PowerDomain

For additional topology examples, see Managing Topologies.

Step 5: Validate and import the topology

Validate the topology file against the DPS schema and import it. Importing only registers the topology and its inventory in the DPS database — no policy is applied to any BMC yet, so this step is safe to run on a live fleet.

dpsctl topology validate topology.json # Confirms the topology matches the DPS topology schema
dpsctl topology import topology.json

Troubleshooting: If topology validation or import fails, see Topology Validation and Import.

Step 6: Run the BMC health check for every node in the topology

Before activating, run the deep BMC prerequisite health check against every BMC in the imported pilot topology. This check is broader than a reachability ping: it validates Redfish endpoint reachability, firmware inventory, power-limit write and read-back behavior, telemetry sampling, GPU power-policy state, and in-band/out-of-band power drift across every node. Passing --topology runs the check against every node in that topology in a single call. The topology only needs to be imported — not active — for this check to run.

On large clusters the probe can take up to an hour. dpsctl verify bmc-health start starts the check asynchronously and returns a task_id. This server-side task continues running even after dpsctl exits. Use the task_id to query the status of the health check:

dpsctl verify bmc-health start --topology maxlps-pilot --force-writes --expected-edpp-pct 100 --samples-per-telemetry 2500 --telemetry-interval 500ms
# Query the status until the health check is done
dpsctl verify bmc-health status <task-id>
# Once done, generate a report
dpsctl verify bmc-health report <task-id> --wait --output json > bmc-health-maxlps-pilot.json
dpsctl verify bmc-health report <task-id> --summary-only
▼ Example Full GB300 Report: All Nodes Healthy

Step 7: Review BMC health results

This run is also where you collect BMC latency metrics for the pilot. From bmc-health-maxlps-pilot.json, capture the following latency summaries for every node:

  • endpoint_latency_ms
  • telemetry_latency_ms
  • power_get_latency_ms
  • power_set_latency_ms
  • edpp_get_latency_ms
  • wpps_get_latency_ms

For MaxLPS readiness, target p50 below 0.5 seconds, p95 below 0.7 seconds, p99 below 0.8 seconds, and max near 1 second for read and write paths. If p99 is greater than 1 to 2 seconds, document the latency measurements and assess cluster-wide BMC client connection pressure before activation.

Prepare other BMC clients before you run this check. Confirm the BMC client connection requirements from the prerequisites are still in place:

  • For GB200 and GB300 systems, the NVIDIA BMC and DPS teams recommend no more than four BMC client connections, including two DPS client connections.
  • Use session token authentication instead of basic authentication.
  • Use keep-alive to maintain the connection.

In the summary report generated in Step 6, confirm:

  • status.ok is true.
  • cluster_summary.total_nodes is 72, matching the node count in maxlps-pilot.
  • cluster_summary.failed is 0.
  • cluster_summary.nodes_unreachable is 0.
  • issues contains no SEVERITY_ERROR entries.

Known B200/B300 firmware-validation exception: on systems with B200 or B300 GPUs, DPS 0.8.x can report a FIRMWARE_VALIDATION_FAILED issue with message: firmware validation failed even when the BMC is usable. If this is the only SEVERITY_ERROR for the affected B200 or B300 node, cluster_summary.nodes_unreachable is still 0, and the rest of the health report is clean, document the exception and continue with the runbook.

If any of these do not hold, other than the known B200/B300 firmware-validation exception described here, the issues list identifies what to fix. Each entry includes the affected node, the resource within that node, an issue code, the observed value, the threshold it violated, and a human-readable message. Fix the affected nodes and re-run the health check until the report is clean before activating the topology.

Troubleshooting: For issue-specific remediation, see BMC Health Check.

Step 8: Activate the topology

Activate the topology as a temporary end-to-end gate. Activation exercises the full DPS-to-BMC set-policy path against every node so any device-model or policy-application issue that the BMC health review in Step 7 could not catch surfaces now, at setup time, instead of immediately before the 4-rack managed run in Part 7. The topology will be deactivated in Part 3 so the unmanaged baseline in Part 5 runs with DPS out of the control path, and re-activated later in this runbook.

dpsctl topology activate --topology maxlps-pilot # Must match the name given in topology.json

Troubleshooting: If topology activation fails, see Topology Activation.

Step 9: Verify topology activation

dpsctl tp list

Confirm the topology appears and leaf_node_names lists all 72 compute nodes.

Part 3: Deactivate the Topology Before the Baseline Run

Outcome: The topology remains imported but is inactive, so the baseline runs without DPS in the power-control path.

Part 2 activated the topology once to expose setup or BMC problems early. Deactivate it now for the unmanaged baseline.

Step 1: Deactivate the topology

dpsctl topology deactivate --topology maxlps-pilot

Step 2: Verify no topology is active

dpsctl tp list --active

Confirm no topologies are listed as active. The topology definition itself remains imported and will be re-activated at the start of Part 6.

Part 4: Preparing for Testing

Outcome: The workload is controlled and repeatable, all evidence streams are durable and queryable, and the steady-state selection rule is fixed before either run.

Stop if: The baseline state is not clean, clocks are not aligned, any required evidence stream is missing, or the workload configuration has not been archived.

Workload requirements

The baseline is unmanaged: DPS remains installed, but no topology or resource group is active.

  • dpsctl tp list --active returns no active topology.
  • dpsctl rg list returns no pilot resource group.
  • The workload is representative production inference, not training, gpu-burn, or a pure power stress test.
  • Model, framework, batching, input distribution, and load-generator mode are fixed for both runs.
  • The aggregate target-QPS schedule across all load generators for open-loop testing, or aggregate configured concurrency for closed-loop testing, is fixed for both runs.
  • Request timeout, retry, cancellation, warm-up, run-duration, and benchmark stop rules are fixed for both runs.
  • The pilot racks do not host unrelated workloads during either comparison window.

Load and benchmark validity contract

Record these controls before either run. A change to any row creates a new experiment and invalidates a direct comparison.

Control Requirement
Aggregate offered load Define one fleet-wide QPS schedule. If load generators are configured per replica or node, record every configured value and verify their sum is the same in both runs. Do not hold per-node QPS constant when node count changes.
Closed-loop load Define one fleet-wide concurrency value and distribute it across clients without changing the total.
Capacity sweep Use the same starting load, step size, dwell time, and stop criteria. Report offered QPS, completed QPS, and goodput at every step.
Request timeout Set one explicit client timeout and retain its configuration. If the client uses MLPINF_REQUEST_TIMEOUT_S, record its effective value. Do not mix no-timeout and finite-timeout results.
Retry and cancellation Keep retry count, retry backoff, cancellation behavior, and whether timed-out requests continue server-side identical. Count original attempts separately from retries.
Benchmark validity Keep warm-up, total duration, steady-state rule, error policy, completion rule, and any tail-skip or force-complete behavior identical.
Request phases Emit a stable request ID and UTC timestamps for issue, server admission or dequeue, prefill start, prefill end, first token, completion, timeout, and cancellation. If a phase does not exist in the stack, record it as not applicable rather than omitting it.

Issue, admission or dequeue, prefill, and first-token timestamps are required to attribute TTFT. If they cannot be collected, report TTFT as an outcome only and mark its queue-versus-prefill attribution indeterminate.

Use the same steady-state selection rule for every run:

  1. Run for at least one hour.
  2. Trim ramp-up and cool-down.
  3. Confirm per-rack power and aggregate tokens/sec have flattened without a monotonic climb or large step change.
  4. Select a contiguous window of at least 30 minutes. Extend the run if no such window exists.
  5. Record the exact UTC start and end timestamps and apply them to every metric and log query.

Stage evidence collection and retention

Use identical collectors, intervals, queries, retention, and configuration-capture procedures for both runs.

Evidence Required setup Ready when
GPU and Grace CPU telemetry Follow Capture 1-second GPU and CPU telemetry with DCGM Exporter. Use the pinned collector CSV and a 1,000 ms DCGM watch interval. A current sample with raw timestamps and labels is visible from each required target.
Non-Grace host CPU utilization On B200, H100, or another non-Grace host, use node exporter or an equivalent host collector at a 1-second interval as described in the DCGM companion. Aggregate and per-core host CPU utilization are retained for the same UTC window as the GPU data.
DPS and MaxLPS controller logs Route dps-server and MaxLPS controller stdout and stderr to Loki or equivalent persistent storage. The controller’s container identifier is prs. Retain the events listed in Capture per-update debug evidence. Both containers and the required events are queryable by namespace, pod, container, and UTC range. Events can be joined by task/request ID where available and always by UTC timestamp, node, and GPU ID.
Workload logs Persist inference server, application, router, and load-generator logs with stable request IDs. Required request-phase timestamps, errors, retries, timeouts, queue events, throughput, and latency are queryable.
Workload configuration Archive the job, manifest, or launch command; image digest; model and dataset versions; framework; batching; ISL/OSL; aggregate QPS schedule or aggregate concurrency; replicas; placement; timeout/retry/stop rules; and non-secret environment values. The snapshot is stored beside the run metadata, contains no secrets, and identifies every benchmark-validity control.
Effective control state Export topology, entities, policies, device models, effective settings, resource-group state, release version, and deployed images immediately before each run. The state bundle is timestamped and stored beside that run’s evidence.

Do not use kubectl logs as the system of record. Pod replacement and rotation can remove local logs.

For the diagnostic interval, persist detailed dps-server span events with 100% sampling so queue, dispatch, retry, and Redfish request/response data are available for debugging. The 0.8.x runtime identifiers are TRACING_GRPC_PROVIDER_URL, TRACING_SAMPLING_RATE=1, and TRACING_EVENT_DETAIL=detailed; apply them through the site’s deployment overlay and retain the resulting events in the site’s persistent observability backend. Configure the MaxLPS controller log level so New power limits: entries are retained. Prove the collection path with one controlled limit update before starting the baseline.

Archive effective control state

Run this after the intended activation or deactivation steps and immediately before each workload run. Use a unique RUN_ID, such as baseline-20260709T210000Z or managed-20260709T230000Z.

export RUN_ID='<run-id>'
export RUN_DIR="${HOME}/maxlps-pilot/${RUN_ID}"
mkdir -p "${RUN_DIR}"

dpsctl topology export \
  --topology maxlps-pilot \
  --entities \
  --policies \
  >"${RUN_DIR}/topology-entities-policies.json"
dpsctl settings list >"${RUN_DIR}/effective-settings.json"
dpsctl tp list >"${RUN_DIR}/topologies.json"
dpsctl tp list --active >"${RUN_DIR}/active-topologies.json"
dpsctl rg list >"${RUN_DIR}/resource-groups.json"
dpsctl device list >"${RUN_DIR}/device-models.json"
dpsctl server-version >"${RUN_DIR}/server-version.json"
dpsctl --version >"${RUN_DIR}/dpsctl-version.txt"
helm list --namespace dps --output json >"${RUN_DIR}/helm-releases.json"
kubectl get pods --namespace dps \
  --output custom-columns='POD:.metadata.name,CONTAINERS:.spec.containers[*].name,IMAGES:.spec.containers[*].image,IMAGE_IDS:.status.containerStatuses[*].imageID' \
  >"${RUN_DIR}/deployed-images.txt"

The topology export can contain internal BMC URLs and secret names. Store the original in the protected evidence package and redact those fields before sharing it outside the operating team. Also archive the non-secret deployment values used for the run; never place passwords, tokens, or Secret contents in the evidence package.

What to collect

Category Capture for the full run and steady-state window
Service Aggregate offered and completed QPS, goodput, output and total tokens/sec, latency percentiles, errors, retries, timeouts, cancellations, queue depth, and every required request-phase timestamp.
Workload Archived configuration, validity controls, and persistent application, router, and load-generator logs.
Power Per-rack and aggregate draw, steady-state averages, and per-GPU draw distribution.
Limit-update debugging Per-update requested and returned limits, clamp diagnostics, queue/dispatch/retry timing, Redfish request/response data, and DCGM limits keyed by node and GPU.
Run context Effective control-state bundle, total duration, steady-state start and end, nodes actually under load, clock authority, and EDPp settings per node.

Use the same definitions and clock authority across trusted BMC, rack, DCGM, inference, and facility sources. Launch through the existing orchestrator; this runbook does not replace Slurm, Kubernetes, Triton, or another workload path.

Part 5: Running the Baseline Test

Outcome: A complete, unmanaged 3-rack reference run with a valid steady-state window and durable evidence package.

  1. Reconfirm that no topology or resource group is active.
  2. Archive the effective control state using Part 4.
  3. Start DCGM, host CPU, power, workload, actuation-trace, and log collection.
  4. Launch the workload on 3 racks (54 compute nodes) through the existing orchestrator.
  5. Record the number of nodes actually under load; Part 6 uses this count to calculate policy.
  6. Run for at least one hour and select the steady-state window using Part 4.
  7. Stop collection only after the workload ends, then verify every artifact is readable.

Part 6: Create and Activate DPS Resource Group

Outcome: The topology and maxlps-pilot resource group are active across all 72 nodes with DPM, MaxLPS power steering, and Shared GPU enabled.

Stop if: Topology activation fails, the policy calculation is wrong, membership is incomplete, or the MaxLPS control loop does not emit ongoing limit updates.

Step Action Gate
1–2 Reactivate and verify the imported topology. maxlps-pilot is active.
3 Calculate policy from actual baseline and managed node counts. The 54-to-72 example resolves to 75% of MaxP.
4–5 Create the resource group and add all compute nodes. DPM, MaxLPS power steering, and Shared GPU are enabled; membership contains all 72 nodes.
6–7 Activate synchronously and inspect state. Activation is complete with no unresolved per-node failure.
8 Inspect the MaxLPS controller stream. New power-limit blocks continue after activation.

Step 1: Re-activate the topology

The topology was deactivated in Part 3 so the baseline could run with DPS out of the control path. Re-activate it now before creating the resource group; the topology is already imported, so this is the only setup step required:

dpsctl topology activate --topology maxlps-pilot

Troubleshooting: If topology activation fails, see Topology Activation.

Step 2: Verify the topology is active

dpsctl tp list --active

Confirm maxlps-pilot appears as the active topology.

Step 3: Calculate the resource group policy

Calculate the resource group policy from the ratio between the unmanaged baseline node count and the managed MaxLPS node count. The goal is to preserve the baseline power envelope while the MaxLPS run uses more nodes.

Use the following formula:

test policy = (baseline node count / MaxLPS node count) * MaxP

As a percentage of MaxP:

policy percentage = (baseline node count / MaxLPS node count) * 100

For the 3-rack baseline and 4-rack MaxLPS run in this runbook:

54 baseline nodes / 72 MaxLPS nodes * 100 = 75

Use the matching policy preset from the example topology: GB300-Per-75 for GB300 NVL72 or GB200-Per-75 for GB200 NVL72.

If your baseline or MaxLPS run uses a different node count, calculate the percentage from your actual counts. If the result does not match an existing preset, create or select a policy that matches the calculated watt value instead of defaulting to 75%.

For a custom policy, multiply each 100% (MaxP) policy limit by the calculated ratio, including Node and GPU entries. Add the custom policy to the topology file’s Policies list, then follow the update workflow in Managing Topologies before continuing. Confirm the policy is available with dpsctl policy list.

Step 4: Create the resource group

Create one resource group containing every compute node in the topology, with DPM, MaxLPS power steering, and shared-GPU all enabled. The --prs-enabled flag is the CLI implementation flag for MaxLPS power steering. The --shared-gpu-enable flag binds the resource group’s GPU power consumption to a shared, group-wide budget rather than per-node caps. Set --policy to the policy you calculated in Step 3. The example below uses the 54-node baseline to 72-node MaxLPS ratio from this runbook.

dpsctl resource-group create \
  --resource-group maxlps-pilot \
  --external-id 1 \
  --dpm-enable \
  --prs-enabled \
  --shared-gpu-enable \
  --policy GB300-Per-75 # For GB200, use GB200-Per-75; for custom ratios, use your calculated policy.

Troubleshooting: If resource group creation fails, see Resource Group Creation.

--external-id is required and must be a 64-bit integer. It is the handle a workload scheduler or provisioning system would normally use to correlate this resource group with one of its own records. In the scheduler-independent configuration this runbook describes, no external system owns the resource group — pick any positive integer and keep a note of it for your own records. DPS does not enforce uniqueness on this value, but using a distinct ID per resource group makes dpsctl rg list output easier to read once you have more than one.

Step 5: Add every compute node

Add every compute node in the topology to the resource group. Substitute your real node names for gb300-nvl-001-compute01,gb300-nvl-001-compute02,...:

dpsctl resource-group add \
  --resource-group maxlps-pilot \
  --entities "gb300-nvl-001-compute01,gb300-nvl-001-compute02,gb300-nvl-001-compute03..."

See dpsctl resource-group add for additional flags (partial-update behavior, strict policy, etc.).

Troubleshooting: If adding resources fails, see Adding Resources to the Resource Group.

Step 6: Activate the resource group

Activate the resource group. Use --sync so the command blocks until every node has its policy applied:

dpsctl resource-group activate \
  --resource-group maxlps-pilot \
  --allow-reprovision \
  --sync

See dpsctl resource-group activate for the full activation flag set, including strict-policy and partial-timeout options.

Step 7: Verify resource group activation

dpsctl rg list --active

Confirm the maxlps-pilot resource group appears with these API implementation fields:

  • activation_status: active
  • dpm_enable: true
  • prs_enabled: true
  • shared_gpu_enable: true
  • resource_names listing all 72 compute nodes

Troubleshooting: If resource group activation fails, see Resource Group Activation.

Step 8: Verify the MaxLPS control loop is updating GPU limits

After the resource group is active, confirm the MaxLPS controller is running control-loop iterations and writing new GPU limits. Its Kubernetes label and container identifiers are both prs:

kubectl get pods -n dps -l app=prs
kubectl logs -n dps <prs-pod-name> -c prs --since=10m

Look for repeated Devices power draw:, New power limits:, and Loop latency: blocks. The node names, timestamps, and watt values will differ, but a healthy MaxLPS control loop should look similar to this:

[2026-06-12 16:42:10,214][DEBUG] Devices power draw:
   node_name                 device_type  device_index  domain_name   min_watts  max_watts  value_watts
0  gb300-nvl-001-compute01  gpu          0             maxlps-pilot  200.0      1200.0     842.7
1  gb300-nvl-001-compute01  gpu          1             maxlps-pilot  200.0      1200.0     816.4
2  gb300-nvl-001-compute02  gpu          0             maxlps-pilot  200.0      1200.0     905.2

[2026-06-12 16:42:10,319][DEBUG] New power limits:
   node_name                 device_type  device_index  next_limit
0  gb300-nvl-001-compute01  gpu          0             985.3
1  gb300-nvl-001-compute01  gpu          1             962.1
2  gb300-nvl-001-compute02  gpu          0             1034.8

[2026-06-12 16:42:11,002][INFO] Loop latency: 0.68 s. About to sleep: 5.32 s

The important signal is that the MaxLPS controller continues to emit new New power limits: blocks after activation.

Troubleshooting: If the control loop does not emit ongoing power-limit updates, see MaxLPS Control-Loop Logs.

Part 7: 4-Rack Inference Test Under DPS

Outcome: A complete 4-rack managed run with a steady-state window comparable to the baseline and all managed-control evidence retained.

Stop if: The workload configuration differs beyond rack count and the control path, the evidence gate fails, or power compliance is lost.

The intended experimental changes between Part 5 and Part 7 are exactly two: all 4 pilot racks are under load, and DPS is now actively managing their power. Keep the workload, model, framework, batching, input distribution, load-generator mode, aggregate offered load, timeout rules, and operating envelope controlled.

Because node count and power-management behavior change together, this comparison evaluates the overall pilot configuration. It does not isolate the effect of dynamic MaxLPS redistribution from the static power reduction required to place 72 nodes under the same envelope.

The two runs differ on exactly two dimensions:

Dimension Baseline (Part 5) Managed (Part 7)
Racks Under Load 3 racks (54 compute nodes) 4 racks (72 compute nodes)
DPS Control Path Not in control path; no topology and no resource group active In control path; topology maxlps-pilot active and resource group maxlps-pilot active
Operating Power Envelope Same Same
Aggregate Offered Load Same fleet-wide target-QPS schedule, or same fleet-wide configured concurrency Same fleet-wide target-QPS schedule, or same fleet-wide configured concurrency
Timeout, Retry, and Stop Rules Same Same
Workload, Model, Framework, Batching, Input Distribution, and Load-Generator Mode Same Same

The Operating Power Envelope row is held by the topology in the managed run: the active PDU-Root PowerDomain already sets the same 3-rack operating limit used by the baseline, and DPS enforces that limit across all 4 racks through the resource group’s shared-GPU policy.

With the topology active and the pilot resource group (maxlps-pilot) in place—created and maintained without integration to your workload schedulers or provisioning systems—re-run the same workload you used for the 3-rack baseline—same model, same framework, same request pattern—but now across all 4 racks (72 nodes), with DPS in the control path.

As in Part 5, this runbook does not prescribe how you launch the workload. Use the same orchestration tooling (Slurm, Kubernetes inference operator, Triton, your own job runner) you used for the baseline. The only intended changes between the two runs are the number of racks under load and whether DPS is enforcing power policy — everything else must stay the same so the comparison is meaningful.

Workload requirements

The 4-rack run must be directly comparable to the 3-rack baseline. In addition to the baseline’s requirements:

  • It is the same workload as the baseline. Use the same model, framework, batch size, input distribution, load-generator mode, timeout/retry rules, and benchmark stop criteria.
  • Aggregate offered load is controlled. For an open-loop test, use the same fleet-wide target-QPS schedule—not the same QPS per node or replica. For a closed-loop test, use the same fleet-wide configured concurrency. Record the per-generator values and verify their sum before launch.
  • All 4 pilot racks are under load in isolation. No other workloads should be sharing these racks or competing for the same facility power budget during the run. The same operating envelope that constrained the 3-rack baseline should constrain the 4-rack run.
  • DPS is in the control path. Confirm the pilot topology from Part 2 and the active resource group from Part 6 are in place (dpsctl tp list, dpsctl rg list) and that the BMC health gate from Part 2 still passes (dpsctl verify bmc-health start --topology maxlps-pilot --wait --summary-only) before launching the workload.
  • It produces a comparable steady-state window. DPS’s per-GPU caps are set dynamically from telemetry, so the run needs enough duration for the control loop to settle into steady-state redistribution, not just the workload itself. Apply the same steady-state window procedure defined in Part 4 under “Workload requirements”, and make the resulting window at least as long as the one used for the 3-rack baseline.

Enable DPS Prometheus metrics

dps-server exports operational metrics in Prometheus text format on its HTTP service at /metrics. Enable scraping for the pilot so the DPS-side signals called out in What to record below — per-GPU cap movement and group-sum allocation versus observed aggregate draw — land in the same time-series store as the workload and BMC telemetry you captured in Part 4, and can be cross-checked against the steady-state window with PromQL. The chart-managed ServiceMonitor (dps.serviceMonitor.enabled: true) is the recommended path for in-cluster Prometheus and is applied through the Helm upgrade flow in the Deployment Guide; see Prometheus Monitoring for the full setup, including external Prometheus and TLS variants.

Note: Land this Helm change once, before the 4-rack managed run starts so the steady-state window in What to record is fully covered by Prometheus samples.

Capture per-update debug evidence

Prometheus summaries alone cannot explain a node- or GPU-specific limit transition. During a predeclared diagnostic interval inside the steady-state window, retain the following data for every limit update and for every node or GPU associated with a latency, timeout, or power anomaly.

Collect Required fields and events
Requested limit Timestamped New power limits: entry with node, GPU ID, and next_limit.
Limit result UpdateGPUPolicies task/request ID, requested watts, returned set_limit, and any prs.update_gpu_policies.budget_clamped event or clamped: requested ... set ... diagnostic.
Processing timing Per-device reconciler.enqueue, reconciler.dispatch, queue_wait, retry, reconciler.response_received, and reconciler.dequeue events with UTC timestamp, request ID, device ID, and queue size.
Device request Redfish request start/end or duration, response status, node/GPU, and error details.
Device limit telemetry DCGM_FI_DEV_POWER_MGMT_LIMIT and DCGM_FI_DEV_ENFORCED_POWER_LIMIT for the same node/GPU and UTC interval.
Workload timing Request issue, admission/dequeue, prefill start/end, first token, completion/timeout, and cancellation timestamps for requests overlapping the limit update.

Preserve trace ID, task ID, and request ID wherever emitted. Where records do not share an identifier, correlate by node, GPU ID, and the tightest common UTC interval and document the join rule. A DCGM limit alone does not show the requested limit or whether a request was clamped.

If any required data is missing, report the available power and workload observations but mark the cause of the transition or workload effect indeterminate.

Use reconciler metrics only as summaries

reconciler_request_duration_seconds measures a device request from its first dispatch through reconciler completion, including response wait time and retries. It does not include time waiting in the reconciler queue before first dispatch. For MaxLPS GPU-limit updates, record the rolling 15-minute observation count and mean lifecycle latency throughout the workload, then evaluate the resulting panels over the selected steady-state window:

sum(increase(reconciler_request_duration_seconds_count{request_type="gpu_policies_update"}[15m]))
sum(rate(reconciler_request_duration_seconds_sum{request_type="gpu_policies_update"}[15m]))
/
sum(rate(reconciler_request_duration_seconds_count{request_type="gpu_policies_update"}[15m]))

The count is the number of completed lifecycle observations, not a success count. The mean combines nodes and requests and cannot diagnose a specific event. Use the event-level chain above for queue time, node/GPU attribution, clamp behavior, retries, and readback. Use request_type="resource_group_update" when measuring resource-group update requests instead.

Do not use histogram_quantile() on this metric for sub-5-second p50, p95, or p99 values in DPS 0.8.3. This release uses default histogram buckets whose first useful upper bound is 5 seconds, so interpolation inside that bucket is too coarse for sub-second control-path claims. Use the raw p50, p95, and p99 endpoint statistics from the BMC health report for BMC preflight latency, and use request traces or logs when individual lifecycle timing is required.

Running the workload

Archive the effective control state using Part 4, start every evidence stream, then launch the workload across all 4 pilot racks (72 compute nodes) using the same orchestration tooling and submission flow used for the 3-rack baseline. DPS is now in the control path and will actively redistribute per-GPU power headroom across the resource group in response to live BMC telemetry.

What to record

Capture every Part 4 signal over a comparable steady-state window, plus the managed-only evidence below.

Managed-only evidence What it proves
Controller next_limit by node/GPU over time Shows requested dynamic redistribution.
DPS requested watts, effective set_limit, and clamp diagnostics Shows whether the requested target was accepted or reduced before dispatch.
Reconciler, Redfish, and readback events by request/node/GPU Shows queue time, dispatch timing, BMC outcome, and applied limit.
GPU allocation versus observed aggregate draw Shows whether the shared group sum is being enforced.
DPS /metrics series Cap movement and group-sum behavior can be cross-checked with PromQL.
Persistent dps-server and MaxLPS controller (prs container) logs The control loop continued emitting power-limit updates through the managed window.
Workload logs and configuration diff Rack count and the DPS control path were the only intended changes.

Any other workload configuration difference invalidates a direct comparison unless it is analyzed as another experimental variable.

Use the 1-second DCGM collection configured in Part 4 for durable, workload-correlated GPU and Grace CPU telemetry. The following nvidia-smi command is optional supplemental diagnostics on representative nodes; it does not replace the DCGM capture:

nvidia-smi \
  --query-gpu=timestamp,index,power.draw,power.limit,utilization.gpu \
  --format=csv \
  --loop-ms=200 > gpu-metrics.csv

The 200 ms value is the polling cadence, not a guarantee that the underlying power and utilization sensors refresh every 200 ms. Repeated readings are expected when a sensor’s sample period is slower, and high-rate polling can add overhead. Keep clocks synchronized, retain the raw CSV, and correlate it with applied limits, completions, queue depth, TTFT, TPOT, and timeout counts over aligned timestamps.

Repeat the managed run at -15 when needed

Complete and preserve the -30 managed run first. If its workload performance is unacceptable, change only MaxLPS headroom at runtime and verify the effective value before repeating the managed run:

dpsctl settings update --set prs_headroom_percent=-15
dpsctl settings list \
  | jq -e '.settings.prs_headroom_percent == "-15"'

The command persists -15 and starts a background control-domain refresh; no pod or server restart is required. Before starting the repeat workload, confirm the DPS server logs show the configuration-change refresh and that the MaxLPS control loop resumes ongoing power-limit updates. If the refresh fails, follow MaxLPS Headroom Updates and return to -30 before continuing. Because headroom is deployment-wide, perform this repeat only on an isolated pilot deployment or in the coordinated change window established in Part 1, Step 4.

Keep the resource-group policy, topology, workload, model, framework, batching, input distribution, load-generator mode, controlled load parameter, and measurement window unchanged. For an open-loop test, reuse the same target-QPS schedule; for a closed-loop test, reuse the same configured concurrency. This makes prs_headroom_percent the only intended difference between the two managed runs. Do not tune the resource-group policy in the same comparison.

Part 8: Evaluate Metrics

Outcome: A documented pass, partial pass, or fail decision backed by aligned compliance, throughput, latency, and efficiency evidence.

Compare only the steady-state windows selected using Part 4. Do not compare headline numbers unless request mix, workload configuration, load-generator mode, and controlled load parameter match.

Pilot success quick-check

Before reading the KPI tables, confirm the run satisfies every pass condition below. A miss on any row means the pilot did not meet its goals on this configuration.

Check Pass condition Source KPI in this Part
Experiment validity Aggregate offered-load schedule, timeout/retry rules, benchmark stop rules, request mix, and workload configuration match between compared runs Load and benchmark validity contract
Compliance Zero new compliance events in the managed run’s steady-state window Compliance events (Data center power and compliance KPIs)
Throughput capacity At a load that constrains the baseline, managed-run completed throughput is greater without violating service or power limits Aggregate output tokens/sec (Inference service KPIs)
Service stability Error rate matches baseline within ±0.1 percentage points Success / error rate (Inference service KPIs)
Envelope adherence Overall observed power ≤ binding data center constraint for ~100% of steady-state samples Overall observed power (Data center power and compliance KPIs)
Attribution evidence Required request-phase timestamps and per-update debug evidence are complete, or causal attribution is explicitly marked indeterminate Capture per-update debug evidence

Time alignment

  • Clock authority. Synchronize every sampling agent — BMC/BMS exporter, DCGM, inference server, load generator — to the cluster’s NTP/chrony source (typically the cluster head node or site-wide NTP server) before either run. All timestamps in this Part are interpreted relative to that authority. If a sampling agent cannot sync to the cluster source, document the offset and treat its data as advisory, not authoritative.
  • Use the same wall-clock sampling interval (or finer on the managed run) for power and service metrics.
  • If you change the client mode, prompt distribution, batching, aggregate offered-load schedule, aggregate concurrency, timeout/retry rules, or benchmark stop criteria, treat the run as a new experiment; do not attribute deltas to DPS.

Data center power and compliance KPIs

The definitions below assume you can obtain an overall facility or pilot-segment power budget (the data center constraint), provisioned electrical or thermal capacity for the GPUs or systems under test (from nameplate TDP, derated caps, or your capacity model), and metered or telemetry-derived actual draw for the same boundary. Map each to your monitoring source (BMS, rack PDU, BCM, or aggregated BMC).

KPI Definition What to expect in a successful pilot
Compliance events Distinct count of intervals (or events) where overall observed power exceeds any applicable data center constraint (contract limit, breaker headroom, pilot PowerDomain budget—use the tightest binding constraint you track). Target: zero new compliance events in the managed run’s steady-state window. The baseline may already be clean; the managed run must not introduce excursions.
Overall provisioned power Total system power (provisioned) — sum (or budgeted envelope) of provisioned capacity for the systems in scope (for example aggregate GPU or node TDP limits you could enforce statically). Between runs: provisioned total for the 4-rack fleet can be higher than for the 3-rack baseline because more hardware sits in the topology, while the same pilot envelope caps observed aggregate draw. You are measuring whether you provisioned more compute than naive sizing would have allowed under that envelope.
Overall observed power Total system power (observed utilization) — metered or telemetry-summed draw for the same boundary as provisioned and constraints. Must remain ≤ the binding data center constraint for essentially 100% of steady-state samples on the managed run (allow rare spikes only if your constraint definition already includes agreed crest factor). Expect observed at or below the pilot envelope you set in topology.
Capacity efficiency Power budget minus total system power (provisioned). Interpret as headroom between envelope and what you provisioned on paper. With MaxLPS-style sizing, this number is often smaller than in conservative MaxP-only planning because you deliberately provision closer to the envelope—that is expected if compliance stays clean.
Provisioning efficiency Total system power (provisioned) minus total system power (observed utilization). Interpret as stranded provisioned headroom not appearing as draw. You often want this gap to shrink when inference loads the fleet more evenly; a large persistent gap under load suggests under-utilization or caps binding unevenly.
Resource group efficiency Aggregated average power (W) divided by aggregated provisioned TDP limit (W) for all members of the pilot resource group, using the same time window. Ratios in roughly 0.35–0.85 are common for bursty inference once warm; values sustained below 0.25 under declared steady load suggest the workload or caps are not exercising the fleet, while values sustained above 0.95 leave little margin for spikes—watch compliance.
Node efficiency Node average power (W) divided by node provisioned TDP limit (W) per compute node, then compare distributions across racks. Expect a similar spread to baseline at matched load, with managed runs sometimes showing higher averages on more nodes because the control loop moves limits toward demand. Flag nodes stuck below 0.2 while peers approach limits—possible mapping, cooling, or telemetry issues.

If your tooling cannot yet compute one of the aggregates exactly as defined, document the substitute signal (for example sum of BMC inlet metrics vs. rack PDU) and apply it consistently on both runs.

Inference service KPIs

These metrics come from the inference stack (vLLM, Triton, TensorRT-LLM, proprietary routers, etc.), not from DPS. Prefer server-reported token and latency counters over client-only estimates.

Interpret fixed-QPS and fixed-concurrency tests separately

A fixed-concurrency client usually submits replacement work as earlier requests complete. In an open-loop fixed-QPS test, the client continues offering requests at the configured rate independently of prior completions. For example, the MLPerf Inference Server scenario schedules queries according to a Poisson distribution at the target QPS.

Let offered_rate be the offered request rate and service_capacity be the sustainable completion rate. If power limits reduce service_capacity below offered_rate, requests accumulate faster than they complete. As offered_rate approaches service_capacity, ordinary variability can also make queue depth and tail latency grow rapidly. TTFT includes the wait before first-token delivery, so queueing and prefill pressure can make TTFT grow nonlinearly even when time per output token (TPOT) changes much less. TPOT is the average spacing between generated output tokens after the first token; an inter-token-latency distribution is related but is not automatically the same statistic.

For open-loop fixed-QPS runs, report aggregate offered or scheduled QPS, completed QPS, success and timeout counts, queue depth, admission/dequeue timestamps, prefill start/end, TTFT, and TPOT together. A large TTFT increase is evidence of queueing only when those signals support it; it is not by itself proof that DPS added equivalent control-path latency. Correlate the same timestamps with the per-update debug evidence.

KPI Definition What to expect in a successful pilot
Aggregate output tokens/sec Total output tokens completed per second across all replicas in scope (cluster-wide), measured over the steady-state window. Exclude failed generations if your stack exposes goodput separately. At an offered load that constrains the baseline, the managed run should complete more work without violating the power envelope or service limits. If both runs complete all offered work, equal throughput is expected; use a controlled QPS or concurrency sweep to compare capacity instead of inferring capacity from an underloaded point.
Aggregate total tokens/sec Input + output tokens per second if your economics or SLAs track both. Should move roughly in proportion to output tokens/sec unless the prompt length distribution changed.
Tokens/sec per concurrent user (or per active session) Aggregate output tokens/sec divided by a defined concurrency measure (open sessions, active clients, or in-flight requests—pick one and keep it fixed). Proxy for interactive saturation. At fixed concurrency, expect within ~±15% of the baseline value if power is not the dominant bottleneck; a large sustained drop warrants checking for cap-induced queueing or straggler GPUs.
Time to first token (TTFT) Latency from the load generator’s issue timestamp, request acceptance, or another explicitly named boundary to first output token. Use the same boundary in both runs and report p50/p95/p99. Expect p95 within ~1.2× of baseline p95 and p99 not more than ~1.5× baseline p99 when configuration is healthy. Investigate larger regressions together with offered/completed QPS, queue depth, timeouts, applied limits, and control-path latency.
Time per output token (TPOT) / inter-token latency (ITL) TPOT is the average interval across generated output tokens after first-token delivery. ITL is commonly a distribution of individual token intervals; name the statistic your tooling emits. Compare the same statistic and percentile or average between runs. TTFT growing much more than TPOT or ITL points toward queueing or prefill pressure, but confirm with queue and admission data.
End-to-end request latency Wall time for full response. Same percentile bands as TTFT/ITL for streaming workloads; for non-streaming, this is often the primary user-facing metric—use the same ~1.2× / ~1.5× p95/p99 gates unless your SLO says otherwise.
Success / error rate Fraction of requests completing without server-side error. Match baseline within ±0.1 percentage points on error rate; regressions here invalidate throughput comparisons.

The ±15%, 1.2×, and 1.5× bands above are starting heuristics for a first pilot read—not substitute contractual SLOs. Replace them with your own targets where you have historical production data.

How to document the outcome

  1. Compliance table — List each binding constraint, observed max, and count of excursions (baseline vs. managed).
  2. Throughput and latency table — Baseline vs. managed for aggregate tokens/sec, tokens/sec per concurrent user, and p50/p95/p99 for TTFT (or TTFT + E2E).
  3. Efficiency plots — Time series for overall observed vs. constraint, resource group efficiency, and (optional) per-rack node efficiency histograms.
  4. DPS overlay — Plot controller-requested, DPS-effective, and applied/read-back GPU limits against observed GPU power, keyed to the same timestamps as inference metrics.

Use the following one-page layout for the final comparison output. Each section should be a single paragraph, table, or plot — anything longer belongs in an appendix.

  1. Run metadata and validity controls — date, operator, state-bundle path, topology, resource group, policy, workload/model/framework, aggregate offered-load schedule, timeout/retry values, and benchmark stop rules.
  2. Steady-state windows — start/end timestamps for the baseline run and the managed run, plus total run duration.
  3. Clock authority — name of the NTP/chrony source used and any sampling agents whose offset was outside it.
  4. Pilot success quick-check — a copy of the Pilot success quick-check table from this Part with the actual pass/fail per row.
  5. Compliance summary — binding constraints, observed maxima, and excursion counts (baseline vs. managed).
  6. Power KPI table — baseline vs. managed values for each row of Data center power and compliance KPIs.
  7. Service KPI table — baseline vs. managed values for each row of Inference service KPIs.
  8. Per-update debug evidence — requested and returned limits, clamp diagnostics, processing timing, device request results, and correlated DCGM and workload timestamps for each sampled rack and every anomaly.
  9. Plots — overall observed power vs. constraint over time; requested, effective, and applied GPU limits vs. observed GPU power; host CPU utilization; and optional per-rack node-efficiency histograms.
  10. Verdict and next action — pass / fail / partial pass, plus the single next change to make if not a pass (per the discipline in Next Steps).

If compliance stays clean, aggregate output tokens/sec rises materially, and latency percentiles remain inside your agreed bands, you have a documented case that the pilot configuration met its goals. If not, capture the same package of telemetry before changing topology, policy, or workload shape so the next iteration is evidence-driven.

Cleanup

Outcome: The pilot resource group is deleted, the topology is inactive, original MaxLPS settings are restored, and sampled GPUs report their default power limits.

Reverse the configuration in this order:

  1. Delete the resource group.
  2. Deactivate the topology.
  3. Restore and verify the original MaxLPS settings.
  4. Verify representative GPU limits on every rack.
  5. Optionally uninstall DPS.

Step 1: Delete the resource group

DPS does not expose a standalone resource-group deactivate subcommand — deleting an active resource group automatically deactivates it first. See dpsctl resource-group delete.

dpsctl resource-group delete --resource-group maxlps-pilot

Step 2: Deactivate the topology

dpsctl topology deactivate --topology maxlps-pilot

See dpsctl topology deactivate for details.

At this point DPS is no longer in the power-control path for the pilot racks.

Step 3: Restore the original MaxLPS settings

Restore both values saved in Part 1, Step 4, then verify they were persisted:

ORIGINAL_MAXLPS_ENABLED="$(
  jq -er '.settings.prs_enabled' maxlps-pilot-original-settings.json
)"
ORIGINAL_MAXLPS_HEADROOM_PERCENT="$(
  jq -er '.settings.prs_headroom_percent' \
    maxlps-pilot-original-settings.json
)"
dpsctl settings update \
  --set "prs_enabled=${ORIGINAL_MAXLPS_ENABLED}" \
  --set "prs_headroom_percent=${ORIGINAL_MAXLPS_HEADROOM_PERCENT}"
dpsctl settings list \
  | jq -e \
    --arg enabled "${ORIGINAL_MAXLPS_ENABLED}" \
    --arg headroom "${ORIGINAL_MAXLPS_HEADROOM_PERCENT}" \
    '
      .settings.prs_enabled == $enabled and
      .settings.prs_headroom_percent == $headroom
    '

Keep or remove the saved-settings file according to your run-record retention policy only after the readback succeeds. Because the pilot resource group and topology are no longer active, restoring the global settings cannot change their GPU limits. If the restore changes headroom while MaxLPS power steering remains enabled, DPS asynchronously refreshes any other active control domains in the deployment; a no-op restore does not. Use the same coordinated change window as the pilot updates.

Step 4: Verify per-GPU caps returned to default

DPS does not actively reset per-GPU power.limit on deactivate; the pilot’s racks should already be at their default limit because the topology is no longer enforcing one, but a lingering cap from a failed activation, a manual nvidia-smi -pl ..., or another tool will not be cleared by dpsctl topology deactivate. Confirm on a representative compute node from each pilot rack:

nvidia-smi --query-gpu=index,power.default_limit,power.limit --format=csv

Every GPU should report power.limit equal to power.default_limit (within rounding). If any GPU shows a lower power.limit, reset it explicitly before moving on:

sudo nvidia-smi -i <gpu-index> -pl <default-limit-watts>

Repeat on each affected GPU. Do not proceed to Next Steps or to the optional uninstall below until every sampled GPU reads back its default limit.

Step 5: (Optional) Uninstall DPS

If you have more tuning passes planned (different policies or additional workloads from Next Steps), skip this section. DPS can remain installed indefinitely; with the resource group deleted and the topology deactivated, it is not in the power-control path and is safe to leave in place between runs.

Only if you are done with the pilot and want to remove DPS entirely:

helm uninstall dps -n dps

Optionally remove the namespace and any BMC secrets created for the pilot:

kubectl delete namespace dps

Next Steps

The pilot topology and decoupled resource group are intentionally small so you can prove behavior end-to-end. After Part 8 shows the outcome you want, work the three priorities below in order, re-running the compliance and service KPI captures from Part 8 after each change.

1. Model a larger, more complex topology

Expand beyond the 4-rack sketch to full rows, feeds, and redundancy paths your facility actually runs: additional PowerDomain boundaries, floor and rack PDUs, diversity groups, and any devices that were simplified or omitted here. The goal is to practice the same import → activate → verify loop on data that resembles a future data center rollout, so surprises surface in the lab—not the first production change window.

See Topologies for the entity model and Managing Topologies for operational procedures. Re-run the deployment-health gate from Part 1 and the BMC health gate (dpsctl verify bmc-health start --topology <name> --wait --summary-only) from Part 2 after each major topology change.

2. Tune power budgets and overprovisioning carefully

Many pilots still show observed draw below 100% of the binding budget during steady inference: that is normal when burstiness or conservative caps leave slack. Tune the topology model before you expand the pilot:

  • Topology PowerDomain — The operating envelope is expressed as the domain’s OperatingLimit (the cap DPS enforces against).
  • Topology overprovisioning — Overprovisioning is not a Helm chart setting: it is the optional numeric field OverProvisioningPercentage on the PowerDomain entity in your topology JSON. Per the topology JSON schema (PowerDomain definition), provisioning headroom scales with the operating limit as provisioning limit = operating limit × (1 + OverProvisioningPercentage / 100)—see the schema for exact typing, constraints, and the dependency that OverProvisioningPercentage requires an OperatingLimit. After editing the topology, re-import and activate it following Managing Topologies, then re-validate metrics before comparing runs.

Use small steps on OperatingLimit and OverProvisioningPercentage so DPS can hand out a little more effective GPU budget—only while aggregate telemetry and facility constraints prove you remain safe.

Treat this as an experiment matrix, not a single knob turn:

  • Change one family of settings at a time (PowerDomain’s OperatingLimit or PowerDomain’s OverProvisioningPercentage), apply it through the topology workflow, and re-run a comparable workload window.
  • Stay within roughly 10% of the true electrical or contractual limit on the binding constraint when you first push outward—enough to learn sensitivity without jumping past derated margins you have not modeled here.
  • Observe heavily: the same compliance, efficiency, and inference tables from Part 8 should improve or at least not regress; any increase in compliance events or tail latency is a signal to roll back that change before stacking the next one.

3. Experiment with different workloads

Repeat measurement passes with other inference models, batch sizes, concurrency levels, and prompt length distributions that matter for your road map—including bursty or seasonal patterns if you can simulate them. The goal is to learn where token throughput and latency stay acceptable as power policy moves, not to optimize a single demo model forever. Non-inference loads (training, stress tools) remain poor substitutes for the envelope story this runbook targets.

Once those three are stable, common production-readiness follow-ons are: integrating scheduling and provisioning (Managing Resource Groups); promoting Part 8 plots into dashboards and SLOs; running multi-hour or multi-day soaks; validating the topology model against as-built power; defining change control for PowerDomain OperatingLimit and OverProvisioningPercentage; and rehearsing failure and maintenance drills with DPS active. Sequence them by risk — topology fidelity and compliance first, then throughput tuning, then integration and operations.

    Copyright © 2026  NVIDIA Corporation