Rehearse the MaxLPS Power Pilot With the SDK Simulator#

Use the DPS SDK Pilot profile to practice the MaxLPS power pilot without physical hardware. Deploy a local Kubernetes cluster, configure DPS, replay a power trace, and compare simulated GPU power across a three-rack baseline and a four-rack managed run.

This guide complements Run the MaxLPS Power Pilot, which remains the procedure for evaluating real systems. For the power-management model, refer to MaxLPS Power Management.

What the Rehearsal Measures#

The SDK runs DPS and Power Steering against simulated hardware. Use the following boundaries when interpreting results.

Evidence

What You Can Learn

DPS application telemetry

Observe actual API health, topology state, resource-group membership, provisioned load, Redfish requests, and control-loop activity.

Power Steering telemetry

Observe actual controller scheduling, allocations, errors, and limit requests and confirmations against simulated endpoints.

GPU and node power, utilization, and limits

Observe values produced by the selected simulation model.

The built-in nemotron-12b workload replays a power trace. It does not launch an inference model. Job completion and simulated GPU utilization do not measure tokens per second, time to first token, or request latency.

A completed rehearsal demonstrates the configuration workflow and control behavior. The comparison shows three racks with workload activity in the baseline and four under MaxLPS, against the same configured power envelope.

Step 1: Obtain and Deploy the Pilot Profile#

Download and extract the DPS SDK customer package from the NVIDIA NVOnline Portal. Run the commands in this guide from the extracted SDK directory.

Prepare your host before deployment:

  • Use Linux (Ubuntu or Debian) or macOS, with Task installed and a running Docker daemon accessible to your user.

  • Allow dependency downloads and access to the container and chart registries referenced by the package, including nvcr.io. Resolve any image-pull or registry authentication errors before continuing.

  • Allow the setup tasks to install tools and configure /etc/hosts with elevated privileges. macOS setup uses Homebrew.

  • Make local ports 80, 443, and 5001 available for the cluster and registry. The workload helper also uses port 18443 for a temporary simulator connection.

  • Install curl and jq separately if needed. task setup does not install these helper dependencies.

  • Provide enough Docker memory, CPU, and disk for the monitoring stack, DPS, and simulator. The general SDK requirements in SDK Simulator User Guide are a starting point; a Pilot-specific minimum has not been established. Check for pending Pods, memory pressure, and restarts.

Install the SDK dependencies first:

task setup

Setup installs or checks Docker, kubectl, K3D, Helm, uv, and the Docker-based dpsctl wrapper. On Linux, the wrapper is installed in /usr/local/bin. On macOS, add its installation directory to your shell path before deployment:

export PATH="$HOME/.local/bin:$PATH"

Confirm the tools and deploy the Pilot profile:

command -v task docker kubectl helm dpsctl curl jq
docker info
task deploy:pilot
task simulator:wait-for-dps-server

The deployment creates the dps-sdk K3D cluster and installs DPS, Power Steering, a BMC simulator with a workload-model sidecar, Prometheus, Grafana, and the web UI. It does not import the pilot topology or run the paired workloads for you.

Configure the CLI endpoint and authenticate explicitly:

export DPSCTL_HOST=api.dps.sdk
export DPSCTL_PORT=80
export DPSCTL_INSECURE_TLS_SKIP_VERIFY=true
dpsctl login --username dps

Enter the local SDK password, dps. These connection settings apply to the local SDK deployment. The packaged wrapper runs the CLI image through Docker and maps SDK ingress hostnames inside the container.

Check the deployment and record the server version:

kubectl --context k3d-dps-sdk get pods -n dps
dpsctl server-version
dpsctl verify

Inspect reported status and diagnostics, including any status.ok field. Confirm that the server, agent, authentication, database, simulator, and monitoring components are available. Record the package version, CLI image tag from scripts/dpsctl-wrapper.sh, and deployed component images with your results.

The SDK does not deploy NVIDIA Base Command Manager, so its skipped connectivity check is expected. Verification can also report a missing dps-ui ingress because the SDK chart names it dps-sdk-ui. Check the actual ingresses and open the UI to distinguish that naming mismatch from a failure. Do not ignore API, authentication, database, or Pod readiness failures.

Use the following local services throughout the rehearsal.

Service

Address

Credentials

DPS API

api.dps.sdk:80

dps / dps

DPS web UI

Local UI

dps / dps

Grafana

Local Grafana

admin / dps

Prometheus

Local Prometheus

None

If a service does not resolve, inspect the cluster ingresses and rerun the hostname configuration task:

kubectl --context k3d-dps-sdk get ingress -n dps
task deploy:etc:hosts

Step 2: Import and Activate the Pilot Topology#

The Pilot profile uses the following inventory and configured limits.

Item

Configuration

Simulated hardware

Four GB300 racks, each with 18 compute nodes and 72 GPUs.

Total inventory

72 compute nodes and 288 GPUs. Names follow gb300-nvl-<rack>-compute<node>, from rack 001 through 004.

Baseline workload

54 nodes and 216 GPUs across racks 001 through 003. Rack 004 remains idle.

Managed workload

All 72 nodes and 288 GPUs across racks 001 through 004.

Root domain

PDU-Root, with OperatingLimit.PowerValue of 405,000 W.

Rack limits

135,000 W each, represented by FloorPDU-Rack01 through FloorPDU-Rack04.

Static load

7,416 W per rack, included in the topology budget.

The four racks share a root envelope equal to three rack limits.

Validate, import, and activate the topology:

dpsctl topology validate sim/maxlps-pilot.json
dpsctl topology import sim/maxlps-pilot.json
dpsctl topology activate --topology maxlps-pilot
dpsctl topology list --active

Confirm that maxlps-pilot is active and that each operation reports success. If import reports an existing topology, follow Step 9: Retain Results and Clean Up before starting a new paired run.

Check the inventory and simulated node connectivity:

dpsctl topology list-entities
dpsctl check connection --topology maxlps-pilot

Resolve inaccessible nodes or simulator DNS errors before proceeding.

Step 3: Confirm Monitoring With the Topology Active#

Open Prometheus and check its scrape targets. Confirm that DPS, Power Steering, and the simulator are being scraped successfully. Then run the following expressions in the Prometheus query view to verify the data required for the comparison.

  1. Confirm that the topology is present and active:

    topology_service_active_status{topology_name="maxlps-pilot"}

  2. Confirm that 72 distinct nodes have topology metadata:

    count(count by (device_id) (device_total_power_watts{
  device_type="ComputerSystem",topology_name="maxlps-pilot"
}))

  3. Confirm that 288 GPU power series are available:

    count(max by (Hostname, gpu, UUID) (DCGM_FI_DEV_POWER_USAGE{
  Hostname=~"gb300-nvl-00[1-4]-compute.*"
}))

  4. Confirm utilization samples for all four racks, including idle GPUs:

    DCGM_FI_DEV_GPU_UTIL{Hostname=~"gb300-nvl-00[1-4]-compute.*"}

  5. Confirm a root target of 405,000 W without a schedule or target override:

    nvgrid_system_status_load_target_watts{feed_tag="PDU-Root"}

Allow several collection cycles after activation. The Pilot values configure a 5-second DPS scrape interval, a 10-second simulator scrape interval, and 5-second topology, grid, and node polling. Verify Power Steering’s actual scrape interval in Prometheus. Healthy scrape targets do not guarantee that every required metric or dashboard query is populated.

Use the DPS Server Health Dashboard for application diagnostics. The Datacenter and Resource Group dashboards are optional views for inventory, allocation, and membership checks. Refer to DPS Sample Dashboards for query and power-boundary details.

Step 4: Deactivate and Restore Simulator Defaults#

Remove DPS from the power-control path before the baseline:

dpsctl topology deactivate --topology maxlps-pilot
dpsctl topology list --active
dpsctl resource-group list --active

Confirm that maxlps-pilot is absent from the active topology list and no resource group manages the pilot nodes. The simulator can retain power limits written during activation. Deactivation alone does not establish an unmanaged baseline.

Restart the Pilot simulator to restore its defaults:

kubectl --context k3d-dps-sdk rollout restart \
  deploy/bmc-surrogate-simulator-pilot -n dps
kubectl --context k3d-dps-sdk rollout status \
  deploy/bmc-surrogate-simulator-pilot -n dps --timeout=240s

After the simulator becomes ready, wait for fresh scrapes from the replacement Pod. Query the effective GPU limits in Prometheus:

DCGM_FI_DEV_POWER_MGMT_LIMIT{Hostname=~"gb300-nvl-00[1-4]-compute.*"}

The bundled GB300 simulator defaults to 1,400 W per GPU. Confirm that all 288 GPU limits report that value, rather than retained managed caps. Exclude old Pod series or wait for them to become stale before counting samples. Check GPU power, utilization, and DPS node metadata again with the expressions in Step 3: Confirm Monitoring With the Topology Active.

If limits remain capped, confirm that the topology is inactive and no other resource group or simulation is writing limits. Resolve that condition and repeat the reset before collecting a baseline. This reset is the simulator equivalent of restoring default GPU limits in the hardware procedure.

Step 5: Prepare Comparable Observation Windows#

Create a rehearsal record before starting the workloads. Retain the following information for both phases:

  • Package, CLI, server, controller, simulator, and dashboard versions.

  • Topology JSON, Pilot values, root and rack limits, and the MaxLPS bundle.

  • Trace name (nemotron-12b), TIME_SCALE, job identifiers, and node sets.

  • Collection and scrape intervals, workload start and end times, and selected observation windows in UTC.

  • Grafana datasource, topology, comparison offset, and active-GPU threshold.

Keep the trace, time scale, topology envelope, and collection settings consistent across both runs. The 54-node and 72-node replays exercise different simulated fleet sizes. They do not establish matched aggregate inference request demand.

The following commands use the task default, TIME_SCALE=1. A larger value accelerates replay. At TIME_SCALE=1, each run takes about 30 minutes. At TIME_SCALE=20, each run takes about 1 to 2 minutes. Choose a value that leaves enough time to observe metrics and controller behavior, and use it for both runs. The helper polls progress every 10 seconds and defaults to a 3-hour completion timeout. Record the baseline window during the run so you can select a matching managed window.

Use the same measured power envelope in both phases: GPU draw across all 72 topology nodes, including idle GPUs in rack four during the baseline. CPU power and static rack loads are outside the dashboard’s GPU draw, although they consume part of the modeled topology budget.

Step 6: Run the Three-Rack Baseline#

Recheck that the topology is inactive, no pilot group is active, and simulator limits are restored. Record a UTC timestamp and start the baseline:

date -u '+%Y-%m-%dT%H:%M:%SZ'
task pilot:workload:baseline TIME_SCALE=1 JOB_ID=pilot-baseline
date -u '+%Y-%m-%dT%H:%M:%SZ'

The helper submits the trace to 54 nodes across racks 001 through 003 and waits for completion. Expect confirmation that 54 nodes were accepted, progress updates, and a completion message.

If submission fails, check simulator readiness, curl, jq, port 18443, and the reported HTTP response. If progress times out, retain diagnostics and inspect the workload and caps before repeating.

Each submission helper stops existing simulator jobs before starting its job. Run the phases sequentially, without another simulation in parallel. If metrics are absent or caps were retained, correct the issue and repeat the baseline before proceeding.

Step 7: Enable MaxLPS and Run the Managed Phase#

Reactivate the topology and inspect the server’s bundled policy:

dpsctl topology activate --topology maxlps-pilot
dpsctl topology list --active
dpsctl policy bundle list
dpsctl policy bundle get MaxLPS

Confirm successful activation, then create, populate, and activate the resource group:

dpsctl resource-group create --resource-group maxlps-pilot --external-id 1 \
  --dpm-enable --prs-enabled --shared-gpu-enable --policy-bundle MaxLPS
dpsctl resource-group add --resource-group maxlps-pilot \
  --entities "$(task pilot:nodes:72)"
dpsctl resource-group activate --resource-group maxlps-pilot \
  --allow-reprovision --sync
dpsctl resource-group list --active

Inspect status and diagnostics from each command. Confirm the following fields for the active group before submitting the workload.

Field

Expected Value

group_name

maxlps-pilot

activation_status

active

policy_bundle

MaxLPS

dpm_enable, prs_enabled, shared_gpu_enable

All true

resource_names

All 72 pilot compute nodes.

Run the managed phase with the same time scale as the baseline:

date -u '+%Y-%m-%dT%H:%M:%SZ'
task pilot:workload:managed TIME_SCALE=1 JOB_ID=pilot-managed
date -u '+%Y-%m-%dT%H:%M:%SZ'

Confirm acceptance and completion for 72 nodes. Preserve the run output and group configuration with your timestamps.

Step 8: Compare the Results in Grafana#

After both runs, open DPS MaxP Window Comparison in Grafana. Comparison means the earlier baseline window. Current means the selected managed window, even after both runs have finished.

Configure the page with your recorded windows:

  1. Select the provisioned Prometheus datasource under Metrics source.

  2. Select maxlps-pilot under Datacenter / topology.

  3. Set Grafana’s time picker to an absolute range covering the managed observation window. Use UTC to match your recorded timestamps.

  4. Set Comparison window to the positive time difference between the managed and baseline window starts. The entire current range shifts backward by this offset. Verify that shifting the managed end by the same amount reaches the recorded baseline end.

  5. Set Active GPU threshold to More than 5%. The count uses each GPU’s average utilization over the window. Any activity can include the simulated idle fourth rack because idle utilization is nonzero.

  6. Confirm that both sets of summary cards, node power charts, and four named rack bars populate. Save the dashboard’s absolute URL and a screenshot.

The comparison offset is the difference between window starts. It is not the baseline duration or the gap between runs. The dropdown defaults to one day and accepts custom Prometheus durations, such as 2m or 125s.

The following example illustrates equal-duration windows separated by a 2-minute offset. Use your own timestamps and corresponding loaded trace portions; these example times do not identify an acceptance window.

Selection

Example Value

Baseline observation

October 14, 9:04:30 p.m. to 9:05:30 p.m. UTC.

Managed observation and Grafana absolute range

October 14, 9:06:30 p.m. to 9:07:30 p.m. UTC.

Comparison window

2m

Datacenter / topology

maxlps-pilot

Active GPU threshold

More than 5%

You can also supply the custom offset as var-baseline_offset in the saved dashboard URL. For this example, retain the following query parameters, alongside the absolute from and to parameters and datasource selection:

  • var-topology=maxlps-pilot

  • var-baseline_offset=2m

  • var-active_gpu_util_threshold=5

Interpret the Power Envelope#

Use the following definitions when reading the comparison.

Signal

Meaning

Power envelope

Operating target from the Pilot’s single PowerDomain, PDU-Root.

GPU draw and node power charts

DCGM_FI_DEV_POWER_USAGE across all 288 GPUs. A node chart sums its GPU draw, rather than total BMC-reported node power.

Active GPUs

Count of GPUs whose window-average DCGM_FI_DEV_GPU_UTIL exceeds the selected threshold.

Mean power utilization

Mean GPU draw divided by the root envelope. It is not a percentage of GPU maximum power or an applied GPU cap.

Mean power headroom

Root envelope minus mean GPU draw. CPU power and rack static loads still consume part of the modeled budget.

Observed power change by rack

Relative change in mean GPU draw from the baseline, displayed as a percentage. An initially idle rack can have a large relative increase.

Power utilization change by rack

Change in mean GPU draw divided by that rack’s 135 kW target, in percentage points. The panel displays a percent symbol.

Active GPU change by rack

Absolute count change. The fleet Active GPU change card instead shows a percentage.

The peak utilization value sums individual GPU peak powers, which need not occur simultaneously. It is not an exact simultaneous fleet peak.

Check the Rehearsal Outcome#

Use the following observations together with successful workload completion and verified resource-group configuration.

Observation

Expected Result in Loaded Windows

Power envelope

405 kW in both windows. In the inactive baseline, this is a reference target rather than evidence of enforcement.

Active GPU counts

Comparison active GPUs is 216, Current active GPUs is 288, and Active GPU change is approximately +33.3%.

Rack activity

Four named rack bars populate. Active GPU change by rack is zero for racks one through three and +72 for rack four.

Simulated power behavior

Rack four contributes loaded GPU draw in the managed window. The first three racks can draw less under managed caps.

Exact power changes depend on trace phase, caps, and window selection. Do not require a particular reduction percentage.

Use the following checks for unexpected results.

Symptom

Recovery

Empty comparison cards

Check the absolute range, offset, and availability of baseline telemetry. The default 1-day offset usually does not reach a run you just recorded.

Empty current cards

Check the managed window, datasource, topology, and the required metric series from Step 3.

Equal active GPU counts

Select More than 5%, inspect the loaded windows, and verify which node set received each workload.

Missing rack bars or wrong topology

Check topology selection, DPS node metadata, hostname matching, and the packaged dashboard version.

Zero or unexpected envelope

Check PDU-Root, schedules, target overrides, and root-domain metadata.

Power appears capped in the baseline

Restore simulator defaults and repeat the baseline, then select a matching managed window.

Step 9: Retain Results and Clean Up#

If desired, preserve the configuration and version record, workload output, active group status, membership, and both run timestamps. Retain the selected observation windows, computed offset, Grafana variables, saved absolute URL, and a screenshot or exported comparison. A dashboard view does not itself retain an evidence package.

Delete the resource group, then deactivate the topology:

dpsctl resource-group delete --resource-group maxlps-pilot
dpsctl topology deactivate --topology maxlps-pilot
dpsctl resource-group list --active
dpsctl topology list --active

Confirm success in the output and active lists. For another rehearsal, repeat the simulator reset in Step 4: Deactivate and Restore Simulator Defaults before collecting a new baseline. You can reuse the imported topology; skip its import step on a repeat run.

When you finish with the local environment, optionally remove the SDK cluster. This deletes its workloads and locally retained monitoring data:

task deploy:cluster:delete

Optional Follow-On: Move to the Hardware Pilot#

Use MaxLPS Power Pilot to plan the real-system evaluation and Run the MaxLPS Power Pilot to execute it.