Troubleshoot the MaxLPS power pilot#

Choose the question that matches the failed pilot step. Return to MaxLPS Power Pilot to choose another pilot step. For failures outside this pilot, refer to Troubleshooting.

Why Does DPS Deployment Verification Fail?#

Use this section when dpsctl verify reports an unhealthy component during Step 1 of the runbook.

To recheck a single component after a fix, pass its flag — for example dpsctl verify --database. Any combination of --dps-server, --database, --auth, --ui, and --bcm is valid.

  • dps_server unhealthy — Check the DPS server pod is Running; kubectl -n dps logs statefulset/dps-server usually shows the underlying error.

  • database unhealthy — Check the Postgres pod and the database connection secret; a failed ping is almost always reachability or credentials, not load.

  • auth unhealthy — For LDAP, verify ServerURL, BindDN, BindPassword, and CA/client certificates. For JWT, verify the private key file is mounted and readable.

  • ui unhealthy or missing — Confirm the dps-ui Ingress exists in the same namespace as the DPS server with at least one rule and a non-empty host.

  • bcm unhealthy — If BCM is expected, confirm the credential Secret, network path, and API access. If BCM is intentionally omitted, refer to Deployment Component Unhealthy for the no-BCM output and gate this step on the other components.

Why Does Topology Validation or Import Fail?#

Use this section when dpsctl topology validate <topology-file>.json or dpsctl topology import <topology-file>.json fails during Part 2, Step 5 of the runbook.

Local JSON syntax errors and JSON-schema failures print instance location (or line, column, and offset) and do not produce validation_errors. A non-empty top-level Policies array is rejected before the server call with legacy topology policies are not supported; create policy bundles with dpsctl policy bundle upsert.

When dpsctl topology validate reaches ValidateTopology, it prints ValidationError entries. Match failures by the error field:

  • invalid_model — an entity Type/Model combination or topology structure is not valid for the device registry. Refer to Import a Topology.

  • device_not_found — an entity’s Type/Model pair is not in the DPS device registry (typo in Model, or the device was not seeded). Run dpsctl device list and correct the Model string to an exact match.

  • invalid_name — an entity or topology name does not satisfy the 3–63-character alphanumeric and hyphen rule.

  • legacy topology policies are not supported — the JSON contains a non-empty legacy Policies array. Remove it; DPS 0.9 rejects topology-embedded policies.

  • invalid_secret_name — the Redfish.SecretName is not a valid Kubernetes Secret name. Use 2–253 lowercase letters, digits, hyphens, or dots, and start and end with a lowercase letter or digit.

  • duplicate_entity — two entities share the same Name in the file. Make each node name unique.

  • referenced_entity_not_found / referenced_topology_entity_not_found — a topology entry lists a child that has no matching entity block. Compare the Topology.Entities[].Children list against the top-level Entities list.

  • self_reference / circular_dependency — a topology entity lists itself or creates a cycle through its children. Rebuild the parent/child chain so every leaf is reached exactly once from the root.

  • disconnected_graph — one or more compute nodes are not reachable from the topology root. Every compute node must be a descendant of the topology root entity.

  • invalid_connection — a parent entity cannot legally have the given child device type (for example, a rack entity listing another rack as a child). Correct the parent/child relationship to match the reference topology examples.

Fix the file, then rerun dpsctl topology validate until it prints Topology validation passed.

If dpsctl topology import <topology-file>.json fails:

  • failed to create topology — Run dpsctl topology list first. If maxlps-pilot already exists, verify whether it is the expected topology; retrying the unchanged import cannot resolve the name collision. If the name is absent, collect the server and database logs.

  • failed to add topology entities — Check the DPS server and dps-postgresql logs, then retry only after identifying and correcting the failure.

  • Entity create stream errors — dpsctl topology import creates missing entities before topology creation. Inspect the returned gRPC status and server logs.

BMC Health Check#

Use this section when dpsctl verify bmc-health reports issues during Part 2, Step 6 or Part 2, Step 7 of the runbook.

Use the issues list in the dpsctl verify bmc-health report as the starting point. Each issue includes a node, resource, code, observed value, threshold, and message. The BMC health check guide covers safe mode selection, report interpretation, and recovery. The dpsctl verify bmc-health reference documents every command flag and the asynchronous task model.

After fixing an affected node, scope the retry to that node before rerunning the full topology check:

dpsctl verify bmc-health start --topology maxlps-pilot --nodes <node-name> --force-writes --expected-edpp-pct 100 --samples-per-telemetry 2500 --telemetry-interval 500ms --per-node-timeout 1h --wait --summary-only

If a power-write failure looks like slow asynchronous BMC readback rather than a hard incompatibility, rerun the affected node with a larger write-resolution timeout. Keep the telemetry interval unchanged; increasing it extends the time between telemetry samples and does not extend the power-write readback window.

dpsctl verify bmc-health start --topology maxlps-pilot --nodes <node-name> --force-writes --expected-edpp-pct 100 --samples-per-telemetry 2500 --telemetry-interval 500ms --write-resolution-timeout 5s --per-node-timeout 1h --wait

BMC Unreachable or Firmware Validation Failed#

Issue codes: BMC_UNREACHABLE, FIRMWARE_VALIDATION_FAILED

The BMC did not respond to the Redfish service root, the fallback BMC ping failed, or firmware validation failed.

On systems with B200 or B300 GPUs, a standalone FIRMWARE_VALIDATION_FAILED issue with message: firmware validation failed is a known DPS 0.8.x health-check false failure. If the affected B200 or B300 node is still reachable, this is the only SEVERITY_ERROR for that node, and no other BMC health checks failed, document the exception and continue the runbook. This exception will be removed in a future DPS version. Treat FIRMWARE_VALIDATION_FAILED on other hardware, or any firmware failure combined with BMC_UNREACHABLE or another SEVERITY_ERROR, as a blocker.

To remediate:

  1. Confirm BMC network reachability from the cluster.

  2. Confirm the Kubernetes BMC credential secret for the affected node.

  3. Confirm the Day 0 BMC compatibility assessment from the runbook prerequisites.

  4. Attempt a BMC reset if compatibility and credentials are correct.

  5. Review the hardware model if the compatibility assessment shows the required GB200 or GB300 Redfish endpoints but the model is different. DPS might support the node through a custom device plugin. For example, a Supermicro BMC based on NVBMC and OEM extensions for HGX B300 can expose the environment metrics endpoints needed for TGP power settings and metrics, and might be supportable under the GB200 plugin with a custom device definition.

  6. Contact the NVIDIA DPS support team if the node appears supportable through a custom device definition.

  7. Exclude the node if the BMC remains unreachable. DPS cannot operate on the affected node.

Power Limit Drift#

Issue code: IB_OOB_LIMIT_DRIFT

The in-band and out-of-band BMC power limits do not match.

To remediate:

  1. Rerun a node-scoped read-only health check to confirm the drift. Telemetry interval and write-resolution settings do not change the in-band and out-of-band comparison.

  2. Query the current, default, minimum, and maximum power limits before choosing the reset value:

    nvidia-smi -q -d POWER | grep -E 'Default Power Limit|Max Power Limit|Min Power Limit|Current Power Limit'

  3. Reset the in-band GPU power limits to the default or maximum TGP:

    sudo nvidia-smi -pl <watts>      # Set all GPUs.
sudo nvidia-smi -i 0 -pl <watts> # Set one GPU.

  4. Rerun the node-scoped health check.

  5. Attempt a BMC reset or node restart if drift persists.

  6. Exclude the node if the issue cannot be resolved. DPS cannot operate on the affected node.

Power Write or Restore Mismatch#

Issue codes: POWER_WRITE_READBACK_MISMATCH, POWER_LIMIT_RESTORE_FAILED

DPS wrote a power limit through the BMC, but the readback did not converge to the requested value, or the probe could not verify restoration to the original limit.

To remediate:

  1. Rerun the affected node with --write-resolution-timeout 5s and inspect the full power_writes evidence. Do not increase --telemetry-interval for a write-readback failure.

  2. Use the DPS BMC credentials to manually test Redfish setpoint writes on the affected node if the issue remains.

  3. Follow the Redfish API guide examples for PowerLimitWatts.SetPoint.

  4. Verify TGP, TMP where available, and TCP where available.

  5. Attempt a BMC reset if manual writes fail or do not read back correctly.

  6. Exclude the node if the issue cannot be resolved. DPS cannot operate on the affected node.

WPPS or EDPp State#

Issue codes: WPPS_PROFILES_ACTIVE, WPPS_INACCESSIBLE, WPPS_RESET_FAILED, EDPP_BELOW_REFERENCE

Workload power profile settings (WPPS) or EDPp settings differ from the expected default state. No workload power profile should be active, and EDPp should report the expected current value of 100% for this pilot gate. WPPS can also affect the EDPp setpoint.

To remediate:

  1. Reset WPPS through both in-band and out-of-band paths where applicable.

  2. Reset EDPp directly where supported if EDPp remains below 100%.

  3. Attempt a BMC reset if the settings continue to persist.

  4. Document the node, active settings, and measured values if no immediate fix is available. These settings affect MaxLPS performance metrics and baselines.

BMC Health#

In DPS 0.9, both dpsctl check connection and dpsctl verify bmc-health start require an active topology. If either reports no agent configuration available, confirm the topology is active and the agent is configured. Follow Activate the Topology Before Checking BMCs, then Deactivate the Topology for the Baseline before running the unmanaged baseline.

For BMC compatibility, access mapping, health-check commands, latency tuning, and issue-specific recovery, refer to BMC Readiness and Health Guide.

What If Topology Activation Fails?#

Use this section when dpsctl topology activate --topology maxlps-pilot fails during Step 3 or Step 6 of the runbook.

  • topology maxlps-pilot is already active — activation was rerun after it succeeded. Run dpsctl tp list --active to verify, then continue without deactivating the topology.

  • topology <name> not found / load-topology error — the value passed to --topology does not match any imported topology. Confirm the Topology.Name in the imported file and rerun dpsctl tp list.

What If Resource Group Creation Fails?#

Use this section when dpsctl resource-group create fails during Step 6 of the runbook.

  • resource group already exists — a resource group named maxlps-pilot is already in the database. Run dpsctl rg list and verify its configuration. Reuse it only if it is the expected pilot group; otherwise, stop and choose a different name.

  • resource-group name validation error — the --resource-group value contains disallowed characters. Use lowercase letters, digits, and -, matching the naming convention used elsewhere in this runbook.

  • policy bundle "MaxLPS" not found — Run dpsctl policy bundle list and confirm that MaxLPS appears. If it does not, stop and reconcile the DPS installation before retrying. Do not replace it with a pilot-specific bundle.

  • failed to create resource group — generic database failure. Check the DPS pod logs and the dps-postgresql pod, then retry once the database is healthy.

What If I Cannot Add Nodes to the Resource Group?#

Use this section when dpsctl resource-group add fails during Step 6 of the runbook.

  • no active topologies — the topology activation step in Step 6 was skipped or rolled back. Run dpsctl tp list --active, then return to that runbook step instead of activating from this troubleshooting page.

  • some entities are not in the active topology — one or more node names in --entities are not part of the active topology. Compare the list against dpsctl tp list --active and correct the request. If the active topology omitted an intended node, create and activate a corrected replacement topology.

  • some entities are already in the resource group — those nodes are already members of maxlps-pilot. Remove the duplicate names from the add request.

What If Resource Group Activation Fails?#

Use this section when dpsctl resource-group activate fails during Step 6 of the runbook.

  • resource group maxlps-pilot is already active — activation was rerun after it succeeded. Run dpsctl rg list --active to verify, then continue without deleting the resource group.

  • resource group maxlps-pilot has no devices — the resource-group add step did not land. Run dpsctl rg list and confirm resource_names is populated before retrying.

  • power budget exceeded for resource group maxlps-pilot, but reprovision is false — admission detected a new or worsened overload in the proposed device or ancestor allocation. Capture the effective policy, active resource groups, topology limits, and reported device metadata. The Power Steering headroom setting does not determine admission.

  • failed to activate resource group with per-node failures in the --sync response — run dpsctl verify bmc-health start --topology maxlps-pilot --nodes <node1>,<node2> --skip-writes --wait --summary-only against the failing nodes. If overall status.ok is false, the server cleans up partially activated nodes. If the request succeeds under its partial-activation rule, accepted nodes remain active. Query dpsctl rg list --active before deciding which nodes require remediation.

How Do I Check Power Steering?#

Use this section when the Power Steering controller is not Ready, reports reconciliation errors, or the observed BMC/DCGM limit time series does not move during Step 6 of the runbook.

Do not use the older New power limits: log phrase as a health check. Instead:

  1. Verify the resource group is active with Power Steering enabled.

  2. For the bundled deployment, confirm the Power Steering settings are enabled in the Helm values. Refer to DPS Power Steering Controller.

  3. Verify the controller pod is Ready, then check its logs for errors:

    kubectl get pods -n dps \
  -l app.kubernetes.io/component=controller,app.kubernetes.io/part-of=power-steering
kubectl logs -n dps \
  -l app.kubernetes.io/component=controller,app.kubernetes.io/part-of=power-steering \
  -c controller --since=15m --tail=200

  4. Confirm at least one GPU limit changes in the BMC or DCGM time series while the workload is running. If limits remain static, capture the resource-group configuration, controller logs, and matching telemetry window before changing policy or workload settings.