layout-3 layout-6 layout-1 layout-top
layout-top layout-6 layout-1

Troubleshooting the NVL72 inference power pilot

Use this page when a command in the Inference power pilot: NVL72 fleet power management runbook fails or returns an unhealthy state. Start with the section that matches the step you were running.

Deployment Verification

Use this section when dpsctl verify reports an unhealthy component during Part 1, Step 3 of the runbook.

To re-check a single component after a fix, pass its flag — for example dpsctl verify --database. Any combination of --dps-server, --database, --auth, --ui, and --bcm is valid.

  • dps_server unhealthy — Check the DPS server pod is Running; kubectl -n dps logs statefulset/dps-server usually shows the underlying error.
  • database unhealthy — Check the Postgres pod and the database connection secret; a failed ping is almost always reachability or credentials, not load.
  • auth unhealthy — For LDAP, verify ServerURL, BindDN, BindPassword, and CA/client certificates. For JWT, verify the private key file is mounted and readable.
  • ui unhealthy or missing — Confirm the dps-ui Ingress exists in the same namespace as the DPS server with at least one rule and a non-empty host.
  • bcm unhealthy — Confirm the BCM credential secret is present, the URL is reachable from the cluster, and the account has API access. If BCM is not deployed, expect skipped: true instead.

Topology Validation and Import

Use this section when dpsctl topology validate topology.json or dpsctl topology import topology.json fails during Part 2, Step 5 of the runbook.

dpsctl topology validate prints a list of ValidationError entries; the error field identifies the class of failure. The most common ones on a hand-authored topology file:

  • invalid_model — the file does not match the topology JSON schema (missing top-level Topology or Entities, wrong field casing, malformed JSON). Re-check against Managing Topologies.
  • device_not_found — an entity’s Type/Model pair is not in the DPS device registry (typo in Model, or the device was not seeded). Run dpsctl device list and correct the Model string to an exact match.
  • invalid_name — an entity, topology, or policy name has disallowed characters. Use lowercase letters, digits, and -.
  • invalid_secret_name — the Redfish.SecretName contains characters that are not valid for a Kubernetes Secret name. Use lowercase letters, digits, -, and ..
  • duplicate_entity — two entities share the same Name in the file. Make each node name unique.
  • referenced_entity_not_found / referenced_topology_entity_not_found — a topology entry lists a child that has no matching entity block. Compare the Topology.Entities[].Children list against the top-level Entities list.
  • self_reference / circular_dependency — a topology entity lists itself or creates a cycle through its children. Rebuild the parent/child chain so every leaf is reached exactly once from the root.
  • disconnected_graph — one or more compute nodes are not reachable from the topology root. Every compute node must be a descendant of the topology root entity.
  • invalid_connection — a parent entity cannot legally have the given child device type (for example, a rack entity listing another rack as a child). Correct the parent/child relationship to match the reference topology examples.

Fix the file, then re-run dpsctl topology validate until it prints Topology validation passed.

If dpsctl topology import topology.json fails:

  • failed to create topology / failed to upsert entities / failed to add topology entities — generic database failure. Check the DPS pod logs and the dps-postgresql pod, then retry once the database is healthy.

BMC Health Check

Use this section when dpsctl verify bmc-health reports issues during Part 2, Step 6 or Part 2, Step 7 of the runbook.

Use the issues list in the dpsctl verify bmc-health report as the starting point. Each issue includes a node, resource, code, observed value, threshold, and message. The dpsctl verify bmc-health reference documents the command flags and asynchronous task model.

After fixing an affected node, scope the retry to that node before re-running the full topology check:

dpsctl verify bmc-health start --topology maxlps-pilot --nodes <node-name> --force-writes --expected-edpp-pct 100 --samples-per-telemetry 2500 --telemetry-interval 500ms --wait --summary-only

If a failure looks like slow BMC readback rather than a hard incompatibility, rerun the check with a larger telemetry interval and write-resolution timeout:

dpsctl verify bmc-health start --topology maxlps-pilot --force-writes --expected-edpp-pct 100 --samples-per-telemetry 2500 --telemetry-interval 5s --write-resolution-timeout 5s --wait --summary-only

BMC Unreachable or Firmware Validation Failed

Issue codes: BMC_UNREACHABLE, FIRMWARE_VALIDATION_FAILED

The BMC did not respond to the Redfish service root, the fallback BMC ping failed, or firmware validation failed.

On systems with B200 or B300 GPUs, a standalone FIRMWARE_VALIDATION_FAILED issue with message: firmware validation failed is a known DPS 0.8.x health-check false failure. If the affected B200 or B300 node is still reachable, this is the only SEVERITY_ERROR for that node, and no other BMC health checks failed, document the exception and continue the runbook. This exception will be removed in a future DPS version. Treat FIRMWARE_VALIDATION_FAILED on other hardware, or any firmware failure combined with BMC_UNREACHABLE or another SEVERITY_ERROR, as a blocker.

To remediate:

  1. Confirm BMC network reachability from the cluster.
  2. Confirm the Kubernetes BMC credential secret for the affected node.
  3. Confirm the Day 0 BMC compatibility assessment from the runbook prerequisites.
  4. Attempt a BMC reset if compatibility and credentials are correct.
  5. Review the hardware model if the compatibility assessment shows the required GB200 or GB300 Redfish endpoints but the model is different. DPS might support the node through a custom device plugin. For example, a Supermicro BMC based on NVBMC and OEM extensions for HGX B300 can expose the environment metrics endpoints needed for TGP power settings and metrics, and might be supportable under the GB200 plugin with a custom device definition.
  6. Contact the NVIDIA DPS support team if the node appears supportable through a custom device definition.
  7. Exclude the node if the BMC remains unreachable. DPS cannot operate on the affected node.

Power Limit Drift

Issue code: IB_OOB_LIMIT_DRIFT

The in-band and out-of-band BMC power limits do not match.

To remediate:

  1. Rerun the health check with --telemetry-interval 5s and --write-resolution-timeout 5s to rule out slow BMC readback.

  2. Query the current, default, minimum, and maximum power limits before choosing the reset value:

    nvidia-smi -q -d POWER | grep -E 'Default Power Limit|Max Power Limit|Min Power Limit|Current Power Limit'

  3. Reset the in-band GPU power limits to the default or maximum TGP:

    sudo nvidia-smi -pl <watts>      # Set all GPUs.
sudo nvidia-smi -i 0 -pl <watts> # Set one GPU.

  4. Rerun the node-scoped health check.

  5. Attempt a BMC reset or node restart if drift persists.

  6. Exclude the node if the issue cannot be resolved. DPS cannot operate on the affected node.

Power Write or Restore Mismatch

Issue codes: POWER_WRITE_READBACK_MISMATCH, POWER_LIMIT_RESTORE_FAILED

DPS wrote a power limit through the BMC, but the readback did not converge to the requested value, or the probe could not verify restoration to the original limit.

To remediate:

  1. Rerun the health check with --telemetry-interval 5s and --write-resolution-timeout 5s.
  2. Use the DPS BMC credentials to manually test Redfish setpoint writes on the affected node if the issue remains.
  3. Follow the Redfish API guide examples for PowerLimitWatts.SetPoint.
  4. Verify TGP, TMP where available, and TCP where available.
  5. Attempt a BMC reset if manual writes fail or do not read back correctly.
  6. Exclude the node if the issue cannot be resolved. DPS cannot operate on the affected node.

WPPS or EDPp State

Issue codes: WPPS_PROFILES_ACTIVE, WPPS_INACCESSIBLE, WPPS_RESET_FAILED, EDPP_BELOW_REFERENCE

Workload power profile settings (WPPS) or EDPp settings differ from the expected default state. No workload power profile should be active, and EDPp should report the expected current value of 100% for this pilot gate. WPPS can also affect the EDPp setpoint.

To remediate:

  1. Reset WPPS through both in-band and out-of-band paths where applicable.
  2. Reset EDPp directly where supported if EDPp remains below 100%.
  3. Attempt a BMC reset if the settings continue to persist.
  4. Document the node, active settings, and measured values if no immediate fix is available. These settings affect MaxLPS performance metrics and baselines.

BMC Latency

Issue codes: TELEMETRY_LATENCY_MAX_HIGH, TELEMETRY_LATENCY_AVG_HIGH, POWER_GET_LATENCY_AVG_HIGH, POWER_SET_LATENCY_AVG_HIGH, POWER_LATENCY_HIGH, EDPP_GET_LATENCY_AVG_HIGH, EDPP_LATENCY_HIGH

Latency failures appear in issue codes and latency summaries.

To remediate:

  1. Use the saved full report from the health-check step to review per-node latency summaries.

  2. Compare the summaries against the MaxLPS readiness targets from the health-check step in the pilot runbook.

  3. Treat the BMC health-check issue-code thresholds as coarse failure thresholds: 5 seconds average for power GET, power SET, EDPp GET, and telemetry latency, and 10 seconds maximum telemetry latency.

  4. Tune the MaxLPS control-loop interval from the highest measured p99 across the BMC telemetry and power-control paths so the controller does not run faster than the BMCs can reliably answer. As an initial value, use the greater of the 10-second chart default or the measured p99 rounded up to whole seconds. For example, keep 10 when the highest p99 is 8 seconds, and use 30 when it is around 30 seconds. The prs key below is the chart’s implementation identifier.

    prs:
  config:
    loopIntervalSeconds: 10

    Changing the MaxLPS control-loop interval changes how quickly power steering responds to workload and telemetry changes. Record the value used for each pilot run because it can affect workload-performance comparison in Part 8.

  5. Assess cluster-wide BMC client connection pressure if latency exceeds the readiness targets.

  6. Keep GB200 and GB300 systems at no more than four BMC client connections, including two DPS client connections.

  7. Use session token authentication instead of basic authentication, and use keep-alive to maintain the connection.

  8. Review the DMTF Redfish Host Interface Specification for Redfish connection guidance.

No Nodes Probed

Issue code: NO_NODES_PROBED

The server could not build a usable BMC health report for any requested node.

To remediate:

  1. Confirm the topology name.
  2. Confirm the --nodes values.
  3. Confirm that the requested nodes belong to the imported topology.
  4. Rerun the health check.

Topology Activation

Use this section when dpsctl topology activate --topology maxlps-pilot fails during Part 2, Step 8 or Part 6, Step 1 of the runbook.

  • topology maxlps-pilot is already active — activate was re-run after it succeeded. Skip ahead to dpsctl tp list to verify, or deactivate first with dpsctl topology deactivate --topology maxlps-pilot.
  • topology <name> not found / load-topology error — the value passed to --topology does not match any imported topology. Confirm the name used in topology.json and re-run dpsctl tp list to see what DPS has.

Resource Group Creation

Use this section when dpsctl resource-group create fails during Part 6, Step 4 of the runbook.

  • resource group already exists — an RG named maxlps-pilot is already in the database. Run dpsctl rg list to confirm, then either reuse it or remove it first with dpsctl resource-group delete --resource-group maxlps-pilot.
  • resource-group name validation error — the --resource-group value contains disallowed characters. Use lowercase letters, digits, and -, matching the naming convention used elsewhere in this runbook.
  • failed to create resource group — generic database failure. Check the DPS pod logs and the dps-postgresql pod, then retry once the database is healthy.

Adding Resources to the Resource Group

Use this section when dpsctl resource-group add fails during Part 6, Step 5 of the runbook.

  • no active topologies — the topology activation step in Part 6, Step 1 was skipped or rolled back. Run dpsctl tp list --active and activate the pilot topology before retrying.
  • some entities are not in the active topology — one or more node names in --entities are not part of the active topology. Compare the list against dpsctl tp list --active and correct the typo, or reimport the topology JSON.
  • some entities are already in a resource group — those nodes belong to another RG. Remove them from the other RG first, or delete that RG.

Resource Group Activation

Use this section when dpsctl resource-group activate fails during Part 6, Step 6 of the runbook.

  • resource group maxlps-pilot is already active — activate was re-run after it succeeded. Skip ahead to the verify step, or remove the RG first with dpsctl resource-group delete (DPS has no standalone deactivate subcommand).
  • resource group maxlps-pilot has no devices — the resource-group add step did not land. Run dpsctl rg list and confirm resource_names is populated before retrying.
  • power budget exceeded for resource group maxlps-pilot, but reprovision is false — the requested resource-group policy cannot be allocated within the current topology constraints. Lower the resource-group policy, free or reprovision existing allocations, or correct the topology budget.
  • failed to activate resource group with per-node failures in the --sync response — the BMCs for those nodes rejected the set-limit request. Re-run dpsctl verify bmc-health start --topology maxlps-pilot --nodes <node1>,<node2> --wait --summary-only against the failing nodes to confirm BMC reachability, credentials, and power-control health, then retry activation. Nodes that succeeded stay active; only the failing nodes need to be resolved.

MaxLPS Control-Loop Logs

Use this section when the MaxLPS controller logs do not show ongoing limit updates during Part 6, Step 8 of the runbook. The controller’s Kubernetes label and container identifiers are both prs.

The important signal is that the MaxLPS controller continues to emit new New power limits: blocks after activation. If its logs only show startup messages, or if New power limits: stops appearing while the workload is running:

  1. Verify the resource group is active with prs_enabled: true. This is the API’s implementation field for MaxLPS power steering.

  2. Confirm prs.enabled: true in the Helm values so the MaxLPS controller is deployed.

  3. Confirm the persisted runtime switch is enabled; a previously stored value overrides the Helm dps.prs.enabled default after restart:

    dpsctl settings list | jq -e '.settings.prs_enabled == "true"'

  4. Check the MaxLPS controller logs for errors.

MaxLPS Headroom Updates

Use this section when a MaxLPS runtime prs_headroom_percent update is rejected, its effective value is unexpected, or the background control-domain refresh does not complete during the -15 repeat.

Check the effective value

Read the persisted, effective setting instead of relying on the Helm values file:

dpsctl settings list \
  | jq '{
      prs_enabled: .settings.prs_enabled,
      prs_headroom_percent: .settings.prs_headroom_percent
    }'

Both values are persisted. Stored values override the Helm implementation defaults (dps.prs.enabled and dps.prs.headroomPercent) after restart, so a restart does not roll back a runtime change. Confirm prs_enabled is true before expecting a headroom update to affect active MaxLPS control domains. See dpsctl settings list for details.

Correct an invalid value

prs_headroom_percent must be a finite number from -50 through 100, inclusive. Values outside that range, non-numeric strings, NaN, and infinite values are rejected. Correct the value and retry, for example:

dpsctl settings update --set prs_headroom_percent=-30

See Power-steering behavior for the canonical tuning semantics. Do not compensate for a rejected headroom value by changing the resource-group policy at the same time.

Check the asynchronous control-domain refresh

A successful settings response confirms that the submitted value was stored. When it changes the effective headroom value and prs_enabled is true, active MaxLPS control domains are updated asynchronously, so inspect the DPS server logs separately:

kubectl logs -n dps statefulset/dps-server --since=10m \
  | grep -E 'PRS config changed|finished prs upsert|failed to re-upsert PRS|failed to stage commit and start domain|Prs add or update returned error'

The expected start signal is the implementation log message PRS config changed, re-triggering PRS upsert. For each active domain, finished prs upsert confirms completion. Errors such as failed to stage commit and start domain or Prs add or update returned error mean the setting is persisted but the MaxLPS control domain did not accept the new value. Inspect the nested error and the controller logs for the stage or commit rejection before continuing the workload. If the submitted value was already effective, the write is a no-op and no new start or completion message is expected.

Roll back to the last known-good value

If the background control-domain refresh fails, restore the last known-good value rather than restarting the server:

LAST_KNOWN_GOOD_MAXLPS_HEADROOM_PERCENT=-30
dpsctl settings update \
  --set "prs_headroom_percent=${LAST_KNOWN_GOOD_MAXLPS_HEADROOM_PERCENT}"
dpsctl settings list \
  | jq -e --arg expected "${LAST_KNOWN_GOOD_MAXLPS_HEADROOM_PERCENT}" \
    '.settings.prs_headroom_percent == $expected'

If the rollback changes the effective value while MaxLPS power steering is enabled, wait for a new PRS config changed implementation log message and a successful finished prs upsert for every active domain before resuming the test. During final pilot cleanup, restore both pre-pilot MaxLPS settings saved in Part 1, Step 4, which may differ from true and -30.

    Copyright © 2026  NVIDIA Corporation