layout-3 layout-6 layout-1 layout-top
layout-top layout-6 layout-1

Troubleshooting the NVL72 inference power pilot

Use this page when a command in the Inference power pilot: NVL72 fleet power management runbook fails or returns an unhealthy state. Start with the section that matches the step you were running.

Deployment Verification

Use this section when dpsctl verify reports an unhealthy component during Part 1, Step 3 of the runbook.

To re-check a single component after a fix, pass its flag — for example dpsctl verify --database. Any combination of --dps-server, --database, --auth, --ui, and --bcm is valid.

  • dps_server unhealthy — Check the DPS server pod is Running; kubectl -n dps logs statefulset/dps-server usually shows the underlying error.
  • database unhealthy — Check the Postgres pod and the database connection secret; a failed ping is almost always reachability or credentials, not load.
  • auth unhealthy — For LDAP, verify ServerURL, BindDN, BindPassword, and CA/client certificates. For JWT, verify the private key file is mounted and readable.
  • ui unhealthy or missing — Confirm the dps-ui Ingress exists in the same namespace as the DPS server with at least one rule and a non-empty host.
  • bcm unhealthy — Confirm the BCM credential secret is present, the URL is reachable from the cluster, and the account has API access. If BCM is not deployed, expect skipped: true instead.

Topology Validation and Import

Use this section when dpsctl topology validate topology.json or dpsctl topology import topology.json fails during Part 2, Step 5 of the runbook.

dpsctl topology validate prints a list of ValidationError entries; the error field identifies the class of failure. The most common ones on a hand-authored topology file:

  • invalid_model — the file does not match the topology JSON schema (missing top-level Topology or Entities, wrong field casing, malformed JSON). Re-check against Managing Topologies.
  • device_not_found — an entity’s Type/Model pair is not in the DPS device registry (typo in Model, or the device was not seeded). Run dpsctl device list and correct the Model string to an exact match.
  • invalid_name — an entity, topology, or policy name has disallowed characters. Use lowercase letters, digits, and -.
  • invalid_secret_name — the Redfish.SecretName contains characters that are not valid for a Kubernetes Secret name. Use lowercase letters, digits, -, and ..
  • duplicate_entity — two entities share the same Name in the file. Make each node name unique.
  • referenced_entity_not_found / referenced_topology_entity_not_found — a topology entry lists a child that has no matching entity block. Compare the Topology.Entities[].Children list against the top-level Entities list.
  • self_reference / circular_dependency — a topology entity lists itself or creates a cycle through its children. Rebuild the parent/child chain so every leaf is reached exactly once from the root.
  • disconnected_graph — one or more compute nodes are not reachable from the topology root. Every compute node must be a descendant of the topology root entity.
  • invalid_connection — a parent entity cannot legally have the given child device type (for example, a rack entity listing another rack as a child). Correct the parent/child relationship to match the reference topology examples.

Fix the file, then re-run dpsctl topology validate until it prints Topology validation passed.

If dpsctl topology import topology.json fails:

  • failed to create topology / failed to upsert entities / failed to add topology entities — generic database failure. Check the DPS pod logs and the dps-postgresql pod, then retry once the database is healthy.

BMC Health Check

Use this section when dpsctl verify bmc-health reports issues during Part 2, Step 6 or Part 2, Step 7 of the runbook.

Use the issues list in the dpsctl verify bmc-health report as the starting point. Each issue includes a node, resource, code, observed value, threshold, and message. The dpsctl verify bmc-health reference documents the command flags and asynchronous task model.

After fixing an affected node, start an asynchronous diagnostic retry scoped to that node. Use 100 telemetry samples for the diagnostic run, then retrieve the full report separately so per-node latency detail is preserved:

dpsctl verify bmc-health start \
  --topology maxlps-pilot \
  --nodes <node-name> \
  --force-writes \
  --expected-edpp-pct 100 \
  --samples-per-telemetry 100 \
  --telemetry-interval 500ms

dpsctl verify bmc-health report <task-id> \
  --wait \
  --poll-interval 5s \
  --output json > bmc-health-node.json

After the diagnostic retry passes, re-run the full-topology gate from the pilot runbook before activation. If the report indicates slow or timeout-censored BMC operations, follow BMC Latency before retrying.

BMC Unreachable or Firmware Validation Failed

Issue codes: BMC_UNREACHABLE, FIRMWARE_VALIDATION_FAILED

The BMC did not respond to the Redfish service root, the fallback BMC ping failed, or firmware validation failed.

On systems with B200 or B300 GPUs, a standalone FIRMWARE_VALIDATION_FAILED issue with message: firmware validation failed is a known DPS 0.8.x health-check false failure. If the affected B200 or B300 node is still reachable, this is the only SEVERITY_ERROR for that node, and no other BMC health checks failed, document the exception and continue the runbook. This exception will be removed in a future DPS version. Treat FIRMWARE_VALIDATION_FAILED on other hardware, or any firmware failure combined with BMC_UNREACHABLE or another SEVERITY_ERROR, as a blocker.

To remediate:

  1. Confirm BMC network reachability from the cluster.
  2. Confirm the Kubernetes BMC credential secret for the affected node.
  3. Confirm the Day 0 BMC compatibility assessment from the runbook prerequisites.
  4. Attempt a BMC reset if compatibility and credentials are correct.
  5. Review the hardware model if the compatibility assessment shows the required GB200 or GB300 Redfish endpoints but the model is different. DPS might support the node through a custom device plugin. For example, a Supermicro BMC based on NVBMC and OEM extensions for HGX B300 can expose the environment metrics endpoints needed for TGP power settings and metrics, and might be supportable under the GB200 plugin with a custom device definition.
  6. Contact the NVIDIA DPS support team if the node appears supportable through a custom device definition.
  7. Exclude the node if the BMC remains unreachable. DPS cannot operate on the affected node.

Power Limit Drift

Issue code: IB_OOB_LIMIT_DRIFT

The in-band and out-of-band BMC power limits do not match.

To remediate:

  1. Follow BMC Latency to rule out slow or timeout-censored BMC readback.

  2. Query the current, default, minimum, and maximum power limits before choosing the reset value:

    nvidia-smi -q -d POWER | grep -E 'Default Power Limit|Max Power Limit|Min Power Limit|Current Power Limit'

  3. Reset the in-band GPU power limits to the default or maximum TGP:

    sudo nvidia-smi -pl <watts>      # Set all GPUs.
sudo nvidia-smi -i 0 -pl <watts> # Set one GPU.

  4. Rerun the node-scoped health check.

  5. Attempt a BMC reset or node restart if drift persists.

  6. Exclude the node if the issue cannot be resolved. DPS cannot operate on the affected node.

Power Write or Restore Mismatch

Issue codes: POWER_WRITE_READBACK_MISMATCH, POWER_LIMIT_RESTORE_FAILED

DPS wrote a power limit through the BMC, but the readback did not converge to the requested value, or the probe could not verify restoration to the original limit.

To remediate:

  1. Follow BMC Latency to rule out slow or timeout-censored BMC operations.
  2. Use the DPS BMC credentials to manually test Redfish setpoint writes on the affected node if the issue remains.
  3. Follow the Redfish API guide examples for PowerLimitWatts.SetPoint.
  4. Verify TGP, TMP where available, and TCP where available.
  5. Attempt a BMC reset if manual writes fail or do not read back correctly.
  6. Exclude the node if the issue cannot be resolved. DPS cannot operate on the affected node.

WPPS or EDPp State

Issue codes: WPPS_PROFILES_ACTIVE, WPPS_INACCESSIBLE, WPPS_RESET_FAILED, EDPP_BELOW_REFERENCE

Workload power profile settings (WPPS) or EDPp settings differ from the expected default state. No workload power profile should be active, and EDPp should report the expected current value of 100% for this pilot gate. WPPS can also affect the EDPp setpoint.

To remediate:

  1. Reset WPPS through both in-band and out-of-band paths where applicable.
  2. Reset EDPp directly where supported if EDPp remains below 100%.
  3. Attempt a BMC reset if the settings continue to persist.
  4. Document the node, active settings, and measured values if no immediate fix is available. These settings affect MaxLPS performance metrics and baselines.

BMC Latency

Issue codes: TELEMETRY_LATENCY_MAX_HIGH, TELEMETRY_LATENCY_AVG_HIGH, POWER_GET_LATENCY_AVG_HIGH, POWER_SET_LATENCY_AVG_HIGH, POWER_LATENCY_HIGH, EDPP_GET_LATENCY_AVG_HIGH, EDPP_LATENCY_HIGH

Latency failures appear in issue codes and latency summaries. Diagnose the full request path before tuning the MaxLPS control loop. A report whose p99 or maximum duration is pinned near a configured timeout does not measure the BMC’s true tail latency.

To remediate:

  1. Use the saved full report from the health-check step to review each node’s endpoint_latency_ms, telemetry_latency_ms, power_get_latency_ms, power_set_latency_ms, edpp_get_latency_ms, and wpps_get_latency_ms summaries. Do not use a summary-only report for latency diagnosis.

  2. Check whether the latency distribution is censored by the deployed Redfish timeouts. In DPS 0.8.3, both Redfish operation timeouts default to 1500ms, while the request timeout defaults to 30s:

    • dps.redfish.connection.get.operationTimeout caps a complete GET-lane service operation, including retries.
    • dps.redfish.connection.actuation.operationTimeout caps a complete actuation-lane service operation, including retries.
    • dps.redfish.connection.requestTimeout caps an individual Redfish HTTP request.
    • --write-resolution-timeout is separate: it controls how long the health check waits for a power-limit write to converge during readback.

    If p99 or maximum duration clusters near 1500ms, successful calls are missing, or the report contains deadline errors, do not use those percentiles to size the MaxLPS loop. The true latency extends beyond the configured operation timeout.

  3. Temporarily raise only the Redfish operation and request timeouts high enough to exceed the expected BMC tail with margin. For example, when investigating BMC responses expected to complete within 30 seconds, add the following values to the same values.yaml used for the current deployment:

    dps:
  redfish:
    connection:
      get:
        operationTimeout: 35s
      actuation:
        operationTimeout: 35s
      requestTimeout: 40s

    Keep the initial rerun representative of normal operation. Leave dps.bmcReadConcurrency and dps.bmcWriteConcurrency at their default of 4, dps.redfish.connection.metricsClients at its default of 1, and dps.redfish.connection.actuationClients at its default of 4. Change concurrency only in a separate, clearly labeled isolation run when testing for connection pressure.

    Record the original values, apply the updated file with the same chart version used by the deployment, and wait for the server rollout:

    helm upgrade <release-name> <chart-reference> \
  --namespace <namespace> \
  --version <installed-chart-version> \
  --values values.yaml

kubectl rollout status statefulset/dps-server --namespace <namespace>

    See the Deployment Guide for the standard chart upgrade workflow.

  4. Re-run a full-topology diagnostic with the normal server fan-out and a write-resolution window that covers the expected write tail. Start the task without --wait, then wait while retrieving the full report:

    dpsctl verify bmc-health start \
  --topology maxlps-pilot \
  --force-writes \
  --expected-edpp-pct 100 \
  --write-resolution-timeout 35s \
  --samples-per-telemetry 100 \
  --telemetry-interval 500ms

dpsctl verify bmc-health report <task-id> \
  --wait \
  --poll-interval 5s \
  --output json > bmc-health-latency.json

    Omit --nodes to cover every topology node and omit --concurrency to use the server default of 16 nodes in flight. Health-check --concurrency is separate from dps.bmcReadConcurrency and dps.bmcWriteConcurrency, which control fan-out within DPS operations. For DPS 0.8.3, the Redfish settings above do not extend the stock five-minute BMC-health per-entity deadline. Use the 100-sample run for latency diagnosis; a 2,500-sample run at a 500 ms interval cannot complete inside that deadline. A shorter diagnostic run does not replace the pilot’s required health gate. If an exact 2,500-sample BMC-health report is required on 0.8.3, contact NVIDIA support; extending that deadline requires a different DPS image because the chart has no per-entity timeout value.

  5. Confirm that the rerun is uncensored before interpreting it: endpoint attempts and successes should agree, timeout errors should be absent, and p99 and maximum durations should remain below the configured operation and request timeouts. Treat connection resets or proxy errors separately from slow successful BMC responses. The report’s telemetry_samples field is the requested sample count, not proof that every sample completed; verify attempts, successes, and errors as well.

  6. Compare the uncensored summaries against the MaxLPS readiness targets from the health-check step in the pilot runbook. Treat the BMC health-check issue-code thresholds as coarse failure thresholds: 5 seconds average for power GET, power SET, EDPp GET, and telemetry latency, and 10 seconds maximum telemetry latency.

  7. Size the MaxLPS control-loop interval from the highest valid p99 across the BMC telemetry and power-control paths so the controller does not run faster than the BMCs can reliably answer. Use the greater of the 10-second chart default or the measured p99 rounded up to whole seconds. For example, use 30 when the highest valid p99 is approximately 30 seconds. The prs key below is the chart’s implementation identifier.

    prs:
  config:
    loopIntervalSeconds: 30

    Changing the MaxLPS control-loop interval changes how quickly power steering responds to workload and telemetry changes. Record the value used for each pilot run because it can affect workload-performance comparison in Part 8.

    Throughput limitation: Raising the Redfish timeouts and lengthening the MaxLPS control-loop interval can prevent premature failures and align the loop with BMC response time, but they do not make a slow BMC respond faster. When BMC latency exceeds the readiness targets, telemetry and actuation feedback are slower, and the expected MaxLPS throughput improvement is not guaranteed. Treat throughput as a measured pilot outcome, not an assumed MaxLPS benefit, and compare aggregate throughput and workload latency with the unmanaged baseline before accepting the configuration.

  8. Assess cluster-wide BMC client connection pressure if latency exceeds the readiness targets. Keep GB200 and GB300 systems at no more than four BMC client connections, including two DPS client connections. Use session token authentication instead of basic authentication, and use keep-alive to maintain the connection. Review the DMTF Redfish Host Interface Specification for Redfish connection guidance.

  9. After diagnosis, update the managed values.yaml deliberately:

    • If BMC or network remediation brings latency back inside the original timeouts, restore the original Redfish timeout values.
    • If a documented pilot exception accepts a slower BMC envelope, keep the Redfish operation and request timeouts above that envelope and retain the matching MaxLPS loop interval. Do not restore an operation timeout below the accepted BMC tail, because doing so reintroduces censored metrics and failed operations.

    Apply the final values with helm upgrade, wait for the server rollout, and archive the effective timeout and loop settings with the pilot results.

No Nodes Probed

Issue code: NO_NODES_PROBED

The server could not build a usable BMC health report for any requested node.

To remediate:

  1. Confirm the topology name.
  2. Confirm the --nodes values.
  3. Confirm that the requested nodes belong to the imported topology.
  4. Rerun the health check.

Topology Activation

Use this section when dpsctl topology activate --topology maxlps-pilot fails during Part 2, Step 8 or Part 6, Step 1 of the runbook.

  • topology maxlps-pilot is already active — activate was re-run after it succeeded. Skip ahead to dpsctl tp list to verify, or deactivate first with dpsctl topology deactivate --topology maxlps-pilot.
  • topology <name> not found / load-topology error — the value passed to --topology does not match any imported topology. Confirm the name used in topology.json and re-run dpsctl tp list to see what DPS has.

Resource Group Creation

Use this section when dpsctl resource-group create fails during Part 6, Step 4 of the runbook.

  • resource group already exists — an RG named maxlps-pilot is already in the database. Run dpsctl rg list to confirm, then either reuse it or remove it first with dpsctl resource-group delete --resource-group maxlps-pilot.
  • resource-group name validation error — the --resource-group value contains disallowed characters. Use lowercase letters, digits, and -, matching the naming convention used elsewhere in this runbook.
  • failed to create resource group — generic database failure. Check the DPS pod logs and the dps-postgresql pod, then retry once the database is healthy.

Adding Resources to the Resource Group

Use this section when dpsctl resource-group add fails during Part 6, Step 5 of the runbook.

  • no active topologies — the topology activation step in Part 6, Step 1 was skipped or rolled back. Run dpsctl tp list --active and activate the pilot topology before retrying.
  • some entities are not in the active topology — one or more node names in --entities are not part of the active topology. Compare the list against dpsctl tp list --active and correct the typo, or reimport the topology JSON.
  • some entities are already in a resource group — those nodes belong to another RG. Remove them from the other RG first, or delete that RG.

Resource Group Activation

Use this section when dpsctl resource-group activate fails during Part 6, Step 6 of the runbook.

  • resource group maxlps-pilot is already active — activate was re-run after it succeeded. Skip ahead to the verify step, or remove the RG first with dpsctl resource-group delete (DPS has no standalone deactivate subcommand).
  • resource group maxlps-pilot has no devices — the resource-group add step did not land. Run dpsctl rg list and confirm resource_names is populated before retrying.
  • power budget exceeded for resource group maxlps-pilot, but reprovision is false — the requested resource-group policy cannot be allocated within the current topology constraints. Lower the resource-group policy, free or reprovision existing allocations, or correct the topology budget.
  • failed to activate resource group with per-node failures in the --sync response — the BMCs for those nodes rejected the set-limit request. Re-run dpsctl verify bmc-health start --topology maxlps-pilot --nodes <node1>,<node2> --wait --summary-only against the failing nodes to confirm BMC reachability, credentials, and power-control health, then retry activation. Nodes that succeeded stay active; only the failing nodes need to be resolved.

MaxLPS Control-Loop Logs

Use this section when the MaxLPS controller logs do not show ongoing limit updates during Part 6, Step 8 of the runbook. The controller’s Kubernetes label and container identifiers are both prs.

The important signal is that the MaxLPS controller continues to emit new New power limits: blocks after activation. If its logs only show startup messages, or if New power limits: stops appearing while the workload is running:

  1. Verify the resource group is active with prs_enabled: true. This is the API’s implementation field for MaxLPS power steering.

  2. Confirm prs.enabled: true in the Helm values so the MaxLPS controller is deployed.

  3. Confirm the persisted runtime switch is enabled; a previously stored value overrides the Helm dps.prs.enabled default after restart:

    dpsctl settings list | jq -e '.settings.prs_enabled == "true"'

  4. Check the MaxLPS controller logs for errors.

MaxLPS Headroom Updates

Use this section when a MaxLPS runtime prs_headroom_percent update is rejected, its effective value is unexpected, or the background control-domain refresh does not complete during the -15 repeat.

Check the effective value

Read the persisted, effective setting instead of relying on the Helm values file:

dpsctl settings list \
  | jq '{
      prs_enabled: .settings.prs_enabled,
      prs_headroom_percent: .settings.prs_headroom_percent
    }'

Both values are persisted. Stored values override the Helm implementation defaults (dps.prs.enabled and dps.prs.headroomPercent) after restart, so a restart does not roll back a runtime change. Confirm prs_enabled is true before expecting a headroom update to affect active MaxLPS control domains. See dpsctl settings list for details.

Correct an invalid value

prs_headroom_percent must be a finite number from -50 through 100, inclusive. Values outside that range, non-numeric strings, NaN, and infinite values are rejected. Correct the value and retry, for example:

dpsctl settings update --set prs_headroom_percent=-30

See Power-steering behavior for the canonical tuning semantics. Do not compensate for a rejected headroom value by changing the resource-group policy at the same time.

Check the asynchronous control-domain refresh

A successful settings response confirms that the submitted value was stored. When it changes the effective headroom value and prs_enabled is true, active MaxLPS control domains are updated asynchronously, so inspect the DPS server logs separately:

kubectl logs -n dps statefulset/dps-server --since=10m \
  | grep -E 'PRS config changed|finished prs upsert|failed to re-upsert PRS|failed to stage commit and start domain|Prs add or update returned error'

The expected start signal is the implementation log message PRS config changed, re-triggering PRS upsert. For each active domain, finished prs upsert confirms completion. Errors such as failed to stage commit and start domain or Prs add or update returned error mean the setting is persisted but the MaxLPS control domain did not accept the new value. Inspect the nested error and the controller logs for the stage or commit rejection before continuing the workload. If the submitted value was already effective, the write is a no-op and no new start or completion message is expected.

Roll back to the last known-good value

If the background control-domain refresh fails, restore the last known-good value rather than restarting the server:

LAST_KNOWN_GOOD_MAXLPS_HEADROOM_PERCENT=-30
dpsctl settings update \
  --set "prs_headroom_percent=${LAST_KNOWN_GOOD_MAXLPS_HEADROOM_PERCENT}"
dpsctl settings list \
  | jq -e --arg expected "${LAST_KNOWN_GOOD_MAXLPS_HEADROOM_PERCENT}" \
    '.settings.prs_headroom_percent == $expected'

If the rollback changes the effective value while MaxLPS power steering is enabled, wait for a new PRS config changed implementation log message and a successful finished prs upsert for every active domain before resuming the test. During final pilot cleanup, restore both pre-pilot MaxLPS settings saved in Part 1, Step 4, which may differ from true and -30.

    Copyright © 2026  NVIDIA Corporation