BMC Readiness and Health Guide#

Use this guide to prepare Baseboard Management Controllers (BMCs) for DPS and resolve BMC health-check issues. Complete the BMC health gate before you enable DPS power control for a pilot or production resource group.

An unmanaged baseline can proceed when its compute nodes and metrics collection are ready. Do not include a node with an unresolved BMC health issue in a managed DPS run.

Confirm BMC Compatibility #

Before you deploy or configure DPS, confirm that every target BMC exposes the Redfish endpoints required for its hardware. Refer to Redfish API for the endpoints DPS uses to collect telemetry and control GPU, CPU, module, or node power.

For GB200 NVL and GB300 NVL systems, use BMC firmware 25.0.0 or later. Record the BMC service’s advertised RedfishVersion and verify support for the required endpoints; endpoint support is the readiness gate for the pilot. Record the following for each BMC:

Manufacturer and BMC type, such as OEM-based or NVBMC-based.
BMC firmware build and Redfish version.
Supported and unsupported Redfish endpoints.

If an endpoint is missing or behaves differently, capture a sanitized API response and engage NVIDIA support before you add that node to a managed DPS group. Tools such as redfish-crawler can help collect that evidence.

Limit BMC Client Pressure #

For GB200 and GB300 systems, use no more than four simultaneous BMC client connections, including two DPS connections. Configure every client to use session-token authentication and keep-alive connections instead of basic authentication or repeated session creation. Refer to the DMTF Redfish Host Interface Specification for connection guidance.

Connect DPS to BMCs #

For direct BMC access, create a Kubernetes credential Secret for every node and reference its name from that node’s Redfish.SecretName field in the topology. Set Redfish.URL to the BMC endpoint. Do not put BMC credentials in topology JSON. Refer to Credentials and Secrets Configuration for the supported credential stores and Secret formats.

DPS 0.9 routes device-management requests through a connected DPS Agent. The chart leaves agent.enabled as false, so enable and configure an agent before you run a BMC health check or apply hardware policy. Set the topology entities’ AgentId fields to the matching agent.agentID. For a MaxLPS pilot, follow the agent deployment and mTLS configuration in Step 1 of the pilot runbook.

If you use a Redfish proxy, complete Redfish Proxy Configuration instead. That guide covers proxy routing, authentication, TLS, and topology routing fields. When the proxy owns backend BMC authentication, do not create DPS-owned BMC credential Secrets for that path.

Activate the Topology Before Checking BMCs #

Note

DPS 0.9 requires an active topology for dpsctl verify bmc-health start and dpsctl check connection. Activate the target topology first, even for read-only checks or checks scoped with --topology or --nodes.

Check the active topology with dpsctl topology list --active. If the imported target topology is inactive, activate it before either BMC command:

dpsctl topology activate --topology <topology-name>
dpsctl topology list --active

Activation applies power policies to the nodes. For the MaxLPS pilot, perform it before the baseline workload and leave resource-group creation until after the health gate and baseline. Follow Activate the Topology Before Checking BMCs.

Run the BMC Health Check #

With the target topology active, run the BMC health check. Keep it active until the check finishes and you have saved the report. The check validates Redfish reachability, firmware inventory, power-limit write and readback behavior, telemetry sampling, GPU power-policy state, and in-band and out-of-band power drift.

The full check creates a server task and can write and restore BMC power limits. Obtain authorization for those writes before you start it. It is single-flight across the server, so record the returned task_id and use it to query an active health check rather than starting an overlapping one.

dpsctl verify bmc-health start \
  --topology <topology-name> \
  --force-writes \
  --expected-edpp-pct 100 \
  --samples-per-telemetry 2500 \
  --telemetry-interval 500ms \
  --per-node-timeout 1h

dpsctl verify bmc-health status <task-id>
dpsctl verify bmc-health report <task-id> --wait --output json \
  > bmc-health-report.json
dpsctl verify bmc-health report <task-id> --summary-only

Full checks can take several hours. --per-node-timeout applies to each node, not the whole check. Size it for the sample count multiplied by the telemetry interval, plus the firmware, power, EDPp, WPPS, and write phases. Fleet duration also depends on node count and concurrency. For this 2,500-sample command, sampling alone takes about 21 minutes per node, so the default 5m is too short. Refer to Run a BMC Health Check for the operator procedure and report interpretation, and to bmc-health for every flag and the asynchronous task model. The server task continues if the local dpsctl process exits. Use status or report with the task_id to query its progress.

Before an unmanaged baseline, deactivate the topology and verify default GPU limits as described in Deactivate the Topology for the Baseline. For subsequent health checks in 0.9, reactivate the topology first.

Review the Health Report #

Save the full JSON report and record these per-node latency summaries:

endpoint_latency_ms
telemetry_latency_ms
power_get_latency_ms
power_set_latency_ms
edpp_get_latency_ms
wpps_get_latency_ms

For a MaxLPS-ready BMC, target p50 below 0.5 s, p95 below 0.7 s, p99 below 0.8 s, and maximum latency near 1 s on read and write paths. If p99 exceeds 1 to 2 s, record the measurements and assess BMC client pressure before you start the managed workload.

Before continuing, confirm all of the following:

status.ok is true.
The report covers every expected node.
cluster_summary.failed and cluster_summary.nodes_unreachable are 0.
issues contains no SEVERITY_ERROR entries.

Each issue records a node, resource, code, observed value, threshold, and message. Resolve the issue and rerun a scoped check before you rerun the full topology check.

Account for Slow BMCs #

Slow BMCs delay the telemetry used for dynamic power management and the readback that verifies power-limit changes. First run a node-scoped read-only check with a longer telemetry interval:

dpsctl verify bmc-health start \
  --topology <topology-name> \
  --nodes <node-name> \
  --skip-writes \
  --samples-per-telemetry 10 \
  --telemetry-interval 5s \
  --wait --summary-only

--telemetry-interval spaces telemetry reads; it does not extend the underlying request deadline. --write-resolution-timeout applies only to the post-PATCH readback window of a write-cycle check. It is capped at 60 s and does not make an unreachable BMC respond faster.

For a confirmed write or readback issue, an authorized operator can run a node-scoped write-cycle check with a longer readback window:

dpsctl verify bmc-health start \
  --topology <topology-name> \
  --nodes <node-name> \
  --force-writes \
  --expected-edpp-pct 100 \
  --samples-per-telemetry 2500 \
  --telemetry-interval 5s \
  --write-resolution-timeout 5s \
  --per-node-timeout 4h \
  --wait --summary-only

Adjust DPS Redfish Timeouts #

For an agent-enabled DPS deployment, the Helm chart exposes separate Redfish operation timeouts for reads and power-control writes. The rendered default is 1500ms for both. Use the health report to adjust only the path that is consistently slow but healthy:

dps.redfish.connection.get.operationTimeout covers health checks, telemetry, and other GET calls.
dps.redfish.connection.actuation.operationTimeout covers power-limit and workload-profile writes.

Set a measured value just above the affected path’s healthy-node p99, while keeping dps.redfish.connection.requestTimeout above either operation timeout. For example, if both read and write latency are healthy but exceed the default:

dps:
  redfish:
    connection:
      get:
        operationTimeout: 3s
      actuation:
        operationTimeout: 3s

Use dialTimeout only when connecting to the BMC is slow, and sessionCreateTimeout only when Redfish session creation or refresh is slow. Do not raise these limits to hide an unreachable BMC, authentication failure, or a failed power write. Apply the values through the approved DPS Helm release procedure, then rerun the scoped BMC health check and record the effective values with the report.

The chart renders these values into dps-agent; they apply when agent.enabled: true. For a Redfish proxy, also verify the proxy and proxy-to-BMC timeouts. Proxy mode uses a shared DPS-to-proxy connection path, so its connection limits are configured separately.

For materially slow but otherwise healthy BMCs, set and record the Power Steering scheduler interval through the approved Helm deployment procedure. Use 0 to retain the 10 s Power Steering default, or a value of at least 5 seconds. If BMC latency p99 exceeds 10 s, start with 10; when it is near 30 s, start with 30 and document the choice. Do not change this setting as a BMC preflight; use it only after the health report confirms slow BMCs.

power-steering:
  enabled: true
dps:
  prs:
    enabled: true
    schedulerIntervalSeconds: 10

Verify that the controller is ready and that the BMC or DCGM limit time series changes under load. Record the scheduler interval with the health report. If it changes after an unmanaged baseline, repeat the baseline and managed pair before comparing results.

BMC Health FAQ #

Warning

Resetting a BMC, restarting a node, or changing any GPU power setting through an in-band or out-of-band path can interrupt service or change active power limits. Before you take any of these actions, obtain explicit authorization, schedule a site-approved maintenance window, and prepare a documented rollback plan.

Why Is the BMC Unreachable?#

Issue code: BMC_UNREACHABLE

The BMC did not respond to the Redfish service root, or the fallback BMC ping failed. In 0.9, first confirm that the target topology is active and the DPS Agent is ready; checks require both before they can reach a BMC.

Confirm BMC network reachability from the cluster.
Confirm the credential Secret and its Redfish.SecretName mapping.
Recheck the BMC compatibility inventory and the required Redfish endpoints.
Attempt a BMC reset after compatibility and credentials are confirmed.
If a different hardware model exposes the required endpoints, capture the evidence and engage NVIDIA support about a custom device plugin.
Exclude the node from the managed group if the issue remains unresolved.

Why Did Firmware Validation Fail?#

Issue code: FIRMWARE_VALIDATION_FAILED

The fallback health check could not validate the BMC firmware for the selected device model. Recheck the affected node’s model, BMC firmware, Redfish version, and required endpoints against the compatibility inventory. If the inventory is correct and the issue persists, retain the sanitized report and engage NVIDIA support before adding the node to a managed group.

The B200 and B300 false-failure exception documented for DPS 0.8.x does not apply to this DPS 0.9 guide. The GB200 and GB300 deep health check records firmware inventory rather than using it as a health-check gate. Do not treat a FIRMWARE_VALIDATION_FAILED error from a fallback check as an exception.

Why Do In-Band and Out-of-Band Limits Differ?#

Issue code: IB_OOB_LIMIT_DRIFT

Preserve the original report, then run the node-scoped write-cycle check to rule out slow readback.

Query the current, default, minimum, and maximum GPU power limits:

nvidia-smi -q -d POWER | grep -E 'Default Power Limit|Max Power Limit|Min Power Limit|Current Power Limit'

With authorization, reset the affected in-band limit to the default or maximum TGP value.
```
sudo nvidia-smi -pl <watts>
sudo nvidia-smi -i 0 -pl <watts>
```
Rerun the node-scoped check. If drift remains, attempt a BMC reset or node restart. Exclude the node if it cannot return to a healthy state.

Why Did a Power Write or Restore Fail?#

Issue codes: POWER_WRITE_READBACK_MISMATCH, POWER_LIMIT_RESTORE_FAILED

Run the authorized node-scoped write-cycle check with a longer telemetry interval and readback window.
If it still fails, use the BMC credentials to test the Redfish setpoint write on that node. Refer to Redfish API.
Verify TGP, and TMP or TCP where the platform supports them.
Attempt a BMC reset if the write does not read back correctly. Exclude the node if it remains unresolved.

Why Does the Report Flag WPPS or EDPp?#

Issue codes: WPPS_PROFILES_ACTIVE, WPPS_INACCESSIBLE, WPPS_RESET_FAILED, EDPP_BELOW_REFERENCE

The pilot expects no active workload power profile and an EDPp value of 100%. Reset WPPS through in-band and out-of-band paths where applicable. Reset EDPp where the platform supports it, then rerun the health check. If the settings persist, record the node and values because they affect managed-run performance and baseline comparison.

Why Does the Report Flag Latency?#

Issue codes include TELEMETRY_LATENCY_MAX_HIGH, TELEMETRY_LATENCY_AVG_HIGH, POWER_GET_LATENCY_AVG_HIGH, POWER_SET_LATENCY_AVG_HIGH, POWER_LATENCY_HIGH, EDPP_GET_LATENCY_AVG_HIGH, and EDPP_LATENCY_HIGH.

Use the saved report to identify the affected node and endpoint. The issue-code thresholds are coarse failure thresholds: 5 s average for power GET, power SET, EDPp GET, and telemetry latency, and 10 s maximum telemetry latency. Follow Account for Slow BMCs, then reduce BMC client pressure before you change DPS settings.

Why Did the Report Probe No Nodes?#

Issue code: NO_NODES_PROBED

Confirm the topology name, requested node names, and that those nodes belong to the imported topology. Then rerun the scoped health check.

Why Are Metrics Missing or a Setpoint Write Unauthorized?#

Confirm that the selected device model supports the required Redfish endpoints and that the BMC credentials have read and write access. Capture the report’s node, endpoint, and error message, then check routing and any Redfish proxy configuration. Do not weaken authentication or force another write to diagnose an authorization failure.

Collect Evidence Before Escalation #

Keep the full BMC health JSON report, its task_id, DPS and dpsctl versions, the relevant Helm values, and a sanitized window of controller or server logs. Also retain the BMC firmware and Redfish inventory, latency summaries, and affected node names. Redact passwords, tokens, BMC credentials, certificate private keys, and sensitive endpoint details before sharing them.

For control-plane metrics collection, refer to Prometheus Monitoring. Prometheus metrics can supplement the health report, but they do not replace the per-node BMC health gate.

For the full MaxLPS workflow, return to Run the MaxLPS Power Pilot.