Run a BMC Health Check#

Use dpsctl verify bmc-health to test baseboard management controller (BMC) reachability and collect a structured health report for an imported topology. Depending on the device plugin and selected mode, the check can also exercise firmware inventory, power-limit controls, telemetry, Electrical Design Point (EDPp), and Workload Power Profile Settings (WPPS). It compares in-band and out-of-band power-limit values where supported.

For every command and flag, refer to the dpsctl verify bmc-health reference.

Prerequisites#

Before you start, confirm the following conditions:

dpsctl is installed, authenticated, and aligned with the DPS server version.
The topology is validated and imported. Refer to Managing Topologies. It does not need to be active.
Every intended node belongs to the topology and has a configured power plugin and BMC credential Secret. Refer to Credentials and Secrets Configuration for credential setup.
No other BMC health check is running on the server. The server permits one check at a time, even when two checks target different topologies or nodes.
You know whether the result is for diagnosis, a readiness gate, or an approved write-enabled validation. These uses require different evidence.

Start with a small node scope when you are validating a new device model, credential configuration, or operator procedure.

Choose a Mode#

The current server default is read-only when the request does not specify a write mode. Pass a mode flag anyway so saved commands and automation state the intended behavior explicitly.

Selection	Behavior
`--skip-writes`	Reads health data without power-limit write cycles or WPPS remediation. Use this mode first and for general diagnosis.
No mode flag	Uses the server default, which is currently read-only. This behavior can change between versions.
`--force-writes`	Requests power-limit write and readback cycles and WPPS remediation from a device plugin that supports write-enabled deep checks. Use only after the write-safety preflight.

--skip-writes and --force-writes are mutually exclusive.

Run a Read-Only Check#

Run a blocking summary check when you need the overall result and issue list:

dpsctl verify bmc-health start \
  --topology <topology-name> \
  --skip-writes \
  --wait \
  --summary-only

The command prints progress to standard error and the report to standard output. Without --prs-bmc-latency-check, a completed report with status.ok: false is still printed, and the command exits with status 1. With the PRS latency check, the CLI prints the BMC report followed by the PRS artifact, and the final exit status reflects the PRS artifact. Inspect the BMC report’s status.ok value separately.

Use the asynchronous flow when the check can outlive your terminal session or when you need the full report:

dpsctl verify bmc-health start --topology <topology-name> --skip-writes
dpsctl verify bmc-health status <task-id>
dpsctl verify bmc-health report <task-id> --wait --output json > bmc-health-report.json

Recover Interrupted Polling#

Save the task_id from start. The task runs on the server after dpsctl disconnects. A local interruption, including Ctrl-C during --wait, stops only polling. Reattach with status or report --wait.

The server retains a completed task for the configured --zombie-lifetime, which defaults to 10m. A server restart discards both running and completed task state. There is no dpsctl verify bmc-health cancel command.

Run an Approved Write-Enabled Check#

--force-writes can change BMC state. Use it only with applicable operator approval and an approved procedure that defines the maintenance window, recovery action, and acceptable node scope. The flag does not override a device plugin’s safety policy, suspend effective DPS policy, or disable other power controllers. Some plugins remain read-only. If the selected device plugin does not support the deep BMC health check, DPS checks only BMC reachability and firmware.

Complete this preflight before you use write mode:

Place the selected nodes in an idle or otherwise approved maintenance state.
Review the active and effective policy and its hardware and hierarchical constraints for the selected scope. If enforcement can reapply or redistribute power during the probe, quiesce it through the approved procedure or account for that behavior in the result.
Prevent other operators and power-management controllers from changing power limits or WPPS on the selected nodes during the check.
Run and retain a full read-only report as the baseline.
Confirm that you can restore the captured power limits through the approved operational path if automated restoration fails.
Confirm that write-enabled WPPS remediation is acceptable for the selected scope. The probe does not restore WPPS profiles that it disables.

Start with one node and one in-flight node probe:

dpsctl verify bmc-health start \
  --topology <topology-name> \
  --nodes <node-name> \
  --force-writes \
  --concurrency 1 \
  --wait

For each supported GPU and CPU component, the write-cycle probe performs the following actions:

Reads and records the original power-limit set point.
Moves the set point in 1 W steps, up to three steps away from the original while staying within the reported bounds.
Reads after every write until the requested value appears or the --write-resolution-timeout expires.
Walks the set point back and issues a final best-effort write to the captured original value.
Reads the value again and records whether restoration succeeded.

The default readback window is 1s. Increase --write-resolution-timeout only when observed BMC commit latency justifies it. A larger value does not correct an unreachable or stuck BMC.

When a device plugin runs write-enabled deep checks and finds active WPPS profiles, it can disable them and verifies that the enforced mask becomes empty. A WPPS_PROFILES_ACTIVE issue can remain in the same run because the profiles were active when observed.

Warning

Treat POWER_LIMIT_RESTORE_FAILED as an unresolved state change. Keep the node out of service, compare original_w with restored_value_w, and restore and verify the limit through the approved recovery procedure before returning the node to use.

Set Sampling and Timeouts#

The following defaults affect run duration and BMC load:

Option	Current Behavior
`--samples-per-telemetry`	`0` uses the server default of 10 samples per node. The server accepts at most 2,500.
`--telemetry-interval`	The CLI default is `500ms` between samples. This option does not control power-write readback.
`--write-resolution-timeout`	The default is `1s` per write readback, with a server maximum of `60s`.
`--per-node-timeout`	The default is `5m`. An explicit value must be from `5m` through `48h`.
`--concurrency`	`0` uses the server default of 16 concurrent node probes. The server maximum is 1,024.
`--expected-edpp-pct`	An unset value uses the server floor of `100`. An explicit value must be from 1 through 200.

When you increase the sample count or interval, set --per-node-timeout high enough for all requested samples plus firmware, power, EDPp, WPPS, and write phases. A node that reaches this timeout is reported as BMC_UNREACHABLE, and a later response does not replace the timeout result.

Lower concurrency when the fleet has tight BMC session or connection limits. Higher concurrency increases simultaneous BMC and agent load; it does not change the server-wide limit of one BMC health task.

Interpret the Report#

Read the report from the outside in:

Field	Interpretation
`status`	The overall result. `ok: false` means at least one error-severity issue or failed node. Warnings alone can leave `ok: true`.
`cluster_summary`	Scope, pass/fail counts, unreachable nodes, and cluster latency distributions. Confirm the counts match the intended topology or node subset.
`issues`	Actionable findings. Use `severity`, `code`, `node`, `resource`, `observed`, and `threshold` together. Do not automate against the human-readable message alone.
`node_reports`	Per-node evidence, including reachability, firmware, endpoint statistics, power settings and writes, EDPp, WPPS, drift, and telemetry. This field is omitted by `--summary-only`.
`endpoint_stats`	Attempt, success, failure, and latency counts for each probed endpoint.
`telemetry_samples` and `telemetry_samples_completed`	Requested and attempted telemetry reads. Also inspect `telemetry_completion_reason`.
`power_writes`	Write-cycle evidence. Require `mismatches: 0`, `restored: true`, and `restored_value_w` equal to `original_w`.

The JSON formatter omits fields that have their default values. For example, a clean report can omit an empty issues list and zero-valued failed, nodes_unreachable, and mismatches fields. An omitted false-valued WPPS field such as reset_attempted also means false. Do not apply this rule to an entire evidence section: an omitted power_writes list means that the report contains no write-cycle evidence.

The report does not identify the requested write mode. The presence of power_writes confirms that power write cycles ran, but their absence does not distinguish --skip-writes, a device plugin that remained read-only, and a plugin without write support. Record the exact command options with the report. When those options show --skip-writes, power_writes must be absent. When they show --force-writes for a plugin expected to support write-enabled deep checks, require power_writes evidence or treat the requested check as not tested. Older server versions can also omit newer telemetry completion fields; align the client and server versions when your acceptance criteria require those fields.

A device plugin without deep BMC health support falls back to BMC reachability and firmware validation. The final cluster report does not expose the internal deep-probe marker. If expected node_reports fields are absent, treat those checks as not tested instead of inferring that they passed.

The probe also treats some data as optional. For example, a missing in-band one-shot power value prevents that drift comparison but does not by itself create an issue. A clean status is therefore not a substitute for confirming that the report contains every field required by your readiness criteria.

Apply Safety Criteria#

For a full readiness gate, confirm all of the following conditions:

status.ok is true.
cluster_summary.total_nodes and nodes_attempted match the intended scope, and failed and nodes_unreachable are 0.
The report has no error-severity issues. Review and disposition every warning even when the process exits with status 0.
Every required node contains evidence for the checks that its device model is expected to support.
Telemetry sampling completed. Failed individual samples produce TELEMETRY_SAMPLES_FAILED; an incomplete sampling loop produces TELEMETRY_SAMPLING_INCOMPLETE.
Average telemetry, power GET, power SET, and EDPp GET latency do not exceed 5s, and maximum telemetry latency does not exceed 10s.
EDPp is at or above the selected floor, WPPS is accessible with no active profiles, and required in-band and out-of-band values do not drift.
For a write-enabled gate on a plugin that supports write-enabled deep checks, every write cycle has zero mismatches and verified restoration. A second run confirms that any WPPS remediation left the node in the intended state.

The built-in thresholds are coarse product checks. Apply any stricter release, platform, or site acceptance criteria separately.

Handle Expected Outcomes and Failures#

Use the following table to decide the next action:

Outcome	Meaning and Action
`start` returns `ALREADY_EXISTS`	Another server-wide check is active. The public error does not provide its task ID. Use a previously saved ID if you own the run; otherwise, coordinate with the operator who started it or wait for it to finish.
`report` returns `done: false`	The task is still running. The command can exit successfully because no health decision exists yet. Use `status` or `report --wait`.
`status` is `Failed` or the report has `error_message`	The task failed before it produced a usable report. Preserve the task ID and inspect the DPS server logs.
`status` or `report` returns `NOT_FOUND`	The ID is invalid, the completed task expired, or the server restarted. For a write-enabled run, verify node state before starting again.
`BMC_UNREACHABLE`	The BMC, agent, device registration, power plugin, dispatch, or per-node timeout path failed. Start with BMC or Redfish Calls Fail.
`POWER_WRITE_READBACK_MISMATCH`	A written value did not converge before the readback window. Keep the node out of the readiness set and investigate the write path.
`POWER_LIMIT_RESTORE_FAILED`	Automated restoration was not verified. Apply the approved recovery procedure before another run or workload.
`WPPS_PROFILES_ACTIVE`	Profiles were active at probe time. Read-only mode leaves them unchanged; a device plugin running write-enabled deep checks attempts to disable them. Review the recorded profile IDs and rerun after remediation.
A completed report has `status.ok: false`	Without `--prs-bmc-latency-check`, `start --wait` and a completed `report` print the report, then exit with status `1`. Use the structured report as the decision input.
`--prs-bmc-latency-check` is enabled	The CLI prints the BMC report followed by the PRS artifact. The final exit status reflects the PRS artifact, so inspect the BMC report’s `status.ok` value separately.
The report contains warnings only	The command can exit with status `0`. Review warnings such as `IB_OOB_LIMIT_DRIFT` and `TELEMETRY_SAMPLES_FAILED`; do not rely on the exit status alone.
The server returns `NOT_IMPLEMENTED` or an unknown BMC-health method	The server can predate the client command. Align `dpsctl` and the server to the same release before retrying.