Troubleshooting#

Use this guide when DPS returns an error, a health check fails, or a component does not behave as expected. Start with Quick Diagnosis, then jump to the symptom that matches what you observed.

This page covers operator-facing failures across installation, authentication, topology, resource groups, baseboard management controller (BMC) connectivity, the database, and Power Reservation Steering (PRS). For step-by-step recovery during the NVL72 inference power pilot, refer to Troubleshoot the MaxLPS power pilot.

Safety and Evidence#

Preserve evidence before you change production state:

Capture the full dpsctl JSON response, including status.diag_msg and status.error.code. That code value is the DPS error code, such as AUTHENTICATION_FAILED or DATABASE_ERROR. Prefer the code over matching the entire message string, because messages can change between releases. On the gRPC wire the same value appears as ErrorInfo.reason; dpsctl JSON does not emit a reason field for that code.
Record the DPS server version (dpsctl server-version), the dpsctl client version, the Helm chart version, the Kubernetes namespace, and the topology or resource-group name.
Collect only the log lines that surround the failure. Redact passwords, tokens, BMC credentials, certificate private keys, and any workload or tenant identifiers that are not required to diagnose the issue.
Prefer read-only commands (dpsctl verify, dpsctl check connection, dpsctl topology list, kubectl get, kubectl logs) until you have identified a documented remediation.
This guide does not authorize mutating commands such as dpsctl topology activate, dpsctl topology deactivate, dpsctl topology remove, dpsctl resource-group activate, dpsctl resource-group delete, BMC reset, nvidia-smi -pl, kubectl rollout restart, or helm upgrade. Use an approved procedure that defines preconditions, recovery verification, and rollback. Topology deactivation and resource-group deletion do not restore GPU Total Graphics Power (TGP).

Quick Diagnosis#

Run these checks in order. If a check fails, use the matching section. The UI Ingress is not in the control path for BMC, Redfish, or PRS operations, so an unhealthy ui result does not prevent you from collecting evidence for those incidents.

Confirm --host, --port, and TLS flags match the deployed API endpoint. Refer to the dpsctl global options.
Authenticate before any other API call. dpsctl server-version and dpsctl verify require a session.
```
dpsctl login
```
If login fails, go to Cannot Connect or Log In. Do not run dpsctl verify --auth until login succeeds.

Confirm the client can reach the API:

dpsctl --host <dps-api-host> --port <grpc-port> server-version

Check component health:
```
dpsctl verify
```
The response can include dps_server, database, auth, ui, and bcm ComponentStatus objects. Default-valued fields are omitted from dpsctl JSON, so use status.ok and the fields that are present instead of matching an expected set of false values. On a deployment without a registered Base Command Manager (BCM) check, the bcm object contains only "name": "BCM", and status.ok is false with a generic deployment-verification diagnostic. To recheck one component, pass its flag, for example dpsctl verify --database.
If BMC or Redfish symptoms remain, confirm the intended topology appears in dpsctl topology list --active=true before checking node reachability. DPS 0.9 requires an active topology for this check, even with --topology. If it is inactive, skip this check and follow Managing Topologies through your approved activation procedure.
```
dpsctl check connection --topology <topology-name>
```

Inspect Kubernetes objects in the DPS namespace (default dps):

kubectl get pods -n dps
kubectl logs -n dps -l app=dps-server --all-containers --since=15m --tail=200

Errors and Diagnosis#

Cannot Connect or Log In#

Use this section when dpsctl cannot open a gRPC channel, dpsctl login fails, or commands return authentication or authorization codes.

Reported codes can include AUTHENTICATION_FAILED, AUTHENTICATION_INVALID_CREDENTIALS, AUTHENTICATION_USER_NOT_FOUND, AUTHENTICATION_SERVICE_UNAVAILABLE, AUTHORIZATION_FAILED, NETWORK_ERROR, and DEADLINE_EXCEEDED.

Use these diagnosis steps:

Confirm --host, --port, and TLS flags (--ca-cert-path, --client-cert-path, --client-key-path) match the deployed API endpoint. Refer to the dpsctl global options.
For username and password login, confirm the account exists in the authentication backend and belongs to a DPS role group. Refer to User Accounts.
After login succeeds, if dpsctl verify --auth reports healthy: false, check LDAP ServerURL, BindDN, BindPassword, and CA or client certificates, or confirm the JWT private key is mounted and readable.
If the authentication service is temporarily unavailable, wait and retry. Persistent AUTHENTICATION_SERVICE_UNAVAILABLE is an administrator issue, not a client credential typo.
If login succeeds but a later command returns AUTHORIZATION_FAILED, the user is authenticated without the required role. Adjust LDAP or OIDC group membership rather than retrying the same credentials.

Deployment Component Unhealthy#

Use this section when dpsctl verify reports healthy: false for a component after Deploy DPS.

The following table maps each component to evidence to collect:

Component	What to check	Potential cause
`dps_server`	`kubectl get pods -n dps` and `kubectl logs -n dps -l app=dps-server --all-containers --since=15m --tail=200`	Pod not `Running`, crash loop, or API not reachable from the client
`database`	Postgres pod, connection secret, and Database TLS	Reachability, credentials, or TLS handshake failure (`DATABASE_ERROR`)
`auth`	LDAP or JWT configuration and certificates	Bind failure, missing key material, or authentication service down
`ui`	`dps-ui` Ingress in the same namespace, with a host and at least one rule	Missing Ingress, DNS, or TLS for the UI hostname
`bcm`	Whether BCM is part of the deployment, then its credential secret and network path	If no BCM check is registered, the `bcm` object contains only `"name": "BCM"` and makes `status.ok` `false`. If BCM is expected, this result indicates that the integration is not registered.

After an approved configuration change, rerun only the affected component, for example dpsctl verify --dps-server --database.

Topology Validation or Import Fails#

Use this section when dpsctl topology validate or dpsctl topology import fails. The validate command returns ValidationError entries whose error field identifies the failure class. Import performs the same semantic checks during topology creation, but reports them as a gRPC error rather than returning the validation_errors array.

This guide covers the following validation classes:

invalid_model — the file does not match the topology JSON schema.
device_not_found — an entity Type and Model pair is not in the device registry. Run dpsctl device list and correct Model to an exact match.
invalid_name / invalid_secret_name — disallowed characters in an entity, topology, policy, or Kubernetes Secret name.
duplicate_entity — two entities share the same Name.
referenced_entity_not_found / disconnected_graph / circular_dependency — the parent and child graph is incomplete or cyclic.
invalid_connection — a parent device type cannot legally own the listed child.

Fix the file and rerun dpsctl topology validate until it reports success, then import. If import returns failed to create topology, run dpsctl topology list first. An existing topology with the same name produces that message and DATABASE_ERROR; retrying the unchanged import cannot resolve the collision. If the name is not present, collect dpsctl verify --database and server logs to distinguish connectivity or persistence failures.

For the full validation list in the NVL72 pilot, refer to Why Does Topology Validation or Import Fail?. For schema and import guidance, refer to Import a Topology.

Topology or Resource Group Lifecycle Fails#

Use this section when dpsctl topology activate, dpsctl resource-group add, or related commands fail.

Reported codes can include NOT_FOUND, ALREADY_EXISTS, VALIDATION_ERROR, NODE_NOT_IN_TOPOLOGY, DEVICE_NOT_IN_RESOURCE_GROUP, and INVALID_OPERATION.

Use these diagnosis steps:

topology <name> not found or NOT_FOUND — the --topology value does not match an imported topology. Run dpsctl topology list.
topology <name> is already active — the topology is already the control path. Confirm with dpsctl topology list --active=true and skip activate. Do not deactivate as a diagnosis step. Deactivate does not restore GPU TGP.
no active topologies / INVALID_OPERATION when adding resource-group members — run dpsctl topology list --active=true and record the intended topology. Do not activate a topology from this general diagnosis path.
some entities are not in the active topology / INVALID_OPERATION — correct the node names against the active topology. If the topology omitted intended nodes, create a corrected replacement topology or use the CreateUpdateTopology API update operation.
resource(s) not in topology / NODE_NOT_IN_TOPOLOGY — the listed names are absent from the topology. Correct --entities against dpsctl topology list.
some entities are already in the resource group / ALREADY_EXISTS — those nodes are already members of this group. Remove duplicates from the request.
some entities are reserved by another allocating resource group — the code depends on the operation. Adding resources returns ALREADY_EXISTS; activating a resource group returns INVALID_OPERATION. Record the conflicting group metadata and stop. Do not deactivate another group from this general diagnosis path.
resource(s) not in resource group / DEVICE_NOT_IN_RESOURCE_GROUP — a remove request listed nodes that are not members of this group. Correct the names; do not delete another resource group.
power budget exceeded ... reprovision is false — admission detected a new or worsened overload in the proposed device or ancestor allocation. Capture the effective policy, active resource groups, topology limits, and the reported device metadata. dps.prs.headroomPercent does not determine this admission result. Do not change policy caps or topology budgets from this general diagnosis path.
Per-node failures in a --sync activate response — those BMCs rejected the set-limit request. With the topology active, run a node-scoped read-only BMC health check:
```
dpsctl verify bmc-health start --topology <topology-name> --nodes <node1>,<node2> --skip-writes --wait --summary-only
```
Check the overall status.ok value after the task completes. If it is false, the server cleans up partially activated nodes. If the request succeeds under its partial-activation rule, accepted nodes remain active. The default rule requires at least 80% of hosts. In either case, query the resource group again instead of inferring final state from individual request results.

Refer to Managing Resource Groups for the intended lifecycle, and to What If Resource Group Creation Fails? in the NVL72 runbook for the exact pilot command failures.

BMC or Redfish Calls Fail#

Use this section when topology activation, resource-group activation, or dpsctl check connection reports BMC failures, or when codes such as REDFISH_REQUEST_UNAUTHORIZED, REDFISH_REQUEST_ERROR, or NETWORK_ERROR appear.

DPS 0.9 requires the topology to be active before dpsctl check connection or dpsctl verify bmc-health start can probe its nodes. Passing --topology or --skip-writes does not bypass this requirement. If the command returns INVALID_OPERATION with no agent configuration available, check dpsctl topology list --active. If the intended topology is inactive, complete the activation procedure from Managing Topologies before retrying. Continue network and credential diagnosis without these probes if activation has failed.

Use these diagnosis steps:

Confirm BMC network reachability from the cluster and that the Kubernetes BMC credential secret exists for the node. Refer to Credentials and Secrets Configuration.
REDFISH_REQUEST_UNAUTHORIZED — the BMC rejected the credentials or the session. Rotate or correct the secret; do not embed passwords in topology JSON.
Connection timeouts — confirm routing, firewalls, and any Redfish Proxy Configuration.
For a read-only deep probe of firmware, power-limit drift, Workload Power Profiles Settings (WPPS), Electrical Design Point (EDPp), and BMC latency, start a node-scoped check. Bare dpsctl verify bmc-health only lists subcommands. Do not pass --force-writes from this page:
```
dpsctl verify bmc-health start --topology <topology-name> --nodes <node-name> --skip-writes --wait --summary-only
```
Use Run a BMC Health Check to interpret the report and decide whether a write-enabled check is appropriate. Issue-specific BMC reset and in-band nvidia-smi -pl remediation in BMC Health Check applies only to the NVL72 inference power pilot. Issue codes such as BMC_UNREACHABLE, IB_OOB_LIMIT_DRIFT, and POWER_WRITE_READBACK_MISMATCH are documented in BMC Health FAQ. Use the write-cycle check, BMC reset, and in-band nvidia-smi -pl only when you have the authorization required for those changes.

Mutating BMC remediations (reset, in-band nvidia-smi -pl) change GPU power limits. Do not run them from this general troubleshooting path.

Database Errors#

Use this section when dpsctl verify --database is unhealthy, pods crash on Postgres connect, or commands return DATABASE_ERROR.

Use these diagnosis steps:

Confirm the Postgres pod or external instance is running and that the connection secret matches global.postgresql.
Confirm network connectivity from dps-server to the database host and port.
For TLS failures, inspect dps_db_tls_handshake_failures_total and the reason label (cert_parse, cert_verify, hostname_mismatch, protocol, auth_rejected). Refer to Troubleshooting.
After rotating CA or client certificates, record the mounted Secret version and the dps-server pod start time. The process must reload changed key pairs, but this guide does not prescribe a restart without an approved deployment rollback.
Review Postgres logs and dps-server logs for the same timestamp. Do not paste connection strings or passwords into an evidence bundle.

PRS Is Not Updating GPU Limits#

Use this section when the Power Steering controller is not Ready, reports reconciliation errors, or the observed BMC/DCGM limit time series does not move under load.

Use these diagnosis steps:

Confirm the resource group is active with prs_enabled: true.
For the bundled Power Steering deployment, confirm both power-steering.enabled: true and dps.prs.enabled: true in the Helm values. An external Power Steering deployment can disable the subchart only when dps.prs.hostPort and dps.prs.controllerHostPort point to the external services.

Check the Power Steering controller logs:

kubectl logs -n dps \
  -l app.kubernetes.io/component=controller,app.kubernetes.io/part-of=power-steering \
  -c controller --since=15m --tail=200

If BMC latency is high, record dps.prs.schedulerIntervalSeconds and the measured latency. Do not change the scheduler interval from this general diagnosis path. Refer to How Do I Check Power Steering? and Account for Slow BMCs.

Error Code Index#

dpsctl JSON responses put the lookup key in status.error.code. On the gRPC wire, the same value is ErrorInfo.reason with domain dcpower.nvidia.com. Use the code, not the full message. The following table maps codes covered by this guide to a diagnosis section:

Code	Start here
`AUTHENTICATION_FAILED`	Cannot Connect or Log In
`AUTHENTICATION_INVALID_CREDENTIALS`	Cannot Connect or Log In
`AUTHENTICATION_USER_NOT_FOUND`	Cannot Connect or Log In
`AUTHENTICATION_SERVICE_UNAVAILABLE`	Cannot Connect or Log In
`AUTHORIZATION_FAILED`	Cannot Connect or Log In
`NETWORK_ERROR`	Cannot Connect or Log In or BMC or Redfish Calls Fail
`DEADLINE_EXCEEDED`	Quick Diagnosis; then BMC latency if the call was a Redfish operation
`DATABASE_ERROR`	Database Errors
`FILE_ERROR`	Confirm the referenced file exists, is readable, and is not truncated
`NOT_FOUND`	Topology or Resource Group Lifecycle Fails
`ALREADY_EXISTS`	Topology or Resource Group Lifecycle Fails
`INVALID_OPERATION`	Topology or Resource Group Lifecycle Fails
`VALIDATION_ERROR`	Topology Validation or Import Fails
`NODE_NOT_IN_TOPOLOGY`	Topology or Resource Group Lifecycle Fails
`DEVICE_NOT_IN_RESOURCE_GROUP`	Topology or Resource Group Lifecycle Fails
`REDFISH_REQUEST_UNAUTHORIZED`	BMC or Redfish Calls Fail
`REDFISH_REQUEST_ERROR`	BMC or Redfish Calls Fail
`INVALID_CONFIGURATION` / `INVALID_ARGUMENT`	Check Helm values, topology JSON, and command flags against the linked task guide
`INTERNAL_ERROR`	Collect redacted `dps-server` logs and escalate

Codes such as INVALID_POLICY_LIMIT_MIN and POWER_USAGE_EXCEEDED indicate a policy or allocation constraint. Capture the effective policy and reported device metadata instead of retrying the same activation request unchanged.

Escalation#

Stop mutation attempts and use your approved escalation process when any of the following conditions is true:

The error code is INTERNAL_ERROR, or it is not listed in this guide, and read-only diagnosis did not identify a safe fix.
A documented remediation failed, or the system state is worse after the attempt.
A BMC remains unreachable, unauthorized, or incompatible with the required firmware after the checks in BMC or Redfish Calls Fail. For the NVL72 inference power pilot, continue with BMC Health FAQ.
The next step would mutate power limits, firmware, or cluster configuration without a documented rollback.
You need a custom device definition for hardware that is not in the shipped device registry.

Do not keep applying mutating commands to gather more data. Preserve the last successful and last failed outputs.

What to Include#

Prepare a redacted bundle that contains:

DPS server version, dpsctl version, Helm chart version, and Kubernetes namespace.
The failing command, timestamp (UTC), topology name, resource-group name, and node names.
The customer-visible error code (status.error.code in dpsctl JSON) and diag_msg, with secrets removed.
dpsctl verify output and, when relevant, dpsctl check connection or a BMC health-check summary.
Redacted excerpts of dps-server, database, authentication, or PRS logs around the failure.
Hardware model, BMC firmware, and whether the node is in the NVL72 pilot topology.

Never include passwords, tokens, private keys, full Secret YAML, or unredacted BMC URLs that embed credentials.

This guide does not define a customer support channel, support severity levels, or response times. Use your organization’s approved incident and NVIDIA support process. If no approved NVIDIA case-intake route is established, preserve the evidence bundle and contact the owner of your DPS deployment.

Troubleshooting#

Safety and Evidence#

Quick Diagnosis#

Errors and Diagnosis#

Cannot Connect or Log In#

Deployment Component Unhealthy#

Topology Validation or Import Fails#

Topology or Resource Group Lifecycle Fails#

BMC or Redfish Calls Fail#

Database Errors#

PRS Is Not Updating GPU Limits#

Error Code Index#

Escalation#

What to Include#

Related Documentation#