Adding Nodes to DPS#

Overview#

This guide provides step-by-step instructions for adding new compute nodes to an existing DPS (Dynamic Power Software) configuration. The process involves creating BMC credentials, updating the topology configuration, and ensuring the new nodes are properly integrated into the power management system.

Prerequisites#

  • DPS server running and accessible

  • dpsctl installed and authenticated

  • Existing topology already configured and active

  • Access to the new node’s BMC (Baseboard Management Controller)

  • BMC credentials for the new node

  • Device specifications for the new node type (if not already defined)

Step 1: Verify Current Configuration#

Before adding new nodes, verify your current DPS configuration:

# Check current topology status
dpsctl topology list

# List current entities
dpsctl topology list-entities

# Check active topology
dpsctl topology list --active=true

Example Output:

{
  "topologies": [
    {
      "topology_name": "datacenter",
      "is_active": true,
      "leaf_node_names": ["node001", "node002"]
    }
  ]
}

Step 2: Create BMC Credentials Secret#

DPS requires BMC credentials to communicate with each node. Copy the example in Kubernetes Secret Store to node003-secret.yaml, then set metadata.name to the exact Redfish.SecretName used for the node.

After replacing the placeholders, apply the Secret through your approved workflow. For a direct Kubernetes workflow:

kubectl apply -f node003-secret.yaml

Confirm that the object exists and has the selector label:

kubectl get secrets --namespace dps \
  --selector app.kubernetes.io/component=bmc-credentials

Step 3: Verify Device Specifications#

Ensure the device specification for your new node type exists in DPS:

# List available device specifications
dpsctl device list

If the device specification does not exist, import it:

# Import device specifications (if needed)
dpsctl device upsert devices.yaml

Example device specification for DGX GB200:

- type: ComputerSystem
  description: NVIDIA GB200 Compute Tray (Bianca)
  model: DGX_GB200
  spec:
    devices:
      - type: CPU
        model: Grace
        count: 2
      - type: GPU
        model: GB200
        count: 4
    minLoadWatts: 1000
    maxLoadWatts: 6667
    processorModulesCount: 2

Step 4: Create a Replacement Topology Configuration#

Create a topology file that includes the existing nodes and the new nodes. Give the topology a new name because dpsctl topology import rejects a name that already exists.

Option A: Export Current Topology and Modify#

# Export current topology
dpsctl topology export --topology datacenter > current-topology.json

Edit the exported file to add the new node and change Topology.Name:

{
  "Entities": [
    {
      "Type": "PowerDomain",
      "Name": "PD-A",
      "OperatingLimit": {
        "PowerValue": {"Value": 1150000, "Type": "W"},
        "PowerFactor": 0.9
      }
    },
    {
      "Type": "ComputerSystem",
      "Model": "DGX_GB200",
      "Name": "node001",
      "Redfish": {
        "URL": "https://node001-bmc.example.com",
        "SecretName": "node001"
      }
    },
    {
      "Type": "ComputerSystem",
      "Model": "DGX_GB200",
      "Name": "node002",
      "Redfish": {
        "URL": "https://node002-bmc.example.com",
        "SecretName": "node002"
      }
    },
    {
      "Type": "ComputerSystem",
      "Model": "DGX_GB200",
      "Name": "node003",
      "Redfish": {
        "URL": "https://node003-bmc.example.com",
        "SecretName": "node003"
      }
    }
  ],
  "Topology": {
    "Name": "datacenter-expanded",
    "Entities": [
      {
        "Name": "PD-A",
        "Children": ["node001", "node002", "node003"]
      },
      {
        "Name": "node001"
      },
      {
        "Name": "node002"
      },
      {
        "Name": "node003"
      }
    ]
  }
}

Option B: Create a New Topology File#

Create updated-topology.json with all nodes, including the new node. Use a topology name that does not already exist.

Step 5: Validate the Updated Topology#

Validate the topology file before importing:

# Validate the updated topology
dpsctl topology validate updated-topology.json

Expected Output:

{
  "status": {
    "ok": true,
    "diag_msg": "Topology validation passed"
  }
}

If validation fails, fix the errors before proceeding. For validation and import details, refer to Import a Topology.

Step 6: Import the Replacement Topology#

Import the validated topology:

dpsctl topology import updated-topology.json

Expected Output:

{
  "status": {
    "ok": true,
    "diag_msg": "Success"
  }
}

Import creates the missing node entity and a new inactive topology. It leaves the current topology active. Because topology JSON does not contain a policy bundle field, import does not copy the current topology’s persisted selection. The server resolves the replacement topology selection from the current global_policy_bundle. If you must preserve a different explicit selection, create the replacement through CreateUpdateTopology and set CreateUpdateTopologyRequest.CreateRequest.policy_bundle.

Step 7: Verify the Import#

Verify that the replacement topology contains the new node and remains inactive. An unfiltered dpsctl topology list still includes the current active topology. Confirm the replacement with --active=false. JSON output omits is_active when the value is false. The persisted policy_bundle is the current global_policy_bundle value, MaxLPS on a default deployment.

dpsctl topology list-entities
dpsctl topology list --active=false

Expected Output:

{
  "topologies": [
    {
      "topology_name": "datacenter-expanded",
      "leaf_node_names": ["node001", "node002", "node003"],
      "policy_bundle": "MaxLPS"
    }
  ]
}

Step 8: Switch to the Replacement Topology#

Deactivate the current topology, then activate the replacement. Activation configures the replacement topology’s nodes on the agents.

dpsctl topology deactivate --topology datacenter
dpsctl topology activate --topology datacenter-expanded --ping-hosts

Flags:

  • --ping-hosts Test connectivity of all hosts (recommended)

  • --at-least-percent-hosts Minimum percent of hosts that must be reachable (default: 50)

Note

DPS 0.9 requires an active topology before dpsctl check connection or dpsctl verify bmc-health start can check nodes. Passing --topology does not bypass this requirement. Confirm the replacement topology appears in dpsctl topology list --active, and reactivate it if it was deactivated during the update:

dpsctl topology activate --topology datacenter-expanded

Step 9: Test Node Connectivity#

After activation, test connectivity to the new node’s BMC:

dpsctl check connection --topology datacenter-expanded --nodes node003

Expected Output:

{
  "total_nodes": 1,
  "success_nodes": 1
}

You can also test connectivity to all nodes in the topology:

dpsctl check connection --topology datacenter-expanded

Expected Output:

{
  "total_nodes": 3,
  "success_nodes": 3
}

Step 10: Verify Power Management#

Verify that the new node is properly integrated into power management:

# Check node status
dpsctl check status

# Confirm that the selected policy bundle exists
dpsctl policy bundle list

Troubleshooting#

Common Issues and Solutions#

1. BMC Connection Failures#

Symptoms:

  • Node connection check fails

  • Power policy application errors

Solutions:

  • Verify BMC credentials are correct

  • Check network connectivity to BMC

  • Ensure BMC is accessible from DPS server

  • Verify BMC Redfish API is enabled

# Test BMC connectivity manually
curl -k -u admin:password https://node003-bmc.example.com/redfish/v1

2. Device Specification Not Found#

Symptoms:

  • Validation errors about unknown device type/model

Solutions:

  • Import the required device specifications

  • Verify the device type and model match existing specifications

# Import device specifications
dpsctl device upsert devices.yaml

3. Secret Not Found#

Symptoms:

  • Authentication errors when connecting to BMC

Solutions:

  • Verify the secret name matches the SecretName in the topology

  • Check that the secret is in the correct namespace

  • Ensure the secret contains the correct bmc key

# Verify secret exists
kubectl get secret node003 -n dps

# Confirm the Secret has the required key without printing its value
kubectl describe secret node003 -n dps

4. Topology Name Already Exists#

Symptoms:

  • Import fails with failed to create topology.

Solutions:

  • Choose a new Topology.Name for the replacement topology.

  • Run dpsctl topology list to confirm that the name is unused.

  • Revalidate and import the file.

5. Power Policy Application Issues#

Symptoms:

  • Nodes not responding to power policy changes

  • Power limit errors

Solutions:

  • Verify the node supports the specified power policy plugin

  • Check that the BMC supports the required Redfish endpoints

  • Ensure the device specification includes the correct powerPolicyPlugin