Skip to navigation

Power Control

View as MarkdownOpen in Claude

Read and change node power state, and run the coordinated rack cold reboot.

RPCExecutionSummary
SetPowerStateSynchronousSetPowerState controls the power state of a single node.
BatchSetPowerStateSynchronousBatchSetPowerState controls power for caller-supplied nodes without requiring them to be registered in RMS inventory.
GetPowerStateSynchronousGetPowerState retrieves the current power state of a node.
BatchGetPowerStateSynchronousBatchGetPowerState retrieves the current power state for caller-supplied nodes without requiring them to be registered in RMS inventory.
ExecuteColdRebootAsynchronous, one jobRun the full rack cold reboot sequence on caller-supplied nodes as one asynchronous job.

For definitions of execution modes and job polling, refer to Async jobs.

SetPowerState

rack_manager.RackManager/SetPowerState

SetPowerState controls the power state of a single node. Supports power on, power off, and power reset operations.

PropertyValue
ExecutionSynchronous. The response carries the full result.
TargetRegistered inventory. Reads or changes nodes and racks stored in RMS inventory.
Behavior detailsOperations guide
Protobuf referenceSetPowerState

Request: SetPowerStateRequest

FieldTypeDescription
node_idstringTarget node identifier
operationPowerOperationPower operation to perform
rack_idstringRack containing the target node

Response: SetPowerStateResponse

FieldTypeDescription
statusReturnCodeRETURN_CODE_SUCCESS if power operation completed, RETURN_CODE_FAILURE otherwise

Errors: Refer to Conventions for how RMS reports failures. This RPC also returns these gRPC errors:

  • INVALID_ARGUMENT: The operation value is not a valid PowerOperation.

Replace each <placeholder> with a real value. Define RMS_GRPCURL_FLAGS and RMS_ADDR as described in Getting started.

grpcurl $RMS_GRPCURL_FLAGS \
-d '{
"node_id": "<node_id>",
"operation": "POWER_OPERATION_OFF",
"rack_id": "<rack_id>"
}' \
"$RMS_ADDR" rack_manager.RackManager/SetPowerState

The rack_manager_client test client offers an equivalent command:

rack_manager_client <host> <port> power <rack_id> <node_id> <operation>

BatchSetPowerState

rack_manager.RackManager/BatchSetPowerState

BatchSetPowerState controls power for caller-supplied nodes without requiring them to be registered in RMS inventory.

PropertyValue
ExecutionSynchronous. The response carries the full result.
TargetCaller-supplied nodes. The request carries each node’s endpoints and credentials, and RMS does not store them.
Behavior detailsOperations guide
Protobuf referenceBatchSetPowerState

Request: BatchSetPowerStateRequest

FieldTypeDescription
nodesNodeSetCaller-supplied node connection details
operationPowerOperationPower operation to perform

Response: BatchSetPowerStateResponse

FieldTypeDescription
responseNodeBatchResponse-

Errors: Refer to Conventions for how RMS reports failures. This RPC also returns these gRPC errors:

  • INVALID_ARGUMENT: The operation value is not a valid PowerOperation.

Replace each <placeholder> with a real value. Define RMS_GRPCURL_FLAGS and RMS_ADDR as described in Getting started.

grpcurl $RMS_GRPCURL_FLAGS \
-d '{
"nodes": {
"nodes": [
{
"node_id": "<node_id>",
"rack_id": "<rack_id>",
"type": "COMPUTE_GB200_NVIDIA",
"bmc_endpoint": {
"interface": {
"ip_address": "<ip_address>",
"mac_address": "<mac_address>",
"host_name": "<host_name>"
},
"port": 0,
"credentials": {
"user_pass": {
"username": "<username>",
"password": "<password>"
}
}
},
"host_endpoint": {
"interface": {
"ip_address": "<ip_address>",
"mac_address": "<mac_address>",
"host_name": "<host_name>"
},
"port": 0,
"credentials": {
"user_pass": {
"username": "<username>",
"password": "<password>"
}
}
},
"node_descriptor": {
"attributes": {
"<key>": "<attributes>"
}
},
"additional_host_endpoints": [
{
"interface": {
"ip_address": "<ip_address>",
"mac_address": "<mac_address>",
"host_name": "<host_name>"
},
"port": 0,
"credentials": {
"user_pass": {
"username": "<username>",
"password": "<password>"
}
}
}
]
}
]
},
"operation": "POWER_OPERATION_OFF"
}' \
"$RMS_ADDR" rack_manager.RackManager/BatchSetPowerState

The rack_manager_client test client offers an equivalent command:

rack_manager_client <host> <port> batch_set_power_state <nodes.csv> <operation>

GetPowerState

rack_manager.RackManager/GetPowerState

GetPowerState retrieves the current power state of a node. Returns the current state (ON, OFF, or UNKNOWN).

PropertyValue
ExecutionSynchronous. The response carries the full result.
TargetRegistered inventory. Reads or changes nodes and racks stored in RMS inventory.
Behavior detailsOperations guide
Protobuf referenceGetPowerState

Request: GetPowerStateRequest

FieldTypeDescription
node_idstringTarget node identifier
rack_idstringRack containing the target node

Response: GetPowerStateResponse

FieldTypeDescription
statusReturnCodeRETURN_CODE_SUCCESS if query completed, RETURN_CODE_FAILURE otherwise
node_idstringNode identifier (echoed from request)
pstatestringCurrent power state: “ON”, “OFF”, or “UNKNOWN”
rack_idstringRack identifier (echoed from request)

Errors: Refer to Conventions for how RMS reports failures.

Replace each <placeholder> with a real value. Define RMS_GRPCURL_FLAGS and RMS_ADDR as described in Getting started.

grpcurl $RMS_GRPCURL_FLAGS \
-d '{
"node_id": "<node_id>",
"rack_id": "<rack_id>"
}' \
"$RMS_ADDR" rack_manager.RackManager/GetPowerState

The rack_manager_client test client offers an equivalent command:

rack_manager_client <host> <port> status <rack_id> <node_id>

BatchGetPowerState

rack_manager.RackManager/BatchGetPowerState

BatchGetPowerState retrieves the current power state for caller-supplied nodes without requiring them to be registered in RMS inventory. Returns per-node power state along with batch status.

PropertyValue
ExecutionSynchronous. The response carries the full result.
TargetCaller-supplied nodes. The request carries each node’s endpoints and credentials, and RMS does not store them.
Behavior detailsOperations guide
Protobuf referenceBatchGetPowerState

Request: BatchGetPowerStateRequest

FieldTypeDescription
nodesNodeSetCaller-supplied node connection details

Response: BatchGetPowerStateResponse

FieldTypeDescription
responseNodeBatchResponse-
node_power_statesrepeated NodePowerStatePower state for each successfully queried node

Errors: Refer to Conventions for how RMS reports failures.

Replace each <placeholder> with a real value. Define RMS_GRPCURL_FLAGS and RMS_ADDR as described in Getting started.

grpcurl $RMS_GRPCURL_FLAGS \
-d '{
"nodes": {
"nodes": [
{
"node_id": "<node_id>",
"rack_id": "<rack_id>",
"type": "COMPUTE_GB200_NVIDIA",
"bmc_endpoint": {
"interface": {
"ip_address": "<ip_address>",
"mac_address": "<mac_address>",
"host_name": "<host_name>"
},
"port": 0,
"credentials": {
"user_pass": {
"username": "<username>",
"password": "<password>"
}
}
},
"host_endpoint": {
"interface": {
"ip_address": "<ip_address>",
"mac_address": "<mac_address>",
"host_name": "<host_name>"
},
"port": 0,
"credentials": {
"user_pass": {
"username": "<username>",
"password": "<password>"
}
}
},
"node_descriptor": {
"attributes": {
"<key>": "<attributes>"
}
},
"additional_host_endpoints": [
{
"interface": {
"ip_address": "<ip_address>",
"mac_address": "<mac_address>",
"host_name": "<host_name>"
},
"port": 0,
"credentials": {
"user_pass": {
"username": "<username>",
"password": "<password>"
}
}
}
]
}
]
}
}' \
"$RMS_ADDR" rack_manager.RackManager/BatchGetPowerState

The rack_manager_client test client offers an equivalent command:

rack_manager_client <host> <port> batch_get_power_state <nodes.csv>

ExecuteColdReboot

rack_manager.RackManager/ExecuteColdReboot

ExecuteColdReboot orchestrates the full DGX GB200 rack COLD reboot sequence on caller-supplied nodes (no inventory registration required): gracefully shut down the compute trays over BMC, power off the power shelves, wait for capacitor discharge, power the shelves back on, wait for the compute/switch tray BMCs and switch NVOS to come up, then power on the compute nodes. Returns a coordinator job ID immediately; poll GetJobStatus on it: while running, execution_state is RUNNING and state_description reports the current stage by its rack-reboot runbook step number(s) (e.g. “Rack reboot step 7-8: …”); on completion or failure the full per-stage stage timeline is written to result_json. Runbook steps 9 (switch fabric provisioning) and 13 (post-reboot host/GPU verification) are out of scope. The sequence is fail-fast with no automated rollback.

The sequence stops at the first failed stage and does not roll back. The response job_id identifies the coordinator job.

PropertyValue
ExecutionAsynchronous, one job. The response returns a job ID and the work continues after the call returns.
Job trackingPoll GetJobStatus with the job ID in response.job_id.
Job resultresult_json in the job status holds the full per-stage timeline of the reboot sequence, written on completion or failure.
TargetCaller-supplied nodes. The request carries each node’s endpoints and credentials, and RMS does not store them.
Behavior detailsOperations guide
Protobuf referenceExecuteColdReboot

Request: ExecuteColdRebootRequest

FieldTypeDescription
nodesNodeSetParticipating devices: compute + power-shelf + switch trays (full NodeInfo)
domainoptional stringOptional NVL domain selecting switch mTLS material for the NVOS-readiness probe
prestage_shutdown_floor_secondsoptional uint32Min wait after issuing the compute graceful shutdown before probing (default 0)
prestage_shutdown_poll_max_secondsoptional uint32Max wait for compute to power down before the shelves are cut (default 300)
discharge_floor_secondsoptional uint32Min wait after shelf power-off before probing (default 180)
discharge_poll_max_secondsoptional uint32Max wait for BMCs to become unreachable (default 600)
bmc_up_floor_secondsoptional uint32Min wait after shelf power-on before probing (default 120)
bmc_up_poll_max_secondsoptional uint32Max wait for BMCs to become reachable (default 600)
nvos_hello_backoff_secondsoptional uint32Base delay for the exponential backoff between switch nvue-hello attempts (default 30)
compute_boot_wait_secondsoptional uint32Fixed wait for the compute OS to boot after power-on (default 300); the BMC cannot report OS readiness
poll_interval_secondsoptional uint32Poll cadence for all stages (default 15)

Response: ExecuteColdRebootResponse

FieldTypeDescription
responseNodeBatchResponseresponse.job_id is the coordinator job ID for GetJobStatus

Errors: Refer to Conventions for how RMS reports failures.

Replace each <placeholder> with a real value. Define RMS_GRPCURL_FLAGS and RMS_ADDR as described in Getting started.

grpcurl $RMS_GRPCURL_FLAGS \
-d '{
"nodes": {
"nodes": [
{
"node_id": "<node_id>",
"rack_id": "<rack_id>",
"type": "COMPUTE_GB200_NVIDIA",
"bmc_endpoint": {
"interface": {
"ip_address": "<ip_address>",
"mac_address": "<mac_address>",
"host_name": "<host_name>"
},
"port": 0,
"credentials": {
"user_pass": {
"username": "<username>",
"password": "<password>"
}
}
},
"host_endpoint": {
"interface": {
"ip_address": "<ip_address>",
"mac_address": "<mac_address>",
"host_name": "<host_name>"
},
"port": 0,
"credentials": {
"user_pass": {
"username": "<username>",
"password": "<password>"
}
}
},
"node_descriptor": {
"attributes": {
"<key>": "<attributes>"
}
},
"additional_host_endpoints": [
{
"interface": {
"ip_address": "<ip_address>",
"mac_address": "<mac_address>",
"host_name": "<host_name>"
},
"port": 0,
"credentials": {
"user_pass": {
"username": "<username>",
"password": "<password>"
}
}
}
]
}
]
},
"domain": "<domain>",
"prestage_shutdown_floor_seconds": 0,
"prestage_shutdown_poll_max_seconds": 0,
"discharge_floor_seconds": 0,
"discharge_poll_max_seconds": 0,
"bmc_up_floor_seconds": 0,
"bmc_up_poll_max_seconds": 0,
"nvos_hello_backoff_seconds": 0,
"compute_boot_wait_seconds": 0,
"poll_interval_seconds": 0
}' \
"$RMS_ADDR" rack_manager.RackManager/ExecuteColdReboot