Rack State Machine interaction with Machine, Switch, and Power Shelf

This document defines the combined state machines for Machine (each compute tray / managed host lifecycle), Switch (each NVLink switch), Power Shelf (each power shelf), and Rack (collection of machines, switches, and power shelves). The diagram below shows all four and the transitions between the Rack state machine and the child device state machines.

For switch-only and power-shelf-only diagrams, see Switch State Diagram and Power Shelf State Diagram. For on-demand rack maintenance API details, see On-Demand Rack Maintenance.

Combined State Diagram (Machine, Switch, Power Shelf, Rack)

Rack State Machine Flow

The Rack state machine represents a collection of machines, switches, and power shelves. The rack lifecycle runs in coordination with child device state machines: the rack tracks when its children are created and ready, drives maintenance (firmware upgrade, NVOS update, NMX cluster configuration, optional power sequencing), validates the rack via RVS, and reaches Ready when validation completes and all trays are healthy.

High-Level Rack Diagram

Rack State Definitions

Created (R_Created)

• Entry: Site operator enters the expected rack; Site Explorer creates the rack entity when a machine, switch, or power shelf with the same rack_id is discovered.
• Exit: When expected devices are present and discovery can proceed, the rack moves to Discovering.

Discovering (R_Discovering)

• Entry: From Created when discovery starts. Also re-entered from Ready when topology_changed is set (tray replacement).
• Exit: When every machine in the rack is ManagedHostState::Ready or Assigned, every switch is SwitchControllerState::Ready, and every power shelf is Ready per the rack profile capability counts, the rack transitions to Maintenance at FirmwareUpgrade(Start).

The rack waits until all child devices reach ready before starting the first maintenance cycle.

Maintenance (R_Maintenance)

• Entry: From Discovering after all children are ready, from Ready when reprovision_requested or maintenance_requested is set, or from Error when on-demand maintenance is requested.
• Exit:
  • To Validating (Pending) when maintenance reaches Completed.
  • To Error on timeout or unrecoverable failure.

Sub-state flow (activities may be skipped based on MaintenanceScope.activities):

FirmwareUpgrade(Start -> WaitForComplete)
  -> NVOSUpdate(Start -> WaitForComplete)
  -> ConfigureNmxCluster(Start -> WaitForSwitchCertificateJob
                               -> WaitForScaleUpFabricManagerJob
                               -> WaitForPrimarySwitchCertificateJob (optional telemetry))
  -> PowerSequence (optional)
  -> Completed
  -> Validating(Pending)

During automatic maintenance, a missing rack-profile firmware_object skips both update phases. If a switch in the maintenance scope is already in ReProvisioning::WaitingForNVOSUpgrade, the rack transitions to Error instead of skipping the NVOS phase.

ConfigureNmxCluster sub-states

Start submits one ConfigureSwitchCertificate request containing every rack switch. The request always binds nvue_api and also binds the explicitly configured gNMI telemetry interface. After that job completes, the rack submits the unchanged RMS ScaleUpFabricManager V2 request. RMS selects the primary and reconciles the fabric and binds NMX-C to the current NVUE material. NICo persists the observed primary. When scale_up_fabric_telemetry is in the effective service list, NICo submits a certificate request for that primary. Each submitted parent RMS job ID is persisted while NICo polls the corresponding operation. The RMS V2 deployment must rebind NMX-C on every run, including when its existing Hello check succeeds; otherwise a rotation can leave NMX-C on the old certificate.

Validating (R_Validating)

• Entry: From Maintenance(Completed).
• Exit:
  • To Ready when validation reaches Validated and all child devices are ready.
  • To Error on terminal validation failure.

Sub-states: Pending → InProgress → Partial / FailedPartial → Validated or Failed. RVS drives transitions by writing rv.run-id and partition result labels on rack machines.

Ready (R_Ready)

• Entry: From Validating when validation completes and every tray is healthy.
• Exit:
  • To Maintenance when reprovision_requested or maintenance_requested is set.
  • To Discovering when topology_changed is set (tray replacement).
  • To Error when any child switch, power shelf, or machine enters a terminal failure state.

The rack is fully operational. While ready, it monitors child health and accepts reprovisioning or on-demand maintenance requests.

Error (R_Error)

• Entry: From Maintenance, Validating, or Ready on failure.
• Exit:
  • To Ready when a component-origin error is marked for automatic recovery and all child devices are healthy again.
  • To Maintenance when on-demand maintenance is requested from error state.

Maintenance-origin errors remain in Error until a new maintenance request is submitted. Errors without recovery metadata retain automatic component-readiness recovery.

Deleting (R_Deleting)

• Entry: When the rack is marked deleted.
• Exit: Terminal delete.

Switch Interaction with Rack

The Rack state machine drives or observes the Switch state machine as follows:

These cross-state dependencies are shown in the Combined State Diagram.

Switch State Machine Flow (summary)

The Switch state machine runs on each switch. The lifecycle runs from creation through initialization, certificate configuration, password rotation, slot/tray fetch, validation, BOM validation, and Ready. From Ready a switch can enter operator Maintenance, rack-driven ReProvisioning, Deleting, or Error.

Bring-up flow:

Created -> Initializing -> Configuring -> FetchInfo -> Validating -> BomValidating -> Ready

Rack reprovisioning flow (when continue_after_firmware_upgrade is true):

Ready -> ReProvisioning(WaitingForRackFirmwareUpgrade
                      -> WaitingForNVOSUpgrade
                      -> WaitingForNMXCConfigure) -> Ready

See Switch State Diagram for the full switch FSM.

Machine Interaction with Rack

The Rack state machine drives or observes the Machine (compute) state machine as follows:

These cross-state dependencies are shown in the Combined State Diagram.

Power Shelf Interaction with Rack

See Power Shelf State Diagram for the power-shelf FSM.

Recovering from M_HostReprovision::FailedFirmwareUpgrade

A compute machine that fails its host firmware upgrade lands in the M_HostReprovision::FailedFirmwareUpgrade substate. There are two ways out:

1. Automatic retry. While retry_count < MAX_FIRMWARE_UPGRADE_RETRIES and the configured host_firmware_upgrade_retry_interval has elapsed since the failure, the machine state handler automatically transitions back to CheckingFirmwareV2 and re-attempts the upgrade.
2. Fresh Host Reprovision request. The Rack state machine (or an operator via trigger_host_reprovisioning) can issue a brand-new Host Reprovision request at any time. The new request overwrites host_reprovisioning_requested with started_at = None; the FailedFirmwareUpgrade handler detects this fresh request and restarts the upgrade flow from CheckingFirmwareV2 with retry_count reset to 0, mirroring the way ManagedHostState::Ready kicks off a Host Reprovision (including the host-fw-update health-report alert merge). Rack-level requests (initiator prefixed with rack-) instead enter WaitingForRackFirmwareUpgrade.

This guarantees the Rack can always drive a stuck compute out of FailedFirmwareUpgrade without waiting for the retry backoff, which is important during R_Maintenance where the Rack must converge all computes back to M_Ready before progressing to R_Validation.

Tray Replacement (External Event)

When a tray (compute machine) in a rack is physically replaced, the rack topology changes. This is an external event that triggers the following state machine transitions:

1. Old machine: The replaced machine is removed from the rack. Its Machine state machine terminates (deletion path).
2. New machine: Site Explorer detects the new tray and creates a new machine entity in M_Created. The new machine progresses through its ingestion states toward M_Ready.
3. Rack: The rack detects the topology change (topology_changed) and transitions from R_Ready → R_Discovering. In R_Discovering the rack waits until the new machine reaches M_Ready (and all other machines, switches, and power shelves remain ready), then proceeds through R_Maintenance → R_Validation → R_Ready as in the normal flow.

This ensures that any replaced hardware is fully discovered, provisioned, and validated before the rack returns to an operational ready state.

How the data is organized

A rack is the top-level entity. Every machine (compute tray), every switch, and every power shelf belongs to exactly one rack.

• Each rack has a unique identifier (the rack ID).
• Each child device stores the rack ID of the rack it belongs to.
• Each switch has switch_reprovisioning_requested and per-cycle status fields (firmware_upgrade_status, nvos_update_status, fabric_manager_status) that the rack state machine sets during maintenance.

When a site operator enters an expected rack (with a rack ID and rack type), Site Explorer creates the rack entity as soon as it discovers machines, switches, or power shelves that share the same rack ID. From that point the rack tracks its children through their respective state machines until the entire rack reaches a ready state.

Implementation

• Rack state type: RackState in crates/api-model/src/rack.rs.
• Rack handlers: crates/rack-controller/src/.
• Switch state type: SwitchControllerState in crates/api-model/src/switch/mod.rs.
• Switch handlers: crates/switch-controller/src/.
• On-demand maintenance API: On-Demand Rack Maintenance.