Switch State Diagram | NVIDIA Infra Controller

This document describes the Finite State Machine (FSM) for Switches in NICo: lifecycle from creation through configuration, validation, ready, optional rack-level reprovisioning, operator maintenance, decommissioning, and deletion.

High-Level Overview

The main flow shows the primary states and transitions:

States

State	Description
Created	Switch record exists in NICo; awaiting first controller tick.
Initializing	Controller associates discovered NVOS interfaces and waits for the expected switch’s primary NVOS interface. Sub-state: `WaitForOsMachineInterface`.
Configuring	Switch is being configured. Sub-states: `DisableMtls` (`Start` → `WaitForComplete { job_id }`), `ConfigureCertificate` (`Start` → `WaitForComplete { job_id }`), then `RotateOsPassword`. Refer to Switch Certificate Configuration.
FetchInfo	Controller fetches rack placement info (slot and tray) from the component manager backend and persists it on the switch record. Warn-and-continue on lookup failure.
Validating	Switch is being validated. Sub-state: `ValidationComplete`.
BomValidating	BOM (Bill of Materials) validation. Sub-state: `BomValidationComplete`.
Ready	Switch is ready for use. Can be deleted, enter decommissioning, enter operator maintenance, or enter rack-level reprovisioning.
Decommissioning	Switch is being removed from managed service. NICo suppresses discovery and DHCP, factory-resets NVOS and the BMC, and deletes managed credentials. Ends in terminal sub-state `Decommissioned`.
Maintenance	Operator-requested maintenance in progress. Operation is carried in `operation`: `PowerOn`, `PowerOff`, `Reset`, or `ReconfigureCertificate`. Certificate reconfiguration uses `configure_certificate` sub-states (`Start` → `WaitForComplete { job_id }`).
ReProvisioning	Rack-driven maintenance in progress. Sub-states: `WaitingForRackFirmwareUpgrade`, `WaitingForNVOSUpgrade`, `WaitingForNMXCConfigure`. The rack state machine sets per-switch status fields and clears `switch_reprovisioning_requested` when the cycle completes.
Error	Switch is in error (for example firmware upgrade failed, certificate job failed, or NVOS MAC conflict). Can transition to `Deleting` or `Maintenance`; otherwise waits for manual intervention.
Deleting	Switch is being removed; ends in final delete (terminal).

Transitions (by trigger)

From	To	Trigger / Condition
(create)	Created	Switch created
Created	Initializing (`WaitForOsMachineInterface`)	Controller processes switch
Initializing (`WaitForOsMachineInterface`)	Configuring (`DisableMtls` `Start`)	Primary NVOS interface associated for expected switch
Configuring (`DisableMtls` `Start`)	Configuring (`DisableMtls` `WaitForComplete`)	Component manager returns RMS `BatchDisableSwitchMtls` job ID
Configuring (`DisableMtls` `Start`)	Configuring (`ConfigureCertificate` `Start`)	No `rack_id`, no component manager, no services, or backend does not support the operation (skip disable step)
Configuring (`DisableMtls` `Start`)	Error	Missing BMC MAC or endpoint, or the submission outcome is unknown
Configuring (`DisableMtls` `WaitForComplete`)	Configuring (`ConfigureCertificate` `Start`)	RMS disable job and every child `Completed`
Configuring (`DisableMtls` `WaitForComplete`)	Error	RMS job `Failed`, or its outcome cannot be observed
Initializing (`WaitForOsMachineInterface`)	Error	Expected switch declares no NVOS ethernet ports, has no resolvable primary port, or declares a MAC owned by another switch
Configuring (`ConfigureCertificate` `Start`)	Configuring (`ConfigureCertificate` `WaitForComplete`)	Component manager returns RMS job ID (`domain_name` = `rack_id`)
Configuring (`ConfigureCertificate` `Start`)	Configuring (`RotateOsPassword`)	No `rack_id` or no component manager (skip certificate step)
Configuring (`ConfigureCertificate` `WaitForComplete`)	Configuring (`RotateOsPassword`)	RMS job `Completed`
Configuring (`ConfigureCertificate` `WaitForComplete`)	Error	RMS job `Failed`
Configuring (`RotateOsPassword`)	FetchInfo	NVOS credentials stored or already in the credential store
Configuring (`RotateOsPassword`)	Error	No expected switch or missing BMC MAC
FetchInfo	Validating (`ValidationComplete`)	Slot/tray lookup attempted (always advances)
Validating (`ValidationComplete`)	BomValidating (`BomValidationComplete`)	Validation complete
BomValidating (`BomValidationComplete`)	Ready	BOM validation complete
Ready	Deleting	`deleted` set (marked for deletion)
Ready	Decommissioning (`SuppressingSiteExplorer`)	`decommission_requested` is set
Ready	Maintenance	`switch_maintenance_requested` is set
Ready	ReProvisioning (`WaitingForRackFirmwareUpgrade`)	`switch_reprovisioning_requested` with `initiator` prefixed `rack-`
Ready	Error	Unknown `switch_reprovisioning_requested` initiator
Maintenance (`PowerOn` / `PowerOff` / `Reset`)	Ready	BMC operation complete; maintenance request cleared
Maintenance (`PowerOn` / `PowerOff` / `Reset`)	Error	BMC operation failed
Maintenance (`ReconfigureCertificate` `Start`)	Maintenance (`ReconfigureCertificate` `WaitForComplete`)	Component manager returns RMS job ID
Maintenance (`ReconfigureCertificate` `WaitForComplete`)	Ready	RMS job `Completed`; maintenance request cleared
Maintenance (`ReconfigureCertificate` `WaitForComplete`)	Error	RMS job `Failed`
ReProvisioning (`WaitingForRackFirmwareUpgrade`)	ReProvisioning (`WaitingForNVOSUpgrade`)	`firmware_upgrade_status` terminal `Completed` and `continue_after_firmware_upgrade` is true
ReProvisioning (`WaitingForRackFirmwareUpgrade`)	Ready	`firmware_upgrade_status` terminal `Completed` and `continue_after_firmware_upgrade` is false
ReProvisioning (`WaitingForRackFirmwareUpgrade`)	Error	`firmware_upgrade_status` terminal `Failed`
ReProvisioning (`WaitingForNVOSUpgrade`)	ReProvisioning (`WaitingForNMXCConfigure`)	`nvos_update_status` terminal `Completed`
ReProvisioning (`WaitingForNVOSUpgrade`)	Error	`nvos_update_status` terminal `Failed`
ReProvisioning (`WaitingForNMXCConfigure`)	Ready	`fabric_manager_status` reports running
ReProvisioning (`WaitingForNMXCConfigure`)	Error	`fabric_manager_status` reports an error message
ReProvisioning (any sub-state)	Ready	Parent rack entered `Error`; rack-level reprovision request cleared
Error	Deleting	`deleted` set (marked for deletion)
Error	Maintenance	`switch_maintenance_requested` is set
Decommissioning (`SuppressingSiteExplorer`)	Decommissioning (`FactoryResetNvos`)	Site Explorer acknowledges the BMC suppression
Decommissioning (`FactoryResetNvos`)	Decommissioning (`WaitingForNvosFactoryReset { job_id }`)	RMS returns a reset job ID; the controller persists it for polling
Decommissioning (`FactoryResetNvos`)	Decommissioning (`NvosFactoryResetOutcomeUnknown { error }`)	Submission returns `OperationOutcomeUnknown`; operator recovery is required
Decommissioning (`WaitingForNvosFactoryReset { job_id }`)	Same sub-state	RMS reports `Pending`, or a transient polling error occurs; poll the same job again
Decommissioning (`WaitingForNvosFactoryReset { job_id }`)	Same sub-state	RMS reports `Failed`; handler reports `ManualInterventionRequired`
Decommissioning (`WaitingForNvosFactoryReset { job_id }`)	Decommissioning (`SuppressingNvosDhcp`)	RMS reports `Completed`
Decommissioning (`NvosFactoryResetOutcomeUnknown { error }`)	Same sub-state	Handler reports `ManualInterventionRequired`; no automatic reset resubmission
Decommissioning (`SuppressingNvosDhcp`)	Decommissioning (`RebootingSwitch`)	NVOS DHCP suppression is recorded
Decommissioning (`RebootingSwitch`)	Decommissioning (`WaitingForNvosDhcpAcknowledgement`)	Redfish accepts `ForceRestart`
Decommissioning (`WaitingForNvosDhcpAcknowledgement`)	Same sub-state	NVOS DHCP acknowledgement is required and not yet recorded
Decommissioning (`WaitingForNvosDhcpAcknowledgement`)	Decommissioning (`SuppressingBmcDhcp`)	No NVOS DHCP acknowledgement pending
Decommissioning (`SuppressingBmcDhcp`)	Decommissioning (`FactoryResetBmc`)	BMC DHCP suppression is recorded
Decommissioning (`FactoryResetBmc`)	Decommissioning (`WaitingForBmcDhcpAcknowledgement`)	Redfish accepts the BMC factory-reset request
Decommissioning (`WaitingForBmcDhcpAcknowledgement`)	Same sub-state	BMC DHCP acknowledgement is required and not yet recorded
Decommissioning (`WaitingForBmcDhcpAcknowledgement`)	Decommissioning (`DeletingManagedCredentials`)	No BMC DHCP acknowledgement pending
Decommissioning (`DeletingManagedCredentials`)	Decommissioning (`Decommissioned`)	Managed BMC and NVOS credentials and convergence records are deleted
Deleting	(end)	Final delete committed

Rack-Level Re-Provisioning

Rack maintenance drives switch reprovisioning by setting switch_reprovisioning_requested with an initiator prefixed rack-. The switch controller then walks these sub-states in order when continue_after_firmware_upgrade is true (the default):

WaitingForRackFirmwareUpgrade -> WaitingForNVOSUpgrade -> WaitingForNMXCConfigure -> Ready

The rack state machine updates per-switch firmware_upgrade_status, nvos_update_status, and fabric_manager_status while the switch waits in each sub-state. If the parent rack enters Error, the switch aborts reprovisioning and returns to Ready.

On-Demand Maintenance Operations

The Maintenance state is entered when switch_maintenance_requested is posted (from Ready or Error). Supported operations:

PowerOn — powers the switch on via the component manager.
PowerOff — powers the switch off.
Reset — force-restarts the switch.
ReconfigureCertificate — reinstalls or rotates NVOS mTLS certificates via component manager / RMS. Uses the same async certificate flow as bring-up; see Switch Certificate Configuration.

Decommissioning

Managed-switch decommissioning requires the RMS switch backend and starts only from Ready. After Site Explorer acknowledges suppression, the controller submits the NVOS factory reset, persists the returned job ID, and polls that job until RMS reports Completed. It then suppresses NVOS DHCP, requests a switch reboot through Redfish, and completes the NVOS DHCP wait before suppressing BMC DHCP and resetting the BMC. It then completes the BMC DHCP wait before deleting managed credentials. Either endpoint can complete its wait without acknowledgement. Refer to DHCP Suppression Acknowledgement for the shared rule.

Transient handler errors leave the switch in its current sub-state for a later retry. A failed RMS reset job reports ManualInterventionRequired while retaining the job ID in WaitingForNvosFactoryReset. An ambiguous submission outcome enters NvosFactoryResetOutcomeUnknown, which requires operator recovery and does not automatically resubmit the reset.

Refer to Decommission Managed Switches for the operator procedure and intended post-reset state.

Implementation

State type: SwitchControllerState in crates/api-model/src/switch/mod.rs.
Handlers: crates/switch-controller/src/ — one module per top-level state (created, initializing, configuring, fetch_info, validating, bom_validating, ready, decommissioning, maintenance, reprovisioning, error_state, deleting).
Certificate configuration design: switch_configure_certificate.md.
Orchestration: SwitchStateHandler in handler.rs delegates to the handler for the current controller_state.