Managed Host State Diagrams

This document contains the complete Finite State Machine (FSM) that illustrates the lifecycle of NICo managed hosts from discovery through ingestion through instance assignment and management.

High-Level Overview

The main flow shows the primary states and transitions between them:

Failed recovery semantics

FailureCause::BiosSetupFailed is set when BIOS setup retries are exhausted during ingestion or instance deprovisioning. The Failed handler polls is_bios_setup and auto-recovers into SetBootOrder once the BMC reports success.

DPU Discovery State Details (DpuDiscoveringState)

Shows the complete DPU discovery and configuration process:

DPU Initialization State Details (DpuInitState)

Shows DPU initialization including BFB installation:

Host Initialization State Details (HostInitState)

Shows host initialization including boot order and UEFI setup:

BOM Validation State Details (BomValidating)

Shows the BOM (Bill of Materials) validation process:

Machine Validation State Details (ValidationState)

Shows the machine validation process:

Ready State Details (Ready)

For hosts already provisioned through DPF, a reprovisioning request can remain in Ready while NICo checks compatibility or retries a failed DPU restart. Refer to DPU Firmware Updates for the start checks, saved start timestamps, and retry behavior behind the reprovisioning transitions in these diagrams.

Shows what can happen in Ready state:

Decommissioning State Details

Decommissioning starts only from Ready and follows this top-level sequence:

DeconfiguringHost disables lockdown, unlocks managed SuperNICs, resets UEFI settings, and clears the host UEFI password. DeconfiguringDpus removes DPF resources when applicable and installs the vanilla preingestion.bfb on every DPU.

The OOB DHCP wait checks the host and DPU OOB interface MACs, and the BMC DHCP wait checks the host and DPU BMC MACs. Each DHCP wait advances after every MAC it checks has a recorded suppression acknowledgement or qualifies to advance without one. Refer to DHCP Suppression Acknowledgement for the shared rule.

Refer to Decommission Managed Hosts and DPUs for the operator procedure, platform limitations, and intended post-reset state.

Instance Assignment State Details (InstanceState)

Shows the complete instance assignment and management flow:

Network Readiness Gates

During initial provisioning, WaitingForNetworkConfig waits for all required network configuration observations to match their desired versions. For a host with associated DPUs, it also waits for aggregate health alerts that prevent host state changes to clear. A host without associated DPUs skips both the DPU observation and aggregate health checks in this state.

The normal agent report gives a p0 transport failure the combination of BgpPeeringTor and PreventAllocations. The PXE gate matches that probe ID and classification; it does not inspect the alert target. A host Replace report takes precedence for this check. Without one, the primary DPU Replace report takes precedence over that DPU’s Merge reports. Refer to DPU ToR Uplink Health for the uplink policy and classifications.

WaitingForRebootToReady repeats the aggregate health and primary p0 checks immediately before the normal PXE ForceRestart. This closes the interval in which health can change after the configuration check. Hosts without managed DPUs skip this PXE signal check, including one supplied by a host Replace report, but still receive the aggregate health check. In WaitingForNetworkConfig, a host Replace report can supply the matching signal before NICo looks for a primary DPU, including on a host without managed DPUs.

Normal success proceeds directly from WaitingForNetworkConfig to WaitingForRebootToReady. WaitingForStorageConfig and WaitingForExtensionServicesConfig remain only so NICo can recover hosts that persisted those states on an older release.

The following requests bypass these initial provisioning readiness checks:

• Instance deletion proceeds through its required restart without waiting for network health.
• An explicit custom iPXE request proceeds through its requested restart.

Live Network Update Resource Release

A live network update keeps old network resources until the required DPU reports and health checks pass. It does not restart the host or apply the primary p0 PXE gate.

For a host with associated DPUs, NICo checks these conditions in order:

1. Every DPU in the host topology reports the current managed-host network configuration version. A missing DPU snapshot, observation, or version does not count as an acknowledgement.
2. The number of instance network observations matches the number of DPUs required by the current instance configuration. Every recorded instance observation reports the current instance network configuration version.
3. The aggregate health report has no PreventHostStateChanges alert.

A host without associated DPUs skips these acknowledgement and aggregate health checks.

NetworkConfigUpdate/WaitingForConfigSynced runs the checks first. If they pass, the host enters NetworkConfigUpdate/ReleaseOldResources, which repeats them. Another DPU report can arrive after NICo saves the state change.

Missing or stale acknowledgements cause a wait. Blocking health causes a handler error. In either case, NICo keeps the same state, old resources, and pending update request, then checks again on a later iteration. Instance deletion does not bypass these checks for a pending live update.

After the repeated checks pass, NICo completes cleanup in one transaction. It releases old addresses and unused VPC loopbacks, marks removed generated segments for deletion, and clears the pending update request. The host returns to Assigned/Ready. To troubleshoot a pending update, refer to Diagnose a Live Network Update.

A DPU that loses all tenant interfaces still belongs to the host topology. It must acknowledge its Admin configuration before NICo can release the old resources.

Until that DPU reports again, its old instance observation can leave more instance observations than required DPUs, even if its managed-host version still matches. If applying Admin fails, the instance observation can disappear without proving that the old tenant configuration was removed. A healthy report or an absent instance observation is therefore not enough: NICo still needs the managed-host acknowledgement.

NICo uses the versions and health that the agents report. It does not independently verify hardware forwarding. Refer to Wait for Removed DPUs Before Releasing Instance Network Resources for background on this removed-DPU scenario.

Host Reprovision State Details (HostReprovisionState)

Shows the host firmware reprovision process:

DPU Reprovision State Details (DpuReprovisionState)

Shows the DPU firmware reprovision process:

WaitingForCleanup State Details

Measuring and PostAssignedMeasuring State Details

Shows the attestation measurement process.

Failed State