Firmware

See Operations Overview for the conventions (in-band failures, registered vs. ephemeral targets) and the async job model that the asynchronous firmware RPCs below build on.

Firmware - inventory queries

Synchronous reads of installed firmware.

BatchGetFirmwareInventory is read-only and does not add the supplied nodes to RMS inventory. The test client accepts the standard direct-operation node CSV:

./rack_manager_client localhost 8801 --insecure \
  batch_get_firmware_inventory nodes.csv

Each CSV row supplies the node identity, endpoint, credentials, and node type. Switch rows use the host/NVUE endpoint; compute and power-shelf rows use the BMC endpoint. Successful inventories and per-node failures are both preserved when only part of a batch succeeds.

Firmware - asynchronous updates

These start firmware updates and return job IDs. Poll them with GetFirmwareJobStatus. A new update is admitted only if the target node is idle; a second job for the same (rack_id, node_id) is rejected as “update in progress.”

For NVFWUPD-backed updates, a node may select a configured expected hardware inventory with NodeDescriptor.attributes["inventory_profile"]. RMS validates the profile before admitting a child job and asks NVFWUPD to recheck the AP inventory before every target update and retry. A mismatch fails with failed-precondition details listing missing and present APs. Firmware version verification does not perform this inventory check.

GB200 in-band ConnectX updates

NVIDIA and Wiwynn GB200 compute nodes can update CX7, CX8, and BF3 NIC firmware through the host OS with Flint. The node registration must provide host SSH credentials and BMC credentials.

Before device discovery or flashing, RMS makes the host boot requirement explicit. It powers on a host that is not already on and waits up to six minutes for SSH and the OS service manager to become ready. If the OS does not boot, RMS performs one BMC auxiliary AC cycle, powers the host on, and repeats the six-minute boot wait once. Authentication and host-key failures fail immediately because a power cycle cannot correct them.

If MST enumeration or a per-device Flint query fails before image staging or flashing begins, RMS performs one auxiliary AC cycle, boots the host again, and repeats the complete device discovery once. A second discovery failure ends the job; RMS never uses this recovery path after a Flint flash has started.

When the node’s expected-inventory profile contains flint_devices counts, RMS also compares them with the physical CX7, CX8, and BF3 NICs reported by MST. Counts are per family and are not assumed to be four. A mismatch uses the same single pre-flash recovery cycle and must match on the second discovery. RMS repeats the count check during post-activation version verification.

After a successful flash, RMS follows the GB200 compute activation sequence: it forces the host off and requires Redfish to report PowerState=Off within one minute, performs the auxiliary AC cycle, waits three minutes for stabilization, powers the host on, and requires PowerState=On within one minute. If the first AC-cycle attempt does not reach On, RMS repeats the AC-cycle, stabilization, and power-on steps once before failing the job.

For a GB200 compute node with a configured host endpoint, activation is not complete until the host OS is reachable and ready again. RMS waits up to six minutes; if the OS does not boot, it performs one additional auxiliary AC cycle, waits three minutes, powers the host on, and repeats the six-minute OS readiness check once. Post-activation Flint version verification begins only after this boot check succeeds.

Supermicro GB300 sequencing

For the descriptor-only Supermicro GB300 compute type, firmware-object apply selects artifacts by filename from the manifest rather than by location order: the HMC *nosbios*.fwpkg, the BIOS*.bin, and the host-BMC *OBMC*.bin. RMS applies them in that order. With activation enabled it performs a full compute power cycle after nosbios and after BIOS; the host-BMC update runs last and does not add another host AC cycle. A combined nosbios-plus-BIOS request is rejected when activation is disabled because the required intermediate cycle would be skipped.

Firmware objects

A firmware object is a catalog entry built from a firmware manifest JSON: a software object tree plus RMS-parsed component and artifact metadata, scoped to a hardware_type, optionally marked default, and only available once RMS has downloaded all referenced artifacts. See the sample manifest.

component_filters restrict which components apply to which node type. CX7, CX8, and BF3 NIC artifacts are opt-in. If one firmware-object apply selects both Flint and out-of-band components for a GB200 compute node, RMS executes them as stages in one child job: it flashes and activates all in-band targets first, then processes and activates the out-of-band targets. A failed or cancelled in-band stage prevents the out-of-band stage from starting. Direct firmware-update RPCs continue to reject mixed in-band and out-of-band targets.