Configure Firmware Versions

View as Markdown

NICo needs a site-specific definition of acceptable firmware before it can detect drift or choose an upgrade target. Host firmware and DPU NIC firmware use different configuration models. Rack and component updates use a target supplied by the request or selected by their backend.

The firmware catalog answers what firmware NICo recognizes and should install. Start there: define host firmware through the Host Firmware Config API or legacy metadata. Later sections cover DPU baselines and the site controls that determine when NICo looks for work and how much work it may start.

Host firmware catalog

Each host firmware definition is associated with a BMC vendor and host model. The definition can come from the Host Firmware Config API, a legacy metadata.toml file, or the static host_models configuration. NICo merges these sources into one effective catalog.

During site exploration, Redfish reports the BMC vendor and host model. NICo uses that pair to select the corresponding definition from the catalog. A definition can contain several components, such as BMC, UEFI, CPLD, HGX BMC, GPU, NIC, or CX7 firmware.

A definition contains model-level settings and one or more component entries:

SettingScopeMeaning
vendorModelBMC vendor to which the definition applies.
modelModelHost model to which the definition applies. Together, vendor and model identify the definition.
componentsModelGroups the component definitions, keyed by component type.
orderingModelLists the components in the order NICo should update them.
explicit_start_neededModelWhen enabled, drift alone does not start the update.
typeComponentIdentifies the component as BMC, UEFI, CEC, NIC, CPLD, HGX BMC, GPU, CX7, or a combined BMC and UEFI package. In legacy TOML, the type is the key below components.
current_version_reported_asComponentRegular expression used to find the component in the firmware inventory reported by the BMC. The API derives this matcher from the vendor, model, and component type.
preingest_upgrade_when_belowComponentSets the threshold for pre-ingestion firmware work. A lower reported version triggers an update when global automatic updates are enabled.
known_firmwareComponentLists the versions NICo can install and the artifacts used to install them.
versionFirmware versionVersion string NICo expects the firmware inventory to report after installation.
defaultFirmware versionMarks the desired version used for drift detection after ingestion.
preingestion_exclusive_configFirmware versionMarks a version for use during pre-ingestion without making it the steady-state default.
filesFirmware versionStructured list of firmware artifacts. Each entry provides a filename or URL and can include a SHA-256 digest. The API exposes this list as artifacts.
install_only_specifiedFirmware versionTells Redfish to install only the component identified by the firmware definition when the package contains multiple components.
power_drains_neededFirmware versionNumber of full power-drain cycles required to activate the firmware after installation.
pre_update_resetsFirmware versionRequests the platform-specific reset sequence before NICo starts the installation.
scriptFirmware versionLegacy path that hands the update to a local script instead of using the normal Redfish upload path.

current_version_reported_as is matched against firmware inventory entry identifiers, not against version strings. For example, ^Installed-.*__iDRAC. selects the iDRAC entry from a Dell Redfish firmware inventory. NICo then reads the installed version from that matching entry.

Legacy metadata also accepts the older artifact fields filename, filenames, url, and checksum; new API entries use URL-based artifacts. The data model still parses mandatory_upgrade_from_priority and scout, but the current upgrade workflow does not use them as firmware-policy controls, so they are not listed as active settings here.

Minimum and default versions

The pre-ingestion minimum is an upgrade trigger, not the upgrade target. When a host is below the minimum and global automatic updates are enabled, NICo normally installs the version marked as the default. A firmware version can instead set preingestion_exclusive_config = true when the platform needs a different target during ingestion.

For example, assume the BMC minimum is 7.00.00.00 and the default is 7.10.30.00:

Reported versionResult
6.50.00.00Below the minimum. With automatic updates enabled, NICo updates the BMC during pre-ingestion, using 7.10.30.00 as the target.
7.05.00.00Meets the minimum, so ingestion can continue. After ingestion it is still drifted from 7.10.30.00.
7.10.30.00Meets the minimum and matches the desired version. No firmware update is needed.

Refer to Pre-ingestion Firmware Updates for minimum-version enforcement and Managed Host Firmware Updates for post-ingestion drift handling.

Configure host firmware through the API

The current way to configure host firmware is the Create or Update Host Firmware Config operation:

PUT /v2/org/:org/nico/firmware-config/host

The request is site-scoped and keyed by (vendor, model). NICo stores the configuration in host_firmware_config and uses it on subsequent discovery, pre-ingestion, drift detection, and upgrade passes. This allows a site to change its firmware catalog without changing the NICo deployment or placing metadata files on the Core filesystem.

The endpoint requires Provider Admin authorization. The following request defines one BMC version for a Dell PowerEdge R760:

$curl -X PUT "https://<api-url>/v2/org/<org>/nico/firmware-config/host" \
> -H "Authorization: Bearer <token>" \
> -H "Content-Type: application/json" \
> -d '{
> "siteId": "<site-id>",
> "vendor": "Dell",
> "model": "PowerEdge R760",
> "ordering": ["BMC"],
> "components": [
> {
> "type": "BMC",
> "preingestUpgradeWhenBelow": "7.00.00.00",
> "firmware": [
> {
> "version": "7.10.30.00",
> "default": true,
> "artifacts": [
> { "url": "https://<firmware-repository>/<artifact>" }
> ]
> }
> ]
> }
> ]
> }'

The request does not contain an inventory-matching regular expression. Core derives that expression from its supported vendor, model, and component mappings. The request is rejected when no mapping exists.

PUT is an upsert, not a complete replacement:

  • The first request for a vendor and model must provide an ordering that contains every configured component.
  • Each component must have exactly one default firmware version after the request is merged.
  • Later requests can add components or firmware versions. Versions are merged by version string, and omitted components remain unchanged.
  • Marking a new version as the default clears the previous default for that component.

The Delete Host Firmware Config operation removes the runtime entry. If the same vendor and model also exists in static or legacy metadata, that lower-priority definition becomes effective again.

Legacy metadata.toml configuration

Before the Host Firmware Config API was introduced, host firmware definitions were deployed as metadata.toml files. This is the legacy configuration path. It remains supported for existing deployments, but new host firmware definitions should use the API.

firmware_global.firmware_directory points to a directory containing one subdirectory per metadata entry. NICo looks for a file named metadata.toml in each of those subdirectories. For example:

/opt/nico/firmware/
`-- dell-r760-bmc-7.10.30/
`-- metadata.toml

Each file describes one vendor and model and can define one or more firmware components. The equivalent legacy definition for the API example above is:

1vendor = "Dell"
2model = "PowerEdge R760"
3ordering = ["bmc"]
4
5[components.bmc]
6current_version_reported_as = "^Installed-.*__iDRAC."
7preingest_upgrade_when_below = "7.00.00.00"
8
9[[components.bmc.known_firmware]]
10version = "7.10.30.00"
11url = "https://<firmware-repository>/<artifact>"
12default = true

NICo also supports host definitions embedded in the static host_models configuration. This is the base catalog supplied with the NICo deployment and uses the same firmware data model as metadata.toml.

NICo builds the effective host firmware catalog in this order:

AppliedSource
FirstStatic host_models configuration.
NextLegacy metadata.toml files under firmware_global.firmware_directory.
Last (highest precedence)Runtime host_firmware_config entries created through the API.

Higher-priority sources overlay matching vendor, model, and component entries; they do not discard unrelated lower-priority entries. For example, if a metadata.toml file defines BMC 7.00.00.00 as the default and an API request adds BMC 7.10.30.00 as the default, both remain known versions, but 7.10.30.00 becomes the effective target.

Use the CLI to inspect the effective default versions after all three sources have been merged:

$nico-admin-cli -a <core-api-url> firmware show

DPU firmware

DPU firmware does not use the Host Firmware Config API. Its site configuration has two related parts: an accepted-version list that triggers automatic work, and a model catalog used to verify the result.

SettingDefaultMeaning
dpu_config.dpu_nic_firmware_initial_update_enabledfalseCompatibility setting for initial DPU setup. The current state machine records it but does not use it to suppress BFB installation or verification.
dpu_config.dpu_nic_firmware_reprovision_update_enabledtrueEnables automatic DPU NIC drift selection in Machine Update Manager. Explicit DPU reprovisioning remains available when disabled.
dpu_config.dpu_nic_firmware_update_versionsBuilt-in BF2 and BF3 NIC versionsAccepted NIC versions used by the classic automatic drift detector.
dpu_config.dpu_modelsBuilt-in BF2 and BF3 definitionsBMC, ERoT/CEC, NIC, and UEFI baselines associated with each DPU model.
dpu_config.dpu_enable_secure_bootfalseAllows classic reprovisioning to install the BFB through Redfish and enable secure boot when every attached DPU supports that path. Otherwise NICo uses UEFI HTTP network installation.

Automatic DPU NIC trigger

NICo checks the reported DPU NIC version against dpu_config.dpu_nic_firmware_update_versions:

1[dpu_config]
2dpu_nic_firmware_update_versions = ["32.43.1014", "32.44.1030"]

This is an accepted-version list rather than a single default. A DPU reporting either version in the example is considered compliant. A version outside the list is drifted and, when DPU NIC updates are enabled, makes the DPU eligible for reprovisioning. Allowing more than one version is useful during a staged rollout where both the old and new versions must temporarily remain valid.

dpu_nic_firmware_reprovision_update_enabled enables this automatic drift module. It does not prevent an operator from requesting DPU reprovisioning.

Post-reprovision baseline

The dpu_models catalog describes the component versions expected after a classic DPU reprovision. Entries are keyed by NICo’s normalized DPU model, currently bluefield2 or bluefield3, and use the same firmware structures as the static host catalog:

1[dpu_config.dpu_models.bluefield3]
2vendor = "Nvidia"
3model = "Bluefield 3 SmartNIC Main Card"
4ordering = ["bmc", "cec"]
5
6[dpu_config.dpu_models.bluefield3.components.nic]
7current_version_reported_as = "DPU_NIC"
8
9[[dpu_config.dpu_models.bluefield3.components.nic.known_firmware]]
10version = "32.44.1030"
11default = true

The built-in catalog defines BMC, CEC/ERoT, NIC, and UEFI entries for each model. The current reprovisioning verifier compares BMC, CEC/ERoT, and NIC. It selects the last non-pre-ingestion-only known version for each component; in normal configurations, keep one steady-state entry for each component.

NICo applies these versions by installing or booting the deployed BlueField Bundle, not by uploading the known_firmware entries as independent packages. The accepted NIC list, DPU model catalog, and firmware carried by forge.bfb must therefore be updated together. For example, if the BFB installs NIC version 32.44.1030, that version must be accepted by dpu_nic_firmware_update_versions and should be the NIC baseline in the matching dpu_models entry.

Refer to DPU firmware updates for how NICo acts on this configuration.

Operational controls

The settings in this section are fields in NICo Core’s site configuration. They control the automatic paths; they do not define component target versions.

Managed-machine scheduling

Machine Update Manager uses one capacity budget for managed-host and DPU firmware work:

SettingDefaultMeaning
machine_update_run_interval300Seconds between Machine Update Manager passes.
machine_updater.max_concurrent_machine_updates_absoluteunsetMaximum number of distinct hosts that may be in machine-update workflows. Set this to a positive value to allow automatic managed-host or DPU work.
machine_updater.max_concurrent_machine_updates_percentunsetIntended percentage-based capacity. The current implementation calculates this incorrectly; leave it unset and use the absolute limit.
machine_updater.instance_autoreboot_period.startunsetStart of a one-time UTC window in which an assigned host may consume an existing update request without tenant approval.
machine_updater.instance_autoreboot_period.endunsetEnd of that one-time UTC window.

For example, this allows at most five managed hosts to be in update workflows and gives assigned hosts a four-hour automatic reboot window:

1machine_update_run_interval = 300
2
3[machine_updater]
4max_concurrent_machine_updates_absolute = 5
5instance_autoreboot_period.start = "2026-08-15T01:00:00Z"
6instance_autoreboot_period.end = "2026-08-15T05:00:00Z"

The autoreboot period is not the per-host firmware window used by models with explicit_start_needed. Operators open that window through the managed-host API. Refer to Scheduling gates.

Firmware execution

firmware_global controls host firmware execution and the separate pre-ingestion manager:

SettingDefaultMeaning
firmware_global.autoupdatefalseEnables pre-ingestion upgrades and supplies the default automatic-update policy for managed hosts. A per-machine policy can override the managed-host default.
firmware_global.run_interval30sInterval between pre-ingestion manager passes.
firmware_global.max_uploads4Shared limit for concurrent host firmware uploads.
firmware_global.concurrency_limit16Maximum number of pre-ingestion endpoints processed concurrently. This is separate from Machine Update Manager capacity.
firmware_global.firmware_directory/opt/nico/firmware, otherwise /opt/carbide/firmwareDirectory containing legacy metadata and local firmware artifacts.
firmware_global.firmware_download_cache_directory/mnt/persistence/fw/download-cacheWritable cache for downloaded firmware artifacts.
firmware_global.host_firmware_upgrade_retry_interval60mDelay before the managed-host state machine retries a failed host firmware upgrade.
firmware_global.instance_updates_manual_taggingSee belowWhen true, automatic host selection is limited to unassigned hosts in top-level Ready. When false, assigned hosts can also receive a pending request.
firmware_global.no_reset_retriesfalseDisables the normal retry handling for BMC reset failures during pre-ingestion and managed-host firmware work.
firmware_global.hgx_bmc_gpu_reboot_delay30sDelay after a GPU reboot before NICo accesses the HGX BMC again.
firmware_global.requires_manual_upgradefalseAdds a manually confirmed upgrade gate for MNNVL-capable hosts.
firmware_global.max_concurrent_bfb_copies10Maximum concurrent BFB copies performed by the pre-ingestion manager.

Set instance_updates_manual_tagging explicitly. When a present firmware_global table omits the field, deserialization uses true; when the entire table is omitted, the current whole-structure default uses false.

A staging configuration can make the policy explicit while leaving site-wide automatic selection disabled:

1[firmware_global]
2autoupdate = false
3run_interval = "30s"
4max_uploads = 4
5concurrency_limit = 16
6host_firmware_upgrade_retry_interval = "60m"
7instance_updates_manual_tagging = true
8requires_manual_upgrade = false
9max_concurrent_bfb_copies = 10

The legacy host_enable_autoupdate and host_disable_autoupdate fields do not provide a reliable model allowlist. Use the per-machine automatic-update policy instead.

Roll out a new baseline

Treat a baseline change as a site-wide rollout, even when the first update is a canary:

  1. Update the canonical configuration source. Do not maintain a second live version table in this guide or a site runbook.
  2. Make every referenced host artifact available. For DPU firmware, update the BFB, accepted NIC versions, and dpu_models together.
  3. Check host component ordering, defaults, pre-ingestion thresholds, and inventory matchers. Then inspect the merged host catalog with firmware show.
  4. Configure a nonzero shared capacity, explicit-start windows, and assigned-host approval policy. Keep site-wide automatic selection disabled while validating one host through a per-machine host policy or an explicit DPU reprovisioning request.
  5. After verification, enable the intended site-wide automatic policy.
  6. Monitor the applicable workflow until inventory reports the configured versions and the host has returned to service.

Changing a pre-ingestion threshold while firmware_global.autoupdate is enabled can affect every matching endpoint on the next pre-ingestion pass. Validate the catalog and artifact availability before applying that change.

Rack and component firmware

Rack and component firmware do not use either of the configuration models above. Their target version comes from the update request or from the default selected by the rack or component backend. Refer to Rack and Tray Firmware Updates.