Configure Firmware Versions v2.1

NICo needs a site-specific definition of acceptable firmware before it can detect drift or choose an upgrade target. Host firmware and DPU NIC firmware use different configuration models. Rack and component updates use a target supplied by the request or selected by their backend.

The firmware catalog answers what firmware NICo recognizes and should install. Start there: define host firmware through the Host Firmware Config API or legacy metadata. Later sections cover DPU baselines and the site controls that determine when NICo looks for work and how much work it may start.

Host firmware catalog

Each host firmware definition is associated with a BMC vendor and host model. The definition can come from the Host Firmware Config API, a legacy metadata.toml file, or the static host_models configuration. NICo merges these sources into one effective catalog.

During site exploration, Redfish reports the BMC vendor and host model. NICo uses that pair to select the corresponding definition from the catalog. A definition can contain several components, such as BMC, UEFI, CPLD, HGX BMC, GPU, NIC, or CX7 firmware.

A definition contains model-level settings and one or more component entries:

current_version_reported_as is matched against firmware inventory entry identifiers, not against version strings. For example, ^Installed-.*__iDRAC. selects the iDRAC entry from a Dell Redfish firmware inventory. NICo then reads the installed version from that matching entry.

Legacy metadata also accepts the older artifact fields filename, filenames, url, and checksum; new API entries use URL-based artifacts. The data model still parses mandatory_upgrade_from_priority and scout, but the current upgrade workflow does not use them as firmware-policy controls, so they are not listed as active settings here.

Minimum and default versions

The pre-ingestion minimum is an upgrade trigger, not the upgrade target. When a host is below the minimum and global automatic updates are enabled, NICo normally installs the version marked as the default. A firmware version can instead set preingestion_exclusive_config = true when the platform needs a different target during ingestion.

For example, assume the BMC minimum is 7.00.00.00 and the default is 7.10.30.00:

Refer to Pre-ingestion Firmware Updates for minimum-version enforcement and Managed Host Firmware Updates for post-ingestion drift handling.

Configure host firmware through the API

The current way to configure host firmware is the Create or Update Host Firmware Config operation:

PUT /v2/org/:org/nico/firmware-config/host

The request is site-scoped and keyed by (vendor, model). NICo stores the configuration in host_firmware_config and uses it on subsequent discovery, pre-ingestion, drift detection, and upgrade passes. This allows a site to change its firmware catalog without changing the NICo deployment or placing metadata files on the Core filesystem.

The endpoint requires Provider Admin authorization. The following request defines one BMC version for a Dell PowerEdge R760:

curl -X PUT "https://<api-url>/v2/org/<org>/nico/firmware-config/host" \
  -H "Authorization: Bearer <token>" \
  -H "Content-Type: application/json" \
  -d '{
    "siteId": "<site-id>",
    "vendor": "Dell",
    "model": "PowerEdge R760",
    "ordering": ["BMC"],
    "components": [
      {
        "type": "BMC",
        "preingestUpgradeWhenBelow": "7.00.00.00",
        "firmware": [
          {
            "version": "7.10.30.00",
            "default": true,
            "artifacts": [
              { "url": "https://<firmware-repository>/<artifact>" }
            ]
          }
        ]
      }
    ]
  }'

The request does not contain an inventory-matching regular expression. Core derives that expression from its supported vendor, model, and component mappings. The request is rejected when no mapping exists.

PUT is an upsert, not a complete replacement:

• The first request for a vendor and model must provide an ordering that contains every configured component.
• Each component must have exactly one default firmware version after the request is merged.
• Later requests can add components or firmware versions. Versions are merged by version string, and omitted components remain unchanged.
• Marking a new version as the default clears the previous default for that component.

The Delete Host Firmware Config operation removes the runtime entry. If the same vendor and model also exists in static or legacy metadata, that lower-priority definition becomes effective again.

Legacy metadata.toml configuration

Before the Host Firmware Config API was introduced, host firmware definitions were deployed as metadata.toml files. This is the legacy configuration path. It remains supported for existing deployments, but new host firmware definitions should use the API.

firmware_global.firmware_directory points to a directory containing one subdirectory per metadata entry. NICo looks for a file named metadata.toml in each of those subdirectories. For example:

/opt/nico/firmware/
`-- dell-r760-bmc-7.10.30/
    `-- metadata.toml

Each file describes one vendor and model and can define one or more firmware components. The equivalent legacy definition for the API example above is:

vendor = "Dell"
model = "PowerEdge R760"
ordering = ["bmc"]
[components.bmc]
current_version_reported_as = "^Installed-.*__iDRAC."
preingest_upgrade_when_below = "7.00.00.00"
[[components.bmc.known_firmware]]
version = "7.10.30.00"
url = "https://<firmware-repository>/<artifact>"
default = true

NICo also supports host definitions embedded in the static host_models configuration. This is the base catalog supplied with the NICo deployment and uses the same firmware data model as metadata.toml.

NICo builds the effective host firmware catalog in this order:

Higher-priority sources overlay matching vendor, model, and component entries; they do not discard unrelated lower-priority entries. For example, if a metadata.toml file defines BMC 7.00.00.00 as the default and an API request adds BMC 7.10.30.00 as the default, both remain known versions, but 7.10.30.00 becomes the effective target.

Use the CLI to inspect the effective default versions after all three sources have been merged:

nico-admin-cli -a <core-api-url> firmware show

ConnectX-7 InfiniBand Firmware

NICo v2.0 and later use Scout to update CX7 firmware on hosts reported with vendor Nvidia, model DGXH100, and firmware inventory entries named CX7_<number>.

To prepare the update:

1. Publish the NVIDIA-signed CX7 firmware artifact at an HTTPS URL that Scout can access from the discovery environment.

   Use the DGX H100/H200 firmware guide to select an artifact compatible with every CX7 adapter in the host. The artifact’s SHA-256 digest is optional; when provided, it must be 64 hexadecimal characters and NICo uses it to verify the download. NICo does not require a particular repository path or filename.

2. Call the Host Firmware Config API with the following:

   • vendor: Nvidia
   • model: DGXH100
   • Component type: Cx7
   • version: 28.47.2682
   • default: true
   • powerDrainsNeeded: 1
   • One artifact containing the HTTPS url
   • sha256: (optional) the artifact’s 64-character SHA-256 digest to enable download verification

Use Monitor and verify. A host is updated only when every CX7_<number> entry reports the target version, the host has returned to Ready or Assigned/Ready, and its reprovisioning request is gone.

Other host models require their own inventory mapping and Scout script. To add support for another model:

• Follow the contributing guide.
• Add the catalog component regex for FirmwareComponentType::Cx7 to the host firmware inventory mappings for the model.
• Add the model’s script and metadata under the Scout firmware script assets.

The existing DGX H100 script might work on other servers, but it has not been tested on them; validate it on the new platform before reusing it.

DPU firmware

DPU firmware does not use the Host Firmware Config API. Its site configuration has two related parts: an accepted-version list that triggers automatic work, and a model catalog used to verify the result.

Automatic DPU NIC trigger

NICo checks the reported DPU NIC version against dpu_config.dpu_nic_firmware_update_versions:

[dpu_config]
dpu_nic_firmware_update_versions = ["32.43.1014", "32.44.1030"]

This is an accepted-version list rather than a single default. A DPU reporting either version in the example is considered compliant. A version outside the list is drifted and, when DPU NIC updates are enabled, makes the DPU eligible for reprovisioning. Allowing more than one version is useful during a staged rollout where both the old and new versions must temporarily remain valid.

dpu_nic_firmware_reprovision_update_enabled enables this automatic drift module. It does not prevent an operator from requesting DPU reprovisioning.

Post-reprovision baseline

The dpu_models catalog describes the component versions expected after a classic DPU reprovision. Entries are keyed by NICo’s normalized DPU model, currently bluefield2 or bluefield3, and use the same firmware structures as the static host catalog:

[dpu_config.dpu_models.bluefield3]
vendor = "Nvidia"
model = "Bluefield 3 SmartNIC Main Card"
ordering = ["bmc", "cec"]
[dpu_config.dpu_models.bluefield3.components.nic]
current_version_reported_as = "DPU_NIC"
[[dpu_config.dpu_models.bluefield3.components.nic.known_firmware]]
version = "32.44.1030"
default = true

The built-in catalog defines BMC, CEC/ERoT, NIC, and UEFI entries for each model. The current reprovisioning verifier compares BMC, CEC/ERoT, and NIC. It selects the last non-pre-ingestion-only known version for each component; in normal configurations, keep one steady-state entry for each component.

NICo applies these versions by installing or booting the deployed BlueField Bundle, not by uploading the known_firmware entries as independent packages. The accepted NIC list, DPU model catalog, and firmware carried by forge.bfb must therefore be updated together. For example, if the BFB installs NIC version 32.44.1030, that version must be accepted by dpu_nic_firmware_update_versions and should be the NIC baseline in the matching dpu_models entry.

Refer to DPU firmware updates for how NICo acts on this configuration.

Operational controls

The settings in this section are fields in NICo Core’s site configuration. They control the automatic paths; they do not define component target versions.

Managed-machine scheduling

Machine Update Manager uses one capacity budget for managed-host and DPU firmware work:

For example, this allows at most five managed hosts to be in update workflows and gives assigned hosts a four-hour automatic reboot window:

machine_update_run_interval = 300
[machine_updater]
max_concurrent_machine_updates_absolute = 5
instance_autoreboot_period.start = "2026-08-15T01:00:00Z"
instance_autoreboot_period.end = "2026-08-15T05:00:00Z"

The autoreboot period is not the per-host firmware window used by models with explicit_start_needed. Operators open that window through the managed-host API. Refer to Scheduling gates.

Firmware execution

firmware_global controls host firmware execution and the separate pre-ingestion manager:

Set instance_updates_manual_tagging explicitly. When a present firmware_global table omits the field, deserialization uses true; when the entire table is omitted, the current whole-structure default uses false.

A staging configuration can make the policy explicit while leaving site-wide automatic selection disabled:

[firmware_global]
autoupdate = false
run_interval = "30s"
max_uploads = 4
concurrency_limit = 16
host_firmware_upgrade_retry_interval = "60m"
instance_updates_manual_tagging = true
requires_manual_upgrade = false
max_concurrent_bfb_copies = 10

The legacy host_enable_autoupdate and host_disable_autoupdate fields do not provide a reliable model allowlist. Use the per-machine automatic-update policy instead.

Firmware downloads

These settings are planned for NICo v2.4. They require a Core build containing the download-limit implementation, with this reference tracked in the documentation companion. Earlier builds do not enforce these limits.

The optional [firmware_global.downloads] section limits URL artifacts fetched by Core for managed-host Redfish upgrades and non-BFB pre-ingestion upgrades. Both paths share one worker pool per Core process. Scout, Core-local scripts, filename-only artifacts, and pre-ingestion BFB delivery bypass this downloader.

Omit the section or individual fields to use their defaults. This also applies to existing configurations: with defaults, HTTP responses over 4 GiB fail, and HTTP requests have a 1,800-second total deadline. Core validates the settings at startup; changes require a Core restart.

The following table lists the download settings and their inclusive ranges:

All values are positive integers; write timeouts as seconds, not duration strings such as "30s". Both connect_timeout_seconds and read_timeout_seconds must be less than or equal to request_timeout_seconds. Core rejects invalid values at startup instead of disabling the limit.

The read timer starts with the request and runs through receipt of final response headers, including connection establishment. It then resets after each body read makes progress. Whichever active timer expires first ends the HTTP attempt. Time spent writing received bytes counts toward the total request deadline, but the deadline cannot interrupt a filesystem operation. A blocked filesystem can keep the download in progress and retain its worker slot beyond that deadline.

Unknown or misspelled keys produce a warning and are ignored; an omitted setting keeps its default. Set top-level deny_unknown_fields = true to reject unknown keys at startup. Refer to the NICo API configuration reference for this shared loading policy.

The following example explicitly sets all download defaults:

[firmware_global.downloads]
max_concurrent_downloads = 4
max_download_bytes = 4294967296
connect_timeout_seconds = 30
read_timeout_seconds = 60
request_timeout_seconds = 1800

Use site TOML for these settings. Site configuration overrides base configuration; CARBIDE_API_ environment configuration takes precedence over both. For Helm deployments, add the section to the existing nico-api.siteConfig.nicoApiSiteConfig TOML with nico-api.siteConfig.enabled enabled. Keep the rest of the TOML: replacing this Helm string replaces its entire contents. The chart does not set download limits, so Core’s defaults apply when omitted. Changing this site configuration through Helm rolls the Core pods, applying the restart-required settings.

Active HTTP downloads can hold up to 16 GiB of partial files per Core process at the defaults, or 256 GiB at the hard maxima. Retained cache and file:// URL copies require additional storage. Choose limits for your artifact sizes, source performance, and available cache storage; these are not measured site capacity guarantees.

When all workers are busy, Core defers the artifact check to the next poll. This applies even when the artifact is already cached or a recorded download failure is waiting to be reported. Core does not queue the check, and it does not count saturation as a firmware failure. The workflow remains waiting for artifact availability; the debug message “Firmware download workers are busy; deferring artifact check” identifies saturation. A slot remains occupied until verification and cleanup finish, even after the requesting controller iteration stops waiting. HTTP deadlines do not bound local copying, checksum verification, or cleanup.

An HTTP timeout, size-limit violation, or transfer error prevents publication of the partial file. Managed-host Redfish upgrades enter FailedFirmwareUpgrade and follow the firmware execution retry policy, including firmware_global.host_firmware_upgrade_retry_interval and the applicable retry budget. Pre-ingestion logs a warning and retries on later passes. Retries download from the start rather than resume. The carbide_firmware_download_duration_seconds_count metric records finished attempts by outcome; HTTP deadlines appear as fetch or transfer failures.

For these downloads, Core verifies a configured SHA-256 digest but does not require one. file:// URL artifacts share worker capacity but do not use the HTTP size or timeout limits.

Download limits are separate from firmware upload limits, BFB copy limits, and firmware execution concurrency. They do not limit retained cache storage.

Roll out a new baseline

Treat a baseline change as a site-wide rollout, even when the first update is a canary:

1. Update the canonical configuration source. Do not maintain a second live version table in this guide or a site runbook.
2. Make every referenced host artifact available. For DPU firmware, update the BFB, accepted NIC versions, and dpu_models together.
3. Check host component ordering, defaults, pre-ingestion thresholds, and inventory matchers. Then inspect the merged host catalog with firmware show.
4. Configure a nonzero shared capacity, explicit-start windows, and assigned-host approval policy. Keep site-wide automatic selection disabled while validating one host through a per-machine host policy or an explicit DPU reprovisioning request.
5. After verification, enable the intended site-wide automatic policy.
6. Monitor the applicable workflow until inventory reports the configured versions and the host has returned to service.

Changing a pre-ingestion threshold while firmware_global.autoupdate is enabled can affect every matching endpoint on the next pre-ingestion pass. Validate the catalog and artifact availability before applying that change.

Rack and component firmware

Rack and component firmware do not use either of the configuration models above. Their target version comes from the update request or from the default selected by the rack or component backend. Refer to Rack and Tray Firmware Updates.

Rack profile firmware object

A rack profile can specify one firmware-object JSON document to use as the default input for automatic rack firmware and switch NVOS updates:

[rack_profiles.NVL72]
product_family = "gb200"
[rack_profiles.NVL72.firmware_object]
url = "https://firmware.example.com/objects/nvl72.json"
fetch_timeout = "30s"
access_token_credential = "nvl72-artifacts"
[rack_profiles.NVL72.rack_capabilities.compute]
vendor = "NVIDIA"
count = 18
[rack_profiles.NVL72.rack_capabilities.switch]
vendor = "NVIDIA"
count = 9
[rack_profiles.NVL72.rack_capabilities.power_shelf]
vendor = "LiteOn"
count = 8

When firmware_object is configured for a profile with switches, the SOT JSON must contain an NVOS image whose firmware type matches rack_hardware_class. NICo requests prod when rack_hardware_class is omitted. RMS records an asynchronous update failure when the document does not contain the required image. After all NVOS image jobs complete or fail, NICo uses RMS to verify or restore the desired NVOS admin password on each selected switch before the rack leaves the NVOS update phase. Without firmware_object, NICo skips both automatic update phases. An explicit maintenance request can supply a firmware object instead. If no firmware object is available while a selected switch is in WaitingForNVOSUpgrade for a reprovision request whose initiator is rack-{rack_id}, the rack transitions to Error instead of skipping the NVOS phase.

The url field identifies the document location. The optional fetch_timeout field accepts duration strings such as 30s and 60s and defaults to 30s. Use seconds for this request timeout, although the parser accepts other duration units such as milliseconds (ms), minutes (m), and hours (h). The optional access_token_credential field names a credential that contains a firmware artifact access token. Store the token by running nico-admin-cli credential firmware-access-token set. NICo reads the credential when compute-tray pre-ingestion starts and forwards the token to RMS. The rack profile contains no secret. When the field is omitted, NICo sends the RMS no-auth sentinel. The rack state-machine firmware path keeps its existing token behavior.