Is this page helpful?

Improvements#

This section describes the improvements in each release.

Release 2.0.0 GA#

This release includes the following features:
- NVLink Down reset improvements
- Enhanced Fabric state reporting for GPUs with misprogrammed CBC EEPROM
- Enhanced GFM API error reporting
- Single Switch Tray recovery with adaptive bandwidth
  
  These features require the following minimum versions:
- GPU driver 580 TRD6 or later
- 2.0 recommended SW/FW stack
Refer to the NVOS Release Notes in the NVOS NVOnline posting for additional information about the improvements in this release.
This release includes security updates. Refer to the Partner Security Bulletin: NVIDIA Blackwell Data Center Products - June 2026.
This driver release resolves two conditions where the GPU driver could report XID 109 spuriously:

On B300 and later GPUs, a driver issue could occur when multiple processes were using the GPU while ECC counters were being monitored through NVML or nvidia-smi. In rare instances, one of the processes could stop making progress, causing the driver to report XID 109 and terminate the affected process.

On Hopper and later GPUs, when a process encounters a GPU error, such as a device-side assert(), the process is terminated as expected. However, if GPU core dumps are enabled, CUDA_WAITS_ON_EXCEPTION is set, or a debugger is attached, process teardown could time out and cause the driver to report XID 109. Other processes resume after the teardown notification.
NCCL Hangs with Coherent GPU Memory Mode

Resolved an issue where enabling NVreg_CoherentGPUMemoryMode=driver on GB200 caused NCCL applications to hang. A CDMM mode detection bug caused the driver to incorrectly detect the mode as NUMA, which affected dma-buf mapping because this mode is handled differently. The driver now detects the mode correctly. (Fixed in R580 TRD7.)
Traffic after compute tray swaps.

Resolved an issue where traffic could fail after compute tray swaps across locations within a partition. This fix supports the documented customer workflow for compute tray swaps within a partition.
NVLink retraining during GPU probes.

Resolved an issue where NVLink retraining could conflict with GPU probe request and response handling within a partition. This conflict could disrupt workloads when retraining and GPU probe operations occurred at the same time.
Multicast workloads fail after a factory reset.

Resolved an issue where multicast workloads could fail after a factory reset because GPU handles were not recognized correctly.
NVLink range logging.

Corrected Fabric Manager log output so NVLink link ranges are reported accurately for easier diagnosis.
SError After CBB Timeout

Resolved an issue where handling a CBB timeout to PCIe configuration space could incorrectly trigger an Arm processor SError and kernel panic.
ATF Workaround for Arm Neoverse V2 Erratum 2743011

Resolved a logic error in the ATF workaround for Arm Neoverse V2 erratum 2743011. The error could prevent the workaround from being applied even when the boot message indicated that it was active.
SKU Authentication Bypass in PLDM firmware update.

Resolved a security vulnerability in the PLDM firmware update state machine where a command sequence could allow an adversary to flash firmware intended for a different hardware SKU without passing authentication checks. The fix updates the pending trigger table when firmware data is written to flash, preventing the authentication bypass.
Backup-Partition Wear-Out from PLDM Cancel Flow.

Resolved an issue where repeated PLDM UpdateComponent -> write -> CancelUpdateComponent cycles could leave partial flash data and consume erase cycles on the same backup-partition sectors. The fix adds a rate-limit check for UpdateComponent.
ERoT hangs from VDM Multipacket buffer overflow.

Resolved an issue where oversized VDM middle fragments could overflow the composer buffer, causing the overflow path to return true with the output packet pointer left as nullptr. Because the VDM driver path lacked the null pointer check used by the PLDM and SPDM paths, this could result in a null pointer dereference and an approximately 4 GB memcpy, causing the ERoT to hang.
Potential code execution in ERoT DOT provisioning path.

The fix applies the correctly bounded logic from auth_metadata_buffer() to the parallel update_metadata_sig() DOT path.
InfoROM Read-Only Strap Policy Bypass in PLDM.

Resolved an issue where pldmfw_cmd_pass_component_table() recorded a component ID in the pass_comp_id() table before evaluating the strap gate. The fix updates pass_comp_id() only after the component ID check completes.
ERoT denial of service in Non-MCTP PLDM adapter.

Resolved an issue in the non-MCTP PLDM adapter. The fix eliminates the vector that could disrupt root-of-trust attestation and firmware update services.
Fixed an issue with NVBMC GPU*_PWR0 sensors reporting ‘na’ when the GPU power is > 1300 Watts.
SPDM CPU attestation fails when MCTP response exceeds one packet. Resolved an issue where SPDM CPU attestation could fail with a certificate-length mismatch when an MCTP response was one byte longer than a single-packet message. In this case, ERoT marked the first packet of the multi-packet response with EOM set to true. Multi-packet response length handling now correctly accounts for this case.
Glacier ERoT firmware detects SPI flash devices without hardware write protection.

Added support in Glacier ERoT firmware to detect and report Serial Peripheral Interface (SPI) flash devices that do not support hardware write protection through the SRWD/SRP bit.
FirmwareState and BackgroundCopyStatus consistency in DOT volatile mode.

Resolved an issue in DOT Volatile Mode where, after a background copy completed, BackgroundCopyStatus could report Completed while the active slot FirmwareState still reported WriteInProgress and the inactive slot incorrectly reported Activated. NSM now prioritizes the AP boot-complete slot when determining the active slot, keeping active and inactive slot reporting aligned.
Added support for CEC1736 N05 CPN.

Added support for the CEC1736 N05 CPN, including a dedicated S8 key set for the N05 part.
FPGA-ERoT I2C driver strength reduced to prevent SDA undershoot.

Resolved an issue on HGX-B200 systems where the FPGA-ERoT I2C target driver strength could cause the SDA line at the customer BMC chip to undershoot below the -0.5 V GND limit during read operations. No functional failures were observed in NVIDIA lab measurements, but customer testing reproduced the signal-integrity violation. ERoT firmware now supports a new VDM command to reduce the I2C driver output strength for the FPGA-ERoT at address 0x52.
Improved ERoT firmware upgrade logs for revoked security keys.

Improved ERoT firmware upgrade logging when an upgrade fails with Security keys revoked. The logs now provide clearer information for key revocation and rollback-protection cases.
EC firmware debug key revocation uses updated NSM cache.

Resolved an issue where EC firmware debug key revocation could fail because NSM used stale cached key index and permission values after boot. NSM could return 0x0 instead of the production-configuration values key_idx=1 and key_revocation_info=0x01, causing firmware to determine that no revocation was needed. NSM now populates the cache from flash before processing key-revocation commands.
NVLOOM workloads fail after removing an unallocated GPU from a partition.

Resolved an issue where NVLOOM workloads could fail after approximately 30 minutes when two 1-GPU VMs were running in a 3-GPU partition and the third, unallocated GPU was removed from the partition. The failure could trigger a kgspRcAndNotifyAllChannels_IMPL() crash and log an Xid 94 error in one VM. This issue is fixed in the 6.17.0-1015-nvidia-64k kernel and later.
GPU reset operations no longer require power cycle after NVSwitch tray reboot.

Resolved an issue where GPU reset operations could fail with kernel Xid errors, such as GSP timeout, when a subset of NVSwitch trays rebooted while an NVLink SHARP workload was running. In some cases, recovery required a rack-level AC or DC power cycle.
NVSwitch ERoT forced recovery works on Juliet switch board.

Resolved an issue where forcing a functional NVSwitch ERoT into recovery mode and then force updating it did not work on the Juliet switch board. The forced recovery and update flow now supports this case. Standard ERoT functionality, failed-ERoT recovery, and normal firmware update mechanisms were not affected.
RDMA NCCL failures resolved after RDM-GFM communication loss.

Resolved an issue where, in rare cases, communication loss between RDM and GFM could cause a memory leak in NVLSM and lead to RDMI NCCL failures. NVLSM now handles this communication-loss scenario without requiring nmx-c processes to be restarted.
Micron NVMe drives remain available behind CX8 with PCIe max payload configuration.

Resolved an issue where Micron M.2 and E1.S NVMe drives could become unavailable on PCIe when the system used the NVIDIA reference SBIOS and all installed drives were Micron. The SBIOS now programs the PCIe root ports with the proper PCIe Max Payload setting. GB200 systems with CX7 were not affected.
OS image PCIe errors and system reboots are resolved.

Resolved an issue where a PCIe surprise link down between the Grace root port and NIC could make the NIC unreachable, cause cascading service failures in Kubernetes, Cilium, Node Cache, and Node Exporter which leads to a kernel panic after function-level reset (FLR) and PCIe retrain recovery attempts failed. This issue is fixed in DOCA 3.4.0. Also resolved an issue where, during kdump kernel invocation after the kernel panic, the Grace SMMU could enter a global abort state, terminate in-flight transactions, emit Poison TLPs, and cause Completion Timeouts. This issue is fixed in the 7.0.0-1015-nvidia-64k kernel and later.
HMC memory no longer fills during update startup.

Resolved an issue where repeated calls to the temp sensor shell script could leak small amounts of memory and eventually deplete HMC memory when starting an update. This could trigger the out-of-memory (OOM) killer and cause the HMC to crash or behave unexpectedly. The memory leak in the temp sensor shell script is fixed.behave unexpectedly. The memory leak in the temp sensor shell script is fixed.
Deepseek 1N job causing Xid 48 and fatal node crash.

Deepseek workload fails causing an OS crash with RM driver 580.159.01. This is not an issue with RM driver 580.126.20 TRD7 or RM driver 580.126.16 TRD6.

Workaround: Perform a GPU reset or driver unload/reload in between workloads.
DPC limit from BMC incorrectly changed GPU power under power brake.

Applying a DPC limit from the BMC could alter GPU power consumption while the power brake was engaged. Limits are now deferred so they take effect only after the power brake disengages, as intended.
One or more GPU IRoT(s) not found.

On GPU I3C tasks, the implementation had an issue where unknown tasks would lead to failure. This issue is resolved and implemented in the I3C recovery flow.
Unsupported Target GPU T.Limit sensor disabled.

The Target GPU T.Limit sensor is not supported but could still report an invalid threshold of 0xFF. Recent firmware disables this sensor so the invalid value is no longer returned.
Clear Max Observed Power NSM API ignored aggregate clear (0xFF).

The Clear Max Observed Power NSM API (message type 3, command code 5) did not accept the aggregate argument 0xFF to clear all max power readouts. Added support for that will clear the aggregate argument driver branch 595.28 and newer.
XID 150/154 immediately after GPU reset (MSE communication initialization).

During reboot stress testing (22+ hours), XID 150 and 154 could rarely appear immediately after a GPU reset. A race condition in the MSE communications path could leave a message request to MSE in an “empty but incomplete” state. Without recovery for that state, the communications flow did not reinitialize properly and MSE became unable to process new requests. Recovery handling for this state has been added.
MCU firmware update can fail on a CX8 port when MCU with TRNG issue fails to boot.

On CX8, the MCU did not always update to the correct firmware on one port. When the MCU had a TRNG-related issue, it could fail to boot, which caused the firmware update to fail. The update path was made more resilient by adding multiple retries so the MCU can complete the update when boot is delayed or initially unsuccessful.
Enhance Logging for C2C training and memory training failure.

This release includes logging enhancements to allow more debug information in ROT log.
Redfish telemetry on HMC interrupted by GPU OOB NSM timeouts.

Resolved an intermittent timeout between the HMC and GPU OOB firmware that disrupted Redfish telemetry when NSM replies were delayed. The underlying timing issue is fixed in VBIOS firmware, restoring stable telemetry collection.
CPU cores may get stuck in firmware leading to a CPU softlock under the operating system.

Resolved issues in ATF, RAS, and SatMC firmware that could lead to a CPU core getting stuck in the firmware, causing a CPU softlock under the operating system.
Hard lockup due to uncorrectable error consumed by GSP.

Fixed an issue in RAS_FW that caused a CPU core to be stuck in an infinite loop.
Null-valued GPU sensors.

Under some conditions, internal processing of requests can take more time than expected. During this time, another request may be received from the BMC. In this scenario, OOBHUB will respond to the wrong request if a response for the first request is sent. Implemented a mechanism to drop the request if it exceeds the timeout threshold.
Intermittent GPU boot failure.

Fixed a GPU boot failure that occurred when FLR was issued while NVLink was training.
“Route Unhealthy” state remains True after reroute().

Repeated NVLink in-band messages could exhaust credits in the driver because of a credit-handling bug. In that situation, the route could stay in “Route Unhealthy” even after a reroute. The driver’s credit handling has been corrected so that credits are not exhausted in this scenario and the route returns to a healthy state after reroute.
NMX-C unhealthy, GFM gRPC times out.

Resolved an issue where multiple threads in the GFM had a circular dependency on locks, causing a deadlock. GFM could not make progress in that state, which led to NMX-C appearing unhealthy and GFM gRPC calls timing out. The deadlock has been removed so GFM can run normally and gRPC requests complete.
Node swaps no longer require fabric restart (GUID-based partitions).

For default GUID-based partitions, tray replacement now correctly removes the old tray’s GUIDs and adds the new tray’s GUIDs, so a fabric restart is no longer required after a node swap. For user-defined GUID-based partitions, the partition create and add-GPU APIs now enforce the NVLink domain GPU limit so partition size cannot exceed the domain limit.
GPU containment with no related data, XIDs, C2C, or NVLink errors.

Improved the logging of the GPU Containment CPER to include additional register dumps that are useful when analyzing failures.
C2C bandwidth out of range, unexpected interrupt and buffer mismatch.

Fixed an issue in which UEFI-reserved memory may be included in the OS’s available memory map when EGM is enabled.
CPU core soft lockup.

Fixed multiple issues in SBIOS firmware that could cause a core to get stuck in the firmware for extended periods of time, leading to the OS reporting soft lockups.
NVRASTool v1.2.1 OOB full error injection – 7 fail, 5 pass when run individually.

HMC now acknowledges GPIO alerts from both primary and secondary FPGA. Resolved a synchronization mismatch between the FPGA and HMC firmware: the FPGA drove separate GPIO alerts for primary and secondary controllers, while the HMC only monitored an aggregated signal from the primary FPGA. Alerts from the secondary FPGA were therefore not acknowledged by the HMC. HMC firmware has been updated to monitor GPIO alerts from both the primary and secondary FPGA so all controller alerts are acknowledged.
Failed to get HGX_CPU_ sensors.

Resolved this issue by creating associations to CPU inventory path in entity-manager without relying on SmbiosMDR service.
Redfish reporting “disabled” status for multiple CPU ports.

Fixed an issue in SatMC that causes PCIe ports to incorrectly report “disabled” in Redfish reporting.
GPU-PGOOD is missing in the metric report.

Added the GPU’s status.state property for GPU-PGOOD, which was previously missing in the GPU ProcessorMetrics report.
GPU-Status CPER logged at runtime with Redfish fatal severity.

Changed the GPU-Status CPER severity from recoverable to informational to ensure that this informational event is handled correctly by some operating systems.
MemorySpareChannelPresence reported as false when information is unavailable.

When MemorySpareChannel information could not be read (for example when SatMC is down), the HMC set the Presence (state) to false by default, which incorrectly suggested that no spare channel was present. The default is now unknown in the backend and null in Redfish when the information is unavailable, so the state correctly reflects that the value is unknown rather than false.
No P2P after reboot following Xid119 on FW1.2.1 (tray swap/replacement).

When a tray was replaced or swapped, the internally used systemImageGuid stayed with the physical location instead of moving with the tray. That led to duplicate nodeIds, corrupted remaps, and failed P2P or workload failures (including no P2P after reboot following XID 119 on FW 1.2.1). Fabric Manager now uses a software-based approach called “gpu set” to identify GPUs that belong to a single tray. This correctly handles trays that have been moved to a different location and new GPU trays added in a previous location, eliminating the duplicate nodeId and remap corruption issues and restoring P2P after tray swap or replacement.
Node swaps no longer require fabric restart.

Resolved an issue in how GPUs on the same tray are identified. In one code path, GPUs could use an uninitialized nodeId, which incorrectly grouped them with uninitialized GPUs from other trays and led to REMAP and P2P failures, often requiring a fabric restart after node swaps. Tray identification is now correct in that path, so REMAP and P2P no longer fail for this reason and node swaps do not require a fabric restart.
RemoveGpusFromPartition no longer fails when partition is already unhealthy.

Previously, if an NVLink partition’s routing was unhealthy, the NMX Controller failed all partition-modification API calls (add or remove GPU), so no changes could be made until the partition was healthy again. The controller now fails and reverts only when the API call itself would make the partition unhealthy. RemoveGpusFromPartition (and add-GPU calls) can therefore succeed when the partition is already unhealthy, allowing administrators to modify the partition in that state.
RFE support for reduced bandwidth when a switch tray is offline.

When a switch tray goes offline (due to a fault or admin action), the fabric can now continue operating using the remaining switches at reduced bandwidth. In earlier releases, the NVLink domain became non-operational in this case. This feature currently supports a single switch tray being absent, provided that tray is not the primary switch tray running the NMX Controller.
RFE: GPU NVLink traffic when moving GPUs across partitions without GPU reset.

This enhancement to Multi-Node NVLink partitioning allows administrators to move GPUs between NVLink partitions for multi-tenancy without requiring a GPU reset or GPU resident service restart to enable peer-to-peer (P2P) traffic over NVLink.

The behavior is controlled by the new Fabric Manager option MNNVL_NVLINK_PARTITION_MODE, which supports two modes:
- Multi-tenancy NVLink Partition Mode (Default value = 1): A GPU reset is required when a GPU is moved between partitions. In this mode, nvidia-smi -q shows GPU Recover.
  
  Action: Drain and Reset until the reset is performed.
- Trusted NVLink Partition Mode (Value = 2): GPUs can be moved between partitions and use P2P traffic immediately, without a reset or service restart. In this mode, nvidia-smi -q shows GPU Recover.
  
  Action: None during the move.
Changes to this setting take effect after an NMX-C restart.
CreatePartition with duplicate GPUs did not return PARTITION_EXISTS.

Fixed an issue where Fabric Manager did not return PARTITION_EXISTS when creating a partition that would reuse GPUs from an existing partition. The cause was default resiliency mode not being set on creation, so the existence check compared unset vs set values and failed to match. Default resiliency mode is now set correctly on partition creation, so the correct error is returned.
Reroute returns generic GRPC error (NMX_ST_GENERIC_ERROR).

Fixed an issue where the partition reroute API returned NMX_ST_GENERIC_ERROR for most failures. New error codes have been added so that specific failure scenarios return the appropriate code, improving diagnostics and error handling.
Fabric Manager uplink calculation for partially populated topologies.

Resolved an issue where Fabric Manager (FM) could miscalculate required uplinks in partially populated topologies, so probe could fail even when enough links were present. The uplink calculation now correctly accounts for available links in these topologies, allowing probe to succeed when the required number of uplinks is satisfied.
GFM underflow when allocating multicast resources to a partition.

Fixed an issue where GFM could miscalculate remaining multicast teams when allocating MC resources to a partition due to an unsigned-integer underflow in the availability check; remaining resources are now computed correctly.
Multiple control plane status description string.

The control plane status was incorrectly set to CONFIG_ERROR_ADDITIONAL_CHASSIS_DETECTED when only one chassis was discovered, even though the topology file indicated two chassis. The control plane status description is now set according to the actual discovered chassis count.
GFM log flooding due to constant topology updates.

Admin state logging is optimized so that admin states are logged only when they change. This prevents GFM log flooding caused by constant topology updates.
NV-SW Fabric Manager not reflecting correct GPU health/state.

GPU health states were incorrectly reported as healthy through the NMX-C API when the compute node was powered off or rebooted. The fix checks for the presence of GPU nodes before computing their health, so the Fabric Manager now reflects the correct state.
RFE - Enhance GFM APIs to GPU/Switch/Node NVLink states and transaction IDs.

NMX-C APIs now return structured, detailed error information for failed fabric and partition operations instead of only a generic error code. NMX-C APIs can now be used as the main interface for fabric troubleshooting and no longer need to manually correlate cryptic GFM logs.
Restricting PMU reset controls.

Allowing external PMU reset at elevated priority could, in some cases, expose a timeout bug. This change disallows external reset requests at elevated priority to eliminate that risk.
Adding a GPU to a location-based partition failed.

A bug in SM caused adding a GPU to a location-based partition to fail. A bug in GFM cleanup logic prevented subsequent add operations from succeeding. This fix ensures GFM cleans up correctly after failures so that later add attempts succeed. The SM bug is fixed in SM 1.3.1.
RemoveGpusFromPartition failed with NMX_ST_GENERIC_ERROR.

Removed the constraint that prevented removing GPUs with active multicast teams from a partition. Those GPUs can now be removed and are marked as “Needs Reset.”
Partition remained unhealthy after NMX-C restart trunk link routing was lost.

Previously, when a partition became unhealthy because of a trunk link routing issue, restarting NMX-C or rebooting the switch that hosts NMX-C did not trigger a reroute. Users had to manually start the reroute to restore the partition. With this fix, restarting NMX-C or rebooting the hosting switch automatically triggers a reroute of trunk links and restores the partition to a healthy state when no trunk link issues persist.
GPU_THERM_WARN_INT-ERROR during CSST at safe temperatures.

Adjusted temperature-sensor warning thresholds so THERM_WARN_INT is not reported when the GPU is within safe operating limits.
Support for MSE LFM to notify GFM alive state.

In multi-node systems, GPUs previously timed out on NVLink detection if compute trays started before GFM/NMX-C, requiring reboots or GPU resets when NMX-C came online causing downtime.

MSE 13.1 introduces a heartbeat-based notification in LFM to detect GFM connectivity and immediately alert RM to trigger fabric probing, replacing timeout-based retries. This enables GPUs to establish NVLink connectivity with late-starting NMX-C without resets or reboots, eliminating operational disruption.
The system cannot boot up.

Fixed an issue that may prevent boot when power the brake is asserted.
[5394666] FWERROR reported in PEI.

Fixed an issue that would cause all FWERROR CPERs to report PEI mode.
RoT HGX_ERoT_BMC_0 (HMC) certificate extraction issues.

Resolved this issue by adding retry for certificate request when truncated response is received in spdm connection handler.
HMC reported invalid data (SlotIndex and SlotLocation).

Resolved this issue by adding retry to get UUID command and enhanced MCTP rediscovery.
HMC BMCweb hardcoded check for update header in MultipartHeader.

Changed the content-disposition field of multipart firmware update to non case sensitive. Prior to this change, the field “Content-Disposition” required exact casing to get recognized.
Host based copies regression with 1.1.

The nvbandwidth tool now supports testing with Huge Pages for host memory. Users can enable this feature by adding the –useHugePages flag to the command line. This enhancement will be available starting with the nvbandwidth v0.9 release.
Graceful handling of empty sensor list.

When the host is off, CPUs are expected to have no sensors or metrics. Previously, this returned an internal error instead of an empty list. Added a fix to remove the error response and now it returns an empty list when no sensors are found.
Telemetry missing for the power brake.

Added additional throttle reason updates for hardware slowdown, thermal slowdown, power brake slowdown, and sync boost in the NsmProcessorThrottleReason::updateReading function. This improves the processor’s performance interface by providing more detailed throttle reasons.
HIP-1 down training to 1x1 after the DC cycle with the BM BIOS flavor.

Introduced a SatMC PLDM effector to allow out-of-band software to trigger a warm reset of the CPU.
Reporting NVLink BW performance variations across all nodes.

The cudaLaunchAttributeNVLinkUtilCentricScheduling launch attribute, accessible via cudaLaunchKernelEx() to indicate that a kernel should prioritize maximizing NVLink bandwidth utilization. When applied to communication-intensive kernels, such as those used by NCCL, CUDA automatically adjusts scheduling to better saturate NVLink bandwidth across modern GPU architectures. This enhancement helps mitigate performance variability in collective communication workloads observed on newer hardware platforms.
System occasionally does not report BERT error after injecting a CSF UE.

Added support for reading records by index and consuming multiple RAS log records. This enables more reliable error reporting by capturing multiple records that may have been generated between collections.
MSE implementation to support node description setting in GPU.

Previously, the NMX-T Prometheus interface returned hardcoded values (“GB100 Nvidia Technologies”) for node_description and host_name, preventing the ability to identify individual GB100 nodes and correlate metrics in OCI monitoring. MSE 14.14 now supports GET/SET_HOST_NAME messages to set hostnames in NetIR. Monitoring systems now display correct hostnames.

BMC and HMC Improvements#

This release also includes the following BMC and HMC improvements:

`bios-settings-mgr`#

Corrected ScureCurrentBoot to SecureCurrentBoot in Redfish and the BIOS manager.

`openbmc`#

(HMC) Removed manufacturing script-exec and parse-yaml recipes.
(HMC) Updated mbedtls to 3.6 to fix CVE vulnerabilities in SPDM.
(HMC) Fixed HMC Redfish Status.State incorrectly reporting Starting after boot.
(BMC) Flushed journal logs to eMMC on aux power cycle and after firmware update to prevent log loss.
(HMC) Fixed memory leak in hmc-temp-sensors daemon.
Disabled ssh pubkey auth to prevent ssh login after debug token removal (CVE-2025-23337).
Removed slp-lite service (CVE-2024-41660).
Added maximum recovery attempts limit for BMC/HMC filesystem recovery.
Improved FPGA1 FW recovery process.
If host power is off during PCIe switch update, ignored update failure and returned success to make it consistent with GPU FW update behavior.
(DGXOPENBMC) Reapplied built-in user settings and group permissions from ROFS after firmware update.
Fixed issues with PCIe switch on/off power flow.
Updated HMC to Kernel 6.12.41.
Added new metrics for GPU, CPU, HMC, and FPGA.
Differentiated BMC reset types: Enabled kernel watchdog pretimeout panic to distinguish between normal reset, kernel panic, and watchdog timeout.
Fixed Redfish event SSE filtering when message ID is missing.
Fixed randomly generated password for bootstrap account not always complying with password complexity requirements.
Selected correct Metric Report Definition Metrics depending on CX7 or CX8 configuration.
Added missing virtual I2C buses to FRU device service blocklist.
Added support for GB200 NVL4.
Fixed missing support for alerts from the second FPGA.
Fixed CX7/CX8/BF3 FW RelatedItems in Redfish.
Fixed redundant CPU id issue in CPU Metric Report Definition.
Removed service account from BMC and disabled -S option for SCP.
Removed service account from BMC.
Disabled SCP -S option.
Added support for GB200 NVL4.
Added support for additional CX8 variants.
Enabled kernel watchdog timeout detection SEL logging.
Fixed issues with firewall rules preventing access to SSH ports 2201-2203 to connect to the Grace CPU UARTs.
Enabled IPMI command filtering based on host state. Added allow list for CPU/SBIOS with ProvisionedHostAllowlist configuration.
Updated 1RU exhaust and pdb inlet thermal setpoints based on latest thermal recommendations.
Added support for additional GB200 variant.
Fixed issues with missing ipmi oem command support with GB200/GB300 DGX builds.
Fixed CPU max power setting in IPMI/Redfish causing incorrect sensor value reporting.
Updated BF3 temperature sensor reporting to use PLDM and instead of i2c which fixed occasional crash of hwmontempsensor service.
Fixed issues with PDB FRU being occasionally lost during host reboot.
Fixed issues with psusensor service slowly leaking thread descriptors
Enabled ERoT log collection via I2C fallback if PLDM methods fail.
Fixed issue with built-in admin account becoming unusable if deletion of the account is attempted. Now throw 403 errors on deletion attempts.
Disabled deprecated faultlog in bmcweb since this was moved to EventLog.
Fixed issue with LastResetTime not always being synced with chassis power state.
Fixed issue with missing shutdown options displayed in the WebUI.
SSIF interface reliability enhancements.
Enabled OOB clearing UEFI variables without reboot.
Enhanced WebUI SOL to include host power state indicator.
Enhanced WebUI to filter non-present E1.S drives as status absent by default instead of status critical.
Enhanced error handling, logging, and reporting when the emmc log partition is corrupted or unavailable.
Modified CPLD FW update to not report error if host main power is off.
Fixed kernel watchdog timer reset reporting and logging.
Fixed issue with OpenOCD service not always auto-disabling when JTAG interface is removed.
Enabled 10-second fan controller watchdog timer to ramp fans to 100% if BMC fails to pet the watchdog.
Fixed kernel crash issue with attempted access to eeprom devices when in incorrect host power state.
Fixed kernel APSEED driver issues with unhandled i2c IRQs.
Fixed issue with CPER event logging.
Fixed Redfish BMC and HMC product name reporting.
Fixed missing CPU FW RelatedItem in Redfish.
Fixed Redfish FW Inventory HGX_FW_CPLD RelatedItem pointing to incorrect Chassis item.
Fixed Redfish related items for firmware inventory entries.
Patched BMC kernel to include SSIF workaround for data mismatch handling to prevent i2c-tegra driver crashes in host OS.
Fixed various missing RelatedItems in Redfish FW Inventory.
Fixed WebUI FW Update pages to allow FW update in any host power state with clearer warnings about what may not be supported in different power states.
Fixed BMC ERoT FW version occasionally missing after DC power cycle.
Added support for additional factory diags tests.
Added support for NVL4.
Fixed issue with BMC ERoT serial number missing after HMC reboot.
Fixed issue with missing CPU sensors in certain corner cases.
Fixed occasional phosphor-create-chassis-poweron-log service failure during soft reboots.
Fixed issue with user creation through Redfish when password complexity requirements were not met.
Added EffectiveError to ProcessorPortMetrics MRD.
Fixed issue with ProcessorPortMetrics containing removed properties.
Fixed missing HGX_MetricProperties for GPU.
Added support for additional GB300 variant.
Fixed issue with ERoT SPI access post FPGA0 recovery if AC power cycle was not performed post recovery.
Fixed issue with missing GPU PN and SN reporting.
Fixed intermittent issue with pldmd and fansensor coredumps.
Added logic to persist host side MAC address of BMC-to-host USB Ethernet adapter link.
Added additional CX8 Power Telemetry sensors.
Fixed issue with missing GPU thermal sensor readings.
Fixed issue with certain NVMe drive sensors being lost after DC cycle.
Updated thermal setpoints for ConnectX ports.
Fixed intermittent issue with false 0 RPM critical FAN readings events.
Fixed intermittent issue with invalid fan critical threshold events being reported after cycling AUX power.
Fixed missing Redfish CX8 power telemetry metrics.
Updated thermal recommendations for PDB sensors.
Fixed intermittent issue with failing to log BMC reset reason after BMC reboot.
Report NVLink LED state in Redfish.
Fixed intermittent issue with chassis-poweron-log service failing to start after BMC reboot.
Enabled OOB boot variable clear (that is, CMOS clear).
Aspeed RTC only re-initializes on cold boot so warm reboot preserves time and HMC stays synchronized with the BMC.
Enabled network adapter device protection across D-Bus, NSM, and Redfish.
Added configured warning/critical/shutdown temperature limits for Quanta PDB HSC sensors.
Included FPGA I3C APB register dump output in HMC FPGA diagnostics.
Added GPU PCIe port inventory in entity-manager so Redfish processor GPU ports expose PCIe_0 as well as NVLink.

`dbus-sensors`#

Fixed intermittent issue with fansensor service coredumping during host power cycle.
Fixed intermittent issue with nvmesensor service coredumping after bmc reboot.
Fixed intermittent issue with nvmesensor falsely reporting -1 temperature.
(HMC) Fixed intermittent issue with lpcsnoop and nvidia-fpga1-ready services failing to start.

`entity-manager`#

Fixed intermittent issue with duplicate path handling leading to coredumps.

`libmctp`#

(BMC and HMC) Fixed OOB heap read in libmctp Get UUID response handlers that could leak adjacent heap memory via Redfish.
(BMC and HMC) Fixed OOB heap reads in multiple MCTP control response handlers.
(DGXOPENBMC) Enabled systemd journal APIs for MCTP logging.
Fixed issue with duplicate MCTP control packet handling causing CX8 discovery to fail.
Fixed occasional issue with CX7/CX8 Temp reporting as 4000/8000C due to rare error handling issue with MCTP over I2C that could lead to duplicate messages being transmitted over the bus. This could occasionally lead to PLDM message response confusion.
Capped consecutive SPI AP receive timeouts so mctp-demux can return to its main loop and satisfy systemd watchdog.
Removed Redfish logging for restart notify failure for the ERoT in certain spurious cases.

`linux`#

(HMC) Moved to Linux kernel 6.12.73.
(BMC) Hold BMC kernel at 6.12.9 revision due to kernel instability. Kernel will be updated after the root cause is found.
Aspeed RTC only re-initializes on cold boot so warm reboot preserves time and HMC stays synchronized with the BMC.

`nsmd`#

(BMC and HMC) Fixed buffer overflow vulnerabilities in libnsm response decoders.
(HMC) Fixed invalid (NaN) sensor readings reported as valid 0C data by setting the telemetry stale bit.
(HMC) Fixed GPU assembly FRU fields (part number, serial number) returning blank after a DC power cycle.
(HMC) Fixed PowerSmoothing AdminOverride mask issue where it was not fully cleared after ApplyAdminOverrides.
Various NSMD bugfixes and performance improvements
Enabled various missing telemetry items.
(HMC) Fixed missing power brake telemetry.
(HMC) Fixed issue with GPU utilization metrics.
(HMC) DGXOPENBMC Reduced High Latency in Redfish Workload Power Profile Management.
(HMC) Reduced High Latency in Redfish Workload Power Profile Management.

`nvidia-code-mgmt`#

(HMC) Fixed heap OOB read in firmware image signature verifier when parsing an undersized firmware component.
(HMC) Fixed heap out-of-bounds read in firmware image hash verification caused by a 32-bit integer overflow.
Fixed issue with fw-status crashing sometimes when changing power states.
(BMC) Fixed issue when using targeted FW update. PCIeSwitch would always be updated if contained in the bundle even if not part of the targeted update.
Fixed issue when using targeted FW update. PCIeSwitch would always be updated if contained in the bundle even if not part of the targeted update.
(HMC) Fixed issue with BMC firmware update in ERoT-less systems.

`bmcweb`#

(BMC) Fixed Redfish SSE subscriptions not starting the BMC/HMC event aggregation listener, causing an SSE event stream to receive no events until a separate EventDestination subscription was created.
(BMC) Fixed incorrect task-progress status check for the SPI erase operation.
(BMC and HMC) Fixed Redfish Service Validator conformance errors in OEM ActionInfo and schema resources.
(BMC) Added missing odata.type to SPDM signed measurements attestation response.
(BMC) Fixed Redfish log entry collections to sort numerically by ID, correcting broken log pagination.
(BMC and HMC) Fixed CPER event severity code handling in the SSE and event log paths.
Enabled IPMI command filtering based on host state. Added allow list for CPU/SBIOS with ProvisionedHostAllowlist configuration.
Fixed issue with Empty Part Number reporting in Redfish.
Fixed issue with incorrect handling of trailing “/” in Redfish URIs.
Fixed issue with secondary SPI missing in inventory.
Corrected ScureCurrentBoot to SecureCurrentBoot in Redfish and the BIOS manager.
Fixed missing power cycle option reporting in uxPowerResetActionInfo.
(BMC and HMC) Fixed issue with POST action timeout and Redfish aggregation.
(BMC and HMC) Improved memory usage during firmware update.
Fixed issue with occasional missing events with Redfish Event subscriptions.
Fixed issue that could cause phosphor-networkd to crash and be exploited to cause a DoS of BMC services.
Fixed issue with missing CPU sensors in certain corner cases.
Fixed issue with incorrect RF return codes when NVMe drive cannot be accessed.
Added new LsbIBGUID property.
Fixed issue with RF aggregation when using VariableSpiRead.
Fixed Redfish conformance issue with SSD temperature reporting when value is unavailable.
Fixed issue with CPU sensor reporting when the host is powered off.
Fixed memory leak in certain cases with Redfish aggregation.
(BMC and HMC) Fixed display of EDPp persistence field value.
(BMC and HMC) Fixed issue which contains links and HGX_ProcessorModule.
(HMC) Report NVLink LED state in Redfish.
Fixed Redfish location of LeakDetector policy ReactionDelaySeconds PATCH operations to comply with schema.
(HMC) Fixed missing Redfish OriginOfCondition field in certain event log entries.
(BMC) Fixed issue with Redfish server not closing inactive TCP sessions.
Enable network adapter device protection across D-Bus, NSM, and Redfish.
Fixed intermittent corrupted MetricReportDefinition property names under aggregation (URL fragment use-after-free when rebuilding URIs).

`nvidia-ipmi-oem`#

Fixed randomly generated password for bootstrap account not always complying with password complexity requirements.

`nvidia-power-manager`#

(DGXOPENBMC) If host power is off during PCIe switch update, ignored update failure and returned success to make it consistent with GPU FW update behavior.

`nv-shmem`#

Enabled various missing telemetry items.
Fixed missing PCIe gen6 type in pcie-type map.

`phosphor-bmc-code-mgmt`#

Added logic to persist host side MAC address of BMC-to-host USB Ethernet adapter link.

`phosphor-dbus-interfaces`#

Fixed issue that could cause phosphor-networkd to crash and be exploited to cause a DoS of BMC services.
Fixed issue with missing CPU sensors in certain corner cases.
(HMC) Report NVLink LED state in Redfish.
Enabled network adapter device protection across D-Bus, NSM, and Redfish.

`phosphor-host-ipmid`#

Enabled IPMI command filtering based on host state.
Fixed issue with BIOS reset not being reliably cleared.

`phosphor-logging`#

Fixed issue when handling corrupted data logging files.

`phosphor-time-manager`#

Changed ERoT timesync failure event log to informational rather than critical since it does not impact operation.

`phosphor-networkd`#

Fixed issue that could cause phosphor-networkd to crash and be exploited to cause a DoS of BMC services.

`phosphor-state-manager`#

(DGXOPENBMC) Validated cpu boot done signal.

`phosphor-user-manager`#

Fixed issue with built-in admin account becoming unusable if deletion of the account is attempted. Now throw 403 errors on deletion attempts.

`pldm`#

(DGXOPENBMC) Fixed issue with logging FW_UPDATE_BLOCKED messages in Redfish.
(HMC) Fixed issue with GPU FW update not being skipped without error when the host is powered off.

`spdm`#

(HMC) Updatef mbedtls to 3.6 to fix CVE vulnerabilities in SPDM.
(BMC and HMC) Fixrf OOB read in SPDM NEGOTIATE_ALGORITHMS request decoder by bound-checking algorithm counts.
(BMC and HMC) Fixed the issue where SPDM requests failed promptly with a clear error when a device was unreachable instead of hanging for minutes.
(BMC and HMC) Fixed OOB heap read in SPDM CERTIFICATE response decoder.
(BMC and HMC) Update mbedTLS crypto library to version 3.6 for the SPDM daemon.

Release 1.3.10#

Refer to the NVOS Release Notes in the NVOS NVOnline posting for additional information about the improvements in this release.
This release includes security updates. Refer to the Partner Security Bulletin: NVIDIA Blackwell Data Center Products - June 2026.
GPU recovery after failed AddGpusToPartition call.

Resolved an issue where a GPU could fail to rejoin the NVLink fabric after a failed AddGpusToPartition API call during an add or recovery flow.
Multicast team teardown during HA events.

Resolved an issue where multicast persistence handling could prematurely tear down multicast teams during high availability (HA) resiliency events.
OpenBMC HMC memory leak when starting an update.

Resolved an issue where the hardware temperature sensor service could leak memory on the OpenBMC HMC when an update was started.
SError after HSM_FABRIC fault.

Resolved an issue that could cause an SError to occur after an HSM_FABRIC error. The fault is now handled correctly without raising a spurious SError.
Disabled HBM SBE reporting by default.

HBM SBEs are now disabled by default to avoid duplicate reporting as they are already reported via XIDs. Otherwise, use the BCT knob to enable this reporting to continue double reporting of HBM SBEs.
Neoverse V2 Erratum 2618597.

Included Arm’s suggested errata workaround for 2618597.
Resolved an issue with SDEI error reporting.

Save NS context before existing RAS handling of SDEI events to avoid TLS pointer corruption.
XID 150/154 immediately after GPU reset (MSE communication initialization).

During reboot stress testing (22+ hours), XID 150 and 154 could rarely appear immediately after a GPU reset. A race condition in the MSE communications path could leave a message request to MSE in an “empty but incomplete” state. Without recovery for that state, the communications flow did not reinitialize properly and MSE became unable to process new requests. Recovery handling for this state has been added.
Some GPUs may fail to ramp to full TGP when power smoothing ramp-up is enabled, resulting in reduced compute performance.

Implemented a VBIOS fix to update the ramp-up logic to allow the ramp-up.
Enhance Logging for C2C training and memory training failure.

This release includes logging enhancements to allow more debug information in ROT log.
Intermittent GPU boot failure.

Fixed a GPU boot failure that occurred when FLR was issued while NVLink was training.
Micron NVMe drives remain available behind CX8 with PCIe max payload configuration.

Resolved an issue where Micron M.2 and E1.S NVMe drives could become unavailable on PCIe when the system used the NVIDIA reference SBIOS and all installed drives were Micron. The SBIOS now programs the PCIe root ports with the proper PCIe Max Payload setting. GB200 systems with CX7 were not affected.
NVSwitch CPU reboots with MGMT board VR failure.

Added an improvement to avoid an NVSwitch CPU reboot due to a BMC forced/unexpected reboot.
XID 145 due to incorrect routing configured by NVLSM (36x2 only).

Resolved an issue where NVLink Subnet Manager (NVLSM) configured incorrect routing in switches in one rack of 36x2 towards GPU ALID when GPU is belonging to another rack of 36x2.
Power smoothing minimum power floor increase.

This enhancement to power smoothing now allows the Minimum Power Floor (MPF) value to be configured to 95% (previously 90%).
Workarounds for Arm Neoverse-V2 erratum 3841324 and 3442699.

Implemented suggested workarounds for two Arm erratum that previously caused incorrect instructions from being executed.
Some GPUs may fail to ramp to full TGP when power smoothing ramp-up is enabled, resulting in reduced compute performance.

Implemented a VBIOS fix to update the ramp-up logic to allow the ramp-up.

Improvements#

Release 2.0.0 GA#

BMC and HMC Improvements#

bios-settings-mgr#

openbmc#

dbus-sensors#

entity-manager#

libmctp#

linux#

nsmd#

nvidia-code-mgmt#

bmcweb#

nvidia-ipmi-oem#

nvidia-power-manager#

nv-shmem#

phosphor-bmc-code-mgmt#

phosphor-dbus-interfaces#

phosphor-host-ipmid#

phosphor-logging#

phosphor-time-manager#

phosphor-networkd#

phosphor-state-manager#

phosphor-user-manager#

pldm#

spdm#

Release 1.3.10#

`bios-settings-mgr`#

`openbmc`#

`dbus-sensors`#

`entity-manager`#

`libmctp`#

`linux`#

`nsmd`#

`nvidia-code-mgmt`#

`bmcweb`#

`nvidia-ipmi-oem`#

`nvidia-power-manager`#

`nv-shmem`#

`phosphor-bmc-code-mgmt`#

`phosphor-dbus-interfaces`#

`phosphor-host-ipmid`#

`phosphor-logging`#

`phosphor-time-manager`#

`phosphor-networkd`#

`phosphor-state-manager`#

`phosphor-user-manager`#

`pldm`#

`spdm`#