Is this page helpful?

Improvements#

This section provides information about the improvements in each release.

Release 1.0.10#

  1. Refer to the NVOS release notes in the NVOS NVOnline posting for additional information about the improvements in this release.

  2. This release includes security updates. Refer to the Partner Security Bulletin: NVIDIA Blackwell Data Center Products - June 2026.

  3. GFM may crash when MAD messages cannot be sent.

    Fixed a Fabric Manager stability issue where failed MAD message sends could cause GFM to crash during fabric operations.

  4. GPU recovery after failed AddGpusToPartition call.

    Resolved an issue where a GPU could fail to rejoin the NVLink fabric after a failed AddGpusToPartition API call during an add or recovery flow. This fix improves partition recovery after service management errors.

  5. Multicast team teardown during HA events.

    Resolved an issue where multicast persistence handling could prematurely tear down multicast teams during high availability (HA) resiliency events. This fix improves multicast workload stability during those events.

  6. OpenBMC HMC memory leak when starting an update.

    Resolved an issue where the hardware temperature sensor service could leak memory on the OpenBMC HMC when an update was started. The leak could fill HMC memory and cause the HMC to crash.

  7. SError after HSM_FABRIC fault.

    Resolved an issue that could cause an SError to occur after an HSM_FABRIC error. The fault is now handled correctly without raising a spurious SError.

  8. Disabled HBM SBE reporting by default.

    HBM SBE reporting is now disabled by default to avoid duplicate reporting because HBM SBEs are already reported through XIDs. Use the BCT setting to enable this reporting if you want to continue double reporting of HBM SBEs.

  9. Neoverse V2 Erratum 2618597.

    Included Arm’s suggested errata workaround for 2618597.

  10. Resolved an issue with SDEI error reporting.

    The firmware now saves NS context before existing RAS handling of SDEI events to avoid TLS pointer corruption.

  11. XID 150/154 immediately after GPU reset (MSE communication initialization).

    During reboot stress testing (22+ hours), XID 150 and 154 could rarely appear immediately after a GPU reset. A race condition in the MSE communications path could leave a message request to MSE in an “empty but incomplete” state. Without recovery for that state, the communications flow did not reinitialize properly and MSE became unable to process new requests. Recovery handling for this state has been added.

  12. Some GPUs may fail to ramp to full TGP when power smoothing ramp-up is enabled, resulting in reduced compute performance.

    Implemented a VBIOS fix to update the ramp-up logic so affected GPUs can complete ramp-up.

  13. Enhanced logging for C2C training and memory training failure.

    This release includes logging enhancements to provide additional debug information in the ROT log.

  14. Intermittent GPU boot failure.

    Fixed a GPU boot failure that occurred when FLR was issued while NVLink was training.

Release 1.0.6#

  1. MSE Uptime crash ~67 days (XID 150).

    After extended uptime (approximately 60+ days), systems can exhibit NVLink task scheduling behavior that leads to GPU driver hangs and, when using R580 driver or later, appearance of XID errors 150 and 154 in the kernel log. The overflow handling in the GPU’s NVLink management microcode within the VBIOS firmware was fixed to avoid this issue going forward.

  2. Intermittent CX8 to GPU link instability causing CTO and other PCIe errors.

    Stabilized the CX8 and GPU Gen6 link operation with the application of improved PHY settings for receiver gain and PLL stability.

  3. GPU PM L1 stability improvement.

    Improved the GPU implementation of PCI-PM L1 flow through setting changes that now ensure continuous PLL stability.

  4. Tray swaps no longer leave stale GPU state with the system manager, so newly added GPUs are correctly detected and appear in the partition.

  5. Resolved an issue where, after a tray swap, the partition showed only 72 GPUs and the nv CLI showed 0.0.0.0.

    Fixed issues around identifying GPUs belonging to a Compute Node.

  6. Fixed switch configuration on NVLink systems that incorrectly allowed short transient congestion conditions to cause spurious timeout events and unnecessary port disables (XID149.33).

  7. Elevated rate of GPU PMU halted and thermal issues reported.

    Resolved a race condition between the driver and VBIOS that could cause communication failures, resulting in elevated rates of GPU PMU halted and thermal issues. The VBIOS update implements locking between response paths to prevent this condition.

  8. Improved NVLink reliability with updated NVLink reduction and accounting for HLL.

    This fix prevents unnecessary analysis of false positives (where an error is indicated but no actual fault exists), reducing wasted effort and improving diagnostic accuracy.

  9. Enable DHCP6 on eth0.

    This fix ensures that DHCP6 is enabled and running.