Improvements

This section describes the improvements in each release.

Release 1.3.10

1. Refer to the NVOS release notes in the NVOS NVOnline posting for additional information about the improvements in this release.

2. This release includes security updates. Refer to the Partner Security Bulletin: NVIDIA Blackwell Data Center Products - June 2026.

3. GPU recovery after failed AddGpusToPartition call.

   Resolved an issue where a GPU could fail to rejoin the NVLink fabric after a failed AddGpusToPartition API call during an add or recovery flow.

4. Multicast team teardown during HA events.

   Resolved an issue where multicast persistence handling could prematurely tear down multicast teams during high availability (HA) resiliency events.

5. OpenBMC HMC memory leak when starting an update.

   Resolved an issue where the hardware temperature sensor service could leak memory on the OpenBMC HMC when an update was started.

6. SError after HSM_FABRIC fault.

   Resolved an issue that could cause an SError to occur after an HSM_FABRIC error. The fault is now handled correctly without raising a spurious SError.

7. Disabled HBM SBE reporting by default.

   HBM SBE reporting is now disabled by default to avoid duplicate reporting because HBM SBEs are already reported through XIDs. Use the BCT setting to enable this reporting if you want to continue double reporting of HBM SBEs.

8. Neoverse V2 Erratum 2618597.

   Included Arm’s suggested errata workaround for 2618597.

9. Resolved an issue with SDEI error reporting.

   The firmware now saves NS context before existing RAS handling of SDEI events to avoid TLS pointer corruption.

10. XID 150/154 immediately after GPU reset (MSE communication initialization).

    During reboot stress testing (22+ hours), XID 150 and 154 could rarely appear immediately after a GPU reset. A race condition in the MSE communications path could leave a message request to MSE in an “empty but incomplete” state. Without recovery for that state, the communications flow did not reinitialize properly and MSE became unable to process new requests. Recovery handling for this state has been added.

11. Some GPUs may fail to ramp to full TGP when power smoothing ramp-up is enabled, resulting in reduced compute performance.

    Implemented a VBIOS fix to update the ramp-up logic so affected GPUs can complete ramp-up.

12. Enhanced logging for C2C training and memory training failure.

    This release includes logging enhancements to provide additional debug information in the ROT log.

13. Intermittent GPU boot failure.

    Fixed a GPU boot failure that occurred when FLR was issued while NVLink was training.

14. Micron NVMe drives remain available behind CX8 with PCIe max payload configuration.

    Resolved an issue where Micron M.2 and E1.S NVMe drives could become unavailable on PCIe when the system used the NVIDIA reference SBIOS and all installed drives were Micron. The SBIOS now programs the PCIe root ports with the proper PCIe Max Payload setting. GB200 systems with CX7 were not affected.

15. NVSwitch CPU reboots with MGMT board VR failure.

    Added an improvement to avoid an NVSwitch CPU reboot caused by a forced or unexpected BMC reboot.

16. XID 145 due to incorrect routing configured by NVLSM (36x2 only).

    Resolved an issue where NVLink Subnet Manager (NVLSM) configured incorrect routing in switches in one rack of 36x2 toward GPU ALID when the GPU belongs to another rack of 36x2.

17. Power smoothing minimum power floor increase.

    This enhancement to power smoothing now allows the Minimum Power Floor (MPF) value to be configured to 95% (previously 90%).

18. Workarounds for Arm Neoverse-V2 errata 3841324 and 3442699.

    Implemented suggested workarounds for two Arm errata that previously caused incorrect instructions to be executed.

Release 1.3.6

1. MSE Uptime crash ~67 days (XID 150).

   After extended uptime (approximately 60+ days), systems can exhibit NVLink task scheduling behavior that leads to GPU driver hangs and, when using R580 driver or later, appearance of XID errors 150 and 154 in the kernel log. The overflow handling in the GPU’s NVLink management microcode within the VBIOS firmware is now fixed for this release.

2. GFM does not handle partition state changes from SM to Error.

   Resolved an issue where the Global Fabric Manager (GFM) did not properly propagate partition error states from the Subnet Manager to the NMX-C GetPartitionInfoList() API. Unhealthy partition states are now correctly reported.

3. NETIR dumps missing, libnvidia-ml.so error in NV-Bug-Report.

   Fixed an issue where running the mst gpu add command failed due to broken symbolic links, resulting in missing NETIR dumps.

4. BMC RF server does not close inactive TCP sessions, resulting in loss of RF access.

   Resolved an issue where idle HTTPS sessions accumulated on the server without being closed promptly, resulting in an increased number of established but unused connections.

5. GPU PMU and thermal issues caused by driver-VBIOS race condition.

   Resolved a race condition between the driver and VBIOS that caused communication failures, leading to GPU PMU halted errors and thermal issues. The VBIOS update adds locking between response paths to prevent this condition.