Known Issues#
This section lists the known issues in this release.
Trusted NVLink partition is not supported in this release.
Adaptive Bandwidth Mode and NVLink retraining require a TRD11 driver or later.
cuSOLVERMp 0.9.0 workloads may encounter GPU errors.
Workloads that use cuSOLVERMp 0.9.0 may encounter a GPU error during execution. In affected cases, driver logs may report XID 31 followed by XID 109.
Workaround: Update the workload to use the latest cuSOLVERMp.
Intermittent CPU-only HPL performance variability.
On some systems, certain 2.0 release candidate firmware stacks may show intermittent CPU-only HPL performance instability, including higher run-to-run variation and lower sustained FP64 results than expected.
Workaround: There is currently no workaround and this issue will be fixed in a future release.
Intermittent topology-related failure during GPU partition recovery after reset.
During GPU partition recovery after reset, an intermittent race between Fabric Manager and the switch manager can cause premature reset handling and a topology-related failure before recovery fully completes.
Workaround: There is currently no workaround and this issue will be fixed in a future release.
Intermittent NVLink SHARP multicast registration failure on GB300 multi-node systems.
On multi-node systems, NVLink SHARP multicast setup may intermittently fail after abnormal partition deletion, causing multicast memory registration to fail with an invalid-argument error and leading to
ncclCommWindowRegisterfailure.Workaround: Avoid using a partition ID that was used previously.
Performance degradation for 256 MB NCCL AllReduce messages with MNNVL disabled.
On RoCE configurations running NCCL 2.30.7 with MNNVL disabled, the NCCL tuner may select NVLSTree/Simple instead of Ring/Simple for 256 MB AllReduce messages. This selection can cause lower AllReduce performance for affected workloads.
Workaround: Set
NCCL_ALGO=RINGandNCCL_PROTO=SIMPLEfor affected workloads.Multi-property PATCH to Redfish AccountService returns HTTP 400 for one out-of-range value.
When a single PATCH request updates multiple Redfish AccountService properties and one value is below the enforced security minimum, such as AccountLockoutThreshold or AccountLockoutDuration, the valid properties are still applied. However, the response returns HTTP 400 instead of the HTTP 200 response with a per-property error required by the Redfish specification.
Workaround: There is currently no workaround and this issue will be fixed in a future release.
FPGA may permanently assert power brake after a secondary module overt fault.
If a thermal OVERT condition occurs on the secondary module, the FPGA may incorrectly assert power brake permanently. This condition can reduce system performance on subsequent power-ups.
Workaround: AC power cycle the compute tray to clear the condition.
BMC ERoT serial number may fail to populate in Redfish (< 0.5%).
Workaround: This does not cause any operational issues. Perform a graceful restart of the BMC to correct the serial number display.
In rare instances, (< .01 %) BMC management interface may fail to train phy link.
In rare instances ( < 0.01%), the BMC management interface phy may not link correctly with a switch.
Workaround: Reboot the BMC through the host with
ipmitool mc reset cold, or use Redfish through the BMC-to-host USB Ethernet interface:curl -k -u "user:password" -X POST https://${bmcip}/redfish/v1/Managers/BMC_0/Actions/Manager.Reset -d '{"ResetType": "ForceRestart"}'
Unexpected PCB sensor data was detected at
/redfish/v1/Chassis/HGX_Chassis_0/Sensors.Renamed the PCB-Temp sensor to Exhaust-Temp sensor. While the PCB sensor no longer exists in the RF sensor list, the PCB temp events still exist and increment on exhaust temp event occurrence.
Workaround: There is currently no workaround, and this issue will be fixed in a future release.
NeighborMTUDiscards property is not defined in MetricReportDefinitions
NeighborMTUDiscards property is not defined in MetricReportDefinitions but exists in the Metric Reports.
Workaround: This property will be removed in a future release. Property read will fail - do not use.
When an incorrect static topology file (for example 2x36) is specified on a 1x72 topology, FM log displays duplicate control plane error states
When an incorrect static topology file (for example 2x36) is specified on a 1x72 topology, both CONFIG_ERROR_ADDITIONAL_CHASSIS_DETECTED and CONFIG_ERROR_MISSING_CHASSIS are set.
Workaround: There is currently no workaround, and this issue will be fixed in a future NVOS release.
Inconsistent GPU memory reported by NSM,
nvidia-smi, and RedfishMultiple management interfaces report different total GPU memory values: NSM Type 3 (0x0C) and
nvidia-smi(-q -d MEMORY) return 284,208 MiB, while Redfish TotalMemorySizeMiB returns 285,324 MiB. This occurs when querying inventory or memory capacity from NSM,nvidia-smi, MODS, and Redfish.Reporting inconsistency may cause confusion in monitoring, inventory reconciliation, or capacity planning. There is no data-loss or security impact identified.
Workaround: There is currently no workaround, and this issue will be fixed in a future release.
No mechanism to disable the host IPMI interface
This firmware release does not provide a mechanism to disable the host IPMI interface and any privileged user running on a compute tray can send IPMI commands to the BMC on the same compute tray.
Workaround: A future BMC firmware release will provide an interface to restrict host IPMI commands. In the meantime, necessary access controls need to be implemented on the compute trays to limit host privileged access by users not authorized to access the BMC.
Fan Control and Leak Detector user configured settings that are modified through Redfish PATCH API reset to their defaults after a BMC firmware update.
Some Fan Control and Leak Detector properties are configurable through the PATCH method on Redfish. For example, you can use the following API request to configure whether the BMC shuts down the chassis when it detects a leak:
curl -s -k -u ${USER}:${PASSWORD} https://${BMCIP}/redfish/v1/Chassis/Chassis_0/Oem/Nvidia/Policies/LeakDetectionPolicy --request PATCH -d '{"PolicyEnabled":true}'
This setting will persist through BMC resets and tray power cycles, but will not currently survive a BMC firmware update.
Workaround: Check and reapply any desired settings after a BMC firmware update.
Liteon PowerShelf FW Update Task Monitoring Does Not Fully Track Progress for PMC and PSU Firmware
During Liteon PMC firmware updates performed in Immediate mode, the update process does not report task progress. While the update is in progress, the BMC becomes temporarily inaccessible, which prevents accurate monitoring. A similar limitation exists during PSU firmware updates, where task monitoring is not fully reliable due to an indexing issue from 0-5 to 1-6 in PMC FW 1.3.10. The firmware updates complete successfully for both PSU and PMC FW. The issue is specifically with task monitoring, which can affect users who rely on automation or scripted checks. This limitation is known in Liteon PMC firmware versions 1.3.10 and 1.3.9.
Workaround: There is currently no definitive workaround. Wait approximately 10 minutes for the PMC to reboot and the BMC to become accessible again after initiating the update. The task monitoring issue is expected to be resolved in the upcoming Liteon PMC firmware release 1.3.11.
Leak detector voltage values read through the compute tray BMC via IPMI are clamped to 0.165 and 1.815.
For leak presence/absence, use the standard leak detection interfaces under
/redfish/v1/Chassis/Chassis_0/ThermalSubsystem/LeakDetection/LeakDetectors. Do not use raw sensor voltage as the primary leak indicator.Workaround: To read the correct voltage when the sensor is outside that range, use Redfish (for example,
/redfish/v1/Chassis/Chassis_0/Sensors) instead of IPMI.