Is this page helpful?

Known Issues#

This section lists the known issues in this release.

Trusted NVLink partition is not supported in this release.
Adaptive Bandwidth Mode and NVLink retraining require a TRD11 driver or later.
Compute node firmware update from an older version (any version before 0.9) will require an intermediate upgrade step before upgrading to the latest version of firmware.

There is a GPU security change that requires customers to flash from v0.6 -> v0.8 -> v0.9 -> 1.x

Workaround: As an interim step, go to the v0.8 GPU firmware.
Firmware Update task completed 100% but says critical.

The full log causes the task status update to fail.

Workaround: The fw update was successful, no workaround is needed. To confirm that the status changes to successful, clear the eMMC and run it again:

https://${bmcip}/redfish/v1/Managers/HGX_BMC_0/Actions/Oem/eMMC.SecureErase
GPU firmware cannot be updated until after SBIOS boots into UEFI.

The Grace PCIe PERST signal causes the GPU firmware’s MCTP stack to not work reliably until the UEFI stage of boot (PERST deasserted).

Workaround: Wait for the SBIOS to boot into UEFI to run a firmware update.
The HMC_BMC_0 Status is stuck in a Starting state.

The HMC MCTP service signaling of “start” was not being triggered properly to move the task to the completed state.

Workaround: Check the HMC_READY GPI and FirmwareInventory in Redfish for another way to determine whether the HMC is ready.
ERoTs are not responding to MCTP.

A secure boot task cancels background copy. When the background copy was cancelled, the ERoT was responding to MCTP with errors. This issue with EROT v1.4.7 was addressed in EROT v1.4.8 and in the GB200 NVL72 GB200NVL72_P4975_CX7_Release_0.9.00 full release.

Workaround: Flash back to ERoT .262 or move forward to ERoT 1.4.8.

Multiple AX cycles might be required to get ERoT “unstuck” and allow a firmware update. To avoid race conditions, disable BMC Auto power on.
FPGA ERoT 1 is missing in the DOT Redfish APIs.

All Incoming PG548 (Bianca) boards from NVIDIA L6 have one sockmet (1P) SBIOS. During L10 manufacturing, if a system boots with two 1P SBIOS, module 0 boots but module 1 does not, which triggers the HMC to only build Redfish for one of the two modules.

Workaround: Disable BMC Auto power on and do not let the system boot until a proper 2S SBIOS is flashed. After a system enters this failed state, the HMC should not show module 1 in Redfish until after an HMC factory reset and AUX power cycle is performed.
HMC Telemetry Missing in Redfish.

This issue impacts some partner BMCs due to an I2C contention issue that is reading the HMC FRU on I2C bus 3. The HMC FRU is on a shared I2C bus between the BMC and the HMC and can lead to a multi-master collision.

Workaround: Before accessing the FRU and checking Redfish to the HMC, ensure that the BMC is properly checking for HMC_READY.
SBIOS flash interruption during a Grace power cycle.

During an SBIOS flash, if Grace is reset, the ERoT will abort the flash and give SPI priority to boot the CPU. This causes Boot Chan 0 to become corrupted. SBIOS boots from Boot Chain 1, and a background copy will recover the redundancy.

Workaround: Complete one of the following tasks:
```
- Ensure that the SBIOS flash is finished before resetting Grace.

- The BMC can hold Grace in reset by asserting `SYS_RST` until the flash is done.
```
FPGA ERoT 1 is missing in the DOT Redfish APIs.

All Incoming PG548 (Bianca) boards from NVIDIA L6 have one socket (1P) SBIOS. During L10 manufacturing, if a system boots with two 1P SBIOS, module 0 boots but module 1 does not, this triggers the HMC to only build Redfish for one of the two modules.

Workaround: Disable the BMC Auto power on and do not allow the system boot until a proper 2S SBIOS is flashed. After a system gets in this failed state, the HMC will not show “module 1” in Redfish until after an HMC Factory Reset and an AUX power cycle is performed.
The SBIOS will not boot on after an L2 reset.

If Boot Chain is corrupt or will not boot, the Grace/EROT will try to fall back to Boot Chain 1. On an earlier version of the SBIOS, Socket 0 successfully resets, but Socket 1 fails with a SPI contention.

Workaround: Ensure that Boot Chain 0 contains a proper image.
Parallel GPU resets led to unexpected system crashes a single digit percentage of time.

A GPIO-based GPU reset depends on a four-way handshake that is based on the GPIO pins being in sync. However, there are corner cases where this handshake can get out of sync, which might cause RAS_FW to make an incorrect assumption about the reset status of the GPU, and cause a fault.

Workaround: If RAS_FW detects that the GPU init status is questionable, RAS_FW will request an additional reset from users.
cuSOLVERMp 0.9.0 workloads may encounter GPU errors.

Workloads that use cuSOLVERMp 0.9.0 may encounter a GPU error during execution. In affected cases, driver logs may report XID 31 followed by XID 109.

Workaround: Update the workload to use the latest cuSOLVERMp.
Intermittent CPU-only HPL performance variability.

On some systems, certain 2.0 release candidate firmware stacks may show intermittent CPU-only HPL performance instability, including higher run-to-run variation and lower sustained FP64 results than expected.

Workaround: There is currently no workaround and this issue will be fixed in a future release.
Intermittent topology-related failure during GPU partition recovery after reset.

During GPU partition recovery after reset, an intermittent race between Fabric Manager and the switch manager can cause premature reset handling and a topology-related failure before recovery fully completes.

Workaround: There is currently no workaround and this issue will be fixed in a future release.
Intermittent NVLink SHARP multicast registration failure on GB300 multi-node systems.

On multi-node systems, NVLink SHARP multicast setup may intermittently fail after abnormal partition deletion, causing multicast memory registration to fail with an invalid-argument error and leading to ncclCommWindowRegister failure.

Workaround: Avoid using a partition ID that was used previously.
Performance degradation for 256 MB NCCL AllReduce messages with MNNVL disabled.

On RoCE configurations running NCCL 2.30.7 with MNNVL disabled, the NCCL tuner may select NVLSTree/Simple instead of Ring/Simple for 256 MB AllReduce messages. This selection can cause lower AllReduce performance for affected workloads.

Workaround: Set NCCL_ALGO=RING and NCCL_PROTO=SIMPLE for affected workloads.
Multi-property PATCH to Redfish AccountService returns HTTP 400 for one out-of-range value.

When a single PATCH request updates multiple Redfish AccountService properties and one value is below the enforced security minimum, such as AccountLockoutThreshold or AccountLockoutDuration, the valid properties are still applied. However, the response returns HTTP 400 instead of the HTTP 200 response with a per-property error required by the Redfish specification.

Workaround: There is currently no workaround and this issue will be fixed in a future release.
FPGA may permanently assert power brake after a secondary module overt fault.

If a thermal OVERT condition occurs on the secondary module, the FPGA may incorrectly assert power brake permanently. This condition can reduce system performance on subsequent power-ups.

Workaround: AC power cycle the compute tray to clear the condition.
BMC ERoT serial number may fail to populate in Redfish (< 0.5%).

Workaround: This does not cause any operational issues. Perform a graceful restart of the BMC to correct the serial number display.
GPU re-introduced the following correction of location programming. Link status is not updated.

Fabric Manager detects GPUs that have invalid or duplicate locations, usually indicating an EEPROM mis-programming and does not initialize these GPUs. In the duplicated location scenario, both GPUs are introduced into the cluster when the information is corrected. The port states may not be updated in this scenario for these GPUs being introduced, causing messaging and/or NVLS failures.

Workaround: Perform a GPU reset of the GPUs being introduced to correct this error.
On certain GB200 NVSwitch platforms, the switch reboot history may display multiple unexpected reboots with the reason shown as Platform reset (retrieved using the NVOS command: nv show sys reboot -o json | jq -r .reason). The behavior is caused by a race condition in a driver, which can be triggered when the internal InfiniBand debug utility iblinkinfo runs on NVOS. NVIDIA plans to include a fix in a future NVOS release.
coRIM should reference the value of FWID tcg-dice-TcbInfo in AliasKeyCert.

The FWID[0] field of DiceTcbInfo in the GPU iRoT DICE certificate contains the FSP FMC measurement. This value is written to an RTS hardware register (MSR 2), with hardware recording additional state information to compute the value to be stored in the register. The final register value is reported in SPDM measurement block Index 4.

The following Python code shows how to generate the measurement block 4 value in the GPU CoMID based on the FWID value contained in the DICE certificate.

In the following example:
- GPU CoMID value for measurement block 4 is:
  
  "sha-384;gOQ20Htc9drxCgG6pdBzpQ6SHuQY9YVsV78zyOP9GG3Vg6N6TyUnmIqz/zchkTkM"
  
  which is
  
  "80E436D07B5CF5DAF10A01BAA5D073A50E921EE418F5856C57BF33C8E3FD186DD583A37A4F2527988AB3FF372191390C"
  
  in hexadecimal.
- The DICE FWID[0] value is:
  
  "BB5FEC93F7EC3E89204A21F4B1AF6B941E92A2B5550DA06009C06455439CF16C6EEAC3234DCBB3A4AEAC5F6E9A07B5C4"
  
  The example Python code is:
```
import hashlib
fwid = bytes.fromhex('BB5FEC93F7EC3E89204A21F4B1AF6B941E92A2B5550DA06009C06455439CF16C6EEAC3234DCBB3A4AEAC5F6E9A07B5C4')
msr = bytearray(48)
msr_buf = bytearray([0x0, 0x0, 0x11, 0x0])
msr_buf.extend(fwid)
msr_buf.extend(msr)
msr = hashlib.sha384(msr_buf).digest()
print(msr.hex())
```
  Output:
```
80e436d07b5cf5daf10a01baa5d073a50e921ee418f5856c57bf33c8e3fd186dd583a37a4f2527988ab3ff372191390c
```

DGX Issues#

After an Aux or PDU power cycle, the BMC may fail to restore the host to previous power state

If PowerRestorePolicy is set to LastState, the BMC may occasionally fail to restore to the previous host power state after an Aux or PDU power cycle. For example, if the previous host state was On, the BMC may not turn on the host automatically after power is restored.

Workaround: If there is a fixed desired power state for the host after the power is restored, you can set PowerRestorePolicy to either AlwaysOn or AlwaysOff. Otherwise, you can send an additional power-on or power-off command after the power is restored to achieve the desired host power state.
Leak detection voltage sensors do not show the Degraded state

When the BMC gets an out-of-range reading from the voltage-based leak detection sensor, it marks the Leak Detector’s Status:State as Degraded. The corresponding Voltage Sensor does not currently align its Status:State property with the Leak Detector to reflect the Degraded value.

Workaround: It is recommended to use the Leak Detector Redfish URIs to get the most accurate state of the leak detector system. The sensor will only provide the raw voltage value for diagnostic purposes.

Leak Detector URIs:

/redfish/v1/Chassis/Chassis_0/ThermalSubsystem/LeakDetection/LeakDetectors/Chassis_0_LeakDetector_0_ColdPlate

/redfish/v1/Chassis/Chassis_0/ThermalSubsystem/LeakDetection/LeakDetectors/Chassis_0_LeakDetector_0_Manifold

/redfish/v1/Chassis/Chassis_0/ThermalSubsystem/LeakDetection/LeakDetectors/Chassis_0_LeakDetector_1_ColdPlate

/redfish/v1/Chassis/Chassis_0/ThermalSubsystem/LeakDetection/LeakDetectors/Chassis_0_LeakDetector_1_Manifold
No mechanism to disable the host IPMI interface

This firmware release does not provide a mechanism to disable the host IPMI interface and any privileged user running on a compute tray can send IPMI commands to the BMC on the same compute tray.

Workaround: A future BMC firmware release will provide an interface to restrict host IPMI commands. In the meantime, necessary access controls need to be implemented on the compute trays to limit host privileged access by users not authorized to access the BMC.
BMC SKU ID is sometimes not available through Redfish

The BMC’s SKU ID should be available through the following Redfish URI under the “SKU” property: /redfish/v1/Chassis/BMC_0

Due to a current issue on the BMC, this property may not be available on all systems. Workaround: There is currently no known workaround. The plan is to resolve this issue for the next release.
Can not set IPv6 address through Redfish

There is a current BMC bug where setting the IPv6 through the following Redfish URI fails: /redfish/v1/Managers/BMC_0/NetworkProtocol

The command will return the following message: “Message”: “The property ‘Address or Port’ with the requested value of ‘“2620:10d:c0a3:103::9:7554”’ could not be written because the value does not meet the constraints of the implementation.” This is a known issue and will be addressed in the next release.

Workaround: There is currently no known workaround. The plan is to resolve this issue for the next release.
No error Redfish response when credentials are not provided

If no credentials are provided in a Redfish command, the BMC currently does not return an explicit message indicating the lack of credentials. A 401 Unauthorized response is still returned, and access is prevented. This only applies if no credentials are provided. If credentials are provided but are incorrect, then the BMC will return a message indicating invalid credentials.

Workaround: The return code can be checked to confirm the status of the command. The 401 will indicate to the user that access was unauthorized.
Redfish GPU_0/Ports/NVLink_0, the property “LinkState” value does not change after a DC Cycle.

NVLinks that were previously disabled through Redfish continue to report the LinkState as Enabled and LinkStatus as LinkUp.

Workaround: Use host reporting tools (nvidia-smi/NVML/DCGM) to fetch the NVLink status for disabled links.
Leak detector voltage values read through the compute tray BMC via IPMI are clamped to 0.165 and 1.815.

For leak presence/absence, use the standard leak detection interfaces under /redfish/v1/Chassis/Chassis_0/ThermalSubsystem/LeakDetection/LeakDetectors. Do not use raw sensor voltage as the primary leak indicator.

Workaround: To read the correct voltage when the sensor is outside that range, use Redfish (for example, /redfish/v1/Chassis/Chassis_0/Sensors) instead of IPMI.