R590 TRD1 Release Notes

Overview

This release consists of the NVIDIA CUDA Toolkit version 13.1, which is paired with the NVIDIA Data Center GPU Drivers version 590.48.01.

The following features are supported in this software release:

• Single GPU Passthrough (SPT CC)

• Hopper Multiple GPU Passthrough with Protected PCIe (PPCIe)

• Blackwell Multiple GPU Passthrough (MPT CC) NEW

For more information about these features, refer to Feature Summary.

For additional information, refer to the following documentation:

• Platform Compatibility Matrix for more information about supported platforms, drivers, firmware, and Secure AI modes

• Secure AI Deployment Guide for details about setting up the confidential computing system

• Secure AI Operations Guide for best practices and CUDA developer considerations

Before you deploy workloads, NVIDIA recommends that users use good practices, such as performing regular attestations.

Feature Summary

This section provides information about the CC features in this release. For more details on Secure AI modes, security posture, and capabilities, please refer to the NVIDIA Secure AI with Blackwell and Hopper GPUs whitepaper.

Single GPU Passthrough (SPT CC)

NVIDIA Trusted Computing support for NVIDIA Hopper GPUs was first introduced with the Hopper Single GPU Passthrough with a Bounce Buffer (SPT CC) mode. SPT CC mode support for Blackwell GPUs was also added in subsequent releases. In this mode, one GPU can be passed through for each Confidential VM (CVM). Multiple CVMs may be hosted on the same node, each with one GPU passed through. A bounce buffer stages encrypted data transfers between the GPU device and the CVM.

Following are the hardware SKUs that support SPT CC mode:

• NVIDIA H100 PCIe

• NVIDIA H800 PCIe

• NVIDIA H100NVL

• NVIDIA H800NVL

• NVIDIA H200NVL

• HGX H100 8-GPU 80GB (Air Cooled)

• HGX H100 4-GPU 80GB HBM3 (Partner Cooled)

• HGX H800 8-GPU 80GB (Air Cooled)

• HGX H800 8-GPU 80GB (Partner Cooled)

• HGX H100 4-GPU 64GB HBM2e (Partner Cooled)

• HGX H100 8-GPU 96GB (Air Cooled)

• HGX H100 4-GPU 94GB HBM2e (Partner Cooled)

• HGX H20A HBM3 96GB 8-GPU (Air Cooled)

• HGX H20 141GB HBM3e 8-GPU (Air Cooled)

• HGX H200 8-GPU 141GB (Air Cooled)

• HGX B200, 8-GPU, SXM6 180GB HBM3e, AC

• RTX PRO 6000 Blackwell Server Edition

• HGX B200-850, 8-GPU, SXM6 180GB HBM3e, AC

Refer to the Intel TDX - Confidential Computing Deployment Guide and AMD SNP - Confidential Computing Deployment Guide for more information.

Table 1. Component Versions to Enable SPT CC Mode

Hopper Multiple GPU Passthrough (Protected PCIe)

Trusted Computing support in the PPCIe mode is available with Hopper-architecture GPUs and Intel CPUs with TDX/AMD CPUs with SEV/SEV-SNP technology in an Ubuntu KVM/QEMU environment.

In the PPCIe mode, multiple Hopper-architecture GPUs interconnected by NVSwitch or NVLink can be passed through to one CVM. As in the SPT CC mode, a bounce buffer is used to stage encrypted data transfers between the GPU device and CVM over the PCI Express bus. In this mode, GPU-GPU communications over the NVLink or NVSwitch interconnect are not encrypted.

Following are the hardware SKUs that support PPCIe mode:

• HGX H100 8-GPU 80GB (Air Cooled)

• HGX H800 8-GPU 80GB (Air Cooled)

• HGX H100 8-GPU 96GB (Air Cooled)

• HGX H20A HBM3 96GB 8-GPU (Air Cooled)

• HGX H200 8-GPU 141GB (Air Cooled)

Table 2. Component Versions to Enable PPCIe

Blackwell Multiple GPU Passthrough (MPT CC)

This release introduces support for Blackwell Multiple GPU Passthrough with a Bounce Buffer (MPT CC) mode. In this mode, up to eight GPUs can be passed through for each Confidential VM (CVM). A bounce buffer stages encrypted data transfers between the GPU device and CVM. GPUs that are part of the same CVM can communicate peer-to-peer over encrypted NVLink connections.

Following are the hardware SKUs that support MPT CC mode:

• HGX B200, 8-GPU, SXM6 180GB HBM3e, AC

• HGX B200-850, 8-GPU, SXM6 180GB HBM3e, AC

Table 3. Component Versions to Enable MPT CC Mode for Blackwell GPUs

Known Issues

• The key rotation feature is not supported with PPCIe. A sophisticated attacker with physical or logical superuser access to the system can act as a passive adversary to capture the ciphertext and execute an attempt to break it or the key.

  Workaround: Review the latest research on the effects of extreme AES key usage and cryptographic wear out to determine your requirements for an attacker advantage. To create a new set of encryption keys in PPCIe mode, you must terminate and launch your CVMs again.

• In systems with multiple GPUs interconnected by NVLink Switch interconnects, the driver registry key NVreg_RegistryDwords="RmNvswitchGpioDetect=2" must be used if any of the virtual machines has only a single GPU passed through with CC-mode set to OFF.

  Workaround: None.

• With Blackwell CC Modes, the number of decoder sessions is restricted. Because of a software bug, you cannot use more than 190 sessions when a single CUDA context is shared among all decoder sessions or more than 28 sessions when one CUDA context per decoder session is created.

  Workaround: None. This issue will be fixed in a future release.

• Performance is degraded with NCCL 2.26.2 and Protected PCIe. Applications might encounter up to 10% slowdown with NCCL 2.26.2.

  Workaround: Use NCCL 2.26.3 or a newer version with the fix.

• IV exhaustion crashes the application in PPCIe mode. The H100 CC modes use a 96-bit deterministic IV for each virtual copy engine that is used to transfer data between the GPU and CPU. When this IV space is exhausted, transfers fail to complete.

  Workaround: Rotate the keys often in supported modes. If the keys are not rotated often, restart the CVM.

• The GPU-Ready bit is always set when devtools mode is enabled, which may lead to confusion with the requirement to set the GPU-Ready bit when running MPT with devtools mode enabled.

  Workaround: When in full CC-on modes, the driver does not accept any workloads until after the Attestation SDK, or the users, manually enable a GPU-Ready bit.

Note: This bit is already enabled in devtools mode.

Users should use best practices by attesting the GPU before performing any work. The GPUs booted in devtools mode are clearly identified, and attestation fails.

• With HGX Hopper firmware 1.6.0, there is an increased risk of the GPU or NVLink Switch falling off the PCIe bus during DC power cycling. This issue was resolved in the 1.7.0 firmware release.

  Workaround: Reboot the system to bring the missing devices back on the PCIe bus. To avoid this issue, use HGX Hopper firmware 1.8.0 or newer.

• NVIDIA Performance Primitives (NPP) might not work. NPP uses optimized coding to extract the maximum performance from commonly used transforms and calculations as part of the leverage pinned host memory, which is not supported in CC.