R570 TRD1 Release Notes

Overview

This release consists of the NVIDIA CUDA Toolkit version 12.8, which is paired with the NVIDIA Data Center GPU Drivers version 570.86.15.

The following features are supported in this software release:

• The Protected PCIe (PPCIe) mode and Single GPU Passthrough (SPT) for NVIDIA H100 GPUs.

• Key Rotation in the SPT mode.

Refer to Feature Summary for more information about the supported Confidential Computing (CC) modes for H100 GPUs.

Before you deploy workloads, NVIDIA recommends that users use good practices, such as performing regular attestations.

Feature Summary

Confidential Computing

This section provides information about the CC features in this release.

Hopper Single GPU Passthrough with a Bounce Buffer

NVIDIA Trusted Computing support for NVIDIA Hopper GPUs was first introduced with the Hopper Single GPU Passthrough with a Bounce Buffer (SPT CC) mode. In this mode, one GPU can be passed through for each Confidential VM (CVM). A bounce buffer stages encrypted data transfers between the GPU device and CVM. Refer to the Intel TDX - Confidential Computing Deployment Guide and AMD SNP - Confidential Computing Deployment Guide for more information.

Table 1. Component Versions to Enable the SPT CC Mode

Protected PCIe

This section provides information about the PPCIe features in this release.

Eight Hopper GPUs with Four NVSwitch Passthrough

Trusted Computing support in the PPCIe mode is available only with the Hopper GPUs and Intel CPUs with TDX technology in an Ubuntu KVM/QEMU environment.

In the PPCIe mode, multiple NVSwitch/NVLink interconnected Hopper GPUs can be passed through to one CVM. As in the SPT CC mode, a bounce buffer is used to stage encrypted data transfers between the GPU device and CVM over the PCI Express bus. In this mode, GPU-GPU communications over the NVLink/NVSwitch are not encrypted (refer to the Protected PCIe Deployment Guide PDF file, which is a part of this posting, for more information).

Table 2. Component Versions to Enable PPCIe

Limitations

This section provides a list of the known limitations in this release.

Limitations in the Hopper SPT CC Mode

• Only one GPU per CVM is allowed.

  • Only one CVM is permitted even in systems with multiple GPUs.

  • This limitation is temporary and is expected to be resolved in a future release.

• With a maximum of one GPU passed through per CVM, operations that involve multiple GPUs, such as P2P communications, are not supported.

Limitations in the Hopper PPCIe Mode

• Hopper PPCIe is limited to HGX 8-way Air cooled systems, where the eight GPUs and four NVSwitches are passed through to one VM. Other topologies are not supported.

• NVIDIA NCCL is the only supported GPU communication library.

• In the PPCIe mode, when the source or destination operand are imported, GPU memory allocations on a device that is not visible to the process, the host-to-device, or device-to-host copies might fail asynchronously with cudaErrorLaunchFailure.

• In the PPCIe mode, using cooperative_groups::multi_grid_group::sync in kernels launched with cudaLaunchCooperativeKernelMultiDevice results in the kernel failing with cudaErrorIllegalAddress.

• CUDA Interprocess Communication (IPC) is not supported in PPCIe mode.

• Developer tools such as NVIDIA Nsight for profiling are not supported in PPCIe mode

Limitations in the SPT CC and PPCIe Modes

This section provides information about the limitations that apply to the SPT CC and PPCIe modes.

The following runtime APIs are incompatible with CC:

• Host memory registration. The following CPU memory pinning operations are not allowed in CC mode:

  • cudaHostRegister

  • cudaHostUnregister

• cudaMemcpy calls that describe an HtoA or AtoH copy.

The following Host-to-Array and Array-to-Host copies are not supported because of the potential requirement for a conversion between pitch-linear and block-linear access patterns of the CUArray memory type during the secure copy operation:

• cudaMemcpy2DFromArray

• cudaMemcpy2DFromArrayAsync

• cudaMemcpy2DToArray

• cudaMemcpy2DToArrayAsync

• cudaMemcpy3D

• cudaMemcpy3DAsync

• cudaMemcpy3DPeer

• CUDA External Resource Interoperability. The following APIs are not supported because an external resource interaction with a trusted execution environment is not permitted:

  • cudaImportExternalMemory

  • cudaExternalMemoryGetMappedBuffer

  • cudaExternalMemoryGetMappedMipmappedArray

  • cudaDestroyExternalMemory

  • cudaFreeMipmappedArray

  • cudaImportExternalSemaphore

  • cudaSignalExternalSemaphoresAsync

  • cudaWaitExternalSemaphoresAsync

  • cudaDestroyExternalSemaphore

  • cudaGraphAddExternalSemaphoresSignalNode

  • cudaGraphAddExternalSemaphoresWaitNode

  • cudaGraphExecExternalSemaphoresSignalNodeSetParams

  • cudaGraphExecExternalSemaphoresWaitNodeSetParams

  • cudaGraphExternalSemaphoresSignalNodeGetParams

  • cudaGraphExternalSemaphoresSignalNodeSetParams

  • cudaGraphExternalSemaphoresWaitNodeGetParams

  • cudaGraphExternalSemaphoresWaitNodeSetParams

The following Driver APIs are incompatible with CC:

• Host memory registration. The following CPU memory pinning operations are not allowed in CC mode:

  • cuMemHostRegister

  • cuMemHostUnregister

• cuMemcpy calls that describe an HtoA or AtoH copy. The following Host-to-Array and Array-to-Host copies are not supported because of the potential requirement for a conversion between pitch-linear and block-linear access patterns of the CUArray memory type during the secure copy operation:

  • cuMemcpy2DUnaligned

  • cuMemcpyAtoH

  • cuMemcpyAtoHAsync

  • cuMemcpyHtoA

  • cuMemcpyHtoAAsync

• cuStream memory operation calls passing pointers allocated using cudaMallocHost, cudaHostAlloc, cuMemAllocHost APIs, and their graph counterparts:

  • cuStreamBatchMemOp

  • cuStreamBatchMemOp_v2

  • cuStreamWaitValue32

  • cuStreamWaitValue32_v2

  • cuStreamWaitValue64

  • cuStreamWaitValue64_v2

  • cuStreamWriteValue32

  • cuStreamWriteValue32_v2

  • cuStreamWriteValue64

  • cuStreamWriteValue64_v2

  • cuGraphAddBatchMemOpNode

  • cuGraphBatchMemOpNodeGetParams

  • cuGraphBatchMemOpNodeSetParams

  • CuGraphExecBatchMemOpNodeSetParams

• CUDA External Resource Interoperability. The following APIs are not supported as external resource interaction with a trusted execution environment is not permitted:

  • cuImportExternalMemory

  • cuExternalMemoryGetMappedBuffer

  • cuExternalMemoryGetMappedMipmappedArray

  • cuDestroyExternalMemory

  • cuFreeMipmappedArray

  • cuImportExternalSemaphore

  • cuSignalExternalSemaphoresAsync

  • cuWaitExternalSemaphoresAsync

  • cuDestroyExternalSemaphore

  • cuGraphAddExternalSemaphoresSignalNode

  • cuGraphAddExternalSemaphoresWaitNode

  • cuGraphExecExternalSemaphoresSignalNodeSetParams

  • cuGraphExecExternalSemaphoresWaitNodeSetParams

  • cuGraphExternalSemaphoresSignalNodeGetParams

  • cuGraphExternalSemaphoresSignalNodeSetParams

  • cuGraphExternalSemaphoresWaitNodeGetParams

  • cuGraphExternalSemaphoresWaitNodeSetParams

The following CUDA capabilities are incompatible with CC:

• CUDA/Graphics interop, specifically APIs to enable interop with EGL, VDPAU, OpenGL, DirectX, OptiX, and Vulkan.

• GPUDirect RDMA.

• The CUDA Programmatic Dependent Launch and Synchronization feature will not show expected overlaps in the primary and secondary kernel executions. A program that uses these APIs should functionally succeed in CC modes.

The following CUDA samples are expected to fail when you run them in the CC mode:

• convolutionTexture

• dct8x8

• lineOfSight

• simpleCubemapTexture

• simpleLayeredTexture

• simplePitchLinearTexture

• simpleStream

• simpleTexture

• simpleTextureDrv

• watershedSegmentationNPP

The following CUDA samples are expected to fail in the PPCIe mode:

• simpleIPC

• cudaCompressibleMemory

• p2pBandwithLatencyTest

The following CUDA Runtime APIs are not supported with CC in this release but might be enabled in a future release:

• cudaEventElapsedTime

• cudaEventCreateWithFlags where Flags is set to cudaEventBlockingSync

The following CUDA capabilities are not supported with CC in this release but might be enabled in a future release:

• CUDA Multi Process Service (MPS).

• CUDA Toolkit minor version compatibility.

• CUDA Forward Compatibility.

Known Issues

• A key rotation feature is not supported with PPCIe. A sophisticated attacker with physical or logical superuser access to the system can act as a passive adversary to capture the ciphertext and execute an attempt to break it or the key.

  Workaround: Users should review the latest research on the effects of extreme AES key usage and the cryptographic wear out to determine their requirements for an attacker advantage. To create a new set of encryption keys in PPCIe mode, users must terminate and launch their CVMs again.

• IV exhaustion will crash the application in PPCIe mode. The H100 CC modes use a 96-bit deterministic IV for each virtual copy engine that is used to transfer data between the GPU and CPU. When this IV space is exhausted, transfers will fail to complete.

  Workaround: Rotate the keys often in supported modes. If the keys are not rotated often, restart the CVM.

• GPU-Ready bit is set when the devtools mode is enabled.

  Workaround: When in full CC-on modes, the driver will not accept any workloads until after the Attestation SDK, or the users, manually enable a GPU-Ready bit.

Note This bit is already enabled in the Devtools mode.

Users should use best practices by attesting the GPU before performing any work. The GPUs booted in devtools mode will be clearly identified, and the attestation will fail.

• With HGX Firmware 1.6.0, there is an increased risk of the GPU or NVSwitch becoming disconnected from the PCIe bus during DC power cycling. HGX Firmware 1.6.1 removed support for PPCIe due to this issue. This issue was resolved in the 1.7.0 firmware release.

  Workaround: A system reboot would need to be performed to bring the missing devices back on the PCIe bus

• NVIDIA Performance Primitives might not work.

NVIDIA Performance Primitives (NPP) uses optimized coding to extract the maximum performance from commonly used transforms/calculations as part of the leverage pinned host memory, which is not supported in CC.