Configuring the Guest | NVIDIA Confidential Computing

The Virtual Machine Administrator receives a CVM with GPUs attached by the hypervisor and configures the NVIDIA driver in the guest. This persona might not know the BIOS or host OS configuration. Most users start here.

The examples on this page continue from an Ubuntu 24.04 guest that you created by following Creating a Confidential VM. If your system administrators provided the CVM, the output can differ slightly, but the steps are the same.

Enabling LKCA

The NVIDIA driver requires the Linux Kernel Crypto API (LKCA) in every Confidential Computing mode.

Create the /etc/modprobe.d/nvidia-lkca.conf file with the following line:

install nvidia /sbin/modprobe ecdsa_generic; /sbin/modprobe ecdh; /sbin/modprobe --ignore-install nvidia

Update the initramfs and reboot:
```
$ sudo update-initramfs -u
$ sudo reboot
```

Installing the NVIDIA Driver and CUDA Toolkit

NVIDIA recommends that you install the driver by using the package manager. The open GPU kernel modules are the open-source version of the NVIDIA kernel driver. The source code is available in the NVIDIA Linux Open GPU Kernel Module repository.

Install the CUDA repository keyring:

$ wget https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2404/x86_64/cuda-keyring_1.1-1_all.deb
$ sudo dpkg -i cuda-keyring_1.1-1_all.deb
$ sudo apt-get update

Install the CUDA Toolkit:

$ sudo apt-get -y install cuda-toolkit-13-1

Install the driver:
```
$ sudo apt install nvidia-driver-open
```

Starting Fabric Manager for Hopper PPCIe Mode

If the GPUs use multi-GPU PPCIe mode, start Fabric Manager in the guest and confirm that Fabric Manager is running:

$ sudo systemctl enable nvidia-fabricmanager
$ sudo systemctl start nvidia-fabricmanager
$ systemctl status nvidia-fabricmanager

Enabling Persistence Mode

When the NVIDIA driver loads, the driver establishes a secure SPDM session with the GPU and exchanges ephemeral encryption keys with the GPU.

When no process uses the GPU, the NVIDIA kernel driver normally tears down the device state. In the Confidential Computing modes, the teardown destroys the shared secret and the keys from the SPDM session. For security, the GPU doesn’t allow a new SPDM session until a function-level reset (FLR) resets and scrubs the GPU.

Persistence mode prevents the NVIDIA kernel driver from tearing down the device state. The nvidia-persistenced daemon provides persistence mode. The daemon doesn’t use device resources and only holds a reference to the NVIDIA device state.

Determine whether nvidia-persistenced runs in persistence mode:

$ ps -aux | grep nvidia-persistenced

If the output includes --uvm-persistence-mode, nvidia-persistenced runs in persistence mode, and you can skip to Validating the GPU State:

nvidia-+     797  0.0  0.0   5472  1852 ?        Ss   17:23   0:00 /usr/bin/nvidia-persistenced --user nvidia-persistenced --uvm-persistence-mode --verbose

If the output includes --no-persistence-mode, nvidia-persistenced runs without persistence mode:

nvidia-+     797  0.0  0.0   5472  1852 ?        Ss   17:23   0:00 /usr/bin/nvidia-persistenced --user nvidia-persistenced --no-persistence-mode --verbose

If the output shows only the grep command, nvidia-persistenced isn’t running:

user       25944  0.0  0.0   4032  2180 pts/0    S+   18:52   0:00 grep --color=auto nvidia-persistenced

Edit /usr/lib/systemd/system/nvidia-persistenced.service and change --no-persistence-mode to --uvm-persistence-mode in the ExecStart line:
```
ExecStart=/usr/bin/nvidia-persistenced --user nvidia-persistenced --uvm-persistence-mode --verbose
```

Reload the systemd configuration, enable the nvidia-persistenced service, and reboot the guest:

$ sudo systemctl daemon-reload
$ sudo systemctl enable nvidia-persistenced.service
$ sudo reboot

Validating the GPU State

With the driver in persistence mode, confirm that the GPU runs in the Confidential Computing mode that you expect. The nvidia-smi command reports CC mode and PPCIe mode differently. The following example shows a GPU in PPCIe mode on an Intel TDX system:

$ nvidia-smi conf-compute -q
==============NVSMI CONF-COMPUTE LOG==============
    CC State                   : OFF
    Multi-GPU Mode             : Protected PCIe
    CPU CC Capabilities        : INTEL TDX
    GPU CC Capabilities        : CC Capable
    CC GPUs Ready State        : Not Ready
$ nvidia-smi conf-compute -f
CC status: OFF
$ nvidia-smi conf-compute -d
DevTools Mode: OFF
$ nvidia-smi conf-compute -mgm
Multi-GPU Mode: Protected PCIe

For multi-GPU modes, also confirm that Fabric Manager runs, that NVIDIA NVLink is up, and that NVLink peer-to-peer access is enabled:

$ nvidia-smi topo -m
        GPU0  GPU1  GPU2  GPU3  GPU4  GPU5  GPU6  GPU7  CPU Affinity  NUMA Affinity  GPU NUMA ID
GPU0     X    NV18  NV18  NV18  NV18  NV18  NV18  NV18  0-15          0              N/A
GPU1    NV18   X    NV18  NV18  NV18  NV18  NV18  NV18  0-15          0              N/A
GPU2    NV18  NV18   X    NV18  NV18  NV18  NV18  NV18  0-15          0              N/A
GPU3    NV18  NV18  NV18   X    NV18  NV18  NV18  NV18  0-15          0              N/A
GPU4    NV18  NV18  NV18  NV18   X    NV18  NV18  NV18  0-15          0              N/A
GPU5    NV18  NV18  NV18  NV18  NV18   X    NV18  NV18  0-15          0              N/A
GPU6    NV18  NV18  NV18  NV18  NV18  NV18   X    NV18  0-15          0              N/A
GPU7    NV18  NV18  NV18  NV18  NV18  NV18  NV18   X    0-15          0              N/A
$ nvidia-smi topo -p2p n
        GPU0  GPU1  GPU2  GPU3  GPU4  GPU5  GPU6  GPU7
GPU0     X    OK    OK    OK    OK    OK    OK    OK
GPU1    OK     X    OK    OK    OK    OK    OK    OK
GPU2    OK    OK     X    OK    OK    OK    OK    OK
GPU3    OK    OK    OK     X    OK    OK    OK    OK
GPU4    OK    OK    OK    OK     X    OK    OK    OK
GPU5    OK    OK    OK    OK    OK     X    OK    OK
GPU6    OK    OK    OK    OK    OK    OK     X    OK
GPU7    OK    OK    OK    OK    OK    OK    OK     X

The guest CVM is now configured for CC mode or PPCIe mode. After you install the driver, maintain the following requirements:

Keep persistence mode enabled for the NVIDIA driver.
After the driver unloads, the driver doesn’t load again until the GPU is reset. Reset the GPU with a physical function-level reset (PF-FLR) or reboot the system before you load the driver again.

Continue with Attesting the GPU.