Managing the DGX A100 Self-Encrypting Drives
The NVIDIA DGX OS software supports the ability to manage self-encrypting drives (SEDs), including setting an Authentication Key for locking and unlocking the drives on NVIDIA DGX™ A100 systems. You can manage only the SED data drives.
The software cannot be used to manage OS drives even if they are SED-capable.
Overview
The SED management software is in the nv-disk-encrypt
package.
The software supports the following configurations:
NVIDIA DGX A100 systems where all data drives are self-encrypting drives.
Only SEDs used as data drives are supported.
The software will not manage SEDs that are OS drives.
The software provides the following functionality:
Identifies eligible drives on the system.
Allows you to you assign Authentication Keys (passwords) for each SED as part of the initialization process.
Alternatively, the software can generate random passwords for each drive.
The passwords are stored in a password-protected vault on the system.
Once initialized, SEDs are locked upon power loss, such as a system shutdown or drive removal.
Locked drives get unlocked after power is restored and the root file system is mounted.
Provides functionality to export the vault.
Provides functionality for erasing the drives.
Provides the ability to revert the initialization.
Installing the Software
Use the package manager to install the nv-disk-encrypt
package and, optionally, the TPM2 tools package, and reboot the system. You need the TPM tools package if you plan to use the TPM2 to store security keys.
Update the packages.
$ sudo apt update
Install
nv-disk-encrypt
.$ sudo apt install -y nv-disk-encrypt
(Optional) Install
tpm2-tools
.$ sudo apt install -y tpm2-tools
Reboot.
$ sudo reboot
If you plan to use TPM2, enable it. Refer to “Configuring Trusted Computing” for more information.
Configuring Trusted Computing
Here is some information about the controls that are required to configure Trusted Computing (TC).
The DGX A100 system BIOS provides setup controls for configuring the following TC features:
Trusted Platform Module
The NVIDIA DGX A100 incorporates Trusted Platform Module 2.0 (TPM 2.0) which can be enabled from the system BIOS and used in conjunction with the nv-disk-encrypt tool. After being enabled, the
nv-disk-encrypt
tool uses the TPM for encryption and stores the vault and SED authentication keys on the TPM instead of on the file system. Using the TPM is preferred because this allows the vault data to persist even if the system is reimaged.Block SID
Certain drives shipped with the DGX A100 system might support the Block SID authentication feature. Block SID authentication prevents malicious actors from taking ownership of drives and blocks others from using the drives. By default, the DGX BIOS will send the Block SID request. On such setups, you will need to enable the Disable Block Sid feature in the BIOS before proceeding with the initialization steps.
Determining Whether Drives Support SID
The drive model is a good indicator of whether the drive supports this feature. Issue the following and look for the KCM6DRUL3T84
model string:
$ sudo nvme list
Node SN Model ...
---------------- -------------------- ---------------------
/dev/nvme0n1 70H0A0AHTTHR KCM6DRUL3T84 ...
/dev/nvme1n1 70H0A007TTHR KCM6DRUL3T84
Enabling the TPM and Preventing the BIOS from Sending Block SID Requests
This section provides instructions to enable the TPM and prevent the SBIOS from sending Block SID request. Each task is independent, so you can select which task to complete.
Reboot the DGX A100, then press [Del] or [F2] at the NVIDIA splash screen to enter the BIOS Setup.
Navigate to the Advanced tab on the top menu, then scroll to Trusted Computing and press [Enter].
To enable TPM, scroll to Security Device and switch the setting to Enabled.
To disable Block SID, scroll to Disable Block Sid, then switch to Enabled.
Save and exit the BIOS Setup to continue the boot process.
If you disabled Block SID, you will be prompted to accept the request to disable issuing a Block SID Authentication command.
Press F10 at the prompt.
After the system boots, you can proceed to initialize drive encryption.
Initializing the System for Drive Encryption
Here is some information about how to initialize the system for drive encryption.
Note
Before initializing drive encryption, review the information in Configuring Trusted Computing and follow the configuration instructions if needed.
Initialize the system for drive encryption using the nv-disk-encrypt command.
$ sudo nv-disk-encrypt init [-k <your-vault-password>] [-f <path/to/json-file>] [-g] [-r]
Here is a list of the options:
-k: Lets you create the vault password in the command.
Otherwise, the software will prompt you to create a password before proceeding.
-f: Lets you specify a JSON file that contains a mapping of passwords to drives.
Refer to Example 1: Passing in the JSON File for further instructions.
-g: Generates random salt values (stored in
/etc/nv-disk-encrypt/.dgxenc.salt
) for each drive password.Salt values are characters added to a password for enhanced password security. NVIDIA strongly recommends using this option for best security, otherwise the software will use a default salt value instead of a randomly generated one.
-r: Generates random passwords for each drive.
This avoids the need to create a JSON file or the need to enter a password one by one during the initialization.
Enabling Drive Locking
After initializing the system for SED management, issue the following command, which uses the nv-disk-encrypt
command to enable drive locking.
$ sudo nv-disk-encrypt lock
After initializing the system and enabling drive locking, the drives will become locked when they lose power. The system will automatically unlock each drive when power is restored to the system and the system is rebooted.
Initialization Examples
Example 1: Passing in the JSON File
The following instructions in this section describe a method to specify the drive/password mapping ahead of time. This method is useful for initializing several drives at a time and avoids the need to enter the password for each drive after issuing the initialization command, or if you want control of the passwords.
Determining Which Drives Can be Managed as Self-Encrypting
Here is some information about how you can determine which drives can be managed as self-encrypting.
Review the storage layout of the DGX system to determine which drives are eligible to be managed as SEDs.
$ sudo nv-disk-encrypt info
The default output shows which drives can be used for encryption and which drives cannot. The following status information is provided:
SED capable: Is this a self-encrypting drive?
Boot disk: Is this drive currently being used as a boot drive? Does it contain the root filesystem?
Locked: Is this drive currently in the locked state? Is it able to accept I/O?. It can only be in this state after the following conditions have been met:
Locking has been enabled (nv-disk-encrypt init, followed by
nv-disk-encrypt init
lock)The drive is coming back from power-off.
The user queries this state prior to it being (automatically) unlocked.
Lock Enabled: Are locks enabled on this drive? It will be in this state after initialization (
nv- disk-encrypt init
).MBR done: This setting is only relevant for drives that support MBR shadowing. On drives that support this feature, this will report ‘Y’ after initialization (
nv-disk-encrypt init
)
MBR done: This setting is only relevant for drives that support MBR shadowing. On drives that support this feature, this will report ‘Y’ after initialization (nv-disk-encrypt init
)
The following example output snippet shows drives than can be used for encryption. Notice SED capable = Y and Boot disk = N.
The following example output snippet shows drives than cannot be used for encryption. Notice SED capable = Y and Boot disk = Y, or SED capable = N.
Alternatively, you can specify the output be presented in JSON format by using the -j
option.
$ sudo nv-disk-encrypt info -j
In this case, drives that can be used for encryption are indicate by the following:
"sed_capable": true "used_for_boot": false
And drives that cannot be used for encryption are indicated by one of the following:
"sed_capable": true "used_for_boot": true
Or
"sed_capable": false
Creating the Drive/Password Mapping JSON Files and Using it to Initialize the System
You can initialize the system by creating the drive and password map the JSON files.
Create a JSON file that lists all the eligible SED-capable drives that you want to manage.
Note
These are the list of drives that you obtained from Determining Which Drives Can be Managed as Self-Encrypting.
The following example shows the format of the JSON file.
{ "/dev/nvme2n1": "<your-password>", "/dev/nvme3n1": "<your-password>", "/dev/nvme4n1": "<your-password>", "/dev/nvme5n1": "<your-password>", }
Ensure that you follow the syntax exactly.
Passwords must consist of only upper-case letters, lower-case letters, digits, and/or the following special characters: ~ : @ % ^ + = _ ,
Initialize the system and then enable locking.
The following command assumes you have placed the JSON file in the /tmp directory.
$ sudo nv-disk-encrypt init -f /tmp/<your-file>.json -g $ sudo nv-disk-encrypt lock
When prompted, enter a password for the vault.
Passwords must consist of only upper-case letters, lower-case letters, digits, and/or the following special characters: ~ : @ % ^ + = _ ,
Delete the JSON file in the temporary location for security.
Example 2: Generating Random Passwords
The commands in this topic use the -k
and -r
options so that you are not prompted to enter passwords. You pass the vault password into the command and then the command instructs the tool to generate random passwords for each drive.
The vault password must consist of only upper-case letters, lower-case letters, digits, and/or the following special-characters: ~ : @ % ^ + = _ ,
$ sudo nv-disk-encrypt init -k <your-vault-password> -g -r
$ sudo nv-disk-encrypt lock
Example 3: Specifying Passwords One at a Time When Prompted
If there are a small number of drives, or you do not want to create a JSON file, issue the following command.
$ sudo nv-disk-encrypt init -g
$ sudo nv-disk-encrypt lock
The software prompts you to enter a password for the vault and then a password for each eligible SED.
Passwords must consist of only upper-case letters, lower-case letters, digits, and/or the following special characters: ~ : @ % ^ + = _ ,
Disabling Drive Locking
To disable drive locking at any time after you initialize, run the following command: $ sudo nv-disk-encrypt disable
This command disables locking on all drives. You can run the initial set up again at any time after this process is complete.
Enabling Drive Locking
After initializing the system for SED management, issue the following command, which uses the nv-disk-encrypt
command to enable drive locking.
$ sudo nv-disk-encrypt lock
After initializing the system and enabling drive locking, the drives will become locked when they lose power. The system will automatically unlock each drive when power is restored to the system and the system is rebooted.
Exporting the Vault
Here is some information about how to export the vault.
To export all drive keys out to a file, use the export function. This requires that you pass in the vault password.
$ sudo nv-disk-encrypt export -k <your-vault-password>
Writing vault data to /tmp/secrets.out
The /tmp/secrets.out
file contains the mapping of disk serial numbers to drive passwords.
Erasing Your Data
Here is some information about how you can erase your data.
Warning
Be aware when executing this that all data will be lost. On DGX A100 systems, these drives generally form a RAID 0 array, and this array will also be destroyed when you perform an erase.
After initializing the system for SED management, use the nv-disk-encrypt command to erase data on your drives after stopping cachefilesd and unmounting the RAID array as follows.
Completely stop the RAID.
$ systemctl stop cachefilesd $ sudo umount /raid $ sudo mdadm --stop /dev/md1
Perform the erase.
$ sudo nv-disk-encrypt erase
This command does the following:
Sets the drives in an unlocked state.
Disables locking on the drives.
Removes the RAID 0 array configuration.
To rebuild the RAID array, issue the following command:
$ sudo /usr/bin/configure_raid_array.py -c -f
Clearing the TPM
If you’ve lost the password to your TPM, you will not be able to access its contents. In this case, the only way to regain access to the TPM is to clear the TPM’s contents. After clearing the TPM, you will need to re-initialize the vault and SED authentication keys.
Reboot the DGX A100, then press [Del] or [F2] at the NVIDIA splash screen to enter the BIOS Setup.
Navigate to the Advanced tab on the top menu, scroll to Trusted Computing, and press [Enter].
Clear TPM2.
Scroll to Trusted Computing and press [Enter].
Scroll to Pending Operation and press [Enter].
Select TPM Clear at the Pending Operation popup and press [Enter].
Save and exit the BIOS Setup.
Changing Disk Passwords, Adding Disks, or Replacing Disks
The same steps are needed for changing or rotating passwords, adding disks, or replacing disks.
Context for the current task.
Disable SED management.
$ sudo nv-disk-encrypt disable
Add or replace drives as needed and then rebuild the RAID array.
Refer to the NVIDIA DGX A100 Service Manual for more information.
Enable SED management and assign passwords per the instructions in Initializing the System for Drive Encryption.
Recovering From Lost Keys
NVIDIA recommends backing up your keys and storing them in a secure location. If you’ve lost the key used to initialize and lock your drives, you will not be able to unlock the drive again. If this happens, the only way to recover is to perform a factory-reset, which will result in a loss of data.
SED drives come with a PSID printed on the label; this value can only be obtained by physically examining the drive as exemplified in the following image.
Specify the PSID to reset the drive using the following sedutil-cli command:
$ sudo sedutil-cli --yesIreallywanttoERASEALLmydatausingthePSID <your-drive-psid> /dev/ nvme3n1