Creating Persistent SSD Disks for Datasets

GCP recommends using Persistent SSD Disks for Compute Engine storage. A minimum of 1 TB of storage is recommended for storing deep learning datasets. However, a much larger disk or a software RAID, using mdadm, can be used to create a volume with multiple SSD Persistent Disks for achieving the the maximum performance supported by GCP on a Compute Engine instance. See instructions on how to set up software RAID on local disks. Persistent SSD disks can also be set up for software RAID using the same instructions.

Creating a Persistent SSD Dataset Disk and Attaching it to the VM Instance from the GCP Console

You can create a persistent SSD dataset disk from the GCP console as follows.
  1. Log on to the Google Cloud Platform.
  2. Create the SSD disk.
    1. Click Compute Engine-> Disks in the left-side navigation pane.
    2. Click Create Disk from the top of the page.

    3. Specify the following:
      • Zone: Select the same zone as the VM instance you created.
      • Disk Type: SSD persistent disk
      • Source type: None (blank disk)
      • Size: At least 1024 GB

      If you choose to provide your own encryption key, You must provide a key that is a 256-bit string encoded in RFC 4648 standard base64 to Compute Engine. See Customer-Supplied-Encryption-Keys for details on how to provide a custom Encryption Key globally for all your operations.
    4. Click Create when done.
  3. Attach the disk to the VM instance.
    1. Go to the Compute Engine->VM Instance page.
    2. Click your VM instance from the list.
    3. Click Stop.

      You must stop a running VM Instance as changes cannot be performed when the instance is running.

    4. Click Edit.
    5. Scroll down to the Additional Disks and click + Add Item.
    6. Under Name, select the disk that you created and want to attach to the VM instance.
    7. Click Save.
    8. Start the VM instance.
Refer to the section Creating a Filesystem and Mounting the Volume for instructions on formatting and mounting the drive for uploading datasets.

Creating a Persistent SSD Dataset Disk and Attaching it to the VM Instance from the gcloud CLI

  1. Create the disk using the following command.
    $ gcloud compute disks create ngc-ssd --zone <zone> --description “<your-description>” --type=pd-ssd --size=1000GM
  2. Attach the disk to a VM instance using the following command..
    $ gcloud compute instances attach-disk <instance-name> --disk ngc-ssd --zone <zone>

Creating a File System and Mounting the Volume

Refer to the Formatting and mounting a persistent disk section of the Google Cloud how-to guide Adding or Resizing Persistent Disks for instructions on creating a file system and mounting the volume.

Deleting a Persistent SSD Disk

Using gcloud CLI

$ gcloud compute instances detach-disk <instance-name> --disk ngc-ssd --zone <zone>

Using the GCP Console

Be aware that once you delete a Persistent SSD Disk, you cannot undelete it.

  1. Click the disk to delete from the Compute Engine->Disks page.
  2. On the top of the page, click Delete.

  3. Click Delete at the Delete a disk confirmation dialog. .