Using the Amazon Elastic File System (EFS) for Persistent Data Storage

You can create an elastic file system (EFS)  from the AWS Console. EFS is suggested since a single EFS volume can be simultaneously accessed by multiple concurrent instances, and can be expanded beyond the initial size.

Note: For working with a large number of small files, such as in a dataset, elastic block storage (EBS) offers better performance. For instructions on setting up and using EBS, see the section Using the Amazon Elastic Block Storage (EBS) for Persistent Data Storage.

EFS is available in most regions with Amazon EC2 P3 instances.

Creating an EFS

  1. Open the EFS Console. Go to the main AWS console and select EFS under the "Storage" section.
  2. Click Create file system.
  3. At the Configure file system access page, make sure the Security group that you created with NFS access is selected, then click Next Step.
  4. Configure optional settings:
    1. Enter a value for the Name key

      This is the name of your EFS.

    2. Under Performance mode, select General purpose

      The default setting will work for many workloads. You can experiment with other settings that are better suited to your workload.

    3. Encryption is not needed if you are working with public datasets. For private datasets it's optional and involves additional setup steps. Read the AWS documentation if you need encryption.
    4. Click Next step.
  5. Review the options and then click Create File System.

Mounting an EFS

  1. Open the EFS Console

    Go to the main AWS console and select EFS under the "Storage" section.

  2. Select your EFS to expand its details.
  3. Click Amazon EC2 mount instructions.

    The console displays instructions specific to the selected EFS file system. The nfs-common package is already installed in the NVIDIA AMI, so only the mount step is required. The snippet that AWS shows assumes mounting to an "efs" directory, but the other values including DNS are correct.

    There are several documents linked from the mounting instructions explaining advanced mounting options and troubleshooting.

    To mount your EFS automatically every time you stop and then start the instance, you  need to edit the /etc/fstab file that is provided in the instance. Refer to the Amazon documentation Mounting Automatically

Deleting an EFS

Be aware that once you delete an EFS, you cannot undelete it.

  1. Open the EFS Console.

    Go to the main AWS console and select EFS under the "Storage" section.

  2. Select your EFS.
  3. Using the "Actions" dropdown, select Delete file system.

  4. In the confirmation dialog, enter the file system ID, and then click Delete File System.

Managing an EFS Using AWS CLI

It is recommended that you use the AWS Console for EFS management. If you need to manage EFS file systems with the CLI, NVIDIA has created scripts available on GitHub at https://github.com/nvidia/ngc-examples.

These scripts will let you perform basic EFS management and can serve as the basis for further automation.