Using File Storage for Datasets

You can create persistent file storage from the Azure Dashboard. File storage is useful for sharing datasets across multiple VMs.

Note: For working with a large number of small files, such as in a dataset, Premium Storage SSDs offer better performance. For instructions on setting up and using Premium Storage SSDs, see the section Using Premium Storage SSD for Datasets.

Creating a Storage Account

You need a storage account to set up your file storage.
  1. Log in to the Azure portal.
  2. Click All services from the left side menu, then click Storage accounts (under the STORAGE section).

  3. Click Add.
  4. Enter information as follows:
    • Name: enter any name.
    • Deployment model: Resource Manager
    • Account kind: Select based on your storage needs and cost.
    • Select the subscription in which you want to create the new storage account.
    • Resource group: Use existing
    • Select the geographic region.
  5. Click Create to create the storage account.
When you create a storage account, Azure generates two 512-bit storage access keys, which are used for authentication when the storage account is accessed.

Creating File Storage

  1. Click Storage accounts from the left side menu, then click your storage account.
  2. Under Services, click Files.
  3. Click +File share, then enter a name for you storage and required quota and click OK.
  4. Note the keys and use it to mount.
  5. Ensure that the region selected above is same as the location of your GPU Cloud VM.

Mounting the File Storage

  1. Once file storage created, establish an SSH connection to your VM.
  2. Mount the share to the local directory (pass your StorageAccount and key as passwords).
    sudo mount -t cifs \
     //myCifsAccount.file.core.windows.net/cifsShare01 /cifs \
     -o vers=3.0,username=myCifsAccount,password=mystorageaccountkey,\
     dir_mode=0777,file_mode=0777 

    or, for higher performance, use the following.

    sudo mount -t cifs \
     //myCifsAccount.file.core.windows.net/cifsShare01 /cifs \
     -o vers=2.1,username=myCifsAccount,password=mystorageaccountkey,\
     dir_mode=0777,file_mode=0777,cache=strict 
  3. [Optional] To persist the mount through reboots, an entry to /etc/fstab.
    //myCifsAccount.file.core.windows.net/cifsShare01 /mymountpoint cifs vers=3.0,username=myCifsAccount,password=StorageAccountKeyEndingIn==,dir_mode=0777,file_mode=0777
    

    or, for higher performance, use the following.

    //myCifsAccount.file.core.windows.net/cifsShare01 /mymountpoint cifs vers=2.1,username=myCifsAccount,password=StorageAccountKeyEndingIn==,dir_mode=0777,file_mode=0777,cache=strict
  4. Copy the dataset over.
    scp -r local_dataset_dir/ <username>@<Azure_VM_Instance>:/data
The Storage account and mount should be in the same region.

Deleting File Storage

Be aware that once you delete your File Storage, you cannot undelete it.

  1. Open the Azure Dashboard and select Storage accounts from the left side menu.
  2. Locate and click your storage account.
  3. Click Files from the Services section.

  4. From the list, click the file share to delete.

  5. Click Delete share from the top menu.

  6. Follow the instructions in the Delete file share pane and then click OK.