6. Storage User Guide#

DGX Cloud Create has a storage control feature currently designed to provide these functions:

Storage class encapsulation:
- a standard set of storage classes to encapsulate shared storage across the CSP storage variants, allowing customers to clearly understand their quota management
Support for object storage mounts:
- an API for mounting object storage into customer workloads
Volume protection:
- an API for the retention and deletion of provisioned storage volumes

6.1. Storage Classes#

The storage control feature provides the following three custom storage classes:

dgxc-enterprise-file: the highest performance shared filesystem
dgxc-standard-file: a lower performance filesystem that supports smaller volumes
- This will be optional for CSPs where the enterprise filesystem already supports smaller volume sizes. It is currently not applicable for AWS.
dgxc-standard-object: POSIX mounted object storage

DGXC Storage Class	CSP	CSP Driver
dgxc-enterprise-file	AWS GCP	FSX for Lustre Filestore (zonal-rwx)
dgxc-standard-file	AWS GCP	not applicable Filestore (basic-hdd)
dgxc-standard-object	AWS GCP	AWS Mountpoint GCS FUSE

6.1.1. Enterprise Storage Profiles#

These are the CSP storage classes that are encapsulated by dgxc-enterprise-file.

AWS

AWS-based clusters leverage shared storage provided by FSx Lustre.

Service Tier	Storage Class	PVC Size	Deployment	Access Mode
FSx Lustre (SSD)	lustre-sc	12-160 TiB	PERSISTENT 2	read/write/many

All other PVC storage classes (none, gp2, ebs) are unsupported.

GCP

GCP-based clusters leverage shared storage provided by Google Filestore with the following capabilities.

Service Tier	GKE Storage Class	GKE PVC Size	Deployment	Access Mode
Zonal	zonal-rwx	10-100 TiB	Zonal	read/write/many

All other PVC storage classes (none, standard, standard-rwo, standard-rwx, premium-rwo, premium-rwx, enterprise-rwx, enterprise-multishare-rwx) are unsupported.

6.2. Object Storage#

Integrated object storage can be provisioned for both AWS and GCP. In both cases, you will first need the OIDC URL for the cluster. This should be provided to you by your TAM as part of onboarding.

Both AWS Mountpoint and GCP Google Cloud Storage are POSIX compatible object storage interfaces. However, it is important to understand differences to non-object filesystems as they relate to the creation, modification, and deletion of files as objects. Also, in general, object storage has performance characteristics that are not well-suited to many small files but instead is optimal for fewer large files such as datasets.

Note

The following sets of instructions for working with the dgxc-standard-object storage class are specific to AWS and GCP respectively. They use different methods for IAM and OIDC configuration. Also, you will be using multiple CLI:

the CSP CLI (aws and gcloud) to interact with your CSP account
kubectl to interact with the DGX Cloud Create Kubernetes cluster

6.2.1. Creating a New AWS Object Storage Resource#

The following steps for creating a new AWS object resource in DGX Cloud Create involve multiple interactions with the AWS console, the AWS CLI, and kubectl in the DGX Cloud Create cluster.

The first step is to create a new bucket if necessary in the AWS console at https://us-east-1.console.aws.amazon.com/s3 Obviously this is dependent on the region where you want to provision the S3 bucket so update the URL or change region within the console after logging in. In the following instructions, we will use a $BUCKET_NAME of “my-s3-bucket-$ACCOUNT_ID”.

Note

S3 bucket names exist at a global scope so it is important to ensure that the name of your bucket is reasonably unique. Appending the $ACCOUNT_ID helps provide a unique name.

The next step is to create an OIDC identity provider in IAM at https://us-east-1.console.aws.amazon.com/s3 Again, set the region as appropriate.

The details of this will be the following:

Provider type: OpenID Connect
Provider URL: <the OIDC URL from your TAM>
Audience: sts.amazonaws.com

Next, use the following to capture your AWS account ID to a variable.

ACCOUNT_ID=$(aws sts get-caller-identity --query "Account" --output text)
echo "ACCOUNT ID: $ACCOUNT_ID"

Once the S3 bucket and OIDC identity provider are created, we will proceed with setting up Bash environment variables specific to the AWS-related requirements. Specify a different $BUCKET_NAME if you used a custom name in the AWS console, ensuring it is globally unique.

BUCKET_NAME="my-s3-bucket-$ACCOUNT_ID"
S3_REGION="<Customer S3 region; for example: us-east-1>"
POLICY_NAME="s3-example-policy"
ROLE_NAME="s3-example-role"
ROLE_DESCRIPTION="Role for testing AWS S3"
OIDC_URL="<OIDC URL provided by your TAM>"

Note that the $S3_REGION will depend on where you provisioned your bucket.

We will create a new project in the NVIDIA Run:ai UI that will reflect the Kubernetes namespace for this example. Again, NVIDIA Run:ai by default will prepend this project asset with runai- so the project s3-example in NVIDIA Run:ai will become the Kubernetes namespace runai-s3-example.

Next we define some variables for the Kubernetes-related resources that will be used later. Specify a different $BUCKET_NAME if you used a custom name in the AWS console.

NAMESPACE="runai-s3-example"
DATA_SOURCE_NAME="s3-example-data-source-1"
PVC_NAME="s3-example-pvc-1"
MOUNT_PATH="/mnt/s3_example"
SERVICE_ACCOUNT="default"

Now we will create a JSON file that defines an AWS Security Token Service (STS) role policy using the OIDC URL provided by your TAM. Note the use of Bash variable manipulation to omit the scheme from the URL. Then we create the role using the AWS CLI.

cat >aws-role.json <<EOF
{
    "Version": "2012-10-17",
    "Statement": [
     {
     "Effect": "Allow",
     "Principal": {
       "Federated": "arn:aws:iam::$ACCOUNT_ID:oidc-provider/${OIDC_URL#https://}"
     },
     "Action": "sts:AssumeRoleWithWebIdentity",
     "Condition": {
       "StringLike": {
          "${OIDC_URL#https://}:aud": "sts.amazonaws.com",
          "${OIDC_URL#https://}:sub": "system:serviceaccount:$NAMESPACE:$SERVICE_ACCOUNT"
          }
        }
     }
    ]
}
EOF

aws iam create-role --role-name "$ROLE_NAME"\
    --assume-role-policy-document file://aws-role.json \
    --description "$ROLE_DESCRIPTION"

Now we define and create the AWS policy we want for accessing the S3 bucket in another JSON file.

cat >aws-policy.json <<EOF
{
   "Version": "2012-10-17",
   "Statement": [
   {
   "Sid": "AllowS3ReadWrite",
   "Effect": "Allow",
   "Action": [
     "s3:ListBucket",
     "s3:GetBucketLocation"
     ],
   "Resource": "arn:aws:s3:::$BUCKET_NAME"
   },
   {
   "Effect": "Allow",
   "Action": [
      "s3:PutObject",
      "s3:GetObject",
      "s3:DeleteObject"
     ],
   "Resource": "arn:aws:s3:::$BUCKET_NAME/*"
   }
  ]
}
EOF

aws iam create-policy --policy-name "$POLICY_NAME" \
    --policy-document file://aws-policy.json

Now that we have created both a role and bucket access policy we will attach the policy to that role.

aws iam attach-role-policy --policy-arn arn:aws:iam::"$ACCOUNT_ID":policy/"$POLICY_NAME" \
    --role-name "$ROLE_NAME"

Kubernetes has the concept of service accounts as resources which are essentially non-human identities that can interact with the API server and other resources. Every namespace (NVIDIA Run:ai project) is provided a default service account which is used by pods and other resources if no other service account is specified. Here we will annotate the default service account with the AWS account and role that we created so that the pod that uses the PVC Data Source can access the bucket.

kubectl annotate sa default -n "$NAMESPACE" \
       eks.amazonaws.com/role-arn=arn:aws:iam::"$ACCOUNT_ID":role/"$ROLE_NAME"

Finally, we can define and create a NvStorageLocation resource that binds the bucket to a PVC Data Source. Note that the NvStorageLocation spec currently requires some arbitrary capacity that is unrelated to the size of the object storage.

Here is a list valid mount options for AWS Mountpoint.

Option	Description
–read-only	Mount file system in read-only mode
–allow-delete	Allow delete operations on file system
–allow-overwrite	Allow overwrite operations on file system
–auto-unmount	Automatically unmount on exit
–allow-root	Allow root user to access file system
–allow-other	Allow other users, including root, to access file system
–uid <UID>	Owner UID [default: current user’s UID]
–gid <GID>	Owner GID [default: current user’s GID]
–dir-mode <DIR_MODE>	Directory permissions [default: 0755]
–file-mode <FILE_MODE>	File permissions [default: 0644]

cat >aws-object-data-source.yaml <<EOF
apiVersion: "storage.dgxc.nvidia.com/v1beta1"
kind: NvStorageLocation
metadata:
  name: "$DATA_SOURCE_NAME"
  namespace: "$NAMESPACE"
spec:
  description: "An example NvStorageLocation for an AWS bucket"
  volumeName: "$PVC_NAME"
  mountPath: "$MOUNT_PATH"
  pvc:
    objectSpec:
      endpointUrl: "https://s3.$S3_REGION.amazonaws.com"
      bucket: "$BUCKET_NAME"
    mountOptions:
    # Set a user and group ID if desired
    - uid=1000
    - gid=1000
    # The following is an example of a mount option we might choose to apply
    # allowing other users access to the filesystem
    - allow-other
    # Currently some capacity needs to be specified
    capacity: "1Gi"
EOF

kubectl apply -f aws-object-data-source.yaml

6.2.1.1. Multiple Pods and S3 Bucket Combinations for AWS#

In the case that one pod needs to access two or more S3 buckets, we can add multiple buckets like we defined in the file aws-policy.json and then attach the single policy to the role. In the case that an S3 bucket needs to be accessed by different pods in different namespaces, we need to provide an IAM role like the one in aws-role.json for different namespaces (and possibly service accounts if not using default).

6.2.1.2. Mount Semantics for AWS S3#

AWS Mountpoint supports creating new objects in your S3 bucket by allowing writes to new files. If the --allow-overwrite flag is set at startup time, Mountpoint also supports replacing existing objects by allowing writes to existing files, but only when the O_TRUNC flag is used at open time to truncate the existing file. In both cases, writes must always start from the beginning of the file and must be made sequentially.

Mountpoint allows creating new directories with commands like mkdir. Creating a new directory is a local operation and no changes are made to your S3 bucket. A new directory will only be visible to other clients once a file has been written and uploaded inside it. You cannot remove or rename an existing directory with Mountpoint. However, you can remove a new directory created locally if no files have been written inside it. Mountpoint does not support hard or symbolic links.

Here are more detailed semantics for Mountpoint.

6.2.2. Creating a New GCP GCS Object Storage Resource#

Navigate to https://console.cloud.google.com/storage/browser in your GCP project and create a new bucket called gcs-example-bucket. Once the GCS bucket is created, we will proceed with setting up some bash environment variables for the specifics of the GCP-related variables we require. Specify a different $BUCKET_NAME if you used a custom name in the console.

BUCKET_NAME="gcs-example-bucket"
PROJ_ID="<Customer-owned Project ID>"
PROJ_NUM="<Customer-owned Project Number>"
PROJ_REGION="<Customer GCS region; for example: us-east5-a>"
ID_POOL="gcs-example-oidc-pool"
ID_POOL_PROV="${ID_POOL}-provider"
ID_ROLE_DESC="A workload identity pool for GCS"
OIDC_URL="<The OIDC URL provided by your TAM>"

We will create a new project in the NVIDIA Run:ai UI that will reflect the Kubernetes namespace for this example. Again, NVIDIA Run:ai by default will prepend this project asset with runai- so the project gcs-example in NVIDIA Run:ai will become the Kubernetes namespace runai-gcs-example.

Next we define some variables for the Kubernetes-related resources that will be used later.

NAMESPACE="runai-gcs-example"
DATA_SOURCE_NAME="gcs-example-data-source-1"
PVC_NAME="gcs-example-pvc-1"
MOUNT_PATH="/mnt/gcs_example"
SERVICE_ACCOUNT="default"

Use the gcloud CLI to authenticate then set your project for the subsequent instructions.

gcloud config set project $PROJ_ID

Next we will create a workload identity pool and an OIDC provider for that pool using the OIDC URL provided to you by your TAM as part of cluster onboarding.

gcloud iam workload-identity-pools create "$ID_POOL" \
       --location="global" --description="$ID_POOL_DESC" \
       --display-name="$ID_POOL"

gcloud iam workload-identity-pools providers create-oidc "$ID_POOL_PROV" \
       --location="global" --workload-identity-pool="$ID_POOL" \
       --issuer-uri="$OIDC_URL" --attribute-mapping="google.subject=assertion.sub"

Now we will create a Kubernetes config map from the workload identity pool credentials. This config map will be used by the NvStorageLocation resource we define later.

gcloud iam workload-identity-pools \
       create-cred-config projects/$PROJ_NUM/locations/global/workloadIdentityPools/$ID_POOL/providers/$ID_POOL_PROV \
       --credential-source-file=/var/run/service-account/token \
       --credential-source-type=text --output-file=/tmp/credential-configuration.json

kubectl create configmap $CONFIG_MAP --from-file /tmp/credential-configuration.json \
        --namespace $NAMESPACE

Now we will bind the Kubernetes service account for the pod that will load the PVC to the GCS bucket previously created.

gcloud storage buckets add-iam-policy-binding gs://$BUCKET_NAME  \
       --member "principal://iam.googleapis.com/projects/$PROJECT_NUM/locations/global/workloadIdentityPools/$ID_POOL/subject/system:serviceaccount:$NAMESPACE:$SERVICE_ACCOUNT  \
       --role $ROLE_NAME

Finally, we will create a NvStorageLocation resource using the following YAML in the Kubernetes cluster. Note that the NvStorageLocation spec currently requires some arbitrary capacity that is unrelated to the size of the object storage.

Google has a variety of FUSE mount options for GCS listed here.

cat >gcp-object-data-source.yaml <<EOF
apiVersion: 'storage.dgxc.nvidia.com/v1beta1'
kind: NvStorageLocation
metadata:
  name: "$DATA_SOURCE_NAME"
  namespace: "$NAMESPACE"
spec:
  description: "An example NvStorageLocation for a GCP bucket"
  mountPath: "$MOUNT_PATH"
pvc:
  # Currently, some capacity needs to be specified
  capacity: "5Gi"
  mountOptions:
    # Set user and group ID as desired
    - uid=1001
    - gid=1001
    # Here we might employ yet another FUSE mount option
    #- only-dir=<PrefixName>
  objectSpec:
    bucket: "$BUCKET_NAME"
    endpointUrl: gcp
  parameters:
    # Needed for OIDC workload identity federation
    gcs-oidc-configmap: "$CONFIG_MAP"
volumeName: "$PVC_NAME"
EOF

kubectl apply -f gcp-object-data-source.yaml

6.2.2.1. Multiple Pods and Bucket Combinations for GCP GCS#

You can import multiple GCS buckets in the same workload. The workload identity pool creation steps do not need to be repeated for multiple buckets. You should create just one workload identity pool. Then you would update the IAM configuration of each GCS bucket to assign the appropriate role to the IAM principal corresponding to the workload identity pool (as described above).

Once the IAM policy of all the buckets are updated, users can then create multiple NVStorageLocation resources, corresponding to each GCS bucket. Note that the bucket details in the NVStorageLocation would be different for different buckets however the gcs-oidc-configmap parameter should be the same in all those NVStorageLocation objects. These NVStorageLocation objects would result in the creation of multiple “Data Sources” in the Run:Ai UI by which you can then associate these data sources to various workloads.

6.2.2.2. Mount Semantics for GCS#

Google Cloud Storage FUSE downloads the entire backing object’s contents from Cloud Storage for modifications to existing file objects. The contents are stored in a local temporary file whose location is controlled by the flag --temp-dir. When the file is closed, Cloud Storage FUSE writes the contents of the local file back to Cloud Storage as a new object generation, and deletes the temporary file. The modification of even a single bit of an object results in the full re-upload of the object. The exception is if an append is done to the end of a file, where the original file is at least 2MB, then only the appended content is uploaded.

Objects are first written to the same temporary directory as mentioned previously for new file creation. Upon closing the file, it is then written to the Cloud Storage bucket. Since new and modified files are fully staged in the local temporary directory until they are written out to Cloud Storage, it is important to ensure that there is enough free space available to accommodate staged content when writing large files.

Here are more detailed semantics for FUSE.

6.2.3. Destroying an Object Storage Resource#

The previously created NvStorageLocation resources can be destroyed by using the same YAML to delete it from the cluster.

# for the AWS resource
kubectl delete -f aws-object-data-source.yaml
# or for the GCP resource
kubectl delete -f gcp-object-data-source.yaml

This will not destroy the CSP bucket itself but instead remove the PVC for it.

6.3. Volume Protection#

In the interest of data protection, DGX Cloud Create modifies any PersistentVolume request by default to have a retention policy of “Retain”, regardless of whether the underlying StorageClass has a default policy of “Delete”. This effectively means that all volumes will remain even after being released by a pod. This ensures that important data will not be inadvertently lost due to default Kubernetes cluster storage policies. However, it also means that these volumes can accumulate over time and continue to count against your CSP project storage quota.

Note

The following steps require the privileges described at Advanced Kubernetes Usage for Admins.

You can use the same RunaiDGXCStorage resource mentioned in Managing Your Storage Utilization (CLI) to change the retention policy of any volume. After you have definitively identified a volume that can be deleted, use the following command:

kubectl edit runaidgxcstorages -n dgxc-tenant-cluster-policies

This command will launch your editor, where you can modify the retention policy for any existing volume. For example:

spec:
  instances:
    pvc-0ffa54ec-049b-4ad8-847e-8476b44e18ca:
      name: pvc-0ffa54ec-049b-4ad8-847e-8476b44e18ca
      persistentVolumeReclaimPolicy: Retain

This could be changed to:

spec:
  instances:
    pvc-0ffa54ec-049b-4ad8-847e-8476b44e18ca:
      name: pvc-0ffa54ec-049b-4ad8-847e-8476b44e18ca
      persistentVolumeReclaimPolicy: Delete

Once you save your changes, the volume will be deleted when it is no longer used. Note that a PersistentVolume will not be deleted until all its pod references are gone. In other words, there are no more pods in the cluster referring to the volume claim.