GKE(Google Kubernetes Engine)

NVIDIA AI Enterprise 3.1 or later

Google Kubernetes Engine (GKE) provides a managed environment for deploying, managing, and scaling your containerized applications using Google infrastructure. NVIDIA AI Enterprise, the end-to-end software of the NVIDIA AI platform, is supported to run on GKE. The GKE environment consists of multiple machines (specifically, Compute Engine instances) grouped together to form a cluster. This guide provides details for deploying and running NVIDIA AI Enterprise on GKE clusters with NVIDIA GPU Accelerated nodes.

Note

The NVIDIA Terraform Modules offer an easy way to deploy Managed Kubernetes clusters that can be supported by NVIDIA AI Enterprise when used with supported OS and GPU Operator versions.

  • NVIDIA AI Enterprise License via BYOL or a Private Offer

  • Install Helm

  • Install the Google Cloud CLI

    Note

    NVIDIA recommends installing the Google Cloud CLI via download of the linux google-cloud-cli-xxx.x.x-linux-x86_64.tar.gz file. If a different OS distribution is being utilized, please continue to follow Google’s instructions for your desired OS.

    cloud-gke-01.png


  • Google Cloud account with Google Kubernetes Engine Admin Role and Kubernetes Engine Cluster Admin Role for more information please refer to Google’s IAM Policies

  • Ubuntu Nodes

Run the below comannds to install the GKE components.

Copy
Copied!
            

./google-cloud-sdk/bin/gcloud components install beta ./google-cloud-sdk/bin/gcloud components install kubectl ./google-cloud-sdk/bin/gcloud components update

Run the below command to create a GKE cluster.

Copy
Copied!
            

./google-cloud-sdk/bin/gcloud beta container --project <Google-Project-ID> clusters create <GKE-Cluster-Name> --zone us-west1-a --release-channel "regular" --machine-type "n1-standard-4" --accelerator "type=nvidia-tesla-t4,count=1" --image-type "UBUNTU_CONTAINERD" --disk-type "pd-standard" --disk-size "1000" --no-enable-intra-node-visibility --metadata disable-legacy-endpoints=true --max-pods-per-node "110" --num-nodes "1" --logging=SYSTEM,WORKLOAD --monitoring=SYSTEM --enable-ip-alias --no-enable-intra-node-visibility --default-max-pods-per-node "110" --no-enable-master-authorized-networks --tags=nvidia-ingress-all

Note
  1. <Google-Project-ID>: You will find the Project ID within the Google console dashboard Settings and update accordingly.

    cloud-gke-02.png


  2. If the network name is not “default” then get the network information from the console and append the below value with the appropriate network and subnetwork name.

Copy
Copied!
            

--network "<Google Kubernetes Network Name>" --subnetwork "<Google Kubernetes SubNetwork Name>" Example: --network "projects/<GKE-Project-ID>/global/networks/<GKE-Network-Name>" --subnetwork "projects/<GKE-Project-ID>/regions/us-west1/subnetworks/<GKE-Network-Name>"

Run the below command to get the kubeconfig credentials to the local system.

Copy
Copied!
            

$ export USE_GKE_GCLOUD_AUTH_PLUGIN=True ./google-cloud-sdk/bin/gcloud container clusters get-credentials <GKE-Cluster-Name> --zone us-west1-a

Run the below command to verify the node information

Copy
Copied!
            

kubectl get nodes -o wide

Example output result

Copy
Copied!
            

NAME STATUS ROLES AGE VERSION INTERNAL-IP EXTERNAL-IP OS-IMAGE KERNEL-VERSION CONTAINER-RUNTIME gke-<GKE-Cluster-Name>-default-pool-db9e3df9-r0jf Ready <none> 5m15s v1.25.6 192.168.50.108 13.57.187.63 Ubuntu 20.04.6 LTS 5.15.0-1033-gke containerd://1.6.12

Create a resource quota file with the below command.

Copy
Copied!
            

cat <<EOF | tee resourcequota.yaml apiVersion: v1 kind: ResourceQuota metadata: name: gpu-operator-quota namespace: gpu-operator spec: hard: pods: 100 scopeSelector: matchExpressions: - operator: In scopeName: PriorityClass values: - system-node-critical - system-cluster-critical EOF

Run the below commands to create a namespace and resource quota to the namespace on GKE cluster.

Copy
Copied!
            

./google-cloud-sdk/bin/kubectl create ns gpu-operator ./google-cloud-sdk/bin/kubectl apply -f resourcequota.yaml ./google-cloud-sdk/bin/kubectl get ResourceQuota -n gpu-operator

Example output result:

Copy
Copied!
            

NAME AGE REQUEST LIMIT gke-resource-quotas 24s count/ingresses.extensions: 0/100, count/ingresses.networking.k8s.io: 0/100, count/jobs.batch: 0/5k, pods: 0/1500, services: 0/500 gpu-operator-quota 21s pods: 0/100

For more information, please refer to GKE’s quotas and limits overview

Verify the default Pod Security Policies on GKE with below command.

Copy
Copied!
            

./google-cloud-sdk/bin/kubectl get psp

Example Output result:

Copy
Copied!
            

Warning: policy/v1beta1 PodSecurityPolicy is deprecated in v1.21+, unavailable in v1.25+ NAME PRIV CAPS SELINUX RUNASUSER FSGROUP SUPGROUP READONLYROOTFS VOLUMES gce.gke-metrics-agent false RunAsAny RunAsAny RunAsAny RunAsAny false hostPath,secret,configMap

Now that the cluster and appropriate resources are created, the NVIDIA GPU Operator can be installed

First we will access our NGC API Key.

Log into your NGC account and generate a new API Key or locate your existing API key. Please refer to the Accessing NGC section of the Appendix.

Generate an API key for accessing the catalog

Next you must generate an API Key that will give you access to the Enterprise catalog.

Navigate to the user account icon in the top right corner and select Setup.

cloud-03.png

cloud-04.png

Select Get API key to open the Setup > API Key page.

Select Generate API Key to generate your API key.

Select Confirm to generate the key, and copy it from the bottom of the page. NGC does not save your key, so store it in a secure place.

cloud-05.png

Note

Generating a new API Key invalidates the previously generated key.


Add the Helm repo and update with the below commands.

Copy
Copied!
            

helm repo add nvidia https://helm.ngc.nvidia.com/nvaie --username='$oauthtoken' --password=<YOUR API KEY> helm repo update

Create a NGC Secret with your NGC API key on “gpu-operator” namespace as per below.

Copy
Copied!
            

./google-cloud-sdk/bin/kubectl create secret docker-registry ngc-secret \ --docker-server=nvcr.io/nvaie --docker-username=\$oauthtoken \ --docker-password=<NGC-API-KEY> \ --docker-email=<NGC-email> -n gpu-operator

Create an empty gridd.conf file, then create a configmap with NVIDIA vGPU Licence token file as per below

Copy
Copied!
            

./google-cloud-sdk/bin/kubectl create configmap licensing-config -n gpu-operator --from-file=./client_configuration_token.tok --from-file=./gridd.conf

Note

The configmap will look for the file client_configuration_token.tok, if your token is in a different form such as client_configuration_token_date_xx_xx.tok, then please run the below command:

Copy
Copied!
            

mv client_configuration_token_date_xx_xx.tok client_configuration_token.tok

Install the GPU Operator from Enterprise Catalog with License token and driver repository.

Copy
Copied!
            

helm install gpu-operator nvidia/gpu-operator-3-0 --version 22.9.1 --set driver.repository=nvcr.io/nvaie,driver.licensingConfig.configMapName=licensing-config,psp.enabled=true --namespace gpu-operator

Important

Ensure that you have the correct role as either a Kubernetes Engine Admin or Kubernetes Engine Cluster Admin in order to install the GPU Operator. Please refer to Create IAM policies if you lack the required permissions.

Once installed, please wait at least 5 minutes and verify that all the pods are either running or completed as per below.

Copy
Copied!
            

./google-cloud-sdk/bin/kubectl get pods -n gpu-operator NAME READY STATUS RESTARTS AGE gpu-feature-discovery-fzgv9 1/1 Running 0 6m1s gpu-operator-69f476f875-w4hwr 1/1 Running 0 6m29s gpu-operator-node-feature-discovery-master-84c7c7c6cf-hxlk4 1/1 Running 0 6m29s gpu-operator-node-feature-discovery-worker-86bbx 1/1 Running 0 6m29s nvidia-container-toolkit-daemonset-c7k5p 1/1 Running 0 6m nvidia-cuda-validator-qjcsf 0/1 Completed 0 59s nvidia-dcgm-exporter-9tggn 1/1 Running 0 6m nvidia-device-plugin-daemonset-tpx9z 1/1 Running 0 6m nvidia-device-plugin-validator-gz85d 0/1 Completed 0 44s nvidia-driver-daemonset-jwzx8 1/1 Running 0 6m9s nvidia-operator-validator-qj57n 1/1 Running 0 6m

Verify the NVIDIA GPU Driver loaded with below command.

Copy
Copied!
            

./google-cloud-sdk/bin/kubectl exec -it nvidia-driver-daemonset-jwzx8 -n gpu-operator -- nvidia-smi

Copy
Copied!
            

Defaulted container "nvidia-driver-ctr" out of: nvidia-driver-ctr, k8s-driver-manager (init) Tue Feb 14 22:24:31 2023 +-----------------------------------------------------------------------------+ | NVIDIA-SMI 520.60.13 Driver Version: 520.60.13 CUDA Version: 12.0 | |-------------------------------+----------------------+----------------------+ | GPU Name Persistence-M| Bus-Id Disp.A | Volatile Uncorr. ECC | | Fan Temp Perf Pwr:Usage/Cap| Memory-Usage | GPU-Util Compute M. | | | | MIG M. | |===============================+======================+======================| | 0 Tesla T4 On | 00000000:00:04.0 Off | 0 | | N/A 51C P8 17W / 70W | 0MiB / 15360MiB | 0% Default | | | | N/A | +-------------------------------+----------------------+----------------------+ +-----------------------------------------------------------------------------+ | Processes: | | GPU GI CI PID Type Process name GPU Memory | | ID ID Usage | |=============================================================================| | No running processes found | +-----------------------------------------------------------------------------+

Note

nvidia-driver-daemonset-xxxxx will be different within your own environment for the above command to verify the NVIDIA vGPU Driver.

Verify the NVIDIA vGPU license information with below command

Copy
Copied!
            

./google-cloud-sdk/bin/kubectl exec -it nvidia-driver-daemonset-jwzx8 -n gpu-operator-resources -- nvidia-smi -q

Create a docker-regirty secret. This will be used in a custom yaml to pull containers from the Enterprise Catalog.

Copy
Copied!
            

./google-cloud-sdk/bin/kubectl create secret docker-registry regcred --docker-server=nvcr.io/nvaie --docker-username=\$oauthtoken --docker-password=<YOUR_NGC_KEY> --docker-email=<your_email_id> -n default

Create a custom yaml file to deploy an NVIDIA AI Enterprise Container and run sample training code.

Copy
Copied!
            

nano pytoch-mnist.yaml

Paste the below contents into the file and save

Copy
Copied!
            

--- apiVersion: apps/v1 kind: Deployment metadata: name: pytorch-mnist labels: app: pytorch-mnist spec: replicas: 1 selector: matchLabels: app: pytorch-mnist template: metadata: labels: app: pytorch-mnist spec: containers: - name: pytorch-container image: nvcr.io/nvaie/pytorch-2-0:22.02-nvaie-2.0-py3 command: - python args: - /workspace/examples/upstream/mnist/main.py resources: requests: nvidia.com/gpu: 1 limits: nvidia.com/gpu: 1 imagePullSecrets: - name: regcred

Check the status of the pod.

Copy
Copied!
            

./google-cloud-sdk/bin/kubectl get pods

View the output of the sample mnist training job.

Copy
Copied!
            

./google-cloud-sdk/bin/kubectl logs -l app=pytorch-mnist

The output will look similar to this.

Copy
Copied!
            

~$ ./google-cloud-sdk/bin/kubectl logs -l app=pytorch-mnist Train Epoch: 7 [55680/60000 (93%)] Loss: 0.040756 Train Epoch: 7 [56320/60000 (94%)] Loss: 0.028230 Train Epoch: 7 [56960/60000 (95%)] Loss: 0.019917 Train Epoch: 7 [57600/60000 (96%)] Loss: 0.005957 Train Epoch: 7 [58240/60000 (97%)] Loss: 0.003768 Train Epoch: 7 [58880/60000 (98%)] Loss: 0.277371 Train Epoch: 7 [59520/60000 (99%)] Loss: 0.115487 Test set: Average loss: 0.0270, Accuracy: 9913/10000 (99%)

Run the below command to delete the GKE cluster

Copy
Copied!
            

./google-cloud-sdk/bin/gcloud beta container clusters delete <cluster-name> --zone <zone-name>

© Copyright 2022-2023, NVIDIA. Last updated on Nov 7, 2023.