NVIDIA AI Enterprise 3.1 or later
Google Kubernetes Engine (GKE) provides a managed environment for deploying, managing, and scaling your containerized applications using Google infrastructure. NVIDIA AI Enterprise, the end-to-end software of the NVIDIA AI platform, is supported to run on GKE. The GKE environment consists of multiple machines (specifically, Compute Engine instances) grouped together to form a cluster. This guide provides details for deploying and running NVIDIA AI Enterprise on GKE clusters with NVIDIA GPU Accelerated nodes.
NVIDIA AI Enterprise License via BYOL or a Private Offer
Install the Google Cloud CLI
NoteNVIDIA recommends installing the Google Cloud CLI via download of the linux google-cloud-cli-xxx.x.x-linux-x86_64.tar.gz file. If a different OS distribution is being utilized, please continue to follow Google’s instructions for your desired OS.
Google Cloud account with Google Kubernetes Engine Admin Role and Kubernetes Engine Cluster Admin Role for more information please refer to Google’s IAM Policies
Ubuntu Nodes
Run the below comannds to install the GKE components.
./google-cloud-sdk/bin/gcloud components install beta
./google-cloud-sdk/bin/gcloud components install kubectl
./google-cloud-sdk/bin/gcloud components update
Run the below command to create a GKE cluster.
./google-cloud-sdk/bin/gcloud beta container --project <Google-Project-ID> clusters create <GKE-Cluster-Name> --zone us-west1-a --release-channel "regular" --machine-type "n1-standard-4" --accelerator "type=nvidia-tesla-t4,count=1" --image-type "UBUNTU_CONTAINERD" --disk-type "pd-standard" --disk-size "1000" --no-enable-intra-node-visibility --metadata disable-legacy-endpoints=true --max-pods-per-node "110" --num-nodes "1" --logging=SYSTEM,WORKLOAD --monitoring=SYSTEM --enable-ip-alias --no-enable-intra-node-visibility --default-max-pods-per-node "110" --no-enable-master-authorized-networks --tags=nvidia-ingress-all
<Google-Project-ID>: You will find the Project ID within the Google console dashboard Settings and update accordingly.
If the network name is not “default” then get the network information from the console and append the below value with the appropriate network and subnetwork name.
--network "<Google Kubernetes Network Name>" --subnetwork "<Google Kubernetes SubNetwork Name>"
Example:
--network "projects/<GKE-Project-ID>/global/networks/<GKE-Network-Name>" --subnetwork "projects/<GKE-Project-ID>/regions/us-west1/subnetworks/<GKE-Network-Name>"
Run the below command to get the kubeconfig credentials to the local system.
$ export USE_GKE_GCLOUD_AUTH_PLUGIN=True
./google-cloud-sdk/bin/gcloud container clusters get-credentials <GKE-Cluster-Name> --zone us-west1-a
Run the below command to verify the node information
kubectl get nodes -o wide
Example output result
NAME STATUS ROLES AGE VERSION INTERNAL-IP EXTERNAL-IP OS-IMAGE KERNEL-VERSION CONTAINER-RUNTIME
gke-<GKE-Cluster-Name>-default-pool-db9e3df9-r0jf Ready <none> 5m15s v1.25.6 192.168.50.108 13.57.187.63 Ubuntu 20.04.6 LTS 5.15.0-1033-gke containerd://1.6.12
Create a resource quota file with the below command.
cat <<EOF | tee resourcequota.yaml
apiVersion: v1
kind: ResourceQuota
metadata:
name: gpu-operator-quota
namespace: gpu-operator
spec:
hard:
pods: 100
scopeSelector:
matchExpressions:
- operator: In
scopeName: PriorityClass
values:
- system-node-critical
- system-cluster-critical
EOF
Run the below commands to create a namespace and resource quota to the namespace on GKE cluster.
./google-cloud-sdk/bin/kubectl create ns gpu-operator
./google-cloud-sdk/bin/kubectl apply -f resourcequota.yaml
./google-cloud-sdk/bin/kubectl get ResourceQuota -n gpu-operator
Example output result:
NAME AGE REQUEST LIMIT
gke-resource-quotas 24s count/ingresses.extensions: 0/100, count/ingresses.networking.k8s.io: 0/100, count/jobs.batch: 0/5k, pods: 0/1500, services: 0/500
gpu-operator-quota 21s pods: 0/100
For more information, please refer to GKE’s quotas and limits overview
Verify the default Pod Security Policies on GKE with below command.
./google-cloud-sdk/bin/kubectl get psp
Example Output result:
Warning: policy/v1beta1 PodSecurityPolicy is deprecated in v1.21+, unavailable in v1.25+
NAME PRIV CAPS SELINUX RUNASUSER FSGROUP SUPGROUP READONLYROOTFS VOLUMES
gce.gke-metrics-agent false RunAsAny RunAsAny RunAsAny RunAsAny false hostPath,secret,configMap
Now that the cluster and appropriate resources are created, the NVIDIA GPU Operator can be installed
First we will access our NGC API Key.
Log into your NGC account and generate a new API Key or locate your existing API key. Please refer to the Accessing NGC section of the Appendix.
Generate an API key for accessing the catalog
Next you must generate an API Key that will give you access to the Enterprise catalog.
Navigate to the user account icon in the top right corner and select Setup.
Select Get API key to open the Setup > API Key page.
Select Generate API Key to generate your API key.
Select Confirm to generate the key, and copy it from the bottom of the page. NGC does not save your key, so store it in a secure place.
Generating a new API Key invalidates the previously generated key.
Add the Helm repo and update with the below commands.
helm repo add nvidia https://helm.ngc.nvidia.com/nvaie --username='$oauthtoken' --password=<YOUR API KEY>
helm repo update
Create a NGC Secret with your NGC API key on “gpu-operator” namespace as per below.
./google-cloud-sdk/bin/kubectl create secret docker-registry ngc-secret \
--docker-server=nvcr.io/nvaie --docker-username=\$oauthtoken \
--docker-password=<NGC-API-KEY> \
--docker-email=<NGC-email> -n gpu-operator
Create an empty gridd.conf file, then create a configmap with NVIDIA vGPU Licence token file as per below
./google-cloud-sdk/bin/kubectl create configmap licensing-config -n gpu-operator --from-file=./client_configuration_token.tok --from-file=./gridd.conf
The configmap will look for the file client_configuration_token.tok, if your token is in a different form such as client_configuration_token_date_xx_xx.tok, then please run the below command:
mv client_configuration_token_date_xx_xx.tok client_configuration_token.tok
Install the GPU Operator from Enterprise Catalog with License token and driver repository.
helm install gpu-operator nvidia/gpu-operator-3-0 --version 22.9.1 --set driver.repository=nvcr.io/nvaie,driver.licensingConfig.configMapName=licensing-config,psp.enabled=true --namespace gpu-operator
Ensure that you have the correct role as either a Kubernetes Engine Admin or Kubernetes Engine Cluster Admin in order to install the GPU Operator. Please refer to Create IAM policies if you lack the required permissions.
Once installed, please wait at least 5 minutes and verify that all the pods are either running or completed as per below.
./google-cloud-sdk/bin/kubectl get pods -n gpu-operator
NAME READY STATUS RESTARTS AGE
gpu-feature-discovery-fzgv9 1/1 Running 0 6m1s
gpu-operator-69f476f875-w4hwr 1/1 Running 0 6m29s
gpu-operator-node-feature-discovery-master-84c7c7c6cf-hxlk4 1/1 Running 0 6m29s
gpu-operator-node-feature-discovery-worker-86bbx 1/1 Running 0 6m29s
nvidia-container-toolkit-daemonset-c7k5p 1/1 Running 0 6m
nvidia-cuda-validator-qjcsf 0/1 Completed 0 59s
nvidia-dcgm-exporter-9tggn 1/1 Running 0 6m
nvidia-device-plugin-daemonset-tpx9z 1/1 Running 0 6m
nvidia-device-plugin-validator-gz85d 0/1 Completed 0 44s
nvidia-driver-daemonset-jwzx8 1/1 Running 0 6m9s
nvidia-operator-validator-qj57n 1/1 Running 0 6m
Verify the NVIDIA GPU Driver loaded with below command.
./google-cloud-sdk/bin/kubectl exec -it nvidia-driver-daemonset-jwzx8 -n gpu-operator -- nvidia-smi
Defaulted container "nvidia-driver-ctr" out of: nvidia-driver-ctr, k8s-driver-manager (init)
Tue Feb 14 22:24:31 2023
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 520.60.13 Driver Version: 520.60.13 CUDA Version: 12.0 |
|-------------------------------+----------------------+----------------------+
| GPU Name Persistence-M| Bus-Id Disp.A | Volatile Uncorr. ECC |
| Fan Temp Perf Pwr:Usage/Cap| Memory-Usage | GPU-Util Compute M. |
| | | MIG M. |
|===============================+======================+======================|
| 0 Tesla T4 On | 00000000:00:04.0 Off | 0 |
| N/A 51C P8 17W / 70W | 0MiB / 15360MiB | 0% Default |
| | | N/A |
+-------------------------------+----------------------+----------------------+
+-----------------------------------------------------------------------------+
| Processes: |
| GPU GI CI PID Type Process name GPU Memory |
| ID ID Usage |
|=============================================================================|
| No running processes found |
+-----------------------------------------------------------------------------+
nvidia-driver-daemonset-xxxxx will be different within your own environment for the above command to verify the NVIDIA vGPU Driver.
Verify the NVIDIA vGPU license information with below command
./google-cloud-sdk/bin/kubectl exec -it nvidia-driver-daemonset-jwzx8 -n gpu-operator-resources -- nvidia-smi -q
Create a docker-regirty secret. This will be used in a custom yaml to pull containers from the Enterprise Catalog.
./google-cloud-sdk/bin/kubectl create secret docker-registry regcred --docker-server=nvcr.io/nvaie --docker-username=\$oauthtoken --docker-password=<YOUR_NGC_KEY> --docker-email=<your_email_id> -n default
Create a custom yaml file to deploy an NVIDIA AI Enterprise Container and run sample training code.
nano pytoch-mnist.yaml
Paste the below contents into the file and save
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: pytorch-mnist
labels:
app: pytorch-mnist
spec:
replicas: 1
selector:
matchLabels:
app: pytorch-mnist
template:
metadata:
labels:
app: pytorch-mnist
spec:
containers:
- name: pytorch-container
image: nvcr.io/nvaie/pytorch-2-0:22.02-nvaie-2.0-py3
command:
- python
args:
- /workspace/examples/upstream/mnist/main.py
resources:
requests:
nvidia.com/gpu: 1
limits:
nvidia.com/gpu: 1
imagePullSecrets:
- name: regcred
Check the status of the pod.
./google-cloud-sdk/bin/kubectl get pods
View the output of the sample mnist training job.
./google-cloud-sdk/bin/kubectl logs -l app=pytorch-mnist
The output will look similar to this.
~$ ./google-cloud-sdk/bin/kubectl logs -l app=pytorch-mnist
Train Epoch: 7 [55680/60000 (93%)] Loss: 0.040756
Train Epoch: 7 [56320/60000 (94%)] Loss: 0.028230
Train Epoch: 7 [56960/60000 (95%)] Loss: 0.019917
Train Epoch: 7 [57600/60000 (96%)] Loss: 0.005957
Train Epoch: 7 [58240/60000 (97%)] Loss: 0.003768
Train Epoch: 7 [58880/60000 (98%)] Loss: 0.277371
Train Epoch: 7 [59520/60000 (99%)] Loss: 0.115487
Test set: Average loss: 0.0270, Accuracy: 9913/10000 (99%)
Run the below command to delete the GKE cluster
./google-cloud-sdk/bin/gcloud beta container clusters delete <cluster-name> --zone <zone-name>