TAO Toolkit v5.2.0

The following is used to deploy or update the TAO Toolkit API service on an existing Kubernetes cluster. One can use the following to enable HTTPS and enforce user authentication to enable secure multi-tenancy. You do not need these steps if you followed one of the Platform Setup and do not wish to enable secure multi-tenancy.

One can shutdown an already deployed TAO Toolkit service.


helm delete tao-toolkit-api

One must use the provided Helm chart to deploy TAO Toolkit service.


helm fetch --username='$oauthtoken' --password=<YOUR API KEY> mkdir tao-toolkit-api && tar -zxvf tao-toolkit-api-5.2.0.tgz -C tao-toolkit-api

If needed, one can customize the deployment by updating the chart’s tao-toolkit-api/values.yaml.

  • image is the location of the TAO Toolkit API container image

  • host, tlsSecret are for enabling HTTPS and enforcing user authentication, enabling secure multi-tenancy

  • corsOrigin is for enabling CORS and setting origin

  • authClientID is reserved for future NVIDIA Starfleet authentication

  • imagePullSecret is the secret name that you setup to access Nvidia’s registry

  • imagePullPolicy is set to Always fetch from instead of using locally cached image

  • storageClassName is the storage class created by your K8s Storage Provisioner. On bare-metal deployment it is nfs-client, and on AWS EKS can be standard. Not providing a value would make your deployment use your K8s cluster’s default storage class

  • storageAccessMode is set to ReadWriteMany to reuse allocated storage between deployments, or ReadWriteOnce to create a new storage at every deployment

  • storageSize is ignored by many Storage Provisioners. But here would be where to set your shared storage size

  • backend is the platform used for training jobs. Defaults to local-k8s

  • maxNumGpuPerNode is the number of GPU assigned to each job. Multi-node training is not supported, you are limited to the number of GPUs within a cluster node

  • telemetryOptOut can be set if you want to opt-out from NVIDIA to collect anonymous usage metrics

Example for creating a tlsSecret:


openssl req -x509 -sha256 -nodes -days 365 -newkey rsa:2048 -keyout tls.key -out tls.crt -subj "/" --addext "subjectAltName =" kubectl create secret tls tls-secret --key tls.key --cert tls.crt --namespace default

Then deploy the API service.


helm install tao-toolkit-api tao-toolkit-api/ --namespace default

One can validate the deployment by looking for the Ready and Completed states.


kubectl get pods -n default

To debug a deployment. look for events toward the bottom of the following command.


kubectl describe pods tao-toolkit-api -n default

Common issues are:

  • GPU Operator or Storage Provisioner pods not in Ready or Completed states

  • Missing or invalid imagepullsecret

Previous Setup
© Copyright 2024, NVIDIA. Last updated on Mar 18, 2024.