Kubernetes Deployment#

To start with a simple kubernetes deployment of an Audio2Face cluster, you can use our quick deployment script.

It assumes the following dependencies:

microk8s
microk8s GPU addon running the NVIDIA GPU operator.
A valid NGC token to connect to NVIDIA’s image repository as well as a configured NGC CLI.

Make sure you have an NVAIE Access and your Personal Key.

Note

If this is the first time you are deploying A2F and your host machine already have a kubernetes setup, please remove ~/.kube/ folder. You may make a backup of this folder if required:

$ mv ~/.kube ~/.kube.bk

1. Install microk8s, GPU addon, set local path provisioner and set secrets#

Run the following command to install microk8s with the GPU addon with snap.

$ if [ -d $HOME/.kube ]; then
          echo "kubernetes already setup; skipping microk8s installation; please rm -rf $HOME/.kube to force installation"
  else
          sudo snap install microk8s --revision=5891 --classic
          sudo microk8s enable gpu --version v23.6.1
          sudo snap install --classic kubectl
          sudo snap install helm --classic
          sudo usermod -a -G microk8s ${USER}
          mkdir -p $HOME/.kube
          sudo microk8s config > $HOME/.kube/config
          sudo chown $(id -u):$(id -g) $HOME/.kube/config
          sudo microk8s stop
          sudo microk8s refresh-certs --cert ca.crt
          sudo microk8s start
          sudo microk8s status --wait-ready
  fi

Then to avoid using sudo run:

$ newgrp microk8s

Set local path provisioner by running:

$ curl https://raw.githubusercontent.com/rancher/local-path-provisioner/v0.0.23/deploy/local-path-storage.yaml | sed 's/^  name: local-path$/  name: mdx-local-path/g' | microk8s kubectl apply -f -

Export the NGC API KEY in an environment variable:

$ export NGC_API_KEY=<value>

Delete any pre-existing secrets and set the required secrets with your valid NGC_API_KEY:

$ microk8s kubectl delete secret --ignore-not-found ngc-docker-reg-secret
$ microk8s kubectl delete secret --ignore-not-found ngc-api-key-secret
$ microk8s kubectl create secret docker-registry ngc-docker-reg-secret --docker-username='$oauthtoken' --docker-password=$NGC_API_KEY --docker-server=nvcr.io
$ microk8s kubectl create secret generic ngc-api-key-secret --from-literal=NGC_API_KEY=$NGC_API_KEY

1. Download and install the helm chart#

Use the following command to create a new directory and download the Helm chart into it:

$ export HELM_VERSION=1.2.0
$ mkdir a2f-3d-nim/ && cd a2f-3d-nim/
$ microk8s helm fetch https://helm.ngc.nvidia.com/nim/nvidia/charts/audio2face-3d-"$HELM_VERSION".tgz --username='$oauthtoken' --password=$NGC_API_KEY
$ tar xvf audio2face-3d-$HELM_VERSION.tgz

The Helm chart includes observability features for the A2F-3D service. For additional details about observability, refer to this page: Observability.

You can then install the Helm chart by executing the following command:

$ microk8s helm install a2f-3d-nim audio2face-3d/

2. Optional - Expose the pod IP publicly#

You need to expose the pod ID publicly in order to access the A2F-3D NIM from another machine. If this is not your usecase, you can skip this step.

The following yaml file defines the set of ports for communication between external systems and the a2f-a2f-deployment-a2f-service pod in the Kubernetes cluster. Read the comments to understand the file structure.

You can adjust the external ports as needed, ensuring they fall within the 30000–32767 range. To apply the YAML file and expose the specified ports, run the following command:

$ sudo microk8s kubectl apply -f service_expose.yaml

Then you need to delete Audio2Face-3D pod to restart it.

$ microk8s kubectl delete pods <audio2face_pod_name>

The <audio2face_pod_name> can be found by running:

$ microk8s kubectl get pods
NAME                                  READY   STATUS             RESTARTS       AGE
a2f-a2f-deployment-xxx-xxx            1/1     Running            0              1h

Here it is: a2f-a2f-deployment-xxx-xxx

3. Check it is up and running#

Execute the following command to check if the app was deployed successfully. Please note that the command output may vary.

$ microk8s kubectl get pods
  NAME                                 READY   STATUS    RESTARTS   AGE
  a2f-a2f-deployment-94d89f979-v6n24   0/1     Init:0/1  0          53s

Note

If you see the status being ImagePullBackOff, then reconfigure the secrets. Make sure your NGC_API_KEY is valid.

Note

If you see the status being Pending, then you need to set again the local path provisioner:

$ curl https://raw.githubusercontent.com/rancher/local-path-provisioner/v0.0.23/deploy/local-path-storage.yaml | sed 's/^  name: local-path$/  name: mdx-local-path/g' | microk8s kubectl apply -f -

Wait some time until STATUS changes to Running.

To run a client application, you will need the external IP and port to connect to. You can obtain this information by executing the following command:

$ microk8s kubectl get svc
  NAME                             TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)                       AGE
  a2f-a2f-deployment-a2f-service   ClusterIP   <ip>            <none>        50010/TCP,9464/TCP,8000/TCP   56m

You can then try to interact with the Audio2Face-3D using for example the sample app provided.

Uninstall the app#

To uninstall the app, run:

$ microk8s helm uninstall a2f-3d-nim