Deploying Riva ASR Service on AWS EKS

This is a sample for deploying and scaling Riva ASR Service on Amazon Web Services (AWS) Elastic Kubernetes Service (EKS) with traefik based load balancing. It includes the following steps:

  1. Downloading and modifying the Riva API Helm Chart to add a node selector and to be a headless service.

  2. Downloading and modifying the Traefik Helm Chart to add a node selector and expose it on the cluster-internal IP.

  3. Defining the client service and ingress route.

  4. Defining cluster config and launching the cluster via eksctl.

  5. Benchmarking

  6. Scaling the cluster and the Riva ASR Service.

This sample assumes that:

  • The system has access to Riva via NGC (check via the Docker login

  • The system has pre-installed eksctl, helm and kubectl.

This sample has been tested on: eksctl (0.35.0), helm (v3.4.1), kubectl (v1.17.0), and traefik (2.2.8, API version v2).


Each step can have validation, debug, cleanup, and monitoring steps associated with each step in case you’d like to re-do that step.

Downloading and Modifying the Riva API Helm Chart

  1. Download and untar the Riva API Helm Chart. Replace VERSION_TAG with the specific version needed.

    $ export NGC_API_KEY=<<replace with your NGC_API_KEY>>
    $ export VERSION_TAG="1.8.0-beta"
    $ helm fetch${VERSION_TAG}.tgz --username='$oauthtoken' --password=$NGC_API_KEY
    $ tar -xvzf riva-api-${VERSION_TAG}.tgz
  2. Within the riva-api folder, modify the following files:

    • values.yaml

      • Set all services to false except the ASR service (TTS, NLP).

      • For ngcModelConfigs.asr, select the model to use for ASR; optionally comment out the rest.

    • templates/deployment.yaml

      • Within spec.template.spec add:

        - key: gpu-type
            operator: Equal
            value: v100
            effect: NoSchedule
      • This tells the service to deploy on a node-group named gpu-linux-workers and also restricts it to V100 GPU type.

    • templates/service.yaml

      • Within spec, replace type: {{ .Values.service.type }} with clusterIP: None.

      • We have now made this a headless service (overriding the .Values.service.type originally set to be LoadBalancer in values.yaml).

Downloading and Modifying the Traefik Helm Chart

  1. Download and untar the Traefik Helm Chart.

    $ helm repo add traefik
    $ helm repo update
    $ helm fetch traefik/traefik
    $ tar -zxvf traefik-*.tgz
  2. Within the traefik folder, modify the following files:

    • values.yaml

      • Change service.type from LoadBalancer to ClusterIP. This will expose the service on a cluster-internal IP.

      • Set nodeSelector to { cpu-linux-lb }. Similar to what we did for Riva API Service, this will tell the Traefik Service to run on the cpu-linux-lb node-group.

Defining the Client Service and Ingress Route

  1. Pull the riva-api-client container from Riva NGC.

  2. Service to deploy on the cpu-linux-client node-group. The client deployment.yaml looks like the following:

    apiVersion: apps/v1
    kind: Deployment
      name: ss-client
        app: "rivaasrclient"
      namespace: riva
      replicas: 1
        app: "rivaasrclient"
            app: "rivaasrclient"
            - name: imagepullsecret
            - name: riva-client
              image: "|NgcOrgTeam|/riva-speech-client:|VersionNum|"
              command: ["/bin/bash"]
              args: ["-c", "while true; do sleep 5; done"]

    With all the individual services ready to go, we need to define an ingress route that will enable the traefik load balancer to balance the incoming requests across multiple riva-api services. The code below shows how riva-ingress.yaml is defined. This relies on a DNS entry matching the Host clause. Currently, this looks for riva.nvda. Replace or add a DNS resolution suitable for the deployment environment.

    kind: IngressRoute
      name: riva-ingressroute
      namespace: riva
        - web
        - match: Host(`riva.nvda`)
          kind: Rule
            - name: riva-riva-api
              port: 50051
              scheme: h2c

Defining and Launching the EKS Cluster

So far, we’ve talked about 3 node-groups, cpu-linux-client, cpu-linux-lb and gpu-linux-workers.

  1. Set each of these node-groups in our cluster.

    • cpu-linux-client We want to use m5.2xlarge (general purpose) instances with minimum size 1 and maximum size 4.

    • cpu-linux-lb We want to use one c5.24xlarge (compute intensive) instance.

    • gpu-linux-workers We want to use p3.2xlarge (single V100 GPU) with minimum size 1 and maximum size 4.

  2. Build and launch a configuration that defines each of these node-groups and save to a file called eks_launch_conf.yaml.

    kind: ClusterConfig
      name: riva-cluster
      region: us-west-2
      version: "1.17"
      - name: gpu-linux-workers
        labels: { role: workers }
        instanceType: p3.2xlarge
        minSize: 1
        maxSize: 8
        volumeSize: 100
        privateNetworking: true
          allow: true
      - name: cpu-linux-clients
        labels: { role: clients }
        instanceType: m5.2xlarge
        minSize: 1
        maxSize: 4
        volumeSize: 100
        privateNetworking: true
          allow: true
      - name: cpu-linux-lb
        labels: { role: loadbalancers }
        instanceType: c5.24xlarge
        desiredCapacity: 1
        volumeSize: 100
        privateNetworking: true
          allow: true
  3. Launch the cluster with the above config.

    $ eksctl create cluster -f eks_launch_conf.yaml

    As a result of this command, you should see some changes in your default Kubernetes configuration file, and the nodes should start showing up in Kubernetes. Here is how to check:

    $ cat .kube/config
    $ kubectl get pods -A
    $ kubectl get nodes --show-labels
    $ kubectl get nodes --selector role=workers
    $ kubectl get nodes --selector role=clients
    $ kubectl get nodes --selector role=loadbalancers
  4. After the cluster is up-and-running, it is time to launch the services.

    # setup namespaces
    $ kubectl create namespace riva
    # ngc api key setup and secrets setup, if not already set by the helm chart
    $ export NGC_API_KEY=<<NGC_API_KEY>>
       # install gpu operator
    $ helm repo add nvdp
    $ helm repo update
    $ helm install \
    --generate-name \
    --set failOnInitError=false \
    # install riva
    $ cd riva-api-nvidia
    $ helm install --namespace riva riva --set ngcCredentials.password=`echo -n $NGC_API_KEY | base64 -w0` --set modelRepoGenerator.modelDeployKey=`echo -n tlt_encode | base64 -w0` --set riva.speechServices.asr=true --set riva.speechServices.tts=true --set riva.speechServices.nlp=true
    $ cd ..
    # debug
    $ export pod=`kubectl get pods -n riva | cut -d " " -f 1|grep riva`
    $ kubectl describe pod -n riva $pod
    # watch logs
    $ kubectl logs -n riva -f $pod -c riva-speech-api
    # install traefik
    $ cd traefik/
    $ helm install traefik traefik -n riva
    $ cd ..
    # install client
    $ cd traefik/
    $ kubectl apply -f deployment.yaml -n riva
    $ cd ..
    # ingress route apply
    $ cd traefik/
    $ kubectl apply -f riva-ingress.yaml -n riva
    $ cd ..

Running the Benchmarks

After all the services are up-and-running, we can benchmark by stepping into the client container and send requests to the load balancer.

Here is how the services look like:

$ kubectl get svc -A
NAMESPACE     NAME                TYPE        CLUSTER-IP     EXTERNAL-IP   PORT(S)                                          AGE
default       kubernetes          ClusterIP     <none>        443/TCP                                          53m
riva        riva-riva-api   ClusterIP   None           <none>        8000/TCP,8001/TCP,8002/TCP,50051/TCP,60051/TCP   91s
riva        traefik             ClusterIP   <none>        80/TCP,443/TCP                                   68s
kube-system   kube-dns            ClusterIP    <none>        53/UDP,53/TCP

And here are the pods:

$ kubectl get pods -A
NAMESPACE     NAME                                    READY   STATUS    RESTARTS   AGE
riva        riva-riva-api-5d8f5c7dd6-vkd49      2/2     Running   0          6m33s
riva        ss-client-7ff77cbb76-djt5q              1/1     Running   0          6m2s
riva        traefik-5fb6c8bb47-mlxsg                1/1     Running   0          6m10s
kube-system   aws-node-fgm52                          1/1     Running   0          51m
kube-system   aws-node-hbwfn                          1/1     Running   0          50m
kube-system   aws-node-xltx6                          1/1     Running   0          51m
kube-system   coredns-5946c5d67c-5w8bv                1/1     Running   0          57m
kube-system   coredns-5946c5d67c-f728c                1/1     Running   0          57m
kube-system   kube-proxy-hpp6p                        1/1     Running   0          50m
kube-system   kube-proxy-t4dvb                        1/1     Running   0          51m
kube-system   kube-proxy-v2ttk                        1/1     Running   0          51m
kube-system   nvidia-device-plugin-1611946093-vgg2f   1/1     Running   0          6m46s
kube-system   nvidia-device-plugin-1611946093-w6969   1/1     Running   0          6m46s
kube-system   nvidia-device-plugin-1611946093-w7sw4   1/1     Running   0          6m46s
  1. Run the benchmarks.

    # exec into the client
    $ export clnt=`kubectl get pods -n riva | cut -d " " -f 1| grep ss-client`
    $ kubectl exec --stdin --tty $clnt /bin/bash -n riva
    # setup fqdn inside the ss-client container with Traefik svc IP
    $ kubectl get svc -A
    $ echo ' riva.nvda' >> /etc/hosts
    # test connectivity, exec into the client and run the following
    $ riva_streaming_asr_client --audio_file=/work/wav/vad_test_files/2094-142345-0010.wav --automatic_punctuation=false --riva_uri=riva.nvda:80
    # run benchmark
    $ for i in `seq 5`; do /usr/local/bin/riva_streaming_asr_client --num_parallel_requests=512 --num_iterations=2048 --audio_file=/work/wav/test/1272-135031-0000.wav --interim_results=false --automatic_punctuation=false --print_transcripts=false --chunk_duration_ms=800 --riva_uri=riva.nvda:80; done | tee output_config1_max_throughtput
  2. Monitor the GPU usage. Navigate into any of the riva-api pod (riva-speech container) in a separate terminal.

    # to monitor GPU usage
    $ kubectl exec --stdin --tty  /bin/bash -n riva -c riva-speech
    $ watch -n0.1 nvidia-smi

Scaling and Deleting the Cluster

The cluster and services can be scaled using the following commands:

# scaling the nodegroups
$ eksctl scale nodegroup --name=gpu-linux-workers --cluster=riva-cluster --nodes=8 --region=us-west-2 # or use the EKS UI

# now scale the riva api
$ kubectl scale deployments/riva-riva-api --replicas=8 -n riva

For deleting the cluster, run:

$ eksctl delete cluster riva-cluster --region=us-west-2