Launch UCF Studio

This section provides pointers and guides to get started building UCF applications using UCF Studio. It starts with building and deploying simple text based chatbot application using Speech skills and Dialog manager services. Next it explains how upgrade same application to speech based chatbot. This section should give good overview of using UCF Studio.

  1. Run the following command in a terminal:

    $ ucf_studio
    

Note

If running remotely via SSH, make sure that correct DISPLAY variable is exported before running the command.

UCF Studio can also be run remotely using other methods - Remote Access.

  1. Once Studio starts, you should see the following home screen:

UCF Studio Home

Repo Setup

Let’s start with the repository setup. NGC API Key needs to be set in order to get access to the NGC Repositories.

  1. Get the NGC API Key - Refer to Generating Your NGC API Key.

  2. Click on Registry from the Menu Bar at the top and then on Repo Setup.

UCF Studio - Repo Setup Menu
  1. This will open the Repo Setup dialog popup. Enter your NGC API Key and click on Ok.

UCF Studio - Repo Setup Dialog

It will take a few seconds for the operation to complete. If the entered API Key is valid, list of NGC Repositories will be populated and a success message popup will be shown.

Sync Microservices

Next we will synchronize microservices from NGC repository to list available microservices and view their information.

  1. Click on Registry from the Menu Bar at the top and then on Fast Repo Setup.

UCF Studio - Fast Repo Sync Menu
  1. This will open the Repo Sync popup dialog. Select UCF from the drop down list.

UCF Studio - Repo Sync Dialog
  1. Click on Sync. A progress dialog will pop up. The process may take up to a few seconds. Once the sync is complete, a success confirmation dialog will be displayed and the microservice list on the left will be populated.

UCF Studio - Microservice List

Create an Reference Application

This section demonstrates how to create a Text based Chatbot using UCF Studio with microservices provided by NVIDIA. Some instructions on this page are specific to the applications being created, refer to Creating an Application for more detailed information on creating applications.

Prerequisites

Refer to Development System and Deployment System for requirements and instructions to setup development and deployment systems.

Obtain API keys for the following services, as microservices used in this reference application depends on them:

Create New Graph

We can now start creating the application graph.

  1. Click on Create Graph at the center of the application.

UCF Studio - Create Graph
  1. This will display pop-up dialog for Graph Name. Enter a name like ucf-text-bot and click on Ok.

UCF Studio - Create Graph Name
  1. Click on Ok to open a blank canvas on which microservices can be added.

Search for Microservices and their Details

  1. You may browse the available microservices using the list on the left

UCF Studio - Microservice List
  1. Microservices categorization can be changed for easier browsing

UCF Studio - Microservice categorization
  1. Microservice can be filtered and searched using text

UCF Studio - Microservice Filtering
  1. Select a microservice from the list to show its details on the right panel

UCF Studio - Microservice Details

Add Microservices

1. Microservices can be added to the application by selecting them from the list on the left and drag-dropping them on to the canvas.

UCF Studio - Adding Microservices
  1. For this reference application, the following microservices will be required:

  • Riva Speech Skills

  • BotMaker Dialog Manager

  • BotMaker NLP Server

  • BotMaker Fulfillment Server

  • BotMaker Editor

  • BotMaker Text Web App

  1. After adding above microservices, the graph now looks like the following:

UCF Studio - Adding Microservices

Connect Microservices

Microservice can be connected by connecting their endpoints. Start dragging from the egress endpoint (ports on the right side of a microservice) to the ingress endpoint (ports on the left side of a microservice).

UCF Studio - Connecting Microservices

Configure Microservices

  1. Parameters can be set by selecting a microservice on the canvas and setting the parameters from the “Properties” window on the right.

UCF Studio - Setting Microservice Parameters
  1. For convenience, parameters can also be loaded from a file. Save the following content to a file (e.g. params.yaml.)

'BotMaker Dialog Manager':
  configNgcPath: eevaigoeixww/staging/botmaker-quickstart:23.10.2-ea
  botConfigName: bots/misty/misty_bot_config.yaml
  imagePullSecrets:
    - name: ngc-docker-reg-secret

'Riva Speech Skills':
  ucfVisibleGpus: [0]
  modelRepoGenerator:
    clearAllRMIRSAndModels: false
    #> description: Flag to clear artifacts and models before downloading and deploying
    ngcModelConfigs:
      tritonGroup0:
        models:
        #> description: List of NGC models for deployment
        - nvidia/riva/rmir_nlp_named_entity_recognition_bert_base:2.13.0
        - eevaigoeixww/conversational-ai/rmir_nlp_weather_intent_slot_bert_base:2.0.0-ea
  persistentVolumeClaim:
    keepPVC: True
  imagePullSecrets:
    - name: ngc-docker-reg-secret

'BotMaker NLP Server':
  ucfVisibleGpus: [0]
  configNgcPath: "eevaigoeixww/staging/botmaker-quickstart:23.10.2-ea"
  modelConfigPath: "bots/misty/model_config.yaml"
  imagePullSecrets:
    - name: ngc-docker-reg-secret

'BotMaker Fulfillment Server':
  configNgcPath: "eevaigoeixww/staging/botmaker-quickstart:23.10.2-ea"
  fmconfigPath: "bots/misty/fulfillment_config.yaml"
  imagePullSecrets:
    - name: ngc-docker-reg-secret

'BotMaker Text Web App':
  imagePullSecrets:
    - name: ngc-docker-reg-secret

'BotMaker Editor':
  imagePullSecrets:
    - name: ngc-docker-reg-secret
  1. The API keys can be obtained from:

  1. To load the above parameters file, use Load Parameters button from the actions bar.

UCF Studio - Load Parameters
  1. This will open a file browser, navigate to the parameters file and load it.

UCF Studio - Load Parameters File
  1. Set secrets on the microservices. First click the Graph Configuration button from the actions bar to open the graph configuration.

UCF Studio - Open Graph Configuration

Next select the Secrets window, click on Add to add a new secret. Enter details as shown below:

UCF Studio - Add Secret

Now select Riva Speech Skills microservice on the canvas, this will show the Secrets section in the Property window on the right. For the ngc-api-key-secret, select ngc-api-key-secret from the dropdown.

UCF Studio - Set Secret

Do the same for the BotMaker Dialog Manager, BotMaker NLP Server, and BotMaker Fulfillment Server.

Save the Graph

  1. Click on the Save Graph icon from the actions bar

UCF Studio - Save Graph
  1. This will open a file browser. Navigate to the directory of choice and save the graph.

Build the Reference Application

  1. Click on the Build the current Graph icon from the actions bar.

UCF Studio - Build Graph
  1. This will open a file browser. Navigate to the directory of choice and click on Okay.

  2. Building an application may take up to a few minutes. Once it is done, a success pop-up dialog will be shown containing the path where the deployable helm chart for the application has been generated.

UCF Studio - Build Graph
  1. Refer to the Deploy the Reference Application for instructions on deploying the helm chart.

Update the Reference Application

This section demonstrates how to create a Speech based Chatbot using UCF Studio with microservices provided by NVIDIA. These are incremental instructions on top to follow after creating the Create an Reference Application. This demonstrates how increasingly complex functionality can be added by just changing a few things in the application using UCF Studio.

Some instructions on this page are specific to the applications being created, refer to Creating an Application for more detailed information on creating applications.

Open Existing Graph

  1. If the UCF Text Bot Application application graph is not already open, open it by clicking on the Open Graph button at the center of the application or Open Graph button on the action bar.

UCF Studio - Open Graph
  1. This will open a file browser, navigate to the UCF Text Bot Application application graph and open it. This will load the application graph in the canvas. If some of the microservices are out of view, right-click on the canvas to open the context menu and click on Focus on All to automatically pan and zoom the canvas so that all microservices are visible.

UCF Studio - Graph Focus All

Add / Replace Microservices

For this application:

  • BotMaker Text Web App should be replaced by BotMaker Speech Web App

  • Additional BotMaker Bot Controller microservice will be required

  1. To do this select the BotMaker Text Web App microservice instance on the canvas and then right-click on it to open the context menu and click on Delete Selection to delete the microservice.

UCF Studio - Delete Microservice
  1. Next, add the BotMaker Speech Web App and BotMaker Bot Controller microservices by searching for them in the microservice list on the left and drag-dropping them on to the canvas. This is how the graph should look now:

UCF Studio - Add Speech Microservices

Connect Microservices

  • bot-controller egress endpoint of BotMaker Speech Web App needs to be connected to grpc-api ingress endpoint of BotMaker Bot Controller

  • bot-editor egress endpoint of BotMaker Speech Web App needs to be connected to http-api ingress endpoint of BotMaker Editor

  • dialog-manager egress endpoint of BotMaker Bot Controller needs to be connected to dialog-manager ingress endpoint of BotMaker Dialog Manager

  • riva egress endpoint of BotMaker Bot Controller needs to be connected to riva-speech-api ingress endpoint of Riva Speech Skills

Do this by dragging from the egress endpoints to the ingress endpoints as shown in Connect Microservices. The graph should now look as below:

UCF Studio - Connect Speech Microservices

Load Parameters

  1. Save the following content to a file (e.g. params.yaml):

'BotMaker Dialog Manager':
   configNgcPath: eevaigoeixww/staging/botmaker-quickstart:23.10.2-ea
   botConfigName: bots/misty/misty_bot_config.yaml
   imagePullSecrets:
     - name: ngc-docker-reg-secret

 'Riva Speech Skills':
   ucfVisibleGpus: [0]
   modelRepoGenerator:
     clearAllRMIRSAndModels: false
     #> description: Flag to clear artifacts and models before downloading and deploying
     ngcModelConfigs:
       tritonGroup0:
         models:
         #> description: List of NGC models for deployment
         - nvidia/riva/rmir_nlp_named_entity_recognition_bert_base:2.13.0
         - eevaigoeixww/conversational-ai/rmir_nlp_weather_intent_slot_bert_base:2.0.0-ea
         - eevaigoeixww/staging/asr_conformer_en_us_streaming_throughput_flashlight_vad:2.13.0-tokkio #english
         - nvidia/riva/rmir_tts_fastpitch_hifigan_en_us:2.13.0
   persistentVolumeClaim:
     keepPVC: True
   imagePullSecrets:
     - name: ngc-docker-reg-secret

 'BotMaker NLP Server':
   ucfVisibleGpus: [0]
   configNgcPath: "eevaigoeixww/staging/botmaker-quickstart:23.10.2-ea"
   modelConfigPath: "bots/misty/model_config.yaml"
   imagePullSecrets:
     - name: ngc-docker-reg-secret

 'BotMaker Fulfillment Server':
   configNgcPath: "eevaigoeixww/staging/botmaker-quickstart:23.10.2-ea"
   fmconfigPath: "bots/misty/fulfillment_config.yaml"
   imagePullSecrets:
       - name: ngc-docker-reg-secret

 'BotMaker Speech Web App':
   imagePullSecrets:
     - name: ngc-docker-reg-secret

 'BotMaker Editor':
   imagePullSecrets:
     - name: ngc-docker-reg-secret

 'BotMaker Bot Controller':
   pipeline: speech_lite
   configNgcPath: "eevaigoeixww/staging/botmaker-quickstart:23.10.2-ea"
   speechConfigPath: "bots/misty/speech_config.yaml"
   imagePullSecrets:
     - name: ngc-docker-reg-secret
  1. Next, follow acquire API keys and load parameters as shown in UCF Text Bot Application section.

  2. Set secrets on BotMaker Bot Controller microservice as shown in UCF Text Bot Application - Set Secrets.

Update Application Information

  1. click the Graph Configuration button from the actions bar to open the graph configuration.

UCF Studio - Open Graph Configuration
  1. This will open the Graph Configuration section at the bottom.

UCF Studio - Graph Configuration Window
  1. Change the application name to ucf-speech-bot and optionally update the description.

Save and Build the Application

  1. From the the File menu at the top, select Save Graph As to the save the graph to a new file.

UCF Studio - Graph Save As
  1. This will open a file browser, navigate to a directory of choice and save the graph.

  2. Follow instructions in Build the Reference Application in the UCF Text Bot Application application section to build the application.

Deploy the Reference Application

Deploy on local MicroK8S cluster

  1. Create Kubernetes secret required to pull the images:

$ microk8s kubectl create secret docker-registry ngc-docker-reg-secret --docker-server=nvcr.io --docker-username='$oauthtoken' --docker-password=<NGC_API_KEY>

If another name is used for the secret instead of ngc-docker-reg-secret, same should be updated in the application and the app should be rebuilt.

  1. Create Kubernetes secret for the NGC API Key:

$ microk8s kubectl create secret generic ngc-api-key-secret --from-literal=NGC_API_KEY=<NGC_API_KEY>

If another secret name is used instead of ngc-api-key-secret or another key is used instead of NGC_API_KEY, same should be updated in the application and the app should be rebuilt.

  1. Install the Local Path Provisioner if not already installed:

$ curl https://raw.githubusercontent.com/rancher/local-path-provisioner/v0.0.23/deploy/local-path-storage.yaml | sed 's/^  name: local-path$/  name: mdx-local-path/g' | microk8s kubectl apply -f -
  1. Deploy the application helm chart:

$ microk8s helm3 install <release-name> <app-output-helm-chart>

Deploy application on AWS

  1. Copy helm chart to AWS Instance:

scp -r -i /path/to/<your_key>.pem /path/to/<app-output-helm-chart>/   ubuntu@<aws_instance_ip>:
  1. Install the Local Path Provisioner (on AWS Instance):

$ curl https://raw.githubusercontent.com/rancher/local-path-provisioner/v0.0.23/deploy/local-path-storage.yaml | sed 's/^  name: local-path$/  name: mdx-local-path/g' | kubectl apply -f -
  1. Create Kubernetes secret required to pull the images (on AWS Instance):

$ kubectl create secret docker-registry ngc-docker-reg-secret --docker-server=nvcr.io --docker-username='$oauthtoken' --docker-password=<NGC_API_KEY>

If another name is used for the secret instead of ngc-docker-reg-secret, same should be updated in the application and the app should be rebuilt.

  1. Create Kubernetes secret for the NGC API Key (on AWS Instance):

$ kubectl create secret generic ngc-api-key-secret --from-literal=NGC_API_KEY=<NGC_API_KEY>
  1. If another secret name is used instead of ngc-api-key-secret or another key is used instead of NGC_API_KEY, same should be updated in the application and the app should be rebuilt.

  2. Deploy the application helm chart (on AWS Instance):

$ helm install <release-name> <app-output-helm-chart>

Interact with the Reference Application

UCF Text Bot Application

Watch Pods:

Wait for all the application microservice pods to go into the running state

MicroK8s:

$ microk8s kubectl get pod
NAME                                                  READY   STATUS    RESTARTS     AGE
botmaker-dialog-manager-deployment-6d9d6fd949-88f7f   1/1     Running   0            28m
botmaker-text-web-app-deployment-8499c4b857-xjwcg     1/1     Running   0            28m
riva-speech-skills-deployment-74cc4c9457-mfm98        1/1     Running   0            28m

AWS:

$ kubectl get pods
NAME                                                  READY   STATUS    RESTARTS   AGE
botmaker-dialog-manager-deployment-ffdd98965-49rbk    1/1     Running   0          48m
botmaker-text-web-app-deployment-ddccf454-wsxh6       1/1     Running   0          48m
riva-speech-skills-deployment-7856cc499f-9bnkf        1/1     Running   0          48m

Service List

Get the list of services associated with the application

MicroK8s:

$ microk8s kubectl get svc
NAME                                       TYPE        CLUSTER-IP       EXTERNAL-IP   PORT(S)              AGE
botmaker-dialog-manager-deployment-svc     ClusterIP   10.152.183.47    <none>        5000/TCP             32m
botmaker-text-web-app-deployment-service   NodePort    10.152.183.132   <none>        5001:31842/TCP       32m
kubernetes                                 ClusterIP   10.152.183.1     <none>        443/TCP              67d
riva-speech-skills-deployment-svc          ClusterIP   10.152.183.21    <none>        50051/TCP,8001/TCP   32m   29m

AWS:

$ kubectl get svc
NAME                                        TYPE        CLUSTER-IP       EXTERNAL-IP   PORT(S)              AGE
botmaker-dialog-manager-deployment-svc      ClusterIP   10.102.53.150    <none>        5000/TCP             49m
botmaker-text-web-app-deployment-service    NodePort    10.96.23.19      <none>        5001:31666/TCP       49m
kubernetes                                  ClusterIP   10.96.0.1        <none>        443/TCP              151m
riva-speech-skills-deployment-svc           ClusterIP   10.101.176.114   <none>        50051/TCP,8001/TCP   49m

Accessing the WebApp

The service of interest is botmaker-text-web-app-deployment-service. We will need the Node Port associated with the service. This port number will change for every deployment.

MicroK8s:

NodePort: 31842

Open http://<MicroK8s-Host-IP>:<Botmaker-Text-Web-App-NodePort>/bot in a browser to start interacting with the bot.

AWS:

NodePort: 31666

We will have to open this port to the internet so that we can access it.

  1. Go to your instance’s Security tab and navigate to the security group. Edit the Inbound Rule to accept connections to the NodePort (in this case 31666) from Anywhere-IPv4 (0.0.0.0/0).

  2. After saving open https://<aws_instance_ip>:<NodePort>/bot in a browser to start interacting with the bot.

Application WebUI

On opening the URL, you should be greeted with the following UI:

UCF Text Bot App User Interface
  • You can enter the question/statement in the text box and click on Submit.

  • The bot answers to questions/statements like “Hello”, “How are you?”, “How is the weather in New York?”

UCF Speech Bot

Follow the steps in UCF Text Bot Application to wait till all pods are running and to get the list of services.

Accessing the WebApp

The service of interest is botmaker-speech-webapp-deployment-service. We will need the Node Port associated with the service. This port number will change for every deployment.

MicroK8s:

NodePort: 31746

Open https://<MicroK8s-Host-IP>:<Botmaker-Speech-WebApp-NodePort>/bot in a browser to start interacting with the bot.

AWS:

NodePort: 31666

We will have to open this port to the internet so that we can access it.

  1. Go to your instance’s Security tab and navigate to the security group. Edit the Inbound Rule to accept connections to the NodePort (in this case 31666) from Anywhere-IPv4 (0.0.0.0/0).

  2. After saving open https://<aws_instance_ip>:<NodePort>/bot in a browser to start interacting with the bot.

Application WebUI

On opening the URL, you should be greeted with the following UI:

UCF Speech Bot App User Interface
  • After giving permissions to your microphone, you should be able to provide speech input in the WebApp.

  • Click on the green mic button to talk to the bot.

  • Click on the yellow “Unmute Speech” button to hear audio replies from the bot.

  • Alternatively, you can enter the question/statement in the text box and click on Submit and/or view the replies in text format

  • The bot answers to questions/statements like “Hello”, “How are you?”, “How is the weather in New York?”