Virtual Assistant (with Google Dialogflow)¶

This Virtual Assistant (with Google Dialogflow) sample application demonstrates the integration of Google Dialogflow and Riva Speech Services in the form of a weather chatbot web application.

In this sample, we use Riva for ASR and TTS and Google Dialogflow for NLP and Dialog Management (DM).

Demo Video¶

To see how the Riva and Google Dialogflow weather chatbot service works, the demo video can be found here.

Implementation¶

At a high-level, the integration takes advantage of the native API support of Google Dialogflow and gRPC support in Riva. The Weatherbot Client coordinates the workflow with Riva Services and Dialogflow, then interacts with the end-user via a web UI. There are three primary parts to this solution; Riva AI Services, Dialogflow Weatherbot, and the Weatherbot Client application.

Here is the implementation at a high-level:

Riva AI Services
- Exposes Speech Services (ASR/NLP/TTS) over gRPC endpoints.
- Needs a GPU.
Riva and Dialogflow Chatbot
- Dialogflow Weatherbot
  Exposes API endpoints to communicate with the chatbot.
  
  Takes user text as input and returns a response.
  
  Responsible for fulfillment, when needed.
  
  Runs on GCP.
- Weatherbot Client application
  Includes the Riva Client Python library.
  
  Communicates with Riva AI Services and Dialogflow Weatherbot over gRPC and REST API endpoints respectively.
  
  Pipelines ASR, NLP, TTS, and dialog manager functionalities.
  
  Contains the Weatherbot Client application (web UI and web service).
  
  Does not need GPUs.

Architecture¶

The above diagram shows the architecture of the Riva and Dialogflow Weatherbot. Audio input from the user is collected through the microphone by the web UI of the Weatherbot Client application. The input audio from the user is sent to Riva AI Services for ASR, by the client application. Riva AI Services returns the transcribed text back to the Client application. The transcribed text from Riva AI Services is then sent to the Dialogflow Weatherbot (running on GCP). The Dialogflow Weatherbot returns the appropriate response for the text. The response text is then sent to Riva AI Services for TTS. A voice response is returned back to the client application, which is then played on the user’s speakers by the web UI.

Code Structure¶

This section shows the high-level code structure of the Weatherbot Client application (in Riva and Dialogflow Chatbot).

asr.py
- This file contains the functionality to make the gRPC call to Riva ASR, using the Riva Python Client libraries with the audio snippet and returns the text transcript.
- ASR is used in streaming mode
dialogflow.py
- This file contains the functionality to make an API call to Dialogflow, with the user input and sender ID and returns a text response obtained by Dialogflow.
tts.py and tts_stream.py
- These files contain the functionality to make the gRPC call to Riva TTS, using the Riva Python Client libraries, with a text snippet, and returns the corresponding audio speech.
- TTS can be used in either Batch or Streaming mode, depending on whether tts.py or tts_stream.py is used. This can be set by changing the import statements in lines 12 and 13 in the virtual-assistant-dialogflow/dialogflow-riva-weatherbot-webapp/riva/chatbot/chatbot.py script.
chatbot.py
- This file contains the Chatbot class which is responsible for pipelining all the ASR, TTS and Dialogflow operations.
- Creates one instance of the Chatbot class per conversation.

Running the Demo¶

Start Riva Speech Services per Riva Quick Start Guide.
Run Riva and the Dialogflow Virtual Assistant.
1. Run the Riva Sample container.
  Create a directory to hold the code for Riva and the Dialogflow Virtual Assistant.
  
  mkdir riva-dialogflow-va-temp
  
  Pull the Riva Sample container.
  
  docker pull nvcr.io/nvidia/riva/riva-speech-client:1.9.0-beta-samples
  
  Run the Riva Sample container.
  
  docker run -it --rm -p 6006:6006 -v <Path to riva-dialogflow-va-temp>:/riva-dialogflow-va-temp nvcr.io/nvidia/riva/riva-speech-client:1.9.0-beta-samples /bin/bash
  
  Navigate to the samples/virtual-assistant-dialogflow directory.
  
  cd samples/virtual-assistant-dialogflow
  
  Modify the API endpoint setting per the Network Configuration section.
2. Set up Google Dialogflow. The entire set up process for Dialogflow can take some time to complete. We have tried to complete as much of the set up as possible in the Docker container, however, the following steps must be completed.
  Read through Dialogflow Basics and About the Google Cloud Console to better understand the basics of Dialogflow.
  
  Follow the steps in Create a Project and Enable the API from the Dialogflow setup. Enable Billing and Enable audit logs are not needed for this demo.
  
  Follow the Set up Authentication instructions. When done, run the command in the Use the service account key file in your environment step in the Riva Samples container.
  
  export GOOGLE_APPLICATION_CREDENTIALS="<Path to key json file>"
  
  Install and initialize the Cloud SDK in the container, except for initilializing gcloud CLI. In the Riva Samples container, run:
  
  gcloud init
  
  During this command, you will need to provide your Project ID. To find your Project ID, perform the following steps:
  
  In the Google Cloud Platform (GCP) Dashboard, select your project from the top-left drop-down, found on the right side of the GCP banner.
  
  Under the DASHBOARD tab, the Project ID can be found in the Project Info section.
  
  Complete Test the SDK and authentication.
  
  gcloud auth application-default print-access-token
  
  Skip Install the Dialogflow client library. This step has been completed in the Riva Samples container, therefore, no action is needed.
  
  In config.py, update PROJECT_ID parameter with your project ID. To find your Project ID, perform the following steps:
  
  In the Google Cloud Platform (GCP) Dashboard, select your project from the top-left drop-down, found on the right side of the GCP banner.
  
  Under the DASHBOARD tab, the Project ID can be found in the Project Info section.
3. Initialize and start the Dialogflow Weatherbot.
  Copy the code from the Samples container to the host system.
  
  cp -r /workspace/samples/virtual-assistant-dialogflow/* /riva-dialogflow-va-temp/
  
  Follow the steps here to create an agent.
  
  Click the Setting button next to the agent name in the Dialogflow console. Under the Export and Import tab, choose Restore From ZIP and upload the zipped folder from your host at <Path to riva-dialogflow-va-temp>/dialogflow-weatherbot/dialogflow-weatherbot.zip.
  
  Add fullfillment.
  
  Open the Fulfillment section and enable the Inline Editor in the Dialogflow console.
  
  Copy and paste the contents of the <Path to riva-dialogflow-va-temp>/dialogflow-weatherbot/fulfillment/index.js into index.js under the Inline Editor.
  
  Copy and paste the contents of the <Path to riva-dialogflow-va-temp>/dialogflow-weatherbot/fulfillment/package.json into package.json under the Inline Editor.
  
  In index.js, at line 4, update the weatherstack_APIkey with your Weatherstack API key. A new Weatherstack API key can be obtained from here.
4. Start Riva and the Dialogflow chatbot client.
  Activate the chatbot client Python environment.
  
  . /pythonenvs/dialogflow/bin/activate
  
  Navigate to the chatbot client folder.
  
  cd dialogflow-riva-weatherbot-webapp
  
  Start the chatbot web server.
  
  python3 main.py
5. Open a Web UI on a web browser.
  https://<riva chatbot server host IP>:6006/rivaWeather
  For example:
  https://0.0.0.0:6006/rivaWeather

Network Configuration¶

Update the config.py script for inter-service communication.

# uncomment and populate the section below
riva_config = {
    ...
    "RIVA_SPEECH_API_URL": "[riva speech service host IP]:50051",
    ...

In the case where one is using _localhost_, the "RIVA_SPEECH_API_URL" must be set to the local machine’s IP, instead of _localhost_.

Sample Conversations¶

Sample 1¶

Virtual Assistant: Hi, welcome to RIVA weather service. How can I help you?

User: How’s the weather?

Virtual Assistant: For which location?

User: In Seattle.

Virtual Assistant: It is Partly cloudy in Seattle at the moment. The temperature is 16 degrees, the humidity is 97 percent and the wind speed is 0 miles per hour.

Sample 2¶

Virtual Assistant: Hi, welcome to RIVA weather service. How can I help you?

User: How’s the weather in Paris?

Virtual Assistant: It is Clear in Paris at the moment. The temperature is 20 degrees, the humidity is 73 percent and the wind speed is 9 miles per hour.

Sample 3¶

Virtual Assistant: Hi, welcome to RIVA weather service. How can I help you?

User: Hi

Virtual Assistant: Hey! How can I help you?

User: What is your name?

Virtual Assistant: My friends call me Ja-Di.

Sample 4¶

Virtual Assistant: Hi, welcome to RIVA weather service. How can I help you?

User: Good morning

Virtual Assistant: Hey! How can I help you?

User: See you around

Virtual Assistant: Bye

Limitations¶

The provided samples are not complete virtual assistants; but are intended as simple examples of how to build basic task-oriented chatbots with Riva. Consequently, the intent classifier and slot filling models have been trained with small amounts of data and are not expected to be highly accurate.
The Dialogflow Weatherbot sample supports intents for cloudy, humidity, rainfall, snow, sunny, temperature, weather and windy checks. It does not support general conversational queries or other domains.
The Dialogflow Weatherbot supports only 1 slot for city. It doesn’t take into account the day associated with the query while providing the response.
This sample supports up to four concurrent users. This restriction is not because of Riva, but because of the web framework (Flask and Flask-ScoketIO) that is being used by the client web application. The socket connection to stream audio to (TTS) and from (ASR) the user is unable to sustain more than four concurrent socket connections.

License¶

For applicable licenses, refer to the License section.