Virtual Assistant (with Google Dialogflow)
Contents

Virtual Assistant (with Google Dialogflow)#
This Virtual Assistant (with Google Dialogflow) sample application demonstrates the integration of Google Dialogflow and Riva Speech Services in the form of a weather chatbot web application.
In this sample, we use Riva for ASR and TTS and Google Dialogflow for NLP and Dialog Management (DM).
Demo Video#
To see how the Riva and Google Dialogflow weather chatbot service works, the demo video can be found here.
Implementation#
At a high-level, the integration takes advantage of the native API support of Google Dialogflow and gRPC support in Riva. The Weatherbot Client coordinates the workflow with Riva Services and Dialogflow, then interacts with the end-user via a web UI. There are three primary parts to this solution; Riva AI Services, Dialogflow Weatherbot, and the Weatherbot Client application.
Here is the implementation at a high-level:
Riva AI Services
Exposes Speech Services (ASR/NLP/TTS) over gRPC endpoints.
Needs a GPU.
Riva and Dialogflow Chatbot
Dialogflow Weatherbot
Exposes API endpoints to communicate with the chatbot.
Takes user text as input and returns a response.
Responsible for fulfillment, when needed.
Runs on GCP.
Weatherbot Client application
Includes the Riva Client Python library.
Communicates with Riva AI Services and Dialogflow Weatherbot over gRPC and REST API endpoints respectively.
Pipelines ASR, NLP, TTS, and dialog manager functionalities.
Contains the Weatherbot Client application (web UI and web service).
Does not need GPUs.
Architecture#
The above diagram shows the architecture of the Riva and Dialogflow Weatherbot. Audio input from the user is collected through the microphone by the web UI of the Weatherbot Client application. The input audio from the user is sent to Riva AI Services for ASR, by the client application. Riva AI Services returns the transcribed text back to the Client application. The transcribed text from Riva AI Services is then sent to the Dialogflow Weatherbot (running on GCP). The Dialogflow Weatherbot returns the appropriate response for the text. The response text is then sent to Riva AI Services for TTS. A voice response is returned back to the client application, which is then played on the user’s speakers by the web UI.
Code Structure#
This section shows the high-level code structure of the Weatherbot Client application (in Riva and Dialogflow Chatbot).
asr.py
This file contains the functionality to make the gRPC call to Riva ASR, using the Riva Python Client libraries with the audio snippet and returns the text transcript.
ASR is used in streaming mode
dialogflow.py
This file contains the functionality to make an API call to Dialogflow, with the user input and sender ID and returns a text response obtained by Dialogflow.
tts.py
andtts_stream.py
These files contain the functionality to make the gRPC call to Riva TTS, using the Riva Python Client libraries, with a text snippet, and returns the corresponding audio speech.
TTS can be used in either Batch or Streaming mode, depending on whether
tts.py
ortts_stream.py
is used. This can be set by changing the import statements in lines 12 and 13 in thevirtual-assistant-dialogflow/dialogflow-riva-weatherbot-webapp/riva/chatbot/chatbot.py
script.
chatbot.py
This file contains the Chatbot class which is responsible for pipelining all the ASR, TTS and Dialogflow operations.
Creates one instance of the Chatbot class per conversation.
Requirements and Setup#
Requirements:#
Before you try running the Riva client, ensure you meet the following requirements:
You have access and are logged into NVIDIA NGC. For step-by-step instructions, refer to the NGC Getting Started Guide.
Python 3.8 (Support for other Python versions will be added in a future release).
Setup:#
Clone Riva Sample Apps repository:
git clone https://github.com/nvidia-riva/sample-apps.git
Enter Riva and Dialogflow Virtual Assistant directory:
cd sample-apps/virtual-assistant-dialogflow
Create parent directory for the Python virtual environment we will be using for this sample:
mkdir pythonenvs
Create and activate Python virtual environment for this sample. We will be using this virtual environment for all dependent libraries, including Riva client library, weatherbot web application libraries, and Google Dialogflow client library:
python3 -m venv pythonenvs/dialogflow
. pythonenvs/dialogflow/bin/activate
Install the libraries necessary for the virtual assistant, including the Riva client library:
Upgrade
pip
:
pip3 install -U pip
Install Riva client libraries:
Download the Riva Quick Start scripts, if not already done.
x.y.z
is the Riva Speech Skills version number - The latest Riva version number can be found in the Riva Quick Start Guide’s Local Deploymnent using Quick Start Scripts section
ngc registry resource download-version "nvidia/riva/riva_quickstart:x.y.z"
Install the Riva client library.
cd riva_quickstart_v<x.y.z> pip install riva_api-<x.y.z>-py3-none-any.whl
Install weatherbot web application dependencies.
requirements.txt
captures all Python dependencies needed for weatherbot web application:
pip3 install -r requirements.txt # For Python 3.8
Set up Google Dialogflow. The entire set up process for Dialogflow consists of multiple steps and can take some time to complete.
At the end of this step, you would have setup a Google Project, installed and initialized gcloud CLI, and installed the Google Dialogflow client library in thedialogflow
virtual environment that we created in the previous step.In
virtual-assistant-dialogflow
’sconfig.py
, updatePROJECT_ID
parameter with your project ID. To find your Project ID, perform the following steps:In the Google Cloud Platform (GCP) Dashboard, select your project from the top-left drop-down, found on the right side of the GCP banner.
Under the DASHBOARD tab, the Project ID can be found in the Project Info section.
Deactivate Weatherbot web application’s
dialogflow
Python virtual environment:
deactivate
Running the Demo#
Start the Riva Speech Server, if not already done. Follow the steps in the Riva Quick Start Guide.
Navigate to the Riva and Dialogflow Virtual Assistant directory in the Riva sample-apps github repository that you cloned in the Setup section’s, step 1.
cd sample-apps/virtual-assistant-dialogflow
Modify the API endpoint setting in the code base for inter-service communication - Update the
config.py
script for inter-service communication.
riva_config = {
...
"RIVA_SPEECH_API_URL": "[riva speech service host IP]:50051",
...
}
For example:
# uncomment and populate the section below
riva_config = {
...
"RIVA_SPEECH_API_URL": "10.20.30.40:50051",
...
}
Initialize and start the Dialogflow Weatherbot.
Follow the steps here to create an agent.
Click the Setting button next to the agent name in the Dialogflow console. Under the Export and Import tab, choose Restore From ZIP and upload the zipped folder from your host at
sample-apps/virtual-assistant-dialogflow/dialogflow-weatherbot/dialogflow-weatherbot.zip
.Add fullfillment.
Open the Fulfillment section and enable the Inline Editor in the Dialogflow console.
Copy and paste the contents of the
sample-apps/virtual-assistant-dialogflow/dialogflow-weatherbot/fulfillment/index.js
intoindex.js
under the Inline Editor.Copy and paste the contents of the
sample-apps/virtual-assistant-dialogflow/dialogflow-weatherbot/fulfillment/package.json
intopackage.json
under the Inline Editor.In
index.js
, at line 4, update theweatherstack_APIkey
with your Weatherstack API key. A new Weatherstack API key can be obtained from here.
Start Riva and Dialogflow virtual assistant web application.
If not already done, update
config.py
with the Dialogflow weatherbot’s Project ID as described in the Setup section, step 7.Activate the
dialogflow
Python environment.
. /pythonenvs/dialogflow/bin/activate
Navigate to the chatbot web application folder.
cd dialogflow-riva-weatherbot-webapp
Start the chatbot web server.
python3 main.py
Open the interface in a web browser
https://<riva chatbot server host IP>:6006/rivaWeather
, e.g. https://0.0.0.0:6006/rivaWeather.
Sample Conversations#
Sample 1#
Virtual Assistant: Hi, welcome to RIVA weather service. How can I help you?
User: How’s the weather?
Virtual Assistant: For which location?
User: In Seattle.
Virtual Assistant: It is Partly cloudy in Seattle at the moment. The temperature is 16 degrees, the humidity is 97 percent and the wind speed is 0 miles per hour.
Sample 2#
Virtual Assistant: Hi, welcome to RIVA weather service. How can I help you?
User: How’s the weather in Paris?
Virtual Assistant: It is Clear in Paris at the moment. The temperature is 20 degrees, the humidity is 73 percent and the wind speed is 9 miles per hour.
Sample 3#
Virtual Assistant: Hi, welcome to RIVA weather service. How can I help you?
User: Hi
Virtual Assistant: Hey! How can I help you?
User: What is your name?
Virtual Assistant: My friends call me Ja-Di.
Sample 4#
Virtual Assistant: Hi, welcome to RIVA weather service. How can I help you?
User: Good morning
Virtual Assistant: Hey! How can I help you?
User: See you around
Virtual Assistant: Bye
Limitations#
The provided samples are not complete virtual assistants; but are intended as simple examples of how to build basic task-oriented chatbots with Riva. Consequently, the intent classifier and slot filling models have been trained with small amounts of data and are not expected to be highly accurate.
The Dialogflow Weatherbot sample supports intents for cloudy, humidity, rainfall, snow, sunny, temperature, weather and windy checks. It does not support general conversational queries or other domains.
The Dialogflow Weatherbot supports only 1 slot for
city
. It doesn’t take into account the day associated with the query while providing the response.This sample supports up to four concurrent users. This restriction is not because of Riva, but because of the web framework (Flask and Flask-ScoketIO) that is being used by the client web application. The socket connection to stream audio to (TTS) and from (ASR) the user is unable to sustain more than four concurrent socket connections.
License#
The NVIDIA Riva License Agreement is included with the product. Licenses are also available along with the model application zip file. By pulling and using the Riva SDK container, downloading models, or using the sample applications here, you accept the terms and conditions of these licenses.
Here is the license information for the libraries we are using in this sample:
google-cloud-dialogflow - The License for this library can be found here.
actions-on-google - The License for this library can be found here.
firebase-admin - The License for this library can be found here.
firebase-functions - The License for this library can be found here.
dialogflow - The License for this library can be found here.
dialogflow-fulfillment - The License for this library can be found here.