Virtual Assistant (with Rasa)¶
This Virtual Assistant (with Rasa) sample application demonstrates the integration of Rasa and the Riva Speech Service in the form of a weather chatbot web application. This sample is available in two options:
Option 1: Riva ASR + Riva TTS + Riva NLP + Rasa dialog manager
Option 2: Riva ASR + Riva TTS + Rasa NLU + Rasa dialog manager
To learn more about Rasa, follow this link.
Implementation¶
At a high-level, the integration takes advantage of the native API support of Rasa and gRPC support of Riva. The Weatherbot Client coordinates the workflow with Riva Services and Rasa, then interacts with the end-user via a web UI.
Container 1: Riva AI Services
Exposes Speech Services (ASR/NLP/TTS) over gRPC endpoints
Needs a GPU
Container 2: Riva Samples’ Riva-Rasa Chatbot
Functionality
Exposes Rasa functionality over API endpoints. Responsible for dialog management with Rasa as well as NLP with Riva or NLU with Rasa
Contains the Weatherbot Client application (web UI and web service)
Rasa
Up to two instances
Rasa Server
Rasa Action (if applicable)
Does not need GPUs but can be deployed on GPUs for performance
Weatherbot Client
Includes the Riva Python library installed
Communicates with Riva AI Services and Rasa over gRPC and REST API endpoints respectively
Pipelines ASR, NLP, TTS, and dialog manager functionalities
Does not need GPUs
Code Structure¶
This section shows the high-level code structure of the Weatherbot Client (in Riva Samples’ Riva-Rasa Chatbot).
asr.py
This file contains the functionality to make the gRPC call to Riva ASR, using the Riva Python Client libraries, with the audio snippet, and returns the text transcript.
ASR is used in streaming mode
rasa.py
This file contains the functionality to make an API call to Rasa, with the user input and sender ID, and returns a text response obtained by processing the Rasa response object.
tts.py
andtts\_stream.py
These files contain the functionality to make the gRPC call to Riva TTS, using the Riva Python Client libraries, with a text snippet, and returns the corresponding audio speech.
TTS can be used in either
Batch
orStreaming
mode, depending on whethertts.py
ortts\_stream.py
is used. This can be set by changing the import statements in lines 12 and 13 invirtual-assistant-rasa/rasa-riva-weatherbot-webapp/riva/chatbot/chatbot.py
.
chatbot.py
This file contains the
Chatbot
class which is responsible for pipelining all the ASR, TTS, and Rasa operations.Creates one instance of the
Chatbot
class per conversation.Pipeline is as follows:
ASR is used in
Streaming
mode, therefore, it’s a background operation. By default, we are listening to the audio of the user who is speaking into the microphone.ASR calls Rasa with the transcribed text either automatically when the user stops talking (when the
AutoSubmit
flag is enabled) or when the Submit button is hit.When called, Rasa internally calls Riva NLP or Rasa NLU with the transcribed text to fetch the intents and slots which is then used by the Rasa dialog manager to get the final response text. The response text is returned.
TTS is automatically called with Rasa’s response text, if
System Speech
is not muted.TTS gets the audio snippet and plays it back to the user on the speakers.
Network Configuration¶
There are two locations in the code base that have to be configured for inter-service communication:
virtual-assistant-rasa/rasa-weatherbot/endpoints.yml
# uncomment and populate the section below action_endpoint: url: "http://[rasa server host IP]:5055/webhook"
For example:
# uncomment and populate the section below action_endpoint: url: "http://10.20.30.40:5055/webhook"
virtual-assistant-rasa/config.py
# uncomment and populate the section below riva_config = { ... "RIVA_SPEECH_API_URL": "[riva speech service host IP]:50051", ... # uncomment and populate the section below rasa_config = { ... "RASA_API_URL": "[rasa server host IP]:5005", ...
In the case where one is using
localhost
the[rasa server host IP]
must be set to the local machine’s IP, instead oflocalhost
.If the Riva Service, Rasa server, and the Weatherbot Client are all running on the same machine, it would be:
# uncomment and populate the section below riva_config = { ... "RIVA_SPEECH_API_URL": "10.20.30.40:50051", ... # uncomment and populate the section below rasa_config = { ... "RASA_API_URL": "10.20.30.40:5005", ...
If the Riva Service runs on a GPU that is supported by a remote machine, and Rasa and the Weatherbot Client are running on a local machine, it would be:
# uncomment and populate the section below riva_config = { ... "RIVA_SPEECH_API_URL": "10.20.30.40:50051", ... # uncomment and populate the section below rasa_config = { ... "RASA_API_URL": "20.40.60.80:5005", ...
Running the Demo¶
Start the Riva Speech Server. Follow the steps in the Quick Start Guide.
Start the Rasa Action server.
Run the Riva Sample container.
Pull the Riva Sample container.
docker pull nvcr.io/nvidia/riva/riva-speech-client:1.9.0-beta-samplesRun the Riva Sample container.
docker run -it --rm -p 5055:5055 nvcr.io/nvidia/riva/riva-speech-client:1.9.0-beta-samples /bin/bashNavigate to the
samples/virtual-assistant-rasa
directory.cd samples/virtual-assistant-rasaModify the API endpoint setting, if needed, per the Network Configuration section. Ensure you modify the API endpoints for each of the three container instances.
Start the Rasa Action server.
Open the
config.py
script. In the dictionary on the right side of theriva_config
variable, update theWEATHERSTACK_ACCESS_KEY
field with your Weatherstack API key. A new Weatherstack API key can be obtained here.Activate the Rasa Python environment.
. /pythonenvs/rasa/bin/activateNavigate to the
rasa-weatherbot
directory.cd rasa-weatherbotRun the Rasa Action server.
rasa run actions --actions actionsStart the Rasa server in a different terminal.
Run the Riva Sample container.
Pull the Riva Sample container.
docker pull nvcr.io/nvidia/riva/riva-speech-client:1.9.0-beta-samplesRun the Riva Sample container.
docker run -it --rm -p 5005:5005 nvcr.io/nvidia/riva/riva-speech-client:1.9.0-beta-samples /bin/bashNavigate to the
cd samples/virtual-assistant-rasa
directory.cd samples/virtual-assistant-rasaModify the API endpoint setting, if needed, per the Network Configuration section. Ensure you modify the API endpoints for each of the three container instances.
Start the Rasa server.
Activate the Rasa Python environment.
. /pythonenvs/rasa/bin/activateNavigate to the
rasa-weatherbot
directory.cd rasa-weatherbotRun the Rasa training.
For Riva NLP: Train the Rasa Core model.
rasa train -c config/config_rivanlp.yml \ -d domain/domain_rivanlp.yml \ --out models/models_rivanlp/ \ --data data/nlu_rivanlp.yml \ data/rules_rivanlp.yml \ data/stories_rivanlp.ymlFor Rasa NLU: Train the Rasa NLU and Rasa Core models.
rasa train -c config/config_rasanlp.yml \ -d domain/domain_rasanlp.yml \ --out models/models_rasanlp/ \ --data data/nlu_rasanlp.yml \ data/rules_rasanlp.yml data/stories_rasanlp.ymlRun the Rasa server.
For Riva NLP.
rasa run -m models/models_rivanlp/ --enable-api \ --log-file out.log --endpoints endpoints.ymlFor Rasa NLU.
rasa run -m models/models_rasanlp/ --enable-api \ --log-file out.log --endpoints endpoints.ymlStart the Riva Chatbot server. Configure the API endpoints for both Riva Speech Service and Rasa. Refer to the Network Configuration section for more information.
Run the Riva Sample container.
Pull the Riva Sample container.
docker pull nvcr.io/nvidia/riva/riva-speech-client:1.9.0-beta-samplesRun the Riva Sample container.
docker run -it --rm -p 5555:5555 nvcr.io/nvidia/riva/riva-speech-client:1.9.0-beta-samples /bin/bashNavigate to the
samples/virtual-assistant-rasa
directory.cd samples/virtual-assistant-rasaModify the API endpoint setting, if needed, per the Network Configuration section. Ensure you modify the API endpoints for each of the three container instances.
Start the Riva Chatbot Client.
Activate the Chatbot Client Python environment.
. /pythonenvs/client/bin/activateNavigate to the
rasa-riva-weatherbot-webapp
directory.cd rasa-riva-weatherbot-webappStarting the Chatbot Client server.
python3 main.pyOpen the web UI on a web browser.
https://[riva chatbot server host IP]:5555/rivaWeatherFor example:
[https://0.0.0.0:5555/rivaWeather](https://0.0.0.0:5555/rivaWeather)
Sample Conversations¶
Rasa NLU: Sample 1¶
Virtual Assistant: Hi, welcome to RIVA weather service. How can I help you?
User: How’s the weather?
Virtual Assistant: For which location?
User: In Seattle.
Virtual Assistant: It is Haze, Smoke in Seattle at the moment. The temperature is 31 degrees, the humidity is 30 percent and the wind speed is 20 miles per hour.
Rasa NLU: Sample 2¶
Virtual Assistant: Hi, welcome to RIVA weather service. How can I help you?
User: How’s the weather in Paris?
Virtual Assistant: It is Clear in Paris at the moment. The temperature is 20 degrees, the humidity is 73 percent and the wind speed is 9 miles per hour.
Rasa NLU: Sample 3¶
Virtual Assistant: Hi, welcome to RIVA weather service. How can I help you?
User: hi
Virtual Assistant: Hey! How are you?
User: who are you?
Virtual Assistant: I am a bot, powered by Rasa.
Rasa NLU: Sample 4¶
Virtual Assistant: Hi, welcome to RIVA weather service. How can I help you?
User: Good Morning
Virtual Assistant: Hey! How are you?
User: I am doing great
Virtual Assistant: Great, carry on!
User: See you around
Virtual Assistant: Bye
Riva NLP: Sample 1¶
Virtual Assistant: Hi, welcome to RIVA weather service. How can I help you?
User: How’s the weather?
Virtual Assistant: For which location?
User: In Seattle.
Virtual Assistant: It is Haze, Smoke in Seattle at the moment. The temperature is 31 degrees, the humidity is 30 percent and the wind speed is 20 miles per hour.
Riva NLP: Sample 2¶
Virtual Assistant: Hi, welcome to RIVA weather service. How can I help you?
User: How’s the weather in Paris?
Virtual Assistant: It is Clear in Paris at the moment. The temperature is 20 degrees, the humidity is 73 percent and the wind speed is 9 miles per hour.
Riva NLP: Sample 3¶
Virtual Assistant: Hi, welcome to RIVA weather service. How can I help you?
User: Is it raining in Austin?
Virtual Assistant: The precipitation is 0.4 inches in Austin at the moment.
User: What is the temperature in Toronto today?
Virtual Assistant: It is 4 degree celsius in Toronto at the moment.
Limitations¶
The provided samples are not complete virtual assistants, but are intended as simple examples of how to build basic task-oriented chatbots with Riva. Consequently, the intent classifier and slot filling models have been trained with small amounts of data and are not expected to be highly accurate.
The Rasa virtual assistant sample supports intents for weather, temperature, rain, humidity, sunny, cloudy and snowfall checks. It does not support general conversational queries or other domains.
Both the Riva NLP and Rasa NLU samples support only 1 slot for city. Neither takes into account the day associated with the query.
Although the Rasa servers and the chatbot servers can be hosted on different machines, the provided code does not support independent scaling of the servers.
These samples support up to four concurrent users. This restriction is not because of Riva, but because of the web framework (Flask and Flask-ScoketIO) that is being used. The socket connection to stream audio to (TTS) and from (ASR) the user is unable to sustain more than four concurrent socket connections.
The Rasa virtual assistant is not optimized for low latency in case of multiple concurrent users.
Some erratic issues have been observed with the Rasa sample on the Firefox browser. The most common issue is the TTS output being taken in as input by ASR for certain microphone gain values.