Virtual Assistant (with Rasa)¶

This Virtual Assistant (with Rasa) sample application demonstrates the integration of Rasa and the Jarvis Speech Service in the form of a weather chatbot web application. This sample is available in two options:

Option 1: Jarvis ASR + Jarvis TTS + Jarvis NLP + Rasa dialog manager

Option 2: Jarvis ASR + Jarvis TTS + Rasa NLU + Rasa dialog manager

To learn more about Rasa, follow this link.

Demo Video¶

To see how the Jarvis weather chatbot service works, the demo video can be found here.

Implementation¶

At a high-level, the integration takes advantage of the native API support of Rasa and gRPC support of Jarvis. The Weatherbot Client coordinates the workflow with Jarvis Service and Rasa, then interacts with the end-user via a web UI.

Container 1: Jarvis AI Services
- Exposes Speech Services (ASR/NLP/TTS) over gRPC endpoints
- Needs a GPU
Container 2: Jarvis Samples’ Jarvis-Rasa Chatbot
- Functionality
  Exposes Rasa functionality over API endpoints. Responsible for dialog management with Rasa as well as NLP with Jarvis or NLU with Rasa
  
  Contains the Weatherbot Client application (web UI and web service)
- Rasa
  Up to two instances
  
  Rasa Server
  
  Rasa Action (if applicable)
  
  Does not need GPUs but can be deployed on GPUs for performance
- Weatherbot Client
  Includes the Jarvis Python library installed
  
  Communicates with Jarvis AI Services and Rasa over gRPC and REST API endpoints respectively
  
  Pipelines ASR, NLP, TTS, and dialog manager functionalities
  
  Does not need GPUs

Architecture¶

Jarvis ASR + Jarvis TTS + Jarvis NLP + Rasa dialog manager

Jarvis ASR + Jarvis TTS + Rasa NLU + Rasa dialog manager

Code Structure¶

In this section, we will discuss the high-level code structure of the Weatherbot Client (in Jarvis Samples’ Jarvis-Rasa Chatbot).

asr.py
- This file contains the functionality to make the gRPC call to Jarvis ASR, using the Jarvis Python Client libraries, with the audio snippet, and returns the text transcript.
- ASR is used in streaming mode
rasa.py
- This file contains the functionality to make an API call to Rasa, with the user input and sender ID, and returns a text response obtained by processing the Rasa response object.
tts.py and tts\_stream.py
- These files contain the functionality to make the gRPC call to Jarvis TTS, using the Jarvis Python Client libraries, with a text snippet, and returns the corresponding audio speech.
- TTS can be used in either Batch or Streaming mode, depending on whether tts.py or tts\_stream.py is used. This can be set by changing the import statements in lines 3 and 4 in rasa-jarvis-weatherbot/rasa-jarvis-weatherbot-webapp/chatbot.py.
chatbot.py
- This file contains the Chatbot class which is responsible for pipelining all the ASR, TTS, and Rasa operations.
- Creates one instance of the Chatbot class per conversation.
- Pipeline is as follows:
  ASR is used in Streaming mode, therefore, it’s a background operation. By default, we are listening to the audio of the user who is speaking into the microphone.
  
  ASR calls Rasa with the transcribed text either automatically when the user stops talking (when the AutoSubmit flag is enabled) or when the Submit button is hit.
  
  When called, Rasa internally calls Jarvis NLP or Rasa NLU with the transcribed text to fetch the intents and slots which is then used by the Rasa Dialog Manager to get the final response text. The response text is returned.
  
  TTS is automatically called with Rasa’s response text, if System Speech is not muted.
  
  TTS gets the audio snippet and plays it back to the user on the speakers.

Network Configuration¶

There are two locations in the code base that have to be configured for inter-service communication:

rasa-chatbot/rasa-weatherbot/endpoints.yml

# uncomment and populate the section below
action_endpoint:
    url: "http://[rasa server host IP]:5055/webhook"

Example:

# uncomment and populate the section below
action_endpoint:
    url: "http://10.20.30.40:5055/webhook"

rasa-chatbot/config.py

# uncomment and populate the section below
jarvis_config = {
    ...
    "JARVIS_SPEECH_API_URL": "[jarvis speech service host IP]:50051",
    ...

# uncomment and populate the section below
rasa_config = {
    ...
    "RASA_API_URL": "[rasa server host IP]:5005",
    ...

In the case where one is using localhost the [rasa server host IP] must be set to the local machine’s IP, instead of localhost.

Example:

If the Jarvis Service, Rasa server, and the Weatherbot Client are all running on the same machine, it would be:

# uncomment and populate the section below
jarvis_config = {
    ...
    "JARVIS_SPEECH_API_URL": "10.20.30.40:50051",
    ...

# uncomment and populate the section below
rasa_config = {
    ...
    "RASA_API_URL": "10.20.30.40:5005",
    ...

If the Jarvice Service runs on a GPU that is supported by a remote machine, and Rasa and the Weatherbot Client are running on a local machine, it would be:

# uncomment and populate the section below
jarvis_config = {
    ...
    "JARVIS_SPEECH_API_URL": "10.20.30.40:50051",
    ...

# uncomment and populate the section below
rasa_config = {
    ...
    "RASA_API_URL": "20.40.60.80:5005",
    ...

Running the Demo¶

Start the Jarvis Speech Server. Follow the steps in the Quick Start Guide.
Start the Rasa Action server.
Run the Jarvis Sample container.
Pull the Jarvis Sample container.
docker pull nvcr.io/nvidia/jarvis/jarvis-speech-client:1.0.0-b.2-samples
Run the Jarvis Sample container.
docker run -it --rm -p 5055:5055 nvcr.io/nvidia/jarvis/jarvis-speech-client:1.0.0-b.2-samples /bin/bash
Navigate to the samples/rasa-chatbot directory.
cd samples/rasa-chatbot
Modify the API endpoint setting, if needed, per the Network Configuration section. Ensure you modify the API endpoints for each of the three container instances.
Start the Rasa Action server.
Activate the Rasa Python environment
. /pythonenvs/rasa/bin/activate
Navigate to the rasa-weatherbot directory.
cd rasa-weatherbot
Run the Rasa Action server.
rasa run actions --actions actions
Start the Rasa server in a different terminal.
Run the Jarvis Sample container.
Pull the Jarvis Sample container.
docker pull nvcr.io/nvidia/jarvis/jarvis-speech-client:1.0.0-b.2-samples
Run the Jarvis Sample container.
docker run -it --rm -p 5005:5005 nvcr.io/nvidia/jarvis/jarvis-speech-client:1.0.0-b.2-samples /bin/bash
Navigate to the cd samples/rasa-chatbot directory.
samples/rasa-chatbot
Modify the API endpoint setting, if needed, per the Network Configuration section. Ensure you modify the API endpoints for each of the three container instances.
Start the Rasa server.
Activate the Rasa Python environment.
. /pythonenvs/rasa/bin/activate
Navigate to the rasa-weatherbot directory.
cd rasa-weatherbot
Running Rasa training.
For Jarvis NLP: Training the Rasa Core model.
rasa train -c config/config_jarvisnlp.yml \
    -d domain/domain_jarvisnlp.yml \
    --out models/models_jarvisnlp/ \
    --data data/nlu_jarvisnlp.yml \
    data/rules_jarvisnlp.yml \
    data/stories_jarvisnlp.yml
For Rasa NLU: Training the Rasa NLU and Rasa Core models.
rasa train -c config/config_rasanlp.yml \
    -d domain/domain_rasanlp.yml \
    --out models/models_rasanlp/ \
    --data data/nlu_rasanlp.yml \
    data/rules_rasanlp.yml data/stories_rasanlp.yml
Running the Rasa server.
For Jarvis NLP.
rasa run -m models/models_jarvisnlp/ --enable-api \
    --log-file out.log --endpoints endpoints.yml
For Rasa NLU.
rasa run -m models/models_rasanlp/ --enable-api \
    --log-file out.log --endpoints endpoints.yml
Start the Jarvis Chatbot server. Configure the API endpoints for both Jarvis Speech Service and Rasa. Refer to the Network Configuration section for more information.
Run the Jarvis Sample container.
Pull the Jarvis Sample container.
docker pull nvcr.io/nvidia/jarvis/jarvis-speech-client:1.0.0-b.2-samples
Run the Jarvis Sample container.
docker run -it --rm -p 5555:5555 -p 50051:50051 -p 60051:60051 nvcr.io/nvidia/jarvis/jarvis-speech-client:1.0.0-b.2-samples /bin/bash
Navigate to the samples/rasa-chatbot directory.
cd samples/rasa-chatbot
Modify the API endpoint setting, if needed, per the Network Configuration section. Ensure you modify the API endpoints for each of the three container instances.
Start the Jarvis Chatbot Client.
Activate the Chatbot Client Python environment.
. /pythonenvs/client/bin/activate
Navigate to the rasa-jarvis-weatherbot-webapp directory.
cd rasa-jarvis-weatherbot-webapp
Starting the Chatbot Client server.
python3 main.py
Open the web UI on a web browser.
https://[jarvis chatbot server host IP]:5555/JarvisWeather
Example:
[https://0.0.0.0:5555/jarvisWeather](https://0.0.0.0:5555/jarvisWeather)

Sample Conversations¶

Virtual Assistant: Hi, welcome to JARVIS weather service. How can I help you?

User: How’s the weather?

Virtual Assistant: For which location?

User: In Seattle.

Virtual Assistant: It is Haze, Smoke in Seattle at the moment. The temperature is 31 degrees, the humidity is 30 percent and the wind speed is 20 miles per hour.

Virtual Assistant: Hi, welcome to JARVIS weather service. How can I help you?

User: How’s the weather in Paris?

Virtual Assistant: It is Clear in Paris at the moment. The temperature is 20 degrees, the humidity is 73 percent and the wind speed is 9 miles per hour.

Virtual Assistant: Hi, welcome to JARVIS weather service. How can I help you?

User: hi

Virtual Assistant: Hey! How are you?

User: who are you?

Virtual Assistant: I am a bot, powered by Rasa.

Virtual Assistant: Hi, welcome to JARVIS weather service. How can I help you?

User: Good Morning

Virtual Assistant: Hey! How are you?

User: I am doing great

Virtual Assistant: Great, carry on!

User: See you around

Virtual Assistant: Bye

Virtual Assistant: Hi, welcome to JARVIS weather service. How can I help you?

User: How’s the weather?

Virtual Assistant: For which location?

User: In Seattle.

Virtual Assistant: It is Haze, Smoke in Seattle at the moment. The temperature is 31 degrees, the humidity is 30 percent and the wind speed is 20 miles per hour.

Virtual Assistant: Hi, welcome to JARVIS weather service. How can I help you?

User: How’s the weather in Paris?

Virtual Assistant: It is Clear in Paris at the moment. The temperature is 20 degrees, the humidity is 73 percent and the wind speed is 9 miles per hour.

Virtual Assistant: Hi, welcome to JARVIS weather service. How can I help you?

User: Is it raining in Austin?

Virtual Assistant: The precipitation is 0.4 inches in Austin at the moment.

User: What is the temperature in Toronto today?

Virtual Assistant: It is 4 degree celsius in Toronto at the moment.

Limitations¶

The provided samples are not complete virtual assistants, but are intended as simple examples of how to build basic task-oriented chatbots with Jarvis. Consequently, the intent classifier and slot filling models have been trained with small amounts of data and are not expected to be highly accurate.
The Rasa virtual assistant sample supports intents for weather, temperature, rain, humidity, sunny, cloudy and snowfall checks. It does not support general conversational queries or other domains.
Both the Jarvis NLP and Rasa NLU samples support only 1 slot for city. Neither takes into account the day associated with the query.
Although the Rasa servers and the chatbot servers can be hosted on different machines, the provided code does not support independent scaling of the servers.
These samples support up to four concurrent users. This restriction is not because of Jarvis, but because of the web framework (Flask and Flask-ScoketIO) that is being used. The socket connection to stream audio to (TTS) and from (ASR) the user is unable to sustain more than four concurrent socket connections.
The Rasa virtual assistant is not optimized for low latency in case of multiple concurrent users.
Some erratic issues have been observed with the Rasa sample on the Firefox browser. The most common issue is the TTS output being taken in as input by ASR for certain microphone gain values.

License¶

End User License Agreement is included with the product. Licenses are also available along with the model application zip file. By pulling and using the Jarvis SDK container and download models, you accept the terms and conditions of these licenses. This project uses Rasa and Rasa-SDK, the license information for which can be found below: