Building a Bot using Colang 2.0 and Event Interface#

Before you build a bot using Colang 2.0 and the Event Interface, ensure you have followed the steps in the Docker Environment section.

NVIDIA ACE Agent is an SDK, which helps you to build your domain conversational AI agent using Large Language Models (LLM). In this tutorial, you will learn how to work with the ACE Agent and how to create a simple bot that makes use of Colang 2.0 and asynchronous event processing. The bot features:

Different response types - The bot will make use of gestures, utterances and showing information on a UI.
LLM integration - The bot will make different use of LLMs to provide contextual answers and to simplify user input handling.
Proactivity - The bot will be proactive and will try to engage the user if no reply is given.
Interruptibility - The user can interrupt the bot at any time.
Back-channeling - The bot can react in real time based on ongoing user input to make your interactions interesting.
Business Logic - The bot can utilize custom python functions or API calls to integrate business logic.

Before you get started, there are a two key terminologies that you need to know.

Colang - Colang is the dialog modeling language used by ACE Agent. Colang makes it easy to define and control the behavior of your conversational system, especially in situations where deterministic behavior is required. ACE Agent is built on top of NVIDIA NeMo Guardrails which also uses the Colang language. The current ACE Agent release supports two different versions (Colang 1.0 and Colang 2.0-beta). In this tutorial, we are going to use Colang 2.0-beta syntax. Colang 1.0 will be deprecated in future releases. This tutorial requires no prior knowledge about Colang as we will walk you through all the changes step by step. For more information about Colang, refer to the Getting Started documentation.

Event interface - The ACE Agent event interface provides an asynchronous, event-based interface to interact with bots written in Colang 2.0. The interface allows you greater flexibility in managing your interaction between the user and the bot due to its asynchronous design. Using the interface, you can easily build interactive systems that break turn-taking behavior (user speaks, bot speaks, user speaks…) and you can handle multiple actions going on at the same time (for example, bot speaks, makes a gesture to emphasize a point, user is looking around and interrupts bot after a short while).

The event interface is best suited with more complex interactive systems. An example for this would be an interactive avatar system where you not only want to control the bot responses but also gestures, postures, sounds, showing pictures, and so on. The NVIDIA Tokkio reference application (part of ACE) is a great starting point for how to build a real interactive avatar system using the event interface from ACE Agent. For more information, NVIDIA Tokkio demonstrates such interactions with a 3D avatar running accessible from web browsers.

Prerequisites#

Setup the Event Simulator packaged as part of the Quick Start resource at clients/event_client/. To run the Event Simulator script, you need to install a few Python packages. It is recommended that you create a new Python environment to do so.
cd clients/event_client/ python3 -m venv sim-env source sim-env/bin/activate pip install -r requirements.txt

Test the template.

Open two different terminals.
In one terminal, start the ACE Agent with the bot template packaged in the Quick Start resource at samples/event_interface_tutorial_bot.
Set the OpenAI API key environment variable.
export OPENAI_API_KEY=...

export BOT_PATH=samples/event_interface_tutorial_bot/step_0
source deploy/docker/docker_init.sh
docker compose -f deploy/docker/docker-compose.yml up event-bot -d
In a separate terminal, start the Event Simulator CLI.
# Make sure that you are in the correct folder and that you have activated the python environment
cd clients/event_client/
source sim-env/bin/activate

python event_client.py

To confirm the test was successful, you should see the message Welcome to the tutorial in the Chat section on the left. Refer to the sample event client section for more information.

Note

You can find the source code for all the individual steps in the Quick Start Resources under samples/event_interface_tutorial_bot/step_x. You can either follow the tutorial step by step making the necessary changes or you can try out the different versions by running the bots in the folder corresponding to each step (as explained in the following topics).

Step 1: Making the Greeting More Interesting#

In this section, you will see how you can use different types of responses to make the greeting or any other response from the bot more interesting. If you connect your bot to a system such as ACE to drive a 2D or 3D avatar, the exact same bot code as outlined below can be used to trigger multimodal responses such as avatar gestures or speech.

If you want to follow along, start with the bot in samples/event_interface_tutorial_bot/step_0 to apply the changes below. The final bot code with all the changes from Step 1 can be found in samples/event_interface_tutorial_bot/step_1.

Change the existing flow bot express greetings at the top of the main.co file. The flow should look similar to:

@meta(bot_intent=True)
flow bot express greeting
  (bot express "Hi there!"
    or bot express "Welcome!"
    or bot express "Hello!")
    and bot gesture "Wave with one hand"

Show the greeting in the UI by adding the statement start scene show short information "Welcome to this tutorial interaction" as $intro_ui in the main flow before the line bot express greeting as shown below.

# The bot greets the user and a welcome message is shown on the UI
  start scene show short information "Welcome to this tutorial interaction" as $intro_ui
  bot express greeting

Restart the updated bot and Event Simulator.

docker compose -f deploy/docker/docker-compose.yml down

export BOT_PATH=samples/event_interface_tutorial_bot/step_1
source deploy/docker/docker_init.sh

docker compose -f deploy/docker/docker-compose.yml up event-bot -d

python event_client.py

In addition to the greeting message, we have a bot gesture shown for two seconds in the Motion area on the left and a UI shown on the right. To make a 3D Interactive Avatar say “Welcome!”, wave into the camera, and display a proper UI in the view, you could use the exact same Colang code.

Step 2: Leveraging LLMs to Answer User Questions#

In this section, you will enable the bot to answer any user question based on a large language model (LLM).

If you want to follow along, start with the bot in samples/event_interface_tutorial_bot/step_1 to apply the changes below. The final bot code with all the changes from Step 2 can be found in samples/event_interface_tutorial_bot/step_2.

Provide general instructions. Open the file samples/event_interface_tutorial_bot/step_2/bot_config.yml. Under instructions is an example for general LLM instructions. Keep these instructions for now, however, you can experiment with different instructions and see how you can change the types of answers the bot provides.

instructions:
    - type: "general"
      content: |
        Below is a conversation between Emma, a helpful interactive avatar assistant (bot), and a user.
        The bot is designed to generate human-like actions based on the user actions that it receives.
      [...]

Enable LLM fallback. Update your main.co file as shown below. You only need to update the CHANGE sections. Ensure you don’t duplicate flows when doing your changes. If you define the same flow twice, the later definition will overwrite the first one. Your main.co file should look like this:

import core
import avatars
import llm

@meta(bot_intent=True)
flow bot express greeting
  (bot express "Hi there!"
    or bot express "Welcome!"
    or bot express "Hello!")
    and bot gesture "Wave with one hand"

# CHANGE 1
# Add two flows to handle ending the conversation: bot express goodbye, user expressed done
@meta(bot_intent=True)
flow bot express goodbye
  (bot express "Goodbye" or bot express "Talk to you soon!") and bot gesture "bowing in goodbye"

@meta(user_intent=True)
flow user expressed done
  user said (regex("(?i).*done.*|.*end.*demo.*|.*exit.*"))

flow handling user requests until user expressed done
  # CHANGE 2
  # This activates LLM-based responses for all unhandled user intents
  activate llm continuation

  # CHANGE 3 (optional)
  # This will generated a variation of the question (variations are generated by the LLM)
  bot say something like "How can I help you today?"

  # CHANGE 4
  # When the user expressed we end the conversation
  user expressed done
  bot express goodbye

# The main flow is the entry point
@meta(exclude_from_llm=True)
flow main

  # Technical flows, see Colang 2.0 documentation for more details
  activate notification of undefined flow start
  activate notification of colang errors
  activate tracking bot talking state

  # The bot greets the user and a welcome message is shown on the UI
  start scene show short information "Welcome to this tutorial interaction" as $intro_ui
  bot express greeting

  # CHANGE 5
  # Start handling user requests
  handling user requests until user expressed done

  # This will prevent the main flow finishing ever
  wait indefinitely

Restart the updated bot and Event Simulator.

docker compose -f deploy/docker/docker-compose.yml down

export BOT_PATH=samples/event_interface_tutorial_bot/step_2
source deploy/docker/docker_init.sh

docker compose -f deploy/docker/docker-compose.yml up event-bot -d

python event_client.py

Ask any questions to the bot that will be answered by the LLM given the provided general instructions.

Step 3: Making the Bot Proactive#

In this section, we will add a proactivity feature to make the conversation with your bot feel more natural. The bot will generate an appropriate utterance when the user has been silent for a specified amount of time. This helps drive the conversation forward and can be used to provide additional information or help to the user.

If you want to follow along, start with the bot in samples/event_interface_tutorial_bot/step_2 to apply the changes below. The final bot code with all the changes from Step 3 can be found in samples/event_interface_tutorial_bot/step_3.

Add a new flow to main.co and activate the flow inside the handling user requests until user expressed done flow.

[...]
[...]
# CHANGE 1: Add new flow that reacts to the user being silent
flow handling user silence
  user was silent 15.0
  llm continue interaction

flow handling user requests until user expressed done
  activate llm continuation
  # CHANGE 2: Activate the new flow
  activate handling user silence

  bot say something like "How can I help you today?"
  user expressed done
  bot express goodbye
[...]

With this, we will leverage the LLM to generate a bot response whenever the user has been silent for at least 15 seconds.

Restart the updated bot and Event Simulator.

docker compose -f deploy/docker/docker-compose.yml down

export BOT_PATH=samples/event_interface_tutorial_bot/step_3
source deploy/docker/docker_init.sh

docker compose -f deploy/docker/docker-compose.yml up event-bot -d

python event_client.py

Look for a timer that is ticking down, in the Utils section. When the timer finishes, the bot will follow up with the user.

Step 4: Interrupting the Bot#

When humans talk to each other we often interrupt each other to clarify certain points or to tell someone that you already know what they are talking about. With the ACE Agent event interface and Colang 2.0 we can easily achieve this with a few small changes to the current bot.

If you want to follow along, start with the bot in samples/event_interface_tutorial_bot/step_3 to apply the changes below. The final bot code with all the changes from Step 4 can be found in samples/event_interface_tutorial_bot/step_4.

Activate the following flow inside the handling user requests until user expressed done flow.

flow handling user requests until user expressed done
  [...]

  # Allow the user to interrupt the bot at anytime
  activate interruption handling bot talking $mode="interrupt"

  bot say something like "How can I help you today?"
  [...]

This flow handles any interruptions by the user.

Ask the bot to tell a story about something, make the bot respond with a long sentence.
While the bot is responding, type something to interrupt it.
Restart the updated bot and Event Simulator.

docker compose -f deploy/docker/docker-compose.yml down

export BOT_PATH=samples/event_interface_tutorial_bot/step_4
source deploy/docker/docker_init.sh

docker compose -f deploy/docker/docker-compose.yml up event-bot -d

python event_client.py

Step 5: Back-Channeling#

Back channeling means the bot might provide short reactions based on user input to make the interaction more engaging. For this tutorial we will use a very simple example: At the end of the interaction we will ask the user to provide an email address. While the user is entering the email address the bot will provide contextual feedback.

If you want to follow along, start with the bot in samples/event_interface_tutorial_bot/step_4 to apply the changes below. The final bot code with all the changes from Step 5 can be found in samples/event_interface_tutorial_bot/step_5.

Add the following flows to the top of the file main.co.

# CHANGE 1: Add two new user intent flows
@meta(user_intent=True)
flow user confirmed
  user has selected choice "yes"
    or user said "yes"
    or user said "ok"
    or user said "that's ok"
    or user said "yes why not"
    or user said "sure"

@meta(user_intent=True)
flow user denied
  user has selected choice "no"
    or user said "no"
    or user said "don't do it"
    or user said "I am not OK with this"
    or user said "cancel"

# CHANGE 2: Add flow that asks user for email address using UI and voice
flow ask for user email
  start VisualChoiceSceneAction(prompt= "Would you share your e-mail?", support_prompts=["You can just type 'yes' or 'no'","Or just click on the buttons below"],choice_type="selection", allow_multiple_choices=False, options= [{"id": "yes", "text": "Yes"}, {"id": "no", "text": "No"}]) as $confirmation_ui
  bot say "I would love to keep in touch. Would you be OK to give me your e-mail address?"
  when user confirmed
    send $confirmation_ui.Stop()
    bot ask "Nice! Please enter a valid email address to continue"
    start VisualFormSceneAction(prompt="Enter valid email",inputs=[{"id": "email", "description": "email address", "value" : ""}]) as $action
    while True
      when VisualFormSceneAction.InputUpdated(interim_inputs=[{"id": "email",  "value" : regex("@$")}])
          bot say "And now only the domain missing!"
      or when VisualFormSceneAction.InputUpdated(interim_inputs=[{"id": "email",  "value" : regex("^[-\w\.]+@([\w-]+\.)+[\w-]{2,4}$")}])
          bot say "Looks like a valid email address to me, just click ok to confirm" and bot gesture "success"
      or when VisualFormSceneAction.ConfirmationUpdated(confirmation_status="confirm")
          bot say "Thank you" and bot gesture "bowing"
          break
      or when VisualFormSceneAction.ConfirmationUpdated(confirmation_status="cancel")
          bot say "OK. Maybe another time."
          break
  or when user denied
    bot say "That is OK"

Add the flow ask for user email at the end of the handling user requests until user expressed done flow before the bot expresses goodbye.

flow handling user requests until user expressed done
  [...]
  user expressed done
  # Run the flow that asks the user for email address and await it
  ask for user email
  bot express goodbye

Restart the updated bot and Event Simulator.

docker compose -f deploy/docker/docker-compose.yml down

export BOT_PATH=samples/event_interface_tutorial_bot/step_5
source deploy/docker/docker_init.sh

docker compose -f deploy/docker/docker-compose.yml up event-bot -d

python event_client.py

Test this interaction by ending the conversation. For example, write I am done into the prompt. This will trigger the flow ask for user email. This flow first asks you if you are Okay to provide your email (you can write the confirmation in the prompt or click on an option in the UI on the right). If you confirm, the email entering prompt will appear on the right in the UI section.

Note

The bot will react if you type the @ as part of your email address.

Step 6: Running Python Code#

Colang allows you to call Python code from within a flow. These are called Python Actions and can contain arbitrary Python code. Python Actions can be particularly useful if you need more complex data processing functionality that cannot be easily done in Colang.

If you want to follow along, start with the bot in samples/event_interface_tutorial_bot/step_5 to apply the changes below. The final bot code with all the changes from Step 6 can be found in samples/event_interface_tutorial_bot/step_6.

Note

All Python Actions run in the Python context of the Colang interpreter (sharing libraries and dependencies). If you need to write more complex code with specific dependencies, refer to Integrating a Plugin Service.

Using Python Actions, we can easily leverage Python functionality inside a flow. Let’s add a flow that counts the words from the user’s sentence.

Create a file called actions.py inside the bot folder, with the following content.

from nemoguardrails.actions.actions import action

@action(name="CountWordsAction")
async def count_words_action(transcript: str) -> int:
    return len(transcript.split())

Update the main.co as shown below (adding a new user intent flow and a flow to handle the word counting requests from the user).

# CHANGE 1: Add a new user intent at the top of main.co (after the imports)
@meta(user_intent=True)
flow user requested to count words in sentence
  user has selected choice "can you please count the words in my next sentence"
    or user said "count how many words are in my next utterance"
    or user said "how many words"

[...]

# CHANGE 2: Add new flow that defines how to handle word counting requests
flow handling word counting requests
  user requested to count words in sentence
  bot say "Sure, please say a sentence and I will count the words"
  user said something as $ref
  $count = await CountWordsAction(transcript=$ref.transcript)
  bot say "There were {$count} words in your sentence."

[...]

flow handling user requests until user expressed done
  [...]
  activate handling bot talking interruption $mode="interrupt"

  # CHANGE 3: Activate the new flow
  activate handling word counting requests

  bot say something like "How can I help you today?"
  [...]

Restart the updated bot and Event Simulator.

docker compose -f deploy/docker/docker-compose.yml down

export BOT_PATH=samples/event_interface_tutorial_bot/step_6
source deploy/docker/docker_init.sh

docker compose -f deploy/docker/docker-compose.yml up event-bot -d

python event_client.py

Step 7: Integrating a Plugin Service#

In this section, we will add support to integrate an external API call with our bot. If the bot that you design depends on external information or needs to invoke certain actions, you may use the plugin service to host more complex Python code, compared to the approach showcased in the previous section using Python Actions.

If you want to follow along, start with the bot in samples/event_interface_tutorial_bot/step_6 to apply the changes below. The final bot code with all the changes from Step 7 can be found in samples/event_interface_tutorial_bot/step_7.

Using plugin services, you can interact with any external API or any custom code using ACE Agent’s Plugin server. For this example, let’s collect information about stocks using the Yahoo Finance API and feed that information into the bot using a custom plugin.

Create a file called plugin_config.yaml. This will contain the details of plugins that need to be deployed by ACE Agent.
Create a plugin directory in the bot config directory.
Add a file named yahoo_fin.py inside the plugin directory.

samples
└── event_interface_tutorial_bot
    └── bot_config.yaml
    └── main.co
    └── plugin
        └── yahoo_fin.py
    └── plugin_config.yaml

Update yahoo_fin.py to get information about stocks using the Yahoo Finance API. For custom plugins, it’s necessary to import the APIRouter object.

import yfinance as yf
from yahoo_fin import stock_info as si
import requests
from typing import Optional
from fastapi import APIRouter

# API to extract stock price
Y_TICKER = "https://query2.finance.yahoo.com/v1/finance/search"
Y_FINANCE = "https://query1.finance.yahoo.com/v7/finance/quote?symbols="

router = APIRouter()

# Prepare headers for requests
session = requests.Session()
user_agent = (
    "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/108.0.0.0 Safari/537.36"
)
session.headers.update({"User-Agent": user_agent})


def get_ticker_symbol_alphavantage(stock_name: str) -> Optional[str]:
    # We do not need actual api key to get ticker info
    # But it is required as placeholder
    api_key = "YOUR_ALPHA_VANTAGE_API_KEY"
    url = f"https://www.alphavantage.co/query?function=SYMBOL_SEARCH&keywords={stock_name}&apikey={api_key}"
    response = requests.get(url)
    data = response.json()

    if "bestMatches" in data and len(data["bestMatches"]) > 0:
        ticker_symbol = data["bestMatches"][0]["1. symbol"]
        return ticker_symbol

    return None


@router.get("/get_ticker")
def get_ticker(company_name: str) -> Optional[str]:
    """
    Take company name returns ticker symbol used for trading
    param
        Args:
            company_name: company name like Microsoft
        Returns:
            Ticker Symbol used for trading like MSFT for microsoft
    """
    try:
        params = {"q": company_name, "quotes_count": 1, "country": "United States"}
        return session.get(url=Y_TICKER, params=params).json()["quotes"][0]["symbol"]
    except Exception as e:
        return get_ticker_symbol_alphavantage(company_name)


@router.get("/get_stock_price")
def get_stock_price(company_name: str) -> Optional[float]:
    """
    get a stock price from yahoo finance api
    """

    try:
        # Find ticker symbol for stock name, eg. Microsoft : MSFT, Nvidia: NVDA
        ticker = get_ticker(company_name)
        live_price = si.get_live_price(ticker)
        return round(live_price, 2)

    except Exception as e:
        print(f"Unable to find stock price of {company_name}")
        return None

Register the plugin into the Plugin server. Update plugin_config.yaml to include the name and path of the new plugin.

config:
  workers: 1
  timeout: 30

plugins:
  - name: stock
    path: "./plugin/yahoo_fin.py"

Update main.co to add and activate the new flow handling stock price questions to utilize the endpoints exposed by this plugin for our bot as shown below. The flow will first use the LLM to extract the company name, and then send the name to the plugin that we’ve created.

define flow provide stock price
  user asks stock price

# CHANGE 1: Add user intent at the top of main.co
@meta(user_intent=True)
flow user asked stock price
  user said "What is the stock price of Microsoft?"
    or user said "How much does an Nvidia stock cost"
    or user said "what is the value of amazon share price?"
    or user said "What is it's stock price?"

[...]

# CHANGE 2: Add flow that handles stock price questions
flow handling stock price questions
  global $last_user_transcript
  user asked stock price

  $company_name = ..."Return the name of the company the user was referring to in quotes \"<company name here>\" or \"unknown\" if the user did not mention a company."
  if $company_name == "unknown"
    bot say "Sorry, I can't understand which company you are referring to here. Can you rephrase your query?"
    return
  else
    $price = await InvokeFulfillmentAction(endpoint="/stock/get_stock_price", company_name=$company_name)
    if not $price
      bot say "Could not find the stock price!"
    else
      bot say "Stock price of {$company_name} is {$price}"

[...]

flow handling user requests until user expressed done
  [...]
  activate handling bot talking interruption $mode="interrupt"

  # CHANGE 3: Activate the flow
  activate handling stock price questions
  [...]

Restart the updated bot and Event Simulator.

docker compose -f deploy/docker/docker-compose.yml down

export BOT_PATH=samples/event_interface_tutorial_bot/step_7
source deploy/docker/docker_init.sh

docker compose -f deploy/docker/docker-compose.yml up event-bot -d

python event_client.py