Building a Bot using Colang 1.0#

Before getting started with this section, ensure you have followed the Docker Environment section.

NVIDIA ACE Agent is an SDK, which helps you to build your domain conversational AI agent using Large Language Models (LLM). In this section, you will learn how to build a simple bot using ACE Agent that can answer questions about the stock market and give real-time stock prices.

Note

If you are completely new to this ecosystem, it will be beneficial to go through the Planning Your Bot section to understand some best practices and the typical steps you need to go through while building a use case using the NVIDIA ACE Agent.

Colang 1.0 will be deprecated in future releases in favor of Colang 2.0 (currently in preview as part of this release). You can follow Building Bot using Colang 2.0 and Event Interface for Colang 2.0 syntax.

Before we get started, there are a few key concepts that you need to understand.

Colang - Colang is the dialog modeling language used by ACE Agent. Colang makes it easy to define and control the behavior of your conversational system, especially in situations where deterministic behavior is required. ACE Agent is built on top of NVIDIA NeMo Guardrails, which also uses the Colang language. The current ACE Agent release supports two different versions (Colang 1.0 and Colang 2.0). In this tutorial, we are going to use Colang 1.0 syntax.

Bot - A conversational AI agent which can respond to any user input. A bot may support both text based and voice based conversations. A bot is defined by a set of YAML configuration and Colang dialog modeling files.

Canonical forms - Shorthand descriptions for user and bot messages that are easier to work with. They define the intent of a group of sentences, making it easier for LLMs to understand and process them as part of a conversation.

define user express greeting
  "Hello"
  "Hi"
  "Wassup?"

define user ask about ace agent
  "What is ace agent?"
  "How do I use ace agent?"

define bot answer about ace agent
  ACE Agent is a SDK which helps developers to build conversational AI agents. You can learn more about ACE Agent in ACE Agent documentation.
The above example defines two canonical forms for user messages: express greeting and ask about ace agent and one canonical form for bot message answer about ace agent. These user message canonical forms are paired with a representative set of sentences that are used to express the intent of the form. This provides context to the LLM, helps it understand the intent of the user’s input, and enables behavior according to specific dialog flows. The bot message canonical forms are paired with a set of sentences with which the bot can respond to the user’s query.

Dialog flows - Descriptions of how the dialog between the user and the bot should unfold. They include sequences of canonical forms for user and bot messages as well as additional logic (for example, branching, context variables, and other types of events). They help guide the behavior of the bot in specific situations.

define flow
    user express greeting
    bot express greeting

define flow
    user ask about ace agent
    bot answer about ace agent
The above example defines two dialog flows with the canonical definition above: express greeting and ask about ace agent. These flows would be activated when the user greets the bot or asks about ACE Agent.

You do not need to specify all possible dialog flows. If you would like deterministic behavior from the bot in a situation, you need to specify the dialog flow. When encountering novel situations that do not fall into any of the defined flows, the generalization capabilities of the LLM help generate new flows to make the bot respond appropriately.

Stock Bot#

In this section, let’s create a basic stock bot that responds to greetings, answers stock questions from a knowledge base, and provides a fallback response for questions about individual stock prices. We already have sample stock as part of the Quick Start resource at ./samples/stock_bot, you can use the same as reference for this tutorial.

Creating a Bot#

Create a my_bots directory to store new bots created as part of the tutorial.
Create a stock_bot directory inside it, with two files in it bot_config.yaml and flows.co. bot_config.yaml stores the bot configurations. flows.co is the Colang file which will define the conversation design.
├── my_bots │ └── stock_bot │ └── bot_config.yaml │ └── flows.co

Specifying Bot Configurations#

bot_config.yaml is the configuration entry point for any bot. Let’s add a few important configuration parameters to this file.

Give the bot a name. In bot_config.yaml, you need to add a unique name of the bot on top of the file. Let’s name the bot as enola.
bot: enola

Add the LLM model which you want to use to drive the bot. We will use the NVIDIA API Catalog mixtral-8x7b-instruct model.

models:
- type: main
    engine: nvidia-ai-endpoints
    model: ai-mixtral-8x7b-instruct
    parameters:
      stop: ["\n"]

Add the general instruction for the LLM (similar to a system prompt) to the bot_config.yaml. The instruction gets appended at the beginning of every prompt.

instructions:
    - type: general
      content: |
        Below is a conversation between a user and a stock faq bot named Enola that provides stock prices of companies and corporations provided by the user.
        It also provides users with information about stocks and the stock market. The bot is factual and concise. Bot informs the user when a company is imaginary.
        While generating a user intent, ensure that you generate an intent completely in lower case.

You can also add the sample conversations to the bot_config.yaml to help the LLM to better learn the conversation tone, style and response verbosity.

sample_conversation: |
user "Hello there!"
    express greeting
bot express greeting
    "Hello! I am Enola, how can I help you today?"

Specifying Colang Dialog Modeling#

In the flows.co file, let’s define canonical forms and the dialog flow, and add a fallback flow.

Define the canonical forms for the user and bot messages.
define user express greeting "Hello" "Hi" "Wassup?" define bot express greeting "Hey there!"
Each canonical form can have multiple examples attached to it. In the case of canonical forms for user messages, this helps the LLM generalize and respond appropriately to similar inputs. In the case of canonical forms for bot messages, the bot will use a random literal message from the given examples.
Define the dialog flows for guiding the bot. You can start by defining a simple flow that greets the user and asks them how they are doing:
define flow greeting user express greeting bot express greeting

Define a flow to answer questions about individual stock prices. For now, let’s provide a static response saying that we don’t have access to real-time information. Add these flows to flows.co.

define user asks stock price
    "What is the stock price of Microsoft?"
    "How much does an Nvidia stock cost"
    "what is the value of amazon stock?"

define bot respond with stock price
    "I'm sorry, but I don't have access to real-time information yet"

define flow provide stock price
    user asks stock price
    bot respond with stock price

Add a knowledge-base that contains some basic stock information. Create a directory called kb under stock_bot. Inside kb, create a file called stock_faq.md with the following contents.
├── my_bots │ └── stock_bot │ └── bot_config.yaml │ └── flows.co │ └── kb │ └── stock_faq.md

Inside kb, create a file called stock_faq.md with the following contents.

This document answers all questions and queries related to stocks and share market.
---
<br>

## What is a stock?

A stock, also known as equity, is a security that represents the ownership of a fraction of the issuing corporation. Units of stock are called "shares" which entitles the owner to a proportion of the corporation's assets and profits equal to how much stock they own.

## High Level Categories of Stocks

There are mainly six criterias under which the stocks are categorized - Market capitalization, Ownership, Fundamentals, Price volatility, Profit sharing and Economic trends.

## What is the share market?

The share market, also known as the stock market, is a platform where buyers and sellers come together to trade publicly listed shares of companies. A share in market parlance is part ownership in a company. So if a company has issued 100 shares and you own 1 share then you own 1% stake in the company. Share market is where shares of different companies are traded.

Add a flow in flows.co to handle generic questions about stocks or the stock market. This flow will use LLM to form a response based on the retrieved context from stock_faq.md.
define user ask stock question "What is a stock?" "Define stocks" "What is the share market?" define flow user ask stock question bot responds with answer
The bot canonical form responds with answer has not been defined anywhere. In Colang, this indicates that the LLM should form the answer.

Add a flow in flows.co to catch off topic queries.

define user ask off topic
    "Can you recommend a place to eat?"
    "Do you know any restaurants?"
    "Can you paint?"

define flow
    user ask off topic
    bot explain cant off topic

define bot explain cant off topic
    "Sorry, I cannot comment on anything which is not relevant to the stock market and stocks."

Testing the Bot#

Set the NVIDIA API key if it is not already set.
export NVIDIA_API_KEY=...
Run the bot in the HTTP interface using Docker Environment.

Set environment variables required for the docker-compose.yml file.

export BOT_PATH=./my_bots/stock_bot/
source deploy/docker/docker_init.sh

Deploy the ACE Agent microservices. Deploy the Chat Engine, Plugin server, and NLP server containers. We haven’t configured bot to use the Plugin server and NLP server yet.
docker compose -f deploy/docker/docker-compose.yml up --build chat-bot -d
Try out the bot using a web browser. You can deploy a sample frontend application with only textual chat support using the following command.
docker compose -f deploy/docker/docker-compose.yml up frontend
Interact with the bot using your browser at http://<YOUR_IP_ADDRESS>:9001.

Here is an example conversation with the bot:

Adding a Plugin Service#

In this section, we will add support to integrate an external API call with our bot. Any custom Python code that needs to be executed to gather information during a dialog flow execution can be integrated with the bots using the plugin service.

You can interact with external API or any custom code using ACE Agent’s Plugin server. Let’s collect information about stocks using the Yahoo Finance API and feed that into the ACE Agent using a custom plugin.

Create a file called plugin_config.yaml. This will contain the details of plugins that need to be deployed by ACE Agent.
Create a plugin directory in the bot config directory.

Add a file named yahoo_fin.py inside the plugin directory.

my_bots
└── stock_bot
    └── bot_config.yaml
    └── flows.co
    └── kb
        └── stock_faq.md
    └── plugin
        └── yahoo_fin.py
    └── plugin_config.yaml

Update yahoo_fin.py to get information about stocks using the Yahoo Finance API. For custom plugins, it’s necessary to import the APIRouter object.

import yfinance as yf
from yahoo_fin import stock_info as si
import requests
from typing import Optional
from fastapi import APIRouter

# API to extract stock price
Y_TICKER = "https://query2.finance.yahoo.com/v1/finance/search"
Y_FINANCE = "https://query1.finance.yahoo.com/v7/finance/quote?symbols="

router = APIRouter()

# Prepare headers for requests
session = requests.Session()
user_agent = (
    "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/108.0.0.0 Safari/537.36"
)
session.headers.update({"User-Agent": user_agent})


def get_ticker_symbol_alphavantage(stock_name: str) -> Optional[str]:
    # We do not need actual api key to get ticker info
    # But it is required as placeholder
    api_key = "YOUR_ALPHA_VANTAGE_API_KEY"
    url = f"https://www.alphavantage.co/query?function=SYMBOL_SEARCH&keywords={stock_name}&apikey={api_key}"
    response = requests.get(url)
    data = response.json()

    if "bestMatches" in data and len(data["bestMatches"]) > 0:
        ticker_symbol = data["bestMatches"][0]["1. symbol"]
        return ticker_symbol

    return None


@router.get("/get_ticker")
def get_ticker(company_name: str) -> Optional[str]:
    """
    Take company name returns ticker symbol used for trading
    param
        Args:
            company_name: company name like Microsoft
        Returns:
            Ticker Symbol used for trading like MSFT for microsoft
    """
    try:
        params = {"q": company_name, "quotes_count": 1, "country": "United States"}
        return session.get(url=Y_TICKER, params=params).json()["quotes"][0]["symbol"]
    except Exception as e:
        return get_ticker_symbol_alphavantage(company_name)


@router.get("/get_stock_price")
def get_stock_price(company_name: str) -> Optional[float]:
    """
    get a stock price from yahoo finance api
    """

    try:
        # Find ticker symbol for stock name, eg. Microsoft : MSFT, Nvidia: NVDA
        ticker = get_ticker(company_name)
        live_price = si.get_live_price(ticker)
        return round(live_price, 2)

    except Exception as e:
        print(f"Unable to find stock price of {company_name}")
        return None

Register the plugin into the Plugin server. Update plugin_config.yaml to include the name and path of the new plugin.

config:
    workers: 1
    timeout: 30

plugins:
    - name: stock
      path: "./plugin/yahoo_fin.py"

Update the flow provide stock price in Colang to utilize the endpoints exposed by this plugin for our bot.

The flow will first use LLM to extract the company name, and then send the name to the plugin that we’ve created. We can extract slots by providing a description of slot values.

define flow provide stock price
  user asks stock price

  # Generate the company name from user's input. If the company name is not specified, return "unknown".
  # Return only the name of the company in quotes, not an expression to calculate the name of the company.
  # For example, if the input is "What is the share price of Amazon?", return "Amazon"
  # For example, if the input is "How much does a share of microsoft cost?", return "microsoft"
  $company_name = ...

  $price = execute plugin(endpoint="/stock/get_stock_price", company_name=$company_name)
  if $price
    bot tell stock price
  else
    bot respond that it could not find the stock price

Re-test the updated bot.
1. Set the NVIDIA API key if it is not already set.
  export NVIDIA_API_KEY=...
2. Run the bot in the HTTP interface using Docker Environment.
3. Set the environment variables required for the docker-compose.yml file.
  export BOT_PATH=./my_bots/stock_bot/ source deploy/docker/docker_init.sh
4. Remove the existing deployment.
  docker compose -f deploy/docker/docker-compose.yml down
5. Deploy the updated ACE Agent microservices along with the Plugin server changes.
  docker compose -f deploy/docker/docker-compose.yml up --build chat-bot -d
6. Deploy the sample frontend application.
  docker compose -f deploy/docker/docker-compose.yml up frontend
7. Interact with the bot using your browser at http://<YOUR_IP_ADDRESS>:9001.
  
  Here is an example conversation with the bot:
  
  Using Colang, it is possible to handle more complex cases, such as fallback responses, contextual queries, profanity handling, and much more. To integrate these features into your bot, refer to the Colang User Guide, as well as the sample Stock Market Bot.
Finally, Add Speech to the Bot.

Adding Speech to a Bot#

In this tutorial, let’s add speech capabilities to the Stock bot we built as part of the tutorial as an example.

Specify the ASR and TTS models that need to be used in the pipeline. Create a file called model_config.yaml and add the following to it:

model_servers:
  - name: riva
    speech_models:
      - nvidia/ucs-ms/rmir_asr_parakeet_1-1b_en_us_str_vad:2.15.0
      - nvidia/riva/rmir_tts_radtts_hifigan_en_us_ipa:2.15.0
    url: localhost:8001

Specify some configurations used by the Chat Controller, the ACE Agent component that conducts the speech pipeline. Create a file called speech_config.yaml with the following contents:

grpc_server:
  nvidia::rrt::BotRuntimeGrpc: # component type
    ip_address: "0.0.0.0"
    port_number: 50055
    virtual_assistant_num_instances: 30
    virtual_assistant_pipeline_idle_threshold_secs: 600
    virtual_assistant_pipeline_idle_handler_wakeup_rate_secs: 10

speech_pipeline_manager: # config name
  SpeechPipelineManager: # component name
    asr_idle_timeout_ms: 200000
    tts_eos_delay_ms: 2000

riva_asr:
  RivaASR:
    server: "localhost:50051"
    word_boost_file_path: "/workspace/config/asr_words_to_boost.txt"
    enable_profanity_filter: false

dialog_manager:
  DialogManager:
    server: "http://localhost:9000"
    use_streaming: true

riva_tts:
  RivaTTS:
    server: "localhost:50051"
    voice_name: "English-US-RadTTS.Female-1"
    language: "en-US"
    ipa_dict: ""
    sample_rate: 44100

riva_logger:
  RivaLogger:
    data_dump_path: "/workspace/log"
    enable_logging: true

Deploy the speech models defined in model_config.yaml.

export BOT_PATH=<PATH TO BOT DIRECTORY>
source deploy/docker/docker_init.sh
docker compose -f deploy/docker/docker-compose.yml up model-utils-speech

Deploy the bot with the speech gRPC interface.

docker compose -f deploy/docker/docker-compose.yml up --build speech-bot -d

Interact with the bot using the Speech sample frontend application.

docker compose -f deploy/docker/docker-compose.yml up frontend-speech

Customize speech support using Word Boosting for ASR, TTS pronunciation with IPA, and a 3rd party TTS.