Plugin Server

Order-Food Plugin

Plugin server allows developers to integrate and execute custom use case and domain specific code as part of the NVIDIA Chat engine pipeline. For Tokkio bot order-food plugin is responsible for interacting with customer menu API and order-management system.

Plugin modules required for a bot can be defined in the config file. For Tokkio, you can find the plugin defined in plugin_config.yaml file.

 plugins:
- name: food_order
  path: plugin/order_food.py
  parameters:
    menu_url: http://localhost:8080
    cart_url: http://localhost:7007
    recommendation_url: http://localhost:7000
    ui_server_url: http://localhost:32362

The defined plugin module can be invoked as part of intent-specific dialog action defined with Colang 2.0. For Tokkio bot, here is an example of calling the plugin server module when a query related to adding an item to the order is received.

flow handle adding items to order
  user added food drink items to order
  log "Add - Starting the add"
  $items_to_add = """Return the items the user wants to add to his order as a list of python dicts based on the last user action. Do not include the items requested in other user actions. If the user did not specify then don't include the size field. If the user did not specify the quantity then don't include the quantity field. Example: [{'name':'coke', 'size': 'large', 'quantity': 2},{'name' : 'burger', 'size': 'medium', 'quantity' : 1}]. Provide a valid Python literal on a single line:"""

  $information = await AddItemAction(items=$items_to_add)

  if $information
     $response_to_user = """This is the result from AddItemAction: '{{$information}}'. Rephrase in simple terms to the user."""
     bot say $response_to_user
  else
     bot say something like "I could not add the requested items. We have a technical issues."

Responsibilities of Tokkio order-food plugin includes:

  • Determine what actions need to be taken to fulfill the user’s request based on the intent and slot related information.

  • Fetch relevant information from the customer specific menu database.

  • Maintain the state of the user’s cart by interacting with Cart Manager to perform actions like adding, removing and replacing items or toppings.

  • Control the state of the UI by showing the options that the user has asked about.

  • Form narratives for items that the user has asked about.

  • Maintain the context of the conversation for a natural flow.

  • Use LLM to generate a suitable response and return it to the Chat engine.


At a high level, the flow of a conversation goes like this:

  1. The Colang runtime runs as part of the chat engine to provide guard-railing and conversational interaction management capabilities. It makes a call to the LLM to perform user/input query parameter extraction and classifies it based on the detected intent.

  2. The plugin server implements the logic to fulfill the classified intent (eg: add an item to the cart) by performing the actions with the help of Cart Manager, Menu server and the UI server, based on the intent and identified items.

  3. The result of plugin server action is passed to the Colang runtime

  4. The Colang runtime makes a call to the LLM to paraphrase the response in a human-friendly manner.

  5. The paraphrased response is then used by the Chat engine to respond to the user request.

You can read about how it interact with other Tokkio QSR microservices in Architecture.

Tokkio Food ordering bot

Internally, the Tokkio food ordering bot is broken into multiple files and components for easier maintenance and readability. In this section, we will give an overview of the main components and their functionalities.

Fulfilment Module

You can find the Plugin server source code on NGC along with other Tokkio submodules.

Colang

This directory contains the logic for using the Colang runtime. The flows to handle an identified intent are defined in various files that are logically separated by functionality.

Order-Food - order_food.py

This is the central module of Tokkio plugin server. The purpose of this file is to:

  1. Initialize all modules during bot startup.

  2. Validate the request received from Colang runtime. If the intent is unsupported or if the session is inactive, the query is rejected.

  3. Call intent specific Intent Handlers to disambiguate context, food items, perform actions and form a response.

Intent Handlers

Each intent supported by the bot requires a file that contains the logic to form requests to search the database or cart and perform actions specific to that intent. Such a file is called the Intent Handler for that intent.

Based on the intent received by Order-Food, the respective intent handler is called to execute the necessary logic to fulfill the query.

Several intents share common steps that need to be performed before the intent-specific code can be executed. Examples of such steps include: filling templates of FoodItems, searching the cart for specific items, fetching specific items from context, etc.

All such common functions are stored in the intent_handlers/utils/common.py file. These can be imported into the specific Intent Handlers where they are needed and used with minimum overhead.

Access Endpoints

For ease of debugging and maintenance, a single file is used to perform actions, search the menu database, and get the state of the cart. This file is called Access Endpoints, as it interacts with the endpoints of containers that are responsible for menu, cart and UI updating.

Each individual intent handler forms filters corresponding to the item to be searched or the action to be performed. These filters are used by the Access Endpoints APIs to actually perform the action/search as needed.

API Guide

Please note that the context variable needs to be provided for using most Plugin Server APIs. An example of context variable is as below for a queries regarding adding or removing items

 1 {
 2    "context": {
 3       "number_of_failed_utterance_actions": 0,
 4       "user_id": "3c2a9f78-0ed7-11ef-8257-bbcff4ccb54a",
 5       "stream_id": "05828057-73c7-4154-9f65-df9eea42f730",
 6       "session_id": "4107ec59-9856-4e54-9c75-46e9695ffca9",
 7       "query_id": "d4534800-93e4-4b26-b9a5-5d1d46956403",
 8       "event_id": "714586fd-807d-439a-8e28-83ca81256749",
 9       "bot_talking_state": false,
10       "last_user_transcript": "Add one cheeseburger to the cart ",
11       "last_user_message": "Add one cheeseburger to the cart. ",
12       "slot_results": {
13             "food_name": [
14                "cheeseburger"
15             ],
16             "food_quantity": [
17                "one"
18             ],
19             "food_size": [
20
21             ]
22       },
23       "linked_slots": {},
24       "food_name": [
25             "cheeseburger"
26       ],
27       "food_quantity": [
28             "one"
29       ]
30    }
31 }

A snippet of APIs exposed by the the QSR Plugin Server is shown below. These APIs can be accessed at http://<deployment_ip>:9002/docs#/ link.

Plugin Server APIs

See QSR Plugin Server APIs for more details.