Use Vision Events with your Bots#

This section explains how you can make your bot react to vision-based user events. Currently, Tokkio supports two types of vision events out of the box:

  • PresenceUserAction events provide information on whether the user is present in the camera view.

  • AttentionUserAction events provide information about the attention or engagement level of the user (e.g., a user can be considered distracted if they look away from the camera).

If your bot should react and interact with these events, you have two options:

  1. [Recommended] You can use the Colang library modules that provide high-level support to work with these events and integrate with the rest of the Colang Standard Library. (Note: the Colang libraries themselves make use of the UMIM action events directly; see point 2.)

  2. You can directly work with the UMIM Action events and write your custom Colang logic. The Working with Actions section from the Colang documentation explains how you can do this, and the two actions PresenceUserAction and AttentionUserAction are explained in detail in the UMIM documentation UMIM documentation.

The rest of this section will describe option (2) and how you can use Colang library modules to make your bot react to vision events.

Note

If you are using one of the reference applications with default settings of Tokkio, the backend and frontend are pre-configured to process the webcam stream to extract these vision-based user events. The two reference bots that ship with these reference applications provide more complete examples of how you may want to integrate reacting to vision events in your bots.

Prerequisites#

Before you start working through this section, make sure that you:

User Presence#

User presence handling is part of the avatar.co Colang standard library and is shipped by default with Colang 2.0. Therefore, you can directly use it in your bots. The following flows are available to interact with user presence.

user became present

Wait for the user to be detected as present (e.g., inside the camera ROI).

Example use:

import avatar
import core

flow main
     user became present
     bot say "Hi there, I can see you now!"
user became absent

Wait for the user to be detected as absent (e.g., out of the camera ROI).

Example use:

import avatar
import core

flow main
     user became absent
     bot say "Goodbye! Until next time!"

User Attention / Engagement#

To integrate handling user attention events in your bot scripts, we provide an example Colang module attention.co.

Install the Colang module#

We provide the Colang module attention.co together with a Python file implementing a few helper methods using Colang Python actions attention.py. You can install the module into your bot following these steps:

  1. Download the attention.co and attention.py files from NVIDIA/ACE

  2. Copy the files into your bot under the following locations:

    your_bot/
     actions/
          actions.py
          ...
     attention.co
     ...

Note

The bots from the two Tokkio reference applications already ship with this library. If you want to integrate it into your own bot script, follow the steps above

tracking user attention

For the automatic handling of user attention events, you need to activate this flow to track user attention levels during the last user utterance. This information will be used to change the functionality of all user said flows such that they will no longer finish when the user says something while being inattentive.

Example:

import avatar
import core
# attention.co is imported automatically, since it is in the same folder as your main.co

flow main
     # Activate the flow at the beginning to make sure user attention events are tracked properly
     activate tracking user attention

     ...
user said (overwritten)

When you include attention.co in your bot folder, it overrides all user said related flows so that these flows only consider user utterances when the user is attentive. You can overwrite the default attention check by overwriting the flow attention checks explained below. For your first test, the default implementation should work well with the Tokkio setup.

Example:

import avatar
import core
# attention.co is imported automatically, since it is in the same folder as your main.co

flow main
     activate tracking user attention

     # Since attention.co overwrites all user said related flows, this line will wait until the user says
     # something while being attentive.
     user said something
     bot say "I heard you and you are attentive"
attention checks $event -> $is_attentive

The attention checks flow is called whenever the system needs to decide if a user utterance was completed while the user was attentive. You can overwrite the default behavior by overwriting this flow in your bot script.

Example:

import avatar
import core
# attention.co is imported automatically, since it is in the same folder as your main.co

@override
flow attention checks $event -> $is_attentive
     # Implement your custom attention logic here
     $is_attentive = True
     return $is_attentive
user said something inattentively

The user said something while being inattentive. Use this flow to let the user know that the bot assumes that the user is not attentive and the utterance will be ignored.

Example:

import avatar
import core
# attention.co is imported automatically, since it is in the same folder as your main.co

flow main
     activate tracking user attention
     when user said something
          bot say "I hear you"
     or when user said something unattentively
          bot say "You seem distracted. Can you repeat?" and bot gesture "asking if something refers to them, being unsure if they're being addressed"
On this page