Modalities

The API provides standard events and actions for the following modalities:

• Speech
• Motion
• Scene
• System
• Integration

Actions for different modalities are independent of each other. These actions can run independently of each other. An interactive system needs to support this. If multiple actions should be executed for the same modality, the policy of that modality governs how the execution / scheduling of these events are handled.

The following modality policies exist in UMIM:

UMIM Modality Policies

Policy	Description	Example Modality
parallel	Multiple actions can run at the same time.	Sound effects, Timers
override	Multiple actions can run at the same time. If an action is running it will be “overwritten”. When the overwriting action finishes the other action will be resumed	Postures, Gestures
skip	If an action is started while there is already an action in progress on the same modality the new action to be started will be started and immediately finished (ActionStarted() followed by ActionFinished()) without an effect on the interactive system
replace	Stops any ongoing action (ActionFinished()) on the same modality and starts the new action as soon as possible

Modality overview

Each interactive system can support different modalities. If a modality is supported, the interactive system needs to support all mandatory actions and events of that modality and may support further optional actions and events as listed in the table below.

UMIM Modality Overview

Modality Group	Modality Name	Policy	Action
Speech	bot_speech	replace	UtteranceBotAction
	user_speech	replace	UtteranceUserAction
Motion	bot_gesture	override	GestureBotAction
	bot_posture	override	PostureBotAction
	bot_position	override	PositionBotAction
	bot_face	override	FacialGestureBotAction
	user_gesture	parallel	GestureUserAction
	user_face	parallel	FacialGestureUserAction
Scene	information	override	VisualInformationSceneAction
	information	override	VisualChoiceSceneAction
	information	override	VisualFormSceneAction
	camera_shot	override	ShotCameraAction
	camera_motion_effect	override	MotionEffectCameraAction
System	user_presence	parallel	PresenceUserAction
	user_engagement	parallel	AttentionUserAction
	bot_expectation	parallel	ExpectationBotAction
	bot_expectation	parallel	ExpectationSignalingBotAction
	time	parallel	TimerBotAction
	web_request	parallel	RestApiCallBotAction