Metropolis User Occupancy and Engagement Alerts#
Overview#
Metropolis Occupancy Alerts module does streaming analytics of vision inference metadata and produces insights in the form of User occupancy alerts.
Metropolis User Engagement Alerts module does streaming analytics of vision inference metadata and produces insights in the form of User attention zone alerts.
These alerts are published to Redis messaging bus.
Design#
Input#
The output of Vision AI micro-service serves as an input for Tokkio User Occupancy and Engagement module. It runs real-time computer vision on every frame from the stream and outputs the corresponding metadata detected in the visionai redis channel. Each metadata frame contains face bounding box and 2D body pose of all the detected people.
Sample metadata frame:
{
"version": "4.0",
"id": 1,
"@timestamp": "YYYY-MM-DDTHH:MM:SS.sssZ",
"sensorId": "SENSOR_ID",
"objects":
[
"0|484.822|125.63|705.183|345.99|Face|#|pose2D|18|nose,266.28452,144.77771,0.83630073|neck,-1.0,-1.0,-1.0|right-shoulder,167.94446,250.14096,0.93399066|right-elbow,93.0017,415.3354,0.63592523|right-wrist,91.006836,471.03665,0.28546536|left-shoulder,379.28296,241.4837,0.8813749|left-elbow,481.14264,360.60825,0.82538456|left-wrist,482.97302,420.16254,0.37851536|right-hip,214.57726,487.93814,0.7985392|right-knee,97.43787,466.14273,0.13412973|right-ankle,230.69476,478.4018,0.03407041|left-hip,357.42007,485.96896,0.8217566|left-knee,466.0811,476.68948,0.07266329|left-ankle,466.70682,444.2228,0.04072526|right-eye,243.17857,119.35192,0.7210652|left-eye,289.328,119.19954,0.6104527|right-ear,218.67596,134.33481,0.5708076|left-ear,316.38135,134.21928,0.87809706"
]
}
Outputs#
Sample User FOV occupancy alert :
{
"id": "073835a1-b33b-4f5f-92af-85564d662fc8",
"sensor": {
"id": "SENSOR_ID",
"type": "camera"
},
"analyticsModule": {
"version": "1.0",
"id": "emdat_fov_alerts",
"source": "Nvidia_eMDX",
"description": "eMDAT fov alerting module"
},
"alerts": [
{
"id": "SENSOR_ID::fov::occupancy_threshold_switch::up::YYYY-MM-DDTHH:MM:SS.sssZ",
"rule_id": "occupancy_threshold_switch_id",
"rule_name": "",
"rule_type": "occupancy_threshold_switch",
"start": "YYYY-MM-DDTHH:MM:SS.sssZ",
"end": "YYYY-MM-DDTHH:MM:SS.sssZ",
"data": {
"type": "count",
"description": "FOV occupancy count greater than 1",
"result": {
"time": "YYYY-MM-DDTHH:MM:SS.sssZ",
"value": 1
}
},
"direction": "up"
}
]
}
Sample User engagement zone change alert
{
"id": "46ecca77-3bf7-4eab-b719-93fcf870f52a",
"sensor": {
"id": "SENSOR_ID",
"type": "camera"
},
"analyticsModule": {
"version": "1.0",
"id": "emdat_userengagement_alerts",
"source": "Nvidia_eMDX",
"description": "eMDAT UserEngagement alerting module"
},
"alerts": [
{
"id": "SENSOR_ID::userengagement::UE_CONFIG_1::TEST_ALERT_RULE_1::zone_change::YYYY-MM-DDTHH:MM:SS.sssZ",
"rule_id": "b8d2ba48-4881-4969-bf9d-4852efdd853b",
"rule_type": "zone_change",
"start": "YYYY-MM-DDTHH:MM:SS.sssZ",
"end": "YYYY-MM-DDTHH:MM:SS.sssZ",
"data": {
"type": "user",
"description": "User's start zone is ENGAGED",
"result": {
"time": "YYYY-MM-DDTHH:MM:SS.sssZ",
"value": "[ENGAGED|DISTRACTED|DISENGAGED]"
}
},
"attributes": [
{
"name": "object_id",
"value": 10023829383
},
{
"name": "from_zone",
"value": "[ENGAGED|DISTRACTED|DISENGAGED]"
},
{
"name": "to_zone",
"value": "[ENGAGED|DISTRACTED|DISENGAGED]"
}
]
}
]
}
Metropolis FOV Occupancy Alerts Concepts#
Field of View (FOV)#
Field of View (FOV) represents the full camera view. FOV analytics module can track movement of objects (e.g People, Face, Vehicle) within the full field of view and provides time series metrics, and alerts related to it.
Occupancy Alerts#
Alerts are raised when either FOV occupancy state changes between [occupied,vacant] due to presence/absence of Face.
Occupancy Alert Rules#
Alerts’ rules can be configured to raise Alerts when the FOV gets occupied or vacant by entry and exit of Face into the FOV area.
Configure FOV Occupancy Alerts Rule#
Alerts’ rules can be configured to raise Alerts when the FOV gets occupied or vacant by entry and exit of objects into the FOV area.
Tokkio App comes pre-configured with a sensor template definition. This is defined in Tokkio deployment application parameters file tokkio-app-params.yaml under the emdx-analytics section as follows.
For any newly discovered sensor this template is used to configure the sensor with the FOV occupancy alert at runtime.
Default configuration:
emdx-analytics:
sensorTemplate:
alert_rules:
fov:
rules:
- count_threshold: 1
id: alert_rule_id_1
rule_type: occupancy_threshold_switch
time_interval: 1
type: fov
parameters:
- name: time_interval_up
value: 1
- name: time_interval_down
value: 2
roi:
rules: []
tripwire:
rules: []
gaze_rois: []
rois: []
tripwires: []
An installation can have multiple sensors over the period of time. The above sensorTemplate is used to configure each new sensor.
A sensor can have alert rules defined for FOV under the property sensorTemplate.alert_rules.fov.rules.
Description: Configures an Alert rule based on change of occupancy above or below count_threshold of detected objects present in the FOV.
Simple use case is when FOV status changes from vacant (occupancy == 0) to occupied (occupancy > 0). The inverse of this is when occupancy changes from occupied (occupancy > 0) to vacant ( occupancy == 0).
Alert Rule Attributes#
- id: This refers to the id of this alert rule. This should be unique across all alert rules for the FOV
FOV can have multiple alert rules associated with it.
count_threshold: Number of counts threshold used to determine status change.
A count_threshold of zero indicates alert to be raised when occupancy changes from zero to greater than zero.
Inverse of this is when occupancy falls to zero from a positive number greater than zero the alert is raised.
A positive non-zero count_threshold (N) means we want to raise the alert when occupancy exceeds N, raise the UP alert (direction=up), while when the occupancy falls to N or below, raise the DOWN (direction=down) alert.
The alert message direction attribute will contain ‘up’ or ‘down’ to indicate above and below threshold crossing events.
time_interval : Time window. Specified in seconds.
Indicate the duration for which the counts need to be above or below the threshold for the alert to be raised.
It is used to deal jitters in occupancy counts and smoothen the publication of alerts in case of such jitters.
Separate value of time_interval for UP and DOWN events can be specified using the parameters extensions. See parameters property description below
rule_type: ‘occupancy_threshold_switch’
type: fixed value fov
parameters: Additional parameters to customize the alert rule.
Parameters are provided to specify separate values of time_interval for UP and DOWN events.
When specified this overrides the default value of UP/DOWN event specified in the top level time_interval property.
For example to set a time_interval of 2 seconds for UP event and 3 seconds for DOWN event add following two parameter to the parameters property
Custom configuration:
- name: time_interval_up value:2 - name:time_interval_down value:3
Helm Installation Time Values Overrides#
While installing the Tokkio App helm chart these values can be overridden to specify a custom FOV alert rule definition in the values.yaml file.
User Engagement Concepts#
Head Pose Estimation#
At its core, head pose estimation is the process of determining the orientation of a person’s head in three-dimensional space relative to a camera. This enables tracking the user attention towards the camera.
Key Steps in Head Pose Estimation:#
Facial Landmark Detection
Movenet Model is used in determining facial landmarks which encompass vital points such as the tip of the nose, the corners of the eyes, and the ears. Extracting these coordinates is a pivotal aspect of the process.
Camera Calibration
Camera calibration involves the determination of the intrinsic parameters of the camera, including focal length and optical center. The system uses default camera calibration intrinsic param values.
Rotation Estimation
Rotation vectors and matrices take the spotlight to estimate the head’s pose. These vectors delineate the head’s rotation around the x, y, and z axes. Transforming these vectors into meaningful angles empowers us to discern the direction in which the head is oriented. This is done by using PnP algorithm. We extract the face normal vector out of the rotation matrix returned by Pnp algorithm.
Head Pose Angle Calculation
Head pose angle represents how much the head is turned away from the camera. It is the dot product b/w the face’s normal vector and the camera’s forward direction. The face’s normal vector points out from the face, and when the face is directly facing the camera, this vector is aligned with the camera’s forward direction. The forward direction of the camera is along the positive z-axis.
Face Angle Orientation
0° - The head is perfectly aligned with the camera (looking directly at it)180° - The head is turned fully backwards (the face is pointing away from the camera)
Zones#
Zones are non-overlapping rings of different face angle ranges. For example, below illustrates three zones
As described above, head pose angle ranges from 0° to 180° where 0° is directly facing the screen and 180° is facing backwards from the screen. This angle is w.r.t to the y-axis. Users can define their own angle ranges for the zones.
The zones names are pre-determined and cannot be changed
Example:
Below is a sample config with 3 Zones
Zone |
From Angle |
To Angle |
Engaged |
0° |
40° |
Distracted |
40° |
100° |
Disengaged |
100° |
180° |
Alerts#
There are 3 types of UE Alert published over Redis:
START ZONE Alert
Publish an alert when the person (appears first in the scene) and his/her initial zone stabilizes for a configurable time window. We publish its stabilized zone value and the time at which the zone stability window started.
Ex:
{ "id": "46ecca77-3bf7-4eab-b719-93fcf870f52a", "sensor": { "id": "SENSOR_ID", "type": "camera" }, "analyticsModule": { "version": "1.0", "id": "emdat_userengagement_alerts", "source": "Nvidia_eMDX", "description": "eMDAT UserEngagement alerting module" } "alerts": [ { "id": "SENSOR_ID::userengagement::UE_CONFIG_1::TEST_ALERT_RULE_1::zone_change::YYYY-MM-DDTHH:MM:SS.sssZ", "rule_id": "b8d2ba48-4881-4969-bf9d-4852efdd853b", "rule_type": "zone_change", "start": "YYYY-MM-DDTHH:MM:SS.sssZ", "end": "YYYY-MM-DDTHH:MM:SS.sssZ", "data": { "type": "user", "description": "User's start zone is ENGAGED", "result": { "time": "YYYY-MM-DDTHH:MM:SS.sssZ", "value": "ENGAGED" } } "attributes": [ { "name": "object_id", "value": 10023829383 }, { "name": "from_zone", "value": "" }, { "name": "to_zone", "value": "ENGAGED" } ] } ] }
ZONE CHANGE Alert
When the person’s head pose angle changes the zone, and stays in the new zone for a configurable time window, we publish the time of zone transition, current zone and the previous zone.
Ex:
{ "id": "46ecca77-3bf7-4eab-b719-93fcf870f52a", "sensor": { "id": "SENSOR_ID", "type": "camera" }, "analyticsModule": { "version": "1.0", "id": "emdat_userengagement_alerts", "source": "Nvidia_eMDX", "description": "eMDAT UserEngagement alerting module" } "alerts": [ { "id": "SENSOR_ID::userengagement::UE_CONFIG_1::TEST_ALERT_RULE_1::zone_change::YYYY-MM-DDTHH:MM:SS.sssZ", "rule_id": "b8d2ba48-4881-4969-bf9d-4852efdd853b", "rule_type": "zone_change", "start": "2YYYY-MM-DDTHH:MM:SS.sssZ", "end": "YYYY-MM-DDTHH:MM:SS.sssZ", "data": { "type": "user", "description": "User changed the zone to DISTRACTED from ENGAGED", "result": { "time": "YYYY-MM-DDTHH:MM:SS.sssZ", "value": "DISTRACTED" } } "attributes": [ { "name": "object_id", "value": 10023829383 }, { "name": "from_zone", "value": "ENGAGED" }, { "name": "to_zone", "value": "DISTRACTED" } ] } ] }
END ZONE Alert
When the person disappears (out of the scene), we publish the last stable zone occupancy, and the time at which the object disappeared.
Ex:
{ "id": "a6ecca77-3bf7-4eab-b719-93fcD870f23a", "sensor": { "id": "SENSOR_ID", "type": "camera" }, "analyticsModule": { "version": "1.0", "id": "emdat_userengagement_alerts", "source": "Nvidia_eMDX", "description": "eMDAT UserEngagement alerting module" } "alerts": [ { "id": "SENSOR_ID::userengagement::UE_CONFIG_1::TEST_ALERT_RULE_1::zone_change::YYYY-MM-DDTHH:MM:SS.sssZ", "rule_id": "b8d2ba48-4881-4969-bf9d-4852efdd853b", "rule_type": "zone_change", "start": "YYYY-MM-DDTHH:MM:SS.sssZ", "end": "YYYY-MM-DDTHH:MM:SS.sssZ", "data": { "type": "user", "description": "User's end zone is DISTRACTED", "result": { "time": "YYYY-MM-DDTHH:MM:SS.sssZ", "value": "DISTRACTED" } } "attributes": [ { "name": "object_id", "value": 10023829383 }, { "name": "from_zone", "value": "DISTRACTED" }, { "name": "to_zone", "value": "" } ] } ] }
User Engagement Configuration#
Following are the configuration which are required to enable user engagment alerts. These are all defined by default but some of the parameters can be modified to fine tune the user experience.
User Engagement Plugin#
UE plugin is defined by default as following in the ‘occupancy-alerts.’ section:
occupancy-alerts:
plugins:
- name: UserEngagement
className: core.analytics.plugins.ue.ue.UserEngagement
config:
resultRetensionWindowMaxSize: 3600
exponentialSmootheningAlpha: 0.4
Zone Config#
UE zone config is defined by default as following in the ‘occupancy-alerts.sensorTemplate’ section as follows:
occupancy-alerts:
sensorTemplate:
user_engagement:
- id: UE_CONFIG_1
zones:
- id: ENGAGED
name: Near to Camera
from_angle: 0
to_angle: 40
- id: DISTRACTED
name: Slightly tilted Away from Camera
from_angle: 40
to_angle: 100
- id: DISENGAGED
name: Away from Camera
from_angle: 100
to_angle: 180
frameBuffer: 2 # temporal stability
angleBuffer: 5 # spatial stability
Note
By default, the 3 zones(Engaged/Distracted/Disengaged) are pre-defined with angles(from_angle/to_angle). The number and id of the zones cannot be changed. However the angles(from_angle, to_angle) can be modified to to suit the user experience needs.
Params frameBuffer and angleBuffer repesent the temporal and spatial stability and can be modified to fine tune the smoothening algorithm. Please refer to section ‘Head Pose Stability’ to understand the smoothening algorithm.
Zone Change Alert Config#
UE zone change alert is defined by default as following in the ‘occupancy-alerts.sensorTemplate.alert_rules’:
occupancy-alerts:
sensorTemplate:
rules:
- id: UE_CONFIG_1 # Should be equal to sensorTemplate.user_engagement.id as defined previous section.
rule_type: zone_change
time_interval: 1 # Zone change alert published only if attention stays in the new zone for 1 sec
name: UE_ALERT_RULE_1
Note: By default, alert config is defined. If required, we can tweak the time_interval param for better stability with increased latency.
Helm Installation Time Values Overrides#
While installing the Tokkio App helm chart these values can be overridden to specify a custom zone configuration in the values.yaml file.
(Advanced) Head Pose Stability#
There can be noise due to jitters in 2D body pose detections, jerky user head motion. This will result in the head pose vector rapidly changing zones or jitter. For a smooth end user experience we need to suppress such noise/jitters and provide stable head pose zone states.
Following are the two algorithms along with their tuning parameters for stabilizing the zone states:
Spatial stability using angle buffer
Algorithm waits until the head pose angle increases/decreases by X° from the zone transition angle. For eg: if the zone transition angle is 40°, and X=5, the head pose angle should change to > 45 or < 35 and meet this condition for a few frames for a valid zone transition. If the head pose angle stays in the 35°-45° band, zone transition doesn’t happen (helps keep noise in check). This is a configurable parameter in the zones configuration.
Temporal stability using frame buffer
Head pose angle should stay continuously in the transition zone for at least ‘Y’ frames for a valid state change
For eg: if the zone transition angle is 40°, and Y=3 the head pose angle should change from >40 or <40 and the condition holds true for at least 3 frames for a valid zone transition (helps keep noise in check). This is a configurable parameter in the zones configuration.