Behavior Analytics

The Behavior Analytics microservice is implemented in Python and leverages the Python Multiprocessing library for parallel processing.

The perception layer processes camera or video input streams, performing sequential frame analysis for object detection and tracking. It generates object embeddings and sends protobuf messages containing frame metadata to a message broker. The pipeline supports multiple message brokers including Kafka, Redis Streams, and MQTT, allowing flexibility based on deployment requirements.

The metadata from the perception layer represents a frame containing object detection bounding boxes, tracking IDs, embedding vectors, and other attributes. This input is processed by the pipeline to generate Behavior data. A Behavior represents an object tracked over time with its associated attributes.

Configurations

The Behavior Analytics microservice requires configuration files that control various aspects of the analytics pipeline. These files allow users to customize the system’s behavior according to their specific requirements.

Application Configuration File

The main configuration file for Behavior Analytics controls message broker settings, sensor configurations, and pipeline parameters. The configuration supports multiple message broker types (Kafka, Redis Streams, MQTT).

Example Warehouse 2D Kafka configuration:

{
  "kafka": {
    "brokers": "kafka:29092",
    "group": "mdx-spatial-analytics-2d-app",
    "consumer": {
      "autoOffsetReset": "latest",
      "enableAutoCommit": false,
      "maxPollIntervalMs": 900000,
      "maxPartitionFetchBytes": 10485760,
      "fetchMaxBytes": 104857600,
      "maxPollRecords": 10000,
      "timeout": 0.01
    },
    "producer": {
      "lingerMs": 0
    },
    "topics": [
      {"name": "raw", "value": "mdx-raw"},
      {"name": "frames", "value": "mdx-frames"},
      {"name": "behavior", "value": "mdx-behavior"},
      {"name": "notification", "value": "mdx-notification"},
      {"name": "events", "value": "mdx-events"},
      {"name": "incidents", "value": "mdx-incidents"}
    ]
  },
  "sensors": [
    {
      "id": "default",
      "configs": [
        {"name": "proximityDetectionEnable", "value": "true"},
        {"name": "proximityDetectionThreshold", "value": "4"},
        {"name": "proximityDetectionCenterClasses", "value": "[\"Forklift\"]"},
        {"name": "proximityDetectionSurroundingClasses", "value": "[\"Person\"]"}
      ]
    }
  ],
  "app": [
    {"name": "sourceType", "value": "kafka"},
    {"name": "sinkType", "value": "kafka"},
    {"name": "numWorkersForBehaviorCreation", "value": "2"},
    {"name": "numWorkersForFrameEnhancement", "value": "2"},
    {"name": "behaviorWatermarkSec", "value": "30"},
    {"name": "behaviorMaxPoints", "value": "200"},
    {"name": "proximityViolationIncidentEnable", "value": "true"},
    {"name": "proximityViolationIncidentThreshold", "value": "2"},
    {"name": "restrictedAreaViolationIncidentEnable", "value": "true"},
    {"name": "confinedAreaViolationIncidentEnable", "value": "true"}
  ]
}

Message Broker Configuration

Kafka Topics

Pipeline Configuration

Playback Configuration

State Management Configuration

Proximity Detection Configuration (Sensor-level)

Incident Detection Configuration

Tripwire Configuration (Sensor-level)

Frame Management Configuration (Sensor-level)

Calibration File

The calibration file defines camera parameters, regions of interest (ROIs), tripwires, and coordinate transformations. Use the Calibration tool to generate this JSON file. For more details, refer to the Calibration documentation. For the calibration JSON structure, see Calibration Schema.

Runtime Configuration Adjustments

This section covers common configuration changes for deploying the Behavior Analytics pipeline.

Modifying Message Broker Settings

To switch between message brokers (Kafka, Redis Streams, MQTT), update the sourceType and sinkType parameters:

{
  "app": [
    {"name": "sourceType", "value": "redisStream"},
    {"name": "sinkType", "value": "redisStream"}
  ]
}

Ensure the corresponding broker configuration section is properly configured (kafka, redisStream, or mqtt).

Configuring Worker Parallelism

Adjust the number of workers based on your deployment’s processing requirements:

{
  "app": [
    {"name": "numWorkersForBehaviorCreation", "value": "4"},
    {"name": "numWorkersForFrameEnhancement", "value": "4"}
  ]
}

Enabling Violation Detection

To enable specific violation types for incident generation:

{
  "app": [
    {"name": "proximityViolationIncidentEnable", "value": "true"},
    {"name": "restrictedAreaViolationIncidentEnable", "value": "true"},
    {"name": "confinedAreaViolationIncidentEnable", "value": "true"},
    {"name": "fovCountViolationIncidentEnable", "value": "true"},
    {"name": "fovCountViolationIncidentObjectThreshold", "value": "5"},
    {"name": "fovCountViolationIncidentObjectType", "value": "Person"}
  ]
}

Configuring Sensor-Specific Settings

Override default configurations for specific sensors:

{
  "sensors": [
    {
      "id": "default",
      "configs": [
        {"name": "proximityDetectionEnable", "value": "false"},
        {"name": "tripwireMinPoints", "value": "5"}
      ]
    },
    {
      "id": "Warehouse_Cam1",
      "configs": [
        {"name": "proximityDetectionEnable", "value": "true"},
        {"name": "proximityDetectionThreshold", "value": "2.0"},
        {"name": "proximityDetectionCenterClasses", "value": "[\"Agility_Digit_Humanoid\"]"},
        {"name": "proximityDetectionSurroundingClasses", "value": "[\"Person\"]"}
      ]
    }
  ]
}

Sensors inherit from the default configuration and override only the specified parameters.

Updating Configuration Without a Restart

Most app and sensors keys can be changed at runtime by publishing an upsert to the mdx-notification topic with the message key behavior-analytics-config. The service validates each item against an allowlist, applies the accepted ones, and publishes an ack carrying the post-apply state and any per-item rejections.

Keys that are read once at startup — sourceType, sinkType, every numWorkersFor* count, and the whole kafka / redisStream / mqtt / coordinateReferenceSystem / inference sections — are rejected by the validator rather than silently ignored, so those changes still require a restart.

Calibration updates use the same topic with the message key calibration. Unlike config updates, calibration updates produce no ack; a rejected payload is visible only in the container logs.

For the full wire contract, the mutable/read-only key lists, and ack semantics, see the Behavior Analytics reference.