ELK#

The ELK stack (Elasticsearch, Logstash, Kibana) provides the data pipeline and analytics infrastructure for all the blueprints. Video analytics data flows from microservices through Kafka or Redis Streams, gets processed by Logstash, indexed into Elasticsearch, and visualized in Kibana.

Elasticsearch#

Elasticsearch is a versatile datastore that provides real-time search and analytics capabilities for large volumes of textual, numerical, geospatial, structured, and unstructured data.

Elasticsearch provides a RESTful API and client libraries for on-demand querying and data manipulation.

Elasticsearch Quick Start Guide can be found here.

Elasticsearch Configuration#

The default Elasticsearch configuration is located at:

deployments/foundational/elk/configs/elasticsearch.yml
Elasticsearch Configuration#

Setting

Default Value

Description

network.host

0.0.0.0

Network interface to bind to for incoming connections.

http.port

9200

HTTP port for REST API access.

xpack.security.enabled

false

Security features are disabled by default for development.

path.data

/tmp/elastic/data

Directory for storing index data.

path.logs

/tmp/elastic/logs

Directory for Elasticsearch log files.

Elasticsearch Indices#

Elasticsearch indices created for the Blueprint applications include:

Elasticsearch indices#

Index Pattern

Description

mdx-raw-*

Index pattern for mdx-raw data, with daily data in separate indices named as mdx-raw-YYYY-MM-dd.

mdx-bev-*

Daily index pattern for mdx-bev data, mdx-bev-YYYY-MM-dd.

mdx-behavior-*

Similar daily index pattern for mdx-behavior data, mdx-behavior-YYYY-MM-dd.

mdx-cluster-labels

Stores cluster IDs, model versions, and cluster names.

mdx-alerts-*

Daily indexing pattern for mdx-alerts data.

mdx-vlm-alerts-*

Daily indexing pattern for VLM-generated alert data.

mdx-events-*

Daily indices for various event data including IN/OUT tripwire events, ENTRY/EXIT ROI events, etc.

mdx-occupancy-reset

Stores occupancy reset data for various locations.

mdx-frames-*

Daily indexing pattern for mdx-frames data.

mdx-mtmc-*

Daily indices for mdx-mtmc data.

mdx-rtls-*

Daily indexing pattern for mdx-rtls data.

mdx-space-utilization-*

Daily indices for space utilization analytics data.

mdx-amr-locations-*

Daily indices for mdx-amr-locations data.

mdx-amr-events-*

Daily indices for AMR events data.

mdx-incidents-*

Daily indices for detected incident data.

mdx-vlm-incidents-*

Daily indices for VLM-detected incidents.

mdx-embed-*

Daily indices for vision embeddings data with dense vector support.

mdx-calibration

Stores current calibration states.

mdx-calibration-audit

Logs each calibration update.

mdx-calibration-images

Stores metadata of calibration images.

mdx-configs

Holds current configurations of various microservices.

mdx-configs-audit

Logs each update made to the microservices’ configurations.

mdx-usd-assets

Stores USD (Universal Scene Description) assets configuration.

mdx-road-network

Stores road network configuration data.

mdx-sensor-lookup

Stores sensor lookup information for coordinate mapping.

Note

You can use the Elasticsearch Python library to query the indices. You can find more details regarding the library and sample code here.

Index Templates#

Index templates automatically apply settings and mappings when new indices are created, ensuring consistency across all video analytics data. Templates are applied based on index name patterns with priority ordering — higher priority templates override lower ones for the same index.

When to review templates:

  • Base template (metropolis_template): Adjust number_of_shards, refresh_interval, or translog.durability for scaling and performance tuning

  • Index-specific templates: Enable or disable field indexing based on your query patterns and storage requirements

For the complete list of templates and their exact mappings, refer to the initialization script:

deployments/foundational/elk/init-scripts/elasticsearch-template-creation.sh

Ingest Pipelines#

Ingest pipelines preprocess documents before indexing. The following pipeline is created during initialization:

insertion-timestamp-pipeline

A pipeline that automatically adds insertion timestamps to documents. Microservices can specify a target field name, and the pipeline populates it with the current UTC timestamp.

The pipeline configuration is located in:

deployments/foundational/elk/init-scripts/elasticsearch-ingest-pipeline-creation.sh

Index Lifecycle Management (ILM)#

Index Lifecycle Management (ILM) is the recommended way to manage Elasticsearch indices in ELK 9.x. ILM allows you to automate index rollover, retention, and deletion based on policies.

Additional information regarding ILM can be found here.

ILM Configuration

The ILM poll interval is configured to 30 seconds (default is 10 minutes) for faster policy execution:

{
  "persistent": {
    "indices.lifecycle.poll_interval": "30s"
  }
}

ILM Policies

The following ILM policies are created during initialization:

ILM Policy Reference#

Policy Name

Current Retention Period

Index Pattern

mdx-raw-ilm-policy

4 hours

mdx-raw-* indices

mdx-bev-ilm-policy

4 hours

mdx-bev-* indices

mdx-behavior-ilm-policy

4 hours

mdx-behavior-* indices

mdx-alerts-ilm-policy

4 hours

mdx-alerts-* indices

mdx-vlm-alerts-ilm-policy

4 hours

mdx-vlm-alerts-* indices

mdx-events-ilm-policy

4 hours

mdx-events-* indices

mdx-frames-ilm-policy

4 hours

mdx-frames-* indices

mdx-mtmc-ilm-policy

4 hours

mdx-mtmc-* indices

mdx-rtls-ilm-policy

4 hours

mdx-rtls-* indices

mdx-space-utilization-ilm-policy

4 hours

mdx-space-utilization-* indices

mdx-amr-locations-ilm-policy

4 hours

mdx-amr-locations-* indices

mdx-amr-events-ilm-policy

4 hours

mdx-amr-events-* indices

mdx-incidents-ilm-policy

4 hours

mdx-incidents-* indices

mdx-vlm-incidents-ilm-policy

4 hours

mdx-vlm-incidents-* indices

mdx-embed-ilm-policy

8 hours

mdx-embed-* indices (longer retention for embeddings)

Note

  • ILM is used to manage index retention and delete old indices.

  • By default, indices are deleted every 4 hours to manage disk space on development systems. The mdx-embed index has 8-hour retention due to the value of embedding data.

  • If the system has sufficient storage capacity, then the data retention period can be increased.

  • To modify the retention duration, edit the ILM policy configuration in:

    deployments/foundational/elk/init-scripts/elasticsearch-ilm-policy-creation.sh

Modifying Retention Periods

To change the retention period for an index, update the ILM policy in the initialization script:

# Example: Change mdx-behavior retention to 24 hours
create_ilm_policy 'mdx-behavior-ilm-policy' \
  '{"policy":{"phases":{"delete":{"min_age":"24h","actions":{"delete":{}}}}}}'

Logstash#

Logstash is a powerful data processing pipeline that helps collect data from source, transform, and ship data to sink. It provides a flexible configuration system for handling various input sources, data transformations, and output destinations.

Logstash reads data from Kafka or Redis Streams and indexes it into Elasticsearch.

Additional information regarding Logstash can be found here.

Logstash Configuration#

The default Logstash configuration is located at:

deployments/foundational/elk/configs/logstash.yml
Logstash Settings#

Setting

Default Value

Description

pipeline.workers

1

Number of workers for pipeline processing.

pipeline.ordered

true

Ensures events are processed in order.

pipeline.ecs_compatibility

disabled

Disables Elastic Common Schema compatibility for legacy format.

xpack.monitoring.elasticsearch.hosts

["http://localhost:9200"]

Elasticsearch hosts for monitoring data.

api.http.host

0.0.0.0

HTTP API binding address.

Kafka vs Redis Streams#

Logstash can consume data from either Kafka or Redis Streams. Two configurations are provided to support different deployment scenarios:

  • Kafka: Datacenter deployments with higher throughput requirements and sufficient infrastructure

  • Redis Streams: Edge deployments with minimal memory footprint and simpler operational overhead

    Input Source Comparison#

    Feature

    Kafka

    Redis Streams

    Throughput

    Optimized for high throughput with batching

    Optimized for low latency with smaller batches

    Memory Footprint

    Higher memory requirements for broker and consumers

    Lightweight, suitable for resource-constrained environments

    Message Retention

    Configurable retention with replay support

    Memory-based with optional persistence (AOF/RDB)

    Consumer Groups

    Native support with offset tracking

    Native support via XREADGROUP

    Setup Complexity

    Single node or cluster (KRaft mode)

    Single instance or Redis Cluster

    Recommended For

    Datacenter and cloud deployments

    Edge deployments with limited resources

Kafka Input Plugin#

For Kafka-based deployments, Logstash uses the built-in Kafka input plugin with Protocol Buffer decoding.

For a complete list of Kafka topics, their data formats, and class names, see Kafka topics.

For complete configuration, refer to:

deployments/foundational/elk/configs/mdx-kafka-logstash.conf

Redis Stream Input Plugin#

For Redis Streams support, a custom Logstash input plugin (logstash-input-redis_stream) is provided that offers:

  • Consumer Group Support - Uses XREADGROUP for reliable message delivery with acknowledgments

  • Binary Data Support - Reads raw binary data from Redis Streams

  • Protobuf Decoding - Native support for decoding Protocol Buffer messages

  • Automatic Reconnection - Handles connection failures with automatic retry

For a complete list of Redis streams, their data formats, and class names, see Redis streams.

For complete configuration, refer to:

deployments/foundational/elk/configs/mdx-redis-logstash.conf

Configuration Options:

Option

Type

Default

Description

host

string

"localhost"

Redis server hostname

port

number

6379

Redis server port

stream_key

string

"mystream"

Redis stream key to read from

group

string

"logstash_group"

Consumer group name

batch_size

number

100

Maximum entries to read per batch

data_field

string

"value"

Field name containing the message data

data_codec

hash

{}

Data decoding configuration:

  • Default {} uses plain text (UTF-8)

  • For protobuf: set type => "protobuf" with class_name and optionally class_file

Kibana#

Kibana is a powerful visualization and analytics platform that works seamlessly with Elasticsearch. It provides an intuitive interface for exploring the data, visualizing the data using dashboards, and managing your Elasticsearch data.

Additional information regarding Kibana can be found here.

Kibana Configuration#

The default Kibana configuration for the deployment is located at:

deployments/foundational/elk/configs/kibana.yml
Kibana Settings#

Setting

Default Value

Description

server.host

0.0.0.0

Network interface for Kibana server.

server.shutdownTimeout

5s

Graceful shutdown timeout period.

elasticsearch.hosts

["http://localhost:9200"]

Elasticsearch cluster endpoints.

monitoring.ui.container.elasticsearch.enabled

true

Enables container-aware monitoring UI.