ELK

The ELK stack (Elasticsearch, Logstash, Kibana) provides the data pipeline and analytics infrastructure for all the blueprints. Video analytics data flows from microservices through Kafka or Redis Streams, gets processed by Logstash, indexed into Elasticsearch, and visualized in Kibana.

The Docker deployment uses Elasticsearch 9.4.4, Logstash 9.4.4, and Kibana 9.4.4.

Elasticsearch

Elasticsearch is a versatile datastore that provides real-time search and analytics capabilities for large volumes of textual, numerical, geospatial, structured, and unstructured data.

Elasticsearch provides a RESTful API and client libraries for on-demand querying and data manipulation.

Refer to the Elasticsearch Quick Start Guide.

Elasticsearch Configuration

The default Elasticsearch configuration is located at:

deploy/docker/services/infra/elk/elasticsearch/configs/elasticsearch.yml

Elasticsearch with GPU

Elasticsearch 9.4.4 also supports GPU-based dense vector embedding. To enable it, do the following.

Prerequisites

• Valid Elasticsearch license (required for GPU-based dense vector indexing). Purchase from Elastic subscriptions.
• NVIDIA GPU with supported driver and Docker NVIDIA Container Toolkit (nvidia-container-toolkit) installed.

Elasticsearch license and GPU verification

After Elasticsearch is running, you can use the following. Replace <ip> with your Elasticsearch host (e.g. localhost or the node IP). Port is 9200 if unchanged.

• Check current license:

curl --location 'http://<ip>:9200/_license'

• Apply or update the Elasticsearch license (supply your license JSON as the request body; use --data @path/to/license.json to send a file):

curl --location --request PUT 'http://<ip>:9200/_license?acknowledge=true' \
--header 'Content-Type: application/json' \
--data '@path/to/license.json'

• Check whether GPU is in use (inspect node settings):

curl --location 'http://<ip>:9200/_nodes/settings?pretty=null'

1. Elasticsearch Dockerfiles

The infra service provides the default Elasticsearch Dockerfile and a sample GPU Dockerfile under deploy/docker/services/infra/Dockerfiles/:

• elasticsearch.Dockerfile: Default CPU Elasticsearch image.
• elasticsearch-gpu.Dockerfile: Sample GPU-enabled Elasticsearch Dockerfile for dense vector indexing.

The sample GPU Dockerfile uses a multi-stage build with Elasticsearch and CUDA base images, adds cuvs (GPU-accelerated vector search), and lets Docker Compose build the Elasticsearch container with GPU indexing support.

2. Infra compose (deploy/docker/services/infra/compose.yml)

In deploy/docker/services/infra/compose.yml, find services.elasticsearch.build.dockerfile in the main elasticsearch service and update only that field. Use Dockerfiles/elasticsearch${ELASTICSEARCH_MODE:+-${ELASTICSEARCH_MODE}}.Dockerfile so the value resolves to Dockerfiles/elasticsearch.Dockerfile by default and Dockerfiles/elasticsearch-gpu.Dockerfile when ELASTICSEARCH_MODE=gpu.

elasticsearch-init-container:
  build:
    context: $VSS_APPS_DIR/services/infra
    dockerfile: Dockerfiles/elastic-init.Dockerfile
    network: "host"
  profiles: ["bp_wh_2d","bp_wh_kafka_2d${MINIMAL_PROFILE:+_extended}","bp_wh_kafka_3d${MINIMAL_PROFILE:+_extended}","bp_wh_redis_2d${MINIMAL_PROFILE:+_extended}","bp_wh_redis_3d${MINIMAL_PROFILE:+_extended}","playback_kafka_2d","playback_kafka_3d","playback_redis_2d","playback_redis_3d","bp_developer_search_2d","bp_developer_lvs_2d","bp_developer_alerts_2d_cv", "bp_developer_alerts_2d_vlm"]
  network_mode: "host"
  container_name: vss-elasticsearch-init
  environment:
    - BP_PROFILE=${BP_PROFILE}
    - ELASTICSEARCH_CONNECTION_MAX_ATTEMPTS=${ELASTICSEARCH_CONNECTION_MAX_ATTEMPTS:-20}
    - ELASTICSEARCH_ILM_MIN_AGE=${ELASTICSEARCH_ILM_MIN_AGE:-4h}
    - ELASTICSEARCH_RTVI_CV_EMBEDDINGS_DIM=${ELASTICSEARCH_RTVI_CV_EMBEDDINGS_DIM:-1536}
    - ELASTICSEARCH_VISION_LLM_EMBEDDINGS_DIM=${ELASTICSEARCH_VISION_LLM_EMBEDDINGS_DIM:-768}
  command: bash -c "
    /opt/mdx/init-scripts/elasticsearch-ilm-policy-creation.sh &&
    /opt/mdx/init-scripts/elasticsearch-template-creation.sh &&
    /opt/mdx/init-scripts/elasticsearch-ingest-pipeline-creation.sh
    "
  depends_on:
    - elasticsearch

3. Environment

In your deployment environment (e.g., .env or export before running compose):

• Set ELASTICSEARCH_MODE=gpu to build and run the GPU image.
• Optionally set ELASTICSEARCH_GPU_DEVICE_ID when Elasticsearch should use a GPU other than 0. For example, set ELASTICSEARCH_GPU_DEVICE_ID=1 to reserve GPU 1.

Add the following GPU reservation block under the main elasticsearch service in deploy/docker/services/infra/compose.yml:

deploy:
  resources:
    reservations:
      devices:
        - capabilities:
            - gpu
          device_ids:
            - "${ELASTICSEARCH_GPU_DEVICE_ID:-0}"

After updating the Dockerfile and compose, rebuild and start the stack so the Elasticsearch container uses the GPU image and has access to the specified GPU. To verify GPU usage, when dense vectors are being indexed, use nvtop or nvidia-smi.

Elasticsearch Indices

Elasticsearch indices created for the Blueprint applications include:

Caption and summary indices use a collection name rather than the Kafka topic name or a daily suffix:

• If the message includes info.collection_name, Logstash uses that value as the index name. For example, collection_name: "warehouse-summary" writes to warehouse-summary.
• Otherwise, Logstash prefixes the stream ID with default_, replacing hyphens, slashes, backslashes, and spaces with underscores. For example, streamId: "camera-01" writes to default_camera_01.

Index Templates

Index templates automatically apply settings and mappings when new indices are created, ensuring consistency across all video analytics data. Templates are applied based on index name patterns with priority ordering — higher priority templates override lower ones for the same index.

When to review templates:

• Base template (metropolis_template): Adjust number_of_shards, refresh_interval, or translog.durability for scaling and performance tuning
• Index-specific templates: Enable or disable field indexing based on your query patterns and storage requirements

For the complete list of templates and their exact mappings, refer to the initialization script:

deploy/docker/services/infra/elk/elasticsearch/init-scripts/elasticsearch-template-creation.sh

Ingest Pipelines

Ingest pipelines preprocess documents before indexing. The following pipeline is created during initialization:

insertion-timestamp-pipeline

A pipeline that automatically adds insertion timestamps to documents. Microservices can specify a target field name, and the pipeline populates it with the current UTC timestamp.

The pipeline configuration is located in:

deploy/docker/services/infra/elk/elasticsearch/init-scripts/elasticsearch-ingest-pipeline-creation.sh

Index Lifecycle Management (ILM)

Index Lifecycle Management (ILM) is the recommended way to manage Elasticsearch indices in ELK 9.x. ILM allows you to automate index rollover, retention, and deletion based on policies.

Refer to Index lifecycle management for additional ILM information.

ILM Configuration

The ILM poll interval is configured to 30 seconds (default is 10 minutes) for faster policy execution:

{
  "persistent": {
    "indices.lifecycle.poll_interval": "30s"
  }
}

ILM Policies

The following ILM policies are created during initialization:

deploy/docker/services/infra/elk/elasticsearch/init-scripts/elasticsearch-ilm-policy-creation.sh

Modifying Retention Periods

To change the retention period for an index, update the ILM policy in the initialization script:

# Example: Change mdx-behavior retention to 24 hours
create_ilm_policy 'mdx-behavior-ilm-policy' \
  '{"policy":{"phases":{"delete":{"min_age":"24h","actions":{"delete":{}}}}}}'

Logstash

Logstash is a powerful data processing pipeline that helps collect data from source, transform, and ship data to sink. It provides a flexible configuration system for handling various input sources, data transformations, and output destinations.

Logstash reads data from Kafka or Redis Streams and indexes it into Elasticsearch.

The Docker deployment uses Logstash 9.4.4.

Refer to the Logstash introduction for additional information.

Logstash Configuration

The default Logstash configuration is located at:

deploy/docker/services/infra/elk/logstash/configs/logstash.yml

Kafka vs Redis Streams

Logstash can consume data from either Kafka or Redis Streams. Two configurations are provided to support different deployment scenarios:

• Kafka: Datacenter deployments with higher throughput requirements and sufficient infrastructure
• Redis Streams: Edge deployments with minimal memory footprint and simpler operational overhead

Kafka Input Plugin

For Kafka-based deployments, Logstash uses the built-in Kafka input plugin with Protocol Buffer decoding.

For a complete list of Kafka topics, their data formats, and class names, see Kafka topics.

Configuration Options:

For complete configuration, refer to:

deploy/docker/services/infra/elk/logstash/pipelines/kafka/mdx-logstash.conf

Redis Stream Input Plugin

For Redis Streams support, a custom Logstash input plugin (logstash-input-redis_stream) is provided that offers:

• Consumer Group Support - Uses XREADGROUP for reliable message delivery with acknowledgments
• Binary Data Support - Reads raw binary data from Redis Streams
• Protobuf Decoding - Native support for decoding Protocol Buffer messages
• Automatic Reconnection - Handles connection failures with automatic retry

For a complete list of Redis streams, their data formats, and class names, see Redis streams.

For complete configuration, refer to:

deploy/docker/services/infra/elk/logstash/pipelines/redis/mdx-logstash.conf

Configuration Options:

Kibana

Kibana is a powerful visualization and analytics platform that works seamlessly with Elasticsearch. It provides an intuitive interface for exploring the data, visualizing the data using dashboards, and managing your Elasticsearch data.

The Docker deployment uses Kibana 9.4.4.

Refer to the Kibana documentation for additional information.

Kibana Configuration

The default Kibana configuration for the deployment is located at:

deploy/docker/services/infra/elk/kibana/configs/kibana.yml

Kibana Dashboards

Kibana dashboards combine multiple visualizations (tables, histograms, charts) into a single view for exploring and analyzing Elasticsearch data. Some deployments (for example, the 2D and 3D Warehouse Vision AI Profiles) include visualizations, and dashboards.

For more on creating and using dashboards, see the Kibana documentation on dashboards. For building individual visualizations, see Create visualizations.

FAQ

ELK Services Timeouts or Startup Health Check Failures

The Docker health checks for ELK services are defined in the deployment compose file:

video-search-and-summarization/deploy/docker/services/infra/compose.yml

From the deployment repository root, this is deploy/docker/services/infra/compose.yml. If Kibana shows Request timeout: "Request timed out", inspect the relevant service logs before changing Kibana settings.

A common root cause is Elasticsearch disk pressure. In Kibana Discover, this can appear as Unable to retrieve search results with Request timeout: "Request timed out". Check Elasticsearch allocation and disk watermarks before tuning Kibana timeouts. See the Elasticsearch disk watermark FAQ below.

If Docker marks Elasticsearch, Kibana, or Logstash as unhealthy during startup, check the container logs to confirm the service is starting slowly rather than failing because of a configuration issue. On slower hosts, increase the affected service’s healthcheck.retries value in this compose file, for example from 60 to 90 or 120, and redeploy. This only gives the ELK service more time to pass its startup health check; it does not fix ingestion or indexing issues.

Elasticsearch Disk Watermarks and Read-Only Indices

Elasticsearch uses disk-based shard allocation to keep data nodes from filling their disks. One common symptom is Kibana Discover failing with Unable to retrieve search results and Request timeout: "Request timed out". The default watermarks are:

Use these commands to inspect disk allocation and the active watermark settings. Replace <ip> with your Elasticsearch host, such as localhost or the node IP.

curl --location "http://<ip>:9200/_cat/allocation?v"
curl --location "http://<ip>:9200/_cluster/settings?include_defaults=true&filter_path=**.disk.watermark*"

If flood stage is reached, fix the disk pressure first. Common options are:

• Remove unnecessary local data or logs from the Elasticsearch host.
• Remove unnecessary containers which are not being used.
• Add disk capacity or add Elasticsearch capacity so shards have somewhere safe to move.

After disk usage drops below the high watermark, Elasticsearch normally removes the read-only block automatically. If the block remains after disk pressure is fixed, clear it manually:

curl --location --request PUT "http://<ip>:9200/*/_settings?expand_wildcards=all" \
  --header "Content-Type: application/json" \
  --data "{\"index.blocks.read_only_allow_delete\": null}"

Avoid increasing or disabling disk watermarks as a long-term fix. Doing so can allow Elasticsearch to fill the disk completely, which may cause indexing failures, service instability, or data loss.