Skip to navigation
DeepStream Key Features

DeepStream OpenTelemetry Support

View as Markdown

DeepStream provides OpenTelemetry support via nvmultiurisrcbin, enabling the export of pipeline performance metrics to observability platforms like Prometheus and Grafana.

OpenTelemetry support is available for both x86 (DeepStream 9.0 onwards) and Jetson platforms. The Gst-nvdslogger plugin collects performance data (FPS, latency, frame numbers) from the GStreamer pipeline. This data is stored in shared memory structures managed by the nvds_rest_metrics library and exported to OpenTelemetry collectors using OTLP/HTTP.

OpenTelemetry is supported through nvmultiurisrcbin. To activate it, incorporate nvmultiurisrcbin into your pipeline.

Configuration

Configure OpenTelemetry using environment variables:

Environment VariableDescription
OTEL_SDK_DISABLEDSet to "true" to disable all telemetry (default: "false")
OTEL_SERVICE_NAMEService identifier (e.g., "rtvi-cv")
OTEL_EXPORTER_OTLP_ENDPOINTCollector base URL (e.g., "http://otel-collector:4318")
OTEL_METRIC_EXPORT_INTERVALMetric export interval in milliseconds (default: 60000)
OTEL_METRICS_EXPORTERExport destination: "console", "otlp", or "none" (default: "otlp")

Set the following parameters in the deepstream-test5 application configuration file (/opt/nvidia/deepstream/deepstream/sources/apps/sample_apps/deepstream-test5/configs/test5_config_file_nvmultiurisrcbin_src_list_attr_all.txt):

[tiled-display]
enable=3
[sinkN]
nvdslogger=1

Supported Prometheus Metrics

Stream Performance Metrics

Metric NameDescriptionTypical Value
stream_fpsFrames per second processed for each stream25-30 (depends on source)
stream_latency_millisecondsEnd-to-end pipeline latency in milliseconds30-100ms (lower is better)
stream_frame_numberCurrent frame number being processed for each streamMonotonically increasing
stream_countTotal number of active streams being processedBased on configuration

System Resource Metrics

Metric NameDescription
cpu_utilizationCPU utilization percentage across all cores
gpu_utilizationGPU compute utilization percentage
ram_memory_gbSystem RAM memory usage in gigabytes
gpu_memory_gbGPU memory usage in gigabytes

gpu_memory_gb is not applicable on aarch64 devices (e.g., Jetson Thor) as they use unified memory, so it returns -1.

OpenTelemetry Collector Configuration

To filter out inactive stream metrics, add the following processor to your collector configuration:

processors:
  filter/drop_inactive_streams:
    error_mode: ignore
    metrics:
      datapoint:
        - 'metric.name == "stream_fps" and value_double == -1.0'
        - 'metric.name == "stream_latency" and value_double == -1.0'
        - 'metric.name == "stream_frame_number" and value_int == -1'

If exporting to Prometheus, set metric_expiration >= otlp-interval to drop stale metrics:

exporters:
  prometheus:
    endpoint: "0.0.0.0:8889"
    metric_expiration: 4s

For steps to set up and run the OpenTelemetry Collector, refer to the DeepStream OpenTelemetry Collector README.

References

For more information about OpenTelemetry, refer to the following resources: