Telemetry Streaming and Integration with Elasticsearch, Logstash and Kibana (ELK) Stack

NVIDIA Telemetry Agent User Manual v2.7.20

ELK is the acronym for three open source tools: Elasticsearch, Logstash, and Kibana.

  1. Elasticsearch – the database where the data (for example, counters) is stored to be viewed later

  2. Logstash – an engine that collects data from multiple sources and forwards it to the Elasticsearch tool; it runs with a configuration file that specifies the input (e.g. UDP with a specific port) and the data format

  3. Kibana – the visualization tool that receives the data from the database and displays it in graphs

    Telemetry-ELK.png

To forward data to the Logstash server (to be stored in the Elasticsearch database), the port that the server listens on and the IP address of that machine should be specified in the collector settings of the telemetry.

Run the Logstash to start receiving new data to be added to the Elasticsearch database:

Copy
Copied!
            

/usr/share/logstash/bin/logstash -f /tmp/counter.conf

The following is the content of the counters.conf file, which is reading data from the UDP port and forwarding it to the Elasticsearch database:

image2019-7-8_12-46-40.png

Perform the following steps to run the telemetry agent from NEO, and send data to the port that the Logstash tool is listening on:

  1. Login to NEO (http://<neo_server_ip>/neo).

  2. Remove the previous running instances from the agent.

    Start_Agent.png

  3. Run a new instance.

    Stop_Agent.png

  4. Go to “Settings” on the main NEO menu, click the Telemetry tab, and add your data collector. Choose a name for the collector and insert the port that the Logstash will listen on, and the IP of the machine on which the Logstash is installed.

    image2019-7-8_12-48-37.png

Viewing Data on Kibana Tool

The Kibana tool obtains the data stored in the Elasticsearch database and uses indices to determine which groups of data to display. By default, the Logstash stores all the data received on the same day with the same index in the Elasticsearch, using the following name format: “logstash-<today date>”.

Therefore, the first time that data is inserted, when logging into Kibana (

:5601); the index pattern that matches the stored indices must be defined.

image2019-7-8_12-50-3.png

Adding New Visualization

To add a new visualization:

  1. Go to the “Visualize” tab, and click the “+” button.

  2. Select the desired visualization chart.

    image2019-7-8_12-50-31.png

    To expose the rate counters, it is recommended to select a line chart, as it shows how the counter is changing with time. The wizard will ask you which index pattern to use. You may select the saved index pattern (logs*).

    image2019-7-8_12-50-47.png

    You may change the Y and X axes parameters. For example:

    • On the Y-axis, you can use the aggregation "average" for frames_transmitted_ok_rate

    • On the X-axis, you can use "Date Histogram" and "Field"

      image2019-7-8_12-51-11.png

      In the above example, the selected port is "sw1p9". If no port is specified, the average of the frame_recieved_ok_rate value of all ports will be shown.

      To add a filter, click the "Add a filter +" button. You may add a filter to show a specific port (e.g. "sw1p11" only). You can also save the visualization and reload it later.

      image2019-7-8_12-52-17.png

© Copyright 2023, NVIDIA. Last updated on Nov 16, 2023.