Telemetry

NVIDIA UFM Enterprise User Manual v6.17.1 Download PDF

UFM Telemetry allows the collection and monitoring of InfiniBand fabric port statistics, such as network bandwidth, congestion, errors, latency, and more.

UFM provides a range of telemetry capabilities:

  • Real-time monitoring views

  • Monitoring of multiple attributes

  • Intelligent Counters for error and congestion counters

  • InfiniBand port-based error counters

  • InfiniBand congestion XmitWait counter-based congestion measurement

  • InfiniBand port-based bandwidth data

The telemetry session panels support the following actions:

  • Rearrangement via a straightforward drag-and-drop function

  • Resizing by hovering over the panel's border

UFM Telemetry data is collected via UFM telemetry instances invoked during UFM startup.

Telemetry Instance

Description

REST API

High-Frequency (Primary) Telemetry Instance

A default telemetry session that collects a predefined set of ~30 counters covering bandwidth, congestion, and error metrics, which UFM analyzes and reports.

These counters are used for:

  • Default Telemetry Session - An ongoing session used by the UFM to display UFM WebUI dashboard charts information and for monitoring and analyzing ports threshold events (the session interval is 30 secs by default)

  • Real-Time Telemetry - allows users to define live telemetry sessions for monitoring small subsets of devices or ports and a selected set of counters. For more information, refer to Telemetry - User-Defined Sessions

  • Historical Telemetry - based on the primary telemetry and collects statistical data from all fabric ports and stores them in an internal UFM SQLite database (the session interval is 5 mins by default)

For Default and Real-time Telemetry: Monitoring REST API

For Historical Telemetry: History Telemetry Sessions REST API

Low-Frequency (Secondary) Telemetry Instance

Operates automatically upon UFM startup, offering an extended scope of 120 counters. For a list of the Secondary Telemetry Fields, refer to Low-Frequency (Secondary) Telemetry Fields.

N/A

For direct telemetry endpoint access, which exposes the list of supported counters:

For the High-Frequency (Primary) Telemetry Instance, run the following command:

Copy
Copied!
            

curl http://r-ufm114:9001/csv/cset/converted_enterprise

For the Low-Frequency (Secondary) Telemetry Instance, run the following command:

Copy
Copied!
            

curl http://r-ufm114:9002/csv/xcset/low_freq_debug

Storage Considerations

UFM periodically collects fabric port statistics and saves them in its SQLite database. Before starting up UFM Enterprise, please consider the following disk space utilization for various fabric sizes and duration.
The measurements in the table below were taken with sampling interval set to once per 30 seconds.

Note

Be aware that the default sampling rate is once per 300 seconds. Disk utilization calculation should be adjusted accordingly.

Number of Nodes

Ports per Node

Storage per Hour

Storage per 15 Days

Storage per 30 Days

16

8

1.6 MB

576 MB (0.563 GB)

1152 MB (1.125 GB)

100

8

11 MB

3960 MB (3.867 GB)

7920 MB (7.734 GB)

500

8

50 MB

18000 MB (17.58 GB)

36000 MB (35.16 GB)

1000

8

100 MB

36000 MB (35.16 GB)

72000 MB (70.31 GB)


© Copyright 2024, NVIDIA. Last updated on Jun 18, 2024.