NVIDIA UFM Telemetry Documentation v1.9
v1.9

Data Collection

NVIDIA® UFM® Telemetry uses the configuration file launch_ibdiagnet_config.ini to control the process of collecting the data. It collects two types of data: Cable info and port counters.

Port counters are collected periodically by setting the parameter sample_rate in seconds.

By default cable info data will not be collected. To enable its collection, add the following flag:

Copy
Copied!
            

plugin_env_CLX_EXPORT_API_DISABLE_CABLEINFO=0

When enabled, cable info data is collected, by default, on every run. It is possible to change the collection frequency to be once every num_iterations using the following setting:

Copy
Copied!
            

plugin_env_CLX_EXPORT_API_CABLE_RUN_ONCE=1

To work with the collected data, you may use the Telemetry CLI, which can be accessed as follows:

Copy
Copied!
            

./bin/clxcli CollectX: set_data_root /tmp/clx_data CollectX: set_data_template {{year}}/{{month}}{{day}}/{{hash1023}}/{{source}}/{{tag}}{{id}}.bin

Cable info data is collected based on a weekly schedule, set with the parameter cable_info_schedule. Time parameter is in the format "day/hrs:mins". For daily collection, it is "hrs:mins".

It is possible to collect the data multiple times during the week. To do that use a comma to separate the times at which collection is to take place. For example,

  • cable_info_schedule= 5/00:00 – collects cable info data on 5th day of the week at midnight

  • cable_info_schedule= 12:00 – collects cable info data midnight at 12:00 every day

  • cable_info_schedule= 5/00:00,12:00 – combines the previous two examples

To work with the collected data, you may use the Telemetry CLI, which can be accessed as follows:

Copy
Copied!
            

[root@r-ufm145 ~]# docker exec -it ufm-telemetry clxcli Read configuration from: /opt/mellanox/collectx/etc/collectx.ini agx_data_root = /data Loaded 2 schemas from /data/schema/schema*.json   CollectX:

The main commands to query and retrieve cable info data are cable_times and cable_info.

  • cables_times – dump times and file names of cable info data files, and you can redirect the output to a file

  • cable_info – dump cable info for a given date or range of dates

The following presents the help menu of the cable_time command:

Copy
Copied!
            

CollectX: help cable_times   Usage: cable_times [TIME] [out=]   [TIME] is one the following: date= past=n[hours|days]   Description: Dump times and file names of cable info data files   Examples: cable_times cable_times date=jun04 cable_times past=15d out=out.csv

Example for cable_time command:

Copy
Copied!
            

CollectX: cable_times Opened 202 files in 0.05 seconds   Cable -----   idx Date Time Filename --- ---------------- ----------------------------------- 1 2020-07-26 04:13 /…/cables_1595725983912963.bin 3 2020-07-26 04:28 /…/cables_1595726884030804.bin

Help menu of cable_info command:

Copy
Copied!
            

CollectX: help cable_info   Usage: cable_info [TIME] [out=]   [TIME] is one of the following: last date= past=n[hours|days] [out=] is to specify output file (optional)   Description: Dump cable info for a given date or range of dates. If "last" arg is given, dumps only the last file. If "out=" file name specified, data will be also dumped to that file.   Examples: cable_info filename cable_info file=filename cable_info last cable_info date=jun04 cable_info past=15d out=cable_info.csv

Example for cable_info command:

Copy
Copied!
            

cable_info /…/cables_1595764809124997.bin   time,source,timestamp,port,lid,guid,port_name,vendor,oui,pn,sn,rev,length,type,supportedspeed,temperature,powerclass, nominalbitrate,cdrenabletxrx,inputeq,outputamp,outputemp,fw_version,attenuation_2.5_5_7_12,rx_power_type, rx_power.1.mw,rx_power.1.dbm,rx_power.2.mw,rx_power.2.dbm,rx_power.3.mw,rx_power.3.dbm,rx_power.4.mw, rx_power.4.dbm,tx_bias.1,tx_bias.2,tx_bias.3,tx_bias.4,tx_power.1.mw,tx_power.1.dbm,tx_power.2.mw,tx_power.2.dbm,tx_power.3.mw,tx_power.3.dbm,tx_power.4.mw,tx_power.4.dbm,cdr_tx_rx_loss_indicator,adaptive_equalization_fault,tx_rx_lol_indicator,temperature_alarm_and_warning,voltage_alarm_and_warning,rx_power_alarm_and_warning,tx_bias_alarm_and_warning,tx_power_alarm_and_warning,diag_supply_voltage,transmitter_technology,eth_com_codes_ext,datacode,lot,tx_adaptive_equalization_freeze,rx_output_disable,tx_adaptive_equalization_enable, 2020-07-26T15:00:12.742710,cable_info,1595764812742710,1,117,0x248a0703008b20ec,ufm-hercules-01/U1/P1,Mellanox,0x2c9,MC2207130-002,MT1442VS07035,A3,2 m,Copper cable- unequalized,SDR/DDR/QDR/FDR,N/A,1,0,N/A N/A,N/A,N/A,N/A,N/A,5 8 11 0,OMA,0.0,-999.999023438,0.0,-999.999023438,0.0,-999.999023438,0.0,-999.999023438,*,*,*,*,0.0,-999.999023438,0.0,-999.999023438,0.0,-999.999023438,0.0,-999.999023438,0,0,0,0,0,0,0,0,0,160,0,14-11-27,8224,0,0x0,0x0, 2020-07-26T15:00:12.742710,cable_info,1595764812742710,1,104,0xe41d2d0300109610,msib-e2edmz-02/U1/P1,Mellanox,0x2c9,MC2207130-002,MT1442VS07035,A3,2 m,Copper cable- unequalized,SDR/DDR/QDR/FDR,N/A,1,0,N/A N/A,N/A,N/A,N/A,N/A,5 8 11 0,OMA,0.0,-999.999023438,0.0,-999.999023438,0.0,-999.999023438,0.0,-999.999023438,*,*,*,*,0.0,-999.999023438,0.0,-999.999023438,0.0,-999.999023438,0.0,-999.999023438,0,0,0,0,0,0,0,0,0,160,0,14-11-27,8224,0,0x0,0x0, 2020-07-26T15:00:12.742710,cable_info,1595764812742710,3,104,0xe41d2d0300109610,msib-e2edmz-02/U1/P3,Mellanox,0x2c9,MC2207130-002,MT1411VS08914,A3,2 m,Copper cable- unequalized,SDR/DDR/QDR/FDR,N/A,1,0,N/A N/A,N/A,N/A,N/A,N/A,5 8 11 0,OMA,0.0,-999.999023438,0.0,-999.999023438,0.0,-999.999023438,0.0,-999.999023438,*,*,*,*,0.0,-999.999023438,0.0,-999.999023438,0.0,-999.999023438,0.0,-999.999023438,0,0,0,0,0,0,0,0,0,160,0,14-03-25,8224,0,0x0,0x0, 2020-07-26T15:00:12.742710,cable_info,1595764812742710,1,187,0xe41d2d03005d2250,ip-forwarder/U1/P1,Mellanox,0x2c9,MC2207130-002,MT1411VS08914,A3,2 m,Copper cable- unequalized,SDR/DDR/QDR/FDR,N/A,1,0,N/A N/A,N/A,N/A,N/A,N/A,5 8 11 0,OMA,0.0,-999.999023438,0.0,-999.999023438,0.0,-999.999023438,0.0,-999.999023438,*,*,*,*,0.0,-999.999023438,0.0,-999.999023438,0.0,-999.999023438,0.0,-999.999023438,0,0,0,0,0,0,0,0,0,160,0,14-03-25,8224,0,0x0,0x0,

The port_counters command is used to extract data in CSV format. It dumps counters matching a given text fragment or "counterset" for a date or range of dates.

Following is the help menu of port_counters command:

Copy
Copied!
            

CollectX: help port_counters   Usage: port_counters [TAGS] [TIME] [out=]   [TAGS] is a list of countersets/name fragments. [TIME] can be specified as: date= past=n[hours|days] from= to= [out=] to specify output file (optional)   Description: Dump port_counters matching a given fragment/counterset for a given date or range of dates.   Example: port_counters error past=10m port_counters error date=jul16 out=error_dump.csv

The following is an example of a port_counters command run:

Copy
Copied!
            

CollectX: port_counters past=300s   idx,time,ts,node,port_num,PortMultiCastXmitPktsExtended,PortUniCastXmitPktsExtended,PortXmitConstraintErrorsExtended, PortXmitDataExtended,PortXmitDiscardsExtended,PortXmitPktsExtended,PortXmitWaitExtended,port_xmit_constraint_errors, port_xmit_data,port_xmit_discard,port_xmit_pkts,port_xmit_wait, 0,2020-07-27T09:57:02,1595833022873349,0xb8599f0300355d6e,1,0,0,0,0,0,0,0,0,0,0,0,0, 1,2020-07-27T09:57:02,1595833022873363,0xb8599f0300355d6e,2,0,0,0,0,0,0,26700642,0,0,0,0,26700642, 2,2020-07-27T09:57:02,1595833022873374,0xb8599f0300355d6e,3,0,9881980474,0,4540230161607,0,9881980474,0,0,0,0,0,0, 3,2020-07-27T09:57:02,1595833022873396,0xb8599f0300355d6e,4,0,0,0,1339496959094,0,0,21581722,0,0,0,0,21581722, 4,2020-07-27T09:57:02,1595833022873408,0xb8599f0300355d6e,9,24766362454,0,0,0,0,0,0,0,0,0,0,0, 5,2020-07-27T09:57:02,1595833022873419,0xb8599f0300355d6e,10,0,54725808986,0,8222892412792,0,29959446532,33957262,0,0,0,0,33957262, …

The switch_temperature command is used to dump switch temperature info for a given date or range of dates into CSV files.

The following presents the help menu of the switch_temperature command:

Copy
Copied!
            

CollectX: help switch_temperature   Usage: switch_temperature [TIME] [out=]   [TIME] is one of the following: last date= past=n[hours|days] [out=] is to specify output file (optional)   Description: Dump switch temperature info for a given date or range of dates. If "out=" file name specified, data will be also dumped to that file.   Examples: switch_temperature filename switch_temperature file=filename switch_temperature date=apr21 switch_temperature past=15d out=switch_temperature.csv

The following is an example of a switch_temperature command run:

Copy
Copied!
            

CollectX: switch_temperature past=10m out=switch_temperature.csv time,source,timestamp,node_guid,sensor_index,mtmp_sensor_name,temperature,max_temperature, 12T17:05:16.332772,0xe41d2d030003e450,1649783116332772,0xe41d2d030003e450,0,,47,51, 2022-04-12T17:05:16.332772,0xe41d2d030003e450,1649783116332772,0xe41d2d030003e450,1,,30,33, 2022-04-12T17:05:16.332772,0xe41d2d030003e450,1649783116332772,0xe41d2d030003e450,2,,33,37, 2022-04-12T17:05:16.332772,0xec0d9a0300b41a50,1649783116332772,0xec0d9a0300b41a50,0,,58,66, 2022-04-12T17:05:16.332772,0xec0d9a0300b41a50,1649783116332772,0xec0d9a0300b41a50,1,,27,31, 2022-04-12T17:05:16.332772,0xec0d9a0300b41a50,1649783116332772,0xec0d9a0300b41a50,2,,33,37, …

The switch_fans command is used to dump switch fans info for a given date or range of dates into CSV files.

The following presents the help menu of the switch_fans command:

Copy
Copied!
            

CollectX: help switch_fans   Usage: switch_fans [TIME] [out=]   [TIME] is one of the following: last date= past=n[hours|days] [out=] is to specify output file (optional)   Description: Dump switch fans info for a given date or range of dates. If "out=" file name specified, data will be also dumped to that file.   Examples: switch_fans filename switch_fans file=filename switch_fans date=jun04 switch_fans past=15d out=switch_fans.csv

The following is an example of a switch_fans command run:

Copy
Copied!
            

CollectX: switch_fans past=10m out=switch_fans.csv   time,source,timestamp,node_guid,sensor_index,fan_speed, 2020-10-04T17:36:05.287397,0xe41d2d0300169e40,1601822165287397,0xe41d2d0300169e40,1,10288, 2020-10-04T17:36:05.287402,0xe41d2d0300169e40,1601822165287402,0xe41d2d0300169e40,2,8823, 2020-10-04T17:36:05.287403,0xe41d2d0300169e40,1601822165287403,0xe41d2d0300169e40,3,10608, 2020-10-04T17:36:05.287404,0xe41d2d0300169e40,1601822165287404,0xe41d2d0300169e40,4,9118, …

The switch_general command is used to dump general switch info for a given date or range of dates into CSV files.

The following presents the help menu of switch_ general command:

Copy
Copied!
            

CollectX: help switch_general   Usage: switch_general [TIME] [out=]   [TIME] is one of the following: last date= past=n[hours|days] [out=] is to specify output file (optional)   Description: Dump switch general info for a given date or range of dates. If "out=" file name specified, data will be also dumped to that file.   Examples: switch_general filename switch_general file=filename switch_general date=jun04 switch_general past=15d out=switch_general.csv

The following is an example of a switch_general command run:

Copy
Copied!
            

CollectX: switch_general past=10m out=switch_general.csv   time,source,timestamp,node_guid,serial_number,part_number,revision,product_name,random_fdb_cap,linear_fdb_cap,linear_fdb_top,mcast_fdb_cap,optimized_slvl_mapping,port_state_change,life_time_value,def_mcast_not_pri_port,def_mcast_pri_port,def_port,part_enf_cap,lids_per_port,mcast_fdb_top,enp0,filter_raw_outb_cap,filter_raw_inb_cap,outb_enf_cap,inb_enf_cap, 2020-10-25T11:41:05.183039,0xe41d2d0300169e40,1603618865183039,0xe41d2d0300169e40,MT1510X10802,MSB7700-EB2F,A6,Scorpion IB EDR,0,49152,7936,16383,1,1,19,255,255,0,32,0,49183,1,1,1,1,1, 2020-10-25T11:42:05.559284,0xe41d2d0300169e40,1603618925559284,0xe41d2d0300169e40,MT1510X10802,MSB7700-EB2F,A6,Scorpion IB EDR,0,49152,7936,16383,1,1,19,255,255,0,32,0,49183,1,1,1,1,1, 2020-10- …

amBER is an output format designed for debugging a cluster in its bringup stage.

The following shows the help menu of the generate amBER report command:

Copy
Copied!
            

CollectX: generate_amber_ib_csv past=1h out=amber_ib.csv

For example:

Copy
Copied!
            

CollectX: help generate_amber_ib_csv Usage: generate_amber_ib_csv TIME [report_type=] [out=] [show_raw_data=]   TIME can be specified as: date= past=n[hours|days] : relative to the current time on the server. from= to=   [out=] to specify output file   [show_raw_data=t|f] boolean to show raw data as is. Default: f   Description: Dump amBER IB report for a given date or range of dates   Example: generate_amber_ib_csv past=10m generate_amber_ib_csv date=jul16 out=amber_ib.csv   TIME: from='sep 23, 2021 16:05:00' from='2021-09-23 16:05:00'

© Copyright 2023, NVIDIA. Last updated on Sep 5, 2023.