image image image image image


On This Page

Prometheus Endpoint

UFM Telemetry can expose an http or https endpoint to allow simple and effective integration with monitoring systems that work in poll mode and support Prometheus, CSV, or JSON data formats. The endpoint provides only the last data sample. The user cannot obtain statistics for time points in the past.

Supported Formats

An http endpoint provides data in Prometheus format by default. It also supports JSON and CSV formats. The user can request the desired format using a URL prefix, as shown in the table below.

Data FormatURL Prefix

Data Filtering

An http endpoint can provide all sampled data using the default /metrics URL. The filtering functionality described in the Cset/Fset Filtering section is also supported. To use it place <name>.cset or <name>.fset file in appropriate folders. This folder should be stated in  configuration file. See section "Configuring Data Polling Endpoint" for more details.

A filter file name is included in the URL to request that the data be filtered through the particular .cset/.fset file the user intends. For example, if there are two filter files named name1.cset and name2.cset, then URLs /name1 (or /cset/name1 ) and /name2 (or /cset/name2) can be used to get filtered output described in these files accordingly.

The URL prefixes /cset and /fset can also be used to specify which filter file is meant.


File Extension

Folder Parameter in Configuration File





If the cset folder is not explicitly specified in the configuration file, then the cset directory is set the same as the fset directory.




If the fset folder is not explicitly specified in the configuration file, then the fset directory is set the same as the cset directory.

If a URL prefix is not specified, then the filter file will be searched under both cset and fset folders. If they both have files with the same names, then both filters will be applied.

URL Prefixes Priority

A bre are a set URL prefixes can be used to manipulate data output. It is important to use the prefixes in the correct order as they have assigned priorities. The table below shows URL prefixes priority assignments with examples:



Link Examples






Used to show labels from metadata files


/json, /csv





Used to specify output format


/cset, /fset













Used to specify which type of filer file should be applied

Configuring Data Polling Endpoint

To configure the Prometheus endpoint, the keys listed below need to be set in the launch_ibdiagnet_config.ini file.

plugin_env_PROMETHEUS_INDEXES           	port_num
plugin_env_PROMETHEUS_FSET_INDEXES      	port,lid,guid
plugin_env_PROMETHEUS_CSET_DIR          	/config/prometheus_configs/cset

There are several options related to configuring the HTTP polling endpoint. The key plugin_env_PROMETHEUS_ENDPOINT is used to configure the IP interface for endpoint binding. The “” part in the setting above means that any of the host's valid IP addresses can be used. Note that the user can also specify the host's IP address explicitly. 

The plugin_env_PROMETHEUS_ENDPOINT key also configures the data transport. For regular HTTP, prefix to http. To send over a TLS connection, set the prefix to https, set the above mandatory parameters (keys), and select the existing security keys as follows.

A DH (key exchange protoon) file can also be specified if needed as follows:


To use custom labels for Prometheus statistics, a metadata file is used. For details about labels and label file format, see sections "Prometheus Labels" and "Prometheus Label Generation".

There are several options that allow configuring metadata. The file containing the labels used in Prometheus generation is set as follows:


The user can create the metadata file upon system setup or use a script to generate it automatically via script, using the following parameter:

plugin_env_CLX_METADATA_COMMAND=/opt/mellanox/collectx/telem/bin/gen_metadata --fabric compute --file /var/log/ibdiagnet2.ibnetdiscover --output /config/labels.txt

In the above example, the script generates metadata from /var/log/ibdiagnet2.ibnetdiscover. If the user wishes to create the label file manually, the above option should be commented out to prevent periodic overwriting of the content of the metadata file.

By default, the Prometheus endpoint provides statistics with the collection timestamps. The user can decide whether counter values will be passed with or without timestamps by setting the plugin_env_PROMETHEUS_SHOW_TIMESTAMPS parameter to T (true) or F (false), respectively. For example, to send counter values without timestamps, set the parameter as follows:


To use data filters folders with counter set and field sets, the directories where the files are stored should be configured as follows:


Any parameters not explicitly documented should not be changed and should be considered read-only.

Prometheus Labels

For use cases such as UFM Enterprise or UFM Cyber AI where the network topology is known, a human-readable name can be presented based on the GUID.

# TYPE PortXmitDataExtended counter
# TYPE PortXmitPktsExtended counter
PortXmitDataExtended{source="0x0002c90300f172a0", node_guid="2c90300f172a0", port_guid="2c90300f172a2", port_num="2"}  85554128244 1628683905941
PortXmitPktsExtended{source="0x0002c90300f172a0", node_guid="2c90300f172a0", port_guid="2c90300f172a2", port_num="2"}  1188251785  1628683905941

For integration with third-party applications, labels which are more human-readable may be generated using a labels metadata file, as described below.

Prometheus Label Generation

To generate custom labels, a file containing key-value pairs is used. When the keys are matched, the key-value pairs added to the Prometheus labels are generated.

The following is an example of the format of a labels metadata file:

ec0d9a0300b41a50_36|port_id|ec0d9a0300b41a50_36|device_name|SwitchIB Mellanox Technologies|device_type|switch|fabric|compute|hostname||node_desc||level|leaf|peer_level|server
ec0d9a0300b41a50_37|port_id|ec0d9a0300b41a50_37|device_name|SwitchIB Mellanox Technologies|device_type|switch|fabric|compute|hostname||node_desc||level|leaf|peer_level|
ec0d9a0300b41a58_1|port_id|ec0d9a0300b41a58_1|device_name||device_type|switch|fabric|compute|hostname|aggregation|node_desc|aggregation node|level||peer_level|leaf
98039b0300640b92_1|port_id|98039b0300640b92_1|device_name||device_type|host|fabric|compute|hostname|agx-1|node_desc|agx-1 mlx5_0|level|server|peer_level|leaf
98039b0300640c22_1|port_id|98039b0300640c22_1|device_name||device_type|host|fabric|compute|hostname|agx-2|node_desc|agx-2 mlx5_0|level|server|peer_level|leaf
0002c90300f172a0_2|port_id|0002c90300f172a0_2|device_name||device_type|host|fabric|compute|hostname|agx-3|node_desc|agx-3 mlx4_0|level|server|peer_level|leaf
98039b0300640b9a_1|port_id|98039b0300640b9a_1|device_name||device_type|host|fabric|compute|hostname|agx-3|node_desc|agx-3 mlx5_0|level|server|peer_level|leaf

The following is an example of the generated Prometheus output:

# TYPE infiniband_port_xmit_data_bytes counter
# TYPE infiniband_port_rcv_data_bytes counter
# TYPE infiniband_link_error_recovery_events counter
# TYPE infiniband_link_downed_events counter
# TYPE infiniband_cbw gauge
infiniband_port_xmit_data_bytes       {port_id="0002c90300f172a0_2", ADDITIONAL_LABELS}  82218360540 1628602711924
infiniband_port_rcv_data_bytes        {port_id="0002c90300f172a0_2", ADDITIONAL_LABELS}  82218429458 1628602711924
infiniband_link_error_recovery_events {port_id="0002c90300f172a0_2", ADDITIONAL_LABELS}            0 1628602711924
infiniband_link_downed_events         {port_id="0002c90300f172a0_2", ADDITIONAL_LABELS}            0 1628602711924
infiniband_cbw                        {port_id="0002c90300f172a0_2", ADDITIONAL_LABELS}}           0 1628602711924

where ADDITIONAL_LABELS include:
	node_desc="agx-3 mlx5_0"

To enable this functionality, the following additional keys need to be configured:

plugin_env_CLX_EXPORT_API_IBNETDISCOVER_RUN_ONCE 1    # Without this, the script cannot generate the human readable names, nor the level and peer_level.
plugin_env_CLX_METADATA_FILE /path/to/labels/file
plugin_env_CLX_METADATA_COMMAND "python3 /opt/mellanox/collectx/telem/bin/  --fabric compute --file /var/log/ibdiagnet2.ibnetdiscover -o /path/to/labels/file"

To test, the curl command can be used as follows:

[root@jazz11 /]# curl --silent  IP_ADDR_OF_HOST:9100/metrics |egrep   "xmit|rcv" |  tail
port_xmit_discard{device_name="",device_type="host",fabric="compute",hostname="jazz32",level="server",node_desc="jazz32 mlx5_2",peer_level="leaf",port_id="ec0d9a0300c04a54_1"}  0 1629194120043
port_rcv_switch_relay_errors{device_name="",device_type="host",fabric="compute",hostname="jazz32",level="server",node_desc="jazz32 mlx5_2",peer_level="leaf",port_id="ec0d9a0300c04a54_1"}  0 1629194120043
port_rcv_constraint_errors{device_name="",device_type="host",fabric="compute",hostname="jazz32",level="server",node_desc="jazz32 mlx5_2",peer_level="leaf",port_id="ec0d9a0300c04a54_1"}  0 1629194120043
port_xmit_constraint_errors{device_name="",device_type="host",fabric="compute",hostname="jazz32",level="server",node_desc="jazz32 mlx5_2",peer_level="leaf",port_id="ec0d9a0300c04a54_1"}  0 1629194120043