NVIDIA® UFM® Telemetry adds the ability to stream to multiple destinations using Fluent Bit. The streaming implementation can stream to any Fluent Bit export plugin, with the "Forward" plugin being particularly useful as it allows sending data to a customer-maintained Fluent Bit or FluentD instance which the customer can then configure as based on their requirements.

Exporting Data Using Fluent Bit Export

To export collected data from the UFM Telemetry docker image:

Load, configure, and run the docker image. See the details in the "Software Management" chapter.

Connect to "ufm-telemetry docker bash".

[root@r-ufm ~]# sudo docker exec -it ufm-telemetry bash

Configure/create export files *.exp in export directory /config/fluent_bit_configs/ and set enable=1 for plugins you want to run. Please see details in the "Export Files" section.
Enable Fluent Bit export by setting plugin_env_FLUENT_BIT_EXPORT_ENABLE=1 in /config/launch_ibdiagnet_config.ini.
```
[root@r-ufm ~]# vi /telemetry.config/launch_ibdiagnet_config.ini
…
[fluentbit_export]

plugin_env_FLUENT_BIT_EXPORT_ENABLE=1
plugin_env_FLUENT_BIT_CONFIG_DIR=/telemetry.config/fluent_bit_configs
plugin_env_LD_LIBRARY_PATH=/opt/mellanox/collectx/lib
...
```
Alternatively, you may do this using the configuration script configure_ufm_telemetry_target.py by running:
```
[root@r-ufm ~]# /config/configure_ufm_telemetry_target.py enable-streaming
```
This changes the value of the plugin_env_FLUENT_BIT_EXPORT_ENABLE parameter in the launch_ibdiagnet_config.ini file. See section "Controlling Fluent Bit Streaming" for more details.
Run destination programs that will receive data. See more details in the "Data Forwarding" section.
See the data on the receiving side.

Ibdiagnet will collect and export data periodically as configured by launch_ibdiagnet_config.ini file using the sample_rate parameter.

Export Files

Export destinations are set by configuring .exp files or creating new ones. All export files are placed in the export configuration folder /config/fluent_bit_configs. The easiest way to start is to use documented example exp-files for the following plugins:

forward
stdout
stdout_raw (this plugin is presented only in the Fluent Bit version installed in the UFM Telemetry docker image)

All plugins are disabled by default. To enable a plugin, set enable=1.

Export File Configuration Details

Each export destination has the following fields:

name – configuration name
plugin_name – Fluent Bit plugin name
enable – 1 or 0 values to enable/disable this destination
host – the host for Fluent Bit plugin
port – port for Fluent Bit plugin
msgpack_data_layout – the msgpacked data format. Default is flb_std. The other option is custom. See section "Msgpack Data Layout" for details.
plugin_key=val – key-value pairs of Fluent Bit plugin parameter (optional)
counterset/fieldset – file paths (optional). See the details in section "Cset/Fset Filtering".

Use "#" to comment line.

Msgpack Data Layout

Data layout can be configured using .exp files by setting "msgpack_data_layout=layout".

Two layouts are available:

"flb_std" data layout is an array of 2 fields: timestamp double value and a plain dictionary (key-value pairs). The standard layout is appropriate for all Fluent Bit plugins. For example:
```
[timestamp_val, {"timestamp"->ts_val, type=>"counters/events", "source"=>"source_val", "key_1"=>val_1, "key_2"=>val_2,...}]
```
"custom" data layout is a dictionary of meta-fields and counter fields. Values are placed into a separate plain dictionary. Custom data format can be dumped with "stdout_raw" output plugin of fluent-bit installed or can be forwarded with "forward" output plugin.
Counters example:
```
{"timestamp"=>timestamp_val, "type"=>"counters", "source"=>"source_val", "values"=> {"key_1"=>val_1, "key_2"=>val_2,...}}
```
Events example:
```
{"timestamp"=>timestamp_val, "type"=>"events", "type_name"=>"type_name_val", "source"=>" source_val", "values"=>{"key_1"=>val_1, "key_2"=>val_2,...}}
```

Cset/Fset Filtering

Each export file can optionally use one cset and one fset file to filter UFM Telemetry counters and events data.

Cset file contains tokens per line to filter data with "type"="counters".
Fset contains several blocks started with the header line [event_type_name] and tokens under that header. Fset file is used to filter data with "type"="events".
- Event type names can be prefixed to apply the same tokens to all fitting types. For example, to filter all ethtool events use [ethtool_event_*].

If several tokens are needed to be matched simultaneously use "tok1+tok2+tok3". Exclusive tokens are available too: line "tok1+tok2-tok3-tok4" will filter names that match both tok1 and tok2 and do not match tok3 or tok4.

For more details see documentation in the files ufm_enterprise.cset and ufm_ enterprise.fset under /config/fluent_bit_configs.

The following is the content of /config/fluent_bit_configs/ufm_enterprise.cset:

# put tokens on separate lines

# Tokens are the actual name 'fragments' to be matched
#     port$         # match names ending with token "port"
#     ^port         # match names starting with token "port"
#     ^port$        # include name that is exact token "port
#     port+xmit     # match names that contain both tokens "port" and "xmit"
#     port-support  # match names that contain the token "port" and do not match the "-" token "support"
#
# Tip: To disable counter export put a single token line that fits nothing

# List of available counters:
#
#node_guid
#port_guid
#port_num
#lid
#link_down_counter
#link_error_recovery_counter
#symbol_error_counter
#port_rcv_remote_physical_errors
#port_rcv_errors
#port_xmit_discard
#port_rcv_switch_relay_errors
#excessive_buffer_errors
…

The following is the content of /config/fluent_bit_configs/ufm_enterprise.fset:

# Put your events here

# Usage:
#
# [type_name_1]
# tokens
# [type_name_2]
# tokens
# [type_name_3]
# tokens
# ...

# Tokens are the actual name 'fragments' to be matched
#     port$         # match names ending with token "port"
#     ^port         # match names starting with token "port"
#     ^port$        # include name that is exact token "port
#     port+xmit     # match names that contain both tokens "port" and "xmit"
#     port-support  # match names that contain the token "port" and do not match the "-" token "support"

# The next example will export the whole "switch_fan" events and events "CableInfo" filtered with token "port" :
# [switch_fan]
#
# [CableInfo]
# port

# To know which event type names are available use one of these options:
#     1. Check export and find field "type_name"=>"switch_temperature"
#            OR
#     2. Open log file "/tmp/ibd/ibdiagnet2_port_counters.log" and find event types are printed to log:
#        ...
#        [info] type [CableInfo] is type of interest
#        [info] type [switch_temperature] is type of interest
#        [info] type [switch_fan] is type of interest
#        [info] type [switch_general] is type of interest
#        ...

# Corner cases:
# 1. Empty fset file will export all events.
# 2. Tokens written above/without [event_type] will be ignored.
# 3. If cannot open fset file, warning will be printed, all event types will be exported.

Quick Start Guide for FluentD

Connect to a remote Linux machine via SSH and ensure docker is installed and started on it.
```
[root@r-ufm ~]# sudo service docker start
```

Pull FluentD image:

[root@r-ufm ~]# sudo docker pull fluentd

Create a configuration file for fluentd container.

[root@r-ufm ~]# export fluentd_dir=/tmp/fluentd
[root@r-ufm ~]# mkdir -p $ fluentd_dir
[root@r-ufm ~]# vim $ fluentd_dir/config.conf #fill it with next configuration

<source>
    @type forward
    bind 0.0.0.0
    port 24432
</source>
 
<match ufm_telemetry>
    @type stdout
</match>

Start fluentd collector container.

[root@r-ufm ~]# sudo docker run -it --rm --network host -v $fluentd_dir:/fluentd/etc fluentd -c /fluentd/etc/config.conf -v

For more details refer to "FluentD" on docker hub.

Data Forwarding

Follow the instructions under "Quick Start Guide for FluentD" to prepare remote host with a running FluentD.

Follow the instructions under "Exporting Data Using Fluent Bit Export" to prepare UFM Telemetry with Fluent Bit export capability and ensure it matches the following configurations:

Fluent Bit is enabled (plugin_env_FLUENT_BIT_EXPORT_ENABLE=1) in the launch_ibdiagnet_config.ini file:

[root@r-ufm ~]# grep -a2 fluent /config/launch_ibdiagnet_config.ini

[fluentbit_export]
plugin_env_FLUENT_BIT_EXPORT_ENABLE=1
plugin_env_FLUENT_BIT_CONFIG_DIR=/telemetry.config/fluent_bit_configs
plugin_env_LD_LIBRARY_PATH=/opt/mellanox/collectx/lib

Prepare a forward.exp file to send data to remote host where fluentd is running:

[root@r-ufm ~]# cat /config/fluent_bit_configs/forward.exp

name=ufm-enterprise
enable=1
plugin_name=forward
host=10.209.36.248 # Remote host IP where fluentd is running
port=24432
 
plugin_tag_match_pair=ufm_telemetry

Verify that data is streamed from the CollectX Telemetry plugin and is received on the FluentD collector.

UFM Telemetry Configuration Script

A script to facilitate the configuration of UFM Telemetry is located under the path /config/configure_ufm_telemetry_target.py.

The script is used to set and show sample rate duration, enable and disable streaming capabilities, add, remove, update, enable, disable and review target destinations to receive counters and cable info data, and import filters defined in files to filter streamed data.

[root@r-ufm ~]# /config/configure_ufm_telemetry_target.py -h
 usage: configure_ufm_telemetry_target.py <command> [<args>]

positional arguments:
  {add-target,show-target,remove-target,enable-target,enable-streaming,disable-target,disable-streaming,modify-target,import-filter-file,disable-filter-file,set-sample-rate,show-sample-rate}
                        Commands
    add-target          Add a telemetry target
    show-target         Show telemetry target(s)
    remove-target       Remove a telemetry target
    enable-target       Enable a telemetry target
    enable-streaming    Enable telemetry streaming
    disable-target      Disable a telemetry target
    disable-streaming   Disable telemetry streaming
    modify-target       Modify a telemetry target
    import-filter-file  Import a telemetry target filter file
    disable-filter-file
                        Disable telemetry target filter file
    set-sample-rate     Set telemetry sample rate
    show-sample-rate    Show telemetry sample rate

optional arguments:
  -h, --help            show this help message and exit
  -V, --version         Print version information

Controlling Fluent Bit Streaming

Fluent Bit data streaming is disabled by default. You may enable it by using the script argument enable-streaming (disable-streaming to disable). This changes the value of the plugin_env_FLUENT_BIT_EXPORT_ENABLE parameter in the launch_ibdiagnet_config.ini file.

[root@r-ufm ~]# grep plugin_env_FLUENT_BIT_EXPORT_ENABLE /config/launch_ibdiagnet_config.ini
plugin_env_FLUENT_BIT_EXPORT_ENABLE=0
[root@r-ufm ~]# /config/configure_ufm_telemetry_target.py enable-streaming
[root@r-ufm ~]# grep plugin_env_FLUENT_BIT_EXPORT_ENABLE /config/launch_ibdiagnet_config.ini
plugin_env_FLUENT_BIT_EXPORT_ENABLE=1

Controlling Target Destinations

You can add, remove, update, enable, disable and review many target destinations to receive counters and cable info data.

Use the flag -h to see the details of any operation.

Adding Destination Target

The parameter add-target adds and enables a destination target.

[root@r-ufm ~]# /config/configure_ufm_telemetry_target.py add-target -h
usage: configure_ufm_telemetry_target.py <command> [<args>] add-target
       [-h] -n <[A-Za-z0-9_-] Name size: 32> -H <IPv4> -p <1-65535> -m
       {extended,standard}

optional arguments:
  -h, --help            show this help message and exit
  -n <[A-Za-z0-9_-] Name size: 32>, --target-name <[A-Za-z0-9_-] Name size: 32>
                        Target name
  -H <IPv4>, --target-host <IPv4>
                        IPv4 address
  -p <1-65535>, --target-port <1-65535>
                        Port number
  -m {extended,standard}, --target-message-type {extended,standard}

For example:

[root@r-ufm ~]# /config/configure_ufm_telemetry_target.py add-target --target-name ufm-telemetry --target-host 10.212.145.6 --target-port 24453 -m standard

Displaying Destination Target Details

The parameter show-target displays the details of a destination target.

[root@r-ufm ~]#[root@r-ufm ~]# /config/configure_ufm_telemetry_target.py add-target -h
usage: configure_ufm_telemetry_target.py <command> [<args>] add-target
       [-h] -n <[A-Za-z0-9_-] Name size: 32> -H <IPv4> -p <1-65535> -m
       {extended,standard}

optional arguments:
  -h, --help            show this help message and exit
  -n <[A-Za-z0-9_-] Name size: 32>, --target-name <[A-Za-z0-9_-] Name size: 32>
                        Target name
  -H <IPv4>, --target-host <IPv4>
                        IPv4 address
  -p <1-65535>, --target-port <1-65535>
                        Port number
  -m {extended,standard}, --target-message-type {extended,standard}

For example:

[root@r-ufm ~]# /config/configure_ufm_telemetry_target.py show-target --target-name ufm-telemetry
Enabled:     Yes
     Name:               ufm-telemetry
     Enabled:            Yes
     Host:               10.212.145.6
     Port:               24453
     Message Type:       Standard

Disabling Destination Target

The parameter disable-target disables a destination target.

[root@r-ufm ~]# /config/configure_ufm_telemetry_target.py disable-target -h
usage: configure_ufm_telemetry_target.py <command> [<args>] disable-target
       [-h] -n TARGET_NAME

optional arguments:
  -h, --help            show this help message and exit
  -n TARGET_NAME, --target-name TARGET_NAME

For example:

[root@r-ufm ~]# /config/configure_ufm_telemetry_target.py disable-target --target-name ufm-telemetry
[root@r-ufm ~]# /config/configure_ufm_telemetry_target.py show-target --target-name ufm-telemetry
Enabled:     Yes
     Name:               ufm-telemetry
     Enabled:            No
     Host:               10.212.145.6
     Port:               24453
     Message Type:       Standard

Enabling Destination Target

The parameter enable-target enables a destination target.

[root@r-ufm ~]# /config/configure_ufm_telemetry_target.py enable-target -h
usage: configure_ufm_telemetry_target.py <command> [<args>] enable-target
       [-h] -n TARGET_NAME

optional arguments:
  -h, --help            show this help message and exit
  -n TARGET_NAME, --target-name TARGET_NAME

For example:

[root@r-ufm ~]# /config/configure_ufm_telemetry_target.py enable-target --target-name ufm-telemetry
[root@r-ufm ~]# /config/configure_ufm_telemetry_target.py show-target --target-name ufm-telemetry
Enabled:     Yes
     Name:               ufm-telemetry
     Enabled:            Yes
     Host:               10.212.145.6
     Port:               24453
     Message Type:       Standard

Modifying Destination Target

The parameter modify-target modifies a destination target.

[root@r-ufm ~]# /config/configure_ufm_telemetry_target.py modify-target -h
usage: configure_ufm_telemetry_target.py <command> [<args>] modify-target
       [-h] -n TARGET_NAME [-H <IPv4>] [-p <1-65535>] [-m {extended,standard}]

optional arguments:
  -h, --help            show this help message and exit
  -n TARGET_NAME, --target-name TARGET_NAME
  -H <IPv4>, --target-host <IPv4>
                        IPv4 address
  -p <1-65535>, --target-port <1-65535>
                        Port number
  -m {extended,standard}, --target-message-type {extended,standard}

For example:

[root@r-ufm ~]# /config/configure_ufm_telemetry_target.py modify-target --target-name ufm-telemetry --target-host 10.212.145.7 --target-port 24455 -m standard
[root@r-ufm ~]# /config/configure_ufm_telemetry_target.py show-target --target-name ufm-telemetry
Enabled:     Yes
     Name:               ufm-telemetry
     Enabled:            Yes
     Host:               10.212.145.7
     Port:               24455
     Message Type:       Standard

Removing Destination Target

The parameter remove-target removes a destination target.

[root@r-ufm ~]# /config/configure_ufm_telemetry_target.py remove-target -h
usage: configure_ufm_telemetry_target.py <command> [<args>] remove-target
       [-h] -n TARGET_NAME

optional arguments:
  -h, --help            show this help message and exit
  -n TARGET_NAME, --target-name TARGET_NAME

For example:

[root@r-ufm ~]# /config/configure_ufm_telemetry_target.py remove-target --target-name ufm-telemetry
[root@r-ufm ~]# /config/configure_ufm_telemetry_target.py show-target --target-name ufm-telemetry
Enabled:     Yes
Target ufm-telemetry is missing. Please add it first.

Data Filtration

The configure_ufm_telemetry_target.py script allows users to import filter files to enable filtering streamed data and to disable filter options.

Enabling Data Filtration

To enable filtration of the streamed counters and cable info data, users must create a file containing the appropriate RegEx patterns (one pattern per line to extract the required parameters data).

[root@r-ufm ~]# cat ~/counters_filter
lm_counter
Errors

Then they must import the filter file to a destination, specifying the type of data (counters or cable info) using the parameter import-filter-file.

[root@r-ufm ~]# /config/configure_ufm_telemetry_target.py import-filter-file -h
usage: configure_ufm_telemetry_target.py <command> [<args>] import-filter-file
       [-h] -n TARGET_NAME -t {counters,fields} -f FILE_PATH

optional arguments:
  -h, --help            show this help message and exit
  -n TARGET_NAME, --target-name TARGET_NAME
  -t {counters,fields}, --target-filter-type {counters,fields}
  -f FILE_PATH, --file-path FILE_PATH

For example, to enable filtering streamed data and create filters:

[root@r-ufm ~]# /config/configure_ufm_telemetry_target.py import-filter-file --target-name ufm-telemetry --target-filter-type counters --file-path  ~/counters_filter

On the target destination side, users will receive all the counters include one of texts (lm_counterm Errors).

Disabling Data Filtration

The parameter disable-filter-file disables an imported filtering file.

[root@r-ufm ~]# /config/configure_ufm_telemetry_target.py disable-filter-file -h
usage: configure_ufm_telemetry_target.py <command> [<args>] disable-filter-file
       [-h] -n TARGET_NAME -t {counters,fields}

optional arguments:
  -h, --help            show this help message and exit
  -n TARGET_NAME, --target-name TARGET_NAME
  -t {counters,fields}, --target-filter-type {counters,fields}

For example:

[root@r-ufm ~]# /config/configure_ufm_telemetry_target.py disable-filter-file --target-name ufm-telemetry --target-filter-type counters

On the target destination side, users will receive all the counters without filtering.

Content

Space Tools

On This Page

Exporting Data Using Fluent Bit Export

Export Files

Export File Configuration Details

Msgpack Data Layout

Cset/Fset Filtering

Quick Start Guide for FluentD

Data Forwarding

UFM Telemetry Configuration Script

Controlling Fluent Bit Streaming

Controlling Target Destinations

Adding Destination Target

Displaying Destination Target Details

Disabling Destination Target

Enabling Destination Target

Modifying Destination Target

Removing Destination Target

Data Filtration

Enabling Data Filtration

Disabling Data Filtration