Certain providers or components are unable to execute properly within the container due to various container limitations. Therefore, they would have to perform remote collection or execution.

#disable-provider=sysfs

enable-provider=sysfs
disable-provider=sysfs.eth

• ib_port counters:

  {hca_name}:{port_num}:ib_port_state
  {hca_name}:{port_num}:VL15_dropped
  {hca_name}:{port_num}:excessive_buffer_overrun_errors
  {hca_name}:{port_num}:link_downed
  {hca_name}:{port_num}:link_error_recovery
  {hca_name}:{port_num}:local_link_integrity_errors
  {hca_name}:{port_num}:multicast_rcv_packets
  {hca_name}:{port_num}:multicast_xmit_packets
  {hca_name}:{port_num}:port_rcv_constraint_errors
  {hca_name}:{port_num}:port_rcv_data
  {hca_name}:{port_num}:port_rcv_errors
  {hca_name}:{port_num}:port_rcv_packets
  {hca_name}:{port_num}:port_rcv_remote_physical_errors
  {hca_name}:{port_num}:port_rcv_switch_relay_errors
  {hca_name}:{port_num}:port_xmit_constraint_errors
  {hca_name}:{port_num}:port_xmit_data
  {hca_name}:{port_num}:port_xmit_discards
  {hca_name}:{port_num}:port_xmit_packets
  {hca_name}:{port_num}:port_xmit_wait
  {hca_name}:{port_num}:symbol_error
  {hca_name}:{port_num}:unicast_rcv_packets
  {hca_name}:{port_num}:unicast_xmit_packets

• ib_hw counters:

  {hca_name}:{port_num}:hw_state
  {hca_name}:{port_num}:hw_duplicate_request
  {hca_name}:{port_num}:hw_implied_nak_seq_err
  {hca_name}:{port_num}:hw_lifespan
  {hca_name}:{port_num}:hw_local_ack_timeout_err
  {hca_name}:{port_num}:hw_out_of_buffer
  {hca_name}:{port_num}:hw_out_of_sequence
  {hca_name}:{port_num}:hw_packet_seq_err   
  {hca_name}:{port_num}:hw_req_cqe_error
  {hca_name}:{port_num}:hw_req_cqe_flush_error
  {hca_name}:{port_num}:hw_req_remote_access_errors   
  {hca_name}:{port_num}:hw_req_remote_invalid_request   
  {hca_name}:{port_num}:hw_resp_cqe_error
  {hca_name}:{port_num}:hw_resp_cqe_flush_error
  {hca_name}:{port_num}:hw_resp_local_length_error
  {hca_name}:{port_num}:hw_resp_remote_access_errors
  {hca_name}:{port_num}:hw_rnr_nak_retry_err   
  {hca_name}:{port_num}:hw_rx_atomic_requests   
  {hca_name}:{port_num}:hw_rx_dct_connect
  {hca_name}:{port_num}:hw_rx_icrc_encapsulated
  {hca_name}:{port_num}:hw_rx_read_requests   
  {hca_name}:{port_num}:hw_rx_write_requests

• ib_mr_cache counters:
  Note:n ranges from 0 to 24.

  {hca_name}:mr_cache:size_{n}:cur
  {hca_name}:mr_cache:size_{n}:limit
  {hca_name}:mr_cache:size_{n}:miss
  mlx5_0:mr_cache:size_{n}:size

• eth counters:

  {hca_name}:{device_name}:eth_collisions
  {hca_name}:{device_name}:eth_multicast
  {hca_name}:{device_name}:eth_rx_bytes
  {hca_name}:{device_name}:eth_rx_compressed
  {hca_name}:{device_name}:eth_rx_crc_errors
  {hca_name}:{device_name}:eth_rx_dropped
  {hca_name}:{device_name}:eth_rx_errors
  {hca_name}:{device_name}:eth_rx_fifo_errors
  {hca_name}:{device_name}:eth_rx_frame_errors
  {hca_name}:{device_name}:eth_rx_length_errors
  {hca_name}:{device_name}:eth_rx_missed_errors
  {hca_name}:{device_name}:eth_rx_nohandler
  {hca_name}:{device_name}:eth_rx_over_errors
  {hca_name}:{device_name}:eth_rx_packets
  {hca_name}:{device_name}:eth_tx_aborted_errors
  {hca_name}:{device_name}:eth_tx_bytes
  {hca_name}:{device_name}:eth_tx_carrier_errors
  {hca_name}:{device_name}:eth_tx_compressed
  {hca_name}:{device_name}:eth_tx_dropped
  {hca_name}:{device_name}:eth_tx_errors
  {hca_name}:{device_name}:eth_tx_fifo_errors
  {hca_name}:{device_name}:eth_tx_heartbeat_errors
  {hca_name}:{device_name}:eth_tx_packets
  {hca_name}:{device_name}:eth_tx_window_errors

• BlueField-2 hwmon counters:

  {hwmon_name}:{l3cache}:CYCLES
  {hwmon_name}:{l3cache}:HITS_BANK0
  {hwmon_name}:{l3cache}:HITS_BANK1
  {hwmon_name}:{l3cache}:MISSES_BANK0
  {hwmon_name}:{l3cache}:MISSES_BANK1
  {hwmon_name}:{pcie}:IN_C_BYTE_CNT
  {hwmon_name}:{pcie}:IN_C_PKT_CNT
  {hwmon_name}:{pcie}:IN_NP_BYTE_CNT
  {hwmon_name}:{pcie}:IN_NP_PKT_CNT
  {hwmon_name}:{pcie}:IN_P_BYTE_CNT
  {hwmon_name}:{pcie}:IN_P_PKT_CNT
  {hwmon_name}:{pcie}:OUT_C_BYTE_CNT
  {hwmon_name}:{pcie}:OUT_C_PKT_CNT
  {hwmon_name}:{pcie}:OUT_NP_BYTE_CNT
  {hwmon_name}:{pcie}:OUT_NP_PKT_CNT
  {hwmon_name}:{pcie}:OUT_P_PKT_CNT
  {hwmon_name}:{tile}:MEMORY_READS
  {hwmon_name}:{tile}:MEMORY_WRITES
  {hwmon_name}:{tile}:MSS_NO_CREDIT
  {hwmon_name}:{tile}:VICTIM_WRITE
  {hwmon_name}:{tilenet}:CDN_DIAG_C_OUT_OF_CRED
  {hwmon_name}:{tilenet}:CDN_REQ
  {hwmon_name}:{tilenet}:DDN_REQ
  {hwmon_name}:{tilenet}:NDN_REQ
  {hwmon_name}:{trio}:TDMA_DATA_BEAT
  {hwmon_name}:{trio}:TDMA_PBUF_MAC_AF
  {hwmon_name}:{trio}:TDMA_RT_AF
  {hwmon_name}:{trio}:TPIO_DATA_BEAT
  {hwmon_name}:{triogen}:TX_DAT_AF
  {hwmon_name}:{triogen}:TX_DAT_AF

• BlueField-3 hwmon counters:

  {hwmon_name}:{llt}:GDC_BANK0_RD_REQ
  {hwmon_name}:{llt}:GDC_BANK1_RD_REQ
  {hwmon_name}:{llt}:GDC_BANK0_WR_REQ
  {hwmon_name}:{llt}:GDC_BANK1_WR_REQ
  {hwmon_name}:{llt_miss}:GDC_MISS_MACHINE_RD_REQ
  {hwmon_name}:{llt_miss}:GDC_MISS_MACHINE_WR_REQ
  {hwmon_name}:{mss}:SKYLIB_DDN_TX_FLITS
  {hwmon_name}:{mss}:SKYLIB_DDN_RX_FLITS

• BlueField-3 bf_ptm counters:

  bf:ptm:active_power_profile
  bf:ptm:atx_power_available
  bf:ptm:core_temp
  bf:ptm:ddr_temp
  bf:ptm:error_state
  bf:ptm:power_envelope
  bf:ptm:power_throttling_event_count
  bf:ptm:power_throttling_state
  bf:ptm:thermal_throttling_event_count
  bf:ptm:thermal_throttling_state
  bf:ptm:throttling_state
  bf:ptm:total_power
  bf:ptm:vr0_power
  bf:ptm:vr1_power

Ethtool counters is the generated list of counters which corresponds to Ethtool utility. Counters are generated on a per-device basis. See this community post for more information on mlx5 ethtool counters.

fluent_aggr listens on a port for Fluent Bit Forward protocol input connections. Received data can be streamed via a Fluent Bit exporter.

fluent-aggr-port=42442 

prometheus_aggr polls data from a list of Prometheus endpoints.

Each endpoint is listed in the following format:

prometheus_aggr_endpoint.{N}={host_name},{host_port_url},{poll_inteval_msec}

Aggregated data can be exported via a Prometheus aggregator exporter endpoint.

ifconfig collects network interface data. To enable, set:

enable-provider=ifconfig

prometheus-fset-indexes=name

name
rx_packets
tx_packets
rx_bytes
tx_bytes
rx_errors
tx_errors
rx_dropped
tx_dropped
multicast
collisions
rx_length_errors
rx_over_errors
rx_crc_errors
rx_frame_errors
rx_fifo_errors
rx_missed_errors
tx_aborted_errors
tx_carrier_errors
tx_fifo_errors
tx_heartbeat_errors
tx_window_errors
rx_compressed
tx_compressed
rx_nohandler

hcaperf collects HCA performance data. Since it requires access to an RDMA device, it must use remote collection on the DPU. On the host, the user runs the container in privileged mode and RDMA device mount.

The counter list is device dependent.

The nvidia-smi provider collects GPU and GPU process information provided by the system management interface.

This provider is supported only on x86_64 hosts with installed GPUs. All GPU cards supported by nvidia-smi are supported by this provider.

The counter list is GPU dependent. Additionally, per-process information is collected for the first 20 (by default) nvidia_smi_max_processes processes.

Counters can be either collected as string data "as is" in nvidia-smi or converted to numbers when nvsmi_with_numeric_fields is set.

enable-provider=nvidia-smi

# Optional parameters:
#nvidia_smi_max_processes=20
#nvsmi_with_numeric_fields=1 

The dcgm provider collects GPU information provided by the NVIDIA data center GPU manager (DCGM) API.

This provider is supported only on x86_64 hosts with installed GPUs, and requires running the nv-hostengine service (refer to DCGM documentation for details).

enable-provider=dcgm

dcgm_events_enable_common_fields=1
#dcgm_events_enable_prof_fields=0
#dcgm_events_enable_ecc_fields=0
#dcgm_events_enable_nvlink_fields=0
#dcgm_events_enable_nvswitch_fields=0
#dcgm_events_enable_vgpu_fields=0

The data writer is disabled by default to save space on BlueField. Steps for activating data write during debug can be found under section Enabling Data Write.

tree /opt/mellanox/doca/services/telemetry/data/
/opt/mellanox/doca/services/telemetry/data/
├── {year}
│   └── {mmdd}
│        └── {hash}
│             ├── {source_id}
│             │   └── {source_tag}{timestamp}.bin
│             └── {another_source_id}
│                  └── {another_source_tag}{timestamp}.bin
└── schema
    └── schema_{MD5_digest}.json

crictl exec -it <Container ID> /opt/mellanox/collectx/bin/clx_read -s /data/schema /data/path/to/datafile.bin

{
    "timestamp": 1634815738799728,
    "event_number": 0,
    "iter_num": 0,
    "string_number": 0,
    "example_string": "example_str_1"
}
{
    "timestamp": 1634815738799768,
    "event_number": 1,
    "iter_num": 0,
    "string_number": 1,
    "example_string": "example_str_2"
}
…

The Prometheus endpoint keeps the most recent data to be pulled by the Prometheus server and is enabled by default.

curl -s http://0.0.0.0:9100/metrics

counter_name {list of label fields} counter_value timestamp

Prometheus is configured as a part of dts_config.ini.

prometheus=http://0.0.0.0:9100

# Comma-separated counter set description for Prometheus indexing:
#prometheus-indexes=idx1,idx2

# Comma-separated fieldset description for prometheus indexing
#prometheus-fset-indexes=idx1,idx2 

prometheus-fset-indexes=device_name

If the fset index is not set, the data from p1 overwrites p0's data.

#prometheus-ignore-names=counter_name1,counter_name_2 

For quick filtering of data by tag, the Prometheus exporter supports being provided with a comma-separated list of data source tags to be ignored.

prometheus-ignore-tags=FI_metrics

/opt/mellanox/doca/services/telemetry/config/fluent_bit_configs

Fluent Bit allows exporting data via "Forward" protocol which connects to the Fluent Bit/FluentD instance on customer side.

NetFlow exporter must be used when data is collected as NetFlow packets from the telemetry client applications. In this case, DOCA Telemetry NetFlow API sends NetFlow data packages to DTS via IPC. DTS uses NetFlow exporter to send data to the NetFlow collector (3rd party service).

To enable NetFlow exporter, set netflow-collector-ip and netflow-collector-port in dts_config.ini. netflow-collector-ip could be set either to IP or an address.

For additional information, refer to the dts_config.ini file.

Please download Prometheus for your platform.

Prometheus is configured via command-line flags and a configuration file, prometheus.yml.

Please download and install Grafana for your platform.