Performance Monitoring Counters

The performance modules in NVIDIA® BlueField® are present in several hardware blocks and each block has a certain set of supported events.

The mlx_pmc driver provides access to all of these performance modules through a sysfs interface. The driver creates a directory under /sys/class/hwmon under which each of the blocks explained above has a subdirectory. Please note that all directories under /sys/class/hwmon are named as "hwmon<N>" where N is the hwmon device number corresponding to the device. This is assigned by Linux and could change with the addition of more devices to the hwmon class. Each hwmon directory has a "name" node which can be used to identify the correct device. In this case, reading the "name" file should return "bfperf".

The hardware blocks that include performance modules are:

• Tile (block containing 2 cores and a shared L2 cache) has 2 sets of counters, one set for HNF and HNF_NET events. These are present as "tile" and "tilenet" directories in the sysfs interface of the driver.

• TRIO (PCIe root complex) has 3 sets of counters, one each for TRIO, SMGEN and PCIE TLR events. The sysfs directories for these are called "trio", "triogen" and "pcie" respectively.

• MSS (memory sub-system containing the memory controller and L3 cache)

• GIC and SMMU with one set of counters each for the SMGEN events. These are simply labelled "gic" and "smmu" respectively.

The number of Tile, TRIO and MSS blocks depends on the system. There is a maximum of 8 Tile, 3 TRIO and 2 MSS blocks in BlueField, and this is added as a suffix to the sysfs directory names. For example, this is a list of directories present in a BlueField-2 system:

            

            
ubuntu@dpu:/$ ls /sys/class/hwmon/hwmon0/
device l3cachehalf0 pcie0 smmu0 tile1 tilenet0 tilenet3 triogen0
ecc l3cachehalf1 pcie1 subsystem tile2 tilenet1 trio0 triogen1
gic0 name power  tile0 tile3 tilenet2 trio1 uevent
        

The PCIe TLR statistics for each trio are under the "pcie" block.

The performance data of the BlueField hardware is collected using two mechanisms:

1. Programming hardware counters to monitor specific events

2. Reading registers that hold performance/event statistics

All blocks except "ecc" and "pcie" use the mechanism 1.

Using Hardware Counters

For blocks that use hardware counters to collect data, each counter present in the block is represented by "event<N>" and "counter<N>" sysfs files.

For example:

            

            
ubuntu@dpu:/$ ls /sys/class/hwmon/hwmon0/tile0/
counter0 counter1 counter2 counter3 event0 event1 event2 event3 event_list
        

An event<N> and counter<N> pair can be used to program and monitor events. The "event_list" sysfs file displays the list of events supported by that block along with the hexadecimal value corresponding to each event.

Use the echo command to write the event number to the event<N> file, and use the cat command to read the counter value from the corresponding counter (counter<N>).

The counters are enabled individually once the event number is written to the corresponding event file. However, the L3 cache performance counters cannot be enabled or disabled individually and can only be triggered or stopped all at the same time.

So in the example provided, all 4 event files may be programmed with the necessary event numbers and then the "enable" file may be used to start the counters. Writing 0 to the enable file stops the counters while 1 starts them.

Reading Registers

For "ecc" and "pcie" blocks, the counters cannot be started or stopped by the user, instead the statistics are automatically collected by HW and stored in registers. These register names are exposed within the directory and can be read by the user at any time.

SMGEN Performance Module

Tile HNF Performance Module

TRIO Performance Module

L3 Cache Performance Module

PCIe TLR Statistics

Tile HNFNET Performance Module

To program a counter to monitor one of the events from the event list, the event name or number needs to be written to the corresponding event file.

Let us call the /sys/class/hwmon/hwmon<N> folder corresponding to this driver as "BFPERF_DIR".

For example, to monitor the event HNF_REQUESTS (0x45) on "tile2" using counter 3:

            

            
$ echo 0x45 > <BFPERF_DIR>/tile2/event3
        

Or:

            

            
$ echo HNF_REQUESTS > <BFPERF_DIR>/tile2/event3
        

Once this is done, counter3 resets the counter and starts monitoring the number of HNF_REQUESTS.

To read the counter value, just run:

            

            
$ cat <BFPERF_DIR>/tile2/counter3
        

To see what event is currently being monitored by a counter, just read the corresponding event file to get the event name and number.

            

            
$ cat <BFPERF_DIR>/tile2/event3
        

In this case, reading the event3 file returns "0x45: HNF_REQUESTS".

To clear the counter, write 0 to the counter file.

            

            
$ echo 0 > <BFPERF_DIR>/tile2/counter3
        

This resets the accumulator and the counter continues monitoring the same event that has previously been programmed, but starts the count from 0 again. Writing non-zero values to the counter files is not allowed.

To stop monitoring an event, write "0xff" to the corresponding event file.

This is slightly different for the l3cache blocks due to the restriction that all counters can only be enabled, disabled, or reset together. So once the event is written to the event file, the counters will have to be enabled to start monitoring their respective events by writing "1" to the "enable" file. Writing "0" to this file will stop all the counters. The most reliable way to get accurate counter values would be by disabling the counters after a certain time period and then proceeding to read the counter values.

For blocks that have performance statistics registers (mechanism 2), all of these statistics are directly made available to be read or reset.

For example, to read the number of incoming posted packets to TRIO2:

            

            
$ cat <BFPERF_DIR>/pcie2/IN_P_PKT_CNT
        

The count can be reset to 0 by writing 0 to the same file. Again, non-zero writes to these files are not allowed.

Simple Network Management Protocol is a UDP-based protocol which allows remote monitoring and configuration of certain system statistics and parameters. SNMP is used by agents and managers to send and retrieve information. An agent is a software process that responds to SNMP queries to provide status and statistics about a network node. A manager is an application that manages SNMP agents on a network by issuing requests, getting responses, and listening for and processing agent-issued traps.

System statistics and parameters are collectively known in SNMP terminology as variables. Each variable is uniquely identified by an object identifier (OID). A variable may be either scalar or columnar. A scalar variable is a singleton. A columnar variable is part of a two-dimensional collection of variables known as a table. In a table a row represents one record or instance of an object.

SNMP uses a collection of text files, called MIB (Management Information Base), to describe variables and traps that a server may offer and maps OIDs to human-readable names. All MIBs fit into a universal OID hierarchy. There is a point in the tree under which all vendor MIBs are rooted: .1.3.6.1.4.1 (.iso.org.dod.internet.private.enterprises). Each vendor should have an enterprise number registered with IANA (Internet Assigned Numbers Authority). For NVIDIA, this number is 33049.

Performance Monitoring via SNMP Subagent

Depending on a counter's type, there are two types of tables to display the hardware counters:

1. A 4-column (block ID, counter ID, event, and counter value) table. The number of rows for each table is determined according to the equation <number_of_blocks>*<number_of_counters>.
   For example, the MSS module has 2 blocks and 4 hardware counters, so the total amount of rows in table should be 2*4 = 8 rows.
   To program a counter, the block ID, counter ID and event name or number must be set using SNMP SET commands. For example:

   Copy

   Copied!

               
   
               
   $ snmpset -v2c -c mlnx 192.168.100.2 MLNX-PMC-MIB::mlnxPmcMssBlockIdSet.0 u 0
   $ snmpset -v2c -c mlnx 192.168.100.2 MLNX-PMC-MIB::mlnxPmcMssCounterIdSet.0 u 3
   $ snmpset -v2c -c mlnx 192.168.100.2 MLNX-PMC-MIB::mlnxPmcMssEventIdSet.0 s 0xc0
           

   Or using a single SNMP SET format:

   Copy

   Copied!

               
   
               
   $ snmpset -v2c -c mlnx 192.168.100.2 MLNX-PMC-MIB::mlnxPmcMssBlockIdSet.0 u 0 MLNX-PMC-MIB::mlnxPmcMssCounterIdSet.0 u 0 MLNX-PMC-MIB::mlnxPmcMssEventIdSet.0 s 0xc0
           

   This command starts monitoring the event 0xc0 (RXREQ_MSS) on counter /sys/class/hwmon/hwmon0/mss0/counter3.

   To stop monitoring, set 0x0 to the corresponding event. For example:

   Copy

   Copied!

               
   
               
   $ snmpset -v2c -c mlnx 192.168.100.2 MLNX-PMC-MIB::mlnxPmcMssBlockIdSet.0 u 0
   $ snmpset -v2c -c mlnx 192.168.100.2 MLNX-PMC-MIB::mlnxPmcMssCounterIdSet.0 u 3
   $ snmpset -v2c -c mlnx 192.168.100.2 MLNX-PMC-MIB::mlnxPmcMssEventIdSet.0 s 0x0
           

   The following is an output example for the MSS module:

   Copy

   Copied!

               
   
               
   $ snmptable -v2c -c mlnx 192.168.100.2 MLNX-PMC-MIB::mlnxPmcMssTable
   SNMP table: MLNX-PMC-MIB::mlnxPmcMssTable
   mlnxPmcMssBlockId mlnxPmcMssCounterId mlnxPmcMssEvent mlnxPmcMssCounters
                   0                   0           0xc3          9485515546
                   0                   1           0xc2          3718241246
                   0                   2           0xc1          3718118806
                   0                   3           0xc0          6601319766
                   1                   0           0xc0          5675145425
                   1                   1           0xc1          3001880054
                   1                   2           0xc2          3001805288
                   1                   3           0xc3          8347484949
           

2. A 3-column (block ID, event, counter value) table.
   The number of rows for each table is determined according to the equation <number_of_blocks>*<number_of_events>. For example, PCIe module has 3 blocks and 15 events, so the total amount of rows in the table should be 3*15 = 45 rows.
   The following is an output example for PCIe module:

   Collapse Source

   Copy

   Copied!

               
   
               
   $ snmptable -v2c -c mlnx 192.168.100.2 MLNX-PMC-MIB::mlnxPmcPcieTable
   SNMP table: MLNX-PMC-MIB::mlnxPmcPcieTable
   mlnxPmcPcieBlockId mlnxPmcPcieEvent mlnxPmcPcieCounters
                    0    IN_P_BYTE_CNT                   0
                    0   OUT_C_BYTE_CNT                   0
                    0    IN_NP_PKT_CNT                   0
                    0   OUT_P_BYTE_CNT              125312
                    0  OUT_NP_BYTE_CNT                   0
                    0     IN_C_PKT_CNT               17260
                    0   IN_NP_BYTE_CNT                   0
                    0     IN_P_PKT_CNT                   0
                    0    OUT_C_PKT_CNT                   0
                    0   OUT_NP_PKT_CNT               17260
                    0    OUT_P_PKT_CNT                1958
                    0    IN_C_BYTE_CNT                   0
                    1    IN_P_BYTE_CNT                   0
                    1   OUT_C_BYTE_CNT                   0
                    1    IN_NP_PKT_CNT                   0
                    1   OUT_P_BYTE_CNT                   0
                    1  OUT_NP_BYTE_CNT                   0
                    1     IN_C_PKT_CNT                   1
                    1   IN_NP_BYTE_CNT                   0
                    1     IN_P_PKT_CNT                   0
                    1    OUT_C_PKT_CNT                   0
                    1   OUT_NP_PKT_CNT                   1
                    1    OUT_P_PKT_CNT                   0
                    1    IN_C_BYTE_CNT                   0
                    2    IN_P_BYTE_CNT            95284696
                    2   OUT_C_BYTE_CNT           264050560
                    2    IN_NP_PKT_CNT             4026124
                    2   OUT_P_BYTE_CNT              116872
                    2  OUT_NP_BYTE_CNT           200243467
                    2     IN_C_PKT_CNT              366859
                    2   IN_NP_BYTE_CNT                   0
                    2     IN_P_PKT_CNT             1575547
                    2    OUT_C_PKT_CNT             4065099
                    2   OUT_NP_PKT_CNT              366859
                    2    OUT_P_PKT_CNT               28984
                    2    IN_C_BYTE_CNT             1362368