RoCE

In NetQ, use either the UI or CLI to monitor RoCE in your network.

The syntax for RoCE commands is:

netq [<hostname>] show roce-counters [<text-port>] tx | rx [roce | general] [around <text-time>] [json]
netq [<hostname>] show roce-config [<text-port>] [around <text-time>] [json]
netq [<hostname>] show roce-counters pool [json]
netq [<hostname>] show events [message_type tca_roce]
netq [<hostname>] show events [message_type roceconfig]

View the RoCE Configuration

To view the RoCE configuration, run netq show roce-config.

show roce-config

View RoCE Counters

You can view the following RoCE counters for a given switch through the UI or CLI:

  • Rx and Tx counters
  • General, CNP, and RoCE-specific counters
  • Counter pools
  • Port-specific counters

View Rx Counters

  1. To view Rx counters, open the large switch card, then click the RoCE icon ().
  2. Expand the card to the largest size, then select RoCE Counters from the side menu:
full-size switch card with RoCe Counters tab selected

To view general and CNP Rx counters, run netq show roce-counters rx general:

cumulus@switch:~$ netq show roce-counters rx general

Matching roce records:
Hostname          Interface            PG packets           PG bytes             no buffer discard    buffer usage         buffer max usage     PG usage             PG max usage
----------------- -------------------- -------------------- -------------------- -------------------- -------------------- -------------------- -------------------- --------------------
switch            swp1s1               1627273              152582910            0                    0                    1                    0                    1
switch            swp1s2               1627273              152582910            0                    0                    1                    0                    1
switch            swp63s1              1618361              160178796            0                    0                    2                    0                    2
switch            swp1s0               1627273              152582910            0                    0                    1                    0                    1
switch            swp63s3              1618361              160178796            0                    0                    2                    0                    2
switch            swp1s3               1627273              152582910            0                    0                    1                    0                    1
switch            swp63s0              1094532              120228456            0                    0                    1                    0                    1
switch            swp63s2              1618361              160178796            0                    0                    2                    0                    2

To view RoCE-specific Rx counters, run netq show roce-counters rx roce:

cumulus@switch:~$ netq show roce-counters rx roce

Matching roce records:
Hostname          Interface       PG packets   PG bytes     no buffer discard  PFC pause packets  PFC pause duration buffer usage buffer max usage   PG usage     PG max usage
----------------- --------------- ------------ ------------ ------------------ ------------------ ------------------ ------------ ------------------ ------------ ---------------
switch            swp1s1          0            0            0                  0                  0                  0            0                  0            0
switch            swp1s2          0            0            0                  0                  0                  0            0                  0            0
switch            swp63s1         0            0            0                  0                  0                  0            0                  0            0
switch            swp1s0          0            0            0                  0                  0                  0            0                  0            0
switch            swp63s3         0            0            0                  0                  0                  0            0                  0            0
switch            swp1s3          0            0            0                  0                  0                  0            0                  0            0
switch            swp63s0         0            0            0                  0                  0                  0            0                  0            0
switch            swp63s2         0            0            0                  0                  0                  0            0                  0            0

View Tx Counters

  1. To view Tx counters, open the large switch card, then click the RoCE icon ().
  2. Expand the card to the largest size, then select RoCE Counters from the side menu.
  3. Select Tx from the toggle above the table:
switch card with table displaying Rx and Tx RoCe toggle

To view general and CNP Tx counters, run netq show roce-counters tx general:

cumulus@switch:~$ netq show roce-counters tx general 

Matching roce records:
Hostname          Interface       ECN marked packets   TC packets   TC bytes     unicast no buffer discard buffer usage buffer max usage   TC usage     TC max usage
----------------- --------------- -------------------- ------------ ------------ ------------------------- ------------ ------------------ ------------ ------------
switch            swp1s1          0                    0            0            0                         0            0                  0            0
switch            swp1s2          0                    0            0            0                         0            0                  0            0
switch            swp63s1         0                    0            0            0                         0            0                  0            0
switch            swp1s0          0                    0            0            0                         0            0                  0            0
switch            swp63s3         0                    0            0            0                         0            0                  0            0
switch            swp1s3          0                    0            0            0                         0            0                  0            0
switch            swp63s0         0                    0            0            0                         0            0                  0            0
switch            swp63s2         0                    0            0            0                         0            0                  0            0
cumulus@switch      :~$ 

To view RoCE-specific Tx counters, run netq show roce-counters tx roce:

cumulus@switch:~$ netq show roce-counters tx roce 

Matching roce records:
Hostname          Interface       TC packets TC bytes   unicast no buffer discard PFC pause packets  PFC pause duration buffer usage buffer max usage   TC usage   TC max usage
----------------- --------------- ---------- ---------- ------------------------- ------------------ ------------------ ------------ ------------------ ---------- ---------------
switch            swp1s1          0          0          0                         0                  0                  0            0                  0          0
switch            swp1s2          0          0          0                         0                  0                  0            0                  0          0
switch            swp63s1         0          0          0                         0                  0                  0            0                  0          0
switch            swp1s0          0          0          0                         0                  0                  0            0                  0          0
switch            swp63s3         0          0          0                         0                  0                  0            0                  0          0
switch            swp1s3          0          0          0                         0                  0                  0            0                  0          0
switch            swp63s0         0          0          0                         0                  0                  0            0                  0          0
switch            swp63s2         0          0          0                         0                  0                  0            0                  0          0

View RoCE Counter Pools

  1. To view RoCE counter pools, open the large switch card, then click the RoCE icon ().
  2. Switch to the full-screen card, then click RoCE Counters. Look for these columns: Lossy Default Ingress Size, RoCE Reserved Ingress Size, Lossy Default Egress Size, and RoCE Reserved Egress Size.

To view the RoCE counter pools, run netq show roce-counters pool:

cumulus@switch:~$ netq show roce-counters pool 

Matching roce records:
Hostname          Lossy Default Ingress Size     Roce Reserved Ingress Size     Lossy Default Egress Size      Roce Reserved Egress Size
----------------- ------------------------------ ------------------------------ ------------------------------ ------------------------------
switch            104823                         104823                         104823                         104823

View Counters for a Specific Switch Port

  1. Open the large switch card, then click the RoCE icon ().
  2. Select a port from the list on the left:
switch card displaying list of ports

To view counters for a specific switch port, include the switch name with the command:

cumulus@switch:~$ netq show roce-counters swp1s1 rx general 

Matching roce records:
Hostname          Interface            PG packets           PG bytes             no buffer discard    buffer usage         buffer max usage     PG usage             PG max usage
----------------- -------------------- -------------------- -------------------- -------------------- -------------------- -------------------- -------------------- --------------------
switch            swp1s1               1643392              154094520            0                    0                    1                    0                    1

View Results from a Time in the Past

To view counters for a time period in the past:

  1. Open the large switch card, then click the RoCE icon ().
  2. Click in the header and select a different time period.

Use around with any RoCE-related command to view counters from a previous time period:

cumulus@switch:~$ netq show roce-counters swp1s1 rx general around 1h

Matching roce records:
Hostname          Interface            PG packets           PG bytes             no buffer discard    buffer usage         buffer max usage     PG usage             PG max usage
----------------- -------------------- -------------------- -------------------- -------------------- -------------------- -------------------- -------------------- --------------------
switch            swp1s1               661                  61856                0                    0                    1                    0                    1

Disable RoCE Monitoring

To disable RoCE monitoring:

  1. Edit /etc/netq/commands/cl4-netq-commands.yml and comment out the following lines:

     cumulus@netq-ts:~$ sudo nano /etc/netq/commands/cl4-netq-commands.yml
    
     #- period: "60"
     #  key: "roce"
     #  isactive: true
     #  command: "/usr/lib/cumulus/mlxcmd --json roce counters"
     #  parser: "local"
    
  2. Delete the /var/run/netq/netq_commands.yml file:

     cumulus@netq-ts:~$ sudo rm /var/run/netq/netq_commands.yml
    
  3. Restart the NetQ Agent:

    cumulus@netq-ts:~$ netq config agent restart