Configure and Monitor What Just Happened

The What Just Happened (WJH) feature, available on NVIDIA switches, streams detailed and contextual telemetry data for analysis. This provides real-time visibility into problems in the network, such as hardware packet drops due to buffer congestion, incorrect routing, and ACL or layer 1 problems. You must have Cumulus Linux 4.0.0 or later, SONiC 202012 or later, and NetQ 2.4.0 or later to take advantage of this feature.

For a list of supported WJH events, refer to the WJH Event Messages Reference.

When you combine WJH capabilities with NetQ, you have the ability to home in on losses, anywhere in the fabric, from a single management console. You can:

• View any current or historic drop information, including the reason for the drop
• Identify problematic flows or endpoints, and pin-point exactly where communication is failing in the network

cumulus@<hostname>:~$ sudo apt-get update
cumulus@<hostname>:~$ sudo apt-get install -y netq-agent
cumulus@<hostname>:~$ sudo netq config restart agent
cumulus@<hostname>:~$ sudo apt-get install -y netq-apps
cumulus@<hostname>:~$ sudo netq config restart cli

Configure the WJH Feature

WJH is enabled by default on NVIDIA switches and Cumulus Linux 4.0.0 requires no configuration; however, you must enable the NetQ Agent to collect the data.

To enable WJH in NetQ on any switch or server:

1. Configure the NetQ Agent on the NVIDIA switch.

   cumulus@switch:~$ sudo netq config add agent wjh
   

2. Restart the NetQ Agent to start collecting the WJH data.

   cumulus@switch:~$ sudo netq config restart agent
   

When you finish viewing the WJH metrics, you might want to stop the NetQ Agent from collecting WJH data to reduce network traffic. Use netq config del agent wjh followed by netq config restart agent to disable the WJH feature on the given switch.

Configure Latency and Congestion Thresholds

WJH latency and congestion metrics depend on threshold settings to trigger the events. WJH measures packet latency as the time spent inside a single system (switch). WJH measures congestion as a percentage of buffer occupancy on the switch. When specified, WJH triggers events when they cross high and/or low thresholds.

To configure these thresholds, run:

netq config add agent wjh-threshold (latency|congestion) <text-tc-list> <text-port-list> <text-th-hi> <text-th-lo>

You can specify multiple traffic classes and multiple ports by separating the classes or ports by a comma (no spaces).

This example creates latency thresholds for Class 3 traffic on port swp1 where the upper threshold is 10 and the lower threshold is 1.

cumulus@switch:~$ sudo netq config add agent wjh-threshold latency 3 swp1 10 1

This example creates congestion thresholds for Class 4 traffic on port swp1 where the upper threshold is 200 and the lower threshold is 10.

cumulus@switch:~$ sudo netq config add agent wjh-threshold congestion 4 swp1 200 10

Configure Filters

You can filter the WJH events at the NetQ Agent before the NetQ system processes it. You perform filtering on a drop-type basis. You can filter the drop type further by specifying one or more drop reasons or severity. Filter events by creating a NetQ Configuration profile in the NetQ UI or using the netq config add agent wjh-drop-filter command in the NetQ CLI.

For a complete list of drop types and reasons, refer to the WJH Event Messages Reference.

netq config add agent wjh-drop-filter drop-type <text-wjh-drop-type> [drop-reasons <text-wjh-drop-reasons>] [severity <text-drop-severity-list>]

cumulus@switch:~$ sudo netq config add agent wjh-drop-filter drop-type l1

cumulus@switch:~$ sudo netq config add agent wjh-drop-filter drop-type l1 drop-reasons BAD_SIGNAL_INTEGRITY

cumulus@switch:~$ sudo netq config add agent wjh-drop-filter drop-type router severity Warning

cumulus@netq-ts:~$ netq config add agent wjh-drop-filter drop-type router drop-reasons BLACKHOLE_ROUTE

cumulus@netq-ts:~$ netq config add agent wjh-drop-filter drop-type router drop-reasons SRC_IP_IS_IN_CLASS_E

View What Just Happened Metrics

You can view the WJH metrics from the NetQ UI or the NetQ CLI.

NetQ UI

1. Click (main menu).

2. Click What Just Happened under the Network column.

   This view displays events based on conditions detected in the data plane. Each drop category contains the most recent 1000 events from the last 24 hours.

3. By default the layer 1 drops are shown. Click one of the other drop categories to view those drops for all devices.

NetQ CLI

Run one of the following commands:

netq [<hostname>] show wjh-drop <text-drop-type> [ingress-port <text-ingress-port>] [severity <text-severity>] [reason <text-reason>] [src-ip <text-src-ip>] [dst-ip <text-dst-ip>] [proto <text-proto>] [src-port <text-src-port>] [dst-port <text-dst-port>] [src-mac <text-src-mac>] [dst-mac <text-dst-mac>] [egress-port <text-egress-port>] [traffic-class <text-traffic-class>] [rule-id-acl <text-rule-id-acl>] [between <text-time> and <text-endtime>] [around <text-time>] [json]
netq [<hostname>] show wjh-drop [ingress-port <text-ingress-port>] [severity <text-severity>] [details] [between <text-time> and <text-endtime>] [around <text-time>] [json]

Use the various options to restrict the output accordingly.

This example uses the first form of the command to show drops on switch leaf03 for the past week.

cumulus@switch:~$ netq leaf03 show wjh-drop between now and 7d
Matching wjh records:
Drop type          Aggregate Count
------------------ ------------------------------
L1                 560
Buffer             224
Router             144
L2                 0
ACL                0
Tunnel             0

This example uses the second form of the command to show drops on switch leaf03 for the past week including the drop reasons.

cumulus@switch:~$ netq leaf03 show wjh-drop details between now and 7d

Matching wjh records:
Drop type          Aggregate Count                Reason
------------------ ------------------------------ ---------------------------------------------
L1                 556                            None
Buffer             196                            WRED
Router             144                            Blackhole route
Buffer             14                             Packet Latency Threshold Crossed
Buffer             14                             Port TC Congestion Threshold
L1                 4                              Oper down

This example shows the drops seen at layer 2 across the network.

cumulus@mlx-2700-03:mgmt:~$ netq show wjh-drop l2
Matching wjh records:
Hostname          Ingress Port             Reason                                        Agg Count          Src Ip           Dst Ip           Proto  Src Port         Dst Port         Src Mac            Dst Mac            First Timestamp                Last Timestamp
----------------- ------------------------ --------------------------------------------- ------------------ ---------------- ---------------- ------ ---------------- ---------------- ------------------ ------------------ ------------------------------ ----------------------------
mlx-2700-03       swp1s2                   Port loopback filter                          10                 27.0.0.19        27.0.0.22        0      0                0                00:02:00:00:00:73  0c:ff:ff:ff:ff:ff  Mon Dec 16 11:54:15 2019       Mon Dec 16 11:54:15 2019
mlx-2700-03       swp1s2                   Source MAC equals destination MAC             10                 27.0.0.19        27.0.0.22        0      0                0                00:02:00:00:00:73  00:02:00:00:00:73  Mon Dec 16 11:53:17 2019       Mon Dec 16 11:53:17 2019
mlx-2700-03       swp1s2                   Source MAC equals destination MAC             10                 0.0.0.0          0.0.0.0          0      0                0                00:02:00:00:00:73  00:02:00:00:00:73  Mon Dec 16 11:40:44 2019       Mon Dec 16 11:40:44 2019

The following two examples include the severity of a drop event (error, warning or notice) for ACLs and routers.

cumulus@switch:~$ netq show wjh-drop acl
Matching wjh records:
Hostname          Ingress Port             Reason                                        Severity         Agg Count          Src Ip           Dst Ip           Proto  Src Port         Dst Port         Src Mac            Dst Mac            Acl Rule Id            Acl Bind Point               Acl Name         Acl Rule         First Timestamp                Last Timestamp
----------------- ------------------------ --------------------------------------------- ---------------- ------------------ ---------------- ---------------- ------ ---------------- ---------------- ------------------ ------------------ ---------------------- ---------------------------- ---------------- ---------------- ------------------------------ ----------------------------
leaf01            swp2                     Ingress router ACL                            Error            49                 55.0.0.1         55.0.0.2         17     8492             21423            00:32:10:45:76:89  00:ab:05:d4:1b:13  0x0                    0                                                              Tue Oct  6 15:29:13 2020       Tue Oct  6 15:29:39 2020

cumulus@switch:~$ netq show wjh-drop router
Matching wjh records:
Hostname          Ingress Port             Reason                                        Severity         Agg Count          Src Ip           Dst Ip           Proto  Src Port         Dst Port         Src Mac            Dst Mac            First Timestamp                Last Timestamp
----------------- ------------------------ --------------------------------------------- ---------------- ------------------ ---------------- ---------------- ------ ---------------- ---------------- ------------------ ------------------ ------------------------------ ----------------------------
leaf01            swp1                     Blackhole route                               Notice           36                 46.0.1.2         47.0.2.3         6      1235             43523            00:01:02:03:04:05  00:06:07:08:09:0a  Tue Oct  6 15:29:13 2020       Tue Oct  6 15:29:47 2020

Collect WJH Data Using gNMI

You can use gNMI, the gRPC network management interface, to collect What Just Happened data from the NetQ Agent and export it to your own gNMI client.

The client should use the following YANG model as a reference:

module WjhDropAggregate {
    // Entrypoint container interfaces.
    // Path --> interfaces/interface[name]/wjh/aggregate/[acl, l2, router, tunnel, buffer]/reasons/reason[id, severity]/drop[]
    // Path for L1 --> interfaces/interface[name]/wjh/aggregate/l1/drop[]

    namespace "http://nvidia.com/yang/what-just-happened-config";
    prefix "wjh_drop_aggregate";

    container interfaces {
        config false;
        list interface {
            key "name";
            leaf name {
                type string;
                mandatory "true";
                description "Interface name";
            }
            uses wjh_aggregate;
        }
    }

    grouping wjh_aggregate {
    description "Top-level grouping for What-just happened data.";
        container wjh {
            container aggregate {
                container l1 {
                    uses l1-drops;
                }
                container l2 {
                    list reason {
                        key "id severity";
                        uses reason-key;
                        uses l2-drops;
                    }
                }
                container router {
                    list reason {
                        key "id severity";
                        uses reason-key;
                        uses router-drops;
                    }
                }
                container tunnel {
                    list reason {
                        key "id severity";
                        uses reason-key;
                        uses tunnel-drops;
                    }
                }
                container acl {
                    list reason {
                        key "id severity";
                        uses reason-key;
                        uses acl-drops;
                    }
                }
                container buffer {
                    list reason {
                        key "id severity";
                        uses reason-key;
                        uses buffer-drops;
                    }
                }
            }
        }
    }

    grouping reason-key {
        leaf id {
            type uint32;
            mandatory "true";
            description "reason ID";
        }
        leaf severity {
            type string;
            mandatory "true";
            description "Severity";
        }
    }

    grouping reason_info {
        leaf reason {
                type string;
                mandatory "true";
                description "Reason name";
        }
        leaf drop_type {
            type string;
            mandatory "true";
            description "reason drop type";
        }
        leaf ingress_port {
            type string;
            mandatory "true";
            description "Ingress port name";
        }
        leaf ingress_lag {
            type string;
            description "Ingress LAG name";
        }
        leaf egress_port {
            type string;
            description "Egress port name";
        }
        leaf agg_count {
            type uint64;
            description "Aggregation count";
        }
        leaf severity {
            type string;
            description "Severity";
        }
        leaf first_timestamp {
            type uint64;
            description "First timestamp";
        }
        leaf end_timestamp {
            type uint64;
            description "End timestamp";
        }
    }

    grouping packet_info {
        leaf smac {
            type string;
            description "Source MAC";
        }
        leaf dmac {
            type string;
            description "Destination MAC";
        }
        leaf sip {
            type string;
            description "Source IP";
        }
        leaf dip {
            type string;
            description "Destination IP";
        }
        leaf proto {
            type uint32;
            description "Protocol";
        }
        leaf sport {
            type uint32;
            description "Source port";
        }
        leaf dport {
            type uint32;
            description "Destination port";
        }
    }

    grouping l1-drops {
        description "What-just happened drops.";
        list drop {
            leaf ingress_port {
                type string;
                description "Ingress port";
            }
            leaf is_port_up {
                type boolean;
                description "Is port up";
            }
            leaf port_down_reason {
                type string;
                description "Port down reason";
            }
            leaf description {
                type string;
                description "Description";
            }
            leaf state_change_count {
                type uint64;
                description "State change count";
            }
            leaf symbol_error_count {
                type uint64;
                description "Symbol error count";
            }
            leaf crc_error_count {
                type uint64;
                description "CRC error count";
            }
            leaf first_timestamp {
                type uint64;
                description "First timestamp";
            }
            leaf end_timestamp {
                type uint64;
                description "End timestamp";
            }
            leaf timestamp {
                type uint64;
                description "Timestamp";
            }
        }
    }
    grouping l2-drops {
        description "What-just happened drops.";
        list drop {
            uses reason_info;
            uses packet_info;
        }
    }

    grouping router-drops {
        description "What-just happened drops.";
        list drop {
            uses reason_info;
            uses packet_info;
        }
    }

    grouping tunnel-drops {
        description "What-just happened drops.";
        list drop {
            uses reason_info;
            uses packet_info;
        }
    }

    grouping acl-drops {
        description "What-just happened drops.";
        list drop {
            uses reason_info;
            uses packet_info;
            leaf acl_rule_id {
                type uint64;
                description "ACL rule ID";
            }
            leaf acl_bind_point {
                type uint32;
                description "ACL bind point";
            }
            leaf acl_name {
                type string;
                description "ACL name";
            }
            leaf acl_rule {
                type string;
                description "ACL rule";
            }
        }
    }

    grouping buffer-drops {
        description "What-just happened drops.";
        list drop {
            uses reason_info;
            uses packet_info;
            leaf traffic_class {
                type uint32;
                description "Traffic Class";
            }
            leaf original_occupancy {
                type uint32;
                description "Original occupancy";
            }
            leaf original_latency {
                type uint64;
                description "Original latency";
            }
        }
    }
}

Supported Features

In this release, the gNMI agent supports capability and stream subscribe requests for WJH events.

Configure the gNMI Agent

To configure the gNMI agent, you need to enable it on every switch you want to use with gNMI. Optionally, you can update these default settings:

• Log level: The default log level is info. You can change the default setting to debug, warning, or error.
• Default gNMI port: The gNMI agent listens over port 9339 by default. You can change this setting in case you use that port in another application.

The /etc/netq/netq.yml file stores the configuration.

To configure the gNMI agent on a switch:

1. Enable the gNMI agent:

   cumulus@switch:~$ netq config add agent gnmi-enable true
   

2. If you want to change the default log level to something other than info (choose from debug, warning, or error), run:

   cumulus@switch:~$ netq config add agent gnmi-log-level [debug|warning|error]
   

3. If you want to change the default port over which the gNMI agent listens, run:

   cumulus@switch:~$ netq config add agent gnmi-port <gnmi_port>
   

4. Restart the NetQ Agent:

   cumulus@switch:~$ netq config restart agent
   

gNMI Client Requests

You use your gNMI client on a host server to request capabilities and data the agent is subscribed to.

To make a subscribe request, run:

[root@host ~]# /path/to/your/gnmi_client/gnmi_client -gnmi_address 10.209.37.84 -request subscribe
2021/05/06 18:33:10.142160 host gnmi_client[24847]: INFO: 10.209.37.84:9339: ready for streaming
2021/05/06 18:33:10.220814 host gnmi_client[24847]: INFO: sync response received: sync_response:true
2021/05/06 18:33:16.813000 host gnmi_client[24847]: INFO: update received [interfaces interface swp8 wjh aggregate l2 reason 209 error]: {"Drop":[{"AggCount":1,"Dip":"1.2.0.0","Dmac":"00:bb:cc:11:22:32","Dport":0,"DropType":"L2","EgressPort":"","EndTimestamp":1620326044,"FirstTimestamp":1620326044,"IngressLag":"","IngressPort":"swp8","Proto":0,"Reason":"Source MAC is multicast","Severity":"Error","Sip":"10.213.1.242","Smac":"ff:ff:ff:ff:ff:ff","Sport":0}],"Id":209,"Severity":"Error"}
2021/05/06 18:33:16.815068 host gnmi_client[24847]: INFO: update received [interfaces interface swp8 wjh aggregate l2 reason 209 error]: {"Drop":[{"AggCount":1,"Dip":"1.2.0.0","Dmac":"00:bb:cc:11:22:32","Dport":0,"DropType":"L2","EgressPort":"","EndTimestamp":1620326044,"FirstTimestamp":1620326044,"IngressLag":"","IngressPort":"swp8","Proto":0,"Reason":"Source MAC is multicast","Severity":"Error","Sip":"10.213.2.242","Smac":"ff:ff:ff:ff:ff:ff","Sport":0}],"Id":209,"Severity":"Error"}
2021/05/06 18:33:16.818896 host gnmi_client[24847]: INFO: update received [interfaces interface swp8 wjh aggregate l2 reason 209 error]: {"Drop":[{"AggCount":1,"Dip":"1.2.0.0","Dmac":"00:bb:cc:11:22:32","Dport":0,"DropType":"L2","EgressPort":"","EndTimestamp":1620326044,"FirstTimestamp":1620326044,"IngressLag":"","IngressPort":"swp8","Proto":0,"Reason":"Source MAC is multicast","Severity":"Error","Sip":"10.213.3.242","Smac":"ff:ff:ff:ff:ff:ff","Sport":0}],"Id":209,"Severity":"Error"}
2021/05/06 18:33:16.823091 host gnmi_client[24847]: INFO: update received [interfaces interface swp8 wjh aggregate l2 reason 209 error]: {"Drop":[{"AggCount":1,"Dip":"1.2.0.0","Dmac":"00:bb:cc:11:22:32","Dport":0,"DropType":"L2","EgressPort":"","EndTimestamp":1620326044,"FirstTimestamp":1620326044,"IngressLag":"","IngressPort":"swp8","Proto":0,"Reason":"Source MAC is multicast","Severity":"Error","Sip":"10.213.4.242","Smac":"ff:ff:ff:ff:ff:ff","Sport":0}],"Id":209,"Severity":"Error"}
...

To request the capabilities, run:

[root@host ~]# /path/to/your/gnmi_client/gnmi_client/gnmi_client -gnmi_address 10.209.37.84 -request capabilities
2021/05/06 18:36:31.285648 host gnmi_client[25023]: INFO: 10.209.37.84:9339: ready for streaming
2021/05/06 18:36:31.355944 host gnmi_client[25023]: INFO: capability response: supported_models:{name:"WjhDropAggregate"  organization:"NVIDIA"  version:"0.1"}  supported_encodings:JSON  supported_encodings:JSON_IETF

Use Only the gNMI Agent

It is possible (although it is not recommended) to collect data using only the gNMI agent, and not the NetQ Agent. However, this sends data only to gNMI and not to NetQ.

To use only gNMI for data collection, disable the NetQ Agent, which is always enabled by default:

cumulus@switch:~$ netq config add agent opta-enable false

WJH Drop Reasons

The data NetQ sends to the gNMI agent is in the form of WJH drop reasons. The reasons are generated by the SDK and are stored in the /usr/etc/wjh_lib_conf.xml file on the switch and. Use this file as a guide to filter for specific reason types (L1, ACL, and so forth), reason IDs, and/or event severities.

L1 Drop Reasons

L2 Drop Reasons

Router Drop Reasons

Tunnel Drop Reasons

ACL Drop Reasons

Buffer Drop Reasons

Related Information