gNMI Streaming

gNMI Support on Cumulus Linux

You can use gNMI, the gRPC network management interface, to collect system resource, interface, and counter information from Cumulus Linux and export it to your own gNMI client.

Configure the gNMI Agent

To configure the gNMI agent, you need to enable it on every switch you want to use with gNMI. Optionally, you can update the default gNMI port. The gNMI agent listens over port 9339 by default. You can change this setting in case you use that port in another application:

The /etc/netq/netq.yml file stores the configuration.

To configure the gNMI agent on a switch:

Enable the gNMI agent:

cumulus@switch:~$ netq config add agent gnmi-enable true

If you want to change the default port over which the gNMI agent listens, run:
```
cumulus@switch:~$ netq config add agent gnmi-port <gnmi_port>
```

Restart the NetQ Agent:

cumulus@switch:~$ netq config restart agent

Use Only the gNMI Agent

It is possible (although it is not recommended) to collect data using only the gNMI agent, and not the NetQ Agent. However, this sends data only to gNMI and not to NetQ.

To use only gNMI for data collection, disable the NetQ Agent, which is always enabled by default:

cumulus@switch:~$ netq config add agent opta-enable false

You cannot disable both the NetQ Agent and the gNMI agent.

If you enable both the gNMI agent and the traditional NetQ agent on Cumulus Linux, the data from OpenConfig models and the nvidia-if-ethernet-ext model are not sent to gNMI if a NetQ server is not reachable. WJH data continues streaming to gNMI in this state. If you are only using gNMI and a NetQ telemetry server does not exist, you should disable the NetQ agent by setting opta-enable to false.

Supported Models

Cumulus Linux supports the following OpenConfig models:

Model	Supported Data
openconfig-interfaces	Name, Operstatus, AdminStatus, IfIndex, MTU, LoopbackMode, Enabled
openconfig-if-ethernet	AutoNegotiate, PortSpeed, MacAddress, NegotiatedPortSpeed, Counters
openconfig-if-ethernet-ext	Frame size counters
openconfig-system	Memory, CPU

gNMI clients can also use the following model for extended ethernet counters:

▼ nvidia-if-ethernet-ext

module nvidia-if-ethernet-counters-ext {
    // xPath --> /interfaces/interface[name=*]/ethernet/counters/state/

   namespace "http://nvidia.com/yang/nvidia-ethernet-counters";
   prefix "nvidia-if-ethernet-counters-ext";


  // import some basic types
  import openconfig-interfaces { prefix oc-if; }
  import openconfig-if-ethernet { prefix oc-eth; }
  import openconfig-yang-types { prefix oc-yang; }


  revision "2021-10-12" {
    description
      "Initial revision";
    reference "1.0.0.";
  }

  grouping ethernet-counters-ext {

    leaf alignment-error {
      type oc-yang:counter64;
    }

    leaf in-acl-drops {
      type oc-yang:counter64;
    }

    leaf in-buffer-drops {
      type oc-yang:counter64;
    }

    leaf in-dot3-frame-errors {
      type oc-yang:counter64;
    }

    leaf in-dot3-length-errors {
      type oc-yang:counter64;
    }

    leaf in-l3-drops {
      type oc-yang:counter64;
    }

    leaf in-pfc0-packets {
      type oc-yang:counter64;
    }

    leaf in-pfc1-packets {
      type oc-yang:counter64;
    }

    leaf in-pfc2-packets {
      type oc-yang:counter64;
    }

    leaf in-pfc3-packets {
      type oc-yang:counter64;
    }

    leaf in-pfc4-packets {
      type oc-yang:counter64;
    }

    leaf in-pfc5-packets {
      type oc-yang:counter64;
    }

    leaf in-pfc6-packets {
      type oc-yang:counter64;
    }

    leaf in-pfc7-packets {
      type oc-yang:counter64;
    }

    leaf out-non-q-drops {
      type oc-yang:counter64;
    }

    leaf out-pfc0-packets {
      type oc-yang:counter64;
    }

    leaf out-pfc1-packets {
      type oc-yang:counter64;
    }

    leaf out-pfc2-packets {
      type oc-yang:counter64;
    }

    leaf out-pfc3-packets {
      type oc-yang:counter64;
    }

    leaf out-pfc4-packets {
      type oc-yang:counter64;
    }

    leaf out-pfc5-packets {
      type oc-yang:counter64;
    }

    leaf out-pfc6-packets {
      type oc-yang:counter64;
    }

    leaf out-pfc7-packets {
      type oc-yang:counter64;
    }

    leaf out-q0-wred-drops {
      type oc-yang:counter64;
    }

    leaf out-q1-wred-drops {
      type oc-yang:counter64;
    }

    leaf out-q2-wred-drops {
      type oc-yang:counter64;
    }

    leaf out-q3-wred-drops {
      type oc-yang:counter64;
    }

    leaf out-q4-wred-drops {
      type oc-yang:counter64;
    }

    leaf out-q5-wred-drops {
      type oc-yang:counter64;
    }

    leaf out-q6-wred-drops {
      type oc-yang:counter64;
    }

    leaf out-q7-wred-drops {
      type oc-yang:counter64;
    }

    leaf out-q8-wred-drops {
      type oc-yang:counter64;
    }

    leaf out-q9-wred-drops {
      type oc-yang:counter64;
    }

    leaf out-q-drops {
      type oc-yang:counter64;
    }

    leaf out-q-length {
      type oc-yang:counter64;
    }

    leaf out-wred-drops {
      type oc-yang:counter64;
    }

    leaf symbol-errors {
      type oc-yang:counter64;
    }

    leaf out-tx-fifo-full {
      type oc-yang:counter64;
    }

  }

  augment "/oc-if:interfaces/oc-if:interface/oc-eth:ethernet/" +
    "oc-eth:state/oc-eth:counters" {
      uses ethernet-counters-ext;
  }

}

SONiC only supports collection of WJH data with gNMI.

Collect WJH Data Using gNMI

You can export What Just Happened data from the NetQ Agent to your own gNMI client.

The client should use the following YANG model as a reference:

▼ nvidia-if-wjh-drop-aggregate

module nvidia-wjh {
    // Entrypoint /oc-if:interfaces/oc-if:interface
    //
    // xPath L1     --> interfaces/interface[name=*]/wjh/aggregate/l1
    // xPath L2     --> /interfaces/interface[name=*]/wjh/aggregate/l2/reasons/reason[id=*][severity=*]
    // xPath Router --> /interfaces/interface[name=*]/wjh/aggregate/router/reasons/reason[id=*][severity=*]
    // xPath Tunnel --> /interfaces/interface[name=*]/wjh/aggregate/tunnel/reasons/reason[id=*][severity=*]
    // xPath Buffer --> /interfaces/interface[name=*]/wjh/aggregate/buffer/reasons/reason[id=*][severity=*]
    // xPath ACL    --> /interfaces/interface[name=*]/wjh/aggregate/acl/reasons/reason[id=*][severity=*]

    import openconfig-interfaces { prefix oc-if; }

    namespace "http://nvidia.com/yang/what-just-happened-config";
    prefix "nvidia-wjh";

    revision "2021-10-12" {
        description
            "Initial revision";
        reference "1.0.0.";
    }

    augment "/oc-if:interfaces/oc-if:interface" {
        uses interfaces-wjh;
    }

    grouping interfaces-wjh {
        description "Top-level grouping for What-just happened data.";
        container wjh {
            container aggregate {
                container l1 {
                    container state {
                        leaf drop {
                            type string;
                            description "Drop list based on wjh-drop-types module encoded in JSON";
                        }
                    }
                }
                container l2 {
                    uses reason-drops;
                }
                container router {
                    uses reason-drops;
                }
                container tunnel {
                    uses reason-drops;
                }
                container acl {
                    uses reason-drops;
                }
                container buffer {
                    uses reason-drops;
                }
            }
        }
    }

    grouping reason-drops {
        container reasons {
            list reason {
                key "id severity";
                leaf id {
                    type leafref {
                        path "../state/id";
                    }
                    description "reason ID";
                }
                leaf severity {
                    type leafref {
                        path "../state/severity";
                    }
                    description "Reason severity";
                }
                container state {
                    leaf id {
                        type uint32;
                        description "Reason ID";
                    }
                    leaf name {
                        type string;
                        description "Reason name";
                    }
                    leaf severity {
                        type string;
                        mandatory "true";
                        description "Reason severity";
                    }
                    leaf drop {
                        type string;
                        description "Drop list based on wjh-drop-types module encoded in JSON";
                    }
                }
            }
        }
    }
}

module wjh-drop-types {
    namespace "http://nvidia.com/yang/what-just-happened-config-types";
    prefix "wjh-drop-types";

    container l1-aggregated {
        uses l1-drops;
    }
    container l2-aggregated {
        uses l2-drops;
    }
    container router-aggregated {
        uses router-drops;
    }
    container tunnel-aggregated {
        uses tunnel-drops;
    }
    container acl-aggregated {
        uses acl-drops;
    }
    container buffer-aggregated {
        uses buffer-drops;
    }

    grouping reason-key {
        leaf id {
            type uint32;
            mandatory "true";
            description "reason ID";
        }
        leaf severity {
            type string;
            mandatory "true";
            description "Severity";
        }
    }

    grouping reason_info {
        leaf reason {
                type string;
                mandatory "true";
                description "Reason name";
        }
        leaf drop_type {
            type string;
            mandatory "true";
            description "reason drop type";
        }
        leaf ingress_port {
            type string;
            mandatory "true";
            description "Ingress port name";
        }
        leaf ingress_lag {
            type string;
            description "Ingress LAG name";
        }
        leaf egress_port {
            type string;
            description "Egress port name";
        }
        leaf agg_count {
            type uint64;
            description "Aggregation count";
        }
        leaf severity {
            type string;
            description "Severity";
        }
        leaf first_timestamp {
            type uint64;
            description "First timestamp";
        }
        leaf end_timestamp {
            type uint64;
            description "End timestamp";
        }
    }

    grouping packet_info {
        leaf smac {
            type string;
            description "Source MAC";
        }
        leaf dmac {
            type string;
            description "Destination MAC";
        }
        leaf sip {
            type string;
            description "Source IP";
        }
        leaf dip {
            type string;
            description "Destination IP";
        }
        leaf proto {
            type uint32;
            description "Protocol";
        }
        leaf sport {
            type uint32;
            description "Source port";
        }
        leaf dport {
            type uint32;
            description "Destination port";
        }
    }

    grouping l1-drops {
        description "What-just happened drops.";
        leaf ingress_port {
            type string;
            description "Ingress port";
        }
        leaf is_port_up {
            type boolean;
            description "Is port up";
        }
        leaf port_down_reason {
            type string;
            description "Port down reason";
        }
        leaf description {
            type string;
            description "Description";
        }
        leaf state_change_count {
            type uint64;
            description "State change count";
        }
        leaf symbol_error_count {
            type uint64;
            description "Symbol error count";
        }
        leaf crc_error_count {
            type uint64;
            description "CRC error count";
        }
        leaf first_timestamp {
            type uint64;
            description "First timestamp";
        }
        leaf end_timestamp {
            type uint64;
            description "End timestamp";
        }
        leaf timestamp {
            type uint64;
            description "Timestamp";
        }
    }
    grouping l2-drops {
        description "What-just happened drops.";
        uses reason_info;
        uses packet_info;
    }

    grouping router-drops {
        description "What-just happened drops.";
        uses reason_info;
        uses packet_info;
    }

    grouping tunnel-drops {
        description "What-just happened drops.";
        uses reason_info;
        uses packet_info;
    }

    grouping acl-drops {
        description "What-just happened drops.";
        uses reason_info;
        uses packet_info;
        leaf acl_rule_id {
            type uint64;
            description "ACL rule ID";
        }
        leaf acl_bind_point {
            type uint32;
            description "ACL bind point";
        }
        leaf acl_name {
            type string;
            description "ACL name";
        }
        leaf acl_rule {
            type string;
            description "ACL rule";
        }
    }

    grouping buffer-drops {
        description "What-just happened drops.";
        uses reason_info;
        uses packet_info;
        leaf traffic_class {
            type uint32;
            description "Traffic Class";
        }
        leaf original_occupancy {
            type uint32;
            description "Original occupancy";
        }
        leaf original_latency {
            type uint64;
            description "Original latency";
        }
    }
}

Supported Features

In this release, the gNMI agent supports capability and stream subscribe requests for WJH events.

WJH Drop Reasons

The data NetQ sends to the gNMI agent is in the form of WJH drop reasons. The reasons are generated by the SDK and are stored in the /usr/etc/wjh_lib_conf.xml file on the switch and. Use this file as a guide to filter for specific reason types (L1, ACL, and so forth), reason IDs, and/or event severities.

L1 Drop Reasons

Reason ID	Reason	Description
10021	Port admin down	Validate port configuration
10022	Auto-negotiation failure	Set port speed manually, disable auto-negotiation
10023	Logical mismatch with peer link	Check cable/transceiver
10024	Link training failure	Check cable/transceiver
10025	Peer is sending remote faults	Replace cable/transceiver
10026	Bad signal integrity	Replace cable/transceiver
10027	Cable/transceiver is not supported	Use supported cable/transceiver
10028	Cable/transceiver is unplugged	Plug cable/transceiver
10029	Calibration failure	Check cable/transceiver
10030	Cable/transceiver bad status	Check cable/transceiver
10031	Other reason	Other L1 drop reason

L2 Drop Reasons

Reason ID	Reason	Severity	Description
201	MLAG port isolation	Notice	Expected behavior
202	Destination MAC is reserved (DMAC=01-80-C2-00-00-0x)	Error	Bad packet was received from the peer
203	VLAN tagging mismatch	Error	Validate the VLAN tag configuration on both ends of the link
204	Ingress VLAN filtering	Error	Validate the VLAN membership configuration on both ends of the link
205	Ingress spanning tree filter	Notice	Expected behavior
206	Unicast MAC table action discard	Error	Validate MAC table for this destination MAC
207	Multicast egress port list is empty	Warning	Validate why IGMP join or multicast router port does not exist
208	Port loopback filter	Error	Validate MAC table for this destination MAC
209	Source MAC is multicast	Error	Bad packet was received from peer
210	Source MAC equals destination MAC	Error	Bad packet was received from peer

Router Drop Reasons

Reason ID	Reason	Severity	Description
301	Non-routable packet	Notice	Expected behavior
302	Blackhole route	Warning	Validate routing table for this destination IP
303	Unresolved neighbor/next hop	Warning	Validate ARP table for the neighbor/next hop
304	Blackhole ARP/neighbor	Warning	Validate ARP table for the next hop
305	IPv6 destination in multicast scope FFx0:/16	Notice	Expected behavior - packet is not routable
306	IPv6 destination in multicast scope FFx1:/16	Notice	Expected behavior - packet is not routable
307	Non IP packet	Notice	Destination MAC is the router, packet is not routable
308	Unicast destination IP but multicast destination MAC	Error	Bad packet was received from the peer
309	Destination IP is loopback address	Error	Bad packet was received from the peer
310	Source IP is multicast	Error	Bad packet was received from the peer
311	Source IP is in class E	Error	Bad packet was received from the peer
312	Source IP is loopback address	Error	Bad packet was received from the peer
313	Source IP is unspecified	Error	Bad packet was received from the peer
314	Checksum or IPver or IPv4 IHL too short	Error	Bad cable or bad packet was received from the peer
315	Multicast MAC mismatch	Error	Bad packet was received from the peer
316	Source IP equals destination IP	Error	Bad packet was received from the peer
317	IPv4 source IP is limited broadcast	Error	Bad packet was received from the peer
318	IPv4 destination IP is local network (destination=0.0.0.0/8)	Error	Bad packet was received from the peer
320	Ingress router interface is disabled	Warning	Validate your configuration
321	Egress router interface is disabled	Warning	Validate your configuration
323	IPv4 routing table (LPM) unicast miss	Warning	Validate routing table for this destination IP
324	IPv6 routing table (LPM) unicast miss	Warning	Validate routing table for this destination IP
325	Router interface loopback	Warning	Validate the interface configuration
326	Packet size is larger than router interface MTU	Warning	Validate the router interface MTU configuration
327	TTL value is too small	Warning	Actual path is longer than the TTL

Tunnel Drop Reasons

Reason ID	Reason	Severity	Description
402	Overlay switch - Source MAC is multicast	Error	The peer sent a bad packet
403	Overlay switch - Source MAC equals destination MAC	Error	The peer sent a bad packet
404	Decapsulation error	Error	The peer sent a bad packet

ACL Drop Reasons

Reason ID	Reason	Severity	Description
601	Ingress port ACL	Notice	Validate ACL configuration
602	Ingress router ACL	Notice	Validate ACL configuration
603	Egress router ACL	Notice	Validate ACL configuration
604	Egress port ACL	Notice	Validate ACL configuration

Buffer Drop Reasons

Reason ID	Reason	Severity	Description
503	Tail drop	Warning	Monitor network congestion
504	WRED	Warning	Monitor network congestion
505	Port TC Congestion Threshold Crossed	Notice	Monitor network congestion
506	Packet Latency Threshold Crossed	Notice	Monitor network congestion

gNMI presentation to IETF

gNMI Client Requests

You can use your gNMI client on a host server to request capabilities and data the agent is subscribed to.

The following example shows a gNMI client request for interface speed:

gnmi_client -target_addr 10.209.37.121:9339 -xpath "/interfaces/interface[name=swp1]/ethernet/state/port-speed" -once
{
   "Response": {
      "Update": {
         "update": [
            {
               "val": {
                  "Value": {
                     "StringVal": "SPEED_40GB"
                  }
               },
               "path": {
                  "elem": [
                     {
                        "name": "state"
                     },
                     {
                        "name": "port-speed"
                     }
                  ]
               }
            }
         ],
         "timestamp": 1636910588085654861,
         "prefix": {
            "target": "netq",
            "elem": [
               {
                  "name": "interfaces"
               },
               {
                  "name": "interface",
                  "key": {
                     "name": "swp1"
                  }
               },
               {
                  "name": "ethernet"
               }
            ]
         }
      }
   }
}

The following example shows a gNMI client request for WJH drop data:

gnmi_client -target_addr 10.209.37.121:9339 -xpath "/interfaces/interface[name=swp8]/wjh/aggregate/l2/reasons/reason[id=210]"
{
   "Response": {
      "Update": {
         "update": [
            {
               "val": {
                  "Value": {
                     "StringVal": "[{
									  "IngressPort": "swp8",
									  "DropType": "L2",
									  "Reason": "Source MAC equals destination MAC",
									  "Severity": "Error",
									  "Smac": "00:02:10:00:00:01",
									  "Dmac": "00:02:10:00:00:01",
									  "Proto": 6,
									  "Sport": 15,
									  "Dport": 16,
									  "Sip": "1.1.1.1"
									  "Dip": "2.2.2.2",
									  "AggCount": 192,
									  "FirstTimestamp": 1636907412,
									  "EndTimestamp": 1636907432,
								   }]"

                  }
               },
               "path": {
                  "elem": [
                     {
                        "name": "state"
                     },
                     {
                        "name": "drop"
                     }
                  ]
               }
            }
         ],
         "prefix": {
            "elem": [
               {
                  "name": "interfaces"
               },
               {
                  "key": {
                     "name": "swp8"
                  },
                  "name": "interface"
               },
               {
                  "name": "wjh"
               },
               {
                  "name": "aggregate"
               },
               {
                  "name": "l2"
               },
               {
                  "name": "reasons"
               },
               {
                  "key" : {
                     "severity": "error",
                     "id": "210"
                  },
                  "name" : "reason"
               }
            ],
            "target": "netq"
         },
         "timestamp": 1636907442362981645
      }
   }
}

gNMI Streaming

gNMI Support on Cumulus Linux

Configure the gNMI Agent

Use Only the gNMI Agent

Supported Models

Collect WJH Data Using gNMI

Supported Features

WJH Drop Reasons

L1 Drop Reasons

L2 Drop Reasons

Router Drop Reasons

Tunnel Drop Reasons

ACL Drop Reasons

Buffer Drop Reasons

gNMI Client Requests

Products

Solutions

Learn

gNMI Streaming

gNMI Support on Cumulus Linux

Configure the gNMI Agent

Use Only the gNMI Agent

Supported Models

Collect WJH Data Using gNMI

Supported Features

WJH Drop Reasons

L1 Drop Reasons

L2 Drop Reasons

Router Drop Reasons

Tunnel Drop Reasons

ACL Drop Reasons

Buffer Drop Reasons

Related Information

gNMI Client Requests