If you are using the current version of Cumulus Linux, the content on this page may not be up to date. The current version of the documentation is available here. If you are redirected to the main page of the user guide, then this page may have been renamed; please search for it there.

Resource Diagnostics Using cl-resource-query

You can use the cl-resource-query command or the NCLU net show system asic command to retrieve information about host entries, MAC entries, layer 2 and layer 3 routes, and ECMP routes that are in use. Because Cumulus Linux synchronizes routes between the kernel and the switching silicon, if the required resource pools in hardware fill up, new kernel routes can cause existing routes to move from being fully allocated to being partially allocated. To avoid this, monitor the routes in the hardware to keep them below the ASIC limits. For example, on a Broadcom Tomahawk switch, the limits are as follows:

routes: 8192 <<<< if all routes are IPv6, or 65536 if all routes are IPv4
route mask limit 64
host_routes: 73728
ecmp_nhs: 16327
ecmp_nhs_per_route: 52

This translates to about 314 routes with ECMP nexthops, if every route has the maximum ECMP nexthops.

To monitor the routes in Cumulus Linux hardware, use the cl-resource-query command. The results vary between switches running on different chipsets.

The example below shows cl-resource-query results for a Broadcom Tomahawk switch:

cumulus@switch:~$ sudo cl-resource-query
IPv4/IPv6 host entries:                 0,   0% of maximum value  40960
IPv4 neighbors:                         0
IPv6 neighbors:                         0
IPv4 route entries:                     4,   0% of maximum value  65536
IPv6 route entries:                     8,   0% of maximum value   8192
IPv4 Routes:                            4
IPv6 Routes:                            8
Total Routes:                          12,   0% of maximum value  65536
ECMP nexthops:                          0,   0% of maximum value  16327
MAC entries:                            1,   0% of maximum value  40960
Total Mcast Routes:                     0,   0% of maximum value  20480
Ingress ACL entries:                  195,  12% of maximum value   1536
Ingress ACL counters:                 195,  12% of maximum value   1536
Ingress ACL meters:                    21,   1% of maximum value   2048
Ingress ACL slices:                     6, 100% of maximum value      6
Egress ACL entries:                    58,  11% of maximum value    512
Egress ACL counters:                   58,   5% of maximum value   1024
Egress ACL meters:                     29,   5% of maximum value    512
Egress ACL slices:                      2, 100% of maximum value      2
Ingress ACL ipv4_mac filter table:     36,  14% of maximum value    256 (allocated: 256)
Ingress ACL ipv6 filter table:         29,  11% of maximum value    256 (allocated: 256)
Ingress ACL mirror table:               0,   0% of maximum value      0 (allocated: 0)
Ingress ACL 8021x filter table:         0,   0% of maximum value      0 (allocated: 0)
Ingress PBR ipv4_mac filter table:      0,   0% of maximum value      0 (allocated: 0)
Ingress PBR ipv6 filter table:          0,   0% of maximum value      0 (allocated: 0)
Ingress ACL ipv4_mac mangle table:      0,   0% of maximum value      0 (allocated: 0)
Ingress ACL ipv6 mangle table:          0,   0% of maximum value      0 (allocated: 0)
Egress ACL ipv4_mac filter table:      29,  11% of maximum value    256 (allocated: 256)
Egress ACL ipv6 filter table:           0,   0% of maximum value      0 (allocated: 0)
ACL L4 port range checkers:             2,   6% of maximum value     32

The example below shows cl-resource-query results for a Broadcom Trident II switch:

cumulus@switch:~$ sudo cl-resource-query
IPv4/IPv6 host entries:                 0,   0% of maximum value  16384
IPv4 neighbors:                         0
IPv6 neighbors:                         0
IPv4 route entries:                     0,   0% of maximum value 131072
IPv6 route entries:                     1,   0% of maximum value  20480
IPv4 Routes:                            0
IPv6 Routes:                            1
Total Routes:                           1,   0% of maximum value 131072
ECMP nexthops:                          0,   0% of maximum value  16346
MAC entries:                            0,   0% of maximum value  32768
Total Mcast Routes:                     0,   0% of maximum value   8192
Ingress ACL entries:                  130,   6% of maximum value   2048
Ingress ACL counters:                  86,   4% of maximum value   2048
Ingress ACL meters:                    21,   0% of maximum value   4096
Ingress ACL slices:                     4,  66% of maximum value      6
Egress ACL entries:                    58,  11% of maximum value    512
Egress ACL counters:                   58,   5% of maximum value   1024
Egress ACL meters:                     29,   5% of maximum value    512
Egress ACL slices:                      2, 100% of maximum value      2
Ingress ACL ipv4_mac filter table:     36,   7% of maximum value    512 (allocated: 256)
Ingress ACL ipv6 filter table:         29,   3% of maximum value    768 (allocated: 512)
Ingress ACL mirror table:               0,   0% of maximum value      0 (allocated: 0)
Ingress ACL 8021x filter table:         0,   0% of maximum value      0 (allocated: 0)
Ingress PBR ipv4_mac filter table:      0,   0% of maximum value      0 (allocated: 0)
Ingress PBR ipv6 filter table:          0,   0% of maximum value      0 (allocated: 0)
Ingress ACL ipv4_mac mangle table:      0,   0% of maximum value      0 (allocated: 0)
Ingress ACL ipv6 mangle table:          0,   0% of maximum value      0 (allocated: 0)
Egress ACL ipv4_mac filter table:      29,  11% of maximum value    256 (allocated: 256)
Egress ACL ipv6 filter table:           0,   0% of maximum value      0 (allocated: 0)
ACL L4 port range checkers:             2,   8% of maximum value     24

On a switch with a Spectrum ASIC, the cl-resource-query command shows the number of TCAM entries used by the different types of ACL resources.

The example below shows cl-resource-query results for an NVIDIA Mellanox Spectrum switch:

cumulus@switch:~$ sudo cl-resource-query
IPv4 host entries:                      0,   0% of maximum value  32768
IPv6 host entries:                      0,   0% of maximum value  16384
IPv4 neighbors:                         0
IPv6 neighbors:                         0
IPv4 route entries:                     0,   0% of maximum value  65536
IPv6 route entries:                     7,   0% of maximum value  28672
IPv4 Routes:                            0
IPv6 Routes:                            7
Total Routes:                           7,   0% of maximum value  94208
ECMP nexthops:                          0,   0% of maximum value   4101
MAC entries:                            0,   0% of maximum value  40960
Total Mcast Routes:                     0,   0% of maximum value    400
Ingress ACL entries:                    0,   0% of maximum value      0
Ingress ACL counters:                   0,   0% of maximum value      0
Ingress ACL meters:                     0,   0% of maximum value      0
Ingress ACL slices:                     0,   0% of maximum value      0
Egress ACL entries:                     0,   0% of maximum value      0
Egress ACL counters:                    0,   0% of maximum value      0
Egress ACL meters:                      0,   0% of maximum value      0
Egress ACL slices:                      0,   0% of maximum value      0
Ingress ACL ipv4_mac filter table:      0,   0% of maximum value      0 (allocated: 0)
Ingress ACL ipv6 filter table:          0,   0% of maximum value      0 (allocated: 0)
Ingress ACL mirror table:               0,   0% of maximum value      0 (allocated: 0)
Ingress ACL 8021x filter table:         0,   0% of maximum value      0 (allocated: 0)
Ingress PBR ipv4_mac filter table:      0,   0% of maximum value      0 (allocated: 0)
Ingress PBR ipv6 filter table:          0,   0% of maximum value      0 (allocated: 0)
Ingress ACL ipv4_mac mangle table:      0,   0% of maximum value      0 (allocated: 0)
Ingress ACL ipv6 mangle table:          0,   0% of maximum value      0 (allocated: 0)
Egress ACL ipv4_mac filter table:       0,   0% of maximum value      0 (allocated: 0)
Egress ACL ipv6 filter table:           0,   0% of maximum value      0 (allocated: 0)
ACL L4 port range checkers:             0,   0% of maximum value      0
ACL Regions:                            4,   1% of maximum value    400
ACL 18B Rules Key:                      2,   0% of maximum value   2256
ACL 32B Rules Key:                      0,   0% of maximum value   1024
ACL 54B Rules Key:                      2,   0% of maximum value   1024

Ingress ACL and Egress ACL entries show the counts in single wide (not double-wide). For information about ACL entries, see Estimate the Number of ACL Rules.