Access Control List Configuration
Cumulus Linux provides several tools to configure ACLs:
- NVUE is a Cumulus Linux-specific userspace tool you can use to configure custom ACLs.
iptables
,ip6tables
, andebtables
are Linux userspace tools you use to administer filtering rules for IPv4 packets, IPv6 packets, and Ethernet frames (layer 2 using MAC addresses).cl-acltool
is a Cumulus Linux-specific userspace tool you can use to administer filtering rules and configure default ACLs.cl-acltool
operates on various configuration files and usesiptables
,ip6tables
, andebtables
to install rules into the kernel. In addition,cl-acltool
programs rules in hardware for switch port interfaces, whichiptables
,ip6tables
andebtables
cannot do on their own.
Traffic Rules
Chains
ACLs in Cumulus Linux classify and control packets to, from, and across the switch, asserting policies at layer 2, 3 and 4 of the OSI model by inspecting packet and frame headers according to a list of rules. The iptables
, ip6tables
, and ebtables
userspace applications provide syntax you use to define rules.
The rules inspect or operate on packets at several points (chains) in the life of the packet through the system:
- PREROUTING touches packets before the switch routes them.
- INPUT touches packets after the switch determines that the packets are for the local system but before the control plane software receives them.
- FORWARD touches transit traffic as it moves through the switch.
- OUTPUT touches packets from the control plane software before they leave the switch.
- POSTROUTING touches packets immediately before they leave the switch but after a routing decision.
Tables
When you build rules to affect the flow of traffic, tables can access the individual chains. Linux provides three tables by default:
- Filter classifies traffic or filters traffic
- NAT applies Network Address Translation rules
- Mangle alters packets as they move through the switch
Each table has a set of default chains that modify or inspect packets at different points of the path through the switch. Chains contain the individual rules to influence traffic.
Rules
Rules classify the traffic you want to control. You apply rules to chains, which attach to tables.
Rules have several different components:
- Table: The first argument is the table.
- Chain: The second argument is the chain. Each table supports several different chains. See Tables above.
- Matches: The third argument is the match. You can specify multiple matches in a single rule. However, the more matches you use in a rule, the more memory the rule consumes.
- Jump: The jump specifies the target of the rule; what action to take if the packet matches the rule. If you omit this option in a rule, matching the rule has no effect on the packet, but the counters on the rule increment.
- Targets: The target is a user-defined chain (other than the one this rule is in), one of the special built-in targets that decides the fate of the packet immediately (like DROP), or an extended target. See Supported Rule Types below for different target examples.
How Rules Parse and Apply
The switch reads all the rules from each chain from iptables
, ip6tables
, and ebtables
and enters them in order into either the filter table or the mangle table. The switch reads the rules from the kernel in the following order:
- IPv6 (
ip6tables
) - IPv4 (
iptables
) ebtables
When you combine and put rules into one table, the order determines the relative priority of the rules; iptables
and ip6tables
have the highest precedence and ebtables
has the lowest.
The Linux packet forwarding construct is an overlay for how the silicon underneath processes packets. Be aware of the following:
The switch silicon reorders rules when
switchd
writes to the ASIC, whereas traditionaliptables
execute the list of rules in order.All rules, except for POLICE and SETCLASS rules, are terminating; after a rule matches, the action occurs and no more rules process.
When processing traffic, rules affecting the FORWARD chain that specify an ingress interface process before rules that match on an egress interface. As a workaround, rules that only affect the egress interface can have an ingress interface wildcard (only swp+ and bond+) that matches any interface you apply so that you can maintain order of operations with other input interface rules. For example, with the following rules:
-A FORWARD -i swp1 -j ACCEPT -A FORWARD -o swp1 -j ACCEPT <-- This rule processes LAST (because of egress interface matching) -A FORWARD -i swp2 -j DROP
If you modify the rules like this, they process in order:
-A FORWARD -i swp1 -j ACCEPT -A FORWARD -i swp+ -o $PORTA -j ACCEPT <-- These rules are performed in order (because of wildcard match on the ingress interface) -A FORWARD -i swp2 -j DROP
When using rules that do a mangle and a filter lookup for a packet, Cumulus Linux processes them in parallel and combines the action.
If there is no ingress interface or egress interface match, Cumulus Linux installs FORWARD chain rules in ingress by default.
When using the OUTPUT chain, you must assign rules to the source. For example, if you assign a rule to the switch port in the direction of traffic but the source is a bridge (VLAN), the rule does not affect the traffic and you must apply it to the bridge.
If you need to apply a rule to all transit traffic, use the FORWARD chain, not the OUTPUT chain.
The switch puts
ebtable
rules into either the IPv4 or IPv6 memory space depending on whether the rule uses IPv4 or IPv6 to make a decision. The switch only puts layer 2 rules that match the MAC address into the IPv4 memory space.
Rule Placement in Memory
INPUT and ingress (FORWARD -i
) rules occupy the same memory space. A rule counts as ingress if you set the -i
option. If you set both input and output options (-i
and -o
), the switch considers the rule as ingress and occupies that memory space. For example:
-A FORWARD -i swp1 -o swp2 -s 10.0.14.2 -d 10.0.15.8 -p tcp -j ACCEPT
If you set an output flag with the INPUT chain, you see an error. For example:
-A FORWARD,INPUT -i swp1 -o swp2 -s 10.0.14.2 -d 10.0.15.8 -p tcp -j ACCEPT
error: line 2 : output interface specified with INPUT chain error processing rule '-A FORWARD,INPUT -i swp1 -o swp2 -s 10.0.14.2 -d 10.0.15.8 -p tcp -j ACCEPT'
If you remove the -o
option and the interface, it is a valid rule.
Nonatomic Update Mode and Atomic Update Mode
Cumulus Linux enables atomic update mode by default. However, this mode limits the number of ACL rules that you can configure.
To increase the number of configurable ACL rules, configure the switch to operate in nonatomic mode.
Instead of reserving 50% of your TCAM space for atomic updates, nonatomic mode runs incremental updates that use available free space to write the new TCAM rules, then swap over to the new rules. Cumulus Linux deletes the old rules and frees up the original TCAM space. If there is insufficient free space to complete this task, the regular nonatomic (non-incremental) update runs, which interrupts traffic.
Nonatomic updates offer better scaling because all TCAM resources actively impact traffic. With atomic updates, half of the hardware resources are on standby and do not actively impact traffic.
Incremental nonatomic updates are table based, so they do not interrupt network traffic when you install new rules. The rules map to the following tables and update in this order:
mirror
(ingress only)ipv4-mac
(can be both ingress and egress)ipv6
(ingress only)
The incremental nonatomic update operation follows this order:
- Updates are incremental, one table at a time without stopping traffic.
- Cumulus Linux checks if the rules in a table are different from installation time; if a table does not have any changes, it does not reinstall the rules.
- If there are changes in a table, the new rules populate in new groups or slices in hardware, then that table switches over to the new groups or slices.
- Finally, old resources for that table free up. This process repeats for each of the tables listed above.
- If there are insufficient resources to hold both the new rule set and old rule set, Cumulus Linux tries regular nonatomic mode, which interrupts network traffic.
- If the regular nonatomic update fails, Cumulus Linux reverts back to the previous rules.
To set nonatomic mode:
cumulus@switch:~$ nv set system acl mode non-atomic
cumulus@switch:~$ nv config apply
- On Spectrum-2 and later, NVUE reloads
switchd
after you run and apply thenv set system acl mode
command. Reloadingswitchd
does not interrupt network services. - On Spectrum 1, NVUE restarts
switchd
after you run and apply thenv set system acl mode
command. Restartingswitchd
causes all network ports to reset in addition to resetting the switch hardware configuration.
Edit the
/etc/cumulus/switchd.conf
file to addacl.non_atomic_update_mode = TRUE
:cumulus@switch:~$ sudo nano /etc/cumulus/switchd.conf ... acl.non_atomic_update_mode = TRUE
On Spectrum-2 and later, reload
switchd
for the changes to take effect. Reloadingswitchd
does not interrupt network services.cumulus@switch:~$ sudo systemctl reload switchd.service
On Spectrum 1, restart
switchd
for the changes to take effect. Restartingswitchd
causes all network ports to reset in addition to resetting the switch hardware configuration.cumulus@switch:~$ sudo systemctl restart switchd.service
During regular non-incremental nonatomic updates, traffic stops, then continues after all the new configuration is in the hardware.
iptables, ip6tables, and ebtables
Do not use iptables
, ip6tables
, ebtables
directly; installed rules only apply to the Linux kernel and Cumulus Linux does not hardware accelerate. When you run cl-acltool -i
, Cumulus Linux resets all rules and deletes anything that is not in /etc/cumulus/acl/policy.conf
.
For example, the following rule appears to work:
cumulus@switch:~$ sudo iptables -A INPUT -p icmp --icmp-type echo-request -j DROP
The cl-acltool -L
command shows the rule:
cumulus@switch:~$ sudo cl-acltool -L ip
-------------------------------
Listing rules of type iptables:
-------------------------------
TABLE filter :
Chain INPUT (policy ACCEPT 72 packets, 5236 bytes)
pkts bytes target prot opt in out source destination
0 0 DROP icmp -- any any anywhere anywhere icmp echo-request
However, Cumulus Linux does not synchronize the rule to hardware. Running cl-acltool -i
or reboot
removes the rule without replacing it. To ensure that Cumulus Linux hardware accelerates all rules that can be in hardware, add them to /etc/cumulus/acl/policy.conf
and install them with the cl-acltool -i
command.
Estimate the Number of Rules
To estimate the number of rules you can create from an ACL entry, first determine if the ACL entry is ingress or egress. Then, determine if the entry is an IPv4-mac
or IPv6
type rule. This determines the slice to which the rule belongs. Use the following to determine how many entries the switch uses for each type.
By default, each entry occupies one double wide entry, except if the entry is one of the following:
An entry with multiple comma-separated input interfaces splits into one rule for each input interface. For example, this entry splits into two rules:
-A FORWARD -i swp1s0,swp1s1 -p icmp -j ACCEPT
An entry with multiple comma-separated output interfaces splits into one rule for each output interface. This entry splits into two rules:
-A FORWARD -i swp+ -o swp1s0,swp1s1 -p icmp -j ACCEPT
An entry with both input and output comma-separated interfaces splits into one rule for each combination of input and output interface. This entry splits into four rules:
-A FORWARD -i swp1s0,swp1s1 -o swp1s2,swp1s3 -p icmp -j ACCEPT
An entry with multiple layer 4 port ranges splits into one rule for each range. For example, this entry splits into two rules:
-A FORWARD -i swp+ -p tcp -m multiport --dports 1050:1051,1055:1056 -j ACCEPT
You can only use port ranges for ingress rules.
Match on VLAN IDs on Layer 2 Interfaces
You can match on VLAN IDs on layer 2 interfaces for ingress rules. The following example matches on a VLAN and DSCP class, and sets the internal class of the packet. For extended matching on IP fields, combine this rule with ingress iptable rules.
[ebtables]
-A FORWARD -p 802_1Q --vlan-id 100 -j mark --mark-set 102
[iptables]
-A FORWARD -i swp31 -m mark --mark 102 -m dscp --dscp-class CS1 -j SETCLASS --class 2
- Cumulus Linux reserves
mark
values between 0 and 100; for example, if you use--mark-set 10
, you see an error. Use mark values between 101 and 4196. - You cannot mark multiple VLANs with the same value.
- If you enable EVPN-MH and configure VLAN match rules in ebtables with a
mark
target, the ebtables rule might overwrite themark
set by traffic class rules you configure for EVPN-MH on ingress. Egress EVPN MH traffic class rules that match the ingress traffic classmark
might not get hit. To work around this issue, add ebtable rules toACCEPT
the packets already marked by EVPN-MH traffic class rules on ingress.
Install and Manage ACL Rules with NVUE
Instead of crafting a rule by hand, then installing it with cl-acltool
, you can use NVUE commands. Cumulus Linux converts the commands to the /etc/cumulus/acl/policy.d/50_nvue.rules
file. The rules you create with NVUE are independent of the default files /etc/cumulus/acl/policy.d/00control_plane.rules
and 99control_plane_catch_all.rules
.
Cumulus Linux 5.0 and later uses the -t mangle -A PREROUTING
chain for ingress rules and the -t mangle -A POSTROUTING
chain for egress rules instead of the - A FORWARD
chain used in previous releases.
Consider the following iptables
rule:
-t mangle -A PREROUTING -i swp1 -s 10.0.14.2/32 -d 10.0.15.8/32 -p tcp -j ACCEPT
To create this rule with NVUE, follow the steps below. NVUE adds all options in the rule automatically.
Set the rule type, the matching protocol, source IP address and port, destination IP address and port, and the action. You must provide a name for the rule (EXAMPLE1 in the commands below):
cumulus@switch:~$ nv set acl EXAMPLE1 type ipv4 cumulus@switch:~$ nv set acl EXAMPLE1 rule 10 match ip protocol tcp cumulus@switch:~$ nv set acl EXAMPLE1 rule 10 match ip source-ip 10.0.14.2/32 cumulus@switch:~$ nv set acl EXAMPLE1 rule 10 match ip tcp source-port ANY cumulus@switch:~$ nv set acl EXAMPLE1 rule 10 match ip dest-ip 10.0.15.8/32 cumulus@switch:~$ nv set acl EXAMPLE1 rule 10 match ip tcp dest-port ANY cumulus@switch:~$ nv set acl EXAMPLE1 rule 10 action permit
For firewall IPv4 type ACLs on the control plane, you can match on the hashlimit module (
hashimit
), the connection state (connection-state
), and the recent module (recent-list
). Refer to Firewall Rules.Apply the rule to an inbound or outbound interface with the
nv set interface <interface> acl
command.- For rules affecting the -t mangle -A PREROUTING chain (-A FORWARD in previous releases), apply the rule to an inbound or outbound interface: For example:
cumulus@switch:~$ nv set interface swp1 acl EXAMPLE1 inbound cumulus@switch:~$ nv config apply
- For rules affecting the INPUT or OUPUT chain (-A INPUT or -A OUTPUT), apply the rule to a control plane interface. For example:
cumulus@switch:~$ nv set interface swp1 acl EXAMPLE1 inbound control-plane cumulus@switch:~$ nv config apply
To see the installed rule, either examine the /etc/cumulus/acl/policy.d/50_nvue.rules
file or run the NVUE nv show acl <rule-name> rule <ID>
command:
cumulus@switch:~$ sudo cat /etc/cumulus/acl/policy.d/50_nvue.rules
[iptables]
## ACL EXAMPLE1 in dir inbound on interface swp1 ##
-t mangle -A PREROUTING -i swp1 -s 10.0.14.2/32 -d 10.0.15.8/32 -p tcp -j ACCEPT
...
cumulus@switch:~$ nv show acl EXAMPLE1 rule 10
operational applied
------------------- ------------ ------------
match
ip
source-ip 10.0.14.2/32 10.0.14.2/32
dest-ip 10.0.15.8/32 10.0.15.8/32
protocol tcp tcp
tcp
[source-port] ANY ANY
[dest-port] ANY ANY
To remove this rule, run the nv unset acl <acl-name>
and nv unset interface <interface> acl <acl-name>
commands. These commands delete the rule from the /etc/cumulus/acl/policy.d/50_nvue.rules
file.
cumulus@switch:~$ nv unset acl EXAMPLE1
cumulus@switch:~$ nv unset interface swp1 acl EXAMPLE1
cumulus@switch:~$ nv config apply
To show ACL statistics per interface, such as the total number of bytes that match the ACL rule, run the nv show interface <interface-id> acl <acl-id> statistics
or nv show interface <interface-id> acl <acl-id> statistics <rule-id>
command.
To see the list of all NVUE ACL commands, run the nv list-commands acl
command.
Install and Manage ACL Rules with cl-acltool
You can manage Cumulus Linux ACLs with cl-acltool
. Rules write first to the iptables
chains, as described above, and then synchronize to hardware through switchd
.
To examine the current state of chains and list all installed rules, run:
cumulus@switch:~$ sudo cl-acltool -L all
-------------------------------
Listing rules of type iptables:
-------------------------------
TABLE filter :
Chain INPUT (policy ACCEPT 432K packets, 31M bytes)
pkts bytes target prot opt in out source destination
0 0 DROP all -- swp+ any 240.0.0.0/5 anywhere
0 0 DROP all -- swp+ any 127.0.0.0/8 anywhere
0 0 DROP all -- swp+ any base-address.mcast.net/4 anywhere
0 0 DROP all -- swp+ any 255.255.255.255 anywhere
0 0 ACCEPT all -- swp+ any anywhere anywhere
Chain FORWARD (policy ACCEPT 0 packets, 0 bytes)
pkts bytes target prot opt in out source destination
Chain OUTPUT (policy ACCEPT 457K packets, 35M bytes)
pkts bytes target prot opt in out source destination
...
To list installed rules using native iptables
, ip6tables
and ebtables
, use the -L
option with the respective commands:
cumulus@switch:~$ sudo iptables -L
cumulus@switch:~$ sudo ip6tables -L
cumulus@switch:~$ sudo ebtables -L
To remove all installed rules, run:
cumulus@switch:~$ sudo cl-acltool -F all
To remove only the IPv4 iptables
rules, run:
cumulus@switch:~$ sudo cl-acltool -F ip
If the install fails, ACL rules in the kernel and hardware roll back to the previous state. You also see errors from programming rules in the kernel or ASIC.
Install Packet Filtering (ACL) Rules
cl-acltool
takes access control list (ACL) rule input in files. Each ACL policy file includes iptables
, ip6tables
and ebtables
categories under the tags [iptables]
, [ip6tables]
and [ebtables]
. You must assign each rule in an ACL policy to one of the rule categories.
See man cl-acltool(5)
for ACL rule details. For iptables
rule syntax, see man iptables(8)
. For ip6tables
rule syntax, see man ip6tables(8)
. For ebtables
rule syntax, see man ebtables(8)
.
See man cl-acltool(5)
and man cl-acltool(8)
for more details on using cl-acltool
.
By default:
- ACL policy files are in
/etc/cumulus/acl/policy.d/
. - All
*.rules
files in/etc/cumulus/acl/policy.d/
directory are also in/etc/cumulus/acl/policy.conf
. - All files in the
policy.conf
file install when the switch boots up. - The
policy.conf
file expects rule files to have a.rules
suffix as part of the file name.
The following shows an example ACL policy file:
[iptables]
-A INPUT -i swp1 -p tcp --dport 80 -j ACCEPT
-A FORWARD -i swp1 -p tcp --dport 80 -j ACCEPT
[ip6tables]
-A INPUT -i swp1 -p tcp --dport 80 -j ACCEPT
-A FORWARD -i swp1 -p tcp --dport 80 -j ACCEPT
[ebtables]
-A INPUT -p IPv4 -j ACCEPT
-A FORWARD -p IPv4 -j ACCEPT
You can use wildcards or variables to specify chain and interface lists.
- You can only use swp+ and bond+ as wildcard names.
- swp+ rules apply as an aggregate, not per port. If you want to apply per port policing, specify a specific port instead of the wildcard.
INGRESS = swp+
INPUT_PORT_CHAIN = INPUT,FORWARD
[iptables]
-A $INPUT_PORT_CHAIN -i $INGRESS -p tcp --dport 80 -j ACCEPT
[ip6tables]
-A $INPUT_PORT_CHAIN -i $INGRESS -p tcp --dport 80 -j ACCEPT
[ebtables]
-A INPUT -p IPv4 -j ACCEPT
You can write ACL rules for the system into multiple files under the default /etc/cumulus/acl/policy.d/
directory. The ordering of rules during installation follows the sort order of the files according to their file names.
Use multiple files to stack rules. The example below shows two rule files that separate rules for management and datapath traffic:
cumulus@switch:~$ ls /etc/cumulus/acl/policy.d/
00sample_mgmt.rules 01sample_datapath.rules
cumulus@switch:~$ cat /etc/cumulus/acl/policy.d/00sample_mgmt.rules
INGRESS_INTF = swp+
INGRESS_CHAIN = INPUT
[iptables]
# protect the switch management
-A $INGRESS_CHAIN -i $INGRESS_INTF -s 10.0.14.2 -d 10.0.15.8 -p tcp -j ACCEPT
-A $INGRESS_CHAIN -i $INGRESS_INTF -s 10.0.11.2 -d 10.0.12.8 -p tcp -j ACCEPT
-A $INGRESS_CHAIN -i $INGRESS_INTF -d 10.0.16.8 -p udp -j DROP
cumulus@switch:~$ cat /etc/cumulus/acl/policy.d/01sample_datapath.rules
INGRESS_INTF = swp+
INGRESS_CHAIN = INPUT, FORWARD
[iptables]
-A $INGRESS_CHAIN -i $INGRESS_INTF -s 192.0.2.5 -p icmp -j ACCEPT
-A $INGRESS_CHAIN -i $INGRESS_INTF -s 192.0.2.6 -d 192.0.2.4 -j DROP
-A $INGRESS_CHAIN -i $INGRESS_INTF -s 192.0.2.2 -d 192.0.2.8 -j DROP
Install all ACL policies under a directory:
cumulus@switch:~$ sudo cl-acltool -i -P ./rules
Reading files under rules
Reading rule file ./rules/01_http_rules.txt ...
Processing rules in file ./rules/01_http_rules.txt ...
Installing acl policy ...
Done.
Apply all rules and policies included in /etc/cumulus/acl/policy.conf
:
cumulus@switch:~$ sudo cl-acltool -i
Specify the Policy Files to Install
By default, Cumulus Linux installs any .rules
file you configure in /etc/cumulus/acl/policy.d/
. To add other policy files to an ACL, you need to include them in /etc/cumulus/acl/policy.conf
. For example, for Cumulus Linux to install a rule in a policy file called 01_new.datapathacl
, add include /etc/cumulus/acl/policy.d/01_new.rules
to policy.conf
:
cumulus@switch:~$ sudo nano /etc/cumulus/acl/policy.conf
#
# This file is a master file for acl policy file inclusion
#
# Note: This is not a file where you list acl rules.
#
# This file can contain:
# - include lines with acl policy files
# example:
# include <filepath>
#
# see manpage cl-acltool(5) and cl-acltool(8) for how to write policy files
#
include /etc/cumulus/acl/policy.d/01_new.datapathacl
Hardware Limitations for ACL Rules
The maximum number of rules that the switch hardware can store depends on:
- The combination of IPv4 and IPv6 rules; Cumulus Linux does not support the maximum number of rules for both IPv4 and IPv6 simultaneously.
- The number of default rules that Cumulus Linux provides.
- Whether the rules apply on ingress or egress.
- Whether the rules are in atomic or nonatomic mode; Cumulus Linux uses nonatomic mode rules when you enable nonatomic updates.
- Other resources that share the same table space, such as multicast route entries and internal VLAN counters.
If you exceed the maximum number of rules or run out of related memory resources for the ACL table, cl-acltool -i
generates one of the following errors:
error: hw sync failed (sync_acl hardware installation failed) Rolling back .. failed.
error: hw sync failed (Bulk counter init failed with No More Resources). Rolling back ..
To troubleshoot this issue and manage resources with high VLAN and ACL scale, refer to Troubleshooting ACL Rule Installation Failures.
NVIDIA Spectrum switches use a TCAM or ATCAM to quickly look up various tables that include ACLs, multicast routes, and certain internal VLAN counters. Depending on the size of the network ACLs, multicast routes, and VLAN counters, you might need to adjust some parameters to fit your network requirements into the tables.
TCAM Profiles on Spectrum 1
The NVIDIA Spectrum 1 ASIC (model numbers 2xx0) has one common TCAM space for both ingress and egress ACLs, which the switch also uses for multicast route entries.
Cumulus Linux controls the ACL and multicast route entry scale on NVIDIA Spectrum 1 switches with different TCAM profiles in combination with the ACL atomic and nonatomic update setting.
Profile | Atomic Mode IPv4 Rules | Atomic Mode IPv6 Rules | Nonatomic Mode IPv4 Rules | Nonatomic Mode IPv6 Rules | Multicast Route Entries |
---|---|---|---|---|---|
default | 500 | 250 | 1000 | 500 | 1000 |
ipmc-heavy | 750 | 500 | 1500 | 1000 | 8500 |
acl-heavy | 1750 | 1000 | 3500 | 2000 | 450 |
ipmc-max | 1000 | 500 | 2000 | 1000 | 13000 |
ip-acl-heavy | 6000 | 0 | 12000 | 0 | 0 |
- Even though the table above specifies the ip-acl-heavy profile supports no IPv6 rules, Cumulus Linux does not prevent you from configuring IPv6 rules. However, there is no guarantee that IPv6 rules work under the ip-acl-heavy profile.
- The ip-acl-heavy profile shows an updated number of supported atomic mode and nonatomic mode IPv4 rules. The previously published numbers were 7500 for atomic mode and 15000 for nonatomic mode IPv4 rules.
To configure the profile you want to use, set the tcam_resource.profile
parameter in the /etc/mlx/datapath/tcam_profile.conf
file, then restart switchd
:
cumulus@switch:~$ sudo nano /etc/mlx/datapath/tcam_profile.conf
...
tcam_resource.profile = ipmc-max
cumulus@switch:~$ sudo systemctl restart switchd.service
Spectrum 1 TCAM resource profiles that control ACLs and multicast route scale are different from forwarding resource profiles that control MAC table, IPv4, and IPv6 entry scale.
ATCAM on Spectrum-2 and Later
Switches with Spectrum-2 and later use a newer KVD scheme and an ATCAM design that is more flexible and allows a higher ACL scale than Spectrum 1. There is no TCAM resource profile on Spectrum-2 and later.
The following table shows the tested ACL rule limits. Because the KVD and ATCAM share space with forwarding table entries, multicast route entries, and VLAN flow counters, these ACL limits might vary based on your use of other tables.
These limits are valid when using any Spectrum-2 and later forwarding profile, except for the l2-heavy-3
and v6-lpm-heavy1
profiles, which reduce the ACL scale significantly.
For Spectrum-2 and later, all profiles support the same number of rules.
Atomic Mode IPv4 Rules | Atomic Mode IPv6 Rules | Nonatomic Mode IPv4 Rules | Nonatomic Mode IPv6 Rules |
---|---|---|---|
12500 | 6250 | 25000 | 12500 |
For information about nonatomic and atomic mode, refer to Nonatomic Update Mode and Atomic Update Mode.
ATCAM Resource Exhaustion
If you see error messages similar to No More Resources .. Rolling back
when you try to apply ACLs, refer to Troubleshooting ACL Rule Installation Failures for information on troublshooting and managing resources.
Supported Rule Types
The iptables
/ip6tables
/ebtables
construct tries to layer the Linux implementation on top of the underlying hardware but they are not always directly compatible. The following shows the supported rules for chains in iptables
, ip6tables
and ebtables
.
To learn more about any of the options shown in the tables below, run iptables -h [name of option]
. The same help syntax works for options for ip6tables
and ebtables
.
root@leaf1# ebtables -h tricolorpolice
...
tricolorpolice option:
--set-color-mode STRING setting the mode in blind or aware
--set-cir INT setting committed information rate in kbits per second
--set-cbs INT setting committed burst size in kbyte
--set-pir INT setting peak information rate in kbits per second
--set-ebs INT setting excess burst size in kbyte
--set-conform-action-dscp INT setting dscp value if the action is accept for conforming packets
--set-exceed-action-dscp INT setting dscp value if the action is accept for exceeding packets
--set-violate-action STRING setting the action (accept/drop) for violating packets
--set-violate-action-dscp INT setting dscp value if the action is accept for violating packets
Supported chains for the filter table:
INPUT FORWARD OUTPUT
iptables and ip6tables Rule Support
Rule Element | Supported | Unsupported |
---|---|---|
Matches | Src/Dst, IP protocol In/out interface IPv4: ecn, icmp, frag, ttl, IPv6: icmp6, hl, IP common: tcp (with flags), udp, multiport, DSCP, addrtype | Rules with input/output Ethernet interfaces do not apply Inverse matches |
Standard Targets | ACCEPT, DROP | RETURN, QUEUE, STOP, Fall Thru, Jump |
Extended Targets | LOG (IPv4/IPv6); UID is not supported for LOG TCP SEQ, TCP options or IP options ULOG SETQOS DSCP Unique to Cumulus Linux: SPAN ERSPAN (IPv4/IPv6) POLICE TRICOLORPOLICE SETCLASS |
ebtables Rule Support
Rule Element | Supported | Unsupported |
---|---|---|
Matches | ether type input interface/wildcard output interface/wildcard Src/Dst MAC IP: src, dest, tos, proto, sport, dport IPv6: tclass, icmp6: type, icmp6: code range, src/dst addr, sport, dport 802.1p (CoS) VLAN | Inverse matches Proto length |
Standard Targets | ACCEPT, DROP | RETURN, CONTINUE, Jump, Fall Thru |
Extended Targets | ULOG LOG Unique to Cumulus Linux: SPAN ERSPAN POLICE TRICOLORPOLICE SETCLASS |
Other Unsupported Rules
- Rules that have no matches and accept all packets in a chain are currently ignored.
- Chain default rules (that are ACCEPT) are also ignored.
Considerations
Splitting rules across the ingress TCAM and the egress TCAM causes the ingress IPv6 part of the rule to match packets going to all destinations, which can interfere with the regular expected linear rule match in a sequence. For example:
A higher rule can prevent a lower rule from matching:
Rule 1: -A FORWARD -o vlan100 -p icmp6 -j ACCEPT
Rule 2: -A FORWARD -o vlan101 -p icmp6 -s 01::02 -j ACCEPT
Rule 1 matches all icmp6 packets from to all out interfaces in the ingress TCAM.
This prevents rule 2 from matching, which is more specific but with a different out interface. Make sure to put more specific matches above more general matches even if the output interfaces are different.
When you have two rules with the same output interface, the lower rule might match depending on the presence of the previous rules.
Rule 1: -A FORWARD -o vlan100 -p icmp6 -j ACCEPT
Rule 2: -A FORWARD -o vlan101 -s 00::01 -j DROP
Rule 3: -A FORWARD -o vlan101 -p icmp6 -j ACCEPT
Rule 3 still matches for an icmp6 packet with sip 00:01 going out of vlan101. Rule 1 interferes with the normal function of rule 2 and/or rule 3.
When you have two adjacent rules with the same match and different output interfaces, such as:
Rule 1: -A FORWARD -o vlan100 -p icmp6 -j ACCEPT
Rule 2: -A FORWARD -o vlan101 -p icmp6 -j DROP
Rule 2 never matches on ingress. Both rules share the same mark.
Common Examples
Data Plane Policers
You can configure quality of service for traffic on the data plane. By using QoS policers, you can rate limit traffic so incoming packets get dropped if they exceed specified thresholds.
Counters on POLICE ACL rules in iptables
do not show dropped packets due to those rules.
The following example rate limits the incoming traffic on swp1 to 400 packets per second with a burst of 200 packets per second:
cumulus@switch:~$ nv set acl example1 type ipv4
cumulus@switch:~$ nv set acl example1 rule 10 action police
cumulus@switch:~$ nv set acl example1 rule 10 action police mode packet
cumulus@switch:~$ nv set acl example1 rule 10 action police burst 200
cumulus@switch:~$ nv set acl example1 rule 10 action police rate 400
cumulus@switch:~$ nv set interface swp1 acl example1 inbound
cumulus@switch:~$ nv config apply
Use the POLICE
target with iptables
. POLICE
takes these arguments:
--set-rate value
specifies the maximum rate in kilobytes (KB) or packets.--set-burst value
specifies the number of packets or kilobytes (KB) allowed to arrive sequentially.--set-mode string
sets the mode in KB (kilobytes) or pkt (packets) for rate and burst size.
For example, to rate limit the incoming traffic on swp1 to 400 packets per second with a burst of 200 packets per second and set this rule in your appropriate .rules
file:
-t mangle -A PREROUTING -i swp1 -j POLICE --set-mode pkt --set-rate 400 --set-burst 200
Control Plane Policers
You can configure quality of service for traffic on the control plane and rate limit traffic so incoming packets drop if they exceed certain thresholds in the following ways:
- Run NVUE commands.
- Edit the
/etc/cumulus/control-plane/policers.conf
file.
Cumulus Linux 5.0 and later no longer uses INPUT chain rules to configure control plane policers.
To configure control plane policers:
- Set the burst rate for the trap group with the
nv set system control-plane policer <trap-group> burst <value>
command. The burst rate is the number of packets or kilobytes (KB) allowed to arrive sequentially. - Set the forwarding rate for the trap group with the
nv set system control-plane policer <trap-group> rate <value>
command. The forwarding rate is the maximum rate in kilobytes (KB) or packets.
The trap group can be: arp
, bfd
, pim-ospf-rip
, bgp
, clag
, icmp-def
, dhcp-ptp
, igmp
, ssh
, icmp6-neigh
, icmp6-def-mld
, lacp
, lldp
, rpvst
, eapol
, ip2me
, acl-log
, nat
, stp
, l3-local
, span-cpu
, catch-all
, or NONE
.
The following example changes the PIM trap group forwarding rate and burst rate to 400 packets per second, and the IGMP trap group forwarding rate to 400 packets per second and burst rate to 200 packets per second:
cumulus@switch:~$ nv set system control-plane policer pim-ospf-rip rate 400
cumulus@switch:~$ nv set system control-plane policer pim-ospf-rip burst 400
cumulus@switch:~$ nv set system control-plane policer pim-ospf-rip state on
cumulus@switch:~$ nv set system control-plane policer igmp rate 400
cumulus@switch:~$ nv set system control-plane policer igmp burst 200
cumulus@switch:~$ nv config apply
To rate limit traffic using the /etc/cumulus/control-plane/policers.conf
file, you:
- Enable an individual policer for a trap group (set
enable
toTRUE
). - Set the policer rate in packets per second. The forwarding rate is the maximum rate in kilobytes (KB) or packets.
- Set the policer burst rate in packets per second. The burst rate is the number of packets or kilobytes (KB) allowed to arrive sequentially.
After you edit the /etc/cumulus/control-plane/policers.conf
file, you must reload the file with the /usr/lib/cumulus/switchdctl --load /etc/cumulus/control-plane/policers.conf
command.
When enable
is FALSE for a trap group, the trap group and catch-all
trap group have a shared policer. When enable
is TRUE, Cumulus Linux creates an individual policer for the trap group.
The following example changes the PIM trap group forwarding rate and burst rate to 400 packets per second, and the IGMP trap group forwarding rate to 400 packets per second and burst rate to 200 packets per second:
cumulus@switch:~$ sudo nano /etc/cumulus/control-plane/policers.conf
...
copp.pim_ospf_rip.enable = TRUE
copp.pim_ospf_rip.rate = 400
copp.pim_ospf_rip.burst = 400
...
copp.igmp.enable = TRUE
copp.igmp.rate = 400
copp.igmp.burst = 200
...
cumulus@switch:~$ /usr/lib/cumulus/switchdctl --load /etc/cumulus/control-plane/policers.conf
To show the control plane police configuration and statistics, run the NVUE nv show system control-plane policer --view=brief
command.
Cumulus Linux provides default control plane policer values. You can adjust these values to accommodate higher scale requirements for specific protocols as needed.
Control Plane ACLs
You can configure control plane ACLs to apply a single rule for all packets forwarded to the CPU regardless of the source interface or destination interface on the switch. Control plane ACLs allow you to regulate traffic forwarded to applications on the switch with more granularity than traps and to configure ACLs to block SSH from specific addresses or subnets.
Cumulus Linux applies inbound control plane ACLs in the INPUT chain and outbound control plane ACLs in the OUTPUT chain.
Cumulus Linux does not support a deny all control plane rule. This type of rule blocks traffic for interprocess communication and impacts overall system functionality.
The following example command applies the input control plane ACL called ACL1.
cumulus@switch:~$ nv set system control-plane acl ACL1 inbound
cumulus@switch:~$ nv config apply
The following example command applies the output control plane ACL called ACL2.
cumulus@switch:~$ nv set system control-plane acl ACL2 outbound
cumulus@switch:~$ nv config apply
To show statistics for all control-plane ACLs, run the nv show system control-plane acl
command:
cumulus@switch:~$ nv show system control-plane acl
ACL Name Rule ID In Packets In Bytes Out Packets Out Bytes
--------- ------- ---------- -------- ----------- ---------
acl1 1 0 0 0 0
65535 0 0 0 0
acl2 1 0 0 0 0
65535 0 0 0 0
To show statistics for a specific control-plane ACL, run the nv show system control-plane acl <acl_name> statistics
command:
cumulus@switch:~$ nv show system control-plane acl ACL1 statistics
Rule In Packet In Byte Out Packet Out Byte Summary
---- --------- ------- ---------- -------- ---------------------------
1 0 0 Bytes 0 0 Bytes match.ip.dest-ip: 9.1.2.3
2 0 0 Bytes 0 0 Bytes match.ip.source-ip: 7.8.2.3
Set DSCP on Transit Traffic
The following examples use the mangle table to modify the packet as it transits the switch. DSCP is in decimal notation in the examples below.
[iptables]
#Set SSH as high priority traffic.
-t mangle -A PREROUTING -i swp+ -p tcp -m multiport --dports 22 -j SETQOS --set-dscp 46
#Set everything coming in swp1 as AF13
-t mangle -A PREROUTING -i swp1 -j SETQOS --set-dscp 14
#Set Packets destined for 10.0.100.27 as best effort
-t mangle -A PREROUTING -i swp+ -d 10.0.100.27/32 -j SETQOS --set-dscp 0
#Example using a range of ports for TCP traffic
-t mangle -A PREROUTING -i swp+ -s 10.0.0.17/32 -d 10.0.100.27/32 -p tcp -m multiport --sports 10000:20000 -m multiport --dports 10000:20000 -j SETQOS --set-dscp 34
Apply the rule:
cumulus@switch:~$ sudo cl-acltool -i
To set SSH as high priority traffic:
cumulus@switch:~$ nv set acl EXAMPLE1 type ipv4
cumulus@switch:~$ nv set acl EXAMPLE1 rule 10 match ip protocol tcp
cumulus@switch:~$ nv set acl EXAMPLE1 rule 10 match ip tcp dest-port 22
cumulus@switch:~$ nv set acl EXAMPLE1 rule 10 action set dscp 46
cumulus@switch:~$ nv set interface swp1-48 acl EXAMPLE1 inbound
cumulus@switch:~$ nv config apply
To set everything coming in swp1 as AF13:
cumulus@switch:~$ nv set acl EXAMPLE1 type ipv4
cumulus@switch:~$ nv set acl EXAMPLE1 rule 10 action set dscp 14
cumulus@switch:~$ nv set interface swp1 acl EXAMPLE1 inbound
cumulus@switch:~$ nv config apply
To set Packets destined for 10.0.100.27 as best effort:
cumulus@switch:~$ nv set acl EXAMPLE1 type ipv4
cumulus@switch:~$ nv set acl EXAMPLE1 rule 10 match ip dest-ip 10.0.100.27/32
cumulus@switch:~$ nv set acl EXAMPLE1 rule 10 action set dscp 0
cumulus@switch:~$ nv set interface swp1-48 acl EXAMPLE1 inbound
cumulus@switch:~$ nv config apply
To use a range of ports for TCP traffic:
cumulus@switch:~$ nv set acl EXAMPLE1 type ipv4
cumulus@switch:~$ nv set acl EXAMPLE1 rule 10 match ip protocol tcp
cumulus@switch:~$ nv set acl EXAMPLE1 rule 10 match ip source-ip 10.0.0.17/32
cumulus@switch:~$ nv set acl EXAMPLE1 rule 10 match ip tcp source-port 10000:20000
cumulus@switch:~$ nv set acl EXAMPLE1 rule 10 match ip dest-ip 10.0.100.27/32
cumulus@switch:~$ nv set acl EXAMPLE1 rule 10 match ip tcp dest-port 10000:20000
cumulus@switch:~$ nv set acl EXAMPLE1 rule 10 action set dscp 34
cumulus@switch:~$ nv set interface swp1-48 acl EXAMPLE1 inbound
cumulus@switch:~$ nv config apply
To specify all ports on the switch in NVUE (swp+ in an iptables rule), you must set the range of interfaces on the switch as in the examples above (nv set interface swp1-48
). This command creates as many rules in the /etc/cumulus/acl/policy.d/50_nvue.rules
file as the number of interfaces in the range you specify.
Filter Specific TCP Flags
The example rule below drops ingress IPv4 TCP packets when you set the SYN bit and reset the RST, ACK, and FIN bits. The rule applies inbound on interface swp1. After configuring this rule, you cannot establish new TCP sessions that originate from ingress port swp1. You can establish TCP sessions that originate from any other port.
-t mangle -A PREROUTING -i swp1 -p tcp --tcp-flags ACK,SYN,FIN,RST SYN -j DROP
Apply the rule:
cumulus@switch:~$ sudo cl-acltool -i
cumulus@switch:~$ nv set acl EXAMPLE1 type ipv4
cumulus@switch:~$ nv set acl EXAMPLE1 rule 20 match ip protocol tcp
cumulus@switch:~$ nv set acl EXAMPLE1 rule 20 match ip tcp flags syn
cumulus@switch:~$ nv set acl EXAMPLE1 rule 20 match ip tcp mask rst
cumulus@switch:~$ nv set acl EXAMPLE1 rule 20 match ip tcp mask syn
cumulus@switch:~$ nv set acl EXAMPLE1 rule 20 match ip tcp mask fin
cumulus@switch:~$ nv set acl EXAMPLE1 rule 20 match ip tcp mask ack
cumulus@switch:~$ nv set acl EXAMPLE1 rule 20 action deny
cumulus@switch:~$ nv set interface swp1 acl EXAMPLE1 inbound
cumulus@switch:~$ nv config apply
Control Who Can SSH into the Switch
Run the following commands to control who can SSH into the switch. In the following example, 10.10.10.1/32 is the interface IP address (or loopback IP address) of the switch and 10.255.4.0/24 can SSH into the switch.
-A INPUT -i swp+ -s 10.255.4.0/24 -d 10.10.10.1/32 -j ACCEPT
-A INPUT -i swp+ -d 10.10.10.1/32 -j DROP
Apply the rule:
cumulus@switch:~$ sudo cl-acltool -i
cumulus@switch:~$ nv set acl example2 type ipv4
cumulus@switch:~$ nv set acl example2 rule 10 match ip source-ip 10.255.4.0/24
cumulus@switch:~$ nv set acl example2 rule 10 match ip dest-ip 10.10.10.1/32
cumulus@switch:~$ nv set acl example2 rule 10 action permit
cumulus@switch:~$ nv set acl example2 rule 20 match ip source-ip ANY
cumulus@switch:~$ nv set acl example2 rule 20 match ip dest-ip 10.10.10.1/32
cumulus@switch:~$ nv set acl example2 rule 20 action deny
cumulus@switch:~$ nv set system control-plane acl example2 inbound
cumulus@switch:~$ nv config apply
Block Traffic towards the eth0 Interface
To block traffic towards the eth0 interface, apply an ACL on the system control plane instead of on the eth0 interface. The following example creates an ACL called DENY-IN that blocks traffic from ingressing eth0 with source IP address 192.168.200.10:
cumulus@switch:~$ nv set acl DENY-IN rule 10 action deny
cumulus@switch:~$ nv set acl DENY-IN rule 10 match ip source-ip 192.168.200.10
cumulus@switch:~$ nv set acl DENY-IN type ipv4
cumulus@switch:~$ nv set system control-plane acl DENY-IN inbound
cumulus@switch:~$ nv config apply
Match on ECN Bits in the TCP IP Header
ECN allows end-to-end notification of network congestion without dropping packets. You can add ECN rules to match on the ECE, CWR, and ECT flags in the TCP IPv4 header.
By default, ECN rules match a packet with the bit set. You can reverse the match by using an explanation point (!).
Match on the ECE Bit
After an endpoint receives a packet with the CE bit set by a router, it sets the ECE bit in the returning ACK packet to notify the other endpoint that it needs to slow down.
To match on the ECE bit:
Create a rules file in the /etc/cumulus/acl/policy.d
directory and add the following rule under [iptables]
:
cumulus@switch:~$ sudo nano /etc/cumulus/acl/policy.d/30-tcp-flags.rules
[iptables]
-t mangle -A PREROUTING -i swp1 -p tcp -m ecn --ecn-tcp-ece -j ACCEPT
Apply the rule:
cumulus@switch:~$ sudo cl-acltool -i
cumulus@switch:~$ nv set acl example2 type ipv4
cumulus@switch:~$ nv set acl example2 rule 10 match ip protocol tcp
cumulus@switch:~$ nv set acl example2 rule 10 match ip ecn flags tcp-ece
cumulus@switch:~$ nv set acl example2 rule 10 action permit
cumulus@switch:~$ nv set interface swp1 acl example2 inbound
cumulus@switch:~$ nv config apply
Match on the CWR Bit
The CWR bit notifies the other endpoint of the connection that it received and reacted to an ECE.
To match on the CWR bit:
Create a rules file in the /etc/cumulus/acl/policy.d
directory and add the following rule under [iptables]
:
cumulus@switch:~$ sudo nano /etc/cumulus/acl/policy.d/30-tcp-flags.rules
[iptables]
-t mangle -A PREROUTING -i swp1 -p tcp -m ecn --ecn-tcp-cwr -j ACCEPT
Apply the rule:
cumulus@switch:~$ sudo cl-acltool -i
cumulus@switch:~$ nv set acl example2 type ipv4
cumulus@switch:~$ nv set acl example2 rule 10 match ip protocol tcp
cumulus@switch:~$ nv set acl example2 rule 10 match ip ecn flags tcp-cwr
cumulus@switch:~$ nv set acl example2 rule 10 action permit
cumulus@switch:~$ nv set interface swp1 acl example2 inbound
cumulus@switch:~$ nv config apply
Match on the ECT Bit
The ECT codepoints negotiate if the connection is ECN capable by setting one of the two bits to 1. Routers also use the ECT bit to indicate that they are experiencing congestion by setting both the ECT codepoints to 1.
To match on the ECT bit:
Create a rules file in the /etc/cumulus/acl/policy.d
directory and add the following rule under [iptables]
:
cumulus@switch:~$ sudo nano /etc/cumulus/acl/policy.d/30-tcp-flags.rules
[iptables]
-t mangle -A PREROUTING -i swp1 -p tcp -m ecn --ecn-ip-ect 1 -j ACCEPT
Apply the rule:
cumulus@switch:~$ sudo cl-acltool -i
cumulus@switch:~$ nv set acl example2 type ipv4
cumulus@switch:~$ nv set acl example2 rule 10 match ip protocol tcp
cumulus@switch:~$ nv set acl example2 rule 10 match ip ecn ip-ect 1
cumulus@switch:~$ nv set acl example2 rule 10 action permit
cumulus@switch:~$ nv set interface swp1 acl example2 inbound
cumulus@switch:~$ nv config apply
Example Configuration
The following example demonstrates how Cumulus Linux applies several different rules.
Egress Rule
The following rule blocks any TCP traffic with destination port 200 going through leaf01 to server01 (rule 1 in the diagram above).
[iptables]
-t mangle -A POSTROUTING -o swp1 -p tcp -m multiport --dports 200 -j DROP
cumulus@switch:~$ nv set acl EXAMPLE1 type ipv4
cumulus@switch:~$ nv set acl EXAMPLE1 rule 10 match ip protocol tcp
cumulus@switch:~$ nv set acl EXAMPLE1 rule 10 match ip tcp dest-port 200
cumulus@switch:~$ nv set acl EXAMPLE1 rule 10 action deny
cumulus@switch:~$ nv set interface swp1 acl EXAMPLE1 outbound
cumulus@switch:~$ nv config apply
Ingress Rule
The following rule blocks any UDP traffic with source port 200 going from server01 through leaf01 (rule 2 in the diagram above).
[iptables]
-t mangle -A PREROUTING -i swp1 -p udp -m multiport --sports 200 -j DROP
cumulus@switch:~$ nv set acl EXAMPLE1 type ipv4
cumulus@switch:~$ nv set acl EXAMPLE1 rule 10 match ip protocol udp
cumulus@switch:~$ nv set acl EXAMPLE1 rule 10 match ip udp source-port 200
cumulus@switch:~$ nv set acl EXAMPLE1 rule 10 action deny
cumulus@switch:~$ nv set interface swp1 acl EXAMPLE1 inbound
cumulus@switch:~$ nv config apply
Input Rule
The following rule blocks any UDP traffic with source port 200 and destination port 50 going from server02 to the leaf02 control plane (rule 3 in the diagram above).
[iptables]
-A INPUT -i swp2 -p udp -m multiport --dports 50 -j DROP
cumulus@switch:~$ nv set acl EXAMPLE1 type ipv4
cumulus@switch:~$ nv set acl EXAMPLE1 rule 10 match ip protocol udp
cumulus@switch:~$ nv set acl EXAMPLE1 rule 10 match ip udp dest-port 50
cumulus@switch:~$ nv set acl EXAMPLE1 rule 10 action deny
cumulus@switch:~$ nv set interface swp2 acl EXAMPLE1 inbound control-plane
cumulus@switch:~$ nv config apply
Output Rule
The following rule blocks any TCP traffic with source port 123 and destination port 123 going from leaf02 to server02 (rule 4 in the diagram above).
[iptables]
-A OUTPUT -o swp2 -p tcp -m multiport --sports 123 -m multiport --dports 123 -j DROP
cumulus@switch:~$ nv set acl EXAMPLE1 type ipv4
cumulus@switch:~$ nv set acl EXAMPLE1 rule 10 match ip protocol tcp
cumulus@switch:~$ nv set acl EXAMPLE1 rule 10 match ip tcp source-port 123
cumulus@switch:~$ nv set acl EXAMPLE1 rule 10 match ip tcp dest-port 123
cumulus@switch:~$ nv set acl EXAMPLE1 rule 10 action deny
cumulus@switch:~$ nv set interface swp2 acl EXAMPLE1 outbound control-plane
cumulus@switch:~$ nv config apply
Layer 2 Rules (ebtables)
The following rule blocks any traffic with source MAC address 00:00:00:00:00:12 and destination MAC address 08:9e:01:ce:e2:04 going from any switch port egress or ingress.
[ebtables]
-A FORWARD -s 00:00:00:00:00:12 -d 08:9e:01:ce:e2:04 -j DROP
cumulus@switch:~$ nv set acl EXAMPLE type mac
cumulus@switch:~$ nv set acl EXAMPLE rule 10 match mac source-mac 00:00:00:00:00:12
cumulus@switch:~$ nv set acl EXAMPLE rule 10 match mac dest-mac 08:9e:01:ce:e2:04
cumulus@switch:~$ nv set acl EXAMPLE rule 10 action deny
cumulus@switch:~$ nv set interface swp1-48 acl EXAMPLE inbound
cumulus@switch:~$ nv config apply
Considerations
Not All Rules Supported
Cumulus Linux does not support all iptables
, ip6tables
, or ebtables
rules. Refer to Supported Rules for specific rule support.
ACL Log Policer Limits Traffic
To protect the CPU from overloading, Cumulus Linux limits traffic copied to the CPU to 1 packet per second by an ACL Log Policer.
Bridge Traffic Limitations
Bridge traffic that matches LOG ACTION rules do not log to syslog; the kernel and hardware identify packets using different information.
You Cannot Forward Log Actions
You cannot forward logged packets. The hardware cannot both forward a packet and send the packet to the control plane (or kernel) for logging. A log action must also have a drop action.
SPAN Sessions that Reference an Outgoing Interface
SPAN sessions that reference an outgoing interface create mirrored packets based on the ingress interface before the routing/switching decision. See SPAN Sessions that Reference an Outgoing Interface and Use the CPU Port as the SPAN Destination in the Network Troubleshooting section.
iptables Interactions with cl-acltool
Because Cumulus Linux is a Linux operating system, you can use the iptables
commands. However, consider using cl-acltool
instead for the following reasons:
- Without using
cl-acltool
, rules do not install into hardware. - Running
cl-acltool -i
(the installation command) resets all rules and deletes anything that is not in the/etc/cumulus/acl/policy.conf
file.
For example, running the following command works:
cumulus@switch:~$ sudo iptables -A INPUT -p icmp --icmp-type echo-request -j DROP
The rules appear when you run cl-acltool -L
:
cumulus@switch:~$ sudo cl-acltool -L ip
-------------------------------
Listing rules of type iptables:
-------------------------------
TABLE filter :
Chain INPUT (policy ACCEPT 72 packets, 5236 bytes)
pkts bytes target prot opt in out source destination
0 0 DROP icmp -- any any anywhere anywhere icmp echo-request
However, running cl-acltool -i
or reboot
removes them. To ensure that Cumulus Linux can hardware accelerate all rules that can be in hardware, place them in the /etc/cumulus/acl/policy.conf
file, then run cl-acltool -i
.
Where to Assign Rules
- If you assign a switch port to a bond, you must assign any egress rules to the bond.
- When using the OUTPUT chain, you must assign rules to the source. For example, if you assign a rule to the switch port in the direction of traffic but the source is a bridge (VLAN), the rule does not affect the traffic and you must apply the rule to the bridge.
- If you need to apply a rule to all transit traffic, use the FORWARD chain, not the OUTPUT chain.
Troubleshooting ACL Rule Installation Failures
On Spectrum-2 and later, in addition to ACLs, items stored in KVD and ATCAM include internal counters for VLANs and interfaces in a bridge. If the network includes more than 1000 VLAN interfaces, the counters might occupy a significant amount of space and reduce the amount of available space for ACLs.
If you use all the ACL space, you might see error messages similar to the following when you try to apply ACLs:
cumulus@switch:$ sudo cl-acltool -i -p 00control_plane.rules
Using user provided rule file 00control_plane.rules
Reading rule file 00control_plane.rules ...
Processing rules in file 00control_plane.rules ...
error: hw sync failed (sync_acl hardware installation failed)
Installing acl policy... Rolling back ..
failed.
error: hw sync failed (Bulk counter init failed with No More Resources). Rolling back ..
You might also see messages similar to the following in the /var/log/syslog
file:
2023-12-07T16:31:32.386792-05:00 mlx-4700-51 sx_sdk: 1951 [FLOW_COUNTER] [NOTICE ]:
Spectrum_flow_counter_bulk_set: cm_bulk_block_add failed toallocated bulk size 64
You might also see messages similar to the following in the /var/log/switchd.log
file:
2023-12-07T16:31:32.387219-05:00 mlx-4700-51 switchd[7354]: hal_mlx_sdk_counter_wrap.c:366 ERR
sx_api_flow_counter_bulk_set create failed with: No More Resources
2023-12-07T16:31:32.387338-05:00 mlx-4700-51 switchd[7354]: hal_mlx_flx_acl.c:9531 ERR
flow_counter_bulk_set create failed with: No More Resources
2023-12-07T16:31:32.387415-05:00 mlx-4700-51 switchd[7354]: hal_mlx_flx_acl.c:3202 ERR BULK
counter init failed with No More Resources
2023-12-07T16:31:32.387481-05:00 mlx-4700-51 switchd[7354]: hal_mlx_flx_acl.c:2765
hal_mlx_flx_chain_desc_install returned 0
2023-12-07T16:31:32.387554-05:00 mlx-4700-51 switchd[7354]: hal_mlx_flx_acl.c:1981 ERR
acl_plan_install returned 0
2023-12-07T16:31:32.393928-05:00 mlx-4700-51 switchd[7354]: sync_acl.c:225 ERR BULK counter init
failed with No More Resources
2023-12-07T16:31:32.394047-05:00 mlx-4700-51 switchd[7354]: sync_acl.c:6669 ERR BULK counter init
failed with No More Resources
For information on ACL resource limitations, refer to Hardware Limitations for ACL Rules.
On Spectrum-2 and later, you might see resource errors when you try to configure more than 1000 VLAN interfaces because certain VLAN counters share space with ACL memory in the ATCAM.
To free up resources, you can:
Reduce the number of specified VLANs or VLAN interfaces to the number you need in the network.
Free up VLAN flow counter space; edit the
/etc/mlx/datapath/stats.conf
file to uncomment and set thehal.mlx.stats.vlan.enable
option toFALSE
, then reloadswitchd
:cumulus@switch:$ sudo nano /etc/mlx/datapath/stats.conf # Once the stat controls are enabled/disabled, # run 'systemctl reload switchd' for changes to take effect hal.mlx.stats.vlan.enable = FALSE
cumulus@switch:$ sudo systemctl reload switchd.service
The flow counters are internal counters for debugging; you do not see the counters in nv show interface <interface> counters
or cl-netstat
commands.
To see how much space the flow counters consume, examine the Flow Counters
line in the cl-resource-query
output.
ACLs Do not Match when the Output Port on the ACL is a Subinterface
The ACL does not match on packets when you configure a subinterface as the output port. The ACL matches on packets only if the primary port is as an output port. If a subinterface is an output or egress port, the packets match correctly.
For example:
-A FORWARD -o swp49s1.100 -j ACCEPT
Egress ACL Matching on Bonds
Cumulus Linux does not support ACL rules that match on an outbound bond interface. For example, you cannot create the following rule:
[iptables]
-A FORWARD -o <bond_intf> -j DROP
To work around this issue, duplicate the ACL rule on each physical port of the bond. For example:
[iptables]
-A FORWARD -o <bond-member-port-1> -j DROP
-A FORWARD -o <bond-member-port-2> -j DROP
SSH Traffic to the Management VRF
To allow SSH traffic to the management VRF, use -i mgmt
, not -i eth0
. For example:
-A INPUT -i mgmt -s 10.0.14.2/32 -p tcp --dport ssh -j ACCEPT
INPUT Chain Rules and swp+
In INPUT chain rules, the -i swp+
match works only if the destination of the packet is towards a layer 3 swp interface; the match does not work if the packet terminates at an SVI interface (for example, vlan10). To allow traffic towards specific SVIs, use rules without any interface match or rules with individual -i <SVI>
matches.