Connection Tracking Window Validation

Connection Tracking (CT) is the process of tracking a TCP connection between two endpoints (E.g., between server and client). This involves maintaining the state of the connection, identifying state changing packets, identifying malformed packets, identifying packets that do not conform to the protocol or the current connection state, and more.

CT is an important process used in many networking solutions, that require state awareness. These networking solutions include stateful firewalls, the Linux kernel, and more.

Currently, the CT is executed in the software level, but with the world becoming more and more data driven, networking applications are required to run with higher bandwidth, and software is becoming the bottleneck. This is where DPDK CT comes in, we offer the ability to offload the CT process to the hardware, accommodating our speed of light NICs.

To make the CT offload possible, we added hardware support for the operation (CT HW module), and a software rte_flow API that connects the user to the hardware.

The CT offload can track a single connection only, meaning every connection should be offloaded separately.

The HW module can track a given connection by maintaining a structure in HW referred to as CT context, which holds information about a single TCP connection. As such, every connection being tracked requires its own CT context object. However, the HW does not have the ability to initialize the context.

When the HW module receives a packet that is part of a tracked connection, it checks the packet based on that connection’s context, stores the result for later usage (CT result matching), and then updates the context based on the received packet.

This section explains how for a given connection, the context is initialized, and how to associate that connection’s packets with a context object.

The API brings these main changes:

• rte_flow CT Context

  The struct rte_flow_action_conntrackstruct is almost identical to the HW CT context mentioned above. The SW context holds the following two extra fields .peer_port and .is_original_portexplained in details below.

• CT rte_flow Item (RTE_FLOW_ITEM_TYPE_CONNTRACK)

  This item allows the user to insert an rte_flow rule that matches the result of the CT HW module (CT action). To insert this rule, make sure that the packet passes through the CT HW module before reaching this rule. You can match on 5 possible results (valid/state_change/error/disabled/bad_packet) or combinations.

• CT rte_flow Action (RTE_FLOW_ACTION_TYPE_CONNTRACK)

  The action .conf field points to a SW CT context and activates the CT HW module on this rule. When this action is created, the HW creates its own CT context and copies the values of the SW context.

• CT Context Create / Modify

  • Create: rte_flow_action_handle_createfunction creates a CT action effectively initializing the HW CT context. The function also returns a handle to the created action. The handle is used later to insert the action as part of multiple rte_flow rules. Meaning the handle can connect a HW context with the packets that match on the rule.

  • Modify: rte_flow_action_handle_update function allows you to modify the HW context, as well as changing the .is_original_portwhich will be updated separately as explained below.

New Data Structures

struct rte_flow_action_conntrack (SW CT Context)

enum rte_flow_conntrack_state (CT state)

enum rte_flow_conntrack_tcp_last_index (packet type/TCP flags)

struct rte_flow_tcp_dir_param

• The CT action must be wrapped by an action_handle

• Retransmission limit checking cannot work

• The CT action cannot be inserted in group 0

• Cannot track a connection before seeing the first 2 handshake packets (SYN and SYN-ACK)

• No direction validation is done for closing sequence (FIN)

• Maximum number of supported ports with CT action is 16

For the best flow insertion usage experience, the following actions/steps are recommended:

• Use “hint_num_of_rules_log” to configure the maximum expected number of rules in each group, see Hint to the Driver on the Number of Flow Rules. The value is the log number of the expected rules. For example, assuming cross port and each port is going to have 4M rules, the value should be 22. In case of a single port (meaning we add 2 rules to the same table), the expected number of rules is 8M so the value should be 23.

• Reuse CT objects, the usage flow should be as follows upon new connection arrival:

  • Check if there is a free CT object in the app pool. The CT object should match the direction of the original ones that were set when the CT object was created. For example, if the app created a CT with original port = 0 and reply port 1, it can be reused only in the same configuration.

  • If the CT object is reused, the application should call the modify the action with the new requested state.

  • If the app does not have a free CT object, then it should allocate a new one by calling the create function.

The app creates the rules just like before with the CT object. To close a connection, the app removes the rules and adds the CT object to the application pool.

General Comments

1. The export part will be changed in the June release and will be part of the rule creation.

2. A new NV config mode “icm_cache_mode_large_scale_steering” is added to enable less cache misses and improve performance for cases when working with many steering rules.

   This capability is enabled by setting the mlxconfig parameter "ICM_CACHE_MODE"” to "1": mlxconfig -d <device> set ICM_CACHE_MODE=1. Note that this optimization is intended to be used with very large number of steering rules.

3. To save insertion rate time, the CT objects can be pre-allocated at startup.

4. Querying the CT state will have a huge effect on the performance. All measurements are done without the query.

5. Before closing the device, all CT objects should be destroyed.

CT is offloaded using the rte_flow rules. Below is explanation on how to create rules in three different groups:

• Group 0: Has a single rule that matches on a TCP packet and jumps to group 1.

• Group 1: Has two rules, the first one matches on the connection’s 5-tuple, tags the packet with original, applies CT action, and then jumps to group 2. The second rule is similar to the first however, it matches on the reverse 5-tuple, and tags the packet with reply. Both rules share the same CT action (and HW CT context). These rules should be created for every 5-tuple (connection).

• Group 2: Has 6 rules, 4 of them are used to inform the application about an event, while the remaining two rules match on a valid packet. There is a rule for every direction where for each direction the packet will be forwarded to a different hairpin queue.

The flows are as follows:

As explained above, the rules in group 1 are inserted for every new connection. We refer to inserting these rules as offloading a connection.

To offload a connection, you must use the API as follows:

1. Create a SW CT context that matches the connection parameters. It is important to set .is_original_port= 1.

2. Create a CT action that points to the SW context, and then commit it using rte_flow_action_handle_createwhich will return an action_handle.

3. Insert the rule(*) that matches on the 5-tuple of the connection. However, instead of using action CT, you can use the action_handle created and insert that.

4. Modify the CT action that was created in 2, such that .is_original_port= 0. You can modify the action by using rte_flow_action_handle_updateand providing the struct rte_flow_modify_conntrack parameter.

5. Insert the rule(**) that matches on the reverse 5-tuple of the connection. Same as 3.

The below is an example of how the CT context should be initialized (what values to put). In our example we are interested in tracking a TCP connection only after a three-way handshake has passed, as such we will be configuring the CT context based on information gathered from these 3 packets.

The three-way handshake packets are as follows:

handshake_pkts[0] == syn_pkt == S

handshake_pkts[1] == syn_ack_pkt == SA

handshake_pkts[2] == ack_pkt == A

SW CT Context Initial Values