DOCA Flow Connection Tracking

This guide provides an overview and configuration instructions for DOCA Flow CT API.

DOCA Flow Connection-Tracking AKA, ConnTrack (CT) is 5-tuple table usually supporting the following :

• Track 5-tuple sessions (or 6-tuple when having a zone).

• Zone based - virtual tables.

• Aging, remove idle connections.

• Sets Meta-Data for a connection.

• Bi-directional packet handling.

• High connections per second (CPS) rate.

The CT module makes it simple and efficient to track connections by leveraging hardware resources. The module supports both autonomous and managed mode.

DOCA Flow CT pipe handles non-encapsulated TCP and UDP packets. CT pipe only support forward next pipe or miss next pipe, all packets match known connection 6-tuples are forwarded to CT pipe forward pipe, no-matching packets are forwarded to miss pipe. The user application must handle packets accordingly.

DOCA Flow CT API is fundamentally built around four major parts:

• CT module manipulation – configure CT module resources.

• CT connection entry manipulation - add, remove or update connection entries.

• Callbacks - handle async entry processing result.

• Pipe and entry statistics.

Aging

Timeout/aging time is a time in seconds that sets the maximum allowed time for a session to be kept without a packet seen. If the time has passed and no packet was seen the session is considered as terminated.

To support Aging, a dedicated Aging thread is started to poll and check counters for all connections,

Autonomous mode

Multiple CT workers process connections in parallel, connection's lifecycle is controlled by packet state and aging.

CT workers establish and close connections automatically based on connection packet flags.

Packet Meta is defined as following:

uint32_t src : 1 ; /**< Source port in multi-port E-Switch mode */
uint32_t hairpin : 1 ; /**< Subject to forward using hairpin. */
uint32_t type : 2 ; /**< CT packet type: New, End or Update */
uint32_t data : 28 ; /**< Zone set by user or reserved after CT pipe. */

CT table matches Packet Meta and 5-tuples, change type of meta:

• NONE (known) if packet match any connection rule.

• NEW if new TCP or UDP connection.

• END if TCP connection closed.

Packet Meta src and hairpin are used for forwarding pipe and worker to deliver packet.

Managed mode

The user manages thread, parsing and process connection state packets, create or destroy connection by calling DOCA Flow CT management APIs.

CT aging module notify aged connections by calling callbacks.

The user can create connection rules with different pattern, meta or counter for each packet direction.

The user is responsible for defining meta and mask to match and modify.

The user can create one rule of a connection firstly, then create the other rule with API doca_flow_ct_entry_add_dir().

To enable DOCA Flow CT on the DPU, perform the following:

1. Enable "iommu.passthrough" in Linux boot commands or disable SMMU from DPU BIOS

2. Create huge pages.

3. Configure DPU firmware with "LAG_RESOURCE_ALLOCATION=1"

4. Put DPU into E-Switch mode if single port or multi-port E-Switch mode to support two PF ports.

Please reference to unit_test/README.txt for detail steps.

DOCA Flow CT supports actions based on meta and NAT operations.

Each action can be defined as shared or non-shared.

Shared actions

Actions that can be shared between entries. Shared actions are predefined and reused in multiple entries.

The user gets a handle per shared action created and uses this handle as a reference to the action where required.

It is the user responsibility to track shared actions and to remove them when they become irrelevant.

Shared actions are defined using control queue (see struct doca_flow_ct_cfg).

Non-shared actions

Actions provided with their data during entry create/update.

These actions are completely managed by DOCA Flow CT and can't be reused in multiple entries.

Actions templates

The user should provide actions templates during Doca Flow CT pipe creation (as for any other pipe),

Note that only templates for meta and NAT will be accepted (according to struct doca_flow_ct_actions).

During entry create/update, different actions can be provided per direction (different action content and/or different type)

Feature enable

Enable user actions by setting the following:

1. User actions templates (during CT pipe creation).

2. Number of user actions (during DOCA Flow CT init).

For the library API reference, refer to DOCA FLOW and CT API documentation in the NVIDIA DOCA Libraries APIs.

The following sections provide additional details about the library API.

enum doca_flow_ct_flags

DOCA Flow CT configuration optional flags.

Fields:

DOCA_FLOW_CT_FLAG_STATS = 1u << 0

Enable internal pipe counters for packet tracking purposes. Call doca_flow_pipe_dump(<ct_pipe>) to dump counter values, each call will dump values changed.

DOCA_FLOW_CT_FLAG_WORKER_STATS = 1u << 1 ,

Enable worker thread internal debug counter periodical dump. Autonomous mode only.

DOCA_FLOW_CT_FLAG_NO_AGING = 1u << 2 ,

Disable aging feature.

DOCA_FLOW_CT_FLAG_SW_PKT_PARSING = 1u << 3 ,

Enable CT worker software packet parsing to support VLAN, IPv6 options or special tunnel types.

DOCA_FLOW_CT_FLAG_MANAGED = 1u << 4 ,

Enable managed mode which user application is responsible to manage packet handling and call CT API to manipulate CT connection entries.

DOCA_FLOW_CT_FLAG_ASYMMETRIC = 1u << 5 ,

Allows different 6-tuple table definition for origin and reply direction, default to symmetric mode, uses same meta and reverse 5-tuples for reply direction. Managed mode only.

DOCA_FLOW_CT_FLAG_ASYMMETRIC_COUNTER = 1u << 6 ,

Enable different counters for origin and reply direction. Managed mode only.

DOCA_FLOW_CT_FLAG_NO_COUNTER = 1u << 7 ,

Disable counter and aging to save aging thread CPU cycles.

DOCA_FLOW_CT_FLAG_DEFAULT_MISS = 1u << 8 ,

Check TCP SYN flags and UDP in CT miss flow to identify ADD type packets.

DOCA_FLOW_CT_FLAG_WIRE_TO_WIRE = 1u << 9 ,

Hint traffic comes from uplink wire and forwards to uplink wire.

struct direction_cfg

Managed mode configuration for origin or reply direction.

Fields:

bool match_inner;

5-tuple match pattern applies to packet inner layer.

struct doca_flow_meta * zone_match_mask ;

Mask to indicate meta field and bits to match.

struct doca_flow_meta * meta_modify_mask ;

Mask to indicate meta field and bits to modify on connection packet match.

struct doca_flow_ct_cfg

DOCA Flow CT configuration

Fields:

uint32_t nb_arm_queues ;

Number of CT queues. In autonomous mode, also the number of worker threads.

uint32_t nb_ctrl_queues ;

Number of CT control queues, used for defining shared actions.

uint32_t nb_user_actions ;

Max number of user actions supported (shared & non shared).

uint32_t nb_arm_sessions [ DOCA_FLOW_CT_SESSION_MAX ];

Max CT connection number of IPv4 and IPv6.

uint32_t flags ;

CT configuration flags.

struct doca_dev * doca_dev ;

DOCA device.

void * ib_dev ;

Deprecated.

void * ib_pd ;

Deprecated.

uint16_t tcp_timeout_s ;

TCP aging timeout in second. Autonomous mode only.

uint16_t tcp_session_del_s ;

Second to delay when closing a connection to allow final ACK packet pass.

uint16_t udp_timeout_s ;

UDP aging timeout in second. Autonomous mode only.

uint16_t aging_core ;

CPU core ID for CT aging thread to bind.

doca_flow_ct_flow_log_cb flow_log_cb ;

Flow log callback function. When set.

struct doca_flow_ct_aging_ops * aging_ops ;

User defined aging logic callback functions. Fallback to default aging logic.

union {
struct direction_cfg direction [ 2 ];

Managed mode configuration for origin and reply direction.

Below fields are dedicate for autonomous mode:

struct {

enum doca_flow_tun_type tunnel_type ;

Encapsulation tunnel type.

uint16_t vxlan_dst_port ;

VxLAN outer UDP destination port in big endian.

enum doca_flow_ct_hash_type hash_type ;

Type of connection hash table type: NONE or SYMMETRIC_HASH.

uint32_t meta_user_bits ;

User packet meta bits to be owned by the user.

uint32_t meta_action_bits ;

User packet meta bits to be carried by identified connection packet.

struct doca_flow_meta * meta_zone_mask ;

Mask to indicate meta field and bits saving zone information.

struct doca_flow_meta * connection_id_mask ;

Mask to indicate meta field and bits for CT internal connection ID.
};
};

struct doca_flow_ct_actions

DOCA Flow CT action structure is used in the following cases:

1. For defining shared actions. In this case action data is provided by the user. Action handle is returned by DOCA Flow CT.

2. For defining entry with actions. The structure can be filled in two options:

   1. With action handle of a previously created shared action.

   2. With non-shared action data.

Fields:

enum doca_flow_resource_type resource_type ;

Shared/ non-shared action.

union {

uint32_t action_handle ;

Used when creating an entry with a shared action.

struct { Used when creating an entry with non-shared action or when creating a shared action.

uint32_t action_idx ;

Actions template index.

struct doca_flow_meta meta;

Modify meta.

struct doca_flow_header_l4_port l4_port;

UDP or TCP source and destination port.

union {

struct doca_flow_ct_ip4 ip4;

Source and destination IPv4 addresses.

struct doca_flow_ct_ip6 ip6;

Source and destination IPv6 addresses.

};

} data ;

};