DOCA Flow Connection Tracking

This guide provides an overview and configuration instructions for DOCA Flow CT API.

DOCA Flow Connection Tracking (CT) is a 5-tuple table which supports the following:

• Track 5-tuple sessions (or 6-tuple when a zone is available)

• Zone based – virtual tables

• Aging (i.e., removes idle connections)

• Sets metadata for a connection

• Bidirectional packet handling

• High rate of connections per second (CPS)

The CT module makes it simple and efficient to track connections by leveraging hardware resources. The module supports both autonomous and managed mode.

DOCA Flow CT pipe handles non-encapsulated TCP and UDP packets. The CT pipe only supports forward to next pipe or miss to next pipe actions:

• All packets matching known connection 6-tuples are forwarded to the CT's forward pipe

• Non-matching packets are forwarded to the miss pipe

The user application must handle packets accordingly.

The DOCA Flow CT API is built around four major parts:

• CT module manipulation – configuring CT module resources

• CT connection entry manipulation – adding, removing, or updating connection entries

• Callbacks – handling asynchronous entry processing result

• Pipe and entry statistics

Aging

Aging time is a time in seconds that sets the maximum allowed time for a session to be maintained without a packet seen. If that time elapses with no packet being detected, the session is terminated.

To support aging, a dedicated aging thread is started to poll and check counters for all connections.

Autonomous Mode

In this mode, DOCA runs multiple CT workers internally, to handle connections in parallel.

A connection's lifecycle is controlled by the connection state encapsulated in the packet and time-based aging.

CT workers establish and close connections automatically based on the connection's state stored in packet meta.

Packet meta is defined as follows:

            

            
uint32_t src : 1;      /**< Source port in multi-port E-Switch mode */
uint32_t hairpin : 1;  /**< Subject to forward using hairpin. */
uint32_t type : 2;     /**< CT packet type: New, End or Update */
uint32_t data : 28;    /**< Zone set by user or reserved after CT pipe. */
        

• data – CT table matches on packet meta (zone) and 5-tuples

• type – can have the following values:

  • NONE – (known) if packet hit any connection rule

  • NEW – if new TCP or UDP connection

  • END – if TCP connection closed

• src and hairpin – used for forwarding pipe and worker to deliver packet

Managed Mode

The application is responsible for managing the worker threads in this mode, parsing and handling the connection's lifecycle.

Managed mode uses DOCA Flow CT management APIs to create or destroy the connections.

The CT aging module notifies on aged out connections by calling callbacks.

Users can create connection rules with a different pattern, meta, or counter, for each packet direction.

Users can create one rule of a connection first, then create another rule using API doca_flow_ct_entry_add_dir().

DOCA Flow API can be used to process CT entries with a CT-dedicated queue.

• doca_flow_entries_process – process pipe entries in queue

• doca_flow_aging_handle – handle pipe entries aging

DPU

To enable DOCA Flow CT on the DPU, perform the following on the Arm:

1. Enable iommu.passthrough in Linux boot commands ( or disable SMMU from the DPU BIOS):

   1. Run:

      Copy

      Copied!

                  
      
                  
      sudo vim /etc/default/grub
              

   2. Set GRUB_CMDLINE_LINUX="iommu.passthrough=1".

   3. Run:

      Copy

      Copied!

                  
      
                  
      sudo update-grub
      sudo reboot
              

2. Configure DPU firmware with LAG_RESOURCE_ALLOCATION=1:

   Copy

   Copied!

               
   
               
   sudo mlxconfig -d <device-id> s LAG_RESOURCE_ALLOCATION=1
           

   Info

   Retrieve device-id from the output of the mst status -v command. If, under the MST tab, the value is N/A, run the mst start command.

3. Update /etc/mellanox/mlnx-bf.conf as follows:

   Copy

   Copied!

               
   
               
   ALLOW_SHARED_RQ="no"
           

4. Perform power cycle on the host and Arm sides.

5. If working with a single port, set the DPU into e-switch mode:

   Copy

   Copied!

               
   
               
   sudo devlink dev eswitch set pci/<pcie-address> mode switchdev
   sudo devlink dev param set pci/<pcie-address> name esw_multiport value false cmode runtime
           

   Info

   Retrieve pcie-address from the output of the mst status -v command.

6. If working with two PF ports, set the DPU into multi-port e-switch mode (for the 2 PCIe devices):

   Copy

   Copied!

               
   
               
   sudo devlink dev param set pci/<pcie-address> name esw_multiport value true cmode runtime
           

   Info

   Retrieve pcie-address from the output of the mst status -v command.

7. Define huge pages (see DOCA Flow prerequisites).

ConnectX

To enable DOCA Flow CT on the NVIDIA® ConnectX®, perform the following:

1. Configure firmware with LAG_RESOURCE_ALLOCATION=1:

   Copy

   Copied!

               
   
               
   sudo mlxconfig -d <device-id> s LAG_RESOURCE_ALLOCATION=1
           

   Info

   Retrieve device-id from the output of the mst status -v command. If, under the MST tab, the value is N/A, run the mst start command.

2. Perform power cycle.

3. If working with a single port:

   Copy

   Copied!

               
   
               
   sudo devlink dev eswitch set pci/<pcie-address> mode switchdev
   sudo devlink dev param set pci/<pcie-address> name esw_multiport value false cmode runtime
           

   Info

   Retrieve pcie-address from the output of the mst status -v command.

4. If working with two PF ports:

   Copy

   Copied!

               
   
               
   sudo devlink dev eswitch set pci/<pcie-address0> mode switchdev
   sudo devlink dev eswitch set pci/<pcie-address1> mode switchdev
   sudo devlink dev param set pci/<pcie-address0> name esw_multiport value true cmode runtime
   sudo devlink dev param set pci/<pcie-address1> name esw_multiport value true cmode runtime
           

   Info

   Retrieve pcie-address from the output of the mst status -v command.

5. Define huge pages (see DOCA Flow prerequisites).

DOCA Flow CT supports actions based on meta and NAT operations. Each action can be defined as either shared or non-shared.

Shared Actions

Actions that can be shared between entries. Shared actions are predefined and reused in multiple entries.

The user gets a handle per shared action created and uses this handle as a reference to the action where required.

Shared actions are defined using a control queue (see struct doca_flow_ct_cfg).

Non-shared Actions

Actions provided with their data during entry create/update.

These actions are completely managed by DOCA Flow CT and cannot be reused in multiple flows (i.e., NAT operations).

Action Sets in Pipe Creation

When creating a DOCA Flow CT pipe, users must define action sets, just as they would for any other pipe.

Fields in the CT pipe must be marked as CHANGEABLE during pipe creation. This allows the actual criteria for these fields to be specified later during entry creation.

During entry creation or update, different actions can be specified for each direction, allowing variations in action content and/or action type.

Feature Enable

To enable user actions, configure the following parameters:

• User action templates during DOCA Flow CT pipe creation

• Maximum number of user actions (nb_user_actions on DOCA Flow CT init)

Using Actions in Autonomous Mode

Init

Configure the following parameters on doca_flow_ct_init():

• nb_ctrl_queues – number of control queues for defining shared actions

• nb_user_actions – maximum number of actions (shared and non-shared)

• worker_cb – callbacks required to communicate with the user

Create DOCA Flow CT Pipe

Configure actions sets on doca_flow_pipe_create().

Create Shared Actions

Use doca_flow_ct_actions_add_shared() with one of the control queues.

Shared actions can be added at any time before use.

Implement Worker Callbacks

Callbacks are called from each worker thread to acquire synchronization with the user code and on the first packet of a flow.

On doca_flow_ct_rule_pkt_cb:

• Determine how the packet should be treated

• If rules are required, return the actions handles to use

Using Actions in Managed Mode

Init

Configure the following parameters on doca_flow_ct_init():

• nb_ctrl_queues – number of control queues for defining shared actions

• nb_user_actions – maximum number of user actions. Must align to 64. Both shared control queues and non-shared control queues cache action IDs to speed up ID allocation. Each queue may cache a maximum of 1024 IDs. Users must configure the expected number of actions + total queues * 1024. This number cannot exceed the number of actions hardware supports.

Create DOCA Flow CT Pipe

Configure actions sets on doca_flow_pipe_create().

Create Shared Actions

Use doca_flow_ct_actions_add_shared() with one of the control queues.

Shared actions can be added at any time before use.

Add Entry

Entry can be created in one of the following ways:

• Using an action handle of a predefined shared action

• Using action data, which is specific to the flow, not sharable (e.g., for NAT operations)

The entry can have different actions and/or different action types per direction.

Remove Entry

Non-shared actions associated with an entry are implicitly destroyed by DOCA Flow CT.

Shared actions are not destroyed. They can be used by the user until they decide to remove them.

Update Entry

Entry actions can be updated per direction. All combinations of shared/non-shared actions are applicable (e.g., update from shared to non-shared).

DOCA Flow CT allows using a different forward pipe per flow direction.

DOCA Flow CT supports the forward pipe in two levels:

• Pipe level – a single forward pipe defined during DOCA Flow CT pipe creation and used for all entries

• Entry level – forward pipe defined during entry create

• DOCA Flow CT operates in one of the two levels

DOCA CT forward in entry level has the following characteristics:

• Supports only DOCA_FLOW_FWD_PIPE (up to 4 different forward pipes)

• Supports forward pipe per flow direction (both directions can have same/different forward pipe)

• Must set forward pipes on each entry create (no default forward pipe)

Turn on the feature:

1. Create DOCA Flow CT pipe with forward type = DOCA_FLOW_FWD_PIPE and next_pipe = NULL .

2. Call to doca_flow_ct_fwd_register to register forward pipes and get fwd_handles in return.

Using Changeable Forward in Managed Mode

1. Initialize DOCA Flow CT (doca_flow_ct_init​).

2. Register forward pipes (doca_flow_ct_fwd_register​).

   • Define pipes that can be used for forward​

3. Create DOCA Flow CT pipe​ (doca_flow_pipe_create) w ith definition of possible forward pipes.

4. Add entry (doca_flow_ct_add_entry).

   • Set origin and/or reply fwd_handles returned from doca_flow_ct_fwd_register.

5. Update forward for entry direction (doca_flow_ct_update_entry​).

   Note

   Updating forward handle requires setting all other parameters with their previous values.​

Using Changeable Forward in Autonomous Mode

1. Initialize DOCA Flow CT (doca_flow_ct_init​).

2. Register forward pipes (doca_flow_ct_fwd_register​).

   • Define pipes that can be used for forward.

3. Create DOCA Flow CT pipe​ (doca_flow_pipe_create) w ith definition of possible forward pipes.

4. CT workers start to handle traffic.​

5. On the first flow packet, doca_flow_ct_rule_pkt callback​ is called.

   • In this callback, determine if the entry should be created, and which actions and/or forward handles should be used for this entry.

For the library API reference, refer to DOCA Flow and CT API documentation in the DOCA Library APIs.

The following sections provide additional details about the library API.

enum doca_flow_ct_flags

DOCA Flow CT configuration optional flags.

enum doca_flow_ct doca_flow_ct_entry_flags

DOCA Flow CT Entry optional flags.

enum doca_flow_ct_rule_opr

Options for handling flows in autonomous mode with shared actions. The decision is taken on the first flow packet.

struct direction_cfg

Managed mode configuration for origin or reply direction.

struct doca_flow_ct_worker_callbacks

Set of callbacks for using shared actions in autonomous mode.

struct doca_flow_ct_cfg

DOCA Flow CT configuration.

            

            
uint32_t nb_arm_queues;
uint32_t nb_ctrl_queues;
uint32_t nb_user_actions;
uint32_t nb_arm_sessions[DOCA_FLOW_CT_SESSION_MAX];
uint32_t flags;
uint16_t aging_core;
uint16_t aging_query_delay_s;
doca_flow_ct_flow_log_cb flow_log_cb;
struct doca_flow_ct_aging_ops *aging_ops;
uint32_t base_core_id;
uint32_t dup_filter_sz;
union {
                /* Managed mode configuration for origin and reply direction. */
                struct direction_cfg direction[2];
 
                /* Below fields are dedicate for autonomous mode */
                struct {
 						uint16_t tcp_timeout_s;
  						uint16_t tcp_session_del_s;
  						uint16_t udp_timeout_s;
                        enum doca_flow_tun_type tunnel_type;
                        uint16_t vxlan_dst_port;
                        enum doca_flow_ct_hash_type hash_type;
                        uint32_t meta_user_bits;
                        uint32_t meta_action_bits;
                        struct doca_flow_meta *meta_zone_mask;
                        struct doca_flow_meta *connection_id_mask;
                        struct doca_flow_ct_worker_callbacks worker_cb;
               };
       };
        

Where:

struct doca_flow_ct_actions

This structure is used in the following cases:

• For defining shared actions. In this case, action data is provided by the user. The action handle is returned by DOCA Flow CT.

• For defining an entry with actions. The structure can be filled with two options:

  • With action handle of a previously created shared action

  • With non-shared action data

DOCA Flow CT action structure.

            

            
enum doca_flow_resource_type  resource_type;
union {
			   /* Used when creating an entry with a shared action. */
               uint32_t action_handle;
 
               /* Used when creating an entry with non-shared action or when creating a shared action. */
               struct {   
                          uint32_t action_idx;
                          struct doca_flow_meta meta;
                          struct doca_flow_header_l4_port l4_port;
                          union {
                                    struct doca_flow_ct_ip4 ip4;
                                    struct doca_flow_ct_ip6 ip6;
                           };
               } data;
       };
        

Where: