DOCA Flow
This guide describes how to deploy the DOCA Flow library, the philosophy of the DOCA Flow API, and how to use it. The guide is intended for developers writing network function applications that focus on packet processing (such as gateways). It assumes familiarity with the network stack and DPDK.
DOCA Flow is the most fundamental API for building generic packet processing pipes in hardware. The DOCA Flow library provides an API for building a set of pipes, where each pipe consists of match criteria, monitoring, and a set of actions. Pipes can be chained so that after a pipe-defined action is executed, the packet may proceed to another pipe.
Using DOCA Flow API, it is easy to develop hardware-accelerated applications that have a match on up to two layers of packets (tunneled).
MAC/VLAN/ETHERTYPE
IPv4/IPv6
TCP/UDP/ICMP
GRE/VXLAN/GTP-U/ESP/PSP
Metadata
The execution pipe can include packet modification actions such as the following:
Modify MAC address
Modify IP address
Modify L4 (ports)
Strip tunnel
Add tunnel
Set metadata
Encrypt/Decrypt
The execution pipe can also have monitoring actions such as the following:
Count
Policers
The pipe also has a forwarding target which can be any of the following:
Software (RSS to subset of queues)
Port
Another pipe
Drop packets
A DOCA Flow-based application can run either on the host machine or on an NVIDIA® BlueField® DPU target. Flow-based programs require an allocation of huge pages, hence the following commands are required:
echo '1024' | sudo tee -a /sys/kernel/mm/hugepages/hugepages-2048kB/nr_hugepages
sudo mkdir /mnt/huge
sudo mount -t hugetlbfs nodev /mnt/huge
On some operating systems (RockyLinux, OpenEuler, CentOS 8.2) the default huge page size on the DPU (and Arm hosts) is larger than 2MB, often 512MB. Users can check the size of the huge pages on their OS using the following command:
$ grep -i huge /proc/meminfo
AnonHugePages: 0 kB
ShmemHugePages: 0 kB
FileHugePages: 0 kB
HugePages_Total: 4
HugePages_Free: 4
HugePages_Rsvd: 0
HugePages_Surp: 0
Hugepagesize: 524288 kB
Hugetlb: 6291456 kB
In this case, instead of allocating 1024 pages, users should only allocate 4:
echo '4' | sudo tee -a /sys/kernel/mm/hugepages/hugepages-524288kB/nr_hugepages
The following diagram shows how the DOCA Flow library defines a pipe template, receives a packet for processing, creates the pipe entry, and offloads the flow rule in NIC hardware.
Features of DOCA Flow:
User-defined set of matches parser and actions
DOCA Flow pipes can be created or destroyed dynamically
Packet processing is fully accelerated by hardware with a specific entry in a flow pipe
Packets that do not match any of the pipe entries in hardware can be sent to Arm cores for exception handling and then reinjected back to hardware
The DOCA Flow pipe consists of the following components:
Monitor (MON in the diagram) - counts, meters, or mirrors
Modify (MDF in the diagram) - modifies a field
Forward (FWD in the diagram) - forwards to the next stage in packet processing
DOCA Flow organizes pipes into high-level containers named domains to address the specific needs of the underlying architecture.
A key element in defining a domain is the packet direction and a set of allowed actions.
A domain is a pipe attribute (also relates to shared objects)
A domain restricts the set of allowed actions
Transition between domains is well-defined (packets cannot cross domains arbitrarily)
A domain may restrict the sharing of objects between packet directions
Packet direction can restrict the move between domains
List of Steering Domains
DOCA Flow provides the following set of predefined steering domains:
|
Domain |
Description |
|
DOCA_FLOW_PIPE_DOMAIN_DEFAULT |
|
|
DOCA_FLOW_PIPE_DOMAIN_SECURE_INGRESS |
|
|
DOCA_FLOW_PIPE_DOMAIN_EGRESS |
|
|
DOCA_FLOW_PIPE_DOMAIN_SECURE_EGRESS |
|
Domains in VNF Mode
Domains in Switch Mode
You can find more detailed information on DOCA Flow API in NVIDIA DOCA Library APIs.
The pkg-config (*.pc file) for the DOCA Flow library is doca-flow.
The following sections provide additional details about the library API.
doca_flow_cfg
This structure is required input for the DOCA Flow global initialization function doca_flow_init. The user must create and set this structure's fields using the API doca_error_t doca_flow_cfg_create(struct doca_flow_cfg **cfg).
doca_error_t doca_flow_cfg_create(struct doca_flow_cfg **cfg);
cfg – DOCA Flow configuration structure
This function creates and allocates the DOCA Flow configuration structure.
doca_error_t doca_flow_cfg_destroy(struct doca_flow_cfg *cfg);
cfg – DOCA Flow configuration structure
This function destroys and frees the DOCA Flow configuration structure.
doca_error_t doca_flow_cfg_set_pipe_queues(struct doca_flow_cfg *cfg, uint16_t pipe_queues);
cfg – DOCA Flow configuration structure
pipe_queues – The pipe's number of queues for each offload thread
This function sets the number of hardware acceleration control queues. It is expected for the same core to always use the same queue_id. In cases where multiple cores access the API using the same queue_id, the application must use locks between different cores/threads.
doca_error_t doca_flow_cfg_set_nr_counters(struct doca_flow_cfg *cfg, uint32_t nr_counters);
cfg – DOCA Flow configuration structure
nr_counters – The number of counters to configure
This function sets the number of regular (non-shared) counters to configure.
doca_error_t doca_flow_cfg_set_nr_meters(struct doca_flow_cfg *cfg, uint32_t nr_meters);
cfg – DOCA Flow configuration structure
nr_meters – number of traffic meters to configure
This function sets the number of regular (non-shared) traffic meters to configure.
doca_error_t doca_flow_cfg_set_nr_acl_collisions(struct doca_flow_cfg *cfg, uint8_t nr_acl_collisions);
cfg – DOCA Flow configuration structure
nr_acl_collisions – The number of pre-configured collisions
This function sets the number of collisions for the ACL module.
Maximum value is 8
If not set, then default value is 3
doca_error_t doca_flow_cfg_set_mode_args(struct doca_flow_cfg *cfg, const char *mode_args);
cfg – DOCA Flow configuration structure
mode_args – The DOCA Flow architecture mode
This function sets the DOCA Flow architecture mode.
doca_error_t doca_flow_cfg_set_nr_shared_resource(struct doca_flow_cfg *cfg, uint32_t nr_shared_resource, enum doca_flow_shared_resource_type type);
cfg – DOCA Flow configuration structure
nr_shared_resource – Number of shared resource
type – Shared resource type
This function sets the number of shared resource per type:
Type DOCA_FLOW_SHARED_RESOURCE_METER – number of meters that can be shared among flows
Type DOCA_FLOW_SHARED_RESOURCE_COUNT – number of counters that can be shared among flows
Type DOCA_FLOW_SHARED_RESOURCE_RSS – number of RSS that can be shared among flows
Type DOCA_FLOW_SHARED_RESOURCE_IPSEC_SA – number of IPSec SAs actions that can be shared among flows
Type DOCA_FLOW_SHARED_RESOURCE_PSP – number of PSP actions that can be shared among flows
See "Shared Counter Resource" section for more information.
doca_error_t doca_flow_cfg_set_queue_depth(struct doca_flow_cfg *cfg, uint32_t queue_depth);
cfg – DOCA Flow configuration structure
queue_depth – The number of pre-configured queue size
This function sets the n umber of flow rule operations a queue can hold.
This value is preconfigured at port start ( queue_size).
Default value is 128
Passing 0 or not setting the value would set the default value
Maximum value is 1024
doca_error_t doca_flow_cfg_set_cb_pipe_process(struct doca_flow_cfg *cfg, doca_flow_pipe_process_cb cb);
cfg – DOCA Flow configuration structure
cb – Callback for pipe process completion
This function sets the callback function for the entry to be called during doca_flow_entries_process to complete entry operation (i.e., add, update, delete, and aged).
doca_error_t doca_flow_cfg_set_cb_entry_process(struct doca_flow_cfg *cfg, doca_flow_entry_process_cb cb);
cfg – DOCA Flow configuration structure
cb – Callback for entry create/destroy
This function sets the callback for creating/destroying an entry.
doca_error_t doca_flow_cfg_set_cb_shared_resource_unbind(struct doca_flow_cfg *cfg, doca_flow_shared_resource_unbind_cb cb);
cfg – DOCA Flow configuration structure
cb – Callback for unbinding of a shared resource
This function sets the callback for unbinding of a shared resource.
doca_error_t doca_flow_cfg_set_rss_key(struct doca_flow_cfg *cfg, const uint8_t *rss_key, uint32_t rss_key_len);
cfg – DOCA Flow configuration structure
rss_key – RSS hash key
rss_key_len – RSS hash key length (in bytes)
This function sets the RSS key used by default RSS rules.
If not set, the underlying driver's default RSS key is used.
doca_error_t doca_flow_cfg_set_default_rss(struct doca_flow_cfg *cfg, const struct doca_flow_resource_rss_cfg *rss);
cfg – DOCA Flow configuration structure
rss – RSS global configuration
This function sets the global RSS configuration for all ports. It is overridden by the port's RSS configuration. This configuration is used at port start which, in non-isolated mode, creates default RSS rules to forward incoming packets to traffic receive queues (i.e., RXQ) created by the user.
If users creates their own hairpin queues (i.e., doca_flow_port_cfg.dev is not set in "switch" mode), RSS configuration must also be set to reflect the number of traffic receive queues and calculate the correct hairpin queue index internally. Otherwise, packets would not go to the correct hairpin queue.
doca_flow_entry_process_cb
typedef void (*doca_flow_entry_process_cb)(struct doca_flow_pipe_entry *entry,
uint16_t pipe_queue, enum doca_flow_entry_status status,
enum doca_flow_entry_op op, void *user_ctx);
entry [in] – p ointer to pipe entry
pipe_queue [in] – q ueue identifier
status [in] – entry processing status (see doca_flow_entry_status)
op [in] – entry's operation, defined in the following enum:
DOCA_FLOW_ENTRY_OP_ADD – Add entry operation
DOCA_FLOW_ENTRY_OP_DEL – Delete entry operation
DOCA_FLOW_ENTRY_OP_UPD – Update entry operation
DOCA_FLOW_ENTRY_OP_AGED – Aged entry operation
user_ctx [in] – user context as provided to doca_flow_pipe_add_entry
NoteUser context is set once to the value provided to doca_flow_pipe_add_entry (or to any doca_flow_pipe_*_add_entry variant) as the usr_ctx parameter, and is then reused in subsequent callback invocation for all operations. This user context must remain available for all potential invocations of the callback depending on it, as it is memorized as part of the entry and provided each time.
shared_resource_unbind_cb
typedef void (*shared_resource_unbind_cb)(enum engine_shared_resource_type type,
uint32_t shared_resource_id,
struct engine_bindable *bindable);
type [in] – engine shared resource type. Supported types are: meter, counter, rss, crypto, mirror.
shared_resource_id [in] – shared resource; unique ID
bindable [in] – pointer to bindable object (e.g., port, pipe)
doca_flow_resource_rss_cfg
struct doca_flow_resource_rss_cfg {
uint32_t outer_flags;
uint32_t inner_flags;
uint16_t *queues_array;
int nr_queues;
enum doca_flow_rss_hash_function rss_hash_func;
};
outer_flags – RSS offload type on the outermost packet
inner_flags – RSS offload type on the innermost packet
queues_array – pointer to receive queues ID (i.e., [0, 1, 2, 3])
nr_queues – number of receive queues ID (i.e., 4)
rss_hash_func – RSS hash type, DOCA_FLOW_RSS_HASH_FUNCTION_TOEPLITZ or DOCA_FLOW_RSS_HASH_FUNCTION_SYMMETRIC_TOEPLITZ
Noteouter_flags , inner_flags , rss_hash_func are ignored by default RSS rules which use the driver's default values.
doca_flow_port_cfg
This struct is required input for the DOCA Flow port initialization function, doca_flow_port_start. The user must create and set this structure's fields using the following API:
doca_error_t doca_flow_port_cfg_create(struct doca_flow_port_cfg **cfg);
cfg – DOCA Flow port configuration structure
This function creates and allocates the DOCA Flow port configuration structure.
doca_error_t doca_flow_port_cfg_destroy(struct doca_flow_port_cfg *cfg);
cfg – DOCA Flow port configuration structure
This function destroys and frees the DOCA Flow port configuration structure.
doca_error_t doca_flow_port_cfg_set_devargs(struct doca_flow_port_cfg *cfg, const char *devargs);
cfg – DOCA Flow port configuration structure
devargs – A string containing specific configuration port configurations
This function sets specific port configurations.
For usage information of the type and devargs fields, refer to section "Start Point".
doca_error_t doca_flow_port_cfg_set_priv_data_size(struct doca_flow_port_cfg *cfg, uint16_t priv_data_size);
cfg – DOCA Flow port configuration structure
priv_data_size – User private data size
This function sets the user's private data size. Per port, users may define private data where application-specific info can be stored.
doca_error_t doca_flow_port_cfg_set_dev(struct doca_flow_port_cfg *cfg, void *dev);
cfg – DOCA Flow port configuration structure
dev – Device
This function sets the port's doca_dev.
Used to create internal hairpin resource for switch mode and crypto resources
Mandatory for switch mode and to use PSP or IPsec SA shared resources
doca_error_t doca_flow_port_cfg_set_rss_cfg(struct doca_flow_port_cfg *cfg, const struct doca_flow_resource_rss_cfg *rss_cfg);
cfg – DOCA Flow port configuration structure
rss_cfg – RSS configuration
This function sets the port's RSS configuration (optional) used by default RSS rules of this port.
This configuration overrides global RSS configuration in doca_flow_cfg.
This configuration is used to create default RSS rules of this port to forward packets to traffic receive queues (i.e., RXQ) in non-isolated mode.
doca_flow_pipe_cfg
This is a pipe configuration that contains the user-defined template for the packet process. The user must create and set this structure's fields using the following API:
doca_error_t doca_flow_pipe_cfg_create(struct doca_flow_pipe_cfg **cfg, struct doca_flow_port *port);
cfg – DOCA Flow port configuration structure
port – DOCA Flow port
This function creates and allocates the DOCA Flow port configuration structure. It also sets the port of the pipeline.
The port here is a final parameter (i.e., to set another port, the user must first destroy the current pipe configuration).
doca_error_t doca_flow_pipe_cfg_destroy(struct doca_flow_pipe_cfg *cfg);
cfg – DOCA Flow port configuration structure
This function destroys and frees the DOCA Flow pipe configuration structure.
doca_error_t doca_flow_pipe_cfg_set_match(struct doca_flow_pipe_cfg *cfg, const struct doca_flow_match *match, const struct doca_flow_match *match_mask);
cfg – DOCA Flow port configuration structure
match – DOCA Flow match
match_mask – DOCA Flow match mask
This function sets the matcher and the match mask for the pipeline.
Note that:
match is for all pipe types except for DOCA_FLOW_PIPE_HASH where it is ignored
match_mask is only for pipes of types DOCA_FLOW_PIPE_BASIC, DOCA_FLOW_PIPE_CONTROL, DOCA_FLOW_PIPE_HASH, and DOCA_FLOW_PIPE_ORDERED_LIST
If one of the fields is not relevant, user should pass NULL instead.
doca_error_t doca_flow_pipe_cfg_set_actions(struct doca_flow_pipe_cfg *cfg, struct doca_flow_actions *const *actions, struct doca_flow_actions *const *actions_masks, struct doca_flow_action_descs *const *action_descs, size_t nr_actions);
cfg – DOCA Flow pipe configuration structure
actions – DOCA Flow actions array
actions_masks – DOCA Flow actions mask array
action_descs – DOCA Flow actions descriptor array
nr_actions – Number of actions
This function sets the actions, actions mask, and actions descriptor for the pipeline.
Note that:
actions is only for pipes of type DOCA_FLOW_PIPE_BASIC and DOCA_FLOW_PIPE_HASH
actions_masks is only for pipes of type DOCA_FLOW_PIPE_BASIC and DOCA_FLOW_PIPE_HASH
action_descs is only for pipes of type DOCA_FLOW_PIPE_BASIC, DOCA_FLOW_PIPE_CONTROL, and DOCA_FLOW_PIPE_HASH
If one of the fields is not relevant, user should pass NULL instead.
doca_error_t doca_flow_pipe_cfg_set_monitor(struct doca_flow_pipe_cfg *cfg, const struct doca_flow_monitor *monitor);
cfg – DOCA Flow port configuration structure
monitor – DOCA Flow monitor
This function sets the monitor for the pipeline.
Note that the monitor is only for pipes of type DOCA_FLOW_PIPE_BASIC, DOCA_FLOW_PIPE_CONTROL, and DOCA_FLOW_PIPE_HASH.
doca_error_t doca_flow_pipe_cfg_set_ordered_lists(struct doca_flow_pipe_cfg *cfg, struct doca_flow_ordered_list *const *ordered_lists, size_t nr_ordered_lists);
cfg – DOCA Flow port configuration structure
ordered_lists – DOCA Flow ordered lists array
nr_ordered_lists – Number of ordered lists
This function sets an array of ordered list types for the pipeline.
struct doca_flow_ordered_list {
uint32_t idx;
uint32_t size;
const void **elements;
enum doca_flow_ordered_list_element_type *types;
};
idx – List index among the lists of the pipe
At pipe creation, it must match the list position in the array of lists
At entry insertion, it determines which list to use
size – Number of elements in the list
elements – An array of DOCA Flow structure pointers, depending on the types
types – Types of DOCA Flow structures each of the elements is pointing to. This field includes the following ordered list element types:
DOCA_FLOW_ORDERED_LIST_ELEMENT_ACTIONS – Ordered list element is struct doca_flow_actions. The next element is struct doca_flow_action_descs which is associated with the current element.
DOCA_FLOW_ORDERED_LIST_ELEMENT_ACTION_DESCS – Ordered list element is struct doca_flow_action_descs. If the previous element type is ACTIONS, the current element is associated with it. Otherwise, the current element is ordered with regards to the previous one.
DOCA_FLOW_ORDERED_LIST_ELEMENT_MONITOR – Ordered list element is struct doca_flow_monitor
Note that the ordered lists are only for pipes of type DOCA_FLOW_PIPE_ORDERED_LIST.
doca_error_t doca_flow_pipe_cfg_set_name(struct doca_flow_pipe_cfg *cfg, const char *name);
cfg – DOCA Flow port configuration structure
name – Pipe name
This function sets the name of the pipeline.
doca_error_t doca_flow_pipe_cfg_set_type(struct doca_flow_pipe_cfg *cfg, enum doca_flow_pipe_type type);
cfg – DOCA Flow port configuration structure
type – DOCA Flow pipe type
This function sets the type of the pipeline, it includes the following pipe types:
DOCA_FLOW_PIPE_BASIC – Flow pipe
DOCA_FLOW_PIPE_CONTROL – Control pipe
DOCA_FLOW_PIPE_LPM – LPM pipe
DOCA_FLOW_PIPE_ACL – ACL pipe
DOCA_FLOW_PIPE_ORDERED_LIST – Ordered list pipe
DOCA_FLOW_PIPE_HASH – Hash pipe
If not set, then by default, the pipeline's type is DOCA_FLOW_PIPE_BASIC:
doca_error_t doca_flow_pipe_cfg_set_domain(struct doca_flow_pipe_cfg *cfg, enum doca_flow_pipe_domain domain);
cfg – DOCA Flow port configuration structure
domain – DOCA Flow pipe steering domain
This function sets the steering domain of the pipeline, it includes the following domains:
DOCA_FLOW_PIPE_DOMAIN_DEFAULT – Default pipe domain for actions on ingress traffic
DOCA_FLOW_PIPE_DOMAIN_SECURE_INGRESS – Pipe domain for secure actions on ingress traffic
DOCA_FLOW_PIPE_DOMAIN_EGRESS – Pipe domain for actions on egress traffic
DOCA_FLOW_PIPE_DOMAIN_SECURE_EGRESS – Pipe domain for actions on egress traffic
If not set, then by default, the pipeline's steering domain is DOCA_FLOW_PIPE_DOMAIN_DEFAULT.
doca_error_t doca_flow_pipe_cfg_set_is_root(struct doca_flow_pipe_cfg *cfg, bool is_root);
cfg – DOCA Flow pipe configuration structure
is_root – If the pipe is root
This function determines whether the pipeline is root. If true, then the pipe is a root pipe executed on packet arrival. If not set, then by default, the pipeline is not root.
Only one root pipe is allowed per port of any type.
doca_error_t doca_flow_pipe_cfg_set_nr_entries(struct doca_flow_pipe_cfg *cfg, uint32_t nr_entries);
cfg – DOCA Flow port configuration structure
nr_entries – Maximum number of flow rules
This function sets the pipeline's maximum number of flow rules. If not set, then by default, the maximum number of flow rules is 8k.
doca_error_t doca_flow_pipe_cfg_set_is_resizable(struct doca_flow_pipe_cfg *cfg, bool is_resizable);
cfg – DOCA Flow port configuration structure
is_resizable – If the pipe is resizable
This function determines whether the pipeline supports the resize operation. If not set, then by default, the pipeline does not support the resize operation.
doca_error_t doca_flow_pipe_cfg_set_enable_strict_matching(struct doca_flow_pipe_cfg *cfg, bool enable_strict_matching);
cfg – DOCA Flow port configuration structure
enable_strict_matching – If the pipe supports strict matching
This function determines whether the pipeline supports strict matching.
If true, relaxed matching (enabled by default) is disabled for this pipe
If not set, then by default, the pipeline doesn't support strict matching
doca_error_t doca_flow_pipe_cfg_set_dir_info(struct doca_flow_pipe_cfg *cfg, enum doca_flow_direction_info dir_info);
cfg – DOCA Flow port configuration structure
dir_info – DOCA Flow direction info
This function sets the pipeline's direction information:
DOCA_FLOW_DIRECTION_BIDIRECTIONAL – Default for traffic in both directions
DOCA_FLOW_DIRECTION_NETWORK_TO_HOST – Network to host traffic
DOCA_FLOW_DIRECTION_HOST_TO_NETWORK – Host to network traffic
dir_info is supported in Switch Mode only.
dir_info is optional. It can provide potential optimization at the driver layer. Configuring the direction information properly optimizes the traffic steering.
doca_error_t doca_flow_pipe_cfg_set_miss_counter(struct doca_flow_pipe_cfg *cfg, bool miss_counter);
cfg – DOCA Flow port configuration structure
miss_counter – If to enable miss counter
This function determines whether to add a miss counter to the pipe. If true, then the pipe would have a miss counter and the user can query it using doca_flow_query_pipe_miss. If not set, then by default, the miss counter is disabled.
Miss counter may impact performance and should be avoided if not required by the application.
doca_error_t doca_flow_pipe_cfg_set_congestion_level_threshold(struct doca_flow_pipe_cfg *cfg, uint8_t congestion_level_threshold);
cfg – DOCA Flow port configuration structure
congestion_level_threshold – Congestion level threshold
This function sets the congestion threshold for the pipe in percentage (0,100]. If not set, then by default, the congestion threshold is zero.
doca_error_t doca_flow_pipe_cfg_set_user_ctx(struct doca_flow_pipe_cfg *cfg, void *user_ctx);
cfg – DOCA Flow port configuration structure
user_ctx – User context
This function sets the pipeline's user context.
doca_flow_parser_geneve_opt_cfg
This is a parser configuration that contains the user-defined template for a single GENEVE TLV option.
struct doca_flow_parser_geneve_opt_cfg {
enum doca_flow_parser_geneve_opt_mode match_on_class_mode;
doca_be16_t option_class;
uint8_t option_type;
uint8_t option_len;
doca_be32_t data_mask[DOCA_FLOW_GENEVE_DATA_OPTION_LEN_MAX];
};
match_on_class_mode – role of option_class in this option (enum doca_flow_parser_geneve_opt_mode). This field includes the following class modes:
DOCA_FLOW_PARSER_GENEVE_OPT_MODE_IGNORE – class is ignored, its value is neither part of the option identifier nor changeable per pipe/entry
DOCA_FLOW_PARSER_GENEVE_OPT_MODE_FIXED – class is fixed (the class defines the option along with the type)
DOCA_FLOW_PARSER_GENEVE_OPT_MODE_MATCHABLE – class is the field of this option; different values can be matched for the same option (defined by type only)
option_class – option class ID (must be set when class mode is fixed)
option_type – option type
option_len – length of the option data (in 4-byte granularity)
data_mask – mask for indicating which dwords (DWs) should be configured on this option
NoteThis is not a bit mask. Each DW can contain either 0xffffffff for configure or 0x0 for ignore. Other values are not valid.
doca_flow_meta
There is a maximum DOCA_FLOW_META_MAX-byte scratch area which exists throughout the pipeline.
The user can set a value to metadata, copy from a packet field, then match in later pipes. Mask is supported in both match and modification actions.
The user can modify the metadata in different ways based on its actions' masks or descriptors:
ADD – set metadata scratch value from a pipe action or an action of a specific entry. Width is specified by the descriptor.
COPY – copy metadata scratch value from a packet field (including the metadata scratch itself). Width is specified by the descriptor.
NoteIn a real application, it is encouraged to create a union of doca_flow_meta defining the application's scratch fields to use as metadata.
struct doca_flow_meta { union { uint32_t pkt_meta; /**< Shared with application via packet. */ struct { uint32_t lag_port :2; /**< Bits of LAG member port. */ uint32_t type :2; /**< 0: traffic 1: SYN 2: RST 3: FIN. */ uint32_t zone :28; /**< Zone ID for CT processing. */ } ct; }; uint32_t u32[DOCA_FLOW_META_MAX / 4 - 1]; /**< Programmable user data. */ uint32_t mark; /**< Mark id. */ };
pkt_meta – Metadata can be received along with the packet
u32[] – Scratch are u32[]
u32[0] contains the IPsec syndrome in the lower 8 bits if the packet passes the pipe with IPsec crypto action configured in full offload mode:
0 – signifies a successful IPsec operation on the packet
1 – bad replay. Ingress packet sequence number is beyond anti-reply window boundaries.
u32[1] contains the IPsec packet sequence number (lower 32 bits) if the packet passes the pipe with IPsec crypto action configured in full offload mode
mark – O ptional parameter that may be communicated to the software. If it is set and the packet arrives to the software, the value can be examined using the software API.
When DPDK is used, MARK is placed on the struct rte_mbuf (see "Action: MARK" section in official DPDK documentation )
When the Kernel is used, MARK is placed on the struct sk_buff 's MARK field
Some DOCA pipe types (or actions) use several bytes in the scratch area for internal usage. So, if the user has set these bytes in PIPE-1 and read them in PIPE-2, and between PIPE-1 and PIPE-2 there is PIPE-A which also uses these bytes for internal purpose, then these bytes are overwritten by the PIPE-A. This must be considered when designing the pipe tree.
The bytes used in the scratch area are presented by pipe type in the following table:
|
Pipe Type/Action |
Bytes Used in Scratch |
|
orderd_list |
[0, 1, 2, 3] |
|
LPM |
[0, 1, 2, 3] |
|
mirror |
[0, 1, 2, 3] |
|
ACL |
[0, 1, 2, 3] |
|
Fwd from ingress to egress |
[0, 1, 2, 3] |
doca_flow_parser_meta
This structure contains all metadata information which hardware extracts from the packet.
These fields contain read-only hardware data which can be used to match on.
struct doca_flow_parser_meta {
uint32_t port_meta;
uint16_t random;
uint8_t ipsec_syndrome;
uint8_t psp_syndrome;
enum doca_flow_meter_color meter_color;
enum doca_flow_l2_meta outer_l2_type;
enum doca_flow_l3_meta outer_l3_type;
enum doca_flow_l4_meta outer_l4_type;
enum doca_flow_l2_meta inner_l2_type;
enum doca_flow_l3_meta inner_l3_type;
enum doca_flow_l4_meta inner_l4_type;
uint8_t outer_ip_fragmented;
uint8_t inner_ip_fragmented;
uint8_t outer_l3_ok;
uint8_t outer_ip4_checksum_ok;
uint8_t outer_l4_ok;
uint8_t outer_l4_checksum_ok;
uint8_t inner_l3_ok;
uint8_t inner_ip4_checksum_ok;
uint8_t inner_l4_ok;
uint8_t inner_l4_checksum_ok;
};
port_meta – Programmable source vport.
random – Random value to match regardless to packet data/headers content. Application should not assume that this value is kept during the lifetime of the packet. It holds a different random value for each matching.
NoteWhen random matching is used for sampling, the number of entries in the pipe must be 1 (doca_flow_pipe_attr.nb_flows = 1).
ipsec_syndrome – IPsec decrypt/authentication syndrome, Valid syndromes:
DOCA_FLOW_CRYPTO_SYNDROME_OK- IPsec decryption and authentication success
DOCA_FLOW_CRYPTO_ICV_FAIL - IPsec authentication failure
DOCA_FLOW_CRYPTO_BAD_TRAILER - IPsec trailer length exceeded ESP payload
psp_syndrome – PSP decrypt/authentication syndrome, Valid syndromes:
DOCA_FLOW_CRYPTO_SYNDROME_OK- PSP decryption and authentication success
DOCA_FLOW_CRYPTO_ICV_FAIL - PSP authentication failure
DOCA_FLOW_CRYPTO_BAD_TRAILER - PSP trailer overlaps with headers
meter_color – Meter colors (enum doca_flow_meter_color). Valid colors:
DOCA_FLOW_METER_COLOR_GREEN – Meter marking packet color as green
DOCA_FLOW_METER_COLOR_YELLOW – Meter marking packet color as yellow
DOCA_FLOW_METER_COLOR_RED – Meter marking packet color as red.
outer_l2_type – Outer L2 packet type (enum doca_flow_l2_meta). Valid L2 types:
DOCA_FLOW_L2_META_NO_VLAN – No VLAN present
DOCA_FLOW_L2_META_SINGLE_VLAN – Single VLAN present
DOCA_FLOW_L2_META_MULTI_VLAN – Multiple VLAN present
outer_l3_type – Outer L3 packet type (enum doca_flow_l3_meta). Valid L3 types:
DOCA_FLOW_L3_META_NONE – L3 type is none of the below
DOCA_FLOW_L3_META_IPV4 – L3 type is IPv4
DOCA_FLOW_L3_META_IPV6 – L3 type is IPv6
outer_l4_type – Outer L4 packet type (enum doca_flow_l4_meta). Valid L4 types:
DOCA_FLOW_L4_META_NONE – L4 type is none of the below
DOCA_FLOW_L4_META_TCP – L4 type is TCP
DOCA_FLOW_L4_META_UDP – L4 type is UDP
DOCA_FLOW_L4_META_ICMP – L4 type is ICMP or ICMPv6
DOCA_FLOW_L4_META_ESP – L4 type is ESP
inner_l2_type – Inner L2 packet type (enum doca_flow_l2_meta). Valid L2 types:
DOCA_FLOW_L2_META_NO_VLAN – No VLAN present
DOCA_FLOW_L2_META_SINGLE_VLAN – Single VLAN present
DOCA_FLOW_L2_META_MULTI_VLAN – Multiple VLAN present
inner_l3_type – Inner L3 packet type (enum doca_flow_l3_meta). Valid L3 types:
DOCA_FLOW_L3_META_NONE – L3 type is none of the below
DOCA_FLOW_L3_META_IPV4 – L3 type is IPv4
DOCA_FLOW_L3_META_IPV6 – L3 type is IPv6
inner_l4_type – Inner L4 packet type (enum doca_flow_l4_meta). Valid L4 types:
DOCA_FLOW_L4_META_NONE – L4 type is none of the below
DOCA_FLOW_L4_META_TCP – L4 type is TCP
DOCA_FLOW_L4_META_UDP – L4 type is UDP
DOCA_FLOW_L4_META_ICMP – L4 type is ICMP or ICMPv6
DOCA_FLOW_L4_META_ESP – L4 type is ESP
outer_ip_fragmented – Whether outer IP packet is fragmented
inner_ip_fragmented – Whether inner IP packet is fragmented
outer_l3_ok – Whether outer network layer is valid regardless to IPv4 checksum
outer_ip4_checksum_ok – Whether outer IPv4 checksum is valid, packets without outer IPv4 header are taken as invalid checksum
outer_l4_ok – Whether outer transport layer is valid including L4 checksum
outer_l4_checksum_ok – Whether outer transport layer checksum is valid. Packets without outer TCP/UDP header are taken as invalid checksum.
inner_l3_ok – Whether inner network layer is valid regardless to IPv4 checksum
inner_ip4_checksum_ok – Whether inner IPv4 checksum is valid. Packets without inner IPv4 header are taken as invalid checksum.
inner_l4_ok – Whether inner transport layer is valid including L4 checksum
inner_l4_checksum_ok – Whether inner transport layer checksum is valid. Packets without inner TCP/UDP header are taken as invalid checksum.
Matching on either outer_l4_ok=1 or inner_l4_ok=1 means that all L4 checks (length, checksum, etc.) are ok.
Matching on either outer_l4_ok=0 or inner_l4_ok=0 means that all L4 checks are not ok.
It is not possible to match using these fields for cases where a part of the checks is okay and a part is not ok.
doca_flow_header_format
This structure defines each layer of the packet header format.
struct doca_flow_header_format {
struct doca_flow_header_eth eth;
uint16_t l2_valid_headers;
struct doca_flow_header_eth_vlan eth_vlan[DOCA_FLOW_VLAN_MAX];
enum doca_flow_l3_type l3_type;
union {
struct doca_flow_header_ip4 ip4;
struct doca_flow_header_ip6 ip6;
};
enum doca_flow_l4_type_ext l4_type_ext;
union {
struct doca_flow_header_icmp icmp;
struct doca_flow_header_udp udp;
struct doca_flow_header_tcp tcp;
struct doca_flow_header_l4_port transport;
};
};
eth – Ethernet header format, including source and destination MAC address and the Ethernet layer type. If a VLAN header is present then eth.type represents the type following the last VLAN tag.
l2_valid_headers – bitwise OR one of the following options: DOCA_FLOW_L2_VALID_HEADER_VLAN_0, DOCA_FLOW_L2_VALID_HEADER_VLAN_1
eth_vlan – VLAN tag control information for each VLAN header.
l3_type – Layer 3 type, indicates the next layer is IPv4 or IPv6.
ip4 – IPv4 header format.
ip6 – IPv6 header format.
l4_type_ext – The next layer type after the layer 3.
icmp – ICMP header format.
udp – UDP header format.
tcp – TCP header format.
transport – header format for source and destination ports; used for defining match or actions with relaxed matching while not caring about the L4 protocol (whether TCP or UDP). This is used only if the l4_type_ext is DOCA_FLOW_L4_TYPE_EXT_TRANSPORT.
doca_flow_header_ip4
This structure defines IPv4 fields. This structure is relevant only when l3_type is DOCA_DOCA_FLOW_L3_TYPE_IP4.
struct doca_flow_header_ip4 {
doca_be32_t src_ip;
doca_be32_t dst_ip;
uint8_t version_ihl;
uint8_t dscp_ecn;
doca_be16_t total_len;
doca_be16_t identification;
doca_be16_t flags_fragment_offset;
uint8_t next_proto;
uint8_t ttl;
};
src_ip – source IP address.
dst_ip – destination IP address.
version_ihl – IP version (4 bits) and Internet Header Length (4 bits). The IHL part is supported as destination in DOCA_FLOW_ACTION_ADD operation.
dscp_ecn – type of service containing DSCP (6 bits) and ECN.
total_len – total length field. It is supported as destination in DOCA_FLOW_ACTION_ADD operation.
identification – IP fragment identification.
flags_fragment_offset – IP fragment flags (3 bits) and IP fragment offset (13 bits).
next_proto – IP next protocol.
ttl – Time-to-live. It is supported as destination in DOCA_FLOW_ACTION_ADD operation.
doca_flow_header_ip6
This structure defines IPv6 fields. This structure is relevant only when l3_type is DOCA_DOCA_FLOW_L3_TYPE_IP6.
struct doca_flow_header_ip6 {
doca_be32_t src_ip[4];
doca_be32_t dst_ip[4];
uint8_t traffic_class;
doca_be32_t flow_label;
doca_be16_t payload_len;
uint8_t next_proto;
uint8_t hop_limit;
};
src_ip – source IP address.
dst_ip – destination IP address.
traffic_class – traffic class containing DSCP (6 bits) and ECN.
flow_label – Only 20 bits (LSB) are relevant for specific value/mask, but marking it as changeable is done by all 32 bits (0xffffffff).
payload_len – Payload length. It is supported as destination in DOCA_FLOW_ACTION_ADD operation.
next_proto – IP next protocol.
hop_limit – Supported as destination in DOCA_FLOW_ACTION_ADD operation.
doca_flow_tun
This structure defines tunnel headers.
struct doca_flow_tun {
enum doca_flow_tun_type type;
union {
struct {
enum doca_flow_tun_ext_vxlan_type vxlan_type;
union {
uint8_t vxlan_next_protocol;
doca_be16_t vxlan_group_policy_id;
};
doca_be32_t vxlan_tun_id;
};
struct {
enum doca_flow_tun_ext_gre_type gre_type;
doca_be16_t protocol;
union {
struct {
bool key_present;
doca_be32_t gre_key;
};
struct {
doca_be32_t nvgre_vs_id;
uint8_t nvgre_flow_id;
};
};
};
struct {
doca_be32_t gtp_teid;
};
struct {
doca_be32_t esp_spi;
doca_be32_t esp_sn;
};
struct {
struct doca_flow_header_mpls mpls[DOCA_FLOW_MPLS_LABELS_MAX];
};
struct {
struct doca_flow_header_geneve geneve;
union doca_flow_geneve_option geneve_options[DOCA_FLOW_GENEVE_OPT_LEN_MAX];
};
struct {
struct doca_flow_header_psp psp;
};
};
};
type – type of tunnel (enum doca_flow_tun_type). Valid tunnel types:
DOCA_FLOW_TUN_VXLAN – VXLAN tunnel
DOCA_FLOW_TUN_GRE – GRE tunnel with option KEY (optional)
DOCA_FLOW_TUN_GTP – GTP tunnel
DOCA_FLOW_TUN_ESP – ESP tunnel
DOCA_FLOW_TUN_MPLS_O_UDP – MPLS tunnel (supports up to 5 headers)
DOCA_FLOW_TUN_GENEVE – GENEVE header format including option length, VNI, next protocol, and options.
DOCA_FLOW_TUN_PSP – PSP tunnel
vxlan_type – type of VXLAN extension (enum doca_flow_tun_ext_vxlan_type). Valid extension types:
DOCA_FLOW_TUN_EXT_VXLAN_STANDARD – Standard VXLAN tunnel (default)
DOCA_FLOW_TUN_EXT_VXLAN_GPE – VXLAN-GPE
DOCA_FLOW_TUN_EXT_VXLAN_GBP – VXLAN-GBP
vxlan_next_protocol – VXLAN-GPE next protocol
vxlan_group_policy_id – VXLAN-GBP group policy_id
vxlan_tun_vni – VNI (24) + reserved (8)
gre_type – type of GRE extension(enum doca_flow_tun_ext_gre_type). Valid extension types:
DOCA_FLOW_TUN_EXT_GRE_STANDARD – Standard GRE tunnel (default)
DOCA_FLOW_TUN_EXT_GRE_NVGRE – NVGRE
protocol – GRE next protocol
key_present – GRE option KEY is present
gre_key – GRE key option, match on this field only when key_present is true
nvgre_vs_id – NVGRE virtual subnet id(24) + reserved (8)
nvgre_flow_id – NVGRE flow ID
gtp_teid – GTP TEID
esp_spi – IPsec session parameter index
esp_sn – IPsec sequence number
mpls – List of MPLS header format
geneve – GENEVE header format
geneve_options – List DWs describing GENEVE TLV options
psp – PSP header format
The following table details which tunnel types support which operation on the tunnel header:
|
Tunnel Type |
Match 1 |
L2 encap 2 |
L2 decap 2 |
L3 encap 2 |
L3 decap 2 |
Modify 3 |
Copy 4 |
|
DOCA_FLOW_TUN_VXLAN |
✔ |
✔ |
✔ |
✘ |
✘ |
✔ |
✔ |
|
DOCA_FLOW_TUN_GRE |
✔ |
✔ |
✔ |
✔ |
✔ |
✘ |
✘ |
|
DOCA_FLOW_TUN_GTP |
✔ |
✘ |
✘ |
✔ |
✔ |
✔ |
✔ |
|
DOCA_FLOW_TUN_ESP |
✔ |
✘ |
✘ |
✘ |
✘ |
✔ |
✔ |
|
DOCA_FLOW_TUN_MPLS_O_UDP |
✔ |
✘ |
✘ |
✔ |
✔ |
✘ |
✔ |
|
DOCA_FLOW_TUN_GENEVE |
✔ |
✔ |
✔ |
✔ |
✔ |
✔ |
✔ |
|
DOCA_FLOW_TUN_PSP |
✔ |
✘ |
✘ |
✘ |
✘ |
✔ |
✔ |
Support for matching on this tunnel header, configured in the tun field in struct doca_flow_match.
Decapsulation/encapsulation of the tunnel header is to be enabled in struct doca_flow_actions using encap_type and decap_type. It is the user's responsibility to determine whether a tunnel is type L2 or L3. If the user sets settings unaligned with the packets coming, anomalous behavior may occur.
Support for modifying this tunnel header, configured in the tun field in struct doca_flow_actions.
Support for copying fields to/from this tunnel header, configured in struct doca_flow_action_descs.
DOCA Flow Tunnel GENEVE
The DOCA_FLOW_TUN_GENEVE type includes the basic header for GENEVE and an array for GENEVE TLV options.
The GENEVE TLV options must be configured before in parser creation (doca_flow_parser_geneve_opt_create).
doca_flow_header_geneve
This structure defines GENEVE protocol header.
struct doca_flow_header_geneve {
uint8_t ver_opt_len;
uint8_t o_c;
doca_be16_t next_proto;
doca_be32_t vni;
};
ver_opt_len – version (2) + options length (6). The length is expressed in 4-byte multiples, excluding the GENEVE header.
o_c – OAM packet (1) + critical options present (1) + reserved (6).
next_proto – GENEVE next protocol. When GENEVE has options, it describes the protocol after the options.
gre_key – GENEVE VNI (24) + reserved (8).
doca_flow_geneve_option
This object describes a single DW (4-bytes) from the GENEVE option header. It describes either the first DW in the option including class, type, and length, or any other data DW.
union doca_flow_geneve_option {
struct {
doca_be16_t class_id;
uint8_t type;
uint8_t length;
};
doca_be32_t data;
};
class_id – Option class ID
type – Option type
length – Reserved (3) + option data length (5). The length is expressed in 4-byte multiples, excluding the option header.
data – 4 bytes of option data
GENEVE Matching Notes
Option type and length must be provided for each option at pipe creation time in a match structure
When class mode is DOCA_FLOW_PARSER_GENEVE_OPT_MODE_FIXED, option class must also be provided for each option at pipe creation time in a match structure
Option length field cannot be matched
Type field is the option identifier, it must be provided as a specific value upon pipe creation
Option data is taken as changeable when all data is filled with 0xffffffff including DWs which were not configured at parser creation time
In the match_mask structure, the DWs which have not been configured at parser creation time must be zero
GENEVE Modification Notes
When GENEVE option modification is requested, the actions_mask structure must be provided. The option identifiers are provided in the mask structure while the values are provided in actions structure.
The options type and length must be provided for each option at pipe creation time in an action mask structure.
When the class mode is DOCA_FLOW_PARSER_GENEVE_OPT_MODE_FIXED, the option class must also be provided for each option at pipe creation time in an action mask structure.
Since class_id and type mask fields are used for identifiers, their modification is limited:
Specific value 0 is not supported on pipe creation, value 0 is interpreted as ignored. Modifying them to 0 is enabled per entry when they are marked as changeable during pipe creation.
Modifying partial mask is not supported.
The option length field cannot be modified.
The option data is taken as changeable when all the data is filled with 0xffffffff including DWs which were not configured at parser creation time.
In the actions_mask structure, the DWs which have not been configured at parser creation time must be zero.
Only data DW configured during parser creation can be modified.
Modification of the options type and class_id is supported only for options configured with class mode DOCA_FLOW_PARSER_GENEVE_OPT_MODE_MATCHABLE.
The order of options in the action structure does not necessarily reflect their order in the packet or match structure.
GENEVE Encapsulation Notes
The encapsulation size is constant per doca_flow_actions structure. The size is determined by the tun.geneve.ver_opt_len field, so it must be specified at pipe creation time.
The total encapsulation size is limited by 128 bytes. The tun.geneve.ver_opt_len field should follow this limitation according to the requested outer header sizes:
Header Included in Outer
Maximal tun.geneve.ver_opt_len Value
ETH + VLAN + IPV4 + UDP + GENEVE
18
ETH + VLAN + IPV6 + UDP + GENEVE
13
ETH + IPV4 + UDP + GENEVE
19
ETH + IPV6 + UDP + GENEVE
14
Options in encapsulation data do not have to be configured at parser creation time.
When at least one of the encap fields are changeable, GENEVE options are also taken as changeable regardless of values provided during pipe creation time. Thus, GENEVE option values must be provided again for each entry.
GENEVE Decapsulation Notes
The options to decapsulate do not have to be configured at parser creation time.
DOCA Flow Tunnel PSP
doca_flow_header_psp
This structure defines PSP protocol header.
struct doca_flow_header_psp {
uint8_t nexthdr;
uint8_t hdrextlen;
uint8_t res_cryptofst;
uint8_t s_d_ver_v;
doca_be32_t spi;
doca_be64_t iv;
doca_be64_t vc;
};
nexthdr – An IP protocol number, identifying the type of the next header.
hdrextlen – Length of this header in 8-octet units, not including the first 8 octets. When hdrextlen is non-zero, a virtualization cookie and/or other header extension fields may be present.
res_cryptofst – Reserved (2) + crypt offset (6). Crypt offset is the offset from the end of the initialization vector to the start of the encrypted portion of the payload, measured in 4-octet units.
s_d_ver_v – Sample at receiver (1) + drop after sampling (1) + version (4 bits) + V (1 bit) + 1 (always set bit).
spi – A 32-bit value that is used by a receiver to identify the security association (SA) to which an incoming packet is bound. High-order bit of this value indicates which master key is to be used for decryption.
iv – A unique value for each packet sent over a SA.
vc – An optional field, present if and only if V is set. It may contain a VNI or other data, as defined by the implementation of doca_flow_resource_meter_cfg.
All PSP header fields and bits can be matched/modified. Special care must be taken on the mask used to isolated desired bits; especially in byte fields that contains multiple bit fields like s_d_ver_v.
doca_flow_match
This structure is a match configuration that contains the user-defined fields that should be matched on the pipe.
struct doca_flow_match {
uint32_t flags;
struct doca_flow_meta meta;
struct doca_flow_parser_meta parser_meta;
struct doca_flow_header_format outer;
struct doca_flow_tun tun;
struct doca_flow_header_format inner;
};
flags – Match items which are no value needed.
meta – Programmable metadata.
parser_meta – Read-only metadata.
outer – Outer packet header format.
tun – Tunnel info
inner – Inner packet header format.
doca_flow_match_condition
This structure is a "match with compare result" configuration that contains the user-defined fields compare result that should be matched on the pipe. It can only be used for adding control pipe entry.
struct doca_flow_match_condition {
enum doca_flow_compare_op operation;
union {
struct {
struct doca_flow_desc_field a;
struct doca_flow_desc_field b;
uint32_t width;
} field_op;
};
};
operation – Match condition operation
a – Compare field descriptor A. The string field must be specified.
b – Compare field descriptor B. When string field is NULL, B is the immediate value. The value is taken from the field described by the A string in the attached match structure.
width – Compare width
The operation field includes the following compare operations:
DOCA_FLOW_COMPARE_EQ – Match with the compare operation to be equal (A = B)
DOCA_FLOW_COMPARE_NE – Match with the compare operation to be not equal (A ≠ B)
DOCA_FLOW_COMPARE_LT – Match with the compare operation to be less than (A < B)
DOCA_FLOW_COMPARE_LE – Match with the compare operation to be less equal (A ≤ B)
DOCA_FLOW_COMPARE_GT – Match with the compare operation to great than (A > B)
DOCA_FLOW_COMPARE_GE – Match with the compare operation to great equal (A ≥ B)
doca_flow_actions
This structure is a flow actions configuration.
struct doca_flow_actions {
uint8_t action_idx;
uint32_t flags;
enum doca_flow_resource_type decap_type;
union {
struct doca_flow_resource_decap_cfg decap_cfg;
uint32_t shared_decap_id;
};
bool pop;
struct doca_flow_meta meta;
struct doca_flow_header_format outer;
struct doca_flow_tun tun;
enum doca_flow_resource_type encap_type;
union {
struct doca_flow_resource_encap_cfg encap_cfg;
uint32_t shared_encap_id;
};
bool has_push;
struct doca_flow_push_action push;
bool has_crypto_encap;
struct doca_flow_crypto_encap_action crypto_encap;
struct doca_flow_crypto_action crypto;
};
action_idx – Index according to place provided on creation.
flags – Action flags.
decap_type – If decap_type is SHARED, the shared_decap_id takes effect. If NON_SHARED, the decap_cfg takes effect. If NONE, then there is no decap.
decap_cfg – The decap config for decap_type is NON_SHARED.
shared_decap_id – The shared decap ID for decap_type is SHARED.
pop – Pop header while it is set to true.
meta – Modify meta value.
outer – Modify outer header.
tun – Modify tunnel header.
encap_type – If encap_type is SHARED, the shared_encap_id takes effect. If NON_SHARED, the encap_cfg takes effect. If NONE, then there is no encap.
encap_cfg – The encap config for encap_type is NON_SHARED.
shared_encap_id – The shared encap ID for encap_type is SHARED.
has_push – Push header while it is set to true.
push – Push header data information.
has_crypto_encap – Perform packet reformat for crypto protocols while it is set to true. If set to true, the structure doca_flow_crypto_encap_action provides a description for the header and trailer to be inserted or removed.
crypto_encap – Crypto protocols header and trailer data information.
crypto – Contains crypto action information.
Action Order
When setting actions they are executed in the following order:
Crypto (decryption)
Decap
Pop
Meta
Outer
Tun
Push
Encap
Crypto (encryption)
doca_flow_encap_action
This structure is an encapsulation action configuration.
struct doca_flow_encap_action {
struct doca_flow_header_format outer;
struct doca_flow_tun tun;
};
outer – L2/3/4 layers of the outer tunnel header.
L2 - src/dst MAC addresses, ether type, VLAN
L3 - IPv4/6 src/dst IP addresses, TTL/hop_limit, dscp_ecn
L4 - the UDP dst port is determined by the tunnel type
tun – The specific fields of the used tunnel protocol. Supported tunnel types: GRE, GTP, VXLAN, MPLS, GENEVE.
doca_flow_push_action
This structure is a push action configuration.
struct doca_flow_push_action {
enum doca_flow_push_action_type type;
union {
struct doca_flow_header_eth_vlan vlan;
};
};
type – Push action type.
vlan – VLAN data.
The type field includes the following push action types:
DOCA_FLOW_PUSH_ACTION_VLAN – push VLAN.
doca_flow_crypto_encap_action
This structure is a crypto protocol packet reformat action configuration.
struct doca_flow_crypto_encap_action {
enum doca_flow_crypto_encap_action_type action_type;
enum doca_flow_crypto_encap_net_type net_type;
uint16_t icv_size;
uint16_t data_size;
uint8_t encap_data[DOCA_FLOW_CRYPTO_HEADER_LEN_MAX];
};
action_type – Reformat action type.
net_type – Protocol mode, network header type.
icv_size – Integrity check value size, in bytes; defines the trailer size.
data_size – Header size in bytes to be inserted from encap_data.
encap_data – Header data to be inserted.
The action_type field includes the following crypto encap action types:
DOCA_FLOW_CRYPTO_REFORMAT_NONE – No crypto encap action performed.
DOCA_FLOW_CRYPTO_REFORMAT_ENCAP – Add/insert the crypto header and trailer to the packet. data_size and encap_data should provide the data for the headers being inserted.
DOCA_FLOW_CRYPTO_REFORMAT_DECAP – Remove the crypto header and trailer from the packet. data_size and encap_data should provide the data for the headers being inserted for tunnel mode.
The net_type field includes the following protocol/network header types:
DOCA_FLOW_CRYPTO_HEADER_NONE – No header type specified.
DOCA_FLOW_CRYPTO_HEADER_ESP_TUNNEL – ESP tunnel mode header type. On encap, the full tunnel header data for new L2+L3+ESP should be provided. On decap, the new L2 header data should be provided.
DOCA_FLOW_CRYPTO_HEADER_ESP_OVER_IP – ESP transport over IP mode header type. On encap, the data for ESP header being inserted should be provided.
DOCA_FLOW_CRYPTO_HEADER_UDP_ESP_OVER_IP – UDP+ESP transport over IP mode header type. On encap, the data for UDP+ ESP headers being inserted should be provided.
DOCA_FLOW_CRYPTO_HEADER_ESP_OVER_LAN – ESP transport over UDP/TCP mode header type. On encap, the data for ESP header being inserted should be provided.
DOCA_FLOW_CRYPTO_HEADER_PSP_TUNNEL – UDP+PSP tunnel mode header type. On encap, the full tunnel header data for new L2+L3+UDP+PSP should be provided. On decap, the new L2 header data should be provided.
DOCA_FLOW_CRYPTO_HEADER_PSP_OVER_IPV4 – UDP+PSP transport over IPv4 mode header type. On encap, the data for UDP+PSP header being inserted should be provided.
DOCA_FLOW_CRYPTO_HEADER_PSP_OVER_IPV6 – UDP+PSP transport over IPv6 mode header type. On encap, the data for UDP+PSP header being inserted should be provided.
The icv_size field can be either 8, 12 or 16 for ESP, and 16 for PSP.
The data_size field should not exceed DOCA_FLOW_CRYPTO_HEADER_LEN_MAX; can be zero for decap in non-tunnel modes.
doca_flow_crypto_action
This structure is a crypto action configuration to perform packet data encryption and decryption.
struct doca_flow_crypto_action {
enum doca_flow_crypto_action_type action_type;
enum doca_flow_crypto_resource_type resource_type;
union {
struct {
bool sn_en;
} ipsec_sa;
};
uint32_t crypto_id;
};
action_type – Crypto action type.
resource_type – PSP or IPsec resource.
ipsec_sa.sn_en – Enable ESP sequence number generation/checking (IPsec only).
crypto_id – Shared crypto resource ID or Shared PSP/IPsec SA resource ID.
The action_type field includes the following crypto action types:
DOCA_FLOW_CRYPTO_ACTION_NONE – No crypto action specified.
DOCA_FLOW_CRYPTO_ACTION_ENCRYPT – Encrypt packet data according to the chosen protocol.
DOCA_FLOW_CRYPTO_ACTION_DECRYPT – Decrypt packet data according to the chosen protocol.
The resource_type field includes the following protocols supported:
DOCA_FLOW_CRYPTO_RESOURCE_NONE – No crypto resource specified.
DOCA_FLOW_CRYPTO_RESOURCE_IPSEC_SA – IPsec resource.
DOCA_FLOW_CRYPTO_RESOURCE_PSP – PSP resource.
doca_flow_action_descs
This structure describes operations executed on packets matched by the pipe.
Detailed compatibility matrix and usage can be found in section "Summary of Action Types".
struct doca_flow_action_descs {
uint8_t nb_action_desc;
struct doca_flow_action_desc *desc_array;
};
nb_action_desc – Maximum number of action descriptor array (i.e., number of descriptor array elements)
desc_array – Action descriptor array pointer
struct doca_flow_action_desc
struct doca_flow_action_desc {
enum doca_flow_action_type type;
union {
struct {
struct doca_flow_desc_field src;
struct doca_flow_desc_field dst;
unit32_t width;
} field_op;
};
};
type – Action type.
field_op – Field to copy/add source and destination descriptor. Add always applies from field bit 0 .
The type field includes the following forwarding modification types:
DOCA_FLOW_ACTION_AUTO – modification type derived from pipe action
DOCA_FLOW_ACTION_ADD – add from field value or packet field. Supports meta scratch, ipv4_ttl, ipv6_hop, tcp_seq, and tcp_ack.
InfoAdding from packet fields is supported only with NVIDIA® BlueField®-3 and NVIDIA® ConnectX®-7.
DOCA_FLOW_ACTION_COPY – copy field
doca_flow_desc_field
This struct is the flow descriptor's field configuration.
struct doca_flow_desc_field {
const char *field_string;
/**< Field selection by string. */
uint32_t bit_offset;
/**< Field bit offset. */
};
field_string – Field string. Describes which packet field is selected in string format.
bit_offset – Bit offset in the field.
NoteThe complete supported field string could be found in the "Field String Support" appendix.
doca_flow_monitor
This structure is a monitor configuration.
struct doca_flow_monitor {
enum doca_flow_resource_type meter_type;
/**< Type of meter configuration. */
union {
struct {
enum doca_flow_meter_limit_type limit_type;
/**< Meter rate limit type: bytes / packets per second */
uint64_t cir;
/**< Committed Information Rate (bytes/second). */
uint64_t cbs;
/**< Committed Burst Size (bytes). */
} non_shared_meter;
struct {
uint32_t shared_meter_id;
/**< shared meter id */
enum doca_flow_meter_color meter_init_color;
/**< meter initial color */
} shared_meter;
};
enum doca_flow_resource_type counter_type;
/**< Type of counter configuration. */
union {
struct {
uint32_t shared_counter_id;
/**< shared counter id */
} shared_counter;
};
uint32_t shared_mirror_id;
/**< shared mirror id. */
bool aging_enabled;
/**< Specify if aging is enabled */
uint32_t aging_sec;
/**< aging time in seconds.*/
};
meter_type – Defines the type of meter. Meters can be shared, non-shared, or not used at all.
non_shared_meter – non-shared meter params
limit_type – bytes versus packets measurement
cir – committed information rate of non-shared meter
cbs – committed burst size of non-shared meter
shared_meter – shared meter params
shared_meter_id – meter ID that can be shared among multiple pipes
meter_init_colr – the initial color assigned to a packet entering the meter
counter_type – defines the type of counter. Counters can be shared, or not used at all.
shared_counter_id – counter ID that can be shared among multiple pipes
shared_mirror_id – mirror ID that can be shared among multiple pipes
aging_enabled – set to true to enable aging
aging_sec – number of seconds from the last hit after which an entry is aged out
doca_flow_resource_type is defined as follows.:
enum doca_flow_resource_type {
DOCA_FLOW_RESOURCE_TYPE_NONE,
DOCA_FLOW_RESOURCE_TYPE_SHARED,
DOCA_FLOW_RESOURCE_TYPE_NON_SHARED
};
T(c) is the number of available tokens. For each packet where b equals the number of bytes, if t(c)-b≥0 the packet can continue, and tokens are consumed so that t(c)=t(c)-b. If t(c)-b<0, the packet is dropped.
CIR is the maximum bandwidth at which packets continue being confirmed. Packets surpassing this bandwidth are dropped. CBS is the maximum number of bytes allowed to exceed the CIR to be still CIR confirmed. Confirmed packets are handled based on the fwd parameter.
The number of <cir,cbs> pair different combinations is limited to 128.
Metering packets can be individual (i.e., per entry) or shared among multiple entries:
For the individual use case, set meter_type to DOCA_FLOW_RESOURCE_TYPE_NON_SHARED
For the shared use case, set meter_type to DOCA_FLOW_RESOURCE_TYPE_SHARED and shared_meter_id to the meter identifier
Counting packets can be individual (i.e., per entry) or shared among multiple entries:
For the individual use case, set counter type to DOCA_FLOW_RESOURCE_TYPE_SHARED and shared_counter_id to the counter identifier
For the shared use case, use a non-zero shared_counter_id
Mirroring packets can only be used as shared with a non-zero shared_mirror_id.
doca_flow_fwd
This structure is a forward configuration which directs where the packet goes next.
struct doca_flow_fwd {
enum doca_flow_fwd_type type;
union {
struct {
uint32_t rss_outer_flags;
uint32_t rss_inner_flags;
unit32_t *rss_queues;
int num_of_queues;
};
struct {
unit16_t port_id;
};
struct {
struct doca_flow_pipe *next_pipe;
};
struct {
struct doca_flow_pipe *pipe;
uint32_t idx;
} ordered_list_pipe;
struct {
struct doca_flow_target *target;
};
};
};
type – indicates the forwarding type
rss_outer_flags – RSS offload types on the outer-most layer (tunnel or non-tunnel).
rss_inner_flags – RSS offload types on the inner layer of a tunneled packet.
rss_queues – RSS queues array
num_of_queues – number of queues
port_id – destination port ID
next_pipe – next pipe pointer
ordered_list_pipe.pipe – ordered list pipe to select an entry from
ordered_list_pipe.idx – index of the ordered list pipe entry
target - target pointer
The type field includes the forwarding action types defined in the following enum:
DOCA_FLOW_FWD_RSS – forwards packets to RSS
DOCA_FLOW_FWD_PORT – forwards packets to port
DOCA_FLOW_FWD_PIPE – forwards packets to another pipe
DOCA_FLOW_FWD_DROP – drops packets
DOCA_FLOW_FWD_ORDERED_LIST_PIPE - forwards packet to a specific entry in an ordered list pipe
DOCA_FLOW_FWD_TARGET – forwards packets to a target
The rss_outer_flags and rss_inner_flags fields must be configured exclusively (either outer or inner).
Each outer/inner field is a bitwise OR of the RSS fields defined in the following enum:
DOCA_FLOW_RSS_IPV4 – RSS by IPv4 header
DOCA_FLOW_RSS_IPV6 – RSS by IPv6 header
DOCA_FLOW_RSS_UDP – RSS by UDP header
DOCA_FLOW_RSS_TCP – RSS by TCP header
DOCA_FLOW_RSS_ESP – RSS by ESP header
When specifying an RSS L4 type (DOCA_FLOW_RSS_TCP or DOCA_FLOW_RSS_UDP) it must have a bitwise OR with RSS L3 types (DOCA_FLOW_RSS_IPV4 or DOCA_FLOW_RSS_IPV6).
doca_flow_query
This struct is a flow query result.
struct doca_flow_query {
uint64_t total_bytes;
uint64_t total_pkts;
};
The struct doca_flow_query contains the following elements:
total_bytes – total bytes that hit this flow.
total_ptks – total packets that hit this flow.
doca_flow_init
This function is the global initialization function for DOCA Flow.
doca_error_t doca_flow_init(const struct doca_flow_cfg *cfg);
cfg [in] – Pointer to flow config structure.
Returns – DOCA_SUCCESS on success. Error code in case of failure:
DOCA_ERROR_INVALID_VALUE – Received invalid input
DOCA_ERROR_NO_MEMORY – Memory allocation failed
DOCA_ERROR_NOT_SUPPORTED – Unsupported pipe type
DOCA_ERROR_UNKNOWN – Otherwise
doca_flow_init must be invoked first before any other function in this API (except doca_flow_cfg API, to create and set doca_flow_cfg). This is a one-time call used for DOCA Flow initialization and global configurations.
doca_flow_destroy
This function is the global destroy function for DOCA Flow.
void doca_flow_destroy(void);
doca_flow_destroy must be invoked last to stop using DOCA Flow.
doca_flow_port_start
This function starts a port with its given configuration. It creates one port in the DOCA Flow layer, allocates all resources used by this port, and creates the default offload flow rules to redirect packets into software queues .
doca_error_t doca_flow_port_start(const struct doca_flow_port_cfg *cfg,
struct doca_flow_port **port);
cfg [in] – Pointer to DOCA Flow config structure.
port [out] – Pointer to DOCA Flow port handler on success.
Returns – DOCA_SUCCESS on success. Error code in case of failure:
DOCA_ERROR_INVALID_VALUE – Received invalid input
DOCA_ERROR_NO_MEMORY – Memory allocation failed
DOCA_ERROR_NOT_SUPPORTED – Unsupported pipe type
DOCA_ERROR_UNKNOWN – Otherwise
doca_flow_port_start modifies the state of the underlying DPDK port implementing the DOCA port. The DPDK port is stopped, then the flow configuration is applied, calling rte_flow_configure before starting the port again.
doca_flow_port_start must be called before any other DOCA Flow API to avoid conflicts.
In switch mode, the representor port must be stopped before switch port is stopped.
doca_flow_port_stop
This function releases all resources used by a DOCA flow port and removes the port's default offload flow rules.
doca_error_t doca_flow_port_stop(struct doca_flow_port *port);
port [in] – Pointer to DOCA Flow port handler.
doca_flow_port_pair
This function pairs two DOCA ports. After successfully pairing the two ports, traffic received on either port is transmitted via the other port by default.
For a pair of non-representor ports, this operation is required before port-based forwarding flows can be created. It is optional, however, if either port is a representor.
The two paired ports have no order.
A port cannot be paired with itself.
doca_error_t *doca_flow_port_pair(struct doca_flow_port *port,
struct doca_flow_port *pair_port);
port [in] – Pointer to the DOCA Flow port structure.
pair_port [in] – Pointer to another DOCA Flow port structure.
Returns – DOCA_SUCCESS on success. Error code in case of failure:
DOCA_ERROR_INVALID_VALUE – Received invalid input.
DOCA_ERROR_NO_MEMORY – Memory allocation failed.
DOCA_ERROR_UNKNOWN – Otherwise.
doca_flow_pipe_create
This function creates a new pipeline to match and offload specific packets. The pipeline configuration is defined in the doca_flow_pipe_cfg (configured using the doca_flow_pipe_cfg_set_* API). The API creates a new pipe but does not start the hardware offload.
When cfg type is DOCA_FLOW_PIPE_CONTROL, the function creates a special type of pipe that can have dynamic matches and forwards with priority.
doca_error_t
doca_flow_pipe_create(const struct doca_flow_pipe_cfg *cfg,
const struct doca_flow_fwd *fwd,
const struct doca_flow_fwd *fwd_miss,
struct doca_flow_pipe **pipe);
cfg [in] – Pointer to flow pipe config structure.
fwd [in] – Pointer to flow forward config structure.
fwd_miss [in] – Pointer to flow forward miss config structure. NULL for no fwd_miss.
NoteWhen fwd_miss configuration is provided for basic and hash pipes , they are executed on miss. For any other pipe type, the configuration is ignored.
pipe [out] – Pointer to pipe handler on success.
Returns – DOCA_SUCCESS on success. Error code in case of failure:
DOCA_ERROR_INVALID_VALUE – Received invalid input
DOCA_ERROR_NOT_SUPPORTED – Unsupported pipe type
DOCA_ERROR_DRIVER – Driver error
doca_flow_pipe_add_entry
This function adds a new entry to a pipe. When a packet matches a single pipe, it starts hardware offload. The pipe defines which fields to match. This API does the actual hardware offload, with the information from the fields of the input packets.
doca_error_t
doca_flow_pipe_add_entry(uint16_t pipe_queue,
struct doca_flow_pipe *pipe,
const struct doca_flow_match *match,
const struct doca_flow_actions *actions,
const struct doca_flow_monitor *monitor,
const struct doca_flow_fwd *fwd,
unit32_t flags,
void *usr_ctx,
struct doca_flow_pipe_entry **entry);
pipe_queue [in] – Queue identifier
pipe [in] – Pointer to flow pipe
match [in] – Pointer to flow match. Indicates specific packet match information.
actions [in] – Pointer to modify actions. Indicates specific modify information.
monitor [in] – Pointer to monitor actions
fwd [in] – Pointer to flow forward actions
flags [in] – can be set as DOCA_FLOW_WAIT_FOR_BATCH or DOCA_FLOW_NO_WAIT
DOCA_FLOW_WAIT_FOR_BATCH – this entry waits to be pushed to hardware
DOCA_FLOW_NO_WAIT – this entry is pushed to hardware immediately
usr_ctx [in] – Pointer to user context (see note at doca_flow_entry_process_cb)
entry [out] – Pointer to pipe entry handler on success
Returns – DOCA_SUCCESS on success. Error code in case of failure:
DOCA_ERROR_INVALID_VALUE – Received invalid input
DOCA_ERROR_DRIVER – Driver error
Some sanity checks may be omitted to avoid extra delays during flow insertion. For example, when forwarding to a pipe, the next_pipe field of struct doca_flow_fwd must contain a valid pointer. DOCA does not detect misconfigurations like these in the release build of the library.
doca_flow_pipe_update_entry
This function overrides the actions specified when the entry was last updated. If the intent is for some actions to be left unmodified, then the application must pass those as arguments to the update function.
doca_error_t
doca_flow_pipe_update_entry(uint16_t pipe_queue,
struct doca_flow_pipe *pipe,
const struct doca_flow_actions *actions,
const struct doca_flow_monitor *mon,
const struct doca_flow_fwd *fwd,
const enum doca_flow_flags_type flags,
struct doca_flow_pipe_entry *entry);
pipe_queue [in] – Queue identifier.
pipe [in] – Pointer to flow pipe.
actions [in] – Pointer to modify actions. Indicates specific modify information.
mon [in] – Pointer to monitor actions.
fwd [in] – Pointer to flow forward actions.
flags [in] – can be set as DOCA_FLOW_WAIT_FOR_BATCH or DOCA_FLOW_NO_WAIT.
DOCA_FLOW_WAIT_FOR_BATCH – this entry waits to be pushed to hardware.
DOCA_FLOW_NO_WAIT – this entry is pushed to hardware immediately.
entry [in] – Pointer to pipe entry to update.
Returns – DOCA_SUCCESS on success. Error code in case of failure:
DOCA_ERROR_INVALID_VALUE – Received invalid input
DOCA_ERROR_DRIVER – Driver error
doca_flow_pipe_control_add_entry
This function adds a new entry to a control pipe. When a packet matches a single pipe, it starts hardware offload. The pipe defines which fields to match. This API does the actual hardware offload with the information from the fields of the input packets.
doca_error_t
doca_flow_pipe_control_add_entry(uint16_t pipe_queue,
uint32_t priority,
struct doca_flow_pipe *pipe,
const struct doca_flow_match *match,
const struct doca_flow_match *match_mask,
const struct doca_flow_match_condition *condition,
const struct doca_flow_actions *actions,
const struct doca_flow_actions *actions_mask,
const struct doca_flow_action_descs *action_descs,
const struct doca_flow_monitor *monitor,
const struct doca_flow_fwd *fwd,
struct doca_flow_pipe_entry **entry);
pipe_queue [in] – Queue identifier.
priority [in] – Priority value.
pipe [in] – Pointer to flow pipe.
match [in] – Pointer to flow match. Indicates specific packet match information.
match_mask [in] – Pointer to flow match mask information.
condition [in] – Pointer to flow match condition information.
actions [in] – Pointer to modify actions. Indicates specific modify information.
actions_mask [in] – Pointer to modify actions' mask. Indicates specific modify mask information.
action_descs [in] – Pointer to action descriptors.
monitor [in] – Pointer to monitor actions.
fwd [in] – Pointer to flow forward actions.
entry [out] – Pointer to pipe entry handler on success.
Returns – DOCA_SUCCESS on success. Error code in case of failure:
DOCA_ERROR_INVALID_VALUE – Received invalid input
DOCA_ERROR_DRIVER – Driver error
Using a match condition cannot be mixed with exact match. Therefore, when condition is valid match_mask must be NULL.
When condition uses immediate value, the match structure must be provided with the value. Otherwise, match must also be NULL.
doca_flow_pipe_lpm_add_entry
This function adds a new entry to an LPM pipe. This API does the actual hardware offload all entries when flags is set to DOCA_FLOW_NO_WAIT.
doca_error_t
doca_flow_pipe_lpm_add_entry(uint16_t pipe_queue,
struct doca_flow_pipe *pipe,
const struct doca_flow_match *match,
const struct doca_flow_match *match_mask,
const struct doca_flow_actions *actions,
const struct doca_flow_monitor *monitor,
const struct doca_flow_fwd *fwd,
unit32_t flags,
void *usr_ctx,
struct doca_flow_pipe_entry **entry);
pipe_queue [in] – Queue identifier.
pipe [in] – Pointer to flow pipe.
match [in] – Pointer to flow match. Indicates specific packet match information.
match_mask [in] – Pointer to flow match mask information.
actions [in] – Pointer to modify actions. Indicates specific modify information.
monitor [in] – Pointer to monitor actions.
fwd [in] – Pointer to flow FWD actions.
flags [in] – Can be set as DOCA_FLOW_WAIT_FOR_BATCH or DOCA_FLOW_NO_WAIT.
DOCA_FLOW_WAIT_FOR_BATCH – LPM collects this flow entry
DOCA_FLOW_NO_WAIT – LPM adds this entry, builds the LPM software tree, and pushes all entries to hardware immediately
usr_ctx [in] – Pointer to user context (see note at doca_flow_entry_process_cb)
entry [out] – Pointer to pipe entry handler on success.
Returns – DOCA_SUCCESS on success. Error code in case of failure:
DOCA_ERROR_INVALID_VALUE – Received invalid input.
DOCA_ERROR_DRIVER – Driver error.
doca_flow_pipe_lpm_update_entry
This function updates an LPM entry with a new set of actions.
doca_error_t
doca_flow_pipe_lpm_update_entry(uint16_t pipe_queue,
struct doca_flow_pipe *pipe,
const struct doca_flow_actions *actions,
const struct doca_flow_monitor *monitor,
const struct doca_flow_fwd *fwd,
const enum doca_flow_flags_type flags,
struct doca_flow_pipe_entry *entry);
pipe_queue [in] – Queue identifier.
pipe [in] – Pointer to flow pipe.
actions [in] – Pointer to modify actions. Indicates specific modify information.
monitor [in] – Pointer to monitor actions.
fwd [in] – Pointer to flow FWD actions.
flags [in] – Can be set as DOCA_FLOW_WAIT_FOR_BATCH or DOCA_FLOW_NO_WAIT.
DOCA_FLOW_WAIT_FOR_BATCH – LPM collects this flow entry
DOCA_FLOW_NO_WAIT – LPM updates this entry and pushes all entries to hardware immediately
entry [in] – Pointer to pipe entry to update.
Returns – DOCA_SUCCESS on success. Error code in case of failure:
DOCA_ERROR_INVALID_VALUE – Received invalid input.
DOCA_ERROR_DRIVER – Driver error.
doca_flow_pipe_acl_add_entry
This function adds a new entry to an ACL pipe. This API performs the actual hardware offload for all entries when flags is set to DOCA_FLOW_NO_WAIT.
doca_error_t
doca_flow_pipe_acl_add_entry(uint16_t pipe_queue,
struct doca_flow_pipe *pipe,
const struct doca_flow_match *match,
const struct doca_flow_match *match_mask,
uint8_t priority,
const struct doca_flow_fwd *fwd,
unit32_t flags,
void *usr_ctx,
struct doca_flow_pipe_entry **entry);
pipe_queue [in] – Queue identifier.
pipe [in] – Pointer to flow pipe.
match [in] – Pointer to flow match. Indicates specific packet match information.
match_mask [in] – Pointer to flow match mask information.
priority [in] – Priority value.
fwd [in] – Pointer to flow FWD actions.
flags [in] – Can be set as DOCA_FLOW_WAIT_FOR_BATCH or DOCA_FLOW_NO_WAIT.
DOCA_FLOW_WAIT_FOR_BATCH – ACL collects this flow entry
DOCA_FLOW_NO_WAIT – ACL adds this entry, builds the ACL software model, and pushes all entries to hardware immediately
usr_ctx [in] – Pointer to user context (see note at doca_flow_entry_process_cb)
entry [out] – Pointer to pipe entry handler on success.
Returns – DOCA_SUCCESS on success. Error code in case of failure:
DOCA_ERROR_INVALID_VALUE – Received invalid input
DOCA_ERROR_DRIVER – Driver error
doca_flow_pipe_ordered_list_add_entry
This function adds a new entry to an order list pipe. When a packet matches a single pipe, it starts hardware offload. The pipe defines which fields to match. This API does the actual hardware offload, with the information from the fields of the input packets.
doca_error_t
doca_flow_pipe_ordered_list_add_entry(uint16_t pipe_queue,
struct doca_flow_pipe *pipe,
uint32_t idx,
const struct doca_flow_ordered_list *ordered_list,
const struct doca_flow_fwd *fwd,
enum doca_flow_flags_type flags,
void *user_ctx,
struct doca_flow_pipe_entry **entry);
pipe_queue [in] – Queue identifier.
pipe [in] – Pointer to flow pipe.
idx [in] – A unique entry index. It is the user's responsibility to ensure uniqueness.
ordered_list [in] – Pointer to an ordered list structure with pointers to struct doca_flow_actions and struct doca_flow_monitor at the same indices as they were at the pipe creation time. If the configuration contained an element of struct doca_flow_action_descs, the corresponding array element is ignored and can be NULL.
fwd [in] – Pointer to flow FWD actions.
flags [in] – Can be set as DOCA_FLOW_WAIT_FOR_BATCH or DOCA_FLOW_NO_WAIT.
DOCA_FLOW_WAIT_FOR_BATCH – this entry waits to be pushed to hardware
DOCA_FLOW_NO_WAIT – this entry is pushed to hardware immediately
user_ctx [in] – Pointer to user context (see note at doca_flow_entry_process_cb).
entry [out] – Pointer to pipe entry handler to fill.
Returns – DOCA_SUCCESS in case of success. Error code in case of failure:
DOCA_ERROR_INVALID_VALUE – Received invalid input.
DOCA_ERROR_NO_MEMORY – Memory allocation failed.
DOCA_ERROR_DRIVER – Driver error.
doca_flow_pipe_hash_add_entry
This function adds a new entry to a hash pipe. When a packet matches a single pipe, it starts hardware offload. The pipe defines which fields to match. This API does the actual hardware offload with the information from the fields of the input packets.
doca_error_t
doca_flow_pipe_hash_add_entry(uint16_t pipe_queue,
struct doca_flow_pipe *pipe,
uint32_t entry_index,
const struct doca_flow_actions *actions,
const struct doca_flow_monitor *monitor,
const struct doca_flow_fwd *fwd,
const enum doca_flow_flags_type flags,
void *usr_ctx,
struct doca_flow_pipe_entry **entry);
pipe_queue [in] – Queue identifier.
pipe [in] – Pointer to flow pipe.
entry_index [in] – A unique entry index. If the index is not unique, the function returns error.
actions [in] – Pointer to modify actions. Indicates specific modify information.
monitor [in] – Pointer to monitor actions.
fwd [in] – Pointer to flow FWD actions.
flags [in] – Can be set as DOCA_FLOW_WAIT_FOR_BATCH or DOCA_FLOW_NO_WAIT
DOCA_FLOW_WAIT_FOR_BATCH – This entry waits to be pushed to hardware
DOCA_FLOW_NO_WAIT – This entry is pushed to hardware immediately
usr_ctx [in] – Pointer to user context (see note at doca_flow_entry_process_cb)
entry [out] – Pointer to pipe entry handler to fill.
Returns – DOCA_SUCCESS in case of success. Error code in case of failure:
DOCA_ERROR_INVALID_VALUE – Received invalid input.
DOCA_ERROR_DRIVER – Driver error.
doca_flow_pipe_resize
This function resizes a pipe (currently only type DOCA_FLOW_PIPE_CONTROL is supported if, on creation, its configuration had is_resizable == true). The new congestion level is the percentage by which the new pipe size is determined. For example, if there are 800 entries in the pipe and congestion level is 50%, the new size of the pipe would be 1600 entries rounded up to the nearest a power of 2. The nr_entries_changed_cb and entry_relocation_cb are optional callbacks. The first callback informs on the exact new number of entries in the pipe. The second callback informs on entries that have been relocated from the smaller to the resized pipe.
doca_error_t
doca_flow_pipe_resize(struct doca_flow_pipe *pipe,
uint8_t new_congestion_level,
doca_flow_pipe_resize_nr_entries_changed_cb nr_entries_changed_cb,
doca_flow_pipe_resize_entry_relocate_cb entry_relocation_cb);
pipe [in] – Pointer to pipe to be resized
new_congestion_level [in] – Percentage to calculate the new new pipe size based on the current number of entries. The final size may be rounded up to the nearest power of 2.
nr_entries_changed_cb [in] – Pointer to callback when the new pipe size is set.
entry_relocation_cb [in] – Pointer to callback for each entry in the pipe that will be part of the resized pipe.
NoteCallbacks are optional. Either both are set or none.
Returns – DOCA_SUCCESS on success. Error code in case of failure.
doca_flow_pipe_process_cb
This typedef callback defines the received notification API for once congestion is reached or once the resize operation is completed.
enum doca_flow_pipe_op {
DOCA_FLOW_PIPE_OP_CONGESTION_REACHED,
DOCA_FLOW_PIPE_OP_RESIZED,
...
};
enum doca_flow_pipe_status {
DOCA_FLOW_PIPE_STATUS_SUCCESS = 1,
DOCA_FLOW_PIPE_STATUS_ERROR,
};
typedef void (*doca_flow_pipe_process_cb)(struct doca_flow_pipe *pipe,
enum doca_flow_pipe_status status,
enum doca_flow_pipe_op op, void *user_ctx);
pipe [in] – Pointer to the pipe whose process is reported.
status [in] – Process completion status (i.e., success or error).
op [in] – Operation reported in process, such as CONGESTION_REACHED, RESIZED.
user_ctx [in] – Pointer to user pipe context as configured on pipe creation.
NoteCallbacks are optional. Either both are set or none.
Returns – DOCA_SUCCESS on success. Error code in case of failure.
doca_flow_entries_process
This function processes entries in the queue. The application must invoke this function to complete flow rule offloading and to receive the flow rule's operation status.
If doca_flow_pipe_resize() is called, this function must be invoked as well to complete the resize process (i.e., until DOCA_FLOW_OP_PIPE_RESIZED is received as part of doca_flow_pipe_process_cb() callback).
doca_error_t
doca_flow_entries_process(struct doca_flow_port *port,
uint16_t pipe_queue,
uint64_t timeout,
uint32_t max_processed_entries);
port [in] – Pointer to the flow port structure.
pipe_queue [in] – Queue identifier.
timeout [in] – Timeout value in microseconds.
max_processed_entries [in] – Pointer to the flow pipe.
Returns – DOCA_SUCCESS on success. Error code in case of failure:
DOCA_ERROR_DRIVER – Driver error.
doca_flow_entry_status
This function gets the status of pipe entry.
enum doca_flow_entry_status
doca_flow_entry_get_status(struct doca_flow_entry *entry);
entry [in] – Pointer to the flow pipe entry to query.
Returns – Entry's status, defined in the following enum:
DOCA_FLOW_ENTRY_STATUS_IN_PROCESS – the operation is in progress
DOCA_FLOW_ENTRY_STATUS_SUCCESS – the operation completed successfully
DOCA_FLOW_ENTRY_STATUS_ERROR – the operation failed
doca_flow_entry_query
This function queries packet statistics about a specific pipe entry.
The pipe must have been created with the DOCA_FLOW_MONITOR_COUNT flag or the query will return an error.
doca_error_t doca_flow_query_entry(struct doca_flow_pipe_entry *entry, struct doca_flow_query *query_stats);
entry [in] – Pointer to the flow pipe entry to query
query_stats [out] – Pointer to the data retrieved by the query
Returns – DOCA_SUCCESS on success. Error code in case of failure:
DOCA_ERROR_INVALID_VALUE – Received invalid input
DOCA_ERROR_UNKNOWN – Otherwise
doca_flow_query_pipe_miss
This function queries packet statistics about a specific pipe miss flow.
The pipe must have been created with the miss_counter set to true or the query will return an error.
doca_error_t doca_flow_query_pipe_miss(struct doca_flow_pipe *pipe, struct doca_flow_query *query_stats);
pipe [in] – Pointer to the flow pipe to query.
query_stats [out] – Pointer to the data retrieved by the query.
Returns – DOCA_SUCCESS on success. Error code in case of failure:
DOCA_ERROR_INVALID_VALUE – Received invalid input
DOCA_ERROR_UNKNOWN – Otherwise
doca_flow_pipe_update_miss
This function updates a specific pipe's miss flow.
doca_error_t doca_flow_pipe_update_miss(struct doca_flow_pipe *pipe, const struct doca_flow_fwd *fwd_miss);
pipe [in] – Pointer to the flow pipe to update.
fwd_miss [in] – Pointer to a new flow forward miss config structure.
NoteThe forward type must be the same of fwd_miss given in pipe creation, the type itself cannot be updated.
Returns – DOCA_SUCCESS on success. Error code in case of failure:
DOCA_ERROR_INVALID_VALUE – Received invalid input
DOCA_ERROR_NOT_SUPPORTED – Unsupported configuration
DOCA_ERROR_UNKNOWN – Otherwise
doca_flow_aging_handle
This function handles the aging of all the pipes of a given port. It goes over all flows and releases aged flows from being tracked. The user gets a notification in the callback about the aged entries. Since the number of flows can be very large, it can take a significant amount of time to go over all flows, so this function is limited by a time quota. This means it might return without handling all flows which requires the user to call it again.
The pipe needs to have been created with the DOCA_FLOW_MONITOR_COUNT flag or the query will return an error.
int doca_flow_aging_handle(struct doca_flow_port *port,
uint16_t queue,
uint64_t quota);
queue [in] – Queue identifier.
quota [in] – Max time quota in microseconds for this function to handle aging.
Returns
>0 – the number of aged flows filled in entries array.
0 – no aged entries in current call.
-1 – full cycle is done.
doca_flow_mpls_label_encode
This function prepares an MPLS label header in big-endian. Input variables are provided in CPU-endian.
doca_error_t
doca_flow_mpls_label_encode(uint32_t label, uint8_t traffic_class, uint8_t ttl, bool bottom_of_stack,
struct doca_flow_header_mpls *mpls);
label [in] – Label value (20 bits).
traffic_class [in] – Traffic class (3 bits).
ttl [in] – Time to live (8 bits).
bottom_of_stack [in] – Whether this MPLS is bottom-of-stack.
mpls [out] – Pointer to MPLS structure to fill.
Returns – DOCA_SUCCESS on success. Error code in case of failure:
DOCA_ERROR_INVALID_VALUE – Received invalid input.
doca_flow_mpls_label_decode
This function decodes an MPLS label header. Output variables are returned in CPU-endian.
doca_error_t
doca_flow_mpls_label_decode(const struct doca_flow_header_mpls *mpls, uint32_t *label,
uint8_t *traffic_class, uint8_t *ttl, bool *bottom_of_stack);
mpls [in] – Pointer to MPLS structure to decode.
label [out] – Pointer to fill MPLS label value.
traffic_class [out] – Pointer to fill MPLS traffic class value.
ttl [out] – Pointer to fill MPLS TTL value.
bottom_of_stack [out] – Pointer to fill whether this MPLS is bottom-of-stack.
Returns – DOCA_SUCCESS on success. Error code in case of failure:
DOCA_ERROR_INVALID_VALUE – Received invalid input
doca_flow_parser_geneve_opt_create
This function must be called before creation of any pipe using GENEVE option.
This operation is fully supported only when FLEX_PARSER_PROFILE_ENABLE = 8. To do that, run:
mlxconfig -d <device-id> set FLEX_PARSER_PROFILE_ENABLE=8
This function prepares a GENEVE TLV parser for a selected port.
This API is port oriented, but the configuration is done once for all ports under the same physical device. Each port should call this API before using GENEVE options, but it must use the same options in the same order in the list.
Each physical device has 7 DWs for GENEVE TLV options. Each non-zero element in the data_mask array consumes one DW, and choosing a matchable mode per class consumes additional one.
Calling this API for a second port under the same physical device does not consume more DWs as it uses same configuration.
doca_error_t
doca_flow_parser_geneve_opt_create(const struct doca_flow_port *port,
const struct doca_flow_parser_geneve_opt_cfg tlv_list[i],
uint8_t nb_options, struct doca_flow_parser **parser);
port [in] – Pointer to DOCA Flow port.
tlv_list [in] – An array to create GENEVE TLV parser for. The index option in this array is used as an option identifier in the action descriptor string.
nb_options [in] – The number of options in the TLV array.
parser [out] – Pointer to parser handler to fill on success.
Returns – DOCA_SUCCESS on success. Error code in case of failure:
DOCA_ERROR_INVALID_VALUE – Received invalid input
DOCA_ERROR_NO_MEMORY – Memory allocation failed
DOCA_ERROR_NOT_SUPPORTED – Unsupported configuration
DOCA_ERROR_ALREADY_EXIST – Physical device already has parser, by either same or another port
DOCA_ERROR_UNKNOWN – Otherwise
doca_flow_parser_geneve_opt_destroy
This function must be called after the last use of the GENEVE option and before port closing.
This function destroys GENEVE TLV parser.
doca_error_t
doca_flow_parser_geneve_opt_destroy(struct doca_flow_parser *parser);
parser [in] – Pointer to parser to be destroyed.
Returns – DOCA_SUCCESS on success. Error code in case of failure:
DOCA_ERROR_INVALID_VALUE – Received invalid input.
DOCA_ERROR_IN_USE – One of the options is being used by a pipe.
DOCA_ERROR_DRIVER – There is no valid GENEVE TLV parser in this handle.
DOCA_ERROR_UNKNOWN – Otherwise.
doca_flow_get_target
This function gets a target handler.
doca_error_t
doca_flow_get_target(enum doca_flow_target_type, struct doca_flow_target **target);
type [in] – Target type.
target [out] – Pointer to target handler.
Returns – DOCA_SUCCESS on success. Error code in case of failure:
DOCA_ERROR_INVALID_VALUE – Received invalid input.
DOCA_ERROR_NOT_SUPPORTED – Unsupported type.
The type field includes the following target type:
DOCA_FLOW_TARGET_KERNEL
doca_flow_port_switch_get
doca_flow_port_switch_get(const struct doca_flow_port *port);
port [in] – The port for which to get the associated switch port
Returns - The switch port or NULL if none exists
doca_flow_pipe_calc_hash
doca_flow_pipe_calc_hash(struct doca_flow_pipe *pipe, const struct doca_flow_match *match, uint32_t *hash);
pipe [in] – Pointer to flow pipe.
match [in] – Pointer to flow match. Indicates specific packet match information.
hash [out] – Holds the calculation result for a given packet.
doca_flow_crypto_ipsec_resource_handle
This function updates IPsec resources according to hardware state. This function must be called periodically in order to keep a valid state of SA.
doca_flow_crypto_ipsec_resource_handle(struct doca_flow_pipe *pipe, const struct doca_flow_match *match, uint32_t *hash);
port [in] – Port to handle resources.
quota [in] – Max time in microseconds. 0 means no time limit.
max_processed_resources [in] – Max resources to handle. 0 means no resources limit.
doca_flow_crypto_psp_master_key_rotate
This function rotates PSP master key.
After this function is called, a new master key is used to generate pairs of SPI and key.
The old master key remains valid for decryption until another key_rotate is called.
doca_flow_crypto_psp_master_key_rotate(struct doca_flow_port *port);
port [in] – Pointer to DOCA Flow port.
doca_flow_crypto_psp_spi_key_bulk_alloc
This function allocates the memory required for the array of SPI key pairs, based on the key type and the number of pairs.
doca_flow_crypto_psp_spi_key_bulk_alloc(struct doca_flow_port *port,
enum doca_flow_crypto_key_type key_type,
uint32_t nr_spi_keys,
struct doca_flow_crypto_psp_spi_key_bulk **spi_key_bulk);
port [in] – Pointer to DOCA Flow port.
key_type [in] – DOCA_FLOW_CRYPTO_KEY_128 or DOCA_FLOW_CRYPTO_KEY_256.
nr_spi_keys [in] – Array length.
spi_key_bulk [out] – SPI key bulk handler.
doca_flow_crypto_psp_spi_key_bulk_generate
This function fills a bulk with new pairs of SPI and key, based on the key type and number of pairs.
This function can be used more than once on allocated bulk of pairs.
doca_flow_crypto_psp_spi_key_bulk_generate(struct doca_flow_crypto_psp_spi_key_bulk *spi_key_bulk);
spi_key_bulk [in] – SPI key bulk handler.
doca_flow_crypto_psp_spi_key_bulk_get
This function gets a pair of SPI and key for a specific index in the bulk
doca_flow_crypto_psp_spi_key_bulk_get(struct doca_flow_crypto_psp_spi_key_bulk *spi_key_bulk,
uint32_t spi_key_idx,
uint32_t *spi,
uint32_t *key);
spi_key_bulk [in] – SPI key bulk handler.
spi_key_idx [in] – Index in the bulk.
spi [out] – Pointer to the SPI.
key [out] – Pointer to the key.
doca_flow_crypto_psp_spi_key_wipe
This function wipes the memory of a key for specific index in the bulk.
doca_flow_crypto_psp_spi_key_wipe(struct doca_flow_crypto_psp_spi_key_bulk *spi_key_bulk,
uint32_t spi_key_idx);
spi_key_bulk [in] – SPI key bulk handler.
spi_key_idx [in] – Index in the bulk.
doca_flow_crypto_psp_spi_key_bulk_clear
This function clears the bulk data.
After all the keys in bulk have been disposed, if the user generates new bulk in the future, it is recommended to clear the bulk memory with this function.
doca_flow_crypto_psp_spi_key_bulk_clear(struct doca_flow_crypto_psp_spi_key_bulk *spi_key_bulk);
spi_key_bulk [in] – SPI key bulk handler.
doca_flow_crypto_psp_spi_key_bulk_free
This function frees the memory in the bulk.
doca_flow_crypto_psp_spi_key_bulk_free(struct doca_flow_crypto_psp_spi_key_bulk *spi_key_bulk);
spi_key_bulk [in] – SPI key bulk handler.
A shared counter can be used in multiple pipe entries. The following are the steps for configuring and using shared counters.
On doca_flow_init()
Specify the total number of shared counters to be used, nb_shared_counters. This call implicitly defines the shared counters IDs in the range of 0-(nb_shared_counters-1)
.nr_shared_resources = {
[DOCA_FLOW_SHARED_RESOURCE_COUNT] = nb_shared_counters
},
On doca_flow_shared_resource_cfg()
This call can be skipped for shared counters.
On doca_flow_shared_resource_bind()
This call binds a bulk of shared counters IDs to a specific pipe or port.
doca_error_t
doca_flow_shared_resources_bind(enum doca_flow_shared_resource_type type, uint32_t *res_array,
uint32_t res_array_len, void *bindable_obj);
res_array [in] – Array of shared counters IDs to be bound.
res_array_len [in] – Array length.
bindable_obj – Pointer to either a pipe or port.
This call allocates the counter's objects. A counter ID specified in this array can only be used later by the corresponding bindable object (pipe or port).
The following example binds counter IDs 2, 4, and 7 to a pipe. The counters' IDs must be within the range 0-(nb_shared_coutners-1).
uint32_t shared_counters_ids[] = {2, 4, 7};
struct doca_flow_pipe *pipe = ...
doca_flow_shared_resources_bind(
DOCA_FLOW_SHARED_RESOURCE_COUNT,
shared_counters_ids, 3, pipe, &error);
On doca_flow_pipe_add_entry() or Pipe Configuration (struct doca_flow_pipe_cfg)
The shared counter ID is included in the monitor parameter. It must be bound in advance to the pipe object.
struct doca_flow_monitor {
...
uint32_t shared_counter_id;
/**< shared counter id */
...
}
Packets matching the pipe entry are counted on the shared_counter_id. In pipe configuration, the shared_counter_id can be changeable (all FFs) and then the pipe entry holds the specific shared counter ID.
In switch mode, verifying counter domain is skipped.
Querying Bulk of Shared Counter IDs
Use this API:
int
doca_flow_shared_resources_query(enum doca_flow_shared_resource_type type,
uint32_t *res_array,
struct doca_flow_shared_resource_result *query_results_array,
uint32_t array_len,
struct doca_flow_error *error);
res_array [in] – Array of shared counters IDs to be queried.
res_array_len [in] – Array length.
query_results_array [out] – Query results array. The user must have allocated it prior to calling this API.
The type parameter is DOCA_FLOW_SHARED_RESOURCE_COUNT.
On doca_flow_pipe_destroy() or doca_flow_port_stop()
All bound resource IDs of this pipe or port are destroyed.
A shared meter can be used in multiple pipe entries (hardware steering mode support only).
The shared meter action marks a packet with one of three colors: Green, Yellow, and Red. The packet color can then be matched in the next pipe, and an appropriate action may be taken. For example, packets marked in red color are usually dropped. So, the next pipe to meter action may have an entry which matches on red and has fwd type DOCA_FLOW_FWD_DROP.
DOCA Flow supports three marking algorithms based on RFCs: 2697, 2698, and 4115.
RFC 2697 – Single-rate Three Color Marker (srTCM)
CBS (committed burst size) is the bucket size which is granted credentials at a CIR (committed information rate). If CBS overflow occurs, credentials are passed to the EBS (excess burst size) bucket. Packets passing through the meter consume credentials. A packet is marked green if it does not exceed the CBS, yellow if it exceeds the CBS but not the EBS, and red otherwise. A packet can have an initial color upon entering the meter. A pre-colored yellow packet will start consuming credentials from the EBS.
RFC 2698 – Two-rate Three Color Marker (trTCM)
CBS and CIR are defined as in RFC 2697. PBS (peak burst size) is a second bucket which is granted credentials at a PIR (peak information rate). There is no overflow of credentials from the CBS bucket to the PBS bucket. The PIR must be equal to or greater than the CIR. Packets consuming CBS credentials consume PBS credentials as well. A packet is marked red if it exceeds the PIR. Otherwise, it is marked either yellow or green depending on whether it exceeds the CIR or not. A packet can have an initial color upon entering the meter. A pre-colored yellow packet starts consuming credentials from the PBS.
RFC 4115 – trTCM without Peak-rate Dependency
EBS is a second bucket which is granted credentials at a EIR (excess information rate) and gets overflowed credentials from the CBS. For the packet marking algorithm, refer to RFC 4115.
The following sections present the steps for configuring and using shared meters to mark packets.
On doca_flow_init()
Specify the total number of shared meters to be used, nr_shared_meters.
The following call is an example how to initialize both shared counters and meter ranges. This call implicitly defines the shared counter IDs in the range of (0-nr_shared_counters-1) and the shared meter IDs in the range of 0-(nr_shared_meters-1).
struct doca_flow_cfg *cfg;
doca_flow_cfg_create(&cfg);
doca_flow_cfg_set_pipe_queues(cfg, queues);
...
doca_flow_cfg_set_nr_shared_resource(cfg, nr_shared_meters, DOCA_FLOW_SHARED_RESOURCE_METER);
doca_flow_cfg_set_nr_shared_resource(cfg, nr_shared_counters, DOCA_FLOW_SHARED_RESOURCE_COUNT);
doca_flow_init(cfg);
doca_flow_cfg_destroy(cfg);
On doca_flow_shared_resource_cfg()
This call binds a specific meter ID with its committed information rate (CIR) and committed burst size (CBS).
struct doca_flow_resource_meter_cfg {
...
uint64_t cir;
/**< Committed Information Rate (bytes/second). */
uint64_t cbs;
/**< Committed Burst Size (bytes). */
...
};
struct doca_flow_shared_resource_cfg {
union {
struct doca_flow_resource_meter_cfg meter_cfg;
...
};
};
int
doca_flow_shared_resource_cfg(enum doca_flow_shared_resource_type type, uint32_t id,
struct doca_flow_shared_resource_cfg *cfg,
struct doca_flow_error *error);
The following example configures the shared meter ID 5 with a CIR of 0x1000 bytes per second and a CBS of 0x600 bytes:
struct doca_flow_shared_resource_cfg shared_cfg = { 0 };
shared_cfg.meter_cfg.cir = 0x1000;
shared_cfg.meter_cfg.cbs = 0x600;
doca_flow_shared_resource_cfg(DOCA_FLOW_SHARED_RESOURCE_METER, 0x5, &shared_cfg, &error);
The last meter configuration example sets only the CIR and CBS fields (using RFC 2697 algorithm by default).
The following is the full meter configuration struct:
enum doca_flow_meter_algorithm_type {
DOCA_FLOW_METER_ALGORITHM_TYPE_RFC2697,
/**< Single Rate Three Color Marker - IETF RFC 2697. */
DOCA_FLOW_METER_ALGORITHM_TYPE_RFC2698,
/**< Two Rate Three Color Marker - IETF RFC 2698. */
DOCA_FLOW_METER_ALGORITHM_TYPE_RFC4115,
/**< Two Rate Three Color Marker - IETF RFC 4115. */
};
enum doca_flow_meter_limit_type {
DOCA_FLOW_METER_LIMIT_TYPE_BYTES = 0,
/**< Meter parameters per bytes */
DOCA_FLOW_METER_LIMIT_TYPE_PACKETS,
/**< Meter parameters packets */
};
struct doca_flow_resource_meter_cfg {
enum doca_flow_meter_limit_type limit_type;
/**< Meter rate limit type: bytes / packets per second */
enum doca_flow_meter_algorithm_type alg;
/**< Meter algorithm by RFCs */
uint64_t cir;
/**< Committed Information Rate (bytes or packets per second). */
uint64_t cbs;
/**< Committed Burst Size (bytes or packets). */
union {
struct {
uint64_t ebs;
/** Excess Burst Size (EBS) (bytes or packets). */
} rfc2697;
struct {
uint64_t pir;
/**< Peak Information Rate (bytes or packets per seconds). */
uint64_t pbs;
/**< Peak Burst Size (bytes or packets). */
} rfc2698;
struct {
uint64_t eir;
/**< Excess Information Rate (bytes or packets per seconds). */
uint64_t ebs;
/**< Excess Burst Size (EBS) (bytes or packets). */
} rfc4115;
};
};
limit_type – Bytes versus packets measurement.
alg – The meter marking RFC algorithm: 2697, 2698, or 4115.
cir – Committed information rate for shared meter.
cbs – Committed burst size of shared meter.
Pir – Peak information rate of shared meter.
Pbs – Peak burst size of shared meter.
Eir – Excess information rate of shared meter.
Ebs – Excess burst size of shared meter.
On doca_flow_shared_resource_bind()
This call binds a bulk of shared meter IDs to a specific pipe or port.
doca_error_t
doca_flow_shared_resources_bind(enum doca_flow_shared_resource_type type, uint32_t *res_array,
uint32_t res_array_len, void *bindable_obj);
res_array [in] – array of shared meter IDs to be bound
res_array_len [in] – array length
bindable_obj – pointer to either a pipe or port
This call allocates the meter's objects. A meter ID specified in this array can only be used later by the corresponding bindable object (pipe or port).
The following example binds meter IDs 5 and 14 to a pipe. The meter IDs must be within the range 0-(nb_shared_meters-1).
uint32_t shared_meters_ids[] = {5, 14};
struct doca_flow_pipe *pipe = ...
doca_flow_shared_resources_bind(
DOCA_FLOW_SHARED_RESOURCE_METER,
shared_meters_ids, 2, pipe, &error);
On doca_flow_pipe_add_entry() or Pipe Configuration (struct doca_flow_pipe_cfg)
The shared meter ID is included in the monitor parameter. It must be bound in advance to the pipe object.
struct doca_flow_monitor {
...
uint32_t shared_meter_id;
/**< shared meter id */
...
}
Packets matching the pipe entry are metered based on the cir and the cbs parameters related to the shared_meter_id. In the pipe configuration, the shared_meter_id can be changeable (all FFs) and then the pipe entry must hold the specific shared meter ID for that entry.
In switch mode, verifying meter domain is skipped.
Querying Bulk of Shared Meter IDs
There is no direct API to query a shared meter ID. To count the number of packets before a meter object, add a counter (shared or single) and use an API to query it. For an example, see section "Querying Bulk of Shared Counter IDs".
On doca_flow_pipe_destroy() or doca_flow_port_stop()
All bound resource IDs of this pipe or port are destroyed.
A shared RSS can be used in multiple pipe entries.
On doca_flow_init()
Specify the total number of shared RSS to be used, nr_shared_rss. This call implicitly defines the shared RSS IDs in the range between 0-(nr_shared_rss-1).
struct doca_flow_cfg *cfg;
doca_flow_cfg_create(&cfg);
doca_flow_cfg_set_nr_shared_resource(cfg, nr_shared_rss, DOCA_FLOW_SHARED_RESOURCE_RSS);
doca_flow_init(cfg);
doca_flow_cfg_destroy(cfg);
On doca_flow_shared_resource_cfg()
This call configures shared RSS resource.
struct doca_flow_shared_resource_cfg res_cfg;
for (uint8_t i = 0; i < nb_shared_rss; i++) {
res_cfg.rss_cfg.nr_queues = nr_queues;
res_cfg.rss_cfg.flags = flags;
res_cfg.rss_cfg.queues_array = queues_array;
doca_flow_shared_resource_cfg(DOCA_FLOW_SHARED_RESOURCE_RSS, i, &rss_cfg, &error);
}
On doca_flow_shared_resource_bind()
This call binds a bulk of shared RSS to a specific port.
uint32_t shared_rss_ids[] = {2, 4, 7};
struct doca_flow_port *port;
doca_flow_shared_resources_bind(
DOCA_FLOW_SHARED_RESOURCE_RSS,
shared_rss_ids, 3, port);
On doca_flow_pipe_add_entry()
On doca_flow_pipe_create, the user can input NULL as fwd. On doca_flow_pipe_add_entry, the user can input preconfigured shared RSS as fwd by specifying the shared_rss_id.
struct doca_flow_fwd;
fwd.shared_rss_id = 2;
fwd.type = DOCA_FLOW_FWD_RSS;
doca_flow_pipe_add_entry(queue, pipe, match, action, mon, &fwd, flag, usr_ctx, &error);
On doca_flow_port_stop()
All bound shared_rss resource IDs of this port are destroyed.
A shared IPSec SA resource can be used in multiple pipe entries and is intended to create an SA for crypto offloads (i.e., encrypt or decrypt packet data operations).
The following subsections expand on the steps for configuring and using shared IPsec SA resource.
On doca_flow_init()
Specify the total number of shared crypto operations to be used, nr_shared_ipsec_sa. This call implicitly defines the shared IPsec SA IDs in the range of 0-(nr_shared_ipsec_sa-1).
struct doca_flow_cfg *cfg;
doca_flow_cfg_create(&cfg);
doca_flow_cfg_set_pipe_queues(cfg, queues);
...
doca_flow_cfg_set_nr_shared_resource(cfg, nr_shared_ipsec_sa, DOCA_FLOW_SHARED_RESOURCE_IPSEC_SA);
doca_flow_init(cfg);
doca_flow_cfg_destroy(cfg);
On doca_flow_shared_resource_cfg()
This call provides the specific IPsec SA ID with its configuration.
struct doca_flow_crypto_key_cfg {
enum doca_flow_crypto_key_type key_type;
uint32_t *key;
};
struct doca_flow_resource_ipsec_sa_cfg {
struct doca_flow_crypto_key_cfg key_cfg;
uint32_t salt;
uint64_t implicit_iv;
enum doca_flow_crypto_icv_len icv_len;
enum doca_flow_crypto_sn_offload_type sn_offload_type;
enum doca_flow_crypto_replay_win_size win_size;
bool esn_en;
uint64_t sn_initial;
};
struct doca_flow_shared_resource_cfg {
union {
struct doca_flow_resource_ipsec_sa_cfg ipsec_sa_cfg;
...
};
};
int
doca_flow_shared_resource_cfg(enum doca_flow_shared_resource_type type, uint32_t id,
struct doca_flow_shared_resource_cfg *cfg,
struct doca_flow_error *error);
On doca_flow_shared_resource_bind()
This call binds a bulk of shared IPsec SAs IDs to a specific pipe or port.
doca_error_t
doca_flow_shared_resources_bind(enum doca_flow_shared_resource_type type, uint32_t *res_array,
uint32_t res_array_len, void *bindable_obj);
res_array [in] – Array of shared crypto IDs to be bound.
res_array_len [in] – Array length.
bindable_obj – Pointer to either a pipe or port.
This call allocates the IPsec SA's objects. A IPsec SA ID specified in this array can only be used later by the corresponding bindable object (pipe or port).
The following example binds IPsec SA IDs 2, 5, and 7 to a pipe. The cryptos' IDs must be within the range 0-(nb_shared_ipsec_sa-1).
uint32_t shared_ipsec_sa_ids[] = {2, 5, 7};
struct doca_flow_pipe *pipe = ...
doca_flow_shared_resources_bind(
DOCA_FLOW_SHARED_RESOURCE_IPSEC_SA,
shared_ipsec_sa_ids, 3, pipe, &error);
On doca_flow_pipe_add_entry() or Pipe Configuration (struct doca_flow_pipe_cfg)
The shared IPsec SA ID is included in the action parameter. It must be bound in advance to the pipe object.
struct doca_flow_actions {
...
struct {
...
uint32_t crypto_id;
/**< Crypto shared action id */
} crypto;
}
Crypto operations are performed over the packets matching the pipe entry according to the crypto_id configuration. Afterwards, the flow continues from the point specified in the forward part of the pipe configuration. In pipe configuration, the crypto_id should reference the shared crypto object ID to fetch the security pool information required for pipe creation. crypto_id is always considered changeable (regardless of all FFs) and the pipe entry holds the specific shared crypto ID.
The IPsec shared crypto resource can provide extra information in the u32[] fields of doca_flow_meta and the application can establish a match on these fields in the next pipes.
Extra information is provided only for actions with an IPsec object configured in full offload mode.
On doca_flow_pipe_destroy() or doca_flow_port_stop()
All bound crypto resource IDs of this pipe or port are destroyed.
A shared PSP resource can be used in multiple pipe entries and is intended to create SA for encryption offloads operations.
The following subsections expand on the steps for configuring and using shared PSP resource.
On doca_flow_init()
Specifies the total number of shared PSP operations to be used, nr_shared_psp. This call implicitly defines the shared PSP IDs in the range of 0-(nr_shared_psp-1).
struct doca_flow_cfg *cfg;
doca_flow_cfg_create(&cfg);
doca_flow_cfg_set_pipe_queues(cfg, queues);
...
doca_flow_cfg_set_nr_shared_resource(cfg, nr_shared_psp, DOCA_FLOW_SHARED_RESOURCE_PSP);
doca_flow_init(cfg);
doca_flow_cfg_destroy(cfg);
On doca_flow_shared_resource_cfg()
This call provides the specific PSP ID with its configuration.
struct doca_flow_crypto_key_cfg {
enum doca_flow_crypto_key_type key_type;
uint32_t *key;
};
struct doca_flow_resource_psp_cfg {
struct doca_flow_crypto_key_cfg key_cfg;
};
struct doca_flow_shared_resource_cfg {
union {
struct doca_flow_resource_psp_cfg psp_cfg;
...
};
};
int
doca_flow_shared_resource_cfg(enum doca_flow_shared_resource_type type, uint32_t id,
struct doca_flow_shared_resource_cfg *cfg,
struct doca_flow_error *error);
On doca_flow_shared_resource_bind()
This call binds a bulk of shared IPsec SAs IDs to a specific pipe or port.
doca_error_t
doca_flow_shared_resources_bind(enum doca_flow_shared_resource_type type, uint32_t *res_array,
uint32_t res_array_len, void *bindable_obj);
res_array [in] – Array of shared crypto IDs to be bound.
res_array_len [in] – Array length.
bindable_obj – Pointer to either a pipe or port.
This call allocates the PSP objects. A PSP ID specified in this array can only be used later by the corresponding bindable object (pipe or port).
The following example binds PSP IDs 8, 9, 15, and 25 to a pipe. The PSP's IDs must be within the range 0-(nb_shared_psp-1).
uint32_t shared_psp_ids[] = {8, 9, 15, 25};
struct doca_flow_pipe *pipe = ...
doca_flow_shared_resources_bind(
DOCA_FLOW_SHARED_RESOURCE_IPSEC_SA,
shared_ipsec_sa_ids, 4, pipe, &error);
On doca_flow_pipe_add_entry() or Pipe Configuration (struct doca_flow_pipe_cfg)
The shared PSP ID is included in the action parameter. It must be bound in advance to the encrypt pipe object.
struct doca_flow_actions {
...
struct {
...
uint32_t crypto_id;
/**< Crypto shared action id */
} crypto;
}
Crypto operations are performed over the packets matching the pipe entry according to the crypto_id configuration. Afterwards, the flow continues from the point specified in the forward part of the pipe configuration.
On doca_flow_pipe_destroy() or doca_flow_port_stop()
All bound crypto resource IDs of this pipe or port are destroyed.
A shared mirror can be used in multiple pipe entries (hardware steering mode support only). The following are the steps for configuring and using shared mirrors.
On doca_flow_init()
Specify the total number of shared mirrors to be used, nr_shared_mirrors.
The following call is an example for how to initialize both shared counters and mirror ranges. This call implicitly defines the shared counter IDs in the range of 0-(nr_shared_counters-1) and the shared mirror IDs in the range of 0-(nr_shared_mirrors-1).
struct doca_flow_cfg *cfg;
doca_flow_cfg_create(&cfg);
doca_flow_cfg_set_pipe_queues(cfg, queues);
...
doca_flow_cfg_set_nr_shared_resource(cfg, nr_shared_mirrors, DOCA_FLOW_SHARED_RESOURCE_MIRROR);
doca_flow_cfg_set_nr_shared_resource(cfg, nr_shared_counters, DOCA_FLOW_SHARED_RESOURCE_COUNT);
doca_flow_init(cfg);
doca_flow_cfg_destroy(cfg);
On doca_flow_shared_resource_cfg()
This call binds a specific mirror ID with its mirror packet destination and original packet destination.
struct doca_flow_mirror_target {
bool has_encap;
/**< Encap mirrored packets. */
struct doca_flow_encap_action encap;
/**< Encap data. */
struct doca_flow_fwd fwd;
/**< Mirror target, must be filled. */
};
struct doca_flow_resource_mirror_cfg {
int nr_targets;
/**< Mirror target number. */
struct doca_flow_mirror_target *target;
/**< Mirror target pointer. */
struct doca_flow_fwd fwd;
/**< Original packet dst, can be filled optional. */
};
struct doca_flow_shared_resource_cfg {
union {
struct doca_flow_resource_mirror_cfg mirror_cfg;
...
};
};
int
doca_flow_shared_resource_cfg(enum doca_flow_shared_resource_type type, uint32_t id,
struct doca_flow_shared_resource_cfg *cfg,
struct doca_flow_error *error);
The following example configures the shared mirror ID 5 with mirroring the packet to the second hairpin port:
struct doca_flow_shared_resource_cfg shared_cfg = { 0 };
target.fwd.type = DOCA_FLOW_FWD_PORT;
target.fwd.port_id = 0;
shared_cfg.mirror_cfg.nr_targets = 1;
shared_cfg.mirror_cfg.target = ⌖
doca_flow_shared_resource_cfg(DOCA_FLOW_SHARED_RESOURCE_MIRROR, 0x5, &shared_cfg, &error);
On doca_flow_shared_resource_bind()
This call binds a bulk of shared mirror IDs to a specific pipe or port.
doca_error_t
doca_flow_shared_resources_bind(enum doca_flow_shared_resource_type type, uint32_t *res_array,
uint32_t res_array_len, void *bindable_obj);
res_array [in] – array of shared mirror IDs to be bound
res_array_len [in] – array length
bindable_obj – pointer to either a pipe or port
This call allocates the mirror's objects. A mirror ID specified in this array can only be used later by the corresponding bindable object (i.e., pipe or port).
Mirror can only be used with a BASIC pipe.
The following example binds mirror IDs 5 and 14 to a pipe. The mirror IDs must be within the range 0-(nb_shared_mirrors-1)
uint32_t shared_mirrors_ids[] = {5, 14};
struct doca_flow_pipe *pipe = ...
doca_flow_shared_resources_bind(
DOCA_FLOW_SHARED_RESOURCE_MIRROR,
shared_mirrors_ids, 2, pipe, &error);
On doca_flow_pipe_add_entry() or Pipe Configuration (struct doca_flow_pipe_cfg)
The shared mirror ID is included in the monitor parameter. It must be bound in advance to the pipe object.
struct doca_flow_monitor {
...
uint32_t shared_mirror_id;
/**< shared mirror id */
...
}
Packets matching the pipe entry are mirrored to the targets related to the shared_mirror_id. In the pipe configuration, the shared_mirror_id can be changeable (all FFs) and then the pipe entry must hold the specific shared mirror ID for that entry.
Mirror is not allowed to be used on NIC Tx ("hws,VNF,Tx").
Mirror can only be used with a BASIC pipe.
Querying Bulk of Shared Mirror IDs
Query is not supported with mirror.
On doca_flow_pipe_destroy() or doca_flow_port_stop()
All bound resource IDs of this pipe or port are destroyed.
A shared encap can be used in multiple pipe entries (hardware steering mode support only). The following are the steps for configuring and using shared encaps.
On doca_flow_init()
Specifies the total number of shared encaps to be used, nr_shared_encaps. This call implicitly defines the shared encap IDs in the range of 0-(nr_shared_encaps-1) and the shared encap IDs in the range of 0-(nr_shared_encaps-1).
The following call is an example for how to initialize shared encaps:
struct doca_flow_cfg *cfg;
doca_flow_cfg_create(&cfg);
doca_flow_cfg_set_pipe_queues(cfg, queues);
...
doca_flow_cfg_set_nr_shared_resource(cfg, nr_shared_encaps, DOCA_FLOW_SHARED_RESOURCE_ENCAP);
doca_flow_init(cfg);
doca_flow_cfg_destroy(cfg);
On doca_flow_shared_resource_cfg()
This call binds a specific encap ID with its encap configuration:
struct doca_flow_resource_encap_cfg {
bool is_l2;
/**< L2 or L3 tunnel flavor */
struct doca_flow_encap_action encap;
/**< Encap data */
};
struct doca_flow_shared_resource_cfg {
union {
struct doca_flow_resource_encap_cfg encap_cfg;
...
};
};
int
doca_flow_shared_resource_cfg(enum doca_flow_shared_resource_type type, uint32_t id,
struct doca_flow_shared_resource_cfg *cfg);
The following example configures the shared encap ID 5 with VXLAN encap:
struct doca_flow_shared_resource_cfg shared_cfg = { 0 };
shared_cfg.domain = DOCA_FLOW_PIPE_DOMAIN_EGRESS;
shared_cfg.encap_cfg.is_l2 = true;
shared_cfg.encap_cfg.encap.outer.eth = {
.src_mac = {0xaa, 0xbb, 0xcc, 0xdd, 0xee, 0xff},
.dst_mac = {0x11, 0x22, 0x33, 0x44, 0x55, 0x66},
};
shared_cfg.encap_cfg.encap.outer.l3_type = DOCA_FLOW_L3_TYPE_IP4;
shared_cfg.encap_cfg.encap.outer.ip4.src_ip = 0x81818181;
shared_cfg.encap_cfg.encap.outer.ip4.dst_ip = 0x11213141;
shared_cfg.encap_cfg.encap.outer.ip4.ttl = 17;
shared_cfg.encap_cfg.encap.outer.l4_type_ext = DOCA_FLOW_L4_TYPE_EXT_UDP;
shared_cfg.encap_cfg.encap.outer.udp.l4_port.dst_port = RTE_BE16(DOCA_FLOW_VXLAN_DEFAULT_PORT);
shared_cfg.encap_cfg.encap.tun.type = DOCA_FLOW_TUN_VXLAN;
shared_cfg.encap_cfg.encap.tun.vxlan_tun_id = 0x1c;
doca_flow_shared_resource_cfg(DOCA_FLOW_SHARED_RESOURCE_ENCAP, 0x5, &shared_cfg);
On doca_flow_shared_resource_bind()
This call binds a bulk of shared encap IDs to a specific pipe or port:
doca_error_t
doca_flow_shared_resources_bind(enum doca_flow_shared_resource_type type, uint32_t *res_array,
uint32_t res_array_len, void *bindable_obj);
res_array [in] – array of shared encap IDs to be bound
res_array_len [in] – array length
bindable_obj – pointer to either a pipe or port
This call allocates the encaps's objects. An encap ID specified in this array can only be used later by the corresponding bindable object (i.e., pipe or port).
The following example binds encap IDs 5 and 14 to a port. The encap IDs must be within the range 0-(nb_shared_encaps-1)
uint32_t shared_encap_ids[] = {5, 14};
struct doca_flow_port *port = ...
doca_flow_shared_resources_bind(
DOCA_FLOW_SHARED_RESOURCE_ENCAP,
shared_encap_ids, 2, port);
On doca_flow_pipe_add_entry() or Pipe Configuration (struct doca_flow_pipe_cfg)
The shared encap ID is included in the encap config parameter. It must be bound in advance to the port/pipe.
Packets matching the pipe entry are encapped related to the shared_encap_id.
Querying Bulk of Shared Encap IDs
Query is not supported with encap.
On doca_flow_pipe_destroy() or doca_flow_port_stop()
All bound resource IDs of this pipe or port are destroyed.
A shared decap can be used in multiple pipe entries (hardware steering mode support only). The following are the steps for configuring and using shared decaps.
On doca_flow_init()
Specifies the total number of shared decaps to be used, nr_shared_decaps. This call implicitly defines the shared decap IDs in the range of 0-(nr_shared_decaps-1) and the shared decap IDs in the range of 0-(nr_shared_decaps-1).
The following call is an example for how to initialize shared decaps:
struct doca_flow_cfg *cfg;
doca_flow_cfg_create(&cfg);
doca_flow_cfg_set_pipe_queues(cfg, queues);
...
doca_flow_cfg_set_nr_shared_resource(cfg, nr_shared_decaps, DOCA_FLOW_SHARED_RESOURCE_DECAP);
doca_flow_init(cfg);
doca_flow_cfg_destroy(cfg);
On doca_flow_shared_resource_cfg()
This call binds a specific decap ID with its decap configuration:
struct doca_flow_resource_decap_cfg {
bool is_l2;
/**< L2 or L3 tunnel flavor */
struct doca_flow_header_eth eth;
/**< ether head for is_l2 is false */
uint16_t l2_valid_headers;
/**< indicate which headers are valid */
struct doca_flow_header_eth_vlan eth_vlan[DOCA_FLOW_VLAN_MAX];
/**< vlan header array for is_l2 is false */
};
struct doca_flow_shared_resource_cfg {
union {
struct doca_flow_resource_decap_cfg decap_cfg;
...
};
};
int
doca_flow_shared_resource_cfg(enum doca_flow_shared_resource_type type, uint32_t id,
struct doca_flow_shared_resource_cfg *cfg);
The following example configures the shared decap ID 5:
struct doca_flow_shared_resource_cfg shared_cfg = { 0 };
shared_cfg.domain = DOCA_FLOW_PIPE_DOMAIN_EGRESS;
shared_cfg.decap_cfg = (struct doca_flow_resource_decap_cfg) {
.is_l2 = false,
.eth = {.dst_mac = {0x11, 0x21, 0x31, 0x41, 0x51, 0x61},
.type = RTE_BE16(DOCA_FLOW_ETHER_TYPE_IPV4)},};
doca_flow_shared_resource_cfg(DOCA_FLOW_SHARED_RESOURCE_DECAP, 0x5, &shared_cfg);
On doca_flow_shared_resource_bind()
This call binds a bulk of shared decap IDs to a specific pipe or port:
doca_error_t
doca_flow_shared_resources_bind(enum doca_flow_shared_resource_type type, uint32_t *res_array,
uint32_t res_array_len, void *bindable_obj);
res_array [in] – array of shared decap IDs to be bound
res_array_len [in] – array length
bindable_obj – pointer to either a pipe or port
This call allocates the decaps's objects. An decap ID specified in this array can only be used later by the corresponding bindable object (i.e., pipe or port).
The following example binds decap IDs 5 and 14 to a port. The decap IDs must be within the range 0-(nb_shared_decaps-1).
uint32_t shared_decap_ids[] = {5, 14};
struct doca_flow_port *port = ...
doca_flow_shared_resources_bind(
DOCA_FLOW_SHARED_RESOURCE_DECAP,
shared_decap_ids, 2, port);
On doca_flow_pipe_add_entry() or Pipe Configuration (struct doca_flow_pipe_cfg)
The shared decap ID is included in the decap config parameter. It must be bound in advance to the port/pipe.
Packets matching the pipe entry are decapped related to the shared_decap_id.
Querying Bulk of Shared Decap IDs
Query is not supported with decap.
On doca_flow_pipe_destroy() or doca_flow_port_stop()
All bound resource IDs of this pipe or port are destroyed.
Initialization Flow
Before using any DOCA Flow function, it is mandatory to call DOCA Flow initialization, doca_flow_init(), which initializes all resources used by DOCA Flow.
Pipe Mode
This mode ( mode_args) defines the basic traffic in DOCA. It creates some miss rules when a DOCA port initializes. Currently, DOCA supports 3 modes:
vnf
A packet arriving from one of the device's ports is processed, and can be sent to another port. By default, missed packets go to RSS.
The following diagram shows the basic traffic flow in vnf mode. Packet1 firstly misses and is forwarded to host RSS. The app captures this packet and decides how to process it and then creates a pipe entry. Packet2 will hit this pipe entry and do the action, for example, for VXLAN, will do decap, modify, and encap, then is sent out from P1.
switch
Used for internal switching, only representor ports are allowed, for example, uplink representors and SF/VF representors. Packet is forwarded from one port to another. If a packet arrives from an uplink and does not hit the rules defined by the user's pipe, then the packet is received on all RSS queues of the representor of the uplink.
The following diagram shows the basic flow of traffic in switch mode. Packet1 firstly misses to host RSS queues. The app captures this packet and decides to which representor the packet goes, and then sets the rule. Packets hit this rule and go to representor0.
doca_dev field is mandatory in doca_flow_port_cfg (using doca_flow_port_cfg_set_dev()) and isolated mode should be specified.
NoteThe application must avoid initialization of the VF/SF representor ports in DPDK API (i.e., the following functions rte_eth_dev_configure(), rte_eth_rx_queue_setup(), rte_eth_dev_start() must not be called for VF/SF representor ports).
DOCA Flow switch mode unifies all the ports to the switch manager port for traffic management. This means that all the traffic is handled by switch manager port. Users only have to create an RSS pipe on the switch manager port to get the missed traffic, and they should only manage the pipes on the switch manager port. Switch mode can work with two different mode_args configurations: With or without expert. The way to retrieve the miss traffic source's port_id depends on this configuration:
NoteTraffic missed from user's pipe without fwd_miss target specified is sent to kernel.
If expert is not set, the traffic misses to software would be tagged with port_id information in the mbuf CQE field to allow users to deduce the source port_id. Meanwhile, users can set the destination port_id to mbuf meta and the packet is sent out directly to the destination port based on the meta information.
InfoPlease refer to the "Flow Switch to Wire" sample to get more information regarding the port_id management with missed traffic mbuf.
If expert is set, the port_id is not added to the packet. Users can configure the pipes freely to implement their own solution.
NoteTraffic cloned from the VF to the RSS pipe misses its port_id information due to firmware limitation.
remote-vnf
Remote mode is a BlueField mode only, with two physical ports (uplinks). Users must use doca_flow_port_pair to pair one physical port and one of its representors. A packet from this uplink, if it does not hit any rules from the users, is firstly received on this representor. Users must also use doca_flow_port_pair to pair two physical uplinks. If a packet is received from one uplink and hits the rule whose FWD action is to another uplink, then the packets are sent out from it.
The following diagram shows the basic traffic flow in remote-vnf mode. Packet1, from BlueField uplink P0, firstly misses to host VF0. The app captures this packet and decides whether to drop it or forward it to another uplink (P1). Then, using gRPC to set rules on P0, packet2 hits the rule, then is either dropped or is sent out from P1.
Start Point
DOCA Flow API serves as an abstraction layer API for network acceleration. The packet processing in-network function is described from ingress to egress and, therefore, a pipe must be attached to the origin port. Once a packet arrives to the ingress port, it starts the hardware execution as defined by the DOCA API.
doca_flow_port is an opaque object since the DOCA Flow API is not bound to a specific packet delivery API, such as DPDK. The first step is to start the DOCA Flow port by calling doca_flow_port_start(). The purpose of this step is to attach user application ports to the DOCA Flow ports.
When DPDK is used, the following configuration must be provided:
enum doca_flow_port_type type = DOCA_FLOW_PORT_DPDK_BY_ID;
const char *devargs = "1";
The devargs parameter points to a string that has the numeric value of the DPDK port_id in decimal format. The port must be configured and started before calling this API. Mapping the DPDK port to the DOCA port is required to synchronize application ports with hardware ports.
Create Pipe and Pipe Entry
Pipe is a template that defines packet processing without adding any specific hardware rule. A pipe consists of a template that includes the following elements:
Match
Monitor
Actions
Forward
The following diagram illustrates a pipe structure.
The creation phase allows the hardware to efficiently build the execution pipe. After the pipe is created, specific entries can be added. A subset of the pipe may be used (e.g., skipping the monitor completely, just using the counter, etc).
Matching
This section explains the concept of matching. Conceptually, the following logic is followed:
The packet enters the green filter which modifies it by masking it with the value A. The output value, P&A, is then compared to the value B, and if they are equal, then that is a match.
The values of A and B are evaluated according to the values of the pipe configuration and entry configuration fields, according to the tables in sections "Implicit Match" and "Explicit Match".
Setting Pipe Match
Match is a mandatory parameter when creating a pipe. Using the doca_flow_match struct, users must define the packet fields to be matched by the pipe.
For each doca_flow_match field, users select whether the field type is:
Ignore (match any) – the value of the field is ignored in a packet. In other words, match on any value of the field.
Constant – all entries in the pipe have the same value for this field. Users should not put a value for each entry.
Changeable – the value of the field is defined per entry. Users must provide it upon adding an entry.
NoteL4 type, L3 type, and tunnel type cannot be changeable.
The match field type can be defined either implicitly or explicitly using the doca_flow_pipe_cfg_set_match(struct doca_flow_pipe_cfg *cfg, const doca_flow_match *match, const doca_flow_match *match_mask) function. If match_mask == NULL, then it is done implicitly. Otherwise, it is explicit.
In the tables in the following subsections, an example is used of a 16-bit field (such as layer-4 destination port) where:
The same concept would apply to any other field (such as an IP address occupying 32 bits).
P stands for the packet field value
V stands for the pipe match field value
M stands for the pipe mask field value
E stands for the match entry field value
Implicit Match
|
Match Type |
Pipe Match Value (V) |
Pipe Match Mask (M) |
Entry Match Value (E) |
Filter (A) |
Rule (B) |
|
Ignore |
0 |
NULL |
N/A |
0 |
0 |
|
Constant |
0<V<0xffff |
NULL |
N/A |
0xffff |
V |
|
Changeable (per entry) |
0xffff |
NULL |
0≤E≤0xffff |
0xffff |
E |
To match implicitly, the following considerations should be taken into account.
Ignored fields:
Field is zeroed
Pipeline has no comparison on the field
Constant fields
These are fields that have a constant value among all entries. For example, as shown in the following, the tunnel type is VXLAN.
match.tun.type = DOCA_FLOW_TUN_VXLAN;
These fields must only be configured once at pipe build stage, not once per new pipeline entry.
Changeable fields
These are fields whose value may change per entry. For example, the following shows match on a destination IPv4 address of variable per-entry value (outer 5-tuple):
match.outer.ip4.dst_ip = 0xffffffff;
The following is an example of a match, where:
Outer 5-tuple
L3 type is IPv4 – constant among entries by design
L4 type is UDP – constant among entries by design
Tunnel type is DOCA_FLOW_TUN_VXLAN – constant among entries by design
IPv4 destination address varies per entry
UDP destination port is always DOCA_VXLAN_DEFAULT_PORT
VXLAN tunnel ID varies per entry
The rest of the packet fields are ignored
Inner 5-tuple
L3 type is IPv4 – constant among entries by design
L4 type is TCP – constant among entries by design
IPv4 source and destination addresses vary per entry
TCP source and destination ports vary per entry
The rest of the packet fields are ignored
// filter creation
static void build_underlay_overlay_match(struct doca_flow_match *match)
{
//outer
match->outer.l3_type = DOCA_FLOW_L3_TYPE_IP4;
match->outer.l4_type_ext = DOCA_FLOW_L4_TYPE_EXT_UDP;
match->tun.type = DOCA_FLOW_TUN_VXLAN;
match->outer.ip4.dst_ip = 0xffffffff;
match->outer.udp.l4_port.dst_port = DOCA_VXLAN_DEFAULT_PORT;
match->tun.vxlan_tun_id = 0xffffffff;
//inner
match->inner.l3_type = DOCA_FLOW_L3_TYPE_IP4;
match->inner.l4_type_ext = DOCA_FLOW_L4_TYPE_EXT_TCP;
match->inner.ip4.dst_ip = 0xffffffff;
match->inner.ip4.src_ip = 0xffffffff;
match->inner.tcp.l4_port.src_port = 0xffff;
match->inner.tcp.l4_port.dst_port = 0xffff;
}
// create entry specifying specific values to match upon
doca_error_t add_entry(struct doca_flow_pipe *pipe, struct doca_flow_port *port,
struct doca_flow_pipe_entry **entry)
{
struct doca_flow_match match = {};
struct entries_status status = {};
doca_error_t result;
match.outer.ip4.dst_ip = BE_IPV4_ADDR(7, 7, 7, 1);
match.tun.vxlan_tun_id = RTE_BE32(9876);
match.inner.ip4.src_ip = BE_IPV4_ADDR(8, 8, 8, 1);
match.inner.ip4.dst_ip = BE_IPV4_ADDR(9, 9, 9, 1);
match.inner.tcp.l4_port.src_port = rte_cpu_to_be_16(5678);
match.inner.tcp.l4_port.dst_port = rte_cpu_to_be_16(1234);
result = doca_flow_pipe_add_entry(0, pipe, &match, &actions, NULL, NULL, 0, &status, entry);
}
The fields of the doca_flow_meta struct inside the match are not subject to implicit match rules and must be paired with explicit mask values.
Explicit Match
|
Match Type |
Pipe Match Value (V) |
Pipe Match Mask (M) |
Entry Match Value (E) |
Filter (A) |
Rule (B) |
|
Constant |
V!=0xffff |
0<M≤0xffff |
0≤E≤0xffff |
M |
M&V |
|
Changeable |
V==0xffff |
0<M≤0xffff |
0≤E≤0xffff |
M |
M&E |
|
Ignored |
0≤V<0xffff |
M==0 |
0≤E≤0xffff |
0 |
0 |
In this case, there are two doca_flow_match items, the following considerations should be considered:
Ignored fields
M equals zero. This can be seen from the table where the rule equals 0. Since mask is also 0, the resulting packet after the filter is0. Thus, the comparison always succeeds.
match_mask.inner.ip4.dst_ip = 0;
Constant fields
These are fields that have a constant value. For example, as shown in the following, the inner 5-tuple match on IPv4 destination addresses belonging to the 0.0.0.0/24 subnet, and this match is constant among all entries:
// BE_IPV4_ADDR converts 4 numbers A,B,C,D to a big endian representation of IP address A.B.C.D match.inner.ip4.dst_ip = 0; match_mask.inner.ip4.dst_ip = BE_IPV4_ADDR(255, 255, 255, 0);
For example, as shown in the following, the inner 5-tuple match on IPv4 destination addresses belonging to the 1.2.0.0/16 subnet, and this match is constant among all entries. The last two octets of the match.inner.ip4.dst_ip are ignored because the match_mask of 255.255.0.0 is applied:
// BE_IPV4_ADDR converts 4 numbers A,B,C,D to a big endian representation of IP address A.B.C.D match.inner.ip4.dst_ip = BE_IPV4_ADDR(1, 2, 3, 4); match_mask.inner.ip4.dst_ip = BE_IPV4_ADDR(255, 255, 0, 0);
Once a field is defined as constant, the field's value cannot be changed per entry.
TipUsers should set constant fields to zero when adding entries for better code readability.
A more complex example of constant matches may be achieved as follows:
match_mask.outer.tcp.l4_port.dst_port = rte_cpu_to_be_16(0xf0f0); match.outer.tcp.l4_port.dst_port = rte_cpu_to_be_16(0x5020)
The following ports would be matched:
0x5020 - 0x502f
0x5120 - 0x512f
...
0x5f20 - 0x5f2f
Changeable fields
The following example matches on either FTP or TELNET well known port numbers and forwards packets to a server after modifying the destination IP address and destination port numbers. In the example, either FTP or TELNET are forwarded to the same server. FTP is forwarded to port 8000 and TELNET is forwarded to port 9000.
// at Pipe creation
pipe_cfg.attr.name = "PORT_MAPPER";
pipe_cfg.attr.type = DOCA_FLOW_PIPE_BASIC;
match.outer.tcp.l4_port.dst_port = rte_cpu_to_be_16(0xffff); // v
match_mask.outer.tcp.l4_port.dst_port = rte_cpu_to_be_16(0xffff); // M
pipe_cfg.match_mask = &match_mask;
pipe_cfg.match = &match;
actions_arr[0] = &actions;
pipe_cfg.actions = actions_arr;
pipe_cfg.attr.is_root = true;
pipe_cfg.attr.nb_actions = 1;
// Adding entries
// FTP
match.outer.tcp.l4_port.dst_port = rte_cpu_to_be_16(20); // E
actions.outer.ip4.src_ip = server_addr;
actions.outer.tcp.l4_port.dst_port = rte_cpu_to_be_16(8000);
result = doca_flow_pipe_add_entry(0, pipe, &match, &actions, NULL, NULL, 0, &status, entry);
// TELNET
match.outer.tcp.l4_port.dst_port = rte_cpu_to_be_16(23); // E
actions.outer.ip4.src_ip = server_addr;
actions.outer.tcp.l4_port.dst_port = rte_cpu_to_be_16(9000);
result = doca_flow_pipe_add_entry(0, pipe, &match, &actions, NULL, NULL, 0, &status, entry);
Relaxed Match
Relaxed matching is the default working mode in DOCA flow. However, it can be disabled per pipe using the enable_strict_matching pipe attribute. This mode grants the user more control on matching fields such that only explicitly set match fields by the user (either specific or changeable) are matched by the pipe.
Consider the following strict matching mode example. There are three pipes:
Basic pipe A with match.outer.l4_type_ext = DOCA_FLOW_L4_TYPE_EXT_TCP; and match.outer.tcp.flags = 1;
Basic pipe B with match.outer.l4_type_ext = DOCA_FLOW_L4_TYPE_EXT_UDP; and match.outer.udp.l4_port.src_port = 8080;
Control pipe X with two entries to direct TCP traffic to pipe A and UDP to pipe B. The first entry has match.outer.l4_type_ext = DOCA_FLOW_L4_TYPE_EXT_TCP; while the second has match.outer.l4_type_ext = DOCA_FLOW_L4_TYPE_EXT_UDP;.
As a result, the hardware performs match on the L4 header type twice:
First, when the packet enters the filter in control pipe X to decide the next pipe
Second, when the packet enters the filter of pipe A or pipe B to do the match on L4 header fields
With particularly large pipelines, such double matches decrease performance and increase the memory footprint in hardware. Relaxed matching mode gives the user greater control of the match to solve the performance problems.
In relaxed mode, type selectors in the outer, inner, and tun parts of the doca_flow_match are used only for the type cast (or selectors) of the underlying unions. Header-type matches are available using the parser_meta API.
Thus, the aforementioned scenario may be overwritten in the following manner. There are three pipes:
Basic pipe A with match.outer.l4_type_ext = DOCA_FLOW_L4_TYPE_EXT_TCP; and match.outer.tcp.flags = 1;
Basic pipe B with match.outer.l4_type_ext = DOCA_FLOW_L4_TYPE_EXT_UDP; and match.outer.udp.l4_port.src_port = 8080;
Control pipe X with two entries to direct TCP traffic to pipe A and UDP to pipe B. The first entry has match.parser_meta.outer_l4_type = DOCA_FLOW_L4_META_TCP; while the second has match.parser_meta.outer_l4_type = DOCA_FLOW_L4_META_UDP;.
As a result, the hardware performs the L4 header-type match only once, when the packet enters the filter of control pipe. Basic pipes' match.outer.l4_type_ext are used only for the selection of the match.outer.tcp or match.outer.udp structures.
Example
The following code snippet is used to demonstrate relaxed matching mode:
// filter creation
static void build_underlay_overlay_match(struct doca_flow_match *match)
{
//outer
match->outer.l3_type = DOCA_FLOW_L3_TYPE_IP4;
match->outer.l4_type_ext = DOCA_FLOW_L4_TYPE_EXT_UDP;
match->tun.type = DOCA_FLOW_TUN_VXLAN;
match->outer.ip4.dst_ip = 0xffffffff;
match->outer.udp.l4_port.src_port = 22;
match->tun.vxlan_tun_id = 0xffffffff;
}
This match code above is an example of a match where:
With relaxed matching disabled (i.e., enable_strict_matching attribute set to true), the following hardware matches are performed:
L3 type is IPv4 – constant among entries by design
L4 type is UDP – constant among entries by design
Tunnel type is DOCA_FLOW_TUN_VXLAN – constant among entries by design
IPv4 destination address varies per entry
UDP source port is constant among entries
VXLAN tunnel ID varies per entry
The rest of the packet fields are ignored
With relaxed matching enabled (default mode), the following hardware matches are performed :
IPv4 destination address varies per entry
UDP source port is constant among entries
VXLAN tunnel ID varies per entry
In summary, with relaxed matching L3, L4, tunnel protocol types, and similar no longer indicate a match on the specific protocol. They are use d solely as a selector for the relevant header fields. For example, to match on outer.ip4.dst_ip, users must set outer.l3_type = DOCA_FLOW_L3_TYPE_IP4. T hat is, the L3 header is checked for the IPv4 destination address. There is no check that it is of IPv4 type. It is user responsibility to make sure that packets arriving to such a filter indeed have an L3 header of type IPv4 (same goes for L4 UDP header/VXLAN tunnel).
Protocols/Tunnels Type Match
The following section explains how to match on a protocol's and a tunnel's type with relaxed matching.
To match on a specific protocol/tunnel type, consider the following:
To match on an inner/outer L3/L4 protocol type, one can use relevant doca_flow_parser_meta fields (e.g., for outer protocols, parser_meta.outer_l[3,4]_type fields can be used).
To match on a specific tunnel type (e.g., VXLAN/GRE and so on), users should match on a tunnel according to its specification (e.g., for VXLAN, a match on UDP destination port 4789 can be used). Another option is to use the L3 next protocol field (e.g., for IPv4 with next header GRE, one can match on the IPv4 header's next protocol field value to match GRE IP protocol number 47).
Example
Using the aforementioned example, to add the match on the same L3,L4 protocol type and on a VXLAN tunnel with relaxed matching enabled, the following function implementation should be considered:
// filter creation
static void build_underlay_overlay_match(struct doca_flow_match *match)
{
//outer
match->parser_meta.outer_l3_type = DOCA_FLOW_L3_META_IPV4;
match->parser_meta.outer_l4_type = DOCA_FLOW_L4_META_UDP;
match->outer.l3_type = DOCA_FLOW_L3_TYPE_IP4;
match->outer.l4_type_ext = DOCA_FLOW_L4_TYPE_EXT_UDP;
match->tun.type = DOCA_FLOW_TUN_VXLAN;
match->outer.ip4.dst_ip = 0xffffffff;
match->outer.udp.l4_port.src_port = 22;
match->outer.udp.l4_port.dst_port = DOCA_VXLAN_DEFAULT_PORT;
match->tun.vxlan_tun_id = 0xffffffff;
}
The match code above is an example of a match, where:
With relaxed matching disabled (i.e., enable_strict_matching attribute set to true), the following hardware matches are performed:
L3 type is IPv4 – constant among entries by design
L4 type is UDP – constant among entries by design
Tunnel type is DOCA_FLOW_TUN_VXLAN – constant among entries by design
IPv4 destination address varies per entry
UDP source port is always 22
UDP destination port is always DOCA_VXLAN_DEFAULT_PORT
VXLAN tunnel ID varies per entry
The rest of the packet fields are ignored
With relaxed matching enabled (default mode), the following hardware matches are performed:
L3 type is IPv4 – constant among entries by design
L4 type is UDP – constant among entries by design
IPv4 destination address varies per entry
UDP source port is always 22
UDP destination port is always DOCA_VXLAN_DEFAULT_PORT
VXLAN tunnel ID varies per entry
With relaxed matching, if any of the selectors is used without setting a relevant field, the pipe/entry creation would fail with the following error message:
failed building active opcode - active opcode <opcode number> is protocol only
Setting Pipe Actions
Auto-modification
Similarly to setting pipe match, actions also have a template definition.
Similarly to doca_flow_match in the creation phase, only the subset of actions that should be executed per packet are defined. This is done in a similar way to match, namely by classifying a field of doca_flow_match to one of the following:
Ignored field – field is zeroed, modify is not used.
Constant fields – when a field must be modified per packet, but the value is the same for all packets, a one-time value on action definitions can be used
Changeable fields – fields that may have more than one possible value, and the exact values are set by the user per entry
actions.outer.ip4.dst_ip = 0xffffffff
NoteThe action_mask should be set as 0xffffffff and action as 0 if the user wants to configure 0 to this field.
Explicit Modification Type
It is possible to force constant modification or per-entry modification with action mask. For example:
static void
create_constant_modify_actions(struct doca_flow_actions *actions，
struct doca_flow_actions *actions_mask,
struct doca_flow_action_descs *descs)
{
actions->outer.l4_type_ext = DOCA_FLOW_L4_TYPE_EXT_UDP;
actions->outer.udp.src_port = 0x1234;
actions_mask->outer.udp.src_port = 0xffff;
}
Copy Field
The action descriptor can be used to copy between the packet field and metadata. For example:
#define META_U32_BIT_OFFSET(idx) (offsetof(struct doca_flow_meta, u32[(idx)]) << 3)
static void
create_copy_packet_to_meta_actions(struct doca_flow_match *match，
struct doca_flow_action_desc *desc)
{
desc->type = DOCA_FLOW_ACTION_COPY;
desc->field_op.src.field_string = "outer.ipv4.src_ip";
desc->field_op.src.bit_offset = 0;
desc->field_op.dst.field_string = "meta.data";
desc->field_op.dst.bit_offset = META_U32_BIT_OFFSET(1); /* Bit offset of meta.u32[1] */;
}
Multiple Actions List
Creating a pipe is possible using a list of multiple actions. For example:
static void
create_multi_actions_for_pipe_cfg()
{
struct doca_flow_actions *actions_arr[2];
struct doca_flow_actions actions_0 = {0}, actions_1 = {0};
struct doca_flow_pipe_cfg *pipe_cfg;
/* input configurations for actions_0 and actions_1 */
actions_arr[0] = &actions_0;
actions_arr[1] = &actions_1;
doca_flow_pipe_cfg_set_actions(pipe_cfg, actions_arr, NULL, NULL, 2);
}
Summary of Action Types
|
Pipe Creation |
Entry Creation |
Behavior |
|||
|
action_desc |
Pipe Actions |
Pipe Actions Mask |
Entry Actions |
||
|
doca_flow_action_type |
Configuration |
||||
|
DOCA_FLOW_ACTION_AUTO/ action_desc = NULL |
No specific config |
0 |
0 |
N/A |
Field ignored, no modification |
|
0 |
mask != 0 |
N/A |
Apply 0 and mask to all entries |
||
|
val != 0 && val != 0xFF |
mask != 0 |
N/A |
Apply val and mask to all entries |
||
|
val = 0xFF |
mask = 0 |
N/A |
Apply 0xFF to all entries |
||
|
val = 0xFF |
mask != 0 |
Define val per entry |
Apply entry's val and mask |
||
|
DOCA_FLOW_ACTION_ADD Add field value or from src |
Define only the dst field and width |
val != 0 |
N/A |
N/A |
Apply this val to all entries |
|
val == 0 |
N/A |
Define val per entry |
Apply entry's val |
||
|
Define the src and dst fields and width |
Define the source and destination fields.
|
N/A |
N/A |
Add data from src fields to dst for all entries |
|
|
DOCA_FLOW_ACTION_COPY Copy field to another field |
N/A |
Define the source and destination fields.
|
N/A |
N/A |
Copy data between fields for all entries |
Setting Pipe Monitoring
If a meter policer should be used, then it is possible to have the same configuration for all policers on the pipe or to have a specific configuration per entry. The meter policer is determined by the FWD action. If an entry has NULL FWD action, the policer FWD action is taken from the pipe.
If a mirror should be used, mirror can be shared on the pipe or configured to have a specific value per entry.
The monitor also includes the aging configuration, if the aging time is set, this entry ages out if timeout passes without any matching on the entry.
For example:
static void build_entry_monitor(struct doca_flow_monitor *monitor, void *user_ctx)
{
monitor->flags |= DOCA_FLOW_MONITOR_AGING;
monitor->aging_sec = 10;
}
Refer to Pipe Entry Aged Query for more information.
Setting Pipe Forwarding
The FWD (forwarding) action is the last action in a pipe, and it directs where the packet goes next. Users may configure one of the following destinations:
Send to software (representor)
Send to wire
Jump to next pipe
Drop packets
The FORWARDING action may be set for pipe create, but it can also be unique per entry.
A pipe can be defined with constant forwarding (e.g., always send packets on a specific port). In this case, all entries will have the exact same forwarding. If forwarding is not defined when a pipe is created, users must define forwarding per entry. In this instance, pipes may have different forwarding actions.
When a pipe includes meter monitor <cir, cbs>, it must have fwd defined as well as the policer.
If a pipe is created with a dedicate constant mirror with FWD, the pipe FWD can be from a mirror FWD or a pipe FWD and the two FWDs are exclusive. It is not allowed to specify a mirror with a FWD to a pipe with FWD also.
If a mirror FWD is not configured, the FWD is from the pipe configuration. The FWD of the pipe with a mirror cannot be direct RSS, only shared RSS from NULL FWD is allowed.
The following is an RSS forwarding example:
fwd->type = DOCA_FLOW_FWD_RSS;
fwd->rss_queues = queues;
fwd->rss_flags = DOCA_FLOW_RSS_IP | DOCA_FLOW_RSS_UDP;
fwd->num_of_queues = 4;
Queues point to the uint16_t array that contains the queue numbers. When a port is started, the number of queues is defined, starting from zero up to the number of queues minus 1. RSS queue numbers may contain any subset of those predefined queue numbers. For a specific match, a packet may be directed to a single queue by having RSS forwarding with a single queue.
Changeable RSS forwarding is supported. When creating the pipe, the num_of_queues must be set to 0xffffffff, then different forwarding RSS information can be set when adding each entry.
fwd->num_of_queues = 0xffffffff;
The packet is directed to the port. In many instances the complete pipe is executed in the hardware, including the forwarding of the packet back to the wire. The packet never arrives to the software.
Example code for forwarding to port:
struct doca_flow_fwd *fwd = malloc(sizeof(struct doca_flow_fwd));
memset(fwd, 0, sizeof(struct doca_flow_fwd));
fwd->type = DOCA_FLOW_FWD_PORT;
fwd->port_id = port_id; // this should the same port_id that was set in doca_flow_port_cfg_set_devargs()
The type of forwarding is DOCA_FLOW_FWD_PORT and the only data required is the port_id as defined in DOCA_FLOW_PORT.
Changeable port forwarding is also supported. When creating the pipe, the port_id must be set to 0xffff, then different forwarding port_id values can be set when adding each entry.
fwd->port_id = 0xffff;
Basic Pipe Create
Once all parameters are defined, the user should call doca_flow_pipe_create to create a pipe.
The return value of the function is a handle to the pipe. This handle should be given when adding entries to pipe. If a failure occurs, the function returns NULL, and the error reason and message are put in the error argument if provided by the user.
Refer to the NVIDIA DOCA Library APIs to see which fields are optional and may be skipped. It is typically recommended to set optional fields to 0 when not in use. See Miss Pipe and Control Pipe for more information.
Once a pipe is created, a new entry can be added to it. These entries are bound to a pipe, so when a pipe is destroyed, all the entries in the pipe are removed. Please refer to section Pipe Entry for more information.
There is no priority between pipes or entries. The way that priority can be implemented is to match the highest priority first, and if a miss occurs, to jump to the next PIPE. There can be more than one PIPE on a root as long the pipes are not overlapping. If entries overlap, the priority is set according to the order of entries added. So, if two pipes have overlapping matching and PIPE1 has higher priority than PIPE2, users should add an entry to PIPE1 after all entries are added to PIPE2.
Pipe Entry (doca_flow_pipe_add_entry)
An entry is a specific instance inside of a pipe. When defining a pipe, users define match criteria (subset of fields to be matched), the type of actions to be done on matched packets, monitor, and, optionally, the FWD action.
When a user calls doca_flow_pipe_add_entry() to add an entry, they should define the values that are not constant among all entries in the pipe. And if FWD is not defined then that is also mandatory.
DOCA Flow is designed to support concurrency in an efficient way. Since the expected rate is going to be in millions of new entries per second, it is mandatory to use a similar architecture as the data path. Having a unique queue ID per core saves the DOCA engine from having to lock the data structure and enables the usage of multiple queues when interacting with hardware.
Each core is expected to use its own dedicated pipe_queue number when calling doca_flow_pipe_entry. Using the same pipe_queue from different cores causes a race condition and has unexpected results.
Failure Path
Entry insertion can fail in two places, add_entry and add_entry_cb.
When add_entry fails, no cleanup is required.
When add_entry succeeds, a handle is returned to the user. If the subsequent add_entry_cb fails, the user is responsible for releasing the handle through a rm_entry call. This rm_entry call is expected to return DOCA_SUCCESS and is expected to invoke doca_rm_entry_cb with a successful return code.
Pipe Entry Counting
By default, no counter is added. If defined in monitor, a unique counter is added per entry.
Having a counter per entry affects performance and should be avoided if it is not required by the application.
The retrieved statistics are stored in struct doca_flow_query.
Pipe Entry Aged Query
When a user calls doca_flow_aging_handle(), this query is used to get the aged-out entries by the time quota in microseconds. The user callback is invoked by this API with the aged entries.
Since the number of flows can be very large, the query of aged flows is limited by a quota in microseconds. This means that it may return without all flows and requires the user to call it again. When the query has gone over all flows, a full cycle is done.
Pipe Entry With Multiple Actions
Users can define multiple actions per pipe. This gives the user the option to define different actions per entry in the same pipe by providing the action_idx in struct doca_flow_actions.
For example, to create two flows with the same match but with different actions, users can provide two actions upon pipe creation, Action_0 and Action_1, which have indices 0 and 1 respectively in the actions array in the pipe configuration. Action_0 has modify_mac, and Action_1 has modify_ip.
Users can also add two kinds of entries to the pipe, the first one with Action_0 and the second with Action_1. This is done by assigning 0 in the action_idx field in struct doca_flow_actions when creating the first entry and 1 when creating the second one.
Miss Pipe and Control Pipe
Only one root pipe is allowed. If more than one is needed, create a control pipe as root and forward the packets to relevant non-root pipes.
To set priority between pipes, users must use miss-pipes. Miss pipes allow to look up entries associated with pipe X, and if there are no matches, to jump to pipe X+1 and perform a lookup on entries associated with pipe X+1.
The following figure illustrates the hardware table structure:
The first lookup is performed on the table with priority 0. If no hits are found, then it jumps to the next table and performs another lookup.
The way to implement a miss pipe in DOCA Flow is to use a miss pipe in FWD. In struct doca_flow_fwd, the field next_pipe signifies that when creating a pipe, if a fwd_miss is configured then if a packet does not match the specific pipe, steering should jump to next_pipe in fwd_miss.
fwd_miss is of type struct doca_flow_fwd but it only implements two forward types of this struct:
DOCA_FLOW_FWD_PIPE – forwards the packet to another pipe
DOCA_FLOW_FWD_DROP – drops the packet
Other forwarding types (e.g., forwarding to port or sending to RSS queue) are not supported.
next_pipe is defined as doca_flow_pipe and created by doca_flow_pipe_create. To separate miss_pipe and a general one, is_root is introduced in struct doca_flow_pipe_cfg. If is_root is true, it means the pipe is a root pipe executed on packet arrival. Otherwise, the pipe is next_pipe.
When fwd_miss is not null, the packet that does not match the criteria is handled by next_pipe which is defined in fwd_miss.
In internal implementations of doca_flow_pipe_create, if fwd_miss is not null and the forwarding action type of miss_pipe is DOCA_FLOW_FWD_PIPE, a flow with the lowest priority is created that always jumps to the group for the next_pipe of the fwd_miss. Then the flow of next_pipe can handle the packets, or drop the packets if the forwarding action type of miss_pipe is DOCA_FLOW_FWD_DROP.
For example, VXLAN packets are forwarded as RSS and hairpin for other packets. The miss_pipe is for the other packets (non-VXLAN packets) and the match is for general Ethernet packets. The fwd_miss is defined by miss_pipe and the type is DOCA_FLOW_FWD_PIPE. For the VXLAN pipe, it is created by doca_flow_create() and fwd_miss is introduced.
Since, in the example, the jump flow is for general Ethernet packets, it is possible that some VXLAN packets match it and cause conflicts. For example, VXLAN flow entry for ipA is created. A VXLAN packet with ipB comes in, no flow entry is added for ipB, so it hits miss_pipe and is hairpinned.
A control pipe is introduced to handle the conflict. When a user calls doca_flow_create_control_pipe(), the new control pipe is created without any configuration except for the port. Then the user can add different matches with different forwarding and priorities when there are conflicts.
The user can add a control entry by calling doca_flow_control_pipe_add_entry().
priority must be defined as higher than the lowest priority (3) and lower than the highest one (0).
The other parameters represent the same meaning of the parameters in doca_flow_pipe_create. In the example above, a control entry for VXLAN is created. The VLXAN packets with ipB hit the control entry.
doca_flow_pipe_lpm
doca_flow_pipe_lpm uses longest prefix match (LPM) matching. LPM matching is limited to a single field of the match provided by the user at pipe creation (e.g., the outer destination IP). Each entry is consisted of a value and a mask (e.g., 10.0.0.0/8, 10.10.0.0/16, etc). The LPM match is defined as the entry that has the maximum matching bits. For example, using the two entries 10.7.0.0/16 and 10.0.0.0/8, the IP 10.1.9.2 matches on 10.0.0.0/8 and IP 10.7.9.2 matches on 10.7.0.0/16 because 16 bits are the longest prefix matched.
In addition to the longest prefix match logic, LPM supports exact match (EM) logic on the meta.u32. Only index 1 is supported for meta.u32. EM logic can be used for various applications, including filtering known VLAN/VNI/etc. from unknown, by copying the VLAN/VNI/etc. to the meta.u32[1] on pipes before LPM. EM is performed at the same time as LPM matching. That is, logical AND is applied for both conditions: If there is a match on LPM logic, but value in meta.u32[1] is not exactly matched—this is LPM pipe miss.
To enable EM logic in LPM pipe, two steps are required:
Provide match_mask to the LPM pipe creation with meta.u32[1] being fully masked (i.e., UINT32_MAX value). Thus, the match parameter is responsible for the choice of field for LPM logic, while the match_mask parameter is responsible for the enablement of EM logic. Separation into two parameters is done to distinguish which field is for LPM logic and which is for EM logic, when both fields can be used for LPM (e.g., destination IP address and source MAC address).
Per entry, provide values to do exact match using the match structure. match_mask is used only for LPM-related masks and is not involved into EM logic.
EM logic allows inserting many entries with different meta values for the same pair of LPM-related data. Regarding IPv4-based LPM logic with exact match enabled: LPM pipe can have 1.1.1.1/32 with meta 42, 555, 1020. If a packet with 1.1.1.1/32 goes through such an LPM pipe, its meta value is compared against 42, 555, and 1020.
The actions and FWD of the DOCA Flow LPM pipe works the same as the basic DOCA Flow pipe.
The monitor only supports non-shared counters in the LPM pipe.
doca_flow_pipe_lpm insertion max latency can be measured in milliseconds in some cases and, therefore, it is better to insert it from the control path. To get the best insertion performance, entries should be added in large batches.
An LPM pipe cannot be a root pipe. You must create a pipe as root and forward the packets to the LPM pipe.
For monitoring, an LPM pipe only supports non-shared counters and does not support other capabilities of doca_flow_monitor.
doca_flow_pipe_acl
doca_flow_pipe_acl uses a ccess-control list (ACL) matching. ACL matching is five tuple of the doca_flow_match. Each entry consists of a value and a mask (e.g., 10.0.0.0/8, 10.10.0.0/16, etc.) for IP address fields, port range, or specific port in the port fields, protocol, and priority of the entry.
ACL entry port configuration:
Mask port is 0 ==> Any port
Mask port is equal to match port ==> Exact port. Port with mask 0xffff.
Mask port > match port ==> Match port is used as port from and mask port is used as port to
Monitor actions are not supported in ACL. FWD of the DOCA Flow ACL pipe works the same as the basic DOCA Flow pipe.
ACL supports the following types of FWD:
DOCA_FLOW_FWD_PORT
DOCA_FLOW_FWD_PIPE
DOCA_FLOW_FWD_DROP
doca_flow_pipe_lpm insertion max latency can be measured in milliseconds in some cases and, therefore, it is better to insert it from the control path. To get the best insertion performance, entries should be added in large batches.
An ACL pipe can be a root pipe.
An ACL pipe can be in ingress and egress domain.
An ACL pipe must be accessed on a single queue. Different ACL pipes may be accessed on different queues.
Adding an entry to the ACL pipe after sending an entry with flag DOCA_FLOW_NO_WAIT is not supported.
Removing an entry from an ACL pipe is not supported.
doca_flow_pipe_ordered_list
doca_flow_pipe_ordered_list allows the user to define a specific order of actions and multiply the same type of actions (i.e., specific ordering between counter/meter and encap/decap).
An ordered list pipe is defined by an array of actions (i.e., sequences of actions). Each entry can be an instance one of these sequences. An ordered list pipe may consist of up to an array of 8 different actions. The maximum size of each action array is 4 elements. Resource allocation may be optimized when combining multiple action arrays in one ordered list pipe.
doca_flow_pipe_hash
doca_flow_pipe_hash allows the user to insert entries by index. The index represents the packet hash calculation.
An hash pipe gets doca_flow_match only on pipe creation and only mask. The mask provides all fields to be used for hash calculation.
The monitor, actions, actions_descs, and FWD of the DOCA Flow hash pipe works the same as the basic DOCA Flow pipe.
The nb_flows in doca_flow_pipe_attr should be a power of 2.
Hardware Steering Mode
Users can enable hardware steering mode by setting devarg dv_flow_en to 2.
The following is an example of running DOCA with hardware steering mode:
.... –a 03:00.0, dv_flow_en=2 –a 03:00.1, dv_flow_en=2....
The following is an example of running DOCA with software steering mode:
.... –a 03:00.0 –a 03:00.1 ....
The dv_flow_en=2 means that hardware steering mode is enabled.
In the struct doca_flow_cfg, setting mode_args using (doca_flow_cfg_set_mode_args()) represents DOCA applications. If it is set with hws (e.g., "vnf,hws", "switch,hws", "remmote_vnf,hws") then hardware steering mode is enabled.
In switch mode, fdb_def_rule_en=0,vport_match=1,repr_matching_en=0,dv_xmeta_en=4 should be added to DPDK PMD devargs, which makes DOCA Flow switch module take over all the traffic.
To create an entry by calling doca_flow_pipe_add_entry, the parameter flags can be set as DOCA_FLOW_WAIT_FOR_BATCH or DOCA_FLOW_NO_WAIT:
DOCA_FLOW_WAIT_FOR_BATCH means that this flow entry waits to be pushed to hardware. Batch flows then can be pushed only at once. This reduces the push times and enhances the insertion rate.
DOCA_FLOW_NO_WAIT means that the flow entry is pushed to hardware immediately.
The parameter usr_ctx is handled in the callback set in struct doca_flow_cfg.
doca_flow_entries_process processes all the flows in this queue. After the flow is handled and the status is returned, the callback is executed with the status and usr_ctx.
If the user does not set the callback in doca_flow_cfg, the user can get the status using doca_flow_entry_get_status to check if the flow has completed offloading or not.
Isolated Mode
In non-isolated mode (default) any received packets (following an RSS forward, for example) can be processed by the DOCA application, bypassing the kernel. In the same way, the DOCA application can send packets to the NIC without kernel knowledge. This is why, by default, no replies are received when pinging a host with a running DOCA application. If only specific packet types (e.g., DNS packets) should be processed by the DOCA application, while other packets (e.g., ICMP ping) should be handled directly the kernel, then isolated mode becomes relevant.
In isolated mode, packets that match root pipe entries are steered to the DOCA application (as usual) while other packets are received/sent directly by the kernel.
If you plan to create a pipe with matches followed by action/monitor/forward operations, due to functional/performance considerations, it is advised that root pipes entries include the matches followed by a next pipe forward operation. In the next pipe, all the planned matches actions/monitor/forward operations could be specified. Unmatched packets are received and sent by the kernel.
In switch mode, only isolated mode is supported.
To activate isolated mode, two configurations are required:
DOCA configuration: Update the string member mode_args (struct doca_flow_cfg) using doca_flow_cfg_set_mode_args() which represents the DOCA application mode and add "isolated" (separated by comma) to the other mode arguments. For example: doca_flow_cfg_set_mode_args(cfg, "vnf,hws,isolated") doca_flow_cfg_set_mode_args(cfg, "switch,isolated")
DPDK configuration: Set isolated_mode to 1 (struct application_port_config). For example, if DPDK is initialized by the API: dpdk_queues_and_ports_init(struct application_dpdk_config *app_dpdk_config).
struct application_dpdk_config app_dpdk_config = {
.port_config = {
.isolated_mode = 1,
.nb_ports = ...
...
},
...
};
Pipe Resize
The move to HWS improves performance because rule insertion is implemented in hardware rather than software. However, this move imposes additional limitations, such as the need to commit in advance on the size of the pipes (the number of rule entries). For applications that require pipe sizes to grow over time, a static size can be challenging: Committing to a pipe size too small can cause the the application to fail once the number of rule entries exceeds the committed number, and pre-committing to an excessively high number of rules can result in memory over-allocation.
This is where pipe resizing comes in handy. This feature allows the pipe size to increase during runtime with support for all entries in a new resized pipe.
For this release, pipe resizing it is only operational in Control Pipes.
Increasing Pipe Size
It is possible to set a congestion level by percentage (CONGESTION_PERCENTAGE). Once the number of entries in the pipe exceeds this value, a callback is invoked. For example, for a pipe with 1000 entries and a CONGESTION_PERCENTAGE of 80%, the CONGESTION_REACHED callback is invoked after the 800th entry is added.
Following the CONGESTION_REACHED callback, the application should call the pipe resize API (resize()). The following are optional callbacks during the resize callback:
A callback on the new number of entries allocated to the pipe
A callback on each entry that existed in the smaller pipe and is now allocated to the resized pipe
The pipe pointer remains the same for the application to use even after being resized.
Upon completion of the internal transfer of all entries from the small pipe to the resized pipe, a RESIZED callback is invoked.
A CONGESTION_REACHED callback is received exactly once before the RESIZED callback. Receiving another CONGESTION_REACHED only happens after calling resize() and receiving its completion with a RESIZED callback.
List of Callbacks
CONGESTION_REACHED – on the updated number of entries in the pipe (if pipe is resizable)
InfoReceiving a CONGESTION_REACHED callback can occur after adding a small number of entries and for moving entries from a small to resized pipe. The application must always call pipe resize after receiving the CONGESTION_REACHED callback to handle such cases.
RESIZED – upon completion of the resize operation
NoteCalling pipe resize returns immediately. It starts an internal process that ends later with the RESIZED callback.
NR_ENTRIES_CHANGED (optional) – on the new max number of entries in the pipe
ENTRY_RELOCATE (optional) – on each entry moved from the small pipe to the resized pipe
Order of Operations for Pipe Resizing
set a process callback on doca_flow_init():
.pipe_process_cb = <pipe-process-callback>
InfoThis informs on OP_CONGESTION_REACHED and OP_RESIZED operations when applicable.
Set the following pipe attributes on pipe creation:
.is_resizable =
true, .congestion_level_threshold = <CONGESTION_PERCENTAGE>, .user_ctx = <pipe-user-context>Start adding entries:
doca_flow_pipe_control_add_entry()
Once the number of entries in the pipe crosses the congestion threshold, an OP_CONGESTION_REACHED operation callback is received.
Mark the pipe's congestion threshold event and, upon return, call:
doca_flow_pipe_resize()
For this call, add the following parameters:
The new threshold percentage for calculating the new size
A callback on the new pipe size (optional):
doca_flow_pipe_resize_nr_entries_changed_cb nr_entries_changed_cb
A callback on the entries to be transferred to the resized pipe:
doca_flow_pipe_resize_entry_relocate_cb entry_relocation_cb
Call doca_flow_entries_process() to trigger the transfer of entries. At this phase, adding new entries to the pipe is permitted. The entries are added directly to the resized pipe and therefore do not need to be transferred.
Once all entries are transferred, an OP_RESIZED operation callback is received, at which point calling doca_flow_entries_process() can be stopped. Also, at this point a new OP_CONGESTION_REACHED operation callback can be received again.
Hairpin Configuration
In switch mode, if dev is set in struct doca_flow_port_cfg (using doca_flow_port_cfg_set_dev()), then an internal hairpin is created for direct wire-to-wire fwd. Users may specify the hairpin configuration using mode_args. The supported options as follows:
hairpinq_num=[n] – the hairpin queue number
use_huge_mem – determines whether the Tx buffer uses hugepage memory
lock_rx_mem – locks Rx queue memory
Teardown
Pipe Entry Teardown
When an entry is terminated by the user application or ages-out, the user should call the entry destroy function, doca_flow_pipe_rm_entry(). This frees the pipe entry and cancels hardware offload.
Pipe Teardown
When a pipe is terminated by the user application, the user should call the pipe destroy function, doca_flow_pipe_destroy(). This destroys the pipe and the pipe entries that match it.
When all pipes of a port are terminated by the user application, the user should call the pipe flush function, doca_flow_port_pipes_flush(). This destroys all pipes and all pipe entries belonging to this port.
During doca_flow_pipe_destroy() execution, the application must avoid adding/removing entries or checking for aged entries of any other pipes.
Port Teardown
When the port is not used anymore, the user should call the port stop function, doca_flow_port_stop(). This stops the DOCA port, disables the traffic, destroys the port and frees all resources of the port.
Flow Teardown
When the DOCA Flow is not used anymore, the user should call the flow destroy function, doca_flow_destroy(). This releases all the resources used by DOCA Flow.
In situations where there is a port without a pipe defined, or with a pipe defined but without any entry, the default behavior is that all packets arrive to a port in the software.
Once entries are added to the pipe, if a packet has no match then it continues to the port in the software. If it is matched, then the rules defined in the pipe are executed.
If the packet is forwarded in RSS, the packet is forwarded to software according to the RSS definition. If the packet is forwarded to a port, the packet is redirected back to the wire. If the packet is forwarded to the next pipe, then the software attempts to match it with the next pipe.
Note that the number of pipes impacts performance. The longer the number of matches and actions that the packet goes through, the longer it takes the hardware to process it. When there is a very large number of entries, the hardware must access the main memory to retrieve the entry context which increases latency.
DOCA Flow supports trace and debugging of DOCA Flow applications which enable collecting predefined internal key performance indicators (KPIs) and pipeline visualization.
Installation
The set of DOCA's SDK development packages include also a developer-oriented package that includes additional trace and debug features which are not included in the production libraries:
.deb based systems – libdoca-libs-trace
.rpm based systems – doca-libs-trace
These packages install the trace-version of the libraries under the following directories:
.deb based systems – /opt/mellanox/doca/lib/<arch>/trace
.rpm based systems – /opt/mellanox/doca/lib64/trace
Using Trace Libraries
The trace libraries are designed to allow a user to link their existing (production) program to the trace library without needing to recompile the program. To do so, one should simply update the matching environment variable so that the OS will prioritize loading libraries from the above trace directory:
LD_LIBRARY_PATH=/opt/mellanox/doca/lib/aarch64-linux-gnu/trace:${LD_LIBRARY_PATH} doca_ipsec_security_gw <program parameters>
Trace Features
DOCA Log – Trace Level
DOCA's trace logging level (DOCA_LOG_LEVEL_TRACE) is compiled as part of this trace version of the library. That is, any program compiled against the library can activate this additional logging level through DOCA's API or even through DOCA's built-in argument parsing (ARGP) library:
LD_LIBRARY_PATH=/opt/mellanox/doca/lib/aarch64-linux-gnu/trace:${LD_LIBRARY_PATH} doca_ipsec_security_gw <program parameters> --sdk-log-level 70
This section provides DOCA Flow sample implementation on top of the BlueField.
Sample Prerequisites
A DOCA Flow-based program can either run on the host machine or on the BlueField.
Flow-based programs require an allocation of huge pages, hence the following commands are required:
echo '1024' | sudo tee -a /sys/kernel/mm/hugepages/hugepages-2048kB/nr_hugepages
sudo mkdir /mnt/huge
sudo mount -t hugetlbfs nodev /mnt/huge
On some OSs (RockyLinux, OpenEuler, CentOS 8.2), the default huge page size on the BlueField (and Arm hosts) is larger than 2MB, often 512MB. Users can check the size of the huge pages on their OS using the following command:
$ grep -i huge /proc/meminfo
AnonHugePages: 0 kB
ShmemHugePages: 0 kB
FileHugePages: 0 kB
HugePages_Total: 4
HugePages_Free: 4
HugePages_Rsvd: 0
HugePages_Surp: 0
Hugepagesize: 524288 kB
Hugetlb: 6291456 kB
In this case, instead of allocating 1024 pages, users should only allocate 4:
echo '4' | sudo tee -a /sys/kernel/mm/hugepages/hugepages-524288kB/nr_hugepages
Running the Sample
Refer to the following documents:
NVIDIA DOCA Installation Guide for Linux for details on how to install BlueField-related software.
NVIDIA DOCA Troubleshooting Guide for any issue you may encounter with the installation, compilation, or execution of DOCA samples.
To build a given sample:
cd /opt/mellanox/doca/samples/doca_flow/<sample_name> meson /tmp/build ninja -C /tmp/build
NoteThe binary doca_<sample_name> will be created under /tmp/build/.
Sample (e.g., flow_aging) usage:
Usage: doca_flow_aging [DPDK Flags] –- [DOCA Flags] DOCA Flags: -h, --help Print a help synopsis -v, --version Print program version information -l, --log-level Set the (numeric) log level
forthe program <10=DISABLE,20=CRITICAL,30=ERROR,40=WARNING,50=INFO,60=DEBUG,70=TRACE> --sdk-log-level Set the SDK (numeric) log levelforthe program <10=DISABLE,20=CRITICAL,30=ERROR,40=WARNING,50=INFO,60=DEBUG,70=TRACE> -j, --json <path> Parse all command flags from an input json fileFor additional information per sample, use the -h option after the -- separator:
/tmp/build/doca_<sample_name> -- -h
DOCA Flow samples are based on DPDK libraries. Therefore, the user is required to provide DPDK flags. The following is an example from an execution on the DPU:
CLI example for running the samples with "vnf" mode:
/tmp/build/doca_<sample_name> -a auxiliary:mlx5_core.sf.2 -a auxiliary:mlx5_core.sf.3 -- -l 60
CLI example for running the VNF samples with vnf,hws mode:
/tmp/build/doca_<sample_name> -a auxiliary:mlx5_core.sf.2,dv_flow_en=2 -a auxiliary:mlx5_core.sf.3,dv_flow_en=2 -- -l 60
CLI example for running the switch samples with switch,hws mode:
/tmp/build/doca_<sample_name> -- -p 03:00.0 -r sf[2-3] -l 60
NoteWhen running on the BlueField with switch,hws mode , it is not necessary to configure the OVS.
DOCA switch sample hides the extra fdb_def_rule_en=0,vport_match=1,repr_matching_en=0,dv_xmeta_en=4 DPDK devargs with a simple -p and -r to specify the PCIe ID and representor information.
NoteWhen running on the DPU using the command above, sub-functions must be enabled according to the NVIDIA BlueField DPU Scalable Function User Guide.
NoteWhen running on the host, virtual functions must be used according to the instructions in the NVIDIA DOCA Virtual Functions User Guide.
Samples
Flow ACL
This sample illustrates how to use the access-control list (ACL) pipe.
The sample logic includes:
Initializing DOCA Flow by indicating mode_args="vnf,hws" in the doca_flow_cfg struct.
Starting two DOCA Flow ports.
On each port:
Building an ACL pipe that matches changeable:
Source IPv4 address
Destination IPv4 address
Source port
Destination port
Adding four example 5-tuple entries:
The first entry with:
Full mask on source IPv4 address
Full mask on destination IPv4 address
Null mask on source port (any source port)
Null mask on destination port (any destination port)
TCP protocol
Priority 10
Action "deny" (drop action)
The second entry with:
Full mask on source IPv4 address
Full mask on destination IPv4 address
Null mask on source port (any source port)
Value set in mask on destination port is used as part of port range:
Destination port in match is used as port from
Destination port in mask is used as port to
UDP protocol
Priority 50
Action "allow" (forward port action)
The third entry with:
Full mask on source IPv4 address
Full mask on destination IPv4 address
Value set in mask on source port is equal to the source port in match. It is the exact port. ACL uses the port with full mask.
Null mask on destination port (any destination port)
TCP protocol
Priority 40
Action "allow" (forward port action)
The fourth entry with:
24-bit mask on source IPv4 address
24-bit mask on destination IPv4 address
Value set in mask on source port is used as part of port range : source port in match is used as port from, source port in mask is used as port to.
Value set in mask on destination port is equal to the destination port in match. It is the exact port. ACL uses the port with full mask.
TCP protocol
Priority 20
Action "allow" (forward port action)
The sample shows how to run the ACL pipe on ingress and egress domains. To change the domain, use the global parameter flow_acl_sample.c.
Ingress domain: ACL is created as root pipe
Egress domain:
Building a control pipe with one entry that forwards the IPv4 traffic hairpin port.
ACL is created as a root pipe on the hairpin port.
Reference:
/opt/mellanox/doca/samples/doca_flow/flow_acl/flow_acl_sample.c
/opt/mellanox/doca/samples/doca_flow/flow_acl/flow_acl_main.c
/opt/mellanox/doca/samples/doca_flow/flow_acl/meson.build
Flow Aging
This sample illustrates the use of DOCA Flow's aging functionality. It demonstrates how to build a pipe and add different entries with different aging times and user data.
The sample logic includes:
Initializing DOCA Flow with mode_args="vnf,hws" in the doca_flow_cfg struct.
Starting two DOCA Flow port.
On each port:
Building a pipe with changeable 5-tuple match and forward port action.
Adding 10 entries with different 5-tuple match, a monitor with different aging time (5-60 seconds), and setting user data in the monitor. The user data will contain the port ID, entry number, and entry pointer.
Handling aging every 5 seconds and removing each entry after age-out.
Running these commands until all entries age out.
Reference:
/opt/mellanox/doca/samples/doca_flow/flow_aging/flow_aging_sample.c
/opt/mellanox/doca/samples/doca_flow/flow_aging/flow_aging_main.c
/opt/mellanox/doca/samples/doca_flow/flow_aging/meson.build
Flow Control Pipe
This sample shows how to use the DOCA Flow control pipe and decap action.
The sample logic includes:
Initializing DOCA Flow by indicating mode_args="vnf,hws" in the doca_flow_cfg struct.
Starting two DOCA Flow ports.
On each port:
Building VXLAN pipe with match on VNI field, decap action, action descriptor for decap, and forwarding the matched packets to the second port.
Building VXLAN-GPE pipe with match on VNI plus next protocol fields, and forwarding the matched packets to the second port.
Building GRE pipe with match on GRE key field, decap and build eth header actions, action descriptor for decap, and forwarding the matched packets to the second port.
Building NVGRE pipe with match on protocol is 0x6558, vs_id, flow_id, and inner UDP source port fields, and forwarding the matched packets to the second port. This pipe has a higher priority than the GRE pipe. The NVGRE packets are matched first.
Building MPLS pipe with match on third MPLS label field, decap and build eth header actions, action descriptor for decap, and forwarding the matched packets to the second port.
Building a control pipe with the following entries:
If L4 type is UDP and destination port is 4789, forward to VXLAN pipe
If L4 type is UDP and destination port is 4790, forward to VXLAN-GPE pipe
If L4 type is UDP and destination port is 6635, forward to MPLS pipe
If tunnel type and L4 type is GRE, forward to GRE pipe
When any tunnel is decapped, it is user responsibility to identify if it is an L2 or L3 tunnel within the action descriptor. If the tunnel is L3, the complete outer layer, tunnel, and inner L2 are removed and the inner L3 layer is exposed. To keep the packet valid, DOCA Flow automatically encaps the inner packet with an empty ETH header. To make the ETH header valid, the user must modify the L2 source/destination MAC addresses and VLAN may optionally be modified. For example:
actions.decap = true;
/* append eth header after decap GRE tunnel */
SET_MAC_ADDR(actions.outer.eth.src_mac, src_mac[0], src_mac[1], src_mac[2], src_mac[3], src_mac[4], src_mac[5]);
SET_MAC_ADDR(actions.outer.eth.dst_mac, dst_mac[0], dst_mac[1], dst_mac[2], dst_mac[3], dst_mac[4], dst_mac[5]);
actions.outer.l3_type = DOCA_FLOW_L3_TYPE_IP4;
/* identify that the tunnel is of L3 type */
desc_array[0].type = DOCA_FLOW_ACTION_DECAP_ENCAP;
desc_array[0].decap_encap.is_l2 = false;
For a VXLAN tunnel, since VXLAN is a L2 tunnel, the user must indicate that to the DOCA Flow using an action descriptor so it does not execute any automatic actions:
actions.decap = true;
/* identify that the tunnel is of L2 type */
desc_array[0].type = DOCA_FLOW_ACTION_DECAP_ENCAP;
desc_array[0].decap_encap.is_l2 = true;
Reference:
/opt/mellanox/doca/samples/doca_flow/flow_control_pipe/flow_control_pipe_sample.c
/opt/mellanox/doca/samples/doca_flow/flow_control_pipe/flow_control_pipe_main.c
/opt/mellanox/doca/samples/doca_flow/flow_control_pipe/meson.build
Flow Copy to Meta
This sample shows how to use the DOCA Flow copy-to-metadata action to copy the source MAC address and then match on it.
The sample logic includes:
Initializing DOCA Flow by indicating mode_args="vnf,hws" in the doca_flow_cfg struct.
Starting two DOCA Flow ports.
On each port:
Building a pipe with changeable match on meta_data and forwarding the matched packets to the second port.
Adding an entry that matches an example source MAC that has been copied to metadata.
Building a pipe with changeable 5-tuple match, copying source MAC action, and fwd to the first pipe.
Adding example 5-tuple entry to the pipe.
Reference:
/opt/mellanox/doca/samples/doca_flow/flow_copy_to_meta/flow_copy_to_meta_sample.c
/opt/mellanox/doca/samples/doca_flow/flow_copy_to_meta/flow_copy_to_meta_main.c
/opt/mellanox/doca/samples/doca_flow/flow_copy_to_meta/meson.build
Flow Add to Metadata
This sample shows how to use the DOCA Flow add-to-metadata action to accumulate the source IPv4 address for double to meta and then match on the meta.
The sample logic includes:
Initializing DOCA Flow by indicating mode_args="vnf,hws" in the doca_flow_cfg struct.
Starting two DOCA Flow ports.
On each port:
Building a pipe with changeable match on meta_data and forwarding the matched packets to the second port.
Adding an entry that matches an example double of source IPv4 address that has been added to metadata.
Building a pipe with changeable 5-tuple match, copying the source IPv4, and adding the value again to the meta action, and forwarding to the first pipe.
Adding an example 5-tuple entry to the pipe.
Reference:
/opt/mellanox/doca/samples/doca_flow/flow_add_to_meta/flow_add_to_meta_sample.c
/opt/mellanox/doca/samples/doca_flow/flow_add_to_meta/flow_add_to_meta_main.c
/opt/mellanox/doca/samples/doca_flow/flow_add_to_meta/meson.build
Flow Drop
This sample illustrates how to build a pipe with 5-tuple match, forward action drop, and forward miss action to the hairpin pipe. The sample also demonstrates how to dump pipe information to a file and query entry.
The sample logic includes:
Initializing DOCA Flow by indicating mode_args="vnf,hws" in the doca_flow_cfg struct.
Starting two DOCA Flow ports.
On each port:
Building a hairpin pipe with an entry that matches all traffic and forwarding traffic to the second port.
Building a pipe with a changeable 5-tuple match, forwarding action drop, and miss forward to the hairpin pipe. This pipe serves as a root pipe.
Adding an example 5-tuple entry to the drop pipe with a counter as monitor to query the entry later.
Waiting 5 seconds and querying the drop entry (total bytes and total packets).
Dumping the pipe information to a file.
Reference:
/opt/mellanox/doca/samples/doca_flow/flow_drop/flow_drop_sample_sample.c
/opt/mellanox/doca/samples/doca_flow/flow_drop/flow_drop_sample_main.c
/opt/mellanox/doca/samples/doca_flow/flow_drop/meson.build
Flow ECMP
This sample illustrates ECMP feature using a hash pipe.
The sample enables users to determine how many port are included in ECMP distribution:
The number of ports, n, is determined by DPDK device argument representor=sf[0-m] where m=n-1.
CLI example for running this samples with n=4 ports:
/tmp/build/doca_flow_ecmp -- -p 03:00.0 -r sf[0-3] -l 60 --sdk-log-level 60
n should be power of 2. Max supported value is n=8.
The sample logic includes:
Calculate the number of SF representors (n) created by DPDK according to user input.
Initializing DOCA Flow by indicating mode_args="switch,hws" in the doca_flow_cfg structure.
Starting DOCA Flow ports: Physical port and n SF representors.
On switch port:
Constructing a hash pipe that signifies the match_mask structure to compute the hash based on the outer IPv6 flow label field.
Adding n entries to the created pipe, each of which forwards packets to a different port representor.
Waiting 15 seconds and querying the entries.
Print the ECMP results per port (number packets in each port related to total packets).
Reference:
/opt/mellanox/doca/samples/doca_flow/flow_ecmp/flow_ecmp_sample.c
/opt/mellanox/doca/samples/doca_flow/flow_ecmp/flow_ecmp_main.c
/opt/mellanox/doca/samples/doca_flow/flow_ecmp/meson.build
Flow ESP
This sample illustrates how to match match ESP fields in two ways:
Exact match for both esp_spi and esp_en fields using the doca_flow_match structure.
Comparison match for esp_en field using the doca_flow_match_condition structure.
This sample is supported for ConnectX-7, BlueField-3, and above.
The sample logic includes:
Initializing DOCA Flow by indicating mode_args="vnf,hws" in the doca_flow_cfg struct.
Starting two DOCA Flow ports.
On each port:
Building a control pipe with entry that match esp_en > 3 (GT pipe).
Building a control pipe with entry that match esp_en < 3 (LT pipe).
Building a root pipe with changeable next_pipe FWD and esp_spi match along with specific esp_sn match + IPv4 and ESP exitance (matching parser_meta).
Adding example esp_spi = 8 entry to the root pipe which forwards to GT pipe (and miss condition).
Adding example esp_spi = 5 entry to the root pipe which forwards to LT pipe (and hit condition).
Reference:
/opt/mellanox/doca/samples/doca_flow/flow_esp/flow_esp_sample.c
/opt/mellanox/doca/samples/doca_flow/flow_esp/flow_esp_main.c
/opt/mellanox/doca/samples/doca_flow/flow_esp/meson.build
Flow Forward Miss
The sample illustrates how to use FWD miss query and update with or without miss counter.
The sample logic includes:
Initializing DOCA Flow by indicating mode_args="vnf,hws" in the doca_flow_cfg struct.
Starting two DOCA Flow ports.
On each port:
Building a copy pipe with a changeable outer L3 type match and forwarding traffic to the second port.
Add entries doing different copy action depending on the outer L3 type:
IPv4 – copy IHL field into Type Of Service field.
IPv6 – copy Payload Length field into Traffic Class field.
Building a pipe with a IPv4 addresses match, forwarding traffic to the second port, and miss forward to the copy pipe.
Building an IP selector pipe with outer L3 type match, forwarding IPv4 traffic to IPv4 pipe, and miss forward to the copy pipe with miss counter.
Building a root pipe with outer L3 type match, forwarding IPv4 and IPv6 traffic to IP selector pipe, and dropping all other traffic by miss forward with miss counter.
Waiting 5 seconds for first batch of traffic.
On each port:
Querying the miss counters using doca_flow_query_pipe_miss API.
Printing the miss results.
On each port:
Building a push pipe that pushes VLAN header and forwarding traffic to the second port.
Updating both IP selector and IPv4 pipes miss FWD pipe target to push pipe using doca_flow_pipe_update_miss API.
Waiting 5 seconds for second batch of traffic, same flow as before.
On each port:
Querying again the miss counters using doca_flow_query_pipe_miss API.
Printing the miss results again, the results should include miss packets coming either before or after miss action updating.
Reference:
/opt/mellanox/doca/samples/doca_flow/flow_fwd_miss/flow_fwd_miss_sample.c
/opt/mellanox/doca/samples/doca_flow/flow_fwd_miss/flow_fwd_miss_main.c
/opt/mellanox/doca/samples/doca_flow/flow_fwd_miss/meson.build
Flow Forward Target (DOCA_FLOW_TARGET_KERNEL)
The sample illustrates how to use DOCA_FLOW_FWD_TARGET type of forward, as well as the doca_flow_get_target API to obtain an instance of struct doca_flow_target.
The sample logic includes:
Initializing DOCA Flow with "vnf,isolated,hws".
Initializing two ports.
Obtaining an instance of doca_flow_target by calling doca_flow_get_target(DOCA_FLOW_TARGET_KERNEL, &kernel_target);.
On each port, creating:
Non-root basic pipe with 5 tuple match.
If hit – forward the packet to another port.
If miss – forward the packet to the kernel for processing by using the instance of doca_flow_target obtained in previous steps.
Then add a single entry with a specific 5-tuple which is hit, and the rest is forwarded to the kernel.
Root control pipe with a match on outer L3 type being IPv4.
If hit – forward the packet to the non-root pipe.
If miss – drop the packet.
Add a single entry that implements the logic described.
Reference:
/opt/mellanox/doca/samples/doca_flow/flow_fwd_target/flow_fwd_target_sample.c
/opt/mellanox/doca/samples/doca_flow/flow_fwd_target/flow_fwd_target_main.c
/opt/mellanox/doca/samples/doca_flow/flow_fwd_target/meson.build
Flow GENEVE Encap
This sample illustrates how to use DOCA Flow actions to create a GENEVE tunnel.
The sample logic includes:
Initializing DOCA Flow by indicating mode_args="vnf,hws" in the doca_flow_cfg struct.
Starting two DOCA Flow ports.
On each port:
Building ingress pipe with changeable 5-tuple match, copying to pkt_meta action, and forwarding port action.
Building egress pipe with pkt_meta match and 4 different encapsulation actions:
L2 encap without options
L2 encap with options
L3 encap without options
L3 encap with options
Adding example 5-tuple and encapsulation values entries to the pipes.
Reference:
/opt/mellanox/doca/samples/doca_flow/flow_geneve_encap/flow_geneve_encap_sample.c
/opt/mellanox/doca/samples/doca_flow/flow_geneve_encap/flow_geneve_encap_main.c
/opt/mellanox/doca/samples/doca_flow/flow_geneve_encap/meson.build
Flow GENEVE Options
This sample illustrates how to prepare a GENEVE options parser, match on configured options, and decap GENEVE tunnel.
This sample works only with PF. VFs and SFs are not supported.
The sample logic includes:
Initializing DOCA Flow by indicating mode_args="vnf,hws" in the doca_flow_cfg struct.
Starting two DOCA Flow ports.
On each port:
Building GENEVE options parser, same input for all ports.
Building match pipe with GENEVE VNI and options match and forwards decap pipe.
Building decap pipe with more GENEVE options match, and 2 different decapsulation actions:
L2 decap
L3 decap with changeable mac addresses
Adding example GENEVE options and MAC address values entries to the pipes.
Reference:
/opt/mellanox/doca/samples/doca_flow/flow_geneve_opt/flow_geneve_opt_sample.c
/opt/mellanox/doca/samples/doca_flow/flow_geneve_opt/flow_geneve_opt_main.c
/opt/mellanox/doca/samples/doca_flow/flow_geneve_opt/meson.build
Flow Hairpin VNF
This sample illustrates how to build a pipe with 5-tuple match and to forward packets to the other port.
The sample logic includes:
Initializing DOCA Flow by indicating mode_args="vnf,hws" in the doca_flow_cfg struct.
Starting two DOCA Flow ports.
On each port:
Building a pipe with changeable 5-tuple match and forwarding port action.
Adding example 5-tuple entry to the pipe.
Reference:
/opt/mellanox/doca/samples/doca_flow/flow_hairpin_vnf/flow_hairpin_vnf_sample.c
/opt/mellanox/doca/samples/doca_flow/flow_hairpin_vnf/flow_hairpin_vnf_main.c
/opt/mellanox/doca/samples/doca_flow/flow_hairpin_vnf/meson.build
Flow Switch to Wire
This sample illustrates how to build a pipe with 5-tuple match and forward packets from the wire back to the wire.
The sample shows how to build a basic pipe in a switch and hardware steering (HWS) mode. Each pipe contains two entries, each of which forwards matched packets to two different representors.
The sample also demonstrates how to obtain the switch port of a given port using doca_flow_port_switch_get().
The test requires one PF with three representors (either VFs or SFs).
The sample logic includes:
Initializing DOCA Flow by indicating mode_args="switch,hws" in the doca_flow_cfg struct.
Starting DOCA Flow ports with doca_dev in struct doca_flow_port_cfg.
On the switch's PF port:
Building ingress, egress, vport, and RSS pipes with changeable 5-tuple match and forwarding port action.
Adding example 5-tuple entry to the pipe.
The matched traffic goes to its destination port, the missed traffic is handled by the rx_tx function and is sent to a dedicate port based on the protocol.
Ingress pipe:
Entry
0: IP src1.2.3.4/ TCP src1234dst80-> egress pipe Entry1: IP src1.2.3.5/ TCP src1234dst80-> vport pipeEgress pipe (test ingress to egress cross domain):
Entry
0: IP dst8.8.8.8/ TCP src1234dst80-> port0Entry1: IP dst8.8.8.9/ TCP src1234dst80-> port1Entry2: IP dst8.8.8.10/ TCP src1234dst80-> port2Entry3: IP dst8.8.8.11/ TCP src1234dst80-> port3Vport pipe (test ingress direct to vport):
Entry
0: IP dst8.8.8.8/ TCP src1234-> port0Entry1: IP dst8.8.8.9/ TCP src1234-> port1Entry2: IP dst8.8.8.10/ TCP src1234-> port2Entry3: IP dst8.8.8.11/ TCP src1234-> port3RSS pipe (test miss traffic port_id get and destination port_id set):
Entry
0: IPv4 / TCP -> port0Entry0: IPv4 / UDP -> port1Entry0: IPv4 / ICMP -> port2
Reference:
/opt/mellanox/doca/samples/doca_flow/flow_switch_to_wire/flow_switch_to_wire_sample.c
/opt/mellanox/doca/samples/doca_flow/flow_switch_to_wire/flow_switch_to_wire_main.c
/opt/mellanox/doca/samples/doca_flow/flow_switch_to_wire/meson.build
Flow Hash Pipe
This sample illustrates how to build a hash pipe in hardware steering (HWS) mode.
The hash pipe contains two entries, each of which forwards "matched" packets to two different SF representors. For each received packet, the hash pipe calculates the entry index to use based on the IPv4 destination address.
The sample logic includes:
Initializing DOCA Flow by indicating mode_args="switch,hws" in the doca_flow_cfg struct.
Starting DOCA Flow ports: Physical port and two SF representors.
On switch port:
Building a hash pipe while indicating which fields to use to calculate the hash in the struct match_mask.
Adding two entries to the created pipe, each of which forwards packets to a different port representor.
Printing the hash result calculated by the software with the following message: "hash value for" for dest ip = 192.168.1.1.
Reference:
/opt/mellanox/doca/samples/doca_flow/flow_hash_pipe/flow_hash_pipe_sample.c
/opt/mellanox/doca/samples/doca_flow/flow_hash_pipe/flow_hash_pipe_main.c
/opt/mellanox/doca/samples/doca_flow/flow_hash_pipe/meson.build
Flow IPv6 Flow Label
This sample shows how to use DOCA Flow actions to update IPv6 flow label field after encapsulation.
As a side effect, it shows also example for IPv6 + MPLS encapsulation.
The sample logic includes:
Initializing DOCA Flow by indicating mode_args="vnf,hws" in the doca_flow_cfg struct.
Starting two DOCA Flow ports.
On each port:
Building an ingress pipe with changeable L4 type and ports matching, which updates metadata and goes to the peer port.
Adding example UDP/TCP type and ports and metadata values entries to the pipe. This pipe is L3 type agnostic.
Building an egress pipe on the peer port with changeable metadata matching, which encapsulates packets with IPv6 + MPLS headers, and goes to the next pipe.
Adding entries to the pipe, with different encapsulation values for different metadata values.
Building another egress pipe on the peer port with changeable L3 inner type matching, which copies value into outer IPv6 flow label field.
Adding two entries to the pipe:
L3 inner type is IPv6 - copy IPv6 flow label from inner to outer.
L3 inner type is IPv6 - copy outer IPv6 flow label from metadata.
Reference:
/opt/mellanox/doca/samples/doca_flow/flow_ipv6_flow_label/flow_ipv6_flow_label_sample.c
/opt/mellanox/doca/samples/doca_flow/flow_ipv6_flow_label/flow_ipv6_flow_label_main.c
/opt/mellanox/doca/samples/doca_flow/flow_ipv6_flow_label/meson.build
Flow Loopback
This sample illustrates how to implement packet re-injection, or loopback, in VNF mode.
The sample logic includes:
Initializing DOCA Flow by indicating mode_args="vnf,hws" in the doca_flow_cfg struct.
Starting two DOCA Flow ports.
On each port:
Building a UDP pipe that matches a changeable source and destination IPv4 address, while the forwarding component is RSS to queues. Upon match, setting the packet meta on this UDP pipe which is referred to as an RSS_UDP_IP pipe.
Adding one entry to the RSS_UDP_IP pipe that matches a packet with a specific source and destination IPv4 address and setting the meta to 10.
Building a TCP pipe that matches changeable 4-tuple source and destination IPv4 and port addresses, while the forwarding component is RSS to queues (this pipe is called RSS_TCP_IP and it is the root pipe on ingress domain).
Adding one entry to the RSS_TCP_IP pipe, that matches a packet with a specific source and destination port and IPv4 addresses.
On the egress domain, creating the loopback pipe, which is root, and matching TCP over IPv4 with changeable 4-tuple source and destination port and IPv4 addresses, while encapsulating the matched packets with VXLAN tunneling and setting the destination and source MAC addresses to be changeable per entry.
Adding one entry to the loopback pipe with specific values for the match and actions part while setting the destination MAC address to the port to which to inject the packet (in this case, it is the ingress port where the packet arrived).
Starting to receive packets loop and printing the metadata
For packets that were re-injected, metadata equaling 10 is printed
Otherwise, 0 is be printed as metadata (indicating that it is the first time the packet has been encountered)
Reference:
/opt/mellanox/doca/samples/doca_flow/flow_loopback/flow_loopback_sample.c
/opt/mellanox/doca/samples/doca_flow/flow_loopback/flow_loopback_main.c
/opt/mellanox/doca/samples/doca_flow/flow_loopback/meson.build
Flow LPM
This sample illustrates how to use LPM (Longest Prefix Match) pipe
The sample logic includes:
Initializing DOCA Flow by indicating mode_args="vnf,hws" in the doca_flow_cfg struct.
Starting two DOCA Flow ports.
On each port:
Building an LPM pipe that matches changeable source IPv4 address.
Adding two example 5-tuple entries:
The first entry with full mask and forward port action
The second entry with 16-bit mask and drop action
Building a control pipe with one entry that forwards IPv4 traffic to the LPM pipe.
Reference:
/opt/mellanox/doca/samples/doca_flow/flow_lpm/flow_lpm_sample.c
/opt/mellanox/doca/samples/doca_flow/flow_lpm/flow_lpm_main.c
/opt/mellanox/doca/samples/doca_flow/flow_lpm/meson.build
Flow LPM with exact match (EM)
This sample illustrates how to use LPM (Longest Prefix Match) pipe with exact match logic (EM) enabled.
The sample logic includes:
Initializing DOCA Flow by indicating mode_args="vnf,hws" in the doca_flow_cfg struct.
Starting two DOCA Flow ports.
On each port:
Building LPM pipe that matches changeable source IPv4 address (using match) with exact-match logic on meta.u32[1] (using match_mask )
Add five entries to the LPM
Default entry with IPv4 subnet 0 to drop the packets which are unmatched in LPM with EM
Fully masked 1.2.3.4 IPv4 address with meta value 1 to forward to the next port
Fully masked 1.2.3.4 IPv4 address with meta value 2 to forward to the next port
Fully masked 1.2.3.4 IPv4 address with meta value 3 to drop
First 16 bit masked 1.2.0.0 IPv4 address with meta value 3 to forward to the next port
Build basic root pipe which matches everything, copies the outer.eth_vlan0.tci value to the meta.u32[1] and forwards the packet to the LPM pipe
Add single entry to the main pipe
The sample uses the counters to show the packets per entry. Here are the packets that can be used for the test and the expected response of the sample to them:
Ether()/Dot1Q(vlan=1)/IP(src="1.2.3.4") - expected to be forwarded to next port by entry number 1
Ether()/Dot1Q(vlan=2)/IP(src="1.2.3.4") - expected to be forwarded to next port by entry number 2
Ether()/Dot1Q(vlan=3)/IP(src="1.2.3.4") - expected to be dropped by entry number 3
Ether()/Dot1Q(vlan=3)/IP(src="1.2.125.125") - expected to be forwarded to next port by entry number 4
Ether()/Dot1Q(vlan=5)/IP(src="5.5.5.5") - expected to be dropped by entry number 0 (default)
Reference:
/opt/mellanox/doca/samples/doca_flow/flow_lpm_em/flow_lpm_em_sample.c
/opt/mellanox/doca/samples/doca_flow/flow_lpm_em/flow_lpm_em_main.c
/opt/mellanox/doca/samples/doca_flow/flow_lpm_em/meson.build
Flow Modify Header
This sample illustrates how to use DOCA Flow actions to decrease TTL by 1 and modify the destination MAC address.
The sample logic includes:
Initializing DOCA Flow by indicating mode_args="vnf,hws" in the doca_flow_cfg struct.
Starting two DOCA Flow ports.
On each port:
Building a pipe with action dec_ttl=true and changeable mod_dst_mac. The pipe matches IPv4 traffic with a changeable destination IP and forwards the matched packets to the second port.
Adding an entry with an example destination IP and mod_dst_mac value.
Reference:
/opt/mellanox/doca/samples/doca_flow/flow_modify_header/flow_modify_header_sample.c
/opt/mellanox/doca/samples/doca_flow/flow_modify_header/flow_modify_header_main.c
/opt/mellanox/doca/samples/doca_flow/flow_modify_header/meson.build
Flow Monitor Meter
This sample illustrates how to use DOCA Flow monitor meter.
The sample logic includes:
Initializing DOCA Flow by indicating mode_args="vnf,hws" in the doca_flow_cfg struct.
Starting two DOCA Flow ports.
On each port:
Building a pipe with monitor meter flag and changeable 5-tuple match. The pipe forwards the matched packets to the second port.
Adding an entry with an example CIR and CBS values.
Reference:
/opt/mellanox/doca/samples/doca_flow/flow_monitor_meter/flow_monitor_meter_sample.c
/opt/mellanox/doca/samples/doca_flow/flow_monitor_meter/flow_monitor_meter_main.c
/opt/mellanox/doca/samples/doca_flow/flow_monitor_meter/meson.build
Flow Multi-actions
This sample shows how to use a DOCA Flow array of actions in a pipe.
The sample logic includes:
Initializing DOCA Flow by indicating mode_args="vnf,hws" in the doca_flow_cfg struct.
Starting two DOCA Flow ports.
On each port:
Building a pipe with changeable source IP match which forwards the matched packets to the second port and sets different actions in the actions array:
Changeable modify source MAC address
Changeable modify source IP address
Adding two entries to the pipe with different source IP match:
The first entry with an example modify source MAC address.
The second with a modify source IP address.
Reference:
/opt/mellanox/doca/samples/doca_flow/flow_multi_actions/flow_multi_actions_sample.c
/opt/mellanox/doca/samples/doca_flow/flow_multi_actions/flow_multi_actions_main.c
/opt/mellanox/doca/samples/doca_flow/flow_multi_actions/meson.build
Flow Multi-fwd
This sample shows how to use a different forward in pipe entries.
The sample logic includes:
Initializing DOCA Flow by indicating mode_args="vnf,hws" in the doca_flow_cfg struct.
Starting two DOCA Flow ports.
On each port:
Building a pipe with changeable source IP match and sending NULL in the forward.
Adding two entries to the pipe with different source IP match, and different forward:
The first entry with forward to the second port
The second with drop
Reference:
/opt/mellanox/doca/samples/doca_flow/flow_multi_fwd/flow_multi_fwd_sample.c
/opt/mellanox/doca/samples/doca_flow/flow_multi_fwd/flow_multi_fwd_main.c
/opt/mellanox/doca/samples/doca_flow/flow_multi_fwd/meson.build
Flow Ordered List
This sample shows how to use a DOCA Flow ordered list pipe.
The sample logic includes:
Initializing DOCA Flow by indicating mode_args="vnf,hws" in the doca_flow_cfg struct.
Starting two DOCA Flow ports.
On each port:
Building a root pipe with changeable 5-tuple match and forwarding to an ordered list pipe with a changeable index.
Adding two entries to the pipe with an example value sent to a different index in the ordered list pipe.
Building ordered list pipe with two lists, one for each entry:
First list uses meter and then shared counter
Second list uses shared counter and then meter
Waiting 5 seconds and querying the entries (total bytes and total packets).
Reference:
/opt/mellanox/doca/samples/doca_flow/flow_ordered_list/flow_ordered_list_sample.c
/opt/mellanox/doca/samples/doca_flow/flow_ordered_list/flow_ordered_list_main.c
/opt/mellanox/doca/samples/doca_flow/flow_ordered_list/meson.build
Flow Parser Meta
This sample shows how to use some of match.parser_meta fields from 3 families:
IP fragmentation – matching on whether a packet is IP fragmented
Integrity bits – matching on whether a specific protocol is OK (length, checksum etc.)
Packet types – matching on a specific layer packet type
The sample logic includes:
Initializing DOCA Flow by indicating mode_args="vnf,hws" in the doca_flow_cfg struct.
Starting two DOCA Flow ports.
On each port:
Building a root pipe with outer IP fragmentation match:
If a packet is IP fragmented – forward it to the second port regardless of next pipes in the pipeline
If a packet is not IP fragmented – proceed with the the pipeline by forwarding it to integrity pipe
Building an "integrity" pipe with a single entry which continues to the next pipe when:
The outer IPv4 checksum is OK
The inner L3 is OK (incorrect length should be dropped)
Building a "packet type" pipe which forwards packets to the second port when:
The outer L3 type is IPv4
The inner L4 type is either TCP or UDP
Waiting 5 seconds for traffic to arrive.
Reference:
/opt/mellanox/doca/samples/doca_flow/flow_parser_meta/flow_parser_meta_sample.c
/opt/mellanox/doca/samples/doca_flow/flow_parser_meta/flow_parser_meta_main.c
/opt/mellanox/doca/samples/doca_flow/flow_parser_meta/meson.build
Flow Random
This sample shows how to use match.parser_meta.random field for 2 different use-cases:
Sampling – sampling certain percentage of traffic regardless of flow content
Distribution – distributing traffic in 8 different queues
The sample logic includes:
Initializing DOCA Flow by indicating mode_args="vnf,hws" in the doca_flow_cfg struct.
Starting two DOCA Flow ports.
On each port:
Building a root pipe with changeable 5-tuple match and forwarding to specific use-case pipe according to changeable source IP address.
Adding two entries to the pipe with different source IP match, and different forward:
The first entry with forward to the sampling pipe.
The second entry with forward to the distribution pipe.
Building a "sampling" pipe with a single entry and preparing the entry to sample 12.5% of traffic.
Building a "distribution" hash pipe with 8 entries and preparing the entries to get 12.5% of traffic for each queue.
Waiting 15 seconds and querying the entries (total packets after sampling/distribution related to total packets before).
Reference:
/opt/mellanox/doca/samples/doca_flow/flow_random/flow_random_sample.c
/opt/mellanox/doca/samples/doca_flow/flow_random/flow_random_main.c
/opt/mellanox/doca/samples/doca_flow/flow_random/meson.build
Flow RSS ESP
This sample shows how to use DOCA Flow forward RSS according to ESP SPI field, and distribute the traffic between queues.
The sample logic includes:
Initializing DOCA Flow by indicating mode_args="vnf,hws" in the doca_flow_cfg struct.
Starting two DOCA Flow ports.
On each port:
Building a pipe with both L3 and L4 types match, copy the SPI field into packet meta data, and forwarding to RSS with 7 queues.
Adding an entry with both IPv4 and ESP existence matching.
Waiting 15 seconds for traffic to arrived.
On each port:
Calculates the traffic percentage distributed into each port and prints the result.
Printing for each packet its SPI value. (only in debug mode, -l ≥ 60)
Reference:
/opt/mellanox/doca/samples/doca_flow/flow_rss_esp/flow_rss_esp_sample.c
/opt/mellanox/doca/samples/doca_flow/flow_rss_esp/flow_rss_esp_main.c
/opt/mellanox/doca/samples/doca_flow/flow_rss_esp/meson.build
Flow RSS Meta
This sample shows how to use DOCA Flow forward RSS, set meta action, and then retrieve the matched packets in the sample.
The sample logic includes:
Initializing DOCA Flow by indicating mode_args="vnf,hws" in the doca_flow_cfg struct.
Starting two DOCA Flow ports.
On each port:
Building a pipe with a changeable 5-tuple match, forwarding to RSS queue with index 0, and setting changeable packet meta data.
Adding an entry with an example 5-tuple and metadata value to the pipe.
Retrieving the packets on both ports from a receive queue , and printing the packet metadata value.
Reference:
/opt/mellanox/doca/samples/doca_flow/flow_rss_meta/flow_rss_meta_sample.c
/opt/mellanox/doca/samples/doca_flow/flow_rss_meta/flow_rss_meta_main.c
/opt/mellanox/doca/samples/doca_flow/flow_rss_meta/meson.build
Flow Sampling
This sample shows how to sample certain percentage of traffic regardless of flow content using doca_flow_match_condition structure with parser_meta.random.value field string.
This sample is supported for ConnectX-7/BlueField-3 and above.
The sample logic includes:
Initializing DOCA Flow by indicating mode_args="switch,hws" in the doca_flow_cfg struct.
Starting DOCA Flow ports: Physical port and two SF representors.
On switch port:
Building a root pipe with changeable 5-tuple match and forwarding to sampling pipe.
Adding entry with an example 5-tuple to the pipe.
Building a "sampling" control pipe with a single entry.
calculating the requested random value for getting 35% of traffic.
Adding entry with an example condition random value to the pipe.
Waiting 15 seconds and querying the entries (total packets after sampling related to total packets before).
Reference:
/opt/mellanox/doca/samples/doca_flow/flow_sampling/flow_sampling_sample.c
/opt/mellanox/doca/samples/doca_flow/flow_sampling/flow_sampling_main.c
/opt/mellanox/doca/samples/doca_flow/flow_sampling/meson.build
Flow Set Meta
This sample shows how to use the DOCA Flow set metadata action and then match on it.
The sample logic includes:
Initializing DOCA Flow by indicating mode_args="vnf,hws" in the doca_flow_cfg struct.
Starting two DOCA Flow ports.
On each port:
Building a pipe with a changeable match on metadata and forwarding the matched packets to the second port.
Adding an entry that matches an example metadata value.
Building a pipe with changeable 5-tuple match, changeable metadata action, and fwd to the first pipe.
Adding entry with an example 5-tuple and metadata value to the pipe.
Reference:
/opt/mellanox/doca/samples/doca_flow/flow_set_meta/flow_set_meta_sample.c
/opt/mellanox/doca/samples/doca_flow/flow_set_meta/flow_set_meta_main.c
/opt/mellanox/doca/samples/doca_flow/flow_set_meta/meson.build
Flow Shared Counter
This sample shows how to use the DOCA Flow shared counter and query it to get the counter statistics.
The sample logic includes:
Initializing DOCA Flow by indicating mode_args="vnf,hws" in the doca_flow_cfg struct.
Starting two DOCA Flow ports.
On each port:
Binding the shared counter to the port.
Building a pipe with changeable 5-tuple match with UDP protocol, changeable shared counter ID and forwarding the matched packets to the second port.
Adding an entry with an example 5-tuple match and shared counter with ID=port_id.
Building a pipe with changeable 5-tuple match with TCP protocol, changeable shared counter ID and forwarding the matched packets to the second port.
Adding an entry with an example 5-tuple match and shared counter with ID=port_id.
Building a control pipe with the following entries:
If L4 type is UDP, forwards the packets to the UDP pipe
If L4 type is TCP, forwards the packets to the TCP pipe
Waiting 5 seconds and querying the shared counters (total bytes and total packets).
Reference:
/opt/mellanox/doca/samples/doca_flow/flow_shared_counter/flow_shared_counter_sample.c
/opt/mellanox/doca/samples/doca_flow/flow_shared_counter/flow_shared_counter_main.c
/opt/mellanox/doca/samples/doca_flow/flow_shared_counter/meson.build
Flow Shared Meter
This sample shows how to use the DOCA Flow shared meter.
The sample logic includes:
Initializing DOCA Flow by indicating mode_args="vnf,hws" in the doca_flow_cfg struct.
Starting two DOCA Flow ports.
On each port:
Config a shared meter with specific cir and cbs values.
Binding the shared meter to the port.
Building a pipe with a changeable 5-tuple match with UDP protocol, changeable shared meter ID and forwarding the matched packets to the second port.
Adding an entry with an example 5-tuple match and shared meter with ID=port_id.
Building a pipe with a changeable 5-tuple match with TCP protocol, changeable shared meter ID and forwarding the matched packets to the second port.
Adding an entry with an example 5-tuple match and shared meter with ID=port_id.
Building a control pipe with the following entries:
If L4 type is UDP, forwards the packets to the UDP pipe
If L4 type is TCP, forwards the packets to the TCP pipe
Reference:
/opt/mellanox/doca/samples/doca_flow/flow_shared_meter/flow_shared_meter_sample.c
/opt/mellanox/doca/samples/doca_flow/flow_shared_meter/flow_shared_meter_main.c
/opt/mellanox/doca/samples/doca_flow/flow_shared_meter/meson.build
Flow Switch Control Pipe
This sample shows how to use the DOCA Flow control pipe in switch mode.
The sample logic includes:
Initializing DOCA Flow by indicating mode_args="switch,hws" in the doca_flow_cfg struct.
Starting two DOCA Flow ports.
On each port:
Building control pipe with match on VNI field.
Adding two entries to the control pipe, both matching TRANSPORT (UDP or TCP proto) over IPv4 with source port 80 and forwarding to the other port, where the first entry matches destination port 1234 and the second 12345.
Both entries have counters, so that after the successful insertions of both entries, the sample queries those counters to check the number of matched packets per entry.
Reference:
/opt/mellanox/doca/samples/doca_flow/flow_switch_control_pipe/flow_switch_control_pipe_sample.c
/opt/mellanox/doca/samples/doca_flow/flow_switch_control_pipe/flow_switch_control_pipe_main.c
/opt/mellanox/doca/samples/doca_flow/flow_switch_control_pipe/meson.build
Flow Switch – Multiple Switches
This sample illustrates how to use two switches working concurrently on two different physical functions.
It shows how to build a basic pipe in a switch and hardware steering (HWS) mode. Each pipe contains two entries, each of which forwards matched packets to two different representors.
The sample also demonstrates how to obtain the switch port of a given port using doca_flow_port_switch_get().
The test requires two PFs with two (either VF or SF) representors on each.
The sample logic includes:
Initializing DOCA Flow by indicating mode_args="switch,hws" in the doca_flow_cfg struct.
Starting DOCA Flow ports: Two physical ports and two representors each (totaling six ports).
On the switch port:
Building a basic pipe while indicating which fields to match on using struct doca_flow_match match.
Adding two entries to the created pipe, each of which forwards packets to a different port representor.
Reference:
/opt/mellanox/doca/samples/doca_flow/flow_switch/flow_switch_sample.c
/opt/mellanox/doca/samples/doca_flow/flow_switch/flow_switch_main.c
/opt/mellanox/doca/samples/doca_flow/flow_switch/meson.build
Flow Switch – Single Switch
This sample is identical to the previous sample, before the flow switch sample was extended to take advantage of the capabilities of DOCA to support multiple switches concurrently, each based on a different physical device.
The reason we add this original version is that it removes the constraints imposed by the modified flow switch version, allowing to use arbitrary number of representors in the switch configuration.
The logic of this sample is identical to that of the previous sample.
Reference:
/opt/mellanox/doca/samples/doca_flow/flow_switch_single/flow_switch_single_sample.c
/opt/mellanox/doca/samples/doca_flow/flow_switch_single/flow_switch_single_main.c
/opt/mellanox/doca/samples/doca_flow/flow_switch_single/meson.build
Flow Switch (Direction Info)
This sample illustrates how to give a hint to the driver for potential optimizations based on the direction information.
This sample requires a single PF with two representors (either VF or SF).
The sample also demonstrates usage of the match.parser_meta.port_meta to detect by the switch pipe the source from where the packet has arrived.
The sample logic includes:
Initializing DOCA Flow by indicating mode_args="switch,hws" in the doca_flow_cfg struct.
Starting 3 DOCA Flow ports, 1 physical port and 2 representors.
On the switch port:
Network-to-host pipe:
Building basic pipe with a changeable ipv4.next_proto field and configuring the pipe with the hint of direction by setting attr.dir_info = DOCA_FLOW_DIRECTION_NETWORK_TO_HOST.
Adding two entries:
If ipv4.next_proto is TCP, the packet is forwarded to the first representor, to the host.
If ipv4.next_proto is UDP, the packet is forwarder to the second representor, to the host.
Host-to-network pipe:
Building a basic pipe with a match on aa:aa:aa:aa:aa:aa as a source MAC address and configuring a pipe with the hint of direction by setting attr.dir_info = DOCA_FLOW_DIRECTION_HOST_TO_NETWORK.
Adding an entry. If the source MAC is matched, forward the packet to the physical port (i.e., to the network).
Switch pipe:
Building a basic pipe with a changeable parser_meta.port_meta to detect where the packet has arrived from.
Adding 3 entries:
If the packet arrived from port 0 (i.e., the network), forward it to the network-to-host pipe to decide for further logic
If the packet arrived from port 1 (i.e., the host's first representor), forward it to the host-to-network pipe to decide for further logic
If the packet arrived from port 2, (i.e., the host's second representor), forward it to the host-to-network pipe to decide for further logic
Reference:
/opt/mellanox/doca/samples/doca_flow/flow_switch_direction_info/flow_switch_direction_info_sample.c
/opt/mellanox/doca/samples/doca_flow/flow_switch_direction_info/flow_switch_direction_info_main.c
/opt/mellanox/doca/samples/doca_flow/flow_switch_direction_info/meson.build
Flow VXLAN Encap
This sample shows how to use DOCA Flow actions to create a VXLAN tunnel as well as illustrating the usage of matching TCP and UDP packets in the same pipe.
The sample logic includes:
Initializing DOCA Flow by indicating mode_args="vnf,hws" in the doca_flow_cfg struct.
Starting two DOCA Flow ports.
On each port:
Building a pipe with changeable 5-tuple match, encap action, and forward port action.
Adding example 5-tuple and encapsulation values entry to the pipe. Every TCP or UDP over IPv4 packet with the same 5-tuple is matched and encapsulated.
Reference:
/opt/mellanox/doca/samples/doca_flow/flow_vxlan_encap/flow_vxlan_encap_sample.c
/opt/mellanox/doca/samples/doca_flow/flow_vxlan_encap/flow_vxlan_encap_main.c
/opt/mellanox/doca/samples/doca_flow/flow_vxlan_encap/meson.build
Flow Shared Mirror
This sample shows how to use the DOCA Flow shared mirror.
The sample logic includes:
Initializing DOCA Flow by indicating mode_args="vnf,hws" in the doca_flow_cfg struct.
Starting two DOCA Flow ports.
On each port:
Configuring a shared mirror with a clone destination hairpin to the second port.
Binding the shared mirror to the port.
Building a pipe with a changeable 5-tuple match with UDP protocol, changeable shared mirror ID, and forwarding the matched packets to the second port.
Adding an entry with an example 5-tuple match and shared mirror with ID=port_id+1.
Building a pipe with a changeable 5-tuple match with TCP protocol, changeable shared mirror ID, and forwarding the matched packets to the second port.
Adding an entry with an example 5-tuple match and shared mirror with ID=port_id+1.
Building a control pipe with the following entries:
If L4 type is UDP, forwards the packets to the UDP pipe
If L4 type is TCP, forwards the packets to the TCP pipe
Waiting 15 seconds to clone any incoming traffic. Should see the same two packets received on the second port (one from the clone and another from the original).
Reference:
/opt/mellanox/doca/samples/doca_flow/flow_shared_mirror/flow_shared_mirror_sample.c
/opt/mellanox/doca/samples/doca_flow/flow_shared_mirror/flow_shared_mirror_main.c
/opt/mellanox/doca/samples/doca_flow/flow_shared_mirror/meson.build
Flow Match Comparison
This sample shows how to use the DOCA Flow match with a comparison result.
The sample logic includes:
Initializing DOCA Flow by indicating mode_args="vnf,hws" in the doca_flow_cfg struct.
Starting two DOCA Flow ports.
On each port:
Building a pipe with a changeable match on meta_data[0] and forwarding the matched packets to the second port.
Adding an entry that matches on meta_data[0] equal with TCP header length.
Building a control pipe for comparison purpose.
Adding an entry to the control pipe match with comparison result the meta_data[0] value greater than meta_data[1] and forwarding the matched packets to match with the meta pipe.
Building a pipe with a changeable 5-tuple match, copying ipv4.total_len to meta_data[1], and accumulating ipv4.version_ihl << 2 tcp.data_offset << 2 to meta_data[1], then forwarding to the second pipe.
Adding an example 5-tuple entry to the pipe.
Reference:
/opt/mellanox/doca/samples/doca_flow/flow_match_comparison/flow_match_comparison_sample.c
/opt/mellanox/doca/samples/doca_flow/flow_match_comparison/flow_match_comparison_main.c
/opt/mellanox/doca/samples/doca_flow/flow_match_comparison/meson.build
Flow Pipe Resize
This sample shows how the DOCA Flow pipe resize feature behaves as pipe size increases.
The sample logic includes:
Initializing DOCA Flow by indicating mode_args="switch,hws,cpds" in the doca_flow_cfg struct.
InfoBy default, a control pipe's internal tables have a default size of 64 entries. Using the CPDS (control pipe dynamic size) mode, each table's initial size matches the control pipe size.
Starting a PF port with two representors of subfunction (SF) or virtual functions (VF). For example:
./doca_flow_pipe_resize -p 03:00.0 -r sf[0-1] -- -l 60
Starting with a control pipe of a max size of 10 entries then adding 80 entries. Instead of failing on adding the 11th entry, the pipe continues increasing in the following manner:
Receiving a CONGESTION_REACHED callback whenever the number of current entries exceeds a threshold level of 80%.
Calling doca_flow_pipe_resize() with threshold percentage of 50%. Roughly, the new size is calculated as: (current entries) / (50%) rounded up to the nearest power of 2. A callback can indicate the exact number of entries.
Receiving a callback on the exact new calculated size of the pipe:
typedef doca_error_t (*doca_flow_pipe_resize_nr_entries_changed_cb)(void *pipe_user_ctx, uint32_t nr_entries);
Start calling doca_flow_entries_process() in a loop on each thread ID to trigger the entry relocations.
InfoThe loop should continue as long as the resize process was not ended.
Receiving a callback on each entry relocated to the new resized pipe:
typedef doca_error_t (*doca_flow_pipe_resize_entry_relocate_cb)(void *pipe_user_ctx, uint16_t pipe_queue, void *entry_user_ctx, void **new_entry_user_ctx)
Receiving a PIPE_RESIZED callback upon completion of the resize process. At this point, calling doca_flow_entries_process() should stop.
InfoThe resize cycles described above repeats five times increasing the pipe sizes in these steps: 10 -> 16 -> 32 -> 64 -> 128.
InfoThe pipe control entries define a match on increasing destination IP address. The fwd action send packet to the other port.
Waiting 5 seconds to send any traffic that matches the flows and seeing them on the other port.
Reference:
/opt/mellanox/doca/samples/doca_flow/flow_pipe_resize/flow_pipe_resize_sample.c
/opt/mellanox/doca/samples/doca_flow/flow_pipe_resize/flow_pipe_resize_main.c
/opt/mellanox/doca/samples/doca_flow/flow_pipe_resize/meson.build
Flow Entropy
This sample shows how to use the DOCA Flow entropy calculation.
The sample logic includes:
Initializing DOCA Flow by indicating mode_args="switch,hws" in the doca_flow_cfg struct.
Starting one DOCA Flow port.
Configuring the doca_flow_entropy_format structure with 5-tuple values.
Calling to doca_flow_port_calc_entropy to get the calculated entropy.
Logging the calculated entropy.
Reference:
/opt/mellanox/doca/samples/doca_flow/flow_entropy/flow_entropy_sample.c
/opt/mellanox/doca/samples/doca_flow/flow_entropy/flow_entropy_main.c
/opt/mellanox/doca/samples/doca_flow/flow_entropy/meson.build
Flow VXLAN Shared Encap
This sample shows how to use DOCA Flow actions to create a VXLAN tunnel as well as illustrating the usage of matching TCP and UDP packets in the same pipe.
The VXLAN tunnel is created by shared_resource_encap.
The sample logic includes:
Initializing DOCA Flow by indicating mode_args="vnf,hws" in the doca_flow_cfg struct.
Starting two DOCA Flow ports.
On each port:
Configure and bind shared encap resources. The encap resources are for VXLAN encap.
Building a pipe with changeable 5-tuple match, shared_encap_id, and forward port action.
Adding example 5-tuple and encapsulation values entry to the pipe. Every TCP or UDP over IPv4 packet with the same 5-tuple is matched and encapsulated.
Reference:
/opt/mellanox/doca/samples/doca_flow/flow_vxlan_shared_encap/flow_vxlan_shared_encap_sample.c
/opt/mellanox/doca/samples/doca_flow/flow_vxlan_shared_encap/flow_vxlan_shared_encap_main.c
/opt/mellanox/doca/samples/doca_flow/flow_vxlan_shared_encap/meson.build
Supported Field String
The following is a list of all the API fields available for matching criteria and action execution.
|
String Field |
Path in Match Structure |
Set |
Add |
Dst Copy |
Src Copy |
|
meta.data |
meta.pkt_meta meta.u32[i] |
✔ |
✔ |
✔ |
✔ |
|
meta.mark |
meta.mark |
✘ |
✘ |
✘ |
✘ |
|
meta.flags |
flags |
✘ |
✘ |
✘ |
✘ |
|
parser_meta.hash.result |
None. See section "Copy Hash Result" for details. |
✘ |
✘ |
✘ |
✔ |
|
parser_meta.port.id |
parser_meta.port_meta |
✘ |
✘ |
✘ |
✘ |
|
parser_meta.ipsec.syndrome |
parser_meta.ipsec_syndrome |
✘ |
✘ |
✘ |
✘ |
|
parser_meta.psp.syndrome |
parser_meta.psp_syndrome |
✘ |
✘ |
✘ |
✘ |
|
parser_meta.random.value |
parser_meta.random |
✘ |
✘ |
✘ |
✘ |
|
parser_meta.meter.color |
parser_meta.meter_color |
✘ |
✘ |
✘ |
✘ |
|
parser_meta.packet_type.l2_outer |
parser_meta.outer_l2_type |
✘ |
✘ |
✘ |
✘ |
|
parser_meta.packet_type.l3_outer |
parser_meta.outer_l3_type |
✘ |
✘ |
✘ |
✘ |
|
parser_meta.packet_type.l4_outer |
parser_meta.outer_l4_type |
✘ |
✘ |
✘ |
✘ |
|
parser_meta.packet_type.l2_inner |
parser_meta.inner_l2_type |
✘ |
✘ |
✘ |
✘ |
|
parser_meta.packet_type.l3_inner |
parser_meta.inner_l3_type |
✘ |
✘ |
✘ |
✘ |
|
parser_meta.packet_type.l4_inner |
parser_meta.inner_l4_type |
✘ |
✘ |
✘ |
✘ |
|
parser_meta.outer_ip_fragmented.flag |
parser_meta.outer_ip_fragmented |
✘ |
✘ |
✘ |
✘ |
|
parser_meta.inner_ip_fragmented.flag |
parser_meta.inner_ip_fragmented |
✘ |
✘ |
✘ |
✘ |
|
parser_meta.outer_integrity.l3_ok |
parser_meta.outer_l3_ok |
✘ |
✘ |
✘ |
✘ |
|
parser_meta.outer_integrity.ipv4_checksum_ok |
parser_meta.outer_ip4_checksum_ok |
✘ |
✘ |
✘ |
✘ |
|
parser_meta.outer_integrity.l4_ok |
parser_meta.outer_l4_ok |
✘ |
✘ |
✘ |
✘ |
|
parser_meta.inner_integrity.l3_ok |
parser_meta.inner_l3_ok |
✘ |
✘ |
✘ |
✘ |
|
parser_meta.inner_integrity.ipv4_checksum_ok |
parser_meta.inner_ip4_checksum_ok |
✘ |
✘ |
✘ |
✘ |
|
parser_meta.inner_integrity.l4_ok |
parser_meta.inner_l4_ok |
✘ |
✘ |
✘ |
✘ |
|
outer.eth.dst_mac |
outer.eth.dst_mac |
✔ |
✘ |
✔ |
✔ |
|
outer.eth.src_mac |
outer.eth.src_mac |
✔ |
✘ |
✔ |
✔ |
|
outer.eth.type |
outer.eth.type |
✔ |
✘ |
✔ |
✔ |
|
outer.eth_vlan0.tci |
outer.eth_vlan[0].tci |
✔ |
✘ |
✔ |
✔ |
|
outer.eth_vlan1.tci |
outer.eth_vlan[1].tci |
✘ |
✘ |
✘ |
✘ |
|
outer.ipv4.src_ip |
outer.ip4.src_ip |
✔ |
✘ |
✔ |
✔ |
|
outer.ipv4.dst_ip |
outer.ip4.dst_ip |
✔ |
✘ |
✔ |
✔ |
|
outer.ipv4.dscp_ecn |
outer.ip4.dscp_ecn |
✔ |
✘ |
✔ |
✔ |
|
outer.ipv4.next_proto |
outer.ip4.next_proto |
✔ |
✘ |
✔ |
✔ |
|
outer.ipv4.ttl |
outer.ip4.ttl |
✔ |
✔ |
✔ |
✔ |
|
outer.ipv4.version_ihl |
outer.ip4.version_ihl |
✘ |
✘ |
✘ |
✘ |
|
outer.ipv4.total_len |
outer.ip4.total_len |
✘ |
✘ |
✘ |
✘ |
|
outer.ipv6.src_ip |
outer.ip6.src_ip |
✔ |
✘ |
✔ |
✔ |
|
outer.ipv6.dst_ip |
outer.ip6.dst_ip |
✔ |
✘ |
✔ |
✔ |
|
outer.ipv6.traffic_class |
outer.ip6.traffic_class |
✔ |
✘ |
✔ |
✔ |
|
outer.ipv6.next_proto |
outer.ip6.next_proto |
✔ |
✘ |
✔ |
✔ |
|
outer.ipv6.hop_limit |
outer.ip6.hop_limit |
✔ |
✔ |
✔ |
✔ |
|
outer.ipv6.payload_len |
outer.ip6.payload_len |
✘ |
✘ |
✘ |
✘ |
|
outer.udp.src_port |
outer.udp.l4_port.src_port |
✔ |
✘ |
✔ |
✔ |
|
outer.udp.dst_port |
outer.udp.l4_port.dst_port |
✔ |
✘ |
✔ |
✔ |
|
outer.transport.src_port |
outer.transport.src_port |
✔ |
✘ |
✔ |
✔ |
|
outer.transport.dst_port |
outer.transport.dst_port |
✔ |
✘ |
✔ |
✔ |
|
outer.tcp.src_port |
outer.tcp.l4_port.src_port |
✔ |
✘ |
✔ |
✔ |
|
outer.tcp.dst_port |
outer.tcp.l4_port.dst_port |
✔ |
✘ |
✔ |
✔ |
|
outer.tcp.flags |
outer.tcp.flags |
✘ |
✘ |
✘ |
✘ |
|
outer.tcp.data_offset |
outer.tcp.data_offset |
✘ |
✘ |
✘ |
✘ |
|
outer.icmp4.type |
outer.icmp.type |
✘ |
✘ |
✘ |
✘ |
|
outer.icmp4.code |
outer.icmp.code |
✘ |
✘ |
✘ |
✘ |
|
outer.icmp4.ident |
outer.icmp.ident |
✘ |
✘ |
✘ |
✘ |
|
outer.icmp6.type |
outer.icmp.type |
✘ |
✘ |
✘ |
✘ |
|
outer.icmp6.code |
outer.icmp.code |
✘ |
✘ |
✘ |
✘ |
|
tunnel.gre.protocol |
tun.protocol |
✘ |
✘ |
✘ |
✘ |
|
tunnel.gre_key.value |
tun.gre_key |
✘ |
✘ |
✘ |
✘ |
|
tunnel.vxlan.vni |
tun.vxlan_tun_id |
✔ |
✘ |
✔ |
✔ |
|
tunnel.gtp.teid |
tun.gtp_teid |
✔ |
✘ |
✔ |
✔ |
|
tunnel.esp.spi |
tun.esp_spi |
✔ |
✘ |
✔ |
✔ |
|
tunnel.esp.sn |
tun.esp_sn |
✔ |
✘ |
✔ |
✔ |
|
tunnel.mpls[0].label |
tun.mpls[0].label |
✘ |
✘ |
✘ |
✔ |
|
tunnel.mpls[1].label |
tun.mpls[1].label |
✘ |
✘ |
✘ |
✔ |
|
tunnel.mpls[2].label |
tun.mpls[2].label |
✘ |
✘ |
✘ |
✔ |
|
tunnel.mpls[3].label |
tun.mpls[3].label |
✘ |
✘ |
✘ |
✔ |
|
tunnel.mpls[4].label |
tun.mpls[4].label |
✘ |
✘ |
✘ |
✔ |
|
tunnel.geneve.ver_opt_len |
tun.geneve.ver_opt_len |
✘ |
✘ |
✘ |
✘ |
|
tunnel.geneve.o_c |
tun.geneve.o_c |
✘ |
✘ |
✘ |
✘ |
|
tunnel.geneve.next_proto |
tun.geneve.next_proto |
✘ |
✘ |
✘ |
✘ |
|
tunnel.geneve.vni |
tun.geneve.vni |
✔ |
✘ |
✔ |
✔ |
|
tunnel.geneve_opt[i].type |
None. See section "Copy Geneve Options" for details. |
✘ |
✘ |
✔ |
✔ |
|
tunnel.geneve_opt[i].class |
✘ |
✘ |
✔ |
✔ |
|
|
tunnel.geneve_opt[i].data |
✘ |
✘ |
✔ |
✔ |
|
|
tunnel.psp.nexthdr |
tun.psp.nexthdr |
✔ |
✘ |
✔ |
✔ |
|
tunnel.psp.hdrextlen |
tun.psp.hdrextlen |
✔ |
✘ |
✔ |
✔ |
|
tunnel.psp.res_cryptofst |
tun.psp.res_cryptofst |
✔ |
✘ |
✔ |
✔ |
|
tunnel.psp.s_v_ver_v |
tun.psp.s_d_ver_v |
✔ |
✘ |
✔ |
✔ |
|
tunnel.psp.spi |
tun.psp.spi |
✔ |
✘ |
✔ |
✔ |
|
tunnel.psp.iv |
tun.psp.iv |
✔ |
✘ |
✔ |
✔ |
|
tunnel.psp.vni |
tun.psp.vc |
✔ |
✘ |
✔ |
✔ |
|
inner.eth.dst_mac |
inner.eth.dst_mac |
✘ |
✘ |
✘ |
✘ |
|
inner.eth.src_mac |
inner.eth.src_mac |
✘ |
✘ |
✘ |
✘ |
|
inner.eth.type |
inner.eth.type |
✘ |
✘ |
✘ |
✘ |
|
inner.eth_vlan0.tci |
inner.eth_vlan[0].tci |
✘ |
✘ |
✘ |
✘ |
|
inner.eth_vlan1.tci |
inner.eth_vlan[1].tci |
✘ |
✘ |
✘ |
✘ |
|
inner.ipv4.src_ip |
inner.ip4.src_ip |
✘ |
✘ |
✘ |
✘ |
|
inner.ipv4.dst_ip |
inner.ip4.dst_ip |
✘ |
✘ |
✘ |
✘ |
|
inner.ipv4.dscp_ecn |
inner.ip4.dscp_ecn |
✘ |
✘ |
✘ |
✘ |
|
inner.ipv4.next_proto |
inner.ip4.next_proto |
✘ |
✘ |
✘ |
✘ |
|
inner.ipv4.ttl |
inner.ip4.ttl |
✘ |
✘ |
✘ |
✘ |
|
inner.ipv4.version_ihl |
inner.ip4.version_ihl |
✘ |
✘ |
✘ |
✘ |
|
inner.ipv4.total_len |
inner.ip4.total_len |
✘ |
✘ |
✘ |
✘ |
|
inner.ipv6.src_ip |
inner.ip6.src_ip |
✘ |
✘ |
✘ |
✘ |
|
inner.ipv6.dst_ip |
inner.ip6.dst_ip |
✘ |
✘ |
✘ |
✘ |
|
inner.ipv6.traffic_class |
inner.ip6.traffic_class |
✘ |
✘ |
✘ |
✘ |
|
inner.ipv6.next_proto |
inner.ip6.next_proto |
✘ |
✘ |
✘ |
✘ |
|
inner.ipv6.hop_limit |
inner.ip6.hop_limit |
✘ |
✘ |
✘ |
✘ |
|
inner.ipv6.payload_len |
inner.ip6.payload_len |
✘ |
✘ |
✘ |
✘ |
|
inner.udp.src_port |
inner.udp.l4_port.src_port |
✘ |
✘ |
✘ |
✘ |
|
inner.udp.dst_port |
inner.udp.l4_port.dst_port |
✘ |
✘ |
✘ |
✘ |
|
inner.transport.src_port |
inner.transport.src_port |
✘ |
✘ |
✘ |
✘ |
|
inner.transport.dst_port |
inner.transport.dst_port |
✘ |
✘ |
✘ |
✘ |
|
inner.tcp.src_port |
inner.tcp.l4_port.src_port |
✘ |
✘ |
✘ |
✘ |
|
inner.tcp.dst_port |
inner.tcp.l4_port.dst_port |
✘ |
✘ |
✘ |
✘ |
|
inner.tcp.flags |
inner.tcp.flags |
✘ |
✘ |
✘ |
✘ |
|
inner.tcp.data_offset |
inner.tcp.data_offset |
✘ |
✘ |
✘ |
✘ |
|
inner.icmp4.type |
inner.icmp.type |
✘ |
✘ |
✘ |
✘ |
|
inner.icmp4.code |
inner.icmp.code |
✘ |
✘ |
✘ |
✘ |
|
inner.icmp4.ident |
inner.icmp.ident |
✘ |
✘ |
✘ |
✘ |
|
inner.icmp6.type |
inner.icmp.type |
✘ |
✘ |
✘ |
✘ |
|
inner.icmp6.code |
inner.icmp.code |
✘ |
✘ |
✘ |
✘ |
Supported Non-field String
Users can modify fields which are not included in doca_flow_match structure.
Copy Hash Result
Users can copy the the matcher hash calculation into other fields using the "parser_meta.hash" string.
Copy GENEVE Options
User can copy GENEVE option type/class/data using the following strings:
"tunnel.geneve_opt[i].type" – Copy from/to option type (only for option configured with DOCA_FLOW_PARSER_GENEVE_OPT_MODE_MATCHABLE).
"tunnel.geneve_opt[i].class" – Copy from/to option class (only for option configured with DOCA_FLOW_PARSER_GENEVE_OPT_MODE_MATCHABLE).
"tunnel.geneve_opt[i].data" – Copy from/to option data, the bit offset is from the start of the data.
i is the index of the option in tlv_list array provided in doca_flow_parser_geneve_opt_create.