NVIDIA DOCA Eth L2 Forwarding Application Guide

This document provides an Ethernet L2 Forwarding implementation on top of the NVIDIA® BlueField® DPU.

Introduction

Th e Ethernet L2 Forwarding applicatio n is a DOCA Ethernet based application that forwards traffic from a single RX port to a single TX port and vice versa, leveraging DOCA's task/event batching feature for enhanced performance.

The application can run both on the Host and the BlueField, and has two main modes:

Two-sided forwarding – device 1 → device 2 and device 2 → device 1
One-sided forwarding – device 1 → device 2 or device 2 → device 1

The one-sided mode offers better performance, enlarging the packets forwarding rate.

System Design

The Ethernet L2 Forwarding application runs on the host or the BlueField.

image-2024-4-24_19-22-9-1-version-1-modificationdate-1715184477330-api-v2.png

----------------------------------------------------------------------------------------------------------------------

image-2024-4-25_18-4-10-1-version-1-modificationdate-1715184477773-api-v2.png

Application Architecture

The Ethernet L2 Forwarding application runs on top of the DOCA Ethernet API to form an (two/one-sided) L2 forwarding between two ports.

image-2024-4-28_12-3-37-1-version-1-modificationdate-1715184478247-api-v2.png

Two DOCA devices are opened.
Two DOCA mmaps are created.
Two DOCA Flow ports are configured and started, each with a different opened DOCA device.
Two DOCA Ethernet TXQ and RXQ contexts are initialized, each TXQ-RXQ pair with a different opened DOCA device such that traffic is steered from the device to the corresponding RXQ, and from the corresponding TXQ to the device.
Forwarding - Packets received by device x are steered to RXQ x, then allocated to TXQ y and sent by device y (and vice versa).

DOCA Libraries

This application leverages the following DOCA libraries:

For additional information about the used DOCA libraries, please refer to the respective programming guides.

Compiling the Application

Installation

Please refer to the NVIDIA DOCA Installation Guide for Linux for details on how to install BlueField-related software.

Overview

The installation of DOCA's reference applications contains the sources of the applications, alongside the matching compilation instructions. This allows for both compilation of the applications "as-is", as well as provides the ability to modify the sources and then compile the new version of the application. For more information about the applications, as well as development and compilation tips, please refer to the DOCA Applications main guide.

The sources of the application can be found under the application's directory: /opt/mellanox/doca/applications/eth_l2_fwd/.

Compiling All Applications

The applications are all defined under a single meson project, meaning that the default compilation will compile all the DOCA applications.

To build all the applications together, run:

Copy
Copied!

            
            cd /opt/mellanox/doca/applications/
meson /tmp/build 
ninja -C /tmp/build

Note

doca_eth_l2_fwd will be created under /tmp/build/eth_l2_fwd/.

Compiling Only the Current Application

To directly build only the Ethernet L2 Forwarding application:

Copy
Copied!

            
            cd /opt/mellanox/doca/applications/
meson /tmp/build -Denable_all_applications=false -Denable_eth_l2_fwd=true
ninja -C /tmp/build

Note

doca_eth_l2_fwd will be created under /tmp/build/eth_l2_fwd/.

Alternatively, one can set the desired flags in the meson_options.txt file instead of providing them in the compilation command line:
1. Edit the following flags in /opt/mellanox/doca/applications/meson_options.txt:
  - Set enable_all_applications to false
  - Set enable_eth_l2_fwd to true
2. The same compilation commands should be used, as were shown in the previous section:
  Copy
  
  Copied!
```
            
            cd /opt/mellanox/doca/applications/
meson /tmp/build 
ninja -C /tmp/build
        
```
  Note
  
  doca_eth_l2_fwd will be created under /tmp/build/eth_l2_fwd/.

Copy
Copied!

            
            Usage: doca_eth_l2_fwd [DOCA Flags] [Program Flags]
 
DOCA Flags:
  -h, --help                        Print a help synopsis
  -v, --version                     Print program version information
  -l, --log-level                   Set the (numeric) log level for the program <10=DISABLE, 20=CRITICAL, 30=ERROR, 40=WARNING, 50=INFO, 60=DEBUG, 70=TRACE>
  --sdk-log-level                   Set the SDK (numeric) log level for the program <10=DISABLE, 20=CRITICAL, 30=ERROR, 40=WARNING, 50=INFO, 60=DEBUG, 70=TRACE>
  -j, --json <path>                 Parse all command flags from an input json file
 
Program Flags:
  -d, --devs-names <name1,name2>    Set two IB devices names separated by a comma, without spaces.
  -r, --rate <rate>                 Set packets receive rate (in [MB/s]), default is 12500.
  -ps, --pkt-size <size>            Set max packet size (in [B]), default is 1600.
  -t, --time <time>                 Set packet max process time (in [μs]), default is 1.
  -nt, --num-tasks <num>            Set number of tasks per batch, default is 128.
  -nb, --num-batches <num>          Set number of task batches, default is 32.
  -o, --one-sided-forwarding <num>  Set one-sided forwarding: 0 - two-sided forwarding, 1 - device 1 -> device 2, 2 - device 2 -> device 1. default is 0.
  -f, --max-forwardings <num>       Set max forwardings after which the application run will end, default is 0, meaning no limit.

For additional information, please refer to the Command Line Flags section below.

Note

The above usage printout can be printed to the command line using the -h (or --help) options:

Copy
Copied!

            
            ./doca_eth_l2_fwd -h

CLI example for running the application either on the BlueField or on the host:
Copy

Copied!
```
            
            ./doca_eth_l2_fwd -d mlx5_0,mlx5_1
        
```
Note

Both IB devices identifiers ( mlx5_0, mlx5_1 ) should match the identifiers of the desired IB devices.
The application also supports a JSON-based deployment mode, in which all command-line arguments are provided through a JSON file:
Copy

Copied!
```
            
            ./doca_eth_l2_fwd --json [json_file]
        
```
For example:
Copy

Copied!
```
            
            ./doca_eth_l2_fwd --json ./eth_l2_fwd_params.json
        
```
Note

Before execution, please ensure that the used JSON file contains the correct configuration parameters, and especially the desired IB devices names needed for the deployment.

Command Line Flags

Flag Type	Short Flag	Long Flag/JSON Key	Description	JSON Content
General flags	`h`	`help`	Prints a help synopsis	N/A
	`v`	`version`	Prints program version information	N/A
	`l`	`log-level`	Set the log level for the application: DISABLE=10 CRITICAL=20 ERROR=30 WARNING=40 INFO=50 DEBUG=60 TRACE=70 (Requires compilation with Trace level support)	Copy Copied! `"log-level": 60`
	N/A	`sdk-log-level`	Sets the log level for the program: DISABLE=10 CRITICAL=20 ERROR=30 WARNING=40 INFO=50 DEBUG=60 TRACE=70	N/A
	`j`	`json`	Parse all command flags from an input json file	N/A
Program flags	`d`	`devs-names`	Two IB devices names, separated by a comma, without spaces. Note This is a mandatory flag.	Copy Copied! `"devs-names": "mlx5_0,mlx5_1"`
	`r`	`rate`	The rate (in [MB/s]) in which the RX port is expected to receive traffic.	Copy Copied! `"rate": 12500`
	`ps`	`pkt-size`	The maximum size (in [B]) of a received packet.	Copy Copied! `"pkt-size": 1600`
	`t`	`time`	The maximum time taking to process a single packet.	Copy Copied! `"time": 1`
	`nt`	`num-tasks`	The number of tasks to set per a single task batch.	Copy Copied! `"num-tasks": 128`
	`nb`	`num-batches`	The number of task batches to set for the TX side.	Copy Copied! `"num-batches": 32`
	`o`	`one-sided-forwarding`	Flag to set one of 3 options: 0 - Two-sided forwarding. 1 - One-sided forwarding from device 1 to device 2. 2 - One-sided forwarding from device 2 to device 1.	Copy Copied! `"one-sided-forwarding": 0`
	`f`	`max-forwardings`	The maximum number of forwarding	Copy Copied! `"max-forwardings": 32`

Refer to DOCA Arg Parser for more information regarding the supported flags and execution modes.

Troubleshooting

Please refer to the NVIDIA DOCA Troubleshooting Guide for any issue you may encounter with the installation or execution of the DOCA applications.

Application Code Flow

Parse application argument.

Initialize Arg parser resources and register DOCA general parameters.

Copy
Copied!

            
            doca_argp_init();

Register Ethernet L2 Forwarding application parameters.

Copy
Copied!

            
            register_eth_l2_forwarding_params();

Parse the arguments.

Copy
Copied!

            
            doca_argp_start();

Parse DOCA flags.
Parse application parameters.

Execute Ethernet L2 Forwarding application main logic.
Copy

Copied!
```
            
            eth_l2_fwd_execute();
        
```
1. Open the two chosen DOCA devices.
2. Initialize necessary DOCA Core objects.
3. Initialize ETH RXQ/TXQ contexts for the devices.
4. Forward packets.
Clean up application resources.
Copy

Copied!
```
            
            eth_l2_fwd_cleanup();
        
```
1. Stop all contexts and drain tasks.
2. Free all application resources.

Arg parser destroy.

Copy
Copied!

            
            doca_argp_destroy()

References

/opt/mellanox/doca/applications/eth_l2_fwd/
/opt/mellanox/doca/applications/eth_l2_fwd/eth_l2_fwd_params.json

On This Page

NVIDIA DOCA Eth L2 Forwarding Application Guide

Introduction

System Design

Application Architecture

DOCA Libraries

Compiling the Application

Installation

Overview

Compiling All Applications

Compiling Only the Current Application

Troubleshooting

Running the Application

Application Execution

Command Line Flags

Troubleshooting

Application Code Flow

References