NVIDIA DOCA Eth L2 Forwarding Application Guide

1.0

This document provides an Ethernet L2 Forwarding implementation on top of the NVIDIA® BlueField® DPU.

Th e Ethernet L2 Forwarding applicatio n is a DOCA Ethernet based application that forwards traffic from a single RX port to a single TX port and vice versa, leveraging DOCA's task/event batching feature for enhanced performance.

The application can run both on the Host and the BlueField, and has two main modes:

  • Two-sided forwarding – device 1 → device 2 and device 2 → device 1

  • One-sided forwarding – device 1 → device 2 or device 2 → device 1

The one-sided mode offers better performance, enlarging the packets forwarding rate.

The Ethernet L2 Forwarding application runs on the host or the BlueField.

image-2024-4-24_19-22-9-1-version-1-modificationdate-1715184477330-api-v2.png

----------------------------------------------------------------------------------------------------------------------

image-2024-4-25_18-4-10-1-version-1-modificationdate-1715184477773-api-v2.png

The Ethernet L2 Forwarding application runs on top of the DOCA Ethernet API to form an (two/one-sided) L2 forwarding between two ports.

image-2024-4-28_12-3-37-1-version-1-modificationdate-1715184478247-api-v2.png

  1. Two DOCA devices are opened.

  2. Two DOCA mmaps are created.

  3. Two DOCA Flow ports are configured and started, each with a different opened DOCA device.

  4. Two DOCA Ethernet TXQ and RXQ contexts are initialized, each TXQ-RXQ pair with a different opened DOCA device such that traffic is steered from the device to the corresponding RXQ, and from the corresponding TXQ to the device.

  5. Forwarding - Packets received by device x are steered to RXQ x, then allocated to TXQ y and sent by device y (and vice versa).

This application leverages the following DOCA libraries:

For additional information about the used DOCA libraries, please refer to the respective programming guides.

Installation

Please refer to the NVIDIA DOCA Installation Guide for Linux for details on how to install BlueField-related software.

Overview

The installation of DOCA's reference applications contains the sources of the applications, alongside the matching compilation instructions. This allows for both compilation of the applications "as-is", as well as provides the ability to modify the sources and then compile the new version of the application. For more information about the applications, as well as development and compilation tips, please refer to the DOCA Applications main guide.

The sources of the application can be found under the application's directory: /opt/mellanox/doca/applications/eth_l2_fwd/.

Compiling All Applications

The applications are all defined under a single meson project, meaning that the default compilation will compile all the DOCA applications.

To build all the applications together, run:

Copy
Copied!
            

cd /opt/mellanox/doca/applications/ meson /tmp/build ninja -C /tmp/build

Note

doca_eth_l2_fwd will be created under /tmp/build/eth_l2_fwd/.


Compiling Only the Current Application

  1. To directly build only the Ethernet L2 Forwarding application:

    Copy
    Copied!
                

    cd /opt/mellanox/doca/applications/ meson /tmp/build -Denable_all_applications=false -Denable_eth_l2_fwd=true ninja -C /tmp/build

    Note

    doca_eth_l2_fwd will be created under /tmp/build/eth_l2_fwd/.

  2. Alternatively, one can set the desired flags in the meson_options.txt file instead of providing them in the compilation command line:

    1. Edit the following flags in /opt/mellanox/doca/applications/meson_options.txt:

      • Set enable_all_applications to false

      • Set enable_eth_l2_fwd to true

    2. The same compilation commands should be used, as were shown in the previous section:

      Copy
      Copied!
                  

      cd /opt/mellanox/doca/applications/ meson /tmp/build ninja -C /tmp/build

      Note

      doca_eth_l2_fwd will be created under /tmp/build/eth_l2_fwd/.

Troubleshooting

Please refer to the NVIDIA DOCA Troubleshooting Guide for any issue you may encounter with the compilation of the DOCA applications.

Application Execution

The Ethernet L2 Forwarding application is provided in source form, hence a compilation is required before the application can be executed.

  1. Application usage instructions:

    Copy
    Copied!
                

    Usage: doca_eth_l2_fwd [DOCA Flags] [Program Flags]   DOCA Flags: -h, --help Print a help synopsis -v, --version Print program version information -l, --log-level Set the (numeric) log level for the program <10=DISABLE, 20=CRITICAL, 30=ERROR, 40=WARNING, 50=INFO, 60=DEBUG, 70=TRACE> --sdk-log-level Set the SDK (numeric) log level for the program <10=DISABLE, 20=CRITICAL, 30=ERROR, 40=WARNING, 50=INFO, 60=DEBUG, 70=TRACE> -j, --json <path> Parse all command flags from an input json file   Program Flags: -d, --devs-names <name1,name2> Set two IB devices names separated by a comma, without spaces. -r, --rate <rate> Set packets receive rate (in [MB/s]), default is 12500. -ps, --pkt-size <size> Set max packet size (in [B]), default is 1600. -t, --time <time> Set packet max process time (in [μs]), default is 1. -nt, --num-tasks <num> Set number of tasks per batch, default is 128. -nb, --num-batches <num> Set number of task batches, default is 32. -o, --one-sided-forwarding <num> Set one-sided forwarding: 0 - two-sided forwarding, 1 - device 1 -> device 2, 2 - device 2 -> device 1. default is 0. -f, --max-forwardings <num> Set max forwardings after which the application run will end, default is 0, meaning no limit.

    For additional information, please refer to the Command Line Flags section below.

    Note

    The above usage printout can be printed to the command line using the -h (or --help) options:

    Copy
    Copied!
                

    ./doca_eth_l2_fwd -h

  2. CLI example for running the application either on the BlueField or on the host:

    Copy
    Copied!
                

    ./doca_eth_l2_fwd -d mlx5_0,mlx5_1

    Note

    Both IB devices identifiers ( mlx5_0, mlx5_1 ) should match the identifiers of the desired IB devices.

  3. The application also supports a JSON-based deployment mode, in which all command-line arguments are provided through a JSON file:

    Copy
    Copied!
                

    ./doca_eth_l2_fwd --json [json_file]

    For example:

    Copy
    Copied!
                

    ./doca_eth_l2_fwd --json ./eth_l2_fwd_params.json

    Note

    Before execution, please ensure that the used JSON file contains the correct configuration parameters, and especially the desired IB devices names needed for the deployment.

Command Line Flags

Flag Type

Short Flag

Long Flag/JSON Key

Description

JSON Content

General flags

h

help

Prints a help synopsis

N/A

v

version

Prints program version information

N/A

l

log-level

Set the log level for the application:

  • DISABLE=10

  • CRITICAL=20

  • ERROR=30

  • WARNING=40

  • INFO=50

  • DEBUG=60

  • TRACE=70 (Requires compilation with Trace level support)

Copy
Copied!
            

"log-level": 60

N/A

sdk-log-level

Sets the log level for the program:

  • DISABLE=10

  • CRITICAL=20

  • ERROR=30

  • WARNING=40

  • INFO=50

  • DEBUG=60

  • TRACE=70

N/A

j

json

Parse all command flags from an input json file

N/A

Program flags

d

devs-names

Two IB devices names, separated by a comma, without spaces.

Note

This is a mandatory flag.

Copy
Copied!
            

"devs-names": "mlx5_0,mlx5_1"

r

rate

The rate (in [MB/s]) in which the RX port is expected to receive traffic.

Copy
Copied!
            

"rate": 12500

ps

pkt-size

The maximum size (in [B]) of a received packet.

Copy
Copied!
            

"pkt-size": 1600

t

time

The maximum time taking to process a single packet.

Copy
Copied!
            

"time": 1

nt

num-tasks

The number of tasks to set per a single task batch.

Copy
Copied!
            

"num-tasks": 128

nb

num-batches

The number of task batches to set for the TX side.

Copy
Copied!
            

"num-batches": 32

o

one-sided-forwarding

Flag to set one of 3 options:

0 - Two-sided forwarding.

1 - One-sided forwarding from device 1 to device 2.

2 - One-sided forwarding from device 2 to device 1.

Copy
Copied!
            

"one-sided-forwarding": 0

f

max-forwardings

The maximum number of forwarding

Copy
Copied!
            

"max-forwardings": 32

Refer to DOCA Arg Parser for more information regarding the supported flags and execution modes.

Troubleshooting

Please refer to the NVIDIA DOCA Troubleshooting Guide for any issue you may encounter with the installation or execution of the DOCA applications.

  1. Parse application argument.

    1. Initialize Arg parser resources and register DOCA general parameters.

      Copy
      Copied!
                  

      doca_argp_init();

    2. Register Ethernet L2 Forwarding application parameters.

      Copy
      Copied!
                  

      register_eth_l2_forwarding_params();

    3. Parse the arguments.

      Copy
      Copied!
                  

      doca_argp_start();

      1. Parse DOCA flags.

      2. Parse application parameters.

  2. Execute Ethernet L2 Forwarding application main logic.

    Copy
    Copied!
                

    eth_l2_fwd_execute();

    1. Open the two chosen DOCA devices.

    2. Initialize necessary DOCA Core objects.

    3. Initialize ETH RXQ/TXQ contexts for the devices.

    4. Forward packets.

  3. Clean up application resources.

    Copy
    Copied!
                

    eth_l2_fwd_cleanup();

    1. Stop all contexts and drain tasks.

    2. Free all application resources.

  4. Arg parser destroy.

    Copy
    Copied!
                

    doca_argp_destroy()

  • /opt/mellanox/doca/applications/eth_l2_fwd/

  • /opt/mellanox/doca/applications/eth_l2_fwd/eth_l2_fwd_params.json

© Copyright 2024, NVIDIA. Last updated on May 21, 2024.