DOCA Bifurcated Driver Model Reference Application

This page provides an example of a Bifurcated Driver Model implementation using a multi-port eSwitch on top of the NVIDIA® BlueField® DPU.

The Bifurcated Driver Model leverages hardware capabilities to split incoming network traffic based on flow type. This application specifically uses the DOCA Flow library to demonstrate efficient traffic forwarding between Physical Ports (wire) and Linux Ports (kernel).

Port 0 ↔ Port 1 (Data Plane): High-volume traffic matching defined DOCA Flow rules are forwarded directly between Physical Port 0 and Physical Port 1. This bypasses the kernel network stack, remaining in the data plane for fast processing.

Port[0-1]↔ Kernel Pipeline (Control Plane): External traffic coming from Physical Port 0 or Physical Port 1 that is not TCP/UDP (e.g., ARP, ICMPv6, or specific control protocols) is missed and steered to the Linux Driver.

Kernel Port ↔ Kernel Port (Messaging): Internal Messaging traffic from the Linux Port is matched and steered to the Linux Representor Port, utilizing RSS for efficient kernel-space application processing.

The Bifurcated Driver Model Reference Application consists of core logic that manages the DOCA Flow pipes and entries to implement traffic bifurcation.

1. Initialization: The application initializes DPDK and DOCA Flow resources, including ports and queues.

2. DOCA Flow Switch Configuration: setting DOCA Flow in a switch mode.

3. Port Setup: Creating DOCA Flow switch ports corresponding to the physical and Linux representor ports.

4. Default miss behavior: Unmatched packets are forwarded to the kernel, ensuring control plane protocols receive proper kernel processing.

5. Pipe Creation: Creates two pipes to implement the bifurcation logic:

   1. Egress Pipe (EGRESS domain): Matches on packet metadata (pkt_meta) and forwards packets to the corresponding port. Contains three entries:

      • 

      • 

      • 

   2. Ingress Classifier Pipe (DEFAULT domain): Routes traffic based on source port_id and outer L4 protocol type, sets packet metadata, and forwards to either the egress pipe or RSS queues. Contains nine entries:

      • 

      • 

      • 

6. Monitoring Configuration: Each pipe entry is configured with a non-shared counter resource to track packet statistics. The ingress root classifier pipe also enables a miss counter to monitor packets forwarded to the kernel.

7. Entry Processing: All flow entries are processed and offloaded to hardware.

8. Packet Processing: The main application loop demonstrates Linux port packet handling by receiving packets from queue 0, setting appropriate metadata, and transmitting them back through the switch port.

9. Statistics Collection: Upon completion, the application queries and displays per-entry packet counters for both ingress and egress pipes, along with the ingress miss counter, providing visibility into traffic patterns.

This application leverages the following DOCA library:

• DOCA Flow – The core library used to program the hardware eSwitch to create the flow rules for traffic steering and bifurcation.

Refer to the respective programming guide for more information.

Please refer to the DOCA Installation Guide for Linux for details on how to install BlueField-related software.

The installation of DOCA's reference applications contains the sources of the applications, alongside the matching compilation instructions. This allows for compiling the applications "as-is" and provides the ability to modify the sources, then compile a new version of the application.

For more information about the applications as well as development and compilation tips, refer to the DOCA Reference Applications page.

The sources of the application can be found under the application's directory: /opt/mellanox/doca/applications/bifurcated_driver_model/.

Compiling All Applications

To build all DOCA applications together, run:

            

            
cd /opt/mellanox/doca/applications/
meson /tmp/build
ninja -C /tmp/build
        

doca_bifurcated_driver_model is created under /tmp/build/applications/bifurcated_driver_model/.

Recompiling Only the Current Application

To directly build only the Bifurcated Driver Model application:

            

            
cd /opt/mellanox/doca/applications/
meson /tmp/build -Denable_all_applications=false -Denable_bifurcated_driver_model=true
ninja -C /tmp/build
        

doca_bifurcated_driver_model is created under /tmp/build/applications/bifurcated_driver_model/.

The application is based on DOCA Flow and DPDK. Therefore, the user is required to allocate huge pages.

            

            
$ echo '1024' | sudo tee -a /sys/kernel/mm/hugepages/hugepages-2048kB/nr_hugepages
$ sudo mkdir /mnt/huge
$ sudo mount -t hugetlbfs -o pagesize=2M nodev /mnt/huge
        

Application Execution

• Application usage instructions:

  Info

  The application is provided in source form, requiring compilation before execution.

  Copy

  Copied!

              
  
              
  Usage: doca_bifurcated_driver_model [DPDK Flags] -- [DOCA Flags] [Program Flags]
  DOCA Flags:
    -h, --help                        Print a help synopsis
    -v, --version                     Print program version information
    -l, --log-level                   Set the (numeric) log level for the program <10=DISABLE, 20=CRITICAL, 30=ERROR, 40=WARNING, 50=INFO, 60=DEBUG, 70=TRACE>
    --sdk-log-level                   Set the SDK (numeric) log level for the program <10=DISABLE, 20=CRITICAL, 30=ERROR, 40=WARNING, 50=INFO, 60=DEBUG, 70=TRACE>
    -j, --json <path>                 Parse command line flags from an input json file
  Program Flags:
    -a, --device                      device
    -r, --rep                         device representor
    -exp, --expert-mode               set expert mode
          

• This usage printout can be printed to the command line using the -h (or --help) options:

  Copy

  Copied!

              
  
              
  ./doca_bifurcated_driver_model -- -h
          

  Info

  For additional information, refer to section "Command Line Flags".

• CLI example for running the application on BlueField:

  Copy

  Copied!

              
  
              
  cd /opt/mellanox/doca/applications/
  ./bifurcated_driver_model/doca_bifurcated_driver_model
   -- -l 60 -a pci/08:00.0 -r pci/08:00.0,pf0sf0 -r pci/08:00.1
          

  Info

  SFs must be enabled according to the BlueField Scalable Function User Guide.

• Before creating SFs on a specific physical port, it is important to verify the encap mode on the respective PF FDB. The default mode is basic. To check the encap mode, run:

  Copy

  Copied!

              
  
              
  cat /sys/class/net/p0/compat/devlink/encap
          

  In this case, disable encap on the PF FDB before creating the SFs by running:

  Copy

  Copied!

              
  
              
  /opt/mellanox/iproute2/sbin/devlink dev eswitch set pci/0000:03:00.0 mode legacy
  /opt/mellanox/iproute2/sbin/devlink dev eswitch set pci/0000:03:00.1 mode legacy
  echo none > /sys/class/net/p0/compat/devlink/encap
  echo none > /sys/class/net/p1/compat/devlink/encap
  /opt/mellanox/iproute2/sbin/devlink dev eswitch set pci/0000:03:00.0 mode switchdev
  /opt/mellanox/iproute2/sbin/devlink dev eswitch set pci/0000:03:00.1 mode switchdev
          

  Note

  If the encap mode is set to basic then the application fails upon initialization.

• The flag -a pci/08:00.0 with representors -r pci/08:00.0,pf0sf0 -r pci/08:00.1 is mandatory for proper usage of the application.

  • Modifying this flag will result in an unexpected behavior as only 2 ports are supported.

  • The SF number is arbitrary and configurable.

• The SF numbers must match the desired SF devices.

  • The device identifiers must match the desired network devices.

  • For additional information regarding the command line syntax of the device and device representor identifiers, refer to the matching section in the DOCA Arg Parser.

  • For more information, refer to section "Running DOCA Application on Host" in DOCA Virtual Functions User Guide.

Command Line Flags

Troubleshooting

Refer to the NVIDIA BlueField Platform Software Troubleshooting Guide for any issue encountered with the installation or execution of the DOCA applications.

The application initializes resources, sets up the flow pipes, and enters a processing loop.

1. Initialize logging:

   1. doca_log_backend_create_standard();

   2. doca_log_backend_create_with_file_sdk();

   3. doca_log_backend_set_sdk_level();

2. Argument Parsing Initialization:

   1. doca_argp_init(): Initialize arg parser.

   2. register_doca_flow_switch_params(): Register switch applications parameters.

3. Parse all command line arguments.

   1. doca_argp_start();

4. DOCA Flow devices initialization:

   1. init_doca_flow_devs();

5. DPDK & Port Initialization:

   1. dpdk_queues_and_ports_init();

6. DOCA Flow & Application Setup:

   1. bifurcated_driver_model_init():

      1. Initializes DOCA Flow:

         init_doca_flow();

      2. Creates DOCA ports for all physical and Linux representor ports:

         init_doca_flow_switch_ports();

      3. Set default miss behavior:

         doca_flow_port_update_default_miss();

      4. Creates all necessary pipes: Ingress, and Egress pipes, defining the flow structure:

         1. create_egress_pipe();

         2. add_egress_pipe_entries();

         3. create_ingress_classifier_pipe();

         4. add_ingress_classifier_entries();

            1. Data Plane (P0 ↔ P1): 4 entries for TCP/UDP forwarding

            2. Control Plane (Linux → RSS): 5 entries for various protocols

7. Process flow entries:

   1. doca_flow_entries_process();

8. Main processing loop

   1. handle_rx_tx_pkts()

9. Statistics collection:

   1. print_flow_statistics();

      1. doca_flow_resource_query_entry(): Query and display ingress entry counters.

      2. doca_flow_resource_query_pipe_miss(): Query and display ingress miss counter.

10. Cleanup and destroy:

    1. stop_doca_flow_ports()

    2. doca_flow_destroy()

    3. dpdk_queues_and_ports_fini()

    4. dpdk_fini()

    5. destroy_doca_flow_devs()

    6. doca_argp_destroy()