NVIDIA DOCA Storage Zero Copy

This document provides a data storage implementation on top of the NVIDIA® BlueField®.

The zero copy storage reference applications demonstrate a way to use the DOCA Comch, DOCA Core, and DOCA RDMA libraries to leverage hardware acceleration to implement a simple data storage solution that allows for data to be stored and retrieved efficiently without any unnecessary copying of data. In this application suite data is transferred directly from the initiator to the target using DOCA RDMA without the initiator being aware of the storage target.

The zero copy storage reference consists of three applications:

• doca_storage_zero_copy_comch_to_rdma (comch_to_rdma) – isolates the user from the storage target and relays messages between the initiator and the target

• doca_storage_zero_copy_initiator_comch (initiator_comch) – a trivial benchmark to demonstrate configuring and interacting with the storage by either reading or writing data

• doca_storage_zero_copy_target_rdma (target_rdma) - a mock storage application that simply uses a block of memory instead of a physical disk. Performs the RDMA read and write operations to fulfill the initiators requests.

A typical deployment of these applications is shown in the following diagram:

The important aspects to note here are:

• Initiator_comch and comch_to_rdma run on the same physical machine and then communicate with target_rdma via RDMA

• A TCP socket (not shown) is used to allow comch_to_rdma and target_rdma to communicate before and after the data path to handle setup and teardown

• DOCA RDMA is used to provide a high performance data path link between the comch_to_rdma and target_rdma applications

• DOCA Comch client-server (not shown) is used to perform early configuration and control exchanges between initiator_comch and comch_to_rdma, then DOCA Comch consumer-producer objects are used to provide a high performance data path link between the initiator_comch and comch_to_rdma applications

The execution of the applications is broken down into three main phases:

1. Initial configuration and setup.

2. Data path high-performance execution.

3. Teardown.

The following presents a high-level view of these phases and the key activities:

Data path execution involves forming one or more requests containing the following:

• The operation to perform

• The data to use

• Additional user metadata which is returned with the response

These requests are sent from initiator_comch to comch_to_rdma using a DOCA Comch producer. Comch_to_rdma relays this request verbatim to be processed by target_rdma. Target_rdma performs the actual transfer of data between the memory indicated by the request and the memory allocated as a mocked storage.

The following diagram presents the described data flow, where green sections are work performed by the applications and blue sections are work performed by the hardware in the background:

This application leverages the following DOCA libraries:

• DOCA Comch

• DOCA RDMA

Refer to their respective programming guide for more information.

The installation of DOCA's reference applications contains the sources of the applications, alongside the matching compilation instructions. This allows for compiling the applications "as-is" and provides the ability to modify the sources, then compile a new version of the application. For more information about the applications as well as development and compilation tips, refer to the DOCA Reference Applications page.

The sources of the application can be found under the application's directory: /opt/mellanox/doca/applications/storage/.

Compiling All Applications

All DOCA applications are defined under a single meson project. So, by default, the compilation includes all of them.

To build all the applications together, run:

            

            
cd /opt/mellanox/doca/applications/
meson /tmp/build
ninja -C /tmp/build
        

The storage zero copy applications are:

• doca_storage_zero_copy_initiator_comch

• doca_storage_zero_copy_comch_to_rdma

• doca_storage_zero_copy_target_rdma

The applications are built under /tmp/build/storage/.

Compiling Storage Zero Copy Applications Only

To directly build only the storage zero copy applications:

            

            
cd /opt/mellanox/doca/applications/
meson /tmp/build -Denable_all_applications=false -Denable_storage=true
ninja -C /tmp/build
        

Alternatively, the user may set the desired flags in the meson_options.txt file instead of providing them in the compilation command line:

1. Edit the following flags in /opt/mellanox/doca/applications/meson_options.txt:

   • Set enable_all_applications to false

   • Set enable_storage to true

2. Run the following compilation commands :

   Copy

   Copied!

               
   
               
   cd /opt/mellanox/doca/applications/
   meson /tmp/build
   ninja -C /tmp/build
           

   Info

   The storage zero copy applications are created under /tmp/build/storage/.

Troubleshooting

Refer to the NVIDIA DOCA Troubleshooting Guide for any issue encountered with the compilation of the application.

The applications must be initiated in the following order for them to execute correctly:

1. doca_storage_zero_copy_target_rdma

2. doca_storage_zero_copy_comch_to_rdma

3. doca_storage_zero_copy_initiator_comch

Refer to each application's page to learn about the set of parameters that should be provided to each one:

• target_rdma

• comch_to_rdma

• initiator_comch