NVIDIA DOCA NVMe-oF RDMA Target Reference Application Guide

Instructions: Please replace all labels/tags such as with the relevant content from your library/driver name, samples names, etc.

All the text that is marked in RED should be replaced with actual content for the specific guide.

This document provides a <application name> implementation on top of the NVIDIA® BlueField® DPU.

Introduction

A storage target is any application that can access data storage. This includes hard drives, SSDs, network-attached storage (NAS), and cloud storage services.

It serves storage access requests from local or remote initiators.

To allow the storage target to receive remote requests (over the network), the NVMe ( Non-Volatile Memory Express ) over fabrics (NVMeOF) protocol was introduced.

NVMe over Fabrics (NVMe-oF) is a protocol specification designed to extend the capabilities of NVMe storage across network fabrics (for example, RoCEv1/IB RDMA and NVMeTCP).

NVMe-oF extends the concept of storage targets by enabling NVMe commands to be executed over a network, rather than being limited to direct-attached storage.

Storage target application that supports NVMe-oF is a heavy application that consumes a substantial number of CPU cores.

To improve latency, free CPU resources and save power, an offload solution is required. Starting from BF3, the target application shall be offloaded on the DPA.

To abstract the communication between the target offload application (in our case SPDK) integration with the underlying offload accelerator and to ease the integration,

a new DOCA library is introduced (DOCA_STA). The library shall expose a public API that shall be used by the target application (DOCA application) for control-plane and data-plane handling.

While a software-based storage target accesses the NVMe drives directly over the PCIe, the DPA storage target offload accelerator accesses the NVMe drives

over PCIe-P2P (peer-to-peer) topology. To allow such access special kernel configuration is required.

System Design

The file compression application client runs on the host and the server runs on the DPU.

Application Architecture

The file compression application runs on top of the DOCA Comm Channel API to send and receive the file from the host and to the DPU.

app-arch-version-1-modificationdate-1732033374093-api-v2.png

Connection is established on both sides by DOCA Comm Channel API.
Client compresses the data:
- When compress engine is available – submits compress job with DOCA Compress API and sends the result to the server
- When compress engine is unavailable – compresses the data in software
Client sends the number of messages needed to send the compressed content of the file.
Client sends data segments in size of up to 4080 bytes.
Server saves the received data in a buffer and submits a decompress job.
Server sends an ACK message to the client when all parts of the file are received successfully.
Server compares the received checksum to the calculated checksum.
Server writes the decompressed data to an output file.

DOCA Libraries

This application leverages the following DOCA libraries:

For additional information about the used DOCA libraries, please refer to the respective programming guides.

Dependencies

NVIDIA BlueField-3 DPU is required.
Ubuntu 18.04/20.04/22.04 hosts (x86)
Open MPI version 4.1.5rc2 or greater (included in DOCA's installation)

Compiling the Application

Info

Please refer to the NVIDIA DOCA Installation Guide for Linux for details on how to install BlueField-related software.

The installation of DOCA's reference applications contains the sources of the applications, alongside the matching compilation instructions. This allows for compiling the applications "as-is" and provides the ability to modify the sources, then compile a new version of the application.

Tip

For more information about the applications as well as development and compilation tips, refer to the DOCA Reference Applications page.

The sources of the application can be found under the application's directory: /opt/mellanox/doca/applications/file_compression/.

Compiling All Applications

All DOCA applications are defined under a single meson project. So, by default, the compilation includes all of them.

To build all the applications together, run:

Copy
Copied!

            
            cd /opt/mellanox/doca/applications/
meson /tmp/build 
ninja -C /tmp/build

Info

doca_file_compression is created under /tmp/build/file_compression/.

Compiling Only the Current Application

To directly build only the file compression application: (REPLACE WITH FLAG FOR YOUR APP)

Copy
Copied!

            
            cd /opt/mellanox/doca/applications/
meson /tmp/build -Denable_all_applications=false -Denable_file_compression=true
ninja -C /tmp/build

Info

doca_file_compression is created under /tmp/build/file_compression/.

Alternatively, one can set the desired flags in the meson_options.txt file instead of providing them in the compilation command line:
1. Edit the following flags in /opt/mellanox/doca/applications/meson_options.txt:
  - Set enable_all_applications to false
  - Set enable_file_compression to true
2. The same compilation commands should be used, as were shown in the previous section:
  Copy
  
  Copied!
```
            
            cd /opt/mellanox/doca/applications/
meson /tmp/build 
ninja -C /tmp/build
        
```
  Info
  
  doca_file_compression is created under /tmp/build/file_compression/.

Troubleshooting

Please refer to the NVIDIA DOCA Troubleshooting for any issue you may encounter with the compilation of the DOCA applications.

Running the Application

Prerequisites

The IPsec security gateway application is based on DOCA Flow. Therefore, the user is required to allocate huge pages.

Copy
Copied!

            
            $ echo '1024' | sudo tee -a /sys/kernel/mm/hugepages/hugepages-2048kB/nr_hugepages
$ sudo mkdir /mnt/huge
$ sudo mount -t hugetlbfs -o pagesize=2M nodev /mnt/huge

Application Execution

The file compression application is provided in source form, hence a compilation is required before the application can be executed.

Application usage instructions (REPLACE WITH REAL HELP AS PRINTED FROM YOUR APP) :

Copy
Copied!

            
            Usage: doca_file_compression [DOCA Flags] [Program Flags]
 
DOCA Flags:
  -h, --help                        Print a help synopsis
  -v, --version                     Print program version information
  -l, --log-level                   Set the (numeric) log level for the program <10=DISABLE, 20=CRITICAL, 30=ERROR, 40=WARNING, 50=INFO, 60=DEBUG, 70=TRACE>
  --sdk-log-level                   Set the SDK (numeric) log level for the program <10=DISABLE, 20=CRITICAL, 30=ERROR, 40=WARNING, 50=INFO, 60=DEBUG, 70=TRACE>
  -j, --json <path>                 Parse all command flags from an input json file
 
Program Flags:
  -p, --pci-addr                    DOCA Comm Channel device PCI address
  -r, --rep-pci                     DOCA Comm Channel device representor PCI address
  -f, --file                        File to send by the client / File to write by the server
  -t, --timeout                     Application timeout for receiving file content messages, default is 5 sec

For additional information, please refer to the Command Line Flags section below.

Note

The above usage printout can be printed to the command line using the -h (or --help) options: (REPLACE WITH REAL PATH OF YOUR APP)

Copy
Copied!

            
            ./doca_file_compression -h

CLI example for running the application on BlueField: (REPLACE WITH REAL PATH & COMMAND OF YOUR APP)
Copy

Copied!
```
            
            ./doca_file_compression -p 03:00.0 -r 3b:00.0 -f received.txt
        
```
Note

Both the DOCA Comm Channel device PCI address (03:00.0) and the DOCA Comm Channel device representor PCI address (3b:00.0) should match the addresses of the desired PCI devices.
CLI example for running the application on the host: (REPLACE WITH REAL PATH & COMMAND OF YOUR APP)
Copy

Copied!
```
            
            ./doca_file_compression -p 3b:00.0 -f send.txt
        
```
Note

The DOCA Comm Channel device PCI address (3b:00.0) should match the address of the desired PCI device.
The application also supports a JSON-based deployment mode, in which all command-line arguments are provided through a JSON file:
Copy

Copied!
```
            
            ./doca_file_compression --json [json_file]
        
```
For example: (REPLACE WITH REAL PATH & COMMAND OF YOUR APP)
Copy

Copied!
```
            
            ./doca_file_compression --json ./file_compression_params.json
        
```
Note

Before execution, please ensure that the used JSON file contains the correct configuration parameters, and especially the desired PCI addresses needed for the deployment.

Command Line Flags

Flag Type	Short Flag	Long Flag/JSON Key	Description	JSON Content
General flags	`h`	`help`	Prints a help synopsis	N/A
	`v`	`version`	Prints program version information	N/A
	`l`	`log-level`	Set the log level for the application: DISABLE=10 CRITICAL=20 ERROR=30 WARNING=40 INFO=50 DEBUG=60 TRACE=70 (Requires compilation with Trace level support)	Copy Copied! `"log-level": 60`
	N/A	`sdk-log-level`	Sets the log level for the program: DISABLE=10 CRITICAL=20 ERROR=30 WARNING=40 INFO=50 DEBUG=60 TRACE=70	Copy Copied! `"sdk-log-level": 40`
	`j`	`json`	Parse all command flags from an input json file	N/A
Program flags	`f`	`file`	For client – path to the file to be sent For server – path to write the file into Note This is a mandatory flag.	Copy Copied! `"file": "/tmp/data.txt"`
	`p`	`pci-addr`	Comm Channel DOCA device PCIe address Note This is a mandatory flag.	Copy Copied! `"pci-addr": 03:00.1`
	`r`	`rep-pci`	Comm Channel DOCA device representor PCIe address Note This flag is mandatory only on the DPU.	Copy Copied! `"rep-pci": b1:00.1`

Refer to DOCA Arg Parser for more information regarding the supported flags and execution modes.

Troubleshooting

Please refer to the NVIDIA DOCA Troubleshooting for any issue you may encounter with the installation or execution of the DOCA applications.

Application Code Flow

Parse application argument.

Initialize arg parser resources and register DOCA general parameters.

Copy
Copied!

            
            doca_argp_init();

Register file compression application parameters.

Copy
Copied!

            
            register_file_compression_params();

Parse the arguments.

Copy
Copied!

            
            doca_argp_start();

Parse app parameters.

Set endpoint attributes.
Copy

Copied!
```
            
            set_endpoint_properties();
        
```
1. Set maximum message size of 4080 bytes.
2. Set maximum number of messages allowed.

Create comm channel endpoint.

Copy
Copied!

            
            doca_comm_channel_ep_create();

Create endpoint for client/server.

Run client/server main logic.

Copy
Copied!

            
            file_compression_client/server();

Clean up the file compression application.

Copy
Copied!

            
            file_compression_cleanup();

Free all application resources.

Arg parser destroy.

Copy
Copied!

            
            doca_argp_destroy()

References

/opt/mellanox/doca/applications/file_compression/
/opt/mellanox/doca/applications/file_compression/file_compression_params.json

On This Page