NVIDIA DOCA DMA Copy Application Guide

This guide provides an example of a DMA Copy implementation on top of NVIDIA® BlueField® DPU.

Introduction

DOCA DMA (direct memory access) Copy application transfers files (data path), up to the maximum supported size by the hardware, between the DPU and the x86 host using the DOCA DMA Library which provides an API to copy data between DOCA buffers using hardware acceleration, supporting both local and remote memory.

DOCA DMA allows complex memory copy operations to be easily executed in an optimized, hardware-accelerated manner.

System Design

DOCA DMA Copy is designed to run on the instances of the BlueField DPU and x86 host. The DPU application must be the first to spawn as it opens the DOCA Comch server between the two sides on which all the necessary DOCA DMA library configuration files (control path) are transferred.

Application Architecture

DOCA DMA Copy runs on top of DOCA DMA to read/write directly from the host's memory without any user/kernel space context switches, allowing for a fast memory copy.

Flow:

The two sides initiate a short negotiation in which the file size and location are determined.
The host side creates the export descriptor with doca_mmap_export_pci() and sends it with the local buffer address and length on the Comch to the DPU side application.
The DPU side application uses the received export descriptor to create a remote memory map locally with doca_mmap_create_from_export() and the host buffer information to create a remote DOCA buffer.
From this point on, the DPU side application has all the necessary memory information and the DMA copy can take place.

DOCA Libraries

This application leverages the following DOCA libraries:

Refer to their respective programming guide for more information.

Compiling the Application

Info

Please refer to the NVIDIA DOCA Installation Guide for Linux for details on how to install BlueField-related software.

The installation of DOCA's reference applications contains the sources of the applications, alongside the matching compilation instructions. This allows for compiling the applications "as-is" and provides the ability to modify the sources, then compile a new version of the application.

Tip

For more information about the applications as well as development and compilation tips, refer to the DOCA Applications page.

The sources of the application can be found under the application's directory: /opt/mellanox/doca/applications/dma_copy/.

Compiling All Applications

All DOCA applications are defined under a single meson project. So, by default, the compilation includes all of them.

To build all the applications together, run:

Copy
Copied!

            
            cd /opt/mellanox/doca/applications/
meson /tmp/build 
ninja -C /tmp/build

Info

doca_dma_copy is created under /tmp/build/dma_copy/.

Compiling Only the Current Application

To directly build only the DMA Copy application:

Copy
Copied!

            
            cd /opt/mellanox/doca/applications/
meson /tmp/build -Denable_all_applications=false -Denable_dma_copy=true
ninja -C /tmp/build

Info

doca_dma_copy is created under /tmp/build/dma_copy/.

Alternatively, one can set the desired flags in the meson_options.txt file instead of providing them in the compilation command line:

Edit the following flags in /opt/mellanox/doca/applications/meson_options.txt:
- Set enable_all_applications to false
- Set enable_dma_copy to true

Run the following compilation commands :

Copy
Copied!

            
            cd /opt/mellanox/doca/applications/
meson /tmp/build 
ninja -C /tmp/build

Info

doca_dma_copy is created under /tmp/build/dma_copy/.

Copy
Copied!

            
            Usage: doca_dma_copy [DOCA Flags] [Program Flags]
 
DOCA Flags:
  -h, --help                        Print a help synopsis
  -v, --version                     Print program version information
  -l, --log-level                   Set the (numeric) log level for the program <10=DISABLE, 20=CRITICAL, 30=ERROR, 40=WARNING, 50=INFO, 60=DEBUG, 70=TRACE>
  --sdk-log-level                   Set the SDK (numeric) log level for the program <10=DISABLE, 20=CRITICAL, 30=ERROR, 40=WARNING, 50=INFO, 60=DEBUG, 70=TRACE>
  -j, --json <path>                 Parse all command flags from an input json file
 
Program Flags:
  -f, --file                        Full path to file to be copied/created after a successful DMA copy
  -p, --pci-addr                    DOCA Comm Channel device PCI address
  -r, --rep-pci                     DOCA Comm Channel device representor PCI address (needed only on DPU)

Info

This usage printout can be printed to the command line using the -h (or --help) options:

Copy
Copied!

            
            ./doca_dma_copy -h

Info

For additional information, refer to section "Command Line Flags".

CLI example for running the application on the BlueField:
Copy

Copied!
```
            
            ./doca_dma_copy -p 03:00.0 -r 3b:00.0 -f received.txt
        
```
Note

Both the DOCA Comch device PCIe address (03:00.0) and the DOCA Comch device representor PCIe address (3b:00.0) should match the addresses of the desired PCIe devices.
CLI example for running the application on the host:
Copy

Copied!
```
            
            ./doca_dma_copy -p 3b:00.0 -f send.txt
        
```
Note

The DOCA Comch device PCIe address, 3b:00.0, should match the address of the desired PCIe device.
The application also supports a JSON-based deployment mode, in which all command-line arguments are provided through a JSON file:
Copy

Copied!
```
            
            ./doca_dma_copy --json [json_file]
        
```
For example:
Copy

Copied!
```
            
            ./doca_dma_copy --json ./dma_copy_params.json
        
```
Note

Before execution, ensure that the used JSON file contains the correct configuration parameters, and especially the PCIe addresses necessary for the deployment.

Command Line Flags

Flag Type	Short Flag	Long Flag/JSON Key	Description	JSON Content
General flags	`h`	`help`	Print a help synopsis	N/A
	`v`	`version`	Print program version information	N/A
	`l`	`log-level`	Set the log level for the application: DISABLE=10 CRITICAL=20 ERROR=30 WARNING=40 INFO=50 DEBUG=60 TRACE=70 (requires compilation with `TRACE` log level support)	Copy Copied! `"log-level": 60`
	N/A	`sdk-log-level`	Set the log level for the program: DISABLE=10 CRITICAL=20 ERROR=30 WARNING=40 INFO=50 DEBUG=60 TRACE=70	Copy Copied! `"sdk-log-level": 40`
	`j`	`json`	Parse all command flags from an input JSON file	N/A
Program flags	`f`	`file`	Full path to file to be copied/created after a successful copy Note This is a mandatory flag.	Copy Copied! `"file": "/tmp/sample.txt"`
	`p`	`pci-addr`	DOCA Comch device PCIe address. Note This is a mandatory flag.	Copy Copied! `"pci-addr": "b1:00.0"`
	`r`	`rep-pci`	DOCA Comch device representor PCIe address. Note This is a mandatory flag only on the DPU.	Copy Copied! `"rep-pci": "b1:02.0"`

Info

Refer to DOCA Arg Parser for more information regarding the supported flags and execution modes.

Troubleshooting

Refer to the NVIDIA DOCA Troubleshooting Guide for any issue encountered with the installation or execution of the DOCA applications.

Application Code Flow

Parse application argument.

Initialize arg parser resources and register DOCA general parameters.

Copy
Copied!

            
            doca_argp_init();

Register DMA Copy application parameters.

Copy
Copied!

            
            register_dma_copy_params();

Parse the arguments.

Copy
Copied!

            
            doca_argp_start();

Initialize Comch endpoint.
Copy

Copied!
```
            
            init_cc();
        
```
1. Create Comch endpoint.
2. Parse user PCIe address for Comch device.
3. Open Comch DOCA device.
4. Parse user PCIe address for Comch device representor (on DPU side).
5. Open Comch DOCA device representor (on DPU side).
6. Set Comch endpoint properties.
Open the DOCA hardware device from which the copy would be made.
Copy

Copied!
```
            
            open_dma_device();
        
```
1. Parse the PCIe address provided by the user.
2. Create a list of all available DOCA devices.
3. Find the appropriate DOCA device according to specific properties.
4. Open the device.

Create all required DOCA core objects.

Copy
Copied!

            
            create_core_objects();

Initiate DOCA core objects.

Copy
Copied!

            
            init_core_objects();

Start host/DPU DMA Copy.
1. Host side application:
  Copy
  
  Copied!
```
            
            host_start_dma_copy();
        
```
  1. Start negotiation with the DPU side application for the location and size of the file.
  2. Allocate memory for the DMA buffer.
  3. Export the memory map and send the output (export descriptor) to the DPU side application.
  4. Send the host local buffer memory address and length on the Comch to the DPU side application.
  5. Wait for the DPU to notify that DMA Copy ended.
  6. Close all memory objects.
  7. Clean resources.
2. DPU side application:
  Copy
  
  Copied!
```
            
            dpu_start_dma_copy();
        
```
  1. Start negotiation with the host side application for file location and size.
  2. Allocate memory for the DMA buffer.
  3. Receive the export descriptor on the Comch.
  4. Create the DOCA memory map for the remote buffer on the host.
  5. Receive the host buffer information on the Comch.
  6. Create two DOCA buffers, one for the remote (host) buffer and one for the local buffer.
  7. Submit the DMA copy task.
  8. Send a host message to notify that DMA copy ended.
  9. Clean resources.

Destroy Comch.

Copy
Copied!

            
            destroy_cc();

Destroy DOCA core objects.

Copy
Copied!

            
            destroy_core_objects();

Arg parser destroy.

Copy
Copied!

            
            doca_argp_destroy();

References

/opt/mellanox/doca/applications/dma_copy/
/opt/mellanox/doca/applications/dma_copy/dma_copy_params.json

On This Page

NVIDIA DOCA DMA Copy Application Guide

Introduction

System Design

Application Architecture

DOCA Libraries

Compiling the Application

Compiling All Applications

Compiling Only the Current Application

Troubleshooting

Running the Application

Application Execution

Command Line Flags

Troubleshooting

Application Code Flow

References