NVIDIA DOCA DMA Copy Application Guide
This guide provides an example of a DMA Copy implementation on top of NVIDIA® BlueField® DPU.
DOCA DMA (direct memory access) Copy application transfers files (data path), up to the maximum supported size by the hardware, between the DPU and the x86 host using the DOCA DMA Library which provides an API to copy data between DOCA buffers using hardware acceleration, supporting both local and remote memory.
DOCA DMA allows complex memory copy operations to be easily executed in an optimized, hardware-accelerated manner.
DOCA DMA Copy is designed to run on the instances of the BlueField DPU and x86 host. The DPU application must be the first to spawn as it opens the DOCA Comm Channel server between the two sides on which all the necessary DOCA DMA library configuration files (control path) are transferred.
DOCA DMA Copy runs on top of DOCA DMA to read/write directly from the host's memory without any user/kernel space context switches, allowing for a fast memory copy.
Flow:
The two sides initiate a short negotiation in which the file size and location are determined.
The host side creates the export descriptor with doca_mmap_export_pci() and sends it with the local buffer address and length on the Comm Channel to the DPU side application.
The DPU side application uses the received export descriptor to create a remote memory map locally with doca_mmap_create_from_export() and the host buffer information to create a remote DOCA buffer.
From this point on, the DPU side application has all the necessary memory information and the DMA copy can take place.
This application leverages the following DOCA libraries:
Refer to their respective programming guide for more information.
Installation
Please refer to the NVIDIA DOCA Installation Guide for Linux for details on how to install BlueField-related software.
Application Execution
The DMA copy application is provided in both source and binary forms. The binary is located under /opt/mellanox/doca/applications/dma_copy/bin/doca_dma_copy.
Application usage instructions:
Usage: doca_dma_copy [DOCA Flags] [Program Flags] DOCA Flags: -h, --help Print a help synopsis -v, --version Print program version information -l, --log-level Set the (numeric) log level
for
the program <10
=DISABLE,20
=CRITICAL,30
=ERROR,40
=WARNING,50
=INFO,60
=DEBUG,70
=TRACE> --sdk-log-level Set the SDK (numeric) log levelfor
the program <10
=DISABLE,20
=CRITICAL,30
=ERROR,40
=WARNING,50
=INFO,60
=DEBUG,70
=TRACE> -j, --json <path> Parse all command flags from an input json file Program Flags: -f, --file Full path to file to be copied/created after a successful DMA copy -p, --pci-addr DOCA Comm Channel device PCI address -r, --rep-pci DOCA Comm Channel device representor PCI address (needed only on DPU)InfoThis usage printout can be printed to the command line using the -h (or --help) options:
/opt/mellanox/doca/applications/dma_copy/bin/doca_dma_copy -h
InfoFor additional information, refer to section "Command Line Flags".
CLI example for running the application on the BlueField:
/opt/mellanox/doca/applications/dma_copy/bin/doca_dma_copy -p
03
:00.0
-r 3b:00.0
-f received.txtNoteBoth the DOCA Comm Channel device PCIe address (03:00.0) and the DOCA Comm Channel device representor PCIe address (3b:00.0) should match the addresses of the desired PCIe devices.
CLI example for running the application on the host:
/opt/mellanox/doca/applications/dma_copy/bin/doca_dma_copy -p 3b:
00.0
-f send.txtNoteThe DOCA Comm Channel device PCIe address, 3b:00.0, should match the address of the desired PCIe device.
The application also supports a JSON-based deployment mode, in which all command-line arguments are provided through a JSON file:
doca_dma_copy --json [json_file]
For example:
cd /opt/mellanox/doca/applications/dma_copy/bin ./doca_dma_copy --json ./dma_copy_params.json
NoteBefore execution, ensure that the used JSON file contains the correct configuration parameters, and especially the PCIe addresses necessary for the deployment.
Command Line Flags
Flag Type |
Short Flag |
Long Flag/JSON Key |
Description |
JSON Content |
General flags |
h |
help |
Print a help synopsis |
N/A |
v |
version |
Print program version information |
N/A |
|
l |
log-level |
Set the log level for the application:
|
|
|
N/A |
sdk-log-level |
Set the log level for the program:
|
|
|
j |
json |
Parse all command flags from an input JSON file |
N/A |
|
Program flags |
f |
file |
Full path to file to be copied/created after a successful copy Note
This is a mandatory flag.
|
|
p |
pci-addr |
DOCA Comm Channel device PCIe address. Note
This is a mandatory flag.
|
|
|
r |
rep-pci |
DOCA Comm Channel device representor PCIe address. Note
This is a mandatory flag only on the DPU.
|
|
Refer to DOCA Arg Parser for more information regarding the supported flags and execution modes.
Troubleshooting
Refer to the NVIDIA DOCA Troubleshooting Guide for any issue encountered with the installation or execution of the DOCA applications.
In addition to providing the application in binary form, the installation also includes all of the application sources and compilation instructions so as to allow modifying the sources and recompiling the application. For more information about the applications, as well as development and compilation tips, refer to the DOCA Applications page.
The sources of the application can be found under the /opt/mellanox/doca/applications/dma_copy/src directory.
Recompiling All Applications
The applications are all defined under a single meson project, so the default compilation recompiles all the DOCA applications.
To build all the applications together, run:
cd /opt/mellanox/doca/applications/
meson /tmp/build
ninja -C /tmp/build
doca_dma_copy is created under /tmp/build/dma_copy/src/.
Recompiling DMA Copy Application Only
To directly build only the DMA Copy application:
cd /opt/mellanox/doca/applications/
meson /tmp/build -Denable_all_applications=false
-Denable_dma_copy=true
ninja -C /tmp/build
doca_dma_copy is created under /tmp/build/dma_copy/src/.
Alternatively, one can set the desired flags in the meson_options.txt file instead of providing them in the compilation command line:
Edit the following flags in /opt/mellanox/doca/applications/meson_options.txt:
Set enable_all_applications to false
Set enable_dma_copy to true
Run the following compilation commands :
cd /opt/mellanox/doca/applications/ meson /tmp/build ninja -C /tmp/build
Infodoca_dma_copy is created under /tmp/build/dma_copy/src/.
Troubleshooting
Refer to the NVIDIA DOCA Troubleshooting Guide for any issue encountered with the compilation of the application .
Parse application argument.
Initialize arg parser resources and register DOCA general parameters.
doca_argp_init();
Register DMA Copy application parameters.
register_dma_copy_params();
Parse the arguments.
doca_argp_start();
Initialize Comm Channel endpoint.
init_cc();
Create Comm Channel endpoint.
Parse user PCIe address for Comm Channel device.
Open Comm Channel DOCA device.
Parse user PCIe address for Comm Channel device representor (on DPU side).
Open Comm Channel DOCA device representor (on DPU side).
Set Comm Channel endpoint properties.
Open the DOCA hardware device from which the copy would be made.
open_dma_device();
Parse the PCIe address provided by the user.
Create a list of all available DOCA devices.
Find the appropriate DOCA device according to specific properties.
Open the device.
Create all required DOCA core objects.
create_core_objects();
Initiate DOCA core objects.
init_core_objects();
Start host/DPU DMA Copy.
Host side application:
host_start_dma_copy();
Start negotiation with the DPU side application for the location and size of the file.
Allocate memory for the DMA buffer.
Export the memory map and send the output (export descriptor) to the DPU side application.
Send the host local buffer memory address and length on the Comm Channel to the DPU side application.
Wait for the DPU to notify that DMA Copy ended.
Close all memory objects.
Clean resources.
DPU side application:
dpu_start_dma_copy();
Start negotiation with the host side application for file location and size.
Allocate memory for the DMA buffer.
Receive the export descriptor on the Comm Channel.
Create the DOCA memory map for the remote buffer on the host.
Receive the host buffer information on the Comm Channel.
Create two DOCA buffers, one for the remote (host) buffer and one for the local buffer.
Submit the DMA copy task.
Send a host message to notify that DMA copy ended.
Clean resources.
Destroy Comm Channel.
destroy_cc();
Destroy DOCA core objects.
destroy_core_objects();
Arg parser destroy.
doca_argp_destroy();
/opt/mellanox/doca/applications/dma_copy/src
/opt/mellanox/doca/applications/dma_copy/bin/dma_copy_params.json