DOCA Erasure Coding

This guide provides instructions on how to use the DOCA Erasure Coding API .

The DOCA Erasure Coding ( known also as forward error correction or FEC) library provides an API to encode and decode data using hardware acceleration, supporting both host and NVIDIA® BlueField®-3 (and higher) DPU memory regions.

DOCA Erasure Coding recovers lost data fragments by creating generic redundancy fragments (backup). Each redundancy block that the library creates can help recover any block in the original data should a total loss of fragment occur. This increases data redundancy and reduces data overhead.

The library provides an API for executing erasure coding (EC) operations on DOCA buffers residing in either the DPU or host memory.

This document is intended for software developers wishing to accelerate their application's EC memory operations.

Glossary

Familiarize yourself with the following terms to better understand the information in this document:

DOCA Erasure Coding library follows the architecture of a DOCA Core Context, it is recommended read the following sections before:

• DOCA Core Execution Model

• DOCA Core Device

• DOCA Core Memory Subsystem

DOCA Erasure Coding-based applications can run either on the host machine or on the DPU target (NVIDIA® BlueField®-3 and above).

Erasure Coding can only be run with DPU configured in DPU mode as described in NVIDIA BlueField DPU Modes of Operation.

DOCA Erasure Coding is a DOCA Context as defined by DOCA Core. This library leverages the DOCA Core architecture to expose asynchronous tasks/events that are offloaded to hardware.

The following diagram presents a high-level view of the EC transmission flow:

1. M packets are sent from the source (8 in this case).

2. Before the source send them, the source encode the data by adding to it T redundancy packets (4 in this case).

3. The packets are transmitted to the destination in UDP protocol. Some packets are lost and N' packets are received (in this case 4 packets are lost and 8 are received).

4. The destination decodes the data using all the packets available (both original data in green and redundancy data in red) and gets back the M original data packets.

Flows

Regular EC flow consists of the following elements:

1. Creating redundancy blocks from data (EC create).

2. Updating redundancy blocks from updated data (EC update).

3. Recovering data blocks from redundancy blocks (EC recover).

The following sections examine an M:K (where M is the original data and K is redundancy) EC.

Create Redundancy Blocks

The user must perform the following:

1. Input M data blocks via doca_buf (filled with data, each block size B)

2. Output K empty blocks via doca_buf (each block size B)

3. Use DOCA Erasure Coding to create a coding matrix of M by K via doca_buf.

4. Use DOCA Erasure Coding Create task to get the K output redundancy blocks.

   Warning

   This step can be repeated in a stream use case, as the DPU would not be the recovery or update point.

Recover Block

The user must perform the following:

1. Input M-L original blocks via doca_buf (blocks that weren't impaired).

2. Input L≤K (any) redundancy blocks via doca_buf (redundancy blocks originating from create \ update tasks).

3. Input bitmask or array, indicating which blocks to recover.

4. Output L empty blocks via doca_buf (same size of data block).

5. Use DOCA Erasure Coding to create a recover coding matrix of M by L via doca_buf (unique per bitmask).

6. Use DOCA Erasure Coding Recover task to get the L output recovered data blocks.

Objects

Device and Device Representor

The DOCA Erasure Coding library requires a DOCA device to operate. The device is used to access memory and perform the encoding and decoding operations. See DOCA Core Device Discovery.

For same Bluefield card, it does not matter which device is used (PF/VF/SF), as all these devices utilize the same HW component. In case there are multiple cards, then it is possible to create an EC instance per card, providing each instance with a device from a different card.

For accessing memory that is not local (from Host to DPU and vice versa), the DPU side of the application will need to pick a device with appropriate representor. See DOCA Core Device Representor Discovery

The device must stay valid until the EC instance is destroyed.

Memory Buffers

Executing any DOCA EC task requires two DOCA buffers, a source buffer and a destination buffer.

Depending on the allocation pattern of the buffers, refer to the Inventory Types table.

Buffers must not be modified or read during the execution of any task.

To start using the library, first, you need to go through a configuration phase as described in DOCA Core Context Configuration Phase.

This section describes how to configure and start the context, to allow execution of tasks and retrieval of events.

Configurations

The context can be configured to match the application use case.

To find if a configuration is supported, or what the min/max value, please refer to Device Support.

Mandatory Configurations

These configurations are mandatory and must be set by the application before attempting to start the context:

• At least 1 task/event type needs to be configured. See configuration of Tasks.

• A device with appropriate support must be provided on creation.

Device Support

DOCA Erasure Coding needs a device to operate. For picking a device, see DOCA Core Device Discovery.

Erasure Coding can be used in BlueField-3 with some limitations (see architecture). Any device can be used PF/VF/SF.

As device capabilities may change in the future, it is recommended to choose your device using the following methods:

• doca_ec_cap_task_galois_mul_is_supported

• doca_ec_cap_task_create_is_supported

• doca_ec_cap_task_update_is_supported

• doca_ec_cap_task_recover_is_supported

Some devices can allow different capabilities as follows:

• The maximum buffer list length

• The maximum block size

Buffer Support

Tasks support buffers with the following features:

This section describes execution on CPU or DPU using the DOCA Core Progress Engine .

Matrix Generate

All tasks require a coding matrix.

Matrix type

DOCA EC provides 2 matrix types in the following subsections.

Cauchy

Encoding matrix is constructed so that

Where:

• 

• 

• 

• 

Vandermonde

Encoding matrix is constructed so that

Where:

• 

• 

Matrix Functionality

Create

An encoding matrix is necessary for executing the create task, to create redundancy blocks.

The matrices used for updates and recovery are based on an encoding matrix.

The following subsections describe the available options for creating matrices.

Generic

Generic creation, with the doca_ec_matrix_create() function, is used for simple setup using one of matrix types provided by the library.

Input

Custom

Custom creation, with the doca_ec_matrix_create_from_raw() function, is used if the desired type of matrix is not provided by the library.

Input

Update

This matrix is necessary for executing the update task, to update the redundancy blocks after a change in the data blocks.

The matrix is created using the doca_ec_matrix_create_update() function.

Input

Recover

This matrix is necessary for executing the recover task, to recover original data blocks.

The matrix is created using the doca_ec_matrix_create_recover() function.

Input

Tasks

Galois Mul Task

This task executes Galois multiplication between the original blocks and the coding matrix.

Configur1ation

Input

Common input as described in DOCA Core Task.

Output

Common output as described in DOCA Core Task .

Task Successful Completion

After the task completes successfully, the following happens:

• The destination buffer holds a sequence containing all multiplication outcome blocks (e.g., dst_block_1, dst_block_2 , etc.)

• The destination buffer data segment is extended to include the outcome blocks

Task Failed Completion

If the task fails midway:

• The context may enter stopping state if a fatal error occurs

• The source and destination doca_buf objects are not modified

• The destination buffer contents may be modified

Limitations

• The operation is not atomic

• Once the task has been submitted, the source and destination buffer should not be read from/written to

• Source and destination buffers must not overlap

• Other limitations are described in DOCA Core Task

Create Task

This task creates redundancy blocks for the given original data blocks using a given coding matrix.

Configuration

Input

Common input as described in DOCA Core Task.

Output

Common output as described in DOCA Core Task .

Task Successful Completion

After the task completes successfully, the following happens:

• The destination buffer holds a sequence containing all redundancy blocks (rdnc_block_1, rdnc_block_2, etc.)

• The destination buffer data segment is extended to include the redundancy blocks

Task Failed Completion

If the task fails midway:

• The context may enter stopping state if a fatal error occurs

• The source and destination doca_buf objects are not modified

• The destination buffer contents may be modified

Limitations

• The operation is not atomic

• Once the task is submitted, the source and destination buffers should not be read from/written to

• Source and destination buffers must not overlap

• Other limitations are described in DOCA Core Task

Update Task

This task executes updates the redundancy blocks for the given original data blocks, using an update coding matrix.

Configuration

Input

Common input as described in DOCA Core Task.

Output

Common output as described in DOCA Core Task.

Task Successful Completion

After the task completes successfully, the following happens:

• The destination buffer holds a sequence containing the updated redundancy blocks (rdnc_block_1, rdnc_block_2, etc.)

• The destination buffer data segment is extended to include the updated redundancy blocks

Task Failed Completion

If the task fails midway:

• The context may enter stopping state if a fatal error occurs

• The source and destination doca_buf objects is not modified

• The destination buffer contents may be modified

Limitations

• The operation is not atomic

• Once the task has been submitted, the source and destination buffers should not be read from/written to

• Source and destination buffers must not overlap

• Other limitations described in DOCA Core Task

Recover Task

This task executes recovers data blocks for, using given available original data blocks and redundancy blocks and a given coding matrix.

Configuration

Input

Common input as described in DOCA Core Task.

Output

Common output as described in DOCA Core Task.

Task Successful Completion

After the task is completed successfully t he data is transformed to destination.

Task Failed Completion

If the task fails midway:

• The context may enter stopping state if a fatal error occurs

• The source and destination doca_buf objects are not modified

• The destination buffer contents may be modified

Limitations

• The operation is not atomic

• Once the task is submitted, the source and destination buffers should not be read from/written to

• Source and destination must not overlap

• The amount of blocks that can be recovered are limited to the number of redundancy blocks created

• Other limitations are described in DOCA Core Task

This section provides DOCA Erasure Coding sample implementation on top of the BlueField-3 DPU (and higher).

Sample Prerequisites

N/A

Running the Sample

1. Refer to the following documents:

   • NVIDIA DOCA Installation Guide for Linux for details on how to install BlueField-related software.

   • NVIDIA DOCA Troubleshooting Guide for any issue you may encounter with the installation, compilation, or execution of DOCA samples.

2. To build a given sample:

   Copy

   Copied!

               
   
               
   cd /opt/mellanox/doca/samples/doca_erasure_coding/<sample_name>
   meson /tmp/build
   ninja -C /tmp/build
           

   Warning

   The binary doca_<sample_name> is created under /tmp/build/.

3. Sample (e.g., doca_erasure_coding_recover) usage:

   Copy

   Copied!

               
   
               
   Usage: doca_erasure_coding_recover [DOCA Flags] [Program Flags]
    
   DOCA Flags:
     -h, --help                        Print a help synopsis
     -v, --version                     Print program version information
     -l, --log-level                   Set the (numeric) log level for the program <10=DISABLE, 20=CRITICAL, 30=ERROR, 40=WARNING, 50=INFO, 60=DEBUG, 70=TRACE>
     --sdk-log-level                   Set the SDK (numeric) log level for the program <10=DISABLE, 20=CRITICAL, 30=ERROR, 40=WARNING, 50=INFO, 60=DEBUG, 70=TRACE>
     -j, --json <path>                 Parse all command flags from an input JSON file
    
   Program Flags:
     -p, --pci-addr                    DOCA device PCI device address - default: 03:00.0
     -i, --input                       Input file/folder to ec - default: self
     -o, --output                      Output file/folder to ec - default: /tmp
     -b, --both                        Do both (encode & decode) - default: false
     -x, --matrix                      Matrix - {cauchy, vandermonde} - default: cauchy
     -t, --data                        Data block count - default: 2
     -r, --rdnc                        Redundancy block count - default: 2
     -d, --delete_index                Indices of data blocks to delete; comma-separated (i.e., 0,3,4) - default: 0
           

   Warning

   Current BlueField-3 limitations:

   • Data block count range – 1-128

   • Redundancy block count – 1-32

   • Block size – 64B-128MB

4. For additional information per sample, use the -h option:

   Copy

   Copied!

               
   
               
   /tmp/build/doca_<sample_name> -h
           

Samples

Erasure Coding Recover

This sample illustrates how to use DOCA Erasure Coding (EC) library to encode and decode a file block (and entire file).

The sample logic includes 3 steps:

1. Encoding – create redundancy.

2. Deleting – simulating disaster.

3. Decoding – recovering data.

The encode logic includes:

1. Locating a DOCA device.

2. Initializing the required DOCA Core structures, such as the progress engine (PE), memory maps, and buffer inventory.

3. Reading source original data file and splitting it to a specified number of blocks, <data block count>, specified for the sample to the output directory.

4. Populating two DOCA memory maps with a memory range, one for the source data and one for the result.

5. Allocating buffers from DOCA buffer inventory for each memory range.

6. Creating an EC object.

7. Connecting the EC context to the PE.

8. Setting a state change callback function for the PE, with the following logic:

   • Printing a log with every state change

   • Indicating that the user may stop progress the PE once it is back in idle state

9. Setting the configuration to the EC create task, including setting callback functions as follows:

   • Successful completion callback:

     1. Writing the resulting redundancy blocks to the output directory (count is specified by <redundancy block count>).

     2. Freeing the task.

     3. Saving the result of the task and the callback. If there was an error in step a., the relevant error value is saved.

     4. Stopping the context.

   • Failed completion callback:

     1. Saving the result of the task and the callback.

     2. Freeing the task.

     3. Stopping the context.

10. Creating EC encoding matrix by the matrix type specified to the sample.

11. Allocating and s ubmitting an EC create task.

12. Progressing the PE until the context returns to idle state, either as a result of a successful run in which all tasks have been successfully completed, or as a result of a fatal error.

13. Destroying all EC and DOCA Core structures.

The delete logic includes:

1. Deleting the block files specified with <indices of data blocks to delete>.

The decode logic includes:

1. Locating a DOCA device.

2. Initializing the required DOCA Core structures, such as the PE, memory maps, and buffer inventory.

3. Reading the output directory (source remaining data) and determining the block size and which blocks are missing (needing recovery).

4. Populating two DOCA memory maps with a memory range, one for the source data and one for the result.

5. Allocating buffers from DOCA buffer inventory for each memory range.

6. Creating an EC object.

7. Connecting the EC context to the PE.

8. Setting a state change callback function for the PE, with the following logic:

   • Printing a log with every state change

   • Indicating that the user may stop progress the PE once it is back in idle state

9. Setting the configuration to the EC recover task, including setting callback functions as following:

   • Successful completion callback:

     1. Writing the resulting recovered blocks to the output directory.

     2. Writing the recovered file to the output path.

     3. Freeing the task.

     4. Saving the result of the task and the callback. If there was an error in step a., the relevant error value is saved.

     5. Stopping the context.

   • Failed completion callback:

     1. Saving the result of the task and the callback.

     2. Freeing the task.

     3. Stopping the context.

10. Creating EC encoding matrix by the matrix type specified to the sample.

11. Creating EC decoding matrix, with doca_ec_matrix_create_recover(), using the encoding matrix.

12. Allocating and s ubmitting an EC recover task.

13. Progressing the PE until the context returns to idle state, either as a result of a successful run in which all tasks have been successfully completed, or as a result of a fatal error.

14. Destroying all DOCA EC and DOCA Core structures.

References:

• /opt/mellanox/doca/samples/doca_erasure_coding/doca_erasure_coding_recover/erasure_coding_recover_sample.c

• /opt/mellanox/doca/samples/doca_erasure_coding/doca_erasure_coding_recover/erasure_coding_recover_main.c

• /opt/mellanox/doca/samples/doca_erasure_coding/doca_erasure_coding_recover/meson.build