DOCA DPACC Compiler

Contents:

This document describes DOCA DPACC compiler and provides instructions about DPA toolchain setup and usage.

DPACC is a high-level compiler for the DPA processor which compiles code targeted for the data-path accelerator (DPA) processor into a device executable and generates a DPA program.

The DPA program is a host library with interfaces encapsulating the device executable. This DPA program is linked with the host application to generate a host executable. The host executable can invoke the DPA code through FlexIO runtime API.

DPACC uses DPA compiler (dpa-clang) to compile code targeted for DPA. dpa-clang is part of the DPA toolchain package which is an LLVM-based, cross-compiling, bare-metal toolchain. It provides Clang compiler, LLD linker targeting DPA architecture, and other utilities.

Glossary

Offloading Work on DPA

To invoke a DPA function from a host, the following things are required:

• DPA device code – C programs, targeted to run on the DPA. The DPA device code may contain one or more entry functions.

• Host application code – the corresponding host application. Refer to DPA Subsystem for more details

• Runtime – FlexIO or DOCA DPA library provides the runtime

The generated DPA program, when linked with a host application results in a host executable which also contains the device executable. The host application oversees loading the device executable on the device.

DPACC Predefined Macros

DPACC predefines the following macros:

Writing DPA Applications

DPA device code is a C code with some restrictions and special definitions.

FlexIO or DOCA-DPA APIs provide interfaces to DPA.

Language Support

The DPA is programmed using a subset of the C11 language standard. The compiler documents any constructs that are not available. Language constructs, where available, retain their standard definitions.

Restrictions on DPA Code

• Use of C thread local storage is not allowed for any variables

• Identifiers with _dpacc/__dpacc prefix are reserved by the compiler. Use of such identifiers may result in an error or undefined behavior

• DPA processor does not have native floating-point support; floating-point operations are disabled

DPA RPC Functions

A remote procedure call function is a synchronous call that triggers work in DPA and waits for its completion. These functions return a type uint64_t value. They are annotated with a __dpa_rpc__ attribute.

DPA Global Functions

A DPA global function is an event handler device function referenced from the host code. These functions do not return anything. They are annotated with a __dpa_global__ attribute.

For more information, refer to DPA Subsystem.

Characteristics of Annotated Functions

• Global functions must have void return type and RPC functions must have uint64_t return type

• Annotated functions cannot accept C pointers and arrays as arguments (for example, void my_global (int *ptr, int arr[]))

• Annotated functions cannot accept a variable number of arguments

• Inline specifier is not allowed on annotated functions

Handling User-defined Data Types

User-defined data types, when used as global function arguments, require special handling. They must be annotated with a __dpa_global__ attribute.

If the user-defined data type is typedef'd, the typedef statement must be annotated with a __dpa_global__ attribute along the data type itself.

Characteristics of Annotated Types

• They must have a copy of the definition in all translation units where they are used as global function arguments

• They cannot have pointers, variable length arrays, and flexible arrays as members

• Fixed-size arrays as C structure members are supported

• These characteristics apply recursively to any user-defined/typedef'd types that are members of an annotated type

DPACC processes all annotated functions along with annotated types and generates host and device interfaces to facilitate the function launch.

DPA Intrinsics

DPA features such as fences and processor-specific instructions are exposed via intrinsics by the DPA compiler. All intrinsics defined in the header file dpaintrin.h are guarded by the DPA_INTRIN_VERSION_USED macro. The current DPA_INTRIN_VERSION is 1.3.

Example:

            

            
#define DPA_INTRIN_VERSION_USED (DPA_INTRIN_VERSION(1, 3))
#include <dpaintrin.h>
…
__dpa_thread_writeback_window();   // Fence for write barrier
        

For more information, refer to DPA Subsystem.

DPACC Inputs and Outputs

DPACC can produce DPA programs in a single command by accepting all source files as an input. DPACC also offers the flexibility of producing DPA object files or libraries from input files.

DPA object files contain both host-stub objects (DPACC-generated interfaces) and device objects. These DPA object files can later be given to DPACC as input to produce the DPA library.

DPACC can accept the following file types as input:

Based on the mode of operations, DPACC can generate the following output files:

The following provides the commands to generate different kinds of supported output file types for each input file type:

dpacc -hostcc=<cc> -mcpu=<targets> in.c -o libprog.a

dpacc -hostcc=<cc> -mcpu=<targets> in.c -c

dpacc -hostcc=<cc> -mcpu=<targets> in.c -o lib<name> -gen-libs

dpacc -hostcc=<cc> -mcpu=<targets> in.dpa.o -o libprog.a

dpacc -hostcc=<cc> -mcpu=<targets> in.dpa.o -o lib<name> -gen-libs

dpacc -hostcc=<cc> -mcpu=<targets> in.a -o libprog.a

dpacc -hostcc=<cc> -mcpu=<targets> in.a -o lib<name> -gen-libs

DPA Program

DPACC produces a DPA program in compile-and-link mode. A DPA program is a host library which contains:

• DPACC-generated host stubs which registering the DPA application and facilitate invoking an entry-point function from the host application

• Device executable, generated by DPACC by compiling and linking input DPA device code

DPA program library must be linked with the host application that contains appropriate runtime APIs to load the device executable onto the device.

A DPA program can contain device executables for multiple targets. All the target-specific device executables are encapsulated in a special fatbinary container format and this container is embedded as a section into the host object which is present in the host library.

DPA Object

DPACC produces DPA object files in compile-only mode. A DPA object is a host object file which has a similar layout to the DPA program where it contains DPACC-generated host stubs and device object which is generated by compiling input device code.

A DPA object can contain device objects for multiple targets, similar to a DPA program. All the target-specific device objects are encapsulated in a special fatbinary container format and this container is embedded as a section into the host object.

DPA Library

A DPA library is a collection of two separate libraries:

• DPA host library: contains host interface objects corresponding to the device objects in a DPA device library

• DPA device library: contains device objects generated by compiling the input device code files

The DPA device library is consumed by DPACC during DPA-program generation and the DPA host library can optionally be linked with other host code and be distributed as the host library. Both libraries are generated as static archives.

The host library is a static archive. The device library is a file of specialized fatbinary container format type which contains archives of device objects built for different targets.

The 'Device Archive' box in the preceding figure indicates the fatbinary container. The fatbinary container should be treated as an opaque object by the end-user and should not be manipulated in any way.

DPACC Trajectory

The following diagram illustrates DPACC compile-and-link mode trajectory.

Modes of Operation

In each of the below modes, DPACC accepts single or multiple target names through the mcpu option and builds the output such that all the mentioned targets are supported.

Compile-and-link Mode

This is a one-step mode that accepts C source files or DPA object files and produces the DPA program. Specifying the output library name is mandatory in this mode.

Example commands:

            

            
$ dpacc in1.c in2.c -o myLib1.a -hostcc=gcc -mcpu=nv-dpa-bf3                               # Takes C sources to produce myLib1.a library which supports a single target - nv-dpa-bf3
$ dpacc in3.dpa.o in4.dpa.o -o myLib2.a -hostcc=gcc -mcpu=nv-dpa-bf3,nv-dpa-cx8            # Takes DPA object files to produce myLib2.a library which supports multiple targets - nv-dpa-bf3 and nv-dpa-cx8
$ dpacc in1.c in3.dpa.o -o myLib3.a -hostcc=gcc -mcpu=nv-dpa-bf3,nv-dpa-cx7,nv-dpa-cx8     # Takes C source and DPA object to produce myLib3.a library which supports multiple targets - nv-dpa-bf3, nv-dpa-cx7 and nv-dpa-cx8
        

Compile-only Mode

This mode accepts C source code and produces .dpa.o object files. These files can be given to DPACC to produce the DPA program. The mode is invoked by the --compile or -c option.

To display the object file name, use the --output-file or -o option.

Example commands:

            

            
$ dpacc -c input1.c -hostcc=gcc -mcpu=nv-dpa-cx7                                     # Produces input1.dpa.o which supports a single target - nv-dpa-cx7
$ dpacc -c input2.c -o myObj.dpa.o -hostcc=gcc -mcpu=nv-dpa-cx8,nv-dpa-cx7           # Produces myObj.dpa.o which supports multiple targets - nv-dpa-cx7 and nv-dpa-cx8
$ dpacc -c input3.c input4.c -hostcc=gcc -mcpu=nv-dpa-bf3,nv-dpa-cx7,nv-dpa-cx8      # Produces input3.dpa.o and input4.dpa.o which support multiple targets - nv-dpa-bf3, nv-dpa-cx7 and nv-dpa-cx8
        

Library Generation Mode

This mode accepts C source files or DPA object files and produces the DPA program. Specifying the output DPA library name is mandatory in this mode.

Example commands:

            

            
$ dpacc in1.c in2.c -o libdummy1 -hostcc=gcc -mcpu=nv-dpa-cx8 -gen-libs                                    # Takes C sources to produce a DPA-Library (libdummy1_host.a and libdummy_device.a archives) which supports a single target - nv-dpa-cx8
$ dpacc in3.dpa.o in4.dpa.o -o libdummy2 -hostcc=gcc -mcpu=nv-dpa-cx8,nv-dpa-bf3 -gen-libs                 # Takes DPA object files to produce a DPA-Library (libdummy2_host.a and libdummy2_device.a archives) which supports multiple targets - nv-dpa-bf3 and nv-dpa-cx8
$ dpacc in1.c in3.dpa.o -o outdir/libdummy3 -hostcc=gcc -mcpu=nv-dpa-bf3,nv-dpa-cx7,nv-dpa-cx8 -gen-libs   # Takes C source and DPA object to produce a DPA-Library (outdir/libdummy3_host.a and outdir/libdummy3_device.a archives) which supports multiple targets - nv-dpa-bf3, nv-dpa-cx7 and nv-dpa-cx8
        

To execute DOCA DPACC compiler:

            

            
Usage: dpacc <list-of-input-files> -hostcc=<path> -mcpu=<targets> [other options]
Helper Flags:
  -h, --help                          Print help information about DPACC
  -V, --version                       Print DPACC version information
  -v, --verbose                       List the compilation commands generated by this invocation while also executing every command in verbose mode
  -dryrun, --dryrun                   Only list the compilation commands generated by DPACC, without executing them
  -keep, --keep                       Keep all intermediate files that are generated during internal compilation steps in the current directory
  -keep-dir, --keep-dir               Keep all intermediate files that are generated during internal compilation steps in the given directory
  -optf, --options-file <file>,...    Include command line options from the specified file
        

Mandatory Arguments

Commonly Used Arguments

DPA Hardware Architectures

The following table indicates the DPA architectures, the associated values supported in the compiler through the -mcpu option, and the macros defined by the compiler to identify these architectures.

Because ConnectX-7 and Bluefield-3 share the same DPA hardware, nv-dpa-cx7 is treated as an alias of nv-dpa-bf3 by the compiler.

Link Compatibility

Only relocatable objects which are link-compatible are allowed to be linked together. Incompatible objects are errored out during linking. The toolchain version of the linker should be same as the toolchain version of the compiler which produced the objects.

If two architectures A and B are link-compatible and B is newer than A, objects built for target A can be linked to build an app for target B. However, the inverse—that is, linking objects built for target B to build an app for target A—is not valid.

Bluefield-3/ConnectX-7 and ConnectX-8 are link-compatible: objects built for Bluefield-3/ConnectX-7 can be linked together to build an application for ConnectX-8.

Architecture Macros

The compiler defines identifier macros for each version of DPA hardware, as described in DPA Hardware Architectures. Each identifier macro will have a unique integer value which is strictly greater than that of macros for older DPA CPU models. Known aliases such as Bluefield-3 DPA and ConnectX-7 DPA share the same integer value. The macro __NV_DPA is defined to the value of current compilation target. This can be used to write device code specific to a DPA hardware generation as shown below:

            

            
#if __NV_DPA == __NV_DPA_BF3
// Code for Bluefield-3 here
#elif __NV_DPA > __NV_DPA_BF3
// Code for devices after Bluefield-3 here
#endif
        

Note: The ordering established by value of hardware version identifier macros do not imply an ordering of features supported by hardware. It is the user's responsibility to ensure that features used in the code specific for a DPA version are actually supported on the hardware.

LTO Usage Guidelines

Restrictions

• Only the default linker script is supported with LTO

• Using options -fPIC/-fpic/-shared/-mcmodel=large through -devicecc-options is not supported when LTO is enabled

• Fat bitcode objects containing both LLVM bitcode and ELF representation are not supported

• Thin LTO is not supported

Compatibility

During compilation, LLVM generates the object as bitcode intermediate representation (IR) when LTO is enabled instead of ELF representation. The bitcode IR generated by the DPA compiler is only guaranteed to be compatible within the same version. The toolchain version of the compiler which builds the objects involved in link-time optimization (enabled with -flto) and the toolchain version of the linker which performs LTO must be the same.

Deprecated Features

• The -ldpa option which links with DOCA-DPA libraries is deprecated and will be removed in future releases. Use the -ldoca_dpa option instead.

Examples

This section provides some common use cases of DPACC and showcases the dpacc command.

Building Libraries

This example shows how to build DPA libraries using DPACC. Libraries for DPA typically contain two archives, one for the host and one for the device.

            

            
dpacc input.c -hostcc=gcc -mcpu=nv-dpa-bf3 -o lib<name> -gen-libs -hostcc-options="-fPIC"
        

This command generates the output files lib<name>_host.a and lib<name>_device.a.

The host stub archive can be linked with other host code to generate a shared/static host library.

• Generating a static host library:

  Copy

  Copied!

              
  
              
  ar x lib<name>_host.a                   # Extract objects to generate *.o
  ar cr lib<name>.a <*src.host.o> *.o     # Generate final static archive with all objects
          

• Generating a shared host library:

  Copy

  Copied!

              
  
              
  gcc -shared -o lib<name>.so <*src.host.o> -Wl,-whole-archive -l<name>_host -Wl,-no-whole-archive       # Link the generated archive to build a shared library
          

Linking with DPA Device Library

The DPA device library generated by DPACC using -gen-libs as part of a DPA library can be consumed by DPACC using the -device-libs option.

            

            
dpacc input.c -hostcc=gcc -mcpu=nv-dpa-bf3 -o libInput.a -device-libs="-L <path-to-library> -l<libName>"
        

Enabling Link-time Optimizations

Link-time optimizations can be enabled using -flto along with an optimization level specified for device compilation.

            

            
dpacc input1.c -hostcc=gcc -mcpu=nv-dpa-bf3 -c -flto -devicecc-options="-O2" 
dpacc input2.c -hostcc=gcc -mcpu=nv-dpa-bf3 -c -flto -devicecc-options="-O2"   
dpacc -mcpu=nv-dpa-bf3 input1.dpa.o input2.dpa.o -hostcc=gcc -o libInput.a
        

Including Headers

This example includes headers for device compilation using devicecc-options and host compilation using hostcc-options. You may also specify headers for any compilation on both the host and device side using the -I option.

            

            
dpacc input.c -hostcc=gcc -mcpu=nv-dpa-bf3 -o libInput.a -I <common-headers-path> -devicecc-options="-I <device-headers-path>" -hostcc-options="-I <host-headers-path>"
        

Dump Targets Supported by a Fatbinary File

The targets supported by a fatbinary file can be dumped using dpa-fatbin. This tool can also dump the targets supported by a DPA library from the device archive.

            

            
dpa-fatbin --list libfoo_device.a
dpa-fatbin --list device_exec.fatbin
        

Dump Target of Device ELF file

The target for which a device ELF file is built can be dumped using dpa-objdump.

            

            
dpa-objdump --file-headers foo.o
        

Generating Output as Source Code

DPACC provides the option -src-output to generate output as host-source code. This source can be compiled by the host compiler to generate functionally equivalent output which DPACC would have generated directly.

This example shows how to build various outputs of DPACC as source using this option and how to compile the generated source.

DPA Program Source

Generate DPA program source to DPACC:

            

            
dpacc input.c -hostcc=gcc -mcpu=nv-dpa-bf3 -o libfoo.c -src-output
        

Compile the generated source using the host compiler to generate an object and build an archive with this object. The macro __DPACC_SRC_TARGET__ must be defined when building this object to remove unnecessary code when building from source.

            

            
$ gcc libfoo.c -c -I /opt/mellanox/flexio/include -Wno-attributes -Wno-pedantic -Wno-unused-parameter -Wno-return-type -Wno-implicit-function-declaration -D__DPACC_SRC_TARGET__
$ ar cr libfoo.a libfoo.o
        

DPA Library Source

Generate DPA library source passing this option to DPACC

            

            
dpacc input.c -hostcc=gcc -mcpu=nv-dpa-bf3 -o libfoo -gen-libs -src-output 
        

This generates the device archive libfoo_device.a and host code files libfoo.lib.c, input.dpa.c. The host archive of DPA library is generated by compiling these sources and building an archive. The __DPACC_SRC_TARGET__ macro needs to be defined in this case to remove unnecessary code.

            

            
$ gcc libfoo.lib.c input.dpa.c -c -I /opt/mellanox/flexio/include -Wno-attributes -Wno-pedantic -Wno-unused-parameter -Wno-return-type -Wno-implicit-function-declaration -D__DPACC_SRC_TARGET__
$ ar cr libfoo_host.a libfoo.lib.o input.dpa.o
        

DPA Object Source

Generate DPA object source passing this option to DPACC:

            

            
dpacc input.c -hostcc=gcc -mcpu=nv-dpa-bf3 -c -src-output 
        

This generates a single file, input.dpa.c. Compile the host file to generate an object.

            

            
gcc input.dpa.c -c -I /opt/mellanox/flexio/include -Wno-attributes -Wno-pedantic -Wno-unused-parameter -Wno-return-type -Wno-implicit-function-declaration
        

DPA Compiler Usage

DPA-Compiler is an LLVM based compiler which is used by DPACC internally for compiling and linking the device code files. User specified options can be passed to the compiler and linker through the DPACC options --devicecc-optionsand --devicelink-options respectively.

Refer to the following resources for additional options:

• Clang compiler user manual

• Clang command line argument reference

• Target-dependent compilation options

• LLD command line reference

dpacc-extract Command Line Options

dpacc-extract extracts a device executable out of a DPA program or a host executable containing DPA program(s).

To execute dpacc-extract:

            

            
Usage: dpacc-extract <input-file> -o=<output-file> [other options]
Helper Flags:
 
  -o, --output-file                   Specify name of the output file
  -app-name, --app-name <name>        Specify name of the DPA application to extract
  -mcpu, --mcpu <target>              Specify name of the device for which the application is to be extracted
  -info, --info                       List apps and supported targets
  -h, --help                          Print help information about dpacc-extract
  -V, --version                       Print dpacc-extract version information
  -optf, --options-file <file>,...    Include command line options from the specified file
        

Mandatory arguments:

Objdump Command Line Options

The dpa-objdumputility prints the contents of object files and final linked images named on the command line.

For more information, refer to the Objdump command line reference.

Archiver Command Line Options

dpa-ar is a Unix ar-compatible archiver.

For more information, refer to the Archiver command line reference.

NM Tool Command Line Options

The dpa-nm utility lists the names of symbols from object files and archives.

For more information, refer to the NM tool command line reference.

Miscellaneous Notes

• Objects produced by LLD are not compatible with those generated by any other linker.

• Ensure that there is at least one reference to the device entry point function in the DPA program from the host application so that during linking, the DPA program is not silently discarded by the host linker because it is not referenced.

Changes in DPACC 1.11.0

What's New

• Added info mode in dpacc-extract which dumps information about DPA programs and executables

• Added new macro, extract_fields , which can be used to copy chunks of bytes from source buffer to destination buffer

• Added support for __atomic builtins using amocas on ConnectX-8

• Fixed an issue where an unknown DPA-ID warning is generated when building the DPA library from DPA objects produced by DPACC 1.9.0 or earlier

• Changed versioning format from x.y.z-b to x.y.z.b

Limitations

• Extract fields macro:

  • Destination buffer size and offsets used as arguments must be a multiple of 8

  • Source and destination buffers must be 8-byte aligned