NVIDIA DOCA DPACC Compiler

This document describes DOCA DPACC compiler and instructions about DPA toolchain setup and usage.

DPACC is a high-level compiler for the DPA processor which compiles code targeted for the data-path accelerator (DPA) processor into a device executable and generates a DPA program.

The DPA program is a host library with interfaces encapsulating the device executable. This DPA program is linked with the host application to generate a host executable. The host executable can invoke the DPA code through FlexIO runtime API.

DPACC uses DPA compiler (dpa-clang) to compile code targeted for DPA. dpa-clang is part of the DPA toolchain package which is an LLVM-based cross-compiling bare-metal toolchain. It provides Clang compiler, LLD linker targeting DPA architecture, and other utilities.

Glossary

Term

Definition

Device

DPA as present on the BlueField DPU

Host

CPU that launches the device code to run on the DPA

Device function

Any C function that runs on the DPA device

DPA global function

Device function that is the point of entry when offloading any work on DPA

Host compiler

Compiler used to compile the code targeting the host CPU

Device compiler

Compiler used to compile code targeting the DPA

DPA program

Host library that encapsulates the DPA device executable (.elf) and host stubs which are used to access the device executable


Offloading Work on DPA

dpacc-offloading-version-1-modificationdate-1702941557653-api-v2.png

To invoke a DPA function from host, the following things are required:

  • DPA device code – C programs, targeted to run on the DPA. DPA device code may contain one or more entry functions.

  • Host application code – the corresponding host application. Please refer to DPA Subsystem for more details

  • Runtime – FlexIO or DOCA DPA library provides the runtime

The generated DPA program, when linked with A host application, results in a host executable which also contains the device executable. The host application oversees loading the device executable on the device.

DPACC Predefined Macros

DPACC predefines the following macros:

Macro

Description

__DPA__

Defined when compiling device code file

__DPA_MAJOR__

Defined to the major version number of DPACC

__DPA_MINOR__

Defined to the minor version number of DPACC

__DPA_PATCH__

Defined to the patch version number of DPACC


Writing DPA Applications

DPA device code is a C code with some restrictions and special definitions.

FlexIO or DOCA-DPA APIs provide interfaces to DPA.

Language Support

The DPA is programmed using a subset of the C11 language standard. The compiler documents any constructs that are not available. Language constructs, where available, retain their standard definitions.

Restrictions on DPA Code

  • Use of C thread local storage is not allowed for any variables

  • Identifiers with _dpacc prefix are reserved by the compiler. Use of such identifiers may result in an error or undefined behavior

  • DPA processor does not have native floating-point support; use of floating point operations is disabled

DPA RPC Functions

A remote procedure call function is a synchronous call that triggers work in DPA and waits for its completion. These functions return a type uint64_t value. They are annotated with a __dpa_rpc__ attribute.

DPA Global Functions

A DPA global function is an event handler device function referenced from the host code. These functions do not return anything. They are annotated with a __dpa_global__ attribute.

For more information, refer to the DPA Subsystem.

Characteristics of Annotated Functions

  • Global functions must have void return type and RPC functions must have uint64_t return type

  • Annotated functions cannot accept C pointers and arrays as arguments (e.g., void my_global (int *ptr, int arr[]))

  • Annotated functions cannot accept a variable number of arguments

  • Inline specifier is not allowed on annotated functions

Handling User-defined Data Types

User-defined data types, when used as global function arguments, require special handling. They must be annotated with a __dpa_global__ attribute.

If the user-defined data type is typedef'd, the typedef statement must be annotated with a __dpa_global__ attribute along the data type itself.

Characteristics of Annotated Types

  • They must have a copy of the definition in all translation units where they are used as global function arguments

  • They cannot have pointers, variable length arrays, and flexible arrays as members

  • Fixed-size arrays as C structure members are supported

  • These characteristics apply recursively to any user-defined/typedef'd types that are members of an annotated type

DPACC processes all annotated functions along with annotated types and generates host and device interfaces to facilitate the function launch.

DPA Intrinsics

DPA features such as fences and processor-specific instructions are exposed via intrinsics by the DPA compiler. All intrinsics defined in the header file dpaintrin.h are guarded by the DPA_INTRIN_VERSION_USED macro. The current DPA_INTRIN_VERSION is 1.3.

Example:

Copy
Copied!
            

#define DPA_INTRIN_VERSION_USED (DPA_INTRIN_VERSION(1, 3)) #include <dpaintrin.h> … __dpa_thread_writeback_window(); // Fence for write barrier

For more information, please refer to DPA Subsystem.

Package

Instructions

Host compiler

Compiler specified through hostcc option. Both gcc and clang are supported.

Device compiler

The default device compiler is the "DPA compiler". Installing the DPACC package also installs the DPA compiler binaries dpa-clang, dpa-ar, dpa-nm and dpa-objdump.

FlexIO SDK and C library

Available as part of the DOCA software package. DPA toolchain does not provide C library and corresponding headers. Users are expected to use the C library for DPA from the FlexIO SDK.

Supported Versions

  • DPACC version 1.6.0

  • See DPA Subsystem for other component versions

DPACC Inputs and Outputs

DPACC can produce DPA programs in a single command by accepting all source files as input. DPACC also offers the flexibility of producing DPA object files or libraries from input files.

DPA object files contain both host stub objects (DPACC-generated interfaces) and device objects. These DPA object files can later be given to DPACC as input to produce the DPA library.

Phase

Option Name

Default Output File Name

Compile input device code files to DPA object files

--compile or -c

.dpa.o appended to the name of each input source file

Compile and link the input device code files/DPA object files, and produce a DPA program

No specific option

No default name, output file name must be specified

Compile and build DPA library from input device code files/DPA object files

--gen-libs or -gen-libs

No default name, output library name must be specified

DPACC can accept the following file types as input:

Input File Extension

File Type

Description

.c

C source file

DPA device code

.dpa.o

DPA object file

Object file generated by DPACC, containing both host and device objects

.a

DPA object archive

An archive of DPA object files. User can generate this archive from DPACC-generated DPA objects.

Based on the mode of operations, DPACC can generate the following output files:

Output File Type

Input Files

DPA object file

C source files

DPA program

C source files, DPA object files, and/or DPA object archives

DPA library

(DPA host library and DPA device library)

C source files, DPA object files, and/or DPA object archives

The following provides the commands to generate different kinds of supported output file types for each input file type:

Input

Output

DPACC Command

C source file

DPA program

dpacc -hostcc=gcc in.c -o libprog.a

DPA object

dpacc -hostcc=gcc in.c -c

DPA library

dpacc -hostcc=gcc in.c -o lib<name> -gen-libs

DPA object

DPA program

dpacc -hostcc=gcc in.dpa.o -o libprog.a

DPA library

dpacc -hostcc=gcc in.dpa.o -o lib<name> -gen-libs

DPA object archive

DPA program

dpacc -hostcc=gcc in.a -o libprog.a

DPA library

dpacc -hostcc=gcc in.a -o lib<name> -gen-libs

DPA Program

DPACC produces a DPA program in compile-and-link mode. A DPA program is a host library which contains:

  • DPACC-generated host stubs which facilitate invoking a DPA global function from the host application

  • Device executable, generated by DPACC by compiling input DPA device code

DPA program library must be linked with the host application that contains appropriate runtime APIs to load the device executable onto DPA memory.

DPA Object

DPACC produces DPA object files in compile-only mode. A DPA object is an object file for the host machine. In a DPA object, the device object generated by compiling the input device code file is placed inside a specific section of the generated host stubs object. This process is repeated for each input file.

DPACC_Output-version-1-modificationdate-1702941560593-api-v2.PNG


DPA Library

A DPA library is a collection of two individual libraries:

  • DPA device library – contains device objects generated from input files

  • DPA host library – contains host interface objects corresponding to the device objects in DPA device library

The DPA device library is consumed by DPACC during DPA-program generation and the DPA host library can optionally be linked with other host code and be distributed as the host library. Both libraries are generated as static archives.

dpa-lib-version-1-modificationdate-1702941560260-api-v2.png

DPACC Trajectory

The following diagram illustrates DPACC compile-and-link mode trajectory.

dpacc_trajectory-version-1-modificationdate-1702941557270-api-v2.jpg


Modes of Operation

Compile-and-link Mode

This is a one-step mode that accepts C source files or DPA object files and produces the DPA program. Specifying the output library name is mandatory in this mode.

Example commands:

Copy
Copied!
            

$ dpacc in1.c in2.c -o myLib1.a -hostcc=gcc # Takes C sources to produce myLib1.a library $ dpacc in3.dpa.o in4.dpa.o -o myLib2.a -hostcc=gcc # Takes DPA object files to produce myLib2.a library $ dpacc in1.c in3.dpa.o -o myLib3.a -hostcc=gcc # Takes C source and DPA object to produce myLib3.a library


Compile-only Mode

This mode accepts C source code and produces .dpa.o object files. These files can be given to DPACC to produce the DPA program. The mode is invoked by the --compile or -c option.

The user can explicitly provide the output object file name using the --output-file or -o option.

Example commands:

Copy
Copied!
            

$ dpacc -c input1.c -hostcc=gcc                 # Produces input1.dpa.o $ dpacc -c input3.c input4.c -hostcc=gcc        # Produces input3.dpa.o and input4.dpa.o $ dpacc -c input2.c -o myObj.dpa.o -hostcc=gcc  # Produces myObj.dpa.o


Library Generation Mode

This mode accepts C source files or DPA object files and produces the DPA program. Specifying the output DPA library name is mandatory in this mode.

Example commands:

Copy
Copied!
            

$ dpacc in1.c in2.c -o libdummy1 -hostcc=gcc -gen-libs # Takes C sources to produce libdummy1_host.a and libdummy_device.a archives $ dpacc in3.dpa.o in4.dpa.o -o libdummy2 -hostcc=gcc -gen-libs # Takes DPA object files to produce libdummy2_host.a and libdummy2_device.a archives $ dpacc in1.c in3.dpa.o -o outdir/libdummy3 -hostcc=gcc -gen-libs # Takes C source and DPA object to produce outdir/libdummy3_host.a and outdir/libdummy3_device.a archives

To execute DOCA DPACC compiler:

Copy
Copied!
            

Usage: dpacc <list-of-input-files> -hostcc=<path> [other options] Helper Flags: -h, --help                 Print help information about DPACC -V, --version               Print DPACC version information -v, --verbose               List the compilation commands generated by this invocation while also executing every command in verbose mode  -dryrun, --dryrun           Only list the compilation commands generated by DPACC, without executing them -keep, --keep Keep all intermediate files that are generated during internal compilation steps in the current directory -keep-dir, --keep-dir Keep all intermediate files that are generated during internal compilation steps in the given directory -optf, --options-file <file>,... Include command line options from the specified file

Mandatory Arguments

Flag

DPACC Mode

Description

List of one or more input files

All

List of C source files or DPA object file names. Specifying at least one input file is mandatory. A file with an unknown extension is treated as a DPA object file.

-hostcc, --hostcc <path>

All

Specify the host compiler. This is typically the native compiler present on the host system.

-o, --output-file <file>

Compile-and-link/library generation

Specify name and location of the output file.


Commonly Used Arguments

Tip

Use --help option for a list of all supported options.

Flag

Description

-app-name, --app-name <name>

Specify DPA application name for the DPA program. This option is required if multiple DPA programs are part of a host application because each DPA application must have a unique name. Default name is __dpa_a_out.

-flto, --flto

Enable link-time optimization (LTO) for device code. Specify this option during compilation along with an optimization level in devicecc-options.

-devicecc-options, --devicecc-options <options>,...

Specify the list of options to pass to the device compiler.

-devicelink-options, --devicelink-options <options>,...

Specify the list of options to pass during device linking stage.

-device-libs, --device-libs '-L<path> -l<name>',...

Specify the list of device libraries including their names (in -l) and their paths (in -L). FlexIO libraries are linked by default.

-I, --common-include-path <path>,...

Specify include search paths common to host and device code compilation. FlexIO headers paths are included by DPACC by default.

-o, --output-file <file>

Specify name and location of the output file.

  • Compile-only mode – name of the output DPA object file. If not specified, .dpa.o is generated for each .c file.

  • Compiler-and-link mode – name of the output DPA program. This is a mandatory option in compiler-and-link mode.

  • Library generation mode – name of the output library. This is a mandatory option for this mode. Output files <name>_device.a and <name>_host.a are generated.

-hostcc-options, --hostcc-options <options>,...

Specify the list of options to pass to the host compiler.

-gen-libs, --gen-libs

Generate a DPA library from input files

-ldpa, --ldpa

Link with DOCA DPA library

Warning

The devicecc-options option allows passing any option to the device compiler. However, passing options that prevent compilation of the input file may lead to unexpected behavior (e.g., -devicecc-options="-version" makes the device compiler print the version and not process input files).

Warning

Incompatible options that affect DPA global function argument sizes during DPACC invocation and host application compilation may lead to undefined behavior during execution (e.g., passing -hostcc-options="-fshort-enums" to DPACC and missing this option when building the host application).


LTO Usage Guidelines

Restrictions

  • Only the default linker script is supported with LTO

  • Using options -fPIC/-fpic/-shared/-mcmodel=large through -devicecc-options is not supported when LTO is enabled

  • Fat objects containing both LLVM bitcode and ELF representation are not supported

  • Thin LTO is not supported

Compatibility

During compilation, LLVM generates the object as bitcode IR (intermediate representation) when LTO is enabled instead of ELF representation. The bitcode IR generated by the DPA compiler is only guaranteed to be compatible within the same version of DPACC. All objects involved in link-time optimization (enabled with -flto) must be built with the same version of DPACC.

Examples

This section provides some common use cases of DPACC and showcases the dpacc command.

Building Libraries

This example shows how to build DPA libraries using DPACC. Libraries for DPA typically contain two archives, one for the host and one for the device.

Copy
Copied!
            

dpacc input.c -hostcc=gcc -o lib<name> -gen-libs -hostcc-options="-fPIC"

This command generates the output files lib<name>_host.a and lib<name>_device.a.

The host stub archive can be linked with other host code to generate a shared/static host library.

  • Generating a static host library:

    Copy
    Copied!
                

    ar x lib<name>_host.a # Extract objects to generate *.o ar cr lib<name>.a <*src.host.o> *.o # Generate final static archive with all objects

  • Generating a shared host library:

    Copy
    Copied!
                

    gcc -shared -o lib<name>.so <*src.host.o> -Wl,-whole-archive -l<name>_host -Wl,-no-whole-archive # Link the generated archive to build a shared library

Linking with DPA Device Library

The DPA device library generated by DPACC using -gen-libs as part of a DPA library can be consumed by DPACC using the -device-libs option.

Copy
Copied!
            

dpacc input.c -hostcc=gcc -o libInput.a -device-libs="-L <path-to-library> -l<libName>"


Link-time optimizations can be enabled using -flto along with an optimization level specified for device compilation.

Copy
Copied!
            

dpacc input1.c -hostcc=gcc -c -flto -devicecc-options="-O2" dpacc input2.c -hostcc=gcc -c -flto -devicecc-options="-O2" dpacc input1.dpa.o input2.dpa.o -hostcc=gcc -o libInput.a


Including Headers

This example includes headers for device compilation using devicecc-options and host compilation using hostcc-options. You may also specify headers for any compilation on both the host and device side using the -I option.

Copy
Copied!
            

dpacc input.c -hostcc=gcc -o libInput.a -I <common-headers-path> -devicecc-options="-I <device-headers-path>" -hostcc-options="-I <host-headers-path>"

DPA Compiler Usage

dpa-clang is a compiler driver for accessing the Clang/LLVM compiler, assembler, and linker which accepts C code files or object files and generates an output according to different usage modes.

Warning

Invoking the compiler, assembler, or linker directly may lead to unexpected errors.

Refer to the following resources for more detailed information on Clang:

Compiler Driver Command-line Options

Copy
Copied!
            

dpa-clang <list-of-input-files> [other-options]


Linker Command Line Options

LLD is the default linker provided in the DPA toolchain. Linker-related options are passed to through the compiler driver.

Copy
Copied!
            

dpa-clang -Wl,<linker-option>

For more information, please refer to the LLD command line reference.

dpacc-extract Command Line Options

dpacc-extract is a tool for extracting a device executable out of a DPA program or a host executable containing DPA program(s).

To execute dpacc-extract:

Copy
Copied!
            

Usage: dpacc-extract <input-file> -o=<output-file> [other options] Helper Flags:   -o, --output-file Specify name of the output file -app-name, --app-name <name> Specify name of the DPA application to extract -h, --help                 Print help information about dpacc-extract -V, --version               Print dpacc-extract version information -optf, --options-file <file>,... Include command line options from the specified file

Mandatory arguments:

Flag

Description

Input file

DPA program or host executable containing DPA program. Specifying one input file is mandatory.

-o, --output-file <file>

Specify name and location of the output device executable.

-app-name, --app-name <name>

Specify name of the DPA application to extract. Mandatory if input file has multiple DPA apps.


Objdump Command Line Options

The dpa-objdump utility prints the contents of object files and final linked images named on the command line.

For more information, please refer to the Objdump command line reference.

Commonly used dpa-objdump options:

Flag

Description

--mcpu=nv-dpa-bf3

Option to choose micro-architecture for DPA processor. nv-dpa-bf3 is the default CPU for dpa-objdump.


Archiver Command Line Options

dpa-ar is a Unix ar-compatible archiver.

For more information, please refer to the Archiver command line reference.

NM Tool Command Line Options

The dpa-nm utility lists the names of symbols from object files and archives.

For more information, please refer to the NM tool command line reference.

Common Compiler Options

Flag

Description

--mcpu=nv-dpa-bf3

Option to choose micro-architecture and ABI for DPA processor. nv-dpa-bf3 is the default CPU for the compiler.

-mrelax/-mno-relax

Option to enable/disable linker relaxations.

-I <dir>

Option to include header files present in <dir>.


Common Linker Options

Flag

Description

-Wl,-L <path-to-library> -Wl,-l<library-name>

Option to link against libraries

Warning

Linker options are provided through the compiler driver dpa-clang.

Warning

The LLD linker script is honored in addition to the default configuration rather than replacing the whole configuration like in GNU lD. Hence, additional options may be required to override some default behaviors.


Debugging Options

Flag

Description

-fdebug-macro

Option to emit macro debugging information. This option enables macro-debugging similar to GCC option -g3.


Miscellaneous Notes

  • Objects produced by LLD are not compatible with those generated by any other linker.

  • The default debugging standard of the DPA compiler is DWARFv5. GDB versions <10.1 have issues processing some DWARFv5 features. Use the option -devicecc-options="-gdwarf-4" with DPACC to debug with GDB versions <10.1.

© Copyright 2023, NVIDIA. Last updated on Feb 9, 2024.