DOCA Management

This guide provides instructions on how to use the DOCA Management API for developing applications for device management and configuration.

DOCA Management provides an API to manage NVIDIA® BlueField® DPU and host devices and their associated functions (PFs, VFs, and SFs). The library offers an interface for device configuration, capability management, and direct firmware control.

Using DOCA Management, administrators and applications can dynamically configure and query device settings, enable or disable capabilities, and manage device behavior at runtime through a standardized API that communicates directly with device firmware via the Linux kernel's fwctl framework.

This document is intended for software developers and system administrators wishing to manage and configure DPU and host devices programmatically, control device capabilities, and integrate device management into their applications.

DOCA Management library uses the Linux kernel’s fwctl framework and as such requires the following kernel drivers:

• fwctl.ko – fwctl device firmware access framework

• mlx5_fwctl.ko – mlx5 ConnectX fwctl driver

The library also relies on DOCA Core Device for device and device representor discovery. It is recommended to read the following section before proceeding:

• DOCA Core Device

This is the first release of DOCA Management library.

DOCA Management-based applications can run on the host machine and on the NVIDIA® BlueField® DPU. Certain configurations (e.g., configuring device capabilities) can only be performed through the device’s representor, and thus must be executed where the representor resides.

DOCA Management leverages DOCA Core Device for device and device representor discovery. The library operates through management device contexts and management device representor contexts, which correspond to DOCA devices and DOCA device representors, respectively.

The management contexts are the main objects of the library. They hold a management interface for the device and are passed to other APIs of the library to specify the device to operate on.

The library provides dedicated APIs for various configuration operations, allowing querying and setting device configuration parameters. Additionally, a generic API enables advanced users to send arbitrary configuration commands directly to the device firmware when specific abstractions are unavailable.

Objects

Device and Device Representor Management Context

The DOCA Management device context (doca_mgmt_dev_ctx) represents a management interface to a DOCA device. This context is used to perform device-level management operations. Creating the management device context requires a DOCA device, which must remain valid until the management device context is destroyed.

Similarly, the DOCA Management device representor context (doca_mgmt_dev_rep_ctx) represents a management interface to a DOCA device representor. This context is used to perform management operations for the device represented by the representor. Creating the management device representor context requires a DOCA Management device context (of the representor’s parent device) and a DOCA device representor, which must remain valid until the management device representor context is destroyed.

The management device context and management device representor context function as handles that are passed to other DOCA Management APIs to specify the target device. Some DOCA Management APIs accept a management device context, while others accept a management device representor context.

Dedicated Management Handles

The library provides dedicated handles for specific management operations. Each handle is an opaque structure that encapsulates configuration attributes and provides APIs to both configure the handle (via setters/getters) and apply/query these attributes to or from the device (via set/get operations).

The following table lists the currently supported handles:

The following section describes the expected usage flow of the various APIs of the DOCA Management library.

Device Context Creation

To start using the library, a DOCA Management device context must be created.

The following steps describe how to create a management device context:

1. Open the doca_dev that should be configured.

   For information about opening a DOCA device, see DOCA Core Device Discovery.

2. Create the corresponding doca_mgmt_dev_ctx by calling doca_mgmt_dev_ctx_create() with the opened doca_dev.

3. The created doca_mgmt_dev_ctx can now be used to configure the device or to create a DOCA Management device representor context for one of its representors (as explained in Device Representor Context Creation).

4. When the application finishes using the doca_mgmt_dev_ctx, destroy it using doca_mgmt_dev_ctx_destroy(). The doca_dev must remain valid until the doca_mgmt_dev_ctx is destroyed.

Device Representor Context Creation

Some configurations (such as configuring device capabilities) can be done only via the device representor. For this, a DOCA Management device representor context must be created.

The following steps describe how to create a management device representor context:

1. Create a doca_mgmt_dev_ctx for the parent device of the representor (as explained in Device Context Creation). For example, if VF0 of PF0 should be configured, create a doca_mgmt_dev_ctx for PF0.

2. Open the doca_dev_rep that should be configured. For information about opening a DOCA device representor, see DOCA Core Device Representor Discovery.

3. Create the corresponding doca_mgmt_dev_rep_ctx by calling doca_mgmt_dev_rep_ctx_create() with the created doca_mgmt_dev_ctx and the opened doca_dev_rep.

4. The created doca_mgmt_dev_rep_ctx can now be used to configure the device represented by the representor.

5. When the application finishes using the doca_mgmt_dev_rep_ctx, destroy it using doca_mgmt_dev_rep_ctx_destroy(). Both the doca_dev_rep and doca_mgmt_dev_ctx must remain valid until the doca_mgmt_dev_rep_ctx is destroyed.

Device Capabilities General Configuration

Configuring device capabilities is done via the device representor (doca_mgmt_dev_rep_ctx).

Getting General Capabilities

1. Create a doca_mgmt_dev_rep_ctx for the device representor that should be configured (as explained in Device Representor Context Creation).

2. Create a doca_mgmt_device_caps_general handle by calling doca_mgmt_device_caps_general_create().

3. Query the general capabilities of the device by calling doca_mgmt_device_caps_general_get() with the created doca_mgmt_dev_rep_ctx and doca_mgmt_device_caps_general.

4. Get the queried attributes by calling the getters of the doca_mgmt_device_caps_general handle. Currently, only the data direct attribute is available and it can be retrieved using doca_mgmt_device_caps_general_get_data_direct().

5. When the application finishes using the doca_mgmt_device_caps_general handle, destroy it using doca_mgmt_device_caps_general_destroy().

Setting General Capabilities

• Create a doca_mgmt_dev_rep_ctx for the device representor that should be configured (as explained in Device Representor Context Creation).

• Create a doca_mgmt_device_caps_general handle by calling doca_mgmt_device_caps_general_create().

• Set the desired attributes by calling the setters of the doca_mgmt_device_caps_general handle. Currently, only the data direct attribute is available and it can be set by calling doca_mgmt_device_caps_general_set_data_direct() with the desired value.

• Set the general capabilities of the device by calling doca_mgmt_device_caps_general_set() with the created doca_mgmt_dev_rep_ctx and doca_mgmt_device_caps_general.

• When the application finishes using the doca_mgmt_device_caps_general handle, destroy it using doca_mgmt_device_caps_general_destroy().

Congestion Control Global Status Configuration

Configuring the congestion control global status is done via the device (doca_mgmt_dev_ctx) and can be configured only for PFs.

Getting Congestion Control Global Status

1. Create a doca_mgmt_dev_ctx for the device that should be configured (as explained in Device Context Creation).

2. Create a doca_mgmt_cc_global_status handle by calling doca_mgmt_cc_global_status_create().

3. Set the priority and protocol attributes to specify which congestion control entity to query.

   This is done by calling doca_mgmt_cc_global_status_set_priority() and doca_mgmt_cc_global_status_set_protocol() with the desired priority (0–7) and protocol (RP or NP).

4. Query the congestion control global status of the device by calling doca_mgmt_cc_global_status_get() with the created doca_mgmt_dev_ctx and doca_mgmt_cc_global_status.

5. Get the queried enabled attribute by calling doca_mgmt_cc_global_status_get_enabled().

6. When the application finishes using the doca_mgmt_cc_global_status handle, destroy it using doca_mgmt_cc_global_status_destroy().

Setting Congestion Control Global Status

1. Create a doca_mgmt_dev_ctx for the device that should be configured (as explained in Device Context Creation).

2. Create a doca_mgmt_cc_global_status handle by calling doca_mgmt_cc_global_status_create().

3. Set the priority and protocol attributes to specify which congestion control entity to configure.

   This is done by calling doca_mgmt_cc_global_status_set_priority() and doca_mgmt_cc_global_status_set_protocol() with the desired priority (0–7) and protocol (RP or NP).

4. Set the enabled attribute by calling doca_mgmt_cc_global_status_set_enabled() with the desired value (1 to enable, 0 to disable).

5. Set the congestion control global status of the device by calling doca_mgmt_cc_global_status_set() with the created doca_mgmt_dev_ctx and doca_mgmt_cc_global_status.

6. When the application finishes using the doca_mgmt_cc_global_status handle, destroy it using doca_mgmt_cc_global_status_destroy().

Raw Command

For advanced use cases, the library provides a raw command API that allows sending arbitrary firmware commands directly to the device.

The following steps detail how to send a raw command to a device:

1. Create a doca_mgmt_dev_ctx for the device that should receive the command (as explained in Device Context Creation).

2. Prepare the input payload buffer according to the specific command requirements.

3. Allocate an output payload buffer to receive the command response.

4. Call doca_mgmt_raw_cmd() with the following parameters:

   • The created doca_mgmt_dev_ctx

   • A command identifier string (for logging)

   • The command scope

   • Pointer to the input payload buffer and its size

   • Pointer to the output payload buffer and its size

5. Check the return value to verify that the command executed successfully.

6. Parse the output payload buffer according to the specific command response format.

The command scope parameter specifies the scope of the command being performed and must match the fwctl scope assigned to it by the mlx5_fwctl driver.

The mapping between DOCA and fwctl command scopes is as follows:

This section describes DOCA Management samples based on the DOCA Management library. The samples demonstrate how to use the DOCA Management API for the following device configuration tasks:

• Configure device data direct capability.

• Configure device congestion control global status.

Running the Samples

Refer to the following documents:

• DOCA Installation Guide for Linux for details on how to install BlueField-related software.

• NVIDIA BlueField Platform Software Troubleshooting Guide for any issue you may encounter with the installation, compilation, or execution of DOCA samples.

To build a given sample, run the following command. If you downloaded the sample from GitHub, update the path in the first line to reflect the location of the sample file:

            

            
cd /opt/mellanox/doca/samples/doca_mgmt/<sample_name>
meson /tmp/build
ninja -C /tmp/build
        

Sample usage (e.g., doca_mgmt_data_direct):

            

            
Usage: doca_mgmt_data_direct [Program Commands] [DOCA Flags]
 
Program Commands:
  get                               Get data direct capability
  set                               Set data direct capability
 
DOCA Flags:
  -h, --help                        Print a help synopsis
  -v, --version                     Print program version information
  -l, --log-level                   Set the (numeric) log level for the program <10=DISABLE, 20=CRITICAL, 30=ERROR, 40=WARNING, 50=INFO, 60=DEBUG, 70=TRACE>
  --sdk-log-level                   Set the SDK (numeric) log level for the program <10=DISABLE, 20=CRITICAL, 30=ERROR, 40=WARNING, 50=INFO, 60=DEBUG, 70=TRACE>
  -j, --json <path>                 Parse command line flags from an input json file
        

For additional information per sample, use the -h option:

            

            
/tmp/build/doca_<sample_name> -h
/tmp/build/doca_<sample_name> get -h
/tmp/build/doca_<sample_name> set -h
        

Samples

Management Data Direct

The NVIDIA® ConnectX®-8 device may expose a side DMA engine as an additional PCIe PF called the Data Direct device. This additional device allows access to data buffers through multiple PCIe data path interfaces. By default, the data direct capability is disabled for VFs and SFs and must be explicitly enabled to allow data direct usage for a specific VF or SF.

This sample demonstrates how to set and get the data direct capability for VFs and SFs using the DOCA Management library.

The sample supports two commands:

• get – Displays the current data direct capability state.

• set – Enables or disables the data direct capability.

For both commands, the device (VF or SF) to operate on must be specified.

Specifying the Device

• For VFs:

  1. Specify the VF PCIe address (e.g., 0000:08:00.2) using the -v or --vf-pci-addr parameter.

  2. Specify the VF representor using the -r or --rep parameter in the format pci/<parent_pf_pci_address>,pf<pfnum>vf<vfnum>. Example for VF 0000:08:00.2, where VF number = 0 and parent PF = 0000:08:00.0 (PF number 0):

     Copy

     Copied!

                 
     
                 
     pci/0000:08:00.0,pf0vf0
             

• For SFs, specify the SF representor using the -r or --rep parameter in the format pci/<parent_pf_pci_address>,pf<pfnum>sf<sfnum>. Example for SF number 88 and parent PF 0000:08:00.0 (PF number 0):

  Copy

  Copied!

              
  
              
  pci/0000:08:00.0,pf0sf88
          

Set Command

The -e or --enabled parameter must be specified with true or false values to indicate whether the data direct capability should be enabled or disabled, respectively.

Example Executions

• Get data direct state for a VF with PCIe address 0000:08:00.2:

  Copy

  Copied!

              
  
              
  $ /tmp/build/doca_mgmt_data_direct get --vf-pci-addr 0000:08:00.2
          

• Enable data direct for a VF (uninitialized):

  Copy

  Copied!

              
  
              
  $ /tmp/build/doca_mgmt_data_direct set --vf-pci-addr 0000:08:00.2 --enabled true
          

• Disable data direct for a VF (uninitialized):

  Copy

  Copied!

              
  
              
  $ /tmp/build/doca_mgmt_data_direct set --vf-pci-addr 0000:08:00.2 --enabled false
          

• Enable data direct using VF representor:

  Copy

  Copied!

              
  
              
  $ /tmp/build/doca_mgmt_data_direct set --rep pci/0000:08:00.0,pf0vf0 --enabled true
          

• Enable data direct for SF representor:

  Copy

  Copied!

              
  
              
  $ /tmp/build/doca_mgmt_data_direct set --rep pci/0000:08:00.0,pf0sf88 --enabled true
          

Sample Logic

• Get Data Direct:

  1. Opening the DOCA device and a DOCA device representor that were specified in the command line arguments.

  2. Creating a DOCA management device context for the DOCA device and a DOCA management device representor context for the DOCA device representor.

  3. Creating a device caps general handle.

  4. Executing get operation through the DOCA management device caps general API.

  5. Retrieving the data direct capability for the specified representor interface.

  6. Cleaning up all DOCA management and device structures.

• Set Data Direct:

  1. Opening the DOCA device that was specified in the command line arguments.

  2. Creating a DOCA management device context for the DOCA device and a DOCA management device representor context for the DOCA device representor.

  3. Creating a device caps general handle and setting data direct attribute.

  4. Executing set operation through the DOCA management device caps general API.

  5. Cleaning up all DOCA management and device structures.

Reference:

• /opt/mellanox/doca/samples/doca_mgmt/mgmt_data_direct/mgmt_data_direct_sample.c

• /opt/mellanox/doca/samples/doca_mgmt/mgmt_data_direct/mgmt_data_direct_main.c

• /opt/mellanox/doca/samples/doca_mgmt/mgmt_data_direct/meson.build

Management Congestion Control Global Status

This sample demonstrates how to set and get congestion control global status for different priorities and protocols (NP/RP) on a DOCA device using the DOCA Management library.

The sample logic includes two main operations:

• Get Congestion Control Global Status:

  1. Opening the DOCA device that was specified in the command line arguments.

  2. Creating a DOCA management device context for the DOCA device.

  3. Creating a congestion control global status handle and setting priority and protocol attributes.

  4. Executing get operation through the DOCA management congestion control global status API.

  5. Retrieving the congestion control global status enabled attribute for the specified priority and protocol.

  6. Cleaning up all DOCA management and device structures.

• Set Congestion Control Global Status:

  1. Opening the DOCA device that was specified in the command line arguments.

  2. Creating a DOCA management device context for the DOCA device.

  3. Creating a congestion control global status handle and setting priority, protocol and enabled attributes.

  4. Executing set operation through the DOCA management congestion control global status API.

  5. Cleaning up all DOCA management and device structures.

Reference:

• /opt/mellanox/doca/samples/doca_mgmt/mgmt_cc_global_status/mgmt_cc_global_status_sample.c

• /opt/mellanox/doca/samples/doca_mgmt/mgmt_cc_global_status/mgmt_cc_global_status_main.c

• /opt/mellanox/doca/samples/doca_mgmt/mgmt_cc_global_status/meson.build