NVIDIA DOCA Secure Channel Application Guide

This guide provides a secure channel implementation on top of NVIDIA® BlueField® DPU.

The DOCA Secure Channel reference application leverages the DOCA Comch API which creates a secure, network independent communication channel between the host and the NVIDIA BlueField DPU.

Comch allows the host to control services on the DPU, activate certain offloads, or exchange messages using client-server/producer-consumer framework.

The client (host) side can communicate only with one server at a time while the server side is able to communicate with multiple clients.

The API allows communication between any PF/VF/SF on the host to the server located on the DPU.

Once a client-server connection is established, multiple producer-consumer instances can be spawned to transfer large amount of data in a first-in-first-out (FIFO) style queue.

Secure channel allows the user to select the message size and amount to be exchanged between the client and the server to simulate heavy load on the channel.

A secure channel application runs on client mode (host) and server mode (DPU). Once a channel is open, messages can flow from both sides.

The secure channel application runs on top of the DOCA Comch API.

The application begins by establishing a client-server connection between host (client) and DPU (server). Once this connection is established, the application on both sides spawns two new threads:

• Consumer – The consumer thread registers a number of post_recv buffers with the opposite end of the connection

• Producer – The producer thread, after it detects the consumer, populates these post_recv buffers with data that can then be read by the consumer

Once the consumer has processed a received buffer, the buffer is reposted (via task submit) to again be populated by the opposite producer.

When the producer has sent the requested number of messages, and the consumer has received the same amount, both threads exit and the main client-server connection is destroyed.

The flow is described in the following diagram:

This application leverages the following DOCA library:

• DOCA Comch

Refer to its respective programming guide for more information.

The installation of DOCA's reference applications contains the sources of the applications, alongside the matching compilation instructions. This allows for compiling the applications "as-is" and provides the ability to modify the sources, then compile a new version of the application.

The sources of the application can be found under the application's directory: /opt/mellanox/doca/applications/secure_channel/.

Compiling All Applications

All DOCA applications are defined under a single meson project. So, by default, the compilation includes all of them.

To build all the applications together, run:

            

            
cd /opt/mellanox/doca/applications/
meson /tmp/build 
ninja -C /tmp/build
        

Compiling Only the Current Application

To directly build only the secure channel application:

            

            
cd /opt/mellanox/doca/applications/
meson /tmp/build -Denable_all_applications=false -Denable_secure_channel=true
ninja -C /tmp/build
        

Alternatively, users can set the desired flags in the meson_options.txt file instead of providing them in the compilation command line:

1. Edit the following flags in /opt/mellanox/doca/applications/meson_options.txt:

   • Set enable_all_applications to false

   • Set enable_secure_channel to true

2. Run the following compilation commands :

   Copy

   Copied!

               
   
               
   cd /opt/mellanox/doca/applications/
   meson /tmp/build 
   ninja -C /tmp/build
           

   Info

   doca_secure_channel is created under /tmp/build/secure_channel/.

Troubleshooting

Refer to the NVIDIA DOCA Troubleshooting Guide for any issue encountered with the compilation of the application .

Application Execution

The secure channel application is provided in source form. Therefore, a compilation is required before the application can be executed.

1. Application usage instructions:

   Copy

   Copied!

               
   
               
   Usage: doca_secure_channel [DOCA Flags] [Program Flags]
    
   DOCA Flags:
     -h, --help                        Print a help synopsis
     -v, --version                     Print program version information
     -l, --log-level                   Set the (numeric) log level for the program <10=DISABLE, 20=CRITICAL, 30=ERROR, 40=WARNING, 50=INFO, 60=DEBUG, 70=TRACE>
     --sdk-log-level                   Set the SDK (numeric) log level for the program <10=DISABLE, 20=CRITICAL, 30=ERROR, 40=WARNING, 50=INFO, 60=DEBUG, 70=TRACE>
     -j, --json <path>                 Parse all command flags from an input json file
    
   Program Flags:
     -s, --msg-size                    Message size to be sent
     -n, --num-msgs                    Number of messages to be sent
     -p, --pci-addr                    DOCA Comm Channel device PCI address
     -r, --rep-pci                     DOCA Comm Channel device representor PCI address (needed only on DPU)
           

   Info

   This usage printout can be printed to the command line using the -h (or --help) options:

   Copy

   Copied!

               
   
               
   ./doca_secure_channel -h
           

   Info

   For additional information, refer to section "Command Line Flags".

2. CLI example for running the application on the BlueField:

   Copy

   Copied!

               
   
               
   ./doca_secure_channel -s 256 -n 10 -p 03:00.0 -r 3b:00.0
           

   Note

   Both the DOCA Comch device PCIe address (03:00.0) and the DOCA Comch device representor PCIe address (3b:00.0) should match the addresses of the desired PCIe devices. Users may use the DOCA Capabilities Print Tool to find out their device and device representor PCIe addresses.

3. CLI example for running the application on the host:

   Copy

   Copied!

               
   
               
   ./doca_secure_channel -s 256 -n 10 -p 3b:00.0
           

   Note

   The DOCA Comch device PCIe address (3b:00.0) should match the address of the desired PCIe device.

4. The application also supports a JSON-based deployment mode, in which all command-line arguments are provided through a JSON file:

   Copy

   Copied!

               
   
               
   ./doca_secure_channel --json [json_file]
           

   For example:

   Copy

   Copied!

               
   
               
   ./doca_secure_channel --json ./sc_params.json
           

   Note

   Before execution, ensure that the used JSON file contains the correct configuration parameters, and especially the PCIe addresses necessary for the deployment.

Command Line Flags

            

            
"log-level": 60
        

            

            
"sdk-log-level": 40
        

            

            
"msg-size": 128
        

            

            
"num-msgs": 256
        

            

            
"pci-addr": 03:00.1
        

            

            
"rep-pci": b1:00.1
        

Troubleshooting

Refer to the NVIDIA DOCA Troubleshooting Guide for any issue encountered with the installation or execution of the DOCA applications .

1. Parse application argument.

   1. Initialize the arg parser resources and register DOCA general parameters.

      Copy

      Copied!

                  
      
                  
      doca_argp_init();
              

   2. Register secure channel application parameters.

      Copy

      Copied!

                  
      
                  
      register_secure_channel_params();
              

   3. Parse application parameters:

      Copy

      Copied!

                  
      
                  
      doca_argp_start();
              

   4. Establish the client-server control path connection:

      Copy

      Copied!

                  
      
                  
      comch_utils_fast_path_init();
              

      1. Create a doca_comch_client or doca_comch_server depending on where the application is run (x86 host or DPU Arm cores).

      2. Register a callback which is triggered when new consumers are created on the opposite end.

      3. Progress the connection until it becomes established.

2. Run main logic.

   Copy

   Copied!

               
   
               
   sc_start();
           

   1. Send a message on the control channel telling the opposite end how many fastpath messages it intends to send and their length.

   2. Wait to receive a similar message from the opposite side.

   3. Start a producer thread:

      1. Create and starts a doca_comch_producer.

      2. Wait until a consumer has been registered from the opposite end of the control channel.

      3. Create a doca_buf of input message length to send repeatedly.

      4. Submit the max number of doca_comch_producer_task_send tasks each containing the doca_buf.

      5. Resubmit each task from its completion callback until the requested number of tasks are sent.

      6. Destroy producer and ends thread.

   4. Start a consumer thread

      1. Create and starts a doca_comch_consumer.

      2. Progress until consumer has been fully registered with the control channel connection.

      3. Create doca_bufs of negotiated size to receive data on.

      4. Submit the max number of doca_comch_consumer_task_post_recv tasks with each of the allocated doca_bufs.

      5. Resubmit each post_recv buffer from its callback when it has been populated by a producer.

      6. Destroy consumer and thread when it has received the communicated number of fastpath messages.

   5. Sends (DPU) or waits (host) on a message to indicate that all fastpath messages have completed and the Comch control connection can be destroyed.

• /opt/mellanox/doca/applications/secure_channel/

• /opt/mellanox/doca/applications/secure_channel/sc_params.json