DOCA Time Sync Application Guide

This guide provides an example of a Time Sync implementation on top of NVIDIA® BlueField® DPU.

The DOCA Time Sync reference application demonstrates how to trigger and correlate events across processors operating on different clock domains within a BlueField DPU environment.

The application triggers events on the x86 host, the BlueField Arm cores, and the Data Path Accelerator (DPA) subsystem. It then leverages the DOCA Clock cross-timestamping functionality (available in DOCA Core) to correlate these events against the NVIDIA® ConnectX® real-time clock (RTC). This allows the application to determine the precise chronological ordering of events occurring across the different subsystems.

            

            
sudo mlxconfig -d <device> set REAL_TIME_CLOCK_ENABLE=1
        

The DOCA Time Sync architecture consists of two distinct applications:

• DPU application (doca_time_sync_dpu) – Runs on the BlueField DPU Arm cores.

• Host application (doca_time_sync_host) – Runs on the x86 host system.

Operational workflow:

1. The DPU application must be started first. It establishes a DOCA Comch server to listen for incoming connections.

2. The Host application connects to the Comch server on the DPU.

3. Event triggering:

   1. Host to Arm: The Host application sends a message via the Comch connection to the DPU Arm cores, triggering the first event.

   2. Arm to DPA: The DPU application loads a DOCA DPA kernel and communicates with it to trigger a second event on the DPA subsystem.

4. All event timestamps are relayed back to the Host application, where they are correlated and ordered based on the common DOCA Clock.

The DOCA Time Sync application generates a sequence of four events across three distinct processors, inserting variable delays between each step to simulate real-world processing latency.

Event Sequence

The application tracks the following four events:

1. x86 Host: Packages and sends a request message to the DPU

2. BlueField Arm Cores: Receives the message from the Host

3. DPA Subsystem: Executes a Remote Procedure Call (RPC) triggered by the Arm cores

4. x86 Host: Receives the response message from the DPU

Clock Synchronization and Correlation

Each event records a timestamp using its processor's local clock:

• x86 Host & Arm Cores: Use the system RTC.

• DPA Subsystem: Uses its internal local timer.

To order these events chronologically, all timestamps are correlated against a single common reference clock: the ConnectX RTC embedded in the BlueField DPU.

Host and Arm Core Synchronization

Both the x86 and DPU applications utilize the DOCA Clock cross-timestamping library (part of DOCA Core). This library captures the local clock time and the common NIC clock time simultaneously to establish a precise correlation.

DPA Subsystem Synchronization

The DPA kernel can only capture time using its local timer. To synchronize this local DPA timestamp with the common NIC clock, the Host application performs a retrospective calculation:

1. Uses DOCA Clock to capture the current relationship between the NIC clock and the DPA timer.

2. Determines the duration (in seconds/nanoseconds) between the current DPA time and the recorded event time.

3. Subtracts this delta from the current NIC time to derive the precise NIC time when the DPA event occurred.

Output Logging

The Host application aggregates all data and outputs it to a log file (time_sync.log). Each entry includes:

• Synchronized time: The calculated time on the common NIC RTC.

• Local time: The raw timestamp from the processor's local clock.

• Accuracy: The margin of error for the synchronization.

• Event description: A label identifying the specific event step.\

This application leverages the following DOCA libraries:

• DOCA Comch

• DOCA DPA

Refer to their respective programming guide for more information.

DOCA reference applications are installed with full source code and build instructions. This allows you to compile them as-is or modify the source code to create custom versions.

The source code for the application is located in the following directory:

            

            
/opt/mellanox/doca/applications/time_sync/
        

Compiling All Applications

All DOCA applications are defined under a single Meson project. By default, the build process compiles all of them.

To build all applications:

            

            
cd /opt/mellanox/doca/applications/
meson /tmp/build
ninja -C /tmp/build
        

Compiling Only the Current Application

To reduce build time, you can configure Meson to build only the Time Sync application.

Option 1: Command Line Configuration

Run the following commands to disable all applications and explicitly enable Time Sync:

            

            
cd /opt/mellanox/doca/applications/
meson /tmp/build -Denable_all_applications=false -Denable_time_sync=true
ninja -C /tmp/build
        

Option 2: Configuration File

edit the configuration file directly:

1. Edit /opt/mellanox/doca/applications/meson_options.txt.

   • Set enable_all_applications to false

   • Set enable_time_sync to true

2. Run the standard compilation commands:

   Copy

   Copied!

               
   
               
   cd /opt/mellanox/doca/applications/
   meson /tmp/build
   ninja -C /tmp/build
           

Application Execution

The Time Sync application is distributed as source code and must be compiled before execution.

Running on x86 Host

• Usage syntax:

  Copy

  Copied!

              
  
              
  Usage: doca_time_sync_host [DOCA Flags] [Program Flags]
   
  DOCA Flags:
    -h, --help                        Print a help synopsis
    -v, --version                     Print program version information
    -l, --log-level                   Set the (numeric) log level for the program <10=DISABLE, 20=CRITICAL, 30=ERROR, 40=WARNING, 50=INFO, 60=DEBUG, 70=TRACE>
    --sdk-log-level                   Set the SDK (numeric) log level for the program <10=DISABLE, 20=CRITICAL, 30=ERROR, 40=WARNING, 50=INFO, 60=DEBUG, 70=TRACE>
    --log-filter                      Filter logs from specific modules, separated by comma
    -j, --json <path>                 Parse command line flags from an input json file
   
  Program Flags:
    -p, --pci-addr                    DOCA device PCI address
    -d, --delay                       Delay (msecs) to insert between event triggers (optional)
          

• Example execution:

  Copy

  Copied!

              
  
              
  sudo ./doca_time_sync_host -p 3b:00.0 -d 1000
          

  Note

  The application requires sudo (root privileges) to access cross-timestamping system calls.

  Note

  Ensure 3b:00.0 matches your specific device's PCIe address.

Running on BlueField DPU

• Usage syntax:

  Copy

  Copied!

              
  
              
  Usage: doca_time_sync_dpu [DOCA Flags] [Program Flags]
   
  DOCA Flags:
    -h, --help                        Print a help synopsis
    -v, --version                     Print program version information
    -l, --log-level                   Set the (numeric) log level for the program <10=DISABLE, 20=CRITICAL, 30=ERROR, 40=WARNING, 50=INFO, 60=DEBUG, 70=TRACE>
    --sdk-log-level                   Set the SDK (numeric) log level for the program <10=DISABLE, 20=CRITICAL, 30=ERROR, 40=WARNING, 50=INFO, 60=DEBUG, 70=TRACE>
    --log-filter                      Filter logs from specific modules, separated by comma
    -j, --json <path>                 Parse command line flags from an input json file
   
  Program Flags:
    -p, --pci-addr                    DOCA device PCI address
    -r, --repr-addr                   DOCA device representor PCI address (optional)
          

• Example execution:

  Copy

  Copied!

              
  
              
  sudo ./doca_time_sync_dpu -p 03:00.0 -r 3b:00.0
          

  Note

  The application requires sudo (root privileges) to access cross-timestamping system calls.

  Note

  Ensure the device address (03:00.0) and representor address (3b:00.0) match your system configuration.

Command Line Flags

General Flags

Host Program Flags

DPU Program Flags

Troubleshooting

Refer to the NVIDIA BlueField Platform Software Troubleshooting Guide for any issue encountered with the compilation, installation, or execution of the DOCA applications.

Common

1. Parse application argument.

   1. Initialize arg parser resources and register DOCA general parameters.

      Copy

      Copied!

                  
      
                  
      doca_argp_init();
              

   2. Register Time Sync application parameters.

      Copy

      Copied!

                  
      
                  
      time_sync_common_reg_params();
              

   3. Parse the arguments.

      Copy

      Copied!

                  
      
                  
      doca_argp_start();
              

2. Open DOCA devices for use in the application:

   Copy

   Copied!

               
   
               
   time_sync_common_open_dev_with_caps();
    
   // DPU only
   time_sync_common_open_repr();
           

   1. Parse PCIe address for associated DOCA device

   2. Verify the selected device has the required capabilities

      1. Comch support

      2. Real-time NIC clock support

      3. DPA timer support

      4. DOCA DPA support (DPU only)

      5. Repr Support (DPU only)

   3. Open valid device

   4. On DPU, parse and open repr device

3. Create a DOCA Clock Context:

   Copy

   Copied!

               
   
               
   time_sync_common_create_clock();
           

4. Run Host or DPU-specific code.

5. Destroy DOCA Clock Context:

   Copy

   Copied!

               
   
               
   time_sync_common_destroy_clock();
           

6. Close DOCA devices:

   Copy

   Copied!

               
   
               
   time_sync_common_close_devs();
           

7. Destroy Arg Parser:

   Copy

   Copied!

               
   
               
   doca_argp_destroy();
           

Host (x86) App

1. Initialize Comch client.

   Copy

   Copied!

               
   
               
   time_sync_host_init_comch_client();
           

   1. Create progress engine

   2. Create DOCA Comch Client context

   3. Configure taskpool/callbacks for sending and receiving messages

   4. Start Comch Client

2. Run main loop.

   Copy

   Copied!

               
   
               
   time_sync_host_main_loop();
           

   1. Progress until Client is fully connected to Comch Server on DPU

   2. Get event time (on host and NIC) before sending a message

   3. Create and send a message to DPU containing the input delay time in milliseconds

   4. Wait to receive a response from the DPU containing DPU and DPA event times

   5. Get event time (on host and NIC) of message receive

   6. Convert received DPA time to NIC time using cross-timestamping functions

   7. Log the local and synchronized time of all events to 'time_sync.log'

3. Clean up Comch Client

   Copy

   Copied!

               
   
               
   time_sync_host_close_comch_client();
           

   1. Stop the DOCA Comch Client

   2. Progress until connection is fully shut down and context is IDLE

   3. Destroy Client context and progress engine

DPU App

1. Load DPA application.

   Copy

   Copied!

               
   
               
   time_sync_dpu_load_dpa_app();
           

   1. Create a new DOCA DPA context

   2. Add an app to the context (app with given name is compiled alongside DPU app - source in /opt/mellanox/doca/applications/time_sync/dpu_device/time_sync_dev.c)

   3. Start the DPA context

2. Initiate Comch Server.

   Copy

   Copied!

               
   
               
   time_sync_dpu_init_comch_server();
           

   1. Create progress engine

   2. Create DOCA Comch Server context

   3. Configure taskpool/callbacks for sending and receiving messages

   4. Configure callback for connection events

   5. Start Comch Server

3. Run main loop.

   Copy

   Copied!

               
   
               
   time_sync_dpu_main_loop();
           

   1. Progress until an x86 client has established a connection to the Comch Server

   2. Receive a message from Client

      1. Extract the delay from request message

      2. Sleep for 'delay' milliseconds

      3. Get event time (on ARM and NIC) of message receive

      4. Sleep for 'delay' milliseconds

      5. Trigger a remote procedure call to the loaded DPA app which will return its local time from running kernel

      6. Sleep for 'delay' milliseconds

      7. package event times from ARM/DPA and send response message back to host

   3. Progress until x86 has finished and closed Client Comch connection

4. Clean up Comch Server

   Copy

   Copied!

               
   
               
   time_sync_dpu_close_comch_server();
           

   1. Stop the DOCA Comch Server

   2. Progress until context is IDLE

   3. Destroy Server context and progress engine

5. Clean up DPA application

   Copy

   Copied!

               
   
               
   time_sync_dpu_unload_dpa_app();
           

   1. Stop the DOCA DPA context

   2. Destroy DPA context

• /opt/mellanox/doca/applications/time_sync/