DOCA Flow Tune Tool

DOCA Flow Tune is a powerful, one-stop-shop solution, providing visibility and analysis capabilities for DOCA Flow programs.

DOCA Flow Tune is a one-stop-shop solution which allows developers to visualize their traffic steering pipelines, have a live monitor of software Key Performance Indicators (KPIs) as well as hardware counters, and gain valuable performance insights about their DOCA-Flow-based program.

DOCA Flow Tune is especially useful for the following scenarios:

• Aiding developers during the development of their traffic steering pipeline by providing visualization of the pipeline, and later also performance insights about the designed pipeline

• Aiding developers in pre-production environment by providing live monitoring of the performance indicators of the program on both software and hardware levels, and helping detect possible bottlenecks/critical paths so they are addressed before deployment to production environments

• Aiding administrators monitor the program in production by providing live monitoring as well as high-rate hardware counters to be used when analyzing a possible deployment/setup issue

The tool operates in three distinct modes, Monitor, Analyze, and Visualize, which are presented in the following subsections.

Monitor Mode Overview

This mode collects and displays both hardware counters and software KPIs in real time (as extracted from the running DOCA Flow program and the underlying setup), providing a comprehensive view of the system's performance:

This information can also be exported to a CSV file for further analysis.

Analyze Mode Overview

The analyze mode supports the ability to dump the internal steering pipeline state to be used by Visualize mode.

Visualize Mode Overview

This mode allows users to produce a graphical representation of their steering pipeline (as built using the DOCA Flow API), allowing developers to quickly understand their program's pipeline, and compare it with their intended architecture.

The following is an example from the DOCA IPsec Security Gateway reference application:

Mermaid Graph

• Graph contains the following shapes:

  • Square - represents a port.

  • Hexagon - represents a pipe.

  • Arrow - represent a possible connection between pipes and ports.

• Pipe data is divided into different layers, layers can be stacked one over the other.

  • Base layer - includes pipe general attributes (name, label, type, domain, port id), mask, monitor and actions data. This layer is the default layer to be used in case that no other layer was provided when running DOCA Flow Tune.

  • Cost - includes the minimum and maximum cost that a packet can incur while being processed in a pipe. This layer is an additional layer on top of the base layer.

  • Critical Path - highlights the path with the highest cost in the entire pipeline. If more than one is the highest, only one is highlighted.

  • Resources - Includes pipe resources (number of counters, number of meters, actions memory). This layer is an additional layer on top of the base layer.

DOCA Flow Tune depends on the following DOCA SDK libraries:

• DOCA Flow

• DOCA Telemetry

• DOCA 3.0.0 and higher.

• For optimal experience, it is recommended to comply with the prerequisites of all the listed dependencies, and especially with their recommended firmware versions.

To execute DOCA Flow Tune tool:

            

            
Usage: doca_flow_tune [Program Commands] [DOCA Flags] [Program Flags]
 
Program Commands:
  analyze                           Run Flow Tune in Analyze mode
  monitor                           Run Flow Tune in Monitor mode
  visualize                         Run Flow Tune in Visualize mode
 
DOCA Flags:
  -h, --help                        Print a help synopsis
  -v, --version                     Print program version information
  -l, --log-level                   Set the (numeric) log level for the program <10=DISABLE, 20=CRITICAL, 30=ERROR, 40=WARNING, 50=INFO, 60=DEBUG, 70=TRACE>
  --sdk-log-level                   Set the SDK (numeric) log level for the program <10=DISABLE, 20=CRITICAL, 30=ERROR, 40=WARNING, 50=INFO, 60=DEBUG, 70=TRACE>
  -j, --json <path>                 Parse all command flags from an input json file
        

            

            
doca_flow_tune -h
        

            

            
doca_flow_tune monitor -h
        

Monitor Command

The monitor command presents software KPIs and hardware counters. Each component offers various options, which can be specified in the configuration file under the monitor section, or through the CLI.

            

            
Usage: doca_flow_tune monitor [Program Commands] [DOCA Flags] [Program Flags]
 
Program Commands:
  background                        Collect software key performance indicators and hardware counters on the background
 
DOCA Flags:
  -h, --help                        Print a help synopsis
  -l, --log-level                   Set the (numeric) log level for the program <10=DISABLE, 20=CRITICAL, 30=ERROR, 40=WARNING, 50=INFO, 60=DEBUG, 70=TRACE>
  --sdk-log-level                   Set the SDK (numeric) log level for the program <10=DISABLE, 20=CRITICAL, 30=ERROR, 40=WARNING, 50=INFO, 60=DEBUG, 70=TRACE>
 
Program Flags:
  --enable-csv                      Enable dumping data to CSV file
  --disable-csv                     Disable dumping data to CSV file
  --csv-file-name                   CSV file name to create
  --hw-profile                      Register hardware profile {basic, full}
  --sw-profile                      Register software profile
  -f, --cfg-file                    JSON configuration file
        

Supported sub-commands:

• background – This subcommand allows performing CSV dumping without displaying the output on the screen. This is useful for scenarios where one wants to log counters without cluttering the terminal. It also supports high-rate dumping for hardware counters which may be activated using the --high-rate flag.

  Copy

  Copied!

              
  
              
  Usage: doca_flow_tune monitor background [DOCA Flags] [Program Flags]
   
  DOCA Flags:
    -h, --help                        Print a help synopsis
    -l, --log-level                   Set the (numeric) log level for the program <10=DISABLE, 20=CRITICAL, 30=ERROR, 40=WARNING, 50=INFO, 60=DEBUG, 70=TRACE>
    --sdk-log-level                   Set the SDK (numeric) log level for the program <10=DISABLE, 20=CRITICAL, 30=ERROR, 40=WARNING, 50=INFO, 60=DEBUG, 70=TRACE>
   
  Program Flags:
    --high-rate                       Enable dumping hardware counters data to CSV file in high rate
    --hw-profile                      Register hardware profile {basic, full}
    --sw-profile                      Register software profile
          

CLI Examples

• To launch the monitor command with a given configuration file:

  Copy

  Copied!

              
  
              
  doca_flow_tune monitor -f /tmp/flow_tune_cfg.json
          

• To launch the monitor command with both a given configuration file and a CLI parameter for specifying the desired hardware counters profile:

  Copy

  Copied!

              
  
              
  doca_flow_tune monitor -f /tmp/flow_tune_cfg.json --hw-profile basic
          

• To launch the monitor command with the background subcommand and the request to perform a high rate collection and export for the hardware counters:

  Copy

  Copied!

              
  
              
  doca_flow_tune monitor -f /tmp/flow_tune_cfg.json background --high-rate
          

  Note

  The tool silently creates and updates the flow_tune.csv file.

Analyze Command

The analyze command runs a specified set of analysis methods over the target DOCA Flow program. The analysis supports the ability to export a JSON description of the steering pipeline, as is used by the visualize command, and could later be used for future analysis methods (both online or offline).

            

            
Usage: doca_flow_tune analyze export [DOCA Flags] [Program Flags]
 
DOCA Flags:
  -h, --help                        Print a help synopsis
  -l, --log-level                   Set the (numeric) log level for the program <10=DISABLE, 20=CRITICAL, 30=ERROR, 40=WARNING, 50=INFO, 60=DEBUG, 70=TRACE>
  --sdk-log-level                   Set the SDK (numeric) log level for the program <10=DISABLE, 20=CRITICAL, 30=ERROR, 40=WARNING, 50=INFO, 60=DEBUG, 70=TRACE>
 
Program Flags:
  --file-name                       File name on which the pipeline information will be saved
  -f, --cfg-file                    JSON configuration file
        

Supported subcommands:

• export – This command allows the tool to export a running DOCA Flow program's pipeline into a JSON file. This file is the main input for other features of the tool, such as the graphical visualization.

  Note

  The export subcommand is currently mandatory.

CLI Examples

• To launch the analyze command without a configuration file:

  Copy

  Copied!

              
  
              
  doca_flow_tune analyze export
          

  The JSON file is stored into its default path.

• To launch the analyze command with a given configuration file that specifies the desired values for all needed configurations:

  Copy

  Copied!

              
  
              
  doca_flow_tune analyze export -f /tmp/flow_tune_cfg.json
          

• To launch the analyze command with a configuration file while also configuring the output path for the exported JSON file through the CLI:

  Copy

  Copied!

              
  
              
  doca_flow_tune analyze export -f /tmp/flow_tune_cfg.json --file-name my_program_pipeline_desc.json
          

  The exported pipeline is stored as my_program_pipeline_desc.json into the chosen/default output directory.

Visualize Command

The visualize command visualizes the steering pipeline of a given DOCA Flow program. The command works on a given JSON file as input. This file can either be generated by the analyze export command or queried dynamically from a running program, in which case the command would dump the pipeline from the program and then generate the visualization output file.

            

            
Usage: doca_flow_tune visualize [DOCA Flags] [Program Flags]
 
DOCA Flags:
  -h, --help                        Print a help synopsis
  -l, --log-level                   Set the (numeric) log level for the program <10=DISABLE, 20=CRITICAL, 30=ERROR, 40=WARNING, 50=INFO, 60=DEBUG, 70=TRACE>
  --sdk-log-level                   Set the SDK (numeric) log level for the program <10=DISABLE, 20=CRITICAL, 30=ERROR, 40=WARNING, 50=INFO, 60=DEBUG, 70=TRACE>
 
Program Flags:
  --pipeline-desc                   Input JSON file that represents the Flow application pipeline
  --file-name                       File name on which the visualization information will be saved
  -f, --cfg-file                    JSON configuration file
  --cost                            Show cost
  --critical-path                   Show critical path
  --resources                       Show resources
        

CLI Examples

• Launching the visualize command without the configuration file leads to a live query of the pipeline against the running DOCA Flow program. Since no input JSON file was provided, it automatically dumps a JSON file of the running DOCA Flow program to use for the mermaid:

  Copy

  Copied!

              
  
              
  doca_flow_tune visualize
          

  Example output:

  Copy

  Copied!

              
  
              
  2025-04-09 15:06:44 - flow_tune - INFO - Pipeline description file not provided or doesn't exist, dumping pipeline description to /tmp/tmp_pipeline_desc.json
  2025-04-09 15:06:44 - flow_tune - INFO - Flow program pipeline information was exported to "/tmp/tmp_pipeline_desc.json"
  2025-04-09 15:06:44 - flow_tune - INFO - Mermaid graph exported to /tmp/flow_tune/flow_tune_pipeline_vis.md
  2025-04-09 15:06:44 - flow_tune - INFO - DOCA Flow Tune finished successfully
          

• Launching the visualize command with a given configuration file that specifies the desired values for all needed configurations:

  Copy

  Copied!

              
  
              
  doca_flow_tune visualize -f /tmp/flow_tune_cfg.json
          

• To launch the visualize command with a configuration file while configuring the output path for the Mermaid file through the CLI and providing an offline pipeline file:

  Copy

  Copied!

              
  
              
  doca_flow_tune visualize -f /tmp/flow_tune_cfg.json --file-name my_program_pipeline_viz.md --pipeline-desc my_program_pipeline_desc.json
          

  The exported Mermaid file is stored as my_program_pipeline_viz.md into the chosen/default output directory. Because the pipeline description file is explicitly provided, this command could be used offline, as it would not need connection with the DOCA Flow program to visualize.

  Note

  To launch the visualize command with the --critical-path or --cost flags, "low_level_info" is required to be true in the configuration file provided:

  Copy

  Copied!

              
  
              
  doca_flow_tune visualize --cost -f /tmp/flow_tune_cfg.json # flow_tune_cfg.json has "low_level_info": true in the analyze section
          

  or that the offline pipeline JSON file was dumped with "low_level_info" true :\

  Copy

  Copied!

              
  
              
  doca_flow_tune visualize --cost --pipeline-desc my_program_pipeline_desc.json # my_program_pipeline_desc.json was dumped with "low_level_info": true
          

DOCA Flow Tune has a configuration file which allows customizing various settings.

Config File Default Values

If a configuration file is not provided, DOCA Flow Tune uses its default values for fields which are mandatory.

In Monitor mode, if a software KPIs or hardware counters query is not needed, removing the hardware or software fields from the configuration file disables the respective feature.

Custom Config File

Instead of using default configuration values, users can create a file of their own and provide a file path when running DOCA Flow Tune (-f/--cfg-file).

Once used, DOCA Flow Tune loads all provided values directly from the file, while the rest of the fields (if any) use their respective default values.

Overriding Config Values from CLI

Setting s ome of the fields in the configuration file is supported through CLI using the --file-name flag. If used, the provided values from the CLI would override the values of the fields from the configuration file. This allows for easier configuration of common values without the need to create a new custom file or to modify an existing configuration file.

Common Configuration Values

Some sections of the configuration file are shared between multiple runtime modes of DOCA Flow Tune (i.e., Monitor, Analyze, Visualize) and generally have to do with the output/input file paths and interaction with the live DOCA Flow program.

            

            
{
...  	
 	"outputs_directory": "/tmp/flow_tune/",
...
	"network": {
		"server_uds": "/tmp/tune_server.sock",
		"uds_directory": "/var/run/doca/flow_tune/"
	},
...
}
        

Output Directory

outputs_directory defines the main directory on which all output products are saved. This field does not have a default value. If no value is provided, DOCA Flow Tune files are saved at the following directories:

• CSV file – /var/log/doca/flow_tune/

• Analyze export pipeline description file – /tmp/flow_tune/

• Pipeline visualization file – /tmp/flow_tune/

Connection to DOCA Flow Tune Server

Some features of DOCA Flow Tune work by interacting with a live DOCA Flow based program. This is enabled through a server that is running in the background as part of the DOCA Flow library, and requires all of the following to be applied:

1. DOCA Flow based program should explicitly enable the server.

   Info

   More information is available in the relevant DOCA Flow Tune Server programming guide.

2. The DOCA-Flow-based program should run using the trace-enabled DOCA Flow library.

   Info

   More information is available in the "Debug and Trace Features" section of the DOCA Flow programming guide.

DOCA Flow Tune should be configured in a way to allow it to connect to the matching server. This can be done by modifying the following variables under the network section of the configuration file:

• server_uds – DOCA Tune Server Unix Domain Socket (UDS) path. Default value is /tmp/tune_server.sock.

• uds_directory – Directory on which all local UDSs are created. Default value is /var/run/doca/flow_tune/.

Hardware Counters

This table provides the supported hardware counters and their associated profiles.

Counter Name	Description	Unit	Profile Basic	Profile Full	Notes
RX Packet Rate	The number of received packets per second	pkt/sec
RX Bandwidth	The data transfer rate based on the number of packets received per second	Gb/s
RX Packet Average Size	The average size of received data packets	Bytes
TX Packet Rate	The number of packets transmitted per second	pkt/sec
TX Bandwidth	The data transfer rate based on the number of packets transmitted per second	Gb/s
TX Packet Average Size	The average size of transmitted data packets	Bytes
RX SW Drops	The number of dropped packets due to a lack of WQE for the associated QPs/RQs (excluding hairpin QPs/RQs) Info Supported only on NVIDIA® ConnectX®-7 and above.	drops/sec			If drops are observed, this may be because the software was unable to process all received packets. Consider reducing CPU processing time or increasing the number of utilized cores and queues.
Hairpin Drops	The number of dropped packets due to a lack of WQE for the associated hairpin QPs/RQs Info Supported only on NVIDIA® ConnectX®-7 and above.	drops/sec			If drops are observed, the Tx packet processing is probably causing a bottleneck. Consider simplifying the process or adjusting the number or size of hairpin queues, or implementing locking mechanisms.
RX HW Drops	The number of packets discarded due to no available data or descriptor buffers in the RX buffer	drops/sec			If drops are observed, the Rx packet processing is probably causing a bottleneck. Consider simplifying it.
ICM Cache Miss Rate	The rate of data requests that miss in the ICM (interconnect context memory) cache	events/sec
ICM Cache Miss per Packet	The number of data requests that miss per packet	events/pkt
PCIe Inbound Bandwidth	The number of bits received from the PCIe toward the device per second.	Gb/s			PCIe counters are supported only on the host side
PCIe Outbound Bandwidth	The number of bits transmitted from the device toward the PCIe per second	Gb/s
PCIe AVG Read latency	The average PCIe read latency for all read data	nsec
PCIe Max Latency	T he maximum latency (in nanoseconds) for a single PCIe read from the device	nsec
PCIe Min Latency	T he minimum latency (in nanoseconds) for a single PCIe read from the device	nsec
RX Hops Per Packet	The number of Hops per packet in RX domain	hops/pkt
TX Hops Per Packet	The number of Hops per packet in TX domain	hops/pkt

Software Key Performance Indicators

This table provides the supported software KPIs and their associated profiles.

Action Resources Counters

The actions resources section shows the following counters per port, including the total allocated and how much are in use:

Configuration

CSV Format

The CSV format stores two types of rows, specific to each counter module:

• Hardware Counter Rows (Module ID=0)

  Module ID	HW Counter ID	Counter Value	Timestamp
  0	1	8	142623139459
  0	2	197503959728	142623139459

  • Module ID – Hardware module identifier

  • HW Counter ID – Unique identifier for the hardware counter

  • Counter Value – Counter value

  • Timestamp – Hardware timestamp

    Hardware Counter ID Mapping

• Software KPI Rows (Module ID=1)

  Module ID	Port ID	SW Counter Type	Counter Value	Timestamp
  1	0	Queue 0 Insertion Rate	34511	1727345744137828
  1	1	Queue 0 Insertion Rate	37050	1727345755137828
  1	0	Action Resource 64B Allocated Shared Counters	4	1727345755137828
  1	0	Action Resource 64B Used Shared Counters	3	1727345755137828

  • Module ID – Software module identifier

  • Port ID – Software port ID

  • SW KPI Type – KPI type

  • KPI Value – KPI value

  • Timestamp – Software timestamp

Configuration File

DOCA Flow Tune's configuration file consists of two main parts of relevance for Monitor mode:

• csv dump object

• monitor configuration object

The following is an example for both sections:

            

            
{
...
 	 "csv": {
		"enable": false,
		"file_name": "flow_tune.csv",
		"max_size_bytes": 1000000,
		"max_files": 1
	},
 
...
 
 	 "monitor": {
		"screen_mode": "dark",	// modes: {light, dark}
		"hardware": {
			"pci_addresses": [
				"b1:00.0",
				"b1:00.1"
			],
			"profile": "full"  // profiles: {basic, full}
		},
		"software": [
			{
				"flow_port_id": 0,
				"profiles": [
					"entries_ops_rates"  // profiles: {entries_ops_rates, action_resources}
				]
			},
			{
				"flow_port_id": 1,
				"profiles": [
					"entries_ops_rates"
				]
			}
		]
	}
...
}
        

CSV Configuration Section

CSV dumping allows exporting the hardware and software counters collected by the tool into a CSV file for further analysis or record keeping. This is particularly useful for logging performance metrics over time.

How to Enable CSV Dumping

To enable CSV dumping, modify the configuration in the JSON file as follows:

            

            
{
  "csv": {
    "enable": true,
    "file_name": "flow_tune.csv",
    "max_size_bytes": 1000000,
    "max_files": 1
  }
}
        

The supported fields are:

• enable – Set to true to enable CSV dumping or false to disable it. Default value is false.

• file_name – The name of the CSV file where the data will be saved.

• max_size_bytes – The maximum size (in bytes) of the CSV file. Once this limit is reached, a new file is created based on the max_files setting.

• max_files – The maximum number of CSV files to keep. When this limit is reached, the oldest files are deleted.

CSV dumping can also be enabled or disabled from the CLI using the --enable-csv or --disable-csv flags, respectively. For example:

            

            
doca_flow_tune monitor -f /tmp/flow_tune_cfg.json --enable-csv
        

Additionally, the CSV filename can be updated by using the --csv-file-name flag, for example:

            

            
doca_flow_tune monitor -f /tmp/flow_tune_cfg.json --csv-file-name "counters_dump.csv"
        

Monitor Configuration Section

Screen Mode

The Monitor module supports two screen modes: dark and light.

Hardware

The hardware section includes the pci_addresses and profile fields:

• The pci_addresses field expects an array of PCIe addresses for NIC ports. The tool uses these addresses to retrieve the corresponding NIC device and the desired port IDs.

  Note

  PCIe addresses must belong to the same device.

  Info

  The tool supports up to two ports per device.

• The profile field expects to receive either a basic or full profile.

  • basic profile – includes packet- and port-related counters (i.e., Bandwidth, Packets Per Second, Average Packet Size, Packet Drops)

  • full profile – includes all the basic counters and adds additional debug counters (e.g., ICMC and PCIe counters)

    Info

    For more information about the counters please refer to section "Hardware Counters".

The hardware counters profile can be set from the CLI by adding --hw-profile. For example:

            

            
doca_flow_tune monitor -f /tmp/flow_tune_cfg.json --hw-profile basic
        

Software

The software section includes the flow_port_id and profiles fields:

• flow_port_id field – expects a single DOCA Flow port identification number. Flow port ID should be set by the DOCA Flow program, by calling the doca_flow_port_cfg_set_devargs() API call with a proper port ID string.

• profiles field – expects to receive one or more supported profiles.

  • entries_ops_rates profile – includes both insertion and deletion rates KPIs

  • action_resources profile - includes data on the counters, meters, and actions memory that were allocated and are in use.

The software profile can be set from the CLI by adding --sw-profile, for example:

            

            
doca_flow_tune monitor -f /tmp/flow_tune_cfg.json --sw-profile entries_ops_rates
        

Analyze mode gathers (and later analyzes) information in order to assist users to better understand and debug their DOCA-Flow-based program.

Pipeline Export

This tool export an internal state of the DOCA-Flow-based program in a proprietary JSON format. This allows the tool to provide offline information about a given program which can be later be analyzed. One such example is the ability to visualize the pipeline of the target program without having said program run on real hardware.

While the pipeline export operation is meant to encode all relevant information for future analysis, the format itself is proprietary and is only meant to be consumed by other DOCA tools.

Supported Pipe types:

• Basic

• Control

• Hash

• CT (partially supported):

  • Pipe level FWD and FWD miss.

  • Basic data

Viewing the Pipeline

After running DOCA Flow Tune tool with the visualize command, an output Mermaid file is generated. The content of the file is the raw format for the markdown markup language (known as "Mermaid" format). This is a widely supported industry standard for visualization purposes. An example for an online graphical editor for the Mermaid format is the Mermaid Live website.

Visualization Layers

The following diagrams were taken from the DOCA IPsec Security Gateway reference application.

Base Layer

Base layer is the default layer in case no other layer was provided by the user. Layer responsibility is to present on the Mermaid pipeline subgraph nodes and links important data that was provided to the DOCA Flow library through its API.

Pipe node data:

• Basic pipe attributes

  • Name

  • Label

  • Domain

  • Type

  • Root or not

• Pipe masking.

• Pipe actions.

• Pipe monitor resources.

Pipe link (arch) data:

• In case of Control pipe, each link contains the match value.

• For all other pipes, in case of FWD_MISS link an indication will appear.

Resources Layer

Resources layer is the additional layer on top of base layer, can enable it through visualization command line with --resources flag.

This layer focuses on the following resources:

• Counters

• Meters

• Actions memory

• Number of entries

Pipe node data:

• For all pipe types will have, the total actions in use by the entries

• Beside control pipe, other pipes include the resources per action

Cost Layer

Cost layer is the additional layer on top of base layer, can enable it through visualization command line with --cost flag.

The cost's layer objective is to approximate that number of cache misses a packet will incur while being processed in a pipe. It includes for each pipe the minimum and maximum number of cache misses.

Critical Path Layer

Critical path layer highlights the most critical path. A path is defined as most critical if it has the most cache misses. If more than one path has the most cache misses, a random one is highlighted.

Known Issues

• In some cases, control pipe links match data is missing.

• In some cases, forward miss links match data is missing.

• Match values for the following fields are not supported:

  • outer.l2_type

  • outer_l3_type

  • outer_l4_type

  • tun.gre.protocol

  • tun.key_present

• In some cases, when visualizing with critical path, multiple or broken paths appear.

Actions Memory Calculation

DOCA Flow API require setting the max action memory that will be allocated using doca_flow_port_cfg_set_actions_mem_size() function.

Application can get the best number to set by using one of the following strategies :

Measured Strategy:

In this strategy the application is configured with the max action memory supported by the system.

While the application is running with the expected load, the tool should be run in Monitor mode.

The final stage is to replace the value of actions memory configured in the application with the used actions memory that was reported by the tool, and that matches the requirements from DOCA Flow's respective API.

Calculated Strategy:

In this strategy the application is configured with small action memory size.

After the application created the pipes and before entries are added, the tool should be run in Visualization mode with resource layer.

Now the developer should calculate the need value by calculating the expected action memory per pipe by multiply the number of entries expected

in this pipe with the actions memory reported per entry, then set the action memory to the sum of memory from all the pipes.

Counter Observation

By running the tool in monitor mode, can observe hardware/software counters and view their values per second. This shows real-time performance metrics and helps in identifying bottlenecks.

Program Pipeline Visualization

By running the tool in visualization mode, can get a draw for DOCA Flow program pipeline, which allows debugging the code during development. This helps in understanding the flow of execution, identifying issues, and optimizing the code effectively.

Identifying "bottle neck" Pipes

By running the tool in visualization mode with --cost and --critical-path, we can get a see the pipes in the program that are slowing down the packet processing, and think about how to minimize them.

Telemetry fwctl driver is not loaded

Error

When running the DOCA Flow Tune in Monitor mode, the following log messages are encountered at startup:

            

            
[DOCA][WRN][priv_doca_telemetry_fwctl.cpp:121][priv_doca_telemetry_fwctl_find_device_by_pci] Failed finding fwctl device: Opening directory /sys/class/fwctl/ failed. Make sure you have the fwctl driver loaded
[DOCA][ERR][priv_doca_telemetry_fwctl.cpp:201][priv_doca_telemetry_fwctl_open_by_devinfo] devinfo 0x55c572286520: Failed to open fwctl device: Failed to find matching fwctl device
        

Solution

The DOCA Telemetry SDK uses the fwctl driver to query the hardware counters, so it is essential to have it installed and loaded.

Step 1: Verify the Driver Installation

First, check if the driver is installed as follow:

• Debian/Ubuntu:

  Copy

  Copied!

              
  
              
  $ sudo apt list --installed | grep fwctl 
          

• RHEL:

  Copy

  Copied!

              
  
              
  $ sudo yum list installed | grep fwctl 
          

If the driver is not installed, install it by running the following commands:

• Debian/Ubuntu:

  Copy

  Copied!

              
  
              
  $ sudo apt search fwctl 
  >> <fwctl-package-name>/....
   
  $ sudo apt install -y <fwctl-package-name>
          

• RHEL:

  Copy

  Copied!

              
  
              
  $ sudo yum search fwctl 
  >> <fwctl-package-name>/....
   
  $ apt/yum install -y <fwctl-package-name>
          

Step 2: Check if the Driver is Loaded

After installing the driver, verify that it is loaded by executing:

            

            
$ sudo lsmod | grep fwctl
        

You should see output similar to:

            

            
> mlx5_fwctl 20480 0
> fwctl 16384 1 mlx5_fwctl
> mlx5_core 2134016 2 mlx5_fwctl,mlx5_ib
> mlx_compat 69632 14 rdma_cm,ib_ipoib,mlxdevm,mlxfw,mlx5_fwctl,iw_cm,ib_umad,fwctl,ib_core,rdma_ucm,ib_uverbs,mlx5_ib,ib_cm,mlx5_core
        

If the driver is not loaded, load it by running:

            

            
$ sudo modprobe mlx5_fwctl
        

Mermaid visualization in Visual Studio Code

Visual Studio Code provides extensions to view Mermaid markdown format, these extensions can be used to view the Mermaid output from DOCA Flow Tune tool.

However, for these extension to work, the Mermaid file should be modified with Mermaid opening and closure lines as follows:

            

            
```mermaid
<original_mermaid_file_content>
```
        

Limited feature set – could not detect a running DOCA Flow program

Error

When running DOCA Flow Tune, the following log message is encountered at startup, followed by some features failing to work/load:

            

            
[DOCA][WRN][flow_tune.cpp:195][get_flow_app_data] Could not detect a running DOCA Flow program, some features will be impacted
        

Solution

Some features of DOCA Flow Tune work by interacting with a live DOCA-Flow-based program. This is enabled through a server running in the background as part of the DOCA Flow library, and requires all of the following to be applied:

• DOCA-Flow-based program should explicitly enable the server. More information is available in the DOCA Flow Tune Server programming guide.

• DOCA-Flow-based program should run using the "trace enabled" DOCA Flow library. More information is available in the "Debug and Trace Features" section of the DOCA Flow programming guide.\

visualize --cost or --critical-path isn't working

Error

When running DOCA Flow Tune visualize mode with --cost or --critical-path, the following error message is encountered, and the tool exists with error.

            

            
2025-04-14 12:24:20 - flow_tune - ERROR - Low level info need to be enabled to visualize cost or critical path
2025-04-14 12:24:20 - flow_tune - ERROR - Failed to export mermaid graph
2025-04-14 12:24:20 - flow_tune - ERROR - DOCA Flow Tune failed during "visualize" command: Invalid input
2025-04-14 12:24:20 - flow_tune - ERROR - DOCA Flow Tune finished with errors
        

Solution

To use --cost or --crtitical-path, "low_level_info" is required to be true in the configuration file provided:

            

            
doca_flow_tune visualize --cost -f /tmp/flow_tune_cfg.json # flow_tune_cfg.json has "low_level_info": true in the analyze section
        

or that the offline pipeline JSON file was dumped with "low_level_info" true :

            

            
doca_flow_tune visualize --cost --pipeline-desc my_program_pipeline_desc.json # my_program_pipeline_desc.json was dumped with "low_level_info": true
        

            

            
{
	"outputs_directory": "/tmp/flow_tune/",
    "logging": {
		"developer_log": "/var/log/doca/flow_tune/flow_tune_dev.log",
		"operational_log": "/var/log/doca/flow_tune/flow_tune.log"
	},
	"network": {
		"server_uds": "/tmp/tune_server.sock",
		"uds_directory": "/var/run/doca/flow_tune/"
	},
	"csv": {
		"enable": false,
		"file_name": "flow_tune.csv",
		"max_size_bytes": 1000000000,
		"max_files": 1
	},
	"analyze": {
		"file_name": "flow_tune_pipeline_desc.json",
	    "low_level_info": false
	},
	"visualize": {
		"pipeline_desc_file": "/tmp/flow_tune/flow_tune_pipeline_desc.json",  // Non-mandatory field
		"file_name": "flow_tune_pipeline_vis.md"
	},
	"monitor": {
		"screen_mode": "light",
		"hardware": {
			"pci_addresses": [
				"08:00.0",
				"08:00.1"
			],
			"profile": "full"
		},
		"software": [
			{
				"flow_port_id": 0,
				"profiles": [
					"entries_ops_rates",
					"action_resources"
				]
			},
			{
				"flow_port_id": 1,
				"profiles": [
					"entries_ops_rates",
					"action_resources"
				]
			}
		]
	}
}
        

Where:

• outputs_directory – Main directory on which all output products are saved. This field does not have a default value. If no value is provided, DOCA Flow Tune files are saved at the following directories:

  • CSV file – /var/log/doca/flow_tune/

  • Analyze export pipeline description file – /tmp/flow_tune/

  • Pipeline visualization file – /tmp/flow_tune/

• logging - Log files directory.

  • developer_log - developer level log file.

  • operational_log - user level log file.

• network

  • server_uds – DOCA Tune Server Unix Domain Socket (UDS) path. Default value is /tmp/tune_server.sock.

  • uds_directory – Directory on which all local UDS is created. Default value is /var/run/doca/flow_tune/.

• csv

  • enable – true if information should be saved into a CSV file. Default value is false.

  • file_name – CSV filename. Default value is flow_tune.csv.

  • max_size_bytes – CSV file maximum size in bytes. When the limit is reached, a new file is created. Default value is 1Gb.

  • max_files – Maximum CSV files to create. Default value is 1.

• analyze

  • file_name – Flow program pipeline description filename. File is created under outputs_directory path. Default value is flow_tune_pipeline_desc.json.

  • low_level_info - Pipeline description JSON file to include low level information.

• visualize

  • pipeline_desc_file – Flow program pipeline description input file path. This file is the product of the analyze export command. If not provided, then the visualize command triggers a creation of it (a run of analyze export).

  • file_name – Flow program pipeline visualization filename. File is created under the outputs_directory path. Default value is flow_tune_pipeline_vis.md.

• monitor

  • screen_mode – Monitor command theme to be used. Default value is light.

  • hardware

    • pci_addresses – List of PCIe addresses which DOCA Flow Tune should inspect.

    • profile – Hardware profile to be used for each PCIe address given. Default value is full.

  • software

    • flow_port_id – Flow program port identification number which DOCA Flow Tune should inspect.

    • profiles – List of software profiles to be used for the specific port identification number given. Default value is [entries_ops_rates].