DOCA Perftest

DOCA Perftest is an RDMA benchmarking tool designed for compute clusters, enabling fine-tuned evaluation of bandwidth, message rate, and latency across a wide range of RDMA operations.

DOCA Perftest enables evaluation of RDMA performance in both simple client-server setups and large-scale cluster scenarios. It offers fine-grained control over benchmarking parameters and supports diverse RDMA verbs and configurations.

Key features:

• Comprehensive RDMA benchmarking – Supports bandwidth, message rate, and latency measurements for various RDMA verbs

• Unified application – A single tool for all RDMA operations, with broad configurability and CUDA support for GPUDirect RDMA

• Cluster-wide testing – Execute distributed benchmarks from a central host across multiple nodes

• Flexible scenario configuration – Define complex multi-node, multi-benchmark tests using a JSON configuration file

• Command-line simplicity – Quickly execute single-node benchmarks with a straightforward CLI interface

• Synchronized execution – Ensures accurate timing across multi-node and multi-process benchmarks

DOCA Perftest streamlines the performance validation process for RDMA-based applications.

DOCA Perftest relies on the following components, provided with the DOCA SDK:

• rdma-core

• libibverbs

• OpenMPI

• doca-verbs (optional)

• DOCA version 3.0.0 or later

• Optional:

  • Passwordless SSH setup across all servers for multi-node testing

  • CUDA 12.8 or later, required for GPUDirect RDMA benchmarks

DOCA Perftest supports the following usage modes to accommodate both basic and advanced benchmarking needs:

• Simple benchmarks – Use command-line arguments directly on the client and server nodes to perform quick bandwidth or latency tests.

  This mode is ideal for testing individual parameters or small-scale deployments.

• Complex scenarios – Use a JSON input file to define multi-node, multi-benchmark configurations.

  This mode enables synchronized, distributed benchmarking across multiple systems with centralized result aggregation.

Simple Point-To-Point Benchmarks

To run a basic RDMA performance test, execute doca_perftest on both the server and client nodes using command-line parameters.

Example usage:

• Client (requestor):

  Copy

  Copied!

              
  
              
  doca_perftest -N 3 -d mlx5_0 -c UC -v send -m bw -s 1024 -D 10 -q 50 --server_name remote-server-name
          

Parameter breakdown:

• -N 3 – Launches 3 synchronized processes with automatic core selection

  (Use -C to specify cores explicitly.)

• -d mlx5_0 – Uses the local InfiniBand device mlx5_0

• -c UC – Sets the RDMA connection type to Unreliable Connection (UC)

• -v send – Uses the send RDMA verb

• -m bw – Measures bandwidth

• -s 1024 – Sets the message size to 1024 bytes

• -D 10 – Sets the test duration to 10 seconds

• -q 50 – Allocates 50 queue pairs per process

• --server_name remote-server-name – Specifies the remote server hostname

  Info

  If passwordless SSH is configured, the client automatically invokes the server instance. If the server is already running manually, the client detects it and skips the invocation.

The following options are available via -h / --help:

            

            
doca_perftest -h
        

1. The options -N/--num_processes and -C/--cores are mutually exclusive. If neither is set, 1 process is used and a single core is auto-selected. ⤶   ⤶

2. The options -i/--iterations and -D/--duration are mutually exclusive. If neither is set, 1 process is used and a single core is auto-selected. ⤶   ⤶

Complex Scenarios

For advanced benchmarking scenarios involving multiple hosts, users can define tests using a structured JSON input file. Example:

            

            
doca_perftest -in path_to_scenario_file.json
        

Capabilities:

• Automatically deploys and coordinates execution across all defined hosts

• Synchronized test initiation across all nodes

• Collects and aggregates results on the invoking node

Use cases:

• Cluster-wide RDMA performance testing

• Multi-benchmark test suites

• Automation and repeatability of complex test setups

Notes:

• JSON examples can be found in /usr/share/doc/doca-perftest/examples

• It is recommended to start from an example and customize as needed

Bandwidth

Bandwidth tests measure the total data throughput and message-handling efficiency across all active test processes.

Metrics collected:

• Message Rate (Mpps) – Total number of Completion Queue Entries (CQEs) processed per second across all processes. Indicates how efficiently the system handles many small messages.

• Bandwidth (Gb/s) – Total data transfer rate, calculated as bandwidth = message_rate × message_size. Measures sustained throughput.

How it is measured:

• The result is aggregated across all active test processes

• Concurrency is controlled using:

  • -co (CLI)

  • "cores" field (JSON)

• The test duration is averaged across processes for consistency

Interpretation tips:

• High message rate, low bandwidth → Likely using small message sizes

• High bandwidth, moderate message rate → Larger messages or fewer CQEs

• Results help determine network saturation, queue depth tuning, and core scaling

Latency

Latency tests measure the time taken for a message to travel across the network. The direction and scope of the measurement depend on the RDMA verb.

RDMA verb modes:

• Send/Receive – Measures one-way latency (client → server)

• Write – Measures round-trip latency (client → server → client), i.e., ping-pong

Metrics collected:

• Minimum latency – Fastest observed transmission

• Maximum latency – Longest time taken, reflects worst-case delays

• Mean latency – Arithmetic average across all iterations

• Median latency – Middle value; less affected by outliers than the mean

• Standard deviation – Indicates variability; smaller is more consistent

• 99% tail latency – 99% of messages completed within this latency

• 99.9% tail latency – Extreme edge-case analysis; only 0.1% of messages exceed this

How it is measured:

• Uses RDMA verbs in a loop with tightly synchronized messaging

• Time measurements are taken on the sender side for accuracy

• Results are aggregated and reported per test

Interpretation tips:

• Low mean/median, but high max/tail → Potential jitter or queuing delays

• Low standard deviation → Reliable, predictable performance

• Use tail latency (99%, 99.9%) for SLA validation and real-time applications

Important notes:

• DOCA Perftest offers improved Write latency accuracy compared to the legacy perftest tool

• Differences in latency measurement logic may exist between tools—compare results carefully