In the handshake phase, the client and server decide on which cipher suites they will use, and exchange keys and certificates according to the following flow:

1. Client hello, provides the server at a minimum with the following:
   • A key exchange algorithm, to determine how symmetric keys are exchanged
   • An authentication or digital signature algorithm, which dictates how server authentication and client authentication (if required) are implemented
   • A bulk encryption cipher, which is used to encrypt the data
   • A hash/MAC (message authentication code) function, which determines how data integrity checks are carried out
   • The version of the protocol it understands
   • The cipher suites it is capable of working with
   • A unique random number, which is important to guard against replay attacks
2. Server hello:
   • Selects a cipher suite
   • Generates its own random number
   • Assigns a session ID to the TLS connection
   • Sends enough information to complete a key exchange—most often, this means sending a certificate including an RSA public key
3. Client:
   • Responsible for completing the key exchange using the information the server provided

kTLS has 3 modes of operation:

• SW – all operation is handled in kernel (i.e., handshake, encryption, decryption)
• HW-offload (the focus of this guide) – handshake and error handling are performed in software. Packets are encrypted/decrypted in hardware. In this case, there is an additional offload from the kernel to the hardware.
• HW-record – all operations are handled by the hardware (driver and firmware) including the handshake. It also handles its own TCP session. This option is currently not supported.

It is necessary to learn the following terms before proceeding:

• The transport interface send (TIS) object is responsible for performing all transport-related operations of the transmit side. Messages from Send Queues (SQs) get segmented and transmitted by the TIS including all transport required implications. For example, in the case of a large send offload, the TIS is responsible for the segmentation. The ConnectX HW uses a TIS object to save and access the TLS crypto information and state of an offloaded Tx kTLS connection.
• The transport interface receive (TIR) object is responsible for performing all transport-related operations on the receive side. TIR performs the packet processing and reassembly and is also responsible for demultiplexing the packets into different receive queues (RQs).
• Both TIS and TIR hold the data encryption key (DEK).

To check if the BlueField or ConnectX have crypto acceleration, run the following command from host:

host> mst start # turn on mst driver
host> flint -d <device under /dev/mst/ directory> dc | grep Crypto

The output should include Crypto Enabled. For example:

host> flint -d  /dev/mst/mt41686_pciconf0 dc | grep Crypto
....
;;Description = NVIDIA BlueField-2 E-Series Eng. sample DPU; 200GbE single-port QSFP56; PCIe Gen4 x16; Secure Boot Disabled; Crypto Enabled; 16GB on-board DDR; 1GbE OOB management
....

Operating system must be either:

• FreeBSD 13.0+.
• A Linux distribution built on Linux kernel version 5.3 or later for Tx support and version 5.9 or later for Rx support. We recommend using the latest version when possible for the best available optimizations.

To check the current kernel version on the host, run:

host> uname -r

The kernel must be configured to support TLS. The options TLS_DEVICE and MLX5_TLS must be set to y. To check if it is configured, run:

host> cat /boot/config-$(uname -r) | grep TLS

Example output:

host> cat /boot/config-5.4.0-121-generic | grep TLS
...
CONFIG_TLS_DEVICE=y
CONFIG_MLX5_TLS=y
...

If the current kernel does not support one of the settings, you may want to change configurations and recompile, or build a new kernel.

Schematic flow for building a Linux kernel:

# Enter Linux kernel directory downloaded (usually in /usr/src/)
host> make menuconfig # in options set TLS_DEVICE=y, MLX5_TLS=y. Setting location in the menu, can be found by searching the setting- pressing '/' and typing setting.
host> make -j <num of cores> && make -j <num of cores> modules_install && make -j <num of cores> install
# Update grub to new configured kernel and than reboot

host> mst start      # If mst driver is not loaded
host> mst status -v

NVIDIA's netdev interfaces are found be under the NET column.

For example:

host> mst status -v
....
DEVICE_TYPE             MST                           PCI       RDMA            NET                       NUMA
BlueField2(rev:0)       /dev/mst/mt41686_pciconf0.1   b1:00.1   mlx5_1          net-ens5f1                1
 
BlueField2(rev:0)       /dev/mst/mt41686_pciconf0     b1:00.0   mlx5_0          net-ens5f0                1 

In this example, the interfaces ens5f1 and ens5f0 are NVIDIA's netdev interfaces.

To check if the offload option is on or off, run:

host> ethtool -k $iface | grep tls

Example output:

tls-hw-tx-offload: on
tls-hw-rx-offload: off
tls-hw-record: off [fixed]

To turn Tx offload on or off:

host> ethtool -K $iface tls-hw-tx-offload <on | off>

To turn Rx offload on or off:

host> ethtool -K $iface tls-hw-rx-offload <on | off>

When the host is connected to a BlueField device, an OVS bridge must be configured on the BlueField so traffic passes bidirectionally from host to uplink. If no OVS bridge is configured, the host is isolated from the network (see diagram above).

To configure the OVS bridge on BlueField, run the following commands on BlueField:

dpu> for br in $(ovs-vsctl list-br); do ovs-vsctl del-br $br; done           # erasing existing bridges
dpu> ovs-vsctl add-br ovs-br0 && ovs-vsctl add-port ovs-br0 p0 && ovs-vsctl add-port ovs-br0 pf0hpf
dpu> ovs-vsctl add-br ovs-br1 && ovs-vsctl add-port ovs-br1 p1 && ovs-vsctl add-port ovs-br1 pf1hpf
dpu> ovs-vsctl set Open_vSwitch . other_config:hw-offload=true && systemctl restart openvswitch-switch

Where p0/p1 are the uplink interfaces and pf0hpf/pf1hpf are the interfaces facing the host.

OpenSSL is an all-around cryptography library that offers open-source application of the TLS protocol. It is the main library for using kTLS and other applications since Nginx depends on it as their base library.

kTLS is supported only in OpenSSL version 3.0.0 or higher, and only on the supported kernel versions. The supported OpenSSL version is available for download from distro packages, or it can be downloaded and compiled from the OpenSSL GitHub.

Check the version of the default OpenSSL:

host> openssl version

Follow OpenSSL installation instructions from their supplied guides. During the configuration process, make sure to configure the enable-ktls option before building it by running from within OpenSSL directory (works in version 3.0 and higher). For example:

host> ./Configure linux-$(uname -p) enable-ktls --prefix=/var/tmp/ssl --openssldir=/var/tmp/ssl # add "threads" as well for multithread support

Check if kTLS is enabled in OpenSSL by running the following command from within the OpenSSL directory, and check whether ktls is listed under Enabled features:

host> perl configdata.pm --dump | less

If OpenSSL has been downloaded manually, the OpenSSL executable would be located in the /<openssl-dir>/apps/ directory. For example, checking the version (from within OpenSSL directory) is done using the command ./apps/openssl version.

The NVIDIA accelerated IO (XLIO) software library boosts the performance of TCP/IP applications based on Nginx (e.g., CDN, DoH) and storage solutions as part of SPDK. XLIO is a user-space software library that exposes standard socket APIs with kernel-bypass architecture, enabling a hardware-based direct copy between an application's user-space memory and the network interface. In particular, XLIO can boost the performance of applications that use the kTLS hardware offload as OpenSSL and Nginx. Read more about XLIO in the NVIDIA XLIO Documentation and XLIO TLS HW-offload over kTLS in the TLS HW Offload section.

The server side should create a certificate and key. The client can also use a certificate, but it is not necessary for this test case. Run the following command in the installed OpenSSL directory and fill in all the requested details:

host> openssl req -x509 -newkey rsa:2048 -keyout key.pem -out cert.pem -days 365 -nodes

The following files are created:

• key.pem – private-key file used to generate the CSR and, later, to secure and verify connections using the certificate
• cert.pem – certificate signing request (CSR) file used to order your SSL certificate and, later, to encrypt messages that only its corresponding private key can decrypt

host> openssl s_server -key key.pem -cert cert.pem -tls1_2 -cipher ECDHE-RSA-AES128-GCM-SHA256 -accept 443 -ktls

host> openssl s_client -connect 4.4.4.4:443 -tls1_2 

Where 4.4.4.4 is the IP of the remote server.

The following example checks kTLS hardware offload on the tested setup by tracking Rx and Tx TLS on device counters:

host> ethtool -S $iface | grep -i 'tx_tls_encrypted\|rx_tls_decrypted' # ($iface is the interface that offloads)

To check kTLS over kernel counters:

host> cat /proc/net/tls_stat

Output example:

host> cat /proc/net/tls_stat
TlsCurrTxSw                             0               # Current Tx connections opened in SW mode
TlsCurrRxSw                             0               # Current Rx connections opened in SW mode
TlsCurrTxDevice                         0               # Current Tx connections opened in HW-offload mode
TlsCurrRxDevice                         0               # Current Rx connections opened in HW-offload mode
TlsTxSw                                 2323828         # Accumulated number of Tx connections opened in SW mode
TlsRxSw                                 1               # Accumulated number of Rx connections opened in SW mode
TlsTxDevice                             12203652        # Accumulated number of Tx connections opened in HW-offload mode
TlsRxDevice                             0               # Accumulated number of Rx connections opened in HW-offload mode
TlsDecryptError                         0               # Failed record decryption (e.g., due to incorrect authentication tag)
TlsRxDeviceResync                       0               # Rx resyncs sent to HW's handling cryptography
TlsDecryptRetry                         0               # All Rx records re-decrypted due to TLS_RX_EXPECT_NO_PAD misprediction
TlsRxNoPadViolation                     0               # Data Rx records re-decrypted due to TLS_RX_EXPECT_NO_PAD misprediction