Note: Before proceeding with this section, make sure to follow the procedure in Enabling IPsec Full Offload for both DPUs.

This section configures OVS IPsec VXLAN tunnel which automatically generates the swanctl.conf files and runs strongSwan (the IPsec daemon). The following figure illustrates an example with two BlueField DPUs, Left and Right, operating with a secured VXLAN channel.

Two BlueField DPUs are required to build an OVS IPsec tunnel between the two hosts, Right and Left.

The OVS IPsec tunnel configures an unaware IPsec connection between the two hosts' InfiniBand devices. For the sake of this example, the host's InfiniBand network device is HOST_PF, and the DPU's host representor is PF_REP and the DPU's physical function PF.

# host_ip1=1.1.1.1
# host_ip2=1.1.1.2
# HOST_PF=ens7np0
# ip1=192.168.50.1
# ip2=192.168.50.2
# PF=p0
# PF_REP=pf0hpf

# ibdev2netdev 
mlx5_0 port 1 ==> ens7np0 (Down)
mlx5_1 port 1 ==> ens8np1 (Down)

1. Configure IP addresses for the HOST_PFs of both hosts (x86):
   1. On host_1:

      # ifconfig $HOST_PF $host_ip1/24 up

   2. On host_2:

      # ifconfig $HOST_PF $host_ip2/24 up

      Note: Step 1 is the only command that is performed on the host, the rest of the commands are performed on the Arm (DPU) side.
2. Configure IP addresses for the PFs of both Arms:
   1. On Arm_1:

      # ifconfig $PF $ip1/24 up

   2. On Arm_2:

      # ifconfig $PF $ip2/24 up

3. Enable TC offloading for the PF. Run the following on both Arm_1 and Arm_2:

   # ethtool -K $PF hw-tc-offload on

4. Disable host PF as the port owner from Arm. Run the following on both Arm_1 and Arm_2:

   # mlxprivhost -d /dev/mst/mt${pciconf} --disable_port_owner r

   Note: To get ${pciconf}, run the following on the Arm:

   # ls --color=never /dev/mst/ | grep --color=never '^m.*f0$' | cut -c 3-

   For example:

   # mlxprivhost -d /dev/mst/mt41686_pciconf0 --disable_port_owner r

5. Start Open vSwitch. If your operating system is Ubuntu, run the following on both Arm_1 and Arm_2:

   # service openvswitch-switch start

   If your operating system is CentOS, run the following on both Arm_1 and Arm_2:

   # service openvswitch restart

6. Start OVS IPsec service. Run the following on both Arm_1 and Arm_2:

   # systemctl start openvswitch-ipsec.service

7. Set up OVS bridges in both DPUs. Run the following on both Arm_1 and Arm_2:

   # ovs-vsctl add-br vxlan-br
   # ovs-vsctl add-port ovs-br $PF_REP
   # ovs-vsctl set Open_vSwitch . other_config:hw-offload=true

   Note: Configuring other_config:hw-offload=true sets IPsec full offload. Setting it to false sets software IPsec. Make sure that IPsec devlink's mode is set back to none for software IPsec. This is done by reverting the configurations in Enabling IPsec Full Offload.
   Note: The MTU of the tunnel interface (PF) should be at least 50 bytes larger than the MTU of the endpoints of the tunnels above (PF_REP) to account for the size of the VXLAN tunnel header. For example, if the MTU of PF_REP is 1500 then the MTU of PF should at least be 1550. To configure the MTU of the PF:

   # ifconfig $PF mtu $PF_MTU up

8. Set up IPsec tunnel on the OVS bridge.
   Three authentication methods are possible. Select your preferred method and follow the steps relevant to it. Note that the last two authentication methods require you to create certificates (self-signed or certificate authority certificates).

   Note: After the IPsec tunnel is set up using one of the three methods of authentication, strongSwan configuration is done automatically and the swanctl.conf files are generated and strongSwan runs automatically.

Now the IPsec connection should be established.

strongSwan configures IPSec HW full offload using a new value added to its configuration file swanctl.conf. The file should be placed under sysconfdir which by default can be found at /etc/swanctl/swanctl.conf.

In this example, 192.168.50.1 is used for the left PF uplink and 192.168.50.2 for the right PF uplink.

connections {
   BFL-BFR {
      local_addrs  = 192.168.50.1
      remote_addrs = 192.168.50.2
 
      local {
        auth = psk
        id = host1
      }
      remote {
        auth = psk
        id = host2
      }
 
      children {
         bf {
            local_ts = 192.168.50.1/24 [udp/4789]
            remote_ts = 192.168.50.2/24 [udp/4789]
            esp_proposals = aes128gcm128-x25519
            mode = transport
            policies_fwd_out = yes
            hw_offload = full
         }
      }
      version = 2
      mobike = no
      reauth_time = 0
      proposals = aes128-sha256-x25519
   }
}
 
secrets {
   ike-BF {
      id-host1 = host1
      id-host2 = host2
      secret = 0sv+NkxY9LLZvwj4qCC2o/gGrWDF2d21jL
   }
   }

The BFB installation will place two example swanctl.conf files for BFL and BFR (BFL.swanctl.conf and BFR.swanctl.conf respectively) in the strongSwan conf.d directory. Each node should have only one swanctl.conf file in its strongSwan conf.d directory.

Note that:

• "hw_offload = full" is responsible for configuring IPSec HW full offload
• Full offload support has been added to the existing hw_offload field and preserves backward compatibility.

  For your reference:

  Value	Description
  no	Do not configure HW offload, fail if not supported
  yes	Configure crypto HW offload if supported by the kernel, fail if not supported (Existing)
  auto	Configure crypto HW offload if supported by the kernel, do not fail (Existing)
  full	Configure full HW offload if supported by the kernel, fail if not supported (New)

• Whenever the value of hw_offload is changed, strongSwan configuration must be reloaded.
• Switching to crypto HW offload requires setting up devlink/ipsec_mode to none beforehand.
• Switching to full HW offload requires setting up devlink/ipsec_mode to full beforehand.
• [udp/4789] is crucial for instructing strongSwan to IPSec only VXLAN communication.
• Full HW offload can only be done on what is streamed over VXLAN.

This subsection explains how to configure and set an IPsec connection using the script. To configure the IPsec connection, you need two DPUs, referred to as the initiator and receiver machines. There are no differences between the two machines except that the initiator is the one that initiates the connection between the two (and should run the script after the receiver).

The script is located under /opt/mellanox/doca/applications/east_west_overlay_encryption/bin/east_west_overlay_encryption.sh.

1. strongSwan IPsec full version can be found here (tag: 5.9.6bf).
2. Install dependencies mentioned here. libgmp-dev is missing from that list, so make sure to install that as well.
3. Git clone https://github.com/Mellanox/strongswan.git.
4. Git checkout BF-5.9.6.
5. Run autogen.sh within the strongSwan repo.
6. Run the following:

   configure --enable-openssl --disable-random --prefix=/usr/local --sysconfdir=/etc --enable-systemd
   make
   make install

   Notes:

   • --enable-systemd enables the systemd service for strongSwan present inside the GitHub repo (see step 3) at init/systemd-starter/strongswan-starter.service.in. This service file is meant for Ubuntu, Debian and Yocto distributions. For CentOS, the contents of the above file must be replaced by the one present in systemd-conf/strongswan-starter.service.in.centos (inside the GitHub repo) before running the configure script above.
   • When building strongSwan on your own, the openssl.cnf.mlnx file, required for PK and RNG HW offload via OpenSSL plugin, is not installed. It must be copied over manually from GitHub repo inside the openssl-conf directory. See section "Running Strongswan Example" for important notes.
   • The openssl.cnf.mlnx file references PKA engine shared objects. libpka (version 1.3 or later) and openssl (version 1.1.1) must be installed for this to work.