Full Offload and VDPA
Before following the instructions in this chapter, please read the following chapters in the latest RedHat Network Functions Virtualization Planning and Configuration Guide:
Chapter 7: Planning Your OVS-DPDK Deployment
Chapter 8: Configuring an OVS-DPDK Deployment
Use the ovs-hw-offload.yaml file that is available in the following location:
/usr/share/openstack-tripleo-heat-templates/environments/ovs-hw-offload.yaml
Configure the file over VXLAN setup in the following way:
parameter_defaults: NeutronTunnelTypes:
'vxlan'
NeutronNetworkType: - flat - vxlan NeutronOVSFirewallDriver: openvswitch OvsDpdkDriverType: mlx5_core NovaPCIPassthrough: - devname:"enp3s0f1"
physical_network:null
ComputeOvsDpdkParameters: KernelArgs:"default_hugepagesz=1GB hugepagesz=1G hugepages=16 iommu=pt intel_iommu=on isolcpus=-5,7-11,14-17,19-23"
TunedProfileName:"cpu-partitioning"
IsolCpusList:"2-5,7-11,14-17,19-23"
NovaReservedHostMemory:4096
OvsDpdkSocketMemory:"1024,1024"
OvsDpdkMemoryChannels:"4"
OvsHwOffload: True OvsPmdCoreList:"4,7,8,16,19,20"
NovaComputeCpuDedicatedSet:2
,3
,5
,9
,10
,11
,14
,15
,17
,21
,22
,23
OvsDpdkCoreList:"0,1,6,12,13,18"
NovaComputeCpuSharedSet:0
,1
,6
,12
,13
,18
Configure the interface names in the /usr/share/openstack-tripleo-heat-templates/network/config/single-nic-vlans/control.yaml file, by adding the following code to move the tenant network from VLAN on a bridge to a separated interface:
-type:
interface
name: <interface_name> addresses: -ip_netmask: get_param: TenantIpSubnetConfigure the interface names in the /usr/share/openstack-tripleo-heat-templates/network/config/single-nic-vlans/compute.yaml file, by adding the following code to move the tenant network from VLAN on a bridge to a separated interface:
- type: ovs_user_bridge name: br-link members: - type: ovs_dpdk_port name: dpdk0 driver: mlx5_core members: - type:
interface
name: enp3s0f1 primary:true
use_dhcp:false
addresses: - ip_netmask: get_param: TenantIpSubnet - type: sriov_pf name: enp3s0f1 link_mode: switchdev numvfs:8
use_dhcp:false
Create a new role for the compute node, and change it to ComputeOvsDpdk:
$ openstack overcloud roles generate -o roles_data.yaml Controller ComputeOvsDpdk
Update the ~/cloud-names.yaml file accordingly. You may refer to the following example:
parameter_defaults: ComputeOvsDpdkCount:
2
OvercloudComputeOvsDpdkFlavor: compute ControllerCount:1
OvercloudControllerFlavor: controlAssign the compute.yaml file to the ComputeOvsDpdk role. Update the /usr/share/openstack-tripleo-heat-templates/environments/net-single-nic-with-vlans.yaml file, by adding the following line:
OS::TripleO::ComputeOvsDpdk::Net::SoftwareConfig: ../network/config/single-nic-vlans/compute.yaml
To customize overcloud images with MOFED, run:
$ sudo su
$ yum install -y libguestfs-tools
$ export LIBGUESTFS_BACKEND=direct
$ cd /home/stack/images/
$ wget https://www.mellanox.com/downloads/ofed/MLNX_OFED-5.1-2.3.7.1/MLNX_OFED_LINUX-5.1-2.3.7.1-rhel8.2-x86_64.tgz
$ virt-copy-in -a overcloud-full.qcow2 MLNX_OFED_LINUX-5.1
-2.3
.7.1
-rhel8.2
-x86_64.tgz /tmp
$ virt-customize -v -a overcloud-full.qcow2 --run-command 'yum install pciutils tcl tcsh pkgconf-pkg-config unbound gcc-gfortran make tk -y'
$ virt-customize -v -a overcloud-full.qcow2 --run-command 'cd /tmp && tar -xf MLNX_OFED_LINUX-5.1-2.3.7.1-rhel8.2-x86_64.tgz && rm -rf /tmp/MLNX_OFED_LINUX-5.1-2.3.7.1-rhel8.2-x86_64.tgz'
$ virt-customize -v -a overcloud-full.qcow2 --run-command '/tmp/MLNX_OFED_LINUX-5.1-2.3.7.1-rhel8.2-x86_64/mlnxofedinstall --force --ovs-dpdk'
$ virt-customize -v -a overcloud-full.qcow2 --run-command 'systemctl enable openvswitch'
$ virt-customize -v -a overcloud-full.qcow2 --run-command '/etc/init.d/openibd restart'
$ virt-customize -a overcloud-full.qcow2 --selinux-relabel
For OVS-DPDK with SR-IOV, all VFS must be bound before creating VMS. For this purpose, you may use this patch, in os-net-config. Since, it is not merged yet, apply it manually on the overcloud image, as shown in the following example:
$ cat << EOF > os-net-config-sriov-bind
#!/bin/python3
import
sys
from os_net_config.sriov_bind_config import
main
if
__name__ == "__main__"
:
sys.exit(main())
EOF
$ chmod 755
os-net-config-sriov-bind
$ virt-copy-in -a overcloud-full.qcow2 os-net-config-sriov-bind /usr/bin/
$ virt-customize -v -a overcloud-full.qcow2 --run-command "yum install patch -y "
$ wget https://raw.githubusercontent.com/wmousa/os-net-config-bind/main/os-net-config.patch
$ virt-copy-in -a overcloud-full.qcow2 os-net-config.patch /usr/lib/python3.6
/site-packages/
$ virt-customize -v -a overcloud-full.qcow2 --run-command "cd /usr/lib/python3.6/site-packages/ && patch -p1 < os-net-config.patch"
Deploy the overcloud using the appropriate templates and yaml files from /usr/share/openstack-tripleo-heat-templates/, as shown in the following example:
openstack overcloud deploy\
--templates /usr/share/openstack-tripleo-heat-templates/\
--libvirt-type kvm \
-r /home/stack/roles_data.yaml\
--timeout 240
\
-e /usr/share/openstack-tripleo-heat-templates/environments/disable-telemetry.yaml \
-e /usr/share/openstack-tripleo-heat-templates/environments/podman.yaml\
--validation-warnings-fatal \
-e /home/stack/cloud-names.yaml\
-e /home/stack/overcloud_storage_params.yaml\
-e /home/stack/containers-prepare-parameter.yaml\
-e /usr/share/openstack-tripleo-heat-templates/environments/network-isolation.yaml \
-e /home/stack/network-environment.yaml \
-e /usr/share/openstack-tripleo-heat-templates/environments/net-single-nic-with-vlans.yaml\
-e /home/stack/overcloud-selinux-config.yaml \
-e /usr/share/openstack-tripleo-heat-templates/environments/services/neutron-ovs.yaml\
-e /usr/share/openstack-tripleo-heat-templates/environments/services/neutron-ovs-dpdk.yaml\
-e /usr/share/openstack-tripleo-heat-templates/environments/ovs-hw-offload.yaml\
-e /usr/share/openstack-tripleo-heat-templates/environments/net-single-nic-with-vlans.yaml
Run the following on all compute nodes:
$ podman exec -it -u root nova_compute bash $ git clone https:
//github.com/wmousa/openstack-ovs-dpdk.git
$ yum install patch -y $ cd /usr/lib/python3.6
/site-packages/ $ patch -p1 < /openstack-ovs-dpdk/ussuri/nova.patch $ patch -p1 < /openstack-ovs-dpdk/ussuri/os-vif.patch $ exit $ podman restart nova_compute $ mkdir -p /var/lib/vhost_sockets/ $ chmod775
/var/lib/vhost_sockets/ $ chown qemu:hugetlbfs /var/lib/vhost_sockets/Run the following on all controller nodes:
$ podman exec -it -u root neutron_api bash $ git clone https:
//github.com/mellanox/openstack-ovs-dpdk.git
$ yum install patch -y $ cd /usr/lib/python3.6
/site-packages/ $ patch -p1 < /openstack-ovs-dpdk/ussuri/neutron.patch $ exit
To boot the VM on the undercloud machine, perform the following steps:
Load the overcloudrc configuration:
$ source ./overcloudrc
Create a flavor:
$ openstack flavor create --ram
1024
--vcpus1
--property hw:mem_page_size=1GB --public
dpdk.1gCreate a “cirrios” image:
$ openstack image create --
public
--file cirros-mellanox_eth.img --disk-format qcow2 --container-format bare mellanoxCreate a network:
$ openstack network create
private
--provider-network-type vxlan --shareCreate a subnet:
$ openstack subnet create private_subnet --dhcp --network
private
--subnet-range11.11
.11.0
/24
To boot a VM with an SR-IOV port, use the following command after creating the SR-IOV port accordingly:
For the first VM:
$ direct_port1=`openstack port create direct1 --vnic-type=direct --network
private
--binding-profile'{"capabilities":["switchdev"]}'
| grep' id '
| awk'{print $4}'
` $ openstack server create --flavor dpdk.1g --image mellanox --nic port-id=$direct_port1 vm1For the second VM:
$ direct_port2=`openstack port create direct1 --vnic-type=direct --network
private
--binding-profile'{"capabilities":["switchdev"]}'
| grep' id '
| awk'{print $4}'
` $ openstack server create --flavor dpdk.1g --image mellanox --nic port-id=$direct_port2 vm2
To boot a VM with a VDPA port, use the following command after creating the VDPA port accordingly:
For the first VM:
$ virtio_port0=`openstack port create virtio_port --vnic-type=virtio-forwarder --network
private
| grep' id '
| awk'{print $4}'
` $ openstack server create --flavor dpdk.1g --image mellanox --nic port-id=$virtio_port0 --availability-zone nova:overcloud-computesriov-0
.localdomain vm0For the second VM:
$ virtio_port1=`openstack port create virtio_port --vnic-type=virtio-forwarder --network
private
| grep' id '
| awk'{print $4}'
` $ openstack server create --flavor dpdk.1g --image mellanox --nic port-id=$virtio_port1 --availability-zone nova:overcloud-computesriov-0
.localdomain vm1