Introduction
This document presents a complete Reference Design Guide (RDG) for accelerated, high-performance and ultra-efficient OpenStack based Network Function Virtualization Infrastructure (NFVi) platform.
The solution comprises of the following main components:
- Red Hat OpenStack Platform (OSP): The leading commercially supported distribution of open source OpenStack software
- Nuage Software Define Networking (SDN): The leading commercially supported SDN solution
- Nvidia interconnect solutions: The leading supplier of smart data center interconnect
This solution takes advantage of OVS offload, an open source technology which allows offloading the SDN data plane from running on the server's CPU (typically x86) to an NVIDIA® Nvidia® SmartNIC. This allow the Virtual Network Function (VNF) applications, or any other application, to enjoy high throughput, high message rate and low and predictable latency while freeing up CPU cores from processing host networking stack.
This document provides a brief overview of the technologies, highlights new functionalities and instructs the user on how to deploy a full working cluster.
References
Red Hat and Mellanox Deliver Agile NFV Cloud with Unprecedented Performance
Accelerate Nuage VSP with Mellanox Offloads: VXLAN, ASAP2 and OVS DPDK
Nuage Networks VSP Powered by Mellanox VXLAN Offloads - Video
Components Overview
Nuage VSP (Virtualized services platform) consists of three components:
- Nuage VSD - Virtualized Server Directory provide the network policy and analytics engine
- Nuage VSC - Virtualized Services Controller delivers real time networking control plane
- Nuage OVRS - Offloading Virtual Routing and Switching is the software forwarding plane
Nvidia component:
- Intelligent NVIDIA® ConnectX®-5 adapter cards introduce new acceleration engines for maximizing high-performance. The cards are capable of very high message-rate providing the highest performance and most flexible solution for the most demanding applications and markets.
- ASAP2 - Accelerated Switch and Packet Processing® solution combines the performance and efficiency hardware with the flexibility of virtual switching software, leaning on eSwitch (NIC Embedded Switch), which is responsible for complex packet processing supporting OpenFlow forwarding rules.
Nvidia switch SN2410 - Provides the most predictable, highest-performance 100GbE ToR switching platform for the growing demands of today’s data centers.
Powered by the Spectrum® ASIC and packed with 48 ports running at 25GbE, the SN2410 carries a whopping throughput of 4Tb/s with a landmark of 5.95Bpps processing.
Red Hat OSP (OpenStack Platform) main components:
- OpenStack Platform Director - A toolset for installing and managing a complete OpenStack environment.
- Openstack Platform Controller - Controller nodes are responsible for hosting the core services in a Red Hat OpenStack platform environment.
- Server Openstack Nova Compute - Responsible for building a disk image, launching an instance, calls to check the instance's state, attaching persistent storage, and terminating the instance.
Solution Overview
VSD composes a network policy scope and sends it to VSC by the XMPP messaging protocol. The policy is interpreted by VSC to OpenFLow protocol and is communicated to OVRS for execution, resulting in actual flow based data plane.
These forwarding data plane flows can be offloaded to Nvidia SmartNIC and can be stored in hardware due to the ASAP2 technology, accelerating packet forwarding and reducing CPU processing from the local host. All supported and wrapped in Red Hat Openstack platform.
Offloading Basic Architecture and Feature Spec
How it works?
First packet hits the slow path, then the flow is composed by Nuage SDN (VSD+VSC) layer and is inserted into OVRS-vSwitch through the control plane using OpenFlow protocol.
The newly learned flow is simultaneously offloaded to Nvidia SmartNIC.
Subsequent packets that hit the SmartNIC and match this specific offloaded flow are automatically forwarded to guest VM via VF interface (blue line) and bypass the traditional kernel forward path (red line).
This unique way holds the ability to maintain Nuage SDN layer intact while achieving high performances with full line rate bandwidth and high packet rate per each VNF (Virtual Network Function).
ASAP2 VF-LAG
The ASAP2 VF-LAG feature provides transparent redundancy and doubles throughput for individual VF interface by offloading BOND functionality to Nvidia's SmartNIC hardware.
OVRS seamlessly responsible for offloading BOND configuration with open APIs.
LAG LACP (Link Aggregation Control Protocol) and LAG HA (High-Availability) are transparently supported modes with offloaded LAG.
Solution Logical Design
Multi node scale deployment of each network role share common Management network.
Tenant network mainly contains user data traffic and in-band control messaging of compute node OVRS to VSC server using the OpenFlow protocol. Openstack controller APIs are maintained and accessible via a dedicated network.”
Bill of Materials
Solution Physical Design / Wiring
Network subnets and IPs ranges are documented in network-environment .yaml.
VSD and VSC communicate over the Tenant network. In real production environment, it is recommended to share VSD and VSC communication with a dedicated management network.
The below is a Nvidia Switch configuration example:
Host Configuration
Ensure that VT-d and SR-IOV are enabled in BIOS per server.
Deployment Guide
Host Deployment and Configuration
Before we begin
- This post highlights and points only public Nuage documentation for installing all components covering Red Hat and Nvidia.
Nuage Documentation - In order to get the most updated licensed Nuage packages and documentation, please contact Nokia at:
nuage-smartnic-support@list.nokia.com - To install OSP 13 director, the server needs to be registered and licensed under Red Hat account. For more information:
Contact Red Hat
1. Install Openstack Director on the Undercloud System
Follow Nuage guidelines for Red Hat official documentation: Install openstack director on the undercloud system.
The below is an undercloud.conf file example for this deployment guide:
[DEFAULT] # IP information for the interface on the Undercloud that will be # handling the PXE boots and DHCP for Overcloud instances. The IP # portion of the value will be assigned to the network interface # defined by local_interface, with the netmask defined by the prefix # portion of the value. (string value) local_ip = 192.168.24.1/24 # Virtual IP or DNS address to use for the public endpoints of # Undercloud services. Only used with SSL. (string value) # Deprecated group/name - [DEFAULT]/undercloud_public_vip undercloud_public_host = 192.168.24.2 # Virtual IP or DNS address to use for the admin endpoints of # Undercloud services. Only used with SSL. (string value) # Deprecated group/name - [DEFAULT]/undercloud_admin_vip undercloud_admin_host = 192.168.24.3 # DNS nameserver(s) to use for the undercloud node. (list value) undercloud_nameservers = 10.7.77.192,10.7.77.135 # List of ntp servers to use. (list value) undercloud_ntp_servers = 192.114.62.250 # Network interface on the Undercloud that will be handling the PXE # boots and DHCP for Overcloud instances. (string value) local_interface = enp132s0f0 # Whether to install the TripleO UI. (boolean value) enable_ui = false # Whether to install requirements to run the TripleO validations. # (boolean value) enable_validations = true # Maximum number of attempts the scheduler will make when deploying # the instance. You should keep it greater or equal to the number of # bare metal nodes you expect to deploy at once to work around # potential race condition when scheduling. (integer value) # Minimum value: 1 scheduler_max_attempts = 10 [auth] [ctlplane-subnet] # # From instack-undercloud # # Network CIDR for the Neutron-managed subnet for Overcloud instances. # (string value) # Deprecated group/name - [DEFAULT]/network_cidr cidr = 192.168.24.0/24 # Start of DHCP allocation range for PXE and DHCP of Overcloud # instances on this network. (string value) # Deprecated group/name - [DEFAULT]/dhcp_start dhcp_start = 192.168.24.5 # End of DHCP allocation range for PXE and DHCP of Overcloud instances # on this network. (string value) # Deprecated group/name - [DEFAULT]/dhcp_end dhcp_end = 192.168.24.24 # Temporary IP range that will be given to nodes on this network # during the inspection process. Should not overlap with the range # defined by dhcp_start and dhcp_end, but should be in the same ip # subnet. (string value) # Deprecated group/name - [DEFAULT]/inspection_iprange inspection_iprange = 192.168.24.100,192.168.24.120 # Network gateway for the Neutron-managed network for Overcloud # instances on this network. (string value) # Deprecated group/name - [DEFAULT]/network_gateway gateway = 192.168.24.1 # The network will be masqueraded for external access. (boolean value) masquerade = true
2. Download Nuage Source Code
Follow the instructions in Downloading Nuage source code.
Note: Make sure to set git branch to version 13.605.1.
3. Pull Nuage and Red Hat Docker Containers
Follow the instructions in Prepare the Containers, sections 3.1 and 3.2
Notes:
- Make sure you have followed the steps from Red Hat OSP13 user guide to register containers in local repository in
Obtaining container images and configuring local registry Section 5.5: "Using the undercloud as a local registry" - The local registry IP is 192.168.24.1 in this this example deployment
- In nuage_container_config.yaml set Nuage release to: 6-0-5 and 'latest' tag.
4. Register and Inspect the Bare Metal Nodes
Follow the instructions in Register and inspect the bare metal nodes
Note: Make sure you have followed the steps from Red Hat OSP13 user guide to register bare metal nodes Registering Nodes for the Overcloud, Section 6.1: "Registering Nodes for the Overcloud".
The below is an instackenv.json file example for this deployment guide:
{ "nodes": [ { "pm_password": "3tango11", "name": "control-0", "pm_addr": "10.7.214.205", "capabilities": "profile:control,boot_option:local", "pm_type": "pxe_ipmitool", "pm_user": "rcon" }, { "pm_password": "3tango11", "name": "compute-0", "pm_addr": "10.7.214.208", "capabilities": "profile:computeovrs,boot_option:local", "pm_type": "pxe_ipmitool", "pm_user": "rcon" }, { "pm_password": "3tango11", "name": "compute-1", "pm_addr": "10.7.214.209", "capabilities": "profile:computeovrs,boot_option:local", "pm_type": "pxe_ipmitool", "pm_user": "rcon" } ] }
5. Download the Nuage vsp rpms and Create a Yum Repository
Follow the instructions in download the nuage vsp rpms and create a yum repository in order to identify the required packages.
The below is a full list of all the required packages divided by groups:
Nuage common packages: nuage-bgp, nuage-metadata-agent, nuage-openstack-neutronclient, nuage-puppet-modules, python-openvswitch-nuage
Nuage ovrs packages: mstflint, nuage-openvswitch-ovrs
Extra packages (Of Red Hat official repositories) : boost-filesystem, boost-regex, boost-system, python3, python3-libs, python3-pip, python3-setuptools, createrepo, deltarpm, libconfig, lldpad, perl-JSON, perl-Sys-Syslog, protobuf-c, python-deltarpm, glibc, keyutils-libs, krb5-libs, libcom_err, libgcc, libselinux, libsepol, libstdc++, libverto, nss-softokn-freebl, openssl-libs, pcre, zlib
Note: In order to get the most updated licensed Nuage packages please contact Nokia at:
nuage-smartnic-support@list.nokia.com
Creating local repository for Nuage packages:
- Create two directories per each package group, as follows:
/var/www/html/nuage_osp13_6.0.5/nuage_common/
/var/www/html/nuage_osp13_6.0.5/nuage_ovrs Copy each package to its dedicated group directory, as the list of required packages specifies.
- Create a file named nuage_ospd13.repo and place it in:
/home/stack/nuage-ospdirector/image-patching/nuage_image_patching_scripts
Example of nuage_ospd13.repo file is available in this section. Install createrepo util.
- Execute 'createrepo' per each directory created in step 1.
The below is nuage_ospd13.repo file example for this deployment guide:
######################################################################################################################## ############## Repositories which user want to enable by default on the overcloud image while patching ################# ######################################################################################################################## # This repo should contain following packages: # nuage-bgp # nuage-puppet-modules # nuage-openstack-neutronclient # python-openvswitch-nuage # nuage-metadata-agent [nuage_common] name=nuage_osp13_6.0.5 baseurl=http://192.168.24.1/nuage_osp13_6.0.5/nuage_common enabled=1 gpgcheck=0 ######################################################################################################################## ############## Repositories which we recommend user to disable by default ############################################## ######################################################################################################################## # This repo should contain following packages: # nuage-openvswitch-ovrs # mstflint [nuage_ovrs] name=nuage_osp13_6.0.5 baseurl=http://192.168.24.1/nuage_osp13_6.0.5/nuage_ovrs enabled=1 gpgcheck=0
6. Modify the Overcloud Image
Follow the instructions in modify the overcloud image after following the prior settings in this section.
Note: Make sure to copy overcloud image from ~/images/overcloud-full.qcow2 to:
/home/stack/nuage-ospdirector/image-patching/nuage_image_patching_scripts
Prior settings
- Edit /home/stack/nuage-ospdirector/image-patching/nuage_image_patching_scripts/nuage_patching_config.yaml file with:
RepoFile: 'nuage_ospd13.repo'
DeploymentType: ["ovrs"] - In order to download the extra packages, edit the nuage_patching_config.yaml file and add Red Hat account details.
This way the overcloud image can fetch the packages from the Red Hat subscribed repositories.
Here is an example for the required parameters:RhelUserName: 'abc' RhelPassword: '***' RhelPool: '1234567890123445'
7. Create the Dataplane Roles and Update the Node Profiles
Follow the instructions in Create the dataplane roles and update the node profiles.
Generate the nuage_roles_data.yaml file as follows:
Openstack overcloud roles generate --roles-path /home/stack/nuage-tripleo-heat-templates/roles -o /home/stack/nuage-tripleo-heat-templates/templates/nuage_roles_data.yaml Controller ComputeOvrs
The below is an a node-info.yaml file example for this deployment guide:
parameter_defaults: OvercloudControllerFlavor: control ControllerCount: 1 OvercloudComputeOvrsFlavor: computeovrs ComputeOvrsCount: 2
8. Generate a CMS ID for Openstack Installation
Follow the instructions in Generate a cms-id for openstack deployment.
Note:
- Make sure your OSP Director has a network connectivity to VSD server through public NIC 1 as the Solution Physical design / wiring section describes.
Command line example for generating CMS-ID:
python configure_vsd_cms_id.py --server 10.7.215.50:8443 --serverauth csproot:csproot --organization csp --auth_resource /me --serverssl True --base_uri /nuage/api/v5_0
9. Customize the Environment Files
Follow the instructions in customize the environment files.
Note:
Based on Solution Physical design / wiring section.
- In this deployment VSD IP for tenant NIC 1 is 192.168.10.101
- In this deployment VSC IP for tenant NIC 1 is 192.168.10.102
The below is a neutron-nuage-config.yaml and nova-nuage-config.yaml file examples for this deployment guide:
# A Heat environment file which can be used to enable a # a Neutron Nuage backend on the controller, configured via puppet resource_registry: OS::TripleO::Services::NeutronDhcpAgent: OS::Heat::None OS::TripleO::Services::NeutronL3Agent: OS::Heat::None OS::TripleO::Services::NeutronMetadataAgent: OS::Heat::None OS::TripleO::Services::NeutronOvsAgent: OS::Heat::None OS::TripleO::Services::ComputeNeutronOvsAgent: OS::Heat::None # Override the NeutronMl2PluginBase to use Nuage inside Docker container OS::TripleO::Docker::NeutronMl2PluginBase: ../puppet/services/neutron-plugin-ml2-nuage.yaml OS::TripleO::Services::NeutronCorePlugin: ../docker/services/neutron-plugin-ml2-nuage.yaml parameter_defaults: NeutronNuageNetPartitionName: 'default_name' NeutronNuageVSDIp: '192.168.10.101:8443' NeutronNuageVSDUsername: 'csproot' NeutronNuageVSDPassword: 'csproot' NeutronNuageVSDOrganization: 'csp' NeutronNuageBaseURIVersion: 'v6_0' NeutronNuageCMSId: '4e455172-014d-4fa7-8a99-63efa0f6d79c' UseForwardedFor: true NeutronPluginMl2PuppetTags: 'neutron_plugin_ml2,neutron_plugin_nuage' NeutronServicePlugins: 'NuagePortAttributes,NuageAPI,NuageL3,trunk' NeutronDBSyncExtraParams: '--config-file /etc/neutron/neutron.conf --config-file /etc/neutron/plugins/ml2/ml2_conf.ini --config-file /etc/neutron/plugins/nuage/plugin.ini' NeutronTypeDrivers: 'vxlan' NeutronNetworkType: 'vxlan' NeutronMechanismDrivers: "nuage" NeutronPluginExtensions: 'nuage_subnet,nuage_port,port_security' NeutronFlatNetworks: '*' NovaOVSBridge: 'alubr0' NeutronMetadataProxySharedSecret: 'NuageNetworksSharedSecret' NeutronTunnelIdRanges: '' NeutronNetworkVLANRanges: '' NeutronVniRanges: '1001:2000' HeatEnginePluginDirs: ['/usr/lib/python2.7/site-packages/nuage-heat/'] HorizonCustomizationModule: 'nuage_horizon.customization' HorizonVhostExtraParams: add_listen: True priority: 10 access_log_format: '%a %l %u %t \"%r\" %>s %b \"%%{}{Referer}i\" \"%%{}{User-Agent}i\"' directories: [{'allow_override': ['None'], 'path': '/usr/lib/python2.7/site-packages/nuage_horizon', 'require': 'all granted', 'options': ['FollowSymLinks']}] aliases: [{'alias': '%{root_url}/static/nuage', 'path': '/usr/lib/python2.7/site-packages/nuage_horizon/static'}, {'alias': '%{root_url}/static', 'path': '/usr/share/openstack-dashboard/static'}] ControllerExtraConfig: neutron::config::server_config: DEFAULT/ipam_driver: value: nuage_internal DEFAULT/enable_snat_by_default: value: false neutron::config::plugin_nuage_config: RESTPROXY/nuage_pat: value: legacy_disabled
# Heat environment file which can be used to enable # Nuage backend on the compute, configured via puppet resource_registry: OS::TripleO::Services::ComputeNeutronCorePlugin: ../puppet/services/nuage-compute-vrs.yaml parameter_defaults: NuageActiveController: '192.168.10.102' NuageStandbyController: '0.0.0.0' NuageBridgeMTU: '9000' NovaIPv6: 'True' NuageMetadataProxySharedSecret: 'NuageNetworksSharedSecret' NuageNovaApiEndpoint: 'internalURL' NovaOVSBridge: 'alubr0' NovaComputeLibvirtType: 'kvm' NovaComputeLibvirtVifDriver: 'nova.virt.libvirt.vif.LibvirtGenericVIFDriver' # VrsExtraConfigs can be used to configure extra parameters in /etc/default/openvswitch # For example to set "NETWORK_UPLINK_INTF" see below sample: # VrsExtraConfigs: {"NETWORK_UPLINK_INTF": "eno1"} VrsExtraConfigs: {} NuageGpgKeys: []
10. Assign Nuage Controller Role
Follow the instructions in nuage controller role controller.
To extract controller node-uuid assist this command
openstack baremetal node list
11. Assign Nuage Offloaded VRS Nodes Role
Follow the instructions in offload vrs role computeovrs.
Notes:
To extract compute node-uuid, run the below command
openstack baremetal node list
- For ConnectX firmeware, see firmware downloads.
- Execute 'createrepo' for firmware directory /var/www/html/FW_<VERSION>, once the firmware bin file is downloaded.
- This deployment uses firmware version 16.26.1040
- This deployment uses two Nvidia Bond interfaces. Ports are specifed in mellanox-environment.yaml file.
Interfaces names must suite the deployed target servers OS naming.
The below is a mellanox-environment.yaml file example for this deployment guide:
resource_registry: OS::TripleO::ComputeSriov::NodeUserData: ../firstboot/mellanox_fw_update.yaml parameter_defaults: ################ # Nic's params # ################ MellanoxTenantPort1: "ens2f0" MellanoxTenantPort2: "ens2f1" MellanoxInfraPort1: "ens1f0" MellanoxInfraPort2: "ens1f1" ######################## # FIRST Boot FW config # ######################## BIN_DIR_URL: "http://192.168.24.1/FW_16_26_1040/" NUM_OF_VFS: 16 SRIOV_EN: True ESWITCH_IPV4_TTL_MODIFY_ENABLE: True PRIO_TAG_REQUIRED_EN: True
12. Network Isolation
Follow the instructions in the network isolation section: "VF lag with VLANs for Nvidia ConnectX-5 NICs".
Notes:
BondInterfaceOvsOptions is set to bond LACP in network-environment.yaml
- ovs-hw-offload.yaml contains the names of Nvidia Tenat interfaces for computeovrs servers.
- Make sure ovs-hw-offload.yaml contains the following line (See file example):
resource_registry:
OS::TripleO::Services::NuageComputeOvrs: ../puppet/services/nuage-compute-ovrs.yaml - ovs-hw-offload.yaml NodeRootPassword is set with default OS root password.
The below are computeovrs.yaml, controller.yaml, ovs-hw-offload.yaml network-environment.yaml file examples for this deployment guide:
parameter_defaults: # This section is where deployment-specific configuration is done # CIDR subnet mask length for provisioning network ControlPlaneSubnetCidr: '24' # Gateway router for the provisioning network (or Undercloud IP) ControlPlaneDefaultRoute: 192.168.24.1 EC2MetadataIp: 192.168.24.1 # Generally the IP of the Undercloud # Customize the IP subnets to match the local environment StorageNetCidr: '172.16.1.0/24' StorageMgmtNetCidr: '172.16.3.0/24' InternalApiNetCidr: '172.16.2.0/24' TenantNetCidr: '192.168.10.0/24' ExternalNetCidr: '172.16.4.0/24' ManagementNetCidr: '10.0.1.0/24' # Customize the VLAN IDs to match the local environment StorageNetworkVlanID: 30 StorageMgmtNetworkVlanID: 40 InternalApiNetworkVlanID: 20 TenantNetworkVlanID: 50 ExternalNetworkVlanID: 10 ManagementNetworkVlanID: 60 StorageAllocationPools: [{'start': '172.16.1.200', 'end': '172.16.1.250'}] StorageMgmtAllocationPools: [{'start': '172.16.3.200', 'end': '172.16.3.250'}] InternalApiAllocationPools: [{'start': '172.16.2.200', 'end': '172.16.2.250'}] TenantAllocationPools: [{'start': '192.168.10.50', 'end': '192.168.10.80'}] # Leave room if the external network is also used for floating IPs ExternalAllocationPools: [{'start': '172.16.4.100', 'end': '172.16.4.150'}] ManagementAllocationPools: [{'start': '10.0.1.4', 'end': '10.0.1.250'}] # Gateway routers for routable networks ExternalInterfaceDefaultRoute: '172.16.4.1' # Define the DNS servers (maximum 2) for the overcloud nodes DnsServers: ["8.8.8.8","10.7.77.192"] # The tunnel type for the tenant network (vxlan or gre). Set to '' to disable tunneling. NeutronTunnelTypes: 'vxlan' # Customize bonding options, e.g. "mode=4 lacp_rate=1 updelay=1000 miimon=100" # for Linux bonds w/LACP, or "bond_mode=active-backup" for OVS active/backup. BondInterfaceOvsOptions: "mode=4 miimon=100 lacp_rate=1 xmit_hash_policy=layer3+4" TenantPortEthtoolOptions: "-L ${DEVICE} combined 40; -X ${DEVICE} equal 38; -U ${DEVICE} flow-type tcp4 src-port 6633 action 39 loc 1; -U ${DEVICE} flow-type tcp4 src-port 7406 action 39 loc 2; -U ${DEVICE} flow-type tcp4 src-port 39090 action 39 loc 3;-U ${DEVICE} flow-type tcp4 src-port 179 action 39 loc 4"
heat_template_version: queens description: > Software Config to drive os-net-config with 2 bonded nics on a bridge with VLANs attached for the ComputeSriov role. parameters: ControlPlaneIp: default: '' description: IP address/subnet on the ctlplane network type: string StorageIpSubnet: default: '' description: IP address/subnet on the storage network type: string StorageMgmtIpSubnet: default: '' description: IP address/subnet on the storage_mgmt network type: string InternalApiIpSubnet: default: '' description: IP address/subnet on the internal_api network type: string TenantIpSubnet: default: '' description: IP address/subnet on the tenant network type: string ExternalIpSubnet: default: '' description: IP address/subnet on the external network type: string ManagementIpSubnet: default: '' description: IP address/subnet on the management network type: string BondInterfaceOvsOptions: default: bond_mode=active-backup description: 'The ovs_options or bonding_options string for the bond interface. Set things like lacp=active and/or bond_mode=balance-slb for OVS bonds or like mode=4 for Linux bonds using this option.' type: string StorageNetworkVlanID: default: 30 description: Vlan ID for the storage network traffic. type: number StorageMgmtNetworkVlanID: default: 40 description: Vlan ID for the storage_mgmt network traffic. type: number InternalApiNetworkVlanID: default: 20 description: Vlan ID for the internal_api network traffic. type: number TenantNetworkVlanID: default: 50 description: Vlan ID for the tenant network traffic. type: number ExternalNetworkVlanID: default: 10 description: Vlan ID for the external network traffic. type: number ManagementNetworkVlanID: default: 60 description: Vlan ID for the management network traffic. type: number ControlPlaneDefaultRoute: # Override this via parameter_defaults description: The default route of the control plane network. type: string ExternalInterfaceDefaultRoute: default: '192.168.24.2' description: default route for the external network type: string ControlPlaneSubnetCidr: # Override this via parameter_defaults default: '24' description: The subnet CIDR of the control plane network. type: string DnsServers: # Override this via parameter_defaults default: [] description: A list of DNS servers (2 max for some implementations) that will be added to resolv.conf. type: comma_delimited_list EC2MetadataIp: # Override this via parameter_defaults description: The IP address of the EC2 metadata server. type: string MellanoxTenantPort1: description: Mellanox Tenant Port1 type: string MellanoxTenantPort2: description: Mellanox Tenant Port2 type: string MellanoxInfraPort1: description: Mellanox Infra Port1 type: string MellanoxInfraPort2: description: Mellanox Infra Port2 type: string TenantPortEthtoolOptions: description: Port ethtool options for Control Plane Protectionn type: string resources: OsNetConfigImpl: type: OS::Heat::SoftwareConfig properties: group: script config: str_replace: template: get_file: /usr/share/openstack-tripleo-heat-templates/network/scripts/run-os-net-config.sh params: $network_config: network_config: - type: interface name: eno1 use_dhcp: false addresses: - ip_netmask: list_join: - / - - get_param: ControlPlaneIp - get_param: ControlPlaneSubnetCidr routes: - ip_netmask: 169.254.169.254/32 next_hop: get_param: EC2MetadataIp - default: true next_hop: get_param: ControlPlaneDefaultRoute # Linux bond for Offload VRS Compute nodes - type: linux_bond name: infra-bond dns_servers: get_param: DnsServers bonding_options: get_param: BondInterfaceOvsOptions members: - type: interface name: get_param: MellanoxInfraPort1 - type: interface name: get_param: MellanoxInfraPort2 - type: vlan device: infra-bond vlan_id: get_param: StorageNetworkVlanID addresses: - ip_netmask: get_param: StorageIpSubnet - type: vlan device: infra-bond vlan_id: get_param: InternalApiNetworkVlanID addresses: - ip_netmask: get_param: InternalApiIpSubnet - type: linux_bond name: tenant-bond dns_servers: get_param: DnsServers bonding_options: get_param: BondInterfaceOvsOptions members: - type: sriov_pf name: get_param: MellanoxTenantPort1 link_mode: switchdev numvfs: 8 promisc: true use_dhcp: false primary: true ethtool_opts: get_param: TenantPortEthtoolOptions - type: sriov_pf name: get_param: MellanoxTenantPort2 link_mode: switchdev numvfs: 8 promisc: true use_dhcp: false ethtool_opts: get_param: TenantPortEthtoolOptions - type: vlan device: tenant-bond vlan_id: get_param: TenantNetworkVlanID addresses: - ip_netmask: get_param: TenantIpSubnet outputs: OS::stack_id: description: The OsNetConfigImpl resource. value: get_resource: OsNetConfigImpl
heat_template_version: queens description: > Software Config to drive os-net-config with 2 bonded nics on a bridge with VLANs attached for the Controller role. parameters: ControlPlaneIp: default: '' description: IP address/subnet on the ctlplane network type: string StorageIpSubnet: default: '' description: IP address/subnet on the storage network type: string StorageMgmtIpSubnet: default: '' description: IP address/subnet on the storage_mgmt network type: string InternalApiIpSubnet: default: '' description: IP address/subnet on the internal_api network type: string TenantIpSubnet: default: '' description: IP address/subnet on the tenant network type: string ExternalIpSubnet: default: '' description: IP address/subnet on the external network type: string ManagementIpSubnet: default: '' description: IP address/subnet on the management network type: string BondInterfaceOvsOptions: default: bond_mode=active-backup description: 'The ovs_options or bonding_options string for the bond interface. Set things like lacp=active and/or bond_mode=balance-slb for OVS bonds or like mode=4 for Linux bonds using this option.' type: string StorageNetworkVlanID: default: 30 description: Vlan ID for the storage network traffic. type: number StorageMgmtNetworkVlanID: default: 40 description: Vlan ID for the storage_mgmt network traffic. type: number InternalApiNetworkVlanID: default: 20 description: Vlan ID for the internal_api network traffic. type: number TenantNetworkVlanID: default: 50 description: Vlan ID for the tenant network traffic. type: number ExternalNetworkVlanID: default: 10 description: Vlan ID for the external network traffic. type: number ManagementNetworkVlanID: default: 60 description: Vlan ID for the management network traffic. type: number ControlPlaneDefaultRoute: # Override this via parameter_defaults description: The default route of the control plane network. type: string ExternalInterfaceDefaultRoute: default: '192.168.24.2' description: default route for the external network type: string ControlPlaneSubnetCidr: # Override this via parameter_defaults default: '24' description: The subnet CIDR of the control plane network. type: string DnsServers: # Override this via parameter_defaults default: [] description: A list of DNS servers (2 max for some implementations) that will be added to resolv.conf. type: comma_delimited_list EC2MetadataIp: # Override this via parameter_defaults description: The IP address of the EC2 metadata server. type: string MellanoxInfraPort1: description: Mellanox Infra Port1 type: string MellanoxInfraPort2: description: Mellanox Infra Port2 type: string resources: OsNetConfigImpl: type: OS::Heat::SoftwareConfig properties: group: script config: str_replace: template: get_file: /usr/share/openstack-tripleo-heat-templates/network/scripts/run-os-net-config.sh params: $network_config: network_config: - type: interface name: eno1 use_dhcp: false addresses: - ip_netmask: list_join: - / - - get_param: ControlPlaneIp - get_param: ControlPlaneSubnetCidr routes: - ip_netmask: 169.254.169.254/32 next_hop: get_param: EC2MetadataIp # Linux bond for Controller nodes - type: linux_bond name: infra-bond dns_servers: get_param: DnsServers bonding_options: get_param: BondInterfaceOvsOptions members: - type: interface name: get_param: MellanoxInfraPort1 - type: interface name: get_param: MellanoxInfraPort2 - type: vlan device: infra-bond vlan_id: get_param: StorageNetworkVlanID addresses: - ip_netmask: get_param: StorageIpSubnet - type: vlan device: infra-bond vlan_id: get_param: StorageMgmtNetworkVlanID addresses: - ip_netmask: get_param: StorageMgmtIpSubnet - type: vlan device: infra-bond vlan_id: get_param: InternalApiNetworkVlanID addresses: - ip_netmask: get_param: InternalApiIpSubnet - type: vlan device: infra-bond vlan_id: get_param: TenantNetworkVlanID addresses: - ip_netmask: get_param: TenantIpSubnet - type: vlan device: infra-bond vlan_id: get_param: ExternalNetworkVlanID addresses: - ip_netmask: get_param: ExternalIpSubnet routes: - default: true next_hop: get_param: ExternalInterfaceDefaultRoute outputs: OS::stack_id: description: The OsNetConfigImpl resource. value: get_resource: OsNetConfigImpl
# A Heat environment file that enables OVS Hardware Offload in the overcloud. resource_registry: OS::TripleO::Services::NuageComputeOvrs: ../puppet/services/nuage-compute-ovrs.yaml parameter_defaults: NovaSchedulerDefaultFilters: ['RetryFilter','AvailabilityZoneFilter','RamFilter','ComputeFilter','ComputeCapabilitiesFilter','ImagePropertiesFilter','ServerGroupAntiAffinityFilter','ServerGroupAffinityFilter','PciPassthroughFilter'] NovaSchedulerAvailableFilters: ["nova.scheduler.filters.all_filters","nova.scheduler.filters.pci_passthrough_filter.PciPassthroughFilter"] NodeRootPassword: "3tango" # Kernel arguments for ComputeOvrs node ComputeOvrsParameters: KernelArgs: "intel_iommu=on iommu=pt pci=realloc" #NOTE: By default TunedProfileName is set to "cpu-partitioning" in ovrs-role.yaml. # If IsolCpusList is not set in your environment, then leave TunedProfileName below to set to empty string. # If planning on setting IsolCpusList in your environment # 1. You can comment the below line to set TunedProfileName to "cpu-partitioning" or # 2. You can pass your custom Tuned Profile to apply to the host TunedProfileName: "" OvsHwOffload: True # Mapping of SR-IOV PF interface to neutron physical_network. # In case of Vxlan/GRE physical_network should be null. # In case of flat/vlan the physical_network should as configured in neutron. NovaPCIPassthrough: - devname: "ens2f0" physical_network: null - devname: "ens2f1" physical_network: null
13. Deploy the Overcloud
Follow the instructions in Deploy the overcloud, Section 3: "For VF lag with VLANs for CX-5 NICs".
openstack overcloud deploy --templates -r /home/stack/nuage-tripleo-heat-templates/templates/nuage_roles_data.yaml -e /home/stack/templates/node-info.yaml -e /home/stack/templates/overcloud_images.yaml -e /home/stack/nuage-tripleo-heat-templates/environments/nuage_overcloud_images.yaml -e /usr/share/openstack-tripleo-heat-templates/environments/network-isolation.yaml -e /home/stack/nuage-tripleo-heat-templates/environments/network-environment.yaml -e /home/stack/nuage-tripleo-heat-templates/environments/net-bond-with-vlans.yaml -e /home/stack/nuage-tripleo-heat-templates/environments/neutron-nuage-config.yaml -e /home/stack/nuage-tripleo-heat-templates/environments/nova-nuage-config.yaml -e /home/stack/nuage-tripleo-heat-templates/environments/mellanox-environment.yaml -e /home/stack/nuage-tripleo-heat-templates/environments/ovs-hw-offload.yaml -e /usr/share/openstack-tripleo-heat-templates/environments/host-config-and-reboot.yaml --ntp-server 192.168.24.1
For your convenience, all template files are attached:
Files OSP directory locations:
/home/stack/nuage-tripleo-heat-templates/templates/nuage_roles_data.yaml
/home/stack/templates/node-info.yaml
/home/stack/templates/overcloud_images.yaml
/home/stack/nuage-tripleo-heat-templates/environments/nuage_overcloud_images.yaml
/home/stack/nuage-tripleo-heat-templates/environments/network-environment.yaml
/home/stack/nuage-tripleo-heat-templates/environments/net-bond-with-vlans.yaml
/home/stack/nuage-tripleo-heat-templates/environments/neutron-nuage-config.yaml
/home/stack/nuage-tripleo-heat-templates/environments/nova-nuage-config.yaml
/home/stack/nuage-tripleo-heat-templates/environments/mellanox-environment.yaml
/home/stack/nuage-tripleo-heat-templates/environments/ovs-hw-offload.yaml
/home/stack/nuage-tripleo-heat-templates/network/config/bond-with-vlans/controller.yaml
/home/stack/nuage-tripleo-heat-templates/network/config/bond-with-vlans/computeovrs.yaml
Application Deployment and Configuration
This document does not cover VSD and VSC server installation and configuration. It is assumed those are up and running based on Nuage documentation.
Please review this exception before going forward:
Make sure VSD user which is defined as 'NeutronNuageVSDUsername' in neutron-nuage-config.yaml is also defined as part of the CMS group in VSD. Not doing so results in VM spawning failure.
To do so, get the username fromneutron-nuage-config.yaml (in this example the username is 'csproot'), open VSD WebUI 'Platform Configuration' and move the user right to the 'Assigned Users' side.
- Open VSD WebUI
- Click on the Platform Configuration icon in the tor-right corner
- Click Settings
- Click on the Groups icon on the left, choose CMS Group and then press the Assign button
- Now select the desired user, click the arrow to move it to the right and press Submit.
Finally, you should get to this:
Configure a Basic Flavor
source overcloudrc openstack flavor create m1.small --id 1 --ram 4096 --disk 20 --vcpus 4
Create VSD a Managed Network
Enter OpenStack WebUi.
Successful deployment points you to the Openstack dashboard.
/home/stack/overcloudrc file contains user and password login credentials.Under Openstack WebUI Project→Network Tab, click on "create network":
Give it a name and set the desired checkboxes:
Choose Subnet Type to be "VSD Managed Subnet (auto)"
Choose Organization, Domain, Zone and Subnet from the WebUI options. It is assumed that these parameters are already configured in the VSD together with active policy.
Subnet fields are populated automatically. Click next and the network creation is completed.
Create 'switchdev' Port
Create a port with 'switchdev' capabilities from CLI and attach it to the desired network created as describe in the previous section.
openstack port create direct1 --vnic-type=direct --no-security-group --network Net1 --binding-profile '{"capabilities":["switchdev"]}' openstack port create direct2 --vnic-type=direct --no-security-group --network Net1 --binding-profile '{"capabilities":["switchdev"]}'
Launch to VMs on Separate Hypervisors
Launch VMs from CLI and insert the port ID and image name per each VM:
openstack server create --flavor m1.small --image <image_name> --nic port-id=<port1_ID> --availability-zone nova:overcloud-computesriov-0.localdomain vm1 openstack server create --flavor m1.small --image <image_name> --nic port-id=<port2_ID> --availability-zone nova:overcloud-computesriov-1.localdomain vm2
Deployment Verification
Initiate ping request between the two running VMs to see if you get a reply.
Set continuous ping requests and move to hypervisor CLI for checking offloading mechanism.
This example shows offloaded flows of outgoing and returning ICMP request/return packets.
The VXLAN interface holds the incoming flow and the representor port holds the outgoing flow.
Dump OVRS Offloaded Rules
[root@overcloud-computesriov-0 ~]sudo ovs-dpctl dump-flows type=offloaded --name in_port(ens2f0_6),eth(dst=fa:16:3e:81:a2:b2),eth_type(0x8100),vlan(vid=0,pcp=0),encap(eth_type(0x0800),ipv4(src=10.44.0.0/255.255.0.0,dst=0.0.0.0/1.0.0.0,proto=1,tos=0/0x3)), packets:2808, bytes:438048, used:0.120s, actions:set(tunnel(tun_id=0x8cfcb6,dst=192.168.10.52,tp_dst=4789,flags(key))),vxlan_sys_4789 tunnel(tun_id=0x8cfcb6,src=192.168.10.52,dst=192.168.10.50,ttl=64,flags(+key)),in_port(vxlan_sys_4789),eth(dst=fa:16:3e:01:78:98),eth_type(0x8100),vlan(vid=0,pcp=0),encap(eth_type(0x0800),ipv4(dst=0.0.0.0/1.0.0.0,proto=1)), packets:2808, bytes:286416, used:0.130s, actions:ens2f0_6
Dump TC VXLAN Interface Offloaded Rules
[root@overcloud-computesriov-0 ~]tc filter show dev vxlan_sys_4789: parent ffff: filter protocol 802.1Q pref 4 flower chain 0 handle 0x8fb vlan_id 0 vlan_prio 0 vlan_ethtype ip dst_mac 68:54:ed:00:0d:0b src_mac c6:17:f1:d8:4e:32 eth_type ipv4 ip_proto udp ip_tos 0x0/ff ip_ttl 0x3d/ff dst_ip 17.0.11.108 src_ip 49.0.11.108 dst_port 10001 src_port 1000 in_hw in_hw_count 1 action order 1: tunnel_key set src_ip 0.0.0.0 dst_ip 3.1.1.102 key_id 7708303 dst_port 4789 nocsum ttl 64 pipe index 9472 ref 1 bind 1 action order 2: pedit action pipe keys 5 index 9472 ref 1 bind 1 key #0 at ipv4+8: val 3c000000 mask 00ffffff key #1 at eth+4: val 0000fa7b mask ffff0000 key #2 at eth+8: val aea37243 mask 00000000 key #3 at eth+0: val 00000301 mask 00000000 key #4 at eth+4: val 01660000 mask 0000ffff action order 3: csum (iph, udp) action pipe index 9472 ref 1 bind 1 action order 4: mirred (Egress Redirect to device vxlan_sys_4789) stolen index 9472 ref 1 bind 1 cookie c81b9f7c3c4cac8437509ba179c02c83
Dump TC Representor Interface Offloaded Rules
[root@overcloud-computesriov-0 ~]tc filter show dev ens2f0_6: parent ffff: filter protocol 802.1Q pref 2 flower chain 0 filter protocol 802.1Q pref 4 flower chain 0 filter protocol 802.1Q pref 5 flower chain 0 filter protocol 802.1Q pref 6 flower chain 0 filter protocol 802.1Q pref 7 flower chain 0 filter protocol 802.1Q pref 7 flower chain 0 handle 0x1 vlan_id 0 vlan_ethtype ip dst_mac fa:16:3e:81:a2:b2 eth_type ipv4 ip_proto icmp ip_tos 0/3 dst_ip 10.44.27.17/1.0.0.0 src_ip 10.44.27.20/16 in_hw action order 1: tunnel_key set src_ip 0.0.0.0 dst_ip 192.168.10.52 key_id 9239734 dst_port 4789 nocsum ttl 64 pipe index 1 ref 1 bind 1 action order 2: mirred (Egress Redirect to device vxlan_sys_4789) stolen index 1 ref 1 bind 1
Performance Testing
Performance test deployment is performed over a single compute node and the XIA server, both connected to Nvidia switch.
The compute node is a device under test running TestPMD over two VFs and is connected to bond VF-LAG interface.
IXIA injects flat UDP packets towards Nvidia switch which encapsulate the packets over the VXLAN tunnel on top of logical LACP bond.
VF-LAG bond intercepts the packets and forwards them to VF1 and VF2 connected to testPMD VM, which forwards the packets from VF1 to VF2 and from VF2 to VF1 back to IXIA server.
Number of flows is determined by the range of source and destination IPs.
Test Results
The below table presents bandwidth and packets per second while each column maintains different number of flows in hardware starting from 64 to 400,000.
About the Author
Amir Zeidner For the past several years, Amir has worked as a Solutions Architect primarily in the Telco space, leading advanced solutions to answer 5G, NFV, and SDN networking infrastructures requirements. Amir’s expertise in data plane acceleration technologies, such as Accelerated Switching and Network Processing (ASAP²) and DPDK, together with a deep knowledge of open source cloud-based infrastructures, allows him to promote and deliver unique end-to-end NVIDIA Networking solutions throughout the Telco world.
Related Documents