This document presents a complete Reference Design Guide (RDG) for accelerated, high-performance and ultra-efficient OpenStack based Network Function Virtualization Infrastructure (NFVi) platform.
The solution comprises of the following main components:
- Red Hat OpenStack Platform (OSP): The leading commercially supported distribution of open source OpenStack software
- Nuage Software Define Networking (SDN): The leading commercially supported SDN solution
- Nvidia interconnect solutions: The leading supplier of smart data center interconnect
This solution takes advantage of OVS offload, an open source technology which allows offloading the SDN data plane from running on the server's CPU (typically x86) to an NVIDIA® Nvidia® SmartNIC. This allow the Virtual Network Function (VNF) applications, or any other application, to enjoy high throughput, high message rate and low and predictable latency while freeing up CPU cores from processing host networking stack.
This document provides a brief overview of the technologies, highlights new functionalities and instructs the user on how to deploy a full working cluster.
Nuage VSP (Virtualized services platform) consists of three components:
- Nuage VSD - Virtualized Server Directory provide the network policy and analytics engine
- Nuage VSC - Virtualized Services Controller delivers real time networking control plane
- Nuage OVRS - Offloading Virtual Routing and Switching is the software forwarding plane
- Intelligent NVIDIA® ConnectX®-5 adapter cards introduce new acceleration engines for maximizing high-performance. The cards are capable of very high message-rate providing the highest performance and most flexible solution for the most demanding applications and markets.
- ASAP2 - Accelerated Switch and Packet Processing® solution combines the performance and efficiency hardware with the flexibility of virtual switching software, leaning on eSwitch (NIC Embedded Switch), which is responsible for complex packet processing supporting OpenFlow forwarding rules.
Nvidia switch SN2410 - Provides the most predictable, highest-performance 100GbE ToR switching platform for the growing demands of today’s data centers.
Powered by the Spectrum® ASIC and packed with 48 ports running at 25GbE, the SN2410 carries a whopping throughput of 4Tb/s with a landmark of 5.95Bpps processing.
Red Hat OSP (OpenStack Platform) main components:
- OpenStack Platform Director - A toolset for installing and managing a complete OpenStack environment.
- Openstack Platform Controller - Controller nodes are responsible for hosting the core services in a Red Hat OpenStack platform environment.
- Server Openstack Nova Compute - Responsible for building a disk image, launching an instance, calls to check the instance's state, attaching persistent storage, and terminating the instance.
VSD composes a network policy scope and sends it to VSC by the XMPP messaging protocol. The policy is interpreted by VSC to OpenFLow protocol and is communicated to OVRS for execution, resulting in actual flow based data plane.
These forwarding data plane flows can be offloaded to Nvidia SmartNIC and can be stored in hardware due to the ASAP2 technology, accelerating packet forwarding and reducing CPU processing from the local host. All supported and wrapped in Red Hat Openstack platform.
Offloading Basic Architecture and Feature Spec
How it works?
First packet hits the slow path, then the flow is composed by Nuage SDN (VSD+VSC) layer and is inserted into OVRS-vSwitch through the control plane using OpenFlow protocol.
The newly learned flow is simultaneously offloaded to Nvidia SmartNIC.
Subsequent packets that hit the SmartNIC and match this specific offloaded flow are automatically forwarded to guest VM via VF interface (blue line) and bypass the traditional kernel forward path (red line).
This unique way holds the ability to maintain Nuage SDN layer intact while achieving high performances with full line rate bandwidth and high packet rate per each VNF (Virtual Network Function).
The ASAP2 VF-LAG feature provides transparent redundancy and doubles throughput for individual VF interface by offloading BOND functionality to Nvidia's SmartNIC hardware.
OVRS seamlessly responsible for offloading BOND configuration with open APIs.
LAG LACP (Link Aggregation Control Protocol) and LAG HA (High-Availability) are transparently supported modes with offloaded LAG.
Solution Logical Design
Multi node scale deployment of each network role share common Management network.
Tenant network mainly contains user data traffic and in-band control messaging of compute node OVRS to VSC server using the OpenFlow protocol. Openstack controller APIs are maintained and accessible via a dedicated network.”
Bill of Materials
Solution Physical Design / Wiring
Network subnets and IPs ranges are documented in network-environment .yaml.
VSD and VSC communicate over the Tenant network. In real production environment, it is recommended to share VSD and VSC communication with a dedicated management network.
The below is a Nvidia Switch configuration example:
Ensure that VT-d and SR-IOV are enabled in BIOS per server.
Host Deployment and Configuration
Before we begin
- This post highlights and points only public Nuage documentation for installing all components covering Red Hat and Nvidia.
- In order to get the most updated licensed Nuage packages and documentation, please contact Nokia at:
- To install OSP 13 director, the server needs to be registered and licensed under Red Hat account. For more information:
Contact Red Hat
1. Install Openstack Director on the Undercloud System
Follow Nuage guidelines for Red Hat official documentation: Install openstack director on the undercloud system.
The below is an undercloud.conf file example for this deployment guide:
2. Download Nuage Source Code
Follow the instructions in Downloading Nuage source code.
Note: Make sure to set git branch to version 13.605.1.
3. Pull Nuage and Red Hat Docker Containers
Follow the instructions in Prepare the Containers, sections 3.1 and 3.2
- Make sure you have followed the steps from Red Hat OSP13 user guide to register containers in local repository in
Obtaining container images and configuring local registry Section 5.5: "Using the undercloud as a local registry"
- The local registry IP is 192.168.24.1 in this this example deployment
- In nuage_container_config.yaml set Nuage release to: 6-0-5 and 'latest' tag.
4. Register and Inspect the Bare Metal Nodes
Follow the instructions in Register and inspect the bare metal nodes
Note: Make sure you have followed the steps from Red Hat OSP13 user guide to register bare metal nodes Registering Nodes for the Overcloud, Section 6.1: "Registering Nodes for the Overcloud".
The below is an instackenv.json file example for this deployment guide:
5. Download the Nuage vsp rpms and Create a Yum Repository
Follow the instructions in download the nuage vsp rpms and create a yum repository in order to identify the required packages.
The below is a full list of all the required packages divided by groups:
Nuage common packages: nuage-bgp, nuage-metadata-agent, nuage-openstack-neutronclient, nuage-puppet-modules, python-openvswitch-nuage
Nuage ovrs packages: mstflint, nuage-openvswitch-ovrs
Extra packages (Of Red Hat official repositories) : boost-filesystem, boost-regex, boost-system, python3, python3-libs, python3-pip, python3-setuptools, createrepo, deltarpm, libconfig, lldpad, perl-JSON, perl-Sys-Syslog, protobuf-c, python-deltarpm, glibc, keyutils-libs, krb5-libs, libcom_err, libgcc, libselinux, libsepol, libstdc++, libverto, nss-softokn-freebl, openssl-libs, pcre, zlib
Note: In order to get the most updated licensed Nuage packages please contact Nokia at:
Creating local repository for Nuage packages:
- Create two directories per each package group, as follows:
Copy each package to its dedicated group directory, as the list of required packages specifies.
- Create a file named nuage_ospd13.repo and place it in:
Example of nuage_ospd13.repo file is available in this section.
Install createrepo util.
What is createrepo ?
For more information, see http://yum.baseurl.org/wiki/RepoCreate.html
- Execute 'createrepo' per each directory created in step 1.
The below is nuage_ospd13.repo file example for this deployment guide:
6. Modify the Overcloud Image
Follow the instructions in modify the overcloud image after following the prior settings in this section.
Note: Make sure to copy overcloud image from ~/images/overcloud-full.qcow2 to:
- Edit /home/stack/nuage-ospdirector/image-patching/nuage_image_patching_scripts/nuage_patching_config.yaml file with:
- In order to download the extra packages, edit the nuage_patching_config.yaml file and add Red Hat account details.
This way the overcloud image can fetch the packages from the Red Hat subscribed repositories.
Here is an example for the required parameters:
RhelUserName: 'abc' RhelPassword: '***' RhelPool: '1234567890123445'
7. Create the Dataplane Roles and Update the Node Profiles
Follow the instructions in Create the dataplane roles and update the node profiles.
Generate the nuage_roles_data.yaml file as follows:
The below is an a node-info.yaml file example for this deployment guide:
8. Generate a CMS ID for Openstack Installation
Follow the instructions in Generate a cms-id for openstack deployment.
- Make sure your OSP Director has a network connectivity to VSD server through public NIC 1 as the Solution Physical design / wiring section describes.
Command line example for generating CMS-ID:
9. Customize the Environment Files
Follow the instructions in customize the environment files.
Based on Solution Physical design / wiring section.
- In this deployment VSD IP for tenant NIC 1 is 192.168.10.101
- In this deployment VSC IP for tenant NIC 1 is 192.168.10.102
The below is a neutron-nuage-config.yaml and nova-nuage-config.yaml file examples for this deployment guide:
10. Assign Nuage Controller Role
Follow the instructions in nuage controller role controller.
To extract controller node-uuid assist this command
11. Assign Nuage Offloaded VRS Nodes Role
Follow the instructions in offload vrs role computeovrs.
To extract compute node-uuid, run the below command
- For ConnectX firmeware, see firmware downloads.
- Execute 'createrepo' for firmware directory /var/www/html/FW_<VERSION>, once the firmware bin file is downloaded.
- This deployment uses firmware version 16.26.1040
- This deployment uses two Nvidia Bond interfaces. Ports are specifed in mellanox-environment.yaml file.
Interfaces names must suite the deployed target servers OS naming.
The below is a mellanox-environment.yaml file example for this deployment guide:
12. Network Isolation
Follow the instructions in the network isolation section: "VF lag with VLANs for Nvidia ConnectX-5 NICs".
BondInterfaceOvsOptions is set to bond LACP in network-environment.yaml
- ovs-hw-offload.yaml contains the names of Nvidia Tenat interfaces for computeovrs servers.
- Make sure ovs-hw-offload.yaml contains the following line (See file example):
- ovs-hw-offload.yaml NodeRootPassword is set with default OS root password.
The below are computeovrs.yaml, controller.yaml, ovs-hw-offload.yaml network-environment.yaml file examples for this deployment guide:
13. Deploy the Overcloud
Follow the instructions in Deploy the overcloud, Section 3: "For VF lag with VLANs for CX-5 NICs".
For your convenience, all template files are attached:
Files OSP directory locations:
Application Deployment and Configuration
This document does not cover VSD and VSC server installation and configuration. It is assumed those are up and running based on Nuage documentation.
Please review this exception before going forward:
Make sure VSD user which is defined as 'NeutronNuageVSDUsername' in neutron-nuage-config.yaml is also defined as part of the CMS group in VSD. Not doing so results in VM spawning failure.
To do so, get the username fromneutron-nuage-config.yaml (in this example the username is 'csproot'), open VSD WebUI 'Platform Configuration' and move the user right to the 'Assigned Users' side.
- Open VSD WebUI
- Click on the Platform Configuration icon in the tor-right corner
- Click Settings
- Click on the Groups icon on the left, choose CMS Group and then press the Assign button
- Now select the desired user, click the arrow to move it to the right and press Submit.
Finally, you should get to this:
Configure a Basic Flavor
Create VSD a Managed Network
Enter OpenStack WebUi.
Successful deployment points you to the Openstack dashboard./home/stack/overcloudrc file contains user and password login credentials.
Under Openstack WebUI Project→Network Tab, click on "create network":
Give it a name and set the desired checkboxes:
Choose Subnet Type to be "VSD Managed Subnet (auto)"
Choose Organization, Domain, Zone and Subnet from the WebUI options. It is assumed that these parameters are already configured in the VSD together with active policy.
Subnet fields are populated automatically. Click next and the network creation is completed.
Create 'switchdev' Port
Create a port with 'switchdev' capabilities from CLI and attach it to the desired network created as describe in the previous section.
Launch to VMs on Separate Hypervisors
Launch VMs from CLI and insert the port ID and image name per each VM:
Initiate ping request between the two running VMs to see if you get a reply.
Set continuous ping requests and move to hypervisor CLI for checking offloading mechanism.
This example shows offloaded flows of outgoing and returning ICMP request/return packets.
The VXLAN interface holds the incoming flow and the representor port holds the outgoing flow.
Dump OVRS Offloaded Rules
Dump TC VXLAN Interface Offloaded Rules
Dump TC Representor Interface Offloaded Rules
Performance test deployment is performed over a single compute node and the XIA server, both connected to Nvidia switch.
The compute node is a device under test running TestPMD over two VFs and is connected to bond VF-LAG interface.
IXIA injects flat UDP packets towards Nvidia switch which encapsulate the packets over the VXLAN tunnel on top of logical LACP bond.
VF-LAG bond intercepts the packets and forwards them to VF1 and VF2 connected to testPMD VM, which forwards the packets from VF1 to VF2 and from VF2 to VF1 back to IXIA server.
Number of flows is determined by the range of source and destination IPs.
The below table presents bandwidth and packets per second while each column maintains different number of flows in hardware starting from 64 to 400,000.
About the Author
Amir Zeidner For the past several years, Amir has worked as a Solutions Architect primarily in the Telco space, leading advanced solutions to answer 5G, NFV, and SDN networking infrastructures requirements. Amir’s expertise in data plane acceleration technologies, such as Accelerated Switching and Network Processing (ASAP²) and DPDK, together with a deep knowledge of open source cloud-based infrastructures, allows him to promote and deliver unique end-to-end NVIDIA Networking solutions throughout the Telco world.
For the past several years, Amir has worked as a Solutions Architect primarily in the Telco space, leading advanced solutions to answer 5G, NFV, and SDN networking infrastructures requirements. Amir’s expertise in data plane acceleration technologies, such as Accelerated Switching and Network Processing (ASAP²) and DPDK, together with a deep knowledge of open source cloud-based infrastructures, allows him to promote and deliver unique end-to-end NVIDIA Networking solutions throughout the Telco world.