image image image image image image



On This Page

Introduction

This document presents a complete Reference Design Guide (RDG) for accelerated, high-performance and ultra-efficient OpenStack based Network Function Virtualization Infrastructure (NFVi) platform.

The solution comprises of the following main components:

This solution takes advantage of OVS offload, an open source technology which allows offloading the SDN data plane from running on the server's CPU (typically x86) to an NVIDIA® Nvidia® SmartNIC. This allow the Virtual Network Function (VNF) applications, or any other application, to enjoy high throughput, high message rate and low and predictable latency while freeing up CPU cores from processing host networking stack.

This document provides a brief overview of the technologies, highlights new functionalities and instructs the user on how to deploy a full working cluster.

References

Red Hat and Mellanox Deliver Agile NFV Cloud with Unprecedented Performance

Accelerate Nuage VSP with Mellanox Offloads: VXLAN, ASAP2 and OVS DPDK

Nuage Networks VSP Powered by Mellanox VXLAN Offloads - Video

Components Overview

Nuage VSP (Virtualized services platform) consists of three components:

  • Nuage VSD - Virtualized Server Directory provide the network policy and analytics engine
  • Nuage VSC - Virtualized Services Controller delivers real time networking control plane 
  • Nuage OVRS - Offloading Virtual Routing and Switching is the software forwarding plane

Nvidia component:

  • Intelligent NVIDIA® ConnectX®-5 adapter cards introduce new acceleration engines for maximizing high-performance. The cards are capable of very high message-rate providing the highest performance and most flexible solution for the most demanding applications and markets.
  • ASAP2 - Accelerated Switch and Packet Processing® solution combines the performance and efficiency hardware with the flexibility of virtual switching software, leaning on eSwitch (NIC Embedded Switch), which is responsible for complex packet processing supporting OpenFlow forwarding rules. 

  • Nvidia switch SN2410 - Provides the most predictable, highest-performance 100GbE ToR switching platform for the growing demands of today’s data centers.
    Powered by the Spectrum® ASIC and packed with 48 ports running at 25GbE, the SN2410 carries a whopping throughput of 4Tb/s with a landmark of 5.95Bpps processing.

Red Hat OSP (OpenStack Platform) main components:

  • OpenStack Platform Director - A toolset for installing and managing a complete OpenStack environment.
  • Openstack Platform Controller - Controller nodes are responsible for hosting the core services in a Red Hat OpenStack platform environment.
  • Server Openstack Nova Compute - Responsible for building a disk image, launching an instance, calls to check the instance's state, attaching persistent storage, and terminating the instance. 

 

Solution Overview

VSD composes a network policy scope and sends it to VSC by the XMPP messaging protocol. The policy is interpreted by VSC to OpenFLow protocol and is communicated to OVRS for execution, resulting in actual flow based data plane.

These forwarding data plane flows can be offloaded to Nvidia SmartNIC and can be stored in hardware due to the ASAP2 technology, accelerating packet forwarding and reducing CPU processing from the local host. All supported and wrapped in Red Hat Openstack platform.

Offloading Basic Architecture and Feature Spec 

How it works?

First packet hits the slow path, then the flow is composed by Nuage SDN (VSD+VSC) layer and is inserted into OVRS-vSwitch through the control plane using OpenFlow protocol.
The newly learned flow is simultaneously offloaded to Nvidia SmartNIC.
Subsequent packets that hit the SmartNIC and match this specific offloaded flow are automatically forwarded to guest VM via VF interface (blue line) and bypass the traditional kernel forward path (red line).
This unique way holds the ability to maintain Nuage SDN layer intact while achieving high performances with full line rate bandwidth and high packet rate per each VNF (Virtual Network Function).

ASAP2 VF-LAG 

The ASAP2 VF-LAG feature provides transparent redundancy and doubles throughput for individual VF interface by offloading BOND functionality to Nvidia's SmartNIC hardware.

OVRS seamlessly responsible for offloading BOND configuration with open APIs.

LAG LACP (Link Aggregation Control Protocol) and LAG HA (High-Availability) are transparently supported modes with offloaded LAG.

Solution Logical Design

Multi node scale deployment of each network role share common Management network.

Tenant network mainly contains user data traffic and in-band control messaging of compute node OVRS to VSC server using the OpenFlow protocol. Openstack controller APIs are maintained and accessible via a dedicated network.”


Bill of Materials

 

Solution Physical Design / Wiring

Network subnets and IPs ranges are documented in network-environment .yaml.

VSD and VSC communicate over the Tenant network. In real production environment, it is recommended to share VSD and VSC communication with a dedicated management network.


The below is a Nvidia Switch configuration example:


Host Configuration

Ensure that VT-d and SR-IOV are enabled in BIOS per server.

Deployment Guide

Host Deployment and Configuration


Before we begin

  • This post highlights and points only public Nuage documentation for installing all components covering Red Hat and Nvidia.
    Nuage Documentation
  • In order to get the most updated licensed Nuage packages and documentation, please contact Nokia at:
    nuage-smartnic-support@list.nokia.com
  • To install OSP 13 director, the server needs to be registered and licensed under Red Hat account. For more information:
    Contact Red Hat

1. Install Openstack Director on the Undercloud System

Follow Nuage guidelines for Red Hat official documentation: Install openstack director on the undercloud system.

Note: local_interface in undercloud.conf file stands for OSP Director “Control NIC 2” interface which is connected to “Provisioning / Control Plane” network, as shown in the “Solution physical design / wiring” figure.

The below is an undercloud.conf file example for this deployment guide:

undercloud.conf 
[DEFAULT]

# IP information for the interface on the Undercloud that will be
# handling the PXE boots and DHCP for Overcloud instances.  The IP
# portion of the value will be assigned to the network interface
# defined by local_interface, with the netmask defined by the prefix
# portion of the value. (string value)
local_ip = 192.168.24.1/24

# Virtual IP or DNS address to use for the public endpoints of
# Undercloud services. Only used with SSL. (string value)
# Deprecated group/name - [DEFAULT]/undercloud_public_vip
undercloud_public_host = 192.168.24.2

# Virtual IP or DNS address to use for the admin endpoints of
# Undercloud services. Only used with SSL. (string value)
# Deprecated group/name - [DEFAULT]/undercloud_admin_vip
undercloud_admin_host = 192.168.24.3

# DNS nameserver(s) to use for the undercloud node. (list value)
undercloud_nameservers = 10.7.77.192,10.7.77.135

# List of ntp servers to use. (list value)
undercloud_ntp_servers = 192.114.62.250

# Network interface on the Undercloud that will be handling the PXE
# boots and DHCP for Overcloud instances. (string value)
local_interface = enp132s0f0

# Whether to install the TripleO UI. (boolean value)
enable_ui = false

# Whether to install requirements to run the TripleO validations.
# (boolean value)
enable_validations = true

# Maximum number of attempts the scheduler will make when deploying
# the instance. You should keep it greater or equal to the number of
# bare metal nodes you expect to deploy at once to work around
# potential race condition when scheduling. (integer value)
# Minimum value: 1
scheduler_max_attempts = 10

[auth]

[ctlplane-subnet]

#
# From instack-undercloud
#

# Network CIDR for the Neutron-managed subnet for Overcloud instances.
# (string value)
# Deprecated group/name - [DEFAULT]/network_cidr
cidr = 192.168.24.0/24

# Start of DHCP allocation range for PXE and DHCP of Overcloud
# instances on this network. (string value)
# Deprecated group/name - [DEFAULT]/dhcp_start
dhcp_start = 192.168.24.5

# End of DHCP allocation range for PXE and DHCP of Overcloud instances
# on this network. (string value)
# Deprecated group/name - [DEFAULT]/dhcp_end
dhcp_end = 192.168.24.24

# Temporary IP range that will be given to nodes on this network
# during the inspection process. Should not overlap with the range
# defined by dhcp_start and dhcp_end, but should be in the same ip
# subnet. (string value)
# Deprecated group/name - [DEFAULT]/inspection_iprange
inspection_iprange = 192.168.24.100,192.168.24.120

# Network gateway for the Neutron-managed network for Overcloud
# instances on this network. (string value)
# Deprecated group/name - [DEFAULT]/network_gateway
gateway = 192.168.24.1

# The network will be masqueraded for external access. (boolean value)
masquerade = true

2. Download Nuage Source Code

Follow the instructions in Downloading Nuage source code.

Note: Make sure to set git branch to version 13.605.1.

3. Pull Nuage and Red Hat Docker Containers 

Follow the instructions in Prepare the Containers, sections 3.1 and 3.2

Notes:

  • Make sure you have followed the steps from Red Hat OSP13 user guide to register containers in local repository in
    Obtaining container images and configuring local registry  Section 5.5: "Using the undercloud as a local registry"
  • The local registry IP is 192.168.24.1 in this this example deployment 
  • In nuage_container_config.yaml set Nuage release to: 6-0-5 and 'latest' tag.

4. Register and Inspect the Bare Metal Nodes

Follow the instructions in Register and inspect the bare metal nodes

Note: Make sure you have followed the steps from Red Hat OSP13 user guide to register bare metal nodes Registering Nodes for the Overcloud, Section 6.1: "Registering Nodes for the Overcloud".

The below is an instackenv.json file example for this deployment guide:

instackenv.json 
{
    "nodes": [
        {
            "pm_password": "3tango11",
            "name": "control-0",
            "pm_addr": "10.7.214.205",
            "capabilities": "profile:control,boot_option:local",
            "pm_type": "pxe_ipmitool",
            "pm_user": "rcon"
        },
        {
            "pm_password": "3tango11",
            "name": "compute-0",
            "pm_addr": "10.7.214.208",
            "capabilities": "profile:computeovrs,boot_option:local",
            "pm_type": "pxe_ipmitool",
            "pm_user": "rcon"
        },
		{
            "pm_password": "3tango11",
            "name": "compute-1",
            "pm_addr": "10.7.214.209",
            "capabilities": "profile:computeovrs,boot_option:local",
            "pm_type": "pxe_ipmitool",
            "pm_user": "rcon"
        }
    ]
}

5. Download the Nuage vsp rpms and Create a Yum Repository

Follow the instructions in download the nuage vsp rpms and create a yum repository in order to identify the required packages.

The below is a full list of all the required packages divided by groups:
Nuage common packages: nuage-bgp, nuage-metadata-agent, nuage-openstack-neutronclient, nuage-puppet-modules, python-openvswitch-nuage
Nuage ovrs packages: mstflint, nuage-openvswitch-ovrs
Extra packages (Of Red Hat official repositories) : boost-filesystem, boost-regex, boost-system, python3, python3-libs, python3-pip, python3-setuptools, createrepo, deltarpm, libconfig, lldpad, perl-JSON, perl-Sys-Syslog, protobuf-c, python-deltarpm, glibc, keyutils-libs, krb5-libs, libcom_err, libgcc, libselinux, libsepol, libstdc++, libverto, nss-softokn-freebl, openssl-libs, pcre, zlib

Note: In order to get the most updated licensed Nuage packages please contact Nokia at:
nuage-smartnic-support@list.nokia.com

Creating local repository for Nuage packages:

  1.  Create two directories per each package group, as follows:
    /var/www/html/nuage_osp13_6.0.5/nuage_common/
    /var/www/html/nuage_osp13_6.0.5/nuage_ovrs

  2. Copy each package to its dedicated group directory, as the list of required packages specifies.

  3. Create a file named nuage_ospd13.repo and place it in: 
    /home/stack/nuage-ospdirector/image-patching/nuage_image_patching_scripts
    Example of nuage_ospd13.repo file is available in this section.

  4. Install createrepo util.

    What is createrepo ?

    For more information, see http://yum.baseurl.org/wiki/RepoCreate.html

  5. Execute 'createrepo' per each directory created in step 1.

The below is nuage_ospd13.repo file example for this deployment guide:

nuage_ospd13.repo 
########################################################################################################################
############## Repositories which user want to enable by default on the overcloud image while patching #################
########################################################################################################################
 
# This repo should contain following packages:
# nuage-bgp
# nuage-puppet-modules
# nuage-openstack-neutronclient
# python-openvswitch-nuage
# nuage-metadata-agent

[nuage_common]
name=nuage_osp13_6.0.5
baseurl=http://192.168.24.1/nuage_osp13_6.0.5/nuage_common
enabled=1
gpgcheck=0
 
########################################################################################################################
############## Repositories which we recommend user to disable by default ##############################################
########################################################################################################################
 
# This repo should contain following packages:
# nuage-openvswitch-ovrs
# mstflint

[nuage_ovrs]
name=nuage_osp13_6.0.5
baseurl=http://192.168.24.1/nuage_osp13_6.0.5/nuage_ovrs
enabled=1
gpgcheck=0

6. Modify the Overcloud Image

Follow the instructions in modify the overcloud image after following the prior settings in this section.

Note: Make sure to copy overcloud image from ~/images/overcloud-full.qcow2 to:
/home/stack/nuage-ospdirector/image-patching/nuage_image_patching_scripts

Prior settings

  • Edit /home/stack/nuage-ospdirector/image-patching/nuage_image_patching_scripts/nuage_patching_config.yaml file with:
    RepoFile: 'nuage_ospd13.repo'
    DeploymentType: ["ovrs"]
  • In order to download the extra packages, edit the nuage_patching_config.yaml file and add Red Hat account details.
    This way the overcloud image can fetch the packages from the Red Hat subscribed repositories. 
    Here is an example for the required parameters:
    RhelUserName: 'abc'
RhelPassword: '***'
RhelPool: '1234567890123445'

7. Create the Dataplane Roles and Update the Node Profiles

Follow the instructions in Create the dataplane roles and update the node profiles.

Generate the nuage_roles_data.yaml file as follows:

Openstack overcloud roles generate --roles-path /home/stack/nuage-tripleo-heat-templates/roles -o /home/stack/nuage-tripleo-heat-templates/templates/nuage_roles_data.yaml Controller ComputeOvrs

The below is an a node-info.yaml file example for this deployment guide:

node-info.yaml 
parameter_defaults:
  OvercloudControllerFlavor: control
  ControllerCount: 1
  OvercloudComputeOvrsFlavor: computeovrs
  ComputeOvrsCount: 2

8. Generate a CMS ID for Openstack Installation

Follow the instructions in Generate a cms-id for openstack deployment.

Note:

  • Make sure your OSP Director has a network connectivity to VSD server through public NIC 1 as the Solution Physical design / wiring section describes.   

  • Command line example for generating CMS-ID:

    python configure_vsd_cms_id.py --server 10.7.215.50:8443 --serverauth csproot:csproot --organization csp --auth_resource /me --serverssl True --base_uri /nuage/api/v5_0

9. Customize the Environment Files

Follow the instructions in customize the environment files.

Note:

Based on Solution Physical design / wiring section.

  • In this deployment VSD IP for tenant NIC 1 is 192.168.10.101 
  • In this deployment VSC IP for tenant NIC 1 is 192.168.10.102

The below is a neutron-nuage-config.yaml and nova-nuage-config.yaml file examples for this deployment guide:

neutron-nuage-config.yaml 
# A Heat environment file which can be used to enable a
# a Neutron Nuage backend on the controller, configured via puppet
resource_registry:
  OS::TripleO::Services::NeutronDhcpAgent: OS::Heat::None
  OS::TripleO::Services::NeutronL3Agent: OS::Heat::None
  OS::TripleO::Services::NeutronMetadataAgent: OS::Heat::None
  OS::TripleO::Services::NeutronOvsAgent: OS::Heat::None
  OS::TripleO::Services::ComputeNeutronOvsAgent: OS::Heat::None

  # Override the NeutronMl2PluginBase to use Nuage inside Docker container
  OS::TripleO::Docker::NeutronMl2PluginBase: ../puppet/services/neutron-plugin-ml2-nuage.yaml
  OS::TripleO::Services::NeutronCorePlugin: ../docker/services/neutron-plugin-ml2-nuage.yaml

parameter_defaults:
  NeutronNuageNetPartitionName: 'default_name'
  NeutronNuageVSDIp: '192.168.10.101:8443'
  NeutronNuageVSDUsername: 'csproot'
  NeutronNuageVSDPassword: 'csproot'
  NeutronNuageVSDOrganization: 'csp'
  NeutronNuageBaseURIVersion: 'v6_0'
  NeutronNuageCMSId: '4e455172-014d-4fa7-8a99-63efa0f6d79c'
  UseForwardedFor: true
  NeutronPluginMl2PuppetTags: 'neutron_plugin_ml2,neutron_plugin_nuage'
  NeutronServicePlugins: 'NuagePortAttributes,NuageAPI,NuageL3,trunk'
  NeutronDBSyncExtraParams: '--config-file /etc/neutron/neutron.conf --config-file /etc/neutron/plugins/ml2/ml2_conf.ini --config-file /etc/neutron/plugins/nuage/plugin.ini'
  NeutronTypeDrivers: 'vxlan'
  NeutronNetworkType: 'vxlan'
  NeutronMechanismDrivers: "nuage"
  NeutronPluginExtensions: 'nuage_subnet,nuage_port,port_security'
  NeutronFlatNetworks: '*'
  NovaOVSBridge: 'alubr0'
  NeutronMetadataProxySharedSecret: 'NuageNetworksSharedSecret'
  NeutronTunnelIdRanges: ''
  NeutronNetworkVLANRanges: ''
  NeutronVniRanges: '1001:2000'
  HeatEnginePluginDirs: ['/usr/lib/python2.7/site-packages/nuage-heat/']
  HorizonCustomizationModule: 'nuage_horizon.customization'
  HorizonVhostExtraParams:
    add_listen: True
    priority: 10
    access_log_format: '%a %l %u %t \"%r\" %>s %b \"%%{}{Referer}i\" \"%%{}{User-Agent}i\"'
    directories: [{'allow_override': ['None'], 'path': '/usr/lib/python2.7/site-packages/nuage_horizon', 'require': 'all granted', 'options': ['FollowSymLinks']}]
    aliases: [{'alias': '%{root_url}/static/nuage', 'path': '/usr/lib/python2.7/site-packages/nuage_horizon/static'}, {'alias': '%{root_url}/static', 'path': '/usr/share/openstack-dashboard/static'}]
  ControllerExtraConfig:
    neutron::config::server_config:
      DEFAULT/ipam_driver:
        value: nuage_internal
      DEFAULT/enable_snat_by_default:
        value: false
    neutron::config::plugin_nuage_config:
      RESTPROXY/nuage_pat:
        value: legacy_disabled
nova-nuage-config.yaml 
# Heat environment file which can be used to enable
# Nuage backend on the compute, configured via puppet
resource_registry:
  OS::TripleO::Services::ComputeNeutronCorePlugin: ../puppet/services/nuage-compute-vrs.yaml

parameter_defaults:
  NuageActiveController: '192.168.10.102'
  NuageStandbyController: '0.0.0.0'
  NuageBridgeMTU: '9000'
  NovaIPv6: 'True'
  NuageMetadataProxySharedSecret: 'NuageNetworksSharedSecret'
  NuageNovaApiEndpoint: 'internalURL'
  NovaOVSBridge: 'alubr0'
  NovaComputeLibvirtType: 'kvm'
  NovaComputeLibvirtVifDriver: 'nova.virt.libvirt.vif.LibvirtGenericVIFDriver'
  # VrsExtraConfigs can be used to configure extra parameters in /etc/default/openvswitch
  # For example to set "NETWORK_UPLINK_INTF" see below sample:
  # VrsExtraConfigs: {"NETWORK_UPLINK_INTF": "eno1"}
  VrsExtraConfigs: {}
  NuageGpgKeys: []

10. Assign Nuage Controller Role

Follow the instructions in nuage controller role controller.

To extract controller node-uuid assist this command

openstack baremetal node list

11. Assign Nuage Offloaded VRS Nodes Role

Follow the instructions in offload vrs role computeovrs.

Notes:

  • To extract compute node-uuid, run the below command

    openstack baremetal node list
  • For ConnectX firmeware, see firmware downloads.
  • Execute 'createrepo' for firmware directory /var/www/html/FW_<VERSION>, once the firmware bin file is downloaded.
  • This deployment uses firmware version 16.26.1040
  • This deployment uses two Nvidia Bond interfaces. Ports are specifed in mellanox-environment.yaml file. 
    Interfaces names must suite the deployed target servers OS naming. 

The below is a mellanox-environment.yaml file example for this deployment guide:

mellanox-environment.yaml 
resource_registry:
  OS::TripleO::ComputeSriov::NodeUserData: ../firstboot/mellanox_fw_update.yaml

parameter_defaults:
  ################
  # Nic's params #
  ################
  MellanoxTenantPort1: "ens2f0"
  MellanoxTenantPort2: "ens2f1"
  MellanoxInfraPort1: "ens1f0"
  MellanoxInfraPort2: "ens1f1"

  ########################
  # FIRST Boot FW config #
  ########################

  BIN_DIR_URL: "http://192.168.24.1/FW_16_26_1040/"
  NUM_OF_VFS: 16
  SRIOV_EN: True
  ESWITCH_IPV4_TTL_MODIFY_ENABLE: True
  PRIO_TAG_REQUIRED_EN: True

12. Network Isolation

Follow the instructions in the network isolation section: "VF lag with VLANs for Nvidia ConnectX-5 NICs".

Notes:

  • BondInterfaceOvsOptions is set to bond LACP in network-environment.yaml 

  • ovs-hw-offload.yaml contains the names of Nvidia Tenat interfaces for computeovrs servers.
  • Make sure ovs-hw-offload.yaml contains the following line (See file example):

    resource_registry:
    OS::TripleO::Services::NuageComputeOvrs: ../puppet/services/nuage-compute-ovrs.yaml

  • ovs-hw-offload.yaml NodeRootPassword is set with default OS root password. 


The below are computeovrs.yaml,  controller.yaml, ovs-hw-offload.yaml network-environment.yaml file examples for this deployment guide:

network-environment.yaml 
parameter_defaults:
  # This section is where deployment-specific configuration is done
  # CIDR subnet mask length for provisioning network
  ControlPlaneSubnetCidr: '24'
  # Gateway router for the provisioning network (or Undercloud IP)
  ControlPlaneDefaultRoute: 192.168.24.1
  EC2MetadataIp: 192.168.24.1  # Generally the IP of the Undercloud
  # Customize the IP subnets to match the local environment
  StorageNetCidr: '172.16.1.0/24'
  StorageMgmtNetCidr: '172.16.3.0/24'
  InternalApiNetCidr: '172.16.2.0/24'
  TenantNetCidr: '192.168.10.0/24'
  ExternalNetCidr: '172.16.4.0/24'
  ManagementNetCidr: '10.0.1.0/24'
  # Customize the VLAN IDs to match the local environment
  StorageNetworkVlanID: 30
  StorageMgmtNetworkVlanID: 40
  InternalApiNetworkVlanID: 20
  TenantNetworkVlanID: 50
  ExternalNetworkVlanID: 10
  ManagementNetworkVlanID: 60
  StorageAllocationPools: [{'start': '172.16.1.200', 'end': '172.16.1.250'}]
  StorageMgmtAllocationPools: [{'start': '172.16.3.200', 'end': '172.16.3.250'}]
  InternalApiAllocationPools: [{'start': '172.16.2.200', 'end': '172.16.2.250'}]
  TenantAllocationPools: [{'start': '192.168.10.50', 'end': '192.168.10.80'}]
  # Leave room if the external network is also used for floating IPs
  ExternalAllocationPools: [{'start': '172.16.4.100', 'end': '172.16.4.150'}]
  ManagementAllocationPools: [{'start': '10.0.1.4', 'end': '10.0.1.250'}]
  # Gateway routers for routable networks
  ExternalInterfaceDefaultRoute: '172.16.4.1'
  # Define the DNS servers (maximum 2) for the overcloud nodes
  DnsServers: ["8.8.8.8","10.7.77.192"]
  # The tunnel type for the tenant network (vxlan or gre). Set to '' to disable tunneling.
  NeutronTunnelTypes: 'vxlan'
  # Customize bonding options, e.g. "mode=4 lacp_rate=1 updelay=1000 miimon=100"
  # for Linux bonds w/LACP, or "bond_mode=active-backup" for OVS active/backup.
  BondInterfaceOvsOptions: "mode=4 miimon=100 lacp_rate=1 xmit_hash_policy=layer3+4"
  TenantPortEthtoolOptions: "-L ${DEVICE} combined 40; -X ${DEVICE} equal 38; -U ${DEVICE} flow-type tcp4 src-port 6633 action 39 loc 1; -U ${DEVICE} flow-type tcp4 src-port 7406 action 39 loc 2; -U ${DEVICE} flow-type tcp4 src-port 39090 action 39 loc 3;-U ${DEVICE} flow-type tcp4 src-port 179 action 39 loc 4"
computeovrs.yaml 
heat_template_version: queens
description: >
  Software Config to drive os-net-config with 2 bonded nics on a bridge with VLANs attached for the ComputeSriov role.
parameters:
  ControlPlaneIp:
    default: ''
    description: IP address/subnet on the ctlplane network
    type: string
  StorageIpSubnet:
    default: ''
    description: IP address/subnet on the storage network
    type: string
  StorageMgmtIpSubnet:
    default: ''
    description: IP address/subnet on the storage_mgmt network
    type: string
  InternalApiIpSubnet:
    default: ''
    description: IP address/subnet on the internal_api network
    type: string
  TenantIpSubnet:
    default: ''
    description: IP address/subnet on the tenant network
    type: string
  ExternalIpSubnet:
    default: ''
    description: IP address/subnet on the external network
    type: string
  ManagementIpSubnet:
    default: ''
    description: IP address/subnet on the management network
    type: string
  BondInterfaceOvsOptions:
    default: bond_mode=active-backup
    description: 'The ovs_options or bonding_options string for the bond
      interface. Set things like lacp=active and/or bond_mode=balance-slb
      for OVS bonds or like mode=4 for Linux bonds using this option.'
    type: string
  StorageNetworkVlanID:
    default: 30
    description: Vlan ID for the storage network traffic.
    type: number
  StorageMgmtNetworkVlanID:
    default: 40
    description: Vlan ID for the storage_mgmt network traffic.
    type: number
  InternalApiNetworkVlanID:
    default: 20
    description: Vlan ID for the internal_api network traffic.
    type: number
  TenantNetworkVlanID:
    default: 50
    description: Vlan ID for the tenant network traffic.
    type: number
  ExternalNetworkVlanID:
    default: 10
    description: Vlan ID for the external network traffic.
    type: number
  ManagementNetworkVlanID:
    default: 60
    description: Vlan ID for the management network traffic.
    type: number
  ControlPlaneDefaultRoute: # Override this via parameter_defaults
    description: The default route of the control plane network.
    type: string
  ExternalInterfaceDefaultRoute:
    default: '192.168.24.2'
    description: default route for the external network
    type: string
  ControlPlaneSubnetCidr: # Override this via parameter_defaults
    default: '24'
    description: The subnet CIDR of the control plane network.
    type: string
  DnsServers: # Override this via parameter_defaults
    default: []
    description: A list of DNS servers (2 max for some implementations) that will be added to resolv.conf.
    type: comma_delimited_list
  EC2MetadataIp: # Override this via parameter_defaults
    description: The IP address of the EC2 metadata server.
    type: string
  MellanoxTenantPort1:
    description: Mellanox Tenant Port1
    type: string
  MellanoxTenantPort2:
    description: Mellanox Tenant Port2
    type: string
  MellanoxInfraPort1:
    description: Mellanox Infra Port1
    type: string
  MellanoxInfraPort2:
    description: Mellanox Infra Port2
    type: string
  TenantPortEthtoolOptions:
          description: Port ethtool options for Control Plane Protectionn
          type: string
resources:
  OsNetConfigImpl:
    type: OS::Heat::SoftwareConfig
    properties:
      group: script
      config:
        str_replace:
          template:
            get_file: /usr/share/openstack-tripleo-heat-templates/network/scripts/run-os-net-config.sh
          params:
            $network_config:
              network_config:
              - type: interface
                name: eno1
                use_dhcp: false
                addresses:
                - ip_netmask:
                    list_join:
                    - /
                    - - get_param: ControlPlaneIp
                      - get_param: ControlPlaneSubnetCidr
                routes:
                - ip_netmask: 169.254.169.254/32
                  next_hop:
                    get_param: EC2MetadataIp
                - default: true
                  next_hop:
                    get_param: ControlPlaneDefaultRoute
              # Linux bond for Offload VRS Compute nodes
              - type: linux_bond
                name: infra-bond
                dns_servers:
                  get_param: DnsServers
                bonding_options:
                  get_param: BondInterfaceOvsOptions
                members:
                - type: interface
                  name:
                    get_param: MellanoxInfraPort1
                - type: interface
                  name:
                    get_param: MellanoxInfraPort2
              - type: vlan
                device: infra-bond
                vlan_id:
                  get_param: StorageNetworkVlanID
                addresses:
                - ip_netmask:
                    get_param: StorageIpSubnet
              - type: vlan
                device: infra-bond
                vlan_id:
                  get_param: InternalApiNetworkVlanID
                addresses:
                - ip_netmask:
                    get_param: InternalApiIpSubnet
              - type: linux_bond
                name: tenant-bond
                dns_servers:
                  get_param: DnsServers
                bonding_options:
                  get_param: BondInterfaceOvsOptions
                members:
                - type: sriov_pf
                  name:
                    get_param: MellanoxTenantPort1
                  link_mode: switchdev
                  numvfs: 8
                  promisc: true
                  use_dhcp: false
                  primary: true
                  ethtool_opts:
                    get_param: TenantPortEthtoolOptions
                - type: sriov_pf
                  name:
                    get_param: MellanoxTenantPort2
                  link_mode: switchdev
                  numvfs: 8
                  promisc: true
                  use_dhcp: false
                  ethtool_opts:
                    get_param: TenantPortEthtoolOptions
              - type: vlan
                device: tenant-bond
                vlan_id:
                  get_param: TenantNetworkVlanID
                addresses:
                - ip_netmask:
                    get_param: TenantIpSubnet
outputs:
  OS::stack_id:
    description: The OsNetConfigImpl resource.
    value:
      get_resource: OsNetConfigImpl
controller.yaml 
heat_template_version: queens
description: >
  Software Config to drive os-net-config with 2 bonded nics on a bridge with VLANs attached for the Controller role.
parameters:
  ControlPlaneIp:
    default: ''
    description: IP address/subnet on the ctlplane network
    type: string
  StorageIpSubnet:
    default: ''
    description: IP address/subnet on the storage network
    type: string
  StorageMgmtIpSubnet:
    default: ''
    description: IP address/subnet on the storage_mgmt network
    type: string
  InternalApiIpSubnet:
    default: ''
    description: IP address/subnet on the internal_api network
    type: string
  TenantIpSubnet:
    default: ''
    description: IP address/subnet on the tenant network
    type: string
  ExternalIpSubnet:
    default: ''
    description: IP address/subnet on the external network
    type: string
  ManagementIpSubnet:
    default: ''
    description: IP address/subnet on the management network
    type: string
  BondInterfaceOvsOptions:
    default: bond_mode=active-backup
    description: 'The ovs_options or bonding_options string for the bond
      interface. Set things like lacp=active and/or bond_mode=balance-slb
      for OVS bonds or like mode=4 for Linux bonds using this option.'
    type: string
  StorageNetworkVlanID:
    default: 30
    description: Vlan ID for the storage network traffic.
    type: number
  StorageMgmtNetworkVlanID:
    default: 40
    description: Vlan ID for the storage_mgmt network traffic.
    type: number
  InternalApiNetworkVlanID:
    default: 20
    description: Vlan ID for the internal_api network traffic.
    type: number
  TenantNetworkVlanID:
    default: 50
    description: Vlan ID for the tenant network traffic.
    type: number
  ExternalNetworkVlanID:
    default: 10
    description: Vlan ID for the external network traffic.
    type: number
  ManagementNetworkVlanID:
    default: 60
    description: Vlan ID for the management network traffic.
    type: number
  ControlPlaneDefaultRoute: # Override this via parameter_defaults
    description: The default route of the control plane network.
    type: string
  ExternalInterfaceDefaultRoute:
    default: '192.168.24.2'
    description: default route for the external network
    type: string
  ControlPlaneSubnetCidr: # Override this via parameter_defaults
    default: '24'
    description: The subnet CIDR of the control plane network.
    type: string
  DnsServers: # Override this via parameter_defaults
    default: []
    description: A list of DNS servers (2 max for some implementations) that will be added to resolv.conf.
    type: comma_delimited_list
  EC2MetadataIp: # Override this via parameter_defaults
    description: The IP address of the EC2 metadata server.
    type: string
  MellanoxInfraPort1:
    description: Mellanox Infra Port1
    type: string
  MellanoxInfraPort2:
    description: Mellanox Infra Port2
    type: string

resources:
  OsNetConfigImpl:
    type: OS::Heat::SoftwareConfig
    properties:
      group: script
      config:
        str_replace:
          template:
            get_file: /usr/share/openstack-tripleo-heat-templates/network/scripts/run-os-net-config.sh
          params:
            $network_config:
              network_config:
              - type: interface
                name: eno1
                use_dhcp: false
                addresses:
                - ip_netmask:
                    list_join:
                    - /
                    - - get_param: ControlPlaneIp
                      - get_param: ControlPlaneSubnetCidr
                routes:
                - ip_netmask: 169.254.169.254/32
                  next_hop:
                    get_param: EC2MetadataIp
              # Linux bond for Controller nodes
              - type: linux_bond
                name: infra-bond
                dns_servers:
                  get_param: DnsServers
                bonding_options:
                  get_param: BondInterfaceOvsOptions
                members:
                - type: interface
                  name:
                    get_param: MellanoxInfraPort1
                - type: interface
                  name:
                    get_param: MellanoxInfraPort2
              - type: vlan
                device: infra-bond
                vlan_id:
                  get_param: StorageNetworkVlanID
                addresses:
                - ip_netmask:
                    get_param: StorageIpSubnet
              - type: vlan
                device: infra-bond
                vlan_id:
                  get_param: StorageMgmtNetworkVlanID
                addresses:
                - ip_netmask:
                    get_param: StorageMgmtIpSubnet
              - type: vlan
                device: infra-bond
                vlan_id:
                  get_param: InternalApiNetworkVlanID
                addresses:
                - ip_netmask:
                    get_param: InternalApiIpSubnet
              - type: vlan
                device: infra-bond
                vlan_id:
                  get_param: TenantNetworkVlanID
                addresses:
                - ip_netmask:
                    get_param: TenantIpSubnet
              - type: vlan
                device: infra-bond
                vlan_id:
                  get_param: ExternalNetworkVlanID
                addresses:
                - ip_netmask:
                    get_param: ExternalIpSubnet
                routes:
                - default: true
                  next_hop:
                    get_param: ExternalInterfaceDefaultRoute

outputs:
  OS::stack_id:
    description: The OsNetConfigImpl resource.
    value:
      get_resource: OsNetConfigImpl
ovs-hw-offload.yaml 
# A Heat environment file that enables OVS Hardware Offload in the overcloud.
resource_registry:
  OS::TripleO::Services::NuageComputeOvrs: ../puppet/services/nuage-compute-ovrs.yaml

parameter_defaults:

  NovaSchedulerDefaultFilters: ['RetryFilter','AvailabilityZoneFilter','RamFilter','ComputeFilter','ComputeCapabilitiesFilter','ImagePropertiesFilter','ServerGroupAntiAffinityFilter','ServerGroupAffinityFilter','PciPassthroughFilter']
  NovaSchedulerAvailableFilters: ["nova.scheduler.filters.all_filters","nova.scheduler.filters.pci_passthrough_filter.PciPassthroughFilter"]
  NodeRootPassword: "3tango"

  # Kernel arguments for ComputeOvrs node
  ComputeOvrsParameters:
    KernelArgs: "intel_iommu=on iommu=pt pci=realloc"
    #NOTE: By default TunedProfileName is set to "cpu-partitioning" in ovrs-role.yaml.
    # If IsolCpusList is not set in your environment, then leave TunedProfileName below to set to empty string.
    # If planning on setting IsolCpusList in your environment
    #   1. You can comment the below line to set TunedProfileName to "cpu-partitioning" or
    #   2. You can pass your custom Tuned Profile to apply to the host
    TunedProfileName: ""
    OvsHwOffload: True
    # Mapping of SR-IOV PF interface to neutron physical_network.
    # In case of Vxlan/GRE physical_network should be null.
    # In case of flat/vlan the physical_network should as configured in neutron.
    NovaPCIPassthrough:
      - devname: "ens2f0"
        physical_network: null
      - devname: "ens2f1"
        physical_network: null

13. Deploy the Overcloud

Follow the instructions in Deploy the overcloud, Section 3: "For VF lag with VLANs for CX-5 NICs".

Overcloud deploy full command
openstack overcloud deploy --templates -r /home/stack/nuage-tripleo-heat-templates/templates/nuage_roles_data.yaml -e /home/stack/templates/node-info.yaml -e /home/stack/templates/overcloud_images.yaml -e /home/stack/nuage-tripleo-heat-templates/environments/nuage_overcloud_images.yaml -e /usr/share/openstack-tripleo-heat-templates/environments/network-isolation.yaml -e /home/stack/nuage-tripleo-heat-templates/environments/network-environment.yaml -e /home/stack/nuage-tripleo-heat-templates/environments/net-bond-with-vlans.yaml -e /home/stack/nuage-tripleo-heat-templates/environments/neutron-nuage-config.yaml -e /home/stack/nuage-tripleo-heat-templates/environments/nova-nuage-config.yaml -e /home/stack/nuage-tripleo-heat-templates/environments/mellanox-environment.yaml -e /home/stack/nuage-tripleo-heat-templates/environments/ovs-hw-offload.yaml -e /usr/share/openstack-tripleo-heat-templates/environments/host-config-and-reboot.yaml --ntp-server 192.168.24.1

For your convenience, all template files are attached:

yaml_config.tar.gz

Files OSP directory locations:

/home/stack/nuage-tripleo-heat-templates/templates/nuage_roles_data.yaml
/home/stack/templates/node-info.yaml
/home/stack/templates/overcloud_images.yaml
/home/stack/nuage-tripleo-heat-templates/environments/nuage_overcloud_images.yaml
/home/stack/nuage-tripleo-heat-templates/environments/network-environment.yaml
/home/stack/nuage-tripleo-heat-templates/environments/net-bond-with-vlans.yaml
/home/stack/nuage-tripleo-heat-templates/environments/neutron-nuage-config.yaml
/home/stack/nuage-tripleo-heat-templates/environments/nova-nuage-config.yaml
/home/stack/nuage-tripleo-heat-templates/environments/mellanox-environment.yaml
/home/stack/nuage-tripleo-heat-templates/environments/ovs-hw-offload.yaml
/home/stack/nuage-tripleo-heat-templates/network/config/bond-with-vlans/controller.yaml
/home/stack/nuage-tripleo-heat-templates/network/config/bond-with-vlans/computeovrs.yaml

Application Deployment and Configuration

This document does not cover VSD and VSC server installation and configuration. It is assumed those are up and running based on Nuage documentation.

Please  review this exception before going forward:

Make sure VSD user which is defined as 'NeutronNuageVSDUsername' in neutron-nuage-config.yaml is also defined as part of the CMS group in VSD. Not doing so results in VM spawning failure.

To do so, get the username fromneutron-nuage-config.yaml (in this example the username is 'csproot'), open VSD WebUI 'Platform Configuration' and move the user right to the 'Assigned Users' side. 

  1. Open VSD WebUI

  2. Click on the Platform Configuration icon in the tor-right corner


  3. Click Settings


  4. Click on the Groups icon on the left, choose CMS Group and then press the Assign button


  5. Now select the desired user, click the arrow to move it to the right and press Submit.


Finally, you should get to this: 

Configure a Basic Flavor

Flavor configuration
source overcloudrc
openstack flavor create m1.small --id 1 --ram 4096 --disk 20 --vcpus 4

Create VSD a Managed Network

  1. Enter OpenStack WebUi.

    Successful deployment points you to the Openstack dashboard.

    /home/stack/overcloudrc file contains user and password login credentials.


  2. Under Openstack WebUI Project→Network Tab, click on "create network":

  3. Give it a name and set the desired checkboxes:

  4. Choose Subnet Type to be "VSD Managed Subnet (auto)"



  5. Choose Organization, Domain, Zone and Subnet from the WebUI options. It is assumed that these parameters are already configured in the VSD together with active policy.


  6. Subnet fields are populated automatically. Click next and the network creation is completed.

Create 'switchdev' Port

Create a port with 'switchdev' capabilities from CLI and attach it to the desired network created as describe in the previous section.

Port creation
openstack port create direct1 --vnic-type=direct --no-security-group --network Net1 --binding-profile '{"capabilities":["switchdev"]}'
openstack port create direct2 --vnic-type=direct --no-security-group --network Net1 --binding-profile '{"capabilities":["switchdev"]}'

Launch to VMs on Separate Hypervisors 

Launch VMs from CLI and insert the port ID and image name per each VM:

Launch VMs
openstack server create  --flavor m1.small --image <image_name> --nic port-id=<port1_ID> --availability-zone nova:overcloud-computesriov-0.localdomain vm1
openstack server create  --flavor m1.small --image <image_name> --nic port-id=<port2_ID> --availability-zone nova:overcloud-computesriov-1.localdomain vm2

Deployment Verification

Initiate ping request between the two running VMs to see if you get a reply.

Set continuous ping requests and move to hypervisor CLI for checking offloading mechanism.

This example shows offloaded flows of outgoing and returning ICMP request/return packets.

The VXLAN interface holds the incoming flow and the representor port holds the outgoing flow.

Dump OVRS Offloaded Rules

[root@overcloud-computesriov-0 ~]sudo ovs-dpctl dump-flows type=offloaded --name
in_port(ens2f0_6),eth(dst=fa:16:3e:81:a2:b2),eth_type(0x8100),vlan(vid=0,pcp=0),encap(eth_type(0x0800),ipv4(src=10.44.0.0/255.255.0.0,dst=0.0.0.0/1.0.0.0,proto=1,tos=0/0x3)), packets:2808, bytes:438048, used:0.120s, actions:set(tunnel(tun_id=0x8cfcb6,dst=192.168.10.52,tp_dst=4789,flags(key))),vxlan_sys_4789
tunnel(tun_id=0x8cfcb6,src=192.168.10.52,dst=192.168.10.50,ttl=64,flags(+key)),in_port(vxlan_sys_4789),eth(dst=fa:16:3e:01:78:98),eth_type(0x8100),vlan(vid=0,pcp=0),encap(eth_type(0x0800),ipv4(dst=0.0.0.0/1.0.0.0,proto=1)), packets:2808, bytes:286416, used:0.130s, actions:ens2f0_6

Dump TC VXLAN Interface Offloaded Rules

[root@overcloud-computesriov-0 ~]tc filter show dev vxlan_sys_4789: parent ffff:  
filter protocol 802.1Q pref 4 flower chain 0 handle 0x8fb
  vlan_id 0
  vlan_prio 0
  vlan_ethtype ip
  dst_mac 68:54:ed:00:0d:0b
  src_mac c6:17:f1:d8:4e:32
  eth_type ipv4
  ip_proto udp
  ip_tos 0x0/ff
  ip_ttl 0x3d/ff
  dst_ip 17.0.11.108
  src_ip 49.0.11.108
  dst_port 10001
  src_port 1000
  in_hw in_hw_count 1
        action order 1: tunnel_key  set
        src_ip 0.0.0.0
        dst_ip 3.1.1.102
        key_id 7708303
        dst_port 4789
        nocsum
        ttl 64 pipe
         index 9472 ref 1 bind 1

        action order 2:  pedit action pipe keys 5
         index 9472 ref 1 bind 1
         key #0  at ipv4+8: val 3c000000 mask 00ffffff
         key #1  at eth+4: val 0000fa7b mask ffff0000
         key #2  at eth+8: val aea37243 mask 00000000
         key #3  at eth+0: val 00000301 mask 00000000
         key #4  at eth+4: val 01660000 mask 0000ffff

        action order 3: csum (iph, udp) action pipe
        index 9472 ref 1 bind 1

        action order 4: mirred (Egress Redirect to device vxlan_sys_4789) stolen
        index 9472 ref 1 bind 1
        cookie c81b9f7c3c4cac8437509ba179c02c83

Dump TC Representor Interface Offloaded Rules

[root@overcloud-computesriov-0 ~]tc filter show dev ens2f0_6: parent ffff:  
filter protocol 802.1Q pref 2 flower chain 0
 filter protocol 802.1Q pref 4 flower chain 0
 filter protocol 802.1Q pref 5 flower chain 0
 filter protocol 802.1Q pref 6 flower chain 0
 filter protocol 802.1Q pref 7 flower chain 0
 filter protocol 802.1Q pref 7 flower chain 0 handle 0x1
   vlan_id 0
   vlan_ethtype ip
   dst_mac fa:16:3e:81:a2:b2
   eth_type ipv4
   ip_proto icmp
   ip_tos 0/3
   dst_ip 10.44.27.17/1.0.0.0
   src_ip 10.44.27.20/16
   in_hw
         action order 1: tunnel_key set
         src_ip 0.0.0.0
         dst_ip 192.168.10.52
         key_id 9239734
         dst_port 4789
         nocsum
         ttl 64 pipe
         index 1 ref 1 bind 1
  
         action order 2: mirred (Egress Redirect to device vxlan_sys_4789) stolen
         index 1 ref 1 bind 1


Performance Testing

Performance test deployment is performed over a single compute node and the XIA server, both connected to Nvidia switch.

The compute node is a device under test running TestPMD over two VFs and is connected to bond VF-LAG interface.

IXIA injects flat UDP packets towards Nvidia switch which encapsulate the packets over the VXLAN tunnel on top of logical LACP bond.

VF-LAG bond intercepts the packets and forwards them to VF1 and VF2 connected to testPMD VM, which forwards the packets from VF1 to VF2 and from VF2 to VF1 back to IXIA server.

Number of flows is determined by the range of source and destination IPs.

Test Results

The below table presents bandwidth and packets per second while each column maintains different number of flows in hardware starting from 64 to 400,000.


About the Author


Amir Zeidner

For the past several years, Amir has worked as a Solutions Architect primarily in the Telco space, leading advanced solutions to answer 5G, NFV, and SDN networking infrastructures requirements. Amir’s expertise in data plane acceleration technologies, such as Accelerated Switching and Network Processing (ASAP²) and DPDK, together with a deep knowledge of open source cloud-based infrastructures, allows him to promote and deliver unique end-to-end NVIDIA Networking solutions throughout the Telco world.







Related Documents



Notice

This document is provided for information purposes only and shall not be regarded as a warranty of a certain functionality, condition, or quality of a product. Neither NVIDIA Corporation nor any of its direct or indirect subsidiaries and affiliates (collectively: “NVIDIA”) make any representations or warranties, expressed or implied, as to the accuracy or completeness of the information contained in this document and assumes no responsibility for any errors contained herein. NVIDIA shall have no liability for the consequences or use of such information or for any infringement of patents or other rights of third parties that may result from its use. This document is not a commitment to develop, release, or deliver any Material (defined below), code, or functionality.
NVIDIA reserves the right to make corrections, modifications, enhancements, improvements, and any other changes to this document, at any time without notice.
Customer should obtain the latest relevant information before placing orders and should verify that such information is current and complete.
NVIDIA products are sold subject to the NVIDIA standard terms and conditions of sale supplied at the time of order acknowledgement, unless otherwise agreed in an individual sales agreement signed by authorized representatives of NVIDIA and customer (“Terms of Sale”). NVIDIA hereby expressly objects to applying any customer general terms and conditions with regards to the purchase of the NVIDIA product referenced in this document. No contractual obligations are formed either directly or indirectly by this document.
NVIDIA products are not designed, authorized, or warranted to be suitable for use in medical, military, aircraft, space, or life support equipment, nor in applications where failure or malfunction of the NVIDIA product can reasonably be expected to result in personal injury, death, or property or environmental damage. NVIDIA accepts no liability for inclusion and/or use of NVIDIA products in such equipment or applications and therefore such inclusion and/or use is at customer’s own risk.
NVIDIA makes no representation or warranty that products based on this document will be suitable for any specified use. Testing of all parameters of each product is not necessarily performed by NVIDIA. It is customer’s sole responsibility to evaluate and determine the applicability of any information contained in this document, ensure the product is suitable and fit for the application planned by customer, and perform the necessary testing for the application in order to avoid a default of the application or the product. Weaknesses in customer’s product designs may affect the quality and reliability of the NVIDIA product and may result in additional or different conditions and/or requirements beyond those contained in this document. NVIDIA accepts no liability related to any default, damage, costs, or problem which may be based on or attributable to: (i) the use of the NVIDIA product in any manner that is contrary to this document or (ii) customer product designs.
No license, either expressed or implied, is granted under any NVIDIA patent right, copyright, or other NVIDIA intellectual property right under this document. Information published by NVIDIA regarding third-party products or services does not constitute a license from NVIDIA to use such products or services or a warranty or endorsement thereof. Use of such information may require a license from a third party under the patents or other intellectual property rights of the third party, or a license from NVIDIA under the patents or other intellectual property rights of NVIDIA.
Reproduction of information in this document is permissible only if approved in advance by NVIDIA in writing, reproduced without alteration and in full compliance with all applicable export laws and regulations, and accompanied by all associated conditions, limitations, and notices.
THIS DOCUMENT AND ALL NVIDIA DESIGN SPECIFICATIONS, REFERENCE BOARDS, FILES, DRAWINGS, DIAGNOSTICS, LISTS, AND OTHER DOCUMENTS (TOGETHER AND SEPARATELY, “MATERIALS”) ARE BEING PROVIDED “AS IS.” NVIDIA MAKES NO WARRANTIES, EXPRESSED, IMPLIED, STATUTORY, OR OTHERWISE WITH RESPECT TO THE MATERIALS, AND EXPRESSLY DISCLAIMS ALL IMPLIED WARRANTIES OF NONINFRINGEMENT, MERCHANTABILITY, AND FITNESS FOR A PARTICULAR PURPOSE. TO THE EXTENT NOT PROHIBITED BY LAW, IN NO EVENT WILL NVIDIA BE LIABLE FOR ANY DAMAGES, INCLUDING WITHOUT LIMITATION ANY DIRECT, INDIRECT, SPECIAL, INCIDENTAL, PUNITIVE, OR CONSEQUENTIAL DAMAGES, HOWEVER CAUSED AND REGARDLESS OF THE THEORY OF LIABILITY, ARISING OUT OF ANY USE OF THIS DOCUMENT, EVEN IF NVIDIA HAS BEEN ADVISED OF THE POSSIBILITY OF SUCH DAMAGES. Notwithstanding any damages that customer might incur for any reason whatsoever, NVIDIA’s aggregate and cumulative liability towards customer for the products described herein shall be limited in accordance with the Terms of Sale for the product.

Trademarks
NVIDIA, the NVIDIA logo, and Mellanox are trademarks and/or registered trademarks of NVIDIA Corporation and/or Mellanox Technologies Ltd. in the U.S. and in other countries. Other company and product names may be trademarks of the respective companies with which they are associated.

Copyright
© 2022 NVIDIA Corporation & affiliates. All Rights Reserved.