Created on Aug 14, 2022
Scope
The following Reference Deployment Guide (RDG) demonstrates the process of deploying and configuring NVIDIA accelerated networking with Red Hat OpenShift Container Platform (OCP) cluster version 4.10.18. The deployment is performed on a single VMware vSphere 7.0.3 cluster over NVIDIA® accelerated end-to-end 25/100 Gbp/s Ethernet solution. This setup is capable of running RDMA and DPDK-based applications. VMware’s vSAN over RDMA will be used as a shared storage for the Red Hat OpenShift Container Platform workloads.
Abbreviations and Acronyms
Term | Definition | Term | Definition |
|---|---|---|---|
| DAC | Direct Attached Cable | RDMA | Remote Direct Memory Access |
DHCP | Dynamic Host Configuration Protocol | RoCE | RDMA over Converged Ethernet |
| DPDK | Data Plane Development Kit | SDN | Software-Defined Network |
| CNI | Container Network Interface | SDS | Software-Defined Storage |
| IB | InfiniBand | SR-IOV | Single Root I/O Virtualization |
| NFD | Node Feature Discovery Operator | VDS | vSphere Distributed Switch |
| NOS | Network Operation System | VF | Virtual Function |
| OCP | Red Hat® OpenShift Container Platform® | VM | Virtual Machine |
Introduction
Provisioning Red Hat OCP cluster for running RDMA and DPDK-based workloads over VMware vSphere environment may prove to be challenging. Proper design, software and hardware components selection could be a gating task toward a successful deployment.
This guide provides step-by-step instructions on how to deploy a Red Hat OCP cluster by using OCP Assisted Installer on a single vSphere cluster, including technology overview, design, components selection and deployment steps. The Assisted Installer is a user-friendly installation solution offered on the Red Hat Hybrid Cloud Console. The Assisted Installer focuses on bare metal and vSphere infrastructures.
VMware’s vSANoRDMA is now fully qualified, and available as of the ESXi 7.0 U2 release, making it ready for deployments.
In this document, we will be using the NVIDIA Network Operator, which is responsible for deploying and configuring, with a Host Device Network mode. This allows to run RDMA and DPDK workloads on an OCP Cluster Worker Node.
References
- What is RDMA?
- RDMA over Converged Ethernet (RoCE)
- ConnectX® Ethernet Driver for VMware® ESXi Server
Product Documentation for OpenShift Container Platform Container Platform 4.10
- VMware vSAN 7 Design Guide
- NVIDIA Network Operator
Solution Architecture
Key Components and Technologies
Flexible form-factors with 16 to 128 physical ports, supporting 1GbE through 400GbE speeds.
Based on a ground-breaking silicon technology optimized for performance and scalability, NVIDIA Spectrum switches are ideal for building high-performance, cost-effective, and efficient Cloud Data Center Networks, Ethernet Storage Fabric, and Deep Learning Interconnects.
NVIDIA combines the benefits of NVIDIA Spectrum™ switches, based on an industry-leading application-specific integrated circuit (ASIC) technology, with a wide variety of modern network operating system choices, including NVIDIA Cumulus® Linux, SONiC and NVIDIA Onyx®.
NVIDIA® Cumulus® Linux is the industry's most innovative open network operating system that allows you to automate, customize, and scale your data center network like no other.
NVIDIA ConnectX SmartNICs
10/25/40/50/100/200 and 400G Ethernet Network Adapters
The industry-leading NVIDIA® ConnectX® family of smart network interface cards (SmartNICs) offer advanced hardware offloads and accelerations.
NVIDIA Ethernet adapters enable the highest ROI and lowest Total Cost of Ownership for hyperscale, public and private clouds, storage, machine learning, AI, big data, and telco platforms.
NVIDIA LinkX Cables
The NVIDIA® LinkX® product family of cables and transceivers provides the industry’s most complete line of 10, 25, 40, 50, 100, 200, and 400GbE in Ethernet and 100, 200 and 400Gb/s InfiniBand products for Cloud, HPC, hyperscale, Enterprise, telco, storage and artificial intelligence, data center applications.
RDMA
RDMA is a technology that allows computers in a network to exchange data without involving the processor, cache or operating system of either computer.
Like locally based DMA, RDMA improves throughput and performance and frees up compute resources.
(RoCE) or InfiniBand over Ethernet (IBoE)[1] is a network protocol that allows remote direct memory access (RDMA) over an Ethernet network. It does this by encapsulating an InfiniBand (IB) transport packet over Ethernet.
- NVIDIA Network Operator
An analog to the NVIDIA GPU Operator, the NVIDIA Network Operator simplifies scale-out network design for Kubernetes by automating aspects of network deployment and configuration that would otherwise require manual work. It loads the required drivers, libraries, device plugins, and CNIs on any cluster node with an NVIDIA network interface. Paired with the NVIDIA GPU Operator, the Network Operator enables GPUDirect RDMA, a key technology that accelerates cloud-native AI workloads by orders of magnitude. The NVIDIA Network Operator uses Kubernetes CRD and the Operator Framework to provision the host software needed for enabling accelerated networking.
Logical Design
The described setup uses one vSphere cluster that includes 4 ESXi servers connected to two NVIDIA® Spectrum® SN2010 Ethernet switches (Management, Ingress and Egress traffic), and one NVIDIA® Spectrum® SN2100 Ethernet switch (High speed vSAN, RDMA and DPDK traffic).
vCenter, Infra Node and Router VMs will be placed on the same cluster.
Network Design
The Cluster networking is configured with the following networks:
Corporate (Management) Network (VLAN 1610 - 192.168.1.x/24)
The network responsible for connectivity between the ESXi VMkernel Management interface, vCenter and other Management VMs.DHCP and DNS services are required. The components installation and configuration are not covered in this guide.- vMotion Network (VLAN 1611 – 192.168.11.0/24)
This is where the ESXi vMotion VMkernel interfaces will reside. - OCP Network (VLAN 1677 - 192.168.77.0/24) – OpenShift Management network.
- vSAN Network (VLAN 1630 – 192.168.30.0/24)
This is where the ESXi vSAN VMkernel interfaces will reside. - RDMA Network (VLAN 1631 – 192.168.3.0/24)
This is where the ESXi vSAN VMkernel interfaces will reside.
vSphere Distributed Switches Design
Internet access for the OCP
In our guide, an OCP cluster is deployed in a non-Air Gap environment and Internet access is required.
OpenShift Container Platform Networking
Software Stack Components
This guide assumes that the following software and drivers are installed:
- VMware ESXi 7.0.3, build 17630552
- VMware vCenter 7.0.3, build 17694817
- Distributed Switch 7.0.3
- RedHat OCP 4.10.18
- NVIDIA® ConnectX® Driver for VMware ESXi Server v4.21.71.101
- NVIDIA® ConnectX®-6DX FW version 22.32.2004
- NVIDIA® ConnectX®-6LX FW version 26.32.1010
- Network Operating System (NOS): NVIDIA Cumulus™ v5.1
Bill of Materials
The following hardware setup is utilized in the vSphere environment described in this guide:
Cluster Compute/Storage:
VM | CPU | MEM | DISK |
Compute Cluster vCenter (based on Small) | 4 | 20GB | 48GB |
Infra Node | 2 | 4GB | 50GB |
Control Node x 3 | 8 | 32GB | 120GB |
Worker Node x 3 | 8 | 32GB | 120GB |
Deployment and Configuration
The following topics are outside the scope of this document:
- ESXi Installation
- vCenter Installation
- Virtual Data Center Configuration
- vSphere Cluster Configuration
- Adding ESXi Hosts to Clusters
Sample of our lab environment:
Wiring
This document covers highly available VMware vSphere cluster deployment.
Network/Fabric
Prerequisites
- Switch OS
NVIDIA® Cumulus® Linux v5.1 - Network adapter
NVIDIA ConnectX-6 Lx and ConnectX-6 Dx adapter cards.
Latest firmware and Driver versions. For more information, please see:
Hosts Network Configuration
This table provides details on the ESXi server, switches names and their network configuration.
Server | Server Name | IP and NICs | |
|---|---|---|---|
| High-Speed Ethernet Network | Management Network | ||
| ESXi-01 | sl01w01esx11 | vmk1: 192.168.11.111 (vMotion) vmk2: 192.168.130.111 (vSAN) | vmk0: 192.168.1.111 From Corporate DHCP (reserved) |
| ESXi-02 | sl01w01esx12 | vmk1: 192.168.11.112 (vMotion) vmk2: 192.168.130.112 (vSAN) | vmk0: 192.168.1.112 From Corporate DHCP (reserved) |
| ESXi-03 | sl01w01esx13 | vmk1: 192.168.11.113(vMotion) vmk2: 192.168.130.113 (vSAN) | vmk0: 192.168.1.113 From Corporate DHCP (reserved) |
| ESXi-04 | sl01w01esx14 | vmk1: 192.168.11.114(vMotion) vmk2: 192.168.130.114 (vSAN) | vmk0: 192.168.1.114 From Corporate DHCP (reserved) |
Leaf-01 | clx-swx-033 | - | 192.168.1.233 |
| Leaf-02 | clx-swx-034 | - | 192.168.1.234 |
| Higt-Speed-01 | clx-swx-035 | - | 192.168.1.235 |
| vCenter (VM) | sl01w01vc01 | - | 192.168.1.25 From Corporate DHCP (reserved) |
Infra Node (VM) | sl01w01depl | - | 192.168.77.201 |
| Router Node (VM) | sl01w01pfsense | - | WAN - 192.168.1.202 LAN - 192.168.77.254 |
| Control Node 01 (VM) | control01 | - | 192.168.77.11 From Infra Node DHCP (reserved) |
| Control Node 01 (VM) | control02 | - | 192.168.77.12 From Infra Node DHCP (reserved) |
| Control Node 01 (VM) | control03 | - | 192.168.77.13 From Infra Node DHCP (reserved) |
| Worker Node 01 (VM) | worker01 | - | 192.168.77.21 From Infra Node DHCP (reserved) |
| Worker Node 02 (VM) | worker02 | - | 192.168.77.22 From Infra Node DHCP (reserved) |
| Worker Node 03 (VM) | worker03 | - | 192.168.77.23 From Infra Node DHCP (reserved) |
Network Switch Configuration
ESXi to Leaf Connection
Port Channel and VLAN Configuration
Run the following commands on both Leaf NVIDIA SN2010 switches in the Supervisor Cluster to configure port channel and VLAN.
Sample for Leaf-01 switch configuration (named clx-swx-033):
cumulus@clx-swx-033:mgmt:~$sudo nv set system hostname clx-swx-033 cumulus@clx-swx-033:mgmt:~$sudo nv set interface lo ip address 10.10.10.1/32 cumulus@clx-swx-033:mgmt:~$sudo nv set interface swp1-22 type swp cumulus@clx-swx-033:mgmt:~$sudo nv set interface swp1-4 bridge domain br_default cumulus@clx-swx-033:mgmt:~$sudo nv set bridge domain br_default vlan 1610 cumulus@clx-swx-033:mgmt:~$sudo nv set bridge domain br_default vlan 1611 cumulus@clx-swx-033:mgmt:~$sudo nv set bridge domain br_default vlan 1677 cumulus@clx-swx-033:mgmt:~$sudo nv set vrf default router static 0.0.0.0/0 address-family ipv4-unicast cumulus@clx-swx-033:mgmt:~$sudo nv set interface peerlink bond member swp21-22 cumulus@clx-swx-033:mgmt:~$sudo nv set mlag mac-address 44:38:39:BE:EF:AA cumulus@clx-swx-033:mgmt:~$sudo nv set mlag backup 10.10.10.2 cumulus@clx-swx-033:mgmt:~$sudo nv set mlag peer-ip linklocal cumulus@clx-swx-033:mgmt:~$sudo nv config apply cumulus@clx-swx-033:mgmt:~$sudo nv config save
Sample for Leaf-02 switch configuration (named clx-swx-034):
cumulus@clx-swx-034:mgmt:~$sudo nv set system hostname clx-swx-034 cumulus@clx-swx-034:mgmt:~$sudo nv set interface lo ip address 10.10.10.2/32 cumulus@clx-swx-034:mgmt:~$sudo nv set interface swp1-22 type swp cumulus@clx-swx-034:mgmt:~$sudo nv set interface swp1-4 bridge domain br_default cumulus@clx-swx-034:mgmt:~$sudo nv set bridge domain br_default vlan 1610 cumulus@clx-swx-034:mgmt:~$sudo nv set bridge domain br_default vlan 1611 cumulus@clx-swx-034:mgmt:~$sudo nv set bridge domain br_default vlan 1677 cumulus@clx-swx-034:mgmt:~$sudo nv set vrf default router static 0.0.0.0/0 address-family ipv4-unicast cumulus@clx-swx-034:mgmt:~$sudo nv set interface peerlink bond member swp21-22 cumulus@clx-swx-034:mgmt:~$sudo nv set mlag mac-address 44:38:39:BE:EF:AA cumulus@clx-swx-034:mgmt:~$sudo nv set mlag backup 10.10.10.1 cumulus@clx-swx-034:mgmt:~$sudo nv set mlag peer-ip linklocal cumulus@clx-swx-034:mgmt:~$sudo nv config apply cumulus@clx-swx-034:mgmt:~$sudo nv config save
Port Channel and VLAN Configuration on High Speed NVIDIA SN2100 Switch
Run the following commands on the High Speed switch in the vSphere Cluster to configure port channel and VLAN.
Sample for the Higt-Speed-01(clx-swx-035):
cumulus@clx-swx-035:mgmt:~$sudo nv set system hostname clx-swx-035 cumulus@clx-swx-035:mgmt:~$sudo nv set interface swp1-16 type swp cumulus@clx-swx-035:mgmt:~$sudo nv set interface swp1-16 bridge domain br_default cumulus@clx-swx-035:mgmt:~$sudo nv set bridge domain br_default vlan 1630 cumulus@clx-swx-035:mgmt:~$sudo nv set bridge domain br_default vlan 1631 cumulus@clx-swx-035:mgmt:~$sudo nv set qos roce enable on cumulus@clx-swx-035:mgmt:~$sudo nv config apply cumulus@clx-swx-035:mgmt:~$sudo nv config save
Enable RDMA over Converged Ethernet Lossless (with PFC and ETS) on a High Speed SN2100 Switch
RoCE transport is utilized to accelerate vSAN networking. To get the highest possible results, the network is configured to be lossless.
Run the following commands on the High-Speed switch to configure a lossless network for NVIDIA Cumulus:
cumulus@clx-swx-035:mgmt:~$sudo nv set qos roce enable on cumulus@clx-swx-035:mgmt:~$sudo nv config apply cumulus@clx-swx-035:mgmt:~$sudo nv config save
To check RoCE configuration, run the following command:
ESXi Host and vSphere Cluster Configuration
Prerequisites
- Host BIOS
Verify that an SR-IOV supported server platform is being used and review the BIOS settings in the server platform vendor documentation to enable SR-IOV in the BIOS.
Verify that VT-d/IOMMU is enabled in the BIOS. Physical server configuration
All ESXi servers must have the same PCIe placement for the NIC and expose the same interface name.Experience with Kubernetes/OpenShift
Familiarization with the Kubernetes Cluster architecture is essential.- Privileges
Verify that you have an account privilege to read and create the required resources. Using an account that has global administrative privileges is the simplest way to access all the necessary permissions on the cluster. For more information, see Chapter 20. Installing on vSphere OpenShift Container Platform Container Platform 4.10 | Red Hat Customer Portal - Verify that NTP is configured and works properly in your environment.
- Create and configure 2 VMware VDS by using the following document - How-to: Configure a vSphere Distributed Switch with NVIDIA network fabric.
Two VDS will be used in the environment:SL-WL01-DS01 with the following port groups:
SL-WL01-MGMT-VLAN1610-DPG
SL-WL01-vMotion-VLAN611-DPG
SL-WL01-OCP-VLAN1677-DPG
SL-WL01-DS02 with the following port groups:
SL-WL01-vSAN-VLAN1630-DPG
SL-WL01-RDMA-VLAN1631-DPG
Sample:
Create and configure a VMware vSAN RDMA cluster by using the following document - RDG: VMware vSAN over RoCE on VMware vSphere 7.0 U3.
Enable SR-IOV on an ESXi host
NVIDIA Network Operator leverages Kubernetes CRDs and Operator SDK to manage networking-related components to enable fast networking and RDMA for workloads in the OCP cluster. The fast network is a secondary network of the K8s cluster for applications that require high bandwidth or low latency.
In OCP Cluster it is possible to use Dynamic DirectPath I/O to assign a multiple PCI passthrough or SR-IOV devices to a Kubernetes Workload VM.
To make it work, enable SR-IOV capability on a ConnectX-6 Dx network adapter.
To Enable SR-IOV:- Launch the vSphere Web Client, and connect to a vCenter Server instance.
- Navigate to a an ESXi host, and select ConfigurePCI Devices. Click on ALL PCI DEVICES. Click on Filter.
Type Mellanox and click on Vendor Name.
Select a ConnectX-6 Dx NIC.
- Click on CONFIGURE SR-IOV.
- Enable SR-IOV, and set the number of Virtual functions (VF). Click OK.
- Click on PASSTHROUGH-ENABLED DEVICES to verify that 8 VFs were enabled.
Create 6 VMs for 3 Control and 3 Worker OpenShift Clusters with the following configuration:
On the Customize hardware tab, click VM Options → Advanced.- Click Boot Options. Set the Boot Delay to 9999 milliseconds.
Scroll down and click Edit Configuration. On the Configuration Parameters window, click Add Configuration Params. Define the following parameter name and value:
disk.EnableUUID: Specify TRUE.
Click OK twice for Control VMs.
For worker VM, click OK. Go back to the Virtual Hardware tab and add an additional SR-IOV Network Adapter by clicking on ADD NEW DEVICE and selecting Network Adapter.
On New Network click on Browse...
Select an RDMA network port group. Click OK.
Open New Network and select Adapter Type: SR-IOV passthrough and Physical Function (for example, vmnic4 was configured previously with VFs). Click OK.
- Click Boot Options. Set the Boot Delay to 9999 milliseconds.
OCP Cluster Installation
Infra node Prerequisites
DHCP and DNS services are required for OCP installation. In this case, an Infra Node is used to run the services.
An additional ntp-servers configuration is required for a DHCP service on an OCP management network.
The components installation and configuration are not covered in this guide.
For more information, see Networking requirements for user-provisioned infrastructure.
The OCP cluster installation requires two VIP addresses:
- The API address is used to access the cluster API.
- The Ingress address is used for cluster ingress traffic.
These IP addresses must be provided to the installation program when installing the OCP cluster.
In our environment, we use the following IPs and DNS records:
| Component | IP | Record | Description |
|---|---|---|---|
| Kubernetes API | 192.168.77.210 | api.ocp4.vwd.clx | A DNS A/AAAA or CNAME record, and a DNS PTR record, to identify the API load balancer. These records must be resolvable by both clients external to the cluster and from all the nodes within the cluster. |
| Ingress | 192.168.77.211 | *.apps.ocp4.vwd.clx | A wildcard DNS A/AAAA or CNAME record that refers to the application ingress load balancer. The application ingress load balancer targets the machines that run the Ingress Controller PODs. The Ingress Controller PODs run on the compute machines by default. These records must be resolvable by both clients external to the cluster and from all the nodes within the cluster. For example, |
| control01 | 192.168.77.11 | control01.ocp4.vwd.clx | DNS A/AAAA or CNAME records and DNS PTR records to identify each machine for the control plane nodes. These records must be resolvable by the nodes within the cluster. |
| control02 | 192.168.77.12 | control02.ocp4.vwd.clx | DNS A/AAAA or CNAME records and DNS PTR records to identify each machine for the control plane nodes. These records must be resolvable by the nodes within the cluster. |
| control03 | 192.168.77.13 | control03.ocp4.vwd.clx | DNS A/AAAA or CNAME records and DNS PTR records to identify each machine for the control plane nodes. These records must be resolvable by the nodes within the cluster. |
| worker01 | 192.168.77.21 | worker01.ocp4.vwd.clx | DNS A/AAAA or CNAME records and DNS PTR records to identify each machine for the worker nodes. These records must be resolvable by the nodes within the cluster. |
| worker02 | 192.168.77.22 | worker02.ocp4.vwd.clx | DNS A/AAAA or CNAME records and DNS PTR records to identify each machine for the worker nodes. These records must be resolvable by the nodes within the cluster. |
worker03 | 192.168.77.23 | worker03.ocp4.vwd.clx | DNS A/AAAA or CNAME records and DNS PTR records to identify each machine for the worker nodes. These records must be resolvable by the nodes within the cluster. |
Generating an SSH Public Key
To generate a key, please run the following command on the Infra Node:
# ssh-keygen
OCP Cluster Deployment
You can install the OpenShift Container Platform on vSphere by using the Assisted Installer, an Installer-Provisioned or User-Provisioned Infrastructure.
This document describes OCP on-premises VMs installation using the Assisted Installer.
Follow the steps outlined in the section to install OpenShift on vSphere.
- Open a web browser on Infra Node (in this case, AlmaLinux with UI is used), connect the Red Hat Hybrid Cloud site and log into the console with your credentials.
- Using the left menu bar, select OpenShift:
- Click on Create Cluster.
- Select Datacenter and scroll down the window.
- Select vSphere.
- Select the Assisted Installer installation type.
- In the Cluster details open window, provide Cluster Name and Base Domain. From the drop list, select the OpenShift version that you wish to install, and click on Next.
Example:
- In the Host discovery window, click on Add hosts.
- In the Add hosts window, select the Full Image file option. Generate, copy and paste an SSH public key (the key was generated in Infra Node in our case), and click on Generate Discovery ISO.
Examples:
- In the opened window, click on Download Discovery ISO.
- Login to the vCenter.
- Navigate to your control01 VM.
- Attach the Download Discovery ISO to your VM.
- Launch Remote Console.
- And Power On a VM.
- Click ESC, and select CD-ROM Drive in the Boot Menu.
- Click Enter.
- Repeat Steps 12-17 for 5 additional VMs.
- Go back to the RedHat Hybrid Cloud Console web page. After 1-2 minutes, you will be able to see all 6 hosts in "Ready" status.
Review the CPU, Memory and Disk hosts configuration.
Select Role for each Host and click Next.
Example:
- In the Networking opened window, provide the Machine network, API IP and Ingress IP. Click Next.
Example:
- Review the configuration and click on Install cluster.
The OCP Installation process will start and will take approximately 30 minutes.
- Once the installation process is complete, you will be able to Download the kubeconfig file.
Copy the file on your Infra Node to the /root/.kube/config file.
Infra Node Console[root@sl01lw01depl ~]# cp kubeconfig /root/.kube/config [oot@sl01lw01depl ~]# export KUBECONFIG=/root/.kube/config
- To connect to the installed cluster, use the following provided information:
To verify that the OCP cluster is installed properly, open the Infra Node console and open a web browser with the provided Web Console URL, Username and Password.
Make sure that the Cluster Status is ok.
Using the left menu bar, expand the Compute section and select the Nodes Status. 
Click on each node to see that all PODs on the node are running without errors.
To check the cluster status from the ssh console:
Go to the Infra Node ssh console, and run the following command to see the nodes status:
[root@sl01lw01depl ocp4-helpernode]# oc get nodes -o wide NAME STATUS ROLES AGE VERSION INTERNAL-IP EXTERNAL-IP OS-IMAGE KERNEL-VERSION CONTAINER-RUNTIME control01.ocp4.vwd.clx Ready master 2d v1.23.5+3afdacb 192.168.77.11 <none> Red Hat Enterprise Linux CoreOS 410.84.202206080346-0 (Ootpa) 4.18.0-305.49.1.el8_4.x86_64 cri-o://1.23.3-3.rhaos4.10.git5fe1720.el8 control02.ocp4.vwd.clx Ready master 2d v1.23.5+3afdacb 192.168.77.12 <none> Red Hat Enterprise Linux CoreOS 410.84.202206080346-0 (Ootpa) 4.18.0-305.49.1.el8_4.x86_64 cri-o://1.23.3-3.rhaos4.10.git5fe1720.el8 control03.ocp4.vwd.clx Ready master 2d v1.23.5+3afdacb 192.168.77.13 <none> Red Hat Enterprise Linux CoreOS 410.84.202206080346-0 (Ootpa) 4.18.0-305.49.1.el8_4.x86_64 cri-o://1.23.3-3.rhaos4.10.git5fe1720.el8 worker01.ocp4.vwd.clx Ready worker 2d v1.23.5+3afdacb 192.168.77.21 <none> Red Hat Enterprise Linux CoreOS 410.84.202206080346-0 (Ootpa) 4.18.0-305.49.1.el8_4.x86_64 cri-o://1.23.3-3.rhaos4.10.git5fe1720.el8 worker02.ocp4.vwd.clx Ready worker 2d v1.23.5+3afdacb 192.168.77.22 <none> Red Hat Enterprise Linux CoreOS 410.84.202206080346-0 (Ootpa) 4.18.0-305.49.1.el8_4.x86_64 cri-o://1.23.3-3.rhaos4.10.git5fe1720.el8 worker03.ocp4.vwd.clx Ready worker 2d v1.23.5+3afdacb 192.168.77.23 <none> Red Hat Enterprise Linux CoreOS 410.84.202206080346-0 (Ootpa) 4.18.0-305.49.1.el8_4.x86_64 cri-o://1.23.3-3.rhaos4.10.git5fe1720.el8
To see all running cluster operators status, run the following command:
[root@sl01lw01depl ocp4-helpernode]# oc get clusteroperators NAME VERSION AVAILABLE PROGRESSING DEGRADED SINCE MESSAGE authentication 4.10.18 True False False 135m baremetal 4.10.18 True False False 2d cloud-controller-manager 4.10.18 True False False 2d cloud-credential 4.10.18 True False False 2d cluster-autoscaler 4.10.18 True False False 2d config-operator 4.10.18 True False False 2d console 4.10.18 True False False 2d csi-snapshot-controller 4.10.18 True False False 2d dns 4.10.18 True False False 2d etcd 4.10.18 True False False 2d image-registry 4.10.18 True False False 2d ingress 4.10.18 True False False 2d insights 4.10.18 True False False 2d kube-apiserver 4.10.18 True False False 2d kube-controller-manager 4.10.18 True False False 2d kube-scheduler 4.10.18 True False False 2d kube-storage-version-migrator 4.10.18 True False False 2d machine-api 4.10.18 True False False 2d machine-approver 4.10.18 True False False 2d machine-config 4.10.18 True False False 2d marketplace 4.10.18 True False False 2d monitoring 4.10.18 True False False 2d network 4.10.18 True False False 2d node-tuning 4.10.18 True False False 2d openshift-apiserver 4.10.18 True False False 2d openshift-controller-manager 4.10.18 True False False 24h openshift-samples 4.10.18 True False False 2d operator-lifecycle-manager 4.10.18 True False False 2d operator-lifecycle-manager-catalog 4.10.18 True False False 2d operator-lifecycle-manager-packageserver 4.10.18 True False False 2d service-ca 4.10.18 True False False 2d storage 4.10.18 True False False 2d
To see all running PODs status, run the following command:
Installing the Node Feature Discovery (NFD) Operator
The OpenShift Node Feature Discovery (NFD) manages the detection of hardware features and their configuration in the OpenShift Container Platform. The NFD labels the host with node-specific attributes. Therefore, the NFD is a prerequisite for the NVIDIA GPU Operator. As a cluster administrator, you can install the NFD Operator using the Red Hat OperatorHub catalog in the OpenShift Container Platform web console.
Using the left menu bar, expand the Operators section, and select the OperatorHub.
Use the search bar to search for the Node Feature Discovery. Two items should be found as results.
Select the operator that is tagged as the RedHat operator. This is the version supported by Red Hat.
Click Install.
The next menu allows you to determine how and where the operator will be installed. You would rarely have to change any of these options.
Click Install to continue with the defaults.
- Wait while the NFD operator is being installed. Once you see the “Installed operator - ready for use” status, click View Operator. Notice that you are now in the openshift-nfd project that was created as part of the operator installation.
Click the Create Instance button to create a new NFD object.
The next menu provides options to configure how the NFD operator will scan your cluster. For the NFD operator, you would rarely have to change any of the default settings. Click Create to instantiate the NFD resource on your cluster.
You will be able to see the installation progress in the Infra Node ssh console. It can take 5 minutes for the NFD installation.
In the horizontal menu bar, select NodeFeatureDiscovery. You should be able to see one listed instance titled nfd-instance. Wait for the status to show that the deployment is finished.
Enabling a Cluster-wide Entitlement
The NVIDIA Network Operator deploys several PODs that manage and enable NVIDIA High Speed Networking for use in the OpenShift Container Platform. Some of these PODs require packages that are not available by default in the Universal Base Image (UBI) that the OpenShift Container Platform uses. To make packages available, you must enable cluster-wide entitled container builds in OpenShift. To do that, follow the instructions in Enabling a Cluster-wide Entitlement.
By running oc get mcp/worker command. You can see MachineConfigPool object monitor that the MCP is updated, not updating or degraded, so all the MachineConfig resources have been successfully applied to the nodes and you can proceed to validate the cluster. 
Installing the NVIDIA Network Operator
Please see the below steps for Network Operator deployment using Web UI.
Before starting, create a namespace for the Network Operator Resources by running [root@sl01lw01depl ocp4-helpernode]# oc create ns nvidia-operator-resources
Using the left menu bar, expand the Operators section, and select the OperatorHub.
Use the search bar to search for NVIDIA. Two items should be found.
Select the NVIDIA Network Operator.
- Click Install.
- The next menu allows you to define how and where the operator will be installed. You would rarely have to change any of these options.
Click Install to continue with the default values.
- Wait while the NVIDIA Network operator is being installed. Once you see the “Installed operator - ready for use” status, click View Operator.
- Click the Create Instance button to create a new NicClusterPolicy object.
Click on YAML view, and change to the following:
NicClusterPolicy sampleapiVersion: mellanox.com/v1alpha1 kind: NicClusterPolicy metadata: name: nic-cluster-policy spec: ofedDriver: image: mofed livenessProbe: initialDelaySeconds: 30 periodSeconds: 30 readinessProbe: initialDelaySeconds: 10 periodSeconds: 30 repository: nvcr.io/nvidia/mellanox startupProbe: initialDelaySeconds: 10 periodSeconds: 20 upgradePolicy: autoUpgrade: false drain: deleteEmptyDir: true enable: true force: true podSelector: '' timeoutSeconds: 300 maxParallelUpgrades: 1 version: 5.7-1.0.2.0 sriovDevicePlugin: image: sriov-network-device-plugin repository: ghcr.io/k8snetworkplumbingwg version: a765300344368efbf43f71016e9641c58ec1241b config: | { "resourceList": [ { "resourcePrefix": "nvidia.com", "resourceName": "host_dev", "selectors": { "vendors": ["15b3"], "isRdma": true } } ] }Click Create.
- Wait for the status to show State ready.
- In the horizontal menu bar, select HostDeviceNetwork, and click on Create HostDeviceNetwork to create a new object.
Click on YAML view, and change to the following:
HostDeviceNetwork sampleapiVersion: mellanox.com/v1alpha1 kind: HostDeviceNetwork metadata: name: hostdev-net spec: networkNamespace: "default" resourceName: "host_dev" ipam: | { "type": "whereabouts", "range": "192.168.3.225/28" }- Click Create.
- Wait for the status to show State ready.
Running the Application
Now it is possible to deploy a sample POD.
[root@sl01lw01depl ocp4-helpernode]# vim podrdma.yaml
apiVersion: v1
kind: Pod
metadata:
name: rdma-test-pod-1
annotations:
k8s.v1.cni.cncf.io/networks: hostdev-net
spec:
restartPolicy: OnFailure
containers:
- image: harbor.mellanox.com/nbu-solutions-labs/ubuntu-mlnx-inbox:20.04
name: rdma-test-ctr
securityContext:
capabilities:
add: [ "IPC_LOCK" ]
resources:
limits:
nvidia.com/host_dev: '1'
requests:
nvidia.com/host_dev: '1'
command:
- sh
- -c
- sleep infinity
A sample of harbor.mellanox.com/nbu-solutions-labs/ubuntu-mlnx-inbox:20.04 docker file:
FROM ubuntu:20.04
# Ubuntu 20.04 docker container with inbox Mellanox drivers
WORKDIR /tmp/
ENV DEBIAN_FRONTEND=noninteractive
RUN apt-get clean -y && apt-get -y update && apt-get install -y apt-utils udev vim bash && apt-get -y upgrade
RUN apt-get install -y iproute2 rdma-core libibmad5 ibutils ibverbs-utils infiniband-diags perftest \
mstflint strace iputils-ping
RUN ln -fs /usr/share/zoneinfo/America/New_York /etc/localtime
RUN dpkg-reconfigure --frontend noninteractive tzdata && apt-get clean all -y
CMD bash
Run the following command to create the POD:
[root@sl01lw01depl ocp4-helpernode]# oc apply -f podrdma.yaml pod/hostdev-test-pod created
To verify that the POD is running, run:
[root@sl01lw01depl ocp4-helpernode]# oc get pods -o wide NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES rdma-test-pod-1 1/1 Running 0 10s 10.129.3.80 worker02.ocp4.vwd.clx <none> <none>
Get a shell for the running container.
[root@sl01lw01depl ocp4-helpernode]# oc exec -it rdma-test-pod-1 -- bash
Check the available network interfaces in the POD:
root@rdma-test-pod-1:/tmp# rdma link
link mlx5_0/1 state ACTIVE physical_state LINK_UP netdev net1
root@rdma-test-pod-1:/tmp# ip a s
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
valid_lft forever preferred_lft forever
inet6 ::1/128 scope host
valid_lft forever preferred_lft forever
3: eth0@if2890: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc noqueue state UP group default
link/ether 0a:58:0a:81:03:50 brd ff:ff:ff:ff:ff:ff link-netnsid 0
inet 10.129.3.80/23 brd 10.129.3.255 scope global eth0
valid_lft forever preferred_lft forever
inet6 fe80::c05d:6ff:fe08:84ad/64 scope link
valid_lft forever preferred_lft forever
161: net1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default qlen 1000
link/ether 00:50:56:a2:9b:7f brd ff:ff:ff:ff:ff:ff
inet 192.168.3.225/28 brd 192.168.3.239 scope global net1
valid_lft forever preferred_lft forever
inet6 fe80::250:56ff:fea2:9b7f/64 scope link
valid_lft forever preferred_lft forever
Verification
Check the RDMA
To check the RDMA, deploy the second POD. Open an additional console window, and run the following:
[root@sl01lw01depl ocp4-helpernode]# vim podrdma2.yaml
apiVersion: v1
kind: Pod
metadata:
name: rdma-test-pod-2
annotations:
k8s.v1.cni.cncf.io/networks: hostdev-net
spec:
restartPolicy: OnFailure
containers:
- image: harbor.mellanox.com/nbu-solutions-labs/ubuntu-mlnx-inbox:20.04
name: rdma-test-ctr
securityContext:
capabilities:
add: [ "IPC_LOCK" ]
resources:
limits:
nvidia.com/host_dev: '1'
requests:
nvidia.com/host_dev: '1'
command:
- sh
- -c
- sleep infinity
Run the following command to run the second POD:
[root@sl01lw01depl ocp4-helpernode]# oc apply -f podrdma2.yaml pod/hostdev-test-pod-2 created
Verify that two PODs are running on different Worker Nodes.
[root@sl01lw01depl ocp4-helpernode]# oc get pods -o wide NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES rdma-test-pod-1 1/1 Running 0 102s 10.129.3.80 worker02.ocp4.vwd.clx <none> <none> rdma-test-pod-2 1/1 Running 0 83s 10.128.2.18 worker03.ocp4.vwd.clx <none> <none>
As you can see, the first rdma-test-pod-1 POD is running on the worker02 worker, and the second rdma-test-pod-2 POD is running on the worker03 worker.
Get a shell for the second running container.
[root@sl01lw01depl ocp4]# oc exec -it rdma-test-pod-2 -- bash
Check the available network interfaces in the POD.
root@rdma-test-pod-1:/tmp# rdma link
link mlx5_0/1 state ACTIVE physical_state LINK_UP netdev net1
root@hostdev-test-pod-2:/tmp# ip a s
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
valid_lft forever preferred_lft forever
inet6 ::1/128 scope host
valid_lft forever preferred_lft forever
3: eth0@if30: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc noqueue state UP group default
link/ether 0a:58:0a:80:02:12 brd ff:ff:ff:ff:ff:ff link-netnsid 0
inet 10.128.2.18/23 brd 10.128.3.255 scope global eth0
valid_lft forever preferred_lft forever
inet6 fe80::3415:caff:fed9:b936/64 scope link
valid_lft forever preferred_lft forever
26: net1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default qlen 1000
link/ether 00:50:56:a2:2e:a2 brd ff:ff:ff:ff:ff:ff
inet 192.168.3.226/28 brd 192.168.3.239 scope global net1
valid_lft forever preferred_lft forever
inet6 fe80::250:56ff:fea2:2ea2/64 scope link
valid_lft forever preferred_lft forever
Now you can run the ib_write_bw (InfiniBand write bandwidth) tool as part of the Perftest Package by running the following:
Go back to the first running container shell window, and run:
root@rdma-test-pod-1:/tmp# ib_write_bw -F -d mlx5_0 --report_gbits ************************************ * Waiting for client to connect... * ************************************
Back to the additional running container shell window. And run.
root@rdma-test-pod-2:/tmp# ib_write_bw -F 192.168.3.225 -d mlx5_0 --report_gbits
Result:
On Client side.
---------------------------------------------------------------------------------------
RDMA_Write BW Test
Dual-port : OFF Device : mlx5_0
Number of qps : 1 Transport type : IB
Connection type : RC Using SRQ : OFF
TX depth : 128
CQ Moderation : 100
Mtu : 1024[B]
Link type : Ethernet
GID index : 2
Max inline data : 0[B]
rdma_cm QPs : OFF
Data ex. method : Ethernet
---------------------------------------------------------------------------------------
local address: LID 0000 QPN 0x00c8 PSN 0xae9396 RKey 0x020b05 VAddr 0x007f1fd458a000
GID: 00:00:00:00:00:00:00:00:00:00:255:255:192:168:03:226
remote address: LID 0000 QPN 0x00c8 PSN 0x5fc1b8 RKey 0x020b05 VAddr 0x007f1c19487000
GID: 00:00:00:00:00:00:00:00:00:00:255:255:192:168:03:225
---------------------------------------------------------------------------------------
#bytes #iterations BW peak[Gb/sec] BW average[Gb/sec] MsgRate[Mpps]
65536 5000 90.52 90.52 0.172651
---------------------------------------------------------------------------------------
On Server side.
************************************
* Waiting for client to connect... *
************************************
---------------------------------------------------------------------------------------
RDMA_Write BW Test
Dual-port : OFF Device : mlx5_0
Number of qps : 1 Transport type : IB
Connection type : RC Using SRQ : OFF
CQ Moderation : 100
Mtu : 1024[B]
Link type : Ethernet
GID index : 2
Max inline data : 0[B]
rdma_cm QPs : OFF
Data ex. method : Ethernet
---------------------------------------------------------------------------------------
local address: LID 0000 QPN 0x00c8 PSN 0x5fc1b8 RKey 0x020b05 VAddr 0x007f1c19487000
GID: 00:00:00:00:00:00:00:00:00:00:255:255:192:168:03:225
remote address: LID 0000 QPN 0x00c8 PSN 0xae9396 RKey 0x020b05 VAddr 0x007f1fd458a000
GID: 00:00:00:00:00:00:00:00:00:00:255:255:192:168:03:226
---------------------------------------------------------------------------------------
#bytes #iterations BW peak[Gb/sec] BW average[Gb/sec] MsgRate[Mpps]
65536 5000 90.52 90.52 0.172651
---------------------------------------------------------------------------------------
The performance results listed in this document are indicative and should not be considered as formal performance targets for NVIDIA products.
To run DPDK application, please see the following document: RDG: DPDK Applications on SR-IOV Enabled Kubernetes Cluster with NVIDIA Network Operator.
Done!
Authors
| Boris Kovalev Boris Kovalev has worked for the past several years as a Solutions Architect, focusing on NVIDIA Networking/Mellanox technology, and is responsible for complex machine learning, Big Data and advanced VMware-based cloud research and design. Boris previously spent more than 20 years as a senior consultant and solutions architect at multiple companies, most recently at VMware. He has written multiple reference designs covering VMware, machine learning, Kubernetes, and container solutions which are available at the Mellanox Documents website. |
| Vitaliy Razinkov Over the past few years, Vitaliy Razinkov has been working as a Solutions Architect on the NVIDIA Networking team, responsible for complex Kubernetes/OpenShift and Microsoft's leading solutions, research and design. He previously spent more than 25 years in senior positions at several companies. Vitaliy has written several reference design guides on Microsoft technologies, RoCE/RDMA accelerated machine learning in Kubernetes/OpenShift, and container solutions, all of which are available on the NVIDIA Networking Documentation website. |
Related Documents
