NVIDIA Docs Hub  NVIDIA Networking  Networking Solutions  RDG for vSphere Distributed Services Engine (Project Monterey) deployment over NVIDIA BlueField-2 DPU
RDG for vSphere Distributed Services Engine (Project Monterey) deployment over NVIDIA BlueField-2 DPU

On This Page

RDG for vSphere Distributed Services Engine (Project Monterey) deployment over NVIDIA BlueField-2 DPU

Created on Jul 01,2023

Scope

The following R eference D eployment G uide ( RDG ) explains how to install and configure VMware vSphere 8.0b Distributed Services Engine (DSE) with VMware NSX 4.1 version on a single vSphere cluster over Dell NVIDIA® BlueField Data Processing Units (DPU) and end-to-end NVIDIA 25Gbps Ethernet solution. This setup is capable of running all types of workloads included DPDK-based applications. VMware’s vSAN Express Storage Architecture (ESA) will be used as a share storage for the environment.

Abbreviations and Acronyms

Term

Definition

Term

Definition

DAC

Direct Attached Cable

MFT

NVIDIA Mellanox Firmware Tools

DCUI

Direct Console User Interface

NOS

Network Operation System

DHCP

Dynamic Host Configuration Protocol

OOB

Out-of-band

DPDK

Data Plane Development Kit

OSA

Original Storage Architecture

DPU

Data Processing Unit

RDG

R eference D eployment G uide

DSE

Distributed Services Engine

SR-IOV

Single Root Input/Output Virtualization

EDP

Enhanced Data Path

VF

Virtual Function

ESA

Express Storage Architecture

VM

Virtual Machine

Introduction

Contemporary applications are being built using a microservices architecture, where individual services are deployed on servers spread throughout the datacenter. Although this approach has numerous advantages, it also results in an increase in network traffic and places a significant burden on network communications. Tasks associated with traffic routing, network overlays (for Geneve or VXLAN), security (such as a distributed firewall), telemetry, storage, and remote management can consume more than 30% of CPU resources on each virtualized server. This, in turn, can affect the availability of server CPU resources for running end-user workloads.

One way to resolve this inefficiency is to offload network and security services to a DPU (Data Processing Unit) that contains purpose-built engines for accelerating these services. In 2022, VMware launched VMware vSphere® Distributed Services Engine™(previously known as “Project Monterey”) which allows vSphere 8 to offload and accelerate NSX network and security services to a DPU such as NVIDIA BlueField-2. This offload frees up host CPU cores to run end-user workloads while accelerating these services on NVIDIA BlueField-2 engines allows for high performance network, security, and storage services.

Starting with vSphere 8, VMware and NVIDIA introduce a VMware vSphere Distributed Switch and VMware NSX networking running on the NVIDIA BlueField-2 DPU.

The NVIDIA BlueField-2 DPU provides a new control point for scaling infrastructure functions and enables security controls that are separate from the workload domain.

This guide provides step-by-step instructions to deploy and configure VMware ESXi and NVIDIA BlueField-2 DPU, vCenter, and NSX manager to support a DSE environment on a single vSphere cluster including technology overview, design, component selection, and deployment steps.

References

Solution Architecture

Key Components and Technologies

  • NVIDIA Spectrum Ethernet Switches

    Flexible form-factors with 16 to 128 physical ports, supporting 1GbE through 400GbE speeds.

    Based on a ground-breaking silicon technology optimized for performance and scalability, NVIDIA Spectrum switches are ideal for building high-performance, cost-effective, and efficient Cloud Data Center Networks, Ethernet Storage Fabric, and Deep Learning Interconnects.

    NVIDIA combines the benefits of NVIDIA Spectrum™ switches, based on an industry-leading application-specific integrated circuit (ASIC) technology, with a wide variety of modern network operating system choices, including NVIDIA Cumulus® Linux , SONiC and NVIDIA Onyx®.

  • NVIDIA Cumulus Linux

    NVIDIA® Cumulus® Linux is the industry's most innovative open network operating system that allows you to automate, customize, and scale your data center network like no other.

  • BlueField® data processing unit (DPU)

    The NVIDIA® BlueField® data processing unit (DPU) ignites unprecedented innovation for modern data centers. By offloading, accelerating, and isolating a broad range of advanced networking, storage, and security services, BlueField DPUs provide a secure and accelerated infrastructure for any workload in any environment, from cloud to a data center, to an edge. BlueField DPUs combine powerful computing, full infrastructure-on-chip programmability, and high-performance networking for addressing the most demanding workloads.

  • NVIDIA LinkX Cables

    The NVIDIA® LinkX® product family of cables and transceivers provides the industry’s most complete line of 10, 25, 40, 50, 100, 200, and 400GbE in Ethernet and 100, 200 and 400Gb/s InfiniBand products for Cloud, HPC, hyperscale, Enterprise, telco, storage and artificial intelligence, data center applications.

  • VMware vSphere on DPUs , enabled by vSphere Distributed Services Engine modernizes virtual infrastructure by offloading functions from CPU to DPU. It enables modern distributed workloads to take advantage of resource savings, accelerated networking, and enhanced workload security, while also improving DPU lifecycle management with workflows integrated into vSphere.

  • VMware vSphere Distributed Switch (VDS) provides a centralized interface from which you can configure, monitor and administer virtual machine access switching for the entire data center. The VDS provides simplified Virtual Machine network configuration, enhanced network monitoring and troubleshooting capabilities.

  • VMware NSX provides an agile software-defined infrastructure to build cloud-native application environments.

    NSX focuses on providing networking, security, automation, and operational simplicity for emerging application frameworks and architectures that have heterogeneous endpoint environments and technology stacks. NSX Data Center supports cloud-native applications, bare metal workloads, multi-hypervisor environments, public clouds, and multiple clouds. NSX Data Center is designed for management, operation, and consumption by development organizations. NSX Data Center allows IT teams and development teams to select the technologies best suited for their applications.

  • VMware vSAN Express Storage Architecture (ESA) represents a massive step forward in the capabilities of the solution. This is an optional, alternative storage architecture (OSA) to the vSAN original storage architecture also found in vSAN 8. The best way to think of the vSAN Express Storage Architecture is as a new way to process and store data. It is an optional, alternative architecture in vSAN that is designed to achieve all-new levels of efficiency, scalability, and performance. The ESA is optimized to exploit the full potential of the very latest in hardware, and unlocks new capabilities.

  • Distributed Services Engine (DSE) introduces a new data path that is only available with DSE. With vSphere Distributed Services Engine, you can use Uniform pass-through (UPTv2) and/or MUX mode to improve performance and reduce network hops and CPU resources on x86 servers. The administrator can fine-tune the offloading and acceleration behavior using two modes: MUX Mode and UPTv2.

    MUX Mode is the default mode and does some processing on the x86 host. It provides higher flexibility and has fewer requirements than UPTv2.

    UPTv2 completely offloads all network and security processing to the DPU and provides higher performance. It requires Guest Memory Reservation and a specific VMXNET3 driver version. UPTv2 supported guest operating systems for DSE - see Distributed Services Engine (DSE) UPTv2 VMXNET3 driver requirements.

    Both modes provide accelerated networking services for VLAN and overlay networking, as well as offloading TCP Segment Offload (TSO), Checksum, and Receive Side Scaling (RSS).

    To configure the new EDP with either MUX Mode or UPTv2, you need to deploy NSX Manager.

    UPTv2 requires publishing virtualized device functions (VFs), like SR-IOV. The number of VMs that can be connected depends on the number of VFs published by the device vendor. UPTv2 supports core vSphere features like vMotion for VMs that use it.

Logical Design

The setup uses one vSphere cluster that includes 3 ESXi servers connected to single high speed ethernet switch for VM Applications, vMotion and High speed vSAN traffics.
The Management traffic uses dedicated 1GbE switch.

This setup will explain how to configure VMs (Virtual Machine) network connectivity SR-IOV/MUX and UPT types. 

VMware vCenter and NSX Manager VMs will be placed on a separate Management cluster which is not in the scope of the document.

In our deployment example we use NVIDIA® Spectrum® SN3700 Ethernet switch and NVIDIA 100GbE to 4x25GbE (QSFP28 to 4xSFP28) Direct Attach Copper Splitter Cables for high speed ethernet network. Management traffic is served by NVIDIA® SN2201 Ethernet switch.

Currently, only one Dell NVIDIA BlueField-2 DPU may be used per host. To provide High Availability solution you can add NVIDIA ConnectX-6DX network card for vMotion and High speed vSAN traffics. Please follow the RDG for VMware vSAN ESA over NVIDIA RoCE on VMware vSphere 8.0 - Solutions - NVIDIA Networking Docs guide.

Logical_Design.png

Host Network Design

Host_Network_Design.png

Bill of Materials

BoM.png

vSphere Cluster Compute/Storage

VM

CPU

MEM

DISK

vCenter

4

21GB

50GB

NSX Manager

12

48GB

300GB

Deployment and Configuration

Wiring

wiring.png

Setup Configuration

Before starting the configuration process, make sure you are familiar with VMware vSphere, vCenter and NSX deployment and management procedures.

The installation process requires administrator privileges on the target machines.

In addition, the following components are required:

Network

Prerequisites

Switches OS

NVIDIA Cumulus 5 .3.

Management Network

DHCP and DNS services are required.

Warning

The components' installation and configuration are not covered by this guide.

Network Switch Configuration

In the document we will use the following networks.

  • Management Network (VLAN 130 – 10.130.0.0/16) – Management VMs such as the vCenter and NSX Manager location

  • ESXi Management Network (VLAN 236 – 10.236.0.0/16) – ESXi VMkernel interfaces location

  • vMotion Network (VLAN 1620 – 192.168.20.0/24) – ESXi vMotion VMkernel interfaces location

  • vSAN Network (VLAN 1630 – 192.168.30.0/24) – ESXi vSAN VMkernel interfaces location

  • NSX Geneve Overlay Network for ESXi Hosts (VLAN 1640- 192.168.40.0/24) – Used by the Geneve Overlay Tunnel endpoints VMkernel interfaces on the ESXi Hosts aka vmk10.

  • SR-IOV Network (VLAN 1650 – 192.168.50.0/24) – Virtual Machines with Virtual Function interface location

Switches Configuration

Run the following commands on both switches in the vSphere Cluster to configure the VLAN .

Sample on SN2201 switch.

Switch console

Copy
Copied!
            

            
nv set interface swp1-48 bridge domain br_default
nv set bridge domain br_default vlan 130
nv set bridge domain br_default untagged 130
nv config apply
nv config save

Sample on SN3700 switch.

Switch console

Copy
Copied!
            

            
nv set interface swp1-32 bridge domain br_default
nv set bridge domain br_default vlan 236,1620,1630,1640,1650
nv set bridge domain br_default untagged 236
nv config apply
nv config save

Hosts Preparation

Warning

Hosts in the vSphere Cluster must be configured before a vSphere Distributed Services Engine can be configured.

Host setup preparation:

  1. 3 x Dell R750 physical server installed with Dell NVIDIA BlueField-2 DPUs in slot 2. To make sure, perform the following:

    1. Access iDRAC via the web browser.

      Enter_to_iDRAC_via_browser.PNG

    2. From the System → Overview → PCI Slots, select PCIe Slot 2.

      Sample.

      0JNDCM__Camelantis_DPU_Install_10.PNG

  2. Each NVIDIA BlueField-2 DPU is connected to server through Paige card, which is connected to server by UART connection and to NVIDIA BlueField-2 DPUs by NC-SI cable.

  3. Installer privileges: The installation requires administrator privileges on the target machine.

  4. Management network connectivity to host and Dell NVIDIA BlueField-2 DPU.

  5. High speed network connectivity to 2 Dell NVIDIA BlueField-2 DPU ports.

  6. Ensure that your server is configured with boot mode: UEFI boot mode.

  7. VERY IMPORTANT!: Ensure BIOS, iDRAC, CPLD, Dell NVIDIA BlueField-2 BlueField-2 DPU BIOS and NIC firmware versions and update to the required ones.
    To make sure, perform the following:

    1. Access the VMware Compatibility Guide via the web browser and select your server.

      Sample.

      Dell_server_BIOS_support_01.png

    2. Choose ESXi 8.0 in the VMware Product Name option to see the certified BIOS/iDRAC/CPLD and DPU NIC firmware versions.

      Sample.

      Dell_server_BIOS_support_02.png

    3. Access iDRAC via the web browser.

      Enter_to_iDRAC_via_browser.PNG

    4. From the System → Inventory, select Firmware Inventory and check BIOS, iDRAC, CPLD, BlueField-2 DPU BIOS and NIC FW versions.
      Sample.

      Ensure_BIOS_IDRAC_FW_versions_01.png

      ...

      Ensure_BIOS_IDRAC_FW_versions_02.png

    5. To update the component/s.

      1. Enter to Dell Support portal via the web browser.

      2. Select your server → Drivers&Downloads → OS → Category.

        Dell_BIOS_Update_01_.png

      3. Select the components to download and click on Download Selected.

        Dell_BIOS_Update_02.png

      4. Access iDRAC via the web browser.

        Enter_to_iDRAC_via_browser.PNG

      5. Navigate to the Maintenance → System Update → Manual Update, select Local in the Location Type and click on Choose File.

        Important

        Recommended order to update the components:

        1. BIOS

        2. iDRAC

        3. CPLD

        4. BIOS of BlueField-2 DPU

        5. BlueField-2 DPU NIC firmware

        Dell_BIOS_Update_03.png

      6. Select the downloaded DUP (.exe) file and click Open.

        Dell_BIOS_Update_04.png

      7. Click on Upload.

      8. Select the package and click on Install/Install and Reboot.

        Dell_BIOS_Update_05.png

      9. Wait for the update to complete and recheck the component version.

      10. Repeat the process for other updated components.

  8. Ensure OS to iDRAC Pass-through is enabled and works. Otherwise, BlueField-2 DPU will not be detected during ESXi/ESXio install.

    1. Access iDRAC via the web browser.

      Enter_to_iDRAC_via_browser.PNG

    2. Navigate to the iDRAC Settings → Connectivity → OS to iDRAC Pass-through.

      1. Select:

        - Enabled in the State

        - USB NIC in the Pass-through Mode

        Sample.

        OS_to_IDRAC_.png

      2. Click on Test network connection.

        OS_to_IDRAC_02.png

  9. Ensure optimal performance:

    1. Access iDRAC via the web browser.

      Enter_to_iDRAC_via_browser.PNG

    2. From the Configuration → BIOS Settings, select System Profile Settings.

      Dell_perf_tuning_01.png

    3. Set the System Profile to Performance Per Watt (OS).

    4. Set Workload Profile to Telco Optimized Profile.

      Dell_perf_tuning_02.png

  10. On iDRAC, check and enable DPUBootSynchronization, DPUTrust.

    1. Login to the iDRAC console by running #ssh root@<hostname>-ilo or by any other method you use, and press Enter.

    2. Check DPUBootSynchronization and DPUTrust. Expected values for both are Enabled.

      iDRAc console

      Copy
      Copied!
                  
      
            
      racadm>>get system.PCISlot.2
...
DPUBootSynchronization=Enabled
...
DPUTrust=Enabled
...

    3. Enable DPUBootSynchronization, run:

      iDRAc console

      Copy
      Copied!
                  
      
            
      racadm>>set system.pcislot.2.DPUBootSynchronization 1
[Key=system.Embedded.1#PCISlot.2]
Object value modified successfully

    4. Enable DPUTrust, run:

      iDRAc console

      Copy
      Copied!
                  
      
            
      racadm>>set system.pcislot.2.DPUTrust 1
[Key=system.Embedded.1#PCISlot.2]
Object value modified successfully

    5. Check and confirm it is enabled.

      iDRAc console

      Copy
      Copied!
                  
      
            
      racadm>>get system.PCISlot.2
...
DPUBootSynchronization=Enabled
...
DPUTrust=Enabled
...

    6. COLD BOOT the x86 host.

  11. Ensure access to the BlueField-2 DPU console through iDRAC, and check BMC and OOB NAC addresses.

    Note: These addresses you need to provide to your System Administrators for DHCP records creating.

    1. Login to the iDRAC console by running #ssh root@<hostname>-ilo or by any other method you use and press Enter.

    2. Use iDRAC for the SmartNOC console access.

      iDRAc console

      Copy
      Copied!
                  
      
            
      racadm>>console dpu1
dpu-bmc login: root
Password:
root@dpu-bmc:~#

      Warning

      First-time login credentials are: root/0penBmc.

      When prompted to change the password, change it to a new one.

      dpu_bmc_password_policy.png

    3. Check the MAC/IP addresses on BMC.

      Sample.

      iDRAc console

      Copy
      Copied!
                  
      
            
      root@dpu-bmc:~# ip a s
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host
       valid_lft forever preferred_lft forever
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast qlen 1000
    link/ether 94:6d:ae:2f:61:c0 brd ff:ff:ff:ff:ff:ff
    inet 169.254.20.165/16 brd 169.254.255.255 scope link eth0
       valid_lft forever preferred_lft forever
    inet 10.236.153.19/16 brd 10.236.255.255 scope global dynamic eth0
       valid_lft 41432sec preferred_lft 41432sec
    inet6 fdfd:fdfd:10:236:966d:aeff:fe2f:61c0/64 scope global dynamic
       valid_lft 2591999sec preferred_lft 604799sec
    inet6 fe80::966d:aeff:fe2f:61c0/64 scope link
       valid_lft forever preferred_lft forever=

      In our case the eth0 interface has MAC - 94:6d:ae:2f:61:c0 and IP - 10.236.153.19 addresses.

    4. Connect to OOB by running the command below and press Enter.

      iDRAc console

      Copy
      Copied!
                  
      
            
      root@dpu-bmc:~# obmc-console-client
ubuntu@clx-host-153-oob:~$

      Warning

      First-time OOB login credentials are: ubuntu/ubuntu.

      When prompted to change the password, change it to a new one.

    5. Check the MAC/IP addresses on BMC.Sample.

      iDRAc console

      Copy
      Copied!
                  
      
            
      ubuntu@clx-host-153-oob:~$ ip a s
...
3: oob_net0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UP group default qlen 1000
    link/ether 94:6d:ae:2f:61:cc brd ff:ff:ff:ff:ff:ff
    inet 10.236.153.18/16 brd 10.236.255.255 scope global dynamic noprefixroute oob_net0
       valid_lft 72374sec preferred_lft 72374sec
    inet6 fe80::966d:aeff:fe2f:61bc/64 scope link
       valid_lft forever preferred_lft forever
...

      In our case, the oob_net0 interface has MAC - 94:6d:ae:2f:61:cc and IP - 10.236.153.18 addresses.

      Warning

      To quit the iDRAC console, press the following key combinations: Ctrl+6 and Ctrl+\ or Ctrl+^ and Ctrl+\

      To quit the BMC console, press the following key combination: ENTER and ~ and .

Hosts Network Configuration

This table provides details of the ESXi server, switches names and their network configuration.

SL-WL01-Cluster01 vSphere Cluster

Server

Server

Name

IP and NICs

High-Speed Ethernet Network

Management Networks

10.236.0.0/16

10.130.0.0/16

ESXi-01

clx-host-153

vmk0: 169.254.0.2 (BMC_Network)

vmk2: 192.168.20.153 (vMotion)

vmk3: 192.168.30.153 (vSAN)

vmk10: From IP Pool 192.168.40.0/24 (NSX Host TEP)

vmk0: 10.236.153.1

From DHCP (reserved)

ESXi-01-BF

clx-host-153-oob

clx-host-153-bmc

vmk0: 10.236.153.18 (OOB, ESXio)

BMC: 10.236.153.19

From DHCP (reserved)

ESXi-02

clx-host-154

vmk0: 169.254.0.2 (BMC_Network)

vmk2: 192.168.20.154 (vMotion)

vmk3: 192.168.30.154 (vSAN)

vmk10: From IP Pool 192.168.40.0/24 (NSX Host TEP)

vmk0: 10.236.155.1

From DHCP (reserved)

ESXi-02-BF

clx-host-154-oob

clx-host-154-bmc

vmk0: 10.236.154.128 (OOB, ESXio)

BMC: 10.236.154.129

From DHCP (reserved)

ESXi-03

clx-host-155

vmk0: 169.254.0.2 (BMC_Network)

vmk2: 192.168.20.155 (vMotion)

vmk3: 192.168.30.155 (vSAN)

vmk10: From IP Pool 192.168.40.0/24 (NSX Host TEP)

vmk0: 10.236.155.1

From DHCP (reserved)

ESXi-03-BF

clx-host-155-oob

clx-host-155-bmc

vmk0: 10.236.155.18 (OOB, ESXio)

BMC: 10.236.155.19

From DHCP (reserved)

Leaf-01

clx-swx-033

10.130.250.233

Leaf-03

clx-swx-034

10.130.250.234

vCenter (VM)

sl01w01vc01

10.130.250.115

NSX Manager 01 (VM)

sl01w01nsx01

10.130.250.140

PV-VM-01

clx-vm-pv-01

10.10.10.2

10.236.153.2

PV-VM-02

clx-vm-pv-02

10.10.10.3

10.236.153.3

SRIOV-VM-01

clx-vm-sriov-01

10.10.10.4

10.236.153.4

SRIOV-VM-02

clx-vm-sriov-02

10.10.10.5

10.236.153.5

UPT-VM-01

clx-vm-upt-01

10.10.10.6

10.236.153.6

UPT-VM-02

clx-vm-upt-02

10.10.10.7

10.236.153.7

ESXi/ESXio Installation using a Unified Image

Before you start, make sure the following required items are available:

  • ESXi/ESXio IPs and FQDNs registered in DNS/DHCP

  • Management network netmask and gateway

  • NTP servers

  • Password for ESXi/ESXio root user

To install the unified ESXi image on the BlueField-2 DPU equipped Dell server:

  1. Download the ESXi/ESXio server hypervisor ISO file from https://customerconnect.vmware.com/ website .

  2. Access the iDRAC web interface and open the Virtual Console.

    Enter_to_iDRAC_via_browser2.PNG

  3. In the Virtual Console, select Virtual Media.

    step_1.png

  4. Click on Connect Virtual Media.

    step_1b.png

  5. Click on Choose File.

    step_2.png

  6. Select the image file.

    step_3.png

  7. Click on Map Device and afterwards on Close.

    step_4.png

  8. In the Virtual Console window, select Boot.

    step_5.png

  9. From the Boot Controls popup window, select Virtual CD/DVD/ISO.

    step_6.png

  10. On Confirm Boot Action popup window, click Yes.

    step_7.png

  11. On the Virtual Console window select Power.

    step_8.png

  12. On Confirm Power Action popup window, click Yes.

    step_12.png

  13. From the Power Controls popup window, select Power On System or Power Cycle System (cold boot).

    step_9.png

  14. Let the system boot up (take several minutes) and reach the VMware ESXi 8.0.0 Installation. Press Enter to start the installation.

    step_13.png

  15. Press F11 to accept the End User License Agreement (EULA).

    step_14.png

  16. At the Installer Configuration window, ensure both the Install/Upgrade ESXi and the detected Dell NVIDIA Bluefield-2 Dual Port 25 GbE SFP Crypto DPU 2 are selected, and press Enter.

    step_15.png

  17. Select the disk from the list where ESXi operating system will be installed and press Enter.

    In our case, we selected the DELLBOSS VD.

    step_16.png

  18. Keep the default for keyboard layout - US Default, press Enter to proceed.

    step_17.png

  19. Provide a root password. After password match confirmation press Enter.

    Warning

    ESXio password on newer builds will be same as password set for ESXi.

    step_18.png

  20. Ensure the installation on both the Dell BOSS card and the BlueField-2 DPU, and press F11.

    step_19.png

  21. Press Enter to reboot once the installation in completed on both on ESXi and BlueField-2 DPU.

    The installation is performed first on the BlueField-2 DPU and then the host.

    step_20.png

  22. Upon successful ESXi installation, press Enter to reboot.

    step_21.png

  23. Upon successful ESXi installation on x86 and ESXi on BlueField-2 DPU (ESXio), the Server then reboots into ESXi booted on x86 and ESXio is booted on BlueField-2 DPU.
    Now your system is ready for use, both the ESXi on x86 and the ESXio on BlueField-2 DPU.
    Below is the DCUI screen, which will be configured in next steps.

    step_22.png

  24. Press F2 to configure the ESXi. Authentication is required here, Enter an authorized login name root and password (you set during install) for the ESXi host.

    step_22a.png

  25. Select Configure Management Network and ensure/apply the appropriate host network configuration.

    step_22b.png

    step_22c.png

    step_22d.png

    step_22f1.png

    step_22h.png

    step_22i.png

  26. Check and Enable ESXi Shell and SSH on ESXi (x86).

    step_22k.png

    step_22l.png

    step_22m.png

  27. Connect to the ESXi console and run sshdpu to access to the ESXio on the BlueField-2 DPU DCUI screen.

    Warning

    You can enable ssh on ESXio by running the following command on ESXi (x86) console.

    [root@clx-host-154:~] vim-cmd combinersvc/dpu_services/set_policy TSM-SSH on vmdpu0

    [root@clx-host-154:~] vim-cmd combinersvc/dpu_services/start TSM-SSH vmdpu0

    ESXi Host Console

    Copy
    Copied!
                
    
            
    [root@clx-host-154:~] sshdpu
ssh: connect to host 169.254.100.2 port 22: Connection refused
[root@clx-host-154:~] vim-cmd combinersvc/dpu_services/set_policy TSM-SSH on vmdpu0
[root@clx-host-154:~] vim-cmd combinersvc/dpu_services/start TSM-SSH vmdpu0
[root@clx-host-154:~] sshdpu
hostfile_replace_entries: link /.ssh/known_hosts to /.ssh/known_hosts.old: Function not implemented
update_known_hosts: hostfile_replace_entries failed for /.ssh/known_hosts: Function not implemented
The time and date of this login have been sent to the system logs.
 
WARNING:
   All commands run on the ESXi shell are logged and may be included in
   support bundles. Do not provide passwords directly on the command line.
   Most tools can prompt for secrets or accept them from standard input.
 
VMware offers supported, powerful system administration tools.  Please
see www.vmware.com/go/sysadmintools for details.
 
The ESXi Shell can be disabled by an administrative user. See the
vSphere Security documentation for more information.
[root@clx-host-154-oob:~] dcui

  28. Press F2 to configure the ESXi. Authentication is required here.

    step_23a.png

    Warning

    You can access and login to the ESXio shell on BlueField-2 DPU console DCUI using following keys

    Ctrl+G Ctrl+B 2 Enter

    You can switch back to DCUI window using following keys

    Ctrl+G Ctrl+B 3 Enter

  29. Enter an authorized login name root and password (you set during install) for the ESXi host.

    step_24.png

  30. Select Configure Management Network and ensure/apply the appropriate host network configuration.

    step_24a.png

    step_24b.png

    step_24c.png

    step_24d.png

    step_24e.png

    step_24f.png

  31. Check and Enable ESXi Shell and SSH on ESXi (x86).

    step_24g.png

    step_24h.png

vCenter Deployment

Before you start please make sure the following required items are available:

  • vCenter IP and FQDN registered in DNS/DHCP

  • Management network netmask and gateway

  • NTP servers

  • Make sure nslookup is working for vCenter hostname and IP Address

    vCenter_install_00.png

  • Password for vCenter root user

  • Password for administrator@vsphere.local user

To install the unified ESXi image on BlueField-2 DPU equipped Dell server:

  1. Download VCSA 8.0 ISO from https://customerconnect.vmware.com/ website and mounted it on the windows system.

  2. Execute Installer.exe application from CD\DVD Drive <VMware VCSA>:\vcsa-ui-installer\win32 ISO folder location.

    vCenter_install_01.png

  3. Choose “Install” and click Next on the Introduction screen.

    vCenter_install_02.png

    vCenter_install_03.png

  4. Accept the End user license agreement and click Next.

    vCenter_install_04.png

  5. Specify the target where the vCenter server appliance will be deployed. The target can be the ESXi host or an existing vCenter server.

    In this deployment we will be using a management cluster vCenter server.

    Once you have completed the form, click Next to continue.

    vCenter_install_05.png

  6. A Certificate Warning might appear. Click Yes to continue.

    vCenter_install_06.png

  7. In the Select folder , select a datacenter or VM folder to create the vCenter Server VM and click Next.

    vCenter_install_07.png

  8. In the Select compute resource, s elect an ESXi server where the vCenter server appliance will be deployed.

    vCenter_install_08.png

  9. Specify the VM settings for the vCenter Server. Click Next to continue.

    - VM Name - Name of the Virtual Machine

    - Set root password - Provide a Password

    - Confirm root password - Confirm the root password

    vCenter_install_09.png

  10. Select the VCSA deployment size for your environment. In this deployment we will be using “Small” for deployment size and “Default” for Storage Size.

  11. Click Next to continue.

    vCenter_install_10.png

    Warning

    See Hardware Requirements for the vCenter Server Appliance and required information for Deploying a vCenter Server Appliance for more information about the deployment size.

  12. Select the datastore to identify the storage location for this vCenter server in Select datastore . Click Next to continue.

    vCenter_install_11.png

    Warning

    For this environment, we selected “Enable Thin Disk Mode”. If you enable thin disk mode, the datastore size should have a minimum of 25GB space.

  13. Configure the network settings of the VCSA.

    Please provide following information:

    - Network - Select the port group to be used for the VCSA and ping the ESXi management network

    - IP Version - Select either IPv4 or IPv6

    - IP assignment - Select either Static or dhcp

    - FQDN - Provide the fully qualified domain name (FQDN) or IP address of the VCSA (vcentername.domain.com or x.x.x.x)

    - IP Address - Provide the IP address of the VCSA (x.x.x.x)

    - Subnet Mask or Prefix Length - Provide the Subnet mask or Prefix Length of the VCSA network (x.x.x.x or xx)

    - Default Gateway - Provide the Default Gateway (x.x.x.x)

    - DNS Servers - Provide the DNS server IP address for the VCSA (x.x.x.x,x.x.x.x)

    - Common Ports - Leave Defaults unless you need to customize them for your enviroment

  14. Click Next to continue.

    vCenter_install_12.png

  15. Review your configuration and click Finish to initiate stage 1 of deploying the VCSA.

    vCenter_install_13.png

  16. Wait until the vCenter server appliance deployment is completed.

    Note, this might take awhile.

    vCenter_install_14.png

  17. Click Continue to proceed with stage 2 of the deployment process once stage 1 is completed successfully.

    vCenter_install_15.png

    Warning

    If you exit the setup installer window, you can continue with the vCenter Server setup at any time by logging in to the vCenter Server Management Interface (VAMI) https://vCenter_FQDN_or_IP:5480/

  18. Click Next . to proceed with Stage 2: setting up the vCenter Server.

    vCenter_install_16.png

  19. Set the NTP and SSH configuration for the VCSA and click Next to continue.

    vCenter_install_17.png

  20. Choose your SSO Configuration and click Next to continue. In this deployment we will be creating a new SSO domain.

    vCenter_install_18.png

  21. Configure CEIP and click Next to continue.

    vCenter_install_19.png

  22. Review Ready to complete screen and verify configuration. Click Finish to initiate stage 2.

    vCenter_install_20.png

  23. After completing successfully stage 2, you will be presented with a URL. Click the URL to launch the vSphere Client. You can now close the vCenter Server Installer.

    vCenter_install_21.png

  24. At the vCenter server page, accept the SSL certificate.

  25. Click Launch vSphere Client.

    vCenter_install_22.png

  26. Log into vSphere client using administrator@vsphere.local account.

    vCenter__Configuration_01.png

  27. You have deployed the vCenter Server Appliance successfully.

    vCenter__Configuration_02.png

vSphere Configuration

Create vSphere Datacenter, Cluster and add the ESXi Host to the Cluster

  1. Create a new Datacenter, navigate to Home and then Hosts and Clusters.

  2. Right-click the mouse on the vCenter name to show the menu and select New Datacenter.

    Create_Datacenter_01.png

  3. Name the new Datacenter.
    For example: SL-Datacenter.

    Create_Datacenter_02.png

  4. Right-click on the Datacenter name to show the menu and to create a New Cluster.

    Create_Cluster_01.png

  5. In the Basic menu in the New Cluster window:

    1. Name the new Cluster (e.g, SL-WL01-Cluster01).

    2. Make sure the image the settings are set as shown in the figure below.

  6. Click Next.

    Create_Cluster_02.png

  7. In the Image menu in the New Cluster window, make sure the right ESXi Version is set, then click Next.

    Create_Cluster_03.png

  8. In the Review menu in the New Cluster window, make sure the settings are correct, then click Finish.

    Create_Cluster_04.png

  9. Add the ESXi host to the Cluster.

    Create_Cluster_05.png

  10. Right-click on the Cluster name to show the menu, and select Add and Manage Hosts.

    add_hosts_01.png

  11. In the Select task menu, add the ESXi host that was created (name, user root, and password). If the user and password are the same for all ESXi hosts, you can enable the option to use the same credentials for all ESXi hosts (no need to add to each one).

  12. Click Next.

    add_hosts_02.png

  13. In the Select Hosts menu, Select all ESXi hosts and import all certificates from the ESXi host to Center.

  14. Click OK.

    add_hosts_03.png

    1. In the Host Summary menu check for warnings. In our case, we have an warning that explains that one VM already exists in one of the ESXi hosts and is powered on. Such notification is harmless and just informed us about the ESXi host we are adding to the Center.

    2. In the Import Image menu, select Don't import an image. Click Next.

      add_hosts_04.png

  15. In the Manage physical adapters menu, check the ESXi host information. If all is ok, click Finish and add the ESXi hosts to your Cluster.

    add_hosts_05.png

    If there is no problem with the IP/FQDN, or the network of your ESXi hosts, vCenter will start to import them to the Cluster.

    Create_Cluster_and_add_hosts.png

  16. Select the ESXi host and in Summary tab check the BlueField-2 DPU Information.

    ESXIO.png

Add Additional MFT Depot to the Cluster Image

To load the MFT depot to the VMware Lifecycle Manager (vLCM).

  1. Download MFT v4.22.1 LTS (required for vSphere 8.0b) from the web.

    Cluster_Configuration_04a.png

  2. Open a browser, connect to vSphere web interface at https:// , and login with the administrator@vsphere.local account.

    vCenter__Configuration_01.png

  3. On the top left menu, click on the tree lines and select Lifecycle Manager.

    Cluster_Configuration_04.png

  4. Open the Action dropdown menu and select Import Updates.

    Cluster_Configuration_05.png

  5. Click Browse.

    Cluster_Configuration_06.png

  6. At the opened popup, select the MFT NMST file that you downloaded and then click Open.

    Cluster_Configuration_07.png

  7. Repeat steps 4 to 6 for the second depot bundle.

    Cluster_Configuration_07b.png

    Cluster_Configuration_08.png

  8. At the Inventory tab, select the cluster, and then select the Updates tab.

  9. Select the Image and then click Edit.

    Cluster_Configuration_09.png

  10. Click Show details.

    Cluster_Configuration_10.png

  11. Click ADD COMPONENTS.

    Cluster_Configuration_11.png

  12. Select the MFT files and click SELECT.

    Cluster_Configuration_12.png

  13. Click SAVE.

    Cluster_Configuration_13.png

  14. Compliancy check will starting automatically,

    Cluster_Configuration_14.png

  15. Click REMEDIATE ALL to start the MFT installation on the hosts.

    Cluster_Configuration_15.png

  16. Click START REMEDIATION.

    Cluster_Configuration_16.png

    Cluster_Configuration_17.png

    Cluster_Configuration_18.png

  17. Manually power off vCLS VMs on the host to set a host to Maintenance mode, if needed.

    Cluster_Configuration_19.png

    Cluster_Configuration_20.png

    Cluster_Configuration_21.png

All host have now MFT tools installed.

Cluster_Configuration_22.png

SR-IOV Enabling on the ESXi Host

To set up an SR-IOV environment, the following is required:

  1. Make sure that SR-IOV is enabled in the BIOS of the specific server.

  2. Access iDRAC via the web browser.

    Enter_to_iDRAC_via_browser.PNG

  3. From the Confuguration → BIOS Settings → Integrated Devices, s et the SR-IOV Global Enable to Enabled .

    SRIOV_Enabled.png

  4. Open a browser, connect to the vSphere web interface at https://<vcenter_fqdn>, and login with the administrator@vsphere.local account.

    Cluster_Configuration_00a.png

  5. At the Inventory tab, click on a ESXi host, and select Configure → Hardware → PCI Devices, and click on ALL PCI DEVICES.

    SRIOV_enabling_01.png

  6. Filter by Vendor Name.

    SRIOV_enabling_02.png

  7. Select first port (in our deployment) and click CONFIGURE SR-IOV.

    SRIOV_enabling_03.png

  8. At the Configure SR-IOV menu, Enable SR-IOV and select number of Virtual functions(VF).

  9. Click on OK.

    SRIOV_enabling_04.png

  10. Now you can see 8 Virtual functions.

    SRIOV_enabling_05.png

  11. To see the VF in host console. Log into the ESXi console with root permissions and r un the following commands.

    SRIOV_enabling_06.png

    SRIOV_enabling_07.png

NTP Configuration

Verify that in your environment the NTP is configured and works properly on both the host and vCenter.

NTP_on_ESXi.PNG

NTP_on_vCenter.PNG

Create and Configure a Regular (non-offloaded) Distributed Virtual Switch for vMotion and vSAN

A regular (non-offloaded) Distributed Virtual Switch with portgroups for the vMotion, and vSAN traffic need to be created to ensure consistent network settings between the hosts.

  1. Launch the vSphere Web Client and connect to a vCenter Server instance.

    vCenter__Configuration_01.png

  2. On the vSphere Web Client home screen, choose the Inventory Networking tab.

  3. Right click on the datacenter (SL-Datacenter), select the Distributed Switch, then click New Distributed Switch (see image below) to launch the New VDS creation wizard.

    New_vds.png

  4. In the Name and location menu, set the name (SL-WL01-DS01) of the new distributed switch and select the location within the vCenter inventory where you would like to store the new VDS (a data center object or a folder).

  5. Click NEXT.

    New_vds_01.png

  6. In the Select version menu, select the 8.0.0 – ESXi 8.0.0 and later switch version.

  7. Click NEXT.

    11d_UPT_Switch_Config_03.png

  8. In the Configure settings menu:

    1. set the Network Offloads compatibility as None

    2. set the Number of uplinks as 2

    3. uncheck the Create a default port group box

  9. Click NEXT.

    11d_vSAN_Switch_Config_04.png

  10. In the Ready to complete menu, click Finish.

    11d_vSAN_Switch_Config_05.png

  11. Set the MTU and Discovery protocol for the newly created distributed switch.

    Right-click the new distributed switch in the list of objects and select Settings → Edit Settings... from the Actions menu.

    11d_vSAN_Switch_Config_06.png

  12. In the SL_W01-DS01 Distributed Switch - Edit Settings dialog box, in the Advanced tab, set:

    1. MTU to 9000

    2. Discovery protocol Type to Link Layer Discovery Protocol Operation

    3. Operation to Both

  13. Click OK.

    11d_vSAN_Switch_Config_07.png

Adding Hosts to VDS

To add an ESXi host to an existing VDS:

  1. Launch the vSphere Web Client, and connect to the vCenter Server instance.

  2. Navigate to the list of Hosts in the SL-WL01-Cluster01 and select ESXi host.

  3. Select Configure → Networking → Physical adapters.

  4. Check the network ports that you are going to use. In this case, vmnic1 will used.

    11c_vSAN_config_Adding_host_01.png

  5. Navigate to the list of distributed switches.

  6. Right-click the new distributed switch in the list of objects and select Add and Manage Hosts from the Actions menu.

  7. In the Select task menu, check the Add hosts option and click NEXT .

    11c_vSAN_config_Adding_host_02.PNG

  8. In the Select Hosts menu, select all the ESXi hosts you would like to add to the V DS .

  9. Click NEXT.

    11c_vSAN_config_Adding_host_05.png

  10. In the Manage physical adapters menu click on the Adapters on all hosts tab and c onfigure vmnic1 in an ESXi host as Uplink 1 for the VDS .

    Warning

    In case you added the ConnectX-6DX please select the card ports as Uplink1 and 2.

    11c_vSAN_config_Adding_host_06.png

  11. In the Manage VMkernel adapters and Migrate VM networking menus click NEXT to continue.

    11c_vSAN_config_Adding_host_07.png

  12. In the Ready to complete menu click FINISH.

    11c_vSAN_config_Adding_host_09.png

Creating Distributed Port Groups for vMotion vSAN Traffic

  1. Right-click the Distributed switch, and select Distributed Port Group → New Distributed Port Group.

    11c_vSAN_config_Adding_Port_Group_00.png

  2. In the Name and location menu, set the Name as <vMotion-VLAN1620-DPG> and click NEXT.

    11c_vSAN_config_Adding_Port_Group_01_Vmotion.png

  3. In the Configure settings menu:

    1. set the VLAN type as VLAN

    2. set the VLAN ID as 1620

    3. uncheck the Customize default policies configuration checkbox

  4. Click NEXT

    11c_vSAN_config_Adding_Port_Group_02_Vmotion.png

  5. In the Ready to complete menu, make sure all the changes are OK and then click FINSIH.

    11c_vSAN_config_Adding_Port_Group_03_Vmotion.png

  6. Repeat steps 1 to 4 for the vSAN.

Adding a VMkernel Network for vMotion and vSAN Port Group

To add VMkernel adapters for distributed port groups, follow the steps below.

  1. Right click the distributed port group and select Add VMkernel Adapters.

    11c_vSAN_config_Adding_vmkernel_00_Vmotion.png

  2. In the Select hosts menu, select the hosts and click NEXT.

    11c_vSAN_config_Adding_vmkernel_01_Vmotion.png

  3. In the Configure VMkernel adapter menu, select vVMotion and Provisioning in Available services, and click NEXT.

    11c_vSAN_config_Adding_vmkernel_02_Vmotion.png

  4. In the IPv4 settings menu, set the Network Settings and Gateway details, and click NEXT.

    11c_vSAN_config_Adding_vmkernel_03_Vmotion.png

  5. In the Ready to complete menu, click FINISH.

    11c_vSAN_config_Adding_vmkernel_04_Vmotion.png

  6. Repeat steps 1 to 5 for the vSAN.

Create and Configure a VMware vSAN ESA

To create and configure a VMware vSAN ESA cluster please use the following document RDG for VMware vSAN ESA over NVIDIA RoCE on VMware vSphere 8.0 - Solutions - NVIDIA Networking Docs

Warning

RDMA is not supported in VMware vSphere 8.0b Distributed Services Engine. To run RDMA vSAN you can use additional ConnectX-6 Dx network card and use the card for vSAN RDMA traffic.

Samples of vSAN configuration screens.

vSAN_config_01.png

vSAN_config_02.png

vSAN_config_03.png

vSAN_config_04.png

vSAN_config_05.png

vSAN_config_06.png

vSAN_config_07.png

Enable DRS and HA on Cluster

To enable DRS and HA on the SL-WL01-Cluster01 vSphere Cluster.

  1. Open a browser, connect to vSphere web interface at https://<vcenter_fqdn>, and login with the administrator@vsphere.local account.

  2. On the Inventory tab, select the cluster (SL-WL01-Cluster01), select Configure, select vSphere DRS, then select Edit….

  3. On the Edit Cluster Settings page, enable vSphere DRS, then click Ok.

    DRS_enable.png

    DRS_On.PNG

  4. Check and apply LCM compliance.

    Cluster_Configuration_00.png

To enable HA on the SL-WL01-Cluster01 vSphere Cluster.

  1. Open a browser, connect to vSphere web interface at https://<vcenter_fqdn>, and login with the administrator@vsphere.local account.

  2. On the Inventory tab, select the cluster (SL-WL01-Cluster01), select Configure, select vSphere DRS, then select Edit….

  3. Enable vSphere HA, and click OK.

    HA_on.jpg

  4. Check and apply LCM compliance.

    Cluster_Configuration_00.png

Create and Configure an Offloaded Distributed Virtual Switch

A regular distributed switch with portgroups for the vMotion, and vSAN traffic should be created to ensure consistent network settings between the hosts.

  1. Launch the vSphere Web Client and connect to a vCenter Server instance.

    vCenter__Configuration_01.png

  2. On the vSphere Web Client home screen, click on Inventory Networking tab.

  3. Right click on the datacenter (SL-Datacenter), select the Distributed Switch, then click New Distributed Switch (see image below) to launch the New VDS creation wizard.

    11d_UPT_Switch_Config_01.png

  4. In the Name and location menu, set the name (SL-WL01-DS02) of the new distributed switch and select the location within the vCenter inventory where you would like to store the new VDS (a data center object or a folder).

  5. Click NEXT.

    11d_UPT_Switch_Config_02.png

  6. In the Switch version menu, select 8.0.0 – ESXi 8.0.0 and later.

  7. Click NEXT.

    11d_UPT_Switch_Config_03.png

  8. In the Configure settings menu:

    1. set the Network Offloads compatibility as NNVDIA BlueField

    2. set the Number of uplinks as 2

    3. uncheck the Create a default port group box

  9. Click NEXT.

    11d_UPT_Switch_Config_04.png

  10. In the Ready to complete menu, click Finish.

    11d_UPT_Switch_Config_05.png

  11. Set the MTU and Discovery protocol for the newly created distributed switch.

  12. Right-click the new distributed switch in the list of objects and select Settings → Edit Settings... from the Actions menu.

    11d_UPT_Switch_Config_07.png

  13. In the SL_W01-DS02 Distributed Switch - Edit Settings dialog box, in the Advanced tab, set:

    1. MTU to 9000

    2. Discovery protocol Type to Link Layer Discovery Protocol Operation

    3. Operation to Both

  14. Click OK.

    11d_UPT_Switch_Config_08.png

Adding Hosts to VDS

To add an ESXi host to an existing VDS:

  1. Launch the vSphere Web Client, and connect to the vCenter Server instance.

  2. Navigate to the list of Hosts in the SL-WL01-Cluster01 and select ESXi host.

  3. Select Configure → Networking → Physical adapters.

  4. Check the network ports that you are going to use. In this case, vmnic0 will used.

  5. Navigate to the list of distributed switches.

  6. Right-click the new distributed switch in the list of objects and select Add and Manage Hosts from the Actions menu.

    add_hosts_012.png

  7. In the Select task menu, check the Add hosts option and click NEXT .

    add_hosts_022.png

  8. In the Select hosts menu, select all the ESXi hosts you would like to add to the V DS .

  9. Click NEXT.

    add_hosts_03.png

  10. In the Manage physical adapters menu, select the Adapters on all hosts tab and c onfigure vmnic0 in an ESXi host as Uplink 1 for the VDS .

    Warning

    In case you added the ConnectX-6DX please select the card first port (vmnic0) as Uplink1 and second port (vmnic1) as Uplink2.

    Add_hosts_06.png

  11. In the Manage VMkernel adapters and Migrate VM networking menus click NEXT to continue.

    Add_hosts_07.png

  12. In the Ready to complete menu, click FINISH.

    Add_hosts_08.png

Creating Distributed Port Groups for Virtual Machines SR-IOV Traffic

  1. Right-click on Distributed switch, and select Distributed Port Group → New Distributed Port Group.

    Add_hosts_09.png

  2. In the Name and location menu, set the Name (SL-WL01-SRIOV-1650) and click NEXT.

    Add_SRIOV_PG_01.png

  3. In the Configure settings menu:

    1. set the VLAN type as VLAN

    2. set VLAN ID as 1650

    3. check the Customize default policies configuration checkbox

  4. Click NEXT.

    Add_SRIOV_PG_02.png

  5. In the Security menu click NEXT.

    Add_PG_04.png

  6. In the Traffic shaping menu click NEXT .

    Add_PG_05b.png

  7. In the Teaming and failover menu, set Uplink 1 as active and Uplink 2 as standby.

  8. Click NEXT.

    Add_PG_06.png

  9. In the Monitoring menu, set NetFlow to Disabled , and click NEXT.

    Add_PG_07.png

  10. In the Miscellaneous menu, set Block All Ports to No , and click NEXT.

    Add_PG_08.png

  11. In the Ready to complete menu review all the changes before you click FINSIH.

    Add_SRIOV_PG_09.png

NSX Manager Deployment and Configuration

NSX Manager Deployment

Prerequisites

Below are the prerequisites for deploying NSX Manager.

  • Necessary resources exist on the cluster to deploy NSX-Manager appliance

  • Port group for Management network (it is preferable to keep on the same network as vCenter)

  • 4 free IPs from management network for NSX Managers (in this guide only one appliance is deployed)

  • NSX License

  • Create DNS A rerecords for NSX-Manager Hostnames

  • Management network netmask and gateway

  • NTP servers

  • Make sure nslookup is working for vCenter hostname and IP Address

    NSX_Install_00.png

  • Password for the NSX root user

  • Password for the NSX admin user

  • Password for the NSX audit user

Network Latency Requirements

  • The maximum network latency between NSX Managers in a NSX Manager cluster is 10ms

  • The maximum network latency between NSX Managers and Transport Nodes is 150ms

Storage Requirements

  • The maximum disk access latency is under 10ms

  • It is recommended that NSX Managers are placed on a shared storage

  • Storage should be highly available to avoid a storage outage causing all NSX Manager file systems to be placed into read-only mode upon a storage failure event

    Please see the documentation for your storage technology on how to optimally design a highly available storage solution.

Deployment

  1. Download the NSX manager OVA from VMware Customer Connect portal.

    NSX_Download_01.png

    NSX_Download_02.png

  2. Login to the Management Cluster vCenter.

    vCenter__Configuration_01.png

  3. Click Host and Clusters, right click on a ESXi where you would like the NSX Manager to be deployed at and click Deploy OVF Template.

    NSX_Install_01.png

  4. Select Local file → browse the OVA file for NSX Unified appliance .

    NSX_Install_02.png

    NSX_Install_03.png

  5. Click NEXT.

  6. Assign a Virtual machine name (sl-wl01-nsx01) and select the location were you want to place the NSX manager and click NEXT.

    NSX_Install_04.png

  7. Select the compute resource whe re you want to place your NSX manager ( i.e., the cluster, resource pool, or the host), check its compatibility, and c lick NEXT.

    NSX_Install_05.png

  8. Review the details and c lick NEXT.

    NSX_Install_06.png

  9. Select deployment size (Medium), click NEXT .

    NSX_Install_07.png

    Warning

    When selecting "Small" as the deployment configuration, some of the services do not work. Always use "Medium" or "Large".

  10. Select the Storage/datastore where NSX manager should be placed and c lick NEXT.

    NSX_Install_08.png

  11. Select the Management network and click NEXT to move to the "Customize Template" screen .

    NSX_Install_09.png

  12. Specify the System GRUB root, System root account’s, admin and audit user passwords.

  13. S croll down .

    Warning

    Strong password is required.

    NSX_Install_10.png

    NSX_Install_11.png

    NSX_Install_12.png

  14. Provide the Hostname, Rolename (NSX Manager has 3 roles, as seen below), and Management networking details.

  15. Scroll down .

    NSX_Install_13.png

  16. Assign the DNS and NTP details and choose whether SSH needs to be enabled on the NSX Manager.

  17. Click NEXT .

    NSX_Install_14.png

  18. Review the details and click FINISH . Wait for the NSX manager appliance to be successfully deployed.

    NSX_Install_15.png

  19. The NSX Manager will be deployed after about 5-10 minutes .

    Once the appliance is successfully deployed, Power on and wait for at least 15 minutes for all the services to come up.

  20. Once the services are up, a login prompt is shown. At this point, wait 10 more minutes as backend services need to start the web-portal-related services.

    NSXt_depl_16.PNG

Post-Deployment Health Checks

Once done, do the following post-deployment steps:

  1. Open the console and try to log in with admin credentials using putty. You will see the version number and the role.

    NSXt_depl_17.png

  2. Verify the network configuration by using the command below.

    NSX manager console

    Copy
    Copied!
                
    
            
    get interface eth0

    NSXt_depl_18.png

  3. Check the services status by running the following command.

    NSX manager console

    Copy
    Copied!
                
    
            
    get services

    NSXt_depl_19.png

    Warning

    Liagent, migration-coordinator and SNMP services are not started by default.

NSX Manager Configuration

  1. Log in to NSX manager UI login page by using the URL " https:// ".

    NSXT_config_01.png

  2. Accept the EULA on the first login to the NSX manager.

    NSXT_config_02.png

  3. Decide whether or not to join the Customer Experience Improvement Program.

    NSXT_config_03.png

  4. Take a quick tour, in case you need.

    NSX_Config_03b.png

  5. Finally, the NSX manager is presented.

    NSXT_config_04.png

  6. Navigate to System → Licenses and click on +ADD LICENSE.

    NSXT_config_05.png

  7. Add a license and click ADD.

    NSXT_config_06.png

    NSXT_config_07.png

Adding a Compute Manager

  1. To add a compute manager, navigate to System → Fabric → Compute Managers, click ADD COMPUTE MANAGER to add vCenter as compute manager.

    Add_Computer_Manager_01.png

  2. Fill in the vCenter details → click ADD.

    Add_Computer_Manager_02.png

  3. Click ADD , when presented with the vCenter server thumbprint window.

    Add_Computer_Manager_03.png

  4. Wait for registration to complete. T he status will be changed to registered and connection status to "Up".

    Add_Computer_Manager_04.png


Create an Uplink Profile

An uplink profile defines the policies for links from a hypervisor hosts to the NSX logical switches or from NSX Edge nodes to top-of-rack switches. The settings defined by these profiles may include teaming policies, active/standby links, transport VLAN ID, and MTU settings. Uplink profiles enables consistent configuration of identical capabilities for network adapters across multiple hosts and nodes. By default, there are two uplink profiles that are provided with NSX and cannot be edited, that is why new profiles for the Edge uplink should be created (for hosts’ uplinks as well).

To create a Host uplink profile in NSX Manager, do the following:

  1. Navigate to System → Fabric → Profiles → Uplink Profiles → +ADD PROFILE.

    Uplink_Profile_01.png

  2. Assign a name to the profile and fill the description.

  3. Under Teamings, set the Teaming Policy to Failover Order.

  4. Set the Active Uplinks to uplink-1.

  5. The Transport VLAN will be an Overlay VLAN ID (in the example provided,1640) since these uplinks are connected directly to the hosts and need to be tagged accordingly. MTU not fill .

  6. Click ADD.

    Uplink_Profile_02.png

    Uplink_Profile_03.png

Create a Transport Zone

Transport Zones dictate which hosts (and consequently which VMs) can participate in the use of a particular network. There are two types of transport zones: an overlay and a VLAN.

  • The overlay transport zone is used by both host transport nodes and NSX Edges and is responsible for communication over the overlay network.

  • The VLAN transport zone is used by the NSX Edge for its VLAN uplinks.
    Both types create an N-VDS on the host or Edge to allow for virtual-to-physical packet flow by binding logical router uplinks and downlinks to physical NICs.

To create Transport Zone with Overlay Traffic Type

  1. To create a Transport Zone in the NSX Manager, navigate to System → Fabric → Transport Zones → +ADD ZONE.

    Transport_Zone_01.png

  2. Provide a Name and select the Traffic Type as Overlay (Geneve) and then click ADD.

    Transport_Zone_02.png

    Transport_Zone_03.png

Create a NSX TEPs IP Pool

Each transport node (i.e., hypervisors) is assigned with an IP address for the TEP interface. DHCP, Static IP List, and IP address Pool can all be used for assigning IP addresses for the TEP (Tunnel Endpoint) interfaces. To configure the ESXi hosts as Transport nodes, NSX IP Pool will be created to assign IP addresses for the TEP interfaces.

To create an IP Pool, in the NSX Manager, do the following:

  1. Navigate to Networking → IP Address Pools → ADD IP ADDRESS POOL.

    NSX_IP_Pool_01.png

  2. Specify the name and description of the IP address pool. Click Set.

    NSX_IP_Pool_02.png

  3. Click ADD SUBNET and select IP Ranges.

    NSX_IP_Pool_03.png

  4. Specify the IP Address Ranges, along with CIDR and Gateway IP, address for the specified IP ranges. Click ADD.

    NSX_IP_Pool_04.png

  5. Click Apply.

    NSX_IP_Pool_05.png

  6. Click SAVE to create the IP Pool.

    NSX_IP_Pool_06.png

  7. Once the IP Pool is created, the status will be changed to Success.

    NSX_IP_Pool_07.png

Installing NSX Components on a Host Transport Node(ESXi/ESXio).

  1. Navigate to System → Fabric → Hosts → Clusters. Select SL-WL01-Cluster01. Click Configure NSX.

    Transport_Nodes_01.png

  2. In the Transport Node Profile, click the three dots menu to Create a New Transport Nodes Profile.

    Transport_Nodes_02.png

    1. Specify the Name → SL-WL01-TNP-EDP-STD and description, and click the Set.

      Transport_Nodes_02b.png

    2. Click ADD HOST SWITCH.

    3. Select Name of vCenter → sl-wl01-vc01 and VDS → SL-WL01-DS02.

    4. Select Transport Zones were created before → SL-WL01-Overlay-TZ.

    5. Select the Uplink Profile → Uplink-profile-01.

    6. Select the IP Assignment (TEP) → Use IP Pool.

    7. Select the IP Pool → SL-WL01-TEP-IPPool.

    8. Select Mode → Enhanced Datapath - Standard.

      Warning

      VDS Modes:

      • Standard applies to all the supported hosts.

      • Enhanced Datapath - Standard is an interrupt driven variant of the Enhanced Data Path mode.

      • Enhanced Datapath - Performance is the Enhanced Data Path switch mode. This mode provides accelerated networking performance, but also introduces additional prerequisites. In order to benefit from this mode, workloads must be compiled with DPDK and use VMNET3 for their vNIC.

    9. In Team Policy Uplink Mapping Select uplink-1 → Uplink 1.

      Warning

      In case you added the ConnectX-6DX please select uplink-1 → Uplink 1 and uplink-2 → Uplink 2.

    10. Click ADD.

      Transport_Nodes_03.png

  3. Click APPLY.

    Transport_Nodes_03b.png

  4. Click SAVE.

    Transport_Nodes_03c.png

  5. Select the Transport Nodes Profile we created (SL-WL01-TNP-EDP-STD) and click SAVE.

    Transport_Nodes_04.png

    You can see the installation is progressing.

    Transport_Nodes_05.png

  6. In the vSphere client.

    Transport_Nodes_06.png

  7. Wait until the hosts are successfully installed, the NSX Configuration status is “ Success ” and the Status indicators are “ Up ”.

    Transport_Nodes_07.png

  8. In the Inventory tab, select the cluster (SL-WL01-Cluster01), select Updates, select Image. Image compliance should show compliant prior to moving on to next steps.

    Transport_Nodes_08.png

Create an NSX Tier-1 Gateway

  1. Navigate to Networking -> Tier-1 Gateway and click on Add Tier-1 Gateway.

    Add_tier_1_GW_01.png

  2. Provide a name for the Tier-1 Gateway (T1-GW) and click on Save to complete creation of the Tier-1 Gateway.

    Add_tier_1_GW_02.png

  3. Select NO in response to Do you want to continue configuring this Tier-1 Gateway?

    Add_tier_1_GW_03.png

  4. The T1-GW Tier-1 Gateway is successfully created.

    Add_tier_1_GW_04.png

Create an Overlay Segment with Offload Switches

  1. Navigate to Networking -> Segments and click on Add Segments,

    Add_segment_01.png

  2. Provide a name for the Segment (SL-WL01-Seg01), Connected Gateway (T1-GW), Transport Zone (SL-WL01-Overlay-TZ), Subnets: 10.10.10.1/24.

  3. Click on Save to complete creation of the Segment,

    Add_segment_02.png

  4. Select NO in response to Do you want to continue configuring this Segment?.

    Add_segment_03.png

  5. The SL-WL01-Seg01 segment is successfully created.