Enhanced Network Stack (ENS), also appears as Enhanced Data Path is a networking stack mode, which when configured provides superior network performance. It is primarily targeted for NFV workloads, which requires the performance benefits provided by this mode. ENS utilizes DPDK Poll Mode driver model and significantly improves packet rate and latency for small message sizes.
Current driver can operate in both ENS and legacy (slow path) modes. Device mode of operation is determined automatically based on Virtual Switch mode. Once the uplink is attached to NSX-T Virtual Distribute Switch (N-VDS) which is configured for ENS, the driver will be re-attached to the device and will initialize it to work with ENS.
Please follow VMWare documentation for N-VDS configuration instructions:
To achieve best performance, it is recommended to have N-VDS assigned with NUMA node which is local to Mellanox NIC. NUMA node and amount of logical cores for enhanced data- path processing is selected during the initial N-VDS configuration.
For further information, refer to: https://docs.vmware.com/en/VMware-NSX-T-Data-Center/2.3/com.vmware.nsxt.install.doc/GUID-D7CA778B-6554-4A23-879D- 4BC336E01031.html#GUID-D7CA778B-6554-4A23-879D-4BC336E01031
To find out the number for NUMA nodes on your host, run the “esxcli hardware memory get” command.
To find out the NIC’s affinity, run the following command: vsish -e cat /net/pNics/<vmnicX>/properties | grep -i numa
The device supports Flow Processing Offload (FPO), meaning it can offload host's CPU by taking flow related tasks from it and thus enabling it to perform other tasks to achieve better overall performance. FPO must be configured on the DVS with one or more of the supported models (Model 1 Level 1, Model 1 Level 2 and Model 2A). The OS than performs negotiation with the device regarding the best FPO operational model(s) to be used.
The supported ENS models by the device (Model 1 Level 1, Model 1 Level 2 and Model 2A) come with the following restrictions:
If the Virtual Functions are configured for the device, it will support FPO only if the OS configured ENS is set to work in Model 2
In ESXi7.0u1, the device will support FPO in Model 1 if the Virtual Functions are not configured for the device and the module parameter ens_fallback_model is set to 1. Otherwise the device will not support FPO
The device does not support Model 2A at the same time with Model 1 Level 2. If during negotiation, the OS recommends that the device will operate in Model 2A and Model 1 Level 2, the device will refuse to work in Model 1 Level 2
If the device operates in Model 2A, then RDMA will not be supported in the hypervisor but only in the SR-IOV VMs. In this case, RDMA will be supported in the VMs with overlay configuration (the same as for the Ethernet device)
For further information on ENS configuration instructions, please see VMWare documentation:
The following are the current ENS limitations:
The device does not support SR-IOV when attached to ENS N-VDS. In such configuration, max_vfs module parameter for the ENS port will be ignored and no Virtual Functions will be created for this port. Meaning, if we have 2-port devices with vmnic4 and vmnic5 uplinks connected to a regular ENS and ENS DVS respectively, no VFs will be created for vmnic5 PF
RSS NetQ is currently not supported in ENS mode
VXLAN is currently not supported in ENS mode
GENEVE hardware offload enables the traditional offloads to be performed on the encapsulated traffic. With ConnectX-4/ConnectX-5 family adapter cards, data center operators can decouple the overlay network layer from the physical NIC performance, thus achieving native performance in the new network architecture.
Configuring GENEVE Hardware Offload
GENEVE hardware offload includes:
TX: Calculates the Inner L3/L4 and the Outer L3 checksum
RX:
Checks the Inner L3/L4 and the Outer L3 checksum
Maps the GENEVE traffic to an RX queue according to:
Inner destination MAC address
Outer destination MAC address
GENEVE VNI
GENEVE hardware offload is disabled by default.
To enable it, run the “geneve_offload_enable” parameter.
GENEVE configuration is done in the ESXi environment via VMware NSX manager. For additional NSX information, please refer to VMware documentation, see: http://pubs.vmware.com/NSX-62/index.jsp#com.vmware.nsx.install.doc/GUID-D8578F6E-A40C-493A-9B43-877C2B75ED52.html.