Network Direct is a user-mode programming interface specification for Remote Direct Memory Access (RDMA). RDMA is provided by RDMA-enabled network adapters. Because Network Direct is fabric agnostic, it can be used on InfiniBand, iWARP, and RoCE. Network Direct allows RDMA-enabled network interface card manufacturers to expose the RDMA functionality of their network adapters in Windows.
RDMA is a kernel bypass technique which makes it possible to transfer large amounts of data quite rapidly. Because the transfer is performed by the DMA engine on the network adapter, the CPU is not used for the memory movement, which frees the CPU to perform other work.
Network Direct is widely used for High-Performance Computing (HPC) applications in which computational workloads are distributed to large numbers of servers for parallel processing. In addition, various financial markets trading workloads also require extremely low latency and extremely high message rates, which RDMA can provide.
The Network Direct Interface (NDI) architecture provides application developers with a networking interface that enables zero-copy data transfers between applications, kernel-bypass I/O generation and completion processing, and one-sided data transfer operations. NDI is supported by Microsoft and is the recommended method to write an RDMA application. NDI exposes the advanced capabilities of the NVIDIA® networking devices and allows applications to leverage advances of RDMA. Both RoCE and InfiniBand (IB) can implement NDI.
For further information please refer to: http://msdn.microsoft.com/en-us/library/cc904397(v=vs.85).aspx
To run the test, follow the steps below:
- Connect two servers to NVIDIA® adapters.
- Verify ping between the two servers.
- Configure the RoCE version to be:
- Linux side - V2
- Windows side - V2
- Verify that ROCE udp_port is the same on the two servers. For the registry key, refer to RoCE Options section.
- Select the server side and the client side, and run accordingly:
nd_rping/rping -s [-v -V -d] [-S size] [-C count] [-a addr] [-p port]
nd_rping/rping -c [-v -V -d] [-S size] [-C count] -a addr [-p port]
Persistent server mode allowing multiple connections
Displays ping data to stdout every test cycle
Validates ping data every test cycle
Shows debug prints to stdout
Indicates ping data size - must be < (64*1024)
Indicates the number of ping cycles to perform
rping -v -s -a <IP address> -C 10
nd_rping -v -c -a <same IP as above> -C 10
Using Network Direct with NVIDIA® Adapters
In order to use Network Direct with NVIDIA® adapters, NVIDIA® ND Provider should be installed in Windows. The tool can be used to remove, install and list OFA NetworkDirect providers.
> ndinstall -h
Install/remove the specified/default providers.
Provider must be one of the following names:
|[-l]||List OFA ND providers|
"ndinstall -i"to install all available NVIDIA® ND Providers.
Installing mlx5nd provider: successful Installing mlx5nd2 provider: successful Current providers: 0000001001 - Hyper-V RAW 0000001006 - MSAFD Tcpip [TCP/IP] 0000001007 - MSAFD Tcpip [UDP/IP] 0000001008 - MSAFD Tcpip [RAW/IP] 0000001009 - MSAFD Tcpip [TCP/IPv6] 0000001010 - MSAFD Tcpip [UDP/IPv6] 0000001011 - MSAFD Tcpip [RAW/IPv6] 0000001016 - RSVP TCPv6 Service Provider 0000001017 - RSVP TCP Service Provider 0000001018 - RSVP UDPv6 Service Provider 0000001019 - RSVP UDP Service Provider 0000001055 - NDv1 Provider for Mellanox WinOF-2 0000001056 - NDv2 Provider for Mellanox WinOF-2
"ndinstall -l"to see a list of installed ND Providers:
Current providers: 0000001001 - Hyper-V RAW 0000001006 - MSAFD Tcpip [TCP/IP] 0000001007 - MSAFD Tcpip [UDP/IP] 0000001008 - MSAFD Tcpip [RAW/IP] 0000001009 - MSAFD Tcpip [TCP/IPv6] 0000001010 - MSAFD Tcpip [UDP/IPv6] 0000001011 - MSAFD Tcpip [RAW/IPv6] 0000001016 - RSVP TCPv6 Service Provider 0000001017 - RSVP TCP Service Provider 0000001018 - RSVP UDPv6 Service Provider 0000001019 - RSVP UDP Service Provider 0000001055 - NDv1 Provider for Mellanox WinOF-2 0000001056 - NDv2 Provider for Mellanox WinOF-2
In the example above you can see that NDv1 and NDv2 NVIDIA® Providers are installed.