Network Direct is a user-mode programming interface specification for Remote Direct Memory Access (RDMA). RDMA is provided by RDMA-enabled network adapters. Because Network Direct is fabric agnostic, it can be used on InfiniBand, iWARP, and RoCE. Network Direct allows RDMA-enabled network interface card manufacturers to expose the RDMA functionality of their network adapters in Windows.
RDMA is a kernel bypass technique which makes it possible to transfer large amounts of data quite rapidly. Because the transfer is performed by the DMA engine on the network adapter, the CPU is not used for the memory movement, which frees the CPU to perform other work.
Network Direct is widely used for High-Performance Computing (HPC) applications in which computational workloads are distributed to large numbers of servers for parallel processing. In addition, various financial markets trading workloads also require extremely low latency and extremely high message rates, which RDMA can provide.
The Network Direct Interface (NDI) architecture provides application developers with a networking interface that enables zero-copy data transfers between applications, kernel-bypass I/O generation and completion processing, and one-sided data transfer operations. NDI is supported by Microsoft and is the recommended method to write an RDMA application. NDI exposes the advanced capabilities of the NVIDIA® networking devices and allows applications to leverage advances of RDMA. Both RoCE and InfiniBand (IB) can implement NDI.
For further information please refer to: http://msdn.microsoft.com/en-us/library/cc904397(v=vs.85).aspx
To run the test, follow the steps below:
Connect two servers to NVIDIA® adapters.
Verify ping between the two servers.
Configure the RoCE version to be:
Linux side - V2
Windows side - V2
Verify that ROCE udp_port is the same on the two servers. For the registry key, refer to RoCE Options section.
Select the server side and the client side, and run accordingly:
Server:
nd_rping/rping -s [-v -V -d] [-S size] [-C count] [-a addr] [-p port]
Client:
nd_rping/rping -c [-v -V -d] [-S size] [-C count] -a addr [-p port]
Executable Options:
Letter
Description
-s
Server side
-P
Persistent server mode allowing multiple connections
-c
Client side
-a
Address
-p
Port
Debug Extensions:
Letter
Description
-v
Displays ping data to stdout every test cycle
-V
Validates ping data every test cycle
-d
Shows debug prints to stdout
-S
Indicates ping data size - must be < (64*1024)
-C
Indicates the number of ping cycles to perform
Example:
Linux server:rping -v -s -a <IP address> -C
10Windows client:
nd_rping -v -c -a <same IP as above> -C
10
In order to use Network Direct with NVIDIA® adapters, NVIDIA® ND Provider should be installed in Windows. The tool can be used to remove, install and list OFA NetworkDirect providers.
Usage:
> ndinstall -h
where:
[-[i|r] [provider]] | Install/remove the specified/default providers. Provider must be one of the following names:
|
[-l] | List OFA ND providers |
[-h] | This text |
Run "ndinstall -i" to install all available NVIDIA® ND Providers.
Installing mlx5nd provider: successful Installing mlx5nd2 provider: successful Current providers:
0000001001- Hyper-V RAW0000001006- MSAFD Tcpip [TCP/IP]0000001007- MSAFD Tcpip [UDP/IP]0000001008- MSAFD Tcpip [RAW/IP]0000001009- MSAFD Tcpip [TCP/IPv6]0000001010- MSAFD Tcpip [UDP/IPv6]0000001011- MSAFD Tcpip [RAW/IPv6]0000001016- RSVP TCPv6 Service Provider0000001017- RSVP TCP Service Provider0000001018- RSVP UDPv6 Service Provider0000001019- RSVP UDP Service Provider0000001055- NDv1 ProviderforMellanox WinOF-20000001056- NDv2 ProviderforMellanox WinOF-2Run "ndinstall -l" to see a list of installed ND Providers:
Current providers:
0000001001- Hyper-V RAW0000001006- MSAFD Tcpip [TCP/IP]0000001007- MSAFD Tcpip [UDP/IP]0000001008- MSAFD Tcpip [RAW/IP]0000001009- MSAFD Tcpip [TCP/IPv6]0000001010- MSAFD Tcpip [UDP/IPv6]0000001011- MSAFD Tcpip [RAW/IPv6]0000001016- RSVP TCPv6 Service Provider0000001017- RSVP TCP Service Provider0000001018- RSVP UDPv6 Service Provider0000001019- RSVP UDP Service Provider0000001055- NDv1 ProviderforMellanox WinOF-20000001056- NDv2 ProviderforMellanox WinOF-2
In the example above you can see that NDv1 and NDv2 NVIDIA® Providers are installed.