image image image image image

On This Page

Network Direct is a user-mode programming interface specification for Remote Direct Memory Access (RDMA). RDMA is provided by RDMA-enabled network adapters. Because Network Direct is fabric agnostic, it can be used on InfiniBand, iWARP, and RoCE. Network Direct allows RDMA-enabled network interface card manufacturers to expose the RDMA functionality of their network adapters in Windows.
RDMA is a kernel bypass technique which makes it possible to transfer large amounts of data quite rapidly. Because the transfer is performed by the DMA engine on the network adapter, the CPU is not used for the memory movement, which frees the CPU to perform other work.
Network Direct is widely used for High-Performance Computing (HPC) applications in which computational workloads are distributed to large numbers of servers for parallel processing. In addition, various financial markets trading workloads also require extremely low latency and extremely high message rates, which RDMA can provide.

The Network Direct Interface (NDI) architecture provides application developers with a networking interface that enables zero-copy data transfers between applications, kernel-bypass I/O generation and completion processing, and one-sided data transfer operations. NDI is supported by Microsoft and is the recommended method to write an RDMA application. NDI exposes the advanced capabilities of the NVIDIA® networking devices and allows applications to leverage advances of RDMA. Both RoCE and InfiniBand (IB) can implement NDI.

For further information please refer to: http://msdn.microsoft.com/en-us/library/cc904397(v=vs.85).aspx

Test Running

To run the test, follow the steps below:

  1. Connect two servers to NVIDIA® adapters.
  2. Verify ping between the two servers.
  3. Configure the RoCE version to be:
    • Linux side - V2
    • Windows side - V2
    • Verify that ROCE udp_port is the same on the two servers. For the registry key, refer to RoCE Options section.
  4. Select the server side and the client side, and run accordingly:

    • Server:

      nd_rping/rping -s [-v -V -d] [-S size] [-C count] [-a addr] [-p port]

    • Client:

      nd_rping/rping -c [-v -V -d] [-S size] [-C count] -a addr [-p port]

      Executable Options:

      LetterDescription

      -s

      Server side

      -P

      Persistent server mode allowing multiple connections

      -c

      Client side

      -a

      Address

      -p

      Port

      Debug Extensions:

      LetterDescription

      -v

      Displays ping data to stdout every test cycle

      -V

      Validates ping data every test cycle

      -d

      Shows debug prints to stdout

      -S

      Indicates ping data size - must be < (64*1024)

      -C

      Indicates the number of ping cycles to perform

      Example:
      Linux server: 

      rping -v -s -a <IP address> -C 10

      Windows client:

      nd_rping -v -c -a <same IP as above> -C 10

Using Network Direct with NVIDIA® Adapters

In order to use Network Direct with NVIDIA® adapters, NVIDIA® ND Provider should be installed in Windows. The tool can be used to remove, install and list OFA NetworkDirect providers.

Usage:

> ndinstall -h

where:

[-[i|r] [provider]] 

Install/remove the specified/default providers.

Provider must be one of the following names:

  • mlx4nd
  • mlx4nd2
  • mlx5nd
  • mlx5nd2
  • <blank> use the default ND providers
[-l]List OFA ND providers
[-h]

This text

  • Run "ndinstall -i" to install all available NVIDIA® ND Providers.

    Installing mlx5nd provider: successful
Installing mlx5nd2 provider: successful

Current providers:
        0000001001 - Hyper-V RAW
        0000001006 - MSAFD Tcpip [TCP/IP]
        0000001007 - MSAFD Tcpip [UDP/IP]
        0000001008 - MSAFD Tcpip [RAW/IP]
        0000001009 - MSAFD Tcpip [TCP/IPv6]
        0000001010 - MSAFD Tcpip [UDP/IPv6]
        0000001011 - MSAFD Tcpip [RAW/IPv6]
        0000001016 - RSVP TCPv6 Service Provider
        0000001017 - RSVP TCP Service Provider
        0000001018 - RSVP UDPv6 Service Provider
        0000001019 - RSVP UDP Service Provider
        0000001055 - NDv1 Provider for Mellanox WinOF-2
        0000001056 - NDv2 Provider for Mellanox WinOF-2

  • Run "ndinstall -l" to see a list of installed ND Providers:

    Current providers:
        0000001001 - Hyper-V RAW
        0000001006 - MSAFD Tcpip [TCP/IP]
        0000001007 - MSAFD Tcpip [UDP/IP]
        0000001008 - MSAFD Tcpip [RAW/IP]
        0000001009 - MSAFD Tcpip [TCP/IPv6]
        0000001010 - MSAFD Tcpip [UDP/IPv6]
        0000001011 - MSAFD Tcpip [RAW/IPv6]
        0000001016 - RSVP TCPv6 Service Provider
        0000001017 - RSVP TCP Service Provider
        0000001018 - RSVP UDPv6 Service Provider
        0000001019 - RSVP UDP Service Provider
        0000001055 - NDv1 Provider for Mellanox WinOF-2
        0000001056 - NDv2 Provider for Mellanox WinOF-2

In the example above you can see that NDv1 and NDv2 NVIDIA® Providers are installed.