Network Direct Interface

Linux Kernel Upstream Release Notes v6.5

Network Direct is a user-mode programming interface specification for Remote Direct Memory Access (RDMA). RDMA is provided by RDMA-enabled network adapters. Because Network Direct is fabric agnostic, it can be used on InfiniBand, iWARP, and RoCE. Network Direct allows RDMA-enabled network interface card manufacturers to expose the RDMA functionality of their network adapters in Windows.
RDMA is a kernel bypass technique which makes it possible to transfer large amounts of data quite rapidly. Because the transfer is performed by the DMA engine on the network adapter, the CPU is not used for the memory movement, which frees the CPU to perform other work.
Network Direct is widely used for High-Performance Computing (HPC) applications in which computational workloads are distributed to large numbers of servers for parallel processing. In addition, various financial markets trading workloads also require extremely low latency and extremely high message rates, which RDMA can provide.

The Network Direct Interface (NDI) architecture provides application developers with a networking interface that enables zero-copy data transfers between applications, kernel-bypass I/O generation and completion processing, and one-sided data transfer operations. NDI is supported by Microsoft and is the recommended method to write an RDMA application. NDI exposes the advanced capabilities of the NVIDIA® networking devices and allows applications to leverage advances of RDMA. Both RoCE and InfiniBand (IB) can implement NDI.

For further information please refer to: http://msdn.microsoft.com/en-us/library/cc904397(v=vs.85).aspx

Procedure_Heading_Icon.PNG

To run the test, follow the steps below:

  1. Connect two servers to NVIDIA® adapters.

  2. Verify ping between the two servers.

  3. Configure the RoCE version to be:

    • Linux side - V2

    • Windows side - V2

    • Verify that ROCE udp_port is the same on the two servers. For the registry key, refer to RoCE Options section.

  4. Select the server side and the client side, and run accordingly:

  • Server:

    Copy
    Copied!
                

    nd_rping/rping -s [-v -V -d] [-S size] [-C count] [-a addr] [-p port]

  • Client:

    Copy
    Copied!
                

    nd_rping/rping -c [-v -V -d] [-S size] [-C count] -a addr [-p port]

    Executable Options:

    Letter

    Description

    -s

    Server side

    -P

    Persistent server mode allowing multiple connections

    -c

    Client side

    -a

    Address

    -p

    Port

    Debug Extensions:

    Letter

    Description

    -v

    Displays ping data to stdout every test cycle

    -V

    Validates ping data every test cycle

    -d

    Shows debug prints to stdout

    -S

    Indicates ping data size - must be < (64*1024)

    -C

    Indicates the number of ping cycles to perform

    Example:
    Linux server:

    Copy
    Copied!
                

    rping -v -s -a <IP address> -C 10

    Windows client:

    Copy
    Copied!
                

    nd_rping -v -c -a <same IP as above> -C 10

In order to use Network Direct with NVIDIA® adapters, NVIDIA® ND Provider should be installed in Windows. The tool can be used to remove, install and list OFA NetworkDirect providers.

Usage:

Copy
Copied!
            

> ndinstall -h

where:

[-[i|r] [provider]]

Install/remove the specified/default providers.

Provider must be one of the following names:

  • mlx4nd

  • mlx4nd2

  • mlx5nd

  • mlx5nd2

  • <blank> use the default ND providers

[-l]

List OFA ND providers

[-h]

This text

  • Run "ndinstall -i" to install all available NVIDIA® ND Providers.

    Copy
    Copied!
                

    Installing mlx5nd provider: successful Installing mlx5nd2 provider: successful   Current providers: 0000001001 - Hyper-V RAW 0000001006 - MSAFD Tcpip [TCP/IP] 0000001007 - MSAFD Tcpip [UDP/IP] 0000001008 - MSAFD Tcpip [RAW/IP] 0000001009 - MSAFD Tcpip [TCP/IPv6] 0000001010 - MSAFD Tcpip [UDP/IPv6] 0000001011 - MSAFD Tcpip [RAW/IPv6] 0000001016 - RSVP TCPv6 Service Provider 0000001017 - RSVP TCP Service Provider 0000001018 - RSVP UDPv6 Service Provider 0000001019 - RSVP UDP Service Provider 0000001055 - NDv1 Provider for Mellanox WinOF-2 0000001056 - NDv2 Provider for Mellanox WinOF-2

  • Run "ndinstall -l" to see a list of installed ND Providers:

    Copy
    Copied!
                

    Current providers: 0000001001 - Hyper-V RAW 0000001006 - MSAFD Tcpip [TCP/IP] 0000001007 - MSAFD Tcpip [UDP/IP] 0000001008 - MSAFD Tcpip [RAW/IP] 0000001009 - MSAFD Tcpip [TCP/IPv6] 0000001010 - MSAFD Tcpip [UDP/IPv6] 0000001011 - MSAFD Tcpip [RAW/IPv6] 0000001016 - RSVP TCPv6 Service Provider 0000001017 - RSVP TCP Service Provider 0000001018 - RSVP UDPv6 Service Provider 0000001019 - RSVP UDP Service Provider 0000001055 - NDv1 Provider for Mellanox WinOF-2 0000001056 - NDv2 Provider for Mellanox WinOF-2

In the example above you can see that NDv1 and NDv2 NVIDIA® Providers are installed.

© Copyright 2023, NVIDIA. Last updated on Sep 5, 2023.