NVIDIA Networking Server-Side Preboot Drivers (Flexboot/UEFI) User Manual v13.0
NVIDIA ConnectX-5 Adapter Cards Firmware Release Notes v16.35.3502 LTS

Case I: InfiniBand Ports

PreBoot drivers require that the DHCP server runs on a machine which supports IP over IB. Prior to installing DHCP, make sure that MLNX_OFED is already installed on your DHCP server – see www.mellanox.com.
ISC DHCP of versions above 3.1.3 have native support for IPoIB clients. If using DHCP v3.1.3, the provided patch must be deployed. Newer versions do not require patching.
To download and install other supported DHCP server versions, please download MLNX_OFED driver from www.mellanox.com → Products → InfiniBand/VPI Drivers → Linux SW/Drivers and refer to docs/dhcp/README.
Standard DHCP fields holding MAC addresses are not large enough to contain an IPoIB hardware address. To overcome this problem, DHCP over InfiniBand messages convey a client identifier field (in DHCP option 61) used to identify the DHCP session. This client identifier field can be used to associate an IP address with a client identifier value, such that the DHCP server will grant the same IP address to any client that conveys this client identifier.

Warning

In the DHCP discover packets, Flexboot sends "hlen" 6 rather than 0. "chaddr" will be the ethernet mac address rather than zeros.
In the DHCP server, the GUID in the client identifier can still be used to identify clients.

For ConnectX Family Devices

When a FlexBoot client boots, it sends the DHCP server various information including its DHCP client identifier. This identifier is used to distinguish between the various DHCP sessions. The value of the client identifier is composed of a prefix — ff:00:00:00:00:00:02:00:00:02:c9:00 — and an 8-byte port GUID (all separated by colons and represented in hexadecimal digits).

Extracting the Port GUID – Method I

  1. Start mst.
    host1# mst start
    host1# mst status

  2. The following MFT commands assume that the MFT package has been installed on the client machine.Obtain the Port GUID using the device name. The device name will be of the form: /dev/mst/ mt<dev_id>_pci{_cr0|conf0}.
    flint -d <MST_DEVICE_NAME> q

Assuming FlexBoot is connected via Port 1, then the Port GUID is 00:02:c9:03:00:00:10:39

Extracting the Port GUID – Method II

An alternative method for obtaining the port GUID involves booting the client machine via Flex- Boot. This requires having a Subnet Manager running on one of the machines in the InfiniBand subnet. The 8 bytes can be captured from the boot session as shown in the figure below.

worddavce83b9ce30e0506a82e6d6ba6f80882f.png

Placing Client Identifiers in /etc/dhcpd.conf

The following is an excerpt of a /etc/dhcpd.conf example file showing the format of representing a client machine for the DHCP server

Copy
Copied!
            

host host1 { next-server 11.4.3.7; filename "pxelinux.0"; fixed-address 11.4.3.130; option dhcp-client-identifier = ff:00:00:00:00:00:02:00:00:02:c9:00:00:02:c9:03:00:0c:78:11; }

Warning

This section is applicable only when the Initiator's IP address is empty.

For the DHCP server to provide configuration records for clients, an appropriate configuration file needs to be created. By default, the DHCP server looks for a configuration file called dhcpd.conf under /etc. You can either edit this file or create a new one and provide its full path to the DHCP server using the -cf flag. See a file example at docs/dhcpd.conf of this package.

The DHCP server must run on a machine which has loaded the IPoIB module.

Copy
Copied!
            

dhcpd <IB network interface name> -d 

Example:

Copy
Copied!
            

host1# dhcpd ib0 –d

Warning

A DHCP client can be used if you need to prepare a diskless machine with an IB driver. See Step 11 under Example: Adding an IB Driver to initrd (Linux).

© Copyright 2023, NVIDIA. Last updated on May 23, 2023.