NVIDIA Networking Server-Side Preboot Drivers (Flexboot/UEFI) User Manual v18
NVIDIA Docs Hub  NVIDIA Networking Server-Side Preboot Drivers (Flexboot/UEFI) User Manual v18  Diskless Machines
NVIDIA ConnectX-5 Adapter Cards Firmware Release Notes v16.35.3502 LTS

On This Page

Diskless Machines

FlexBoot supports booting diskless machines.

Warning

The following configuration insturctions apply only to operation systems which do not include an inbox driver, and should be skipped in case an OS with an inbox driver is used.

To enable using an IB/ETH driver, the initrd image must include a device driver module and be configured to load that driver. This can be achieved by adding the device driver module into the initrd image and loading it.

The 'initrd' image of some Linux distributions such as SuSE Linux Enterprise Server and Red Hat Enterprise Linux, cannot be edited prior or during the installation process.
If you need to install Linux distributions over Flexboot, please replace your 'initrd'
images with the images found at: www.mellanox.com → Products → InfiniBand/VPI Drivers → FlexBoot (Download Tab).
All OSes which does not include inbox driver should do the following.
OSes which include the inbox driver should skip these configurations

InfiniBand Ports

The IB driver requires loading the following modules in the specified order:

  • ib_addr.ko

  • ib_core.ko

  • ib_mad.ko

  • ib_sa.ko

  • ib_cm.ko

  • ib_uverbs.ko

  • ib_ucm.ko

  • ib_umad.ko

  • iw_cm.ko

  • rdma_cm.ko

  • rdma_ucm.ko

  • mlx4_core.ko

  • mlx4_ib.ko

  • ib_mthca.ko

  • ipoib_helper.ko – this module is not required for all OS kernels. Please check the release notes.

  • ib_ipoib.ko

Example: Adding an IB Driver to initrd (Linux)

Prerequisites

  • The FlexBoot image is already programmed on the HCA card

  • The DHCP server is installed, configured and connected to the client machine

  • An initrd file.

  • To add an IB driver into initrd, you need to copy the IB modules to the diskless image. Your machine needs to be pre-installed with a MLNX_OFED for Linux ISO image (available for download from www.mellanox.com → Products → InfiniBand/VPI Drivers → Linux SW/ Drivers) that is appropriate for the kernel version the diskless image will run.

The remainder of this section assumes that MLNX_OFED has been installed on your machine.

Adding the IB Driver to the initrd File

Warning

The following procedure modifies critical files used in the boot procedure. It must be executed by users with expertise in the boot process. Improper application of this procedure may prevent the diskless machine from booting.

  1. Back up your current initrd file.

  2. Make a new working directory and change to it.

    Copy
    Copied!
                
    
            
    host1$ mkdir /tmp/initrd_ib 
host1$ cd /tmp/initrd_ib

  3. Extract the initrd.

    Copy
    Copied!
                
    
            
    host1$ gzip -dc <initrd image> | cpio -id
The initrd files are now located at: /tmp/initrd_ib

  4. Create a directory for the InfiniBand modules and copy them.

    Copy
    Copied!
                
    
            
    host1$ mkdir -p /tmp/initrd_ib/lib/modules/ib
host1$ cd /lib/modules/`uname -r`/updates/kernel/drivers
host1$ cp infiniband/core/ib_addr.ko /tmp/initrd_ib/lib/modules/ib 
host1$ cp infiniband/core/ib_core.ko /tmp/initrd_ib/lib/modules/ib 
host1$ cp infiniband/core/ib_mad.ko /tmp/initrd_ib/lib/modules/ib 
host1$ cp infiniband/core/ib_sa.ko /tmp/initrd_ib/lib/modules/ib 
host1$ cp infiniband/core/ib_cm.ko /tmp/initrd_ib/lib/modules/ib 
host1$ cp infiniband/core/ib_uverbs.ko /tmp/initrd_ib/lib/modules/ib 
host1$ cp infiniband/core/ib_ucm.ko /tmp/initrd_ib/lib/modules/ib 
host1$ cp infiniband/core/ib_umad.ko /tmp/initrd_ib/lib/modules/ib 
host1$ cp infiniband/core/iw_cm.ko /tmp/initrd_ib/lib/modules/ib 
host1$ cp infiniband/core/rdma_cm.ko /tmp/initrd_ib/lib/modules/ib 
host1$ cp infiniband/core/rdma_ucm.ko /tmp/initrd_ib/lib/modules/ib 
host1$ cp net/mlx4/mlx4_core.ko /tmp/initrd_ib/lib/modules/ib
host1$ cp infiniband/hw/mlx4/mlx4_ib.ko /tmp/initrd_ib/lib/modules/ib host1$ cp infiniband/hw/mthca/ib_mthca.ko /tmp/initrd_ib/lib/modules/ib host1$ cp infiniband/ulp/ipoib/ipoib_helper.ko /tmp/initrd_ib/lib/modules/ib 
host1$ cp infiniband/ulp/ipoib/ib_ipoib.ko /tmp/initrd_ib/lib/modules/ib

  5. Load the IPv6 module. If you do not have it in your initrd, add it using the following command.

    Copy
    Copied!
                
    
            
    host1$ cp /lib/modules/`uname -r`/kernel/net/ipv6/ipv6.ko
/tmp/initrd_ib/lib/modules

  6. Load the modules using the insmod executable. If you do not have it in your initrd, add it using the following command.

    Copy
    Copied!
                
    
            
    host1$ cp /sbin/insmod /tmp/initrd_ib/sbin/

  7. [Optional] Give your IB device a static IP address.

    Copy
    Copied!
                
    
            
    host1$ cp /sbin/ifconfig /tmp/initrd_ib/sbin

  8. [Optional] Obtain an IP address for the IB device through DHCP. Copy the DHCP client which was compiled specifically to support IB.
    To continue with this step, DHCP client needs to be already installed on the machine you are working with.
    Copy the DHCP client file and all the relevant files as described below.

    Copy
    Copied!
                
    
            
    host1# cp <path to DHCP client>/dhclient /tmp/initrd_ib/sbin 
host1# cp <path to DHCP client>/dhclient-script /tmp/initrd_ib/sbin 
host1# mkdir -p /tmp/initrd_ib/var/state/dhcp
host1# touch /tmp/initrd_ib/var/state/dhcp/dhclient.leases 
host1# cp /bin/uname /tmp/initrd_ib/bin
host1# cp /usr/bin/expr /tmp/initrd_ib/bin 
host1# cp /sbin/ifconfig /tmp/initrd_ib/bin 
host1# cp /bin/hostname /tmp/initrd_ib/bin

  9. Create a configuration file for the DHCP client and place it under/tmp/initrd_ib/sbin.
    Example of the dclient.conf

    Copy
    Copied!
                
    
            
    The value indicates a hexadecimal number
For a ConnectX® device interface "ib0" {send dhcp-client-identifier ff:00:00:00:00:00:02:00:00:02:c9:00:00:02:c9:03:00:00:10:39;
}

  10. Add the commands for loading the copied modules into the file init at the point you wish the IB driver to be loaded.
    Please pay attention to the following commands order (for loading modules) as it is critical.
    Loading "ipoib_helper.ko" is not required for all OS kernels.

    Copy
    Copied!
                
    
            
    echo "loading ipv6"
/sbin/insmod /lib/modules/ipv6.ko echo "loading IB driver"
/sbin/insmod /lib/modules/ib/ib_addr.ko
/sbin/insmod /lib/modules/ib/ib_core.ko
/sbin/insmod /lib/modules/ib/ib_mad.ko
/sbin/insmod /lib/modules/ib/ib_sa.ko
/sbin/insmod /lib/modules/ib/ib_cm.ko
/sbin/insmod /lib/modules/ib/ib_uverbs.ko
/sbin/insmod /lib/modules/ib/ib_ucm.ko
/sbin/insmod /lib/modules/ib/ib_umad.ko
/sbin/insmod /lib/modules/ib/iw_cm.ko
/sbin/insmod /lib/modules/ib/rdma_cm.ko
/sbin/insmod /lib/modules/ib/rdma_ucm.ko
/sbin/insmod /lib/modules/ib/mlx4_core.ko
/sbin/insmod /lib/modules/ib/mlx4_ib.ko
/sbin/insmod /lib/modules/ib/ib_mthca.ko

    In case of interoperability issues between iSCSI and Large Receive Offload (LRO), set the "ib_ipoib.ko lro" to "0" to disable LRO.

    Copy
    Copied!
                
    
            
    /sbin/insmod /lib/modules/ib/ib_ipoib.ko lro=0

  11. Assign an IP address to the IB device. Add either a call to ifconfig or to the DHCP client in the init file after loading the modules. If you want to use the DHCP client, then you need to add a call to the DHCP client in the init file after loading the IB modules.
    For example:

    Copy
    Copied!
                
    
            
    /sbin/dhclient -cf /sbin/dhclient.conf ib1

  12. Save the init file.

  13. Close initrd.

    Copy
    Copied!
                
    
            
    host1$ cd /tmp/initrd_ib
host1$ find ./ | cpio -H newc -o > /tmp/new_initrd_ib.img 
host1$ gzip /tmp/new_init_ib.img

  14. Copy the modified initrd (including the IB driver) which is located at/tmp/new_init_ib.img.gz and rename it properly.

  15. Back up your current initrd file.

  16. Make a new working directory and change to it.

  17. Extract the initrd image.

  18. Create a directory for the ConnectX EN modules and copy them.

  19. Load the modules using the insmod executable. If you do not have it in your initrd, add it using the following command.

  20. Give your Ethernet device a static IP address.

  21. Add the commands for loading the copied modules into the file init at the point you wish the Ethernet driver to be loaded.

  22. Assign an IP address to the ConnectX IB network interface.

  23. Save the init file.

  24. Close initrd.

  25. Copy the modified initrd (including the Ethernet driver) which is located at/tmp/new_init_en.img.gz and rename it properly.

Ethernet Ports

The Ethernet driver requires loading the following modules in the specified order – see the exam- ple below:

  • mlx4_core.ko

  • mlx4_en.ko

Example: Adding an Ethernet Driver to initrd (Linux)

Prerequisites

  • The FlexBoot image is already programmed on the adapter card.

  • The DHCP server is installed, configured and connected to the client machine.

  • An initrd file.

  • To add an Ethernet driver into initrd, you need to copy the Ethernet modules to the diskless image. Your machine needs to be pre-installed with a MLNX_EN Linux Driver (available for download from www.mellanox.com  Products  Ethernet Drivers) that is appropriate for the kernel version the diskless image will run.

Adding the Ethernet Driver to the initrd File

The following procedure modifies critical files used in the boot procedure. It must be executed by users with expertise in the boot process. Improper application of this procedure may prevent the diskless machine from booting.

  1. Back up your current initrd file.

  2. Make a new working directory and change to it.

    Copy
    Copied!
                
    
            
    host1$ mkdir /tmp/initrd_en 
host1$ cd /tmp/initrd_en

  3. Extract the initrd image.

    Copy
    Copied!
                
    
            
    host1$ gzip -dc <initrd image> | cpio -id 
The initrd files can be located at: /tmp/initrd_en

  4. Create a directory for the ConnectX EN modules and copy them.

    Copy
    Copied!
                
    
            
    host1$ mkdir -p /tmp/initrd_en/lib/modules/mlnx_en 
host1$ cd /lib/modules/`uname -r`/updates/kernel/drivers
host1$ cp net/mlx4/mlx4_core.ko /tmp/initrd_en/lib/modules/mlnx_en 
host1$ cp net/mlx4/mlx4_en.ko /tmp/initrd_en/lib/modules/mlnx_en

  5. Load the modules using the insmod executable. If you do not have it in your initrd, add it using the following command.

    Copy
    Copied!
                
    
            
    host1$ cp /sbin/insmod /tmp/initrd_en/sbin/

  6. [Optional] Give your Ethernet device a static IP address.

    Copy
    Copied!
                
    
            
    host1$ cp /sbin/ifconfig /tmp/initrd_en/sbin

  7. Add the commands for loading the copied modules into the file init at the point you wish the Ethernet driver to be loaded.
    Please pay attention to the following commands order (for loading modules) as it is critical.

    Copy
    Copied!
                
    
            
    echo "loading Mellanox ConnectX EN driver"
/sbin/insmod lib/modules/mlnx_en/mlx4_core.ko
/sbin/insmod lib/modules/mlnx_en/mlx4_en.ko

  8. Assign an IP address to the ConnectX EB network interface.

  9. Save the init file.

  10. Close initrd.

    Copy
    Copied!
                
    
            
    host1$ cd /tmp/initrd_en
host1$ find ./ | cpio -H newc -o > /tmp/new_initrd_en.img 
host1$ gzip /tmp/new_init_en.img

  11. Copy the modified initrd (including the Ethernet driver) which is located at/tmp/new_init_en.img.gz and rename it properly.

  12. Back up your current initrd file.

  13. Make a new working directory and change to it.

  14. Extract the initrd image.

  15. Create a directory for the ConnectX EN modules and copy them.

  16. Load the modules using the insmod executable. If you do not have it in your initrd, add it using the following command.

  17. Give your Ethernet device a static IP address.

  18. Add the commands for loading the copied modules into the file init at the point you wish the Ethernet driver to be loaded.

  19. Assign an IP address to the ConnectX EN network interface.

  20. Save the init file.

  21. Close initrd.

  22. Copy the modified initrd (including the Ethernet driver) which is located at/tmp/new_init_en.img.gz and rename it properly.

© Copyright 2023, NVIDIA. Last updated on Oct 11, 2023.