Diskless Machines
FlexBoot supports booting diskless machines.
The following configuration insturctions apply only to operation systems which do not include an inbox driver, and should be skipped in case an OS with an inbox driver is used.
To enable using an IB/ETH driver, the initrd image must include a device driver module and be configured to load that driver. This can be achieved by adding the device driver module into the initrd image and loading it.
The 'initrd' image of some Linux distributions such as SuSE Linux Enterprise Server and Red Hat Enterprise Linux, cannot be edited prior or during the installation process.
If you need to install Linux distributions over Flexboot, please replace your 'initrd'
images with the images found at: www.mellanox.com → Products → InfiniBand/VPI Drivers → FlexBoot (Download Tab).
All OSes which does not include inbox driver should do the following.
OSes which include the inbox driver should skip these configurations
The IB driver requires loading the following modules in the specified order:
ib_addr.ko
ib_core.ko
ib_mad.ko
ib_sa.ko
ib_cm.ko
ib_uverbs.ko
ib_ucm.ko
ib_umad.ko
iw_cm.ko
rdma_cm.ko
rdma_ucm.ko
mlx4_core.ko
mlx4_ib.ko
ib_mthca.ko
ipoib_helper.ko – this module is not required for all OS kernels. Please check the release notes.
ib_ipoib.ko
Example: Adding an IB Driver to initrd (Linux)
Prerequisites
The FlexBoot image is already programmed on the HCA card
The DHCP server is installed, configured and connected to the client machine
An initrd file.
To add an IB driver into initrd, you need to copy the IB modules to the diskless image. Your machine needs to be pre-installed with a MLNX_OFED for Linux ISO image (available for download from www.mellanox.com → Products → InfiniBand/VPI Drivers → Linux SW/ Drivers) that is appropriate for the kernel version the diskless image will run.
The remainder of this section assumes that MLNX_OFED has been installed on your machine.
Adding the IB Driver to the initrd File
The following procedure modifies critical files used in the boot procedure. It must be executed by users with expertise in the boot process. Improper application of this procedure may prevent the diskless machine from booting.
Back up your current initrd file.
Make a new working directory and change to it.
host1$ mkdir /tmp/initrd_ib host1$ cd /tmp/initrd_ib
Extract the initrd.
host1$ gzip -dc <initrd image> | cpio -id The initrd files are now located at: /tmp/initrd_ib
Create a directory for the InfiniBand modules and copy them.
host1$ mkdir -p /tmp/initrd_ib/lib/modules/ib host1$ cd /lib/modules/`uname -r`/updates/kernel/drivers host1$ cp infiniband/core/ib_addr.ko /tmp/initrd_ib/lib/modules/ib host1$ cp infiniband/core/ib_core.ko /tmp/initrd_ib/lib/modules/ib host1$ cp infiniband/core/ib_mad.ko /tmp/initrd_ib/lib/modules/ib host1$ cp infiniband/core/ib_sa.ko /tmp/initrd_ib/lib/modules/ib host1$ cp infiniband/core/ib_cm.ko /tmp/initrd_ib/lib/modules/ib host1$ cp infiniband/core/ib_uverbs.ko /tmp/initrd_ib/lib/modules/ib host1$ cp infiniband/core/ib_ucm.ko /tmp/initrd_ib/lib/modules/ib host1$ cp infiniband/core/ib_umad.ko /tmp/initrd_ib/lib/modules/ib host1$ cp infiniband/core/iw_cm.ko /tmp/initrd_ib/lib/modules/ib host1$ cp infiniband/core/rdma_cm.ko /tmp/initrd_ib/lib/modules/ib host1$ cp infiniband/core/rdma_ucm.ko /tmp/initrd_ib/lib/modules/ib host1$ cp net/mlx4/mlx4_core.ko /tmp/initrd_ib/lib/modules/ib host1$ cp infiniband/hw/mlx4/mlx4_ib.ko /tmp/initrd_ib/lib/modules/ib host1$ cp infiniband/hw/mthca/ib_mthca.ko /tmp/initrd_ib/lib/modules/ib host1$ cp infiniband/ulp/ipoib/ipoib_helper.ko /tmp/initrd_ib/lib/modules/ib host1$ cp infiniband/ulp/ipoib/ib_ipoib.ko /tmp/initrd_ib/lib/modules/ib
Load the IPv6 module. If you do not have it in your initrd, add it using the following command.
host1$ cp /lib/modules/`uname -r`/kernel/net/ipv6/ipv6.ko /tmp/initrd_ib/lib/modules
Load the modules using the insmod executable. If you do not have it in your initrd, add it using the following command.
host1$ cp /sbin/insmod /tmp/initrd_ib/sbin/
[Optional] Give your IB device a static IP address.
host1$ cp /sbin/ifconfig /tmp/initrd_ib/sbin
[Optional] Obtain an IP address for the IB device through DHCP. Copy the DHCP client which was compiled specifically to support IB.
To continue with this step, DHCP client needs to be already installed on the machine you are working with.
Copy the DHCP client file and all the relevant files as described below.
host1# cp <path to DHCP client>/dhclient /tmp/initrd_ib/sbin host1# cp <path to DHCP client>/dhclient-script /tmp/initrd_ib/sbin host1# mkdir -p /tmp/initrd_ib/var/state/dhcp host1# touch /tmp/initrd_ib/var/state/dhcp/dhclient.leases host1# cp /bin/uname /tmp/initrd_ib/bin host1# cp /usr/bin/expr /tmp/initrd_ib/bin host1# cp /sbin/ifconfig /tmp/initrd_ib/bin host1# cp /bin/hostname /tmp/initrd_ib/bin
Create a configuration file for the DHCP client and place it under/tmp/initrd_ib/sbin.
Example of the dclient.conf
The value indicates a hexadecimal number For a ConnectX® device
interface
"ib0"
{send dhcp-client-identifier ff:00
:00
:00
:00
:00
:02
:00
:00
:02
:c9:00
:00
:02
:c9:03
:00
:00
:10
:39
; }Add the commands for loading the copied modules into the file init at the point you wish the IB driver to be loaded.
Please pay attention to the following commands order (for loading modules) as it is critical.
Loading "ipoib_helper.ko" is not required for all OS kernels.
echo
"loading ipv6"
/sbin/insmod /lib/modules/ipv6.ko echo"loading IB driver"
/sbin/insmod /lib/modules/ib/ib_addr.ko /sbin/insmod /lib/modules/ib/ib_core.ko /sbin/insmod /lib/modules/ib/ib_mad.ko /sbin/insmod /lib/modules/ib/ib_sa.ko /sbin/insmod /lib/modules/ib/ib_cm.ko /sbin/insmod /lib/modules/ib/ib_uverbs.ko /sbin/insmod /lib/modules/ib/ib_ucm.ko /sbin/insmod /lib/modules/ib/ib_umad.ko /sbin/insmod /lib/modules/ib/iw_cm.ko /sbin/insmod /lib/modules/ib/rdma_cm.ko /sbin/insmod /lib/modules/ib/rdma_ucm.ko /sbin/insmod /lib/modules/ib/mlx4_core.ko /sbin/insmod /lib/modules/ib/mlx4_ib.ko /sbin/insmod /lib/modules/ib/ib_mthca.koIn case of interoperability issues between iSCSI and Large Receive Offload (LRO), set the "ib_ipoib.ko lro" to "0" to disable LRO.
/sbin/insmod /lib/modules/ib/ib_ipoib.ko lro=
0
Assign an IP address to the IB device. Add either a call to ifconfig or to the DHCP client in the init file after loading the modules. If you want to use the DHCP client, then you need to add a call to the DHCP client in the init file after loading the IB modules.
For example:
/sbin/dhclient -cf /sbin/dhclient.conf ib1
Save the init file.
Close initrd.
host1$ cd /tmp/initrd_ib host1$ find ./ | cpio -H newc -o > /tmp/new_initrd_ib.img host1$ gzip /tmp/new_init_ib.img
Copy the modified initrd (including the IB driver) which is located at/tmp/new_init_ib.img.gz and rename it properly.
Back up your current initrd file.
Make a new working directory and change to it.
Extract the initrd image.
Create a directory for the ConnectX EN modules and copy them.
Load the modules using the insmod executable. If you do not have it in your initrd, add it using the following command.
Give your Ethernet device a static IP address.
Add the commands for loading the copied modules into the file init at the point you wish the Ethernet driver to be loaded.
Assign an IP address to the ConnectX IB network interface.
Save the init file.
Close initrd.
Copy the modified initrd (including the Ethernet driver) which is located at/tmp/new_init_en.img.gz and rename it properly.
The Ethernet driver requires loading the following modules in the specified order – see the exam- ple below:
mlx4_core.ko
mlx4_en.ko
Example: Adding an Ethernet Driver to initrd (Linux)
Prerequisites
The FlexBoot image is already programmed on the adapter card.
The DHCP server is installed, configured and connected to the client machine.
An initrd file.
To add an Ethernet driver into initrd, you need to copy the Ethernet modules to the diskless image. Your machine needs to be pre-installed with a MLNX_EN Linux Driver (available for download from www.mellanox.com Products Ethernet Drivers) that is appropriate for the kernel version the diskless image will run.
Adding the Ethernet Driver to the initrd File
The following procedure modifies critical files used in the boot procedure. It must be executed by users with expertise in the boot process. Improper application of this procedure may prevent the diskless machine from booting.
Back up your current initrd file.
Make a new working directory and change to it.
host1$ mkdir /tmp/initrd_en host1$ cd /tmp/initrd_en
Extract the initrd image.
host1$ gzip -dc <initrd image> | cpio -id The initrd files can be located at: /tmp/initrd_en
Create a directory for the ConnectX EN modules and copy them.
host1$ mkdir -p /tmp/initrd_en/lib/modules/mlnx_en host1$ cd /lib/modules/`uname -r`/updates/kernel/drivers host1$ cp net/mlx4/mlx4_core.ko /tmp/initrd_en/lib/modules/mlnx_en host1$ cp net/mlx4/mlx4_en.ko /tmp/initrd_en/lib/modules/mlnx_en
Load the modules using the insmod executable. If you do not have it in your initrd, add it using the following command.
host1$ cp /sbin/insmod /tmp/initrd_en/sbin/
[Optional] Give your Ethernet device a static IP address.
host1$ cp /sbin/ifconfig /tmp/initrd_en/sbin
Add the commands for loading the copied modules into the file init at the point you wish the Ethernet driver to be loaded.
Please pay attention to the following commands order (for loading modules) as it is critical.
echo
"loading Mellanox ConnectX EN driver"
/sbin/insmod lib/modules/mlnx_en/mlx4_core.ko /sbin/insmod lib/modules/mlnx_en/mlx4_en.koAssign an IP address to the ConnectX EB network interface.
Save the init file.
Close initrd.
host1$ cd /tmp/initrd_en host1$ find ./ | cpio -H newc -o > /tmp/new_initrd_en.img host1$ gzip /tmp/new_init_en.img
Copy the modified initrd (including the Ethernet driver) which is located at/tmp/new_init_en.img.gz and rename it properly.
Back up your current initrd file.
Make a new working directory and change to it.
Extract the initrd image.
Create a directory for the ConnectX EN modules and copy them.
Load the modules using the insmod executable. If you do not have it in your initrd, add it using the following command.
Give your Ethernet device a static IP address.
Add the commands for loading the copied modules into the file init at the point you wish the Ethernet driver to be loaded.
Assign an IP address to the ConnectX EN network interface.
Save the init file.
Close initrd.
Copy the modified initrd (including the Ethernet driver) which is located at/tmp/new_init_en.img.gz and rename it properly.