eMMC Backup and Restore
The complete setup process can be time-consuming. Fortunately, the filesystem installed on one BlueField® can be directly used on another BlueField system. Therefore, the fastest, most efficient way to install CentOS onto a BlueField system is to restore the eMMC image backup from another BlueField image.
Before backing up the eMMC, all of its partitions need to be unmounted to avoid data corruption. Unmounting the root partition of the CentOS is impractical, therefore using the initial Yocto running entirely on memory is a good option.
If the DPU is currently running CentOS, issue a shutdown command so that the kernel unmounts the entire filesystem:
[root@localhost ~]# shutdown -h now [ OK ] Started Show Plymouth Power Off Screen. [ OK ] Stopped Dynamic System Tuning Daemon. [ OK ] Stopped target Network. Stopping LSB: Bring up/down networking... [ OK ] Stopped LSB: Bring up/down networking. Stopping Network Manager... [ OK ] Stopped Network Manager. …… Unmounting /run/user/0... Unmounting /mnt... Unmounting /boot/efi... [ OK ] Deactivated swap /dev/disk/by-path/platform-PRP0001:00-part3. [ OK ] Deactivated swap /dev/disk/by-partu...4fa-7c6a-4fd4-a795-84415d19f840. [ OK ] Deactivated swap /dev/disk/by-id/mmc-R1J56L_0x353c1019-part3. [ OK ] Deactivated swap /dev/mmcblk0p3. [ OK ] Deactivated swap /dev/disk/by-uuid/...291-1ad6-4e3a-b5b4-9087950a3296. [ OK ] Unmounted /run/user/0. [ OK ] Unmounted /boot/efi. Unmounting /boot... [ OK ] Unmounted /mnt. [14370.028599] XFS (mmcblk0p2): Unmounting Filesystem [ OK ] Unmounted /boot. [ OK ] Reached target Unmount All Filesystems. [ OK ] Stopped target Local File Systems (Pre). [ OK ] Stopped Remount Root and Kernel File Systems. …… [14370.787777] reboot: Power down ERROR: System Off: operation not handled. PANIC at PC : 0x0000000000459b9c
On the host, push the install.bfb through the RShim boot device for the BlueField to boot up running the Yocto mini system entirely on memory:
[root@bu-lab02 ~]# cat /root/BlueField-<version>/sample/install.bfb > /dev/rshim0/boot
Once the mini-Yocto has finished booting, bring up the interface which is selected to copy over the eMMC image to the host. Any working network interface can be used, in this example the representor interface is used as it offers a faster transfer speed (using the RShim network interface is also a good option).
On the BlueField side:
root@bluefield:~# ifconfig rep0-0 192.168.200.2 up
On the host side:
[root@bu-lab02 ~]# ifconfig p4p1 192.168.200.1/24 up [root@bu-lab02 ~]# ping 192.168.200.2 PING 192.168.200.2 (192.168.200.2) 56(84) bytes of data. 64 bytes from 192.168.200.2: icmp_seq=1 ttl=64 time=0.281 ms 64 bytes from 192.168.200.2: icmp_seq=2 ttl=64 time=0.073 ms ^C --- 192.168.200.2 ping statistics --- 2 packets transmitted, 2 received, 0% packet loss, time 999ms rtt min/avg/max/mdev = 0.073/0.177/0.281/0.104 ms
Check if netcat is working properly. This is the tool that is used to pipe data across networks.
Set up a netcat server on the host to listen to port 12345, and let it send the message “Hello from host” to the client:
[root@bu-lab02 ~]# echo "Hello from host" | nc -l 12345
On BlueField, send the message “Hello from BlueField” to the server:
root@bluefield:~# echo "Hello from BlueField" | nc 192.168.200.1 12345 Hello from host
On the host, the nc command completes and prints out “Hello from BlueField”:
[root@bu-lab02 ~]# echo "Hello from host" | nc -l 12345 Hello from BlueField
This may fail since the iptables forbid listening to the port. If that happens, flush the rules by running:
iptables -F
Forcing nc to use IPv4 addresses might resolve the issue:
[root@bu-lab02 ~]# nc -4 -l 12345
Back up the eMMC image from the BlueField to the host. Set up the host to listen on port 12345, compress what it receives and store it to a file:
[root@bu-lab02 ~]# nc -l 12345 | pv | gzip -1 > /backup.dd.gz
NoteThe “pv” command is optional. It is used to monitor the progress of the backup. The backup should finish when the total data consumed is 13.8G, which is approximately 6 minutes if using the representor port.
On BlueField, read the entire eMMC boot partition with the “dd” command and pass it to the host:
root@bluefield:~# dd if=/dev/mmcblk0 bs=64M | nc 192.168.200.1 12345
If the “pv” command is used, it will start showing the transfer speed and data:
[root@bu-lab02 ~]# nc -l 12345 | pv | gzip -1 > / backup.dd.gz 7.69GiB 0:04:44 [64.4MiB/s] [ <=> ]
Once this is complete, the generated backip.dd.gz file on the host is the compressed eMMC image of the BlueField system.
NoteNote that the backup does not include the eMMC boot partitions, as they are actually physically separate partitions on the eMMC device.
If nc is not usable for any reason, the same thing can be accomplished via the “ssh” command. It will take longer due to the encryption/decryption overhead of SSH. On the host, to get the same results, run:
ssh 192.168.200.2 "dd if=/dev/mmcblk0 bs=64M" | pv | gzip -1 > /backup.dd.gz
To restore the eMMC, the BlueField system cannot be using the eMMC when recovering it, thus the mini Yocto running entirely on memory is the solution.
Push the install.bfb for it to boot from memory and set up the network interface that is going to be used. This step is the same as when backing up the eMMC image. Instead of transferring the image from the BlueField to the host, it is done the other way around.
Set up the host to extract the image and set up a netcat server to send the image:
[root@bu-lab02 ~]# zcat /backup.dd.gz | pv | nc -l 12345
On the BlueField side, retrieve the image using netcat and write it to the eMMC:
root@bluefield:~# nc 192.168.200.1 12345 | dd of=/dev/mmcblk0 bs=64M
This can also be done with SSH. To have the same effect, on the host, run:
zcat /backup.dd.gz | pv | ssh 192.168.200.2 dd of=/dev/mmcblk0 bs=64M
When this is complete, the UEFI persistent variable needs to be set up so that UEFI knows where to boot grub from. This can be done using the efibootmgr tool included in the mini Yocto:
root@bluefield:~# mount -t efivarfs none /sys/firmware/efi/efivars root@bluefield:~# efibootmgr -c -d /dev/mmcblk0 -p 1 -l "\EFI\centos\grubaa64.efi" -L "CentOS 7.4"
Alternatively, if this is not done, at boot time UEFI would stop at the boot menu and you would have to go to the UEFI console and use the UEFI console bcfg command to achieve the same affect:
Shell> bcfg boot add 0 FS0:\EFI\centos\shim.efi "CentOS 7.4"
Exit the shell and select “continue booting”, and UEFI resumes the next stage of the booting process.
As mentioned before, this does not update the eMMC boot partitions. Therefore, if this process is used to deploy a new DPU, the boot partitions should also be updated by running the bfrec script from within the mini Yocto:
root@bluefield:~# /opt/mlnx/scripts/bfrec
In addition, if OFED is already installed on the restored system, update the firmware of the BlueField to the matching one. This can be done when entering into the restored image via:
[root@localhost ~]# /opt/mellanox/mlnx-fw-updater/firmware/mlxfwmanager_sriov_dis_aarch64