Installing Tools on Grace Hopper MGX System#
This chapter describes how to manually install the required kernel, driver, and tools on the host. This is a one-time installation and can be skipped if the system has been configured already.
In the following sequence of steps, the target host is Supermicro Grace Hopper MGX System.
Depending on the release, tools that are installed in this section may need to be upgraded in the Installing and Upgrading cuBB section.
After everything is installed and updated, refer to the cuBB Quick Start Guide for how to use cuBB.
Supermicro Grace Hopper MGX Configuration#
Supermicro Server SKU: ARS-111GL-NHR (Config 2)
Top View:
Back View:
Cable Connection#
Host OS Internet Connection#
The BF3 NICs are reserved for fronthaul and backhaul connections; a USB-to-Ethernet dongle connected to the back USB port is recommended for the host OS internet connection.
E2E Test Connection#
To run an end-to-end test with an O-RU, BF3 fronthaul port #0 or port #1 must be connected to the fronthaul switch. Make sure PTP is configured to use the port connected to the fronthaul switch. The following diagram shows a typical E2E connection in O-RAN LLS-C3 topology.
cuBB Test Connection#
To run a cuBB end-to-end test with TestMAC and an RU emulator, an R750 RU emulator is recommended for use with the Grace Hopper MGX system. The BF3 NIC (P/N: 900-9D3B6-00CV-AA0) should be installed in slot 7 of the R750 server, as shown in the following picture.
To provision the R750 RU emulator, follow the instructions at Installing Tools on Dell R750. Because the R750 RU emulator has no GPU, the Installing CUDA Driver can be skipped. Note that the PCI addresses of the BF3 ports are ca:00.0 and ca:00.1 on the R750 RU emulator.
$ lshw -c network -businfo
Bus info Device Class Description
==========================================================
pci@0000:04:00.0 eno8303 network NetXtreme BCM5720 Gigabit Etherne
pci@0000:04:00.1 eno8403 network NetXtreme BCM5720 Gigabit Etherne
pci@0000:ca:00.0 aerial00 network MT43244 BlueField-3 integrated Co
pci@0000:ca:00.1 aerial01 network MT43244 BlueField-3 integrated Co
The Mellanox 200GbE direct attach copper cable is required to connect the Grace Hopper MGX and R750 RU emulator to run more than 10 cells. The 100GbE direct attach copper cable should be able to support 10C 59c BFP9 but it is not going to work for 20C 60c BFP9.
To run the RU emulator on R750 + BF3, update the RU emulator YAML as follows:
sed -i "s/ul_core_list.*/ul_core_list: [5,7,9,11,13,15,17,19,21,23,25,27,29,31,33,35,37,39,41,43]/" $RU_YAML
sed -i "s/dl_core_list.*/dl_core_list: [4,6,8,10,12,14,16,18,20,22,24,26,28,30,32,34,36,38,40,42]/" $RU_YAML
sed -i "s/aerial_fh_split_rx_tx_mempool.*/aerial_fh_split_rx_tx_mempool: 1/" $RU_YAML
sed -i "s/low_priority_core.*/low_priority_core: 45/" $RU_YAML
System Firmware Upgrade#
During the first boot, log in to the BMC to check the firmware inventory. Go to Dashboard -> Maintenance -> Firmware Management -> Inventory to see the current firmware versions.
Here is the list of the minimum required versions. Upgrade the firmware to the following or newer versions, if your system has older firmware.
Component |
Firmware Version |
Firmware filename |
|---|---|---|
BMC |
1.05.04 |
BMC_SCMAST2600-ROT20-2501MS_20250814_01.05.04_STDsp.bin |
BIOS |
2.1a |
BIOS_G1SMH-G-1D31-SYS0001_20250527_2.1a_STDsp.bin |
FPGA |
F2.00.9A |
FPGA_MBD-G1SMH-G-10XX1D31_20240820_F2.00.9A_STDsp.bin |
VBIOS |
96.00.8D.00.03 |
g530_0206_888__96008D0003-prod.fwpkg |
EROT |
01.03.0252.0000_n01 |
cec1736-ecfw-01.03.0252.0000-n01-rel-prod.fwpkg |
CPLD Motherboard Misc |
0F |
CPLD_XO3-GP03E0-10XX03E0_20240823_0F.XX.XX_STDsp.jed |
The recommended firmware update sequence is:
Power off host
Update BMC
Update CPLD motherboard misc
Update CPU ERoT
Update FPGA
A/C power cycle
Update BIOS
Update VBIOS
Reboot or Power cycle
To update the firmware for a specific component, go to Dashboard -> Maintenance -> Firmware Management -> Update then select the component icon -> Next -> Select File -> Upload -> Update. For example, select BMC and its firmware file as follows:
For a non-BMC firmware update, the update is queued in the task list and applied on the next boot.
Install Ubuntu 24.04 Server#
Download the Ubuntu server 24.04 ISO image for ARM-based systems from https://ubuntu.com/download/server/arm. Before installing the system OS, prepare a bootable USB drive that contains the OS image or configure the virtual media in the BMC for remote installation. Also verify that a USB-to-Ethernet dongle is connected to the back USB port for host internet access.
There are two ways to configure the virtual media. One is to share the OS ISO image by Windows network sharing or Samba sharing on Linux. Then go to BMC Dashboard -> Configuration -> Virtual Media to enter the virtual media connection information, including the share host IP address, image path, username, and password. After the connection info is saved, click the Link icon to connect.
Another way to configure virtual media is to select the Virtual Media icon from the remote console and then mount the OS ISO image to the virtual CD/DVD drive.
Reboot the system after the virtual media is configured and connected. Press F11 to enter the BIOS boot menu and select UEFI: USB CD/DVD Drive to boot with the virtual media.
Follow the Ubuntu installation process with the notable selections below:
Continue in rich mode
Continue without updating
Ubuntu Server
Install OpenSSH server
When the installation is done, the console shows Install complete and Reboot now. Reboot the system and check the following:
Check if the system time is correct to avoid apt update error.
Run the following commands to set the date and time via NTP once (this will not enable the NTP service):
$ sudo apt-get install ntpdate
$ sudo ntpdate -s pool.ntp.org
Check if the GPU and NIC are detected by the OS.
Use the following commands to determine whether the GPU and NIC are detected by the OS:
$ lspci | grep -i nvidia
# GH200 GPU
0009:01:00.0 3D controller: NVIDIA Corporation Device 2342 (rev a1)
$ lspci | grep -i mellanox
# The first BF3 NIC (Fronthaul NIC)
0000:01:00.0 Ethernet controller: Mellanox Technologies MT43244 BlueField-3 integrated ConnectX-7 network controller (rev 01)
0000:01:00.1 Ethernet controller: Mellanox Technologies MT43244 BlueField-3 integrated ConnectX-7 network controller (rev 01)
0000:01:00.2 DMA controller: Mellanox Technologies MT43244 BlueField-3 SoC Management Interface (rev 01)
# The second BF3 NIC (Backhaul NIC)
0002:01:00.0 Ethernet controller: Mellanox Technologies MT43244 BlueField-3 integrated ConnectX-7 network controller (rev 01)
0002:01:00.1 Ethernet controller: Mellanox Technologies MT43244 BlueField-3 integrated ConnectX-7 network controller (rev 01)
0002:01:00.2 DMA controller: Mellanox Technologies MT43244 BlueField-3 SoC Management Interface (rev 01)
Use the following command to change the hostname:
$ sudo hostnamectl set-hostname NEW_HOSTNAME
To display the GRUB menu during boot, create /etc/default/grub.d/menu.cfg with the following content:
Automated Host Provisioning (Alpha)#
After completing the platform prerequisites above, installing the operating system, and configuring management-network access, you can follow the Aerial CUDA-Accelerated RAN (ACAR) cubb_scripts/install/README.md to automate the supported host software installation steps described in the remaining sections.
Note this automated host installer is an Alpha feature. If it does not support your system or does not complete successfully, follow the manual installation steps below.
Configure the Network Interfaces#
The following installation steps need an internet connection. Ensure that you have the proper netplan config for your local network.
The network interface names could change after reboot. To ensure persistent network interface
names after reboot, create persistent network link files under /etc/systemd/network, one for each interface.
To find the MAC address of the BlueField-3 NIC, run lshw to check for network devices and look for the ConnectX-7 entries.
$ sudo apt-get install jq -y
$ sudo lshw -json -C network | jq '.[] | "\(.product), MAC: \(.serial)"' | grep "ConnectX-7"
"MT43244 BlueField-3 integrated ConnectX-7 network controller, MAC: 94:6d:ae:ww:ww:ww"
"MT43244 BlueField-3 integrated ConnectX-7 network controller, MAC: 94:6d:ae:xx:xx:xx"
"MT43244 BlueField-3 integrated ConnectX-7 network controller, MAC: 94:6d:ae:yy:yy:yy"
"MT43244 BlueField-3 integrated ConnectX-7 network controller, MAC: 94:6d:ae:zz:zz:zz"
Create files under /etc/systemd/network/ with the desired interface name and the MAC address found in the previous step.
Note
The rest of this document assumes that the aerial00 and aerial01 interfaces are connected to the RU emulator for cuBB testing or to the fronthaul switch for E2E testing, and that aerial00 is used for PTP.
$ sudo nano /etc/systemd/network/20-aerial00.link
[Match]
MACAddress=94:6d:ae:ww:ww:ww
[Link]
Name=aerial00
$ sudo nano /etc/systemd/network/20-aerial01.link
[Match]
MACAddress=94:6d:ae:xx:xx:xx
[Link]
Name=aerial01
$ sudo nano /etc/systemd/network/20-aerial02.link
[Match]
MACAddress=94:6d:ae:yy:yy:yy
[Link]
Name=aerial02
$ sudo nano /etc/systemd/network/20-aerial03.link
[Match]
MACAddress=94:6d:ae:zz:zz:zz
[Link]
Name=aerial03
To apply the change:
$ sudo netplan apply
Disable Auto Upgrade#
Edit the /etc/apt/apt.conf.d/20auto-upgrades system file and change "1" to "0" on both lines.
This prevents the installed version of the low latency kernel from being accidentally changed with a
subsequent software upgrade.
$ sudo sed -i 's/"1"/"0"/g' /etc/apt/apt.conf.d/20auto-upgrades
Disable the fwupd-refresh timer to prevent fwupdmgr from automatically checking for any updates.
$ sudo systemctl mask fwupd-refresh.timer
Disable Nouveau#
Disable the Nouveau open-source GPU driver before installing the NVIDIA driver:
Regenerate the kernel initramfs:
sudo update-initramfs -u
Note
The Nouveau blacklist takes effect after reboot. Complete the remaining kernel configuration steps before rebooting.
Install NVIDIA Optimized Ubuntu Kernel#
Run the following commands to install the NVIDIA optimized Ubuntu kernel.
$ sudo apt update
# NOTE: This will install the specific kernel version, not the latest NVIDIA optimized kernel.
$ sudo apt install -y linux-image-6.17.0-1018-nvidia-64k
Then, update the GRUB to change the default boot kernel. The version to use here depends on the latest version that was installed with the previous command:
# Update grub to change the default boot kernel
$ sudo sed -i 's/^GRUB_DEFAULT=.*/GRUB_DEFAULT="Advanced options for Ubuntu>Ubuntu, with Linux 6.17.0-1018-nvidia-64k"/' /etc/default/grub
Configure Linux Kernel Command-line#
Ensure the iommu.passthrough=y kernel parameter is NOT passed to the kernel. This parameter prevents the GPU driver from loading, so it must be removed if it is present.
Verify that the parameter is present by running:
$ grep iommu.passthrough=y /proc/cmdline
If the parameter is present, find the file that contains this parameter and remove it. For example:
# Remove iommu.passthrough=y from the found file
$ grep -rlZ 'iommu.passthrough=y' /etc/default/grub* | xargs -0 --no-run-if-empty sudo sed -i 's/ iommu.passthrough=y//'
To set kernel command-line parameters, edit the GRUB_CMDLINE_LINUX parameter in the grub
file /etc/default/grub.d/cmdline.cfg and append or update the parameters described below. The following kernel parameters
are optimized for GH200. To automatically append the grub file with these parameters, enter this command:
Note
The hugepage size is 512MB which is optimized for the 64k page size kernel on ARM.
Apply the Changes and Reboot to Load the Kernel#
$ sudo update-grub
$ sudo reboot
After rebooting, enter the following command to verify that the kernel command-line parameters are configured properly:
$ uname -r
6.17.0-1018-nvidia-64k
$ cat /proc/cmdline
BOOT_IMAGE=/vmlinuz-6.17.0-1018-nvidia-64k root=/dev/mapper/ubuntu--vg-ubuntu--lv ro pci=realloc=off pci=pcie_bus_safe default_hugepagesz=512M hugepagesz=512M hugepages=48 tsc=reliable processor.max_cstate=0 audit=0 idle=poll rcu_nocb_poll nosoftlockup irqaffinity=0-3 isolcpus=managed_irq,domain,4-64 nohz_full=4-64 rcu_nocbs=4-64 earlycon module_blacklist=nouveau acpi_power_meter.force_cap_on=y numa_balancing=disable init_on_alloc=0 preempt=none
Enter this command to verify if hugepages are enabled:
$ grep -i huge /proc/meminfo
AnonHugePages: 0 kB
ShmemHugePages: 0 kB
FileHugePages: 0 kB
HugePages_Total: 48
HugePages_Free: 48
HugePages_Rsvd: 0
HugePages_Surp: 0
Hugepagesize: 524288 kB
Hugetlb: 25165824 kB
Install Dependency Packages#
Enter these commands to install the prerequisite packages:
$ sudo apt-get update
$ sudo apt-get install -y build-essential linux-headers-$(uname -r) dkms unzip ipmitool linuxptp pv apt-utils net-tools
Install DOCA OFED and Mellanox Firmware Tools on the Host#
Check if there is an existing MOFED installed on the host system.
$ ofed_info -s
OFED-internal-25.10-1.7.1:
Uninstall MOFED if it is present, following the DOCA Installation Guide for Linux.
$ for f in $( dpkg --list | grep -E 'doca|flexio|dpa-gdbserver|dpa-stats|dpaeumgmt' | awk '{print $2}' ); do echo $f ; sudo apt remove --purge $f -y ; done
$ sudo /usr/sbin/ofed_uninstall.sh --force
$ sudo apt-get autoremove
Enter the following commands to install DOCA OFED.
# Install DOCA OFED
$ wget https://www.mellanox.com/downloads/DOCA/DOCA_v3.3.0/host/doca-host_3.3.0-088000-26.01-ubuntu2404_arm64.deb
$ sudo dpkg -i doca-host_3.3.0-088000-26.01-ubuntu2404_arm64.deb
$ sudo apt update
$ sudo apt install -y doca-tools doca-ofed mlnx-fw-updater
$ sudo systemctl restart rshim
# The mlnx-fw-updater should update the NIC FW automatically if it detects the newer version is available.
Attempting to perform Firmware update...
Querying Mellanox devices firmware ...
# To check what version of OFED you have installed
$ ofed_info -s
OFED-internal-26.01-1.0.0:
The Mellanox firmware tools are already installed through the DOCA-HOST package. Run the following command to verify the installation.
$ apt list --installed | grep mft
kernel-mft-dkms/DOCA-HOST-3.3.0,now 4.35.0.159-1 all [installed,automatic]
mft-mlx5/DOCA-HOST-3.3.0,now 4.35.0-159 arm64 [installed,automatic]
mft-nvredfish/DOCA-HOST-3.3.0,now 4.35.0-159 arm64 [installed,automatic]
mft/DOCA-HOST-3.3.0,now 4.35.0-159 arm64 [installed,automatic]
$ sudo mst version
mst, mft 4.35.0-159. Git SHA Hash: 004525846
$ sudo mst start
# check NIC PCIe bus addresses and network interface names
$ sudo mst status -v
MST modules:
------------
MST PCI module is not loaded
MST PCI configuration module loaded
PCI devices:
------------
DEVICE_TYPE MST PCI RDMA NET NUMA
BlueField3(rev:1) /dev/mst/mt41692_pciconf0 0000:01:00.0 mlx5_0 net-aerial00 0
BlueField3(rev:1) /dev/mst/mt41692_pciconf0.1 0000:01:00.1 mlx5_1 net-aerial01 0
BlueField3(rev:1) /dev/mst/mt41692_pciconf1 0002:01:00.0 mlx5_2 net-aerial02 0
BlueField3(rev:1) /dev/mst/mt41692_pciconf1.1 0002:01:00.1 mlx5_3 net-aerial03 0
Enter these commands to check the link status of port 0:
# Here is an example if the port 0 of fronthaul NIC is connected to another server or switch via a 200GbE DAC cable.
$ sudo mlxlink -d /dev/mst/mt41692_pciconf0
Operational Info
----------------
State : Active
Physical state : LinkUp
Speed : 200G
Width : 4x
FEC : Standard_RS-FEC - (544,514)
Loopback Mode : No Loopback
Auto Negotiation : ON
Supported Info
--------------
Enabled Link Speed (Ext.) : 0x00003ff2 (200G_2X,200G_4X,100G_1X,100G_2X,100G_4X,50G_1X,50G_2X,40G,25G,10G,1G)
Supported Cable Speed (Ext.) : 0x000017f2 (200G_4X,100G_2X,100G_4X,50G_1X,50G_2X,40G,25G,10G,1G)
Troubleshooting Info
--------------------
Status Opcode : 0
Group Opcode : N/A
Recommendation : No issue was observed
Tool Information
----------------
Firmware Version : 32.47.1088
amBER Version : 6.4
MFT Version : 4.35.0-159
Follow the instructions at Configure BF3 NIC to configure the BF3 NIC.
Install CUDA Driver#
If the system has an older driver installed, unload the current driver modules and uninstall the old driver, using the following:
# Unload the current driver modules
$ for m in $(lsmod | awk "/^[^[:space:]]*(nvidia|nv_|gdrdrv)/ {print \$1}"); do echo Unload $m...; sudo rmmod $m; done
# Remove the driver if it was installed by runfile installer before.
$ sudo /usr/bin/nvidia-uninstall
Create the driver module config with the following recommended settings:
Run the following commands to install the NVIDIA open-source GPU kernel driver (OpenRM).
# Install NVIDIA GPU driver
$ wget https://us.download.nvidia.com/tesla/610.57.04/NVIDIA-Linux-aarch64-610.57.04.run
$ sudo sh NVIDIA-Linux-aarch64-610.57.04.run --silent -m kernel-open
# Verify that the driver is loaded successfully
$ nvidia-smi
+-----------------------------------------------------------------------------------------+
| NVIDIA-SMI 610.57.04 KMD Version: 610.57.04 CUDA UMD Version: 13.3 |
+-----------------------------------------+------------------------+----------------------+
| GPU Name Persistence-M | Bus-Id Disp.A | Volatile Uncorr. ECC |
| Fan Temp Perf Pwr:Usage/Cap | Memory-Usage | GPU-Util Compute M. |
| | | MIG M. |
|=========================================+========================+======================|
| 0 NVIDIA GH200 480GB On | 00000009:01:00.0 Off | 0 |
| N/A 32C P0 119W / 900W | 0MiB / 97871MiB | 0% Default |
| | | Disabled |
+-----------------------------------------+------------------------+----------------------+
+-----------------------------------------------------------------------------------------+
| Processes: |
| GPU GI CI PID Type Process name GPU Memory |
| ID ID Usage |
|=========================================================================================|
| No running processes found |
+-----------------------------------------------------------------------------------------+
Install GDRCopy Driver#
Run the following commands to install the GDRCopy driver. If the system has an older version installed, remove the old driver first.
Warning
GDRCopy driver must be installed after the CUDA driver.
# Check the installed GDRCopy driver version
$ apt list --installed | grep gdrdrv-dkms
# Remove the driver, if you have the older version installed.
$ sudo apt purge gdrdrv-dkms
$ sudo apt autoremove
# Install GDRCopy driver
$ wget https://developer.download.nvidia.com/compute/redist/gdrcopy/CUDA%2013.0/ubuntu24_04/aarch64/gdrdrv-dkms_2.6-1_arm64.Ubuntu24_04.deb
$ sudo dpkg -i gdrdrv-dkms_2.6-1_arm64.Ubuntu24_04.deb
Install Docker CE#
The full official instructions for installing Docker CE can be found here: https://docs.docker.com/engine/install/ubuntu/#install-docker-engine. The following instructions are one supported way of installing Docker CE:
$ sudo apt-get update
$ sudo apt-get install -y ca-certificates curl gnupg
$ sudo install -m 0755 -d /etc/apt/keyrings
$ curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo gpg --dearmor -o /etc/apt/keyrings/docker.gpg
$ sudo chmod a+r /etc/apt/keyrings/docker.gpg
$ echo "deb [arch=$(dpkg --print-architecture) signed-by=/etc/apt/keyrings/docker.gpg] https://download.docker.com/linux/ubuntu $(. /etc/os-release && echo "$VERSION_CODENAME") stable" | sudo tee /etc/apt/sources.list.d/docker.list > /dev/null
$ sudo apt-get update
$ sudo apt-get install -y docker-ce docker-ce-cli containerd.io docker-buildx-plugin docker-compose-plugin
$ sudo docker run --rm hello-world
Install the NVIDIA Container Toolkit#
Locate and follow the NVIDIA Container Toolkit installation instructions.
Alternatively, use the following instructions to install the NVIDIA Container Toolkit.
Warning
Install the NVIDIA GPU driver (CUDA driver) and verify it with nvidia-smi before installing the NVIDIA Container Toolkit.
$ curl -fsSL https://nvidia.github.io/libnvidia-container/gpgkey | sudo gpg --dearmor -o /usr/share/keyrings/nvidia-container-toolkit-keyring.gpg
$ curl -s -L https://nvidia.github.io/libnvidia-container/stable/deb/nvidia-container-toolkit.list | sed 's#deb https://#deb [signed-by=/usr/share/keyrings/nvidia-container-toolkit-keyring.gpg] https://#g' | sudo tee /etc/apt/sources.list.d/nvidia-container-toolkit.list
$ sudo apt-get update
$ sudo apt-get install -y nvidia-container-toolkit
$ sudo nvidia-ctk runtime configure --runtime=docker
$ sudo systemctl restart docker
$ sudo docker run --rm --runtime=nvidia --gpus all ubuntu nvidia-smi
Note
If you have nvidia-container-toolkit installed on the existing system, check the version by running the nvidia-ctk --version
command. If it is older than 1.17.4, run the following commands to upgrade to the current version:
$ nvidia-ctk --version
NVIDIA Container Toolkit CLI version 1.14.4
commit: d167812ce3a55ec04ae2582eff1654ec812f42e1
$ sudo apt update
$ sudo apt-get install -y nvidia-container-toolkit
$ nvidia-ctk --version
NVIDIA Container Toolkit CLI version 1.19.1
commit: 09ceee5dde66ba9ce25c7cc69b1ebd5e6e3266fa
Update BF3 BFB Image and NIC Firmware#
Note
The following instructions are for BF3 NIC (OPN: 900-9D3B6-00CV-A; PSID: MT_0000000884) specifically.
There is no need to switch to DPU mode if using the BFB image below.
This BFB image will update the NIC firmware automatically.
Check whether the RShim service is running by using
sudo systemctl status rshim. If it is not, restart the service by usingsudo systemctl restart rshim.
# Enable MST
$ sudo mst start
$ sudo mst status
MST modules:
------------
MST PCI module is not loaded
MST PCI configuration module loaded
MST devices:
------------
/dev/mst/mt41692_pciconf0 - PCI configuration cycles access.
domain:bus:dev.fn=0000:01:00.0 addr.reg=88 data.reg=92 cr_bar.gw_offset=-1
Chip revision is: 01
/dev/mst/mt41692_pciconf1 - PCI configuration cycles access.
domain:bus:dev.fn=0002:01:00.0 addr.reg=88 data.reg=92 cr_bar.gw_offset=-1
Chip revision is: 01
# Download the BF3 BFB image
$ wget https://content.mellanox.com/BlueField/FW-Bundle/bf-fwbundle-3.3.0-202_26.01-prod.bfb
# Update the BFB image of the 1st BF3
$ sudo bfb-install -r rshim0 -b bf-fwbundle-3.3.0-202_26.01-prod.bfb
# Update the BFB image of the 2nd BF3
$ sudo bfb-install -r rshim1 -b bf-fwbundle-3.3.0-202_26.01-prod.bfb
Pushing bfb
Collecting BlueField booting status. Press Ctrl+C to stop…
INFO[PSC]: PSC BL1 START
INFO[BL2]: start
INFO[BL2]: boot mode (rshim)
INFO[BL2]: VDD_CPU: 870 mV
INFO[BL2]: VDDQ: 1120 mV
INFO[BL2]: DDR POST passed
INFO[BL2]: UEFI loaded
INFO[BL31]: start
INFO[BL31]: lifecycle GA Secured
INFO[BL31]: runtime
INFO[BL31]: MB ping success
INFO[UEFI]: Partial NIC
INFO[UEFI]: eMMC init
INFO[UEFI]: eMMC probed
INFO[UEFI]: UPVS valid
INFO[UEFI]: PMI: updates started
INFO[UEFI]: PMI: total updates: 1
INFO[UEFI]: PMI: updates completed, status 0
INFO[UEFI]: PCIe enum start
INFO[UEFI]: PCIe enum end
INFO[BL31]: Partial NIC
INFO[BL31]: power capping disabled
INFO[UEFI]: UEFI Secure Boot (disabled)
INFO[UEFI]: PK configured
INFO[UEFI]: Redfish enabled
INFO[UEFI]: exit Boot Service
INFO[MISC]: Erasing eMMC drive: /dev/mmcblk0
INFO[MISC]: Erasing NVME drive: /dev/nvme0n1
INFO[MISC]: Ubuntu installation started
INFO[MISC]: Installing OS image
INFO[MISC]: Ubuntu installation completed
INFO[MISC]: Updating NIC firmware...
INFO[MISC]: NIC firmware update done: 32.48.1080
INFO[MISC]: Installation finished
# Wait 10 minutes to ensure the card initializes properly after the BFB installation
$ sleep 600
# NOTE: Requires a full power cycle from host with cold boot
$ sudo ipmitool power cycle
# Verify NIC FW version after reboot
$ sudo mst start
$ sudo flint -d /dev/mst/mt41692_pciconf0 q
Image type: FS4
FW Version: 32.48.1000
FW Release Date: 11.2.2026
Product Version: 32.48.1000
Rom Info: type=UEFI Virtio net version=21.4.13 cpu=AMD64,AARCH64
type=UEFI Virtio blk version=22.4.14 cpu=AMD64,AARCH64
type=UEFI version=14.41.14 cpu=AMD64,AARCH64
type=PXE version=3.9.101 cpu=AMD64
Description: UID GuidsNumber
Base GUID: 9c63c00300e56d14 38
Base MAC: 9c63c0e56d14 38
Image VSD: N/A
Device VSD: N/A
PSID: MT_0000000884
Security Attributes: secure-fw
Run the following commands to configure the BF3 NIC:
# Setting BF3 port to Ethernet mode (not Infiniband)
$ sudo mlxconfig -d /dev/mst/mt41692_pciconf0 --yes set LINK_TYPE_P1=2
$ sudo mlxconfig -d /dev/mst/mt41692_pciconf0 --yes set LINK_TYPE_P2=2
$ sudo mlxconfig -d /dev/mst/mt41692_pciconf0 --yes set INTERNAL_CPU_MODEL=1
$ sudo mlxconfig -d /dev/mst/mt41692_pciconf0 --yes set INTERNAL_CPU_PAGE_SUPPLIER=EXT_HOST_PF
$ sudo mlxconfig -d /dev/mst/mt41692_pciconf0 --yes set INTERNAL_CPU_ESWITCH_MANAGER=EXT_HOST_PF
$ sudo mlxconfig -d /dev/mst/mt41692_pciconf0 --yes set INTERNAL_CPU_IB_VPORT0=EXT_HOST_PF
$ sudo mlxconfig -d /dev/mst/mt41692_pciconf0 --yes set INTERNAL_CPU_OFFLOAD_ENGINE=DISABLED
$ sudo mlxconfig -d /dev/mst/mt41692_pciconf0 --yes set CQE_COMPRESSION=1
$ sudo mlxconfig -d /dev/mst/mt41692_pciconf0 --yes set PROG_PARSE_GRAPH=1
$ sudo mlxconfig -d /dev/mst/mt41692_pciconf0 --yes set ACCURATE_TX_SCHEDULER=1
$ sudo mlxconfig -d /dev/mst/mt41692_pciconf0 --yes set FLEX_PARSER_PROFILE_ENABLE=4
$ sudo mlxconfig -d /dev/mst/mt41692_pciconf0 --yes set REAL_TIME_CLOCK_ENABLE=1
$ sudo mlxconfig -d /dev/mst/mt41692_pciconf0 --yes set EXP_ROM_VIRTIO_NET_PXE_ENABLE=0
$ sudo mlxconfig -d /dev/mst/mt41692_pciconf0 --yes set EXP_ROM_VIRTIO_NET_UEFI_ARM_ENABLE=0
$ sudo mlxconfig -d /dev/mst/mt41692_pciconf0 --yes set EXP_ROM_VIRTIO_NET_UEFI_x86_ENABLE=0
$ sudo mlxconfig -d /dev/mst/mt41692_pciconf0 --yes set EXP_ROM_VIRTIO_BLK_UEFI_ARM_ENABLE=0
$ sudo mlxconfig -d /dev/mst/mt41692_pciconf0 --yes set EXP_ROM_VIRTIO_BLK_UEFI_x86_ENABLE=0
# NOTE: Requires a full power cycle from host with cold boot
$ sudo ipmitool power cycle
#Enable MST
$ sudo mst start
# Verify that the NIC FW changes have been applied
$ sudo mlxconfig -d /dev/mst/mt41692_pciconf0 q | grep "CQE_COMPRESSION\|PROG_PARSE_GRAPH\|ACCURATE_TX_SCHEDULER\|FLEX_PARSER_PROFILE_ENABLE\|REAL_TIME_CLOCK_ENABLE\|INTERNAL_CPU_MODEL\|LINK_TYPE_P1\|LINK_TYPE_P2\|INTERNAL_CPU_PAGE_SUPPLIER\|INTERNAL_CPU_ESWITCH_MANAGER\|INTERNAL_CPU_IB_VPORT0\|INTERNAL_CPU_OFFLOAD_ENGINE"
INTERNAL_CPU_MODEL EMBEDDED_CPU(1)
INTERNAL_CPU_PAGE_SUPPLIER EXT_HOST_PF(1)
INTERNAL_CPU_ESWITCH_MANAGER EXT_HOST_PF(1)
INTERNAL_CPU_IB_VPORT0 EXT_HOST_PF(1)
INTERNAL_CPU_OFFLOAD_ENGINE DISABLED(1)
FLEX_PARSER_PROFILE_ENABLE 4
PROG_PARSE_GRAPH True(1)
ACCURATE_TX_SCHEDULER True(1)
CQE_COMPRESSION AGGRESSIVE(1)
REAL_TIME_CLOCK_ENABLE True(1)
LINK_TYPE_P1 ETH(2)
LINK_TYPE_P2 ETH(2)
Install ptp4l and phc2sys#
Versions of ptp4l prior to 4.0 do not support dual-port PTP. Run the following commands to install ptp4l 4.2 from source:
$ sudo apt remove linuxptp
$ wget https://github.com/richardcochran/linuxptp/archive/refs/tags/v4.2.tar.gz
$ tar -xzf v4.2.tar.gz
$ cd linuxptp-4.2/
$ make
$ sudo make install prefix=/usr sbindir=/usr/sbin
Enter the following commands to configure ptp4l. This example assumes that aerial00 is the
NIC interface used for PTP.
Create /etc/ptp.conf:
Create /etc/systemd/system/ptp4l.service:
Reload systemd and start the service:
Verify that the PTP service is active and that one server has become the master clock, as shown below:
$ sudo systemctl status ptp4l.service
● ptp4l.service - Precision Time Protocol (PTP) service
Loaded: loaded (/etc/systemd/system/ptp4l.service; enabled; vendor preset: enabled)
Active: active (running) since Fri 2024-08-30 01:25:57 UTC; 2min 16s ago
Docs: man:ptp4l
Main PID: 3404 (ptp4l)
Tasks: 1 (limit: 598789)
Memory: 2.6M
CPU: 126ms
CGroup: /system.slice/ptp4l.service
└─3404 /usr/sbin/ptp4l -f /etc/ptp.conf
Aug 30 01:25:57 r750-01 ptp4l[3404]: [14.291] port 0: INITIALIZING to LISTENING on INIT_COMPLETE
Aug 30 01:25:57 r750-01 ptp4l[3404]: [14.291] port 1: link down
Aug 30 01:25:57 r750-01 ptp4l[3404]: [14.291] port 1: LISTENING to FAULTY on FAULT_DETECTED (FT_UNSPECIFIED)
Aug 30 01:25:57 r750-01 ptp4l[3404]: [14.323] selected local clock a088c2.fffe.47be40 as best master
Aug 30 01:25:57 r750-01 ptp4l[3404]: [14.323] port 1: assuming the grand master role
Aug 30 01:26:56 r750-01 ptp4l[3404]: [73.338] port 1: link up
Aug 30 01:26:56 r750-01 ptp4l[3404]: [73.368] port 1: FAULTY to LISTENING on INIT_COMPLETE
Aug 30 01:26:57 r750-01 ptp4l[3404]: [73.860] port 1: LISTENING to MASTER on ANNOUNCE_RECEIPT_TIMEOUT_EXPIRES
Aug 30 01:26:57 r750-01 ptp4l[3404]: [73.860] selected local clock a088c2.fffe.47be40 as best master
Aug 30 01:26:57 r750-01 ptp4l[3404]: [73.860] port 1: assuming the grand master role
The other becomes the secondary (follower) clock, as shown below:
$ sudo systemctl status ptp4l.service
● ptp4l.service - Precision Time Protocol (PTP) service
Loaded: loaded (/etc/systemd/system/ptp4l.service; enabled; vendor preset: enabled)
Active: active (running) since Fri 2024-08-30 01:29:33 UTC; 47s ago
Docs: man:ptp4l
Process: 1509 ExecStartPre=ifconfig aerial00 up (code=exited, status=0/SUCCESS)
Process: 3069 ExecStartPre=ethtool --set-priv-flags aerial00 tx_port_ts on (code=exited, status=0/SUCCESS)
Process: 3755 ExecStartPre=ethtool -A aerial00 rx off tx off (code=exited, status=0/SUCCESS)
Process: 3822 ExecStartPre=ifconfig aerial01 up (code=exited, status=0/SUCCESS)
Process: 3827 ExecStartPre=ethtool --set-priv-flags aerial01 tx_port_ts on (code=exited, status=0/SUCCESS)
Process: 3862 ExecStartPre=ethtool -A aerial01 rx off tx off (code=exited, status=0/SUCCESS)
Main PID: 3870 (ptp4l)
Tasks: 1 (limit: 73247)
Memory: 9.2M
CPU: 183ms
CGroup: /system.slice/ptp4l.service
└─3870 /usr/sbin/ptp4l -f /etc/ptp.conf
Aug 30 01:30:12 aerial-mgx-cg1-01 ptp4l[3870]: [107.479] rms 3 max 6 freq +9551 +/- 12 delay -94 +/- 0
Aug 30 01:30:13 aerial-mgx-cg1-01 ptp4l[3870]: [108.479] rms 3 max 6 freq +9556 +/- 10 delay -94 +/- 0
Aug 30 01:30:14 aerial-mgx-cg1-01 ptp4l[3870]: [109.479] rms 3 max 4 freq +9552 +/- 13 delay -94 +/- 0
Aug 30 01:30:15 aerial-mgx-cg1-01 ptp4l[3870]: [110.479] rms 3 max 6 freq +9556 +/- 12 delay -94 +/- 1
Aug 30 01:30:16 aerial-mgx-cg1-01 ptp4l[3870]: [111.479] rms 3 max 7 freq +9558 +/- 14 delay -94 +/- 0
Aug 30 01:30:17 aerial-mgx-cg1-01 ptp4l[3870]: [112.479] rms 4 max 7 freq +9567 +/- 12 delay -94 +/- 0
Aug 30 01:30:18 aerial-mgx-cg1-01 ptp4l[3870]: [113.479] rms 3 max 5 freq +9569 +/- 7 delay -94 +/- 0
Aug 30 01:30:19 aerial-mgx-cg1-01 ptp4l[3870]: [114.479] rms 3 max 6 freq +9574 +/- 8 delay -94 +/- 1
Aug 30 01:30:20 aerial-mgx-cg1-01 ptp4l[3870]: [115.479] rms 3 max 5 freq +9577 +/- 9 delay -94 +/- 0
Aug 30 01:30:21 aerial-mgx-cg1-01 ptp4l[3870]: [116.479] rms 4 max 7 freq +9583 +/- 12 delay -94 +/- 0
Enter the commands to turn off NTP:
$ sudo timedatectl set-ntp false
$ timedatectl
Local time: Fri 2024-08-30 01:30:36 UTC
Universal time: Fri 2024-08-30 01:30:36 UTC
RTC time: Fri 2024-08-30 01:30:36
Time zone: Etc/UTC (UTC, +0000)
System clock synchronized: no
NTP service: inactive
RTC in local TZ: no
Run PHC2SYS as service:
PHC2SYS is used to synchronize the system clock to the PTP hardware clock (PHC) on the NIC.
Set the PTP NIC, such as aerial00, as the source clock and the system clock
(CLOCK_REALTIME) as the slave clock.
If more than one phc2sys instance is already running, stop the existing sessions before
updating the service file.
Create or update /etc/systemd/system/phc2sys.service with the following contents:
Note
PTP is based on TAI time and the system clock is synchronized to TAI time scale with the above PHC2SYS settings.
The current offset between UTC and TAI is 37 seconds (leap seconds) and TAI is ahead of UTC by this amount.
If there is a need to change the system clock to UTC time on DU, the first ExecStart with -O 0 should be commented out and the second ExecStart with -w should be uncommented assuming the PTP and GrandMaster are properly configured.
After the PHC2SYS config file is changed, run the following:
$ sudo systemctl daemon-reload
$ sudo systemctl restart phc2sys.service
# Set to start automatically on reboot
$ sudo systemctl enable phc2sys.service
# check that the service is active and has converged to a low rms value (<30) and that the correct NIC has been selected (aerial00):
$ sudo systemctl status phc2sys.service
● phc2sys.service - Synchronize system clock or PTP hardware clock (PHC)
Loaded: loaded (/etc/systemd/system/phc2sys.service; enabled; vendor preset: enabled)
Active: active (running) since Fri 2024-08-30 01:31:35 UTC; 18min ago
Docs: man:phc2sys
Process: 3871 ExecStartPre=sleep 2 (code=exited, status=0/SUCCESS)
Main PID: 4006 (sh)
Tasks: 2 (limit: 73247)
Memory: 6.0M
CPU: 3.628s
CGroup: /system.slice/phc2sys.service
├─4006 /bin/sh -c "/usr/sbin/phc2sys -s aerial00 -c CLOCK_REALTIME -n 24 -O 0 -R 256 -u 256"
└─4012 /usr/sbin/phc2sys -s /dev/ptp2 -c CLOCK_REALTIME -n 24 -O 0 -R 256 -u 256
Aug 30 01:48:09 aerial-mgx-c1-01 phc2sys[4012]: [1184.489] CLOCK_REALTIME rms 8 max 22 freq +5522 +/- 47 delay 480 +/- 0
Aug 30 01:48:10 aerial-mgx-c1-01 phc2sys[4012]: [1185.505] CLOCK_REALTIME rms 7 max 19 freq +5542 +/- 30 delay 480 +/- 2
Aug 30 01:48:11 aerial-mgx-c1-01 phc2sys[4012]: [1186.521] CLOCK_REALTIME rms 7 max 19 freq +5530 +/- 36 delay 480 +/- 0
Aug 30 01:48:12 aerial-mgx-c1-01 phc2sys[4012]: [1187.537] CLOCK_REALTIME rms 7 max 19 freq +5534 +/- 43 delay 480 +/- 2
Aug 30 01:48:13 aerial-mgx-c1-01 phc2sys[4012]: [1188.553] CLOCK_REALTIME rms 9 max 22 freq +5557 +/- 64 delay 480 +/- 0
Aug 30 01:48:14 aerial-mgx-c1-01 phc2sys[4012]: [1189.569] CLOCK_REALTIME rms 9 max 23 freq +5516 +/- 52 delay 480 +/- 0
Aug 30 01:48:15 aerial-mgx-c1-01 phc2sys[4012]: [1190.586] CLOCK_REALTIME rms 7 max 19 freq +5538 +/- 32 delay 480 +/- 0
Aug 30 01:48:16 aerial-mgx-c1-01 phc2sys[4012]: [1191.602] CLOCK_REALTIME rms 7 max 19 freq +5534 +/- 27 delay 480 +/- 0
Aug 30 01:48:17 aerial-mgx-c1-01 phc2sys[4012]: [1192.618] CLOCK_REALTIME rms 8 max 18 freq +5538 +/- 42 delay 480 +/- 0
Aug 30 01:48:18 aerial-mgx-c1-01 phc2sys[4012]: [1193.634] CLOCK_REALTIME rms 8 max 20 freq +5547 +/- 47 delay 480 +/- 0
Verify that the system clock is synchronized:
$ timedatectl
Local time: Fri 2024-08-30 01:48:25 UTC
Universal time: Fri 2024-08-30 01:48:25 UTC
RTC time: Fri 2024-08-30 01:48:25
Time zone: Etc/UTC (UTC, +0000)
System clock synchronized: yes
NTP service: inactive
RTC in local TZ: no
Dual-port PTP#
Dual-port PTP is supported when both FH ports (for example, aerial00 and aerial01) receive the PTP signal from a timing source, such as a Primary Reference Time Clock (PRTC) or Grandmaster. Install linuxptp 4.2 as described in the previous section.
When dual-port PTP is configured and PTP services are enabled, linuxptp selects one FH port as RS_SLAVE (preferred timing reference) and the other as RS_PASSIVE (alternate timing reference). If the RS_SLAVE port loses the PTP signal, the RS_PASSIVE port assumes the RS_SLAVE role. When the PTP signal is restored on the preferred timing reference, that port resumes the RS_SLAVE role.
The phc2sys service applies the timing from the RS_SLAVE port to the Grace Hopper server system clock.
Enabling Dual-port PTP#
Add both FH ports to /etc/ptp.conf.
The assignment of RS_SLAVE and RS_PASSIVE does not depend on the order in which the ports are listed.
The following is an example of the contents of /etc/ptp.conf:
[global]
dataset_comparison G.8275.x
G.8275.defaultDS.localPriority 128
maxStepsRemoved 255
logAnnounceInterval -3
logSyncInterval -4
logMinDelayReqInterval -4
G.8275.portDS.localPriority 128
network_transport L2
domainNumber 24
tx_timestamp_timeout 30
clientOnly 1
clock_servo pi
step_threshold 1.0
egressLatency 28
pi_proportional_const 4.65
pi_integral_const 0.1
announceReceiptTimeout 3
delay_mechanism E2E
network_transport L2
[aerial01]
[aerial00]
Create or edit /etc/systemd/system/phc2sys.service with the following example contents:
[Unit]
Description=Synchronize system clock or PTP hardware clock (PHC)
Documentation=man:phc2sys
Requires=ptp4l.service
After=ptp4l.service
[Service]
Restart=always
RestartSec=5s
Type=simple
# Gives ptp4l a chance to stabilize
ExecStartPre=sleep 2
# Dual Port PTP
ExecStart=/bin/sh -c "/usr/sbin/phc2sys -a -r -n 24 -R 256 -u 256 -l 7"
[Install]
WantedBy=multi-user.target
Set Up the Boot Configuration Service#
Create the directory /usr/local/bin and create the /usr/local/bin/nvidia.sh file to run the commands on every reboot.
Note
The command for nvidia-smi -lgc expects just one GPU device (-i 0). This needs to be modified if the system uses more than one GPU.
The mode must be set to 1 for the GH200 so that it can utilize the max clock rate; otherwise, it is limited to 1830MHz with the default mode=0.
Create the rcu_affinity_manager.sh script referenced in the nvidia.sh script. This file can be found in the Aerial CUDA-Accelerated RAN source code at NVIDIA/aerial-cuda-accelerated-ran. After downloading the source, ensure the rcu_affinity_manager.sh file is copied to /usr/local/bin/rcu_affinity_manager.sh.
$ wget raw.githubusercontent.com/NVIDIA/aerial-cuda-accelerated-ran/refs/heads/main/cubb_scripts/infra/rcu_affinity_manager.sh
$ sudo cp rcu_affinity_manager.sh /usr/local/bin/rcu_affinity_manager.sh
$ sudo chmod +x /usr/local/bin/rcu_affinity_manager.sh
Create a system service file to be loaded after network interfaces are up.
Create a system service file for nvidia-persistenced to be run at startup.
Note
This file was created following the sample from /usr/share/doc/NVIDIA_GLX-1.0/samples/nvidia-persistenced-init.tar.bz2
Then set the file permissions, reload the systemd daemon, enable the service, restart the service when installing for the first time, and check the status.
$ sudo chmod 744 /usr/local/bin/nvidia.sh
$ sudo chmod 664 /etc/systemd/system/nvidia.service
$ sudo chmod 664 /etc/systemd/system/nvidia-persistenced.service
$ sudo systemctl daemon-reload
$ sudo systemctl enable nvidia-persistenced.service
$ sudo systemctl enable nvidia.service
$ sudo systemctl restart nvidia.service
$ sudo systemctl restart nvidia-persistenced.service
$ sudo systemctl status nvidia.service
$ sudo systemctl status nvidia-persistenced.service
The output of the last command should look like this:
$ sudo systemctl status nvidia.service
○ nvidia.service
Loaded: loaded (/etc/systemd/system/nvidia.service; enabled; vendor preset: enabled)
Active: inactive (dead) since Wed 2026-03-25 06:14:33 UTC; 1s ago
Process: 261646 ExecStart=/usr/local/bin/nvidia.sh (code=exited, status=0/SUCCESS)
Main PID: 261646 (code=exited, status=0/SUCCESS)
CPU: 3.560s
Mar 25 06:14:33 server nvidia.sh[265497]: CPU 3: : 1 RCU processes
Mar 25 06:14:33 server nvidia.sh[265497]: CPU 65:: 1 RCU processes
Mar 25 06:14:33 server nvidia.sh[265497]: CPU 66:: 1 RCU processes
Mar 25 06:14:33 server nvidia.sh[265497]: CPU 67:: 1 RCU processes
Mar 25 06:14:33 server nvidia.sh[265497]: CPU 68:: 1 RCU processes
Mar 25 06:14:33 server nvidia.sh[265497]: CPU 69:: 1 RCU processes
Mar 25 06:14:33 server nvidia.sh[265497]: CPU 71:: 1 RCU processes
Mar 25 06:14:33 server nvidia.sh[262191]: === End of RCU Process Affinity Information ===
Mar 25 06:14:33 server systemd[1]: nvidia.service: Deactivated successfully.
Mar 25 06:14:33 server systemd[1]: nvidia.service: Consumed 3.560s CPU time.
$ sudo systemctl status nvidia-persistenced.service
● nvidia-persistenced.service - NVIDIA Persistence Daemon
Loaded: loaded (/etc/systemd/system/nvidia-persistenced.service; enabled; vendor preset: enabled)
Active: active (running) since Wed 2026-03-25 06:17:41 UTC; 2s ago
Process: 265541 ExecStart=/usr/bin/nvidia-persistenced (code=exited, status=0/SUCCESS)
Main PID: 265542 (nvidia-persiste)
Tasks: 1 (limit: 146803)
Memory: 1.3M
CPU: 6ms
CGroup: /system.slice/nvidia-persistenced.service
└─265542 /usr/bin/nvidia-persistenced
Mar 25 06:17:41 server systemd[1]: Starting NVIDIA Persistence Daemon...
Mar 25 06:17:41 server nvidia-persistenced[265542]: Started (265542)
Mar 25 06:17:41 server systemd[1]: Started NVIDIA Persistence Daemon.
Validating software-component versions and system configurations#
Before running Aerial, make sure that your software-component versions and system configurations meet the required specifications. For more information, refer to the System Configuration Validation Script.