Part 3. Configure the gNB Server
To install the Aerial tools, follow the the cuBB installation guide; refer to the Software Manifest for a link to the cuBB documentation.
To install the Aerial cuBB tools, follow the Gigabyte cuBB installation guide.
The ARC-OTA thread-to-core assignment functionality is different from a standard Aerial installation.
Layer 1 threads need to be isolated in a monolithic block and have been moved, with the rest
left to layers 2 and higher: Cores 2-6 are used as cuPHY worker cores; core 7 is used for
the cuPHY lowprio thread; core 8 is used for the cuPHY timer thread; and core 9 is used for PTP and
PHC2SYS.
Configure Linux Kernel Command-Line for ARC-OTA
To set kernel command-line parameters, edit the GRUB_CMDLINE_LINUX_DEFAULT parameter in the grub
file /etc/default/grub and modify the following parameters.
The following kernel parameters are optimized for the Gigabyte server with 24 cores Xeon Gold 6240R and 96GB memory. For ARC-OTA, typically it is optimal to configure the gNB to isolate cores 2 to 10 for Aerial, and leave the other cores for OAI L2+.
default_hugepagesz=1G
hugepagesz=1G
hugepages=16
tsc=reliable
clocksource=tsc
intel_idle.max_cstate=0
mce=ignore_ce processor.max_cstate=0
intel_pstate=disable
audit=0
idle=poll
isolcpus=2-10
rcu_nocb_poll
nosoftlockup
iommu=off
intel_iommu=off
irqaffinity=0-1,22-23
To automatically append the grub file with these changes, use this command:
sudo sed -i 's/^GRUB_CMDLINE_LINUX_DEFAULT="[^"]*/& default_hugepagesz=1G hugepagesz=1G hugepages=16 tsc=reliable clocksource=tsc intel_idle.max_cstate=0 mce=ignore_ce processor.max_cstate=0 intel_pstate=disable audit=0 idle=poll isolcpus=2-10 rcu_nocb_poll nosoftlockup iommu=off intel_iommu=off irqaffinity=0-1,22-23/' /etc/default/grub
Apply the Changes
Use the following commands to apply the changes and reboot to load the kernel.
sudo update-grub sudo reboot
After rebooting, enter the following command to check whether the system has booted into the low-latency kernel:
uname -r 5.15.0-1042-nvidia-lowlatency
Enter this command to check that the kernel command-line parameters are configured correctly:
cat /proc/cmdline BOOT_IMAGE=/vmlinuz-5.15.0-1042-nvidia-lowlatency root=/dev/mapper/ubuntu--vg-ubuntu--lv ro default_hugepagesz=1G hugepagesz=1G hugepages=16 tsc=reliable clocksource=tsc intel_idle.max_cstate=0 mce=ignore_ce processor.max_cstate=0 intel_pstate=disable audit=0 idle=poll isolcpus=2-10 rcu_nocb_poll nosoftlockup iommu=off intel_iommu=off irqaffinity=0-1,22-23
Change Core for ptp4l and phc2sys
Edit the /lib/systemd/system/ptp4l.service file as follows:
ExecStart=taskset -c 9 /usr/sbin/ptp4l -f /etc/ptp.conf
Edit the /lib/systemd/system/phc2sys.service file as follows:
ExecStart=/bin/sh -c "taskset -c 9 /usr/sbin/phc2sys -s /dev/ptp$(ethtool -T ens6f0 | grep PTP | awk '{print $4}')-c CLOCK_REALTIME -n 24 -O 0 -R 256 -u 256"
To install the Aerial cuBB tools, follow the Dell R750 cuBB installation guide.
The ARC-OTA thread-to-core assignment functionality is different from a standard Aerial installation.
Layer 1 threads need to be isolated in a monolithic block and have been moved, with the rest
left to layers 2 and higher. Cores [5,7,9,11,13] are used as cuPHY workers cores. Core 17 is
used for the cuPHY lowprio thread; core 15 is used for the cuPHY timer thread; and core 19
is used for PTP and PHC2SYS.
Configure the Linux Kernel Command Line for ARC-OTA
To set kernel command-line parameters, modify the following parameters under the GRUB_CMDLINE_LINUX_DEFAULT
parameter in the grub file (/etc/default/grub).
pci=realloc=off
default_hugepagesz=1G
hugepagesz=1G
hugepages=16
tsc=reliable
clocksource=tsc
intel_idle.max_cstate=0
mce=ignore_ce
processor.max_cstate=0
intel_pstate=disable
audit=0
idle=poll
rcu_nocb_poll
nosoftlockup
iommu=off
irqaffinity=0-3,44-47
isolcpus=5,7,9,11,13,15,17,19,21
noht
To automatically append the grub file with these changes, use this command:
sudo sed -i 's/^GRUB_CMDLINE_LINUX_DEFAULT="[^"]*/& pci=realloc=off default_hugepagesz=1G hugepagesz=1G hugepages=16 tsc=reliable clocksource=tsc intel_idle.max_cstate=0 mce=ignore_ce processor.max_cstate=0 intel_pstate=disable audit=0 idle=poll rcu_nocb_poll nosoftlockup iommu=off irqaffinity=0-3,44-47 isolcpus=5,7,9,11,13,15,17,19,21 noht/' /etc/default/grub
Apply the Changes and Load the Kernel
Use the following commands to apply the command-line changes and reboot the system.
sudo update-grub sudo reboot
After rebooting, use the following command to check whether the system has booted into the low-latency kernel:
uname -r 5.15.0-1042-nvidia-lowlatency
Enter this command to check that the kernel command-line parameters are configured correctly:
cat /proc/cmdline BOOT_IMAGE=/vmlinuz-5.15.0-1042-nvidia-lowlatency root=/dev/mapper/ubuntu--vg-ubuntu--lv ro pci=realloc=off default_hugepagesz=1G hugepagesz=1G hugepages=16 tsc=reliable clocksource=tsc intel_idle.max_cstate=0 mce=ignore_ce processor.max_cstate=0 intel_pstate=disable audit=0 idle=poll rcu_nocb_poll nosoftlockup iommu=off irqaffinity=0-3,44-47 isolcpus=5,7,9,11,13,15,17,19,21 noht
Change the Core for ptp4l and phc2sys
Edit the /lib/systemd/system/ptp4l.service file:
ExecStart=taskset -c 19 /usr/sbin/ptp4l -f /etc/ptp.conf
Edit the /lib/systemd/system/phc2sys.service file:
ExecStart=/bin/sh -c "taskset -c 19 /usr/sbin/phc2sys -s /dev/ptp$(ethtool -T enp204s0f1np1 | grep PTP | awk '{print $4}')-c CLOCK_REALTIME -n 24 -O 0 -R 256 -u 256"
To install the Aerial cuBB tools, follow the same steps as the Grace Hopper MGX installation guide.
The ARC-OTA thread-to-core assignment functionality is different from a standard Aerial installation.
Layer 1 threads need to be isolated in a monolithic block and have been moved, with the rest
left to layers 2 and higher. Cores [5,7,9,11,13] are used as cuPHY worker cores. Core 17 is
used for the cuPHY lowprio thread; core 15 is used for the cuPHY timer thread; and core 41
is used for PTP and PHC2SYS.
Configure the Linux Kernel Command Line for ARC-OTA
To set kernel command-line parameters, edit the GRUB_CMDLINE_LINUX parameter in the grub
file /etc/default/grub.d/cmdline.cfg and append or update the parameters described below. The following kernel parameters
are optimized for GH200. To automatically append the grub file with these parameters, use this command:
cat <<"EOF" | sudo tee /etc/default/grub.d/cmdline.cfg
GRUB_CMDLINE_LINUX="$GRUB_CMDLINE_LINUXpci=realloc=off pci=pcie_bus_safe default_hugepagesz=512M hugepagesz=512M hugepages=32 tsc=reliable processor.max_cstate=0 audit=0 idle=poll rcu_nocb_poll nosoftlockup irqaffinity=0 isolcpus=managed_irq,domain,4-47 nohz_full=4-47 rcu_nocbs=4-47 earlycon module_blacklist=nouveau acpi_power_meter.force_cap_on=y numa_balancing=disable init_on_alloc=0 preempt=none"
EOF
Apply the Changes and Load the Kernel
Use the following commands to apply the command-line changes and reboot the system.
sudo update-grub sudo reboot
After rebooting, use the following command to check whether the system has booted into the low-latency kernel:
uname -r 6.2.0-1012-nvidia-64k
Enter this command to check that the kernel command-line parameters are configured correctly:
cat /proc/cmdline BOOT_IMAGE=/vmlinuz-6.2.0-1012-nvidia-64k root=/dev/mapper/ubuntu--vg-ubuntu--lv ro pci=realloc=off pci=pcie_bus_safe default_hugepagesz=512M hugepagesz=512M hugepages=32 tsc=reliable processor.max_cstate=0 audit=0 idle=poll rcu_nocb_poll nosoftlockup irqaffinity=0 isolcpus=managed_irq,domain,4-47 nohz_full=4-47 rcu_nocbs=4-47 earlycon module_blacklist=nouveau acpi_power_meter.force_cap_on=y numa_balancing=disable init_on_alloc=0 preempt=none
Change the Core for ptp4l and phc2sys
Edit the /lib/systemd/system/ptp4l.service file:
ExecStart=taskset -c 41 /usr/sbin/ptp4l -f /etc/ptp.conf
Edit the /lib/systemd/system/phc2sys.service file:
ExecStart=/bin/sh -c "taskset -c 41 /usr/sbin/phc2sys -s /dev/ptp\$(ethtool -T aerial00 | grep PTP | awk '{print \$4}') -c CLOCK_REALTIME -n 24 -O 0 -R 256 -u 256"