To install the Aerial tools follow the Aerial devkit cuBB installation guide.
The ARC-OTA thread-to-core assignment functionality is different from standard Aerial installation. Layer 1 threads need to be isolated in a monolithic block and have been moved, with the rest left to layers 2 and higher. Cores 2-6 are used as cuPHY workers cores. Core 7 is used for the cuPHY lowprio thread. Core 8 is used for the cuPHY timer thread, and core 9 for PTP and PHC2SYS.
Configure Linux Kernel Command-Line for ARC-OTA
To set kernel command-line parameters, edit the GRUB_CMDLINE_LINUX_DEFAULT parameter in the grub
file /etc/default/grub andmodify the following parameters.
The following kernel parameters are optimized for the Aerial DevKit with 24 cores Xeon Gold 6240R and 96GB memory. For ARC-OTA, typically it is optimal to configure the gNB to isolate core 2 to 10 for Aerial, and leave the other cores for use by OAI L2+.
To automatically append the grub file with these changes, enter this command:
$ sudo sed -i 's/^GRUB_CMDLINE_LINUX_DEFAULT="[^"]*/& default_hugepagesz=1G hugepagesz=1G hugepages=16 tsc=reliable clocksource=tsc intel_idle.max_cstate=0 mce=ignore_ce processor.max_cstate=0 intel_pstate=disable audit=0 idle=poll isolcpus=2-10 rcu_nocb_poll nosoftlockup iommu=off intel_iommu=off irqaffinity=0-1,22-23/' /etc/default/grub
Apply the Changes and Reboot to Load the Kernel
$ sudo update-grub
$ sudo reboot
After rebooting, enter the following command to check whether the system has booted into the low-latency kernel:
$ uname -r
5.15.0-1042-nvidia-lowlatency
Enter this command to check that the kernel command-line parameters are configured properly:
$ cat /proc/cmdline
BOOT_IMAGE=/vmlinuz-5.15.0-1042-nvidia-lowlatency root=/dev/mapper/ubuntu--vg-ubuntu--lv ro default_hugepagesz=1G hugepagesz=1G hugepages=16 tsc=reliable clocksource=tsc intel_idle.max_cstate=0 mce=ignore_ce processor.max_cstate=0 intel_pstate=disable audit=0 idle=poll isolcpus=2-10 rcu_nocb_poll nosoftlockup iommu=off intel_iommu=off irqaffinity=0-1,22-23
Change Core for ptp4l and phc2sys
Edit /lib/systemd/system/ptp4l.service:
ExecStart=taskset -c 9 /usr/sbin/ptp4l -f /etc/ptp.conf
Edit /lib/systemd/system/phc2sys.service:
ExecStart=/bin/sh -c "taskset -c 9 /usr/sbin/phc2sys -s /dev/ptp$(ethtool -T ens6f0 | grep PTP | awk '{print $4}')-c CLOCK_REALTIME -n 24 -O 0 -R 256 -u 256"
To install the Aerial tools, follow the Dell R750 cuBB installation guide.
The ARC-OTA thread-to-core assignment functionality is different from standard Aeriald installation. Layer 1 threads need to be isolated in a monolithic block and have been moved, with the rest left to layers 2 and higher. Cores 5,7,9,11,13 are used as cuPHY workers cores. Core 17 is used for the cuPHY lowprio thread. Core 15 is used for the cuPHY timer thread, and core 19 for PTP and PHC2SYS.
Configure the Linux Kernel Command Line for ARC-OTA
To set kernel command-line parameters, modify the following parameters under the``GRUB_CMDLINE_LINUX_DEFAULT``
parameter in the grub file (/etc/default/grub).
To automatically append the grub file with these changes, use this command:
$ sudo sed -i 's/^GRUB_CMDLINE_LINUX_DEFAULT="[^"]*/& pci=realloc=off default_hugepagesz=1G hugepagesz=1G hugepages=16 tsc=reliable clocksource=tsc intel_idle.max_cstate=0 mce=ignore_ce processor.max_cstate=0 intel_pstate=disable audit=0 idle=poll rcu_nocb_poll nosoftlockup iommu=off irqaffinity=0-3,44-47 isolcpus=5,7,9,11,13,15,17,19,21 noht/' /etc/default/grub
Apply the Changes and Load the Kernel
Use the following commands to apply the command-line changes and reboot the system.
$ sudo update-grub
$ sudo reboot
After rebooting, enter the following command to check whether the system has booted into the low-latency kernel:
$ uname -r
5.15.0-1042-nvidia-lowlatency
Enter this command to check that the kernel command-line parameters are configured properly:
$ cat /proc/cmdline
BOOT_IMAGE=/vmlinuz-5.15.0-1042-nvidia-lowlatency root=/dev/mapper/ubuntu--vg-ubuntu--lv ro pci=realloc=off default_hugepagesz=1G hugepagesz=1G hugepages=16 tsc=reliable clocksource=tsc intel_idle.max_cstate=0 mce=ignore_ce processor.max_cstate=0 intel_pstate=disable audit=0 idle=poll rcu_nocb_poll nosoftlockup iommu=off irqaffinity=0-3,44-47 isolcpus=5,7,9,11,13,15,17,19,21 noht
Change the Core for ptp4l and phc2sys
Edit the /lib/systemd/system/ptp4l.service file:
ExecStart=taskset -c 19 /usr/sbin/ptp4l -f /etc/ptp.conf
Edit the /lib/systemd/system/phc2sys.service file:
ExecStart=/bin/sh -c "taskset -c 19 /usr/sbin/phc2sys -s /dev/ptp$(ethtool -T enp204s0f1np1 | grep PTP | awk '{print $4}')-c CLOCK_REALTIME -n 24 -O 0 -R 256 -u 256"