Part 2. Configure the Network Hardware#

Note

Refer to the NVIDIA SDK Manager resources for setup and installation of ARC-OTA.

The network hardware is configured in the following steps.

  1. Setup the GrandMaster

  2. Setup the switch

  3. Setup PTP

  4. Setup Foxconn O-RU

Chapter 2.1 Setup the Qulsar GrandMaster#

Step 1.#

Follow the Qulsar User Guide to set up the MGMT connection.

../../_images/image11.png

Step 2.#

Set the operating mode to GNSS Only, and other fields as such, then run Start Engine.

../../_images/image2.png

Step 3.#

Enable the ports on the GrandMaster with the 8275.1 Profile configurations.

../../_images/image3.png

Step 4.#

Configure the clock configs as follows:

../../_images/image4.png

Step 5.#

Ensure the GPS configuration values are unchanged from the QG2 default settings.

../../_images/image5.png

Step 6.#

Verify that the GPS Signal reaches the GrandMaster.

../../_images/image6.png

Chapter 2.2 Switch Setup#

Chapter 2.2.1 Dell Switch#

The following example uses these VLAN 2 settings:

  • RUs are on ports 1 and 7

  • GrandMaster is on port 5

  • CN is on ports 11 and 12

  • gNB ports are connected to ports 49 and 51

  1. Set up MGMT access to the switch (in this case 172.168.20.67):

    OS10# configure terminal
    OS10(config)#
    interface mgmt1/1/1
    no shutdown
    no ip address dhcp
    ip address 172.16.204.67/22
    exit
    
  2. Use SSH to access admin@172.168.204.67.

  3. Set the speed to 10G for port groups 1 and 2.

    OS10(config)#
    port-group 1/1/1
    mode Eth 10g-4x
    exit
    port-group 1/1/2
    mode Eth 10g-4x
    exit
    
  4. Enable PTP on the switch.

    OS10# configure terminal
    OS10(config)#
    ptp clock boundary profile g8275.1
    ptp domain 24
    ptp system-time enable
    !
    
  5. Configure the GrandMaster port.

    OS10(config)#
    interface ethernet 1/1/5:1
    no shutdown
    no switchport
    ip address 169.254.2.1/24
    flowcontrol receive off
    ptp delay-req-min-interval -4
    ptp enable
    ptp sync-interval -4
    ptp transport layer2
    exit
    

    After some time, the following values will print:

    <165>1 2023-05-09T07:49:22.625584+00:00 OS10 dn_alm 1021 - - Node.1-Unit.1:PRI [event], Dell EMC (OS10) %PTP_SYSTEM_TIME_NOT_SET: System time is not set. System time will be set when the clock is.
    <165>1 2023-05-09T07:51:22.312557+00:00 OS10 dn_alm 1021 - - Node.1-Unit.1:PRI [event], Dell EMC (OS10) %PTP_CLOCK_PHASE_LOCKED: Clock servo is phase locked.
    <165>1 2023-05-09T07:51:22.313081+00:00 OS10 dn_alm 1021 - - Node.1-Unit.1:PRI [event], Dell EMC (OS10) %PTP_SYSTEM_TIME_UPDATE_STARTED: System time update service is started. Update interval: 60 minutes.
    <165>1 2023-05-09T07:51:59.334346+00:00 OS10 dn_alm 1021 - - Node.1-Unit.1:PRI [event], Dell EMC (OS10) %ALM_CLOCK_UPDATE: Clock changed MESSAGE=apt-daily.timer: Adding 6h 36min 18.719270s random time.
    <165>1 2023-05-09T07:57:27.254181+00:00 OS10 dn_alm 1021 - - Node.1-Unit.1:PRI [event], Dell EMC (OS10) %ALM_CLOCK_UPDATE: Clock changed MESSAGE=apt-daily.timer: Adding 4h 31mi
    
  6. Configure the Fronthaul Network Configuration by creating a VLAN.

    Note

    If you choose to use a different VLAN, you must modify the Aerial YAML file and O-RU configuration. C- and U-planes use the same VLAN.

    Create “VLAN 2”.

    OS10(config)#
    interface vlan 2
    OS10(conf-if-vl-2)#
    <165>1 2023-03-16T16:51:36.458730+00:00 OS10 dn_alm 813 - - Node.1-Unit.1:PRI [event], Dell EMC (OS10) %IFM_ASTATE_UP: Interface admin state up :vlan2
    
    OS10(conf-if-vl-2)# show configuration
    !
    interface vlan2
    no shutdown
    OS10(conf-if-vl-2)# exit
    
  7. Configure the RU, gNB, CN, and MEC ports.

    Interfaces that are configured to be slower than their maximum speed have a :1 appended to their name. This applies to ports in port groups 1 and 2.

    no shutdown
    switchport mode trunk
    switchport trunk allowed vlan 2
    mtu 8192
    flowcontrol receive off
    ptp enable
    ptp transport layer2
    ptp role timeTransmitter
    exit
    
  8. Check the PTP status.

    OS10# show ptp | no-more
    PTP Clock : Boundary
    Clock Identity : b0:4f:13:ff:ff:46:63:5f
    GrandMaster Clock Identity : fc:af:6a:ff:fe:02:bc:8d
    Clock Mode : One-step
    Clock Quality
    Class : 135
    Accuracy : <=100ns
    Offset Log Scaled Variance : 65535
    Domain : 24
    Priority1 : 128
    Priority2 : 128
    Profile : G8275-1(Local-Priority:-128)
    Steps Removed : 1
    Mean Path Delay(ns) : 637
    Offset From Master(ns) : 1
    Number of Ports : 8
    ----------------------------------------------------------------------------
    Interface State Port Identity
    ----------------------------------------------------------------------------
    Ethernet1/1/1:1 Master b0:4f:13:ff:ff:46:63:5f:1
    Ethernet1/1/3:1 Master b0:4f:13:ff:ff:46:63:5f:3
    Ethernet1/1/5:1 Slave b0:4f:13:ff:ff:46:63:5f:5
    Ethernet1/1/7:1 Master b0:4f:13:ff:ff:46:63:5f:8
    Ethernet1/1/11 Master b0:4f:13:ff:ff:46:63:5f:4
    Ethernet1/1/49 Master b0:4f:13:ff:ff:46:63:5f:9
    Ethernet1/1/51 Master b0:4f:13:ff:ff:46:63:5f:10
    Ethernet1/1/54 Master b0:4f:13:ff:ff:46:63:5f:2
    ----------------------------------------------------------------------------
    Number of slave ports :1
    Number of master ports :7
    
  9. Save the switch configuration:

    copy running-configuration startup-configuration
    

Chapter 2.2.2 Fibrolan Falcon RX Setup#

Although the Fibrolan switch has not been qualified in the NVIDIA lab, OAI labs incorporate the following configuration and switch for interoperability.

../../_images/fibrolan_1.png

To get started, follow the Fibrolan Getting Started Guide.

In this setup, the Qulsar GrandMaster is connected to port 4, the Aerial cuBB to port 17, and the Foxconn O-RU to port 16 (C/U plane) and port 15 (S/M plane). You can ignore all other ports in the figures[A][B] below.

VLAN Setup#

The following assumes that the VLAN tag is 2 for both the control plane and the user plane of the O-RAN CU plane. VLAN tag 80 is used for everything else.

Open the configuration page of the Fibrolan switch, then go to Configuration > VLANs. Port 4 (the Qulsar GrandMaster) needs to be set to “Access” mode, with the port VLAN set to 80.

../../_images/fibrolan_2.png

Figure A - VLAN Setup#

Use the same configuration for port 15 (RU S/M plane).

Configure ports 16 and 17 as follows:

  • Mode: “Trunk”

  • Port: VLAN 80

  • Untag Port VLAN

  • Allowed VLANs: 2, 80

../../_images/fibrolan_3.png

Figure B - VLAN Setup#

DHCP Setup#

The RU M-plane requires you to set up a DHCP server. Go to Configuration > DHCP > Server > Pool and create a new DHCP server with the following settings:

../../_images/fibrolan_4.png

PTP Setup on gNB#

For the PTP setup, follow the Fibrolan PTP Boundary Clock Configuration guide and use the following settings:

  • Device Type: “Ord-Bound”

  • Profile: “G8275.1”

  • Clock domain: 24

  • VLAN: 80

Also make sure you enable the used ports (in this case, 4, 15, 16, and 17).

Hybrid mode is recommended as the sync mode.

If everything is configured correctly, the SyncCenter should show green.

../../_images/fibrolan_5.png

Chapter 2.3 PTP Setup#

These commands assume that PTP4L runs on the ens6f0 NIC interface and uses CPU core 9. Core clash can cause problems, so if a different core is being used, it must not be used by L1 or L2+.

Verify Inbound PTP Packets#

Typically, you should see packets with ethertype 0x88f7 on the selected interface.

sudo tcpdump -i ens6f0 -c 5 | grep ethertype
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on ens6f1, link-type EN10MB (Ethernet), capture size 262144 bytes
13:27:41.291503 48:b0:2d:63:83:ac (oui Unknown) > 01:1b:19:00:00:00 (oui Unknown), ethertype Unknown (0x88f7), length 60:
13:27:41.291503 48:b0:2d:63:83:ac (oui Unknown) > 01:1b:19:00:00:00 (oui Unknown), ethertype Unknown (0x88f7), length 60:
13:27:41.296727 c4:5a:b1:14:1a:c6 (oui Unknown) > 01:1b:19:00:00:00 (oui Unknown), ethertype Unknown (0x88f7), length 78:
13:27:41.296784 c4:5a:b1:14:1a:c6 (oui Unknown) > 01:1b:19:00:00:00 (oui Unknown), ethertype Unknown (0x88f7), length 60:
13:27:41.306316 08:c0:eb:71:e7:d5 (oui Unknown) > 01:1b:19:00:00:00 (oui Unknown), ethertype Unknown (0x88f7), length 58:

Create ptp4l Configuration File#

Paste these commands into the shell to create the three configuration files:

cat <<EOF | sudo tee /etc/ptp.conf
[global]
priority1 128
priority2 128
domainNumber 24
tx_timestamp_timeout 30
dscp_event 46
dscp_general 46
logging_level 6
verbose 1
use_syslog 0
logMinDelayReqInterval 1
[ens6f0]
logAnnounceInterval -3
announceReceiptTimeout 3
logSyncInterval -4
logMinDelayReqInterval -4
delay_mechanism E2E
network_transport L2
EOF

cat <<EOF | sudo tee /lib/systemd/system/ptp4l.service
[Unit]
Description=Precision Time Protocol (PTP) service
Documentation=man:ptp4l

[Service]
Restart=always
RestartSec=5s
Type=simple
ExecStart=/usr/bin/taskset -c 9 /usr/sbin/ptp4l -f /etc/ptp.conf

[Install]
WantedBy=multi-user.target
EOF

Create phc2sys Configuration File#

# If more than one instance is already running, kill the existing
# PHC2SYS sessions.

# Command used can be found in /lib/systemd/system/phc2sys.service
# Update the ExecStart line to the following, assuming ens6f0 interface is used.
  sudo nano /lib/systemd/system/phc2sys.service

[Unit]
Description=Synchronize system clock or PTP hardware clock (PHC)
Documentation=man:phc2sys
After=ntpdate.service
Requires=ptp4l.service
After=ptp4l.service

[Service]
Restart=always
RestartSec=5s
Type=simple
ExecStart=/bin/sh -c "taskset -c 9 /usr/sbin/phc2sys -s /dev/ptp$(ethtool -T ens6f0 | grep PTP | awk '{print $4}') -c CLOCK_REALTIME -n 24 -O 0 -R 256 -u 256"

[Install]
WantedBy=multi-user.target

Enable and Start phc2sys and ptp4l#

After changing the configuration files, they need to be reloaded, enabled, and restarted. These services can be restarted if they don’t sync.

sudo systemctl daemon-reload
sudo systemctl enable ptp4l.service
sudo systemctl enable phc2sys.service

sudo systemctl restart phc2sys.service ptp4l.service

# check that the service is active and has low rms value (<30):
systemctl status ptp4l.service phc2sys.service
● ptp4l.service - Precision Time Protocol (PTP) service
     Loaded: loaded (/lib/systemd/system/ptp4l.service; enabled; vendor preset: enabled)
     Active: active (running) since Tue 2023-05-09 13:21:12 UTC; 14s ago
       Docs: man:ptp4l
   Main PID: 6962 (ptp4l)
      Tasks: 1 (limit: 94588)
     Memory: 544.0K
     CGroup: /system.slice/ptp4l.service
             └─6962 /usr/sbin/ptp4l -f /etc/ptp.conf

May 09 13:21:17 aerial-rf-gb-gnb taskset[6962]: ptp4l[15552.609]: rms   15 max   32 freq   -639 +/-  25 delay   211 +/-   1
May 09 13:21:18 aerial-rf-gb-gnb taskset[6962]: ptp4l[15553.609]: rms   21 max   29 freq   -583 +/-  12 delay   210 +/-   1
May 09 13:21:19 aerial-rf-gb-gnb taskset[6962]: ptp4l[15554.609]: rms   11 max   21 freq   -576 +/-   8 delay   211 +/-   1
May 09 13:21:20 aerial-rf-gb-gnb taskset[6962]: ptp4l[15555.609]: rms    6 max   13 freq   -579 +/-   8 delay   211 +/-   1
May 09 13:21:21 aerial-rf-gb-gnb taskset[6962]: ptp4l[15556.609]: rms    4 max    7 freq   -578 +/-   6 delay   212 +/-   0
May 09 13:21:22 aerial-rf-gb-gnb taskset[6962]: ptp4l[15557.609]: rms    5 max   11 freq   -589 +/-   6 delay   213 +/-   1
May 09 13:21:23 aerial-rf-gb-gnb taskset[6962]: ptp4l[15558.609]: rms    6 max   12 freq   -593 +/-   8 delay   210 +/-   1
May 09 13:21:24 aerial-rf-gb-gnb taskset[6962]: ptp4l[15559.609]: rms    3 max    7 freq   -587 +/-   5 delay   211 +/-   1
May 09 13:21:25 aerial-rf-gb-gnb taskset[6962]: ptp4l[15560.609]: rms    5 max   12 freq   -582 +/-   7 delay   212 +/-   1
May 09 13:21:26 aerial-rf-gb-gnb taskset[6962]: ptp4l[15561.609]: rms    4 max    7 freq   -587 +/-   7 delay   213 +/-   1 phc2sys.service - Synchronize system clock or PTP hardware clock (PHC)
     Loaded: loaded (/lib/systemd/system/phc2sys.service; enabled; vendor preset: enabled)
     Active: active (running) since Tue 2023-05-09 13:21:12 UTC; 14s ago
       Docs: man:phc2sys
   Main PID: 6963 (phc2sys)
      Tasks: 1 (limit: 94588)
     Memory: 572.0K
     CGroup: /system.slice/phc2sys.service
             └─6963 /usr/sbin/phc2sys -a -r -n 24 -R 256 -u 256

May 09 13:21:17 aerial-rf-gb-gnb phc2sys[6963]: [15553.320] CLOCK_REALTIME rms   42 max   79 freq  +8240 +/- 368 delay  1762 +/-  16
May 09 13:21:18 aerial-rf-gb-gnb phc2sys[6963]: [15554.336] CLOCK_REALTIME rms   35 max   64 freq  +8091 +/- 303 delay  1754 +/-  13
May 09 13:21:19 aerial-rf-gb-gnb phc2sys[6963]: [15555.352] CLOCK_REALTIME rms   27 max   52 freq  +8218 +/- 224 delay  1752 +/-  13
May 09 13:21:20 aerial-rf-gb-gnb phc2sys[6963]: [15556.368] CLOCK_REALTIME rms   21 max   49 freq  +8153 +/- 152 delay  1758 +/-  16
May 09 13:21:21 aerial-rf-gb-gnb phc2sys[6963]: [15557.384] CLOCK_REALTIME rms   17 max   39 freq  +8149 +/- 125 delay  1761 +/-  16
May 09 13:21:22 aerial-rf-gb-gnb phc2sys[6963]: [15558.400] CLOCK_REALTIME rms   14 max   33 freq  +8185 +/- 101 delay  1750 +/-  14
May 09 13:21:23 aerial-rf-gb-gnb phc2sys[6963]: [15559.416] CLOCK_REALTIME rms   12 max   32 freq  +8138 +/-  63 delay  1752 +/-  13
May 09 13:21:24 aerial-rf-gb-gnb phc2sys[6963]: [15560.431] CLOCK_REALTIME rms   11 max   43 freq  +8171 +/-  54 delay  1756 +/-  15
May 09 13:21:25 aerial-rf-gb-gnb phc2sys[6963]: [15561.447] CLOCK_REALTIME rms   10 max   32 freq  +8163 +/-  38 delay  1762 +/-  16
May 09 13:21:26 aerial-rf-gb-gnb phc2sys[6963]: [15562.463] CLOCK_REALTIME rms    9 max   23 freq  +8162 +/-  17 delay  1761 +/-  16

Disable NTP#

Use these commands to turn off NTP:

sudo timedatectl set-ntp false
timedatectl
Local time: Thu 2022-02-03 22:30:58 UTC
           Universal time: Thu 2022-02-03 22:30:58 UTC
                 RTC time: Thu 2022-02-03 22:30:58
                Time zone: Etc/UTC (UTC, +0000)
System clock synchronized: no
              NTP service: inactive
          RTC in local TZ: no

Verify System Clock Synchronization#

Make NTP inactive and synchronize the system clock:

timedatectl
Local time: Thu 2022-02-03 22:30:58 UTC
           Universal time: Thu 2022-02-03 22:30:58 UTC
                 RTC time: Thu 2022-02-03 22:30:58
                Time zone: Etc/UTC (UTC, +0000)
System clock synchronized: yes
              NTP service: inactive
          RTC in local TZ: no

Chapter 2.4 Set up the Foxconn ORU#

Tip

There is a tutorial video for setting up the Foxconn ORU.

../../_images/image7.png

Foxconn RPQN-7801E

Connections and Settings

image8

Connections:

  • 10SFP: C/U plane (will support S/M plane after firmware upgrade)

  • 1G RJ45: S/M plane

  • 10G RJ45: POE only

  • Micro-USB: USB to serial for debugging (115200, 8, 1, none, flow control off)

GrandMaster settings (Qulsar):

  • PTP timing port: Disable VLAN

  • Two steps: OFF

  • Domain number: 24 (needs to configured on O-RU)

  • IPv4, Unicast, etc.

/home/root/sdcard/RRHconfig_xran.xml:

  • RRH_PTPV2_GRAND_MASTER_IP = 20.0.0.8

  • RRH_PTPV2_SUB_DOMAIN_NUM = 24

  • C/U plane VLAN tag

  • RRH_LO_FREQUENCY_KHZ = 3750000

Configure VLAN and IP Address on the gNB Server#

  1. Add these commands to the server startup script (/etc/rc.local) so they are automatically run on reboot.

  2. Configure these settings on the fronthaul port.

  3. You must use IP addresses that do not match those in the example below:

sudo ip link add link ens6f0 name ens6f0.2 type vlan id 2
sudo ip addr add 169.254.1.103/24 dev ens6f0.2
sudo ip link set up ens6f0.2

O-RU M-Plane Setup#

  1. Add the following to the bottom of /etc/profile and comment out the line with set_qse.sh if it already exists. Set the interface initially to eth0 for firmware version 1, and to qse-eth after upgrading to firmware version 2 or greater.

    interface=eth0
    vlanid=2
    ipLastOctet=20
    
    ip link add link ${interface} name ${interface}.$vlanid type vlan id $vlanid
    ip addr flush dev ${interface}
    ip addr add 169.254.0.0/24 dev ${interface}
    ip addr add 169.254.1.${ipLastOctet}/24 dev ${interface}.$vlanid
    ip link set up ${interface}.$vlanid
    
  2. Reboot the O-RU using the command ./reboot.sh and check the network configuration:

    # ip r
    169.254.1.0/24 dev eth0.2 src 169.254.1.20
    

Update O-RU Configuration#

Note

If you are using the CBRS O-RU (Foxconn RPQN-4800E), refer to the note below for the modified configuration.

  1. Update the O-RU configuration in /home/root/test/RRHconfig_xran.xml.

    root@arria10:~/test# grep -v '<!-' RRHconfig_xran.xml
    RRH_DST_MAC_ADDR = 08:c0:eb:71:e7:d4 # To match fronthaul interface of DU
    RRH_SRC_MAC_ADDR = 6C:AD:AD:00:04:6C # To match qse-eth of RU
    RRH_EN_EAXC_ID = 0
    RRH_EAXC_ID_TYPE1 = 0x0, 0x1, 0x2, 0x3
    RRH_EAXC_ID_TYPE3 = 0x8, 0x9, 0xA, 0xB
    RRH_EN_SPC = 1
    RRH_RRH_LTE_OR_NR = 1
    RRH_TRX_EN_BIT_MASK = 0x0f
    RRH_RF_EN_BIT_MASK = 0x0f
    RRH_CMPR_HDR_PRESENT = 0
    RRH_CMPR_TYPE = 1, 1
    RRH_CMPR_BIT_LENGTH = 9, 9
    RRH_UL_INIT_SYM_ID = 0
    RRH_TX_TRUNC_BITS = 4
    RRH_RX_TRUNC_BITS = 4
    RRH_MAX_PRB = 273
    RRH_C_PLANE_VLAN_TAG = 0x0002 #To match vlan id set in cuphycontroller yaml file
    RRH_U_PLANE_VLAN_TAG = 0x0002 #To match vlan id set in cuphycontroller yaml file
    RRH_SLOT_TICKS_IN_SEC = 2000
    RRH_SLOT_PERIOD_IN_SAMPLE = 61440
    RRH_LO_FREQUENCY_KHZ = 3750000, 0
    RRH_TX_POWER = 24, 24
    RRH_TX_ATTENUATION = 12.0, 12.0, 12.0, 12.0
    RRH_RX_ATTENUATION = 0.0, 0.0, 0.0, 0.0
    RRH_BB_GENERAL_CTRL = 0x0, 0x0, 0x0, 0x0
    RRH_RF_GENERAL_CTRL = 0x3, 0x1, 0x0, 0x0
    RRH_PTPV2_GRAND_MASTER_MODE = 3
    RRH_PTPV2_JITTER_LEVEL = 0
    RRH_PTPV2_VLAN_ID = 0
    RRH_PTPV2_IP_MODE = 4
    RRH_PTPV2_GRAND_MASTER_IP = 192.167.27.150
    RRH_PTPV2_SUB_DOMAIN_NUM = 24
    RRH_PTPV2_ACCEPTED_CLOCK_CLASS = 135
    RRH_TRACE_PERIOD = 10
    RRH_DL_IQ_SCALING = 0x1001
    RRH_CFR_PEAK_THRESHOLD = 0.5
    

    Note

    In Foxconn firmware version 2.6.9, the configuration file is located in /home/root/sdcard.

    Note

    The above configuration was taken from an ORU running firmware 3.1.15.

    Note

    If you’re using the CBRS O-RU (Foxconn RPQN-4800E), the above parameters should be modified as follows:

    • n78:

      RRH_LO_FREQUENCY_KHZ = 3750000, 0

    • n48 (CBRS):

      RRH_LO_FREQUENCY_KHZ = 3649140, 0

  2. Reboot O-RU.

    cd /home/root/test/
    
    ./reboot
    
  3. Run the following to enable the configuration:

    cd /home/root/test/
    
    ./init_rrh_config_enable_cuplane
    
  4. To see the ORU status run the following script.

    cd /home/root/test/
    
    ./chk_con.sh