E2E gNodeB on MIG#
This page covers how to set up E2E gNodeB on MIG.
Setting up MIG for Aerial#
Check GPU Device availability#
To check the available GPUs on the system and get the GPU-ID, run the nvidia-smi -L command.
$ nvidia-smi -L
GPU 0: NVIDIA GH200 480GB (UUID: GPU-51c12aab-5ee1-2f10-a4b7-6baacfec5e31)
Partition GPUs#
Run the
nvidia-smi -i <GPU_ID> -mig 1command to enable MIG mode on the GPU(s).Note
If
-i <GPU_ID>is not specified, then MIG mode is applied to all the GPUs on the system.$ sudo nvidia-smi -i 0 -mig 1 Enabled MIG Mode for GPU 00000009:01:00.0 All done.
Check the available partition options using the
nvidia-smi mig -lgipcommand.The following example displays the results from GH.
$ sudo nvidia-smi mig -lgip +-----------------------------------------------------------------------------+ | GPU instance profiles: | | GPU Name ID Instances Memory P2P SM DEC ENC | | Free/Total GiB CE JPEG OFA | |=============================================================================| | 0 MIG 1g.12gb 19 7/7 11.00 No 16 1 0 | | 1 1 0 | +-----------------------------------------------------------------------------+ | 0 MIG 1g.12gb+me 20 1/1 11.00 No 16 1 0 | | 1 1 1 | +-----------------------------------------------------------------------------+ | 0 MIG 1g.24gb 15 4/4 23.00 No 26 1 0 | | 1 1 0 | +-----------------------------------------------------------------------------+ | 0 MIG 2g.24gb 14 3/3 23.00 No 32 2 0 | | 2 2 0 | +-----------------------------------------------------------------------------+ | 0 MIG 3g.48gb 9 2/2 46.50 No 60 3 0 | | 3 3 0 | +-----------------------------------------------------------------------------+ | 0 MIG 4g.48gb 5 1/1 46.50 No 64 4 0 | | 4 4 0 | +-----------------------------------------------------------------------------+ | 0 MIG 7g.96gb 0 1/1 93.00 No 132 7 0 | | 8 7 1 | +-----------------------------------------------------------------------------+
Slice the GPU using the
nvidia-smi mig -cgi <PROFILE> -Ccommand.The following example uses one Profile with 4g and one with 3g.
$ sudo nvidia-smi mig -cgi 9,5 -C Successfully created GPU instance ID 2 on GPU 0 using profile MIG 3g.48gb (ID 9) Successfully created compute instance ID 0 on GPU 0 GPU instance ID 2 using profile MIG 3g.48gb (ID 2) Successfully created GPU instance ID 1 on GPU 0 using profile MIG 4g.48gb (ID 5) Successfully created compute instance ID 0 on GPU 0 GPU instance ID 1 using profile MIG 4g.48gb (ID 3)
Check the GPU partitions using the
nvidia-smi -Lcommand.The following example displays the results from GH.
$ nvidia-smi -L GPU 0: NVIDIA GH200 480GB (UUID: GPU-51c12aab-5ee1-2f10-a4b7-6baacfec5e31) MIG 4g.48gb Device 0: (UUID: MIG-e9f0fa8c-548f-5fc5-aa58-51ef34c2816a) MIG 3g.48gb Device 1: (UUID: MIG-fcc563dc-5c8d-5de2-a448-439bde80400c)
Note
MIG mode is not persistent over reboots, so you may need to run above commands after each reboot.
Disabling MIG#
To disable MIG , use the nvidia-smi -i <GPU_ID> -mig 0 command.
The following example shows how to disable MIG on GH:
$ sudo nvidia-smi -i 0 -mig 0
Bringing up cuBB with a MIG Instance#
Start the cuBB Container#
To start the L1 container with a specific MIG instance, pass the variable argument specifying the MIG instance UUID to docker run.
The following example brings up the cuBB container on a MIG4g.48gb device with the UUID MIG-3c64afd5-8e80-53c3-99c8-79e84da3f895.
$ sudo docker run --restart unless-stopped -dP --runtime=nvidia -e NVIDIA_VISIBLE_DEVICES="MIG-3c64afd5-8e80-53c3-99c8-79e84da3f895" --network host --shm-size=4096m --privileged -it --device=/dev/gdrdrv:/dev/gdrdrv -v /lib/modules:/lib/modules -v /dev/hugepages:/dev/hugepages --userns=host --ipc=host -v /usr/src:/usr/src -v /home/aerial/nfs:/root -v /home/aerial/nfs:/cuBBSrc -v /home/aerial/nfs:/home/aerial/nfs -e CUDA_VISIBLE_DEVICES=<UUID of MIG-3> --name 25-3-mig gitlab-master.nvidia.com:5005/gputelecom/container/aerial-cuda-accelerated-ran:25-3-rc3 bash
If successful, the above command creates a container with the name “25-3-mig”.
Enter the cuBB Container#
Enter the cuBB container using the following command:
$ docker exec -it 25-3-mig /bin/bash
Check the GPU Device Inside the Container#
Check the GPU device inside the container using the following command. The MIG4g.48gB device should be visible in the container if the above command was used to create the container.
$ nvidia-smi -L
GPU 0: NVIDIA GH200 480GB (UUID: GPU-9b84da3e-9704-6b26-257b-a0665d400ac9)
MIG 4g.48gb Device 0: (UUID: MIG-3c64afd5-8e80-53c3-99c8-79e84da3f895)
$ nvidia-smi
Mon Nov 17 23:43:16 2025
+-----------------------------------------------------------------------------------------+
| NVIDIA-SMI 575.57.08 Driver Version: 575.57.08 CUDA Version: 12.9 |
|-----------------------------------------+------------------------+----------------------+
| GPU Name Persistence-M | Bus-Id Disp.A | Volatile Uncorr. ECC |
| Fan Temp Perf Pwr:Usage/Cap | Memory-Usage | GPU-Util Compute M. |
| | | MIG M. |
|=========================================+========================+======================|
| 0 NVIDIA GH200 480GB On | 00000009:01:00.0 Off | On |
| N/A 33C P0 84W / 900W | N/A | N/A Default |
| | | Enabled |
+-----------------------------------------+------------------------+----------------------+
+-----------------------------------------------------------------------------------------+
| MIG devices: |
+------------------+----------------------------------+-----------+-----------------------+
| GPU GI CI MIG | Memory-Usage | Vol| Shared |
| ID ID Dev | BAR1-Usage | SM Unc| CE ENC DEC OFA JPG |
| | | ECC| |
|==================+==================================+===========+=======================|
| 0 1 0 0 | 64MiB / 47616MiB | 64 0 | 4 0 4 0 4 |
| | 0MiB / 0MiB | | |
+------------------+----------------------------------+-----------+-----------------------+
+-----------------------------------------------------------------------------------------+
| Processes: |
| GPU GI CI PID Type Process name GPU Memory |
| ID ID Usage |
|=========================================================================================|
| No running processes found |
+-----------------------------------------------------------------------------------------+
Start cuPHYController/L1 Binaries#
These instructions assume the cuBB has been compiled. If not, use the following commands inside the cuBB container:
cmake -Bbuild -GNinja -DCMAKE_TOOLCHAIN_FILE=cuPHY/cmake/toolchains/grace-cross
cmake --build build -j$(nproc --all)
Update the mps_sm_* configurations in cuphycontroller_P5G_FXN.yaml to the following values:
Note
The mps_sm_* values should not exceed the maximum SM for the MIG device used by cuBB container. In this example, the SM of the MIG4g.48gb device is 64, so the mps_sm_* values
do not exceed this number.
mps_sm_pusch: 60
mps_sm_pucch: 2
mps_sm_prach: 2
mps_sm_ul_order: 20
mps_sm_pdsch: 62
mps_sm_pdcch: 10
mps_sm_pbch: 2
mps_sm_gpu_comms: 16
mps_sm_srs: 16
Create the following run-l1.sh script to stop/start the MPS and run cuPHYController.
#!/bin/bash
# This script to be used after getting into the docker image
export cuBB_SDK=$(pwd)
export CUDA_DEVICE_MAX_CONNECTIONS=8
export CUDA_MPS_PIPE_DIRECTORY=/var
export CUDA_MPS_LOG_DIRECTORY=/var
# Stop existing MPS
echo "Stop existing mps"
sudo -E echo quit | sudo -E nvidia-cuda-mps-control
# Start MPS
echo "Start mps"
sudo -E nvidia-cuda-mps-control -d
sudo -E echo start_server -uid 0 | sudo -E nvidia-cuda-mps-control
sudo -E ${cuBB_SDK}/cuPHY-CP/cuphycontroller/config/cuphycontroller_P5G_FXN.yaml
Adding Routes on CN and PDN#
Adding a PDN Route on CN#
Navigate to the
/sbinfolder on the CN machine and create a script namedadd-route.sh.Add the following contents to the
add-route.shscript. The PDN server IP is given as169.254.200.1; modify this value as needed based on your PDN IP setup.#!/bin/bash container_id=`docker ps | grep dataplane | awk '{print$1}'` echo "*************** Adding route to PDN inside VPP ***************" echo -e "\n" docker exec -it $container_id bash -c "vppctl ip route add 0.0.0.0/0 via 169.254.200.1 net1" echo -e "\n" echo "*************** Checking added route ***************" echo -e "\n" docker exec -it $container_id bash -c "vppctl show ip fib"
Provide full permissions permissions for the script:
chmod 777 add-route.shRun the script:
./add-route.sh.Note
This route may get deleted at some point, in which case you will need to run the
add-route.shscript again. If CUE cannot connect to internet, this is an indication that the route was deleted on the CN.
Adding Routes on PDN to enable Internet#
The PDN server has 2 IP addresses:
PDN VM Interface
IP:
192.168.122.11Interface name:
enp6s0
PDN server Interface: The IP of this interface is configured on the CN machine.
IP:
169.254.200.1Interface name:
enp1s0
Add the first route for a UE IP range of
21.21.21.*.iptables -t nat -A POSTROUTING -s 21.21.21.0/24 -p all -j SNAT --to-source 192.168.122.11
Create a script named
internet_enable.shwith the content below.Note
Ensure the WANIF and LANIF are set properly.
#! /bin/bash IPTABLES=/sbin/iptables WANIF='enp6s0' LANIF='enp1s0' # enable ip forwarding in the kernel echo 'Enabling Kernel IP forwarding...' /bin/echo 1 > /proc/sys/net/ipv4/ip_forward # flush rules and delete chains echo 'Flushing rules and deleting existing chains...' $IPTABLES -F $IPTABLES -X # enable masquerading to allow LAN internet access echo 'Enabling IP Masquerading and other rules...' $IPTABLES -t nat -A POSTROUTING -o $LANIF -j MASQUERADE $IPTABLES -A FORWARD -i $LANIF -o $WANIF -m state --state RELATED,ESTABLISHED -j ACCEPT $IPTABLES -A FORWARD -i $WANIF -o $LANIF -j ACCEPT $IPTABLES -t nat -A POSTROUTING -o $WANIF -j MASQUERADE $IPTABLES -A FORWARD -i $WANIF -o $LANIF -m state --state RELATED,ESTABLISHED -j ACCEPT $IPTABLES -A FORWARD -i $LANIF -o $WANIF -j ACCEPT echo 'Done.' $IPTABLES -X # enable masquerading to allow LAN internet access echo 'Enabling IP Masquerading and other rules...' $IPTABLES -t nat -A POSTROUTING -o $LANIF -j MASQUERADE $IPTABLES -A FORWARD -i $LANIF -o $WANIF -m state --state RELATED,ESTABLISHED -j ACCEPT $IPTABLES -A FORWARD -i $WANIF -o $LANIF -j ACCEPT $IPTABLES -t nat -A POSTROUTING -o $WANIF -j MASQUERADE $IPTABLES -A FORWARD -i $WANIF -o $LANIF -m state --state RELATED,ESTABLISHED -j ACCEPT $IPTABLES -A FORWARD -i $LANIF -o $WANIF -j ACCEPT echo 'Done.'
Provide full permissions permissions for the script:
chmod 777 internet_enable.shRun the script:
./internet_enable.sh
Note
You may need to add a proper nameserver entry in /etc/netplan/00-installer-config.yaml to ping the outside Internet. To get the
DNS Server name, use the following command:
aerial@iperf-cn-vm:~$ systemd-resolve --status | grep "DNS Servers"
DNS Servers: 10.110.8.18