Enable 25 Gigabit Ethernet on QSFP Port#
In the Jetson AGX Thor Developer Kit, the QSFP port can support both 10 Gigabit Ethernet (10GbE) and 25 Gigabit Ethernet (25GbE). The default is 10GbE.
The QSFP port can support only one configuration at a time. To enable 25GbE, edit the flashing configuration file jetson-agx-thor-devkit.conf and set ODMDATA as follows:
ODMDATA="uphy1-config-8,mgbe0-speed-3,mgbe1-speed-3,mgbe2-speed-3,mgbe3-speed-3";
For the Jetson Thor T4000 module, which has three MGBE ports, edit jetson-agx-thor-t4000.conf and set ODMDATA as follows:
ODMDATA="uphy1-config-8,mgbe0-speed-3,mgbe1-speed-3,mgbe2-speed-3";
Then reflash the device.
How to Optimize the Performance of 25GbE#
Run the following commands to enable NAPI for 25GbE on each port.
echo 1 > /sys/devices/platform/bus@0/a808a10000.ethernet/net/mgbe0_0/threaded
echo 1 > /sys/devices/platform/bus@0/a808e10000.ethernet/net/mgbe3_0/threaded
echo 1 > /sys/devices/platform/bus@0/a808b10000.ethernet/net/mgbe1_0/threaded
echo 1 > /sys/devices/platform/bus@0/a808d10000.ethernet/net/mgbe2_0/threaded
Optimizing Performance of 25 Gigabit Ethernet on QSFP Port#
It is possible to achieve up to 70 Gbps aggregate network throughput using the 4 QSFP ports available in the Jetson AGX Thor Developer Kit.
Connect two Jetson AGX Thor Developer Kits to each other back-to-back through the QSFP port. Make sure that it is a 4X25Gbps cable.
For testing 4×10Gbps, the QSFP cable used was:
6COMGIGA 40GBASE-SR4 QSFP+ 850nm 150m DDM MMF Transceiver, 40 Gigabit QSFP+ MPO Multi-Mode Module Compatible with Cisco QSFP-40G-SR4
https://www.amazon.com/dp/B0CXXC9LYV?ref_=pe_153780060_1227138240&th=1
For testing 4×25Gbps, the QSFP cable used was:
MCP1600 QSFP28 Copper Cable
Perform these steps after the Linux kernel boots up.
Set up 4 MGBE network interfaces with IP addresses and 9K MTU.
sudo ifconfig mgbe0_0 down mtu 9000 up sudo ifconfig mgbe1_0 down mtu 9000 up sudo ifconfig mgbe2_0 down mtu 9000 up sudo ifconfig mgbe3_0 down mtu 9000 up
Enable Linux kernel threaded NAPI mode.
sudo sh -c "echo 1 > /sys/devices/platform/bus@0/a808a10000.ethernet/net/mgbe0_0/threaded" sudo sh -c "echo 1 > /sys/devices/platform/bus@0/a808b10000.ethernet/net/mgbe1_0/threaded" sudo sh -c "echo 1 > /sys/devices/platform/bus@0/a808d10000.ethernet/net/mgbe2_0/threaded" sudo sh -c "echo 1 > /sys/devices/platform/bus@0/a808e10000.ethernet/net/mgbe3_0/threaded"
Boost clocks on the Jetson AGX Thor CPU.
sudo jetson_clocksEnable deferred DMA unmap for MGBE.
# MGBE0_0 GROUP=$(basename $(readlink /sys/class/net/mgbe0_0/device/iommu_group)) ls /sys/kernel/iommu_groups/$GROUP/devices echo DMA-FQ | sudo tee /sys/kernel/iommu_groups/$GROUP/type echo GROUP=$GROUP cat /sys/kernel/iommu_groups/$GROUP/type # MGBE1_0 GROUP=$(basename $(readlink /sys/class/net/mgbe1_0/device/iommu_group)) ls /sys/kernel/iommu_groups/$GROUP/devices echo DMA-FQ | sudo tee /sys/kernel/iommu_groups/$GROUP/type echo GROUP=$GROUP cat /sys/kernel/iommu_groups/$GROUP/type # MGBE2_0 GROUP=$(basename $(readlink /sys/class/net/mgbe2_0/device/iommu_group)) ls /sys/kernel/iommu_groups/$GROUP/devices echo DMA-FQ | sudo tee /sys/kernel/iommu_groups/$GROUP/type echo GROUP=$GROUP cat /sys/kernel/iommu_groups/$GROUP/type # MGBE3_0 GROUP=$(basename $(readlink /sys/class/net/mgbe3_0/device/iommu_group)) ls /sys/kernel/iommu_groups/$GROUP/devices echo DMA-FQ | sudo tee /sys/kernel/iommu_groups/$GROUP/type echo GROUP=$GROUP cat /sys/kernel/iommu_groups/$GROUP/type
If dynamic domain update does not work, unbind the driver, set the IOMMU group type, and bind the driver again.
Run the network receiver application on the 4 MGBE network interfaces. Replace
192.168.100.10with the IP address of the Jetson board that receives data.# Use -S on the iperf3 client to send using different TX DMA channels. sudo killall iperf3 iperf3 -s -B 192.168.100.10 -p 10010 & iperf3 -s -B 192.168.100.10 -p 10011 & iperf3 -s -B 192.168.100.10 -p 10012 & iperf3 -s -B 192.168.100.10 -p 10013 & iperf3 -s -B 192.168.100.10 -p 10014 & iperf3 -s -B 192.168.100.10 -p 10015 & iperf3 -s -B 192.168.100.10 -p 10016 & iperf3 -s -B 192.168.100.10 -p 10017 & iperf3 -s -B 192.168.101.10 -p 10110 & iperf3 -s -B 192.168.101.10 -p 10111 & iperf3 -s -B 192.168.101.10 -p 10112 & iperf3 -s -B 192.168.101.10 -p 10113 & iperf3 -s -B 192.168.101.10 -p 10114 & iperf3 -s -B 192.168.101.10 -p 10115 & iperf3 -s -B 192.168.101.10 -p 10116 & iperf3 -s -B 192.168.101.10 -p 10117 & iperf3 -s -B 192.168.102.10 -p 10210 & iperf3 -s -B 192.168.102.10 -p 10211 & iperf3 -s -B 192.168.102.10 -p 10212 & iperf3 -s -B 192.168.102.10 -p 10213 & iperf3 -s -B 192.168.102.10 -p 10214 & iperf3 -s -B 192.168.102.10 -p 10215 & iperf3 -s -B 192.168.102.10 -p 10216 & iperf3 -s -B 192.168.102.10 -p 10217 & iperf3 -s -B 192.168.103.10 -p 10310 & iperf3 -s -B 192.168.103.10 -p 10311 & iperf3 -s -B 192.168.103.10 -p 10312 & iperf3 -s -B 192.168.103.10 -p 10313 & iperf3 -s -B 192.168.103.10 -p 10314 & iperf3 -s -B 192.168.103.10 -p 10315 & iperf3 -s -B 192.168.103.10 -p 10316 & iperf3 -s -B 192.168.103.10 -p 10317 &
Run the network transmitter application on the 4 MGBE network interfaces. Use multiple DMA channels while transmitting, and use multiple sockets for each MGBE network interface and TX DMA channel combination. Replace
192.168.100.10with the IP address of the Jetson board that receives data.Create a shell script named
iperf3_tx.shwith the following commands:#!/bin/bash # iperf3 client (TCP TX) # Use -S to send over multiple TX DMA channels. # MGBE0_0 iperf3 -c 192.168.100.10 -p 10010 -i 1 -t 10 -S 0 $1 $2 $3 $4 $5 $6 $7 $8 $9 & iperf3 -c 192.168.100.10 -p 10011 -i 1 -t 10 -S 32 $1 $2 $3 $4 $5 $6 $7 $8 $9 & iperf3 -c 192.168.100.10 -p 10012 -i 1 -t 10 -S 64 $1 $2 $3 $4 $5 $6 $7 $8 $9 & iperf3 -c 192.168.100.10 -p 10013 -i 1 -t 10 -S 96 $1 $2 $3 $4 $5 $6 $7 $8 $9 & iperf3 -c 192.168.100.10 -p 10014 -i 1 -t 10 -S 128 $1 $2 $3 $4 $5 $6 $7 $8 $9 & iperf3 -c 192.168.100.10 -p 10015 -i 1 -t 10 -S 160 $1 $2 $3 $4 $5 $6 $7 $8 $9 & iperf3 -c 192.168.100.10 -p 10016 -i 1 -t 10 -S 192 $1 $2 $3 $4 $5 $6 $7 $8 $9 & iperf3 -c 192.168.100.10 -p 10017 -i 1 -t 10 -S 224 $1 $2 $3 $4 $5 $6 $7 $8 $9 & # MGBE1_0 iperf3 -c 192.168.101.10 -p 10110 -i 1 -t 10 -S 0 $1 $2 $3 $4 $5 $6 $7 $8 $9 & iperf3 -c 192.168.101.10 -p 10111 -i 1 -t 10 -S 32 $1 $2 $3 $4 $5 $6 $7 $8 $9 & iperf3 -c 192.168.101.10 -p 10112 -i 1 -t 10 -S 64 $1 $2 $3 $4 $5 $6 $7 $8 $9 & iperf3 -c 192.168.101.10 -p 10113 -i 1 -t 10 -S 96 $1 $2 $3 $4 $5 $6 $7 $8 $9 & iperf3 -c 192.168.101.10 -p 10114 -i 1 -t 10 -S 128 $1 $2 $3 $4 $5 $6 $7 $8 $9 & iperf3 -c 192.168.101.10 -p 10115 -i 1 -t 10 -S 160 $1 $2 $3 $4 $5 $6 $7 $8 $9 & iperf3 -c 192.168.101.10 -p 10116 -i 1 -t 10 -S 192 $1 $2 $3 $4 $5 $6 $7 $8 $9 & iperf3 -c 192.168.101.10 -p 10117 -i 1 -t 10 -S 224 $1 $2 $3 $4 $5 $6 $7 $8 $9 & # MGBE2_0 iperf3 -c 192.168.102.10 -p 10210 -i 1 -t 10 -S 0 $1 $2 $3 $4 $5 $6 $7 $8 $9 & iperf3 -c 192.168.102.10 -p 10211 -i 1 -t 10 -S 32 $1 $2 $3 $4 $5 $6 $7 $8 $9 & iperf3 -c 192.168.102.10 -p 10212 -i 1 -t 10 -S 64 $1 $2 $3 $4 $5 $6 $7 $8 $9 & iperf3 -c 192.168.102.10 -p 10213 -i 1 -t 10 -S 96 $1 $2 $3 $4 $5 $6 $7 $8 $9 & iperf3 -c 192.168.102.10 -p 10214 -i 1 -t 10 -S 128 $1 $2 $3 $4 $5 $6 $7 $8 $9 & iperf3 -c 192.168.102.10 -p 10215 -i 1 -t 10 -S 160 $1 $2 $3 $4 $5 $6 $7 $8 $9 & iperf3 -c 192.168.102.10 -p 10216 -i 1 -t 10 -S 192 $1 $2 $3 $4 $5 $6 $7 $8 $9 & iperf3 -c 192.168.102.10 -p 10217 -i 1 -t 10 -S 224 $1 $2 $3 $4 $5 $6 $7 $8 $9 & # MGBE3_0 iperf3 -c 192.168.103.10 -p 10310 -i 1 -t 10 -S 0 $1 $2 $3 $4 $5 $6 $7 $8 $9 & iperf3 -c 192.168.103.10 -p 10311 -i 1 -t 10 -S 32 $1 $2 $3 $4 $5 $6 $7 $8 $9 & iperf3 -c 192.168.103.10 -p 10312 -i 1 -t 10 -S 64 $1 $2 $3 $4 $5 $6 $7 $8 $9 & iperf3 -c 192.168.103.10 -p 10313 -i 1 -t 10 -S 96 $1 $2 $3 $4 $5 $6 $7 $8 $9 & iperf3 -c 192.168.103.10 -p 10314 -i 1 -t 10 -S 128 $1 $2 $3 $4 $5 $6 $7 $8 $9 & iperf3 -c 192.168.103.10 -p 10315 -i 1 -t 10 -S 160 $1 $2 $3 $4 $5 $6 $7 $8 $9 & iperf3 -c 192.168.103.10 -p 10316 -i 1 -t 10 -S 192 $1 $2 $3 $4 $5 $6 $7 $8 $9 & iperf3 -c 192.168.103.10 -p 10317 -i 1 -t 10 -S 224 $1 $2 $3 $4 $5 $6 $7 $8 $9 &
For approximately 70 Gbps total TCP throughput, run:
./iperf3_tx.sh -b 0 -P 24 | grep send | grep SUM
For approximately 70 Gbps total UDP throughput, run:
./iperf3_tx.sh -u -b 0 -P 24 | grep send | grep SUM
Verifying 4×25 Gbps Aggregate Throughput on the QSFP Port#
Prerequisite: Follow the instructions in Optimizing Performance of 25 Gigabit Ethernet on QSFP Port.
UDP (4×25Gbps, 9K MTU)#
For UDP, with 4×25Gbps network interfaces set to 9K MTU, run the following command:
./iperf3_tx.sh -u -b 0 -P 24 | grep send | grep SUM
The approximate expected aggregate throughput is 70 Gbps, using 4 MGBE interfaces, 8 IP TOS traffic classes, and 24 sockets per MGBE/IP TOS combination, for a total of 768 sockets.
Add the throughput values for all 32 sender lines, converting any Mbits/sec values to Gbits/sec before adding them. The total should be approximately 70 Gbps.
TCP (4×25 Gbps, 9K MTU)#
For TCP, with 4×25Gbps network interfaces set to 9K MTU, run the following command:
./iperf3_tx.sh -b 0 -P 24 | grep send | grep SUM
The approximate expected aggregate throughput is 70 Gbps, using 4 MGBE interfaces, 8 IP TOS traffic classes, and 24 sockets per MGBE/IP TOS combination, for a total of 768 sockets.
Add the throughput values for all 32 sender lines, converting any Mbits/sec values to Gbits/sec before adding them. The total should be approximately 70 Gbps.