PCIe Endpoint Mode#
Applies to the Jetson Orin series.
For Jetson Thor, refer to PCIe Endpoint Mode on Jetson Thor.
Use Tegra VNET to establish an Ethernet interface over PCIe between a Jetson Orin endpoint (EP) and a root port (RP) system running Jetson Linux 39.2.2. You can assign IP addresses to these interfaces and exchange network traffic over the PCIe link.
Jetson Linux provides the following software components:
A PCIe endpoint controller driver that configures the controller to operate as an endpoint.
The Tegra VNET endpoint function driver,
pci-epf-tegra-vnet, which exposes a virtual network device through the endpoint.The root port driver,
tegra_vnet, which provides the corresponding network interface on the root port system.The Linux PCIe endpoint subsystem, which provides the
configfsinterface for creating an endpoint function and binding it to a controller.
The following source paths are relative to Linux_for_Tegra/source/. To obtain the sources for your release, refer to Obtaining the Kernel Sources.
Component |
Source path |
|---|---|
Orin PCIe controller driver |
|
Tegra VNET endpoint function driver |
|
Tegra VNET root port driver |
|
PCIe endpoint subsystem |
|
Hardware Requirements#
You need the following hardware:
A Jetson AGX Orin, Jetson Orin NX, or Jetson Orin Nano device to act as the endpoint.
A second Jetson Orin device to act as the root port.
A PCIe cable and adapters appropriate for the connectors on both carrier boards.
A Linux host to flash the endpoint device.
These instructions use developer kits running the same Jetson Linux release. The endpoint uses PCIe C5 on Jetson AGX Orin or PCIe C4 on Jetson Orin NX/Nano. Keep the root port system in its root port configuration.
Before flashing, prepare the release package and root file system and connect the endpoint device to the flashing host in Force Recovery Mode, as described in Quick Start.
Warning
The following flash commands erase existing data on the target storage. Back up any data you need before flashing.
Flashing PCIe as Endpoint on a Jetson AGX Orin Series System#
Run this procedure on the flashing host from the Linux_for_Tegra/ directory. The command uses the developer kit’s eMMC storage.
Open
jetson-agx-orin-devkit.confin a text editor.Add the following override at the end of the file to select
nvhs-uphy-config-1:ODMDATA="gbe-uphy-config-22,nvhs-uphy-config-1,hsio-uphy-config-0,gbe0-enable-10g,hsstp-lane-map-3";
Flash the endpoint device:
sudo ./l4t_initrd_flash.sh --erase-all jetson-agx-orin-devkit internal
Remove the override you added to
jetson-agx-orin-devkit.conf.This restores the configuration for subsequent root port flashes. It does not change the endpoint configuration already flashed to the device.
Flashing PCIe as Endpoint on a Jetson Orin NX/Nano Series System#
Run this procedure on the flashing host from the Linux_for_Tegra/ directory. The command uses NVMe storage on the developer kit.
Open
jetson-orin-nano-devkit.confin a text editor.Add the following override at the end of the file to select
hsio-uphy-config-41:ODMDATA="gbe-uphy-config-8,hsstp-lane-map-3,hsio-uphy-config-41";
Flash the endpoint device:
sudo ./l4t_initrd_flash.sh --erase-all jetson-orin-nano-devkit internal
Remove the override you added to
jetson-orin-nano-devkit.conf.This restores the configuration for subsequent root port flashes. It does not change the endpoint configuration already flashed to the device.
Connecting and Configuring the Devices#
Before you start, ensure that pci-epf-tegra-vnet is available on the endpoint and tegra_vnet is available on the root port system. If you use a custom kernel, build and install the NVIDIA out-of-tree modules for that kernel as described in Kernel Customization.
Important
Run the following configuration and test commands in a root shell on each device. Use sudo -i to open a root shell. Prefixing a command that uses shell redirection with sudo does not give the redirection root privileges.
With both devices powered off, connect them using the PCIe cable and adapters.
Boot the endpoint device. Leave the root port system powered off until you have enabled the endpoint function.
On the endpoint, select the controller address for your device. Run only the command for your platform.
For Jetson AGX Orin, use PCIe C5:
PCIE_EP_ADDR=141a0000
For Jetson Orin NX/Nano, use PCIe C4:
PCIE_EP_ADDR=14160000
Load the endpoint function driver:
modprobe pci-epf-tegra-vnetCheck the endpoint controller directory:
ls /sys/kernel/config/pci_ep/controllers/Confirm that
<controller-address>.pcie-epis present, where<controller-address>is the value you selected forPCIE_EP_ADDR. Ifconfigfsis not mounted, mount it withmount -t configfs none /sys/kernel/configand check again. If the controller is still missing, verify the endpoint flash configuration before continuing.Create a Tegra VNET endpoint function:
cd /sys/kernel/config/pci_ep mkdir functions/pci_epf_tvnet/func1
Configure the function’s MSI interrupt count:
echo 16 > functions/pci_epf_tvnet/func1/msi_interrupts
The driver supplies the NVIDIA vendor ID
0x10deand the network device ID0x2296.Bind the function to the selected controller:
ln -s /sys/kernel/config/pci_ep/functions/pci_epf_tvnet/func1 \ controllers/${PCIE_EP_ADDR}.pcie-ep/
Start the endpoint controller:
echo 1 > controllers/${PCIE_EP_ADDR}.pcie-ep/start
Boot the root port system.
On the root port system, load the network driver:
modprobe tegra_vnetVerify that the root port system enumerated the Tegra VNET device:
lspci -nn -d 10de:2296
The device listing must contain
[10de:2296]. If no device is listed, check the PCIe connection, endpoint configuration, and boot order before configuring the network interfaces.
For details about the endpoint configuration interface, refer to kernel/kernel-noble/Documentation/PCI/endpoint/pci-endpoint-cfs.rst in the release source package.
Bringing up an Ethernet Interface over PCIe#
Use the following procedure after both drivers are loaded. The interface name can differ between the endpoint and the root port system; do not assume that either interface is named eth1.
On each device, identify the network interface created by the Tegra VNET driver:
ip -brief link
To check an interface’s driver, replace
<interface-name>with its name and run:readlink -f /sys/class/net/<interface-name>/device/driver
The driver path ends in
pci_epf_tvneton the endpoint andtvneton the root port.In each device’s root shell, set
VNET_IFto its local Tegra VNET interface name:VNET_IF=<interface-name>
Replace
<interface-name>with the interface name on that device. If NetworkManager or another network manager controls the interface, configure it not to manage this interface during the manual test so that it does not replace the static address or start DHCP.Bring up the endpoint interface:
ip link set dev "${VNET_IF}" up
Bring up the root port interface:
ip link set dev "${VNET_IF}" up
Assign an address to the endpoint interface:
ip address add 192.168.2.1/24 dev "${VNET_IF}"
Assign an address to the root port interface:
ip address add 192.168.2.2/24 dev "${VNET_IF}"
The example uses 192.168.2.0/24. If that subnet is already in use on either system, choose an unused subnet and adjust both addresses and the test commands. These interface settings apply to the current boot.
Testing Procedures#
Prepare for Testing#
On each device, check the address and link state of its Tegra VNET interface:
ip -brief address show dev "${VNET_IF}"
ip -s link show dev "${VNET_IF}"
Confirm that both interfaces are up and have different IP addresses in the same subnet.
Execution#
From the endpoint, test connectivity to the root port:
ping -I "${VNET_IF}" -c 4 192.168.2.2
From the root port, test connectivity to the endpoint:
ping -I "${VNET_IF}" -c 4 192.168.2.1
Successful replies in both directions verify IP connectivity over the Tegra VNET interfaces. If the PCIe device is enumerated but traffic fails, check that both network interfaces are up, the addresses are correct, and the local firewall permits ICMP. Check dmesg on both devices for driver errors.