PCIe Endpoint Mode#

Applies to the Jetson Orin series.

For Jetson Thor, refer to PCIe Endpoint Mode on Jetson Thor.

Use Tegra VNET to establish an Ethernet interface over PCIe between a Jetson Orin endpoint (EP) and a root port (RP) system running Jetson Linux 39.2.2. You can assign IP addresses to these interfaces and exchange network traffic over the PCIe link.

Jetson Linux provides the following software components:

  • A PCIe endpoint controller driver that configures the controller to operate as an endpoint.

  • The Tegra VNET endpoint function driver, pci-epf-tegra-vnet, which exposes a virtual network device through the endpoint.

  • The root port driver, tegra_vnet, which provides the corresponding network interface on the root port system.

  • The Linux PCIe endpoint subsystem, which provides the configfs interface for creating an endpoint function and binding it to a controller.

The following source paths are relative to Linux_for_Tegra/source/. To obtain the sources for your release, refer to Obtaining the Kernel Sources.

Component

Source path

Orin PCIe controller driver

kernel/kernel-noble/drivers/pci/controller/dwc/pcie-tegra194.c

Tegra VNET endpoint function driver

nvidia-oot/drivers/pci/endpoint/functions/pci-epf-tegra-vnet.c

Tegra VNET root port driver

nvidia-oot/drivers/net/ethernet/nvidia/pcie/tegra_vnet.c

PCIe endpoint subsystem

kernel/kernel-noble/drivers/pci/endpoint/

Hardware Requirements#

You need the following hardware:

  • A Jetson AGX Orin, Jetson Orin NX, or Jetson Orin Nano device to act as the endpoint.

  • A second Jetson Orin device to act as the root port.

  • A PCIe cable and adapters appropriate for the connectors on both carrier boards.

  • A Linux host to flash the endpoint device.

These instructions use developer kits running the same Jetson Linux release. The endpoint uses PCIe C5 on Jetson AGX Orin or PCIe C4 on Jetson Orin NX/Nano. Keep the root port system in its root port configuration.

Before flashing, prepare the release package and root file system and connect the endpoint device to the flashing host in Force Recovery Mode, as described in Quick Start.

Warning

The following flash commands erase existing data on the target storage. Back up any data you need before flashing.

Flashing PCIe as Endpoint on a Jetson AGX Orin Series System#

Run this procedure on the flashing host from the Linux_for_Tegra/ directory. The command uses the developer kit’s eMMC storage.

  1. Open jetson-agx-orin-devkit.conf in a text editor.

  2. Add the following override at the end of the file to select nvhs-uphy-config-1:

    ODMDATA="gbe-uphy-config-22,nvhs-uphy-config-1,hsio-uphy-config-0,gbe0-enable-10g,hsstp-lane-map-3";
    
  3. Flash the endpoint device:

    sudo ./l4t_initrd_flash.sh --erase-all jetson-agx-orin-devkit internal
    
  4. Remove the override you added to jetson-agx-orin-devkit.conf.

    This restores the configuration for subsequent root port flashes. It does not change the endpoint configuration already flashed to the device.

Flashing PCIe as Endpoint on a Jetson Orin NX/Nano Series System#

Run this procedure on the flashing host from the Linux_for_Tegra/ directory. The command uses NVMe storage on the developer kit.

  1. Open jetson-orin-nano-devkit.conf in a text editor.

  2. Add the following override at the end of the file to select hsio-uphy-config-41:

    ODMDATA="gbe-uphy-config-8,hsstp-lane-map-3,hsio-uphy-config-41";
    
  3. Flash the endpoint device:

    sudo ./l4t_initrd_flash.sh --erase-all jetson-orin-nano-devkit internal
    
  4. Remove the override you added to jetson-orin-nano-devkit.conf.

    This restores the configuration for subsequent root port flashes. It does not change the endpoint configuration already flashed to the device.

Connecting and Configuring the Devices#

Before you start, ensure that pci-epf-tegra-vnet is available on the endpoint and tegra_vnet is available on the root port system. If you use a custom kernel, build and install the NVIDIA out-of-tree modules for that kernel as described in Kernel Customization.

Important

Run the following configuration and test commands in a root shell on each device. Use sudo -i to open a root shell. Prefixing a command that uses shell redirection with sudo does not give the redirection root privileges.

  1. With both devices powered off, connect them using the PCIe cable and adapters.

  2. Boot the endpoint device. Leave the root port system powered off until you have enabled the endpoint function.

  3. On the endpoint, select the controller address for your device. Run only the command for your platform.

    For Jetson AGX Orin, use PCIe C5:

    PCIE_EP_ADDR=141a0000
    

    For Jetson Orin NX/Nano, use PCIe C4:

    PCIE_EP_ADDR=14160000
    
  4. Load the endpoint function driver:

    modprobe pci-epf-tegra-vnet
    
  5. Check the endpoint controller directory:

    ls /sys/kernel/config/pci_ep/controllers/
    

    Confirm that <controller-address>.pcie-ep is present, where <controller-address> is the value you selected for PCIE_EP_ADDR. If configfs is not mounted, mount it with mount -t configfs none /sys/kernel/config and check again. If the controller is still missing, verify the endpoint flash configuration before continuing.

  6. Create a Tegra VNET endpoint function:

    cd /sys/kernel/config/pci_ep
    mkdir functions/pci_epf_tvnet/func1
    
  7. Configure the function’s MSI interrupt count:

    echo 16 > functions/pci_epf_tvnet/func1/msi_interrupts
    

    The driver supplies the NVIDIA vendor ID 0x10de and the network device ID 0x2296.

  8. Bind the function to the selected controller:

    ln -s /sys/kernel/config/pci_ep/functions/pci_epf_tvnet/func1 \
      controllers/${PCIE_EP_ADDR}.pcie-ep/
    
  9. Start the endpoint controller:

    echo 1 > controllers/${PCIE_EP_ADDR}.pcie-ep/start
    
  10. Boot the root port system.

  11. On the root port system, load the network driver:

    modprobe tegra_vnet
    
  12. Verify that the root port system enumerated the Tegra VNET device:

    lspci -nn -d 10de:2296
    

    The device listing must contain [10de:2296]. If no device is listed, check the PCIe connection, endpoint configuration, and boot order before configuring the network interfaces.

For details about the endpoint configuration interface, refer to kernel/kernel-noble/Documentation/PCI/endpoint/pci-endpoint-cfs.rst in the release source package.

Bringing up an Ethernet Interface over PCIe#

Use the following procedure after both drivers are loaded. The interface name can differ between the endpoint and the root port system; do not assume that either interface is named eth1.

  1. On each device, identify the network interface created by the Tegra VNET driver:

    ip -brief link
    

    To check an interface’s driver, replace <interface-name> with its name and run:

    readlink -f /sys/class/net/<interface-name>/device/driver
    

    The driver path ends in pci_epf_tvnet on the endpoint and tvnet on the root port.

  2. In each device’s root shell, set VNET_IF to its local Tegra VNET interface name:

    VNET_IF=<interface-name>
    

    Replace <interface-name> with the interface name on that device. If NetworkManager or another network manager controls the interface, configure it not to manage this interface during the manual test so that it does not replace the static address or start DHCP.

  3. Bring up the endpoint interface:

    ip link set dev "${VNET_IF}" up
    
  4. Bring up the root port interface:

    ip link set dev "${VNET_IF}" up
    
  5. Assign an address to the endpoint interface:

    ip address add 192.168.2.1/24 dev "${VNET_IF}"
    
  6. Assign an address to the root port interface:

    ip address add 192.168.2.2/24 dev "${VNET_IF}"
    

The example uses 192.168.2.0/24. If that subnet is already in use on either system, choose an unused subnet and adjust both addresses and the test commands. These interface settings apply to the current boot.

Testing Procedures#

Prepare for Testing#

On each device, check the address and link state of its Tegra VNET interface:

ip -brief address show dev "${VNET_IF}"
ip -s link show dev "${VNET_IF}"

Confirm that both interfaces are up and have different IP addresses in the same subnet.

Execution#

  1. From the endpoint, test connectivity to the root port:

    ping -I "${VNET_IF}" -c 4 192.168.2.2
    
  2. From the root port, test connectivity to the endpoint:

    ping -I "${VNET_IF}" -c 4 192.168.2.1
    

Successful replies in both directions verify IP connectivity over the Tegra VNET interfaces. If the PCIe device is enumerated but traffic fails, check that both network interfaces are up, the addresses are correct, and the local firewall permits ICMP. Check dmesg on both devices for driver errors.