PCIe Endpoint Mode on Jetson Thor#
Applies to the Jetson Thor series.
Use PCIe endpoint mode to connect a Jetson Thor endpoint (EP) to a root port (RP) system. This topic describes how to configure the endpoint, run the Tegra VNET sample to exchange network traffic, and run the DMA sample to test data transfers.
The procedures use two Jetson Thor systems running the same Jetson Linux release. For an x86 root port, refer to Using an x86 Root Port before configuring the endpoint.
Jetson Linux provides a PCIe controller driver, endpoint function drivers, and corresponding root port drivers. The Linux PCIe endpoint subsystem provides the configfs interface that you use to create a function and bind it to a controller.
The following source paths are relative to Linux_for_Tegra/source/. To obtain matching sources, refer to Obtaining the Kernel Sources.
Component |
Source path |
|---|---|
Thor endpoint controller driver |
|
Tegra VNET endpoint function driver |
|
Tegra VNET root port driver |
|
DMA endpoint function driver |
|
DMA root port driver |
|
DMA library API |
|
Hardware and Software Requirements#
You need the following hardware:
A Jetson Thor system on a carrier board that exposes the selected PCIe controller for endpoint use.
A second Jetson Thor system with a compatible root port connection.
PCIe cables and adapters that provide the required lane, reset, ground, and reference clock connections for both boards.
A Linux host to build the device tree and flash the endpoint.
Keep the root port system in its root port configuration. Select a controller and UPHY lane configuration that are available on your module and carrier board. Do not reconfigure a port that provides the endpoint’s boot storage unless you have configured another boot device.
The following table identifies the Thor endpoint controllers. It does not specify connector availability on a carrier board.
Controller |
Controller address |
BPMP device tree node |
UPHY endpoint property |
|---|---|---|---|
C2 |
|
|
|
C4 |
|
|
|
C5 |
|
|
|
The configuration examples use C4 with a common reference clock supplied by the root port. For C2 or C5, update the BPMP node, UPHY endpoint property, kernel device tree nodes, pin configuration, and controller address together. For separate reference clocks with independent spread spectrum (SRIS), configure both systems and the board’s clock routing as described in Enable PCIe in a Customer CVB Design.
Before you start:
Prepare the BSP and root file system for your board as described in Quick Start.
Use matching kernel, NVIDIA out-of-tree modules, device trees, and firmware from your release. The endpoint needs
pcie-tegra264-ep,tegra-pcie-dma-lib, and the function driver for the selected sample. The root port needs the matching sample driver.If you use a custom kernel, build and install its modules as described in Kernel Customization. Ensure that PCI endpoint support, endpoint
configfssupport, anddebugfsare enabled.Install
pciutilson the root port forlspci. Installiperf3on both systems if you plan to run the network throughput tests.
Configuring and Flashing the Endpoint#
Endpoint operation requires consistent BPMP, UPHY, kernel device tree, and pin configuration. Setting pcie-mode alone does not complete the configuration.
Run the following procedure on the flashing host. Replace <board> with the flash configuration for your module, carrier board, and boot storage. Use your board’s existing UPHY lane configuration as the starting point.
Open the board configuration file used by
Linux_for_Tegra/<board>.conf.Check the order of sourced configuration files so that a later assignment does not replace your
ODMDATAchanges.Configure C4 for endpoint operation with a common clock:
ODMDATA+=",pcie-c4-endpoint-enable,pcie@4_status=okay_clk-scheme=0_pcie-mode=2"
Add this line after the existing
ODMDATAassignment. Preserve the board’s lane and storage settings. If the configuration already contains settings forpcie@4, update those settings instead of adding conflicting entries. Thepcie-c4-endpoint-enableproperty selects the UPHY endpoint configuration;pcie-mode=2selects the BPMP controller mode.Update the board’s kernel device tree to disable the C4 root port node and enable the C4 endpoint node:
/ { bus@0 { pcie@a808460000 { status = "disabled"; }; pcie-ep@a808460000 { status = "okay"; }; }; };
Apply these overrides to the existing controller definitions. Preserve the controller’s register, interrupt, reset, IOMMU, and MSI properties. If your board uses a dynamic overlay to select endpoint mode, verify that it applies these changes for the selected
ODMDATA. Do not assume that every board supplies such an overlay.Verify that the endpoint node retains
nvidia,host1x = <&host1x>;for the Tegra VNET sample.Tegra VNET uses this phandle for notifications from the root port. The DMA sample does not require the VNET notification mechanism.
Configure the endpoint reset pin and reference clock routing for your carrier board.
Configure
PEX_L*_RST_Nas a GPIO input, and ensure that the endpoint node’sreset-gpiosand pin control properties reference the correct pin. Configure any clock buffer or clock selection GPIOs required by your board. For pin configuration details, refer to Enable PCIe in a Customer CVB Design.Build and install the updated device tree files in the BSP.
Follow the Building the DTBs procedure in Kernel Customization for kernel device trees. Include any updated pinmux and BCT files in the flash configuration. If you changed the BPMP device tree directly, also build and install the BPMP DTB selected by the board configuration; building kernel DTBs does not build the BPMP DTB.
Put the endpoint device into Force Recovery Mode.
From
Linux_for_Tegra/, flash the endpoint:Warning
Flashing can erase data on the target storage. Back up any data you need before flashing.
sudo ./l4t_initrd_flash.sh <board> internal
For board selection and storage options, refer to Flashing Support.
Keep a separate copy of the endpoint board configuration if you also flash root port systems. Changing the host-side configuration after flashing does not change the configuration already installed on the endpoint.
Connecting and Configuring the Devices#
Use the following procedure to run the Tegra VNET sample. Start from a boot in which no other endpoint function is bound to the selected controller.
Important
Run the configuration and test commands in a root shell on each target. Use sudo -i to open a root shell. Prefixing a command that uses shell redirection with sudo does not give the redirection root privileges.
With both systems powered off, connect the PCIe cable and adapters.
Boot the endpoint. Leave the root port system powered off until the endpoint function is ready.
On the endpoint, select the controller address. For C4, run:
PCIE_EP_ADDR=a808460000
Load the endpoint controller, DMA library, and VNET function drivers:
modprobe pcie-tegra264-ep modprobe tegra-pcie-dma-lib modprobe pci-epf-tegra-vnet
Check that the endpoint controller is available:
ls /sys/kernel/config/pci_ep/controllers/Confirm that
${PCIE_EP_ADDR}.pcie-epis present. Ifconfigfsis not mounted, mount it withmount -t configfs none /sys/kernel/config. If the controller is still missing, checkdmesgand the device tree used at boot. For C4,pcie@a808460000must be disabled andpcie-ep@a808460000must be enabled.Create the VNET endpoint function:
cd /sys/kernel/config/pci_ep mkdir functions/pci_epf_tvnet/func1
Configure the MSI interrupt count:
echo 16 > functions/pci_epf_tvnet/func1/msi_interrupts
The Thor driver supplies vendor ID
0x10de, device ID0x22d7, and the network device class. Keep these defaults so that the root port driver can identify the function.Bind the function to the controller:
ln -s /sys/kernel/config/pci_ep/functions/pci_epf_tvnet/func1 \ controllers/${PCIE_EP_ADDR}.pcie-ep/
Start the endpoint controller:
echo 1 > controllers/${PCIE_EP_ADDR}.pcie-ep/start
Boot the root port system.
On the root port, ensure that the DMA test driver is not loaded:
lsmod | grep tegra_pcie_dma_test
If it is loaded, unload it with
modprobe -r tegra-pcie-dma-testbefore loading VNET. The DMA test driver can match the same PCI device ID and must not claim the VNET device. If you have configured it to load automatically, remove that configuration for the VNET test.Load the VNET root port driver:
modprobe tegra-pcie-dma-lib modprobe tegra_vnet
Verify enumeration and driver binding:
lspci -Dnnk -d 10de:22d7
Confirm that the device is listed and
Kernel driver in useistvnet. If the device is missing, check the physical connection, reset and clock configuration, and boot order. If another driver owns the device, resolve the driver binding before configuring the network.
For details about configfs, refer to kernel/kernel-noble/Documentation/PCI/endpoint/pci-endpoint-cfs.rst in the source package.
Bringing up an Ethernet Interface over PCIe#
The interface names can differ between systems. The following example uses 192.168.2.0/24. If that subnet is already in use, choose another subnet and update both addresses and the test commands.
On each system, identify the interface created by VNET:
ip -brief link readlink -f /sys/class/net/<interface-name>/device/driver
Replace
<interface-name>with the interface you are checking. The driver path ends inpci_epf_tvneton the endpoint andtvneton the root port.In each system’s root shell, set its local interface name:
VNET_IF=<interface-name>
If a network manager controls this interface, configure it not to manage the interface during the manual test so that it does not replace the static address or start DHCP.
On each system, set the interface MTU to 1500 for the initial test:
ip link set dev "${VNET_IF}" mtu 1500
On each system, bring up the interface:
ip link set dev "${VNET_IF}" up
On the endpoint, assign its IP address:
ip address add 192.168.2.1/24 dev "${VNET_IF}"
On the root port, assign its IP address:
ip address add 192.168.2.2/24 dev "${VNET_IF}"
These settings apply to the current boot. Keep both interfaces at the same MTU when changing the test configuration.
Testing Procedures#
Prepare for Testing#
On each system, check the address, link state, and counters:
ip -brief address show dev "${VNET_IF}"
ip -s link show dev "${VNET_IF}"
Confirm that both interfaces are up and have different addresses in the same subnet. Allow ICMP and the iperf3 test traffic through the local firewall. Use a separate console to monitor dmesg -w on each system during the tests.
Execution#
On the endpoint, test connectivity to the root port:
ping -I "${VNET_IF}" -c 4 192.168.2.2
On the root port, test connectivity to the endpoint:
ping -I "${VNET_IF}" -c 4 192.168.2.1
On the root port, start the throughput test server:
iperf3 -s -B 192.168.2.2
On the endpoint, test TCP traffic from the endpoint to the root port:
iperf3 -c 192.168.2.2 -B 192.168.2.1 -t 30
On the endpoint, test TCP traffic in the reverse direction:
iperf3 -c 192.168.2.2 -B 192.168.2.1 -R -t 30
On the endpoint, run an initial UDP test at 100 Mbps:
iperf3 -c 192.168.2.2 -B 192.168.2.1 -u -b 100M -t 30
Successful ping replies verify IP connectivity. Record TCP throughput and UDP loss and jitter, and check both systems for driver errors and increasing error counters. Throughput depends on the negotiated PCIe speed and width, MTU, system configuration, and test parameters; these commands do not specify a minimum performance requirement.
Running the DMA Sample#
The DMA sample uses kernel drivers and debugfs rather than a userspace test executable. Run it separately from VNET. The following procedure uses a Thor root port and a Thor endpoint with the same controller configuration used for VNET.
Shut down both systems after completing any VNET tests.
Boot the endpoint without creating the VNET function.
In an endpoint root shell, select the controller and load the DMA sample:
PCIE_EP_ADDR=a808460000 modprobe pcie-tegra264-ep modprobe tegra-pcie-dma-lib modprobe pci-epf-dma-test
Check the endpoint controller directory:
ls /sys/kernel/config/pci_ep/controllers/Mount
configfsfirst if needed, as described in the VNET procedure. Confirm that${PCIE_EP_ADDR}.pcie-epis present before continuing.Create the DMA endpoint function:
cd /sys/kernel/config/pci_ep mkdir functions/tegra_pcie_dma_epf/func1
The Thor DMA sample supplies device ID
0x22d7and a memory device class. Do not bind the VNET function to this controller at the same time.Configure the MSI interrupt count:
echo 16 > functions/tegra_pcie_dma_epf/func1/msi_interrupts
Bind the function to the controller:
ln -s /sys/kernel/config/pci_ep/functions/tegra_pcie_dma_epf/func1 \ controllers/${PCIE_EP_ADDR}.pcie-ep/
Start the endpoint controller:
echo 1 > controllers/${PCIE_EP_ADDR}.pcie-ep/start
Boot the root port system.
In a root port root shell, load the DMA sample driver:
modprobe tegra-pcie-dma-lib modprobe tegra-pcie-dma-test
Verify that the DMA driver owns the endpoint device:
lspci -Dnnk -d 10de:22d7
Confirm that
Kernel driver in useispcie_dma_host. Wait for the driver to initialize the shared buffers before starting a transfer. The endpoint reportsRP DMA address is nullif this initialization has not completed.On the endpoint, mount
debugfsif it is not already mounted:mountpoint -q /sys/kernel/debug || mount -t debugfs none /sys/kernel/debug
Select the endpoint sample’s debug directory:
cd /sys/kernel/debug/${PCIE_EP_ADDR}.pcie-ep_epf_dma_test
Configure a small asynchronous write test on channel 0:
echo 4096 > dma_size echo 4 > nents echo 10 > stress_count echo 0x11 > edma_ch
dma_sizeis the number of bytes per descriptor,nentsis the descriptor count, andstress_countis the iteration count. Inedma_ch, bits 0 through 3 select asynchronous operation and bits 4 through 7 enable the corresponding channels.Start the test:
cat edmalib_testCheck
dmesgfor asynchronous completion with status0. A submission message alone does not establish that the transfer completed. Check for DMA timeouts, SMMU faults, and PCIe errors on both systems.After the write test completes, run the read test:
echo 0x08000011 > edma_ch cat edmalib_test
Bit 27 selects DMA reads from root port memory to the endpoint.
After the read test completes, run the CRC test:
echo 0x100000f0 > edma_ch cat edmalib_test
The CRC mode selects four synchronous channels and transfers 4 KiB per channel for one iteration. Check for
CRC check passin the endpoint kernel log.Release the sample’s DMA resources after testing:
echo 0 > stress_count cat edmalib_test
For the test flags and transfer limits, refer to nvidia-oot/include/linux/tegra-pcie-edma-test-common.h in your release’s source package. Do not run endpoint and root port DMA tests concurrently on the same DMA engine.
Using an x86 Root Port#
An x86 root port requires a compatible host driver built for its architecture and running kernel. You cannot load a Jetson Arm64 module on x86. Verify that the host driver’s PCI ID and class table, shared data structures, and DMA library match the endpoint driver. The preceding DMA procedure assumes a Thor root port; it does not provide an x86 DMA driver build procedure.
When connecting a Thor endpoint to a non-Tegra root port, configure the endpoint’s SMMU stream IDs for that host’s requester ID (RID). Preserve the controller’s default stream ID and add the stream ID for the root port. Do not replace the default ID or change the XDMA stream ID to the root port RID, because the endpoint’s local DMA interrupt path uses the default ID.
For example, a root port at 0000:64:00.0 has RID 0x6400. For a C4 endpoint, the corresponding property is:
iommus = <&smmu1_mmu 0x40000>, <&smmu1_mmu 0x46400>;
This example is specific to that root port. Identify the root port from the host’s PCIe topology, rather than using the endpoint’s own bus, device, and function number. Recheck the mapping when changing the host, slot, or topology. Apply the device tree change and reflash before loading a host driver that accesses the endpoint’s BARs.
Troubleshooting#
Symptom |
Checks |
|---|---|
The endpoint controller is missing from |
Check the controller module, probe messages, and the device tree used at boot. Confirm that the RP node is disabled and the EP node is enabled. |
The root port does not enumerate the endpoint. |
Check that the endpoint function was started before the root port booted. Check PERST#, REFCLK, lane routing, and the selected UPHY configuration. |
The PCI device is present, but the expected driver is not bound. |
Check the PCI ID and class, module version, and |
VNET interfaces exist, but traffic fails. |
Check IP addresses, link state, MTU, routes, and firewall rules. Check the |
BAR accesses fail or SMMU faults occur with an x86 root port. |
Check the default and host-specific stream IDs against the actual root port topology. Refer to Using an x86 Root Port. |
For PCIe link diagnostics, refer to Configure the PCIe Controller.