PCIe Endpoint Mode¶

Applies to the Jetson Xavier NX or the Jetson AGX Xavier and the Jetson AGX Orin series.

Jetson Linux contains the following software support for PCIe endpoint mode:

A Linux kernel device driver for the PCIe endpoint controller.

This driver configures the PCIe controller as an endpoint, and provides an interface for higher-level software to select the configuration that is exposed to the PCIe bus.

The source code for this driver is available at the following paths in the Jetson Linux kernel source package:
```
kernel-5.10/drivers/pci/controller/dwc/pcie-tegra194.c
kernel-5.10/drivers/pci/controller/dwc/
```
A sample Linux kernel PCIe endpoint function driver.

This driver configures properties of the PCIe endpoint, such as BAR count and size, and IRQ count. It also implements any runtime functionality of the endpoint.

The sample driver is a trivial implementation, useful for demonstration purposes only. It just exposes some endpoint RAM to the PCIe bus. It does not interact with the host or with other parts of the endpoint software. You must implement your own PCIe endpoint function driver according to the needs of your application.

The source code for this driver is available at the following path in the Jetson Linux kernel source package:
```
nvidia/drivers/pci/endpoint/functions/pci-epf-nv-test.c
```
A Linux kernel PCIe endpoint subsystem.

The source code for this subsystem is available at the following path in the Jetson Linux kernel source package:
```
kernel-5.10/drivers/pci/endpoint
```

Hardware Requirements¶

To use PCIe endpoint support, you need the following hardware:

A Jetson Xavier NX or AGX Xavier and AGX Orin series device, which is running Jetson Linux, to act as the PCIe endpoint.
Another computer system to act as the PCIe root port.

These instructions assume that you are using a second Jetson Xavier NX or AGX Xavier or AGX Orin series device that is running Jetson Linux, but you can use any standard x86-64 PC that is running Linux.
Cables to connect the two devices.

See the NVIDIA Jetson AGX Xavier PCIe Endpoint Design Guidelines (DA-09357) application note for more information.

Note

The Jetson Xavier NX Developer Kit supports an endpoint mode at the M.2 Key M slot. An additional adapter is required to convert the M.2 Key M to a CEM slot.

Flashing the PCIe Endpoint on a Jetson NX Xavier Series System¶

Important

The commands in this procedure must be run as the root user. Some commands use shell I/O redirection and will not operate correctly if run by using sudo.

A Jetson system must be flashed in a specific way to enable PCIe endpoint mode. Use the following steps to flash the endpoint system:

In the extracted Jetson Linux release directory, edit the p3668.conf.common file.
Write down the ODMDATA property’s original value and set bit 12 of that value.

For example, you can change the value from 0xB8190000 to 0xB8191000.
Run the following command to reflash the device:
```
sudo ./flash.sh jetson-xavier-nx-devkit mmcblk0p1
```
This step completely erases data that was previously stored on the Jetson system.
Edit the p3668.conf.common file again and restore the ODMDATA property’s original value.

This step ensures that systems that are flashed in the future will operate in PCIe root port mode.

Flashing the PCIe Endpoint on a Jetson AGX Xavier Series System¶

A Jetson device must be flashed in a specific way to enable PCIe endpoint mode:

In the extracted Jetson Linux release directory, edit the p2972-0000.conf.common file.
Set the bit 12 of the ODMDATA value, for example, you can change it from 0x09190000 to 0x09191000.
Run this command to reflash the device:
```
# sudo ./flash.sh jetson-xavier mmcblk0p1
```
This step completely erases data that was previously stored on the Jetson device.
Edit the p2972-0000.conf.common file again and restore the ODMDATA property’s original value of 0x09190000.

This step ensures that devices that were flashed in the future will operate in PCIe root port mode.

Flashing the PCIe Endpoint on a Jetson AGX Orin Series System¶

A Jetson device must be flashed in a specific way to enable PCIe endpoint mode:

In the extracted Jetson Linux release directory, edit the p3701.conf.common file.

#. Set the nvhs-uphy-config of the ODMDATA value, for example: ODMDATA="**gbe-uphy-config-22,nvhs-uphy-config-0,hsio-uphy-config-0**,gbe0-enable-10g,hsstp-lane-map-3"; to ODMDATA="**gbe-uphy-config-22,nvhs-uphy-config-1,hsio-uphy-config-0**,gbe0-enable-10g,hsstp-lane-map-3";

Run this command to reflash the device:
```
# sudo ./flash.sh jetson-agx-orin-devkit mmcblk0p1
```
This step completely erases data that was previously stored on the Jetson device.
Edit the p3701.conf.common file again and restore the ODMDATA property’s original value.

This step ensures that devices that were flashed in the future will operate in PCIe root port mode.

Connecting and Configuring the Devices¶

Connect the devices using the appropriate PCIe cable.
Boot the endpoint device.

Run these commands to configure and enable PCIe endpoint mode:

# cd /sys/kernel/config/pci_ep/
# mkdir functions/pci_epf_nv_test/func1
# echo 0x10de > functions/pci_epf_nv_test/func1/vendorid
# echo 0x0001 > functions/pci_epf_nv_test/func1/deviceid
# ln -s functions/pci_epf_nv_test/func1 controllers/141a0000.pcie_ep/
# echo 1 > controllers/141a0000.pcie_ep/start

For additional details, read the following file in the Jetson Linux kernel source package:

kernel-5.10/Documentation/PCI/endpoint/pci-endpoint-cfs.rst

Boot the root port system.

Testing Procedures¶

Use the following procedures to test PCIe endpoint support.

Prepare for Testing¶

The sample PCIe endpoint function driver exposes a page of RAM to the root port system, which gives both systems access to a shared page of RAM.

Note

The following procedure demonstrates a simple method to transfer data between the two systems through the shared RAM.

Note

In 5.x release, Linux kernel has enhanced security, if you want to access shared RAM need to add CONFIG_STRICT_DEVMEM=n to tegra_defconfig and recompile the kernel

To install required utilities, run this command on the root port system and the endpoint device:
```
# apt install busybox pciutils
```

On the endpoint device, determine the physical address of the RAM that is accessed via the endpoint’s BAR:

# dmesg | grep pci_epf_nv_test

This command prints messages similar to the following:

[ 38.338101] pci_epf_nv_test pci_epf_nv_test.0: BAR0 RAM phys: 0x4307b8000
[ 38.338113] pci_epf_nv_test pci_epf_nv_test.0: BAR0 RAM IOVA: 0xffff0000
[ 38.338138] pci_epf_nv_test pci_epf_nv_test.0: BAR0 RAM virt: 0xffffff800b3dc000

Write the BAR0 RAM phys address.

Accessing the Shared RAM on the Endpoint Device¶

You can test with any tool that allows you to read and write memory locations directly. This procedure uses BusyBox.

To read the RAM, enter this command:
```
# busybox devmem ``<BAR0_RAM_phys>``
```
Where <BAR0_RAM_phys> is the address you noted from the dmesg command’s output.

To write the RAM, enter this command:

# busybox devmem 0x4307b8000 32 0xfa950000

To allow the PCIe endpoint to respond to PCIe memory accesses on the root port system, enter this command:
```
# setpci -s 0005:01:00.0 COMMAND=0x02
```
This command is required only if no Linux kernel device driver binds to the PCIe device and enables the device.

Accessing the Shared RAM on the Root Port System¶

To determine the PCIe address allocated to the shared RAM (the endpoint’s BAR), enter this command:

# lspci -v

This command prints messages similar to the following:

0005:01:00.0 RAM memory: NVIDIA Corporation Device 0001
Flags: fast devsel, IRQ 255
Memory at 3a300000 (32-bit, non-prefetchable) [disabled] [size=64K]
Memory at 1c00000000 (64-bit, prefetchable) [disabled] [size=128K]
Memory at 3a200000 (64-bit, non-prefetchable) [disabled] [size=1M]
Capabilities: [40] Power Management version 3
Capabilities: [50] MSI: Enable- Count=1/1 Maskable+ 64bit-
Capabilities: [70] Express Endpoint, MSI 00
Capabilities: [b0] MSI-X: Enable- Count=8 Masked-
Capabilities: [100] Advanced Error Reporting
Capabilities: [148] #19
Capabilities: [168] #26
Capabilities: [190] #27
Capabilities: [1b8] Latency Tolerance Reporting
Capabilities: [1c0] L1 PM Substates
Capabilities: [1d0] Vendor Specific Information: ID=0002 Rev=4 Len=100 <?>
Capabilities: [2d0] Vendor Specific Information: ID=0001 Rev=1 Len=038 <?>
Capabilities: [308] #25
Capabilities: [314] Precision Time Measurement
Capabilities: [320] Vendor Specific Information: ID=0003 Rev=1 Len=054 <?>

Write down the first Memory at address (the BAR0 address). The CPU on the root port system might access this address directly, and this access type modifies the RAM on the endpoint device. To read the RAM, enter this command:

# busybox devmem 0x3a300000

To write the RAM, enter this command:

# busybox devmem 0x3a300000 32 0xfa950000

Testing Bidirectional Data Transfer¶

Enter the following sequence of commands:

On the endpoint device:

# busybox devmem 0x4307b8000 32 0x98765432

On the root port system:
```
# busybox devmem 0x3a300000
```
Observe that the memory location contains the value 0x98765432, which was written by the endpoint device.

On the root port system:

# busybox devmem 0x3a300000 32 0x12345678

On the endpoint device:
```
# busybox devmem 0x4307b8000
```
Observe that the memory location contains the value 0x12345678, which was written by the root port system.

Bringing up an Ethernet Interface over PCIe¶

Note

In 5x release, EDMA interrupt is enabled by default. Polling on DMA_DONE in Tegra virtual Ethernet have a side effect, it will cause a warn print in controller interrupt handler because DMA interrupt is cleared during polling.Enable asynchronous DMA for which EDMA interrupt handling will be done and eliminated warning error. For the PCIe virtual network we are only using EP EDMA, so only need to apply the following patch to EP side.

diff --git a/kernel/kernel-5.10/drivers/pci/controller/dwc/pcie-tegra194.c b/kernel/kernel-5.10/drivers/pci/controller/dwc/pcie-tegra194.c
index 7fde1a4d6..981bbac02 100644
--- a/kernel/kernel-5.10/drivers/pci/controller/dwc/pcie-tegra194.c
+++ b/kernel/kernel-5.10/drivers/pci/controller/dwc/pcie-tegra194.c
@@ -2164,7 +2164,9 @@ static void tegra_pcie_enable_legacy_interrupts(struct pcie_port *pp)
val |= APPL_INTR_EN_L1_8_INTX_EN;
val |= APPL_INTR_EN_L1_8_AUTO_BW_INT_EN;
val |= APPL_INTR_EN_L1_8_BW_MGT_INT_EN;
+#if 0
val |= APPL_INTR_EN_L1_8_EDMA_INT_EN;
+#endif
if (IS_ENABLED(CONFIG_PCIEAER))
val |= APPL_INTR_EN_L1_8_AER_INT_EN;
appl_writel(pcie, val, APPL_INTR_EN_L1_8_0);
@@ -3569,9 +3571,11 @@ static void pex_ep_event_pex_rst_deassert(struct tegra_pcie_dw *pcie)
val |= APPL_INTR_EN_L1_0_0_RDLH_LINK_UP_INT_EN;
appl_writel(pcie, val, APPL_INTR_EN_L1_0_0);

+#if 0
val = appl_readl(pcie, APPL_INTR_EN_L1_8_0);
val |= APPL_INTR_EN_L1_8_EDMA_INT_EN;
appl_writel(pcie, val, APPL_INTR_EN_L1_8_0);
+#endif

if (pcie->enable_cdm_check) {
val = appl_readl(pcie, APPL_INTR_EN_L0_0);
@@ -3880,6 +3884,7 @@ tegra_pcie_ep_get_features(struct dw_pcie_ep *ep)
return &tegra_pcie_epc_features;
}

+#if 0
/* Reserve BAR0_BASE + BAR0_MSI_OFFSET of size SZ_64K as MSI page */
static int tegra_pcie_ep_set_bar(struct dw_pcie_ep *ep, u8 func_no,
                                 struct pci_epf_bar *epf_bar)
@@ -3903,11 +3908,11 @@ static int tegra_pcie_ep_set_bar(struct dw_pcie_ep *ep, u8 func_no,

return 0;
}
+#endif

static struct dw_pcie_ep_ops pcie_ep_ops = {
.raise_irq = tegra_pcie_ep_raise_irq,
.get_features = tegra_pcie_ep_get_features,
-       .set_bar = tegra_pcie_ep_set_bar,
};

static int tegra_pcie_config_ep(struct tegra_pcie_dw *pcie,

Connect the systems using the appropriate PCIe cable.
Boot the endpoint device.

Enter the following commands to configure and enable PCIe endpoint mode:

# cd /sys/kernel/config/pci_ep/
# mkdir functions/pci_epf_tvnet/func1
# echo 16 > functions/pci_epf_tvnet/func1/msi_interrupts
# ln -s functions/pci_epf_tvnet/func1 controllers/141a0000.pcie_ep/
# echo 1 > controllers/141a0000.pcie_ep/start

Boot the root port system.
Enter the lspci command on the root port system to verify that the PCIe link is up.

The console output should include a message like, PCIe device with vendor id: 0x10de and device id: 0x2296.
Bring up both interfaces to establish an Ethernet link between the systems and assign the interfaces dynamic or static IP addresses.

For example, enter the following commands, in the order shown, to bring up the Ethernet interfaces on /dev/eth1 and assign static IP addresses:
1. On the endpoint device: ifconfig eth1 up
2. On the root port system: ifconfig eth1 up
3. On the endpoint device: ifconfig eth1 192.168.2.1
4. On the root port system: ifconfig eth1 192.168.2.2

Custom Endpoint Function Driver¶

The customer is expected to implement their own PCIe endpoint function driver based on their application need.

Sample example of a Linux kernel PCIe endpoint function driver:

pci-epf-nv-test driver provides the configuration of the PCIe endpoint, such as BAR count and size, IRQ count, etc.

This driver is a minimal example, useful for demonstration purposes only. The driver does not interact with the host or with any other part of the endpoint software at run time. It simply exposes some endpoint RAM to the PCIe bus. The customer is expected to implement their own PCIe endpoint function driver according to the needs of their application.

The source code for this driver is available in <TOP>/kernel/nvidia/drivers/pci/endpoint/functions/pci-epf-nv-test.c file in the Jetson LinuxL4T kernel source package.

Linux kernel PCIe endpoint subsystem:

The PCIe endpoint subsystem provides common PCIe endpoint support code. It binds together endpoint controller drivers and endpoint function drivers.

The source code for this subsystem is available in the <TOP>/kernel/kernel-5.10/drivers/pci/endpoint/ directory in the Jetson Linux kernel source package.