Jetson AGX Xavier PCIe Endpoint Mode

Todo

Does this apply to Orin? What about Xavier NX… it uses the same processor as AGX Xavier, so why not?

NVIDIA® Jetson™ Linux contains the following software support for PCIe endpoint mode:

  • A Linux kernel device driver for the PCIe endpoint controller. This driver configures the PCIe controller as an endpoint, and provides an interface for higher-level software to select the configuration that is exposed to the PCIe bus.

    Source code for this driver is available at the following paths in the Jetson Linux kernel source package:

    nvidia/drivers/pci/dwc/pcie-tegra.c
kernel-4.9/drivers/pci/dwc/

  • A sample Linux kernel PCIe endpoint function driver. This driver configures properties of the PCIe endpoint, such as BAR count and size, and IRQ count. It also implements any runtime functionality of the endpoint.

    The sample driver is a trivial implementation, useful for demonstration purposes only. It just exposes some endpoint RAM to the PCIe bus. It does not interact with the host or with other parts of the endpoint software. You must implement your own PCIe endpoint function driver according to the needs of your application.

    Source code for this driver is available at the following path in the Jetson Linux kernel source package:

    nvidia/drivers/pci/endpoint/functions/pci-epf-nv-test.c

  • A Linux kernel PCIe endpoint subsystem. The PCIe endpoint subsystem provides common PCIe endpoint support code. It binds together endpoint controller drivers and endpoint function drivers.

    Source code for this subsystem is available at the following path in the Jetson Linux kernel source package:

    kernel/kernel-4.9/drivers/pci/endpoint/

Hardware Requirements

You need the following hardware to use PCIe endpoint support:

  • An NVIDIA® Jetson AGX Xavier™ series device running Jetson Linux, to act as the PCIe endpoint.

  • Another computer system to act as the PCIe root port. These instructions assume that you are using a second Jetson AGX Xavier series device running Jetson Linux, but any standard x86-64 PC running Linux will act almost identically.

  • Cables to connect the two devices. See the NVIDIA application note NVIDIA Jetson AGX Xavier PCIe Endpoint Design Guidelines (DA-09357) for details.

Todo

Where? It’s not in the J.D.C.

Commands Must Be Run as Root

All commands described in these instructions must be run as the root user. Some commands use shell I/O redirection, and will not operate correctly if run using sudo.

To flash the PCIe endpoint on a Jetson AGX Xavier series device

A Jetson device must be flashed in a specific way to enable PCIe endpoint mode:

  1. In the extracted Jetson Linux release directory, edit p2972-0000.conf.common. Set bit 12 of the ODMDATA value, i.e. change it from 0x09190000 to 0x09191000.

    Todo

    Here and throughout the Developer Guide, we refer to this software in a variety of ways. Here we started with “the Jetson Linux kernel source package”; now we’ve switched to “the extracted Jetson Linux release directory.” Let’s standardize on one description that is concise and clear.

  2. Run this command to reflash the device:

    # sudo ./flash.sh jetson-xavier mmcblk0p1

    Todo

    I’m using the ‘#’ prompt in accord with the instruction to run all commands as root. But why does the command use sudo? Unless we’re ignoring the caution we just gave, there should be no need for that.

    Note that this completely erases any data previously stored on the Jetson device.

  3. Edit p2972-0000.conf.common again and restore the ODMDATA property’s original value of 0x09190000. This ensures that any devices flashed in the future operate in PCIe root port mode.

To connect and configure the devices

  1. Connect the devices using the appropriate PCIe cable.

  2. Boot the endpoint device.

  3. Run these commands to configure and enable PCIe endpoint mode:

    # cd /sys/kernel/config/pci_ep/
# mkdir functions/pci_epf_nv_test/func1
# echo 0x10de > functions/pci_epf_nv_test/func1/vendorid
# echo 0x0001 > functions/pci_epf_nv_test/func1/deviceid
# ln -s functions/pci_epf_nv_test/func1 controllers/141a0000.pcie_ep/
# echo 1 > controllers/141a0000.pcie_ep/start

    For additional details, read the following file in the Jetson Linux kernel source package:

    kernel-4.9/Documentation/PCI/endpoint/pci-endpoint-cfs.txt

  4. Boot the root port system. You must execute this step after step 3 above.

    Todo

    Is there some specific reason for “You must execute…”? The steps in a numbered-list procedure are always executed in order.

Testing Procedures

Use the following procedures to test PCIe endpoint support.

To prepare for testing

The sample PCIe endpoint function driver exposes a page of RAM to the root port system. This gives both systems access to a shared page of RAM. This section demonstrates a simple method for transferring data between the two systems through the shared RAM.

  1. Run this command on both the root port system and the endpoint device to install required utilities:

    # apt install busybox pciutils

    Todo

    Assuming the “run as root” instruction is operative, does it apply to the root port system as well as the endpoint device?

  2. On the endpoint device, determine the physical address of the RAM that is accessed via the endpoint’s BAR:

    # dmesg | grep pci_epf_nv_test

    This command prints messages similar to the following:

    [ 38.338101] pci_epf_nv_test pci_epf_nv_test.0: BAR0 RAM phys: 0x4307b8000
[ 38.338113] pci_epf_nv_test pci_epf_nv_test.0: BAR0 RAM IOVA: 0xffff0000
[ 38.338138] pci_epf_nv_test pci_epf_nv_test.0: BAR0 RAM virt: 0xffffff800b3dc000

    Make a note of the “BAR0 RAM phys” address.

To access the shared RAM on the endpoint device

You can test with any tool that allows you to read and write memory locations directly. This procedure uses BusyBox.

  • To read the RAM, enter this command:

    # busybox devmem ``<BAR0_RAM_phys>``

    Where <BAR0_RAM_phys> is the address you noted from the dmesg command’s output.

  • To write the RAM, enter this command:

    # busybox devmem 0x4307b8000 32 0xfa950000

  • To allow the PCIe endpoint to respond to PCIe memory accesses on the root port system, enter this command:

    # setpci -s 0005:01:00.0 COMMAND=0x02

    Note that this command is required only if no Linux kernel device driver binds to the PCIe device and enables the device.

    Todo

    That paragraph’s meaning is unclear. The new term “PCIe device” adds to the uncertainty. I assume it’s a variant of “endpoint device,” but maybe it doesn’t make sense to me because it’s a distinct term that needs to be explained.

To access shared RAM on the root port system

  • To determine the PCIe address allocated to the shared RAM (the endpoint’s BAR), enter this command:

    # lspci -v

    This command prints messages similar to the following:

    0005:01:00.0 RAM memory: NVIDIA Corporation Device 0001
Flags: fast devsel, IRQ 255
Memory at 3a300000 (32-bit, non-prefetchable) [disabled] [size=64K]
Memory at 1c00000000 (64-bit, prefetchable) [disabled] [size=128K]
Memory at 3a200000 (64-bit, non-prefetchable) [disabled] [size=1M]
Capabilities: [40] Power Management version 3
Capabilities: [50] MSI: Enable- Count=1/1 Maskable+ 64bit-
Capabilities: [70] Express Endpoint, MSI 00
Capabilities: [b0] MSI-X: Enable- Count=8 Masked-
Capabilities: [100] Advanced Error Reporting
Capabilities: [148] #19
Capabilities: [168] #26
Capabilities: [190] #27
Capabilities: [1b8] Latency Tolerance Reporting
Capabilities: [1c0] L1 PM Substates
Capabilities: [1d0] Vendor Specific Information: ID=0002 Rev=4 Len=100 <?>
Capabilities: [2d0] Vendor Specific Information: ID=0001 Rev=1 Len=038 <?>
Capabilities: [308] #25
Capabilities: [314] Precision Time Measurement
Capabilities: [320] Vendor Specific Information: ID=0003 Rev=1 Len=054 <?>

    Make a note of the first “Memory at” address (the BAR0 address). The CPU on the root port system may access this address directly. Any such access modifies RAM on the endpoint device. To read the RAM, enter this command:

    # busybox devmem 0x3a300000

  • To write the RAM, enter this command:

    # busybox devmem 0x3a300000 32 0xfa950000

To test bidirectional data transfer

Enter this sequence of commands:

  1. On the endpoint device:

    # busybox devmem 0x4307b8000 32 0x98765432

  2. On the root port system:

    # busybox devmem 0x3a300000

    Observe that the memory location contains the value 0x98765432, which was written by the endpoint device.

  3. On the root port system:

    # busybox devmem 0x3a300000 32 0x12345678

  4. On the endpoint device:

    # busybox devmem 0x4307b8000

    Observe that the memory location contains the value 0x12345678, which was written by the root port system.

To bring up an Ethernet interface over PCIE

  1. Connect the systems using the appropriate PCIe cable.

  2. Boot the endpoint device.

  3. Enter the following commands to configure and enable PCIe endpoint mode:

    # cd /sys/kernel/config/pci_ep/
# mkdir functions/pci_epf_tvnet/func1
# echo 16 > functions/pci_epf_tvnet/func1/msi_interrupts
# ln -s functions/pci_epf_tvnet/func1 controllers/141a0000.pcie_ep/
# echo 1 > controllers/141a0000.pcie_ep/start

  4. Boot the root port system.

  5. Enter the lspci command on the root port system to verify that the PCIe link is up. The console output should include a message like, “PCIe device with vendor id: 0x10de and device id: 0x2296.”

  6. Bring up both interfaces to establish an Ethernet link between the systems. Then assign them dynamic or static IP addresses.

    For example, enter the following commands, in the order shown, to bring up the Ethernet interfaces on /dev/eth1 and assign static IP addresses:

    1. On the endpoint device: ifconfig eth1 up

    2. On the root port system: ifconfig eth1 up

    3. On the endpoint device: ifconfig eth1 192.168.2.1

    4. On the root port system: ifconfig eth1 192.168.2.2