NVIDIA Tegra
NVIDIA DRIVE OS 5.1 Linux SDK

Developer Guide
5.1.6.0 Release


 
Non-Transparent Bridging and PCIe Interface Communication
 
Overview
Non-Crosslink NTB connection for Linux
Crosslink NTB connection for Linux
This topic describes using Non-Transparent Bridge (NTB) for inter-domain communication through PCIe interfaces.
Overview
A limitation of the PCI Express (PCIe) architectural model is that it allows only a single root, and that the root and all of the End Points (EP) must share a common address space. In many applications there is a need to interconnect two independent PCI domains. A Non-Transparent Bridge (NTB) enables this inter-domain communication, facilitating communication between devices in different switch partitions. This ability enables both hosts and EPs to initiate transactions to hosts and/or EPs in another switch partition.
An NTB or NT Interconnection consists of two or more PCI functions each defined by a Type 0 PCI header that are interconnected by a bridging function. The Type 0 PCI functions are referred to as Non-Transparent EPs (NT EP). Each partition can have at most one NT EP.
The Microsemi’s PM8534 PFX switch on Pegasus board can be used to establish NTB connection which will allow one Root Complex device on board to establish and communicate to another Root Complex device on same board or another Pegasus board.
There are two types of NTB connection topologies which are supported on Pegasus board:
1. Non-Crosslink NTB:
When NTB connection is established between Xavier A and Xavier B Root Complex on Pegasus.
2. Crosslink NTB:
When NTB connection is established between Xavier A of one Pegasus board to Xavier A of another Pegasus board.
The physical PCIe connection between two Pegasus boards is created using miniSAS HD cables connected on x16 PCIe Connector slot.
The following image shows PCIe connector cables connected on Pegasus:
C:\Users\asite\Desktop\pcie_connector.jpg
Non-Crosslink NTB connection for Linux
Use the steps listed below to establish NTB connection between the two Xaviers on same Pegasus board. These steps also explain the use of an example NTB client driver NTB_NETDEV which will expose virtual ethernet interfaces. Any traffic on virtual ethernet interface will pass through PCIe.
1. Update the switch firmware on board to bootleg firmware which supports NTB.
The path of SDK/PDK firmware file to be flashed is as as follows:
<TOP>/drive-t186ref-foundation/firmwares/bin/t19x/microsemi/pcie/bins/ firmware/pm74602_pfx_01080D58_315391/pm74602_pfx_01080D58/images/switchtec_pfx.pmc
 
For steps to update firmware, see Flash PCIe Switch Configuration/Firmware (with UART) under Flashing Basics section.
2. Flash usecase4.0 config file on PM8534 switch of both boards.
The path of the SDK configuration file to be flashed is as follows:
<TOP>/drive-t186ref-foundation/firmwares/bin/t19x/microsemi/pcie/bins/e3550/config_files/pm8534/usecase4.0.pmc
 
For steps to update firmware, see Flash PCIe Switch Configuration/Firmware (with UART) under Flashing Basics section.
3. Power cycle the board after booting check lspci on Xavier A and Xavier B to confirm if link training is successful and both hosts can see switch.
root@tegra-ubuntu:~# lspci
0000:00:00.0 PCI bridge: NVIDIA Corporation Device 1ad0 (rev a1)
0000:01:00.0 PCI bridge: PMC-Sierra Inc. Device 8534
0000:01:00.1 Bridge: PMC-Sierra Inc. Device 8534
0000:02:00.0 PCI bridge: PMC-Sierra Inc. Device 8534
0000:02:01.0 PCI bridge: PMC-Sierra Inc. Device 8534
0000:03:00.0 Ethernet controller: Device 1d6a:d107 (rev 02)
0000:04:00.0 RAID bus controller: Marvell Technology Group Ltd. 88SE9480 SAS/SATA 6Gb/s RAID controller (rev c3)
0001:00:00.0 PCI bridge: NVIDIA Corporation Device 1ad1 (rev a1)
0001:01:00.0 Ethernet controller: Device 1d6a:0001 (rev 02)
 
4. After booting the host, insert the following modules if not auto loaded.
root@tegra-ubuntu:~# modprobe ntb
root@tegra-ubuntu:~# modprobe switchtec
root@tegra-ubuntu:~# modprobe ntb_hw_switchtec
root@tegra-ubuntu:~# modprobe ntb_transport
[ 559.033375] Software Queue-Pair Transport over NTB, version 4
[ 559.034097] switchtec switchtec0: ntb link up
root@tegra-ubuntu:~# modprobe ntb_netdev
[ 602.604719] switchtec 0000:01:00.1: NTB Transport QP 0 created
[ 602.605397] switchtec 0000:01:00.1: eth1 created
 
5. Assign IPs to new interfaces detected on both the hosts (assuming interface eth1).
Host 1:
root@tegra-ubuntu:~# ifconfig eth1 up
root@tegra-ubuntu:~# ifconfig eth1 192.168.1.11
Host 2:
root@tegra-ubuntu:~# ifconfig eth1 up
root@tegra-ubuntu:~# ifconfig eth1 192.168.1.12
When ethernet interfaces on both hosts are up, you can find a “link ready” print in kernel logs.
[ 754.142617] IPv6: ADDRCONF(NETDEV_UP): eth1: link is not ready
[ 754.142776] switchtec 0000:01:00.1: qp 0: Link Up
[ 754.143314] IPv6: ADDRCONF(NETDEV_CHANGE): eth1: link becomes ready
 
6. Run ping command from either host to other host to verify the connection.
Ping from Host 2 to Host 1:
root@tegra-ubuntu:~# ping 192.168.1.11
PING 192.168.1.11 (192.168.1.11) 56(84) bytes of data.
64 bytes from 192.168.1.11: icmp_seq=1 ttl=64 time=3.20 ms
64 bytes from 192.168.1.11: icmp_seq=2 ttl=64 time=0.558 ms
64 bytes from 192.168.1.11: icmp_seq=3 ttl=64 time=0.535 ms
--- 192.168.1.11 ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 2003ms
 
Crosslink NTB connection for Linux
Use the steps listed below to establish NTB connection between two Pegasus boards. Both Pegasus boards are required to be connected using miniSAS-HD cables on PCIe Connector Slot.
The following steps also explain the use of an example NTB client driver NTB_NETDEV which will expose virtual ethernet interfaces. Any traffic on virtual ethernet interface will pass through PCIe.
1. Update the switch firmware on both boards to bootleg firmware which supports NTB.
The path of the SDK firmware file to be flashed is as follows:
<TOP>/drive-t186ref-foundation/firmwares/bin/t19x/microsemi/pcie/bins/ firmware/pm74602_pfx_01080D58_315391/pm74602_pfx_01080D58/images/switchtec_pfx.pmc
 
For information on updating firmware, see Flash/Update PCIe Switch Configuration/Firmware (with UART) under Flashing Basics section.
2. Flash usecase7.0 config file on PM8534 switch of both boards.
The path of SDK configuration file to be flashed is as follows:
<TOP>/drive-t186ref-foundation/firmwares/bin/t19x/microsemi/pcie/bins/e3550/config_files/pm8534/usecase7.0.pmc
 
For information on updaing firmware, see Flash/Update PCIe Switch Configuration/Firmware (with UART) under Flashing Basics section.
3. Power cycle both the boards. After booting Xavier A, check lspci on both hosts to confirm if link training is successful and hosts can see PM8534 switch.
0000:00:00.0 PCI bridge: NVIDIA Corporation Device 1ad0 (rev a1)
0000:01:00.0 PCI bridge: PMC-Sierra Inc. Device 8534
0000:01:00.1 Bridge: PMC-Sierra Inc. Device 8534
0000:02:00.0 PCI bridge: PMC-Sierra Inc. Device 8534
0000:02:01.0 PCI bridge: PMC-Sierra Inc. Device 8534
0000:02:02.0 PCI bridge: PMC-Sierra Inc. Device 8534
0000:02:03.0 PCI bridge: PMC-Sierra Inc. Device 8534
0000:04:00.0 Ethernet controller: Device 1d6a:d107 (rev 02)
0000:05:00.0 RAID bus controller: Marvell Technology Group Ltd. 88SE9480 SAS/SATA 6Gb/s RAID controller (rev c3)
0000:06:00.0 Ethernet controller: Device 1d6a:d107 (rev 02)
0001:00:00.0 PCI bridge: NVIDIA Corporation Device 1ad1 (rev a1)
0001:01:00.0 Ethernet controller: Device 1d6a:0001 (rev 02)
 
4. After booting the host, insert the following modules if they are not auto loaded.
root@tegra-ubuntu:~# modprobe ntb
root@tegra-ubuntu:~# modprobe switchtec
root@tegra-ubuntu:~# modprobe ntb_hw_switchtec
root@tegra-ubuntu:~# modprobe ntb_transport
[ 559.033375] Software Queue-Pair Transport over NTB, version 4
[ 559.034097] switchtec switchtec0: ntb link up
root@tegra-ubuntu:~# modprobe ntb_netdev
[ 602.604719] switchtec 0000:01:00.1: NTB Transport QP 0 created
[ 602.605397] switchtec 0000:01:00.1: eth1 created
 
5. Assign IPs to new interfaces detected on both hosts (assuming eth1 interface).
Host 1:
root@tegra-ubuntu:~# ifconfig eth1 up
root@tegra-ubuntu:~# ifconfig eth1 192.168.1.11
Host 2:
root@tegra-ubuntu:~# ifconfig eth1 up
root@tegra-ubuntu:~# ifconfig eth1 192.168.1.12
One should notice a link ready print in kernel logs when ethernet interfaces on both hosts are up.
[ 754.142617] IPv6: ADDRCONF(NETDEV_UP): eth1: link is not ready
[ 754.142776] switchtec 0000:01:00.1: qp 0: Link Up
[ 754.143314] IPv6: ADDRCONF(NETDEV_CHANGE): eth1: link becomes ready
 
6. Run ping command from one of the hosts to other host to verify the connection.
Ping from Host 2 to Host 1:
root@tegra-ubuntu:~# ping 192.168.1.11
PING 192.168.1.11 (192.168.1.11) 56(84) bytes of data.
64 bytes from 192.168.1.11: icmp_seq=1 ttl=64 time=3.20 ms
64 bytes from 192.168.1.11: icmp_seq=2 ttl=64 time=0.558 ms
64 bytes from 192.168.1.11: icmp_seq=3 ttl=64 time=0.535 ms
--- 192.168.1.11 ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 2003ms
rtt min/avg/max/mdev = 0.535/1.431/3.201/1.251 ms