Important notes
Receiver operators, including RoceReceiverOp and LinuxReceiverOperator specifications include:
Always return the most recently received video frame. This operator will never return a video frame older than 1 frame time. If the pipeline is busy and is unavailable to accept incoming video frames, older frame data is replaced with newer frame data. If the frame request occurs while another frame is being received, this operator returns the last completed video frame.
Never return the same video frame twice. If the pipeline is faster than a video reception time, the pipeline will block until the next incoming frame is complete.
PTP timestamps, sent by HSB on both the start-of-frame and the end-of-frame condition, are available by looking at received metadata. When PTP is properly configured, this provides a reliable mechanism for determining the actual latency for processing or this displaying this data. Note that PTP is only supported for IGX configurations.
RoceReceiverOp uses RDMA with special caveats that are likely to change in future versions:
There is no protection against rewriting the video buffer while the pipeline is using it. The current implementation allocates enough GPU memory to receive two video frames, and the transmitter is configured to alternate between these two buffers. Video pipelines usually start by copying the data from the receiver buffer into another region of memory (via CsiToBayerOp)–this minimizes the time during which an overwrite hazard can occur–but scheduling these operations is dependent on CPU availability. CudaCRCOp can be used to verify that received data has not been corrupted; as long as it computes the same value that’s found in the receiver metadata, then you’ve proven that the memory contents have not been updated.
For systems where the 192.168.0.0/24 network is unavailable, you can use the
hololink-set-ip command to reconfigure sensor bridge IP addresses. Sensor bridge
devices transmit enumeration messages based on the BOOTP protocol; the host can reply
with a request to set the IP address to a specific value. Programs in the Holoscan
sensor bridge host software all accept a --hololink=<IP-address> command line
parameter to look for the device with the given IP address instead of the default
192.168.0.2.
First, locate the MAC ID and local interface that the sensor bridge port is connected to
with the hololink-enumerate command. Within the
demo container:
$ hololink-enumerate
mac_id=48:B0:2D:EE:84:DA hsb_ip_version=0x2507 fpga_crc=0xffff ip_address=192.168.0.2 fpga_uuid=889b7ce3-65a5-4247-8b05-4ff1904c3359 serial_number=10030032828115 interface=enP5p3s0f0np0 board=hololink-lite
mac_id=48:B0:2D:EE:84:DB hsb_ip_version=0x2507 fpga_crc=0xffff ip_address=192.168.0.3 fpga_uuid=889b7ce3-65a5-4247-8b05-4ff1904c3359 serial_number=10030032828115 interface=enP5p3s0f1np1 board=hololink-lite
mac_id=48:B0:2D:EE:84:DA hsb_ip_version=0x2507 fpga_crc=0xffff ip_address=192.168.0.2 fpga_uuid=889b7ce3-65a5-4247-8b05-4ff1904c3359 serial_number=10030032828115 interface=enP5p3s0f0np0 board=hololink-lite
mac_id=48:B0:2D:EE:84:DB hsb_ip_version=0x2507 fpga_crc=0xffff ip_address=192.168.0.3 fpga_uuid=889b7ce3-65a5-4247-8b05-4ff1904c3359 serial_number=10030032828115 interface=enP5p3s0f1np1 board=hololink-lite
This configuration has two network ports, with MAC ID 48:B0:2D:EE:84:DA connected to the local enP5p3s0f0np0 device; 48:B0:2D:EE:84:DB is connected to enP5p3s0f1np1. These connect to the same sensor bridge device, as shown by the common serial number. Note that these messages will be observed by the local system regardless of the IP addresses of local network devices.
For our example, we’ll set up this configuration:
Host eth0 will be configured to 192.168.200.101/24
Host eth1 will be configured to 192.168.200.102/24
Sensor bridge port 48:B0:2D:EE:84:DA will use IP address 192.168.200.2
Sensor bridge port 48:B0:2D:EE:84:DB will use IP address 192.168.200.3
Explicit routes are added with enP5p3s0f0np0 to 192.168.200.2 and enP5p3s0f1np1 to 192.168.200.3
First, follow the setup instructions to configure the host IP addresses and
routes for your expected configuration– replace the references to 192.168.0.x with the
addresses you expect to use. Then, use the hololink-set-ip command to reconfigure the
sensor bridge device ports. To configure the target IP address examples above, in the
demo container,
$ hololink-set-ip 48:B0:2D:EE:84:DA 192.168.200.2 48:B0:2D:EE:84:DB 192.168.200.3
Setting the following addresses:
48:B0:2D:EE:84:DB to 192.168.200.3
48:B0:2D:EE:84:DA to 192.168.200.2
Running in daemon mode; run with '--one-time' to exit after configuration.
Updating mac=48:B0:2D:EE:84:DA from ip=192.168.0.2 to new_ip=192.168.200.2
Updating mac=48:B0:2D:EE:84:DB from ip=192.168.0.3 to new_ip=192.168.200.3
Found mac_id=48:B0:2D:EE:84:DA using peer_ip=192.168.200.2
Found mac_id=48:B0:2D:EE:84:DB using peer_ip=192.168.200.3
hololink-set-ip accepts a list of MAC ID and IP address pairs. Because the sensor
bridge IP address configuration is not stored in nonvolatile memory, hololink-set-ip
runs as a daemon, and must be running whenever this configuration is desired. When
hololink-set-ip sees an enumeration message from a device with a listed MAC ID but a
different IP address, it will reply with a request to set the desired IP address– this
accommodates IP address reverting on power cycle or reset. Following this, pinging the
target IP address now works:
$ ping 192.168.200.2
PING 192.168.200.2 (192.168.200.2) 56(84) bytes of data.
64 bytes from 192.168.200.2: icmp_seq=1 ttl=64 time=0.215 ms
64 bytes from 192.168.200.2: icmp_seq=2 ttl=64 time=0.352 ms
64 bytes from 192.168.200.2: icmp_seq=3 ttl=64 time=0.365 ms
^C
...
$ ping 192.168.200.3
PING 192.168.200.3 (192.168.200.3) 56(84) bytes of data.
64 bytes from 192.168.200.3: icmp_seq=1 ttl=64 time=0.218 ms
64 bytes from 192.168.200.3: icmp_seq=2 ttl=64 time=0.323 ms
64 bytes from 192.168.200.3: icmp_seq=3 ttl=64 time=0.281 ms
^C
Note that the routing must be set up correctly for the ping commands to work as expected; for examples on how to permanently configure routes, see the Host setup page. Here is more specific information on sensor bridge IP address reconfiguration.
Please use one of the following cables to connect the sensor bridge SPF+ to the IGX devkit or DGX Spark QSFP port:
Fiber optic cable with optick to SFP+ and SFP+ to QSFP adapters
Copper SFP+ to SFP+ with an additional SFP+ to QSFP adapter
Please use the following cables to connect the sensor bridge SFP+ to the Jetson AGX Orin devkit 10G Ethernet port:
All Cables and adapters are available for purchase online - please note that the links above are only for demonstration purposes and should not be considered as a purchase recommendation.
A new channel enumeration metadata parameter “mtu” is added, which can be adjusted by
calling the DataChannel.use_mtu() method. By default, the 1500 Ethernet MTU is used.
If the application suggests a different value, then the code will do one of these
things:
use the specific value you’ve passed in
use an adjusted value based on e.g. GPU page size, NIC constraints, or RDMA requirements
or stick with its default value, ignoring this suggestion entirely
If a new size is used, then it’s guaranteed to be no larger than the suggested MTU. Note that if HSB sends packets with an MTU that is outside the receiver’s acceptable size (e.g. due to hardware or OS configured limits) then it’s likely that no data plane traffic would be observed. This also applies for data plane packets traversing switches or other network interconnects: all the components in the data path must support the MTU being requested.
Configuring the receiver to accept larger MTUs usually require OS level configuration of
MTU on the receiving network interface. Host setup instructions now
include recommendations to execute e.g.
sudo nmcli connection modify hololink-$EN0 802-3-ethernet.mtu 4096, for all HSB
connected interfaces, to allow for a larger receiver MTU. Note that you must
reinitialize that network interface (e.g. sudo nmcli con down hololink-$EN0 and
sudo nmcli con up hololink-$EN0) for that setting to take effect.
ConnectX-7, included in IGX configurations, only supports a maximum MTU of 4096–it does not support the industry standard 9000 byte jumbo size. For this reason, larger packets than 4k have not been tested. AGX Orin’s network interface doesn’t support an MTU larger than 1500.
When choosing an MTU value, note that HSB data plane packets include some overhead: 74 bytes for ROCE or 46 for P1722B COE, and the remainder of the MTU is available for payload. If you want 256 bytes of payload in a ROCE configuration, you need to specify an MTU of 330, which is the payload size plus the 74 bytes of overhead. HSB host code normally reduces the requested MTU so that the payload size aligns with GPU cache pages (128 bytes); so payload sizes that aren’t even multiples of 128 are usually reduced to meet that alignment.