Getting Started

Host Setup

View as Markdown

Holoscan sensor bridge is supported on the following configurations:

  • IGX systems configured with IGX OS 1.1.2+ with CX7 SmartNIC devices.
  • AGX Orin systems running JP6.2.1+. In this configuration, the on-board Ethernet controller is used with the Linux kernel network stack for data I/O; all network I/O is performed by the CPU without network acceleration.
  • AGX Thor systems running JP7.2 with MGBE SmartNIC device and CoE transport.
  • DGX Spark systems running DGX OS 7.2.3+ with CX7 SmartNIC devices.
  • x86 Linux workstations with an NVIDIA discrete GPU. Two host receive paths are supported: unaccelerated Linux socket-based receivers (used primarily for development or testing purposes) and an accelerated RoCE path that requires a ConnectX network adapter on the link to the sensor bridge. Note that this configuration is not part of the primary reference platforms; GPUDirect RDMA into GPU memory on the RoCE path may appear available from the GPU and driver even when the PCIe path on the host does not support it, in which case you must disable GPU VRAM frame buffers at build time (see the x86 Linux tab).

After the Holoscan sensor bridge board is set up, configure a few prerequisites in your host system. While holoscan sensor bridge demonstration applications usually run in a container, these commands are all to be executed outside the container, on the host system directly. These configurations are remembered across power cycles and therefore only need to be set up once.

  • Install git-lfs

    Some data files in the Holoscan sensor bridge source repository use GIT LFS.

    sudo apt-get update
    sudo apt-get install -y git-lfs

  • Grant your user permission to the docker subsystem:

    sudo usermod -aG docker $USER

    Reboot the computer to activate this setting.

Next, follow the directions on the appropriate tab below to configure your host system.

  • Update the GPU driver to 535.309.01 and install IGX 1.1.3 by running the following.

    sudo tee /etc/apt/preferences.d/nvidia-driver-pin >/dev/null <<'EOF'
    Package: nvidia-* libnvidia-*
    Pin: version 535.309.01*
    Pin-Priority: 1001
    EOF
    sudo apt-get purge "nvidia-kernel-*"
    sudo apt-get update && sudo apt-get install -y nvidia-driver-535-open=535.309.01-0ubuntu0.22.04.1
    sudo apt dist-upgrade

    Reboot the machine so the new driver loads:

    sudo reboot

    After rebooting, confirm the new driver is active:

    ➜ nvidia-smi
    +---------------------------------------------------------------------------------------+
    | NVIDIA-SMI 535.309.01             Driver Version: 535.309.01   CUDA Version: 12.2     |
    |-----------------------------------------+----------------------+----------------------+
    | GPU  Name                 Persistence-M | Bus-Id        Disp.A | Volatile Uncorr. ECC |
    | Fan  Temp   Perf          Pwr:Usage/Cap |         Memory-Usage | GPU-Util  Compute M. |
    |                                         |                      |               MIG M. |
    |=========================================+======================+======================|
    |   0  NVIDIA RTX 6000 Ada Gene...    Off | 00000005:09:00.0 Off |                  Off |
    | 30%   34C    P8              20W / 300W |    189MiB / 49140MiB |      0%      Default |
    |                                         |                      |                  N/A |
    +-----------------------------------------+----------------------+----------------------+

  • Determine the name of the network device associated with the first CX7 port. This is the rightmost QSFP port when looking at the back of the IGX unit.

IGX QSFP0
  ls /sys/class/infiniband
  roceP5p3s0f0 roceP5p3s0f1

This will produce a list of CX7 ports; your device names may vary. The lowest numbered one, in this case roceP5p3s0f0, is the first CX7 port.

Check the CX7 firmware version using one of the devices names that appear on your machine from the command above.

  cat /sys/class/infiniband/roceP5p3s0f0/fw_ver
  28.39.3004

If the version is 28.38.1026 or lower, please download the appropriate “ConnectX-7 Firmware” iso and upgrade the CX7 firmware. Make sure to reboot the machine and re-check the firmware has loaded to the newer one.

Assign the name of the first CX7 port to the variable $IN0.

  LC_COLLATE=C IN=(/sys/class/infiniband/*)
  IN0=`basename ${IN[0]}`
  echo $IN0
  roceP5p3s0f0

Next, determine which host ethernet port is associated with that device, and assign that to the variable $EN0, which we’ll use later during network configuration.

  EN0=`basename /sys/class/infiniband/$IN0/device/net/*`
  echo $EN0
  enP5p3s0f0np0

In summary, the host network interface associated with $IN0 (roceP5p3s0f0) is $EN0 (enP5p3s0f0np0); your specific device names may vary.

  • IGX OS uses NetworkManager to configure network interfaces. By default, the sensor bridge device uses the address 192.168.0.2 for the first port. Set up your first ethernet device ($EN0) to use the address 192.168.0.101 with a permanent route to 192.168.0.2 using the recommended MTU for use with RoCE: (Here is more information about configuring your system if you cannot use the 192.168.0.0/24 network in this way.)
  sudo nmcli con add con-name hololink-$EN0 ifname $EN0 type ethernet ip4 192.168.0.101/24
  sudo nmcli connection modify hololink-$EN0 +ipv4.routes 192.168.0.2/32
  sudo nmcli connection modify hololink-$EN0 ethtool.ring-rx 4096
  sudo nmcli connection modify hololink-$EN0 802-3-ethernet.mtu 4200
  sudo nmcli connection up hololink-$EN0

Apply power to the sensor bridge device, ensure that it’s properly connected, then ping 192.168.0.2 to check connectivity:

  ping 192.168.0.2
  PING 192.168.0.2 (192.168.0.2) 56(84) bytes of data.
  64 bytes from 192.168.0.2: icmp_seq=1 ttl=64 time=0.225 ms
  64 bytes from 192.168.0.2: icmp_seq=2 ttl=64 time=0.081 ms
  64 bytes from 192.168.0.2: icmp_seq=3 ttl=64 time=0.088 ms
  64 bytes from 192.168.0.2: icmp_seq=4 ttl=64 time=0.132 ms
  ^C
  --- 192.168.0.2 ping statistics ---
  4 packets transmitted, 4 received, 0% packet loss, time 3057ms
  rtt min/avg/max/mdev = 0.081/0.131/0.225/0.057 ms
  • The second SFP+ connector on the sensor bridge device is used to transmit data acquired from the second camera on a stereo camera module (like the IMX274). By default, the sensor bridge device uses the address 192.168.0.3 for that second port. Connect the second IGX QSFP port (indicated with the red arrow below) to the second SFP+ port on the sensor bridge device.
IGX QSFP1

Let’s refer to these as $IN1 and $EN1. Given the commands to assign $IN0 and $EN0 above,

  IN1=`basename ${IN[1]}`
  echo $IN1
  EN1=`basename /sys/class/infiniband/$IN1/device/net/*`
  echo $EN1
  roceP5p3s0f1
  enP5p3s0f1np1

As above, your device names may be different. Configure the second QSFP network port with an appropriate address and permanent route:

  sudo nmcli con add con-name hololink-$EN1 ifname $EN1 type ethernet ip4 192.168.0.102/24
  sudo nmcli connection modify hololink-$EN1 +ipv4.routes 192.168.0.3/32
  sudo nmcli connection modify hololink-$EN1 ethtool.ring-rx 4096
  sudo nmcli connection modify hololink-$EN1 802-3-ethernet.mtu 4200
  sudo nmcli connection up hololink-$EN1

Now test the second connection with ping 192.168.0.3:

  ping 192.168.0.3
  PING 192.168.0.3 (192.168.0.3) 56(84) bytes of data.
  64 bytes from 192.168.0.3: icmp_seq=1 ttl=64 time=0.210 ms
  64 bytes from 192.168.0.3: icmp_seq=2 ttl=64 time=0.271 ms
  64 bytes from 192.168.0.3: icmp_seq=3 ttl=64 time=0.181 ms
  64 bytes from 192.168.0.3: icmp_seq=4 ttl=64 time=0.310 ms
  64 bytes from 192.168.0.3: icmp_seq=5 ttl=64 time=0.258 ms
  ^C
  --- 192.168.0.3 ping statistics ---
  5 packets transmitted, 5 received, 0% packet loss, time 4102ms
  rtt min/avg/max/mdev = 0.181/0.246/0.310/0.045 ms

When the second port is configured, the first port should continue to respond to pings as appropriate.

Now, for all configurations,

  • Configure the network receiver buffer for running unaccelerated network examples: Holoscan sensor bridge supports running Li VB1940, IMX274, and TauroTech DA326 Hawk using unaccelerated network Linux sockets. For best performance of these examples increase the Linux sockets network receiver buffer:

    echo 'net.core.rmem_max = 31326208' | sudo tee /etc/sysctl.d/52-hololink-rmem_max.conf
    sudo sysctl -p /etc/sysctl.d/52-hololink-rmem_max.conf

  • Make sure that $EN0 is set to the name of the Ethernet controller connected to HSB. Some installations require reboots which will clear this value, so be sure and configure it appropriately for your configuration per the instructions above.

  • Enable PTP on $EN0. This synchronizes the timestamps reported with received data with the host time.

    Run the phc2sys tool at boot time. This synchronizes the clock in $EN0 with the system clock. First, install the linuxptp tool.

    sudo apt update && sudo apt install -y linuxptp

    Next, set up a systemd service file that will run phc2sys.

    PHC2SYS_SERVICE=/etc/systemd/system/phc2sys-$EN0.service
    cat <<EOF | sudo tee $PHC2SYS_SERVICE >/dev/null
    [Unit]
    Description=Copy system time to $EN0
    Requires=NetworkManager.service
    After=NetworkManager.service
    After=timemaster.service
    [Service]
    Type=simple
    ExecStartPre=timeout 3m bash -c "until [ \"\$(nmcli -g GENERAL.STATE device show $EN0)\" = \"100 (connected)\" ]; do sleep 1; done"
    ExecStart=/usr/sbin/phc2sys -c $EN0 -s CLOCK_REALTIME -O 0 -S 0.0001
    [Install]
    WantedBy=multi-user.target
    EOF

    Configure it for execution at startup, and start it now.

    sudo chmod u+x $PHC2SYS_SERVICE
    sudo systemctl enable phc2sys-$EN0.service
    sudo systemctl start phc2sys-$EN0.service

Next, run ptp4l to send PTP SYNC messages to $EN0.

cat <<EOF | sudo tee /etc/linuxptp/hsb-ptp.conf >/dev/null
# This configuration is appropriate for NVIDIA Holoscan sensor bridge
# applications, where PTP messages are sent over L2 and a 1/2 second interval.
[global]
logSyncInterval -1
logMinDelayReqInterval -1
network_transport L2
EOF

Set up a systemd service file for this.

PTP4L_SERVICE=/etc/systemd/system/ptp4l-$EN0.service
cat <<EOF | sudo tee $PTP4L_SERVICE >/dev/null
[Unit]
Description=Send PTP SYNC messages to $EN0
After=phc2sys-$EN0.service
 
[Service]
Type=simple
ExecStart=/usr/sbin/ptp4l -i $EN0 -f /etc/linuxptp/hsb-ptp.conf
 
[Install]
WantedBy=multi-user.target
EOF

Finally, run it.

sudo chmod u+x $PTP4L_SERVICE
sudo systemctl enable ptp4l-$EN0.service
sudo systemctl start ptp4l-$EN0.service

  • For Orin iGPU with Cuda 12 configurations only (Jetpack 6 and IGX 1.X): Install NVIDIA DLA compiler. Applications using inference need this at initialization time; some OS images for iGPU don’t include it.

    sudo apt update && sudo apt install -y nvidia-l4t-dla-compiler

  • Log in to Nvidia GPU Cloud (NGC) with your developer account:

      docker login nvcr.io
      Username: $oauthtoken
      Password: <Your token key to NGC>
      WARNING! Your password will be stored unencrypted in /home/<user>/.docker/config.json.
      Configure a credential helper to remove this warning. See
      https://docs.docker.com/engine/reference/commandline/login/#credentials-store
      Login Succeeded

Now proceed to build and test the Holoscan Sensor Bridge container.