doc/tutorials/recording/dwx_HighThroughputRecording.md

Go to the documentation of this file.

# Copyright (c) 2019-2020, NVIDIA CORPORATION.  All rights reserved.

@page dwx_recording_devguide_high_throughput_recording High Throughput Recording
@tableofcontents

This section provides guidance on developing strategies for recording the huge
amounts of data that multiple cameras can produce.

@section dwx_recording_devguide_high_determining_speed Determining the Required the USB Disk Speed

Because of storage limitations with the platform's local file system, sensor
data is stored on an external USB 3.0 drive. But the drive must be fast enough
to keep up with the amount of data that your sensors generate. For example, four
cameras generating RAW data at 30 frames per second can exceed the bandwidth of
most SSD USB drives.

The required write speed of the drives depends on the number of sensors and the
recording format. If you are recording RAW data, calculate the required drive
speed as:

    width * height * 2 * fps = bytes/second
RAW recording consumes most of the bandwidth.

@section dwx_recording_devguide_high_options Options for Capturing High-Throughput Data

The following table provides suggestions for recording more than three cameras generating RAW.

| Cameras generating RAW | Framerate | Options |
|------------------------|-----------|---------|
| 3-4 | 30 | Attach all cameras to a single Xavier and mount a USB on that Xavier. |
| 6-8 | 30 | <ul><li>Attach 3-4 cameras to Xavier A and mount a USB on that Xavier.</li><li>Do the same for Xavier B.</li><li>Use PTP or gPTP to synchronize camera timestamps between the Xavier cameras.</li></ul>|
| 12  | 30 | Use two NVIDIA DRIVE platforms. For each platform, follow the steps as for 6-8 cameras at 30 FPS. |

@section dwx_recording_devguide_high_achieving_max_throughput Achieving the Maximum USB Throughput

You should be able to achieve USB3.0-SSD write speeds between 400 and 500-
megabytes per second, depending on the brand of the USB3.0 SSD being used. That
speed is adequate for three cameras generating RAW data. If you need to store
more data than your USB device can handle, consider distributing recording
across the different Xavier processors. For more information, see
@ref dwx_devguide_rec_distrec in this guide.

NVIDIA has tested extensively with Samsung 850 EVO and Samsung 750 EVO SSDs,
where the SSDs are formatted with the EXT4 file system. The following are
guidelines for ensuring optimal disk speed.

The system automatically mounts the USB drive.

### To initialize your USB disk (Linux)

1. Enable the CFQ schedule for the USB disk:

       # echo cfq > /sys/block/sda/queue/scheduler
   The directory /sys/block/device/queue/iosched contains files that allow the
   administrator to retrieve and set tunable values related to the I/O scheduler.

2. Change the memory limit for the I/O scheduler.

       # echo 1000 > /sys/module/usbcore/parameters/usbfs_memory_mb
3. Confirm you successfully updated the memory limit.

       # cat /sys/module/usbcore/parameters/usbfs_memory_mb

### To optimize your platform to store recorded data (Linux)

1. Locate and edit the following file:

       /etc/sysctl.conf
2. Modify the file to have the following settings. These are persistent changes
   and are preserved between reboots.
```
       net.core.rmem_default = 1048576
       net.core.rmem_max = 10485760
       net.core.wmem_default = 1048576
       net.core.wmem_max = 10485760
       net.core.netdev_max_backlog = 30000
       net.ipv4.ipfrag_high_thresh = 8388608
       net.ipv6.conf.all.disable_ipv6 = 1
       vm.dirty_background_ratio = 5
       vm.dirty_ratio = 50
```

### To determine the write speed of the mounted jbod disk (Linux)

- Enter:

       # dd if=/dev/zero of=/media/NVidia/folder#/file bs=1G count=10
  The JBod disk is mounted on:

       /media/NVidia/