doc/tutorials/recording/dwx_DistributedRecording.md

Go to the documentation of this file.

# Copyright (c) 2018, NVIDIA CORPORATION.  All rights reserved.

@page dwx_devguide_rec_distrec Distributed Recording
@tableofcontents

This section provides guidance on synchroneously recording data on sensors
distributed across multiple NVIDIA DRIVE<sup>&trade;</sup> platforms.

The diagram below shows two NVIDIA DRIVE platforms with the recording
application, each collecting their own data. For each recorder, data is stored
with a timestamp that records the time, in milliseconds. Data between the two
NVIDIA DRIVE platforms are sycnronized based on timestamps. Multiple sensors
allow for higher accuracy, and the exact timestamps ensure precise
synchronization between the devices.

![](dwx_distributed_recording_app_overview.jpg)

Both the @ref dwx_recorder_textui_tool and @ref dwx_gui_recording2_tool
can support distributed recording with systems reachable by the network.
The front-end instance of these tools launches  multiple back-end instances on
connected Xaviers and synchronizes them over UNIX pipes.

@section dwx_devguide_rec_distrec_basic Basic Steps

@subsection dwx_recorder_usecase2_step_1 Step 1: Set up Xavier A & B
On each Xavier:
1. Verify the Sensors Are Collecting Data in @ref dwx_recording_devguide_basic_recording.
2. Create a DriveWorks Rig Configuration File for the sensor set.
3. Attach eSATA/USB storage.

@subsection dwx_devguide_rec_distrec_step2 Step 2: Collect Rig Files
On the master Xavier (either A or B), collect rig files for both Xavier A and B,
and arrange them in the following directory structure:

    <rig_directory>
    |___10.42.0.28.json
    |___10.42.0.29.json

Where each rig file name contains the IP address of that Xavier.

@subsection dwx_devguide_rec_distrec_step3 Step 3: Synchronize and then run
1. Synchronize the Xavier clocks as described in Synchronizing system clocks in
   these release notes.
2. On the master Xavier, start the recording application, specifying the rig corresponding
   configuration file directory:

       # tools/recorder-tui <rig_directory>
   Recording is automatically toggled on Xavier B, when toggled on Xavier A.

@section dwx_devguide_rec_distrec_synch Sensor Synchronization

NVIDIA DRIVE platforms can record sensor data for various sensor groups on the
same Xavier or for various groups on different Xavier processors or NVIDIA DRIVE
platforms. The sensor setting in the rig configuration file effects sensor
timestamp synchronization. In addition, for recordings on different Xavier
processors, other factors affect timestamp synchronization.

@subsection dwx_devguide_rec_distrec_sync_same_1 Synchronizing for Distributed Recording
By default, the recorder tool does not provide synchronization in a distributed
recording setting because PTP is off. But if you synchronize the system clocks
with PTP or gPTP, then you can get synchronization within +/- 0.5 frames. If you
record when PTP is off, recording timestamps are based on the current Xavier
time, which might be not the real time when the recording takes place.

Cross-CSI sync is enabled automatically makes all cameras capture at the same time,
HW wise. This will enable the external FSYNC feature of the deserializer and allow
the Xavier PWM output to drive the camera sync for multiple camera groups

#### About synchronization
There are two types of synchronization.
- Synchronization of system clocks - Ensures that the timestamp from one Xavier
  is the same time as another Xavier.

  With PTP or gPTP, you can achieve synchronization of system clocks.
- Synchronization of camera frames - Ensures that the time of the capture
  is the same among multiple cameras.

Synchronization of camera frames is not possible. During distributed recording,
the Xavier processors control separate deserializers. They are not connected by
the same PWM. Even though clocks are synchronized, sensor timing cannot be
synchronized. As a result, the actual timing of the image capture cannot be
synchronized, meaning that synchronization will be off by at most +/- 0.5
frames.

#### How gPTP and PTP affect synchronization of system clocks

The NVIDIA DRIVE PDK gPTP application provides the NVIDIA DRIVE platform
support for 802.1AS conforming time synchronization messages for hardware time
synchronization. gPTP is for multi-Xavier use cases (such as distributed
recording), which requires time scales of each Xavier to be synchronized with
each other. NVIDIA recommends enabling gPTP at all times.

#### Synchronizing System Clocks

Use either PTP or gPTP to synchronize between one Xavier as master and 3 other
Xaviers as slaves. Depending on the execution timing, you should have +/- 0.5
frames difference between sensor timestamps when measuring the same camera with
same framerate.

If PTP is running, have all peers start/stop recording at a predetermined
timestamp value. For guidance on using gPTP, search the
_NVIDIA DRIVE OS 5.1 Development Guide_ for `PTP`.

@subsection dwx_devguide_rec_distrec_testing Testing Sensor Synchronization

You cannot compare the timestamp of the first frame to see if PTP sync worked.
Recording does not start synchronously on both Xaviers. Instead, you must
correlate images and then compare the timestamps on those images.

#### To test sensor synchronization
1. Observe with cameras from two Xaviers that are recording the same content,
   for example a video playing on a monitor.
2. Find corresponding frames in both video streams, by comparing the frame content.
3. Compare the timestamp of both frames.
4. Ensure the timestamps are within +/- 0.5 frames.