GXF Stream Sync¶
GXF Stream Sync is responsible for synchronization across two CUDA codelets without involving CPU wait. When two CUDA codelets are used, the first CUDA codelet that generates the data or triggers the CUDA kernel is called as the signaler. The second CUDA codelet that waits for the data or for the CUDA job that was submitted by the upstream codelet is called as the waiter. Signaling and waiting is based on a single synchronization object. Signaler and waiter both make use of the same synchronization object. CUDA stream is associated with the signaler and the waiter. The synchronization object provides APIs for signaling and waiting mechanisms.
The signaler codelet upon submitting all the work on a specific CUDA stream, will call the signalSemaphore API of synchronization object. Internally GXF stream sync will make use of a fence to track the completion of the tasks submitted on the CUDA stream. Signaling happens asynchronously on the GPU and the signalSemaphore API returns immediately. signalSemaphore will make use of the same CUDA stream on which the work was submitted. Signaler is also responsible for allocating the synchronization object and passes the same as message entity to the waiter.
The waiter codelet will issue a call to waitSemaphore and submit its own work to the same CUDA stream on which the signaler codelet submitted the work or it may make use of another CUDA stream. GXF stream sync will wait until the fence is signaled which ensures that the work submitted by the signaler codelet is complete. Waiting happens asynchronously on the GPU and the waitSemaphore API returns immediately.
The below figure depicts concept of signaler and waiter
Figure: Synchronization across two CUDA codelets
Extension for synchronization across two CUDA modules without a CPU wait.
Component which helps to achieve synchronization across two CUDA codelets without involving CPU wait. Holds a synchronization object that can be used by the signaler and the waiter.
Component ID: 0011bee7-5d53-43ee-aafa-61485a436bc4
Base Type: nvidia::gxf::Component
Defined in: gxf/stream/stream_nvscisync.hpp
Parameter indicating the type of signaler.
Parameter indicating the type of waiter.
Device id on which signaler is running.
Device id on which waiter is running.
GXF Stream Sync Workflow¶
Cuda to Cuda codelet communication happens with the help of message.
At the Signaler codelet¶
Add StreamSync handle to the mesage.
Get the streamsync Handle.
Allocate Sync Object based on the signaler and waiter
Set cuda Stream for signaler and waiter
Submit work of signaler codelet on CUDA stream.
Signal Semaphore (Asynchronous call)
At the Waiter Codelet¶
Receive the message
Find the streamsync handle
Wait Semaphore (Asynchronous call)
Submit the work of waiter codelet on CUDA stream.
Now wait will happen on the GPU asynchronously
Below example describes on how to make use of GXF Stream Sync in the application.
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61
--- name: global components: - name: cuda_dot_pool type: nvidia::gxf::BlockMemoryPool parameters: storage_type: 1 # cuda block_size: 16384 num_blocks: 10 - name: stream_sync_cuda_to_cuda type: nvidia::gxf::StreamSync parameters: signaler: 1 # Cuda signaler waiter: 3 # Cuda waiter --- name: stream_tensor_generator components: - name: cuda_out type: nvidia::gxf::DoubleBufferTransmitter - name: generator type: nvidia::gxf::stream::test::StreamTensorGeneratorNew parameters: cuda_tx: cuda_out cuda_tensor_pool: global/cuda_pool stream_sync: global/stream_sync_cuda_to_cuda - type: nvidia::gxf::DownstreamReceptiveSchedulingTerm parameters: transmitter: cuda_out min_size: 1 - type: nvidia::gxf::CountSchedulingTerm parameters: count: 50 --- components: - type: nvidia::gxf::Connection parameters: source: stream_tensor_generator/cuda_out target: cuda_dotproduct/rx --- name: cuda_dotproduct components: - name: rx type: nvidia::gxf::DoubleBufferReceiver parameters: capacity: 2 - name: tx type: nvidia::gxf::DoubleBufferTransmitter - type: nvidia::gxf::MessageAvailableSchedulingTerm parameters: receiver: rx min_size: 1 - type: nvidia::gxf::DownstreamReceptiveSchedulingTerm parameters: transmitter: tx min_size: 1 - type: nvidia::gxf::stream::test::CublasDotProductNew parameters: rx: rx tx: tx tensor_pool: global/cuda_dot_pool ---