CudaExtension

Extension for CUDA operations.

• UUID: d63a98fa-7882-11eb-a917-b38f664f399c

• Version: 2.0.0

• Author: NVIDIA

• License: LICENSE

nvidia::gxf::CudaStream

Holds and provides access to native cudaStream_t.

nvidia::gxf::CudaStream handle must be allocated by nvidia::gxf::CudaStreamPool. Its lifecycle is valid until explicitly recycled through nvidia::gxf::CudaStreamPool.releaseStream() or implicitly until nvidia::gxf::CudaStreamPool is deactivated.

You may call stream() to get the native cudaStream_t handle, and to submit GPU operations. After the submission, GPU takes over the input tensors/buffers and keeps them in use. To prevent the host carelessly releasing these in-use buffers, CUDA Codelet needs to call record(event, input_entity, sync_cb) to extend input_entity’s lifecycle until the GPU completely consumes it. Alternatively, you may call record(event, event_destroy_cb) for native cudaEvent_t operations and free in-use resource via event_destroy_cb.

It is required to have a nvidia::gxf::CudaStreamSync in the graph pipeline after all the CUDA operations. See more details in nvidia::gxf::CudaStreamSync.

• Component ID: 5683d692-7884-11eb-9338-c3be62d576be

• Defined in: gxf/cuda/cuda_stream.hpp

nvidia::gxf::CudaStreamId

Holds CUDA stream Id to deduce nvidia::gxf::CudaStream handle.

stream_cid should be nvidia::gxf::CudaStream component id.

• Component ID: 7982aeac-37f1-41be-ade8-6f00b4b5d47c

• Defined in: gxf/cuda/cuda_stream_id.hpp

nvidia::gxf::CudaEvent

Holds and provides access to native cudaEvent_t handle.

When a nvidia::gxf::CudaEvent is created, you’ll need to initialize a native cudaEvent_t through init(flags, dev_id), or set third party event through initWithEvent(event, dev_id, free_fnc). The event keeps valid until deinit is called explicitly otherwise gets recycled in destructor.

• Component ID: f5388d5c-a709-47e7-86c4-171779bc64f3

• Defined in: gxf/cuda/cuda_event.hpp

nvidia::gxf::CudaStreamPool

CudaStream allocation.

You must explicitly call allocateStream() to get a valid nvidia::gxf::CudaStream handle. This component would hold all the its allocated nvidia::gxf::CudaStream entities until releaseStream(stream) is called explicitly or the CudaStreamPool component is deactivated.

• Component ID: 6733bf8b-ba5e-4fae-b596-af2d1269d0e7

• Base Type: nvidia::gxf::Allocator

Parameters

dev_id

GPU device id.

• Flags: GXF_PARAMETER_FLAGS_NONE

• Type: GXF_PARAMETER_TYPE_INT32

• Default Value: 0

stream_flags

Flag values to create CUDA streams.

• Flags: GXF_PARAMETER_FLAGS_NONE

• Type: GXF_PARAMETER_TYPE_INT32

• Default Value: 0

stream_priority

Priority values to create CUDA streams.

• Flags: GXF_PARAMETER_FLAGS_NONE

• Type: GXF_PARAMETER_TYPE_INT32

• Default Value: 0

reserved_size

User-specified file name without extension.

• Flags: GXF_PARAMETER_FLAGS_NONE

• Type: GXF_PARAMETER_TYPE_INT32

• Default Value: 1

max_size

Maximum Stream Size.

• Flags: GXF_PARAMETER_FLAGS_NONE

• Type: GXF_PARAMETER_TYPE_INT32

• Default Value: 0, no limitation.

nvidia::gxf::CudaStreamSync

Synchronize all CUDA streams which are carried by message entities.

This codelet is required to get connected in the graph pipeline after all CUDA ops codelets. When a message entity is received, it would find all of the nvidia::gxf::CudaStreamId in that message, and extract out each nvidia::gxf::CudaStream. With each CudaStream handle, it synchronizes all previous nvidia::gxf::CudaStream.record() events, along with all submitted GPU operations before this point.

• Component ID: 0d1d8142-6648-485d-97d5-277eed00129c

• Base Type: nvidia::gxf::Codelet

Parameters

rx

Receiver to receive all messages carrying nvidia::gxf::CudaStreamId.

• Flags: GXF_PARAMETER_FLAGS_NONE

• Type: GXF_PARAMETER_TYPE_HANDLE

• Handle Type: nvidia::gxf::Receiver

tx

Transmitter to send messages to downstream.

• Flags: GXF_PARAMETER_FLAGS_OPTIONAL

• Type: GXF_PARAMETER_TYPE_HANDLE

• Handle Type: nvidia::gxf::Transmitter