CudaExtension

Extension for CUDA operations.

  • UUID: d63a98fa-7882-11eb-a917-b38f664f399c

  • Version: 2.4.0

  • Author: NVIDIA

  • License: LICENSE

Components

nvidia::gxf::CudaStream

Holds and provides access to native cudaStream_t.

nvidia::gxf::CudaStream handle must be allocated by nvidia::gxf::CudaStreamPool. Its lifecycle is valid until explicitly recycled through nvidia::gxf::CudaStreamPool.releaseStream() or implicitly until nvidia::gxf::CudaStreamPool is deactivated.

You may call stream() to get the native cudaStream_t handle, and to submit GPU operations. After the submission, GPU takes over the input tensors/buffers and keeps them in use. To prevent host carelessly releasing these in-use buffers, CUDA Codelet needs to call record(event, input_entity, sync_cb) to extend input_entity’s lifecycle until GPU completely consumes it. Alternatively, you may call record(event, event_destroy_cb) for native cudaEvent_t operations and free in-use resource via event_destroy_cb.

It is required to have a nvidia::gxf::CudaStreamSync in the graph pipeline after all the CUDA operations. See more details in nvidia::gxf::CudaStreamSync

  • Component ID: 5683d692-7884-11eb-9338-c3be62d576be

  • Defined in: gxf/cuda/cuda_stream.hpp

nvidia::gxf::CudaStreamId

Holds CUDA stream Id to deduce nvidia::gxf::CudaStream handle.

stream_cid should be nvidia::gxf::CudaStream component id.

  • Component ID: 7982aeac-37f1-41be-ade8-6f00b4b5d47c

  • Defined in: gxf/cuda/cuda_stream_id.hpp

nvidia::gxf::CudaEvent

Holds and provides access to native cudaEvent_t handle.

When a nvidia::gxf::CudaEvent is created, you’ll need to initialize a native cudaEvent_t through init(flags, dev_id), or set third party event through initWithEvent(event, dev_id, free_fnc). The event keeps valid until deinit is called explicitly otherwise gets recycled in destructor.

  • Component ID: f5388d5c-a709-47e7-86c4-171779bc64f3

  • Defined in: gxf/cuda/cuda_event.hpp

nvidia::gxf::CudaStreamPool

CudaStream allocation.

You must explicitly call allocateStream() to get a valid nvidia::gxf::CudaStream handle. This component would hold all the its allocated nvidia::gxf::CudaStream entities until releaseStream(stream) is called explicitly or the CudaStreamPool component is deactivated.

  • Component ID: 6733bf8b-ba5e-4fae-b596-af2d1269d0e7

  • Base Type: nvidia::gxf::Allocator

Parameters

dev_id

GPU device id.

  • Flags: GXF_PARAMETER_FLAGS_NONE

  • Type: GXF_PARAMETER_TYPE_INT32

  • Default Value: 0


stream_flags

Flag values to create CUDA streams.

  • Flags: GXF_PARAMETER_FLAGS_NONE

  • Type: GXF_PARAMETER_TYPE_INT32

  • Default Value: 0


stream_priority

Priority values to create CUDA streams.

  • Flags: GXF_PARAMETER_FLAGS_NONE

  • Type: GXF_PARAMETER_TYPE_INT32

  • Default Value: 0


reserved_size

User-specified file name without extension.

  • Flags: GXF_PARAMETER_FLAGS_NONE

  • Type: GXF_PARAMETER_TYPE_INT32

  • Default Value: 1

max_size

Maximum Stream Size.

  • Flags: GXF_PARAMETER_FLAGS_NONE

  • Type: GXF_PARAMETER_TYPE_INT32

  • Default Value: 0, no limitation.

nvidia::gxf::CudaStreamSync

Synchronize all CUDA streams which are carried by message entities.

This codelet is required to get connected in the graph pipeline after all CUDA ops codelets. When a message entity is received, it would find all of the nvidia::gxf::CudaStreamId in that message, and extract out each nvidia::gxf::CudaStream. With each CudaStream handle, it synchronizes all previous nvidia::gxf::CudaStream.record() events, along with all submitted GPU operations before this point.

Note

CudaStreamSync must be set in the graph when nvidia::gxf::CudaStream.record() is used, otherwise it may cause memory leak.

  • Component ID: 0d1d8142-6648-485d-97d5-277eed00129c

  • Base Type: nvidia::gxf::Codelet

Parameters

rx

Receiver to receive all messages carrying nvidia::gxf::CudaStreamId.

  • Flags: GXF_PARAMETER_FLAGS_NONE

  • Type: GXF_PARAMETER_TYPE_HANDLE

  • Handle Type: nvidia::gxf::Receiver


tx

Transmitter to send messages to downstream.

  • Flags: GXF_PARAMETER_FLAGS_OPTIONAL

  • Type: GXF_PARAMETER_TYPE_HANDLE

  • Handle Type: nvidia::gxf::Transmitter