1.2. cuDLA API

This section describes the application programming interface of the cuDLA driver.

Functions

cudlaStatus cudlaCreateDevice(uint64_t const device, cudlaDevHandle *const devHandle, uint32_t const flags): Create a device handle.
cudlaStatus cudlaDestroyDevice(cudlaDevHandle const devHandle): Destroy device handle.
cudlaStatus cudlaDeviceGetAttribute(cudlaDevHandle const devHandle, cudlaDevAttributeType const attrib, cudlaDevAttribute *const pAttribute): Get cuDLA device attributes.
cudlaStatus cudlaDeviceGetCount(uint64_t *const pNumDevices): Get device count.
cudlaStatus cudlaGetLastError(cudlaDevHandle const devHandle): Gets the last asynchronous error in task execution.
cudlaStatus cudlaGetNvSciSyncAttributes(uint64_t *attrList, uint32_t const flags): Get cuDLA's NvSciSync attributes.
cudlaStatus cudlaGetVersion(uint64_t *const version): Returns the version number of the library.
cudlaStatus cudlaImportExternalMemory(cudlaDevHandle const devHandle, const cudlaExternalMemoryHandleDesc *const desc, uint64_t **const devPtr, uint32_t const flags): Imports external memory into cuDLA.
cudlaStatus cudlaImportExternalSemaphore(cudlaDevHandle const devHandle, const cudlaExternalSemaphoreHandleDesc *const desc, uint64_t **const devPtr, uint32_t const flags): Imports external semaphore into cuDLA.
cudlaStatus cudlaMemRegister(cudlaDevHandle const devHandle, const uint64_t *const ptr, size_t const size, uint64_t **const devPtr, uint32_t const flags): Registers the CUDA memory to DLA engine.
cudlaStatus cudlaMemUnregister(cudlaDevHandle const devHandle, const uint64_t *const devPtr): Unregisters the input memory from DLA engine.
cudlaStatus cudlaModuleGetAttributes(cudlaModule const hModule, cudlaModuleAttributeType const attrType, cudlaModuleAttribute *const attribute): Get DLA module attributes.
cudlaStatus cudlaModuleLoadFromMemory(cudlaDevHandle const devHandle, const uint8_t *const pModule, size_t const moduleSize, cudlaModule *const hModule, uint32_t const flags): Load a DLA module.
cudlaStatus cudlaModuleUnload(cudlaModule const hModule, uint32_t const flags): Unload a DLA module.
cudlaStatus cudlaSetTaskTimeoutInMs(cudlaDevHandle const devHandle, uint32_t const timeout): Set task timeout in millisecond.
cudlaStatus cudlaSubmitTask(cudlaDevHandle const devHandle, const cudlaTask *const ptrToTasks, uint32_t const numTasks, void *const stream, uint32_t const flags): Submits the inference operation on DLA.

1.2.1. Functions

cudlaStatus cudlaCreateDevice(uint64_t const device, cudlaDevHandle *const devHandle, uint32_t const flags)

Create a device handle.

Creates an instance of a cuDLA device which can be used to submit DLA operations. The application can create the handle in hybrid or standalone mode. In hybrid mode, the current set GPU device is used by this API to decide the association of the created DLA device handle. This function returns cudlaErrorUnsupportedOperation if the current set GPU device is a dGPU as cuDLA is not supported on dGPU presently. cuDLA supports 16 cuDLA device handles per DLA HW instance.

A scratch-memory configuration is OR-ed with the mode:

CUDLA_SCRATCH_MEMORY_SHARED_STATIC - Modules loaded on this handle share a single fixed-size pinned Local DRAM scratch pool instead of each allocating a private buffer. Loadables that do not fit the pool fall back to a private buffer. This flag must be OR-ed with a device mode (CUDLA_CUDA_DLA or CUDLA_STANDALONE).

Parameters

device – [in] - Device number (can be 0 or 1).
devHandle – [out] - Pointer to hold the created cuDLA device handle.
flags – [in] - Flags controlling device creation. Valid values for flags are:
- CUDLA_CUDA_DLA - In this mode, cuDLA serves as a programming model extension of CUDA wherein DLA work can be submitted using CUDA constructs.
- CUDLA_STANDALONE - In this mode, cuDLA works standalone without any interaction with CUDA.

Returns

cudlaSuccess, cudlaErrorOutOfResources, cudlaErrorInvalidParam, cudlaErrorIncompatibleDlaSWVersion, cudlaErrorCreationFailed, cudlaErrorCuda, cudlaErrorUmd, cudlaErrorUnsupportedOperation, cudlaErrorDriverNotFound

cudlaStatus cudlaDestroyDevice(cudlaDevHandle const devHandle)

Destroy device handle.

Destroys the instance of the cuDLA device which was created with cudlaCreateDevice. Before destroying the handle, it is important to ensure that all the tasks submitted previously to the device are completed. Failure to do so can lead to application crashes.

In hybrid mode, cuDLA internally performs memory allocations with CUDA using the primary context. As a result, before destroying or resetting a CUDA primary context, it is mandatory that all cuDLA device initializations are destroyed.

Note

This API can return task execution errors from previous DLA task submissions.

Parameters: devHandle – [in] - A valid device handle.
Returns: cudlaSuccess, cudlaErrorInvalidDevice, cudlaErrorCuda, cudlaErrorUmd, cudlaErrorDriverNotFound

cudlaStatus cudlaDeviceGetAttribute(cudlaDevHandle const devHandle, cudlaDevAttributeType const attrib, cudlaDevAttribute *const pAttribute)

Get cuDLA device attributes.

UVA addressing between CUDA and DLA requires special support in the underlying kernel mode drivers. Applications are expected to query the cuDLA runtime to check if the current version of cuDLA supports UVA addressing.

Note

This API can return task execution errors from previous DLA task submissions.

Parameters

devHandle – [in] - The input cuDLA device handle.
attrib – [in] - The attribute that is being requested.
pAttribute – [out] - The output pointer where the attribute will be available.

Returns

cudlaSuccess, cudlaErrorInvalidParam, cudlaErrorInvalidDevice, cudlaErrorUmd, cudlaErrorInvalidAttribute, cudlaErrorDriverNotFound

cudlaStatus cudlaDeviceGetCount(uint64_t *const pNumDevices)

Get device count.

Get number of DLA devices available to use.

Parameters: pNumDevices – [out] - The number of DLA devices will be available in this variable upon successful completion.
Returns: cudlaSuccess, cudlaErrorInvalidParam, cudlaErrorUmd, cudlaErrorIncompatibleDlaSWVersion, cudlaErrorDriverNotFound

cudlaStatus cudlaGetLastError(cudlaDevHandle const devHandle)

Gets the last asynchronous error in task execution.

The DLA tasks execute asynchronously on the DLA HW. As a result, the status of the task execution is not known at the time of task submission. The status of the task executed by the DLA HW most recently for the particular device handle can be queried using this interface.

Note that a return code of cudlaSuccess from this function does not necessarily imply that most recent task executed successfully. Since this function returns immediately, it can only report the status of the tasks at the snapshot of time when it is called. To be guaranteed of task completion, applications must synchronize on the submitted tasks in hybrid or standalone modes and then call this API to check for errors.

Parameters: devHandle – [in] - A valid device handle.
Returns: cudlaSuccess, cudlaErrorInvalidDevice, cudlaErrorDlaErrInvalidInput, cudlaErrorDlaErrInvalidPreAction, cudlaErrorDlaErrNoMem, cudlaErrorDlaErrProcessorBusy, cudlaErrorDlaErrTaskStatusMismatch, cudlaErrorDlaErrEngineTimeout, cudlaErrorDlaErrDataMismatch, cudlaErrorUnknown, cudlaErrorDriverNotFound

cudlaStatus cudlaGetNvSciSyncAttributes(uint64_t *attrList, uint32_t const flags)

Get cuDLA’s NvSciSync attributes.

Gets the NvSciSync’s attributes in the attribute list created by the application.

cuDLA supports three types of NvSciSync object primitives -

Sync point
Regular semaphore
Deterministic semaphore cuDLA prioritizes sync point primitive over regular and deterministic semaphore primitives by default and sets these priorities in the NvSciSync attribute list.

For Deterministic semaphore, NvSciSync attribute list used to create the NvSciSync object must have value of NvSciSyncAttrKey_RequireDeterministicFences key set to true.

cuDLA also supports Timestamp feature on NvSciSync objects. Waiter can request for this by setting NvSciSync attribute “NvSciSyncAttrKey_WaiterRequireTimestamps” as true.

In the event of failed NvSci initialization this function would return cudlaErrorUnsupportedOperation. This function can return cudlaErrorNvSci or cudlaErrorInvalidAttribute in certain cases when the underlying NvSci operation fails.

This API updates the input nvSciSyncAttrList with values equivalent to the following public attribute key-values:

NvSciSyncAttrKey_RequiredPerm is set to

NvSciSyncAccessPerm_SignalOnly if value of flag is set to CUDLA_NVSCISYNC_ATTR_WAIT.
NvSciSyncAccessPerm_WaitOnly if value of flag is set to CUDLA_NVSCISYNC_ATTR_SIGNAL.
NvSciSyncAccessPerm_WaitSignal if value of flag is set to CUDLA_NVSCISYNC_ATTR_SIGNAL | CUDLA_NVSCISYNC_ATTR_WAIT.

As NvSciSyncAttrKey_RequiredPerm is internally set by cuDLA, setting this value by the application is disallowed.

Note

Users of cuDLA can only append attributes to output attrList using NvSci API, modifying already populated values of the output attrList can result in undefined behavior.

Parameters

attrList – [out] - Attribute list created by the application.
flags – [in] - Applications can use this flag to specify how they intend to use the NvSciSync object created from the attrList. The valid values of flags can be one of the following (or an OR of these values):
- CUDLA_NVSCISYNC_ATTR_WAIT, specifies that the application intend to use the NvSciSync object created using this attribute list as a waiter in cuDLA and therefore needs cuDLA to fill waiter specific NvSciSyncAttr.
- CUDLA_NVSCISYNC_ATTR_SIGNAL, specifies that the application intend to use the NvSciSync object created using this attribute list as a signaler in cuDLA and therefore needs cuDLA to fill signaler specific NvSciSyncAttr.

Returns

cudlaSuccess, The API call returned with no errors.
cudlaErrorInvalidParam, This API call failed because invalid parameter attrList was passed.
cudlaErrorUnsupportedOperation, This error code indicates that the API call failed because the operation is not supported in hybrid mode.
cudlaErrorInvalidAttribute, The API call failed as parameter attrList has invalid values.
cudlaErrorNvSci, This error code indicates error in the NvSci operation as part of the API call.
::cudlaErrorNotPermittedOperation, This error code indicates that the API call is not permitted when DRIVE OS is in Operational state.
cudlaErrorUnknown, This error code indicates that an unknown error has occurred.
cudlaErrorDriverNotFound, This indicates that the cuDLA driver library cannot be loaded.

cudlaStatus cudlaGetVersion(uint64_t *const version)

Returns the version number of the library.

cuDLA is semantically versioned. This function will return the version as 1000000*major + 1000*minor + patch.

Parameters: version – [out] - cuDLA library version will be available in this variable upon successful execution.
Returns: cudlaSuccess, cudlaErrorInvalidParam, cudlaErrorDriverNotFound

cudlaStatus cudlaImportExternalMemory(cudlaDevHandle const devHandle, const cudlaExternalMemoryHandleDesc *const desc, uint64_t **const devPtr, uint32_t const flags)

Imports external memory into cuDLA.

Imports the allocated external memory by registering it with DLA. After successful registration, the returned pointer can be used in a task submit.

On Tegra, cuDLA supports importing NvSciBuf objects in standalone mode only. In the event of failed NvSci initialization (either due to usage of this API in hybrid mode or an issue in the NvSci library initialization), this function would return cudlaErrorUnsupportedOperation. This function can return cudlaErrorNvSci or cudlaErrorInvalidAttribute in certain cases when the underlying NvSci operation fails.

Note

cuDLA only supports importing NvSciBuf objects of type NvSciBufType_RawBuffer or NvSciBufType_Tensor. Importing NvSciBuf object of any other type can result in an undefined behaviour.

Note

This API may result in undefined behavior if the address being registered is not 32-byte aligned. The input pointer devPtr must always satisfy the condition ((devPtr & 0x1F) == 0)

Note

This API can return task execution errors from previous DLA task submissions.

Parameters

devHandle – [in] - A valid device handle.
desc – [in] - Contains description about allocated external memory.
devPtr – [out] - The output pointer where the mapping will be available.
flags – [in] - Application can use this flag to specify the memory access permissions of the memory that needs to be registered with DLA. The valid values of flags can be one of the following:
- CUDLA_READ_WRITE_PERM, specifies that the external memory needs to be registered with DLA as read-write memory.
- CUDLA_READ_ONLY_PERM, specifies that the external memory needs to be registered with DLA as read-only memory.
- CUDLA_TASK_STATISTICS, specifies that the external memory needs to be registered with DLA for layerwise statistics.

Returns

cudlaSuccess, cudlaErrorInvalidParam, cudlaErrorInvalidDevice, cudlaErrorUnsupportedOperation, cudlaErrorNvSci, cudlaErrorInvalidAttribute, cudlaErrorMemoryRegistered, cudlaErrorUmd, cudlaErrorDriverNotFound

cudlaStatus cudlaImportExternalSemaphore(cudlaDevHandle const devHandle, const cudlaExternalSemaphoreHandleDesc *const desc, uint64_t **const devPtr, uint32_t const flags)

Imports external semaphore into cuDLA.

Imports the allocated external semaphore by registering it with DLA. After successful registration, the returned pointer can be used in a task submission to signal synchronization objects.

On Tegra, cuDLA supports importing NvSciSync objects in standalone mode only. NvSciSync object primitives that cuDLA supports are sync point and deterministic semaphore.

cuDLA also supports Timestamp feature on NvSciSync objects, using which the user can get a snapshot of DLA clock at which a particular fence is signaled. At any point in time there are only 512 valid timestamp buffers that can be associated with fences. For example, If User has created 513 fences from a single NvSciSync object with timestamp enabled then the timestamp buffer associated with 1st fence is same as with 513th fence.

In the event of failed NvSci initialization (either due to usage of this API in hybrid mode or an issue in the NvSci library initialization), this function would return cudlaErrorUnsupportedOperation. This function can return cudlaErrorNvSci or cudlaErrorInvalidAttribute in certain cases when the underlying NvSci operation fails.

Note

This API can return task execution errors from previous DLA task submissions.

Parameters

devHandle – [in] - A valid device handle.
desc – [in] - Contains sempahore object.
devPtr – [out] - The output pointer where the mapping will be available.
flags – [in] - Reserved for future. Must be set to 0.

Returns

cudlaSuccess, cudlaErrorInvalidParam, cudlaErrorInvalidDevice, cudlaErrorUnsupportedOperation, cudlaErrorNvSci, cudlaErrorInvalidAttribute, cudlaErrorMemoryRegistered, cudlaErrorDriverNotFound

cudlaStatus cudlaMemRegister(cudlaDevHandle const devHandle, const uint64_t *const ptr, size_t const size, uint64_t **const devPtr, uint32_t const flags)

Registers the CUDA memory to DLA engine.

As part of registration, a system mapping is created whereby the DLA HW can access the underlying CUDA memory. The resultant mapping is available in devPtr and applications must use this mapping while referring this memory in submit operations.

This function will return cudlaErrorInvalidAddress if the pointer or size to be registered is invalid. In addition, if the input pointer was already registered, then this function will return cudlaErrorMemoryRegistered. Attempting to re-register memory does not cause any irrecoverable error in cuDLA and applications can continue to use cuDLA APIs even after this error has occurred.

Note

This API may result in undefined behavior if the address being registered is not 32-byte aligned. The input pointer ptr must always satisfy the condition ((ptr & 0x1F) == 0)

Note

This API can return task execution errors from previous DLA task submissions.

Parameters

devHandle – [in] - A valid cuDLA device handle create by a previous call to cudlaCreateDevice.
ptr – [in] - The CUDA pointer to be registered.
size – [in] - The size of the mapping i.e the number of bytes from ptr that must be mapped.
devPtr – [out] - The output pointer where the mapping will be available.
flags – [in] - Applications can use this flag to control several aspects of the registration process. The valid values of flags can be one of the following (or an OR of these values):
- 0, default
- CUDLA_TASK_STATISTICS, specifies that the external memory needs to be registered with DLA for layerwise statistics.

Returns

cudlaSuccess, cudlaErrorInvalidDevice, cudlaErrorInvalidParam, cudlaErrorInvalidAddress, cudlaErrorCuda, cudlaErrorUmd, cudlaErrorOutOfResources, cudlaErrorMemoryRegistered, cudlaErrorUnsupportedOperation, cudlaErrorDriverNotFound

cudlaStatus cudlaMemUnregister(cudlaDevHandle const devHandle, const uint64_t *const devPtr)

Unregisters the input memory from DLA engine.

The system mapping that enables the DLA HW to access the memory is removed. This mapping could have been created by a previous call to cudlaMemRegister , cudlaImportExternalMemory or cudlaImportExternalSemaphore.

Note

This API can return task execution errors from previous DLA task submissions.

Parameters

devHandle – [in] - A valid cuDLA device handle create by a previous call to cudlaCreateDevice.
devPtr – [in] - The pointer to be unregistered.

Returns

cudlaSuccess, cudlaErrorInvalidDevice, cudlaErrorInvalidAddress, cudlaErrorUmd, cudlaErrorDriverNotFound

cudlaStatus cudlaModuleGetAttributes(cudlaModule const hModule, cudlaModuleAttributeType const attrType, cudlaModuleAttribute *const attribute)

Get DLA module attributes.

Get module attributes from the loaded module. This API returns cudlaErrorInvalidDevice if the module is not loaded in any device.

Note

This API can return task execution errors from previous DLA task submissions.

Parameters

hModule – [in] - The input DLA module.
attrType – [in] - The attribute type that is being requested.
attribute – [out] - The output pointer where the attribute will be available.

Returns

cudlaSuccess, cudlaErrorInvalidParam, cudlaErrorInvalidModule, cudlaErrorInvalidDevice, cudlaErrorUmd, cudlaErrorInvalidAttribute, cudlaErrorUnsupportedOperation, cudlaErrorDriverNotFound

cudlaStatus cudlaModuleLoadFromMemory(cudlaDevHandle const devHandle, const uint8_t *const pModule, size_t const moduleSize, cudlaModule *const hModule, uint32_t const flags)

Load a DLA module.

Loads the module into the current device handle.

Multiple loadables are not allowed to load onto single cuDLA device handle.
A Loadable can only be loaded once in cuDLA device handle lifecycle.

Note

This API can return task execution errors from previous DLA task submissions.

Parameters

devHandle – [in] - The input cuDLA device handle. The module will be loaded in the context of this handle.
pModule – [in] - A pointer to an in-memory module.
moduleSize – [in] - The size of the module.
hModule – [out] - The address in which the loaded module handle will be available upon successful execution.
flags – [in] - Applications can use this flag to specify how the module is going to be used. The valid values of flags can be one of the following:
- CUDLA_MODULE_DEFAULT, Default value which is 0.
- CUDLA_MODULE_ENABLE_FAULT_DIAGNOSTICS, Application can specify this flag to load a module that is used for performing fault diagnostics for DLA HW. With this flag set, the pModule and moduleSize parameters shall be NULL and 0 as the diagnostics module is loaded internally.

Returns

cudlaSuccess, cudlaErrorInvalidDevice, cudlaErrorInvalidParam, cudlaErrorOutOfResources, cudlaErrorUnsupportedOperation, cudlaErrorUmd, cudlaErrorDriverNotFound

cudlaStatus cudlaModuleUnload(cudlaModule const hModule, uint32_t const flags)

Unload a DLA module.

Unload the module from the device handle that it was loaded into. This API returns cudlaErrorInvalidDevice if the module is not loaded into a valid device.

Note

This API can return task execution errors from previous DLA task submissions.

Parameters

hModule – [in] - Handle to the loaded module.
flags – [in] - Reserved for future. Must be set to 0.

Returns

cudlaSuccess, cudlaErrorInvalidParam, cudlaErrorInvalidDevice, cudlaErrorInvalidModule, cudlaErrorUmd, cudlaErrorDriverNotFound

cudlaStatus cudlaSetTaskTimeoutInMs(cudlaDevHandle const devHandle, uint32_t const timeout)

Set task timeout in millisecond.

Set task timeout in ms for each device handle. cuDLA sets 30 seconds as default timeout value if user doesn’t explicitly set the timeout.

In case , device handle is invalid or timeout is 0 or timeout is greater than 1000 sec, this function would return cudlaErrorInvalidParam otherwise cudlaSuccess.

Note

This API can return task execution errors from previous DLA task submissions.

Parameters

devHandle – [in] - A valid device handle.
timeout – [in] - task timeout value in ms.

Returns

cudlaSuccess, cudlaErrorInvalidParam, cudlaErrorDriverNotFound

cudlaStatus cudlaSubmitTask(cudlaDevHandle const devHandle, const cudlaTask *const ptrToTasks, uint32_t const numTasks, void *const stream, uint32_t const flags)

Submits the inference operation on DLA.

This operation takes in a sequence of tasks and submits them to the DLA HW for execution in the same sequence as they appear in the input task array. The input and output tensors (and statistics buffer if used) are assumed to be pre-registered using cudlaMemRegister (in hybrid mode) or cudlaImportExternalMemory (in standalone mode). Failure to do so can result in this function returning cudlaErrorInvalidAddress.

The stream parameter must be specified as the CUDA stream on which the DLA task is submitted for execution in hybrid mode. In standalone mode, this parameter must be passed as NULL and failure to do so will result in this function returning cudlaErrorInvalidParam.

The cudlaTask structure has a provision to specify wait and signal events that cuDLA must wait on and signal respectively as part of cudlaSubmitTask(). In hybrid mode, the waitEvents and signalEvents fields of the cudlaTask structure must be set to NULL. Wait and signal events are only supported in standalone mode. Failure to set these fields to NULL in hybrid mode will result in this function returning cudlaErrorUnsupportedOperation.

In standalone mode, each submitted task will wait for all its wait events to be signaled before beginning execution and will provide a signal event (if one is requested for during cudlaSubmitTask) that the application (or any other entity) can wait on to ensure that the submitted task has completed execution. In cuDLA 1.0, only NvSciSync fences are supported as part of wait events. Furthermore, only NvSciSync objects (registered as part of cudlaImportExternalSemaphore) can be signaled as part of signal events and the fence corresponding to the signaled event is returned as part of cudlaSubmitTask.

Event-Only submissions - In standalone mode, if inputTensor and outputTensor fields are set to NULL inside the cudlaTask structure, the task submission is interpreted as an enqueue of wait and signal events that must be considered for subsequent task submission. No actual task submission is done. Multiple such subsequent task submissions with NULL fields in the input/outputTensor fields will overwrite the list of wait and signal events to be considered. In other words, the latest non-null wait events and/or latest non-null signal events before a non-null submission are considered for subsequent actual task submission. During an actual task submit in standalone mode, the effective wait events and signal events that will be considered are what the application sets using NULL data submissions and what is set for that particular task submission in the waitEvents and signalEvents fields. The wait events set as part of NULL data submission are considered as dependencies for only the first task and the signal events set as part of NULL data submission are signaled when the last task of task list is complete. All constraints that apply to waitEvents and signalEvents individually (as described below) are also applicable to the combined list.

Configurations allowed in cuDLA for a task submission