Typedef TRITONSERVER_InferenceRequestReleaseFn_t

Typedef Documentation

typedef void (*TRITONSERVER_InferenceRequestReleaseFn_t)(TRITONSERVER_InferenceRequest *request, const uint32_t flags, void *userp)

Type for inference request release callback function.

The callback indicates what type of release is being performed on the request and for some of these the callback function takes ownership of the TRITONSERVER_InferenceRequest object. The ‘userp’ data is the data provided as ‘request_release_userp’ in the call to TRITONSERVER_InferenceRequestSetReleaseCallback.

One or more flags will be specified when the callback is invoked, and the callback must take the following actions:

  • TRITONSERVER_REQUEST_RELEASE_ALL: The entire inference request is being released and ownership is passed to the callback function. Triton will not longer access the ‘request’ object itself nor any input tensor data associated with the request. The callback should free or otherwise manage the ‘request’ object and all associated tensor data.

Note that currently TRITONSERVER_REQUEST_RELEASE_ALL should always be set when the callback is invoked but in the future that may change, so the callback should explicitly check for the flag before taking ownership of the request object.