Function TRTSERVER_ServerInferAsync

Function Documentation

TRTSERVER_Error *TRTSERVER_ServerInferAsync(TRTSERVER_Server *server, TRTSERVER_TraceManager *trace_manager, TRTSERVER_InferenceRequestProvider *request_provider, TRTSERVER_ResponseAllocator *response_allocator, void *response_allocator_userp, TRTSERVER_InferenceCompleteFn_t complete_fn, void *complete_userp)

Perform inference using the meta-data and inputs supplied by the request provider.

The caller retains ownership of ‘request_provider’ but may release it by calling TRTSERVER_InferenceRequestProviderDelete once this function returns.

Return

a TRTSERVER_Error indicating success or failure.

Parameters

  • server: The inference server object.

  • trace_manager: The trace manager object for this request, or nullptr if no tracing.

  • request_provider: The request provider for the request.

  • response_allocator: The TRTSERVER_ResponseAllocator to use to allocate buffers to hold inference results.

  • response_allocator_userp: User-provided pointer that is delivered to the response allocator’s allocation function.

  • complete_fn: The function called when the inference completes.

  • complete_userp: User-provided pointer that is delivered to the completion function.