Class InferGrpcContext

Inheritance Relationships

Base Type

Class Documentation

class InferGrpcContext : public nvidia::inferenceserver::client::InferContext

InferGrpcContext is the gRPC instantiation of InferContext.

Public Functions

~InferGrpcContext()
Error Run(std::vector<std::unique_ptr<Result>> *results)

Send a synchronous request to the inference server to perform an inference to produce results for the outputs specified in the most recent call to SetRunOptions().

The Result objects holding the output values are returned in the same order as the outputs are specified in the options.

Return
Error object indicating success or failure.
Parameters
  • results: Returns Result objects holding inference results.

Error AsyncRun(std::shared_ptr<Request> *async_request)

Send an asynchronous request to the inference server to perform an inference to produce results for the outputs specified in the most recent call to SetRunOptions().

Return
Error object indicating success or failure.
Parameters
  • async_request: Returns a Request object that can be used to retrieve the inference results for the request.

Error GetAsyncRunResults(std::vector<std::unique_ptr<Result>> *results, const std::shared_ptr<Request> &async_request, bool wait)

Get the results of the asynchronous request referenced by ‘async_request’.

The Result objects holding the output values are returned in the same order as the outputs are specified in the options when AsyncRun() was called.

Return
Error object indicating success or failure. Success will be returned only if the request has been completed succesfully. UNAVAILABLE will be returned if ‘wait’ is false and the request is not ready.
Parameters
  • results: Return Result objects holding inference results.
  • async_request: Request handle to retrieve results.
  • wait: If true, block until the request completes. Otherwise, return immediately.

Public Static Functions

static Error Create(std::unique_ptr<InferContext> *ctx, const std::string &server_url, const std::string &model_name, int model_version = -1, bool verbose = false)

Create context that performs inference for a model using gRPC protocol.

Return
Error object indicating success or failure.
Parameters
  • ctx: Returns a new InferGrpcContext object.
  • server_url: The inference server name and port.
  • model_name: The name of the model to get status for.
  • model_version: The version of the model to use for inference, or -1 to indicate that the latest (i.e. highest version number) version should be used.
  • verbose: If true generate verbose output when contacting the inference server.