Struct InferOptions

Struct Documentation

struct InferOptions

Structure to hold options for Inference Request.

Public Functions

InferOptions(const std::string &model_name)

Public Members

std::string model_name_

The name of the model to run inference.

std::string model_version_

The version of the model to use while running inference.

The default value is an empty string which means the server will select the version of the model based on its internal policy.

std::string request_id_

An identifier for the request.

If specified will be returned in the response. Default value is an empty string which means no request_id will be used.

uint64_t sequence_id_

The unique identifier for the sequence being represented by the object.

Default value is 0 which means that the request does not belong to a sequence.

bool sequence_start_

Indicates whether the request being added marks the start of the sequence.

Default value is False. This argument is ignored if ‘sequence_id’ is 0.

bool sequence_end_

Indicates whether the request being added marks the end of the sequence.

Default value is False. This argument is ignored if ‘sequence_id’ is 0.

uint64_t priority_

Indicates the priority of the request.

Priority value zero indicates that the default priority level should be used (i.e. same behavior as not specifying the priority parameter). Lower value priorities indicate higher priority levels. Thus the highest priority level is indicated by setting the parameter to 1, the next highest is 2, etc. If not provided, the server will handle the request using default setting for the model.

uint64_t timeout_

The timeout value for the request, in microseconds.

If the request cannot be completed within the time the server can take a model-specific action such as terminating the request. If not provided, the server will handle the request using default setting for the model.