Function TRITONSERVER_ServerModelBatchProperties

Function Documentation

TRITONSERVER_Error *TRITONSERVER_ServerModelBatchProperties(TRITONSERVER_Server *server, const char *model_name, const int64_t model_version, uint32_t *flags, void **voidp)

Get the batch properties of the model.

The properties are communicated by a flags value and an (optional) object returned by ‘voidp’.

  • TRITONSERVER_BATCH_UNKNOWN: Triton cannot determine the batching properties of the model. This means that the model does not support batching in any way that is useable by Triton. The returned ‘voidp’ value is nullptr.

  • TRITONSERVER_BATCH_FIRST_DIM: The model supports batching along the first dimension of every input and output tensor. Triton schedulers that perform batching can automatically batch inference requests along this dimension. The returned ‘voidp’ value is nullptr.

Return

a TRITONSERVER_Error indicating success or failure.

Parameters

  • server: The inference server object.

  • model_name: The name of the model.

  • model_version: The version of the model. If -1 then the server will choose a version based on the model’s policy.

  • flags: Returns flags indicating the batch properties of the model.

  • voidp: If non-nullptr, returns a point specific to the ‘flags’ value.