Parameters Extension
Contents
Parameters Extension#
This document describes Triton’s parameters extension. The parameters extension allows an inference request to provide custom parameters that cannot be provided as inputs. Because this extension is supported, Triton reports “parameters” in the extensions field of its Server Metadata. This extension uses the optional “parameters” field in the KServe Protocol in HTTP and GRPC.
The following parameters are reserved for Triton’s usage and should not be used as custom parameters:
sequence_id
priority
timeout
sequence_start
sequence_end
All the keys that start with “triton_” prefix.
headers
When using both GRPC and HTTP endpoints, you need to make sure to not use the reserved parameters list to avoid unexpected behavior. The reserved parameters are not accessible in the Triton C-API.
HTTP/REST#
The following example shows how a request can include custom parameters.
POST /v2/models/mymodel/infer HTTP/1.1
Host: localhost:8000
Content-Type: application/json
Content-Length: <xx>
{
"parameters" : { "my_custom_parameter" : 42 }
"inputs" : [
{
"name" : "input0",
"shape" : [ 2, 2 ],
"datatype" : "UINT32",
"data" : [ 1, 2, 3, 4 ]
}
],
"outputs" : [
{
"name" : "output0",
}
]
}
GRPC#
The parameters
field in the
ModelInferRequest message can be used to send custom parameters.