Capabilities¶
The following table shows which backends support each major inference server feature. See Datatypes for information on data-types supported by each backend.
Feature |
TensorRT |
TensorFlow |
Caffe2 |
ONNX Runtime |
PyTorch |
Custom |
|---|---|---|---|---|---|---|
Multi-GPU |
Yes |
Yes |
Yes |
Yes |
Yes |
Yes |
Multi-Model |
Yes |
Yes |
Yes |
Yes |
Yes |
Yes |
Batching |
Yes |
Yes |
Yes |
Yes |
Yes |
Yes |
Dynamic Batching |
Yes |
Yes |
Yes |
Yes |
Yes |
Yes |
Sequence Batching |
Yes |
Yes |
Yes |
Yes |
Yes |
Yes |
Variable-Size Tensors |
Yes |
Yes |
Yes |
Yes |
Yes |
Yes |
Shape Tensor |
Yes |
|||||
Tensor Reshape |
Yes |
Yes |
Yes |
Yes |
Yes |
Yes |
String Datatype |
Yes |
Yes |
Yes |
|||
HTTP API |
Yes |
Yes |
Yes |
Yes |
Yes |
Yes |
GRPC API |
Yes |
Yes |
Yes |
Yes |
Yes |
Yes |
GRPC Streaming API |
Yes |
Yes |
Yes |
Yes |
Yes |
Yes |
Ensembling |
Yes |
Yes |
Yes |
Yes |
Yes |
Yes |
Shared Memory API |
Yes |
Yes |
Yes |
Yes |
Yes |
Yes |
CUDA Shared Memory API |
Yes |
Yes |
Yes |
Yes |
Yes |
Yes |