NVIDIA Triton Inference Server

NVIDIA Triton Inference Server provides a cloud inferencing solution optimized for NVIDIA GPUs. The server provides an inference service via an HTTP/REST or GRPC endpoint, allowing remote clients to request inferencing for any model being managed by the server. For edge deployments, Triton Server is also available as a shared library with an API that allows the full functionality of the server to be included directly in an application.

What’s New in 2.2.0

  • TensorFlow 2.x is now supported in addition to TensorFlow 1.x. See the Frameworks Support Matrix for the supported TensorFlow versions. The version of TensorFlow used can be selected when launching Triton with the –backend-config=tensorflow,version=<version> flag. Set <version> to 1 or 2 to select TensorFlow1 or TensorFlow2 respectively. By default TensorFlow 1 is used.

  • Add inference request timeout option to Python and C++ client libraries.

  • GRPC inference protocol updated to fix performance regression.

  • Explicit major/minor versioning added to TRITONSERVER and TRITONBACKED APIs.

  • New CMake option TRITON_CLIENT_SKIP_EXAMPLES to disable building the client examples.


  • Multiple framework support. The server can manage any number and mix of models (limited by system disk and memory resources). Supports TensorRT, TensorFlow GraphDef, TensorFlow SavedModel, ONNX, PyTorch, and Caffe2 NetDef model formats. Both TensorFlow 1.x and TensorFlow 2.x are supported. Also supports TensorFlow-TensorRT and ONNX-TensorRT integrated models. Variable-size input and output tensors are allowed if supported by the framework. See Capabilities for detailed support information for each framework.

  • Concurrent model execution support. Multiple models (or multiple instances of the same model) can run simultaneously on the same GPU.

  • Batching support. For models that support batching, Triton Server can accept requests for a batch of inputs and respond with the corresponding batch of outputs. Triton Server also supports multiple scheduling and batching algorithms that combine individual inference requests together to improve inference throughput. These scheduling and batching decisions are transparent to the client requesting inference.

  • Custom backend support. Triton Server allows individual models to be implemented with custom backends instead of by a deep-learning framework. With a custom backend a model can implement any logic desired, while still benefiting from the GPU support, concurrent execution, dynamic batching and other features provided by the server.

  • Ensemble support. An ensemble represents a pipeline of one or more models and the connection of input and output tensors between those models. A single inference request to an ensemble will trigger the execution of the entire pipeline.

  • Multi-GPU support. Triton Server can distribute inferencing across all system GPUs.

  • Triton Server provides multiple modes for model management. These model management modes allow for both implicit and explicit loading and unloading of models without requiring a server restart.

  • Model repositories may reside on a locally accessible file system (e.g. NFS), in Google Cloud Storage or in Amazon S3.

  • HTTP/REST and GRPC inference protocols based on the community developed KFServing protocol.

  • Readiness and liveness health endpoints suitable for any orchestration or deployment framework, such as Kubernetes.

  • Metrics indicating GPU utilization, server throughput, and server latency.

  • C library inferface allows the full functionality of Triton Server to be included directly in an application.

Indices and tables