Triton Inference Server

riton Inference Server provides a cloud inferencing solution optimized for both CPUs and GPUs. Triton provides an inference service via an HTTP/REST or GRPC endpoint, allowing remote clients to request inferencing for any model being managed by the server. For edge deployments, Triton is also available as a shared library with a C API that allows the full functionality of Triton to be included directly in an application.

What’s New In 2.3.0

• Python Client library is now a pip package available from the NVIDIA pypi index. See https://github.com/triton-inference-server/server/blob/master/docs/client_libraries.md#download-using-python-package-installer-pip for more information.

• Fix a performance issue with the HTTP/REST protocol and the Python client library that caused reduced performance when outputs were not requested explicitly in an inference request.

• Fix some bugs in reporting of statistics for ensemble models.

• GRPC updated to version 1.25.0.

Features

• Multiple framework support. Triton can manage any number and mix of models (limited by system disk and memory resources). Supports TensorRT, TensorFlow GraphDef, TensorFlow SavedModel, ONNX, PyTorch, and Caffe2 NetDef model formats. Both TensorFlow 1.x and TensorFlow 2.x are supported. Also supports TensorFlow-TensorRT and ONNX-TensorRT integrated models. Variable-size input and output tensors are allowed if supported by the framework. See Capabilities for information for each framework.

• Concurrent model execution support. Multiple models (or multiple instances of the same model) can run simultaneously on the same GPU.

• Batching support. For models that support batching, Triton can accept requests for a batch of inputs and respond with the corresponding batch of outputs. Triton also supports multiple scheduling and batching algorithms that combine individual inference requests together to improve inference throughput. These scheduling and batching decisions are transparent to the client requesting inference.

• Custom backend support. Triton allows individual models to be implemented with custom backends instead of by a deep-learning framework. With a custom backend a model can implement any logic desired, while still benefiting from the CPU and GPU support, concurrent execution, dynamic batching and other features provided by Triton.

• Ensemble support. An ensemble represents a pipeline of one or more models and the connection of input and output tensors between those models. A single inference request to an ensemble will trigger the execution of the entire pipeline.

• Multi-GPU support. Triton can distribute inferencing across all system GPUs.

• Triton provides multiple modes for model management. These model management modes allow for both implicit and explicit loading and unloading of models without requiring a server restart.

• Model repositories may reside on a locally accessible file system (e.g. NFS), in Google Cloud Storage or in Amazon S3.

• HTTP/REST and GRPC inference protocols based on the community developed KFServing protocol.

• Readiness and liveness health endpoints suitable for any orchestration or deployment framework, such as Kubernetes.

• Metrics indicating GPU utilization, server throughput, and server latency. The metrics are provided in Prometheus data format.

• C library inferface allows the full functionality of Triton to be included directly in an application.

User Guide

• Quickstart
  • Prerequisites
  • Using A Prebuilt Docker Container
  • Building With Docker
  • Building With CMake
  • Run Triton Inference Server
  • Verify Triton Is Running Correctly
  • Getting The Client Examples
  • Running The Image Classification Example
• Installing Triton
• Running Triton
  • Example Model Repository
  • Running Triton On A System With A GPU
  • Running Triton On A System Without A GPU
  • Running Triton Without Docker
  • Checking Triton Status
• Client Examples
  • Getting the Client Examples
  • Simple Example Applications
  • Image Classification Example Application
  • Ensemble Image Classification Example Application
• Client Libraries
  • Getting the Client Libraries
  • Building Your Own Client
  • Client Library API
• Model Repository
  • Model Repository Locations
  • Repository Layout
  • Modifying the Model Repository
  • Model Versions
  • Framework Model Definition
• Model Configuration
  • Minimal Model Configuration
  • Generated Model Configuration
  • Datatypes
  • Reshape
  • Shape Tensors
  • Version Policy
  • Instance Groups
  • Scheduling And Batching
  • Optimization Policy
  • Model Warmup
• Models And Schedulers
  • Stateless Models
  • Stateful Models
  • Ensemble Models
• Model Management
  • Model Control Mode NONE
  • Model Control Mode EXPLICIT
  • Model Control Mode POLL
• Optimization
  • Optimization Settings
  • Model Analyzer
  • perf_client
  • Server Trace
• Metrics

Developer Guide

• Architecture
  • Concurrent Model Execution
• Backends
  • Backend Shared Library
  • Triton Backend API
  • Example Backends
  • Legacy Custom Backends
• Custom Operations
  • TensorRT
  • TensorFlow
  • PyTorch
  • ONNX
• HTTP/REST and GRPC API
• Library API
• Building
  • Building Triton
  • Building A Custom Backend
  • Building the Client Libraries and Examples
  • Building the Documentation
• Testing
  • Generate QA Model Repositories
  • Build QA Container
  • Run QA Container
• Contributing
  • Coding Convention

Reference

• FAQ
  • What are the advantages of running a model with Triton Inference Server compared to running directly using the model’s framework API?
  • Can Triton Inference Server run on systems that don’t have GPUs?
  • Can Triton Inference Server be used in non-Docker environments?
  • Do you provide client libraries for languages other than C++ and Python?
  • How would you use Triton Inference Server within the AWS environment?
  • How do I measure the performance of my model running in the Triton Inference Server?
  • How can I fully utilize the GPU with Triton Inference Server?
  • If I have a server with multiple GPUs should I use one Triton Inference Server to manage all GPUs or should I use multiple inference servers, one for each GPU?
• Capabilities
• Protobuf API
  • GRPC API
  • Model Configuration
• C++ API
  • Class Hierarchy
  • File Hierarchy
  • Full API
• Python API
  • GRPC Client
  • HTTP/REST Client
  • Client Utilities
  • Shared Memory Utilities

Indices and tables

• Index