NVIDIA TensorRT Inference Server

The NVIDIA TensorRT Inference Server (TRTIS) provides a cloud inferencing solution optimized for NVIDIA GPUs. The server provides an inference service via an HTTP or gRPC endpoint, allowing remote clients to request inferencing for any model being managed by the server.

What’s New In 0.10.0 Beta

• Multiple framework support. The server can manage any number and mix of models (limited by system disk and memory resources). Supports TensorRT, TensorFlow GraphDef, TensorFlow SavedModel and Caffe2 NetDef model formats. Also supports TensorFlow-TensorRT integrated models.
• Custom backend support. TRTIS allows individual models to be implemented with custom backends instead of by a deep-learning framework. With a custom backend a model can implement any logic desired, while still benefiting from the GPU support, concurrent execution, dynamic batching and other features provided by TRTIS.
• TRTIS monitors the model repository for any change and dynamically reloads the model(s) when necessary, without requiring a server restart. Models and model versions can be added and removed, and model configurations can be modified while the server is running.
• Multi-GPU support. TRTIS can distribute inferencing across all system GPUs.
• Concurrent model execution support. Multiple models (or multiple instances of the same model) can run simultaneously on the same GPU.
• Batching support. For models that support batching, the server can accept requests for a batch of inputs and respond with the corresponding batch of outputs. TRTIS also supports dynamic batching where individual inference requests are dynamically combined together to improve inference throughput. Dynamic batching is transparent to the client requesting inference.
• Model repositories may reside on a locally accessible file system (e.g. NFS) or in Google Cloud Storage.
• Readiness and liveness health endpoints suitable for any orchestration or deployment framework, such as Kubernetes.
• Metrics indicating GPU utiliization, server throughput, and server latency.

User Guide

• Quickstart
• Installing the Server
  • Installing Prebuilt Containers
• Running the Server
  • Example Model Repository
  • Running The Inference Server
• Client Libraries and Examples
  • Building the Client Libraries and Examples
  • Image Classification Example Application
  • Performance Example Application
  • Client API
• Model Repository
  • Modifying the Model Repository
  • Model Versions
  • Framework Model Definition
  • Custom Backends
• Model Configuration
  • Generated Model Configuration
  • Version Policy
  • Instance Groups
  • Dynamic Batching
  • Optimization Policy
• Inference Server API
  • Health
  • Status
  • Inference
• Metrics

Developer Guide

• Architecture
  • Concurrent Model Execution
• Contributing
  • Coding Convention
• Building
  • Building the Server
  • Building the Client Libraries and Examples
  • Building the Documentation
• Testing
  • Generate QA Model Repository
  • Build QA Container
  • Run QA Container

API Reference

• Protobuf API
  • HTTP/GRPC API
  • Model Configuration
  • Status
• C++ API
  • Class Hierarchy
  • File Hierarchy
  • Full API
• Python API
  • Client

Indices and tables

• Index