NVIDIA TensorRT Inference Server

The NVIDIA TensorRT Inference Server provides a cloud inferencing solution optimized for NVIDIA GPUs. The server provides an inference service via an HTTP or gRPC endpoint, allowing remote clients to request inferencing for any model being managed by the server.

What’s New In 1.2.0

  • Ensembling is now available. An ensemble represents a pipeline of one or more models and the connection of input and output tensors between those models. A single inference request to an ensemble will trigger the execution of the entire pipeline.

  • Added Helm chart that deploys a single TensorRT Inference Server into a Kubernetes cluster.

  • The client Makefile now supports building for both Ubuntu 18.04 and Ubuntu 16.04. The Python wheel produced from the build is now compatible with both Python2 and Python3.

  • The perf_client application now has a –percentile flag that can be used to report latencies instead of reporting average latency (which remains the default). For example, using –percentile=99 causes perf_client to report the 99th percentile latency.

  • The perf_client application now has a -z option to use zero-valued input tensors instead of random values.

  • Improved error reporting of incorrect input/output tensor names for TensorRT models.

  • Added –allow-gpu-metrics option to enable/disable reporting of GPU metrics.


  • Multiple framework support. The server can manage any number and mix of models (limited by system disk and memory resources). Supports TensorRT, TensorFlow GraphDef, TensorFlow SavedModel and Caffe2 NetDef model formats. Also supports TensorFlow-TensorRT integrated models. Variable-size input and output tensors are allowed if supported by the framework. See Capabilities for detailed support information for each framework.

  • Concurrent model execution support. Multiple models (or multiple instances of the same model) can run simultaneously on the same GPU.

  • Batching support. For models that support batching, the server can accept requests for a batch of inputs and respond with the corresponding batch of outputs. The inference server also supports multiple scheduling and batching algorithms that combine individual inference requests together to improve inference throughput. These scheduling and batching decisions are transparent to the client requesting inference.

  • Custom backend support. The inference server allows individual models to be implemented with custom backends instead of by a deep-learning framework. With a custom backend a model can implement any logic desired, while still benefiting from the GPU support, concurrent execution, dynamic batching and other features provided by the server.

  • Multi-GPU support. The server can distribute inferencing across all system GPUs.

  • The inference server monitors the model repository for any change and dynamically reloads the model(s) when necessary, without requiring a server restart. Models and model versions can be added and removed, and model configurations can be modified while the server is running.

  • Model repositories may reside on a locally accessible file system (e.g. NFS) or in Google Cloud Storage.

  • Readiness and liveness health endpoints suitable for any orchestration or deployment framework, such as Kubernetes.

  • Metrics indicating GPU utilization, server throughput, and server latency.

Indices and tables