NVIDIA TensorRT Inference Server

The NVIDIA TensorRT Inference Server provides a cloud inferencing solution optimized for NVIDIA GPUs. The server provides an inference service via an HTTP or GRPC endpoint, allowing remote clients to request inferencing for any model being managed by the server.

What’s New in 1.9.0

  • The model configuration now includes a model warmup option. This option provides the ability to tune and optimize the model before inference requests are received, avoiding initial inference delays. This option is especially useful for frameworks like TensorFlow that perform network optimization in response to the initial inference requests. Models can be warmed-up with one or more synthetic or realistic workloads before they become ready in the server.

  • An enhanced sequence batcher now has multiple scheduling strategies. A new Oldest strategy integrates with the dynamic batcher to enable improved inference performance for models that don’t require all inference requests in a sequence to be routed to the same batch slot.

  • The perf_client now has an option to generate requests using a realistic poisson distribution or a user provided distribution.

  • A new repository API (available in the shared library API, HTTP, and GRPC) returns an index of all models available in the model repositories) visible to the server. This index can be used to see what models are available for loading onto the server.

  • The server status returned by the server status API now includes the timestamp of the last inference request received for each model.

  • Inference server tracing capabilities are now documented in the Optimization section of the User Guide. Tracing support is enhanced to provide trace for ensembles and the contained models.

  • A community contributed Dockerfile is now available to build the TensorRT Inference Server clients on CentOS.


  • Multiple framework support. The server can manage any number and mix of models (limited by system disk and memory resources). Supports TensorRT, TensorFlow GraphDef, TensorFlow SavedModel, ONNX, PyTorch, and Caffe2 NetDef model formats. Also supports TensorFlow-TensorRT integrated models. Variable-size input and output tensors are allowed if supported by the framework. See Capabilities for detailed support information for each framework.

  • Concurrent model execution support. Multiple models (or multiple instances of the same model) can run simultaneously on the same GPU.

  • Batching support. For models that support batching, the server can accept requests for a batch of inputs and respond with the corresponding batch of outputs. The inference server also supports multiple scheduling and batching algorithms that combine individual inference requests together to improve inference throughput. These scheduling and batching decisions are transparent to the client requesting inference.

  • Custom backend support. The inference server allows individual models to be implemented with custom backends instead of by a deep-learning framework. With a custom backend a model can implement any logic desired, while still benefiting from the GPU support, concurrent execution, dynamic batching and other features provided by the server.

  • Ensemble support. An ensemble represents a pipeline of one or more models and the connection of input and output tensors between those models. A single inference request to an ensemble will trigger the execution of the entire pipeline.

  • Multi-GPU support. The server can distribute inferencing across all system GPUs.

  • The inference server provides multiple modes for model management. These model management modes allow for both implicit and explicit loading and unloading of models without requiring a server restart.

  • Model repositories may reside on a locally accessible file system (e.g. NFS), in Google Cloud Storage or in Amazon S3.

  • Readiness and liveness health endpoints suitable for any orchestration or deployment framework, such as Kubernetes.

  • Metrics indicating GPU utilization, server throughput, and server latency.

  • C library inferface allows the full functionality of the inference server to be included directly in an application.

Indices and tables