TensorRT Inference Server Release 19.06

The TensorRT Inference Server container image, release 19.06, is available on NGC and is open source on GitHub.

Contents of the TensorRT inference server container

The TensorRT Inference Server Docker image contains the inference server executable and related shared libraries in /opt/tensorrtserver.

The container also includes the following:

Driver Requirements

Release 19.06 is based on NVIDIA CUDA 10.1.168, which requires NVIDIA Driver release 418.xx. However, if you are running on Tesla (Tesla V100, Tesla P4, Tesla P40, or Tesla P100), you may use NVIDIA driver release 384.111+ or 410. The CUDA driver's compatibility package only supports particular drivers. For a complete list of supported drivers, see the CUDA Application Compatibility topic. For more information, see CUDA Compatibility and Upgrades.

GPU Requirements

Release 19.06 supports CUDA compute capability 6.0 and higher. This corresponds to GPUs in the Pascal, Volta, and Turing families. Specifically, for a list of GPUs that this compute capability corresponds to, see CUDA GPUs. For additional support details, see Deep Learning Frameworks Support Matrix.

Key Features and Enhancements

This Inference Server release includes the following key features and enhancements.
  • The inference server container image version 19.06 is based on NVIDIA TensorRT Inference Server 1.3.0, TensorFlow 1.13.1, ONNX Runtime 0.4.0, and Caffe2 0.8.2.
  • Latest version of NVIDIA CUDA 10.1.168 including cuBLAS
  • Latest version of NVIDIA NCCL 2.4.7
  • Added ONNX Runtime as a new backend. The ONNX Runtime backend allows the inference server to directly run ONNX models without requiring conversion to Caffe2 or TensorRT.
  • HTTP health port may be specified independently of inference and status HTTP port with --http-health-port flag.
  • Fixed bug in perf_client that caused high CPU usage by client that could lower the measured inference/sec in some cases.
  • Ubuntu 16.04 with May 2019 updates (see Announcements)


In the next release, we will no longer support Ubuntu 16.04. Release 19.07 will instead support Ubuntu 18.04.

Known Issues

  • Google Cloud Storage (GCS) support is not available in this release. Support for GCS will be re-enabled in the 19.07 release.