Triton Inference Server Release 19.08

The TensorRT Inference Server container image, release 19.08, is available on NGC and is open source on GitHub.

Contents of the Triton inference server container

The TensorRT Inference Server Docker image contains the inference server executable and related shared libraries in /opt/tensorrtserver.

The container also includes the following:

Driver Requirements

Release 19.08 is based on NVIDIA CUDA 10.1.243, which requires NVIDIA Driver release 418.87. However, if you are running on Tesla (Tesla V100, Tesla P4, Tesla P40, or Tesla P100), you may use NVIDIA driver release 384.111+ or 410. The CUDA driver's compatibility package only supports particular drivers. For a complete list of supported drivers, see the CUDA Application Compatibility topic. For more information, see CUDA Compatibility and Upgrades.

GPU Requirements

Release 19.08 supports CUDA compute capability 6.0 and higher. This corresponds to GPUs in the Pascal, Volta, and Turing families. Specifically, for a list of GPUs that this compute capability corresponds to, see CUDA GPUs. For additional support details, see Deep Learning Frameworks Support Matrix.

Key Features and Enhancements

This Inference Server release includes the following key features and enhancements.
  • The inference server container image version 19.08 is based on NVIDIA TensorRT Inference Server 1.5.0, TensorFlow 1.14.0, ONNX Runtime 0.4.0, and PyTorch 1.2.0a0.
  • Added a new execution mode allows the inference server to start without loading any models from the model repository. Model loading and unloading is then controlled by a new GRPC/HTTP model control API.
  • Added a new instance-group mode allows TensorFlow models that explicitly distribute inferencing across multiple GPUs to run in that manner in the inference server.
  • Improved input/output tensor reshape to allow variable-sized dimensions in tensors being reshaped.
  • Added a C++ wrapper around the custom backend C API to simplify the creation of custom backends. This wrapper is included in the custom backend SDK.
  • Improved the accuracy of the compute statistic reported for inference requests. Previously the compute statistic included some additional time beyond the actual compute time.
  • The performance client, perf_client, now reports more information for ensemble models, including statistics for all contained models and the entire ensemble.
  • Latest version of NVIDIA CUDA 10.1.243 including cuBLAS 10.2.1.243
  • Latest version of NVIDIA cuDNN 7.6.2
  • Latest version of NVIDIA NCCL 2.4.8
  • Latest version of MLNX_OFED +4.0
  • Latest version of OpenMPI 3.1.4
  • Ubuntu 18.04 with July 2019 updates

Known Issues

There are no known issues in this release.