Triton Inference Server Release 22.09
The Triton Inference Server container image, release 22.09, is available on NGC and is open source on GitHub.
Contents of the Triton Inference Server container
The Triton Inference Server Docker image contains the inference server executable and related shared libraries in /opt/tritonserver.
For the list of what the container includes, refer to Deep Learning Frameworks Support Matrix.
Driver Requirements
Release 22.09 is based on CUDA 11.8.0, which requires NVIDIA Driver release 520 or later. However, if you are running on a data center GPU (for example, T4 or any other data center GPU), you can use NVIDIA driver release 450.51 (or later R450), 470.57 (or later R470), 510.47 (or later R510), or 515.65 (or later R515).
The CUDA driver's compatibility package only supports particular drivers. Thus, users should upgrade from all R418, R440, and R460 drivers, which are not forward-compatible with CUDA 11.8. For a complete list of supported drivers, see the CUDA Application Compatibility topic. For more information, see CUDA Compatibility and Upgrades.GPU Requirements
Release 22.09 supports CUDA compute capability 6.0 and later. This corresponds to GPUs in the NVIDIA Pascal, NVIDIA Volta™, NVIDIA Turing™, NVIDIA Ampere architecture, and NVIDIA Hopper™ architecture families. For a list of GPUs to which this compute capability corresponds, see CUDA GPUs. For additional support details, see Deep Learning Frameworks Support Matrix.
Key Features and Enhancements
This Inference Server release includes the following key features and enhancements.
- Added developer tools Github repository that provides a simplified interface for users to interact with the Triton Core shared library. These developer tools are in beta and are subject to change.
- Added CPU metrics reporting in Triton’s Prometheus metrics endpoint.
- Added logging protocol extension for users to change logging configuration dynamically.
- Users can specify the custom plugins to be loaded for TensorRT backend through command line option in addition to LD_PRELOAD.
- Enabled auto-completion for OpenVINO backend.
- Enabled Python backend to log messages through Triton’s logger.
- Added quick search algorithm to Model Analyzer to drastically reduce search time.
- Added GPU metrics gathering to Perf Analyzer, which is also used by Model Analyzer to improve accuracy of those metrics.
- Refer to the 22.09 column of the Frameworks Support Matrix for container image versions on which the 22.09 inference server container is based.
NVIDIA Triton Inference Server Container Versions
The following table shows what versions of Ubuntu, CUDA, Triton Inference Server, and NVIDIA TensorRT™ are supported in each of the NVIDIA containers for Triton Inference Server. For older container versions, refer to the Frameworks Support Matrix.
Known Issues
- In certain rare cases with specific backends, Triton server may crash with segmentation fault when exiting. Preliminary analysis shows that there might be a race condition in clean up of backend/model/instance state objects. Exact root cause is still unknown.
- Triton's TensorRT support depends on the CUDA event synchronization. In some rare cases the events may be triggered earlier than expected, causing Triton to overwrite input tensors while they are still in use and leading to corrupt input data being used for inference. If you encounter accuracy issues with your TensorRT model, you can work-around the issue by enabling the output_copy_stream option in your model's configuration.
- When using a custom operator for the PyTorch backend, the operator may not be loaded due to undefined Python library symbols. This can be work-around by specifying Python library in LD_PRELOAD.
- Auto-complete may cause an increase in server start time. To avoid a start time increase, users can provide the full model configuration and launch the server with --disable-auto-complete-config.
- Auto-complete does not support PyTorch models due to lack of metadata in the model. It can only verify that the number of inputs and the input names matches what is specified in the model configuration. There is no model metadata about the number of outputs and datatypes. Related PyTorch bug:https://github.com/pytorch/pytorch/issues/38273
- Perf Analyzer stability criteria has been changed which may result in reporting instability for scenarios that were previously considered stable. This change has been made to improve the accuracy of Perf Analyzer results. If you observe this message, it can be resolved by increasing the --measurement-interval in the time windows mode or --measurement-request-count in the count windows mode.
- Triton Client PIP wheels for Arm SBSA are not available
from PyPI and pip will install an incorrect Jetson
version of Triton Client library for Arm SBSA.
The correct client wheel file can be pulled directly from the Arm SBSA SDK image and manually installed.
- Traced models in PyTorch seem to create overflows when
int8 tensor values are transformed to int32 on the
GPU.
Refer to https://github.com/pytorch/pytorch/issues/66930 for more information.
- Triton cannot retrieve GPU metrics with MIG-enabled GPU devices (A100 and A30).
- Triton metrics might not work if the host machine is running a separate DCGM agent on bare-metal or in a container.