Triton Inference Server Release 22.08
The Triton Inference Server container image, release 22.08, is available on NGC and is open source on GitHub.
Contents of the Triton Inference Server container
The Triton Inference Server Docker image contains the inference server executable and related shared libraries in /opt/tritonserver.
For the list of what the container includes, refer to Deep Learning Frameworks Support Matrix.
Driver Requirements
Release 22.08 is based on CUDA 11.7 Update 1, which requires NVIDIA Driver release 515 or later. However, if you are running on a data center GPU (for example, T4 or any other data center GPU), you can use NVIDIA driver release 450.51 (or later R450), 470.57 (or later R470), or 510.47 (or later R510).
The CUDA driver's compatibility package only supports particular drivers. Thus, users should upgrade from all R418, R440, and R460 drivers, which are not forward-compatible with CUDA 11.7. For a complete list of supported drivers, see the CUDA Application Compatibility topic. For more information, see CUDA Compatibility and Upgrades.GPU Requirements
Release 22.08 supports CUDA compute capability 6.0 and later. This corresponds to GPUs in the NVIDIA Pascal, NVIDIA Volta™, NVIDIA Turing™, and NVIDIA Ampere Architecture GPU families. For a list of GPUs to which this compute capability corresponds, see CUDA GPUs. For additional support details, see Deep Learning Frameworks Support Matrix.
Key Features and Enhancements
This Inference Server release includes the following key features and enhancements.
- New support for multiple cloud credentials has been enabled. This feature is in beta and is subject to change.
- Models using custom backends which implement auto-complete configuration, can be loaded without an explicit config.pbtxt file if they are named in the form <model_name>.<backend_name>.
- Users can specify a maximum memory limit when loading models onto the GPU with the new --model-load-gpu-limit tritonserver option and the TRITONSERVER_ServerOptionsSetModelLoadDeviceLimit C API function.
- Added new documentation, Performance Tuning, with a step by step guide to optimize models for production.
- From this release onwards, Triton will default to TensorFlow version 2.X. TensorFlow version 1.X can still be manually specified using backend config.
- PyTorch backend has improved performance by using a separate CUDA Stream for each model instance when the instance kind is GPU.
- Model Analyzer's profile subcommand now analyzes the results after Profile is completed. Usage of the Analyze subcommand is deprecated. Refer to the Model Analyzer documentation for further details.
- Refer to the 22.08 column of the Frameworks Support Matrix for container image versions on which the 22.08 inference server container is based.
NVIDIA Triton Inference Server Container Versions
The following table shows what versions of Ubuntu, CUDA, Triton Inference Server, and NVIDIA TensorRT™ are supported in each of the NVIDIA containers for Triton Inference Server. For older container versions, refer to the Frameworks Support Matrix.
Known Issues
- There is no Jetpack release for 22.08; the latest JetPack release is 22.07.
- Auto-complete may cause an increase in server start time. To avoid a start time increase, users can provide the full model configuration and launch the server with --disable-auto-complete-config.
- When auto-completing some model configs, backends may generate a model config even though there is not enough metadata (for example, Graphdef models for TensorFlow Backend). The user will see the model successfully load but fail to inference. In this case, the user should provide the full model configuration for these models or use the --disable-auto-complete-config CLI option to show which models fail to load.
- Auto-complete does not support PyTorch models due to lack of metadata in the model. It can only verify that the number of inputs and the input names matches what is specified in the model configuration. There is no model metadata about the number of outputs and datatypes. Related PyTorch bug:https://github.com/pytorch/pytorch/issues/38273
- Running inference on multiple TensorRT model instances in Triton may fail with signal(6). The issue is expected to be fixed in a future release. Details can be found in https://github.com/triton-inference-server/server/issues/4566.
- Perf Analyzer stability criteria has been changed which may result in reporting instability for scenarios that were previously considered stable. This change has been made to improve the accuracy of Perf Analyzer results. If you observe this message, it can be resolved by increasing the --measurement-interval in the time windows mode or --measurement-request-count in the count windows mode.
- Unlike previously noted, 22.07 is the last release that defaults to TensorFlow version 1. From 22.08 onwards Triton will change the default TensorFlow version to 2.x.
- Triton Client PIP wheels for Arm SBSA are not available
from PyPI and pip will install an incorrect Jetson
version of Triton Client library for Arm SBSA.
The correct client wheel file can be pulled directly from the Arm SBSA SDK image and manually installed.
- Traced models in PyTorch seem to create overflows when
int8 tensor values are transformed to int32 on the
GPU.
Refer to https://github.com/pytorch/pytorch/issues/66930 for more information.
- Triton cannot retrieve GPU metrics with MIG-enabled GPU devices (A100 and A30).
- Triton metrics might not work if the host machine is running a separate DCGM agent on bare-metal or in a container.
- Model Analyzer reported values for GPU utilization and GPU power are known to be inaccurate and generally lower than reality.