Triton Inference Server Release 22.07
The Triton Inference Server container image, release 22.07, is available on NGC and is open source on GitHub.
Contents of the Triton Inference Server container
The Triton Inference Server Docker image contains the inference server executable and related shared libraries in /opt/tritonserver.
For the list of what the container includes, refer to Deep Learning Frameworks Support Matrix.
Driver Requirements
Release 22.07 is based on CUDA 11.7 Update 1 Preview, which requires NVIDIA Driver release 515 or later. However, if you are running on a data center GPU (for example, T4 or any other data center GPU), you can use NVIDIA driver release 450.51 (or later R450), 470.57 (or later R470), or 510.47 (or later R510).
The CUDA driver's compatibility package only supports particular drivers. Thus, users should upgrade from all R418, R440, and R460 drivers, which are not forward-compatible with CUDA 11.7. For a complete list of supported drivers, see the CUDA Application Compatibility topic. For more information, see CUDA Compatibility and Upgrades.GPU Requirements
Release 22.07 supports CUDA compute capability 6.0 and later. This corresponds to GPUs in the NVIDIA Pascal, NVIDIA Volta™, NVIDIA Turing™, and NVIDIA Ampere Architecture GPU families. For a list of GPUs to which this compute capability corresponds, see CUDA GPUs. For additional support details, see Deep Learning Frameworks Support Matrix.
Key Features and Enhancements
This Inference Server release includes the following key features and enhancements.
- Auto-Complete is enabled by default. The --strict-model-config CLI option has been soft deprecated; use the new --disable-auto-complete-config option instead.
- New example backend demonstrating Business Logic Scripting in C++.
- Users can provide values for init_ops variables in TensorFlow TF1.x GraphDef models through JSON file.
- New asyncio compatible API to the Python GRPC/HTTP APIs.
- Added a thread pool to reduce service downtime for concurrently loading models. The thread pool size is configurable with the new --model-load-thread-count tritonserver option. You can find more information here.
- Model Analyzer now doesn't require config.pbtxt file for models that can be auto-completed in Triton.
- Refer to the 22.07 column of the Frameworks Support Matrix for container image versions on which the 22.07 inference server container is based.
NVIDIA Triton Inference Server Container Versions
The following table shows what versions of Ubuntu, CUDA, Triton Inference Server, and NVIDIA TensorRT™ are supported in each of the NVIDIA containers for Triton Inference Server. For older container versions, refer to the Frameworks Support Matrix.
Known Issues
- JetPack release will be published later in the month in order to align with Jetpack SDK public availability.
- Auto-complete could cause an increase in server start time. To avoid a start time increase, users should provide the full model configuration
- When auto-completing some model configs, backends may generate a model config even though there is not enough metadata (for example, Graphdef models for TensorFlow Backend). The user will see the model successfully load but fail to inference. In this case, the user should provide the full model configuration for these models or use the --disable-auto-complete-config CLI option to show which models fail to load.
- Can't do autocomplete for PyTorch models, due to not enough metadata.Can only verify that the number of inputs is correct and the input names match what is specified in the model configuration. There is no info about the number of outputs and datatypes. Related PyTorch bug:https://github.com/pytorch/pytorch/issues/38273.
- Running inference on multiple TensorRT model instances in Triton may fail with signal(6). The issue is expected to be fixed in a future release. Details can be found in https://github.com/triton-inference-server/server/issues/4566.
- Perf Analyzer stability criteria has been changed which may result in reporting instability for scenarios that were previously considered stable. This change has been made to improve the accuracy of Perf Analyzer results. If you observe this message, it can be resolved by increasing the --measurement-interval in the time windows mode or --measurement-request-count in the count windows mode.
- Unlike previously noted, 22.07 is the last release that defaults to TensorFlow version 1. From 22.08 onwards Triton will change the default TensorFlow version to 2.x.
- Triton Client PIP wheels for Arm SBSA are not available
from PyPI and pip will install an incorrect Jetson
version of Triton Client library for Arm SBSA.
The correct client wheel file can be pulled directly from the Arm SBSA SDK image and manually installed.
- Traced models in PyTorch seem to create overflows when
int8 tensor values are transformed to int32 on the
GPU.
Refer to https://github.com/pytorch/pytorch/issues/66930 for more information.
- Triton cannot retrieve GPU metrics with MIG-enabled GPU devices (A100 and A30).
- Triton metrics might not work if the host machine is running a separate DCGM agent on bare-metal or in a container.
- Model Analyzer reported values for GPU utilization and GPU power are known to be inaccurate and generally lower than reality.