TensorRT Container Release Notes
The TensorRT container is an easy to use container for TensorRT development. The container allows you to build, modify, and execute TensorRT samples. These release notes provide a list of key features, packaged software in the container, software enhancements and improvements, and known issues. The TensorRT container is released monthly to provide you with the latest NVIDIA deep learning software libraries and GitHub code contributions that have been sent upstream. The libraries and contributions have all been tested, tuned, and optimized.
For a complete view of the supported software and specific versions that are packaged with the frameworks based on the container image, see the Frameworks Support Matrix.
1. TensorRT Overview
The core of NVIDIA® TensorRT™ is a C++ library that facilitates high-performance inference on NVIDIA graphics processing units (GPUs). TensorRT takes a trained network, which consists of a network definition and a set of trained parameters, and produces a highly optimized runtime engine that performs inference for that network.
You can describe a TensorRT network by using a C++ or Python API, or you can import an existing Caffe, ONNX, or TensorFlow model by using one of the provided parsers.
TensorRT provides APIs through C++ and Python that help express deep learning models by using the Network Definition API or load a predefined model by using parsers that allows TensorRT to optimize and run them on an NVIDIA GPU. TensorRT applies graph optimizations, layer fusion, and other optimizations, while also finding the fastest implementation of that model by leveraging a diverse collection of highly optimized kernels. TensorRT also supplies a runtime that you can use to execute this network on all NVIDIA’s GPUs from the NVIDIA Pascal™ generation onwards.
TensorRT also includes optional high-speed, mixed precision capabilities that were introduced in Tegra X1 and were extended with the NVIDIA Pascal, NVIDIA Volta™, and NVIDIA Turing™ architectures.
The TensorRT container allows TensorRT samples to be built, modified, and executed. For more information about the TensorRT samples, see the TensorRT Sample Support Guide.
For a complete list of installation options and instructions, refer to Installing TensorRT.
2. Pulling A Container
- Install Docker.
- For NVIDIA DGX™ users, see Preparing to use NVIDIA Containers Getting Started Guide.
- For non-DGX users, see NVIDIA® GPU Cloud™ (NGC) container registry installation documentation based on your platform.
- Ensure that you have access and can log in to the NGC container registry.
Refer to NGC Getting Started Guide for more information.
The deep learning frameworks, the NGC Docker containers, and the deep learning framework containers are stored in the nvcr.io/nvidia repository.
3. Running TensorRT
Before you can run an NGC deep learning framework container, your Docker environment must support NVIDIA GPUs. To run a container, issue the appropriate command as explained in Running A Container and specify the registry, repository, and tags.
- The Docker engine loads the image into a container which runs the software.
- You define the runtime resources of the container by including the
additional flags and settings that are used with the command.
These flags and settings are described in Running A Container.
- The GPUs are explicitly defined for the Docker container, which defaults to
all GPUs, but can be specified by using the
NVIDIA_VISIBLE_DEVICES environment variable.
For more information, refer to the nvidia-docker documentation.
Note: Starting in Docker 19.03, complete the steps below.
The method implemented in your system depends on the DGX OS version that you installed (for DGX systems), the NGC Cloud Image that was provided by a Cloud Service Provider, or the software that you installed to prepare to run NGC containers on TITAN PCs, Quadro PCs, or NVIDIA Virtual GPUs (vGPUs).
- Issue the command for the applicable release of the container that you want.
The following command assumes that you want to pull the latest container.
docker pull nvcr.io/nvidia/tensorrt:24.02-py3
- Open a command prompt and paste the pull command.
Ensure that the pull process successfully completes before you proceed to step 3.
- Run the container image.
- If you have Docker 19.03 or later, a typical command to
launch the container is:
docker run --gpus all -it --rm -v local_dir:container_dir nvcr.io/nvidia/tensorrt:<xx.xx>-py<x>
- If you have Docker 19.02 or earlier, a typical command to
launch the container is:
nvidia-docker run -it --rm -v local_dir:container_dir nvcr.io/nvidia/tensorrt:<xx.xx>-py<x>
- If you have Docker 19.03 or later, a typical command to
launch the container is:
- To extend the TensorRT container, select one of the following options:
- Add to or modify the source code in this container and run your customized version.
- To add additional packages, use docker build to add
your customizations on top of this container.
Note: NVIDIA recommends using the docker build option for ease of migration to later versions of the TensorRT container.
4. TensorRT Release 24.07
The NVIDIA container image for TensorRT, release 24.07, is available on NGC.
Contents of the TensorRT container
- The TensorRT C++ samples and C++ API documentation.
- Build the samples can be by running make in the /workspace/tensorrt/samples directory.
- The resulting executables are in the /workspace/tensorrt/bin directory.
- The C++ API documentation is in the /workspace/tensorrt/doc/cpp directory.
- The TensorRT Python samples and Python API documentation.
- The Python samples are in the
/workspace/tensorrt/samples/python directory.
Refer to the respective README documents for more samples.
- Many Python samples can be run by using python
<script.py> -d /workspace/tensorrt/data.
For example:
python onnx_resnet50.py -d /workspace/tensorrt/data
- The Python API documentation is in the /workspace/tensorrt/doc/python directory.
- The Python samples are in the
/workspace/tensorrt/samples/python directory.
- TensorRT 8.6.3.
The ONNX parser and plug-in libraries that are bundled with this container are built from TensorRT Open Source Software.
- Ubuntu 22.04Note: Container image 24.07-py3 contains Python 3.10.
- NVIDIA CUDA 12.5.0.23
- NVIDIA cuBLAS 12.5.2.13
- NVIDIA cuDNN 9.1.0.70
- NVIDIA NCCL 2.21.5Note: Although NCCL is packaged in the container, it does not affect TensorRT nor inferencing in any way.
- rdma-core 39.0
- OpenMPI 4.1.4+
- OpenUCX 1.15.0
- GDRCopy 2.3
- NVIDIA HPC-X 2.19
- Nsight Compute 2024.2.0.16
- Nsight Systems 2024.2.3.38
Driver Requirements
Release 24.07 is based on NVIDIA CUDA 12.5.0.23, which requires NVIDIA Driver release 545 or later. However, if you are running on a data center GPU (for example, T4 or any other data center GPU), you can use NVIDIA driver release 470.57 (or later R470), 525.85 (or later R525), 535.86 (or later R535), or 545.23 (or later R545).
The CUDA driver's compatibility package only supports particular drivers. Thus, users should upgrade from all R418, R440, R450, R460, R510, and R520 drivers, which are not forward-compatible with CUDA 12.3. For a complete list of supported drivers, see the CUDA Application Compatibility topic. For more information, see CUDA Compatibility and Upgrades.
GPU Requirements
Release 24.07 supports CUDA compute capability 6.0 and later. This corresponds to GPUs in the NVIDIA Pascal, NVIDIA Volta™, NVIDIA Turing™, NVIDIA Ampere architecture, NVIDIA Hopper™, and NVIDIA Ada Lovelace architecture families. For a list of GPUs to which this compute capability corresponds, see CUDA GPUs. For additional support details, see the Deep Learning Frameworks Support Matrix.
Key Features and Enhancements
- TensorRT container image version 24.07 is based on TensorRT 10.1.
For a list of the features and enhancements that were introduced in this version of TensorRT, refer to the TensorRT release notes.
- All dependencies on cuDNN have been removed from the TensorRT starting with the 8.6.3 release to reduce the overall container size. Any TensorRT features which depend on cuDNN, which are primarily some plugins and samples, will not work with this release.
- Latest version of Ubuntu 22.04 with October 2023 updates.
Announcements
- Starting with the 23.11 release, TensorRT containers supporting iGPU architectures are published, and run on Jetson devices. Please refer to the Frameworks Support Matrix for information regarding which iGPU hardware/software is supported by which container.
- NVIDIA Deep Learning Profiler (DLProf) v1.8, which was included in the 21.12
container, was the last release of DLProf.
Starting with the 22.01 container, DLProf is no longer included. It can still be manually installed by using a pip wheel on the nvidia-pyindex.
Obtaining Missing Data Files
Some samples require data files that are not included in the TensorRT container because of licensing restrictions, or because they are too large. Samples that do not include the required data files include a README.md file in the corresponding source directory that provides information about how to obtain the necessary data files.
Installing Required Python Modules
- To complete some of the samples, you might want to first run the Python setup script.
- If you need to install the missing Python modules and their dependencies, run the /opt/tensorrt/python/python_setup.sh script.
Installing Open Source Components
A script has been added to clone, build, and replace the provided plug-in, the Caffe parser, and the ONNX parser libraries with the open source ones that are based on the 24.07 tag on the official TensorRT open source repository.
To install the open source components in the container, run the following script:
/opt/tensorrt/install_opensource.sh -b main
For more information, see GitHub: TensorRT.
Limitations
- Accelerating Inference In TensorFlow with
TensorRT (TF-TRT) is not supported in the TensorRT containers.
Use the TensorFlow Container to accelerate through TF-TRT instead.
- Torch-TensorRT is not supported in the TensorRT containers.
Use the PyTorch Container to accelerate through Torch-TRT instead.
NVIDIA TensorRT Container Versions
The following table shows what versions of Ubuntu, CUDA, and TensorRT are supported in each of the NVIDIA containers for TensorRT. For older container versions, refer to the Frameworks Support Matrix.
Known Issues
- The onnx_graphsurgeon Python module on ARM Server systems is not compatible with ONNX version 1.11.0, which is normally recommended for the included TensorRT release. You will instead need to use ONNX version 1.15.0 to resolve a possible segmentation fault.
- With r545 or r550 drivers, some models may run into "Unspecified Launch Failure" during engine building. This can be worked around by downgrading the driver version to r535.
- TensorRT’s version compatibility feature has not been extensively tested and is therefore not supported with TensorRT 8.6.3. This TensorRT release is a special release that removes cuDNN as a dependency. Version compatibility between TensorRT 8.6.1 and future versions as documented will still be supported.
- Due to removing TensorRT’s dependency on cuDNN the following networks may
show performance regressions:
- BasicUnet
- DynUnet
- HighResNet
- StableDiffusion VAE-encoder
- StableDiffusion VAE-decoder
5. TensorRT Release 24.06
The NVIDIA container image for TensorRT, release 24.06, is available on NGC.
Contents of the TensorRT container
- The TensorRT C++ samples and C++ API documentation.
- Build the samples can be by running make in the /workspace/tensorrt/samples directory.
- The resulting executables are in the /workspace/tensorrt/bin directory.
- The C++ API documentation is in the /workspace/tensorrt/doc/cpp directory.
- The TensorRT Python samples and Python API documentation.
- The Python samples are in the
/workspace/tensorrt/samples/python directory.
Refer to the respective README documents for more samples.
- Many Python samples can be run by using python
<script.py> -d /workspace/tensorrt/data.
For example:
python onnx_resnet50.py -d /workspace/tensorrt/data
- The Python API documentation is in the /workspace/tensorrt/doc/python directory.
- The Python samples are in the
/workspace/tensorrt/samples/python directory.
- TensorRT 8.6.3.
The ONNX parser and plug-in libraries that are bundled with this container are built from TensorRT Open Source Software.
- Ubuntu 22.04Note: Container image 24.06-py3 contains Python 3.10.
- NVIDIA CUDA 12.5.0.23
- NVIDIA cuBLAS 12.5.2.13
- NVIDIA cuDNN 9.1.0.70
- NVIDIA NCCL 2.21.5Note: Although NCCL is packaged in the container, it does not affect TensorRT nor inferencing in any way.
- rdma-core 39.0
- OpenMPI 4.1.4+
- OpenUCX 1.15.0
- GDRCopy 2.3
- NVIDIA HPC-X 2.19
- Nsight Compute 2024.2.0.16
- Nsight Systems 2024.2.3.38
Driver Requirements
Release 24.06 is based on NVIDIA CUDA 12.5.0.23, which requires NVIDIA Driver release 545 or later. However, if you are running on a data center GPU (for example, T4 or any other data center GPU), you can use NVIDIA driver release 470.57 (or later R470), 525.85 (or later R525), 535.86 (or later R535), or 545.23 (or later R545).
The CUDA driver's compatibility package only supports particular drivers. Thus, users should upgrade from all R418, R440, R450, R460, R510, and R520 drivers, which are not forward-compatible with CUDA 12.3. For a complete list of supported drivers, see the CUDA Application Compatibility topic. For more information, see CUDA Compatibility and Upgrades.
GPU Requirements
Release 24.06 supports CUDA compute capability 6.0 and later. This corresponds to GPUs in the NVIDIA Pascal, NVIDIA Volta™, NVIDIA Turing™, NVIDIA Ampere architecture, NVIDIA Hopper™, and NVIDIA Ada Lovelace architecture families. For a list of GPUs to which this compute capability corresponds, see CUDA GPUs. For additional support details, see the Deep Learning Frameworks Support Matrix.
Key Features and Enhancements
- TensorRT container image version 24.06 is based on TensorRT 10.1.
For a list of the features and enhancements that were introduced in this version of TensorRT, refer to the TensorRT release notes.
- All dependencies on cuDNN have been removed from the TensorRT starting with the 8.6.3 release to reduce the overall container size. Any TensorRT features which depend on cuDNN, which are primarily some plugins and samples, will not work with this release.
- Latest version of Ubuntu 22.04 with October 2023 updates.
Announcements
- Starting with the 23.11 release, TensorRT containers supporting iGPU architectures are published, and run on Jetson devices. Please refer to the Frameworks Support Matrix for information regarding which iGPU hardware/software is supported by which container.
- NVIDIA Deep Learning Profiler (DLProf) v1.8, which was included in the 21.12
container, was the last release of DLProf.
Starting with the 22.01 container, DLProf is no longer included. It can still be manually installed by using a pip wheel on the nvidia-pyindex.
Obtaining Missing Data Files
Some samples require data files that are not included in the TensorRT container because of licensing restrictions, or because they are too large. Samples that do not include the required data files include a README.md file in the corresponding source directory that provides information about how to obtain the necessary data files.
Installing Required Python Modules
- To complete some of the samples, you might want to first run the Python setup script.
- If you need to install the missing Python modules and their dependencies, run the /opt/tensorrt/python/python_setup.sh script.
Installing Open Source Components
A script has been added to clone, build, and replace the provided plug-in, the Caffe parser, and the ONNX parser libraries with the open source ones that are based on the 24.06 tag on the official TensorRT open source repository.
To install the open source components in the container, run the following script:
/opt/tensorrt/install_opensource.sh -b main
For more information, see GitHub: TensorRT.
Limitations
- Accelerating Inference In TensorFlow with
TensorRT (TF-TRT) is not supported in the TensorRT containers.
Use the TensorFlow Container to accelerate through TF-TRT instead.
- Torch-TensorRT is not supported in the TensorRT containers.
Use the PyTorch Container to accelerate through Torch-TRT instead.
NVIDIA TensorRT Container Versions
The following table shows what versions of Ubuntu, CUDA, and TensorRT are supported in each of the NVIDIA containers for TensorRT. For older container versions, refer to the Frameworks Support Matrix.
Known Issues
- The onnx_graphsurgeon Python module on ARM Server systems is not compatible with ONNX version 1.11.0, which is normally recommended for the included TensorRT release. You will instead need to use ONNX version 1.15.0 to resolve a possible segmentation fault.
- With r545 or r550 drivers, some models may run into "Unspecified Launch Failure" during engine building. This can be worked around by downgrading the driver version to r535.
- TensorRT’s version compatibility feature has not been extensively tested and is therefore not supported with TensorRT 8.6.3. This TensorRT release is a special release that removes cuDNN as a dependency. Version compatibility between TensorRT 8.6.1 and future versions as documented will still be supported.
- Due to removing TensorRT’s dependency on cuDNN the following networks may
show performance regressions:
- BasicUnet
- DynUnet
- HighResNet
- StableDiffusion VAE-encoder
- StableDiffusion VAE-decoder
6. TensorRT Release 24.05
The NVIDIA container image for TensorRT, release 24.05, is available on NGC.
Contents of the TensorRT container
- The TensorRT C++ samples and C++ API documentation.
- Build the samples can be by running make in the /workspace/tensorrt/samples directory.
- The resulting executables are in the /workspace/tensorrt/bin directory.
- The C++ API documentation is in the /workspace/tensorrt/doc/cpp directory.
- The TensorRT Python samples and Python API documentation.
- The Python samples are in the
/workspace/tensorrt/samples/python directory.
Refer to the respective README documents for more samples.
- Many Python samples can be run by using python
<script.py> -d /workspace/tensorrt/data.
For example:
python onnx_resnet50.py -d /workspace/tensorrt/data
- The Python API documentation is in the /workspace/tensorrt/doc/python directory.
- The Python samples are in the
/workspace/tensorrt/samples/python directory.
- TensorRT 8.6.3.
The ONNX parser and plug-in libraries that are bundled with this container are built from TensorRT Open Source Software.
- Ubuntu 22.04Note: Container image 24.05-py3 contains Python 3.10.
- NVIDIA CUDA 12.4.1
- NVIDIA cuBLAS 12.4.5.8
- NVIDIA cuDNN 9.1.0.70
- NVIDIA NCCL 2.21.5Note: Although NCCL is packaged in the container, it does not affect TensorRT nor inferencing in any way.
- rdma-core 39.0
- OpenMPI 4.1.4+
- OpenUCX 1.15.0
- GDRCopy 2.3
- NVIDIA HPC-X 2.19
- Nsight Compute 2024.1.1.4
- Nsight Systems 2024.2.1.106
Driver Requirements
Release 24.05 is based on NVIDIA CUDA 12.4.0.41, which requires NVIDIA Driver release 545 or later. However, if you are running on a data center GPU (for example, T4 or any other data center GPU), you can use NVIDIA driver release 470.57 (or later R470), 525.85 (or later R525), 535.86 (or later R535), or 545.23 (or later R545).
The CUDA driver's compatibility package only supports particular drivers. Thus, users should upgrade from all R418, R440, R450, R460, R510, and R520 drivers, which are not forward-compatible with CUDA 12.3. For a complete list of supported drivers, see the CUDA Application Compatibility topic. For more information, see CUDA Compatibility and Upgrades.
GPU Requirements
Release 24.05 supports CUDA compute capability 6.0 and later. This corresponds to GPUs in the NVIDIA Pascal, NVIDIA Volta™, NVIDIA Turing™, NVIDIA Ampere architecture, NVIDIA Hopper™, and NVIDIA Ada Lovelace architecture families. For a list of GPUs to which this compute capability corresponds, see CUDA GPUs. For additional support details, see the Deep Learning Frameworks Support Matrix.
Key Features and Enhancements
- TensorRT container image version 24.05 is based on TensorRT 8.6.3.1.
For a list of the features and enhancements that were introduced in TensorRT 8.6, refer to the TensorRT release notes.
- All dependencies on cuDNN have been removed from the TensorRT 8.6.3 release to reduce the overall container size. Any TensorRT features which depend on cuDNN, which are primarily some plugins and samples, will not work with this release.
- Latest version of Ubuntu 22.04 with October 2023 updates.
Announcements
- Starting with the 23.11 release, TensorRT containers supporting iGPU architectures are published, and run on Jetson devices. Please refer to the Frameworks Support Matrix for information regarding which iGPU hardware/software is supported by which container.
- NVIDIA Deep Learning Profiler (DLProf) v1.8, which was included in the 21.12
container, was the last release of DLProf.
Starting with the 22.01 container, DLProf is longer included. It can still be manually installed by using a pip wheel on the nvidia-pyindex.
Obtaining Missing Data Files
Some samples require data files that are not included in the TensorRT container because of licensing restrictions, or because they are too large. Samples that do not include the required data files include a README.md file in the corresponding source directory that provides information about how to obtain the necessary data files.
Installing Required Python Modules
- To complete some of the samples, you might want to first run the Python setup script.
- If you need to install the missing Python modules and their dependencies, run the /opt/tensorrt/python/python_setup.sh script.
Installing Open Source Components
A script has been added to clone, build, and replace the provided plug-in, the Caffe parser, and the ONNX parser libraries with the open source ones that are based on the 24.05 tag on the official TensorRT open source repository.
To install the open source components in the container, run the following script:
/opt/tensorrt/install_opensource.sh -b main
For more information, see GitHub: TensorRT.
Limitations
- Accelerating Inference In TensorFlow with
TensorRT (TF-TRT) is not supported in the TensorRT containers.
Use the TensorFlow Container to accelerate through TF-TRT instead.
- Torch-TensorRT is not supported in the TensorRT containers.
Use the PyTorch Container to accelerate through Torch-TRT instead.
NVIDIA TensorRT Container Versions
The following table shows what versions of Ubuntu, CUDA, and TensorRT are supported in each of the NVIDIA containers for TensorRT. For older container versions, refer to the Frameworks Support Matrix.
Known Issues
- The onnx_graphsurgeon Python module on ARM Server systems is not compatible with ONNX version 1.11.0, which is normally recommended for the included TensorRT release. You will instead need to use ONNX version 1.15.0 to resolve a possible segmentation fault.
- With r545 or r550 drivers, some models may run into "Unspecified Launch Failure" during engine building. This can be worked around by downgrading the driver version to r535.
- TensorRT’s version compatibility feature has not been extensively tested and is therefore not supported with TensorRT 8.6.3. This TensorRT release is a special release that removes cuDNN as a dependency. Version compatibility between TensorRT 8.6.1 and future versions as documented will still be supported.
- Due to removing TensorRT’s dependency on cuDNN the following networks may
show performance regressions:
- BasicUnet
- DynUnet
- HighResNet
- StableDiffusion VAE-encoder
- StableDiffusion VAE-decoder
7. TensorRT Release 24.04
The NVIDIA container image for TensorRT, release 24.04, is available on NGC.
Contents of the TensorRT container
- The TensorRT C++ samples and C++ API documentation.
- Build the samples can be by running make in the /workspace/tensorrt/samples directory.
- The resulting executables are in the /workspace/tensorrt/bin directory.
- The C++ API documentation is in the /workspace/tensorrt/doc/cpp directory.
- The TensorRT Python samples and Python API documentation.
- The Python samples are in the
/workspace/tensorrt/samples/python directory.
Refer to the respective README documents for more samples.
- Many Python samples can be run by using python
<script.py> -d /workspace/tensorrt/data.
For example:
python onnx_resnet50.py -d /workspace/tensorrt/data
- The Python API documentation is in the /workspace/tensorrt/doc/python directory.
- The Python samples are in the
/workspace/tensorrt/samples/python directory.
- TensorRT 8.6.3.
The ONNX parser and plug-in libraries that are bundled with this container are built from TensorRT Open Source Software.
- Ubuntu 22.04Note: Container image 24.04-py3 contains Python 3.10.
- NVIDIA CUDA 12.4
- NVIDIA cuBLAS 12.4.5.8
- NVIDIA cuDNN 9.1.0.70
- NVIDIA NCCL 2.21.5Note: Although NCCL is packaged in the container, it does not affect TensorRT nor inferencing in any way.
- rdma-core 39.0
- OpenMPI 4.1.4+
- OpenUCX 1.15.0
- GDRCopy 2.3
- NVIDIA HPC-X 2.18
- Nsight Compute 2024.1.1.4
- Nsight Systems 2024.2.1.106
Driver Requirements
Release 24.04 is based on NVIDIA CUDA 12.4.0.41, which requires NVIDIA Driver release 545 or later. However, if you are running on a data center GPU (for example, T4 or any other data center GPU), you can use NVIDIA driver release 470.57 (or later R470), 525.85 (or later R525), 535.86 (or later R535), or 545.23 (or later R545).
The CUDA driver's compatibility package only supports particular drivers. Thus, users should upgrade from all R418, R440, R450, R460, R510, and R520 drivers, which are not forward-compatible with CUDA 12.3. For a complete list of supported drivers, see the CUDA Application Compatibility topic. For more information, see CUDA Compatibility and Upgrades.
GPU Requirements
Release 24.04 supports CUDA compute capability 6.0 and later. This corresponds to GPUs in the NVIDIA Pascal, NVIDIA Volta™, NVIDIA Turing™, NVIDIA Ampere architecture, NVIDIA Hopper™, and NVIDIA Ada Lovelace architecture families. For a list of GPUs to which this compute capability corresponds, see CUDA GPUs. For additional support details, see the Deep Learning Frameworks Support Matrix.
Key Features and Enhancements
- TensorRT container image version 24.04 is based on TensorRT 8.6.3.1.
For a list of the features and enhancements that were introduced in TensorRT 8.6, refer to the TensorRT release notes.
- All dependencies on cuDNN have been removed from the TensorRT 8.6.3 release to reduce the overall container size. Any TensorRT features which depend on cuDNN, which are primarily some plugins and samples, will not work with this release.
- Latest version of Ubuntu 22.04 with October 2023 updates.
Announcements
- Starting with the 23.11 release, TensorRT containers supporting iGPU architectures are published, and run on Jetson devices. Please refer to the Frameworks Support Matrix for information regarding which iGPU hardware/software is supported by which container.
- NVIDIA Deep Learning Profiler (DLProf) v1.8, which was included in the 21.12
container, was the last release of DLProf.
Starting with the 22.01 container, DLProf is longer included. It can still be manually installed by using a pip wheel on the nvidia-pyindex.
Obtaining Missing Data Files
Some samples require data files that are not included in the TensorRT container because of licensing restrictions, or because they are too large. Samples that do not include the required data files include a README.md file in the corresponding source directory that provides information about how to obtain the necessary data files.
Installing Required Python Modules
- To complete some of the samples, you might want to first run the Python setup script.
- If you need to install the missing Python modules and their dependencies, run the /opt/tensorrt/python/python_setup.sh script.
Installing Open Source Components
A script has been added to clone, build, and replace the provided plug-in, the Caffe parser, and the ONNX parser libraries with the open source ones that are based on the 24.04 tag on the official TensorRT open source repository.
To install the open source components in the container, run the following script:
/opt/tensorrt/install_opensource.sh -b main
For more information, see GitHub: TensorRT.
Limitations
- Accelerating Inference In TensorFlow with
TensorRT (TF-TRT) is not supported in the TensorRT containers.
Use the TensorFlow Container to accelerate through TF-TRT instead.
- Torch-TensorRT is not supported in the TensorRT containers.
Use the PyTorch Container to accelerate through Torch-TRT instead.
NVIDIA TensorRT Container Versions
The following table shows what versions of Ubuntu, CUDA, and TensorRT are supported in each of the NVIDIA containers for TensorRT. For older container versions, refer to the Frameworks Support Matrix.
Known Issues
- The onnx_graphsurgeon Python module on ARM Server systems is not compatible with ONNX version 1.11.0, which is normally recommended for the included TensorRT release. You will instead need to use ONNX version 1.15.0 to resolve a possible segmentation fault.
- With r545 or r550 drivers, some models may run into "Unspecified Launch Failure" during engine building. This can be worked around by downgrading the driver version to r535.
- TensorRT’s version compatibility feature has not been extensively tested and is therefore not supported with TensorRT 8.6.3. This TensorRT release is a special release that removes cuDNN as a dependency. Version compatibility between TensorRT 8.6.1 and future versions as documented will still be supported.
- Due to removing TensorRT’s dependency on cuDNN the following networks may
show performance regressions:
- BasicUnet
- DynUnet
- HighResNet
- StableDiffusion VAE-encoder
- StableDiffusion VAE-decoder
8. TensorRT Release 24.03
The NVIDIA container image for TensorRT, release 24.03, is available on NGC.
Contents of the TensorRT container
- The TensorRT C++ samples and C++ API documentation.
- Build the samples can be by running make in the /workspace/tensorrt/samples directory.
- The resulting executables are in the /workspace/tensorrt/bin directory.
- The C++ API documentation is in the /workspace/tensorrt/doc/cpp directory.
- The TensorRT Python samples and Python API documentation.
- The Python samples are in the
/workspace/tensorrt/samples/python directory.
Refer to the respective README documents for more samples.
- Many Python samples can be run by using python
<script.py> -d /workspace/tensorrt/data.
For example:
python onnx_resnet50.py -d /workspace/tensorrt/data
- The Python API documentation is in the /workspace/tensorrt/doc/python directory.
- The Python samples are in the
/workspace/tensorrt/samples/python directory.
- TensorRT 8.6.3.
The ONNX parser and plug-in libraries that are bundled with this container are built from TensorRT Open Source Software.
- Ubuntu 22.04Note: Container image 24.03-py3 contains Python 3.10.
- NVIDIA CUDA 12.4.0.41
- NVIDIA cuBLAS 12.4.2.65
- NVIDIA cuDNN 9.0.0.306
- NVIDIA NCCL 2.20Note: Although NCCL is packaged in the container, it does not affect TensorRT nor inferencing in any way.
- rdma-core 39.0
- OpenMPI 4.1.4+
- OpenUCX 1.15.0
- GDRCopy 2.3
- NVIDIA HPC-X 2.18
- Nsight Compute 2024.1.0.13
- Nsight Systems 2024.2.1.38
Driver Requirements
Release 24.03 is based on NVIDIA CUDA 12.4.0.41, which requires NVIDIA Driver release 545 or later. However, if you are running on a data center GPU (for example, T4 or any other data center GPU), you can use NVIDIA driver release 470.57 (or later R470), 525.85 (or later R525), 535.86 (or later R535), or 545.23 (or later R545).
The CUDA driver's compatibility package only supports particular drivers. Thus, users should upgrade from all R418, R440, R450, R460, R510, and R520 drivers, which are not forward-compatible with CUDA 12.3. For a complete list of supported drivers, see the CUDA Application Compatibility topic. For more information, see CUDA Compatibility and Upgrades.
GPU Requirements
Release 24.03 supports CUDA compute capability 6.0 and later. This corresponds to GPUs in the NVIDIA Pascal, NVIDIA Volta™, NVIDIA Turing™, NVIDIA Ampere architecture, NVIDIA Hopper™, and NVIDIA Ada Lovelace architecture families. For a list of GPUs to which this compute capability corresponds, see CUDA GPUs. For additional support details, see the Deep Learning Frameworks Support Matrix.
Key Features and Enhancements
- TensorRT container image version 24.03 is based on TensorRT 8.6.3.1.
For a list of the features and enhancements that were introduced in TensorRT 8.6, refer to the TensorRT 8.6 release notes.
- All dependencies on cuDNN have been removed from the TensorRT 8.6.3 release to reduce the overall container size. Any TensorRT features which depend on cuDNN, which are primarily some plugins and samples, will not work with this release.
- Latest version of Ubuntu 22.04 with October 2023 updates.
Announcements
- Starting with the 23.11 release, TensorRT containers supporting iGPU architectures are published, and run on Jetson devices. Please refer to the Frameworks Support Matrix for information regarding which iGPU hardware/software is supported by which container.
- NVIDIA Deep Learning Profiler (DLProf) v1.8, which was included in the 21.12
container, was the last release of DLProf.
Starting with the 22.01 container, DLProf is longer included. It can still be manually installed by using a pip wheel on the nvidia-pyindex.
Obtaining Missing Data Files
Some samples require data files that are not included in the TensorRT container because of licensing restrictions, or because they are too large. Samples that do not include the required data files include a README.md file in the corresponding source directory that provides information about how to obtain the necessary data files.
Installing Required Python Modules
- To complete some of the samples, you might want to first run the Python setup script.
- If you need to install the missing Python modules and their dependencies, run the /opt/tensorrt/python/python_setup.sh script.
Installing Open Source Components
A script has been added to clone, build, and replace the provided plug-in, the Caffe parser, and the ONNX parser libraries with the open source ones that are based on the 24.03 tag on the official TensorRT open source repository.
To install the open source components in the container, run the following script:
/opt/tensorrt/install_opensource.sh -b main
For more information, see GitHub: TensorRT.
Limitations
- Accelerating Inference In TensorFlow with
TensorRT (TF-TRT) is not supported in the TensorRT containers.
Use the TensorFlow Container to accelerate through TF-TRT instead.
- Torch-TensorRT is not supported in the TensorRT containers.
Use the PyTorch Container to accelerate through Torch-TRT instead.
NVIDIA TensorRT Container Versions
The following table shows what versions of Ubuntu, CUDA, and TensorRT are supported in each of the NVIDIA containers for TensorRT. For older container versions, refer to the Frameworks Support Matrix.
Known Issues
- The onnx_graphsurgeon Python module on ARM Server systems is not compatible with ONNX version 1.11.0, which is normally recommended for the included TensorRT release. You will instead need to use ONNX version 1.15.0 to resolve a possible segmentation fault.
- With r545 or r550 drivers, some models may run into "Unspecified Launch Failure" during engine building. This can be worked around by downgrading the driver version to r535.
- TensorRT’s version compatibility feature has not been extensively tested and is therefore not supported with TensorRT 8.6.3. This TensorRT release is a special release that removes cuDNN as a dependency. Version compatibility between TensorRT 8.6.1 and future versions as documented will still be supported.
- Due to removing TensorRT’s dependency on cuDNN the following networks may
show performance regressions:
- BasicUnet
- DynUnet
- HighResNet
- StableDiffusion VAE-encoder
- StableDiffusion VAE-decoder
9. TensorRT Release 24.02
The NVIDIA container image for TensorRT, release 24.02, is available on NGC.
Contents of the TensorRT container
- The TensorRT C++ samples and C++ API documentation.
- Build the samples can be by running make in the /workspace/tensorrt/samples directory.
- The resulting executables are in the /workspace/tensorrt/bin directory.
- The C++ API documentation is in the /workspace/tensorrt/doc/cpp directory.
- The TensorRT Python samples and Python API documentation.
- The Python samples are in the
/workspace/tensorrt/samples/python directory.
Refer to the respective README documents for more samples.
- Many Python samples can be run by using python
<script.py> -d /workspace/tensorrt/data.
For example:
python onnx_resnet50.py -d /workspace/tensorrt/data
- The Python API documentation is in the /workspace/tensorrt/doc/python directory.
- The Python samples are in the
/workspace/tensorrt/samples/python directory.
- TensorRT 8.6.3.
The ONNX parser and plug-in libraries that are bundled with this container are built from TensorRT Open Source Software.
- Ubuntu 22.04Note: Container image 24.01-py3 contains Python 3.10.
- NVIDIA CUDA® 12.3.2
- NVIDIA cuBLAS 12.3.4.1
- NVIDIA cuDNN 9.0.0.306
- NVIDIA NCCL 2.19.4Note: Although NCCL is packaged in the container, it does not affect TensorRT nor inferencing in any way.
- rdma-core 39.0
- OpenMPI 4.1.4+
- OpenUCX 1.15.0
- GDRCopy 2.3
- NVIDIA HPC-X 2.16rc4
- Nsight Compute 2023.3.1.1
- Nsight Systems 2023.4.1.97
Driver Requirements
Release 24.02 is based on CUDA 12.3.2, which requires NVIDIA Driver release 545 or later. However, if you are running on a data center GPU (for example, T4 or any other data center GPU), you can use NVIDIA driver release 470.57 (or later R470), 525.85 (or later R525), 535.86 (or later R535), or 545.23 (or later R545).
The CUDA driver's compatibility package only supports particular drivers. Thus, users should upgrade from all R418, R440, R450, R460, R510, and R520 drivers, which are not forward-compatible with CUDA 12.3. For a complete list of supported drivers, see the CUDA Application Compatibility topic. For more information, see CUDA Compatibility and Upgrades.
GPU Requirements
Release 24.02 supports CUDA compute capability 6.0 and later. This corresponds to GPUs in the NVIDIA Pascal, NVIDIA Volta™, NVIDIA Turing™, NVIDIA Ampere architecture, NVIDIA Hopper™, and NVIDIA Ada Lovelace architecture families. For a list of GPUs to which this compute capability corresponds, see CUDA GPUs. For additional support details, see the Deep Learning Frameworks Support Matrix.
Key Features and Enhancements
- TensorRT container image version 24.02 is based on TensorRT 8.6.3.1.
For a list of the features and enhancements that were introduced in TensorRT 8.6, refer to the TensorRT 8.6 release notes.
- All dependencies on cuDNN have been removed from the TensorRT 8.6.3 release to reduce the overall container size. Any TensorRT features which depend on cuDNN, which are primarily some plugins and samples, will not work with this release.
- Latest version of Ubuntu 22.04 with October 2023 updates.
Announcements
- Starting with the 23.11 release, TensorRT containers supporting iGPU architectures are published, and run on Jetson devices. Please refer to the Frameworks Support Matrix for information regarding which iGPU hardware/software is supported by which container.
- NVIDIA Deep Learning Profiler (DLProf) v1.8, which was included in the 21.12
container, was the last release of DLProf.
Starting with the 22.01 container, DLProf is longer included. It can still be manually installed by using a pip wheel on the nvidia-pyindex.
Obtaining Missing Data Files
Some samples require data files that are not included in the TensorRT container because of licensing restrictions, or because they are too large. Samples that do not include the required data files include a README.md file in the corresponding source directory that provides information about how to obtain the necessary data files.
Installing Required Python Modules
- To complete some of the samples, you might want to first run the Python setup script.
- If you need to install the missing Python modules and their dependencies, run the /opt/tensorrt/python/python_setup.sh script.
Installing Open Source Components
A script has been added to clone, build, and replace the provided plug-in, the Caffe parser, and the ONNX parser libraries with the open source ones that are based on the 24.01 tag on the official TensorRT open source repository.
To install the open source components in the container, run the following script:
/opt/tensorrt/install_opensource.sh -b main
For more information, see GitHub: TensorRT.
Limitations
- Accelerating Inference In TensorFlow with
TensorRT (TF-TRT) is not supported in the TensorRT containers.
Use the TensorFlow Container to accelerate through TF-TRT instead.
- Torch-TensorRT is not supported in the TensorRT containers.
Use the PyTorch Container to accelerate through Torch-TRT instead.
NVIDIA TensorRT Container Versions
The following table shows what versions of Ubuntu, CUDA, and TensorRT are supported in each of the NVIDIA containers for TensorRT. For older container versions, refer to the Frameworks Support Matrix.
Known Issues
- The onnx_graphsurgeon Python module on ARM Server systems is not compatible with ONNX version 1.11.0, which is normally recommended for the included TensorRT release. You will instead need to use ONNX version 1.15.0 to resolve a possible segmentation fault.
- With r545 or r550 drivers, some models may run into "Unspecified Launch Failure" during engine building. This can be worked around by downgrading the driver version to r535.
- TensorRT’s version compatibility feature has not been extensively tested and is therefore not supported with TensorRT 8.6.3. This TensorRT release is a special release that removes cuDNN as a dependency. Version compatibility between TensorRT 8.6.1 and future versions as documented will still be supported.
- Due to removing TensorRT’s dependency on cuDNN the following networks may
show performance regressions:
- BasicUnet
- DynUnet
- HighResNet
- StableDiffusion VAE-encoder
- StableDiffusion VAE-decoder
10. TensorRT Release 24.01
The NVIDIA container image for TensorRT, release 24.01, is available on NGC.
Contents of the TensorRT container
- The TensorRT C++ samples and C++ API documentation.
- Build the samples can be by running make in the /workspace/tensorrt/samples directory.
- The resulting executables are in the /workspace/tensorrt/bin directory.
- The C++ API documentation is in the /workspace/tensorrt/doc/cpp directory.
- The TensorRT Python samples and Python API documentation.
- The Python samples are in the
/workspace/tensorrt/samples/python directory.
Refer to the respective README documents for more samples.
- Many Python samples can be run by using python
<script.py> -d /workspace/tensorrt/data.
For example:
python onnx_resnet50.py -d /workspace/tensorrt/data
- The Python API documentation is in the /workspace/tensorrt/doc/python directory.
- The Python samples are in the
/workspace/tensorrt/samples/python directory.
- TensorRT 8.6.1.6.
The ONNX parser and plug-in libraries that are bundled with this container are built from TensorRT Open Source Software.
- Ubuntu 22.04Note: Container image 24.01-py3 contains Python 3.10.
- NVIDIA CUDA® 12.3.2
- NVIDIA cuBLAS 12.3.4.1
- NVIDIA cuDNN 8.9.7.29
- NVIDIA NCCL 2.19.4Note: Although NCCL is packaged in the container, it does not affect TensorRT nor inferencing in any way.
- rdma-core 39.0
- OpenMPI 4.1.4+
- OpenUCX 1.15.0
- GDRCopy 2.3
- NVIDIA HPC-X 2.16rc4
- Nsight Compute 2023.3.1.1
- Nsight Systems 2023.3.4.1.97
Driver Requirements
Release 24.01 is based on CUDA 12.3.2, which requires NVIDIA Driver release 545 or later. However, if you are running on a data center GPU (for example, T4 or any other data center GPU), you can use NVIDIA driver release 470.57 (or later R470), 525.85 (or later R525), 535.86 (or later R535), or 545.23 (or later R545).
The CUDA driver's compatibility package only supports particular drivers. Thus, users should upgrade from all R418, R440, R450, R460, R510, and R520 drivers, which are not forward-compatible with CUDA 12.3. For a complete list of supported drivers, see the CUDA Application Compatibility topic. For more information, see CUDA Compatibility and Upgrades.
GPU Requirements
Release 24.01 supports CUDA compute capability 6.0 and later. This corresponds to GPUs in the NVIDIA Pascal, NVIDIA Volta™, NVIDIA Turing™, NVIDIA Ampere architecture, NVIDIA Hopper™, and NVIDIA Ada Lovelace architecture families. For a list of GPUs to which this compute capability corresponds, see CUDA GPUs. For additional support details, see the Deep Learning Frameworks Support Matrix.
Key Features and Enhancements
- TensorRT container image version 24.01 is based on TensorRT 8.6.1.6.
For a list of the features and enhancements that were introduced in TensorRT 8.6.1, refer to the TensorRT 8.6 release notes.
- Ubuntu 22.04 with October 2023 updates.
Announcements
- Starting with the 23.11 release, TensorRT containers supporting iGPU architectures are published, and run on Jetson devices. Please refer to the Frameworks Support Matrix for information regarding which iGPU hardware/software is supported by which container.
- NVIDIA Deep Learning Profiler (DLProf) v1.8, which was included in the 21.12
container, was the last release of DLProf.
Starting with the 22.01 container, DLProf is longer included. It can still be manually installed by using a pip wheel on the nvidia-pyindex.
Obtaining Missing Data Files
Some samples require data files that are not included in the TensorRT container because of licensing restrictions, or because they are too large. Samples that do not include the required data files include a README.md file in the corresponding source directory that provides information about how to obtain the necessary data files.
Installing Required Python Modules
- To complete some of the samples, you might want to first run the Python setup script.
- If you need to install the missing Python modules and their dependencies, run the /opt/tensorrt/python/python_setup.sh script.
Installing Open Source Components
A script has been added to clone, build, and replace the provided plug-in, the Caffe parser, and the ONNX parser libraries with the open source ones that are based on the 24.01 tag on the official TensorRT open source repository.
To install the open source components in the container, run the following script:
/opt/tensorrt/install_opensource.sh -b main
For more information, see GitHub: TensorRT.
Limitations
- Accelerating Inference In TensorFlow with
TensorRT (TF-TRT) is not supported in the TensorRT containers.
Use the TensorFlow Container to accelerate through TF-TRT instead.
- Torch-TensorRT is not supported in the TensorRT containers.
Use the PyTorch Container to accelerate through Torch-TRT instead.
NVIDIA TensorRT Container Versions
The following table shows what versions of Ubuntu, CUDA, and TensorRT are supported in each of the NVIDIA containers for TensorRT. For older container versions, refer to the Frameworks Support Matrix.
Known Issues
None.
12. TensorRT Release 23.12
The NVIDIA container image for TensorRT, release 23.12, is available on NGC.
Contents of the TensorRT container
- The TensorRT C++ samples and C++ API documentation.
- Build the samples can be by running make in the /workspace/tensorrt/samples directory.
- The resulting executables are in the /workspace/tensorrt/bin directory.
- The C++ API documentation is in the /workspace/tensorrt/doc/cpp directory.
- The TensorRT Python samples and Python API documentation.
- The Python samples are in the
/workspace/tensorrt/samples/python directory.
Refer to the respective README documents for more samples.
- Many Python samples can be run by using python
<script.py> -d /workspace/tensorrt/data.
For example:
python onnx_resnet50.py -d /workspace/tensorrt/data
- The Python API documentation is in the /workspace/tensorrt/doc/python directory.
- The Python samples are in the
/workspace/tensorrt/samples/python directory.
- TensorRT 8.6.1.6.
The ONNX parser and plug-in libraries that are bundled with this container are built from TensorRT Open Source Software.
- Ubuntu 22.04Note: Container image 23.12-py3 contains Python 3.10.
- NVIDIA CUDA® 12.3.2
- NVIDIA cuBLAS 12.3.4.1
- NVIDIA cuDNN 8.9.7.29
- NVIDIA NCCL 2.19.3Note: Although NCCL is packaged in the container, it does not affect TensorRT nor inferencing in any way.
- rdma-core 39.0
- OpenMPI 4.1.4+
- OpenUCX 1.15.0
- GDRCopy 2.3
- NVIDIA HPC-X 2.16
- Nsight Compute 2023.3.1.1
- Nsight Systems 2023.3.4.1
Driver Requirements
Release 23.12 is based on CUDA 12.3.2, which requires NVIDIA Driver release 545 or later. However, if you are running on a data center GPU (for example, T4 or any other data center GPU), you can use NVIDIA driver release 450.51 (or later R450), 470.57 (or later R470), 510.47 (or later R510), or 525.85 (or later R525) 535.86 (or later R535), or 545.23 (or later R545).
The CUDA driver's compatibility package only supports particular drivers. Thus, users should upgrade from all R418, R440, R460, and R520 drivers, which are not forward-compatible with CUDA 12.3. For a complete list of supported drivers, see the CUDA Application Compatibility topic. For more information, see CUDA Compatibility and Upgrades.GPU Requirements
Release 23.12 supports CUDA compute capability 6.0 and later. This corresponds to GPUs in the NVIDIA Kepler, Maxwell, NVIDIA Pascal, NVIDIA Volta™, NVIDIA Turing™, NVIDIA Ampere architecture, NVIDIA Hopper™, and NVIDIA Ada Lovelace architecture families. For a list of GPUs to which this compute capability corresponds, see CUDA GPUs. For additional support details, see the Deep Learning Frameworks Support Matrix.
Key Features and Enhancements
- TensorRT container image version 23.12 is based on TensorRT 8.6.1.6.
For a list of the features and enhancements that were introduced in TensorRT 8.6.1, refer to the TensorRT 8.6 release notes.
- Ubuntu 22.04 with October 2023 updates.
Announcements
- Starting with the 23.11 release, TensorRT containers supporting iGPU architectures are published, and run on Jetson devices. Please refer to the Frameworks Support Matrix for information regarding which iGPU hardware/software is supported by which container.
- NVIDIA Deep Learning Profiler (DLProf) v1.8, which was included in the 21.12
container, was the last release of DLProf.
Starting with the 22.01 container, DLProf is longer included. It can still be manually installed by using a pip wheel on the nvidia-pyindex.
Obtaining Missing Data Files
Some samples require data files that are not included in the TensorRT container because of licensing restrictions, or because they are too large. Samples that do not include the required data files include a README.md file in the corresponding source directory that provides information about how to obtain the necessary data files.
Installing Required Python Modules
- To complete some of the samples, you might want to first run the Python setup script.
- If you need to install the missing Python modules and their dependencies, run the /opt/tensorrt/python/python_setup.sh script.
Installing Open Source Components
A script has been added to clone, build, and replace the provided plug-in, the Caffe parser, and the ONNX parser libraries with the open source ones that are based on the 23.12 tag on the official TensorRT open source repository.
To install the open source components in the container, run the following script:
/opt/tensorrt/install_opensource.sh -b main
For more information, see GitHub: TensorRT.
Limitations
- Accelerating Inference In TensorFlow with
TensorRT (TF-TRT) is not supported in the TensorRT containers.
Use the TensorFlow Container to accelerate through TF-TRT instead.
- Torch-TensorRT is not supported in the TensorRT containers.
Use the PyTorch Container to accelerate through Torch-TRT instead.
NVIDIA TensorRT Container Versions
The following table shows what versions of Ubuntu, CUDA, and TensorRT are supported in each of the NVIDIA containers for TensorRT. For older container versions, refer to the Frameworks Support Matrix.
Known Issues
None.
TensorRT Release 23.11
The NVIDIA container image for TensorRT, release 23.11, is available on NGC.
Contents of the TensorRT container
- The TensorRT C++ samples and C++ API documentation.
- Build the samples can be by running make in the /workspace/tensorrt/samples directory.
- The resulting executables are in the /workspace/tensorrt/bin directory.
- The C++ API documentation is in the /workspace/tensorrt/doc/cpp directory.
- The TensorRT Python samples and Python API documentation.
- The Python samples are in the
/workspace/tensorrt/samples/python directory.
Refer to the respective README documents for more samples.
- Many Python samples can be run by using python
<script.py> -d /workspace/tensorrt/data.
For example:
python onnx_resnet50.py -d /workspace/tensorrt/data
- The Python API documentation is in the /workspace/tensorrt/doc/python directory.
- The Python samples are in the
/workspace/tensorrt/samples/python directory.
- TensorRT 8.6.1.6.
The ONNX parser and plug-in libraries that are bundled with this container are built from TensorRT Open Source Software.
- Ubuntu 22.04Note: Container image 23.11-py3 contains Python 3.10.
- NVIDIA CUDA® 12.3.0
- NVIDIA cuBLAS 12.3.2.1
- NVIDIA cuDNN 8.9.6
- NVIDIA NCCL 2.19.3Note: Although NCCL is packaged in the container, it does not affect TensorRT nor inferencing in any way.
- rdma-core 39.0
- OpenMPI 4.1.4+
- OpenUCX 1.15.0
- GDRCopy 2.3
- NVIDIA HPC-X 2.16
- Nsight Compute 2023.3.0.12
- Nsight Systems 2023.3.1.92
Driver Requirements
Release 23.11 is based on CUDA 12.3.0, which requires NVIDIA Driver release 545 or later. However, if you are running on a data center GPU (for example, T4 or any other data center GPU), you can use NVIDIA driver release 450.51 (or later R450), 470.57 (or later R470), 510.47 (or later R510), or 525.85 (or later R525) 535.86 (or later R535), or 545.23 (or later R545).
The CUDA driver's compatibility package only supports particular drivers. Thus, users should upgrade from all R418, R440, R460, and R520 drivers, which are not forward-compatible with CUDA 12.3. For a complete list of supported drivers, see the CUDA Application Compatibility topic. For more information, see CUDA Compatibility and Upgrades.GPU Requirements
Release 23.11 supports CUDA compute capability 6.0 and later. This corresponds to GPUs in the NVIDIA Kepler, Maxwell, NVIDIA Pascal, NVIDIA Volta™, NVIDIA Turing™, NVIDIA Ampere architecture, NVIDIA Hopper™, and NVIDIA Ada Lovelace architecture families. For a list of GPUs to which this compute capability corresponds, see CUDA GPUs. For additional support details, see the Deep Learning Frameworks Support Matrix.
Key Features and Enhancements
- TensorRT container image version 23.11 is based on TensorRT 8.6.1.6.
For a list of the features and enhancements that were introduced in TensorRT 8.6.1, refer to the TensorRT 8.6 release notes.
- Ubuntu 22.04 with October 2023 updates.
Announcements
- Starting with the 23.11 release, TensorRT containers supporting iGPU architectures are published, and run on Jetson devices. Please refer to the Frameworks Support Matrix for information regarding which iGPU hardware/software is supported by which container.
- NVIDIA Deep Learning Profiler (DLProf) v1.8, which was included in the 21.12
container, was the last release of DLProf.
Starting with the 22.01 container, DLProf is longer included. It can still be manually installed by using a pip wheel on the nvidia-pyindex.
Obtaining Missing Data Files
Some samples require data files that are not included in the TensorRT container because of licensing restrictions, or because they are too large. Samples that do not include the required data files include a README.md file in the corresponding source directory that provides information about how to obtain the necessary data files.
Installing Required Python Modules
- To complete some of the samples, you might want to first run the Python setup script.
- If you need to install the missing Python modules and their dependencies, run the /opt/tensorrt/python/python_setup.sh script.
Installing Open Source Components
A script has been added to clone, build, and replace the provided plug-in, the Caffe parser, and the ONNX parser libraries with the open source ones that are based on the 23.11 tag on the official TensorRT open source repository.
To install the open source components in the container, run the following script:
/opt/tensorrt/install_opensource.sh -b main
For more information, see GitHub: TensorRT.
Limitations
- Accelerating Inference In TensorFlow with
TensorRT (TF-TRT) is not supported in the TensorRT containers.
Use the TensorFlow Container to accelerate through TF-TRT instead.
- Torch-TensorRT is not supported in the TensorRT containers.
Use the PyTorch Container to accelerate through Torch-TRT instead.
NVIDIA TensorRT Container Versions
The following table shows what versions of Ubuntu, CUDA, and TensorRT are supported in each of the NVIDIA containers for TensorRT. For older container versions, refer to the Frameworks Support Matrix.
Known Issues
None.
13. TensorRT Release 23.10
The NVIDIA container image for TensorRT, release 23.10, is available on NGC.
Contents of the TensorRT container
- The TensorRT C++ samples and C++ API documentation.
- Build the samples can be by running make in the /workspace/tensorrt/samples directory.
- The resulting executables are in the /workspace/tensorrt/bin directory.
- The C++ API documentation is in the /workspace/tensorrt/doc/cpp directory.
- The TensorRT Python samples and Python API documentation.
- The Python samples are in the
/workspace/tensorrt/samples/python directory.
Refer to the respective README documents for more samples.
- Many Python samples can be run by using python
<script.py> -d /workspace/tensorrt/data.
For example:
python onnx_resnet50.py -d /workspace/tensorrt/data
- The Python API documentation is in the /workspace/tensorrt/doc/python directory.
- The Python samples are in the
/workspace/tensorrt/samples/python directory.
- TensorRT 8.6.1.6.
The ONNX parser and plug-in libraries that are bundled with this container are built from TensorRT Open Source Software.
- Ubuntu 22.04Note: Container image 23.10-py3 contains Python 3.10.
- NVIDIA CUDA® 12.2.2
- NVIDIA cuBLAS 12.2.5.6
- NVIDIA cuDNN 8.9.5
- NVIDIA NCCL 2.19.3Note: Although NCCL is packaged in the container, it does not affect TensorRT nor inferencing in any way.
- rdma-core 39.0
- OpenMPI 4.1.4+
- OpenUCX 1.15.0
- GDRCopy 2.3
- NVIDIA HPC-X 2.16
- Nsight Compute 2023.2.1.3
- Nsight Systems 2023.3.1.92
Driver Requirements
Release 23.10 is based on CUDA 12.2.2, which requires NVIDIA Driver release 535 or later. However, if you are running on a data center GPU (for example, T4 or any other data center GPU), you can use NVIDIA driver release 450.51 (or later R450), 470.57 (or later R470), 510.47 (or later R510), or 525.85 (or later R525), or 535.86 (or later R535).
The CUDA driver's compatibility package only supports particular drivers. Thus, users should upgrade from all R418, R440, R460, and R520 drivers, which are not forward-compatible with CUDA 12.2. For a complete list of supported drivers, see the CUDA Application Compatibility topic. For more information, see CUDA Compatibility and Upgrades.GPU Requirements
Release 23.10 supports CUDA compute capability 6.0 and later. This corresponds to GPUs in the NVIDIA Kepler, Maxwell, NVIDIA Pascal, NVIDIA Volta™, NVIDIA Turing™, NVIDIA Ampere architecture, NVIDIA Hopper™, and NVIDIA Ada Lovelace architecture families. For a list of GPUs to which this compute capability corresponds, see CUDA GPUs. For additional support details, see the Deep Learning Frameworks Support Matrix.
Key Features and Enhancements
- TensorRT container image version 23.10 is based on TensorRT 8.6.1.6.
For a list of the features and enhancements that were introduced in TensorRT 8.6.1, refer to the TensorRT 8.6 release notes.
- Ubuntu 22.04 with September 2023 updates.
Announcements
- NVIDIA Deep Learning Profiler (DLProf) v1.8, which was included in the 21.12
container, was the last release of DLProf.
Starting with the 22.01 container, DLProf is longer included. It can still be manually installed by using a pip wheel on the nvidia-pyindex.
Obtaining Missing Data Files
Some samples require data files that are not included in the TensorRT container because of licensing restrictions, or because they are too large. Samples that do not include the required data files include a README.md file in the corresponding source directory that provides information about how to obtain the necessary data files.
Installing Required Python Modules
- To complete some of the samples, you might want to first run the Python setup script.
- If you need to install the missing Python modules and their dependencies, run the /opt/tensorrt/python/python_setup.sh script.
Installing Open Source Components
A script has been added to clone, build, and replace the provided plug-in, the Caffe parser, and the ONNX parser libraries with the open source ones that are based on the 22.05 tag on the official TensorRT open source repository.
To install the open source components in the container, run the following script:
/opt/tensorrt/install_opensource.sh -b main
For more information, see GitHub: TensorRT.
Limitations
- Accelerating Inference In TensorFlow with
TensorRT (TF-TRT) is not supported in the TensorRT containers.
Use the TensorFlow Container to accelerate through TF-TRT instead.
- Torch-TensorRT is not supported in the TensorRT containers.
Use the PyTorch Container to accelerate through Torch-TRT instead.
NVIDIA TensorRT Container Versions
The following table shows what versions of Ubuntu, CUDA, and TensorRT are supported in each of the NVIDIA containers for TensorRT. For older container versions, refer to the Frameworks Support Matrix.
Known Issues
None.
14. TensorRT Release 23.09
The NVIDIA container image for TensorRT, release 23.09, is available on NGC.
Contents of the TensorRT container
- The TensorRT C++ samples and C++ API documentation.
- Build the samples can be by running make in the /workspace/tensorrt/samples directory.
- The resulting executables are in the /workspace/tensorrt/bin directory.
- The C++ API documentation is in the /workspace/tensorrt/doc/cpp directory.
- The TensorRT Python samples and Python API documentation.
- The Python samples are in the
/workspace/tensorrt/samples/python directory.
Refer to the respective README documents for more samples.
- Many Python samples can be run by using python
<script.py> -d /workspace/tensorrt/data.
For example:
python onnx_resnet50.py -d /workspace/tensorrt/data
- The Python API documentation is in the /workspace/tensorrt/doc/python directory.
- The Python samples are in the
/workspace/tensorrt/samples/python directory.
- TensorRT 8.6.1.6.
The ONNX parser and plug-in libraries that are bundled with this container are built from TensorRT Open Source Software.
- Ubuntu 22.04Note: Container image 23.09-py3 contains Python 3.10.
- NVIDIA CUDA® 12.2.1
- NVIDIA cuBLAS 12.2.5.6
- NVIDIA cuDNN 8.9.5
- NVIDIA NCCL 2.18.5Note: Although NCCL is packaged in the container, it does not affect TensorRT nor inferencing in any way.
- rdma-core 39.0
- OpenMPI 4.1.4+
- OpenUCX 1.15.0
- GDRCopy 2.3
- NVIDIA HPC-X 2.16
- Nsight Compute 2023.2.1.3
- Nsight Systems 2023.3.1.92
Driver Requirements
Release 23.09 is based on CUDA 12.2.1, which requires NVIDIA Driver release 535 or later. However, if you are running on a data center GPU (for example, T4 or any other data center GPU), you can use NVIDIA driver release 450.51 (or later R450), 470.57 (or later R470), 510.47 (or later R510), or 525.85 (or later R525), or 535.86 (or later R535).
The CUDA driver's compatibility package only supports particular drivers. Thus, users should upgrade from all R418, R440, R460, and R520 drivers, which are not forward-compatible with CUDA 12.2. For a complete list of supported drivers, see the CUDA Application Compatibility topic. For more information, see CUDA Compatibility and Upgrades.GPU Requirements
Release 23.09 supports CUDA compute capability 6.0 and later. This corresponds to GPUs in the NVIDIA Kepler, Maxwell, NVIDIA Pascal, NVIDIA Volta™, NVIDIA Turing™, NVIDIA Ampere architecture, NVIDIA Hopper™, and NVIDIA Ada Lovelace architecture families. For a list of GPUs to which this compute capability corresponds, see CUDA GPUs. For additional support details, see the Deep Learning Frameworks Support Matrix.
Key Features and Enhancements
- TensorRT container image version 23.09 is based on TensorRT 8.6.1.6.
For a list of the features and enhancements that were introduced in TensorRT 8.6.1, refer to the TensorRT 8.6 release notes.
- Ubuntu 22.04 with August 2023 updates.
Announcements
- NVIDIA Deep Learning Profiler (DLProf) v1.8, which was included in the 21.12
container, was the last release of DLProf.
Starting with the 22.01 container, DLProf is longer included. It can still be manually installed by using a pip wheel on the nvidia-pyindex.
Obtaining Missing Data Files
Some samples require data files that are not included in the TensorRT container because of licensing restrictions, or because they are too large. Samples that do not include the required data files include a README.md file in the corresponding source directory that provides information about how to obtain the necessary data files.
Installing Required Python Modules
- To complete some of the samples, you might want to first run the Python setup script.
- If you need to install the missing Python modules and their dependencies, run the /opt/tensorrt/python/python_setup.sh script.
Installing Open Source Components
A script has been added to clone, build, and replace the provided plug-in, the Caffe parser, and the ONNX parser libraries with the open source ones that are based on the 22.05 tag on the official TensorRT open source repository.
To install the open source components in the container, run the following script:
/opt/tensorrt/install_opensource.sh -b main
For more information, see GitHub: TensorRT.
Limitations
- Accelerating Inference In TensorFlow with
TensorRT (TF-TRT) is not supported in the TensorRT containers.
Use the TensorFlow Container to accelerate through TF-TRT instead.
- Torch-TensorRT is not supported in the TensorRT containers.
Use the PyTorch Container to accelerate through Torch-TRT instead.
NVIDIA TensorRT Container Versions
The following table shows what versions of Ubuntu, CUDA, and TensorRT are supported in each of the NVIDIA containers for TensorRT. For older container versions, refer to the Frameworks Support Matrix.
Known Issues
None.
15. TensorRT Release 23.08
The NVIDIA container image for TensorRT, release 23.08, is available on NGC.
Contents of the TensorRT container
- The TensorRT C++ samples and C++ API documentation.
- Build the samples can be by running make in the /workspace/tensorrt/samples directory.
- The resulting executables are in the /workspace/tensorrt/bin directory.
- The C++ API documentation is in the /workspace/tensorrt/doc/cpp directory.
- The TensorRT Python samples and Python API documentation.
- The Python samples are in the
/workspace/tensorrt/samples/python directory.
Refer to the respective README documents for more samples.
- Many Python samples can be run by using python
<script.py> -d /workspace/tensorrt/data.
For example:
python onnx_resnet50.py -d /workspace/tensorrt/data
- The Python API documentation is in the /workspace/tensorrt/doc/python directory.
- The Python samples are in the
/workspace/tensorrt/samples/python directory.
- TensorRT 8.6.1.6.
The ONNX parser and plug-in libraries that are bundled with this container are built from TensorRT Open Source Software.
- Ubuntu 22.04Note: Container image 23.08-py3 contains Python 3.10.
- NVIDIA CUDA® 12.2.1
- NVIDIA cuBLAS 12.2.5.1
- NVIDIA cuDNN 8.9.4
- NVIDIA NCCL 2.18.3Note: Although NCCL is packaged in the container, it does not affect TensorRT nor inferencing in any way.
- rdma-core 39.0
- OpenMPI 4.1.4+
- OpenUCX 1.15.0
- GDRCopy 2.3
- NVIDIA HPC-X 2.15
- Nsight Compute 2023.2.1.3
- Nsight Systems 2023.2.3.1001
Driver Requirements
Release 23.08 is based on CUDA 12.2.1, which requires NVIDIA Driver release 535 or later. However, if you are running on a data center GPU (for example, T4 or any other data center GPU), you can use NVIDIA driver release 450.51 (or later R450), 470.57 (or later R470), 510.47 (or later R510), or 525.85 (or later R525), or 535.86 (or later R535).
The CUDA driver's compatibility package only supports particular drivers. Thus, users should upgrade from all R418, R440, R460, and R520 drivers, which are not forward-compatible with CUDA 12.2. For a complete list of supported drivers, see the CUDA Application Compatibility topic. For more information, see CUDA Compatibility and Upgrades.GPU Requirements
Release 23.08 supports CUDA compute capability 6.0 and later. This corresponds to GPUs in the NVIDIA Kepler, Maxwell, NVIDIA Pascal, NVIDIA Volta™, NVIDIA Turing™, NVIDIA Ampere architecture, NVIDIA Hopper™, and NVIDIA Ada Lovelace architecture families. For a list of GPUs to which this compute capability corresponds, see CUDA GPUs. For additional support details, see the Deep Learning Frameworks Support Matrix.
Key Features and Enhancements
- TensorRT container image version 23.08 is based on TensorRT 8.6.1.6.
For a list of the features and enhancements that were introduced in TensorRT 8.6.1, refer to the TensorRT 8.6 release notes.
- Ubuntu 22.04 with July 2023 updates.
Announcements
- NVIDIA Deep Learning Profiler (DLProf) v1.8, which was included in the 21.12
container, was the last release of DLProf.
Starting with the 22.01 container, DLProf is longer included. It can still be manually installed by using a pip wheel on the nvidia-pyindex.
Obtaining Missing Data Files
Some samples require data files that are not included in the TensorRT container because of licensing restrictions, or because they are too large. Samples that do not include the required data files include a README.md file in the corresponding source directory that provides information about how to obtain the necessary data files.
Installing Required Python Modules
- To complete some of the samples, you might want to first run the Python setup script.
- If you need to install the missing Python modules and their dependencies, run the /opt/tensorrt/python/python_setup.sh script.
Installing Open Source Components
A script has been added to clone, build, and replace the provided plug-in, the Caffe parser, and the ONNX parser libraries with the open source ones that are based on the 22.05 tag on the official TensorRT open source repository.
To install the open source components in the container, run the following script:
/opt/tensorrt/install_opensource.sh -b main
For more information, see GitHub: TensorRT.
Limitations
- Accelerating Inference In TensorFlow with
TensorRT (TF-TRT) is not supported in the TensorRT containers.
Use the TensorFlow Container to accelerate through TF-TRT instead.
- Torch-TensorRT is not supported in the TensorRT containers.
Use the PyTorch Container to accelerate through Torch-TRT instead.
NVIDIA TensorRT Container Versions
The following table shows what versions of Ubuntu, CUDA, and TensorRT are supported in each of the NVIDIA containers for TensorRT. For older container versions, refer to the Frameworks Support Matrix.
Known Issues
None.
16. TensorRT Release 23.07
The NVIDIA container image for TensorRT, release 23.07, is available on NGC.
Contents of the TensorRT container
- The TensorRT C++ samples and C++ API documentation.
- Build the samples can be by running make in the /workspace/tensorrt/samples directory.
- The resulting executables are in the /workspace/tensorrt/bin directory.
- The C++ API documentation is in the /workspace/tensorrt/doc/cpp directory.
- The TensorRT Python samples and Python API documentation.
- The Python samples are in the
/workspace/tensorrt/samples/python directory.
Refer to the respective README documents for more samples.
- Many Python samples can be run by using python
<script.py> -d /workspace/tensorrt/data.
For example:
python onnx_resnet50.py -d /workspace/tensorrt/data
- The Python API documentation is in the /workspace/tensorrt/doc/python directory.
- The Python samples are in the
/workspace/tensorrt/samples/python directory.
- TensorRT 8.6.1.6.
The ONNX parser and plug-in libraries that are bundled with this container are built from TensorRT Open Source Software.
- Ubuntu 22.04Note: Container image 23.07-py3 contains Python 3.10.
- NVIDIA CUDA® 12.1.1
- NVIDIA cuBLAS 12.1.3.1
- NVIDIA cuDNN 8.9.3
- NVIDIA NCCL 2.18.3Note: Although NCCL is packaged in the container, it does not affect TensorRT nor inferencing in any way.
- rdma-core 39.0
- OpenMPI 4.1.4+
- OpenUCX 1.15.0
- GDRCopy 2.3
- NVIDIA HPC-X 2.15
- Nsight Compute 2023.1.1.4
- Nsight Systems 2023.2.3.1001
Driver Requirements
Release 23.07 is based on CUDA 12.1.1, which requires NVIDIA Driver release 530 or later. However, if you are running on a data center GPU (for example, T4 or any other data center GPU), you can use NVIDIA driver release 450.51 (or later R450), 470.57 (or later R470), 510.47 (or later R510), 515.65 (or later R515), 525.85 (or later R525), or 530.30 (or later R530).
The CUDA driver's compatibility package only supports particular drivers. Thus, users should upgrade from all R418, R440, R460, and R520 drivers, which are not forward-compatible with CUDA 12.1. For a complete list of supported drivers, see the CUDA Application Compatibility topic. For more information, see CUDA Compatibility and Upgrades.GPU Requirements
Release 23.07 supports CUDA compute capability 3.5 and later. This corresponds to GPUs in the NVIDIA Kepler, Maxwell, NVIDIA Pascal, NVIDIA Volta™, NVIDIA Turing™, NVIDIA Ampere architecture, NVIDIA Hopper™, and NVIDIA Ada Lovelace architecture families. For a list of GPUs to which this compute capability corresponds, see CUDA GPUs. For additional support details, see the Deep Learning Frameworks Support Matrix.
Key Features and Enhancements
- TensorRT container image version 23.07 is based on TensorRT 8.6.1.6.
For a list of the features and enhancements that were introduced in TensorRT 8.6.1, refer to the TensorRT 8.6 release notes.
- Ubuntu 22.04 with June 2023 updates.
Announcements
- NVIDIA Deep Learning Profiler (DLProf) v1.8, which was included in the 21.12
container, was the last release of DLProf.
Starting with the 22.01 container, DLProf is longer included. It can still be manually installed by using a pip wheel on the nvidia-pyindex.
Obtaining Missing Data Files
Some samples require data files that are not included in the TensorRT container because of licensing restrictions, or because they are too large. Samples that do not include the required data files include a README.md file in the corresponding source directory that provides information about how to obtain the necessary data files.
Installing Required Python Modules
- To complete some of the samples, you might want to first run the Python setup script.
- If you need to install the missing Python modules and their dependencies, run the /opt/tensorrt/python/python_setup.sh script.
Installing Open Source Components
A script has been added to clone, build, and replace the provided plug-in, the Caffe parser, and the ONNX parser libraries with the open source ones that are based on the 22.05 tag on the official TensorRT open source repository.
To install the open source components in the container, run the following script:
/opt/tensorrt/install_opensource.sh -b main
For more information, see GitHub: TensorRT.
Limitations
- Accelerating Inference In TensorFlow with
TensorRT (TF-TRT) is not supported in the TensorRT containers.
Use the TensorFlow Container to accelerate through TF-TRT instead.
- Torch-TensorRT is not supported in the TensorRT containers.
Use the PyTorch Container to accelerate through Torch-TRT instead.
NVIDIA TensorRT Container Versions
The following table shows what versions of Ubuntu, CUDA, and TensorRT are supported in each of the NVIDIA containers for TensorRT. For older container versions, refer to the Frameworks Support Matrix.
Known Issues
None.
17. TensorRT Release 23.06
The NVIDIA container image for TensorRT, release 23.06, is available on NGC.
Contents of the TensorRT container
- The TensorRT C++ samples and C++ API documentation.
- Build the samples can be by running make in the /workspace/tensorrt/samples directory.
- The resulting executables are in the /workspace/tensorrt/bin directory.
- The C++ API documentation is in the /workspace/tensorrt/doc/cpp directory.
- The TensorRT Python samples and Python API documentation.
- The Python samples are in the
/workspace/tensorrt/samples/python directory.
Refer to the respective README documents for more samples.
- Many Python samples can be run by using python
<script.py> -d /workspace/tensorrt/data.
For example:
python onnx_resnet50.py -d /workspace/tensorrt/data
- The Python API documentation is in the /workspace/tensorrt/doc/python directory.
- The Python samples are in the
/workspace/tensorrt/samples/python directory.
- TensorRT 8.6.1.6.
The ONNX parser and plug-in libraries that are bundled with this container are built from TensorRT Open Source Software.
- Ubuntu 22.04Note: Container image 23.06-py3 contains Python 3.10.
- NVIDIA CUDA® 12.1.1
- NVIDIA cuBLAS 12.1.3.1
- NVIDIA cuDNN 8.9.2
- NVIDIA NCCL 2.18.1Note: Although NCCL is packaged in the container, it does not affect TensorRT nor inferencing in any way.
- rdma-core 39.0
- OpenMPI 4.1.4+
- OpenUCX 1.15.0
- GDRCopy 2.3
- NVIDIA HPC-X 2.15
- Nsight Compute 2023.1.1.4
- Nsight Systems 2023.2.3.1001
Driver Requirements
Release 23.06 is based on CUDA 12.1.1, which requires NVIDIA Driver release 530 or later. However, if you are running on a data center GPU (for example, T4 or any other data center GPU), you can use NVIDIA driver release 450.51 (or later R450), 470.57 (or later R470), 510.47 (or later R510), 515.65 (or later R515), 525.85 (or later R525), or 530.30 (or later R530).
The CUDA driver's compatibility package only supports particular drivers. Thus, users should upgrade from all R418, R440, R460, and R520 drivers, which are not forward-compatible with CUDA 12.1. For a complete list of supported drivers, see the CUDA Application Compatibility topic. For more information, see CUDA Compatibility and Upgrades.GPU Requirements
Release 23.06 supports CUDA compute capability 3.5 and later. This corresponds to GPUs in the NVIDIA Kepler, Maxwell, NVIDIA Pascal, NVIDIA Volta™, NVIDIA Turing™, NVIDIA Ampere architecture, NVIDIA Hopper™, and NVIDIA Ada Lovelace architecture families. For a list of GPUs to which this compute capability corresponds, see CUDA GPUs. For additional support details, see the Deep Learning Frameworks Support Matrix.
Key Features and Enhancements
- TensorRT container image version 23.06 is based on TensorRT 8.6.1.6.
For a list of the features and enhancements that were introduced in TensorRT 8.6.1, refer to the TensorRT 8.6 release notes.
- Ubuntu 22.04 with May 2023 updates.
Announcements
- NVIDIA Deep Learning Profiler (DLProf) v1.8, which was included in the 21.12
container, was the last release of DLProf.
Starting with the 22.01 container, DLProf is longer included. It can still be manually installed by using a pip wheel on the nvidia-pyindex.
Obtaining Missing Data Files
Some samples require data files that are not included in the TensorRT container because of licensing restrictions, or because they are too large. Samples that do not include the required data files include a README.md file in the corresponding source directory that provides information about how to obtain the necessary data files.
Installing Required Python Modules
- To complete some of the samples, you might want to first run the Python setup script.
- If you need to install the missing Python modules and their dependencies, run the /opt/tensorrt/python/python_setup.sh script.
Installing Open Source Components
A script has been added to clone, build, and replace the provided plug-in, the Caffe parser, and the ONNX parser libraries with the open source ones that are based on the 22.05 tag on the official TensorRT open source repository.
To install the open source components in the container, run the following script:
/opt/tensorrt/install_opensource.sh -b main
For more information, see GitHub: TensorRT.
Limitations
- Accelerating Inference In TensorFlow with
TensorRT (TF-TRT) is not supported in the TensorRT containers.
Use the TensorFlow Container to accelerate through TF-TRT instead.
- Torch-TensorRT is not supported in the TensorRT containers.
Use the PyTorch Container to accelerate through Torch-TRT instead.
NVIDIA TensorRT Container Versions
The following table shows what versions of Ubuntu, CUDA, and TensorRT are supported in each of the NVIDIA containers for TensorRT. For older container versions, refer to the Frameworks Support Matrix.
Known Issues
None.
18. TensorRT Release 23.05
The NVIDIA container image for TensorRT, release 23.05, is available on NGC.
Contents of the TensorRT container
- The TensorRT C++ samples and C++ API documentation.
- Build the samples can be by running make in the /workspace/tensorrt/samples directory.
- The resulting executables are in the /workspace/tensorrt/bin directory.
- The C++ API documentation is in the /workspace/tensorrt/doc/cpp directory.
- The TensorRT Python samples and Python API documentation.
- The Python samples are in the
/workspace/tensorrt/samples/python directory.
Refer to the respective README documents for more samples.
- Many Python samples can be run by using python
<script.py> -d /workspace/tensorrt/data.
For example:
python onnx_resnet50.py -d /workspace/tensorrt/data
- The Python API documentation is in the /workspace/tensorrt/doc/python directory.
- The Python samples are in the
/workspace/tensorrt/samples/python directory.
- TensorRT 8.6.1.2.
The ONNX parser and plug-in libraries that are bundled with this container are built from TensorRT Open Source Software.
- Ubuntu 22.04Note: Container image 23.05-py3 contains Python 3.10.
- NVIDIA CUDA® 12.1.1
- NVIDIA cuBLAS 12.1.3.1
- NVIDIA cuDNN 8.9.1.23
- NVIDIA NCCL 2.18.1Note: Although NCCL is packaged in the container, it does not affect TensorRT nor inferencing in any way.
- rdma-core 36.0
- OpenMPI 4.1.4+
- OpenUCX 1.14.0
- GDRCopy 2.3
- NVIDIA HPC-X 2.14
- Nsight Compute 2023.1.1.4
- Nsight Systems 2023.2
Driver Requirements
Release 23.05 is based on CUDA 12.1.1, which requires NVIDIA Driver release 530 or later. However, if you are running on a data center GPU (for example, T4 or any other data center GPU), you can use NVIDIA driver release 450.51 (or later R450), 470.57 (or later R470), 510.47 (or later R510), 515.65 (or later R515), 525.85 (or later R525), or 530.30 (or later R530).
The CUDA driver's compatibility package only supports particular drivers. Thus, users should upgrade from all R418, R440, R460, and R520 drivers, which are not forward-compatible with CUDA 12.1. For a complete list of supported drivers, see the CUDA Application Compatibility topic. For more information, see CUDA Compatibility and Upgrades.GPU Requirements
Release 23.05 supports CUDA compute capability 3.5 and later. This corresponds to GPUs in the NVIDIA Kepler, Maxwell, NVIDIA Pascal, NVIDIA Volta™, NVIDIA Turing™, NVIDIA Ampere architecture, NVIDIA Hopper™, and NVIDIA Ada Lovelace architecture families. For a list of GPUs to which this compute capability corresponds, see CUDA GPUs. For additional support details, see the Deep Learning Frameworks Support Matrix.
Key Features and Enhancements
- TensorRT container image version 23.05 is based on TensorRT 8.6.1.2.
For a list of the features and enhancements that were introduced in TensorRT 8.6.1, refer to the TensorRT 8.6 release notes.
- Ubuntu 22.04 with April 2023 updates.
Announcements
- Starting with the 22.05 release, the TensorRT container is available for the Arm
SBSA platform.
For example, when you pull the nvcr.io/nvidia/tensorrt:22.05-py3 Docker image on an Arm SBSA machine, the Arm-specific image is automatically fetched.
- NVIDIA Deep Learning Profiler (DLProf) v1.8, which was included in the 21.12
container, was the last release of DLProf.
Starting with the 22.01 container, DLProf is longer included. It can still be manually installed by using a pip wheel on the nvidia-pyindex.
Obtaining Missing Data Files
Some samples require data files that are not included in the TensorRT container because of licensing restrictions, or because they are too large. Samples that do not include the required data files include a README.md file in the corresponding source directory that provides information about how to obtain the necessary data files.
Installing Required Python Modules
- To complete some of the samples, you might want to first run the Python setup script.
- If you need to install the missing Python modules and their dependencies, run the /opt/tensorrt/python/python_setup.sh script.
Installing Open Source Components
A script has been added to clone, build, and replace the provided plug-in, the Caffe parser, and the ONNX parser libraries with the open source ones that are based on the 22.05 tag on the official TensorRT open source repository.
To install the open source components in the container, run the following script:
/opt/tensorrt/install_opensource.sh -b main
For more information, see GitHub: TensorRT.
Limitations
- Accelerating Inference In TensorFlow with
TensorRT (TF-TRT) is not supported in the TensorRT containers.
Use the TensorFlow Container to accelerate through TF-TRT instead.
- Torch-TensorRT is not supported in the TensorRT containers.
Use the PyTorch Container to accelerate through Torch-TRT instead.
NVIDIA TensorRT Container Versions
The following table shows what versions of Ubuntu, CUDA, and TensorRT are supported in each of the NVIDIA containers for TensorRT. For older container versions, refer to the Frameworks Support Matrix.
Known Issues
None.
19. TensorRT Release 23.04
The NVIDIA container image for TensorRT, release 23.04, is available on NGC.
Contents of the TensorRT container
- The TensorRT C++ samples and C++ API documentation.
- Build the samples can be by running make in the /workspace/tensorrt/samples directory.
- The resulting executables are in the /workspace/tensorrt/bin directory.
- The C++ API documentation is in the /workspace/tensorrt/doc/cpp directory.
- The TensorRT Python samples and Python API documentation.
- The Python samples are in the
/workspace/tensorrt/samples/python directory.
Refer to the respective README documents for more samples.
- Many Python samples can be run by using python
<script.py> -d /workspace/tensorrt/data.
For example:
python onnx_resnet50.py -d /workspace/tensorrt/data
- The Python API documentation is in the /workspace/tensorrt/doc/python directory.
- The Python samples are in the
/workspace/tensorrt/samples/python directory.
- TensorRT 8.6.1.
The ONNX parser and plug-in libraries that are bundled with this container are built from TensorRT Open Source Software.
- Ubuntu 20.04Note: Container image 23.04-py3 contains Python 3.8.
- NVIDIA CUDA® 12.1.0
- NVIDIA cuBLAS 12.1.3
- NVIDIA cuDNN 8.9.0
- NVIDIA NCCL 2.17.1Note: Although NCCL is packaged in the container, it does not affect TensorRT nor inferencing in any way.
- rdma-core 36.0
- OpenMPI 4.1.4+
- OpenUCX 1.14.0
- GDRCopy 2.3
- NVIDIA HPC-X 2.13
- Nsight Compute 2023.1.0.15
- Nsight Systems 2023.1.1.127
Driver Requirements
Release 23.04 is based on CUDA 12.1.0, which requires NVIDIA Driver release 530 or later. However, if you are running on a data center GPU (for example, T4 or any other data center GPU), you can use NVIDIA driver release 450.51 (or later R450), 470.57 (or later R470), 510.47 (or later R510), 515.65 (or later R515), 525.85 (or later R525), or 530.30 (or later R530).
The CUDA driver's compatibility package only supports particular drivers. Thus, users should upgrade from all R418, R440, R460, and R520 drivers, which are not forward-compatible with CUDA 12.1. For a complete list of supported drivers, see the CUDA Application Compatibility topic. For more information, see CUDA Compatibility and Upgrades.GPU Requirements
Release 23.04 supports CUDA compute capability 3.5 and later. This corresponds to GPUs in the NVIDIA Kepler, Maxwell, NVIDIA Pascal, NVIDIA Volta™, NVIDIA Turing™, NVIDIA Ampere architecture, NVIDIA Hopper™, and NVIDIA Ada Lovelace architecture families. For a list of GPUs to which this compute capability corresponds, see CUDA GPUs. For additional support details, see the Deep Learning Frameworks Support Matrix.
Key Features and Enhancements
- TensorRT container image version 23.04 is based on TensorRT 8.6.1.
For a list of the features and enhancements that were introduced in TensorRT 8.6.1, refer to the TensorRT 8.6 release notes.
- Ubuntu 20.04 with March 2023 updates.
Announcements
- Starting with the 22.05 release, the TensorRT container is available for the Arm
SBSA platform.
For example, when you pull the nvcr.io/nvidia/tensorrt:22.05-py3 Docker image on an Arm SBSA machine, the Arm-specific image is automatically fetched.
- NVIDIA Deep Learning Profiler (DLProf) v1.8, which was included in the 21.12
container, was the last release of DLProf.
Starting with the 22.01 container, DLProf is longer included. It can still be manually installed by using a pip wheel on the nvidia-pyindex.
Obtaining Missing Data Files
Some samples require data files that are not included in the TensorRT container because of licensing restrictions, or because they are too large. Samples that do not include the required data files include a README.md file in the corresponding source directory that provides information about how to obtain the necessary data files.
Installing Required Python Modules
- To complete some of the samples, you might want to first run the Python setup script.
- If you need to install the missing Python modules and their dependencies, run the /opt/tensorrt/python/python_setup.sh script.
Installing Open Source Components
A script has been added to clone, build, and replace the provided plug-in, the Caffe parser, and the ONNX parser libraries with the open source ones that are based on the 22.05 tag on the official TensorRT open source repository.
To install the open source components in the container, run the following script:
/opt/tensorrt/install_opensource.sh -b main
For more information, see GitHub: TensorRT.
Limitations
- Accelerating Inference In TensorFlow with
TensorRT (TF-TRT) is not supported in the TensorRT containers.
Use the TensorFlow Container to accelerate through TF-TRT instead.
- Torch-TensorRT is not supported in the TensorRT containers.
Use the PyTorch Container to accelerate through Torch-TRT instead.
NVIDIA TensorRT Container Versions
The following table shows what versions of Ubuntu, CUDA, and TensorRT are supported in each of the NVIDIA containers for TensorRT. For older container versions, refer to the Frameworks Support Matrix.
Known Issues
None.
20. TensorRT Release 23.03
The NVIDIA container image for TensorRT, release 23.03, is available on NGC.
Contents of the TensorRT container
- The TensorRT C++ samples and C++ API documentation.
- Build the samples can be by running make in the /workspace/tensorrt/samples directory.
- The resulting executables are in the /workspace/tensorrt/bin directory.
- The C++ API documentation is in the /workspace/tensorrt/doc/cpp directory.
- The TensorRT Python samples and Python API documentation.
- The Python samples are in the
/workspace/tensorrt/samples/python directory.
Refer to the respective README documents for more samples.
- Many Python samples can be run by using python
<script.py> -d /workspace/tensorrt/data.
For example:
python onnx_resnet50.py -d /workspace/tensorrt/data
- The Python API documentation is in the /workspace/tensorrt/doc/python directory.
- The Python samples are in the
/workspace/tensorrt/samples/python directory.
- TensorRT 8.5.3.
The ONNX parser and plug-in libraries that are bundled with this container are built from TensorRT Open Source Software.
- Ubuntu 20.04Note: Container image 23.03-py3 contains Python 3.8.
- NVIDIA CUDA® 12.1.0
- NVIDIA cuBLAS from CUDA 12.1.0
- NVIDIA cuDNN 8.8.1.3
- NVIDIA NCCL 2.17.1Note: Although NCCL is packaged in the container, it does not affect TensorRT nor inferencing in any way.
- rdma-core 36.0
- OpenMPI 4.1.4+
- OpenUCX 1.14.0
- GDRCopy 2.3
- NVIDIA HPC-X 2.13
- Nsight Compute 2023.1.0.15
- Nsight Systems 2023.1.1.127
Driver Requirements
Release 23.03 is based on CUDA 12.1.0, which requires NVIDIA Driver release 530 or later. However, if you are running on a data center GPU (for example, T4 or any other data center GPU), you can use NVIDIA driver release 450.51 (or later R450), 470.57 (or later R470), 510.47 (or later R510), 515.65 (or later R515), 525.85 (or later R525), or 530.30 (or later R530).
The CUDA driver's compatibility package only supports particular drivers. Thus, users should upgrade from all R418, R440, R460, and R520 drivers, which are not forward-compatible with CUDA 12.1. For a complete list of supported drivers, see the CUDA Application Compatibility topic. For more information, see CUDA Compatibility and Upgrades.GPU Requirements
Release 23.03 supports CUDA compute capability 3.5 and later. This corresponds to GPUs in the NVIDIA Kepler, Maxwell, NVIDIA Pascal, NVIDIA Volta™, NVIDIA Turing™, NVIDIA Ampere architecture, and NVIDIA Hopper™ architecture families. For a list of GPUs to which this compute capability corresponds, see CUDA GPUs. For additional support details, see the Deep Learning Frameworks Support Matrix.
Key Features and Enhancements
- TensorRT container image version 23.03 is based on TensorRT 8.5.3.
For a list of the features and enhancements that were introduced in TensorRT 8.5.3, refer to the TensorRT 8.5 release notes.
- Ubuntu 20.04 with February 2023 updates.
Announcements
- Starting with the 22.05 release, the TensorRT container is available for the Arm
SBSA platform.
For example, when you pull the nvcr.io/nvidia/tensorrt:22.05-py3 Docker image on an Arm SBSA machine, the Arm-specific image is automatically fetched.
- NVIDIA Deep Learning Profiler (DLProf) v1.8, which was included in the 21.12
container, was the last release of DLProf.
Starting with the 22.01 container, DLProf is longer included. It can still be manually installed by using a pip wheel on the nvidia-pyindex.
Obtaining Missing Data Files
Some samples require data files that are not included in the TensorRT container because of licensing restrictions, or because they are too large. Samples that do not include the required data files include a README.md file in the corresponding source directory that provides information about how to obtain the necessary data files.
Installing Required Python Modules
- To complete some of the samples, you might want to first run the Python setup script.
- If you need to install the missing Python modules and their dependencies, run the /opt/tensorrt/python/python_setup.sh script.
Installing Open Source Components
A script has been added to clone, build, and replace the provided plug-in, the Caffe parser, and the ONNX parser libraries with the open source ones that are based on the 22.05 tag on the official TensorRT open source repository.
To install the open source components in the container, run the following script:
/opt/tensorrt/install_opensource.sh -b main
For more information, see GitHub: TensorRT.
Limitations
- Accelerating Inference In TensorFlow with
TensorRT (TF-TRT) is not supported in the TensorRT containers.
Use the TensorFlow Container to accelerate through TF-TRT instead.
- Torch-TensorRT is not supported in the TensorRT containers.
Use the PyTorch Container to accelerate through Torch-TRT instead.
NVIDIA TensorRT Container Versions
The following table shows what versions of Ubuntu, CUDA, and TensorRT are supported in each of the NVIDIA containers for TensorRT. For older container versions, refer to the Frameworks Support Matrix.
Known Issues
None.
21. TensorRT Release 23.02
The NVIDIA container image for TensorRT, release 23.02, is available on NGC.
Contents of the TensorRT container
- The TensorRT C++ samples and C++ API documentation.
- Build the samples can be by running make in the /workspace/tensorrt/samples directory.
- The resulting executables are in the /workspace/tensorrt/bin directory.
- The C++ API documentation is in the /workspace/tensorrt/doc/cpp directory.
- The TensorRT Python samples and Python API documentation.
- The Python samples are in the
/workspace/tensorrt/samples/python directory.
Refer to the respective README documents for more samples.
- Many Python samples can be run by using python
<script.py> -d /workspace/tensorrt/data.
For example:
python onnx_resnet50.py -d /workspace/tensorrt/data
- The Python API documentation is in the /workspace/tensorrt/doc/python directory.
- The Python samples are in the
/workspace/tensorrt/samples/python directory.
- TensorRT 8.5.3.
The ONNX parser and plug-in libraries that are bundled with this container are built from TensorRT Open Source Software.
- Ubuntu 20.04Note: Container image 23.02-py3 contains Python 3.8.
- NVIDIA CUDA® 12.0.1
- NVIDIA cuBLAS from CUDA 12.0.1
- NVIDIA cuDNN 8.7.0
- NVIDIA NCCL 2.16.5Note: Although NCCL is packaged in the container, it does not affect TensorRT nor inferencing in any way.
- rdma-core 36.0
- OpenMPI 4.1.4+
- OpenUCX 1.14.0
- GDRCopy 2.3
- NVIDIA HPC-X 2.13
- Nsight Compute 2022.4.1.6
- Nsight Systems 2022.5.1.93
Driver Requirements
Release 23.02 is based on CUDA 12.0.1, which requires NVIDIA Driver release 525 or later. However, if you are running on a data center GPU (for example, T4 or any other data center GPU), you can use NVIDIA driver release 450.51 (or later R450), 470.57 (or later R470), 510.47 (or later R510), 515.65 (or later R515), or 525.85 (or later R525).
The CUDA driver's compatibility package only supports particular drivers. Thus, users should upgrade from all R418, R440, R460, and R520 drivers, which are not forward-compatible with CUDA 12.0. For a complete list of supported drivers, see the CUDA Application Compatibility topic. For more information, see CUDA Compatibility and Upgrades.GPU Requirements
Release 23.02 supports CUDA compute capability 3.5 and later. This corresponds to GPUs in the NVIDIA Kepler, Maxwell, NVIDIA Pascal, NVIDIA Volta™, NVIDIA Turing™, NVIDIA Ampere architecture, and NVIDIA Hopper™ architecture families. For a list of GPUs to which this compute capability corresponds, see CUDA GPUs. For additional support details, see the Deep Learning Frameworks Support Matrix.
Key Features and Enhancements
- TensorRT container image version 23.02 is based on TensorRT 8.5.3.
For a list of the features and enhancements that were introduced in TensorRT 8.5.3, refer to the TensorRT 8.5 release notes.
- Ubuntu 20.04 with January 2023 updates.
Announcements
- Starting with the 22.05 release, the TensorRT container is available for the Arm
SBSA platform.
For example, when you pull the nvcr.io/nvidia/tensorrt:22.05-py3 Docker image on an Arm SBSA machine, the Arm-specific image is automatically fetched.
- NVIDIA Deep Learning Profiler (DLProf) v1.8, which was included in the 21.12
container, was the last release of DLProf.
Starting with the 22.01 container, DLProf is longer included. It can still be manually installed by using a pip wheel on the nvidia-pyindex.
Obtaining Missing Data Files
Some samples require data files that are not included in the TensorRT container because of licensing restrictions, or because they are too large. Samples that do not include the required data files include a README.md file in the corresponding source directory that provides information about how to obtain the necessary data files.
Installing Required Python Modules
- To complete some of the samples, you might want to first run the Python setup script.
- If you need to install the missing Python modules and their dependencies, run the /opt/tensorrt/python/python_setup.sh script.
Installing Open Source Components
A script has been added to clone, build, and replace the provided plug-in, the Caffe parser, and the ONNX parser libraries with the open source ones that are based on the 22.05 tag on the official TensorRT open source repository.
To install the open source components in the container, run the following script:
For more information, see GitHub: TensorRT.
Limitations
- Accelerating Inference In TensorFlow with
TensorRT (TF-TRT) is not supported in the TensorRT containers.
Use the TensorFlow Container to accelerate through TF-TRT instead.
- Torch-TensorRT is not supported in the TensorRT containers.
Use the PyTorch Container to accelerate through Torch-TRT instead.
NVIDIA TensorRT Container Versions
The following table shows what versions of Ubuntu, CUDA, and TensorRT are supported in each of the NVIDIA containers for TensorRT. For older container versions, refer to the Frameworks Support Matrix.
Known Issues
None.
22. TensorRT Release 23.01
The NVIDIA container image for TensorRT, release 23.01, is available on NGC.
Contents of the TensorRT container
- The TensorRT C++ samples and C++ API documentation.
- Build the samples can be by running make in the /workspace/tensorrt/samples directory.
- The resulting executables are in the /workspace/tensorrt/bin directory.
- The C++ API documentation is in the /workspace/tensorrt/doc/cpp directory.
- The TensorRT Python samples and Python API documentation.
- The Python samples are in the
/workspace/tensorrt/samples/python directory.
Refer to the respective README documents for more samples.
- Many Python samples can be run by using python
<script.py> -d /workspace/tensorrt/data.
For example:
python onnx_resnet50.py -d /workspace/tensorrt/data
- The Python API documentation is in the /workspace/tensorrt/doc/python directory.
- The Python samples are in the
/workspace/tensorrt/samples/python directory.
- TensorRT 8.5.2.2.
The ONNX parser and plug-in libraries that are bundled with this container are built from TensorRT Open Source Software.
- Ubuntu 20.04Note: Container image 23.01-py3 contains Python 3.8.
- NVIDIA CUDA® 12.0.1
- NVIDIA cuBLAS from CUDA 12.0.1
- NVIDIA cuDNN 8.7.0
- NVIDIA NCCL 2.16.5Note: Although NCCL is packaged in the container, it does not affect TensorRT nor inferencing in any way.
- rdma-core 36.0
- OpenMPI 4.1.4+
- OpenUCX 1.14.0
- GDRCopy 2.3
- NVIDIA HPC-X 2.13
- Nsight Compute 2022.4.1.6
- Nsight Systems 2022.5.1
Driver Requirements
Release 23.01 is based on CUDA 12.0.1, which requires NVIDIA Driver release 525 or later. However, if you are running on a data center GPU (for example, T4 or any other data center GPU), you can use NVIDIA driver release 450.51 (or later R450), 470.57 (or later R470), 510.47 (or later R510), 515.65 (or later R515), or 525.85 (or later R525).
The CUDA driver's compatibility package only supports particular drivers. Thus, users should upgrade from all R418, R440, and R460 drivers, which are not forward-compatible with CUDA 12.0. For a complete list of supported drivers, see the CUDA Application Compatibility topic. For more information, see CUDA Compatibility and Upgrades.GPU Requirements
Release 23.01 supports CUDA compute capability 3.5 and later. This corresponds to GPUs in the NVIDIA Kepler, Maxwell, NVIDIA Pascal, NVIDIA Volta™, NVIDIA Turing™, NVIDIA Ampere architecture, and NVIDIA Hopper™ architecture families. For a list of GPUs to which this compute capability corresponds, see CUDA GPUs. For additional support details, see the Deep Learning Frameworks Support Matrix.