Container Release Notes

TensorRT Container Release Notes

The TensorRT container is an easy to use container for TensorRT development. The container allows you to build, modify, and execute TensorRT samples. These release notes provide a list of key features, packaged software in the container, software enhancements and improvements, and known issues. The TensorRT container is released monthly to provide you with the latest NVIDIA deep learning software libraries and GitHub code contributions that have been sent upstream. The libraries and contributions have all been tested, tuned, and optimized.

For a complete view of the supported software and specific versions that are packaged with the frameworks based on the container image, see the Frameworks Support Matrix.

The core of NVIDIA® TensorRT™ is a C++ library that facilitates high-performance inference on NVIDIA graphics processing units (GPUs). TensorRT takes a trained network consisting of a network definition and a set of trained parameters and produces a highly optimized runtime engine that performs inference for that network.

TensorRT provides APIs via C++ and Python that help to express deep learning models via the Network Definition API or load a pre-defined model via the ONNX parser that allows TensorRT to optimize and run them on an NVIDIA GPU. TensorRT applies graph optimizations layer fusions, among other optimizations, while also finding the fastest implementation of that model leveraging a diverse collection of highly optimized kernels. TensorRT also supplies a runtime that you can use to execute this network on all of NVIDIA’s GPUs from the NVIDIA Turing™ generation onwards.

TensorRT includes optional high-speed mixed-precision capabilities with the NVIDIA Turing™, NVIDIA Ampere, NVIDIA Ada Lovelace, and NVIDIA Hopper™ architectures.

The TensorRT container allows TensorRT samples to be built, modified, and executed. For more information about the TensorRT samples, see the TensorRT Sample Support Guide.

For a complete list of installation options and instructions, refer to Installing TensorRT.

About this task

Before you can pull a container from the NGC container registry:

• Install Docker.
  • For NVIDIA DGX™ users, see Preparing to use NVIDIA Containers Getting Started Guide.
  • For non-DGX users, see NVIDIA^® GPU Cloud™ (NGC) container registry installation documentation based on your platform.
• Ensure that you have access and can log in to the NGC container registry.

  Refer to NGC Getting Started Guide for more information.

The deep learning frameworks, the NGC Docker containers, and the deep learning framework containers are stored in the nvcr.io/nvidia repository.

Before you begin

Before you can run an NGC deep learning framework container, your Docker environment must support NVIDIA GPUs. To run a container, issue the appropriate command as explained in Running A Container and specify the registry, repository, and tags.

About this task

On a system with GPU support for NGC containers, when you run a container, the following occurs:

• The Docker engine loads the image into a container which runs the software.
• You define the runtime resources of the container by including the additional flags and settings that are used with the command.

  These flags and settings are described in Running A Container.

• The GPUs are explicitly defined for the Docker container, which defaults to all GPUs, but can be specified by using the NVIDIA_VISIBLE_DEVICES environment variable.

  For more information, refer to the nvidia-docker documentation.

  Note:

  Starting in Docker 19.03, complete the steps below.

The method implemented in your system depends on the DGX OS version that you installed (for DGX systems), the NGC Cloud Image that was provided by a Cloud Service Provider, or the software that you installed to prepare to run NGC containers on TITAN PCs, Quadro PCs, or NVIDIA Virtual GPUs (vGPUs).

Procedure

1. Issue the command for the applicable release of the container that you want.

   The following command assumes that you want to pull the latest container.

   Copy

   Copied!

               
   
               
   docker pull nvcr.io/nvidia/tensorrt:25.01-py3
           

2. Open a command prompt and paste the pull command.

   Ensure that the pull process successfully completes before you proceed to step 3.

3. Run the container image.
   • Copy

     Copied!

                 
     
                 
     docker run --gpus all -it --rm nvcr.io/nvidia/tensorrt:<xx.xx>-py<x>
             

     
     

4. To extend the TensorRT container, select one of the following options:
   • Add to or modify the source code in this container and run your customized version.
   • To add additional packages, use docker build to add your customizations on top of this container.
     Note:

     NVIDIA recommends using the docker build option for ease of migration to later versions of the TensorRT container.

     
     

The NVIDIA container image for TensorRT, release 25.02, is available on NGC.

Contents of the TensorRT container

This container includes the following:

• The TensorRT C++ samples and C++ API documentation.
  • Build the samples can be by running make in the /workspace/tensorrt/samples directory.
  • The resulting executables are in the /workspace/tensorrt/bin directory.
  • The C++ API documentation is in the /workspace/tensorrt/doc/cpp directory.
• The TensorRT Python samples and Python API documentation.
  • The Python samples are in the /workspace/tensorrt/samples/python directory.

    Refer to the respective README documents for more samples.

  • Many Python samples can be run by using python <script.py> -d /workspace/tensorrt/data. For example:
    Copy

    Copied!

                
    
                
    python onnx_resnet50.py -d /workspace/tensorrt/data
            

  • The Python API documentation is in the /workspace/tensorrt/doc/python directory.
• TensorRT 10.8.0.43.

  The ONNX parser and plug-in libraries that are bundled with this container are built from TensorRT Open Source Software.

The container also includes the following:

• Ubuntu 24.04
  Note:

  Container image 25.01-py3 contains Python 3.12.

• NVIDIA CUDA 12.8.0.38
• NVIDIA cuBLAS 12.8.3.14
• NVIDIA cuDNN 9.7.1.26
• NVIDIA NCCL 2.25.1
  Note:

  Although NCCL is packaged in the container, it does not affect TensorRT nor inferencing in any way.

• rdma-core 39.0
• OpenMPI 4.1.7
• OpenUCX 1.18.0
• GDRCopy 2.4.1
• NVIDIA HPC-X 2.21
• Nsight Compute 2025.1.0.14
• Nsight Systems 2025.1.1.65

Driver Requirements

Release 25.02 is based on CUDA 12.8.0.38 which requires NVIDIA Driver release 570 or later. However, if you are running on a data center GPU (for example, B100, L40, or any other data center GPU), you can use NVIDIA driver release 470.57 (or later R470), 525.85 (or later R525), 535.86 (or later R535), or 550.54 (or later R550) in forward-compatibility mode.

The CUDA driver's compatibility package only supports particular drivers. Thus, users should upgrade from all R418, R440, R450, R460, R510, R520, R530, R545 and R555 and R560 drivers, which are not forward-compatible with CUDA 12.8. For a complete list of supported drivers, see the CUDA Application Compatibility topic. For more information, see CUDA Compatibility and Upgrades.

GPU Requirements

Release 25.01 supports CUDA compute capability 7.5 and later. This corresponds to GPUs in the NVIDIA Turing™, NVIDIA Ampere architecture, NVIDIA Hopper™, and NVIDIA Ada Lovelace architecture families. For a list of GPUs to which this compute capability corresponds, see CUDA GPUs. For additional support details, see the Deep Learning Frameworks Support Matrix.

Key Features and Enhancements

This TensorRT container release includes the following key features and enhancements.

• TensorRT container image version 25.02 is based on TensorRT 10.8.

  For a list of the features and enhancements that were introduced in this version of TensorRT, refer to the TensorRT release notes.

• All dependencies on cuDNN have been removed from the TensorRT starting with the 8.6.3 release to reduce the overall container size. Any TensorRT features which depend on cuDNN, which are primarily some plugins and samples, will not work with this release.

Announcements

• Starting with the 23.11 release, TensorRT containers supporting iGPU architectures are published, and run on Jetson devices. Please refer to the Frameworks Support Matrix for information regarding which iGPU hardware/software is supported by which container.
• NVIDIA Deep Learning Profiler (DLProf) v1.8, which was included in the 21.12 container, was the last release of DLProf.

  Starting with the 22.01 container, DLProf is no longer included. It can still be manually installed by using a pip wheel on the nvidia-pyindex.

Obtaining Missing Data Files

Some samples require data files that are not included in the TensorRT container because of licensing restrictions, or because they are too large. Samples that do not include the required data files include a README.md file in the corresponding source directory that provides information about how to obtain the necessary data files.

Installing Required Python Modules

• To complete some of the samples, you might want to first run the Python setup script.
• If you need to install the missing Python modules and their dependencies, run the /opt/tensorrt/python/python_setup.sh script.

Installing Open Source Components

A script has been added to clone, build, and replace the provided plug-in, and the ONNX parser libraries with the open source ones that are based on the 24.10 tag on the official TensorRT open source repository.

To install the open source components in the container, run the following script:

/opt/tensorrt/install_opensource.sh -b main

For more information, see GitHub: TensorRT.

Limitations

• Accelerating Inference In TensorFlow with TensorRT (TF-TRT) is not supported in the TensorRT containers.

  Use the TensorFlow Container to accelerate through TF-TRT instead.

• Torch-TensorRT is not supported in the TensorRT containers.

  Use the PyTorch Container to accelerate through Torch-TRT instead.

NVIDIA TensorRT Container Versions

The following table shows what versions of Ubuntu, CUDA, and TensorRT are supported in each of the NVIDIA containers for TensorRT. For older container versions, refer to the Frameworks Support Matrix.

Known Issues

• None.

The NVIDIA container image for TensorRT, release 25.01, is available on NGC.

Contents of the TensorRT container

This container includes the following:

• The TensorRT C++ samples and C++ API documentation.
  • Build the samples can be by running make in the /workspace/tensorrt/samples directory.
  • The resulting executables are in the /workspace/tensorrt/bin directory.
  • The C++ API documentation is in the /workspace/tensorrt/doc/cpp directory.
• The TensorRT Python samples and Python API documentation.
  • The Python samples are in the /workspace/tensorrt/samples/python directory.

    Refer to the respective README documents for more samples.

  • Many Python samples can be run by using python <script.py> -d /workspace/tensorrt/data. For example:
    Copy

    Copied!

                
    
                
    python onnx_resnet50.py -d /workspace/tensorrt/data
            

  • The Python API documentation is in the /workspace/tensorrt/doc/python directory.
• TensorRT 10.8.0.43.

  The ONNX parser and plug-in libraries that are bundled with this container are built from TensorRT Open Source Software.

The container also includes the following:

• Ubuntu 24.04
  Note:

  Container image 25.01-py3 contains Python 3.12.

• NVIDIA CUDA 12.8.0.038
• NVIDIA cuBLAS 12.8.3.14
• NVIDIA cuDNN 9.7.0.66
• NVIDIA NCCL 2.25.1
  Note:

  Although NCCL is packaged in the container, it does not affect TensorRT nor inferencing in any way.

• rdma-core 39.0
• OpenMPI 4.1.7
• OpenUCX 1.18.0
• GDRCopy 2.4.1
• NVIDIA HPC-X 2.21
• Nsight Compute 2025.1.0.14
• Nsight Systems 2024.6.2.225

Driver Requirements

Release 25.01 is based on CUDA 12.8.0 which requires NVIDIA Driver release 570 or later. However, if you are running on a data center GPU (for example, B100, L40, or any other data center GPU), you can use NVIDIA driver release 470.57 (or later R470), 525.85 (or later R525), 535.86 (or later R535), or 550.54 (or later R550) in forward-compatibility mode.

The CUDA driver's compatibility package only supports particular drivers. Thus, users should upgrade from all R418, R440, R450, R460, R510, R520, R530, R545 and R555 and R560 drivers, which are not forward-compatible with CUDA 12.8. For a complete list of supported drivers, see the CUDA Application Compatibility topic. For more information, see CUDA Compatibility and Upgrades.

GPU Requirements

Release 25.01 supports CUDA compute capability 7.5 and later. This corresponds to GPUs in the NVIDIA Turing™, NVIDIA Ampere architecture, NVIDIA Hopper™, and NVIDIA Ada Lovelace architecture families. For a list of GPUs to which this compute capability corresponds, see CUDA GPUs. For additional support details, see the Deep Learning Frameworks Support Matrix.

Key Features and Enhancements

This TensorRT container release includes the following key features and enhancements.

• TensorRT container image version 25.01 is based on TensorRT 10.7.

  For a list of the features and enhancements that were introduced in this version of TensorRT, refer to the TensorRT release notes.

• All dependencies on cuDNN have been removed from the TensorRT starting with the 8.6.3 release to reduce the overall container size. Any TensorRT features which depend on cuDNN, which are primarily some plugins and samples, will not work with this release.

Announcements

• Starting with the 23.11 release, TensorRT containers supporting iGPU architectures are published, and run on Jetson devices. Please refer to the Frameworks Support Matrix for information regarding which iGPU hardware/software is supported by which container.
• NVIDIA Deep Learning Profiler (DLProf) v1.8, which was included in the 21.12 container, was the last release of DLProf.

  Starting with the 22.01 container, DLProf is no longer included. It can still be manually installed by using a pip wheel on the nvidia-pyindex.

Obtaining Missing Data Files

Some samples require data files that are not included in the TensorRT container because of licensing restrictions, or because they are too large. Samples that do not include the required data files include a README.md file in the corresponding source directory that provides information about how to obtain the necessary data files.

Installing Required Python Modules

• To complete some of the samples, you might want to first run the Python setup script.
• If you need to install the missing Python modules and their dependencies, run the /opt/tensorrt/python/python_setup.sh script.

Installing Open Source Components

A script has been added to clone, build, and replace the provided plug-in, and the ONNX parser libraries with the open source ones that are based on the 24.10 tag on the official TensorRT open source repository.

To install the open source components in the container, run the following script:

/opt/tensorrt/install_opensource.sh -b main

For more information, see GitHub: TensorRT.

Limitations

• Accelerating Inference In TensorFlow with TensorRT (TF-TRT) is not supported in the TensorRT containers.

  Use the TensorFlow Container to accelerate through TF-TRT instead.

• Torch-TensorRT is not supported in the TensorRT containers.

  Use the PyTorch Container to accelerate through Torch-TRT instead.

NVIDIA TensorRT Container Versions

The following table shows what versions of Ubuntu, CUDA, and TensorRT are supported in each of the NVIDIA containers for TensorRT. For older container versions, refer to the Frameworks Support Matrix.

Known Issues

• None.

The NVIDIA container image for TensorRT, release 24.12, is available on NGC.

Contents of the TensorRT container

This container includes the following:

• The TensorRT C++ samples and C++ API documentation.
  • Build the samples can be by running make in the /workspace/tensorrt/samples directory.
  • The resulting executables are in the /workspace/tensorrt/bin directory.
  • The C++ API documentation is in the /workspace/tensorrt/doc/cpp directory.
• The TensorRT Python samples and Python API documentation.
  • The Python samples are in the /workspace/tensorrt/samples/python directory.

    Refer to the respective README documents for more samples.

  • Many Python samples can be run by using python <script.py> -d /workspace/tensorrt/data. For example:
    Copy

    Copied!

                
    
                
    python onnx_resnet50.py -d /workspace/tensorrt/data
            

  • The Python API documentation is in the /workspace/tensorrt/doc/python directory.
• TensorRT 10.7.

  The ONNX parser and plug-in libraries that are bundled with this container are built from TensorRT Open Source Software.

The container also includes the following:

• Ubuntu 24.04
  Note:

  Container image 24.12-py3 contains Python 3.12.

• NVIDIA CUDA 12.6.3
• NVIDIA cuBLAS 12.6.4.1
• NVIDIA cuDNN 9.6.0.74
• NVIDIA NCCL 2.3
  Note:

  Although NCCL is packaged in the container, it does not affect TensorRT nor inferencing in any way.

• rdma-core 39.0
• OpenMPI 4.1.7
• OpenUCX 1.18.0
• GDRCopy 2.4.1
• NVIDIA HPC-X 2.21
• Nsight Compute 2024.3.2.3
• Nsight Systems 2024.7.1.84

Driver Requirements

Release 24.12 is based on CUDA 12.6.3 which requires NVIDIA Driver release 560 or later. However, if you are running on a data center GPU (for example, T4 or any other data center GPU), you can use NVIDIA driver release 470.57 (or later R470), 525.85 (or later R525), 535.86 (or later R535), or 545.23 (or later R545).

The CUDA driver's compatibility package only supports particular drivers. Thus, users should upgrade from all R418, R440, R450, R460, R510, R520, R530, R545 and R555 drivers, which are not forward-compatible with CUDA 12.6. For a complete list of supported drivers, see the CUDA Application Compatibility topic. For more information, see CUDA Compatibility and Upgrades.

GPU Requirements

Release 24.12 supports CUDA compute capability 7.5 and later. This corresponds to GPUs in the NVIDIA Turing™, NVIDIA Ampere architecture, NVIDIA Hopper™, and NVIDIA Ada Lovelace architecture families. For a list of GPUs to which this compute capability corresponds, see CUDA GPUs. For additional support details, see the Deep Learning Frameworks Support Matrix.

Key Features and Enhancements

This TensorRT container release includes the following key features and enhancements.

• TensorRT container image version 24.12 is based on TensorRT 10.7.

  For a list of the features and enhancements that were introduced in this version of TensorRT, refer to the TensorRT release notes.

• All dependencies on cuDNN have been removed from the TensorRT starting with the 8.6.3 release to reduce the overall container size. Any TensorRT features which depend on cuDNN, which are primarily some plugins and samples, will not work with this release.

Announcements

• Starting with the 23.11 release, TensorRT containers supporting iGPU architectures are published, and run on Jetson devices. Please refer to the Frameworks Support Matrix for information regarding which iGPU hardware/software is supported by which container.
• NVIDIA Deep Learning Profiler (DLProf) v1.8, which was included in the 21.12 container, was the last release of DLProf.

  Starting with the 22.01 container, DLProf is no longer included. It can still be manually installed by using a pip wheel on the nvidia-pyindex.

Obtaining Missing Data Files

Some samples require data files that are not included in the TensorRT container because of licensing restrictions, or because they are too large. Samples that do not include the required data files include a README.md file in the corresponding source directory that provides information about how to obtain the necessary data files.

Installing Required Python Modules

• To complete some of the samples, you might want to first run the Python setup script.
• If you need to install the missing Python modules and their dependencies, run the /opt/tensorrt/python/python_setup.sh script.

Installing Open Source Components

A script has been added to clone, build, and replace the provided plug-in, and the ONNX parser libraries with the open source ones that are based on the 24.10 tag on the official TensorRT open source repository.

To install the open source components in the container, run the following script:

/opt/tensorrt/install_opensource.sh -b main

For more information, see GitHub: TensorRT.

Limitations

• Accelerating Inference In TensorFlow with TensorRT (TF-TRT) is not supported in the TensorRT containers.

  Use the TensorFlow Container to accelerate through TF-TRT instead.

• Torch-TensorRT is not supported in the TensorRT containers.

  Use the PyTorch Container to accelerate through Torch-TRT instead.

NVIDIA TensorRT Container Versions

The following table shows what versions of Ubuntu, CUDA, and TensorRT are supported in each of the NVIDIA containers for TensorRT. For older container versions, refer to the Frameworks Support Matrix.

Known Issues

• None.

The NVIDIA container image for TensorRT, release 24.11, is available on NGC.

Contents of the TensorRT container

This container includes the following:

• The TensorRT C++ samples and C++ API documentation.
  • Build the samples can be by running make in the /workspace/tensorrt/samples directory.
  • The resulting executables are in the /workspace/tensorrt/bin directory.
  • The C++ API documentation is in the /workspace/tensorrt/doc/cpp directory.
• The TensorRT Python samples and Python API documentation.
  • The Python samples are in the /workspace/tensorrt/samples/python directory.

    Refer to the respective README documents for more samples.

  • Many Python samples can be run by using python <script.py> -d /workspace/tensorrt/data. For example:
    Copy

    Copied!

                
    
                
    python onnx_resnet50.py -d /workspace/tensorrt/data
            

  • The Python API documentation is in the /workspace/tensorrt/doc/python directory.
• TensorRT 10.6.0.26.

  The ONNX parser and plug-in libraries that are bundled with this container are built from TensorRT Open Source Software.

The container also includes the following:

• Ubuntu 24.04
  Note:

  Container image 24.11-py3 contains Python 3.12.

• NVIDIA CUDA 12.6.3
• NVIDIA cuBLAS 12.6.4.1
• NVIDIA cuDNN 9.5.1.17
• NVIDIA NCCL 2.23.4
  Note:

  Although NCCL is packaged in the container, it does not affect TensorRT nor inferencing in any way.

• rdma-core 39.0
• OpenMPI 4.1.7
• OpenUCX 1.18.0
• GDRCopy 2.4.1
• NVIDIA HPC-X 2.21
• Nsight Compute 2024.3.2.3
• Nsight Systems 2024.6.1.90

Driver Requirements

Release 24.11 is based on CUDA 12.6.3 which requires NVIDIA Driver release 560 or later. However, if you are running on a data center GPU (for example, T4 or any other data center GPU), you can use NVIDIA driver release 470.57 (or later R470), 525.85 (or later R525), 535.86 (or later R535), or 545.23 (or later R545).

The CUDA driver's compatibility package only supports particular drivers. Thus, users should upgrade from all R418, R440, R450, R460, R510, R520, R530, R545 and R555 drivers, which are not forward-compatible with CUDA 12.6. For a complete list of supported drivers, see the CUDA Application Compatibility topic. For more information, see CUDA Compatibility and Upgrades.

GPU Requirements

Release 24.11 supports CUDA compute capability 7.5 and later. This corresponds to GPUs in the NVIDIA Turing™, NVIDIA Ampere architecture, NVIDIA Hopper™, and NVIDIA Ada Lovelace architecture families. For a list of GPUs to which this compute capability corresponds, see CUDA GPUs. For additional support details, see the Deep Learning Frameworks Support Matrix.

Key Features and Enhancements

This TensorRT container release includes the following key features and enhancements.

• TensorRT container image version 24.11 is based on TensorRT 10.6.0.26.

  For a list of the features and enhancements that were introduced in this version of TensorRT, refer to the TensorRT release notes.

• All dependencies on cuDNN have been removed from the TensorRT starting with the 8.6.3 release to reduce the overall container size. Any TensorRT features which depend on cuDNN, which are primarily some plugins and samples, will not work with this release.

Announcements

• Starting with the 23.11 release, TensorRT containers supporting iGPU architectures are published, and run on Jetson devices. Please refer to the Frameworks Support Matrix for information regarding which iGPU hardware/software is supported by which container.
• NVIDIA Deep Learning Profiler (DLProf) v1.8, which was included in the 21.12 container, was the last release of DLProf.

  Starting with the 22.01 container, DLProf is no longer included. It can still be manually installed by using a pip wheel on the nvidia-pyindex.

Obtaining Missing Data Files

Some samples require data files that are not included in the TensorRT container because of licensing restrictions, or because they are too large. Samples that do not include the required data files include a README.md file in the corresponding source directory that provides information about how to obtain the necessary data files.

Installing Required Python Modules

• To complete some of the samples, you might want to first run the Python setup script.
• If you need to install the missing Python modules and their dependencies, run the /opt/tensorrt/python/python_setup.sh script.

Installing Open Source Components

A script has been added to clone, build, and replace the provided plug-in, and the ONNX parser libraries with the open source ones that are based on the 24.10 tag on the official TensorRT open source repository.

To install the open source components in the container, run the following script:

/opt/tensorrt/install_opensource.sh -b main

For more information, see GitHub: TensorRT.

Limitations

• Accelerating Inference In TensorFlow with TensorRT (TF-TRT) is not supported in the TensorRT containers.

  Use the TensorFlow Container to accelerate through TF-TRT instead.

• Torch-TensorRT is not supported in the TensorRT containers.

  Use the PyTorch Container to accelerate through Torch-TRT instead.

NVIDIA TensorRT Container Versions

The following table shows what versions of Ubuntu, CUDA, and TensorRT are supported in each of the NVIDIA containers for TensorRT. For older container versions, refer to the Frameworks Support Matrix.

Known Issues

• None.

The NVIDIA container image for TensorRT, release 24.10, is available on NGC.

Contents of the TensorRT container

This container includes the following:

• The TensorRT C++ samples and C++ API documentation.
  • Build the samples can be by running make in the /workspace/tensorrt/samples directory.
  • The resulting executables are in the /workspace/tensorrt/bin directory.
  • The C++ API documentation is in the /workspace/tensorrt/doc/cpp directory.
• The TensorRT Python samples and Python API documentation.
  • The Python samples are in the /workspace/tensorrt/samples/python directory.

    Refer to the respective README documents for more samples.

  • Many Python samples can be run by using python <script.py> -d /workspace/tensorrt/data. For example:
    Copy

    Copied!

                
    
                
    python onnx_resnet50.py -d /workspace/tensorrt/data
            

  • The Python API documentation is in the /workspace/tensorrt/doc/python directory.
• TensorRT 10.5.0.18.

  The ONNX parser and plug-in libraries that are bundled with this container are built from TensorRT Open Source Software.

The container also includes the following:

• Ubuntu 22.04
  Note:

  Container image 24.10-py3 contains Python 3.10.

• NVIDIA CUDA 12.6.2
• NVIDIA cuBLAS 12.6.3.3
• NVIDIA cuDNN 9.5.0.50
• NVIDIA NCCL 2.22.3
  Note:

  Although NCCL is packaged in the container, it does not affect TensorRT nor inferencing in any way.

• rdma-core 39.0
• OpenMPI 4.1.7
• OpenUCX 1.15.0
• GDRCopy 2.3
• NVIDIA HPC-X 2.20
• Nsight Compute 2024.3.2.3
• Nsight Systems 2024.6.1.90

Driver Requirements

Release 24.10 is based on CUDA 12.6.2 which requires NVIDIA Driver release 560 or later. However, if you are running on a data center GPU (for example, T4 or any other data center GPU), you can use NVIDIA driver release 470.57 (or later R470), 525.85 (or later R525), 535.86 (or later R535), or 545.23 (or later R545).

The CUDA driver's compatibility package only supports particular drivers. Thus, users should upgrade from all R418, R440, R450, R460, R510, R520, R530, R545 and R555 drivers, which are not forward-compatible with CUDA 12.6. For a complete list of supported drivers, see the CUDA Application Compatibility topic. For more information, see CUDA Compatibility and Upgrades.

GPU Requirements

Release 24.10 supports CUDA compute capability 7.5 and later. This corresponds to GPUs in the NVIDIA Turing™, NVIDIA Ampere architecture, NVIDIA Hopper™, and NVIDIA Ada Lovelace architecture families. For a list of GPUs to which this compute capability corresponds, see CUDA GPUs. For additional support details, see the Deep Learning Frameworks Support Matrix.

Key Features and Enhancements

This TensorRT container release includes the following key features and enhancements.

• TensorRT container image version 24.10 is based on TensorRT 10.5.0.18.

  For a list of the features and enhancements that were introduced in this version of TensorRT, refer to the TensorRT release notes.

• All dependencies on cuDNN have been removed from the TensorRT starting with the 8.6.3 release to reduce the overall container size. Any TensorRT features which depend on cuDNN, which are primarily some plugins and samples, will not work with this release.

Announcements

• Starting with the 23.11 release, TensorRT containers supporting iGPU architectures are published, and run on Jetson devices. Please refer to the Frameworks Support Matrix for information regarding which iGPU hardware/software is supported by which container.
• NVIDIA Deep Learning Profiler (DLProf) v1.8, which was included in the 21.12 container, was the last release of DLProf.

  Starting with the 22.01 container, DLProf is no longer included. It can still be manually installed by using a pip wheel on the nvidia-pyindex.

Obtaining Missing Data Files

Some samples require data files that are not included in the TensorRT container because of licensing restrictions, or because they are too large. Samples that do not include the required data files include a README.md file in the corresponding source directory that provides information about how to obtain the necessary data files.

Installing Required Python Modules

• To complete some of the samples, you might want to first run the Python setup script.
• If you need to install the missing Python modules and their dependencies, run the /opt/tensorrt/python/python_setup.sh script.

Installing Open Source Components

A script has been added to clone, build, and replace the provided plug-in, and the ONNX parser libraries with the open source ones that are based on the 24.10 tag on the official TensorRT open source repository.

To install the open source components in the container, run the following script:

/opt/tensorrt/install_opensource.sh -b main

For more information, see GitHub: TensorRT.

Limitations

• Accelerating Inference In TensorFlow with TensorRT (TF-TRT) is not supported in the TensorRT containers.

  Use the TensorFlow Container to accelerate through TF-TRT instead.

• Torch-TensorRT is not supported in the TensorRT containers.

  Use the PyTorch Container to accelerate through Torch-TRT instead.

NVIDIA TensorRT Container Versions

The following table shows what versions of Ubuntu, CUDA, and TensorRT are supported in each of the NVIDIA containers for TensorRT. For older container versions, refer to the Frameworks Support Matrix.

Known Issues

• None.

The NVIDIA container image for TensorRT, release 24.09, is available on NGC.

Contents of the TensorRT container

This container includes the following:

• The TensorRT C++ samples and C++ API documentation.
  • Build the samples can be by running make in the /workspace/tensorrt/samples directory.
  • The resulting executables are in the /workspace/tensorrt/bin directory.
  • The C++ API documentation is in the /workspace/tensorrt/doc/cpp directory.
• The TensorRT Python samples and Python API documentation.
  • The Python samples are in the /workspace/tensorrt/samples/python directory.

    Refer to the respective README documents for more samples.

  • Many Python samples can be run by using python <script.py> -d /workspace/tensorrt/data. For example:
    Copy

    Copied!

                
    
                
    python onnx_resnet50.py -d /workspace/tensorrt/data
            

  • The Python API documentation is in the /workspace/tensorrt/doc/python directory.
• TensorRT 10.4.0.26.

  The ONNX parser and plug-in libraries that are bundled with this container are built from TensorRT Open Source Software.

The container also includes the following:

• Ubuntu 22.04
  Note:

  Container image 24.08-py3 contains Python 3.10.

• NVIDIA CUDA 12.6.1
• NVIDIA cuBLAS 12.6.3.1
• NVIDIA cuDNN 9.4.0.58
• NVIDIA NCCL 2.22.3
  Note:

  Although NCCL is packaged in the container, it does not affect TensorRT nor inferencing in any way.

• rdma-core 39.0
• OpenMPI 4.1.7
• OpenUCX 1.15.0
• GDRCopy 2.3
• NVIDIA HPC-X 2.20
• Nsight Compute 2024.3.1.2
• Nsight Systems 2024.4.2.133

Driver Requirements

Release 24.09 is based on CUDA 12.6.1 which requires NVIDIA Driver release 560 or later. However, if you are running on a data center GPU (for example, T4 or any other data center GPU), you can use NVIDIA driver release 470.57 (or later R470), 525.85 (or later R525), 535.86 (or later R535), or 545.23 (or later R545).

The CUDA driver's compatibility package only supports particular drivers. Thus, users should upgrade from all R418, R440, R450, R460, R510, R520, R530, R545 and R555 drivers, which are not forward-compatible with CUDA 12.6. For a complete list of supported drivers, see the CUDA Application Compatibility topic. For more information, see CUDA Compatibility and Upgrades.

GPU Requirements

Release 24.08 supports CUDA compute capability 6.0 and later. This corresponds to GPUs in the NVIDIA Pascal, NVIDIA Volta™, NVIDIA Turing™, NVIDIA Ampere architecture, NVIDIA Hopper™, and NVIDIA Ada Lovelace architecture families. For a list of GPUs to which this compute capability corresponds, see CUDA GPUs. For additional support details, see the Deep Learning Frameworks Support Matrix.

Key Features and Enhancements

This TensorRT container release includes the following key features and enhancements.

• TensorRT container image version 24.08 is based on TensorRT 10.3.0.26.

  For a list of the features and enhancements that were introduced in this version of TensorRT, refer to the TensorRT release notes.

• All dependencies on cuDNN have been removed from the TensorRT starting with the 8.6.3 release to reduce the overall container size. Any TensorRT features which depend on cuDNN, which are primarily some plugins and samples, will not work with this release.
• Latest version of Ubuntu 22.04 with October 2023 updates.

Announcements

• Starting with the 23.11 release, TensorRT containers supporting iGPU architectures are published, and run on Jetson devices. Please refer to the Frameworks Support Matrix for information regarding which iGPU hardware/software is supported by which container.
• NVIDIA Deep Learning Profiler (DLProf) v1.8, which was included in the 21.12 container, was the last release of DLProf.

  Starting with the 22.01 container, DLProf is no longer included. It can still be manually installed by using a pip wheel on the nvidia-pyindex.

Obtaining Missing Data Files

Some samples require data files that are not included in the TensorRT container because of licensing restrictions, or because they are too large. Samples that do not include the required data files include a README.md file in the corresponding source directory that provides information about how to obtain the necessary data files.

Installing Required Python Modules

• To complete some of the samples, you might want to first run the Python setup script.
• If you need to install the missing Python modules and their dependencies, run the /opt/tensorrt/python/python_setup.sh script.

Installing Open Source Components

A script has been added to clone, build, and replace the provided plug-in, the Caffe parser, and the ONNX parser libraries with the open source ones that are based on the 24.07 tag on the official TensorRT open source repository.

To install the open source components in the container, run the following script:

/opt/tensorrt/install_opensource.sh -b main

For more information, see GitHub: TensorRT.

Limitations

• Accelerating Inference In TensorFlow with TensorRT (TF-TRT) is not supported in the TensorRT containers.

  Use the TensorFlow Container to accelerate through TF-TRT instead.

• Torch-TensorRT is not supported in the TensorRT containers.

  Use the PyTorch Container to accelerate through Torch-TRT instead.

NVIDIA TensorRT Container Versions

The following table shows what versions of Ubuntu, CUDA, and TensorRT are supported in each of the NVIDIA containers for TensorRT. For older container versions, refer to the Frameworks Support Matrix.

Known Issues

• The onnx_graphsurgeon Python module on ARM Server systems is not compatible with ONNX version 1.11.0, which is normally recommended for the included TensorRT release. You will instead need to use ONNX version 1.15.0 to resolve a possible segmentation fault.
• With r545 or r550 drivers, some models may run into "Unspecified Launch Failure" during engine building. This can be worked around by downgrading the driver version to r535.
• TensorRT’s version compatibility feature has not been extensively tested and is therefore not supported with TensorRT 8.6.3. This TensorRT release is a special release that removes cuDNN as a dependency. Version compatibility between TensorRT 8.6.1 and future versions as documented will still be supported.
• Due to removing TensorRT’s dependency on cuDNN the following networks may show performance regressions:
  • BasicUnet
  • DynUnet
  • HighResNet
  • StableDiffusion VAE-encoder
  • StableDiffusion VAE-decoder

The NVIDIA container image for TensorRT, release 24.08, is available on NGC.

Contents of the TensorRT container

This container includes the following:

• The TensorRT C++ samples and C++ API documentation.
  • Build the samples can be by running make in the /workspace/tensorrt/samples directory.
  • The resulting executables are in the /workspace/tensorrt/bin directory.
  • The C++ API documentation is in the /workspace/tensorrt/doc/cpp directory.
• The TensorRT Python samples and Python API documentation.
  • The Python samples are in the /workspace/tensorrt/samples/python directory.

    Refer to the respective README documents for more samples.

  • Many Python samples can be run by using python <script.py> -d /workspace/tensorrt/data. For example:
    Copy

    Copied!

                
    
                
    python onnx_resnet50.py -d /workspace/tensorrt/data
            

  • The Python API documentation is in the /workspace/tensorrt/doc/python directory.
• TensorRT 10.3.0.26.

  The ONNX parser and plug-in libraries that are bundled with this container are built from TensorRT Open Source Software.

The container also includes the following:

• Ubuntu 22.04
  Note:

  Container image 24.08-py3 contains Python 3.10.

• NVIDIA CUDA 12.6
• NVIDIA cuBLAS 12.6.0.22
• NVIDIA cuDNN 9.3.0.75
• NVIDIA NCCL 2.22.3
  Note:

  Although NCCL is packaged in the container, it does not affect TensorRT nor inferencing in any way.

• rdma-core 39.0
• OpenMPI 4.1.7
• OpenUCX 1.15.0
• GDRCopy 2.3
• NVIDIA HPC-X 2.19
• Nsight Compute 2024.3.0.15
• Nsight Systems 2024.4.2.133

Driver Requirements

Release 24.08 is based on CUDA 12.6 which requires NVIDIA Driver release 560 or later. However, if you are running on a data center GPU (for example, T4 or any other data center GPU), you can use NVIDIA driver release 470.57 (or later R470), 525.85 (or later R525), 535.86 (or later R535), or 545.23 (or later R545).

The CUDA driver's compatibility package only supports particular drivers. Thus, users should upgrade from all R418, R440, R450, R460, R510, R520, R530, R545 and R555 drivers, which are not forward-compatible with CUDA 12.6. For a complete list of supported drivers, see the CUDA Application Compatibility topic. For more information, see CUDA Compatibility and Upgrades.

GPU Requirements

Release 24.08 supports CUDA compute capability 6.0 and later. This corresponds to GPUs in the NVIDIA Pascal, NVIDIA Volta™, NVIDIA Turing™, NVIDIA Ampere architecture, NVIDIA Hopper™, and NVIDIA Ada Lovelace architecture families. For a list of GPUs to which this compute capability corresponds, see CUDA GPUs. For additional support details, see the Deep Learning Frameworks Support Matrix.

Key Features and Enhancements

This TensorRT container release includes the following key features and enhancements.

• TensorRT container image version 24.08 is based on TensorRT 10.3.0.26.

  For a list of the features and enhancements that were introduced in this version of TensorRT, refer to the TensorRT release notes.

• All dependencies on cuDNN have been removed from the TensorRT starting with the 8.6.3 release to reduce the overall container size. Any TensorRT features which depend on cuDNN, which are primarily some plugins and samples, will not work with this release.
• Latest version of Ubuntu 22.04 with October 2023 updates.

Announcements

• Starting with the 23.11 release, TensorRT containers supporting iGPU architectures are published, and run on Jetson devices. Please refer to the Frameworks Support Matrix for information regarding which iGPU hardware/software is supported by which container.
• NVIDIA Deep Learning Profiler (DLProf) v1.8, which was included in the 21.12 container, was the last release of DLProf.

  Starting with the 22.01 container, DLProf is no longer included. It can still be manually installed by using a pip wheel on the nvidia-pyindex.

Obtaining Missing Data Files

Some samples require data files that are not included in the TensorRT container because of licensing restrictions, or because they are too large. Samples that do not include the required data files include a README.md file in the corresponding source directory that provides information about how to obtain the necessary data files.

Installing Required Python Modules

• To complete some of the samples, you might want to first run the Python setup script.
• If you need to install the missing Python modules and their dependencies, run the /opt/tensorrt/python/python_setup.sh script.

Installing Open Source Components

A script has been added to clone, build, and replace the provided plug-in, the Caffe parser, and the ONNX parser libraries with the open source ones that are based on the 24.07 tag on the official TensorRT open source repository.

To install the open source components in the container, run the following script:

/opt/tensorrt/install_opensource.sh -b main

For more information, see GitHub: TensorRT.

Limitations

• Accelerating Inference In TensorFlow with TensorRT (TF-TRT) is not supported in the TensorRT containers.

  Use the TensorFlow Container to accelerate through TF-TRT instead.

• Torch-TensorRT is not supported in the TensorRT containers.

  Use the PyTorch Container to accelerate through Torch-TRT instead.

NVIDIA TensorRT Container Versions

The following table shows what versions of Ubuntu, CUDA, and TensorRT are supported in each of the NVIDIA containers for TensorRT. For older container versions, refer to the Frameworks Support Matrix.

Known Issues

• The onnx_graphsurgeon Python module on ARM Server systems is not compatible with ONNX version 1.11.0, which is normally recommended for the included TensorRT release. You will instead need to use ONNX version 1.15.0 to resolve a possible segmentation fault.
• With r545 or r550 drivers, some models may run into "Unspecified Launch Failure" during engine building. This can be worked around by downgrading the driver version to r535.
• TensorRT’s version compatibility feature has not been extensively tested and is therefore not supported with TensorRT 8.6.3. This TensorRT release is a special release that removes cuDNN as a dependency. Version compatibility between TensorRT 8.6.1 and future versions as documented will still be supported.
• Due to removing TensorRT’s dependency on cuDNN the following networks may show performance regressions:
  • BasicUnet
  • DynUnet
  • HighResNet
  • StableDiffusion VAE-encoder
  • StableDiffusion VAE-decoder

The NVIDIA container image for TensorRT, release 24.07, is available on NGC.

Contents of the TensorRT container

This container includes the following:

• The TensorRT C++ samples and C++ API documentation.
  • Build the samples can be by running make in the /workspace/tensorrt/samples directory.
  • The resulting executables are in the /workspace/tensorrt/bin directory.
  • The C++ API documentation is in the /workspace/tensorrt/doc/cpp directory.
• The TensorRT Python samples and Python API documentation.
  • The Python samples are in the /workspace/tensorrt/samples/python directory.

    Refer to the respective README documents for more samples.

  • Many Python samples can be run by using python <script.py> -d /workspace/tensorrt/data. For example:
    Copy

    Copied!

                
    
                
    python onnx_resnet50.py -d /workspace/tensorrt/data
            

  • The Python API documentation is in the /workspace/tensorrt/doc/python directory.
• TensorRT 10.2.0.19.

  The ONNX parser and plug-in libraries that are bundled with this container are built from TensorRT Open Source Software.

The container also includes the following:

• Ubuntu 22.04
  Note:

  Container image 24.07-py3 contains Python 3.10.

• NVIDIA CUDA 12.5.1
• NVIDIA cuBLAS 12.5.3.2
• NVIDIA cuDNN 9.2.1.18
• NVIDIA NCCL 2.22.3
  Note:

  Although NCCL is packaged in the container, it does not affect TensorRT nor inferencing in any way.

• rdma-core 39.0
• OpenMPI 4.1.7
• OpenUCX 1.15.0
• GDRCopy 2.3
• NVIDIA HPC-X 2.19
• Nsight Compute 2024.2.1.2
• Nsight Systems 2024.4.2.133

Driver Requirements

Release 24.07 is based on CUDA 12.5.1 which requires NVIDIA Driver release 555 or later. However, if you are running on a data center GPU (for example, T4 or any other data center GPU), you can use NVIDIA driver release 470.57 (or later R470), 525.85 (or later R525), 535.86 (or later R535), or 545.23 (or later R545).

The CUDA driver's compatibility package only supports particular drivers. Thus, users should upgrade from all R418, R440, R450, R460, R510, R520 and R545 drivers, which are not forward-compatible with CUDA 12.5. For a complete list of supported drivers, see the CUDA Application Compatibility topic. For more information, see CUDA Compatibility and Upgrades.

GPU Requirements

Release 24.07 supports CUDA compute capability 6.0 and later. This corresponds to GPUs in the NVIDIA Pascal, NVIDIA Volta™, NVIDIA Turing™, NVIDIA Ampere architecture, NVIDIA Hopper™, and NVIDIA Ada Lovelace architecture families. For a list of GPUs to which this compute capability corresponds, see CUDA GPUs. For additional support details, see the Deep Learning Frameworks Support Matrix.

Key Features and Enhancements

This TensorRT container release includes the following key features and enhancements.

• TensorRT container image version 24.07 is based on TensorRT 10.2.

  For a list of the features and enhancements that were introduced in this version of TensorRT, refer to the TensorRT release notes.

• All dependencies on cuDNN have been removed from the TensorRT starting with the 8.6.3 release to reduce the overall container size. Any TensorRT features which depend on cuDNN, which are primarily some plugins and samples, will not work with this release.
• Latest version of Ubuntu 22.04 with October 2023 updates.

Announcements

• Starting with the 23.11 release, TensorRT containers supporting iGPU architectures are published, and run on Jetson devices. Please refer to the Frameworks Support Matrix for information regarding which iGPU hardware/software is supported by which container.
• NVIDIA Deep Learning Profiler (DLProf) v1.8, which was included in the 21.12 container, was the last release of DLProf.

  Starting with the 22.01 container, DLProf is no longer included. It can still be manually installed by using a pip wheel on the nvidia-pyindex.

Obtaining Missing Data Files

Some samples require data files that are not included in the TensorRT container because of licensing restrictions, or because they are too large. Samples that do not include the required data files include a README.md file in the corresponding source directory that provides information about how to obtain the necessary data files.

Installing Required Python Modules

• To complete some of the samples, you might want to first run the Python setup script.
• If you need to install the missing Python modules and their dependencies, run the /opt/tensorrt/python/python_setup.sh script.

Installing Open Source Components

A script has been added to clone, build, and replace the provided plug-in, the Caffe parser, and the ONNX parser libraries with the open source ones that are based on the 24.07 tag on the official TensorRT open source repository.

To install the open source components in the container, run the following script:

/opt/tensorrt/install_opensource.sh -b main

For more information, see GitHub: TensorRT.

Limitations

• Accelerating Inference In TensorFlow with TensorRT (TF-TRT) is not supported in the TensorRT containers.

  Use the TensorFlow Container to accelerate through TF-TRT instead.

• Torch-TensorRT is not supported in the TensorRT containers.

  Use the PyTorch Container to accelerate through Torch-TRT instead.

NVIDIA TensorRT Container Versions

The following table shows what versions of Ubuntu, CUDA, and TensorRT are supported in each of the NVIDIA containers for TensorRT. For older container versions, refer to the Frameworks Support Matrix.

Known Issues

• The onnx_graphsurgeon Python module on ARM Server systems is not compatible with ONNX version 1.11.0, which is normally recommended for the included TensorRT release. You will instead need to use ONNX version 1.15.0 to resolve a possible segmentation fault.
• With r545 or r550 drivers, some models may run into "Unspecified Launch Failure" during engine building. This can be worked around by downgrading the driver version to r535.
• TensorRT’s version compatibility feature has not been extensively tested and is therefore not supported with TensorRT 8.6.3. This TensorRT release is a special release that removes cuDNN as a dependency. Version compatibility between TensorRT 8.6.1 and future versions as documented will still be supported.
• Due to removing TensorRT’s dependency on cuDNN the following networks may show performance regressions:
  • BasicUnet
  • DynUnet
  • HighResNet
  • StableDiffusion VAE-encoder
  • StableDiffusion VAE-decoder

The NVIDIA container image for TensorRT, release 24.06, is available on NGC.

Contents of the TensorRT container

This container includes the following:

• The TensorRT C++ samples and C++ API documentation.
  • Build the samples can be by running make in the /workspace/tensorrt/samples directory.
  • The resulting executables are in the /workspace/tensorrt/bin directory.
  • The C++ API documentation is in the /workspace/tensorrt/doc/cpp directory.
• The TensorRT Python samples and Python API documentation.
  • The Python samples are in the /workspace/tensorrt/samples/python directory.

    Refer to the respective README documents for more samples.

  • Many Python samples can be run by using python <script.py> -d /workspace/tensorrt/data. For example:
    Copy

    Copied!

                
    
                
    python onnx_resnet50.py -d /workspace/tensorrt/data
            

  • The Python API documentation is in the /workspace/tensorrt/doc/python directory.
• TensorRT 8.6.3.

  The ONNX parser and plug-in libraries that are bundled with this container are built from TensorRT Open Source Software.

The container also includes the following:

• Ubuntu 22.04
  Note:

  Container image 24.06-py3 contains Python 3.10.

• NVIDIA CUDA 12.5.0.23
• NVIDIA cuBLAS 12.5.2.13
• NVIDIA cuDNN 9.1.0.70
• NVIDIA NCCL 2.21.5
  Note:

  Although NCCL is packaged in the container, it does not affect TensorRT nor inferencing in any way.

• rdma-core 39.0
• OpenMPI 4.1.4+
• OpenUCX 1.15.0
• GDRCopy 2.3
• NVIDIA HPC-X 2.19
• Nsight Compute 2024.2.0.16
• Nsight Systems 2024.2.3.38

Driver Requirements

Release 24.06 is based on NVIDIA CUDA 12.5.0.23, which requires NVIDIA Driver release 545 or later. However, if you are running on a data center GPU (for example, T4 or any other data center GPU), you can use NVIDIA driver release 470.57 (or later R470), 525.85 (or later R525), 535.86 (or later R535), or 545.23 (or later R545).

The CUDA driver's compatibility package only supports particular drivers. Thus, users should upgrade from all R418, R440, R450, R460, R510, and R520 drivers, which are not forward-compatible with CUDA 12.3. For a complete list of supported drivers, see the CUDA Application Compatibility topic. For more information, see CUDA Compatibility and Upgrades.

GPU Requirements

Release 24.06 supports CUDA compute capability 6.0 and later. This corresponds to GPUs in the NVIDIA Pascal, NVIDIA Volta™, NVIDIA Turing™, NVIDIA Ampere architecture, NVIDIA Hopper™, and NVIDIA Ada Lovelace architecture families. For a list of GPUs to which this compute capability corresponds, see CUDA GPUs. For additional support details, see the Deep Learning Frameworks Support Matrix.

Key Features and Enhancements

This TensorRT container release includes the following key features and enhancements.

• TensorRT container image version 24.06 is based on TensorRT 10.1.

  For a list of the features and enhancements that were introduced in this version of TensorRT, refer to the TensorRT release notes.

• All dependencies on cuDNN have been removed from the TensorRT starting with the 8.6.3 release to reduce the overall container size. Any TensorRT features which depend on cuDNN, which are primarily some plugins and samples, will not work with this release.
• Latest version of Ubuntu 22.04 with October 2023 updates.

Announcements

• Starting with the 23.11 release, TensorRT containers supporting iGPU architectures are published, and run on Jetson devices. Please refer to the Frameworks Support Matrix for information regarding which iGPU hardware/software is supported by which container.
• NVIDIA Deep Learning Profiler (DLProf) v1.8, which was included in the 21.12 container, was the last release of DLProf.

  Starting with the 22.01 container, DLProf is no longer included. It can still be manually installed by using a pip wheel on the nvidia-pyindex.

Obtaining Missing Data Files

Some samples require data files that are not included in the TensorRT container because of licensing restrictions, or because they are too large. Samples that do not include the required data files include a README.md file in the corresponding source directory that provides information about how to obtain the necessary data files.

Installing Required Python Modules

• To complete some of the samples, you might want to first run the Python setup script.
• If you need to install the missing Python modules and their dependencies, run the /opt/tensorrt/python/python_setup.sh script.

Installing Open Source Components

A script has been added to clone, build, and replace the provided plug-in, the Caffe parser, and the ONNX parser libraries with the open source ones that are based on the 24.06 tag on the official TensorRT open source repository.

To install the open source components in the container, run the following script:

/opt/tensorrt/install_opensource.sh -b main

For more information, see GitHub: TensorRT.

Limitations

• Accelerating Inference In TensorFlow with TensorRT (TF-TRT) is not supported in the TensorRT containers.

  Use the TensorFlow Container to accelerate through TF-TRT instead.

• Torch-TensorRT is not supported in the TensorRT containers.

  Use the PyTorch Container to accelerate through Torch-TRT instead.

NVIDIA TensorRT Container Versions

The following table shows what versions of Ubuntu, CUDA, and TensorRT are supported in each of the NVIDIA containers for TensorRT. For older container versions, refer to the Frameworks Support Matrix.

Known Issues

• The onnx_graphsurgeon Python module on ARM Server systems is not compatible with ONNX version 1.11.0, which is normally recommended for the included TensorRT release. You will instead need to use ONNX version 1.15.0 to resolve a possible segmentation fault.
• With r545 or r550 drivers, some models may run into "Unspecified Launch Failure" during engine building. This can be worked around by downgrading the driver version to r535.
• TensorRT’s version compatibility feature has not been extensively tested and is therefore not supported with TensorRT 8.6.3. This TensorRT release is a special release that removes cuDNN as a dependency. Version compatibility between TensorRT 8.6.1 and future versions as documented will still be supported.
• Due to removing TensorRT’s dependency on cuDNN the following networks may show performance regressions:
  • BasicUnet
  • DynUnet
  • HighResNet
  • StableDiffusion VAE-encoder
  • StableDiffusion VAE-decoder

The NVIDIA container image for TensorRT, release 24.05, is available on NGC.

Contents of the TensorRT container

This container includes the following:

• The TensorRT C++ samples and C++ API documentation.
  • Build the samples can be by running make in the /workspace/tensorrt/samples directory.
  • The resulting executables are in the /workspace/tensorrt/bin directory.
  • The C++ API documentation is in the /workspace/tensorrt/doc/cpp directory.
• The TensorRT Python samples and Python API documentation.
  • The Python samples are in the /workspace/tensorrt/samples/python directory.

    Refer to the respective README documents for more samples.

  • Many Python samples can be run by using python <script.py> -d /workspace/tensorrt/data. For example:
    Copy

    Copied!

                
    
                
    python onnx_resnet50.py -d /workspace/tensorrt/data
            

  • The Python API documentation is in the /workspace/tensorrt/doc/python directory.
• TensorRT 8.6.3.

  The ONNX parser and plug-in libraries that are bundled with this container are built from TensorRT Open Source Software.

The container also includes the following:

• Ubuntu 22.04
  Note:

  Container image 24.05-py3 contains Python 3.10.

• NVIDIA CUDA 12.4.1
• NVIDIA cuBLAS 12.4.5.8
• NVIDIA cuDNN 9.1.0.70
• NVIDIA NCCL 2.21.5
  Note:

  Although NCCL is packaged in the container, it does not affect TensorRT nor inferencing in any way.

• rdma-core 39.0
• OpenMPI 4.1.4+
• OpenUCX 1.15.0
• GDRCopy 2.3
• NVIDIA HPC-X 2.19
• Nsight Compute 2024.1.1.4
• Nsight Systems 2024.2.1.106

Driver Requirements

Release 24.05 is based on NVIDIA CUDA 12.4.0.41, which requires NVIDIA Driver release 545 or later. However, if you are running on a data center GPU (for example, T4 or any other data center GPU), you can use NVIDIA driver release 470.57 (or later R470), 525.85 (or later R525), 535.86 (or later R535), or 545.23 (or later R545).

The CUDA driver's compatibility package only supports particular drivers. Thus, users should upgrade from all R418, R440, R450, R460, R510, and R520 drivers, which are not forward-compatible with CUDA 12.3. For a complete list of supported drivers, see the CUDA Application Compatibility topic. For more information, see CUDA Compatibility and Upgrades.

GPU Requirements

Release 24.05 supports CUDA compute capability 6.0 and later. This corresponds to GPUs in the NVIDIA Pascal, NVIDIA Volta™, NVIDIA Turing™, NVIDIA Ampere architecture, NVIDIA Hopper™, and NVIDIA Ada Lovelace architecture families. For a list of GPUs to which this compute capability corresponds, see CUDA GPUs. For additional support details, see the Deep Learning Frameworks Support Matrix.

Key Features and Enhancements

This TensorRT container release includes the following key features and enhancements.

• TensorRT container image version 24.05 is based on TensorRT 8.6.3.1.

  For a list of the features and enhancements that were introduced in TensorRT 8.6, refer to the TensorRT release notes.

• All dependencies on cuDNN have been removed from the TensorRT 8.6.3 release to reduce the overall container size. Any TensorRT features which depend on cuDNN, which are primarily some plugins and samples, will not work with this release.
• Latest version of Ubuntu 22.04 with October 2023 updates.

Announcements

• Starting with the 23.11 release, TensorRT containers supporting iGPU architectures are published, and run on Jetson devices. Please refer to the Frameworks Support Matrix for information regarding which iGPU hardware/software is supported by which container.
• NVIDIA Deep Learning Profiler (DLProf) v1.8, which was included in the 21.12 container, was the last release of DLProf.

  Starting with the 22.01 container, DLProf is longer included. It can still be manually installed by using a pip wheel on the nvidia-pyindex.

Obtaining Missing Data Files

Some samples require data files that are not included in the TensorRT container because of licensing restrictions, or because they are too large. Samples that do not include the required data files include a README.md file in the corresponding source directory that provides information about how to obtain the necessary data files.

Installing Required Python Modules

• To complete some of the samples, you might want to first run the Python setup script.
• If you need to install the missing Python modules and their dependencies, run the /opt/tensorrt/python/python_setup.sh script.

Installing Open Source Components

A script has been added to clone, build, and replace the provided plug-in, the Caffe parser, and the ONNX parser libraries with the open source ones that are based on the 24.05 tag on the official TensorRT open source repository.

To install the open source components in the container, run the following script:

/opt/tensorrt/install_opensource.sh -b main

For more information, see GitHub: TensorRT.

Limitations

• Accelerating Inference In TensorFlow with TensorRT (TF-TRT) is not supported in the TensorRT containers.

  Use the TensorFlow Container to accelerate through TF-TRT instead.

• Torch-TensorRT is not supported in the TensorRT containers.

  Use the PyTorch Container to accelerate through Torch-TRT instead.

NVIDIA TensorRT Container Versions

The following table shows what versions of Ubuntu, CUDA, and TensorRT are supported in each of the NVIDIA containers for TensorRT. For older container versions, refer to the Frameworks Support Matrix.

Known Issues

• The onnx_graphsurgeon Python module on ARM Server systems is not compatible with ONNX version 1.11.0, which is normally recommended for the included TensorRT release. You will instead need to use ONNX version 1.15.0 to resolve a possible segmentation fault.
• With r545 or r550 drivers, some models may run into "Unspecified Launch Failure" during engine building. This can be worked around by downgrading the driver version to r535.
• TensorRT’s version compatibility feature has not been extensively tested and is therefore not supported with TensorRT 8.6.3. This TensorRT release is a special release that removes cuDNN as a dependency. Version compatibility between TensorRT 8.6.1 and future versions as documented will still be supported.
• Due to removing TensorRT’s dependency on cuDNN the following networks may show performance regressions:
  • BasicUnet
  • DynUnet
  • HighResNet
  • StableDiffusion VAE-encoder
  • StableDiffusion VAE-decoder

The NVIDIA container image for TensorRT, release 24.04, is available on NGC.

Contents of the TensorRT container

This container includes the following:

• The TensorRT C++ samples and C++ API documentation.
  • Build the samples can be by running make in the /workspace/tensorrt/samples directory.
  • The resulting executables are in the /workspace/tensorrt/bin directory.
  • The C++ API documentation is in the /workspace/tensorrt/doc/cpp directory.
• The TensorRT Python samples and Python API documentation.
  • The Python samples are in the /workspace/tensorrt/samples/python directory.

    Refer to the respective README documents for more samples.

  • Many Python samples can be run by using python <script.py> -d /workspace/tensorrt/data. For example:
    Copy

    Copied!

                
    
                
    python onnx_resnet50.py -d /workspace/tensorrt/data
            

  • The Python API documentation is in the /workspace/tensorrt/doc/python directory.
• TensorRT 8.6.3.

  The ONNX parser and plug-in libraries that are bundled with this container are built from TensorRT Open Source Software.

The container also includes the following:

• Ubuntu 22.04
  Note:

  Container image 24.04-py3 contains Python 3.10.

• NVIDIA CUDA 12.4
• NVIDIA cuBLAS 12.4.5.8
• NVIDIA cuDNN 9.1.0.70
• NVIDIA NCCL 2.21.5
  Note:

  Although NCCL is packaged in the container, it does not affect TensorRT nor inferencing in any way.

• rdma-core 39.0
• OpenMPI 4.1.4+
• OpenUCX 1.15.0
• GDRCopy 2.3
• NVIDIA HPC-X 2.18
• Nsight Compute 2024.1.1.4
• Nsight Systems 2024.2.1.106

Driver Requirements

Release 24.04 is based on NVIDIA CUDA 12.4.0.41, which requires NVIDIA Driver release 545 or later. However, if you are running on a data center GPU (for example, T4 or any other data center GPU), you can use NVIDIA driver release 470.57 (or later R470), 525.85 (or later R525), 535.86 (or later R535), or 545.23 (or later R545).

The CUDA driver's compatibility package only supports particular drivers. Thus, users should upgrade from all R418, R440, R450, R460, R510, and R520 drivers, which are not forward-compatible with CUDA 12.3. For a complete list of supported drivers, see the CUDA Application Compatibility topic. For more information, see CUDA Compatibility and Upgrades.

GPU Requirements

Release 24.04 supports CUDA compute capability 6.0 and later. This corresponds to GPUs in the NVIDIA Pascal, NVIDIA Volta™, NVIDIA Turing™, NVIDIA Ampere architecture, NVIDIA Hopper™, and NVIDIA Ada Lovelace architecture families. For a list of GPUs to which this compute capability corresponds, see CUDA GPUs. For additional support details, see the Deep Learning Frameworks Support Matrix.

Key Features and Enhancements

This TensorRT container release includes the following key features and enhancements.

• TensorRT container image version 24.04 is based on TensorRT 8.6.3.1.

  For a list of the features and enhancements that were introduced in TensorRT 8.6, refer to the TensorRT release notes.

• All dependencies on cuDNN have been removed from the TensorRT 8.6.3 release to reduce the overall container size. Any TensorRT features which depend on cuDNN, which are primarily some plugins and samples, will not work with this release.
• Latest version of Ubuntu 22.04 with October 2023 updates.

Announcements

• Starting with the 23.11 release, TensorRT containers supporting iGPU architectures are published, and run on Jetson devices. Please refer to the Frameworks Support Matrix for information regarding which iGPU hardware/software is supported by which container.
• NVIDIA Deep Learning Profiler (DLProf) v1.8, which was included in the 21.12 container, was the last release of DLProf.

  Starting with the 22.01 container, DLProf is longer included. It can still be manually installed by using a pip wheel on the nvidia-pyindex.

Obtaining Missing Data Files

Some samples require data files that are not included in the TensorRT container because of licensing restrictions, or because they are too large. Samples that do not include the required data files include a README.md file in the corresponding source directory that provides information about how to obtain the necessary data files.

Installing Required Python Modules

• To complete some of the samples, you might want to first run the Python setup script.
• If you need to install the missing Python modules and their dependencies, run the /opt/tensorrt/python/python_setup.sh script.

Installing Open Source Components

A script has been added to clone, build, and replace the provided plug-in, the Caffe parser, and the ONNX parser libraries with the open source ones that are based on the 24.04 tag on the official TensorRT open source repository.

To install the open source components in the container, run the following script:

/opt/tensorrt/install_opensource.sh -b main

For more information, see GitHub: TensorRT.

Limitations

• Accelerating Inference In TensorFlow with TensorRT (TF-TRT) is not supported in the TensorRT containers.

  Use the TensorFlow Container to accelerate through TF-TRT instead.

• Torch-TensorRT is not supported in the TensorRT containers.

  Use the PyTorch Container to accelerate through Torch-TRT instead.

NVIDIA TensorRT Container Versions

The following table shows what versions of Ubuntu, CUDA, and TensorRT are supported in each of the NVIDIA containers for TensorRT. For older container versions, refer to the Frameworks Support Matrix.

Known Issues

• The onnx_graphsurgeon Python module on ARM Server systems is not compatible with ONNX version 1.11.0, which is normally recommended for the included TensorRT release. You will instead need to use ONNX version 1.15.0 to resolve a possible segmentation fault.
• With r545 or r550 drivers, some models may run into "Unspecified Launch Failure" during engine building. This can be worked around by downgrading the driver version to r535.
• TensorRT’s version compatibility feature has not been extensively tested and is therefore not supported with TensorRT 8.6.3. This TensorRT release is a special release that removes cuDNN as a dependency. Version compatibility between TensorRT 8.6.1 and future versions as documented will still be supported.
• Due to removing TensorRT’s dependency on cuDNN the following networks may show performance regressions:
  • BasicUnet
  • DynUnet
  • HighResNet
  • StableDiffusion VAE-encoder
  • StableDiffusion VAE-decoder

The NVIDIA container image for TensorRT, release 24.03, is available on NGC.

Contents of the TensorRT container

This container includes the following:

• The TensorRT C++ samples and C++ API documentation.
  • Build the samples can be by running make in the /workspace/tensorrt/samples directory.
  • The resulting executables are in the /workspace/tensorrt/bin directory.
  • The C++ API documentation is in the /workspace/tensorrt/doc/cpp directory.
• The TensorRT Python samples and Python API documentation.
  • The Python samples are in the /workspace/tensorrt/samples/python directory.

    Refer to the respective README documents for more samples.

  • Many Python samples can be run by using python <script.py> -d /workspace/tensorrt/data. For example:
    Copy

    Copied!

                
    
                
    python onnx_resnet50.py -d /workspace/tensorrt/data
            

  • The Python API documentation is in the /workspace/tensorrt/doc/python directory.
• TensorRT 8.6.3.

  The ONNX parser and plug-in libraries that are bundled with this container are built from TensorRT Open Source Software.

The container also includes the following:

• Ubuntu 22.04
  Note:

  Container image 24.03-py3 contains Python 3.10.

• NVIDIA CUDA 12.4.0.41
• NVIDIA cuBLAS 12.4.2.65
• NVIDIA cuDNN 9.0.0.306
• NVIDIA NCCL 2.20
  Note:

  Although NCCL is packaged in the container, it does not affect TensorRT nor inferencing in any way.

• rdma-core 39.0
• OpenMPI 4.1.4+
• OpenUCX 1.15.0
• GDRCopy 2.3
• NVIDIA HPC-X 2.18
• Nsight Compute 2024.1.0.13
• Nsight Systems 2024.2.1.38

Driver Requirements

Release 24.03 is based on NVIDIA CUDA 12.4.0.41, which requires NVIDIA Driver release 545 or later. However, if you are running on a data center GPU (for example, T4 or any other data center GPU), you can use NVIDIA driver release 470.57 (or later R470), 525.85 (or later R525), 535.86 (or later R535), or 545.23 (or later R545).

The CUDA driver's compatibility package only supports particular drivers. Thus, users should upgrade from all R418, R440, R450, R460, R510, and R520 drivers, which are not forward-compatible with CUDA 12.3. For a complete list of supported drivers, see the CUDA Application Compatibility topic. For more information, see CUDA Compatibility and Upgrades.

GPU Requirements

Release 24.03 supports CUDA compute capability 6.0 and later. This corresponds to GPUs in the NVIDIA Pascal, NVIDIA Volta™, NVIDIA Turing™, NVIDIA Ampere architecture, NVIDIA Hopper™, and NVIDIA Ada Lovelace architecture families. For a list of GPUs to which this compute capability corresponds, see CUDA GPUs. For additional support details, see the Deep Learning Frameworks Support Matrix.

Key Features and Enhancements

This TensorRT container release includes the following key features and enhancements.

• TensorRT container image version 24.03 is based on TensorRT 8.6.3.1.

  For a list of the features and enhancements that were introduced in TensorRT 8.6, refer to the TensorRT 8.6 release notes.

• All dependencies on cuDNN have been removed from the TensorRT 8.6.3 release to reduce the overall container size. Any TensorRT features which depend on cuDNN, which are primarily some plugins and samples, will not work with this release.
• Latest version of Ubuntu 22.04 with October 2023 updates.

Announcements

• Starting with the 23.11 release, TensorRT containers supporting iGPU architectures are published, and run on Jetson devices. Please refer to the Frameworks Support Matrix for information regarding which iGPU hardware/software is supported by which container.
• NVIDIA Deep Learning Profiler (DLProf) v1.8, which was included in the 21.12 container, was the last release of DLProf.

  Starting with the 22.01 container, DLProf is longer included. It can still be manually installed by using a pip wheel on the nvidia-pyindex.

Obtaining Missing Data Files

Some samples require data files that are not included in the TensorRT container because of licensing restrictions, or because they are too large. Samples that do not include the required data files include a README.md file in the corresponding source directory that provides information about how to obtain the necessary data files.

Installing Required Python Modules

• To complete some of the samples, you might want to first run the Python setup script.
• If you need to install the missing Python modules and their dependencies, run the /opt/tensorrt/python/python_setup.sh script.

Installing Open Source Components

A script has been added to clone, build, and replace the provided plug-in, the Caffe parser, and the ONNX parser libraries with the open source ones that are based on the 24.03 tag on the official TensorRT open source repository.

To install the open source components in the container, run the following script:

/opt/tensorrt/install_opensource.sh -b main

For more information, see GitHub: TensorRT.

Limitations

• Accelerating Inference In TensorFlow with TensorRT (TF-TRT) is not supported in the TensorRT containers.

  Use the TensorFlow Container to accelerate through TF-TRT instead.

• Torch-TensorRT is not supported in the TensorRT containers.

  Use the PyTorch Container to accelerate through Torch-TRT instead.

NVIDIA TensorRT Container Versions

The following table shows what versions of Ubuntu, CUDA, and TensorRT are supported in each of the NVIDIA containers for TensorRT. For older container versions, refer to the Frameworks Support Matrix.

Known Issues

• The onnx_graphsurgeon Python module on ARM Server systems is not compatible with ONNX version 1.11.0, which is normally recommended for the included TensorRT release. You will instead need to use ONNX version 1.15.0 to resolve a possible segmentation fault.
• With r545 or r550 drivers, some models may run into "Unspecified Launch Failure" during engine building. This can be worked around by downgrading the driver version to r535.
• TensorRT’s version compatibility feature has not been extensively tested and is therefore not supported with TensorRT 8.6.3. This TensorRT release is a special release that removes cuDNN as a dependency. Version compatibility between TensorRT 8.6.1 and future versions as documented will still be supported.
• Due to removing TensorRT’s dependency on cuDNN the following networks may show performance regressions:
  • BasicUnet
  • DynUnet
  • HighResNet
  • StableDiffusion VAE-encoder
  • StableDiffusion VAE-decoder

The NVIDIA container image for TensorRT, release 24.02, is available on NGC.

Contents of the TensorRT container

This container includes the following:

• The TensorRT C++ samples and C++ API documentation.
  • Build the samples can be by running make in the /workspace/tensorrt/samples directory.
  • The resulting executables are in the /workspace/tensorrt/bin directory.
  • The C++ API documentation is in the /workspace/tensorrt/doc/cpp directory.
• The TensorRT Python samples and Python API documentation.
  • The Python samples are in the /workspace/tensorrt/samples/python directory.

    Refer to the respective README documents for more samples.

  • Many Python samples can be run by using python <script.py> -d /workspace/tensorrt/data. For example:
    Copy

    Copied!

                
    
                
    python onnx_resnet50.py -d /workspace/tensorrt/data
            

  • The Python API documentation is in the /workspace/tensorrt/doc/python directory.
• TensorRT 8.6.3.

  The ONNX parser and plug-in libraries that are bundled with this container are built from TensorRT Open Source Software.

The container also includes the following:

• Ubuntu 22.04
  Note:

  Container image 24.01-py3 contains Python 3.10.

• NVIDIA CUDA® 12.3.2
• NVIDIA cuBLAS 12.3.4.1
• NVIDIA cuDNN 9.0.0.306
• NVIDIA NCCL 2.19.4
  Note:

  Although NCCL is packaged in the container, it does not affect TensorRT nor inferencing in any way.

• rdma-core 39.0
• OpenMPI 4.1.4+
• OpenUCX 1.15.0
• GDRCopy 2.3
• NVIDIA HPC-X 2.16rc4
• Nsight Compute 2023.3.1.1
• Nsight Systems 2023.4.1.97

Driver Requirements

Release 24.02 is based on CUDA 12.3.2, which requires NVIDIA Driver release 545 or later. However, if you are running on a data center GPU (for example, T4 or any other data center GPU), you can use NVIDIA driver release 470.57 (or later R470), 525.85 (or later R525), 535.86 (or later R535), or 545.23 (or later R545).

The CUDA driver's compatibility package only supports particular drivers. Thus, users should upgrade from all R418, R440, R450, R460, R510, and R520 drivers, which are not forward-compatible with CUDA 12.3. For a complete list of supported drivers, see the CUDA Application Compatibility topic. For more information, see CUDA Compatibility and Upgrades.

GPU Requirements

Release 24.02 supports CUDA compute capability 6.0 and later. This corresponds to GPUs in the NVIDIA Pascal, NVIDIA Volta™, NVIDIA Turing™, NVIDIA Ampere architecture, NVIDIA Hopper™, and NVIDIA Ada Lovelace architecture families. For a list of GPUs to which this compute capability corresponds, see CUDA GPUs. For additional support details, see the Deep Learning Frameworks Support Matrix.

Key Features and Enhancements

This TensorRT container release includes the following key features and enhancements.

• TensorRT container image version 24.02 is based on TensorRT 8.6.3.1.

  For a list of the features and enhancements that were introduced in TensorRT 8.6, refer to the TensorRT 8.6 release notes.

• All dependencies on cuDNN have been removed from the TensorRT 8.6.3 release to reduce the overall container size. Any TensorRT features which depend on cuDNN, which are primarily some plugins and samples, will not work with this release.
• Latest version of Ubuntu 22.04 with October 2023 updates.

Announcements

• Starting with the 23.11 release, TensorRT containers supporting iGPU architectures are published, and run on Jetson devices. Please refer to the Frameworks Support Matrix for information regarding which iGPU hardware/software is supported by which container.
• NVIDIA Deep Learning Profiler (DLProf) v1.8, which was included in the 21.12 container, was the last release of DLProf.

  Starting with the 22.01 container, DLProf is longer included. It can still be manually installed by using a pip wheel on the nvidia-pyindex.

Obtaining Missing Data Files

Some samples require data files that are not included in the TensorRT container because of licensing restrictions, or because they are too large. Samples that do not include the required data files include a README.md file in the corresponding source directory that provides information about how to obtain the necessary data files.

Installing Required Python Modules

• To complete some of the samples, you might want to first run the Python setup script.
• If you need to install the missing Python modules and their dependencies, run the /opt/tensorrt/python/python_setup.sh script.

Installing Open Source Components

A script has been added to clone, build, and replace the provided plug-in, the Caffe parser, and the ONNX parser libraries with the open source ones that are based on the 24.01 tag on the official TensorRT open source repository.

To install the open source components in the container, run the following script:

/opt/tensorrt/install_opensource.sh -b main

For more information, see GitHub: TensorRT.

Limitations

• Accelerating Inference In TensorFlow with TensorRT (TF-TRT) is not supported in the TensorRT containers.

  Use the TensorFlow Container to accelerate through TF-TRT instead.

• Torch-TensorRT is not supported in the TensorRT containers.

  Use the PyTorch Container to accelerate through Torch-TRT instead.

NVIDIA TensorRT Container Versions

The following table shows what versions of Ubuntu, CUDA, and TensorRT are supported in each of the NVIDIA containers for TensorRT. For older container versions, refer to the Frameworks Support Matrix.

Known Issues

• The onnx_graphsurgeon Python module on ARM Server systems is not compatible with ONNX version 1.11.0, which is normally recommended for the included TensorRT release. You will instead need to use ONNX version 1.15.0 to resolve a possible segmentation fault.
• With r545 or r550 drivers, some models may run into "Unspecified Launch Failure" during engine building. This can be worked around by downgrading the driver version to r535.
• TensorRT’s version compatibility feature has not been extensively tested and is therefore not supported with TensorRT 8.6.3. This TensorRT release is a special release that removes cuDNN as a dependency. Version compatibility between TensorRT 8.6.1 and future versions as documented will still be supported.
• Due to removing TensorRT’s dependency on cuDNN the following networks may show performance regressions:
  • BasicUnet
  • DynUnet
  • HighResNet
  • StableDiffusion VAE-encoder
  • StableDiffusion VAE-decoder

The NVIDIA container image for TensorRT, release 24.01, is available on NGC.

Contents of the TensorRT container

This container includes the following:

• The TensorRT C++ samples and C++ API documentation.
  • Build the samples can be by running make in the /workspace/tensorrt/samples directory.
  • The resulting executables are in the /workspace/tensorrt/bin directory.
  • The C++ API documentation is in the /workspace/tensorrt/doc/cpp directory.
• The TensorRT Python samples and Python API documentation.
  • The Python samples are in the /workspace/tensorrt/samples/python directory.

    Refer to the respective README documents for more samples.

  • Many Python samples can be run by using python <script.py> -d /workspace/tensorrt/data. For example:
    Copy

    Copied!

                
    
                
    python onnx_resnet50.py -d /workspace/tensorrt/data
            

  • The Python API documentation is in the /workspace/tensorrt/doc/python directory.
• TensorRT 8.6.1.6.

  The ONNX parser and plug-in libraries that are bundled with this container are built from TensorRT Open Source Software.

The container also includes the following:

• Ubuntu 22.04
  Note:

  Container image 24.01-py3 contains Python 3.10.

• NVIDIA CUDA® 12.3.2
• NVIDIA cuBLAS 12.3.4.1
• NVIDIA cuDNN 8.9.7.29
• NVIDIA NCCL 2.19.4
  Note:

  Although NCCL is packaged in the container, it does not affect TensorRT nor inferencing in any way.

• rdma-core 39.0
• OpenMPI 4.1.4+
• OpenUCX 1.15.0
• GDRCopy 2.3
• NVIDIA HPC-X 2.16rc4
• Nsight Compute 2023.3.1.1
• Nsight Systems 2023.3.4.1.97

Driver Requirements

Release 24.01 is based on CUDA 12.3.2, which requires NVIDIA Driver release 545 or later. However, if you are running on a data center GPU (for example, T4 or any other data center GPU), you can use NVIDIA driver release 470.57 (or later R470), 525.85 (or later R525), 535.86 (or later R535), or 545.23 (or later R545).

The CUDA driver's compatibility package only supports particular drivers. Thus, users should upgrade from all R418, R440, R450, R460, R510, and R520 drivers, which are not forward-compatible with CUDA 12.3. For a complete list of supported drivers, see the CUDA Application Compatibility topic. For more information, see CUDA Compatibility and Upgrades.

GPU Requirements

Release 24.01 supports CUDA compute capability 6.0 and later. This corresponds to GPUs in the NVIDIA Pascal, NVIDIA Volta™, NVIDIA Turing™, NVIDIA Ampere architecture, NVIDIA Hopper™, and NVIDIA Ada Lovelace architecture families. For a list of GPUs to which this compute capability corresponds, see CUDA GPUs. For additional support details, see the Deep Learning Frameworks Support Matrix.

Key Features and Enhancements

This TensorRT container release includes the following key features and enhancements.

• TensorRT container image version 24.01 is based on TensorRT 8.6.1.6.

  For a list of the features and enhancements that were introduced in TensorRT 8.6.1, refer to the TensorRT 8.6 release notes.

• Ubuntu 22.04 with October 2023 updates.

Announcements

• Starting with the 23.11 release, TensorRT containers supporting iGPU architectures are published, and run on Jetson devices. Please refer to the Frameworks Support Matrix for information regarding which iGPU hardware/software is supported by which container.
• NVIDIA Deep Learning Profiler (DLProf) v1.8, which was included in the 21.12 container, was the last release of DLProf.

  Starting with the 22.01 container, DLProf is longer included. It can still be manually installed by using a pip wheel on the nvidia-pyindex.

Obtaining Missing Data Files

Some samples require data files that are not included in the TensorRT container because of licensing restrictions, or because they are too large. Samples that do not include the required data files include a README.md file in the corresponding source directory that provides information about how to obtain the necessary data files.

Installing Required Python Modules

• To complete some of the samples, you might want to first run the Python setup script.
• If you need to install the missing Python modules and their dependencies, run the /opt/tensorrt/python/python_setup.sh script.

Installing Open Source Components

A script has been added to clone, build, and replace the provided plug-in, the Caffe parser, and the ONNX parser libraries with the open source ones that are based on the 24.01 tag on the official TensorRT open source repository.

To install the open source components in the container, run the following script:

/opt/tensorrt/install_opensource.sh -b main

For more information, see GitHub: TensorRT.

Limitations

• Accelerating Inference In TensorFlow with TensorRT (TF-TRT) is not supported in the TensorRT containers.

  Use the TensorFlow Container to accelerate through TF-TRT instead.

• Torch-TensorRT is not supported in the TensorRT containers.

  Use the PyTorch Container to accelerate through Torch-TRT instead.

NVIDIA TensorRT Container Versions

The following table shows what versions of Ubuntu, CUDA, and TensorRT are supported in each of the NVIDIA containers for TensorRT. For older container versions, refer to the Frameworks Support Matrix.

Known Issues

None.

The NVIDIA container image for TensorRT, release 23.12, is available on NGC.

Contents of the TensorRT container

This container includes the following:

• The TensorRT C++ samples and C++ API documentation.
  • Build the samples can be by running make in the /workspace/tensorrt/samples directory.
  • The resulting executables are in the /workspace/tensorrt/bin directory.
  • The C++ API documentation is in the /workspace/tensorrt/doc/cpp directory.
• The TensorRT Python samples and Python API documentation.
  • The Python samples are in the /workspace/tensorrt/samples/python directory.

    Refer to the respective README documents for more samples.

  • Many Python samples can be run by using python <script.py> -d /workspace/tensorrt/data. For example:
    Copy

    Copied!

                
    
                
    python onnx_resnet50.py -d /workspace/tensorrt/data
            

  • The Python API documentation is in the /workspace/tensorrt/doc/python directory.
• TensorRT 8.6.1.6.

  The ONNX parser and plug-in libraries that are bundled with this container are built from TensorRT Open Source Software.

The container also includes the following:

• Ubuntu 22.04
  Note:

  Container image 23.12-py3 contains Python 3.10.

• NVIDIA CUDA® 12.3.2
• NVIDIA cuBLAS 12.3.4.1
• NVIDIA cuDNN 8.9.7.29
• NVIDIA NCCL 2.19.3
  Note:

  Although NCCL is packaged in the container, it does not affect TensorRT nor inferencing in any way.

• rdma-core 39.0
• OpenMPI 4.1.4+
• OpenUCX 1.15.0
• GDRCopy 2.3
• NVIDIA HPC-X 2.16
• Nsight Compute 2023.3.1.1
• Nsight Systems 2023.3.4.1

Driver Requirements

Release 23.12 is based on CUDA 12.3.2, which requires NVIDIA Driver release 545 or later. However, if you are running on a data center GPU (for example, T4 or any other data center GPU), you can use NVIDIA driver release 450.51 (or later R450), 470.57 (or later R470), 510.47 (or later R510), or 525.85 (or later R525) 535.86 (or later R535), or 545.23 (or later R545). The CUDA driver's compatibility package only supports particular drivers. Thus, users should upgrade from all R418, R440, R460, and R520 drivers, which are not forward-compatible with CUDA 12.3. For a complete list of supported drivers, see the CUDA Application Compatibility topic. For more information, see CUDA Compatibility and Upgrades.

GPU Requirements

Release 23.12 supports CUDA compute capability 6.0 and later. This corresponds to GPUs in the NVIDIA Kepler, Maxwell, NVIDIA Pascal, NVIDIA Volta™, NVIDIA Turing™, NVIDIA Ampere architecture, NVIDIA Hopper™, and NVIDIA Ada Lovelace architecture families. For a list of GPUs to which this compute capability corresponds, see CUDA GPUs. For additional support details, see the Deep Learning Frameworks Support Matrix.

Key Features and Enhancements

This TensorRT container release includes the following key features and enhancements.

• TensorRT container image version 23.12 is based on TensorRT 8.6.1.6.

  For a list of the features and enhancements that were introduced in TensorRT 8.6.1, refer to the TensorRT 8.6 release notes.

• Ubuntu 22.04 with October 2023 updates.

Announcements

• Starting with the 23.11 release, TensorRT containers supporting iGPU architectures are published, and run on Jetson devices. Please refer to the Frameworks Support Matrix for information regarding which iGPU hardware/software is supported by which container.
• NVIDIA Deep Learning Profiler (DLProf) v1.8, which was included in the 21.12 container, was the last release of DLProf.

  Starting with the 22.01 container, DLProf is longer included. It can still be manually installed by using a pip wheel on the nvidia-pyindex.

Obtaining Missing Data Files

Some samples require data files that are not included in the TensorRT container because of licensing restrictions, or because they are too large. Samples that do not include the required data files include a README.md file in the corresponding source directory that provides information about how to obtain the necessary data files.

Installing Required Python Modules

• To complete some of the samples, you might want to first run the Python setup script.
• If you need to install the missing Python modules and their dependencies, run the /opt/tensorrt/python/python_setup.sh script.

Installing Open Source Components

A script has been added to clone, build, and replace the provided plug-in, the Caffe parser, and the ONNX parser libraries with the open source ones that are based on the 23.12 tag on the official TensorRT open source repository.

To install the open source components in the container, run the following script:

/opt/tensorrt/install_opensource.sh -b main

For more information, see GitHub: TensorRT.

Limitations

• Accelerating Inference In TensorFlow with TensorRT (TF-TRT) is not supported in the TensorRT containers.

  Use the TensorFlow Container to accelerate through TF-TRT instead.

• Torch-TensorRT is not supported in the TensorRT containers.

  Use the PyTorch Container to accelerate through Torch-TRT instead.

NVIDIA TensorRT Container Versions

The following table shows what versions of Ubuntu, CUDA, and TensorRT are supported in each of the NVIDIA containers for TensorRT. For older container versions, refer to the Frameworks Support Matrix.

Known Issues

None.

The NVIDIA container image for TensorRT, release 23.11, is available on NGC.

Contents of the TensorRT container

This container includes the following:

• The TensorRT C++ samples and C++ API documentation.
  • Build the samples can be by running make in the /workspace/tensorrt/samples directory.
  • The resulting executables are in the /workspace/tensorrt/bin directory.
  • The C++ API documentation is in the /workspace/tensorrt/doc/cpp directory.
• The TensorRT Python samples and Python API documentation.
  • The Python samples are in the /workspace/tensorrt/samples/python directory.

    Refer to the respective README documents for more samples.

  • Many Python samples can be run by using python <script.py> -d /workspace/tensorrt/data. For example:
    Copy

    Copied!

                
    
                
    python onnx_resnet50.py -d /workspace/tensorrt/data
            

  • The Python API documentation is in the /workspace/tensorrt/doc/python directory.
• TensorRT 8.6.1.6.

  The ONNX parser and plug-in libraries that are bundled with this container are built from TensorRT Open Source Software.

The container also includes the following:

• Ubuntu 22.04
  Note:

  Container image 23.11-py3 contains Python 3.10.

• NVIDIA CUDA® 12.3.0
• NVIDIA cuBLAS 12.3.2.1
• NVIDIA cuDNN 8.9.6
• NVIDIA NCCL 2.19.3
  Note:

  Although NCCL is packaged in the container, it does not affect TensorRT nor inferencing in any way.

• rdma-core 39.0
• OpenMPI 4.1.4+
• OpenUCX 1.15.0
• GDRCopy 2.3
• NVIDIA HPC-X 2.16
• Nsight Compute 2023.3.0.12
• Nsight Systems 2023.3.1.92

Driver Requirements

Release 23.11 is based on CUDA 12.3.0, which requires NVIDIA Driver release 545 or later. However, if you are running on a data center GPU (for example, T4 or any other data center GPU), you can use NVIDIA driver release 450.51 (or later R450), 470.57 (or later R470), 510.47 (or later R510), or 525.85 (or later R525) 535.86 (or later R535), or 545.23 (or later R545). The CUDA driver's compatibility package only supports particular drivers. Thus, users should upgrade from all R418, R440, R460, and R520 drivers, which are not forward-compatible with CUDA 12.3. For a complete list of supported drivers, see the CUDA Application Compatibility topic. For more information, see CUDA Compatibility and Upgrades.

GPU Requirements

Release 23.11 supports CUDA compute capability 6.0 and later. This corresponds to GPUs in the NVIDIA Kepler, Maxwell, NVIDIA Pascal, NVIDIA Volta™, NVIDIA Turing™, NVIDIA Ampere architecture, NVIDIA Hopper™, and NVIDIA Ada Lovelace architecture families. For a list of GPUs to which this compute capability corresponds, see CUDA GPUs. For additional support details, see the Deep Learning Frameworks Support Matrix.

Key Features and Enhancements

This TensorRT container release includes the following key features and enhancements.

• TensorRT container image version 23.11 is based on TensorRT 8.6.1.6.

  For a list of the features and enhancements that were introduced in TensorRT 8.6.1, refer to the TensorRT 8.6 release notes.

• Ubuntu 22.04 with October 2023 updates.

Announcements

• Starting with the 23.11 release, TensorRT containers supporting iGPU architectures are published, and run on Jetson devices. Please refer to the Frameworks Support Matrix for information regarding which iGPU hardware/software is supported by which container.
• NVIDIA Deep Learning Profiler (DLProf) v1.8, which was included in the 21.12 container, was the last release of DLProf.

  Starting with the 22.01 container, DLProf is longer included. It can still be manually installed by using a pip wheel on the nvidia-pyindex.

Obtaining Missing Data Files

Some samples require data files that are not included in the TensorRT container because of licensing restrictions, or because they are too large. Samples that do not include the required data files include a README.md file in the corresponding source directory that provides information about how to obtain the necessary data files.

Installing Required Python Modules

• To complete some of the samples, you might want to first run the Python setup script.
• If you need to install the missing Python modules and their dependencies, run the /opt/tensorrt/python/python_setup.sh script.

Installing Open Source Components

A script has been added to clone, build, and replace the provided plug-in, the Caffe parser, and the ONNX parser libraries with the open source ones that are based on the 23.11 tag on the official TensorRT open source repository.

To install the open source components in the container, run the following script:

/opt/tensorrt/install_opensource.sh -b main

For more information, see GitHub: TensorRT.

Limitations

• Accelerating Inference In TensorFlow with TensorRT (TF-TRT) is not supported in the TensorRT containers.

  Use the TensorFlow Container to accelerate through TF-TRT instead.

• Torch-TensorRT is not supported in the TensorRT containers.

  Use the PyTorch Container to accelerate through Torch-TRT instead.

NVIDIA TensorRT Container Versions

The following table shows what versions of Ubuntu, CUDA, and TensorRT are supported in each of the NVIDIA containers for TensorRT. For older container versions, refer to the Frameworks Support Matrix.

Known Issues

None.