2. Installing TensorRT
- requires sudo or root privileges to install
- provides no flexibility as to which location TensorRT is installed into
- requires that the CUDA Toolkit and cuDNN have also been installed using Debian
- does not allow more than one minor version of TensorRT to be installed at the same time
TensorRT versions: TensorRT is a product made up of separately versioned components. The version on the product conveys important information about the significance of new features while the library version conveys information about the compatibility or incompatibility of the API.
2.1. Debian Installation
Developer Installation: The following instructions set up a full TensorRT development environment with samples, documentation and both the C++ and Python API.
Refer to Download and Run SDK Manager for instructions on setting up your development environment with NVIDIA SDK Manager.
Go to https://developer.nvidia.com/nvidia-sdk-manager for more information about using SDK Manager.
Using The NVIDIA Machine Learning Network Repo For RPM Installation
- Install the NVIDIA CUDA network repository installation package.
os="ubuntu1x04" cuda="x.y.z" wget https://developer.download.nvidia.com/compute/cuda/repos/${os}/x86_64/cuda-repo-${os}_${cuda}-1_amd64.deb sudo dpkg -i cuda-repo-*.debWhere:- OS version: ubuntu1x04 is 1804
- CUDA version: x.y.z is 10.2.89
- Install the NVIDIA Machine Learning network repository installation
package.
os="ubuntu1x04" wget https://developer.download.nvidia.com/compute/machine-learning/repos/${os}/x86_64/nvidia-machine-learning-repo-${os}_1.0.0-1_amd64.deb sudo dpkg -i nvidia-machine-learning-repo-*.deb sudo apt-get update - Install the TensorRT package that fits your particular needs.
- For only running TensorRT C++ applications:
sudo apt-get install libnvinfer6 libnvonnxparsers6 libnvparsers6 libnvinfer-plugin6
- For also building TensorRT C++ applications:
sudo apt-get install libnvinfer-dev libnvonnxparsers-dev libnvparsers-dev libnvinfer-plugin-dev
- For running TensorRT Python applications:
sudo apt-get install python-libnvinfer python3-libnvinfer
- For only running TensorRT C++ applications:
- When using the NVIDIA Machine Learning network repository, Ubuntu will by
default install TensorRT for the latest CUDA version. The following commands
will install libnvinfer6 for an older CUDA version and hold the
libnvinfer6 package at this version. Replace
6.x.x with your version of TensorRT and
cudax.x with your CUDA version for your install.
version="6.x.x-1+cudax.x" sudo apt-get install libnvinfer6=${version} libnvonnxparsers6=${version} libnvparsers6=${version} libnvinfer-plugin6=${version} libnvinfer-dev=${version} libnvonnxparsers-dev=${version} libnvparsers-dev=${version} libnvinfer-plugin-dev=${version} python-libnvinfer=${version} python3-libnvinfer=${version} sudo apt-mark hold libnvinfer6 libnvonnxparsers6 libnvparsers6 libnvinfer-plugin6 libnvinfer-dev libnvonnxparsers-dev libnvparsers-dev libnvinfer-plugin-dev python-libnvinfer python3-libnvinferIf you want to upgrade to the latest version of TensorRT or the latest version of CUDA, then you can unhold the libnvinfer6 package using the following command.sudo apt-mark unhold libnvinfer6 libnvonnxparsers6 libnvparsers6 libnvinfer-plugin6 libnvinfer-dev libnvonnxparsers-dev libnvparsers-dev libnvinfer-plugin-dev python-libnvinfer python3-libnvinfer
You may need to repeat these steps for libcudnn7 to prevent cuDNN from being updated to the latest CUDA version. Refer to the TensorRT 6.3.1 Release Notes in the DRIVE OS 5.2.0.0 SDK/PDK Release Notes for the specific version of cuDNN that was tested with your version of TensorRT. Example commands for downgrading and holding the cuDNN version can be found in the Safety Supported Samples And Tools section. Refer to the CUDA installation section of the DRIVE OS 5.2.0.0 SDK Development Guide.
If both the NVIDIA Machine Learning network repository and a TensorRT local repository are enabled at the same time you may observe package conflicts with either TensorRT or cuDNN. You will need to configure APT so that it prefers local packages over network packages. You can do this by creating a new file at /etc/apt/preferences.d/local-repo with the following lines:Package: * Pin: origin "" Pin-Priority: 1001
Note: This preference change will affect more than just TensorRT in the unlikely event that you have other repositories which are also not downloaded over HTTP(S). To revert APT to its original behavior simply remove the newly created file.
2.2. Additional Installation Methods
- TensorRT container
- The TensorRT container provides an easy method for deploying TensorRT with all necessary dependencies already packaged in the container. For information about installing TensorRT via a container, see the TensorRT Container Release Notes.
- JetPack
- JetPack bundles all Jetson platform software, including TensorRT. Use it to
flash your Jetson Developer Kit with the latest OS image, install NVIDIA
SDKs, and jump-start your development environment. For information about
installing TensorRT through JetPack, see the JetPack documentation.
For JetPack downloads, see Develop: Jetpack.
- NVIDIA DriveWorks
- With every release, TensorRT delivers features to make the DRIVE Development
Platform an excellent computing platform for Autonomous Driving. For more
information about installing TensorRTs through DriveWorks, see the DriveWorks documentation.
For DriveWorks downloads, see NVIDIA Developer: Drive Downloads.
3. Safety Supported Samples And Tools
| Title | TensorRT Sample Name | Description |
|---|---|---|
| dlaSafetyBuilder | dlaSafetyBuilder | A tool to generate the NvMedia DLA loadable from models without having to develop your own application. |
| dlaSafetyRuntime | dlaSafetyRuntime | A tool to load a DLA loadable and run inference using safety certified NvMedia DLA APIs. |
| "Hello World" For TensorRT Safety | sampleSafeMNIST | Consists of two parts; build and infer. The build part of this sample demonstrates how to use the builder flag for safety. The inference part of this sample demonstrates how to use the safe runtime, engine and execution context. |
3.1. Safety C++ Samples
3.2. “Hello World” For TensorRT Safety
What does this sample do?
This sample, sampleSafeMNIST, consists of two parts; build and infer. The build part of this sample demonstrates how to use the builder IBuilderConfig::setEngineCapability() flag for safety. The inference part of this sample demonstrates how to use the safe runtime, engine and execution context.
The build part builds a safe version of a TensorRT engine and saves it into a binary file, then the infer part loads the prebuilt safe engine and performs inference on an input image. The infer part uses the safety header proxy, with the CMakeLists.txt file demonstrating how to build it against the safety subset. This sample can be run in FP16 and INT8 modes.
- Perform the basic setup and initialization of TensorRT using the Caffe parser
- Import A Caffe Model Using The C++ Parser API
- Preprocess the input and store the result in a managed buffer
- Build An Engine In C++
- Serialize A Model In C++
- Perform Inference In C++
For step-by-step instructions, refer to the DRIVE OS 5.2.0.0 TensorRT 6.3.1 API Reference PDF in the DRIVE OS 5.2.0.0 SDK product package.
Where is this sample located?
This sample is maintained under the /usr/src/tensorrt/samples/sampleSafeMNIST directory. If using the Debian or RPM package, the sample is located at /usr/src/tensorrt/samples/sampleSafeMNIST. If using the tar or zip package, the sample is at <extracted_path>/samples/sampleSafeMNIST.
4. Cross Compiling Samples for AArch64 Users
4.1. Prerequisites
- Install the CUDA cross-platform toolkit for the corresponding target and set the
environment variable CUDA_INSTALL_DIR.
$ export CUDA_INSTALL_DIR="your cuda install dir"
Where CUDA_INSTALL_DIR is set to /usr/local/cuda by default.
- Install the cuDNN cross-platform libraries for the corresponding target and set the
environment variable CUDNN_INSTALL_DIR.
$ export CUDNN_INSTALL_DIR="your cudnn install dir"
Where CUDNN_INSTALL_DIR is set to CUDA_INSTALL_DIR by default.
- Install the TensorRT cross compilation Debian packages for the corresponding
target.
Note: If you are using the tar file release for the target platform, then you can safely skip this step. The tar file release already includes the cross compile libraries so no additional packages are required.
- AArch64 QNX
- libnvinfer-dev-cross-qnx
- AArch64 Linux
- libnvinfer-dev-cross-aarch64
4.2. Building Samples for AArch64 QNX
$ export QNX_HOST=/path/to/your/qnx/toolchains/host/linux/x86_64 $ export QNX_TARGET=/path/to/your/qnx/toolchain/target/qnx7
$ cd /path/to/TensorRT/samples $ make TARGET=qnx
4.3. Building Samples for AArch64 Linux
$ sudo apt-get install g++-aarch64-linux-gnu
$ cd /path/to/TensorRT/samples $ make TARGET=aarch64
5. Upgrading TensorRT
5.1. Ubuntu Users
6. Uninstalling TensorRT
- Uninstall libnvinfer6 which was installed using the Debian
package.
sudo apt-get purge "libnvinfer*"
- Uninstall uff-converter-tf and
graphsurgeon-tf, which were also installed using the Debian
package.
sudo apt-get purge graphsurgeon-tf
The uff-converter-tf will also be removed with the above command.
You can use the following command to uninstall uff-converter-tf and not remove graphsurgeon-tf, however, it is no longer required.sudo apt-get purge uff-converter-tf
You can later use autoremove to uninstall graphsurgeon-tf as well.sudo apt-get autoremove
- Uninstall the Python TensorRT wheel file.
If using Python 2.7:
sudo pip2 uninstall tensorrt
If using Python 3.x:sudo pip3 uninstall tensorrt
- Uninstall the Python UFF wheel file.
If using Python 2.7:
sudo pip2 uninstall uff
If using Python 3.x:sudo pip3 uninstall uff
- Uninstall the Python GraphSurgeon wheel file.
If using Python 2.7:
sudo pip2 uninstall graphsurgeon
If using Python 3.x:sudo pip3 uninstall graphsurgeon
7. Installing PyCUDA
- Maps all of CUDA into Python.
- Enables run-time code generation (RTCG) for flexible, fast, automatically tuned codes.
- Added robustness: automatic management of object lifetimes, automatic error checking
- Added convenience: comes with ready-made on-GPU linear algebra, reduction, scan.
- Add-on packages for FFT and LAPACK available.
- Fast. Near-zero wrapping overhead.
pip install 'pycuda>=2019.1.1'If you encounter any issues with PyCUDA usage after installing PyCUDA with the above command, you may need to recompile it yourself. For more information, see Installing PyCUDA on Linux.