Is this page helpful?

TensorRT-RTX 1.7 Release Notes#

These Release Notes apply to Linux and Windows users. This release includes several fixes from the previous TensorRT-RTX releases and additional changes.

Key Features and Enhancements#

  • Reduced build-time memory usage with weight placeholders: TensorRT-RTX 1.7 supports null weight placeholders in the weight-stripped engine build workflow. A placeholder declares only the data type and element count of the weights it stands in for, so the builder avoids loading the weight payload and reduces its internal build-time allocations. Refer to Reducing Build-Time Memory Usage with Weight Placeholders.

  • Weights offloading: TensorRT-RTX 1.7 can unload an inactive engine’s weight memory so that another engine can use the released GPU memory, then restore those weights later without rebuilding or refitting the engine. Restoration based on CUDA virtual memory maps application-provided GPU memory directly into the engine’s weight address range, so no extra copy is needed. Refer to Offloading and Restoring Engine Weights.

  • Improved kernel performance and memory efficiency: TensorRT-RTX 1.7 includes kernel and fusion optimizations across a range of AI workloads. FP8 and INT8 weight-only-quantized convolutions benefit from smaller engines and lower peak GPU memory usage. Reduced CPU processing time also enables faster engine refitting for some workloads.

  • Edge-LLM integration: TensorRT-RTX 1.7 is a supported backend for Edge-LLM on x86-64 Linux and Windows, including workloads that use speculative decoding.

  • Improved JIT performance and host-memory efficiency: TensorRT-RTX 1.7 reduces host-side latency when creating execution contexts during just-in-time (JIT) compilation and when serializing and deserializing the runtime cache. Execution contexts and runtime caches also use less host memory.

Compatibility#

For comprehensive platform compatibility, hardware requirements, and feature availability information, refer to the TensorRT-RTX Support Matrix. The support matrix provides detailed information about supported operating systems, CUDA versions, GPU architectures, precision modes, compiler requirements, and ONNX operator support for TensorRT-RTX 1.7.

Limitations#

Warning

  • Timing cache deprecated: Using ITimingCache forces a GPU query during ahead-of-time compilation and has no effect on the built engine. The APIs were deprecated in 1.2 and will be removed in a future update.

  • Engines are not forward-compatible: TensorRT-RTX engines must be rebuilt when upgrading the TensorRT-RTX runtime library.

  • Using a timing cache through ITimingCache and related APIs forces the ahead-of-time compilation step to query your system for a GPU (that is, prevents use of CPU-only ahead-of-time compilation). This has been true since the initial release (version 1.0). The timing cache has no effect on the built engine in TensorRT-RTX. The timing cache APIs were deprecated in 1.2 and will be removed in a future update. They remain in place to avoid breaking source or binary compatibility. Applications should stop using timing cache APIs in preparation for their removal.

  • If the cache size grows too large (larger than 100 MB), it might require more overhead to serialize and deserialize from disk. If it negatively affects performance, delete the cache file and recreate one.

  • TensorRT-RTX engines are not forward-compatible with other versions of the TensorRT-RTX runtime. Ensure that any TensorRT-RTX engines you produce are run using the runtime from the same version of TensorRT-RTX that was used to generate the engine.

  • While TensorRT-RTX supports Turing (CUDA compute capability 7.5), a TensorRT-RTX engine created with default Compute Capability settings will produce an engine with support for the NVIDIA Ampere architecture and later GPUs, therefore excluding Turing. It is recommended to build a separate engine specifically for Turing to achieve the best performance. Creating a single engine with support for Turing and later GPUs will lead to less performant inference on the NVIDIA Ampere architecture and later GPUs due to technical limitations of the engine format.

Deprecated API Lifetime#

  • APIs deprecated in TensorRT-RTX 1.7 will be retained until 10/2027.

  • APIs deprecated in TensorRT-RTX 1.6 will be retained until 7/2027.

  • APIs deprecated in TensorRT-RTX 1.5 will be retained until 5/2027.

  • APIs deprecated in TensorRT-RTX 1.4 will be retained until 3/2027.

  • APIs deprecated in TensorRT-RTX 1.3 will be retained until 12/2026.

  • APIs deprecated in TensorRT-RTX 1.2 will be retained until 10/2026.

Fixed Issues#

  • Removed the following internal symbols on Windows. They were exported by tensorrt_rtx_1_6.dll and earlier, were never intended for application use, and are no longer present in tensorrt_rtx_1_7.dll. Applications that referenced them must stop doing so:

    • ?disableInternalBuildFlags@nvinfer1@@YAXAEAVINetworkDefinition@1@_K@Z

    • ?enableInternalBuildFlags@nvinfer1@@YAXAEAVINetworkDefinition@1@_K@Z

    • ?getInternalBuildFlags@nvinfer1@@YA_KAEBVINetworkDefinition@1@@Z

    • ?setDebugOutput@nvinfer1@@YAXAEAVIExecutionContext@1@PEAVIDebugOutput@1@@Z

    • ?setInternalBuildFlags@nvinfer1@@YAXAEAVINetworkDefinition@1@_K@Z

    • nvinfer1DisableInternalBuildFlags

    • nvinfer1EnableInternalBuildFlags

    • nvinfer1GetInternalBuildFlags

    • nvinfer1SetInternalBuildFlags

  • Fixed a performance issue on Windows that affected models containing certain concatenation followed by convolution patterns.

  • Fixed an issue that could cause CMake to attempt to import a non-existent file in projects that use find_package(TensorRT-RTX). The issue affected Windows. (Reported in GitHub issue #4851.)

  • Fixed an issue that could cause CMake to report a missing file cmake/NvInferVersion.h in projects that use find_package(TensorRT-RTX). The issue affected the Windows and Linux packages. (Reported in GitHub issue #40.)

  • Fixed an issue that could cause an error to appear for certain supported INT8 convolutions.

  • Decreased excess CPU usage when refitting TensorRT-RTX engines.

  • Decreased engine size and peak process VRAM for FP8 and INT8 weight-only-quantization (WoQ) convolutions.

  • Improved the performance of certain 1D convolution layers on the Turing architecture.

  • Fixed an accuracy issue that could occur when using causal masking with the Attention APIs.

  • Fixed numerical overflow and underflow in ONNX ReduceLogSumExp, and overflow in FP16 LpPool, GlobalLpPool, LpNormalization, and ReduceLogSum, which could produce infinite or inaccurate outputs even when the expected results were within range.

  • Improved the error messaging for unsupported graphs on the Turing architecture.

  • Fixed a concurrency issue that could occur rarely when using multiple TensorRT-RTX engines in a single CUDA context.

  • Fixed an issue that could cause inference to hang on models with strided deconvolution layers for specific input shapes. The issue affected GPUs with compute capability 8.0 and later. (Reported in GitHub issue #37.)

  • Improved the fusion patterns for fused multi-head attention (FMHA).

  • Improved the error message when an engine runs on a GPU whose compute capability the built engine does not target, for example on a multi-GPU machine with an engine built for --computeCapabilities=120 but run on an RTX 40-series GPU.

  • Added support for ONNX Squeeze with omitted axes on inputs with dynamic shapes. Dimensions known to be 1 in the network definition are removed; remaining dimensions must not become 1 at runtime. This resolves the parser error Cannot infer squeeze dimensions from a dynamic shape! for supported inputs.

  • Improved the performance of certain 1D deconvolution layers.

  • Fixed an issue that could cause the engine build to fail on Windows with an illegal memory access error when a network was refittable and contained a loop.

  • Improved the GEMV heuristics.

  • Reduced the engine build time for all model architectures across Linux and Windows.

  • Fixed a performance issue on layers containing a convolution followed by constant elementwise multiplication or division operations. The issue affected all GPU architectures.

  • Fixed a performance issue on layers containing a chain of pointwise operations followed by a convolution. The issue affected all GPU architectures.

Known Issues#

Functional

  • NonMaxSuppression, NonZero, and Multinomial layers are not supported.

  • Only the WDDM driver for Windows is supported. The TCC driver for Windows (refer to Tesla Compute Cluster (TCC)) is unsupported and might fail with the following error.

    [E] Error[1]: [defaultAllocator.cpp::nvinfer1::internal::DefaultAllocator::allocateAsync::48] Error Code 1: Cuda Runtime (operation not supported)

    For instructions on changing the driver mode, refer to the NVIDIA Nsight Visual Studio Edition documentation.

  • By default, TensorRT-RTX uses cudaMallocAsync() for GPU memory allocation. In some environments, such as CUDA in Graphics (CiG) mode or unified-memory systems such as NVIDIA RTX Spark, allocations can fail even when physical GPU memory remains available. Upgrading to the latest CUDA driver can reduce the likelihood of this issue. To avoid these allocator-specific failures, provide a custom IGpuAllocator implementation backed by cudaMalloc(). Refer to the synchronous GPU allocator sample for an example. This workaround can slightly reduce performance and disables CUDA Graph support.

  • Convolutions and deconvolutions with both non-unit strides and dilations are not supported for all precisions. Non-unit strided convolutions and deconvolutions, and non-unit dilated convolutions and deconvolutions are supported.

  • On Windows with CUDA driver R615 or earlier, TensorRT-RTX inference can fail with CUDA_ERROR_INVALID_VALUE when it is launched after a DirectML RecordDispatch call in the same process. To work around this, limit the usable kernel shared memory in the builder configuration.

    config->setMemoryPoolLimit(nvinfer1::MemoryPoolType::kTACTIC_SHARED_MEMORY, 99 * 1024);

Performance

  • Use of the CPU-only ahead-of-time (AOT) feature can lead to reduced performance for some models. Affected applications achieve the best performance if they instead perform ahead-of-time compilation on-device, targeted to the specific end-user machine. This can be done with the --useGPU flag for the tensorrt_rtx binary, or when using the APIs by setting the compute capabilities to contain only kCURRENT using IBuilderConfig::setComputeCapability(). Measure performance with both approaches to determine which is best for your application. This performance discrepancy is targeted for resolution in a future release.

  • Some models might experience additional microsecond-scale launch overhead when CUDA Graphs are enabled.