Is this page helpful?

Migrating Safety Runtime Code from TensorRT 10.x to 11.x#

This page describes how to update safety runtime code when you migrate from TensorRT 10.x to 11.x.

TensorRT 11.x changes the safe engine build and load workflow. Safe engines now use a matched companion .so that contains generated executable code required by the safe engine. Applications must preserve the safe engine and companion .so as a pair and pass the companion .so path when creating the safety graph.

TensorRT 11.x also removes frontend safety scope validation. In 10.x, the builder performed a static pre-build check against a “Minimal Safety Scope” allowlist before engine building began. In 11.x, this check is removed and the build-time compiler is the sole authority for safety scope enforcement. As a result, safety build errors now appear as tactic failures with layer-level traceback rather than pre-build scope rejections.

These changes affect runtime graph creation, builder API usage, safe engine artifact handling, builder flag usage, the behavior of isNetworkSupported(), error handling patterns, and trtexec command-line options. The kernel checker tool also adds MLIR validation in this release. Each section below pairs 10.x and 11.x C++ snippets where applicable.

Migrating Safe Engines to the Companion .so Workflow#

Important

This section is only applicable when using the TensorRT 11.3 and newer safety runtime, which is only available on NVIDIA DriveOS 7.2.6 and newer releases.

Each safe engine now has a matching companion shared object (.so) file. The companion .so contains generated executable host and device code required by the safe engine. Rebuild safe engines when you upgrade to TensorRT 11.3 and newer, and update build scripts, manifests, copy steps, CI artifacts, and deployment packages so that they preserve the safe engine and companion .so as a matched pair.

Runtime graph creation fails if the specified companion .so path is missing or is not an absolute path. Graph creation also fails if the specified .so file is invalid, contains a pairing token that does not match the safe engine, or cannot be loaded. In production QNX Safety deployments, the companion .so is executable content. Deploy it in a trusted location on the target according to the applicable DriveOS safety and security requirements.

Linking the companion shared library (.so) file at TensorRT safe engine build time requires a host C/C++ compiler and linker that match the target runtime platform. Use an x86-64 Linux toolchain for x86 and safety proxy builds, and the matching QNX SDP and qcc cross toolchain for QNX targets with matching license. For more information on installing the QNX SDP, refer to Configuring NVIDIA DriveOS QNX in the NVIDIA DriveOS QNX Installation Guide.

Updating createTRTGraph() Calls#

In the TensorRT 11.3 and newer safety runtime, createTRTGraph() requires an absolute companion .so path for every safe engine. Engines built by earlier releases that did not produce a companion .so must be rebuilt.

Updating createTRTGraph() Calls (Before / After / Summary)

Before (TensorRT 10.x)

1nvinfer2::safe::ITRTGraph* graph = nullptr;
2nvinfer2::safe::ErrorCode code = nvinfer2::safe::createTRTGraph(
3    graph,
4    engineBuffer,
5    engineBufferSize,
6    recorder,
7    /*trtManagedScratch*/ true,
8    allocator);

After (TensorRT 11.3 and newer)

 1nvinfer2::safe::ITRTGraph* graph = nullptr;
 2char const* companionSoPath = "/proc/boot/resnet50_safe.so";
 3nvinfer2::safe::ErrorCode code = nvinfer2::safe::createTRTGraph(
 4    graph,
 5    engineBuffer,
 6    engineBufferSize,
 7    companionSoPath,
 8    recorder,
 9    /*trtManagedScratch*/ true,
10    allocator);

Summary of Changes

  • createTRTGraph() requires companionSoPath for every safe engine.

  • The path must be absolute.

  • Pass the companion .so that was built for the safe engine you are loading.

  • Rebuild engines from earlier releases that did not produce a companion .so.

  • Handle graph-creation errors that come from companion .so load failures.

Updating trtexec Workflows#

Safe trtexec build and inference workflows now need both the safe engine and the companion .so. Use --saveEngineSo to name the companion .so written during build. Use --loadEngineSo to name the companion .so loaded during inference.

Updating trtexec Workflows (Before / After / Summary)

Before (TensorRT 10.x)

1trtexec \
2  --onnx=model.onnx \
3  --safe \
4  --saveEngine=resnet50_safe.engine \
5  --skipInference

After (TensorRT 11.3 and newer)

1trtexec \
2  --onnx=model.onnx \
3  --safe \
4  --saveEngine=resnet50_safe.engine \
5  --saveEngineSo=resnet50_safe.so \
6  --skipInference

Load the safe engine with trtexec:

1trtexec \
2  --safe \
3  --loadEngine=resnet50_safe.engine \
4  --loadEngineSo=resnet50_safe.so

Load the safe engine with trtexec_safe:

1trtexec_safe \
2  --loadEngine=resnet50_safe.engine \
3  --loadEngineSo=resnet50_safe.so

Summary of Changes

  • Safe builds now produce a safe engine and a companion .so.

  • --saveEngineSo=<file> writes the companion .so to <file> during a trtexec safe engine build. If omitted, trtexec writes the companion .so to <saveEngine>.so.

  • --loadEngineSo=<file> loads the companion .so from <file> during trtexec or trtexec_safe inference. If omitted, trtexec and trtexec_safe look for the companion .so at <loadEngine>.so.

  • The default appends .so to the full engine path. For example, resnet50_safe.engine defaults to resnet50_safe.engine.so.

  • trtexec requires --safe when you use --saveEngineSo or --loadEngineSo.

Updating Direct C++ Safe Engine Builds#

If your application builds safe engines directly with the TensorRT C++ builder API, update the build path to use buildSerializedSafeNetwork(). The ordinary serialized-network APIs cannot return the required companion .so and now return an error for safe engine builds.

Updating Direct C++ Safe Engine Builds (Before / After / Summary)

Before (TensorRT 10.x)

1auto serializedNetwork = SampleUniquePtr<nvinfer1::IHostMemory>(
2    builder->buildSerializedNetwork(*network, *config));

After (TensorRT 11.3 and newer)

1auto artifacts = SampleUniquePtr<nvinfer1::ISafeSerializedNetworkArtifacts>(
2    builder->buildSerializedSafeNetwork(*network, *config, /*emitKernelBlob=*/false));
3
4auto const* serializedNetwork = artifacts->getSerializedNetwork();
5auto const* companionSo = artifacts->getCompanionSo();

Summary of Changes

  • Use buildSerializedSafeNetwork() for safety builds.

  • Save both the serialized safe engine and the companion .so.

  • Treat the returned artifacts as a matched pair.

  • If your workflow needs kernel text, retrieve it from the returned artifacts.

Adapting to Build-Time Safety Scope Validation#

TensorRT 11.x removes pre-build safety scope validation and relies exclusively on build-time enforcement. In 10.x, setting BuilderFlag::kSAFETY_SCOPE triggered a pre-compile check against a static “Minimal Safety Scope” allowlist. In 11.x, this pre-build check is removed and the safety scope is now enforced entirely during engine building.

Update your code in three areas: builder configuration, build failure handling, and isNetworkSupported() usage.

BuilderFlag::kSAFETY_SCOPE is Deprecated#

BuilderFlag::kSAFETY_SCOPE is retained for API compatibility but has no effect in TensorRT 11.x. Setting it no longer triggers pre-build validation. You can leave existing calls in place or remove them; either way, the flag is silently ignored.

BuilderFlag::kSAFETY_SCOPE is Deprecated (Before / After / Summary)

Before (TensorRT 10.x)

1config->setFlag(BuilderFlag::kSAFETY_SCOPE);
2// Triggers static per-layer supportsSafety() checks before engine building.
3// Returns an error at network definition time for out-of-scope layers.
4auto serialized = builder->buildSerializedNetwork(*network, *config);

After (TensorRT 11.x)

1// kSAFETY_SCOPE is now a no-op. Remove it or leave it; it has no effect.
2// Safety validation occurs exclusively at build time.
3auto artifacts = builder->buildSerializedSafeNetwork(*network, *config, /*emitKernelBlob=*/false);

Summary of Changes

  • BuilderFlag::kSAFETY_SCOPE is deprecated and ignored.

  • Safety build failures now surface as build-time tactic errors (for example, No tactic available) with layer-level traceback, rather than pre-build scope errors.

isNetworkSupported() No Longer Validates Safety Scope#

In TensorRT 10.x, IBuilder::isNetworkSupported() performed static pre-build safety scope checks when kSAFETY_SCOPE was set. In 11.x, the function is simplified: it checks only architectural constraint violations (for example, hybrid DLA/GPU mode conflicts and kSAFETY_SCOPE flag combinations). A true return value does not guarantee a successful build.

isNetworkSupported() No Longer Validates Safety Scope (Before / After / Summary)

Before (TensorRT 10.x)

1// isNetworkSupported() checked static safety scope rules
2// and could be used as a fast pre-build safety gate.
3if (!builder->isNetworkSupported(*network, *config)) {
4    // Network was outside the static Minimal Safety Scope.
5    return false;
6}
7auto serialized = builder->buildSerializedNetwork(*network, *config);

After (TensorRT 11.x)

 1// isNetworkSupported() checks only architectural constraints (flag compatibility,
 2// hybrid DLA/GPU mode). It does NOT validate safety scope.
 3// Use buildSerializedSafeNetwork() for a definitive safety determination.
 4if (!builder->isNetworkSupported(*network, *config)) {
 5   // Network has an invalid configuration (e.g., incompatible builder flags).
 6   return false;
 7}
 8auto artifacts = builder->buildSerializedSafeNetwork(*network, *config, /*emitKernelBlob=*/false);
 9if (!artifacts) {
10   // Build failed; build-time safety checks rejected the network.
11   // Inspect error log for layer-level traceback.
12   return false;
13}

Summary of Changes

  • isNetworkSupported() no longer performs per-layer safety scope checks.

  • If you relied on isNetworkSupported() as a definitive safety gate, switch to buildSerializedSafeNetwork() for a complete build-time determination.

  • For safety engine builds in TensorRT 11.3 and newer, use buildSerializedSafeNetwork() rather than buildSerializedNetwork(). Refer to Updating Direct C++ Safe Engine Builds.

  • Tensor volume limit and boolean tensor checks have been removed from isNetworkSupported() (these are now validated at build-time).

Interpreting Build-Time Safety Build Failures#

Because all safety scope enforcement now occurs during engine building, build failures manifest as build-time errors rather than pre-build scope rejections. When a network is outside the certified scope, the builder reports a No tactic available error that traces back to the originating layer.

For safety engine builds in TensorRT 11.3 and newer, the same build-time errors surface from buildSerializedSafeNetwork(). Refer to Updating Direct C++ Safe Engine Builds.

Review your error handling code to process these build-time messages:

1auto artifacts = builder->buildSerializedSafeNetwork(*network, *config, /*emitKernelBlob=*/false);
2if (!artifacts) {
3   // Examine the error log for messages such as:
4   //   "Autotuner: no tactics to implement operation: ... layers=[ONNX Layer: <LayerName>]"
5   // Use this layer name to identify the unsupported operation and consult the
6   // TensorRT Safety Delta Document for known limitations relative to standard scope.
7}

Removed trtexec Flag --restricted#

Warning

The --restricted flag has been removed in TensorRT 11.x. Using it will cause trtexec to exit with an error.

The --restricted trtexec flag, which previously enabled pre-build safety scope validation during engine builds, has been removed. Safety restrictions can no longer be applied during standard engine building. Remove --restricted from any build scripts or CI pipelines that reference it.

Kernel Checker Tool Improvements#

Important

This section is only applicable when using the TensorRT 11.x safety runtime, which is only available on NVIDIA DriveOS 7.x.

The TensorRT kernel checker tool adds new checks to validate kernels in MLIR form, in addition to the existing checks that validate kernels in CUDA C++ form. Refer to the NVIDIA Deep Learning Inference SEooC 2.2 Safety Tools Manual V0.1 for more information.

Recommendation to Use Safety Proxy for Early Bring-Up#

For safety workflows, begin developing with the TensorRT safety proxy runtime on x86 for early bring-up rather than with the TensorRT standard runtime. The standard and safety runtime APIs differ significantly, as described in the Migrating Safety Runtime Code from TensorRT 8.x to 10.x section.

Deprecated trtexec_safe Flags and Replacements#

The following trtexec_safe flags have been deprecated in 11.x but are still accepted. Each entry shows the deprecated flag and its replacement.

--useCudaGraph

Enabled by default; flag accepted but has no effect. A deprecation warning is issued. Use --noCudaGraph to disable CUDA graph usage.

--separateProfileRun

Always enabled: flag accepted but has no effect. A deprecation warning is issued. This flag will be removed in a future release.

Adapting to Auxiliary CUDA Stream Support#

Important

This section is only applicable when using the TensorRT 11.x safety runtime, which is only available on NVIDIA DriveOS 7.x.

The TensorRT 11.x safety runtime now supports auxiliary CUDA streams, lifting the TensorRT 10.x restriction that forced every safety engine to execute on a single stream. By default, the builder may produce engines that require one or more auxiliary streams; when it does, the application is responsible for allocating, registering, and destroying those streams at runtime.

Warning

Existing TensorRT 10.x safety application code that loads a serialized engine and calls executeAsync() without first registering auxiliary streams may fail at runtime if the engine was built with maxAuxStreams > 0.

There are two migration paths:

  • Preserve 10.x single-stream behavior by setting maxAuxStreams to 0 at build time. No runtime code changes are required.

  • Adopt multi-stream execution by querying the engine’s auxiliary-stream count and registering streams before the first executeAsync() call.

Note

The standard (non-safety) runtime documents the equivalent APIs in the Within-Inference Multi-Streaming section. The key safety-specific differences are:

  • The application must allocate and register auxiliary streams; the safety runtime does not auto-create them.

  • The runtime APIs are on nvinfer2::safe::ITRTGraph (not IExecutionContext) and live in NvInferSafeRuntime.h.

Preserving 10.x Single-Stream Behavior#

When maxAuxStreams is omitted at build time, the builder may select a non-zero value through internal heuristics. To guarantee single-stream behavior across releases, set the flag explicitly to 0 at build time:

1auto config = builder->createBuilderConfig();
2config->setMaxAuxStreams(0);  // produces a single-stream engine
3// ... rest of build configuration ...

The same flag is available in trtexec:

trtexec --onnx=model.onnx --maxAuxStreams=0 ...

Adopting Auxiliary Streams#

If your workflow allows multi-stream engines, the aux-stream-specific steps to add to your runtime code are:

  1. Query the number of auxiliary streams the engine expects (after creating the safety graph from the serialized engine).

  2. Allocate that many CUDA streams with the cudaStreamNonBlocking flag.

  3. Register the streams through setAuxStreams() before the first call to executeAsync(). This is an INIT-phase operation (before any inference launches).

  4. Destroy the streams after the final sync() call.

If getNbAuxStreams() returns 0, the engine is single-stream and no auxiliary-stream handling is required. You can skip the allocation and registration steps.

The relevant API lives in namespace nvinfer2::safe in NvInferSafeRuntime.h. Pass the companion .so path required by TensorRT 11.3 and newer. Refer to Updating createTRTGraph() Calls.

 1   // 1. Create the safety graph from the serialized engine.
 2   nvinfer2::safe::ITRTGraph* graph = nullptr;
 3   char const* companionSoPath = "/path/to/engine.so";
 4   nvinfer2::safe::createTRTGraph(graph, planData, planSize, companionSoPath, recorder, true);
 5
 6   // 2. Query the number of auxiliary streams this engine requires.
 7   int32_t nbAuxStreams = 0;
 8   graph->getNbAuxStreams(nbAuxStreams);
 9
10   // 3. Allocate that many non-blocking streams. The application owns
11   //    the lifetime of these streams.
12   std::vector<cudaStream_t> auxStreams(nbAuxStreams);
13   for (int32_t i = 0; i < nbAuxStreams; ++i)
14   {
15      cudaStreamCreateWithFlags(&auxStreams[i], cudaStreamNonBlocking);
16   }
17
18   // 4. Register the streams BEFORE the first executeAsync call.
19   //    The count must exactly match getNbAuxStreams().
20   graph->setAuxStreams(auxStreams.data(), nbAuxStreams);
21
22   // 5. Create the main inference stream.
23   cudaStream_t mainStream;
24   cudaStreamCreateWithFlags(&mainStream, cudaStreamNonBlocking);
25
26   // 6. Run inference.
27   graph->executeAsync(mainStream);
28
29   // 7. Wait for the full inference to complete.
30   graph->sync();
31
32   // 8. Cleanup: must occur AFTER the final sync().
33   for (auto s : auxStreams)
34   {
35      cudaStreamDestroy(s);
36   }
37   cudaStreamDestroy(mainStream);
38   nvinfer2::safe::destroyTRTGraph(graph);

Using Debug Overlay Filesystem for Safe Engine Building on QNX Safety#

Starting in TensorRT 11.0.1, components of the TensorRT safety builder are shipped in the debug overlay filesystem on QNX Safety. Specifically, the timing_server executable and certain dependent libraries are mounted in the debug overlay filesystem in the /nvidia_overlay directory rather than /usr/libnvidia, where they previously shipped. Other platforms, such as QNX Standard and x86, are unchanged.

To mount the debug overlay, specify debug_sr as the platform configuration table (PCT) argument to bind_partitions, as shown in the TensorRT 11.0.1 Installation Guide for DriveOS 7.2.5:

./make/bind_partitions -b <board_variant> qnx -p debug_sr

The debug overlay is not needed for production runtime. The libnvinfer_safe.so* libraries are still available at /usr/libnvidia.

Similarly, the --remoteAutoTuningConfig flag to trtexec should refer to /nvidia_overlay:

--remoteAutoTuningConfig="ssh://root:root@${TARGET_IP}:22?remote_exec_path=/nvidia_overlay/timing_server&remote_lib_path=/nvidia_overlay:/usr/libnvidia&dump_remote_stdout=on&dump_remote_stderr=on&verbose=on"