Is this page helpful?

TensorRT-RTX API Capture and Replay#

TensorRT-RTX API Capture and Replay streamlines the process of reproducing and debugging issues within your applications. It allows you to record the engine-building phase of an application and later replay the engine-building steps, without needing to re-run the original application or access the model’s source code. [1]

This process is facilitated by two key components:

Capture Shim (libtensorrt_shim.so): This is a library that you can preload or drop into your application. It works by intercepting all TensorRT-RTX API calls made during the network-build phase. These intercepted calls, along with any associated constants, are then saved as a pair of files: a JSON file for the API calls and a BIN file for the constants.

Player (tensorrt_player): This is a standalone executable that takes the recorded JSON and BIN files generated by the Capture Shim and uses them to rebuild the TensorRT-RTX engine. This means you can recreate the engine that was built during the original application run, differing only in details related to timing differences during auto-tuning, aiding significantly in troubleshooting and debugging. Or, use it to recreate an engine-build failure.

Getting Started#

Important

The Capture and Replay feature is restricted to Linux in this release.

There are two ways to run the capture step:

LD_PRELOAD approach: simplest method, works for most applications.
Drop-in replacement approach: required for tensorrt_rtx and any application that dynamically loads TensorRT-RTX via dlopen/dlsym.

LD_PRELOAD (recommended)

Preload the Capture Shim into your application. If your application uses dlopen and dlsym to load the TensorRT-RTX library and map its C-functions (exposed by extern C) to the process address space, use the drop-in replacement approach instead.

export TRT_SHIM_NVINFER_LIB_NAME=<path to libtensorrt_rtx.so> # optional
export TRT_SHIM_OUTPUT_JSON_FILE=<output JSON path>
LD_PRELOAD=libtensorrt_shim.so <your-application-command-line>

Drop-in replacement

Replace the libtensorrt_rtx.so library being loaded by the app with the Capture Shim and redirect it to the original library. This approach is required for tensorrt_rtx and any application that dynamically loads TensorRT-RTX. Capturing the TensorRT-RTX Python API and Python ONNX parser does not require this approach. The Capture Shim (libtensorrt_shim.so) is installed alongside the TensorRT-RTX libraries.

mv <TRT-RTX-lib-path>/libtensorrt_rtx.so.<major> \
  <TRT-RTX-lib-path>/libtensorrt_rtx_orig.so.<major>

cp <TRT-RTX-lib-path>/libtensorrt_shim.so \
  <TRT-RTX-lib-path>/libtensorrt_rtx.so.<major>

TRT_SHIM_OUTPUT_JSON_FILE=<JSON-file-path> \
TRT_SHIM_NVINFER_LIB_NAME=<TRT-RTX-lib-path>/libtensorrt_rtx_orig.so.<major> \
<your-command>

Player

tensorrt_player -j <output JSON file> -o <output engine file>

Multi-Network Support#

TensorRT-RTX API Capture and Replay supports capturing multiple networks within a single process. This capability allows you to record the engine-building phase for applications that create and build multiple TensorRT-RTX networks, such as applications with ensemble models or multi-stage inference pipelines.

How It Works#

Each network created via create_network() (Python) / createNetworkV2 (C++) is assigned a unique network ID.
Objects (such as tensors and layers) are tracked per-network to ensure proper isolation.
Weights are stored with network-scoped identifiers, allowing the same weight pointer to be reused across different networks without conflicts.
The player automatically manages network context during replay.

Capture Example#

The following example demonstrates capturing multiple networks, including interleaved operations:

import tensorrt_rtx as trt

logger = trt.Logger(trt.Logger.WARNING)
builder = trt.Builder(logger)

# Create first network
network1 = builder.create_network()
input1 = network1.add_input("input1", trt.float32, (1, 3, 224, 224))
# ... add layers ...
network1.mark_output(output1)

# Create second network (interleaved operations are supported)
network2 = builder.create_network()
input2 = network2.add_input("input2", trt.float32, (1, 64))
# ... add layers ...
network2.mark_output(output2)

# Build both networks
config1 = builder.create_builder_config()
engine1 = builder.build_serialized_network(network1, config1)

config2 = builder.create_builder_config()
engine2 = builder.build_serialized_network(network2, config2)

Replay with Multiple Networks#

When replaying a multi-network capture, the player generates one engine file per network. The output files are named with indices appended:

tensorrt_player -j capture.json -o output.engine

This produces:

output.engine0 – Engine for the first network
output.engine1 – Engine for the second network, and so on.

Best Practices#

Use descriptive output file names when capturing multiple networks to help identify which capture session produced the files.
If you need to replay only specific networks, consider capturing them in separate processes or sessions.

Capture Tool Configuration#

Optional Environment Variables

Table 16 Optional Environment Variables#
Environment Variables	Description	Type	Default Value
`TRT_SHIM_OUTPUT_JSON_FILE`	Path to save the captured `.json` file. A corresponding `.bin` file will be created in the same directory.	`string`	`capture.json`
`TRT_SHIM_NVINFER_LIB_NAME`	Intercepted TensorRT-RTX library name. If unset, `dlopen(<lib_name>)` finds the library using its normal search rules (a full path is also allowed).	`string`	`libtensorrt_rtx.so`
`TRT_SHIM_DUMP_API`	Print `enter` and `exit` messages for every API function call.	`bool`	`false`
`TRT_SHIM_PRINT_WELCOME`	Print `Welcome to TensorRT Shim` at the start of the run.	`bool`	`false`
`TRT_SHIM_INLINE_WEIGHTS_LOWER_EQUAL_THAN`	Inline weights into the `.json` instead of a separate `.bin` when their size (in elements) is ≤ this threshold.	`int`	`8`
`TRT_SHIM_MARK_AS_RANDOM_WEIGHTS_GREATER_EQUAL_THAN`	Skip saving weights with an element count ≥ this threshold (they will be marked as random).	`int`	`Max int`
`TRT_SHIM_FLUSH_AFTER_EVERY_CALL`	Flush captured calls to the file after every API call instead of aggregating them. Useful for debugging crashes during engine building.	`bool`	`false`
`TRT_SHIM_FLUSH_ON_BUILD`	Flush captures calls to the file after every `buildSerializedNetwork` call. Useful for capturing incremental progress in multi-network sessions.	`bool`	`false`

Known Limitations#

Supports Linux x86_64. Windows is not supported in this release.
tensorrt_rtx --saveEngine flag is not supported.

Flush Behavior#

The captured data is flushed (written to disk) under any of the following conditions:

Process exit - flush occurs automatically when the process completes.
Every API call - flush after each captured call when TRT_SHIM_FLUSH_AFTER_EVERY_CALL is set.
On build - flush after each buildSerializedNetwork call when TRT_SHIM_FLUSH_ON_BUILD is set.

On each flush, the entire .json file is overwritten, while the .bin file is appended.

Next Steps#

C++ API Documentation: Build and run engines programmatically
Best Practices: Measure and optimize inference performance
Using the Native Runtime API: Run inference with the C++ or Python API

Footnotes