CPU-Only AOT and TensorRT-RTX Engines

The CPU-only ahead-of-time (AOT) feature is enabled by default. When you build an engine, TensorRT-RTX does not require a GPU device to be present. The engine is portable to both Windows and Linux operating systems with NVIDIA RTX GPUs that have compute capability 8.6 or later (NVIDIA Ampere architecture and later).

All model weights are stored inside the engine by default. NVIDIA Turing devices are not supported without further configuration. This section provides details about the Compute Capability API and instructions for building a weightless engine for deployment with minimal storage footprint.

Table 17 CPU AOT engine types and when to use them

Engine type	GPU at AOT build	Weights in serialized file	When to use
Default CPU-only AOT	Not required	Full weights embedded	Standard NVIDIA RTX deployment; portable engine for the NVIDIA Ampere architecture and later (8.6+) by default
Explicit compute capability	Not required (unless kCURRENT)	Full weights embedded	Target Turing (7.5) or lock to specific CC list; refer to the Support Matrix for precision limits
Weightless + refit (kSTRIP_PLAN + kREFIT)	Required; at most one CC	Stripped; refit at first launch	Minimal installer footprint; distribute small engine and restore weights on the client

Compute Capability

Compute capability defines the hardware features and supported instructions for each NVIDIA GPU architecture. For information regarding the compute capability of your GPU, refer to NVIDIA CUDA GPU Compute Capability.

By default, an engine built by TensorRT-RTX can be run on GPU devices with compute capability 8.6, 8.9, 12.0, 12.1, and later. To build engines for Turing devices, a compute capability of 7.5 needs to be specified through the API.

Specifying Target Compute Capabilities

You can specify one or more compute capabilities through IBuilderConfig. The following example shows how to set a single target compute capability of 7.5 for Turing RTX devices.

C++

IBuilderConfig* config = builder->createBuilderConfig();
config->setNbComputeCapabilities(1);
config->setComputeCapability(ComputeCapability::kSM75, 0);

Python

import tensorrt_rtx as trt

builder_config = builder.create_builder_config()
builder_config.num_compute_capabilities = 1
builder_config.set_compute_capability(trt.ComputeCapability.SM75, 0)

Table 18 Compute capability configuration options

Configuration	AOT build behavior	When to use
Default (no CC set)	CPU-only AOT; engine runs on CC 8.6, 8.9, 12.0, 12.1, and later	Most RTX apps without Turing-only requirements
Explicit CC list (setComputeCapability / --computeCapabilities)	Engine limited to listed architectures	Turing (7.5) support or multi-arch targeting with known trade-offs
`ComputeCapability::kCURRENT` / --useGpu	Requires GPU; compiles for the device in the build environment	Development tuning on a specific GPU; must be the only CC target

You can use the ComputeCapability::kCURRENT flag to turn off CPU-only AOT and use the current GPU device in the environment as the target to compile an engine. ComputeCapability::kCURRENT is supported only when it is the sole target compute capability.

When you set one or multiple compute capabilities, the engine you build can only run on specified target devices.

Warning

If you specify Turing (compute capability 7.5) as one of multiple targets, the resulting engine is not guaranteed to be performant on the NVIDIA Ampere architecture or later devices. Build separate engines per architecture for best performance.

The tensorrt_rtx tool provides two ways to set this from the command line. Use --useGpu for the equivalent of ComputeCapability::kCURRENT, or --computeCapabilities=<list_of_CCs> to pass one or more compute capabilities.

Compile Models with Hardware-Specific Data Types

When hardware-specific data types appear in the network definition, TensorRT-RTX will either emit a warning message or an error based on how the API is called by users. For per-architecture precision support, refer to the Support Matrix. If you are porting a model that uses reduced-precision types, also refer to Working with Quantized Types.

Commonly used hardware-specific data types include FP4, FP8, BF16, and INT4. FP16, FP32, and other integer types are supported on all supported hardware.

The behavior of TensorRT-RTX depends on whether one or more compute capabilities are specified:

Table 19 Hardware-specific data type behavior by compute capability configuration

CC configuration	Default precision behavior	Failure mode
Default (none specified)	FP8 models: NVIDIA Ada Lovelace and later (8.9+). FP4 models: NVIDIA Blackwell architecture and later (12.0+). All other models: NVIDIA Ampere architecture and later (8.6+).	Runtime failure on hardware below the implied minimum
One or more CCs specified	Engine is limited to the specified compute capabilities	Build error recorded by IErrorRecorder if the model uses a data type unsupported by any specified CC

Weightless Engines

An engine without weights helps minimize the storage footprint for deployment. Weight-stripping build configuration can be enabled, so that TensorRT-RTX enables refit only for constant weights that do not impact the builder’s ability to optimize and produce an engine with the same runtime performance as a non-refittable engine. Those weights are then omitted from the serialized engine, resulting in a small engine file that can be refitted at runtime using custom weights or weights from the ONNX model.

Note

When weight-stripping build configuration is enabled, a GPU device is required for the AOT build and at most one compute capability can be set through the builder config.

Building a Weightless Engine

The tensorrt_rtx tool provides flags --stripWeights and --refit to enable the weight-stripping build configuration. The corresponding builder flags are kSTRIP_PLAN and kREFIT.

C++

...
config->setFlag(BuilderFlag::kSTRIP_PLAN);
config->setFlag(BuilderFlag::kREFIT);
builder->buildSerializedNetwork(*network, *config);

Python

...
config.flags |= 1 << int(trt.BuilderFlag.STRIP_PLAN)
config.flags |= 1 << int(trt.BuilderFlag.REFIT)
builder.build_serialized_network(network, config)

After the engine is built, save the engine file and distribute it to the installer.

Refitting a Weightless Engine

On the client side, when launching the network for the first time, the user can refit all the weights back to the engine. Since all the weights in the engine were removed, each weight needs to be updated one by one. After all weights are updated, save the full TensorRT-RTX engine file to be used in the application for future inference.

C++

IRefitter* refitter = createInferRefitter(*engine, gLogger);
int32_t const n = refitter->getAllWeights(0, nullptr);
for (int32_t i = 0; i < n; ++i) {
    refitter->setNamedWeights(weightsNames[i], Weights{...});
}
auto serializationConfig = SampleUniquePtr<nvinfer1::ISerializationConfig>(cudaEngine->createSerializationConfig());
auto serializationFlag = serializationConfig->getFlags();
serializationFlag &= ~(1<<static_cast<uint32_t>(nvinfer1::SerializationFlag::kEXCLUDE_WEIGHTS));
serializationConfig->setFlags(serializationFlag);
// hostMemory will contain the full engine
auto hostMemory = SampleUniquePtr<nvinfer1::IHostMemory>(cudaEngine->serializeWithConfig(*serializationConfig));

Python

refitter = trt.Refitter(engine, TRT_LOGGER)
all_weights = refitter.get_all()
for name in all_weights:
    refitter.set_named_weights(name, weights[name])
serialization_config = engine.create_serialization_config()
serialization_config.flags &= ~(1 << int(trt.SerializationFlag.EXCLUDE_WEIGHTS))
binary = engine.serialize_with_config(serialization_config)

Refitting a Weightless Engine Directly with ONNX Models

When working with weight-stripped engines created from ONNX models, the refit process can be done automatically with the IParserRefitter class from the ONNX parser library. The following code shows how to create the class and run the refit process.

C++

IRefitter* refitter = createInferRefitter(*engine, gLogger);
IParserRefitter* parserRefitter = createParserRefitter(*refitter, gLogger);
bool const parsed = parserRefitter->refitFromFile("path_to_onnx_model");
bool const refitSuccess = parsed && refitter->refitCudaEngine();

Python

refitter = trt.Refitter(engine, TRT_LOGGER)
parser_refitter = trt.OnnxParserRefitter(refitter, TRT_LOGGER)
result = parser_refitter.refit_from_file("path_to_onnx_model")
refit_success = refitter.refit_cuda_engine()

Reducing Build-Time Memory Usage with Weight Placeholders

When you build a weight-stripped engine with fine-grained refit, you can pass null weight placeholders in place of the real weight data. A placeholder declares only the data type and element count, so the builder never holds the full weights in host or device memory, which lowers peak memory use during the build. You supply the real weights when you refit, so an engine built this way must be refitted before inference.

The workflow requires both BuilderFlag::kSTRIP_PLAN and BuilderFlag::kREFIT_INDIVIDUAL, and each placeholder must be named with the canonical Weights descriptor that TensorRT-RTX returns. For the placeholder rules, the weight types that accept them, and C++ and Python examples, refer to Reducing Build-Time Memory Usage with Weight Placeholders in the NVIDIA TensorRT documentation.

Offloading and Restoring Engine Weights

After you refit a weightless engine, use IWeightsManager to save its loaded weights, release their physical GPU memory, and restore them later without rebuilding or refitting the engine. An inactive engine can therefore give up its weight memory so that another engine can use it. The saved weight data is opaque and must be preserved unchanged.

Create the manager from the engine with createWeightsManager(). The engine cannot run inference between the point where you unload its weights and the point where its complete weight range is restored.

Setup for the Examples in This Section

The save, unload, and restore examples that follow are successive steps in one workflow rather than independent snippets. They assume engine is the refitted engine produced earlier on this page, and they reuse the single CUDA stream created here. Establish this setup before you run any of them.

The Python examples call the CUDA driver and runtime APIs through the CUDA Python bindings. Install them before you run the examples:

python3 -m pip install cuda-python

C++

#include <cuda.h>
#include <cuda_runtime.h>
#include <fstream>
#include <memory>
#include <stdexcept>
#include <vector>

#include "NvInfer.h"

cudaStream_t stream{};
if (cudaStreamCreate(&stream) != cudaSuccess)
{
    throw std::runtime_error("Failed to create a CUDA stream");
}

Python

import ctypes

from cuda.bindings import driver as cuda
from cuda.bindings import runtime as cudart

import tensorrt_rtx as trt

_, stream = cudart.cudaStreamCreate()

Note

The cuMem* calls in Restoring the Weights belong to the CUDA driver API, which is why the C++ setup includes cuda.h in addition to cuda_runtime.h, and why the Python setup imports cuda.bindings.driver in addition to cuda.bindings.runtime.

Saving the Weights

To save the weights directly to storage, implement IStreamWriter and pass it to saveToStream(). TensorRT-RTX writes exactly getSize() bytes and synchronizes the CUDA stream before it passes each chunk to the writer.

C++

class FileWriter final : public nvinfer1::IStreamWriter
{
public:
    explicit FileWriter(char const* path)
        : mFile(path, std::ios::binary)
    {
    }

    int64_t write(void const* data, int64_t size) override
    {
        mFile.write(static_cast<char const*>(data), size);
        return mFile ? size : -1;
    }

private:
    std::ofstream mFile;
};

std::unique_ptr<nvinfer1::IWeightsManager> weightsManager{
    engine->createWeightsManager()};

FileWriter writer{"engine.weights"};
if (!weightsManager->saveToStream(writer, stream))
{
    throw std::runtime_error("Failed to save engine weights");
}

Python

class FileWriter(trt.IStreamWriter):
    def __init__(self, path):
        trt.IStreamWriter.__init__(self)
        self.file = open(path, "wb")

    def write(self, data: bytes) -> int:
        return self.file.write(data)


weights_manager = engine.create_weights_manager()
writer = FileWriter("engine.weights")
try:
    if not weights_manager.save_to_stream(writer, stream):
        raise RuntimeError("Failed to save engine weights")
finally:
    writer.file.close()

To keep the saved weights in application memory instead, allocate a host or device buffer of exactly getSize() bytes and pass it to saveToBuffer(). Keep the destination valid until the CUDA stream completes.

C++

int64_t const weightsSize = weightsManager->getSize();
std::vector<std::byte> weightsBackup(weightsSize);

if (!weightsManager->saveToBuffer(
        weightsBackup.data(),
        weightsSize,
        nvinfer1::TensorLocation::kHOST,
        stream))
{
    throw std::runtime_error("Failed to save engine weights");
}
cudaStreamSynchronize(stream);

Python

weights_size = weights_manager.get_size()
weights_backup = ctypes.create_string_buffer(weights_size)

if not weights_manager.save_to_buffer(
    ctypes.addressof(weights_backup),
    weights_size,
    trt.TensorLocation.HOST,
    stream,
):
    raise RuntimeError("Failed to save engine weights")
cudart.cudaStreamSynchronize(stream)

Unloading the Weights

After you save the weights, call unload() to release their physical GPU memory while preserving the engine’s virtual weight address range. Order all earlier engine work on the supplied CUDA stream.

C++

if (!weightsManager->unload(stream))
{
    throw std::runtime_error("Failed to unload engine weights");
}

Python

if not weights_manager.unload(stream):
    raise RuntimeError("Failed to unload engine weights")

Restoring the Weights

To restore the weights with CUDA virtual memory, first create a physical GPU allocation with cuMemCreate(). Temporarily map the allocation into the application’s address space, copy the saved weights into it, and then remove the temporary mapping.

Note

cuMemCreate() and cuMemMap() require a size that is a multiple of the minimum CUDA virtual memory allocation granularity, so round the saved weight size up before you allocate. Copy and restore only the getSize() bytes that TensorRT-RTX saved.

C++

int device{};
cudaGetDevice(&device);

CUmemAllocationProp properties{};
properties.type = CU_MEM_ALLOCATION_TYPE_PINNED;
properties.location.type = CU_MEM_LOCATION_TYPE_DEVICE;
properties.location.id = device;
properties.requestedHandleTypes = CU_MEM_HANDLE_TYPE_NONE;

size_t granularity{};
cuMemGetAllocationGranularity(
    &granularity, &properties, CU_MEM_ALLOC_GRANULARITY_MINIMUM);

size_t const mappedSize
    = ((weightsSize + granularity - 1) / granularity) * granularity;

CUmemGenericAllocationHandle allocation{};
cuMemCreate(&allocation, mappedSize, &properties, 0);

CUdeviceptr temporaryAddress{};
cuMemAddressReserve(
    &temporaryAddress, mappedSize, granularity, 0, 0);
cuMemMap(
    temporaryAddress, mappedSize, 0, allocation, 0);

CUmemAccessDesc access{};
access.location = properties.location;
access.flags = CU_MEM_ACCESS_FLAGS_PROT_READWRITE;
cuMemSetAccess(temporaryAddress, mappedSize, &access, 1);

cudaMemcpyAsync(
    reinterpret_cast<void*>(temporaryAddress),
    weightsBackup.data(),
    weightsSize,
    cudaMemcpyHostToDevice,
    stream);
cudaStreamSynchronize(stream);

cuMemUnmap(temporaryAddress, mappedSize);
cuMemAddressFree(temporaryAddress, mappedSize);

Python

_, device = cudart.cudaGetDevice()

properties = cuda.CUmemAllocationProp()
properties.type = cuda.CUmemAllocationType.CU_MEM_ALLOCATION_TYPE_PINNED
properties.location.type = cuda.CUmemLocationType.CU_MEM_LOCATION_TYPE_DEVICE
properties.location.id = device
properties.requestedHandleTypes = (
    cuda.CUmemAllocationHandleType.CU_MEM_HANDLE_TYPE_NONE
)

_, granularity = cuda.cuMemGetAllocationGranularity(
    properties,
    cuda.CUmemAllocationGranularity_flags.CU_MEM_ALLOC_GRANULARITY_MINIMUM,
)

mapped_size = (
    (weights_size + granularity - 1) // granularity
) * granularity

_, allocation = cuda.cuMemCreate(mapped_size, properties, 0)
_, temporary_address = cuda.cuMemAddressReserve(
    mapped_size, granularity, cuda.CUdeviceptr(0), 0
)
cuda.cuMemMap(temporary_address, mapped_size, 0, allocation, 0)

access = cuda.CUmemAccessDesc()
access.location = properties.location
access.flags = cuda.CUmemAccess_flags.CU_MEM_ACCESS_FLAGS_PROT_READWRITE
cuda.cuMemSetAccess(temporary_address, mapped_size, [access], 1)

cudart.cudaMemcpyAsync(
    int(temporary_address),
    ctypes.addressof(weights_backup),
    weights_size,
    cudart.cudaMemcpyKind.cudaMemcpyHostToDevice,
    stream,
)
cudart.cudaStreamSynchronize(stream)

cuda.cuMemUnmap(temporary_address, mapped_size)
cuda.cuMemAddressFree(temporary_address, mapped_size)

To restore from an application buffer, call restoreFromBuffer(). TensorRT-RTX allocates the physical GPU memory and copies the saved data into the engine’s reserved virtual address range. Keep the source buffer valid until the stream operation completes.

C++

if (!weightsManager->restoreFromBuffer(
        weightsBackup.data(),
        weightsSize,
        nvinfer1::TensorLocation::kHOST,
        stream))
{
    throw std::runtime_error("Failed to restore engine weights");
}

Python

if not weights_manager.restore_from_buffer(
    ctypes.addressof(weights_backup),
    weights_size,
    trt.TensorLocation.HOST,
    stream,
):
    raise RuntimeError("Failed to restore engine weights")

As an alternative to copying from a buffer, call restoreFromVmmAllocation() to map the initialized physical allocation directly into the engine’s virtual weight address range. Mapping does not copy the weight data. Retain the allocation handle if the physical memory will be reused after the weights are unloaded again.

C++

if (!weightsManager->restoreFromVmmAllocation(
        static_cast<uint64_t>(allocation),
        /* offset */ 0,
        weightsSize,
        stream))
{
    throw std::runtime_error("Failed to restore engine weights");
}

Python

if not weights_manager.restore_from_vmm_allocation(
    int(allocation),
    0,
    weights_size,
    stream,
):
    raise RuntimeError("Failed to restore engine weights")

Deferred Weights Loading

Use deferred weights loading to deserialize a TensorRT-RTX engine without allocating the engine weights in host or GPU memory. During execution context creation, TensorRT-RTX just-in-time (JIT) compiles the required kernels from the metadata in the weightless engine. This behavior lowers peak host and GPU memory use during deserialization. It is useful when an application deserializes several engines before running any of them, such as a multi-model application that keeps many engines resident.

Warning

Deferred weights loading and weight streaming are mutually exclusive in the current release. Enabling both features results in an API usage error.

Because JIT compilation runs during createExecutionContext(), you can optionally attach an IRuntimeCache to the runtime configuration and serialize the compiled kernels to disk. This approach lets multiple models populate runtime caches in parallel without loading weights into host or GPU memory. The optional cache steps are marked in the following example. Keep the IRuntimeConfig and IRuntimeCache alive while contexts created from them are in use. For more information, refer to Working with Runtime Cache.

C++

IRuntime* runtime = createInferRuntime(gLogger);
runtime->setDeferredWeightsLoading(true);
ICudaEngine* engine = runtime->deserializeCudaEngine(blob, size);

// [Optional] Attach a runtime cache to save the JIT-compiled
// kernels without loading weights. The config and cache must
// outlive contexts created from them.
IRuntimeConfig* config = engine->createRuntimeConfig();
IRuntimeCache* cache = config->createRuntimeCache();
config->setRuntimeCache(*cache);

IExecutionContext* context = engine->createExecutionContext(config);

// [Optional] Serialize the runtime cache into host memory for
// saving to disk.
IHostMemory* hostMemory = cache->serialize();

// Before the first inference, load the weights into GPU memory
// from the same plan blob.
engine->loadWeights(blob, size);
context->enqueueV3(stream);

Python

runtime = trt.Runtime(TRT_LOGGER)
runtime.defer_weights_loading = True
engine = runtime.deserialize_cuda_engine(blob)

# [Optional] Attach a runtime cache to save the JIT-compiled
# kernels without loading weights. Keep config and cache alive
# while contexts created from them are in use.
config = engine.create_runtime_config()
cache = config.create_runtime_cache()
config.set_runtime_cache(cache)

context = engine.create_execution_context(config)

# [Optional] Save the compiled kernels to disk.
with cache.serialize() as serialized_cache:
    with open("runtime.cache", "wb") as cache_file:
        cache_file.write(serialized_cache)

# Before the first inference, load the weights into GPU memory
# from the same plan blob.
engine.load_weights(blob)
assert engine.weights_loaded
context.execute_async_v3(stream)

Loading Weights Directly to the GPU

To transfer weights directly from storage to the GPU without staging a complete copy in host memory, use loadWeightsAsync() with an IStreamReaderV2 that serves the engine plan bytes on demand. loadWeightsAsync() does not synchronize the stream. Enqueue dependent work on the same stream, and synchronize the stream explicitly when needed.

C++

// myReader implements IStreamReaderV2 and serves the engine plan
// bytes on demand.
engine->loadWeightsAsync(myReader, stream);
context->enqueueV3(stream);
// ...
cudaStreamSynchronize(stream);

Python

engine.load_weights_async(my_reader, int(stream))
context.execute_async_v3(int(stream))
# ...
cudart.cudaStreamSynchronize(stream)

Next Steps

• Porting from TensorRT: Deploying CPU-only ahead-of-time engines in your application

• Simultaneous Compute and Graphics: Running inference alongside graphics workloads

• Best Practices: Measuring and optimizing inference performance