Is this page helpful?

Simultaneous Compute and Graphics#

TensorRT-RTX provides support for inference while simultaneously running graphics workloads (simultaneous compute and graphics = SCG). The most common use case is video games, where generative AI features such as NPC dialog generation typically execute concurrently with 3D rendering. When preparing a model to run in SCG mode, take the following steps:

Top-down flow diagram from Application with DirectX 12 and TensorRT-RTX through CUDA-in-Graphics to RTX GPU for simultaneous compute and graphics

Shared Memory Limitation#

NVIDIA CUDA compute kernels running concurrently with graphics workloads are limited to 48 KiB or less shared memory on the NVIDIA Ampere and Ada Lovelace architectures (compute capability 8.x). On the NVIDIA Blackwell architecture and beyond (compute capability 10.x and later), you can increase the limits to 84 KiB at the cost of L1 cache. Whether this happens depends on the priority of the DirectX compute command queue:

When running a low-priority queue for computation (D3D12_COMMAND_QUEUE_PRIORITY_NORMAL), the limit is 48 KiB, even on Blackwell. This is appropriate for heavy concurrent rendering workloads that require full access to the L1 cache.
When running a high-priority computation queue (D3D12_COMMAND_QUEUE_PRIORITY_HIGH), the limit is 84 KiB on Blackwell and subsequent architectures. This situation is appropriate for relatively light graphics workloads that do not require a large L1 cache.

Note

Simultaneous graphics and compute workloads are not supported for the NVIDIA Turing architecture (compute capability 7.5 or less).
This shared memory limitation restricts the choice of CUDA kernels for performing inference, which will generally result in reduced performance compared to running standalone inference.
For details on specifying target compute capabilities during engine builds, refer to the Compute Capability section.

Choose one of the following options to implement this constraint.

Option 1: Using the `tensorrt_rtx` Executable#

When building a TensorRT-RTX engine with the tensorrt_rtx executable for deploying in SCG mode, add the following flag:

When deploying on NVIDIA Ampere or NVIDIA Ada Lovelace:
```
--memPoolSize=tacticSharedMem:48K
```
When deploying on NVIDIA Blackwell and later, assuming light concurrent graphics workload:
```
--memPoolSize=tacticSharedMem:84K
```

These are the maximum shared memory limits that will allow the engine to work in SCG mode. Lower limits are possible (for example, --memPoolSize=tacticSharedMem:32K) but will typically degrade performance and are therefore not recommended.

If you want to set the limits programmatically (for example, as part of an installation script), the nvidia-smi command-line tool can be used to query the compute capability of the current system, for example in Microsoft Windows PowerShell:

 $nvsmiPath = (Get-Command nvidia-smi.exe -ErrorAction SilentlyContinue).Source

 if (-not $nvsmiPath){
     Throw "'nvidia-smi' not found: please install or add to PATH"
 }

 $ccString = & $nvsmiPath --query-gpu=compute_cap --format=csv,noheader
 if (-not $ccString){
     Throw "Error querying compute capability: ensure GPU is installed"
 }
 # Take first GPU if multiple are installed
 $ccString = $ccString.Trim().Split("`n")[0]

 # Parse the major version
 $majorVersion = [int]($ccString.Split(".")[0])

 # Assume light concurrent graphics workload otherwise use 48K limit even on Blackwell
 if ($majorVersion -ge 10){
     Write-Host "Blackwell or later (CC $ccString)"
     $scgFlag = "--memPoolSize=tacticSharedMem:84K"
 } else if ($majorVersion -eq 8){
     Write-Host "Ampere / Ada (CC $ccString)"
     $scgFlag = "--memPoolSize=tacticSharedMem:48K"
 } else {
     Throw "Unsupported CC $ccString"
 }

 tensorrt_rtx.exe --onnx=example_model.onnx --saveEngine=example_engine.trt $scgFlag

Option 2: Using the C++ or Python API#

Both the C++ and the Python API allow setting the same memory limits programmatically. These APIs accept limits in bytes rather than MiB.

C++

// C++, assuming a single installed GPU
cudaDeviceProp deviceProp;
cudaGetDeviceProperties(&deviceProp, 0);
unsigned memLimit{};
if (deviceProp.major >= 10){ // Blackwell
    memLimit = 84 * 1024;
} else if (deviceProp.major == 8){ // Ada or Ampere
    memLimit = 48 * 1024;
} else {
    throw "SCG not supported for this compute capability";
}
auto logger = getLogger();
auto builder = createInferBuilder(*logger);
auto config = builder->createBuilderConfig();
config->setMemoryPoolLimit(MemoryPoolType::kTACTIC_SHARED_MEMORY, memLimit);
// assemble your network…
if (builder->isNetworkSupported(*network, *config)){
    auto hostMem = builder->buildSerializedNetwork(*network, *config);
    // write to a file
    std::ofstream output_file("example_engine.trt", std::ios::binary);
    output_file.write(hostMem->data(), hostMem->size());
    output_file.close();
}

Python

import pycuda.autoinit
import pycuda.driver as cuda
import tensorrt_rtx as trt_rtx

device = cuda.Context.get_device()
major, minor = device.compute_capability()
if major >= 10: # Blackwell
    memLimit = 84 * 1024
elif major >= 8: # Ada or Ampere
    memLimit = 48 * 1024
else:
    raise("SCG not supported for this Compute Capability")
logger = trt_rtx.Logger()
builder = trt_rtx.Builder(logger)
config = builder.create_builder_config()
config.set_memory_pool_limit(trt_rtx.MemoryPoolType.TACTIC_SHARED_MEMORY, memLimit)
if builder.is_network_supported(network, config):
    hostMem = builder.build_serialized_network(network, config)
    with open("example_engine.trt", "wb") as fout:
        fout.write(hostMem)

CUDA Context in CiG Mode#

Before starting inference at runtime, ensure that the current CUDA context runs in CUDA-in-Graphics (CiG) mode. Refer to the CUDA Driver API documentation for more information.

Important

CiG mode is available in CUDA version 12.6 and later (NVIDIA RTX driver 555 and later). It does not work with earlier CUDA or driver versions.

Note

While recent driver versions support CiG with the Vulkan graphics API, TensorRT-RTX supports only DirectX operability in this release. Vulkan support is planned for a future release.

The following code demonstrates how to enable CiG in the CUDA context.

Note

The cuCtxCreate_v4 function automatically associates the newly created CUDA context with the current thread. Therefore, you can use the following code for normal setup of TensorRT-RTX for inference.

// Get CUDA device ID, assuming single device on current system
int deviceIndex;
cudaGetDevice(&deviceIndex);
CUdevice device;
cuDeviceGet(&device, deviceIndex);
// Check that CiG is supported for the current hardware with DirectX
int isSupported;
cuDeviceGetAttribute(&isSupported, CU_DEVICE_ATTRIBUTE_D3D12_CIG_SUPPORTED, device);
if (!isSupported){
    throw "Simultaneous graphics / compute not supported for current hardware";
}
// Create DirectX device
ID3D12Device *dxDevice;
D3D12CreateDevice(nullptr, D3D_FEATURE_LEVEL_11_0, IID_PPV_ARGS(&dxDevice));

// Create two DirectX command queues for graphics and computation
D3D12_COMMAND_QUEUE_DESC graphicsQueueDesc{};
graphicsQueueDesc.Flags = D3D12_COMMAND_QUEUE_FLAG_NONE;
graphicsQueueDesc.Type = D3D12_COMMAND_LIST_TYPE_DIRECT;
void* graphicsQueue;
dxDevice->CreateCommandQueue(&graphicsQueueDesc, IID_PPV_ARGS(&graphicsQueue));

D3D12_COMMAND_QUEUE_DESC computeQueueDesc{};
computeQueueDesc.Flags = D3D12_COMMAND_QUEUE_FLAG_NONE;
computeQueueDesc.Type = D3D12_COMMAND_LIST_TYPE_COMPUTE;
computeQueueDesc.Priority = D3D12_COMMAND_QUEUE_PRIORITY_HIGH;
void* dxComputeQueue;
dxDevice->CreateCommandQueue(&computeQueueDesc, IID_PPV_ARGS(&dxComputeQueue));

// Set context creation parameters
CUctxCigParam ctxCigParam;
ctxCigParam.sharedDataType = CIG_DATA_TYPE_D3D12_COMMAND_QUEUE;
ctxCigParam.sharedData = dxComputeQueue;
CUcontextCreateParams ctxCreateParams;
// Exactly one of execAffinityParams and cigParams must be non-null
ctxCreateParams.execAffinityParams = nullptr;
ctxCreateParams.numExecAffinityParams = 0;
ctxCreateParams.cigParams = &ctxCigParam;
// Create a CUDA context with CiG enabled
CUcontext ctx;
cuCtxCreate_v4(&ctx, &ctxCreateParams, 0, device);
// Verify that the current CUDA context has CiG enabled
size_t limit;
cuCtxGetLimit(&limit, CU_LIMIT_CIG_ENABLED);
if (limit == 0){
    throw "Enabling CiG in the current CUDA context failed";
}

Next Steps#

CPU-Only AOT and TensorRT-RTX Engines: Specifying compute capabilities for engine builds
Best Practices: Measuring and optimizing inference performance in CiG mode
Working with CUDA Graphs: Reducing kernel launch overhead with CUDA graphs

Simultaneous Compute and Graphics#

Shared Memory Limitation#

Option 1: Using the tensorrt_rtx Executable#

Option 2: Using the C++ or Python API#

CUDA Context in CiG Mode#

Next Steps#

Option 1: Using the `tensorrt_rtx` Executable#