nvinfer1::IRuntimeConfig Class Reference

A class for runtime configuration. This class is used during execution context creation. More...

#include <NvInferRuntime.h>

Inheritance diagram for nvinfer1::IRuntimeConfig:

Public Member Functions
virtual	~IRuntimeConfig () noexcept=default
void	setExecutionContextAllocationStrategy (ExecutionContextAllocationStrategy strategy) noexcept
	Set the execution context allocation strategy. Default value is kSTATIC. More...
ExecutionContextAllocationStrategy	getExecutionContextAllocationStrategy () const noexcept
	Get the execution context allocation strategy. More...
IRuntimeCache *	createRuntimeCache () const noexcept
	Create an empty Runtime cache. More...
bool	setRuntimeCache (IRuntimeCache const &cache) noexcept
	Set Runtime cache to the runtime config. Enables Runtime caching. More...
IRuntimeCache *	getRuntimeCache () const noexcept
	Get the Runtime cache from the runtime config. More...
void	setDynamicShapesKernelSpecializationStrategy (DynamicShapesKernelSpecializationStrategy dynamicShapesKernelSpecializationStrategy) noexcept
	Set the dynamic shape kernel specialization strategy for this config. More...
DynamicShapesKernelSpecializationStrategy	getDynamicShapesKernelSpecializationStrategy () const noexcept
	Return the dynamic shape specialization strategy of this config. More...
bool	setCudaGraphStrategy (CudaGraphStrategy strategy) noexcept
	Set the strategy used for CUDA graphs for JIT (Just-In-Time) inference. Default is kDISABLED. More...
CudaGraphStrategy	getCudaGraphStrategy () const noexcept
	Return the strategy used for CUDA graphs for JIT (Just-In-Time) inference. More...

Protected Attributes
apiv::VRuntimeConfig *	mImpl

Additional Inherited Members
Protected Member Functions inherited from nvinfer1::INoCopy
	INoCopy ()=default
virtual	~INoCopy ()=default
	INoCopy (INoCopy const &other)=delete
INoCopy &	operator= (INoCopy const &other)=delete
	INoCopy (INoCopy &&other)=delete
INoCopy &	operator= (INoCopy &&other)=delete

Detailed Description

A class for runtime configuration. This class is used during execution context creation.

See also

IRuntime, IBuilderConfig

Constructor & Destructor Documentation

~IRuntimeConfig()

virtual nvinfer1::IRuntimeConfig::~IRuntimeConfig ( )

virtualdefaultnoexcept

Member Function Documentation

createRuntimeCache()

IRuntimeCache * nvinfer1::IRuntimeConfig::createRuntimeCache ( ) const

inlinenoexcept

Create an empty Runtime cache.

Returns

A fresh IRuntimeCache object.

getCudaGraphStrategy()

CudaGraphStrategy nvinfer1::IRuntimeConfig::getCudaGraphStrategy ( ) const

inlinenoexcept

Return the strategy used for CUDA graphs for JIT (Just-In-Time) inference.

Returns

The CUDA graph strategy.

getDynamicShapesKernelSpecializationStrategy()

DynamicShapesKernelSpecializationStrategy nvinfer1::IRuntimeConfig::getDynamicShapesKernelSpecializationStrategy ( ) const

inlinenoexcept

Return the dynamic shape specialization strategy of this config.

Returns

dynamicShapesKernelSpecializationStrategy The dynamic shape specialization strategy.

getExecutionContextAllocationStrategy()

ExecutionContextAllocationStrategy nvinfer1::IRuntimeConfig::getExecutionContextAllocationStrategy ( ) const

inlinenoexcept

Get the execution context allocation strategy.

Returns

The execution context allocation strategy.

getRuntimeCache()

IRuntimeCache * nvinfer1::IRuntimeConfig::getRuntimeCache ( ) const

inlinenoexcept

Get the Runtime cache from the runtime config.

Returns

The Runtime cache.

setCudaGraphStrategy()

bool nvinfer1::IRuntimeConfig::setCudaGraphStrategy ( CudaGraphStrategy  strategy )

inlinenoexcept

Set the strategy used for CUDA graphs for JIT (Just-In-Time) inference. Default is kDISABLED.

Note

Even if the strategy was set to kWHOLE_GRAPH_CAPTURE, RTX CUDA graphs may be disabled internally during runtime under following circumstances:

• The given stream does not allow graph capture.
• The allocation strategy is blocking.
• Some layers in the engine use data-dependent dynamic shapes or on-device control flow.
• The engine is streaming weights. If CUDA graph is not supported for above or any other reason, it will silently ignore the set strategy and run inference without CUDA graphs.

Warning

If the stream is being captured elsewhere, RTX CUDA graphs will be disabled.

Parameters

strategy

The CUDA graph strategy.

Returns

True if the strategy was within range and set successfully, false otherwise.

setDynamicShapesKernelSpecializationStrategy()

void nvinfer1::IRuntimeConfig::setDynamicShapesKernelSpecializationStrategy ( DynamicShapesKernelSpecializationStrategy  dynamicShapesKernelSpecializationStrategy )

inlinenoexcept

Set the dynamic shape kernel specialization strategy for this config.

Parameters

dynamicShapesKernelSpecializationStrategy

The dynamic shape specialization strategy.

setExecutionContextAllocationStrategy()

void nvinfer1::IRuntimeConfig::setExecutionContextAllocationStrategy ( ExecutionContextAllocationStrategy  strategy )

inlinenoexcept

Set the execution context allocation strategy. Default value is kSTATIC.

Parameters

strategy

The execution context allocation strategy.

setRuntimeCache()

bool nvinfer1::IRuntimeConfig::setRuntimeCache ( IRuntimeCache const &  cache )

inlinenoexcept

Set Runtime cache to the runtime config. Enables Runtime caching.

Parameters

cache

The Runtime cache to set.

Member Data Documentation

mImpl

apiv::VRuntimeConfig* nvinfer1::IRuntimeConfig::mImpl

protected

---

The documentation for this class was generated from the following file:

• NvInferRuntime.h