an object for parsing ONNX models into a TensorRT network definition More...

#include <NvOnnxParser.h>

Public Member Functions
virtual bool	parse (void const serialized_onnx_model, size_t serialized_onnx_model_size, const char model_path=nullptr) noexcept=0
	Parse a serialized ONNX model into the TensorRT network. This method has very limited diagnostics. If parsing the serialized model fails for any reason (e.g. unsupported IR version, unsupported opset, etc.) it the user responsibility to intercept and report the error. To obtain a better diagnostic, use the parseFromFile method below. More...

virtual bool	parseFromFile (char const *onnxModelFile, int verbosity) noexcept=0
	Parse an onnx model file, which can be a binary protobuf or a text onnx model calls parse method inside. More...

virtual bool	supportsOperator (const char *op_name) const noexcept=0
	Returns whether the specified operator may be supported by the parser. More...

virtual int	getNbErrors () const noexcept=0
	Get the number of errors that occurred during prior calls to `parse`. More...

virtual IParserError const *	getError (int index) const noexcept=0
	Get an error that occurred during prior calls to `parse`. More...

virtual void	clearErrors () noexcept=0
	Clear errors from prior calls to `parse`. More...

virtual	~IParser () noexcept=default

virtual char const const	getUsedVCPluginLibraries (int64_t &nbPluginLibs) const noexcept=0
	Query the plugin libraries needed to implement operations used by the parser in a version-compatible engine. More...

virtual void	setFlags (OnnxParserFlags onnxParserFlags) noexcept=0
	Set the parser flags. More...

virtual OnnxParserFlags	getFlags () const noexcept=0
	Get the parser flags. Defaults to 0. More...

virtual void	clearFlag (OnnxParserFlag onnxParserFlag) noexcept=0
	clear a parser flag. More...

virtual void	setFlag (OnnxParserFlag onnxParserFlag) noexcept=0
	Set a single parser flag. More...

virtual bool	getFlag (OnnxParserFlag onnxParserFlag) const noexcept=0
	Returns true if the parser flag is set. More...

virtual nvinfer1::ITensor const *	getLayerOutputTensor (char const *name, int64_t i) noexcept=0
	Return the i-th output ITensor object for the ONNX layer "name". More...

virtual bool	supportsModelV2 (void const serializedOnnxModel, size_t serializedOnnxModelSize, char const modelPath=nullptr) noexcept=0
	Check whether TensorRT supports a particular ONNX model. If the function returns True, one can proceed to engine building without having to call `parse` or `parseFromFile`. Results can be queried through `getNbSubgraphs`, `isSubgraphSupported`, `getSubgraphNodes`. More...

virtual int64_t	getNbSubgraphs () noexcept=0
	Get the number of subgraphs. Calling this function before calling `supportsModelV2` results in undefined behavior. More...

virtual bool	isSubgraphSupported (int64_t const index) noexcept=0
	Returns whether the subgraph is supported. Calling this function before calling `supportsModelV2` results in undefined behavior. More...

virtual int64_t *	getSubgraphNodes (int64_t const index, int64_t &subgraphLength) noexcept=0
	Get the nodes of the specified subgraph. Calling this function before calling `supportsModelV2` results in undefined behavior. More...

virtual bool	loadModelProto (void const serializedOnnxModel, size_t serializedOnnxModelSize, char const modelPath=nullptr) noexcept=0
	Load a serialized ONNX model into the parser. Unlike the parse() or parseFromFile() functions, this function does not immediately convert the model into a TensorRT INetworkDefinition. Using this function allows users to provide their own initializers for the ONNX model through the loadInitializer() function. More...

virtual bool	loadInitializer (char const name, void const data, size_t size) noexcept=0
	Prompt the ONNX parser to load an initializer with user-provided binary data. The lifetime of the data must exceed the lifetime of the parser. More...

virtual bool	parseModelProto () noexcept=0
	Begin the parsing and conversion process of the loaded ONNX model into a TensorRT INetworkDefinition. More...

virtual bool	setBuilderConfig (const nvinfer1::IBuilderConfig *const builderConfig) noexcept=0
	Set the BuilderConfig for the parser. More...

virtual void	setRefitObserver (IRefitterObserver *observer) noexcept=0
	Set or clear an optional observer notified once per refittable weight during parsing. More...

Detailed Description

an object for parsing ONNX models into a TensorRT network definition

Warning: If the ONNX model has a graph output with the same name as a graph input, the output will be renamed by prepending "__".; Do not inherit from this class, as doing so will break forward-compatibility of the API and ABI.

Constructor & Destructor Documentation

◆ ~IParser()

virtual nvonnxparser::IParser::~IParser ( )

virtualdefaultnoexcept

Member Function Documentation

◆ clearErrors()

virtual void nvonnxparser::IParser::clearErrors ( )

pure virtualnoexcept

Clear errors from prior calls to parse.

See also: getNbErrors() getError() IParserError

◆ clearFlag()

virtual void nvonnxparser::IParser::clearFlag ( OnnxParserFlag onnxParserFlag )

pure virtualnoexcept

clear a parser flag.

clears the parser flag from the enabled flags.

See also: setFlags()

◆ getError()

virtual IParserError const * nvonnxparser::IParser::getError ( int index ) const

pure virtualnoexcept

Get an error that occurred during prior calls to parse.

See also: getNbErrors() clearErrors() IParserError

◆ getFlag()

virtual bool nvonnxparser::IParser::getFlag ( OnnxParserFlag onnxParserFlag ) const

pure virtualnoexcept

Returns true if the parser flag is set.

See also: getFlags()

Returns: True if flag is set, false if unset.

◆ getFlags()

virtual OnnxParserFlags nvonnxparser::IParser::getFlags ( ) const

pure virtualnoexcept

Get the parser flags. Defaults to 0.

Returns: The parser flags as a bitmask.

See also: setFlags()

◆ getLayerOutputTensor()

virtual nvinfer1::ITensor const * nvonnxparser::IParser::getLayerOutputTensor	(	char const *	name,
		int64_t	i
	)

pure virtualnoexcept

Return the i-th output ITensor object for the ONNX layer "name".

Return the i-th output ITensor object for the ONNX layer "name". If "name" is not found or i is out of range, return nullptr. In the case of multiple nodes sharing the same name this function will return the output tensors of the first instance of the node in the ONNX graph.

Parameters

name	The name of the ONNX layer.
i	The index of the output. i must be in range [0, layer.num_outputs).

◆ getNbErrors()

virtual int nvonnxparser::IParser::getNbErrors ( ) const

pure virtualnoexcept

Get the number of errors that occurred during prior calls to parse.

See also: getError() clearErrors() IParserError

◆ getNbSubgraphs()

virtual int64_t nvonnxparser::IParser::getNbSubgraphs ( )

pure virtualnoexcept

Get the number of subgraphs. Calling this function before calling supportsModelV2 results in undefined behavior.

Returns: Number of subgraphs.

◆ getSubgraphNodes()

virtual int64_t * nvonnxparser::IParser::getSubgraphNodes	(	int64_t const	index,
		int64_t &	subgraphLength
	)

pure virtualnoexcept

Get the nodes of the specified subgraph. Calling this function before calling supportsModelV2 results in undefined behavior.

Parameters

index	Index of the subgraph.
subgraphLength	Returns the length of the subgraph as reference.

Returns: Pointer to the subgraph nodes array. This pointer is owned by the Parser.

◆ getUsedVCPluginLibraries()

virtual char const *const * nvonnxparser::IParser::getUsedVCPluginLibraries ( int64_t & nbPluginLibs ) const

pure virtualnoexcept

Query the plugin libraries needed to implement operations used by the parser in a version-compatible engine.

This provides a list of plugin libraries on the filesystem needed to implement operations in the parsed network. If you are building a version-compatible engine using this network, provide this list to IBuilderConfig::setPluginsToSerialize to serialize these plugins along with the version-compatible engine, or, if you want to ship these plugin libraries externally to the engine, ensure that IPluginRegistry::loadLibrary is used to load these libraries in the appropriate runtime before deserializing the corresponding engine.

Parameters

[out] nbPluginLibs Returns the number of plugin libraries in the array, or -1 if there was an error.

Returns: Array of nbPluginLibs C-strings describing plugin library paths on the filesystem if nbPluginLibs > 0, or nullptr otherwise. This array is owned by the IParser, and the pointers in the array are only valid until the next call to parse() or parseFromFile().

◆ isSubgraphSupported()

virtual bool nvonnxparser::IParser::isSubgraphSupported ( int64_t const index )

pure virtualnoexcept

Returns whether the subgraph is supported. Calling this function before calling supportsModelV2 results in undefined behavior.

Parameters

index Index of the subgraph.

Returns: Whether the subgraph is supported.

◆ loadInitializer()

virtual bool nvonnxparser::IParser::loadInitializer	(	char const *	name,
		void const *	data,
		size_t	size
	)

pure virtualnoexcept

Prompt the ONNX parser to load an initializer with user-provided binary data. The lifetime of the data must exceed the lifetime of the parser.

All user-provided initializers must be provided prior to calling refitModelProto().

This function can be called multiple times to specify the names of multiple initializers.

Calling this function with an initializer previously specified will overwrite the previous instance.

This function will return false if initializer validation fails. Possible validation errors are:

This function was called prior to loadModelProto().
The requested initializer was not found in the model.
The size of the data provided is different from the corresponding initializer in the model.

Parameters

name	Name of the initializer.
data	Binary data containing the values of the initializer.
size	Size of the initializer in bytes.

Returns: true if the initializer was loaded successfully

See also: loadModelProto()

◆ loadModelProto()

virtual bool nvonnxparser::IParser::loadModelProto	(	void const *	serializedOnnxModel,
		size_t	serializedOnnxModelSize,
		char const *	modelPath = `nullptr`
	)

pure virtualnoexcept

Load a serialized ONNX model into the parser. Unlike the parse() or parseFromFile() functions, this function does not immediately convert the model into a TensorRT INetworkDefinition. Using this function allows users to provide their own initializers for the ONNX model through the loadInitializer() function.

Only one model can be loaded at a time. Subsequent calls to loadModelProto() will result in an error.

To begin the conversion of the model into a TensorRT INetworkDefinition, use parseModelProto().

Parameters

serializedOnnxModel	Pointer to the serialized ONNX model. Can be freed after this function returns.
serializedOnnxModelSize	Size of the serialized ONNX model in bytes.
modelPath	Absolute path to the model file for loading external weights if required.

Returns: true if the model was loaded successfully

See also: getNbErrors() getError()

◆ parse()

virtual bool nvonnxparser::IParser::parse	(	void const *	serialized_onnx_model,
		size_t	serialized_onnx_model_size,
		const char *	model_path = `nullptr`
	)

pure virtualnoexcept

Parse a serialized ONNX model into the TensorRT network. This method has very limited diagnostics. If parsing the serialized model fails for any reason (e.g. unsupported IR version, unsupported opset, etc.) it the user responsibility to intercept and report the error. To obtain a better diagnostic, use the parseFromFile method below.

Parameters

serialized_onnx_model	Pointer to the serialized ONNX model. Can be freed after this function returns.
serialized_onnx_model_size	Size of the serialized ONNX model in bytes
model_path	Absolute path to the model file for loading external weights if required

Returns: true if the model was parsed successfully

See also: getNbErrors() getError()

◆ parseFromFile()

virtual bool nvonnxparser::IParser::parseFromFile	(	char const *	onnxModelFile,
		int	verbosity
	)

pure virtualnoexcept

Parse an onnx model file, which can be a binary protobuf or a text onnx model calls parse method inside.

Parameters

onnxModelFile	name
verbosity	Level

Returns: true if the model was parsed successfully

◆ parseModelProto()

virtual bool nvonnxparser::IParser::parseModelProto ( )

pure virtualnoexcept

Begin the parsing and conversion process of the loaded ONNX model into a TensorRT INetworkDefinition.

Returns: true if conversion was successful

See also: getNbErrors() getError() loadModelProto() loadModelProtoFromFile()

◆ setBuilderConfig()

virtual bool nvonnxparser::IParser::setBuilderConfig ( const nvinfer1::IBuilderConfig *const builderConfig )

pure virtualnoexcept

Set the BuilderConfig for the parser.

Returns: true if the IBuilderConfig was set successfully, false otherwise.

◆ setFlag()

virtual void nvonnxparser::IParser::setFlag ( OnnxParserFlag onnxParserFlag )

pure virtualnoexcept

Set a single parser flag.

Add the input parser flag to the already enabled flags.

See also: setFlags()

◆ setFlags()

virtual void nvonnxparser::IParser::setFlags ( OnnxParserFlags onnxParserFlags )

pure virtualnoexcept

Set the parser flags.

The flags are listed in the OnnxParserFlag enum.

Parameters

OnnxParserFlags The flags used when parsing an ONNX model.

Note: This function will override the previous set flags, rather than bitwise ORing the new flag.

See also: getFlags()

◆ setRefitObserver()

virtual void nvonnxparser::IParser::setRefitObserver ( IRefitterObserver * observer )

pure virtualnoexcept

Set or clear an optional observer notified once per refittable weight during parsing.

When attached, the parser emits one RefitRecord per network weight it names via INetworkDefinition::setWeightsName, at the moment the weight is created during parse/parseModelProto. This is the same record schema emitted by IParserRefitter::setRefitObserver during refit, produced without building or deserializing an engine. Because the network has not been built yet, the records are a candidate superset of the built engine's refittable weights: the builder may fold or absorb some of them. Consumers replaying the records against an engine must skip records whose trtName the engine's nvinfer1::IRefitter does not report in getAllWeights, and rely on refitCudaEngine's missing-weights check for coverage.

TensorRT-RTX only: When an observer is attached to the parser associated with a BuilderConfig that sets kREFIT_INDIVIDUAL and kSTRIP_PLAN, the parser will provide placeholder weights for the listed initializers of the following ONNX nodes:

Conv kernels and biases
Gemm operands and bias
MatMul operands
DequantizeLinear scales
DequantizeLinear FP8 or FP4 data
TRT_FP8DequantizeLinear/TRT_MXFP8DequantizeLinear FP8 data A placeholder weight is represented by nvinfer1::Weights with the same type and count as the original initializer, but with nullptr values. All placeholder weights are expected to be refit prior to inference.

A placeholder weight will not be produced under the following circumstances:

If the initializer is also a graph input or output
If the initializer is referenced by a nested graph

Placeholders will not be created when importing a model whose producer name is "TensorRT". Networks containing placeholders must be built with IBuilder::buildSerializedNetwork().

Records are only valid if the subsequent parse call returns true.

May be called any time before parse / parseModelProto. Pass nullptr to detach. The observer must outlive the parse call, or be detached before destruction. Ownership remains with the caller.

See also: IRefitterObserver IParserRefitter::setRefitObserver

◆ supportsModelV2()

virtual bool nvonnxparser::IParser::supportsModelV2	(	void const *	serializedOnnxModel,
		size_t	serializedOnnxModelSize,
		char const *	modelPath = `nullptr`
	)

pure virtualnoexcept

Check whether TensorRT supports a particular ONNX model. If the function returns True, one can proceed to engine building without having to call parse or parseFromFile. Results can be queried through getNbSubgraphs, isSubgraphSupported, getSubgraphNodes.

Parameters

serializedOnnxModel	Pointer to the serialized ONNX model. Can be freed after this function returns.
serializedOnnxModelSize	Size of the serialized ONNX model in bytes
modelPath	Absolute path to the model file for loading external weights if required

Returns: true if the model is supported

◆ supportsOperator()

virtual bool nvonnxparser::IParser::supportsOperator ( const char * op_name ) const

pure virtualnoexcept

Returns whether the specified operator may be supported by the parser.

Note that a result of true does not guarantee that the operator will be supported in all cases (i.e., this function may return false-positives).

Parameters

op_name The name of the ONNX operator to check for support

The documentation for this class was generated from the following file:

NvOnnxParser.h

Public Member Functions

Detailed Description

Constructor & Destructor Documentation

◆ ~IParser()

Member Function Documentation

◆ clearErrors()

◆ clearFlag()

◆ getError()

◆ getFlag()

◆ getFlags()

◆ getLayerOutputTensor()

◆ getNbErrors()

◆ getNbSubgraphs()

◆ getSubgraphNodes()

◆ getUsedVCPluginLibraries()

◆ isSubgraphSupported()

◆ loadInitializer()

◆ loadModelProto()

◆ parse()

◆ parseFromFile()

◆ parseModelProto()

◆ setBuilderConfig()

◆ setFlag()

◆ setFlags()

◆ setRefitObserver()

◆ supportsModelV2()

◆ supportsOperator()