GStreamer Plugin Details
Gst-nvinfer
The Gst-nvinfer plugin does inferencing on input data using NVIDIA® TensorRT™.
The plugin accepts batched NV12/RGBA buffers from upstream. The NvDsBatchMeta structure must already be attached to the Gst Buffers.
The low-level library (libnvds_infer) operates on any of INT8 RGB, BGR, or GRAY data with dimension of Network Height and Network Width.
The Gst-nvinfer plugin performs transforms (format conversion and scaling), on the input frame based on network requirements, and passes the transformed data to the low-level library.
The low-level library preprocesses the transformed frames (performs normalization and mean subtraction) and produces final float RGB/BGR/GRAY planar data which is passed to the TensorRT engine for inferencing. The output type generated by the low-level library depends on the network type.
The pre-processing function is:
Where:
• x is the input pixel value. It is an int8 with range [0,255].
• mean is the corresponding mean value, read either from the mean file or as offsets[c], where c is the channel to which the input pixel belongs, and offsets is the array specified in the configuration file. It is a float.
• net-scale-factor is the pixel scaling factor specified in the configuration file. It is a float.
• y is the corresponding output pixel value. It is a float.
Gst-nvinfer currently works on the following type of networks:
• Multi-class object detection
• Multi-label classification
• Segmentation
The Gst-nvinfer plugin can work in two modes:
• Primary mode: Operates on full frames
• Secondary mode: Operates on objects added in the meta by upstream components
When the plugin is operating as a secondary classifier along with the tracker, it tries to improve performance by avoiding re-inferencing on the same objects in every frame. It does this by caching the classification output in a map with the object’s unique ID as the key. The object is inferred upon only when it is first seen in a frame (based on its object ID) or when the size (bounding box area) of the object increases by 20% or more. This optimization is possible only when the tracker is added as an upstream element.
Detailed documentation of the TensorRT interface is available at:
The plugin supports the IPlugin interface for custom layers. Refer to section IPlugin Interface for details.
The plugin also supports the interface for custom functions for parsing outputs of object detectors and initialization of non-image input layers in cases where there are more than one input layer.
Refer to sources/includes/nvdsinfer_custom_impl.h for the custom method implementations for custom models.
Downstream components receive a Gst Buffer with unmodified contents plus the metadata created from the inference output of the Gst-nvinfer plugin.
The plugin can be used for cascaded inferencing. That is, it can perform primary inferencing directly on input data, then perform secondary inferencing on the results of primary inferencing, and so on. See the sample application deepstream-test2 for more details.
Inputs and Outputs
This section summarizes the inputs, outputs, and communication facilities of the Gst-nvinfer plugin.
• Inputs
• Gst Buffer
• NvDsBatchMeta (attaching NvDsFrameMeta)
• Caffe Model and Caffe Prototxt
• ONNX
• UFF file
• TLT Encoded Model and Key
• Offline: Supports engine files generated by Transfer Learning Toolkit SDK Model converters
• Layers: Supports all layers supported by TensorRT, see:
• Control parameters: Gst-nvinfer gets control parameters from a configuration file. You can specify this by setting the property config-file-path. For details, see Gst-nvinfer File Configuration Specifications. Other control parameters that can be set through GObject properties are:
• Batch size
• Inference interval
• Attach inference tensor outputs as buffer metadata
• The parameters set through the GObject properties override the parameters in the Gst-nvinfer configuration file.
• Outputs
• Gst Buffer
• Depending on network type and configured parameters, one or more of:
• NvDsObjectMeta
• NvDsClassifierMeta
• NvDsInferSegmentationMeta
• NvDsInferTensorMeta
Features
The following table summarizes the features of the plugin.
Features of the Gst-invinfer plugin | |||
|---|---|---|---|
Feature | Description | Release | |
Explicit Full Dimension Network Support | Refer to https://docs.nvidia.com/deeplearning/sdk/tensorrt-developer-guide/index.html#work_dynamic_shapes for more details | DS 5.0 | |
Non-maximum Suppression (NMS) | DS 5.0 | ||
On-the-fly model update (Engine file only) | Update the model-engine-file on-the-fly in a running pipeline | DS 5.0 | |
Transfer-Learning-Toolkit encoded model support | — | DS 4.0 | |
Gray input model support | Support for models with single channel gray input | DS 4.0 | |
Tensor output as meta | Raw tensor output is attached as meta data to Gst Buffers and flowed through the pipeline | DS 4.0 | |
Segmentation model | Supports segmentation model | DS 4.0 | |
Maintain input aspect ratio | Configurable support for maintaining aspect ratio when scaling input frame to network resolution | DS 4.0 | |
Custom cuda engine creation interface | Interface for generating CUDA engines from TensorRT INetworkDefinition and IBuilder APIs instead of model files | DS 4.0 | |
Caffe Model support | — | DS 2.0 | |
UFF Model support | — | DS 3.0 | |
ONNX Model support | — | DS 3.0 | |
Multiple modes of operation | Support for cascaded inferencing | DS 2.0 | |
Asynchronous mode of operation for secondary inferencing | Infer asynchronously for secondary classifiers | DS 2.0 | |
Grouping using CV::Group rectangles | For detector bounding box clustering | DS 2.0 | |
Configurable batch-size processing | User can configure batch size for processing | DS 2.0 | |
No Restriction on number of output blobs | Supports any number of output blobs | DS 3.0 | |
Configurable number of detected classes (detectors) | Supports configurable number of detected classes | DS 3.0 | |
Support for Classes: configurable (> 32) | Support any number of classes | DS 3.0 | |
Application access to raw inference output | Application can access inference output buffers for user specified layer | DS 3.0 | |
Support for single shot detector (SSD) | — | DS 3.0 | |
Secondary GPU Inference Engines (GIEs) operate as detector on primary bounding box | Support secondary inferencing as detector | DS 2.0 | |
Multiclass secondary support | Support multiple classifier network outputs | DS 2.0 | |
Grouping using DBSCAN | For detector bounding box clustering | DS 3.0 | |
Loading an external lib containing IPlugin implementation for custom layers (IPluginCreator & IPluginFactory) | Supports loading (dlopen() ) a library containing IPlugin implementation for custom layers | DS 3.0 | |
Multi GPU | Select GPU on which we want to run inference | DS 2.0 | |
Detection width height configuration | Filter out detected objects based on min/max object size threshold | DS 2.0 | |
Allow user to register custom parser | Supports final output layer bounding box parsing for custom detector network | DS 2.0 | |
Bounding box filtering based on configurable object size | Supports inferencing in secondary mode objects meeting min/max size threshold | DS 2.0 | |
Configurable operation interval | Interval for inferencing (number of batched buffers skipped) | DS 2.0 | |
Select Top and bottom regions of interest (RoIs) | Removes detected objects in top and bottom areas | DS 2.0 | |
Operate on Specific object type (Secondary mode) | Process only objects of define classes for secondary inferencing | DS 2.0 | |
Configurable blob names for parsing bounding box (detector) | Support configurable names for output blobs for detectors | DS 2.0 | |
Allow configuration file input | Support configuration file as input (mandatory in DS 3.0) | DS 2.0 | |
Allow selection of class id for operation | Supports secondary inferencing based on class ID | DS 2.0 | |
Support for Full Frame Inference: Primary as a classifier | Can work as classifier as well in primary mode | DS 2.0 | |
Multiclass secondary support | Support multiple classifier network outputs | DS 2.0 | |
Secondary GIEs operate as detector on primary bounding box Support secondary inferencing as detector | — | DS 2.0 | |
Supports FP16, FP32 and INT8 models FP16 and INT8 are platform dependent | — | DS 2.0 | |
Supports TensorRT Engine file as input | — | DS 2.0 | |
Inference input layer initialization Initializing non-video input layers in case of more than one input layers | — | DS 3.0 | |
Support for FasterRCNN | — | DS 3.0 | |
Support for Yolo detector (YoloV3/V3-tiny/V2/V2-tiny) | — | DS 4.0 | |
Support for yolov3-ssp detector | — | DS 5.0 | |
Gst-nvinfer File Configuration Specifications
The Gst-nvinfer configuration file uses a “Key File” format described in:
The [property] group configures the general behavior of the plugin. It is the only mandatory group.
The [class-attrs-all] group configures detection parameters for all classes.
The [class-attrs-<class-id>] group configures detection parameters for a class specified by <class-id>. For example, the [class-attrs-23] group configures detection parameters for class ID 23. This type of group has the same keys as [class-attrs-all].
The following two tables respectively describe the keys supported for [property] groups and [class-attrs-…] groups.
Gst-nvinfer plugin, [property] group, supported keys | |||||
|---|---|---|---|---|---|
Network Types / Applicable to GIEs | |||||
Property | Meaning | Type and Range | Example Notes | (Primary/Seconday) | |
num-detected-classes | Number of classes detected by the network | Integer, >0 | num-detected-classes=91 | Detector Both | |
net-scale-factor | Pixel normalization factor | Float, >0.0 | net-scale-factor=0.031 | All Both | |
model-file | Pathname of the caffemodel file. Not required if model-engine-file is used | String | model-file=/home/ubuntu/model.caffemodel | All Both | |
proto-file | Pathname of the prototxt file. Not required if model-engine-file is used | String | proto-file=/home/ubuntu/model.prototxt | All Both | |
int8-calib-file | Pathname of the INT8 calibration file for dynamic range adjustment with an FP32 model | String | int8-calib-file=/home/ubuntu/int8_calib | All Both | |
batch-size | Number of frames or objects to be inferred together in a batch | Integer, >0 | batch-size=30 | All Both | |
model-engine-file | Pathname of the serialized model engine file | String | model-engine-file=/home/ubuntu/model.engine | All Both | |
uff-file | Pathname of the UFF model file | String | uff-file=/home/ubuntu/model.uff | All Both | |
onnx-file | Pathname of the ONNX model file | String | onnx-file=/home/ubuntu/model.onnx | All Both | |
enable-dbscan | Indicates whether to use DBSCAN or the OpenCV groupRectangles() function for grouping detected objects. DEPRECATED. Use cluster-mode instead. | Boolean | enable-dbscan=1 | Detector Both | |
labelfile-path | Pathname of a text file containing the labels for the model | String | labelfile-path=/home/ubuntu/model_labels.txt | Detector & classifier Both | |
mean-file | Pathname of mean data file (PPM format) | String | mean-file=/home/ubuntu/model_meanfile.ppm | All Both | |
gie-unique-id | Unique ID to be assigned to the GIE to enable the application and other elements to identify detected bounding boxes and labels | Integer, >0 | gie-unique-id=2 | All Both | |
operate-on-gie-id | Unique ID of the GIE on whose metadata (bounding boxes) this GIE is to operate on | Integer, >0 | operate-on-gie-id=1 | All Both | |
operate-on-class-ids | Class IDs of the parent GIE on which this GIE is to operate on | Semicolon delimited integer array | operate-on-class-ids=1;2 Operates on objects with class IDs 1, 2 generated by parent GIE | All Both | |
interval | Specifies the number of consecutive batches to be skipped for inference | Integer, >0 | interval=1 | All Primary | |
input-object-min-width | Secondary GIE infers only on objects with this minimum width | Integer, ≥0 | input-object-min-width=40 | All Secondary | |
input-object-min-height | Secondary GIE infers only on objects with this minimum height | Integer, ≥0 | input-object-min-height=40 | All Secondary | |
input-object-max-width | Secondary GIE infers only on objects with this maximum width | Integer, ≥0 | input-object-max-width=256 0 disables the threshold | All Secondary | |
input-object-max-height | Secondary GIE infers only on objects with this maximum height | Integer, ≥0 | input-object-max-height=256 0 disables the threshold | All Secondary | |
uff-input-dims | Dimensions of the UFF model | channel; height; width; input-order All integers, ≥0 | input-dims=3;224;224;0 Possible values for input-order are: 0: NCHW 1: NHWC | All Both | |
network-mode | Data format to be used by inference | Integer 0: FP32 1: INT8 2: FP16 | network-mode=0 | All Both | |
offsets | Array of mean values of color components to be subtracted from each pixel. Array length must equal the number of color components in the frame. The plugin multiplies mean values by net-scale-factor. | Semicolon delimited float array, all values ≥0 | offsets=77.5;21.2;11.8 | All Both | |
output-blob-names | Array of output layer names | Semicolon delimited string array | For detector: output-blob-names=coverage;bbox For multi-label classifiers: output-blob-names=coverage_attrib1;coverage_attrib2 | All Both | |
parse-bbox-func-name | Name of the custom bounding box parsing function. If not specified, Gst-nvinfer uses the internal function for the resnet model provided by the SDK. | String | parse-bbox-func-name=parse_bbox_resnet | Detector Both | |
custom-lib-path | Absolute pathname of a library containing custom method implementations for custom models | String | custom-lib-path=/home/ubuntu/libresnet_custom_impl.so | All Both | |
model-color-format | Color format required by the model. | Integer 0: RGB 1: BGR 2: GRAY | model-color-format=0 | All Both | |
classifier-async-mode | Enables inference on detected objects and asynchronous metadata attachments. Works only when tracker-ids are attached. Pushes buffer downstream without waiting for inference results. Attaches metadata after the inference results are available to next Gst Buffer in its internal queue. | Boolean | classifier-async-mode=1 | Classifier Secondary | |
process-mode | Mode (primary or secondary) in which the element is to operate on | Integer 1=Primary 2=Secondary | gie-mode=1 | All Both | |
classifier-threshold | Minimum threshold label probability. The GIE outputs the label having the highest probability if it is greater than this threshold | Float, ≥0 | classifier-threshold=0.4 | Classifier Both | |
uff-input-blob-name | Name of the input blob in the UFF file | String | uff-input-blob-name=Input_1 | All Both | |
secondary-reinfer-interval | Reinference interval for objects, in frames | Integer, ≥0 | secondary-reinfer-interval=15 | Classifier Secondary | |
output-tensor-meta | Gst-nvinfer attaches raw tensor output as Gst Buffer metadata. | Boolean | output-tensor-meta=1 | All Both | |
enable-dla | Indicates whether to use the DLA engine for inferencing. Note: DLA is supported only on NVIDIA® Jetson AGX Xavier™. Currently work in progress. | Boolean | enable-dla=1 | All Both | |
use-dla-core | DLA core to be used. Note: Supported only on Jetson AGX Xavier. Currently work in progress. | Integer, ≥0 | use-dla-core=0 | All Both | |
network-type | Type of network | Integer 0: Detector 1: Classifier 2: Segmentation | network-type=1 | All Both | |
maintain-aspect-ratio | Indicates whether to maintain aspect ratio while scaling input. | Boolean | maintain-aspect-ratio=1 | All Both | |
parse-classifier-func-name | Name of the custom classifier output parsing function. If not specified, Gst-nvinfer uses the internal parsing function for softmax layers. | String | parse-classifier-func-name=parse_bbox_softmax | Classifier Both | |
custom-network-config | Pathname of the configuration file for custom networks available in the custom interface for creating CUDA engines. | String | custom-network-config=/home/ubuntu/network.config | All Both | |
tlt-encoded-model | Pathname of the Transfer Learning Toolkit (TLT) encoded model. | String | tlt-encoded-model=/home/ubuntu/model.etlt | All Both | |
tlt-model-key | Key for the TLT encoded model. | String | tlt-model-key=abc | All Both | |
segmentation-threshold | Confidence threshold for the segmentation model to output a valid class for a pixel. If confidence is less than this threshold, class output for that pixel is −1. | Float, ≥0.0 | segmentation-threshold=0.3 | Segmentation Both | |
workspace-size | Workspace size to be used by the engine, in MB | Integer, >0 | workspace-size=45 | All Both | |
force-implicit-batch-dim | When a network supports both implicit batch dimension and full dimension, force the implicit batch dimension mode. | Boolean | force-implicit-batch-dim=1 | All Both | |
infer-dims | Binding dimensions to set on the image input layer. | channel; height; width | infer-dims=3;224;224 | All Both | |
engine-create-func-name | Name of the custom TensorRT CudaEngine creation function. Refer to the “Custom Model Implementation Interface” section for details | String | engine-create-func-name=NvDsInferYoloCudaEngineGet | All Both | |
cluster-mode | Clustering algorithm to use. Refer to the next table for configuring the algorithm specific parameters. | Integer 0: OpenCV groupRectangles() 1: DBSCAN 2: Non Maximum Suppression 3:No clustering | cluster-mode=2 | Detector Both | |
filter-out-class-ids | Filter out detected objects belonging specified class-ids | Semicolon delimited integer array | filter-out-class-ids=1;2 | Detector Both | |
Gst-nvinfer plugin, [class-attrs-...] groups, supported keys | ||||
|---|---|---|---|---|
Detector or Classifier / Applicable to GIEs | ||||
Name | Description | Type and Range | Example Notes | (Primary/Seconday) |
threshold | Detection threshold | Float, ≥0 | threshold=0.5 | Object detector Both |
eps | Epsilon values for OpenCV grouprectangles() function and DBSCAN algorithm | Float, ≥0 | eps=0.2 | Object detector Both |
group-threshold | Threshold value for rectangle merging for OpenCV grouprectangles() function | Integer, ≥0 | group-threshold=1 0 disables the clustering functionality | Object detector Both |
minBoxes | Minimum number of points required to form a dense region for DBSCAN algorithm | Integer, ≥0 | minBoxes=1 0 disables the clustering functionality | Object detector Both |
dbscan-min-score | Minimum sum of confidence of all the neighbors in a cluster for it to be considered a valid cluster. | Float, ≥0 | dbscan-min-score=0.7 | Object detector Both |
nms-iou-threshold | Maximum IOU score between two proposals after which the proposal with the lower confidence will be rejected. | Float, ≥0 | nms-iou-threshold=0.2 | Object detector Both |
roi-top-offset | Offset of the RoI from the top of the frame. Only objects within the RoI are output. | Integer, ≥0 | roi-top-offset=200 | Object detector Both |
roi-bottom-offset | Offset of the RoI from the bottom of the frame. Only objects within the RoI are output. | Integer, ≥0 | roi-bottom-offset=200 | Object detector Both |
detected-min-w | Minimum width in pixels of detected objects to be output by the GIE | Integer, ≥0 | detected-min-w=64 | Object detector Both |
detected-min-h | Minimum height in pixels of detected objects to be output by the GIE | Integer, ≥0 | detected-min-h=64 | Object detector Both |
detected-max-w | Maximum width in pixels of detected objects to be output by the GIE | Integer, ≥0 | detected-max-w=200 0 disables the property | Object detector Both |
detected-max-h | Maximum height in pixels of detected objects to be output by the GIE | Integer, ≥0 | detected-max-h=200 0 disables the property | Object detector Both |
Gst Properties
The values set through Gst properties override the values of properties in the configuration file. The application does this for certain properties that it needs to set programmatically.
The following table describes the Gst-nvinfer plugin’s Gst properties.
Gst-nvinfer plugin, Gst properties | |||
|---|---|---|---|
Property | Meaning | Type and Range | Example Notes |
config-file-path | Absolute pathname of configuration file for the Gst-nvinfer element | String | config-file-path=config_infer_primary.txt |
process-mode | Infer Processing Mode 1=Primary Mode 2=Secondary Mode | Integer, 1 or 2 | process-mode=1 |
unique-id | Unique ID identifying metadata generated by this GIE | Integer, 0 to 4,294,967,295 | unique-id=1 |
infer-on-gie-id | See operate-on-gie-id in the configuration file table | Integer, 0 to 4,294,967,295 | infer-on-gie-id=1 |
operate-on-class-ids | See operate-on-class-ids in the configuration file table | An array of colon- separated integers (class-ids) | operate-on-class-ids=1:2:4 |
filter-out-class-ids | See filter-out-class-ids in the configuration file table | Semicolon delimited integer array | filter-out-class-ids=1;2 |
model-engine-file | Absolute pathname of the pre-generated serialized engine file for the mode | String | model-engine-file=model_b1_fp32.engine |
batch-size | Number of frames/objects to be inferred together in a batch | Integer, 1 – 4,294,967,295 | batch-size=4 |
Interval | Number of consecutive batches to be skipped for inference | Integer, 0 to 32 | interval=0 |
gpu-id | Device ID of GPU to use for pre-processing/inference (dGPU only) | Integer, 0-4,294,967,295 | gpu-id=1 |
raw-output-file-write | Pathname of raw inference output file | Boolean | raw-output-file-write=1 |
raw-output-generated-callback | Pointer to the raw output generated callback function | Pointer | Cannot be set through gst-launch |
raw-output-generated-userdata | Pointer to user data to be supplied with raw-output-generated-callback | Pointer | Cannot be set through gst-launch |
output-tensor-meta | Indicates whether to attach tensor outputs as meta on GstBuffer. | Boolean | output-tensor-meta=0 |
Tensor Metadata
The Gst-nvinfer plugin can attach raw output tensor data generated by a TensorRT inference engine as metadata. It is added as an NvDsInferTensorMeta in the frame_user_meta_list member of NvDsFrameMeta for primary (full-frame) mode, or in the obj_user_meta_list member of NvDsObjectMeta for secondary (object) mode.
To read or parse inference raw tensor data of output layers
1. Enable property output-tensor-meta, or enable the same-named attribute in the configuration file for the Gst-nvinfer plugin.
2. When operating as primary GIE, NvDsInferTensorMeta is attached to each frame’s (each NvDsFrameMeta object’s) frame_user_meta_list. When operating as secondary GIE, NvDsInferTensorMeta is attached to each each NvDsObjectMeta object’s obj_user_meta_list.
Metadata attached by Gst-nvinfer can be accessed in a GStreamer pad probe attached downstream from the Gst-nvinfer instance.
3. The NvDsInferTensorMeta object’s metadata type is set to NVDSINFER_TENSOR_OUTPUT_META. To get this metadata you must iterate over the NvDsUserMeta user metadata objects in the list referenced by frame_user_meta_list or obj_user_meta_list.
For more information about Gst-infer tensor metadata usage, see the source code in sources/apps/sample_apps/deepstream_infer_tensor_meta-test.cpp, provided in the DeepStream SDK samples.
Segmentation Metadata
The Gst-nvinfer plugin attaches the output of the segmentation model as user meta in an instance of NvDsInferSegmentationMeta with meta_type set to NVDSINFER_SEGMENTATION_META. The user meta is added to the frame_user_meta_list member of NvDsFrameMeta for primary (full-frame) mode, or the obj_user_meta_list member of NvDsObjectMeta for secondary (object) mode.
For guidance on how to access user metadata, see User/Custom Metadata Addition Inside NvDsMatchMeta and Tensor Metadata, above.
Gst-nvinferserver
The Gst-nvinferserver plugin does inferencing on input data using NVIDIA® TensorRT Inference Server (TRT-IS). Refer to https://docs.nvidia.com/deeplearning/sdk/tensorrt-inference-server-guide/docs/index.html for TensorRT Inference Server documentation.
The plugin accepts batched NV12/RGBA buffers from upstream. The NvDsBatchMeta structure must already be attached to the Gst Buffers.
The low-level library (libnvds_infer_server) operates on any of NV12 or RGBA buffers.
The Gst-nvinferserver plugin passes the input batched buffers to the low-level library and waits for the results to be available. Meanwhile, it keeps queuing input buffers to the low-level library as they are recieved. Once the results are available from the low-level library, the plugin translates and attaches the results back in to gst-buffer for downstream plugins.
The low-level library preprocesses the transformed frames (performs color conversion and scaling, normalization and mean subtraction) and produces final FP32/FP16/INT8/UINT8/INT16/UINT16/INT32/UINT32 RGB/BGR/GRAY planar/packed data which is passed to the TensorRT Inference server(TRT-IS) for inferencing. The output type generated by the low-level library depends on the network type.
The pre-processing function is:
Where:
• x is the input pixel value. It is an uint8 with range [0,255].
• mean is the corresponding mean value, read either from the mean file or as offsets[c], where c is the channel to which the input pixel belongs, and offsets is the array specified in the configuration file. It is a float.
• net-scale-factor is the pixel scaling factor specified in the configuration file. It is a float.
• y is the corresponding output pixel value. It can be of type float / half / int8 / uint8 / int16 / uint16 / int32 / uint32.
• Take specific example for uint8 to int8 conversion. set netscalefactor = 1.0 and mean = [128, 128, 128]. then the function is as such
Gst-nvinferserver currently works on the following type of networks:
• Multi-class object detection
• Multi-label classification
• Segmentation
The Gst-nvinferserver plugin can work in two modes:
• Primary mode: Operates on full frames
• Secondary mode: Operates on objects added in the meta by upstream components
When the plugin is operating as a secondary classifier in async mode along with the tracker, it tries to improve performance by avoiding re-inferencing on the same objects in every frame. It does this by caching the classification output in a map with the object’s unique ID as the key. The object is inferred upon only when it is first seen in a frame (based on its object ID) or when the size (bounding box area) of the object increases by 20% or more. This optimization is possible only when the tracker is added as an upstream element.
Detailed documentation of the TensorRT Inference Server(TRT-IS) is available at:
https://docs.nvidia.com/deeplearning/sdk/tensorrt-inference-server-guide/docs/index.html
The plugin supports TRT-IS features along with multiple deep-learning frameworks such as TensorRT, TensorFlow (GraphDef / SavedModel), ONNX and PyTorch on Telsa platforms. On Jetson, it also supports TensorRT and TensorFlow (GraphDef / SavedModel). Tensorflow and Onnx can configured with TensorRT acceleration. For details, see Framework-Specific Optimization.
The plugin needs a configurable a model repository root directory path where all the models need to reside. All the plugin instances in a single process must share the same model root. For details, see Model Repository. Each model also need a specific config.pbtxt file in its subdirectory. For details, see Model Configuration.
The plugin supports TRT-IS ensemble mode in the case users need do preprocessing or postprocessing with TRT-IS custom backend.
The plugin also supports the interface for custom functions for parsing outputs of object detectors, classifiers and initialization of non-image input layers in cases where there are more than one input layer.
Refer to sources/includes/nvdsinfer_custom_impl.h for the custom method implementations for custom models.
Downstream components receive a Gst Buffer with unmodified contents plus the metadata created from the inference output of the Gst-nvinferserver plugin.
The plugin can be used for cascaded inferencing. That is, it can perform primary inferencing directly on input data, then perform secondary inferencing on the results of primary inferencing, and so on. This is similar with Gst-nvinfer, see more details in Gst-nvinfer.
Inputs and Outputs
This section summarizes the inputs, outputs, and communication facilities of the Gst-nvinfer plugin.
• Inputs
• Gst Buffer
• NvDsBatchMeta (attaching NvDsFrameMeta)
• Model repository directory path (model_repo.root)
• Runtime model file with config.pbtxt file in model repository
• Control parameters:
• Gst-nvinferserver gets control parameters from a configuration file. You can specify this by setting the property config-file-path. For details, see Gst-nvinferserver File Configuration Specifications. Other control parameters that can be set through GObject properties are:
• Batch size
• Process mode
• Unique id
• Inference on GIE id and operate on class ids [secondary mode only]
• Inference interval
• raw output generated callback funtion
• The parameters set through the GObject properties override the parameters in the Gst-nvinfer configuration file.
• Outputs
• Gst Buffer
• Depending on network type and configured parameters, one or more of:
• NvDsObjectMeta
• NvDsClassifierMeta
• NvDsInferSegmentationMeta
• NvDsInferTensorMeta
Features
The following table summarizes the features of the plugin.
Features of the Gst-invinferserver plugin | ||||
Feature | dGPU | Jetson | Release | |
Gst-invinferserver Running on Host | No | Yes | DS 5.0 | |
Running on Docker Image | Yes | No | DS 5.0 | |
DS Preprocessing: Network input format: RGB/BGR/Gray | Yes | Yes | DS 5.0 | |
DS Preprocessing: Network input data types FP32/FP16/UINT8/INT8/UINT16/INT16/UINT32/INT32 | Yes | Yes | DS 5.0 | |
DS Preprocessing: Network input tensor orders NCHW / NHWC | Yes | Yes | DS 5.0 | |
Mem: Cuda(GPU) buf-sharing for Input Tensors | Yes | Yes | DS 5.0 | |
Mem: Cuda Memory(GPU / CPU-pinned) for output tensors | Yes | Yes | DS 5.0 | |
Backend: TensorRT runtime (plan engine file) | Yes | Yes | DS 5.0 | |
Backend: Tensorflow Runtime CPU/GPU (graphdef/savedmodel) | Yes | Yes | DS 5.0 | |
Backend: Tensorflow Runtime with TF-TRT acceleration | Yes | Yes | DS 5.0 | |
Backend: ONNX Runtime | Yes | No | DS 5.0 | |
Backend: ONNX Runtime with ONNX-TRT acceleration | Yes | No | DS 5.0 | |
Backend: Pytorch Runtime | Yes | No | DS 5.0 | |
Backend: Caffe2 Runtime | Yes (very few tests) | No | DS 5.0 | |
Postprocessing: DS Detection / Classification/ Segmentation | Yes | Yes | DS 5.0 | |
Postprocessing: DS Detection cluster method: NMS / GroupRectangle / DBSCan / None | Yes | Yes | DS 5.0 | |
Postprocessing: custom parsing (NvDsInferParseCustomTfSSD) | Yes | Yes | DS 5.0 | |
Postprocessing: TRT-IS native classification | Yes | Yes | DS 5.0 | |
TRT-IS Ensemble Mode(TRTIS preproc/postproc) with specified media-format(RGB/BGR/Gray) with Cuda GPU buffer as inputs | Yes | No | DS 5.0 | |
Postprocessing: Metadata NvDsInferTensorMeta | Yes | Yes | DS 5.0 | |
deepstream-app: pipeline works with PGIE / SGIE / nvtracker | Yes | Yes | DS 5.0 | |
Sample App: deepstream-segmentation-test / deepstream-infer-tensor-meta-test | Yes | Yes | DS 5.0 | |
Gst-nvinferserver File Configuration Specifications
The Gst-nvinferserver configuration file uses prototxt format described in:
https://developers.google.com/protocol-buffers
The protobuf message structures of this configuration file are defined by nvdsinferserver_plugin.proto and nvdsinferserver_config.proto. All the basic data-type values are set to 0 or false from protobuf’s guide. map, arrays and oneof are set to empty by default. See more details for each message definition.
• The message PluginControl in nvdsinferserver_plugin.proto is the entry point for this config-file.
• The message InferenceConfig configures the low-level settings for libnvds_infer_server.
• The message PluginControl::InputControl configures the input buffers, objects filtering policy for model inference.
• The message PluginControl::OutputControl configures inference output policy for detections and raw tensor metadata.
• The message BackendParams configures backend input/output layers and TRT-IS settings in InferenceConfig .
• The message PreProcessParams configures network preprocessing information in InferenceConfig .
• The message PostProcessParams configures the output tensor parsing methods such as detection, classification, segmentation and others in InferenceConfig .
• There are also other messages (e.g. CustomLib, ExtraControl, ...) and enum types (e.g. MediaFormat, TensorOrder, ...) defined in the proto file for miscellaneous settings for InferenceConfig and PluginControl.
Gst-nvinferserver plugin, message PluginControl definition details | |||||
Network Types / Applicable to GIEs | |||||
Fields | Meaning | Type and Range | Example Notes | (Primary/Seconday) | |
infer_config | Low-level libnvds_infer_server inference configuration settings | InferenceConfig | infer_config { … } see details in InferenceConfig | All Both | |
input_control | Control plugin input buffers, objects filtering policy for inference | PluginControl ::InputControl | input_control{ process_mode: PROCESS_MODE_FULL_FRAME } see details in InputControl | All Both | |
output_control | Control plugin output metadata filtering policy after inference | PluginControl ::OutputControl | output_control { … } see details in OutputControl | All Both | |
Gst-nvinferserver plugin, message PluginControl::InputControl definition details | |||||
Network Types / Applicable to GIEs | |||||
Property | Meaning | Type and Range | Example Notes | (Primary/Seconday) | |
process_mode | Processing mode, selected from PluginControl::ProcessMode. In deepstream-app PGIE uses PROCESS_MODE_FULL_FRAME by default, SGIE use PROCESS_MODE_CLIP_OBJECTS by default | enum PluginControl::ProcessMode | process_mode: PROCESS_MODE_FULL_FRAME | All Both | |
operate_on_gie_id | Unique ID of the GIE on whose metadata (bounding boxes) this GIE is to operate on | int32, >=0, valid gie-id. -1, disable gie-id check, inference on all GIE Ids | operate_on_gie_id: 1 | All Secondary | |
operate_on_class_ids | Class IDs of the parent GIE on which this GIE is to operate on | Comma delimited int32 array | operate_on_class_ids: [1, 2] Operates on objects with class IDs 1, 2 generated by parent GIE | All Secondary | |
interval | Specifies the number of consecutive, batches to be skipped for inference. default is 0 | uint32 | interval: 1 | All Primary | |
async_mode | Enables inference on detected objects and asynchronous metadata attachments. Works only when tracker-ids are attached. Pushes buffer downstream without waiting for inference results. Attaches metadata after the inference | bool | async_mode: false | Classifier Secondary | |
object_control | input object filter settings | PluginControl::InputObjectControl | object_control { bbox_filter { min_width: 64 min_height: 64 } } see details in InputObjectControl | All Secondary | |
Gst-nvinferserver plugin, message PluginControl::OutputControl definition details | |||||
Network Types / Applicable to GIEs | |||||
Property | Meaning | Type and Range | Example Notes | (Primary/Seconday) | |
output_tensor_meta | Enable attaching Inference output tensor metadata, tensor buffer pointer for host only | bool | output_tensor_meta: false | All Both | |
detect_control | Specifies detection output filter policy | PluginControl::OutputDetectionControl | detect_control { default_filter { bbox_filter { min_width: 32 min_height: 32 } } } see details in OutputDetectionControl | Detector Both | |
Gst-nvinferserver plugin, message PluginControl::InputObjectControl definition details | |||||
Network Types / Applicable to GIEs | |||||
Fields | Meaning | Type and Range | Example Notes | (Primary/Seconday) | |
bbox_filter | boudingbox filter | PluginControl::BBoxFilter | bbox_filter { min_width: 32 min_height: 32 } see details in BBoxFilter | All Secondary | |
Gst-nvinferserver plugin, message PluginControl::BBoxFilter definition details for Input and Output controls | |||||
Network Types / Applicable to GIEs | |||||
Fields | Meaning | Type and Range | Example Notes | (Primary/Seconday) | |
min_width | boudingbox minimum width | uint32 | min_width: 64 | All Both | |
min_height | boudingbox minimum height | uint32 | min_height: 64 | All Both | |
max_width | boudingbox maximum width, default 0, max_width is ignored | uint32 | max_width: 640 | All Both | |
max_height | boudingbox maximum height. default 0, max_height is ignored | uint32 | max_height: 640 | All Both | |
Gst-nvinferserver plugin, message PluginControl::OutputDetectionControl definition details | |||||
Network Types / Applicable to GIEs | |||||
Fields | Meaning | Type and Range | Example Notes | (Primary/Seconday) | |
default_filter | default detection filter for output controls | PluginControl::DetectClassFilter | default_filter { bbox_filter { min_width: 32 min_height: 32 } } see details in DetectClassFilter | All Both | |
specific_class_filters | specifies detection filters per class to replace default filter | map<uint32, DetectClassFilter> | specific_class_filters: [ { key: 1, value {...} }, { key: 2, value {...} } ] | All Both | |
Gst-nvinferserver plugin, message PluginControl::DetectClassFilter definition details | |||||
Network Types / Applicable to GIEs | |||||
Fields | Meaning | Type and Range | Example Notes | (Primary/Seconday) | |
bbox_filter | detection bounding box filter | PluginControl::BBoxFilter | bbox_filter { min_width: 64 min_height: 64 } | Detection Both | |
roi_top_offset | Offset of the RoI from the top of the frame. Only objects within the RoI are output. | uint32 | roi_top_offset: 128 | Detection Both | |
roi_bottom_offset | Offset of the RoI from the bottom of the frame. Only objects within the RoI are output. | uint32 | roi_bottom_offset: | Detection Both | |
border_color | specify border color for detection bounding boxes | PluginControl::Color | border_color { r: 1.0 g: 0.0 b: 0.0 a: 1.0 } | Detection Both | |
bg_color | specify backgrournd color for detection bounding boxes | PluginControl::Color | border_color { r: 0.0 g: 1.0 b: 0.0 a: 0.5 } | Detection Both | |
Gst-nvinferserver plugin, message PluginControl::Color definition details | |||||
Network Types / Applicable to GIEs | |||||
Fields | Meaning | Type and Range | Example Notes | (Primary/Seconday) | |
r | Red color value | float. Range[0.0, 1.0] | r: 0.5 | All Both | |
g | Green color value | float. Range[0.0, 1.0] | g: 0.5 | All Both | |
b | Blue color value | float. Range[0.0, 1.0] | b: 0.3 | All Both | |
a | Alpha blending value | float. Range[0.0, 1.0] | a: 1.0 | All Both | |
The message InferenceConfig defines all the low-level structure fields in nvdsinferserver_config.proto. It has major settings for inference backend, network preprocessing and postprocessing.
message InferenceConfig definition details | ||||
Detector or Classifier / Applicable to GIEs | ||||
Name | Description | Type and Range | Example Notes | (Primary/Seconday) |
unique_id | Unique ID identifying metadata generated by this GIE | uint32, ≥0 | unique_id: 1 | All Both |
gpu_ids | Device IDs of GPU to use for pre-processing/inference (single GPU supported only) | int32 array, ≥0 | gpu_ids: [0] | All Both |
max_batch_size | Max number of frames/objects to be inferred together in a batch | uint32, ≥0 | max_batch_size: 1 | All Both |
backend | Inference backend settings | BackendParams | backend { trt_is { ... } } see details in BackendParams | All Both |
preprocess | Network preprocessing setting for color conversion， scale and normalization | PreProcessParams | preprocess { normalize { … } } see details in PreProcessParams | All Both |
postprocess | Inference output tensor parsing methods such as detection, classification, segmentation and others | PostProcessParams | postprocess { detection {...} } see details in PostProcessParams | All Both |
custom_lib | Specify custom lib path for custom parsing functions and preloads, optional | CustomLib | custom_lib { path : ./libcustom_parsing.so } | All Both |
extra | extra controls for inference config. | ExtraControl | extra { output_buffer_pool_size: 2 } see details in ExtraControl | All Both |
message BackendParams definition details | ||||
Detector or Classifier / Applicable to GIEs | ||||
Name | Description | Type and Range | Example Notes | (Primary/Seconday) |
inputs | Backend input layer settings, optional | InputLayer arrays | see details in InputLayer | All Both |
outputs | Backend output layer settings, optional | OutputLayer arrays | see details in OutputLayer | All Both |
trt_is | backend of TRT-IS(TensorRT Inference Server) settings | TrtISParams | see details in TrtISParams | All Both |
message InputLayer definition details | ||||
Detector or Classifier / Applicable to GIEs | ||||
Name | Description | Type and Range | Example Notes | (Primary/Seconday) |
name | input tensor name | string | name: “input_0” | All Both |
dims | input tensor shape, optional. only required if backend cannot figure out fixed input shapes | int32 array, > 0 | dims: [299, 299, 3] | All Both |
data_type | enum TensorDataType with types: TENSOR_DT_NONE, TENSOR_DT_FP32, TENSOR_DT_FP16, TENSOR_DT_INT8, TENSOR_DT_INT16, TENSOR_DT_INT32, TENSOR_DT_UINT8, TENSOR_DT_UINT16, TENSOR_DT_UINT32 Default TENSOR_DT_NONE, usually can be deduced from TRT-IS model config.pbtxt | TensorDataType | data_type: TENSOR_DT_FP32 | All Both |
message OutputLayer definition details | ||||
Detector or Classifier / Applicable to GIEs | ||||
Name | Description | Type and Range | Example Notes | (Primary/Seconday) |
name | output tensor name | string | name: “detection_boxes” | All Both |
message TrtISParams definition details | ||||
Detector or Classifier / Applicable to GIEs | ||||
Name | Description | Type and Range | Example Notes | (Primary/Seconday) |
model_name | TRT-IS inference model name | string | model_name: “ssd_inception_graphdef” | All Both |
version | TRT-IS model version number. -1, latest version number. >0, reserved for specific version number in future version | int64 | version: -1 | All Both |
model_repo | TRT-IS model repository settings. Note, all model_repo settings must be same in single process | TrtISParams::ModelRepo | model_repo { root: “../trtis_model_repo” log_level: 2 } see more details in ModelRepo | All Both |
message TrtISParams::ModelRepo definition details | ||||
Detector or Classifier / Applicable to GIEs | ||||
Name | Description | Type and Range | Example Notes | (Primary/Seconday) |
root | TRT-IS inference model repository directory path | string | root: “../trtis_model_repo” | All Both |
log_level | TRT-IS log output levels | uint32; 0, ERROR; 1, WARNING; 2, INFO; >=3, VERBOSE Level | log_level: 1 | All Both |
strict_model_config | enable TRT-IS strict model configuration, see details in TRT-IS Generated Model Configuration. Suggest to set value true | bool | strict_model_config: true | All Both |
tf_gpu_memory_fraction | TensorFlow GPU memory fraction per process. Valid for Tensorflow models only. Default 0 means no GPU memory limitation. Suggest to tune a proper value(e.g. 0.6) in case Tensorflow use up whole GPU memory | float, Range (0, 1.0] | tf_gpu_memory_fraction: 0.6 | All Both |
tf_disable_soft_placement | Disable TensorFlow soft-placement of operators. It’s enabled by default. | bool | tf_disable_soft_placement: false | All Both |
message PreProcessParams definition details | ||||
Detector or Classifier / Applicable to GIEs | ||||
Name | Description | Type and Range | Example Notes | (Primary/Seconday) |
network_format | enum MediaFormat with formats: MEDIA_FORMAT_NONE IMAGE_FORMAT_RGB IMAGE_FORMAT_BGR IMAGE_FORMAT_GRAY . use IMAGE_FORMAT_RGB by default. | MediaFormat | network_format: IMAGE_FORMAT_RGB | All Both |
tensor_order | enum TensorOrder with order types: TENSOR_ORDER_NONE, TENSOR_ORDER_LINEAR, TENSOR_ORDER_NHWC. It can deduce the value from backend layers info if set to TENSOR_ORDER_NONE | TensorOrder | tensor_order: TENSOR_ORDER_NONE | All Both |
tensor_name | Specify the tensor name for the preprocessing buffer. This is in the case when multiple input tensors in a single network. | string; Optional | tensor_name: “input_0” | All Both |
maintain_aspect_ratio | Indicates whether to maintain aspect ratio while scaling input. | int32; 0 or 1 | maintain_aspect_ratio: 0 | All Both |
normalize | Network input tesor normalization settings for scale-factors, offsets and mean-subtraction | PreProcessParams::ScaleNormalize | normalize { scale_factor: 1.0 channel_offsets: [0, 0, 0] } see details in PreProcessParams::ScaleNormalize | |
message PreProcessParams::ScaleNormalize definition details | ||||
Detector or Classifier / Applicable to GIEs | ||||
Name | Description | Type and Range | Example Notes | (Primary/Seconday) |
scale_factor | Pixel normalization factor | float | scale_factor: 0.0078 | All Both |
channel_offsets | Array of mean values of color components to be subtracted from each pixel. Array length must equal the number of color components in the frame. The plugin multiplies mean values by scale_factor. | float array, Optional | channel_offsets: [77.5, 21.2, 11.8] | All Both |
mean_file | Pathname of mean data file (PPM format) | string; Optional | mean_file: “./model_meanfile.ppm” | All Both |
message PostProcessParams definition details | ||||
Detector or Classifier / Applicable to GIEs | ||||
Name | Description | Type and Range | Example Notes | (Primary/Seconday) |
labelfile_path | Pathname of a text file containing the labels for the model | string | labelfile_path: “=/home/ubuntu/model_labels.txt” | All Both |
oneof process_type | Indicates one of the postprocessing type detection; classification; segmentation; other; | None | N/A | All Both |
detection | Specify detection parameters for the network. It must be oneof process_type | DetectionParams | detection { num_detected_classes: 4 simple_cluster { threshold: 0.2 } } see details in DetectionParams | Detector Both |
classification | Specify classification parameters for the network It is neof process_type | ClassificationParams | classification { threshold: 0.6 } see details in ClassificationParams | Classifier Both |
segmentation | Specify segmentation parameters for the network It is oneof process_type | SegmentationParams | segmentation { threshold: 0.2 } | Segmentation Both |
other | Specify other network parameters. This is for user-defined networks and usually coexists with output_control.output_tensor_meta: true. Tensor output data would be attached into GstBuffer. Data can be parsed in application. User can increase extra.output_buffer_pool_size if need to hold metadata longer. It is oneof process_type | OtherNetworkParams | other {} see details in OtherNetworkParams | Others Both |
trtis_classification | Specify TRT-IS classification parameters for the network It is oneof process_type | TrtIsClassifyParams | trtis_classification { topk: 1 } see details in TrtIsClassifyParams | Classifier Both |
message DetectionParams definition details | ||||
Detector or Classifier / Applicable to GIEs | ||||
Name | Description | Type and Range | Example Notes | (Primary/Seconday) |
num_detected_classes | Define number of classes detected by the network | int32, > 0 | num_detected_classes:4 | Detector Both |
per_class_params | Map of specific detection parameters per class. Key-value follows <class_id: per_class_params> order. | map<int32, PerClassParams>; Optional | per_class_params { { key: 1, value { pre_threshold : 0.4} }, { key: 2, value { pre_threshold : 0.5} } } see details for PerClassParams | Detector Both |
custom_parse_bbox_func | Name of the custom bounding box parsing function. If not specified, Gst-nvinferserver uses the internal function for the resnet model provided by the SDK. If specified, also need to set custom_lib to load custom library. | string; | custom_parse_bbox_func: "NvDsInferParseCustomTfSSD" | Detector Both |
oneof clustering_policy | Indicates one of the clustering policies from nms; dbscan; group_rectangle; simple_cluster; | None | N/A | Detector Both |
nms | Indicates clustering bounding boxes by Non-Maximum-Suppression method detected objects. It is oneof clustering_policy | Nms | nms { confidence_threshold: 0.3 iou_threshold: 0.4 } see details in Nms | Detector Both |
dbscan | Indicates clustering bounding boxes by DBSCAN method for detected objects. It is oneof clustering_policy | DbScan | dbscan { pre_threshold: 0.3 eps: 0.7 min_boxes: 3 } see details in DbScan | Detector Both |
group_rectangle | Indicates clustering bounding boxes by groupRectangles() function for grouping detected objects It is oneof clustering_policy | GroupRectangle | group_rectangle { confidence_threshold: 0.2 group_threshold: 2 eps: 0.2 } | Detector Both |
simple_cluster | Indicates simple clustering method by outlier boxes through threshold | SimpleCluster | simple_cluster { threshold: 0.2 } | Detector Both |
message DetectionParams::PerClassParams definition details | ||||
Detector or Classifier / Applicable to GIEs | ||||
Name | Description | Type and Range | Example Notes | (Primary/Seconday) |
pre_threshold | Define confidence threshold per class | float | pre_threshold:0.3 | Detector Both |
message DetectionParams::Nms definition details | ||||
Detector or Classifier / Applicable to GIEs | ||||
Name | Description | Type and Range | Example Notes | (Primary/Seconday) |
confidence_threshold | Detection score less this threshold would be rejected | float | confidence_threshold:0.5 | Detector Both |
iou_threshold | Maximum IOU score between two proposals after which the proposal with the lower confidence will be rejected. | float | iou_threshold: 0.3 | Detector Both |
topk | Specify top k detection results to keep after nms | int32, >= 0 | topk: 2; value 0, means keep all. | Detector Both |
message DetectionParams::DbScan definition details | ||||
Detector or Classifier / Applicable to GIEs | ||||
Name | Description | Type and Range | Example Notes | (Primary/Seconday) |
pre_threshold | Detection score less this threshold would be rejected before DBSCAN clustering | float | pre_threshold:0.2 | Detector Both |
eps | DBSCAN epsilon to control merging of overlapping boxes. | float | eps: 0.7 | Detector Both |
min_boxes | Minimum boxes in DBSCAN cluster to be considered an object | int32, > 0 | min_boxes: 3; | Detector Both |
min_score | Minimum score in DBSCAN cluster for it to be considered as an object | float | min_score: 0.7 Default value is 0 | Detector Both |
message DetectionParams::GroupRectangle definition details | ||||
Detector or Classifier / Applicable to GIEs | ||||
Name | Description | Type and Range | Example Notes | (Primary/Seconday) |
confidence_threshold | Detection score less this threshold would be rejected | float | confidence_threshold:0.2 | Detector Both |
group_threshold | Threshold value for rectangle merging for OpenCV grouprectangles() function | int32; >= 0 | group_threshold: 1 | Detector Both |
eps | Epsilon to control merging of overlapping boxes | float | eps: 0.2 | Detector Both |
message DetectionParams::SimpleCluster definition details | ||||
Detector or Classifier / Applicable to GIEs | ||||
Name | Description | Type and Range | Example Notes | (Primary/Seconday) |
threshold | Detection score less this threshold would be rejected | float | confidence_threshold:0.6 | Detector Both |
message ClassificationParams definition details | ||||
Detector or Classifier / Applicable to GIEs | ||||
Name | Description | Type and Range | Example Notes | (Primary/Seconday) |
threshold | Classification score less this threshold would be rejected | float | threshold: 0.5 | Classifier Both |
custom_parse_classifier_func | Name of the custom classifier output parsing function. If not specified, Gst-nvinfer uses the internal parsing function with NCHW tensor order for softmax layers. User can reshape other output tensor order to NCHW in TRTIS config.pbtxt to run internal parsing. If specified, also need to set custom_lib to load custom library. | string | parse-classifier-func-name: “parse_bbox_softmax” | Classifier Both |
message SegmentationParams definition details | ||||
Detector or Classifier / Applicable to GIEs | ||||
Name | Description | Type and Range | Example Notes | (Primary/Seconday) |
threshold | Segmentation score less this threshold would be rejected | float | threshold: 0.5 | Segmentation Both |
message OtherNetworkParams definition details | ||||
Detector or Classifier / Applicable to GIEs | ||||
Name | Description | Type and Range | Example Notes | (Primary/Seconday) |
type_name | Specify a user-defined network name | string; Optinal | type_name: “face” | Others Both |
message TrtIsClassifyParams definition details | ||||
Detector or Classifier / Applicable to GIEs | ||||
Name | Description | Type and Range | Example Notes | (Primary/Seconday) |
topk | Specify top k elements need to keep from TRT-IS’s native classification | uint32; >=0 | topk : 1 Value 0 or empty would keep the top 1 result. | Classifier Both |
threshold | Classification score less this threshold would be rejected | float | threshold: 0.5 | Classifier Both |
message CustomLib definition details | ||||
Detector or Classifier / Applicable to GIEs | ||||
Name | Description | Type and Range | Example Notes | (Primary/Seconday) |
path | Pathname point a custom library for preload | string | path: “/home/ubuntu/lib_custom_impl.so” | All Both |
message ExtraControl definition details | ||||
Detector or Classifier / Applicable to GIEs | ||||
Name | Description | Type and Range | Example Notes | (Primary/Seconday) |
copy_input_to_host_buffers | Enable to copy input tensor data to host buffers. If enabled, input tensor would attached as NvDsInferTensorMeta into GstBuffer with output tensors together | bool | copy_input_to_host_buffers: false | All Both |
output_buffer_pool_size | Specify the buffer pool size for each output tensor. When infer_config.postprocess.other is specified or output_control.output_tensor_meta is enabled, the ouptut tensor would be attached as NvDsInferTensorMeta into GstBuffer | int32; Range [2, 10] | output_buffer_pool_size: 4 | All Both |
Gst Properties
The values set through Gst properties override the values of properties in the configuration file. The application does this for certain properties that it needs to set programmatically. If user set property though plugin, these values would replace the original value in config files.
The following table describes the Gst-nvinferserver plugin’s Gst properties.
Gst-nvinferserver plugin, Gst properties | |||
Property | Meaning | Type and Range | Example Notes |
config-file-path | Absolute pathname of configuration file for the Gst-nvinfer element | String | config-file-path=config_infer_primary.txt |
process-mode | Infer Processing Mode (0):None, (1)FullFrame, (2)ClipObject. If set, it could replace input_control.process_mode | Integer, 0, 1 or 2 | process-mode=1 |
unique-id | Unique ID identifying metadata generated by this GIE. If set, it could replace infer_config.unique_id | Integer, 0 to 4,294,967,295 | unique-id=1 |
infer-on-gie-id | See input_control.operate_on_gie_id in the configuration file table | Integer, 0 to 4,294,967,295 | infer-on-gie-id=1 |
operate-on-class-ids | See input_control.operate_on_class_ids in the configuration file table | An array of colon- separated integers (class-ids) | operate-on-class-ids=1:2:4 |
batch-size | Number of frames/objects to be inferred together in a batch. If set, it could replace infer_config.max_batch_size | Integer, 1 – 4,294,967,295 | batch-size=4 |
Interval | Number of consecutive batches to be skipped for inference If set, it could replace input_control.interval | Integer, 0 to 32 | interval=0 |
raw-output-generated-callback | Pointer to the raw output generated callback function | Pointer | Cannot be set through gst-launch |
raw-output-generated-userdata | Pointer to user data to be supplied with raw-output-generated-callback | Pointer | Cannot be set through gst-launch |
Tensor Metadata
The Gst-nvinferserver plugin can attach raw output tensor data generated by the inference backend as metadata. It is added as an NvDsInferTensorMeta in the frame_user_meta_list member of NvDsFrameMeta for primary (full-frame) mode, or in the obj_user_meta_list member of NvDsObjectMeta for secondary (object) mode. It use same metadata structure with Gst-nvinfer plugin.
Note: Gst-nvinferserver plugin does not attach device buffer pointer NvDsInferTensorMeta::attach out_buf_ptrs_dev at this moment.
To read or parse inference raw tensor data of output layers
1. Enable
output_control { output_tensor_meta : true }
Or update
infer_config { postprocess { other {} } }
fields in the configuration file for the Gst-nvinferserver plugin.
2. When operating as primary GIE, NvDsInferTensorMeta is attached to each frame’s (each NvDsFrameMeta object’s) frame_user_meta_list. When operating as secondary GIE, NvDsInferTensorMeta is attached to each NvDsObjectMeta object’s obj_user_meta_list.
Metadata attached by Gst-nvinfer can be accessed in a GStreamer pad probe attached downstream from the Gst-nvinfer instance.
3. The NvDsInferTensorMeta object’s metadata type is set to NVDSINFER_TENSOR_OUTPUT_META. To get this metadata you must iterate over the NvDsUserMeta user metadata objects in the list referenced by frame_user_meta_list or obj_user_meta_list.
For more information about Gst-infer tensor metadata usage, see the source code in sources/apps/sample_apps/deepstream_infer_tensor_meta-test.cpp, provided in the DeepStream SDK samples.
Segmentation Metadata
The Gst-nvinferserver plugin attaches the output of the segmentation model as user meta in an instance of NvDsInferSegmentationMeta with meta_type set to NVDSINFER_SEGMENTATION_META. The user meta is added to the frame_user_meta_list member of NvDsFrameMeta for primary (full-frame) mode, or the obj_user_meta_list member of NvDsObjectMeta for secondary (object) mode.
For guidance on how to access user metadata, see User/Custom Metadata Addition Inside NvDsMatchMeta and Tensor Metadata, above.