NVIDIA DeepStream SDK API Reference

6.0 Release
 All Data Structures Namespaces Files Functions Variables Typedefs Enumerations Enumerator Macros Groups Pages
Data Structures
Inference Interface API

Detailed Description

Defines the DeepStream inference interface API.

In C++, defines the NvDsInferContext class.

The DeepStream inference API "NvDsInfer" provides methods to initialize and deinitialize the inference engine, pre-process the input frames as required by the network, and parse the output from the raw tensor buffers.

Both C and C++ interfaces are available, with the C interface being a simple wrapper over the C++ interface.

You can create an opaque handle to an instance of the context required by the API by calling the factory function createNvDsInferContext() or NvDsInferContext_Create(). Both functions accept an instance of NvDsInferContextInitParams to initialize the context. Both let you specify a logging callback to get detailed information about failures and warnings.

Initialization parameters allow you to configure the network data type, network type (Detector, Classifier, or Other), preprocessing parameters (mean subtraction and normalization), model-related parameters like Caffe/Uff/Onnx model file paths, output layer names, etc.

Batches of frames can be queued for inferencing, using NvDsInferContext::queueInputBatch() or NvDsInferContext_QueueInputBatch(). The input frame memories must be accessible to the GPU device configured during initialization. You can provide an asynchronous callback function to return the input buffers to the caller as soon as the input is consumed.

Inference output can be dequeued using NvDsInferContext::dequeueOutputBatch() or NvDsInferContext_DequeueOutputBatch(). The order of dequeued outputs corresponds to the input queueing order. In case of failure, the output of the batch is lost. The dequeued output must be released back to the context using NvDsInferContext::releaseBatchOutput() or NvDsInferContext_ReleaseBatchOutput() to free the associated memory and return the output layer buffers for reuse by the context.

Detectors output an array of detected objects for each frame in the batch. Classifiers classify entire frames and output an array of attributes for each frame. Segmentation classifies each pixel in the frame. A special network type (Other) has been provided whose output layers are not parsed. The caller can parse the device and host output layer buffers. You can also use this network type with the Gst-infer plugin to flow the output buffers as metadata.

Other methods and functions get parsed labels from a label's file and properties of all layers bound by the inference engine.

You can extend the Gst-nvinfer API using the custom method implementations. Refer to the Custom Method Implementations section for more details.

Data Structures

struct  NvDsInferDetectionParams
 Holds detection and bounding box grouping parameters. More...
 
struct  _NvDsInferContextInitParams
 Holds the initialization parameters required for the NvDsInferContext interface. More...
 
struct  NvDsInferContextBatchInput
 Holds information about one batch to be inferred. More...
 
struct  NvDsInferContextBatchPreprocessedInput
 
struct  NvDsInferObject
 Holds information about one detected object. More...
 
struct  NvDsInferDetectionOutput
 Holds information on all objects detected by a detector network in one frame. More...
 
struct  NvDsInferClassificationOutput
 Holds information on all attributes classifed by a classifier network for one frame. More...
 
struct  NvDsInferSegmentationOutput
 Holds information parsed from segmentation network output for one frame. More...
 
struct  NvDsInferFrameOutput
 Holds the information inferred by the network on one frame. More...
 
struct  NvDsInferContextBatchOutput
 Holds the output for all of the frames in a batch (an array of frame), and related buffer information. More...
 

NvDsInferContext API common types and functions.

This section describes the common types and functions for both the C and C++ interfaces for the NvDsInferContext class.

enum  NvDsInferNetworkMode {
  NvDsInferNetworkMode_FP32,
  NvDsInferNetworkMode_INT8,
  NvDsInferNetworkMode_FP16
}
 Defines internal data formats used by the inference engine. More...
 
enum  NvDsInferNetworkType {
  NvDsInferNetworkType_Detector,
  NvDsInferNetworkType_Classifier,
  NvDsInferNetworkType_Segmentation,
  NvDsInferNetworkType_InstanceSegmentation,
  NvDsInferNetworkType_Other = 100
}
 Defines network types. More...
 
enum  NvDsInferFormat {
  NvDsInferFormat_RGB,
  NvDsInferFormat_BGR,
  NvDsInferFormat_GRAY,
  NvDsInferFormat_RGBA,
  NvDsInferFormat_BGRx,
  NvDsInferFormat_Tensor,
  NvDsInferFormat_Unknown = 0xFFFFFFFF
}
 Defines color formats. More...
 
enum  NvDsInferTensorOrder {
  NvDsInferTensorOrder_kNCHW,
  NvDsInferTensorOrder_kNHWC,
  NvDsInferTensorOrder_kNC,
  NvDsInferUffOrder_kNCHW =("Use NvDsInferTensorOrder_kNCHW instead") = NvDsInferTensorOrder_kNCHW,
  NvDsInferUffOrder_kNHWC =("Use NvDsInferTensorOrder_kNCWH instead") = NvDsInferTensorOrder_kNHWC,
  NvDsInferUffOrder_kNC =("Use NvDsInferTensorOrder_kNC instead") = NvDsInferTensorOrder_kNC
}
 Defines UFF input layer orders. More...
 
enum  NvDsInferClusterMode {
  NVDSINFER_CLUSTER_GROUP_RECTANGLES = 0,
  NVDSINFER_CLUSTER_DBSCAN,
  NVDSINFER_CLUSTER_NMS,
  NVDSINFER_CLUSTER_DBSCAN_NMS_HYBRID,
  NVDSINFER_CLUSTER_NONE
}
 Enum for clustering mode for detectors. More...
 
typedef struct
_NvDsInferContextInitParams 		NvDsInferContextInitParams
 Holds the initialization parameters required for the NvDsInferContext interface. More...
 
typedef void(* NvDsInferContextReturnInputAsyncFunc )(void *data)
 Defines a callback function type for asynchronously returning the input client buffers to the NvDsInferContext client. More...
 
typedef struct INvDsInferContext * NvDsInferContextHandle
 An opaque pointer type to be used as a handle for a context instance. More...
 
typedef void(* NvDsInferContextLoggingFunc )(NvDsInferContextHandle handle, unsigned int uniqueID, NvDsInferLogLevel logLevel, const char *logMessage, void *userCtx)
 Type declaration for a logging callback. More...
 
void NvDsInferContext_ResetInitParams (NvDsInferContextInitParams *initParams)
 Resets a context parameter structure to default values. More...
 
NvDsInferContext_GetStatusName
is deprecated Use
NvDsInferStatus2Str instead
const char * 		NvDsInferContext_GetStatusName (NvDsInferStatus status)
 Gets the string name of the status. More...
 
#define _PATH_MAX   4096
 Maximum length of a file path parameter. More...
 
#define _MAX_CHANNELS   4
 Defines the maximum number of channels supported by the API for image input layers. More...
 
#define _MAX_STR_LENGTH   1024
 Defines the maximum length of string parameters. More...
 
#define NVDSINFER_MAX_BATCH_SIZE   1024
 Defines the maximum batch size supported by nvdsinfer. More...
 
#define NVDSINFER_MIN_OUTPUT_BUFFERPOOL_SIZE   2
 Defines the minimum number of sets of output buffers that must be allocated. More...
 
#define NvDsInferUffOrder
 

NvDsInferContext API C-interface

This section describes the C interface for the NvDsInferContext class.

NvDsInferStatus NvDsInferContext_Create (NvDsInferContextHandle *handle, NvDsInferContextInitParams *initParams, void *userCtx, NvDsInferContextLoggingFunc logFunc)
 Creates a new NvDsInferContext object with specified initialization parameters. More...
 
void NvDsInferContext_Destroy (NvDsInferContextHandle handle)
 Destroys an NvDsInferContext instance and releases its resources. More...
 
NvDsInferStatus NvDsInferContext_QueueInputBatch (NvDsInferContextHandle handle, NvDsInferContextBatchInput *batchInput)
 Queues a batch of input frames for preprocessing and inferencing. More...
 
NvDsInferStatus NvDsInferContext_DequeueOutputBatch (NvDsInferContextHandle handle, NvDsInferContextBatchOutput *batchOutput)
 Dequeues output for a batch of frames. More...
 
void NvDsInferContext_ReleaseBatchOutput (NvDsInferContextHandle handle, NvDsInferContextBatchOutput *batchOutput)
 Frees the memory associated with the batch output and releases the set of host buffers back to the context for reuse. More...
 
void NvDsInferContext_GetNetworkInfo (NvDsInferContextHandle handle, NvDsInferNetworkInfo *networkInfo)
 Gets network input information. More...
 
unsigned int NvDsInferContext_GetNumLayersInfo (NvDsInferContextHandle handle)
 Gets the number of the bound layers of the inference engine in an NvDsInferContext instance. More...
 
void NvDsInferContext_FillLayersInfo (NvDsInferContextHandle handle, NvDsInferLayerInfo *layersInfo)
 Fills an input vector with information about all of the bound layers of the inference engine in an NvDsInferContext instance. More...
 
const char * NvDsInferContext_GetLabel (NvDsInferContextHandle handle, unsigned int id, unsigned int value)
 Gets the string label associated with the class ID for detectors and the attribute ID and attribute value for classifiers. More...
 

Macro Definition Documentation

#define _MAX_CHANNELS   4

Defines the maximum number of channels supported by the API for image input layers.

Definition at line 100 of file nvdsinfer_context.h.

#define _MAX_STR_LENGTH   1024

Defines the maximum length of string parameters.

Definition at line 103 of file nvdsinfer_context.h.

#define _PATH_MAX   4096

Maximum length of a file path parameter.

Definition at line 96 of file nvdsinfer_context.h.

#define NVDSINFER_MAX_BATCH_SIZE   1024

Defines the maximum batch size supported by nvdsinfer.

Definition at line 106 of file nvdsinfer_context.h.

#define NVDSINFER_MIN_OUTPUT_BUFFERPOOL_SIZE   2

Defines the minimum number of sets of output buffers that must be allocated.

Definition at line 110 of file nvdsinfer_context.h.

#define NvDsInferUffOrder
Value:
_Pragma \
("GCC warning \"'NvDsInferUffOrder' macro is deprecated. Use NvDsInferTensorOrder instead.\"") \
NvDsInferTensorOrder
NvDsInferTensorOrder
NvDsInferTensorOrder
Defines UFF input layer orders.
Definition: nvdsinfer_context.h:170

Definition at line 179 of file nvdsinfer_context.h.

Typedef Documentation

typedef struct INvDsInferContext* NvDsInferContextHandle

An opaque pointer type to be used as a handle for a context instance.

Definition at line 584 of file nvdsinfer_context.h.

typedef struct _NvDsInferContextInitParams NvDsInferContextInitParams

Holds the initialization parameters required for the NvDsInferContext interface.

typedef void(* NvDsInferContextLoggingFunc)(NvDsInferContextHandle handle, unsigned int uniqueID, NvDsInferLogLevel logLevel, const char *logMessage, void *userCtx)

Type declaration for a logging callback.

The callaback log NvDsInferContext messages.

Parameters
[in]handleThe handle of the NvDsInferContext instance that generated the log.
[in]uniqueIDUnique ID of the NvDsInferContext instance that generated the log.
[in]logLevelLevel of the log.
[in]funcNameA pointer to the name of the function that generated the log.
[in]logMessageA pointer to the log message string.
[in]userCtxAn opaque pointer to the user context, supplied when creating the NvDsInferContext instance.

Definition at line 602 of file nvdsinfer_context.h.

typedef void(* NvDsInferContextReturnInputAsyncFunc)(void *data)

Defines a callback function type for asynchronously returning the input client buffers to the NvDsInferContext client.

Parameters
[in]dataAn opaque pointer provided to the input queueing function through NvDsInferContextBatchInput.

Definition at line 418 of file nvdsinfer_context.h.

Enumeration Type Documentation

enum NvDsInferClusterMode

Enum for clustering mode for detectors.

Enumerator
NVDSINFER_CLUSTER_GROUP_RECTANGLES 
NVDSINFER_CLUSTER_DBSCAN 
NVDSINFER_CLUSTER_NMS 
NVDSINFER_CLUSTER_DBSCAN_NMS_HYBRID 
NVDSINFER_CLUSTER_NONE 

Definition at line 222 of file nvdsinfer_context.h.

enum NvDsInferFormat

Defines color formats.

Enumerator
NvDsInferFormat_RGB 

Specifies 24-bit interleaved R-G-B format.

NvDsInferFormat_BGR 

Specifies 24-bit interleaved B-G-R format.

NvDsInferFormat_GRAY 

Specifies 8-bit Luma format.

NvDsInferFormat_RGBA 

Specifies 32-bit interleaved R-G-B-A format.

NvDsInferFormat_BGRx 

Specifies 32-bit interleaved B-G-R-x format.

NvDsInferFormat_Tensor 

NCHW planar.

NvDsInferFormat_Unknown 

Definition at line 150 of file nvdsinfer_context.h.

enum NvDsInferNetworkMode

Defines internal data formats used by the inference engine.

Enumerator
NvDsInferNetworkMode_FP32 
NvDsInferNetworkMode_INT8 
NvDsInferNetworkMode_FP16 

Definition at line 115 of file nvdsinfer_context.h.

enum NvDsInferNetworkType

Defines network types.

Enumerator
NvDsInferNetworkType_Detector 

Specifies a detector.

Detectors find objects and their coordinates, and their classes in an input frame.

NvDsInferNetworkType_Classifier 

Specifies a classifier.

Classifiers classify an entire frame into one of several classes.

NvDsInferNetworkType_Segmentation 

Specifies a segmentation network.

A segmentation network classifies each pixel into one of several classes.

NvDsInferNetworkType_InstanceSegmentation 

Specifies a instance segmentation network.

A instance segmentation network detects objects, bounding box and mask for objects, and their classes in an input frame

NvDsInferNetworkType_Other 

Specifies other.

Output layers of an "other" network are not parsed by NvDsInferContext. This is useful for networks that produce custom output. Output can be parsed by the NvDsInferContext client or can be combined with the Gst-nvinfer feature to flow output tensors as metadata.

Definition at line 125 of file nvdsinfer_context.h.

enum NvDsInferTensorOrder

Defines UFF input layer orders.

Enumerator
NvDsInferTensorOrder_kNCHW 
NvDsInferTensorOrder_kNHWC 
NvDsInferTensorOrder_kNC 
NvDsInferUffOrder_kNCHW 
NvDsInferUffOrder_kNHWC 
NvDsInferUffOrder_kNC 

Definition at line 170 of file nvdsinfer_context.h.

Function Documentation

NvDsInferStatus NvDsInferContext_Create ( NvDsInferContextHandle *  handle,
NvDsInferContextInitParams *  initParams,
void *  userCtx,
NvDsInferContextLoggingFunc  logFunc 
)

Creates a new NvDsInferContext object with specified initialization parameters.

Parameters
[out]handleA pointer to an NvDsInferContext handle.
[in]initParamsA pointer to a parameter structure to be used to initialize the context.
[in]userCtxA pointer to an opaque user context, with callbacks, generated by the NvDsInferContext instance.
[in]logFuncA log callback for the instance.
Returns
NVDSINFER_SUCCESS if creation was successful, or an error status otherwise.
NvDsInferStatus NvDsInferContext_DequeueOutputBatch ( NvDsInferContextHandle  handle,
NvDsInferContextBatchOutput *  batchOutput 
)

Dequeues output for a batch of frames.

See also
NvDsInferContext::dequeueOutputBatch() for details.
Parameters
[in]handleA handle to an NvDsInferContext instance.
[in,out]batchOutputA reference to the batch output structure to which output is to be appended.
Returns
NVDSINFER_SUCCESS if dequeueing was successful, or an error status otherwise.
void NvDsInferContext_Destroy ( NvDsInferContextHandle  handle)

Destroys an NvDsInferContext instance and releases its resources.

Parameters
[in]handleThe handle to the NvDsInferContext instance to be destroyed.
void NvDsInferContext_FillLayersInfo ( NvDsInferContextHandle  handle,
NvDsInferLayerInfo *  layersInfo 
)

Fills an input vector with information about all of the bound layers of the inference engine in an NvDsInferContext instance.

The size of the array must be at least the value returned by NvDsInferContext_GetNumLayersInfo().

Parameters
[in]handleA handle to an NvDsInferContext instance.
[in,out]layersInfoA pointer to an array of NvDsInferLayerInfo structures to be filled by the function.
const char* NvDsInferContext_GetLabel ( NvDsInferContextHandle  handle,
unsigned int  id,
unsigned int  value 
)

Gets the string label associated with the class ID for detectors and the attribute ID and attribute value for classifiers.

The string is owned by the context; the caller may not modify or free it.

Parameters
[in]handleA handle to an NvDsInferContext instance.
[in]idClass ID for detectors, or attribute ID for classifiers.
[in]valueAttribute value for classifiers; set to 0 for detectors.
Returns
A pointer to a string label. The memory is owned by the context.
void NvDsInferContext_GetNetworkInfo ( NvDsInferContextHandle  handle,
NvDsInferNetworkInfo *  networkInfo 
)

Gets network input information.

Parameters
[in]handleA handle to an NvDsInferContext instance.
[in,out]networkInfoA pointer to an NvDsInferNetworkInfo structure.
unsigned int NvDsInferContext_GetNumLayersInfo ( NvDsInferContextHandle  handle)

Gets the number of the bound layers of the inference engine in an NvDsInferContext instance.

Parameters
[in]handleA handle to an NvDsInferContext instance.
Returns
The number of bound layers of the inference engine.
NvDsInferContext_GetStatusName is deprecated Use NvDsInferStatus2Str instead const char* NvDsInferContext_GetStatusName ( NvDsInferStatus  status)

Gets the string name of the status.

Parameters
[in]statusAn inference status code.
Returns
A pointer to a string containing the status's name, or NULL if the status is unrecognized. Memory is owned by the function; the caller may not free it.
NvDsInferStatus NvDsInferContext_QueueInputBatch ( NvDsInferContextHandle  handle,
NvDsInferContextBatchInput *  batchInput 
)

Queues a batch of input frames for preprocessing and inferencing.

See also
NvDsInferContext::queueInputBatch() for details.
Parameters
[in]handleA handle to an NvDsInferContext instance.
[in]batchInputA reference to a batch input structure.
Returns
NVDSINFER_SUCCESS if preprocessing and queueing were successful, or an error status otherwise.
void NvDsInferContext_ReleaseBatchOutput ( NvDsInferContextHandle  handle,
NvDsInferContextBatchOutput *  batchOutput 
)

Frees the memory associated with the batch output and releases the set of host buffers back to the context for reuse.

Parameters
[in]handleA handle to an NvDsInferContext instance.
[in]batchOutputA pointer to an NvDsInferContextBatchOutput structure filled by NvDsInferContext_DequeueOutputBatch().
void NvDsInferContext_ResetInitParams ( NvDsInferContextInitParams *  initParams)

Resets a context parameter structure to default values.

Parameters
[in]initParamsA pointer to a context parameter structure.