AR SDK#

Creating a Triton Object and Connecting to an ARSDK Enabled Triton Server#

NvAR_TritonServer is a predefined type in the SDK, which represents a connection to the server. The first step is to create an instance of this type and call the NvAR_ConnectTritonServer to connect to a server. The NvAR_ConnectTritonServer function accepts a string indicating the URL (IP and the port) of the gRPC service on the server. In the same application, it is possible to connect to multiple servers by creating multiple instances.

NvCV_Status nvcv_err = NVCV_SUCCESS;

NvAR_TritonServer server_handle;

char server_url[] = "127.0.0.1:8001";

nvcv_err = NvAR_ConnectTritonServer(server_url, &server_handle);

Creating a Feature Instance#

NvAR_FeatureHandle is a predefined type in the SDK, which represents a feature. To instantiate a handle for a feature to run on a server, NvAR_CreateTriton API is called. This is different from NvAR_Create API which creates a feature to be run locally. Only feature instances created using NvAR_CreateTriton can run features on the server and use the batching APIs. If a feature handle is instantiated using NvAR_Create, its usage is limited to exactly the same as the non-Triton enabled SDKs.

The NvAR_CreateTriton API takes feature ID as input which indicates what feature to instantiate. It could be NvAR_Feature_FaceDetection, NvAR_Feature_LandmarkDetection, NvAR_Feature_GazeRedirection, or NvAR_Feature_LivePortrait. The next step is to assign the feature instance to a server instance using NvAR_SetTritonServer. Several feature instances may be assigned to a single server instance, however one feature instance may be associated with only one server instance.

NvAR_FeatureHandle feature_handle;

nvcv_err = NvAR_CreateTriton(NvAR_Feature_FaceDetection, &feature_handle);

nvcv_err = NvAR_SetTritonServer(feature_handle, server_handle);

Getting and Setting Feature Properties#

Once the feature instance is instantiated and assigned to a server, feature configurations may be set using the setters and verified using the getters. For example, the following code sets the temporal flag. The features support the same configurations as the non-Triton SDKs.

nvcv_err = NvAR_SetU32(feature_handle, NvAR_Parameter_Config(Temporal), 0xFFFFFFFF);

Loading the Feature#

NvAR_Load is called to initialize the feature after the configurations have been set.

nvcv_err = NvAR_Load(feature_handle);

Creating State Objects#

The SDK uses state objects to identify and track the input video stream in a batched input. One state object is associated with one input video stream. Therefore, if the application intends to process N video streams, N state objects must be created. The SDK specifies NvAR_StateHandle type for the state objects which are instantiated using NvAR_AllocateState as shown below. Note that even if the application is processing one video stream, it is necessary to create one state object to represent that video stream.

int num_input_video_streams = 10;

std::vector<NvAR_StateHandle> state_handles(num_input_video_streams);

for (int i=0; i<num_input_video_streams; i++) {

   nvcv_err = NvAR_AllocateState(feature_handle, &state_handles[i]);

}

It is not necessary to create all the state objects at once, new state objects may be created during runtime as needed.

Setting Input and Output#

The process of setting input and output buffers for the features is the same as the non-Triton enabled SDKs. The only difference is that when the application is trying to send a batch of input (more than one video stream concurrently) to the server, it is necessary to provide batched buffers.

Batched NvCVImage Object Buffers#

A batched NvCVImage points to a contiguous buffer that contains multiple images with the same structure. The Triton enabled SDK library will transfer NvCVImage to and back from the server using gRPC if the NvCVImage buffer is allocated on the CPU and using CUDA shared memory if the NvCVImage buffer is allocated on the GPU.

There are tighter restrictions on the type of images supplied to the Triton Client, relative to the native SDK:

  • The pixel format is restricted to BGR, BGRA, and A; RGB and RGBA are not accommodated.

  • The images must have a zero-gap pitch. This can be achieved by specifying 1 as the alignment, when calling the NvCVImage constructor or NvCVImage_Alloc, NvCVImage_Realloc.

Batch Utilities#

The following utility functions in BatchUtilities.cpp help you to work with image batches:

  • AllocateBatchBuffer() can be used to allocate a buffer for a batch of images.

  • NthImage() can be used to set a view into the nth image in a batched buffer.

  • ComputeImageBytes()can be used to determine the number of bytes for each image to advance the pixel pointer from one image to the next.

  • TransferToNthImage()makes it easy to call NvCVImage_Transfer() to set one of the images in a batch.

  • TransferFromNthImage()makes it easy to call NvCVImage_Transfer() to copy one of the images in a batch to a regular image.

  • TransferToBatchImage()transfers multiple images from different locations to a batched image.

  • TransferFromBatchImage()can be used to retrieve the images in a batch to different images in different locations.

  • TransferBatchImage() transfers all images in a batch to another compatible batch of images.

The last three functions can also be accomplished by repeatedly calling the Nth image APIs, but the source code illustrates an alternative method of accessing images in a batch.

Allocation of Batched Buffers#

To allocate batched buffers, call the AllocateBatchBuffer() function, which will allocate an image that is N times taller than the prototypical image.

Note

The allocation cannot always be interpreted this way, especially if the pixels are planar.

The purpose of this function is mainly to provide storage and then dispose of this storage when the NvCVImage goes out of scope or its destructor is called.

You can use your own method to allocate the storage for the batched images. The image that the AllocateBatchBuffer() yields is only used for bookkeeping and is never used in any of the SDK APIs.

Note

The SDK APIs only require an NvCVImage descriptor for the first image.

Setting the Batched Images#

The API only takes the image descriptors for the first image in a batch. The following sample allocates the src and dst batch buffers and sets the input and outputs via the views of the first image in each batch buffer.

NvCVImage srcBatch, dstBatch, nthSrc, nthDst;
AllocateBatchBuffer(&srcBatch, batchSize, srcWidth, srcHeight,
                    ...);
AllocateBatchBuffer(&dstBatch, batchSize, dstWidth, dstHeight,
                    ...);
NthImage(0, srcHeight, &srcBatch, &nthSrc);
NthImage(0, dstHeight, &dstBatch, &nthDst);
NvAR_SetObject(effect, NvAR_Parameter_Output(Image), &nthSrc, sizeof(NvCVImage));
NvAR_SetObject(effect, NvAR_Parameter_Output(Image), &nthDst, sizeof(NvCVImage));

Because the image descriptors are copied into the SDK, this can be simplified to the following:

NvCVImage srcBatch, dstBatch, nth;
AllocateBatchBuffer(&srcBatch, batchSize, srcWidth, srcHeight,
                    ...);
AllocateBatchBuffer(&dstBatch, batchSize, dstWidth, dstHeight,
                    ...);
NvAR_SetObject(effect, NvAR_Parameter_Input(Image),
               NthImage(0, srcHeight, &srcBatch, &nth), sizeof(NvCVImage));
NvAR_SetObject(effect, NvAR_Parameter_Output(Image),
               NthImage(0, dstHeight, &dstBatch, &nth), sizeof(NvCVImage));

The other images in the batch are computed by advancing the pixel pointer by the size of each image.

The other aspect of setting the batched images is determining how to set the pixel values. Each image in the batch is accessible by calling the NthImage() function:

NthImage(n, imageHeight, &batchImage, &nthImage);

You can then use the same techniques that were used for other NvCVImages on the recently initialized nthImage view. As previously suggested, NthImage() is just a thin wrapper around NvCVImage_InitView() and can be used instead. The NvCVImage_Transfer() functions can be used to copy pixels to and from the batch.

Batched bounding boxes#

Batched bounding boxes is an array of NvAR_BBoxes structure, where each item’s boxes field points to an array of NvAR_Rect. It can be set as follows:

unsigned batch_size = 3;

std::vector<NvAR_BBoxes> output_bboxes(batch_size);

std::vector<std::vector<NvAR_Rect>> output_bboxes_data(batch_size);

for (unsigned i = 0; i < batch_size; i++) {

   output_bboxes_data[i].resize(25);

   output_bboxes_data[i].assign(25, { 0.f, 0.f, 0.f, 0.f });

   output_bboxes[i].boxes = output_bboxes_data[i].data();

   output_bboxes[i].max_boxes = kMaxBoxes;

   output_bboxes[i].num_boxes = 1;

}

Batched bounding boxes type is set in the SDK by providing the starting address of the buffer:

NvAR_SetObject(feature_handle, NvAR_Parameter_Output(BoundingBoxes),
output_bboxes.data(), sizeof(NvAR_BBoxes)))

Batched float, NvAR_Point2f, NvAR_Point3f, and NvAR_Quaternion#

Batched float, NvAR_Point2f, NvAR_Point2f, and NvAR_Quaternion type buffers are contiguous arrays big enough to hold all of the input or the output in the batch.

They can be allocated as follows:

unsigned batch_size = 3;

std::vector<float> confidence(CONFIDENCE_SIZE*batch_size);
//one item inthe batch is CONFIDENCE_SIZE big

std::vector<NvAR_Point2f> keypoints_2d(KEYPOINTS2D_SIZE*batch_size);
//one item in the batch is KEYPOINTS2D_SIZE big

std::vector<NvAR_Point3f> keypoints_3d(KEYPOINTS3D_SIZE*batch_size);
//one item in the batch is KEYPOINTS3D_SIZE big

std::vector<NvAR_Quaternion> pose(batch_size);
// One item in the batch is of size 1

They are set in the SDK by providing the starting address of the buffer.

Example:

nvcv_err = NvAR_SetObject(feature_handle, NvAR_Parameter_Output(Pose),
pose.data(), sizeof(NvAR_Quaternion));

Setting a Batch of State Objects#

The SDK uses batch size parameter, NvAR_Parameter_Config(BatchSize), to determine the number of items in the input/output buffers and the batch of states parameter, NvAR_Parameter_InOut(State), to determine which item in the batched input and output buffer belongs to which input video stream. It is necessary to set these parameters before the NvAR_Run call.

A batch of states array is an array of NvAR_StateHandle pointers. In the following example, a batch size of 3 is used and a batch of states is created with pointers to the p-th, the q-th and the r-th items in the state_handles array. This signifies that all input/output buffers contain data in the same order. Since each state corresponds to an input video stream, it signifies that the 0-th, the 1-st, and the 2-nd items in the buffers belong to the p-th, the q-th and the r-th input video streams.

A batch cannot contain more than one item from a single input video stream, so a batch of states should not have duplicate state objects.

Even when processing only one video, it is necessary to create a batch of states array with one item and set the batch size to 1.

unsigned batch_size = 3;

NvAR_StateHandle\* batch_of_states[] = {&state_handles[p],
&state_handles[q], &state_handles[r]};

nvcv_err = NvAR_SetU32(feature_handle, NvAR_Parameter_Config(BatchSize),
batch_size);

nvcv_err = NvAR_SetObject(feature_handle, NvAR_Parameter_InOut(State),
batch_of_states, batch_size);

The NvAR_Parameter_Config(BatchSize) and the NvAR_Parameter_InOut(State) can be changed before every NvAR_Run call.

The size of the batch need not be equal to the total number of input video streams.

Running the Feature#

NvAR_Run runs the feature on the server. The call is asynchronous and non-blocking.

nvcv_err = NvAR_Run(feature_handle);

Waiting for the Result#

NvAR_SynchronizeTriton will block the execution and wait until the results are available after the NvAR_Run call. When this function exits, the results will be populated in the output buffers.

nvcv_err = NvAR_SynchronizeTriton(feature_handle);

Destroying the State Objects#

NvAR_DeallocateState destroys the state objects.

for (int i=0; i<num_input_video_streams; i++) {

   nvcv_err = NvAR_DeallocateState(feature_handle, state_handles[i]);

}

Destroying the Feature Instance#

NvAR_Destroy destroys the feature instance.

nvcv_err = NvAR_Destroy(feature_handle);

Destroying the Triton Object#

NvAR_DisconnectTritonServer disconnects from the server and destroys the server object.

nvcv_err = NvAR_DisconnectTritonServer(server_handle);