Batching When Using State Variables#

For Webcam Denoising#

Webcam denoising filters use state variables to track the input video streams to remove temporal noise. A batched input (srcBatch), however, can contain frames from multiple videos in arbitrary number and order.

When you use webcam denoising, you need to complete some additional steps before NvVFX_Run() to provide information about the ordering and the video source of the images in the batch. This process allows each input video stream to be properly tracked.

One state variable tracks one video stream, so you need to:

  1. Create one state variable per video stream in your application.

    If you have n input video streams, you need to create n state variables.

  2. Create an array, batchOfStates, to hold the memory addresses of a batch of state variables. The size of this array will be equal to the batch size.

Note

The size of the batch need not be equal to the number of input video streams, and the batch can contain frames from different video streams in arbitrary order. A batch can also contain multiple frames from the same video stream, but the frames must be arranged in the batch in chronological order.

This array holds the addresses of state variables in the order that corresponds to the video source of the images in the batched input, srcBatch. If the nth image in the batch arises from the mth video stream, the nth element in this array holds the address of the mth state variable.

For example, assume that the batch size is six and that the batched input, srcBatch, contains the frames from input stream #N-1, input stream #q, input stream #q, input stream #1, input stream #p, and input stream #0 in this order. You need to copy the address of the corresponding state variable in batchOfStates and pass this array to the SDK using NvVFX_SetObject():

If the batch size or the order of the source of images in the batch changes, the size and the contents of batchOfStates can be changed and set using NvVFX_SetObject() before every NvVFX_Run().

Example Code#

The example code is in the following files:

  • BatchEffectApp.cpp, which provides the functional example code and includes a list of image files and the code to process these files as a batch.

  • BatchDenoiseEffectApp.cpp, which provides the functional example code to use batching with Webcam Denoising. The code accepts multiple video files as the input and processes the frames from the input videos in batches of the user-specified size.

For AI Green Screen#

To keep temporal consistency, the AI Green Screen filter uses state variables to track the input video streams.

A batched input, srcBatch, can contain frames from multiple videos in an arbitrary number and order. However, when you create a batched input, the following constraints must be met:

  • A batched input should not have multiple frames from the same video stream.

  • Frames from the same video stream must be passed chronologically in a separate batch.

When you use an AI green screen filter, you must provide information about the order and the video source of the images in the batch. Before calling NvVFX_Run(), you need to complete some additional steps. This process allows each input video stream to be properly tracked.

  • Set the maximum number of concurrent input video streams by calling `NvVFX_SetU32() with the NVVFX_MAX_NUMBER_STREAMS selector string.

    This set method must be called before the NvVFX_Load() method, which initializes the effect.

One state variable tracks one video stream, so you need to complete the following steps:

  1. In your application, create one state variable per video stream.

    If you have n input video streams, you need to create n state variables:

    NvVFX_StateObjectHandle stateObjectHandle;

vfxErr = NvVFX_AllocateState(effectHandle, &stateObjectHandle);

  2. The SDK user can query the number of active state variables by calling the NvVFX_GetU32 function with the NVVFX_STATE_COUNT parameter selector.

    The number of active state variables will be assigned to the provided parameter:

    vfxErr = NvVFX_GetU32(effectHandle, NVVFX_STATE_COUNT, &numActiveStateVariables);

  3. Create an array, batchOfStates, to hold the memory addresses of a batch of state variables. The size of this array will be equal to the batch size.

Note

The size of the batch should be less than or equal to the number of input video streams, because only one frame per stream can be in each batch. Due to a different number of video streams in each batch, the number of frames in a batch can differ between batches. A batch can have a frame from different video streams in an arbitrary order.

This array holds the addresses of state variables in the order that corresponds to the video source of the images in srcBatch. If the nth image in the batch arises from the mth video stream, the nth element in this array will hold the address of the mth state variable.

For example, assume that the batch size is five and srcBatch contains the frames from input stream #N-1, input stream #q, input stream #1, input stream #p, and input stream #0 in this order. You need to copy the address of the corresponding state variable in batchOfStates and pass this array to the SDK by using NvVFX_SetObject:

NvVFX_StateObjectHandle batchOfStates[] = {arrayOfStates[N-1],
arrayOfStates[q], arrayOfStates[1], arrayO fStates[p], arrayOfStates[0]};

vfxErr = NvVFX_SetStateObjectHandleArray(effectHandle, NVVFX_STATE,
batchOfStates);

If the batch size or the order of the source of images in the batch changes, the size and the contents of batchOfStates can be changed and set by using the NvVFX_SetStateObjectHandleArray function before every NvVFX_Run().

Example Code#

The example code is in the BatchAigsEffectApp.cpp file, and this file provides the functional sample code that allows you to use batching with AI Green Screen. The code accepts multiple video files as the input and processes the frames from the input videos in batches that are equal to the number of video files.