Create and Set State Variables#

For Webcam Denoising#

Webcam denoising uses a state variable to track the input video stream to remove temporal noise.

The SDK user is responsible for completing the following tasks:

  • Create the state variable.

    1. Query the size of the state variable by calling NvVFX_GetU32() with the NVVFX_STATE_SIZE selector string:

      unsigned int stateSizeInBytes;

vfxErr = NvVFX_GetU32(effectHandle, NVVFX_STATE_SIZE, &stateSizeInBytes);

    2. Allocate the necessary space for the state variable in the GPU by using cudaMalloc():

      void\* state[1];

cudaMalloc(&state[0], stateSizeInBytes);

    3. Initialize the state variable to 0 by using cudaMemset():

      cudaMemset(state[0], 0, stateSizeInByte);

  • Pass the state variable to the SDK.

    To pass the state variable to the SDK, use NvVFX_SetObject() with the NVVFX_STATE selector string:

    vfxErr = NvVFX_SetObject(effectHandle, NVVFX_STATE, (void*)state);

  • Release the state variable memory.

    After the state variable has been initialized and set, the filter can be run on an image or a video. After the state variable has completed the processing of the original input, you can reuse the variable with another image or video.

    However, before you can use it on a new input, reset the state variable to 0 by using cudaMemset(). When the state variable is no longer in use, release the memory that was allocated for the state variable by using cudaFree().

For AI Green Screen#

The AI green screen uses a state variable to track the input video stream to keep the temporal consistency, and the SDK provides NvVFX_StateObjectHandle to represent a handle to the state variable.

The SDK user is responsible for completing the following tasks:

  • Create the state variable.

    Allocate a state variable by calling the NvVFX_AllocateState function, which returns NvCV_Status.

    The function completes the following steps:

    1. Allocates the state variable.

    2. Resets the state variable.

    3. Assigns the handle in the second parameter, which the SDK user can store for additional processing.

  • Pass the state variable to the SDK.

    To pass the state variable to the SDK, use the NvVFX_SetStateObjectHandleArray function with the NVVFX_STATE selector string.

  • Reuse the state variable for another input video stream.

    After the state variable has been initialized and set, the filter can be run on an image or a video. After the state variable has completed the processing of the original input, you can reuse the variable with another image or video.

    Before you can use it on a new input, reset the state variable by calling the NvVFX_ResetState function, which returns NvCV_Status.

  • When the state variable is no longer in use, release the state variable by calling the NvVFX_DeallocateState function.

Note

Refer to Batching for AI Green Screen for more information about how to track multiple input streams concurrently.

Setting the Input and Output Image Buffers#

Each filter takes a GPU NvCVImage structure as input and produces the result in a GPU NvCVImage structure. For more information about NvCVImage, refer to the NvCVImage API Guide. These images are GPU buffers accepted by the filter. The application provides input and output buffers to the filter by setting them as required parameters.

The video effect filter requires the input to be provided in a GPU buffer. If the original buffer is of type CPU/GPU or is in planar format, it must be converted as explained in Transferring Images Between CPU and GPU Buffers.

Here is a list of the currently used formats:

  • AI green screen: BGRu8 chunky → Au8

  • Background Blur: BGRu8 chunky + Au8 chunky → BGRu8 chunky

  • Upscale: RGBAu8 chunky → RGBAu8 chunky

  • ArtifactReduction: BGRf32 planar normalized → BGRf32 planar normalized

  • SuperRes: BGRf32 planar normalized → BGRf32 planar normalized

  • Transfer: anything → anything

  • Denoise: BGRf32 planar normalized → BGRf32 planar normalized

Note

BGRu8 chunky refers to a 24-bit pixel, with each B,G, and R pixel component being 8-bit. Similarly, RGBAu8 chunky refers to a 32-bit pixel, with each B,G, R and A pixel component being 8 bits. In contrast, BGRf32 planar refers to the floating-point precision for each pixel component; for example, each of the B, G and R pixel components occupy 32 bits. However, because these are planar images, these are not compact 96-bit pixels, and there are three 32-bit planes where each component of a particular pixel might be separated by megabytes.

For each image buffer, call the NvVFX_SetImage() function, and specify the following information as parameters:

  • The filter handle that was created as explained in Creating a Video Effect Filter.

  • The selector string that denotes the type of buffer that you are creating:

  • For the input image buffer, use NVVFX_INPUT_IMAGE.

  • For the output (mask) image buffer, use NVVFX_OUTPUT_IMAGE.

  • The address of the NvCVImage object created for the input or output image.

  • For Background blur, use NVVFX_INPUT_IMAGE_1 for passing the second input, which is the segmentation mask.

This example creates an input image buffer:

NvCVImage srcGpuImage;

...

vfxErr = NvCVImage_Alloc(&srcGpuImage, 960, 540, NVCV_BGR, NVCV_U8,
NVCV_CHUNKY, NVCV_GPU, 1)

...

vfxErr = NvVFX_SetImage(effectHandle, NVVFX_INPUT_IMAGE, &srcGpuImage);

This example creates an output image buffer:

NvCVImage srcGpuImage;

...

vfxErr = NvCVImage_Alloc(&dstGpuImage, 960, 540, NVCV_A, NVCV_U8,
NVCV_CHUNKY, NVCV_GPU, 1))

...

vfxErr = NvVFX_SetImage(effectHandle, NVVFX_OUTPUT_IMAGE, &dstGpuImage);

Setting and Getting Other Parameters of a Video Effect Filter#

Before loading and running a video effect filter, set any other parameters that the filter requires. NVIDIA Video Effects SDK provides type-safe set accessor functions for this purpose. If you need the value of a parameter that has a set accessor function, use the corresponding get accessor function.

In the call to each set and get accessor function, provide the following information as parameters:

  • The filter handle that was created as explained in Creating a Video Effect Filter.

  • The selector string for the parameter that you want to access.

  • The value that you want to set or a pointer to a location in which to store the value that you want to get.

Example: Setting the Filter Mode for AI Green Screen#

The AI green screen filter supports the following modes of operation:

  • Quality mode with chairs segmented as the foreground (Mode0), which provides the highest quality result (default).

  • Performance mode with chairs segmented as the foreground (Mode1), which provides the fastest performance.

  • Quality mode with chairs segmented as the background (Mode2), which provides the highest quality result (default)

  • Performance mode with chairs segmented as the background (Mode3), which provides the fastest performance

Call the NvVFX_SetU32() function and specify the following information as parameters:

  • The filter handle that was created (refer to Creating a Video Effect Filter).

  • The selector string NVVFX_MODE.

  • An integer that denotes the mode of operation that you want:

    • 0: Quality mode with chairs in the foreground

    • 1: Performance mode with chairs in the foreground

    • 2: Quality mode with chairs in the background

    • 3: Performance mode with chairs in the background

This example sets the mode to Performance with chairs in the foreground:

vfxErr = NvVFX_SetU32 (effectHandle, NVVFX_MODE, 1);

Example: Enable CUDA Graph Optimization for AI Green Screen#

The AI green screen filter supports CUDA graph optimization. It can be enabled by setting the correct value for NVVFX_CUDA_GRAPH.

Call the NvVFX_SetU32() function, specifying the following information as parameters:

  • The filter handle that was created as explained in Creating a Video Effect Filter.

  • The selector string NVVFX_CUDA_GRAPH.

  • An integer that denotes the CUDA graph optimization state:

    • 0: OFF

    • 1: ON

  • The default state of CUDA Graph optimization is OFF for AI green screen filter:

    vfxErr = NvVFX_SetU32 (effectHandle, NVVFX_CUDA_GRAPH, 1);

    The set method must be called before the NvVFX_Load() method that initializes the AIGS effect.

  • The first call to NvVFX_Run() initializes CUDA Graph if it is enabled. If there is an error during graph initialization, the call returns NVCV_ERR_CUDA.

  • If the set method is called after the initialization, the NvVFX_Run() call returns the NVCV_ERR_INITIALIZATION error code.

Enabling the CUDA Graph optimization reduces the kernel launch overhead and might improve overall performance. To verify the improvement, we recommend that developers test their application with CUDA Graph turned off and on.

Example: Optimize Memory Allocations for AI Green Screen#

The AI green screen filter works on images of any width and height, but an increase in the resolution will cause the filter to allocate the necessary GPU memory.

The AI green screen filter supports the NVVFX_MAX_INPUT_WIDTH and NVVFX_MAX_INPUT_HEIGHT parameters, which can optimize memory allocations during NvVFX_Run().

Call the NvVFX_SetU32() function and specify the following information as parameters:

  • The filter handle that was created as explained in Creating a Video Effect Filter.

  • The selector string NVVFX_MAX_INPUT_WIDTH and NVVFX_MAX_INPUT_HEIGHT.

  • Integer values for the width and height; for example, 1920 and 1080.

  • The set method must be called before the NvVFX_Load() method, which initializes the AIGS effect.

No additional memory will be allocated by the filter up to the values set by using the parameter selector strings above.

Summary of Video Effects SDK Accessor Functions#

Table 2-1. Summary of NVIDIA Video Effects SDK Accessor Functions

Parameter Type

Data Type

Set and Get Accessor Function

32-bit unsigned integer

unsigned int

NvVFX_SetU32() NvVFX_GetU32()

32-bit signed integer

int

NvVFX_SetS32() NvVFX_GetS32()

Single-precision (32-bit) floating-point number

float

NvVFX_SetF32() NvVFX_GetF32()

Double-precision (64-bit) floating point number

double

NvVFX_SetF64() NvVFX_GetF64()

64-bit unsigned integer

unsigned long long

NvVFX_SetU64() NvVFX_GetU64()

Image buffer

NvCVImage

NvVFX_SetImage() NvVFX_GetImage()

Object

void

NvVFX_SetObject() NvVFX_GetObject()

Character string

const char*

NvVFX_SetString() NvVFX_GetString()

CUDA stream

CUstream

NvVFX_SetCudaStream() NvVFX_GetCudaStream()

Getting Information About a Filter and its Parameters#

To get information about a filter and its parameters, call the NvVFX_GetString() function, specifying the NVVFX_INFO type of the NvVFX_ParameterSelector type definition.

Getting a List of All Available Effects#

To get a list of the available effects, call the NvVFX_GetString() function, specifying NULL for the NvVFX_Handle object handle.