AI Green Screen#

The AI green screen filter segments a video or still image into foreground and background regions. The following modes of operation are supported:

  • Quality mode with chairs segmented as the foreground (Mode0), which gives the highest quality result.

    • Images must be at least 288 pixels high and at least 512 pixels wide.

    • This mode is the default.

  • Performance mode with chairs segmented as the foreground (Mode1), which gives the fastest performance.

    • Some degradation in quality may be observed.

    • Images must be at least 288 pixels high and at least 512 pixels wide.

  • Quality mode with chairs segmented as the background (Mode2), which gives the highest quality result.

    • Images must be at least 288 pixels high and at least 512 pixels wide.

    • This mode is the default.

  • Performance mode with chairs segmented as the background (Mode3), which gives the fastest performance.

    • Some degradation in quality may be observed.

    • Images must be at least 288 pixels high and at least 512 pixels wide.

    For best results, the aspect ratio of the image file must be 16:9. The filter works on images with other aspect ratios.

    The filter’s input/output is as follows:

  • The input should be provided in a GPU buffer in BGR interleaved, where each pixel is a 24-bit unsigned char value, and, therefore, 8-bit per pixel component.

  • The output of the filter is written to an 8-bit (grayscale) GPU buffer.

Refer to Sample Applications Reference for more information about a sample application that you can use to experience this effect.

The AI green screen filter processes an input image by performing the following sequence of operations:

  1. The filter classifies the pixels in the video based on the filter’s confidence:

  • Pixels that are part of a human are classified as foreground pixels.

  • All other pixels are classified as background pixels.

  1. After classifying the pixels, the filter generates an 8-bit alpha mask with the same resolution as the input image.

    The alpha values are determined from the filter’s confidence that the pixel belongs to the class to which it was assigned.

    • Pixels with a high foreground confidence are assigned alpha values close to 255.

    • Pixels with a high background confidence are assigned alpha values close to zero.

    Downstream processes can combine the alpha mask generated by the AI green screen filter with the original image to generate effects. For example, a 24-bit BGR input image can be combined with the mask to create a 32-bit image with an alpha channel.

    The alpha values can be determined in one of the following ways:

    • Thresholds can be applied during the process to force the alpha channel of background pixels to be 0 and foreground pixels to be 255.

    • The original alpha values can be maintained to create semi-transparency based on confidence.

    The 32-bit image can then be composited onto a tertiary image to generate an image in which the background pixels of the original image are replaced with the background pixels of the tertiary image.

Note

The AI green screen effect achieves its best results on videos that are recorded by one person sitting in front of a camera. The feature will not perform well on full-body videos, multiple persons in the scene, or camera angles that deviate too much from a front-facing camera.

Note

To keep temporal consistency, the AI green screen effect uses a state variable to track the input video stream. Refer to Creating and Setting State Variables For AI Green Screen for more information about how to track the input stream.