Video Relighting#

The video relighting filter reilluminates a person in a video to match the target lighting condition. It accepts an input, matte, and HDRI image, and produces a relit output image of the foreground and an optional reprojected HDRI image for use as a background. The background pixels of the relit output are not valid and must be composited over a chosen background. The input, matte, and outputs should be the same size. The HDRI should have a 2:1 aspect ratio to avoid distortion.

  • The input image is chunky BGR or RGB U8 in CUDA memory, from a video stream. The filter adjusts the lighting of the foreground portion of this image.

  • The matte image is A or Y U8 in CUDA memory, produced by a grayscale segmenter like the AI Green Screen filter, and selects the foreground. Both the video relighting filter and the AI Green Screen filter receive the same input image, with the video relighting filter additionally using the AI Green Screen Filter’s output.

  • A High Dynamic Range Image (HDRI), specified as RGB or BGR F32, provides a lighting environment. This is an equirectangular 360°x180° projection of the world onto a sphere. Pixels of light emitters will be significantly brighter than those of light reflectors. The HDRI can be stored on the CPU or GPU, as it is immediately processed into a form suitable for lighting computations. The HDRI should have a 2:1 aspect ratio; if not, an NVCV_ERR_WRONGSIZE error is generated, and the image will be stretched to 2:1 to cover the entire sphere, usable for video relighting.

  • If the HDRI is not 2:1 and the optional reprojected output is chosen, distortion will occur due to stretching.

  • The primary output image is an RGB or BGR U8 image, typically part of a video stream, showing the relit foreground with invalid background pixels.

  • There is an optional second image available, if storage is provided for it. This is the HDR image, reprojected to a plane based on the pan angle and vertical field of view (VFOV) parameters, then tone-mapped from high dynamic range to low.

  • The relit foreground image needs to be subsequently composited over a background of your choice, such as:

    • The original input image, effectively relighting the foreground with a nearby light that does not affect the lighting of the faraway background.

    • The reprojected HDRI, effectively placing the foreground into that environment with compatible lighting.

    • An unrelated image.

    • Blurred versions of any of the above.

There are also several parameters that can be adjusted to achieve the desired look:

  • The pan angle rotates the environment to change the lighting. It is also used to render the optional reprojected HDRI. The default is 0 radians.

  • The VFOV adjusts how near or far the background appears to be when outputting the projected HDR for use as a backdrop. The default is π/3 radians (60°).

  • The quality and nature of the lighting are directly controlled by the chosen HDRI. Some sample HDRIs are provided with the Sample App, including a studio light of several color temperatures.