Is this page helpful?

Video Frame Generation#

Video Frame Generation (VFG) is an AI-powered temporal interpolation filter that synthesizes intermediate frames between two consecutive source frames. The previous frame represents time t=0 and the current frame represents time t=1. Each call to NvVFX_Run() produces one generated frame at a requested temporal position strictly between the two inputs.

VFG does not perform spatial scaling. The previous frame, current frame, and generated frame must have the same dimensions and use the same encoding family.

Operating Modes#

VFG provides two mutually exclusive ways to select the temporal position of the generated frame:

Mode

Configuration

Generated Position

Multiplier

Set FrameMultiplier to M in the range 2 through 8, and set FrameIndex to i in the range 1 through M-1 before each run.

t = i / M

Explicit timestep

Set FrameMultiplier to 0, and set Timestep to a value in the open interval (0.0, 1.0) before each run.

t = Timestep

For example, multiplier mode with M=4 requires three calls to NvVFX_Run(), using frame indices 1, 2, and 3. The calls generate frames at t=0.25, t=0.5, and t=0.75. VFG writes each result to the same output buffer, so the application must consume or copy the result before the next run.

Setting FrameMultiplier, including setting it to 0, resets FrameIndex and Timestep. Set a new per-output selector before the next run.

Model Modes#

VFG provides the following model modes:

Value

Mode

Description

0

Low

Selects the lowest-complexity model.

1

Medium

Selects the balanced model. This is the default.

2

High

Selects the highest-complexity model.

Set the model mode before calling NvVFX_Load(). Changing it after loading the effect requires another call to NvVFX_Load().

Parameters#

VFG provides the following feature-specific parameters:

Parameter

Type

Default

Description

NVVFXVIDEOFRAMEGENERATION_MODE

U32

1

Selects the Low (0), Medium (1), or High (2) model.

NVVFXVIDEOFRAMEGENERATION_FRAME_MULTIPLIER

U32

0

Selects explicit timestep mode (0) or multiplier mode (2-8).

NVVFXVIDEOFRAMEGENERATION_FRAME_INDEX

U32

Unset

Selects an intermediate frame in the range 1 through M-1 when multiplier mode is active.

NVVFXVIDEOFRAMEGENERATION_TIMESTEP

F32

Unset

Selects a temporal position in (0.0, 1.0) when explicit timestep mode is active.

NVVFXVIDEOFRAMEGENERATION_SHOT_CHANGE

U32

0

Set to 1 when the two inputs cross a shot change or other discontinuity. VFG bypasses interpolation and copies the current frame to the output. The value persists until the application sets it to 0.

NVVFXVIDEOFRAMEGENERATION_AUTOMATIC_SHOT_CHANGE_DETECTION_ENABLED

U32

1

Enables (1) or disables (0) automatic shot-change detection. A manual shot-change value of 1 takes priority.

Set NVVFX_CUDA_STREAM, NVVFX_INPUT_WIDTH, NVVFX_INPUT_HEIGHT, and the model mode before calling NvVFX_Load(). Only a batch size of 1 is supported.

Input and Output Requirements#

Bind the previous frame with NVVFX_INPUT_IMAGE_0, the current frame with NVVFX_INPUT_IMAGE_1, and the generated frame with NVVFX_OUTPUT_IMAGE.

VFG supports the following image encodings:

  • RGBA or BGRA interleaved images with 8-bit unsigned components.

  • RGB10A2 interleaved images packed in a 32-bit component.

All three images must use either the 8-bit encoding family or RGB10A2; mixing the two families is not supported. The images can reside in CUDA device memory (NVCV_CUDA or NVCV_GPU) or mapped pinned-host memory (NVCV_CPU_PINNED).

Hardware Requirements#

VFG supports these platforms and GPUs:

  • Windows and Linux x86-64: NVIDIA Ada and Blackwell architecture GPUs.

  • Linux x86-64 additionally supports NVIDIA Hopper architecture GPUs.

  • Windows ARM64: NVIDIA RTX Spark.