Is this page helpful?

NVIDIA Augmented Reality (AR) SDK User Guide#

Download this guide as a PDF

The NVIDIA Augmented Reality SDK (AR SDK) is a comprehensive collection of AI-powered features for real-time modeling and tracking of human faces and bodies from video.

The AR SDK enables developers to build state-of-the-art video processing applications with AI-powered features, such as body pose estimation, face detection, landmark tracking, gaze redirection, and facial expressions. The SDK is powered by NVIDIA graphics processing units (GPUs) with Tensor Cores, supporting high throughput and low latency processing.

The AR SDK has the following features:

  • Face detection and tracking detects, localizes, and tracks human faces in images or videos by using bounding boxes.

  • Facial landmark detection and tracking predicts and tracks the pixel locations of human facial landmark points and head poses in images or videos. It can predict 68 and 126 landmark points. The 68 detected facial landmarks follow the Multi-PIE 68 point mark-up information in Facial point annotations. The 126 facial landmark points detector can predict more points on the cheeks, the eyes, and on laugh lines.

  • Legacy 3D Body Pose Tracking predicts and tracks 34 body keypoints in 2D and 3D from images or videos. It outputs joint rotations and supports multiple people in full-body and upper-body images or videos. For new applications, use 3D Body Pose Estimation. Refer to Appendix B: 3D Body Pose Keypoint Format for keypoint details.

  • 3D Body Pose Estimation estimates body pose from video frames by using caller-provided tracking boxes. It predicts 77 body keypoints in 2D and in 3D camera coordinates, and it can output rest pose, root pose, and joint rotations. Refer to Appendix B: 3D Body Pose Keypoint Format for the keypoint format.

  • Eye Contact estimates the gaze angles of a person in an image or video and redirects the gaze to make it frontal. It can operate in two modes. One mode where head pose and gaze angles are estimated in camera coordinates without any redirection and another where, in addition to estimation, the eyes of the person are redirected to make eye contact with the camera within a permissible range.

  • Facial Expression Estimation estimates face expression coefficients from the provided facial landmarks.

  • LipSync animates a person’s video using an audio input by animating the lip motion to match that of the audio.

  • Active Speaker Detection identifies which person in a video is currently speaking by analyzing both video frames and synchronized audio tracks. It supports multiple audio tracks and outputs face bounding boxes with tracking IDs and speaking status.

Note

The Windows SDK supports x64 systems and Windows on Arm (ARM64) systems with a supported iGPU such as RTX Spark. On Windows on Arm, LipSync and Active Speaker Detection are not supported, and only the Body Pose Estimation / Body Detection path is available.