AR SDK System Guide

The AR SDK System Guide provides the requirements for the hardware, the software, and the sample application, how to install the SDK, and how to build and run the sample applications.

1. NVIDIA AR SDK for Windows

NVIDIA® AR SDK enables real-time modeling and tracking of human faces from video.

The SDK is powered by NVIDIA graphics processing units (GPUs) with Tensor Cores, and as a result, the algorithm throughput is greatly accelerated, and latency is reduced.

NVIDIA AR SDK has the following features:
  • Face detection and tracking (Beta) detects, localizes, and tracks human faces in images or videos by using bounding boxes.
  • Facial landmark detection and tracking (Beta) predicts and tracks the pixel locations of human facial landmark points and head poses in images or videos.

    It can predict 68 and 126 landmark points. The 68 detected facial landmarks follow the Multi-PIE 68 point mark-ups information in Facial point annotations. The 126 facial landmark points detector can predict more points on the cheeks, the eyes, and on laugh lines.

  • Face 3D mesh and tracking (Beta) reconstructs and tracks a 3D human face and its head pose from the provided facial landmarks.
  • 3D Body Pose tracking (Beta) predicts and tracks the 3D human pose from images or videos.

    It predicts 34 keypoints of body pose in 2D and 3D. See Appendix B for the keypoints.

  • NVIDIA AR SDK can be used in a wide variety of applications, such as augmented reality, beautification, 3D face animation, modeling, and so on. NVIDIA AR SDK provides sample applications that demonstrate the features listed above in real time by using a webcam or offline videos.

1.1. Getting Started with NVIDIA AR SDK for Windows

This section provides information about how to get started with the AR SDK for Windows.

1.1.1. Hardware and Software Requirements

NVIDIA AR SDK requires specific NVIDIA GPUs, a specific version of the Windows OS, and other associated software on which the SDK depends.

This SDK is designed and optimized for client-side application integration and for local deployment. We do not officially support the testing, experimentation, deployment of this SDK to a datacenter/cloud environment.

1.1.1.1. Hardware Requirements

NVIDIA AR SDK is compatible with GPUs that are based on the NVIDIA Turing™, the NVIDIA Ampere™ architecture, and have Tensor Cores.

1.1.1.2. Software Requirements

NVIDIA AR SDK requires a specific version of the Windows OS and other associated software on which the SDK depends. The NVIDIA CUDA® and TensorRTTM dependencies are bundled with the SDK Installer. See Installing NVIDIA AR SDK and Associated Software.
Table 1. Software Requirements
Software Required Version
Windows OS 64-bit Windows 10 or later
Microsoft Visual Studio 2015 (MSVC14.0) or later
CMake 3.12 or later
NVIDIA Graphics Driver for Windows 465.89 or later

1.2. Installing the AR SDK and the Associated Software

Here is some information about installing the SDK.

The SDK is distributed in the following forms:
  • The development SDK package

    The development package includes everything in the SDK including the API headers, runtime dependencies, and sample apps.

  • The redistributable SDK package.

    The redistributable package is more convenient if your application only wants to integrate SDK API headers and ask end users to download and install the SDK runtime dependencies.

    To develop applications, because the essential contents in these two packages are the same, you can use either package.

    The redistributable SDK package comprises the following parts:
    • An open-source repository that includes the SDK API headers, the sample applications and their dependency libraries, and a proxy file to enable compilation without the SDK DLLs.
    • An installer that installs the following SDK runtime dependencies:
      • The DLLs
      • The models
      • The SDK dependency libraries

    The install location is the C:\Program Files\NVIDIA Corporation\NVIDIA AR SDK\ directory.

    For an application that is built on the SDK, the developer can package the runtime dependencies into the application or require application users to use the SDK installer.

Note: The source code and sample applications are in the development package and are hosted on GitHub at https://github.com/NVIDIA/MAXINE-AR-SDK.

To use the SDK redistributable package, download the source code from GitHub and install the SDK binaries.

Note: If you are using a development package, you can ignore this step.

The sample app source code demonstrates how to integrate API headers and call the SDK APIs. The sample app also includes the nvARProxy.cpp file that is used to link against the SDK DLL without requiring an import library (.lib) file. With this file, you can compile the open-source code independently of the SDK installer. However, the SDK runtime dependencies are still required to load the runtime dependencies, the DLLs, and models.

1.3. Configure NVIDIA AR SDK with a 3D Morphable Face Model

The Face 3D Mesh Tracking feature requires a 3D Morphable Face Model (3DMM), and the AR SDK does not include a 3DMM. Therefore, if you are using the Face 3D Mesh Tracking feature, you must configure the AR SDK with 3DMM.

Note: To configure the AR SDK with 3DMM, you can use the Surrey Face Model or your own model.

The AR SDK provides the ConvertSurreyFaceModel.exe utility to convert 3DMM files to the NVIDIA.nvf format that is required by the SDK.

1.3.1. Using the Surrey Face Model

Here are the steps to use the Surrey face model.

  1. Download the following Surrey Face Model files from the eos project page on GitHub:
    • sfm_shape_3448.bin
    • expression_blendshapes_3448.bin
    • sfm_3448_edge_topology.json
    • Sfm_model_contours.json
    • ibug_to_sfm.txt
  2. Convert the downloaded files to the NVIDIA .nvf format.
    tools/ConvertSurreyFaceModel.exe
    --shape=path/sfm_shape_3448.bin
    --blend_shape=path/expression_blendshapes_3448.bin
    --topology=path/sfm_3448_edge_topology.json
    --contours=path/sfm_model_contours.json
    --ibug=path/ibug_to_sfm.txt
    --out=output-path/face_model0.nvf
    Note: The ConvertSurreyFaceModel.exe file is distributed in the https://github.com/nvidia/MAXINE-AR-SDK GitHub repo.
    Here are the variables:
    path
    The full or relative path to the folder that contains the Surrey Face Model files that you downloaded.
    output-path

    The full or relative path to the folder where the output .nvf format file should be written.

    The sample application provided with the AR SDK requires that the model file be named face_model0.nvf.

    If you use the development SDK package, place the face_model0.nvf file in the bin\models folder. If you use the redistributable SDK package, place the face_model0.nvf file in the %Program Files%\NVIDIA Corporation\NVIDIA AR SDK\models folder that was created by the SDK Installer.

1.3.2.  Using your own 3DMM

Here are the steps to use your own 3DMM.

Write a model natively in the format that is defined in NVIDIA 3DMM File Format.

The sample application that is provided with the AR SDK requires that the model file be named face_model0.nvf.

  • If you use the development SDK package, place the face_model0.nvf file in the bin\models folder.
  • If you use the redistributable SDK package, place the face_model0.nvf file in the %Program Files%\NVIDIA Corporation\NVIDIA AR SDK\models folder that was created by the SDK Installer.

1.4. Building the Sample Applications on Windows

The open source repository includes the source code to build the sample applications, and the nvARProxy.cpp proxy file to enable compilation without explicitly linking against the SDK DLL.
Note: To download the models and runtime dependencies required by the features, you need to run the SDK Installer.
  1. In the root folder of the downloaded source code, start the CMake GUI and specify the source folder and a build folder for the binary files.
  2. For the source folder, ensure that the path ends in OSS.
  3. For the build folder, ensure that the path ends in OSS/build. Use CMake to configure and generate the Visual Studio solution file.
  4. Click Configure.
  5. When prompted to confirm that CMake can create the build folder, click OK.
  6. Select Visual Studio for the generator and x64 for the platform.
  7. To complete configuring the Visual Studio solution file, click Finish.
  8. To generate the Visual Studio Solution file, click Generate.
  9. Verify that the build folder contains the NvAR_SDK.sln file. Use Visual Studio to generate the FaceTrack.exe and BodyTrack.exe files from the NvAR_SDK.sln file.
  10. In CMake, to open Visual Studio, click Open Project.
  11. In Visual Studio, select Build > Build Solution.

1.5. Running the Sample Applications on Windows

The SDK provides the FaceTrack and the BodyTrack sample applications, and the prebuilt FaceTrack.exe and BodyTrack.exe files are provided with the AR SDK.

Before you run the application, connect a camera to the computer on which you plan to run the sample application.

Note: The application uses the video feed from this camera.
  1. Open a Command Prompt window.
  2. To run a sample application, complete one of the following steps:
    • For the FaceTrack sample application, from the samples\FaceTrack folder, under the root folder of the AR SDK, execute the run.bat file.
    • To run the BodyTrack sample application, from the samples\BodyTrack folder, under the root folder of the AR SDK, execute the run.bat file.

    This command launches an OpenCV window with the camera feed. For FaceTrack, it draws a 3D face mesh over the largest detected face. For BodyTrack, it draws a Body Pose skeleton over the detected person.

1.6. Sample Applications for the SDK

FaceTrack is a sample Windows application that demonstrates the face tracking, landmark tracking, and 3D mesh tracking features of the AR SDK. Similarly, BodyTrack is a sample Windows application that demonstrates the 3D Body Pose Tracking feature of the SDK.

1.6.1.  FaceTrack Sample Application

This section provides information about the FaceTrack sample application.

1.6.1.1. Command-Line Arguments for the FaceTrack Sample Application

Here is a list of the command-line arguments for the FaceTrack sample application.

--model_path=path
Specifies the path to the models.
--landmarks_126[=(true|false)]
Specifies whether to set the number of landmark points to 126 or 68.
  • true: set number of landmarks to 126.
  • false: set number of landmarks to 68.
--temporal[=(true|false)]
Optimizes the results for temporal input frames. If the input is a video, set this value to true.
--offline_mode[=(true|false)]
Specifies whether to use offline video or an online camera video as the input.
  • true: Use offline video as the input.
  • false: Use an online camera as the input.
--capture_outputs[=(true|false)]
If --offline_mode=false, specifies whether to enable the following features:
  • Toggling video capture on and off by pressing the C key.
  • Saving an image frame by pressing the S key.

Additionally, a result file that contains the detected landmarks and /or face boxes is written at the time of capture.

If --offline_mode=true, this argument is ignored.

--cam_res=[width x] height
  • If --offline_mode=false, specifies the camera resolution. width is optional.
    If omitted, width is computed from height to give an aspect ratio of 4:3, for example:
    --cam_res=640x480 or --cam_res=480.
  • If --offline_mode=true, this argument is ignored.
--in_file=file
--in=file
  • If --offline_mode=true, specifies the input video file.
  • If --offline_mode=false, this argument is ignored.
--out_file=file
--out=file
  • If --offline_mode=true, specifies the output video file.
  • If --offline_mode=false, this argument is ignored.

1.6.1.2. Keyboard Controls for the FaceTrack Sample Application

The FaceTrack sample application provides keyboard controls to change the runtime behavior of the application.

Here is a list of these controls:
  • 1 selects the face-tracking-only mode and shows only the bounding boxes.
  • 2 selects the face and landmark tracking mode and shows only landmarks.
  • 3 selects face, landmark, and 3D mesh tracking mode and shows only 3D face meshes.
  • W toggles the selected visualization mode on and off.
  • F toggles the frame rate display.
  • C toggles video saving on and off.
    • When video saving is toggled off, a file is saved with the captured video with a result file that contains the detected face box and/or landmarks.
    • This control is enabled only if --offline_mode=false and --capture_outputs=true.
  • S saves an image and a result file.

This control is enabled only if --offline_mode=false and --capture_outputs=true.

1.6.2. BodyTrack Sample Application

This section provides information about the BodyTrack sample application.

1.6.2.1. Command-Line Arguments for the BodyTrack Sample Application

Here is a list of the command-line arguments for the FaceTrack sample application.

--model_path=path
Specifies the path to the models.
--mode[=(0|1)]
Specifies whether to select High Performance mode or High Quality mode.
  • 0: set mode to High Quality.
  • 1: set mode to High Performance.
--app_mode[=(0|1)]
Specifies whether to select Body Detection or Body Pose Detection.
  • 0: set mode to Body Detection.
  • 1: set mode to Body Pose Detection.
--temporal[=(true|false)]
Optimizes the results for temporal input frames. If the input is a video, set this value to true.
--use_cuda_graph[=(true|false)]
Uses CUDA Graphs to improve performance. CUDA graph reduces the overhead of the GPU operation submission of 3D body tracking.
--offline_mode[=(true|false)]
Specifies whether to use offline video or an online camera video as the input.
  • true: Use offline video as the input.
  • false: Use an online camera as the input.
--capture_outputs[=(true|false)]
  • If --offline_mode=false, specifies whether to enable the following features:
    • Toggling video capture on and off by pressing the C key.
    • Saving an image frame by pressing the S key.

    Additionally, a result file that contains the detected landmarks and /or face boxes is written at the time of capture.

  • If --offline_mode=true, this argument is ignored.
--cam_res=[width x] height
  • If --offline_mode=false, specifies the camera resolution.

    width is optional. If omitted, width is computed from height to give an aspect ratio of 4:3.

    For example:

    --cam_res=640x480 or --cam_res=480.
  • If --offline_mode=true, this argument is ignored.
--in_file=file
--in=file
  • If --offline_mode=true, specifies the input video file.
  • If --offline_mode=false, this argument is ignored.
--out_file=file
--out=file
  • If --offline_mode=true, specifies the output video file.
  • If --offline_mode=false, this argument is ignored.

1.6.2.2. Keyboard Controls for the BodyTrack Sample Application

The BodyTrack sample application provides keyboard controls to change the runtime behavior of the application.

Here is a list of these controls:
  • 1 selects the body-tracking-only mode and shows only the bounding boxes.
  • 2 selects the body and body pose tracking mode and shows the bounding boxes and body pose keypoints.
  • W toggles the selected visualization mode on and off.
  • F toggles the frame rate display.
  • C toggles video saving on and off.
    • When video saving is toggled off, a file is saved with the captured video with a result file that contains the detected face box and/or landmarks.
    • This control is enabled only if --offline_mode=false and --capture_outputs=true.
  • S saves an image and a result file.

This control is enabled only if --offline_mode=false and --capture_outputs=true.

1.7. Environment Variables

Here is information about the environmental variables in the AR SDK.

The environmental variables are as follows:
  • NVAR_MODEL_DIR

    If the application has not provided the path to the models directory by setting the NvAR_Parameter_Config_ModelDir string, the SDK tries to load the models from the path in the NVAR_MODEL_DIR environment variable. The SDK installer sets NVAR_MODEL_DIR to %ProgramFiles%\NVIDIA Corporation\NVIDIA AR SDK\models.

  • NV_AR_SDK_PATH

    By default, applications that use the SDK will try to load the SDK DLL and its dependencies from the SDK install directory, for example, %ProgramFiles%\NVIDIA Corporation\NVIDIA AR SDK\models. Applications might also want to include and load the SDK DLL and its dependencies directly from the application folder.

    To prevent the files from being loaded from the install directory, the application can set this environment variable to USE_APP_PATH. If NV_AR_SDK_PATH is set to USE_APP_PATH, instead of loading the binaries from the Program Files install directory, the SDK follows the standard OS search order to load the binaries, for example, the app folder followed by the PATH environment variable.

    Note: Set this variable only for the application process. Setting this variable as a user or a system variable affects other applications that use the SDK.

Notices

Notice

This document is provided for information purposes only and shall not be regarded as a warranty of a certain functionality, condition, or quality of a product. NVIDIA Corporation (“NVIDIA”) makes no representations or warranties, expressed or implied, as to the accuracy or completeness of the information contained in this document and assumes no responsibility for any errors contained herein. NVIDIA shall have no liability for the consequences or use of such information or for any infringement of patents or other rights of third parties that may result from its use. This document is not a commitment to develop, release, or deliver any Material (defined below), code, or functionality.

NVIDIA reserves the right to make corrections, modifications, enhancements, improvements, and any other changes to this document, at any time without notice.

Customer should obtain the latest relevant information before placing orders and should verify that such information is current and complete.

NVIDIA products are sold subject to the NVIDIA standard terms and conditions of sale supplied at the time of order acknowledgement, unless otherwise agreed in an individual sales agreement signed by authorized representatives of NVIDIA and customer (“Terms of Sale”). NVIDIA hereby expressly objects to applying any customer general terms and conditions with regards to the purchase of the NVIDIA product referenced in this document. No contractual obligations are formed either directly or indirectly by this document.

NVIDIA products are not designed, authorized, or warranted to be suitable for use in medical, military, aircraft, space, or life support equipment, nor in applications where failure or malfunction of the NVIDIA product can reasonably be expected to result in personal injury, death, or property or environmental damage. NVIDIA accepts no liability for inclusion and/or use of NVIDIA products in such equipment or applications and therefore such inclusion and/or use is at customer’s own risk.

NVIDIA makes no representation or warranty that products based on this document will be suitable for any specified use. Testing of all parameters of each product is not necessarily performed by NVIDIA. It is customer’s sole responsibility to evaluate and determine the applicability of any information contained in this document, ensure the product is suitable and fit for the application planned by customer, and perform the necessary testing for the application in order to avoid a default of the application or the product. Weaknesses in customer’s product designs may affect the quality and reliability of the NVIDIA product and may result in additional or different conditions and/or requirements beyond those contained in this document. NVIDIA accepts no liability related to any default, damage, costs, or problem which may be based on or attributable to: (i) the use of the NVIDIA product in any manner that is contrary to this document or (ii) customer product designs.

No license, either expressed or implied, is granted under any NVIDIA patent right, copyright, or other NVIDIA intellectual property right under this document. Information published by NVIDIA regarding third-party products or services does not constitute a license from NVIDIA to use such products or services or a warranty or endorsement thereof. Use of such information may require a license from a third party under the patents or other intellectual property rights of the third party, or a license from NVIDIA under the patents or other intellectual property rights of NVIDIA.

Reproduction of information in this document is permissible only if approved in advance by NVIDIA in writing, reproduced without alteration and in full compliance with all applicable export laws and regulations, and accompanied by all associated conditions, limitations, and notices.

THIS DOCUMENT AND ALL NVIDIA DESIGN SPECIFICATIONS, REFERENCE BOARDS, FILES, DRAWINGS, DIAGNOSTICS, LISTS, AND OTHER DOCUMENTS (TOGETHER AND SEPARATELY, “MATERIALS”) ARE BEING PROVIDED “AS IS.” NVIDIA MAKES NO WARRANTIES, EXPRESSED, IMPLIED, STATUTORY, OR OTHERWISE WITH RESPECT TO THE MATERIALS, AND EXPRESSLY DISCLAIMS ALL IMPLIED WARRANTIES OF NONINFRINGEMENT, MERCHANTABILITY, AND FITNESS FOR A PARTICULAR PURPOSE. TO THE EXTENT NOT PROHIBITED BY LAW, IN NO EVENT WILL NVIDIA BE LIABLE FOR ANY DAMAGES, INCLUDING WITHOUT LIMITATION ANY DIRECT, INDIRECT, SPECIAL, INCIDENTAL, PUNITIVE, OR CONSEQUENTIAL DAMAGES, HOWEVER CAUSED AND REGARDLESS OF THE THEORY OF LIABILITY, ARISING OUT OF ANY USE OF THIS DOCUMENT, EVEN IF NVIDIA HAS BEEN ADVISED OF THE POSSIBILITY OF SUCH DAMAGES. Notwithstanding any damages that customer might incur for any reason whatsoever, NVIDIA’s aggregate and cumulative liability towards customer for the products described herein shall be limited in accordance with the Terms of Sale for the product.

VESA DisplayPort

DisplayPort and DisplayPort Compliance Logo, DisplayPort Compliance Logo for Dual-mode Sources, and DisplayPort Compliance Logo for Active Cables are trademarks owned by the Video Electronics Standards Association in the United States and other countries.

HDMI

HDMI, the HDMI logo, and High-Definition Multimedia Interface are trademarks or registered trademarks of HDMI Licensing LLC.

ARM

ARM, AMBA and ARM Powered are registered trademarks of ARM Limited. Cortex, MPCore and Mali are trademarks of ARM Limited. All other brands or product names are the property of their respective holders. "ARM" is used to represent ARM Holdings plc; its operating company ARM Limited; and the regional subsidiaries ARM Inc.; ARM KK; ARM Korea Limited.; ARM Taiwan Limited; ARM France SAS; ARM Consulting (Shanghai) Co. Ltd.; ARM Germany GmbH; ARM Embedded Technologies Pvt. Ltd.; ARM Norway, AS and ARM Sweden AB.

OpenCL

OpenCL is a trademark of Apple Inc. used under license to the Khronos Group Inc.

Trademarks

NVIDIA, the NVIDIA logo, and cuBLAS, CUDA, CUDA Toolkit, cuDNN, DALI, DIGITS, DGX, DGX-1, DGX-2, DGX Station, DLProf, GPU, JetPack, Jetson, Kepler, Maxwell, NCCL, Nsight Compute, Nsight Systems, NVCaffe, NVIDIA Ampere GPU architecture, NVIDIA Deep Learning SDK, NVIDIA Developer Program, NVIDIA GPU Cloud, NVLink, NVSHMEM, PerfWorks, Pascal, SDK Manager, T4, Tegra, TensorRT, TensorRT Inference Server, Tesla, TF-TRT, Triton Inference Server, Turing, and Volta are trademarks and/or registered trademarks of NVIDIA Corporation in the United States and other countries. Other company and product names may be trademarks of the respective companies with which they are associated.