NVIDIA In-Game Inference AI Plugins

Abstract

This document explains the organization and setup process for the NVIGI AI plugins included in this release pack. After reading you should be able to:

  • Understand the organization of the files and directories that make up the plugins in this pack

  • Set up a development environment for building the included AI plugins from source (as part of learning the basics of how to use NVIGI)

Introduction

In-Game Inferencing (NVIGI) is an open-sourced cross-platform solution that simplifies integration of the latest NVIDIA and other provider’s technologies into applications and games. This framework allows developers to easily implement one single integration and enable multiple technologies supported by the hardware vendor or cloud provider. Supported technologies include AI inference but, due to the generic nature of the NVIGI SDK, can be expanded to graphics or any other field of interest.

For high-level details of NVIGI, as well as links to the official downloads for the product, see NVIDIA In-Game Inferencing

This directory contains the main AI plugins for integrating In-Game Inferencing-based AI workflows into your application.

IMPORTANT: For important changes and bug fixes included in the current release, please see the release notes at the end of this document BEFORE use.

Prerequisites

NVIGI version 1.7.0

Minimum

Recommended

GPU

RTX 30x0/A600

RTX 4080/4090 or RTX 5080/5090

VRAM

8GB

12GB

Windows

Win11 22H2 (version 22H2 - 10.0.22621)

Graphics Driver

r555.85

Development Tools

VS Code or VS2022 with SDK 10.0.22621+

NOTE: Some plugins only support RTX 40x0 and above, and will not be available on RTX 30x0.

NVIGI Components

NVIGI consists of several independent components, of which this SDK directory is only one. Normally, several of these components are used together, and in many cases, developers will actually receive a combined, “standard-layout” pack that is easier to use out of the box. This SDK directory contains plugins focused upon AI inference, especially speech recognition, large language models, and embeddings.

The SDK Plugins pack/tree rely upon and require the NVIGI core APIs, which are shipped as the nvigi_core Plugin Development Kit (PDK) or Runtime Pack. The two packs have different use-cases with the plugins:

  • Applications that want to use NVIGI-based AI features will need the NVIGI Core Runtime Pack, which contains the core DLLs and other runtime components needed to build and run NVIGI-based applications along with pre-built plugins.

  • Developers who want to build their own plugins need the NVIGI Core PDK.

These packs can be found either as:

  • Runtime Pack: A part of any NVIGI binary SDK pack, downloadable fom NVIDIA In-Game Inferencing. In this case, the NVIGI Core Runtime Pack will be located in a subdirectory of this SDK directory called nvigi_core

  • PDK: A pre-built download of the NVIGI Core PDK, downloadable from the NVIGI Source Core Repo as a release artifact with each release.

  • PDK or Runtime Pack: A locally cloned and built copy of the nvigi_core git repository. See the NVIGI Source Root Repo for instructions on how to build and package the core.

IMPORTANT: The NVIGI 1.7.0 RTX Spark Developer Preview:

  • Does NOT support building for the ARM64 target the core libraries or the plugins themselves from the full source posted to GitHub.

  • DOES still support building the entire SDK (inc. core and plugins) x64 target from GitHub source

  • DOES support building the samples provided with the binary releases from the source in the release packs for either platform.

  • An upcoming full release for RTX Spark will add support for building the entire SDK including core libraries and the plugins themselves for ARM64 target from source.

NVIGI core is documented in detail in its own documentation, which should be available in any distribution. Developers should read the documentation in nvigi_core to gain a basic understanding of the overall architecture into which the plugins in this source tree plug into.

Directory Setup

As noted, building the NVIGI plugins SDK from source requires access to a fully-built local copy of the NVIGI Core PDK. We will call the location of this PDK <NVIGI_CORE>.

In the following instructions,

  • <SDK_ROOT> should be replaced with the full path to your NVIGI SDK directory (the path of this README). In binary packs, this is the root directory.

  • <NVIGI_CORE> should be replaced with the fully path to the NVIGI Core PDK, either pulled pre-built or built from git source. See above for details. In binary packs, this is <SDK_ROOT>/nvigi_core.

  • <SDK_MODELS> refers to the location of the nvigi.models AI models directory.

    • This is available in <SDK_ROOT>/data/nvigi.models

  • <SDK_TEST> refers to the location of the nvigi.test AI test data directory.

    • This is available in <SDK_ROOT>/data/nvigi.test

  • <SDK_PLATFORM>: the target CPU architecture. See Platform Support for information on supported platforms.

Required connection from the SDK to the Core

The NVIGI Core is a distinct NVIGI component, and available for download separately from this NVIGI plugins. The core build required depends upon whether the developer intends to use pre-built plugins with their app (Core Runtime) or intends to build their own plugins and the NVIDIA plugins from source (Core PDK). But, in some cases, core may be shipped or pulled with another distribution to make it easier for developers to get started exploring NVIGI. For example:

  • In the binary packs, the NVIGI Core Runtime Pack is included in a subdirectory of the SDK directory called nvigi_core.

  • In source releases, the core source is its own repo, which is cloned separately from the plugins/SDK repo and a junction link is created from <SDK_ROOT>/nvigi_core to the cloned core repo.

It can be quickly determined if this is the case by looking at the contents of the <SDK_ROOT> directory, and checking if a directory called <SDK_ROOT>/nvigi_core is present. If so, then this SDK pack already ships with the NVIGI Core, and it is not necessary to download it separately.

If this directory is not present, then developers need to obtain an instance of the NVIGI Core separately (either download a pre-built one or build one from git source). Further, a directory junction link from <SDK_ROOT>/nvigi_core to <NVIGI_CORE> needs to be created; this is because the NVIGI plugin SDK build scripts assume the NVIGI Core will be located at <SDK_ROOT>/nvigi_core.

This directory junction link can be done easily by running the following command within a Visual Studio Developer Command Prompt from the <SDK_ROOT> directory:

mklink /j nvigi_core <NVIGI_CORE>

Basic Steps: Summary

NOTE: This is a quick-start checklist for x64 builds. For full instructions covering ARM64 cross-compilation, all build configurations, remote deployment, and debugging, see Building and Debugging.

  1. Generate the project files

    1. Open a Visual Studio Developer Command Prompt to the SDK (a.k.a. <SDK_ROOT>) directory

    2. Run setup.bat (add -arm64 when targeting ARM64; see Building and Debugging for ARM64 prerequisites)

  2. Build the project, which will build the plugins and samples from source

    1. Open <SDK_ROOT>\_project\vs20XX\nvigi.sln in Visual Studio

    2. Select the desired platform and build configuration, then build

  3. Copy the updated binaries into the expected, packaged locations

    1. Open a (or use an existing) Visual Studio Developer Command Prompt to the SDK (a.k.a. <SDK_ROOT>) directory

    2. Run copy_sdk_binaries.bat [-x64|-arm64] [Release|Debug|Production]

  4. Run the sample (assumes that the models have been downloaded as per initial setup instructions)

    1. Open a (or use an existing) Visual Studio Developer Command Prompt to the SDK (a.k.a. <SDK_ROOT>) directory

    2. Run bin\<SDK_PLATFORM>\Release\nvigi.basic.exe --models data/nvigi.models --speaker data/nvigi.test/nvigi.tts/chatterbox/spk_emb/aaron_turbo.json

  5. Debug the sample or run on a remote ARM64 device

Basic Steps: Detailed Breakdown

For the complete, authoritative build, run, and debug instructions — covering both x64 and ARM64, all build configurations, remote deployment, and Visual Studio debugger setup — see Building and Debugging.

The steps below cover the items specific to this SDK plugins tree.

Setup

IMPORTANT: As detailed in an earlier section, there must be either:

  • An instance of the NVIGI Core SDK located at <SDK_ROOT>/nvigi_core (which is the default for some NVIGI SDK packs), or

  • A directory junction link from <SDK_ROOT>/nvigi_core to your built and packaged copy of the NVIGI Core PDK, <NVIGI_CORE>.

Once the Core connection is in place, run setup.bat [-x64|-arm64] from a Visual Studio Developer Command Prompt in the <SDK_ROOT> directory. This script does two things:

  1. The NVIDIA tool packman pulls all build dependencies to the local machine and creates links in the external directory.

  2. premake generates project files in _project\vs20XX.

NOTE: If setup.bat fails with an error from packman, re-run it — there are rare link-creation issues on the initial run.

Samples

Setting up and running the samples as shipped pre-built or built using the instructions above is discussed in the individual sample documentation; see The Command-Line Basic Sample to get started.

Programming Guides

For specific instructions and details on specific NVIGI AI plugins, please see the following guides:

Sample App and Source

A 3D-rendered sample application using In-Game Inferencing may be found in most SDK packs; please see its documentation for more details on how to use it to test NVIDIA In-Game Inference AI Plugins.

Release Notes: