In-Game Inferencing SDK (IGI)

User Guide

• NVIDIA In-Game Inferencing (NVIGI) Developer Pack 1.1.0 Release
  • What is NVIGI?
  • The Current Release of the Developer Pack
  • Where Do I Start?
  • Getting Models
    • Cloud Models
    • Manually-Downloadable, Public Models
  • Important Documentation in the Developer Pack
  • Contents of the Developer Pack
  • Known Issues and Important Notes
    • 1.1.0 Release
    • 1.0.0 Release
    • Beta 1
• NVIGI 3D Sample
  • What is the Sample?
  • Requirements
  • Directory Setup
  • Setting up to Run the Sample
    • Downloading Models Offline
    • Setting up the GPT Cloud Plugin
    • Launching the Sample
    • Using the Sample
      • Main UI Page
      • Model Settings UI Pages
    • Logging from the Sample
  • (Re)Building the Sample
    • NVIGI and the Sample
      • Using Standard-layout Prebuilt NVIGI Components
      • Using Locally-Built NVIGI Components
        • Setting up NVIGI Core
        • Setting up the NVIGI SDK Plugins
        • Making the Links
      • Generating the Build Files
      • Building the Sample
  • Running the Sample in the Debugger
  • Command-line Options
    • More Useful Command Line Arguments:
  • Types of prompts supported
  • Multiple backends support
  • Release Notes:
• Architecture
  • Typed And Versioned Structures
    • Interfaces
  • Core
  • Internal API
  • Versioning
  • Plugin Structure
  • External API
    • Initialization
    • Interface loading
    • Interface unloading
    • Shutdown
• GPU Scheduling for AI
  • Supported plugins
  • Prerequisites
  • A note on performance measurement
  • How to use
  • How to set the relative priority of compute and graphics
    • API
  • CiG and D3D Wrappers (e.g. Streamline)
    • Limitations
  • UnrealEngine 5 example code
• NVIGI - Programming Guide
  • Version 1.1.0 Release
  • Table of Contents
  • INTRODUCTION
  • Key Concepts
  • Typed And Versioned Structures
    • Input Data
    • Output Data
  • Thread Safety
  • Security
    • Digital Signatures
    • Elevated Privileges
  • Core API
    • Initialization
    • Shutdown
    • Interfaces
  • Validation
    • Check If Specific Plugin Is Supported
    • Check For Minimum Required OS Version
    • Check For Minimum Required Adapter Driver Version
    • Check For Minimum Required Adapter Vendor
    • Check For Minimum Required Adapter Architecture
    • Check If Specific Plugin Exports An Interface
  • NVIGI and D3D Wrappers (e.g. Streamline)
  • 3rd Party Dependencies
    • Runtime Scenarios With Dependencies
    • Plugin And Dependencies Deployment Examples
      • Everything In One Location:
      • Split In Multiple Locations
      • Invalid Configurations
      • Collisions With The Host
• NVIGI - Programming Guide For Local And Cloud Inference
  • Version 1.1.0 Release
  • Table of Contents
  • INTRODUCTION
  • Key Concepts
  • Model Repository
  • Obtaining Models
    • Local Execution
      • Prompt Templates for LLM Models
    • Remote Execution
  • Common And Custom Capabilities and Requirements
    • Local Plugins
    • Cloud Plugins
  • Creation Parameters
    • Common and Custom
    • Cloud Plugins
    • Local Plugins
      • Compute In Graphics (CIG)
  • Data Types
  • Input Slots
  • Execution Context
    • Blocking Vs Asynchronous Evaluation
  • Obtaining Results
    • Callback Approach
    • Polling Approach
• NVIDIA In-Game Inference AI Plugins
  • Prerequisites
    • Hardware
    • Windows
    • NVIGI Components
  • DIRECTORY SETUP
  • Basic Steps: Summary
  • Setup
  • Building
  • “Installing” Built Binaries
  • Samples
  • Changing an Existing Project
  • Programming Guides
  • Sample App and Source
  • Release Notes:
• Automatic Speech Recognition (ASR) - Whisper Programming Guide
  • Version 1.1.0 General Access
  • 1.0 INITIALIZE AND SHUTDOWN
  • 2.0 OBTAIN ASR INTERFACE
  • 3.0 CREATE ASR INSTANCE(S)
    • 3.1 OBTAIN CAPABILITIES AND REQUIREMENTS FOR MODEL(S)
      • LOCAL
      • CLOUD (VRAM ignored)
    • 3.2 CREATE MODEL INSTANCE
  • 4.0 AUDIO INPUT
    • 4.1 PROCESS COMPLETE AUDIO CLIP
    • 4.2 STREAM AUDIO
    • 4.3 PREPARE INPUT SLOT
  • 5.0 SETUP CALLBACK TO RECEIVE INFERRED DATA
  • 6.0 PREPARE THE EXECUTION CONTEXT
  • 7.0 ADD ASR INFERENCE TO THE PIPELINE
  • 8.0 DESTROY INSTANCE(S)
  • 9.0 UNLOAD INTERFACE(S)
  • APPENDIX
    • MEMORY TRACKING
      • CUDA
• Generative Pre-Trained Transformers (GPT) Programming Guide
  • 1.0 INITIALIZE AND SHUTDOWN
  • 2.0 OBTAIN GPT INTERFACE(S)
  • 3.0 CREATE GPT INSTANCE(S)
    • 3.1 OBTAIN CAPABILITIES AND REQUIREMENTS FOR MODEL(S)
      • LOCAL
      • CLOUD (VRAM ignored)
    • 3.2 CREATE MODEL INSTANCE
  • 4.0 SETUP CALLBACK TO RECEIVE INFERRED DATA
  • 5.0 PREPARE THE EXECUTION CONTEXT
    • 5.1 INSTRUCT MODE
    • 5.2 INTERACTIVE (CHAT) MODE
    • 5.3 CLOUD
      • 5.3.1 INSTRUCT OR CHAT MODE
      • 5.3.1 FULL CONTROL VIA JSON INPUT SLOT
    • 5.4 OUTPUTS
  • 6.0 ADD GPT INFERENCE TO THE PIPELINE
    • 6.1 INSTRUCT MODE
    • 6.2 INTERACT MODE
  • 7.0 DESTROY INSTANCE(S)
  • 8.0 UNLOAD INTERFACE(S)
  • 9.0 VLM (Visual Lanuage Models)
  • APPENDIX
    • MEMORY TRACKING
      • CUDA
    • CHAT MODE SETUP SUMMARY
• Embedding (EMBED) Programming Guide
  • Version 1.1.0 General Access
  • 1.0 INITIALIZE AND SHUTDOWN
  • 2.0 OBTAIN EMBED INTERFACE(S)
  • 3.0 CREATE EMBED INSTANCE(S)
  • 4.0 SETUP CALLBACK TO RECEIVE INFERRED DATA
  • 5.0 PREPARE THE EXECUTION CONTEXT AND EXECUTE INFERENCE
  • 6.0 Do something with the embedding
    • 6.1 Use it to match a player’s answer to a set of predefined answers.
  • 7.0 DESTROY INSTANCE(S)
  • 8.0 UNLOAD INTERFACE(S)
  • 9.0 Exceptions
• NVIGI Public Source GitHub Pull-and-Build Scripts
  • 01_clone_all.bat
    • Prerequisites
  • 02_build_pack_core.bat
    • Prerequisites
  • 03_build_pack_plugins.bat
    • Prerequisites
  • 04_build_sample.bat
    • Prerequisites
• Creating a Customized Plugin
  • Prerequisites
  • Steps
    • 1. Add a New Plugin Project
    • 2. Update Names and GUIDs in Source Files
    • 3. (Optional) Update Your App to Use the New Plugin
    • 4. Customize Your Plugin!
    • 5. Adding New Structures Or Interfaces
    • 6. Using New Structures Or Interfaces
      • 6.1 Data Structures
      • 6.2 Interface Structures
• 3rd PARTY SOFTWARE
  • abseil-cpp
  • catch2
  • cpp-httplib
  • grpc
  • Mozilla Public License Version 2.0
  • Exhibit A - Source Code Form License Notice
  • Exhibit B - “Incompatible With Secondary Licenses” Notice
  • nlohmann_json
  • protobuf
  • address_sorting
  • benchmark
  • bloaty
  • boringssl
  • OpenSSL License
  • Original SSLeay License
  • Licenses for support code
  • capstone
  • c-ares
  • Clara
  • demumble
  • embeddings.cpp
  • emscripten
  • envoy-api
  • everest
  • freebsd_elf
  • $FreeBSD$
  • GoogleTest
  • jsoncpp
  • libcurl
  • libcxxabi
  • libzip
  • lief_pe
  • lua
  • lunit
  • Lunit License
  • mbedtls
  • Microsoft.AI.DirectML
  • Microsoft.ML.OnnxRuntime.DirectML
  • onnx
  • opencensus-proto
  • OpenFST
  • opentelemetry
  • premake
  • protobuf-mutator
  • protoc-gen-validate
  • Pybind11
  • re2
  • TextFlowCpp
  • Thrax
  • treesitter
  • undname.c
  • upb
  • utf8_range
  • xds
  • xxhash
  • zlib
  • llama.cpp
  • whisper.cpp
  • whisper-small
  • phi-2-GGUF
  • dr_libs
  • Dear ImGui
  • GGML Implementation
  • SBERT GGML Implementation
  • BGE Flag GGML Implementation
  • BGE Weights

orphan

plugins/sdk/docs/Samples.md