The Command-Line RAG Sample

The RAG sample, nvigi.rag shows an example of an LLM-based chatbot that uses RAG to provide the LLM with detailed context from which it should answer questions. The built-in system prompt and provided context text (data/nvigi.test/nvigi.rag/LegendOfDoria_Corpus.txt) file describes a fantasy game world. The user can type in questions that will be passed to the LLM to be answered in-game.

NOTE: This sample makes use of a CUDA-based backend, and therefore will not work on non-NVIDIA hardware. This will be fixed in future releases.

These instructions use the following path placeholders, which refer to locations that differ between a binary developer pack and a GitHub source tree:

  • <SDK_ROOT>: the root of the SDK Plugins tree, which contains the bin directory for the SDK.

  • <SDK_MODELS>: the root of the models tree for the SDK plugins, normally data/nvigi.models

  • <SDK_PLATFORM>: the target CPU architecture. See Platform Support for information on supported platforms.

Download Required Models

The RAG sample requires the following models:

Plugin

Model Name

GUID

nvigi.plugin.embed.ggml.*

E5 Large Unsupervised

5D458A64-C62E-4A9C-9086-2ADBF6B241C7

nvigi.plugin.gpt.ggml.*

Nemotron Mini 4B

8E31808B-C182-4016-9ED8-64804FF5B40D

See the top-level documentation that shipped with your development pack for information on how to download these models.

How to Use the RAG Sample

When run, the sample should launch a console (or use the one from which it was run); it will await user input. The user may do one of two things:

  • Type in “exit” then Enter, which will exit the sample

  • Type in a Question to the LLM, which will answer them based upon context passed in via the provided text file.

Run at Command Line

To run nvigi.rag from the command line, take the following steps:

  1. Open a command prompt in <SDK_ROOT>

  2. Run the command:

bin\<SDK_PLATFORM>\<Configuration>\nvigi.rag.exe <SDK_MODELS> <text file>

  1. In a standard layout binary development pack or GitHub source tree, launching from a current working directory of the <SDK_ROOT> directory, this might be:

bin\<SDK_PLATFORM>\<Configuration>\nvigi.rag.exe data/nvigi.models data/nvigi.test/nvigi.rag/LegendOfDoria_Corpus.txt

Building the Sample

The sample is built as part of the SDK build process. For full build instructions, see Building and Debugging. After building, stage the binaries:

copy_sdk_binaries.bat [-x64|-arm64] [Release|Debug|Production]

This ensures all DLLs and the executable are in the same directory (bin\<SDK_PLATFORM>\<Configuration>, where <SDK_PLATFORM> is the target CPU architecture and is the selected build configuration; see Platform Support for information on supported platforms).

Run in Debugger

For the general procedure and background, see the Building and Debugging guide. The settings specific to this sample are:

Property

Value

VS Project

nvigi/samples/nvigi.rag

Command

<SDK_ROOT>\bin\<SDK_PLATFORM>\<Configuration>\nvigi.rag.exe

Command Arguments

data\nvigi.models data\nvigi.test\nvigi.rag\LegendOfDoria_Corpus.txt

Working Directory

<SDK_ROOT>\bin\<SDK_PLATFORM>\<Configuration>