The Command-Line Reload Sample

The D3D12 model-reloading sample, nvigi.reload shows how a D3D12-based NVIGI application can manually load and unload models for D3D12 GGML-based plugins to/from VRAM. This allows an application to reclaim VRAM when inference is not in use, or to have two model instances ready to quickly swap in and out of VRAM as desired, avoiding having to choose between time-consuming load-from file and keeping multiple models in VRAM at once. It is automatically built as a part of the SDK build. It allows direct typing “to” an LLM as well as the ability to select when to switch models without reloading from file. It shows the current VRAM use along the way. Nore that the VRAM use may not match what is shown as available in other tools such as nvidia-smi. This is because D3D12 manages its own memory, and so once a model is loaded that uses a given amount of memory, when that model is unloaded, the memory is indeed available to other D3D12 app uses (e.g. textures), but D3D12 may not release the memory back to the OS for use by other applications.

These instructions make reference to a set of directories; the location of these directories differ between binary app developer pack and GitHub source. The documentation for each of these define the locations for these directories in the particular use:

  • <SDK_ROOT>: the root of the SDK Plugins tree, which contains the bin directory for the SDK.

  • <SDK_MODELS>: the root of the models tree for the SDK plugins, normally data/nvigi.models

  • <SDK_TEST>: the root of the test data tree for the SDK plugins, normally data/nvigi.test

  • <SDK_PLATFORM>: the target CPU architecture. See Platform Support for information on supported platforms.

Download Required Models

The reload sample requires the following models:

Plugin

Model Name

GUID

nvigi.plugin.gpt.ggml.*

Llama3.2 3b Instruct

01F43B70-CE23-42CA-9606-74E80C5ED0B6

nvigi.plugin.gpt.ggml.*

Nemotron Mini 4B

8E31808B-C182-4016-9ED8-64804FF5B40D

See the top-level documentation that shipped with your development pack for information on how to download these models.

How to Use the Command-Line Reload Sample

When run, the sample should launch a console (or use the one from which it was run); it will await user input. The user may do one of three things at the prompt:

  • Type a chat query as text and press enter. This will be passed directly to the LLM and the LLM response printed.

  • Type an exit command; “Q”, “q” or “quit”. This will cause the sample to exit.

  • Type <unload> and press enter. This will unload all loaded models.

  • Type <llama3> and press enter. This will load the Llama3.2 3b Instruct model if it is not already loaded, or switch to it if it is already loaded. It will unload the other model.

  • Type <nemotron> and press enter. This will load the Nemotron Mini 4B model if it is not already loaded, or switch to it if it is already loaded. It will unload the other model.

Run at Command Line

To run nvigi.reload from the command line, take the following steps (--models is a required argument):

  1. Open a command prompt in <SDK_ROOT>

  2. Run the command:

bin\<SDK_PLATFORM>\<Configuration>\nvigi.reload.exe --models <SDK_MODELS>

  1. In a standard layout binary development pack or GitHub source tree, launching from a current working directory of the <SDK_ROOT> directory, this is:

bin\<SDK_PLATFORM>\<Configuration>\nvigi.reload.exe --models data/nvigi.models

Here are the command line options:

Usage: nvigi.reload [options]

  -m, --models              model repo location (REQUIRED)
  -s, --sdk                 sdk location, (default: exe location)
  --vram                    the amount of vram to use in MB (default: 8192)

Building the Sample

The sample is built as part of the SDK build process. For full build instructions, see Building and Debugging. After building, stage the binaries:

copy_sdk_binaries.bat [-x64|-arm64] [Release|Debug|Production]

This ensures all DLLs and the executable are in the same directory (bin\<SDK_PLATFORM>\<Configuration>, where <SDK_PLATFORM> is the target CPU architecture and is the selected build configuration; see Platform Support for information on supported platforms).

Run in Debugger

For the general procedure and background, see the Building and Debugging guide. The settings specific to this sample are:

Property

Value

VS Project

nvigi/samples/nvigi.reload

Command

<SDK_ROOT>\bin\<SDK_PLATFORM>\<Configuration>\nvigi.reload.exe

Command Arguments

--models data\nvigi.models

Working Directory

<SDK_ROOT>\bin\<SDK_PLATFORM>\<Configuration>