Programming Guide: Automatic Speech Recognition with Riva ASR (GGML)

The focus of this guide is on using NVIGI to integrate the Riva ASR GGML plugin into a graphics application. The plugin runs the Parakeet-TDT family of acoustic models (v2 English, v3 multilingual) on top of the GGML inference library, with backends for CUDA, Vulkan, D3D12, and CPU.

AUDIO FORMAT (HARD REQUIREMENT): All backends consume 16 kHz, 16-bit, mono PCM. The plugin does not resample, downmix, or convert. Anything else is rejected with kResultInvalidParameter.

OFFLINE-ONLY: This release supports synchronous offline transcription of a complete audio buffer. Streaming, word boosting, separate-punctuation models, silence detection, and beam-search / language-model decoding are not exposed. The corresponding fields and structs are simply absent from ASRRivaGGMLCreationParameters / ASRRivaGGMLRuntimeParameters. See appendix E for the exhaustive list.

GPU INFERENCE SCHEDULING PRIORITY: The SetGpuInferenceSchedulingMode call on CUDA + CIG requires NVIDIA driver R575 or newer. On older drivers the call is silently ignored and transcription itself still works. The plugin does not pin a driver version for instance creation.

IMPORTANT: This guide contains pseudo-code. For an end-to-end working reference see source/samples/nvigi.asr.sample/sample_asr.cpp (CLI) and source/samples/nvigi.3d/ (3D / graphics integration).

1.0 Initialize and Shutdown

Load nvigi.core.framework.dll, resolve nvigiInit / nvigiLoadInterface / nvigiUnloadInterface / nvigiShutdown, and tell NVIGI where the plugin DLLs live:

#include <windows.h>
#include <nvigi.h>
#include "nvigi_asr_riva.h"

HMODULE core = LoadLibraryExA(
    (std::string(sdkPath) + "\\nvigi.core.framework.dll").c_str(),
    nullptr, LOAD_LIBRARY_SEARCH_DLL_LOAD_DIR | LOAD_LIBRARY_SEARCH_DEFAULT_DIRS);
auto nvigiInit            = (PFun_nvigiInit*)           GetProcAddress(core, "nvigiInit");
auto nvigiLoadInterface   = (PFun_nvigiLoadInterface*)  GetProcAddress(core, "nvigiLoadInterface");
auto nvigiUnloadInterface = (PFun_nvigiUnloadInterface*)GetProcAddress(core, "nvigiUnloadInterface");
auto nvigiShutdown        = (PFun_nvigiShutdown*)       GetProcAddress(core, "nvigiShutdown");

auto quietLogMessageCallback = [](nvigi::LogType type, const char* msg)
{
    (void)type;
    (void)msg;
};

const char* pluginPaths[] = { sdkPath };
nvigi::Preferences pref{};
pref.logLevel              = nvigi::LogLevel::eVerbose; // emits ASR timing lines; use eOff to suppress logs
pref.showConsole           = false; // keep stdout clean; logs still go to utf8PathToLogsAndData
pref.numPathsToPlugins     = 1;
pref.utf8PathsToPlugins    = pluginPaths;
pref.utf8PathToLogsAndData = sdkPath;
pref.logMessageCallback    = quietLogMessageCallback; // optional no-op callback for file-only logging

nvigiInit(pref, nullptr, nvigi::kSDKVersion);

nvigiInit and nvigiShutdown are once-per-process. Call nvigiShutdown() and FreeLibrary(core) at process exit.

2.0 Obtain the ASR Interface

Each backend ships as its own DLL with its own PluginID. Pick the one matching the DLL you actually loaded:

Backend	DLL	PluginID constant
CUDA	nvigi.plugin.asr.riva-ggml.cuda.dll	nvigi::plugin::asr::riva_ggml::cuda::kId
D3D12	nvigi.plugin.asr.riva-ggml.d3d12.dll	nvigi::plugin::asr::riva_ggml::d3d12::kId
Vulkan	nvigi.plugin.asr.riva-ggml.vk.dll	nvigi::plugin::asr::riva_ggml::vulkan::kId
CPU	nvigi.plugin.asr.riva-ggml.cpu.dll	nvigi::plugin::asr::riva_ggml::cpu::kId

There is no in-process fallback. If the DLL fails to load, that backend is unavailable for the rest of the process — ship more than one DLL and pick at startup if you need to support multiple hardware configurations. See SelectAutoPlugin in the 3D sample for a working selection example.

nvigi::IAutoSpeechRecognition* iasr{};
if (NVIGI_FAILED(result, nvigiGetInterfaceDynamic(
        nvigi::plugin::asr::riva_ggml::cuda::kId, &iasr, nvigiLoadInterface)))
{
    // Handle error here
}

NOTE: An interface can only be obtained for a feature that is available on the user’s system, and remains valid as long as the plugin is loaded.

3.0 Create an ASR Instance

Instance creation does three things you should know about: it loads the GGUF off disk, it builds the GGML compute graph for the chosen backend, and it runs warmup to make the first real inference fast. Expect creation to take a few hundred milliseconds. This is normal — see section 3.4.

3.1 Pick a Model

Both supported models live under data/nvigi.models/nvigi.plugin.asr.riva-ggml/{GUID}/ and are referenced by GUID:

Model	GUID	Languages
Parakeet-TDT-0.6B-v2	{163BD6DC-248D-4238-BC9B-B76D2AB1C3DD}	English (en-US) only
Parakeet-TDT-0.6B-v3	{9739C15F-9A65-484D-B091-FFA4B9DDD0EE}	25 European languages, auto-detected

Pick v2 if your audio is English-only — it has noticeably better WER on English than v3 (see Developer Pack Overview, section 3). Pick v3 if the input might be any of: Bulgarian, Croatian, Czech, Danish, Dutch, English, Estonian, Finnish, French, German, Greek, Hungarian, Italian, Latvian, Lithuanian, Maltese, Polish, Portuguese, Romanian, Russian, Slovak, Slovenian, Spanish, Swedish, Ukrainian. No language tag is needed at runtime — v3 detects automatically.

If you want to enumerate the models a given pack actually has installed instead of hard-coding the GUID, call getCapsAndRequirements after loading the ASR interface and point CommonCreationParameters::utf8PathToModels at the pack’s model repository:

nvigi::ASRRivaGGMLCreationParameters asrParams{};
nvigi::CommonCreationParameters common{};
common.utf8PathToModels = myPathToNVIGIModelRepository;
asrParams.chain(common);

nvigi::ASRRivaGGMLCapabilitiesAndRequirements* caps{};
nvigi::Result result = nvigi::getCapsAndRequirements(iasr, asrParams, &caps);

if (result == nvigi::kResultOk && caps && caps->common)
{
    const nvigi::CommonCapabilitiesAndRequirements& models = *caps->common;

    for (uint32_t i = 0; i < models.numSupportedModels; ++i)
    {
        const char* name = models.supportedModelNames[i];
        const char* guid = models.supportedModelGUIDs[i];
        size_t vramMB = models.modelMemoryBudgetMB[i];
        bool installedLocally =
            !(models.modelFlags[i] & nvigi::kModelFlagRequiresDownload);

        // Use guid as CommonCreationParameters::modelGUID when creating the instance.
    }
}

3.2 Common and ASR-specific Creation Parameters

nvigi::CommonCreationParameters common{};
common.utf8PathToModels = myPathToNVIGIModelRepository; // UTF-8 path containing nvigi.plugin.asr.riva-ggml/<GUID>/
common.modelGUID        = "{163BD6DC-248D-4238-BC9B-B76D2AB1C3DD}"; // v2 English
common.vramBudgetMB     = 1500;   // informational — used by the framework to filter models, not enforced at runtime
common.numThreads       = (int32_t)std::thread::hardware_concurrency(); // honoured only by the CPU backend; GPU backends ignore (see section 3.4)

nvigi::ASRRivaGGMLCreationParameters asrParams{};
if (NVIGI_FAILED(res, asrParams.chain(common)))
{
    // handle error
}
asrParams.backend = nvigi::ASRGGMLBackend::eCUDA; // or eD3D12 / eVulkan / eCPU

The default backend is CPU, and the plugin logs a warning when CPU is used. For GPU acceleration, choose the plugin first (for example CUDA -> renderer-compatible GPU backend -> CPU), then pass the matching explicit backend enum when creating the instance.

ASRRivaGGMLCreationParameters has only this one field today; everything else (model selection, VRAM budget, thread count) lives on the chained CommonCreationParameters. Future capabilities will land here.

3.3 Graphics Interop — Chain a Graphics-parameters Struct

Chain a graphics-parameters struct only when you want one of these specific effects. There is no benefit to chaining “just in case” — without a chained struct each backend creates and manages its own device.

Backend you loaded	Renderer you have	What chaining does	Chain
CUDA	D3D12	CIG (CUDA-In-Graphics) — share queues with renderer	D3D12Parameters
CUDA	Vulkan	CIG — share queues with renderer	VulkanParameters
CUDA	none	nothing — leave it unchained	—
D3D12	D3D12	Reuse the application’s ID3D12Device and queues	D3D12Parameters
Vulkan	Vulkan	Reuse the application’s VkInstance / VkDevice	VulkanParameters
CPU	anything	nothing — chained graphics structs are ignored	—

Minimal D3D12 chain (works for both D3D12-backend device sharing and CUDA + CIG on a D3D12 renderer):

nvigi::D3D12Parameters d3d12Params{};
d3d12Params.device       = myD3D12Device;     // mandatory
d3d12Params.queue        = myDirectQueue;     // optional (graphics queue)
d3d12Params.queueCompute = myComputeQueue;    // mandatory for CIG; same TSG as direct queue
d3d12Params.queueCopy    = myCopyQueue;       // optional
if (NVIGI_FAILED(res, asrParams.chain(d3d12Params)))
{
    // handle error
}

Minimal Vulkan chain (queues are only required for CIG; for plain Vulkan-backend device reuse, instance + physicalDevice + device is enough):

nvigi::VulkanParameters vkParams{};
vkParams.instance        = myVkInstance;
vkParams.physicalDevice  = myVkPhysicalDevice;
vkParams.device          = myVkDevice;
vkParams.queue           = myGraphicsQueue;   // required for CIG; optional otherwise
vkParams.queueCompute    = myComputeQueue;    // required for CIG; optional otherwise
vkParams.queueTransfer   = myTransferQueue;   // optional
if (NVIGI_FAILED(res, asrParams.chain(vkParams)))
{
    // handle error
}

IMPORTANT: Do not chain the same parameter struct into multiple parameter chains. If you create more than one NVIGI instance with shared D3D12Parameters / VulkanParameters, make a copy per chain — re-chaining the same struct produces a malformed list and instance creation fails.

IMPORTANT: Providing a D3D12 / Vulkan device and queue is highly recommended whenever the same process also renders — it avoids creating a parallel device for inference and is required for CIG.

For deep details — Agility SDK requirements, ReBAR flag, the three Vulkan extensions ggml-vulkan needs, allocator callbacks — see appendices A, B, and D.

3.4 Warmup — What createInstance is Doing

The plugin’s built-in createInstance warmup runs five short transcriptions on synthetic 3, 6, 9, 12, and 15 second clips. This is not optional and exists for two reasons:

• On CUDA, GGML captures a CUDA graph per encoder input shape during these inferences. After 5+ distinct shapes, capture is permanently disabled to avoid recapture overhead on every real call. Without warmup, the first real inference pays the capture cost.

• On D3D12 and Vulkan, this is when shader specializations get compiled and pipelines warm.

Two practical consequences:

• createInstance takes a few hundred ms. Don’t call it on the main thread of a frame-rate-sensitive application; the 3D sample creates instances on a worker thread (see NVIGIContext::ReloadASRModel).

• If your app clips are <= ~15 s, built-in warmup is usually enough. If the application normally submits or permits longer clips, issue one additional dummy evaluate() immediately after createInstance at the application’s maximum expected clip length. Use a silent 16 kHz mono buffer so shader/pipeline compilation or CUDA graph/capture work happens off the user-facing path:

const uint32_t maxExpectedSeconds = 30;
std::vector<int16_t> silence(maxExpectedSeconds * 16000);

nvigi::CpuData warmupData{};
warmupData.buffer      = silence.data();
warmupData.sizeInBytes = silence.size() * sizeof(int16_t);

nvigi::InferenceDataAudio warmupAudio{warmupData};
warmupAudio.bitsPerSample = 16;
warmupAudio.samplingRate  = 16000;
warmupAudio.channels      = 1;

std::vector<nvigi::InferenceDataSlot> warmupSlots = {
    {nvigi::kASRDataSlotAudio, warmupAudio}
};
nvigi::InferenceDataSlotArray warmupInputs = {
    warmupSlots.size(), warmupSlots.data()
};

nvigi::ASRRivaGGMLRuntimeParameters warmupRuntime{};

nvigi::InferenceExecutionContext warmupCtx{};
warmupCtx.instance          = asrInstance;
warmupCtx.runtimeParameters = warmupRuntime;
warmupCtx.callback          = asrCallback;
warmupCtx.callbackUserData  = &myCallbackCtx;
warmupCtx.inputs            = &warmupInputs;

asrInstance->evaluate(&warmupCtx);

Two other behaviours worth internalising while we’re here:

• numThreads is honoured only by the CPU backend. GPU backends (CUDA, D3D12, Vulkan) ignore it — per-thread GPU streams don’t buy anything for this workload.

• vramBudgetMB is informational. It is not a hard cap; the plugin does not abort if actual usage exceeds it. It exists so the NVIGI framework can pick an appropriate model. Set it comfortably above the ~1.2 GB resident cost of the Parakeet models.

3.5 Putting it Together

nvigi::InferenceInstance* asrInstance{};
if (NVIGI_FAILED(res, iasr->createInstance(asrParams, &asrInstance)))
{
    LOG("createInstance failed, code %d", res);
}

That’s the entire creation surface. Everything past this point is per-evaluate() work.

4.0 Audio Input

The plugin consumes a complete audio buffer per evaluate() call — there is no streaming mode in this release. The samples include a reusable microphone helper at source/samples/common/audio/AudioRecordingHelper.h that records 16 kHz, 16-bit, mono PCM input. You can use it directly from sample code or roll your own:

AudioRecordingHelper::RecordingInfo* audioInfo = AudioRecordingHelper::StartRecordingAudio();
// ... user speaks ...
nvigi::CpuData audioData;
nvigi::InferenceDataAudio wavData(audioData);
AudioRecordingHelper::StopRecordingAudio(audioInfo, &wavData);

If you record audio yourself, the format is non-negotiable: 16 kHz sample rate, 16-bit signed PCM, single channel. Build the input slot like this:

nvigi::CpuData audioData{};
audioData.buffer      = reinterpret_cast<const uint8_t*>(pcm.data());
audioData.sizeInBytes = pcm.size() * sizeof(int16_t);

nvigi::InferenceDataAudio audio{audioData};
audio.bitsPerSample = 16;
audio.samplingRate  = 16000;
audio.channels      = 1;

5.0 Receive Inferred Data

This plugin is offline-only and emits exactly one terminal callback per evaluate(): either kInferenceExecutionStateDone (transcription is in kASRDataSlotTranscribedText) or kInferenceExecutionStateInvalid (failed; no text). It does not emit kInferenceExecutionStateDataPending or kInferenceExecutionStateDataPartial; those are streaming-only states defined in nvigi_ai.h for plugins that support it.

auto asrCallback = [](const nvigi::InferenceExecutionContext* execCtx,
                      nvigi::InferenceExecutionState state,
                      void* userData) -> nvigi::InferenceExecutionState
{
    auto* userCtx = static_cast<HostASRCallbackCtx*>(userData);

    if (execCtx && execCtx->outputs)
    {
        const nvigi::InferenceDataText* text{};
        if (execCtx->outputs->findAndValidateSlot(nvigi::kASRDataSlotTranscribedText, &text) && text)
        {
            userCtx->transcription = text->getUTF8Text();
        }
    }

    if (state == nvigi::kInferenceExecutionStateDone)
    {
        // Final and only result is ready.
    }

    if (userCtx->needToInterruptInference)
    {
        return nvigi::kInferenceExecutionStateCancel;
    }
    return state;
};

IMPORTANT: Input and output data slots provided in the execution context are only valid during the callback. Copy what you need (getUTF8Text() returns into your own std::string) before returning.

NOTE: Returning kInferenceExecutionStateCancel aborts the in-flight inference. Cleanup is still guaranteed.

The plugin also defines kASRDataSlotPartialText in the public header for forward compatibility with future streaming plugins. It is not populated by the GGML plugin — read only kASRDataSlotTranscribedText here.

6.0 Prepare the Execution Context

6.1 Runtime Parameters

struct ASRRivaGGMLRuntimeParameters {
    // empty — reserved for forward compatibility
};

The struct is intentionally empty. TDT decoding is greedy; there is no sampling strategy, beam size, prompt, temperature, or no-speech threshold to expose. ASRRivaGGMLRuntimeParameters must still be assigned to InferenceExecutionContext::runtimeParameters on every evaluate() call so the chained-struct walk works correctly.

6.2 Building the Execution Context

std::vector<nvigi::InferenceDataSlot> slots = { {nvigi::kASRDataSlotAudio, audio} };
nvigi::InferenceDataSlotArray inputs = { slots.size(), slots.data() };

nvigi::ASRRivaGGMLRuntimeParameters asrRuntime{};

nvigi::InferenceExecutionContext asrContext{};
asrContext.instance          = asrInstance;
asrContext.callback          = asrCallback;
asrContext.callbackUserData  = &myCallbackCtx;
asrContext.inputs            = &inputs;
asrContext.runtimeParameters = asrRuntime;

IMPORTANT: The execution context and all referenced data must remain valid until evaluate() returns (synchronous case) or until the callback fires kInferenceExecutionStateDone / kInferenceExecutionStateInvalid (async case).

7.0 Add ASR Inference to the Pipeline

Pick evaluate() for blocking transcription on the calling thread, or evaluateAsync() to run it on an internal worker. Both deliver the result through the callback.

if (audioReady)
{
    if (offThread)
    {
        // Non-blocking — callback fires from a worker thread.
        if (NVIGI_FAILED(res, asrContext.instance->evaluateAsync(&asrContext)))
        {
            LOG("evaluateAsync failed, code %d", res);
        }
    }
    else
    {
        // Blocking — callback fires before this returns.
        if (NVIGI_FAILED(res, asrContext.instance->evaluate(&asrContext)))
        {
            LOG("evaluate failed, code %d", res);
        }
    }
}

With NVIGI file logging enabled (logLevel != eOff and utf8PathToLogsAndData set), every evaluate() writes one INFO line that tells you exactly where time went. The CLI sample also installs a no-op logMessageCallback and leaves showConsole disabled, so these plugin INFO lines go to nvigi_log_*.txt without being echoed to stdout:

Preprocessing: <ms> | Encoder: <ms> | Decoder: <ms> | Total: <ms>

— mel-spectrogram extraction, FastConformer encoder, TDT decoder loop, end-to-end. When someone says “it’s slow”, this line tells you which one to blame.

NOTE — threading model: One instance, one in-flight evaluate() at a time. The async path serializes overlapping calls into the same instance through an internal std::future, so the second evaluateAsync() waits for the first to finish. If you genuinely need parallel transcription, create one instance per concurrent stream.

IMPORTANT — graphics integration: When the CUDA backend was created with CIG (a D3D12Parameters / VulkanParameters chained), the plugin re-applies the current GPU inference scheduling priority on every evaluate(). Set the priority once via IHWICommon::SetGpuInferenceSchedulingMode (see appendix C) and it takes effect on the next call. This is how the 3D sample lets the user pick “Prioritize Graphics” vs “Prioritize Inference” from the UI.

8.0 Destroy the Instance

Once an instance is no longer needed:

if (NVIGI_FAILED(result, iasr->destroyInstance(asrInstance)))
{
    // Check error
}

9.0 Unload the Interface

if (NVIGI_FAILED(result, nvigiUnloadInterface(
        nvigi::plugin::asr::riva_ggml::cuda::kId, iasr)))
{
    // Check error
}

nvigiShutdown();
FreeLibrary(core);

That’s the whole API surface.

---

Appendix

A. D3D12 graphics interop

Full struct:

struct D3D12Parameters {
    ID3D12Device*        device{};
    ID3D12CommandQueue*  queue{};                            // direct (graphics) queue
    ID3D12CommandQueue*  queueCompute{};                     // mandatory for CIG
    ID3D12CommandQueue*  queueCopy{};
    PFun_createCommittedResource* createCommittedResourceCallback{};
    PFun_destroyResource*         destroyResourceCallback{};
    void* createCommitResourceUserContext{};
    void* destroyResourceUserContext{};
    D3D12ParametersFlags flags{};                            // eDisableReBAR, eComputeQueueSharedWithFrame
};

Device requirements. The host-supplied ID3D12Device must support shader model 6.6 or higher. To guarantee this across Windows OS versions, ship Microsoft Agility SDK 1.600.0 or newer with your executable:

extern "C" __declspec(dllexport) UINT         D3D12SDKVersion = 610;
extern "C" __declspec(dllexport) const char * D3D12SDKPath    = ".\\D3D12\\";

NOTE: A D3D12 folder must be created next to the executable and contain D3D12Core.dll from the Agility SDK. The pack ships this DLL.

In addition to the Agility SDK, dxcompiler.dll and dxil.dll (the DXC compiler and DXIL signer) must be present next to the executable. The ggml-d3d12 backend invokes them at runtime to compile its shaders.

ReBAR / GPU upload heap. The ggml-d3d12 backend opportunistically uses D3D12_HEAP_TYPE_GPU_UPLOAD (Resizable BAR) when the OS, driver, and BIOS all support it. It is faster but optional:

Feature	First supported Windows OS	First supported Agility SDK
GPU upload heap (ReBAR)	Windows 11 Insider Preview Build 26080 or later	1.613.0

IMPORTANT: On some systems ReBAR must be explicitly enabled in the BIOS. Set D3D12ParametersFlags::eDisableReBAR to opt out unconditionally.

CIG-specific caveat. When D3D12Parameters is chained to a CUDA-backend instance for CIG, the compute queue must be in the same Timeslice Group (TSG) as the application’s graphics queue. If it isn’t, CUDA and the renderer will not share scheduling and CIG silently degrades into a non-CIG CUDA context with extra setup. The 3D sample’s Initialize_postDevice is a working reference.

B. Vulkan graphics interop

Full struct:

struct VulkanParameters {
    VkPhysicalDevice physicalDevice{};
    VkDevice         device{};
    VkInstance       instance{};
    VkQueue          queue{};                  // VK_QUEUE_GRAPHICS_BIT
    VkQueue          queueCompute{};           // VK_QUEUE_COMPUTE_BIT only
    VkQueue          queueTransfer{};          // VK_QUEUE_TRANSFER_BIT only
    PFun_allocateMemoryCallback* allocateMemoryCallback{};
    PFun_freeMemoryCallback*     freeMemoryCallback{};
    void* allocateMemoryCallbackUserContext{};
    void* freeMemoryCallbackUserContext{};
};

NOTE: This appendix only matters if you are chaining VulkanParameters. Without a chain, the Vulkan backend creates its own VkInstance / VkDevice with the right configuration and you can ignore everything below.

VkInstance and VkDevice requirements.

• VkInstance must be created with API 1.3.0 or higher.

• VkDevice must be created with VkPhysicalDeviceFeatures2, VkPhysicalDeviceVulkan11Features, and VkPhysicalDeviceVulkan12Features chained into VkDeviceCreateInfo.

Required device extensions when sharing a VkDevice:

Extension	Why
VK_KHR_shader_integer_dot_product	Required by ggml-vulkan’s quantized matmul pipelines. Without it, dispatches crash.
VK_KHR_pipeline_executable_properties	Pipeline introspection.
VK_NV_shader_sm_builtins	NVIDIA-only, SM-aware tuning. Safe to skip on non-NVIDIA.

If certain extensions are not available, appropriate fallbacks are used where possible — except for VK_KHR_shader_integer_dot_product, which is hard-required.

See source/samples/nvigi.3d/src/DeviceManagerOverride/DeviceManagerOverride_VK.cpp for a reference Donut device-manager override that injects these extensions before vkCreateDevice.

C. CUDA + CIG GPU inference scheduling

When the CUDA backend is created with CIG (a D3D12Parameters / VulkanParameters chained), the host can tell the GPU how to balance inference vs. rendering. Three modes are exposed via the HWI-Common interface:

namespace nvigi::SchedulingMode {
    constexpr uint32_t kPrioritizeCompute  = 0;
    constexpr uint32_t kBalance            = 1;
    constexpr uint32_t kPrioritizeGraphics = 2;
}

nvigi::IHWICommon* hwi{};
nvigiGetInterfaceDynamic(nvigi::plugin::hwi::common::kId, &hwi, nvigiLoadInterface);
hwi->SetGpuInferenceSchedulingMode(nvigi::SchedulingMode::kBalance);

Pick kPrioritizeGraphics for frame-rate-critical apps, kPrioritizeCompute for latency-critical voice commands, kBalance for most cases. The setting is re-applied by the plugin on every evaluate(), so changes take effect on the next inference. For GPU-light workloads the difference is small.

DRIVER REQUIREMENT: This call is only effective on NVIDIA driver R575 or newer and when the resolved IHWICuda::getVersion() is >= 2. On older drivers the call is silently ignored — transcription works, the priority hint just doesn’t take effect.

The setting is global to the process for as long as the HWI-Common interface lives. It is a no-op on D3D12, Vulkan, and CPU backends.

D. Memory tracking and allocator interop

All three graphics-parameters structs expose hooks for tracking or overriding GPU memory allocation done by the plugin. They are independent of device sharing — you can chain them with or without supplying queues.

D3D12

NVIGI provides a callback mechanism to track / allocate / free GPU resources, defined in nvigi_d3d12.h:

ID3D12Resource* createCommittedResource(
    ID3D12Device* device, const D3D12_HEAP_PROPERTIES* pHeapProperties,
    D3D12_HEAP_FLAGS HeapFlags, const D3D12_RESOURCE_DESC* pDesc,
    D3D12_RESOURCE_STATES InitialResourceState, const D3D12_CLEAR_VALUE* pOptimizedClearValue,
    void* userContext)
{
    ID3D12Resource* resource = nullptr;
    HRESULT hr = device->CreateCommittedResource(
        pHeapProperties, HeapFlags, pDesc, InitialResourceState,
        pOptimizedClearValue, IID_PPV_ARGS(&resource));
    if (FAILED(hr)) return nullptr;
    // userContext can be used to record allocation metadata
    return resource;
}

void destroyResource(ID3D12Resource* pResource, void* userContext)
{
    pResource->Release();
}

nvigi::D3D12Parameters params = {};
params.createCommittedResourceCallback = createCommittedResource;
params.destroyResourceCallback         = destroyResource;
params.createCommitResourceUserContext = nullptr;
params.destroyResourceUserContext      = nullptr;

NOTE: REFIID riidResource is not a callback parameter — the requested interface is fixed (ID3D12Resource).

Vulkan

Same idea, defined in nvigi_vulkan.h:

VkResult allocateMemoryVK(VkDevice device, VkDeviceSize size, uint32_t memoryTypeIndex,
                          VkDeviceMemory* outMemory)
{
    VkMemoryAllocateInfo allocInfo = {};
    allocInfo.sType           = VK_STRUCTURE_TYPE_MEMORY_ALLOCATE_INFO;
    allocInfo.allocationSize  = size;
    allocInfo.memoryTypeIndex = memoryTypeIndex;
    return vkAllocateMemory(device, &allocInfo, nullptr, outMemory);
}

void freeMemoryVK(VkDevice device, VkDeviceMemory memory)
{
    if (memory != VK_NULL_HANDLE)
        vkFreeMemory(device, memory, nullptr);
}

nvigi::VulkanParameters params = {};
params.allocateMemoryCallback = allocateMemoryVK;
params.freeMemoryCallback     = freeMemoryVK;

CUDA — report vs override

CUDA exposes two pairs of callbacks. The distinction matters for graphics apps:

• Report callbacks (cudaMallocReportCallback, cudaFreeReportCallback) — informational. The plugin still owns the allocation; you receive (ptr, size, user_context) for telemetry / VRAM accounting. Use these when you just want to see what the inference engine is doing.

• Override callbacks (cudaMallocCallback, cudaFreeCallback) — you are the allocator. Return 0 for success, non-zero for failure. Use these to route inference VRAM through your engine’s allocator / pool / suballocator.

void MallocReportCallback(void* ptr, size_t size, void* user_context)
{
    // Track that the plugin allocated `size` bytes at `ptr`.
}

void FreeReportCallback(void* ptr, size_t size, void* user_context)
{
    // Track that the plugin released the allocation at `ptr`.
}

int32_t MallocCallback(void** ptr, size_t size, int device,
                       bool managed, bool hip, void* user_context)
{
    *ptr = myEngineAllocator.alloc(size, device, managed);
    return *ptr ? 0 : -1;
}

int32_t FreeCallback(void* ptr, void* user_context)
{
    return myEngineAllocator.free(ptr) ? 0 : -1;
}

nvigi::CudaParameters params{};

// Pick report-only OR override; using both is unusual.
params.cudaMallocReportCallback    = MallocReportCallback;
params.cudaMallocReportUserContext = &myCounters;
params.cudaFreeReportCallback      = FreeReportCallback;
params.cudaFreeReportUserContext   = &myCounters;

params.cudaMallocCallback          = MallocCallback;
params.cudaMallocUserContext       = &myEngineAllocator;
params.cudaFreeCallback            = FreeCallback;
params.cudaFreeUserContext         = &myEngineAllocator;

For most graphics apps the report-only path is enough — you get accurate VRAM accounting without taking on responsibility for the allocator behaviour the plugin needs.

E. What this plugin does NOT support

These features existed in the previous Riva ASR ORT plugin but are intentionally absent from this release. There are no stub APIs — the corresponding fields are simply not in ASRRivaGGMLCreationParameters / ASRRivaGGMLRuntimeParameters.

Feature	Status
Streaming / partial results	Not in this release. Will return via natively-streaming TDT acoustic models.
Word boosting	Not supported.
Separate auto-punctuation model	Not needed — TDT models emit punctuation natively where present in training data.
Silence detection	Not supported (was streaming-only in the ORT plugin).
Beam search / KenLM language model	Not supported. Decoding is greedy TDT.
Streaming knobs (chunk size, beam, LM)	N/A — no streaming.

F. Troubleshooting

#	Symptom	Likely cause	Fix
1	createInstance returns kResultInvalidParameter	utf8PathToModels or modelGUID is missing, or the GUID does not match a directory under <utf8PathToModels>/nvigi.plugin.asr.riva-ggml/.	Verify the path. GUIDs must be exact, including braces. See section 3.1.
2	createInstance returns kResultInvalidState	Most often a CUDA + CIG setup problem — the chained D3D12Parameters / VulkanParameters is missing a queue, or its compute queue is not in the same TSG as the application’s graphics queue.	Provide a compute queue in the same TSG as the graphics queue. If CIG was not the goal, simply do not chain the graphics struct. See section 3.3 and appendix A.
3	evaluate returns kResultInvalidParameter	Audio is not 16 kHz, 16-bit, mono PCM. The plugin does not convert.	Convert before submitting. See section 4.0.
4	Transcription quality is poor on non-English audio	Using v2 (English-only) on multilingual input.	Switch to v3 (multilingual). See section 3.1.
5	First inference is slow, subsequent ones are fast	Audio longer than ~15 s tripped a fresh CUDA-graph capture outside the warmup range.	Issue one dummy inference close to your longest expected audio length right after createInstance. See section 3.4.
6	Plugin DLL fails to load	The plugin DLL path is not in Preferences::utf8PathsToPlugins, or a shipped runtime dependency isn’t next to the plugin.	Check utf8PathsToPlugins. For per-backend shipping requirements see Developer Pack Overview, section 6.
7	SetGpuInferenceSchedulingMode has no visible effect	NVIDIA driver older than R575, or loaded HWI-CUDA interface older than v2.	Update the driver. The plugin does not refuse to run; the priority hint is silently ignored. See appendix C.

G. Related docs

• Getting Started — prerequisites, first sample run, integration checklist.

• Samples — CLI sample, 3D sample, what each one demonstrates.

• Developer Pack Overview — pack contents, per-backend runtime dependencies, accuracy and performance numbers.