Stable Diffusion (SD) Programming Guide

The focus of this guide is on using In-Game Inferencing to integrate a Stable Diffusion model into an application. The plugin is built on top of stable-diffusion.cpp and supports both txt2img (text-to-image) and img2img (image-to-image) generation.

FORK NOTE: This plugin uses an IGI fork of stable-diffusion.cpp based on upstream commit 484baa41e5e006c52dcd4addc38c830b9489745f. The IGI fork adds optimizations for better graphics integration via CiG (CUDA in Graphics), a D3D12 backend, and Marigold support for predicting roughness and metallic material maps (see 3.7 Material maps (Marigold)).

MIN RUNTIME SPEC: Note that all SD GGML-based backends require a CPU supporting AVX2 instructions when running on x64 platforms. Support for this instruction extension is ubiquitous in modern gaming CPUs, but older hardware may not support it. A discrete GPU is also required for usable performance.

IMPORTANT: This guide contains pseudo code. For up-to-date, copy-pasteable implementations see:

Plugin source: source/plugins/nvigi.sd/ggml/sd.cpp

Public header: source/plugins/nvigi.sd/nvigi_sd.h

Modern C++ wrapper: source/samples/shared/cxx_wrappers/sd/sd.hpp

Test reference: source/plugins/nvigi.sd/ggml/tests.h

IMPORTANT NOTE: D3D12 and Vulkan backends are experimental and might not behave or perform as expected.

IMPORTANT NOTE: The D3D12 backend (nvigi.plugin.sd.ggml.d3d12.dll) is provided only precompiled as a part of the downloadable binary pack (nvigi_pack). It is not possible for developers to compile the D3D12 backend plugin from source in this release.

IMPORTANT NOTE: The D3D12 backend (nvigi.plugin.sd.ggml.d3d12.dll) requires an NVIDIA R580 driver or newer in order to be available at runtime. For optimal performance MS Agility SDK is required, see D3D12 Appendix.

IMPORTANT NOTE: The CUDA backend (nvigi.plugin.sd.ggml.cuda.dll) strongly recommends an NVIDIA R580 driver or newer in order to avoid a potential memory leak if CiG (CUDA in Graphics) is used and the application deletes D3D12 command queues mid-application.

1.0 INITIALIZE AND SHUTDOWN

Please read the Programming Guide located in the NVIGI Core package to learn more about initializing and shutting down the NVIGI SDK.

2.0 OBTAIN SD INTERFACE(S)

The SD plugin ships in three backend variants. Pick the one that matches your renderer:

Backend	PluginID	Notes
CUDA	`nvigi::plugin::sd::ggml::cuda::kId`	NVIDIA only. Supports CIG (CUDA-in-Graphics) interop with D3D12.
D3D12	`nvigi::plugin::sd::ggml::d3d12::kId`	Any vendor. NVIDIA-specific scheduling applied when an NVDA adapter is present.
Vulkan	`nvigi::plugin::sd::ggml::vulkan::kId`	Any vendor.

Two interfaces are exposed by every backend:

IStableDiffusion (alias of InferenceInterface) — used for createInstance / destroyInstance / getCapsAndRequirements / evaluate / evaluateAsync / cancelAsyncEvaluation.
IPolledInferenceInterface — used by the asynchronous flow to poll for results (getResults / releaseResults).

nvigi::IStableDiffusion* isd{};
if (NVIGI_FAILED(res, nvigiGetInterface(nvigi::plugin::sd::ggml::cuda::kId, &isd)))
{
    LOG("NVIGI call failed, code %d", res);
}

// Optional, only needed if you call evaluateAsync()
nvigi::IPolledInferenceInterface* iPolled{};
if (NVIGI_FAILED(res, nvigiGetInterface(nvigi::plugin::sd::ggml::cuda::kId, &iPolled)))
{
    LOG("NVIGI call failed, code %d", res);
}

NOTE: One can only obtain an interface for a feature available on the user’s system. Interfaces remain valid as long as the underlying plugin is loaded.

3.0 CREATE SD INSTANCE(S)

Two structs control how the instance is built:

3.1 `SDCreationParameters` — fixed at instance creation

Field	Default	Meaning
`width`	`512`	Default txt2img output width and image-generation warmup width. Img2img width comes from the input image. For Marigold, the internal processing width.
`height`	`512`	Default txt2img output height and image-generation warmup height. Img2img height comes from the input image. For Marigold, the internal processing height.
`nThreads`	`1`	CPU threads used for non-GPU stages (`-1` = `sd_get_num_physical_cores()`).
`flashAttention`	`true`	Enables flash attention for both diffusion and conditioning stages. Disable it if the selected backend or model does not support it.
`maxVRAM`	`nullptr`	Optional VRAM budget specification in GiB, for example `"8"` or `"cuda0=8,vulkan0=6"`. When null, the plugin derives a budget from `CommonCreationParameters::vramBudgetMB`.
`backend`	`nullptr`	Runtime placement specification, for example `"vae=cpu,diffusion=cuda0"`.
`paramsBackend`	`nullptr`	Parameter-storage placement, for example `"*=cpu"` or `"diffusion=disk,clip=cpu"`.
`streamLayers`	`false`	Enable layer residency/prefetch streaming on top of `maxVRAM` graph segmentation.
`eagerLoad`	`false`	Load all parameters into their parameter backends when the context is created.
`enableMmap`	`true`	Use memory-mapped model files when supported. Disable it to force explicit model reads and parameter-buffer allocation.
`vaeTiling`	disabled	Fixed VAE tiling controls used by instance warmup and every evaluation. The default tile size is 16x16 when tiling is enabled.

3.2 VAE tiling and instance warmup

VAE tiling reduces peak memory used by VAE encode and decode by processing the latent in overlapping tiles. It is fixed when the instance is created so warmup and inference always use the same configuration. Enable it when either configured dimension is 1024 pixels or larger to avoid oversized VAE dispatches:

nvigi::SDCreationParameters sdParams{};
sdParams.width = 1024;
sdParams.height = 1024;
sdParams.vaeTiling.enabled = true;
sdParams.vaeTiling.temporalTiling = false;
sdParams.vaeTiling.tileSizeX = 16;
sdParams.vaeTiling.tileSizeY = 16;
sdParams.vaeTiling.targetOverlap = 0.5f;
sdParams.vaeTiling.extraTilingArgs = nullptr; // e.g. "temporal_tile_frames=4,temporal_tile_overlap=1"

tileSizeX and tileSizeY are latent-space dimensions and default to 16. If either absolute size is less than 4, stable-diffusion.cpp uses its internal fallback tile size of 32. targetOverlap is clamped to the range [0.0, 0.5].

relativeSizeX and relativeSizeY take precedence over the corresponding absolute tile size when greater than zero. A relative value in (0, 1] is the fraction of that latent dimension assigned to a tile; a value greater than 1 specifies the approximate number of tiles across that dimension. Tiling trades additional dispatches and overlap work for lower peak VAE memory use.

temporalTiling and extraTilingArgs mirror stable-diffusion.cpp’s extended tiling API. They are primarily useful for temporal/video VAEs; image-only workloads should leave them disabled and null.

Both image-generation and Marigold instances attempt warmup synchronously during createInstance, using the configured width, height, and vaeTiling. Instance creation therefore takes longer and normally absorbs first-compute initialization. A warmup failure is logged but does not invalidate an otherwise loaded instance. An img2img or reference-edit request whose input dimensions differ from the creation dimensions can still incur shape-specific planning on its first evaluation.

3.3 `CommonCreationParameters`

Field	Meaning
`utf8PathToModels`	Path to the NVIGI model repository root (UTF-8).
`modelGUID`	GUID of the SD model under that repository.
`numThreads`	CPU threads available to the instance.
`vramBudgetMB`	Max VRAM the instance is allowed to consume. It is converted to GiB when `SDCreationParameters::maxVRAM` is null.

The plugin expects a model config under <utf8PathToModels>/configs/nvigi.plugin.sd.ggml.cuda/{GUID}/... declaring at least:

model.diffusion — the diffusion model file (.safetensors, .gguf, or .ckpt).
model.vae (optional) — separate VAE file. If absent, the diffusion file is expected to embed the VAE.
model.llm (optional) — separate text-encoder/LLM file (used by SD3 / Flux-style architectures).

Model cards can declare capabilities.reference_image_editing: true for edit models such as Flux2 Klein. For these models, the plugin routes kSDDataSlotInputImage through stable-diffusion.cpp’s ref_images conditioning path instead of the classic strength-based init_image path. Capability queries expose this as kSDModelFlagUsesReferenceImageEditing in the corresponding model’s modelFlags.

3.4 Backend parameters

If the host renders with D3D12 or Vulkan and wants the SD plugin to share its device, chain the matching struct:

nvigi::D3D12Parameters — device, queue (direct), queueCompute, queueCopy, plus createCommittedResourceCallback / destroyResourceCallback if the host owns memory allocation.
nvigi::VulkanParameters — instance, physicalDevice, device, queue, queueCompute, queueTransfer, plus optional allocateMemoryCallback / freeMemoryCallback.
nvigi::CudaParameters — when the CUDA backend is used outside of CIG and the host wants to pin the work to a specific device.

When a D3D12 device is provided to the CUDA backend, the plugin automatically enables CIG (CUDA-in-Graphics) and queries the active CUDA device from the CIG context.

3.5 Putting it together

nvigi::CommonCreationParameters common{};
common.utf8PathToModels = myPathToNVIGIModelRepository;
common.modelGUID        = "{CAD313FB-5545-4A8D-BF07-F5C260FAAA73}"; // example GUID
common.numThreads       = myNumCPUThreads;
common.vramBudgetMB     = myVRAMBudget;

nvigi::SDCreationParameters sdParams{};
sdParams.width          = 512;
sdParams.height         = 512;
sdParams.flashAttention = true;
sdParams.paramsBackend  = "*=cpu";     // keep model parameters in system RAM
sdParams.backend        = "vae=cpu";   // execute the VAE on CPU
sdParams.streamLayers   = true;        // requires maxVRAM or common.vramBudgetMB
sdParams.eagerLoad      = false;
sdParams.enableMmap     = true;        // default; set false to force explicit model reads

// Chain optional backend parameters (pick one that matches the active backend)
nvigi::D3D12Parameters d3d12{};
d3d12.device       = myDevice;
d3d12.queue        = myDirectQueue;
d3d12.queueCompute = myComputeQueue;
d3d12.queueCopy    = myCopyQueue;

sdParams.chain(common);
common.chain(d3d12);

nvigi::InferenceInstance* sdInstance{};
if (NVIGI_FAILED(res, isd->createInstance(sdParams, &sdInstance)))
{
    LOG("NVIGI call failed, code %d", res);
}

NOTE: Chaining order does not affect functionality but is shown above as sdParams -> common -> backendParams.

3.6 Optional: query capabilities and requirements

Query capabilities before creating an instance when the host chooses models dynamically or needs to preflight a requested model. The query uses the CommonCreationParameters chained to SDCreationParameters:

Set utf8PathToModels to the NVIGI model repository to enumerate its SD model cards.
Set modelGUID to a specific GUID to check that model; set it to nullptr to enumerate candidates.
Set vramBudgetMB to filter local candidates by their advertised memory budget, or to SIZE_MAX together with a null modelGUID to list all discovered model cards.

The capability filter uses CommonCreationParameters::vramBudgetMB. It does not account for SDCreationParameters::maxVRAM, placement strings, streaming, output resolution, or other instance settings; configure those separately before createInstance.

Use the typed helper rather than casting the output pointer manually, and check both the result and returned pointer:

nvigi::CommonCapabilitiesAndRequirements* caps{};
if (NVIGI_FAILED(res, nvigi::getCapsAndRequirements(isd, sdParams, &caps)) || !caps)
{
    LOG("Unable to query SD capabilities, code %d", res);
}
else
{
    for (size_t i = 0; i < caps->numSupportedModels; ++i)
    {
        // All per-model arrays use the same index.
        const nvigi::ModelFlags flags = caps->modelFlags ? caps->modelFlags[i] : 0;
        if (flags & nvigi::kModelFlagRequiresDownload)
            continue; // The model card is known, but its files are not available locally.

        LOG("MODEL: %s (%s), estimated VRAM: %llu MB",
            caps->supportedModelNames[i],
            caps->supportedModelGUIDs[i],
            caps->modelMemoryBudgetMB[i]);

        const bool supportsTxt2Img = (flags & nvigi::kSDModelFlagSupportsTextToImage) != 0;
        const bool supportsImg2Img = (flags & nvigi::kSDModelFlagSupportsImageToImage) != 0;
        const bool supportsMaterials = (flags & nvigi::kSDModelFlagSupportsMaterialMaps) != 0;
        const bool usesReferenceEditing =
            (flags & nvigi::kSDModelFlagUsesReferenceImageEditing) != 0;
        // Use these values to select a model and expose only compatible UI modes.
    }
}

numSupportedModels is the number of entries in the parallel supportedModelGUIDs, supportedModelNames, modelMemoryBudgetMB, and (when non-null) modelFlags arrays. After choosing an entry, copy its GUID to storage that remains valid through instance creation, then assign it to common.modelGUID before calling createInstance. modelMemoryBudgetMB is a model-card estimate for filtering, not a guarantee of peak memory use; actual use also depends on the workload and placement settings described in Memory placement and budget.

supportedBackends describes where this plugin binary runs: InferenceBackendLocations::eGPU for CUDA, D3D12, and Vulkan variants, or InferenceBackendLocations::eCPU for the CPU fallback. It does not select a particular graphics API or allocate a device.

The standard kModelFlagRequiresDownload flag reports local availability. The SD-specific flags in modelFlags describe each model’s supported workflow:

Flag	Meaning
`kSDModelFlagSupportsTextToImage`	The model can generate from a text prompt without an input image.
`kSDModelFlagSupportsImageToImage`	The model accepts an input image for image-to-image generation.
`kSDModelFlagSupportsMaterialMaps`	The model produces Marigold roughness and metallic outputs.
`kSDModelFlagUsesReferenceImageEditing`	An input image is treated as reference-image editing conditioning rather than classic strength-based img2img.

The capability result is owned by the plugin. Do not free it, and copy any fields you need to retain across a later capability query or plugin unload.

3.7 Material maps (Marigold)

The IGI fork adds Marigold intrinsic-image models that predict roughness and metallic material maps from an input image. Create a separate instance whose model config uses the canonical marigold_intrinsics mode and model key; instances reporting the kSDModelFlagSupportsMaterialMaps capability flag write two extra output slots alongside kSDDataSlotOutputImage:

Constant	Type	Direction
`kSDDataSlotOutputRoughnessImage`	`InferenceDataImage`	output
`kSDDataSlotOutputMetallicImage`	`InferenceDataImage`	output

SDRuntimeParameters::steps controls Marigold sampling steps per evaluation. Marigold always uses the DDIM trailing sample method with the discrete scheduler; sampleMethod and scheduler overrides are ignored. SDCreationParameters::width and height select the internal Marigold processing resolution and must be at least 64 and divisible by 64. All returned maps are resized to the original input dimensions.

4.0 RUNTIME PARAMETERS

SDRuntimeParameters are passed per-evaluation through InferenceExecutionContext::runtimeParameters and let you change the sampling behavior between calls without recreating the instance. If the structure is omitted, the plugin uses the same defaults shown below.

Field	Default	Meaning
`strength`	`0.6f`	Classic img2img denoising strength (0.0 = identical to input, 1.0 = ignore input). Ignored for txt2img and reference-image editing models.
`steps`	`4`	Sampling steps.
`cfgScale`	`1.0f`	Classifier-free guidance scale. This default suits distilled / Turbo / LCM models; conventional models may require a higher value such as `7.0f`.
`seed`	`-1`	Random seed. `-1` = use `time(nullptr)`.
`clipSkip`	`1`	CLIP layers to skip (1 = no skip).
`sampleMethod`	`eDefault`	`SDSampleMethod`: `eEuler`, `eEulerA`, `eHeun`, `eDPM2`, `eDPMPP2S_A`, `eDPMPP2M`, `eDPMPP2Mv2`, `eIPNDM`, `eIPNDM_V`, `eLCM`, `eDDIM_Trailing`, `eTCD`, `eRESMultistep`, `eRES2S`, `eERSDE`, `eEulerCFGPP`, `eEulerACFGPP`, `eEulerGE`. `eDefault` asks the model for its preferred sampler. Ignored by `marigold_intrinsics` models.
`scheduler`	`eDefault`	`SDScheduler`: `eDiscrete`, `eKarras`, `eExponential`, `eAYS`, `eGITS`, `eSGMUniform`, `eSimple`, `eSmoothstep`, `eKLOptimal`, `eLCM`, `eBongTangent`, `eLTX2`, `eLogitNormal`, `eFlux2`, `eFlux`, `eBeta`. `eDefault` asks the model and sampler for their preferred scheduler. Ignored by `marigold_intrinsics` models.
`batchCount`	`1`	Number of images to generate in one call.

5.0 INPUT AND OUTPUT SLOTS

The SD plugin uses the standard NVIGI slot system. Slot constants are declared in nvigi_sd.h:

Constant	Type	Direction	Required
`kSDDataSlotPrompt`	`InferenceDataText`	input	yes
`kSDDataSlotNegativePrompt`	`InferenceDataText`	input	optional
`kSDDataSlotInputImage`	`InferenceDataImage`	input	optional — its presence selects the mode
`kSDDataSlotOutputImage`	`InferenceDataImage`	output	yes (auto-allocated if not supplied)

Mode selection rule: if kSDDataSlotInputImage is provided AND has a non-empty CPU buffer, the call is img2img. Otherwise it is txt2img and the requested output size comes from SDCreationParameters::width × SDCreationParameters::height. The actual output can be rounded up as described below. For model cards with capabilities.reference_image_editing: true, that same input selects reference-image editing; the model receives the image as edit conditioning and strength is ignored.

Image data layout (both input and output) is interleaved 8-bit RGB or RGBA, top-to-bottom, with c == 3 or c == 4. When an RGBA input is provided in img2img mode, the alpha channel is preserved and re-attached to the RGB output.

5.1 Resolution alignment and automatic resizing

Stable Diffusion models operate on model-specific spatial multiples. Before txt2img or img2img generation, stable-diffusion.cpp computes:

spatial multiple = VAE scale factor * diffusion-model downsampling factor

It rounds the requested width and height up independently to the next multiple. For img2img, the input image is automatically resized to those aligned dimensions. The generated output therefore reports the aligned width and height; callers should read InferenceDataImage::w and h from the result rather than assume that they equal the requested or input dimensions.

The multiple depends on the loaded model, not the GPU backend. For example, a model with a spatial multiple of 16 maps 101x103 to 112x112, leaves 1024x1024 unchanged, and maps 1500x1500 to 1504x1504. A conventional UNet model with VAE scale factor 8 and diffusion downsampling factor 8 instead uses a multiple of 64.

Marigold uses SDCreationParameters::width and height as its internal processing resolution, while its returned maps retain the original input dimensions. This lets applications select a lower working resolution, such as 512x512 for a 1024x1024 input, without changing the output size.

NOTE: Resolution alignment is independent of VAE tiling. Alignment is automatic and required by the model. VAE tiling is fixed at instance creation.

6.0 SYNCHRONOUS EVALUATION

For synchronous evaluation the host installs a callback that is invoked once when the result is ready (state kInferenceExecutionStateDone).

struct SDOutput { std::vector<uint8_t> data; int w, h, c; bool done = false; };

nvigi::InferenceExecutionState onSDComplete(
    const nvigi::InferenceExecutionContext* ctx,
    nvigi::InferenceExecutionState state,
    void* userData)
{
    auto* out = static_cast<SDOutput*>(userData);
    if (state == nvigi::kInferenceExecutionStateDone && ctx->outputs)
    {
        const nvigi::InferenceDataImage* img{};
        if (ctx->outputs->findAndValidateSlot(nvigi::kSDDataSlotOutputImage, &img) && img)
        {
            auto cpu = nvigi::castTo<nvigi::CpuData>(img->bytes);
            if (cpu && cpu->buffer)
            {
                size_t sz = size_t(img->w) * img->h * img->c;
                out->data.assign(static_cast<const uint8_t*>(cpu->buffer),
                                 static_cast<const uint8_t*>(cpu->buffer) + sz);
                out->w = img->w; out->h = img->h; out->c = img->c;
                out->done = true;
            }
        }
    }
    return state;
}

IMPORTANT: The slot pointers passed to the callback are only valid for the duration of the callback. Copy any data you need to keep.

Build the execution context and call evaluate:

// Inputs
std::string prompt = "A majestic lion in a savanna at sunset, photorealistic";
std::string negPrompt = "blurry, low quality";

nvigi::CpuData promptCpu(prompt.size() + 1, prompt.c_str());
nvigi::InferenceDataText promptData(promptCpu);

nvigi::CpuData negCpu(negPrompt.size() + 1, negPrompt.c_str());
nvigi::InferenceDataText negData(negCpu);

std::vector<nvigi::InferenceDataSlot> inSlots = {
    { nvigi::kSDDataSlotPrompt,         promptData },
    { nvigi::kSDDataSlotNegativePrompt, negData    },
    // For img2img also push: { nvigi::kSDDataSlotInputImage, inputImageData }
};
nvigi::InferenceDataSlotArray inputs{ inSlots.size(), inSlots.data() };

// Runtime params
nvigi::SDRuntimeParameters runtime{};
runtime.steps        = 20;
runtime.cfgScale     = 7.0f;
runtime.seed         = 42;
runtime.sampleMethod = nvigi::SDSampleMethod::eEulerA;
runtime.scheduler    = nvigi::SDScheduler::eKarras;

// Execution context
SDOutput out{};
nvigi::InferenceExecutionContext exec{};
exec.instance          = sdInstance;
exec.inputs            = &inputs;
exec.runtimeParameters = runtime;
exec.callback          = onSDComplete;
exec.callbackUserData  = &out;

if (NVIGI_FAILED(res, sdInstance->evaluate(&exec)))
{
    LOG("SD evaluate failed, code %d", res);
}
// `out.done == true` once the callback fires.

NOTE: Synchronous evaluation blocks the calling thread for the full diffusion run. A txt2img generation at 512×512 with 20 Euler-A steps typically takes a few seconds on a modern GPU.

6.1 img2img example

For img2img, add an InferenceDataImage for kSDDataSlotInputImage. The requested output dimensions follow the input image, not the creation-time width/height. If required by the loaded model, stable-diffusion.cpp resizes the input to aligned dimensions and returns that aligned size; see 5.1 Resolution alignment and automatic resizing.

// inputImageBytes is a CPU buffer with W*H*C bytes (C is 3 or 4)
nvigi::CpuData inputCpu(inputImageBytes.size(), inputImageBytes.data());
nvigi::InferenceDataImage inputImg{};
inputImg.bytes = inputCpu;
inputImg.w     = inputW;
inputImg.h     = inputH;
inputImg.c     = inputChannels;

inSlots.insert(inSlots.begin(), { nvigi::kSDDataSlotInputImage, inputImg });

// Use a lower strength to stay closer to the input
runtime.strength = 0.6f;

7.0 ASYNCHRONOUS EVALUATION (POLLED)

evaluateAsync returns immediately and runs the diffusion on a worker thread. Progress and completion are observed by polling the IPolledInferenceInterface.

IMPORTANT: The InferenceExecutionContext and every slot, parameter, and string buffer it points at must remain alive until the async job completes. Wrap them in a heap-allocated context (e.g. std::shared_ptr) — see the AsyncContext struct in sd.hpp.

auto ctx = std::make_shared<AsyncContext>(); // owns all string/buffer copies
// ... fill ctx->inputs, ctx->runtime, ctx->exec_ctx as in the sync example ...

if (NVIGI_FAILED(res, sdInstance->evaluateAsync(&ctx->exec_ctx)))
{
    LOG("SD evaluateAsync failed, code %d", res);
}

// Poll every frame from the render thread (non-blocking: wait = false)
nvigi::InferenceExecutionState state = nvigi::kInferenceExecutionStateInvalid;
auto r = iPolled->getResults(&ctx->exec_ctx, /*wait*/ false, &state);

if (r == nvigi::kResultNotReady)
{
    // Still running — try again next frame.
}
else if (r == nvigi::kResultOk)
{
    if (state == nvigi::kInferenceExecutionStateDone)
    {
        // Pull the image out of ctx->exec_ctx.outputs the same way as in the sync callback.
    }
    iPolled->releaseResults(&ctx->exec_ctx, state);
}

7.1 Cancelling async evaluation

When using evaluateAsync, call cancelAsyncEvaluation to stop an ongoing image generation early. This is useful when the user changes the prompt, leaves the current context, or starts another request. Cancellation is cooperative and stops generation as soon as the active backend work can be interrupted.

nvigi::Result result = sdInstance->cancelAsyncEvaluation(&ctx->exec_ctx);
if (result == nvigi::kResultCanceled)
{
    LOG("SD generation cancelled");
}
else if (result == nvigi::kResultNoImplementation)
{
    LOG("No async SD evaluation is currently running");
}
else if (NVIGI_FAILED(result))
{
    LOG("Failed to cancel SD evaluation, code %d", result);
}

cancelAsyncEvaluation applies only to work started with evaluateAsync. When cancellation is observed, the completion callback or polling state reports kInferenceExecutionStateCancel. Keep the execution context and its referenced data alive until the operation reaches a terminal state. destroyInstance cancels any pending async job automatically before tearing the context down.

8.0 BACKEND-SPECIFIC NOTES

8.1 CUDA / CIG

When you provide a D3D12Parameters with a non-null queue to the CUDA backend, the plugin enables CIG so that CUDA work shares the same physical GPU context as your D3D12 renderer. The active CUDA device is queried from the CIG context — there is no need to set CudaParameters::device.
The plugin queries the per-instance CUDA streams and applies the global GPU inference scheduling mode (Balance / PrioritizeCompute / PrioritizeGraphics) on every evaluate if IHWICuda v2+ is available.
An NVIDIA R580 driver or newer is strongly recommended to avoid a potential memory leak when CIG is used and the application deletes D3D12 command queues mid-application. Older drivers also log a warning and fall back to default scheduling priorities.

8.2 D3D12

The D3D12 device must support shader model 6.6 or higher. If queues are not provided, the plugin generates its own compute and direct/copy queues.
The plugin auto-detects the adapter vendor through ISystem. On NVIDIA adapters it loads nvigi.plugin.hwi.d3d12 and applies inference scheduling per command list. Without hwi.d3d12 (or with a version below 4) you’ll see a WARN_ONCE and performance may be suboptimal.
ReBAR is enabled by default. Pass D3D12ParametersFlags::eDisableReBAR (v3+) to opt out. For optimal performance the host must also ship the MS Agility SDK — see 13.1 D3D12.
Requires NVIDIA R580 driver or newer to be loaded at runtime.

IMPORTANT: Do NOT chain the same parameters to multiple parameter chains. The recommended approach is to make a copy per chain. For example, creating an ASR and SD instance with shared d3d12Params can result in re-chaining the input parameters the wrong way which then results in failed instance creation.

8.3 Vulkan

Allocation can be delegated to the host via allocateMemoryCallback / freeMemoryCallback in VulkanParameters. If left null, the plugin uses the Vulkan default allocator.
For full Vulkan device requirements (extensions, features, instance API version) see 13.2 Vulkan.

8.4 CPU fallback

If neither CUDA, D3D12, nor Vulkan was selected at compile time, the plugin reports InferenceBackendLocations::eCPU. CPU inference is functional but dramatically slower and is intended only for development or platforms without a usable GPU backend.

9.0 MEMORY PLACEMENT AND BUDGET

Memory requirements depend on the model architecture and quantization, image resolution, batch size, selected backend, and other runtime settings. Measure the target workload instead of selecting placement settings from a nominal GPU memory capacity.

Goal	Configuration
Use the default device placement	Leave `backend` and `paramsBackend` unset. Flash attention is enabled by default; disable it if unsupported.
Apply a device-memory budget	Set `maxVRAM`, or use `CommonCreationParameters::vramBudgetMB` and let the plugin derive `maxVRAM`.
Reduce resident device parameters	Set `paramsBackend = "*=cpu"` to store model parameters in system memory.
Move selected execution stages to the CPU	Set `backend` explicitly, for example `"vae=cpu"`.
Stream model layers within a budget	Set `streamLayers = true` together with `maxVRAM` or `vramBudgetMB`.
Preload assigned parameter storage	Set `eagerLoad = true`; this changes load timing, not the selected placement.
Disable file-backed model mappings	Set `enableMmap = false`; explicit reads can increase persistent host-buffer usage.

10.0 MODERN C++ WRAPPER (RECOMMENDED)

For most application code, prefer the high-level wrapper in sd.hpp. It hides interface lifetime management, slot construction, and async polling behind a small API:

#include "source/samples/shared/cxx_wrappers/sd/sd.hpp"

using namespace nvigi::sd;

ModelConfig cfg{};
cfg.guid       = "{CAD313FB-5545-4A8D-BF07-F5C260FAAA73}";
cfg.model_path = pathToModels;
cfg.set_resolution(1024, 1024)
   .set_threads(numThreads)
   .set_vram(vramBudgetMB)
   .set_flash_attention(true)
   .set_params_backend_spec("*=cpu")
   .set_stream_layers(true);

// Optional: explicitly reduce peak VAE memory use for this instance.
nvigi::SDVaeTilingParameters tiling{};
tiling.enabled       = true;
tiling.tileSizeX     = 16;
tiling.tileSizeY     = 16;
tiling.targetOverlap = 0.5f;
cfg.set_vae_tiling(tiling);

auto instanceResult = Instance::create(
    cfg, d3d12Config, vulkanConfig,
    nvigiLoadInterface, nvigiUnloadInterface,
    Backend::eCUDA);

if (!instanceResult) { LOG("%s", instanceResult.error().what().c_str()); return; }
auto instance = std::move(instanceResult.value());

RuntimeConfig runtime{};
runtime.set_steps(20).set_cfg_scale(7.0f).set_seed(42);

// Synchronous txt2img
auto txt = instance->generate("a cat astronaut on Mars", runtime);

// Synchronous img2img
Image input = ...;
auto img = instance->generate(input, "make it cartoonish", runtime);

// Asynchronous (polling)
auto async = instance->generate_async("a cat astronaut on Mars", runtime).value();
while (!async.is_complete())
{
    if (auto out = async.try_get_result()) { /* use *out */ break; }
    if (async.is_failed()) break;
    std::this_thread::sleep_for(std::chrono::milliseconds(16));
}

11.0 DESTROY INSTANCE(S)

if (NVIGI_FAILED(res, isd->destroyInstance(sdInstance)))
{
    LOG("NVIGI call failed, code %d", res);
}

destroyInstance waits for any in-flight async job to complete (or signals it to cancel) before freeing GPU resources.

12.0 UNLOAD INTERFACE(S)

if (NVIGI_FAILED(res, nvigiUnloadInterface(nvigi::plugin::sd::ggml::cuda::kId, isd)))
{
    LOG("NVIGI call failed, code %d", res);
}
if (iPolled)
{
    nvigiUnloadInterface(nvigi::plugin::sd::ggml::cuda::kId, iPolled);
}

13.0 APPENDIX

13.1 D3D12

When using the D3D12 backend, the host application must create a device which supports shader model 6.6 or higher. To ensure proper support across various Windows OS versions, the recommended approach is to include Microsoft Agility SDK version 1.600.0 or newer with your executable by adding the following code:

extern "C" __declspec(dllexport) UINT         D3D12SDKVersion = 610; // Change this as needed to reflect the version you want to use
extern "C" __declspec(dllexport) const char * D3D12SDKPath    = ".\\D3D12\\";

NOTE: D3D12 folder must be created next to the executable and it must contain D3D12Core.dll which is provided with the Agility SDK.

The additional benefit of including the latest Agility SDK is the performance enhancement which comes with the introduction of the new heap type D3D12_HEAP_TYPE_GPU_UPLOAD. This new feature enables simultaneous CPU and GPU access to VRAM via the Resizable BAR (ReBAR) mechanism — introduced to the DirectX 12 API through the Direct3D Agility SDK and corresponding Windows updates. This allows for more efficient data transfers, reducing the need for CPU-to-GPU copy operations and potentially improving performance in certain scenarios. For more details please visit https://devblogs.microsoft.com/directx/preview-agility-sdk-1-710-0/

Feature	First Supported Windows OS	First Supported Agility SDK Version
GPU UPLOAD HEAP (ReBAR)	Windows 11 Insider Preview Build 26080 or later	1.613.0

IMPORTANT: Please note that on some systems ReBAR must be explicitly enabled in the BIOS.

In addition to the above, it is also required to distribute dxcompiler.dll with your application.

13.2 Vulkan

NOTE: This section is relevant only if the host application is providing nvigi::VulkanParameters to the NVIGI SD plugin.

Here are the Vulkan requirements:

VkInstance must be created with the API 1.3.0 or higher.
VkDevice must be created with VkPhysicalDeviceFeatures2, VkPhysicalDeviceVulkan11Features and VkPhysicalDeviceVulkan12Features chained to the VkDeviceCreateInfo.
The following extensions must be enabled if the physical device supports them:

"VK_EXT_pipeline_robustness",
"VK_KHR_maintenance4",
"VK_EXT_subgroup_size_control",
"VK_KHR_16bit_storage",
"VK_KHR_shader_float16_int8",
"VK_KHR_cooperative_matrix",
"VK_NV_cooperative_matrix2"

NOTE: If certain extensions are not available the appropriate fallbacks will be used if possible.

13.3 Memory Tracking

NVIGI provides callback mechanisms to track GPU resource allocation and freeing for each backend. The SD plugin honors these callbacks identically to the GPT plugin — see GPT Programming Guide §A.2 MEMORY TRACKING for full Vulkan / D3D12 / CUDA callback examples. The relevant fields on the SD parameter structs are:

VulkanParameters::allocateMemoryCallback / freeMemoryCallback
D3D12Parameters::createCommittedResourceCallback / destroyResourceCallback
CudaParameters allocation callbacks (when applicable)

14.0 EXCEPTIONS AND ERRORS

evaluate / evaluateAsync may fail for several reasons. The most common return codes are:

Result	Cause / Action
`kResultOk`	Success.
`kResultCanceled`	The asynchronous evaluation was cancelled.
`kResultInvalidParameter`	Missing prompt slot, null instance, or missing required `CommonCreationParameters`. Check the slot signature and chained structs.
`kResultInvalidState`	The instance is in an error state (e.g. CUDA context construction failed, or `generate_image` returned `false`). Destroy and recreate.
`kResultTimedOut`	An asynchronous worker did not stop during cancellation or cleanup. Treat the instance as unusable and destroy it.
`kResultNoImplementation`	`cancelAsyncEvaluation` called when no async job is running.
`kResultNotReady`	Returned by `getResults(wait=false)` while the worker is still running — not an error, just poll again.
`kResultMissingInterface`	A required HW interface (`hwi.cuda`, `hwi.d3d12`) could not be loaded at register time. Check that the matching plugin DLL is deployed.
`kResultDriverOutOfDate`	Logged as a warning when applying CIG/SCG scheduling on an older driver. The job still runs, but scheduling priorities are not honored.

IMPORTANT: The host app cannot assume the inference callback is invoked on the thread that called evaluate. Async results are produced on a worker thread, and even synchronous callbacks may be invoked on the plugin’s internal thread.