> This page is for Kumo Engine.

> For clean Markdown of any page, append .md to the page URL.
> For a complete documentation index, see https://docs.nvidia.com/sdgm/llms.txt.

# Explainability overview

> Choose the evidence you need from a prediction and a model architecture that can provide it.

Explainable artificial intelligence (XAI) provides evidence that helps a person understand how a model produced a prediction.
Kumo Engine's approach to XAI is built around [NIST's four principles](https://nvlpubs.nist.gov/nistpubs/ir/2021/NIST.IR.8312.pdf): provide an explanation, make it meaningful to its audience, reflect the model's calculation accurately, and require the system to operate only within intended conditions and with sufficient confidence in its output.
For a graph neural network (GNN), that can mean showing the records used, measuring sensitivity to their features, or tracing how information reached the score.
Use this section to choose the evidence your reviewer needs and a compatible model that can produce it.

## Which records shaped this prediction?

In a credit risk model, an account is flagged for review.
If the model is tree-based, feature attributions can show how inputs such as a six-month late-payment count contributed to its output score.
Training that tree on related account and statement tables usually requires preparing one feature row per account.
Summaries can leave out individual statement values, timing, or relationships unless the features explicitly retain them.

A GNN can learn from the statements and their relationships to the account directly.
This preserves relational detail that an aggregate count alone does not contain, giving you another architecture to evaluate for predictive accuracy.
With a compatible explanation method, you can also follow how a particular statement's features reached the account's score through the model.
The review question becomes more specific: which statements shaped this prediction, and through which calculations?

Kumo Engine lets practitioners train and compare preconfigured heterogeneous GNN architectures from scratch over related tables.
Rows become nodes, relationships connect them, and the model combines information from a sampled neighborhood to make a prediction.
The [model plan](/sdgm/kumo-engine/user-code-guide/train-a-model/model-plan-guide) controls the architecture and search space.
Engine's data loader also supports custom PyTorch models; an explanation method must support the actual model you train.

## What XAI exists today

These tools answer different questions about a prediction.
Their explanation values are not interchangeable.

| Approach                                                                                                                                                                      | Output                                                                 | What it means for the credit example                                                                                                                        |
| ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ---------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------- |
| [XGBoost prediction contributions](https://xgboost.readthedocs.io/en/stable/prediction.html)                                                                                  | Feature contributions and a bias term for a selected model output      | Inspect the contribution assigned to the late-payment count supplied to the tree. Recovering the original statements requires the feature's source lineage. |
| [DataRobot individual prediction explanations](https://docs.datarobot.com/en/docs/workbench/nxt-workbench/experiments/experiment-insights/shap-predex.html#expanded-row-view) | Feature values and SHAP explanations for a selected prediction         | Review which features moved the prediction relative to its explanation baseline.                                                                            |
| A graph explanation, such as [PyG's explanation framework](https://pytorch-geometric.readthedocs.io/en/latest/modules/explain.html)                                           | Node, edge, or feature masks, depending on the algorithm               | Inspect selected parts of a graph. A mask alone does not describe the signed contribution of every computational route.                                     |
| Relevant-walk attribution (experimental, not yet released), based on [Schnake et al.](https://arxiv.org/abs/2006.03589v3)                                                     | Signed relevance assigned to supported routes through a compatible GNN | Follow information from a monthly statement through message passing to the selected account output.                                                         |

## Know which Engine output you are reading

The built-in training workflow can produce column gradient rankings, column cohorts, and sampled graph context for eligible tasks and models.
The [Model Explainability guide](/sdgm/kumo-engine/user-code-guide/train-a-model/model-explainability) explains eligibility, artifact retrieval, and preservation.
These fit-time artifacts concern selected test examples; they do not automatically explain every later scoring request.

Relevant-walk attribution applies the approach in Schnake et al. to supported model calculations.
The Engine implementation and evidence captured while producing a prediction are being developed separately from the released training workflow and require a compatible model and a pinned build.
Walk attribution has a narrower CPU, activation, aggregation, and task profile than the general GNN training interface.
It does not apply automatically to a custom PyTorch model, a graph transformer, or every built-in model configuration.
Treat a missing or unsupported explanation as a reason to inspect compatibility and logs, rather than as evidence that a record had no influence.

## What is the bar for XAI?

XAI supports model debugging and review wherever people need to understand predictions.
In high-stakes decisions and regulated environments, transparency and accountability also make it important to retain evidence that a domain reviewer can inspect.
Establish what that reviewer needs before choosing an architecture.
For the flagged account, that might mean identifying the statements available at the prediction cutoff, inspecting their values, and tracing their signed contributions to a named model output.
Select a model that can produce that evidence, then evaluate its predictions and explanations on known cases.
If the explanation cannot answer the review question, change the evidence method or architecture before relying on it.

Kumo Engine's XAI approach uses NIST's four principles as its design and evaluation framework.
The following questions connect each principle to evidence you can inspect:

| Principle            | Ask the reviewer                                 | Evidence to examine                                                                                                                                               |
| -------------------- | ------------------------------------------------ | ----------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| Explanation          | What evidence accompanies this score?            | The selected output, source records, model version, and explanation method.                                                                                       |
| Meaningful           | Can the intended reviewer understand it?         | A case walkthrough connecting statement dates and values to the displayed calculation.                                                                            |
| Explanation accuracy | How faithfully does it describe the calculation? | Method-specific checks, controlled input changes, and agreement with known model behavior. Contributions adding up is only an accounting check.                   |
| Knowledge limits     | When should we rely on the result?               | Task and architecture compatibility, sampling limits, predictive quality, whether confidence is sufficient or unavailable, and cases where the explanation fails. |

Kumo's GNN researchers evaluate candidate methods with controlled models, numerical accounting checks, and case studies that can expose misleading explanations.
The examples distinguish the model's output from the explanation and report where the evidence weakens.
This is a practical evaluation framework, not a claim that a method has been certified against NIST's principles.

## Choose your next step

To start with the released artifacts, use [Model Explainability](/sdgm/kumo-engine/user-code-guide/train-a-model/model-explainability).

### Understand the method

* [Relevance scores and graph walks](/sdgm/kumo-engine/resources/explainability/relevance-scores-and-graph-walks)
* [Gradients and local subgraphs](/sdgm/kumo-engine/resources/explainability/gradients-and-local-subgraphs)

### Interpret a prediction

* [Review a credit default prediction](/sdgm/kumo-engine/resources/explainability/review-a-credit-default-prediction)
* [Explain a product recommendation](/sdgm/kumo-engine/resources/explainability/explain-a-product-recommendation)

Relevant-walk attribution and prediction-time evidence are experimentally available through [NVIDIA/kumo-engine#429](https://github.com/NVIDIA/kumo-engine/pull/429).
Use the PR's source branch to try these capabilities, and visit the [Kumo Engine repository](https://github.com/NVIDIA/kumo-engine/tree/main) for the project overview.