Explainability overview
Explainable artificial intelligence (XAI) provides evidence that helps a person understand how a model produced a prediction. Kumo Engine’s approach to XAI is built around NIST’s four principles: provide an explanation, make it meaningful to its audience, reflect the model’s calculation accurately, and require the system to operate only within intended conditions and with sufficient confidence in its output. For a graph neural network (GNN), that can mean showing the records used, measuring sensitivity to their features, or tracing how information reached the score. Use this section to choose the evidence your reviewer needs and a compatible model that can produce it.
Which records shaped this prediction?
In a credit risk model, an account is flagged for review. If the model is tree-based, feature attributions can show how inputs such as a six-month late-payment count contributed to its output score. Training that tree on related account and statement tables usually requires preparing one feature row per account. Summaries can leave out individual statement values, timing, or relationships unless the features explicitly retain them.
A GNN can learn from the statements and their relationships to the account directly. This preserves relational detail that an aggregate count alone does not contain, giving you another architecture to evaluate for predictive accuracy. With a compatible explanation method, you can also follow how a particular statement’s features reached the account’s score through the model. The review question becomes more specific: which statements shaped this prediction, and through which calculations?
Kumo Engine lets practitioners train and compare preconfigured heterogeneous GNN architectures from scratch over related tables. Rows become nodes, relationships connect them, and the model combines information from a sampled neighborhood to make a prediction. The model plan controls the architecture and search space. Engine’s data loader also supports custom PyTorch models; an explanation method must support the actual model you train.
What XAI exists today
These tools answer different questions about a prediction. Their explanation values are not interchangeable.
Know which Engine output you are reading
The built-in training workflow can produce column gradient rankings, column cohorts, and sampled graph context for eligible tasks and models. The Model Explainability guide explains eligibility, artifact retrieval, and preservation. These fit-time artifacts concern selected test examples; they do not automatically explain every later scoring request.
Relevant-walk attribution applies the approach in Schnake et al. to supported model calculations. The Engine implementation and evidence captured while producing a prediction are being developed separately from the released training workflow and require a compatible model and a pinned build. Walk attribution has a narrower CPU, activation, aggregation, and task profile than the general GNN training interface. It does not apply automatically to a custom PyTorch model, a graph transformer, or every built-in model configuration. Treat a missing or unsupported explanation as a reason to inspect compatibility and logs, rather than as evidence that a record had no influence.
What is the bar for XAI?
XAI supports model debugging and review wherever people need to understand predictions. In high-stakes decisions and regulated environments, transparency and accountability also make it important to retain evidence that a domain reviewer can inspect. Establish what that reviewer needs before choosing an architecture. For the flagged account, that might mean identifying the statements available at the prediction cutoff, inspecting their values, and tracing their signed contributions to a named model output. Select a model that can produce that evidence, then evaluate its predictions and explanations on known cases. If the explanation cannot answer the review question, change the evidence method or architecture before relying on it.
Kumo Engine’s XAI approach uses NIST’s four principles as its design and evaluation framework. The following questions connect each principle to evidence you can inspect:
Kumo’s GNN researchers evaluate candidate methods with controlled models, numerical accounting checks, and case studies that can expose misleading explanations. The examples distinguish the model’s output from the explanation and report where the evidence weakens. This is a practical evaluation framework, not a claim that a method has been certified against NIST’s principles.
Choose your next step
To start with the released artifacts, use Model Explainability.
Understand the method
Interpret a prediction
Relevant-walk attribution and prediction-time evidence are experimentally available through NVIDIA/kumo-engine#429. Use the PR’s source branch to try these capabilities, and visit the Kumo Engine repository for the project overview.