> For clean Markdown of any page, append .md to the page URL.
> For a complete documentation index, see https://docs.nvidia.com/sdgm/llms.txt.
> For AI client integration (Claude Code, Cursor, etc.), connect to the MCP server at https://docs.nvidia.com/sdgm/_mcp/server.

# Diagnose and Improve Training

> Diagnose model quality and training failures, choose a controlled next experiment, and retain evidence for support

Identify a model quality or execution problem, choose one change, and check whether it helps on unchanged validation data. These examples use Kumo Engine in Python and the completed `fit_result` from [Training](/kumo-engine/1.0.0/user-code-guide/train-a-model/training#train-a-model). Detailed history requires the original scratch files. If training failed, start with [Training Did Not Complete](#training-did-not-complete).

## Start with the Symptom

| Symptom                                             | Next step                                                                                                                                  |
| --------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------ |
| Weak validation quality                             | [Check the evaluation](#check-the-evaluation-before-tuning), then [check features and graph coverage](#check-features-and-graph-coverage). |
| Good AUROC but few useful alerts                    | [Check the decision threshold](#check-the-decision-threshold).                                                                             |
| Unexpectedly high quality                           | Check prediction-time availability and split leakage [before tuning](#check-the-evaluation-before-tuning).                                 |
| Training and validation curves diverge or stay flat | [Interpret the curves](#interpret-the-curves) before changing settings.                                                                    |
| A business segment performs poorly                  | [Inspect holdout rows](#read-holdout-predictions) and segment counts.                                                                      |
| Failed or stalled training                          | [Identify the failing stage](#training-did-not-complete).                                                                                  |
| Missing metrics or files                            | [Check available outputs](#know-which-evidence-you-have).                                                                                  |
| Results are hard to explain or reproduce            | [Inspect the trained configuration](#inspect-the-trained-configuration) or [contact support](#prepare-a-support-request).                  |

## Check the Evaluation Before Tuning

* Verify the target, population, time window, and label counts in each split. For temporal predictions, features must be available at prediction time. Unexpectedly high scores warrant checking duplicates and target-derived columns.
* Compare against a simple baseline on the same rows. For binary classification, check positive-class prevalence and precision/recall at the intended threshold. For ranking, compare the same cutoff; for regression, inspect errors in the target's units.
* Use validation to choose settings and test data for the final assessment. If test errors guide further tuning, use a fresh untouched set for the final estimate.

### Check Features and Graph Coverage

Before increasing model capacity, check whether useful information reaches the model:

* Compare label counts, timestamp ranges, and feature availability across training and validation. Inspect fitted encoders for missing or excluded features.
* Measure foreign-key match rates and related-record counts **at each prediction time**. Report the fraction of entities with no eligible history. A relationship that matches in the full source can still provide no usable history after temporal filtering.
* Compare eligible history with [sampled graph context](/kumo-engine/1.0.0/user-code-guide/train-a-model/model-explainability), when available. Increase neighbor sampling only when useful records are being omitted; it cannot repair missing relationships.

As a control, compare with a simple model using prediction-time-available aggregates on the same validation rows. If that control has signal and Kumo does not, investigate feature encoding, relationships, and sampling. If neither has signal, revisit the target and population before increasing capacity.

### Check the Decision Threshold

For binary classification, compare average precision (AP) with positive-class prevalence on the same population. A high AUROC can coexist with poor precision at your alert budget. Measure precision, recall, and alert count at the intended threshold, or within the highest-scoring rows your team can review.

Choose the threshold or review budget using complete, aligned **validation** labels and positive-class scores, then freeze it before test evaluation. `holdout_df()` contains test rows and is not a threshold-tuning dataset. If no rows trigger an alert, precision is undefined, not perfect. Ranking quality also does not establish calibrated probabilities; compare predicted risk with observed positive rates before using scores as risk estimates.

## Know Which Evidence You Have

| Output                                                    | Location      |
| --------------------------------------------------------- | ------------- |
| Final metrics, resolved plan, and lineage                 | Model bundle  |
| Epoch history and experiment results                      | Local scratch |
| Holdout predictions and explanation files, when supported | Local scratch |

Publishing a model does not retain the last two rows. Preserve needed files before scratch is removed; a notebook shutdown or machine replacement can delete them.

## Read Retained Model Metrics

In the training process:

```python
print(fit_result.metrics)
```

In a later process, read `metrics.json` from the complete retained bundle:

```python
import json
from pathlib import Path

bundle = Path("/path/to/retained-models/model-v1")
metrics = json.loads((bundle / "metrics.json").read_text())
print(metrics)
```

| Prefix                           | Evaluation                                             |
| -------------------------------- | ------------------------------------------------------ |
| `train_`                         | Training metrics                                       |
| `val_`                           | Winning search experiment's validation metrics         |
| `test_`, excluding `test_refit_` | Original fitted model on the test split                |
| `test_refit_`                    | Train-and-validation refit on the test split, when run |

A full-data refit trains on the test split and has no independent test evaluation. Missing metrics are not zeros; check configured metrics, evaluated splits, and warnings about undefined values.

## Read Holdout Predictions

These are test predictions from the train-and-validation refit when run, otherwise from the original fit. Use them to describe errors, not to repeatedly tune settings or thresholds.

```python
if fit_result.holdout_path is None:
    raise FileNotFoundError("This fit has no saved holdout predictions.")
print("Truncated:", fit_result.holdout_truncated)
holdout = fit_result.holdout_df()
print(holdout.columns.tolist())
print(holdout.head())
```

`holdout_df()` reads the entire saved frame into memory and fails if scratch was deleted. The saved frame is capped at 1 GiB in memory. `holdout_truncated=True` means rows were omitted; `None` means completeness is unknown. Do not report truncated-frame metrics as full-holdout metrics. Link prediction does not produce this artifact.

For segment analysis, join attributes on the complete example key, including prediction time where applicable. Verify that the join preserves row count, and report segment size and class counts alongside scores.

To retain the rows, choose a new path on durable storage:

```python
from pathlib import Path

output = Path("/path/to/retained-results/run-001/holdout.parquet")
output.parent.mkdir(parents=True, exist_ok=True)
if output.exists():
    raise FileExistsError(output)
holdout.to_parquet(output, index=False)
```

## Select the Completed Fit

Continue in the process that produced `fit_result`. Its bundle identifies the exact fit run, even when scratch contains several trainers and repeated fits.

```python
from pathlib import Path
import json

run_dir = Path(fit_result.bundle_dir).parent
automl_dirs = sorted({
    path.parent
    for path in run_dir.rglob("auto_train_progress.json")
    if path.is_file()
})

if len(automl_dirs) != 1:
    raise RuntimeError(
        f"Expected one AutoML directory for this fit, found {len(automl_dirs)}. "
        f"Inspect {run_dir} before selecting a run."
    )

automl_dir = automl_dirs[0]
selection = json.loads(
    (automl_dir / "best_trainer.json").read_text(encoding="utf-8")
)
selected_trial_name = Path(selection["best_trainer"]).name
selected_trial_dir = automl_dir / selected_trial_name

if (
    not selected_trial_name.startswith("version_")
    or not selected_trial_dir.is_dir()
):
    raise RuntimeError("The selected experiment directory is unavailable.")

print("Fit run:", run_dir)
print("AutoML directory:", automl_dir)
print("Selected experiment:", selected_trial_dir)
```

`selected_trial_dir` identifies the original winning search experiment. Its curves describe that experiment, not a subsequent refit. Use the bundle artifacts and metric prefixes to distinguish the final model from search history.

Record these paths before the Python objects disappear. If you inspect retained scratch in a later process, set `automl_dir` to the exact recorded directory and resolve `selected_trial_dir` from its `best_trainer.json`. Do not select a run only because it was modified most recently.

## Inspect the Trained Configuration

Read the selected fit's artifacts, not `trainer.model_plan`, which controls the next fit and may have changed:

```python
for role in ("resolved_model_plan", "trainer_config", "encoders", "target_encoder"):
    path = Path(fit_result.artifact_paths[role])
    print(role, path)
    print(path.read_text()[:1500])
```

Open the files for complete values. `resolved_model_plan` can retain candidate lists; `trainer_config` records the concrete settings selected for the trainer. Check the tuning metric, evaluation step limits, refit settings, and fitted feature encoders. Change future settings through the [Model Plan Guide](/kumo-engine/1.0.0/user-code-guide/train-a-model/model-plan-guide), not by editing artifact files.

### Read Lineage in a Later Process

Use a complete retained bundle from a trusted publisher, such as the copy created in [Publish and Retrieve Models](/kumo-engine/1.0.0/user-code-guide/train-a-model/publish-and-retrieve-models).
The publication manifest maps artifact roles to paths relative to that bundle:

```python
from pathlib import Path
import json

bundle = Path("/path/to/retained-models/model-v1")
manifest = json.loads(
    (bundle / "publication-manifest.json").read_text(encoding="utf-8")
)
artifacts = {
    entry["role"]: bundle / entry["relative_path"]
    for entry in manifest["artifacts"]
}
lineage = json.loads(artifacts["training_spec"].read_text(encoding="utf-8"))
print("Predictive query:", lineage["predictive_query"])
print("Training snapshot:", lineage["snapshot_id"])
print("Generation settings:", lineage["training_table_generation"])
print("Resolved plan:", artifacts["resolved_model_plan"])
print("Selected configuration:", artifacts["trainer_config"])
print("Feature encoders:", artifacts["encoders"])
print("Target encoder:", artifacts["target_encoder"])
```

The snapshot identifier records provenance; the bundle does not contain the graph data. Use [Publish and Retrieve Models](/kumo-engine/1.0.0/user-code-guide/train-a-model/publish-and-retrieve-models) for validated loading.

## Load Epoch and Experiment Metrics

Continue with `selected_trial_dir` from [Select the Completed Fit](#select-the-completed-fit). This loads one experiment's epoch history and final validation results; install pandas and Matplotlib if needed.

```python
import pandas as pd

progress = json.loads((selected_trial_dir / "train_progress.json").read_text())
rows = []
for epoch, values in (progress.get("metrics") or {}).items():
    values = values or {}
    rows.append({
        "epoch": int(epoch),
        **(values.get("train_metrics") or {}),
        **(values.get("validation_metrics") or {}),
    })
if not rows:
    raise ValueError("No recorded epochs. Check this experiment's training logs.")
history = pd.DataFrame(rows).sort_values("epoch").set_index("epoch")
print(history)
print(pd.read_csv(selected_trial_dir / "val_result.csv"))
```

To compare experiments, repeat for another recorded `version_N` directory. Compare the tuning metric in that experiment's `model_plan.json`; `auto_train_progress.json` contains the combined history. Search curves describe the original experiment, not a subsequent refit.

## Plot a Learning Curve

Choose a metric present in `history`, such as `loss`, `auroc`, or `mae`. Missing values remain gaps, not zeros:

```python
import matplotlib.pyplot as plt

metric = "loss"
columns = [c for c in (f"train_{metric}", f"val_{metric}") if c in history]
values = history[columns].apply(pd.to_numeric, errors="coerce")
values = values.replace([float("inf"), float("-inf")], float("nan"))
values = values.dropna(axis=1, how="all")
if values.empty:
    raise ValueError("No finite values for this metric. Check metrics and evaluation logs.")
values.plot(marker="o", ylabel=metric, title=selected_trial_dir.name, grid=True)
plt.show()
```

Training-batch and validation metrics can use different sampling conditions. A gap alone does not prove overfitting.

## Interpret the Curves

| Pattern                                | Check, then act                                                                                                                          |
| -------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------- |
| Training improves; validation worsens  | Check the selected checkpoint and stopping settings. If the gap persists, test stronger regularization or less capacity.                 |
| Both remain poor                       | Check labels, baseline, encoders, relationships, and available history before increasing capacity or sampling.                           |
| Loss is flat or nonfinite              | Check labels, target scale, and logs. Fix invalid data first; for finite but unstable optimization, test a lower `optimization.base_lr`. |
| Validation oscillates                  | Check sample and class counts and evaluation limits before changing optimization.                                                        |
| Validation improves until the run ends | Check the stopping reason. If the epoch cap ended useful training, test a larger `optimization.max_epochs`.                              |

### Run a Controlled Comparison

Change one setting, keeping the training table, snapshot, splits, evaluation limits, and environment fixed. Use the [Model Plan Guide](/kumo-engine/1.0.0/user-code-guide/train-a-model/model-plan-guide#optimization) for supported settings. Retain the resolved plan, package versions, seed, validation score, and runtime. For a seed-only repeat, fix all candidate lists to the selected settings first; changing the seed can otherwise select a different configuration. Select on validation, then evaluate on test.

## Training Did Not Complete

A failed fit may not return `fit_result`. Start with the exception and last logged stage, and preserve scratch before restarting.

| Signal                             | Next action                                                                                                                                                                            |
| ---------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| Failure before the first epoch     | Resolve the logged source-access, schema, preparation, or encoder error before tuning.                                                                                                 |
| CUDA out of memory                 | Check concurrent GPU use. For training-batch failures, test a smaller `optimization.batch_size`.                                                                                       |
| Process killed or kernel restarted | Check host RAM and termination logs; this does not establish GPU exhaustion.                                                                                                           |
| No space left on device            | Check scratch bytes and inodes. Retain needed artifacts before cleanup.                                                                                                                |
| No new epochs                      | Check the active stage and CPU, GPU, and storage activity. Observe process health and resource counters over a recorded interval; unchanged progress JSON alone does not prove a hang. |

Measure total fit time, not only epoch time: preprocessing, multiple experiments, refits, evaluation, and explanations also take time. Compare timing under similar cache state and concurrent load. Review [Training Job Plan](/kumo-engine/1.0.0/user-code-guide/train-a-model/model-plan-guide#training-job-plan) before reducing work.

## Preserve the Selected History

Retain the complete `automl_dir` separately from the model bundle. It can contain checkpoints and source-derived values, so restrict access and allow space for the copy.

On Linux, use a filesystem that survives the machine. On Databricks, use a restricted Unity Catalog volume:

```python
import shutil

# Set a new destination on retained storage; copytree rejects an existing path.
retained_run = Path("/path/to/retained-training-runs/run-001")
shutil.copytree(automl_dir, retained_run)
```

<details>
  <summary>
    Archive history from a Snowflake notebook
  </summary>

  Reuse `session` and `SCRATCH_DIR` from [Snowflake Notebook Setup](/kumo-engine/1.0.0/get-started-with-kumo-engine/installation/snowflake-notebook#configure-working-storage). Set `ARCHIVE_STAGE` to an existing writable restricted internal stage:

  ```python
  import shutil
  import tempfile
  from uuid import uuid4

  ARCHIVE_STAGE = "@<database>.<schema>.<stage>/training-runs"
  if "<" in ARCHIVE_STAGE:
      raise ValueError("Set ARCHIVE_STAGE to an existing writable internal stage.")

  scratch_root = Path(SCRATCH_DIR)
  archive_dir = Path(tempfile.mkdtemp(prefix="kumo-archive-", dir=scratch_root))
  archive_path = Path(shutil.make_archive(
      str(archive_dir / f"training-{uuid4().hex}"),
      "zip",
      root_dir=automl_dir,
  ))
  uploads = session.file.put(
      f"file://{archive_path}",
      ARCHIVE_STAGE,
      overwrite=False,
      auto_compress=False,
  )
  if not uploads or any(item.status != "UPLOADED" for item in uploads):
      raise RuntimeError("Archive upload did not complete. Local files remain.")

  SAVED_ARCHIVE = f"{ARCHIVE_STAGE.rstrip('/')}/{archive_path.name}"
  print("Record archive path:", SAVED_ARCHIVE)
  ```

  Record `SAVED_ARCHIVE` with the model version. This archive is separate from Snowflake Model Registry publication.
</details>

## Find Model Explanation Files

The `analyze/` directory describes AutoML search, not feature attribution. Use [Model Explainability](/kumo-engine/1.0.0/user-code-guide/train-a-model/model-explainability) for supported column scores and sampled graph context.

## Prepare a Support Request

Send the symptom, exact error and surrounding logs, package versions, hardware and scratch capacity, predictive query, snapshot and split details, selected experiment, resolved plan, and split-labelled metrics. Include what changed between attempts and whether holdout rows were truncated.

Share relevant excerpts through your agreed support channel. Remove credentials and sensitive values; retain full artifacts privately rather than attaching them by default.