Diagnose and Improve Training

Identify a model quality or execution problem, choose one change, and check whether it helps on unchanged validation data. These examples use Kumo Engine in Python and the completed fit_result from Training. Detailed history requires the original scratch files. If training failed, start with Training Did Not Complete.

Start with the Symptom

Check the Evaluation Before Tuning

• Verify the target, population, time window, and label counts in each split. For temporal predictions, features must be available at prediction time. Unexpectedly high scores warrant checking duplicates and target-derived columns.
• Compare against a simple baseline on the same rows. For binary classification, check positive-class prevalence and precision/recall at the intended threshold. For ranking, compare the same cutoff; for regression, inspect errors in the target’s units.
• Use validation to choose settings and test data for the final assessment. If test errors guide further tuning, use a fresh untouched set for the final estimate.

Check Features and Graph Coverage

Before increasing model capacity, check whether useful information reaches the model:

• Compare label counts, timestamp ranges, and feature availability across training and validation. Inspect fitted encoders for missing or excluded features.
• Measure foreign-key match rates and related-record counts at each prediction time. Report the fraction of entities with no eligible history. A relationship that matches in the full source can still provide no usable history after temporal filtering.
• Compare eligible history with sampled graph context, when available. Increase neighbor sampling only when useful records are being omitted; it cannot repair missing relationships.

As a control, compare with a simple model using prediction-time-available aggregates on the same validation rows. If that control has signal and Kumo does not, investigate feature encoding, relationships, and sampling. If neither has signal, revisit the target and population before increasing capacity.

Check the Decision Threshold

For binary classification, compare average precision (AP) with positive-class prevalence on the same population. A high AUROC can coexist with poor precision at your alert budget. Measure precision, recall, and alert count at the intended threshold, or within the highest-scoring rows your team can review.

Choose the threshold or review budget using complete, aligned validation labels and positive-class scores, then freeze it before test evaluation. holdout_df() contains test rows and is not a threshold-tuning dataset. If no rows trigger an alert, precision is undefined, not perfect. Ranking quality also does not establish calibrated probabilities; compare predicted risk with observed positive rates before using scores as risk estimates.

Know Which Evidence You Have

Publishing a model does not retain the last two rows. Preserve needed files before scratch is removed; a notebook shutdown or machine replacement can delete them.

Read Retained Model Metrics

In the training process:

print(fit_result.metrics)

In a later process, read metrics.json from the complete retained bundle:

import json
from pathlib import Path
bundle = Path("/path/to/retained-models/model-v1")
metrics = json.loads((bundle / "metrics.json").read_text())
print(metrics)

A full-data refit trains on the test split and has no independent test evaluation. Missing metrics are not zeros; check configured metrics, evaluated splits, and warnings about undefined values.

Read Holdout Predictions

These are test predictions from the train-and-validation refit when run, otherwise from the original fit. Use them to describe errors, not to repeatedly tune settings or thresholds.

if fit_result.holdout_path is None:
    raise FileNotFoundError("This fit has no saved holdout predictions.")
print("Truncated:", fit_result.holdout_truncated)
holdout = fit_result.holdout_df()
print(holdout.columns.tolist())
print(holdout.head())

holdout_df() reads the entire saved frame into memory and fails if scratch was deleted. The saved frame is capped at 1 GiB in memory. holdout_truncated=True means rows were omitted; None means completeness is unknown. Do not report truncated-frame metrics as full-holdout metrics. Link prediction does not produce this artifact.

For segment analysis, join attributes on the complete example key, including prediction time where applicable. Verify that the join preserves row count, and report segment size and class counts alongside scores.

To retain the rows, choose a new path on durable storage:

from pathlib import Path
output = Path("/path/to/retained-results/run-001/holdout.parquet")
output.parent.mkdir(parents=True, exist_ok=True)
if output.exists():
    raise FileExistsError(output)
holdout.to_parquet(output, index=False)

Select the Completed Fit

Continue in the process that produced fit_result. Its bundle identifies the exact fit run, even when scratch contains several trainers and repeated fits.

from pathlib import Path
import json
run_dir = Path(fit_result.bundle_dir).parent
automl_dirs = sorted({
    path.parent
    for path in run_dir.rglob("auto_train_progress.json")
    if path.is_file()
})
if len(automl_dirs) != 1:
    raise RuntimeError(
        f"Expected one AutoML directory for this fit, found {len(automl_dirs)}. "
        f"Inspect {run_dir} before selecting a run."
    )
automl_dir = automl_dirs[0]
selection = json.loads(
    (automl_dir / "best_trainer.json").read_text(encoding="utf-8")
)
selected_trial_name = Path(selection["best_trainer"]).name
selected_trial_dir = automl_dir / selected_trial_name
if (
    not selected_trial_name.startswith("version_")
    or not selected_trial_dir.is_dir()
):
    raise RuntimeError("The selected experiment directory is unavailable.")
print("Fit run:", run_dir)
print("AutoML directory:", automl_dir)
print("Selected experiment:", selected_trial_dir)

selected_trial_dir identifies the original winning search experiment. Its curves describe that experiment, not a subsequent refit. Use the bundle artifacts and metric prefixes to distinguish the final model from search history.

Record these paths before the Python objects disappear. If you inspect retained scratch in a later process, set automl_dir to the exact recorded directory and resolve selected_trial_dir from its best_trainer.json. Do not select a run only because it was modified most recently.

Inspect the Trained Configuration

Read the selected fit’s artifacts, not trainer.model_plan, which controls the next fit and may have changed:

for role in ("resolved_model_plan", "trainer_config", "encoders", "target_encoder"):
    path = Path(fit_result.artifact_paths[role])
    print(role, path)
    print(path.read_text()[:1500])

Open the files for complete values. resolved_model_plan can retain candidate lists; trainer_config records the concrete settings selected for the trainer. Check the tuning metric, evaluation step limits, refit settings, and fitted feature encoders. Change future settings through the Model Plan Guide, not by editing artifact files.

Read Lineage in a Later Process

Use a complete retained bundle from a trusted publisher, such as the copy created in Publish and Retrieve Models. The publication manifest maps artifact roles to paths relative to that bundle:

from pathlib import Path
import json
bundle = Path("/path/to/retained-models/model-v1")
manifest = json.loads(
    (bundle / "publication-manifest.json").read_text(encoding="utf-8")
)
artifacts = {
    entry["role"]: bundle / entry["relative_path"]
    for entry in manifest["artifacts"]
}
lineage = json.loads(artifacts["training_spec"].read_text(encoding="utf-8"))
print("Predictive query:", lineage["predictive_query"])
print("Training snapshot:", lineage["snapshot_id"])
print("Generation settings:", lineage["training_table_generation"])
print("Resolved plan:", artifacts["resolved_model_plan"])
print("Selected configuration:", artifacts["trainer_config"])
print("Feature encoders:", artifacts["encoders"])
print("Target encoder:", artifacts["target_encoder"])

The snapshot identifier records provenance; the bundle does not contain the graph data. Use Publish and Retrieve Models for validated loading.

Load Epoch and Experiment Metrics

Continue with selected_trial_dir from Select the Completed Fit. This loads one experiment’s epoch history and final validation results; install pandas and Matplotlib if needed.

import pandas as pd
progress = json.loads((selected_trial_dir / "train_progress.json").read_text())
rows = []
for epoch, values in (progress.get("metrics") or {}).items():
    values = values or {}
    rows.append({
        "epoch": int(epoch),
        **(values.get("train_metrics") or {}),
        **(values.get("validation_metrics") or {}),
    })
if not rows:
    raise ValueError("No recorded epochs. Check this experiment's training logs.")
history = pd.DataFrame(rows).sort_values("epoch").set_index("epoch")
print(history)
print(pd.read_csv(selected_trial_dir / "val_result.csv"))

To compare experiments, repeat for another recorded version_N directory. Compare the tuning metric in that experiment’s model_plan.json; auto_train_progress.json contains the combined history. Search curves describe the original experiment, not a subsequent refit.

Plot a Learning Curve

Choose a metric present in history, such as loss, auroc, or mae. Missing values remain gaps, not zeros:

import matplotlib.pyplot as plt
metric = "loss"
columns = [c for c in (f"train_{metric}", f"val_{metric}") if c in history]
values = history[columns].apply(pd.to_numeric, errors="coerce")
values = values.replace([float("inf"), float("-inf")], float("nan"))
values = values.dropna(axis=1, how="all")
if values.empty:
    raise ValueError("No finite values for this metric. Check metrics and evaluation logs.")
values.plot(marker="o", ylabel=metric, title=selected_trial_dir.name, grid=True)
plt.show()

Training-batch and validation metrics can use different sampling conditions. A gap alone does not prove overfitting.

Interpret the Curves

Run a Controlled Comparison

Change one setting, keeping the training table, snapshot, splits, evaluation limits, and environment fixed. Use the Model Plan Guide for supported settings. Retain the resolved plan, package versions, seed, validation score, and runtime. For a seed-only repeat, fix all candidate lists to the selected settings first; changing the seed can otherwise select a different configuration. Select on validation, then evaluate on test.

Training Did Not Complete

A failed fit may not return fit_result. Start with the exception and last logged stage, and preserve scratch before restarting.

Measure total fit time, not only epoch time: preprocessing, multiple experiments, refits, evaluation, and explanations also take time. Compare timing under similar cache state and concurrent load. Review Training Job Plan before reducing work.

Preserve the Selected History

Retain the complete automl_dir separately from the model bundle. It can contain checkpoints and source-derived values, so restrict access and allow space for the copy.

On Linux, use a filesystem that survives the machine. On Databricks, use a restricted Unity Catalog volume:

import shutil
# Set a new destination on retained storage; copytree rejects an existing path.
retained_run = Path("/path/to/retained-training-runs/run-001")
shutil.copytree(automl_dir, retained_run)

import shutil
import tempfile
from uuid import uuid4
ARCHIVE_STAGE = "@<database>.<schema>.<stage>/training-runs"
if "<" in ARCHIVE_STAGE:
    raise ValueError("Set ARCHIVE_STAGE to an existing writable internal stage.")
scratch_root = Path(SCRATCH_DIR)
archive_dir = Path(tempfile.mkdtemp(prefix="kumo-archive-", dir=scratch_root))
archive_path = Path(shutil.make_archive(
    str(archive_dir / f"training-{uuid4().hex}"),
    "zip",
    root_dir=automl_dir,
))
uploads = session.file.put(
    f"file://{archive_path}",
    ARCHIVE_STAGE,
    overwrite=False,
    auto_compress=False,
)
if not uploads or any(item.status != "UPLOADED" for item in uploads):
    raise RuntimeError("Archive upload did not complete. Local files remain.")
SAVED_ARCHIVE = f"{ARCHIVE_STAGE.rstrip('/')}/{archive_path.name}"
print("Record archive path:", SAVED_ARCHIVE)

Find Model Explanation Files

The analyze/ directory describes AutoML search, not feature attribution. Use Model Explainability for supported column scores and sampled graph context.

Prepare a Support Request

Send the symptom, exact error and surrounding logs, package versions, hardware and scratch capacity, predictive query, snapshot and split details, selected experiment, resolved plan, and split-labelled metrics. Include what changed between attempts and whether holdout rows were truncated.

Share relevant excerpts through your agreed support channel. Remove credentials and sensitive values; retain full artifacts privately rather than attaching them by default.