Improve Model Performance

Use a historical backtest to improve a Kumo Relational workflow without changing the basis of comparison. A backtest asks the model to predict from an earlier point in time, then compares those predictions with outcomes that are already known. Change one factor at a time while keeping the entities, labels, and historical cutoff fixed so that changes in the score reflect the configuration you tested.

Before you begin, load the H&M transactions table into a pandas DataFrame, build its relational graph, and configure access to a Kumo Relational endpoint. The examples reuse those objects throughout the workflow.

1. Define the task and ground truth

This example predicts total sales for each article_id during the seven days after an anchor time. The anchor time is the historical cutoff that separates the data available to the model from the future period being predicted. The evaluation includes only articles with sales during the preceding seven days.

import pandas as pd
def total_price_next_7_days(transactions, anchor_time):
    data = transactions.copy()
    data["t_dat"] = pd.to_datetime(data["t_dat"])
    anchor = pd.Timestamp(anchor_time)
    prior = data["t_dat"].between(
        anchor - pd.Timedelta(days=7), anchor, inclusive="right"
    )
    eligible = data.loc[prior].groupby("article_id")["price"].sum()
    eligible_ids = eligible[eligible > 0].index
    future = data["t_dat"].between(
        anchor, anchor + pd.Timedelta(days=7), inclusive="right"
    )
    return (
        data.loc[future]
        .groupby("article_id")["price"]
        .sum()
        .reindex(eligible_ids, fill_value=0.0)
        .rename("actual_value")
        .reset_index()
    )
BACKTEST_ANCHOR_TIME = pd.Timestamp("2020-09-07T00:00:00Z")
labels = total_price_next_7_days(transactions, BACKTEST_ANCHOR_TIME)

The resulting labels DataFrame contains one known outcome, actual_value, for each eligible article_id. Verify that the interval boundaries match your business definition before comparing model configurations.

2. Align PQL with the evaluation population

Store the Predictive Query Language (PQL) query in query so the same prediction task is used for every configuration:

query = """
PREDICT SUM(transactions.price, 0, 7, days)
FOR EACH article.article_id
WHERE SUM(transactions.price, -7, 0, days) > 0
"""

The WHERE clause applies the same eligibility rule used to construct labels, so the query and ground truth cover the same population.

3. Run a key-aligned backtest

This backtest uses mean absolute error (MAE), the average absolute difference between each prediction and its known value. Lower MAE indicates better predictions for this task.

import pandas as pd
from sklearn.metrics import mean_absolute_error
from kumo_relational_client import RelationalClient
def score_predictions(model, labels, query, run_mode="fast", **settings):
    predictions = model.predict(
        query,
        indices=labels["article_id"].tolist(),
        run_mode=run_mode,
        anchor_time=BACKTEST_ANCHOR_TIME,
        **settings,
    )
    scored = labels.merge(
        predictions[["ENTITY", "PREDICTION"]],
        left_on="article_id",
        right_on="ENTITY",
        how="inner",
        validate="one_to_one",
    )
    if len(scored) != len(labels):
        raise ValueError("Some labeled articles did not receive a prediction")
    return mean_absolute_error(scored["actual_value"], scored["PREDICTION"])
with RelationalClient(url="http://localhost:8000") as client:
    model = client.relational(graph)
    baseline_mae = score_predictions(
        model, labels, query, run_mode="fast"
    )

The merge joins predictions to labels by entity key instead of row position. This prevents a silent scoring error if the service returns predictions in a different order from the labels.

4. Tune one factor at a time

Compare configurations in the following order while keeping the evaluation set fixed. The order starts with the task definition, which often matters more than inference settings:

1. Align the PQL target and filters with the production population.
2. Compare the fast, normal, and best run modes, which are presets for the amount of prediction context and the default relational neighborhood size.
3. Test task-relevant neighborhood settings as direct predict() arguments to control how many related rows Kumo Relational samples through table relationships.
4. Test use_prediction_time only when the calendar timestamp itself could provide a useful signal, such as seasonality.
5. Remove noisy columns or add well-justified relational signals.

Do not assume that a larger run mode or neighborhood will always improve accuracy. Measure latency, request size, and the selected task metric on representative data.

Control the context examples

Use predict_task() when you need direct control over the labeled examples supplied to Kumo Relational. These examples provide context that the model uses to make a prediction. Controlling them lets you test how context selection affects prediction quality without changing the graph or prediction task.

When comparing configurations:

• Keep the context and prediction entities fixed.
• Use labels that would have been available at prediction time.
• Compare the custom-context result with the standard PQL-based workflow.
• Do not assume manually selected context will perform better. Evaluate it using the same metric and historical anchor times.

5. Report reproducible results

Record the information needed to reproduce and compare each result:

• Dataset and code revisions
• Entity set and anchor times
• Label construction
• SDK and NVIDIA Inference Microservice (NIM) versions
• Query and prediction settings
• Evaluation metric

The exact H&M measurements previously shown on this page did not have a primary experiment artifact in this repository, so they have been removed rather than presented as reproducible benchmarks.