Improve Model Performance

Improve a Kumo Relational workflow by changing one factor at a time and evaluating each configuration on the same entities, labels, and historical anchor times.

1. Define the task and ground truth

This example predicts total sales for each article_id during the seven days after an anchor time. It includes only articles with sales during the preceding seven days.

import pandas as pd
def total_price_next_7_days(transactions, anchor_time):
    data = transactions.copy()
    data["t_dat"] = pd.to_datetime(data["t_dat"])
    anchor = pd.Timestamp(anchor_time)
    prior = data["t_dat"].between(
        anchor - pd.Timedelta(days=7), anchor, inclusive="right"
    )
    eligible = data.loc[prior].groupby("article_id")["price"].sum()
    eligible_ids = eligible[eligible > 0].index
    future = data["t_dat"].between(
        anchor, anchor + pd.Timedelta(days=7), inclusive="right"
    )
    return (
        data.loc[future]
        .groupby("article_id")["price"]
        .sum()
        .reindex(eligible_ids, fill_value=0.0)
        .rename("actual_value")
        .reset_index()
    )

Verify that the interval boundaries match the business definition before comparing model configurations.

2. Align PQL with the evaluation population

PREDICT SUM(transactions.price, 0, 7, days)
FOR EACH article.article_id
WHERE SUM(transactions.price, -7, 0, days) > 0

The filter keeps the in-context population aligned with the entities used to construct labels.

3. Run a key-aligned backtest

import pandas as pd
from sklearn.metrics import mean_absolute_error
from sdm_client import PredictiveQuery, SDMClient, KumoRelational
def score_predictions(client, schema, query_text, labels, **materialize_kwargs):
    query = PredictiveQuery(query_text, schema)
    request = query.materialize(
        indices=labels["article_id"].tolist(),
        anchor_time=pd.Timestamp("2020-09-07T00:00:00Z"),
        **materialize_kwargs,
    )
    result = client.predict(request, model=KumoRelational())
    predictions = result.to_pandas()
    scored = labels.merge(
        predictions[["id", "prediction"]],
        left_on="article_id",
        right_on="id",
        how="inner",
        validate="one_to_one",
    )
    if len(scored) != len(labels):
        raise ValueError("Some labeled articles did not receive a prediction")
    return mean_absolute_error(scored["actual_value"], scored["prediction"])
with SDMClient(url="http://localhost:8000") as client:
    baseline_mae = score_predictions(
        client, schema, query_text, labels, num_context_samples=1000
    )

Joining by article_id/id prevents a silent metric error when response order differs from label order. If predictions will be summed, score with prediction_statistic="mean" instead of the default median — see Prediction Results.

4. Tune one factor at a time

Compare configurations while keeping the evaluation set fixed:

Align the PQL target and filters with the production population.
Compare different num_context_samples values.
Test task-relevant neighborhood settings (num_neighbors, num_hops) directly on materialize().
Test use_prediction_time only when calendar time can contain a useful signal.
Remove noisy columns or add well-justified relational signals.

Do not assume that a larger context or neighborhood will always improve accuracy. Measure latency, request size, and the selected task metric on representative data.

5. Report reproducible results

Record the dataset revision, entity set, anchor times, label construction, SDK and NIM versions, query, materialize/predict settings, metric, and code revision.