> This page is for Structured Data Models Client.

> For clean Markdown of any page, append .md to the page URL.
> For a complete documentation index, see https://docs.nvidia.com/sdgm/llms.txt.
> For AI client integration (Claude Code, Cursor, etc.), connect to the MCP server at https://docs.nvidia.com/sdgm/_mcp/server.

# Batch Prediction

> Score large Kumo Relational entity lists — what sdm-client batches for you, and what you still chunk yourself

`materialize()` gathers more entities than one call scores as several calls automatically — you do not split `indices` into chunks yourself for an ordinary scoring job.

```python
from sdm_client import PredictiveQuery, SDMClient, KumoRelational

query = PredictiveQuery(
    "PREDICT COUNT(orders.*, 0, 30, days) FOR EACH customers.customer_id",
    schema,
)

with SDMClient(url="http://localhost:8000", max_retries=3) as client:
    request = query.materialize(indices=customer_ids)   # any size up to 100,000
    result = client.predict(request, model=KumoRelational())
```

`materialize()` builds one batch per 10,000 entities (200 for a temporal link prediction task), each with its own sampled context. `predict()` sends one call per batch and merges the answers into one `PredictionResult`. `row_index` counts across the whole request, so `to_pandas()`, which orders by it, gives the entities back in the order you asked for them — not interleaved by batch.

## What is still a size limit

| Limit                                | Value                                                                                              | What happens past it                                                           |
| ------------------------------------ | -------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------ |
| Entities in one `materialize()` call | 100,000, in at most 100 calls of up to 10,000 (or 20,000 at 200/call for temporal link prediction) | Refused before anything is sampled, naming which ceiling you hit               |
| Request body                         | 30 MiB per call, 16 MB on Databricks Model Serving                                                 | Refused at `materialize()` (30 MiB) or on the first call (16 MB gateway limit) |

`materialize()` builds every batch, each with its own context, before the first is sent — the whole population is held in memory at once, which is why the ceiling exists; 100,000 entities is roughly a gigabyte of it.

**For more than 100,000 entities, chunk `indices` yourself** and call `materialize()`/`predict()` once per chunk:

```python
import itertools

def chunks(items, size):
    it = iter(items)
    while chunk := list(itertools.islice(it, size)):
        yield chunk

results = []
with SDMClient(url="http://localhost:8000", max_retries=3) as client:
    for chunk in chunks(customer_ids, 100_000):
        request = query.materialize(indices=chunk)
        results.append(client.predict(request, model=KumoRelational()))
```

## Tune the per-call size with `rows_per_call`

`rows_per_call` lowers the per-call size below the endpoint's own cap — it never raises it. Reach for it when a wide graph's context pushes a batch over the 30 MiB body, or when your deployment's own cap is lower than this client's default:

```python
request = query.materialize(indices=customer_ids, rows_per_call=1000)
```

It is not the fix for every body-size refusal: a batch carries its context as well as its rows, and where the context alone is already over the limit, no batch size is small enough. There, reduce `num_context_samples`, `num_neighbors`, or the columns the schema carries instead.

## Ranking and forecasting are not batched the same way

A `RANK TOP K` query makes one call per candidate, and a forecast one per step — bounded separately by `sdm_client.decompose.MOST_PARTS` (200), not by the entity limits above. More entities than one call scores is refused outright for either, before anything is gathered, because a ranking can't be batched: a candidate reported unscored applies to the whole ranking, not one batch, and a batch whose own context held only one class would wrongly drop it from every row. See [Prediction Types](/sdgm/kumo-relational/user-guide/make-predictions/prediction-types#recommendations-ranking) and [Forecasting](/sdgm/kumo-relational/user-guide/make-predictions/forecasting).

## A failed batch keeps what it already paid for

If a call in the middle of a batched request fails, the ones that already answered are not discarded. The error carries them as `details['answered']` and the rows they cover as `details['rows_answered']` (or `details['parts_answered']`, by step, for a forecast), so you can retry only what's left instead of paying for the whole population again.

## Sessions and batching

`fit()` takes a request of a **single** call — a session pins one context, and a batched request has one context per batch. Use a small initial request to create the session, then score a larger population against the handle.

`materialize(indices=...)` selects prediction rows; it samples labeled context separately from the schema. Even when the initial and later prediction ID lists are disjoint, later prediction rows can still appear in that sampled context with their answers attached.

Before creating the session, identify every customer it may score and pass those IDs as `holdout` to `fit()`. This removes matching rows from the context before it is pinned:

```python
small_request = query.materialize(indices=labeled_customer_ids[:100])
with SDMClient(url="http://localhost:8000") as client:
    with client.fit(
        small_request,
        model=KumoRelational(),
        holdout=customer_ids_to_score,
    ) as fitted:
        big_request = query.materialize(indices=customer_ids_to_score)
        result = fitted.predict(big_request)
```

Pass the full future scoring population, not just IDs that overlap the initial request. Omit `holdout` only if you have independently established that all future scoring rows are excluded from the pinned context.

Keep enough other labeled context after exclusion. If no usable context remains, prepare a separate labeled context or use ordinary `client.predict()` calls instead of a fitted session. See [Holdout](/sdgm/kumo-relational/reference/glossary#h) for the exclusion rule and [Python SDK API](/sdgm/kumo-relational/reference/python-api-reference) for the session interface.

## Production jobs

`SDMClient(max_retries=3)` retries a transient transport failure (429, 502, 503, 504, or a dropped connection) automatically, per call. It does not retry 500, since the endpoint can use that status for a request or model error rather than a transient failure. Record the prediction settings in durable state for a production job: a retry after a timeout can run inference again, so downstream writes should be idempotent and keyed by `id` (or `row_index` when there is no entity key).