Batch Prediction | NVIDIA Structured Data and Graph Models

Batch prediction lets you score a list of Kumo Relational entities, such as customers, with one predict() call. Pass the target entity IDs through indices. For lists above 1,000 IDs, set batch_size so the client splits the IDs into bounded batches and can retry failed batches.

Method	Scale	Use case
Explicit entity in Predictive Query Language (PQL)	1	Query validation and debugging
`indices` without batching	Up to 1,000	Small and medium scoring jobs
`batch_size`	More than 1,000 total	Larger scoring jobs with automatic batching

“More than 1,000 total” does not mean that individual batches are unbounded. Total duration, NVIDIA NIM inference capacity, connector sampling cost, client memory, and result storage still constrain the job.

The examples assume that graph is an initialized relational graph and customer_ids contains values from customers.customer_id.

Submit one batch

from kumo_relational_client import RelationalClient
with RelationalClient(url="http://localhost:8000", max_retries=3) as client:
    result = client.relational(graph).predict(
        "PREDICT COUNT(orders.*, 0, 30, days) "
        "FOR EACH customers.customer_id",
        indices=customer_ids[:1000],
        run_mode="fast",
    )

Process a larger list

with RelationalClient(url="http://localhost:8000") as client:
    result = client.relational(graph).predict(
        "PREDICT COUNT(orders.*, 0, 30, days) "
        "FOR EACH customers.customer_id",
        indices=customer_ids,
        run_mode="fast",
        batch_size="max",
        num_retries=3,
    )

Set batch_size to a positive integer to specify an explicit batch size. The value "max" uses the maximum batch size for the task: 1,000 entities for standard prediction tasks and 200 entities for link-prediction tasks. The num_retries parameter controls how many times predict() retries each failed batch.

For a multi-batch job, the SDK can upload the related data used for prediction once and reuse the resulting session instead of sending the same relational context with every batch. Set KUMO_RELATIONAL_DISABLE_SESSIONS to 1, true, yes, or on (case-insensitive) to force stateless per-batch uploads without changing prediction semantics. Setting random_seed=None also disables session reuse because each unseeded batch resamples its neighborhood.

For production jobs, save the prediction settings in persistent storage so you can reproduce a run. A retry after a timeout can run inference again, so downstream writes should be idempotent: writing the same prediction more than once must not create duplicates or change the result. Key those writes by ENTITY plus ANCHOR_TIMESTAMP when that column is present.