Skip to navigation

Build a Schema

View as Markdown

A Schema combines table metadata with the relationships used to sample predictive context. Build it after preparing the source data and physical column types.

Build a schema from DataFrames

Using the prepared customers, orders, and products DataFrames from Data Requirements:

from sdm_client import Schema
schema = Schema.from_data({
"customers": customers,
"orders": orders,
"products": products,
})

Schema.from_data() builds the schema and infers table metadata and relationships. Treat this inference as a starting point; the following pages explain how to inspect and correct the inferred metadata and relationships. Every table given to from_data() is read into client memory.

Override inference at build time

schema = Schema.from_data(
{"customers": customers, "orders": orders, "products": products},
primary_keys={"customers": "customer_id"},
time_columns={"orders": "order_time"},
relationships=None, # default: infer relationships
)

Pass relationships=[] to build a schema with no relationships and suppress relationship inference. Pass an explicit list of Relationship objects to declare relationships yourself instead of relying on name-matching inference — see Define Relationships.

Schemas over tables that stay in place

For tables too large to load into memory, or to avoid reading a whole warehouse table for a small prediction, describe them with Schema.from_source() instead. No table is read in full, but from_source() is not lazy: building the schema already queries the source for column metadata and a preview of up to 1,000 rows per table (used to infer semantic types), and — unless check_keys=False — a key-uniqueness check that scans each table once. materialize() is the step that samples the rows an actual prediction needs.

from sdm_client import PredictiveQuery, Schema
from sdm_connectors import DatabricksSource
from sdm_connectors.databricks import warehouse
connection = warehouse(warehouse_id, workspace=workspace)
source = DatabricksSource(connection, catalog="main", schema="sales")
schema = Schema.from_source(
source,
{"users": "crm_users", "orders": "shop_orders"},
time_columns={"orders": "ordered_at"},
)
request = PredictiveQuery(
"PREDICT COUNT(orders.*, 0, 30, days) FOR EACH users.user_id", schema
).materialize(indices=[7, 12, 3])

Schema.from_source() takes a Source — DatabricksSource, SnowflakeSource, PostgresSource, DuckDBSource, or SQLiteSource — built from that backend’s own connection. See the connector pages under Connect Data Sources for each. tables maps the names your query uses to the names in the source; leave it out to describe every table the source holds. primary_keys, time_columns, and relationships mean what they mean on from_data(); time_columns={"table": None} says a table has no time column, on either builder.

A schema’s tables all come from one source — a schema cannot mix an in-memory table with a sampled one, or combine two different warehouses. Where tables really do live in two systems, join them at the platform level first (for example, a Databricks catalog that federates Snowflake or PostgreSQL) so one source sees all of them.

check_keys=True is the default for from_source(): each declared or inferred key is checked against every row in the source, one aggregate query per table. Pass check_keys=False to skip that scan when the cost matters and the key is already trusted. A key a database itself enforces, such as a PostgreSQL or DuckDB primary key, skips the scan automatically.

Semantic models (metric views and semantic views)

A Databricks metric view or Snowflake semantic view already states its tables, joins, and columns. Convert it like any other source:

schema = Schema.from_source(source.semantic_model("sales_metrics"))

Only DatabricksSource and SnowflakeSource offer semantic_model(). Its expressions run as SQL under your own credentials, so by default only pure, deterministic built-in functions are allowed; pass trust_expressions=True to keep an expression that calls something else, when you trust whoever wrote the model. schema.conversion_notes records what the conversion resolved — read it, but do not depend on its exact wording.

Next, configure the inferred table metadata.