Amazon S3

Use the S3 connector to read a flat CSV or Parquet object into a pandas DataFrame. Then pass the DataFrame to Schema.from_data() with the other tables in the schema.

Unlike the SQL connectors, S3 has no Source class and no schema catalog: each object is read explicitly into memory, one table at a time.

Installation

pip install "structured-data-models-client[relational,s3]"

Read an object

from sdm_connectors import read
orders = read("s3", path="s3://my-bucket/warehouse/orders.parquet")

The connector supports:

• CSV files with .csv or .txt suffixes, optionally compressed;
• Parquet files with .parquet, .pq, or .parq suffixes; and
• directories containing a Parquet dataset.

For an object without a recognized suffix, pass format="csv" or format="parquet" explicitly.

Configure storage access

Authentication and filesystem options are passed to pandas through storage_options:

orders = read(
    "s3",
    path="s3://my-bucket/warehouse/orders.parquet",
    storage_options={
        "key": "<access-key-id>",
        "secret": "<secret-access-key>",
    },
)

Do not store credentials in source code. Prefer the ambient AWS credential chain, an instance or workload role, or values loaded from your platform’s secret manager. Omit storage_options when the runtime is already configured to authenticate to S3.

Build a schema

Read each table and build the schema from the resulting DataFrames:

from sdm_connectors import read
from sdm_client import Schema
customers = read("s3", path="s3://my-bucket/customers.parquet")
orders = read("s3", path="s3://my-bucket/orders.parquet")
schema = Schema.from_data({"customers": customers, "orders": orders})

Review the inferred metadata and relationships before prediction. Reading an S3 object materializes that complete object into client memory.

Troubleshooting

• MissingBackendError: Install structured-data-models-client[s3] in the active environment.
• INVALID_CONNECTOR_ARGS: Confirm that path is an s3:// URI and the file has a supported suffix, or pass format explicitly.
• NOT_FOUND: Confirm the bucket, object key, region, and credentials.
• READ_FAILED: Verify the object format and the permissions granted to the active AWS identity.

See Build a Schema to configure and validate the resulting tables.